2024-03-21T10:11:42.431 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-21T10:11:42.432 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612235 --owner scheduled_vshankar@teuthology --verbose --name vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} -- /home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612235/orig.config.yaml 2024-03-21T10:11:42.491 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-21T10:11:42.587 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612235 branch: wip-vshankar-testing-20240320.053002 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} email: vshankar@redhat.com first_in_suite: false job_id: '7612235' kernel: client: branch: testing flavor: default kdb: true sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-vshankar-testing-20240320.053002 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: 2edfee68ad7243bed422a049b6f5f58619300f11 subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount dry_run: true install: ceph: flavor: default sha1: 2edfee68ad7243bed422a049b6f5f58619300f11 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20240320.053002 sha1: 2edfee68ad7243bed422a049b6f5f58619300f11 owner: scheduled_vshankar@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 8073 sha1: 2edfee68ad7243bed422a049b6f5f58619300f11 sleep_before_teardown: 0 subset: 203/1024 suite: fs suite_branch: wip-vshankar-testing-20240320.053002 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_2edfee68ad7243bed422a049b6f5f58619300f11/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 2edfee68ad7243bed422a049b6f5f58619300f11 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi031.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAHtE/wE/RvnJ0NC6xiejpORRY5Lgaz/LyoHHQv/b5T0xb1IbwCQ6a4X06dQtR/RLGmqaliachMDeain5quaeyo= smithi040.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHNbw+RtJDgDGG+sGMwMbSkwF8iXd5VS40DfjBQeH1mxRs+wPxILdy8NbI9qL1xHjRxQkukrrLQQMh1SiHejrEs= smithi070.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHY8qxMgdgd6aVrQl/VWAAFd1U+hRQgCkw1lEhtjt/nL77gWzBfZsYNKriRLWt9xAUs+INvo0t5wbKBh0ArgyR0= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/iogen.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-20_11:21:18 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-21T10:11:42.587 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_2edfee68ad7243bed422a049b6f5f58619300f11/qa; will attempt to use it 2024-03-21T10:11:42.588 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_2edfee68ad7243bed422a049b6f5f58619300f11/qa/tasks 2024-03-21T10:11:42.588 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-21T10:11:42.590 INFO:teuthology.task.internal:Checking packages... 2024-03-21T10:11:42.611 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '2edfee68ad7243bed422a049b6f5f58619300f11' 2024-03-21T10:11:42.611 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-21T10:11:42.612 INFO:teuthology.packaging:ref: None 2024-03-21T10:11:42.612 INFO:teuthology.packaging:tag: None 2024-03-21T10:11:42.612 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240320.053002 2024-03-21T10:11:42.612 INFO:teuthology.packaging:sha1: 2edfee68ad7243bed422a049b6f5f58619300f11 2024-03-21T10:11:42.612 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-vshankar-testing-20240320.053002 2024-03-21T10:11:42.840 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2231.g2edfee68 2024-03-21T10:11:42.842 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-21T10:11:42.848 INFO:teuthology.task.internal:no buildpackages task found 2024-03-21T10:11:42.848 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-21T10:11:42.855 INFO:teuthology.task.internal:Saving configuration 2024-03-21T10:11:42.876 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-21T10:11:42.881 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-21T10:11:42.903 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612235', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 10:06:48.832840', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAHtE/wE/RvnJ0NC6xiejpORRY5Lgaz/LyoHHQv/b5T0xb1IbwCQ6a4X06dQtR/RLGmqaliachMDeain5quaeyo='} 2024-03-21T10:11:42.931 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612235', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 10:06:48.834260', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHNbw+RtJDgDGG+sGMwMbSkwF8iXd5VS40DfjBQeH1mxRs+wPxILdy8NbI9qL1xHjRxQkukrrLQQMh1SiHejrEs='} 2024-03-21T10:11:42.949 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612235', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 10:06:48.831191', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHY8qxMgdgd6aVrQl/VWAAFd1U+hRQgCkw1lEhtjt/nL77gWzBfZsYNKriRLWt9xAUs+INvo0t5wbKBh0ArgyR0='} 2024-03-21T10:11:42.949 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-21T10:11:42.955 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-21T10:11:42.955 INFO:teuthology.task.internal:roles: ubuntu@smithi040.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-21T10:11:42.955 INFO:teuthology.task.internal:roles: ubuntu@smithi070.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-21T10:11:42.955 INFO:teuthology.run_tasks:Running task console_log... 2024-03-21T10:11:43.063 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe179104160>, signals=[15]) 2024-03-21T10:11:43.063 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-21T10:11:43.069 INFO:teuthology.task.internal:Opening connections... 2024-03-21T10:11:43.069 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com 2024-03-21T10:11:43.071 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T10:11:43.178 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi040.front.sepia.ceph.com 2024-03-21T10:11:43.179 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T10:11:43.279 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi070.front.sepia.ceph.com 2024-03-21T10:11:43.280 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T10:11:43.366 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-21T10:11:43.373 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2024-03-21T10:11:43.391 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2024-03-21T10:11:43.392 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2024-03-21T10:11:43.449 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="CentOS Stream" 2024-03-21T10:11:43.449 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="9" 2024-03-21T10:11:43.449 INFO:teuthology.orchestra.run.smithi031.stdout:ID="centos" 2024-03-21T10:11:43.450 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="rhel fedora" 2024-03-21T10:11:43.450 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="9" 2024-03-21T10:11:43.450 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el9" 2024-03-21T10:11:43.450 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-21T10:11:43.450 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2024-03-21T10:11:43.450 INFO:teuthology.orchestra.run.smithi031.stdout:LOGO="fedora-logo-icon" 2024-03-21T10:11:43.450 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-21T10:11:43.450 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://centos.org/" 2024-03-21T10:11:43.450 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-21T10:11:43.450 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-21T10:11:43.450 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-21T10:11:43.451 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2024-03-21T10:11:43.480 DEBUG:teuthology.orchestra.run.smithi040:> uname -m 2024-03-21T10:11:43.498 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2024-03-21T10:11:43.498 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2024-03-21T10:11:43.556 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="CentOS Stream" 2024-03-21T10:11:43.556 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="9" 2024-03-21T10:11:43.556 INFO:teuthology.orchestra.run.smithi040.stdout:ID="centos" 2024-03-21T10:11:43.556 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE="rhel fedora" 2024-03-21T10:11:43.557 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="9" 2024-03-21T10:11:43.557 INFO:teuthology.orchestra.run.smithi040.stdout:PLATFORM_ID="platform:el9" 2024-03-21T10:11:43.557 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-21T10:11:43.557 INFO:teuthology.orchestra.run.smithi040.stdout:ANSI_COLOR="0;31" 2024-03-21T10:11:43.557 INFO:teuthology.orchestra.run.smithi040.stdout:LOGO="fedora-logo-icon" 2024-03-21T10:11:43.557 INFO:teuthology.orchestra.run.smithi040.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-21T10:11:43.557 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://centos.org/" 2024-03-21T10:11:43.557 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-21T10:11:43.557 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-21T10:11:43.557 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-21T10:11:43.557 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2024-03-21T10:11:43.579 DEBUG:teuthology.orchestra.run.smithi070:> uname -m 2024-03-21T10:11:43.596 INFO:teuthology.orchestra.run.smithi070.stdout:x86_64 2024-03-21T10:11:43.596 DEBUG:teuthology.orchestra.run.smithi070:> cat /etc/os-release 2024-03-21T10:11:43.652 INFO:teuthology.orchestra.run.smithi070.stdout:NAME="CentOS Stream" 2024-03-21T10:11:43.652 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION="9" 2024-03-21T10:11:43.652 INFO:teuthology.orchestra.run.smithi070.stdout:ID="centos" 2024-03-21T10:11:43.652 INFO:teuthology.orchestra.run.smithi070.stdout:ID_LIKE="rhel fedora" 2024-03-21T10:11:43.652 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION_ID="9" 2024-03-21T10:11:43.652 INFO:teuthology.orchestra.run.smithi070.stdout:PLATFORM_ID="platform:el9" 2024-03-21T10:11:43.652 INFO:teuthology.orchestra.run.smithi070.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-21T10:11:43.653 INFO:teuthology.orchestra.run.smithi070.stdout:ANSI_COLOR="0;31" 2024-03-21T10:11:43.653 INFO:teuthology.orchestra.run.smithi070.stdout:LOGO="fedora-logo-icon" 2024-03-21T10:11:43.653 INFO:teuthology.orchestra.run.smithi070.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-21T10:11:43.653 INFO:teuthology.orchestra.run.smithi070.stdout:HOME_URL="https://centos.org/" 2024-03-21T10:11:43.653 INFO:teuthology.orchestra.run.smithi070.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-21T10:11:43.653 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-21T10:11:43.653 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-21T10:11:43.653 INFO:teuthology.lock.ops:Updating smithi070.front.sepia.ceph.com on lock server 2024-03-21T10:11:43.671 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-21T10:11:43.682 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-21T10:11:43.687 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-21T10:11:43.687 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest 2024-03-21T10:11:43.690 DEBUG:teuthology.orchestra.run.smithi040:> test '!' -e /home/ubuntu/cephtest 2024-03-21T10:11:43.692 DEBUG:teuthology.orchestra.run.smithi070:> test '!' -e /home/ubuntu/cephtest 2024-03-21T10:11:43.708 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-21T10:11:43.775 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-21T10:11:43.775 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph) 2024-03-21T10:11:43.777 DEBUG:teuthology.orchestra.run.smithi040:> test -z $(ls -A /var/lib/ceph) 2024-03-21T10:11:43.780 DEBUG:teuthology.orchestra.run.smithi070:> test -z $(ls -A /var/lib/ceph) 2024-03-21T10:11:43.886 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-21T10:11:43.966 INFO:teuthology.run_tasks:Running task kernel... 2024-03-21T10:11:43.980 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'} 2024-03-21T10:11:43.980 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-03-21T10:11:43.980 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-03-21T10:11:43.981 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-21T10:11:43.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T10:11:43.998 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-21T10:11:44.249 DEBUG:teuthology.task.kernel:sha1 for client.0 is 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-21T10:11:44.249 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5ec3e6e88b2e2783e09498588bb1f433ae9c8986"... 2024-03-21T10:11:44.250 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2024-03-21T10:11:44.267 INFO:teuthology.orchestra.run.smithi031.stdout:5.14.0-361.el9.x86_64 2024-03-21T10:11:44.268 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-21T10:11:44.268 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-21T10:11:44.268 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum upgrade -y linux-firmware 2024-03-21T10:11:46.376 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-21T10:11:47.802 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-21T10:11:49.575 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 5.7 MB/s | 8.0 MB 00:01 2024-03-21T10:11:51.610 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-21T10:12:04.876 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 651 kB/s | 6.1 MB 00:09 2024-03-21T10:12:06.610 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-03-21T10:12:07.777 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-21T10:12:13.044 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 49 kB/s | 1.7 kB 00:00 2024-03-21T10:12:15.366 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-21T10:12:15.375 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-21T10:12:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-03-21T10:12:15.376 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-21T10:12:15.376 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrading: 2024-03-21T10:12:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-21T10:12:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:12:15.376 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-03-21T10:12:15.376 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-03-21T10:12:15.376 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrade 1 Package 2024-03-21T10:12:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:12:15.376 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 375 M 2024-03-21T10:12:15.376 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-03-21T10:12:32.643 INFO:teuthology.orchestra.run.smithi031.stdout:linux-firmware-20240219-143.el9.noarch.rpm 22 MB/s | 375 MB 00:17 2024-03-21T10:12:32.644 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-03-21T10:12:32.644 INFO:teuthology.orchestra.run.smithi031.stdout:Total 22 MB/s | 375 MB 00:17 2024-03-21T10:12:32.644 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-03-21T10:12:32.750 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-03-21T10:12:32.750 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-03-21T10:12:35.232 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-03-21T10:12:35.232 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-03-21T10:12:41.840 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-03-21T10:12:42.125 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-21T10:12:42.212 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-21T10:12:43.341 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-21T10:12:43.341 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-21T10:12:43.721 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-21T10:12:43.721 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:12:43.722 INFO:teuthology.orchestra.run.smithi031.stdout:Upgraded: 2024-03-21T10:12:43.722 INFO:teuthology.orchestra.run.smithi031.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-21T10:12:43.722 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:12:43.722 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-21T10:12:44.036 INFO:teuthology.task.kernel:Downloading kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on client.0... 2024-03-21T10:12:44.037 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-21T10:12:44.188 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/ 2024-03-21T10:12:44.188 DEBUG:teuthology.orchestra.run.smithi031:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-21T10:12:44.192 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi031.front.sepia.ceph.com to complete... 2024-03-21T10:13:12.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21 10:13:12 URL:https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm [336309687/336309687] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-21T10:13:12.219 INFO:teuthology.orchestra.run.smithi031.stderr:FINISHED --2024-03-21 10:13:12-- 2024-03-21T10:13:12.219 INFO:teuthology.orchestra.run.smithi031.stderr:Total wall clock time: 28s 2024-03-21T10:13:12.220 INFO:teuthology.orchestra.run.smithi031.stderr:Downloaded: 1 files, 321M in 27s (11.7 MB/s) 2024-03-21T10:13:12.222 INFO:teuthology.task.kernel:Installing kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on client.0... 2024-03-21T10:13:12.222 DEBUG:teuthology.orchestra.run.smithi031:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-21T10:13:14.258 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying... ######################################## 2024-03-21T10:13:14.321 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing... ######################################## 2024-03-21T10:13:14.405 INFO:teuthology.orchestra.run.smithi031.stdout:Updating / installing... 2024-03-21T10:13:22.253 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-6.8.0_g5ec3e6e88b2e-1 ######################################## 2024-03-21T10:13:36.547 INFO:teuthology.orchestra.run.smithi031.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-21T10:13:59.885 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi031.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-21T10:13:59.885 DEBUG:teuthology.orchestra.run.smithi031:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-21T10:13:59.961 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e 2024-03-21T10:13:59.961 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/System.map 2024-03-21T10:13:59.961 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/config 2024-03-21T10:13:59.961 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel 2024-03-21T10:13:59.961 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch 2024-03-21T10:13:59.961 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86 2024-03-21T10:13:59.961 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto 2024-03-21T10:13:59.961 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-21T10:13:59.961 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-21T10:13:59.961 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-21T10:13:59.962 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-21T10:13:59.962 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-21T10:13:59.962 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-21T10:13:59.962 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-21T10:13:59.962 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-21T10:13:59.962 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-21T10:13:59.962 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-21T10:13:59.962 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-21T10:13:59.962 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-21T10:13:59.962 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-21T10:13:59.962 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-21T10:13:59.962 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-21T10:13:59.962 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-21T10:13:59.962 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-21T10:13:59.963 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-21T10:13:59.963 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-21T10:13:59.963 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-21T10:13:59.963 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel 2024-03-21T10:13:59.963 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu 2024-03-21T10:13:59.963 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce 2024-03-21T10:13:59.963 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-21T10:13:59.963 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm 2024-03-21T10:13:59.963 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-21T10:13:59.963 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-21T10:13:59.963 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm.ko 2024-03-21T10:13:59.963 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto 2024-03-21T10:13:59.963 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ansi_cprng.ko 2024-03-21T10:13:59.963 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/anubis.ko 2024-03-21T10:13:59.964 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/arc4.ko 2024-03-21T10:13:59.964 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx 2024-03-21T10:13:59.964 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_memcpy.ko 2024-03-21T10:13:59.964 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_pq.ko 2024-03-21T10:13:59.964 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-21T10:13:59.964 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_tx.ko 2024-03-21T10:13:59.964 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_xor.ko 2024-03-21T10:13:59.964 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/raid6test.ko 2024-03-21T10:13:59.964 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authenc.ko 2024-03-21T10:13:59.964 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authencesn.ko 2024-03-21T10:13:59.964 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blake2b_generic.ko 2024-03-21T10:13:59.964 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_common.ko 2024-03-21T10:13:59.965 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_generic.ko 2024-03-21T10:13:59.965 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/camellia_generic.ko 2024-03-21T10:13:59.965 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast5_generic.ko 2024-03-21T10:13:59.965 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast6_generic.ko 2024-03-21T10:13:59.965 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast_common.ko 2024-03-21T10:13:59.965 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ccm.ko 2024-03-21T10:13:59.965 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cmac.ko 2024-03-21T10:13:59.965 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crc32_generic.ko 2024-03-21T10:13:59.965 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_engine.ko 2024-03-21T10:13:59.965 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_user.ko 2024-03-21T10:13:59.965 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cts.ko 2024-03-21T10:13:59.965 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/des_generic.ko 2024-03-21T10:13:59.965 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/dh_generic.ko 2024-03-21T10:13:59.966 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/echainiv.ko 2024-03-21T10:13:59.966 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/essiv.ko 2024-03-21T10:13:59.966 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/fcrypt.ko 2024-03-21T10:13:59.966 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/khazad.ko 2024-03-21T10:13:59.966 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4.ko 2024-03-21T10:13:59.966 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4hc.ko 2024-03-21T10:13:59.966 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/md4.ko 2024-03-21T10:13:59.966 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/michael_mic.ko 2024-03-21T10:13:59.966 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcbc.ko 2024-03-21T10:13:59.966 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcrypt.ko 2024-03-21T10:13:59.966 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/rmd160.ko 2024-03-21T10:13:59.966 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/seed.ko 2024-03-21T10:13:59.966 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/serpent_generic.ko 2024-03-21T10:13:59.966 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tcrypt.ko 2024-03-21T10:13:59.967 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tea.ko 2024-03-21T10:13:59.967 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_common.ko 2024-03-21T10:13:59.967 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_generic.ko 2024-03-21T10:13:59.967 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/vmac.ko 2024-03-21T10:13:59.967 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/wp512.ko 2024-03-21T10:13:59.967 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xcbc.ko 2024-03-21T10:13:59.967 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xor.ko 2024-03-21T10:13:59.967 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xxhash_generic.ko 2024-03-21T10:13:59.967 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers 2024-03-21T10:13:59.967 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi 2024-03-21T10:13:59.967 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-21T10:13:59.967 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_pad.ko 2024-03-21T10:13:59.967 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/custom_method.ko 2024-03-21T10:13:59.968 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/ec_sys.ko 2024-03-21T10:13:59.968 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/platform_profile.ko 2024-03-21T10:13:59.968 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbs.ko 2024-03-21T10:13:59.968 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbshc.ko 2024-03-21T10:13:59.968 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/video.ko 2024-03-21T10:13:59.968 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata 2024-03-21T10:13:59.968 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/acard-ahci.ko 2024-03-21T10:13:59.968 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ahci_platform.ko 2024-03-21T10:13:59.968 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ata_generic.ko 2024-03-21T10:13:59.968 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/libahci_platform.ko 2024-03-21T10:13:59.968 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_acpi.ko 2024-03-21T10:13:59.968 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ali.ko 2024-03-21T10:13:59.968 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_amd.ko 2024-03-21T10:13:59.968 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_artop.ko 2024-03-21T10:13:59.969 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atiixp.ko 2024-03-21T10:13:59.969 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atp867x.ko 2024-03-21T10:13:59.969 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd640.ko 2024-03-21T10:13:59.969 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd64x.ko 2024-03-21T10:13:59.969 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cypress.ko 2024-03-21T10:13:59.969 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_efar.ko 2024-03-21T10:13:59.969 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt366.ko 2024-03-21T10:13:59.969 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt37x.ko 2024-03-21T10:13:59.969 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-21T10:13:59.969 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-21T10:13:59.969 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it8213.ko 2024-03-21T10:13:59.969 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it821x.ko 2024-03-21T10:13:59.969 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_jmicron.ko 2024-03-21T10:13:59.969 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_marvell.ko 2024-03-21T10:13:59.970 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_mpiix.ko 2024-03-21T10:13:59.970 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_netcell.ko 2024-03-21T10:13:59.970 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ninja32.ko 2024-03-21T10:13:59.970 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87410.ko 2024-03-21T10:13:59.970 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87415.ko 2024-03-21T10:13:59.970 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_oldpiix.ko 2024-03-21T10:13:59.970 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_opti.ko 2024-03-21T10:13:59.970 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_optidma.ko 2024-03-21T10:13:59.970 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pcmcia.ko 2024-03-21T10:13:59.970 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-21T10:13:59.970 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-21T10:13:59.970 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_piccolo.ko 2024-03-21T10:13:59.970 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_rdc.ko 2024-03-21T10:13:59.970 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sch.ko 2024-03-21T10:13:59.971 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_serverworks.ko 2024-03-21T10:13:59.971 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sil680.ko 2024-03-21T10:13:59.971 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sis.ko 2024-03-21T10:13:59.971 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sl82c105.ko 2024-03-21T10:13:59.971 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_triflex.ko 2024-03-21T10:13:59.971 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_via.ko 2024-03-21T10:13:59.971 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pdc_adma.ko 2024-03-21T10:13:59.971 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_inic162x.ko 2024-03-21T10:13:59.971 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_mv.ko 2024-03-21T10:13:59.971 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_nv.ko 2024-03-21T10:13:59.971 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_promise.ko 2024-03-21T10:13:59.971 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_qstor.ko 2024-03-21T10:13:59.971 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil.ko 2024-03-21T10:13:59.971 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil24.ko 2024-03-21T10:13:59.972 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sis.ko 2024-03-21T10:13:59.972 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_svw.ko 2024-03-21T10:13:59.972 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sx4.ko 2024-03-21T10:13:59.972 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_uli.ko 2024-03-21T10:13:59.972 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_via.ko 2024-03-21T10:13:59.972 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_vsc.ko 2024-03-21T10:13:59.972 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay 2024-03-21T10:13:59.972 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay/ks0108.ko 2024-03-21T10:13:59.972 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base 2024-03-21T10:13:59.972 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap 2024-03-21T10:13:59.972 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-21T10:13:59.972 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-21T10:13:59.972 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma 2024-03-21T10:13:59.973 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma/bcma.ko 2024-03-21T10:13:59.973 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block 2024-03-21T10:13:59.973 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe 2024-03-21T10:13:59.973 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe/aoe.ko 2024-03-21T10:13:59.973 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/brd.ko 2024-03-21T10:13:59.973 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd 2024-03-21T10:13:59.973 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd/drbd.ko 2024-03-21T10:13:59.973 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/floppy.ko 2024-03-21T10:13:59.973 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/loop.ko 2024-03-21T10:13:59.973 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx 2024-03-21T10:13:59.973 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-21T10:13:59.973 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/nbd.ko 2024-03-21T10:13:59.973 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk 2024-03-21T10:13:59.973 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk/null_blk.ko 2024-03-21T10:13:59.974 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/pktcdvd.ko 2024-03-21T10:13:59.974 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/rbd.ko 2024-03-21T10:13:59.974 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/virtio_blk.ko 2024-03-21T10:13:59.974 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback 2024-03-21T10:13:59.974 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-21T10:13:59.974 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkfront.ko 2024-03-21T10:13:59.974 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram 2024-03-21T10:13:59.974 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram/zram.ko 2024-03-21T10:13:59.974 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char 2024-03-21T10:13:59.974 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hangcheck-timer.ko 2024-03-21T10:13:59.974 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random 2024-03-21T10:13:59.974 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-21T10:13:59.974 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-21T10:13:59.975 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-21T10:13:59.975 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/via-rng.ko 2024-03-21T10:13:59.975 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-21T10:13:59.975 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi 2024-03-21T10:13:59.975 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-21T10:13:59.975 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-21T10:13:59.975 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-21T10:13:59.975 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-21T10:13:59.975 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-21T10:13:59.975 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-21T10:13:59.975 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/lp.ko 2024-03-21T10:13:59.975 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave 2024-03-21T10:13:59.975 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave/mwave.ko 2024-03-21T10:13:59.975 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ppdev.ko 2024-03-21T10:13:59.976 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tlclk.ko 2024-03-21T10:13:59.976 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm 2024-03-21T10:13:59.976 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm.ko 2024-03-21T10:13:59.976 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-21T10:13:59.976 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-21T10:13:59.976 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-21T10:13:59.976 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-21T10:13:59.976 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-21T10:13:59.976 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/uv_mmtimer.ko 2024-03-21T10:13:59.976 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/virtio_console.ko 2024-03-21T10:13:59.976 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq 2024-03-21T10:13:59.976 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-21T10:13:59.976 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-21T10:13:59.976 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-21T10:13:59.977 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-21T10:13:59.977 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-21T10:13:59.977 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-21T10:13:59.977 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto 2024-03-21T10:13:59.977 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp 2024-03-21T10:13:59.977 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-21T10:13:59.977 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp.ko 2024-03-21T10:13:59.977 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel 2024-03-21T10:13:59.977 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat 2024-03-21T10:13:59.977 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common 2024-03-21T10:13:59.977 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-21T10:13:59.977 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-21T10:13:59.977 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-21T10:13:59.977 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-aes.ko 2024-03-21T10:13:59.978 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-sha.ko 2024-03-21T10:13:59.978 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio 2024-03-21T10:13:59.978 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-21T10:13:59.978 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca 2024-03-21T10:13:59.978 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca/dca.ko 2024-03-21T10:13:59.978 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq 2024-03-21T10:13:59.978 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-21T10:13:59.978 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma 2024-03-21T10:13:59.978 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw 2024-03-21T10:13:59.978 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-21T10:13:59.978 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat 2024-03-21T10:13:59.978 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-21T10:13:59.978 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac 2024-03-21T10:13:59.979 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/amd64_edac.ko 2024-03-21T10:13:59.979 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/e752x_edac.ko 2024-03-21T10:13:59.979 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/edac_mce_amd.ko 2024-03-21T10:13:59.979 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3000_edac.ko 2024-03-21T10:13:59.979 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3200_edac.ko 2024-03-21T10:13:59.979 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5100_edac.ko 2024-03-21T10:13:59.979 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5400_edac.ko 2024-03-21T10:13:59.979 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7300_edac.ko 2024-03-21T10:13:59.979 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7core_edac.ko 2024-03-21T10:13:59.979 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i82975x_edac.ko 2024-03-21T10:13:59.979 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/ie31200_edac.ko 2024-03-21T10:13:59.979 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/sb_edac.ko 2024-03-21T10:13:59.979 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/x38_edac.ko 2024-03-21T10:13:59.980 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware 2024-03-21T10:13:59.980 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/edd.ko 2024-03-21T10:13:59.980 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-21T10:13:59.980 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio 2024-03-21T10:13:59.980 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-ich.ko 2024-03-21T10:13:59.980 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-21T10:13:59.980 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid 2024-03-21T10:13:59.980 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-appleir.ko 2024-03-21T10:13:59.980 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-aureal.ko 2024-03-21T10:13:59.980 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-axff.ko 2024-03-21T10:13:59.980 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-dr.ko 2024-03-21T10:13:59.980 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elecom.ko 2024-03-21T10:13:59.980 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elo.ko 2024-03-21T10:13:59.980 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-emsff.ko 2024-03-21T10:13:59.980 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gaff.ko 2024-03-21T10:13:59.981 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gt683r.ko 2024-03-21T10:13:59.981 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gyration.ko 2024-03-21T10:13:59.981 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-21T10:13:59.981 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-21T10:13:59.981 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtekff.ko 2024-03-21T10:13:59.981 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-hyperv.ko 2024-03-21T10:13:59.981 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-icade.ko 2024-03-21T10:13:59.981 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-keytouch.ko 2024-03-21T10:13:59.981 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-kye.ko 2024-03-21T10:13:59.981 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lcpower.ko 2024-03-21T10:13:59.981 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-led.ko 2024-03-21T10:13:59.981 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lenovo.ko 2024-03-21T10:13:59.981 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-21T10:13:59.981 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-21T10:13:59.981 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-multitouch.ko 2024-03-21T10:13:59.982 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-ortek.ko 2024-03-21T10:13:59.982 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-penmount.ko 2024-03-21T10:13:59.982 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-petalynx.ko 2024-03-21T10:13:59.982 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-picolcd.ko 2024-03-21T10:13:59.982 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-pl.ko 2024-03-21T10:13:59.982 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-plantronics.ko 2024-03-21T10:13:59.982 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-primax.ko 2024-03-21T10:13:59.982 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-rmi.ko 2024-03-21T10:13:59.982 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-21T10:13:59.982 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-common.ko 2024-03-21T10:13:59.982 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-21T10:13:59.982 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-21T10:13:59.982 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-21T10:13:59.982 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-21T10:13:59.983 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-21T10:13:59.983 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-21T10:13:59.983 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-21T10:13:59.983 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-21T10:13:59.983 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-21T10:13:59.983 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat.ko 2024-03-21T10:13:59.983 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-saitek.ko 2024-03-21T10:13:59.983 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-samsung.ko 2024-03-21T10:13:59.983 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-21T10:13:59.983 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sjoy.ko 2024-03-21T10:13:59.983 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sony.ko 2024-03-21T10:13:59.983 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-speedlink.ko 2024-03-21T10:13:59.983 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-steelseries.ko 2024-03-21T10:13:59.983 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sunplus.ko 2024-03-21T10:13:59.983 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-21T10:13:59.983 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tivo.ko 2024-03-21T10:13:59.984 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tmff.ko 2024-03-21T10:13:59.984 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-topseed.ko 2024-03-21T10:13:59.984 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-twinhan.ko 2024-03-21T10:13:59.984 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-uclogic.ko 2024-03-21T10:13:59.984 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-waltop.ko 2024-03-21T10:13:59.984 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-wiimote.ko 2024-03-21T10:13:59.984 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-xinmo.ko 2024-03-21T10:13:59.984 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zpff.ko 2024-03-21T10:13:59.984 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zydacron.ko 2024-03-21T10:13:59.984 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/uhid.ko 2024-03-21T10:13:59.984 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/wacom.ko 2024-03-21T10:13:59.984 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv 2024-03-21T10:13:59.984 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_balloon.ko 2024-03-21T10:13:59.984 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_utils.ko 2024-03-21T10:13:59.985 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_vmbus.ko 2024-03-21T10:13:59.985 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon 2024-03-21T10:13:59.985 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru.ko 2024-03-21T10:13:59.985 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru3.ko 2024-03-21T10:13:59.985 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-21T10:13:59.985 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7414.ko 2024-03-21T10:13:59.985 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7418.ko 2024-03-21T10:13:59.985 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adc128d818.ko 2024-03-21T10:13:59.985 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1025.ko 2024-03-21T10:13:59.985 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1026.ko 2024-03-21T10:13:59.985 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1029.ko 2024-03-21T10:13:59.985 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1031.ko 2024-03-21T10:13:59.985 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm9240.ko 2024-03-21T10:13:59.985 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ads7828.ko 2024-03-21T10:13:59.985 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7410.ko 2024-03-21T10:13:59.985 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7411.ko 2024-03-21T10:13:59.986 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7462.ko 2024-03-21T10:13:59.986 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7470.ko 2024-03-21T10:13:59.986 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7475.ko 2024-03-21T10:13:59.986 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7x10.ko 2024-03-21T10:13:59.986 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/amc6821.ko 2024-03-21T10:13:59.986 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/applesmc.ko 2024-03-21T10:13:59.986 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asb100.ko 2024-03-21T10:13:59.986 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asc7621.ko 2024-03-21T10:13:59.986 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-21T10:13:59.986 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/atxp1.ko 2024-03-21T10:13:59.986 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/coretemp.ko 2024-03-21T10:13:59.986 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-21T10:13:59.986 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dme1737.ko 2024-03-21T10:13:59.986 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds1621.ko 2024-03-21T10:13:59.986 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds620.ko 2024-03-21T10:13:59.987 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc1403.ko 2024-03-21T10:13:59.987 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc6w201.ko 2024-03-21T10:13:59.987 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71805f.ko 2024-03-21T10:13:59.987 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71882fg.ko 2024-03-21T10:13:59.987 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f75375s.ko 2024-03-21T10:13:59.987 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fam15h_power.ko 2024-03-21T10:13:59.987 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fschmd.ko 2024-03-21T10:13:59.987 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g760a.ko 2024-03-21T10:13:59.987 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g762.ko 2024-03-21T10:13:59.987 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl518sm.ko 2024-03-21T10:13:59.987 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl520sm.ko 2024-03-21T10:13:59.987 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-21T10:13:59.987 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5500_temp.ko 2024-03-21T10:13:59.987 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5k_amb.ko 2024-03-21T10:13:59.988 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmaem.ko 2024-03-21T10:13:59.988 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmpex.ko 2024-03-21T10:13:59.988 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina209.ko 2024-03-21T10:13:59.988 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina2xx.ko 2024-03-21T10:13:59.988 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/it87.ko 2024-03-21T10:13:59.988 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k10temp.ko 2024-03-21T10:13:59.988 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k8temp.ko 2024-03-21T10:13:59.988 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lineage-pem.ko 2024-03-21T10:13:59.988 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm63.ko 2024-03-21T10:13:59.988 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm73.ko 2024-03-21T10:13:59.988 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm75.ko 2024-03-21T10:13:59.988 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm77.ko 2024-03-21T10:13:59.988 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm78.ko 2024-03-21T10:13:59.988 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm80.ko 2024-03-21T10:13:59.988 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm83.ko 2024-03-21T10:13:59.988 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm85.ko 2024-03-21T10:13:59.989 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm87.ko 2024-03-21T10:13:59.989 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm90.ko 2024-03-21T10:13:59.989 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm92.ko 2024-03-21T10:13:59.989 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm93.ko 2024-03-21T10:13:59.989 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95234.ko 2024-03-21T10:13:59.989 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95241.ko 2024-03-21T10:13:59.989 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95245.ko 2024-03-21T10:13:59.989 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc2945.ko 2024-03-21T10:13:59.989 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4151.ko 2024-03-21T10:13:59.989 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4215.ko 2024-03-21T10:13:59.989 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4222.ko 2024-03-21T10:13:59.989 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4245.ko 2024-03-21T10:13:59.989 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4260.ko 2024-03-21T10:13:59.989 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4261.ko 2024-03-21T10:13:59.989 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max16065.ko 2024-03-21T10:13:59.989 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1619.ko 2024-03-21T10:13:59.990 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1668.ko 2024-03-21T10:13:59.990 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max197.ko 2024-03-21T10:13:59.990 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6639.ko 2024-03-21T10:13:59.990 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6650.ko 2024-03-21T10:13:59.990 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6697.ko 2024-03-21T10:13:59.990 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/mcp3021.ko 2024-03-21T10:13:59.990 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6683.ko 2024-03-21T10:13:59.990 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775-core.ko 2024-03-21T10:13:59.990 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775.ko 2024-03-21T10:13:59.990 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct7802.ko 2024-03-21T10:13:59.990 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87360.ko 2024-03-21T10:13:59.990 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87427.ko 2024-03-21T10:13:59.990 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pcf8591.ko 2024-03-21T10:13:59.990 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus 2024-03-21T10:13:59.990 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-21T10:13:59.991 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-21T10:13:59.991 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-21T10:13:59.991 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-21T10:13:59.991 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-21T10:13:59.991 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-21T10:13:59.991 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-21T10:13:59.991 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-21T10:13:59.991 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-21T10:13:59.991 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-21T10:13:59.991 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-21T10:13:59.991 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-21T10:13:59.991 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/powr1220.ko 2024-03-21T10:13:59.991 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5627.ko 2024-03-21T10:13:59.991 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5636.ko 2024-03-21T10:13:59.991 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-21T10:13:59.991 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht15.ko 2024-03-21T10:13:59.991 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht21.ko 2024-03-21T10:13:59.992 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/shtc1.ko 2024-03-21T10:13:59.992 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sis5595.ko 2024-03-21T10:13:59.992 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47b397.ko 2024-03-21T10:13:59.992 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m1.ko 2024-03-21T10:13:59.992 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m192.ko 2024-03-21T10:13:59.992 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/thmc50.ko 2024-03-21T10:13:59.992 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp102.ko 2024-03-21T10:13:59.992 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp103.ko 2024-03-21T10:13:59.992 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp401.ko 2024-03-21T10:13:59.992 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp421.ko 2024-03-21T10:13:59.992 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via-cputemp.ko 2024-03-21T10:13:59.992 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via686a.ko 2024-03-21T10:13:59.992 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt1211.ko 2024-03-21T10:13:59.992 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt8231.ko 2024-03-21T10:13:59.992 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627ehf.ko 2024-03-21T10:13:59.993 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627hf.ko 2024-03-21T10:13:59.993 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83781d.ko 2024-03-21T10:13:59.993 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83791d.ko 2024-03-21T10:13:59.993 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83792d.ko 2024-03-21T10:13:59.993 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83793.ko 2024-03-21T10:13:59.993 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83795.ko 2024-03-21T10:13:59.993 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l785ts.ko 2024-03-21T10:13:59.993 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l786ng.ko 2024-03-21T10:13:59.993 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c 2024-03-21T10:13:59.993 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos 2024-03-21T10:13:59.993 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-21T10:13:59.993 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-21T10:13:59.993 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses 2024-03-21T10:13:59.993 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-21T10:13:59.993 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-21T10:13:59.994 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-21T10:13:59.994 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-21T10:13:59.994 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-21T10:13:59.994 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-21T10:13:59.994 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-21T10:13:59.994 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-21T10:13:59.994 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-21T10:13:59.994 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-21T10:13:59.994 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-21T10:13:59.994 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-21T10:13:59.994 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-21T10:13:59.994 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-21T10:13:59.994 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-21T10:13:59.994 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-21T10:13:59.995 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-21T10:13:59.995 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-21T10:13:59.995 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-21T10:13:59.995 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-21T10:13:59.995 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-21T10:13:59.995 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-21T10:13:59.995 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-21T10:13:59.995 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-dev.ko 2024-03-21T10:13:59.995 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-mux.ko 2024-03-21T10:13:59.995 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-21T10:13:59.995 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-smbus.ko 2024-03-21T10:13:59.995 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-stub.ko 2024-03-21T10:13:59.995 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband 2024-03-21T10:13:59.995 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core 2024-03-21T10:13:59.995 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-21T10:13:59.996 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_core.ko 2024-03-21T10:13:59.996 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-21T10:13:59.996 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-21T10:13:59.996 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-21T10:13:59.996 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-21T10:13:59.996 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-21T10:13:59.996 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw 2024-03-21T10:13:59.996 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4 2024-03-21T10:13:59.996 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-21T10:13:59.996 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4 2024-03-21T10:13:59.996 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-21T10:13:59.996 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5 2024-03-21T10:13:59.996 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-21T10:13:59.996 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca 2024-03-21T10:13:59.996 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-21T10:13:59.997 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp 2024-03-21T10:13:59.997 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib 2024-03-21T10:13:59.997 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-21T10:13:59.997 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser 2024-03-21T10:13:59.997 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-21T10:13:59.997 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert 2024-03-21T10:13:59.997 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-21T10:13:59.997 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp 2024-03-21T10:13:59.997 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-21T10:13:59.997 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt 2024-03-21T10:13:59.997 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-21T10:13:59.997 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input 2024-03-21T10:13:59.997 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport 2024-03-21T10:13:59.997 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-21T10:13:59.997 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-21T10:13:59.998 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/gameport.ko 2024-03-21T10:13:59.998 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/joydev.ko 2024-03-21T10:13:59.998 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard 2024-03-21T10:13:59.998 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-21T10:13:59.998 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse 2024-03-21T10:13:59.998 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/appletouch.ko 2024-03-21T10:13:59.998 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/bcm5974.ko 2024-03-21T10:13:59.998 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/cyapatp.ko 2024-03-21T10:13:59.998 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-21T10:13:59.998 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/sermouse.ko 2024-03-21T10:13:59.998 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-21T10:13:59.998 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-21T10:13:59.998 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-21T10:13:59.999 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4 2024-03-21T10:13:59.999 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-21T10:13:59.999 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio 2024-03-21T10:13:59.999 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/altera_ps2.ko 2024-03-21T10:13:59.999 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/arc_ps2.ko 2024-03-21T10:13:59.999 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-21T10:13:59.999 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/serio_raw.ko 2024-03-21T10:13:59.999 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/sparse-keymap.ko 2024-03-21T10:13:59.999 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds 2024-03-21T10:13:59.999 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-blinkm.ko 2024-03-21T10:13:59.999 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm3530.ko 2024-03-21T10:13:59.999 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lp3944.ko 2024-03-21T10:13:59.999 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-ss4200.ko 2024-03-21T10:13:59.999 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger 2024-03-21T10:13:59.999 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-21T10:13:59.999 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-21T10:14:00.000 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-21T10:14:00.000 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-21T10:14:00.000 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-21T10:14:00.000 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-21T10:14:00.000 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-21T10:14:00.000 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-21T10:14:00.000 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-21T10:14:00.000 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md 2024-03-21T10:14:00.000 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache 2024-03-21T10:14:00.000 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache/bcache.ko 2024-03-21T10:14:00.000 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-bio-prison.ko 2024-03-21T10:14:00.000 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache-smq.ko 2024-03-21T10:14:00.000 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache.ko 2024-03-21T10:14:00.000 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-crypt.ko 2024-03-21T10:14:00.000 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-delay.ko 2024-03-21T10:14:00.000 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-flakey.ko 2024-03-21T10:14:00.001 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-log-userspace.ko 2024-03-21T10:14:00.001 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-multipath.ko 2024-03-21T10:14:00.001 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-queue-length.ko 2024-03-21T10:14:00.001 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-raid.ko 2024-03-21T10:14:00.001 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-round-robin.ko 2024-03-21T10:14:00.001 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-service-time.ko 2024-03-21T10:14:00.001 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-switch.ko 2024-03-21T10:14:00.001 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-thin-pool.ko 2024-03-21T10:14:00.001 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-verity.ko 2024-03-21T10:14:00.001 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data 2024-03-21T10:14:00.001 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-21T10:14:00.001 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid0.ko 2024-03-21T10:14:00.001 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid1.ko 2024-03-21T10:14:00.001 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid10.ko 2024-03-21T10:14:00.001 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid456.ko 2024-03-21T10:14:00.001 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message 2024-03-21T10:14:00.002 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion 2024-03-21T10:14:00.002 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptbase.ko 2024-03-21T10:14:00.002 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptctl.ko 2024-03-21T10:14:00.002 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptfc.ko 2024-03-21T10:14:00.002 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptlan.ko 2024-03-21T10:14:00.002 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptsas.ko 2024-03-21T10:14:00.002 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptscsih.ko 2024-03-21T10:14:00.002 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptspi.ko 2024-03-21T10:14:00.002 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd 2024-03-21T10:14:00.002 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_ich.ko 2024-03-21T10:14:00.002 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_sch.ko 2024-03-21T10:14:00.002 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/mfd-core.ko 2024-03-21T10:14:00.002 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/sm501.ko 2024-03-21T10:14:00.002 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/viperboard.ko 2024-03-21T10:14:00.002 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/vx855.ko 2024-03-21T10:14:00.003 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/wl1273-core.ko 2024-03-21T10:14:00.003 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc 2024-03-21T10:14:00.003 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl 2024-03-21T10:14:00.003 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-21T10:14:00.003 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds9802als.ko 2024-03-21T10:14:00.003 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds990x.ko 2024-03-21T10:14:00.003 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/bh1770glc.ko 2024-03-21T10:14:00.003 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710 2024-03-21T10:14:00.003 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710/cb710.ko 2024-03-21T10:14:00.003 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom 2024-03-21T10:14:00.003 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/at24.ko 2024-03-21T10:14:00.003 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-21T10:14:00.003 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/max6875.ko 2024-03-21T10:14:00.003 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/enclosure.ko 2024-03-21T10:14:00.004 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/hpilo.ko 2024-03-21T10:14:00.004 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29003.ko 2024-03-21T10:14:00.004 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29020.ko 2024-03-21T10:14:00.004 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d 2024-03-21T10:14:00.004 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-21T10:14:00.004 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-21T10:14:00.004 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei 2024-03-21T10:14:00.004 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-me.ko 2024-03-21T10:14:00.004 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-txe.ko 2024-03-21T10:14:00.004 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei.ko 2024-03-21T10:14:00.004 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru 2024-03-21T10:14:00.004 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-21T10:14:00.004 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp 2024-03-21T10:14:00.004 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-21T10:14:00.005 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-21T10:14:00.005 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-21T10:14:00.005 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_7xx1.ko 2024-03-21T10:14:00.005 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_core.ko 2024-03-21T10:14:00.005 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tsl2550.ko 2024-03-21T10:14:00.005 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_balloon.ko 2024-03-21T10:14:00.005 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci 2024-03-21T10:14:00.005 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-21T10:14:00.005 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net 2024-03-21T10:14:00.005 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding 2024-03-21T10:14:00.005 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding/bonding.ko 2024-03-21T10:14:00.005 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa 2024-03-21T10:14:00.005 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53 2024-03-21T10:14:00.005 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-21T10:14:00.005 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-21T10:14:00.006 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-21T10:14:00.006 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx 2024-03-21T10:14:00.006 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-21T10:14:00.006 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dummy.ko 2024-03-21T10:14:00.006 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/eql.ko 2024-03-21T10:14:00.006 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet 2024-03-21T10:14:00.006 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com 2024-03-21T10:14:00.006 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-21T10:14:00.006 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-21T10:14:00.006 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-21T10:14:00.006 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-21T10:14:00.006 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390 2024-03-21T10:14:00.006 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-21T10:14:00.006 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-21T10:14:00.007 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-21T10:14:00.007 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-21T10:14:00.007 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec 2024-03-21T10:14:00.007 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-21T10:14:00.007 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere 2024-03-21T10:14:00.007 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-21T10:14:00.007 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon 2024-03-21T10:14:00.007 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-21T10:14:00.007 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera 2024-03-21T10:14:00.007 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-21T10:14:00.007 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd 2024-03-21T10:14:00.007 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-21T10:14:00.007 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-21T10:14:00.007 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-21T10:14:00.008 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros 2024-03-21T10:14:00.008 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx 2024-03-21T10:14:00.008 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-21T10:14:00.008 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-21T10:14:00.008 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-21T10:14:00.008 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-21T10:14:00.008 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-21T10:14:00.008 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx 2024-03-21T10:14:00.008 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-21T10:14:00.008 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-21T10:14:00.008 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom 2024-03-21T10:14:00.008 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-21T10:14:00.008 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-21T10:14:00.008 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-21T10:14:00.009 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-21T10:14:00.009 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-21T10:14:00.009 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet 2024-03-21T10:14:00.009 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-21T10:14:00.009 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-21T10:14:00.009 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade 2024-03-21T10:14:00.009 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna 2024-03-21T10:14:00.009 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-21T10:14:00.009 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio 2024-03-21T10:14:00.009 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-21T10:14:00.009 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-21T10:14:00.009 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-21T10:14:00.009 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-21T10:14:00.009 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-21T10:14:00.009 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-21T10:14:00.009 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-21T10:14:00.010 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-21T10:14:00.010 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-21T10:14:00.010 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-21T10:14:00.010 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco 2024-03-21T10:14:00.010 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic 2024-03-21T10:14:00.010 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-21T10:14:00.010 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec 2024-03-21T10:14:00.010 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip 2024-03-21T10:14:00.010 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-21T10:14:00.010 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-21T10:14:00.010 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-21T10:14:00.010 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-21T10:14:00.010 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-21T10:14:00.010 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-21T10:14:00.011 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink 2024-03-21T10:14:00.011 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-21T10:14:00.011 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-21T10:14:00.011 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dnet.ko 2024-03-21T10:14:00.011 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex 2024-03-21T10:14:00.011 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet 2024-03-21T10:14:00.011 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-21T10:14:00.011 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ethoc.ko 2024-03-21T10:14:00.011 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/fealnx.ko 2024-03-21T10:14:00.011 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel 2024-03-21T10:14:00.011 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-21T10:14:00.011 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000 2024-03-21T10:14:00.011 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-21T10:14:00.011 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e 2024-03-21T10:14:00.011 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-21T10:14:00.012 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k 2024-03-21T10:14:00.012 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-21T10:14:00.012 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e 2024-03-21T10:14:00.012 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-21T10:14:00.012 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf 2024-03-21T10:14:00.012 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-21T10:14:00.012 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb 2024-03-21T10:14:00.012 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-21T10:14:00.012 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf 2024-03-21T10:14:00.012 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-21T10:14:00.012 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-21T10:14:00.012 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-21T10:14:00.012 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-21T10:14:00.012 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-21T10:14:00.013 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/jme.ko 2024-03-21T10:14:00.013 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell 2024-03-21T10:14:00.013 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-21T10:14:00.013 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-21T10:14:00.013 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-21T10:14:00.013 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox 2024-03-21T10:14:00.013 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-21T10:14:00.013 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-21T10:14:00.013 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-21T10:14:00.013 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-21T10:14:00.013 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-21T10:14:00.013 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-21T10:14:00.013 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel 2024-03-21T10:14:00.013 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-21T10:14:00.013 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom 2024-03-21T10:14:00.014 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-21T10:14:00.014 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-21T10:14:00.014 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi 2024-03-21T10:14:00.014 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-21T10:14:00.014 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-21T10:14:00.014 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion 2024-03-21T10:14:00.014 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-21T10:14:00.014 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia 2024-03-21T10:14:00.014 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-21T10:14:00.014 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines 2024-03-21T10:14:00.014 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-21T10:14:00.014 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-21T10:14:00.014 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic 2024-03-21T10:14:00.015 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-21T10:14:00.015 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-21T10:14:00.015 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-21T10:14:00.015 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-21T10:14:00.015 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-21T10:14:00.015 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc 2024-03-21T10:14:00.015 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-21T10:14:00.015 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek 2024-03-21T10:14:00.015 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-21T10:14:00.015 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-21T10:14:00.015 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-21T10:14:00.015 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-21T10:14:00.015 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker 2024-03-21T10:14:00.015 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-21T10:14:00.016 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc 2024-03-21T10:14:00.016 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-21T10:14:00.016 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan 2024-03-21T10:14:00.016 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-21T10:14:00.016 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis 2024-03-21T10:14:00.016 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-21T10:14:00.016 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-21T10:14:00.016 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc 2024-03-21T10:14:00.016 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-21T10:14:00.016 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-21T10:14:00.016 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-21T10:14:00.016 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-21T10:14:00.016 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro 2024-03-21T10:14:00.016 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-21T10:14:00.017 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-21T10:14:00.017 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-21T10:14:00.017 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun 2024-03-21T10:14:00.017 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-21T10:14:00.017 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-21T10:14:00.017 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-21T10:14:00.017 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-21T10:14:00.017 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti 2024-03-21T10:14:00.017 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-21T10:14:00.017 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti 2024-03-21T10:14:00.017 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-21T10:14:00.017 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via 2024-03-21T10:14:00.017 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-21T10:14:00.017 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-21T10:14:00.017 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet 2024-03-21T10:14:00.018 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-21T10:14:00.018 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-21T10:14:00.018 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom 2024-03-21T10:14:00.018 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-21T10:14:00.018 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/geneve.ko 2024-03-21T10:14:00.018 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv 2024-03-21T10:14:00.018 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-21T10:14:00.018 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154 2024-03-21T10:14:00.018 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-21T10:14:00.018 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ifb.ko 2024-03-21T10:14:00.018 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan 2024-03-21T10:14:00.018 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-21T10:14:00.018 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvlan.ko 2024-03-21T10:14:00.018 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvtap.ko 2024-03-21T10:14:00.019 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio 2024-03-21T10:14:00.019 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio.ko 2024-03-21T10:14:00.019 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-21T10:14:00.019 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-21T10:14:00.019 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-21T10:14:00.019 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mii.ko 2024-03-21T10:14:00.019 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/net_failover.ko 2024-03-21T10:14:00.019 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/netconsole.ko 2024-03-21T10:14:00.019 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/nlmon.ko 2024-03-21T10:14:00.019 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs 2024-03-21T10:14:00.019 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-21T10:14:00.019 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-21T10:14:00.019 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy 2024-03-21T10:14:00.020 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/amd.ko 2024-03-21T10:14:00.020 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-21T10:14:00.020 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-21T10:14:00.020 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-21T10:14:00.020 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm87xx.ko 2024-03-21T10:14:00.020 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/broadcom.ko 2024-03-21T10:14:00.020 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/cicada.ko 2024-03-21T10:14:00.020 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/davicom.ko 2024-03-21T10:14:00.020 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/dp83640.ko 2024-03-21T10:14:00.020 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/et1011c.ko 2024-03-21T10:14:00.020 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/icplus.ko 2024-03-21T10:14:00.020 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/lxt.ko 2024-03-21T10:14:00.020 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/marvell.ko 2024-03-21T10:14:00.020 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/micrel.ko 2024-03-21T10:14:00.021 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/national.ko 2024-03-21T10:14:00.021 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/phylink.ko 2024-03-21T10:14:00.021 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/qsemi.ko 2024-03-21T10:14:00.021 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/realtek.ko 2024-03-21T10:14:00.021 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/smsc.ko 2024-03-21T10:14:00.021 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/ste10Xp.ko 2024-03-21T10:14:00.021 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/vitesse.ko 2024-03-21T10:14:00.021 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp 2024-03-21T10:14:00.021 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-21T10:14:00.021 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_async.ko 2024-03-21T10:14:00.021 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-21T10:14:00.021 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-21T10:14:00.021 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-21T10:14:00.021 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-21T10:14:00.021 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppoe.ko 2024-03-21T10:14:00.022 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppox.ko 2024-03-21T10:14:00.022 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pptp.ko 2024-03-21T10:14:00.022 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip 2024-03-21T10:14:00.022 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slhc.ko 2024-03-21T10:14:00.022 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slip.ko 2024-03-21T10:14:00.022 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/sungem_phy.ko 2024-03-21T10:14:00.022 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tap.ko 2024-03-21T10:14:00.022 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team 2024-03-21T10:14:00.022 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team.ko 2024-03-21T10:14:00.022 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-21T10:14:00.022 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-21T10:14:00.022 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-21T10:14:00.023 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_random.ko 2024-03-21T10:14:00.023 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-21T10:14:00.023 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tun.ko 2024-03-21T10:14:00.023 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/veth.ko 2024-03-21T10:14:00.023 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/virtio_net.ko 2024-03-21T10:14:00.023 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3 2024-03-21T10:14:00.023 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-21T10:14:00.023 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan 2024-03-21T10:14:00.023 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan/vxlan.ko 2024-03-21T10:14:00.023 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback 2024-03-21T10:14:00.023 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-21T10:14:00.023 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netfront.ko 2024-03-21T10:14:00.023 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb 2024-03-21T10:14:00.023 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb/ntb.ko 2024-03-21T10:14:00.024 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme 2024-03-21T10:14:00.024 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host 2024-03-21T10:14:00.024 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme-core.ko 2024-03-21T10:14:00.024 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme.ko 2024-03-21T10:14:00.024 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport 2024-03-21T10:14:00.024 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport.ko 2024-03-21T10:14:00.024 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_cs.ko 2024-03-21T10:14:00.024 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_pc.ko 2024-03-21T10:14:00.024 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_serial.ko 2024-03-21T10:14:00.024 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci 2024-03-21T10:14:00.024 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug 2024-03-21T10:14:00.024 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-21T10:14:00.024 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie 2024-03-21T10:14:00.024 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-21T10:14:00.024 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/xen-pcifront.ko 2024-03-21T10:14:00.025 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia 2024-03-21T10:14:00.025 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/i82092.ko 2024-03-21T10:14:00.025 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pd6729.ko 2024-03-21T10:14:00.025 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-21T10:14:00.025 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl 2024-03-21T10:14:00.025 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel 2024-03-21T10:14:00.025 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-21T10:14:00.025 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform 2024-03-21T10:14:00.025 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86 2024-03-21T10:14:00.025 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-21T10:14:00.025 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acerhdf.ko 2024-03-21T10:14:00.025 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-21T10:14:00.025 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-21T10:14:00.025 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-21T10:14:00.025 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-21T10:14:00.026 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-21T10:14:00.026 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-21T10:14:00.026 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-21T10:14:00.026 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-21T10:14:00.026 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-21T10:14:00.026 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/hdaps.ko 2024-03-21T10:14:00.026 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel 2024-03-21T10:14:00.026 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-21T10:14:00.026 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel_ips.ko 2024-03-21T10:14:00.026 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-21T10:14:00.026 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-21T10:14:00.026 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-21T10:14:00.026 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-21T10:14:00.027 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-21T10:14:00.027 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-21T10:14:00.027 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-21T10:14:00.027 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-21T10:14:00.027 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-21T10:14:00.027 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi.ko 2024-03-21T10:14:00.027 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap 2024-03-21T10:14:00.027 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-21T10:14:00.027 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-21T10:14:00.027 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps 2024-03-21T10:14:00.027 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients 2024-03-21T10:14:00.027 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-21T10:14:00.027 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-21T10:14:00.027 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps_parport.ko 2024-03-21T10:14:00.027 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/pps_core.ko 2024-03-21T10:14:00.028 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp 2024-03-21T10:14:00.028 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp.ko 2024-03-21T10:14:00.028 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp_kvm.ko 2024-03-21T10:14:00.028 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc 2024-03-21T10:14:00.028 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-21T10:14:00.028 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-21T10:14:00.028 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-21T10:14:00.028 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-21T10:14:00.028 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-21T10:14:00.028 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-21T10:14:00.028 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-21T10:14:00.028 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-21T10:14:00.028 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-21T10:14:00.028 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-21T10:14:00.029 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-em3027.ko 2024-03-21T10:14:00.029 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-21T10:14:00.029 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-21T10:14:00.029 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-21T10:14:00.029 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-21T10:14:00.029 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-21T10:14:00.029 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-21T10:14:00.029 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max6900.ko 2024-03-21T10:14:00.029 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-21T10:14:00.029 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-21T10:14:00.029 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-21T10:14:00.029 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-21T10:14:00.029 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-21T10:14:00.029 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-21T10:14:00.029 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-21T10:14:00.030 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-21T10:14:00.030 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-21T10:14:00.030 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-21T10:14:00.030 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-21T10:14:00.030 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-21T10:14:00.030 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-x1205.ko 2024-03-21T10:14:00.030 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi 2024-03-21T10:14:00.030 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-9xxx.ko 2024-03-21T10:14:00.030 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-sas.ko 2024-03-21T10:14:00.030 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-xxxx.ko 2024-03-21T10:14:00.030 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/BusLogic.ko 2024-03-21T10:14:00.030 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/a100u2w.ko 2024-03-21T10:14:00.030 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid 2024-03-21T10:14:00.030 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-21T10:14:00.031 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/advansys.ko 2024-03-21T10:14:00.031 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx 2024-03-21T10:14:00.031 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-21T10:14:00.031 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-21T10:14:00.031 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/am53c974.ko 2024-03-21T10:14:00.031 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr 2024-03-21T10:14:00.031 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-21T10:14:00.031 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/atp870u.ko 2024-03-21T10:14:00.031 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi 2024-03-21T10:14:00.031 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-21T10:14:00.031 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa 2024-03-21T10:14:00.031 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa/bfa.ko 2024-03-21T10:14:00.031 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc 2024-03-21T10:14:00.031 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-21T10:14:00.031 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i 2024-03-21T10:14:00.032 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-21T10:14:00.032 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ch.ko 2024-03-21T10:14:00.032 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor 2024-03-21T10:14:00.032 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-21T10:14:00.032 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi 2024-03-21T10:14:00.032 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-21T10:14:00.032 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-21T10:14:00.032 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-21T10:14:00.032 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-21T10:14:00.032 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-21T10:14:00.032 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dc395x.ko 2024-03-21T10:14:00.032 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler 2024-03-21T10:14:00.032 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-21T10:14:00.032 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-21T10:14:00.033 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-21T10:14:00.033 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-21T10:14:00.033 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dmx3191d.ko 2024-03-21T10:14:00.033 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r 2024-03-21T10:14:00.033 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-21T10:14:00.033 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esp_scsi.ko 2024-03-21T10:14:00.033 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe 2024-03-21T10:14:00.033 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-21T10:14:00.033 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-21T10:14:00.033 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic 2024-03-21T10:14:00.033 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic/fnic.ko 2024-03-21T10:14:00.033 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hpsa.ko 2024-03-21T10:14:00.033 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hptiop.ko 2024-03-21T10:14:00.033 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hv_storvsc.ko 2024-03-21T10:14:00.033 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/initio.ko 2024-03-21T10:14:00.034 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ipr.ko 2024-03-21T10:14:00.034 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ips.ko 2024-03-21T10:14:00.034 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci 2024-03-21T10:14:00.034 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci/isci.ko 2024-03-21T10:14:00.034 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-21T10:14:00.034 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-21T10:14:00.034 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc 2024-03-21T10:14:00.034 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc/libfc.ko 2024-03-21T10:14:00.034 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi.ko 2024-03-21T10:14:00.034 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-21T10:14:00.034 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas 2024-03-21T10:14:00.034 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas/libsas.ko 2024-03-21T10:14:00.034 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc 2024-03-21T10:14:00.034 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-21T10:14:00.034 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid 2024-03-21T10:14:00.035 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid.ko 2024-03-21T10:14:00.035 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-21T10:14:00.035 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-21T10:14:00.035 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-21T10:14:00.035 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas 2024-03-21T10:14:00.035 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-21T10:14:00.035 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas 2024-03-21T10:14:00.035 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-21T10:14:00.035 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvumi.ko 2024-03-21T10:14:00.035 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001 2024-03-21T10:14:00.035 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-21T10:14:00.035 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pmcraid.ko 2024-03-21T10:14:00.035 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla1280.ko 2024-03-21T10:14:00.035 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx 2024-03-21T10:14:00.035 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-21T10:14:00.036 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-21T10:14:00.036 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx 2024-03-21T10:14:00.036 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-21T10:14:00.036 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/raid_class.ko 2024-03-21T10:14:00.036 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_debug.ko 2024-03-21T10:14:00.036 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-21T10:14:00.036 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-21T10:14:00.036 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-21T10:14:00.036 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-21T10:14:00.036 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-21T10:14:00.036 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ses.ko 2024-03-21T10:14:00.036 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/st.ko 2024-03-21T10:14:00.036 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/stex.ko 2024-03-21T10:14:00.036 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2 2024-03-21T10:14:00.037 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-21T10:14:00.037 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/virtio_scsi.ko 2024-03-21T10:14:00.037 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-21T10:14:00.037 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/wd719x.ko 2024-03-21T10:14:00.037 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb 2024-03-21T10:14:00.037 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb/ssb.ko 2024-03-21T10:14:00.037 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target 2024-03-21T10:14:00.037 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi 2024-03-21T10:14:00.037 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-21T10:14:00.037 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback 2024-03-21T10:14:00.037 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-21T10:14:00.037 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_file.ko 2024-03-21T10:14:00.037 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_iblock.ko 2024-03-21T10:14:00.037 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_mod.ko 2024-03-21T10:14:00.038 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_pscsi.ko 2024-03-21T10:14:00.038 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_user.ko 2024-03-21T10:14:00.038 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc 2024-03-21T10:14:00.038 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-21T10:14:00.038 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal 2024-03-21T10:14:00.038 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel 2024-03-21T10:14:00.038 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal 2024-03-21T10:14:00.038 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-21T10:14:00.038 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-21T10:14:00.038 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-21T10:14:00.038 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-21T10:14:00.038 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-21T10:14:00.038 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-21T10:14:00.038 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-21T10:14:00.039 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-21T10:14:00.039 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-21T10:14:00.039 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-21T10:14:00.039 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-21T10:14:00.039 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-21T10:14:00.039 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-21T10:14:00.039 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-21T10:14:00.039 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-21T10:14:00.039 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-21T10:14:00.039 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-21T10:14:00.039 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-21T10:14:00.039 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty 2024-03-21T10:14:00.039 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless 2024-03-21T10:14:00.040 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-21T10:14:00.040 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_gsm.ko 2024-03-21T10:14:00.040 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_hdlc.ko 2024-03-21T10:14:00.040 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/nozomi.ko 2024-03-21T10:14:00.040 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial 2024-03-21T10:14:00.040 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250 2024-03-21T10:14:00.040 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-21T10:14:00.040 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/arc_uart.ko 2024-03-21T10:14:00.040 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm 2024-03-21T10:14:00.040 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-21T10:14:00.040 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs 2024-03-21T10:14:00.040 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core 2024-03-21T10:14:00.040 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-21T10:14:00.040 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host 2024-03-21T10:14:00.040 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-21T10:14:00.041 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio 2024-03-21T10:14:00.041 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio.ko 2024-03-21T10:14:00.041 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_aec.ko 2024-03-21T10:14:00.041 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_cif.ko 2024-03-21T10:14:00.041 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_pci_generic.ko 2024-03-21T10:14:00.041 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_sercos3.ko 2024-03-21T10:14:00.041 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb 2024-03-21T10:14:00.041 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class 2024-03-21T10:14:00.041 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-acm.ko 2024-03-21T10:14:00.041 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-21T10:14:00.041 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usblp.ko 2024-03-21T10:14:00.041 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usbtmc.ko 2024-03-21T10:14:00.041 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host 2024-03-21T10:14:00.041 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-21T10:14:00.042 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image 2024-03-21T10:14:00.042 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/mdc800.ko 2024-03-21T10:14:00.042 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/microtek.ko 2024-03-21T10:14:00.042 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc 2024-03-21T10:14:00.042 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/adutux.ko 2024-03-21T10:14:00.042 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/appledisplay.ko 2024-03-21T10:14:00.042 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi26.ko 2024-03-21T10:14:00.042 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi62.ko 2024-03-21T10:14:00.042 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ezusb.ko 2024-03-21T10:14:00.042 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/idmouse.ko 2024-03-21T10:14:00.042 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/iowarrior.ko 2024-03-21T10:14:00.042 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-21T10:14:00.042 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ldusb.ko 2024-03-21T10:14:00.042 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/legousbtower.ko 2024-03-21T10:14:00.043 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga 2024-03-21T10:14:00.043 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-21T10:14:00.043 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-21T10:14:00.043 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usb3503.ko 2024-03-21T10:14:00.043 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usblcd.ko 2024-03-21T10:14:00.043 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-21T10:14:00.043 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/uss720.ko 2024-03-21T10:14:00.043 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/yurex.ko 2024-03-21T10:14:00.043 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy 2024-03-21T10:14:00.043 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-generic.ko 2024-03-21T10:14:00.043 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial 2024-03-21T10:14:00.043 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/aircable.ko 2024-03-21T10:14:00.043 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ark3116.ko 2024-03-21T10:14:00.043 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-21T10:14:00.043 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ch341.ko 2024-03-21T10:14:00.044 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cp210x.ko 2024-03-21T10:14:00.044 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cyberjack.ko 2024-03-21T10:14:00.044 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-21T10:14:00.044 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-21T10:14:00.044 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/empeg.ko 2024-03-21T10:14:00.044 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-21T10:14:00.044 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-21T10:14:00.044 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-21T10:14:00.044 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_ti.ko 2024-03-21T10:14:00.044 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipaq.ko 2024-03-21T10:14:00.044 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipw.ko 2024-03-21T10:14:00.044 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ir-usb.ko 2024-03-21T10:14:00.044 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-21T10:14:00.044 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan.ko 2024-03-21T10:14:00.044 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-21T10:14:00.044 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-21T10:14:00.045 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-21T10:14:00.045 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mct_u232.ko 2024-03-21T10:14:00.045 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7720.ko 2024-03-21T10:14:00.045 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7840.ko 2024-03-21T10:14:00.045 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/navman.ko 2024-03-21T10:14:00.045 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/omninet.ko 2024-03-21T10:14:00.045 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/opticon.ko 2024-03-21T10:14:00.045 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/option.ko 2024-03-21T10:14:00.045 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/oti6858.ko 2024-03-21T10:14:00.045 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/pl2303.ko 2024-03-21T10:14:00.045 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcaux.ko 2024-03-21T10:14:00.045 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcserial.ko 2024-03-21T10:14:00.045 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/quatech2.ko 2024-03-21T10:14:00.045 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/safe_serial.ko 2024-03-21T10:14:00.045 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/sierra.ko 2024-03-21T10:14:00.045 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-21T10:14:00.046 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ssu100.ko 2024-03-21T10:14:00.046 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/symbolserial.ko 2024-03-21T10:14:00.046 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-21T10:14:00.046 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-21T10:14:00.046 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_debug.ko 2024-03-21T10:14:00.046 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-21T10:14:00.046 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/visor.ko 2024-03-21T10:14:00.046 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/whiteheat.ko 2024-03-21T10:14:00.046 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-21T10:14:00.046 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage 2024-03-21T10:14:00.046 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/uas.ko 2024-03-21T10:14:00.046 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-21T10:14:00.046 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-21T10:14:00.047 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-21T10:14:00.047 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-21T10:14:00.047 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-21T10:14:00.047 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-21T10:14:00.047 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-21T10:14:00.047 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-karma.ko 2024-03-21T10:14:00.047 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-21T10:14:00.047 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-21T10:14:00.047 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-21T10:14:00.047 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-21T10:14:00.047 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-21T10:14:00.047 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/usb-storage.ko 2024-03-21T10:14:00.047 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip 2024-03-21T10:14:00.047 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-21T10:14:00.047 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-21T10:14:00.048 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-21T10:14:00.048 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio 2024-03-21T10:14:00.048 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci 2024-03-21T10:14:00.048 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-21T10:14:00.048 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-21T10:14:00.048 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio.ko 2024-03-21T10:14:00.048 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-21T10:14:00.048 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost 2024-03-21T10:14:00.048 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost.ko 2024-03-21T10:14:00.048 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-21T10:14:00.048 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_net.ko 2024-03-21T10:14:00.048 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_scsi.ko 2024-03-21T10:14:00.048 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video 2024-03-21T10:14:00.048 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight 2024-03-21T10:14:00.048 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/apple_bl.ko 2024-03-21T10:14:00.049 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lcd.ko 2024-03-21T10:14:00.049 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-21T10:14:00.049 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio 2024-03-21T10:14:00.049 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio.ko 2024-03-21T10:14:00.049 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_balloon.ko 2024-03-21T10:14:00.049 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_mmio.ko 2024-03-21T10:14:00.049 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci.ko 2024-03-21T10:14:00.049 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-21T10:14:00.049 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-21T10:14:00.049 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_ring.ko 2024-03-21T10:14:00.049 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1 2024-03-21T10:14:00.049 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters 2024-03-21T10:14:00.049 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2482.ko 2024-03-21T10:14:00.050 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2490.ko 2024-03-21T10:14:00.050 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves 2024-03-21T10:14:00.050 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-21T10:14:00.050 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-21T10:14:00.050 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-21T10:14:00.050 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-21T10:14:00.050 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-21T10:14:00.050 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-21T10:14:00.050 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-21T10:14:00.050 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-21T10:14:00.050 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-21T10:14:00.050 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-21T10:14:00.050 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-21T10:14:00.050 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/wire.ko 2024-03-21T10:14:00.050 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog 2024-03-21T10:14:00.051 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-21T10:14:00.051 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-21T10:14:00.051 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-21T10:14:00.051 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/hpwdt.ko 2024-03-21T10:14:00.051 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/i6300esb.ko 2024-03-21T10:14:00.051 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-21T10:14:00.051 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-21T10:14:00.051 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ib700wdt.ko 2024-03-21T10:14:00.051 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ibmasr.ko 2024-03-21T10:14:00.051 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-21T10:14:00.051 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-21T10:14:00.051 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it87_wdt.ko 2024-03-21T10:14:00.051 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/machzwd.ko 2024-03-21T10:14:00.051 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/nv_tco.ko 2024-03-21T10:14:00.052 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-21T10:14:00.052 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-21T10:14:00.052 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-21T10:14:00.052 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-21T10:14:00.052 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/softdog.ko 2024-03-21T10:14:00.052 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-21T10:14:00.052 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/via_wdt.ko 2024-03-21T10:14:00.052 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-21T10:14:00.052 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-21T10:14:00.052 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-21T10:14:00.052 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wdt_pci.ko 2024-03-21T10:14:00.052 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/xen_wdt.ko 2024-03-21T10:14:00.052 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen 2024-03-21T10:14:00.052 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-21T10:14:00.053 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-evtchn.ko 2024-03-21T10:14:00.053 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntalloc.ko 2024-03-21T10:14:00.053 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntdev.ko 2024-03-21T10:14:00.053 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback 2024-03-21T10:14:00.053 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-21T10:14:00.053 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-privcmd.ko 2024-03-21T10:14:00.053 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs 2024-03-21T10:14:00.053 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-21T10:14:00.053 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs 2024-03-21T10:14:00.053 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p 2024-03-21T10:14:00.053 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p/9p.ko 2024-03-21T10:14:00.053 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs 2024-03-21T10:14:00.053 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs/affs.ko 2024-03-21T10:14:00.053 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs 2024-03-21T10:14:00.054 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs/befs.ko 2024-03-21T10:14:00.054 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/binfmt_misc.ko 2024-03-21T10:14:00.054 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs 2024-03-21T10:14:00.054 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs/btrfs.ko 2024-03-21T10:14:00.054 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles 2024-03-21T10:14:00.054 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles/cachefiles.ko 2024-03-21T10:14:00.054 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph 2024-03-21T10:14:00.054 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph/ceph.ko 2024-03-21T10:14:00.054 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda 2024-03-21T10:14:00.054 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda/coda.ko 2024-03-21T10:14:00.054 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs 2024-03-21T10:14:00.054 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs/cramfs.ko 2024-03-21T10:14:00.054 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm 2024-03-21T10:14:00.054 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm/dlm.ko 2024-03-21T10:14:00.055 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs 2024-03-21T10:14:00.055 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-21T10:14:00.055 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs 2024-03-21T10:14:00.055 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs/f2fs.ko 2024-03-21T10:14:00.055 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat 2024-03-21T10:14:00.055 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/fat.ko 2024-03-21T10:14:00.055 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/msdos.ko 2024-03-21T10:14:00.055 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/vfat.ko 2024-03-21T10:14:00.055 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse 2024-03-21T10:14:00.055 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/cuse.ko 2024-03-21T10:14:00.055 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/fuse.ko 2024-03-21T10:14:00.055 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2 2024-03-21T10:14:00.055 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2/gfs2.ko 2024-03-21T10:14:00.055 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs 2024-03-21T10:14:00.056 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs/hfs.ko 2024-03-21T10:14:00.056 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus 2024-03-21T10:14:00.056 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus/hfsplus.ko 2024-03-21T10:14:00.056 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs 2024-03-21T10:14:00.056 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs/isofs.ko 2024-03-21T10:14:00.056 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs 2024-03-21T10:14:00.056 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs/jfs.ko 2024-03-21T10:14:00.056 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd 2024-03-21T10:14:00.056 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd/lockd.ko 2024-03-21T10:14:00.056 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix 2024-03-21T10:14:00.056 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix/minix.ko 2024-03-21T10:14:00.056 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs 2024-03-21T10:14:00.056 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs/netfs.ko 2024-03-21T10:14:00.056 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs 2024-03-21T10:14:00.056 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout 2024-03-21T10:14:00.057 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-21T10:14:00.057 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout 2024-03-21T10:14:00.057 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-21T10:14:00.057 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout 2024-03-21T10:14:00.057 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-21T10:14:00.057 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfs.ko 2024-03-21T10:14:00.057 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv3.ko 2024-03-21T10:14:00.057 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv4.ko 2024-03-21T10:14:00.057 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common 2024-03-21T10:14:00.057 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/grace.ko 2024-03-21T10:14:00.057 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/nfs_acl.ko 2024-03-21T10:14:00.057 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd 2024-03-21T10:14:00.057 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd/nfsd.ko 2024-03-21T10:14:00.057 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2 2024-03-21T10:14:00.058 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2/nilfs2.ko 2024-03-21T10:14:00.058 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls 2024-03-21T10:14:00.058 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-celtic.ko 2024-03-21T10:14:00.058 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-centeuro.ko 2024-03-21T10:14:00.058 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-croatian.ko 2024-03-21T10:14:00.058 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-cyrillic.ko 2024-03-21T10:14:00.058 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-gaelic.ko 2024-03-21T10:14:00.058 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-greek.ko 2024-03-21T10:14:00.058 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-iceland.ko 2024-03-21T10:14:00.058 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-inuit.ko 2024-03-21T10:14:00.058 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-roman.ko 2024-03-21T10:14:00.058 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-romanian.ko 2024-03-21T10:14:00.058 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-turkish.ko 2024-03-21T10:14:00.058 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1250.ko 2024-03-21T10:14:00.059 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1251.ko 2024-03-21T10:14:00.059 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1255.ko 2024-03-21T10:14:00.059 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp737.ko 2024-03-21T10:14:00.059 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp775.ko 2024-03-21T10:14:00.059 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp850.ko 2024-03-21T10:14:00.059 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp852.ko 2024-03-21T10:14:00.059 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp855.ko 2024-03-21T10:14:00.059 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp857.ko 2024-03-21T10:14:00.059 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp860.ko 2024-03-21T10:14:00.059 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp861.ko 2024-03-21T10:14:00.059 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp862.ko 2024-03-21T10:14:00.059 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp863.ko 2024-03-21T10:14:00.059 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp864.ko 2024-03-21T10:14:00.059 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp865.ko 2024-03-21T10:14:00.059 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp866.ko 2024-03-21T10:14:00.060 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp869.ko 2024-03-21T10:14:00.060 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp874.ko 2024-03-21T10:14:00.060 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp932.ko 2024-03-21T10:14:00.060 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp936.ko 2024-03-21T10:14:00.060 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp949.ko 2024-03-21T10:14:00.060 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp950.ko 2024-03-21T10:14:00.060 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_euc-jp.ko 2024-03-21T10:14:00.060 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-1.ko 2024-03-21T10:14:00.060 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-13.ko 2024-03-21T10:14:00.060 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-14.ko 2024-03-21T10:14:00.060 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-15.ko 2024-03-21T10:14:00.060 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-2.ko 2024-03-21T10:14:00.060 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-3.ko 2024-03-21T10:14:00.060 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-4.ko 2024-03-21T10:14:00.060 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-5.ko 2024-03-21T10:14:00.061 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-6.ko 2024-03-21T10:14:00.061 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-7.ko 2024-03-21T10:14:00.061 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-9.ko 2024-03-21T10:14:00.061 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-r.ko 2024-03-21T10:14:00.061 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-ru.ko 2024-03-21T10:14:00.061 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-u.ko 2024-03-21T10:14:00.061 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-21T10:14:00.061 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_utf8.ko 2024-03-21T10:14:00.061 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2 2024-03-21T10:14:00.061 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster 2024-03-21T10:14:00.061 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-21T10:14:00.061 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm 2024-03-21T10:14:00.061 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-21T10:14:00.061 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs 2024-03-21T10:14:00.062 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-21T10:14:00.062 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2.ko 2024-03-21T10:14:00.062 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-21T10:14:00.062 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-21T10:14:00.062 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-21T10:14:00.062 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs 2024-03-21T10:14:00.062 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs/overlay.ko 2024-03-21T10:14:00.062 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore 2024-03-21T10:14:00.062 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore/ramoops.ko 2024-03-21T10:14:00.062 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs 2024-03-21T10:14:00.062 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs/reiserfs.ko 2024-03-21T10:14:00.062 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs 2024-03-21T10:14:00.062 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs/romfs.ko 2024-03-21T10:14:00.062 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb 2024-03-21T10:14:00.063 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client 2024-03-21T10:14:00.063 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client/cifs.ko 2024-03-21T10:14:00.063 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common 2024-03-21T10:14:00.063 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_arc4.ko 2024-03-21T10:14:00.063 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_md4.ko 2024-03-21T10:14:00.063 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs 2024-03-21T10:14:00.063 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs/squashfs.ko 2024-03-21T10:14:00.063 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv 2024-03-21T10:14:00.063 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv/sysv.ko 2024-03-21T10:14:00.063 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf 2024-03-21T10:14:00.063 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf/udf.ko 2024-03-21T10:14:00.063 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs 2024-03-21T10:14:00.063 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs/ufs.ko 2024-03-21T10:14:00.063 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs 2024-03-21T10:14:00.064 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs/xfs.ko 2024-03-21T10:14:00.064 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel 2024-03-21T10:14:00.064 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu 2024-03-21T10:14:00.064 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu/rcutorture.ko 2024-03-21T10:14:00.064 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/torture.ko 2024-03-21T10:14:00.064 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace 2024-03-21T10:14:00.064 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-21T10:14:00.064 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib 2024-03-21T10:14:00.064 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/asn1_encoder.ko 2024-03-21T10:14:00.064 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc-itu-t.ko 2024-03-21T10:14:00.064 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc8.ko 2024-03-21T10:14:00.064 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto 2024-03-21T10:14:00.064 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libarc4.ko 2024-03-21T10:14:00.064 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libdes.ko 2024-03-21T10:14:00.065 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/libcrc32c.ko 2024-03-21T10:14:00.065 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lru_cache.ko 2024-03-21T10:14:00.065 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4 2024-03-21T10:14:00.065 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4_compress.ko 2024-03-21T10:14:00.065 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4hc_compress.ko 2024-03-21T10:14:00.065 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math 2024-03-21T10:14:00.065 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math/cordic.ko 2024-03-21T10:14:00.065 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6 2024-03-21T10:14:00.065 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6/raid6_pq.ko 2024-03-21T10:14:00.065 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon 2024-03-21T10:14:00.065 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-21T10:14:00.065 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_bm.ko 2024-03-21T10:14:00.065 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_fsm.ko 2024-03-21T10:14:00.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_kmp.ko 2024-03-21T10:14:00.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd 2024-03-21T10:14:00.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd/zstd_compress.ko 2024-03-21T10:14:00.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm 2024-03-21T10:14:00.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm/hwpoison-inject.ko 2024-03-21T10:14:00.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net 2024-03-21T10:14:00.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan 2024-03-21T10:14:00.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/6lowpan.ko 2024-03-21T10:14:00.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_dest.ko 2024-03-21T10:14:00.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_fragment.ko 2024-03-21T10:14:00.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_hop.ko 2024-03-21T10:14:00.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-21T10:14:00.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_mobility.ko 2024-03-21T10:14:00.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_routing.ko 2024-03-21T10:14:00.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_udp.ko 2024-03-21T10:14:00.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802 2024-03-21T10:14:00.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/garp.ko 2024-03-21T10:14:00.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/mrp.ko 2024-03-21T10:14:00.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/p8022.ko 2024-03-21T10:14:00.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/psnap.ko 2024-03-21T10:14:00.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/stp.ko 2024-03-21T10:14:00.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q 2024-03-21T10:14:00.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q/8021q.ko 2024-03-21T10:14:00.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p 2024-03-21T10:14:00.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet.ko 2024-03-21T10:14:00.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_fd.ko 2024-03-21T10:14:00.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_rdma.ko 2024-03-21T10:14:00.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_virtio.ko 2024-03-21T10:14:00.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv 2024-03-21T10:14:00.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv/batman-adv.ko 2024-03-21T10:14:00.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge 2024-03-21T10:14:00.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/br_netfilter.ko 2024-03-21T10:14:00.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/bridge.ko 2024-03-21T10:14:00.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter 2024-03-21T10:14:00.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-21T10:14:00.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-21T10:14:00.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-21T10:14:00.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-21T10:14:00.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-21T10:14:00.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-21T10:14:00.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-21T10:14:00.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-21T10:14:00.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-21T10:14:00.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-21T10:14:00.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-21T10:14:00.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-21T10:14:00.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-21T10:14:00.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-21T10:14:00.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-21T10:14:00.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-21T10:14:00.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-21T10:14:00.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-21T10:14:00.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-21T10:14:00.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-21T10:14:00.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtables.ko 2024-03-21T10:14:00.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-21T10:14:00.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-21T10:14:00.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-21T10:14:00.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph 2024-03-21T10:14:00.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph/libceph.ko 2024-03-21T10:14:00.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core 2024-03-21T10:14:00.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/failover.ko 2024-03-21T10:14:00.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/pktgen.ko 2024-03-21T10:14:00.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp 2024-03-21T10:14:00.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp.ko 2024-03-21T10:14:00.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_diag.ko 2024-03-21T10:14:00.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv4.ko 2024-03-21T10:14:00.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv6.ko 2024-03-21T10:14:00.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver 2024-03-21T10:14:00.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver/dns_resolver.ko 2024-03-21T10:14:00.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa 2024-03-21T10:14:00.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/dsa_core.ko 2024-03-21T10:14:00.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_brcm.ko 2024-03-21T10:14:00.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_dsa.ko 2024-03-21T10:14:00.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_none.ko 2024-03-21T10:14:00.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_trailer.ko 2024-03-21T10:14:00.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154 2024-03-21T10:14:00.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan 2024-03-21T10:14:00.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-21T10:14:00.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154.ko 2024-03-21T10:14:00.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-21T10:14:00.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4 2024-03-21T10:14:00.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ah4.ko 2024-03-21T10:14:00.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/esp4.ko 2024-03-21T10:14:00.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/fou.ko 2024-03-21T10:14:00.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/gre.ko 2024-03-21T10:14:00.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/inet_diag.ko 2024-03-21T10:14:00.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_gre.ko 2024-03-21T10:14:00.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_tunnel.ko 2024-03-21T10:14:00.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_vti.ko 2024-03-21T10:14:00.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipcomp.ko 2024-03-21T10:14:00.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipip.ko 2024-03-21T10:14:00.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter 2024-03-21T10:14:00.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-21T10:14:00.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-21T10:14:00.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-21T10:14:00.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-21T10:14:00.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-21T10:14:00.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-21T10:14:00.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-21T10:14:00.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-21T10:14:00.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-21T10:14:00.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-21T10:14:00.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-21T10:14:00.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-21T10:14:00.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-21T10:14:00.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-21T10:14:00.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-21T10:14:00.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-21T10:14:00.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-21T10:14:00.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-21T10:14:00.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-21T10:14:00.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-21T10:14:00.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-21T10:14:00.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_bic.ko 2024-03-21T10:14:00.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_dctcp.ko 2024-03-21T10:14:00.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_diag.ko 2024-03-21T10:14:00.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_highspeed.ko 2024-03-21T10:14:00.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_htcp.ko 2024-03-21T10:14:00.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_hybla.ko 2024-03-21T10:14:00.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_illinois.ko 2024-03-21T10:14:00.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_lp.ko 2024-03-21T10:14:00.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_scalable.ko 2024-03-21T10:14:00.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_vegas.ko 2024-03-21T10:14:00.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_veno.ko 2024-03-21T10:14:00.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_westwood.ko 2024-03-21T10:14:00.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_yeah.ko 2024-03-21T10:14:00.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tunnel4.ko 2024-03-21T10:14:00.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_diag.ko 2024-03-21T10:14:00.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_tunnel.ko 2024-03-21T10:14:00.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-21T10:14:00.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6 2024-03-21T10:14:00.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ah6.ko 2024-03-21T10:14:00.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/esp6.ko 2024-03-21T10:14:00.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/fou6.ko 2024-03-21T10:14:00.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_tunnel.ko 2024-03-21T10:14:00.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-21T10:14:00.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_vti.ko 2024-03-21T10:14:00.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ipcomp6.ko 2024-03-21T10:14:00.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter 2024-03-21T10:14:00.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-21T10:14:00.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-21T10:14:00.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-21T10:14:00.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-21T10:14:00.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-21T10:14:00.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-21T10:14:00.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-21T10:14:00.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-21T10:14:00.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-21T10:14:00.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-21T10:14:00.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-21T10:14:00.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-21T10:14:00.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-21T10:14:00.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-21T10:14:00.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-21T10:14:00.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-21T10:14:00.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-21T10:14:00.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-21T10:14:00.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-21T10:14:00.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-21T10:14:00.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-21T10:14:00.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-21T10:14:00.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-21T10:14:00.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-21T10:14:00.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/sit.ko 2024-03-21T10:14:00.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/tunnel6.ko 2024-03-21T10:14:00.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-21T10:14:00.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key 2024-03-21T10:14:00.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key/af_key.ko 2024-03-21T10:14:00.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp 2024-03-21T10:14:00.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_core.ko 2024-03-21T10:14:00.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-21T10:14:00.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_eth.ko 2024-03-21T10:14:00.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip.ko 2024-03-21T10:14:00.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip6.ko 2024-03-21T10:14:00.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_netlink.ko 2024-03-21T10:14:00.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ppp.ko 2024-03-21T10:14:00.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc 2024-03-21T10:14:00.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc/llc.ko 2024-03-21T10:14:00.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154 2024-03-21T10:14:00.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154/mac802154.ko 2024-03-21T10:14:00.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls 2024-03-21T10:14:00.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls/mpls_gso.ko 2024-03-21T10:14:00.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter 2024-03-21T10:14:00.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset 2024-03-21T10:14:00.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set.ko 2024-03-21T10:14:00.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-21T10:14:00.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-21T10:14:00.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-21T10:14:00.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-21T10:14:00.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-21T10:14:00.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-21T10:14:00.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-21T10:14:00.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-21T10:14:00.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-21T10:14:00.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-21T10:14:00.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-21T10:14:00.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-21T10:14:00.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-21T10:14:00.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-21T10:14:00.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-21T10:14:00.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs 2024-03-21T10:14:00.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-21T10:14:00.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-21T10:14:00.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-21T10:14:00.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-21T10:14:00.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-21T10:14:00.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-21T10:14:00.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-21T10:14:00.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-21T10:14:00.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-21T10:14:00.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-21T10:14:00.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-21T10:14:00.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-21T10:14:00.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-21T10:14:00.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-21T10:14:00.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conncount.ko 2024-03-21T10:14:00.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack.ko 2024-03-21T10:14:00.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-21T10:14:00.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-21T10:14:00.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-21T10:14:00.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-21T10:14:00.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-21T10:14:00.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-21T10:14:00.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-21T10:14:00.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-21T10:14:00.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-21T10:14:00.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-21T10:14:00.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-21T10:14:00.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-21T10:14:00.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-21T10:14:00.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table.ko 2024-03-21T10:14:00.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-21T10:14:00.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_log_syslog.ko 2024-03-21T10:14:00.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat.ko 2024-03-21T10:14:00.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-21T10:14:00.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-21T10:14:00.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_irc.ko 2024-03-21T10:14:00.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_sip.ko 2024-03-21T10:14:00.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-21T10:14:00.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-21T10:14:00.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_tables.ko 2024-03-21T10:14:00.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink.ko 2024-03-21T10:14:00.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-21T10:14:00.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_log.ko 2024-03-21T10:14:00.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-21T10:14:00.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-21T10:14:00.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_chain_nat.ko 2024-03-21T10:14:00.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_compat.ko 2024-03-21T10:14:00.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_ct.ko 2024-03-21T10:14:00.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-21T10:14:00.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fib.ko 2024-03-21T10:14:00.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-21T10:14:00.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_hash.ko 2024-03-21T10:14:00.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_limit.ko 2024-03-21T10:14:00.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_log.ko 2024-03-21T10:14:00.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_masq.ko 2024-03-21T10:14:00.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_nat.ko 2024-03-21T10:14:00.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_numgen.ko 2024-03-21T10:14:00.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_queue.ko 2024-03-21T10:14:00.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_quota.ko 2024-03-21T10:14:00.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_redir.ko 2024-03-21T10:14:00.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject.ko 2024-03-21T10:14:00.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject_inet.ko 2024-03-21T10:14:00.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_socket.ko 2024-03-21T10:14:00.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_tunnel.ko 2024-03-21T10:14:00.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_AUDIT.ko 2024-03-21T10:14:00.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-21T10:14:00.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-21T10:14:00.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-21T10:14:00.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CT.ko 2024-03-21T10:14:00.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_DSCP.ko 2024-03-21T10:14:00.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HL.ko 2024-03-21T10:14:00.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HMARK.ko 2024-03-21T10:14:00.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-21T10:14:00.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LED.ko 2024-03-21T10:14:00.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LOG.ko 2024-03-21T10:14:00.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-21T10:14:00.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NETMAP.ko 2024-03-21T10:14:00.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFLOG.ko 2024-03-21T10:14:00.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-21T10:14:00.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_RATEEST.ko 2024-03-21T10:14:00.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-21T10:14:00.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_SECMARK.ko 2024-03-21T10:14:00.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-21T10:14:00.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-21T10:14:00.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TEE.ko 2024-03-21T10:14:00.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TPROXY.ko 2024-03-21T10:14:00.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TRACE.ko 2024-03-21T10:14:00.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_addrtype.ko 2024-03-21T10:14:00.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_bpf.ko 2024-03-21T10:14:00.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cgroup.ko 2024-03-21T10:14:00.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cluster.ko 2024-03-21T10:14:00.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_comment.ko 2024-03-21T10:14:00.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connbytes.ko 2024-03-21T10:14:00.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlabel.ko 2024-03-21T10:14:00.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlimit.ko 2024-03-21T10:14:00.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connmark.ko 2024-03-21T10:14:00.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_conntrack.ko 2024-03-21T10:14:00.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cpu.ko 2024-03-21T10:14:00.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dccp.ko 2024-03-21T10:14:00.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_devgroup.ko 2024-03-21T10:14:00.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dscp.ko 2024-03-21T10:14:00.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ecn.ko 2024-03-21T10:14:00.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_esp.ko 2024-03-21T10:14:00.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hashlimit.ko 2024-03-21T10:14:00.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_helper.ko 2024-03-21T10:14:00.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hl.ko 2024-03-21T10:14:00.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipcomp.ko 2024-03-21T10:14:00.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_iprange.ko 2024-03-21T10:14:00.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipvs.ko 2024-03-21T10:14:00.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_l2tp.ko 2024-03-21T10:14:00.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_length.ko 2024-03-21T10:14:00.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_limit.ko 2024-03-21T10:14:00.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mac.ko 2024-03-21T10:14:00.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mark.ko 2024-03-21T10:14:00.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_multiport.ko 2024-03-21T10:14:00.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nat.ko 2024-03-21T10:14:00.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nfacct.ko 2024-03-21T10:14:00.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_osf.ko 2024-03-21T10:14:00.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_owner.ko 2024-03-21T10:14:00.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_physdev.ko 2024-03-21T10:14:00.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_pkttype.ko 2024-03-21T10:14:00.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_policy.ko 2024-03-21T10:14:00.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_quota.ko 2024-03-21T10:14:00.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_rateest.ko 2024-03-21T10:14:00.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_realm.ko 2024-03-21T10:14:00.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_recent.ko 2024-03-21T10:14:00.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_sctp.ko 2024-03-21T10:14:00.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_set.ko 2024-03-21T10:14:00.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_socket.ko 2024-03-21T10:14:00.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_state.ko 2024-03-21T10:14:00.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_statistic.ko 2024-03-21T10:14:00.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_string.ko 2024-03-21T10:14:00.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_tcpmss.ko 2024-03-21T10:14:00.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_time.ko 2024-03-21T10:14:00.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_u32.ko 2024-03-21T10:14:00.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink 2024-03-21T10:14:00.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink/netlink_diag.ko 2024-03-21T10:14:00.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh 2024-03-21T10:14:00.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh/nsh.ko 2024-03-21T10:14:00.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch 2024-03-21T10:14:00.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/openvswitch.ko 2024-03-21T10:14:00.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-geneve.ko 2024-03-21T10:14:00.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-gre.ko 2024-03-21T10:14:00.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-vxlan.ko 2024-03-21T10:14:00.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet 2024-03-21T10:14:00.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet/af_packet_diag.ko 2024-03-21T10:14:00.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds 2024-03-21T10:14:00.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds.ko 2024-03-21T10:14:00.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_rdma.ko 2024-03-21T10:14:00.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_tcp.ko 2024-03-21T10:14:00.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched 2024-03-21T10:14:00.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_csum.ko 2024-03-21T10:14:00.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_gact.ko 2024-03-21T10:14:00.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_mirred.ko 2024-03-21T10:14:00.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_nat.ko 2024-03-21T10:14:00.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_pedit.ko 2024-03-21T10:14:00.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_police.ko 2024-03-21T10:14:00.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_simple.ko 2024-03-21T10:14:00.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_skbedit.ko 2024-03-21T10:14:00.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_vlan.ko 2024-03-21T10:14:00.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_basic.ko 2024-03-21T10:14:00.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_bpf.ko 2024-03-21T10:14:00.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_flow.ko 2024-03-21T10:14:00.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_fw.ko 2024-03-21T10:14:00.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_route.ko 2024-03-21T10:14:00.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_u32.ko 2024-03-21T10:14:00.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_cmp.ko 2024-03-21T10:14:00.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_ipset.ko 2024-03-21T10:14:00.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_meta.ko 2024-03-21T10:14:00.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_nbyte.ko 2024-03-21T10:14:00.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_text.ko 2024-03-21T10:14:00.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_u32.ko 2024-03-21T10:14:00.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_choke.ko 2024-03-21T10:14:00.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_codel.ko 2024-03-21T10:14:00.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_drr.ko 2024-03-21T10:14:00.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_fq.ko 2024-03-21T10:14:00.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_gred.ko 2024-03-21T10:14:00.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hfsc.ko 2024-03-21T10:14:00.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hhf.ko 2024-03-21T10:14:00.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_htb.ko 2024-03-21T10:14:00.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_ingress.ko 2024-03-21T10:14:00.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio.ko 2024-03-21T10:14:00.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio_lib.ko 2024-03-21T10:14:00.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_multiq.ko 2024-03-21T10:14:00.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_netem.ko 2024-03-21T10:14:00.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_pie.ko 2024-03-21T10:14:00.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_plug.ko 2024-03-21T10:14:00.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_prio.ko 2024-03-21T10:14:00.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_qfq.ko 2024-03-21T10:14:00.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_red.ko 2024-03-21T10:14:00.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfb.ko 2024-03-21T10:14:00.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfq.ko 2024-03-21T10:14:00.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_tbf.ko 2024-03-21T10:14:00.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_teql.ko 2024-03-21T10:14:00.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp 2024-03-21T10:14:00.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp.ko 2024-03-21T10:14:00.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp_diag.ko 2024-03-21T10:14:00.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc 2024-03-21T10:14:00.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss 2024-03-21T10:14:00.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-21T10:14:00.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-21T10:14:00.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/sunrpc.ko 2024-03-21T10:14:00.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma 2024-03-21T10:14:00.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-21T10:14:00.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc 2024-03-21T10:14:00.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/diag.ko 2024-03-21T10:14:00.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/tipc.ko 2024-03-21T10:14:00.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix 2024-03-21T10:14:00.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix/unix_diag.ko 2024-03-21T10:14:00.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock 2024-03-21T10:14:00.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-21T10:14:00.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-21T10:14:00.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock.ko 2024-03-21T10:14:00.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-21T10:14:00.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-21T10:14:00.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm 2024-03-21T10:14:00.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-21T10:14:00.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security 2024-03-21T10:14:00.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys 2024-03-21T10:14:00.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys 2024-03-21T10:14:00.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-21T10:14:00.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys 2024-03-21T10:14:00.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys/trusted.ko 2024-03-21T10:14:00.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt 2024-03-21T10:14:00.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib 2024-03-21T10:14:00.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib/irqbypass.ko 2024-03-21T10:14:00.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias 2024-03-21T10:14:00.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias.bin 2024-03-21T10:14:00.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin 2024-03-21T10:14:00.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.alias.bin 2024-03-21T10:14:00.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.bin 2024-03-21T10:14:00.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.modinfo 2024-03-21T10:14:00.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep 2024-03-21T10:14:00.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep.bin 2024-03-21T10:14:00.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.devname 2024-03-21T10:14:00.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.order 2024-03-21T10:14:00.097 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.softdep 2024-03-21T10:14:00.097 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols 2024-03-21T10:14:00.097 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols.bin 2024-03-21T10:14:00.097 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-21T10:14:00.101 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g5ec3e6e88b2e 2024-03-21T10:14:00.101 DEBUG:teuthology.orchestra.run.smithi031:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-21T10:14:00.125 INFO:teuthology.orchestra.run.smithi031.stdout:postinstall scriptlet (using /bin/sh): 2024-03-21T10:14:00.126 INFO:teuthology.orchestra.run.smithi031.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-21T10:14:00.126 INFO:teuthology.orchestra.run.smithi031.stdout: /usr/bin/kernel-install add 6.8.0-g5ec3e6e88b2e /lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-21T10:14:00.126 INFO:teuthology.orchestra.run.smithi031.stdout:fi 2024-03-21T10:14:00.126 INFO:teuthology.orchestra.run.smithi031.stdout:for file in vmlinuz System.map config; do 2024-03-21T10:14:00.126 INFO:teuthology.orchestra.run.smithi031.stdout: if ! cmp --silent "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e"; then 2024-03-21T10:14:00.126 INFO:teuthology.orchestra.run.smithi031.stdout: cp "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e" 2024-03-21T10:14:00.126 INFO:teuthology.orchestra.run.smithi031.stdout: fi 2024-03-21T10:14:00.126 INFO:teuthology.orchestra.run.smithi031.stdout:done 2024-03-21T10:14:00.126 INFO:teuthology.orchestra.run.smithi031.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-21T10:14:00.126 INFO:teuthology.orchestra.run.smithi031.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-21T10:14:00.126 INFO:teuthology.orchestra.run.smithi031.stdout:new-kernel-pkg --remove 6.8.0-g5ec3e6e88b2e --rminitrd --initrdfile=/boot/initramfs-6.8.0-g5ec3e6e88b2e.img 2024-03-21T10:14:00.126 INFO:teuthology.orchestra.run.smithi031.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-21T10:14:00.126 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-install remove 6.8.0-g5ec3e6e88b2e 2024-03-21T10:14:00.127 INFO:teuthology.orchestra.run.smithi031.stdout:fi 2024-03-21T10:14:00.127 INFO:teuthology.orchestra.run.smithi031.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-21T10:14:00.127 INFO:teuthology.orchestra.run.smithi031.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-21T10:14:00.127 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/update-bootloader --remove 6.8.0-g5ec3e6e88b2e 2024-03-21T10:14:00.127 INFO:teuthology.orchestra.run.smithi031.stdout:fi 2024-03-21T10:14:00.128 DEBUG:teuthology.orchestra.run.smithi031:> sudo rpm -qi grub2-tools 2024-03-21T10:14:00.214 INFO:teuthology.orchestra.run.smithi031.stdout:Name : grub2-tools 2024-03-21T10:14:00.215 INFO:teuthology.orchestra.run.smithi031.stdout:Epoch : 1 2024-03-21T10:14:00.215 INFO:teuthology.orchestra.run.smithi031.stdout:Version : 2.06 2024-03-21T10:14:00.215 INFO:teuthology.orchestra.run.smithi031.stdout:Release : 68.el9 2024-03-21T10:14:00.215 INFO:teuthology.orchestra.run.smithi031.stdout:Architecture: x86_64 2024-03-21T10:14:00.215 INFO:teuthology.orchestra.run.smithi031.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-21T10:14:00.215 INFO:teuthology.orchestra.run.smithi031.stdout:Group : Unspecified 2024-03-21T10:14:00.215 INFO:teuthology.orchestra.run.smithi031.stdout:Size : 8264254 2024-03-21T10:14:00.215 INFO:teuthology.orchestra.run.smithi031.stdout:License : GPLv3+ 2024-03-21T10:14:00.215 INFO:teuthology.orchestra.run.smithi031.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-21T10:14:00.215 INFO:teuthology.orchestra.run.smithi031.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-21T10:14:00.215 INFO:teuthology.orchestra.run.smithi031.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-21T10:14:00.215 INFO:teuthology.orchestra.run.smithi031.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-21T10:14:00.216 INFO:teuthology.orchestra.run.smithi031.stdout:Packager : builder@centos.org 2024-03-21T10:14:00.216 INFO:teuthology.orchestra.run.smithi031.stdout:Vendor : CentOS 2024-03-21T10:14:00.216 INFO:teuthology.orchestra.run.smithi031.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-21T10:14:00.216 INFO:teuthology.orchestra.run.smithi031.stdout:Summary : Support tools for GRUB. 2024-03-21T10:14:00.216 INFO:teuthology.orchestra.run.smithi031.stdout:Description : 2024-03-21T10:14:00.216 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:14:00.216 INFO:teuthology.orchestra.run.smithi031.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-21T10:14:00.216 INFO:teuthology.orchestra.run.smithi031.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-21T10:14:00.216 INFO:teuthology.orchestra.run.smithi031.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-21T10:14:00.216 INFO:teuthology.orchestra.run.smithi031.stdout:hardware devices. 2024-03-21T10:14:00.216 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:14:00.216 INFO:teuthology.orchestra.run.smithi031.stdout:This subpackage provides tools for support of all platforms. 2024-03-21T10:14:00.217 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-21T10:14:00.217 INFO:teuthology.task.kernel:Updating grub on smithi031 to boot 6.8.0-g5ec3e6e88b2e 2024-03-21T10:14:00.217 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-21T10:14:00.774 INFO:teuthology.orchestra.run.smithi031.stderr:Generating grub configuration file ... 2024-03-21T10:14:02.126 INFO:teuthology.orchestra.run.smithi031.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-21T10:14:02.162 INFO:teuthology.orchestra.run.smithi031.stderr:done 2024-03-21T10:14:02.165 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2024-03-21T10:14:02.180 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.UOruetGi9K 2024-03-21T10:14:02.180 DEBUG:teuthology.orchestra.run.smithi031:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.UOruetGi9K 2024-03-21T10:14:02.252 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.UOruetGi9K 2024-03-21T10:14:02.481 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.UOruetGi9K is 6KB 2024-03-21T10:14:02.530 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.UOruetGi9K 2024-03-21T10:14:02.545 DEBUG:teuthology.orchestra.run.smithi031:> sudo /bin/ls /boot/loader/entries || true 2024-03-21T10:14:02.613 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-21T10:14:02.613 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-21T10:14:02.613 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-21T10:14:02.613 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-6.8.0-g5ec3e6e88b2e.conf 2024-03-21T10:14:02.614 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-g5ec3e6e88b2e 2024-03-21T10:14:02.827 DEBUG:teuthology.orchestra.run.smithi031:> sudo shutdown -r now 2024-03-21T10:14:32.829 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-21T10:14:32.830 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-21T10:14:32.831 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T10:14:51.166 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-21T10:15:00.168 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-21T10:15:00.169 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T10:15:18.558 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-21T10:15:30.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-21T10:15:30.560 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T10:15:33.630 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-03-21T10:15:48.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-21T10:15:48.631 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T10:15:48.951 DEBUG:teuthology.orchestra.run.smithi031:> true 2024-03-21T10:15:49.488 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-21T10:15:49.488 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-g5ec3e6e88b2e"... 2024-03-21T10:15:49.488 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2024-03-21T10:15:49.525 INFO:teuthology.orchestra.run.smithi031.stdout:6.8.0-g5ec3e6e88b2e 2024-03-21T10:15:49.525 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g5ec3e6e88b2e vs 6.8.0-g5ec3e6e88b2e 2024-03-21T10:15:49.525 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-21T10:15:49.525 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-21T10:15:50.526 DEBUG:teuthology.parallel:result is None 2024-03-21T10:15:50.527 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-21T10:15:50.533 INFO:teuthology.task.internal:Creating test directory... 2024-03-21T10:15:50.533 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-21T10:15:50.538 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-21T10:15:50.540 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-21T10:15:50.576 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-21T10:15:50.585 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-21T10:15:50.591 INFO:teuthology.task.internal:Creating archive directory... 2024-03-21T10:15:50.591 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-21T10:15:50.625 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-21T10:15:50.627 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-21T10:15:50.843 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-21T10:15:50.850 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-21T10:15:50.850 DEBUG:teuthology.orchestra.run.smithi040:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-21T10:15:50.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T10:15:50.867 DEBUG:teuthology.orchestra.run.smithi070:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-21T10:15:50.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T10:15:50.884 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-21T10:15:50.891 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-21T10:15:50.911 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-21T10:15:50.960 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T10:15:50.966 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T10:15:50.983 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T10:15:50.990 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T10:15:51.033 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T10:15:51.108 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T10:15:51.113 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-21T10:15:51.120 INFO:teuthology.task.internal:Configuring sudo... 2024-03-21T10:15:51.120 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-21T10:15:51.125 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-21T10:15:51.127 DEBUG:teuthology.orchestra.run.smithi070:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-21T10:15:51.213 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-21T10:15:51.221 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-21T10:15:51.221 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-21T10:15:51.226 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-21T10:15:51.229 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-21T10:15:51.268 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T10:15:51.345 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T10:15:51.435 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T10:15:51.513 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T10:15:51.602 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-21T10:15:51.602 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-21T10:15:51.684 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T10:15:51.729 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T10:15:51.788 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T10:15:51.874 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T10:15:51.924 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-21T10:15:51.924 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-21T10:15:51.998 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T10:15:52.039 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T10:15:52.104 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T10:15:52.148 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T10:15:52.191 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-21T10:15:52.191 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-21T10:15:52.260 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart 2024-03-21T10:15:52.265 DEBUG:teuthology.orchestra.run.smithi040:> sudo service rsyslog restart 2024-03-21T10:15:52.267 DEBUG:teuthology.orchestra.run.smithi070:> sudo service rsyslog restart 2024-03-21T10:15:52.310 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-21T10:15:52.337 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-21T10:15:52.426 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-21T10:15:52.715 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-21T10:15:52.724 INFO:teuthology.task.internal:Starting timer... 2024-03-21T10:15:52.724 INFO:teuthology.run_tasks:Running task pcp... 2024-03-21T10:15:52.734 INFO:teuthology.run_tasks:Running task selinux... 2024-03-21T10:15:52.743 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-21T10:15:52.743 DEBUG:teuthology.orchestra.run.smithi031:> sudo service auditd rotate 2024-03-21T10:15:52.966 INFO:teuthology.orchestra.run.smithi031.stdout:Rotating logs: 2024-03-21T10:15:52.975 DEBUG:teuthology.orchestra.run.smithi040:> sudo service auditd rotate 2024-03-21T10:15:53.035 INFO:teuthology.orchestra.run.smithi040.stdout:Rotating logs: 2024-03-21T10:15:53.036 DEBUG:teuthology.orchestra.run.smithi070:> sudo service auditd rotate 2024-03-21T10:15:53.090 INFO:teuthology.orchestra.run.smithi070.stdout:Rotating logs: 2024-03-21T10:15:53.092 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-21T10:15:53.092 DEBUG:teuthology.orchestra.run.smithi031:> /usr/sbin/getenforce 2024-03-21T10:15:53.177 INFO:teuthology.orchestra.run.smithi031.stdout:Permissive 2024-03-21T10:15:53.178 DEBUG:teuthology.orchestra.run.smithi040:> /usr/sbin/getenforce 2024-03-21T10:15:53.208 INFO:teuthology.orchestra.run.smithi040.stdout:Permissive 2024-03-21T10:15:53.208 DEBUG:teuthology.orchestra.run.smithi070:> /usr/sbin/getenforce 2024-03-21T10:15:53.229 INFO:teuthology.orchestra.run.smithi070.stdout:Permissive 2024-03-21T10:15:53.230 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi031.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi040.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi070.front.sepia.ceph.com': 'permissive'} 2024-03-21T10:15:53.230 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-21T10:15:53.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T10:15:53.378 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-21T10:15:53.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T10:15:53.411 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-21T10:15:53.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T10:15:53.439 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-21T10:15:53.439 DEBUG:teuthology.orchestra.run.smithi031:> sudo /usr/sbin/setenforce permissive 2024-03-21T10:15:53.592 DEBUG:teuthology.orchestra.run.smithi040:> sudo /usr/sbin/setenforce permissive 2024-03-21T10:15:53.641 DEBUG:teuthology.orchestra.run.smithi070:> sudo /usr/sbin/setenforce permissive 2024-03-21T10:15:53.672 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-21T10:15:53.684 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-21T10:15:53.689 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-21T10:15:53.822 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-21T10:15:53.840 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-21T10:15:53.841 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi031.front.sepia.ceph.com,smithi040.front.sepia.ceph.com,smithi070.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-21T10:22:13.501 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi031.front.sepia.ceph.com'), Remote(name='ubuntu@smithi040.front.sepia.ceph.com'), Remote(name='ubuntu@smithi070.front.sepia.ceph.com')] 2024-03-21T10:22:13.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-21T10:22:13.504 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T10:22:13.651 DEBUG:teuthology.orchestra.run.smithi031:> true 2024-03-21T10:22:13.810 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-21T10:22:13.811 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-03-21T10:22:13.811 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T10:22:13.915 DEBUG:teuthology.orchestra.run.smithi040:> true 2024-03-21T10:22:13.987 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-03-21T10:22:13.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-03-21T10:22:13.988 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T10:22:14.079 DEBUG:teuthology.orchestra.run.smithi070:> true 2024-03-21T10:22:14.151 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-03-21T10:22:14.152 INFO:teuthology.run_tasks:Running task clock... 2024-03-21T10:22:14.165 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-21T10:22:14.165 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-21T10:22:14.166 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T10:22:14.173 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-21T10:22:14.173 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T10:22:14.175 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-21T10:22:14.176 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T10:22:14.213 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-21T10:22:14.228 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-21T10:22:14.234 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-21T10:22:14.246 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-21T10:22:14.271 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-21T10:22:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: ntpd: command not found 2024-03-21T10:22:14.277 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: ntpd: command not found 2024-03-21T10:22:14.290 INFO:teuthology.orchestra.run.smithi040.stdout:506 Cannot talk to daemon 2024-03-21T10:22:14.291 INFO:teuthology.orchestra.run.smithi070.stdout:506 Cannot talk to daemon 2024-03-21T10:22:14.307 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-21T10:22:14.310 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-21T10:22:14.324 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-21T10:22:14.324 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-21T10:22:14.330 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-21T10:22:14.368 INFO:teuthology.orchestra.run.smithi070.stderr:bash: line 1: ntpq: command not found 2024-03-21T10:22:14.372 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-21T10:22:14.372 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2024-03-21T10:22:14.372 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T10:22:14.372 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T10:22:14.372 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T10:22:14.372 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T10:22:14.377 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: ntpq: command not found 2024-03-21T10:22:14.381 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-21T10:22:14.382 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2024-03-21T10:22:14.382 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T10:22:14.382 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T10:22:14.382 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T10:22:14.382 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T10:22:14.426 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: ntpd: command not found 2024-03-21T10:22:14.473 INFO:teuthology.orchestra.run.smithi031.stdout:506 Cannot talk to daemon 2024-03-21T10:22:14.526 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-21T10:22:14.582 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-21T10:22:14.723 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2024-03-21T10:22:14.732 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-21T10:22:14.732 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-03-21T10:22:14.732 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T10:22:14.732 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T10:22:14.732 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T10:22:14.732 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T10:22:14.734 INFO:teuthology.run_tasks:Running task install... 2024-03-21T10:22:14.743 DEBUG:teuthology.task.install:project ceph 2024-03-21T10:22:14.743 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '2edfee68ad7243bed422a049b6f5f58619300f11'}} 2024-03-21T10:22:14.743 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '2edfee68ad7243bed422a049b6f5f58619300f11'} 2024-03-21T10:22:14.743 INFO:teuthology.task.install:Using flavor: default 2024-03-21T10:22:14.749 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-21T10:22:14.749 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-21T10:22:14.749 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2edfee68ad7243bed422a049b6f5f58619300f11', 'tag': None, 'wait_for_package': False} 2024-03-21T10:22:14.750 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2edfee68ad7243bed422a049b6f5f58619300f11 2024-03-21T10:22:14.751 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2edfee68ad7243bed422a049b6f5f58619300f11', 'tag': None, 'wait_for_package': False} 2024-03-21T10:22:14.752 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2edfee68ad7243bed422a049b6f5f58619300f11 2024-03-21T10:22:14.753 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2edfee68ad7243bed422a049b6f5f58619300f11', 'tag': None, 'wait_for_package': False} 2024-03-21T10:22:14.753 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2edfee68ad7243bed422a049b6f5f58619300f11 2024-03-21T10:22:14.908 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/centos/9/flavors/default/ 2024-03-21T10:22:14.908 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2231.g2edfee68 2024-03-21T10:22:14.912 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/centos/9/flavors/default/ 2024-03-21T10:22:14.912 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2231.g2edfee68 2024-03-21T10:22:15.016 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-21T10:22:15.017 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-21T10:22:15.017 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-21T10:22:15.030 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-21T10:22:15.030 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-21T10:22:15.031 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-21T10:22:15.053 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-21T10:22:15.053 DEBUG:teuthology.orchestra.run.smithi040:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2edfee68ad7243bed422a049b6f5f58619300f11/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-21T10:22:15.059 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-21T10:22:15.060 DEBUG:teuthology.orchestra.run.smithi070:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2edfee68ad7243bed422a049b6f5f58619300f11/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-21T10:22:15.084 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/centos/9/flavors/default/ 2024-03-21T10:22:15.084 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2231.g2edfee68 2024-03-21T10:22:15.127 DEBUG:teuthology.orchestra.run.smithi040:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-21T10:22:15.128 DEBUG:teuthology.orchestra.run.smithi070:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-21T10:22:15.192 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-21T10:22:15.192 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-21T10:22:15.192 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-21T10:22:15.208 DEBUG:teuthology.orchestra.run.smithi070:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-21T10:22:15.218 DEBUG:teuthology.orchestra.run.smithi040:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-21T10:22:15.249 INFO:teuthology.orchestra.run.smithi040.stdout:check_obsoletes = 1 2024-03-21T10:22:15.251 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum clean all 2024-03-21T10:22:15.276 INFO:teuthology.orchestra.run.smithi070.stdout:check_obsoletes = 1 2024-03-21T10:22:15.277 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum clean all 2024-03-21T10:22:15.299 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-21T10:22:15.299 DEBUG:teuthology.orchestra.run.smithi031:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2edfee68ad7243bed422a049b6f5f58619300f11/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-21T10:22:15.442 DEBUG:teuthology.orchestra.run.smithi031:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-21T10:22:15.500 INFO:teuthology.orchestra.run.smithi040.stdout:74 files removed 2024-03-21T10:22:15.513 INFO:teuthology.orchestra.run.smithi070.stdout:74 files removed 2024-03-21T10:22:15.528 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-21T10:22:15.540 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-21T10:22:15.583 DEBUG:teuthology.orchestra.run.smithi031:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-21T10:22:15.680 INFO:teuthology.orchestra.run.smithi031.stdout:check_obsoletes = 1 2024-03-21T10:22:15.684 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum clean all 2024-03-21T10:22:16.054 INFO:teuthology.orchestra.run.smithi031.stdout:74 files removed 2024-03-21T10:22:16.093 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-21T10:22:16.470 INFO:teuthology.orchestra.run.smithi040.stdout:ceph packages for x86_64 131 kB/s | 83 kB 00:00 2024-03-21T10:22:16.479 INFO:teuthology.orchestra.run.smithi070.stdout:ceph packages for x86_64 128 kB/s | 83 kB 00:00 2024-03-21T10:22:16.887 INFO:teuthology.orchestra.run.smithi040.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-21T10:22:16.896 INFO:teuthology.orchestra.run.smithi070.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-21T10:22:17.004 INFO:teuthology.orchestra.run.smithi031.stdout:ceph packages for x86_64 216 kB/s | 83 kB 00:00 2024-03-21T10:22:17.347 INFO:teuthology.orchestra.run.smithi070.stdout:ceph source packages 4.5 kB/s | 1.9 kB 00:00 2024-03-21T10:22:17.421 INFO:teuthology.orchestra.run.smithi040.stdout:ceph source packages 3.8 kB/s | 1.9 kB 00:00 2024-03-21T10:22:17.713 INFO:teuthology.orchestra.run.smithi031.stdout:ceph noarch packages 16 kB/s | 10 kB 00:00 2024-03-21T10:22:18.074 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS-9 - AppStream 9.6 MB/s | 6.7 MB 00:00 2024-03-21T10:22:18.182 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS-9 - AppStream 9.1 MB/s | 6.7 MB 00:00 2024-03-21T10:22:18.289 INFO:teuthology.orchestra.run.smithi031.stdout:ceph source packages 3.8 kB/s | 1.9 kB 00:00 2024-03-21T10:22:18.916 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-21T10:22:19.508 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS-9 - BaseOS 6.0 MB/s | 2.0 MB 00:00 2024-03-21T10:22:19.640 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-21T10:22:20.265 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-03-21T10:22:20.560 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - BaseOS 5.8 MB/s | 2.0 MB 00:00 2024-03-21T10:22:20.690 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-21T10:22:22.202 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-21T10:22:22.682 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-21T10:22:26.380 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 1.5 MB/s | 8.0 MB 00:05 2024-03-21T10:22:26.400 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - CRB 8.6 MB/s | 6.1 MB 00:00 2024-03-21T10:22:26.641 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - CRB 18 MB/s | 6.1 MB 00:00 2024-03-21T10:22:28.134 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 16 kB 00:00 2024-03-21T10:22:28.359 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-21T10:22:31.660 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 6.5 MB/s | 21 MB 00:03 2024-03-21T10:22:33.096 INFO:teuthology.orchestra.run.smithi070.stdout:Extra Packages for Enterprise Linux 4.3 MB/s | 21 MB 00:04 2024-03-21T10:22:36.810 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-21T10:22:38.050 INFO:teuthology.orchestra.run.smithi070.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-21T10:22:39.231 INFO:teuthology.orchestra.run.smithi040.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-21T10:22:39.232 INFO:teuthology.orchestra.run.smithi040.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-21T10:22:39.244 INFO:teuthology.orchestra.run.smithi040.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-21T10:22:39.245 INFO:teuthology.orchestra.run.smithi040.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-21T10:22:39.246 INFO:teuthology.orchestra.run.smithi040.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-21T10:22:39.247 INFO:teuthology.orchestra.run.smithi040.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-21T10:22:39.249 INFO:teuthology.orchestra.run.smithi040.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-21T10:22:39.251 INFO:teuthology.orchestra.run.smithi040.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-21T10:22:39.252 INFO:teuthology.orchestra.run.smithi040.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-21T10:22:39.337 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-03-21T10:22:39.357 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================================ 2024-03-21T10:22:39.358 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-03-21T10:22:39.358 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================================ 2024-03-21T10:22:39.358 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-03-21T10:22:39.358 INFO:teuthology.orchestra.run.smithi040.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-21T10:22:39.358 INFO:teuthology.orchestra.run.smithi040.stdout: ceph x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 6.6 k 2024-03-21T10:22:39.358 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 5.4 M 2024-03-21T10:22:39.358 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 890 k 2024-03-21T10:22:39.359 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 145 k 2024-03-21T10:22:39.359 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 1.5 M 2024-03-21T10:22:39.359 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 140 k 2024-03-21T10:22:39.359 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 1.8 M 2024-03-21T10:22:39.359 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 7.4 M 2024-03-21T10:22:39.359 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 49 k 2024-03-21T10:22:39.359 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 9.9 M 2024-03-21T10:22:39.359 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 48 M 2024-03-21T10:22:39.359 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 264 k 2024-03-21T10:22:39.359 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 750 k 2024-03-21T10:22:39.359 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-mirror x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 220 k 2024-03-21T10:22:39.359 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-shell x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 27 k 2024-03-21T10:22:39.359 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-top noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 18 k 2024-03-21T10:22:39.359 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-21T10:22:39.360 INFO:teuthology.orchestra.run.smithi040.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-21T10:22:39.360 INFO:teuthology.orchestra.run.smithi040.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-21T10:22:39.360 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 31 k 2024-03-21T10:22:39.360 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2 x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 747 k 2024-03-21T10:22:39.360 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 127 k 2024-03-21T10:22:39.360 INFO:teuthology.orchestra.run.smithi040.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-21T10:22:39.360 INFO:teuthology.orchestra.run.smithi040.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-21T10:22:39.360 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-21T10:22:39.360 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-21T10:22:39.360 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-21T10:22:39.360 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-21T10:22:39.360 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 161 k 2024-03-21T10:22:39.360 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 320 k 2024-03-21T10:22:39.361 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 297 k 2024-03-21T10:22:39.361 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 99 k 2024-03-21T10:22:39.361 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 85 k 2024-03-21T10:22:39.361 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 3.1 M 2024-03-21T10:22:39.361 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 171 k 2024-03-21T10:22:39.361 INFO:teuthology.orchestra.run.smithi040.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-21T10:22:39.361 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrading: 2024-03-21T10:22:39.361 INFO:teuthology.orchestra.run.smithi040.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-21T10:22:39.361 INFO:teuthology.orchestra.run.smithi040.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-21T10:22:39.361 INFO:teuthology.orchestra.run.smithi040.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-21T10:22:39.361 INFO:teuthology.orchestra.run.smithi040.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-21T10:22:39.361 INFO:teuthology.orchestra.run.smithi040.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-21T10:22:39.361 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-21T10:22:39.362 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-21T10:22:39.362 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-21T10:22:39.362 INFO:teuthology.orchestra.run.smithi040.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-21T10:22:39.362 INFO:teuthology.orchestra.run.smithi040.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-21T10:22:39.362 INFO:teuthology.orchestra.run.smithi040.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-21T10:22:39.362 INFO:teuthology.orchestra.run.smithi040.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-21T10:22:39.362 INFO:teuthology.orchestra.run.smithi040.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-21T10:22:39.362 INFO:teuthology.orchestra.run.smithi040.stdout: librados2 x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 3.4 M 2024-03-21T10:22:39.362 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1 x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 3.1 M 2024-03-21T10:22:39.362 INFO:teuthology.orchestra.run.smithi040.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-21T10:22:39.362 INFO:teuthology.orchestra.run.smithi040.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-21T10:22:39.362 INFO:teuthology.orchestra.run.smithi040.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-21T10:22:39.362 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-21T10:22:39.363 INFO:teuthology.orchestra.run.smithi040.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-21T10:22:39.363 INFO:teuthology.orchestra.run.smithi040.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-21T10:22:39.363 INFO:teuthology.orchestra.run.smithi040.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-21T10:22:39.363 INFO:teuthology.orchestra.run.smithi040.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-21T10:22:39.363 INFO:teuthology.orchestra.run.smithi040.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-21T10:22:39.363 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-03-21T10:22:39.363 INFO:teuthology.orchestra.run.smithi040.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-21T10:22:39.363 INFO:teuthology.orchestra.run.smithi040.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-21T10:22:39.363 INFO:teuthology.orchestra.run.smithi040.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-21T10:22:39.363 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 21 M 2024-03-21T10:22:39.363 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 34 k 2024-03-21T10:22:39.363 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 2.3 M 2024-03-21T10:22:39.363 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 248 k 2024-03-21T10:22:39.364 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 4.7 M 2024-03-21T10:22:39.364 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 17 M 2024-03-21T10:22:39.364 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 16 k 2024-03-21T10:22:39.364 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 25 k 2024-03-21T10:22:39.364 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-21T10:22:39.364 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-21T10:22:39.364 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-21T10:22:39.364 INFO:teuthology.orchestra.run.smithi040.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-21T10:22:39.364 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-21T10:22:39.364 INFO:teuthology.orchestra.run.smithi040.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-21T10:22:39.364 INFO:teuthology.orchestra.run.smithi040.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-21T10:22:39.364 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-21T10:22:39.364 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-21T10:22:39.364 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 163 k 2024-03-21T10:22:39.365 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-21T10:22:39.365 INFO:teuthology.orchestra.run.smithi040.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-21T10:22:39.365 INFO:teuthology.orchestra.run.smithi040.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-21T10:22:39.365 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-21T10:22:39.365 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-21T10:22:39.365 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1 x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 501 k 2024-03-21T10:22:39.365 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-21T10:22:39.365 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2 x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 5.1 M 2024-03-21T10:22:39.365 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-21T10:22:39.365 INFO:teuthology.orchestra.run.smithi040.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-21T10:22:39.365 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-21T10:22:39.365 INFO:teuthology.orchestra.run.smithi040.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-21T10:22:39.365 INFO:teuthology.orchestra.run.smithi040.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-21T10:22:39.365 INFO:teuthology.orchestra.run.smithi040.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-21T10:22:39.366 INFO:teuthology.orchestra.run.smithi040.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-21T10:22:39.366 INFO:teuthology.orchestra.run.smithi040.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-21T10:22:39.366 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-21T10:22:39.366 INFO:teuthology.orchestra.run.smithi040.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-21T10:22:39.366 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-21T10:22:39.366 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-21T10:22:39.366 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-21T10:22:39.366 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-21T10:22:39.366 INFO:teuthology.orchestra.run.smithi040.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-21T10:22:39.366 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-21T10:22:39.366 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-21T10:22:39.366 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-21T10:22:39.366 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 45 k 2024-03-21T10:22:39.367 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 131 k 2024-03-21T10:22:39.367 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-21T10:22:39.367 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-21T10:22:39.367 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-21T10:22:39.367 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-21T10:22:39.367 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-21T10:22:39.367 INFO:teuthology.orchestra.run.smithi040.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-21T10:22:39.367 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-21T10:22:39.367 INFO:teuthology.orchestra.run.smithi040.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-21T10:22:39.367 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-21T10:22:39.367 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-21T10:22:39.367 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-21T10:22:39.367 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-21T10:22:39.367 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-21T10:22:39.368 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-21T10:22:39.368 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-21T10:22:39.368 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-21T10:22:39.368 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-21T10:22:39.368 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-21T10:22:39.368 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-21T10:22:39.368 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-21T10:22:39.368 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-21T10:22:39.368 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-21T10:22:39.368 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-21T10:22:39.368 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-21T10:22:39.369 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-21T10:22:39.369 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-21T10:22:39.369 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-21T10:22:39.369 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-21T10:22:39.369 INFO:teuthology.orchestra.run.smithi040.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-21T10:22:39.369 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-21T10:22:39.369 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-21T10:22:39.369 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-21T10:22:39.369 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-21T10:22:39.369 INFO:teuthology.orchestra.run.smithi040.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-21T10:22:39.369 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-21T10:22:39.369 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-21T10:22:39.370 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-21T10:22:39.370 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-21T10:22:39.370 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-21T10:22:39.370 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-21T10:22:39.370 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-21T10:22:39.370 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-21T10:22:39.370 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-21T10:22:39.370 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-21T10:22:39.370 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-21T10:22:39.370 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-21T10:22:39.370 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-21T10:22:39.370 INFO:teuthology.orchestra.run.smithi040.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-21T10:22:39.370 INFO:teuthology.orchestra.run.smithi040.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-21T10:22:39.371 INFO:teuthology.orchestra.run.smithi040.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-21T10:22:39.371 INFO:teuthology.orchestra.run.smithi040.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-21T10:22:39.371 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-21T10:22:39.371 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-21T10:22:39.371 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-21T10:22:39.371 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-21T10:22:39.371 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-21T10:22:39.371 INFO:teuthology.orchestra.run.smithi040.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-21T10:22:39.371 INFO:teuthology.orchestra.run.smithi040.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-21T10:22:39.371 INFO:teuthology.orchestra.run.smithi040.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-21T10:22:39.371 INFO:teuthology.orchestra.run.smithi040.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-21T10:22:39.371 INFO:teuthology.orchestra.run.smithi040.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-21T10:22:39.371 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-21T10:22:39.372 INFO:teuthology.orchestra.run.smithi040.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-21T10:22:39.372 INFO:teuthology.orchestra.run.smithi040.stdout:Installing weak dependencies: 2024-03-21T10:22:39.372 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-21T10:22:39.372 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-21T10:22:39.372 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-03-21T10:22:39.372 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================================ 2024-03-21T10:22:39.372 INFO:teuthology.orchestra.run.smithi040.stdout:Install 150 Packages 2024-03-21T10:22:39.372 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrade 24 Packages 2024-03-21T10:22:39.372 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-21T10:22:39.373 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 224 M 2024-03-21T10:22:39.373 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-03-21T10:22:39.895 INFO:teuthology.orchestra.run.smithi040.stdout:(1/174): ceph-19.0.0-2231.g2edfee68.el9.x86_64. 40 kB/s | 6.6 kB 00:00 2024-03-21T10:22:40.104 INFO:teuthology.orchestra.run.smithi040.stdout:(2/174): ceph-fuse-19.0.0-2231.g2edfee68.el9.x8 4.2 MB/s | 890 kB 00:00 2024-03-21T10:22:40.162 INFO:teuthology.orchestra.run.smithi040.stdout:(3/174): ceph-immutable-object-cache-19.0.0-223 2.4 MB/s | 145 kB 00:00 2024-03-21T10:22:40.378 INFO:teuthology.orchestra.run.smithi070.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-21T10:22:40.378 INFO:teuthology.orchestra.run.smithi070.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-21T10:22:40.390 INFO:teuthology.orchestra.run.smithi070.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-21T10:22:40.391 INFO:teuthology.orchestra.run.smithi070.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-21T10:22:40.392 INFO:teuthology.orchestra.run.smithi070.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-21T10:22:40.393 INFO:teuthology.orchestra.run.smithi070.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-21T10:22:40.395 INFO:teuthology.orchestra.run.smithi070.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-21T10:22:40.397 INFO:teuthology.orchestra.run.smithi070.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-21T10:22:40.398 INFO:teuthology.orchestra.run.smithi070.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-21T10:22:40.437 INFO:teuthology.orchestra.run.smithi040.stdout:(4/174): ceph-mds-19.0.0-2231.g2edfee68.el9.x86 8.4 MB/s | 2.3 MB 00:00 2024-03-21T10:22:40.476 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-21T10:22:40.488 INFO:teuthology.orchestra.run.smithi070.stdout:============================================================================================ 2024-03-21T10:22:40.488 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-03-21T10:22:40.488 INFO:teuthology.orchestra.run.smithi070.stdout:============================================================================================ 2024-03-21T10:22:40.488 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-03-21T10:22:40.489 INFO:teuthology.orchestra.run.smithi070.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-21T10:22:40.489 INFO:teuthology.orchestra.run.smithi070.stdout: ceph x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 6.6 k 2024-03-21T10:22:40.489 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 5.4 M 2024-03-21T10:22:40.489 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-fuse x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 890 k 2024-03-21T10:22:40.489 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 145 k 2024-03-21T10:22:40.489 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 1.5 M 2024-03-21T10:22:40.489 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 140 k 2024-03-21T10:22:40.489 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 1.8 M 2024-03-21T10:22:40.489 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 7.4 M 2024-03-21T10:22:40.489 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-rook noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 49 k 2024-03-21T10:22:40.489 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-radosgw x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 9.9 M 2024-03-21T10:22:40.489 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-test x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 48 M 2024-03-21T10:22:40.489 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-volume noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 264 k 2024-03-21T10:22:40.489 INFO:teuthology.orchestra.run.smithi070.stdout: cephadm noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 750 k 2024-03-21T10:22:40.490 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-mirror x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 220 k 2024-03-21T10:22:40.490 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-shell x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 27 k 2024-03-21T10:22:40.490 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-top noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 18 k 2024-03-21T10:22:40.490 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-21T10:22:40.490 INFO:teuthology.orchestra.run.smithi070.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-21T10:22:40.490 INFO:teuthology.orchestra.run.smithi070.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-21T10:22:40.490 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-devel x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 31 k 2024-03-21T10:22:40.490 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs2 x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 747 k 2024-03-21T10:22:40.490 INFO:teuthology.orchestra.run.smithi070.stdout: librados-devel x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 127 k 2024-03-21T10:22:40.490 INFO:teuthology.orchestra.run.smithi070.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-21T10:22:40.490 INFO:teuthology.orchestra.run.smithi070.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-21T10:22:40.490 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-21T10:22:40.490 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-21T10:22:40.491 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-21T10:22:40.491 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-21T10:22:40.491 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cephfs x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 161 k 2024-03-21T10:22:40.491 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 320 k 2024-03-21T10:22:40.491 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rbd x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 297 k 2024-03-21T10:22:40.491 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rgw x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 99 k 2024-03-21T10:22:40.491 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-fuse x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 85 k 2024-03-21T10:22:40.491 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-mirror x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 3.1 M 2024-03-21T10:22:40.491 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-nbd x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 171 k 2024-03-21T10:22:40.491 INFO:teuthology.orchestra.run.smithi070.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-21T10:22:40.491 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrading: 2024-03-21T10:22:40.491 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-21T10:22:40.491 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-21T10:22:40.492 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-21T10:22:40.492 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-21T10:22:40.492 INFO:teuthology.orchestra.run.smithi070.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-21T10:22:40.492 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-21T10:22:40.492 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-21T10:22:40.492 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-21T10:22:40.492 INFO:teuthology.orchestra.run.smithi070.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-21T10:22:40.492 INFO:teuthology.orchestra.run.smithi070.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-21T10:22:40.492 INFO:teuthology.orchestra.run.smithi070.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-21T10:22:40.492 INFO:teuthology.orchestra.run.smithi070.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-21T10:22:40.492 INFO:teuthology.orchestra.run.smithi070.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-21T10:22:40.492 INFO:teuthology.orchestra.run.smithi070.stdout: librados2 x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 3.4 M 2024-03-21T10:22:40.492 INFO:teuthology.orchestra.run.smithi070.stdout: librbd1 x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 3.1 M 2024-03-21T10:22:40.493 INFO:teuthology.orchestra.run.smithi070.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-21T10:22:40.493 INFO:teuthology.orchestra.run.smithi070.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-21T10:22:40.493 INFO:teuthology.orchestra.run.smithi070.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-21T10:22:40.493 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-21T10:22:40.493 INFO:teuthology.orchestra.run.smithi070.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-21T10:22:40.493 INFO:teuthology.orchestra.run.smithi070.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-21T10:22:40.493 INFO:teuthology.orchestra.run.smithi070.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-21T10:22:40.493 INFO:teuthology.orchestra.run.smithi070.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-21T10:22:40.493 INFO:teuthology.orchestra.run.smithi070.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-21T10:22:40.493 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-03-21T10:22:40.493 INFO:teuthology.orchestra.run.smithi070.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-21T10:22:40.493 INFO:teuthology.orchestra.run.smithi070.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-21T10:22:40.493 INFO:teuthology.orchestra.run.smithi070.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-21T10:22:40.493 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-common x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 21 M 2024-03-21T10:22:40.494 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 34 k 2024-03-21T10:22:40.494 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mds x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 2.3 M 2024-03-21T10:22:40.494 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 248 k 2024-03-21T10:22:40.494 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mon x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 4.7 M 2024-03-21T10:22:40.494 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-osd x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 17 M 2024-03-21T10:22:40.494 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 16 k 2024-03-21T10:22:40.494 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-selinux x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 25 k 2024-03-21T10:22:40.494 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-21T10:22:40.494 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-21T10:22:40.494 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-21T10:22:40.494 INFO:teuthology.orchestra.run.smithi070.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-21T10:22:40.494 INFO:teuthology.orchestra.run.smithi070.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-21T10:22:40.494 INFO:teuthology.orchestra.run.smithi070.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-21T10:22:40.494 INFO:teuthology.orchestra.run.smithi070.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-21T10:22:40.495 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-21T10:22:40.495 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-21T10:22:40.495 INFO:teuthology.orchestra.run.smithi070.stdout: libcephsqlite x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 163 k 2024-03-21T10:22:40.495 INFO:teuthology.orchestra.run.smithi070.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-21T10:22:40.495 INFO:teuthology.orchestra.run.smithi070.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-21T10:22:40.495 INFO:teuthology.orchestra.run.smithi070.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-21T10:22:40.495 INFO:teuthology.orchestra.run.smithi070.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-21T10:22:40.495 INFO:teuthology.orchestra.run.smithi070.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-21T10:22:40.495 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1 x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 501 k 2024-03-21T10:22:40.495 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-21T10:22:40.495 INFO:teuthology.orchestra.run.smithi070.stdout: librgw2 x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 5.1 M 2024-03-21T10:22:40.495 INFO:teuthology.orchestra.run.smithi070.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-21T10:22:40.495 INFO:teuthology.orchestra.run.smithi070.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-21T10:22:40.495 INFO:teuthology.orchestra.run.smithi070.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-21T10:22:40.496 INFO:teuthology.orchestra.run.smithi070.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-21T10:22:40.496 INFO:teuthology.orchestra.run.smithi070.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-21T10:22:40.496 INFO:teuthology.orchestra.run.smithi070.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-21T10:22:40.496 INFO:teuthology.orchestra.run.smithi070.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-21T10:22:40.496 INFO:teuthology.orchestra.run.smithi070.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-21T10:22:40.496 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-21T10:22:40.496 INFO:teuthology.orchestra.run.smithi070.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-21T10:22:40.496 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-21T10:22:40.496 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-21T10:22:40.496 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-21T10:22:40.496 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-21T10:22:40.496 INFO:teuthology.orchestra.run.smithi070.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-21T10:22:40.496 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-21T10:22:40.496 INFO:teuthology.orchestra.run.smithi070.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-21T10:22:40.497 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-21T10:22:40.497 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 45 k 2024-03-21T10:22:40.497 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 131 k 2024-03-21T10:22:40.497 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-21T10:22:40.497 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-21T10:22:40.497 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-21T10:22:40.497 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-21T10:22:40.497 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-21T10:22:40.497 INFO:teuthology.orchestra.run.smithi070.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-21T10:22:40.497 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-21T10:22:40.497 INFO:teuthology.orchestra.run.smithi070.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-21T10:22:40.497 INFO:teuthology.orchestra.run.smithi070.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-21T10:22:40.497 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-21T10:22:40.497 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-21T10:22:40.498 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-21T10:22:40.498 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-21T10:22:40.498 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-21T10:22:40.498 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-21T10:22:40.498 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-21T10:22:40.498 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-21T10:22:40.498 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-21T10:22:40.498 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-21T10:22:40.498 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-21T10:22:40.498 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-21T10:22:40.498 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-21T10:22:40.498 INFO:teuthology.orchestra.run.smithi070.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-21T10:22:40.498 INFO:teuthology.orchestra.run.smithi070.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-21T10:22:40.498 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-21T10:22:40.499 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-21T10:22:40.499 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-21T10:22:40.499 INFO:teuthology.orchestra.run.smithi070.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-21T10:22:40.499 INFO:teuthology.orchestra.run.smithi070.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-21T10:22:40.499 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-21T10:22:40.499 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-21T10:22:40.499 INFO:teuthology.orchestra.run.smithi070.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-21T10:22:40.499 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-21T10:22:40.499 INFO:teuthology.orchestra.run.smithi070.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-21T10:22:40.499 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-21T10:22:40.499 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-21T10:22:40.499 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-21T10:22:40.499 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-21T10:22:40.499 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-21T10:22:40.499 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-21T10:22:40.500 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-21T10:22:40.500 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-21T10:22:40.500 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-21T10:22:40.500 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-21T10:22:40.500 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-21T10:22:40.500 INFO:teuthology.orchestra.run.smithi070.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-21T10:22:40.500 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-21T10:22:40.500 INFO:teuthology.orchestra.run.smithi070.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-21T10:22:40.500 INFO:teuthology.orchestra.run.smithi070.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-21T10:22:40.500 INFO:teuthology.orchestra.run.smithi070.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-21T10:22:40.501 INFO:teuthology.orchestra.run.smithi070.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-21T10:22:40.501 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-21T10:22:40.501 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-21T10:22:40.501 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-21T10:22:40.501 INFO:teuthology.orchestra.run.smithi070.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-21T10:22:40.501 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-21T10:22:40.501 INFO:teuthology.orchestra.run.smithi070.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-21T10:22:40.501 INFO:teuthology.orchestra.run.smithi070.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-21T10:22:40.501 INFO:teuthology.orchestra.run.smithi070.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-21T10:22:40.501 INFO:teuthology.orchestra.run.smithi070.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-21T10:22:40.501 INFO:teuthology.orchestra.run.smithi070.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-21T10:22:40.501 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-21T10:22:40.501 INFO:teuthology.orchestra.run.smithi070.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-21T10:22:40.501 INFO:teuthology.orchestra.run.smithi070.stdout:Installing weak dependencies: 2024-03-21T10:22:40.502 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-21T10:22:40.502 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-21T10:22:40.502 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-03-21T10:22:40.502 INFO:teuthology.orchestra.run.smithi070.stdout:============================================================================================ 2024-03-21T10:22:40.502 INFO:teuthology.orchestra.run.smithi070.stdout:Install 150 Packages 2024-03-21T10:22:40.502 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrade 24 Packages 2024-03-21T10:22:40.502 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-21T10:22:40.503 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 224 M 2024-03-21T10:22:40.503 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-03-21T10:22:40.613 INFO:teuthology.orchestra.run.smithi040.stdout:(5/174): ceph-mgr-19.0.0-2231.g2edfee68.el9.x86 8.5 MB/s | 1.5 MB 00:00 2024-03-21T10:22:41.096 INFO:teuthology.orchestra.run.smithi040.stdout:(6/174): ceph-mon-19.0.0-2231.g2edfee68.el9.x86 9.7 MB/s | 4.7 MB 00:00 2024-03-21T10:22:41.318 INFO:teuthology.orchestra.run.smithi070.stdout:(1/174): ceph-19.0.0-2231.g2edfee68.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2024-03-21T10:22:41.380 INFO:teuthology.orchestra.run.smithi040.stdout:(7/174): ceph-common-19.0.0-2231.g2edfee68.el9. 12 MB/s | 21 MB 00:01 2024-03-21T10:22:41.913 INFO:teuthology.orchestra.run.smithi040.stdout:(8/174): ceph-base-19.0.0-2231.g2edfee68.el9.x8 2.5 MB/s | 5.4 MB 00:02 2024-03-21T10:22:42.022 INFO:teuthology.orchestra.run.smithi040.stdout:(9/174): ceph-selinux-19.0.0-2231.g2edfee68.el9 232 kB/s | 25 kB 00:00 2024-03-21T10:22:42.285 INFO:teuthology.orchestra.run.smithi070.stdout:(2/174): ceph-base-19.0.0-2231.g2edfee68.el9.x8 4.2 MB/s | 5.4 MB 00:01 2024-03-21T10:22:42.368 INFO:teuthology.orchestra.run.smithi070.stdout:(3/174): ceph-immutable-object-cache-19.0.0-223 1.7 MB/s | 145 kB 00:00 2024-03-21T10:22:42.443 INFO:teuthology.orchestra.run.smithi070.stdout:(4/174): ceph-fuse-19.0.0-2231.g2edfee68.el9.x8 791 kB/s | 890 kB 00:01 2024-03-21T10:22:42.514 INFO:teuthology.orchestra.run.smithi040.stdout:(10/174): ceph-radosgw-19.0.0-2231.g2edfee68.el 8.7 MB/s | 9.9 MB 00:01 2024-03-21T10:22:42.685 INFO:teuthology.orchestra.run.smithi070.stdout:(5/174): ceph-mds-19.0.0-2231.g2edfee68.el9.x86 7.3 MB/s | 2.3 MB 00:00 2024-03-21T10:22:42.689 INFO:teuthology.orchestra.run.smithi040.stdout:(11/174): ceph-osd-19.0.0-2231.g2edfee68.el9.x8 11 MB/s | 17 MB 00:01 2024-03-21T10:22:42.714 INFO:teuthology.orchestra.run.smithi040.stdout:(12/174): cephfs-mirror-19.0.0-2231.g2edfee68.e 1.1 MB/s | 220 kB 00:00 2024-03-21T10:22:42.740 INFO:teuthology.orchestra.run.smithi040.stdout:(13/174): cephfs-shell-19.0.0-2231.g2edfee68.el 543 kB/s | 27 kB 00:00 2024-03-21T10:22:42.765 INFO:teuthology.orchestra.run.smithi040.stdout:(14/174): libcephfs-devel-19.0.0-2231.g2edfee68 622 kB/s | 31 kB 00:00 2024-03-21T10:22:42.815 INFO:teuthology.orchestra.run.smithi040.stdout:(15/174): libcephfs2-19.0.0-2231.g2edfee68.el9. 9.7 MB/s | 747 kB 00:00 2024-03-21T10:22:42.840 INFO:teuthology.orchestra.run.smithi040.stdout:(16/174): libcephsqlite-19.0.0-2231.g2edfee68.e 2.1 MB/s | 163 kB 00:00 2024-03-21T10:22:42.866 INFO:teuthology.orchestra.run.smithi040.stdout:(17/174): librados-devel-19.0.0-2231.g2edfee68. 2.5 MB/s | 127 kB 00:00 2024-03-21T10:22:42.916 INFO:teuthology.orchestra.run.smithi040.stdout:(18/174): libradosstriper1-19.0.0-2231.g2edfee6 6.5 MB/s | 501 kB 00:00 2024-03-21T10:22:43.000 INFO:teuthology.orchestra.run.smithi040.stdout:(19/174): python3-ceph-argparse-19.0.0-2231.g2e 542 kB/s | 45 kB 00:00 2024-03-21T10:22:43.092 INFO:teuthology.orchestra.run.smithi040.stdout:(20/174): python3-ceph-common-19.0.0-2231.g2edf 1.4 MB/s | 131 kB 00:00 2024-03-21T10:22:43.184 INFO:teuthology.orchestra.run.smithi040.stdout:(21/174): python3-cephfs-19.0.0-2231.g2edfee68. 1.7 MB/s | 161 kB 00:00 2024-03-21T10:22:43.237 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-21T10:22:43.275 INFO:teuthology.orchestra.run.smithi040.stdout:(22/174): python3-rados-19.0.0-2231.g2edfee68.e 3.4 MB/s | 320 kB 00:00 2024-03-21T10:22:43.367 INFO:teuthology.orchestra.run.smithi040.stdout:(23/174): librgw2-19.0.0-2231.g2edfee68.el9.x86 10 MB/s | 5.1 MB 00:00 2024-03-21T10:22:43.377 INFO:teuthology.orchestra.run.smithi070.stdout:(6/174): ceph-mon-19.0.0-2231.g2edfee68.el9.x86 6.8 MB/s | 4.7 MB 00:00 2024-03-21T10:22:43.393 INFO:teuthology.orchestra.run.smithi040.stdout:(24/174): python3-rbd-19.0.0-2231.g2edfee68.el9 2.5 MB/s | 297 kB 00:00 2024-03-21T10:22:43.418 INFO:teuthology.orchestra.run.smithi040.stdout:(25/174): python3-rgw-19.0.0-2231.g2edfee68.el9 1.9 MB/s | 99 kB 00:00 2024-03-21T10:22:43.419 INFO:teuthology.orchestra.run.smithi070.stdout:(7/174): ceph-mgr-19.0.0-2231.g2edfee68.el9.x86 1.5 MB/s | 1.5 MB 00:00 2024-03-21T10:22:43.444 INFO:teuthology.orchestra.run.smithi040.stdout:(26/174): rbd-fuse-19.0.0-2231.g2edfee68.el9.x8 1.7 MB/s | 85 kB 00:00 2024-03-21T10:22:43.527 INFO:teuthology.orchestra.run.smithi040.stdout:(27/174): rbd-nbd-19.0.0-2231.g2edfee68.el9.x86 2.0 MB/s | 171 kB 00:00 2024-03-21T10:22:43.619 INFO:teuthology.orchestra.run.smithi040.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2231.g 371 kB/s | 34 kB 00:00 2024-03-21T10:22:43.727 INFO:teuthology.orchestra.run.smithi040.stdout:(29/174): rbd-mirror-19.0.0-2231.g2edfee68.el9. 10 MB/s | 3.1 MB 00:00 2024-03-21T10:22:43.744 INFO:teuthology.orchestra.run.smithi040.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2231.g2edfee6 1.1 MB/s | 140 kB 00:00 2024-03-21T10:22:44.020 INFO:teuthology.orchestra.run.smithi040.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2231.g2edfe 6.1 MB/s | 1.8 MB 00:00 2024-03-21T10:22:44.095 INFO:teuthology.orchestra.run.smithi040.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2231.g2e 3.2 MB/s | 248 kB 00:00 2024-03-21T10:22:44.170 INFO:teuthology.orchestra.run.smithi040.stdout:(33/174): ceph-mgr-rook-19.0.0-2231.g2edfee68.e 658 kB/s | 49 kB 00:00 2024-03-21T10:22:44.237 INFO:teuthology.orchestra.run.smithi040.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2231.g2 248 kB/s | 16 kB 00:00 2024-03-21T10:22:44.329 INFO:teuthology.orchestra.run.smithi040.stdout:(35/174): ceph-volume-19.0.0-2231.g2edfee68.el9 2.8 MB/s | 264 kB 00:00 2024-03-21T10:22:44.505 INFO:teuthology.orchestra.run.smithi040.stdout:(36/174): cephadm-19.0.0-2231.g2edfee68.el9.noa 4.2 MB/s | 750 kB 00:00 2024-03-21T10:22:44.588 INFO:teuthology.orchestra.run.smithi040.stdout:(37/174): cephfs-top-19.0.0-2231.g2edfee68.el9. 219 kB/s | 18 kB 00:00 2024-03-21T10:22:44.872 INFO:teuthology.orchestra.run.smithi040.stdout:(38/174): ceph-mgr-diskprediction-local-19.0.0- 6.5 MB/s | 7.4 MB 00:01 2024-03-21T10:22:44.919 INFO:teuthology.orchestra.run.smithi070.stdout:(8/174): ceph-common-19.0.0-2231.g2edfee68.el9. 5.3 MB/s | 21 MB 00:03 2024-03-21T10:22:44.961 INFO:teuthology.orchestra.run.smithi070.stdout:(9/174): ceph-selinux-19.0.0-2231.g2edfee68.el9 608 kB/s | 25 kB 00:00 2024-03-21T10:22:44.997 INFO:teuthology.orchestra.run.smithi040.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 1.6 MB/s | 662 kB 00:00 2024-03-21T10:22:45.089 INFO:teuthology.orchestra.run.smithi040.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 4.2 MB/s | 920 kB 00:00 2024-03-21T10:22:45.114 INFO:teuthology.orchestra.run.smithi040.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 910 kB/s | 106 kB 00:00 2024-03-21T10:22:45.173 INFO:teuthology.orchestra.run.smithi040.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 545 kB/s | 32 kB 00:00 2024-03-21T10:22:45.198 INFO:teuthology.orchestra.run.smithi040.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 3.6 MB/s | 307 kB 00:00 2024-03-21T10:22:45.348 INFO:teuthology.orchestra.run.smithi040.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 114 kB/s | 17 kB 00:00 2024-03-21T10:22:45.414 INFO:teuthology.orchestra.run.smithi040.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2024-03-21T10:22:45.430 INFO:teuthology.orchestra.run.smithi040.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 126 kB/s | 10 kB 00:00 2024-03-21T10:22:45.472 INFO:teuthology.orchestra.run.smithi040.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-21T10:22:45.498 INFO:teuthology.orchestra.run.smithi040.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 658 kB/s | 44 kB 00:00 2024-03-21T10:22:45.511 INFO:teuthology.orchestra.run.smithi070.stdout:(10/174): ceph-radosgw-19.0.0-2231.g2edfee68.el 4.7 MB/s | 9.9 MB 00:02 2024-03-21T10:22:45.565 INFO:teuthology.orchestra.run.smithi040.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-03-21T10:22:45.591 INFO:teuthology.orchestra.run.smithi040.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 6.1 MB/s | 578 kB 00:00 2024-03-21T10:22:45.628 INFO:teuthology.orchestra.run.smithi070.stdout:(11/174): cephfs-mirror-19.0.0-2231.g2edfee68.e 1.9 MB/s | 220 kB 00:00 2024-03-21T10:22:45.632 INFO:teuthology.orchestra.run.smithi040.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 705 kB/s | 46 kB 00:00 2024-03-21T10:22:45.658 INFO:teuthology.orchestra.run.smithi040.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.1 MB/s | 282 kB 00:00 2024-03-21T10:22:45.700 INFO:teuthology.orchestra.run.smithi040.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 187 kB 00:00 2024-03-21T10:22:45.712 INFO:teuthology.orchestra.run.smithi070.stdout:(12/174): cephfs-shell-19.0.0-2231.g2edfee68.el 325 kB/s | 27 kB 00:00 2024-03-21T10:22:45.725 INFO:teuthology.orchestra.run.smithi040.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2024-03-21T10:22:45.759 INFO:teuthology.orchestra.run.smithi040.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 587 kB/s | 34 kB 00:00 2024-03-21T10:22:45.795 INFO:teuthology.orchestra.run.smithi070.stdout:(13/174): libcephfs-devel-19.0.0-2231.g2edfee68 377 kB/s | 31 kB 00:00 2024-03-21T10:22:45.976 INFO:teuthology.orchestra.run.smithi040.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 19 MB/s | 4.7 MB 00:00 2024-03-21T10:22:46.017 INFO:teuthology.orchestra.run.smithi040.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 5.9 MB/s | 1.5 MB 00:00 2024-03-21T10:22:46.051 INFO:teuthology.orchestra.run.smithi040.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-21T10:22:46.085 INFO:teuthology.orchestra.run.smithi040.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.0 MB/s | 134 kB 00:00 2024-03-21T10:22:46.154 INFO:teuthology.orchestra.run.smithi070.stdout:(14/174): libcephfs2-19.0.0-2231.g2edfee68.el9. 2.0 MB/s | 747 kB 00:00 2024-03-21T10:22:46.304 INFO:teuthology.orchestra.run.smithi070.stdout:(15/174): libcephsqlite-19.0.0-2231.g2edfee68.e 1.1 MB/s | 163 kB 00:00 2024-03-21T10:22:46.335 INFO:teuthology.orchestra.run.smithi040.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 20 MB/s | 5.7 MB 00:00 2024-03-21T10:22:46.369 INFO:teuthology.orchestra.run.smithi040.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 3.6 MB/s | 1.0 MB 00:00 2024-03-21T10:22:46.385 INFO:teuthology.orchestra.run.smithi040.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.6 MB/s | 78 kB 00:00 2024-03-21T10:22:46.388 INFO:teuthology.orchestra.run.smithi070.stdout:(16/174): librados-devel-19.0.0-2231.g2edfee68. 1.5 MB/s | 127 kB 00:00 2024-03-21T10:22:46.653 INFO:teuthology.orchestra.run.smithi040.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-21T10:22:46.654 INFO:teuthology.orchestra.run.smithi070.stdout:(17/174): libradosstriper1-19.0.0-2231.g2edfee6 1.8 MB/s | 501 kB 00:00 2024-03-21T10:22:46.677 INFO:teuthology.orchestra.run.smithi040.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 84 kB/s | 24 kB 00:00 2024-03-21T10:22:46.730 INFO:teuthology.orchestra.run.smithi040.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 363 kB/s | 18 kB 00:00 2024-03-21T10:22:46.745 INFO:teuthology.orchestra.run.smithi040.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-03-21T10:22:46.787 INFO:teuthology.orchestra.run.smithi040.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 541 kB/s | 32 kB 00:00 2024-03-21T10:22:47.004 INFO:teuthology.orchestra.run.smithi040.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-03-21T10:22:47.029 INFO:teuthology.orchestra.run.smithi040.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 902 kB/s | 217 kB 00:00 2024-03-21T10:22:47.063 INFO:teuthology.orchestra.run.smithi040.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 174 kB 00:00 2024-03-21T10:22:47.080 INFO:teuthology.orchestra.run.smithi040.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-21T10:22:47.122 INFO:teuthology.orchestra.run.smithi040.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 707 kB/s | 41 kB 00:00 2024-03-21T10:22:47.138 INFO:teuthology.orchestra.run.smithi040.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.0 MB/s | 238 kB 00:00 2024-03-21T10:22:47.189 INFO:teuthology.orchestra.run.smithi040.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.0 MB/s | 132 kB 00:00 2024-03-21T10:22:47.206 INFO:teuthology.orchestra.run.smithi040.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.1 MB/s | 211 kB 00:00 2024-03-21T10:22:47.248 INFO:teuthology.orchestra.run.smithi040.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 815 kB/s | 47 kB 00:00 2024-03-21T10:22:47.282 INFO:teuthology.orchestra.run.smithi040.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 567 kB/s | 43 kB 00:00 2024-03-21T10:22:47.397 INFO:teuthology.orchestra.run.smithi070.stdout:(18/174): ceph-test-19.0.0-2231.g2edfee68.el9.x 20 MB/s | 48 MB 00:02 2024-03-21T10:22:47.447 INFO:teuthology.orchestra.run.smithi070.stdout:(19/174): python3-ceph-argparse-19.0.0-2231.g2e 910 kB/s | 45 kB 00:00 2024-03-21T10:22:47.497 INFO:teuthology.orchestra.run.smithi070.stdout:(20/174): python3-ceph-common-19.0.0-2231.g2edf 2.6 MB/s | 131 kB 00:00 2024-03-21T10:22:47.550 INFO:teuthology.orchestra.run.smithi070.stdout:(21/174): python3-cephfs-19.0.0-2231.g2edfee68. 3.2 MB/s | 161 kB 00:00 2024-03-21T10:22:47.587 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-21T10:22:47.606 INFO:teuthology.orchestra.run.smithi070.stdout:(22/174): python3-rados-19.0.0-2231.g2edfee68.e 5.4 MB/s | 320 kB 00:00 2024-03-21T10:22:47.656 INFO:teuthology.orchestra.run.smithi070.stdout:(23/174): python3-rbd-19.0.0-2231.g2edfee68.el9 5.8 MB/s | 297 kB 00:00 2024-03-21T10:22:47.690 INFO:teuthology.orchestra.run.smithi040.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 95 kB/s | 39 kB 00:00 2024-03-21T10:22:47.706 INFO:teuthology.orchestra.run.smithi070.stdout:(24/174): python3-rgw-19.0.0-2231.g2edfee68.el9 1.9 MB/s | 99 kB 00:00 2024-03-21T10:22:47.757 INFO:teuthology.orchestra.run.smithi070.stdout:(25/174): rbd-fuse-19.0.0-2231.g2edfee68.el9.x8 1.7 MB/s | 85 kB 00:00 2024-03-21T10:22:47.965 INFO:teuthology.orchestra.run.smithi070.stdout:(26/174): rbd-mirror-19.0.0-2231.g2edfee68.el9. 15 MB/s | 3.1 MB 00:00 2024-03-21T10:22:48.041 INFO:teuthology.orchestra.run.smithi070.stdout:(27/174): librgw2-19.0.0-2231.g2edfee68.el9.x86 3.7 MB/s | 5.1 MB 00:01 2024-03-21T10:22:48.049 INFO:teuthology.orchestra.run.smithi040.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-03-21T10:22:48.066 INFO:teuthology.orchestra.run.smithi070.stdout:(28/174): rbd-nbd-19.0.0-2231.g2edfee68.el9.x86 1.7 MB/s | 171 kB 00:00 2024-03-21T10:22:48.074 INFO:teuthology.orchestra.run.smithi040.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 107 kB/s | 41 kB 00:00 2024-03-21T10:22:48.091 INFO:teuthology.orchestra.run.smithi070.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2231.g 678 kB/s | 34 kB 00:00 2024-03-21T10:22:48.116 INFO:teuthology.orchestra.run.smithi040.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 300 kB 00:00 2024-03-21T10:22:48.116 INFO:teuthology.orchestra.run.smithi070.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2231.g2edfee6 2.7 MB/s | 140 kB 00:00 2024-03-21T10:22:48.141 INFO:teuthology.orchestra.run.smithi040.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 843 kB/s | 56 kB 00:00 2024-03-21T10:22:48.183 INFO:teuthology.orchestra.run.smithi040.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 939 kB/s | 63 kB 00:00 2024-03-21T10:22:48.209 INFO:teuthology.orchestra.run.smithi040.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 659 kB/s | 44 kB 00:00 2024-03-21T10:22:48.242 INFO:teuthology.orchestra.run.smithi040.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 79 kB 00:00 2024-03-21T10:22:48.276 INFO:teuthology.orchestra.run.smithi040.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-21T10:22:48.368 INFO:teuthology.orchestra.run.smithi040.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 9.1 MB/s | 1.1 MB 00:00 2024-03-21T10:22:48.393 INFO:teuthology.orchestra.run.smithi040.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 879 kB/s | 103 kB 00:00 2024-03-21T10:22:48.435 INFO:teuthology.orchestra.run.smithi040.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 1.8 MB/s | 124 kB 00:00 2024-03-21T10:22:48.477 INFO:teuthology.orchestra.run.smithi040.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-21T10:22:48.517 INFO:teuthology.orchestra.run.smithi070.stdout:(31/174): ceph-mgr-diskprediction-local-19.0.0- 18 MB/s | 7.4 MB 00:00 2024-03-21T10:22:48.578 INFO:teuthology.orchestra.run.smithi070.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2231.g2e 4.2 MB/s | 248 kB 00:00 2024-03-21T10:22:48.578 INFO:teuthology.orchestra.run.smithi040.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-21T10:22:48.617 INFO:teuthology.orchestra.run.smithi070.stdout:(33/174): ceph-mgr-dashboard-19.0.0-2231.g2edfe 3.4 MB/s | 1.8 MB 00:00 2024-03-21T10:22:48.627 INFO:teuthology.orchestra.run.smithi040.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-03-21T10:22:48.642 INFO:teuthology.orchestra.run.smithi070.stdout:(34/174): ceph-mgr-rook-19.0.0-2231.g2edfee68.e 736 kB/s | 49 kB 00:00 2024-03-21T10:22:48.661 INFO:teuthology.orchestra.run.smithi040.stdout:(93/174): libgfortran-11.4.1-3.el9.x86_64.rpm 4.3 MB/s | 803 kB 00:00 2024-03-21T10:22:48.667 INFO:teuthology.orchestra.run.smithi070.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2231.g2 328 kB/s | 16 kB 00:00 2024-03-21T10:22:48.686 INFO:teuthology.orchestra.run.smithi040.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-03-21T10:22:48.693 INFO:teuthology.orchestra.run.smithi070.stdout:(36/174): ceph-volume-19.0.0-2231.g2edfee68.el9 5.2 MB/s | 264 kB 00:00 2024-03-21T10:22:48.703 INFO:teuthology.orchestra.run.smithi040.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 517 kB/s | 21 kB 00:00 2024-03-21T10:22:48.728 INFO:teuthology.orchestra.run.smithi040.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 608 kB/s | 25 kB 00:00 2024-03-21T10:22:48.752 INFO:teuthology.orchestra.run.smithi070.stdout:(37/174): cephfs-top-19.0.0-2231.g2edfee68.el9. 312 kB/s | 18 kB 00:00 2024-03-21T10:22:48.782 INFO:teuthology.orchestra.run.smithi040.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 4.6 MB/s | 245 kB 00:00 2024-03-21T10:22:48.802 INFO:teuthology.orchestra.run.smithi070.stdout:(38/174): cephadm-19.0.0-2231.g2edfee68.el9.noa 5.5 MB/s | 750 kB 00:00 2024-03-21T10:22:48.873 INFO:teuthology.orchestra.run.smithi040.stdout:(98/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 24 MB/s | 4.1 MB 00:00 2024-03-21T10:22:48.898 INFO:teuthology.orchestra.run.smithi040.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.1 MB/s | 248 kB 00:00 2024-03-21T10:22:48.923 INFO:teuthology.orchestra.run.smithi040.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 4.2 MB/s | 216 kB 00:00 2024-03-21T10:22:48.948 INFO:teuthology.orchestra.run.smithi040.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 695 kB/s | 35 kB 00:00 2024-03-21T10:22:49.005 INFO:teuthology.orchestra.run.smithi040.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 545 kB/s | 44 kB 00:00 2024-03-21T10:22:49.041 INFO:teuthology.orchestra.run.smithi040.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.0 MB/s | 94 kB 00:00 2024-03-21T10:22:49.127 INFO:teuthology.orchestra.run.smithi070.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 1.7 MB/s | 662 kB 00:00 2024-03-21T10:22:49.152 INFO:teuthology.orchestra.run.smithi070.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 2.8 MB/s | 920 kB 00:00 2024-03-21T10:22:49.178 INFO:teuthology.orchestra.run.smithi070.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 106 kB 00:00 2024-03-21T10:22:49.211 INFO:teuthology.orchestra.run.smithi070.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.1 MB/s | 307 kB 00:00 2024-03-21T10:22:49.216 INFO:teuthology.orchestra.run.smithi040.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 106 kB/s | 22 kB 00:00 2024-03-21T10:22:49.228 INFO:teuthology.orchestra.run.smithi070.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 630 kB/s | 32 kB 00:00 2024-03-21T10:22:49.320 INFO:teuthology.orchestra.run.smithi070.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 187 kB/s | 17 kB 00:00 2024-03-21T10:22:49.412 INFO:teuthology.orchestra.run.smithi070.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 15 MB/s | 3.0 MB 00:00 2024-03-21T10:22:49.437 INFO:teuthology.orchestra.run.smithi070.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 88 kB/s | 10 kB 00:00 2024-03-21T10:22:49.487 INFO:teuthology.orchestra.run.smithi070.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 159 kB 00:00 2024-03-21T10:22:49.513 INFO:teuthology.orchestra.run.smithi070.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 589 kB/s | 44 kB 00:00 2024-03-21T10:22:49.563 INFO:teuthology.orchestra.run.smithi070.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-21T10:22:49.614 INFO:teuthology.orchestra.run.smithi070.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 5.6 MB/s | 578 kB 00:00 2024-03-21T10:22:49.639 INFO:teuthology.orchestra.run.smithi070.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 617 kB/s | 46 kB 00:00 2024-03-21T10:22:49.650 INFO:teuthology.orchestra.run.smithi040.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.4 MB/s | 863 kB 00:00 2024-03-21T10:22:49.680 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 21 kB/s | 16 kB 00:00 2024-03-21T10:22:49.681 INFO:teuthology.orchestra.run.smithi070.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.1 MB/s | 282 kB 00:00 2024-03-21T10:22:49.700 INFO:teuthology.orchestra.run.smithi040.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-03-21T10:22:49.706 INFO:teuthology.orchestra.run.smithi070.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2024-03-21T10:22:49.748 INFO:teuthology.orchestra.run.smithi070.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2024-03-21T10:22:49.773 INFO:teuthology.orchestra.run.smithi070.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 511 kB/s | 34 kB 00:00 2024-03-21T10:22:49.800 INFO:teuthology.orchestra.run.smithi040.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 267 kB/s | 19 kB 00:00 2024-03-21T10:22:50.015 INFO:teuthology.orchestra.run.smithi070.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 18 MB/s | 4.7 MB 00:00 2024-03-21T10:22:50.057 INFO:teuthology.orchestra.run.smithi070.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 5.4 MB/s | 1.5 MB 00:00 2024-03-21T10:22:50.076 INFO:teuthology.orchestra.run.smithi040.stdout:(108/174): libarrow-9.0.0-10.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-03-21T10:22:50.101 INFO:teuthology.orchestra.run.smithi040.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 685 kB/s | 309 kB 00:00 2024-03-21T10:22:50.151 INFO:teuthology.orchestra.run.smithi040.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 345 kB/s | 26 kB 00:00 2024-03-21T10:22:50.174 INFO:teuthology.orchestra.run.smithi070.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 5.0 MB/s | 811 kB 00:00 2024-03-21T10:22:50.177 INFO:teuthology.orchestra.run.smithi040.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 646 kB/s | 49 kB 00:00 2024-03-21T10:22:50.199 INFO:teuthology.orchestra.run.smithi070.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 945 kB/s | 134 kB 00:00 2024-03-21T10:22:50.227 INFO:teuthology.orchestra.run.smithi040.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 897 kB/s | 67 kB 00:00 2024-03-21T10:22:50.269 INFO:teuthology.orchestra.run.smithi040.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-03-21T10:22:50.274 INFO:teuthology.orchestra.run.smithi070.stdout:(60/174): protobuf-3.14.0-13.el9.x86_64.rpm 14 MB/s | 1.0 MB 00:00 2024-03-21T10:22:50.311 INFO:teuthology.orchestra.run.smithi040.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-03-21T10:22:50.386 INFO:teuthology.orchestra.run.smithi040.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 579 kB/s | 43 kB 00:00 2024-03-21T10:22:50.420 INFO:teuthology.orchestra.run.smithi040.stdout:(116/174): python3-asyncssh-2.13.2-5.el9.noarch 3.6 MB/s | 548 kB 00:00 2024-03-21T10:22:50.450 INFO:teuthology.orchestra.run.smithi070.stdout:(61/174): postgresql-server-13.11-1.el9.x86_64. 23 MB/s | 5.7 MB 00:00 2024-03-21T10:22:50.453 INFO:teuthology.orchestra.run.smithi040.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 482 kB/s | 32 kB 00:00 2024-03-21T10:22:50.475 INFO:teuthology.orchestra.run.smithi070.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 392 kB/s | 78 kB 00:00 2024-03-21T10:22:50.487 INFO:teuthology.orchestra.run.smithi040.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 224 kB/s | 15 kB 00:00 2024-03-21T10:22:50.529 INFO:teuthology.orchestra.run.smithi040.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-21T10:22:50.579 INFO:teuthology.orchestra.run.smithi040.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-03-21T10:22:50.604 INFO:teuthology.orchestra.run.smithi040.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-03-21T10:22:50.646 INFO:teuthology.orchestra.run.smithi040.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 528 kB/s | 35 kB 00:00 2024-03-21T10:22:50.667 INFO:teuthology.orchestra.run.smithi070.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 27 MB/s | 5.8 MB 00:00 2024-03-21T10:22:50.672 INFO:teuthology.orchestra.run.smithi040.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 3.0 MB/s | 203 kB 00:00 2024-03-21T10:22:50.692 INFO:teuthology.orchestra.run.smithi070.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 113 kB/s | 24 kB 00:00 2024-03-21T10:22:50.751 INFO:teuthology.orchestra.run.smithi070.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 310 kB/s | 18 kB 00:00 2024-03-21T10:22:50.755 INFO:teuthology.orchestra.run.smithi040.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-03-21T10:22:50.784 INFO:teuthology.orchestra.run.smithi070.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.5 MB/s | 144 kB 00:00 2024-03-21T10:22:50.809 INFO:teuthology.orchestra.run.smithi070.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 540 kB/s | 32 kB 00:00 2024-03-21T10:22:50.889 INFO:teuthology.orchestra.run.smithi040.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 8.4 MB/s | 2.0 MB 00:00 2024-03-21T10:22:50.914 INFO:teuthology.orchestra.run.smithi040.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 67 kB/s | 11 kB 00:00 2024-03-21T10:22:50.964 INFO:teuthology.orchestra.run.smithi040.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 235 kB/s | 18 kB 00:00 2024-03-21T10:22:50.985 INFO:teuthology.orchestra.run.smithi070.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 4.8 MB 00:00 2024-03-21T10:22:50.990 INFO:teuthology.orchestra.run.smithi040.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 308 kB/s | 23 kB 00:00 2024-03-21T10:22:51.010 INFO:teuthology.orchestra.run.smithi070.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.1 MB/s | 217 kB 00:00 2024-03-21T10:22:51.040 INFO:teuthology.orchestra.run.smithi040.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-03-21T10:22:51.043 INFO:teuthology.orchestra.run.smithi070.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 174 kB 00:00 2024-03-21T10:22:51.060 INFO:teuthology.orchestra.run.smithi070.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-21T10:22:51.074 INFO:teuthology.orchestra.run.smithi040.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 235 kB/s | 20 kB 00:00 2024-03-21T10:22:51.085 INFO:teuthology.orchestra.run.smithi070.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 989 kB/s | 41 kB 00:00 2024-03-21T10:22:51.111 INFO:teuthology.orchestra.run.smithi070.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-21T10:22:51.132 INFO:teuthology.orchestra.run.smithi040.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-03-21T10:22:51.136 INFO:teuthology.orchestra.run.smithi070.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-21T10:22:51.162 INFO:teuthology.orchestra.run.smithi070.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-21T10:22:51.166 INFO:teuthology.orchestra.run.smithi040.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 502 kB/s | 46 kB 00:00 2024-03-21T10:22:51.187 INFO:teuthology.orchestra.run.smithi070.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 950 kB/s | 47 kB 00:00 2024-03-21T10:22:51.208 INFO:teuthology.orchestra.run.smithi040.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-03-21T10:22:51.212 INFO:teuthology.orchestra.run.smithi070.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 855 kB/s | 43 kB 00:00 2024-03-21T10:22:51.242 INFO:teuthology.orchestra.run.smithi040.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 767 kB/s | 58 kB 00:00 2024-03-21T10:22:51.284 INFO:teuthology.orchestra.run.smithi040.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-21T10:22:51.317 INFO:teuthology.orchestra.run.smithi040.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-03-21T10:22:51.368 INFO:teuthology.orchestra.run.smithi040.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.1 MB/s | 90 kB 00:00 2024-03-21T10:22:51.379 INFO:teuthology.orchestra.run.smithi070.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 234 kB/s | 39 kB 00:00 2024-03-21T10:22:51.393 INFO:teuthology.orchestra.run.smithi040.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 375 kB/s | 28 kB 00:00 2024-03-21T10:22:51.443 INFO:teuthology.orchestra.run.smithi040.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 410 kB/s | 31 kB 00:00 2024-03-21T10:22:51.468 INFO:teuthology.orchestra.run.smithi040.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-03-21T10:22:51.519 INFO:teuthology.orchestra.run.smithi040.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 786 kB/s | 59 kB 00:00 2024-03-21T10:22:51.544 INFO:teuthology.orchestra.run.smithi040.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 475 kB/s | 36 kB 00:00 2024-03-21T10:22:51.582 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-21T10:22:51.594 INFO:teuthology.orchestra.run.smithi040.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 679 kB/s | 51 kB 00:00 2024-03-21T10:22:51.619 INFO:teuthology.orchestra.run.smithi040.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-21T10:22:51.646 INFO:teuthology.orchestra.run.smithi070.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-03-21T10:22:51.663 INFO:teuthology.orchestra.run.smithi070.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 146 kB/s | 41 kB 00:00 2024-03-21T10:22:51.670 INFO:teuthology.orchestra.run.smithi040.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-21T10:22:51.688 INFO:teuthology.orchestra.run.smithi070.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-21T10:22:51.703 INFO:teuthology.orchestra.run.smithi040.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-03-21T10:22:51.705 INFO:teuthology.orchestra.run.smithi070.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.3 MB/s | 56 kB 00:00 2024-03-21T10:22:51.722 INFO:teuthology.orchestra.run.smithi070.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-21T10:22:51.739 INFO:teuthology.orchestra.run.smithi070.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-21T10:22:51.745 INFO:teuthology.orchestra.run.smithi040.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 266 kB/s | 20 kB 00:00 2024-03-21T10:22:51.764 INFO:teuthology.orchestra.run.smithi070.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2024-03-21T10:22:51.780 INFO:teuthology.orchestra.run.smithi040.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-21T10:22:51.790 INFO:teuthology.orchestra.run.smithi070.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-21T10:22:51.832 INFO:teuthology.orchestra.run.smithi070.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 17 MB/s | 1.1 MB 00:00 2024-03-21T10:22:51.846 INFO:teuthology.orchestra.run.smithi040.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-21T10:22:51.857 INFO:teuthology.orchestra.run.smithi070.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-03-21T10:22:51.882 INFO:teuthology.orchestra.run.smithi070.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-21T10:22:51.899 INFO:teuthology.orchestra.run.smithi070.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 4.4 MB/s | 187 kB 00:00 2024-03-21T10:22:52.066 INFO:teuthology.orchestra.run.smithi070.stdout:(91/174): ceph-osd-19.0.0-2231.g2edfee68.el9.x8 1.9 MB/s | 17 MB 00:08 2024-03-21T10:22:52.124 INFO:teuthology.orchestra.run.smithi070.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 775 kB/s | 187 kB 00:00 2024-03-21T10:22:52.138 INFO:teuthology.orchestra.run.smithi040.stdout:(150/174): librados2-19.0.0-2231.g2edfee68.el9. 9.5 MB/s | 3.4 MB 00:00 2024-03-21T10:22:52.150 INFO:teuthology.orchestra.run.smithi070.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-03-21T10:22:52.183 INFO:teuthology.orchestra.run.smithi070.stdout:(94/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.8 MB/s | 803 kB 00:00 2024-03-21T10:22:52.188 INFO:teuthology.orchestra.run.smithi040.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-21T10:22:52.217 INFO:teuthology.orchestra.run.smithi070.stdout:(95/174): python3-requests-2.25.1-8.el9.noarch. 1.3 MB/s | 125 kB 00:00 2024-03-21T10:22:52.255 INFO:teuthology.orchestra.run.smithi040.stdout:(152/174): librbd1-19.0.0-2231.g2edfee68.el9.x8 7.7 MB/s | 3.1 MB 00:00 2024-03-21T10:22:52.431 INFO:teuthology.orchestra.run.smithi040.stdout:(153/174): NetworkManager-1.47.2-1.el9.x86_64.r 9.5 MB/s | 2.3 MB 00:00 2024-03-21T10:22:52.472 INFO:teuthology.orchestra.run.smithi040.stdout:(154/174): NetworkManager-libnm-1.47.2-1.el9.x8 8.5 MB/s | 1.8 MB 00:00 2024-03-21T10:22:52.498 INFO:teuthology.orchestra.run.smithi040.stdout:(155/174): NetworkManager-team-1.47.2-1.el9.x86 631 kB/s | 42 kB 00:00 2024-03-21T10:22:52.523 INFO:teuthology.orchestra.run.smithi040.stdout:(156/174): NetworkManager-tui-1.47.2-1.el9.x86_ 4.8 MB/s | 248 kB 00:00 2024-03-21T10:22:52.548 INFO:teuthology.orchestra.run.smithi040.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-03-21T10:22:52.565 INFO:teuthology.orchestra.run.smithi040.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 871 kB/s | 36 kB 00:00 2024-03-21T10:22:52.590 INFO:teuthology.orchestra.run.smithi040.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 4.5 MB/s | 192 kB 00:00 2024-03-21T10:22:52.616 INFO:teuthology.orchestra.run.smithi040.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 5.0 MB/s | 256 kB 00:00 2024-03-21T10:22:52.641 INFO:teuthology.orchestra.run.smithi040.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 457 kB/s | 23 kB 00:00 2024-03-21T10:22:52.667 INFO:teuthology.orchestra.run.smithi040.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 108 kB 00:00 2024-03-21T10:22:52.692 INFO:teuthology.orchestra.run.smithi040.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 155 kB 00:00 2024-03-21T10:22:52.717 INFO:teuthology.orchestra.run.smithi040.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 136 kB 00:00 2024-03-21T10:22:52.742 INFO:teuthology.orchestra.run.smithi040.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-03-21T10:22:52.768 INFO:teuthology.orchestra.run.smithi040.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 573 kB/s | 29 kB 00:00 2024-03-21T10:22:52.810 INFO:teuthology.orchestra.run.smithi040.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 18 MB/s | 1.2 MB 00:00 2024-03-21T10:22:52.826 INFO:teuthology.orchestra.run.smithi070.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 32 kB/s | 21 kB 00:00 2024-03-21T10:22:52.851 INFO:teuthology.orchestra.run.smithi070.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 38 kB/s | 25 kB 00:00 2024-03-21T10:22:52.885 INFO:teuthology.orchestra.run.smithi040.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 18 MB/s | 2.2 MB 00:00 2024-03-21T10:22:52.944 INFO:teuthology.orchestra.run.smithi040.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 17 MB/s | 2.3 MB 00:00 2024-03-21T10:22:52.969 INFO:teuthology.orchestra.run.smithi040.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 5.4 MB/s | 465 kB 00:00 2024-03-21T10:22:52.994 INFO:teuthology.orchestra.run.smithi040.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 623 kB/s | 31 kB 00:00 2024-03-21T10:22:53.036 INFO:teuthology.orchestra.run.smithi040.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 16 MB/s | 1.1 MB 00:00 2024-03-21T10:22:53.061 INFO:teuthology.orchestra.run.smithi040.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.1 MB/s | 74 kB 00:00 2024-03-21T10:22:53.884 INFO:teuthology.orchestra.run.smithi070.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 232 kB/s | 245 kB 00:01 2024-03-21T10:22:53.917 INFO:teuthology.orchestra.run.smithi070.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 232 kB/s | 248 kB 00:01 2024-03-21T10:22:54.551 INFO:teuthology.orchestra.run.smithi070.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 55 kB/s | 35 kB 00:00 2024-03-21T10:22:54.826 INFO:teuthology.orchestra.run.smithi070.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 230 kB/s | 216 kB 00:00 2024-03-21T10:22:55.176 INFO:teuthology.orchestra.run.smithi070.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 71 kB/s | 44 kB 00:00 2024-03-21T10:22:55.268 INFO:teuthology.orchestra.run.smithi070.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 247 kB/s | 22 kB 00:00 2024-03-21T10:22:55.510 INFO:teuthology.orchestra.run.smithi070.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.5 MB/s | 863 kB 00:00 2024-03-21T10:22:55.618 INFO:teuthology.orchestra.run.smithi070.stdout:(105/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 118 kB/s | 94 kB 00:00 2024-03-21T10:22:55.719 INFO:teuthology.orchestra.run.smithi070.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 2.6 MB/s | 548 kB 00:00 2024-03-21T10:22:55.760 INFO:teuthology.orchestra.run.smithi070.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 469 kB/s | 19 kB 00:00 2024-03-21T10:22:55.802 INFO:teuthology.orchestra.run.smithi070.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.6 MB/s | 309 kB 00:00 2024-03-21T10:22:55.953 INFO:teuthology.orchestra.run.smithi070.stdout:(109/174): libarrow-9.0.0-10.el9.x86_64.rpm 23 MB/s | 4.4 MB 00:00 2024-03-21T10:22:55.978 INFO:teuthology.orchestra.run.smithi070.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 148 kB/s | 26 kB 00:00 2024-03-21T10:22:56.003 INFO:teuthology.orchestra.run.smithi070.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 973 kB/s | 49 kB 00:00 2024-03-21T10:22:56.045 INFO:teuthology.orchestra.run.smithi070.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 3.6 MB/s | 151 kB 00:00 2024-03-21T10:22:56.070 INFO:teuthology.orchestra.run.smithi070.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 731 kB/s | 67 kB 00:00 2024-03-21T10:22:56.104 INFO:teuthology.orchestra.run.smithi070.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 14 MB/s | 838 kB 00:00 2024-03-21T10:22:56.146 INFO:teuthology.orchestra.run.smithi070.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.0 MB/s | 43 kB 00:00 2024-03-21T10:22:56.188 INFO:teuthology.orchestra.run.smithi070.stdout:(116/174): python3-asyncssh-2.13.2-5.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-03-21T10:22:56.213 INFO:teuthology.orchestra.run.smithi070.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 481 kB/s | 32 kB 00:00 2024-03-21T10:22:56.247 INFO:teuthology.orchestra.run.smithi070.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 255 kB/s | 15 kB 00:00 2024-03-21T10:22:56.272 INFO:teuthology.orchestra.run.smithi070.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-03-21T10:22:56.314 INFO:teuthology.orchestra.run.smithi070.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 7.1 MB/s | 303 kB 00:00 2024-03-21T10:22:56.356 INFO:teuthology.orchestra.run.smithi070.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 848 kB/s | 35 kB 00:00 2024-03-21T10:22:56.382 INFO:teuthology.orchestra.run.smithi070.stdout:(122/174): python3-cherrypy-18.6.1-2.el9.noarch 2.6 MB/s | 358 kB 00:00 2024-03-21T10:22:56.406 INFO:teuthology.orchestra.run.smithi070.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 3.9 MB/s | 203 kB 00:00 2024-03-21T10:22:56.450 INFO:teuthology.orchestra.run.smithi070.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 3.4 MB/s | 144 kB 00:00 2024-03-21T10:22:56.499 INFO:teuthology.orchestra.run.smithi070.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 17 MB/s | 2.0 MB 00:00 2024-03-21T10:22:56.516 INFO:teuthology.orchestra.run.smithi070.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-03-21T10:22:56.541 INFO:teuthology.orchestra.run.smithi070.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 423 kB/s | 18 kB 00:00 2024-03-21T10:22:56.566 INFO:teuthology.orchestra.run.smithi070.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-03-21T10:22:56.592 INFO:teuthology.orchestra.run.smithi070.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 388 kB/s | 19 kB 00:00 2024-03-21T10:22:56.617 INFO:teuthology.orchestra.run.smithi070.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-03-21T10:22:56.650 INFO:teuthology.orchestra.run.smithi070.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 18 MB/s | 1.0 MB 00:00 2024-03-21T10:22:56.676 INFO:teuthology.orchestra.run.smithi070.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 791 kB/s | 46 kB 00:00 2024-03-21T10:22:56.701 INFO:teuthology.orchestra.run.smithi070.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-03-21T10:22:56.743 INFO:teuthology.orchestra.run.smithi070.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 865 kB/s | 58 kB 00:00 2024-03-21T10:22:56.776 INFO:teuthology.orchestra.run.smithi070.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-21T10:22:56.802 INFO:teuthology.orchestra.run.smithi070.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 281 kB/s | 16 kB 00:00 2024-03-21T10:22:56.819 INFO:teuthology.orchestra.run.smithi070.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 2.1 MB/s | 90 kB 00:00 2024-03-21T10:22:56.844 INFO:teuthology.orchestra.run.smithi070.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 674 kB/s | 28 kB 00:00 2024-03-21T10:22:56.869 INFO:teuthology.orchestra.run.smithi070.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 613 kB/s | 31 kB 00:00 2024-03-21T10:22:56.894 INFO:teuthology.orchestra.run.smithi070.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-03-21T10:22:56.920 INFO:teuthology.orchestra.run.smithi070.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-03-21T10:22:56.945 INFO:teuthology.orchestra.run.smithi070.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 716 kB/s | 36 kB 00:00 2024-03-21T10:22:56.970 INFO:teuthology.orchestra.run.smithi070.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-03-21T10:22:56.996 INFO:teuthology.orchestra.run.smithi070.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-21T10:22:57.021 INFO:teuthology.orchestra.run.smithi070.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-03-21T10:22:57.046 INFO:teuthology.orchestra.run.smithi070.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 8.3 MB/s | 427 kB 00:00 2024-03-21T10:22:57.071 INFO:teuthology.orchestra.run.smithi070.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-03-21T10:22:57.097 INFO:teuthology.orchestra.run.smithi070.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-03-21T10:22:57.147 INFO:teuthology.orchestra.run.smithi070.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-03-21T10:22:57.540 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-21T10:22:57.614 INFO:teuthology.orchestra.run.smithi070.stdout:(150/174): librbd1-19.0.0-2231.g2edfee68.el9.x8 6.7 MB/s | 3.1 MB 00:00 2024-03-21T10:22:57.673 INFO:teuthology.orchestra.run.smithi070.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 5.4 MB/s | 321 kB 00:00 2024-03-21T10:22:57.814 INFO:teuthology.orchestra.run.smithi070.stdout:(152/174): NetworkManager-1.47.2-1.el9.x86_64.r 16 MB/s | 2.3 MB 00:00 2024-03-21T10:22:57.948 INFO:teuthology.orchestra.run.smithi070.stdout:(153/174): NetworkManager-libnm-1.47.2-1.el9.x8 14 MB/s | 1.8 MB 00:00 2024-03-21T10:22:58.007 INFO:teuthology.orchestra.run.smithi070.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 722 kB/s | 42 kB 00:00 2024-03-21T10:22:58.090 INFO:teuthology.orchestra.run.smithi070.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.9 MB/s | 248 kB 00:00 2024-03-21T10:22:58.149 INFO:teuthology.orchestra.run.smithi070.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-03-21T10:22:58.199 INFO:teuthology.orchestra.run.smithi070.stdout:(157/174): librados2-19.0.0-2231.g2edfee68.el9. 3.1 MB/s | 3.4 MB 00:01 2024-03-21T10:22:58.241 INFO:teuthology.orchestra.run.smithi070.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 397 kB/s | 36 kB 00:00 2024-03-21T10:22:58.291 INFO:teuthology.orchestra.run.smithi070.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.0 MB/s | 192 kB 00:00 2024-03-21T10:22:58.333 INFO:teuthology.orchestra.run.smithi070.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.7 MB/s | 256 kB 00:00 2024-03-21T10:22:58.350 INFO:teuthology.orchestra.run.smithi070.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 394 kB/s | 23 kB 00:00 2024-03-21T10:22:58.417 INFO:teuthology.orchestra.run.smithi070.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 108 kB 00:00 2024-03-21T10:22:58.451 INFO:teuthology.orchestra.run.smithi070.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.5 MB/s | 155 kB 00:00 2024-03-21T10:22:58.509 INFO:teuthology.orchestra.run.smithi070.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 136 kB 00:00 2024-03-21T10:22:58.626 INFO:teuthology.orchestra.run.smithi070.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 364 kB/s | 64 kB 00:00 2024-03-21T10:22:58.735 INFO:teuthology.orchestra.run.smithi070.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 127 kB/s | 29 kB 00:00 2024-03-21T10:22:58.794 INFO:teuthology.orchestra.run.smithi070.stdout:(167/174): openssl-devel-3.0.7-27.el9.x86_64.rp 643 kB/s | 4.1 MB 00:06 2024-03-21T10:22:58.886 INFO:teuthology.orchestra.run.smithi070.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 4.7 MB/s | 1.2 MB 00:00 2024-03-21T10:22:58.986 INFO:teuthology.orchestra.run.smithi070.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 8.6 MB/s | 2.2 MB 00:00 2024-03-21T10:22:59.061 INFO:teuthology.orchestra.run.smithi070.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 8.6 MB/s | 2.3 MB 00:00 2024-03-21T10:22:59.086 INFO:teuthology.orchestra.run.smithi070.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 311 kB/s | 31 kB 00:00 2024-03-21T10:22:59.120 INFO:teuthology.orchestra.run.smithi070.stdout:(172/174): util-linux-core-2.37.4-18.el9.x86_64 1.9 MB/s | 465 kB 00:00 2024-03-21T10:22:59.203 INFO:teuthology.orchestra.run.smithi070.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 7.6 MB/s | 1.1 MB 00:00 2024-03-21T10:22:59.745 INFO:teuthology.orchestra.run.smithi070.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 113 kB/s | 74 kB 00:00 2024-03-21T10:22:59.751 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-03-21T10:22:59.751 INFO:teuthology.orchestra.run.smithi070.stdout:Total 12 MB/s | 224 MB 00:19 2024-03-21T10:23:00.219 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-03-21T10:23:00.438 INFO:teuthology.orchestra.run.smithi031.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-21T10:23:00.440 INFO:teuthology.orchestra.run.smithi031.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-21T10:23:00.467 INFO:teuthology.orchestra.run.smithi031.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-21T10:23:00.468 INFO:teuthology.orchestra.run.smithi031.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-21T10:23:00.470 INFO:teuthology.orchestra.run.smithi031.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-21T10:23:00.472 INFO:teuthology.orchestra.run.smithi031.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-21T10:23:00.475 INFO:teuthology.orchestra.run.smithi031.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-21T10:23:00.479 INFO:teuthology.orchestra.run.smithi031.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-21T10:23:00.480 INFO:teuthology.orchestra.run.smithi031.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-21T10:23:00.490 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-03-21T10:23:00.490 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-03-21T10:23:00.580 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-21T10:23:00.606 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2024-03-21T10:23:00.606 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-03-21T10:23:00.606 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2024-03-21T10:23:00.606 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-03-21T10:23:00.607 INFO:teuthology.orchestra.run.smithi031.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-21T10:23:00.607 INFO:teuthology.orchestra.run.smithi031.stdout: ceph x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 6.6 k 2024-03-21T10:23:00.607 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 5.4 M 2024-03-21T10:23:00.607 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 890 k 2024-03-21T10:23:00.607 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 145 k 2024-03-21T10:23:00.607 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 1.5 M 2024-03-21T10:23:00.607 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 140 k 2024-03-21T10:23:00.607 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 1.8 M 2024-03-21T10:23:00.607 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 7.4 M 2024-03-21T10:23:00.607 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 49 k 2024-03-21T10:23:00.607 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 9.9 M 2024-03-21T10:23:00.607 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 48 M 2024-03-21T10:23:00.607 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 264 k 2024-03-21T10:23:00.608 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 750 k 2024-03-21T10:23:00.608 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-mirror x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 220 k 2024-03-21T10:23:00.608 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-shell x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 27 k 2024-03-21T10:23:00.608 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-top noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 18 k 2024-03-21T10:23:00.608 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-21T10:23:00.608 INFO:teuthology.orchestra.run.smithi031.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-21T10:23:00.608 INFO:teuthology.orchestra.run.smithi031.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-21T10:23:00.608 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 31 k 2024-03-21T10:23:00.608 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2 x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 747 k 2024-03-21T10:23:00.608 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 127 k 2024-03-21T10:23:00.608 INFO:teuthology.orchestra.run.smithi031.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-21T10:23:00.608 INFO:teuthology.orchestra.run.smithi031.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-21T10:23:00.609 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-21T10:23:00.609 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-21T10:23:00.609 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-21T10:23:00.609 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-21T10:23:00.609 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 161 k 2024-03-21T10:23:00.609 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 320 k 2024-03-21T10:23:00.609 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 297 k 2024-03-21T10:23:00.609 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 99 k 2024-03-21T10:23:00.609 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 85 k 2024-03-21T10:23:00.609 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 3.1 M 2024-03-21T10:23:00.609 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 171 k 2024-03-21T10:23:00.609 INFO:teuthology.orchestra.run.smithi031.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-21T10:23:00.610 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrading: 2024-03-21T10:23:00.610 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-21T10:23:00.610 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-21T10:23:00.610 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-21T10:23:00.610 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-21T10:23:00.610 INFO:teuthology.orchestra.run.smithi031.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-21T10:23:00.610 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-21T10:23:00.610 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-21T10:23:00.610 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-21T10:23:00.610 INFO:teuthology.orchestra.run.smithi031.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-21T10:23:00.610 INFO:teuthology.orchestra.run.smithi031.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-21T10:23:00.611 INFO:teuthology.orchestra.run.smithi031.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-21T10:23:00.611 INFO:teuthology.orchestra.run.smithi031.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-21T10:23:00.611 INFO:teuthology.orchestra.run.smithi031.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-21T10:23:00.611 INFO:teuthology.orchestra.run.smithi031.stdout: librados2 x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 3.4 M 2024-03-21T10:23:00.611 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1 x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 3.1 M 2024-03-21T10:23:00.611 INFO:teuthology.orchestra.run.smithi031.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-21T10:23:00.611 INFO:teuthology.orchestra.run.smithi031.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-21T10:23:00.611 INFO:teuthology.orchestra.run.smithi031.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-21T10:23:00.611 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-21T10:23:00.611 INFO:teuthology.orchestra.run.smithi031.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-21T10:23:00.611 INFO:teuthology.orchestra.run.smithi031.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-21T10:23:00.611 INFO:teuthology.orchestra.run.smithi031.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-21T10:23:00.611 INFO:teuthology.orchestra.run.smithi031.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-21T10:23:00.611 INFO:teuthology.orchestra.run.smithi031.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-21T10:23:00.612 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-03-21T10:23:00.612 INFO:teuthology.orchestra.run.smithi031.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-21T10:23:00.612 INFO:teuthology.orchestra.run.smithi031.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-21T10:23:00.612 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-21T10:23:00.612 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 21 M 2024-03-21T10:23:00.612 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 34 k 2024-03-21T10:23:00.612 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 2.3 M 2024-03-21T10:23:00.612 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 248 k 2024-03-21T10:23:00.612 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 4.7 M 2024-03-21T10:23:00.612 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 17 M 2024-03-21T10:23:00.612 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2231.g2edfee68.el9 ceph-noarch 16 k 2024-03-21T10:23:00.612 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 25 k 2024-03-21T10:23:00.612 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-21T10:23:00.612 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-21T10:23:00.613 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-21T10:23:00.613 INFO:teuthology.orchestra.run.smithi031.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-21T10:23:00.613 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-21T10:23:00.613 INFO:teuthology.orchestra.run.smithi031.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-21T10:23:00.613 INFO:teuthology.orchestra.run.smithi031.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-21T10:23:00.613 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-21T10:23:00.613 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-21T10:23:00.613 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 163 k 2024-03-21T10:23:00.613 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-21T10:23:00.613 INFO:teuthology.orchestra.run.smithi031.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-21T10:23:00.613 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-21T10:23:00.613 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-21T10:23:00.613 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-21T10:23:00.613 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1 x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 501 k 2024-03-21T10:23:00.614 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-21T10:23:00.614 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2 x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 5.1 M 2024-03-21T10:23:00.614 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-21T10:23:00.614 INFO:teuthology.orchestra.run.smithi031.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-21T10:23:00.614 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-21T10:23:00.614 INFO:teuthology.orchestra.run.smithi031.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-21T10:23:00.614 INFO:teuthology.orchestra.run.smithi031.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-21T10:23:00.614 INFO:teuthology.orchestra.run.smithi031.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-21T10:23:00.614 INFO:teuthology.orchestra.run.smithi031.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-21T10:23:00.614 INFO:teuthology.orchestra.run.smithi031.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-21T10:23:00.614 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-21T10:23:00.614 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-21T10:23:00.615 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-21T10:23:00.615 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-21T10:23:00.615 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-21T10:23:00.615 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-21T10:23:00.615 INFO:teuthology.orchestra.run.smithi031.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-21T10:23:00.615 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-21T10:23:00.615 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-21T10:23:00.615 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-21T10:23:00.615 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 45 k 2024-03-21T10:23:00.615 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common x86_64 2:19.0.0-2231.g2edfee68.el9 ceph 131 k 2024-03-21T10:23:00.615 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-21T10:23:00.615 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-21T10:23:00.615 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-21T10:23:00.615 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-21T10:23:00.616 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-21T10:23:00.616 INFO:teuthology.orchestra.run.smithi031.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-21T10:23:00.616 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-21T10:23:00.616 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-21T10:23:00.616 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-21T10:23:00.616 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-21T10:23:00.616 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-21T10:23:00.616 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-21T10:23:00.616 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-21T10:23:00.616 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-21T10:23:00.616 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-21T10:23:00.616 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-21T10:23:00.616 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-21T10:23:00.617 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-21T10:23:00.617 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-21T10:23:00.617 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-21T10:23:00.617 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-21T10:23:00.617 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-21T10:23:00.617 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-21T10:23:00.617 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-21T10:23:00.617 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-21T10:23:00.617 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-21T10:23:00.618 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-21T10:23:00.618 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-21T10:23:00.618 INFO:teuthology.orchestra.run.smithi031.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-21T10:23:00.618 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-21T10:23:00.618 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-21T10:23:00.618 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-21T10:23:00.618 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-21T10:23:00.618 INFO:teuthology.orchestra.run.smithi031.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-21T10:23:00.618 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-21T10:23:00.618 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-21T10:23:00.619 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-21T10:23:00.619 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-21T10:23:00.619 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-21T10:23:00.619 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-21T10:23:00.619 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-21T10:23:00.619 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-21T10:23:00.619 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-21T10:23:00.619 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-21T10:23:00.619 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-21T10:23:00.619 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-21T10:23:00.619 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-21T10:23:00.619 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-21T10:23:00.619 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-21T10:23:00.620 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-21T10:23:00.620 INFO:teuthology.orchestra.run.smithi031.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-21T10:23:00.620 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-21T10:23:00.620 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-21T10:23:00.620 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-21T10:23:00.620 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-21T10:23:00.620 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-21T10:23:00.620 INFO:teuthology.orchestra.run.smithi031.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-21T10:23:00.620 INFO:teuthology.orchestra.run.smithi031.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-21T10:23:00.620 INFO:teuthology.orchestra.run.smithi031.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-21T10:23:00.620 INFO:teuthology.orchestra.run.smithi031.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-21T10:23:00.620 INFO:teuthology.orchestra.run.smithi031.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-21T10:23:00.620 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-21T10:23:00.621 INFO:teuthology.orchestra.run.smithi031.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-21T10:23:00.621 INFO:teuthology.orchestra.run.smithi031.stdout:Installing weak dependencies: 2024-03-21T10:23:00.621 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-21T10:23:00.621 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:23:00.621 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-03-21T10:23:00.621 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2024-03-21T10:23:00.621 INFO:teuthology.orchestra.run.smithi031.stdout:Install 150 Packages 2024-03-21T10:23:00.621 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrade 24 Packages 2024-03-21T10:23:00.621 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:23:00.622 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 224 M 2024-03-21T10:23:00.622 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-03-21T10:23:01.436 INFO:teuthology.orchestra.run.smithi040.stdout:(174/174): ceph-test-19.0.0-2231.g2edfee68.el9. 2.5 MB/s | 48 MB 00:19 2024-03-21T10:23:01.443 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-03-21T10:23:01.443 INFO:teuthology.orchestra.run.smithi040.stdout:Total 10 MB/s | 224 MB 00:22 2024-03-21T10:23:01.717 INFO:teuthology.orchestra.run.smithi031.stdout:(1/174): ceph-19.0.0-2231.g2edfee68.el9.x86_64. 21 kB/s | 6.6 kB 00:00 2024-03-21T10:23:01.937 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-03-21T10:23:02.208 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-03-21T10:23:02.209 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-03-21T10:23:02.241 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-03-21T10:23:02.241 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-03-21T10:23:02.809 INFO:teuthology.orchestra.run.smithi031.stdout:(2/174): ceph-common-19.0.0-2231.g2edfee68.el9. 15 MB/s | 21 MB 00:01 2024-03-21T10:23:02.850 INFO:teuthology.orchestra.run.smithi031.stdout:(3/174): ceph-fuse-19.0.0-2231.g2edfee68.el9.x8 786 kB/s | 890 kB 00:01 2024-03-21T10:23:02.884 INFO:teuthology.orchestra.run.smithi031.stdout:(4/174): ceph-immutable-object-cache-19.0.0-223 1.9 MB/s | 145 kB 00:00 2024-03-21T10:23:03.001 INFO:teuthology.orchestra.run.smithi031.stdout:(5/174): ceph-mgr-19.0.0-2231.g2edfee68.el9.x86 13 MB/s | 1.5 MB 00:00 2024-03-21T10:23:03.293 INFO:teuthology.orchestra.run.smithi031.stdout:(6/174): ceph-mon-19.0.0-2231.g2edfee68.el9.x86 16 MB/s | 4.7 MB 00:00 2024-03-21T10:23:03.801 INFO:teuthology.orchestra.run.smithi031.stdout:(7/174): ceph-base-19.0.0-2231.g2edfee68.el9.x8 2.2 MB/s | 5.4 MB 00:02 2024-03-21T10:23:04.035 INFO:teuthology.orchestra.run.smithi031.stdout:(8/174): ceph-mds-19.0.0-2231.g2edfee68.el9.x86 2.0 MB/s | 2.3 MB 00:01 2024-03-21T10:23:04.067 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-03-21T10:23:04.067 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-03-21T10:23:04.285 INFO:teuthology.orchestra.run.smithi031.stdout:(9/174): ceph-osd-19.0.0-2231.g2edfee68.el9.x86 17 MB/s | 17 MB 00:00 2024-03-21T10:23:04.310 INFO:teuthology.orchestra.run.smithi031.stdout:(10/174): ceph-selinux-19.0.0-2231.g2edfee68.el 92 kB/s | 25 kB 00:00 2024-03-21T10:23:04.453 INFO:teuthology.orchestra.run.smithi031.stdout:(11/174): cephfs-mirror-19.0.0-2231.g2edfee68.e 1.5 MB/s | 220 kB 00:00 2024-03-21T10:23:04.553 INFO:teuthology.orchestra.run.smithi031.stdout:(12/174): cephfs-shell-19.0.0-2231.g2edfee68.el 273 kB/s | 27 kB 00:00 2024-03-21T10:23:04.645 INFO:teuthology.orchestra.run.smithi031.stdout:(13/174): libcephfs-devel-19.0.0-2231.g2edfee68 345 kB/s | 31 kB 00:00 2024-03-21T10:23:04.995 INFO:teuthology.orchestra.run.smithi031.stdout:(14/174): libcephfs2-19.0.0-2231.g2edfee68.el9. 2.1 MB/s | 747 kB 00:00 2024-03-21T10:23:05.103 INFO:teuthology.orchestra.run.smithi031.stdout:(15/174): libcephsqlite-19.0.0-2231.g2edfee68.e 1.5 MB/s | 163 kB 00:00 2024-03-21T10:23:05.204 INFO:teuthology.orchestra.run.smithi031.stdout:(16/174): librados-devel-19.0.0-2231.g2edfee68. 1.2 MB/s | 127 kB 00:00 2024-03-21T10:23:05.404 INFO:teuthology.orchestra.run.smithi031.stdout:(17/174): libradosstriper1-19.0.0-2231.g2edfee6 2.5 MB/s | 501 kB 00:00 2024-03-21T10:23:06.896 INFO:teuthology.orchestra.run.smithi031.stdout:(18/174): ceph-test-19.0.0-2231.g2edfee68.el9.x 18 MB/s | 48 MB 00:02 2024-03-21T10:23:06.938 INFO:teuthology.orchestra.run.smithi031.stdout:(19/174): python3-ceph-argparse-19.0.0-2231.g2e 1.1 MB/s | 45 kB 00:00 2024-03-21T10:23:07.080 INFO:teuthology.orchestra.run.smithi031.stdout:(20/174): librgw2-19.0.0-2231.g2edfee68.el9.x86 3.1 MB/s | 5.1 MB 00:01 2024-03-21T10:23:07.114 INFO:teuthology.orchestra.run.smithi031.stdout:(21/174): python3-ceph-common-19.0.0-2231.g2edf 754 kB/s | 131 kB 00:00 2024-03-21T10:23:07.205 INFO:teuthology.orchestra.run.smithi031.stdout:(22/174): python3-cephfs-19.0.0-2231.g2edfee68. 1.3 MB/s | 161 kB 00:00 2024-03-21T10:23:07.239 INFO:teuthology.orchestra.run.smithi031.stdout:(23/174): python3-rados-19.0.0-2231.g2edfee68.e 2.5 MB/s | 320 kB 00:00 2024-03-21T10:23:07.314 INFO:teuthology.orchestra.run.smithi031.stdout:(24/174): python3-rgw-19.0.0-2231.g2edfee68.el9 1.3 MB/s | 99 kB 00:00 2024-03-21T10:23:07.356 INFO:teuthology.orchestra.run.smithi031.stdout:(25/174): python3-rbd-19.0.0-2231.g2edfee68.el9 1.9 MB/s | 297 kB 00:00 2024-03-21T10:23:07.415 INFO:teuthology.orchestra.run.smithi031.stdout:(26/174): rbd-fuse-19.0.0-2231.g2edfee68.el9.x8 857 kB/s | 85 kB 00:00 2024-03-21T10:23:07.507 INFO:teuthology.orchestra.run.smithi031.stdout:(27/174): rbd-nbd-19.0.0-2231.g2edfee68.el9.x86 1.8 MB/s | 171 kB 00:00 2024-03-21T10:23:07.674 INFO:teuthology.orchestra.run.smithi031.stdout:(28/174): ceph-radosgw-19.0.0-2231.g2edfee68.el 2.5 MB/s | 9.9 MB 00:03 2024-03-21T10:23:07.699 INFO:teuthology.orchestra.run.smithi031.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2231.g 177 kB/s | 34 kB 00:00 2024-03-21T10:23:07.791 INFO:teuthology.orchestra.run.smithi031.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2231.g2edfee6 1.2 MB/s | 140 kB 00:00 2024-03-21T10:23:08.216 INFO:teuthology.orchestra.run.smithi031.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2231.g2edfe 3.5 MB/s | 1.8 MB 00:00 2024-03-21T10:23:08.275 INFO:teuthology.orchestra.run.smithi031.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2231.g2e 4.2 MB/s | 248 kB 00:00 2024-03-21T10:23:08.359 INFO:teuthology.orchestra.run.smithi031.stdout:(33/174): rbd-mirror-19.0.0-2231.g2edfee68.el9. 3.1 MB/s | 3.1 MB 00:01 2024-03-21T10:23:08.384 INFO:teuthology.orchestra.run.smithi031.stdout:(34/174): ceph-mgr-rook-19.0.0-2231.g2edfee68.e 458 kB/s | 49 kB 00:00 2024-03-21T10:23:08.451 INFO:teuthology.orchestra.run.smithi031.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2231.g2 180 kB/s | 16 kB 00:00 2024-03-21T10:23:08.484 INFO:teuthology.orchestra.run.smithi031.stdout:(36/174): ceph-volume-19.0.0-2231.g2edfee68.el9 2.6 MB/s | 264 kB 00:00 2024-03-21T10:23:08.551 INFO:teuthology.orchestra.run.smithi031.stdout:(37/174): cephfs-top-19.0.0-2231.g2edfee68.el9. 276 kB/s | 18 kB 00:00 2024-03-21T10:23:08.743 INFO:teuthology.orchestra.run.smithi031.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 3.4 MB/s | 662 kB 00:00 2024-03-21T10:23:08.819 INFO:teuthology.orchestra.run.smithi031.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 12 MB/s | 920 kB 00:00 2024-03-21T10:23:08.860 INFO:teuthology.orchestra.run.smithi031.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2024-03-21T10:23:08.902 INFO:teuthology.orchestra.run.smithi031.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.3 MB/s | 307 kB 00:00 2024-03-21T10:23:08.936 INFO:teuthology.orchestra.run.smithi031.stdout:(42/174): cephadm-19.0.0-2231.g2edfee68.el9.noa 1.5 MB/s | 750 kB 00:00 2024-03-21T10:23:08.961 INFO:teuthology.orchestra.run.smithi031.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 546 kB/s | 32 kB 00:00 2024-03-21T10:23:09.055 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-03-21T10:23:09.062 INFO:teuthology.orchestra.run.smithi031.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 24 MB/s | 3.0 MB 00:00 2024-03-21T10:23:09.087 INFO:teuthology.orchestra.run.smithi031.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 158 kB/s | 17 kB 00:00 2024-03-21T10:23:09.145 INFO:teuthology.orchestra.run.smithi031.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 122 kB/s | 10 kB 00:00 2024-03-21T10:23:09.165 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-21T10:23:09.188 INFO:teuthology.orchestra.run.smithi031.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 159 kB 00:00 2024-03-21T10:23:09.199 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-21T10:23:09.213 INFO:teuthology.orchestra.run.smithi031.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 680 kB/s | 44 kB 00:00 2024-03-21T10:23:09.238 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-21T10:23:09.247 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-21T10:23:09.255 INFO:teuthology.orchestra.run.smithi031.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.8 MB/s | 661 kB 00:00 2024-03-21T10:23:09.296 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-21T10:23:09.297 INFO:teuthology.orchestra.run.smithi031.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 7.4 MB/s | 578 kB 00:00 2024-03-21T10:23:09.322 INFO:teuthology.orchestra.run.smithi031.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 760 kB/s | 46 kB 00:00 2024-03-21T10:23:09.349 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-21T10:23:09.356 INFO:teuthology.orchestra.run.smithi031.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2024-03-21T10:23:09.383 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-21T10:23:09.389 INFO:teuthology.orchestra.run.smithi031.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 2.9 MB/s | 187 kB 00:00 2024-03-21T10:23:09.414 INFO:teuthology.orchestra.run.smithi031.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.8 MB/s | 217 kB 00:00 2024-03-21T10:23:09.417 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-21T10:23:09.440 INFO:teuthology.orchestra.run.smithi031.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 718 kB/s | 34 kB 00:00 2024-03-21T10:23:09.458 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-21T10:23:09.690 INFO:teuthology.orchestra.run.smithi031.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 17 MB/s | 4.7 MB 00:00 2024-03-21T10:23:09.749 INFO:teuthology.orchestra.run.smithi031.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 5.1 MB/s | 1.5 MB 00:00 2024-03-21T10:23:09.779 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-21T10:23:09.824 INFO:teuthology.orchestra.run.smithi031.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 6.6 MB/s | 811 kB 00:00 2024-03-21T10:23:09.834 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : librados2-2:19.0.0-2231.g2edfee68.el9.x86_64 10/198 2024-03-21T10:23:09.858 INFO:teuthology.orchestra.run.smithi031.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 1.3 MB/s | 134 kB 00:00 2024-03-21T10:23:09.947 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados2-2:19.0.0-2231.g2edfee68.el9.x86_64 10/198 2024-03-21T10:23:09.969 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephfs2-2:19.0.0-2231.g2edfee68.el9.x86_64 11/198 2024-03-21T10:23:10.043 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephfs2-2:19.0.0-2231.g2edfee68.el9.x86_64 11/198 2024-03-21T10:23:10.073 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rados-2:19.0.0-2231.g2edfee68.el9.x86_64 12/198 2024-03-21T10:23:10.083 INFO:teuthology.orchestra.run.smithi031.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 22 MB/s | 5.7 MB 00:00 2024-03-21T10:23:10.111 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-21T10:23:10.134 INFO:teuthology.orchestra.run.smithi031.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 3.8 MB/s | 1.0 MB 00:00 2024-03-21T10:23:10.184 INFO:teuthology.orchestra.run.smithi031.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 855 kB/s | 78 kB 00:00 2024-03-21T10:23:10.289 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-21T10:23:10.384 INFO:teuthology.orchestra.run.smithi031.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 5.8 MB 00:00 2024-03-21T10:23:10.405 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-21T10:23:10.409 INFO:teuthology.orchestra.run.smithi031.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 117 kB/s | 24 kB 00:00 2024-03-21T10:23:10.443 INFO:teuthology.orchestra.run.smithi031.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 564 kB/s | 18 kB 00:00 2024-03-21T10:23:10.460 INFO:teuthology.orchestra.run.smithi031.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 144 kB 00:00 2024-03-21T10:23:10.485 INFO:teuthology.orchestra.run.smithi031.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 779 kB/s | 32 kB 00:00 2024-03-21T10:23:10.582 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephsqlite-2:19.0.0-2231.g2edfee68.el9.x86_64 16/198 2024-03-21T10:23:10.677 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2231.g2edfee68.el9.x86_64 16/198 2024-03-21T10:23:10.694 INFO:teuthology.orchestra.run.smithi031.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-03-21T10:23:10.699 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libradosstriper1-2:19.0.0-2231.g2edfee68.el9.x86 17/198 2024-03-21T10:23:10.727 INFO:teuthology.orchestra.run.smithi031.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 937 kB/s | 217 kB 00:00 2024-03-21T10:23:10.761 INFO:teuthology.orchestra.run.smithi031.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 2.7 MB/s | 174 kB 00:00 2024-03-21T10:23:10.763 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2231.g2edfee68.el9.x86 17/198 2024-03-21T10:23:10.786 INFO:teuthology.orchestra.run.smithi031.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.2 MB/s | 69 kB 00:00 2024-03-21T10:23:10.803 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-21T10:23:10.811 INFO:teuthology.orchestra.run.smithi031.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 845 kB/s | 41 kB 00:00 2024-03-21T10:23:10.826 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-21T10:23:10.841 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-21T10:23:10.875 INFO:teuthology.orchestra.run.smithi031.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 2.7 MB/s | 238 kB 00:00 2024-03-21T10:23:10.912 INFO:teuthology.orchestra.run.smithi031.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 132 kB 00:00 2024-03-21T10:23:10.946 INFO:teuthology.orchestra.run.smithi031.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.0 MB/s | 211 kB 00:00 2024-03-21T10:23:10.979 INFO:teuthology.orchestra.run.smithi031.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 711 kB/s | 47 kB 00:00 2024-03-21T10:23:11.013 INFO:teuthology.orchestra.run.smithi031.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 644 kB/s | 43 kB 00:00 2024-03-21T10:23:11.025 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-21T10:23:11.044 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-03-21T10:23:11.058 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-21T10:23:11.082 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-21T10:23:11.148 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-21T10:23:11.182 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-21T10:23:11.185 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-21T10:23:11.213 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-21T10:23:11.223 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-21T10:23:11.266 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-21T10:23:11.293 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-21T10:23:11.328 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-21T10:23:11.353 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-21T10:23:11.361 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-21T10:23:11.386 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-21T10:23:11.429 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-21T10:23:11.431 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-21T10:23:11.447 INFO:teuthology.orchestra.run.smithi031.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 89 kB/s | 39 kB 00:00 2024-03-21T10:23:11.488 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-21T10:23:11.695 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-21T10:23:11.775 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-21T10:23:11.805 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-21T10:23:11.820 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librados2-2:19.0.0-2231.g2edfee68.el9.x86_64 10/198 2024-03-21T10:23:11.822 INFO:teuthology.orchestra.run.smithi031.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 19 MB/s | 16 MB 00:00 2024-03-21T10:23:11.847 INFO:teuthology.orchestra.run.smithi031.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 105 kB/s | 41 kB 00:00 2024-03-21T10:23:11.881 INFO:teuthology.orchestra.run.smithi031.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.2 MB/s | 300 kB 00:00 2024-03-21T10:23:11.906 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-21T10:23:11.931 INFO:teuthology.orchestra.run.smithi031.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 703 kB/s | 56 kB 00:00 2024-03-21T10:23:11.945 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:19.0.0-2231.g2edfee68.el9.x86_64 10/198 2024-03-21T10:23:11.951 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-21T10:23:11.965 INFO:teuthology.orchestra.run.smithi031.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 761 kB/s | 63 kB 00:00 2024-03-21T10:23:11.978 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-21T10:23:11.979 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs2-2:19.0.0-2231.g2edfee68.el9.x86_64 11/198 2024-03-21T10:23:11.990 INFO:teuthology.orchestra.run.smithi031.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 770 kB/s | 44 kB 00:00 2024-03-21T10:23:12.014 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-21T10:23:12.015 INFO:teuthology.orchestra.run.smithi031.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.6 MB/s | 79 kB 00:00 2024-03-21T10:23:12.049 INFO:teuthology.orchestra.run.smithi031.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.1 MB/s | 241 kB 00:00 2024-03-21T10:23:12.064 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephfs2-2:19.0.0-2231.g2edfee68.el9.x86_64 11/198 2024-03-21T10:23:12.088 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-21T10:23:12.092 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rados-2:19.0.0-2231.g2edfee68.el9.x86_64 12/198 2024-03-21T10:23:12.099 INFO:teuthology.orchestra.run.smithi031.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-21T10:23:12.116 INFO:teuthology.orchestra.run.smithi031.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.9 MB/s | 103 kB 00:00 2024-03-21T10:23:12.135 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-21T10:23:12.141 INFO:teuthology.orchestra.run.smithi031.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 3.0 MB/s | 124 kB 00:00 2024-03-21T10:23:12.142 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-21T10:23:12.175 INFO:teuthology.orchestra.run.smithi031.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 187 kB 00:00 2024-03-21T10:23:12.180 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-21T10:23:12.245 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-21T10:23:12.276 INFO:teuthology.orchestra.run.smithi031.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-21T10:23:12.279 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-21T10:23:12.319 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-21T10:23:12.323 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-21T10:23:12.334 INFO:teuthology.orchestra.run.smithi031.stdout:(92/174): libgfortran-11.4.1-3.el9.x86_64.rpm 5.0 MB/s | 803 kB 00:00 2024-03-21T10:23:12.355 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-21T10:23:12.374 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-21T10:23:12.376 INFO:teuthology.orchestra.run.smithi031.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.9 MB/s | 195 kB 00:00 2024-03-21T10:23:12.383 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-21T10:23:12.402 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephsqlite-2:19.0.0-2231.g2edfee68.el9.x86_64 16/198 2024-03-21T10:23:12.418 INFO:teuthology.orchestra.run.smithi031.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-03-21T10:23:12.454 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-21T10:23:12.485 INFO:teuthology.orchestra.run.smithi031.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 200 kB/s | 21 kB 00:00 2024-03-21T10:23:12.489 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ceph-common-2:19.0.0-2231.g2edfee68.el9. 39/198 2024-03-21T10:23:12.507 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2231.g2edfee68.el9.x86_64 16/198 2024-03-21T10:23:12.510 INFO:teuthology.orchestra.run.smithi031.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 279 kB/s | 25 kB 00:00 2024-03-21T10:23:12.531 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ceph-argparse-2:19.0.0-2231.g2edfee68.el 40/198 2024-03-21T10:23:12.536 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libradosstriper1-2:19.0.0-2231.g2edfee68.el9.x86 17/198 2024-03-21T10:23:12.560 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cephfs-2:19.0.0-2231.g2edfee68.el9.x86_6 41/198 2024-03-21T10:23:12.600 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2231.g2edfee68.el9.x86 17/198 2024-03-21T10:23:12.629 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-21T10:23:12.631 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-21T10:23:12.654 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-21T10:23:12.670 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-21T10:23:12.683 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-21T10:23:12.756 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-21T10:23:12.769 INFO:teuthology.orchestra.run.smithi031.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 952 kB/s | 245 kB 00:00 2024-03-21T10:23:12.813 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-21T10:23:12.844 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-21T10:23:12.876 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-21T10:23:12.890 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-21T10:23:12.902 INFO:teuthology.orchestra.run.smithi031.stdout:(98/174): ceph-mgr-diskprediction-local-19.0.0- 1.4 MB/s | 7.4 MB 00:05 2024-03-21T10:23:12.944 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-21T10:23:12.969 INFO:teuthology.orchestra.run.smithi031.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.2 MB/s | 248 kB 00:00 2024-03-21T10:23:12.976 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-21T10:23:13.045 INFO:teuthology.orchestra.run.smithi031.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 468 kB/s | 35 kB 00:00 2024-03-21T10:23:13.086 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-21T10:23:13.121 INFO:teuthology.orchestra.run.smithi031.stdout:(101/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 592 kB/s | 44 kB 00:00 2024-03-21T10:23:13.156 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-21T10:23:13.179 INFO:teuthology.orchestra.run.smithi031.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-03-21T10:23:13.222 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-21T10:23:13.297 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-21T10:23:13.346 INFO:teuthology.orchestra.run.smithi031.stdout:(103/174): qatlib-23.11.0-1.el9.x86_64.rpm 490 kB/s | 216 kB 00:00 2024-03-21T10:23:13.446 INFO:teuthology.orchestra.run.smithi031.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 84 kB/s | 22 kB 00:00 2024-03-21T10:23:13.467 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-21T10:23:13.518 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-21T10:23:13.548 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-21T10:23:13.618 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-21T10:23:13.632 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-21T10:23:13.668 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-21T10:23:13.676 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-21T10:23:13.708 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-21T10:23:13.730 INFO:teuthology.orchestra.run.smithi031.stdout:(105/174): openssl-devel-3.0.7-27.el9.x86_64.rp 3.3 MB/s | 4.1 MB 00:01 2024-03-21T10:23:13.744 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-21T10:23:13.757 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-21T10:23:13.772 INFO:teuthology.orchestra.run.smithi031.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.0 MB/s | 863 kB 00:00 2024-03-21T10:23:13.777 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-21T10:23:13.795 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librados-devel-2:19.0.0-2231.g2edfee68.el9.x86_6 53/198 2024-03-21T10:23:13.804 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-21T10:23:13.833 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-21T10:23:13.840 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-21T10:23:13.926 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-21T10:23:13.986 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-21T10:23:14.006 INFO:teuthology.orchestra.run.smithi031.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 84 kB/s | 19 kB 00:00 2024-03-21T10:23:14.024 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-21T10:23:14.055 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-21T10:23:14.067 INFO:teuthology.orchestra.run.smithi031.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 885 kB/s | 548 kB 00:00 2024-03-21T10:23:14.100 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-21T10:23:14.139 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-21T10:23:14.173 INFO:teuthology.orchestra.run.smithi031.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 248 kB/s | 26 kB 00:00 2024-03-21T10:23:14.183 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-21T10:23:14.206 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-21T10:23:14.214 INFO:teuthology.orchestra.run.smithi031.stdout:(110/174): gperftools-libs-2.9.1-2.el9.x86_64.r 639 kB/s | 309 kB 00:00 2024-03-21T10:23:14.215 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-21T10:23:14.242 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-21T10:23:14.257 INFO:teuthology.orchestra.run.smithi031.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 586 kB/s | 49 kB 00:00 2024-03-21T10:23:14.277 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-21T10:23:14.315 INFO:teuthology.orchestra.run.smithi031.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 673 kB/s | 67 kB 00:00 2024-03-21T10:23:14.316 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-21T10:23:14.321 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-21T10:23:14.356 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-common-2:19.0.0-2231.g2edfee68.el9. 39/198 2024-03-21T10:23:14.357 INFO:teuthology.orchestra.run.smithi031.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-03-21T10:23:14.400 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-argparse-2:19.0.0-2231.g2edfee68.el 40/198 2024-03-21T10:23:14.428 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cephfs-2:19.0.0-2231.g2edfee68.el9.x86_6 41/198 2024-03-21T10:23:14.460 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-21T10:23:14.502 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-21T10:23:14.508 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-21T10:23:14.533 INFO:teuthology.orchestra.run.smithi031.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 3.1 MB/s | 548 kB 00:00 2024-03-21T10:23:14.552 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-21T10:23:14.566 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-21T10:23:14.629 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-21T10:23:14.638 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-21T10:23:14.641 INFO:teuthology.orchestra.run.smithi031.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 403 kB/s | 43 kB 00:00 2024-03-21T10:23:14.670 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-21T10:23:14.691 INFO:teuthology.orchestra.run.smithi031.stdout:(116/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.2 MB/s | 838 kB 00:00 2024-03-21T10:23:14.707 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-21T10:23:14.753 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-21T10:23:14.765 INFO:teuthology.orchestra.run.smithi031.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 263 kB/s | 32 kB 00:00 2024-03-21T10:23:14.775 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-21T10:23:14.789 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-21T10:23:14.792 INFO:teuthology.orchestra.run.smithi031.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 150 kB/s | 15 kB 00:00 2024-03-21T10:23:14.830 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-21T10:23:14.841 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-21T10:23:14.859 INFO:teuthology.orchestra.run.smithi031.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-03-21T10:23:14.883 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-21T10:23:14.959 INFO:teuthology.orchestra.run.smithi031.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.0 MB/s | 303 kB 00:00 2024-03-21T10:23:14.993 INFO:teuthology.orchestra.run.smithi031.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 1.8 MB/s | 358 kB 00:00 2024-03-21T10:23:15.015 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-21T10:23:15.051 INFO:teuthology.orchestra.run.smithi031.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 387 kB/s | 35 kB 00:00 2024-03-21T10:23:15.080 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-21T10:23:15.111 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-21T10:23:15.127 INFO:teuthology.orchestra.run.smithi031.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 1.5 MB/s | 203 kB 00:00 2024-03-21T10:23:15.152 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-21T10:23:15.244 INFO:teuthology.orchestra.run.smithi031.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-03-21T10:23:15.245 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-21T10:23:15.277 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-21T10:23:15.317 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-21T10:23:15.336 INFO:teuthology.orchestra.run.smithi031.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 117 kB/s | 11 kB 00:00 2024-03-21T10:23:15.343 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-21T10:23:15.368 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-21T10:23:15.398 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-21T10:23:15.403 INFO:teuthology.orchestra.run.smithi031.stdout:(126/174): python3-grpcio-1.46.7-10.el9.x86_64. 5.8 MB/s | 2.0 MB 00:00 2024-03-21T10:23:15.440 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-21T10:23:15.461 INFO:teuthology.orchestra.run.smithi031.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 142 kB/s | 18 kB 00:00 2024-03-21T10:23:15.478 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-21T10:23:15.520 INFO:teuthology.orchestra.run.smithi031.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 199 kB/s | 23 kB 00:00 2024-03-21T10:23:15.563 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-21T10:23:15.571 INFO:teuthology.orchestra.run.smithi031.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 178 kB/s | 19 kB 00:00 2024-03-21T10:23:15.593 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-21T10:23:15.610 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-21T10:23:15.613 INFO:teuthology.orchestra.run.smithi031.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 213 kB/s | 20 kB 00:00 2024-03-21T10:23:15.619 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-21T10:23:15.650 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-21T10:23:15.692 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-21T10:23:15.708 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-21T10:23:15.713 INFO:teuthology.orchestra.run.smithi031.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 466 kB/s | 46 kB 00:00 2024-03-21T10:23:15.734 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-21T10:23:15.745 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librados-devel-2:19.0.0-2231.g2edfee68.el9.x86_6 53/198 2024-03-21T10:23:15.775 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-21T10:23:15.783 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-21T10:23:15.838 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-21T10:23:15.839 INFO:teuthology.orchestra.run.smithi031.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 634 kB/s | 79 kB 00:00 2024-03-21T10:23:15.956 INFO:teuthology.orchestra.run.smithi031.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 497 kB/s | 58 kB 00:00 2024-03-21T10:23:15.998 INFO:teuthology.orchestra.run.smithi031.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 2.4 MB/s | 1.0 MB 00:00 2024-03-21T10:23:16.008 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-21T10:23:16.029 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-21T10:23:16.057 INFO:teuthology.orchestra.run.smithi031.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-21T10:23:16.072 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-21T10:23:16.090 INFO:teuthology.orchestra.run.smithi031.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 180 kB/s | 16 kB 00:00 2024-03-21T10:23:16.165 INFO:teuthology.orchestra.run.smithi031.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 833 kB/s | 90 kB 00:00 2024-03-21T10:23:16.191 INFO:teuthology.orchestra.run.smithi031.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 282 kB/s | 28 kB 00:00 2024-03-21T10:23:16.191 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-21T10:23:16.199 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-21T10:23:16.237 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-21T10:23:16.258 INFO:teuthology.orchestra.run.smithi031.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 336 kB/s | 31 kB 00:00 2024-03-21T10:23:16.269 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-21T10:23:16.277 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-21T10:23:16.308 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-21T10:23:16.314 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-21T10:23:16.336 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-21T10:23:16.366 INFO:teuthology.orchestra.run.smithi031.stdout:(140/174): libarrow-9.0.0-10.el9.x86_64.rpm 1.9 MB/s | 4.4 MB 00:02 2024-03-21T10:23:16.370 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-21T10:23:16.392 INFO:teuthology.orchestra.run.smithi031.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 445 kB/s | 59 kB 00:00 2024-03-21T10:23:16.409 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-21T10:23:16.417 INFO:teuthology.orchestra.run.smithi031.stdout:(142/174): python3-routes-2.5.1-5.el9.noarch.rp 837 kB/s | 188 kB 00:00 2024-03-21T10:23:16.441 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-21T10:23:16.486 INFO:teuthology.orchestra.run.smithi031.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 303 kB/s | 36 kB 00:00 2024-03-21T10:23:16.497 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-21T10:23:16.518 INFO:teuthology.orchestra.run.smithi031.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 408 kB/s | 51 kB 00:00 2024-03-21T10:23:16.551 INFO:teuthology.orchestra.run.smithi031.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-21T10:23:16.572 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-21T10:23:16.585 INFO:teuthology.orchestra.run.smithi031.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 913 kB/s | 90 kB 00:00 2024-03-21T10:23:16.607 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-21T10:23:16.608 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-21T10:23:16.627 INFO:teuthology.orchestra.run.smithi031.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.9 MB/s | 427 kB 00:00 2024-03-21T10:23:16.640 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-21T10:23:16.652 INFO:teuthology.orchestra.run.smithi031.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 200 kB/s | 20 kB 00:00 2024-03-21T10:23:16.677 INFO:teuthology.orchestra.run.smithi031.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-03-21T10:23:16.681 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-21T10:23:16.708 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-21T10:23:16.797 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-21T10:23:16.810 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-21T10:23:16.875 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-21T10:23:16.878 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-21T10:23:16.886 INFO:teuthology.orchestra.run.smithi031.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 6.1 MB/s | 1.6 MB 00:00 2024-03-21T10:23:16.922 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-21T10:23:16.936 INFO:teuthology.orchestra.run.smithi031.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.4 MB/s | 321 kB 00:00 2024-03-21T10:23:16.949 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-21T10:23:16.953 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-21T10:23:16.985 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-21T10:23:17.000 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-21T10:23:17.022 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-21T10:23:17.070 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-21T10:23:17.098 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-21T10:23:17.105 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-21T10:23:17.147 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-21T10:23:17.148 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-21T10:23:17.191 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-21T10:23:17.253 INFO:teuthology.orchestra.run.smithi031.stdout:(152/174): NetworkManager-1.47.2-1.el9.x86_64.r 7.3 MB/s | 2.3 MB 00:00 2024-03-21T10:23:17.341 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-21T10:23:17.387 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-21T10:23:17.438 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-21T10:23:17.453 INFO:teuthology.orchestra.run.smithi031.stdout:(153/174): NetworkManager-libnm-1.47.2-1.el9.x8 9.2 MB/s | 1.8 MB 00:00 2024-03-21T10:23:17.485 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-21T10:23:17.512 INFO:teuthology.orchestra.run.smithi031.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 736 kB/s | 42 kB 00:00 2024-03-21T10:23:17.582 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-21T10:23:17.596 INFO:teuthology.orchestra.run.smithi031.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.9 MB/s | 248 kB 00:00 2024-03-21T10:23:17.617 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-21T10:23:17.657 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-21T10:23:17.663 INFO:teuthology.orchestra.run.smithi031.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.1 MB/s | 71 kB 00:00 2024-03-21T10:23:17.683 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-21T10:23:17.707 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-21T10:23:17.730 INFO:teuthology.orchestra.run.smithi031.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 552 kB/s | 36 kB 00:00 2024-03-21T10:23:17.774 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-21T10:23:17.821 INFO:teuthology.orchestra.run.smithi031.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.1 MB/s | 192 kB 00:00 2024-03-21T10:23:17.877 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-21T10:23:17.906 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-21T10:23:17.921 INFO:teuthology.orchestra.run.smithi031.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.5 MB/s | 256 kB 00:00 2024-03-21T10:23:17.927 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-21T10:23:17.961 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-21T10:23:17.972 INFO:teuthology.orchestra.run.smithi031.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 467 kB/s | 23 kB 00:00 2024-03-21T10:23:18.009 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-21T10:23:18.020 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-21T10:23:18.039 INFO:teuthology.orchestra.run.smithi031.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 108 kB 00:00 2024-03-21T10:23:18.066 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-21T10:23:18.097 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-21T10:23:18.114 INFO:teuthology.orchestra.run.smithi031.stdout:(162/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 155 kB 00:00 2024-03-21T10:23:18.124 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-21T10:23:18.137 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-21T10:23:18.180 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-21T10:23:18.181 INFO:teuthology.orchestra.run.smithi031.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 136 kB 00:00 2024-03-21T10:23:18.265 INFO:teuthology.orchestra.run.smithi031.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 776 kB/s | 64 kB 00:00 2024-03-21T10:23:18.322 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-21T10:23:18.323 INFO:teuthology.orchestra.run.smithi031.stdout:(165/174): libuuid-2.37.4-18.el9.x86_64.rpm 500 kB/s | 29 kB 00:00 2024-03-21T10:23:18.360 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-21T10:23:18.382 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-21T10:23:18.399 INFO:teuthology.orchestra.run.smithi031.stdout:(166/174): librados2-19.0.0-2231.g2edfee68.el9. 1.9 MB/s | 3.4 MB 00:01 2024-03-21T10:23:18.429 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-21T10:23:18.445 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-21T10:23:18.524 INFO:teuthology.orchestra.run.smithi031.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 6.1 MB/s | 1.2 MB 00:00 2024-03-21T10:23:18.573 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-21T10:23:18.610 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-21T10:23:18.633 INFO:teuthology.orchestra.run.smithi031.stdout:(168/174): librbd1-19.0.0-2231.g2edfee68.el9.x8 1.6 MB/s | 3.1 MB 00:01 2024-03-21T10:23:18.642 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-21T10:23:18.673 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-21T10:23:18.701 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-21T10:23:18.736 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-21T10:23:18.741 INFO:teuthology.orchestra.run.smithi031.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 6.3 MB/s | 2.2 MB 00:00 2024-03-21T10:23:18.775 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-21T10:23:18.783 INFO:teuthology.orchestra.run.smithi031.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 3.0 MB/s | 465 kB 00:00 2024-03-21T10:23:18.806 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-21T10:23:18.834 INFO:teuthology.orchestra.run.smithi031.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 342 kB/s | 31 kB 00:00 2024-03-21T10:23:18.836 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-21T10:23:18.901 INFO:teuthology.orchestra.run.smithi031.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 6.1 MB/s | 2.3 MB 00:00 2024-03-21T10:23:18.916 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-21T10:23:18.946 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-21T10:23:18.951 INFO:teuthology.orchestra.run.smithi031.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 640 kB/s | 74 kB 00:00 2024-03-21T10:23:18.979 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-21T10:23:19.001 INFO:teuthology.orchestra.run.smithi031.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 5.0 MB/s | 1.1 MB 00:00 2024-03-21T10:23:19.009 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-03-21T10:23:19.010 INFO:teuthology.orchestra.run.smithi031.stdout:Total 12 MB/s | 224 MB 00:18 2024-03-21T10:23:19.021 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-21T10:23:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-21T10:23:19.081 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-21T10:23:19.150 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-21T10:23:19.182 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-21T10:23:19.239 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-21T10:23:19.288 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-21T10:23:19.318 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-21T10:23:19.366 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-21T10:23:19.469 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-21T10:23:19.529 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-21T10:23:20.309 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-21T10:23:20.350 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-21T10:23:20.384 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-21T10:23:20.405 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-21T10:23:20.433 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-21T10:23:20.486 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-21T10:23:20.511 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-03-21T10:23:20.517 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-21T10:23:20.525 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-21T10:23:20.548 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-21T10:23:20.568 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-21T10:23:20.609 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-21T10:23:20.763 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-21T10:23:20.790 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-21T10:23:20.871 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-21T10:23:20.906 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-03-21T10:23:20.907 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-03-21T10:23:21.035 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-21T10:23:21.064 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : librbd1-2:19.0.0-2231.g2edfee68.el9.x86_64 124/198 2024-03-21T10:23:21.163 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librbd1-2:19.0.0-2231.g2edfee68.el9.x86_64 124/198 2024-03-21T10:23:21.194 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rbd-2:19.0.0-2231.g2edfee68.el9.x86_64 125/198 2024-03-21T10:23:21.230 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-21T10:23:21.553 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-21T10:23:21.667 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-21T10:23:21.690 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-21T10:23:21.717 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librgw2-2:19.0.0-2231.g2edfee68.el9.x86_64 128/198 2024-03-21T10:23:21.779 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librgw2-2:19.0.0-2231.g2edfee68.el9.x86_64 128/198 2024-03-21T10:23:21.802 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rgw-2:19.0.0-2231.g2edfee68.el9.x86_64 129/198 2024-03-21T10:23:22.967 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-21T10:23:23.058 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-21T10:23:23.089 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-21T10:23:23.120 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-21T10:23:23.147 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-21T10:23:23.415 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-03-21T10:23:23.415 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-03-21T10:23:23.477 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-21T10:23:23.502 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librbd1-2:19.0.0-2231.g2edfee68.el9.x86_64 124/198 2024-03-21T10:23:23.577 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:19.0.0-2231.g2edfee68.el9.x86_64 124/198 2024-03-21T10:23:23.607 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rbd-2:19.0.0-2231.g2edfee68.el9.x86_64 125/198 2024-03-21T10:23:23.644 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-21T10:23:24.062 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-common-2:19.0.0-2231.g2edfee68.el9.x86_64 130/198 2024-03-21T10:23:24.099 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-common-2:19.0.0-2231.g2edfee68.el9.x86_64 130/198 2024-03-21T10:23:24.132 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-21T10:23:24.164 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librgw2-2:19.0.0-2231.g2edfee68.el9.x86_64 128/198 2024-03-21T10:23:24.235 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librgw2-2:19.0.0-2231.g2edfee68.el9.x86_64 128/198 2024-03-21T10:23:24.265 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rgw-2:19.0.0-2231.g2edfee68.el9.x86_64 129/198 2024-03-21T10:23:24.633 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-common-2:19.0.0-2231.g2edfee68.el9.x86_64 130/198 2024-03-21T10:23:24.669 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-base-2:19.0.0-2231.g2edfee68.el9.x86_64 131/198 2024-03-21T10:23:24.741 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-base-2:19.0.0-2231.g2edfee68.el9.x86_64 131/198 2024-03-21T10:23:24.742 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-21T10:23:24.742 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-21T10:23:24.742 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-21T10:23:24.770 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-selinux-2:19.0.0-2231.g2edfee68.el9.x86_64 132/198 2024-03-21T10:23:26.633 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:19.0.0-2231.g2edfee68.el9.x86_64 130/198 2024-03-21T10:23:26.670 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-common-2:19.0.0-2231.g2edfee68.el9.x86_64 130/198 2024-03-21T10:23:27.239 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:19.0.0-2231.g2edfee68.el9.x86_64 130/198 2024-03-21T10:23:27.274 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-base-2:19.0.0-2231.g2edfee68.el9.x86_64 131/198 2024-03-21T10:23:27.340 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-base-2:19.0.0-2231.g2edfee68.el9.x86_64 131/198 2024-03-21T10:23:27.340 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-21T10:23:27.340 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-21T10:23:27.340 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-21T10:23:27.375 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-selinux-2:19.0.0-2231.g2edfee68.el9.x86_64 132/198 2024-03-21T10:23:31.466 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-03-21T10:23:31.649 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-21T10:23:31.694 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-21T10:23:31.724 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-21T10:23:31.768 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-21T10:23:31.859 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-21T10:23:31.961 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-21T10:23:32.025 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-21T10:23:32.056 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-21T10:23:32.154 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-21T10:23:32.570 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-21T10:23:32.604 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librados2-2:19.0.0-2231.g2edfee68.el9.x86_64 10/198 2024-03-21T10:23:32.768 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:19.0.0-2231.g2edfee68.el9.x86_64 10/198 2024-03-21T10:23:32.796 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs2-2:19.0.0-2231.g2edfee68.el9.x86_64 11/198 2024-03-21T10:23:32.917 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs2-2:19.0.0-2231.g2edfee68.el9.x86_64 11/198 2024-03-21T10:23:32.961 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rados-2:19.0.0-2231.g2edfee68.el9.x86_64 12/198 2024-03-21T10:23:33.047 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-21T10:23:33.330 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-21T10:23:33.392 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-21T10:23:33.418 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephsqlite-2:19.0.0-2231.g2edfee68.el9.x86_64 16/198 2024-03-21T10:23:33.570 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2231.g2edfee68.el9.x86_64 16/198 2024-03-21T10:23:33.602 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libradosstriper1-2:19.0.0-2231.g2edfee68.el9.x86 17/198 2024-03-21T10:23:33.718 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2231.g2edfee68.el9.x86 17/198 2024-03-21T10:23:33.771 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-21T10:23:33.804 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-21T10:23:33.867 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-21T10:23:34.197 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-21T10:23:34.245 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-21T10:23:34.287 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-21T10:23:34.621 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-21T10:23:34.792 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-21T10:23:34.883 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-21T10:23:35.074 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-21T10:23:35.161 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-21T10:23:35.543 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-21T10:23:35.731 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-21T10:23:35.866 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-21T10:23:35.911 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-21T10:23:35.947 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-21T10:23:36.008 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-21T10:23:36.122 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-21T10:23:36.198 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-21T10:23:36.254 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-21T10:23:36.359 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-21T10:23:36.400 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-21T10:23:36.426 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2231.g2edfee68.el9.x86_64 132/198 2024-03-21T10:23:36.426 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /sys 2024-03-21T10:23:36.426 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /proc 2024-03-21T10:23:36.426 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /mnt 2024-03-21T10:23:36.426 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /var/tmp 2024-03-21T10:23:36.427 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /home 2024-03-21T10:23:36.427 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /root 2024-03-21T10:23:36.427 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /tmp 2024-03-21T10:23:36.427 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-21T10:23:36.463 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-21T10:23:36.513 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-21T10:23:36.569 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-21T10:23:36.582 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-volume-2:19.0.0-2231.g2edfee68.el9.noarch 133/198 2024-03-21T10:23:36.597 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-volume-2:19.0.0-2231.g2edfee68.el9.noarch 133/198 2024-03-21T10:23:36.597 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:23:36.598 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-21T10:23:36.598 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-21T10:23:36.694 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-21T10:23:36.749 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-common-2:19.0.0-2231.g2edfee68.el9. 39/198 2024-03-21T10:23:36.798 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-argparse-2:19.0.0-2231.g2edfee68.el 40/198 2024-03-21T10:23:36.830 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cephfs-2:19.0.0-2231.g2edfee68.el9.x86_6 41/198 2024-03-21T10:23:36.991 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-21T10:23:37.061 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-21T10:23:37.176 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-21T10:23:37.383 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-21T10:23:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-21T10:23:38.077 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-osd-2:19.0.0-2231.g2edfee68.el9.x86_64 134/198 2024-03-21T10:23:38.102 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-osd-2:19.0.0-2231.g2edfee68.el9.x86_64 134/198 2024-03-21T10:23:38.102 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:23:38.102 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-21T10:23:38.102 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-21T10:23:38.102 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-21T10:23:38.102 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-21T10:23:38.170 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-21T10:23:38.271 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-21T10:23:38.328 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mds-2:19.0.0-2231.g2edfee68.el9.x86_64 135/198 2024-03-21T10:23:38.350 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mds-2:19.0.0-2231.g2edfee68.el9.x86_64 135/198 2024-03-21T10:23:38.351 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:23:38.351 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-21T10:23:38.351 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-21T10:23:38.351 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-21T10:23:38.351 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-21T10:23:38.393 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-21T10:23:38.457 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-21T10:23:38.490 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-21T10:23:38.544 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-21T10:23:38.605 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-21T10:23:38.668 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librados-devel-2:19.0.0-2231.g2edfee68.el9.x86_6 53/198 2024-03-21T10:23:38.715 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-21T10:23:38.880 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mon-2:19.0.0-2231.g2edfee68.el9.x86_64 136/198 2024-03-21T10:23:38.902 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mon-2:19.0.0-2231.g2edfee68.el9.x86_64 136/198 2024-03-21T10:23:38.902 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:23:38.902 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-21T10:23:38.902 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-21T10:23:38.903 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-21T10:23:38.905 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-21T10:23:38.998 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-21T10:23:39.134 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-21T10:23:39.346 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-21T10:23:39.375 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: cephadm-2:19.0.0-2231.g2edfee68.el9.noarch 138/198 2024-03-21T10:23:39.405 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephadm-2:19.0.0-2231.g2edfee68.el9.noarch 138/198 2024-03-21T10:23:39.437 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2231.g2edfee68.e 139/198 2024-03-21T10:23:39.453 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-21T10:23:39.498 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2231.g2edfee68. 140/198 2024-03-21T10:23:39.520 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2231.g2edfee68.el9.noa 141/198 2024-03-21T10:23:39.536 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-21T10:23:39.674 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2231.g2edfee68.el9.x86_64 132/198 2024-03-21T10:23:39.674 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /sys 2024-03-21T10:23:39.674 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /proc 2024-03-21T10:23:39.674 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /mnt 2024-03-21T10:23:39.675 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /var/tmp 2024-03-21T10:23:39.675 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /home 2024-03-21T10:23:39.675 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /root 2024-03-21T10:23:39.675 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /tmp 2024-03-21T10:23:39.675 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-21T10:23:39.753 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2231.g2edfee68.el9.noa 141/198 2024-03-21T10:23:39.760 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-21T10:23:39.788 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2231.g2edfee68.el9.n 142/198 2024-03-21T10:23:39.815 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-21T10:23:39.846 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-volume-2:19.0.0-2231.g2edfee68.el9.noarch 133/198 2024-03-21T10:23:39.862 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-volume-2:19.0.0-2231.g2edfee68.el9.noarch 133/198 2024-03-21T10:23:39.862 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:23:39.862 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-21T10:23:39.862 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-21T10:23:39.899 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-21T10:23:40.001 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-21T10:23:40.049 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-21T10:23:40.089 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-21T10:23:40.162 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-21T10:23:40.214 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-21T10:23:40.279 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-21T10:23:40.344 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-21T10:23:40.557 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-21T10:23:40.618 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-21T10:23:40.655 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-21T10:23:40.729 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-21T10:23:40.743 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2231.g2edfee68.el9.n 142/198 2024-03-21T10:23:40.771 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2231.g2ed 143/198 2024-03-21T10:23:40.867 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-21T10:23:40.870 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2231.g2ed 143/198 2024-03-21T10:23:40.904 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-21T10:23:40.950 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-21T10:23:40.986 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-21T10:23:41.033 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2231.g2edfee68.el 144/198 2024-03-21T10:23:41.037 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-21T10:23:41.057 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-2:19.0.0-2231.g2edfee68.el9.x86_64 145/198 2024-03-21T10:23:41.079 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2231.g2edfee68.el9.x86_64 145/198 2024-03-21T10:23:41.080 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:23:41.080 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-21T10:23:41.080 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-21T10:23:41.080 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-21T10:23:41.080 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-21T10:23:41.124 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-rook-2:19.0.0-2231.g2edfee68.el9.noarch 146/198 2024-03-21T10:23:41.135 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2231.g2edfee68.el9.noarch 146/198 2024-03-21T10:23:41.180 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-2:19.0.0-2231.g2edfee68.el9.x86_64 147/198 2024-03-21T10:23:41.213 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-21T10:23:41.393 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-21T10:23:41.399 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-osd-2:19.0.0-2231.g2edfee68.el9.x86_64 134/198 2024-03-21T10:23:41.426 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-osd-2:19.0.0-2231.g2edfee68.el9.x86_64 134/198 2024-03-21T10:23:41.427 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:23:41.427 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-21T10:23:41.427 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-21T10:23:41.427 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-21T10:23:41.427 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-21T10:23:41.540 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-21T10:23:41.597 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-21T10:23:41.667 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mds-2:19.0.0-2231.g2edfee68.el9.x86_64 135/198 2024-03-21T10:23:41.691 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mds-2:19.0.0-2231.g2edfee68.el9.x86_64 135/198 2024-03-21T10:23:41.691 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:23:41.691 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-21T10:23:41.691 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-21T10:23:41.692 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-21T10:23:41.692 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-21T10:23:42.037 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-21T10:23:42.050 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-21T10:23:42.085 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-radosgw-2:19.0.0-2231.g2edfee68.el9.x86_64 149/198 2024-03-21T10:23:42.106 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2231.g2edfee68.el9.x86_64 149/198 2024-03-21T10:23:42.106 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:23:42.107 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-21T10:23:42.107 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-21T10:23:42.107 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-21T10:23:42.107 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-21T10:23:42.131 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-21T10:23:42.144 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2231.g2edfe 150/198 2024-03-21T10:23:42.159 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mon-2:19.0.0-2231.g2edfee68.el9.x86_64 136/198 2024-03-21T10:23:42.166 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2231.g2edfe 150/198 2024-03-21T10:23:42.166 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:23:42.166 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-21T10:23:42.166 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-21T10:23:42.184 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mon-2:19.0.0-2231.g2edfee68.el9.x86_64 136/198 2024-03-21T10:23:42.184 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:23:42.184 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-21T10:23:42.184 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-21T10:23:42.184 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-21T10:23:42.184 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-21T10:23:42.211 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephfs-mirror-2:19.0.0-2231.g2edfee68.el9.x86_64 151/198 2024-03-21T10:23:42.233 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2231.g2edfee68.el9.x86_64 151/198 2024-03-21T10:23:42.233 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:23:42.233 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-21T10:23:42.233 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-21T10:23:42.233 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-21T10:23:42.233 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-21T10:23:42.278 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-21T10:23:42.396 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-21T10:23:42.459 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-21T10:23:42.545 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-mirror-2:19.0.0-2231.g2edfee68.el9.x86_64 152/198 2024-03-21T10:23:42.567 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2231.g2edfee68.el9.x86_64 152/198 2024-03-21T10:23:42.567 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:23:42.567 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-21T10:23:42.567 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-21T10:23:42.568 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-21T10:23:42.568 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-21T10:23:42.659 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-21T10:23:42.718 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-21T10:23:42.747 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephadm-2:19.0.0-2231.g2edfee68.el9.noarch 138/198 2024-03-21T10:23:42.763 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-21T10:23:42.777 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephadm-2:19.0.0-2231.g2edfee68.el9.noarch 138/198 2024-03-21T10:23:42.799 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-21T10:23:42.809 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2231.g2edfee68.e 139/198 2024-03-21T10:23:42.867 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-21T10:23:42.871 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2231.g2edfee68. 140/198 2024-03-21T10:23:42.900 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2231.g2edfee68.el9.noa 141/198 2024-03-21T10:23:42.916 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-21T10:23:42.985 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-21T10:23:43.042 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-21T10:23:43.089 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-21T10:23:43.149 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2231.g2edfee68.el9.noa 141/198 2024-03-21T10:23:43.176 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2231.g2edfee68.el9.n 142/198 2024-03-21T10:23:43.234 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-21T10:23:43.275 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-21T10:23:43.318 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-21T10:23:43.385 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-21T10:23:43.430 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-21T10:23:43.583 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-21T10:23:43.718 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-21T10:23:43.792 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-21T10:23:43.828 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-21T10:23:43.917 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-21T10:23:44.070 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-21T10:23:44.134 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-21T10:23:44.200 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2231.g2edfee68.el9.n 142/198 2024-03-21T10:23:44.226 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2231.g2ed 143/198 2024-03-21T10:23:44.334 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2231.g2ed 143/198 2024-03-21T10:23:44.505 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2231.g2edfee68.el 144/198 2024-03-21T10:23:44.663 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-2:19.0.0-2231.g2edfee68.el9.x86_64 145/198 2024-03-21T10:23:44.688 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2231.g2edfee68.el9.x86_64 145/198 2024-03-21T10:23:44.688 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:23:44.688 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-21T10:23:44.688 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-21T10:23:44.688 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-21T10:23:44.688 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-21T10:23:44.737 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-rook-2:19.0.0-2231.g2edfee68.el9.noarch 146/198 2024-03-21T10:23:44.749 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2231.g2edfee68.el9.noarch 146/198 2024-03-21T10:23:44.795 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-2:19.0.0-2231.g2edfee68.el9.x86_64 147/198 2024-03-21T10:23:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-21T10:23:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-21T10:23:45.276 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-21T10:23:45.340 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-21T10:23:45.463 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-21T10:23:45.501 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-21T10:23:45.718 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-21T10:23:45.756 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-radosgw-2:19.0.0-2231.g2edfee68.el9.x86_64 149/198 2024-03-21T10:23:45.760 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-21T10:23:45.780 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2231.g2edfee68.el9.x86_64 149/198 2024-03-21T10:23:45.781 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:23:45.781 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-21T10:23:45.781 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-21T10:23:45.781 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-21T10:23:45.781 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-21T10:23:45.798 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-21T10:23:45.823 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2231.g2edfe 150/198 2024-03-21T10:23:45.847 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2231.g2edfe 150/198 2024-03-21T10:23:45.847 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:23:45.847 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-21T10:23:45.847 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-21T10:23:45.898 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-21T10:23:45.899 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephfs-mirror-2:19.0.0-2231.g2edfee68.el9.x86_64 151/198 2024-03-21T10:23:45.923 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2231.g2edfee68.el9.x86_64 151/198 2024-03-21T10:23:45.924 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:23:45.924 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-21T10:23:45.924 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-21T10:23:45.924 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-21T10:23:45.924 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-21T10:23:46.250 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-mirror-2:19.0.0-2231.g2edfee68.el9.x86_64 152/198 2024-03-21T10:23:46.275 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2231.g2edfee68.el9.x86_64 152/198 2024-03-21T10:23:46.275 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:23:46.275 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-21T10:23:46.275 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-21T10:23:46.276 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-21T10:23:46.276 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-21T10:23:46.785 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-21T10:23:46.922 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-21T10:23:47.008 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-test-2:19.0.0-2231.g2edfee68.el9.x86_64 153/198 2024-03-21T10:23:47.049 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-fuse-2:19.0.0-2231.g2edfee68.el9.x86_64 154/198 2024-03-21T10:23:47.088 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-nbd-2:19.0.0-2231.g2edfee68.el9.x86_64 155/198 2024-03-21T10:23:47.115 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-21T10:23:47.298 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephfs-shell-2:19.0.0-2231.g2edfee68.el9.x86_64 157/198 2024-03-21T10:23:47.425 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-21T10:23:47.465 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-fuse-2:19.0.0-2231.g2edfee68.el9.x86_64 159/198 2024-03-21T10:23:47.505 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-21T10:23:47.557 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-21T10:23:47.600 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-21T10:23:47.633 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephfs-devel-2:19.0.0-2231.g2edfee68.el9.x86_ 163/198 2024-03-21T10:23:47.673 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-21T10:23:47.703 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-21T10:23:47.757 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-21T10:23:47.819 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-21T10:23:47.852 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-21T10:23:48.373 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-21T10:23:48.417 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-21T10:23:48.436 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-21T10:23:48.569 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-21T10:23:48.659 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-21T10:23:48.726 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-21T10:23:48.762 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-21T10:23:48.803 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-21T10:23:49.204 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-21T10:23:49.241 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librbd1-2:19.0.0-2231.g2edfee68.el9.x86_64 124/198 2024-03-21T10:23:49.254 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephfs-top-2:19.0.0-2231.g2edfee68.el9.noarch 170/198 2024-03-21T10:23:49.321 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-21T10:23:49.361 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-21T10:23:49.362 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:19.0.0-2231.g2edfee68.el9.x86_64 124/198 2024-03-21T10:23:49.420 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rbd-2:19.0.0-2231.g2edfee68.el9.x86_64 125/198 2024-03-21T10:23:49.443 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-21T10:23:49.479 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-21T10:23:49.482 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-21T10:23:49.507 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-21T10:23:49.533 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-21T10:23:49.560 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-21T10:23:49.560 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-21T10:23:49.630 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-21T10:23:49.660 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-21T10:23:49.660 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-21T10:23:49.745 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-21T10:23:49.771 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-21T10:23:49.813 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-21T10:23:49.838 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-21T10:23:49.863 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-21T10:23:49.896 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-21T10:23:49.936 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-21T10:23:49.968 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-21T10:23:49.993 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-21T10:23:50.019 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-21T10:23:50.044 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-21T10:23:50.050 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-21T10:23:50.069 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-21T10:23:50.075 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-21T10:23:50.076 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-21T10:23:50.110 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librgw2-2:19.0.0-2231.g2edfee68.el9.x86_64 128/198 2024-03-21T10:23:50.212 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librgw2-2:19.0.0-2231.g2edfee68.el9.x86_64 128/198 2024-03-21T10:23:50.246 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rgw-2:19.0.0-2231.g2edfee68.el9.x86_64 129/198 2024-03-21T10:23:50.264 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-21T10:23:50.264 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-21T10:23:50.347 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-21T10:23:50.372 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-21T10:23:50.404 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-21T10:23:50.431 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-21T10:23:50.456 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-21T10:23:50.520 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-21T10:23:50.996 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-test-2:19.0.0-2231.g2edfee68.el9.x86_64 153/198 2024-03-21T10:23:51.038 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-fuse-2:19.0.0-2231.g2edfee68.el9.x86_64 154/198 2024-03-21T10:23:51.068 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-nbd-2:19.0.0-2231.g2edfee68.el9.x86_64 155/198 2024-03-21T10:23:51.096 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-21T10:23:51.287 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephfs-shell-2:19.0.0-2231.g2edfee68.el9.x86_64 157/198 2024-03-21T10:23:51.409 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-21T10:23:51.454 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-fuse-2:19.0.0-2231.g2edfee68.el9.x86_64 159/198 2024-03-21T10:23:51.493 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-21T10:23:51.545 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-21T10:23:51.580 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-21T10:23:51.612 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs-devel-2:19.0.0-2231.g2edfee68.el9.x86_ 163/198 2024-03-21T10:23:51.653 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-21T10:23:51.683 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-21T10:23:51.737 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-21T10:23:51.807 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-21T10:23:51.847 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-21T10:23:52.401 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-21T10:23:52.437 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-21T10:23:52.458 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-21T10:23:52.947 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:19.0.0-2231.g2edfee68.el9.x86_64 130/198 2024-03-21T10:23:52.985 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-common-2:19.0.0-2231.g2edfee68.el9.x86_64 130/198 2024-03-21T10:23:53.360 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephfs-top-2:19.0.0-2231.g2edfee68.el9.noarch 170/198 2024-03-21T10:23:53.441 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-21T10:23:53.483 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-21T10:23:53.566 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-21T10:23:53.603 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-21T10:23:53.628 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-21T10:23:53.647 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:19.0.0-2231.g2edfee68.el9.x86_64 130/198 2024-03-21T10:23:53.653 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-21T10:23:53.680 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-21T10:23:53.680 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-21T10:23:53.697 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-base-2:19.0.0-2231.g2edfee68.el9.x86_64 131/198 2024-03-21T10:23:53.776 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-21T10:23:53.814 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-21T10:23:53.814 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-21T10:23:53.850 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-base-2:19.0.0-2231.g2edfee68.el9.x86_64 131/198 2024-03-21T10:23:53.850 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-21T10:23:53.850 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-21T10:23:53.850 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:23:53.899 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-21T10:23:53.922 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-selinux-2:19.0.0-2231.g2edfee68.el9.x86_64 132/198 2024-03-21T10:23:53.925 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-21T10:23:53.958 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-21T10:23:53.984 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-21T10:23:54.009 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-21T10:23:54.042 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-21T10:23:54.081 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-21T10:23:54.113 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-21T10:23:54.138 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-21T10:23:54.164 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-21T10:23:54.197 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-21T10:23:54.223 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-21T10:23:54.230 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-21T10:23:54.230 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-21T10:23:54.418 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-21T10:23:54.418 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-21T10:23:54.500 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-21T10:23:54.525 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-21T10:23:54.557 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-21T10:23:54.584 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-21T10:23:54.609 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-21T10:23:54.676 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-21T10:23:55.346 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-21T10:23:55.346 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-2:19.0.0-2231.g2edfee68.el9.x86_64 1/198 2024-03-21T10:23:55.346 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-base-2:19.0.0-2231.g2edfee68.el9.x86_64 2/198 2024-03-21T10:23:55.346 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-common-2:19.0.0-2231.g2edfee68.el9.x86_64 3/198 2024-03-21T10:23:55.346 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-fuse-2:19.0.0-2231.g2edfee68.el9.x86_64 4/198 2024-03-21T10:23:55.346 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2231.g2edfe 5/198 2024-03-21T10:23:55.347 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mds-2:19.0.0-2231.g2edfee68.el9.x86_64 6/198 2024-03-21T10:23:55.347 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-2:19.0.0-2231.g2edfee68.el9.x86_64 7/198 2024-03-21T10:23:55.347 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mon-2:19.0.0-2231.g2edfee68.el9.x86_64 8/198 2024-03-21T10:23:55.347 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-osd-2:19.0.0-2231.g2edfee68.el9.x86_64 9/198 2024-03-21T10:23:55.347 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-radosgw-2:19.0.0-2231.g2edfee68.el9.x86_64 10/198 2024-03-21T10:23:55.347 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-selinux-2:19.0.0-2231.g2edfee68.el9.x86_64 11/198 2024-03-21T10:23:55.347 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-test-2:19.0.0-2231.g2edfee68.el9.x86_64 12/198 2024-03-21T10:23:55.347 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephfs-mirror-2:19.0.0-2231.g2edfee68.el9.x86_64 13/198 2024-03-21T10:23:55.347 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephfs-shell-2:19.0.0-2231.g2edfee68.el9.x86_64 14/198 2024-03-21T10:23:55.347 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephfs-devel-2:19.0.0-2231.g2edfee68.el9.x86_ 15/198 2024-03-21T10:23:55.347 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephfs2-2:19.0.0-2231.g2edfee68.el9.x86_64 16/198 2024-03-21T10:23:55.348 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephsqlite-2:19.0.0-2231.g2edfee68.el9.x86_64 17/198 2024-03-21T10:23:55.348 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados-devel-2:19.0.0-2231.g2edfee68.el9.x86_6 18/198 2024-03-21T10:23:55.348 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libradosstriper1-2:19.0.0-2231.g2edfee68.el9.x86 19/198 2024-03-21T10:23:55.348 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librgw2-2:19.0.0-2231.g2edfee68.el9.x86_64 20/198 2024-03-21T10:23:55.348 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2231.g2edfee68.el 21/198 2024-03-21T10:23:55.348 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ceph-common-2:19.0.0-2231.g2edfee68.el9. 22/198 2024-03-21T10:23:55.348 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cephfs-2:19.0.0-2231.g2edfee68.el9.x86_6 23/198 2024-03-21T10:23:55.348 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rados-2:19.0.0-2231.g2edfee68.el9.x86_64 24/198 2024-03-21T10:23:55.348 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rbd-2:19.0.0-2231.g2edfee68.el9.x86_64 25/198 2024-03-21T10:23:55.348 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rgw-2:19.0.0-2231.g2edfee68.el9.x86_64 26/198 2024-03-21T10:23:55.348 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-fuse-2:19.0.0-2231.g2edfee68.el9.x86_64 27/198 2024-03-21T10:23:55.364 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-mirror-2:19.0.0-2231.g2edfee68.el9.x86_64 28/198 2024-03-21T10:23:55.365 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-nbd-2:19.0.0-2231.g2edfee68.el9.x86_64 29/198 2024-03-21T10:23:55.365 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2231.g2edfee68. 30/198 2024-03-21T10:23:55.365 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2231.g2edfee68.el9.noa 31/198 2024-03-21T10:23:55.365 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2231.g2edfee68.el9.n 32/198 2024-03-21T10:23:55.365 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2231.g2ed 33/198 2024-03-21T10:23:55.365 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2231.g2edfee68.el 34/198 2024-03-21T10:23:55.365 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2231.g2edfee68.el9.noarch 35/198 2024-03-21T10:23:55.365 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2231.g2edfee68.e 36/198 2024-03-21T10:23:55.365 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-volume-2:19.0.0-2231.g2edfee68.el9.noarch 37/198 2024-03-21T10:23:55.365 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephadm-2:19.0.0-2231.g2edfee68.el9.noarch 38/198 2024-03-21T10:23:55.365 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephfs-top-2:19.0.0-2231.g2edfee68.el9.noarch 39/198 2024-03-21T10:23:55.365 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-21T10:23:55.365 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-21T10:23:55.366 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-21T10:23:55.366 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-21T10:23:55.366 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-21T10:23:55.366 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-21T10:23:55.366 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-21T10:23:55.366 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-21T10:23:55.366 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-21T10:23:55.366 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-21T10:23:55.366 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-21T10:23:55.366 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-21T10:23:55.366 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-21T10:23:55.366 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-21T10:23:55.366 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-21T10:23:55.367 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-21T10:23:55.367 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-21T10:23:55.367 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-21T10:23:55.367 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-21T10:23:55.367 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-21T10:23:55.367 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-21T10:23:55.367 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-21T10:23:55.367 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-21T10:23:55.367 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-21T10:23:55.367 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-21T10:23:55.367 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-21T10:23:55.367 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-21T10:23:55.367 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-21T10:23:55.368 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-21T10:23:55.368 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-21T10:23:55.368 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-21T10:23:55.368 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-21T10:23:55.368 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-21T10:23:55.368 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-21T10:23:55.368 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-21T10:23:55.368 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-21T10:23:55.368 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-21T10:23:55.368 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-21T10:23:55.368 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-21T10:23:55.368 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-21T10:23:55.368 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-21T10:23:55.368 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-21T10:23:55.369 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-21T10:23:55.369 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-21T10:23:55.369 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-21T10:23:55.369 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-21T10:23:55.369 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-21T10:23:55.369 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-21T10:23:55.369 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-21T10:23:55.369 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-21T10:23:55.369 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-21T10:23:55.369 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-21T10:23:55.369 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-21T10:23:55.369 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-21T10:23:55.369 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-21T10:23:55.370 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-21T10:23:55.370 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-21T10:23:55.370 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-21T10:23:55.370 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-21T10:23:55.370 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-21T10:23:55.370 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-21T10:23:55.370 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-21T10:23:55.370 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-21T10:23:55.370 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-21T10:23:55.370 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-21T10:23:55.370 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-21T10:23:55.370 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-21T10:23:55.370 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-21T10:23:55.370 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-21T10:23:55.371 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-21T10:23:55.371 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-21T10:23:55.371 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-21T10:23:55.371 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-21T10:23:55.371 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-21T10:23:55.371 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-21T10:23:55.371 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-21T10:23:55.371 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-21T10:23:55.371 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-21T10:23:55.371 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-21T10:23:55.371 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-21T10:23:55.371 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-21T10:23:55.371 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-21T10:23:55.371 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-21T10:23:55.372 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-21T10:23:55.372 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-21T10:23:55.372 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-21T10:23:55.372 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-21T10:23:55.372 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-21T10:23:55.372 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-21T10:23:55.372 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-21T10:23:55.372 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-21T10:23:55.372 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-21T10:23:55.372 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-21T10:23:55.372 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-21T10:23:55.372 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-21T10:23:55.372 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-21T10:23:55.372 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-21T10:23:55.373 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-21T10:23:55.373 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-21T10:23:55.373 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-21T10:23:55.373 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-21T10:23:55.373 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-21T10:23:55.373 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-21T10:23:55.373 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-21T10:23:55.373 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-21T10:23:55.373 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-21T10:23:55.373 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-21T10:23:55.373 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-21T10:23:55.373 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-21T10:23:55.373 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-21T10:23:55.373 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-21T10:23:55.374 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados2-2:19.0.0-2231.g2edfee68.el9.x86_64 151/198 2024-03-21T10:23:55.374 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-21T10:23:55.374 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librbd1-2:19.0.0-2231.g2edfee68.el9.x86_64 153/198 2024-03-21T10:23:55.374 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-21T10:23:55.374 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-21T10:23:55.374 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-21T10:23:55.374 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-21T10:23:55.374 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-21T10:23:55.374 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-21T10:23:55.374 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-21T10:23:55.374 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-21T10:23:55.374 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-21T10:23:55.374 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-21T10:23:55.374 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-21T10:23:55.375 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-21T10:23:55.375 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-21T10:23:55.375 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-21T10:23:55.375 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-21T10:23:55.375 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-21T10:23:55.375 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-21T10:23:55.375 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-21T10:23:55.375 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-21T10:23:55.375 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-21T10:23:55.375 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-21T10:23:55.375 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-21T10:23:55.375 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-21T10:23:55.375 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-21T10:23:55.376 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-21T10:23:55.376 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-21T10:23:55.376 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-21T10:23:55.376 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-21T10:23:55.376 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-21T10:23:55.376 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-21T10:23:55.376 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-21T10:23:55.376 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-21T10:23:55.377 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-21T10:23:55.377 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-21T10:23:55.377 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-21T10:23:55.377 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-21T10:23:55.377 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-21T10:23:55.377 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-21T10:23:55.377 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-21T10:23:55.378 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-21T10:23:55.378 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-21T10:23:55.378 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-21T10:23:55.378 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-21T10:23:55.378 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-21T10:23:58.117 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-21T10:23:58.118 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-21T10:23:58.118 INFO:teuthology.orchestra.run.smithi070.stdout:Upgraded: 2024-03-21T10:23:58.118 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-21T10:23:58.118 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-21T10:23:58.118 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-21T10:23:58.118 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-21T10:23:58.118 INFO:teuthology.orchestra.run.smithi070.stdout: acl-2.3.1-4.el9.x86_64 2024-03-21T10:23:58.118 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-21T10:23:58.118 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-21T10:23:58.118 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-21T10:23:58.118 INFO:teuthology.orchestra.run.smithi070.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-21T10:23:58.118 INFO:teuthology.orchestra.run.smithi070.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-21T10:23:58.119 INFO:teuthology.orchestra.run.smithi070.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-21T10:23:58.119 INFO:teuthology.orchestra.run.smithi070.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-21T10:23:58.119 INFO:teuthology.orchestra.run.smithi070.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-21T10:23:58.119 INFO:teuthology.orchestra.run.smithi070.stdout: librados2-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.119 INFO:teuthology.orchestra.run.smithi070.stdout: librbd1-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.119 INFO:teuthology.orchestra.run.smithi070.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-21T10:23:58.119 INFO:teuthology.orchestra.run.smithi070.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-21T10:23:58.119 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-21T10:23:58.119 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-21T10:23:58.119 INFO:teuthology.orchestra.run.smithi070.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-21T10:23:58.119 INFO:teuthology.orchestra.run.smithi070.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-21T10:23:58.119 INFO:teuthology.orchestra.run.smithi070.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-21T10:23:58.119 INFO:teuthology.orchestra.run.smithi070.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-21T10:23:58.119 INFO:teuthology.orchestra.run.smithi070.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-21T10:23:58.120 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-03-21T10:23:58.120 INFO:teuthology.orchestra.run.smithi070.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-21T10:23:58.120 INFO:teuthology.orchestra.run.smithi070.stdout: automake-1.16.2-8.el9.noarch 2024-03-21T10:23:58.120 INFO:teuthology.orchestra.run.smithi070.stdout: bison-3.7.4-5.el9.x86_64 2024-03-21T10:23:58.120 INFO:teuthology.orchestra.run.smithi070.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-21T10:23:58.120 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.120 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.120 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-common-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.120 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-fuse-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.120 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-grafana-dashboards-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:23:58.120 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-immutable-object-cache-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.120 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mds-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.120 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.120 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-cephadm-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:23:58.121 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:23:58.121 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:23:58.121 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:23:58.121 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-rook-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:23:58.121 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mon-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.121 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-osd-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.122 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-prometheus-alerts-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:23:58.122 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-radosgw-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.122 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-selinux-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.122 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-test-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.122 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-volume-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:23:58.122 INFO:teuthology.orchestra.run.smithi070.stdout: cephadm-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:23:58.122 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-mirror-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.123 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-shell-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.123 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-top-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:23:58.123 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-21T10:23:58.123 INFO:teuthology.orchestra.run.smithi070.stdout: flex-2.6.4-9.el9.x86_64 2024-03-21T10:23:58.123 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-21T10:23:58.123 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-21T10:23:58.123 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-21T10:23:58.123 INFO:teuthology.orchestra.run.smithi070.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-21T10:23:58.123 INFO:teuthology.orchestra.run.smithi070.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-21T10:23:58.123 INFO:teuthology.orchestra.run.smithi070.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-21T10:23:58.123 INFO:teuthology.orchestra.run.smithi070.stdout: jq-1.6-16.el9.x86_64 2024-03-21T10:23:58.123 INFO:teuthology.orchestra.run.smithi070.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-21T10:23:58.123 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-21T10:23:58.124 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-21T10:23:58.124 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-devel-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.124 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs2-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.124 INFO:teuthology.orchestra.run.smithi070.stdout: libcephsqlite-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.124 INFO:teuthology.orchestra.run.smithi070.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-21T10:23:58.124 INFO:teuthology.orchestra.run.smithi070.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-21T10:23:58.124 INFO:teuthology.orchestra.run.smithi070.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-21T10:23:58.124 INFO:teuthology.orchestra.run.smithi070.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-21T10:23:58.124 INFO:teuthology.orchestra.run.smithi070.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-21T10:23:58.124 INFO:teuthology.orchestra.run.smithi070.stdout: librados-devel-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.124 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.124 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-21T10:23:58.125 INFO:teuthology.orchestra.run.smithi070.stdout: librgw2-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.125 INFO:teuthology.orchestra.run.smithi070.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-21T10:23:58.125 INFO:teuthology.orchestra.run.smithi070.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-21T10:23:58.125 INFO:teuthology.orchestra.run.smithi070.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-21T10:23:58.125 INFO:teuthology.orchestra.run.smithi070.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-21T10:23:58.125 INFO:teuthology.orchestra.run.smithi070.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-21T10:23:58.125 INFO:teuthology.orchestra.run.smithi070.stdout: lua-5.4.4-4.el9.x86_64 2024-03-21T10:23:58.125 INFO:teuthology.orchestra.run.smithi070.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-21T10:23:58.125 INFO:teuthology.orchestra.run.smithi070.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-21T10:23:58.125 INFO:teuthology.orchestra.run.smithi070.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-21T10:23:58.125 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-21T10:23:58.125 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-21T10:23:58.125 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-21T10:23:58.125 INFO:teuthology.orchestra.run.smithi070.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-21T10:23:58.126 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-13.11-1.el9.x86_64 2024-03-21T10:23:58.126 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-21T10:23:58.126 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-21T10:23:58.126 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-21T10:23:58.126 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-21T10:23:58.126 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-21T10:23:58.126 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-21T10:23:58.126 INFO:teuthology.orchestra.run.smithi070.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-21T10:23:58.126 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-21T10:23:58.126 INFO:teuthology.orchestra.run.smithi070.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-21T10:23:58.126 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-21T10:23:58.129 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.129 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.130 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cephfs-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.130 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-21T10:23:58.130 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-21T10:23:58.130 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-21T10:23:58.130 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-21T10:23:58.130 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-21T10:23:58.130 INFO:teuthology.orchestra.run.smithi070.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-21T10:23:58.130 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-21T10:23:58.130 INFO:teuthology.orchestra.run.smithi070.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-21T10:23:58.130 INFO:teuthology.orchestra.run.smithi070.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-21T10:23:58.130 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-21T10:23:58.130 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-21T10:23:58.131 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-21T10:23:58.131 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-21T10:23:58.131 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-21T10:23:58.131 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-21T10:23:58.131 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-21T10:23:58.131 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-21T10:23:58.131 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-21T10:23:58.131 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-21T10:23:58.131 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-21T10:23:58.131 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-21T10:23:58.131 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-21T10:23:58.131 INFO:teuthology.orchestra.run.smithi070.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-21T10:23:58.131 INFO:teuthology.orchestra.run.smithi070.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-21T10:23:58.131 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-21T10:23:58.132 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-21T10:23:58.132 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-21T10:23:58.132 INFO:teuthology.orchestra.run.smithi070.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-21T10:23:58.132 INFO:teuthology.orchestra.run.smithi070.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-21T10:23:58.132 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-21T10:23:58.132 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ply-3.11-14.el9.noarch 2024-03-21T10:23:58.132 INFO:teuthology.orchestra.run.smithi070.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-21T10:23:58.132 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-21T10:23:58.132 INFO:teuthology.orchestra.run.smithi070.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-21T10:23:58.132 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-21T10:23:58.132 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-21T10:23:58.132 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-21T10:23:58.132 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-21T10:23:58.133 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-21T10:23:58.133 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-21T10:23:58.133 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.133 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rbd-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.133 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-21T10:23:58.133 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-21T10:23:58.133 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-21T10:23:58.133 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rgw-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.133 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-21T10:23:58.133 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-21T10:23:58.133 INFO:teuthology.orchestra.run.smithi070.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-21T10:23:58.133 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-21T10:23:58.133 INFO:teuthology.orchestra.run.smithi070.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-21T10:23:58.133 INFO:teuthology.orchestra.run.smithi070.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-21T10:23:58.134 INFO:teuthology.orchestra.run.smithi070.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-21T10:23:58.134 INFO:teuthology.orchestra.run.smithi070.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-21T10:23:58.134 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-21T10:23:58.134 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-21T10:23:58.134 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-21T10:23:58.134 INFO:teuthology.orchestra.run.smithi070.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-21T10:23:58.134 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-21T10:23:58.134 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-21T10:23:58.134 INFO:teuthology.orchestra.run.smithi070.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-21T10:23:58.134 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-fuse-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.134 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-mirror-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.134 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-nbd-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:23:58.134 INFO:teuthology.orchestra.run.smithi070.stdout: re2-1:20211101-3.el9.x86_64 2024-03-21T10:23:58.134 INFO:teuthology.orchestra.run.smithi070.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-21T10:23:58.135 INFO:teuthology.orchestra.run.smithi070.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-21T10:23:58.135 INFO:teuthology.orchestra.run.smithi070.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-21T10:23:58.135 INFO:teuthology.orchestra.run.smithi070.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-21T10:23:58.135 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-21T10:23:58.135 INFO:teuthology.orchestra.run.smithi070.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-21T10:23:58.135 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-21T10:23:58.135 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-21T10:23:58.607 DEBUG:teuthology.parallel:result is None 2024-03-21T10:23:59.515 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-21T10:23:59.515 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-2:19.0.0-2231.g2edfee68.el9.x86_64 1/198 2024-03-21T10:23:59.515 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-base-2:19.0.0-2231.g2edfee68.el9.x86_64 2/198 2024-03-21T10:23:59.515 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-common-2:19.0.0-2231.g2edfee68.el9.x86_64 3/198 2024-03-21T10:23:59.516 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-fuse-2:19.0.0-2231.g2edfee68.el9.x86_64 4/198 2024-03-21T10:23:59.516 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2231.g2edfe 5/198 2024-03-21T10:23:59.516 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mds-2:19.0.0-2231.g2edfee68.el9.x86_64 6/198 2024-03-21T10:23:59.516 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-2:19.0.0-2231.g2edfee68.el9.x86_64 7/198 2024-03-21T10:23:59.516 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mon-2:19.0.0-2231.g2edfee68.el9.x86_64 8/198 2024-03-21T10:23:59.516 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-osd-2:19.0.0-2231.g2edfee68.el9.x86_64 9/198 2024-03-21T10:23:59.516 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-radosgw-2:19.0.0-2231.g2edfee68.el9.x86_64 10/198 2024-03-21T10:23:59.516 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-selinux-2:19.0.0-2231.g2edfee68.el9.x86_64 11/198 2024-03-21T10:23:59.516 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-test-2:19.0.0-2231.g2edfee68.el9.x86_64 12/198 2024-03-21T10:23:59.516 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephfs-mirror-2:19.0.0-2231.g2edfee68.el9.x86_64 13/198 2024-03-21T10:23:59.516 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephfs-shell-2:19.0.0-2231.g2edfee68.el9.x86_64 14/198 2024-03-21T10:23:59.516 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs-devel-2:19.0.0-2231.g2edfee68.el9.x86_ 15/198 2024-03-21T10:23:59.516 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs2-2:19.0.0-2231.g2edfee68.el9.x86_64 16/198 2024-03-21T10:23:59.516 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephsqlite-2:19.0.0-2231.g2edfee68.el9.x86_64 17/198 2024-03-21T10:23:59.517 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados-devel-2:19.0.0-2231.g2edfee68.el9.x86_6 18/198 2024-03-21T10:23:59.517 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libradosstriper1-2:19.0.0-2231.g2edfee68.el9.x86 19/198 2024-03-21T10:23:59.517 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librgw2-2:19.0.0-2231.g2edfee68.el9.x86_64 20/198 2024-03-21T10:23:59.517 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2231.g2edfee68.el 21/198 2024-03-21T10:23:59.517 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-common-2:19.0.0-2231.g2edfee68.el9. 22/198 2024-03-21T10:23:59.517 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cephfs-2:19.0.0-2231.g2edfee68.el9.x86_6 23/198 2024-03-21T10:23:59.517 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rados-2:19.0.0-2231.g2edfee68.el9.x86_64 24/198 2024-03-21T10:23:59.517 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rbd-2:19.0.0-2231.g2edfee68.el9.x86_64 25/198 2024-03-21T10:23:59.517 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rgw-2:19.0.0-2231.g2edfee68.el9.x86_64 26/198 2024-03-21T10:23:59.517 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-fuse-2:19.0.0-2231.g2edfee68.el9.x86_64 27/198 2024-03-21T10:23:59.517 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-mirror-2:19.0.0-2231.g2edfee68.el9.x86_64 28/198 2024-03-21T10:23:59.517 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-nbd-2:19.0.0-2231.g2edfee68.el9.x86_64 29/198 2024-03-21T10:23:59.517 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2231.g2edfee68. 30/198 2024-03-21T10:23:59.518 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2231.g2edfee68.el9.noa 31/198 2024-03-21T10:23:59.518 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2231.g2edfee68.el9.n 32/198 2024-03-21T10:23:59.518 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2231.g2ed 33/198 2024-03-21T10:23:59.518 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2231.g2edfee68.el 34/198 2024-03-21T10:23:59.518 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2231.g2edfee68.el9.noarch 35/198 2024-03-21T10:23:59.518 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2231.g2edfee68.e 36/198 2024-03-21T10:23:59.518 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-volume-2:19.0.0-2231.g2edfee68.el9.noarch 37/198 2024-03-21T10:23:59.518 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephadm-2:19.0.0-2231.g2edfee68.el9.noarch 38/198 2024-03-21T10:23:59.518 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephfs-top-2:19.0.0-2231.g2edfee68.el9.noarch 39/198 2024-03-21T10:23:59.518 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-21T10:23:59.518 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-21T10:23:59.518 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-21T10:23:59.519 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-21T10:23:59.519 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-21T10:23:59.519 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-21T10:23:59.519 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-21T10:23:59.519 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-21T10:23:59.519 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-21T10:23:59.519 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-21T10:23:59.519 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-21T10:23:59.519 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-21T10:23:59.519 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-21T10:23:59.519 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-21T10:23:59.519 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-21T10:23:59.519 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-21T10:23:59.520 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-21T10:23:59.520 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-21T10:23:59.520 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-21T10:23:59.520 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-21T10:23:59.520 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-21T10:23:59.520 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-21T10:23:59.520 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-21T10:23:59.520 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-21T10:23:59.520 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-21T10:23:59.520 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-21T10:23:59.520 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-21T10:23:59.520 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-21T10:23:59.520 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-21T10:23:59.520 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-21T10:23:59.520 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-21T10:23:59.521 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-21T10:23:59.521 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-21T10:23:59.521 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-21T10:23:59.521 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-21T10:23:59.521 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-21T10:23:59.521 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-21T10:23:59.521 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-21T10:23:59.521 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-21T10:23:59.521 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-21T10:23:59.521 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-21T10:23:59.521 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-21T10:23:59.521 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-21T10:23:59.521 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-21T10:23:59.521 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-21T10:23:59.522 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-21T10:23:59.522 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-21T10:23:59.522 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-21T10:23:59.522 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-21T10:23:59.522 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-21T10:23:59.522 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-21T10:23:59.522 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-21T10:23:59.522 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-21T10:23:59.522 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-21T10:23:59.522 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-21T10:23:59.522 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-21T10:23:59.522 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-21T10:23:59.522 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-21T10:23:59.523 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-21T10:23:59.523 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-21T10:23:59.523 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-21T10:23:59.523 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-21T10:23:59.523 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-21T10:23:59.523 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-21T10:23:59.523 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-21T10:23:59.523 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-21T10:23:59.523 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-21T10:23:59.523 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-21T10:23:59.523 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-21T10:23:59.523 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-21T10:23:59.523 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-21T10:23:59.523 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-21T10:23:59.524 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-21T10:23:59.524 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-21T10:23:59.524 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-21T10:23:59.524 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-21T10:23:59.524 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-21T10:23:59.524 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-21T10:23:59.524 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-21T10:23:59.524 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-21T10:23:59.524 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-21T10:23:59.524 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-21T10:23:59.524 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-21T10:23:59.524 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-21T10:23:59.524 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-21T10:23:59.524 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-21T10:23:59.525 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-21T10:23:59.525 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-21T10:23:59.525 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-21T10:23:59.525 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-21T10:23:59.525 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-21T10:23:59.525 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-21T10:23:59.525 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-21T10:23:59.525 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-21T10:23:59.525 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-21T10:23:59.525 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-21T10:23:59.525 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-21T10:23:59.525 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-21T10:23:59.525 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-21T10:23:59.525 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-21T10:23:59.526 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-21T10:23:59.526 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-21T10:23:59.526 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-21T10:23:59.526 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-21T10:23:59.526 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-21T10:23:59.526 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-21T10:23:59.526 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-21T10:23:59.526 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-21T10:23:59.526 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-21T10:23:59.526 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-21T10:23:59.526 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-21T10:23:59.526 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:19.0.0-2231.g2edfee68.el9.x86_64 151/198 2024-03-21T10:23:59.526 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-21T10:23:59.526 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:19.0.0-2231.g2edfee68.el9.x86_64 153/198 2024-03-21T10:23:59.527 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-21T10:23:59.527 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-21T10:23:59.527 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-21T10:23:59.527 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-21T10:23:59.527 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-21T10:23:59.527 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-21T10:23:59.527 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-21T10:23:59.527 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-21T10:23:59.527 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-21T10:23:59.527 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-21T10:23:59.527 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-21T10:23:59.527 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-21T10:23:59.527 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-21T10:23:59.527 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-21T10:23:59.528 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-21T10:23:59.528 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-21T10:23:59.528 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-21T10:23:59.528 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-21T10:23:59.528 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-21T10:23:59.528 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-21T10:23:59.528 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-21T10:23:59.528 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-21T10:23:59.528 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-21T10:23:59.528 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-21T10:23:59.528 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-21T10:23:59.528 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-21T10:23:59.528 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-21T10:23:59.528 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-21T10:23:59.528 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-21T10:23:59.529 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-21T10:23:59.529 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-21T10:23:59.529 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-21T10:23:59.529 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-21T10:23:59.529 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-21T10:23:59.529 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-21T10:23:59.529 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-21T10:23:59.529 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-21T10:23:59.529 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-21T10:23:59.529 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-21T10:23:59.529 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-21T10:23:59.529 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-21T10:23:59.529 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-21T10:23:59.529 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-21T10:23:59.530 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-21T10:24:02.573 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-21T10:24:02.574 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-21T10:24:02.574 INFO:teuthology.orchestra.run.smithi040.stdout:Upgraded: 2024-03-21T10:24:02.574 INFO:teuthology.orchestra.run.smithi040.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-21T10:24:02.574 INFO:teuthology.orchestra.run.smithi040.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-21T10:24:02.574 INFO:teuthology.orchestra.run.smithi040.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-21T10:24:02.574 INFO:teuthology.orchestra.run.smithi040.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-21T10:24:02.574 INFO:teuthology.orchestra.run.smithi040.stdout: acl-2.3.1-4.el9.x86_64 2024-03-21T10:24:02.574 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-21T10:24:02.575 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-21T10:24:02.575 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-21T10:24:02.575 INFO:teuthology.orchestra.run.smithi040.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-21T10:24:02.575 INFO:teuthology.orchestra.run.smithi040.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-21T10:24:02.575 INFO:teuthology.orchestra.run.smithi040.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-21T10:24:02.575 INFO:teuthology.orchestra.run.smithi040.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-21T10:24:02.575 INFO:teuthology.orchestra.run.smithi040.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-21T10:24:02.575 INFO:teuthology.orchestra.run.smithi040.stdout: librados2-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.575 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.575 INFO:teuthology.orchestra.run.smithi040.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-21T10:24:02.575 INFO:teuthology.orchestra.run.smithi040.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-21T10:24:02.575 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-21T10:24:02.575 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-21T10:24:02.575 INFO:teuthology.orchestra.run.smithi040.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-21T10:24:02.576 INFO:teuthology.orchestra.run.smithi040.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-21T10:24:02.576 INFO:teuthology.orchestra.run.smithi040.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-21T10:24:02.576 INFO:teuthology.orchestra.run.smithi040.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-21T10:24:02.576 INFO:teuthology.orchestra.run.smithi040.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-21T10:24:02.576 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-03-21T10:24:02.576 INFO:teuthology.orchestra.run.smithi040.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-21T10:24:02.576 INFO:teuthology.orchestra.run.smithi040.stdout: automake-1.16.2-8.el9.noarch 2024-03-21T10:24:02.576 INFO:teuthology.orchestra.run.smithi040.stdout: bison-3.7.4-5.el9.x86_64 2024-03-21T10:24:02.576 INFO:teuthology.orchestra.run.smithi040.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-21T10:24:02.576 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.576 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.576 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.576 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.577 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:24:02.577 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.577 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.577 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.577 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:24:02.577 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:24:02.577 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:24:02.577 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:24:02.577 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:24:02.577 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.577 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.577 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:24:02.577 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.578 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.578 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.578 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:24:02.578 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:24:02.578 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-mirror-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.578 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-shell-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.578 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-top-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:24:02.578 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-21T10:24:02.578 INFO:teuthology.orchestra.run.smithi040.stdout: flex-2.6.4-9.el9.x86_64 2024-03-21T10:24:02.578 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-21T10:24:02.578 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-21T10:24:02.578 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-21T10:24:02.579 INFO:teuthology.orchestra.run.smithi040.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-21T10:24:02.579 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-21T10:24:02.579 INFO:teuthology.orchestra.run.smithi040.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-21T10:24:02.579 INFO:teuthology.orchestra.run.smithi040.stdout: jq-1.6-16.el9.x86_64 2024-03-21T10:24:02.579 INFO:teuthology.orchestra.run.smithi040.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-21T10:24:02.579 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-21T10:24:02.579 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-21T10:24:02.579 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.579 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.579 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.579 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-21T10:24:02.579 INFO:teuthology.orchestra.run.smithi040.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-21T10:24:02.579 INFO:teuthology.orchestra.run.smithi040.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-21T10:24:02.579 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-21T10:24:02.580 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-21T10:24:02.580 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.580 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.580 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-21T10:24:02.580 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.580 INFO:teuthology.orchestra.run.smithi040.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-21T10:24:02.580 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-21T10:24:02.580 INFO:teuthology.orchestra.run.smithi040.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-21T10:24:02.580 INFO:teuthology.orchestra.run.smithi040.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-21T10:24:02.580 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-21T10:24:02.580 INFO:teuthology.orchestra.run.smithi040.stdout: lua-5.4.4-4.el9.x86_64 2024-03-21T10:24:02.580 INFO:teuthology.orchestra.run.smithi040.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-21T10:24:02.580 INFO:teuthology.orchestra.run.smithi040.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-21T10:24:02.580 INFO:teuthology.orchestra.run.smithi040.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-21T10:24:02.581 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-21T10:24:02.581 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-21T10:24:02.581 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-21T10:24:02.581 INFO:teuthology.orchestra.run.smithi040.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-21T10:24:02.581 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-13.11-1.el9.x86_64 2024-03-21T10:24:02.581 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-21T10:24:02.581 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-21T10:24:02.581 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-21T10:24:02.581 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-21T10:24:02.581 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-21T10:24:02.581 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-21T10:24:02.581 INFO:teuthology.orchestra.run.smithi040.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-21T10:24:02.581 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-21T10:24:02.581 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-21T10:24:02.582 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-21T10:24:02.582 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.582 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.582 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.582 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-21T10:24:02.582 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-21T10:24:02.582 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-21T10:24:02.582 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-21T10:24:02.582 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-21T10:24:02.582 INFO:teuthology.orchestra.run.smithi040.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-21T10:24:02.582 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-21T10:24:02.583 INFO:teuthology.orchestra.run.smithi040.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-21T10:24:02.583 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-21T10:24:02.583 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-21T10:24:02.583 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-21T10:24:02.583 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-21T10:24:02.583 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-21T10:24:02.583 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-21T10:24:02.583 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-21T10:24:02.583 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-21T10:24:02.583 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-21T10:24:02.583 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-21T10:24:02.583 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-21T10:24:02.583 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-21T10:24:02.584 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-21T10:24:02.584 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-21T10:24:02.584 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-21T10:24:02.584 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-21T10:24:02.584 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-21T10:24:02.584 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-21T10:24:02.584 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-21T10:24:02.584 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-21T10:24:02.584 INFO:teuthology.orchestra.run.smithi040.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-21T10:24:02.584 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-21T10:24:02.584 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ply-3.11-14.el9.noarch 2024-03-21T10:24:02.584 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-21T10:24:02.584 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-21T10:24:02.585 INFO:teuthology.orchestra.run.smithi040.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-21T10:24:02.585 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-21T10:24:02.585 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-21T10:24:02.585 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-21T10:24:02.585 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-21T10:24:02.585 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-21T10:24:02.585 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-21T10:24:02.585 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.585 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.585 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-21T10:24:02.585 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-21T10:24:02.585 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-21T10:24:02.585 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.585 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-21T10:24:02.585 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-21T10:24:02.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-21T10:24:02.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-21T10:24:02.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-21T10:24:02.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-21T10:24:02.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-21T10:24:02.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-21T10:24:02.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-21T10:24:02.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-21T10:24:02.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-21T10:24:02.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-21T10:24:02.586 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-21T10:24:02.587 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-21T10:24:02.587 INFO:teuthology.orchestra.run.smithi040.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-21T10:24:02.587 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.587 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.587 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:02.587 INFO:teuthology.orchestra.run.smithi040.stdout: re2-1:20211101-3.el9.x86_64 2024-03-21T10:24:02.587 INFO:teuthology.orchestra.run.smithi040.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-21T10:24:02.587 INFO:teuthology.orchestra.run.smithi040.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-21T10:24:02.587 INFO:teuthology.orchestra.run.smithi040.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-21T10:24:02.587 INFO:teuthology.orchestra.run.smithi040.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-21T10:24:02.587 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-21T10:24:02.588 INFO:teuthology.orchestra.run.smithi040.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-21T10:24:02.588 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-21T10:24:02.588 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-03-21T10:24:03.037 DEBUG:teuthology.parallel:result is None 2024-03-21T10:24:07.607 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2231.g2edfee68.el9.x86_64 132/198 2024-03-21T10:24:07.607 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /sys 2024-03-21T10:24:07.607 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /proc 2024-03-21T10:24:07.607 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /mnt 2024-03-21T10:24:07.608 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /var/tmp 2024-03-21T10:24:07.608 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /home 2024-03-21T10:24:07.608 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /root 2024-03-21T10:24:07.608 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /tmp 2024-03-21T10:24:07.608 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:24:07.818 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-volume-2:19.0.0-2231.g2edfee68.el9.noarch 133/198 2024-03-21T10:24:07.864 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-volume-2:19.0.0-2231.g2edfee68.el9.noarch 133/198 2024-03-21T10:24:07.864 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:24:07.864 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-21T10:24:07.864 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:24:09.519 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-osd-2:19.0.0-2231.g2edfee68.el9.x86_64 134/198 2024-03-21T10:24:09.608 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-osd-2:19.0.0-2231.g2edfee68.el9.x86_64 134/198 2024-03-21T10:24:09.608 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:24:09.608 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-21T10:24:09.609 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-21T10:24:09.609 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-21T10:24:09.609 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:24:09.869 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mds-2:19.0.0-2231.g2edfee68.el9.x86_64 135/198 2024-03-21T10:24:09.954 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mds-2:19.0.0-2231.g2edfee68.el9.x86_64 135/198 2024-03-21T10:24:09.954 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:24:09.954 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-21T10:24:09.954 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-21T10:24:09.954 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-21T10:24:09.954 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:24:10.478 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mon-2:19.0.0-2231.g2edfee68.el9.x86_64 136/198 2024-03-21T10:24:10.562 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mon-2:19.0.0-2231.g2edfee68.el9.x86_64 136/198 2024-03-21T10:24:10.562 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:24:10.562 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-21T10:24:10.562 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-21T10:24:10.562 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-21T10:24:10.562 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:24:10.680 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-21T10:24:11.282 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:19.0.0-2231.g2edfee68.el9.noarch 138/198 2024-03-21T10:24:11.314 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephadm-2:19.0.0-2231.g2edfee68.el9.noarch 138/198 2024-03-21T10:24:11.365 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2231.g2edfee68.e 139/198 2024-03-21T10:24:11.468 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2231.g2edfee68. 140/198 2024-03-21T10:24:11.504 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2231.g2edfee68.el9.noa 141/198 2024-03-21T10:24:11.849 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2231.g2edfee68.el9.noa 141/198 2024-03-21T10:24:11.887 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2231.g2edfee68.el9.n 142/198 2024-03-21T10:24:13.067 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2231.g2edfee68.el9.n 142/198 2024-03-21T10:24:13.096 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2231.g2ed 143/198 2024-03-21T10:24:13.245 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2231.g2ed 143/198 2024-03-21T10:24:13.418 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2231.g2edfee68.el 144/198 2024-03-21T10:24:13.448 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-2:19.0.0-2231.g2edfee68.el9.x86_64 145/198 2024-03-21T10:24:13.528 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2231.g2edfee68.el9.x86_64 145/198 2024-03-21T10:24:13.528 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:24:13.528 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-21T10:24:13.528 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-21T10:24:13.528 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-21T10:24:13.528 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:24:13.590 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-rook-2:19.0.0-2231.g2edfee68.el9.noarch 146/198 2024-03-21T10:24:13.631 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2231.g2edfee68.el9.noarch 146/198 2024-03-21T10:24:13.687 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-2:19.0.0-2231.g2edfee68.el9.x86_64 147/198 2024-03-21T10:24:14.690 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-21T10:24:14.726 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-radosgw-2:19.0.0-2231.g2edfee68.el9.x86_64 149/198 2024-03-21T10:24:14.809 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2231.g2edfee68.el9.x86_64 149/198 2024-03-21T10:24:14.809 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:24:14.809 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-21T10:24:14.810 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-21T10:24:14.810 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-21T10:24:14.810 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:24:14.859 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2231.g2edfe 150/198 2024-03-21T10:24:14.955 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2231.g2edfe 150/198 2024-03-21T10:24:14.955 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:24:14.955 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-21T10:24:14.956 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:24:15.035 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephfs-mirror-2:19.0.0-2231.g2edfee68.el9.x86_64 151/198 2024-03-21T10:24:15.142 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2231.g2edfee68.el9.x86_64 151/198 2024-03-21T10:24:15.142 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:24:15.142 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-21T10:24:15.142 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-21T10:24:15.142 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-21T10:24:15.143 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:24:15.536 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-mirror-2:19.0.0-2231.g2edfee68.el9.x86_64 152/198 2024-03-21T10:24:15.620 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2231.g2edfee68.el9.x86_64 152/198 2024-03-21T10:24:15.620 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T10:24:15.620 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-21T10:24:15.620 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-21T10:24:15.621 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-21T10:24:15.621 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:24:20.709 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-test-2:19.0.0-2231.g2edfee68.el9.x86_64 153/198 2024-03-21T10:24:20.754 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-fuse-2:19.0.0-2231.g2edfee68.el9.x86_64 154/198 2024-03-21T10:24:20.832 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-nbd-2:19.0.0-2231.g2edfee68.el9.x86_64 155/198 2024-03-21T10:24:20.872 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-21T10:24:21.197 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephfs-shell-2:19.0.0-2231.g2edfee68.el9.x86_64 157/198 2024-03-21T10:24:21.327 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-21T10:24:21.384 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-fuse-2:19.0.0-2231.g2edfee68.el9.x86_64 159/198 2024-03-21T10:24:21.445 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-21T10:24:21.533 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-21T10:24:21.578 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-21T10:24:21.622 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs-devel-2:19.0.0-2231.g2edfee68.el9.x86_ 163/198 2024-03-21T10:24:21.658 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-21T10:24:21.710 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-21T10:24:21.802 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-21T10:24:21.933 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-21T10:24:21.974 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-21T10:24:22.832 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-21T10:24:22.880 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-21T10:24:22.955 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-21T10:24:24.348 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephfs-top-2:19.0.0-2231.g2edfee68.el9.noarch 170/198 2024-03-21T10:24:24.443 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-21T10:24:24.503 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-21T10:24:24.632 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-21T10:24:24.670 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-21T10:24:24.703 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-21T10:24:24.746 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-21T10:24:24.786 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-21T10:24:24.787 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-21T10:24:24.968 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-21T10:24:25.000 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-21T10:24:25.000 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-21T10:24:25.147 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-21T10:24:25.191 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-21T10:24:25.268 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-21T10:24:25.311 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-21T10:24:25.352 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-21T10:24:25.421 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-21T10:24:25.491 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-21T10:24:25.539 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-21T10:24:25.573 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-21T10:24:25.624 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-21T10:24:25.657 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-21T10:24:25.699 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-21T10:24:25.736 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-21T10:24:25.737 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-21T10:24:26.083 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-21T10:24:26.083 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-21T10:24:26.221 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-21T10:24:26.252 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-21T10:24:26.332 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-21T10:24:26.371 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-21T10:24:26.411 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-21T10:24:26.529 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-21T10:24:26.825 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-21T10:24:26.825 INFO:teuthology.orchestra.run.smithi031.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-21T10:24:26.825 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:24:31.602 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-2:19.0.0-2231.g2edfee68.el9.x86_64 1/198 2024-03-21T10:24:31.602 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-base-2:19.0.0-2231.g2edfee68.el9.x86_64 2/198 2024-03-21T10:24:31.602 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-common-2:19.0.0-2231.g2edfee68.el9.x86_64 3/198 2024-03-21T10:24:31.603 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-fuse-2:19.0.0-2231.g2edfee68.el9.x86_64 4/198 2024-03-21T10:24:31.603 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2231.g2edfe 5/198 2024-03-21T10:24:31.603 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mds-2:19.0.0-2231.g2edfee68.el9.x86_64 6/198 2024-03-21T10:24:31.603 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-2:19.0.0-2231.g2edfee68.el9.x86_64 7/198 2024-03-21T10:24:31.603 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mon-2:19.0.0-2231.g2edfee68.el9.x86_64 8/198 2024-03-21T10:24:31.603 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-osd-2:19.0.0-2231.g2edfee68.el9.x86_64 9/198 2024-03-21T10:24:31.603 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-radosgw-2:19.0.0-2231.g2edfee68.el9.x86_64 10/198 2024-03-21T10:24:31.603 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-selinux-2:19.0.0-2231.g2edfee68.el9.x86_64 11/198 2024-03-21T10:24:31.603 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-test-2:19.0.0-2231.g2edfee68.el9.x86_64 12/198 2024-03-21T10:24:31.603 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephfs-mirror-2:19.0.0-2231.g2edfee68.el9.x86_64 13/198 2024-03-21T10:24:31.603 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephfs-shell-2:19.0.0-2231.g2edfee68.el9.x86_64 14/198 2024-03-21T10:24:31.603 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs-devel-2:19.0.0-2231.g2edfee68.el9.x86_ 15/198 2024-03-21T10:24:31.603 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs2-2:19.0.0-2231.g2edfee68.el9.x86_64 16/198 2024-03-21T10:24:31.604 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephsqlite-2:19.0.0-2231.g2edfee68.el9.x86_64 17/198 2024-03-21T10:24:31.604 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados-devel-2:19.0.0-2231.g2edfee68.el9.x86_6 18/198 2024-03-21T10:24:31.604 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libradosstriper1-2:19.0.0-2231.g2edfee68.el9.x86 19/198 2024-03-21T10:24:31.604 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librgw2-2:19.0.0-2231.g2edfee68.el9.x86_64 20/198 2024-03-21T10:24:31.604 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2231.g2edfee68.el 21/198 2024-03-21T10:24:31.604 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-common-2:19.0.0-2231.g2edfee68.el9. 22/198 2024-03-21T10:24:31.604 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cephfs-2:19.0.0-2231.g2edfee68.el9.x86_6 23/198 2024-03-21T10:24:31.604 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rados-2:19.0.0-2231.g2edfee68.el9.x86_64 24/198 2024-03-21T10:24:31.604 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rbd-2:19.0.0-2231.g2edfee68.el9.x86_64 25/198 2024-03-21T10:24:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rgw-2:19.0.0-2231.g2edfee68.el9.x86_64 26/198 2024-03-21T10:24:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-fuse-2:19.0.0-2231.g2edfee68.el9.x86_64 27/198 2024-03-21T10:24:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-mirror-2:19.0.0-2231.g2edfee68.el9.x86_64 28/198 2024-03-21T10:24:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-nbd-2:19.0.0-2231.g2edfee68.el9.x86_64 29/198 2024-03-21T10:24:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2231.g2edfee68. 30/198 2024-03-21T10:24:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2231.g2edfee68.el9.noa 31/198 2024-03-21T10:24:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2231.g2edfee68.el9.n 32/198 2024-03-21T10:24:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2231.g2ed 33/198 2024-03-21T10:24:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2231.g2edfee68.el 34/198 2024-03-21T10:24:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2231.g2edfee68.el9.noarch 35/198 2024-03-21T10:24:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2231.g2edfee68.e 36/198 2024-03-21T10:24:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-volume-2:19.0.0-2231.g2edfee68.el9.noarch 37/198 2024-03-21T10:24:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephadm-2:19.0.0-2231.g2edfee68.el9.noarch 38/198 2024-03-21T10:24:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephfs-top-2:19.0.0-2231.g2edfee68.el9.noarch 39/198 2024-03-21T10:24:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-21T10:24:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-21T10:24:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-21T10:24:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-21T10:24:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-21T10:24:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-21T10:24:31.622 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-21T10:24:31.622 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-21T10:24:31.622 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-21T10:24:31.622 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-21T10:24:31.622 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-21T10:24:31.622 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-21T10:24:31.622 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-21T10:24:31.622 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-21T10:24:31.622 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-21T10:24:31.622 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-21T10:24:31.622 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-21T10:24:31.622 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-21T10:24:31.622 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-21T10:24:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-21T10:24:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-21T10:24:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-21T10:24:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-21T10:24:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-21T10:24:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-21T10:24:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-21T10:24:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-21T10:24:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-21T10:24:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-21T10:24:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-21T10:24:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-21T10:24:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-21T10:24:31.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-21T10:24:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-21T10:24:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-21T10:24:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-21T10:24:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-21T10:24:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-21T10:24:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-21T10:24:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-21T10:24:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-21T10:24:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-21T10:24:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-21T10:24:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-21T10:24:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-21T10:24:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-21T10:24:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-21T10:24:31.624 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-21T10:24:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-21T10:24:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-21T10:24:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-21T10:24:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-21T10:24:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-21T10:24:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-21T10:24:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-21T10:24:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-21T10:24:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-21T10:24:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-21T10:24:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-21T10:24:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-21T10:24:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-21T10:24:31.625 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-21T10:24:31.626 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-21T10:24:31.626 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-21T10:24:31.626 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-21T10:24:31.626 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-21T10:24:31.626 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-21T10:24:31.626 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-21T10:24:31.626 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-21T10:24:31.626 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-21T10:24:31.626 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-21T10:24:31.626 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-21T10:24:31.626 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-21T10:24:31.626 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-21T10:24:31.626 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-21T10:24:31.626 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-21T10:24:31.627 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-21T10:24:31.627 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-21T10:24:31.627 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-21T10:24:31.627 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-21T10:24:31.627 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-21T10:24:31.627 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-21T10:24:31.627 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-21T10:24:31.627 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-21T10:24:31.627 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-21T10:24:31.627 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-21T10:24:31.627 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-21T10:24:31.627 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-21T10:24:31.627 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-21T10:24:31.627 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-21T10:24:31.628 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-21T10:24:31.628 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-21T10:24:31.628 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-21T10:24:31.628 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-21T10:24:31.628 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-21T10:24:31.628 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-21T10:24:31.628 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-21T10:24:31.628 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-21T10:24:31.628 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-21T10:24:31.628 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-21T10:24:31.628 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-21T10:24:31.628 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-21T10:24:31.628 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-21T10:24:31.628 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-21T10:24:31.629 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-21T10:24:31.629 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-21T10:24:31.629 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-21T10:24:31.629 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-21T10:24:31.629 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-21T10:24:31.629 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-21T10:24:31.629 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-21T10:24:31.629 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:19.0.0-2231.g2edfee68.el9.x86_64 151/198 2024-03-21T10:24:31.629 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-21T10:24:31.629 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:19.0.0-2231.g2edfee68.el9.x86_64 153/198 2024-03-21T10:24:31.629 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-21T10:24:31.629 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-21T10:24:31.629 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-21T10:24:31.629 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-21T10:24:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-21T10:24:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-21T10:24:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-21T10:24:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-21T10:24:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-21T10:24:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-21T10:24:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-21T10:24:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-21T10:24:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-21T10:24:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-21T10:24:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-21T10:24:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-21T10:24:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-21T10:24:31.630 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-21T10:24:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-21T10:24:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-21T10:24:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-21T10:24:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-21T10:24:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-21T10:24:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-21T10:24:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-21T10:24:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-21T10:24:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-21T10:24:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-21T10:24:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-21T10:24:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-21T10:24:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-21T10:24:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-21T10:24:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-21T10:24:31.632 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-21T10:24:31.632 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-21T10:24:31.632 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-21T10:24:31.632 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-21T10:24:31.632 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-21T10:24:31.632 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-21T10:24:31.632 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-21T10:24:31.632 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-21T10:24:31.632 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-21T10:24:31.632 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-21T10:24:31.632 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-21T10:24:34.481 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-21T10:24:34.481 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:24:34.481 INFO:teuthology.orchestra.run.smithi031.stdout:Upgraded: 2024-03-21T10:24:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-21T10:24:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-21T10:24:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-21T10:24:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-21T10:24:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: acl-2.3.1-4.el9.x86_64 2024-03-21T10:24:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-21T10:24:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-21T10:24:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-21T10:24:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-21T10:24:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-21T10:24:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-21T10:24:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-21T10:24:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-21T10:24:34.483 INFO:teuthology.orchestra.run.smithi031.stdout: librados2-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.483 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.483 INFO:teuthology.orchestra.run.smithi031.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-21T10:24:34.483 INFO:teuthology.orchestra.run.smithi031.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-21T10:24:34.483 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-21T10:24:34.483 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-21T10:24:34.483 INFO:teuthology.orchestra.run.smithi031.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-21T10:24:34.483 INFO:teuthology.orchestra.run.smithi031.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-21T10:24:34.483 INFO:teuthology.orchestra.run.smithi031.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-21T10:24:34.483 INFO:teuthology.orchestra.run.smithi031.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-21T10:24:34.483 INFO:teuthology.orchestra.run.smithi031.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-21T10:24:34.483 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-03-21T10:24:34.483 INFO:teuthology.orchestra.run.smithi031.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-21T10:24:34.484 INFO:teuthology.orchestra.run.smithi031.stdout: automake-1.16.2-8.el9.noarch 2024-03-21T10:24:34.484 INFO:teuthology.orchestra.run.smithi031.stdout: bison-3.7.4-5.el9.x86_64 2024-03-21T10:24:34.484 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-21T10:24:34.484 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.484 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.484 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.484 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.484 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:24:34.484 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.484 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:24:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:24:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:24:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:24:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:24:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:24:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:24:34.485 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:24:34.486 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-mirror-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.486 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-shell-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.486 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-top-2:19.0.0-2231.g2edfee68.el9.noarch 2024-03-21T10:24:34.486 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-21T10:24:34.486 INFO:teuthology.orchestra.run.smithi031.stdout: flex-2.6.4-9.el9.x86_64 2024-03-21T10:24:34.486 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-21T10:24:34.486 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-21T10:24:34.486 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-21T10:24:34.486 INFO:teuthology.orchestra.run.smithi031.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-21T10:24:34.486 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-21T10:24:34.486 INFO:teuthology.orchestra.run.smithi031.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-21T10:24:34.486 INFO:teuthology.orchestra.run.smithi031.stdout: jq-1.6-16.el9.x86_64 2024-03-21T10:24:34.487 INFO:teuthology.orchestra.run.smithi031.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-21T10:24:34.487 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-21T10:24:34.487 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-21T10:24:34.487 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.487 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.487 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.487 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-21T10:24:34.487 INFO:teuthology.orchestra.run.smithi031.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-21T10:24:34.487 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-21T10:24:34.487 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-21T10:24:34.487 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-21T10:24:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-21T10:24:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-21T10:24:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-21T10:24:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-21T10:24:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-21T10:24:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-21T10:24:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: lua-5.4.4-4.el9.x86_64 2024-03-21T10:24:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-21T10:24:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-21T10:24:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-21T10:24:34.488 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-21T10:24:34.489 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-21T10:24:34.489 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-21T10:24:34.489 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-21T10:24:34.489 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-13.11-1.el9.x86_64 2024-03-21T10:24:34.489 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-21T10:24:34.489 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-21T10:24:34.489 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-21T10:24:34.489 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-21T10:24:34.489 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-21T10:24:34.489 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-21T10:24:34.489 INFO:teuthology.orchestra.run.smithi031.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-21T10:24:34.490 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-21T10:24:34.490 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-21T10:24:34.490 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-21T10:24:34.490 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.490 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.490 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.490 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-21T10:24:34.490 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-21T10:24:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-21T10:24:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-21T10:24:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-21T10:24:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-21T10:24:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-21T10:24:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-21T10:24:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-21T10:24:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-21T10:24:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-21T10:24:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-21T10:24:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-21T10:24:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-21T10:24:34.491 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-21T10:24:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-21T10:24:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-21T10:24:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-21T10:24:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-21T10:24:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-21T10:24:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-21T10:24:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-21T10:24:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-21T10:24:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-21T10:24:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-21T10:24:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-21T10:24:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-21T10:24:34.492 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-21T10:24:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-21T10:24:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-21T10:24:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply-3.11-14.el9.noarch 2024-03-21T10:24:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-21T10:24:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-21T10:24:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-21T10:24:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-21T10:24:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-21T10:24:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-21T10:24:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-21T10:24:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-21T10:24:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-21T10:24:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.493 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-21T10:24:34.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-21T10:24:34.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-21T10:24:34.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-21T10:24:34.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-21T10:24:34.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-21T10:24:34.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-21T10:24:34.494 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-21T10:24:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-21T10:24:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-21T10:24:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-21T10:24:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-21T10:24:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-21T10:24:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-21T10:24:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-21T10:24:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-21T10:24:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-21T10:24:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-21T10:24:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd-2:19.0.0-2231.g2edfee68.el9.x86_64 2024-03-21T10:24:34.495 INFO:teuthology.orchestra.run.smithi031.stdout: re2-1:20211101-3.el9.x86_64 2024-03-21T10:24:34.496 INFO:teuthology.orchestra.run.smithi031.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-21T10:24:34.496 INFO:teuthology.orchestra.run.smithi031.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-21T10:24:34.496 INFO:teuthology.orchestra.run.smithi031.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-21T10:24:34.496 INFO:teuthology.orchestra.run.smithi031.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-21T10:24:34.496 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-21T10:24:34.496 INFO:teuthology.orchestra.run.smithi031.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-21T10:24:34.496 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:24:34.496 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-21T10:24:35.201 DEBUG:teuthology.parallel:result is None 2024-03-21T10:24:35.202 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2edfee68ad7243bed422a049b6f5f58619300f11 2024-03-21T10:24:35.608 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-21T10:24:35.665 INFO:teuthology.orchestra.run.smithi031.stdout:19.0.0-2231.g2edfee68.el9 2024-03-21T10:24:35.665 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2231.g2edfee68.el9 2024-03-21T10:24:35.665 INFO:teuthology.task.install:The correct ceph version 19.0.0-2231.g2edfee68 is installed. 2024-03-21T10:24:35.667 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2edfee68ad7243bed422a049b6f5f58619300f11 2024-03-21T10:24:35.813 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-21T10:24:35.839 INFO:teuthology.orchestra.run.smithi040.stdout:19.0.0-2231.g2edfee68.el9 2024-03-21T10:24:35.840 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2231.g2edfee68.el9 2024-03-21T10:24:35.840 INFO:teuthology.task.install:The correct ceph version 19.0.0-2231.g2edfee68 is installed. 2024-03-21T10:24:35.842 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2edfee68ad7243bed422a049b6f5f58619300f11 2024-03-21T10:24:36.169 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-21T10:24:36.195 INFO:teuthology.orchestra.run.smithi070.stdout:19.0.0-2231.g2edfee68.el9 2024-03-21T10:24:36.195 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2231.g2edfee68.el9 2024-03-21T10:24:36.196 INFO:teuthology.task.install:The correct ceph version 19.0.0-2231.g2edfee68 is installed. 2024-03-21T10:24:36.198 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-21T10:24:36.198 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-21T10:24:36.198 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-21T10:24:36.309 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-21T10:24:36.310 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-21T10:24:36.349 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-21T10:24:36.349 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-21T10:24:36.383 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-21T10:24:36.383 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-21T10:24:36.384 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/daemon-helper 2024-03-21T10:24:36.464 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-21T10:24:36.580 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-21T10:24:36.580 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/daemon-helper 2024-03-21T10:24:36.617 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-21T10:24:36.690 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-21T10:24:36.690 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/daemon-helper 2024-03-21T10:24:36.721 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-21T10:24:36.790 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-21T10:24:36.790 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-21T10:24:36.790 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-21T10:24:36.868 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-21T10:24:36.982 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-21T10:24:36.983 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-21T10:24:37.014 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-21T10:24:37.085 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-21T10:24:37.085 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-21T10:24:37.112 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-21T10:24:37.179 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-21T10:24:37.179 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-21T10:24:37.180 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/stdin-killer 2024-03-21T10:24:37.258 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-21T10:24:37.374 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-21T10:24:37.374 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/stdin-killer 2024-03-21T10:24:37.404 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-21T10:24:37.474 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-21T10:24:37.474 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/stdin-killer 2024-03-21T10:24:37.501 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-21T10:24:37.567 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-21T10:24:37.670 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '2edfee68ad7243bed422a049b6f5f58619300f11', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-21T10:24:37.670 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 2024-03-21T10:24:37.671 INFO:tasks.cephadm:Cluster fsid is 3847408c-e76d-11ee-95cd-87774f69a715 2024-03-21T10:24:37.671 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-21T10:24:37.671 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.31', 'mon.b': '172.21.15.40', 'mon.c': '172.21.15.70'} 2024-03-21T10:24:37.671 INFO:tasks.cephadm:First mon is mon.a on smithi031 2024-03-21T10:24:37.671 INFO:tasks.cephadm:First mgr is x 2024-03-21T10:24:37.671 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-21T10:24:37.671 DEBUG:teuthology.orchestra.run.smithi031:> sudo hostname $(hostname -s) 2024-03-21T10:24:37.752 DEBUG:teuthology.orchestra.run.smithi040:> sudo hostname $(hostname -s) 2024-03-21T10:24:37.782 DEBUG:teuthology.orchestra.run.smithi070:> sudo hostname $(hostname -s) 2024-03-21T10:24:37.809 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-21T10:24:37.810 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2edfee68ad7243bed422a049b6f5f58619300f11 2024-03-21T10:24:38.044 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '2edfee68ad7243bed422a049b6f5f58619300f11', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77791/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2231-g2edfee68', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2231.g2edfee68'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-20 07:25:08.579979', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240320.053002', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-21T10:24:38.193 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-vshankar-testing-20240320.053002, sha1 2edfee68ad7243bed422a049b6f5f58619300f11 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=2edfee68ad7243bed422a049b6f5f58619300f11 2024-03-21T10:24:38.195 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/centos/9/x86_64/flavors/default/cephadm 2024-03-21T10:24:38.195 DEBUG:teuthology.orchestra.run.smithi031:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-21T10:24:38.651 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r--. 1 ubuntu ubuntu 769188 Mar 21 10:24 /home/ubuntu/cephtest/cephadm 2024-03-21T10:24:38.651 DEBUG:teuthology.orchestra.run.smithi040:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-21T10:24:39.059 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r--. 1 ubuntu ubuntu 769188 Mar 21 10:24 /home/ubuntu/cephtest/cephadm 2024-03-21T10:24:39.059 DEBUG:teuthology.orchestra.run.smithi070:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-21T10:24:39.638 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-r--r--. 1 ubuntu ubuntu 769188 Mar 21 10:24 /home/ubuntu/cephtest/cephadm 2024-03-21T10:24:39.639 DEBUG:teuthology.orchestra.run.smithi031:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-21T10:24:39.690 DEBUG:teuthology.orchestra.run.smithi040:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-21T10:24:39.710 DEBUG:teuthology.orchestra.run.smithi070:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-21T10:24:39.744 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 on all hosts... 2024-03-21T10:24:39.744 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 pull 2024-03-21T10:24:39.749 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 pull 2024-03-21T10:24:39.752 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 pull 2024-03-21T10:24:40.565 INFO:teuthology.orchestra.run.smithi040.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11... 2024-03-21T10:24:40.595 INFO:teuthology.orchestra.run.smithi070.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11... 2024-03-21T10:24:41.441 INFO:teuthology.orchestra.run.smithi031.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11... 2024-03-21T10:25:17.938 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-21T10:25:17.938 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_version": "ceph version 19.0.0-2231-g2edfee68 (2edfee68ad7243bed422a049b6f5f58619300f11) squid (dev)", 2024-03-21T10:25:17.938 INFO:teuthology.orchestra.run.smithi040.stdout: "image_id": "237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54", 2024-03-21T10:25:17.938 INFO:teuthology.orchestra.run.smithi040.stdout: "repo_digests": [ 2024-03-21T10:25:17.938 INFO:teuthology.orchestra.run.smithi040.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232" 2024-03-21T10:25:17.938 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-03-21T10:25:17.939 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-21T10:25:18.771 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:25:18.771 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_version": "ceph version 19.0.0-2231-g2edfee68 (2edfee68ad7243bed422a049b6f5f58619300f11) squid (dev)", 2024-03-21T10:25:18.771 INFO:teuthology.orchestra.run.smithi031.stdout: "image_id": "237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54", 2024-03-21T10:25:18.771 INFO:teuthology.orchestra.run.smithi031.stdout: "repo_digests": [ 2024-03-21T10:25:18.771 INFO:teuthology.orchestra.run.smithi031.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232" 2024-03-21T10:25:18.771 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-03-21T10:25:18.771 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:25:24.187 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-03-21T10:25:24.187 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_version": "ceph version 19.0.0-2231-g2edfee68 (2edfee68ad7243bed422a049b6f5f58619300f11) squid (dev)", 2024-03-21T10:25:24.187 INFO:teuthology.orchestra.run.smithi070.stdout: "image_id": "237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54", 2024-03-21T10:25:24.187 INFO:teuthology.orchestra.run.smithi070.stdout: "repo_digests": [ 2024-03-21T10:25:24.187 INFO:teuthology.orchestra.run.smithi070.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232" 2024-03-21T10:25:24.187 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-03-21T10:25:24.187 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-03-21T10:25:24.211 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph 2024-03-21T10:25:24.320 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /etc/ceph 2024-03-21T10:25:24.361 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /etc/ceph 2024-03-21T10:25:24.391 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 777 /etc/ceph 2024-03-21T10:25:24.497 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 777 /etc/ceph 2024-03-21T10:25:24.532 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 777 /etc/ceph 2024-03-21T10:25:24.560 INFO:tasks.cephadm:Writing seed config... 2024-03-21T10:25:24.561 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-21T10:25:24.562 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-21T10:25:24.562 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-21T10:25:24.562 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-21T10:25:24.562 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-21T10:25:24.562 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-21T10:25:24.562 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-21T10:25:24.562 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-21T10:25:24.562 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-21T10:25:24.562 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-21T10:25:24.562 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-21T10:25:24.562 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-21T10:25:24.562 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-21T10:25:24.562 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-21T10:25:24.562 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-21T10:25:24.563 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-21T10:25:24.563 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-21T10:25:24.563 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-21T10:25:24.563 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-21T10:25:24.563 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-21T10:25:24.563 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-21T10:25:24.563 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-21T10:25:24.563 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-21T10:25:24.563 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-21T10:25:24.563 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-21T10:25:24.563 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-21T10:25:24.563 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-21T10:25:24.563 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-21T10:25:24.563 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-21T10:25:24.563 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-21T10:25:24.564 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-21T10:25:24.564 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-21T10:25:24.564 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-21T10:25:24.564 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-21T10:25:24.564 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-21T10:25:24.564 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-21T10:25:24.564 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-21T10:25:24.564 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-21T10:25:24.564 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-21T10:25:24.564 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-21T10:25:24.564 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-21T10:25:24.564 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-21T10:25:24.564 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-21T10:25:24.564 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-03-21T10:25:24.565 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-21T10:25:24.565 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-21T10:25:24.621 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 3847408c-e76d-11ee-95cd-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-21T10:25:24.622 DEBUG:teuthology.orchestra.run.smithi031:mon.a> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mon.a.service 2024-03-21T10:25:24.669 DEBUG:teuthology.orchestra.run.smithi031:mgr.x> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mgr.x.service 2024-03-21T10:25:24.716 INFO:tasks.cephadm:Bootstrapping... 2024-03-21T10:25:24.716 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 -v bootstrap --fsid 3847408c-e76d-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.31 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-21T10:25:24.999 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-03-21T10:25:25.000 INFO:teuthology.orchestra.run.smithi031.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11', '-v', 'bootstrap', '--fsid', '3847408c-e76d-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.31', '--skip-admin-label'] 2024-03-21T10:25:25.000 INFO:teuthology.orchestra.run.smithi031.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-21T10:25:25.000 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying podman|docker is present... 2024-03-21T10:25:25.061 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-21T10:25:25.061 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying lvm2 is present... 2024-03-21T10:25:25.061 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying time synchronization is in place... 2024-03-21T10:25:25.079 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-21T10:25:25.079 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-21T10:25:25.096 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-21T10:25:25.096 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-03-21T10:25:25.115 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout enabled 2024-03-21T10:25:25.133 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout active 2024-03-21T10:25:25.133 INFO:teuthology.orchestra.run.smithi031.stdout:Unit chronyd.service is enabled and running 2024-03-21T10:25:25.133 INFO:teuthology.orchestra.run.smithi031.stdout:Repeating the final host check... 2024-03-21T10:25:25.187 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-21T10:25:25.188 INFO:teuthology.orchestra.run.smithi031.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-21T10:25:25.188 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl is present 2024-03-21T10:25:25.188 INFO:teuthology.orchestra.run.smithi031.stdout:lvcreate is present 2024-03-21T10:25:25.204 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-21T10:25:25.204 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-21T10:25:25.225 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-21T10:25:25.226 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-03-21T10:25:25.249 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout enabled 2024-03-21T10:25:25.271 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout active 2024-03-21T10:25:25.272 INFO:teuthology.orchestra.run.smithi031.stdout:Unit chronyd.service is enabled and running 2024-03-21T10:25:25.272 INFO:teuthology.orchestra.run.smithi031.stdout:Host looks OK 2024-03-21T10:25:25.273 INFO:teuthology.orchestra.run.smithi031.stdout:Cluster fsid: 3847408c-e76d-11ee-95cd-87774f69a715 2024-03-21T10:25:25.273 INFO:teuthology.orchestra.run.smithi031.stdout:Acquiring lock 140426550346704 on /run/cephadm/3847408c-e76d-11ee-95cd-87774f69a715.lock 2024-03-21T10:25:25.274 INFO:teuthology.orchestra.run.smithi031.stdout:Lock 140426550346704 acquired on /run/cephadm/3847408c-e76d-11ee-95cd-87774f69a715.lock 2024-03-21T10:25:25.275 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying IP 172.21.15.31 port 3300 ... 2024-03-21T10:25:25.276 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying IP 172.21.15.31 port 6789 ... 2024-03-21T10:25:25.277 INFO:teuthology.orchestra.run.smithi031.stdout:Base mon IP(s) is [172.21.15.31:3300, 172.21.15.31:6789], mon addrv is [v2:172.21.15.31:3300,v1:172.21.15.31:6789] 2024-03-21T10:25:25.298 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.31 metric 100 2024-03-21T10:25:25.298 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.31 metric 100 2024-03-21T10:25:25.310 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-21T10:25:25.311 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-21T10:25:25.311 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-21T10:25:25.320 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-21T10:25:25.320 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-21T10:25:25.320 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-21T10:25:25.320 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-21T10:25:25.320 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:caba/64 scope link noprefixroute 2024-03-21T10:25:25.320 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-21T10:25:25.322 INFO:teuthology.orchestra.run.smithi031.stdout:Mon IP `172.21.15.31` is in CIDR network `172.21.0.0/20` 2024-03-21T10:25:25.323 INFO:teuthology.orchestra.run.smithi031.stdout:Mon IP `172.21.15.31` is in CIDR network `172.21.0.0/20` 2024-03-21T10:25:25.323 INFO:teuthology.orchestra.run.smithi031.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-21T10:25:25.325 INFO:teuthology.orchestra.run.smithi031.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-21T10:25:25.326 INFO:teuthology.orchestra.run.smithi031.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11... 2024-03-21T10:25:29.555 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stdout 237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 2024-03-21T10:25:29.555 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11... 2024-03-21T10:25:29.555 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Getting image source signatures 2024-03-21T10:25:29.555 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Copying blob sha256:96a207f569d15e36f8e9e3b2feafd8af7b93861f24c2b60ada6c9667dd5405be 2024-03-21T10:25:29.555 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-21T10:25:29.555 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Copying config sha256:237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 2024-03-21T10:25:29.555 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-21T10:25:30.829 INFO:teuthology.orchestra.run.smithi031.stdout:ceph: stdout ceph version 19.0.0-2231-g2edfee68 (2edfee68ad7243bed422a049b6f5f58619300f11) squid (dev) 2024-03-21T10:25:30.830 INFO:teuthology.orchestra.run.smithi031.stdout:Ceph version: ceph version 19.0.0-2231-g2edfee68 (2edfee68ad7243bed422a049b6f5f58619300f11) squid (dev) 2024-03-21T10:25:30.830 INFO:teuthology.orchestra.run.smithi031.stdout:Extracting ceph user uid/gid from container image... 2024-03-21T10:25:32.076 INFO:teuthology.orchestra.run.smithi031.stdout:stat: stdout 167 167 2024-03-21T10:25:32.077 INFO:teuthology.orchestra.run.smithi031.stdout:Creating initial keys... 2024-03-21T10:25:33.455 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph-authtool: stdout AQAcC/xly2skJRAAP7sfY8XZHv9K+kbHkoujAA== 2024-03-21T10:25:34.615 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph-authtool: stdout AQAdC/xlDpYCORAA+dHrZi+ELuwuWfgDIZUohg== 2024-03-21T10:25:36.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph-authtool: stdout AQAfC/xl/WBoGRAAhZ8c4tD5TU4ubyunWd7W6w== 2024-03-21T10:25:36.250 INFO:teuthology.orchestra.run.smithi031.stdout:Creating initial monmap... 2024-03-21T10:25:37.839 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-21T10:25:37.839 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-21T10:25:37.840 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 3847408c-e76d-11ee-95cd-87774f69a715 2024-03-21T10:25:37.840 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-21T10:25:37.840 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool for a [v2:172.21.15.31:3300,v1:172.21.15.31:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-21T10:25:37.840 INFO:teuthology.orchestra.run.smithi031.stdout:setting min_mon_release = quincy 2024-03-21T10:25:37.841 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: set fsid to 3847408c-e76d-11ee-95cd-87774f69a715 2024-03-21T10:25:37.841 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-21T10:25:37.841 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:25:37.841 INFO:teuthology.orchestra.run.smithi031.stdout:Creating mon... 2024-03-21T10:25:39.070 INFO:teuthology.orchestra.run.smithi031.stdout:create mon.a on 2024-03-21T10:25:39.474 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-21T10:25:39.721 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-21T10:25:40.003 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-3847408c-e76d-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-3847408c-e76d-11ee-95cd-87774f69a715.target. 2024-03-21T10:25:40.003 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-3847408c-e76d-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-3847408c-e76d-11ee-95cd-87774f69a715.target. 2024-03-21T10:25:40.899 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-3847408c-e76d-11ee-95cd-87774f69a715@mon.a 2024-03-21T10:25:40.899 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to reset failed state of unit ceph-3847408c-e76d-11ee-95cd-87774f69a715@mon.a.service: Unit ceph-3847408c-e76d-11ee-95cd-87774f69a715@mon.a.service not loaded. 2024-03-21T10:25:41.170 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-3847408c-e76d-11ee-95cd-87774f69a715.target.wants/ceph-3847408c-e76d-11ee-95cd-87774f69a715@mon.a.service → /etc/systemd/system/ceph-3847408c-e76d-11ee-95cd-87774f69a715@.service. 2024-03-21T10:25:41.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:41 smithi031 podman[31278]: 2024-03-21 10:25:41.449606546 +0000 UTC m=+0.032897138 image pull 237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 2024-03-21T10:25:41.993 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-21T10:25:41.993 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2024-03-21T10:25:42.008 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-21T10:25:42.008 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-03-21T10:25:42.008 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2024-03-21T10:25:42.008 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to enable service . firewalld.service is not available 2024-03-21T10:25:42.009 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mon to start... 2024-03-21T10:25:42.009 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mon... 2024-03-21T10:25:42.156 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:41 smithi031 podman[31278]: 2024-03-21 10:25:41.879796961 +0000 UTC m=+0.463087594 container init 47270f9ce54e9a06fe3561adec16ad59274970b50b4f179846be1b26f8346432 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-21T10:25:42.157 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:41 smithi031 podman[31278]: 2024-03-21 10:25:41.890555406 +0000 UTC m=+0.473846040 container start 47270f9ce54e9a06fe3561adec16ad59274970b50b4f179846be1b26f8346432 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, ceph=True) 2024-03-21T10:25:42.157 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:41 smithi031 bash[31278]: 47270f9ce54e9a06fe3561adec16ad59274970b50b4f179846be1b26f8346432 2024-03-21T10:25:42.157 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:41 smithi031 systemd[1]: Started Ceph mon.a for 3847408c-e76d-11ee-95cd-87774f69a715. 2024-03-21T10:25:42.157 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:41 smithi031 ceph-mon[31297]: mkfs 3847408c-e76d-11ee-95cd-87774f69a715 2024-03-21T10:25:42.157 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:41 smithi031 ceph-mon[31297]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-21T10:25:43.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:42 smithi031 ceph-mon[31297]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-21T10:25:43.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:42 smithi031 ceph-mon[31297]: monmap e1: 1 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T10:25:43.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:42 smithi031 ceph-mon[31297]: fsmap 2024-03-21T10:25:43.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:42 smithi031 ceph-mon[31297]: osdmap e1: 0 total, 0 up, 0 in 2024-03-21T10:25:43.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:42 smithi031 ceph-mon[31297]: mgrmap e1: no daemons active 2024-03-21T10:25:43.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:42 smithi031 ceph-mon[31297]: from='client.? 172.21.15.31:0/3602218933' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T10:25:43.541 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout cluster: 2024-03-21T10:25:43.547 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout id: 3847408c-e76d-11ee-95cd-87774f69a715 2024-03-21T10:25:43.547 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-21T10:25:43.548 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-03-21T10:25:43.548 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout services: 2024-03-21T10:25:43.548 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.726011s) 2024-03-21T10:25:43.548 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-21T10:25:43.548 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-21T10:25:43.548 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-03-21T10:25:43.548 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout data: 2024-03-21T10:25:43.548 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-21T10:25:43.548 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-21T10:25:43.548 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-21T10:25:43.548 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout pgs: 2024-03-21T10:25:43.548 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-03-21T10:25:43.549 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.600+0000 7f457c600700 1 Processor -- start 2024-03-21T10:25:43.549 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.600+0000 7f457c600700 1 -- start start 2024-03-21T10:25:43.549 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.601+0000 7f457c600700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4574108580 0x7f457410a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:25:43.549 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.601+0000 7f457c600700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457407be10 con 0x7f4574108580 2024-03-21T10:25:43.549 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.602+0000 7f4579c00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4574108580 0x7f457410a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:25:43.549 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.602+0000 7f4579c00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4574108580 0x7f457410a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40586/0 (socket says 172.21.15.31:40586) 2024-03-21T10:25:43.549 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.602+0000 7f4579c00700 1 -- 172.21.15.31:0/3639436463 learned_addr learned my addr 172.21.15.31:0/3639436463 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:25:43.549 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.603+0000 7f4579c00700 1 -- 172.21.15.31:0/3639436463 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457410aeb0 con 0x7f4574108580 2024-03-21T10:25:43.549 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.603+0000 7f4579c00700 1 --2- 172.21.15.31:0/3639436463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4574108580 0x7f457410a970 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f45680098a0 tx=0x7f4568009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=132e0b589b1de668 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:25:43.549 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.603+0000 7f4573e00700 1 -- 172.21.15.31:0/3639436463 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4568004160 con 0x7f4574108580 2024-03-21T10:25:43.549 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.603+0000 7f4573e00700 1 -- 172.21.15.31:0/3639436463 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4568004300 con 0x7f4574108580 2024-03-21T10:25:43.549 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.603+0000 7f4573e00700 1 -- 172.21.15.31:0/3639436463 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f45680036a0 con 0x7f4574108580 2024-03-21T10:25:43.549 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.604+0000 7f457c600700 1 -- 172.21.15.31:0/3639436463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4574108580 msgr2=0x7f457410a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:25:43.550 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.604+0000 7f457c600700 1 --2- 172.21.15.31:0/3639436463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4574108580 0x7f457410a970 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f45680098a0 tx=0x7f4568009bb0 comp rx=0 tx=0).stop 2024-03-21T10:25:43.550 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.604+0000 7f457c600700 1 -- 172.21.15.31:0/3639436463 shutdown_connections 2024-03-21T10:25:43.550 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.604+0000 7f457c600700 1 --2- 172.21.15.31:0/3639436463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4574108580 0x7f457410a970 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:25:43.550 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.604+0000 7f457c600700 1 -- 172.21.15.31:0/3639436463 >> 172.21.15.31:0/3639436463 conn(0x7f4574101f30 msgr2=0x7f4574104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:25:43.550 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.605+0000 7f457c600700 1 -- 172.21.15.31:0/3639436463 shutdown_connections 2024-03-21T10:25:43.550 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.605+0000 7f457c600700 1 -- 172.21.15.31:0/3639436463 wait complete. 2024-03-21T10:25:43.550 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.606+0000 7f457c600700 1 Processor -- start 2024-03-21T10:25:43.550 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.606+0000 7f457c600700 1 -- start start 2024-03-21T10:25:43.550 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.606+0000 7f457c600700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4574108580 0x7f457419b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:25:43.550 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.606+0000 7f457c600700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457410b320 con 0x7f4574108580 2024-03-21T10:25:43.550 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.606+0000 7f4579c00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4574108580 0x7f457419b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:25:43.550 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.606+0000 7f4579c00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4574108580 0x7f457419b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40590/0 (socket says 172.21.15.31:40590) 2024-03-21T10:25:43.550 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.606+0000 7f4579c00700 1 -- 172.21.15.31:0/3602218933 learned_addr learned my addr 172.21.15.31:0/3602218933 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:25:43.551 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.607+0000 7f4579c00700 1 -- 172.21.15.31:0/3602218933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457419b720 con 0x7f4574108580 2024-03-21T10:25:43.551 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.607+0000 7f4579c00700 1 --2- 172.21.15.31:0/3602218933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4574108580 0x7f457419b1e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4568000c00 tx=0x7f4568004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:25:43.551 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.607+0000 7f4572000700 1 -- 172.21.15.31:0/3602218933 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4568003db0 con 0x7f4574108580 2024-03-21T10:25:43.551 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.607+0000 7f4572000700 1 -- 172.21.15.31:0/3602218933 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4568011630 con 0x7f4574108580 2024-03-21T10:25:43.551 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.607+0000 7f4572000700 1 -- 172.21.15.31:0/3602218933 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4568011950 con 0x7f4574108580 2024-03-21T10:25:43.551 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.607+0000 7f457c600700 1 -- 172.21.15.31:0/3602218933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f457419b9b0 con 0x7f4574108580 2024-03-21T10:25:43.551 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.608+0000 7f457c600700 1 -- 172.21.15.31:0/3602218933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f457419e6a0 con 0x7f4574108580 2024-03-21T10:25:43.551 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.608+0000 7f4572000700 1 -- 172.21.15.31:0/3602218933 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f456801a460 con 0x7f4574108580 2024-03-21T10:25:43.551 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.608+0000 7f4572000700 1 -- 172.21.15.31:0/3602218933 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f456801b780 con 0x7f4574108580 2024-03-21T10:25:43.551 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.609+0000 7f457c600700 1 -- 172.21.15.31:0/3602218933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45740637c0 con 0x7f4574108580 2024-03-21T10:25:43.551 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.613+0000 7f4572000700 1 -- 172.21.15.31:0/3602218933 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f456801ba20 con 0x7f4574108580 2024-03-21T10:25:43.551 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.693+0000 7f457c600700 1 -- 172.21.15.31:0/3602218933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f457402dd30 con 0x7f4574108580 2024-03-21T10:25:43.551 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.694+0000 7f4572000700 1 -- 172.21.15.31:0/3602218933 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f456801bc00 con 0x7f4574108580 2024-03-21T10:25:43.552 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.695+0000 7f457c600700 1 -- 172.21.15.31:0/3602218933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4574108580 msgr2=0x7f457419b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:25:43.552 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.695+0000 7f457c600700 1 --2- 172.21.15.31:0/3602218933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4574108580 0x7f457419b1e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4568000c00 tx=0x7f4568004750 comp rx=0 tx=0).stop 2024-03-21T10:25:43.552 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.695+0000 7f457c600700 1 -- 172.21.15.31:0/3602218933 shutdown_connections 2024-03-21T10:25:43.552 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.695+0000 7f457c600700 1 --2- 172.21.15.31:0/3602218933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4574108580 0x7f457419b1e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:25:43.552 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.695+0000 7f457c600700 1 -- 172.21.15.31:0/3602218933 >> 172.21.15.31:0/3602218933 conn(0x7f4574101f30 msgr2=0x7f45741042b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:25:43.552 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.696+0000 7f457c600700 1 -- 172.21.15.31:0/3602218933 shutdown_connections 2024-03-21T10:25:43.552 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:42.696+0000 7f457c600700 1 -- 172.21.15.31:0/3602218933 wait complete. 2024-03-21T10:25:43.552 INFO:teuthology.orchestra.run.smithi031.stdout:mon is available 2024-03-21T10:25:43.552 INFO:teuthology.orchestra.run.smithi031.stdout:Assimilating anything we can from ceph.conf... 2024-03-21T10:25:44.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:44 smithi031 ceph-mon[31297]: from='client.? 172.21.15.31:0/3140376254' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-21T10:25:44.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:44 smithi031 ceph-mon[31297]: from='client.? 172.21.15.31:0/3140376254' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-21T10:25:44.930 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-03-21T10:25:44.930 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [global] 2024-03-21T10:25:44.930 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout fsid = 3847408c-e76d-11ee-95cd-87774f69a715 2024-03-21T10:25:44.930 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.31:3300,v1:172.21.15.31:6789] 2024-03-21T10:25:44.930 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-21T10:25:44.930 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-21T10:25:44.930 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-21T10:25:44.930 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-21T10:25:44.930 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-03-21T10:25:44.931 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-21T10:25:44.931 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-21T10:25:44.931 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-03-21T10:25:44.931 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [osd] 2024-03-21T10:25:44.931 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-21T10:25:44.931 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-21T10:25:44.931 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-21T10:25:44.931 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-21T10:25:44.931 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-21T10:25:44.931 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.159+0000 7fe469000700 1 Processor -- start 2024-03-21T10:25:44.931 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.159+0000 7fe469000700 1 -- start start 2024-03-21T10:25:44.931 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.159+0000 7fe469000700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe464109aa0 0x7fe464109ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:25:44.931 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.159+0000 7fe469000700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46410a490 con 0x7fe464109aa0 2024-03-21T10:25:44.931 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.160+0000 7fe463400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe464109aa0 0x7fe464109ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:25:44.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.160+0000 7fe463400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe464109aa0 0x7fe464109ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40598/0 (socket says 172.21.15.31:40598) 2024-03-21T10:25:44.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.160+0000 7fe463400700 1 -- 172.21.15.31:0/3874405248 learned_addr learned my addr 172.21.15.31:0/3874405248 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:25:44.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.160+0000 7fe463400700 1 -- 172.21.15.31:0/3874405248 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe46410aca0 con 0x7fe464109aa0 2024-03-21T10:25:44.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.161+0000 7fe463400700 1 --2- 172.21.15.31:0/3874405248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe464109aa0 0x7fe464109ec0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe4540098a0 tx=0x7fe454009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab2f5bf6b7c27589 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:25:44.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.161+0000 7fe462000700 1 -- 172.21.15.31:0/3874405248 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe454004160 con 0x7fe464109aa0 2024-03-21T10:25:44.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.161+0000 7fe462000700 1 -- 172.21.15.31:0/3874405248 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe454004300 con 0x7fe464109aa0 2024-03-21T10:25:44.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.161+0000 7fe462000700 1 -- 172.21.15.31:0/3874405248 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe4540036a0 con 0x7fe464109aa0 2024-03-21T10:25:44.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.162+0000 7fe469000700 1 -- 172.21.15.31:0/3874405248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe464109aa0 msgr2=0x7fe464109ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:25:44.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.162+0000 7fe469000700 1 --2- 172.21.15.31:0/3874405248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe464109aa0 0x7fe464109ec0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe4540098a0 tx=0x7fe454009bb0 comp rx=0 tx=0).stop 2024-03-21T10:25:44.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.162+0000 7fe469000700 1 -- 172.21.15.31:0/3874405248 shutdown_connections 2024-03-21T10:25:44.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.162+0000 7fe469000700 1 --2- 172.21.15.31:0/3874405248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe464109aa0 0x7fe464109ec0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:25:44.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.162+0000 7fe469000700 1 -- 172.21.15.31:0/3874405248 >> 172.21.15.31:0/3874405248 conn(0x7fe46407cee0 msgr2=0x7fe46407d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:25:44.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.162+0000 7fe469000700 1 -- 172.21.15.31:0/3874405248 shutdown_connections 2024-03-21T10:25:44.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.162+0000 7fe469000700 1 -- 172.21.15.31:0/3874405248 wait complete. 2024-03-21T10:25:44.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.163+0000 7fe469000700 1 Processor -- start 2024-03-21T10:25:44.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.163+0000 7fe469000700 1 -- start start 2024-03-21T10:25:44.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.163+0000 7fe469000700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe464109aa0 0x7fe46419d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:25:44.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.163+0000 7fe469000700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46410b1d0 con 0x7fe464109aa0 2024-03-21T10:25:44.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.164+0000 7fe463400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe464109aa0 0x7fe46419d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:25:44.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.164+0000 7fe463400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe464109aa0 0x7fe46419d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40610/0 (socket says 172.21.15.31:40610) 2024-03-21T10:25:44.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.164+0000 7fe463400700 1 -- 172.21.15.31:0/3140376254 learned_addr learned my addr 172.21.15.31:0/3140376254 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:25:44.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.164+0000 7fe463400700 1 -- 172.21.15.31:0/3140376254 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe46419d6e0 con 0x7fe464109aa0 2024-03-21T10:25:44.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.165+0000 7fe463400700 1 --2- 172.21.15.31:0/3140376254 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe464109aa0 0x7fe46419d1a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe454000c00 tx=0x7fe454004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:25:44.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.165+0000 7fe45be00700 1 -- 172.21.15.31:0/3140376254 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe454003db0 con 0x7fe464109aa0 2024-03-21T10:25:44.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.165+0000 7fe45be00700 1 -- 172.21.15.31:0/3140376254 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe454011630 con 0x7fe464109aa0 2024-03-21T10:25:44.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.165+0000 7fe45be00700 1 -- 172.21.15.31:0/3140376254 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe454011950 con 0x7fe464109aa0 2024-03-21T10:25:44.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.165+0000 7fe469000700 1 -- 172.21.15.31:0/3140376254 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe46419d970 con 0x7fe464109aa0 2024-03-21T10:25:44.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.165+0000 7fe469000700 1 -- 172.21.15.31:0/3140376254 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4641a0660 con 0x7fe464109aa0 2024-03-21T10:25:44.934 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.166+0000 7fe45be00700 1 -- 172.21.15.31:0/3140376254 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fe45401a460 con 0x7fe464109aa0 2024-03-21T10:25:44.934 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.166+0000 7fe45be00700 1 -- 172.21.15.31:0/3140376254 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe45401b780 con 0x7fe464109aa0 2024-03-21T10:25:44.934 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.167+0000 7fe469000700 1 -- 172.21.15.31:0/3140376254 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4640637c0 con 0x7fe464109aa0 2024-03-21T10:25:44.934 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.169+0000 7fe45be00700 1 -- 172.21.15.31:0/3140376254 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fe454031020 con 0x7fe464109aa0 2024-03-21T10:25:44.934 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.246+0000 7fe469000700 1 -- 172.21.15.31:0/3140376254 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fe46402dc80 con 0x7fe464109aa0 2024-03-21T10:25:44.934 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.255+0000 7fe45be00700 1 -- 172.21.15.31:0/3140376254 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7fe45401b9d0 con 0x7fe464109aa0 2024-03-21T10:25:44.934 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.257+0000 7fe469000700 1 -- 172.21.15.31:0/3140376254 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe464109aa0 msgr2=0x7fe46419d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:25:44.934 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.257+0000 7fe469000700 1 --2- 172.21.15.31:0/3140376254 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe464109aa0 0x7fe46419d1a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe454000c00 tx=0x7fe454004750 comp rx=0 tx=0).stop 2024-03-21T10:25:44.934 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.257+0000 7fe469000700 1 -- 172.21.15.31:0/3140376254 shutdown_connections 2024-03-21T10:25:44.934 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.257+0000 7fe469000700 1 --2- 172.21.15.31:0/3140376254 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe464109aa0 0x7fe46419d1a0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:25:44.934 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.257+0000 7fe469000700 1 -- 172.21.15.31:0/3140376254 >> 172.21.15.31:0/3140376254 conn(0x7fe46407cee0 msgr2=0x7fe464193db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:25:44.934 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.257+0000 7fe469000700 1 -- 172.21.15.31:0/3140376254 shutdown_connections 2024-03-21T10:25:44.934 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:44.257+0000 7fe469000700 1 -- 172.21.15.31:0/3140376254 wait complete. 2024-03-21T10:25:44.934 INFO:teuthology.orchestra.run.smithi031.stdout:Generating new minimal ceph.conf... 2024-03-21T10:25:46.058 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:45 smithi031 ceph-mon[31297]: from='client.? 172.21.15.31:0/313833997' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:25:47.385 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.562+0000 7fd030c00700 1 Processor -- start 2024-03-21T10:25:47.390 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.562+0000 7fd030c00700 1 -- start start 2024-03-21T10:25:47.390 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.563+0000 7fd030c00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd02c106990 0x7fd02c106db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:25:47.390 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.563+0000 7fd030c00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c107380 con 0x7fd02c106990 2024-03-21T10:25:47.390 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.564+0000 7fd029e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd02c106990 0x7fd02c106db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:25:47.390 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.564+0000 7fd029e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd02c106990 0x7fd02c106db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40624/0 (socket says 172.21.15.31:40624) 2024-03-21T10:25:47.390 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.564+0000 7fd029e00700 1 -- 172.21.15.31:0/164283460 learned_addr learned my addr 172.21.15.31:0/164283460 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:25:47.390 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.564+0000 7fd029e00700 1 -- 172.21.15.31:0/164283460 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd02c107b90 con 0x7fd02c106990 2024-03-21T10:25:47.390 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.565+0000 7fd029e00700 1 --2- 172.21.15.31:0/164283460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd02c106990 0x7fd02c106db0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd01c0098a0 tx=0x7fd01c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74b61dbd6dde513d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:25:47.390 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.565+0000 7fd028a00700 1 -- 172.21.15.31:0/164283460 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd01c004160 con 0x7fd02c106990 2024-03-21T10:25:47.390 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.565+0000 7fd028a00700 1 -- 172.21.15.31:0/164283460 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd01c004300 con 0x7fd02c106990 2024-03-21T10:25:47.391 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.565+0000 7fd028a00700 1 -- 172.21.15.31:0/164283460 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd01c011b50 con 0x7fd02c106990 2024-03-21T10:25:47.391 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.566+0000 7fd030c00700 1 -- 172.21.15.31:0/164283460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd02c106990 msgr2=0x7fd02c106db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:25:47.391 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.566+0000 7fd030c00700 1 --2- 172.21.15.31:0/164283460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd02c106990 0x7fd02c106db0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd01c0098a0 tx=0x7fd01c009bb0 comp rx=0 tx=0).stop 2024-03-21T10:25:47.391 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.566+0000 7fd030c00700 1 -- 172.21.15.31:0/164283460 shutdown_connections 2024-03-21T10:25:47.391 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.566+0000 7fd030c00700 1 --2- 172.21.15.31:0/164283460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd02c106990 0x7fd02c106db0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:25:47.391 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.566+0000 7fd030c00700 1 -- 172.21.15.31:0/164283460 >> 172.21.15.31:0/164283460 conn(0x7fd02c101f30 msgr2=0x7fd02c104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:25:47.391 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.566+0000 7fd030c00700 1 -- 172.21.15.31:0/164283460 shutdown_connections 2024-03-21T10:25:47.391 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.566+0000 7fd030c00700 1 -- 172.21.15.31:0/164283460 wait complete. 2024-03-21T10:25:47.391 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.567+0000 7fd030c00700 1 Processor -- start 2024-03-21T10:25:47.391 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.567+0000 7fd030c00700 1 -- start start 2024-03-21T10:25:47.391 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.568+0000 7fd030c00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd02c106990 0x7fd02c19bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:25:47.391 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.568+0000 7fd030c00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c10af40 con 0x7fd02c106990 2024-03-21T10:25:47.391 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.568+0000 7fd029e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd02c106990 0x7fd02c19bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:25:47.391 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.568+0000 7fd029e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd02c106990 0x7fd02c19bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40630/0 (socket says 172.21.15.31:40630) 2024-03-21T10:25:47.392 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.568+0000 7fd029e00700 1 -- 172.21.15.31:0/313833997 learned_addr learned my addr 172.21.15.31:0/313833997 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:25:47.392 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.569+0000 7fd029e00700 1 -- 172.21.15.31:0/313833997 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd02c19c050 con 0x7fd02c106990 2024-03-21T10:25:47.392 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.569+0000 7fd029e00700 1 --2- 172.21.15.31:0/313833997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd02c106990 0x7fd02c19bb10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd01c009fb0 tx=0x7fd01c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:25:47.392 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.569+0000 7fd022a00700 1 -- 172.21.15.31:0/313833997 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd01c0038b0 con 0x7fd02c106990 2024-03-21T10:25:47.392 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.569+0000 7fd022a00700 1 -- 172.21.15.31:0/313833997 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd01c011e50 con 0x7fd02c106990 2024-03-21T10:25:47.392 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.569+0000 7fd022a00700 1 -- 172.21.15.31:0/313833997 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd01c019550 con 0x7fd02c106990 2024-03-21T10:25:47.392 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.569+0000 7fd030c00700 1 -- 172.21.15.31:0/313833997 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd02c19c2e0 con 0x7fd02c106990 2024-03-21T10:25:47.392 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.570+0000 7fd030c00700 1 -- 172.21.15.31:0/313833997 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd02c19efd0 con 0x7fd02c106990 2024-03-21T10:25:47.392 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.570+0000 7fd022a00700 1 -- 172.21.15.31:0/313833997 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fd01c019b90 con 0x7fd02c106990 2024-03-21T10:25:47.392 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.570+0000 7fd022a00700 1 -- 172.21.15.31:0/313833997 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd01c024730 con 0x7fd02c106990 2024-03-21T10:25:47.392 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.571+0000 7fd030c00700 1 -- 172.21.15.31:0/313833997 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd02c0637c0 con 0x7fd02c106990 2024-03-21T10:25:47.392 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.574+0000 7fd022a00700 1 -- 172.21.15.31:0/313833997 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fd01c003a50 con 0x7fd02c106990 2024-03-21T10:25:47.392 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.651+0000 7fd030c00700 1 -- 172.21.15.31:0/313833997 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fd02c02dd20 con 0x7fd02c106990 2024-03-21T10:25:47.392 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.652+0000 7fd022a00700 1 -- 172.21.15.31:0/313833997 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fd01c0249d0 con 0x7fd02c106990 2024-03-21T10:25:47.392 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.653+0000 7fd030c00700 1 -- 172.21.15.31:0/313833997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd02c106990 msgr2=0x7fd02c19bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:25:47.393 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.653+0000 7fd030c00700 1 --2- 172.21.15.31:0/313833997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd02c106990 0x7fd02c19bb10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd01c009fb0 tx=0x7fd01c011bf0 comp rx=0 tx=0).stop 2024-03-21T10:25:47.393 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.654+0000 7fd030c00700 1 -- 172.21.15.31:0/313833997 shutdown_connections 2024-03-21T10:25:47.393 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.654+0000 7fd030c00700 1 --2- 172.21.15.31:0/313833997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd02c106990 0x7fd02c19bb10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:25:47.393 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.654+0000 7fd030c00700 1 -- 172.21.15.31:0/313833997 >> 172.21.15.31:0/313833997 conn(0x7fd02c101f30 msgr2=0x7fd02c102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:25:47.393 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.654+0000 7fd030c00700 1 -- 172.21.15.31:0/313833997 shutdown_connections 2024-03-21T10:25:47.393 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:45.654+0000 7fd030c00700 1 -- 172.21.15.31:0/313833997 wait complete. 2024-03-21T10:25:47.393 INFO:teuthology.orchestra.run.smithi031.stdout:Restarting the monitor... 2024-03-21T10:25:47.770 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:47 smithi031 systemd[1]: Stopping Ceph mon.a for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:25:48.058 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:47 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-a[31293]: 2024-03-21T10:25:47.916+0000 7fd659800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:25:48.058 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:47 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-a[31293]: 2024-03-21T10:25:47.916+0000 7fd659800700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-21T10:25:48.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:48 smithi031 podman[31541]: 2024-03-21 10:25:48.143213584 +0000 UTC m=+0.411627250 container died 47270f9ce54e9a06fe3561adec16ad59274970b50b4f179846be1b26f8346432 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-a, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-21T10:25:49.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:48 smithi031 podman[31541]: 2024-03-21 10:25:48.719098442 +0000 UTC m=+0.987512126 container cleanup 47270f9ce54e9a06fe3561adec16ad59274970b50b4f179846be1b26f8346432 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-a, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD) 2024-03-21T10:25:49.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:48 smithi031 bash[31541]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-a 2024-03-21T10:25:49.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:49 smithi031 podman[31563]: 2024-03-21 10:25:49.053054598 +0000 UTC m=+0.892757415 container remove 47270f9ce54e9a06fe3561adec16ad59274970b50b4f179846be1b26f8346432 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-a, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-21T10:25:49.709 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:49 smithi031 systemd[1]: ceph-3847408c-e76d-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-21T10:25:49.709 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:49 smithi031 systemd[1]: Stopped Ceph mon.a for 3847408c-e76d-11ee-95cd-87774f69a715. 2024-03-21T10:25:49.709 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:49 smithi031 systemd[1]: ceph-3847408c-e76d-11ee-95cd-87774f69a715@mon.a.service: Consumed 1.808s CPU time. 2024-03-21T10:25:49.709 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:49 smithi031 systemd[1]: Starting Ceph mon.a for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:25:50.002 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:49 smithi031 podman[31655]: 2024-03-21 10:25:49.616175023 +0000 UTC m=+0.040597939 image pull 237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 2024-03-21T10:25:50.002 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:49 smithi031 podman[31655]: 2024-03-21 10:25:49.729166301 +0000 UTC m=+0.153589249 container create aeb8a37c34e6eca49d0cb7d648fd98e198456cb3a6bb14e092ee36f2b2d41433 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-a, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T10:25:50.002 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:49 smithi031 podman[31655]: 2024-03-21 10:25:49.940902772 +0000 UTC m=+0.365325727 container init aeb8a37c34e6eca49d0cb7d648fd98e198456cb3a6bb14e092ee36f2b2d41433 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T10:25:50.002 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:49 smithi031 podman[31655]: 2024-03-21 10:25:49.952175533 +0000 UTC m=+0.376598484 container start aeb8a37c34e6eca49d0cb7d648fd98e198456cb3a6bb14e092ee36f2b2d41433 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-21T10:25:50.002 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: set uid:gid to 167:167 (ceph:ceph) 2024-03-21T10:25:50.002 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: ceph version 19.0.0-2231-g2edfee68 (2edfee68ad7243bed422a049b6f5f58619300f11) squid (dev), process ceph-mon, pid 2 2024-03-21T10:25:50.002 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: pidfile_write: ignore empty --pid-file 2024-03-21T10:25:50.027 INFO:teuthology.orchestra.run.smithi031.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-21T10:25:50.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: load: jerasure load: lrc 2024-03-21T10:25:50.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: RocksDB version: 7.9.2 2024-03-21T10:25:50.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Git sha 0 2024-03-21T10:25:50.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Compile date 2024-03-20 06:23:48 2024-03-21T10:25:50.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: DB SUMMARY 2024-03-21T10:25:50.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: DB Session ID: B3JAAE9QUMJT9XBO8ZIX 2024-03-21T10:25:50.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: CURRENT file: CURRENT 2024-03-21T10:25:50.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: IDENTITY file: IDENTITY 2024-03-21T10:25:50.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-21T10:25:50.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-21T10:25:50.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 100066 ; 2024-03-21T10:25:50.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.error_if_exists: 0 2024-03-21T10:25:50.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.create_if_missing: 0 2024-03-21T10:25:50.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.paranoid_checks: 1 2024-03-21T10:25:50.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-21T10:25:50.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-21T10:25:50.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-21T10:25:50.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.env: 0x55db5b5d9b20 2024-03-21T10:25:50.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.fs: PosixFileSystem 2024-03-21T10:25:50.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.info_log: 0x55db5db95960 2024-03-21T10:25:50.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.max_file_opening_threads: 16 2024-03-21T10:25:50.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.statistics: (nil) 2024-03-21T10:25:50.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.use_fsync: 0 2024-03-21T10:25:50.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.max_log_file_size: 0 2024-03-21T10:25:50.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-21T10:25:50.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-21T10:25:50.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.keep_log_file_num: 1000 2024-03-21T10:25:50.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.recycle_log_file_num: 0 2024-03-21T10:25:50.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.allow_fallocate: 1 2024-03-21T10:25:50.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.allow_mmap_reads: 0 2024-03-21T10:25:50.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.allow_mmap_writes: 0 2024-03-21T10:25:50.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.use_direct_reads: 0 2024-03-21T10:25:50.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-21T10:25:50.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.create_missing_column_families: 0 2024-03-21T10:25:50.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.db_log_dir: 2024-03-21T10:25:50.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.wal_dir: 2024-03-21T10:25:50.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-21T10:25:50.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-21T10:25:50.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-21T10:25:50.314 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-21T10:25:50.314 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-21T10:25:50.314 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-21T10:25:50.314 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.advise_random_on_open: 1 2024-03-21T10:25:50.314 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.db_write_buffer_size: 0 2024-03-21T10:25:50.314 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.write_buffer_manager: 0x55db5db681e0 2024-03-21T10:25:50.314 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-21T10:25:50.314 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-21T10:25:50.314 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-21T10:25:50.315 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.rate_limiter: (nil) 2024-03-21T10:25:50.315 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-21T10:25:50.315 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.wal_recovery_mode: 2 2024-03-21T10:25:50.315 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.enable_thread_tracking: 0 2024-03-21T10:25:50.315 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.enable_pipelined_write: 0 2024-03-21T10:25:50.315 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.unordered_write: 0 2024-03-21T10:25:50.315 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-21T10:25:50.315 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-21T10:25:50.315 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-21T10:25:50.315 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-21T10:25:50.315 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.row_cache: None 2024-03-21T10:25:50.315 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.wal_filter: None 2024-03-21T10:25:50.315 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-21T10:25:50.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.allow_ingest_behind: 0 2024-03-21T10:25:50.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.two_write_queues: 0 2024-03-21T10:25:50.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.manual_wal_flush: 0 2024-03-21T10:25:50.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.wal_compression: 0 2024-03-21T10:25:50.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.atomic_flush: 0 2024-03-21T10:25:50.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-21T10:25:50.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-21T10:25:50.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-21T10:25:50.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.log_readahead_size: 0 2024-03-21T10:25:50.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-21T10:25:50.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.best_efforts_recovery: 0 2024-03-21T10:25:50.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-21T10:25:50.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-21T10:25:50.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.allow_data_in_errors: 0 2024-03-21T10:25:50.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.db_host_id: __hostname__ 2024-03-21T10:25:50.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-21T10:25:50.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.max_background_jobs: 2 2024-03-21T10:25:50.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.max_background_compactions: -1 2024-03-21T10:25:50.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.max_subcompactions: 1 2024-03-21T10:25:50.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-21T10:25:50.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-21T10:25:50.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-21T10:25:50.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.max_total_wal_size: 0 2024-03-21T10:25:50.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-21T10:25:50.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-21T10:25:50.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-21T10:25:50.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-21T10:25:50.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.max_open_files: -1 2024-03-21T10:25:50.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.bytes_per_sync: 0 2024-03-21T10:25:50.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-21T10:25:50.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-21T10:25:50.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.compaction_readahead_size: 0 2024-03-21T10:25:50.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.max_background_flushes: -1 2024-03-21T10:25:50.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Compression algorithms supported: 2024-03-21T10:25:50.319 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-21T10:25:50.319 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: kZSTD supported: 0 2024-03-21T10:25:50.319 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 bash[31655]: aeb8a37c34e6eca49d0cb7d648fd98e198456cb3a6bb14e092ee36f2b2d41433 2024-03-21T10:25:50.319 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: kXpressCompression supported: 0 2024-03-21T10:25:50.319 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: kLZ4HCCompression supported: 1 2024-03-21T10:25:50.319 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: kZlibCompression supported: 1 2024-03-21T10:25:50.319 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: kSnappyCompression supported: 1 2024-03-21T10:25:50.319 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: kLZ4Compression supported: 1 2024-03-21T10:25:50.319 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: kBZip2Compression supported: 0 2024-03-21T10:25:50.319 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-21T10:25:50.319 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-21T10:25:50.319 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-21T10:25:50.319 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-21T10:25:50.319 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-21T10:25:50.320 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.merge_operator: 2024-03-21T10:25:50.320 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.compaction_filter: None 2024-03-21T10:25:50.320 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.compaction_filter_factory: None 2024-03-21T10:25:50.320 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.sst_partitioner_factory: None 2024-03-21T10:25:50.320 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-21T10:25:50.320 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-21T10:25:50.320 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55db5db95a60) 2024-03-21T10:25:50.320 INFO:journalctl@ceph.mon.a.smithi031.stdout: cache_index_and_filter_blocks: 1 2024-03-21T10:25:50.320 INFO:journalctl@ceph.mon.a.smithi031.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-21T10:25:50.320 INFO:journalctl@ceph.mon.a.smithi031.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-21T10:25:50.320 INFO:journalctl@ceph.mon.a.smithi031.stdout: pin_top_level_index_and_filter: 1 2024-03-21T10:25:50.320 INFO:journalctl@ceph.mon.a.smithi031.stdout: index_type: 0 2024-03-21T10:25:50.321 INFO:journalctl@ceph.mon.a.smithi031.stdout: data_block_index_type: 0 2024-03-21T10:25:50.321 INFO:journalctl@ceph.mon.a.smithi031.stdout: index_shortening: 1 2024-03-21T10:25:50.321 INFO:journalctl@ceph.mon.a.smithi031.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-21T10:25:50.321 INFO:journalctl@ceph.mon.a.smithi031.stdout: checksum: 4 2024-03-21T10:25:50.321 INFO:journalctl@ceph.mon.a.smithi031.stdout: no_block_cache: 0 2024-03-21T10:25:50.321 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache: 0x55db5db9b090 2024-03-21T10:25:50.321 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache_name: BinnedLRUCache 2024-03-21T10:25:50.321 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache_options: 2024-03-21T10:25:50.321 INFO:journalctl@ceph.mon.a.smithi031.stdout: capacity : 536870912 2024-03-21T10:25:50.321 INFO:journalctl@ceph.mon.a.smithi031.stdout: num_shard_bits : 4 2024-03-21T10:25:50.321 INFO:journalctl@ceph.mon.a.smithi031.stdout: strict_capacity_limit : 0 2024-03-21T10:25:50.321 INFO:journalctl@ceph.mon.a.smithi031.stdout: high_pri_pool_ratio: 0.000 2024-03-21T10:25:50.321 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache_compressed: (nil) 2024-03-21T10:25:50.321 INFO:journalctl@ceph.mon.a.smithi031.stdout: persistent_cache: (nil) 2024-03-21T10:25:50.321 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_size: 4096 2024-03-21T10:25:50.322 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_size_deviation: 10 2024-03-21T10:25:50.322 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_restart_interval: 16 2024-03-21T10:25:50.322 INFO:journalctl@ceph.mon.a.smithi031.stdout: index_block_restart_interval: 1 2024-03-21T10:25:50.322 INFO:journalctl@ceph.mon.a.smithi031.stdout: metadata_block_size: 4096 2024-03-21T10:25:50.322 INFO:journalctl@ceph.mon.a.smithi031.stdout: partition_filters: 0 2024-03-21T10:25:50.322 INFO:journalctl@ceph.mon.a.smithi031.stdout: use_delta_encoding: 1 2024-03-21T10:25:50.322 INFO:journalctl@ceph.mon.a.smithi031.stdout: filter_policy: bloomfilter 2024-03-21T10:25:50.322 INFO:journalctl@ceph.mon.a.smithi031.stdout: whole_key_filtering: 1 2024-03-21T10:25:50.322 INFO:journalctl@ceph.mon.a.smithi031.stdout: verify_compression: 0 2024-03-21T10:25:50.322 INFO:journalctl@ceph.mon.a.smithi031.stdout: read_amp_bytes_per_bit: 0 2024-03-21T10:25:50.322 INFO:journalctl@ceph.mon.a.smithi031.stdout: format_version: 5 2024-03-21T10:25:50.322 INFO:journalctl@ceph.mon.a.smithi031.stdout: enable_index_compression: 1 2024-03-21T10:25:50.322 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_align: 0 2024-03-21T10:25:50.323 INFO:journalctl@ceph.mon.a.smithi031.stdout: max_auto_readahead_size: 262144 2024-03-21T10:25:50.323 INFO:journalctl@ceph.mon.a.smithi031.stdout: prepopulate_block_cache: 0 2024-03-21T10:25:50.323 INFO:journalctl@ceph.mon.a.smithi031.stdout: initial_auto_readahead_size: 8192 2024-03-21T10:25:50.323 INFO:journalctl@ceph.mon.a.smithi031.stdout: num_file_reads_for_auto_readahead: 2 2024-03-21T10:25:50.323 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.write_buffer_size: 33554432 2024-03-21T10:25:50.323 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.max_write_buffer_number: 2 2024-03-21T10:25:50.323 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.compression: NoCompression 2024-03-21T10:25:50.323 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.bottommost_compression: Disabled 2024-03-21T10:25:50.323 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.prefix_extractor: nullptr 2024-03-21T10:25:50.323 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-21T10:25:50.323 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.num_levels: 7 2024-03-21T10:25:50.323 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-21T10:25:50.323 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-21T10:25:50.323 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-21T10:25:50.324 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-21T10:25:50.324 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-21T10:25:50.324 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-21T10:25:50.324 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-21T10:25:50.324 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-21T10:25:50.324 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-21T10:25:50.324 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-21T10:25:50.325 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-21T10:25:50.325 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-21T10:25:50.325 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-21T10:25:50.325 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.compression_opts.level: 32767 2024-03-21T10:25:50.325 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.compression_opts.strategy: 0 2024-03-21T10:25:50.325 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-21T10:25:50.325 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-21T10:25:50.326 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-21T10:25:50.326 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-21T10:25:50.326 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.compression_opts.enabled: false 2024-03-21T10:25:50.326 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-21T10:25:50.326 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-21T10:25:50.326 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-21T10:25:50.326 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-21T10:25:50.326 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.target_file_size_base: 67108864 2024-03-21T10:25:50.326 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-21T10:25:50.326 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-21T10:25:50.326 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-21T10:25:50.326 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-21T10:25:50.327 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-21T10:25:50.327 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-21T10:25:50.327 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-21T10:25:50.327 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-21T10:25:50.327 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-21T10:25:50.327 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-21T10:25:50.327 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-21T10:25:50.327 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-21T10:25:50.327 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-21T10:25:50.327 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-21T10:25:50.327 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.arena_block_size: 1048576 2024-03-21T10:25:50.327 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-21T10:25:50.327 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-21T10:25:50.327 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.disable_auto_compactions: 0 2024-03-21T10:25:50.328 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-21T10:25:50.328 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-21T10:25:50.328 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-21T10:25:50.328 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-21T10:25:50.328 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-21T10:25:50.328 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-21T10:25:50.328 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-21T10:25:50.328 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-21T10:25:50.328 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-21T10:25:50.328 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-21T10:25:50.328 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.table_properties_collectors: 2024-03-21T10:25:50.328 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.inplace_update_support: 0 2024-03-21T10:25:50.328 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-21T10:25:50.328 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-21T10:25:50.329 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-21T10:25:50.329 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-21T10:25:50.329 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.bloom_locality: 0 2024-03-21T10:25:50.329 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.max_successive_merges: 0 2024-03-21T10:25:50.329 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-21T10:25:50.329 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.paranoid_file_checks: 0 2024-03-21T10:25:50.329 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.force_consistency_checks: 1 2024-03-21T10:25:50.329 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.report_bg_io_stats: 0 2024-03-21T10:25:50.329 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.ttl: 2592000 2024-03-21T10:25:50.329 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-21T10:25:50.329 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-21T10:25:50.329 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-21T10:25:50.329 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.enable_blob_files: false 2024-03-21T10:25:50.329 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.min_blob_size: 0 2024-03-21T10:25:50.330 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.blob_file_size: 268435456 2024-03-21T10:25:50.330 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-21T10:25:50.330 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-21T10:25:50.330 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-21T10:25:50.330 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-21T10:25:50.330 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-21T10:25:50.330 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.blob_file_starting_level: 0 2024-03-21T10:25:50.330 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-21T10:25:50.330 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-21T10:25:50.330 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-21T10:25:50.330 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: efbd8e45-018d-4a3a-9adc-56ae6362480b 2024-03-21T10:25:50.330 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711016750011433, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-21T10:25:50.330 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-21T10:25:50.330 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711016750013301, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 95284, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 316, "table_properties": {"data_size": 93283, "index_size": 283, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13771, "raw_average_key_size": 50, "raw_value_size": 85756, "raw_average_value_size": 311, "num_data_blocks": 11, "num_entries": 275, "num_filter_entries": 275, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711016750, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "efbd8e45-018d-4a3a-9adc-56ae6362480b", "db_session_id": "B3JAAE9QUMJT9XBO8ZIX", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-21T10:25:50.330 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711016750013453, "job": 1, "event": "recovery_finished"} 2024-03-21T10:25:50.331 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-21T10:25:50.331 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-21T10:25:50.331 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55db5dc8e000 2024-03-21T10:25:50.331 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: DB pointer 0x55db5dc78000 2024-03-21T10:25:50.331 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-21T10:25:50.331 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-21T10:25:50.331 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** DB Stats ** 2024-03-21T10:25:50.331 INFO:journalctl@ceph.mon.a.smithi031.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T10:25:50.331 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-21T10:25:50.331 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T10:25:50.331 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T10:25:50.331 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-21T10:25:50.331 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T10:25:50.331 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T10:25:50.331 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-03-21T10:25:50.331 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** Compaction Stats [default] ** 2024-03-21T10:25:50.332 INFO:journalctl@ceph.mon.a.smithi031.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T10:25:50.332 INFO:journalctl@ceph.mon.a.smithi031.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-21T10:25:50.332 INFO:journalctl@ceph.mon.a.smithi031.stdout: L0 2/0 94.89 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 75.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-21T10:25:50.332 INFO:journalctl@ceph.mon.a.smithi031.stdout: Sum 2/0 94.89 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 75.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-21T10:25:50.332 INFO:journalctl@ceph.mon.a.smithi031.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 75.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-21T10:25:50.332 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-03-21T10:25:50.332 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** Compaction Stats [default] ** 2024-03-21T10:25:50.332 INFO:journalctl@ceph.mon.a.smithi031.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T10:25:50.332 INFO:journalctl@ceph.mon.a.smithi031.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T10:25:50.332 INFO:journalctl@ceph.mon.a.smithi031.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 75.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-21T10:25:50.332 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-03-21T10:25:50.332 INFO:journalctl@ceph.mon.a.smithi031.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-21T10:25:50.332 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-03-21T10:25:50.332 INFO:journalctl@ceph.mon.a.smithi031.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T10:25:50.333 INFO:journalctl@ceph.mon.a.smithi031.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-21T10:25:50.333 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-21T10:25:50.333 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-21T10:25:50.333 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-21T10:25:50.333 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-21T10:25:50.333 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative compaction: 0.00 GB write, 12.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T10:25:50.333 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval compaction: 0.00 GB write, 12.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T10:25:50.333 INFO:journalctl@ceph.mon.a.smithi031.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-21T10:25:50.333 INFO:journalctl@ceph.mon.a.smithi031.stdout: Block cache BinnedLRUCache@0x55db5db9b090#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-03-21T10:25:50.333 INFO:journalctl@ceph.mon.a.smithi031.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-21T10:25:50.333 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-03-21T10:25:50.333 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-21T10:25:50.333 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: starting mon.a rank 0 at public addrs [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] at bind addrs [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 3847408c-e76d-11ee-95cd-87774f69a715 2024-03-21T10:25:50.333 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: mon.a@-1(???) e1 preinit fsid 3847408c-e76d-11ee-95cd-87774f69a715 2024-03-21T10:25:50.333 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: mon.a@-1(???).mds e1 new map 2024-03-21T10:25:50.334 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: mon.a@-1(???).mds e1 print_map 2024-03-21T10:25:50.334 INFO:journalctl@ceph.mon.a.smithi031.stdout: e1 2024-03-21T10:25:50.334 INFO:journalctl@ceph.mon.a.smithi031.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-21T10:25:50.334 INFO:journalctl@ceph.mon.a.smithi031.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-21T10:25:50.334 INFO:journalctl@ceph.mon.a.smithi031.stdout: legacy client fscid: -1 2024-03-21T10:25:50.334 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-03-21T10:25:50.334 INFO:journalctl@ceph.mon.a.smithi031.stdout: No filesystems configured 2024-03-21T10:25:50.334 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-21T10:25:50.334 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T10:25:50.334 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T10:25:50.334 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T10:25:50.334 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-21T10:25:50.334 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 systemd[1]: Started Ceph mon.a for 3847408c-e76d-11ee-95cd-87774f69a715. 2024-03-21T10:25:50.334 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-21T10:25:50.335 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: monmap e1: 1 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T10:25:50.335 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: fsmap 2024-03-21T10:25:50.335 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: osdmap e1: 0 total, 0 up, 0 in 2024-03-21T10:25:50.335 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:50 smithi031 ceph-mon[31673]: mgrmap e1: no daemons active 2024-03-21T10:25:51.491 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.651+0000 7fee23400700 1 Processor -- start 2024-03-21T10:25:51.491 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.651+0000 7fee23400700 1 -- start start 2024-03-21T10:25:51.491 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.651+0000 7fee23400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee1c109c80 0x7fee1c10a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:25:51.492 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.652+0000 7fee23400700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1c10a670 con 0x7fee1c109c80 2024-03-21T10:25:51.492 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.652+0000 7fee20a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee1c109c80 0x7fee1c10a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:25:51.493 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.652+0000 7fee20a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee1c109c80 0x7fee1c10a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57566/0 (socket says 172.21.15.31:57566) 2024-03-21T10:25:51.493 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.652+0000 7fee20a00700 1 -- 172.21.15.31:0/355767326 learned_addr learned my addr 172.21.15.31:0/355767326 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:25:51.493 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.653+0000 7fee20a00700 1 -- 172.21.15.31:0/355767326 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1c10ae80 con 0x7fee1c109c80 2024-03-21T10:25:51.493 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.653+0000 7fee20a00700 1 --2- 172.21.15.31:0/355767326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee1c109c80 0x7fee1c10a0a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fee100098a0 tx=0x7fee10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e3dd7b263431ef5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:25:51.494 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.654+0000 7fee1b400700 1 -- 172.21.15.31:0/355767326 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fee10004160 con 0x7fee1c109c80 2024-03-21T10:25:51.494 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.654+0000 7fee1b400700 1 -- 172.21.15.31:0/355767326 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee10004300 con 0x7fee1c109c80 2024-03-21T10:25:51.494 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.654+0000 7fee1b400700 1 -- 172.21.15.31:0/355767326 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fee10011b50 con 0x7fee1c109c80 2024-03-21T10:25:51.494 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.654+0000 7fee23400700 1 -- 172.21.15.31:0/355767326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee1c109c80 msgr2=0x7fee1c10a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:25:51.494 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.654+0000 7fee23400700 1 --2- 172.21.15.31:0/355767326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee1c109c80 0x7fee1c10a0a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fee100098a0 tx=0x7fee10009bb0 comp rx=0 tx=0).stop 2024-03-21T10:25:51.494 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.655+0000 7fee23400700 1 -- 172.21.15.31:0/355767326 shutdown_connections 2024-03-21T10:25:51.494 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.655+0000 7fee23400700 1 --2- 172.21.15.31:0/355767326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee1c109c80 0x7fee1c10a0a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:25:51.494 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.655+0000 7fee23400700 1 -- 172.21.15.31:0/355767326 >> 172.21.15.31:0/355767326 conn(0x7fee1c07d040 msgr2=0x7fee1c07d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:25:51.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.655+0000 7fee23400700 1 -- 172.21.15.31:0/355767326 shutdown_connections 2024-03-21T10:25:51.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.655+0000 7fee23400700 1 -- 172.21.15.31:0/355767326 wait complete. 2024-03-21T10:25:51.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.656+0000 7fee23400700 1 Processor -- start 2024-03-21T10:25:51.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.656+0000 7fee23400700 1 -- start start 2024-03-21T10:25:51.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.656+0000 7fee23400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee1c109c80 0x7fee1c19dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:25:51.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.656+0000 7fee23400700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1c10e230 con 0x7fee1c109c80 2024-03-21T10:25:51.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.657+0000 7fee20a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee1c109c80 0x7fee1c19dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:25:51.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.657+0000 7fee20a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee1c109c80 0x7fee1c19dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57576/0 (socket says 172.21.15.31:57576) 2024-03-21T10:25:51.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.657+0000 7fee20a00700 1 -- 172.21.15.31:0/1655062257 learned_addr learned my addr 172.21.15.31:0/1655062257 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:25:51.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.657+0000 7fee20a00700 1 -- 172.21.15.31:0/1655062257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1c19e150 con 0x7fee1c109c80 2024-03-21T10:25:51.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.657+0000 7fee20a00700 1 --2- 172.21.15.31:0/1655062257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee1c109c80 0x7fee1c19dc10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fee10011670 tx=0x7fee10011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:25:51.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.658+0000 7fee19600700 1 -- 172.21.15.31:0/1655062257 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fee100039c0 con 0x7fee1c109c80 2024-03-21T10:25:51.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.658+0000 7fee19600700 1 -- 172.21.15.31:0/1655062257 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee10011e50 con 0x7fee1c109c80 2024-03-21T10:25:51.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.658+0000 7fee23400700 1 -- 172.21.15.31:0/1655062257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee1c19e3e0 con 0x7fee1c109c80 2024-03-21T10:25:51.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.658+0000 7fee19600700 1 -- 172.21.15.31:0/1655062257 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fee1001a5c0 con 0x7fee1c109c80 2024-03-21T10:25:51.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.658+0000 7fee23400700 1 -- 172.21.15.31:0/1655062257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee1c1a10d0 con 0x7fee1c109c80 2024-03-21T10:25:51.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.659+0000 7fee19600700 1 -- 172.21.15.31:0/1655062257 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fee1001ac50 con 0x7fee1c109c80 2024-03-21T10:25:51.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.659+0000 7fee19600700 1 -- 172.21.15.31:0/1655062257 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fee10019700 con 0x7fee1c109c80 2024-03-21T10:25:51.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.659+0000 7fee23400700 1 -- 172.21.15.31:0/1655062257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee1c0637c0 con 0x7fee1c109c80 2024-03-21T10:25:51.499 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.663+0000 7fee19600700 1 -- 172.21.15.31:0/1655062257 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fee10003b60 con 0x7fee1c109c80 2024-03-21T10:25:51.499 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.740+0000 7fee23400700 1 -- 172.21.15.31:0/1655062257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fee1c1a12a0 con 0x7fee1c109c80 2024-03-21T10:25:51.499 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.744+0000 7fee19600700 1 -- 172.21.15.31:0/1655062257 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fee10024430 con 0x7fee1c109c80 2024-03-21T10:25:51.499 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.745+0000 7fee23400700 1 -- 172.21.15.31:0/1655062257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee1c109c80 msgr2=0x7fee1c19dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:25:51.499 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.745+0000 7fee23400700 1 --2- 172.21.15.31:0/1655062257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee1c109c80 0x7fee1c19dc10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fee10011670 tx=0x7fee10011b60 comp rx=0 tx=0).stop 2024-03-21T10:25:51.499 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.745+0000 7fee23400700 1 -- 172.21.15.31:0/1655062257 shutdown_connections 2024-03-21T10:25:51.499 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.745+0000 7fee23400700 1 --2- 172.21.15.31:0/1655062257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee1c109c80 0x7fee1c19dc10 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:25:51.500 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.745+0000 7fee23400700 1 -- 172.21.15.31:0/1655062257 >> 172.21.15.31:0/1655062257 conn(0x7fee1c07d040 msgr2=0x7fee1c107460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:25:51.500 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.745+0000 7fee23400700 1 -- 172.21.15.31:0/1655062257 shutdown_connections 2024-03-21T10:25:51.500 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:50.745+0000 7fee23400700 1 -- 172.21.15.31:0/1655062257 wait complete. 2024-03-21T10:25:51.500 INFO:teuthology.orchestra.run.smithi031.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-21T10:25:51.529 INFO:teuthology.orchestra.run.smithi031.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-21T10:25:51.529 INFO:teuthology.orchestra.run.smithi031.stdout:Creating mgr... 2024-03-21T10:25:51.531 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-21T10:25:51.532 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-21T10:25:51.951 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-3847408c-e76d-11ee-95cd-87774f69a715@mgr.x 2024-03-21T10:25:51.951 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to reset failed state of unit ceph-3847408c-e76d-11ee-95cd-87774f69a715@mgr.x.service: Unit ceph-3847408c-e76d-11ee-95cd-87774f69a715@mgr.x.service not loaded. 2024-03-21T10:25:52.058 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:51 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1655062257' entity='client.admin' 2024-03-21T10:25:52.234 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-3847408c-e76d-11ee-95cd-87774f69a715.target.wants/ceph-3847408c-e76d-11ee-95cd-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-3847408c-e76d-11ee-95cd-87774f69a715@.service. 2024-03-21T10:25:52.558 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:52 smithi031 systemd[1]: Starting Ceph mgr.x for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:25:52.919 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:52 smithi031 podman[31884]: 2024-03-21 10:25:52.539327925 +0000 UTC m=+0.033186610 image pull 237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 2024-03-21T10:25:52.919 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:52 smithi031 podman[31884]: 2024-03-21 10:25:52.670970067 +0000 UTC m=+0.164828777 container create d2fad9ce65f9c1e0d84494b8af7e9fc45f2757f4157fb5acc46f57c4bdd6287c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-21T10:25:52.919 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:52 smithi031 podman[31884]: 2024-03-21 10:25:52.918515386 +0000 UTC m=+0.412374125 container init d2fad9ce65f9c1e0d84494b8af7e9fc45f2757f4157fb5acc46f57c4bdd6287c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T10:25:53.023 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-21T10:25:53.024 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2024-03-21T10:25:53.039 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-21T10:25:53.040 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-03-21T10:25:53.040 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2024-03-21T10:25:53.040 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to enable service . firewalld.service is not available 2024-03-21T10:25:53.065 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-21T10:25:53.065 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2024-03-21T10:25:53.087 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-21T10:25:53.088 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-03-21T10:25:53.088 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2024-03-21T10:25:53.088 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-21T10:25:53.088 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr to start... 2024-03-21T10:25:53.088 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr... 2024-03-21T10:25:53.242 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:52 smithi031 podman[31884]: 2024-03-21 10:25:52.928602054 +0000 UTC m=+0.422460732 container start d2fad9ce65f9c1e0d84494b8af7e9fc45f2757f4157fb5acc46f57c4bdd6287c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-21T10:25:53.242 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:52 smithi031 bash[31884]: d2fad9ce65f9c1e0d84494b8af7e9fc45f2757f4157fb5acc46f57c4bdd6287c 2024-03-21T10:25:53.242 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:53 smithi031 systemd[1]: Started Ceph mgr.x for 3847408c-e76d-11ee-95cd-87774f69a715. 2024-03-21T10:25:53.838 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:53 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:25:53.727+0000 7f73a57e7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T10:25:54.152 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:53 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2961514534' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-21T10:25:54.153 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:53 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:25:53.858+0000 7f73a57e7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T10:25:54.482 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-03-21T10:25:54.485 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-03-21T10:25:54.485 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsid": "3847408c-e76d-11ee-95cd-87774f69a715", 2024-03-21T10:25:54.485 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "health": { 2024-03-21T10:25:54.485 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-21T10:25:54.485 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-21T10:25:54.485 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-21T10:25:54.486 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:25:54.486 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-21T10:25:54.486 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-21T10:25:54.486 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 0 2024-03-21T10:25:54.486 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-03-21T10:25:54.486 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-21T10:25:54.486 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "a" 2024-03-21T10:25:54.486 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-03-21T10:25:54.486 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-03-21T10:25:54.486 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-21T10:25:54.486 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T10:25:54.486 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-21T10:25:54.486 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-21T10:25:54.487 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:25:54.487 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-21T10:25:54.487 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T10:25:54.487 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-21T10:25:54.487 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-21T10:25:54.487 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-21T10:25:54.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-21T10:25:54.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-21T10:25:54.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-21T10:25:54.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:25:54.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-21T10:25:54.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-21T10:25:54.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-21T10:25:54.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-21T10:25:54.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-21T10:25:54.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-21T10:25:54.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-21T10:25:54.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-21T10:25:54.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-21T10:25:54.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:25:54.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-21T10:25:54.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T10:25:54.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-21T10:25:54.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-21T10:25:54.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:25:54.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-21T10:25:54.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-21T10:25:54.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-21T10:25:54.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-21T10:25:54.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-21T10:25:54.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-21T10:25:54.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "restful" 2024-03-21T10:25:54.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-03-21T10:25:54.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T10:25:54.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:25:54.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-21T10:25:54.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T10:25:54.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modified": "2024-03-21T10:25:41.972019+0000", 2024-03-21T10:25:54.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T10:25:54.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:25:54.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-21T10:25:54.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-03-21T10:25:54.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.745+0000 7ff204800700 1 Processor -- start 2024-03-21T10:25:54.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.746+0000 7ff204800700 1 -- start start 2024-03-21T10:25:54.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.746+0000 7ff204800700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1fc07de80 0x7ff1fc07c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:25:54.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.746+0000 7ff204800700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc07c8b0 con 0x7ff1fc07de80 2024-03-21T10:25:54.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.747+0000 7ff201e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1fc07de80 0x7ff1fc07c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:25:54.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.747+0000 7ff201e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1fc07de80 0x7ff1fc07c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57610/0 (socket says 172.21.15.31:57610) 2024-03-21T10:25:54.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.747+0000 7ff201e00700 1 -- 172.21.15.31:0/501042953 learned_addr learned my addr 172.21.15.31:0/501042953 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:25:54.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.748+0000 7ff201e00700 1 -- 172.21.15.31:0/501042953 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1fc07ca30 con 0x7ff1fc07de80 2024-03-21T10:25:54.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.748+0000 7ff201e00700 1 --2- 172.21.15.31:0/501042953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1fc07de80 0x7ff1fc07c2e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff1f00098a0 tx=0x7ff1f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66d97f0c86e3d0db server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:25:54.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.748+0000 7ff200a00700 1 -- 172.21.15.31:0/501042953 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1f0004160 con 0x7ff1fc07de80 2024-03-21T10:25:54.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.748+0000 7ff200a00700 1 -- 172.21.15.31:0/501042953 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff1f0004300 con 0x7ff1fc07de80 2024-03-21T10:25:54.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.748+0000 7ff200a00700 1 -- 172.21.15.31:0/501042953 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1f0011b50 con 0x7ff1fc07de80 2024-03-21T10:25:54.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.749+0000 7ff204800700 1 -- 172.21.15.31:0/501042953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1fc07de80 msgr2=0x7ff1fc07c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:25:54.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.749+0000 7ff204800700 1 --2- 172.21.15.31:0/501042953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1fc07de80 0x7ff1fc07c2e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff1f00098a0 tx=0x7ff1f0009bb0 comp rx=0 tx=0).stop 2024-03-21T10:25:54.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.749+0000 7ff204800700 1 -- 172.21.15.31:0/501042953 shutdown_connections 2024-03-21T10:25:54.499 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.749+0000 7ff204800700 1 --2- 172.21.15.31:0/501042953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1fc07de80 0x7ff1fc07c2e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:25:54.499 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.749+0000 7ff204800700 1 -- 172.21.15.31:0/501042953 >> 172.21.15.31:0/501042953 conn(0x7ff1fc103080 msgr2=0x7ff1fc1054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:25:54.499 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.749+0000 7ff204800700 1 -- 172.21.15.31:0/501042953 shutdown_connections 2024-03-21T10:25:54.499 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.749+0000 7ff204800700 1 -- 172.21.15.31:0/501042953 wait complete. 2024-03-21T10:25:54.499 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.750+0000 7ff204800700 1 Processor -- start 2024-03-21T10:25:54.499 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.750+0000 7ff204800700 1 -- start start 2024-03-21T10:25:54.499 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.751+0000 7ff204800700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1fc07de80 0x7ff1fc19dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:25:54.499 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.751+0000 7ff204800700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc10e090 con 0x7ff1fc07de80 2024-03-21T10:25:54.499 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.751+0000 7ff201e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1fc07de80 0x7ff1fc19dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:25:54.499 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.751+0000 7ff201e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1fc07de80 0x7ff1fc19dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57626/0 (socket says 172.21.15.31:57626) 2024-03-21T10:25:54.499 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.751+0000 7ff201e00700 1 -- 172.21.15.31:0/2961514534 learned_addr learned my addr 172.21.15.31:0/2961514534 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:25:54.499 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.751+0000 7ff201e00700 1 -- 172.21.15.31:0/2961514534 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1fc19e190 con 0x7ff1fc07de80 2024-03-21T10:25:54.499 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.752+0000 7ff201e00700 1 --2- 172.21.15.31:0/2961514534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1fc07de80 0x7ff1fc19dc50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0011670 tx=0x7ff1f0011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:25:54.500 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.752+0000 7ff1f6a00700 1 -- 172.21.15.31:0/2961514534 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1f00039c0 con 0x7ff1fc07de80 2024-03-21T10:25:54.500 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.752+0000 7ff1f6a00700 1 -- 172.21.15.31:0/2961514534 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff1f0011e50 con 0x7ff1fc07de80 2024-03-21T10:25:54.500 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.752+0000 7ff1f6a00700 1 -- 172.21.15.31:0/2961514534 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1f001a5c0 con 0x7ff1fc07de80 2024-03-21T10:25:54.500 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.752+0000 7ff204800700 1 -- 172.21.15.31:0/2961514534 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1fc19e420 con 0x7ff1fc07de80 2024-03-21T10:25:54.500 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.752+0000 7ff204800700 1 -- 172.21.15.31:0/2961514534 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1fc1a1110 con 0x7ff1fc07de80 2024-03-21T10:25:54.500 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.753+0000 7ff1f6a00700 1 -- 172.21.15.31:0/2961514534 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ff1f001ac00 con 0x7ff1fc07de80 2024-03-21T10:25:54.500 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.753+0000 7ff1f6a00700 1 -- 172.21.15.31:0/2961514534 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff1f0019a50 con 0x7ff1fc07de80 2024-03-21T10:25:54.500 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.754+0000 7ff1ebe00700 1 -- 172.21.15.31:0/2961514534 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1d8005330 con 0x7ff1fc07de80 2024-03-21T10:25:54.500 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.756+0000 7ff1f6a00700 1 -- 172.21.15.31:0/2961514534 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ff1f0003b60 con 0x7ff1fc07de80 2024-03-21T10:25:54.500 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.834+0000 7ff1ebe00700 1 -- 172.21.15.31:0/2961514534 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff1d80058f0 con 0x7ff1fc07de80 2024-03-21T10:25:54.500 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.834+0000 7ff1f6a00700 1 -- 172.21.15.31:0/2961514534 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7ff1f0019cf0 con 0x7ff1fc07de80 2024-03-21T10:25:54.500 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.835+0000 7ff1ebe00700 1 -- 172.21.15.31:0/2961514534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1fc07de80 msgr2=0x7ff1fc19dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:25:54.500 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.835+0000 7ff1ebe00700 1 --2- 172.21.15.31:0/2961514534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1fc07de80 0x7ff1fc19dc50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0011670 tx=0x7ff1f0011b60 comp rx=0 tx=0).stop 2024-03-21T10:25:54.500 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.836+0000 7ff1ebe00700 1 -- 172.21.15.31:0/2961514534 shutdown_connections 2024-03-21T10:25:54.501 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.836+0000 7ff1ebe00700 1 --2- 172.21.15.31:0/2961514534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1fc07de80 0x7ff1fc19dc50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:25:54.501 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.836+0000 7ff1ebe00700 1 -- 172.21.15.31:0/2961514534 >> 172.21.15.31:0/2961514534 conn(0x7ff1fc103080 msgr2=0x7ff1fc103be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:25:54.501 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.836+0000 7ff1ebe00700 1 -- 172.21.15.31:0/2961514534 shutdown_connections 2024-03-21T10:25:54.501 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:53.836+0000 7ff1ebe00700 1 -- 172.21.15.31:0/2961514534 wait complete. 2024-03-21T10:25:54.501 INFO:teuthology.orchestra.run.smithi031.stdout:mgr not available, waiting (1/15)... 2024-03-21T10:25:54.807 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:54 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:25:54.694+0000 7f73a57e7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T10:25:55.128 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:54 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:25:54.819+0000 7f73a57e7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T10:25:55.128 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:55 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:25:54.999+0000 7f73a57e7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T10:25:55.128 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:55 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:25:55.126+0000 7f73a57e7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T10:25:55.557 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:55 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:25:55.242+0000 7f73a57e7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T10:25:55.841 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:55 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:25:55.582+0000 7f73a57e7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T10:25:55.841 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:55 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:25:55.727+0000 7f73a57e7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T10:25:56.190 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:55 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:25:55.840+0000 7f73a57e7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T10:25:56.190 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:55 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:25:55.967+0000 7f73a57e7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T10:25:56.190 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:56 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:25:56.188+0000 7f73a57e7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T10:25:57.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:25:57 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/155061138' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-21T10:25:57.740 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:57 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:25:57.573+0000 7f73a57e7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T10:25:58.023 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:57 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:25:57.753+0000 7f73a57e7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T10:25:58.037 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-03-21T10:25:58.038 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-03-21T10:25:58.038 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsid": "3847408c-e76d-11ee-95cd-87774f69a715", 2024-03-21T10:25:58.038 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "health": { 2024-03-21T10:25:58.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-21T10:25:58.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-21T10:25:58.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-21T10:25:58.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:25:58.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-21T10:25:58.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-21T10:25:58.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 0 2024-03-21T10:25:58.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-03-21T10:25:58.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-21T10:25:58.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "a" 2024-03-21T10:25:58.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-03-21T10:25:58.041 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-03-21T10:25:58.041 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-21T10:25:58.041 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T10:25:58.041 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-21T10:25:58.041 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-21T10:25:58.042 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:25:58.042 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-21T10:25:58.042 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T10:25:58.042 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-21T10:25:58.042 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-21T10:25:58.042 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-21T10:25:58.042 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-21T10:25:58.043 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-21T10:25:58.043 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-21T10:25:58.043 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:25:58.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-21T10:25:58.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-21T10:25:58.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-21T10:25:58.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-21T10:25:58.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-21T10:25:58.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-21T10:25:58.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-21T10:25:58.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-21T10:25:58.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-21T10:25:58.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:25:58.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-21T10:25:58.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T10:25:58.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-21T10:25:58.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-21T10:25:58.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:25:58.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-21T10:25:58.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-21T10:25:58.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-21T10:25:58.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-21T10:25:58.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-21T10:25:58.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-21T10:25:58.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "restful" 2024-03-21T10:25:58.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-03-21T10:25:58.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T10:25:58.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:25:58.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-21T10:25:58.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T10:25:58.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modified": "2024-03-21T10:25:41.972019+0000", 2024-03-21T10:25:58.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T10:25:58.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:25:58.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-21T10:25:58.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-03-21T10:25:58.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.264+0000 7f06b8400700 1 Processor -- start 2024-03-21T10:25:58.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.265+0000 7f06b8400700 1 -- start start 2024-03-21T10:25:58.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.265+0000 7f06b8400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06b007dea0 0x7f06b007c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:25:58.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.265+0000 7f06b8400700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b007c8d0 con 0x7f06b007dea0 2024-03-21T10:25:58.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.265+0000 7f06b5a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06b007dea0 0x7f06b007c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:25:58.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.265+0000 7f06b5a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06b007dea0 0x7f06b007c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55748/0 (socket says 172.21.15.31:55748) 2024-03-21T10:25:58.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.265+0000 7f06b5a00700 1 -- 172.21.15.31:0/657319936 learned_addr learned my addr 172.21.15.31:0/657319936 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:25:58.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.266+0000 7f06b5a00700 1 -- 172.21.15.31:0/657319936 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b007ca50 con 0x7f06b007dea0 2024-03-21T10:25:58.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.266+0000 7f06b5a00700 1 --2- 172.21.15.31:0/657319936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06b007dea0 0x7f06b007c300 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f06a40098a0 tx=0x7f06a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=414623b5428d3adf server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:25:58.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.267+0000 7f06afe00700 1 -- 172.21.15.31:0/657319936 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f06a4004160 con 0x7f06b007dea0 2024-03-21T10:25:58.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.267+0000 7f06afe00700 1 -- 172.21.15.31:0/657319936 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f06a4004300 con 0x7f06b007dea0 2024-03-21T10:25:58.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.267+0000 7f06afe00700 1 -- 172.21.15.31:0/657319936 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f06a4011b50 con 0x7f06b007dea0 2024-03-21T10:25:58.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.267+0000 7f06b8400700 1 -- 172.21.15.31:0/657319936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06b007dea0 msgr2=0x7f06b007c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:25:58.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.267+0000 7f06b8400700 1 --2- 172.21.15.31:0/657319936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06b007dea0 0x7f06b007c300 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f06a40098a0 tx=0x7f06a4009bb0 comp rx=0 tx=0).stop 2024-03-21T10:25:58.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.268+0000 7f06b8400700 1 -- 172.21.15.31:0/657319936 shutdown_connections 2024-03-21T10:25:58.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.268+0000 7f06b8400700 1 --2- 172.21.15.31:0/657319936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06b007dea0 0x7f06b007c300 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:25:58.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.268+0000 7f06b8400700 1 -- 172.21.15.31:0/657319936 >> 172.21.15.31:0/657319936 conn(0x7f06b0103080 msgr2=0x7f06b01054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:25:58.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.268+0000 7f06b8400700 1 -- 172.21.15.31:0/657319936 shutdown_connections 2024-03-21T10:25:58.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.268+0000 7f06b8400700 1 -- 172.21.15.31:0/657319936 wait complete. 2024-03-21T10:25:58.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.269+0000 7f06b8400700 1 Processor -- start 2024-03-21T10:25:58.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.269+0000 7f06b8400700 1 -- start start 2024-03-21T10:25:58.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.269+0000 7f06b8400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06b007dea0 0x7f06b019dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:25:58.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.269+0000 7f06b8400700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b010e070 con 0x7f06b007dea0 2024-03-21T10:25:58.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.270+0000 7f06b5a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06b007dea0 0x7f06b019dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:25:58.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.270+0000 7f06b5a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06b007dea0 0x7f06b019dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55762/0 (socket says 172.21.15.31:55762) 2024-03-21T10:25:58.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.270+0000 7f06b5a00700 1 -- 172.21.15.31:0/155061138 learned_addr learned my addr 172.21.15.31:0/155061138 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:25:58.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.270+0000 7f06b5a00700 1 -- 172.21.15.31:0/155061138 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b019e190 con 0x7f06b007dea0 2024-03-21T10:25:58.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.270+0000 7f06b5a00700 1 --2- 172.21.15.31:0/155061138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06b007dea0 0x7f06b019dc50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f06a4000c00 tx=0x7f06a4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:25:58.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.271+0000 7f06ae000700 1 -- 172.21.15.31:0/155061138 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f06a4003b20 con 0x7f06b007dea0 2024-03-21T10:25:58.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.271+0000 7f06ae000700 1 -- 172.21.15.31:0/155061138 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f06a4003cc0 con 0x7f06b007dea0 2024-03-21T10:25:58.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.271+0000 7f06ae000700 1 -- 172.21.15.31:0/155061138 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f06a4024910 con 0x7f06b007dea0 2024-03-21T10:25:58.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.271+0000 7f06b8400700 1 -- 172.21.15.31:0/155061138 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b019e420 con 0x7f06b007dea0 2024-03-21T10:25:58.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.271+0000 7f06b8400700 1 -- 172.21.15.31:0/155061138 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b01a1110 con 0x7f06b007dea0 2024-03-21T10:25:58.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.271+0000 7f06ae000700 1 -- 172.21.15.31:0/155061138 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f06a4022020 con 0x7f06b007dea0 2024-03-21T10:25:58.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.271+0000 7f06ae000700 1 -- 172.21.15.31:0/155061138 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f06a4011e00 con 0x7f06b007dea0 2024-03-21T10:25:58.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.272+0000 7f06b8400700 1 -- 172.21.15.31:0/155061138 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b00637c0 con 0x7f06b007dea0 2024-03-21T10:25:58.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.275+0000 7f06ae000700 1 -- 172.21.15.31:0/155061138 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f06a401acd0 con 0x7f06b007dea0 2024-03-21T10:25:58.056 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.354+0000 7f06b8400700 1 -- 172.21.15.31:0/155061138 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f06b01a12e0 con 0x7f06b007dea0 2024-03-21T10:25:58.056 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.354+0000 7f06ae000700 1 -- 172.21.15.31:0/155061138 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f06a4032030 con 0x7f06b007dea0 2024-03-21T10:25:58.056 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.356+0000 7f06b8400700 1 -- 172.21.15.31:0/155061138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06b007dea0 msgr2=0x7f06b019dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:25:58.056 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.356+0000 7f06b8400700 1 --2- 172.21.15.31:0/155061138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06b007dea0 0x7f06b019dc50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f06a4000c00 tx=0x7f06a4011c10 comp rx=0 tx=0).stop 2024-03-21T10:25:58.056 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.356+0000 7f06b8400700 1 -- 172.21.15.31:0/155061138 shutdown_connections 2024-03-21T10:25:58.056 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.356+0000 7f06b8400700 1 --2- 172.21.15.31:0/155061138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06b007dea0 0x7f06b019dc50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:25:58.056 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.356+0000 7f06b8400700 1 -- 172.21.15.31:0/155061138 >> 172.21.15.31:0/155061138 conn(0x7f06b0103080 msgr2=0x7f06b0103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:25:58.056 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.356+0000 7f06b8400700 1 -- 172.21.15.31:0/155061138 shutdown_connections 2024-03-21T10:25:58.056 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:25:57.356+0000 7f06b8400700 1 -- 172.21.15.31:0/155061138 wait complete. 2024-03-21T10:25:58.057 INFO:teuthology.orchestra.run.smithi031.stdout:mgr not available, waiting (2/15)... 2024-03-21T10:25:58.301 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:58 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:25:58.179+0000 7f73a57e7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T10:25:58.301 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:58 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:25:58.299+0000 7f73a57e7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T10:25:58.814 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:58 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:25:58.561+0000 7f73a57e7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T10:25:59.197 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:58 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:25:58.812+0000 7f73a57e7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T10:25:59.197 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:59 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:25:59.195+0000 7f73a57e7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T10:25:59.807 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:59 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:25:59.517+0000 7f73a57e7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T10:26:00.205 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:25:59 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:25:59.831+0000 7f73a57e7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T10:26:01.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:00 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1991728938' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-21T10:26:01.058 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:00 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:00.925+0000 7f73a57e7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T10:26:01.424 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-03-21T10:26:01.424 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-03-21T10:26:01.424 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsid": "3847408c-e76d-11ee-95cd-87774f69a715", 2024-03-21T10:26:01.424 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "health": { 2024-03-21T10:26:01.424 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-21T10:26:01.424 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-21T10:26:01.424 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-21T10:26:01.424 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:26:01.424 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-21T10:26:01.424 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-21T10:26:01.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 0 2024-03-21T10:26:01.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-03-21T10:26:01.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-21T10:26:01.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "a" 2024-03-21T10:26:01.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-03-21T10:26:01.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-21T10:26:01.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-21T10:26:01.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T10:26:01.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-21T10:26:01.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-21T10:26:01.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:26:01.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-21T10:26:01.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T10:26:01.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-21T10:26:01.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-21T10:26:01.426 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-21T10:26:01.426 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-21T10:26:01.426 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-21T10:26:01.433 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-21T10:26:01.433 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:26:01.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-21T10:26:01.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-21T10:26:01.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-21T10:26:01.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-21T10:26:01.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-21T10:26:01.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-21T10:26:01.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-21T10:26:01.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-21T10:26:01.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-21T10:26:01.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:26:01.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-21T10:26:01.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T10:26:01.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-21T10:26:01.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-21T10:26:01.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:26:01.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-21T10:26:01.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-21T10:26:01.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-21T10:26:01.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-21T10:26:01.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-21T10:26:01.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-21T10:26:01.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "restful" 2024-03-21T10:26:01.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-03-21T10:26:01.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T10:26:01.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:26:01.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-21T10:26:01.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T10:26:01.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modified": "2024-03-21T10:25:41.972019+0000", 2024-03-21T10:26:01.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T10:26:01.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:26:01.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-21T10:26:01.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-03-21T10:26:01.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.620+0000 7f7791200700 1 Processor -- start 2024-03-21T10:26:01.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.620+0000 7f7791200700 1 -- start start 2024-03-21T10:26:01.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.621+0000 7f7791200700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f778c106370 0x7f778c106790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:01.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.621+0000 7f7791200700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778c07bdf0 con 0x7f778c106370 2024-03-21T10:26:01.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.621+0000 7f7789e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f778c106370 0x7f778c106790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:01.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.621+0000 7f7789e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f778c106370 0x7f778c106790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55766/0 (socket says 172.21.15.31:55766) 2024-03-21T10:26:01.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.621+0000 7f7789e00700 1 -- 172.21.15.31:0/2069058853 learned_addr learned my addr 172.21.15.31:0/2069058853 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:01.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.621+0000 7f7789e00700 1 -- 172.21.15.31:0/2069058853 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778c106cd0 con 0x7f778c106370 2024-03-21T10:26:01.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.622+0000 7f7789e00700 1 --2- 172.21.15.31:0/2069058853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f778c106370 0x7f778c106790 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f777c0098a0 tx=0x7f777c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a50353b266ccfde7 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:01.437 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.622+0000 7f7788a00700 1 -- 172.21.15.31:0/2069058853 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f777c004160 con 0x7f778c106370 2024-03-21T10:26:01.437 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.622+0000 7f7788a00700 1 -- 172.21.15.31:0/2069058853 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f777c004300 con 0x7f778c106370 2024-03-21T10:26:01.437 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.622+0000 7f7788a00700 1 -- 172.21.15.31:0/2069058853 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f777c011b50 con 0x7f778c106370 2024-03-21T10:26:01.437 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.622+0000 7f7791200700 1 -- 172.21.15.31:0/2069058853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f778c106370 msgr2=0x7f778c106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:01.437 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.622+0000 7f7791200700 1 --2- 172.21.15.31:0/2069058853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f778c106370 0x7f778c106790 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f777c0098a0 tx=0x7f777c009bb0 comp rx=0 tx=0).stop 2024-03-21T10:26:01.437 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.623+0000 7f7791200700 1 -- 172.21.15.31:0/2069058853 shutdown_connections 2024-03-21T10:26:01.437 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.623+0000 7f7791200700 1 --2- 172.21.15.31:0/2069058853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f778c106370 0x7f778c106790 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:01.437 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.623+0000 7f7791200700 1 -- 172.21.15.31:0/2069058853 >> 172.21.15.31:0/2069058853 conn(0x7f778c101f70 msgr2=0x7f778c104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:01.437 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.623+0000 7f7791200700 1 -- 172.21.15.31:0/2069058853 shutdown_connections 2024-03-21T10:26:01.437 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.623+0000 7f7791200700 1 -- 172.21.15.31:0/2069058853 wait complete. 2024-03-21T10:26:01.437 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.624+0000 7f7791200700 1 Processor -- start 2024-03-21T10:26:01.437 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.624+0000 7f7791200700 1 -- start start 2024-03-21T10:26:01.437 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.624+0000 7f7791200700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f778c106370 0x7f778c19dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:01.437 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.624+0000 7f7791200700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778c107d60 con 0x7f778c106370 2024-03-21T10:26:01.438 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.625+0000 7f7789e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f778c106370 0x7f778c19dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:01.438 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.625+0000 7f7789e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f778c106370 0x7f778c19dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55780/0 (socket says 172.21.15.31:55780) 2024-03-21T10:26:01.438 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.625+0000 7f7789e00700 1 -- 172.21.15.31:0/1991728938 learned_addr learned my addr 172.21.15.31:0/1991728938 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:01.438 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.625+0000 7f7789e00700 1 -- 172.21.15.31:0/1991728938 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778c19e130 con 0x7f778c106370 2024-03-21T10:26:01.438 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.625+0000 7f7789e00700 1 --2- 172.21.15.31:0/1991728938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f778c106370 0x7f778c19dbf0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f777c011b60 tx=0x7f777c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:01.438 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.626+0000 7f7782a00700 1 -- 172.21.15.31:0/1991728938 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f777c003b20 con 0x7f778c106370 2024-03-21T10:26:01.438 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.626+0000 7f7782a00700 1 -- 172.21.15.31:0/1991728938 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f777c003cc0 con 0x7f778c106370 2024-03-21T10:26:01.438 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.626+0000 7f7782a00700 1 -- 172.21.15.31:0/1991728938 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f777c024840 con 0x7f778c106370 2024-03-21T10:26:01.438 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.626+0000 7f7791200700 1 -- 172.21.15.31:0/1991728938 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f778c19e3c0 con 0x7f778c106370 2024-03-21T10:26:01.438 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.626+0000 7f7791200700 1 -- 172.21.15.31:0/1991728938 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f778c1a10b0 con 0x7f778c106370 2024-03-21T10:26:01.438 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.626+0000 7f7782a00700 1 -- 172.21.15.31:0/1991728938 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f777c022020 con 0x7f778c106370 2024-03-21T10:26:01.438 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.626+0000 7f7782a00700 1 -- 172.21.15.31:0/1991728938 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f777c011e00 con 0x7f778c106370 2024-03-21T10:26:01.438 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.627+0000 7f7791200700 1 -- 172.21.15.31:0/1991728938 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f778c0637c0 con 0x7f778c106370 2024-03-21T10:26:01.438 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.630+0000 7f7782a00700 1 -- 172.21.15.31:0/1991728938 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f777c043dd0 con 0x7f778c106370 2024-03-21T10:26:01.439 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.709+0000 7f7791200700 1 -- 172.21.15.31:0/1991728938 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f778c0008d0 con 0x7f778c106370 2024-03-21T10:26:01.439 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.710+0000 7f7782a00700 1 -- 172.21.15.31:0/1991728938 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7f777c032030 con 0x7f778c106370 2024-03-21T10:26:01.439 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.712+0000 7f7791200700 1 -- 172.21.15.31:0/1991728938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f778c106370 msgr2=0x7f778c19dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:01.439 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.712+0000 7f7791200700 1 --2- 172.21.15.31:0/1991728938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f778c106370 0x7f778c19dbf0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f777c011b60 tx=0x7f777c011c40 comp rx=0 tx=0).stop 2024-03-21T10:26:01.439 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.712+0000 7f7791200700 1 -- 172.21.15.31:0/1991728938 shutdown_connections 2024-03-21T10:26:01.439 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.712+0000 7f7791200700 1 --2- 172.21.15.31:0/1991728938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f778c106370 0x7f778c19dbf0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:01.439 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.712+0000 7f7791200700 1 -- 172.21.15.31:0/1991728938 >> 172.21.15.31:0/1991728938 conn(0x7f778c101f70 msgr2=0x7f778c1948d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:01.439 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.712+0000 7f7791200700 1 -- 172.21.15.31:0/1991728938 shutdown_connections 2024-03-21T10:26:01.439 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:00.712+0000 7f7791200700 1 -- 172.21.15.31:0/1991728938 wait complete. 2024-03-21T10:26:01.439 INFO:teuthology.orchestra.run.smithi031.stdout:mgr not available, waiting (3/15)... 2024-03-21T10:26:02.057 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:01 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:01.671+0000 7f73a57e7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T10:26:02.058 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:01 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:01.804+0000 7f73a57e7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T10:26:02.558 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:02 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:02.093+0000 7f73a57e7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T10:26:02.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:02 smithi031 ceph-mon[31673]: Activating manager daemon x 2024-03-21T10:26:02.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:02 smithi031 ceph-mon[31673]: mgrmap e2: x(active, starting, since 0.0123582s) 2024-03-21T10:26:02.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:02 smithi031 ceph-mon[31673]: from='mgr.14100 172.21.15.31:0/2811004751' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T10:26:02.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:02 smithi031 ceph-mon[31673]: from='mgr.14100 172.21.15.31:0/2811004751' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T10:26:02.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:02 smithi031 ceph-mon[31673]: from='mgr.14100 172.21.15.31:0/2811004751' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T10:26:02.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:02 smithi031 ceph-mon[31673]: from='mgr.14100 172.21.15.31:0/2811004751' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T10:26:02.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:02 smithi031 ceph-mon[31673]: from='mgr.14100 172.21.15.31:0/2811004751' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T10:26:02.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:02 smithi031 ceph-mon[31673]: Manager daemon x is now available 2024-03-21T10:26:02.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:02 smithi031 ceph-mon[31673]: from='mgr.14100 172.21.15.31:0/2811004751' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T10:26:02.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:02 smithi031 ceph-mon[31673]: from='mgr.14100 172.21.15.31:0/2811004751' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T10:26:02.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:02 smithi031 ceph-mon[31673]: from='mgr.14100 172.21.15.31:0/2811004751' entity='mgr.x' 2024-03-21T10:26:03.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:03 smithi031 ceph-mon[31673]: from='mgr.14100 172.21.15.31:0/2811004751' entity='mgr.x' 2024-03-21T10:26:03.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:03 smithi031 ceph-mon[31673]: from='mgr.14100 172.21.15.31:0/2811004751' entity='mgr.x' 2024-03-21T10:26:03.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:03 smithi031 ceph-mon[31673]: mgrmap e3: x(active, since 1.02264s) 2024-03-21T10:26:05.077 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-03-21T10:26:05.078 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-03-21T10:26:05.078 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsid": "3847408c-e76d-11ee-95cd-87774f69a715", 2024-03-21T10:26:05.078 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "health": { 2024-03-21T10:26:05.078 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-21T10:26:05.078 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-21T10:26:05.078 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-21T10:26:05.078 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:26:05.078 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-21T10:26:05.080 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-21T10:26:05.080 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 0 2024-03-21T10:26:05.080 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-03-21T10:26:05.080 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-21T10:26:05.080 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "a" 2024-03-21T10:26:05.080 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-03-21T10:26:05.081 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-21T10:26:05.081 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-21T10:26:05.081 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T10:26:05.081 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-21T10:26:05.081 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-21T10:26:05.081 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:26:05.081 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-21T10:26:05.081 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T10:26:05.081 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-21T10:26:05.081 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-21T10:26:05.081 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-21T10:26:05.090 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-21T10:26:05.090 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-21T10:26:05.090 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-21T10:26:05.090 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:26:05.090 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-21T10:26:05.090 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-21T10:26:05.090 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-21T10:26:05.090 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-21T10:26:05.090 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-21T10:26:05.091 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-21T10:26:05.091 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-21T10:26:05.091 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-21T10:26:05.091 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-21T10:26:05.091 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:26:05.091 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-21T10:26:05.091 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T10:26:05.091 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-21T10:26:05.091 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-21T10:26:05.091 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:26:05.091 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-21T10:26:05.091 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-21T10:26:05.091 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-21T10:26:05.091 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-21T10:26:05.092 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-21T10:26:05.092 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-21T10:26:05.092 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "restful" 2024-03-21T10:26:05.092 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-03-21T10:26:05.092 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T10:26:05.092 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:26:05.092 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-21T10:26:05.092 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T10:26:05.092 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modified": "2024-03-21T10:25:41.972019+0000", 2024-03-21T10:26:05.092 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T10:26:05.092 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-03-21T10:26:05.092 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-21T10:26:05.092 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-03-21T10:26:05.092 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.105+0000 7f90dc800700 1 Processor -- start 2024-03-21T10:26:05.092 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.105+0000 7f90dc800700 1 -- start start 2024-03-21T10:26:05.093 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.106+0000 7f90dc800700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90d4107a80 0x7f90d4107ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:05.093 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.106+0000 7f90dc800700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d4108470 con 0x7f90d4107a80 2024-03-21T10:26:05.093 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.106+0000 7f90d9e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90d4107a80 0x7f90d4107ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:05.093 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.106+0000 7f90d9e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90d4107a80 0x7f90d4107ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55852/0 (socket says 172.21.15.31:55852) 2024-03-21T10:26:05.093 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.106+0000 7f90d9e00700 1 -- 172.21.15.31:0/2525527026 learned_addr learned my addr 172.21.15.31:0/2525527026 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:05.093 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.107+0000 7f90d9e00700 1 -- 172.21.15.31:0/2525527026 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d4108c80 con 0x7f90d4107a80 2024-03-21T10:26:05.093 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.107+0000 7f90d9e00700 1 --2- 172.21.15.31:0/2525527026 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90d4107a80 0x7f90d4107ea0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f90c80098a0 tx=0x7f90c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3358647e12fa3e63 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:05.093 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.108+0000 7f90d8a00700 1 -- 172.21.15.31:0/2525527026 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90c8004160 con 0x7f90d4107a80 2024-03-21T10:26:05.093 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.108+0000 7f90d8a00700 1 -- 172.21.15.31:0/2525527026 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90c8004300 con 0x7f90d4107a80 2024-03-21T10:26:05.093 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.108+0000 7f90d8a00700 1 -- 172.21.15.31:0/2525527026 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90c8011b50 con 0x7f90d4107a80 2024-03-21T10:26:05.093 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.108+0000 7f90dc800700 1 -- 172.21.15.31:0/2525527026 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90d4107a80 msgr2=0x7f90d4107ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:05.093 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.108+0000 7f90dc800700 1 --2- 172.21.15.31:0/2525527026 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90d4107a80 0x7f90d4107ea0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f90c80098a0 tx=0x7f90c8009bb0 comp rx=0 tx=0).stop 2024-03-21T10:26:05.093 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.108+0000 7f90dc800700 1 -- 172.21.15.31:0/2525527026 shutdown_connections 2024-03-21T10:26:05.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.108+0000 7f90dc800700 1 --2- 172.21.15.31:0/2525527026 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90d4107a80 0x7f90d4107ea0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:05.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.108+0000 7f90dc800700 1 -- 172.21.15.31:0/2525527026 >> 172.21.15.31:0/2525527026 conn(0x7f90d4103020 msgr2=0x7f90d4105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:05.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.109+0000 7f90dc800700 1 -- 172.21.15.31:0/2525527026 shutdown_connections 2024-03-21T10:26:05.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.109+0000 7f90dc800700 1 -- 172.21.15.31:0/2525527026 wait complete. 2024-03-21T10:26:05.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.110+0000 7f90dc800700 1 Processor -- start 2024-03-21T10:26:05.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.110+0000 7f90dc800700 1 -- start start 2024-03-21T10:26:05.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.110+0000 7f90dc800700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90d4107a80 0x7f90d419dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:05.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.110+0000 7f90dc800700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d410c030 con 0x7f90d4107a80 2024-03-21T10:26:05.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.110+0000 7f90d9e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90d4107a80 0x7f90d419dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:05.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.111+0000 7f90d9e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90d4107a80 0x7f90d419dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55868/0 (socket says 172.21.15.31:55868) 2024-03-21T10:26:05.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.111+0000 7f90d9e00700 1 -- 172.21.15.31:0/3563453030 learned_addr learned my addr 172.21.15.31:0/3563453030 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:05.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.111+0000 7f90d9e00700 1 -- 172.21.15.31:0/3563453030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d419e180 con 0x7f90d4107a80 2024-03-21T10:26:05.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.111+0000 7f90d9e00700 1 --2- 172.21.15.31:0/3563453030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90d4107a80 0x7f90d419dc40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f90c8011670 tx=0x7f90c8011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:05.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.111+0000 7f90cea00700 1 -- 172.21.15.31:0/3563453030 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90c80039c0 con 0x7f90d4107a80 2024-03-21T10:26:05.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.111+0000 7f90cea00700 1 -- 172.21.15.31:0/3563453030 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90c8011e50 con 0x7f90d4107a80 2024-03-21T10:26:05.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.112+0000 7f90cea00700 1 -- 172.21.15.31:0/3563453030 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90c801a5c0 con 0x7f90d4107a80 2024-03-21T10:26:05.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.112+0000 7f90dc800700 1 -- 172.21.15.31:0/3563453030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d419e410 con 0x7f90d4107a80 2024-03-21T10:26:05.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.112+0000 7f90dc800700 1 -- 172.21.15.31:0/3563453030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d41a1100 con 0x7f90d4107a80 2024-03-21T10:26:05.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.113+0000 7f90dc800700 1 -- 172.21.15.31:0/3563453030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d40637c0 con 0x7f90d4107a80 2024-03-21T10:26:05.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.113+0000 7f90cea00700 1 -- 172.21.15.31:0/3563453030 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 3) v1 ==== 46177+0+0 (secure 0 0 0) 0x7f90c8024430 con 0x7f90d4107a80 2024-03-21T10:26:05.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.114+0000 7f90cea00700 1 --2- 172.21.15.31:0/3563453030 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f90bc039020 0x7f90bc03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:05.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.114+0000 7f90cea00700 1 -- 172.21.15.31:0/3563453030 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f90c804d270 con 0x7f90d4107a80 2024-03-21T10:26:05.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.114+0000 7f90d9400700 1 --2- 172.21.15.31:0/3563453030 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f90bc039020 0x7f90bc03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:05.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.115+0000 7f90d9400700 1 --2- 172.21.15.31:0/3563453030 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f90bc039020 0x7f90bc03b4e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f90c4006fd0 tx=0x7f90c4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:05.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.120+0000 7f90cea00700 1 -- 172.21.15.31:0/3563453030 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f90c800f2e0 con 0x7f90d4107a80 2024-03-21T10:26:05.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.156+0000 7f90cea00700 1 -- 172.21.15.31:0/3563453030 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 4) v1 ==== 46283+0+0 (secure 0 0 0) 0x7f90c802d3f0 con 0x7f90d4107a80 2024-03-21T10:26:05.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.428+0000 7f90dc800700 1 -- 172.21.15.31:0/3563453030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f90d41a12d0 con 0x7f90d4107a80 2024-03-21T10:26:05.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.429+0000 7f90cea00700 1 -- 172.21.15.31:0/3563453030 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f90d41a12d0 con 0x7f90d4107a80 2024-03-21T10:26:05.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.433+0000 7f90dc800700 1 -- 172.21.15.31:0/3563453030 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f90bc039020 msgr2=0x7f90bc03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:05.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.433+0000 7f90dc800700 1 --2- 172.21.15.31:0/3563453030 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f90bc039020 0x7f90bc03b4e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f90c4006fd0 tx=0x7f90c4006e40 comp rx=0 tx=0).stop 2024-03-21T10:26:05.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.433+0000 7f90dc800700 1 -- 172.21.15.31:0/3563453030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90d4107a80 msgr2=0x7f90d419dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:05.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.433+0000 7f90dc800700 1 --2- 172.21.15.31:0/3563453030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90d4107a80 0x7f90d419dc40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f90c8011670 tx=0x7f90c8011b60 comp rx=0 tx=0).stop 2024-03-21T10:26:05.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.433+0000 7f90dc800700 1 -- 172.21.15.31:0/3563453030 shutdown_connections 2024-03-21T10:26:05.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.433+0000 7f90dc800700 1 --2- 172.21.15.31:0/3563453030 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f90bc039020 0x7f90bc03b4e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:05.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.433+0000 7f90dc800700 1 --2- 172.21.15.31:0/3563453030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90d4107a80 0x7f90d419dc40 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:05.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.433+0000 7f90dc800700 1 -- 172.21.15.31:0/3563453030 >> 172.21.15.31:0/3563453030 conn(0x7f90d4103020 msgr2=0x7f90d4103b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:05.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.434+0000 7f90dc800700 1 -- 172.21.15.31:0/3563453030 shutdown_connections 2024-03-21T10:26:05.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:04.434+0000 7f90dc800700 1 -- 172.21.15.31:0/3563453030 wait complete. 2024-03-21T10:26:05.096 INFO:teuthology.orchestra.run.smithi031.stdout:mgr is available 2024-03-21T10:26:05.455 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:05 smithi031 ceph-mon[31673]: mgrmap e4: x(active, since 2s) 2024-03-21T10:26:05.455 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:05 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3563453030' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-21T10:26:06.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:06 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3376216447' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-21T10:26:06.780 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-03-21T10:26:06.780 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [global] 2024-03-21T10:26:06.780 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout fsid = 3847408c-e76d-11ee-95cd-87774f69a715 2024-03-21T10:26:06.780 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-21T10:26:06.780 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-21T10:26:06.780 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-21T10:26:06.780 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-21T10:26:06.780 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-03-21T10:26:06.780 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-21T10:26:06.781 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-21T10:26:06.781 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-03-21T10:26:06.781 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [osd] 2024-03-21T10:26:06.781 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-21T10:26:06.781 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-21T10:26:06.781 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-21T10:26:06.781 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-21T10:26:06.781 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-21T10:26:06.781 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.779+0000 7fa075000700 1 Processor -- start 2024-03-21T10:26:06.786 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.780+0000 7fa075000700 1 -- start start 2024-03-21T10:26:06.786 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.780+0000 7fa075000700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa070081870 0x7fa070081c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:06.787 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.780+0000 7fa075000700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0700821d0 con 0x7fa070081870 2024-03-21T10:26:06.787 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.781+0000 7fa06f400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa070081870 0x7fa070081c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:06.787 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.781+0000 7fa06f400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa070081870 0x7fa070081c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55872/0 (socket says 172.21.15.31:55872) 2024-03-21T10:26:06.787 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.781+0000 7fa06f400700 1 -- 172.21.15.31:0/3524571594 learned_addr learned my addr 172.21.15.31:0/3524571594 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:06.787 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.781+0000 7fa06f400700 1 -- 172.21.15.31:0/3524571594 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa07010f040 con 0x7fa070081870 2024-03-21T10:26:06.787 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.782+0000 7fa06f400700 1 --2- 172.21.15.31:0/3524571594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa070081870 0x7fa070081c90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa0600098a0 tx=0x7fa060009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1669cb5e9da700d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:06.787 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.782+0000 7fa06e000700 1 -- 172.21.15.31:0/3524571594 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa060004160 con 0x7fa070081870 2024-03-21T10:26:06.787 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.782+0000 7fa06e000700 1 -- 172.21.15.31:0/3524571594 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa060004300 con 0x7fa070081870 2024-03-21T10:26:06.787 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.782+0000 7fa06e000700 1 -- 172.21.15.31:0/3524571594 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa060011b50 con 0x7fa070081870 2024-03-21T10:26:06.787 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.782+0000 7fa075000700 1 -- 172.21.15.31:0/3524571594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa070081870 msgr2=0x7fa070081c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:06.787 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.782+0000 7fa075000700 1 --2- 172.21.15.31:0/3524571594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa070081870 0x7fa070081c90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa0600098a0 tx=0x7fa060009bb0 comp rx=0 tx=0).stop 2024-03-21T10:26:06.788 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.783+0000 7fa075000700 1 -- 172.21.15.31:0/3524571594 shutdown_connections 2024-03-21T10:26:06.788 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.783+0000 7fa075000700 1 --2- 172.21.15.31:0/3524571594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa070081870 0x7fa070081c90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:06.788 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.783+0000 7fa075000700 1 -- 172.21.15.31:0/3524571594 >> 172.21.15.31:0/3524571594 conn(0x7fa07007cee0 msgr2=0x7fa07007f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:06.788 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.783+0000 7fa075000700 1 -- 172.21.15.31:0/3524571594 shutdown_connections 2024-03-21T10:26:06.788 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.783+0000 7fa075000700 1 -- 172.21.15.31:0/3524571594 wait complete. 2024-03-21T10:26:06.788 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.784+0000 7fa075000700 1 Processor -- start 2024-03-21T10:26:06.788 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.784+0000 7fa075000700 1 -- start start 2024-03-21T10:26:06.788 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.784+0000 7fa075000700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa070081870 0x7fa0701a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:06.788 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.784+0000 7fa075000700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa070112300 con 0x7fa070081870 2024-03-21T10:26:06.788 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.785+0000 7fa06f400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa070081870 0x7fa0701a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:06.788 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.785+0000 7fa06f400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa070081870 0x7fa0701a1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55876/0 (socket says 172.21.15.31:55876) 2024-03-21T10:26:06.788 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.785+0000 7fa06f400700 1 -- 172.21.15.31:0/3376216447 learned_addr learned my addr 172.21.15.31:0/3376216447 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:06.788 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.785+0000 7fa06f400700 1 -- 172.21.15.31:0/3376216447 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0701a2390 con 0x7fa070081870 2024-03-21T10:26:06.788 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.785+0000 7fa06f400700 1 --2- 172.21.15.31:0/3376216447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa070081870 0x7fa0701a1e50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa060011b60 tx=0x7fa060011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:06.789 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.785+0000 7fa067e00700 1 -- 172.21.15.31:0/3376216447 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa060003970 con 0x7fa070081870 2024-03-21T10:26:06.789 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.786+0000 7fa067e00700 1 -- 172.21.15.31:0/3376216447 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa060003b10 con 0x7fa070081870 2024-03-21T10:26:06.789 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.786+0000 7fa075000700 1 -- 172.21.15.31:0/3376216447 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0701a2620 con 0x7fa070081870 2024-03-21T10:26:06.789 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.786+0000 7fa067e00700 1 -- 172.21.15.31:0/3376216447 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa060024610 con 0x7fa070081870 2024-03-21T10:26:06.789 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.786+0000 7fa075000700 1 -- 172.21.15.31:0/3376216447 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0701a5310 con 0x7fa070081870 2024-03-21T10:26:06.789 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.787+0000 7fa067e00700 1 -- 172.21.15.31:0/3376216447 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 4) v1 ==== 46283+0+0 (secure 0 0 0) 0x7fa06001a9f0 con 0x7fa070081870 2024-03-21T10:26:06.789 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.787+0000 7fa075000700 1 -- 172.21.15.31:0/3376216447 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa070112480 con 0x7fa070081870 2024-03-21T10:26:06.789 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.787+0000 7fa067e00700 1 --2- 172.21.15.31:0/3376216447 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7fa054039130 0x7fa05403b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:06.789 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.787+0000 7fa067e00700 1 -- 172.21.15.31:0/3376216447 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa060011e00 con 0x7fa070081870 2024-03-21T10:26:06.789 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.788+0000 7fa06ea00700 1 --2- 172.21.15.31:0/3376216447 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7fa054039130 0x7fa05403b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:06.789 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.788+0000 7fa06ea00700 1 --2- 172.21.15.31:0/3376216447 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7fa054039130 0x7fa05403b5f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa05c006fd0 tx=0x7fa05c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:06.789 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:05.793+0000 7fa067e00700 1 -- 172.21.15.31:0/3376216447 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa0600248c0 con 0x7fa070081870 2024-03-21T10:26:06.789 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:06.018+0000 7fa075000700 1 -- 172.21.15.31:0/3376216447 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fa0700637c0 con 0x7fa070081870 2024-03-21T10:26:06.789 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:06.020+0000 7fa067e00700 1 -- 172.21.15.31:0/3376216447 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fa06004f230 con 0x7fa070081870 2024-03-21T10:26:06.790 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:06.024+0000 7fa075000700 1 -- 172.21.15.31:0/3376216447 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7fa054039130 msgr2=0x7fa05403b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:06.790 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:06.024+0000 7fa075000700 1 --2- 172.21.15.31:0/3376216447 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7fa054039130 0x7fa05403b5f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa05c006fd0 tx=0x7fa05c006e40 comp rx=0 tx=0).stop 2024-03-21T10:26:06.790 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:06.024+0000 7fa075000700 1 -- 172.21.15.31:0/3376216447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa070081870 msgr2=0x7fa0701a1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:06.790 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:06.024+0000 7fa075000700 1 --2- 172.21.15.31:0/3376216447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa070081870 0x7fa0701a1e50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa060011b60 tx=0x7fa060011c40 comp rx=0 tx=0).stop 2024-03-21T10:26:06.790 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:06.024+0000 7fa075000700 1 -- 172.21.15.31:0/3376216447 shutdown_connections 2024-03-21T10:26:06.790 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:06.024+0000 7fa075000700 1 --2- 172.21.15.31:0/3376216447 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7fa054039130 0x7fa05403b5f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:06.790 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:06.024+0000 7fa075000700 1 --2- 172.21.15.31:0/3376216447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa070081870 0x7fa0701a1e50 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:06.790 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:06.024+0000 7fa075000700 1 -- 172.21.15.31:0/3376216447 >> 172.21.15.31:0/3376216447 conn(0x7fa07007cee0 msgr2=0x7fa07007d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:06.790 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:06.024+0000 7fa075000700 1 -- 172.21.15.31:0/3376216447 shutdown_connections 2024-03-21T10:26:06.790 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:06.024+0000 7fa075000700 1 -- 172.21.15.31:0/3376216447 wait complete. 2024-03-21T10:26:06.790 INFO:teuthology.orchestra.run.smithi031.stdout:Enabling cephadm module... 2024-03-21T10:26:08.058 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:07 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3295544462' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-21T10:26:08.058 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:07 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: ignoring --setuser ceph since I am not root 2024-03-21T10:26:08.058 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:07 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: ignoring --setgroup ceph since I am not root 2024-03-21T10:26:08.766 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:08 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:08.641+0000 7f7140efd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T10:26:08.934 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.464+0000 7f6121000700 1 Processor -- start 2024-03-21T10:26:08.934 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.464+0000 7f6121000700 1 -- start start 2024-03-21T10:26:08.934 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.465+0000 7f6121000700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f611c07e840 0x7f611c07ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:08.934 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.465+0000 7f6121000700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611c07f230 con 0x7f611c07e840 2024-03-21T10:26:08.934 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.465+0000 7f6119e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f611c07e840 0x7f611c07ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:08.935 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.465+0000 7f6119e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f611c07e840 0x7f611c07ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44254/0 (socket says 172.21.15.31:44254) 2024-03-21T10:26:08.935 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.465+0000 7f6119e00700 1 -- 172.21.15.31:0/1469660580 learned_addr learned my addr 172.21.15.31:0/1469660580 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:08.935 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.466+0000 7f6119e00700 1 -- 172.21.15.31:0/1469660580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f611c07fa90 con 0x7f611c07e840 2024-03-21T10:26:08.935 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.466+0000 7f6119e00700 1 --2- 172.21.15.31:0/1469660580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f611c07e840 0x7f611c07ec60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f610c0098a0 tx=0x7f610c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d67deb8deff32d8d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:08.935 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.466+0000 7f6118a00700 1 -- 172.21.15.31:0/1469660580 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f610c004160 con 0x7f611c07e840 2024-03-21T10:26:08.935 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.467+0000 7f6118a00700 1 -- 172.21.15.31:0/1469660580 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f610c004300 con 0x7f611c07e840 2024-03-21T10:26:08.938 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.467+0000 7f6118a00700 1 -- 172.21.15.31:0/1469660580 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f610c011b50 con 0x7f611c07e840 2024-03-21T10:26:08.938 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.467+0000 7f6121000700 1 -- 172.21.15.31:0/1469660580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f611c07e840 msgr2=0x7f611c07ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:08.938 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.467+0000 7f6121000700 1 --2- 172.21.15.31:0/1469660580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f611c07e840 0x7f611c07ec60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f610c0098a0 tx=0x7f610c009bb0 comp rx=0 tx=0).stop 2024-03-21T10:26:08.938 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.467+0000 7f6121000700 1 -- 172.21.15.31:0/1469660580 shutdown_connections 2024-03-21T10:26:08.939 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.467+0000 7f6121000700 1 --2- 172.21.15.31:0/1469660580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f611c07e840 0x7f611c07ec60 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:08.939 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.467+0000 7f6121000700 1 -- 172.21.15.31:0/1469660580 >> 172.21.15.31:0/1469660580 conn(0x7f611c07d040 msgr2=0x7f611c07d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:08.939 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.467+0000 7f6121000700 1 -- 172.21.15.31:0/1469660580 shutdown_connections 2024-03-21T10:26:08.939 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.467+0000 7f6121000700 1 -- 172.21.15.31:0/1469660580 wait complete. 2024-03-21T10:26:08.939 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.468+0000 7f6121000700 1 Processor -- start 2024-03-21T10:26:08.939 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.469+0000 7f6121000700 1 -- start start 2024-03-21T10:26:08.939 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.469+0000 7f6121000700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f611c07e840 0x7f611c1a1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:08.939 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.469+0000 7f6121000700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611c080e50 con 0x7f611c07e840 2024-03-21T10:26:08.939 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.469+0000 7f6119e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f611c07e840 0x7f611c1a1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:08.939 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.469+0000 7f6119e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f611c07e840 0x7f611c1a1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44268/0 (socket says 172.21.15.31:44268) 2024-03-21T10:26:08.939 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.469+0000 7f6119e00700 1 -- 172.21.15.31:0/3295544462 learned_addr learned my addr 172.21.15.31:0/3295544462 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:08.939 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.469+0000 7f6119e00700 1 -- 172.21.15.31:0/3295544462 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f611c1a2510 con 0x7f611c07e840 2024-03-21T10:26:08.939 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.470+0000 7f6119e00700 1 --2- 172.21.15.31:0/3295544462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f611c07e840 0x7f611c1a1fd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f610c011b60 tx=0x7f610c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:08.939 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.470+0000 7f6112a00700 1 -- 172.21.15.31:0/3295544462 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f610c003b20 con 0x7f611c07e840 2024-03-21T10:26:08.940 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.470+0000 7f6112a00700 1 -- 172.21.15.31:0/3295544462 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f610c003cc0 con 0x7f611c07e840 2024-03-21T10:26:08.940 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.470+0000 7f6121000700 1 -- 172.21.15.31:0/3295544462 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f611c1a27a0 con 0x7f611c07e840 2024-03-21T10:26:08.940 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.470+0000 7f6112a00700 1 -- 172.21.15.31:0/3295544462 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f610c024840 con 0x7f611c07e840 2024-03-21T10:26:08.940 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.470+0000 7f6121000700 1 -- 172.21.15.31:0/3295544462 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f611c1a5490 con 0x7f611c07e840 2024-03-21T10:26:08.940 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.471+0000 7f6112a00700 1 -- 172.21.15.31:0/3295544462 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 4) v1 ==== 46283+0+0 (secure 0 0 0) 0x7f610c022020 con 0x7f611c07e840 2024-03-21T10:26:08.940 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.472+0000 7f6121000700 1 -- 172.21.15.31:0/3295544462 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f611c0637c0 con 0x7f611c07e840 2024-03-21T10:26:08.940 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.472+0000 7f6112a00700 1 --2- 172.21.15.31:0/3295544462 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f6100039180 0x7f610003b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:08.940 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.472+0000 7f6112a00700 1 -- 172.21.15.31:0/3295544462 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f610c011e00 con 0x7f611c07e840 2024-03-21T10:26:08.940 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.472+0000 7f6119400700 1 --2- 172.21.15.31:0/3295544462 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f6100039180 0x7f610003b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:08.942 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.473+0000 7f6119400700 1 --2- 172.21.15.31:0/3295544462 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f6100039180 0x7f610003b640 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6108006fd0 tx=0x7f6108006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:08.942 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.477+0000 7f6112a00700 1 -- 172.21.15.31:0/3295544462 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f610c04cc90 con 0x7f611c07e840 2024-03-21T10:26:08.942 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.763+0000 7f6121000700 1 -- 172.21.15.31:0/3295544462 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f611c0008d0 con 0x7f611c07e840 2024-03-21T10:26:08.942 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.817+0000 7f6112a00700 1 -- 172.21.15.31:0/3295544462 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f610c01a7b0 con 0x7f611c07e840 2024-03-21T10:26:08.942 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.820+0000 7f6121000700 1 -- 172.21.15.31:0/3295544462 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f6100039180 msgr2=0x7f610003b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:08.942 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.820+0000 7f6121000700 1 --2- 172.21.15.31:0/3295544462 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f6100039180 0x7f610003b640 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6108006fd0 tx=0x7f6108006e40 comp rx=0 tx=0).stop 2024-03-21T10:26:08.942 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.820+0000 7f6112a00700 1 -- 172.21.15.31:0/3295544462 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mgrmap(e 5) v1 ==== 46294+0+0 (secure 0 0 0) 0x7f610c00b3e0 con 0x7f611c07e840 2024-03-21T10:26:08.942 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.820+0000 7f6121000700 1 -- 172.21.15.31:0/3295544462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f611c07e840 msgr2=0x7f611c1a1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:08.942 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.820+0000 7f6121000700 1 --2- 172.21.15.31:0/3295544462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f611c07e840 0x7f611c1a1fd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f610c011b60 tx=0x7f610c011c40 comp rx=0 tx=0).stop 2024-03-21T10:26:08.942 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.821+0000 7f6121000700 1 -- 172.21.15.31:0/3295544462 shutdown_connections 2024-03-21T10:26:08.943 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.821+0000 7f6121000700 1 --2- 172.21.15.31:0/3295544462 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f6100039180 0x7f610003b640 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:08.943 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.821+0000 7f6121000700 1 --2- 172.21.15.31:0/3295544462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f611c07e840 0x7f611c1a1fd0 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f610c011b60 tx=0x7f610c011c40 comp rx=0 tx=0).stop 2024-03-21T10:26:08.943 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.821+0000 7f6121000700 1 -- 172.21.15.31:0/3295544462 >> 172.21.15.31:0/3295544462 conn(0x7f611c07d040 msgr2=0x7f611c198bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:08.943 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.822+0000 7f6121000700 1 -- 172.21.15.31:0/3295544462 shutdown_connections 2024-03-21T10:26:08.943 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:07.822+0000 7f6121000700 1 -- 172.21.15.31:0/3295544462 wait complete. 2024-03-21T10:26:09.057 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:08 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:08.765+0000 7f7140efd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T10:26:09.058 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:08 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3295544462' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-21T10:26:09.058 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:08 smithi031 ceph-mon[31673]: mgrmap e5: x(active, since 5s) 2024-03-21T10:26:09.807 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:09 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:09.596+0000 7f7140efd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T10:26:09.808 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:09 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:09.724+0000 7f7140efd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T10:26:10.156 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:09 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/645981291' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-21T10:26:10.156 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:09 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:09.907+0000 7f7140efd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T10:26:10.156 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:10 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:10.040+0000 7f7140efd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T10:26:10.156 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:10 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:10.155+0000 7f7140efd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T10:26:10.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-03-21T10:26:10.686 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-21T10:26:10.686 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-21T10:26:10.686 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-21T10:26:10.686 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-21T10:26:10.686 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-03-21T10:26:10.686 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.632+0000 7f64bec00700 1 Processor -- start 2024-03-21T10:26:10.686 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.633+0000 7f64bec00700 1 -- start start 2024-03-21T10:26:10.686 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.633+0000 7f64bec00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64b8107a80 0x7f64b8107ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:10.686 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.633+0000 7f64bec00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b8108470 con 0x7f64b8107a80 2024-03-21T10:26:10.686 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.633+0000 7f64b7e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64b8107a80 0x7f64b8107ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:10.686 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.634+0000 7f64b7e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64b8107a80 0x7f64b8107ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44300/0 (socket says 172.21.15.31:44300) 2024-03-21T10:26:10.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.634+0000 7f64b7e00700 1 -- 172.21.15.31:0/1294620742 learned_addr learned my addr 172.21.15.31:0/1294620742 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:10.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.634+0000 7f64b7e00700 1 -- 172.21.15.31:0/1294620742 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b8108c80 con 0x7f64b8107a80 2024-03-21T10:26:10.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.634+0000 7f64b7e00700 1 --2- 172.21.15.31:0/1294620742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64b8107a80 0x7f64b8107ea0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f64a80098a0 tx=0x7f64a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bdfd25a17588280 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:10.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.635+0000 7f64b6a00700 1 -- 172.21.15.31:0/1294620742 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f64a8004160 con 0x7f64b8107a80 2024-03-21T10:26:10.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.635+0000 7f64b6a00700 1 -- 172.21.15.31:0/1294620742 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f64a8004300 con 0x7f64b8107a80 2024-03-21T10:26:10.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.635+0000 7f64b6a00700 1 -- 172.21.15.31:0/1294620742 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f64a8011b50 con 0x7f64b8107a80 2024-03-21T10:26:10.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.635+0000 7f64bec00700 1 -- 172.21.15.31:0/1294620742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64b8107a80 msgr2=0x7f64b8107ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:10.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.635+0000 7f64bec00700 1 --2- 172.21.15.31:0/1294620742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64b8107a80 0x7f64b8107ea0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f64a80098a0 tx=0x7f64a8009bb0 comp rx=0 tx=0).stop 2024-03-21T10:26:10.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.635+0000 7f64bec00700 1 -- 172.21.15.31:0/1294620742 shutdown_connections 2024-03-21T10:26:10.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.636+0000 7f64bec00700 1 --2- 172.21.15.31:0/1294620742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64b8107a80 0x7f64b8107ea0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:10.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.636+0000 7f64bec00700 1 -- 172.21.15.31:0/1294620742 >> 172.21.15.31:0/1294620742 conn(0x7f64b8103080 msgr2=0x7f64b81054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:10.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.636+0000 7f64bec00700 1 -- 172.21.15.31:0/1294620742 shutdown_connections 2024-03-21T10:26:10.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.636+0000 7f64bec00700 1 -- 172.21.15.31:0/1294620742 wait complete. 2024-03-21T10:26:10.688 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.637+0000 7f64bec00700 1 Processor -- start 2024-03-21T10:26:10.688 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.637+0000 7f64bec00700 1 -- start start 2024-03-21T10:26:10.688 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.637+0000 7f64bec00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64b8107a80 0x7f64b807db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:10.688 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.637+0000 7f64bec00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b810c030 con 0x7f64b8107a80 2024-03-21T10:26:10.688 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.638+0000 7f64b7e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64b8107a80 0x7f64b807db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:10.688 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.638+0000 7f64b7e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64b8107a80 0x7f64b807db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44316/0 (socket says 172.21.15.31:44316) 2024-03-21T10:26:10.688 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.638+0000 7f64b7e00700 1 -- 172.21.15.31:0/645981291 learned_addr learned my addr 172.21.15.31:0/645981291 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:10.688 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.638+0000 7f64b7e00700 1 -- 172.21.15.31:0/645981291 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b807e090 con 0x7f64b8107a80 2024-03-21T10:26:10.688 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.638+0000 7f64b7e00700 1 --2- 172.21.15.31:0/645981291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64b8107a80 0x7f64b807db50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f64a8000c00 tx=0x7f64a8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:10.688 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.639+0000 7f64b4c00700 1 -- 172.21.15.31:0/645981291 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f64a8003b20 con 0x7f64b8107a80 2024-03-21T10:26:10.688 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.639+0000 7f64b4c00700 1 -- 172.21.15.31:0/645981291 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f64a8003cc0 con 0x7f64b8107a80 2024-03-21T10:26:10.688 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.639+0000 7f64bec00700 1 -- 172.21.15.31:0/645981291 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b807c280 con 0x7f64b8107a80 2024-03-21T10:26:10.688 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.639+0000 7f64b4c00700 1 -- 172.21.15.31:0/645981291 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f64a8024910 con 0x7f64b8107a80 2024-03-21T10:26:10.688 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.639+0000 7f64bec00700 1 -- 172.21.15.31:0/645981291 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b807c7c0 con 0x7f64b8107a80 2024-03-21T10:26:10.689 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.640+0000 7f64b4c00700 1 -- 172.21.15.31:0/645981291 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 46294+0+0 (secure 0 0 0) 0x7f64a8022020 con 0x7f64b8107a80 2024-03-21T10:26:10.689 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.640+0000 7f64bec00700 1 -- 172.21.15.31:0/645981291 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b80637c0 con 0x7f64b8107a80 2024-03-21T10:26:10.689 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.640+0000 7f64b4c00700 1 --2- 172.21.15.31:0/645981291 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f649c0391d0 0x7f649c03b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:10.689 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.640+0000 7f64b4c00700 1 -- 172.21.15.31:0/645981291 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f64a8011e00 con 0x7f64b8107a80 2024-03-21T10:26:10.689 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.640+0000 7f64b7400700 1 -- 172.21.15.31:0/645981291 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f649c0391d0 msgr2=0x7f649c03b690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/1981437651 2024-03-21T10:26:10.689 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.641+0000 7f64b7400700 1 --2- 172.21.15.31:0/645981291 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f649c0391d0 0x7f649c03b690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-21T10:26:10.689 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.646+0000 7f64b4c00700 1 -- 172.21.15.31:0/645981291 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f64a8016260 con 0x7f64b8107a80 2024-03-21T10:26:10.689 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.842+0000 7f64b7400700 1 -- 172.21.15.31:0/645981291 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f649c0391d0 msgr2=0x7f649c03b690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/1981437651 2024-03-21T10:26:10.689 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.842+0000 7f64b7400700 1 --2- 172.21.15.31:0/645981291 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f649c0391d0 0x7f649c03b690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-21T10:26:10.689 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.929+0000 7f64bec00700 1 -- 172.21.15.31:0/645981291 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f64b807d0f0 con 0x7f64b8107a80 2024-03-21T10:26:10.689 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.930+0000 7f64b4c00700 1 -- 172.21.15.31:0/645981291 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f64a804cbe0 con 0x7f64b8107a80 2024-03-21T10:26:10.689 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.933+0000 7f64bec00700 1 -- 172.21.15.31:0/645981291 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f649c0391d0 msgr2=0x7f649c03b690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T10:26:10.689 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.933+0000 7f64bec00700 1 --2- 172.21.15.31:0/645981291 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f649c0391d0 0x7f649c03b690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:10.690 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.933+0000 7f64bec00700 1 -- 172.21.15.31:0/645981291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64b8107a80 msgr2=0x7f64b807db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:10.690 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.933+0000 7f64bec00700 1 --2- 172.21.15.31:0/645981291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64b8107a80 0x7f64b807db50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f64a8000c00 tx=0x7f64a8011c10 comp rx=0 tx=0).stop 2024-03-21T10:26:10.690 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.933+0000 7f64bec00700 1 -- 172.21.15.31:0/645981291 shutdown_connections 2024-03-21T10:26:10.690 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.933+0000 7f64bec00700 1 --2- 172.21.15.31:0/645981291 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f649c0391d0 0x7f649c03b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:10.690 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.933+0000 7f64bec00700 1 --2- 172.21.15.31:0/645981291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64b8107a80 0x7f64b807db50 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:10.690 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.933+0000 7f64bec00700 1 -- 172.21.15.31:0/645981291 >> 172.21.15.31:0/645981291 conn(0x7f64b8103080 msgr2=0x7f64b8103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:10.690 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.934+0000 7f64bec00700 1 -- 172.21.15.31:0/645981291 shutdown_connections 2024-03-21T10:26:10.690 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:09.934+0000 7f64bec00700 1 -- 172.21.15.31:0/645981291 wait complete. 2024-03-21T10:26:10.690 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for the mgr to restart... 2024-03-21T10:26:10.690 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr epoch 5... 2024-03-21T10:26:10.769 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:10 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:10.503+0000 7f7140efd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T10:26:10.769 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:10 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:10.648+0000 7f7140efd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T10:26:11.058 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:10 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:10.768+0000 7f7140efd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T10:26:11.058 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:10 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:10.903+0000 7f7140efd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T10:26:11.463 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:11 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:11.133+0000 7f7140efd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T10:26:12.808 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:12 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:12.460+0000 7f7140efd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T10:26:12.808 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:12 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:12.638+0000 7f7140efd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T10:26:13.457 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:13 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:13.069+0000 7f7140efd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T10:26:13.457 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:13 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:13.191+0000 7f7140efd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T10:26:13.457 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:13 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:13.455+0000 7f7140efd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T10:26:14.057 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:13 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:13.709+0000 7f7140efd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T10:26:14.422 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:14 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:14.097+0000 7f7140efd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T10:26:14.422 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:14 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:14.421+0000 7f7140efd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T10:26:15.057 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:14 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:14.736+0000 7f7140efd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T10:26:16.307 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:15 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:15.807+0000 7f7140efd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T10:26:16.807 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:16 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:16.527+0000 7f7140efd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T10:26:16.808 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:16 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:16.662+0000 7f7140efd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T10:26:17.308 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:16 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:16.955+0000 7f7140efd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T10:26:17.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:17 smithi031 ceph-mon[31673]: Active manager daemon x restarted 2024-03-21T10:26:17.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:17 smithi031 ceph-mon[31673]: Activating manager daemon x 2024-03-21T10:26:17.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:17 smithi031 ceph-mon[31673]: osdmap e2: 0 total, 0 up, 0 in 2024-03-21T10:26:17.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:17 smithi031 ceph-mon[31673]: mgrmap e6: x(active, starting, since 0.00870796s) 2024-03-21T10:26:17.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:17 smithi031 ceph-mon[31673]: from='mgr.14120 172.21.15.31:0/4221700501' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T10:26:17.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:17 smithi031 ceph-mon[31673]: from='mgr.14120 172.21.15.31:0/4221700501' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T10:26:17.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:17 smithi031 ceph-mon[31673]: from='mgr.14120 172.21.15.31:0/4221700501' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T10:26:17.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:17 smithi031 ceph-mon[31673]: from='mgr.14120 172.21.15.31:0/4221700501' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T10:26:17.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:17 smithi031 ceph-mon[31673]: from='mgr.14120 172.21.15.31:0/4221700501' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T10:26:17.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:17 smithi031 ceph-mon[31673]: Manager daemon x is now available 2024-03-21T10:26:18.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:18 smithi031 ceph-mon[31673]: Found migration_current of "None". Setting to last migration. 2024-03-21T10:26:18.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:18 smithi031 ceph-mon[31673]: from='mgr.14120 172.21.15.31:0/4221700501' entity='mgr.x' 2024-03-21T10:26:18.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:18 smithi031 ceph-mon[31673]: from='mgr.14120 172.21.15.31:0/4221700501' entity='mgr.x' 2024-03-21T10:26:18.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:18 smithi031 ceph-mon[31673]: from='mgr.14120 172.21.15.31:0/4221700501' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:26:18.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:18 smithi031 ceph-mon[31673]: from='mgr.14120 172.21.15.31:0/4221700501' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:26:18.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:18 smithi031 ceph-mon[31673]: from='mgr.14120 172.21.15.31:0/4221700501' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T10:26:18.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:18 smithi031 ceph-mon[31673]: from='mgr.14120 172.21.15.31:0/4221700501' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T10:26:18.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:18 smithi031 ceph-mon[31673]: from='mgr.14120 172.21.15.31:0/4221700501' entity='mgr.x' 2024-03-21T10:26:18.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:18 smithi031 ceph-mon[31673]: from='mgr.14120 172.21.15.31:0/4221700501' entity='mgr.x' 2024-03-21T10:26:18.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:18 smithi031 ceph-mon[31673]: mgrmap e7: x(active, since 1.01432s) 2024-03-21T10:26:18.682 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-03-21T10:26:18.682 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-21T10:26:18.682 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-21T10:26:18.682 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-03-21T10:26:18.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.253+0000 7f0e51000700 1 Processor -- start 2024-03-21T10:26:18.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.253+0000 7f0e51000700 1 -- start start 2024-03-21T10:26:18.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.253+0000 7f0e51000700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e4c1085a0 0x7f0e4c10a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:18.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.253+0000 7f0e51000700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4c07bdd0 con 0x7f0e4c1085a0 2024-03-21T10:26:18.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.254+0000 7f0e49e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e4c1085a0 0x7f0e4c10a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:18.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.254+0000 7f0e49e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e4c1085a0 0x7f0e4c10a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44332/0 (socket says 172.21.15.31:44332) 2024-03-21T10:26:18.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.254+0000 7f0e49e00700 1 -- 172.21.15.31:0/606760749 learned_addr learned my addr 172.21.15.31:0/606760749 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:18.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.254+0000 7f0e49e00700 1 -- 172.21.15.31:0/606760749 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e4c10aed0 con 0x7f0e4c1085a0 2024-03-21T10:26:18.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.255+0000 7f0e49e00700 1 --2- 172.21.15.31:0/606760749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e4c1085a0 0x7f0e4c10a990 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c0098a0 tx=0x7f0e3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f02cf14e6c7cb4b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:18.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.255+0000 7f0e48a00700 1 -- 172.21.15.31:0/606760749 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0e3c004160 con 0x7f0e4c1085a0 2024-03-21T10:26:18.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.255+0000 7f0e48a00700 1 -- 172.21.15.31:0/606760749 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e3c004300 con 0x7f0e4c1085a0 2024-03-21T10:26:18.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.255+0000 7f0e48a00700 1 -- 172.21.15.31:0/606760749 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0e3c011b50 con 0x7f0e4c1085a0 2024-03-21T10:26:18.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.256+0000 7f0e51000700 1 -- 172.21.15.31:0/606760749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e4c1085a0 msgr2=0x7f0e4c10a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:18.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.256+0000 7f0e51000700 1 --2- 172.21.15.31:0/606760749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e4c1085a0 0x7f0e4c10a990 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c0098a0 tx=0x7f0e3c009bb0 comp rx=0 tx=0).stop 2024-03-21T10:26:18.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.256+0000 7f0e51000700 1 -- 172.21.15.31:0/606760749 shutdown_connections 2024-03-21T10:26:18.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.256+0000 7f0e51000700 1 --2- 172.21.15.31:0/606760749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e4c1085a0 0x7f0e4c10a990 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:18.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.256+0000 7f0e51000700 1 -- 172.21.15.31:0/606760749 >> 172.21.15.31:0/606760749 conn(0x7f0e4c101f70 msgr2=0x7f0e4c104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:18.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.256+0000 7f0e51000700 1 -- 172.21.15.31:0/606760749 shutdown_connections 2024-03-21T10:26:18.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.256+0000 7f0e51000700 1 -- 172.21.15.31:0/606760749 wait complete. 2024-03-21T10:26:18.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.257+0000 7f0e51000700 1 Processor -- start 2024-03-21T10:26:18.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.258+0000 7f0e51000700 1 -- start start 2024-03-21T10:26:18.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.258+0000 7f0e51000700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e4c1085a0 0x7f0e4c19b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:18.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.258+0000 7f0e51000700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4c10be40 con 0x7f0e4c1085a0 2024-03-21T10:26:18.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.258+0000 7f0e49e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e4c1085a0 0x7f0e4c19b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:18.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.258+0000 7f0e49e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e4c1085a0 0x7f0e4c19b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44342/0 (socket says 172.21.15.31:44342) 2024-03-21T10:26:18.689 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.258+0000 7f0e49e00700 1 -- 172.21.15.31:0/3623645577 learned_addr learned my addr 172.21.15.31:0/3623645577 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:18.689 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.259+0000 7f0e49e00700 1 -- 172.21.15.31:0/3623645577 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e4c19bf20 con 0x7f0e4c1085a0 2024-03-21T10:26:18.689 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.259+0000 7f0e49e00700 1 --2- 172.21.15.31:0/3623645577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e4c1085a0 0x7f0e4c19b9e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c011b60 tx=0x7f0e3c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:18.689 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.259+0000 7f0e42a00700 1 -- 172.21.15.31:0/3623645577 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0e3c003b20 con 0x7f0e4c1085a0 2024-03-21T10:26:18.689 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.259+0000 7f0e42a00700 1 -- 172.21.15.31:0/3623645577 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e3c003cc0 con 0x7f0e4c1085a0 2024-03-21T10:26:18.690 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.259+0000 7f0e42a00700 1 -- 172.21.15.31:0/3623645577 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0e3c024840 con 0x7f0e4c1085a0 2024-03-21T10:26:18.690 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.260+0000 7f0e51000700 1 -- 172.21.15.31:0/3623645577 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e4c19c1b0 con 0x7f0e4c1085a0 2024-03-21T10:26:18.690 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.260+0000 7f0e51000700 1 -- 172.21.15.31:0/3623645577 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e4c19eea0 con 0x7f0e4c1085a0 2024-03-21T10:26:18.690 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.261+0000 7f0e42a00700 1 -- 172.21.15.31:0/3623645577 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 46294+0+0 (secure 0 0 0) 0x7f0e3c022020 con 0x7f0e4c1085a0 2024-03-21T10:26:18.690 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.261+0000 7f0e42a00700 1 --2- 172.21.15.31:0/3623645577 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f0e30039180 0x7f0e3003b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:18.690 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.261+0000 7f0e42a00700 1 -- 172.21.15.31:0/3623645577 --> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0e3003bd50 con 0x7f0e30039180 2024-03-21T10:26:18.690 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.261+0000 7f0e42a00700 1 -- 172.21.15.31:0/3623645577 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0e3c011e00 con 0x7f0e4c1085a0 2024-03-21T10:26:18.691 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.261+0000 7f0e49400700 1 -- 172.21.15.31:0/3623645577 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f0e30039180 msgr2=0x7f0e3003b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/1981437651 2024-03-21T10:26:18.691 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.261+0000 7f0e49400700 1 --2- 172.21.15.31:0/3623645577 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f0e30039180 0x7f0e3003b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-21T10:26:18.694 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.462+0000 7f0e49400700 1 -- 172.21.15.31:0/3623645577 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f0e30039180 msgr2=0x7f0e3003b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/1981437651 2024-03-21T10:26:18.694 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.462+0000 7f0e49400700 1 --2- 172.21.15.31:0/3623645577 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f0e30039180 0x7f0e3003b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-21T10:26:18.694 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.863+0000 7f0e49400700 1 -- 172.21.15.31:0/3623645577 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f0e30039180 msgr2=0x7f0e3003b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/1981437651 2024-03-21T10:26:18.694 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:11.863+0000 7f0e49400700 1 --2- 172.21.15.31:0/3623645577 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f0e30039180 0x7f0e3003b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-21T10:26:18.694 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:12.664+0000 7f0e49400700 1 -- 172.21.15.31:0/3623645577 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f0e30039180 msgr2=0x7f0e3003b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/1981437651 2024-03-21T10:26:18.694 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:12.664+0000 7f0e49400700 1 --2- 172.21.15.31:0/3623645577 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f0e30039180 0x7f0e3003b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-21T10:26:18.694 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:14.266+0000 7f0e49400700 1 -- 172.21.15.31:0/3623645577 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f0e30039180 msgr2=0x7f0e3003b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/1981437651 2024-03-21T10:26:18.694 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:14.266+0000 7f0e49400700 1 --2- 172.21.15.31:0/3623645577 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f0e30039180 0x7f0e3003b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-21T10:26:18.694 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:16.963+0000 7f0e42a00700 1 -- 172.21.15.31:0/3623645577 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mgrmap(e 6) v1 ==== 46061+0+0 (secure 0 0 0) 0x7f0e3c024af0 con 0x7f0e4c1085a0 2024-03-21T10:26:18.694 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:16.963+0000 7f0e42a00700 1 -- 172.21.15.31:0/3623645577 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f0e30039180 msgr2=0x7f0e3003b640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T10:26:18.694 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:16.963+0000 7f0e42a00700 1 --2- 172.21.15.31:0/3623645577 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f0e30039180 0x7f0e3003b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:18.695 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:17.966+0000 7f0e42a00700 1 -- 172.21.15.31:0/3623645577 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 7) v1 ==== 46188+0+0 (secure 0 0 0) 0x7f0e3c04ec50 con 0x7f0e4c1085a0 2024-03-21T10:26:18.695 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:17.967+0000 7f0e42a00700 1 --2- 172.21.15.31:0/3623645577 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f0e3003ca70 0x7f0e3003ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:18.695 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:17.967+0000 7f0e42a00700 1 -- 172.21.15.31:0/3623645577 --> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0e3c018e50 con 0x7f0e3003ca70 2024-03-21T10:26:18.695 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:17.970+0000 7f0e49400700 1 --2- 172.21.15.31:0/3623645577 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f0e3003ca70 0x7f0e3003ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:18.695 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:17.972+0000 7f0e49400700 1 --2- 172.21.15.31:0/3623645577 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f0e3003ca70 0x7f0e3003ee60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0e38003a10 tx=0x7f0e38009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:18.695 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:17.975+0000 7f0e42a00700 1 -- 172.21.15.31:0/3623645577 <== mgr.14120 v2:172.21.15.31:6800/4085461895 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f0e3c018e50 con 0x7f0e3003ca70 2024-03-21T10:26:18.695 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:17.988+0000 7f0e51000700 1 -- 172.21.15.31:0/3623645577 --> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f0e4c0637c0 con 0x7f0e3003ca70 2024-03-21T10:26:18.695 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:17.988+0000 7f0e42a00700 1 -- 172.21.15.31:0/3623645577 <== mgr.14120 v2:172.21.15.31:6800/4085461895 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f0e4c0637c0 con 0x7f0e3003ca70 2024-03-21T10:26:18.695 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:17.989+0000 7f0e51000700 1 -- 172.21.15.31:0/3623645577 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f0e3003ca70 msgr2=0x7f0e3003ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:18.695 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:17.989+0000 7f0e51000700 1 --2- 172.21.15.31:0/3623645577 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f0e3003ca70 0x7f0e3003ee60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0e38003a10 tx=0x7f0e38009290 comp rx=0 tx=0).stop 2024-03-21T10:26:18.695 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:17.989+0000 7f0e51000700 1 -- 172.21.15.31:0/3623645577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e4c1085a0 msgr2=0x7f0e4c19b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:18.695 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:17.989+0000 7f0e51000700 1 --2- 172.21.15.31:0/3623645577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e4c1085a0 0x7f0e4c19b9e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c011b60 tx=0x7f0e3c011c40 comp rx=0 tx=0).stop 2024-03-21T10:26:18.696 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:17.989+0000 7f0e51000700 1 -- 172.21.15.31:0/3623645577 shutdown_connections 2024-03-21T10:26:18.697 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:17.989+0000 7f0e51000700 1 --2- 172.21.15.31:0/3623645577 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f0e3003ca70 0x7f0e3003ee60 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:18.697 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:17.989+0000 7f0e51000700 1 --2- 172.21.15.31:0/3623645577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e4c1085a0 0x7f0e4c19b9e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:18.697 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:17.989+0000 7f0e51000700 1 --2- 172.21.15.31:0/3623645577 >> [v2:172.21.15.31:6800/1981437651,v1:172.21.15.31:6801/1981437651] conn(0x7f0e30039180 0x7f0e3003b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:18.697 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:17.989+0000 7f0e51000700 1 -- 172.21.15.31:0/3623645577 >> 172.21.15.31:0/3623645577 conn(0x7f0e4c101f70 msgr2=0x7f0e4c102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:18.697 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:17.990+0000 7f0e51000700 1 -- 172.21.15.31:0/3623645577 shutdown_connections 2024-03-21T10:26:18.697 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:17.990+0000 7f0e51000700 1 -- 172.21.15.31:0/3623645577 wait complete. 2024-03-21T10:26:18.697 INFO:teuthology.orchestra.run.smithi031.stdout:mgr epoch 5 is available 2024-03-21T10:26:18.697 INFO:teuthology.orchestra.run.smithi031.stdout:Setting orchestrator backend to cephadm... 2024-03-21T10:26:19.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:19 smithi031 ceph-mon[31673]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-21T10:26:19.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:19 smithi031 ceph-mon[31673]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-21T10:26:19.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:19 smithi031 ceph-mon[31673]: [21/Mar/2024:10:26:18] ENGINE Bus STARTING 2024-03-21T10:26:19.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:19 smithi031 ceph-mon[31673]: [21/Mar/2024:10:26:18] ENGINE Serving on https://172.21.15.31:7150 2024-03-21T10:26:19.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:19 smithi031 ceph-mon[31673]: [21/Mar/2024:10:26:18] ENGINE Serving on http://172.21.15.31:8765 2024-03-21T10:26:19.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:19 smithi031 ceph-mon[31673]: [21/Mar/2024:10:26:18] ENGINE Bus STARTED 2024-03-21T10:26:19.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:19 smithi031 ceph-mon[31673]: from='mgr.14120 172.21.15.31:0/4221700501' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:26:20.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:20 smithi031 ceph-mon[31673]: mgrmap e8: x(active, since 2s) 2024-03-21T10:26:20.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:20 smithi031 ceph-mon[31673]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:26:20.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:20 smithi031 ceph-mon[31673]: from='mgr.14120 172.21.15.31:0/4221700501' entity='mgr.x' 2024-03-21T10:26:20.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:20 smithi031 ceph-mon[31673]: from='mgr.14120 172.21.15.31:0/4221700501' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:26:20.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.335+0000 7f51ca600700 1 Processor -- start 2024-03-21T10:26:20.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.336+0000 7f51ca600700 1 -- start start 2024-03-21T10:26:20.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.336+0000 7f51ca600700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51c407de60 0x7f51c407c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:20.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.336+0000 7f51ca600700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c407c890 con 0x7f51c407de60 2024-03-21T10:26:20.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.336+0000 7f51c3400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51c407de60 0x7f51c407c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:20.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.337+0000 7f51c3400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51c407de60 0x7f51c407c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38754/0 (socket says 172.21.15.31:38754) 2024-03-21T10:26:20.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.337+0000 7f51c3400700 1 -- 172.21.15.31:0/1919250851 learned_addr learned my addr 172.21.15.31:0/1919250851 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:20.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.337+0000 7f51c3400700 1 -- 172.21.15.31:0/1919250851 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c407ca10 con 0x7f51c407de60 2024-03-21T10:26:20.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.337+0000 7f51c3400700 1 --2- 172.21.15.31:0/1919250851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51c407de60 0x7f51c407c2c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f51b40098a0 tx=0x7f51b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c633165970a382 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:20.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.338+0000 7f51c2a00700 1 -- 172.21.15.31:0/1919250851 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f51b4004160 con 0x7f51c407de60 2024-03-21T10:26:20.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.338+0000 7f51c2a00700 1 -- 172.21.15.31:0/1919250851 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f51b4004300 con 0x7f51c407de60 2024-03-21T10:26:20.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.338+0000 7f51c2a00700 1 -- 172.21.15.31:0/1919250851 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f51b4011b50 con 0x7f51c407de60 2024-03-21T10:26:20.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.338+0000 7f51ca600700 1 -- 172.21.15.31:0/1919250851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51c407de60 msgr2=0x7f51c407c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:20.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.338+0000 7f51ca600700 1 --2- 172.21.15.31:0/1919250851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51c407de60 0x7f51c407c2c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f51b40098a0 tx=0x7f51b4009bb0 comp rx=0 tx=0).stop 2024-03-21T10:26:20.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.338+0000 7f51ca600700 1 -- 172.21.15.31:0/1919250851 shutdown_connections 2024-03-21T10:26:20.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.338+0000 7f51ca600700 1 --2- 172.21.15.31:0/1919250851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51c407de60 0x7f51c407c2c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:20.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.338+0000 7f51ca600700 1 -- 172.21.15.31:0/1919250851 >> 172.21.15.31:0/1919250851 conn(0x7f51c4103080 msgr2=0x7f51c41054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:20.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.339+0000 7f51ca600700 1 -- 172.21.15.31:0/1919250851 shutdown_connections 2024-03-21T10:26:20.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.339+0000 7f51ca600700 1 -- 172.21.15.31:0/1919250851 wait complete. 2024-03-21T10:26:20.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.340+0000 7f51ca600700 1 Processor -- start 2024-03-21T10:26:20.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.340+0000 7f51ca600700 1 -- start start 2024-03-21T10:26:20.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.340+0000 7f51ca600700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51c407de60 0x7f51c419dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:20.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.340+0000 7f51ca600700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c410e0b0 con 0x7f51c407de60 2024-03-21T10:26:20.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.340+0000 7f51c3400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51c407de60 0x7f51c419dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:20.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.340+0000 7f51c3400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51c407de60 0x7f51c419dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38768/0 (socket says 172.21.15.31:38768) 2024-03-21T10:26:20.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.341+0000 7f51c3400700 1 -- 172.21.15.31:0/1809252430 learned_addr learned my addr 172.21.15.31:0/1809252430 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:20.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.341+0000 7f51c3400700 1 -- 172.21.15.31:0/1809252430 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c419e160 con 0x7f51c407de60 2024-03-21T10:26:20.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.341+0000 7f51c3400700 1 --2- 172.21.15.31:0/1809252430 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51c407de60 0x7f51c419dc20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f51b4011b60 tx=0x7f51b4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:20.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.341+0000 7f51c0c00700 1 -- 172.21.15.31:0/1809252430 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f51b4003b20 con 0x7f51c407de60 2024-03-21T10:26:20.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.341+0000 7f51c0c00700 1 -- 172.21.15.31:0/1809252430 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f51b4003cc0 con 0x7f51c407de60 2024-03-21T10:26:20.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.342+0000 7f51c0c00700 1 -- 172.21.15.31:0/1809252430 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f51b4024840 con 0x7f51c407de60 2024-03-21T10:26:20.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.342+0000 7f51ca600700 1 -- 172.21.15.31:0/1809252430 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51c419e3f0 con 0x7f51c407de60 2024-03-21T10:26:20.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.342+0000 7f51ca600700 1 -- 172.21.15.31:0/1809252430 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c41a10e0 con 0x7f51c407de60 2024-03-21T10:26:20.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.343+0000 7f51c0c00700 1 -- 172.21.15.31:0/1809252430 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) v1 ==== 46294+0+0 (secure 0 0 0) 0x7f51b4022020 con 0x7f51c407de60 2024-03-21T10:26:20.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.343+0000 7f51ca600700 1 -- 172.21.15.31:0/1809252430 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c40637c0 con 0x7f51c407de60 2024-03-21T10:26:20.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.343+0000 7f51c0c00700 1 --2- 172.21.15.31:0/1809252430 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f51a80391d0 0x7f51a803b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:20.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.343+0000 7f51c0c00700 1 -- 172.21.15.31:0/1809252430 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f51b4011e00 con 0x7f51c407de60 2024-03-21T10:26:20.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.344+0000 7f51bbe00700 1 --2- 172.21.15.31:0/1809252430 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f51a80391d0 0x7f51a803b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:20.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.344+0000 7f51bbe00700 1 --2- 172.21.15.31:0/1809252430 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f51a80391d0 0x7f51a803b690 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f51ac006fd0 tx=0x7f51ac006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:20.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.349+0000 7f51c0c00700 1 -- 172.21.15.31:0/1809252430 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f51b404cca0 con 0x7f51c407de60 2024-03-21T10:26:20.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.591+0000 7f51ca600700 1 -- 172.21.15.31:0/1809252430 --> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f51c41042a0 con 0x7f51a80391d0 2024-03-21T10:26:20.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.603+0000 7f51c0c00700 1 -- 172.21.15.31:0/1809252430 <== mgr.14120 v2:172.21.15.31:6800/4085461895 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f51c41042a0 con 0x7f51a80391d0 2024-03-21T10:26:20.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.606+0000 7f51ca600700 1 -- 172.21.15.31:0/1809252430 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f51a80391d0 msgr2=0x7f51a803b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:20.368 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.606+0000 7f51ca600700 1 --2- 172.21.15.31:0/1809252430 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f51a80391d0 0x7f51a803b690 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f51ac006fd0 tx=0x7f51ac006e40 comp rx=0 tx=0).stop 2024-03-21T10:26:20.369 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.606+0000 7f51ca600700 1 -- 172.21.15.31:0/1809252430 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51c407de60 msgr2=0x7f51c419dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:20.369 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.606+0000 7f51ca600700 1 --2- 172.21.15.31:0/1809252430 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51c407de60 0x7f51c419dc20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f51b4011b60 tx=0x7f51b4011c40 comp rx=0 tx=0).stop 2024-03-21T10:26:20.369 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.607+0000 7f51ca600700 1 -- 172.21.15.31:0/1809252430 shutdown_connections 2024-03-21T10:26:20.369 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.607+0000 7f51ca600700 1 --2- 172.21.15.31:0/1809252430 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f51a80391d0 0x7f51a803b690 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:20.369 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.607+0000 7f51ca600700 1 --2- 172.21.15.31:0/1809252430 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51c407de60 0x7f51c419dc20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:20.369 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.607+0000 7f51ca600700 1 -- 172.21.15.31:0/1809252430 >> 172.21.15.31:0/1809252430 conn(0x7f51c4103080 msgr2=0x7f51c4103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:20.369 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.607+0000 7f51ca600700 1 -- 172.21.15.31:0/1809252430 shutdown_connections 2024-03-21T10:26:20.369 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:19.607+0000 7f51ca600700 1 -- 172.21.15.31:0/1809252430 wait complete. 2024-03-21T10:26:21.886 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-21T10:26:21.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.921+0000 7f4051a00700 1 Processor -- start 2024-03-21T10:26:21.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.921+0000 7f4051a00700 1 -- start start 2024-03-21T10:26:21.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.921+0000 7f4051a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f404c1084e0 0x7f404c10a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:21.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.921+0000 7f4051a00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404c07be90 con 0x7f404c1084e0 2024-03-21T10:26:21.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.922+0000 7f404aa00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f404c1084e0 0x7f404c10a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:21.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.922+0000 7f404aa00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f404c1084e0 0x7f404c10a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38774/0 (socket says 172.21.15.31:38774) 2024-03-21T10:26:21.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.922+0000 7f404aa00700 1 -- 172.21.15.31:0/519416242 learned_addr learned my addr 172.21.15.31:0/519416242 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.923+0000 7f404aa00700 1 -- 172.21.15.31:0/519416242 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404c10ae10 con 0x7f404c1084e0 2024-03-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.923+0000 7f404aa00700 1 --2- 172.21.15.31:0/519416242 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f404c1084e0 0x7f404c10a8d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f403c0098a0 tx=0x7f403c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bec33ffd0e45c7a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.923+0000 7f4049600700 1 -- 172.21.15.31:0/519416242 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f403c004160 con 0x7f404c1084e0 2024-03-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.924+0000 7f4049600700 1 -- 172.21.15.31:0/519416242 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f403c004300 con 0x7f404c1084e0 2024-03-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.924+0000 7f4049600700 1 -- 172.21.15.31:0/519416242 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f403c011b50 con 0x7f404c1084e0 2024-03-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.924+0000 7f4051a00700 1 -- 172.21.15.31:0/519416242 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f404c1084e0 msgr2=0x7f404c10a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.924+0000 7f4051a00700 1 --2- 172.21.15.31:0/519416242 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f404c1084e0 0x7f404c10a8d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f403c0098a0 tx=0x7f403c009bb0 comp rx=0 tx=0).stop 2024-03-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.925+0000 7f4051a00700 1 -- 172.21.15.31:0/519416242 shutdown_connections 2024-03-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.925+0000 7f4051a00700 1 --2- 172.21.15.31:0/519416242 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f404c1084e0 0x7f404c10a8d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.925+0000 7f4051a00700 1 -- 172.21.15.31:0/519416242 >> 172.21.15.31:0/519416242 conn(0x7f404c101f70 msgr2=0x7f404c104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.925+0000 7f4051a00700 1 -- 172.21.15.31:0/519416242 shutdown_connections 2024-03-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.925+0000 7f4051a00700 1 -- 172.21.15.31:0/519416242 wait complete. 2024-03-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.926+0000 7f4051a00700 1 Processor -- start 2024-03-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.926+0000 7f4051a00700 1 -- start start 2024-03-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.926+0000 7f4051a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f404c1084e0 0x7f404c19b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.926+0000 7f4051a00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404c10be40 con 0x7f404c1084e0 2024-03-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.927+0000 7f404aa00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f404c1084e0 0x7f404c19b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.927+0000 7f404aa00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f404c1084e0 0x7f404c19b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38782/0 (socket says 172.21.15.31:38782) 2024-03-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.927+0000 7f404aa00700 1 -- 172.21.15.31:0/2125318113 learned_addr learned my addr 172.21.15.31:0/2125318113 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.927+0000 7f404aa00700 1 -- 172.21.15.31:0/2125318113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404c19bef0 con 0x7f404c1084e0 2024-03-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.927+0000 7f404aa00700 1 --2- 172.21.15.31:0/2125318113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f404c1084e0 0x7f404c19b9b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f403c011b60 tx=0x7f403c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.928+0000 7f4043400700 1 -- 172.21.15.31:0/2125318113 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f403c003b20 con 0x7f404c1084e0 2024-03-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.928+0000 7f4043400700 1 -- 172.21.15.31:0/2125318113 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f403c003cc0 con 0x7f404c1084e0 2024-03-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.928+0000 7f4043400700 1 -- 172.21.15.31:0/2125318113 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f403c024840 con 0x7f404c1084e0 2024-03-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.928+0000 7f4051a00700 1 -- 172.21.15.31:0/2125318113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f404c19c180 con 0x7f404c1084e0 2024-03-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.928+0000 7f4051a00700 1 -- 172.21.15.31:0/2125318113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f404c19ee70 con 0x7f404c1084e0 2024-03-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.929+0000 7f4043400700 1 -- 172.21.15.31:0/2125318113 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) v1 ==== 46294+0+0 (secure 0 0 0) 0x7f403c022020 con 0x7f404c1084e0 2024-03-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.929+0000 7f4051a00700 1 -- 172.21.15.31:0/2125318113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f404c0637c0 con 0x7f404c1084e0 2024-03-21T10:26:21.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.929+0000 7f4043400700 1 --2- 172.21.15.31:0/2125318113 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f40300391d0 0x7f403003b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:21.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.929+0000 7f4043400700 1 -- 172.21.15.31:0/2125318113 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f403c011e00 con 0x7f404c1084e0 2024-03-21T10:26:21.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.930+0000 7f404a000700 1 --2- 172.21.15.31:0/2125318113 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f40300391d0 0x7f403003b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.931+0000 7f404a000700 1 --2- 172.21.15.31:0/2125318113 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f40300391d0 0x7f403003b690 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4038006fd0 tx=0x7f4038006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:20.936+0000 7f4043400700 1 -- 172.21.15.31:0/2125318113 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f403c01a7b0 con 0x7f404c1084e0 2024-03-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:21.164+0000 7f4051a00700 1 -- 172.21.15.31:0/2125318113 --> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f404c103150 con 0x7f40300391d0 2024-03-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:21.166+0000 7f4043400700 1 -- 172.21.15.31:0/2125318113 <== mgr.14120 v2:172.21.15.31:6800/4085461895 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f404c103150 con 0x7f40300391d0 2024-03-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:21.168+0000 7f4051a00700 1 -- 172.21.15.31:0/2125318113 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f40300391d0 msgr2=0x7f403003b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:21.168+0000 7f4051a00700 1 --2- 172.21.15.31:0/2125318113 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f40300391d0 0x7f403003b690 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4038006fd0 tx=0x7f4038006e40 comp rx=0 tx=0).stop 2024-03-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:21.168+0000 7f4051a00700 1 -- 172.21.15.31:0/2125318113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f404c1084e0 msgr2=0x7f404c19b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:21.168+0000 7f4051a00700 1 --2- 172.21.15.31:0/2125318113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f404c1084e0 0x7f404c19b9b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f403c011b60 tx=0x7f403c011c40 comp rx=0 tx=0).stop 2024-03-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:21.169+0000 7f4051a00700 1 -- 172.21.15.31:0/2125318113 shutdown_connections 2024-03-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:21.169+0000 7f4051a00700 1 --2- 172.21.15.31:0/2125318113 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f40300391d0 0x7f403003b690 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:21.169+0000 7f4051a00700 1 --2- 172.21.15.31:0/2125318113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f404c1084e0 0x7f404c19b9b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:21.169+0000 7f4051a00700 1 -- 172.21.15.31:0/2125318113 >> 172.21.15.31:0/2125318113 conn(0x7f404c101f70 msgr2=0x7f404c102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:21.169+0000 7f4051a00700 1 -- 172.21.15.31:0/2125318113 shutdown_connections 2024-03-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:21.169+0000 7f4051a00700 1 -- 172.21.15.31:0/2125318113 wait complete. 2024-03-21T10:26:21.897 INFO:teuthology.orchestra.run.smithi031.stdout:Generating ssh key... 2024-03-21T10:26:22.246 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:22 smithi031 ceph-mon[31673]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:26:23.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:23 smithi031 ceph-mon[31673]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:26:23.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:23 smithi031 ceph-mon[31673]: Generating ssh key... 2024-03-21T10:26:23.809 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:23 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: Generating public/private rsa key pair. 2024-03-21T10:26:23.809 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:23 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: Your identification has been saved in /tmp/tmparc2lut9/key. 2024-03-21T10:26:23.809 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:23 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: Your public key has been saved in /tmp/tmparc2lut9/key.pub. 2024-03-21T10:26:23.809 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:23 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: The key fingerprint is: 2024-03-21T10:26:23.809 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:23 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: SHA256:Wyqwq+u0JV5FhY0pCrgUtpJ0EECd6rUDhxn1HsspxVk ceph-3847408c-e76d-11ee-95cd-87774f69a715 2024-03-21T10:26:23.809 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:23 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: The key's randomart image is: 2024-03-21T10:26:23.809 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:23 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: +---[RSA 3072]----+ 2024-03-21T10:26:23.809 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:23 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: |=B*o. E. | 2024-03-21T10:26:23.809 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:23 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: |=+oo+ *.. | 2024-03-21T10:26:23.809 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:23 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: |++=. B. | 2024-03-21T10:26:23.809 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:23 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: |o=.o+.+ | 2024-03-21T10:26:23.809 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:23 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: |. +.o=. S . | 2024-03-21T10:26:23.810 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:23 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: | . o.+ + | 2024-03-21T10:26:23.810 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:23 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: | o = . o | 2024-03-21T10:26:23.810 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:23 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: | o = . . | 2024-03-21T10:26:23.810 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:23 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: | .*o. | 2024-03-21T10:26:23.810 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:23 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: +----[SHA256]-----+ 2024-03-21T10:26:24.396 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.548+0000 7f090ca00700 1 Processor -- start 2024-03-21T10:26:24.396 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.548+0000 7f090ca00700 1 -- start start 2024-03-21T10:26:24.396 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.548+0000 7f090ca00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0908081870 0x7f0908081c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:24.396 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.548+0000 7f090ca00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09080821d0 con 0x7f0908081870 2024-03-21T10:26:24.396 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.549+0000 7f0907400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0908081870 0x7f0908081c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:24.396 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.549+0000 7f0907400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0908081870 0x7f0908081c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38784/0 (socket says 172.21.15.31:38784) 2024-03-21T10:26:24.396 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.549+0000 7f0907400700 1 -- 172.21.15.31:0/734437787 learned_addr learned my addr 172.21.15.31:0/734437787 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:24.396 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.549+0000 7f0907400700 1 -- 172.21.15.31:0/734437787 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f090810f040 con 0x7f0908081870 2024-03-21T10:26:24.396 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.549+0000 7f0907400700 1 --2- 172.21.15.31:0/734437787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0908081870 0x7f0908081c90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f08f80098a0 tx=0x7f08f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25021392429e0776 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:24.396 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.550+0000 7f0906a00700 1 -- 172.21.15.31:0/734437787 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f08f8004160 con 0x7f0908081870 2024-03-21T10:26:24.397 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.550+0000 7f0906a00700 1 -- 172.21.15.31:0/734437787 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f08f8004300 con 0x7f0908081870 2024-03-21T10:26:24.397 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.550+0000 7f0906a00700 1 -- 172.21.15.31:0/734437787 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f08f8011b50 con 0x7f0908081870 2024-03-21T10:26:24.397 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.550+0000 7f090ca00700 1 -- 172.21.15.31:0/734437787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0908081870 msgr2=0x7f0908081c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:24.397 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.550+0000 7f090ca00700 1 --2- 172.21.15.31:0/734437787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0908081870 0x7f0908081c90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f08f80098a0 tx=0x7f08f8009bb0 comp rx=0 tx=0).stop 2024-03-21T10:26:24.397 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.550+0000 7f090ca00700 1 -- 172.21.15.31:0/734437787 shutdown_connections 2024-03-21T10:26:24.401 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.550+0000 7f090ca00700 1 --2- 172.21.15.31:0/734437787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0908081870 0x7f0908081c90 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:24.401 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.550+0000 7f090ca00700 1 -- 172.21.15.31:0/734437787 >> 172.21.15.31:0/734437787 conn(0x7f090807cee0 msgr2=0x7f090807f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:24.401 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.551+0000 7f090ca00700 1 -- 172.21.15.31:0/734437787 shutdown_connections 2024-03-21T10:26:24.401 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.551+0000 7f090ca00700 1 -- 172.21.15.31:0/734437787 wait complete. 2024-03-21T10:26:24.401 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.552+0000 7f090ca00700 1 Processor -- start 2024-03-21T10:26:24.401 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.552+0000 7f090ca00700 1 -- start start 2024-03-21T10:26:24.401 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.552+0000 7f090ca00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0908081870 0x7f09081a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:24.402 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.552+0000 7f090ca00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0908112300 con 0x7f0908081870 2024-03-21T10:26:24.402 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.552+0000 7f0907400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0908081870 0x7f09081a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:24.402 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.552+0000 7f0907400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0908081870 0x7f09081a1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38798/0 (socket says 172.21.15.31:38798) 2024-03-21T10:26:24.402 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.552+0000 7f0907400700 1 -- 172.21.15.31:0/2506395913 learned_addr learned my addr 172.21.15.31:0/2506395913 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:24.402 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.553+0000 7f0907400700 1 -- 172.21.15.31:0/2506395913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09081a2390 con 0x7f0908081870 2024-03-21T10:26:24.402 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.553+0000 7f0907400700 1 --2- 172.21.15.31:0/2506395913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0908081870 0x7f09081a1e50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f08f8011b60 tx=0x7f08f8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:24.402 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.553+0000 7f0904c00700 1 -- 172.21.15.31:0/2506395913 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f08f8003970 con 0x7f0908081870 2024-03-21T10:26:24.402 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.553+0000 7f0904c00700 1 -- 172.21.15.31:0/2506395913 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f08f8003b10 con 0x7f0908081870 2024-03-21T10:26:24.402 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.553+0000 7f0904c00700 1 -- 172.21.15.31:0/2506395913 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f08f8024610 con 0x7f0908081870 2024-03-21T10:26:24.402 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.553+0000 7f090ca00700 1 -- 172.21.15.31:0/2506395913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09081a2620 con 0x7f0908081870 2024-03-21T10:26:24.402 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.554+0000 7f090ca00700 1 -- 172.21.15.31:0/2506395913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09081a5310 con 0x7f0908081870 2024-03-21T10:26:24.402 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.555+0000 7f0904c00700 1 -- 172.21.15.31:0/2506395913 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) v1 ==== 46294+0+0 (secure 0 0 0) 0x7f08f80248c0 con 0x7f0908081870 2024-03-21T10:26:24.402 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.555+0000 7f090ca00700 1 -- 172.21.15.31:0/2506395913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0908112480 con 0x7f0908081870 2024-03-21T10:26:24.403 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.555+0000 7f0904c00700 1 --2- 172.21.15.31:0/2506395913 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f08ec0391d0 0x7f08ec03b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:24.403 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.555+0000 7f0904c00700 1 -- 172.21.15.31:0/2506395913 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f08f8011e00 con 0x7f0908081870 2024-03-21T10:26:24.403 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.555+0000 7f08fea00700 1 --2- 172.21.15.31:0/2506395913 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f08ec0391d0 0x7f08ec03b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:24.403 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.556+0000 7f08fea00700 1 --2- 172.21.15.31:0/2506395913 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f08ec0391d0 0x7f08ec03b690 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f08f0006fd0 tx=0x7f08f0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:24.403 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.561+0000 7f0904c00700 1 -- 172.21.15.31:0/2506395913 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f08f800b3e0 con 0x7f0908081870 2024-03-21T10:26:24.403 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:22.790+0000 7f090ca00700 1 -- 172.21.15.31:0/2506395913 --> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f090807e0e0 con 0x7f08ec0391d0 2024-03-21T10:26:24.403 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:23.544+0000 7f0904c00700 1 -- 172.21.15.31:0/2506395913 <== mgr.14120 v2:172.21.15.31:6800/4085461895 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f090807e0e0 con 0x7f08ec0391d0 2024-03-21T10:26:24.403 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:23.546+0000 7f090ca00700 1 -- 172.21.15.31:0/2506395913 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f08ec0391d0 msgr2=0x7f08ec03b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:24.403 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:23.546+0000 7f090ca00700 1 --2- 172.21.15.31:0/2506395913 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f08ec0391d0 0x7f08ec03b690 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f08f0006fd0 tx=0x7f08f0006e40 comp rx=0 tx=0).stop 2024-03-21T10:26:24.403 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:23.546+0000 7f090ca00700 1 -- 172.21.15.31:0/2506395913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0908081870 msgr2=0x7f09081a1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:24.403 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:23.546+0000 7f090ca00700 1 --2- 172.21.15.31:0/2506395913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0908081870 0x7f09081a1e50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f08f8011b60 tx=0x7f08f8011c40 comp rx=0 tx=0).stop 2024-03-21T10:26:24.403 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:23.547+0000 7f090ca00700 1 -- 172.21.15.31:0/2506395913 shutdown_connections 2024-03-21T10:26:24.403 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:23.547+0000 7f090ca00700 1 --2- 172.21.15.31:0/2506395913 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f08ec0391d0 0x7f08ec03b690 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:24.403 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:23.547+0000 7f090ca00700 1 --2- 172.21.15.31:0/2506395913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0908081870 0x7f09081a1e50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:24.404 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:23.547+0000 7f090ca00700 1 -- 172.21.15.31:0/2506395913 >> 172.21.15.31:0/2506395913 conn(0x7f090807cee0 msgr2=0x7f090807d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:24.404 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:23.547+0000 7f090ca00700 1 -- 172.21.15.31:0/2506395913 shutdown_connections 2024-03-21T10:26:24.404 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:23.547+0000 7f090ca00700 1 -- 172.21.15.31:0/2506395913 wait complete. 2024-03-21T10:26:24.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:24 smithi031 ceph-mon[31673]: from='mgr.14120 172.21.15.31:0/4221700501' entity='mgr.x' 2024-03-21T10:26:24.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:24 smithi031 ceph-mon[31673]: from='mgr.14120 172.21.15.31:0/4221700501' entity='mgr.x' 2024-03-21T10:26:25.945 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJUNRYCYm3PhJDatpPDA7cWcOZcWZ1xbEEwhquawL3S/bRslYoumXGK2rL3KYlVlm2alPycHnDRuLuWwGQSOYww3sFAEZePWnT/Rrda7g5qomR6EYrQkALm2ptVbG+mlVvEdCRj96FNKO2MyXIlW+XZIqUAOiJ9KX1MLkhSyU8JQW1no1/3fU107uzcCKB9Q0ENBdKi7DdOW0rO7/dztXwcghKEbZ8p9e5QDe33vJETsj2QB4/g3srcxoIe2pa9e3SNCzS5mKV6/JU1uQng77+WT9KYjxT8dkfOLlcMNouyDhHnFy8iBQX++i8eMDEJVLkykHWy74FTLnJ8CDB3G+iPwA7c2G01A4Aiz14GllLOfczBYdyEpiR1G6xNSKiXbvInjmVx4iAn/uHh+9OhNOJfh2V6ij/hKHpWA7I4tPpiUmymOSK2SvaYPfs3HvYYPIgwc+RzaZ+8S0eaDcn7b/6AphshTcuolkfH2fKZoiQGCVkpLQON+QDM5nokKkRsuk= ceph-3847408c-e76d-11ee-95cd-87774f69a715 2024-03-21T10:26:25.945 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.087+0000 7f3de7e00700 1 Processor -- start 2024-03-21T10:26:25.945 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.087+0000 7f3de7e00700 1 -- start start 2024-03-21T10:26:25.945 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.087+0000 7f3de7e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de8107970 0x7f3de8107d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:25.946 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.087+0000 7f3de7e00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de8108360 con 0x7f3de8107970 2024-03-21T10:26:25.946 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.088+0000 7f3de6a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de8107970 0x7f3de8107d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:25.946 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.088+0000 7f3de6a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de8107970 0x7f3de8107d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38808/0 (socket says 172.21.15.31:38808) 2024-03-21T10:26:25.946 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.088+0000 7f3de6a00700 1 -- 172.21.15.31:0/1081408784 learned_addr learned my addr 172.21.15.31:0/1081408784 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:25.946 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.088+0000 7f3de6a00700 1 -- 172.21.15.31:0/1081408784 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de8108ad0 con 0x7f3de8107970 2024-03-21T10:26:25.946 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.088+0000 7f3de6a00700 1 --2- 172.21.15.31:0/1081408784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de8107970 0x7f3de8107d90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3dd80098a0 tx=0x7f3dd8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1fb85e3883ad19b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:25.946 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.089+0000 7f3de6000700 1 -- 172.21.15.31:0/1081408784 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3dd8004160 con 0x7f3de8107970 2024-03-21T10:26:25.946 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.089+0000 7f3de6000700 1 -- 172.21.15.31:0/1081408784 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3dd8004300 con 0x7f3de8107970 2024-03-21T10:26:25.946 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.089+0000 7f3de6000700 1 -- 172.21.15.31:0/1081408784 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3dd8011b50 con 0x7f3de8107970 2024-03-21T10:26:25.946 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.089+0000 7f3de7e00700 1 -- 172.21.15.31:0/1081408784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de8107970 msgr2=0x7f3de8107d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:25.946 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.089+0000 7f3de7e00700 1 --2- 172.21.15.31:0/1081408784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de8107970 0x7f3de8107d90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3dd80098a0 tx=0x7f3dd8009bb0 comp rx=0 tx=0).stop 2024-03-21T10:26:25.946 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.089+0000 7f3de7e00700 1 -- 172.21.15.31:0/1081408784 shutdown_connections 2024-03-21T10:26:25.946 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.089+0000 7f3de7e00700 1 --2- 172.21.15.31:0/1081408784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de8107970 0x7f3de8107d90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:25.947 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.089+0000 7f3de7e00700 1 -- 172.21.15.31:0/1081408784 >> 172.21.15.31:0/1081408784 conn(0x7f3de8102f50 msgr2=0x7f3de8105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:25.947 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.090+0000 7f3de7e00700 1 -- 172.21.15.31:0/1081408784 shutdown_connections 2024-03-21T10:26:25.947 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.090+0000 7f3de7e00700 1 -- 172.21.15.31:0/1081408784 wait complete. 2024-03-21T10:26:25.947 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.091+0000 7f3de7e00700 1 Processor -- start 2024-03-21T10:26:25.947 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.091+0000 7f3de7e00700 1 -- start start 2024-03-21T10:26:25.947 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.091+0000 7f3de7e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de8107970 0x7f3de807db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:25.947 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.091+0000 7f3de7e00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de810be80 con 0x7f3de8107970 2024-03-21T10:26:25.947 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.092+0000 7f3de6a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de8107970 0x7f3de807db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:25.947 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.092+0000 7f3de6a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de8107970 0x7f3de807db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38812/0 (socket says 172.21.15.31:38812) 2024-03-21T10:26:25.947 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.092+0000 7f3de6a00700 1 -- 172.21.15.31:0/2941240502 learned_addr learned my addr 172.21.15.31:0/2941240502 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:25.949 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.092+0000 7f3de6a00700 1 -- 172.21.15.31:0/2941240502 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de807e090 con 0x7f3de8107970 2024-03-21T10:26:25.949 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.092+0000 7f3de6a00700 1 --2- 172.21.15.31:0/2941240502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de8107970 0x7f3de807db50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8011850 tx=0x7f3dd8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:25.949 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.093+0000 7f3ddfe00700 1 -- 172.21.15.31:0/2941240502 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3dd8003af0 con 0x7f3de8107970 2024-03-21T10:26:25.950 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.093+0000 7f3ddfe00700 1 -- 172.21.15.31:0/2941240502 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3dd8020070 con 0x7f3de8107970 2024-03-21T10:26:25.950 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.093+0000 7f3de7e00700 1 -- 172.21.15.31:0/2941240502 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de807c280 con 0x7f3de8107970 2024-03-21T10:26:25.950 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.093+0000 7f3ddfe00700 1 -- 172.21.15.31:0/2941240502 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3dd80195a0 con 0x7f3de8107970 2024-03-21T10:26:25.950 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.093+0000 7f3de7e00700 1 -- 172.21.15.31:0/2941240502 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de807c7c0 con 0x7f3de8107970 2024-03-21T10:26:25.950 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.094+0000 7f3de7e00700 1 -- 172.21.15.31:0/2941240502 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de80637c0 con 0x7f3de8107970 2024-03-21T10:26:25.950 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.094+0000 7f3ddfe00700 1 -- 172.21.15.31:0/2941240502 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) v1 ==== 46294+0+0 (secure 0 0 0) 0x7f3dd8019850 con 0x7f3de8107970 2024-03-21T10:26:25.950 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.094+0000 7f3ddfe00700 1 --2- 172.21.15.31:0/2941240502 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f3dcc0391d0 0x7f3dcc03b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:25.950 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.095+0000 7f3ddfe00700 1 -- 172.21.15.31:0/2941240502 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f3dd8011e00 con 0x7f3de8107970 2024-03-21T10:26:25.952 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.095+0000 7f3dde000700 1 --2- 172.21.15.31:0/2941240502 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f3dcc0391d0 0x7f3dcc03b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:25.952 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.096+0000 7f3dde000700 1 --2- 172.21.15.31:0/2941240502 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f3dcc0391d0 0x7f3dcc03b690 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0006fd0 tx=0x7f3dd0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:25.953 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.100+0000 7f3ddfe00700 1 -- 172.21.15.31:0/2941240502 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3dd8019b40 con 0x7f3de8107970 2024-03-21T10:26:25.953 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.326+0000 7f3de7e00700 1 -- 172.21.15.31:0/2941240502 --> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f3de81041f0 con 0x7f3dcc0391d0 2024-03-21T10:26:25.953 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.328+0000 7f3ddfe00700 1 -- 172.21.15.31:0/2941240502 <== mgr.14120 v2:172.21.15.31:6800/4085461895 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f3de81041f0 con 0x7f3dcc0391d0 2024-03-21T10:26:25.953 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.330+0000 7f3de7e00700 1 -- 172.21.15.31:0/2941240502 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f3dcc0391d0 msgr2=0x7f3dcc03b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:25.953 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.330+0000 7f3de7e00700 1 --2- 172.21.15.31:0/2941240502 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f3dcc0391d0 0x7f3dcc03b690 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0006fd0 tx=0x7f3dd0006e40 comp rx=0 tx=0).stop 2024-03-21T10:26:25.953 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.330+0000 7f3de7e00700 1 -- 172.21.15.31:0/2941240502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de8107970 msgr2=0x7f3de807db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:25.953 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.330+0000 7f3de7e00700 1 --2- 172.21.15.31:0/2941240502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de8107970 0x7f3de807db50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8011850 tx=0x7f3dd8011c10 comp rx=0 tx=0).stop 2024-03-21T10:26:25.953 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.331+0000 7f3de7e00700 1 -- 172.21.15.31:0/2941240502 shutdown_connections 2024-03-21T10:26:25.953 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.331+0000 7f3de7e00700 1 --2- 172.21.15.31:0/2941240502 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f3dcc0391d0 0x7f3dcc03b690 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:25.953 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.331+0000 7f3de7e00700 1 --2- 172.21.15.31:0/2941240502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de8107970 0x7f3de807db50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:25.953 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.331+0000 7f3de7e00700 1 -- 172.21.15.31:0/2941240502 >> 172.21.15.31:0/2941240502 conn(0x7f3de8102f50 msgr2=0x7f3de8103ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:25.953 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.331+0000 7f3de7e00700 1 -- 172.21.15.31:0/2941240502 shutdown_connections 2024-03-21T10:26:25.954 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:25.331+0000 7f3de7e00700 1 -- 172.21.15.31:0/2941240502 wait complete. 2024-03-21T10:26:25.954 INFO:teuthology.orchestra.run.smithi031.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-21T10:26:25.954 INFO:teuthology.orchestra.run.smithi031.stdout:Adding key to root@localhost authorized_keys... 2024-03-21T10:26:25.954 INFO:teuthology.orchestra.run.smithi031.stdout:Adding host smithi031... 2024-03-21T10:26:26.248 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:26 smithi031 ceph-mon[31673]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:26:27.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:27 smithi031 ceph-mon[31673]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi031", "addr": "172.21.15.31", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:26:29.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:29 smithi031 ceph-mon[31673]: Deploying cephadm binary to smithi031 2024-03-21T10:26:34.414 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Added host 'smithi031' with addr '172.21.15.31' 2024-03-21T10:26:34.414 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.588+0000 7f09e7800700 1 Processor -- start 2024-03-21T10:26:34.414 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.588+0000 7f09e7800700 1 -- start start 2024-03-21T10:26:34.414 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.589+0000 7f09e7800700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09e0081aa0 0x7f09e0081ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:34.414 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.589+0000 7f09e7800700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e0082490 con 0x7f09e0081aa0 2024-03-21T10:26:34.418 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.589+0000 7f09e4e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09e0081aa0 0x7f09e0081ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:34.418 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.589+0000 7f09e4e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09e0081aa0 0x7f09e0081ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38824/0 (socket says 172.21.15.31:38824) 2024-03-21T10:26:34.418 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.589+0000 7f09e4e00700 1 -- 172.21.15.31:0/2405821983 learned_addr learned my addr 172.21.15.31:0/2405821983 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:34.418 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.589+0000 7f09e4e00700 1 -- 172.21.15.31:0/2405821983 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e010f280 con 0x7f09e0081aa0 2024-03-21T10:26:34.418 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.590+0000 7f09e4e00700 1 --2- 172.21.15.31:0/2405821983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09e0081aa0 0x7f09e0081ec0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f09d40098a0 tx=0x7f09d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e160a5149a6ceebe server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:34.418 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.590+0000 7f09df400700 1 -- 172.21.15.31:0/2405821983 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f09d4004160 con 0x7f09e0081aa0 2024-03-21T10:26:34.418 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.590+0000 7f09df400700 1 -- 172.21.15.31:0/2405821983 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f09d4004300 con 0x7f09e0081aa0 2024-03-21T10:26:34.418 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.590+0000 7f09df400700 1 -- 172.21.15.31:0/2405821983 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f09d4011b50 con 0x7f09e0081aa0 2024-03-21T10:26:34.418 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.591+0000 7f09e7800700 1 -- 172.21.15.31:0/2405821983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09e0081aa0 msgr2=0x7f09e0081ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:34.418 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.591+0000 7f09e7800700 1 --2- 172.21.15.31:0/2405821983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09e0081aa0 0x7f09e0081ec0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f09d40098a0 tx=0x7f09d4009bb0 comp rx=0 tx=0).stop 2024-03-21T10:26:34.418 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.591+0000 7f09e7800700 1 -- 172.21.15.31:0/2405821983 shutdown_connections 2024-03-21T10:26:34.418 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.591+0000 7f09e7800700 1 --2- 172.21.15.31:0/2405821983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09e0081aa0 0x7f09e0081ec0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:34.419 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.591+0000 7f09e7800700 1 -- 172.21.15.31:0/2405821983 >> 172.21.15.31:0/2405821983 conn(0x7f09e007d040 msgr2=0x7f09e007f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:34.419 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.591+0000 7f09e7800700 1 -- 172.21.15.31:0/2405821983 shutdown_connections 2024-03-21T10:26:34.419 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.591+0000 7f09e7800700 1 -- 172.21.15.31:0/2405821983 wait complete. 2024-03-21T10:26:34.419 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.592+0000 7f09e7800700 1 Processor -- start 2024-03-21T10:26:34.419 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.592+0000 7f09e7800700 1 -- start start 2024-03-21T10:26:34.419 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.592+0000 7f09e7800700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09e0081aa0 0x7f09e01a1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:34.419 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.592+0000 7f09e7800700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e0112630 con 0x7f09e0081aa0 2024-03-21T10:26:34.419 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.593+0000 7f09e4e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09e0081aa0 0x7f09e01a1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:34.419 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.593+0000 7f09e4e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09e0081aa0 0x7f09e01a1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38826/0 (socket says 172.21.15.31:38826) 2024-03-21T10:26:34.419 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.593+0000 7f09e4e00700 1 -- 172.21.15.31:0/112862153 learned_addr learned my addr 172.21.15.31:0/112862153 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:34.419 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.593+0000 7f09e4e00700 1 -- 172.21.15.31:0/112862153 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e01a2530 con 0x7f09e0081aa0 2024-03-21T10:26:34.419 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.593+0000 7f09e4e00700 1 --2- 172.21.15.31:0/112862153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09e0081aa0 0x7f09e01a1ff0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f09d4000c00 tx=0x7f09d4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:34.419 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.594+0000 7f09dd600700 1 -- 172.21.15.31:0/112862153 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f09d4003b20 con 0x7f09e0081aa0 2024-03-21T10:26:34.419 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.594+0000 7f09dd600700 1 -- 172.21.15.31:0/112862153 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f09d4003cc0 con 0x7f09e0081aa0 2024-03-21T10:26:34.420 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.594+0000 7f09dd600700 1 -- 172.21.15.31:0/112862153 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f09d4024910 con 0x7f09e0081aa0 2024-03-21T10:26:34.420 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.594+0000 7f09e7800700 1 -- 172.21.15.31:0/112862153 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e01a27c0 con 0x7f09e0081aa0 2024-03-21T10:26:34.422 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.594+0000 7f09e7800700 1 -- 172.21.15.31:0/112862153 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e01a54b0 con 0x7f09e0081aa0 2024-03-21T10:26:34.422 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.595+0000 7f09dd600700 1 -- 172.21.15.31:0/112862153 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) v1 ==== 46294+0+0 (secure 0 0 0) 0x7f09d4022020 con 0x7f09e0081aa0 2024-03-21T10:26:34.422 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.595+0000 7f09e7800700 1 -- 172.21.15.31:0/112862153 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e00637c0 con 0x7f09e0081aa0 2024-03-21T10:26:34.422 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.596+0000 7f09dd600700 1 --2- 172.21.15.31:0/112862153 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f09c8039230 0x7f09c803b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:34.422 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.596+0000 7f09dd600700 1 -- 172.21.15.31:0/112862153 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f09d4011e00 con 0x7f09e0081aa0 2024-03-21T10:26:34.422 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.596+0000 7f09dfe00700 1 --2- 172.21.15.31:0/112862153 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f09c8039230 0x7f09c803b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:34.423 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.597+0000 7f09dfe00700 1 --2- 172.21.15.31:0/112862153 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f09c8039230 0x7f09c803b6f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f09cc006fd0 tx=0x7f09cc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:34.423 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.601+0000 7f09dd600700 1 -- 172.21.15.31:0/112862153 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f09d401ad80 con 0x7f09e0081aa0 2024-03-21T10:26:34.423 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:26.833+0000 7f09e7800700 1 -- 172.21.15.31:0/112862153 --> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi031", "addr": "172.21.15.31", "target": ["mon-mgr", ""]}) v1 -- 0x7f09e007e220 con 0x7f09c8039230 2024-03-21T10:26:34.423 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:33.443+0000 7f09dd600700 1 -- 172.21.15.31:0/112862153 <== mgr.14120 v2:172.21.15.31:6800/4085461895 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f09e007e220 con 0x7f09c8039230 2024-03-21T10:26:34.423 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:33.447+0000 7f09e7800700 1 -- 172.21.15.31:0/112862153 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f09c8039230 msgr2=0x7f09c803b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:34.423 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:33.447+0000 7f09e7800700 1 --2- 172.21.15.31:0/112862153 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f09c8039230 0x7f09c803b6f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f09cc006fd0 tx=0x7f09cc006e40 comp rx=0 tx=0).stop 2024-03-21T10:26:34.423 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:33.447+0000 7f09e7800700 1 -- 172.21.15.31:0/112862153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09e0081aa0 msgr2=0x7f09e01a1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:34.423 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:33.447+0000 7f09e7800700 1 --2- 172.21.15.31:0/112862153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09e0081aa0 0x7f09e01a1ff0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f09d4000c00 tx=0x7f09d4011c10 comp rx=0 tx=0).stop 2024-03-21T10:26:34.423 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:33.447+0000 7f09e7800700 1 -- 172.21.15.31:0/112862153 shutdown_connections 2024-03-21T10:26:34.423 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:33.447+0000 7f09e7800700 1 --2- 172.21.15.31:0/112862153 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f09c8039230 0x7f09c803b6f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:34.423 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:33.447+0000 7f09e7800700 1 --2- 172.21.15.31:0/112862153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09e0081aa0 0x7f09e01a1ff0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:34.423 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:33.447+0000 7f09e7800700 1 -- 172.21.15.31:0/112862153 >> 172.21.15.31:0/112862153 conn(0x7f09e007d040 msgr2=0x7f09e007db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:34.423 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:33.448+0000 7f09e7800700 1 -- 172.21.15.31:0/112862153 shutdown_connections 2024-03-21T10:26:34.424 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:33.448+0000 7f09e7800700 1 -- 172.21.15.31:0/112862153 wait complete. 2024-03-21T10:26:34.424 INFO:teuthology.orchestra.run.smithi031.stdout:Deploying unmanaged mon service... 2024-03-21T10:26:34.715 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:34 smithi031 ceph-mon[31673]: from='mgr.14120 172.21.15.31:0/4221700501' entity='mgr.x' 2024-03-21T10:26:34.715 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:34 smithi031 ceph-mon[31673]: Added host smithi031 2024-03-21T10:26:34.715 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:34 smithi031 ceph-mon[31673]: from='mgr.14120 172.21.15.31:0/4221700501' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:26:36.212 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-21T10:26:36.212 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.031+0000 7efe97200700 1 Processor -- start 2024-03-21T10:26:36.212 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.031+0000 7efe97200700 1 -- start start 2024-03-21T10:26:36.214 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.032+0000 7efe97200700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe98109ac0 0x7efe98109ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:36.215 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.032+0000 7efe97200700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9810a4b0 con 0x7efe98109ac0 2024-03-21T10:26:36.215 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.032+0000 7efe95e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe98109ac0 0x7efe98109ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:36.215 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.032+0000 7efe95e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe98109ac0 0x7efe98109ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53712/0 (socket says 172.21.15.31:53712) 2024-03-21T10:26:36.215 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.032+0000 7efe95e00700 1 -- 172.21.15.31:0/3913621156 learned_addr learned my addr 172.21.15.31:0/3913621156 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:36.215 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.033+0000 7efe95e00700 1 -- 172.21.15.31:0/3913621156 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9810ac70 con 0x7efe98109ac0 2024-03-21T10:26:36.215 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.033+0000 7efe95e00700 1 --2- 172.21.15.31:0/3913621156 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe98109ac0 0x7efe98109ee0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efe880098a0 tx=0x7efe88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5024c57dfbb1e93a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:36.215 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.033+0000 7efe94a00700 1 -- 172.21.15.31:0/3913621156 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe88004160 con 0x7efe98109ac0 2024-03-21T10:26:36.215 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.034+0000 7efe94a00700 1 -- 172.21.15.31:0/3913621156 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe88004300 con 0x7efe98109ac0 2024-03-21T10:26:36.215 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.034+0000 7efe94a00700 1 -- 172.21.15.31:0/3913621156 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe88011b50 con 0x7efe98109ac0 2024-03-21T10:26:36.215 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.034+0000 7efe97200700 1 -- 172.21.15.31:0/3913621156 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe98109ac0 msgr2=0x7efe98109ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:36.215 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.034+0000 7efe97200700 1 --2- 172.21.15.31:0/3913621156 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe98109ac0 0x7efe98109ee0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efe880098a0 tx=0x7efe88009bb0 comp rx=0 tx=0).stop 2024-03-21T10:26:36.215 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.034+0000 7efe97200700 1 -- 172.21.15.31:0/3913621156 shutdown_connections 2024-03-21T10:26:36.215 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.034+0000 7efe97200700 1 --2- 172.21.15.31:0/3913621156 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe98109ac0 0x7efe98109ee0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:36.216 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.034+0000 7efe97200700 1 -- 172.21.15.31:0/3913621156 >> 172.21.15.31:0/3913621156 conn(0x7efe9807cf50 msgr2=0x7efe9807d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:36.216 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.035+0000 7efe97200700 1 -- 172.21.15.31:0/3913621156 shutdown_connections 2024-03-21T10:26:36.219 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.035+0000 7efe97200700 1 -- 172.21.15.31:0/3913621156 wait complete. 2024-03-21T10:26:36.219 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.036+0000 7efe97200700 1 Processor -- start 2024-03-21T10:26:36.220 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.036+0000 7efe97200700 1 -- start start 2024-03-21T10:26:36.220 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.036+0000 7efe97200700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe98109ac0 0x7efe9819da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:36.220 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.036+0000 7efe97200700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9810e020 con 0x7efe98109ac0 2024-03-21T10:26:36.220 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.036+0000 7efe95e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe98109ac0 0x7efe9819da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:36.220 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.036+0000 7efe95e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe98109ac0 0x7efe9819da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53722/0 (socket says 172.21.15.31:53722) 2024-03-21T10:26:36.220 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.036+0000 7efe95e00700 1 -- 172.21.15.31:0/990880158 learned_addr learned my addr 172.21.15.31:0/990880158 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:36.220 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.037+0000 7efe95e00700 1 -- 172.21.15.31:0/990880158 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9819dfa0 con 0x7efe98109ac0 2024-03-21T10:26:36.220 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.037+0000 7efe95e00700 1 --2- 172.21.15.31:0/990880158 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe98109ac0 0x7efe9819da60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efe88011850 tx=0x7efe88011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:36.220 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.037+0000 7efe8ea00700 1 -- 172.21.15.31:0/990880158 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe88003af0 con 0x7efe98109ac0 2024-03-21T10:26:36.220 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.037+0000 7efe8ea00700 1 -- 172.21.15.31:0/990880158 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe88020070 con 0x7efe98109ac0 2024-03-21T10:26:36.220 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.037+0000 7efe8ea00700 1 -- 172.21.15.31:0/990880158 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe880195a0 con 0x7efe98109ac0 2024-03-21T10:26:36.220 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.037+0000 7efe97200700 1 -- 172.21.15.31:0/990880158 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe9819e230 con 0x7efe98109ac0 2024-03-21T10:26:36.220 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.038+0000 7efe97200700 1 -- 172.21.15.31:0/990880158 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe981a0f20 con 0x7efe98109ac0 2024-03-21T10:26:36.220 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.039+0000 7efe8ea00700 1 -- 172.21.15.31:0/990880158 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) v1 ==== 46294+0+0 (secure 0 0 0) 0x7efe88019850 con 0x7efe98109ac0 2024-03-21T10:26:36.221 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.039+0000 7efe97200700 1 -- 172.21.15.31:0/990880158 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe980637c0 con 0x7efe98109ac0 2024-03-21T10:26:36.221 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.039+0000 7efe8ea00700 1 --2- 172.21.15.31:0/990880158 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7efe7c0391d0 0x7efe7c03b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:36.221 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.039+0000 7efe8ea00700 1 -- 172.21.15.31:0/990880158 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7efe88011e00 con 0x7efe98109ac0 2024-03-21T10:26:36.221 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.040+0000 7efe95400700 1 --2- 172.21.15.31:0/990880158 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7efe7c0391d0 0x7efe7c03b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:36.221 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.041+0000 7efe95400700 1 --2- 172.21.15.31:0/990880158 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7efe7c0391d0 0x7efe7c03b690 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7efe84006fd0 tx=0x7efe84006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:36.221 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.046+0000 7efe8ea00700 1 -- 172.21.15.31:0/990880158 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efe88019b40 con 0x7efe98109ac0 2024-03-21T10:26:36.221 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.290+0000 7efe97200700 1 -- 172.21.15.31:0/990880158 --> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7efe98107d20 con 0x7efe7c0391d0 2024-03-21T10:26:36.221 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.302+0000 7efe8ea00700 1 -- 172.21.15.31:0/990880158 <== mgr.14120 v2:172.21.15.31:6800/4085461895 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7efe98107d20 con 0x7efe7c0391d0 2024-03-21T10:26:36.221 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.306+0000 7efe97200700 1 -- 172.21.15.31:0/990880158 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7efe7c0391d0 msgr2=0x7efe7c03b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:36.221 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.306+0000 7efe97200700 1 --2- 172.21.15.31:0/990880158 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7efe7c0391d0 0x7efe7c03b690 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7efe84006fd0 tx=0x7efe84006e40 comp rx=0 tx=0).stop 2024-03-21T10:26:36.221 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.306+0000 7efe97200700 1 -- 172.21.15.31:0/990880158 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe98109ac0 msgr2=0x7efe9819da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:36.221 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.306+0000 7efe97200700 1 --2- 172.21.15.31:0/990880158 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe98109ac0 0x7efe9819da60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efe88011850 tx=0x7efe88011c10 comp rx=0 tx=0).stop 2024-03-21T10:26:36.222 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.306+0000 7efe97200700 1 -- 172.21.15.31:0/990880158 shutdown_connections 2024-03-21T10:26:36.222 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.306+0000 7efe97200700 1 --2- 172.21.15.31:0/990880158 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7efe7c0391d0 0x7efe7c03b690 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:36.223 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.306+0000 7efe97200700 1 --2- 172.21.15.31:0/990880158 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe98109ac0 0x7efe9819da60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:36.223 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.306+0000 7efe97200700 1 -- 172.21.15.31:0/990880158 >> 172.21.15.31:0/990880158 conn(0x7efe9807cf50 msgr2=0x7efe98107610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:36.223 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.306+0000 7efe97200700 1 -- 172.21.15.31:0/990880158 shutdown_connections 2024-03-21T10:26:36.223 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:35.307+0000 7efe97200700 1 -- 172.21.15.31:0/990880158 wait complete. 2024-03-21T10:26:36.223 INFO:teuthology.orchestra.run.smithi031.stdout:Deploying unmanaged mgr service... 2024-03-21T10:26:36.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:36 smithi031 ceph-mon[31673]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:26:36.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:36 smithi031 ceph-mon[31673]: Saving service mon spec with placement count:5 2024-03-21T10:26:36.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:36 smithi031 ceph-mon[31673]: from='mgr.14120 172.21.15.31:0/4221700501' entity='mgr.x' 2024-03-21T10:26:38.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:38 smithi031 ceph-mon[31673]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:26:38.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:38 smithi031 ceph-mon[31673]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:26:38.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:38 smithi031 ceph-mon[31673]: Saving service mgr spec with placement count:2 2024-03-21T10:26:38.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:38 smithi031 ceph-mon[31673]: from='mgr.14120 172.21.15.31:0/4221700501' entity='mgr.x' 2024-03-21T10:26:38.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:38 smithi031 ceph-mon[31673]: from='mgr.14120 172.21.15.31:0/4221700501' entity='mgr.x' 2024-03-21T10:26:38.715 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-21T10:26:38.716 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.176+0000 7f6925c00700 1 Processor -- start 2024-03-21T10:26:38.717 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.177+0000 7f6925c00700 1 -- start start 2024-03-21T10:26:38.717 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.177+0000 7f6925c00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6920107970 0x7f6920107d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:38.717 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.177+0000 7f6925c00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6920108360 con 0x7f6920107970 2024-03-21T10:26:38.717 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.177+0000 7f691fe00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6920107970 0x7f6920107d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:38.717 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.177+0000 7f691fe00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6920107970 0x7f6920107d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50478/0 (socket says 172.21.15.31:50478) 2024-03-21T10:26:38.717 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.177+0000 7f691fe00700 1 -- 172.21.15.31:0/256663473 learned_addr learned my addr 172.21.15.31:0/256663473 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:38.717 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.178+0000 7f691fe00700 1 -- 172.21.15.31:0/256663473 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6920108ad0 con 0x7f6920107970 2024-03-21T10:26:38.717 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.178+0000 7f691fe00700 1 --2- 172.21.15.31:0/256663473 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6920107970 0x7f6920107d90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6910009e80 tx=0x7f691000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a416174dbc521a67 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:38.721 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.178+0000 7f691f400700 1 -- 172.21.15.31:0/256663473 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6910017070 con 0x7f6920107970 2024-03-21T10:26:38.721 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.179+0000 7f691f400700 1 -- 172.21.15.31:0/256663473 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f691000b8c0 con 0x7f6920107970 2024-03-21T10:26:38.721 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.179+0000 7f691f400700 1 -- 172.21.15.31:0/256663473 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69100045c0 con 0x7f6920107970 2024-03-21T10:26:38.721 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.179+0000 7f6925c00700 1 -- 172.21.15.31:0/256663473 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6920107970 msgr2=0x7f6920107d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:38.721 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.179+0000 7f6925c00700 1 --2- 172.21.15.31:0/256663473 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6920107970 0x7f6920107d90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6910009e80 tx=0x7f691000b330 comp rx=0 tx=0).stop 2024-03-21T10:26:38.721 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.179+0000 7f6925c00700 1 -- 172.21.15.31:0/256663473 shutdown_connections 2024-03-21T10:26:38.721 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.179+0000 7f6925c00700 1 --2- 172.21.15.31:0/256663473 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6920107970 0x7f6920107d90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:38.721 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.179+0000 7f6925c00700 1 -- 172.21.15.31:0/256663473 >> 172.21.15.31:0/256663473 conn(0x7f6920102f50 msgr2=0x7f6920105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:38.721 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.180+0000 7f6925c00700 1 -- 172.21.15.31:0/256663473 shutdown_connections 2024-03-21T10:26:38.721 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.180+0000 7f6925c00700 1 -- 172.21.15.31:0/256663473 wait complete. 2024-03-21T10:26:38.721 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.181+0000 7f6925c00700 1 Processor -- start 2024-03-21T10:26:38.722 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.181+0000 7f6925c00700 1 -- start start 2024-03-21T10:26:38.722 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.181+0000 7f6925c00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6920107970 0x7f692019dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:38.722 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.181+0000 7f6925c00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692010be80 con 0x7f6920107970 2024-03-21T10:26:38.722 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.182+0000 7f691fe00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6920107970 0x7f692019dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:38.722 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.182+0000 7f691fe00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6920107970 0x7f692019dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50492/0 (socket says 172.21.15.31:50492) 2024-03-21T10:26:38.722 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.182+0000 7f691fe00700 1 -- 172.21.15.31:0/2289370788 learned_addr learned my addr 172.21.15.31:0/2289370788 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:38.722 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.182+0000 7f691fe00700 1 -- 172.21.15.31:0/2289370788 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f692019e010 con 0x7f6920107970 2024-03-21T10:26:38.722 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.182+0000 7f691fe00700 1 --2- 172.21.15.31:0/2289370788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6920107970 0x7f692019dad0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6910004210 tx=0x7f691000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:38.723 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.183+0000 7f691d600700 1 -- 172.21.15.31:0/2289370788 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6910020020 con 0x7f6920107970 2024-03-21T10:26:38.723 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.183+0000 7f691d600700 1 -- 172.21.15.31:0/2289370788 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6910003b50 con 0x7f6920107970 2024-03-21T10:26:38.723 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.183+0000 7f691d600700 1 -- 172.21.15.31:0/2289370788 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6910017040 con 0x7f6920107970 2024-03-21T10:26:38.723 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.183+0000 7f6925c00700 1 -- 172.21.15.31:0/2289370788 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f692019e2a0 con 0x7f6920107970 2024-03-21T10:26:38.723 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.183+0000 7f6925c00700 1 -- 172.21.15.31:0/2289370788 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69201a0f40 con 0x7f6920107970 2024-03-21T10:26:38.723 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.184+0000 7f691d600700 1 -- 172.21.15.31:0/2289370788 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) v1 ==== 46294+0+0 (secure 0 0 0) 0x7f6910023020 con 0x7f6920107970 2024-03-21T10:26:38.723 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.184+0000 7f6925c00700 1 -- 172.21.15.31:0/2289370788 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69200637c0 con 0x7f6920107970 2024-03-21T10:26:38.723 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.184+0000 7f691d600700 1 --2- 172.21.15.31:0/2289370788 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f69040391d0 0x7f690403b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:38.723 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.184+0000 7f691d600700 1 -- 172.21.15.31:0/2289370788 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f691000f050 con 0x7f6920107970 2024-03-21T10:26:38.723 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.185+0000 7f6917400700 1 --2- 172.21.15.31:0/2289370788 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f69040391d0 0x7f690403b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:38.723 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.185+0000 7f6917400700 1 --2- 172.21.15.31:0/2289370788 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f69040391d0 0x7f690403b690 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6908006fd0 tx=0x7f6908006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:38.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.190+0000 7f691d600700 1 -- 172.21.15.31:0/2289370788 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f691002a020 con 0x7f6920107970 2024-03-21T10:26:38.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.440+0000 7f6925c00700 1 -- 172.21.15.31:0/2289370788 --> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f6920104130 con 0x7f69040391d0 2024-03-21T10:26:38.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.454+0000 7f691d600700 1 -- 172.21.15.31:0/2289370788 <== mgr.14120 v2:172.21.15.31:6800/4085461895 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6920104130 con 0x7f69040391d0 2024-03-21T10:26:38.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.456+0000 7f6925c00700 1 -- 172.21.15.31:0/2289370788 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f69040391d0 msgr2=0x7f690403b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:38.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.457+0000 7f6925c00700 1 --2- 172.21.15.31:0/2289370788 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f69040391d0 0x7f690403b690 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6908006fd0 tx=0x7f6908006e40 comp rx=0 tx=0).stop 2024-03-21T10:26:38.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.457+0000 7f6925c00700 1 -- 172.21.15.31:0/2289370788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6920107970 msgr2=0x7f692019dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:38.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.457+0000 7f6925c00700 1 --2- 172.21.15.31:0/2289370788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6920107970 0x7f692019dad0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6910004210 tx=0x7f691000bd50 comp rx=0 tx=0).stop 2024-03-21T10:26:38.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.457+0000 7f6925c00700 1 -- 172.21.15.31:0/2289370788 shutdown_connections 2024-03-21T10:26:38.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.457+0000 7f6925c00700 1 --2- 172.21.15.31:0/2289370788 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f69040391d0 0x7f690403b690 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:38.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.457+0000 7f6925c00700 1 --2- 172.21.15.31:0/2289370788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6920107970 0x7f692019dad0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:38.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.457+0000 7f6925c00700 1 -- 172.21.15.31:0/2289370788 >> 172.21.15.31:0/2289370788 conn(0x7f6920102f50 msgr2=0x7f6920103a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:38.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.457+0000 7f6925c00700 1 -- 172.21.15.31:0/2289370788 shutdown_connections 2024-03-21T10:26:38.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:37.457+0000 7f6925c00700 1 -- 172.21.15.31:0/2289370788 wait complete. 2024-03-21T10:26:40.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:40 smithi031 ceph-mon[31673]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:26:40.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:40 smithi031 ceph-mon[31673]: from='mgr.14120 172.21.15.31:0/4221700501' entity='mgr.x' 2024-03-21T10:26:40.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:40 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2198476866' entity='client.admin' 2024-03-21T10:26:40.575 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.303+0000 7f6284a00700 1 Processor -- start 2024-03-21T10:26:40.575 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.303+0000 7f6284a00700 1 -- start start 2024-03-21T10:26:40.575 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.304+0000 7f6284a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6280107a80 0x7f6280107ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:40.575 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.304+0000 7f6284a00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6280108470 con 0x7f6280107a80 2024-03-21T10:26:40.575 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.304+0000 7f627de00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6280107a80 0x7f6280107ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:40.575 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.304+0000 7f627de00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6280107a80 0x7f6280107ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50500/0 (socket says 172.21.15.31:50500) 2024-03-21T10:26:40.575 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.304+0000 7f627de00700 1 -- 172.21.15.31:0/1150997767 learned_addr learned my addr 172.21.15.31:0/1150997767 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:40.575 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.305+0000 7f627de00700 1 -- 172.21.15.31:0/1150997767 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6280108c80 con 0x7f6280107a80 2024-03-21T10:26:40.575 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.305+0000 7f627de00700 1 --2- 172.21.15.31:0/1150997767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6280107a80 0x7f6280107ea0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f62700098a0 tx=0x7f6270009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a83d3cecf7e39d45 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:40.576 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.305+0000 7f627d400700 1 -- 172.21.15.31:0/1150997767 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6270004160 con 0x7f6280107a80 2024-03-21T10:26:40.576 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.306+0000 7f627d400700 1 -- 172.21.15.31:0/1150997767 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6270004300 con 0x7f6280107a80 2024-03-21T10:26:40.576 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.306+0000 7f627d400700 1 -- 172.21.15.31:0/1150997767 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6270011b20 con 0x7f6280107a80 2024-03-21T10:26:40.576 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.306+0000 7f6284a00700 1 -- 172.21.15.31:0/1150997767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6280107a80 msgr2=0x7f6280107ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:40.576 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.306+0000 7f6284a00700 1 --2- 172.21.15.31:0/1150997767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6280107a80 0x7f6280107ea0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f62700098a0 tx=0x7f6270009bb0 comp rx=0 tx=0).stop 2024-03-21T10:26:40.576 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.307+0000 7f6284a00700 1 -- 172.21.15.31:0/1150997767 shutdown_connections 2024-03-21T10:26:40.576 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.307+0000 7f6284a00700 1 --2- 172.21.15.31:0/1150997767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6280107a80 0x7f6280107ea0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:40.576 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.307+0000 7f6284a00700 1 -- 172.21.15.31:0/1150997767 >> 172.21.15.31:0/1150997767 conn(0x7f6280103080 msgr2=0x7f62801054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:40.576 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.307+0000 7f6284a00700 1 -- 172.21.15.31:0/1150997767 shutdown_connections 2024-03-21T10:26:40.576 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.307+0000 7f6284a00700 1 -- 172.21.15.31:0/1150997767 wait complete. 2024-03-21T10:26:40.576 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.308+0000 7f6284a00700 1 Processor -- start 2024-03-21T10:26:40.576 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.308+0000 7f6284a00700 1 -- start start 2024-03-21T10:26:40.576 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.308+0000 7f6284a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6280107a80 0x7f628019dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:40.576 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.308+0000 7f6284a00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628010c030 con 0x7f6280107a80 2024-03-21T10:26:40.576 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.309+0000 7f627de00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6280107a80 0x7f628019dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:40.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.309+0000 7f627de00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6280107a80 0x7f628019dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50514/0 (socket says 172.21.15.31:50514) 2024-03-21T10:26:40.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.309+0000 7f627de00700 1 -- 172.21.15.31:0/2198476866 learned_addr learned my addr 172.21.15.31:0/2198476866 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:40.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.309+0000 7f627de00700 1 -- 172.21.15.31:0/2198476866 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628019e120 con 0x7f6280107a80 2024-03-21T10:26:40.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.310+0000 7f627de00700 1 --2- 172.21.15.31:0/2198476866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6280107a80 0x7f628019dbe0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f62700117e0 tx=0x7f6270011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:40.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.310+0000 7f6276a00700 1 -- 172.21.15.31:0/2198476866 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6270003bf0 con 0x7f6280107a80 2024-03-21T10:26:40.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.310+0000 7f6276a00700 1 -- 172.21.15.31:0/2198476866 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f627001a910 con 0x7f6280107a80 2024-03-21T10:26:40.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.310+0000 7f6284a00700 1 -- 172.21.15.31:0/2198476866 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f628019e3b0 con 0x7f6280107a80 2024-03-21T10:26:40.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.310+0000 7f6276a00700 1 -- 172.21.15.31:0/2198476866 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6270024970 con 0x7f6280107a80 2024-03-21T10:26:40.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.310+0000 7f6284a00700 1 -- 172.21.15.31:0/2198476866 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f628019e810 con 0x7f6280107a80 2024-03-21T10:26:40.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.311+0000 7f6276a00700 1 -- 172.21.15.31:0/2198476866 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) v1 ==== 46294+0+0 (secure 0 0 0) 0x7f6270022020 con 0x7f6280107a80 2024-03-21T10:26:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.311+0000 7f6284a00700 1 -- 172.21.15.31:0/2198476866 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62800637c0 con 0x7f6280107a80 2024-03-21T10:26:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.312+0000 7f6276a00700 1 --2- 172.21.15.31:0/2198476866 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f6264039180 0x7f626403b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.312+0000 7f6276a00700 1 -- 172.21.15.31:0/2198476866 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f627004d930 con 0x7f6280107a80 2024-03-21T10:26:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.312+0000 7f6277e00700 1 --2- 172.21.15.31:0/2198476866 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f6264039180 0x7f626403b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.313+0000 7f6277e00700 1 --2- 172.21.15.31:0/2198476866 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f6264039180 0x7f626403b640 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6268006fd0 tx=0x7f6268006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.318+0000 7f6276a00700 1 -- 172.21.15.31:0/2198476866 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f627000f2e0 con 0x7f6280107a80 2024-03-21T10:26:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.548+0000 7f6284a00700 1 -- 172.21.15.31:0/2198476866 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f62801a10a0 con 0x7f6280107a80 2024-03-21T10:26:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.555+0000 7f6276a00700 1 -- 172.21.15.31:0/2198476866 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f627001ab70 con 0x7f6280107a80 2024-03-21T10:26:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.558+0000 7f6284a00700 1 -- 172.21.15.31:0/2198476866 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f6264039180 msgr2=0x7f626403b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.558+0000 7f6284a00700 1 --2- 172.21.15.31:0/2198476866 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f6264039180 0x7f626403b640 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6268006fd0 tx=0x7f6268006e40 comp rx=0 tx=0).stop 2024-03-21T10:26:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.558+0000 7f6284a00700 1 -- 172.21.15.31:0/2198476866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6280107a80 msgr2=0x7f628019dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.558+0000 7f6284a00700 1 --2- 172.21.15.31:0/2198476866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6280107a80 0x7f628019dbe0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f62700117e0 tx=0x7f6270011ba0 comp rx=0 tx=0).stop 2024-03-21T10:26:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.558+0000 7f6284a00700 1 -- 172.21.15.31:0/2198476866 shutdown_connections 2024-03-21T10:26:40.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.558+0000 7f6284a00700 1 --2- 172.21.15.31:0/2198476866 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f6264039180 0x7f626403b640 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:40.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.558+0000 7f6284a00700 1 --2- 172.21.15.31:0/2198476866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6280107a80 0x7f628019dbe0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:40.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.558+0000 7f6284a00700 1 -- 172.21.15.31:0/2198476866 >> 172.21.15.31:0/2198476866 conn(0x7f6280103080 msgr2=0x7f6280103a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:40.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.558+0000 7f6284a00700 1 -- 172.21.15.31:0/2198476866 shutdown_connections 2024-03-21T10:26:40.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:39.558+0000 7f6284a00700 1 -- 172.21.15.31:0/2198476866 wait complete. 2024-03-21T10:26:42.727 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.325+0000 7f04f2a00700 1 Processor -- start 2024-03-21T10:26:42.727 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.325+0000 7f04f2a00700 1 -- start start 2024-03-21T10:26:42.728 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.326+0000 7f04f2a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04ec07de60 0x7f04ec07c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:42.728 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.326+0000 7f04f2a00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec07c890 con 0x7f04ec07de60 2024-03-21T10:26:42.728 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.326+0000 7f04ebe00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04ec07de60 0x7f04ec07c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:42.728 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.326+0000 7f04ebe00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04ec07de60 0x7f04ec07c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50520/0 (socket says 172.21.15.31:50520) 2024-03-21T10:26:42.728 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.326+0000 7f04ebe00700 1 -- 172.21.15.31:0/395991394 learned_addr learned my addr 172.21.15.31:0/395991394 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:42.728 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.326+0000 7f04ebe00700 1 -- 172.21.15.31:0/395991394 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ec07ca10 con 0x7f04ec07de60 2024-03-21T10:26:42.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.327+0000 7f04ebe00700 1 --2- 172.21.15.31:0/395991394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04ec07de60 0x7f04ec07c2c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0098a0 tx=0x7f04dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20ee94171abf38ba server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:42.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.327+0000 7f04eb400700 1 -- 172.21.15.31:0/395991394 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f04dc004160 con 0x7f04ec07de60 2024-03-21T10:26:42.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.327+0000 7f04eb400700 1 -- 172.21.15.31:0/395991394 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04dc004300 con 0x7f04ec07de60 2024-03-21T10:26:42.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.327+0000 7f04eb400700 1 -- 172.21.15.31:0/395991394 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f04dc011b20 con 0x7f04ec07de60 2024-03-21T10:26:42.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.328+0000 7f04f2a00700 1 -- 172.21.15.31:0/395991394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04ec07de60 msgr2=0x7f04ec07c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:42.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.328+0000 7f04f2a00700 1 --2- 172.21.15.31:0/395991394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04ec07de60 0x7f04ec07c2c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0098a0 tx=0x7f04dc009bb0 comp rx=0 tx=0).stop 2024-03-21T10:26:42.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.328+0000 7f04f2a00700 1 -- 172.21.15.31:0/395991394 shutdown_connections 2024-03-21T10:26:42.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.328+0000 7f04f2a00700 1 --2- 172.21.15.31:0/395991394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04ec07de60 0x7f04ec07c2c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:42.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.328+0000 7f04f2a00700 1 -- 172.21.15.31:0/395991394 >> 172.21.15.31:0/395991394 conn(0x7f04ec103080 msgr2=0x7f04ec1054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:42.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.328+0000 7f04f2a00700 1 -- 172.21.15.31:0/395991394 shutdown_connections 2024-03-21T10:26:42.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.328+0000 7f04f2a00700 1 -- 172.21.15.31:0/395991394 wait complete. 2024-03-21T10:26:42.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.329+0000 7f04f2a00700 1 Processor -- start 2024-03-21T10:26:42.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.329+0000 7f04f2a00700 1 -- start start 2024-03-21T10:26:42.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.329+0000 7f04f2a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04ec07de60 0x7f04ec19dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:42.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.329+0000 7f04f2a00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec10e100 con 0x7f04ec07de60 2024-03-21T10:26:42.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.330+0000 7f04ebe00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04ec07de60 0x7f04ec19dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:42.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.330+0000 7f04ebe00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04ec07de60 0x7f04ec19dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50536/0 (socket says 172.21.15.31:50536) 2024-03-21T10:26:42.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.330+0000 7f04ebe00700 1 -- 172.21.15.31:0/765658785 learned_addr learned my addr 172.21.15.31:0/765658785 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:42.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.330+0000 7f04ebe00700 1 -- 172.21.15.31:0/765658785 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ec19e0e0 con 0x7f04ec07de60 2024-03-21T10:26:42.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.330+0000 7f04ebe00700 1 --2- 172.21.15.31:0/765658785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04ec07de60 0x7f04ec19dba0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0117e0 tx=0x7f04dc011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:42.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.331+0000 7f04e9600700 1 -- 172.21.15.31:0/765658785 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f04dc003ba0 con 0x7f04ec07de60 2024-03-21T10:26:42.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.331+0000 7f04e9600700 1 -- 172.21.15.31:0/765658785 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04dc01a910 con 0x7f04ec07de60 2024-03-21T10:26:42.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.331+0000 7f04e9600700 1 -- 172.21.15.31:0/765658785 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f04dc0248f0 con 0x7f04ec07de60 2024-03-21T10:26:42.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.331+0000 7f04f2a00700 1 -- 172.21.15.31:0/765658785 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04ec19e370 con 0x7f04ec07de60 2024-03-21T10:26:42.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.331+0000 7f04f2a00700 1 -- 172.21.15.31:0/765658785 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04ec19e7d0 con 0x7f04ec07de60 2024-03-21T10:26:42.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.332+0000 7f04e9600700 1 -- 172.21.15.31:0/765658785 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) v1 ==== 46294+0+0 (secure 0 0 0) 0x7f04dc022020 con 0x7f04ec07de60 2024-03-21T10:26:42.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.332+0000 7f04f2a00700 1 -- 172.21.15.31:0/765658785 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04ec0637c0 con 0x7f04ec07de60 2024-03-21T10:26:42.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.333+0000 7f04e9600700 1 --2- 172.21.15.31:0/765658785 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f04d00391d0 0x7f04d003b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:42.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.333+0000 7f04e9600700 1 -- 172.21.15.31:0/765658785 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f04dc04d590 con 0x7f04ec07de60 2024-03-21T10:26:42.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.333+0000 7f04e3e00700 1 --2- 172.21.15.31:0/765658785 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f04d00391d0 0x7f04d003b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:42.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.334+0000 7f04e3e00700 1 --2- 172.21.15.31:0/765658785 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f04d00391d0 0x7f04d003b690 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f04d4006fd0 tx=0x7f04d4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:42.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.338+0000 7f04e9600700 1 -- 172.21.15.31:0/765658785 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f04dc01ab70 con 0x7f04ec07de60 2024-03-21T10:26:42.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.571+0000 7f04f2a00700 1 -- 172.21.15.31:0/765658785 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f04ec1a11a0 con 0x7f04ec07de60 2024-03-21T10:26:42.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.577+0000 7f04e9600700 1 -- 172.21.15.31:0/765658785 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f04dc00f2e0 con 0x7f04ec07de60 2024-03-21T10:26:42.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.580+0000 7f04f2a00700 1 -- 172.21.15.31:0/765658785 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f04d00391d0 msgr2=0x7f04d003b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:42.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.580+0000 7f04f2a00700 1 --2- 172.21.15.31:0/765658785 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f04d00391d0 0x7f04d003b690 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f04d4006fd0 tx=0x7f04d4006e40 comp rx=0 tx=0).stop 2024-03-21T10:26:42.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.580+0000 7f04f2a00700 1 -- 172.21.15.31:0/765658785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04ec07de60 msgr2=0x7f04ec19dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:42.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.580+0000 7f04f2a00700 1 --2- 172.21.15.31:0/765658785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04ec07de60 0x7f04ec19dba0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0117e0 tx=0x7f04dc011ba0 comp rx=0 tx=0).stop 2024-03-21T10:26:42.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.581+0000 7f04f2a00700 1 -- 172.21.15.31:0/765658785 shutdown_connections 2024-03-21T10:26:42.732 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.581+0000 7f04f2a00700 1 --2- 172.21.15.31:0/765658785 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f04d00391d0 0x7f04d003b690 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:42.732 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.581+0000 7f04f2a00700 1 --2- 172.21.15.31:0/765658785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04ec07de60 0x7f04ec19dba0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:42.732 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.581+0000 7f04f2a00700 1 -- 172.21.15.31:0/765658785 >> 172.21.15.31:0/765658785 conn(0x7f04ec103080 msgr2=0x7f04ec103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:42.732 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.581+0000 7f04f2a00700 1 -- 172.21.15.31:0/765658785 shutdown_connections 2024-03-21T10:26:42.732 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:41.581+0000 7f04f2a00700 1 -- 172.21.15.31:0/765658785 wait complete. 2024-03-21T10:26:42.732 INFO:teuthology.orchestra.run.smithi031.stdout:Enabling the dashboard module... 2024-03-21T10:26:42.872 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:42 smithi031 ceph-mon[31673]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:26:42.872 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:42 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/765658785' entity='client.admin' 2024-03-21T10:26:42.872 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:42 smithi031 ceph-mon[31673]: from='mgr.14120 172.21.15.31:0/4221700501' entity='mgr.x' 2024-03-21T10:26:45.016 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:44 smithi031 ceph-mon[31673]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:26:45.016 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:44 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/553513799' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-21T10:26:45.016 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:44 smithi031 ceph-mon[31673]: from='mgr.14120 172.21.15.31:0/4221700501' entity='mgr.x' 2024-03-21T10:26:45.016 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:44 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: ignoring --setuser ceph since I am not root 2024-03-21T10:26:45.017 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:44 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: ignoring --setgroup ceph since I am not root 2024-03-21T10:26:45.359 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.368+0000 7fdd96e00700 1 Processor -- start 2024-03-21T10:26:45.359 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.369+0000 7fdd96e00700 1 -- start start 2024-03-21T10:26:45.359 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.369+0000 7fdd96e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd901085c0 0x7fdd9010a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:45.359 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.369+0000 7fdd96e00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9007bdb0 con 0x7fdd901085c0 2024-03-21T10:26:45.359 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.370+0000 7fdd8fe00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd901085c0 0x7fdd9010a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:45.359 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.370+0000 7fdd8fe00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd901085c0 0x7fdd9010a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50542/0 (socket says 172.21.15.31:50542) 2024-03-21T10:26:45.359 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.370+0000 7fdd8fe00700 1 -- 172.21.15.31:0/803810938 learned_addr learned my addr 172.21.15.31:0/803810938 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:45.359 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.370+0000 7fdd8fe00700 1 -- 172.21.15.31:0/803810938 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd9010aef0 con 0x7fdd901085c0 2024-03-21T10:26:45.359 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.371+0000 7fdd8fe00700 1 --2- 172.21.15.31:0/803810938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd901085c0 0x7fdd9010a9b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdd800098a0 tx=0x7fdd80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b20c04211d98d85c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:45.359 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.371+0000 7fdd8ea00700 1 -- 172.21.15.31:0/803810938 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd80004160 con 0x7fdd901085c0 2024-03-21T10:26:45.359 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.371+0000 7fdd8ea00700 1 -- 172.21.15.31:0/803810938 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd80004300 con 0x7fdd901085c0 2024-03-21T10:26:45.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.371+0000 7fdd8ea00700 1 -- 172.21.15.31:0/803810938 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd80011b20 con 0x7fdd901085c0 2024-03-21T10:26:45.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.372+0000 7fdd96e00700 1 -- 172.21.15.31:0/803810938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd901085c0 msgr2=0x7fdd9010a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:45.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.372+0000 7fdd96e00700 1 --2- 172.21.15.31:0/803810938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd901085c0 0x7fdd9010a9b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdd800098a0 tx=0x7fdd80009bb0 comp rx=0 tx=0).stop 2024-03-21T10:26:45.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.372+0000 7fdd96e00700 1 -- 172.21.15.31:0/803810938 shutdown_connections 2024-03-21T10:26:45.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.372+0000 7fdd96e00700 1 --2- 172.21.15.31:0/803810938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd901085c0 0x7fdd9010a9b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:45.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.372+0000 7fdd96e00700 1 -- 172.21.15.31:0/803810938 >> 172.21.15.31:0/803810938 conn(0x7fdd90101f70 msgr2=0x7fdd90104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:45.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.372+0000 7fdd96e00700 1 -- 172.21.15.31:0/803810938 shutdown_connections 2024-03-21T10:26:45.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.372+0000 7fdd96e00700 1 -- 172.21.15.31:0/803810938 wait complete. 2024-03-21T10:26:45.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.373+0000 7fdd96e00700 1 Processor -- start 2024-03-21T10:26:45.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.374+0000 7fdd96e00700 1 -- start start 2024-03-21T10:26:45.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.374+0000 7fdd96e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd901085c0 0x7fdd90107e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:45.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.374+0000 7fdd96e00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9010be90 con 0x7fdd901085c0 2024-03-21T10:26:45.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.374+0000 7fdd8fe00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd901085c0 0x7fdd90107e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:45.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.374+0000 7fdd8fe00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd901085c0 0x7fdd90107e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50548/0 (socket says 172.21.15.31:50548) 2024-03-21T10:26:45.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.374+0000 7fdd8fe00700 1 -- 172.21.15.31:0/553513799 learned_addr learned my addr 172.21.15.31:0/553513799 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:45.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.375+0000 7fdd8fe00700 1 -- 172.21.15.31:0/553513799 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd90108380 con 0x7fdd901085c0 2024-03-21T10:26:45.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.375+0000 7fdd8fe00700 1 --2- 172.21.15.31:0/553513799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd901085c0 0x7fdd90107e40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdd800117e0 tx=0x7fdd80011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:45.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.375+0000 7fdd8cc00700 1 -- 172.21.15.31:0/553513799 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd80003ba0 con 0x7fdd901085c0 2024-03-21T10:26:45.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.375+0000 7fdd8cc00700 1 -- 172.21.15.31:0/553513799 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd8001a910 con 0x7fdd901085c0 2024-03-21T10:26:45.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.375+0000 7fdd96e00700 1 -- 172.21.15.31:0/553513799 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd90106570 con 0x7fdd901085c0 2024-03-21T10:26:45.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.375+0000 7fdd8cc00700 1 -- 172.21.15.31:0/553513799 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd800248f0 con 0x7fdd901085c0 2024-03-21T10:26:45.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.375+0000 7fdd96e00700 1 -- 172.21.15.31:0/553513799 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd90106a30 con 0x7fdd901085c0 2024-03-21T10:26:45.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.377+0000 7fdd96e00700 1 -- 172.21.15.31:0/553513799 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd900637c0 con 0x7fdd901085c0 2024-03-21T10:26:45.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.377+0000 7fdd8cc00700 1 -- 172.21.15.31:0/553513799 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) v1 ==== 46294+0+0 (secure 0 0 0) 0x7fdd80022020 con 0x7fdd901085c0 2024-03-21T10:26:45.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.377+0000 7fdd8cc00700 1 --2- 172.21.15.31:0/553513799 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7fdd740391d0 0x7fdd7403b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:45.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.377+0000 7fdd8cc00700 1 -- 172.21.15.31:0/553513799 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fdd8004d650 con 0x7fdd901085c0 2024-03-21T10:26:45.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.377+0000 7fdd8f400700 1 --2- 172.21.15.31:0/553513799 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7fdd740391d0 0x7fdd7403b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:45.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.379+0000 7fdd8f400700 1 --2- 172.21.15.31:0/553513799 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7fdd740391d0 0x7fdd7403b690 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c006fd0 tx=0x7fdd7c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:45.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.384+0000 7fdd8cc00700 1 -- 172.21.15.31:0/553513799 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdd8000b3e0 con 0x7fdd901085c0 2024-03-21T10:26:45.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:43.682+0000 7fdd96e00700 1 -- 172.21.15.31:0/553513799 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fdd90106d20 con 0x7fdd901085c0 2024-03-21T10:26:45.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:44.592+0000 7fdd8cc00700 1 -- 172.21.15.31:0/553513799 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fdd90106d20 con 0x7fdd901085c0 2024-03-21T10:26:45.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:44.595+0000 7fdd8cc00700 1 -- 172.21.15.31:0/553513799 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mgrmap(e 9) v1 ==== 46307+0+0 (secure 0 0 0) 0x7fdd80024a90 con 0x7fdd901085c0 2024-03-21T10:26:45.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:44.596+0000 7fdd96e00700 1 -- 172.21.15.31:0/553513799 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7fdd740391d0 msgr2=0x7fdd7403b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:45.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:44.596+0000 7fdd96e00700 1 --2- 172.21.15.31:0/553513799 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7fdd740391d0 0x7fdd7403b690 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c006fd0 tx=0x7fdd7c006e40 comp rx=0 tx=0).stop 2024-03-21T10:26:45.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:44.596+0000 7fdd96e00700 1 -- 172.21.15.31:0/553513799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd901085c0 msgr2=0x7fdd90107e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:45.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:44.596+0000 7fdd96e00700 1 --2- 172.21.15.31:0/553513799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd901085c0 0x7fdd90107e40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdd800117e0 tx=0x7fdd80011ba0 comp rx=0 tx=0).stop 2024-03-21T10:26:45.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:44.597+0000 7fdd96e00700 1 -- 172.21.15.31:0/553513799 shutdown_connections 2024-03-21T10:26:45.363 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:44.597+0000 7fdd96e00700 1 --2- 172.21.15.31:0/553513799 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7fdd740391d0 0x7fdd7403b690 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:45.363 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:44.597+0000 7fdd96e00700 1 --2- 172.21.15.31:0/553513799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd901085c0 0x7fdd90107e40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:45.363 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:44.597+0000 7fdd96e00700 1 -- 172.21.15.31:0/553513799 >> 172.21.15.31:0/553513799 conn(0x7fdd90101f70 msgr2=0x7fdd90102a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:45.363 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:44.598+0000 7fdd96e00700 1 -- 172.21.15.31:0/553513799 shutdown_connections 2024-03-21T10:26:45.363 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:44.598+0000 7fdd96e00700 1 -- 172.21.15.31:0/553513799 wait complete. 2024-03-21T10:26:45.595 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:45 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:45.492+0000 7f8a978dd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T10:26:45.595 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:45 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/553513799' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-21T10:26:45.595 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:45 smithi031 ceph-mon[31673]: mgrmap e9: x(active, since 27s) 2024-03-21T10:26:45.990 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:45 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:45.618+0000 7f8a978dd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T10:26:46.557 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:46 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:46.472+0000 7f8a978dd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T10:26:46.858 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:46 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2320275320' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-21T10:26:46.859 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:46 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:46.597+0000 7f8a978dd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T10:26:46.859 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:46 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:46.782+0000 7f8a978dd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T10:26:47.261 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:46 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:46.914+0000 7f8a978dd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T10:26:47.261 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:47 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:47.028+0000 7f8a978dd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T10:26:47.521 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:47 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:47.376+0000 7f8a978dd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T10:26:47.808 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:47 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:47.520+0000 7f8a978dd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T10:26:47.808 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:47 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:47.635+0000 7f8a978dd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T10:26:47.808 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:47 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:47.762+0000 7f8a978dd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T10:26:48.088 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-03-21T10:26:48.088 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-21T10:26:48.088 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-21T10:26:48.089 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-21T10:26:48.089 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-21T10:26:48.089 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-03-21T10:26:48.089 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.987+0000 7f2115e00700 1 Processor -- start 2024-03-21T10:26:48.089 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.987+0000 7f2115e00700 1 -- start start 2024-03-21T10:26:48.089 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.988+0000 7f2115e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2110107a80 0x7f2110107ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:48.089 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.988+0000 7f2115e00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2110108470 con 0x7f2110107a80 2024-03-21T10:26:48.097 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.988+0000 7f210ea00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2110107a80 0x7f2110107ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:48.097 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.988+0000 7f210ea00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2110107a80 0x7f2110107ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50580/0 (socket says 172.21.15.31:50580) 2024-03-21T10:26:48.097 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.988+0000 7f210ea00700 1 -- 172.21.15.31:0/2697504772 learned_addr learned my addr 172.21.15.31:0/2697504772 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:48.097 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.989+0000 7f210ea00700 1 -- 172.21.15.31:0/2697504772 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2110108c80 con 0x7f2110107a80 2024-03-21T10:26:48.097 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.989+0000 7f210ea00700 1 --2- 172.21.15.31:0/2697504772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2110107a80 0x7f2110107ea0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2100009e80 tx=0x7f210000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caa386e9c151dc57 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:48.097 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.990+0000 7f210d600700 1 -- 172.21.15.31:0/2697504772 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2100017070 con 0x7f2110107a80 2024-03-21T10:26:48.098 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.990+0000 7f210d600700 1 -- 172.21.15.31:0/2697504772 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f210000b8c0 con 0x7f2110107a80 2024-03-21T10:26:48.098 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.990+0000 7f210d600700 1 -- 172.21.15.31:0/2697504772 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f21000045c0 con 0x7f2110107a80 2024-03-21T10:26:48.098 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.990+0000 7f2115e00700 1 -- 172.21.15.31:0/2697504772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2110107a80 msgr2=0x7f2110107ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:48.098 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.990+0000 7f2115e00700 1 --2- 172.21.15.31:0/2697504772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2110107a80 0x7f2110107ea0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2100009e80 tx=0x7f210000b330 comp rx=0 tx=0).stop 2024-03-21T10:26:48.098 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.990+0000 7f2115e00700 1 -- 172.21.15.31:0/2697504772 shutdown_connections 2024-03-21T10:26:48.098 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.990+0000 7f2115e00700 1 --2- 172.21.15.31:0/2697504772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2110107a80 0x7f2110107ea0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:48.098 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.990+0000 7f2115e00700 1 -- 172.21.15.31:0/2697504772 >> 172.21.15.31:0/2697504772 conn(0x7f2110103080 msgr2=0x7f21101054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:48.098 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.991+0000 7f2115e00700 1 -- 172.21.15.31:0/2697504772 shutdown_connections 2024-03-21T10:26:48.098 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.991+0000 7f2115e00700 1 -- 172.21.15.31:0/2697504772 wait complete. 2024-03-21T10:26:48.098 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.991+0000 7f2115e00700 1 Processor -- start 2024-03-21T10:26:48.098 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.992+0000 7f2115e00700 1 -- start start 2024-03-21T10:26:48.098 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.992+0000 7f2115e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2110107a80 0x7f211019dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:48.098 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.992+0000 7f2115e00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211010c030 con 0x7f2110107a80 2024-03-21T10:26:48.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.992+0000 7f210ea00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2110107a80 0x7f211019dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:48.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.992+0000 7f210ea00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2110107a80 0x7f211019dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50588/0 (socket says 172.21.15.31:50588) 2024-03-21T10:26:48.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.992+0000 7f210ea00700 1 -- 172.21.15.31:0/2320275320 learned_addr learned my addr 172.21.15.31:0/2320275320 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:48.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.993+0000 7f210ea00700 1 -- 172.21.15.31:0/2320275320 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f211019e150 con 0x7f2110107a80 2024-03-21T10:26:48.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.993+0000 7f210ea00700 1 --2- 172.21.15.31:0/2320275320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2110107a80 0x7f211019dc10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2100009fd0 tx=0x7f2100003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:48.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.993+0000 7f2114a00700 1 -- 172.21.15.31:0/2320275320 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2100020020 con 0x7f2110107a80 2024-03-21T10:26:48.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.993+0000 7f2114a00700 1 -- 172.21.15.31:0/2320275320 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f210000bba0 con 0x7f2110107a80 2024-03-21T10:26:48.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.993+0000 7f2115e00700 1 -- 172.21.15.31:0/2320275320 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f211019e3e0 con 0x7f2110107a80 2024-03-21T10:26:48.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.993+0000 7f2114a00700 1 -- 172.21.15.31:0/2320275320 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2100017040 con 0x7f2110107a80 2024-03-21T10:26:48.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.993+0000 7f2115e00700 1 -- 172.21.15.31:0/2320275320 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21101a10d0 con 0x7f2110107a80 2024-03-21T10:26:48.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.994+0000 7f2115e00700 1 -- 172.21.15.31:0/2320275320 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21100637c0 con 0x7f2110107a80 2024-03-21T10:26:48.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.994+0000 7f2114a00700 1 -- 172.21.15.31:0/2320275320 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) v1 ==== 46307+0+0 (secure 0 0 0) 0x7f2100023020 con 0x7f2110107a80 2024-03-21T10:26:48.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.995+0000 7f2114a00700 1 --2- 172.21.15.31:0/2320275320 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f20f4039230 0x7f20f403b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:48.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.995+0000 7f2114a00700 1 -- 172.21.15.31:0/2320275320 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f210000f050 con 0x7f2110107a80 2024-03-21T10:26:48.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.995+0000 7f210e000700 1 -- 172.21.15.31:0/2320275320 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f20f4039230 msgr2=0x7f20f403b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/4085461895 2024-03-21T10:26:48.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:45.995+0000 7f210e000700 1 --2- 172.21.15.31:0/2320275320 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f20f4039230 0x7f20f403b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-21T10:26:48.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:46.001+0000 7f2114a00700 1 -- 172.21.15.31:0/2320275320 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f210002a020 con 0x7f2110107a80 2024-03-21T10:26:48.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:46.196+0000 7f210e000700 1 -- 172.21.15.31:0/2320275320 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f20f4039230 msgr2=0x7f20f403b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/4085461895 2024-03-21T10:26:48.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:46.196+0000 7f210e000700 1 --2- 172.21.15.31:0/2320275320 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f20f4039230 0x7f20f403b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-21T10:26:48.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:46.300+0000 7f2115e00700 1 -- 172.21.15.31:0/2320275320 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f21101a16a0 con 0x7f2110107a80 2024-03-21T10:26:48.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:46.300+0000 7f2114a00700 1 -- 172.21.15.31:0/2320275320 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f210002d350 con 0x7f2110107a80 2024-03-21T10:26:48.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:46.303+0000 7f2115e00700 1 -- 172.21.15.31:0/2320275320 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f20f4039230 msgr2=0x7f20f403b6f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T10:26:48.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:46.303+0000 7f2115e00700 1 --2- 172.21.15.31:0/2320275320 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f20f4039230 0x7f20f403b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:48.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:46.303+0000 7f2115e00700 1 -- 172.21.15.31:0/2320275320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2110107a80 msgr2=0x7f211019dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:48.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:46.303+0000 7f2115e00700 1 --2- 172.21.15.31:0/2320275320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2110107a80 0x7f211019dc10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2100009fd0 tx=0x7f2100003d00 comp rx=0 tx=0).stop 2024-03-21T10:26:48.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:46.303+0000 7f2115e00700 1 -- 172.21.15.31:0/2320275320 shutdown_connections 2024-03-21T10:26:48.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:46.303+0000 7f2115e00700 1 --2- 172.21.15.31:0/2320275320 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f20f4039230 0x7f20f403b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:48.101 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:46.303+0000 7f2115e00700 1 --2- 172.21.15.31:0/2320275320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2110107a80 0x7f211019dc10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:48.101 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:46.303+0000 7f2115e00700 1 -- 172.21.15.31:0/2320275320 >> 172.21.15.31:0/2320275320 conn(0x7f2110103080 msgr2=0x7f2110103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:48.101 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:46.303+0000 7f2115e00700 1 -- 172.21.15.31:0/2320275320 shutdown_connections 2024-03-21T10:26:48.101 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:46.303+0000 7f2115e00700 1 -- 172.21.15.31:0/2320275320 wait complete. 2024-03-21T10:26:48.101 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for the mgr to restart... 2024-03-21T10:26:48.101 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr epoch 9... 2024-03-21T10:26:48.244 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:47 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:47.985+0000 7f8a978dd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T10:26:49.807 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:49 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:49.329+0000 7f8a978dd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T10:26:49.807 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:49 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:49.508+0000 7f8a978dd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T10:26:50.307 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:49 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:49.950+0000 7f8a978dd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T10:26:50.307 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:50 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:50.073+0000 7f8a978dd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T10:26:50.590 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:50 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:50.336+0000 7f8a978dd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T10:26:50.590 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:50 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:50.589+0000 7f8a978dd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T10:26:50.979 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:50 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:50.978+0000 7f8a978dd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T10:26:51.302 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:51 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:51.301+0000 7f8a978dd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T10:26:52.057 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:51 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:51.616+0000 7f8a978dd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T10:26:53.057 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:52 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:52.689+0000 7f8a978dd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T10:26:53.808 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:53 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:53.410+0000 7f8a978dd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T10:26:53.808 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:53 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:53.544+0000 7f8a978dd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T10:26:54.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:53 smithi031 ceph-mon[31673]: Active manager daemon x restarted 2024-03-21T10:26:54.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:53 smithi031 ceph-mon[31673]: Activating manager daemon x 2024-03-21T10:26:54.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:53 smithi031 ceph-mon[31673]: osdmap e3: 0 total, 0 up, 0 in 2024-03-21T10:26:54.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:53 smithi031 ceph-mon[31673]: mgrmap e10: x(active, starting, since 0.00482746s) 2024-03-21T10:26:54.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:53 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T10:26:54.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:53 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T10:26:54.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:53 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T10:26:54.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:53 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T10:26:54.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:53 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T10:26:54.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:53 smithi031 ceph-mon[31673]: Manager daemon x is now available 2024-03-21T10:26:54.308 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:26:53 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:26:53.838+0000 7f8a978dd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T10:26:55.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:54 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:26:55.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:54 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T10:26:55.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:54 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T10:26:55.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:54 smithi031 ceph-mon[31673]: mgrmap e11: x(active, since 1.00987s) 2024-03-21T10:26:55.884 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-03-21T10:26:55.884 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-21T10:26:55.884 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-21T10:26:55.884 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-03-21T10:26:55.884 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.726+0000 7f0f57c00700 1 Processor -- start 2024-03-21T10:26:55.885 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.726+0000 7f0f57c00700 1 -- start start 2024-03-21T10:26:55.885 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.727+0000 7f0f57c00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f50107a80 0x7f0f50107ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:55.885 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.727+0000 7f0f57c00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f50108470 con 0x7f0f50107a80 2024-03-21T10:26:55.885 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.727+0000 7f0f55200700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f50107a80 0x7f0f50107ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:55.885 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.727+0000 7f0f55200700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f50107a80 0x7f0f50107ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59696/0 (socket says 172.21.15.31:59696) 2024-03-21T10:26:55.885 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.727+0000 7f0f55200700 1 -- 172.21.15.31:0/2287259037 learned_addr learned my addr 172.21.15.31:0/2287259037 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:55.885 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.728+0000 7f0f55200700 1 -- 172.21.15.31:0/2287259037 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f50108c80 con 0x7f0f50107a80 2024-03-21T10:26:55.885 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.728+0000 7f0f55200700 1 --2- 172.21.15.31:0/2287259037 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f50107a80 0x7f0f50107ea0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0f440098a0 tx=0x7f0f44009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d00c561ef3a627db server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:55.885 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.728+0000 7f0f4f400700 1 -- 172.21.15.31:0/2287259037 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0f44004160 con 0x7f0f50107a80 2024-03-21T10:26:55.885 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.728+0000 7f0f4f400700 1 -- 172.21.15.31:0/2287259037 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f44004300 con 0x7f0f50107a80 2024-03-21T10:26:55.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.728+0000 7f0f4f400700 1 -- 172.21.15.31:0/2287259037 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0f44011b20 con 0x7f0f50107a80 2024-03-21T10:26:55.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.729+0000 7f0f57c00700 1 -- 172.21.15.31:0/2287259037 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f50107a80 msgr2=0x7f0f50107ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:55.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.729+0000 7f0f57c00700 1 --2- 172.21.15.31:0/2287259037 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f50107a80 0x7f0f50107ea0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0f440098a0 tx=0x7f0f44009bb0 comp rx=0 tx=0).stop 2024-03-21T10:26:55.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.729+0000 7f0f57c00700 1 -- 172.21.15.31:0/2287259037 shutdown_connections 2024-03-21T10:26:55.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.729+0000 7f0f57c00700 1 --2- 172.21.15.31:0/2287259037 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f50107a80 0x7f0f50107ea0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:55.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.729+0000 7f0f57c00700 1 -- 172.21.15.31:0/2287259037 >> 172.21.15.31:0/2287259037 conn(0x7f0f50103080 msgr2=0x7f0f501054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:55.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.729+0000 7f0f57c00700 1 -- 172.21.15.31:0/2287259037 shutdown_connections 2024-03-21T10:26:55.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.729+0000 7f0f57c00700 1 -- 172.21.15.31:0/2287259037 wait complete. 2024-03-21T10:26:55.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.730+0000 7f0f57c00700 1 Processor -- start 2024-03-21T10:26:55.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.731+0000 7f0f57c00700 1 -- start start 2024-03-21T10:26:55.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.731+0000 7f0f57c00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f50107a80 0x7f0f5019b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:55.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.731+0000 7f0f57c00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5010c030 con 0x7f0f50107a80 2024-03-21T10:26:55.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.731+0000 7f0f55200700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f50107a80 0x7f0f5019b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:55.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.731+0000 7f0f55200700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f50107a80 0x7f0f5019b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59704/0 (socket says 172.21.15.31:59704) 2024-03-21T10:26:55.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.731+0000 7f0f55200700 1 -- 172.21.15.31:0/2806007727 learned_addr learned my addr 172.21.15.31:0/2806007727 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:55.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.732+0000 7f0f55200700 1 -- 172.21.15.31:0/2806007727 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5019bef0 con 0x7f0f50107a80 2024-03-21T10:26:55.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.732+0000 7f0f55200700 1 --2- 172.21.15.31:0/2806007727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f50107a80 0x7f0f5019b9b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0f440117e0 tx=0x7f0f44011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:55.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.732+0000 7f0f4d600700 1 -- 172.21.15.31:0/2806007727 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0f44003bf0 con 0x7f0f50107a80 2024-03-21T10:26:55.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.732+0000 7f0f4d600700 1 -- 172.21.15.31:0/2806007727 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f4401a910 con 0x7f0f50107a80 2024-03-21T10:26:55.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.732+0000 7f0f57c00700 1 -- 172.21.15.31:0/2806007727 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f5019c180 con 0x7f0f50107a80 2024-03-21T10:26:55.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.732+0000 7f0f4d600700 1 -- 172.21.15.31:0/2806007727 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0f44024970 con 0x7f0f50107a80 2024-03-21T10:26:55.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.732+0000 7f0f57c00700 1 -- 172.21.15.31:0/2806007727 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f5019c5e0 con 0x7f0f50107a80 2024-03-21T10:26:55.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.733+0000 7f0f4d600700 1 -- 172.21.15.31:0/2806007727 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) v1 ==== 46307+0+0 (secure 0 0 0) 0x7f0f44022020 con 0x7f0f50107a80 2024-03-21T10:26:55.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.734+0000 7f0f4d600700 1 --2- 172.21.15.31:0/2806007727 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f0f38039230 0x7f0f3803b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:55.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.734+0000 7f0f4d600700 1 -- 172.21.15.31:0/2806007727 --> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0f3803be00 con 0x7f0f38039230 2024-03-21T10:26:55.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.734+0000 7f0f4d600700 1 -- 172.21.15.31:0/2806007727 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0f4404d620 con 0x7f0f50107a80 2024-03-21T10:26:55.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.734+0000 7f0f4fe00700 1 -- 172.21.15.31:0/2806007727 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f0f38039230 msgr2=0x7f0f3803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/4085461895 2024-03-21T10:26:55.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.734+0000 7f0f4fe00700 1 --2- 172.21.15.31:0/2806007727 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f0f38039230 0x7f0f3803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-21T10:26:55.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.935+0000 7f0f4fe00700 1 -- 172.21.15.31:0/2806007727 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f0f38039230 msgr2=0x7f0f3803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/4085461895 2024-03-21T10:26:55.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:48.935+0000 7f0f4fe00700 1 --2- 172.21.15.31:0/2806007727 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f0f38039230 0x7f0f3803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-21T10:26:55.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:49.336+0000 7f0f4fe00700 1 -- 172.21.15.31:0/2806007727 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f0f38039230 msgr2=0x7f0f3803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/4085461895 2024-03-21T10:26:55.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:49.336+0000 7f0f4fe00700 1 --2- 172.21.15.31:0/2806007727 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f0f38039230 0x7f0f3803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-21T10:26:55.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:50.137+0000 7f0f4fe00700 1 -- 172.21.15.31:0/2806007727 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f0f38039230 msgr2=0x7f0f3803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/4085461895 2024-03-21T10:26:55.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:50.137+0000 7f0f4fe00700 1 --2- 172.21.15.31:0/2806007727 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f0f38039230 0x7f0f3803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-21T10:26:55.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:51.739+0000 7f0f4fe00700 1 -- 172.21.15.31:0/2806007727 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f0f38039230 msgr2=0x7f0f3803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/4085461895 2024-03-21T10:26:55.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:51.739+0000 7f0f4fe00700 1 --2- 172.21.15.31:0/2806007727 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f0f38039230 0x7f0f3803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-21T10:26:55.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:53.846+0000 7f0f4d600700 1 -- 172.21.15.31:0/2806007727 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mgrmap(e 10) v1 ==== 46074+0+0 (secure 0 0 0) 0x7f0f4402d3f0 con 0x7f0f50107a80 2024-03-21T10:26:55.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:53.847+0000 7f0f4d600700 1 -- 172.21.15.31:0/2806007727 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f0f38039230 msgr2=0x7f0f3803b6f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T10:26:55.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:53.847+0000 7f0f4d600700 1 --2- 172.21.15.31:0/2806007727 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f0f38039230 0x7f0f3803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:55.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:54.848+0000 7f0f4d600700 1 -- 172.21.15.31:0/2806007727 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 11) v1 ==== 46201+0+0 (secure 0 0 0) 0x7f0f4404f410 con 0x7f0f50107a80 2024-03-21T10:26:55.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:54.849+0000 7f0f4d600700 1 --2- 172.21.15.31:0/2806007727 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0f3803cce0 0x7f0f3803f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:55.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:54.849+0000 7f0f4d600700 1 -- 172.21.15.31:0/2806007727 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0f3803be00 con 0x7f0f3803cce0 2024-03-21T10:26:55.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:54.850+0000 7f0f4fe00700 1 --2- 172.21.15.31:0/2806007727 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0f3803cce0 0x7f0f3803f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:55.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:54.851+0000 7f0f4fe00700 1 --2- 172.21.15.31:0/2806007727 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0f3803cce0 0x7f0f3803f0d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c003a10 tx=0x7f0f3c009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:55.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:54.852+0000 7f0f4d600700 1 -- 172.21.15.31:0/2806007727 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f0f3803be00 con 0x7f0f3803cce0 2024-03-21T10:26:55.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:54.860+0000 7f0f57c00700 1 -- 172.21.15.31:0/2806007727 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f0f500637c0 con 0x7f0f3803cce0 2024-03-21T10:26:55.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:54.860+0000 7f0f4d600700 1 -- 172.21.15.31:0/2806007727 <== mgr.14152 v2:172.21.15.31:6800/3875791078 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f0f500637c0 con 0x7f0f3803cce0 2024-03-21T10:26:55.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:54.861+0000 7f0f57c00700 1 -- 172.21.15.31:0/2806007727 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0f3803cce0 msgr2=0x7f0f3803f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:55.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:54.861+0000 7f0f57c00700 1 --2- 172.21.15.31:0/2806007727 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0f3803cce0 0x7f0f3803f0d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c003a10 tx=0x7f0f3c009270 comp rx=0 tx=0).stop 2024-03-21T10:26:55.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:54.861+0000 7f0f57c00700 1 -- 172.21.15.31:0/2806007727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f50107a80 msgr2=0x7f0f5019b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:55.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:54.861+0000 7f0f57c00700 1 --2- 172.21.15.31:0/2806007727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f50107a80 0x7f0f5019b9b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0f440117e0 tx=0x7f0f44011ba0 comp rx=0 tx=0).stop 2024-03-21T10:26:55.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:54.861+0000 7f0f57c00700 1 -- 172.21.15.31:0/2806007727 shutdown_connections 2024-03-21T10:26:55.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:54.861+0000 7f0f57c00700 1 --2- 172.21.15.31:0/2806007727 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0f3803cce0 0x7f0f3803f0d0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:55.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:54.861+0000 7f0f57c00700 1 --2- 172.21.15.31:0/2806007727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f50107a80 0x7f0f5019b9b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:55.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:54.861+0000 7f0f57c00700 1 --2- 172.21.15.31:0/2806007727 >> [v2:172.21.15.31:6800/4085461895,v1:172.21.15.31:6801/4085461895] conn(0x7f0f38039230 0x7f0f3803b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:55.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:54.861+0000 7f0f57c00700 1 -- 172.21.15.31:0/2806007727 >> 172.21.15.31:0/2806007727 conn(0x7f0f50103080 msgr2=0x7f0f50103af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:55.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:54.861+0000 7f0f57c00700 1 -- 172.21.15.31:0/2806007727 shutdown_connections 2024-03-21T10:26:55.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:54.861+0000 7f0f57c00700 1 -- 172.21.15.31:0/2806007727 wait complete. 2024-03-21T10:26:55.896 INFO:teuthology.orchestra.run.smithi031.stdout:mgr epoch 9 is available 2024-03-21T10:26:55.896 INFO:teuthology.orchestra.run.smithi031.stdout:Generating a dashboard self-signed certificate... 2024-03-21T10:26:56.127 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:55 smithi031 ceph-mon[31673]: [21/Mar/2024:10:26:55] ENGINE Bus STARTING 2024-03-21T10:26:56.127 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:55 smithi031 ceph-mon[31673]: [21/Mar/2024:10:26:55] ENGINE Serving on https://172.21.15.31:7150 2024-03-21T10:26:56.127 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:55 smithi031 ceph-mon[31673]: [21/Mar/2024:10:26:55] ENGINE Serving on http://172.21.15.31:8765 2024-03-21T10:26:56.127 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:55 smithi031 ceph-mon[31673]: [21/Mar/2024:10:26:55] ENGINE Bus STARTED 2024-03-21T10:26:57.042 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:56 smithi031 ceph-mon[31673]: mgrmap e12: x(active, since 2s) 2024-03-21T10:26:57.043 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:56 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:26:58.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-21T10:26:58.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.097+0000 7f6836600700 1 Processor -- start 2024-03-21T10:26:58.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.098+0000 7f6836600700 1 -- start start 2024-03-21T10:26:58.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.098+0000 7f6836600700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f683007de60 0x7f683007c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:58.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.098+0000 7f6836600700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683007c890 con 0x7f683007de60 2024-03-21T10:26:58.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.099+0000 7f682f400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f683007de60 0x7f683007c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:58.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.099+0000 7f682f400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f683007de60 0x7f683007c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56482/0 (socket says 172.21.15.31:56482) 2024-03-21T10:26:58.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.099+0000 7f682f400700 1 -- 172.21.15.31:0/342570270 learned_addr learned my addr 172.21.15.31:0/342570270 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:58.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.099+0000 7f682f400700 1 -- 172.21.15.31:0/342570270 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f683007ca10 con 0x7f683007de60 2024-03-21T10:26:58.364 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.100+0000 7f682f400700 1 --2- 172.21.15.31:0/342570270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f683007de60 0x7f683007c2c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f68200098a0 tx=0x7f6820009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd9b283774717a58 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:58.364 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.100+0000 7f682e000700 1 -- 172.21.15.31:0/342570270 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6820004160 con 0x7f683007de60 2024-03-21T10:26:58.364 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.100+0000 7f682e000700 1 -- 172.21.15.31:0/342570270 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6820004300 con 0x7f683007de60 2024-03-21T10:26:58.364 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.100+0000 7f682e000700 1 -- 172.21.15.31:0/342570270 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6820011b20 con 0x7f683007de60 2024-03-21T10:26:58.364 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.100+0000 7f6836600700 1 -- 172.21.15.31:0/342570270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f683007de60 msgr2=0x7f683007c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:58.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.100+0000 7f6836600700 1 --2- 172.21.15.31:0/342570270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f683007de60 0x7f683007c2c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f68200098a0 tx=0x7f6820009bb0 comp rx=0 tx=0).stop 2024-03-21T10:26:58.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.101+0000 7f6836600700 1 -- 172.21.15.31:0/342570270 shutdown_connections 2024-03-21T10:26:58.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.101+0000 7f6836600700 1 --2- 172.21.15.31:0/342570270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f683007de60 0x7f683007c2c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:58.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.101+0000 7f6836600700 1 -- 172.21.15.31:0/342570270 >> 172.21.15.31:0/342570270 conn(0x7f6830103080 msgr2=0x7f68301054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:58.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.101+0000 7f6836600700 1 -- 172.21.15.31:0/342570270 shutdown_connections 2024-03-21T10:26:58.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.101+0000 7f6836600700 1 -- 172.21.15.31:0/342570270 wait complete. 2024-03-21T10:26:58.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.102+0000 7f6836600700 1 Processor -- start 2024-03-21T10:26:58.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.102+0000 7f6836600700 1 -- start start 2024-03-21T10:26:58.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.102+0000 7f6836600700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f683007de60 0x7f683019dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:58.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.102+0000 7f6836600700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683010e100 con 0x7f683007de60 2024-03-21T10:26:58.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.103+0000 7f682f400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f683007de60 0x7f683019dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:58.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.103+0000 7f682f400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f683007de60 0x7f683019dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56498/0 (socket says 172.21.15.31:56498) 2024-03-21T10:26:58.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.103+0000 7f682f400700 1 -- 172.21.15.31:0/1415868215 learned_addr learned my addr 172.21.15.31:0/1415868215 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:26:58.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.103+0000 7f682f400700 1 -- 172.21.15.31:0/1415868215 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f683019e0e0 con 0x7f683007de60 2024-03-21T10:26:58.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.103+0000 7f682f400700 1 --2- 172.21.15.31:0/1415868215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f683007de60 0x7f683019dba0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f68200117e0 tx=0x7f6820011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:58.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.104+0000 7f6827e00700 1 -- 172.21.15.31:0/1415868215 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6820003ba0 con 0x7f683007de60 2024-03-21T10:26:58.370 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.104+0000 7f6827e00700 1 -- 172.21.15.31:0/1415868215 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f682001a910 con 0x7f683007de60 2024-03-21T10:26:58.370 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.104+0000 7f6827e00700 1 -- 172.21.15.31:0/1415868215 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f68200248f0 con 0x7f683007de60 2024-03-21T10:26:58.371 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.104+0000 7f6836600700 1 -- 172.21.15.31:0/1415868215 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f683019e370 con 0x7f683007de60 2024-03-21T10:26:58.371 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.104+0000 7f6836600700 1 -- 172.21.15.31:0/1415868215 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f683019e7d0 con 0x7f683007de60 2024-03-21T10:26:58.371 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.105+0000 7f6827e00700 1 -- 172.21.15.31:0/1415868215 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 12) v1 ==== 46307+0+0 (secure 0 0 0) 0x7f6820022020 con 0x7f683007de60 2024-03-21T10:26:58.371 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.105+0000 7f6836600700 1 -- 172.21.15.31:0/1415868215 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68300637c0 con 0x7f683007de60 2024-03-21T10:26:58.371 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.105+0000 7f6827e00700 1 --2- 172.21.15.31:0/1415868215 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f6814039230 0x7f681403b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:26:58.371 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.106+0000 7f6827e00700 1 -- 172.21.15.31:0/1415868215 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f682004d7c0 con 0x7f683007de60 2024-03-21T10:26:58.371 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.106+0000 7f682ea00700 1 --2- 172.21.15.31:0/1415868215 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f6814039230 0x7f681403b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:26:58.371 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.107+0000 7f682ea00700 1 --2- 172.21.15.31:0/1415868215 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f6814039230 0x7f681403b6f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f681c006fd0 tx=0x7f681c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:26:58.371 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.111+0000 7f6827e00700 1 -- 172.21.15.31:0/1415868215 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6820017ca0 con 0x7f683007de60 2024-03-21T10:26:58.371 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.347+0000 7f6836600700 1 -- 172.21.15.31:0/1415868215 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f68301041d0 con 0x7f6814039230 2024-03-21T10:26:58.371 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.592+0000 7f6827e00700 1 -- 172.21.15.31:0/1415868215 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f68301041d0 con 0x7f6814039230 2024-03-21T10:26:58.371 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.595+0000 7f6836600700 1 -- 172.21.15.31:0/1415868215 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f6814039230 msgr2=0x7f681403b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:58.372 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.595+0000 7f6836600700 1 --2- 172.21.15.31:0/1415868215 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f6814039230 0x7f681403b6f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f681c006fd0 tx=0x7f681c006e40 comp rx=0 tx=0).stop 2024-03-21T10:26:58.372 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.596+0000 7f6836600700 1 -- 172.21.15.31:0/1415868215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f683007de60 msgr2=0x7f683019dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:26:58.372 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.596+0000 7f6836600700 1 --2- 172.21.15.31:0/1415868215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f683007de60 0x7f683019dba0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f68200117e0 tx=0x7f6820011ba0 comp rx=0 tx=0).stop 2024-03-21T10:26:58.372 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.596+0000 7f6836600700 1 -- 172.21.15.31:0/1415868215 shutdown_connections 2024-03-21T10:26:58.372 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.596+0000 7f6836600700 1 --2- 172.21.15.31:0/1415868215 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f6814039230 0x7f681403b6f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:58.372 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.596+0000 7f6836600700 1 --2- 172.21.15.31:0/1415868215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f683007de60 0x7f683019dba0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:26:58.372 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.596+0000 7f6836600700 1 -- 172.21.15.31:0/1415868215 >> 172.21.15.31:0/1415868215 conn(0x7f6830103080 msgr2=0x7f6830103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:26:58.372 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.596+0000 7f6836600700 1 -- 172.21.15.31:0/1415868215 shutdown_connections 2024-03-21T10:26:58.372 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:57.596+0000 7f6836600700 1 -- 172.21.15.31:0/1415868215 wait complete. 2024-03-21T10:26:58.372 INFO:teuthology.orchestra.run.smithi031.stdout:Creating initial admin user... 2024-03-21T10:26:58.614 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:58 smithi031 ceph-mon[31673]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:26:58.614 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:58 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:26:58.614 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:58 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:00.287 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:59 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:00.287 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:59 smithi031 ceph-mon[31673]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:27:00.287 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:26:59 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:00.305 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$niZ9dIWFlMWY7cr4k.gPCeOyDwHJKBApEsDEyzUeyz18PcN.TWo26", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711016819, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-21T10:27:00.305 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.057+0000 7fd505600700 1 Processor -- start 2024-03-21T10:27:00.305 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.057+0000 7fd505600700 1 -- start start 2024-03-21T10:27:00.305 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.058+0000 7fd505600700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5001084f0 0x7fd50010a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:00.305 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.058+0000 7fd505600700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd50007be90 con 0x7fd5001084f0 2024-03-21T10:27:00.305 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.058+0000 7fd4fea00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5001084f0 0x7fd50010a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:00.306 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.059+0000 7fd4fea00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5001084f0 0x7fd50010a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56514/0 (socket says 172.21.15.31:56514) 2024-03-21T10:27:00.306 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.059+0000 7fd4fea00700 1 -- 172.21.15.31:0/1121481355 learned_addr learned my addr 172.21.15.31:0/1121481355 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:27:00.306 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.059+0000 7fd4fea00700 1 -- 172.21.15.31:0/1121481355 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50010ae20 con 0x7fd5001084f0 2024-03-21T10:27:00.306 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.059+0000 7fd4fea00700 1 --2- 172.21.15.31:0/1121481355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5001084f0 0x7fd50010a8e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd4f00098a0 tx=0x7fd4f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18ac299b3b0ef469 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:00.306 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.060+0000 7fd4fd600700 1 -- 172.21.15.31:0/1121481355 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd4f0004160 con 0x7fd5001084f0 2024-03-21T10:27:00.306 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.060+0000 7fd4fd600700 1 -- 172.21.15.31:0/1121481355 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd4f0004300 con 0x7fd5001084f0 2024-03-21T10:27:00.306 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.060+0000 7fd4fd600700 1 -- 172.21.15.31:0/1121481355 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd4f0011b20 con 0x7fd5001084f0 2024-03-21T10:27:00.306 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.060+0000 7fd505600700 1 -- 172.21.15.31:0/1121481355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5001084f0 msgr2=0x7fd50010a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:00.306 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.060+0000 7fd505600700 1 --2- 172.21.15.31:0/1121481355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5001084f0 0x7fd50010a8e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd4f00098a0 tx=0x7fd4f0009bb0 comp rx=0 tx=0).stop 2024-03-21T10:27:00.306 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.060+0000 7fd505600700 1 -- 172.21.15.31:0/1121481355 shutdown_connections 2024-03-21T10:27:00.306 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.060+0000 7fd505600700 1 --2- 172.21.15.31:0/1121481355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5001084f0 0x7fd50010a8e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:00.307 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.060+0000 7fd505600700 1 -- 172.21.15.31:0/1121481355 >> 172.21.15.31:0/1121481355 conn(0x7fd500101f20 msgr2=0x7fd500104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:00.307 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.061+0000 7fd505600700 1 -- 172.21.15.31:0/1121481355 shutdown_connections 2024-03-21T10:27:00.307 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.061+0000 7fd505600700 1 -- 172.21.15.31:0/1121481355 wait complete. 2024-03-21T10:27:00.307 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.062+0000 7fd505600700 1 Processor -- start 2024-03-21T10:27:00.307 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.062+0000 7fd505600700 1 -- start start 2024-03-21T10:27:00.307 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.062+0000 7fd505600700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5001084f0 0x7fd50019b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:00.307 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.062+0000 7fd505600700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd50010be90 con 0x7fd5001084f0 2024-03-21T10:27:00.307 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.062+0000 7fd4fea00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5001084f0 0x7fd50019b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:00.307 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.062+0000 7fd4fea00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5001084f0 0x7fd50019b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56526/0 (socket says 172.21.15.31:56526) 2024-03-21T10:27:00.307 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.062+0000 7fd4fea00700 1 -- 172.21.15.31:0/148792270 learned_addr learned my addr 172.21.15.31:0/148792270 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:27:00.307 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.063+0000 7fd4fea00700 1 -- 172.21.15.31:0/148792270 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50019bec0 con 0x7fd5001084f0 2024-03-21T10:27:00.308 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.063+0000 7fd4fea00700 1 --2- 172.21.15.31:0/148792270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5001084f0 0x7fd50019b980 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd4f00117e0 tx=0x7fd4f0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:00.308 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.063+0000 7fd4f7400700 1 -- 172.21.15.31:0/148792270 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd4f0003ba0 con 0x7fd5001084f0 2024-03-21T10:27:00.308 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.063+0000 7fd4f7400700 1 -- 172.21.15.31:0/148792270 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd4f001a910 con 0x7fd5001084f0 2024-03-21T10:27:00.308 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.063+0000 7fd4f7400700 1 -- 172.21.15.31:0/148792270 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd4f00248f0 con 0x7fd5001084f0 2024-03-21T10:27:00.308 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.064+0000 7fd505600700 1 -- 172.21.15.31:0/148792270 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd50019c150 con 0x7fd5001084f0 2024-03-21T10:27:00.308 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.064+0000 7fd505600700 1 -- 172.21.15.31:0/148792270 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd50019c5b0 con 0x7fd5001084f0 2024-03-21T10:27:00.308 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.065+0000 7fd4f7400700 1 -- 172.21.15.31:0/148792270 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 12) v1 ==== 46307+0+0 (secure 0 0 0) 0x7fd4f0022020 con 0x7fd5001084f0 2024-03-21T10:27:00.308 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.065+0000 7fd505600700 1 -- 172.21.15.31:0/148792270 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5000637c0 con 0x7fd5001084f0 2024-03-21T10:27:00.308 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.066+0000 7fd4f7400700 1 --2- 172.21.15.31:0/148792270 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd4e4039230 0x7fd4e403b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:00.308 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.066+0000 7fd4f7400700 1 -- 172.21.15.31:0/148792270 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd4f004d6c0 con 0x7fd5001084f0 2024-03-21T10:27:00.308 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.066+0000 7fd4fe000700 1 --2- 172.21.15.31:0/148792270 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd4e4039230 0x7fd4e403b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:00.308 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.067+0000 7fd4fe000700 1 --2- 172.21.15.31:0/148792270 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd4e4039230 0x7fd4e403b6f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec006fd0 tx=0x7fd4ec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:00.309 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.071+0000 7fd4f7400700 1 -- 172.21.15.31:0/148792270 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd4f001ab70 con 0x7fd5001084f0 2024-03-21T10:27:00.309 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.322+0000 7fd505600700 1 -- 172.21.15.31:0/148792270 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fd50019efc0 con 0x7fd4e4039230 2024-03-21T10:27:00.309 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.575+0000 7fd4f7400700 1 -- 172.21.15.31:0/148792270 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fd50019efc0 con 0x7fd4e4039230 2024-03-21T10:27:00.309 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.578+0000 7fd505600700 1 -- 172.21.15.31:0/148792270 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd4e4039230 msgr2=0x7fd4e403b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:00.309 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.578+0000 7fd505600700 1 --2- 172.21.15.31:0/148792270 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd4e4039230 0x7fd4e403b6f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec006fd0 tx=0x7fd4ec006e40 comp rx=0 tx=0).stop 2024-03-21T10:27:00.309 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.578+0000 7fd505600700 1 -- 172.21.15.31:0/148792270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5001084f0 msgr2=0x7fd50019b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:00.309 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.578+0000 7fd505600700 1 --2- 172.21.15.31:0/148792270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5001084f0 0x7fd50019b980 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd4f00117e0 tx=0x7fd4f0011ba0 comp rx=0 tx=0).stop 2024-03-21T10:27:00.309 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.578+0000 7fd505600700 1 -- 172.21.15.31:0/148792270 shutdown_connections 2024-03-21T10:27:00.309 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.578+0000 7fd505600700 1 --2- 172.21.15.31:0/148792270 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd4e4039230 0x7fd4e403b6f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:00.309 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.578+0000 7fd505600700 1 --2- 172.21.15.31:0/148792270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5001084f0 0x7fd50019b980 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:00.309 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.578+0000 7fd505600700 1 -- 172.21.15.31:0/148792270 >> 172.21.15.31:0/148792270 conn(0x7fd500101f20 msgr2=0x7fd5001029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:00.309 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.578+0000 7fd505600700 1 -- 172.21.15.31:0/148792270 shutdown_connections 2024-03-21T10:27:00.310 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:26:59.578+0000 7fd505600700 1 -- 172.21.15.31:0/148792270 wait complete. 2024-03-21T10:27:00.310 INFO:teuthology.orchestra.run.smithi031.stdout:Fetching dashboard port number... 2024-03-21T10:27:01.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:01 smithi031 ceph-mon[31673]: mgrmap e13: x(active, since 6s) 2024-03-21T10:27:01.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:01 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1311026967' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-21T10:27:02.268 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 8443 2024-03-21T10:27:02.268 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.086+0000 7f5ab7200700 1 Processor -- start 2024-03-21T10:27:02.269 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.087+0000 7f5ab7200700 1 -- start start 2024-03-21T10:27:02.269 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.087+0000 7f5ab7200700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab0108540 0x7f5ab010a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:02.269 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.087+0000 7f5ab7200700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab007be30 con 0x7f5ab0108540 2024-03-21T10:27:02.269 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.087+0000 7f5aafe00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab0108540 0x7f5ab010a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:02.274 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.087+0000 7f5aafe00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab0108540 0x7f5ab010a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56540/0 (socket says 172.21.15.31:56540) 2024-03-21T10:27:02.274 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.087+0000 7f5aafe00700 1 -- 172.21.15.31:0/1525709909 learned_addr learned my addr 172.21.15.31:0/1525709909 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:27:02.274 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.088+0000 7f5aafe00700 1 -- 172.21.15.31:0/1525709909 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab010ae70 con 0x7f5ab0108540 2024-03-21T10:27:02.274 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.088+0000 7f5aafe00700 1 --2- 172.21.15.31:0/1525709909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab0108540 0x7f5ab010a930 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5aa00098a0 tx=0x7f5aa0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe7beac078f63451 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:02.274 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.088+0000 7f5aaea00700 1 -- 172.21.15.31:0/1525709909 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5aa0004160 con 0x7f5ab0108540 2024-03-21T10:27:02.274 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.089+0000 7f5aaea00700 1 -- 172.21.15.31:0/1525709909 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5aa0004300 con 0x7f5ab0108540 2024-03-21T10:27:02.274 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.089+0000 7f5aaea00700 1 -- 172.21.15.31:0/1525709909 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5aa0011b20 con 0x7f5ab0108540 2024-03-21T10:27:02.274 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.089+0000 7f5ab7200700 1 -- 172.21.15.31:0/1525709909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab0108540 msgr2=0x7f5ab010a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:02.275 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.089+0000 7f5ab7200700 1 --2- 172.21.15.31:0/1525709909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab0108540 0x7f5ab010a930 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5aa00098a0 tx=0x7f5aa0009bb0 comp rx=0 tx=0).stop 2024-03-21T10:27:02.275 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.089+0000 7f5ab7200700 1 -- 172.21.15.31:0/1525709909 shutdown_connections 2024-03-21T10:27:02.275 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.089+0000 7f5ab7200700 1 --2- 172.21.15.31:0/1525709909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab0108540 0x7f5ab010a930 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:02.275 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.089+0000 7f5ab7200700 1 -- 172.21.15.31:0/1525709909 >> 172.21.15.31:0/1525709909 conn(0x7f5ab0101ef0 msgr2=0x7f5ab0104350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:02.275 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.089+0000 7f5ab7200700 1 -- 172.21.15.31:0/1525709909 shutdown_connections 2024-03-21T10:27:02.275 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.089+0000 7f5ab7200700 1 -- 172.21.15.31:0/1525709909 wait complete. 2024-03-21T10:27:02.275 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.090+0000 7f5ab7200700 1 Processor -- start 2024-03-21T10:27:02.275 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.091+0000 7f5ab7200700 1 -- start start 2024-03-21T10:27:02.275 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.091+0000 7f5ab7200700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab0108540 0x7f5ab019b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:02.275 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.091+0000 7f5ab7200700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab010be80 con 0x7f5ab0108540 2024-03-21T10:27:02.275 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.091+0000 7f5aafe00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab0108540 0x7f5ab019b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:02.275 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.091+0000 7f5aafe00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab0108540 0x7f5ab019b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56548/0 (socket says 172.21.15.31:56548) 2024-03-21T10:27:02.275 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.091+0000 7f5aafe00700 1 -- 172.21.15.31:0/1311026967 learned_addr learned my addr 172.21.15.31:0/1311026967 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:27:02.276 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.091+0000 7f5aafe00700 1 -- 172.21.15.31:0/1311026967 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab019beb0 con 0x7f5ab0108540 2024-03-21T10:27:02.276 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.092+0000 7f5aafe00700 1 --2- 172.21.15.31:0/1311026967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab0108540 0x7f5ab019b970 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5aa00117e0 tx=0x7f5aa0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:02.276 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.092+0000 7f5aacc00700 1 -- 172.21.15.31:0/1311026967 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5aa0003ba0 con 0x7f5ab0108540 2024-03-21T10:27:02.276 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.092+0000 7f5aacc00700 1 -- 172.21.15.31:0/1311026967 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5aa001a910 con 0x7f5ab0108540 2024-03-21T10:27:02.276 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.092+0000 7f5aacc00700 1 -- 172.21.15.31:0/1311026967 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5aa00248f0 con 0x7f5ab0108540 2024-03-21T10:27:02.276 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.092+0000 7f5ab7200700 1 -- 172.21.15.31:0/1311026967 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab019c140 con 0x7f5ab0108540 2024-03-21T10:27:02.276 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.092+0000 7f5ab7200700 1 -- 172.21.15.31:0/1311026967 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab019c5a0 con 0x7f5ab0108540 2024-03-21T10:27:02.276 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.093+0000 7f5ab7200700 1 -- 172.21.15.31:0/1311026967 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab00637c0 con 0x7f5ab0108540 2024-03-21T10:27:02.276 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.094+0000 7f5aacc00700 1 -- 172.21.15.31:0/1311026967 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7f5aa0022020 con 0x7f5ab0108540 2024-03-21T10:27:02.276 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.094+0000 7f5aacc00700 1 --2- 172.21.15.31:0/1311026967 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5a940392d0 0x7f5a9403b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:02.276 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.094+0000 7f5aacc00700 1 -- 172.21.15.31:0/1311026967 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5aa004d850 con 0x7f5ab0108540 2024-03-21T10:27:02.276 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.094+0000 7f5aaf400700 1 --2- 172.21.15.31:0/1311026967 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5a940392d0 0x7f5a9403b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:02.276 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.095+0000 7f5aaf400700 1 --2- 172.21.15.31:0/1311026967 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5a940392d0 0x7f5a9403b790 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c006fd0 tx=0x7f5a9c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:02.276 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.101+0000 7f5aacc00700 1 -- 172.21.15.31:0/1311026967 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5aa001ab70 con 0x7f5ab0108540 2024-03-21T10:27:02.277 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.333+0000 7f5ab7200700 1 -- 172.21.15.31:0/1311026967 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f5ab019f030 con 0x7f5ab0108540 2024-03-21T10:27:02.277 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.334+0000 7f5aacc00700 1 -- 172.21.15.31:0/1311026967 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f5aa0010380 con 0x7f5ab0108540 2024-03-21T10:27:02.277 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.336+0000 7f5ab7200700 1 -- 172.21.15.31:0/1311026967 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5a940392d0 msgr2=0x7f5a9403b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:02.277 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.336+0000 7f5ab7200700 1 --2- 172.21.15.31:0/1311026967 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5a940392d0 0x7f5a9403b790 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c006fd0 tx=0x7f5a9c006e40 comp rx=0 tx=0).stop 2024-03-21T10:27:02.277 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.336+0000 7f5ab7200700 1 -- 172.21.15.31:0/1311026967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab0108540 msgr2=0x7f5ab019b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:02.277 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.336+0000 7f5ab7200700 1 --2- 172.21.15.31:0/1311026967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab0108540 0x7f5ab019b970 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5aa00117e0 tx=0x7f5aa0011ba0 comp rx=0 tx=0).stop 2024-03-21T10:27:02.277 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.337+0000 7f5ab7200700 1 -- 172.21.15.31:0/1311026967 shutdown_connections 2024-03-21T10:27:02.277 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.337+0000 7f5ab7200700 1 --2- 172.21.15.31:0/1311026967 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5a940392d0 0x7f5a9403b790 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:02.277 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.337+0000 7f5ab7200700 1 --2- 172.21.15.31:0/1311026967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab0108540 0x7f5ab019b970 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:02.277 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.337+0000 7f5ab7200700 1 -- 172.21.15.31:0/1311026967 >> 172.21.15.31:0/1311026967 conn(0x7f5ab0101ef0 msgr2=0x7f5ab0102970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:02.277 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.337+0000 7f5ab7200700 1 -- 172.21.15.31:0/1311026967 shutdown_connections 2024-03-21T10:27:02.277 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:01.337+0000 7f5ab7200700 1 -- 172.21.15.31:0/1311026967 wait complete. 2024-03-21T10:27:02.309 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-21T10:27:02.309 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2024-03-21T10:27:02.336 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-21T10:27:02.337 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-03-21T10:27:02.337 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2024-03-21T10:27:02.337 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-21T10:27:02.344 INFO:teuthology.orchestra.run.smithi031.stdout:Ceph Dashboard is now available at: 2024-03-21T10:27:02.344 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:27:02.344 INFO:teuthology.orchestra.run.smithi031.stdout: URL: https://smithi031.front.sepia.ceph.com:8443/ 2024-03-21T10:27:02.344 INFO:teuthology.orchestra.run.smithi031.stdout: User: admin 2024-03-21T10:27:02.344 INFO:teuthology.orchestra.run.smithi031.stdout: Password: q572lgz14f 2024-03-21T10:27:02.344 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:27:02.346 INFO:teuthology.orchestra.run.smithi031.stdout:Saving cluster configuration to /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config directory 2024-03-21T10:27:02.348 INFO:teuthology.orchestra.run.smithi031.stdout:Enabling autotune for osd_memory_target 2024-03-21T10:27:04.424 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.425+0000 7fe8ed400700 1 Processor -- start 2024-03-21T10:27:04.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.425+0000 7fe8ed400700 1 -- start start 2024-03-21T10:27:04.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.426+0000 7fe8ed400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8e81078d0 0x7fe8e8107cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:04.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.426+0000 7fe8ed400700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e8108230 con 0x7fe8e81078d0 2024-03-21T10:27:04.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.426+0000 7fe8e7e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8e81078d0 0x7fe8e8107cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:04.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.426+0000 7fe8e7e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8e81078d0 0x7fe8e8107cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56554/0 (socket says 172.21.15.31:56554) 2024-03-21T10:27:04.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.426+0000 7fe8e7e00700 1 -- 172.21.15.31:0/2152523785 learned_addr learned my addr 172.21.15.31:0/2152523785 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:27:04.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.427+0000 7fe8e7e00700 1 -- 172.21.15.31:0/2152523785 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e81083b0 con 0x7fe8e81078d0 2024-03-21T10:27:04.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.427+0000 7fe8e7e00700 1 --2- 172.21.15.31:0/2152523785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8e81078d0 0x7fe8e8107cf0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe8d80098a0 tx=0x7fe8d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ab67239f5330c98 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:04.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.427+0000 7fe8e6a00700 1 -- 172.21.15.31:0/2152523785 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe8d8004160 con 0x7fe8e81078d0 2024-03-21T10:27:04.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.427+0000 7fe8e6a00700 1 -- 172.21.15.31:0/2152523785 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe8d8004300 con 0x7fe8e81078d0 2024-03-21T10:27:04.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.427+0000 7fe8e6a00700 1 -- 172.21.15.31:0/2152523785 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe8d8011b20 con 0x7fe8e81078d0 2024-03-21T10:27:04.426 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.428+0000 7fe8ed400700 1 -- 172.21.15.31:0/2152523785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8e81078d0 msgr2=0x7fe8e8107cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:04.426 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.428+0000 7fe8ed400700 1 --2- 172.21.15.31:0/2152523785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8e81078d0 0x7fe8e8107cf0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe8d80098a0 tx=0x7fe8d8009bb0 comp rx=0 tx=0).stop 2024-03-21T10:27:04.426 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.428+0000 7fe8ed400700 1 -- 172.21.15.31:0/2152523785 shutdown_connections 2024-03-21T10:27:04.426 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.428+0000 7fe8ed400700 1 --2- 172.21.15.31:0/2152523785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8e81078d0 0x7fe8e8107cf0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:04.426 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.428+0000 7fe8ed400700 1 -- 172.21.15.31:0/2152523785 >> 172.21.15.31:0/2152523785 conn(0x7fe8e8102f50 msgr2=0x7fe8e8105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:04.426 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.428+0000 7fe8ed400700 1 -- 172.21.15.31:0/2152523785 shutdown_connections 2024-03-21T10:27:04.426 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.429+0000 7fe8ed400700 1 -- 172.21.15.31:0/2152523785 wait complete. 2024-03-21T10:27:04.426 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.430+0000 7fe8ed400700 1 Processor -- start 2024-03-21T10:27:04.426 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.430+0000 7fe8ed400700 1 -- start start 2024-03-21T10:27:04.426 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.430+0000 7fe8ed400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8e81078d0 0x7fe8e807db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:04.427 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.430+0000 7fe8ed400700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e810bd40 con 0x7fe8e81078d0 2024-03-21T10:27:04.427 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.431+0000 7fe8e7e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8e81078d0 0x7fe8e807db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:04.427 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.431+0000 7fe8e7e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8e81078d0 0x7fe8e807db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56556/0 (socket says 172.21.15.31:56556) 2024-03-21T10:27:04.427 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.431+0000 7fe8e7e00700 1 -- 172.21.15.31:0/4165462819 learned_addr learned my addr 172.21.15.31:0/4165462819 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:27:04.427 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.431+0000 7fe8e7e00700 1 -- 172.21.15.31:0/4165462819 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e807e090 con 0x7fe8e81078d0 2024-03-21T10:27:04.427 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.431+0000 7fe8e7e00700 1 --2- 172.21.15.31:0/4165462819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8e81078d0 0x7fe8e807db50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe8d80117e0 tx=0x7fe8d8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:04.427 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.432+0000 7fe8e4c00700 1 -- 172.21.15.31:0/4165462819 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe8d8003bf0 con 0x7fe8e81078d0 2024-03-21T10:27:04.427 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.432+0000 7fe8e4c00700 1 -- 172.21.15.31:0/4165462819 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe8d801a910 con 0x7fe8e81078d0 2024-03-21T10:27:04.427 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.432+0000 7fe8ed400700 1 -- 172.21.15.31:0/4165462819 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e807c280 con 0x7fe8e81078d0 2024-03-21T10:27:04.427 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.432+0000 7fe8e4c00700 1 -- 172.21.15.31:0/4165462819 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe8d8024970 con 0x7fe8e81078d0 2024-03-21T10:27:04.427 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.432+0000 7fe8ed400700 1 -- 172.21.15.31:0/4165462819 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e807c740 con 0x7fe8e81078d0 2024-03-21T10:27:04.427 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.433+0000 7fe8ed400700 1 -- 172.21.15.31:0/4165462819 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8e810bec0 con 0x7fe8e81078d0 2024-03-21T10:27:04.428 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.433+0000 7fe8e4c00700 1 -- 172.21.15.31:0/4165462819 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7fe8d8022020 con 0x7fe8e81078d0 2024-03-21T10:27:04.428 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.434+0000 7fe8e4c00700 1 --2- 172.21.15.31:0/4165462819 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe8cc039280 0x7fe8cc03b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:04.428 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.434+0000 7fe8e4c00700 1 -- 172.21.15.31:0/4165462819 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe8d804d790 con 0x7fe8e81078d0 2024-03-21T10:27:04.428 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.434+0000 7fe8e7400700 1 --2- 172.21.15.31:0/4165462819 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe8cc039280 0x7fe8cc03b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:04.428 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.435+0000 7fe8e7400700 1 --2- 172.21.15.31:0/4165462819 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe8cc039280 0x7fe8cc03b740 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4006fd0 tx=0x7fe8d4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:04.428 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.439+0000 7fe8e4c00700 1 -- 172.21.15.31:0/4165462819 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe8d8011e20 con 0x7fe8e81078d0 2024-03-21T10:27:04.428 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.668+0000 7fe8ed400700 1 -- 172.21.15.31:0/4165462819 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fe8e80637c0 con 0x7fe8e81078d0 2024-03-21T10:27:04.428 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.669+0000 7fe8e4c00700 1 -- 172.21.15.31:0/4165462819 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fe8d800b3e0 con 0x7fe8e81078d0 2024-03-21T10:27:04.428 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.672+0000 7fe8ed400700 1 -- 172.21.15.31:0/4165462819 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe8cc039280 msgr2=0x7fe8cc03b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:04.428 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.672+0000 7fe8ed400700 1 --2- 172.21.15.31:0/4165462819 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe8cc039280 0x7fe8cc03b740 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4006fd0 tx=0x7fe8d4006e40 comp rx=0 tx=0).stop 2024-03-21T10:27:04.428 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.672+0000 7fe8ed400700 1 -- 172.21.15.31:0/4165462819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8e81078d0 msgr2=0x7fe8e807db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:04.428 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.672+0000 7fe8ed400700 1 --2- 172.21.15.31:0/4165462819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8e81078d0 0x7fe8e807db50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe8d80117e0 tx=0x7fe8d8011c10 comp rx=0 tx=0).stop 2024-03-21T10:27:04.428 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.672+0000 7fe8ed400700 1 -- 172.21.15.31:0/4165462819 shutdown_connections 2024-03-21T10:27:04.428 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.672+0000 7fe8ed400700 1 --2- 172.21.15.31:0/4165462819 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe8cc039280 0x7fe8cc03b740 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:04.429 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.672+0000 7fe8ed400700 1 --2- 172.21.15.31:0/4165462819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8e81078d0 0x7fe8e807db50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:04.429 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.672+0000 7fe8ed400700 1 -- 172.21.15.31:0/4165462819 >> 172.21.15.31:0/4165462819 conn(0x7fe8e8102f50 msgr2=0x7fe8e81039a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:04.429 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.672+0000 7fe8ed400700 1 -- 172.21.15.31:0/4165462819 shutdown_connections 2024-03-21T10:27:04.429 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:03.672+0000 7fe8ed400700 1 -- 172.21.15.31:0/4165462819 wait complete. 2024-03-21T10:27:06.057 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.013+0000 7fd851400700 1 Processor -- start 2024-03-21T10:27:06.057 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.013+0000 7fd851400700 1 -- start start 2024-03-21T10:27:06.057 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.014+0000 7fd851400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd84c081aa0 0x7fd84c081ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:06.058 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.014+0000 7fd851400700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c082490 con 0x7fd84c081aa0 2024-03-21T10:27:06.059 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.014+0000 7fd84a800700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd84c081aa0 0x7fd84c081ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:06.059 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.014+0000 7fd84a800700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd84c081aa0 0x7fd84c081ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56572/0 (socket says 172.21.15.31:56572) 2024-03-21T10:27:06.059 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.014+0000 7fd84a800700 1 -- 172.21.15.31:0/3439164495 learned_addr learned my addr 172.21.15.31:0/3439164495 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:27:06.059 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.015+0000 7fd84a800700 1 -- 172.21.15.31:0/3439164495 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84c10f280 con 0x7fd84c081aa0 2024-03-21T10:27:06.059 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.015+0000 7fd84a800700 1 --2- 172.21.15.31:0/3439164495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd84c081aa0 0x7fd84c081ec0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd83c0098a0 tx=0x7fd83c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed57e18513d20509 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:06.059 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.015+0000 7fd849400700 1 -- 172.21.15.31:0/3439164495 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd83c004160 con 0x7fd84c081aa0 2024-03-21T10:27:06.059 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.015+0000 7fd849400700 1 -- 172.21.15.31:0/3439164495 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd83c004300 con 0x7fd84c081aa0 2024-03-21T10:27:06.060 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.015+0000 7fd849400700 1 -- 172.21.15.31:0/3439164495 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd83c011b20 con 0x7fd84c081aa0 2024-03-21T10:27:06.060 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.016+0000 7fd851400700 1 -- 172.21.15.31:0/3439164495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd84c081aa0 msgr2=0x7fd84c081ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:06.060 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.016+0000 7fd851400700 1 --2- 172.21.15.31:0/3439164495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd84c081aa0 0x7fd84c081ec0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd83c0098a0 tx=0x7fd83c009bb0 comp rx=0 tx=0).stop 2024-03-21T10:27:06.060 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.016+0000 7fd851400700 1 -- 172.21.15.31:0/3439164495 shutdown_connections 2024-03-21T10:27:06.062 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.016+0000 7fd851400700 1 --2- 172.21.15.31:0/3439164495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd84c081aa0 0x7fd84c081ec0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:06.062 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.016+0000 7fd851400700 1 -- 172.21.15.31:0/3439164495 >> 172.21.15.31:0/3439164495 conn(0x7fd84c07d040 msgr2=0x7fd84c07f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:06.062 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.016+0000 7fd851400700 1 -- 172.21.15.31:0/3439164495 shutdown_connections 2024-03-21T10:27:06.063 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.016+0000 7fd851400700 1 -- 172.21.15.31:0/3439164495 wait complete. 2024-03-21T10:27:06.063 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.017+0000 7fd851400700 1 Processor -- start 2024-03-21T10:27:06.063 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.017+0000 7fd851400700 1 -- start start 2024-03-21T10:27:06.063 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.018+0000 7fd851400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd84c081aa0 0x7fd84c1a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:06.063 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.018+0000 7fd851400700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c112630 con 0x7fd84c081aa0 2024-03-21T10:27:06.063 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.018+0000 7fd84a800700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd84c081aa0 0x7fd84c1a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:06.063 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.018+0000 7fd84a800700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd84c081aa0 0x7fd84c1a1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56574/0 (socket says 172.21.15.31:56574) 2024-03-21T10:27:06.063 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.018+0000 7fd84a800700 1 -- 172.21.15.31:0/1653864184 learned_addr learned my addr 172.21.15.31:0/1653864184 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:27:06.063 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.018+0000 7fd84a800700 1 -- 172.21.15.31:0/1653864184 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84c1a24c0 con 0x7fd84c081aa0 2024-03-21T10:27:06.064 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.019+0000 7fd84a800700 1 --2- 172.21.15.31:0/1653864184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd84c081aa0 0x7fd84c1a1f80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd83c0117e0 tx=0x7fd83c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:06.064 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.019+0000 7fd843400700 1 -- 172.21.15.31:0/1653864184 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd83c003bf0 con 0x7fd84c081aa0 2024-03-21T10:27:06.064 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.019+0000 7fd843400700 1 -- 172.21.15.31:0/1653864184 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd83c01a910 con 0x7fd84c081aa0 2024-03-21T10:27:06.064 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.019+0000 7fd851400700 1 -- 172.21.15.31:0/1653864184 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd84c1a2750 con 0x7fd84c081aa0 2024-03-21T10:27:06.064 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.019+0000 7fd843400700 1 -- 172.21.15.31:0/1653864184 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd83c0249a0 con 0x7fd84c081aa0 2024-03-21T10:27:06.064 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.019+0000 7fd851400700 1 -- 172.21.15.31:0/1653864184 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd84c1a2bb0 con 0x7fd84c081aa0 2024-03-21T10:27:06.064 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.020+0000 7fd851400700 1 -- 172.21.15.31:0/1653864184 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd84c0637c0 con 0x7fd84c081aa0 2024-03-21T10:27:06.064 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.020+0000 7fd843400700 1 -- 172.21.15.31:0/1653864184 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7fd83c022020 con 0x7fd84c081aa0 2024-03-21T10:27:06.064 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.021+0000 7fd843400700 1 --2- 172.21.15.31:0/1653864184 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd830039220 0x7fd83003b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:06.066 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.021+0000 7fd843400700 1 -- 172.21.15.31:0/1653864184 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd83c04d780 con 0x7fd84c081aa0 2024-03-21T10:27:06.066 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.021+0000 7fd849e00700 1 --2- 172.21.15.31:0/1653864184 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd830039220 0x7fd83003b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:06.067 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.022+0000 7fd849e00700 1 --2- 172.21.15.31:0/1653864184 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd830039220 0x7fd83003b6e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd838006fd0 tx=0x7fd838006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:06.067 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.026+0000 7fd843400700 1 -- 172.21.15.31:0/1653864184 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd83c011e20 con 0x7fd84c081aa0 2024-03-21T10:27:06.067 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.370+0000 7fd851400700 1 -- 172.21.15.31:0/1653864184 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fd84c1a5580 con 0x7fd84c081aa0 2024-03-21T10:27:06.067 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.375+0000 7fd843400700 1 -- 172.21.15.31:0/1653864184 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7fd83c02a2a0 con 0x7fd84c081aa0 2024-03-21T10:27:06.067 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.378+0000 7fd851400700 1 -- 172.21.15.31:0/1653864184 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd830039220 msgr2=0x7fd83003b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:06.067 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.378+0000 7fd851400700 1 --2- 172.21.15.31:0/1653864184 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd830039220 0x7fd83003b6e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd838006fd0 tx=0x7fd838006e40 comp rx=0 tx=0).stop 2024-03-21T10:27:06.067 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.378+0000 7fd851400700 1 -- 172.21.15.31:0/1653864184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd84c081aa0 msgr2=0x7fd84c1a1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:06.067 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.378+0000 7fd851400700 1 --2- 172.21.15.31:0/1653864184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd84c081aa0 0x7fd84c1a1f80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd83c0117e0 tx=0x7fd83c011c10 comp rx=0 tx=0).stop 2024-03-21T10:27:06.067 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.378+0000 7fd851400700 1 -- 172.21.15.31:0/1653864184 shutdown_connections 2024-03-21T10:27:06.067 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.378+0000 7fd851400700 1 --2- 172.21.15.31:0/1653864184 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd830039220 0x7fd83003b6e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:06.067 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.378+0000 7fd851400700 1 --2- 172.21.15.31:0/1653864184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd84c081aa0 0x7fd84c1a1f80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:06.067 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.378+0000 7fd851400700 1 -- 172.21.15.31:0/1653864184 >> 172.21.15.31:0/1653864184 conn(0x7fd84c07d040 msgr2=0x7fd84c07da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:06.067 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.379+0000 7fd851400700 1 -- 172.21.15.31:0/1653864184 shutdown_connections 2024-03-21T10:27:06.068 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-03-21T10:27:05.379+0000 7fd851400700 1 -- 172.21.15.31:0/1653864184 wait complete. 2024-03-21T10:27:06.068 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-21T10:27:06.068 INFO:teuthology.orchestra.run.smithi031.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-21T10:27:06.068 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:27:06.068 INFO:teuthology.orchestra.run.smithi031.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-21T10:27:06.068 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:27:06.068 INFO:teuthology.orchestra.run.smithi031.stdout:Or, if you are only running a single cluster on this host: 2024-03-21T10:27:06.068 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:27:06.068 INFO:teuthology.orchestra.run.smithi031.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-21T10:27:06.068 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:27:06.068 INFO:teuthology.orchestra.run.smithi031.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-21T10:27:06.068 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:27:06.068 INFO:teuthology.orchestra.run.smithi031.stdout: ceph telemetry on 2024-03-21T10:27:06.068 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:27:06.068 INFO:teuthology.orchestra.run.smithi031.stdout:For more information see: 2024-03-21T10:27:06.069 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:27:06.069 INFO:teuthology.orchestra.run.smithi031.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-21T10:27:06.069 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:27:06.069 INFO:teuthology.orchestra.run.smithi031.stdout:Bootstrap complete. 2024-03-21T10:27:06.162 INFO:tasks.cephadm:Fetching config... 2024-03-21T10:27:06.163 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-21T10:27:06.163 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-21T10:27:06.255 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-21T10:27:06.256 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-21T10:27:06.256 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-21T10:27:06.298 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-21T10:27:06.298 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-21T10:27:06.298 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-21T10:27:06.427 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-21T10:27:06.428 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-21T10:27:06.428 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-21T10:27:06.528 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-21T10:27:06.529 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJUNRYCYm3PhJDatpPDA7cWcOZcWZ1xbEEwhquawL3S/bRslYoumXGK2rL3KYlVlm2alPycHnDRuLuWwGQSOYww3sFAEZePWnT/Rrda7g5qomR6EYrQkALm2ptVbG+mlVvEdCRj96FNKO2MyXIlW+XZIqUAOiJ9KX1MLkhSyU8JQW1no1/3fU107uzcCKB9Q0ENBdKi7DdOW0rO7/dztXwcghKEbZ8p9e5QDe33vJETsj2QB4/g3srcxoIe2pa9e3SNCzS5mKV6/JU1uQng77+WT9KYjxT8dkfOLlcMNouyDhHnFy8iBQX++i8eMDEJVLkykHWy74FTLnJ8CDB3G+iPwA7c2G01A4Aiz14GllLOfczBYdyEpiR1G6xNSKiXbvInjmVx4iAn/uHh+9OhNOJfh2V6ij/hKHpWA7I4tPpiUmymOSK2SvaYPfs3HvYYPIgwc+RzaZ+8S0eaDcn7b/6AphshTcuolkfH2fKZoiQGCVkpLQON+QDM5nokKkRsuk= ceph-3847408c-e76d-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-21T10:27:06.624 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:06 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1653864184' entity='client.admin' 2024-03-21T10:27:06.665 INFO:teuthology.orchestra.run.smithi031.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJUNRYCYm3PhJDatpPDA7cWcOZcWZ1xbEEwhquawL3S/bRslYoumXGK2rL3KYlVlm2alPycHnDRuLuWwGQSOYww3sFAEZePWnT/Rrda7g5qomR6EYrQkALm2ptVbG+mlVvEdCRj96FNKO2MyXIlW+XZIqUAOiJ9KX1MLkhSyU8JQW1no1/3fU107uzcCKB9Q0ENBdKi7DdOW0rO7/dztXwcghKEbZ8p9e5QDe33vJETsj2QB4/g3srcxoIe2pa9e3SNCzS5mKV6/JU1uQng77+WT9KYjxT8dkfOLlcMNouyDhHnFy8iBQX++i8eMDEJVLkykHWy74FTLnJ8CDB3G+iPwA7c2G01A4Aiz14GllLOfczBYdyEpiR1G6xNSKiXbvInjmVx4iAn/uHh+9OhNOJfh2V6ij/hKHpWA7I4tPpiUmymOSK2SvaYPfs3HvYYPIgwc+RzaZ+8S0eaDcn7b/6AphshTcuolkfH2fKZoiQGCVkpLQON+QDM5nokKkRsuk= ceph-3847408c-e76d-11ee-95cd-87774f69a715 2024-03-21T10:27:06.719 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJUNRYCYm3PhJDatpPDA7cWcOZcWZ1xbEEwhquawL3S/bRslYoumXGK2rL3KYlVlm2alPycHnDRuLuWwGQSOYww3sFAEZePWnT/Rrda7g5qomR6EYrQkALm2ptVbG+mlVvEdCRj96FNKO2MyXIlW+XZIqUAOiJ9KX1MLkhSyU8JQW1no1/3fU107uzcCKB9Q0ENBdKi7DdOW0rO7/dztXwcghKEbZ8p9e5QDe33vJETsj2QB4/g3srcxoIe2pa9e3SNCzS5mKV6/JU1uQng77+WT9KYjxT8dkfOLlcMNouyDhHnFy8iBQX++i8eMDEJVLkykHWy74FTLnJ8CDB3G+iPwA7c2G01A4Aiz14GllLOfczBYdyEpiR1G6xNSKiXbvInjmVx4iAn/uHh+9OhNOJfh2V6ij/hKHpWA7I4tPpiUmymOSK2SvaYPfs3HvYYPIgwc+RzaZ+8S0eaDcn7b/6AphshTcuolkfH2fKZoiQGCVkpLQON+QDM5nokKkRsuk= ceph-3847408c-e76d-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-21T10:27:06.774 INFO:teuthology.orchestra.run.smithi040.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJUNRYCYm3PhJDatpPDA7cWcOZcWZ1xbEEwhquawL3S/bRslYoumXGK2rL3KYlVlm2alPycHnDRuLuWwGQSOYww3sFAEZePWnT/Rrda7g5qomR6EYrQkALm2ptVbG+mlVvEdCRj96FNKO2MyXIlW+XZIqUAOiJ9KX1MLkhSyU8JQW1no1/3fU107uzcCKB9Q0ENBdKi7DdOW0rO7/dztXwcghKEbZ8p9e5QDe33vJETsj2QB4/g3srcxoIe2pa9e3SNCzS5mKV6/JU1uQng77+WT9KYjxT8dkfOLlcMNouyDhHnFy8iBQX++i8eMDEJVLkykHWy74FTLnJ8CDB3G+iPwA7c2G01A4Aiz14GllLOfczBYdyEpiR1G6xNSKiXbvInjmVx4iAn/uHh+9OhNOJfh2V6ij/hKHpWA7I4tPpiUmymOSK2SvaYPfs3HvYYPIgwc+RzaZ+8S0eaDcn7b/6AphshTcuolkfH2fKZoiQGCVkpLQON+QDM5nokKkRsuk= ceph-3847408c-e76d-11ee-95cd-87774f69a715 2024-03-21T10:27:06.793 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJUNRYCYm3PhJDatpPDA7cWcOZcWZ1xbEEwhquawL3S/bRslYoumXGK2rL3KYlVlm2alPycHnDRuLuWwGQSOYww3sFAEZePWnT/Rrda7g5qomR6EYrQkALm2ptVbG+mlVvEdCRj96FNKO2MyXIlW+XZIqUAOiJ9KX1MLkhSyU8JQW1no1/3fU107uzcCKB9Q0ENBdKi7DdOW0rO7/dztXwcghKEbZ8p9e5QDe33vJETsj2QB4/g3srcxoIe2pa9e3SNCzS5mKV6/JU1uQng77+WT9KYjxT8dkfOLlcMNouyDhHnFy8iBQX++i8eMDEJVLkykHWy74FTLnJ8CDB3G+iPwA7c2G01A4Aiz14GllLOfczBYdyEpiR1G6xNSKiXbvInjmVx4iAn/uHh+9OhNOJfh2V6ij/hKHpWA7I4tPpiUmymOSK2SvaYPfs3HvYYPIgwc+RzaZ+8S0eaDcn7b/6AphshTcuolkfH2fKZoiQGCVkpLQON+QDM5nokKkRsuk= ceph-3847408c-e76d-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-21T10:27:06.841 INFO:teuthology.orchestra.run.smithi070.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJUNRYCYm3PhJDatpPDA7cWcOZcWZ1xbEEwhquawL3S/bRslYoumXGK2rL3KYlVlm2alPycHnDRuLuWwGQSOYww3sFAEZePWnT/Rrda7g5qomR6EYrQkALm2ptVbG+mlVvEdCRj96FNKO2MyXIlW+XZIqUAOiJ9KX1MLkhSyU8JQW1no1/3fU107uzcCKB9Q0ENBdKi7DdOW0rO7/dztXwcghKEbZ8p9e5QDe33vJETsj2QB4/g3srcxoIe2pa9e3SNCzS5mKV6/JU1uQng77+WT9KYjxT8dkfOLlcMNouyDhHnFy8iBQX++i8eMDEJVLkykHWy74FTLnJ8CDB3G+iPwA7c2G01A4Aiz14GllLOfczBYdyEpiR1G6xNSKiXbvInjmVx4iAn/uHh+9OhNOJfh2V6ij/hKHpWA7I4tPpiUmymOSK2SvaYPfs3HvYYPIgwc+RzaZ+8S0eaDcn7b/6AphshTcuolkfH2fKZoiQGCVkpLQON+QDM5nokKkRsuk= ceph-3847408c-e76d-11ee-95cd-87774f69a715 2024-03-21T10:27:06.856 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-21T10:27:07.290 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:27:10.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.265+0000 7f433aa00700 1 -- 172.21.15.31:0/4151384285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4334102650 msgr2=0x7f4334104a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:10.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.265+0000 7f433aa00700 1 --2- 172.21.15.31:0/4151384285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4334102650 0x7f4334104a40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4324009910 tx=0x7f4324009c20 comp rx=0 tx=0).stop 2024-03-21T10:27:10.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.265+0000 7f433aa00700 1 -- 172.21.15.31:0/4151384285 shutdown_connections 2024-03-21T10:27:10.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.265+0000 7f433aa00700 1 --2- 172.21.15.31:0/4151384285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4334102650 0x7f4334104a40 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:10.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.265+0000 7f433aa00700 1 -- 172.21.15.31:0/4151384285 >> 172.21.15.31:0/4151384285 conn(0x7f43340fc060 msgr2=0x7f43340fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:10.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.265+0000 7f433aa00700 1 -- 172.21.15.31:0/4151384285 shutdown_connections 2024-03-21T10:27:10.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.265+0000 7f433aa00700 1 -- 172.21.15.31:0/4151384285 wait complete. 2024-03-21T10:27:10.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.266+0000 7f433aa00700 1 Processor -- start 2024-03-21T10:27:10.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.266+0000 7f433aa00700 1 -- start start 2024-03-21T10:27:10.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.267+0000 7f433aa00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4334102650 0x7f4334199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:10.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.267+0000 7f433aa00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4334107bf0 con 0x7f4334102650 2024-03-21T10:27:10.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.267+0000 7f4333e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4334102650 0x7f4334199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:10.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.267+0000 7f4333e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4334102650 0x7f4334199460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52282/0 (socket says 172.21.15.31:52282) 2024-03-21T10:27:10.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.267+0000 7f4333e00700 1 -- 172.21.15.31:0/2577842291 learned_addr learned my addr 172.21.15.31:0/2577842291 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:27:10.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.268+0000 7f4333e00700 1 -- 172.21.15.31:0/2577842291 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43341999a0 con 0x7f4334102650 2024-03-21T10:27:10.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.268+0000 7f4333e00700 1 --2- 172.21.15.31:0/2577842291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4334102650 0x7f4334199460 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4324005950 tx=0x7f4324005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:10.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.268+0000 7f4331600700 1 -- 172.21.15.31:0/2577842291 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4324026070 con 0x7f4334102650 2024-03-21T10:27:10.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.268+0000 7f4331600700 1 -- 172.21.15.31:0/2577842291 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f432400bbd0 con 0x7f4334102650 2024-03-21T10:27:10.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.268+0000 7f433aa00700 1 -- 172.21.15.31:0/2577842291 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4334199c90 con 0x7f4334102650 2024-03-21T10:27:10.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.269+0000 7f4331600700 1 -- 172.21.15.31:0/2577842291 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4324003ad0 con 0x7f4334102650 2024-03-21T10:27:10.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.269+0000 7f433aa00700 1 -- 172.21.15.31:0/2577842291 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f433419c920 con 0x7f4334102650 2024-03-21T10:27:10.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.270+0000 7f4331600700 1 -- 172.21.15.31:0/2577842291 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7f4324029070 con 0x7f4334102650 2024-03-21T10:27:10.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.270+0000 7f433aa00700 1 -- 172.21.15.31:0/2577842291 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4334108370 con 0x7f4334102650 2024-03-21T10:27:10.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.270+0000 7f4331600700 1 --2- 172.21.15.31:0/2577842291 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4318039280 0x7f431803b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:10.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.270+0000 7f4331600700 1 -- 172.21.15.31:0/2577842291 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f432401e070 con 0x7f4334102650 2024-03-21T10:27:10.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.271+0000 7f432be00700 1 --2- 172.21.15.31:0/2577842291 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4318039280 0x7f431803b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:10.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.271+0000 7f432be00700 1 --2- 172.21.15.31:0/2577842291 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4318039280 0x7f431803b740 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f431c006fd0 tx=0x7f431c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:10.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.276+0000 7f4331600700 1 -- 172.21.15.31:0/2577842291 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4324013070 con 0x7f4334102650 2024-03-21T10:27:10.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.503+0000 7f433aa00700 1 -- 172.21.15.31:0/2577842291 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f433402dc20 con 0x7f4334102650 2024-03-21T10:27:10.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.506+0000 7f4331600700 1 -- 172.21.15.31:0/2577842291 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f4324030020 con 0x7f4334102650 2024-03-21T10:27:10.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.510+0000 7f433aa00700 1 -- 172.21.15.31:0/2577842291 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4318039280 msgr2=0x7f431803b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:10.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.510+0000 7f433aa00700 1 --2- 172.21.15.31:0/2577842291 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4318039280 0x7f431803b740 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f431c006fd0 tx=0x7f431c006e40 comp rx=0 tx=0).stop 2024-03-21T10:27:10.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.510+0000 7f433aa00700 1 -- 172.21.15.31:0/2577842291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4334102650 msgr2=0x7f4334199460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:10.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.510+0000 7f433aa00700 1 --2- 172.21.15.31:0/2577842291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4334102650 0x7f4334199460 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4324005950 tx=0x7f4324005030 comp rx=0 tx=0).stop 2024-03-21T10:27:10.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.510+0000 7f433aa00700 1 -- 172.21.15.31:0/2577842291 shutdown_connections 2024-03-21T10:27:10.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.510+0000 7f433aa00700 1 --2- 172.21.15.31:0/2577842291 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4318039280 0x7f431803b740 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:10.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.510+0000 7f433aa00700 1 --2- 172.21.15.31:0/2577842291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4334102650 0x7f4334199460 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:10.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.510+0000 7f433aa00700 1 -- 172.21.15.31:0/2577842291 >> 172.21.15.31:0/2577842291 conn(0x7f43340fc060 msgr2=0x7f43340fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:10.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.510+0000 7f433aa00700 1 -- 172.21.15.31:0/2577842291 shutdown_connections 2024-03-21T10:27:10.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:10.510+0000 7f433aa00700 1 -- 172.21.15.31:0/2577842291 wait complete. 2024-03-21T10:27:11.354 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-21T10:27:11.354 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-21T10:27:11.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:11 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2577842291' entity='client.admin' 2024-03-21T10:27:11.801 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:27:14.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:14 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:14.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:14 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:14.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:14 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi031", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:27:14.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:14 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:14.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:14 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:27:14.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:14 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:14.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.844+0000 7f0ad4e00700 1 -- 172.21.15.31:0/2539559421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ad0103c40 msgr2=0x7f0ad0104060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:14.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.844+0000 7f0ad4e00700 1 --2- 172.21.15.31:0/2539559421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ad0103c40 0x7f0ad0104060 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0009910 tx=0x7f0ac0009c20 comp rx=0 tx=0).stop 2024-03-21T10:27:14.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.844+0000 7f0ad4e00700 1 -- 172.21.15.31:0/2539559421 shutdown_connections 2024-03-21T10:27:14.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.844+0000 7f0ad4e00700 1 --2- 172.21.15.31:0/2539559421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ad0103c40 0x7f0ad0104060 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:14.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.844+0000 7f0ad4e00700 1 -- 172.21.15.31:0/2539559421 >> 172.21.15.31:0/2539559421 conn(0x7f0ad00ff1c0 msgr2=0x7f0ad0101620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:14.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.844+0000 7f0ad4e00700 1 -- 172.21.15.31:0/2539559421 shutdown_connections 2024-03-21T10:27:14.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.844+0000 7f0ad4e00700 1 -- 172.21.15.31:0/2539559421 wait complete. 2024-03-21T10:27:14.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.846+0000 7f0ad4e00700 1 Processor -- start 2024-03-21T10:27:14.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.846+0000 7f0ad4e00700 1 -- start start 2024-03-21T10:27:14.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.846+0000 7f0ad4e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ad0103c40 0x7f0ad0079e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:14.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.847+0000 7f0ad4e00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad0109cd0 con 0x7f0ad0103c40 2024-03-21T10:27:14.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.847+0000 7f0acf400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ad0103c40 0x7f0ad0079e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:14.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.847+0000 7f0acf400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ad0103c40 0x7f0ad0079e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52308/0 (socket says 172.21.15.31:52308) 2024-03-21T10:27:14.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.847+0000 7f0acf400700 1 -- 172.21.15.31:0/2633842501 learned_addr learned my addr 172.21.15.31:0/2633842501 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:27:14.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.847+0000 7f0acf400700 1 -- 172.21.15.31:0/2633842501 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad007a3a0 con 0x7f0ad0103c40 2024-03-21T10:27:14.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.848+0000 7f0acf400700 1 --2- 172.21.15.31:0/2633842501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ad0103c40 0x7f0ad0079e60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0005850 tx=0x7f0ac0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:14.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.848+0000 7f0ac7e00700 1 -- 172.21.15.31:0/2633842501 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ac0026070 con 0x7f0ad0103c40 2024-03-21T10:27:14.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.848+0000 7f0ac7e00700 1 -- 172.21.15.31:0/2633842501 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ac00044a0 con 0x7f0ad0103c40 2024-03-21T10:27:14.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.848+0000 7f0ad4e00700 1 -- 172.21.15.31:0/2633842501 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad007a630 con 0x7f0ad0103c40 2024-03-21T10:27:14.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.848+0000 7f0ac7e00700 1 -- 172.21.15.31:0/2633842501 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ac0003ab0 con 0x7f0ad0103c40 2024-03-21T10:27:14.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.848+0000 7f0ad4e00700 1 -- 172.21.15.31:0/2633842501 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad0076bf0 con 0x7f0ad0103c40 2024-03-21T10:27:14.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.849+0000 7f0ad4e00700 1 -- 172.21.15.31:0/2633842501 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad007a800 con 0x7f0ad0103c40 2024-03-21T10:27:14.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.849+0000 7f0ac7e00700 1 -- 172.21.15.31:0/2633842501 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7f0ac0029070 con 0x7f0ad0103c40 2024-03-21T10:27:14.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.850+0000 7f0ac7e00700 1 --2- 172.21.15.31:0/2633842501 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0ab4039220 0x7f0ab403b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:14.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.850+0000 7f0ac7e00700 1 -- 172.21.15.31:0/2633842501 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0ac001e070 con 0x7f0ad0103c40 2024-03-21T10:27:14.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.850+0000 7f0acea00700 1 --2- 172.21.15.31:0/2633842501 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0ab4039220 0x7f0ab403b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:14.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.851+0000 7f0acea00700 1 --2- 172.21.15.31:0/2633842501 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0ab4039220 0x7f0ab403b6e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0abc006fd0 tx=0x7f0abc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:14.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:14.855+0000 7f0ac7e00700 1 -- 172.21.15.31:0/2633842501 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0ad007a800 con 0x7f0ad0103c40 2024-03-21T10:27:15.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:15.086+0000 7f0ad4e00700 1 -- 172.21.15.31:0/2633842501 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ad0192610 con 0x7f0ab4039220 2024-03-21T10:27:15.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:15.094+0000 7f0ac7e00700 1 -- 172.21.15.31:0/2633842501 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0ad0192610 con 0x7f0ab4039220 2024-03-21T10:27:15.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:15.099+0000 7f0ad4e00700 1 -- 172.21.15.31:0/2633842501 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0ab4039220 msgr2=0x7f0ab403b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:15.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:15.099+0000 7f0ad4e00700 1 --2- 172.21.15.31:0/2633842501 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0ab4039220 0x7f0ab403b6e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0abc006fd0 tx=0x7f0abc006e40 comp rx=0 tx=0).stop 2024-03-21T10:27:15.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:15.099+0000 7f0ad4e00700 1 -- 172.21.15.31:0/2633842501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ad0103c40 msgr2=0x7f0ad0079e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:15.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:15.099+0000 7f0ad4e00700 1 --2- 172.21.15.31:0/2633842501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ad0103c40 0x7f0ad0079e60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0005850 tx=0x7f0ac0005030 comp rx=0 tx=0).stop 2024-03-21T10:27:15.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:15.099+0000 7f0ad4e00700 1 -- 172.21.15.31:0/2633842501 shutdown_connections 2024-03-21T10:27:15.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:15.099+0000 7f0ad4e00700 1 --2- 172.21.15.31:0/2633842501 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0ab4039220 0x7f0ab403b6e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:15.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:15.099+0000 7f0ad4e00700 1 --2- 172.21.15.31:0/2633842501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ad0103c40 0x7f0ad0079e60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:15.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:15.099+0000 7f0ad4e00700 1 -- 172.21.15.31:0/2633842501 >> 172.21.15.31:0/2633842501 conn(0x7f0ad00ff1c0 msgr2=0x7f0ad00ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:15.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:15.099+0000 7f0ad4e00700 1 -- 172.21.15.31:0/2633842501 shutdown_connections 2024-03-21T10:27:15.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:15.099+0000 7f0ad4e00700 1 -- 172.21.15.31:0/2633842501 wait complete. 2024-03-21T10:27:15.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:15 smithi031 ceph-mon[31673]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:27:15.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:15 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:15.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:15 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:27:15.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:15 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:27:15.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:15 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:27:16.036 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi040 2024-03-21T10:27:16.036 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-21T10:27:16.036 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/etc/ceph/ceph.conf 2024-03-21T10:27:16.055 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-21T10:27:16.055 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-21T10:27:16.113 INFO:tasks.cephadm:Adding host smithi040 to orchestrator... 2024-03-21T10:27:16.113 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph orch host add smithi040 2024-03-21T10:27:16.552 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:27:17.047 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:16 smithi031 ceph-mon[31673]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:27:17.047 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:16 smithi031 ceph-mon[31673]: Updating smithi031:/etc/ceph/ceph.conf 2024-03-21T10:27:17.048 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:16 smithi031 ceph-mon[31673]: Updating smithi031:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T10:27:17.773 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:17 smithi031 ceph-mon[31673]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:27:17.773 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:17 smithi031 ceph-mon[31673]: Updating smithi031:/etc/ceph/ceph.client.admin.keyring 2024-03-21T10:27:18.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.621+0000 7fb08e600700 1 -- 172.21.15.31:0/3922295332 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0880745c0 msgr2=0x7fb0880749a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:18.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.621+0000 7fb08e600700 1 --2- 172.21.15.31:0/3922295332 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0880745c0 0x7fb0880749a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb078009910 tx=0x7fb078009c20 comp rx=0 tx=0).stop 2024-03-21T10:27:18.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.621+0000 7fb08e600700 1 -- 172.21.15.31:0/3922295332 shutdown_connections 2024-03-21T10:27:18.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.621+0000 7fb08e600700 1 --2- 172.21.15.31:0/3922295332 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0880745c0 0x7fb0880749a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:18.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.621+0000 7fb08e600700 1 -- 172.21.15.31:0/3922295332 >> 172.21.15.31:0/3922295332 conn(0x7fb0880fd6a0 msgr2=0x7fb0880ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:18.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.621+0000 7fb08e600700 1 -- 172.21.15.31:0/3922295332 shutdown_connections 2024-03-21T10:27:18.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.621+0000 7fb08e600700 1 -- 172.21.15.31:0/3922295332 wait complete. 2024-03-21T10:27:18.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.622+0000 7fb08e600700 1 Processor -- start 2024-03-21T10:27:18.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.622+0000 7fb08e600700 1 -- start start 2024-03-21T10:27:18.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.623+0000 7fb08e600700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0880745c0 0x7fb08819f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:18.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.623+0000 7fb08e600700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08810e420 con 0x7fb0880745c0 2024-03-21T10:27:18.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.623+0000 7fb087400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0880745c0 0x7fb08819f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:18.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.623+0000 7fb087400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0880745c0 0x7fb08819f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46324/0 (socket says 172.21.15.31:46324) 2024-03-21T10:27:18.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.623+0000 7fb087400700 1 -- 172.21.15.31:0/1601711392 learned_addr learned my addr 172.21.15.31:0/1601711392 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:27:18.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.623+0000 7fb087400700 1 -- 172.21.15.31:0/1601711392 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb08819fb80 con 0x7fb0880745c0 2024-03-21T10:27:18.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.624+0000 7fb087400700 1 --2- 172.21.15.31:0/1601711392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0880745c0 0x7fb08819f640 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb078006010 tx=0x7fb078005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:18.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.624+0000 7fb07fe00700 1 -- 172.21.15.31:0/1601711392 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb078026070 con 0x7fb0880745c0 2024-03-21T10:27:18.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.624+0000 7fb07fe00700 1 -- 172.21.15.31:0/1601711392 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb07800bc30 con 0x7fb0880745c0 2024-03-21T10:27:18.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.624+0000 7fb08e600700 1 -- 172.21.15.31:0/1601711392 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb08819fe10 con 0x7fb0880745c0 2024-03-21T10:27:18.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.624+0000 7fb07fe00700 1 -- 172.21.15.31:0/1601711392 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb078003ab0 con 0x7fb0880745c0 2024-03-21T10:27:18.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.624+0000 7fb08e600700 1 -- 172.21.15.31:0/1601711392 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb088199710 con 0x7fb0880745c0 2024-03-21T10:27:18.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.625+0000 7fb08e600700 1 -- 172.21.15.31:0/1601711392 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb088075c70 con 0x7fb0880745c0 2024-03-21T10:27:18.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.625+0000 7fb07fe00700 1 -- 172.21.15.31:0/1601711392 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7fb078029070 con 0x7fb0880745c0 2024-03-21T10:27:18.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.626+0000 7fb07fe00700 1 --2- 172.21.15.31:0/1601711392 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb06c039280 0x7fb06c03b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:18.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.626+0000 7fb07fe00700 1 -- 172.21.15.31:0/1601711392 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb07801e070 con 0x7fb0880745c0 2024-03-21T10:27:18.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.626+0000 7fb086a00700 1 --2- 172.21.15.31:0/1601711392 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb06c039280 0x7fb06c03b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:18.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.627+0000 7fb086a00700 1 --2- 172.21.15.31:0/1601711392 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb06c039280 0x7fb06c03b740 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb074006fd0 tx=0x7fb074006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:18.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.632+0000 7fb07fe00700 1 -- 172.21.15.31:0/1601711392 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb078013070 con 0x7fb0880745c0 2024-03-21T10:27:18.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:18 smithi031 ceph-mon[31673]: Updating smithi031:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T10:27:18.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:18 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:18.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:18 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:18.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:18 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:18.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:18.867+0000 7fb08e600700 1 -- 172.21.15.31:0/1601711392 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi040", "target": ["mon-mgr", ""]}) v1 -- 0x7fb08819ac40 con 0x7fb06c039280 2024-03-21T10:27:20.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:19 smithi031 ceph-mon[31673]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:27:21.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:20 smithi031 ceph-mon[31673]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi040", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:27:21.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:20 smithi031 ceph-mon[31673]: Deploying cephadm binary to smithi040 2024-03-21T10:27:22.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:21 smithi031 ceph-mon[31673]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:27:23.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:23.908+0000 7fb07fe00700 1 -- 172.21.15.31:0/1601711392 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fb08819ac40 con 0x7fb06c039280 2024-03-21T10:27:23.908 INFO:teuthology.orchestra.run.smithi031.stdout:Added host 'smithi040' with addr '172.21.15.40' 2024-03-21T10:27:23.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:23.912+0000 7fb08e600700 1 -- 172.21.15.31:0/1601711392 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb06c039280 msgr2=0x7fb06c03b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:23.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:23.912+0000 7fb08e600700 1 --2- 172.21.15.31:0/1601711392 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb06c039280 0x7fb06c03b740 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb074006fd0 tx=0x7fb074006e40 comp rx=0 tx=0).stop 2024-03-21T10:27:23.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:23.912+0000 7fb08e600700 1 -- 172.21.15.31:0/1601711392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0880745c0 msgr2=0x7fb08819f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:23.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:23.912+0000 7fb08e600700 1 --2- 172.21.15.31:0/1601711392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0880745c0 0x7fb08819f640 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb078006010 tx=0x7fb078005050 comp rx=0 tx=0).stop 2024-03-21T10:27:23.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:23.912+0000 7fb08e600700 1 -- 172.21.15.31:0/1601711392 shutdown_connections 2024-03-21T10:27:23.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:23.912+0000 7fb08e600700 1 --2- 172.21.15.31:0/1601711392 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb06c039280 0x7fb06c03b740 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:23.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:23.912+0000 7fb08e600700 1 --2- 172.21.15.31:0/1601711392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0880745c0 0x7fb08819f640 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:23.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:23.912+0000 7fb08e600700 1 -- 172.21.15.31:0/1601711392 >> 172.21.15.31:0/1601711392 conn(0x7fb0880fd6a0 msgr2=0x7fb088105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:23.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:23.912+0000 7fb08e600700 1 -- 172.21.15.31:0/1601711392 shutdown_connections 2024-03-21T10:27:23.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:23.912+0000 7fb08e600700 1 -- 172.21.15.31:0/1601711392 wait complete. 2024-03-21T10:27:24.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:23 smithi031 ceph-mon[31673]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:27:24.700 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-21T10:27:25.152 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:27:25.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:24 smithi031 ceph-mon[31673]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:27:25.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:24 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:25.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:24 smithi031 ceph-mon[31673]: Added host smithi040 2024-03-21T10:27:25.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:24 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:27:25.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:24 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:26.224 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:26 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:27.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.221+0000 7f5aa2c00700 1 -- 172.21.15.31:0/4210127257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a9c106950 msgr2=0x7f5a9c106d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:27.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.221+0000 7f5aa2c00700 1 --2- 172.21.15.31:0/4210127257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a9c106950 0x7f5a9c106d30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5a90009910 tx=0x7f5a90009c20 comp rx=0 tx=0).stop 2024-03-21T10:27:27.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.221+0000 7f5aa2c00700 1 -- 172.21.15.31:0/4210127257 shutdown_connections 2024-03-21T10:27:27.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.221+0000 7f5aa2c00700 1 --2- 172.21.15.31:0/4210127257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a9c106950 0x7f5a9c106d30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:27.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.221+0000 7f5aa2c00700 1 -- 172.21.15.31:0/4210127257 >> 172.21.15.31:0/4210127257 conn(0x7f5a9c0fbb80 msgr2=0x7f5a9c0fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:27.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.222+0000 7f5aa2c00700 1 -- 172.21.15.31:0/4210127257 shutdown_connections 2024-03-21T10:27:27.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.222+0000 7f5aa2c00700 1 -- 172.21.15.31:0/4210127257 wait complete. 2024-03-21T10:27:27.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.223+0000 7f5aa2c00700 1 Processor -- start 2024-03-21T10:27:27.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.223+0000 7f5aa2c00700 1 -- start start 2024-03-21T10:27:27.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.223+0000 7f5aa2c00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a9c106950 0x7f5a9c1a3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:27.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.223+0000 7f5aa2c00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9c112650 con 0x7f5a9c106950 2024-03-21T10:27:27.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.223+0000 7f5aa1800700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a9c106950 0x7f5a9c1a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:27.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.223+0000 7f5aa1800700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a9c106950 0x7f5a9c1a3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43520/0 (socket says 172.21.15.31:43520) 2024-03-21T10:27:27.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.223+0000 7f5aa1800700 1 -- 172.21.15.31:0/3093047748 learned_addr learned my addr 172.21.15.31:0/3093047748 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:27:27.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.224+0000 7f5aa1800700 1 -- 172.21.15.31:0/3093047748 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a9c1a3c80 con 0x7f5a9c106950 2024-03-21T10:27:27.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.224+0000 7f5aa1800700 1 --2- 172.21.15.31:0/3093047748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a9c106950 0x7f5a9c1a3740 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5a90006010 tx=0x7f5a90005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:27.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.224+0000 7f5a9a000700 1 -- 172.21.15.31:0/3093047748 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5a90026070 con 0x7f5a9c106950 2024-03-21T10:27:27.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.224+0000 7f5a9a000700 1 -- 172.21.15.31:0/3093047748 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a9000bc30 con 0x7f5a9c106950 2024-03-21T10:27:27.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.224+0000 7f5a9a000700 1 -- 172.21.15.31:0/3093047748 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5a900039a0 con 0x7f5a9c106950 2024-03-21T10:27:27.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.224+0000 7f5aa2c00700 1 -- 172.21.15.31:0/3093047748 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a9c1a3f10 con 0x7f5a9c106950 2024-03-21T10:27:27.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.225+0000 7f5aa2c00700 1 -- 172.21.15.31:0/3093047748 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a9c19d900 con 0x7f5a9c106950 2024-03-21T10:27:27.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.226+0000 7f5a9a000700 1 -- 172.21.15.31:0/3093047748 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7f5a90029070 con 0x7f5a9c106950 2024-03-21T10:27:27.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.226+0000 7f5aa2c00700 1 -- 172.21.15.31:0/3093047748 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a9c101040 con 0x7f5a9c106950 2024-03-21T10:27:27.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.226+0000 7f5a9a000700 1 --2- 172.21.15.31:0/3093047748 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5a84039280 0x7f5a8403b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:27.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.226+0000 7f5a9a000700 1 -- 172.21.15.31:0/3093047748 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5a9001e070 con 0x7f5a9c106950 2024-03-21T10:27:27.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.226+0000 7f5aa0e00700 1 --2- 172.21.15.31:0/3093047748 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5a84039280 0x7f5a8403b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:27.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.227+0000 7f5aa0e00700 1 --2- 172.21.15.31:0/3093047748 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5a84039280 0x7f5a8403b740 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5a88006fd0 tx=0x7f5a88006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:27.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.232+0000 7f5a9a000700 1 -- 172.21.15.31:0/3093047748 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5a90013070 con 0x7f5a9c106950 2024-03-21T10:27:27.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.473+0000 7f5aa2c00700 1 -- 172.21.15.31:0/3093047748 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5a9c19ee30 con 0x7f5a84039280 2024-03-21T10:27:27.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.475+0000 7f5a9a000700 1 -- 172.21.15.31:0/3093047748 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f5a9c19ee30 con 0x7f5a84039280 2024-03-21T10:27:27.476 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:27:27.476 INFO:teuthology.orchestra.run.smithi031.stdout:[{"addr": "172.21.15.31", "hostname": "smithi031", "labels": [], "status": ""}, {"addr": "172.21.15.40", "hostname": "smithi040", "labels": [], "status": ""}] 2024-03-21T10:27:27.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.478+0000 7f5aa2c00700 1 -- 172.21.15.31:0/3093047748 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5a84039280 msgr2=0x7f5a8403b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:27.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.478+0000 7f5aa2c00700 1 --2- 172.21.15.31:0/3093047748 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5a84039280 0x7f5a8403b740 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5a88006fd0 tx=0x7f5a88006e40 comp rx=0 tx=0).stop 2024-03-21T10:27:27.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.478+0000 7f5aa2c00700 1 -- 172.21.15.31:0/3093047748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a9c106950 msgr2=0x7f5a9c1a3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:27.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.478+0000 7f5aa2c00700 1 --2- 172.21.15.31:0/3093047748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a9c106950 0x7f5a9c1a3740 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5a90006010 tx=0x7f5a90005030 comp rx=0 tx=0).stop 2024-03-21T10:27:27.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.478+0000 7f5aa2c00700 1 -- 172.21.15.31:0/3093047748 shutdown_connections 2024-03-21T10:27:27.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.478+0000 7f5aa2c00700 1 --2- 172.21.15.31:0/3093047748 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5a84039280 0x7f5a8403b740 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:27.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.478+0000 7f5aa2c00700 1 --2- 172.21.15.31:0/3093047748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a9c106950 0x7f5a9c1a3740 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:27.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.478+0000 7f5aa2c00700 1 -- 172.21.15.31:0/3093047748 >> 172.21.15.31:0/3093047748 conn(0x7f5a9c0fbb80 msgr2=0x7f5a9c0fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:27.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.478+0000 7f5aa2c00700 1 -- 172.21.15.31:0/3093047748 shutdown_connections 2024-03-21T10:27:27.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:27.478+0000 7f5aa2c00700 1 -- 172.21.15.31:0/3093047748 wait complete. 2024-03-21T10:27:27.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:27 smithi031 ceph-mon[31673]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:27:27.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:27 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:28.298 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi070 2024-03-21T10:27:28.298 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-21T10:27:28.298 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/etc/ceph/ceph.conf 2024-03-21T10:27:28.317 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-21T10:27:28.317 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-21T10:27:28.374 INFO:tasks.cephadm:Adding host smithi070 to orchestrator... 2024-03-21T10:27:28.375 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph orch host add smithi070 2024-03-21T10:27:28.855 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:27:29.002 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:28 smithi031 ceph-mon[31673]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T10:27:30.029 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:29 smithi031 ceph-mon[31673]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:27:30.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.978+0000 7f5738400700 1 -- 172.21.15.31:0/342995691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5730100900 msgr2=0x7f5730100ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:30.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.978+0000 7f5738400700 1 --2- 172.21.15.31:0/342995691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5730100900 0x7f5730100ce0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5724009910 tx=0x7f5724009c20 comp rx=0 tx=0).stop 2024-03-21T10:27:30.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.978+0000 7f5738400700 1 -- 172.21.15.31:0/342995691 shutdown_connections 2024-03-21T10:27:30.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.978+0000 7f5738400700 1 --2- 172.21.15.31:0/342995691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5730100900 0x7f5730100ce0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:30.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.978+0000 7f5738400700 1 -- 172.21.15.31:0/342995691 >> 172.21.15.31:0/342995691 conn(0x7f57300fc540 msgr2=0x7f57300fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:30.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.978+0000 7f5738400700 1 -- 172.21.15.31:0/342995691 shutdown_connections 2024-03-21T10:27:30.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.978+0000 7f5738400700 1 -- 172.21.15.31:0/342995691 wait complete. 2024-03-21T10:27:30.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.979+0000 7f5738400700 1 Processor -- start 2024-03-21T10:27:30.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.980+0000 7f5738400700 1 -- start start 2024-03-21T10:27:30.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.980+0000 7f5738400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5730100900 0x7f573019f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:30.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.980+0000 7f5738400700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573010e3e0 con 0x7f5730100900 2024-03-21T10:27:30.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.980+0000 7f5735a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5730100900 0x7f573019f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:30.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.980+0000 7f5735a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5730100900 0x7f573019f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43546/0 (socket says 172.21.15.31:43546) 2024-03-21T10:27:30.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.980+0000 7f5735a00700 1 -- 172.21.15.31:0/600450839 learned_addr learned my addr 172.21.15.31:0/600450839 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:27:30.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.981+0000 7f5735a00700 1 -- 172.21.15.31:0/600450839 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573019fb40 con 0x7f5730100900 2024-03-21T10:27:30.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.981+0000 7f5735a00700 1 --2- 172.21.15.31:0/600450839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5730100900 0x7f573019f600 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5724006010 tx=0x7f5724005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:30.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.981+0000 7f572e000700 1 -- 172.21.15.31:0/600450839 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5724026070 con 0x7f5730100900 2024-03-21T10:27:30.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.981+0000 7f572e000700 1 -- 172.21.15.31:0/600450839 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f572400bc30 con 0x7f5730100900 2024-03-21T10:27:30.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.981+0000 7f5738400700 1 -- 172.21.15.31:0/600450839 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f573019fdd0 con 0x7f5730100900 2024-03-21T10:27:30.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.981+0000 7f572e000700 1 -- 172.21.15.31:0/600450839 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f57240039d0 con 0x7f5730100900 2024-03-21T10:27:30.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.982+0000 7f5738400700 1 -- 172.21.15.31:0/600450839 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57301996d0 con 0x7f5730100900 2024-03-21T10:27:30.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.983+0000 7f572e000700 1 -- 172.21.15.31:0/600450839 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7f5724029070 con 0x7f5730100900 2024-03-21T10:27:30.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.983+0000 7f5738400700 1 -- 172.21.15.31:0/600450839 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f573006a690 con 0x7f5730100900 2024-03-21T10:27:30.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.983+0000 7f572e000700 1 --2- 172.21.15.31:0/600450839 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5718039220 0x7f571803b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:30.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.983+0000 7f572e000700 1 -- 172.21.15.31:0/600450839 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f572401e070 con 0x7f5730100900 2024-03-21T10:27:30.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.983+0000 7f5735000700 1 --2- 172.21.15.31:0/600450839 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5718039220 0x7f571803b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:30.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.984+0000 7f5735000700 1 --2- 172.21.15.31:0/600450839 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5718039220 0x7f571803b6e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f571c006fd0 tx=0x7f571c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:30.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:30.989+0000 7f572e000700 1 -- 172.21.15.31:0/600450839 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5724013070 con 0x7f5730100900 2024-03-21T10:27:31.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:30 smithi031 ceph-mon[31673]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:27:31.058 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:30 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:31.058 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:30 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:31.058 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:30 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:31.058 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:30 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:31.058 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:30 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi040", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:27:31.058 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:30 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:27:31.058 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:30 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:27:31.058 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:30 smithi031 ceph-mon[31673]: Updating smithi040:/etc/ceph/ceph.conf 2024-03-21T10:27:31.058 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:30 smithi031 ceph-mon[31673]: Updating smithi040:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T10:27:31.058 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:30 smithi031 ceph-mon[31673]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-03-21T10:27:31.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:31.224+0000 7f5738400700 1 -- 172.21.15.31:0/600450839 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi070", "target": ["mon-mgr", ""]}) v1 -- 0x7f573019ac00 con 0x7f5718039220 2024-03-21T10:27:32.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:32 smithi031 ceph-mon[31673]: Updating smithi040:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T10:27:32.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:32 smithi031 ceph-mon[31673]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi070", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:27:32.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:32 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:32.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:32 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:32.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:32 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:33.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:33 smithi031 ceph-mon[31673]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:27:33.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:33 smithi031 ceph-mon[31673]: Deploying cephadm binary to smithi070 2024-03-21T10:27:35.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:35 smithi031 ceph-mon[31673]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:27:36.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:36.201+0000 7f572e000700 1 -- 172.21.15.31:0/600450839 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f573019ac00 con 0x7f5718039220 2024-03-21T10:27:36.201 INFO:teuthology.orchestra.run.smithi031.stdout:Added host 'smithi070' with addr '172.21.15.70' 2024-03-21T10:27:36.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:36.205+0000 7f5738400700 1 -- 172.21.15.31:0/600450839 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5718039220 msgr2=0x7f571803b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:36.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:36.205+0000 7f5738400700 1 --2- 172.21.15.31:0/600450839 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5718039220 0x7f571803b6e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f571c006fd0 tx=0x7f571c006e40 comp rx=0 tx=0).stop 2024-03-21T10:27:36.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:36.205+0000 7f5738400700 1 -- 172.21.15.31:0/600450839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5730100900 msgr2=0x7f573019f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:36.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:36.205+0000 7f5738400700 1 --2- 172.21.15.31:0/600450839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5730100900 0x7f573019f600 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5724006010 tx=0x7f5724005030 comp rx=0 tx=0).stop 2024-03-21T10:27:36.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:36.205+0000 7f5738400700 1 -- 172.21.15.31:0/600450839 shutdown_connections 2024-03-21T10:27:36.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:36.205+0000 7f5738400700 1 --2- 172.21.15.31:0/600450839 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5718039220 0x7f571803b6e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:36.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:36.205+0000 7f5738400700 1 --2- 172.21.15.31:0/600450839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5730100900 0x7f573019f600 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:36.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:36.205+0000 7f5738400700 1 -- 172.21.15.31:0/600450839 >> 172.21.15.31:0/600450839 conn(0x7f57300fc540 msgr2=0x7f57300fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:36.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:36.205+0000 7f5738400700 1 -- 172.21.15.31:0/600450839 shutdown_connections 2024-03-21T10:27:36.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:36.205+0000 7f5738400700 1 -- 172.21.15.31:0/600450839 wait complete. 2024-03-21T10:27:37.043 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-21T10:27:37.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:37 smithi031 ceph-mon[31673]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:27:37.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:37 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:37.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:37 smithi031 ceph-mon[31673]: Added host smithi070 2024-03-21T10:27:37.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:37 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:27:37.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:37 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:37.505 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:27:38.659 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:38 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:39.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.537+0000 7f24cac00700 1 -- 172.21.15.31:0/1082047478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24c41083f0 msgr2=0x7f24c41087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:39.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.537+0000 7f24cac00700 1 --2- 172.21.15.31:0/1082047478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24c41083f0 0x7f24c41087d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f24b4009910 tx=0x7f24b4009c20 comp rx=0 tx=0).stop 2024-03-21T10:27:39.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.537+0000 7f24cac00700 1 -- 172.21.15.31:0/1082047478 shutdown_connections 2024-03-21T10:27:39.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.537+0000 7f24cac00700 1 --2- 172.21.15.31:0/1082047478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24c41083f0 0x7f24c41087d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:39.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.537+0000 7f24cac00700 1 -- 172.21.15.31:0/1082047478 >> 172.21.15.31:0/1082047478 conn(0x7f24c4077930 msgr2=0x7f24c4079d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:39.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.538+0000 7f24cac00700 1 -- 172.21.15.31:0/1082047478 shutdown_connections 2024-03-21T10:27:39.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.538+0000 7f24cac00700 1 -- 172.21.15.31:0/1082047478 wait complete. 2024-03-21T10:27:39.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.539+0000 7f24cac00700 1 Processor -- start 2024-03-21T10:27:39.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.539+0000 7f24cac00700 1 -- start start 2024-03-21T10:27:39.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.540+0000 7f24cac00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24c41083f0 0x7f24c41a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:39.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.540+0000 7f24cac00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c41127c0 con 0x7f24c41083f0 2024-03-21T10:27:39.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.540+0000 7f24c3e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24c41083f0 0x7f24c41a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:39.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.540+0000 7f24c3e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24c41083f0 0x7f24c41a38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53088/0 (socket says 172.21.15.31:53088) 2024-03-21T10:27:39.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.540+0000 7f24c3e00700 1 -- 172.21.15.31:0/3625940050 learned_addr learned my addr 172.21.15.31:0/3625940050 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:27:39.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.540+0000 7f24c3e00700 1 -- 172.21.15.31:0/3625940050 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c41a3e30 con 0x7f24c41083f0 2024-03-21T10:27:39.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.541+0000 7f24c3e00700 1 --2- 172.21.15.31:0/3625940050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24c41083f0 0x7f24c41a38f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f24b4005850 tx=0x7f24b4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:39.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.541+0000 7f24c0c00700 1 -- 172.21.15.31:0/3625940050 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f24b4026070 con 0x7f24c41083f0 2024-03-21T10:27:39.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.541+0000 7f24cac00700 1 -- 172.21.15.31:0/3625940050 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24c41a4120 con 0x7f24c41083f0 2024-03-21T10:27:39.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.541+0000 7f24c0c00700 1 -- 172.21.15.31:0/3625940050 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f24b400bbd0 con 0x7f24c41083f0 2024-03-21T10:27:39.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.541+0000 7f24c0c00700 1 -- 172.21.15.31:0/3625940050 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f24b4003970 con 0x7f24c41083f0 2024-03-21T10:27:39.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.541+0000 7f24cac00700 1 -- 172.21.15.31:0/3625940050 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24c419dab0 con 0x7f24c41083f0 2024-03-21T10:27:39.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.542+0000 7f24cac00700 1 -- 172.21.15.31:0/3625940050 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24c41098d0 con 0x7f24c41083f0 2024-03-21T10:27:39.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.542+0000 7f24c0c00700 1 -- 172.21.15.31:0/3625940050 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7f24b4029070 con 0x7f24c41083f0 2024-03-21T10:27:39.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.543+0000 7f24c0c00700 1 --2- 172.21.15.31:0/3625940050 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f24a8039280 0x7f24a803b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:39.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.543+0000 7f24c0c00700 1 -- 172.21.15.31:0/3625940050 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f24b401e070 con 0x7f24c41083f0 2024-03-21T10:27:39.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.543+0000 7f24c3400700 1 --2- 172.21.15.31:0/3625940050 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f24a8039280 0x7f24a803b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:39.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.544+0000 7f24c3400700 1 --2- 172.21.15.31:0/3625940050 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f24a8039280 0x7f24a803b740 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f24b0006fd0 tx=0x7f24b0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:39.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.548+0000 7f24c0c00700 1 -- 172.21.15.31:0/3625940050 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f24b4013070 con 0x7f24c41083f0 2024-03-21T10:27:39.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.785+0000 7f24cac00700 1 -- 172.21.15.31:0/3625940050 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f24c419efe0 con 0x7f24a8039280 2024-03-21T10:27:39.787 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:39 smithi031 ceph-mon[31673]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:27:39.788 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:39 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:39.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.788+0000 7f24c0c00700 1 -- 172.21.15.31:0/3625940050 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+238 (secure 0 0 0) 0x7f24c419efe0 con 0x7f24a8039280 2024-03-21T10:27:39.789 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:27:39.789 INFO:teuthology.orchestra.run.smithi031.stdout:[{"addr": "172.21.15.31", "hostname": "smithi031", "labels": [], "status": ""}, {"addr": "172.21.15.40", "hostname": "smithi040", "labels": [], "status": ""}, {"addr": "172.21.15.70", "hostname": "smithi070", "labels": [], "status": ""}] 2024-03-21T10:27:39.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.792+0000 7f24cac00700 1 -- 172.21.15.31:0/3625940050 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f24a8039280 msgr2=0x7f24a803b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:39.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.792+0000 7f24cac00700 1 --2- 172.21.15.31:0/3625940050 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f24a8039280 0x7f24a803b740 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f24b0006fd0 tx=0x7f24b0006e40 comp rx=0 tx=0).stop 2024-03-21T10:27:39.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.792+0000 7f24cac00700 1 -- 172.21.15.31:0/3625940050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24c41083f0 msgr2=0x7f24c41a38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:39.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.792+0000 7f24cac00700 1 --2- 172.21.15.31:0/3625940050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24c41083f0 0x7f24c41a38f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f24b4005850 tx=0x7f24b4005030 comp rx=0 tx=0).stop 2024-03-21T10:27:39.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.792+0000 7f24cac00700 1 -- 172.21.15.31:0/3625940050 shutdown_connections 2024-03-21T10:27:39.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.792+0000 7f24cac00700 1 --2- 172.21.15.31:0/3625940050 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f24a8039280 0x7f24a803b740 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:39.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.792+0000 7f24cac00700 1 --2- 172.21.15.31:0/3625940050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24c41083f0 0x7f24c41a38f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:39.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.792+0000 7f24cac00700 1 -- 172.21.15.31:0/3625940050 >> 172.21.15.31:0/3625940050 conn(0x7f24c4077930 msgr2=0x7f24c4109dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:39.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.792+0000 7f24cac00700 1 -- 172.21.15.31:0/3625940050 shutdown_connections 2024-03-21T10:27:39.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:39.793+0000 7f24cac00700 1 -- 172.21.15.31:0/3625940050 wait complete. 2024-03-21T10:27:40.527 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-21T10:27:40.527 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-21T10:27:40.983 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:27:41.444 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:41 smithi031 ceph-mon[31673]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T10:27:41.445 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:41 smithi031 ceph-mon[31673]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:27:43.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.033+0000 7f2f25200700 1 -- 172.21.15.31:0/872367248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f200745c0 msgr2=0x7f2f200749a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:43.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.033+0000 7f2f25200700 1 --2- 172.21.15.31:0/872367248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f200745c0 0x7f2f200749a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2f10009910 tx=0x7f2f10009c20 comp rx=0 tx=0).stop 2024-03-21T10:27:43.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.033+0000 7f2f25200700 1 -- 172.21.15.31:0/872367248 shutdown_connections 2024-03-21T10:27:43.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.033+0000 7f2f25200700 1 --2- 172.21.15.31:0/872367248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f200745c0 0x7f2f200749a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:43.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.033+0000 7f2f25200700 1 -- 172.21.15.31:0/872367248 >> 172.21.15.31:0/872367248 conn(0x7f2f200fd670 msgr2=0x7f2f200ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:43.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.033+0000 7f2f25200700 1 -- 172.21.15.31:0/872367248 shutdown_connections 2024-03-21T10:27:43.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.033+0000 7f2f25200700 1 -- 172.21.15.31:0/872367248 wait complete. 2024-03-21T10:27:43.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.034+0000 7f2f25200700 1 Processor -- start 2024-03-21T10:27:43.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.035+0000 7f2f25200700 1 -- start start 2024-03-21T10:27:43.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.035+0000 7f2f25200700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f200745c0 0x7f2f2019f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:43.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.035+0000 7f2f25200700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2010e410 con 0x7f2f200745c0 2024-03-21T10:27:43.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.035+0000 7f2f1de00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f200745c0 0x7f2f2019f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:43.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.035+0000 7f2f1de00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f200745c0 0x7f2f2019f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53116/0 (socket says 172.21.15.31:53116) 2024-03-21T10:27:43.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.035+0000 7f2f1de00700 1 -- 172.21.15.31:0/447100085 learned_addr learned my addr 172.21.15.31:0/447100085 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:27:43.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.036+0000 7f2f1de00700 1 -- 172.21.15.31:0/447100085 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2019fb40 con 0x7f2f200745c0 2024-03-21T10:27:43.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.036+0000 7f2f1de00700 1 --2- 172.21.15.31:0/447100085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f200745c0 0x7f2f2019f600 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2f10006010 tx=0x7f2f10005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:43.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.036+0000 7f2f16a00700 1 -- 172.21.15.31:0/447100085 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f10026070 con 0x7f2f200745c0 2024-03-21T10:27:43.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.036+0000 7f2f16a00700 1 -- 172.21.15.31:0/447100085 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2f1000bc30 con 0x7f2f200745c0 2024-03-21T10:27:43.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.036+0000 7f2f16a00700 1 -- 172.21.15.31:0/447100085 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f100039a0 con 0x7f2f200745c0 2024-03-21T10:27:43.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.036+0000 7f2f25200700 1 -- 172.21.15.31:0/447100085 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f2019fdd0 con 0x7f2f200745c0 2024-03-21T10:27:43.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.037+0000 7f2f25200700 1 -- 172.21.15.31:0/447100085 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f20199780 con 0x7f2f200745c0 2024-03-21T10:27:43.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.038+0000 7f2f16a00700 1 -- 172.21.15.31:0/447100085 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7f2f10029070 con 0x7f2f200745c0 2024-03-21T10:27:43.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.038+0000 7f2f25200700 1 -- 172.21.15.31:0/447100085 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f20075c70 con 0x7f2f200745c0 2024-03-21T10:27:43.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.038+0000 7f2f16a00700 1 --2- 172.21.15.31:0/447100085 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2f04039280 0x7f2f0403b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:43.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.038+0000 7f2f16a00700 1 -- 172.21.15.31:0/447100085 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2f1001e070 con 0x7f2f200745c0 2024-03-21T10:27:43.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.039+0000 7f2f1d400700 1 --2- 172.21.15.31:0/447100085 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2f04039280 0x7f2f0403b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:43.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.039+0000 7f2f1d400700 1 --2- 172.21.15.31:0/447100085 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2f04039280 0x7f2f0403b740 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c006fd0 tx=0x7f2f0c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:43.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.045+0000 7f2f16a00700 1 -- 172.21.15.31:0/447100085 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2f10013070 con 0x7f2f200745c0 2024-03-21T10:27:43.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:43.294+0000 7f2f25200700 1 -- 172.21.15.31:0/447100085 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f2f200637c0 con 0x7f2f200745c0 2024-03-21T10:27:43.296 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:43 smithi031 ceph-mon[31673]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:27:43.296 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:43.296 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:43.296 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:43.296 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:43.296 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi070", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:27:43.296 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:27:43.296 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:27:43.296 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:43 smithi031 ceph-mon[31673]: Updating smithi070:/etc/ceph/ceph.conf 2024-03-21T10:27:43.296 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:43 smithi031 ceph-mon[31673]: Updating smithi070:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T10:27:43.297 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:43 smithi031 ceph-mon[31673]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-03-21T10:27:44.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:44.191+0000 7f2f16a00700 1 -- 172.21.15.31:0/447100085 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f2f10030090 con 0x7f2f200745c0 2024-03-21T10:27:44.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:44.196+0000 7f2f25200700 1 -- 172.21.15.31:0/447100085 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2f04039280 msgr2=0x7f2f0403b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:44.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:44.196+0000 7f2f25200700 1 --2- 172.21.15.31:0/447100085 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2f04039280 0x7f2f0403b740 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c006fd0 tx=0x7f2f0c006e40 comp rx=0 tx=0).stop 2024-03-21T10:27:44.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:44.196+0000 7f2f25200700 1 -- 172.21.15.31:0/447100085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f200745c0 msgr2=0x7f2f2019f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:44.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:44.196+0000 7f2f25200700 1 --2- 172.21.15.31:0/447100085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f200745c0 0x7f2f2019f600 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2f10006010 tx=0x7f2f10005030 comp rx=0 tx=0).stop 2024-03-21T10:27:44.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:44.196+0000 7f2f25200700 1 -- 172.21.15.31:0/447100085 shutdown_connections 2024-03-21T10:27:44.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:44.196+0000 7f2f25200700 1 --2- 172.21.15.31:0/447100085 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2f04039280 0x7f2f0403b740 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:44.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:44.196+0000 7f2f25200700 1 --2- 172.21.15.31:0/447100085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f200745c0 0x7f2f2019f600 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:44.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:44.196+0000 7f2f25200700 1 -- 172.21.15.31:0/447100085 >> 172.21.15.31:0/447100085 conn(0x7f2f200fd670 msgr2=0x7f2f20105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:44.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:44.197+0000 7f2f25200700 1 -- 172.21.15.31:0/447100085 shutdown_connections 2024-03-21T10:27:44.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:27:44.197+0000 7f2f25200700 1 -- 172.21.15.31:0/447100085 wait complete. 2024-03-21T10:27:44.198 INFO:teuthology.orchestra.run.smithi031.stderr:adjusted tunables profile to default 2024-03-21T10:27:44.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:44 smithi031 ceph-mon[31673]: Updating smithi070:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T10:27:44.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:44 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:44.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:44 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:44.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:44 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:44.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:44 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/447100085' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-21T10:27:45.020 INFO:tasks.cephadm:Adding mon.a on smithi031 2024-03-21T10:27:45.021 INFO:tasks.cephadm:Adding mon.b on smithi040 2024-03-21T10:27:45.021 INFO:tasks.cephadm:Adding mon.c on smithi070 2024-03-21T10:27:45.021 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph orch apply mon '3;smithi031:172.21.15.31=a;smithi040:172.21.15.40=b;smithi070:172.21.15.70=c' 2024-03-21T10:27:45.194 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:45 smithi031 ceph-mon[31673]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:27:45.195 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:45 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/447100085' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-21T10:27:45.257 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T10:27:45.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:45 smithi031 ceph-mon[31673]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T10:27:46.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.723+0000 7f2c0c833700 1 -- 172.21.15.70:0/3804007030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c08106a80 msgr2=0x7f2c08106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:46.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.723+0000 7f2c0c833700 1 --2- 172.21.15.70:0/3804007030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c08106a80 0x7f2c08106e60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8009910 tx=0x7f2bf8009c20 comp rx=0 tx=0).stop 2024-03-21T10:27:46.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.723+0000 7f2c0c833700 1 -- 172.21.15.70:0/3804007030 shutdown_connections 2024-03-21T10:27:46.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.723+0000 7f2c0c833700 1 --2- 172.21.15.70:0/3804007030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c08106a80 0x7f2c08106e60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:46.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.723+0000 7f2c0c833700 1 -- 172.21.15.70:0/3804007030 >> 172.21.15.70:0/3804007030 conn(0x7f2c080fc590 msgr2=0x7f2c080fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:46.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.723+0000 7f2c0c833700 1 -- 172.21.15.70:0/3804007030 shutdown_connections 2024-03-21T10:27:46.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.723+0000 7f2c0c833700 1 -- 172.21.15.70:0/3804007030 wait complete. 2024-03-21T10:27:46.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.723+0000 7f2c0c833700 1 Processor -- start 2024-03-21T10:27:46.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.724+0000 7f2c0c833700 1 -- start start 2024-03-21T10:27:46.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.724+0000 7f2c0c833700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c08106a80 0x7f2c081954b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:46.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.724+0000 7f2c0c833700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c08109fc0 con 0x7f2c08106a80 2024-03-21T10:27:46.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.724+0000 7f2c0659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c08106a80 0x7f2c081954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:46.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.724+0000 7f2c0659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c08106a80 0x7f2c081954b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.70:37674/0 (socket says 172.21.15.70:37674) 2024-03-21T10:27:46.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.724+0000 7f2c0659c700 1 -- 172.21.15.70:0/576611096 learned_addr learned my addr 172.21.15.70:0/576611096 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-03-21T10:27:46.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.725+0000 7f2c0659c700 1 -- 172.21.15.70:0/576611096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c081959f0 con 0x7f2c08106a80 2024-03-21T10:27:46.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.725+0000 7f2c0659c700 1 --2- 172.21.15.70:0/576611096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c08106a80 0x7f2c081954b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8005850 tx=0x7f2bf8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:46.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.725+0000 7f2bff7fe700 1 -- 172.21.15.70:0/576611096 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2bf8026070 con 0x7f2c08106a80 2024-03-21T10:27:46.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.725+0000 7f2c0c833700 1 -- 172.21.15.70:0/576611096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c08195ce0 con 0x7f2c08106a80 2024-03-21T10:27:46.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.725+0000 7f2c0c833700 1 -- 172.21.15.70:0/576611096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c08199d30 con 0x7f2c08106a80 2024-03-21T10:27:46.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.726+0000 7f2bff7fe700 1 -- 172.21.15.70:0/576611096 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2bf800bbd0 con 0x7f2c08106a80 2024-03-21T10:27:46.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.726+0000 7f2bff7fe700 1 -- 172.21.15.70:0/576611096 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2bf8003680 con 0x7f2c08106a80 2024-03-21T10:27:46.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.726+0000 7f2c0c833700 1 -- 172.21.15.70:0/576611096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be8005360 con 0x7f2c08106a80 2024-03-21T10:27:46.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.727+0000 7f2bff7fe700 1 -- 172.21.15.70:0/576611096 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7f2bf8029070 con 0x7f2c08106a80 2024-03-21T10:27:46.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.727+0000 7f2bff7fe700 1 --2- 172.21.15.70:0/576611096 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2bf403b4d0 0x7f2bf403d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:46.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.727+0000 7f2bff7fe700 1 -- 172.21.15.70:0/576611096 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2bf801e070 con 0x7f2c08106a80 2024-03-21T10:27:46.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.728+0000 7f2c05d9b700 1 --2- 172.21.15.70:0/576611096 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2bf403b4d0 0x7f2bf403d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:46.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.729+0000 7f2c05d9b700 1 --2- 172.21.15.70:0/576611096 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2bf403b4d0 0x7f2bf403d990 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0006fd0 tx=0x7f2bf0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:46.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.743+0000 7f2bff7fe700 1 -- 172.21.15.70:0/576611096 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2bf807b740 con 0x7f2c08106a80 2024-03-21T10:27:46.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.949+0000 7f2c0c833700 1 -- 172.21.15.70:0/576611096 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi031:172.21.15.31=a;smithi040:172.21.15.40=b;smithi070:172.21.15.70=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f2be8000ca0 con 0x7f2bf403b4d0 2024-03-21T10:27:46.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.964+0000 7f2bff7fe700 1 -- 172.21.15.70:0/576611096 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f2be8000ca0 con 0x7f2bf403b4d0 2024-03-21T10:27:46.965 INFO:teuthology.orchestra.run.smithi070.stdout:Scheduled mon update... 2024-03-21T10:27:46.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.967+0000 7f2c0c833700 1 -- 172.21.15.70:0/576611096 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2bf403b4d0 msgr2=0x7f2bf403d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:46.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.967+0000 7f2c0c833700 1 --2- 172.21.15.70:0/576611096 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2bf403b4d0 0x7f2bf403d990 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0006fd0 tx=0x7f2bf0006e40 comp rx=0 tx=0).stop 2024-03-21T10:27:46.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.967+0000 7f2c0c833700 1 -- 172.21.15.70:0/576611096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c08106a80 msgr2=0x7f2c081954b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:46.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.967+0000 7f2c0c833700 1 --2- 172.21.15.70:0/576611096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c08106a80 0x7f2c081954b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8005850 tx=0x7f2bf8005030 comp rx=0 tx=0).stop 2024-03-21T10:27:46.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.967+0000 7f2c0c833700 1 -- 172.21.15.70:0/576611096 shutdown_connections 2024-03-21T10:27:46.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.967+0000 7f2c0c833700 1 --2- 172.21.15.70:0/576611096 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2bf403b4d0 0x7f2bf403d990 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:46.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.967+0000 7f2c0c833700 1 --2- 172.21.15.70:0/576611096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c08106a80 0x7f2c081954b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:46.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.967+0000 7f2c0c833700 1 -- 172.21.15.70:0/576611096 >> 172.21.15.70:0/576611096 conn(0x7f2c080fc590 msgr2=0x7f2c08108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:46.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.967+0000 7f2c0c833700 1 -- 172.21.15.70:0/576611096 shutdown_connections 2024-03-21T10:27:46.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:46.967+0000 7f2c0c833700 1 -- 172.21.15.70:0/576611096 wait complete. 2024-03-21T10:27:47.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:47 smithi031 ceph-mon[31673]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:27:47.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:47 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:47.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:47 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:27:47.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:47 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:27:47.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:47 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:27:47.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:47 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:47.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:47 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T10:27:47.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:47 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:27:47.690 DEBUG:teuthology.orchestra.run.smithi040:mon.b> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mon.b.service 2024-03-21T10:27:47.693 DEBUG:teuthology.orchestra.run.smithi070:mon.c> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mon.c.service 2024-03-21T10:27:47.695 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-21T10:27:47.696 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-21T10:27:48.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:48 smithi031 ceph-mon[31673]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi031:172.21.15.31=a;smithi040:172.21.15.40=b;smithi070:172.21.15.70=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:27:48.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:48 smithi031 ceph-mon[31673]: Saving service mon spec with placement smithi031:172.21.15.31=a;smithi040:172.21.15.40=b;smithi070:172.21.15.70=c;count:3 2024-03-21T10:27:48.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:48 smithi031 ceph-mon[31673]: Deploying daemon mon.c on smithi070 2024-03-21T10:27:48.661 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T10:27:49.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:49 smithi031 ceph-mon[31673]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:27:51.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.258+0000 7feace3bd700 1 -- 172.21.15.70:0/182395883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feac8102d20 msgr2=0x7feac8103100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:51.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.258+0000 7feace3bd700 1 --2- 172.21.15.70:0/182395883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feac8102d20 0x7feac8103100 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7feab8009910 tx=0x7feab8009c20 comp rx=0 tx=0).stop 2024-03-21T10:27:51.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.258+0000 7feace3bd700 1 -- 172.21.15.70:0/182395883 shutdown_connections 2024-03-21T10:27:51.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.258+0000 7feace3bd700 1 --2- 172.21.15.70:0/182395883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feac8102d20 0x7feac8103100 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:51.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.258+0000 7feace3bd700 1 -- 172.21.15.70:0/182395883 >> 172.21.15.70:0/182395883 conn(0x7feac80776d0 msgr2=0x7feac80fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:51.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.258+0000 7feace3bd700 1 -- 172.21.15.70:0/182395883 shutdown_connections 2024-03-21T10:27:51.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.258+0000 7feace3bd700 1 -- 172.21.15.70:0/182395883 wait complete. 2024-03-21T10:27:51.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.259+0000 7feace3bd700 1 Processor -- start 2024-03-21T10:27:51.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.259+0000 7feace3bd700 1 -- start start 2024-03-21T10:27:51.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.259+0000 7feace3bd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feac8102d20 0x7feac8197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:51.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.259+0000 7feace3bd700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac810c210 con 0x7feac8102d20 2024-03-21T10:27:51.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.260+0000 7feac7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feac8102d20 0x7feac8197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:51.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.260+0000 7feac7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feac8102d20 0x7feac8197740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.70:37704/0 (socket says 172.21.15.70:37704) 2024-03-21T10:27:51.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.260+0000 7feac7fff700 1 -- 172.21.15.70:0/1400213739 learned_addr learned my addr 172.21.15.70:0/1400213739 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-03-21T10:27:51.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.260+0000 7feac7fff700 1 -- 172.21.15.70:0/1400213739 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac8197c80 con 0x7feac8102d20 2024-03-21T10:27:51.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.261+0000 7feac7fff700 1 --2- 172.21.15.70:0/1400213739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feac8102d20 0x7feac8197740 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7feab8006010 tx=0x7feab8005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:51.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.261+0000 7feac57fa700 1 -- 172.21.15.70:0/1400213739 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feab8026070 con 0x7feac8102d20 2024-03-21T10:27:51.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.261+0000 7feace3bd700 1 -- 172.21.15.70:0/1400213739 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac8197f10 con 0x7feac8102d20 2024-03-21T10:27:51.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.261+0000 7feac57fa700 1 -- 172.21.15.70:0/1400213739 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feab800bc30 con 0x7feac8102d20 2024-03-21T10:27:51.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.261+0000 7feac57fa700 1 -- 172.21.15.70:0/1400213739 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feab8003ab0 con 0x7feac8102d20 2024-03-21T10:27:51.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.261+0000 7feace3bd700 1 -- 172.21.15.70:0/1400213739 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac8198370 con 0x7feac8102d20 2024-03-21T10:27:51.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.262+0000 7feace3bd700 1 -- 172.21.15.70:0/1400213739 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac8069e60 con 0x7feac8102d20 2024-03-21T10:27:51.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.262+0000 7feac57fa700 1 -- 172.21.15.70:0/1400213739 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7feab8029070 con 0x7feac8102d20 2024-03-21T10:27:51.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.263+0000 7feac57fa700 1 --2- 172.21.15.70:0/1400213739 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7feab0039220 0x7feab003b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:51.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.263+0000 7feac57fa700 1 -- 172.21.15.70:0/1400213739 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7feab801e070 con 0x7feac8102d20 2024-03-21T10:27:51.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.263+0000 7feac77fe700 1 --2- 172.21.15.70:0/1400213739 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7feab0039220 0x7feab003b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:51.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.264+0000 7feac77fe700 1 --2- 172.21.15.70:0/1400213739 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7feab0039220 0x7feab003b6e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7feabc006fd0 tx=0x7feabc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:51.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.278+0000 7feac57fa700 1 -- 172.21.15.70:0/1400213739 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7feab8013070 con 0x7feac8102d20 2024-03-21T10:27:51.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:51 smithi031 ceph-mon[31673]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:27:51.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.574+0000 7feace3bd700 1 -- 172.21.15.70:0/1400213739 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feac8198aa0 con 0x7feac8102d20 2024-03-21T10:27:51.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.575+0000 7feac57fa700 1 -- 172.21.15.70:0/1400213739 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+745 (secure 0 0 0) 0x7feab8030090 con 0x7feac8102d20 2024-03-21T10:27:51.575 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-21T10:27:51.575 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":1,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","modified":"2024-03-21T10:25:36.790162Z","created":"2024-03-21T10:25:36.790162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-21T10:27:51.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.577+0000 7feace3bd700 1 -- 172.21.15.70:0/1400213739 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7feab0039220 msgr2=0x7feab003b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:51.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.577+0000 7feace3bd700 1 --2- 172.21.15.70:0/1400213739 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7feab0039220 0x7feab003b6e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7feabc006fd0 tx=0x7feabc006e40 comp rx=0 tx=0).stop 2024-03-21T10:27:51.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.577+0000 7feace3bd700 1 -- 172.21.15.70:0/1400213739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feac8102d20 msgr2=0x7feac8197740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:51.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.577+0000 7feace3bd700 1 --2- 172.21.15.70:0/1400213739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feac8102d20 0x7feac8197740 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7feab8006010 tx=0x7feab8005050 comp rx=0 tx=0).stop 2024-03-21T10:27:51.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.578+0000 7feace3bd700 1 -- 172.21.15.70:0/1400213739 shutdown_connections 2024-03-21T10:27:51.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.578+0000 7feace3bd700 1 --2- 172.21.15.70:0/1400213739 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7feab0039220 0x7feab003b6e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:51.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.578+0000 7feace3bd700 1 --2- 172.21.15.70:0/1400213739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feac8102d20 0x7feac8197740 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:51.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.578+0000 7feace3bd700 1 -- 172.21.15.70:0/1400213739 >> 172.21.15.70:0/1400213739 conn(0x7feac80776d0 msgr2=0x7feac8103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:51.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.578+0000 7feace3bd700 1 -- 172.21.15.70:0/1400213739 shutdown_connections 2024-03-21T10:27:51.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:51.578+0000 7feace3bd700 1 -- 172.21.15.70:0/1400213739 wait complete. 2024-03-21T10:27:51.578 INFO:teuthology.orchestra.run.smithi070.stderr:dumped monmap epoch 1 2024-03-21T10:27:52.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:52 smithi031 ceph-mon[31673]: from='client.? 172.21.15.70:0/1400213739' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T10:27:52.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:52 smithi070 ceph-mon[36330]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-21T10:27:53.154 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-21T10:27:53.154 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-21T10:27:53.392 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.c/config 2024-03-21T10:27:57.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.720+0000 7fe765660700 1 -- 172.21.15.70:0/1907437301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe74c0035c0 msgr2=0x7fe74c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:57.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.720+0000 7fe765660700 1 --2- 172.21.15.70:0/1907437301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe74c0035c0 0x7fe74c005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe760101bd0 tx=0x7fe75000feb0 comp rx=0 tx=0).stop 2024-03-21T10:27:57.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.720+0000 7fe765660700 1 -- 172.21.15.70:0/1907437301 shutdown_connections 2024-03-21T10:27:57.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.720+0000 7fe765660700 1 --2- 172.21.15.70:0/1907437301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe74c0035c0 0x7fe74c005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:57.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.720+0000 7fe765660700 1 -- 172.21.15.70:0/1907437301 >> 172.21.15.70:0/1907437301 conn(0x7fe7600fd790 msgr2=0x7fe7600ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:57.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.720+0000 7fe765660700 1 -- 172.21.15.70:0/1907437301 shutdown_connections 2024-03-21T10:27:57.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.720+0000 7fe765660700 1 -- 172.21.15.70:0/1907437301 wait complete. 2024-03-21T10:27:57.721 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:57 smithi070 ceph-mon[36330]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:27:57.721 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:57 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T10:27:57.721 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:57 smithi070 ceph-mon[36330]: mon.a calling monitor election 2024-03-21T10:27:57.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:57 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:27:57.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:57 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:27:57.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:57 smithi070 ceph-mon[36330]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:27:57.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:57 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:27:57.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:57 smithi070 ceph-mon[36330]: mon.c calling monitor election 2024-03-21T10:27:57.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:57 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:27:57.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:57 smithi070 ceph-mon[36330]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:27:57.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:57 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:27:57.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:57 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:27:57.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:57 smithi070 ceph-mon[36330]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-21T10:27:57.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:57 smithi070 ceph-mon[36330]: monmap e2: 2 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],c=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T10:27:57.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:57 smithi070 ceph-mon[36330]: fsmap 2024-03-21T10:27:57.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:57 smithi070 ceph-mon[36330]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T10:27:57.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:57 smithi070 ceph-mon[36330]: mgrmap e13: x(active, since 63s) 2024-03-21T10:27:57.723 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:57 smithi070 ceph-mon[36330]: overall HEALTH_OK 2024-03-21T10:27:57.723 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:57 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:57.723 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:57 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:57.723 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:57 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:57.723 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:57 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T10:27:57.723 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:57 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:27:57.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.721+0000 7fe765660700 1 Processor -- start 2024-03-21T10:27:57.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.721+0000 7fe765660700 1 -- start start 2024-03-21T10:27:57.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.722+0000 7fe765660700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe74c0035c0 0x7fe76019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:57.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.722+0000 7fe765660700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe76019fc90 0x7fe760199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:57.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.722+0000 7fe765660700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe760109d30 con 0x7fe74c0035c0 2024-03-21T10:27:57.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.722+0000 7fe765660700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe760109bb0 con 0x7fe76019fc90 2024-03-21T10:27:57.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.722+0000 7fe7567fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe76019fc90 0x7fe760199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:57.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.722+0000 7fe7567fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe76019fc90 0x7fe760199910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35788/0 (socket says 172.21.15.70:35788) 2024-03-21T10:27:57.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.722+0000 7fe7567fc700 1 -- 172.21.15.70:0/3936071997 learned_addr learned my addr 172.21.15.70:0/3936071997 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-03-21T10:27:57.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.722+0000 7fe7567fc700 1 -- 172.21.15.70:0/3936071997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe76019fc90 msgr2=0x7fe760199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-21T10:27:57.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.722+0000 7fe7567fc700 1 -- 172.21.15.70:0/3936071997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe76019fc90 msgr2=0x7fe760199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-21T10:27:57.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.722+0000 7fe7567fc700 1 --2- 172.21.15.70:0/3936071997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe76019fc90 0x7fe760199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-21T10:27:57.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.722+0000 7fe7567fc700 1 --2- 172.21.15.70:0/3936071997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe76019fc90 0x7fe760199910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-21T10:27:57.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.722+0000 7fe75effd700 1 --2- 172.21.15.70:0/3936071997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe74c0035c0 0x7fe76019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:57.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.723+0000 7fe75effd700 1 -- 172.21.15.70:0/3936071997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe76019fc90 msgr2=0x7fe760199910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T10:27:57.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.723+0000 7fe75effd700 1 --2- 172.21.15.70:0/3936071997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe76019fc90 0x7fe760199910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:57.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.723+0000 7fe75effd700 1 -- 172.21.15.70:0/3936071997 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7601a02d0 con 0x7fe74c0035c0 2024-03-21T10:27:57.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.723+0000 7fe75effd700 1 --2- 172.21.15.70:0/3936071997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe74c0035c0 0x7fe76019f750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe7500059d0 tx=0x7fe7500047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:57.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.723+0000 7fe75cff9700 1 -- 172.21.15.70:0/3936071997 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fe750013b00 con 0x7fe74c0035c0 2024-03-21T10:27:57.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.723+0000 7fe765660700 1 -- 172.21.15.70:0/3936071997 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe760199e50 con 0x7fe74c0035c0 2024-03-21T10:27:57.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.724+0000 7fe765660700 1 -- 172.21.15.70:0/3936071997 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76019a3e0 con 0x7fe74c0035c0 2024-03-21T10:27:57.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.724+0000 7fe75cff9700 1 -- 172.21.15.70:0/3936071997 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe750004500 con 0x7fe74c0035c0 2024-03-21T10:27:57.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.724+0000 7fe75cff9700 1 -- 172.21.15.70:0/3936071997 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fe750003e50 con 0x7fe74c0035c0 2024-03-21T10:27:57.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.725+0000 7fe765660700 1 -- 172.21.15.70:0/3936071997 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7601033e0 con 0x7fe74c0035c0 2024-03-21T10:27:57.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.725+0000 7fe75cff9700 1 -- 172.21.15.70:0/3936071997 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7fe750023070 con 0x7fe74c0035c0 2024-03-21T10:27:57.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.725+0000 7fe75cff9700 1 --2- 172.21.15.70:0/3936071997 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe740039160 0x7fe74003b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:27:57.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.725+0000 7fe75cff9700 1 -- 172.21.15.70:0/3936071997 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe750020050 con 0x7fe74c0035c0 2024-03-21T10:27:57.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.726+0000 7fe7567fc700 1 --2- 172.21.15.70:0/3936071997 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe740039160 0x7fe74003b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:27:57.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.727+0000 7fe7567fc700 1 --2- 172.21.15.70:0/3936071997 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe740039160 0x7fe74003b620 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe748009180 tx=0x7fe748008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:27:57.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:57.741+0000 7fe75cff9700 1 -- 172.21.15.70:0/3936071997 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe750031030 con 0x7fe74c0035c0 2024-03-21T10:27:57.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:57 smithi031 ceph-mon[31673]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:27:57.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:57 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T10:27:57.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:57 smithi031 ceph-mon[31673]: mon.a calling monitor election 2024-03-21T10:27:57.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:57 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:27:57.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:57 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:27:57.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:57 smithi031 ceph-mon[31673]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:27:57.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:57 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:27:57.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:57 smithi031 ceph-mon[31673]: mon.c calling monitor election 2024-03-21T10:27:57.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:57 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:27:57.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:57 smithi031 ceph-mon[31673]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:27:57.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:57 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:27:57.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:57 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:27:57.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:57 smithi031 ceph-mon[31673]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-21T10:27:57.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:57 smithi031 ceph-mon[31673]: monmap e2: 2 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],c=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T10:27:57.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:57 smithi031 ceph-mon[31673]: fsmap 2024-03-21T10:27:57.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:57 smithi031 ceph-mon[31673]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T10:27:57.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:57 smithi031 ceph-mon[31673]: mgrmap e13: x(active, since 63s) 2024-03-21T10:27:57.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:57 smithi031 ceph-mon[31673]: overall HEALTH_OK 2024-03-21T10:27:57.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:57 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:57.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:57 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:57.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:57 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:27:57.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:57 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T10:27:57.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:57 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:27:58.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:58.014+0000 7fe765660700 1 -- 172.21.15.70:0/3936071997 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe76019ae40 con 0x7fe74c0035c0 2024-03-21T10:27:58.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:58.015+0000 7fe75cff9700 1 -- 172.21.15.70:0/3936071997 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1011 (secure 0 0 0) 0x7fe750028030 con 0x7fe74c0035c0 2024-03-21T10:27:58.015 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-21T10:27:58.015 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":2,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","modified":"2024-03-21T10:27:52.301670Z","created":"2024-03-21T10:25:36.790162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:3300","nonce":0},{"type":"v1","addr":"172.21.15.70:6789","nonce":0}]},"addr":"172.21.15.70:6789/0","public_addr":"172.21.15.70:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-21T10:27:58.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:58.017+0000 7fe765660700 1 -- 172.21.15.70:0/3936071997 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe740039160 msgr2=0x7fe74003b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:58.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:58.017+0000 7fe765660700 1 --2- 172.21.15.70:0/3936071997 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe740039160 0x7fe74003b620 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe748009180 tx=0x7fe748008040 comp rx=0 tx=0).stop 2024-03-21T10:27:58.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:58.017+0000 7fe765660700 1 -- 172.21.15.70:0/3936071997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe74c0035c0 msgr2=0x7fe76019f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:27:58.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:58.017+0000 7fe765660700 1 --2- 172.21.15.70:0/3936071997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe74c0035c0 0x7fe76019f750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe7500059d0 tx=0x7fe7500047a0 comp rx=0 tx=0).stop 2024-03-21T10:27:58.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:58.017+0000 7fe765660700 1 -- 172.21.15.70:0/3936071997 shutdown_connections 2024-03-21T10:27:58.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:58.017+0000 7fe765660700 1 --2- 172.21.15.70:0/3936071997 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe740039160 0x7fe74003b620 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:58.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:58.017+0000 7fe765660700 1 --2- 172.21.15.70:0/3936071997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe74c0035c0 0x7fe76019f750 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:58.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:58.017+0000 7fe765660700 1 --2- 172.21.15.70:0/3936071997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe76019fc90 0x7fe760199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:27:58.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:58.017+0000 7fe765660700 1 -- 172.21.15.70:0/3936071997 >> 172.21.15.70:0/3936071997 conn(0x7fe7600fd790 msgr2=0x7fe7600feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:27:58.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:58.017+0000 7fe765660700 1 -- 172.21.15.70:0/3936071997 shutdown_connections 2024-03-21T10:27:58.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:27:58.017+0000 7fe765660700 1 -- 172.21.15.70:0/3936071997 wait complete. 2024-03-21T10:27:58.018 INFO:teuthology.orchestra.run.smithi070.stderr:dumped monmap epoch 2 2024-03-21T10:27:58.545 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:58 smithi070 ceph-mon[36330]: Deploying daemon mon.b on smithi040 2024-03-21T10:27:58.545 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:58 smithi070 ceph-mon[36330]: from='client.? 172.21.15.70:0/3936071997' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T10:27:58.546 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:58 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:27:58.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:58 smithi031 ceph-mon[31673]: Deploying daemon mon.b on smithi040 2024-03-21T10:27:58.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:58 smithi031 ceph-mon[31673]: from='client.? 172.21.15.70:0/3936071997' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T10:27:58.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:58 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:27:59.512 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:27:59 smithi031 ceph-mon[31673]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:27:59.512 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:27:59 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:27:59.301+0000 7f8a4cc00700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-21T10:27:59.571 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-21T10:27:59.571 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-21T10:27:59.815 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.c/config 2024-03-21T10:27:59.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:27:59 smithi070 ceph-mon[36330]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:01.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.278+0000 7f927cf66700 1 -- 172.21.15.70:0/925888227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9278102ed0 msgr2=0x7f92781032b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:28:01.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.278+0000 7f927cf66700 1 --2- 172.21.15.70:0/925888227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9278102ed0 0x7f92781032b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9268009910 tx=0x7f9268009c20 comp rx=0 tx=0).stop 2024-03-21T10:28:01.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.278+0000 7f927cf66700 1 -- 172.21.15.70:0/925888227 shutdown_connections 2024-03-21T10:28:01.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.278+0000 7f927cf66700 1 --2- 172.21.15.70:0/925888227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9278102ed0 0x7f92781032b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:01.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.278+0000 7f927cf66700 1 -- 172.21.15.70:0/925888227 >> 172.21.15.70:0/925888227 conn(0x7f92780fc680 msgr2=0x7f92780feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:28:01.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.278+0000 7f927cf66700 1 -- 172.21.15.70:0/925888227 shutdown_connections 2024-03-21T10:28:01.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.279+0000 7f927cf66700 1 -- 172.21.15.70:0/925888227 wait complete. 2024-03-21T10:28:01.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.279+0000 7f927cf66700 1 Processor -- start 2024-03-21T10:28:01.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.279+0000 7f927cf66700 1 -- start start 2024-03-21T10:28:01.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.279+0000 7f927cf66700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9278102ed0 0x7f927819d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:28:01.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.279+0000 7f927cf66700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f927819db80 0x7f92781977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:28:01.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.280+0000 7f927cf66700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927810c320 con 0x7f9278102ed0 2024-03-21T10:28:01.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.280+0000 7f927cf66700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927810c1a0 con 0x7f927819db80 2024-03-21T10:28:01.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.280+0000 7f9275d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f927819db80 0x7f92781977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:28:01.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.280+0000 7f9275d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f927819db80 0x7f92781977c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35794/0 (socket says 172.21.15.70:35794) 2024-03-21T10:28:01.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.280+0000 7f9275d9b700 1 -- 172.21.15.70:0/2990371205 learned_addr learned my addr 172.21.15.70:0/2990371205 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-03-21T10:28:01.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.280+0000 7f9275d9b700 1 -- 172.21.15.70:0/2990371205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f927819db80 msgr2=0x7f92781977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-21T10:28:01.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.280+0000 7f9275d9b700 1 -- 172.21.15.70:0/2990371205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f927819db80 msgr2=0x7f92781977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-21T10:28:01.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.280+0000 7f9275d9b700 1 --2- 172.21.15.70:0/2990371205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f927819db80 0x7f92781977c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-21T10:28:01.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.280+0000 7f9275d9b700 1 --2- 172.21.15.70:0/2990371205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f927819db80 0x7f92781977c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-21T10:28:01.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.280+0000 7f927659c700 1 --2- 172.21.15.70:0/2990371205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9278102ed0 0x7f927819d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:28:01.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.281+0000 7f927659c700 1 -- 172.21.15.70:0/2990371205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f927819db80 msgr2=0x7f92781977c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T10:28:01.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.281+0000 7f927659c700 1 --2- 172.21.15.70:0/2990371205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f927819db80 0x7f92781977c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:01.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.281+0000 7f927659c700 1 -- 172.21.15.70:0/2990371205 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f927819e1c0 con 0x7f9278102ed0 2024-03-21T10:28:01.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.281+0000 7f927659c700 1 --2- 172.21.15.70:0/2990371205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9278102ed0 0x7f927819d640 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9268014000 tx=0x7f9268005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:28:01.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.281+0000 7f92677fe700 1 -- 172.21.15.70:0/2990371205 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f9268005dc0 con 0x7f9278102ed0 2024-03-21T10:28:01.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.281+0000 7f927cf66700 1 -- 172.21.15.70:0/2990371205 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9278197d60 con 0x7f9278102ed0 2024-03-21T10:28:01.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.281+0000 7f927cf66700 1 -- 172.21.15.70:0/2990371205 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92781982c0 con 0x7f9278102ed0 2024-03-21T10:28:01.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.281+0000 7f92677fe700 1 -- 172.21.15.70:0/2990371205 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f926800be40 con 0x7f9278102ed0 2024-03-21T10:28:01.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.281+0000 7f92677fe700 1 -- 172.21.15.70:0/2990371205 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f926800f460 con 0x7f9278102ed0 2024-03-21T10:28:01.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.282+0000 7f927cf66700 1 -- 172.21.15.70:0/2990371205 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f927806abb0 con 0x7f9278102ed0 2024-03-21T10:28:01.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.283+0000 7f92677fe700 1 -- 172.21.15.70:0/2990371205 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7f926802a070 con 0x7f9278102ed0 2024-03-21T10:28:01.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.283+0000 7f92677fe700 1 --2- 172.21.15.70:0/2990371205 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f9260038fc0 0x7f926003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:28:01.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.283+0000 7f92677fe700 1 -- 172.21.15.70:0/2990371205 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f926801e030 con 0x7f9278102ed0 2024-03-21T10:28:01.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.284+0000 7f9275d9b700 1 --2- 172.21.15.70:0/2990371205 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f9260038fc0 0x7f926003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:28:01.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.285+0000 7f9275d9b700 1 --2- 172.21.15.70:0/2990371205 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f9260038fc0 0x7f926003b480 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f926c009180 tx=0x7f926c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:28:01.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.300+0000 7f92677fe700 1 -- 172.21.15.70:0/2990371205 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9268013070 con 0x7f9278102ed0 2024-03-21T10:28:01.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.310+0000 7f92677fe700 1 -- 172.21.15.70:0/2990371205 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92680227c0 con 0x7f9278102ed0 2024-03-21T10:28:01.334 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:01 smithi040 ceph-mon[36154]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-21T10:28:01.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:01.580+0000 7f927cf66700 1 -- 172.21.15.70:0/2990371205 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9278198540 con 0x7f9278102ed0 2024-03-21T10:28:01.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:01 smithi040 bash[36134]: 5897a61780c0a15ddb1d0ae2703d1b63c5db2466a06c05ec1a434fb317463b41 2024-03-21T10:28:01.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:01 smithi040 systemd[1]: Started Ceph mon.b for 3847408c-e76d-11ee-95cd-87774f69a715. 2024-03-21T10:28:06.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:06.330+0000 7f92677fe700 1 -- 172.21.15.70:0/2990371205 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1275 (secure 0 0 0) 0x7f9268035090 con 0x7f9278102ed0 2024-03-21T10:28:06.330 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-21T10:28:06.330 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":3,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","modified":"2024-03-21T10:28:01.308715Z","created":"2024-03-21T10:25:36.790162Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:3300","nonce":0},{"type":"v1","addr":"172.21.15.70:6789","nonce":0}]},"addr":"172.21.15.70:6789/0","public_addr":"172.21.15.70:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-21T10:28:06.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:06.332+0000 7f927cf66700 1 -- 172.21.15.70:0/2990371205 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f9260038fc0 msgr2=0x7f926003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:28:06.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:06.332+0000 7f927cf66700 1 --2- 172.21.15.70:0/2990371205 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f9260038fc0 0x7f926003b480 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f926c009180 tx=0x7f926c008040 comp rx=0 tx=0).stop 2024-03-21T10:28:06.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:06.332+0000 7f927cf66700 1 -- 172.21.15.70:0/2990371205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9278102ed0 msgr2=0x7f927819d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:28:06.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:06.332+0000 7f927cf66700 1 --2- 172.21.15.70:0/2990371205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9278102ed0 0x7f927819d640 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9268014000 tx=0x7f9268005030 comp rx=0 tx=0).stop 2024-03-21T10:28:06.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:06.332+0000 7f927cf66700 1 -- 172.21.15.70:0/2990371205 shutdown_connections 2024-03-21T10:28:06.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:06.332+0000 7f927cf66700 1 --2- 172.21.15.70:0/2990371205 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f9260038fc0 0x7f926003b480 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:06.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:06.332+0000 7f927cf66700 1 --2- 172.21.15.70:0/2990371205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9278102ed0 0x7f927819d640 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:06.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:06.332+0000 7f927cf66700 1 --2- 172.21.15.70:0/2990371205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f927819db80 0x7f92781977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:06.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:06.332+0000 7f927cf66700 1 -- 172.21.15.70:0/2990371205 >> 172.21.15.70:0/2990371205 conn(0x7f92780fc680 msgr2=0x7f9278105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:28:06.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:06.332+0000 7f927cf66700 1 -- 172.21.15.70:0/2990371205 shutdown_connections 2024-03-21T10:28:06.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:06.332+0000 7f927cf66700 1 -- 172.21.15.70:0/2990371205 wait complete. 2024-03-21T10:28:06.333 INFO:teuthology.orchestra.run.smithi070.stderr:dumped monmap epoch 3 2024-03-21T10:28:06.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:06 smithi031 ceph-mon[31673]: mon.c calling monitor election 2024-03-21T10:28:06.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:06 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T10:28:06.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:06 smithi031 ceph-mon[31673]: mon.a calling monitor election 2024-03-21T10:28:06.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:06 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:28:06.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:06 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:28:06.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:06 smithi031 ceph-mon[31673]: from='client.? 172.21.15.70:0/2990371205' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T10:28:06.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:06 smithi031 ceph-mon[31673]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:06.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:06 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:28:06.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:06 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:28:06.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:06 smithi031 ceph-mon[31673]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:06.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:06 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:28:06.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:06 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:28:06.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:06 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:28:06.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:06 smithi031 ceph-mon[31673]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-21T10:28:06.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:06 smithi031 ceph-mon[31673]: monmap e3: 3 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],b=[v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0],c=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T10:28:06.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:06 smithi031 ceph-mon[31673]: fsmap 2024-03-21T10:28:06.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:06 smithi031 ceph-mon[31673]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T10:28:06.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:06 smithi031 ceph-mon[31673]: mgrmap e13: x(active, since 72s) 2024-03-21T10:28:06.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:06 smithi031 ceph-mon[31673]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-21T10:28:06.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:06 smithi031 ceph-mon[31673]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-21T10:28:06.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:06 smithi031 ceph-mon[31673]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-21T10:28:06.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:06 smithi031 ceph-mon[31673]: mon.b (rank 2) addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] is down (out of quorum) 2024-03-21T10:28:06.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:06 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:06.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:06 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:06.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:06 smithi070 ceph-mon[36330]: mon.c calling monitor election 2024-03-21T10:28:06.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:06 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T10:28:06.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:06 smithi070 ceph-mon[36330]: mon.a calling monitor election 2024-03-21T10:28:06.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:06 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:28:06.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:06 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:28:06.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:06 smithi070 ceph-mon[36330]: from='client.? 172.21.15.70:0/2990371205' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T10:28:06.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:06 smithi070 ceph-mon[36330]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:06.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:06 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:28:06.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:06 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:28:06.827 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:06 smithi070 ceph-mon[36330]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:06.827 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:06 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:28:06.827 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:06 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:28:06.827 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:06 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:28:06.827 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:06 smithi070 ceph-mon[36330]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-21T10:28:06.827 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:06 smithi070 ceph-mon[36330]: monmap e3: 3 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],b=[v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0],c=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T10:28:06.827 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:06 smithi070 ceph-mon[36330]: fsmap 2024-03-21T10:28:06.827 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:06 smithi070 ceph-mon[36330]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T10:28:06.827 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:06 smithi070 ceph-mon[36330]: mgrmap e13: x(active, since 72s) 2024-03-21T10:28:06.827 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:06 smithi070 ceph-mon[36330]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-21T10:28:06.827 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:06 smithi070 ceph-mon[36330]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-21T10:28:06.827 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:06 smithi070 ceph-mon[36330]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-21T10:28:06.827 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:06 smithi070 ceph-mon[36330]: mon.b (rank 2) addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] is down (out of quorum) 2024-03-21T10:28:06.827 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:06 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:06.828 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:06 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:07.241 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-21T10:28:07.241 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-21T10:28:07.551 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:07 smithi070 ceph-mon[36330]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:07.551 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:07 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:07.551 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:07 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:07.551 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:07 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:28:07.551 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:07 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:28:07.730 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:28:07.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:07 smithi031 ceph-mon[31673]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:07.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:07 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:07.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:07 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:07.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:07 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:28:07.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:07 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:28:08.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: mon.c calling monitor election 2024-03-21T10:28:08.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T10:28:08.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: mon.a calling monitor election 2024-03-21T10:28:08.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:28:08.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:28:08.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: from='client.? 172.21.15.70:0/2990371205' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T10:28:08.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:08.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:28:08.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:28:08.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:08.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:28:08.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:28:08.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:28:08.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-21T10:28:08.685 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: monmap e3: 3 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],b=[v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0],c=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T10:28:08.685 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: fsmap 2024-03-21T10:28:08.685 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T10:28:08.685 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: mgrmap e13: x(active, since 72s) 2024-03-21T10:28:08.685 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-21T10:28:08.685 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-21T10:28:08.685 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-21T10:28:08.685 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: mon.b (rank 2) addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] is down (out of quorum) 2024-03-21T10:28:08.685 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:08.685 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:08.685 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:08.685 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:08.685 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:08.686 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:28:08.686 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:08 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:28:09.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:09 smithi031 ceph-mon[31673]: mon.b calling monitor election 2024-03-21T10:28:09.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:09 smithi031 ceph-mon[31673]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:09.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:09 smithi031 ceph-mon[31673]: Updating smithi031:/etc/ceph/ceph.conf 2024-03-21T10:28:09.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:09 smithi031 ceph-mon[31673]: Updating smithi040:/etc/ceph/ceph.conf 2024-03-21T10:28:09.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:09 smithi031 ceph-mon[31673]: Updating smithi070:/etc/ceph/ceph.conf 2024-03-21T10:28:09.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:09 smithi031 ceph-mon[31673]: mon.b calling monitor election 2024-03-21T10:28:09.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:09 smithi031 ceph-mon[31673]: mon.c calling monitor election 2024-03-21T10:28:09.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:09 smithi031 ceph-mon[31673]: mon.a calling monitor election 2024-03-21T10:28:09.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:09 smithi031 ceph-mon[31673]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-21T10:28:09.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:09 smithi031 ceph-mon[31673]: monmap e3: 3 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],b=[v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0],c=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T10:28:09.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:09 smithi031 ceph-mon[31673]: fsmap 2024-03-21T10:28:09.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:09 smithi031 ceph-mon[31673]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T10:28:09.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:09 smithi031 ceph-mon[31673]: mgrmap e13: x(active, since 74s) 2024-03-21T10:28:09.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:09 smithi031 ceph-mon[31673]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-21T10:28:09.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:09 smithi031 ceph-mon[31673]: Cluster is now healthy 2024-03-21T10:28:09.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:09 smithi031 ceph-mon[31673]: overall HEALTH_OK 2024-03-21T10:28:09.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:09 smithi031 ceph-mon[31673]: Updating smithi040:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T10:28:09.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:09 smithi031 ceph-mon[31673]: Updating smithi070:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T10:28:09.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:09.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:09.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:09.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:09.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:09.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:09 smithi040 ceph-mon[36154]: mon.b calling monitor election 2024-03-21T10:28:09.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:09 smithi040 ceph-mon[36154]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:09.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:09 smithi040 ceph-mon[36154]: Updating smithi031:/etc/ceph/ceph.conf 2024-03-21T10:28:09.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:09 smithi040 ceph-mon[36154]: Updating smithi040:/etc/ceph/ceph.conf 2024-03-21T10:28:09.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:09 smithi040 ceph-mon[36154]: Updating smithi070:/etc/ceph/ceph.conf 2024-03-21T10:28:09.435 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:09 smithi040 ceph-mon[36154]: mon.b calling monitor election 2024-03-21T10:28:09.435 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:09 smithi040 ceph-mon[36154]: mon.c calling monitor election 2024-03-21T10:28:09.435 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:09 smithi040 ceph-mon[36154]: mon.a calling monitor election 2024-03-21T10:28:09.435 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:09 smithi040 ceph-mon[36154]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-21T10:28:09.435 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:09 smithi040 ceph-mon[36154]: monmap e3: 3 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],b=[v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0],c=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T10:28:09.435 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:09 smithi040 ceph-mon[36154]: fsmap 2024-03-21T10:28:09.435 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:09 smithi040 ceph-mon[36154]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T10:28:09.435 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:09 smithi040 ceph-mon[36154]: mgrmap e13: x(active, since 74s) 2024-03-21T10:28:09.435 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:09 smithi040 ceph-mon[36154]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-21T10:28:09.435 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:09 smithi040 ceph-mon[36154]: Cluster is now healthy 2024-03-21T10:28:09.435 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:09 smithi040 ceph-mon[36154]: overall HEALTH_OK 2024-03-21T10:28:09.436 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:09 smithi040 ceph-mon[36154]: Updating smithi040:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T10:28:09.436 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:09 smithi040 ceph-mon[36154]: Updating smithi070:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T10:28:09.436 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:09.436 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:09.436 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:09.436 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:09.436 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:09.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:09 smithi070 ceph-mon[36330]: mon.b calling monitor election 2024-03-21T10:28:09.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:09 smithi070 ceph-mon[36330]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:09.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:09 smithi070 ceph-mon[36330]: Updating smithi031:/etc/ceph/ceph.conf 2024-03-21T10:28:09.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:09 smithi070 ceph-mon[36330]: Updating smithi040:/etc/ceph/ceph.conf 2024-03-21T10:28:09.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:09 smithi070 ceph-mon[36330]: Updating smithi070:/etc/ceph/ceph.conf 2024-03-21T10:28:09.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:09 smithi070 ceph-mon[36330]: mon.b calling monitor election 2024-03-21T10:28:09.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:09 smithi070 ceph-mon[36330]: mon.c calling monitor election 2024-03-21T10:28:09.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:09 smithi070 ceph-mon[36330]: mon.a calling monitor election 2024-03-21T10:28:09.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:09 smithi070 ceph-mon[36330]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-21T10:28:09.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:09 smithi070 ceph-mon[36330]: monmap e3: 3 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],b=[v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0],c=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T10:28:09.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:09 smithi070 ceph-mon[36330]: fsmap 2024-03-21T10:28:09.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:09 smithi070 ceph-mon[36330]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T10:28:09.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:09 smithi070 ceph-mon[36330]: mgrmap e13: x(active, since 74s) 2024-03-21T10:28:09.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:09 smithi070 ceph-mon[36330]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-21T10:28:09.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:09 smithi070 ceph-mon[36330]: Cluster is now healthy 2024-03-21T10:28:09.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:09 smithi070 ceph-mon[36330]: overall HEALTH_OK 2024-03-21T10:28:09.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:09 smithi070 ceph-mon[36330]: Updating smithi040:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T10:28:09.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:09 smithi070 ceph-mon[36330]: Updating smithi070:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T10:28:09.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:09.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:09.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:09.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:09.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:09.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.932+0000 7f8cc7800700 1 -- 172.21.15.31:0/1132456038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cc00745c0 msgr2=0x7f8cc00749a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:28:09.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.932+0000 7f8cc7800700 1 --2- 172.21.15.31:0/1132456038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cc00745c0 0x7f8cc00749a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4009910 tx=0x7f8cb4009c20 comp rx=0 tx=0).stop 2024-03-21T10:28:09.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.932+0000 7f8cc7800700 1 -- 172.21.15.31:0/1132456038 shutdown_connections 2024-03-21T10:28:09.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.932+0000 7f8cc7800700 1 --2- 172.21.15.31:0/1132456038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cc00745c0 0x7f8cc00749a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:09.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.932+0000 7f8cc7800700 1 -- 172.21.15.31:0/1132456038 >> 172.21.15.31:0/1132456038 conn(0x7f8cc00fd6a0 msgr2=0x7f8cc00ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:28:09.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.932+0000 7f8cc7800700 1 -- 172.21.15.31:0/1132456038 shutdown_connections 2024-03-21T10:28:09.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.933+0000 7f8cc7800700 1 -- 172.21.15.31:0/1132456038 wait complete. 2024-03-21T10:28:09.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.934+0000 7f8cc7800700 1 Processor -- start 2024-03-21T10:28:09.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.934+0000 7f8cc7800700 1 -- start start 2024-03-21T10:28:09.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.934+0000 7f8cc7800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cc00745c0 0x7f8cc019f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:28:09.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.934+0000 7f8cc7800700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cc019fd00 0x7f8cc0199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:28:09.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.934+0000 7f8cc7800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cc0199ec0 0x7f8cc019a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:28:09.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.934+0000 7f8cc7800700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc010e420 con 0x7f8cc019fd00 2024-03-21T10:28:09.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.934+0000 7f8cc7800700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc010e2a0 con 0x7f8cc0199ec0 2024-03-21T10:28:09.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.934+0000 7f8cc7800700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc010e5a0 con 0x7f8cc00745c0 2024-03-21T10:28:09.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.935+0000 7f8cc4e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cc00745c0 0x7f8cc019f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:28:09.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.935+0000 7f8cbfe00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cc019fd00 0x7f8cc0199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:28:09.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.935+0000 7f8cc4e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cc00745c0 0x7f8cc019f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:54534/0 (socket says 172.21.15.31:54534) 2024-03-21T10:28:09.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.935+0000 7f8cc4e00700 1 -- 172.21.15.31:0/2572438083 learned_addr learned my addr 172.21.15.31:0/2572438083 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:28:09.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.935+0000 7f8cc5800700 1 --2- 172.21.15.31:0/2572438083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cc0199ec0 0x7f8cc019a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:28:09.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.935+0000 7f8cbfe00700 1 -- 172.21.15.31:0/2572438083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cc00745c0 msgr2=0x7f8cc019f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:28:09.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.935+0000 7f8cbfe00700 1 --2- 172.21.15.31:0/2572438083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cc00745c0 0x7f8cc019f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:09.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.935+0000 7f8cbfe00700 1 -- 172.21.15.31:0/2572438083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cc0199ec0 msgr2=0x7f8cc019a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:28:09.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.935+0000 7f8cbfe00700 1 --2- 172.21.15.31:0/2572438083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cc0199ec0 0x7f8cc019a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:09.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.935+0000 7f8cbfe00700 1 -- 172.21.15.31:0/2572438083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc01a6690 con 0x7f8cc019fd00 2024-03-21T10:28:09.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.936+0000 7f8cbfe00700 1 --2- 172.21.15.31:0/2572438083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cc019fd00 0x7f8cc0199980 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8cac00b550 tx=0x7f8cac00b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:28:09.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.936+0000 7f8cbd600700 1 -- 172.21.15.31:0/2572438083 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac004230 con 0x7f8cc019fd00 2024-03-21T10:28:09.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.936+0000 7f8cbd600700 1 -- 172.21.15.31:0/2572438083 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8cac0043d0 con 0x7f8cc019fd00 2024-03-21T10:28:09.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.936+0000 7f8cc7800700 1 -- 172.21.15.31:0/2572438083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc01a68c0 con 0x7f8cc019fd00 2024-03-21T10:28:09.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.936+0000 7f8cbd600700 1 -- 172.21.15.31:0/2572438083 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac011b20 con 0x7f8cc019fd00 2024-03-21T10:28:09.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.936+0000 7f8cc7800700 1 -- 172.21.15.31:0/2572438083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc01a6e00 con 0x7f8cc019fd00 2024-03-21T10:28:09.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.937+0000 7f8cc7800700 1 -- 172.21.15.31:0/2572438083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cc00637c0 con 0x7f8cc019fd00 2024-03-21T10:28:09.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.938+0000 7f8cbd600700 1 -- 172.21.15.31:0/2572438083 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7f8cac011d00 con 0x7f8cc019fd00 2024-03-21T10:28:09.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.938+0000 7f8cbd600700 1 --2- 172.21.15.31:0/2572438083 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f8ca80396b0 0x7f8ca803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:28:09.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.939+0000 7f8cbd600700 1 -- 172.21.15.31:0/2572438083 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8cac04bec0 con 0x7f8cc019fd00 2024-03-21T10:28:09.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.939+0000 7f8cc4e00700 1 --2- 172.21.15.31:0/2572438083 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f8ca80396b0 0x7f8ca803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:28:09.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.940+0000 7f8cc4e00700 1 --2- 172.21.15.31:0/2572438083 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f8ca80396b0 0x7f8ca803bb70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4009750 tx=0x7f8cb400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:28:09.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:09.944+0000 7f8cbd600700 1 -- 172.21.15.31:0/2572438083 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8cac0204e0 con 0x7f8cc019fd00 2024-03-21T10:28:10.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:10.178+0000 7f8cc7800700 1 -- 172.21.15.31:0/2572438083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f8cc01a70f0 con 0x7f8cc019fd00 2024-03-21T10:28:10.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:10.178+0000 7f8cbd600700 1 -- 172.21.15.31:0/2572438083 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+271 (secure 0 0 0) 0x7f8cac0151c0 con 0x7f8cc019fd00 2024-03-21T10:28:10.179 INFO:teuthology.orchestra.run.smithi031.stdout:# minimal ceph.conf for 3847408c-e76d-11ee-95cd-87774f69a715 2024-03-21T10:28:10.179 INFO:teuthology.orchestra.run.smithi031.stdout:[global] 2024-03-21T10:28:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: fsid = 3847408c-e76d-11ee-95cd-87774f69a715 2024-03-21T10:28:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: mon_host = [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-03-21T10:28:10.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:10.181+0000 7f8cc7800700 1 -- 172.21.15.31:0/2572438083 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f8ca80396b0 msgr2=0x7f8ca803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:28:10.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:10.181+0000 7f8cc7800700 1 --2- 172.21.15.31:0/2572438083 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f8ca80396b0 0x7f8ca803bb70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4009750 tx=0x7f8cb400b540 comp rx=0 tx=0).stop 2024-03-21T10:28:10.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:10.181+0000 7f8cc7800700 1 -- 172.21.15.31:0/2572438083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cc019fd00 msgr2=0x7f8cc0199980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:28:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:10.181+0000 7f8cc7800700 1 --2- 172.21.15.31:0/2572438083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cc019fd00 0x7f8cc0199980 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8cac00b550 tx=0x7f8cac00b860 comp rx=0 tx=0).stop 2024-03-21T10:28:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:10.181+0000 7f8cc7800700 1 -- 172.21.15.31:0/2572438083 shutdown_connections 2024-03-21T10:28:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:10.181+0000 7f8cc7800700 1 --2- 172.21.15.31:0/2572438083 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f8ca80396b0 0x7f8ca803bb70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:10.181+0000 7f8cc7800700 1 --2- 172.21.15.31:0/2572438083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cc0199ec0 0x7f8cc019a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:10.181+0000 7f8cc7800700 1 --2- 172.21.15.31:0/2572438083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cc00745c0 0x7f8cc019f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:10.181+0000 7f8cc7800700 1 --2- 172.21.15.31:0/2572438083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cc019fd00 0x7f8cc0199980 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:10.181+0000 7f8cc7800700 1 -- 172.21.15.31:0/2572438083 >> 172.21.15.31:0/2572438083 conn(0x7f8cc00fd6a0 msgr2=0x7f8cc0105f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:28:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:10.181+0000 7f8cc7800700 1 -- 172.21.15.31:0/2572438083 shutdown_connections 2024-03-21T10:28:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:10.181+0000 7f8cc7800700 1 -- 172.21.15.31:0/2572438083 wait complete. 2024-03-21T10:28:10.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:10 smithi040 ceph-mon[36154]: Updating smithi031:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T10:28:10.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:10 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:28:10.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:10 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:10.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:10 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:10.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:10 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:10.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:10 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:10.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:10 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:10.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:10 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:10.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:10 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:10.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:10 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T10:28:10.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:10 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T10:28:10.435 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:10 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:10.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:10 smithi031 ceph-mon[31673]: Updating smithi031:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T10:28:10.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:10 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:28:10.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:10 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:10.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:10 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:10.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:10 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:10.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:10 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:10.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:10 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:10.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:10 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:10.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:10 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:10.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:10 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T10:28:10.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:10 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T10:28:10.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:10 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:10.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:10 smithi070 ceph-mon[36330]: Updating smithi031:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T10:28:10.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:10 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:28:10.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:10 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:10.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:10 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:10.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:10 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:10.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:10 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:10.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:10 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:10.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:10 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:10.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:10 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:10.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:10 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T10:28:10.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:10 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T10:28:10.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:10 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:10.938 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-21T10:28:10.939 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-21T10:28:10.939 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.conf 2024-03-21T10:28:11.031 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-21T10:28:11.031 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-21T10:28:11.161 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-21T10:28:11.161 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.conf 2024-03-21T10:28:11.199 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-21T10:28:11.200 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-21T10:28:11.274 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-21T10:28:11.275 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.conf 2024-03-21T10:28:11.309 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-21T10:28:11.309 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-21T10:28:11.379 INFO:tasks.cephadm:Adding mgr.x on smithi031 2024-03-21T10:28:11.379 INFO:tasks.cephadm:Adding mgr.y on smithi040 2024-03-21T10:28:11.379 INFO:tasks.cephadm:Adding mgr.z on smithi070 2024-03-21T10:28:11.379 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph orch apply mgr '3;smithi031=x;smithi040=y;smithi070=z' 2024-03-21T10:28:11.410 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:11 smithi031 ceph-mon[31673]: Reconfiguring mon.a (unknown last config time)... 2024-03-21T10:28:11.410 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:11 smithi031 ceph-mon[31673]: Reconfiguring daemon mon.a on smithi031 2024-03-21T10:28:11.410 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:11 smithi031 ceph-mon[31673]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:11.410 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:11 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2572438083' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:11.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:11 smithi040 ceph-mon[36154]: Reconfiguring mon.a (unknown last config time)... 2024-03-21T10:28:11.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:11 smithi040 ceph-mon[36154]: Reconfiguring daemon mon.a on smithi031 2024-03-21T10:28:11.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:11 smithi040 ceph-mon[36154]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:11.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:11 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2572438083' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:11.445 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:11 smithi070 ceph-mon[36330]: Reconfiguring mon.a (unknown last config time)... 2024-03-21T10:28:11.445 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:11 smithi070 ceph-mon[36330]: Reconfiguring daemon mon.a on smithi031 2024-03-21T10:28:11.445 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:11 smithi070 ceph-mon[36330]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:11.445 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:11 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2572438083' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:11.651 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.c/config 2024-03-21T10:28:13.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.320+0000 7f65f1146700 1 -- 172.21.15.70:0/1540049742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ec100bf0 msgr2=0x7f65ec101010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:28:13.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.320+0000 7f65f1146700 1 --2- 172.21.15.70:0/1540049742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ec100bf0 0x7f65ec101010 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f65dc009910 tx=0x7f65dc009c20 comp rx=0 tx=0).stop 2024-03-21T10:28:13.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.320+0000 7f65f1146700 1 -- 172.21.15.70:0/1540049742 shutdown_connections 2024-03-21T10:28:13.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.320+0000 7f65f1146700 1 --2- 172.21.15.70:0/1540049742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ec100bf0 0x7f65ec101010 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:13.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.320+0000 7f65f1146700 1 -- 172.21.15.70:0/1540049742 >> 172.21.15.70:0/1540049742 conn(0x7f65ec0fc170 msgr2=0x7f65ec0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:28:13.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.320+0000 7f65f1146700 1 -- 172.21.15.70:0/1540049742 shutdown_connections 2024-03-21T10:28:13.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.320+0000 7f65f1146700 1 -- 172.21.15.70:0/1540049742 wait complete. 2024-03-21T10:28:13.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.321+0000 7f65f1146700 1 Processor -- start 2024-03-21T10:28:13.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.321+0000 7f65f1146700 1 -- start start 2024-03-21T10:28:13.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.321+0000 7f65f1146700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ec100bf0 0x7f65ec06aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:28:13.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.321+0000 7f65f1146700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65ec19a260 0x7f65ec19c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:28:13.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.321+0000 7f65f1146700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65ec19cbe0 0x7f65ec19f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:28:13.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.321+0000 7f65f1146700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65ec106c30 con 0x7f65ec100bf0 2024-03-21T10:28:13.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.321+0000 7f65f1146700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65ec106ab0 con 0x7f65ec19cbe0 2024-03-21T10:28:13.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.321+0000 7f65f1146700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65ec106db0 con 0x7f65ec19a260 2024-03-21T10:28:13.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.321+0000 7f65ea59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65ec19a260 0x7f65ec19c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:28:13.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.321+0000 7f65ea59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65ec19a260 0x7f65ec19c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33138/0 (socket says 172.21.15.70:33138) 2024-03-21T10:28:13.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.321+0000 7f65ea59c700 1 -- 172.21.15.70:0/4223361315 learned_addr learned my addr 172.21.15.70:0/4223361315 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-03-21T10:28:13.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.321+0000 7f65eb59e700 1 --2- 172.21.15.70:0/4223361315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65ec19cbe0 0x7f65ec19f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:28:13.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.322+0000 7f65ead9d700 1 --2- 172.21.15.70:0/4223361315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ec100bf0 0x7f65ec06aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:28:13.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.322+0000 7f65ea59c700 1 -- 172.21.15.70:0/4223361315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65ec19cbe0 msgr2=0x7f65ec19f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:28:13.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.322+0000 7f65ea59c700 1 --2- 172.21.15.70:0/4223361315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65ec19cbe0 0x7f65ec19f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:13.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.322+0000 7f65ea59c700 1 -- 172.21.15.70:0/4223361315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ec100bf0 msgr2=0x7f65ec06aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:28:13.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.322+0000 7f65ea59c700 1 --2- 172.21.15.70:0/4223361315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ec100bf0 0x7f65ec06aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:13.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.322+0000 7f65ea59c700 1 -- 172.21.15.70:0/4223361315 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65ec19f940 con 0x7f65ec19a260 2024-03-21T10:28:13.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.322+0000 7f65ea59c700 1 --2- 172.21.15.70:0/4223361315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65ec19a260 0x7f65ec19c650 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f65e000b780 tx=0x7f65e000ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:28:13.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.322+0000 7f65d3fff700 1 -- 172.21.15.70:0/4223361315 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e0004230 con 0x7f65ec19a260 2024-03-21T10:28:13.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.322+0000 7f65d3fff700 1 -- 172.21.15.70:0/4223361315 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f65e00043d0 con 0x7f65ec19a260 2024-03-21T10:28:13.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.322+0000 7f65f1146700 1 -- 172.21.15.70:0/4223361315 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65ec19fc30 con 0x7f65ec19a260 2024-03-21T10:28:13.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.322+0000 7f65d3fff700 1 -- 172.21.15.70:0/4223361315 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e0011c90 con 0x7f65ec19a260 2024-03-21T10:28:13.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.322+0000 7f65f1146700 1 -- 172.21.15.70:0/4223361315 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65ec1a0120 con 0x7f65ec19a260 2024-03-21T10:28:13.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.323+0000 7f65d3fff700 1 -- 172.21.15.70:0/4223361315 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7f65e001a400 con 0x7f65ec19a260 2024-03-21T10:28:13.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.323+0000 7f65f1146700 1 -- 172.21.15.70:0/4223361315 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65ec0637c0 con 0x7f65ec19a260 2024-03-21T10:28:13.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.323+0000 7f65d3fff700 1 --2- 172.21.15.70:0/4223361315 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f65d40396b0 0x7f65d403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:28:13.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.323+0000 7f65d3fff700 1 -- 172.21.15.70:0/4223361315 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f65e004c190 con 0x7f65ec19a260 2024-03-21T10:28:13.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.324+0000 7f65ead9d700 1 --2- 172.21.15.70:0/4223361315 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f65d40396b0 0x7f65d403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:28:13.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.325+0000 7f65ead9d700 1 --2- 172.21.15.70:0/4223361315 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f65d40396b0 0x7f65d403bb70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f65dc005330 tx=0x7f65dc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:28:13.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.328+0000 7f65d3fff700 1 -- 172.21.15.70:0/4223361315 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f65e0020500 con 0x7f65ec19a260 2024-03-21T10:28:13.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.532+0000 7f65f1146700 1 -- 172.21.15.70:0/4223361315 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi031=x;smithi040=y;smithi070=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f65ec062660 con 0x7f65d40396b0 2024-03-21T10:28:13.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.546+0000 7f65d3fff700 1 -- 172.21.15.70:0/4223361315 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f65ec062660 con 0x7f65d40396b0 2024-03-21T10:28:13.547 INFO:teuthology.orchestra.run.smithi070.stdout:Scheduled mgr update... 2024-03-21T10:28:13.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.548+0000 7f65f1146700 1 -- 172.21.15.70:0/4223361315 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f65d40396b0 msgr2=0x7f65d403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:28:13.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.548+0000 7f65f1146700 1 --2- 172.21.15.70:0/4223361315 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f65d40396b0 0x7f65d403bb70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f65dc005330 tx=0x7f65dc005fb0 comp rx=0 tx=0).stop 2024-03-21T10:28:13.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.548+0000 7f65f1146700 1 -- 172.21.15.70:0/4223361315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65ec19a260 msgr2=0x7f65ec19c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:28:13.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.548+0000 7f65f1146700 1 --2- 172.21.15.70:0/4223361315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65ec19a260 0x7f65ec19c650 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f65e000b780 tx=0x7f65e000ba90 comp rx=0 tx=0).stop 2024-03-21T10:28:13.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.549+0000 7f65f1146700 1 -- 172.21.15.70:0/4223361315 shutdown_connections 2024-03-21T10:28:13.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.549+0000 7f65f1146700 1 --2- 172.21.15.70:0/4223361315 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f65d40396b0 0x7f65d403bb70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:13.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.549+0000 7f65f1146700 1 --2- 172.21.15.70:0/4223361315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65ec19cbe0 0x7f65ec19f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:13.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.549+0000 7f65f1146700 1 --2- 172.21.15.70:0/4223361315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ec100bf0 0x7f65ec06aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:13.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.549+0000 7f65f1146700 1 --2- 172.21.15.70:0/4223361315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65ec19a260 0x7f65ec19c650 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:13.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.549+0000 7f65f1146700 1 -- 172.21.15.70:0/4223361315 >> 172.21.15.70:0/4223361315 conn(0x7f65ec0fc170 msgr2=0x7f65ec0fce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:28:13.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.549+0000 7f65f1146700 1 -- 172.21.15.70:0/4223361315 shutdown_connections 2024-03-21T10:28:13.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:28:13.549+0000 7f65f1146700 1 -- 172.21.15.70:0/4223361315 wait complete. 2024-03-21T10:28:13.560 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:13 smithi070 ceph-mon[36330]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:13.560 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:13 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:13.560 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:13 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:13.560 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:13 smithi070 ceph-mon[36330]: Reconfiguring mon.b (monmap changed)... 2024-03-21T10:28:13.560 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:13 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T10:28:13.560 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:13 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T10:28:13.560 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:13 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:13.560 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:13 smithi070 ceph-mon[36330]: Reconfiguring daemon mon.b on smithi040 2024-03-21T10:28:13.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:13 smithi031 ceph-mon[31673]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:13.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:13 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:13.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:13 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:13.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:13 smithi031 ceph-mon[31673]: Reconfiguring mon.b (monmap changed)... 2024-03-21T10:28:13.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:13 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T10:28:13.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:13 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T10:28:13.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:13 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:13.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:13 smithi031 ceph-mon[31673]: Reconfiguring daemon mon.b on smithi040 2024-03-21T10:28:13.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:13 smithi040 ceph-mon[36154]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:13.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:13 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:13.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:13 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:13.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:13 smithi040 ceph-mon[36154]: Reconfiguring mon.b (monmap changed)... 2024-03-21T10:28:13.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:13 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T10:28:13.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:13 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T10:28:13.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:13 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:13.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:13 smithi040 ceph-mon[36154]: Reconfiguring daemon mon.b on smithi040 2024-03-21T10:28:14.217 DEBUG:teuthology.orchestra.run.smithi040:mgr.y> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mgr.y.service 2024-03-21T10:28:14.220 DEBUG:teuthology.orchestra.run.smithi070:mgr.z> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mgr.z.service 2024-03-21T10:28:14.261 INFO:tasks.cephadm:Deploying OSDs... 2024-03-21T10:28:14.261 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-21T10:28:14.261 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/scratch_devs of=/dev/stdout 2024-03-21T10:28:14.322 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-21T10:28:14.322 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_1 2024-03-21T10:28:14.377 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-21T10:28:14.377 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T10:28:14.377 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 631 Links: 1 2024-03-21T10:28:14.378 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T10:28:14.378 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T10:28:14.378 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-03-21 10:27:11.403446598 +0000 2024-03-21T10:28:14.378 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-03-21 10:21:41.646206426 +0000 2024-03-21T10:28:14.378 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-03-21 10:21:41.646206426 +0000 2024-03-21T10:28:14.378 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-03-21 10:21:41.646206426 +0000 2024-03-21T10:28:14.378 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-21T10:28:14.477 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-03-21T10:28:14.477 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-03-21T10:28:14.477 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.00037879 s, 1.4 MB/s 2024-03-21T10:28:14.481 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-21T10:28:14.538 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_2 2024-03-21T10:28:14.593 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-21T10:28:14.593 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T10:28:14.593 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 651 Links: 1 2024-03-21T10:28:14.593 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T10:28:14.593 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T10:28:14.593 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-03-21 10:27:11.464445593 +0000 2024-03-21T10:28:14.593 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-03-21 10:21:42.209197147 +0000 2024-03-21T10:28:14.593 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-03-21 10:21:42.209197147 +0000 2024-03-21T10:28:14.593 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-03-21 10:21:42.209197147 +0000 2024-03-21T10:28:14.594 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-21T10:28:14.687 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-03-21T10:28:14.687 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-03-21T10:28:14.687 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000323965 s, 1.6 MB/s 2024-03-21T10:28:14.691 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-21T10:28:14.753 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_3 2024-03-21T10:28:14.807 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-21T10:28:14.807 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T10:28:14.807 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 670 Links: 1 2024-03-21T10:28:14.807 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T10:28:14.807 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T10:28:14.807 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-03-21 10:27:11.526444571 +0000 2024-03-21T10:28:14.807 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-03-21 10:21:42.808187276 +0000 2024-03-21T10:28:14.807 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-03-21 10:21:42.808187276 +0000 2024-03-21T10:28:14.807 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-03-21 10:21:42.808187276 +0000 2024-03-21T10:28:14.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:14 smithi031 ceph-mon[31673]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi031=x;smithi040=y;smithi070=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:28:14.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:14 smithi031 ceph-mon[31673]: Saving service mgr spec with placement smithi031=x;smithi040=y;smithi070=z;count:3 2024-03-21T10:28:14.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:14 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:14.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:14 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:14.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:14 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:14.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:14 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T10:28:14.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:14 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T10:28:14.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:14 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:14.809 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-21T10:28:14.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:14 smithi070 ceph-mon[36330]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi031=x;smithi040=y;smithi070=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:28:14.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:14 smithi070 ceph-mon[36330]: Saving service mgr spec with placement smithi031=x;smithi040=y;smithi070=z;count:3 2024-03-21T10:28:14.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:14 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:14.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:14 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:14.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:14 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:14.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:14 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T10:28:14.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:14 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T10:28:14.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:14 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:14.904 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-03-21T10:28:14.905 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-03-21T10:28:14.905 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.00036191 s, 1.4 MB/s 2024-03-21T10:28:14.909 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-21T10:28:14.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:14 smithi040 ceph-mon[36154]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi031=x;smithi040=y;smithi070=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:28:14.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:14 smithi040 ceph-mon[36154]: Saving service mgr spec with placement smithi031=x;smithi040=y;smithi070=z;count:3 2024-03-21T10:28:14.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:14 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:14.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:14 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:14.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:14 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:14.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:14 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T10:28:14.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:14 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T10:28:14.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:14 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:14.968 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_4 2024-03-21T10:28:15.017 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-21T10:28:15.017 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T10:28:15.017 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 690 Links: 1 2024-03-21T10:28:15.017 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T10:28:15.017 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T10:28:15.017 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-03-21 10:27:11.591443501 +0000 2024-03-21T10:28:15.017 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-03-21 10:21:43.404177454 +0000 2024-03-21T10:28:15.018 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-03-21 10:21:43.404177454 +0000 2024-03-21T10:28:15.018 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-03-21 10:21:43.404177454 +0000 2024-03-21T10:28:15.018 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-21T10:28:15.108 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-03-21T10:28:15.109 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-03-21T10:28:15.109 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000373837 s, 1.4 MB/s 2024-03-21T10:28:15.113 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-21T10:28:15.172 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-21T10:28:15.172 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/scratch_devs of=/dev/stdout 2024-03-21T10:28:15.192 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-21T10:28:15.192 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_1 2024-03-21T10:28:15.250 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-21T10:28:15.250 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T10:28:15.250 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-03-21T10:28:15.250 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T10:28:15.250 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T10:28:15.250 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-03-21 10:28:00.773851715 +0000 2024-03-21T10:28:15.250 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-03-21 10:20:45.542070357 +0000 2024-03-21T10:28:15.250 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-03-21 10:20:45.542070357 +0000 2024-03-21T10:28:15.250 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-03-21 10:20:45.542070357 +0000 2024-03-21T10:28:15.250 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-21T10:28:15.323 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-03-21T10:28:15.323 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-03-21T10:28:15.324 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000270025 s, 1.9 MB/s 2024-03-21T10:28:15.325 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-21T10:28:15.383 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_2 2024-03-21T10:28:15.440 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-21T10:28:15.440 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T10:28:15.440 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 681 Links: 1 2024-03-21T10:28:15.440 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T10:28:15.441 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T10:28:15.441 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-03-21 10:28:00.774851703 +0000 2024-03-21T10:28:15.441 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-03-21 10:20:45.899065883 +0000 2024-03-21T10:28:15.441 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-03-21 10:20:45.899065883 +0000 2024-03-21T10:28:15.441 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-03-21 10:20:45.899065883 +0000 2024-03-21T10:28:15.441 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-21T10:28:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-03-21T10:28:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-03-21T10:28:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000215708 s, 2.4 MB/s 2024-03-21T10:28:15.514 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-21T10:28:15.571 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_3 2024-03-21T10:28:15.584 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:15 smithi070 ceph-mon[36330]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:15.585 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:15 smithi070 ceph-mon[36330]: Reconfiguring mon.c (monmap changed)... 2024-03-21T10:28:15.585 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:15 smithi070 ceph-mon[36330]: Reconfiguring daemon mon.c on smithi070 2024-03-21T10:28:15.627 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-21T10:28:15.627 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T10:28:15.627 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 700 Links: 1 2024-03-21T10:28:15.627 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T10:28:15.627 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T10:28:15.627 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-03-21 10:28:00.774851703 +0000 2024-03-21T10:28:15.627 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-03-21 10:20:46.234061685 +0000 2024-03-21T10:28:15.628 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-03-21 10:20:46.234061685 +0000 2024-03-21T10:28:15.628 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-03-21 10:20:46.234061685 +0000 2024-03-21T10:28:15.628 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-21T10:28:15.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:15 smithi040 ceph-mon[36154]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:15.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:15 smithi040 ceph-mon[36154]: Reconfiguring mon.c (monmap changed)... 2024-03-21T10:28:15.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:15 smithi040 ceph-mon[36154]: Reconfiguring daemon mon.c on smithi070 2024-03-21T10:28:15.698 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-03-21T10:28:15.698 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-03-21T10:28:15.699 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000218677 s, 2.3 MB/s 2024-03-21T10:28:15.700 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-21T10:28:15.756 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_4 2024-03-21T10:28:15.812 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-21T10:28:15.813 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T10:28:15.813 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 718 Links: 1 2024-03-21T10:28:15.813 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T10:28:15.813 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T10:28:15.813 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-03-21 10:28:00.775851691 +0000 2024-03-21T10:28:15.813 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-03-21 10:20:46.577057387 +0000 2024-03-21T10:28:15.813 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-03-21 10:20:46.577057387 +0000 2024-03-21T10:28:15.813 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-03-21 10:20:46.577057387 +0000 2024-03-21T10:28:15.813 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-21T10:28:15.883 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-03-21T10:28:15.883 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-03-21T10:28:15.883 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000218788 s, 2.3 MB/s 2024-03-21T10:28:15.884 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-21T10:28:15.942 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-21T10:28:15.942 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/scratch_devs of=/dev/stdout 2024-03-21T10:28:15.958 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-21T10:28:15.958 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_1 2024-03-21T10:28:16.013 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-21T10:28:16.014 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T10:28:16.014 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 662 Links: 1 2024-03-21T10:28:16.014 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T10:28:16.014 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T10:28:16.014 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-03-21 10:27:51.183582008 +0000 2024-03-21T10:28:16.014 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-03-21 10:20:03.300511695 +0000 2024-03-21T10:28:16.014 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-03-21 10:20:03.300511695 +0000 2024-03-21T10:28:16.014 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-03-21 10:20:03.300511695 +0000 2024-03-21T10:28:16.014 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-21T10:28:16.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:15 smithi031 ceph-mon[31673]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:16.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:15 smithi031 ceph-mon[31673]: Reconfiguring mon.c (monmap changed)... 2024-03-21T10:28:16.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:15 smithi031 ceph-mon[31673]: Reconfiguring daemon mon.c on smithi070 2024-03-21T10:28:16.080 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-03-21T10:28:16.080 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-03-21T10:28:16.080 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000113425 s, 4.5 MB/s 2024-03-21T10:28:16.081 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-21T10:28:16.137 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_2 2024-03-21T10:28:16.192 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-21T10:28:16.192 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T10:28:16.192 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-03-21T10:28:16.192 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T10:28:16.192 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T10:28:16.193 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-03-21 10:27:51.183582008 +0000 2024-03-21T10:28:16.193 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-03-21 10:20:03.623501939 +0000 2024-03-21T10:28:16.193 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-03-21 10:20:03.623501939 +0000 2024-03-21T10:28:16.193 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-03-21 10:20:03.623501939 +0000 2024-03-21T10:28:16.193 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-21T10:28:16.260 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-03-21T10:28:16.260 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-03-21T10:28:16.260 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 8.7774e-05 s, 5.8 MB/s 2024-03-21T10:28:16.261 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-21T10:28:16.317 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_3 2024-03-21T10:28:16.372 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-21T10:28:16.373 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T10:28:16.373 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 699 Links: 1 2024-03-21T10:28:16.373 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T10:28:16.373 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T10:28:16.373 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-03-21 10:27:51.183582008 +0000 2024-03-21T10:28:16.373 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-03-21 10:20:03.950492062 +0000 2024-03-21T10:28:16.373 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-03-21 10:20:03.950492062 +0000 2024-03-21T10:28:16.373 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-03-21 10:20:03.950492062 +0000 2024-03-21T10:28:16.373 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-21T10:28:16.437 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-03-21T10:28:16.438 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-03-21T10:28:16.438 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000110986 s, 4.6 MB/s 2024-03-21T10:28:16.439 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-21T10:28:16.497 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_4 2024-03-21T10:28:16.549 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-21T10:28:16.550 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T10:28:16.550 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-03-21T10:28:16.550 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T10:28:16.550 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T10:28:16.550 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-03-21 10:27:51.183582008 +0000 2024-03-21T10:28:16.550 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-03-21 10:20:04.268482456 +0000 2024-03-21T10:28:16.550 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-03-21 10:20:04.268482456 +0000 2024-03-21T10:28:16.550 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-03-21 10:20:04.268482456 +0000 2024-03-21T10:28:16.550 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-21T10:28:16.617 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-03-21T10:28:16.617 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-03-21T10:28:16.617 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 8.6128e-05 s, 5.9 MB/s 2024-03-21T10:28:16.618 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-21T10:28:16.674 INFO:tasks.cephadm:Deploying osd.0 on smithi031 with /dev/vg_nvme/lv_4... 2024-03-21T10:28:16.674 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-21T10:28:16.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:16 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:16.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:16 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:16.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:16 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:28:16.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:16 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:16.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:16 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:28:16.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:16 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:16.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:16 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T10:28:16.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:16 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-21T10:28:16.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:16 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T10:28:16.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:16 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:16.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:16 smithi040 ceph-mon[36154]: Deploying daemon mgr.z on smithi070 2024-03-21T10:28:17.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:16 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:17.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:16 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:17.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:16 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:28:17.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:16 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:17.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:16 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:28:17.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:16 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:17.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:16 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T10:28:17.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:16 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-21T10:28:17.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:16 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T10:28:17.058 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:16 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:17.058 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:16 smithi031 ceph-mon[31673]: Deploying daemon mgr.z on smithi070 2024-03-21T10:28:17.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:16 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:17.076 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:16 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:17.076 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:16 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:28:17.076 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:16 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:17.076 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:16 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:28:17.076 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:16 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:17.076 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:16 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T10:28:17.076 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:16 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-21T10:28:17.076 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:16 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T10:28:17.076 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:16 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:17.076 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:16 smithi070 ceph-mon[36330]: Deploying daemon mgr.z on smithi070 2024-03-21T10:28:17.123 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:28:17.717 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:17 smithi070 systemd[1]: Starting Ceph mgr.z for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:28:17.987 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:17 smithi070 ceph-mon[36330]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:17.987 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:17 smithi070 podman[38181]: 2024-03-21 10:28:17.761257991 +0000 UTC m=+0.108231570 container create 7aeba34584304e32690e724124e328e3002535b235d5629cb6fd57d4a9989b60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-21T10:28:17.987 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:17 smithi070 podman[38181]: 2024-03-21 10:28:17.670945907 +0000 UTC m=+0.017919502 image pull 237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232 2024-03-21T10:28:17.987 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:17 smithi070 podman[38181]: 2024-03-21 10:28:17.929962159 +0000 UTC m=+0.276935739 container init 7aeba34584304e32690e724124e328e3002535b235d5629cb6fd57d4a9989b60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, ceph=True) 2024-03-21T10:28:17.988 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:17 smithi070 podman[38181]: 2024-03-21 10:28:17.932764039 +0000 UTC m=+0.279737610 container start 7aeba34584304e32690e724124e328e3002535b235d5629cb6fd57d4a9989b60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-21T10:28:17.988 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:17 smithi070 ceph-mgr[38200]: -- 172.21.15.70:0/3961334207 <== mon.1 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a0f4bfa5a0 con 0x55a0f4bcec00 2024-03-21T10:28:17.988 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:17 smithi070 bash[38181]: 7aeba34584304e32690e724124e328e3002535b235d5629cb6fd57d4a9989b60 2024-03-21T10:28:18.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:17 smithi031 ceph-mon[31673]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:18.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:17 smithi040 ceph-mon[36154]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:18.325 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:17 smithi070 systemd[1]: Started Ceph mgr.z for 3847408c-e76d-11ee-95cd-87774f69a715. 2024-03-21T10:28:18.825 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:18 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:28:18.470+0000 7fa55dece200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T10:28:18.826 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:18 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:28:18.561+0000 7fa55dece200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T10:28:19.213 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:19 smithi070 ceph-mon[36330]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:19.213 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:19 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:19.214 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:19 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:19.214 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:19 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:19.214 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:19 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T10:28:19.214 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:19 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-21T10:28:19.214 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:19 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T10:28:19.214 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:19 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:19.214 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:19 smithi070 ceph-mon[36330]: Deploying daemon mgr.y on smithi040 2024-03-21T10:28:19.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:19 smithi040 ceph-mon[36154]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:19.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:19 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:19.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:19 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:19.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:19 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:19.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:19 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T10:28:19.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:19 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-21T10:28:19.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:19 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T10:28:19.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:19 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:19.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:19 smithi040 ceph-mon[36154]: Deploying daemon mgr.y on smithi040 2024-03-21T10:28:19.476 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:19 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:28:19.213+0000 7fa55dece200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T10:28:19.476 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:19 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:28:19.305+0000 7fa55dece200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T10:28:19.476 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:19 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:28:19.456+0000 7fa55dece200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T10:28:19.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:19 smithi031 ceph-mon[31673]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:19.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:19 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:19.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:19 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:19.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:19 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:19.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:19 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T10:28:19.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:19 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-21T10:28:19.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:19 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T10:28:19.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:19 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:19.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:19 smithi031 ceph-mon[31673]: Deploying daemon mgr.y on smithi040 2024-03-21T10:28:19.732 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:19 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:28:19.556+0000 7fa55dece200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T10:28:19.732 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:19 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:28:19.641+0000 7fa55dece200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T10:28:20.026 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:19 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:28:19.913+0000 7fa55dece200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T10:28:20.026 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:20 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:28:20.026+0000 7fa55dece200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T10:28:20.129 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-21T10:28:20.129 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-21T10:28:20.130 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-03-21T10:28:20.130 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-03-21T10:28:20.130 INFO:teuthology.orchestra.run.smithi031.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0376044 s, 279 MB/s 2024-03-21T10:28:20.130 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-03-21T10:28:20.325 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:20 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:28:20.114+0000 7fa55dece200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T10:28:20.325 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:20 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:28:20.211+0000 7fa55dece200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T10:28:20.356 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:20 smithi040 systemd[1]: Starting Ceph mgr.y for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:28:20.654 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:20 smithi040 podman[37517]: 2024-03-21 10:28:20.259843637 +0000 UTC m=+0.018553099 image pull 237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232 2024-03-21T10:28:20.654 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:20 smithi040 podman[37517]: 2024-03-21 10:28:20.360797665 +0000 UTC m=+0.119507114 container create 37bcf80653ae1042d392fa5c4deda22bfdb3ee8509c6baeb027242ee05c4cb51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-21T10:28:20.655 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:20 smithi040 podman[37517]: 2024-03-21 10:28:20.57175643 +0000 UTC m=+0.330465893 container init 37bcf80653ae1042d392fa5c4deda22bfdb3ee8509c6baeb027242ee05c4cb51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-21T10:28:20.655 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:20 smithi040 podman[37517]: 2024-03-21 10:28:20.574553853 +0000 UTC m=+0.333263306 container start 37bcf80653ae1042d392fa5c4deda22bfdb3ee8509c6baeb027242ee05c4cb51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-21T10:28:20.655 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:20 smithi040 ceph-mgr[37543]: -- 172.21.15.40:0/3019272345 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5640408fc5a0 con 0x5640408d0800 2024-03-21T10:28:20.655 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:20 smithi040 bash[37517]: 37bcf80653ae1042d392fa5c4deda22bfdb3ee8509c6baeb027242ee05c4cb51 2024-03-21T10:28:20.682 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:20 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:28:20.389+0000 7fa55dece200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T10:28:20.933 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:20 smithi040 systemd[1]: Started Ceph mgr.y for 3847408c-e76d-11ee-95cd-87774f69a715. 2024-03-21T10:28:21.040 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi031:vg_nvme/lv_4 2024-03-21T10:28:21.404 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:21 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:28:21.148+0000 7feb09692200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T10:28:21.405 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:21 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:28:21.248+0000 7feb09692200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T10:28:21.513 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:28:21.736 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:21 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:28:21.381+0000 7fa55dece200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T10:28:21.737 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:21 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:28:21.516+0000 7fa55dece200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T10:28:21.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:21 smithi031 ceph-mon[31673]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:21.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:21 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:21.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:21 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:21.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:21 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:21.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:21 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:21.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:21 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:28:22.051 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:21 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:28:21.951+0000 7feb09692200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T10:28:22.051 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:22 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:28:22.050+0000 7feb09692200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T10:28:22.051 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:21 smithi040 ceph-mon[36154]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:22.051 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:21 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:22.051 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:21 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:22.051 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:21 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:22.051 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:21 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:22.051 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:21 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:28:22.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:21 smithi070 ceph-mon[36330]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:22.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:21 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:22.076 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:21 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:22.076 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:21 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:22.076 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:21 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:22.076 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:21 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:28:22.076 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:21 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:28:21.864+0000 7fa55dece200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T10:28:22.076 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:21 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:28:21.951+0000 7fa55dece200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T10:28:22.303 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:22 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:28:22.199+0000 7feb09692200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T10:28:22.304 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:22 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:28:22.303+0000 7feb09692200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T10:28:22.575 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:22 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:28:22.146+0000 7fa55dece200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T10:28:22.576 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:22 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:28:22.340+0000 7fa55dece200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T10:28:22.680 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:22 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:28:22.406+0000 7feb09692200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T10:28:22.881 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:22 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:28:22.628+0000 7fa55dece200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T10:28:22.881 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:22 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:28:22.881+0000 7fa55dece200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T10:28:22.933 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:22 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:28:22.680+0000 7feb09692200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T10:28:22.933 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:22 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:28:22.793+0000 7feb09692200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T10:28:22.933 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:22 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:28:22.883+0000 7feb09692200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T10:28:23.325 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:23 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:28:23.115+0000 7fa55dece200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T10:28:23.426 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:22 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:28:22.980+0000 7feb09692200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T10:28:23.426 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:23 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:28:23.161+0000 7feb09692200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T10:28:23.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:23 smithi040 ceph-mon[36154]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:23.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:23 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:23.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:23 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:23.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:23 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:23.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:23 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:23.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:23 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:23.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:23 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:28:23.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:23 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:23.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:23 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T10:28:23.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:23 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T10:28:23.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:23 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:23.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:23 smithi031 ceph-mon[31673]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:23.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:23 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:23.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:23 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:23.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:23 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:23.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:23 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:23.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:23 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:23.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:23 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:28:23.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:23 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:23.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:23 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T10:28:23.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:23 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T10:28:23.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:23 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:23.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.687+0000 7f732f600700 1 -- 172.21.15.31:0/3173350332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73281021c0 msgr2=0x7f73281025e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:28:23.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.687+0000 7f732f600700 1 --2- 172.21.15.31:0/3173350332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73281021c0 0x7f73281025e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f731c0099c0 tx=0x7f731c009cd0 comp rx=0 tx=0).stop 2024-03-21T10:28:23.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.688+0000 7f732f600700 1 -- 172.21.15.31:0/3173350332 shutdown_connections 2024-03-21T10:28:23.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.688+0000 7f732f600700 1 --2- 172.21.15.31:0/3173350332 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7328103d60 0x7f732810a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:23.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.688+0000 7f732f600700 1 --2- 172.21.15.31:0/3173350332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73281021c0 0x7f73281025e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:23.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.688+0000 7f732f600700 1 --2- 172.21.15.31:0/3173350332 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73281033c0 0x7f7328103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:23.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.688+0000 7f732f600700 1 -- 172.21.15.31:0/3173350332 >> 172.21.15.31:0/3173350332 conn(0x7f73280fd7a0 msgr2=0x7f73280ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:28:23.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.688+0000 7f732f600700 1 -- 172.21.15.31:0/3173350332 shutdown_connections 2024-03-21T10:28:23.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.688+0000 7f732f600700 1 -- 172.21.15.31:0/3173350332 wait complete. 2024-03-21T10:28:23.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.689+0000 7f732f600700 1 Processor -- start 2024-03-21T10:28:23.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.689+0000 7f732f600700 1 -- start start 2024-03-21T10:28:23.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.690+0000 7f732f600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73281021c0 0x7f7328075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:28:23.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.690+0000 7f732f600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73281033c0 0x7f7328074580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:28:23.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.690+0000 7f732f600700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7328103d60 0x7f7328074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:28:23.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.690+0000 7f732f600700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732810ddf0 con 0x7f7328103d60 2024-03-21T10:28:23.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.690+0000 7f732f600700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732810dc70 con 0x7f73281021c0 2024-03-21T10:28:23.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.690+0000 7f732f600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732810df70 con 0x7f73281033c0 2024-03-21T10:28:23.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.690+0000 7f732cc00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73281021c0 0x7f7328075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:28:23.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.690+0000 7f732cc00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73281021c0 0x7f7328075f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58902/0 (socket says 172.21.15.31:58902) 2024-03-21T10:28:23.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.690+0000 7f732cc00700 1 -- 172.21.15.31:0/1057683897 learned_addr learned my addr 172.21.15.31:0/1057683897 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:28:23.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.690+0000 7f732d600700 1 --2- 172.21.15.31:0/1057683897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7328103d60 0x7f7328074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:28:23.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.690+0000 7f7327e00700 1 --2- 172.21.15.31:0/1057683897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73281033c0 0x7f7328074580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:28:23.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.690+0000 7f732cc00700 1 -- 172.21.15.31:0/1057683897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73281033c0 msgr2=0x7f7328074580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:28:23.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.690+0000 7f732cc00700 1 --2- 172.21.15.31:0/1057683897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73281033c0 0x7f7328074580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:23.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.690+0000 7f732cc00700 1 -- 172.21.15.31:0/1057683897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7328103d60 msgr2=0x7f7328074ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:28:23.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.691+0000 7f732cc00700 1 --2- 172.21.15.31:0/1057683897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7328103d60 0x7f7328074ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:23.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.691+0000 7f732cc00700 1 -- 172.21.15.31:0/1057683897 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7328075370 con 0x7f73281021c0 2024-03-21T10:28:23.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.691+0000 7f732d600700 1 --2- 172.21.15.31:0/1057683897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7328103d60 0x7f7328074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:28:23.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.691+0000 7f732cc00700 1 --2- 172.21.15.31:0/1057683897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73281021c0 0x7f7328075f30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f731c0098e0 tx=0x7f731c009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:28:23.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.691+0000 7f7325600700 1 -- 172.21.15.31:0/1057683897 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c027070 con 0x7f73281021c0 2024-03-21T10:28:23.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.691+0000 7f7325600700 1 -- 172.21.15.31:0/1057683897 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f731c022470 con 0x7f73281021c0 2024-03-21T10:28:23.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.691+0000 7f7325600700 1 -- 172.21.15.31:0/1057683897 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c009e60 con 0x7f73281021c0 2024-03-21T10:28:23.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.691+0000 7f732f600700 1 -- 172.21.15.31:0/1057683897 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73281a2590 con 0x7f73281021c0 2024-03-21T10:28:23.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.691+0000 7f732f600700 1 -- 172.21.15.31:0/1057683897 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73281a29e0 con 0x7f73281021c0 2024-03-21T10:28:23.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.692+0000 7f7325600700 1 -- 172.21.15.31:0/1057683897 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) v1 ==== 46350+0+0 (secure 0 0 0) 0x7f731c02a070 con 0x7f73281021c0 2024-03-21T10:28:23.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.693+0000 7f732f600700 1 -- 172.21.15.31:0/1057683897 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7328068e50 con 0x7f73281021c0 2024-03-21T10:28:23.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.693+0000 7f7325600700 1 --2- 172.21.15.31:0/1057683897 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f7310039460 0x7f731003b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:28:23.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.693+0000 7f7325600700 1 -- 172.21.15.31:0/1057683897 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f731c04d430 con 0x7f73281021c0 2024-03-21T10:28:23.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.693+0000 7f7327e00700 1 --2- 172.21.15.31:0/1057683897 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f7310039460 0x7f731003b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:28:23.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.694+0000 7f7327e00700 1 --2- 172.21.15.31:0/1057683897 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f7310039460 0x7f731003b920 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7318009c60 tx=0x7f7318009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:28:23.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.698+0000 7f7325600700 1 -- 172.21.15.31:0/1057683897 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f731c013070 con 0x7f73281021c0 2024-03-21T10:28:23.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:23 smithi070 ceph-mon[36330]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:23.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:23 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:23.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:23 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:23.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:23 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:23.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:23 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:23.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:23 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:23.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:23 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:28:23.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:23 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:23.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:23 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T10:28:23.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:23 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T10:28:23.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:23 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:23.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:23.946+0000 7f732f600700 1 -- 172.21.15.31:0/1057683897 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f73280625d0 con 0x7f7310039460 2024-03-21T10:28:24.325 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:23 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:28:23.897+0000 7fa55dece200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T10:28:24.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:24 smithi031 ceph-mon[31673]: Reconfiguring mgr.x (unknown last config time)... 2024-03-21T10:28:24.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:24 smithi031 ceph-mon[31673]: Reconfiguring daemon mgr.x on smithi031 2024-03-21T10:28:24.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:24 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:24.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:24 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:28:24.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:24 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:28:24.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:24 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:24.671 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:24 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:28:24.189+0000 7feb09692200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T10:28:24.671 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:24 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:28:24.326+0000 7feb09692200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T10:28:24.671 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:24 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:28:24.670+0000 7feb09692200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T10:28:24.671 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:24 smithi040 ceph-mon[36154]: Reconfiguring mgr.x (unknown last config time)... 2024-03-21T10:28:24.671 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:24 smithi040 ceph-mon[36154]: Reconfiguring daemon mgr.x on smithi031 2024-03-21T10:28:24.671 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:24 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:24.671 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:24 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:28:24.671 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:24 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:28:24.671 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:24 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:24.757 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:24 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:28:24.453+0000 7fa55dece200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T10:28:24.757 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:24 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:28:24.548+0000 7fa55dece200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T10:28:24.757 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:28:24 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:28:24.757+0000 7fa55dece200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T10:28:24.757 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:24 smithi070 ceph-mon[36330]: Reconfiguring mgr.x (unknown last config time)... 2024-03-21T10:28:24.757 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:24 smithi070 ceph-mon[36330]: Reconfiguring daemon mgr.x on smithi031 2024-03-21T10:28:24.757 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:24 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:24.757 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:24 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:28:24.758 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:24 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:28:24.758 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:24 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:24.933 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:24 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:28:24.762+0000 7feb09692200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T10:28:25.433 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:24 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:28:24.963+0000 7feb09692200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T10:28:25.433 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:25 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:28:25.153+0000 7feb09692200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T10:28:25.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:25.455+0000 7f7325600700 1 -- 172.21.15.31:0/1057683897 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 14) v1 ==== 92074+0+0 (secure 0 0 0) 0x7f731c022690 con 0x7f73281021c0 2024-03-21T10:28:25.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:25 smithi031 ceph-mon[31673]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:25.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:25 smithi031 ceph-mon[31673]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:28:25.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:25 smithi031 ceph-mon[31673]: Standby manager daemon z started 2024-03-21T10:28:25.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:25 smithi031 ceph-mon[31673]: from='mgr.? 172.21.15.70:0/577264051' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-21T10:28:25.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:25 smithi031 ceph-mon[31673]: from='mgr.? 172.21.15.70:0/577264051' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T10:28:25.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:25 smithi031 ceph-mon[31673]: from='mgr.? 172.21.15.70:0/577264051' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-21T10:28:25.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:25 smithi031 ceph-mon[31673]: from='mgr.? 172.21.15.70:0/577264051' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T10:28:25.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:25 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:25.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:25 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:25.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:25 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:28:25.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:25 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:25.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:25 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:28:25.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:25 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:25.715 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:25 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:28:25.447+0000 7feb09692200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T10:28:25.715 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:25 smithi040 ceph-mon[36154]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:25.715 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:25 smithi040 ceph-mon[36154]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:28:25.715 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:25 smithi040 ceph-mon[36154]: Standby manager daemon z started 2024-03-21T10:28:25.715 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:25 smithi040 ceph-mon[36154]: from='mgr.? 172.21.15.70:0/577264051' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-21T10:28:25.715 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:25 smithi040 ceph-mon[36154]: from='mgr.? 172.21.15.70:0/577264051' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T10:28:25.715 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:25 smithi040 ceph-mon[36154]: from='mgr.? 172.21.15.70:0/577264051' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-21T10:28:25.715 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:25 smithi040 ceph-mon[36154]: from='mgr.? 172.21.15.70:0/577264051' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T10:28:25.716 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:25 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:25.716 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:25 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:25.716 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:25 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:28:25.716 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:25 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:25.716 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:25 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:28:25.716 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:25 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:25.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:25 smithi070 ceph-mon[36330]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:25.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:25 smithi070 ceph-mon[36330]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:28:25.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:25 smithi070 ceph-mon[36330]: Standby manager daemon z started 2024-03-21T10:28:25.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:25 smithi070 ceph-mon[36330]: from='mgr.? 172.21.15.70:0/577264051' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-21T10:28:25.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:25 smithi070 ceph-mon[36330]: from='mgr.? 172.21.15.70:0/577264051' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T10:28:25.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:25 smithi070 ceph-mon[36330]: from='mgr.? 172.21.15.70:0/577264051' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-21T10:28:25.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:25 smithi070 ceph-mon[36330]: from='mgr.? 172.21.15.70:0/577264051' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T10:28:25.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:25 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:25.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:25 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:25.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:25 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:28:25.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:25 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:25.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:25 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:28:25.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:25 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:26.183 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:25 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:28:25.715+0000 7feb09692200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T10:28:26.183 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:25 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:28:25.957+0000 7feb09692200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T10:28:26.789 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:26 smithi040 ceph-mon[36154]: mgrmap e14: x(active, since 91s), standbys: z 2024-03-21T10:28:26.789 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:26 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-21T10:28:26.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:26 smithi031 ceph-mon[31673]: mgrmap e14: x(active, since 91s), standbys: z 2024-03-21T10:28:26.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:26 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-21T10:28:26.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:26 smithi070 ceph-mon[36330]: mgrmap e14: x(active, since 91s), standbys: z 2024-03-21T10:28:26.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:26 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-21T10:28:27.183 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:26 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:28:26.789+0000 7feb09692200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T10:28:27.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:27 smithi040 ceph-mon[36154]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:27.683 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:27 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:28:27.378+0000 7feb09692200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T10:28:27.683 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:27 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:28:27.478+0000 7feb09692200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T10:28:27.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:27 smithi031 ceph-mon[31673]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:27.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:27 smithi070 ceph-mon[36330]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:28.183 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:28:27 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:28:27.699+0000 7feb09692200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T10:28:28.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:28.487+0000 7f7325600700 1 -- 172.21.15.31:0/1057683897 <== mon.2 v2:172.21.15.40:3300/0 8 ==== mgrmap(e 15) v1 ==== 137798+0+0 (secure 0 0 0) 0x7f731c00fb10 con 0x7f73281021c0 2024-03-21T10:28:28.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:28 smithi031 ceph-mon[31673]: Standby manager daemon y started 2024-03-21T10:28:28.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:28 smithi031 ceph-mon[31673]: from='mgr.? 172.21.15.40:0/3659738825' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-21T10:28:28.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:28 smithi031 ceph-mon[31673]: from='mgr.? 172.21.15.40:0/3659738825' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T10:28:28.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:28 smithi031 ceph-mon[31673]: from='mgr.? 172.21.15.40:0/3659738825' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-21T10:28:28.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:28 smithi031 ceph-mon[31673]: from='mgr.? 172.21.15.40:0/3659738825' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T10:28:28.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:28 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3099044957' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0244d33-00e2-47d5-9f66-34efc138c462"}]: dispatch 2024-03-21T10:28:28.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:28 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3099044957' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0244d33-00e2-47d5-9f66-34efc138c462"}]': finished 2024-03-21T10:28:28.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:28 smithi031 ceph-mon[31673]: osdmap e5: 1 total, 0 up, 1 in 2024-03-21T10:28:28.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:28 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T10:28:28.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:28 smithi070 ceph-mon[36330]: Standby manager daemon y started 2024-03-21T10:28:28.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:28 smithi070 ceph-mon[36330]: from='mgr.? 172.21.15.40:0/3659738825' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-21T10:28:28.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:28 smithi070 ceph-mon[36330]: from='mgr.? 172.21.15.40:0/3659738825' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T10:28:28.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:28 smithi070 ceph-mon[36330]: from='mgr.? 172.21.15.40:0/3659738825' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-21T10:28:28.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:28 smithi070 ceph-mon[36330]: from='mgr.? 172.21.15.40:0/3659738825' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T10:28:28.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:28 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3099044957' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0244d33-00e2-47d5-9f66-34efc138c462"}]: dispatch 2024-03-21T10:28:28.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:28 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3099044957' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0244d33-00e2-47d5-9f66-34efc138c462"}]': finished 2024-03-21T10:28:28.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:28 smithi070 ceph-mon[36330]: osdmap e5: 1 total, 0 up, 1 in 2024-03-21T10:28:28.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:28 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T10:28:28.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:28 smithi040 ceph-mon[36154]: Standby manager daemon y started 2024-03-21T10:28:28.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:28 smithi040 ceph-mon[36154]: from='mgr.? 172.21.15.40:0/3659738825' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-21T10:28:28.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:28 smithi040 ceph-mon[36154]: from='mgr.? 172.21.15.40:0/3659738825' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T10:28:28.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:28 smithi040 ceph-mon[36154]: from='mgr.? 172.21.15.40:0/3659738825' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-21T10:28:28.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:28 smithi040 ceph-mon[36154]: from='mgr.? 172.21.15.40:0/3659738825' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T10:28:28.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:28 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3099044957' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0244d33-00e2-47d5-9f66-34efc138c462"}]: dispatch 2024-03-21T10:28:28.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:28 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3099044957' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0244d33-00e2-47d5-9f66-34efc138c462"}]': finished 2024-03-21T10:28:28.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:28 smithi040 ceph-mon[36154]: osdmap e5: 1 total, 0 up, 1 in 2024-03-21T10:28:28.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:28 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T10:28:29.797 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:29 smithi031 ceph-mon[31673]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:29.798 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:29 smithi031 ceph-mon[31673]: mgrmap e15: x(active, since 94s), standbys: z, y 2024-03-21T10:28:29.798 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:29 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T10:28:29.798 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:29 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3098027889' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:28:29.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:29 smithi070 ceph-mon[36330]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:29.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:29 smithi070 ceph-mon[36330]: mgrmap e15: x(active, since 94s), standbys: z, y 2024-03-21T10:28:29.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:29 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T10:28:29.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:29 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3098027889' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:28:29.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:29 smithi040 ceph-mon[36154]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:29.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:29 smithi040 ceph-mon[36154]: mgrmap e15: x(active, since 94s), standbys: z, y 2024-03-21T10:28:29.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:29 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T10:28:29.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:29 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3098027889' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:28:31.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:31 smithi031 ceph-mon[31673]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:31.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:31 smithi070 ceph-mon[36330]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:31.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:31 smithi040 ceph-mon[36154]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:33.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:33 smithi031 ceph-mon[31673]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:33.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:33 smithi070 ceph-mon[36330]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:33.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:33 smithi040 ceph-mon[36154]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:35.726 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:35 smithi031 ceph-mon[31673]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:35.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:35 smithi070 ceph-mon[36330]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:35.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:35 smithi040 ceph-mon[36154]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:37.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:37 smithi031 ceph-mon[31673]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:37.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:37 smithi070 ceph-mon[36330]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:37.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:37 smithi040 ceph-mon[36154]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:38.783 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:38 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T10:28:38.784 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:38 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:38.784 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:38 smithi031 ceph-mon[31673]: Deploying daemon osd.0 on smithi031 2024-03-21T10:28:38.810 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:38 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T10:28:38.810 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:38 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:38.810 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:38 smithi070 ceph-mon[36330]: Deploying daemon osd.0 on smithi031 2024-03-21T10:28:38.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:38 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T10:28:38.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:38 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:38.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:38 smithi040 ceph-mon[36154]: Deploying daemon osd.0 on smithi031 2024-03-21T10:28:39.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:39 smithi031 ceph-mon[31673]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:39.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:39 smithi070 ceph-mon[36330]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:39.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:39 smithi040 ceph-mon[36154]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:41.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:41 smithi031 ceph-mon[31673]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:41.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:41 smithi070 ceph-mon[36330]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:41.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:41 smithi040 ceph-mon[36154]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:43.676 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:43 smithi031 ceph-mon[31673]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:43.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:43 smithi070 ceph-mon[36330]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:43.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:43 smithi040 ceph-mon[36154]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:45.715 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:45 smithi031 ceph-mon[31673]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:45.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:45 smithi070 ceph-mon[36330]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:45.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:45 smithi040 ceph-mon[36154]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:47.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:47 smithi031 ceph-mon[31673]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:47.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:47 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:47.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:47 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:47.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:47 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:28:47.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:47 smithi070 ceph-mon[36330]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:47.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:47 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:47.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:47 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:47.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:47 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:28:47.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:47 smithi040 ceph-mon[36154]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:47.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:47 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:47.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:47 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:47.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:47 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:28:49.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:49 smithi031 ceph-mon[31673]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:49.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:49 smithi070 ceph-mon[36330]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:49.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:49 smithi040 ceph-mon[36154]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:50.713 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:50 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:50.714 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:50 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:50.714 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:50 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:50.714 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:50 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:28:50.714 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:50 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:50.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:50 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:50.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:50 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:50.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:50 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:50.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:50 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:28:50.735 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:50 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:50.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:50 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:50.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:50 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:50.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:50 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:50.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:50 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:28:50.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:50 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:51.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:51.698+0000 7f7325600700 1 -- 172.21.15.31:0/1057683897 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f73280625d0 con 0x7f7310039460 2024-03-21T10:28:51.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:51 smithi031 ceph-mon[31673]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:51.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:51 smithi031 ceph-mon[31673]: from='osd.0 [v2:172.21.15.31:6802/3089885125,v1:172.21.15.31:6803/3089885125]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T10:28:51.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:51 smithi031 ceph-mon[31673]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T10:28:51.699 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 0 on host 'smithi031' 2024-03-21T10:28:51.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:51.702+0000 7f732f600700 1 -- 172.21.15.31:0/1057683897 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f7310039460 msgr2=0x7f731003b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:28:51.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:51.702+0000 7f732f600700 1 --2- 172.21.15.31:0/1057683897 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f7310039460 0x7f731003b920 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7318009c60 tx=0x7f7318009260 comp rx=0 tx=0).stop 2024-03-21T10:28:51.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:51.702+0000 7f732f600700 1 -- 172.21.15.31:0/1057683897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73281021c0 msgr2=0x7f7328075f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:28:51.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:51.702+0000 7f732f600700 1 --2- 172.21.15.31:0/1057683897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73281021c0 0x7f7328075f30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f731c0098e0 tx=0x7f731c009580 comp rx=0 tx=0).stop 2024-03-21T10:28:51.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:51.702+0000 7f732f600700 1 -- 172.21.15.31:0/1057683897 shutdown_connections 2024-03-21T10:28:51.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:51.702+0000 7f732f600700 1 --2- 172.21.15.31:0/1057683897 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f7310039460 0x7f731003b920 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:51.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:51.702+0000 7f732f600700 1 --2- 172.21.15.31:0/1057683897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7328103d60 0x7f7328074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:51.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:51.702+0000 7f732f600700 1 --2- 172.21.15.31:0/1057683897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73281021c0 0x7f7328075f30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:51.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:51.702+0000 7f732f600700 1 --2- 172.21.15.31:0/1057683897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73281033c0 0x7f7328074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:51.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:51.702+0000 7f732f600700 1 -- 172.21.15.31:0/1057683897 >> 172.21.15.31:0/1057683897 conn(0x7f73280fd7a0 msgr2=0x7f73281066e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:28:51.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:51.702+0000 7f732f600700 1 -- 172.21.15.31:0/1057683897 shutdown_connections 2024-03-21T10:28:51.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:28:51.703+0000 7f732f600700 1 -- 172.21.15.31:0/1057683897 wait complete. 2024-03-21T10:28:51.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:51 smithi070 ceph-mon[36330]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:51.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:51 smithi070 ceph-mon[36330]: from='osd.0 [v2:172.21.15.31:6802/3089885125,v1:172.21.15.31:6803/3089885125]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T10:28:51.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:51 smithi070 ceph-mon[36330]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T10:28:51.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:51 smithi040 ceph-mon[36154]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:51.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:51 smithi040 ceph-mon[36154]: from='osd.0 [v2:172.21.15.31:6802/3089885125,v1:172.21.15.31:6803/3089885125]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T10:28:51.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:51 smithi040 ceph-mon[36154]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T10:28:52.615 DEBUG:teuthology.orchestra.run.smithi031:osd.0> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.0.service 2024-03-21T10:28:52.620 INFO:tasks.cephadm:Deploying osd.1 on smithi040 with /dev/vg_nvme/lv_4... 2024-03-21T10:28:52.620 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-21T10:28:52.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:52 smithi031 ceph-mon[31673]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-21T10:28:52.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:52 smithi031 ceph-mon[31673]: osdmap e6: 1 total, 0 up, 1 in 2024-03-21T10:28:52.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:52 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T10:28:52.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:52 smithi031 ceph-mon[31673]: from='osd.0 [v2:172.21.15.31:6802/3089885125,v1:172.21.15.31:6803/3089885125]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-21T10:28:52.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:52 smithi031 ceph-mon[31673]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-21T10:28:52.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:52 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:52.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:52 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:28:52.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:52 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:52.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:52 smithi070 ceph-mon[36330]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-21T10:28:52.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:52 smithi070 ceph-mon[36330]: osdmap e6: 1 total, 0 up, 1 in 2024-03-21T10:28:52.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:52 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T10:28:52.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:52 smithi070 ceph-mon[36330]: from='osd.0 [v2:172.21.15.31:6802/3089885125,v1:172.21.15.31:6803/3089885125]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-21T10:28:52.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:52 smithi070 ceph-mon[36330]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-21T10:28:52.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:52 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:52.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:52 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:28:52.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:52 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:52.877 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T10:28:52.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:52 smithi040 ceph-mon[36154]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-21T10:28:52.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:52 smithi040 ceph-mon[36154]: osdmap e6: 1 total, 0 up, 1 in 2024-03-21T10:28:52.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:52 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T10:28:52.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:52 smithi040 ceph-mon[36154]: from='osd.0 [v2:172.21.15.31:6802/3089885125,v1:172.21.15.31:6803/3089885125]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-21T10:28:52.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:52 smithi040 ceph-mon[36154]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-21T10:28:52.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:52 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:52.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:52 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:28:52.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:52 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:53.409 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 21 10:28:53 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-0[46368]: 2024-03-21T10:28:53.155+0000 7f479ba00700 -1 osd.0 0 waiting for initial osdmap 2024-03-21T10:28:53.410 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 21 10:28:53 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-0[46368]: 2024-03-21T10:28:53.180+0000 7f4791800700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T10:28:53.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:53 smithi040 ceph-mon[36154]: purged_snaps scrub starts 2024-03-21T10:28:53.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:53 smithi040 ceph-mon[36154]: purged_snaps scrub ok 2024-03-21T10:28:53.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:53 smithi040 ceph-mon[36154]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:53.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:53 smithi040 ceph-mon[36154]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-21T10:28:53.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:53 smithi040 ceph-mon[36154]: osdmap e7: 1 total, 0 up, 1 in 2024-03-21T10:28:53.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:53 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T10:28:53.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:53 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T10:28:53.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:53 smithi040 ceph-mon[36154]: from='osd.0 ' entity='osd.0' 2024-03-21T10:28:53.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:53 smithi031 ceph-mon[31673]: purged_snaps scrub starts 2024-03-21T10:28:53.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:53 smithi031 ceph-mon[31673]: purged_snaps scrub ok 2024-03-21T10:28:53.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:53 smithi031 ceph-mon[31673]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:53.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:53 smithi031 ceph-mon[31673]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-21T10:28:53.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:53 smithi031 ceph-mon[31673]: osdmap e7: 1 total, 0 up, 1 in 2024-03-21T10:28:53.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:53 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T10:28:53.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:53 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T10:28:53.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:53 smithi031 ceph-mon[31673]: from='osd.0 ' entity='osd.0' 2024-03-21T10:28:53.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:53 smithi070 ceph-mon[36330]: purged_snaps scrub starts 2024-03-21T10:28:53.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:53 smithi070 ceph-mon[36330]: purged_snaps scrub ok 2024-03-21T10:28:53.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:53 smithi070 ceph-mon[36330]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:53.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:53 smithi070 ceph-mon[36330]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-21T10:28:53.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:53 smithi070 ceph-mon[36330]: osdmap e7: 1 total, 0 up, 1 in 2024-03-21T10:28:53.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:53 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T10:28:53.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:53 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T10:28:53.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:53 smithi070 ceph-mon[36330]: from='osd.0 ' entity='osd.0' 2024-03-21T10:28:54.702 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:54 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T10:28:54.702 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:54 smithi040 ceph-mon[36154]: osd.0 [v2:172.21.15.31:6802/3089885125,v1:172.21.15.31:6803/3089885125] boot 2024-03-21T10:28:54.702 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:54 smithi040 ceph-mon[36154]: osdmap e8: 1 total, 1 up, 1 in 2024-03-21T10:28:54.702 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:54 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T10:28:54.702 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-21T10:28:54.702 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-21T10:28:54.703 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-03-21T10:28:54.703 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-03-21T10:28:54.703 INFO:teuthology.orchestra.run.smithi040.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110909 s, 945 MB/s 2024-03-21T10:28:54.703 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 2024-03-21T10:28:54.703 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-03-21T10:28:54.754 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:54 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T10:28:54.754 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:54 smithi031 ceph-mon[31673]: osd.0 [v2:172.21.15.31:6802/3089885125,v1:172.21.15.31:6803/3089885125] boot 2024-03-21T10:28:54.754 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:54 smithi031 ceph-mon[31673]: osdmap e8: 1 total, 1 up, 1 in 2024-03-21T10:28:54.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:54 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T10:28:54.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:54 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T10:28:54.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:54 smithi070 ceph-mon[36330]: osd.0 [v2:172.21.15.31:6802/3089885125,v1:172.21.15.31:6803/3089885125] boot 2024-03-21T10:28:54.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:54 smithi070 ceph-mon[36330]: osdmap e8: 1 total, 1 up, 1 in 2024-03-21T10:28:54.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:54 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T10:28:55.299 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi040:vg_nvme/lv_4 2024-03-21T10:28:55.552 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T10:28:55.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:55 smithi031 ceph-mon[31673]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:55.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:55 smithi031 ceph-mon[31673]: osdmap e9: 1 total, 1 up, 1 in 2024-03-21T10:28:55.668 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:55 smithi040 ceph-mon[36154]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:55.669 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:55 smithi040 ceph-mon[36154]: osdmap e9: 1 total, 1 up, 1 in 2024-03-21T10:28:55.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:55 smithi070 ceph-mon[36330]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T10:28:55.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:55 smithi070 ceph-mon[36330]: osdmap e9: 1 total, 1 up, 1 in 2024-03-21T10:28:57.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.021+0000 7f554d45c700 1 -- 172.21.15.40:0/468001598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5548103d60 msgr2=0x7f554810a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:28:57.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.021+0000 7f554d45c700 1 --2- 172.21.15.40:0/468001598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5548103d60 0x7f554810a610 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f554000b0d0 tx=0x7f554000b3e0 comp rx=0 tx=0).stop 2024-03-21T10:28:57.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.021+0000 7f554d45c700 1 -- 172.21.15.40:0/468001598 shutdown_connections 2024-03-21T10:28:57.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.021+0000 7f554d45c700 1 --2- 172.21.15.40:0/468001598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5548103d60 0x7f554810a610 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:57.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.021+0000 7f554d45c700 1 --2- 172.21.15.40:0/468001598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55481021c0 0x7f55481025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:57.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.021+0000 7f554d45c700 1 --2- 172.21.15.40:0/468001598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55481033c0 0x7f5548103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:57.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.021+0000 7f554d45c700 1 -- 172.21.15.40:0/468001598 >> 172.21.15.40:0/468001598 conn(0x7f55480fd7a0 msgr2=0x7f55480ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:28:57.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.021+0000 7f554d45c700 1 -- 172.21.15.40:0/468001598 shutdown_connections 2024-03-21T10:28:57.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.021+0000 7f554d45c700 1 -- 172.21.15.40:0/468001598 wait complete. 2024-03-21T10:28:57.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.021+0000 7f554d45c700 1 Processor -- start 2024-03-21T10:28:57.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.021+0000 7f554d45c700 1 -- start start 2024-03-21T10:28:57.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.022+0000 7f554d45c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55481021c0 0x7f55481983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:28:57.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.022+0000 7f554d45c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55481033c0 0x7f55481988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:28:57.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.022+0000 7f554d45c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5548103d60 0x7f554819f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:28:57.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.022+0000 7f554d45c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554810de80 con 0x7f5548103d60 2024-03-21T10:28:57.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.022+0000 7f554d45c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554810dd00 con 0x7f55481021c0 2024-03-21T10:28:57.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.022+0000 7f554d45c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554810e000 con 0x7f55481033c0 2024-03-21T10:28:57.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.022+0000 7f5546ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55481021c0 0x7f55481983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:28:57.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.022+0000 7f5546ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55481021c0 0x7f55481983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52620/0 (socket says 172.21.15.40:52620) 2024-03-21T10:28:57.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.022+0000 7f5546ffd700 1 -- 172.21.15.40:0/1574899597 learned_addr learned my addr 172.21.15.40:0/1574899597 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-21T10:28:57.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.022+0000 7f55467fc700 1 --2- 172.21.15.40:0/1574899597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55481033c0 0x7f55481988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:28:57.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.022+0000 7f5546ffd700 1 -- 172.21.15.40:0/1574899597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55481033c0 msgr2=0x7f55481988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:28:57.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.022+0000 7f5546ffd700 1 --2- 172.21.15.40:0/1574899597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55481033c0 0x7f55481988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:57.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.022+0000 7f5546ffd700 1 -- 172.21.15.40:0/1574899597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5548103d60 msgr2=0x7f554819f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:28:57.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.022+0000 7f5546ffd700 1 --2- 172.21.15.40:0/1574899597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5548103d60 0x7f554819f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:28:57.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.022+0000 7f5546ffd700 1 -- 172.21.15.40:0/1574899597 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55481a0000 con 0x7f55481021c0 2024-03-21T10:28:57.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.022+0000 7f5546ffd700 1 --2- 172.21.15.40:0/1574899597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55481021c0 0x7f55481983a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f553800b500 tx=0x7f553800b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:28:57.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.022+0000 7f552ffff700 1 -- 172.21.15.40:0/1574899597 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5538015020 con 0x7f55481021c0 2024-03-21T10:28:57.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.023+0000 7f554d45c700 1 -- 172.21.15.40:0/1574899597 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55481a02f0 con 0x7f55481021c0 2024-03-21T10:28:57.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.023+0000 7f552ffff700 1 -- 172.21.15.40:0/1574899597 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5538011d10 con 0x7f55481021c0 2024-03-21T10:28:57.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.023+0000 7f552ffff700 1 -- 172.21.15.40:0/1574899597 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553800f2e0 con 0x7f55481021c0 2024-03-21T10:28:57.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.023+0000 7f554d45c700 1 -- 172.21.15.40:0/1574899597 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55481a0900 con 0x7f55481021c0 2024-03-21T10:28:57.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.024+0000 7f554d45c700 1 -- 172.21.15.40:0/1574899597 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5548068e50 con 0x7f55481021c0 2024-03-21T10:28:57.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.025+0000 7f552ffff700 1 -- 172.21.15.40:0/1574899597 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 15) v1 ==== 137798+0+0 (secure 0 0 0) 0x7f5538018020 con 0x7f55481021c0 2024-03-21T10:28:57.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.025+0000 7f552ffff700 1 --2- 172.21.15.40:0/1574899597 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f55300a2fa0 0x7f55300a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:28:57.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.025+0000 7f552ffff700 1 -- 172.21.15.40:0/1574899597 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f55380cd0c0 con 0x7f55481021c0 2024-03-21T10:28:57.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.026+0000 7f55467fc700 1 --2- 172.21.15.40:0/1574899597 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f55300a2fa0 0x7f55300a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:28:57.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.027+0000 7f55467fc700 1 --2- 172.21.15.40:0/1574899597 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f55300a2fa0 0x7f55300a5460 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f553c005ea0 tx=0x7f553c005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:28:57.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.030+0000 7f552ffff700 1 -- 172.21.15.40:0/1574899597 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f553801f020 con 0x7f55481021c0 2024-03-21T10:28:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:28:57.249+0000 7f554d45c700 1 -- 172.21.15.40:0/1574899597 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f55480625d0 con 0x7f55300a2fa0 2024-03-21T10:28:57.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:57 smithi040 ceph-mon[36154]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:28:57.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:57 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:28:57.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:57 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:28:57.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:57 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:57.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:57 smithi031 ceph-mon[31673]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:28:57.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:57 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:28:57.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:57 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:28:57.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:57 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:57.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:57 smithi070 ceph-mon[36330]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:28:57.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:57 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:28:57.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:57 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:28:57.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:57 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:58.741 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:58 smithi040 ceph-mon[36154]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:28:58.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:58 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:58.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:58 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:58.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:58 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:28:58.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:58 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:58.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:58 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:58.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:58 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:28:58.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:58 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:58.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:58 smithi031 ceph-mon[31673]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:28:58.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:58 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:58.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:58 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:58.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:58 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:28:58.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:58 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:58.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:58 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:58.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:58 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:28:58.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:58 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:58.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:58 smithi070 ceph-mon[36330]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:28:58.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:58 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:58.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:58 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:58.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:58 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:28:58.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:58 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:58.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:58 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:28:58.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:58 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:28:58.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:58 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:28:59.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:59 smithi031 ceph-mon[31673]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:28:59.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:59 smithi031 ceph-mon[31673]: Detected new or changed devices on smithi031 2024-03-21T10:28:59.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:28:59 smithi031 ceph-mon[31673]: Adjusting osd_memory_target on smithi031 to 17095M 2024-03-21T10:28:59.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:59 smithi070 ceph-mon[36330]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:28:59.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:59 smithi070 ceph-mon[36330]: Detected new or changed devices on smithi031 2024-03-21T10:28:59.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:28:59 smithi070 ceph-mon[36330]: Adjusting osd_memory_target on smithi031 to 17095M 2024-03-21T10:28:59.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:59 smithi040 ceph-mon[36154]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:28:59.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:59 smithi040 ceph-mon[36154]: Detected new or changed devices on smithi031 2024-03-21T10:28:59.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:28:59 smithi040 ceph-mon[36154]: Adjusting osd_memory_target on smithi031 to 17095M 2024-03-21T10:29:00.763 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:00 smithi031 ceph-mon[31673]: from='client.? 172.21.15.40:0/2163813116' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76a63504-beab-4354-a6c3-eebc3b16ad03"}]: dispatch 2024-03-21T10:29:00.763 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:00 smithi031 ceph-mon[31673]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76a63504-beab-4354-a6c3-eebc3b16ad03"}]: dispatch 2024-03-21T10:29:00.763 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:00 smithi031 ceph-mon[31673]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76a63504-beab-4354-a6c3-eebc3b16ad03"}]': finished 2024-03-21T10:29:00.763 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:00 smithi031 ceph-mon[31673]: osdmap e10: 2 total, 1 up, 2 in 2024-03-21T10:29:00.764 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:00 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T10:29:00.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:00 smithi070 ceph-mon[36330]: from='client.? 172.21.15.40:0/2163813116' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76a63504-beab-4354-a6c3-eebc3b16ad03"}]: dispatch 2024-03-21T10:29:00.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:00 smithi070 ceph-mon[36330]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76a63504-beab-4354-a6c3-eebc3b16ad03"}]: dispatch 2024-03-21T10:29:00.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:00 smithi070 ceph-mon[36330]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76a63504-beab-4354-a6c3-eebc3b16ad03"}]': finished 2024-03-21T10:29:00.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:00 smithi070 ceph-mon[36330]: osdmap e10: 2 total, 1 up, 2 in 2024-03-21T10:29:00.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:00 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T10:29:00.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:00 smithi040 ceph-mon[36154]: from='client.? 172.21.15.40:0/2163813116' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76a63504-beab-4354-a6c3-eebc3b16ad03"}]: dispatch 2024-03-21T10:29:00.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:00 smithi040 ceph-mon[36154]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76a63504-beab-4354-a6c3-eebc3b16ad03"}]: dispatch 2024-03-21T10:29:00.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:00 smithi040 ceph-mon[36154]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76a63504-beab-4354-a6c3-eebc3b16ad03"}]': finished 2024-03-21T10:29:00.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:00 smithi040 ceph-mon[36154]: osdmap e10: 2 total, 1 up, 2 in 2024-03-21T10:29:00.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:00 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T10:29:01.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:01 smithi031 ceph-mon[31673]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:01.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:01 smithi031 ceph-mon[31673]: from='client.? 172.21.15.40:0/940435592' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:29:01.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:01 smithi070 ceph-mon[36330]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:01.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:01 smithi070 ceph-mon[36330]: from='client.? 172.21.15.40:0/940435592' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:29:01.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:01 smithi040 ceph-mon[36154]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:01.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:01 smithi040 ceph-mon[36154]: from='client.? 172.21.15.40:0/940435592' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:29:03.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:03 smithi031 ceph-mon[31673]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:03.816 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:03 smithi040 ceph-mon[36154]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:03.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:03 smithi070 ceph-mon[36330]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:05.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:05 smithi040 ceph-mon[36154]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:05.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:05 smithi031 ceph-mon[31673]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:05.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:05 smithi070 ceph-mon[36330]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:07.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:07 smithi031 ceph-mon[31673]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:07.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:07 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T10:29:07.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:07 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:29:07.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:07 smithi070 ceph-mon[36330]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:07.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:07 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T10:29:07.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:07 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:29:07.838 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:07 smithi040 ceph-mon[36154]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:07.838 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:07 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T10:29:07.838 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:07 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:29:08.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:08 smithi031 ceph-mon[31673]: Deploying daemon osd.1 on smithi040 2024-03-21T10:29:08.813 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:08 smithi070 ceph-mon[36330]: Deploying daemon osd.1 on smithi040 2024-03-21T10:29:08.841 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:08 smithi040 ceph-mon[36154]: Deploying daemon osd.1 on smithi040 2024-03-21T10:29:09.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:09 smithi040 ceph-mon[36154]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:09.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:09 smithi031 ceph-mon[31673]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:09.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:09 smithi070 ceph-mon[36330]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:11.633 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:11 smithi040 ceph-mon[36154]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:11.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:11 smithi031 ceph-mon[31673]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:11.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:11 smithi070 ceph-mon[36330]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:13.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:13 smithi031 ceph-mon[31673]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:13.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:13 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:13.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:13 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:13.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:13 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:29:13.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:13 smithi070 ceph-mon[36330]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:13.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:13 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:13.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:13 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:13.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:13 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:29:13.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:13 smithi040 ceph-mon[36154]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:13.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:13 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:13.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:13 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:13.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:13 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:29:15.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:15 smithi040 ceph-mon[36154]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:15.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:15 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:15.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:15 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:15.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:15 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:29:15.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:15 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:29:15.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:15 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:15.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:15 smithi031 ceph-mon[31673]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:15.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:15 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:15.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:15 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:15.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:15 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:29:15.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:15 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:29:15.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:15 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:15.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:15 smithi070 ceph-mon[36330]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:15.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:15 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:15.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:15 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:15.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:15 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:29:15.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:15 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:29:15.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:15 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:16.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:29:16.393+0000 7f552ffff700 1 -- 172.21.15.40:0/1574899597 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f55480625d0 con 0x7f55300a2fa0 2024-03-21T10:29:16.393 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 1 on host 'smithi040' 2024-03-21T10:29:16.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:29:16.395+0000 7f554d45c700 1 -- 172.21.15.40:0/1574899597 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f55300a2fa0 msgr2=0x7f55300a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:29:16.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:29:16.395+0000 7f554d45c700 1 --2- 172.21.15.40:0/1574899597 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f55300a2fa0 0x7f55300a5460 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f553c005ea0 tx=0x7f553c005e30 comp rx=0 tx=0).stop 2024-03-21T10:29:16.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:29:16.395+0000 7f554d45c700 1 -- 172.21.15.40:0/1574899597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55481021c0 msgr2=0x7f55481983a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:29:16.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:29:16.395+0000 7f554d45c700 1 --2- 172.21.15.40:0/1574899597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55481021c0 0x7f55481983a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f553800b500 tx=0x7f553800b8c0 comp rx=0 tx=0).stop 2024-03-21T10:29:16.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:29:16.395+0000 7f554d45c700 1 -- 172.21.15.40:0/1574899597 shutdown_connections 2024-03-21T10:29:16.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:29:16.395+0000 7f554d45c700 1 --2- 172.21.15.40:0/1574899597 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f55300a2fa0 0x7f55300a5460 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:29:16.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:29:16.395+0000 7f554d45c700 1 --2- 172.21.15.40:0/1574899597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5548103d60 0x7f554819f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:29:16.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:29:16.395+0000 7f554d45c700 1 --2- 172.21.15.40:0/1574899597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55481021c0 0x7f55481983a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:29:16.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:29:16.395+0000 7f554d45c700 1 --2- 172.21.15.40:0/1574899597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55481033c0 0x7f55481988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:29:16.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:29:16.395+0000 7f554d45c700 1 -- 172.21.15.40:0/1574899597 >> 172.21.15.40:0/1574899597 conn(0x7f55480fd7a0 msgr2=0x7f5548106660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:29:16.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:29:16.395+0000 7f554d45c700 1 -- 172.21.15.40:0/1574899597 shutdown_connections 2024-03-21T10:29:16.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:29:16.395+0000 7f554d45c700 1 -- 172.21.15.40:0/1574899597 wait complete. 2024-03-21T10:29:16.990 DEBUG:teuthology.orchestra.run.smithi040:osd.1> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.1.service 2024-03-21T10:29:16.992 INFO:tasks.cephadm:Deploying osd.2 on smithi070 with /dev/vg_nvme/lv_4... 2024-03-21T10:29:16.992 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-21T10:29:17.235 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.c/config 2024-03-21T10:29:17.660 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:17 smithi070 ceph-mon[36330]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:17.660 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:17 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:17.660 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:17 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:29:17.660 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:17 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:17.660 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:17 smithi070 ceph-mon[36330]: from='osd.1 [v2:172.21.15.40:6800/618840037,v1:172.21.15.40:6801/618840037]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T10:29:17.660 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:17 smithi070 ceph-mon[36330]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T10:29:17.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:17 smithi040 ceph-mon[36154]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:17.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:17 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:17.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:17 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:29:17.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:17 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:17.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:17 smithi040 ceph-mon[36154]: from='osd.1 [v2:172.21.15.40:6800/618840037,v1:172.21.15.40:6801/618840037]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T10:29:17.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:17 smithi040 ceph-mon[36154]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T10:29:17.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:17 smithi031 ceph-mon[31673]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:17.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:17 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:17.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:17 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:29:17.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:17 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:17.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:17 smithi031 ceph-mon[31673]: from='osd.1 [v2:172.21.15.40:6800/618840037,v1:172.21.15.40:6801/618840037]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T10:29:17.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:17 smithi031 ceph-mon[31673]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T10:29:18.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:18 smithi070 ceph-mon[36330]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-21T10:29:18.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:18 smithi070 ceph-mon[36330]: from='osd.1 [v2:172.21.15.40:6800/618840037,v1:172.21.15.40:6801/618840037]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-21T10:29:18.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:18 smithi070 ceph-mon[36330]: osdmap e11: 2 total, 1 up, 2 in 2024-03-21T10:29:18.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:18 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T10:29:18.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:18 smithi070 ceph-mon[36330]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-21T10:29:18.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:18 smithi040 ceph-mon[36154]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-21T10:29:18.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:18 smithi040 ceph-mon[36154]: from='osd.1 [v2:172.21.15.40:6800/618840037,v1:172.21.15.40:6801/618840037]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-21T10:29:18.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:18 smithi040 ceph-mon[36154]: osdmap e11: 2 total, 1 up, 2 in 2024-03-21T10:29:18.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:18 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T10:29:18.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:18 smithi040 ceph-mon[36154]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-21T10:29:18.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:18 smithi031 ceph-mon[31673]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-21T10:29:18.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:18 smithi031 ceph-mon[31673]: from='osd.1 [v2:172.21.15.40:6800/618840037,v1:172.21.15.40:6801/618840037]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-21T10:29:18.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:18 smithi031 ceph-mon[31673]: osdmap e11: 2 total, 1 up, 2 in 2024-03-21T10:29:18.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:18 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T10:29:18.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:18 smithi031 ceph-mon[31673]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-21T10:29:18.933 INFO:journalctl@ceph.osd.1.smithi040.stdout:Mar 21 10:29:18 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-1[40744]: 2024-03-21T10:29:18.856+0000 7fee98b9a700 -1 osd.1 0 waiting for initial osdmap 2024-03-21T10:29:18.933 INFO:journalctl@ceph.osd.1.smithi040.stdout:Mar 21 10:29:18 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-1[40744]: 2024-03-21T10:29:18.860+0000 7fee90985700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T10:29:19.226 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-21T10:29:19.226 INFO:teuthology.orchestra.run.smithi070.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-21T10:29:19.226 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10+0 records in 2024-03-21T10:29:19.226 INFO:teuthology.orchestra.run.smithi070.stderr:10+0 records out 2024-03-21T10:29:19.226 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00845538 s, 1.2 GB/s 2024-03-21T10:29:19.226 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping successful for: 2024-03-21T10:29:19.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:19 smithi040 ceph-mon[36154]: pgmap v73: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:19.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:19 smithi040 ceph-mon[36154]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-03-21T10:29:19.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:19 smithi040 ceph-mon[36154]: osdmap e12: 2 total, 1 up, 2 in 2024-03-21T10:29:19.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:19 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T10:29:19.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:19 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T10:29:19.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:19 smithi040 ceph-mon[36154]: from='osd.1 ' entity='osd.1' 2024-03-21T10:29:19.802 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:19 smithi070 ceph-mon[36330]: pgmap v73: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:19.802 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:19 smithi070 ceph-mon[36330]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-03-21T10:29:19.802 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:19 smithi070 ceph-mon[36330]: osdmap e12: 2 total, 1 up, 2 in 2024-03-21T10:29:19.803 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:19 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T10:29:19.803 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:19 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T10:29:19.803 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:19 smithi070 ceph-mon[36330]: from='osd.1 ' entity='osd.1' 2024-03-21T10:29:19.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:19 smithi031 ceph-mon[31673]: pgmap v73: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:19.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:19 smithi031 ceph-mon[31673]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-03-21T10:29:19.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:19 smithi031 ceph-mon[31673]: osdmap e12: 2 total, 1 up, 2 in 2024-03-21T10:29:19.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:19 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T10:29:19.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:19 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T10:29:19.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:19 smithi031 ceph-mon[31673]: from='osd.1 ' entity='osd.1' 2024-03-21T10:29:19.827 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi070:vg_nvme/lv_4 2024-03-21T10:29:20.061 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.c/config 2024-03-21T10:29:20.489 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:20 smithi070 ceph-mon[36330]: purged_snaps scrub starts 2024-03-21T10:29:20.489 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:20 smithi070 ceph-mon[36330]: purged_snaps scrub ok 2024-03-21T10:29:20.489 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:20 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T10:29:20.489 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:20 smithi070 ceph-mon[36330]: osd.1 [v2:172.21.15.40:6800/618840037,v1:172.21.15.40:6801/618840037] boot 2024-03-21T10:29:20.490 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:20 smithi070 ceph-mon[36330]: osdmap e13: 2 total, 2 up, 2 in 2024-03-21T10:29:20.490 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:20 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T10:29:20.490 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:20 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:20.490 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:20 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:20.490 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:20 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:29:20.490 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:20 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:20.490 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:20 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:29:20.490 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:20 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:29:20.490 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:20 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:20.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:20 smithi040 ceph-mon[36154]: purged_snaps scrub starts 2024-03-21T10:29:20.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:20 smithi040 ceph-mon[36154]: purged_snaps scrub ok 2024-03-21T10:29:20.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:20 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T10:29:20.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:20 smithi040 ceph-mon[36154]: osd.1 [v2:172.21.15.40:6800/618840037,v1:172.21.15.40:6801/618840037] boot 2024-03-21T10:29:20.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:20 smithi040 ceph-mon[36154]: osdmap e13: 2 total, 2 up, 2 in 2024-03-21T10:29:20.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:20 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T10:29:20.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:20 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:20.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:20 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:20.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:20 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:29:20.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:20 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:20.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:20 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:29:20.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:20 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:29:20.684 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:20 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:20.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:20 smithi031 ceph-mon[31673]: purged_snaps scrub starts 2024-03-21T10:29:20.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:20 smithi031 ceph-mon[31673]: purged_snaps scrub ok 2024-03-21T10:29:20.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:20 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T10:29:20.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:20 smithi031 ceph-mon[31673]: osd.1 [v2:172.21.15.40:6800/618840037,v1:172.21.15.40:6801/618840037] boot 2024-03-21T10:29:20.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:20 smithi031 ceph-mon[31673]: osdmap e13: 2 total, 2 up, 2 in 2024-03-21T10:29:20.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:20 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T10:29:20.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:20 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:20.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:20 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:20.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:20 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:29:20.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:20 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:20.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:20 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:29:20.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:20 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:29:20.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:20 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:21.458 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:21 smithi070 ceph-mon[36330]: pgmap v76: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:21.459 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:21 smithi070 ceph-mon[36330]: Detected new or changed devices on smithi040 2024-03-21T10:29:21.459 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:21 smithi070 ceph-mon[36330]: Adjusting osd_memory_target on smithi040 to 17084M 2024-03-21T10:29:21.459 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:21 smithi070 ceph-mon[36330]: osdmap e14: 2 total, 2 up, 2 in 2024-03-21T10:29:21.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.655+0000 7ff79d2c8700 1 -- 172.21.15.70:0/3398493913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7981022c0 msgr2=0x7ff798102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:29:21.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.655+0000 7ff79d2c8700 1 --2- 172.21.15.70:0/3398493913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7981022c0 0x7ff798102720 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff780009990 tx=0x7ff780009ca0 comp rx=0 tx=0).stop 2024-03-21T10:29:21.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.655+0000 7ff79d2c8700 1 -- 172.21.15.70:0/3398493913 shutdown_connections 2024-03-21T10:29:21.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.655+0000 7ff79d2c8700 1 --2- 172.21.15.70:0/3398493913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff798102c60 0x7ff798109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:29:21.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.655+0000 7ff79d2c8700 1 --2- 172.21.15.70:0/3398493913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7981010c0 0x7ff7981014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:29:21.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.655+0000 7ff79d2c8700 1 --2- 172.21.15.70:0/3398493913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7981022c0 0x7ff798102720 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:29:21.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.655+0000 7ff79d2c8700 1 -- 172.21.15.70:0/3398493913 >> 172.21.15.70:0/3398493913 conn(0x7ff7980fc660 msgr2=0x7ff7980feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:29:21.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.655+0000 7ff79d2c8700 1 -- 172.21.15.70:0/3398493913 shutdown_connections 2024-03-21T10:29:21.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.655+0000 7ff79d2c8700 1 -- 172.21.15.70:0/3398493913 wait complete. 2024-03-21T10:29:21.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.656+0000 7ff79d2c8700 1 Processor -- start 2024-03-21T10:29:21.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.656+0000 7ff79d2c8700 1 -- start start 2024-03-21T10:29:21.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.656+0000 7ff79d2c8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7981010c0 0x7ff7981973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:29:21.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.656+0000 7ff79d2c8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7981022c0 0x7ff7981978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:29:21.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.656+0000 7ff79d2c8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff798102c60 0x7ff79819e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:29:21.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.656+0000 7ff79d2c8700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79810cd00 con 0x7ff798102c60 2024-03-21T10:29:21.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.656+0000 7ff79d2c8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79810cb80 con 0x7ff7981022c0 2024-03-21T10:29:21.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.656+0000 7ff79d2c8700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79810ce80 con 0x7ff7981010c0 2024-03-21T10:29:21.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.656+0000 7ff796ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7981010c0 0x7ff7981973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:29:21.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.656+0000 7ff796ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7981010c0 0x7ff7981973a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38994/0 (socket says 172.21.15.70:38994) 2024-03-21T10:29:21.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.656+0000 7ff796ffd700 1 -- 172.21.15.70:0/3958689097 learned_addr learned my addr 172.21.15.70:0/3958689097 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-03-21T10:29:21.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.656+0000 7ff78e7fc700 1 --2- 172.21.15.70:0/3958689097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7981022c0 0x7ff7981978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:29:21.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.656+0000 7ff796ffd700 1 -- 172.21.15.70:0/3958689097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7981022c0 msgr2=0x7ff7981978e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:29:21.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.656+0000 7ff796ffd700 1 --2- 172.21.15.70:0/3958689097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7981022c0 0x7ff7981978e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:29:21.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.656+0000 7ff796ffd700 1 -- 172.21.15.70:0/3958689097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff798102c60 msgr2=0x7ff79819e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:29:21.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.656+0000 7ff796ffd700 1 --2- 172.21.15.70:0/3958689097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff798102c60 0x7ff79819e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:29:21.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.656+0000 7ff796ffd700 1 -- 172.21.15.70:0/3958689097 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79819f000 con 0x7ff7981010c0 2024-03-21T10:29:21.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.657+0000 7ff796ffd700 1 --2- 172.21.15.70:0/3958689097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7981010c0 0x7ff7981973a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff78800d9b0 tx=0x7ff78800dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:29:21.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.657+0000 7ff794ff9700 1 -- 172.21.15.70:0/3958689097 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff788017030 con 0x7ff7981010c0 2024-03-21T10:29:21.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.657+0000 7ff794ff9700 1 -- 172.21.15.70:0/3958689097 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff788004500 con 0x7ff7981010c0 2024-03-21T10:29:21.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.657+0000 7ff79d2c8700 1 -- 172.21.15.70:0/3958689097 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff79819f2f0 con 0x7ff7981010c0 2024-03-21T10:29:21.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.657+0000 7ff794ff9700 1 -- 172.21.15.70:0/3958689097 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff788003aa0 con 0x7ff7981010c0 2024-03-21T10:29:21.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.657+0000 7ff79d2c8700 1 -- 172.21.15.70:0/3958689097 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff79819f8b0 con 0x7ff7981010c0 2024-03-21T10:29:21.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.658+0000 7ff79d2c8700 1 -- 172.21.15.70:0/3958689097 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff798068e50 con 0x7ff7981010c0 2024-03-21T10:29:21.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.659+0000 7ff794ff9700 1 -- 172.21.15.70:0/3958689097 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 15) v1 ==== 137798+0+0 (secure 0 0 0) 0x7ff78801a070 con 0x7ff7981010c0 2024-03-21T10:29:21.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.659+0000 7ff794ff9700 1 --2- 172.21.15.70:0/3958689097 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff7840a2f20 0x7ff7840a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:29:21.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.659+0000 7ff794ff9700 1 -- 172.21.15.70:0/3958689097 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7ff7880cd820 con 0x7ff7981010c0 2024-03-21T10:29:21.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.660+0000 7ff78e7fc700 1 --2- 172.21.15.70:0/3958689097 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff7840a2f20 0x7ff7840a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:29:21.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.661+0000 7ff78e7fc700 1 --2- 172.21.15.70:0/3958689097 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff7840a2f20 0x7ff7840a53e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff78000b5c0 tx=0x7ff780003960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:29:21.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.663+0000 7ff794ff9700 1 -- 172.21.15.70:0/3958689097 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff78800e350 con 0x7ff7981010c0 2024-03-21T10:29:21.672 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:21 smithi040 ceph-mon[36154]: pgmap v76: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:21.672 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:21 smithi040 ceph-mon[36154]: Detected new or changed devices on smithi040 2024-03-21T10:29:21.672 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:21 smithi040 ceph-mon[36154]: Adjusting osd_memory_target on smithi040 to 17084M 2024-03-21T10:29:21.672 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:21 smithi040 ceph-mon[36154]: osdmap e14: 2 total, 2 up, 2 in 2024-03-21T10:29:21.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:21 smithi031 ceph-mon[31673]: pgmap v76: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-21T10:29:21.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:21 smithi031 ceph-mon[31673]: Detected new or changed devices on smithi040 2024-03-21T10:29:21.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:21 smithi031 ceph-mon[31673]: Adjusting osd_memory_target on smithi040 to 17084M 2024-03-21T10:29:21.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:21 smithi031 ceph-mon[31673]: osdmap e14: 2 total, 2 up, 2 in 2024-03-21T10:29:21.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:21.868+0000 7ff79d2c8700 1 -- 172.21.15.70:0/3958689097 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7980625d0 con 0x7ff7840a2f20 2024-03-21T10:29:22.635 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:22 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:29:22.635 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:22 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:29:22.635 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:22 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:29:22.647 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:22 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:29:22.647 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:22 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:29:22.647 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:22 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:29:22.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:22 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:29:22.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:22 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:29:22.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:22 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:29:23.628 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:23 smithi070 ceph-mon[36330]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:29:23.629 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:23 smithi070 ceph-mon[36330]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:23.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:23 smithi040 ceph-mon[36154]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:29:23.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:23 smithi040 ceph-mon[36154]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:23.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:23 smithi031 ceph-mon[31673]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:29:23.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:23 smithi031 ceph-mon[31673]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:24.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:24 smithi040 ceph-mon[36154]: from='client.? 172.21.15.70:0/3049058519' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa191bd6-9a21-4ee2-8adf-8a63c9cfd821"}]: dispatch 2024-03-21T10:29:24.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:24 smithi040 ceph-mon[36154]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa191bd6-9a21-4ee2-8adf-8a63c9cfd821"}]: dispatch 2024-03-21T10:29:24.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:24 smithi040 ceph-mon[36154]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa191bd6-9a21-4ee2-8adf-8a63c9cfd821"}]': finished 2024-03-21T10:29:24.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:24 smithi040 ceph-mon[36154]: osdmap e15: 3 total, 2 up, 3 in 2024-03-21T10:29:24.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:24 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T10:29:24.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:24 smithi070 ceph-mon[36330]: from='client.? 172.21.15.70:0/3049058519' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa191bd6-9a21-4ee2-8adf-8a63c9cfd821"}]: dispatch 2024-03-21T10:29:24.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:24 smithi070 ceph-mon[36330]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa191bd6-9a21-4ee2-8adf-8a63c9cfd821"}]: dispatch 2024-03-21T10:29:24.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:24 smithi070 ceph-mon[36330]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa191bd6-9a21-4ee2-8adf-8a63c9cfd821"}]': finished 2024-03-21T10:29:24.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:24 smithi070 ceph-mon[36330]: osdmap e15: 3 total, 2 up, 3 in 2024-03-21T10:29:24.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:24 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T10:29:24.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:24 smithi031 ceph-mon[31673]: from='client.? 172.21.15.70:0/3049058519' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa191bd6-9a21-4ee2-8adf-8a63c9cfd821"}]: dispatch 2024-03-21T10:29:24.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:24 smithi031 ceph-mon[31673]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa191bd6-9a21-4ee2-8adf-8a63c9cfd821"}]: dispatch 2024-03-21T10:29:24.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:24 smithi031 ceph-mon[31673]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa191bd6-9a21-4ee2-8adf-8a63c9cfd821"}]': finished 2024-03-21T10:29:24.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:24 smithi031 ceph-mon[31673]: osdmap e15: 3 total, 2 up, 3 in 2024-03-21T10:29:24.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:24 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T10:29:25.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:25 smithi040 ceph-mon[36154]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:25.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:25 smithi040 ceph-mon[36154]: from='client.? 172.21.15.70:0/2180258713' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:29:25.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:25 smithi031 ceph-mon[31673]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:25.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:25 smithi031 ceph-mon[31673]: from='client.? 172.21.15.70:0/2180258713' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:29:25.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:25 smithi070 ceph-mon[36330]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:25.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:25 smithi070 ceph-mon[36330]: from='client.? 172.21.15.70:0/2180258713' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:29:27.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:27 smithi040 ceph-mon[36154]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:27.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:27 smithi031 ceph-mon[31673]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:27.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:27 smithi070 ceph-mon[36330]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:29.498 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:29 smithi070 ceph-mon[36330]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:29.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:29 smithi040 ceph-mon[36154]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:29.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:29 smithi031 ceph-mon[31673]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:31.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:31 smithi070 ceph-mon[36330]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:31.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:31 smithi040 ceph-mon[36154]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:31.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:31 smithi031 ceph-mon[31673]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:32.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:32 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T10:29:32.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:32 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:29:32.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:32 smithi040 ceph-mon[36154]: Deploying daemon osd.2 on smithi070 2024-03-21T10:29:32.683 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:32 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T10:29:32.683 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:32 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:29:32.683 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:32 smithi070 ceph-mon[36330]: Deploying daemon osd.2 on smithi070 2024-03-21T10:29:32.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:32 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T10:29:32.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:32 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:29:32.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:32 smithi031 ceph-mon[31673]: Deploying daemon osd.2 on smithi070 2024-03-21T10:29:33.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:33 smithi070 ceph-mon[36330]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:33.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:33 smithi040 ceph-mon[36154]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:33.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:33 smithi031 ceph-mon[31673]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:35.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:35 smithi040 ceph-mon[36154]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:35.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:35 smithi031 ceph-mon[31673]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:35.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:35 smithi070 ceph-mon[36330]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:38.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:37 smithi031 ceph-mon[31673]: pgmap v86: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:38.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:37 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:38.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:37 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:38.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:37 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:29:38.064 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:37 smithi070 ceph-mon[36330]: pgmap v86: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:38.064 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:37 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:38.064 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:37 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:38.064 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:37 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:29:38.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:37 smithi040 ceph-mon[36154]: pgmap v86: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:38.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:37 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:38.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:37 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:38.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:37 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:29:40.039 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:39 smithi070 ceph-mon[36330]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:40.039 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:39 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:40.039 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:39 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:40.039 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:39 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:29:40.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:39 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:29:40.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:39 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:40.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:39 smithi031 ceph-mon[31673]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:40.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:39 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:40.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:39 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:40.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:39 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:29:40.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:39 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:29:40.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:39 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:40.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:39 smithi040 ceph-mon[36154]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:40.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:39 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:40.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:39 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:40.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:39 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:29:40.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:39 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:29:40.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:39 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:40.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:40.405+0000 7ff794ff9700 1 -- 172.21.15.70:0/3958689097 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff7980625d0 con 0x7ff7840a2f20 2024-03-21T10:29:40.404 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 2 on host 'smithi070' 2024-03-21T10:29:40.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:40.407+0000 7ff79d2c8700 1 -- 172.21.15.70:0/3958689097 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff7840a2f20 msgr2=0x7ff7840a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:29:40.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:40.407+0000 7ff79d2c8700 1 --2- 172.21.15.70:0/3958689097 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff7840a2f20 0x7ff7840a53e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff78000b5c0 tx=0x7ff780003960 comp rx=0 tx=0).stop 2024-03-21T10:29:40.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:40.407+0000 7ff79d2c8700 1 -- 172.21.15.70:0/3958689097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7981010c0 msgr2=0x7ff7981973a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:29:40.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:40.407+0000 7ff79d2c8700 1 --2- 172.21.15.70:0/3958689097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7981010c0 0x7ff7981973a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff78800d9b0 tx=0x7ff78800dcc0 comp rx=0 tx=0).stop 2024-03-21T10:29:40.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:40.407+0000 7ff79d2c8700 1 -- 172.21.15.70:0/3958689097 shutdown_connections 2024-03-21T10:29:40.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:40.407+0000 7ff79d2c8700 1 --2- 172.21.15.70:0/3958689097 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff7840a2f20 0x7ff7840a53e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:29:40.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:40.407+0000 7ff79d2c8700 1 --2- 172.21.15.70:0/3958689097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff798102c60 0x7ff79819e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:29:40.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:40.407+0000 7ff79d2c8700 1 --2- 172.21.15.70:0/3958689097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7981010c0 0x7ff7981973a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:29:40.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:40.407+0000 7ff79d2c8700 1 --2- 172.21.15.70:0/3958689097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7981022c0 0x7ff7981978e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:29:40.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:40.407+0000 7ff79d2c8700 1 -- 172.21.15.70:0/3958689097 >> 172.21.15.70:0/3958689097 conn(0x7ff7980fc660 msgr2=0x7ff798105610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:29:40.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:40.407+0000 7ff79d2c8700 1 -- 172.21.15.70:0/3958689097 shutdown_connections 2024-03-21T10:29:40.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:29:40.407+0000 7ff79d2c8700 1 -- 172.21.15.70:0/3958689097 wait complete. 2024-03-21T10:29:40.974 DEBUG:teuthology.orchestra.run.smithi070:osd.2> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.2.service 2024-03-21T10:29:40.976 INFO:tasks.cephadm:Deploying osd.3 on smithi031 with /dev/vg_nvme/lv_3... 2024-03-21T10:29:40.976 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-21T10:29:41.455 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:29:41.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:41 smithi070 ceph-mon[36330]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:41.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:41 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:41.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:41 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:41.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:41 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:29:41.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:41 smithi070 ceph-mon[36330]: from='osd.2 [v2:172.21.15.70:6800/2524184117,v1:172.21.15.70:6801/2524184117]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T10:29:41.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:41 smithi070 ceph-mon[36330]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T10:29:41.662 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:41 smithi031 ceph-mon[31673]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:41.663 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:41 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:41.663 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:41 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:41.663 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:41 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:29:41.663 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:41 smithi031 ceph-mon[31673]: from='osd.2 [v2:172.21.15.70:6800/2524184117,v1:172.21.15.70:6801/2524184117]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T10:29:41.663 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:41 smithi031 ceph-mon[31673]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T10:29:41.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:41 smithi040 ceph-mon[36154]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:41.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:41 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:41.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:41 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:41.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:41 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:29:41.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:41 smithi040 ceph-mon[36154]: from='osd.2 [v2:172.21.15.70:6800/2524184117,v1:172.21.15.70:6801/2524184117]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T10:29:41.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:41 smithi040 ceph-mon[36154]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T10:29:42.662 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:42 smithi070 ceph-mon[36330]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T10:29:42.662 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:42 smithi070 ceph-mon[36330]: from='osd.2 [v2:172.21.15.70:6800/2524184117,v1:172.21.15.70:6801/2524184117]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-03-21T10:29:42.662 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:42 smithi070 ceph-mon[36330]: osdmap e16: 3 total, 2 up, 3 in 2024-03-21T10:29:42.662 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:42 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T10:29:42.662 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:42 smithi070 ceph-mon[36330]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-03-21T10:29:42.662 INFO:journalctl@ceph.osd.2.smithi070.stdout:Mar 21 10:29:42 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-2[41483]: 2024-03-21T10:29:42.388+0000 7f05dda80700 -1 osd.2 0 waiting for initial osdmap 2024-03-21T10:29:42.662 INFO:journalctl@ceph.osd.2.smithi070.stdout:Mar 21 10:29:42 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-2[41483]: 2024-03-21T10:29:42.393+0000 7f05d9873700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T10:29:42.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:42 smithi040 ceph-mon[36154]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T10:29:42.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:42 smithi040 ceph-mon[36154]: from='osd.2 [v2:172.21.15.70:6800/2524184117,v1:172.21.15.70:6801/2524184117]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-03-21T10:29:42.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:42 smithi040 ceph-mon[36154]: osdmap e16: 3 total, 2 up, 3 in 2024-03-21T10:29:42.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:42 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T10:29:42.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:42 smithi040 ceph-mon[36154]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-03-21T10:29:42.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:42 smithi031 ceph-mon[31673]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T10:29:42.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:42 smithi031 ceph-mon[31673]: from='osd.2 [v2:172.21.15.70:6800/2524184117,v1:172.21.15.70:6801/2524184117]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-03-21T10:29:42.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:42 smithi031 ceph-mon[31673]: osdmap e16: 3 total, 2 up, 3 in 2024-03-21T10:29:42.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:42 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T10:29:42.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:42 smithi031 ceph-mon[31673]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-03-21T10:29:43.509 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:43 smithi031 ceph-mon[31673]: pgmap v90: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:43.509 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:43 smithi031 ceph-mon[31673]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-03-21T10:29:43.509 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:43 smithi031 ceph-mon[31673]: osdmap e17: 3 total, 2 up, 3 in 2024-03-21T10:29:43.509 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T10:29:43.509 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T10:29:43.522 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:43 smithi070 ceph-mon[36330]: pgmap v90: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:43.522 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:43 smithi070 ceph-mon[36330]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-03-21T10:29:43.522 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:43 smithi070 ceph-mon[36330]: osdmap e17: 3 total, 2 up, 3 in 2024-03-21T10:29:43.523 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:43 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T10:29:43.523 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:43 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T10:29:43.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:43 smithi040 ceph-mon[36154]: pgmap v90: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:43.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:43 smithi040 ceph-mon[36154]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-03-21T10:29:43.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:43 smithi040 ceph-mon[36154]: osdmap e17: 3 total, 2 up, 3 in 2024-03-21T10:29:43.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:43 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T10:29:43.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:43 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T10:29:44.511 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-21T10:29:44.511 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-21T10:29:44.512 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-03-21T10:29:44.512 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-03-21T10:29:44.512 INFO:teuthology.orchestra.run.smithi031.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0324997 s, 323 MB/s 2024-03-21T10:29:44.512 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-03-21T10:29:44.753 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:44 smithi031 ceph-mon[31673]: purged_snaps scrub starts 2024-03-21T10:29:44.754 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:44 smithi031 ceph-mon[31673]: purged_snaps scrub ok 2024-03-21T10:29:44.754 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:44 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T10:29:44.754 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:44 smithi031 ceph-mon[31673]: osd.2 [v2:172.21.15.70:6800/2524184117,v1:172.21.15.70:6801/2524184117] boot 2024-03-21T10:29:44.754 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:44 smithi031 ceph-mon[31673]: osdmap e18: 3 total, 3 up, 3 in 2024-03-21T10:29:44.754 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:44 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T10:29:44.754 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:44 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:44.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:44 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:44.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:44 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:29:44.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:44 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:44.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:44 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:29:44.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:44 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:29:44.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:44 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:44.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:44 smithi070 ceph-mon[36330]: purged_snaps scrub starts 2024-03-21T10:29:44.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:44 smithi070 ceph-mon[36330]: purged_snaps scrub ok 2024-03-21T10:29:44.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:44 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T10:29:44.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:44 smithi070 ceph-mon[36330]: osd.2 [v2:172.21.15.70:6800/2524184117,v1:172.21.15.70:6801/2524184117] boot 2024-03-21T10:29:44.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:44 smithi070 ceph-mon[36330]: osdmap e18: 3 total, 3 up, 3 in 2024-03-21T10:29:44.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:44 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T10:29:44.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:44 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:44.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:44 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:44.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:44 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:29:44.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:44 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:44.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:44 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:29:44.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:44 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:29:44.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:44 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:44.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:44 smithi040 ceph-mon[36154]: purged_snaps scrub starts 2024-03-21T10:29:44.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:44 smithi040 ceph-mon[36154]: purged_snaps scrub ok 2024-03-21T10:29:44.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:44 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T10:29:44.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:44 smithi040 ceph-mon[36154]: osd.2 [v2:172.21.15.70:6800/2524184117,v1:172.21.15.70:6801/2524184117] boot 2024-03-21T10:29:44.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:44 smithi040 ceph-mon[36154]: osdmap e18: 3 total, 3 up, 3 in 2024-03-21T10:29:44.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:44 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T10:29:44.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:44 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:44.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:44 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:44.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:44 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:29:44.934 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:44 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:44.935 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:44 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:29:44.935 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:44 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:29:44.935 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:44 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:45.480 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi031:vg_nvme/lv_3 2024-03-21T10:29:45.720 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:45 smithi031 ceph-mon[31673]: Detected new or changed devices on smithi070 2024-03-21T10:29:45.720 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:45 smithi031 ceph-mon[31673]: pgmap v93: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:45.720 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:45 smithi031 ceph-mon[31673]: Adjusting osd_memory_target on smithi070 to 17080M 2024-03-21T10:29:45.720 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:45 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-21T10:29:45.720 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:45 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-21T10:29:45.721 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:45 smithi031 ceph-mon[31673]: osdmap e19: 3 total, 3 up, 3 in 2024-03-21T10:29:45.721 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:45 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T10:29:45.907 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:29:45.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:45 smithi040 ceph-mon[36154]: Detected new or changed devices on smithi070 2024-03-21T10:29:45.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:45 smithi040 ceph-mon[36154]: pgmap v93: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:45.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:45 smithi040 ceph-mon[36154]: Adjusting osd_memory_target on smithi070 to 17080M 2024-03-21T10:29:45.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:45 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-21T10:29:45.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:45 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-21T10:29:45.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:45 smithi040 ceph-mon[36154]: osdmap e19: 3 total, 3 up, 3 in 2024-03-21T10:29:45.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:45 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T10:29:46.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:45 smithi070 ceph-mon[36330]: Detected new or changed devices on smithi070 2024-03-21T10:29:46.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:45 smithi070 ceph-mon[36330]: pgmap v93: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-21T10:29:46.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:45 smithi070 ceph-mon[36330]: Adjusting osd_memory_target on smithi070 to 17080M 2024-03-21T10:29:46.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:45 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-21T10:29:46.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:45 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-21T10:29:46.076 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:45 smithi070 ceph-mon[36330]: osdmap e19: 3 total, 3 up, 3 in 2024-03-21T10:29:46.076 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:45 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T10:29:46.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:46 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-21T10:29:46.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:46 smithi040 ceph-mon[36154]: osdmap e20: 3 total, 3 up, 3 in 2024-03-21T10:29:47.020 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:46 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-21T10:29:47.020 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:46 smithi031 ceph-mon[31673]: osdmap e20: 3 total, 3 up, 3 in 2024-03-21T10:29:47.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:46 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-21T10:29:47.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:46 smithi070 ceph-mon[36330]: osdmap e20: 3 total, 3 up, 3 in 2024-03-21T10:29:47.306 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 21 10:29:47 smithi031 sudo[50566]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T10:29:47.307 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 21 10:29:47 smithi031 sudo[50566]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T10:29:47.307 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 21 10:29:47 smithi031 sudo[50566]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T10:29:47.307 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 21 10:29:47 smithi031 sudo[50566]: pam_unix(sudo:session): session closed for user root 2024-03-21T10:29:47.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:47 smithi031 ceph-mon[31673]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:29:47.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:47 smithi031 ceph-mon[31673]: osdmap e21: 3 total, 3 up, 3 in 2024-03-21T10:29:47.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:47 smithi040 ceph-mon[36154]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:29:47.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:47 smithi040 ceph-mon[36154]: osdmap e21: 3 total, 3 up, 3 in 2024-03-21T10:29:48.057 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 21 10:29:47 smithi031 sudo[50589]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T10:29:48.057 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 21 10:29:47 smithi031 sudo[50589]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T10:29:48.057 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 21 10:29:47 smithi031 sudo[50589]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T10:29:48.057 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 21 10:29:47 smithi031 sudo[50589]: pam_unix(sudo:session): session closed for user root 2024-03-21T10:29:48.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:47 smithi070 ceph-mon[36330]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:29:48.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:47 smithi070 ceph-mon[36330]: osdmap e21: 3 total, 3 up, 3 in 2024-03-21T10:29:48.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.243+0000 7f7705800700 1 -- 172.21.15.31:0/2823976118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77001022b0 msgr2=0x7f7700102710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:29:48.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.243+0000 7f7705800700 1 --2- 172.21.15.31:0/2823976118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77001022b0 0x7f7700102710 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f76ec0099c0 tx=0x7f76ec009cd0 comp rx=0 tx=0).stop 2024-03-21T10:29:48.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.243+0000 7f7705800700 1 -- 172.21.15.31:0/2823976118 shutdown_connections 2024-03-21T10:29:48.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.243+0000 7f7705800700 1 --2- 172.21.15.31:0/2823976118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7700102c50 0x7f7700109500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:29:48.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.243+0000 7f7705800700 1 --2- 172.21.15.31:0/2823976118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77001010b0 0x7f77001014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:29:48.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.243+0000 7f7705800700 1 --2- 172.21.15.31:0/2823976118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77001022b0 0x7f7700102710 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:29:48.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.243+0000 7f7705800700 1 -- 172.21.15.31:0/2823976118 >> 172.21.15.31:0/2823976118 conn(0x7f77000fc650 msgr2=0x7f77000fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:29:48.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.243+0000 7f7705800700 1 -- 172.21.15.31:0/2823976118 shutdown_connections 2024-03-21T10:29:48.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.243+0000 7f7705800700 1 -- 172.21.15.31:0/2823976118 wait complete. 2024-03-21T10:29:48.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.245+0000 7f7705800700 1 Processor -- start 2024-03-21T10:29:48.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.245+0000 7f7705800700 1 -- start start 2024-03-21T10:29:48.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.245+0000 7f7705800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77001010b0 0x7f770019a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:29:48.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.245+0000 7f7705800700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f77001022b0 0x7f770019ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:29:48.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.245+0000 7f7705800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7700102c50 0x7f77001a1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:29:48.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.245+0000 7f7705800700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770010cce0 con 0x7f77001022b0 2024-03-21T10:29:48.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.245+0000 7f7705800700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770010cb60 con 0x7f77001010b0 2024-03-21T10:29:48.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.245+0000 7f7705800700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770010ce60 con 0x7f7700102c50 2024-03-21T10:29:48.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.246+0000 7f76fea00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77001010b0 0x7f770019a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:29:48.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.246+0000 7f76fea00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77001010b0 0x7f770019a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:35096/0 (socket says 172.21.15.31:35096) 2024-03-21T10:29:48.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.246+0000 7f76fea00700 1 -- 172.21.15.31:0/2102036973 learned_addr learned my addr 172.21.15.31:0/2102036973 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:29:48.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.246+0000 7f76ff400700 1 --2- 172.21.15.31:0/2102036973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7700102c50 0x7f77001a1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:29:48.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.246+0000 7f76fe000700 1 --2- 172.21.15.31:0/2102036973 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f77001022b0 0x7f770019ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:29:48.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.246+0000 7f76fea00700 1 -- 172.21.15.31:0/2102036973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7700102c50 msgr2=0x7f77001a1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:29:48.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.246+0000 7f76fea00700 1 --2- 172.21.15.31:0/2102036973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7700102c50 0x7f77001a1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:29:48.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.246+0000 7f76fea00700 1 -- 172.21.15.31:0/2102036973 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f77001022b0 msgr2=0x7f770019ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:29:48.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.246+0000 7f76fea00700 1 --2- 172.21.15.31:0/2102036973 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f77001022b0 0x7f770019ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:29:48.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.246+0000 7f76fea00700 1 -- 172.21.15.31:0/2102036973 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77001a2280 con 0x7f77001010b0 2024-03-21T10:29:48.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.246+0000 7f76fea00700 1 --2- 172.21.15.31:0/2102036973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77001010b0 0x7f770019a620 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f76f000e950 tx=0x7f76f000ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:29:48.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.247+0000 7f76f7400700 1 -- 172.21.15.31:0/2102036973 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f000cbd0 con 0x7f77001010b0 2024-03-21T10:29:48.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.247+0000 7f76f7400700 1 -- 172.21.15.31:0/2102036973 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76f000ee90 con 0x7f77001010b0 2024-03-21T10:29:48.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.247+0000 7f7705800700 1 -- 172.21.15.31:0/2102036973 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77001a2570 con 0x7f77001010b0 2024-03-21T10:29:48.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.247+0000 7f76f7400700 1 -- 172.21.15.31:0/2102036973 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f0010430 con 0x7f77001010b0 2024-03-21T10:29:48.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.247+0000 7f7705800700 1 -- 172.21.15.31:0/2102036973 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77001a2ab0 con 0x7f77001010b0 2024-03-21T10:29:48.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.248+0000 7f7705800700 1 -- 172.21.15.31:0/2102036973 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7700068e50 con 0x7f77001010b0 2024-03-21T10:29:48.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.250+0000 7f76f7400700 1 -- 172.21.15.31:0/2102036973 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 15) v1 ==== 137798+0+0 (secure 0 0 0) 0x7f76f0003eb0 con 0x7f77001010b0 2024-03-21T10:29:48.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.251+0000 7f76f7400700 1 --2- 172.21.15.31:0/2102036973 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f76e40a2fa0 0x7f76e40a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:29:48.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.251+0000 7f76f7400700 1 -- 172.21.15.31:0/2102036973 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3237+0+0 (secure 0 0 0) 0x7f76f00ce2c0 con 0x7f77001010b0 2024-03-21T10:29:48.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.251+0000 7f76fe000700 1 --2- 172.21.15.31:0/2102036973 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f76e40a2fa0 0x7f76e40a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:29:48.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.252+0000 7f76fe000700 1 --2- 172.21.15.31:0/2102036973 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f76e40a2fa0 0x7f76e40a5460 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f76ec000c00 tx=0x7f76ec005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:29:48.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.255+0000 7f76f7400700 1 -- 172.21.15.31:0/2102036973 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f76f009c500 con 0x7f77001010b0 2024-03-21T10:29:48.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.493+0000 7f7705800700 1 -- 172.21.15.31:0/2102036973 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f7700107ff0 con 0x7f76e40a2fa0 2024-03-21T10:29:48.682 INFO:journalctl@ceph.osd.1.smithi040.stdout:Mar 21 10:29:48 smithi040 sudo[44883]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T10:29:48.683 INFO:journalctl@ceph.osd.1.smithi040.stdout:Mar 21 10:29:48 smithi040 sudo[44883]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T10:29:48.683 INFO:journalctl@ceph.osd.1.smithi040.stdout:Mar 21 10:29:48 smithi040 sudo[44883]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T10:29:48.683 INFO:journalctl@ceph.osd.1.smithi040.stdout:Mar 21 10:29:48 smithi040 sudo[44883]: pam_unix(sudo:session): session closed for user root 2024-03-21T10:29:48.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:29:48.826+0000 7f76f7400700 1 -- 172.21.15.31:0/2102036973 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f76f0098470 con 0x7f77001010b0 2024-03-21T10:29:48.986 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:48 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:48.986 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:48 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:29:48.987 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:48 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:29:48.987 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:48 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:29:48.987 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:48 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:49.183 INFO:journalctl@ceph.osd.1.smithi040.stdout:Mar 21 10:29:48 smithi040 sudo[44886]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T10:29:49.183 INFO:journalctl@ceph.osd.1.smithi040.stdout:Mar 21 10:29:48 smithi040 sudo[44886]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T10:29:49.183 INFO:journalctl@ceph.osd.1.smithi040.stdout:Mar 21 10:29:48 smithi040 sudo[44886]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T10:29:49.183 INFO:journalctl@ceph.osd.1.smithi040.stdout:Mar 21 10:29:48 smithi040 sudo[44886]: pam_unix(sudo:session): session closed for user root 2024-03-21T10:29:49.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:48 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:49.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:48 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:29:49.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:48 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:29:49.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:48 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:29:49.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:48 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:49.238 INFO:journalctl@ceph.osd.2.smithi070.stdout:Mar 21 10:29:48 smithi070 sudo[45291]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T10:29:49.238 INFO:journalctl@ceph.osd.2.smithi070.stdout:Mar 21 10:29:48 smithi070 sudo[45291]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T10:29:49.238 INFO:journalctl@ceph.osd.2.smithi070.stdout:Mar 21 10:29:48 smithi070 sudo[45291]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T10:29:49.238 INFO:journalctl@ceph.osd.2.smithi070.stdout:Mar 21 10:29:49 smithi070 sudo[45291]: pam_unix(sudo:session): session closed for user root 2024-03-21T10:29:49.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:48 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:49.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:48 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:29:49.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:48 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:29:49.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:48 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:29:49.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:48 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:49.575 INFO:journalctl@ceph.osd.2.smithi070.stdout:Mar 21 10:29:49 smithi070 sudo[45294]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T10:29:49.575 INFO:journalctl@ceph.osd.2.smithi070.stdout:Mar 21 10:29:49 smithi070 sudo[45294]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T10:29:49.575 INFO:journalctl@ceph.osd.2.smithi070.stdout:Mar 21 10:29:49 smithi070 sudo[45294]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T10:29:49.575 INFO:journalctl@ceph.osd.2.smithi070.stdout:Mar 21 10:29:49 smithi070 sudo[45294]: pam_unix(sudo:session): session closed for user root 2024-03-21T10:29:50.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:49 smithi031 sudo[50757]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T10:29:50.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:49 smithi031 sudo[50757]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T10:29:50.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:49 smithi031 sudo[50757]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T10:29:50.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:49 smithi031 sudo[50757]: pam_unix(sudo:session): session closed for user root 2024-03-21T10:29:50.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:50 smithi031 sudo[50764]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T10:29:50.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:50 smithi031 sudo[50764]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T10:29:50.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:50 smithi031 sudo[50764]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T10:29:50.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:50 smithi031 sudo[50764]: pam_unix(sudo:session): session closed for user root 2024-03-21T10:29:50.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:50 smithi031 ceph-mon[31673]: pgmap v98: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:29:50.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:50 smithi031 ceph-mon[31673]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:29:50.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:50 smithi031 ceph-mon[31673]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-21T10:29:50.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:50 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:50.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:50 smithi031 ceph-mon[31673]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T10:29:50.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:50 smithi040 ceph-mon[36154]: pgmap v98: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:29:50.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:50 smithi040 ceph-mon[36154]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:29:50.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:50 smithi040 ceph-mon[36154]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-21T10:29:50.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:50 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:50.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:50 smithi040 ceph-mon[36154]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T10:29:50.685 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:50 smithi070 ceph-mon[36330]: pgmap v98: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:29:50.685 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:50 smithi070 ceph-mon[36330]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:29:50.685 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:50 smithi070 ceph-mon[36330]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-21T10:29:50.685 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:50 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:29:50.685 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:50 smithi070 ceph-mon[36330]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T10:29:50.685 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:50 smithi070 sudo[45297]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T10:29:50.685 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:50 smithi070 sudo[45297]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T10:29:50.685 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:50 smithi070 sudo[45297]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T10:29:50.685 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:50 smithi070 sudo[45297]: pam_unix(sudo:session): session closed for user root 2024-03-21T10:29:51.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:50 smithi070 sudo[45300]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T10:29:51.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:50 smithi070 sudo[45300]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T10:29:51.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:50 smithi070 sudo[45300]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T10:29:51.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:50 smithi070 sudo[45300]: pam_unix(sudo:session): session closed for user root 2024-03-21T10:29:51.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:51 smithi040 sudo[44889]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T10:29:51.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:51 smithi040 sudo[44889]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T10:29:51.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:51 smithi040 sudo[44889]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T10:29:51.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:51 smithi040 sudo[44889]: pam_unix(sudo:session): session closed for user root 2024-03-21T10:29:52.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:51 smithi040 sudo[44892]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T10:29:52.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:51 smithi040 sudo[44892]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T10:29:52.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:51 smithi040 sudo[44892]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T10:29:52.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:51 smithi040 sudo[44892]: pam_unix(sudo:session): session closed for user root 2024-03-21T10:29:52.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:51 smithi040 ceph-mon[36154]: pgmap v99: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:29:52.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:51 smithi040 ceph-mon[36154]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T10:29:52.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:51 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T10:29:52.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:51 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:29:52.184 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:51 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:29:52.184 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:51 smithi040 ceph-mon[36154]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T10:29:52.184 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:51 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T10:29:52.184 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:51 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:29:52.184 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:51 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:29:52.184 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:51 smithi040 ceph-mon[36154]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T10:29:52.184 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:51 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T10:29:52.184 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:51 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:29:52.184 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:51 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:29:52.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:51 smithi031 ceph-mon[31673]: pgmap v99: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:29:52.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:51 smithi031 ceph-mon[31673]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T10:29:52.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:51 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T10:29:52.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:51 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:29:52.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:51 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:29:52.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:51 smithi031 ceph-mon[31673]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T10:29:52.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:51 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T10:29:52.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:51 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:29:52.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:51 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:29:52.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:51 smithi031 ceph-mon[31673]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T10:29:52.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:51 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T10:29:52.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:51 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:29:52.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:51 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:29:52.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:51 smithi070 ceph-mon[36330]: pgmap v99: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:29:52.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:51 smithi070 ceph-mon[36330]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T10:29:52.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:51 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T10:29:52.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:51 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:29:52.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:51 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:29:52.326 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:51 smithi070 ceph-mon[36330]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T10:29:52.326 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:51 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T10:29:52.326 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:51 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:29:52.326 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:51 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:29:52.326 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:51 smithi070 ceph-mon[36330]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T10:29:52.326 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:51 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T10:29:52.326 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:51 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:29:52.326 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:51 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:29:53.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:52 smithi040 ceph-mon[36154]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T10:29:53.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:52 smithi040 ceph-mon[36154]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T10:29:53.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:52 smithi040 ceph-mon[36154]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:29:53.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:52 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/565236298' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edb44507-ffb0-42a4-827b-08461e1bf952"}]: dispatch 2024-03-21T10:29:53.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:52 smithi040 ceph-mon[36154]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edb44507-ffb0-42a4-827b-08461e1bf952"}]: dispatch 2024-03-21T10:29:53.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:52 smithi040 ceph-mon[36154]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "edb44507-ffb0-42a4-827b-08461e1bf952"}]': finished 2024-03-21T10:29:53.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:52 smithi040 ceph-mon[36154]: osdmap e22: 4 total, 3 up, 4 in 2024-03-21T10:29:53.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:52 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T10:29:53.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:52 smithi031 ceph-mon[31673]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T10:29:53.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:52 smithi031 ceph-mon[31673]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T10:29:53.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:52 smithi031 ceph-mon[31673]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:29:53.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:52 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/565236298' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edb44507-ffb0-42a4-827b-08461e1bf952"}]: dispatch 2024-03-21T10:29:53.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:52 smithi031 ceph-mon[31673]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edb44507-ffb0-42a4-827b-08461e1bf952"}]: dispatch 2024-03-21T10:29:53.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:52 smithi031 ceph-mon[31673]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "edb44507-ffb0-42a4-827b-08461e1bf952"}]': finished 2024-03-21T10:29:53.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:52 smithi031 ceph-mon[31673]: osdmap e22: 4 total, 3 up, 4 in 2024-03-21T10:29:53.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:52 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T10:29:53.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:52 smithi070 ceph-mon[36330]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T10:29:53.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:52 smithi070 ceph-mon[36330]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T10:29:53.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:52 smithi070 ceph-mon[36330]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:29:53.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:52 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/565236298' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edb44507-ffb0-42a4-827b-08461e1bf952"}]: dispatch 2024-03-21T10:29:53.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:52 smithi070 ceph-mon[36330]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edb44507-ffb0-42a4-827b-08461e1bf952"}]: dispatch 2024-03-21T10:29:53.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:52 smithi070 ceph-mon[36330]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "edb44507-ffb0-42a4-827b-08461e1bf952"}]': finished 2024-03-21T10:29:53.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:52 smithi070 ceph-mon[36330]: osdmap e22: 4 total, 3 up, 4 in 2024-03-21T10:29:53.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:52 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T10:29:54.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:53 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1766723418' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:29:54.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:53 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1766723418' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:29:54.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:53 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1766723418' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:29:55.157 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:54 smithi031 ceph-mon[31673]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:29:55.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:54 smithi040 ceph-mon[36154]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:29:55.425 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:54 smithi070 ceph-mon[36330]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:29:57.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:56 smithi031 ceph-mon[31673]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 65 KiB/s wr, 6 op/s 2024-03-21T10:29:57.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:56 smithi070 ceph-mon[36330]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 65 KiB/s wr, 6 op/s 2024-03-21T10:29:57.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:56 smithi040 ceph-mon[36154]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 65 KiB/s wr, 6 op/s 2024-03-21T10:29:59.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:29:58 smithi031 ceph-mon[31673]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-03-21T10:29:59.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:29:58 smithi070 ceph-mon[36330]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-03-21T10:29:59.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:29:58 smithi040 ceph-mon[36154]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-03-21T10:30:01.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:00 smithi031 ceph-mon[31673]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-03-21T10:30:01.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:00 smithi031 ceph-mon[31673]: overall HEALTH_OK 2024-03-21T10:30:01.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:00 smithi070 ceph-mon[36330]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-03-21T10:30:01.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:00 smithi070 ceph-mon[36330]: overall HEALTH_OK 2024-03-21T10:30:01.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:00 smithi040 ceph-mon[36154]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-03-21T10:30:01.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:00 smithi040 ceph-mon[36154]: overall HEALTH_OK 2024-03-21T10:30:02.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:01 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T10:30:02.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:01 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:02.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:01 smithi031 ceph-mon[31673]: Deploying daemon osd.3 on smithi031 2024-03-21T10:30:02.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:01 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T10:30:02.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:01 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:02.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:01 smithi070 ceph-mon[36330]: Deploying daemon osd.3 on smithi031 2024-03-21T10:30:02.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:01 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T10:30:02.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:01 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:02.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:01 smithi040 ceph-mon[36154]: Deploying daemon osd.3 on smithi031 2024-03-21T10:30:03.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:02 smithi031 ceph-mon[31673]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-21T10:30:03.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:02 smithi070 ceph-mon[36330]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-21T10:30:03.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:02 smithi040 ceph-mon[36154]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-21T10:30:05.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:04 smithi031 ceph-mon[31673]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 898 B/s rd, 23 KiB/s wr, 2 op/s 2024-03-21T10:30:05.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:04 smithi070 ceph-mon[36330]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 898 B/s rd, 23 KiB/s wr, 2 op/s 2024-03-21T10:30:05.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:04 smithi040 ceph-mon[36154]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 898 B/s rd, 23 KiB/s wr, 2 op/s 2024-03-21T10:30:07.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:06 smithi031 ceph-mon[31673]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-21T10:30:07.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:06 smithi070 ceph-mon[36330]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-21T10:30:07.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:06 smithi040 ceph-mon[36154]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-21T10:30:09.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:08 smithi031 ceph-mon[31673]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:30:09.315 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:08 smithi070 ceph-mon[36330]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:30:09.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:08 smithi040 ceph-mon[36154]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:30:11.598 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:11 smithi031 ceph-mon[31673]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:30:11.599 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:11 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:11.599 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:11 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:11.599 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:11 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:30:11.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:11 smithi040 ceph-mon[36154]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:30:11.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:11 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:11.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:11 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:11.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:11 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:30:11.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:11 smithi070 ceph-mon[36330]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:30:11.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:11 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:11.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:11 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:11.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:11 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:30:13.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:13 smithi031 ceph-mon[31673]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:30:13.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:13 smithi040 ceph-mon[36154]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:30:13.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:13 smithi070 ceph-mon[36330]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:30:14.534 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:14 smithi031 ceph-mon[31673]: from='osd.3 [v2:172.21.15.31:6810/2823036296,v1:172.21.15.31:6811/2823036296]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T10:30:14.535 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:14 smithi031 ceph-mon[31673]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T10:30:14.535 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:14 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:14.535 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:14 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:14.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:14 smithi040 ceph-mon[36154]: from='osd.3 [v2:172.21.15.31:6810/2823036296,v1:172.21.15.31:6811/2823036296]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T10:30:14.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:14 smithi040 ceph-mon[36154]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T10:30:14.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:14 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:14.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:14 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:14.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:14 smithi070 ceph-mon[36330]: from='osd.3 [v2:172.21.15.31:6810/2823036296,v1:172.21.15.31:6811/2823036296]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T10:30:14.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:14 smithi070 ceph-mon[36330]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T10:30:14.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:14 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:14.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:14 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:15.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:15 smithi031 ceph-mon[31673]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:30:15.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:15 smithi031 ceph-mon[31673]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-21T10:30:15.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:15 smithi031 ceph-mon[31673]: osdmap e23: 4 total, 3 up, 4 in 2024-03-21T10:30:15.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:15 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T10:30:15.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:15 smithi031 ceph-mon[31673]: from='osd.3 [v2:172.21.15.31:6810/2823036296,v1:172.21.15.31:6811/2823036296]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-21T10:30:15.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:15 smithi031 ceph-mon[31673]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-21T10:30:15.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:15 smithi040 ceph-mon[36154]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:30:15.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:15 smithi040 ceph-mon[36154]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-21T10:30:15.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:15 smithi040 ceph-mon[36154]: osdmap e23: 4 total, 3 up, 4 in 2024-03-21T10:30:15.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:15 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T10:30:15.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:15 smithi040 ceph-mon[36154]: from='osd.3 [v2:172.21.15.31:6810/2823036296,v1:172.21.15.31:6811/2823036296]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-21T10:30:15.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:15 smithi040 ceph-mon[36154]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-21T10:30:15.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:30:15.793+0000 7f76f7400700 1 -- 172.21.15.31:0/2102036973 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7700107ff0 con 0x7f76e40a2fa0 2024-03-21T10:30:15.793 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 3 on host 'smithi031' 2024-03-21T10:30:15.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:30:15.797+0000 7f7705800700 1 -- 172.21.15.31:0/2102036973 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f76e40a2fa0 msgr2=0x7f76e40a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:30:15.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:30:15.797+0000 7f7705800700 1 --2- 172.21.15.31:0/2102036973 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f76e40a2fa0 0x7f76e40a5460 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f76ec000c00 tx=0x7f76ec005fb0 comp rx=0 tx=0).stop 2024-03-21T10:30:15.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:30:15.797+0000 7f7705800700 1 -- 172.21.15.31:0/2102036973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77001010b0 msgr2=0x7f770019a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:30:15.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:30:15.797+0000 7f7705800700 1 --2- 172.21.15.31:0/2102036973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77001010b0 0x7f770019a620 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f76f000e950 tx=0x7f76f000ed10 comp rx=0 tx=0).stop 2024-03-21T10:30:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:30:15.797+0000 7f7705800700 1 -- 172.21.15.31:0/2102036973 shutdown_connections 2024-03-21T10:30:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:30:15.797+0000 7f7705800700 1 --2- 172.21.15.31:0/2102036973 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f76e40a2fa0 0x7f76e40a5460 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:30:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:30:15.797+0000 7f7705800700 1 --2- 172.21.15.31:0/2102036973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7700102c50 0x7f77001a1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:30:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:30:15.797+0000 7f7705800700 1 --2- 172.21.15.31:0/2102036973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77001010b0 0x7f770019a620 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:30:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:30:15.797+0000 7f7705800700 1 --2- 172.21.15.31:0/2102036973 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f77001022b0 0x7f770019ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:30:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:30:15.797+0000 7f7705800700 1 -- 172.21.15.31:0/2102036973 >> 172.21.15.31:0/2102036973 conn(0x7f77000fc650 msgr2=0x7f77001078e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:30:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:30:15.798+0000 7f7705800700 1 -- 172.21.15.31:0/2102036973 shutdown_connections 2024-03-21T10:30:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:30:15.798+0000 7f7705800700 1 -- 172.21.15.31:0/2102036973 wait complete. 2024-03-21T10:30:15.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:15 smithi070 ceph-mon[36330]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:30:15.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:15 smithi070 ceph-mon[36330]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-21T10:30:15.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:15 smithi070 ceph-mon[36330]: osdmap e23: 4 total, 3 up, 4 in 2024-03-21T10:30:15.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:15 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T10:30:15.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:15 smithi070 ceph-mon[36330]: from='osd.3 [v2:172.21.15.31:6810/2823036296,v1:172.21.15.31:6811/2823036296]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-21T10:30:15.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:15 smithi070 ceph-mon[36330]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-21T10:30:16.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:16 smithi031 ceph-mon[31673]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-21T10:30:16.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:16 smithi031 ceph-mon[31673]: osdmap e24: 4 total, 3 up, 4 in 2024-03-21T10:30:16.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:16 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T10:30:16.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:16 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T10:30:16.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:16 smithi031 ceph-mon[31673]: osd.3 [v2:172.21.15.31:6810/2823036296,v1:172.21.15.31:6811/2823036296] boot 2024-03-21T10:30:16.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:16 smithi031 ceph-mon[31673]: osdmap e25: 4 total, 4 up, 4 in 2024-03-21T10:30:16.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:16 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T10:30:16.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:16 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:16.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:16 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:16.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:16 smithi040 ceph-mon[36154]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-21T10:30:16.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:16 smithi040 ceph-mon[36154]: osdmap e24: 4 total, 3 up, 4 in 2024-03-21T10:30:16.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:16 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T10:30:16.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:16 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T10:30:16.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:16 smithi040 ceph-mon[36154]: osd.3 [v2:172.21.15.31:6810/2823036296,v1:172.21.15.31:6811/2823036296] boot 2024-03-21T10:30:16.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:16 smithi040 ceph-mon[36154]: osdmap e25: 4 total, 4 up, 4 in 2024-03-21T10:30:16.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:16 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T10:30:16.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:16 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:16.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:16 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:16.728 DEBUG:teuthology.orchestra.run.smithi031:osd.3> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.3.service 2024-03-21T10:30:16.733 INFO:tasks.cephadm:Deploying osd.4 on smithi040 with /dev/vg_nvme/lv_3... 2024-03-21T10:30:16.733 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-21T10:30:16.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:16 smithi070 ceph-mon[36330]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-21T10:30:16.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:16 smithi070 ceph-mon[36330]: osdmap e24: 4 total, 3 up, 4 in 2024-03-21T10:30:16.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:16 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T10:30:16.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:16 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T10:30:16.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:16 smithi070 ceph-mon[36330]: osd.3 [v2:172.21.15.31:6810/2823036296,v1:172.21.15.31:6811/2823036296] boot 2024-03-21T10:30:16.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:16 smithi070 ceph-mon[36330]: osdmap e25: 4 total, 4 up, 4 in 2024-03-21T10:30:16.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:16 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T10:30:16.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:16 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:16.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:16 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:17.008 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T10:30:17.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:17 smithi031 ceph-mon[31673]: purged_snaps scrub starts 2024-03-21T10:30:17.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:17 smithi031 ceph-mon[31673]: purged_snaps scrub ok 2024-03-21T10:30:17.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:17 smithi031 ceph-mon[31673]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:30:17.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:17 smithi031 ceph-mon[31673]: osdmap e26: 4 total, 4 up, 4 in 2024-03-21T10:30:17.822 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:17 smithi040 ceph-mon[36154]: purged_snaps scrub starts 2024-03-21T10:30:17.822 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:17 smithi040 ceph-mon[36154]: purged_snaps scrub ok 2024-03-21T10:30:17.822 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:17 smithi040 ceph-mon[36154]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:30:17.822 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:17 smithi040 ceph-mon[36154]: osdmap e26: 4 total, 4 up, 4 in 2024-03-21T10:30:17.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:17 smithi070 ceph-mon[36330]: purged_snaps scrub starts 2024-03-21T10:30:17.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:17 smithi070 ceph-mon[36330]: purged_snaps scrub ok 2024-03-21T10:30:17.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:17 smithi070 ceph-mon[36330]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-21T10:30:17.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:17 smithi070 ceph-mon[36330]: osdmap e26: 4 total, 4 up, 4 in 2024-03-21T10:30:18.503 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:18 smithi040 ceph-mon[36154]: osdmap e27: 4 total, 4 up, 4 in 2024-03-21T10:30:18.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:18 smithi031 ceph-mon[31673]: osdmap e27: 4 total, 4 up, 4 in 2024-03-21T10:30:18.817 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:18 smithi070 ceph-mon[36330]: osdmap e27: 4 total, 4 up, 4 in 2024-03-21T10:30:19.084 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-21T10:30:19.084 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-21T10:30:19.084 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-03-21T10:30:19.084 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-03-21T10:30:19.084 INFO:teuthology.orchestra.run.smithi040.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124999 s, 839 MB/s 2024-03-21T10:30:19.084 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-03-21T10:30:19.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:19 smithi040 ceph-mon[36154]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:19.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:19 smithi040 ceph-mon[36154]: osdmap e28: 4 total, 4 up, 4 in 2024-03-21T10:30:19.768 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi040:vg_nvme/lv_3 2024-03-21T10:30:19.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:19 smithi031 ceph-mon[31673]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:19.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:19 smithi031 ceph-mon[31673]: osdmap e28: 4 total, 4 up, 4 in 2024-03-21T10:30:19.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:19 smithi070 ceph-mon[36330]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:19.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:19 smithi070 ceph-mon[36330]: osdmap e28: 4 total, 4 up, 4 in 2024-03-21T10:30:20.018 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T10:30:20.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:20 smithi040 ceph-mon[36154]: osdmap e29: 4 total, 4 up, 4 in 2024-03-21T10:30:20.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:20 smithi031 ceph-mon[31673]: osdmap e29: 4 total, 4 up, 4 in 2024-03-21T10:30:20.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:20 smithi070 ceph-mon[36330]: osdmap e29: 4 total, 4 up, 4 in 2024-03-21T10:30:21.633 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:21 smithi040 ceph-mon[36154]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:21.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.645+0000 7f0f0c872700 1 -- 172.21.15.40:0/592206905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f08101dd0 msgr2=0x7f0f08102250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:30:21.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.645+0000 7f0f0c872700 1 --2- 172.21.15.40:0/592206905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f08101dd0 0x7f0f08102250 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0f0000b0a0 tx=0x7f0f0000b3b0 comp rx=0 tx=0).stop 2024-03-21T10:30:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.645+0000 7f0f0c872700 1 -- 172.21.15.40:0/592206905 shutdown_connections 2024-03-21T10:30:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.645+0000 7f0f0c872700 1 --2- 172.21.15.40:0/592206905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f08101dd0 0x7f0f08102250 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:30:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.645+0000 7f0f0c872700 1 --2- 172.21.15.40:0/592206905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f08100ab0 0x7f0f08100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:30:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.645+0000 7f0f0c872700 1 --2- 172.21.15.40:0/592206905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f08101410 0x7f0f08101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:30:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.645+0000 7f0f0c872700 1 -- 172.21.15.40:0/592206905 >> 172.21.15.40:0/592206905 conn(0x7f0f080fc6f0 msgr2=0x7f0f080feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:30:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.645+0000 7f0f0c872700 1 -- 172.21.15.40:0/592206905 shutdown_connections 2024-03-21T10:30:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.645+0000 7f0f0c872700 1 -- 172.21.15.40:0/592206905 wait complete. 2024-03-21T10:30:21.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.646+0000 7f0f0c872700 1 Processor -- start 2024-03-21T10:30:21.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.646+0000 7f0f0c872700 1 -- start start 2024-03-21T10:30:21.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.646+0000 7f0f0c872700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f08100ab0 0x7f0f0819a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:30:21.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.646+0000 7f0f0c872700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f08101410 0x7f0f0819aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:30:21.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.646+0000 7f0f0c872700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f08101dd0 0x7f0f081a1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:30:21.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.646+0000 7f0f0c872700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0810fff0 con 0x7f0f08101dd0 2024-03-21T10:30:21.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.646+0000 7f0f0c872700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0810fe70 con 0x7f0f08101410 2024-03-21T10:30:21.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.646+0000 7f0f0c872700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f08110170 con 0x7f0f08100ab0 2024-03-21T10:30:21.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.646+0000 7f0f05d9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f08101410 0x7f0f0819aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:30:21.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.646+0000 7f0f05d9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f08101410 0x7f0f0819aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43388/0 (socket says 172.21.15.40:43388) 2024-03-21T10:30:21.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.646+0000 7f0f05d9b700 1 -- 172.21.15.40:0/332772425 learned_addr learned my addr 172.21.15.40:0/332772425 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-21T10:30:21.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.646+0000 7f0f0659c700 1 --2- 172.21.15.40:0/332772425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f08100ab0 0x7f0f0819a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:30:21.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.647+0000 7f0f05d9b700 1 -- 172.21.15.40:0/332772425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f08100ab0 msgr2=0x7f0f0819a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:30:21.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.647+0000 7f0f05d9b700 1 --2- 172.21.15.40:0/332772425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f08100ab0 0x7f0f0819a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:30:21.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.647+0000 7f0f05d9b700 1 -- 172.21.15.40:0/332772425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f08101dd0 msgr2=0x7f0f081a1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:30:21.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.647+0000 7f0f05d9b700 1 --2- 172.21.15.40:0/332772425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f08101dd0 0x7f0f081a1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:30:21.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.647+0000 7f0f05d9b700 1 -- 172.21.15.40:0/332772425 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f081a21d0 con 0x7f0f08101410 2024-03-21T10:30:21.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.647+0000 7f0f05d9b700 1 --2- 172.21.15.40:0/332772425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f08101410 0x7f0f0819aab0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800c920 tx=0x7f0ef800cc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:30:21.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.647+0000 7f0ef77fe700 1 -- 172.21.15.40:0/332772425 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8007b00 con 0x7f0f08101410 2024-03-21T10:30:21.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.647+0000 7f0ef77fe700 1 -- 172.21.15.40:0/332772425 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ef8004d10 con 0x7f0f08101410 2024-03-21T10:30:21.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.647+0000 7f0f0c872700 1 -- 172.21.15.40:0/332772425 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f081a24c0 con 0x7f0f08101410 2024-03-21T10:30:21.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.647+0000 7f0ef77fe700 1 -- 172.21.15.40:0/332772425 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8005680 con 0x7f0f08101410 2024-03-21T10:30:21.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.647+0000 7f0f0c872700 1 -- 172.21.15.40:0/332772425 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f081a2a00 con 0x7f0f08101410 2024-03-21T10:30:21.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.648+0000 7f0f0c872700 1 -- 172.21.15.40:0/332772425 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f08068e50 con 0x7f0f08101410 2024-03-21T10:30:21.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.649+0000 7f0ef77fe700 1 -- 172.21.15.40:0/332772425 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f0ef8004750 con 0x7f0f08101410 2024-03-21T10:30:21.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.650+0000 7f0ef77fe700 1 --2- 172.21.15.40:0/332772425 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0ef00a2fb0 0x7f0ef00a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:30:21.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.650+0000 7f0ef77fe700 1 -- 172.21.15.40:0/332772425 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3669+0+0 (secure 0 0 0) 0x7f0ef80cde00 con 0x7f0f08101410 2024-03-21T10:30:21.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.650+0000 7f0f0659c700 1 --2- 172.21.15.40:0/332772425 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0ef00a2fb0 0x7f0ef00a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:30:21.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.651+0000 7f0f0659c700 1 --2- 172.21.15.40:0/332772425 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0ef00a2fb0 0x7f0ef00a5470 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0efc009a10 tx=0x7f0efc009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:30:21.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.655+0000 7f0ef77fe700 1 -- 172.21.15.40:0/332772425 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0ef8014030 con 0x7f0f08101410 2024-03-21T10:30:21.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:21 smithi031 ceph-mon[31673]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:21.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:21 smithi070 ceph-mon[36330]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:21.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:21.869+0000 7f0f0c872700 1 -- 172.21.15.40:0/332772425 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f080625d0 con 0x7f0ef00a2fb0 2024-03-21T10:30:22.750 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:22 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:30:22.750 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:22 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:30:22.750 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:22 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:22.751 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:22 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:22.751 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:22 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:22.751 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:22 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:30:22.751 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:22 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:30:22.751 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:22 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:22.751 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:22 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:22.751 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:22 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:30:22.751 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:22 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:22.751 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:22 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:30:22.751 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:22 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:22.751 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:22 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:30:22.751 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:22 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:22.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:22 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:30:22.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:22 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:30:22.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:22 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:22.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:22 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:22.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:22 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:22.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:22 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:30:22.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:22 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:30:22.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:22 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:22.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:22 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:22.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:22 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:30:22.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:22 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:22.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:22 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:30:22.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:22 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:22.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:22 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:30:22.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:22 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:22.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:22 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:30:22.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:22 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:30:22.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:22 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:22.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:22 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:22.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:22 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:22.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:22 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:30:22.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:22 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:30:22.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:22 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:22.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:22 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:22.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:22 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:30:22.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:22 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:22.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:22 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:30:22.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:22 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:22.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:22 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:30:22.826 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:22 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:23.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:23 smithi031 ceph-mon[31673]: from='client.24233 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:30:23.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:23 smithi031 ceph-mon[31673]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-03-21T10:30:23.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:23 smithi031 ceph-mon[31673]: Detected new or changed devices on smithi031 2024-03-21T10:30:23.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:23 smithi031 ceph-mon[31673]: Adjusting osd_memory_target on smithi031 to 8547M 2024-03-21T10:30:23.815 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:23 smithi040 ceph-mon[36154]: from='client.24233 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:30:23.815 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:23 smithi040 ceph-mon[36154]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-03-21T10:30:23.815 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:23 smithi040 ceph-mon[36154]: Detected new or changed devices on smithi031 2024-03-21T10:30:23.815 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:23 smithi040 ceph-mon[36154]: Adjusting osd_memory_target on smithi031 to 8547M 2024-03-21T10:30:23.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:23 smithi070 ceph-mon[36330]: from='client.24233 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:30:23.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:23 smithi070 ceph-mon[36330]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-03-21T10:30:23.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:23 smithi070 ceph-mon[36330]: Detected new or changed devices on smithi031 2024-03-21T10:30:23.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:23 smithi070 ceph-mon[36330]: Adjusting osd_memory_target on smithi031 to 8547M 2024-03-21T10:30:25.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:25 smithi031 ceph-mon[31673]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-21T10:30:25.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:25 smithi031 ceph-mon[31673]: from='client.? 172.21.15.40:0/2623717987' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8cb89153-8c3b-431f-b57c-8863fada76dc"}]: dispatch 2024-03-21T10:30:25.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:25 smithi031 ceph-mon[31673]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8cb89153-8c3b-431f-b57c-8863fada76dc"}]: dispatch 2024-03-21T10:30:25.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:25 smithi031 ceph-mon[31673]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8cb89153-8c3b-431f-b57c-8863fada76dc"}]': finished 2024-03-21T10:30:25.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:25 smithi031 ceph-mon[31673]: osdmap e30: 5 total, 4 up, 5 in 2024-03-21T10:30:25.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:25 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T10:30:25.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:25 smithi031 ceph-mon[31673]: from='client.? 172.21.15.40:0/1406986348' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:30:25.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:25 smithi070 ceph-mon[36330]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-21T10:30:25.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:25 smithi070 ceph-mon[36330]: from='client.? 172.21.15.40:0/2623717987' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8cb89153-8c3b-431f-b57c-8863fada76dc"}]: dispatch 2024-03-21T10:30:25.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:25 smithi070 ceph-mon[36330]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8cb89153-8c3b-431f-b57c-8863fada76dc"}]: dispatch 2024-03-21T10:30:25.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:25 smithi070 ceph-mon[36330]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8cb89153-8c3b-431f-b57c-8863fada76dc"}]': finished 2024-03-21T10:30:25.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:25 smithi070 ceph-mon[36330]: osdmap e30: 5 total, 4 up, 5 in 2024-03-21T10:30:25.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:25 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T10:30:25.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:25 smithi070 ceph-mon[36330]: from='client.? 172.21.15.40:0/1406986348' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:30:25.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:25 smithi040 ceph-mon[36154]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-21T10:30:25.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:25 smithi040 ceph-mon[36154]: from='client.? 172.21.15.40:0/2623717987' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8cb89153-8c3b-431f-b57c-8863fada76dc"}]: dispatch 2024-03-21T10:30:25.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:25 smithi040 ceph-mon[36154]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8cb89153-8c3b-431f-b57c-8863fada76dc"}]: dispatch 2024-03-21T10:30:25.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:25 smithi040 ceph-mon[36154]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8cb89153-8c3b-431f-b57c-8863fada76dc"}]': finished 2024-03-21T10:30:25.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:25 smithi040 ceph-mon[36154]: osdmap e30: 5 total, 4 up, 5 in 2024-03-21T10:30:25.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:25 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T10:30:25.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:25 smithi040 ceph-mon[36154]: from='client.? 172.21.15.40:0/1406986348' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:30:27.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:27 smithi031 ceph-mon[31673]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-21T10:30:27.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:27 smithi070 ceph-mon[36330]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-21T10:30:27.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:27 smithi040 ceph-mon[36154]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-21T10:30:29.800 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:29 smithi040 ceph-mon[36154]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-21T10:30:29.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:29 smithi031 ceph-mon[31673]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-21T10:30:29.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:29 smithi070 ceph-mon[36330]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-21T10:30:31.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:31 smithi040 ceph-mon[36154]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T10:30:31.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:31 smithi031 ceph-mon[31673]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T10:30:31.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:31 smithi070 ceph-mon[36330]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T10:30:32.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:32 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T10:30:32.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:32 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:32.822 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:32 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T10:30:32.822 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:32 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:32.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:32 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T10:30:32.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:32 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:33.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:33 smithi040 ceph-mon[36154]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:33.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:33 smithi040 ceph-mon[36154]: Deploying daemon osd.4 on smithi040 2024-03-21T10:30:33.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:33 smithi031 ceph-mon[31673]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:33.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:33 smithi031 ceph-mon[31673]: Deploying daemon osd.4 on smithi040 2024-03-21T10:30:33.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:33 smithi070 ceph-mon[36330]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:33.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:33 smithi070 ceph-mon[36330]: Deploying daemon osd.4 on smithi040 2024-03-21T10:30:35.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:35 smithi040 ceph-mon[36154]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:35.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:35 smithi031 ceph-mon[31673]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:35.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:35 smithi070 ceph-mon[36330]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:37.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:37 smithi031 ceph-mon[31673]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:37.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:37 smithi070 ceph-mon[36330]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:37.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:37 smithi040 ceph-mon[36154]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:39.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:38 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:39.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:38 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:30:39.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:38 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:39.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:38 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:39.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:38 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:30:39.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:38 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:39.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:38 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:39.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:38 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:30:39.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:38 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:40.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:39 smithi070 ceph-mon[36330]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:40.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:39 smithi040 ceph-mon[36154]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:39 smithi031 ceph-mon[31673]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:40.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:40 smithi040 ceph-mon[36154]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:41.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:41.053+0000 7f0ef77fe700 1 -- 172.21.15.40:0/332772425 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0f080625d0 con 0x7f0ef00a2fb0 2024-03-21T10:30:41.053 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 4 on host 'smithi040' 2024-03-21T10:30:41.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:41.056+0000 7f0f0c872700 1 -- 172.21.15.40:0/332772425 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0ef00a2fb0 msgr2=0x7f0ef00a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:30:41.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:41.056+0000 7f0f0c872700 1 --2- 172.21.15.40:0/332772425 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0ef00a2fb0 0x7f0ef00a5470 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0efc009a10 tx=0x7f0efc009190 comp rx=0 tx=0).stop 2024-03-21T10:30:41.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:41.056+0000 7f0f0c872700 1 -- 172.21.15.40:0/332772425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f08101410 msgr2=0x7f0f0819aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:30:41.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:41.056+0000 7f0f0c872700 1 --2- 172.21.15.40:0/332772425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f08101410 0x7f0f0819aab0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800c920 tx=0x7f0ef800cc30 comp rx=0 tx=0).stop 2024-03-21T10:30:41.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:41.056+0000 7f0f0c872700 1 -- 172.21.15.40:0/332772425 shutdown_connections 2024-03-21T10:30:41.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:41.056+0000 7f0f0c872700 1 --2- 172.21.15.40:0/332772425 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0ef00a2fb0 0x7f0ef00a5470 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:30:41.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:41.056+0000 7f0f0c872700 1 --2- 172.21.15.40:0/332772425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f08101dd0 0x7f0f081a1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:30:41.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:41.056+0000 7f0f0c872700 1 --2- 172.21.15.40:0/332772425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f08100ab0 0x7f0f0819a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:30:41.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:41.056+0000 7f0f0c872700 1 --2- 172.21.15.40:0/332772425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f08101410 0x7f0f0819aab0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:30:41.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:41.056+0000 7f0f0c872700 1 -- 172.21.15.40:0/332772425 >> 172.21.15.40:0/332772425 conn(0x7f0f080fc6f0 msgr2=0x7f0f080fe1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:30:41.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:41.056+0000 7f0f0c872700 1 -- 172.21.15.40:0/332772425 shutdown_connections 2024-03-21T10:30:41.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:30:41.056+0000 7f0f0c872700 1 -- 172.21.15.40:0/332772425 wait complete. 2024-03-21T10:30:41.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:40 smithi031 ceph-mon[31673]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:41.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:40 smithi070 ceph-mon[36330]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:42.089 DEBUG:teuthology.orchestra.run.smithi040:osd.4> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.4.service 2024-03-21T10:30:42.091 INFO:tasks.cephadm:Deploying osd.5 on smithi070 with /dev/vg_nvme/lv_3... 2024-03-21T10:30:42.092 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-21T10:30:42.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:42 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:42.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:42 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:42.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:42 smithi031 ceph-mon[31673]: from='osd.4 [v2:172.21.15.40:6808/4215186309,v1:172.21.15.40:6809/4215186309]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T10:30:42.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:42 smithi031 ceph-mon[31673]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T10:30:42.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:42 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:42.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:42 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:42.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:42 smithi070 ceph-mon[36330]: from='osd.4 [v2:172.21.15.40:6808/4215186309,v1:172.21.15.40:6809/4215186309]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T10:30:42.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:42 smithi070 ceph-mon[36330]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T10:30:42.342 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.c/config 2024-03-21T10:30:42.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:42 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:42.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:42 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:42.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:42 smithi040 ceph-mon[36154]: from='osd.4 [v2:172.21.15.40:6808/4215186309,v1:172.21.15.40:6809/4215186309]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T10:30:42.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:42 smithi040 ceph-mon[36154]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T10:30:43.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:43 smithi040 ceph-mon[36154]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:43.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:43 smithi040 ceph-mon[36154]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-21T10:30:43.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:43 smithi040 ceph-mon[36154]: from='osd.4 [v2:172.21.15.40:6808/4215186309,v1:172.21.15.40:6809/4215186309]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-21T10:30:43.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:43 smithi040 ceph-mon[36154]: osdmap e31: 5 total, 4 up, 5 in 2024-03-21T10:30:43.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:43 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T10:30:43.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:43 smithi040 ceph-mon[36154]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-21T10:30:43.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:43 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:43.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:43 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:43.184 INFO:journalctl@ceph.osd.4.smithi040.stdout:Mar 21 10:30:43 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-4[47697]: 2024-03-21T10:30:43.045+0000 7fd049012700 -1 osd.4 0 waiting for initial osdmap 2024-03-21T10:30:43.184 INFO:journalctl@ceph.osd.4.smithi040.stdout:Mar 21 10:30:43 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-4[47697]: 2024-03-21T10:30:43.050+0000 7fd044e05700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T10:30:43.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:43 smithi031 ceph-mon[31673]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:43.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:43 smithi031 ceph-mon[31673]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-21T10:30:43.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:43 smithi031 ceph-mon[31673]: from='osd.4 [v2:172.21.15.40:6808/4215186309,v1:172.21.15.40:6809/4215186309]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-21T10:30:43.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:43 smithi031 ceph-mon[31673]: osdmap e31: 5 total, 4 up, 5 in 2024-03-21T10:30:43.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T10:30:43.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:43 smithi031 ceph-mon[31673]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-21T10:30:43.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:43.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:43.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:43 smithi070 ceph-mon[36330]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:43.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:43 smithi070 ceph-mon[36330]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-21T10:30:43.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:43 smithi070 ceph-mon[36330]: from='osd.4 [v2:172.21.15.40:6808/4215186309,v1:172.21.15.40:6809/4215186309]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-21T10:30:43.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:43 smithi070 ceph-mon[36330]: osdmap e31: 5 total, 4 up, 5 in 2024-03-21T10:30:43.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:43 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T10:30:43.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:43 smithi070 ceph-mon[36330]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-21T10:30:43.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:43 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:43.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:43 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:44.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:44 smithi040 ceph-mon[36154]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-03-21T10:30:44.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:44 smithi040 ceph-mon[36154]: osdmap e32: 5 total, 4 up, 5 in 2024-03-21T10:30:44.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:44 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T10:30:44.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:44 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T10:30:44.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:44 smithi070 ceph-mon[36330]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-03-21T10:30:44.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:44 smithi070 ceph-mon[36330]: osdmap e32: 5 total, 4 up, 5 in 2024-03-21T10:30:44.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:44 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T10:30:44.326 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:44 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T10:30:44.501 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-21T10:30:44.501 INFO:teuthology.orchestra.run.smithi070.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-21T10:30:44.501 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10+0 records in 2024-03-21T10:30:44.502 INFO:teuthology.orchestra.run.smithi070.stderr:10+0 records out 2024-03-21T10:30:44.502 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00836119 s, 1.3 GB/s 2024-03-21T10:30:44.502 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping successful for: 2024-03-21T10:30:44.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:44 smithi031 ceph-mon[31673]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-03-21T10:30:44.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:44 smithi031 ceph-mon[31673]: osdmap e32: 5 total, 4 up, 5 in 2024-03-21T10:30:44.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:44 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T10:30:44.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:44 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T10:30:45.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:45 smithi070 ceph-mon[36330]: purged_snaps scrub starts 2024-03-21T10:30:45.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:45 smithi070 ceph-mon[36330]: purged_snaps scrub ok 2024-03-21T10:30:45.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:45 smithi070 ceph-mon[36330]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:45.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:45 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T10:30:45.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:45 smithi070 ceph-mon[36330]: osd.4 [v2:172.21.15.40:6808/4215186309,v1:172.21.15.40:6809/4215186309] boot 2024-03-21T10:30:45.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:45 smithi070 ceph-mon[36330]: osdmap e33: 5 total, 5 up, 5 in 2024-03-21T10:30:45.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:45 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T10:30:45.167 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi070:vg_nvme/lv_3 2024-03-21T10:30:45.386 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:45 smithi040 ceph-mon[36154]: purged_snaps scrub starts 2024-03-21T10:30:45.386 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:45 smithi040 ceph-mon[36154]: purged_snaps scrub ok 2024-03-21T10:30:45.386 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:45 smithi040 ceph-mon[36154]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:45.386 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:45 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T10:30:45.386 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:45 smithi040 ceph-mon[36154]: osd.4 [v2:172.21.15.40:6808/4215186309,v1:172.21.15.40:6809/4215186309] boot 2024-03-21T10:30:45.386 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:45 smithi040 ceph-mon[36154]: osdmap e33: 5 total, 5 up, 5 in 2024-03-21T10:30:45.386 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:45 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T10:30:45.402 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.c/config 2024-03-21T10:30:45.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:45 smithi031 ceph-mon[31673]: purged_snaps scrub starts 2024-03-21T10:30:45.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:45 smithi031 ceph-mon[31673]: purged_snaps scrub ok 2024-03-21T10:30:45.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:45 smithi031 ceph-mon[31673]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-21T10:30:45.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:45 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T10:30:45.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:45 smithi031 ceph-mon[31673]: osd.4 [v2:172.21.15.40:6808/4215186309,v1:172.21.15.40:6809/4215186309] boot 2024-03-21T10:30:45.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:45 smithi031 ceph-mon[31673]: osdmap e33: 5 total, 5 up, 5 in 2024-03-21T10:30:45.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:45 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T10:30:46.203 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:46 smithi070 ceph-mon[36330]: osdmap e34: 5 total, 5 up, 5 in 2024-03-21T10:30:46.203 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:46 smithi070 ceph-mon[36330]: Detected new or changed devices on smithi040 2024-03-21T10:30:46.204 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:46 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:46.204 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:46 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:46.204 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:46 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:30:46.204 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:46 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:30:46.204 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:46 smithi070 ceph-mon[36330]: Adjusting osd_memory_target on smithi040 to 8542M 2024-03-21T10:30:46.204 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:46 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:46.204 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:46 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:46.204 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:46 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:30:46.204 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:46 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:46.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:46 smithi040 ceph-mon[36154]: osdmap e34: 5 total, 5 up, 5 in 2024-03-21T10:30:46.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:46 smithi040 ceph-mon[36154]: Detected new or changed devices on smithi040 2024-03-21T10:30:46.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:46 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:46.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:46 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:46.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:46 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:30:46.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:46 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:30:46.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:46 smithi040 ceph-mon[36154]: Adjusting osd_memory_target on smithi040 to 8542M 2024-03-21T10:30:46.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:46 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:46.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:46 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:46.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:46 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:30:46.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:46 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:46.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:46 smithi031 ceph-mon[31673]: osdmap e34: 5 total, 5 up, 5 in 2024-03-21T10:30:46.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:46 smithi031 ceph-mon[31673]: Detected new or changed devices on smithi040 2024-03-21T10:30:46.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:46 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:46.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:46 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:46.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:46 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:30:46.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:46 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:30:46.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:46 smithi031 ceph-mon[31673]: Adjusting osd_memory_target on smithi040 to 8542M 2024-03-21T10:30:46.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:46 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:46.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:46 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:46.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:46 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:30:46.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:46 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:47.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.060+0000 7fc18b16f700 1 -- 172.21.15.70:0/3427050213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1841034a0 msgr2=0x7fc184103900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:30:47.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.060+0000 7fc18b16f700 1 --2- 172.21.15.70:0/3427050213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1841034a0 0x7fc184103900 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc170009990 tx=0x7fc170009ca0 comp rx=0 tx=0).stop 2024-03-21T10:30:47.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.060+0000 7fc18b16f700 1 -- 172.21.15.70:0/3427050213 shutdown_connections 2024-03-21T10:30:47.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.060+0000 7fc18b16f700 1 --2- 172.21.15.70:0/3427050213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc184103e40 0x7fc18410a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:30:47.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.060+0000 7fc18b16f700 1 --2- 172.21.15.70:0/3427050213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1841004c0 0x7fc1841008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:30:47.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.060+0000 7fc18b16f700 1 --2- 172.21.15.70:0/3427050213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1841034a0 0x7fc184103900 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:30:47.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.060+0000 7fc18b16f700 1 -- 172.21.15.70:0/3427050213 >> 172.21.15.70:0/3427050213 conn(0x7fc1840fbe20 msgr2=0x7fc1840fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:30:47.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.060+0000 7fc18b16f700 1 -- 172.21.15.70:0/3427050213 shutdown_connections 2024-03-21T10:30:47.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.061+0000 7fc18b16f700 1 -- 172.21.15.70:0/3427050213 wait complete. 2024-03-21T10:30:47.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.061+0000 7fc18b16f700 1 Processor -- start 2024-03-21T10:30:47.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.061+0000 7fc18b16f700 1 -- start start 2024-03-21T10:30:47.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.061+0000 7fc18b16f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1841004c0 0x7fc1841983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:30:47.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.061+0000 7fc18b16f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1841034a0 0x7fc184198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:30:47.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.061+0000 7fc18b16f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc184103e40 0x7fc18419f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:30:47.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.061+0000 7fc18b16f700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18410de50 con 0x7fc1841004c0 2024-03-21T10:30:47.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.061+0000 7fc18b16f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18410dcd0 con 0x7fc1841034a0 2024-03-21T10:30:47.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.061+0000 7fc18b16f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18410dfd0 con 0x7fc184103e40 2024-03-21T10:30:47.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.061+0000 7fc18970c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc184103e40 0x7fc18419f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:30:47.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.061+0000 7fc18970c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc184103e40 0x7fc18419f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34094/0 (socket says 172.21.15.70:34094) 2024-03-21T10:30:47.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.061+0000 7fc18970c700 1 -- 172.21.15.70:0/178638419 learned_addr learned my addr 172.21.15.70:0/178638419 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-03-21T10:30:47.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.062+0000 7fc183fff700 1 --2- 172.21.15.70:0/178638419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1841034a0 0x7fc184198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:30:47.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.062+0000 7fc18970c700 1 -- 172.21.15.70:0/178638419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1841034a0 msgr2=0x7fc184198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:30:47.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.062+0000 7fc18970c700 1 --2- 172.21.15.70:0/178638419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1841034a0 0x7fc184198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:30:47.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.062+0000 7fc18970c700 1 -- 172.21.15.70:0/178638419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1841004c0 msgr2=0x7fc1841983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:30:47.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.062+0000 7fc18970c700 1 --2- 172.21.15.70:0/178638419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1841004c0 0x7fc1841983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:30:47.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.062+0000 7fc18970c700 1 -- 172.21.15.70:0/178638419 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1841a0030 con 0x7fc184103e40 2024-03-21T10:30:47.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.062+0000 7fc18970c700 1 --2- 172.21.15.70:0/178638419 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc184103e40 0x7fc18419f990 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc17c007d40 tx=0x7fc17c00a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:30:47.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.062+0000 7fc181ffb700 1 -- 172.21.15.70:0/178638419 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c016020 con 0x7fc184103e40 2024-03-21T10:30:47.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.062+0000 7fc181ffb700 1 -- 172.21.15.70:0/178638419 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc17c00ad60 con 0x7fc184103e40 2024-03-21T10:30:47.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.062+0000 7fc181ffb700 1 -- 172.21.15.70:0/178638419 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c011780 con 0x7fc184103e40 2024-03-21T10:30:47.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.062+0000 7fc18b16f700 1 -- 172.21.15.70:0/178638419 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1841a0320 con 0x7fc184103e40 2024-03-21T10:30:47.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.062+0000 7fc18b16f700 1 -- 172.21.15.70:0/178638419 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1841a08e0 con 0x7fc184103e40 2024-03-21T10:30:47.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.063+0000 7fc18b16f700 1 -- 172.21.15.70:0/178638419 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc184068e50 con 0x7fc184103e40 2024-03-21T10:30:47.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.064+0000 7fc181ffb700 1 -- 172.21.15.70:0/178638419 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fc17c019020 con 0x7fc184103e40 2024-03-21T10:30:47.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.065+0000 7fc181ffb700 1 --2- 172.21.15.70:0/178638419 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc1740a3030 0x7fc1740a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:30:47.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.065+0000 7fc181ffb700 1 -- 172.21.15.70:0/178638419 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4101+0+0 (secure 0 0 0) 0x7fc17c0ce400 con 0x7fc184103e40 2024-03-21T10:30:47.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.065+0000 7fc188f0b700 1 --2- 172.21.15.70:0/178638419 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc1740a3030 0x7fc1740a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:30:47.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.066+0000 7fc188f0b700 1 --2- 172.21.15.70:0/178638419 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc1740a3030 0x7fc1740a54f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc18406a6d0 tx=0x7fc178006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:30:47.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.070+0000 7fc181ffb700 1 -- 172.21.15.70:0/178638419 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc17c020020 con 0x7fc184103e40 2024-03-21T10:30:47.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:30:47.273+0000 7fc18b16f700 1 -- 172.21.15.70:0/178638419 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1840625d0 con 0x7fc1740a3030 2024-03-21T10:30:47.273 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:47 smithi070 ceph-mon[36330]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:30:47.273 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:47 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:30:47.273 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:47 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:47.273 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:47 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:30:47.273 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:47 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:47.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:47 smithi040 ceph-mon[36154]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:30:47.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:47 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:30:47.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:47 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:47.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:47 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:30:47.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:47 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:47.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:47 smithi031 ceph-mon[31673]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:30:47.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:47 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:30:47.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:47 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:47.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:47 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:30:47.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:47 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:30:48.227 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:48 smithi070 ceph-mon[36330]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:30:48.228 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:48 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:30:48.228 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:48 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:30:48.228 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:48 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:48.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:48 smithi040 ceph-mon[36154]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:30:48.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:48 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:30:48.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:48 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:30:48.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:48 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:48.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:48 smithi031 ceph-mon[31673]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:30:48.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:48 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:30:48.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:48 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:30:48.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:48 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:49.200 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:49 smithi070 ceph-mon[36330]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:30:49.349 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:49 smithi040 ceph-mon[36154]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:30:49.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:49 smithi031 ceph-mon[31673]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:30:50.366 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:50 smithi070 ceph-mon[36330]: from='client.? 172.21.15.70:0/625128031' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9409505a-d463-408c-a3cd-218380c04d68"}]: dispatch 2024-03-21T10:30:50.367 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:50 smithi070 ceph-mon[36330]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9409505a-d463-408c-a3cd-218380c04d68"}]: dispatch 2024-03-21T10:30:50.367 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:50 smithi070 ceph-mon[36330]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9409505a-d463-408c-a3cd-218380c04d68"}]': finished 2024-03-21T10:30:50.367 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:50 smithi070 ceph-mon[36330]: osdmap e35: 6 total, 5 up, 6 in 2024-03-21T10:30:50.367 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:50 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T10:30:50.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:50 smithi040 ceph-mon[36154]: from='client.? 172.21.15.70:0/625128031' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9409505a-d463-408c-a3cd-218380c04d68"}]: dispatch 2024-03-21T10:30:50.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:50 smithi040 ceph-mon[36154]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9409505a-d463-408c-a3cd-218380c04d68"}]: dispatch 2024-03-21T10:30:50.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:50 smithi040 ceph-mon[36154]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9409505a-d463-408c-a3cd-218380c04d68"}]': finished 2024-03-21T10:30:50.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:50 smithi040 ceph-mon[36154]: osdmap e35: 6 total, 5 up, 6 in 2024-03-21T10:30:50.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:50 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T10:30:50.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:50 smithi031 ceph-mon[31673]: from='client.? 172.21.15.70:0/625128031' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9409505a-d463-408c-a3cd-218380c04d68"}]: dispatch 2024-03-21T10:30:50.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:50 smithi031 ceph-mon[31673]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9409505a-d463-408c-a3cd-218380c04d68"}]: dispatch 2024-03-21T10:30:50.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:50 smithi031 ceph-mon[31673]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9409505a-d463-408c-a3cd-218380c04d68"}]': finished 2024-03-21T10:30:50.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:50 smithi031 ceph-mon[31673]: osdmap e35: 6 total, 5 up, 6 in 2024-03-21T10:30:50.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:50 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T10:30:51.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:51 smithi040 ceph-mon[36154]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:30:51.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:51 smithi040 ceph-mon[36154]: from='client.? 172.21.15.70:0/283878514' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:30:51.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:51 smithi031 ceph-mon[31673]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:30:51.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:51 smithi031 ceph-mon[31673]: from='client.? 172.21.15.70:0/283878514' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:30:51.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:51 smithi070 ceph-mon[36330]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:30:51.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:51 smithi070 ceph-mon[36330]: from='client.? 172.21.15.70:0/283878514' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:30:53.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:53 smithi040 ceph-mon[36154]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:30:53.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:53 smithi031 ceph-mon[31673]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:30:53.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:53 smithi070 ceph-mon[36330]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:30:55.301 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:55 smithi070 ceph-mon[36330]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:30:55.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:55 smithi040 ceph-mon[36154]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:30:55.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:55 smithi031 ceph-mon[31673]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:30:57.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:57 smithi040 ceph-mon[36154]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:30:57.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:57 smithi031 ceph-mon[31673]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:30:57.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:57 smithi070 ceph-mon[36330]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:30:58.134 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:58 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T10:30:58.134 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:58 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:58.134 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:58 smithi070 ceph-mon[36330]: Deploying daemon osd.5 on smithi070 2024-03-21T10:30:58.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:58 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T10:30:58.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:58 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:58.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:58 smithi040 ceph-mon[36154]: Deploying daemon osd.5 on smithi070 2024-03-21T10:30:58.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:58 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T10:30:58.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:58 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:30:58.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:58 smithi031 ceph-mon[31673]: Deploying daemon osd.5 on smithi070 2024-03-21T10:30:59.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:30:59 smithi040 ceph-mon[36154]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:30:59.470 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:30:59 smithi070 ceph-mon[36330]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:30:59.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:30:59 smithi031 ceph-mon[31673]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:31:01.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:01 smithi040 ceph-mon[36154]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:31:01.480 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:01 smithi070 ceph-mon[36330]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:31:01.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:01 smithi031 ceph-mon[31673]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:31:03.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:03 smithi040 ceph-mon[36154]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:31:03.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:03 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:03.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:03 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:03.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:03 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:31:04.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:03 smithi031 ceph-mon[31673]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:31:04.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:03 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:04.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:03 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:04.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:03 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:31:04.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:03 smithi070 ceph-mon[36330]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:31:04.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:03 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:04.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:03 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:04.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:03 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:31:05.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:05 smithi040 ceph-mon[36154]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:31:06.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:05 smithi031 ceph-mon[31673]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:31:06.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:05 smithi070 ceph-mon[36330]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:31:06.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:31:06.160+0000 7fc181ffb700 1 -- 172.21.15.70:0/178638419 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc1840625d0 con 0x7fc1740a3030 2024-03-21T10:31:06.160 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 5 on host 'smithi070' 2024-03-21T10:31:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:31:06.164+0000 7fc18b16f700 1 -- 172.21.15.70:0/178638419 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc1740a3030 msgr2=0x7fc1740a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:31:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:31:06.164+0000 7fc18b16f700 1 --2- 172.21.15.70:0/178638419 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc1740a3030 0x7fc1740a54f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc18406a6d0 tx=0x7fc178006d20 comp rx=0 tx=0).stop 2024-03-21T10:31:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:31:06.164+0000 7fc18b16f700 1 -- 172.21.15.70:0/178638419 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc184103e40 msgr2=0x7fc18419f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:31:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:31:06.164+0000 7fc18b16f700 1 --2- 172.21.15.70:0/178638419 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc184103e40 0x7fc18419f990 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc17c007d40 tx=0x7fc17c00a340 comp rx=0 tx=0).stop 2024-03-21T10:31:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:31:06.164+0000 7fc18b16f700 1 -- 172.21.15.70:0/178638419 shutdown_connections 2024-03-21T10:31:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:31:06.164+0000 7fc18b16f700 1 --2- 172.21.15.70:0/178638419 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc1740a3030 0x7fc1740a54f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:31:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:31:06.164+0000 7fc18b16f700 1 --2- 172.21.15.70:0/178638419 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc184103e40 0x7fc18419f990 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:31:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:31:06.164+0000 7fc18b16f700 1 --2- 172.21.15.70:0/178638419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1841004c0 0x7fc1841983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:31:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:31:06.164+0000 7fc18b16f700 1 --2- 172.21.15.70:0/178638419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1841034a0 0x7fc184198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:31:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:31:06.164+0000 7fc18b16f700 1 -- 172.21.15.70:0/178638419 >> 172.21.15.70:0/178638419 conn(0x7fc1840fbe20 msgr2=0x7fc184106740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:31:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:31:06.164+0000 7fc18b16f700 1 -- 172.21.15.70:0/178638419 shutdown_connections 2024-03-21T10:31:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:31:06.164+0000 7fc18b16f700 1 -- 172.21.15.70:0/178638419 wait complete. 2024-03-21T10:31:06.701 DEBUG:teuthology.orchestra.run.smithi070:osd.5> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.5.service 2024-03-21T10:31:06.703 INFO:tasks.cephadm:Deploying osd.6 on smithi031 with /dev/vg_nvme/lv_2... 2024-03-21T10:31:06.703 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-21T10:31:07.184 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:31:07.415 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:07 smithi070 ceph-mon[36330]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:31:07.415 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:07 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:07.415 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:07 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:07.415 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:07 smithi070 ceph-mon[36330]: from='osd.5 [v2:172.21.15.70:6808/3402830452,v1:172.21.15.70:6809/3402830452]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T10:31:07.415 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:07 smithi070 ceph-mon[36330]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T10:31:07.415 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:07 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:07.415 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:07 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:07.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:07 smithi040 ceph-mon[36154]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:31:07.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:07 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:07.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:07 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:07.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:07 smithi040 ceph-mon[36154]: from='osd.5 [v2:172.21.15.70:6808/3402830452,v1:172.21.15.70:6809/3402830452]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T10:31:07.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:07 smithi040 ceph-mon[36154]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T10:31:07.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:07 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:07.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:07 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:07.535 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:07 smithi031 ceph-mon[31673]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:31:07.535 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:07 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:07.535 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:07 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:07.535 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:07 smithi031 ceph-mon[31673]: from='osd.5 [v2:172.21.15.70:6808/3402830452,v1:172.21.15.70:6809/3402830452]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T10:31:07.536 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:07 smithi031 ceph-mon[31673]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T10:31:07.536 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:07 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:07.536 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:07 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:08.423 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:08 smithi070 ceph-mon[36330]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-21T10:31:08.423 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:08 smithi070 ceph-mon[36330]: from='osd.5 [v2:172.21.15.70:6808/3402830452,v1:172.21.15.70:6809/3402830452]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-03-21T10:31:08.423 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:08 smithi070 ceph-mon[36330]: osdmap e36: 6 total, 5 up, 6 in 2024-03-21T10:31:08.423 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:08 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T10:31:08.423 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:08 smithi070 ceph-mon[36330]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-03-21T10:31:08.423 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:31:08 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:31:08.166+0000 7fc52a022700 -1 osd.5 0 waiting for initial osdmap 2024-03-21T10:31:08.423 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:31:08 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:31:08.172+0000 7fc525e15700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T10:31:08.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:08 smithi040 ceph-mon[36154]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-21T10:31:08.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:08 smithi040 ceph-mon[36154]: from='osd.5 [v2:172.21.15.70:6808/3402830452,v1:172.21.15.70:6809/3402830452]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-03-21T10:31:08.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:08 smithi040 ceph-mon[36154]: osdmap e36: 6 total, 5 up, 6 in 2024-03-21T10:31:08.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:08 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T10:31:08.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:08 smithi040 ceph-mon[36154]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-03-21T10:31:08.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:08 smithi031 ceph-mon[31673]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-21T10:31:08.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:08 smithi031 ceph-mon[31673]: from='osd.5 [v2:172.21.15.70:6808/3402830452,v1:172.21.15.70:6809/3402830452]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-03-21T10:31:08.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:08 smithi031 ceph-mon[31673]: osdmap e36: 6 total, 5 up, 6 in 2024-03-21T10:31:08.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:08 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T10:31:08.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:08 smithi031 ceph-mon[31673]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-03-21T10:31:09.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:09 smithi070 ceph-mon[36330]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:31:09.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:09 smithi070 ceph-mon[36330]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-03-21T10:31:09.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:09 smithi070 ceph-mon[36330]: osdmap e37: 6 total, 5 up, 6 in 2024-03-21T10:31:09.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T10:31:09.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:09 smithi070 ceph-mon[36330]: osd.5 [v2:172.21.15.70:6808/3402830452,v1:172.21.15.70:6809/3402830452] boot 2024-03-21T10:31:09.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:09 smithi070 ceph-mon[36330]: osdmap e38: 6 total, 6 up, 6 in 2024-03-21T10:31:09.326 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T10:31:09.335 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:09 smithi031 ceph-mon[31673]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:31:09.335 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:09 smithi031 ceph-mon[31673]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-03-21T10:31:09.335 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:09 smithi031 ceph-mon[31673]: osdmap e37: 6 total, 5 up, 6 in 2024-03-21T10:31:09.335 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T10:31:09.335 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:09 smithi031 ceph-mon[31673]: osd.5 [v2:172.21.15.70:6808/3402830452,v1:172.21.15.70:6809/3402830452] boot 2024-03-21T10:31:09.335 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:09 smithi031 ceph-mon[31673]: osdmap e38: 6 total, 6 up, 6 in 2024-03-21T10:31:09.335 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T10:31:09.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:09 smithi040 ceph-mon[36154]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-21T10:31:09.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:09 smithi040 ceph-mon[36154]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-03-21T10:31:09.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:09 smithi040 ceph-mon[36154]: osdmap e37: 6 total, 5 up, 6 in 2024-03-21T10:31:09.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T10:31:09.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:09 smithi040 ceph-mon[36154]: osd.5 [v2:172.21.15.70:6808/3402830452,v1:172.21.15.70:6809/3402830452] boot 2024-03-21T10:31:09.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:09 smithi040 ceph-mon[36154]: osdmap e38: 6 total, 6 up, 6 in 2024-03-21T10:31:09.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T10:31:10.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:10 smithi031 ceph-mon[31673]: purged_snaps scrub starts 2024-03-21T10:31:10.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:10 smithi031 ceph-mon[31673]: purged_snaps scrub ok 2024-03-21T10:31:10.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:10 smithi031 ceph-mon[31673]: osdmap e39: 6 total, 6 up, 6 in 2024-03-21T10:31:10.405 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-21T10:31:10.406 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-21T10:31:10.406 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-03-21T10:31:10.406 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-03-21T10:31:10.406 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0603111 s, 174 MB/s 2024-03-21T10:31:10.406 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-03-21T10:31:10.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:10 smithi040 ceph-mon[36154]: purged_snaps scrub starts 2024-03-21T10:31:10.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:10 smithi040 ceph-mon[36154]: purged_snaps scrub ok 2024-03-21T10:31:10.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:10 smithi040 ceph-mon[36154]: osdmap e39: 6 total, 6 up, 6 in 2024-03-21T10:31:10.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:10 smithi070 ceph-mon[36330]: purged_snaps scrub starts 2024-03-21T10:31:10.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:10 smithi070 ceph-mon[36330]: purged_snaps scrub ok 2024-03-21T10:31:10.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:10 smithi070 ceph-mon[36330]: osdmap e39: 6 total, 6 up, 6 in 2024-03-21T10:31:11.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:11 smithi031 ceph-mon[31673]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:11.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:11 smithi031 ceph-mon[31673]: Detected new or changed devices on smithi070 2024-03-21T10:31:11.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:11 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:11.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:11 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:11.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:11 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:31:11.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:11 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:31:11.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:11 smithi031 ceph-mon[31673]: Adjusting osd_memory_target on smithi070 to 8540M 2024-03-21T10:31:11.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:11 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:11.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:11 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:31:11.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:11 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:31:11.338 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:11 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:11.339 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:11 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:31:11.339 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:11 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:31:11.339 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:11 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:31:11.339 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:11 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:11.339 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:11 smithi031 ceph-mon[31673]: osdmap e40: 6 total, 6 up, 6 in 2024-03-21T10:31:11.393 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi031:vg_nvme/lv_2 2024-03-21T10:31:11.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:11 smithi070 ceph-mon[36330]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:11.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:11 smithi070 ceph-mon[36330]: Detected new or changed devices on smithi070 2024-03-21T10:31:11.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:11 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:11.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:11 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:11.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:11 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:31:11.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:11 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:31:11.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:11 smithi070 ceph-mon[36330]: Adjusting osd_memory_target on smithi070 to 8540M 2024-03-21T10:31:11.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:11 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:11.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:11 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:31:11.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:11 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:31:11.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:11 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:11.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:11 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:31:11.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:11 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:31:11.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:11 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:31:11.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:11 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:11.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:11 smithi070 ceph-mon[36330]: osdmap e40: 6 total, 6 up, 6 in 2024-03-21T10:31:11.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:11 smithi040 ceph-mon[36154]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:11.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:11 smithi040 ceph-mon[36154]: Detected new or changed devices on smithi070 2024-03-21T10:31:11.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:11 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:11.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:11 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:11.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:11 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:31:11.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:11 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:31:11.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:11 smithi040 ceph-mon[36154]: Adjusting osd_memory_target on smithi070 to 8540M 2024-03-21T10:31:11.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:11 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:11.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:11 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:31:11.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:11 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:31:11.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:11 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:11.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:11 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:31:11.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:11 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:31:11.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:11 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:31:11.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:11 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:11.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:11 smithi040 ceph-mon[36154]: osdmap e40: 6 total, 6 up, 6 in 2024-03-21T10:31:11.841 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:31:13.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:13 smithi031 ceph-mon[31673]: pgmap v159: 1 pgs: 1 remapped+peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:13.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:13 smithi031 ceph-mon[31673]: osdmap e41: 6 total, 6 up, 6 in 2024-03-21T10:31:13.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:13 smithi070 ceph-mon[36330]: pgmap v159: 1 pgs: 1 remapped+peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:13.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:13 smithi070 ceph-mon[36330]: osdmap e41: 6 total, 6 up, 6 in 2024-03-21T10:31:13.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:13 smithi040 ceph-mon[36154]: pgmap v159: 1 pgs: 1 remapped+peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:13.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:13 smithi040 ceph-mon[36154]: osdmap e41: 6 total, 6 up, 6 in 2024-03-21T10:31:14.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.175+0000 7f2c1b200700 1 -- 172.21.15.31:0/377740576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c14105eb0 msgr2=0x7f2c1410c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:31:14.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.175+0000 7f2c1b200700 1 --2- 172.21.15.31:0/377740576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c14105eb0 0x7f2c1410c760 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2c10009f90 tx=0x7f2c1000b330 comp rx=0 tx=0).stop 2024-03-21T10:31:14.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.175+0000 7f2c1b200700 1 -- 172.21.15.31:0/377740576 shutdown_connections 2024-03-21T10:31:14.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.175+0000 7f2c1b200700 1 --2- 172.21.15.31:0/377740576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c14105eb0 0x7f2c1410c760 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:31:14.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.175+0000 7f2c1b200700 1 --2- 172.21.15.31:0/377740576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c14104310 0x7f2c14104730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:31:14.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.175+0000 7f2c1b200700 1 --2- 172.21.15.31:0/377740576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c14105510 0x7f2c14105970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:31:14.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.175+0000 7f2c1b200700 1 -- 172.21.15.31:0/377740576 >> 172.21.15.31:0/377740576 conn(0x7f2c140ff8b0 msgr2=0x7f2c14101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:31:14.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.175+0000 7f2c1b200700 1 -- 172.21.15.31:0/377740576 shutdown_connections 2024-03-21T10:31:14.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.175+0000 7f2c1b200700 1 -- 172.21.15.31:0/377740576 wait complete. 2024-03-21T10:31:14.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.176+0000 7f2c1b200700 1 Processor -- start 2024-03-21T10:31:14.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.176+0000 7f2c1b200700 1 -- start start 2024-03-21T10:31:14.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.177+0000 7f2c1b200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c14104310 0x7f2c1419a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:31:14.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.177+0000 7f2c1b200700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c14105510 0x7f2c1419a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:31:14.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.177+0000 7f2c1b200700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c14105eb0 0x7f2c141a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:31:14.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.177+0000 7f2c1b200700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1410ff30 con 0x7f2c14105eb0 2024-03-21T10:31:14.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.177+0000 7f2c1b200700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1410fdb0 con 0x7f2c14105510 2024-03-21T10:31:14.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.177+0000 7f2c1b200700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c141100b0 con 0x7f2c14104310 2024-03-21T10:31:14.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.177+0000 7f2c19400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c14105510 0x7f2c1419a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:31:14.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.177+0000 7f2c19400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c14105510 0x7f2c1419a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44458/0 (socket says 172.21.15.31:44458) 2024-03-21T10:31:14.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.177+0000 7f2c19400700 1 -- 172.21.15.31:0/3986998446 learned_addr learned my addr 172.21.15.31:0/3986998446 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:31:14.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.177+0000 7f2c1a800700 1 --2- 172.21.15.31:0/3986998446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c14105eb0 0x7f2c141a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:31:14.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.177+0000 7f2c19e00700 1 --2- 172.21.15.31:0/3986998446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c14104310 0x7f2c1419a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:31:14.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.178+0000 7f2c19400700 1 -- 172.21.15.31:0/3986998446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c14104310 msgr2=0x7f2c1419a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:31:14.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.178+0000 7f2c19400700 1 --2- 172.21.15.31:0/3986998446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c14104310 0x7f2c1419a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:31:14.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.178+0000 7f2c19400700 1 -- 172.21.15.31:0/3986998446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c14105eb0 msgr2=0x7f2c141a1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:31:14.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.178+0000 7f2c19400700 1 --2- 172.21.15.31:0/3986998446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c14105eb0 0x7f2c141a1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:31:14.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.178+0000 7f2c19400700 1 -- 172.21.15.31:0/3986998446 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c141a20f0 con 0x7f2c14105510 2024-03-21T10:31:14.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.178+0000 7f2c19400700 1 --2- 172.21.15.31:0/3986998446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c14105510 0x7f2c1419a9d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2c0400e9a0 tx=0x7f2c0400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:31:14.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.178+0000 7f2c0ea00700 1 -- 172.21.15.31:0/3986998446 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0400ca10 con 0x7f2c14105510 2024-03-21T10:31:14.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.179+0000 7f2c0ea00700 1 -- 172.21.15.31:0/3986998446 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c0400cbb0 con 0x7f2c14105510 2024-03-21T10:31:14.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.179+0000 7f2c1b200700 1 -- 172.21.15.31:0/3986998446 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c141a23e0 con 0x7f2c14105510 2024-03-21T10:31:14.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.179+0000 7f2c0ea00700 1 -- 172.21.15.31:0/3986998446 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c04018710 con 0x7f2c14105510 2024-03-21T10:31:14.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.179+0000 7f2c1b200700 1 -- 172.21.15.31:0/3986998446 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c141a2920 con 0x7f2c14105510 2024-03-21T10:31:14.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.180+0000 7f2c1b200700 1 -- 172.21.15.31:0/3986998446 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c14068e50 con 0x7f2c14105510 2024-03-21T10:31:14.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.182+0000 7f2c0ea00700 1 -- 172.21.15.31:0/3986998446 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f2c040188b0 con 0x7f2c14105510 2024-03-21T10:31:14.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.183+0000 7f2c0ea00700 1 --2- 172.21.15.31:0/3986998446 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2bfc0a2fb0 0x7f2bfc0a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:31:14.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.183+0000 7f2c0ea00700 1 -- 172.21.15.31:0/3986998446 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4533+0+0 (secure 0 0 0) 0x7f2c040cea70 con 0x7f2c14105510 2024-03-21T10:31:14.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.183+0000 7f2c19e00700 1 --2- 172.21.15.31:0/3986998446 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2bfc0a2fb0 0x7f2bfc0a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:31:14.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.184+0000 7f2c19e00700 1 --2- 172.21.15.31:0/3986998446 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2bfc0a2fb0 0x7f2bfc0a5470 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2c08009750 tx=0x7f2c08009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:31:14.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.186+0000 7f2c0ea00700 1 -- 172.21.15.31:0/3986998446 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2c0409c6c0 con 0x7f2c14105510 2024-03-21T10:31:14.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:14.424+0000 7f2c1b200700 1 -- 172.21.15.31:0/3986998446 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c14062690 con 0x7f2bfc0a2fb0 2024-03-21T10:31:14.424 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:14 smithi031 ceph-mon[31673]: osdmap e42: 6 total, 6 up, 6 in 2024-03-21T10:31:14.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:14 smithi070 ceph-mon[36330]: osdmap e42: 6 total, 6 up, 6 in 2024-03-21T10:31:14.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:14 smithi040 ceph-mon[36154]: osdmap e42: 6 total, 6 up, 6 in 2024-03-21T10:31:15.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:15 smithi031 ceph-mon[31673]: pgmap v162: 1 pgs: 1 remapped+peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:15.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:15 smithi031 ceph-mon[31673]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:31:15.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:15 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:31:15.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:15 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:31:15.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:15 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:31:15.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:15 smithi070 ceph-mon[36330]: pgmap v162: 1 pgs: 1 remapped+peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:15.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:15 smithi070 ceph-mon[36330]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:31:15.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:15 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:31:15.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:15 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:31:15.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:15 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:31:15.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:15 smithi040 ceph-mon[36154]: pgmap v162: 1 pgs: 1 remapped+peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:15.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:15 smithi040 ceph-mon[36154]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:31:15.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:15 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:31:15.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:15 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:31:15.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:15 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:31:17.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:17 smithi031 ceph-mon[31673]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-21T10:31:17.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:17 smithi070 ceph-mon[36330]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-21T10:31:17.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:17 smithi040 ceph-mon[36154]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-21T10:31:19.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:19 smithi031 ceph-mon[31673]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-21T10:31:19.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:19 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2806524905' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50"}]: dispatch 2024-03-21T10:31:19.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:19 smithi031 ceph-mon[31673]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50"}]: dispatch 2024-03-21T10:31:19.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:19 smithi031 ceph-mon[31673]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50"}]': finished 2024-03-21T10:31:19.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:19 smithi031 ceph-mon[31673]: osdmap e43: 7 total, 6 up, 7 in 2024-03-21T10:31:19.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:19 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T10:31:19.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:19 smithi070 ceph-mon[36330]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-21T10:31:19.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:19 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2806524905' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50"}]: dispatch 2024-03-21T10:31:19.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:19 smithi070 ceph-mon[36330]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50"}]: dispatch 2024-03-21T10:31:19.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:19 smithi070 ceph-mon[36330]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50"}]': finished 2024-03-21T10:31:19.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:19 smithi070 ceph-mon[36330]: osdmap e43: 7 total, 6 up, 7 in 2024-03-21T10:31:19.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:19 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T10:31:19.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:19 smithi040 ceph-mon[36154]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-21T10:31:19.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:19 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2806524905' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50"}]: dispatch 2024-03-21T10:31:19.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:19 smithi040 ceph-mon[36154]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50"}]: dispatch 2024-03-21T10:31:19.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:19 smithi040 ceph-mon[36154]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50"}]': finished 2024-03-21T10:31:19.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:19 smithi040 ceph-mon[36154]: osdmap e43: 7 total, 6 up, 7 in 2024-03-21T10:31:19.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:19 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T10:31:20.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:20 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/803732238' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:31:20.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:20 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/803732238' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:31:20.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:20 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/803732238' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:31:21.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:21 smithi031 ceph-mon[31673]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-21T10:31:21.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:21 smithi070 ceph-mon[36330]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-21T10:31:21.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:21 smithi040 ceph-mon[36154]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-21T10:31:23.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:23 smithi070 ceph-mon[36330]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-21T10:31:23.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:23 smithi040 ceph-mon[36154]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-21T10:31:23.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:23 smithi031 ceph-mon[31673]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-21T10:31:25.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:25 smithi070 ceph-mon[36330]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T10:31:25.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:25 smithi040 ceph-mon[36154]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T10:31:25.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:25 smithi031 ceph-mon[31673]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T10:31:27.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:27 smithi070 ceph-mon[36330]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:27.631 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:27 smithi031 ceph-mon[31673]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:27.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:27 smithi040 ceph-mon[36154]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:29.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:29 smithi070 ceph-mon[36330]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:29.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:29 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T10:31:29.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:29 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:31:29.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:29 smithi070 ceph-mon[36330]: Deploying daemon osd.6 on smithi031 2024-03-21T10:31:29.626 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:29 smithi031 ceph-mon[31673]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:29.626 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:29 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T10:31:29.626 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:29 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:31:29.626 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:29 smithi031 ceph-mon[31673]: Deploying daemon osd.6 on smithi031 2024-03-21T10:31:29.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:29 smithi040 ceph-mon[36154]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:29.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:29 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T10:31:29.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:29 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:31:29.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:29 smithi040 ceph-mon[36154]: Deploying daemon osd.6 on smithi031 2024-03-21T10:31:31.458 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:31 smithi031 ceph-mon[31673]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:31.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:31 smithi070 ceph-mon[36330]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:31.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:31 smithi040 ceph-mon[36154]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:33.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:33 smithi031 ceph-mon[31673]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:33.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:33 smithi040 ceph-mon[36154]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:33.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:33 smithi070 ceph-mon[36330]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:35.637 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:35 smithi031 ceph-mon[31673]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:35.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:35 smithi040 ceph-mon[36154]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:35.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:35 smithi070 ceph-mon[36330]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:37.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:37 smithi031 ceph-mon[31673]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:37.627 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:37 smithi070 ceph-mon[36330]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:37.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:37 smithi040 ceph-mon[36154]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:38.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:38 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:38.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:38 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:38.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:38 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:31:38.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:38 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:38.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:38 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:38.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:38 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:31:38.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:38 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:38.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:38 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:38.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:38 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:31:39.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:39 smithi031 ceph-mon[31673]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:39.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:39 smithi040 ceph-mon[36154]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:40.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:39 smithi070 ceph-mon[36330]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:41.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:41 smithi031 ceph-mon[31673]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:41.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:41 smithi031 ceph-mon[31673]: from='osd.6 [v2:172.21.15.31:6818/1193497058,v1:172.21.15.31:6819/1193497058]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T10:31:41.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:41 smithi040 ceph-mon[36154]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:41.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:41 smithi040 ceph-mon[36154]: from='osd.6 [v2:172.21.15.31:6818/1193497058,v1:172.21.15.31:6819/1193497058]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T10:31:42.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:41 smithi070 ceph-mon[36330]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:42.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:41 smithi070 ceph-mon[36330]: from='osd.6 [v2:172.21.15.31:6818/1193497058,v1:172.21.15.31:6819/1193497058]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T10:31:42.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:42 smithi040 ceph-mon[36154]: from='osd.6 [v2:172.21.15.31:6818/1193497058,v1:172.21.15.31:6819/1193497058]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T10:31:42.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:42 smithi040 ceph-mon[36154]: osdmap e44: 7 total, 6 up, 7 in 2024-03-21T10:31:42.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:42 smithi040 ceph-mon[36154]: from='osd.6 [v2:172.21.15.31:6818/1193497058,v1:172.21.15.31:6819/1193497058]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-21T10:31:42.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:42 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T10:31:42.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:42 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:42.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:42 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:42.940 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:42 smithi031 ceph-mon[31673]: from='osd.6 [v2:172.21.15.31:6818/1193497058,v1:172.21.15.31:6819/1193497058]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T10:31:42.941 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:42 smithi031 ceph-mon[31673]: osdmap e44: 7 total, 6 up, 7 in 2024-03-21T10:31:42.941 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:42 smithi031 ceph-mon[31673]: from='osd.6 [v2:172.21.15.31:6818/1193497058,v1:172.21.15.31:6819/1193497058]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-21T10:31:42.941 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:42 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T10:31:42.941 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:42 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:42.941 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:42 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:43.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:42 smithi070 ceph-mon[36330]: from='osd.6 [v2:172.21.15.31:6818/1193497058,v1:172.21.15.31:6819/1193497058]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T10:31:43.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:42 smithi070 ceph-mon[36330]: osdmap e44: 7 total, 6 up, 7 in 2024-03-21T10:31:43.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:42 smithi070 ceph-mon[36330]: from='osd.6 [v2:172.21.15.31:6818/1193497058,v1:172.21.15.31:6819/1193497058]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-21T10:31:43.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:42 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T10:31:43.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:42 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:43.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:42 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:43.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:43.226+0000 7f2c0ea00700 1 -- 172.21.15.31:0/3986998446 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2c14062690 con 0x7f2bfc0a2fb0 2024-03-21T10:31:43.228 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 6 on host 'smithi031' 2024-03-21T10:31:43.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:43.230+0000 7f2c1b200700 1 -- 172.21.15.31:0/3986998446 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2bfc0a2fb0 msgr2=0x7f2bfc0a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:31:43.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:43.230+0000 7f2c1b200700 1 --2- 172.21.15.31:0/3986998446 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2bfc0a2fb0 0x7f2bfc0a5470 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2c08009750 tx=0x7f2c08009310 comp rx=0 tx=0).stop 2024-03-21T10:31:43.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:43.230+0000 7f2c1b200700 1 -- 172.21.15.31:0/3986998446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c14105510 msgr2=0x7f2c1419a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:31:43.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:43.230+0000 7f2c1b200700 1 --2- 172.21.15.31:0/3986998446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c14105510 0x7f2c1419a9d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2c0400e9a0 tx=0x7f2c0400ecb0 comp rx=0 tx=0).stop 2024-03-21T10:31:43.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:43.231+0000 7f2c1b200700 1 -- 172.21.15.31:0/3986998446 shutdown_connections 2024-03-21T10:31:43.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:43.231+0000 7f2c1b200700 1 --2- 172.21.15.31:0/3986998446 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2bfc0a2fb0 0x7f2bfc0a5470 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:31:43.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:43.231+0000 7f2c1b200700 1 --2- 172.21.15.31:0/3986998446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c14105eb0 0x7f2c141a1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:31:43.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:43.231+0000 7f2c1b200700 1 --2- 172.21.15.31:0/3986998446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c14104310 0x7f2c1419a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:31:43.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:43.231+0000 7f2c1b200700 1 --2- 172.21.15.31:0/3986998446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c14105510 0x7f2c1419a9d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:31:43.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:43.231+0000 7f2c1b200700 1 -- 172.21.15.31:0/3986998446 >> 172.21.15.31:0/3986998446 conn(0x7f2c140ff8b0 msgr2=0x7f2c141087b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:31:43.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:43.231+0000 7f2c1b200700 1 -- 172.21.15.31:0/3986998446 shutdown_connections 2024-03-21T10:31:43.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:31:43.231+0000 7f2c1b200700 1 -- 172.21.15.31:0/3986998446 wait complete. 2024-03-21T10:31:43.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:43 smithi040 ceph-mon[36154]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:43.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:43 smithi040 ceph-mon[36154]: from='osd.6 [v2:172.21.15.31:6818/1193497058,v1:172.21.15.31:6819/1193497058]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-21T10:31:43.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:43 smithi040 ceph-mon[36154]: osdmap e45: 7 total, 6 up, 7 in 2024-03-21T10:31:43.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:43 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T10:31:43.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:43 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T10:31:43.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:43 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:31:43.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:43 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:31:43.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:43 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:43.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:43 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:43.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:43 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:43.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:43 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:31:44.023 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:43 smithi031 ceph-mon[31673]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:44.023 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:43 smithi031 ceph-mon[31673]: from='osd.6 [v2:172.21.15.31:6818/1193497058,v1:172.21.15.31:6819/1193497058]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-21T10:31:44.023 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:43 smithi031 ceph-mon[31673]: osdmap e45: 7 total, 6 up, 7 in 2024-03-21T10:31:44.023 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T10:31:44.023 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T10:31:44.023 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:31:44.023 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:31:44.023 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:44.023 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:44.024 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:44.024 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:31:44.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:43 smithi070 ceph-mon[36330]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:44.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:43 smithi070 ceph-mon[36330]: from='osd.6 [v2:172.21.15.31:6818/1193497058,v1:172.21.15.31:6819/1193497058]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-21T10:31:44.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:43 smithi070 ceph-mon[36330]: osdmap e45: 7 total, 6 up, 7 in 2024-03-21T10:31:44.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:43 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T10:31:44.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:43 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T10:31:44.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:43 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:31:44.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:43 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:31:44.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:43 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:44.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:43 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:44.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:43 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:44.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:43 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:31:44.101 DEBUG:teuthology.orchestra.run.smithi031:osd.6> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.6.service 2024-03-21T10:31:44.108 INFO:tasks.cephadm:Deploying osd.7 on smithi040 with /dev/vg_nvme/lv_2... 2024-03-21T10:31:44.108 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-21T10:31:44.366 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T10:31:44.745 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:44 smithi040 ceph-mon[36154]: purged_snaps scrub starts 2024-03-21T10:31:44.745 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:44 smithi040 ceph-mon[36154]: purged_snaps scrub ok 2024-03-21T10:31:44.745 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:44 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T10:31:44.745 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:44 smithi040 ceph-mon[36154]: osd.6 [v2:172.21.15.31:6818/1193497058,v1:172.21.15.31:6819/1193497058] boot 2024-03-21T10:31:44.745 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:44 smithi040 ceph-mon[36154]: osdmap e46: 7 total, 7 up, 7 in 2024-03-21T10:31:44.745 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:44 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T10:31:44.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:44 smithi031 ceph-mon[31673]: purged_snaps scrub starts 2024-03-21T10:31:44.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:44 smithi031 ceph-mon[31673]: purged_snaps scrub ok 2024-03-21T10:31:44.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:44 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T10:31:44.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:44 smithi031 ceph-mon[31673]: osd.6 [v2:172.21.15.31:6818/1193497058,v1:172.21.15.31:6819/1193497058] boot 2024-03-21T10:31:44.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:44 smithi031 ceph-mon[31673]: osdmap e46: 7 total, 7 up, 7 in 2024-03-21T10:31:44.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:44 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T10:31:45.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:44 smithi070 ceph-mon[36330]: purged_snaps scrub starts 2024-03-21T10:31:45.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:44 smithi070 ceph-mon[36330]: purged_snaps scrub ok 2024-03-21T10:31:45.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:44 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T10:31:45.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:44 smithi070 ceph-mon[36330]: osd.6 [v2:172.21.15.31:6818/1193497058,v1:172.21.15.31:6819/1193497058] boot 2024-03-21T10:31:45.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:44 smithi070 ceph-mon[36330]: osdmap e46: 7 total, 7 up, 7 in 2024-03-21T10:31:45.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:44 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T10:31:45.783 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:45 smithi040 ceph-mon[36154]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:45.783 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:45 smithi040 ceph-mon[36154]: osdmap e47: 7 total, 7 up, 7 in 2024-03-21T10:31:45.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:45 smithi031 ceph-mon[31673]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:45.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:45 smithi031 ceph-mon[31673]: osdmap e47: 7 total, 7 up, 7 in 2024-03-21T10:31:46.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:45 smithi070 ceph-mon[36330]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-21T10:31:46.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:45 smithi070 ceph-mon[36330]: osdmap e47: 7 total, 7 up, 7 in 2024-03-21T10:31:46.405 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-21T10:31:46.405 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-21T10:31:46.405 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-03-21T10:31:46.405 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-03-21T10:31:46.405 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012587 s, 833 MB/s 2024-03-21T10:31:46.405 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-03-21T10:31:47.124 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi040:vg_nvme/lv_2 2024-03-21T10:31:47.373 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T10:31:47.742 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:47 smithi040 ceph-mon[36154]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:31:48.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:47 smithi031 ceph-mon[31673]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:31:48.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:47 smithi070 ceph-mon[36330]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:31:49.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.164+0000 7fac7d662700 1 -- 172.21.15.40:0/83938717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac781033e0 msgr2=0x7fac78103840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:31:49.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.164+0000 7fac7d662700 1 --2- 172.21.15.40:0/83938717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac781033e0 0x7fac78103840 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fac600099c0 tx=0x7fac60009cd0 comp rx=0 tx=0).stop 2024-03-21T10:31:49.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.164+0000 7fac7d662700 1 -- 172.21.15.40:0/83938717 shutdown_connections 2024-03-21T10:31:49.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.164+0000 7fac7d662700 1 --2- 172.21.15.40:0/83938717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac78103d80 0x7fac7810a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:31:49.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.164+0000 7fac7d662700 1 --2- 172.21.15.40:0/83938717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac781021e0 0x7fac78102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:31:49.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.164+0000 7fac7d662700 1 --2- 172.21.15.40:0/83938717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac781033e0 0x7fac78103840 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:31:49.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.164+0000 7fac7d662700 1 -- 172.21.15.40:0/83938717 >> 172.21.15.40:0/83938717 conn(0x7fac780fd7a0 msgr2=0x7fac780ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:31:49.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.164+0000 7fac7d662700 1 -- 172.21.15.40:0/83938717 shutdown_connections 2024-03-21T10:31:49.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.164+0000 7fac7d662700 1 -- 172.21.15.40:0/83938717 wait complete. 2024-03-21T10:31:49.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.165+0000 7fac7d662700 1 Processor -- start 2024-03-21T10:31:49.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.165+0000 7fac7d662700 1 -- start start 2024-03-21T10:31:49.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.165+0000 7fac7d662700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac781021e0 0x7fac7819a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:31:49.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.165+0000 7fac7d662700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac781033e0 0x7fac7819ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:31:49.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.165+0000 7fac7d662700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac78103d80 0x7fac781a1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:31:49.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.165+0000 7fac7d662700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7810de30 con 0x7fac78103d80 2024-03-21T10:31:49.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.165+0000 7fac7d662700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7810dcb0 con 0x7fac781033e0 2024-03-21T10:31:49.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.165+0000 7fac7d662700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7810dfb0 con 0x7fac781021e0 2024-03-21T10:31:49.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.165+0000 7fac767fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac781033e0 0x7fac7819ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:31:49.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.165+0000 7fac767fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac781033e0 0x7fac7819ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36352/0 (socket says 172.21.15.40:36352) 2024-03-21T10:31:49.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.165+0000 7fac767fc700 1 -- 172.21.15.40:0/368096870 learned_addr learned my addr 172.21.15.40:0/368096870 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-21T10:31:49.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.165+0000 7fac767fc700 1 -- 172.21.15.40:0/368096870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac781021e0 msgr2=0x7fac7819a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:31:49.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.165+0000 7fac76ffd700 1 --2- 172.21.15.40:0/368096870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac781021e0 0x7fac7819a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:31:49.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.165+0000 7fac767fc700 1 --2- 172.21.15.40:0/368096870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac781021e0 0x7fac7819a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:31:49.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.165+0000 7fac767fc700 1 -- 172.21.15.40:0/368096870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac78103d80 msgr2=0x7fac781a1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:31:49.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.165+0000 7fac767fc700 1 --2- 172.21.15.40:0/368096870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac78103d80 0x7fac781a1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:31:49.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.165+0000 7fac767fc700 1 -- 172.21.15.40:0/368096870 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac781a2290 con 0x7fac781033e0 2024-03-21T10:31:49.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.166+0000 7fac767fc700 1 --2- 172.21.15.40:0/368096870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac781033e0 0x7fac7819ab70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fac60003890 tx=0x7fac60003730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:31:49.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.166+0000 7fac6ffff700 1 -- 172.21.15.40:0/368096870 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac60027070 con 0x7fac781033e0 2024-03-21T10:31:49.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.166+0000 7fac7d662700 1 -- 172.21.15.40:0/368096870 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac781a2520 con 0x7fac781033e0 2024-03-21T10:31:49.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.166+0000 7fac6ffff700 1 -- 172.21.15.40:0/368096870 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac60022470 con 0x7fac781033e0 2024-03-21T10:31:49.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.166+0000 7fac6ffff700 1 -- 172.21.15.40:0/368096870 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6000f6a0 con 0x7fac781033e0 2024-03-21T10:31:49.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.166+0000 7fac7d662700 1 -- 172.21.15.40:0/368096870 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac781a29f0 con 0x7fac781033e0 2024-03-21T10:31:49.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.167+0000 7fac7d662700 1 -- 172.21.15.40:0/368096870 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac78068e50 con 0x7fac781033e0 2024-03-21T10:31:49.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.168+0000 7fac6ffff700 1 -- 172.21.15.40:0/368096870 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fac6002a070 con 0x7fac781033e0 2024-03-21T10:31:49.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.169+0000 7fac6ffff700 1 --2- 172.21.15.40:0/368096870 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fac640a2fb0 0x7fac640a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:31:49.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.169+0000 7fac6ffff700 1 -- 172.21.15.40:0/368096870 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4965+0+0 (secure 0 0 0) 0x7fac600cf670 con 0x7fac781033e0 2024-03-21T10:31:49.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.169+0000 7fac76ffd700 1 --2- 172.21.15.40:0/368096870 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fac640a2fb0 0x7fac640a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:31:49.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.170+0000 7fac76ffd700 1 --2- 172.21.15.40:0/368096870 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fac640a2fb0 0x7fac640a5470 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fac78103240 tx=0x7fac68008640 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:31:49.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.173+0000 7fac6ffff700 1 -- 172.21.15.40:0/368096870 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fac60013070 con 0x7fac781033e0 2024-03-21T10:31:49.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:31:49.388+0000 7fac7d662700 1 -- 172.21.15.40:0/368096870 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fac78075ae0 con 0x7fac640a2fb0 2024-03-21T10:31:49.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:49 smithi031 ceph-mon[31673]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:31:49.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:49 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:31:49.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:49 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:31:49.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:49 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:31:49.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:49 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:49.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:49 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:49.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:49 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:31:49.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:49 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:31:49.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:49 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:31:49.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:49 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:49.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:49 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:31:49.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:49 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:31:49.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:49 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:49.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:49 smithi040 ceph-mon[36154]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:31:49.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:49 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:31:49.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:49 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:31:49.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:49 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:31:49.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:49 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:49.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:49 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:49.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:49 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:31:49.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:49 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:31:49.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:49 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:31:49.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:49 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:49.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:49 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:31:49.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:49 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:31:49.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:49 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:50.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:49 smithi070 ceph-mon[36330]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:31:50.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:49 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:31:50.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:49 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:31:50.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:49 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:31:50.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:49 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:50.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:49 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:50.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:49 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:31:50.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:49 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:31:50.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:49 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:31:50.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:49 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:50.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:49 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:31:50.076 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:49 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:31:50.076 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:49 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:31:50.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:50 smithi040 ceph-mon[36154]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:31:50.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:50 smithi040 ceph-mon[36154]: Detected new or changed devices on smithi031 2024-03-21T10:31:50.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:50 smithi040 ceph-mon[36154]: Adjusting osd_memory_target on smithi031 to 5698M 2024-03-21T10:31:51.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:50 smithi031 ceph-mon[31673]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:31:51.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:50 smithi031 ceph-mon[31673]: Detected new or changed devices on smithi031 2024-03-21T10:31:51.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:50 smithi031 ceph-mon[31673]: Adjusting osd_memory_target on smithi031 to 5698M 2024-03-21T10:31:51.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:50 smithi070 ceph-mon[36330]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:31:51.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:50 smithi070 ceph-mon[36330]: Detected new or changed devices on smithi031 2024-03-21T10:31:51.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:50 smithi070 ceph-mon[36330]: Adjusting osd_memory_target on smithi031 to 5698M 2024-03-21T10:31:51.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:51 smithi040 ceph-mon[36154]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:31:52.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:51 smithi031 ceph-mon[31673]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:31:52.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:51 smithi070 ceph-mon[36330]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:31:53.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:52 smithi031 ceph-mon[31673]: from='client.? 172.21.15.40:0/1178678284' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db228a87-29a5-45d4-8fe6-a0a554262eb7"}]: dispatch 2024-03-21T10:31:53.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:52 smithi031 ceph-mon[31673]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db228a87-29a5-45d4-8fe6-a0a554262eb7"}]: dispatch 2024-03-21T10:31:53.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:52 smithi031 ceph-mon[31673]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db228a87-29a5-45d4-8fe6-a0a554262eb7"}]': finished 2024-03-21T10:31:53.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:52 smithi031 ceph-mon[31673]: osdmap e48: 8 total, 7 up, 8 in 2024-03-21T10:31:53.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:52 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T10:31:53.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:52 smithi070 ceph-mon[36330]: from='client.? 172.21.15.40:0/1178678284' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db228a87-29a5-45d4-8fe6-a0a554262eb7"}]: dispatch 2024-03-21T10:31:53.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:52 smithi070 ceph-mon[36330]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db228a87-29a5-45d4-8fe6-a0a554262eb7"}]: dispatch 2024-03-21T10:31:53.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:52 smithi070 ceph-mon[36330]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db228a87-29a5-45d4-8fe6-a0a554262eb7"}]': finished 2024-03-21T10:31:53.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:52 smithi070 ceph-mon[36330]: osdmap e48: 8 total, 7 up, 8 in 2024-03-21T10:31:53.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:52 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T10:31:53.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:52 smithi040 ceph-mon[36154]: from='client.? 172.21.15.40:0/1178678284' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db228a87-29a5-45d4-8fe6-a0a554262eb7"}]: dispatch 2024-03-21T10:31:53.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:52 smithi040 ceph-mon[36154]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db228a87-29a5-45d4-8fe6-a0a554262eb7"}]: dispatch 2024-03-21T10:31:53.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:52 smithi040 ceph-mon[36154]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db228a87-29a5-45d4-8fe6-a0a554262eb7"}]': finished 2024-03-21T10:31:53.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:52 smithi040 ceph-mon[36154]: osdmap e48: 8 total, 7 up, 8 in 2024-03-21T10:31:53.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:52 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T10:31:54.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:53 smithi031 ceph-mon[31673]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:31:54.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:53 smithi031 ceph-mon[31673]: from='client.? 172.21.15.40:0/3887759257' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:31:54.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:53 smithi070 ceph-mon[36330]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:31:54.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:53 smithi070 ceph-mon[36330]: from='client.? 172.21.15.40:0/3887759257' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:31:54.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:53 smithi040 ceph-mon[36154]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:31:54.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:53 smithi040 ceph-mon[36154]: from='client.? 172.21.15.40:0/3887759257' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:31:56.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:55 smithi031 ceph-mon[31673]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:31:56.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:55 smithi070 ceph-mon[36330]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:31:56.104 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:55 smithi040 ceph-mon[36154]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:31:57.716 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:57 smithi040 ceph-mon[36154]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:31:58.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:57 smithi031 ceph-mon[31673]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:31:58.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:57 smithi070 ceph-mon[36330]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:31:59.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:31:59 smithi031 ceph-mon[31673]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:32:00.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:31:59 smithi070 ceph-mon[36330]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:32:00.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:31:59 smithi040 ceph-mon[36154]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:32:00.842 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:00 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T10:32:00.842 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:00 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:01.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:00 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T10:32:01.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:00 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:01.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:00 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T10:32:01.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:00 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:02.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:01 smithi031 ceph-mon[31673]: Deploying daemon osd.7 on smithi040 2024-03-21T10:32:02.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:01 smithi031 ceph-mon[31673]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:32:02.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:01 smithi070 ceph-mon[36330]: Deploying daemon osd.7 on smithi040 2024-03-21T10:32:02.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:01 smithi070 ceph-mon[36330]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:32:02.086 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:01 smithi040 ceph-mon[36154]: Deploying daemon osd.7 on smithi040 2024-03-21T10:32:02.086 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:01 smithi040 ceph-mon[36154]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:32:04.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:03 smithi031 ceph-mon[31673]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:32:04.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:03 smithi070 ceph-mon[36330]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:32:04.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:03 smithi040 ceph-mon[36154]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:32:06.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:05 smithi031 ceph-mon[31673]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:32:06.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:05 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:06.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:05 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:06.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:05 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:32:06.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:05 smithi070 ceph-mon[36330]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:32:06.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:05 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:06.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:05 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:06.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:05 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:32:06.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:05 smithi040 ceph-mon[36154]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:32:06.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:05 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:06.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:05 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:06.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:05 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:32:07.818 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:07 smithi040 ceph-mon[36154]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:32:08.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:07 smithi031 ceph-mon[31673]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:32:08.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:07 smithi070 ceph-mon[36330]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:32:09.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:32:09.377+0000 7fac6ffff700 1 -- 172.21.15.40:0/368096870 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fac78075ae0 con 0x7fac640a2fb0 2024-03-21T10:32:09.377 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 7 on host 'smithi040' 2024-03-21T10:32:09.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:32:09.379+0000 7fac7d662700 1 -- 172.21.15.40:0/368096870 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fac640a2fb0 msgr2=0x7fac640a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:32:09.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:32:09.379+0000 7fac7d662700 1 --2- 172.21.15.40:0/368096870 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fac640a2fb0 0x7fac640a5470 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fac78103240 tx=0x7fac68008640 comp rx=0 tx=0).stop 2024-03-21T10:32:09.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:32:09.379+0000 7fac7d662700 1 -- 172.21.15.40:0/368096870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac781033e0 msgr2=0x7fac7819ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:32:09.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:32:09.379+0000 7fac7d662700 1 --2- 172.21.15.40:0/368096870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac781033e0 0x7fac7819ab70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fac60003890 tx=0x7fac60003730 comp rx=0 tx=0).stop 2024-03-21T10:32:09.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:32:09.379+0000 7fac7d662700 1 -- 172.21.15.40:0/368096870 shutdown_connections 2024-03-21T10:32:09.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:32:09.379+0000 7fac7d662700 1 --2- 172.21.15.40:0/368096870 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fac640a2fb0 0x7fac640a5470 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:32:09.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:32:09.379+0000 7fac7d662700 1 --2- 172.21.15.40:0/368096870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac78103d80 0x7fac781a1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:32:09.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:32:09.379+0000 7fac7d662700 1 --2- 172.21.15.40:0/368096870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac781021e0 0x7fac7819a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:32:09.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:32:09.379+0000 7fac7d662700 1 --2- 172.21.15.40:0/368096870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac781033e0 0x7fac7819ab70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:32:09.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:32:09.379+0000 7fac7d662700 1 -- 172.21.15.40:0/368096870 >> 172.21.15.40:0/368096870 conn(0x7fac780fd7a0 msgr2=0x7fac780743c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:32:09.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:32:09.379+0000 7fac7d662700 1 -- 172.21.15.40:0/368096870 shutdown_connections 2024-03-21T10:32:09.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:32:09.379+0000 7fac7d662700 1 -- 172.21.15.40:0/368096870 wait complete. 2024-03-21T10:32:09.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:09 smithi040 ceph-mon[36154]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:32:09.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:09.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:09.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:09.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:32:09.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:09.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:09.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:09.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:32:09.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:09 smithi040 ceph-mon[36154]: from='osd.7 [v2:172.21.15.40:6816/3825628119,v1:172.21.15.40:6817/3825628119]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T10:32:09.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:09 smithi040 ceph-mon[36154]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T10:32:09.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:09 smithi031 ceph-mon[31673]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:32:09.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:09.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:09.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:09.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:32:09.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:09.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:09.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:09.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:32:09.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:09 smithi031 ceph-mon[31673]: from='osd.7 [v2:172.21.15.40:6816/3825628119,v1:172.21.15.40:6817/3825628119]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T10:32:09.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:09 smithi031 ceph-mon[31673]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T10:32:09.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:09 smithi070 ceph-mon[36330]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:32:09.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:09.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:09.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:09.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:32:09.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:09.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:09.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:09.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:32:09.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:09 smithi070 ceph-mon[36330]: from='osd.7 [v2:172.21.15.40:6816/3825628119,v1:172.21.15.40:6817/3825628119]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T10:32:09.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:09 smithi070 ceph-mon[36330]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T10:32:09.991 DEBUG:teuthology.orchestra.run.smithi040:osd.7> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.7.service 2024-03-21T10:32:09.995 INFO:tasks.cephadm:Deploying osd.8 on smithi070 with /dev/vg_nvme/lv_2... 2024-03-21T10:32:09.995 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-21T10:32:10.240 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.c/config 2024-03-21T10:32:11.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:11 smithi040 ceph-mon[36154]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:32:11.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:11 smithi040 ceph-mon[36154]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-21T10:32:11.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:11 smithi040 ceph-mon[36154]: from='osd.7 [v2:172.21.15.40:6816/3825628119,v1:172.21.15.40:6817/3825628119]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-21T10:32:11.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:11 smithi040 ceph-mon[36154]: osdmap e49: 8 total, 7 up, 8 in 2024-03-21T10:32:11.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:11 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T10:32:11.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:11 smithi040 ceph-mon[36154]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-21T10:32:11.673 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:11 smithi070 ceph-mon[36330]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:32:11.674 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:11 smithi070 ceph-mon[36330]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-21T10:32:11.674 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:11 smithi070 ceph-mon[36330]: from='osd.7 [v2:172.21.15.40:6816/3825628119,v1:172.21.15.40:6817/3825628119]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-21T10:32:11.674 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:11 smithi070 ceph-mon[36330]: osdmap e49: 8 total, 7 up, 8 in 2024-03-21T10:32:11.674 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:11 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T10:32:11.674 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:11 smithi070 ceph-mon[36330]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-21T10:32:11.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:11 smithi031 ceph-mon[31673]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:32:11.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:11 smithi031 ceph-mon[31673]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-21T10:32:11.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:11 smithi031 ceph-mon[31673]: from='osd.7 [v2:172.21.15.40:6816/3825628119,v1:172.21.15.40:6817/3825628119]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-21T10:32:11.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:11 smithi031 ceph-mon[31673]: osdmap e49: 8 total, 7 up, 8 in 2024-03-21T10:32:11.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:11 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T10:32:11.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:11 smithi031 ceph-mon[31673]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-21T10:32:12.180 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-21T10:32:12.180 INFO:teuthology.orchestra.run.smithi070.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-21T10:32:12.180 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10+0 records in 2024-03-21T10:32:12.180 INFO:teuthology.orchestra.run.smithi070.stderr:10+0 records out 2024-03-21T10:32:12.180 INFO:teuthology.orchestra.run.smithi070.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00834608 s, 1.3 GB/s 2024-03-21T10:32:12.180 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 2024-03-21T10:32:12.180 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping successful for: 2024-03-21T10:32:12.183 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:32:11 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:32:11.797+0000 7f38adf74700 -1 osd.7 0 waiting for initial osdmap 2024-03-21T10:32:12.183 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:32:11 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:32:11.812+0000 7f38a5d5f700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T10:32:12.648 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:12 smithi070 ceph-mon[36330]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-03-21T10:32:12.648 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:12 smithi070 ceph-mon[36330]: osdmap e50: 8 total, 7 up, 8 in 2024-03-21T10:32:12.648 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:12 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T10:32:12.648 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:12 smithi070 ceph-mon[36330]: from='osd.7 ' entity='osd.7' 2024-03-21T10:32:12.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:12 smithi040 ceph-mon[36154]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-03-21T10:32:12.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:12 smithi040 ceph-mon[36154]: osdmap e50: 8 total, 7 up, 8 in 2024-03-21T10:32:12.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:12 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T10:32:12.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:12 smithi040 ceph-mon[36154]: from='osd.7 ' entity='osd.7' 2024-03-21T10:32:12.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:12 smithi031 ceph-mon[31673]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-03-21T10:32:12.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:12 smithi031 ceph-mon[31673]: osdmap e50: 8 total, 7 up, 8 in 2024-03-21T10:32:12.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:12 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T10:32:12.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:12 smithi031 ceph-mon[31673]: from='osd.7 ' entity='osd.7' 2024-03-21T10:32:12.841 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi070:vg_nvme/lv_2 2024-03-21T10:32:13.085 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.c/config 2024-03-21T10:32:13.460 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:13 smithi070 ceph-mon[36330]: purged_snaps scrub starts 2024-03-21T10:32:13.460 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:13 smithi070 ceph-mon[36330]: purged_snaps scrub ok 2024-03-21T10:32:13.460 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:13 smithi070 ceph-mon[36330]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:32:13.460 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:13 smithi070 ceph-mon[36330]: osd.7 [v2:172.21.15.40:6816/3825628119,v1:172.21.15.40:6817/3825628119] boot 2024-03-21T10:32:13.460 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:13 smithi070 ceph-mon[36330]: osdmap e51: 8 total, 8 up, 8 in 2024-03-21T10:32:13.460 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:13 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T10:32:13.460 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:13 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:13.460 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:13 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:13.461 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:13 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:32:13.461 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:13 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:32:13.461 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:13 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:32:13.461 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:13 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:13.461 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:13 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:13.461 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:13 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:32:13.461 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:13 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:13.461 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:13 smithi070 ceph-mon[36330]: osdmap e52: 8 total, 8 up, 8 in 2024-03-21T10:32:13.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:13 smithi040 ceph-mon[36154]: purged_snaps scrub starts 2024-03-21T10:32:13.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:13 smithi040 ceph-mon[36154]: purged_snaps scrub ok 2024-03-21T10:32:13.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:13 smithi040 ceph-mon[36154]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:32:13.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:13 smithi040 ceph-mon[36154]: osd.7 [v2:172.21.15.40:6816/3825628119,v1:172.21.15.40:6817/3825628119] boot 2024-03-21T10:32:13.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:13 smithi040 ceph-mon[36154]: osdmap e51: 8 total, 8 up, 8 in 2024-03-21T10:32:13.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:13 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T10:32:13.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:13 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:13.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:13 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:13.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:13 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:32:13.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:13 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:32:13.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:13 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:32:13.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:13 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:13.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:13 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:13.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:13 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:32:13.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:13 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:13.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:13 smithi040 ceph-mon[36154]: osdmap e52: 8 total, 8 up, 8 in 2024-03-21T10:32:13.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:13 smithi031 ceph-mon[31673]: purged_snaps scrub starts 2024-03-21T10:32:13.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:13 smithi031 ceph-mon[31673]: purged_snaps scrub ok 2024-03-21T10:32:13.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:13 smithi031 ceph-mon[31673]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-21T10:32:13.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:13 smithi031 ceph-mon[31673]: osd.7 [v2:172.21.15.40:6816/3825628119,v1:172.21.15.40:6817/3825628119] boot 2024-03-21T10:32:13.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:13 smithi031 ceph-mon[31673]: osdmap e51: 8 total, 8 up, 8 in 2024-03-21T10:32:13.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:13 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T10:32:13.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:13 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:13.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:13 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:13.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:13 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:32:13.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:13 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:32:13.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:13 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:32:13.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:13 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:13.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:13 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:13.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:13 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:32:13.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:13 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:13.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:13 smithi031 ceph-mon[31673]: osdmap e52: 8 total, 8 up, 8 in 2024-03-21T10:32:14.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:14 smithi070 ceph-mon[36330]: Detected new or changed devices on smithi040 2024-03-21T10:32:14.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:14 smithi070 ceph-mon[36330]: Adjusting osd_memory_target on smithi040 to 5694M 2024-03-21T10:32:14.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:14 smithi070 ceph-mon[36330]: osdmap e53: 8 total, 8 up, 8 in 2024-03-21T10:32:14.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:14 smithi040 ceph-mon[36154]: Detected new or changed devices on smithi040 2024-03-21T10:32:14.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:14 smithi040 ceph-mon[36154]: Adjusting osd_memory_target on smithi040 to 5694M 2024-03-21T10:32:14.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:14 smithi040 ceph-mon[36154]: osdmap e53: 8 total, 8 up, 8 in 2024-03-21T10:32:14.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:14 smithi031 ceph-mon[31673]: Detected new or changed devices on smithi040 2024-03-21T10:32:14.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:14 smithi031 ceph-mon[31673]: Adjusting osd_memory_target on smithi040 to 5694M 2024-03-21T10:32:14.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:14 smithi031 ceph-mon[31673]: osdmap e53: 8 total, 8 up, 8 in 2024-03-21T10:32:14.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.827+0000 7f5adf4c3700 1 -- 172.21.15.70:0/3941601297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ad8103d60 msgr2=0x7f5ad810a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:32:14.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.827+0000 7f5adf4c3700 1 --2- 172.21.15.70:0/3941601297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ad8103d60 0x7f5ad810a610 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5ad000b0a0 tx=0x7f5ad000b3b0 comp rx=0 tx=0).stop 2024-03-21T10:32:14.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.827+0000 7f5adf4c3700 1 -- 172.21.15.70:0/3941601297 shutdown_connections 2024-03-21T10:32:14.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.827+0000 7f5adf4c3700 1 --2- 172.21.15.70:0/3941601297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ad8103d60 0x7f5ad810a610 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:32:14.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.827+0000 7f5adf4c3700 1 --2- 172.21.15.70:0/3941601297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad81021c0 0x7f5ad81025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:32:14.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.827+0000 7f5adf4c3700 1 --2- 172.21.15.70:0/3941601297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ad81033c0 0x7f5ad8103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:32:14.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.827+0000 7f5adf4c3700 1 -- 172.21.15.70:0/3941601297 >> 172.21.15.70:0/3941601297 conn(0x7f5ad80fd7a0 msgr2=0x7f5ad80ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:32:14.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.827+0000 7f5adf4c3700 1 -- 172.21.15.70:0/3941601297 shutdown_connections 2024-03-21T10:32:14.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.827+0000 7f5adf4c3700 1 -- 172.21.15.70:0/3941601297 wait complete. 2024-03-21T10:32:14.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.827+0000 7f5adf4c3700 1 Processor -- start 2024-03-21T10:32:14.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.827+0000 7f5adf4c3700 1 -- start start 2024-03-21T10:32:14.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.827+0000 7f5adf4c3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ad81021c0 0x7f5ad81984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:32:14.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.828+0000 7f5adf4c3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad81033c0 0x7f5ad8198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:32:14.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.828+0000 7f5adf4c3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ad8103d60 0x7f5ad819fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:32:14.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.828+0000 7f5adf4c3700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad810dea0 con 0x7f5ad8103d60 2024-03-21T10:32:14.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.828+0000 7f5adf4c3700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad810dd20 con 0x7f5ad81033c0 2024-03-21T10:32:14.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.828+0000 7f5adf4c3700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad810e020 con 0x7f5ad81021c0 2024-03-21T10:32:14.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.828+0000 7f5add25f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ad81021c0 0x7f5ad81984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:32:14.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.828+0000 7f5add25f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ad81021c0 0x7f5ad81984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49924/0 (socket says 172.21.15.70:49924) 2024-03-21T10:32:14.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.828+0000 7f5add25f700 1 -- 172.21.15.70:0/1443940766 learned_addr learned my addr 172.21.15.70:0/1443940766 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-03-21T10:32:14.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.828+0000 7f5adca5e700 1 --2- 172.21.15.70:0/1443940766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad81033c0 0x7f5ad8198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:32:14.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.828+0000 7f5add25f700 1 -- 172.21.15.70:0/1443940766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad81033c0 msgr2=0x7f5ad8198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:32:14.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.828+0000 7f5add25f700 1 --2- 172.21.15.70:0/1443940766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad81033c0 0x7f5ad8198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:32:14.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.828+0000 7f5add25f700 1 -- 172.21.15.70:0/1443940766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ad8103d60 msgr2=0x7f5ad819fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:32:14.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.828+0000 7f5add25f700 1 --2- 172.21.15.70:0/1443940766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ad8103d60 0x7f5ad819fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:32:14.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.828+0000 7f5add25f700 1 -- 172.21.15.70:0/1443940766 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad81a0150 con 0x7f5ad81021c0 2024-03-21T10:32:14.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.828+0000 7f5adca5e700 1 --2- 172.21.15.70:0/1443940766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad81033c0 0x7f5ad8198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:32:14.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.828+0000 7f5add25f700 1 --2- 172.21.15.70:0/1443940766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ad81021c0 0x7f5ad81984f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5ad400b530 tx=0x7f5ad400b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:32:14.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.828+0000 7f5ace7fc700 1 -- 172.21.15.70:0/1443940766 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad4015020 con 0x7f5ad81021c0 2024-03-21T10:32:14.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.828+0000 7f5adf4c3700 1 -- 172.21.15.70:0/1443940766 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad81a0440 con 0x7f5ad81021c0 2024-03-21T10:32:14.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.828+0000 7f5ace7fc700 1 -- 172.21.15.70:0/1443940766 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ad4011d10 con 0x7f5ad81021c0 2024-03-21T10:32:14.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.828+0000 7f5ace7fc700 1 -- 172.21.15.70:0/1443940766 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad400f2e0 con 0x7f5ad81021c0 2024-03-21T10:32:14.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.828+0000 7f5adf4c3700 1 -- 172.21.15.70:0/1443940766 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ad81a09b0 con 0x7f5ad81021c0 2024-03-21T10:32:14.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.829+0000 7f5adf4c3700 1 -- 172.21.15.70:0/1443940766 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ad8068e50 con 0x7f5ad81021c0 2024-03-21T10:32:14.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.830+0000 7f5ace7fc700 1 -- 172.21.15.70:0/1443940766 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f5ad4018020 con 0x7f5ad81021c0 2024-03-21T10:32:14.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.831+0000 7f5ace7fc700 1 --2- 172.21.15.70:0/1443940766 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5ac40a2fb0 0x7f5ac40a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:32:14.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.831+0000 7f5ace7fc700 1 -- 172.21.15.70:0/1443940766 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5426+0+0 (secure 0 0 0) 0x7f5ad40ce670 con 0x7f5ad81021c0 2024-03-21T10:32:14.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.832+0000 7f5adca5e700 1 --2- 172.21.15.70:0/1443940766 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5ac40a2fb0 0x7f5ac40a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:32:14.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.832+0000 7f5adca5e700 1 --2- 172.21.15.70:0/1443940766 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5ac40a2fb0 0x7f5ac40a5470 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8007900 tx=0x7f5ac8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:32:14.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:14.835+0000 7f5ace7fc700 1 -- 172.21.15.70:0/1443940766 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5ad401f020 con 0x7f5ad81021c0 2024-03-21T10:32:15.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:15.040+0000 7f5adf4c3700 1 -- 172.21.15.70:0/1443940766 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ad80625d0 con 0x7f5ac40a2fb0 2024-03-21T10:32:15.668 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:15 smithi070 ceph-mon[36330]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:15.668 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:15 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:32:15.668 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:15 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:32:15.668 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:15 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:15.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:15 smithi040 ceph-mon[36154]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:15.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:15 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:32:15.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:15 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:32:15.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:15 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:15.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:15 smithi031 ceph-mon[31673]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:15.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:15 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:32:15.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:15 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:32:15.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:15 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:16.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:16 smithi040 ceph-mon[36154]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:32:16.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:16 smithi040 ceph-mon[36154]: osdmap e54: 8 total, 8 up, 8 in 2024-03-21T10:32:16.729 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:16 smithi070 ceph-mon[36330]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:32:16.729 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:16 smithi070 ceph-mon[36330]: osdmap e54: 8 total, 8 up, 8 in 2024-03-21T10:32:16.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:16 smithi031 ceph-mon[31673]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:32:16.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:16 smithi031 ceph-mon[31673]: osdmap e54: 8 total, 8 up, 8 in 2024-03-21T10:32:17.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:17 smithi040 ceph-mon[36154]: pgmap v205: 1 pgs: 1 remapped+peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:17.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:17 smithi040 ceph-mon[36154]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-21T10:32:17.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:17 smithi040 ceph-mon[36154]: osdmap e55: 8 total, 8 up, 8 in 2024-03-21T10:32:17.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:17 smithi031 ceph-mon[31673]: pgmap v205: 1 pgs: 1 remapped+peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:17.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:17 smithi031 ceph-mon[31673]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-21T10:32:17.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:17 smithi031 ceph-mon[31673]: osdmap e55: 8 total, 8 up, 8 in 2024-03-21T10:32:17.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:17 smithi070 ceph-mon[36330]: pgmap v205: 1 pgs: 1 remapped+peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:17.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:17 smithi070 ceph-mon[36330]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-21T10:32:17.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:17 smithi070 ceph-mon[36330]: osdmap e55: 8 total, 8 up, 8 in 2024-03-21T10:32:18.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:18 smithi070 ceph-mon[36330]: from='client.? 172.21.15.70:0/1833923998' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59b9fde0-095d-4e0b-87da-ead37bd8ff1b"}]: dispatch 2024-03-21T10:32:18.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:18 smithi070 ceph-mon[36330]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59b9fde0-095d-4e0b-87da-ead37bd8ff1b"}]: dispatch 2024-03-21T10:32:18.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:18 smithi070 ceph-mon[36330]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59b9fde0-095d-4e0b-87da-ead37bd8ff1b"}]': finished 2024-03-21T10:32:18.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:18 smithi070 ceph-mon[36330]: osdmap e56: 9 total, 8 up, 9 in 2024-03-21T10:32:18.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:18 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-21T10:32:18.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:18 smithi040 ceph-mon[36154]: from='client.? 172.21.15.70:0/1833923998' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59b9fde0-095d-4e0b-87da-ead37bd8ff1b"}]: dispatch 2024-03-21T10:32:18.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:18 smithi040 ceph-mon[36154]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59b9fde0-095d-4e0b-87da-ead37bd8ff1b"}]: dispatch 2024-03-21T10:32:18.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:18 smithi040 ceph-mon[36154]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59b9fde0-095d-4e0b-87da-ead37bd8ff1b"}]': finished 2024-03-21T10:32:18.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:18 smithi040 ceph-mon[36154]: osdmap e56: 9 total, 8 up, 9 in 2024-03-21T10:32:18.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:18 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-21T10:32:18.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:18 smithi031 ceph-mon[31673]: from='client.? 172.21.15.70:0/1833923998' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59b9fde0-095d-4e0b-87da-ead37bd8ff1b"}]: dispatch 2024-03-21T10:32:18.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:18 smithi031 ceph-mon[31673]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59b9fde0-095d-4e0b-87da-ead37bd8ff1b"}]: dispatch 2024-03-21T10:32:18.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:18 smithi031 ceph-mon[31673]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59b9fde0-095d-4e0b-87da-ead37bd8ff1b"}]': finished 2024-03-21T10:32:18.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:18 smithi031 ceph-mon[31673]: osdmap e56: 9 total, 8 up, 9 in 2024-03-21T10:32:18.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:18 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-21T10:32:19.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:19 smithi040 ceph-mon[36154]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:19.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:19 smithi040 ceph-mon[36154]: from='client.? 172.21.15.70:0/4062967981' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:32:19.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:19 smithi040 ceph-mon[36154]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-21T10:32:19.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:19 smithi040 ceph-mon[36154]: Cluster is now healthy 2024-03-21T10:32:19.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:19 smithi031 ceph-mon[31673]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:19.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:19 smithi031 ceph-mon[31673]: from='client.? 172.21.15.70:0/4062967981' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:32:19.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:19 smithi031 ceph-mon[31673]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-21T10:32:19.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:19 smithi031 ceph-mon[31673]: Cluster is now healthy 2024-03-21T10:32:19.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:19 smithi070 ceph-mon[36330]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:19.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:19 smithi070 ceph-mon[36330]: from='client.? 172.21.15.70:0/4062967981' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:32:19.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:19 smithi070 ceph-mon[36330]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-21T10:32:19.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:19 smithi070 ceph-mon[36330]: Cluster is now healthy 2024-03-21T10:32:21.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:21 smithi031 ceph-mon[31673]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:21.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:21 smithi070 ceph-mon[36330]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:21.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:21 smithi040 ceph-mon[36154]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:23.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:23 smithi031 ceph-mon[31673]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:23.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:23 smithi070 ceph-mon[36330]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:23.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:23 smithi040 ceph-mon[36154]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:25.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:25 smithi031 ceph-mon[31673]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:25.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:25 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-21T10:32:25.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:25 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:25.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:25 smithi070 ceph-mon[36330]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:25.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:25 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-21T10:32:25.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:25 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:25.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:25 smithi040 ceph-mon[36154]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:25.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:25 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-21T10:32:25.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:25 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:26.802 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:26 smithi070 ceph-mon[36330]: Deploying daemon osd.8 on smithi070 2024-03-21T10:32:26.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:26 smithi031 ceph-mon[31673]: Deploying daemon osd.8 on smithi070 2024-03-21T10:32:26.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:26 smithi040 ceph-mon[36154]: Deploying daemon osd.8 on smithi070 2024-03-21T10:32:27.707 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:27 smithi031 ceph-mon[31673]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:27.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:27 smithi070 ceph-mon[36330]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:27.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:27 smithi040 ceph-mon[36154]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:29.570 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:29 smithi070 ceph-mon[36330]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:29.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:29 smithi031 ceph-mon[31673]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:29.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:29 smithi040 ceph-mon[36154]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:32.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:31 smithi031 ceph-mon[31673]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:32.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:31 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:32.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:31 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:32.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:31 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:32:32.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:31 smithi070 ceph-mon[36330]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:32.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:31 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:32.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:31 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:32.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:31 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:32:32.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:31 smithi040 ceph-mon[36154]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:32.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:31 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:32.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:31 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:32.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:31 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:32:34.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:33 smithi031 ceph-mon[31673]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:34.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:33 smithi070 ceph-mon[36330]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:34.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:33 smithi040 ceph-mon[36154]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:34.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:34.226+0000 7f5ace7fc700 1 -- 172.21.15.70:0/1443940766 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5ad80625d0 con 0x7f5ac40a2fb0 2024-03-21T10:32:34.224 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 8 on host 'smithi070' 2024-03-21T10:32:34.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:34.228+0000 7f5adf4c3700 1 -- 172.21.15.70:0/1443940766 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5ac40a2fb0 msgr2=0x7f5ac40a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:32:34.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:34.228+0000 7f5adf4c3700 1 --2- 172.21.15.70:0/1443940766 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5ac40a2fb0 0x7f5ac40a5470 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8007900 tx=0x7f5ac8008040 comp rx=0 tx=0).stop 2024-03-21T10:32:34.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:34.228+0000 7f5adf4c3700 1 -- 172.21.15.70:0/1443940766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ad81021c0 msgr2=0x7f5ad81984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:32:34.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:34.228+0000 7f5adf4c3700 1 --2- 172.21.15.70:0/1443940766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ad81021c0 0x7f5ad81984f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5ad400b530 tx=0x7f5ad400b8f0 comp rx=0 tx=0).stop 2024-03-21T10:32:34.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:34.228+0000 7f5adf4c3700 1 -- 172.21.15.70:0/1443940766 shutdown_connections 2024-03-21T10:32:34.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:34.228+0000 7f5adf4c3700 1 --2- 172.21.15.70:0/1443940766 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5ac40a2fb0 0x7f5ac40a5470 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:32:34.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:34.228+0000 7f5adf4c3700 1 --2- 172.21.15.70:0/1443940766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ad8103d60 0x7f5ad819fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:32:34.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:34.228+0000 7f5adf4c3700 1 --2- 172.21.15.70:0/1443940766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ad81021c0 0x7f5ad81984f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:32:34.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:34.228+0000 7f5adf4c3700 1 --2- 172.21.15.70:0/1443940766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad81033c0 0x7f5ad8198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:32:34.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:34.228+0000 7f5adf4c3700 1 -- 172.21.15.70:0/1443940766 >> 172.21.15.70:0/1443940766 conn(0x7f5ad80fd7a0 msgr2=0x7f5ad81066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:32:34.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:34.228+0000 7f5adf4c3700 1 -- 172.21.15.70:0/1443940766 shutdown_connections 2024-03-21T10:32:34.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:32:34.228+0000 7f5adf4c3700 1 -- 172.21.15.70:0/1443940766 wait complete. 2024-03-21T10:32:35.148 DEBUG:teuthology.orchestra.run.smithi070:osd.8> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.8.service 2024-03-21T10:32:35.150 INFO:tasks.cephadm:Deploying osd.9 on smithi031 with /dev/vg_nvme/lv_1... 2024-03-21T10:32:35.150 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-21T10:32:35.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:35 smithi031 ceph-mon[31673]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:35.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:35 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:35.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:35 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:35.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:35 smithi031 ceph-mon[31673]: from='osd.8 [v2:172.21.15.70:6816/534880586,v1:172.21.15.70:6817/534880586]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-21T10:32:35.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:35 smithi031 ceph-mon[31673]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-21T10:32:35.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:35 smithi070 ceph-mon[36330]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:35.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:35 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:35.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:35 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:35.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:35 smithi070 ceph-mon[36330]: from='osd.8 [v2:172.21.15.70:6816/534880586,v1:172.21.15.70:6817/534880586]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-21T10:32:35.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:35 smithi070 ceph-mon[36330]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-21T10:32:35.629 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:32:35.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:35 smithi040 ceph-mon[36154]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:35.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:35 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:35.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:35 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:35.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:35 smithi040 ceph-mon[36154]: from='osd.8 [v2:172.21.15.70:6816/534880586,v1:172.21.15.70:6817/534880586]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-21T10:32:35.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:35 smithi040 ceph-mon[36154]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-21T10:32:36.292 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:36 smithi031 ceph-mon[31673]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-21T10:32:36.292 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:36 smithi031 ceph-mon[31673]: from='osd.8 [v2:172.21.15.70:6816/534880586,v1:172.21.15.70:6817/534880586]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-03-21T10:32:36.292 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:36 smithi031 ceph-mon[31673]: osdmap e57: 9 total, 8 up, 9 in 2024-03-21T10:32:36.292 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:36 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-21T10:32:36.292 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:36 smithi031 ceph-mon[31673]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-03-21T10:32:36.293 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:36 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:36.293 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:36 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:36.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:36 smithi070 ceph-mon[36330]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-21T10:32:36.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:36 smithi070 ceph-mon[36330]: from='osd.8 [v2:172.21.15.70:6816/534880586,v1:172.21.15.70:6817/534880586]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-03-21T10:32:36.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:36 smithi070 ceph-mon[36330]: osdmap e57: 9 total, 8 up, 9 in 2024-03-21T10:32:36.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:36 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-21T10:32:36.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:36 smithi070 ceph-mon[36330]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-03-21T10:32:36.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:36 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:36.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:36 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:36.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:36 smithi040 ceph-mon[36154]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-21T10:32:36.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:36 smithi040 ceph-mon[36154]: from='osd.8 [v2:172.21.15.70:6816/534880586,v1:172.21.15.70:6817/534880586]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-03-21T10:32:36.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:36 smithi040 ceph-mon[36154]: osdmap e57: 9 total, 8 up, 9 in 2024-03-21T10:32:36.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:36 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-21T10:32:36.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:36 smithi040 ceph-mon[36154]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-03-21T10:32:36.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:36 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:36.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:36 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:37.074 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:32:36 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:32:36.743+0000 7f45e30c5700 -1 osd.8 0 waiting for initial osdmap 2024-03-21T10:32:37.075 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:32:36 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:32:36.754+0000 7f45daeb0700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T10:32:37.407 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:37 smithi070 ceph-mon[36330]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:37.407 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:37 smithi070 ceph-mon[36330]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-03-21T10:32:37.407 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:37 smithi070 ceph-mon[36330]: osdmap e58: 9 total, 8 up, 9 in 2024-03-21T10:32:37.407 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:37 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-21T10:32:37.407 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:37 smithi070 ceph-mon[36330]: from='osd.8 ' entity='osd.8' 2024-03-21T10:32:37.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:37 smithi031 ceph-mon[31673]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:37.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:37 smithi031 ceph-mon[31673]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-03-21T10:32:37.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:37 smithi031 ceph-mon[31673]: osdmap e58: 9 total, 8 up, 9 in 2024-03-21T10:32:37.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:37 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-21T10:32:37.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:37 smithi031 ceph-mon[31673]: from='osd.8 ' entity='osd.8' 2024-03-21T10:32:37.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:37 smithi040 ceph-mon[36154]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-21T10:32:37.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:37 smithi040 ceph-mon[36154]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-03-21T10:32:37.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:37 smithi040 ceph-mon[36154]: osdmap e58: 9 total, 8 up, 9 in 2024-03-21T10:32:37.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:37 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-21T10:32:37.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:37 smithi040 ceph-mon[36154]: from='osd.8 ' entity='osd.8' 2024-03-21T10:32:38.293 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:38 smithi031 ceph-mon[31673]: purged_snaps scrub starts 2024-03-21T10:32:38.293 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:38 smithi031 ceph-mon[31673]: purged_snaps scrub ok 2024-03-21T10:32:38.293 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:38 smithi031 ceph-mon[31673]: osd.8 [v2:172.21.15.70:6816/534880586,v1:172.21.15.70:6817/534880586] boot 2024-03-21T10:32:38.293 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:38 smithi031 ceph-mon[31673]: osdmap e59: 9 total, 9 up, 9 in 2024-03-21T10:32:38.293 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:38 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-21T10:32:38.293 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:38 smithi031 ceph-mon[31673]: osdmap e60: 9 total, 9 up, 9 in 2024-03-21T10:32:38.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:38 smithi070 ceph-mon[36330]: purged_snaps scrub starts 2024-03-21T10:32:38.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:38 smithi070 ceph-mon[36330]: purged_snaps scrub ok 2024-03-21T10:32:38.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:38 smithi070 ceph-mon[36330]: osd.8 [v2:172.21.15.70:6816/534880586,v1:172.21.15.70:6817/534880586] boot 2024-03-21T10:32:38.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:38 smithi070 ceph-mon[36330]: osdmap e59: 9 total, 9 up, 9 in 2024-03-21T10:32:38.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:38 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-21T10:32:38.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:38 smithi070 ceph-mon[36330]: osdmap e60: 9 total, 9 up, 9 in 2024-03-21T10:32:38.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:38 smithi040 ceph-mon[36154]: purged_snaps scrub starts 2024-03-21T10:32:38.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:38 smithi040 ceph-mon[36154]: purged_snaps scrub ok 2024-03-21T10:32:38.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:38 smithi040 ceph-mon[36154]: osd.8 [v2:172.21.15.70:6816/534880586,v1:172.21.15.70:6817/534880586] boot 2024-03-21T10:32:38.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:38 smithi040 ceph-mon[36154]: osdmap e59: 9 total, 9 up, 9 in 2024-03-21T10:32:38.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:38 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-21T10:32:38.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:38 smithi040 ceph-mon[36154]: osdmap e60: 9 total, 9 up, 9 in 2024-03-21T10:32:39.098 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-21T10:32:39.098 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-21T10:32:39.098 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-03-21T10:32:39.098 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-03-21T10:32:39.098 INFO:teuthology.orchestra.run.smithi031.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0378024 s, 277 MB/s 2024-03-21T10:32:39.098 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-03-21T10:32:39.540 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:39 smithi031 ceph-mon[31673]: pgmap v221: 1 pgs: 1 remapped+peering; 577 KiB data, 299 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:32:39.540 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:39 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:39.540 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:39 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:39.540 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:39 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:32:39.540 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:39 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:32:39.540 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:39 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:32:39.540 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:39 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:39.540 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:39 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:39.540 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:39 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:32:39.540 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:39 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:39.541 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:39 smithi031 ceph-mon[31673]: osdmap e61: 9 total, 9 up, 9 in 2024-03-21T10:32:39.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:39 smithi070 ceph-mon[36330]: pgmap v221: 1 pgs: 1 remapped+peering; 577 KiB data, 299 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:32:39.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:39 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:39.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:39 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:39.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:39 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:32:39.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:39 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:32:39.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:39 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:32:39.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:39 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:39.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:39 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:39.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:39 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:32:39.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:39 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:39.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:39 smithi070 ceph-mon[36330]: osdmap e61: 9 total, 9 up, 9 in 2024-03-21T10:32:39.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:39 smithi040 ceph-mon[36154]: pgmap v221: 1 pgs: 1 remapped+peering; 577 KiB data, 299 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:32:39.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:39 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:39.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:39 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:39.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:39 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:32:39.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:39 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:32:39.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:39 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:32:39.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:39 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:39.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:39 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:39.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:39 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:32:39.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:39 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:39.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:39 smithi040 ceph-mon[36154]: osdmap e61: 9 total, 9 up, 9 in 2024-03-21T10:32:40.137 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi031:vg_nvme/lv_1 2024-03-21T10:32:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:40 smithi031 ceph-mon[31673]: Detected new or changed devices on smithi070 2024-03-21T10:32:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:40 smithi031 ceph-mon[31673]: Adjusting osd_memory_target on smithi070 to 5693M 2024-03-21T10:32:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:40 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:32:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:40 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:40 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:32:40.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:40 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:40.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:40 smithi070 ceph-mon[36330]: Detected new or changed devices on smithi070 2024-03-21T10:32:40.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:40 smithi070 ceph-mon[36330]: Adjusting osd_memory_target on smithi070 to 5693M 2024-03-21T10:32:40.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:40 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:32:40.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:40 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:40.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:40 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:32:40.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:40 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:40.606 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:32:40.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:40 smithi040 ceph-mon[36154]: Detected new or changed devices on smithi070 2024-03-21T10:32:40.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:40 smithi040 ceph-mon[36154]: Adjusting osd_memory_target on smithi070 to 5693M 2024-03-21T10:32:40.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:40 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:32:40.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:40 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:40.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:40 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:32:40.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:40 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:32:41.395 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:41 smithi031 ceph-mon[31673]: pgmap v224: 1 pgs: 1 remapped+peering; 577 KiB data, 299 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:32:41.395 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:41 smithi031 ceph-mon[31673]: osdmap e62: 9 total, 9 up, 9 in 2024-03-21T10:32:41.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:41 smithi070 ceph-mon[36330]: pgmap v224: 1 pgs: 1 remapped+peering; 577 KiB data, 299 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:32:41.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:41 smithi070 ceph-mon[36330]: osdmap e62: 9 total, 9 up, 9 in 2024-03-21T10:32:41.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:41 smithi040 ceph-mon[36154]: pgmap v224: 1 pgs: 1 remapped+peering; 577 KiB data, 299 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:32:41.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:41 smithi040 ceph-mon[36154]: osdmap e62: 9 total, 9 up, 9 in 2024-03-21T10:32:42.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:42 smithi031 ceph-mon[31673]: osdmap e63: 9 total, 9 up, 9 in 2024-03-21T10:32:42.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:42 smithi070 ceph-mon[36330]: osdmap e63: 9 total, 9 up, 9 in 2024-03-21T10:32:42.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:42 smithi040 ceph-mon[36154]: osdmap e63: 9 total, 9 up, 9 in 2024-03-21T10:32:43.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.361+0000 7fe59be00700 1 -- 172.21.15.31:0/2278668001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe594103410 msgr2=0x7fe594103870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:32:43.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.361+0000 7fe59be00700 1 --2- 172.21.15.31:0/2278668001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe594103410 0x7fe594103870 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe5800099c0 tx=0x7fe580009cd0 comp rx=0 tx=0).stop 2024-03-21T10:32:43.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.361+0000 7fe59be00700 1 -- 172.21.15.31:0/2278668001 shutdown_connections 2024-03-21T10:32:43.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.361+0000 7fe59be00700 1 --2- 172.21.15.31:0/2278668001 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe594103db0 0x7fe59410a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:32:43.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.361+0000 7fe59be00700 1 --2- 172.21.15.31:0/2278668001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe594102210 0x7fe594102630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:32:43.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.361+0000 7fe59be00700 1 --2- 172.21.15.31:0/2278668001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe594103410 0x7fe594103870 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:32:43.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.361+0000 7fe59be00700 1 -- 172.21.15.31:0/2278668001 >> 172.21.15.31:0/2278668001 conn(0x7fe5940fd7b0 msgr2=0x7fe5940ffbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:32:43.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.361+0000 7fe59be00700 1 -- 172.21.15.31:0/2278668001 shutdown_connections 2024-03-21T10:32:43.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.361+0000 7fe59be00700 1 -- 172.21.15.31:0/2278668001 wait complete. 2024-03-21T10:32:43.362 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:43 smithi031 ceph-mon[31673]: pgmap v227: 1 pgs: 1 remapped+peering; 577 KiB data, 299 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:32:43.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.362+0000 7fe59be00700 1 Processor -- start 2024-03-21T10:32:43.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.362+0000 7fe59be00700 1 -- start start 2024-03-21T10:32:43.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.363+0000 7fe59be00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe594102210 0x7fe59419a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:32:43.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.363+0000 7fe59be00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe594103410 0x7fe59419ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:32:43.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.363+0000 7fe59be00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe594103db0 0x7fe5941a1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:32:43.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.363+0000 7fe59be00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59410de30 con 0x7fe594103410 2024-03-21T10:32:43.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.363+0000 7fe59be00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59410dcb0 con 0x7fe594102210 2024-03-21T10:32:43.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.363+0000 7fe59be00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59410dfb0 con 0x7fe594103db0 2024-03-21T10:32:43.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.363+0000 7fe598a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe594103410 0x7fe59419ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:32:43.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.363+0000 7fe599400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe594102210 0x7fe59419a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:32:43.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.363+0000 7fe598a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe594103410 0x7fe59419ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53330/0 (socket says 172.21.15.31:53330) 2024-03-21T10:32:43.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.363+0000 7fe598a00700 1 -- 172.21.15.31:0/3778408903 learned_addr learned my addr 172.21.15.31:0/3778408903 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:32:43.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.363+0000 7fe599e00700 1 --2- 172.21.15.31:0/3778408903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe594103db0 0x7fe5941a1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:32:43.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.364+0000 7fe599400700 1 -- 172.21.15.31:0/3778408903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe594103db0 msgr2=0x7fe5941a1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:32:43.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.364+0000 7fe599400700 1 --2- 172.21.15.31:0/3778408903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe594103db0 0x7fe5941a1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:32:43.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.364+0000 7fe599400700 1 -- 172.21.15.31:0/3778408903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe594103410 msgr2=0x7fe59419ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:32:43.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.364+0000 7fe599400700 1 --2- 172.21.15.31:0/3778408903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe594103410 0x7fe59419ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:32:43.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.364+0000 7fe599400700 1 -- 172.21.15.31:0/3778408903 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5941a2290 con 0x7fe594102210 2024-03-21T10:32:43.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.364+0000 7fe599400700 1 --2- 172.21.15.31:0/3778408903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe594102210 0x7fe59419a630 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe58800ef90 tx=0x7fe58800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:32:43.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.364+0000 7fe592000700 1 -- 172.21.15.31:0/3778408903 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58800ca10 con 0x7fe594102210 2024-03-21T10:32:43.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.364+0000 7fe592000700 1 -- 172.21.15.31:0/3778408903 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe588019070 con 0x7fe594102210 2024-03-21T10:32:43.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.364+0000 7fe59be00700 1 -- 172.21.15.31:0/3778408903 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5941a2520 con 0x7fe594102210 2024-03-21T10:32:43.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.364+0000 7fe592000700 1 -- 172.21.15.31:0/3778408903 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5880146e0 con 0x7fe594102210 2024-03-21T10:32:43.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.364+0000 7fe59be00700 1 -- 172.21.15.31:0/3778408903 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5941a2ab0 con 0x7fe594102210 2024-03-21T10:32:43.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.365+0000 7fe59be00700 1 -- 172.21.15.31:0/3778408903 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe594068e50 con 0x7fe594102210 2024-03-21T10:32:43.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.367+0000 7fe592000700 1 -- 172.21.15.31:0/3778408903 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fe588012050 con 0x7fe594102210 2024-03-21T10:32:43.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.369+0000 7fe592000700 1 --2- 172.21.15.31:0/3778408903 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe57c0a3030 0x7fe57c0a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:32:43.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.369+0000 7fe592000700 1 -- 172.21.15.31:0/3778408903 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5829+0+0 (secure 0 0 0) 0x7fe5880ce560 con 0x7fe594102210 2024-03-21T10:32:43.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.369+0000 7fe598a00700 1 --2- 172.21.15.31:0/3778408903 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe57c0a3030 0x7fe57c0a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:32:43.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.370+0000 7fe598a00700 1 --2- 172.21.15.31:0/3778408903 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe57c0a3030 0x7fe57c0a54f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe580000c00 tx=0x7fe580005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:32:43.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.373+0000 7fe592000700 1 -- 172.21.15.31:0/3778408903 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe588010040 con 0x7fe594102210 2024-03-21T10:32:43.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:43 smithi070 ceph-mon[36330]: pgmap v227: 1 pgs: 1 remapped+peering; 577 KiB data, 299 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:32:43.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:32:43.611+0000 7fe59be00700 1 -- 172.21.15.31:0/3778408903 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe594075ae0 con 0x7fe57c0a3030 2024-03-21T10:32:43.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:43 smithi040 ceph-mon[36154]: pgmap v227: 1 pgs: 1 remapped+peering; 577 KiB data, 299 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:32:44.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:44 smithi031 ceph-mon[31673]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:32:44.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:44 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:32:44.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:44 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:32:44.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:44 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:44.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:44 smithi070 ceph-mon[36330]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:32:44.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:44 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:32:44.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:44 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:32:44.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:44 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:44.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:44 smithi040 ceph-mon[36154]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:32:44.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:44 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:32:44.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:44 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:32:44.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:44 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:45.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:45 smithi031 ceph-mon[31673]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-21T10:32:45.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:45 smithi070 ceph-mon[36330]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-21T10:32:45.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:45 smithi040 ceph-mon[36154]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-21T10:32:47.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:47 smithi031 ceph-mon[31673]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-21T10:32:47.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:47 smithi070 ceph-mon[36330]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-21T10:32:47.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:47 smithi040 ceph-mon[36154]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-21T10:32:48.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:48 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/962953432' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77f47061-0c5f-4159-b276-e814935d6c1c"}]: dispatch 2024-03-21T10:32:48.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:48 smithi031 ceph-mon[31673]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77f47061-0c5f-4159-b276-e814935d6c1c"}]: dispatch 2024-03-21T10:32:48.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:48 smithi031 ceph-mon[31673]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77f47061-0c5f-4159-b276-e814935d6c1c"}]': finished 2024-03-21T10:32:48.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:48 smithi031 ceph-mon[31673]: osdmap e64: 10 total, 9 up, 10 in 2024-03-21T10:32:48.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:48 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-21T10:32:48.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:48 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/962953432' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77f47061-0c5f-4159-b276-e814935d6c1c"}]: dispatch 2024-03-21T10:32:48.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:48 smithi070 ceph-mon[36330]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77f47061-0c5f-4159-b276-e814935d6c1c"}]: dispatch 2024-03-21T10:32:48.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:48 smithi070 ceph-mon[36330]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77f47061-0c5f-4159-b276-e814935d6c1c"}]': finished 2024-03-21T10:32:48.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:48 smithi070 ceph-mon[36330]: osdmap e64: 10 total, 9 up, 10 in 2024-03-21T10:32:48.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:48 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-21T10:32:48.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:48 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/962953432' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77f47061-0c5f-4159-b276-e814935d6c1c"}]: dispatch 2024-03-21T10:32:48.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:48 smithi040 ceph-mon[36154]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77f47061-0c5f-4159-b276-e814935d6c1c"}]: dispatch 2024-03-21T10:32:48.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:48 smithi040 ceph-mon[36154]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77f47061-0c5f-4159-b276-e814935d6c1c"}]': finished 2024-03-21T10:32:48.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:48 smithi040 ceph-mon[36154]: osdmap e64: 10 total, 9 up, 10 in 2024-03-21T10:32:48.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:48 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-21T10:32:49.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:49 smithi031 ceph-mon[31673]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-21T10:32:49.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:49 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/832674530' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:32:49.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:49 smithi070 ceph-mon[36330]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-21T10:32:49.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:49 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/832674530' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:32:49.583 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:49 smithi040 ceph-mon[36154]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-21T10:32:49.583 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:49 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/832674530' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:32:51.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:51 smithi031 ceph-mon[31673]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-21T10:32:51.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:51 smithi070 ceph-mon[36330]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-21T10:32:51.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:51 smithi040 ceph-mon[36154]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-21T10:32:53.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:53 smithi031 ceph-mon[31673]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T10:32:53.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:53 smithi070 ceph-mon[36330]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T10:32:53.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:53 smithi040 ceph-mon[36154]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T10:32:55.488 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:55 smithi031 ceph-mon[31673]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:32:55.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:55 smithi070 ceph-mon[36330]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:32:55.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:55 smithi040 ceph-mon[36154]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:32:57.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:57 smithi070 ceph-mon[36330]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:32:57.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:57 smithi040 ceph-mon[36154]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:32:57.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:57 smithi031 ceph-mon[31673]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:32:58.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:58 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-21T10:32:58.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:58 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:58.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:58 smithi070 ceph-mon[36330]: Deploying daemon osd.9 on smithi031 2024-03-21T10:32:58.636 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:58 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-21T10:32:58.636 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:58 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:58.636 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:58 smithi031 ceph-mon[31673]: Deploying daemon osd.9 on smithi031 2024-03-21T10:32:58.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:58 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-21T10:32:58.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:58 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:32:58.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:58 smithi040 ceph-mon[36154]: Deploying daemon osd.9 on smithi031 2024-03-21T10:32:59.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:32:59 smithi070 ceph-mon[36330]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:32:59.575 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:32:59 smithi031 ceph-mon[31673]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:32:59.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:32:59 smithi040 ceph-mon[36154]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:33:01.523 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:01 smithi031 ceph-mon[31673]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:33:01.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:01 smithi070 ceph-mon[36330]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:33:01.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:01 smithi040 ceph-mon[36154]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:33:03.606 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:03 smithi031 ceph-mon[31673]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:33:03.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:03 smithi040 ceph-mon[36154]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:33:03.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:03 smithi070 ceph-mon[36330]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:33:05.465 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:05 smithi031 ceph-mon[31673]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:33:05.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:05 smithi040 ceph-mon[36154]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:33:05.827 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:05 smithi070 ceph-mon[36330]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:33:07.908 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:07 smithi031 ceph-mon[31673]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:33:07.908 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:07 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:07.909 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:07 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:33:07.909 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:07 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:07.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:07 smithi040 ceph-mon[36154]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:33:07.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:07 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:07.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:07 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:33:07.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:07 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:08.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:07 smithi070 ceph-mon[36330]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:33:08.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:07 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:08.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:07 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:33:08.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:07 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:09.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:09 smithi031 ceph-mon[31673]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:33:09.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:09 smithi040 ceph-mon[36154]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:33:10.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:09 smithi070 ceph-mon[36330]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:33:11.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:11 smithi031 ceph-mon[31673]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:33:11.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:11 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:11.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:11 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:11.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:11 smithi031 ceph-mon[31673]: from='osd.9 [v2:172.21.15.31:6826/587047469,v1:172.21.15.31:6827/587047469]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-21T10:33:11.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:11 smithi040 ceph-mon[36154]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:33:11.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:11 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:11.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:11 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:11.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:11 smithi040 ceph-mon[36154]: from='osd.9 [v2:172.21.15.31:6826/587047469,v1:172.21.15.31:6827/587047469]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-21T10:33:12.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:33:12.015+0000 7fe592000700 1 -- 172.21.15.31:0/3778408903 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe594075ae0 con 0x7fe57c0a3030 2024-03-21T10:33:12.015 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 9 on host 'smithi031' 2024-03-21T10:33:12.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:33:12.019+0000 7fe59be00700 1 -- 172.21.15.31:0/3778408903 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe57c0a3030 msgr2=0x7fe57c0a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:33:12.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:33:12.019+0000 7fe59be00700 1 --2- 172.21.15.31:0/3778408903 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe57c0a3030 0x7fe57c0a54f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe580000c00 tx=0x7fe580005fd0 comp rx=0 tx=0).stop 2024-03-21T10:33:12.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:33:12.019+0000 7fe59be00700 1 -- 172.21.15.31:0/3778408903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe594102210 msgr2=0x7fe59419a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:33:12.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:33:12.019+0000 7fe59be00700 1 --2- 172.21.15.31:0/3778408903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe594102210 0x7fe59419a630 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe58800ef90 tx=0x7fe58800c3a0 comp rx=0 tx=0).stop 2024-03-21T10:33:12.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:33:12.019+0000 7fe59be00700 1 -- 172.21.15.31:0/3778408903 shutdown_connections 2024-03-21T10:33:12.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:33:12.019+0000 7fe59be00700 1 --2- 172.21.15.31:0/3778408903 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe57c0a3030 0x7fe57c0a54f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:33:12.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:33:12.019+0000 7fe59be00700 1 --2- 172.21.15.31:0/3778408903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe594103db0 0x7fe5941a1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:33:12.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:33:12.019+0000 7fe59be00700 1 --2- 172.21.15.31:0/3778408903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe594102210 0x7fe59419a630 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:33:12.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:33:12.019+0000 7fe59be00700 1 --2- 172.21.15.31:0/3778408903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe594103410 0x7fe59419ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:33:12.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:33:12.019+0000 7fe59be00700 1 -- 172.21.15.31:0/3778408903 >> 172.21.15.31:0/3778408903 conn(0x7fe5940fd7b0 msgr2=0x7fe5940743c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:33:12.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:33:12.020+0000 7fe59be00700 1 -- 172.21.15.31:0/3778408903 shutdown_connections 2024-03-21T10:33:12.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:33:12.020+0000 7fe59be00700 1 -- 172.21.15.31:0/3778408903 wait complete. 2024-03-21T10:33:12.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:11 smithi070 ceph-mon[36330]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:33:12.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:11 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:12.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:11 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:12.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:11 smithi070 ceph-mon[36330]: from='osd.9 [v2:172.21.15.31:6826/587047469,v1:172.21.15.31:6827/587047469]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-21T10:33:12.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:12 smithi031 ceph-mon[31673]: from='osd.9 [v2:172.21.15.31:6826/587047469,v1:172.21.15.31:6827/587047469]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-21T10:33:12.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:12 smithi031 ceph-mon[31673]: osdmap e65: 10 total, 9 up, 10 in 2024-03-21T10:33:12.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:12 smithi031 ceph-mon[31673]: from='osd.9 [v2:172.21.15.31:6826/587047469,v1:172.21.15.31:6827/587047469]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-21T10:33:12.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:12 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-21T10:33:12.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:12 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:33:12.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:12 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:33:12.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:12 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:12.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:12 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:12.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:12 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:12.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:12 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:33:12.913 DEBUG:teuthology.orchestra.run.smithi031:osd.9> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.9.service 2024-03-21T10:33:12.919 INFO:tasks.cephadm:Deploying osd.10 on smithi040 with /dev/vg_nvme/lv_1... 2024-03-21T10:33:12.919 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-21T10:33:12.953 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:12 smithi040 ceph-mon[36154]: from='osd.9 [v2:172.21.15.31:6826/587047469,v1:172.21.15.31:6827/587047469]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-21T10:33:12.953 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:12 smithi040 ceph-mon[36154]: osdmap e65: 10 total, 9 up, 10 in 2024-03-21T10:33:12.953 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:12 smithi040 ceph-mon[36154]: from='osd.9 [v2:172.21.15.31:6826/587047469,v1:172.21.15.31:6827/587047469]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-21T10:33:12.953 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:12 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-21T10:33:12.953 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:12 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:33:12.953 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:12 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:33:12.953 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:12 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:12.954 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:12 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:12.954 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:12 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:12.954 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:12 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:33:13.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:12 smithi070 ceph-mon[36330]: from='osd.9 [v2:172.21.15.31:6826/587047469,v1:172.21.15.31:6827/587047469]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-21T10:33:13.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:12 smithi070 ceph-mon[36330]: osdmap e65: 10 total, 9 up, 10 in 2024-03-21T10:33:13.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:12 smithi070 ceph-mon[36330]: from='osd.9 [v2:172.21.15.31:6826/587047469,v1:172.21.15.31:6827/587047469]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-21T10:33:13.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:12 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-21T10:33:13.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:12 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:33:13.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:12 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:33:13.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:12 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:13.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:12 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:13.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:12 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:13.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:12 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:33:13.178 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T10:33:13.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:13 smithi031 ceph-mon[31673]: purged_snaps scrub starts 2024-03-21T10:33:13.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:13 smithi031 ceph-mon[31673]: purged_snaps scrub ok 2024-03-21T10:33:13.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:13 smithi031 ceph-mon[31673]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:33:13.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:13 smithi031 ceph-mon[31673]: from='osd.9 [v2:172.21.15.31:6826/587047469,v1:172.21.15.31:6827/587047469]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-21T10:33:13.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:13 smithi031 ceph-mon[31673]: osdmap e66: 10 total, 9 up, 10 in 2024-03-21T10:33:13.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:13 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-21T10:33:13.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:13 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-21T10:33:13.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:13 smithi040 ceph-mon[36154]: purged_snaps scrub starts 2024-03-21T10:33:13.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:13 smithi040 ceph-mon[36154]: purged_snaps scrub ok 2024-03-21T10:33:13.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:13 smithi040 ceph-mon[36154]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:33:13.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:13 smithi040 ceph-mon[36154]: from='osd.9 [v2:172.21.15.31:6826/587047469,v1:172.21.15.31:6827/587047469]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-21T10:33:13.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:13 smithi040 ceph-mon[36154]: osdmap e66: 10 total, 9 up, 10 in 2024-03-21T10:33:13.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:13 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-21T10:33:13.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:13 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-21T10:33:14.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:13 smithi070 ceph-mon[36330]: purged_snaps scrub starts 2024-03-21T10:33:14.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:13 smithi070 ceph-mon[36330]: purged_snaps scrub ok 2024-03-21T10:33:14.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:13 smithi070 ceph-mon[36330]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:33:14.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:13 smithi070 ceph-mon[36330]: from='osd.9 [v2:172.21.15.31:6826/587047469,v1:172.21.15.31:6827/587047469]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-21T10:33:14.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:13 smithi070 ceph-mon[36330]: osdmap e66: 10 total, 9 up, 10 in 2024-03-21T10:33:14.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:13 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-21T10:33:14.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:13 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-21T10:33:14.998 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:14 smithi031 ceph-mon[31673]: osd.9 [v2:172.21.15.31:6826/587047469,v1:172.21.15.31:6827/587047469] boot 2024-03-21T10:33:14.999 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:14 smithi031 ceph-mon[31673]: osdmap e67: 10 total, 10 up, 10 in 2024-03-21T10:33:14.999 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:14 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-21T10:33:15.014 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:14 smithi040 ceph-mon[36154]: osd.9 [v2:172.21.15.31:6826/587047469,v1:172.21.15.31:6827/587047469] boot 2024-03-21T10:33:15.014 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:14 smithi040 ceph-mon[36154]: osdmap e67: 10 total, 10 up, 10 in 2024-03-21T10:33:15.014 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:14 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-21T10:33:15.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:14 smithi070 ceph-mon[36330]: osd.9 [v2:172.21.15.31:6826/587047469,v1:172.21.15.31:6827/587047469] boot 2024-03-21T10:33:15.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:14 smithi070 ceph-mon[36330]: osdmap e67: 10 total, 10 up, 10 in 2024-03-21T10:33:15.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:14 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-21T10:33:15.496 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-21T10:33:15.496 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-21T10:33:15.497 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-03-21T10:33:15.497 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-03-21T10:33:15.497 INFO:teuthology.orchestra.run.smithi040.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0126097 s, 832 MB/s 2024-03-21T10:33:15.497 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 2024-03-21T10:33:15.497 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-03-21T10:33:16.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:15 smithi031 ceph-mon[31673]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:33:16.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:15 smithi031 ceph-mon[31673]: osdmap e68: 10 total, 10 up, 10 in 2024-03-21T10:33:16.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:15 smithi070 ceph-mon[36330]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:33:16.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:15 smithi070 ceph-mon[36330]: osdmap e68: 10 total, 10 up, 10 in 2024-03-21T10:33:16.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:15 smithi040 ceph-mon[36154]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-21T10:33:16.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:15 smithi040 ceph-mon[36154]: osdmap e68: 10 total, 10 up, 10 in 2024-03-21T10:33:16.187 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi040:vg_nvme/lv_1 2024-03-21T10:33:16.489 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T10:33:17.850 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:17 smithi040 ceph-mon[36154]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:18.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:17 smithi031 ceph-mon[31673]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:18.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:17 smithi070 ceph-mon[36330]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:18.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.076+0000 7fbbb8fa2700 1 -- 172.21.15.40:0/2037083371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbb4103540 msgr2=0x7fbbb41039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:33:18.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.076+0000 7fbbb8fa2700 1 --2- 172.21.15.40:0/2037083371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbb4103540 0x7fbbb41039a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbba80099c0 tx=0x7fbba8009cd0 comp rx=0 tx=0).stop 2024-03-21T10:33:18.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.076+0000 7fbbb8fa2700 1 -- 172.21.15.40:0/2037083371 shutdown_connections 2024-03-21T10:33:18.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.076+0000 7fbbb8fa2700 1 --2- 172.21.15.40:0/2037083371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbb4103ee0 0x7fbbb410a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:33:18.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.076+0000 7fbbb8fa2700 1 --2- 172.21.15.40:0/2037083371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbb41004c0 0x7fbbb41008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:33:18.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.076+0000 7fbbb8fa2700 1 --2- 172.21.15.40:0/2037083371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbb4103540 0x7fbbb41039a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:33:18.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.076+0000 7fbbb8fa2700 1 -- 172.21.15.40:0/2037083371 >> 172.21.15.40:0/2037083371 conn(0x7fbbb40fbde0 msgr2=0x7fbbb40fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:33:18.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.076+0000 7fbbb8fa2700 1 -- 172.21.15.40:0/2037083371 shutdown_connections 2024-03-21T10:33:18.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.077+0000 7fbbb8fa2700 1 -- 172.21.15.40:0/2037083371 wait complete. 2024-03-21T10:33:18.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.077+0000 7fbbb8fa2700 1 Processor -- start 2024-03-21T10:33:18.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.077+0000 7fbbb8fa2700 1 -- start start 2024-03-21T10:33:18.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.077+0000 7fbbb8fa2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbb41004c0 0x7fbbb4198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:33:18.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.077+0000 7fbbb8fa2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbb4103540 0x7fbbb4198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:33:18.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.077+0000 7fbbb8fa2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbb4103ee0 0x7fbbb419fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:33:18.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.077+0000 7fbbb8fa2700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb410dec0 con 0x7fbbb4103ee0 2024-03-21T10:33:18.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.077+0000 7fbbb8fa2700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb410dd40 con 0x7fbbb41004c0 2024-03-21T10:33:18.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.077+0000 7fbbb8fa2700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb410e040 con 0x7fbbb4103540 2024-03-21T10:33:18.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.078+0000 7fbbb259c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbb41004c0 0x7fbbb4198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:33:18.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.078+0000 7fbbb259c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbb41004c0 0x7fbbb4198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42938/0 (socket says 172.21.15.40:42938) 2024-03-21T10:33:18.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.078+0000 7fbbb259c700 1 -- 172.21.15.40:0/701252595 learned_addr learned my addr 172.21.15.40:0/701252595 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-21T10:33:18.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.078+0000 7fbbb1d9b700 1 --2- 172.21.15.40:0/701252595 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbb4103540 0x7fbbb4198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:33:18.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.078+0000 7fbbb2d9d700 1 --2- 172.21.15.40:0/701252595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbb4103ee0 0x7fbbb419fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:33:18.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.078+0000 7fbbb259c700 1 -- 172.21.15.40:0/701252595 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbb4103540 msgr2=0x7fbbb4198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:33:18.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.078+0000 7fbbb259c700 1 --2- 172.21.15.40:0/701252595 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbb4103540 0x7fbbb4198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:33:18.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.078+0000 7fbbb259c700 1 -- 172.21.15.40:0/701252595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbb4103ee0 msgr2=0x7fbbb419fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:33:18.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.078+0000 7fbbb259c700 1 --2- 172.21.15.40:0/701252595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbb4103ee0 0x7fbbb419fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:33:18.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.078+0000 7fbbb259c700 1 -- 172.21.15.40:0/701252595 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb41a01a0 con 0x7fbbb41004c0 2024-03-21T10:33:18.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.078+0000 7fbbb259c700 1 --2- 172.21.15.40:0/701252595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbb41004c0 0x7fbbb4198590 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbba400b500 tx=0x7fbba400b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:33:18.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.078+0000 7fbba37fe700 1 -- 172.21.15.40:0/701252595 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba4015020 con 0x7fbbb41004c0 2024-03-21T10:33:18.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.078+0000 7fbbb8fa2700 1 -- 172.21.15.40:0/701252595 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb41a0490 con 0x7fbbb41004c0 2024-03-21T10:33:18.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.078+0000 7fbba37fe700 1 -- 172.21.15.40:0/701252595 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbba4011d10 con 0x7fbbb41004c0 2024-03-21T10:33:18.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.078+0000 7fbba37fe700 1 -- 172.21.15.40:0/701252595 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba400f2e0 con 0x7fbbb41004c0 2024-03-21T10:33:18.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.078+0000 7fbbb8fa2700 1 -- 172.21.15.40:0/701252595 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbb41a0a50 con 0x7fbbb41004c0 2024-03-21T10:33:18.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.079+0000 7fbbb8fa2700 1 -- 172.21.15.40:0/701252595 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbb4068e50 con 0x7fbbb41004c0 2024-03-21T10:33:18.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.081+0000 7fbba37fe700 1 -- 172.21.15.40:0/701252595 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fbba4018020 con 0x7fbbb41004c0 2024-03-21T10:33:18.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.082+0000 7fbba37fe700 1 --2- 172.21.15.40:0/701252595 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fbb9c0a2fb0 0x7fbb9c0a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:33:18.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.082+0000 7fbba37fe700 1 -- 172.21.15.40:0/701252595 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6261+0+0 (secure 0 0 0) 0x7fbba40ce890 con 0x7fbbb41004c0 2024-03-21T10:33:18.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.082+0000 7fbbb1d9b700 1 --2- 172.21.15.40:0/701252595 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fbb9c0a2fb0 0x7fbb9c0a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:33:18.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.083+0000 7fbbb1d9b700 1 --2- 172.21.15.40:0/701252595 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fbb9c0a2fb0 0x7fbb9c0a5470 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbba8000c00 tx=0x7fbba8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:33:18.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.086+0000 7fbba37fe700 1 -- 172.21.15.40:0/701252595 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbba401f020 con 0x7fbbb41004c0 2024-03-21T10:33:18.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:18.302+0000 7fbbb8fa2700 1 -- 172.21.15.40:0/701252595 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbb40625d0 con 0x7fbb9c0a2fb0 2024-03-21T10:33:19.209 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:18 smithi040 ceph-mon[36154]: Detected new or changed devices on smithi031 2024-03-21T10:33:19.209 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:18 smithi040 ceph-mon[36154]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:19.209 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:18 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:19.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:18 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:19.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:18 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:33:19.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:18 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:33:19.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:18 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:33:19.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:18 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:33:19.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:18 smithi040 ceph-mon[36154]: Adjusting osd_memory_target on smithi031 to 4273M 2024-03-21T10:33:19.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:18 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:19.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:18 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:33:19.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:18 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:33:19.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:18 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:19.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:18 smithi040 ceph-mon[36154]: from='client.24374 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:33:19.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:18 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:33:19.210 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:18 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:33:19.211 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:18 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:33:19.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:18 smithi031 ceph-mon[31673]: Detected new or changed devices on smithi031 2024-03-21T10:33:19.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:18 smithi031 ceph-mon[31673]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:19.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:18 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:19.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:18 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:19.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:18 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:33:19.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:18 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:33:19.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:18 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:33:19.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:18 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:33:19.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:18 smithi031 ceph-mon[31673]: Adjusting osd_memory_target on smithi031 to 4273M 2024-03-21T10:33:19.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:18 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:19.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:18 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:33:19.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:18 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:33:19.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:18 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:19.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:18 smithi031 ceph-mon[31673]: from='client.24374 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:33:19.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:18 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:33:19.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:18 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:33:19.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:18 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:33:19.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:18 smithi070 ceph-mon[36330]: Detected new or changed devices on smithi031 2024-03-21T10:33:19.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:18 smithi070 ceph-mon[36330]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:19.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:18 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:19.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:18 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:19.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:18 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:33:19.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:18 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:33:19.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:18 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:33:19.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:18 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:33:19.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:18 smithi070 ceph-mon[36330]: Adjusting osd_memory_target on smithi031 to 4273M 2024-03-21T10:33:19.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:18 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:19.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:18 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:33:19.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:18 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:33:19.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:18 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:19.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:18 smithi070 ceph-mon[36330]: from='client.24374 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:33:19.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:18 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:33:19.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:18 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:33:19.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:18 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:33:21.296 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:20 smithi031 ceph-mon[31673]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:21.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:20 smithi070 ceph-mon[36330]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:21.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:20 smithi040 ceph-mon[36154]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:22.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:21 smithi031 ceph-mon[31673]: from='client.? 172.21.15.40:0/1446854938' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8853dc43-7bd1-4c93-8063-f7982fd7da38"}]: dispatch 2024-03-21T10:33:22.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:21 smithi031 ceph-mon[31673]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8853dc43-7bd1-4c93-8063-f7982fd7da38"}]: dispatch 2024-03-21T10:33:22.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:21 smithi031 ceph-mon[31673]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8853dc43-7bd1-4c93-8063-f7982fd7da38"}]': finished 2024-03-21T10:33:22.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:21 smithi031 ceph-mon[31673]: osdmap e69: 11 total, 10 up, 11 in 2024-03-21T10:33:22.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:21 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-21T10:33:22.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:21 smithi031 ceph-mon[31673]: from='client.? 172.21.15.40:0/1214373145' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:33:22.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:21 smithi070 ceph-mon[36330]: from='client.? 172.21.15.40:0/1446854938' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8853dc43-7bd1-4c93-8063-f7982fd7da38"}]: dispatch 2024-03-21T10:33:22.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:21 smithi070 ceph-mon[36330]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8853dc43-7bd1-4c93-8063-f7982fd7da38"}]: dispatch 2024-03-21T10:33:22.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:21 smithi070 ceph-mon[36330]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8853dc43-7bd1-4c93-8063-f7982fd7da38"}]': finished 2024-03-21T10:33:22.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:21 smithi070 ceph-mon[36330]: osdmap e69: 11 total, 10 up, 11 in 2024-03-21T10:33:22.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:21 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-21T10:33:22.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:21 smithi070 ceph-mon[36330]: from='client.? 172.21.15.40:0/1214373145' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:33:22.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:21 smithi040 ceph-mon[36154]: from='client.? 172.21.15.40:0/1446854938' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8853dc43-7bd1-4c93-8063-f7982fd7da38"}]: dispatch 2024-03-21T10:33:22.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:21 smithi040 ceph-mon[36154]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8853dc43-7bd1-4c93-8063-f7982fd7da38"}]: dispatch 2024-03-21T10:33:22.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:21 smithi040 ceph-mon[36154]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8853dc43-7bd1-4c93-8063-f7982fd7da38"}]': finished 2024-03-21T10:33:22.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:21 smithi040 ceph-mon[36154]: osdmap e69: 11 total, 10 up, 11 in 2024-03-21T10:33:22.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:21 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-21T10:33:22.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:21 smithi040 ceph-mon[36154]: from='client.? 172.21.15.40:0/1214373145' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:33:23.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:23 smithi031 ceph-mon[31673]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:23.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:23 smithi070 ceph-mon[36330]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:23.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:23 smithi040 ceph-mon[36154]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:25.524 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:25 smithi040 ceph-mon[36154]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:25.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:25 smithi031 ceph-mon[31673]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:25.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:25 smithi070 ceph-mon[36330]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:27.549 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:27 smithi040 ceph-mon[36154]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:27.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:27 smithi031 ceph-mon[31673]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:27.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:27 smithi070 ceph-mon[36330]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:29.492 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:29 smithi040 ceph-mon[36154]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:29.492 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:29 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-21T10:33:29.492 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:29 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:33:29.492 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:29 smithi040 ceph-mon[36154]: Deploying daemon osd.10 on smithi040 2024-03-21T10:33:29.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:29 smithi031 ceph-mon[31673]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:29.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:29 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-21T10:33:29.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:29 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:33:29.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:29 smithi031 ceph-mon[31673]: Deploying daemon osd.10 on smithi040 2024-03-21T10:33:29.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:29 smithi070 ceph-mon[36330]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:29.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:29 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-21T10:33:29.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:29 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:33:29.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:29 smithi070 ceph-mon[36330]: Deploying daemon osd.10 on smithi040 2024-03-21T10:33:31.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:31 smithi040 ceph-mon[36154]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:31.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:31 smithi031 ceph-mon[31673]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:31.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:31 smithi070 ceph-mon[36330]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:33.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:33 smithi031 ceph-mon[31673]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:33.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:33 smithi070 ceph-mon[36330]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:33.674 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:33 smithi040 ceph-mon[36154]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:35.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:35 smithi040 ceph-mon[36154]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:35.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:35 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:35.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:35 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:33:35.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:35 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:36.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:35 smithi031 ceph-mon[31673]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:36.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:35 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:36.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:35 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:33:36.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:35 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:36.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:35 smithi070 ceph-mon[36330]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:36.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:35 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:36.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:35 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:33:36.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:35 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:37.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:37 smithi040 ceph-mon[36154]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:38.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:37 smithi031 ceph-mon[31673]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:38.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:37 smithi070 ceph-mon[36330]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:38.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:38.741+0000 7fbba37fe700 1 -- 172.21.15.40:0/701252595 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fbbb40625d0 con 0x7fbb9c0a2fb0 2024-03-21T10:33:38.740 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 10 on host 'smithi040' 2024-03-21T10:33:38.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:38.743+0000 7fbbb8fa2700 1 -- 172.21.15.40:0/701252595 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fbb9c0a2fb0 msgr2=0x7fbb9c0a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:33:38.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:38.743+0000 7fbbb8fa2700 1 --2- 172.21.15.40:0/701252595 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fbb9c0a2fb0 0x7fbb9c0a5470 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbba8000c00 tx=0x7fbba8005fb0 comp rx=0 tx=0).stop 2024-03-21T10:33:38.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:38.743+0000 7fbbb8fa2700 1 -- 172.21.15.40:0/701252595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbb41004c0 msgr2=0x7fbbb4198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:33:38.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:38.743+0000 7fbbb8fa2700 1 --2- 172.21.15.40:0/701252595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbb41004c0 0x7fbbb4198590 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbba400b500 tx=0x7fbba400b810 comp rx=0 tx=0).stop 2024-03-21T10:33:38.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:38.743+0000 7fbbb8fa2700 1 -- 172.21.15.40:0/701252595 shutdown_connections 2024-03-21T10:33:38.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:38.743+0000 7fbbb8fa2700 1 --2- 172.21.15.40:0/701252595 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fbb9c0a2fb0 0x7fbb9c0a5470 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:33:38.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:38.743+0000 7fbbb8fa2700 1 --2- 172.21.15.40:0/701252595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbb4103ee0 0x7fbbb419fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:33:38.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:38.743+0000 7fbbb8fa2700 1 --2- 172.21.15.40:0/701252595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbb41004c0 0x7fbbb4198590 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:33:38.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:38.743+0000 7fbbb8fa2700 1 --2- 172.21.15.40:0/701252595 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbb4103540 0x7fbbb4198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:33:38.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:38.743+0000 7fbbb8fa2700 1 -- 172.21.15.40:0/701252595 >> 172.21.15.40:0/701252595 conn(0x7fbbb40fbde0 msgr2=0x7fbbb4106910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:33:38.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:38.743+0000 7fbbb8fa2700 1 -- 172.21.15.40:0/701252595 shutdown_connections 2024-03-21T10:33:38.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-21T10:33:38.743+0000 7fbbb8fa2700 1 -- 172.21.15.40:0/701252595 wait complete. 2024-03-21T10:33:38.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:38 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:38.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:38 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:38.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:38 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:33:38.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:38 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:33:38.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:38 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:38.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:38 smithi040 ceph-mon[36154]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:38.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:38 smithi040 ceph-mon[36154]: from='osd.10 [v2:172.21.15.40:6824/512108711,v1:172.21.15.40:6825/512108711]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-21T10:33:38.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:38 smithi040 ceph-mon[36154]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-21T10:33:38.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:38 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:38.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:38 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:38.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:38 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:33:39.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:38 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:39.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:38 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:39.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:38 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:33:39.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:38 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:33:39.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:38 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:39.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:38 smithi031 ceph-mon[31673]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:39.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:38 smithi031 ceph-mon[31673]: from='osd.10 [v2:172.21.15.40:6824/512108711,v1:172.21.15.40:6825/512108711]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-21T10:33:39.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:38 smithi031 ceph-mon[31673]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-21T10:33:39.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:38 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:39.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:38 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:39.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:38 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:33:39.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:38 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:39.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:38 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:39.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:38 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:33:39.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:38 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:33:39.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:38 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:39.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:38 smithi070 ceph-mon[36330]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:39.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:38 smithi070 ceph-mon[36330]: from='osd.10 [v2:172.21.15.40:6824/512108711,v1:172.21.15.40:6825/512108711]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-21T10:33:39.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:38 smithi070 ceph-mon[36330]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-21T10:33:39.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:38 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:39.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:38 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:39.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:38 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:33:39.464 DEBUG:teuthology.orchestra.run.smithi040:osd.10> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.10.service 2024-03-21T10:33:39.466 INFO:tasks.cephadm:Deploying osd.11 on smithi070 with /dev/vg_nvme/lv_1... 2024-03-21T10:33:39.467 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-21T10:33:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.c/config 2024-03-21T10:33:40.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:39 smithi040 ceph-mon[36154]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-21T10:33:40.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:39 smithi040 ceph-mon[36154]: from='osd.10 [v2:172.21.15.40:6824/512108711,v1:172.21.15.40:6825/512108711]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-21T10:33:40.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:39 smithi040 ceph-mon[36154]: osdmap e70: 11 total, 10 up, 11 in 2024-03-21T10:33:40.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:39 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-21T10:33:40.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:39 smithi040 ceph-mon[36154]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-21T10:33:40.274 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:39 smithi070 ceph-mon[36330]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-21T10:33:40.274 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:39 smithi070 ceph-mon[36330]: from='osd.10 [v2:172.21.15.40:6824/512108711,v1:172.21.15.40:6825/512108711]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-21T10:33:40.274 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:39 smithi070 ceph-mon[36330]: osdmap e70: 11 total, 10 up, 11 in 2024-03-21T10:33:40.274 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:39 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-21T10:33:40.274 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:39 smithi070 ceph-mon[36330]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-21T10:33:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:39 smithi031 ceph-mon[31673]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-21T10:33:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:39 smithi031 ceph-mon[31673]: from='osd.10 [v2:172.21.15.40:6824/512108711,v1:172.21.15.40:6825/512108711]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-21T10:33:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:39 smithi031 ceph-mon[31673]: osdmap e70: 11 total, 10 up, 11 in 2024-03-21T10:33:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:39 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-21T10:33:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:39 smithi031 ceph-mon[31673]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-21T10:33:40.629 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:33:40 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:33:40.345+0000 7f43ecdfe700 -1 osd.10 0 waiting for initial osdmap 2024-03-21T10:33:40.630 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:33:40 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:33:40.361+0000 7f43e43e8700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T10:33:41.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:40 smithi040 ceph-mon[36154]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-03-21T10:33:41.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:40 smithi040 ceph-mon[36154]: osdmap e71: 11 total, 10 up, 11 in 2024-03-21T10:33:41.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:40 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-21T10:33:41.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:40 smithi040 ceph-mon[36154]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:41.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:40 smithi040 ceph-mon[36154]: from='osd.10 ' entity='osd.10' 2024-03-21T10:33:41.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:40 smithi040 ceph-mon[36154]: osd.10 [v2:172.21.15.40:6824/512108711,v1:172.21.15.40:6825/512108711] boot 2024-03-21T10:33:41.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:40 smithi040 ceph-mon[36154]: osdmap e72: 11 total, 11 up, 11 in 2024-03-21T10:33:41.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:40 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-21T10:33:41.186 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:40 smithi070 ceph-mon[36330]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-03-21T10:33:41.186 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:40 smithi070 ceph-mon[36330]: osdmap e71: 11 total, 10 up, 11 in 2024-03-21T10:33:41.186 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:40 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-21T10:33:41.186 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:40 smithi070 ceph-mon[36330]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:41.186 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:40 smithi070 ceph-mon[36330]: from='osd.10 ' entity='osd.10' 2024-03-21T10:33:41.186 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:40 smithi070 ceph-mon[36330]: osd.10 [v2:172.21.15.40:6824/512108711,v1:172.21.15.40:6825/512108711] boot 2024-03-21T10:33:41.186 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:40 smithi070 ceph-mon[36330]: osdmap e72: 11 total, 11 up, 11 in 2024-03-21T10:33:41.186 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:40 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-21T10:33:41.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:40 smithi031 ceph-mon[31673]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-03-21T10:33:41.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:40 smithi031 ceph-mon[31673]: osdmap e71: 11 total, 10 up, 11 in 2024-03-21T10:33:41.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:40 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-21T10:33:41.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:40 smithi031 ceph-mon[31673]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-21T10:33:41.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:40 smithi031 ceph-mon[31673]: from='osd.10 ' entity='osd.10' 2024-03-21T10:33:41.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:40 smithi031 ceph-mon[31673]: osd.10 [v2:172.21.15.40:6824/512108711,v1:172.21.15.40:6825/512108711] boot 2024-03-21T10:33:41.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:40 smithi031 ceph-mon[31673]: osdmap e72: 11 total, 11 up, 11 in 2024-03-21T10:33:41.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:40 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-21T10:33:41.887 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-21T10:33:41.888 INFO:teuthology.orchestra.run.smithi070.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-21T10:33:41.888 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10+0 records in 2024-03-21T10:33:41.888 INFO:teuthology.orchestra.run.smithi070.stderr:10+0 records out 2024-03-21T10:33:41.888 INFO:teuthology.orchestra.run.smithi070.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00846775 s, 1.2 GB/s 2024-03-21T10:33:41.888 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 2024-03-21T10:33:41.888 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping successful for: 2024-03-21T10:33:42.482 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi070:vg_nvme/lv_1 2024-03-21T10:33:42.724 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.c/config 2024-03-21T10:33:42.751 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:42 smithi070 ceph-mon[36330]: purged_snaps scrub starts 2024-03-21T10:33:42.751 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:42 smithi070 ceph-mon[36330]: purged_snaps scrub ok 2024-03-21T10:33:42.751 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:42 smithi070 ceph-mon[36330]: osdmap e73: 11 total, 11 up, 11 in 2024-03-21T10:33:42.780 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:42 smithi040 ceph-mon[36154]: purged_snaps scrub starts 2024-03-21T10:33:42.780 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:42 smithi040 ceph-mon[36154]: purged_snaps scrub ok 2024-03-21T10:33:42.780 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:42 smithi040 ceph-mon[36154]: osdmap e73: 11 total, 11 up, 11 in 2024-03-21T10:33:42.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:42 smithi031 ceph-mon[31673]: purged_snaps scrub starts 2024-03-21T10:33:42.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:42 smithi031 ceph-mon[31673]: purged_snaps scrub ok 2024-03-21T10:33:42.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:42 smithi031 ceph-mon[31673]: osdmap e73: 11 total, 11 up, 11 in 2024-03-21T10:33:43.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:43 smithi031 ceph-mon[31673]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:33:43.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:43 smithi031 ceph-mon[31673]: osdmap e74: 11 total, 11 up, 11 in 2024-03-21T10:33:43.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:43 smithi031 ceph-mon[31673]: Detected new or changed devices on smithi040 2024-03-21T10:33:43.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:43.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:43.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:33:43.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:33:43.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:33:43.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:33:43.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:43.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:33:43.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:33:43.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:43 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:43.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:43 smithi070 ceph-mon[36330]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:33:43.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:43 smithi070 ceph-mon[36330]: osdmap e74: 11 total, 11 up, 11 in 2024-03-21T10:33:43.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:43 smithi070 ceph-mon[36330]: Detected new or changed devices on smithi040 2024-03-21T10:33:43.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:43 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:43.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:43 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:43.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:43 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:33:43.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:43 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:33:43.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:43 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:33:43.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:43 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:33:43.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:43 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:43.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:43 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:33:43.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:43 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:33:43.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:43 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:43.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:43 smithi040 ceph-mon[36154]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:33:43.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:43 smithi040 ceph-mon[36154]: osdmap e74: 11 total, 11 up, 11 in 2024-03-21T10:33:43.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:43 smithi040 ceph-mon[36154]: Detected new or changed devices on smithi040 2024-03-21T10:33:43.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:43 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:43.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:43 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:43.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:43 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:33:43.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:43 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:33:43.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:43 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:33:43.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:43 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:33:43.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:43 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:43.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:43 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:33:43.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:43 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:33:43.933 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:43 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:33:44.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.428+0000 7f2c4c099700 1 -- 172.21.15.70:0/2530796873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c441010c0 msgr2=0x7f2c441014e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:33:44.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.428+0000 7f2c4c099700 1 --2- 172.21.15.70:0/2530796873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c441010c0 0x7f2c441014e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2c34009990 tx=0x7f2c34009ca0 comp rx=0 tx=0).stop 2024-03-21T10:33:44.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.428+0000 7f2c4c099700 1 -- 172.21.15.70:0/2530796873 shutdown_connections 2024-03-21T10:33:44.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.428+0000 7f2c4c099700 1 --2- 172.21.15.70:0/2530796873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c44102c60 0x7f2c44109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:33:44.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.428+0000 7f2c4c099700 1 --2- 172.21.15.70:0/2530796873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c441010c0 0x7f2c441014e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:33:44.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.428+0000 7f2c4c099700 1 --2- 172.21.15.70:0/2530796873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c441022c0 0x7f2c44102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:33:44.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.428+0000 7f2c4c099700 1 -- 172.21.15.70:0/2530796873 >> 172.21.15.70:0/2530796873 conn(0x7f2c440fc680 msgr2=0x7f2c440feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:33:44.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.429+0000 7f2c4c099700 1 -- 172.21.15.70:0/2530796873 shutdown_connections 2024-03-21T10:33:44.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.429+0000 7f2c4c099700 1 -- 172.21.15.70:0/2530796873 wait complete. 2024-03-21T10:33:44.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.429+0000 7f2c4c099700 1 Processor -- start 2024-03-21T10:33:44.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.429+0000 7f2c4c099700 1 -- start start 2024-03-21T10:33:44.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.429+0000 7f2c4c099700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c441010c0 0x7f2c44197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:33:44.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.429+0000 7f2c4c099700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c441022c0 0x7f2c441979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:33:44.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.429+0000 7f2c4c099700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c44102c60 0x7f2c4419ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:33:44.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.429+0000 7f2c4c099700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4410cdd0 con 0x7f2c441010c0 2024-03-21T10:33:44.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.429+0000 7f2c4c099700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4410cc50 con 0x7f2c441022c0 2024-03-21T10:33:44.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.429+0000 7f2c4c099700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4410cf50 con 0x7f2c44102c60 2024-03-21T10:33:44.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.429+0000 7f2c4a636700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c44102c60 0x7f2c4419ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:33:44.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.429+0000 7f2c4a636700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c44102c60 0x7f2c4419ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57924/0 (socket says 172.21.15.70:57924) 2024-03-21T10:33:44.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.429+0000 7f2c4a636700 1 -- 172.21.15.70:0/2265322684 learned_addr learned my addr 172.21.15.70:0/2265322684 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-03-21T10:33:44.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.430+0000 7f2c49634700 1 --2- 172.21.15.70:0/2265322684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c441022c0 0x7f2c441979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:33:44.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.430+0000 7f2c4a636700 1 -- 172.21.15.70:0/2265322684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c441022c0 msgr2=0x7f2c441979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:33:44.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.430+0000 7f2c4a636700 1 --2- 172.21.15.70:0/2265322684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c441022c0 0x7f2c441979c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:33:44.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.430+0000 7f2c4a636700 1 -- 172.21.15.70:0/2265322684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c441010c0 msgr2=0x7f2c44197480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:33:44.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.430+0000 7f2c4a636700 1 --2- 172.21.15.70:0/2265322684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c441010c0 0x7f2c44197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:33:44.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.430+0000 7f2c4a636700 1 -- 172.21.15.70:0/2265322684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c4419f0e0 con 0x7f2c44102c60 2024-03-21T10:33:44.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.430+0000 7f2c4a636700 1 --2- 172.21.15.70:0/2265322684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c44102c60 0x7f2c4419ea40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c00b9a0 tx=0x7f2c3c00bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:33:44.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.430+0000 7f2c3affd700 1 -- 172.21.15.70:0/2265322684 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c019070 con 0x7f2c44102c60 2024-03-21T10:33:44.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.430+0000 7f2c3affd700 1 -- 172.21.15.70:0/2265322684 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c3c014440 con 0x7f2c44102c60 2024-03-21T10:33:44.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.430+0000 7f2c4c099700 1 -- 172.21.15.70:0/2265322684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c4419f3d0 con 0x7f2c44102c60 2024-03-21T10:33:44.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.430+0000 7f2c3affd700 1 -- 172.21.15.70:0/2265322684 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c0075d0 con 0x7f2c44102c60 2024-03-21T10:33:44.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.430+0000 7f2c4c099700 1 -- 172.21.15.70:0/2265322684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c4419f990 con 0x7f2c44102c60 2024-03-21T10:33:44.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.431+0000 7f2c4c099700 1 -- 172.21.15.70:0/2265322684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c44068e50 con 0x7f2c44102c60 2024-03-21T10:33:44.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.433+0000 7f2c3affd700 1 -- 172.21.15.70:0/2265322684 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f2c3c01c070 con 0x7f2c44102c60 2024-03-21T10:33:44.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.433+0000 7f2c3affd700 1 --2- 172.21.15.70:0/2265322684 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2c300a3030 0x7f2c300a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:33:44.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.434+0000 7f2c3affd700 1 -- 172.21.15.70:0/2265322684 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6694+0+0 (secure 0 0 0) 0x7f2c3c0ce630 con 0x7f2c44102c60 2024-03-21T10:33:44.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.434+0000 7f2c49e35700 1 --2- 172.21.15.70:0/2265322684 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2c300a3030 0x7f2c300a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:33:44.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.435+0000 7f2c49e35700 1 --2- 172.21.15.70:0/2265322684 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2c300a3030 0x7f2c300a54f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2c34003890 tx=0x7f2c34005c20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:33:44.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.437+0000 7f2c3affd700 1 -- 172.21.15.70:0/2265322684 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2c3c09bb40 con 0x7f2c44102c60 2024-03-21T10:33:44.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:33:44.653+0000 7f2c4c099700 1 -- 172.21.15.70:0/2265322684 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c440625d0 con 0x7f2c300a3030 2024-03-21T10:33:44.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:44 smithi031 ceph-mon[31673]: Adjusting osd_memory_target on smithi040 to 4271M 2024-03-21T10:33:44.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:44 smithi031 ceph-mon[31673]: osdmap e75: 11 total, 11 up, 11 in 2024-03-21T10:33:44.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:44 smithi070 ceph-mon[36330]: Adjusting osd_memory_target on smithi040 to 4271M 2024-03-21T10:33:44.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:44 smithi070 ceph-mon[36330]: osdmap e75: 11 total, 11 up, 11 in 2024-03-21T10:33:44.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:44 smithi040 ceph-mon[36154]: Adjusting osd_memory_target on smithi040 to 4271M 2024-03-21T10:33:44.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:44 smithi040 ceph-mon[36154]: osdmap e75: 11 total, 11 up, 11 in 2024-03-21T10:33:45.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:45 smithi031 ceph-mon[31673]: pgmap v270: 1 pgs: 1 remapped+peering; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:33:45.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:45 smithi031 ceph-mon[31673]: osdmap e76: 11 total, 11 up, 11 in 2024-03-21T10:33:45.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:45 smithi031 ceph-mon[31673]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:33:45.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:45 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:33:45.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:45 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:33:45.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:45 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:33:45.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:45 smithi070 ceph-mon[36330]: pgmap v270: 1 pgs: 1 remapped+peering; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:33:45.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:45 smithi070 ceph-mon[36330]: osdmap e76: 11 total, 11 up, 11 in 2024-03-21T10:33:45.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:45 smithi070 ceph-mon[36330]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:33:45.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:45 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:33:45.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:45 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:33:45.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:45 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:33:45.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:45 smithi040 ceph-mon[36154]: pgmap v270: 1 pgs: 1 remapped+peering; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:33:45.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:45 smithi040 ceph-mon[36154]: osdmap e76: 11 total, 11 up, 11 in 2024-03-21T10:33:45.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:45 smithi040 ceph-mon[36154]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:33:45.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:45 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T10:33:45.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:45 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T10:33:45.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:45 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:33:47.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:47 smithi031 ceph-mon[31673]: pgmap v272: 1 pgs: 1 remapped+peering; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:33:47.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:47 smithi031 ceph-mon[31673]: from='client.? 172.21.15.70:0/2259242676' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "84b7c2de-c424-45a6-9583-60097e372a86"}]: dispatch 2024-03-21T10:33:47.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:47 smithi031 ceph-mon[31673]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "84b7c2de-c424-45a6-9583-60097e372a86"}]: dispatch 2024-03-21T10:33:47.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:47 smithi031 ceph-mon[31673]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "84b7c2de-c424-45a6-9583-60097e372a86"}]': finished 2024-03-21T10:33:47.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:47 smithi031 ceph-mon[31673]: osdmap e77: 12 total, 11 up, 12 in 2024-03-21T10:33:47.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:47 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-21T10:33:47.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:47 smithi070 ceph-mon[36330]: pgmap v272: 1 pgs: 1 remapped+peering; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:33:47.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:47 smithi070 ceph-mon[36330]: from='client.? 172.21.15.70:0/2259242676' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "84b7c2de-c424-45a6-9583-60097e372a86"}]: dispatch 2024-03-21T10:33:47.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:47 smithi070 ceph-mon[36330]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "84b7c2de-c424-45a6-9583-60097e372a86"}]: dispatch 2024-03-21T10:33:47.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:47 smithi070 ceph-mon[36330]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "84b7c2de-c424-45a6-9583-60097e372a86"}]': finished 2024-03-21T10:33:47.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:47 smithi070 ceph-mon[36330]: osdmap e77: 12 total, 11 up, 12 in 2024-03-21T10:33:47.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:47 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-21T10:33:47.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:47 smithi040 ceph-mon[36154]: pgmap v272: 1 pgs: 1 remapped+peering; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:33:47.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:47 smithi040 ceph-mon[36154]: from='client.? 172.21.15.70:0/2259242676' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "84b7c2de-c424-45a6-9583-60097e372a86"}]: dispatch 2024-03-21T10:33:47.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:47 smithi040 ceph-mon[36154]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "84b7c2de-c424-45a6-9583-60097e372a86"}]: dispatch 2024-03-21T10:33:47.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:47 smithi040 ceph-mon[36154]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "84b7c2de-c424-45a6-9583-60097e372a86"}]': finished 2024-03-21T10:33:47.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:47 smithi040 ceph-mon[36154]: osdmap e77: 12 total, 11 up, 12 in 2024-03-21T10:33:47.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:47 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-21T10:33:48.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:48 smithi031 ceph-mon[31673]: from='client.? 172.21.15.70:0/3413838160' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:33:48.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:48 smithi070 ceph-mon[36330]: from='client.? 172.21.15.70:0/3413838160' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:33:48.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:48 smithi040 ceph-mon[36154]: from='client.? 172.21.15.70:0/3413838160' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T10:33:49.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:49 smithi070 ceph-mon[36330]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-03-21T10:33:49.833 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:49 smithi040 ceph-mon[36154]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-03-21T10:33:50.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:49 smithi031 ceph-mon[31673]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-03-21T10:33:51.767 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:51 smithi070 ceph-mon[36330]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-21T10:33:51.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:51 smithi040 ceph-mon[36154]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-21T10:33:52.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:51 smithi031 ceph-mon[31673]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-21T10:33:53.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:53 smithi070 ceph-mon[36330]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-21T10:33:53.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:53 smithi040 ceph-mon[36154]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-21T10:33:54.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:53 smithi031 ceph-mon[31673]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-21T10:33:55.694 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:55 smithi070 ceph-mon[36330]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-21T10:33:55.694 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:55 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-21T10:33:55.695 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:55 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:33:55.695 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:55 smithi070 ceph-mon[36330]: Deploying daemon osd.11 on smithi070 2024-03-21T10:33:55.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:55 smithi040 ceph-mon[36154]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-21T10:33:55.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:55 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-21T10:33:55.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:55 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:33:55.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:55 smithi040 ceph-mon[36154]: Deploying daemon osd.11 on smithi070 2024-03-21T10:33:56.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:55 smithi031 ceph-mon[31673]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-21T10:33:56.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:55 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-21T10:33:56.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:55 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:33:56.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:55 smithi031 ceph-mon[31673]: Deploying daemon osd.11 on smithi070 2024-03-21T10:33:57.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:57 smithi070 ceph-mon[36330]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T10:33:57.858 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:57 smithi031 ceph-mon[31673]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T10:33:57.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:57 smithi040 ceph-mon[36154]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-21T10:33:59.674 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:33:59 smithi070 ceph-mon[36330]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:33:59.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:33:59 smithi031 ceph-mon[31673]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:33:59.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:33:59 smithi040 ceph-mon[36154]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:34:01.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:01 smithi070 ceph-mon[36330]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:34:01.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:01 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:01.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:01 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:34:01.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:01 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:01.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:01 smithi040 ceph-mon[36154]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:34:01.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:01 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:01.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:01 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:34:01.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:01 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:02.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:01 smithi031 ceph-mon[31673]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:34:02.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:01 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:02.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:01 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:34:02.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:01 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:03.905 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:03 smithi070 ceph-mon[36330]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:34:03.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:03 smithi040 ceph-mon[36154]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:34:04.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:03 smithi031 ceph-mon[31673]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:34:04.250 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 11 on host 'smithi070' 2024-03-21T10:34:04.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:04.251+0000 7f2c3affd700 1 -- 172.21.15.70:0/2265322684 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f2c440625d0 con 0x7f2c300a3030 2024-03-21T10:34:04.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:04.254+0000 7f2c4c099700 1 -- 172.21.15.70:0/2265322684 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2c300a3030 msgr2=0x7f2c300a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:04.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:04.254+0000 7f2c4c099700 1 --2- 172.21.15.70:0/2265322684 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2c300a3030 0x7f2c300a54f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2c34003890 tx=0x7f2c34005c20 comp rx=0 tx=0).stop 2024-03-21T10:34:04.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:04.254+0000 7f2c4c099700 1 -- 172.21.15.70:0/2265322684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c44102c60 msgr2=0x7f2c4419ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:04.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:04.254+0000 7f2c4c099700 1 --2- 172.21.15.70:0/2265322684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c44102c60 0x7f2c4419ea40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c00b9a0 tx=0x7f2c3c00bd60 comp rx=0 tx=0).stop 2024-03-21T10:34:04.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:04.254+0000 7f2c4c099700 1 -- 172.21.15.70:0/2265322684 shutdown_connections 2024-03-21T10:34:04.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:04.254+0000 7f2c4c099700 1 --2- 172.21.15.70:0/2265322684 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2c300a3030 0x7f2c300a54f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:04.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:04.254+0000 7f2c4c099700 1 --2- 172.21.15.70:0/2265322684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c44102c60 0x7f2c4419ea40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:04.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:04.254+0000 7f2c4c099700 1 --2- 172.21.15.70:0/2265322684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c441010c0 0x7f2c44197480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:04.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:04.254+0000 7f2c4c099700 1 --2- 172.21.15.70:0/2265322684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c441022c0 0x7f2c441979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:04.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:04.254+0000 7f2c4c099700 1 -- 172.21.15.70:0/2265322684 >> 172.21.15.70:0/2265322684 conn(0x7f2c440fc680 msgr2=0x7f2c441055e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:04.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:04.254+0000 7f2c4c099700 1 -- 172.21.15.70:0/2265322684 shutdown_connections 2024-03-21T10:34:04.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:04.254+0000 7f2c4c099700 1 -- 172.21.15.70:0/2265322684 wait complete. 2024-03-21T10:34:04.913 DEBUG:teuthology.orchestra.run.smithi070:osd.11> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.11.service 2024-03-21T10:34:04.916 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-21T10:34:04.916 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-21T10:34:05.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:05 smithi031 ceph-mon[31673]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:34:05.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:05 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:05.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:05 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:05.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:05 smithi031 ceph-mon[31673]: from='osd.11 [v2:172.21.15.70:6824/540305718,v1:172.21.15.70:6825/540305718]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-21T10:34:05.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:05 smithi031 ceph-mon[31673]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-21T10:34:05.387 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:34:05.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:05 smithi070 ceph-mon[36330]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:34:05.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:05 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:05.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:05 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:05.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:05 smithi070 ceph-mon[36330]: from='osd.11 [v2:172.21.15.70:6824/540305718,v1:172.21.15.70:6825/540305718]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-21T10:34:05.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:05 smithi070 ceph-mon[36330]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-21T10:34:05.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:05 smithi040 ceph-mon[36154]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:34:05.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:05 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:05.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:05 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:05.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:05 smithi040 ceph-mon[36154]: from='osd.11 [v2:172.21.15.70:6824/540305718,v1:172.21.15.70:6825/540305718]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-21T10:34:05.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:05 smithi040 ceph-mon[36154]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-21T10:34:06.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:06 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:06.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:06 smithi031 ceph-mon[31673]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-21T10:34:06.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:06 smithi031 ceph-mon[31673]: from='osd.11 [v2:172.21.15.70:6824/540305718,v1:172.21.15.70:6825/540305718]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-03-21T10:34:06.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:06 smithi031 ceph-mon[31673]: osdmap e78: 12 total, 11 up, 12 in 2024-03-21T10:34:06.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:06 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-21T10:34:06.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:06 smithi031 ceph-mon[31673]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-03-21T10:34:06.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:06 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:06.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:06 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:06.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:06 smithi070 ceph-mon[36330]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-21T10:34:06.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:06 smithi070 ceph-mon[36330]: from='osd.11 [v2:172.21.15.70:6824/540305718,v1:172.21.15.70:6825/540305718]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-03-21T10:34:06.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:06 smithi070 ceph-mon[36330]: osdmap e78: 12 total, 11 up, 12 in 2024-03-21T10:34:06.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:06 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-21T10:34:06.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:06 smithi070 ceph-mon[36330]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-03-21T10:34:06.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:06 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:06.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:06 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:06.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:06 smithi040 ceph-mon[36154]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-21T10:34:06.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:06 smithi040 ceph-mon[36154]: from='osd.11 [v2:172.21.15.70:6824/540305718,v1:172.21.15.70:6825/540305718]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-03-21T10:34:06.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:06 smithi040 ceph-mon[36154]: osdmap e78: 12 total, 11 up, 12 in 2024-03-21T10:34:06.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:06 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-21T10:34:06.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:06 smithi040 ceph-mon[36154]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-03-21T10:34:06.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:06 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:06.882 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:34:06 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:34:06.752+0000 7f115fb2c700 -1 osd.11 0 waiting for initial osdmap 2024-03-21T10:34:06.883 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:34:06 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:34:06.766+0000 7f1157917700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T10:34:07.386 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:07 smithi031 ceph-mon[31673]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:34:07.386 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:07 smithi031 ceph-mon[31673]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-03-21T10:34:07.386 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:07 smithi031 ceph-mon[31673]: osdmap e79: 12 total, 11 up, 12 in 2024-03-21T10:34:07.386 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:07 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-21T10:34:07.387 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:07 smithi031 ceph-mon[31673]: from='osd.11 ' entity='osd.11' 2024-03-21T10:34:07.519 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:07 smithi070 ceph-mon[36330]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:34:07.519 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:07 smithi070 ceph-mon[36330]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-03-21T10:34:07.519 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:07 smithi070 ceph-mon[36330]: osdmap e79: 12 total, 11 up, 12 in 2024-03-21T10:34:07.520 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:07 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-21T10:34:07.520 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:07 smithi070 ceph-mon[36330]: from='osd.11 ' entity='osd.11' 2024-03-21T10:34:07.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:07 smithi040 ceph-mon[36154]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:34:07.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:07 smithi040 ceph-mon[36154]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-03-21T10:34:07.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:07 smithi040 ceph-mon[36154]: osdmap e79: 12 total, 11 up, 12 in 2024-03-21T10:34:07.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:07 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-21T10:34:07.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:07 smithi040 ceph-mon[36154]: from='osd.11 ' entity='osd.11' 2024-03-21T10:34:07.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.965+0000 7f207fc00700 1 -- 172.21.15.31:0/2821237246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20781055c0 msgr2=0x7f20781079b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:07.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.965+0000 7f207fc00700 1 --2- 172.21.15.31:0/2821237246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20781055c0 0x7f20781079b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2064009990 tx=0x7f2064009ca0 comp rx=0 tx=0).stop 2024-03-21T10:34:07.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.965+0000 7f207fc00700 1 -- 172.21.15.31:0/2821237246 shutdown_connections 2024-03-21T10:34:07.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.965+0000 7f207fc00700 1 --2- 172.21.15.31:0/2821237246 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2078107ef0 0x7f207810a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:07.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.965+0000 7f207fc00700 1 --2- 172.21.15.31:0/2821237246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2078102c90 0x7f2078105080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:07.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.965+0000 7f207fc00700 1 --2- 172.21.15.31:0/2821237246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20781055c0 0x7f20781079b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:07.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.965+0000 7f207fc00700 1 -- 172.21.15.31:0/2821237246 >> 172.21.15.31:0/2821237246 conn(0x7f20780fc660 msgr2=0x7f20780feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:07.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.966+0000 7f207fc00700 1 -- 172.21.15.31:0/2821237246 shutdown_connections 2024-03-21T10:34:07.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.966+0000 7f207fc00700 1 -- 172.21.15.31:0/2821237246 wait complete. 2024-03-21T10:34:07.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.967+0000 7f207fc00700 1 Processor -- start 2024-03-21T10:34:07.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.967+0000 7f207fc00700 1 -- start start 2024-03-21T10:34:07.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.967+0000 7f207fc00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2078102c90 0x7f207819a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:07.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.967+0000 7f207fc00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20781055c0 0x7f207819abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:07.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.967+0000 7f207fc00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2078107ef0 0x7f20781a1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:07.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.968+0000 7f207fc00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f207810ddf0 con 0x7f2078107ef0 2024-03-21T10:34:07.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.968+0000 7f207fc00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f207810dc70 con 0x7f2078102c90 2024-03-21T10:34:07.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.968+0000 7f207fc00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f207810df70 con 0x7f20781055c0 2024-03-21T10:34:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.968+0000 7f2077e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20781055c0 0x7f207819abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.968+0000 7f2077e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20781055c0 0x7f207819abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:35740/0 (socket says 172.21.15.31:35740) 2024-03-21T10:34:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.968+0000 7f2077e00700 1 -- 172.21.15.31:0/2641484192 learned_addr learned my addr 172.21.15.31:0/2641484192 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:34:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.968+0000 7f207d200700 1 --2- 172.21.15.31:0/2641484192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2078102c90 0x7f207819a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.968+0000 7f207dc00700 1 --2- 172.21.15.31:0/2641484192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2078107ef0 0x7f20781a1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.968+0000 7f207d200700 1 -- 172.21.15.31:0/2641484192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20781055c0 msgr2=0x7f207819abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.968+0000 7f207d200700 1 --2- 172.21.15.31:0/2641484192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20781055c0 0x7f207819abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.968+0000 7f207d200700 1 -- 172.21.15.31:0/2641484192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2078107ef0 msgr2=0x7f20781a1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.968+0000 7f207d200700 1 --2- 172.21.15.31:0/2641484192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2078107ef0 0x7f20781a1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:07.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.968+0000 7f207d200700 1 -- 172.21.15.31:0/2641484192 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20781a2310 con 0x7f2078102c90 2024-03-21T10:34:07.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.968+0000 7f2077e00700 1 --2- 172.21.15.31:0/2641484192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20781055c0 0x7f207819abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:34:07.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.968+0000 7f207dc00700 1 --2- 172.21.15.31:0/2641484192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2078107ef0 0x7f20781a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:34:07.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.969+0000 7f207d200700 1 --2- 172.21.15.31:0/2641484192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2078102c90 0x7f207819a6b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f206c00e9f0 tx=0x7f206c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:07.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.969+0000 7f2075600700 1 -- 172.21.15.31:0/2641484192 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f206c00cbd0 con 0x7f2078102c90 2024-03-21T10:34:07.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.969+0000 7f2075600700 1 -- 172.21.15.31:0/2641484192 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f206c00cd70 con 0x7f2078102c90 2024-03-21T10:34:07.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.969+0000 7f207fc00700 1 -- 172.21.15.31:0/2641484192 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20781a2600 con 0x7f2078102c90 2024-03-21T10:34:07.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.969+0000 7f2075600700 1 -- 172.21.15.31:0/2641484192 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f206c0187a0 con 0x7f2078102c90 2024-03-21T10:34:07.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.969+0000 7f207fc00700 1 -- 172.21.15.31:0/2641484192 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20781a2b90 con 0x7f2078102c90 2024-03-21T10:34:07.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.970+0000 7f207fc00700 1 -- 172.21.15.31:0/2641484192 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2078068e50 con 0x7f2078102c90 2024-03-21T10:34:07.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.972+0000 7f2075600700 1 -- 172.21.15.31:0/2641484192 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f206c0109a0 con 0x7f2078102c90 2024-03-21T10:34:07.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.974+0000 7f2075600700 1 --2- 172.21.15.31:0/2641484192 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f20600a3030 0x7f20600a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:07.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.974+0000 7f2075600700 1 -- 172.21.15.31:0/2641484192 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f206c0cf3e0 con 0x7f2078102c90 2024-03-21T10:34:07.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.974+0000 7f2077e00700 1 --2- 172.21.15.31:0/2641484192 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f20600a3030 0x7f20600a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:07.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.975+0000 7f2077e00700 1 --2- 172.21.15.31:0/2641484192 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f20600a3030 0x7f20600a54f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f206400b7b0 tx=0x7f206400b880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:07.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:07.976+0000 7f2075600700 1 -- 172.21.15.31:0/2641484192 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f206c09c610 con 0x7f2078102c90 2024-03-21T10:34:08.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:08.202+0000 7f207fc00700 1 -- 172.21.15.31:0/2641484192 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f2078062da0 con 0x7f2078102c90 2024-03-21T10:34:08.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:08.203+0000 7f2075600700 1 -- 172.21.15.31:0/2641484192 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) v1 ==== 74+0+133 (secure 0 0 0) 0x7f206c09bf10 con 0x7f2078102c90 2024-03-21T10:34:08.202 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:34:08.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:08.207+0000 7f207fc00700 1 -- 172.21.15.31:0/2641484192 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f20600a3030 msgr2=0x7f20600a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:08.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:08.207+0000 7f207fc00700 1 --2- 172.21.15.31:0/2641484192 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f20600a3030 0x7f20600a54f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f206400b7b0 tx=0x7f206400b880 comp rx=0 tx=0).stop 2024-03-21T10:34:08.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:08.207+0000 7f207fc00700 1 -- 172.21.15.31:0/2641484192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2078102c90 msgr2=0x7f207819a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:08.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:08.207+0000 7f207fc00700 1 --2- 172.21.15.31:0/2641484192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2078102c90 0x7f207819a6b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f206c00e9f0 tx=0x7f206c00ed00 comp rx=0 tx=0).stop 2024-03-21T10:34:08.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:08.207+0000 7f207fc00700 1 -- 172.21.15.31:0/2641484192 shutdown_connections 2024-03-21T10:34:08.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:08.207+0000 7f207fc00700 1 --2- 172.21.15.31:0/2641484192 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f20600a3030 0x7f20600a54f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:08.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:08.207+0000 7f207fc00700 1 --2- 172.21.15.31:0/2641484192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2078107ef0 0x7f20781a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:08.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:08.207+0000 7f207fc00700 1 --2- 172.21.15.31:0/2641484192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2078102c90 0x7f207819a6b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:08.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:08.207+0000 7f207fc00700 1 --2- 172.21.15.31:0/2641484192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20781055c0 0x7f207819abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:08.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:08.207+0000 7f207fc00700 1 -- 172.21.15.31:0/2641484192 >> 172.21.15.31:0/2641484192 conn(0x7f20780fc660 msgr2=0x7f2078102530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:08.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:08.208+0000 7f207fc00700 1 -- 172.21.15.31:0/2641484192 shutdown_connections 2024-03-21T10:34:08.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:08.208+0000 7f207fc00700 1 -- 172.21.15.31:0/2641484192 wait complete. 2024-03-21T10:34:08.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:08 smithi031 ceph-mon[31673]: purged_snaps scrub starts 2024-03-21T10:34:08.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:08 smithi031 ceph-mon[31673]: purged_snaps scrub ok 2024-03-21T10:34:08.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:08 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-21T10:34:08.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:08 smithi031 ceph-mon[31673]: osd.11 [v2:172.21.15.70:6824/540305718,v1:172.21.15.70:6825/540305718] boot 2024-03-21T10:34:08.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:08 smithi031 ceph-mon[31673]: osdmap e80: 12 total, 12 up, 12 in 2024-03-21T10:34:08.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:08 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-21T10:34:08.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:08 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2641484192' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-21T10:34:08.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:08 smithi070 ceph-mon[36330]: purged_snaps scrub starts 2024-03-21T10:34:08.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:08 smithi070 ceph-mon[36330]: purged_snaps scrub ok 2024-03-21T10:34:08.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:08 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-21T10:34:08.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:08 smithi070 ceph-mon[36330]: osd.11 [v2:172.21.15.70:6824/540305718,v1:172.21.15.70:6825/540305718] boot 2024-03-21T10:34:08.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:08 smithi070 ceph-mon[36330]: osdmap e80: 12 total, 12 up, 12 in 2024-03-21T10:34:08.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:08 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-21T10:34:08.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:08 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2641484192' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-21T10:34:08.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:08 smithi040 ceph-mon[36154]: purged_snaps scrub starts 2024-03-21T10:34:08.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:08 smithi040 ceph-mon[36154]: purged_snaps scrub ok 2024-03-21T10:34:08.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:08 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-21T10:34:08.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:08 smithi040 ceph-mon[36154]: osd.11 [v2:172.21.15.70:6824/540305718,v1:172.21.15.70:6825/540305718] boot 2024-03-21T10:34:08.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:08 smithi040 ceph-mon[36154]: osdmap e80: 12 total, 12 up, 12 in 2024-03-21T10:34:08.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:08 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-21T10:34:08.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:08 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2641484192' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-21T10:34:08.946 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1711017247,"num_in_osds":12,"osd_in_since":1711017227,"num_remapped_pgs":0} 2024-03-21T10:34:08.947 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-21T10:34:09.403 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:34:09.824 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:09 smithi031 ceph-mon[31673]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:34:09.824 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:09 smithi031 ceph-mon[31673]: osdmap e81: 12 total, 12 up, 12 in 2024-03-21T10:34:09.824 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:09.824 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:09.824 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:34:09.824 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:34:09.824 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:34:09.824 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:34:09.824 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:09.824 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:09.824 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:34:09.824 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:09.825 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:34:09.825 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:09.825 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:34:09.825 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:10.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:09 smithi070 ceph-mon[36330]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:34:10.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:09 smithi070 ceph-mon[36330]: osdmap e81: 12 total, 12 up, 12 in 2024-03-21T10:34:10.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:10.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:10.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:34:10.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:34:10.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:34:10.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:34:10.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:10.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:10.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:34:10.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:10.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:34:10.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:10.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:34:10.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:10.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:09 smithi040 ceph-mon[36154]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-21T10:34:10.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:09 smithi040 ceph-mon[36154]: osdmap e81: 12 total, 12 up, 12 in 2024-03-21T10:34:10.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:10.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:10.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:34:10.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:34:10.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:34:10.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:34:10.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:10.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:10.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:34:10.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:10.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:34:10.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:10.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:34:10.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:11.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:10 smithi031 ceph-mon[31673]: Detected new or changed devices on smithi070 2024-03-21T10:34:11.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:10 smithi031 ceph-mon[31673]: Adjusting osd_memory_target on smithi070 to 4270M 2024-03-21T10:34:11.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:10 smithi070 ceph-mon[36330]: Detected new or changed devices on smithi070 2024-03-21T10:34:11.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:10 smithi070 ceph-mon[36330]: Adjusting osd_memory_target on smithi070 to 4270M 2024-03-21T10:34:11.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:10 smithi040 ceph-mon[36154]: Detected new or changed devices on smithi070 2024-03-21T10:34:11.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:10 smithi040 ceph-mon[36154]: Adjusting osd_memory_target on smithi070 to 4270M 2024-03-21T10:34:11.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.905+0000 7f650ca00700 1 -- 172.21.15.31:0/1433744429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6508104f90 msgr2=0x7f6508108fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:11.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.905+0000 7f650ca00700 1 --2- 172.21.15.31:0/1433744429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6508104f90 0x7f6508108fe0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f64f4009990 tx=0x7f64f4009ca0 comp rx=0 tx=0).stop 2024-03-21T10:34:11.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.905+0000 7f650ca00700 1 -- 172.21.15.31:0/1433744429 shutdown_connections 2024-03-21T10:34:11.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.905+0000 7f650ca00700 1 --2- 172.21.15.31:0/1433744429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65081096c0 0x7f6508111290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:11.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.905+0000 7f650ca00700 1 --2- 172.21.15.31:0/1433744429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65081045e0 0x7f65081049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:11.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.905+0000 7f650ca00700 1 --2- 172.21.15.31:0/1433744429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6508104f90 0x7f6508108fe0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:11.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.905+0000 7f650ca00700 1 -- 172.21.15.31:0/1433744429 >> 172.21.15.31:0/1433744429 conn(0x7f65080ffe70 msgr2=0x7f6508102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:11.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.905+0000 7f650ca00700 1 -- 172.21.15.31:0/1433744429 shutdown_connections 2024-03-21T10:34:11.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.905+0000 7f650ca00700 1 -- 172.21.15.31:0/1433744429 wait complete. 2024-03-21T10:34:11.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.906+0000 7f650ca00700 1 Processor -- start 2024-03-21T10:34:11.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.907+0000 7f650ca00700 1 -- start start 2024-03-21T10:34:11.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.907+0000 7f650ca00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65081045e0 0x7f65081a07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:11.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.907+0000 7f650ca00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6508104f90 0x7f65081a0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:11.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.907+0000 7f650ca00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65081096c0 0x7f650819a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:11.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.907+0000 7f650ca00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65081146d0 con 0x7f65081045e0 2024-03-21T10:34:11.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.907+0000 7f650ca00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6508114550 con 0x7f6508104f90 2024-03-21T10:34:11.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.907+0000 7f650ca00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6508114850 con 0x7f65081096c0 2024-03-21T10:34:11.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.907+0000 7f6505e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65081045e0 0x7f65081a07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:11.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.908+0000 7f6505e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65081045e0 0x7f65081a07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43116/0 (socket says 172.21.15.31:43116) 2024-03-21T10:34:11.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.908+0000 7f6505e00700 1 -- 172.21.15.31:0/3524889183 learned_addr learned my addr 172.21.15.31:0/3524889183 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:34:11.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.908+0000 7f6506800700 1 --2- 172.21.15.31:0/3524889183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65081096c0 0x7f650819a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:11.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.908+0000 7f6505400700 1 --2- 172.21.15.31:0/3524889183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6508104f90 0x7f65081a0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:11.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.908+0000 7f6505e00700 1 -- 172.21.15.31:0/3524889183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65081096c0 msgr2=0x7f650819a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:11.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.908+0000 7f6505e00700 1 --2- 172.21.15.31:0/3524889183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65081096c0 0x7f650819a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:11.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.908+0000 7f6505e00700 1 -- 172.21.15.31:0/3524889183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6508104f90 msgr2=0x7f65081a0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:11.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.908+0000 7f6505e00700 1 --2- 172.21.15.31:0/3524889183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6508104f90 0x7f65081a0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:11.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.908+0000 7f6505e00700 1 -- 172.21.15.31:0/3524889183 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f650819b110 con 0x7f65081045e0 2024-03-21T10:34:11.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.908+0000 7f6505400700 1 --2- 172.21.15.31:0/3524889183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6508104f90 0x7f65081a0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:34:11.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.908+0000 7f6505e00700 1 --2- 172.21.15.31:0/3524889183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65081045e0 0x7f65081a07a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f64f800e9a0 tx=0x7f64f800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:11.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.909+0000 7f64fea00700 1 -- 172.21.15.31:0/3524889183 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f800ca10 con 0x7f65081045e0 2024-03-21T10:34:11.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.909+0000 7f650ca00700 1 -- 172.21.15.31:0/3524889183 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f650819b400 con 0x7f65081045e0 2024-03-21T10:34:11.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.909+0000 7f64fea00700 1 -- 172.21.15.31:0/3524889183 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64f800cbb0 con 0x7f65081045e0 2024-03-21T10:34:11.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.909+0000 7f64fea00700 1 -- 172.21.15.31:0/3524889183 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f8018680 con 0x7f65081045e0 2024-03-21T10:34:11.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.909+0000 7f650ca00700 1 -- 172.21.15.31:0/3524889183 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65081a7670 con 0x7f65081045e0 2024-03-21T10:34:11.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.910+0000 7f650ca00700 1 -- 172.21.15.31:0/3524889183 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6508068e50 con 0x7f65081045e0 2024-03-21T10:34:11.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.912+0000 7f64fea00700 1 -- 172.21.15.31:0/3524889183 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f64f8010430 con 0x7f65081045e0 2024-03-21T10:34:11.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.914+0000 7f64fea00700 1 --2- 172.21.15.31:0/3524889183 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f64ec0a3030 0x7f64ec0a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:11.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.914+0000 7f64fea00700 1 -- 172.21.15.31:0/3524889183 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f64f80cf590 con 0x7f65081045e0 2024-03-21T10:34:11.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.914+0000 7f6505400700 1 --2- 172.21.15.31:0/3524889183 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f64ec0a3030 0x7f64ec0a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:11.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.915+0000 7f6505400700 1 --2- 172.21.15.31:0/3524889183 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f64ec0a3030 0x7f64ec0a54f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f65081035f0 tx=0x7f64f400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:11.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:11.919+0000 7f64fea00700 1 -- 172.21.15.31:0/3524889183 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f64f809c860 con 0x7f65081045e0 2024-03-21T10:34:12.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:11 smithi031 ceph-mon[31673]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:12.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:11 smithi070 ceph-mon[36330]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:12.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:12.153+0000 7f650ca00700 1 -- 172.21.15.31:0/3524889183 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f650802da30 con 0x7f65081045e0 2024-03-21T10:34:12.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:12.154+0000 7f64fea00700 1 -- 172.21.15.31:0/3524889183 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18337 (secure 0 0 0) 0x7f64f809c160 con 0x7f65081045e0 2024-03-21T10:34:12.153 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:34:12.154 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":81,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","created":"2024-03-21T10:25:41.971117+0000","modified":"2024-03-21T10:34:09.260664+0000","last_up_change":"2024-03-21T10:34:07.757196+0000","last_in_change":"2024-03-21T10:33:47.460406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T10:29:44.553968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0244d33-00e2-47d5-9f66-34efc138c462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6803","nonce":3089885125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6805","nonce":3089885125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6809","nonce":3089885125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6807","nonce":3089885125}]},"public_addr":"172.21.15.31:6803/3089885125","cluster_addr":"172.21.15.31:6805/3089885125","heartbeat_back_addr":"172.21.15.31:6809/3089885125","heartbeat_front_addr":"172.21.15.31:6807/3089885125","state":["exists","up"]},{"osd":1,"uuid":"76a63504-beab-4354-a6c3-eebc3b16ad03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6801","nonce":618840037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6803","nonce":618840037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6807","nonce":618840037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6805","nonce":618840037}]},"public_addr":"172.21.15.40:6801/618840037","cluster_addr":"172.21.15.40:6803/618840037","heartbeat_back_addr":"172.21.15.40:6807/618840037","heartbeat_front_addr":"172.21.15.40:6805/618840037","state":["exists","up"]},{"osd":2,"uuid":"fa191bd6-9a21-4ee2-8adf-8a63c9cfd821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6801","nonce":2524184117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6803","nonce":2524184117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6807","nonce":2524184117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6805","nonce":2524184117}]},"public_addr":"172.21.15.70:6801/2524184117","cluster_addr":"172.21.15.70:6803/2524184117","heartbeat_back_addr":"172.21.15.70:6807/2524184117","heartbeat_front_addr":"172.21.15.70:6805/2524184117","state":["exists","up"]},{"osd":3,"uuid":"edb44507-ffb0-42a4-827b-08461e1bf952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6811","nonce":2823036296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6813","nonce":2823036296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6817","nonce":2823036296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6815","nonce":2823036296}]},"public_addr":"172.21.15.31:6811/2823036296","cluster_addr":"172.21.15.31:6813/2823036296","heartbeat_back_addr":"172.21.15.31:6817/2823036296","heartbeat_front_addr":"172.21.15.31:6815/2823036296","state":["exists","up"]},{"osd":4,"uuid":"8cb89153-8c3b-431f-b57c-8863fada76dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6809","nonce":4215186309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6811","nonce":4215186309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6815","nonce":4215186309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6813","nonce":4215186309}]},"public_addr":"172.21.15.40:6809/4215186309","cluster_addr":"172.21.15.40:6811/4215186309","heartbeat_back_addr":"172.21.15.40:6815/4215186309","heartbeat_front_addr":"172.21.15.40:6813/4215186309","state":["exists","up"]},{"osd":5,"uuid":"9409505a-d463-408c-a3cd-218380c04d68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6809","nonce":3402830452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6811","nonce":3402830452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6815","nonce":3402830452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6813","nonce":3402830452}]},"public_addr":"172.21.15.70:6809/3402830452","cluster_addr":"172.21.15.70:6811/3402830452","heartbeat_back_addr":"172.21.15.70:6815/3402830452","heartbeat_front_addr":"172.21.15.70:6813/3402830452","state":["exists","up"]},{"osd":6,"uuid":"f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6819","nonce":1193497058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6821","nonce":1193497058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6825","nonce":1193497058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6823","nonce":1193497058}]},"public_addr":"172.21.15.31:6819/1193497058","cluster_addr":"172.21.15.31:6821/1193497058","heartbeat_back_addr":"172.21.15.31:6825/1193497058","heartbeat_front_addr":"172.21.15.31:6823/1193497058","state":["exists","up"]},{"osd":7,"uuid":"db228a87-29a5-45d4-8fe6-a0a554262eb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6817","nonce":3825628119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6819","nonce":3825628119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6823","nonce":3825628119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6821","nonce":3825628119}]},"public_addr":"172.21.15.40:6817/3825628119","cluster_addr":"172.21.15.40:6819/3825628119","heartbeat_back_addr":"172.21.15.40:6823/3825628119","heartbeat_front_addr":"172.21.15.40:6821/3825628119","state":["exists","up"]},{"osd":8,"uuid":"59b9fde0-095d-4e0b-87da-ead37bd8ff1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6817","nonce":534880586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6819","nonce":534880586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6823","nonce":534880586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6821","nonce":534880586}]},"public_addr":"172.21.15.70:6817/534880586","cluster_addr":"172.21.15.70:6819/534880586","heartbeat_back_addr":"172.21.15.70:6823/534880586","heartbeat_front_addr":"172.21.15.70:6821/534880586","state":["exists","up"]},{"osd":9,"uuid":"77f47061-0c5f-4159-b276-e814935d6c1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6827","nonce":587047469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6829","nonce":587047469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6833","nonce":587047469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6831","nonce":587047469}]},"public_addr":"172.21.15.31:6827/587047469","cluster_addr":"172.21.15.31:6829/587047469","heartbeat_back_addr":"172.21.15.31:6833/587047469","heartbeat_front_addr":"172.21.15.31:6831/587047469","state":["exists","up"]},{"osd":10,"uuid":"8853dc43-7bd1-4c93-8063-f7982fd7da38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6825","nonce":512108711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6827","nonce":512108711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6831","nonce":512108711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6829","nonce":512108711}]},"public_addr":"172.21.15.40:6825/512108711","cluster_addr":"172.21.15.40:6827/512108711","heartbeat_back_addr":"172.21.15.40:6831/512108711","heartbeat_front_addr":"172.21.15.40:6829/512108711","state":["exists","up"]},{"osd":11,"uuid":"84b7c2de-c424-45a6-9583-60097e372a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6825","nonce":540305718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6827","nonce":540305718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6831","nonce":540305718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6829","nonce":540305718}]},"public_addr":"172.21.15.70:6825/540305718","cluster_addr":"172.21.15.70:6827/540305718","heartbeat_back_addr":"172.21.15.70:6831/540305718","heartbeat_front_addr":"172.21.15.70:6829/540305718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:28:51.477358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:17.405045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:41.626268+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:15.339568+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:42.676996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:07.516999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:42.412273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:10.365041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:35.699120+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:11.733639+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:39.505442+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:34:05.449301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451930554":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/3342588843":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6801/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/1626448389":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/1910059351":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/344822770":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/3811368512":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/1981437651":"2024-03-22T10:26:16.959165+0000","172.21.15.31:6801/1981437651":"2024-03-22T10:26:16.959165+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T10:34:12.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:12.158+0000 7f650ca00700 1 -- 172.21.15.31:0/3524889183 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f64ec0a3030 msgr2=0x7f64ec0a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:12.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:12.158+0000 7f650ca00700 1 --2- 172.21.15.31:0/3524889183 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f64ec0a3030 0x7f64ec0a54f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f65081035f0 tx=0x7f64f400b260 comp rx=0 tx=0).stop 2024-03-21T10:34:12.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:12.158+0000 7f650ca00700 1 -- 172.21.15.31:0/3524889183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65081045e0 msgr2=0x7f65081a07a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:12.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:12.158+0000 7f650ca00700 1 --2- 172.21.15.31:0/3524889183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65081045e0 0x7f65081a07a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f64f800e9a0 tx=0x7f64f800ecb0 comp rx=0 tx=0).stop 2024-03-21T10:34:12.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:12.158+0000 7f650ca00700 1 -- 172.21.15.31:0/3524889183 shutdown_connections 2024-03-21T10:34:12.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:12.158+0000 7f650ca00700 1 --2- 172.21.15.31:0/3524889183 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f64ec0a3030 0x7f64ec0a54f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:12.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:12.158+0000 7f650ca00700 1 --2- 172.21.15.31:0/3524889183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65081096c0 0x7f650819a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:12.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:12.158+0000 7f650ca00700 1 --2- 172.21.15.31:0/3524889183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65081045e0 0x7f65081a07a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:12.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:12.158+0000 7f650ca00700 1 --2- 172.21.15.31:0/3524889183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6508104f90 0x7f65081a0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:12.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:12.158+0000 7f650ca00700 1 -- 172.21.15.31:0/3524889183 >> 172.21.15.31:0/3524889183 conn(0x7f65080ffe70 msgr2=0x7f65081009f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:12.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:12.159+0000 7f650ca00700 1 -- 172.21.15.31:0/3524889183 shutdown_connections 2024-03-21T10:34:12.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:12.159+0000 7f650ca00700 1 -- 172.21.15.31:0/3524889183 wait complete. 2024-03-21T10:34:12.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:11 smithi040 ceph-mon[36154]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:13.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:12 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3524889183' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:34:13.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:12 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3524889183' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:34:13.153 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T10:29:44.553968+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-21T10:34:13.154 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-21T10:34:13.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:12 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3524889183' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:34:13.628 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:34:14.037 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:13 smithi031 ceph-mon[31673]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:14.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:13 smithi070 ceph-mon[36330]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:14.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:13 smithi040 ceph-mon[36154]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:15.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.946+0000 7feeb7200700 1 -- 172.21.15.31:0/3641351530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feeb0102440 msgr2=0x7feeb0102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:15.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.946+0000 7feeb7200700 1 --2- 172.21.15.31:0/3641351530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feeb0102440 0x7feeb0102820 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7feea00099c0 tx=0x7feea0009cd0 comp rx=0 tx=0).stop 2024-03-21T10:34:15.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.947+0000 7feeb7200700 1 -- 172.21.15.31:0/3641351530 shutdown_connections 2024-03-21T10:34:15.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.947+0000 7feeb7200700 1 --2- 172.21.15.31:0/3641351530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feeb0107510 0x7feeb010f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:15.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.947+0000 7feeb7200700 1 --2- 172.21.15.31:0/3641351530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feeb0102440 0x7feeb0102820 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:15.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.947+0000 7feeb7200700 1 --2- 172.21.15.31:0/3641351530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feeb0102df0 0x7feeb0106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:15.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.947+0000 7feeb7200700 1 -- 172.21.15.31:0/3641351530 >> 172.21.15.31:0/3641351530 conn(0x7feeb00792f0 msgr2=0x7feeb0079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:15.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.947+0000 7feeb7200700 1 -- 172.21.15.31:0/3641351530 shutdown_connections 2024-03-21T10:34:15.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.947+0000 7feeb7200700 1 -- 172.21.15.31:0/3641351530 wait complete. 2024-03-21T10:34:15.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.948+0000 7feeb7200700 1 Processor -- start 2024-03-21T10:34:15.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.948+0000 7feeb7200700 1 -- start start 2024-03-21T10:34:15.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.949+0000 7feeb7200700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feeb0102440 0x7feeb01a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:15.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.949+0000 7feeb7200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feeb0102df0 0x7feeb01a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:15.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.949+0000 7feeb7200700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feeb0107510 0x7feeb019a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:15.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.949+0000 7feeb7200700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb0112540 con 0x7feeb0102440 2024-03-21T10:34:15.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.949+0000 7feeb7200700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb01123c0 con 0x7feeb0107510 2024-03-21T10:34:15.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.949+0000 7feeb7200700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb01126c0 con 0x7feeb0102df0 2024-03-21T10:34:15.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.949+0000 7feeafe00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feeb0102440 0x7feeb01a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:15.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.949+0000 7feeaf400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feeb0102df0 0x7feeb01a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:15.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.949+0000 7feeaf400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feeb0102df0 0x7feeb01a0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:35782/0 (socket says 172.21.15.31:35782) 2024-03-21T10:34:15.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.949+0000 7feeb5200700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feeb0107510 0x7feeb019a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:15.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.949+0000 7feeaf400700 1 -- 172.21.15.31:0/2160765030 learned_addr learned my addr 172.21.15.31:0/2160765030 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:34:15.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.949+0000 7feeaf400700 1 -- 172.21.15.31:0/2160765030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feeb0107510 msgr2=0x7feeb019a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:15.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.949+0000 7feeaf400700 1 --2- 172.21.15.31:0/2160765030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feeb0107510 0x7feeb019a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:15.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.949+0000 7feeaf400700 1 -- 172.21.15.31:0/2160765030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feeb0102440 msgr2=0x7feeb01a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:15.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.949+0000 7feeaf400700 1 --2- 172.21.15.31:0/2160765030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feeb0102440 0x7feeb01a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:15.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.949+0000 7feeaf400700 1 -- 172.21.15.31:0/2160765030 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb019b120 con 0x7feeb0102df0 2024-03-21T10:34:15.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.949+0000 7feeb5200700 1 --2- 172.21.15.31:0/2160765030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feeb0107510 0x7feeb019a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:34:15.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.950+0000 7feeafe00700 1 --2- 172.21.15.31:0/2160765030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feeb0102440 0x7feeb01a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:34:15.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.950+0000 7feeaf400700 1 --2- 172.21.15.31:0/2160765030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feeb0102df0 0x7feeb01a0d40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fee9c00e970 tx=0x7fee9c00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:15.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.950+0000 7feeacc00700 1 -- 172.21.15.31:0/2160765030 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9c004a30 con 0x7feeb0102df0 2024-03-21T10:34:15.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.950+0000 7feeacc00700 1 -- 172.21.15.31:0/2160765030 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee9c00ccc0 con 0x7feeb0102df0 2024-03-21T10:34:15.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.950+0000 7feeacc00700 1 -- 172.21.15.31:0/2160765030 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9c005650 con 0x7feeb0102df0 2024-03-21T10:34:15.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.950+0000 7feeb7200700 1 -- 172.21.15.31:0/2160765030 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeb019b410 con 0x7feeb0102df0 2024-03-21T10:34:15.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.950+0000 7feeb7200700 1 -- 172.21.15.31:0/2160765030 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeb01a76d0 con 0x7feeb0102df0 2024-03-21T10:34:15.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.951+0000 7feeb7200700 1 -- 172.21.15.31:0/2160765030 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeb0068e50 con 0x7feeb0102df0 2024-03-21T10:34:15.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.953+0000 7feeacc00700 1 -- 172.21.15.31:0/2160765030 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fee9c004bd0 con 0x7feeb0102df0 2024-03-21T10:34:15.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.955+0000 7feeacc00700 1 --2- 172.21.15.31:0/2160765030 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fee940a2fb0 0x7fee940a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:15.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.955+0000 7feeacc00700 1 -- 172.21.15.31:0/2160765030 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fee9c0cf380 con 0x7feeb0102df0 2024-03-21T10:34:15.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.955+0000 7feeafe00700 1 --2- 172.21.15.31:0/2160765030 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fee940a2fb0 0x7fee940a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:15.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.956+0000 7feeafe00700 1 --2- 172.21.15.31:0/2160765030 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fee940a2fb0 0x7fee940a5470 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7feea0000c00 tx=0x7feea0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:15.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:15.959+0000 7feeacc00700 1 -- 172.21.15.31:0/2160765030 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fee9c09c5b0 con 0x7feeb0102df0 2024-03-21T10:34:16.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:15 smithi031 ceph-mon[31673]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:16.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:15 smithi070 ceph-mon[36330]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:16.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:15 smithi040 ceph-mon[36154]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:16.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:16.185+0000 7feeb7200700 1 -- 172.21.15.31:0/2160765030 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feeb01a7900 con 0x7feeb0102df0 2024-03-21T10:34:16.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:16.186+0000 7feeacc00700 1 -- 172.21.15.31:0/2160765030 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7fee9c09beb0 con 0x7feeb0102df0 2024-03-21T10:34:16.185 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-03-21T10:34:16.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:16.189+0000 7feeb7200700 1 -- 172.21.15.31:0/2160765030 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fee940a2fb0 msgr2=0x7fee940a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:16.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:16.189+0000 7feeb7200700 1 --2- 172.21.15.31:0/2160765030 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fee940a2fb0 0x7fee940a5470 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7feea0000c00 tx=0x7feea0005fb0 comp rx=0 tx=0).stop 2024-03-21T10:34:16.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:16.189+0000 7feeb7200700 1 -- 172.21.15.31:0/2160765030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feeb0102df0 msgr2=0x7feeb01a0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:16.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:16.189+0000 7feeb7200700 1 --2- 172.21.15.31:0/2160765030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feeb0102df0 0x7feeb01a0d40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fee9c00e970 tx=0x7fee9c00ec80 comp rx=0 tx=0).stop 2024-03-21T10:34:16.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:16.190+0000 7feeb7200700 1 -- 172.21.15.31:0/2160765030 shutdown_connections 2024-03-21T10:34:16.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:16.190+0000 7feeb7200700 1 --2- 172.21.15.31:0/2160765030 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fee940a2fb0 0x7fee940a5470 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:16.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:16.190+0000 7feeb7200700 1 --2- 172.21.15.31:0/2160765030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feeb0107510 0x7feeb019a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:16.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:16.190+0000 7feeb7200700 1 --2- 172.21.15.31:0/2160765030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feeb0102440 0x7feeb01a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:16.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:16.190+0000 7feeb7200700 1 --2- 172.21.15.31:0/2160765030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feeb0102df0 0x7feeb01a0d40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:16.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:16.190+0000 7feeb7200700 1 -- 172.21.15.31:0/2160765030 >> 172.21.15.31:0/2160765030 conn(0x7feeb00792f0 msgr2=0x7feeb0100590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:16.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:16.190+0000 7feeb7200700 1 -- 172.21.15.31:0/2160765030 shutdown_connections 2024-03-21T10:34:16.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:16.190+0000 7feeb7200700 1 -- 172.21.15.31:0/2160765030 wait complete. 2024-03-21T10:34:16.968 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:16 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2160765030' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:34:17.050 INFO:tasks.cephadm:Adding mds.a on smithi031 2024-03-21T10:34:17.050 INFO:tasks.cephadm:Adding mds.d on smithi031 2024-03-21T10:34:17.050 INFO:tasks.cephadm:Adding mds.g on smithi031 2024-03-21T10:34:17.050 INFO:tasks.cephadm:Adding mds.j on smithi031 2024-03-21T10:34:17.050 INFO:tasks.cephadm:Adding mds.b on smithi040 2024-03-21T10:34:17.050 INFO:tasks.cephadm:Adding mds.e on smithi040 2024-03-21T10:34:17.050 INFO:tasks.cephadm:Adding mds.h on smithi040 2024-03-21T10:34:17.050 INFO:tasks.cephadm:Adding mds.k on smithi040 2024-03-21T10:34:17.051 INFO:tasks.cephadm:Adding mds.c on smithi070 2024-03-21T10:34:17.051 INFO:tasks.cephadm:Adding mds.f on smithi070 2024-03-21T10:34:17.051 INFO:tasks.cephadm:Adding mds.i on smithi070 2024-03-21T10:34:17.051 INFO:tasks.cephadm:Adding mds.l on smithi070 2024-03-21T10:34:17.051 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph orch apply mds all '12;smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi070=c;smithi070=f;smithi070=i;smithi070=l' 2024-03-21T10:34:17.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:16 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2160765030' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:34:17.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:16 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2160765030' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:34:17.301 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.c/config 2024-03-21T10:34:18.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:17 smithi070 ceph-mon[36330]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:18.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:17 smithi040 ceph-mon[36154]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:18.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:17 smithi031 ceph-mon[31673]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:19.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.053+0000 7f865eeb4700 1 -- 172.21.15.70:0/1912889237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86581055b0 msgr2=0x7f86581079a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:19.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.053+0000 7f865eeb4700 1 --2- 172.21.15.70:0/1912889237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86581055b0 0x7f86581079a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8648009990 tx=0x7f8648009ca0 comp rx=0 tx=0).stop 2024-03-21T10:34:19.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.053+0000 7f865eeb4700 1 -- 172.21.15.70:0/1912889237 shutdown_connections 2024-03-21T10:34:19.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.053+0000 7f865eeb4700 1 --2- 172.21.15.70:0/1912889237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8658107ee0 0x7f865810a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:19.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.053+0000 7f865eeb4700 1 --2- 172.21.15.70:0/1912889237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8658102c80 0x7f8658105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:19.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.053+0000 7f865eeb4700 1 --2- 172.21.15.70:0/1912889237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86581055b0 0x7f86581079a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:19.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.053+0000 7f865eeb4700 1 -- 172.21.15.70:0/1912889237 >> 172.21.15.70:0/1912889237 conn(0x7f86580fc690 msgr2=0x7f86580fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:19.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.053+0000 7f865eeb4700 1 -- 172.21.15.70:0/1912889237 shutdown_connections 2024-03-21T10:34:19.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.053+0000 7f865eeb4700 1 -- 172.21.15.70:0/1912889237 wait complete. 2024-03-21T10:34:19.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.053+0000 7f865eeb4700 1 Processor -- start 2024-03-21T10:34:19.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.053+0000 7f865eeb4700 1 -- start start 2024-03-21T10:34:19.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.054+0000 7f865eeb4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8658102c80 0x7f8658198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:19.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.054+0000 7f865eeb4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86581055b0 0x7f8658198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:19.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.054+0000 7f865eeb4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8658107ee0 0x7f865819f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:19.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.054+0000 7f865eeb4700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865810de60 con 0x7f8658107ee0 2024-03-21T10:34:19.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.054+0000 7f865eeb4700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865810dce0 con 0x7f8658102c80 2024-03-21T10:34:19.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.054+0000 7f865eeb4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865810dfe0 con 0x7f86581055b0 2024-03-21T10:34:19.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.054+0000 7f8657fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86581055b0 0x7f8658198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:19.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.054+0000 7f8657fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86581055b0 0x7f8658198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34312/0 (socket says 172.21.15.70:34312) 2024-03-21T10:34:19.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.054+0000 7f8657fff700 1 -- 172.21.15.70:0/2056342351 learned_addr learned my addr 172.21.15.70:0/2056342351 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-03-21T10:34:19.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.054+0000 7f865cc50700 1 --2- 172.21.15.70:0/2056342351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8658102c80 0x7f8658198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:19.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.054+0000 7f8657fff700 1 -- 172.21.15.70:0/2056342351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8658102c80 msgr2=0x7f8658198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:19.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.054+0000 7f8657fff700 1 --2- 172.21.15.70:0/2056342351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8658102c80 0x7f8658198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:19.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.054+0000 7f8657fff700 1 -- 172.21.15.70:0/2056342351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8658107ee0 msgr2=0x7f865819f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:19.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.054+0000 7f8657fff700 1 --2- 172.21.15.70:0/2056342351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8658107ee0 0x7f865819f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:19.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.054+0000 7f8657fff700 1 -- 172.21.15.70:0/2056342351 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86581a0070 con 0x7f86581055b0 2024-03-21T10:34:19.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.054+0000 7f8657fff700 1 --2- 172.21.15.70:0/2056342351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86581055b0 0x7f8658198950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f86480047b0 tx=0x7f8648009f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:19.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.054+0000 7f8655ffb700 1 -- 172.21.15.70:0/2056342351 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8648027070 con 0x7f86581055b0 2024-03-21T10:34:19.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.055+0000 7f865eeb4700 1 -- 172.21.15.70:0/2056342351 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86581a0300 con 0x7f86581055b0 2024-03-21T10:34:19.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.055+0000 7f8655ffb700 1 -- 172.21.15.70:0/2056342351 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8648022470 con 0x7f86581055b0 2024-03-21T10:34:19.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.055+0000 7f8655ffb700 1 -- 172.21.15.70:0/2056342351 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864800f6c0 con 0x7f86581055b0 2024-03-21T10:34:19.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.055+0000 7f865eeb4700 1 -- 172.21.15.70:0/2056342351 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86581a0830 con 0x7f86581055b0 2024-03-21T10:34:19.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.056+0000 7f865eeb4700 1 -- 172.21.15.70:0/2056342351 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8658068e50 con 0x7f86581055b0 2024-03-21T10:34:19.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.057+0000 7f8655ffb700 1 -- 172.21.15.70:0/2056342351 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f864802a070 con 0x7f86581055b0 2024-03-21T10:34:19.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.057+0000 7f8655ffb700 1 --2- 172.21.15.70:0/2056342351 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f86440a2fb0 0x7f86440a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:19.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.058+0000 7f8655ffb700 1 -- 172.21.15.70:0/2056342351 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f86480cffd0 con 0x7f86581055b0 2024-03-21T10:34:19.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.058+0000 7f865cc50700 1 --2- 172.21.15.70:0/2056342351 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f86440a2fb0 0x7f86440a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:19.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.059+0000 7f865cc50700 1 --2- 172.21.15.70:0/2056342351 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f86440a2fb0 0x7f86440a5470 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f86500060b0 tx=0x7f865000c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:19.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.064+0000 7f8655ffb700 1 -- 172.21.15.70:0/2056342351 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f864805e740 con 0x7f86581055b0 2024-03-21T10:34:19.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.273+0000 7f865eeb4700 1 -- 172.21.15.70:0/2056342351 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi070=c;smithi070=f;smithi070=i;smithi070=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f86580627d0 con 0x7f86440a2fb0 2024-03-21T10:34:19.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.289+0000 7f8655ffb700 1 -- 172.21.15.70:0/2056342351 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f86580627d0 con 0x7f86440a2fb0 2024-03-21T10:34:19.289 INFO:teuthology.orchestra.run.smithi070.stdout:Scheduled mds.all update... 2024-03-21T10:34:19.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.291+0000 7f865eeb4700 1 -- 172.21.15.70:0/2056342351 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f86440a2fb0 msgr2=0x7f86440a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:19.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.291+0000 7f865eeb4700 1 --2- 172.21.15.70:0/2056342351 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f86440a2fb0 0x7f86440a5470 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f86500060b0 tx=0x7f865000c040 comp rx=0 tx=0).stop 2024-03-21T10:34:19.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.291+0000 7f865eeb4700 1 -- 172.21.15.70:0/2056342351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86581055b0 msgr2=0x7f8658198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:19.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.291+0000 7f865eeb4700 1 --2- 172.21.15.70:0/2056342351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86581055b0 0x7f8658198950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f86480047b0 tx=0x7f8648009f50 comp rx=0 tx=0).stop 2024-03-21T10:34:19.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.291+0000 7f865eeb4700 1 -- 172.21.15.70:0/2056342351 shutdown_connections 2024-03-21T10:34:19.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.292+0000 7f865eeb4700 1 --2- 172.21.15.70:0/2056342351 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f86440a2fb0 0x7f86440a5470 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:19.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.292+0000 7f865eeb4700 1 --2- 172.21.15.70:0/2056342351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8658107ee0 0x7f865819f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:19.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.292+0000 7f865eeb4700 1 --2- 172.21.15.70:0/2056342351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8658102c80 0x7f8658198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:19.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.292+0000 7f865eeb4700 1 --2- 172.21.15.70:0/2056342351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86581055b0 0x7f8658198950 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:19.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.292+0000 7f865eeb4700 1 -- 172.21.15.70:0/2056342351 >> 172.21.15.70:0/2056342351 conn(0x7f86580fc690 msgr2=0x7f86581035c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:19.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.292+0000 7f865eeb4700 1 -- 172.21.15.70:0/2056342351 shutdown_connections 2024-03-21T10:34:19.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-21T10:34:19.292+0000 7f865eeb4700 1 -- 172.21.15.70:0/2056342351 wait complete. 2024-03-21T10:34:20.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:19 smithi070 ceph-mon[36330]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:20.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:19 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:20.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:19 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:34:20.073 DEBUG:teuthology.orchestra.run.smithi031:mds.a> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.a.service 2024-03-21T10:34:20.079 DEBUG:teuthology.orchestra.run.smithi031:mds.d> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.d.service 2024-03-21T10:34:20.124 DEBUG:teuthology.orchestra.run.smithi031:mds.g> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.g.service 2024-03-21T10:34:20.143 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:19 smithi031 ceph-mon[31673]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:20.143 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:19 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:20.143 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:19 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:34:20.147 DEBUG:teuthology.orchestra.run.smithi031:mds.j> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.j.service 2024-03-21T10:34:20.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:19 smithi040 ceph-mon[36154]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:20.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:19 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:20.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:19 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:34:20.193 DEBUG:teuthology.orchestra.run.smithi040:mds.b> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.b.service 2024-03-21T10:34:20.196 DEBUG:teuthology.orchestra.run.smithi040:mds.e> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.e.service 2024-03-21T10:34:20.239 DEBUG:teuthology.orchestra.run.smithi040:mds.h> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.h.service 2024-03-21T10:34:20.282 DEBUG:teuthology.orchestra.run.smithi040:mds.k> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.k.service 2024-03-21T10:34:20.325 DEBUG:teuthology.orchestra.run.smithi070:mds.c> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.c.service 2024-03-21T10:34:20.328 DEBUG:teuthology.orchestra.run.smithi070:mds.f> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.f.service 2024-03-21T10:34:20.371 DEBUG:teuthology.orchestra.run.smithi070:mds.i> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.i.service 2024-03-21T10:34:20.414 DEBUG:teuthology.orchestra.run.smithi070:mds.l> sudo journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.l.service 2024-03-21T10:34:20.457 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-21T10:34:20.457 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T10:34:20.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.709+0000 7f217f200640 1 -- 172.21.15.31:0/2340865212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2180108db0 msgr2=0x7f2180109230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:20.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.709+0000 7f217f200640 1 --2- 172.21.15.31:0/2340865212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2180108db0 0x7f2180109230 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2168009860 tx=0x7f216802bb70 comp rx=0 tx=0).stop 2024-03-21T10:34:20.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.709+0000 7f217f200640 1 -- 172.21.15.31:0/2340865212 shutdown_connections 2024-03-21T10:34:20.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.709+0000 7f217f200640 1 --2- 172.21.15.31:0/2340865212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2180109770 0x7f2180110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:20.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.709+0000 7f217f200640 1 --2- 172.21.15.31:0/2340865212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2180108db0 0x7f2180109230 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:20.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.709+0000 7f217f200640 1 --2- 172.21.15.31:0/2340865212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2180107bc0 0x7f2180107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:20.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.709+0000 7f217f200640 1 -- 172.21.15.31:0/2340865212 >> 172.21.15.31:0/2340865212 conn(0x7f2180076b60 msgr2=0x7f2180078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:20.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.709+0000 7f217f200640 1 -- 172.21.15.31:0/2340865212 shutdown_connections 2024-03-21T10:34:20.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.709+0000 7f217f200640 1 -- 172.21.15.31:0/2340865212 wait complete. 2024-03-21T10:34:20.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.710+0000 7f217f200640 1 Processor -- start 2024-03-21T10:34:20.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.711+0000 7f217f200640 1 -- start start 2024-03-21T10:34:20.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.711+0000 7f217f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2180107bc0 0x7f218019de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:20.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.711+0000 7f217f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2180108db0 0x7f218019e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:20.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.711+0000 7f217f200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2180109770 0x7f21801a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:20.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.711+0000 7f217f200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2180113820 con 0x7f2180107bc0 2024-03-21T10:34:20.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.711+0000 7f217f200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21801136a0 con 0x7f2180108db0 2024-03-21T10:34:20.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.711+0000 7f217f200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21801139a0 con 0x7f2180109770 2024-03-21T10:34:20.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.711+0000 7f2175400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2180108db0 0x7f218019e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:20.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.711+0000 7f217e800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2180109770 0x7f21801a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:20.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.711+0000 7f2175400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2180108db0 0x7f218019e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:45434/0 (socket says 172.21.15.31:45434) 2024-03-21T10:34:20.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.711+0000 7f217de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2180107bc0 0x7f218019de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:20.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.711+0000 7f217e800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2180109770 0x7f21801a5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:35824/0 (socket says 172.21.15.31:35824) 2024-03-21T10:34:20.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.711+0000 7f2175400640 1 -- 172.21.15.31:0/625029889 learned_addr learned my addr 172.21.15.31:0/625029889 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:34:20.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.712+0000 7f2175400640 1 -- 172.21.15.31:0/625029889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2180109770 msgr2=0x7f21801a5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:20.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.712+0000 7f2175400640 1 --2- 172.21.15.31:0/625029889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2180109770 0x7f21801a5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:20.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.712+0000 7f2175400640 1 -- 172.21.15.31:0/625029889 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2180107bc0 msgr2=0x7f218019de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:20.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.712+0000 7f2175400640 1 --2- 172.21.15.31:0/625029889 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2180107bc0 0x7f218019de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:20.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.712+0000 7f2175400640 1 -- 172.21.15.31:0/625029889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21801a5ab0 con 0x7f2180108db0 2024-03-21T10:34:20.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.712+0000 7f2175400640 1 --2- 172.21.15.31:0/625029889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2180108db0 0x7f218019e3d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f21680349e0 tx=0x7f2168032450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:20.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.712+0000 7f2177400640 1 -- 172.21.15.31:0/625029889 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2168041030 con 0x7f2180108db0 2024-03-21T10:34:20.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.712+0000 7f2177400640 1 -- 172.21.15.31:0/625029889 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f216802c510 con 0x7f2180108db0 2024-03-21T10:34:20.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.712+0000 7f2177400640 1 -- 172.21.15.31:0/625029889 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2168039040 con 0x7f2180108db0 2024-03-21T10:34:20.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.712+0000 7f217f200640 1 -- 172.21.15.31:0/625029889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21801a5d40 con 0x7f2180108db0 2024-03-21T10:34:20.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.713+0000 7f217f200640 1 -- 172.21.15.31:0/625029889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21801a6220 con 0x7f2180108db0 2024-03-21T10:34:20.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.714+0000 7f217f200640 1 -- 172.21.15.31:0/625029889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f218010eee0 con 0x7f2180108db0 2024-03-21T10:34:20.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.716+0000 7f2177400640 1 -- 172.21.15.31:0/625029889 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f2168044070 con 0x7f2180108db0 2024-03-21T10:34:20.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.717+0000 7f2177400640 1 --2- 172.21.15.31:0/625029889 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f21640a2f90 0x7f21640a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:20.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.717+0000 7f2177400640 1 -- 172.21.15.31:0/625029889 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f21680f0480 con 0x7f2180108db0 2024-03-21T10:34:20.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.717+0000 7f217de00640 1 --2- 172.21.15.31:0/625029889 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f21640a2f90 0x7f21640a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:20.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.718+0000 7f217de00640 1 --2- 172.21.15.31:0/625029889 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f21640a2f90 0x7f21640a5450 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f21700095f0 tx=0x7f2170005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:20.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.721+0000 7f2177400640 1 -- 172.21.15.31:0/625029889 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f21680bd6b0 con 0x7f2180108db0 2024-03-21T10:34:20.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.927+0000 7f217f200640 1 -- 172.21.15.31:0/625029889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21800622a0 con 0x7f2180108db0 2024-03-21T10:34:20.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.929+0000 7f2177400640 1 -- 172.21.15.31:0/625029889 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18337 (secure 0 0 0) 0x7f21680bcfb0 con 0x7f2180108db0 2024-03-21T10:34:20.927 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:34:20.927 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":81,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","created":"2024-03-21T10:25:41.971117+0000","modified":"2024-03-21T10:34:09.260664+0000","last_up_change":"2024-03-21T10:34:07.757196+0000","last_in_change":"2024-03-21T10:33:47.460406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T10:29:44.553968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0244d33-00e2-47d5-9f66-34efc138c462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6803","nonce":3089885125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6805","nonce":3089885125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6809","nonce":3089885125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6807","nonce":3089885125}]},"public_addr":"172.21.15.31:6803/3089885125","cluster_addr":"172.21.15.31:6805/3089885125","heartbeat_back_addr":"172.21.15.31:6809/3089885125","heartbeat_front_addr":"172.21.15.31:6807/3089885125","state":["exists","up"]},{"osd":1,"uuid":"76a63504-beab-4354-a6c3-eebc3b16ad03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6801","nonce":618840037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6803","nonce":618840037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6807","nonce":618840037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6805","nonce":618840037}]},"public_addr":"172.21.15.40:6801/618840037","cluster_addr":"172.21.15.40:6803/618840037","heartbeat_back_addr":"172.21.15.40:6807/618840037","heartbeat_front_addr":"172.21.15.40:6805/618840037","state":["exists","up"]},{"osd":2,"uuid":"fa191bd6-9a21-4ee2-8adf-8a63c9cfd821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6801","nonce":2524184117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6803","nonce":2524184117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6807","nonce":2524184117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6805","nonce":2524184117}]},"public_addr":"172.21.15.70:6801/2524184117","cluster_addr":"172.21.15.70:6803/2524184117","heartbeat_back_addr":"172.21.15.70:6807/2524184117","heartbeat_front_addr":"172.21.15.70:6805/2524184117","state":["exists","up"]},{"osd":3,"uuid":"edb44507-ffb0-42a4-827b-08461e1bf952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6811","nonce":2823036296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6813","nonce":2823036296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6817","nonce":2823036296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6815","nonce":2823036296}]},"public_addr":"172.21.15.31:6811/2823036296","cluster_addr":"172.21.15.31:6813/2823036296","heartbeat_back_addr":"172.21.15.31:6817/2823036296","heartbeat_front_addr":"172.21.15.31:6815/2823036296","state":["exists","up"]},{"osd":4,"uuid":"8cb89153-8c3b-431f-b57c-8863fada76dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6809","nonce":4215186309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6811","nonce":4215186309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6815","nonce":4215186309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6813","nonce":4215186309}]},"public_addr":"172.21.15.40:6809/4215186309","cluster_addr":"172.21.15.40:6811/4215186309","heartbeat_back_addr":"172.21.15.40:6815/4215186309","heartbeat_front_addr":"172.21.15.40:6813/4215186309","state":["exists","up"]},{"osd":5,"uuid":"9409505a-d463-408c-a3cd-218380c04d68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6809","nonce":3402830452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6811","nonce":3402830452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6815","nonce":3402830452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6813","nonce":3402830452}]},"public_addr":"172.21.15.70:6809/3402830452","cluster_addr":"172.21.15.70:6811/3402830452","heartbeat_back_addr":"172.21.15.70:6815/3402830452","heartbeat_front_addr":"172.21.15.70:6813/3402830452","state":["exists","up"]},{"osd":6,"uuid":"f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6819","nonce":1193497058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6821","nonce":1193497058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6825","nonce":1193497058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6823","nonce":1193497058}]},"public_addr":"172.21.15.31:6819/1193497058","cluster_addr":"172.21.15.31:6821/1193497058","heartbeat_back_addr":"172.21.15.31:6825/1193497058","heartbeat_front_addr":"172.21.15.31:6823/1193497058","state":["exists","up"]},{"osd":7,"uuid":"db228a87-29a5-45d4-8fe6-a0a554262eb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6817","nonce":3825628119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6819","nonce":3825628119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6823","nonce":3825628119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6821","nonce":3825628119}]},"public_addr":"172.21.15.40:6817/3825628119","cluster_addr":"172.21.15.40:6819/3825628119","heartbeat_back_addr":"172.21.15.40:6823/3825628119","heartbeat_front_addr":"172.21.15.40:6821/3825628119","state":["exists","up"]},{"osd":8,"uuid":"59b9fde0-095d-4e0b-87da-ead37bd8ff1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6817","nonce":534880586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6819","nonce":534880586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6823","nonce":534880586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6821","nonce":534880586}]},"public_addr":"172.21.15.70:6817/534880586","cluster_addr":"172.21.15.70:6819/534880586","heartbeat_back_addr":"172.21.15.70:6823/534880586","heartbeat_front_addr":"172.21.15.70:6821/534880586","state":["exists","up"]},{"osd":9,"uuid":"77f47061-0c5f-4159-b276-e814935d6c1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6827","nonce":587047469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6829","nonce":587047469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6833","nonce":587047469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6831","nonce":587047469}]},"public_addr":"172.21.15.31:6827/587047469","cluster_addr":"172.21.15.31:6829/587047469","heartbeat_back_addr":"172.21.15.31:6833/587047469","heartbeat_front_addr":"172.21.15.31:6831/587047469","state":["exists","up"]},{"osd":10,"uuid":"8853dc43-7bd1-4c93-8063-f7982fd7da38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6825","nonce":512108711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6827","nonce":512108711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6831","nonce":512108711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6829","nonce":512108711}]},"public_addr":"172.21.15.40:6825/512108711","cluster_addr":"172.21.15.40:6827/512108711","heartbeat_back_addr":"172.21.15.40:6831/512108711","heartbeat_front_addr":"172.21.15.40:6829/512108711","state":["exists","up"]},{"osd":11,"uuid":"84b7c2de-c424-45a6-9583-60097e372a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6825","nonce":540305718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6827","nonce":540305718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6831","nonce":540305718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6829","nonce":540305718}]},"public_addr":"172.21.15.70:6825/540305718","cluster_addr":"172.21.15.70:6827/540305718","heartbeat_back_addr":"172.21.15.70:6831/540305718","heartbeat_front_addr":"172.21.15.70:6829/540305718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:28:51.477358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:17.405045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:41.626268+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:15.339568+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:42.676996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:07.516999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:42.412273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:10.365041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:35.699120+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:11.733639+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:39.505442+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:34:05.449301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451930554":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/3342588843":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6801/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/1626448389":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/1910059351":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/344822770":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/3811368512":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/1981437651":"2024-03-22T10:26:16.959165+0000","172.21.15.31:6801/1981437651":"2024-03-22T10:26:16.959165+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T10:34:20.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.931+0000 7f217f200640 1 -- 172.21.15.31:0/625029889 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f21640a2f90 msgr2=0x7f21640a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:20.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.931+0000 7f217f200640 1 --2- 172.21.15.31:0/625029889 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f21640a2f90 0x7f21640a5450 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f21700095f0 tx=0x7f2170005f90 comp rx=0 tx=0).stop 2024-03-21T10:34:20.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.931+0000 7f217f200640 1 -- 172.21.15.31:0/625029889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2180108db0 msgr2=0x7f218019e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:20.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.931+0000 7f217f200640 1 --2- 172.21.15.31:0/625029889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2180108db0 0x7f218019e3d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f21680349e0 tx=0x7f2168032450 comp rx=0 tx=0).stop 2024-03-21T10:34:20.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.932+0000 7f217f200640 1 -- 172.21.15.31:0/625029889 shutdown_connections 2024-03-21T10:34:20.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.932+0000 7f217f200640 1 --2- 172.21.15.31:0/625029889 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f21640a2f90 0x7f21640a5450 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:20.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.932+0000 7f217f200640 1 --2- 172.21.15.31:0/625029889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2180109770 0x7f21801a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:20.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.932+0000 7f217f200640 1 --2- 172.21.15.31:0/625029889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2180108db0 0x7f218019e3d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:20.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.932+0000 7f217f200640 1 --2- 172.21.15.31:0/625029889 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2180107bc0 0x7f218019de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:20.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.932+0000 7f217f200640 1 -- 172.21.15.31:0/625029889 >> 172.21.15.31:0/625029889 conn(0x7f2180076b60 msgr2=0x7f218010c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:20.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.932+0000 7f217f200640 1 -- 172.21.15.31:0/625029889 shutdown_connections 2024-03-21T10:34:20.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:20.932+0000 7f217f200640 1 -- 172.21.15.31:0/625029889 wait complete. 2024-03-21T10:34:20.952 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T10:29:44.553968+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-21T10:34:20.952 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T10:34:21.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.182+0000 7f3837000640 1 -- 172.21.15.31:0/3097977647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38301093c0 msgr2=0x7f383010b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:21.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.182+0000 7f3837000640 1 --2- 172.21.15.31:0/3097977647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38301093c0 0x7f383010b850 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3828009be0 tx=0x7f382802bf80 comp rx=0 tx=0).stop 2024-03-21T10:34:21.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.182+0000 7f3837000640 1 -- 172.21.15.31:0/3097977647 shutdown_connections 2024-03-21T10:34:21.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.182+0000 7f3837000640 1 --2- 172.21.15.31:0/3097977647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38301093c0 0x7f383010b850 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:21.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.182+0000 7f3837000640 1 --2- 172.21.15.31:0/3097977647 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3830074080 0x7f38300744e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:21.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.182+0000 7f3837000640 1 --2- 172.21.15.31:0/3097977647 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3830075660 0x7f3830073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:21.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.182+0000 7f3837000640 1 -- 172.21.15.31:0/3097977647 >> 172.21.15.31:0/3097977647 conn(0x7f38300fca80 msgr2=0x7f38300feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:21.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.182+0000 7f3837000640 1 -- 172.21.15.31:0/3097977647 shutdown_connections 2024-03-21T10:34:21.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.182+0000 7f3837000640 1 -- 172.21.15.31:0/3097977647 wait complete. 2024-03-21T10:34:21.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:20 smithi040 ceph-mon[36154]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi070=c;smithi070=f;smithi070=i;smithi070=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:34:21.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:20 smithi040 ceph-mon[36154]: Saving service mds.all spec with placement smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi070=c;smithi070=f;smithi070=i;smithi070=l;count:12 2024-03-21T10:34:21.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:20 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:21.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:20 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:34:21.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:20 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:21.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:20 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:21.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:20 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:21.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:20 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:21.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.183+0000 7f3837000640 1 Processor -- start 2024-03-21T10:34:21.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.183+0000 7f3837000640 1 -- start start 2024-03-21T10:34:21.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.184+0000 7f3837000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3830074080 0x7f3830199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:21.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.184+0000 7f3837000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3830075660 0x7f3830199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:21.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.184+0000 7f3837000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38301093c0 0x7f38301a0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:21.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.184+0000 7f3837000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383010f380 con 0x7f3830074080 2024-03-21T10:34:21.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.184+0000 7f3837000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383010f200 con 0x7f3830075660 2024-03-21T10:34:21.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.184+0000 7f3837000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383010f500 con 0x7f38301093c0 2024-03-21T10:34:21.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.184+0000 7f3835c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3830074080 0x7f3830199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:21.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.184+0000 7f3835200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3830075660 0x7f3830199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:21.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.184+0000 7f3835200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3830075660 0x7f3830199ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:45460/0 (socket says 172.21.15.31:45460) 2024-03-21T10:34:21.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.184+0000 7f3835200640 1 -- 172.21.15.31:0/2345105023 learned_addr learned my addr 172.21.15.31:0/2345105023 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:34:21.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.184+0000 7f3836600640 1 --2- 172.21.15.31:0/2345105023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38301093c0 0x7f38301a0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:21.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.185+0000 7f3835200640 1 -- 172.21.15.31:0/2345105023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38301093c0 msgr2=0x7f38301a0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:21.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.185+0000 7f3835200640 1 --2- 172.21.15.31:0/2345105023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38301093c0 0x7f38301a0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:21.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.185+0000 7f3835200640 1 -- 172.21.15.31:0/2345105023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3830074080 msgr2=0x7f3830199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:21.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.185+0000 7f3835200640 1 --2- 172.21.15.31:0/2345105023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3830074080 0x7f3830199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:21.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.185+0000 7f3835200640 1 -- 172.21.15.31:0/2345105023 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38301a1580 con 0x7f3830075660 2024-03-21T10:34:21.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.185+0000 7f3835c00640 1 --2- 172.21.15.31:0/2345105023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3830074080 0x7f3830199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:34:21.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.185+0000 7f3836600640 1 --2- 172.21.15.31:0/2345105023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38301093c0 0x7f38301a0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:34:21.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.185+0000 7f3835200640 1 --2- 172.21.15.31:0/2345105023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3830075660 0x7f3830199ea0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f381c00e9f0 tx=0x7f381c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:21.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.185+0000 7f382e000640 1 -- 172.21.15.31:0/2345105023 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381c00cd80 con 0x7f3830075660 2024-03-21T10:34:21.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.185+0000 7f382e000640 1 -- 172.21.15.31:0/2345105023 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f381c010910 con 0x7f3830075660 2024-03-21T10:34:21.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.185+0000 7f3837000640 1 -- 172.21.15.31:0/2345105023 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38301a1870 con 0x7f3830075660 2024-03-21T10:34:21.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.185+0000 7f382e000640 1 -- 172.21.15.31:0/2345105023 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381c018990 con 0x7f3830075660 2024-03-21T10:34:21.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.185+0000 7f3837000640 1 -- 172.21.15.31:0/2345105023 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38301a1dc0 con 0x7f3830075660 2024-03-21T10:34:21.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.187+0000 7f3837000640 1 -- 172.21.15.31:0/2345105023 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3830108180 con 0x7f3830075660 2024-03-21T10:34:21.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.189+0000 7f382e000640 1 -- 172.21.15.31:0/2345105023 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f381c010ac0 con 0x7f3830075660 2024-03-21T10:34:21.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.190+0000 7f382e000640 1 --2- 172.21.15.31:0/2345105023 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f38180a2f20 0x7f38180a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:21.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.191+0000 7f382e000640 1 -- 172.21.15.31:0/2345105023 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f381c0cf2e0 con 0x7f3830075660 2024-03-21T10:34:21.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.191+0000 7f3835c00640 1 --2- 172.21.15.31:0/2345105023 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f38180a2f20 0x7f38180a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:21.189 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:20 smithi070 ceph-mon[36330]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi070=c;smithi070=f;smithi070=i;smithi070=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:34:21.189 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:20 smithi070 ceph-mon[36330]: Saving service mds.all spec with placement smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi070=c;smithi070=f;smithi070=i;smithi070=l;count:12 2024-03-21T10:34:21.189 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:20 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:21.189 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:20 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:34:21.189 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:20 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:21.189 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:20 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:21.189 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:20 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:21.190 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:20 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:21.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.191+0000 7f3835c00640 1 --2- 172.21.15.31:0/2345105023 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f38180a2f20 0x7f38180a53e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f38240096c0 tx=0x7f3824009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:21.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.195+0000 7f382e000640 1 -- 172.21.15.31:0/2345105023 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f381c09bec0 con 0x7f3830075660 2024-03-21T10:34:21.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:20 smithi031 ceph-mon[31673]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi070=c;smithi070=f;smithi070=i;smithi070=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:34:21.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:20 smithi031 ceph-mon[31673]: Saving service mds.all spec with placement smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi070=c;smithi070=f;smithi070=i;smithi070=l;count:12 2024-03-21T10:34:21.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:20 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:21.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:20 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:34:21.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:20 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:21.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:20 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:21.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:20 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:21.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:20 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:21.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.397+0000 7f3837000640 1 -- 172.21.15.31:0/2345105023 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f38300622a0 con 0x7f3830075660 2024-03-21T10:34:21.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.398+0000 7f382e000640 1 -- 172.21.15.31:0/2345105023 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f381c09ff60 con 0x7f3830075660 2024-03-21T10:34:21.396 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-03-21T10:34:21.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.400+0000 7f3837000640 1 -- 172.21.15.31:0/2345105023 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f38180a2f20 msgr2=0x7f38180a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:21.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.400+0000 7f3837000640 1 --2- 172.21.15.31:0/2345105023 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f38180a2f20 0x7f38180a53e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f38240096c0 tx=0x7f3824009260 comp rx=0 tx=0).stop 2024-03-21T10:34:21.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.401+0000 7f3837000640 1 -- 172.21.15.31:0/2345105023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3830075660 msgr2=0x7f3830199ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:21.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.401+0000 7f3837000640 1 --2- 172.21.15.31:0/2345105023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3830075660 0x7f3830199ea0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f381c00e9f0 tx=0x7f381c00eeb0 comp rx=0 tx=0).stop 2024-03-21T10:34:21.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.401+0000 7f3837000640 1 -- 172.21.15.31:0/2345105023 shutdown_connections 2024-03-21T10:34:21.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.401+0000 7f3837000640 1 --2- 172.21.15.31:0/2345105023 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f38180a2f20 0x7f38180a53e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:21.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.401+0000 7f3837000640 1 --2- 172.21.15.31:0/2345105023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38301093c0 0x7f38301a0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:21.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.401+0000 7f3837000640 1 --2- 172.21.15.31:0/2345105023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3830075660 0x7f3830199ea0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:21.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.401+0000 7f3837000640 1 --2- 172.21.15.31:0/2345105023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3830074080 0x7f3830199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:21.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.401+0000 7f3837000640 1 -- 172.21.15.31:0/2345105023 >> 172.21.15.31:0/2345105023 conn(0x7f38300fca80 msgr2=0x7f3830105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:21.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.401+0000 7f3837000640 1 -- 172.21.15.31:0/2345105023 shutdown_connections 2024-03-21T10:34:21.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.401+0000 7f3837000640 1 -- 172.21.15.31:0/2345105023 wait complete. 2024-03-21T10:34:21.418 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:34:21.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.644+0000 7f4144800640 1 -- 172.21.15.31:0/1649828331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f413c104a80 msgr2=0x7f413c106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:21.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.644+0000 7f4144800640 1 --2- 172.21.15.31:0/1649828331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f413c104a80 0x7f413c106e70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4128009900 tx=0x7f412802bc30 comp rx=0 tx=0).stop 2024-03-21T10:34:21.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.644+0000 7f4144800640 1 -- 172.21.15.31:0/1649828331 shutdown_connections 2024-03-21T10:34:21.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.644+0000 7f4144800640 1 --2- 172.21.15.31:0/1649828331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f413c1073b0 0x7f413c109800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:21.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.644+0000 7f4144800640 1 --2- 172.21.15.31:0/1649828331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f413c104a80 0x7f413c106e70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:21.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.644+0000 7f4144800640 1 --2- 172.21.15.31:0/1649828331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f413c102150 0x7f413c104540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:21.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.644+0000 7f4144800640 1 -- 172.21.15.31:0/1649828331 >> 172.21.15.31:0/1649828331 conn(0x7f413c0fbb40 msgr2=0x7f413c0fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:21.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.644+0000 7f4144800640 1 -- 172.21.15.31:0/1649828331 shutdown_connections 2024-03-21T10:34:21.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.644+0000 7f4144800640 1 -- 172.21.15.31:0/1649828331 wait complete. 2024-03-21T10:34:21.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.645+0000 7f4144800640 1 Processor -- start 2024-03-21T10:34:21.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.645+0000 7f4144800640 1 -- start start 2024-03-21T10:34:21.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.646+0000 7f4144800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f413c102150 0x7f413c101ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:21.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.646+0000 7f4144800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f413c104a80 0x7f413c100100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:21.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.646+0000 7f4144800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f413c1073b0 0x7f413c100640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:21.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.646+0000 7f4144800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413c10d390 con 0x7f413c1073b0 2024-03-21T10:34:21.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.646+0000 7f4144800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413c10d210 con 0x7f413c104a80 2024-03-21T10:34:21.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.646+0000 7f4144800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413c10d510 con 0x7f413c102150 2024-03-21T10:34:21.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.646+0000 7f4141e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f413c102150 0x7f413c101ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:21.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.646+0000 7f4142800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f413c1073b0 0x7f413c100640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:21.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.646+0000 7f4141400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f413c104a80 0x7f413c100100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:21.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.646+0000 7f4141e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f413c102150 0x7f413c101ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:35872/0 (socket says 172.21.15.31:35872) 2024-03-21T10:34:21.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.646+0000 7f4141e00640 1 -- 172.21.15.31:0/1790405038 learned_addr learned my addr 172.21.15.31:0/1790405038 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:34:21.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.646+0000 7f4142800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f413c1073b0 0x7f413c100640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40068/0 (socket says 172.21.15.31:40068) 2024-03-21T10:34:21.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.646+0000 7f4141e00640 1 -- 172.21.15.31:0/1790405038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f413c104a80 msgr2=0x7f413c100100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:21.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.646+0000 7f4141e00640 1 --2- 172.21.15.31:0/1790405038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f413c104a80 0x7f413c100100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:21.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.646+0000 7f4141e00640 1 -- 172.21.15.31:0/1790405038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f413c1073b0 msgr2=0x7f413c100640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:21.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.646+0000 7f4141e00640 1 --2- 172.21.15.31:0/1790405038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f413c1073b0 0x7f413c100640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:21.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.646+0000 7f4141e00640 1 -- 172.21.15.31:0/1790405038 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413c100ed0 con 0x7f413c102150 2024-03-21T10:34:21.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.647+0000 7f4141e00640 1 --2- 172.21.15.31:0/1790405038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f413c102150 0x7f413c101ac0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f413000e9f0 tx=0x7f413000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:21.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.647+0000 7f4136a00640 1 -- 172.21.15.31:0/1790405038 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413000cd80 con 0x7f413c102150 2024-03-21T10:34:21.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.647+0000 7f4136a00640 1 -- 172.21.15.31:0/1790405038 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4130010910 con 0x7f413c102150 2024-03-21T10:34:21.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.647+0000 7f4136a00640 1 -- 172.21.15.31:0/1790405038 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4130018990 con 0x7f413c102150 2024-03-21T10:34:21.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.647+0000 7f4144800640 1 -- 172.21.15.31:0/1790405038 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f413c1a1ae0 con 0x7f413c102150 2024-03-21T10:34:21.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.647+0000 7f4144800640 1 -- 172.21.15.31:0/1790405038 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f413c1a2050 con 0x7f413c102150 2024-03-21T10:34:21.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.648+0000 7f4144800640 1 -- 172.21.15.31:0/1790405038 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f413c0fd630 con 0x7f413c102150 2024-03-21T10:34:21.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.650+0000 7f4136a00640 1 -- 172.21.15.31:0/1790405038 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f4130010ac0 con 0x7f413c102150 2024-03-21T10:34:21.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.651+0000 7f4136a00640 1 --2- 172.21.15.31:0/1790405038 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f41240a2fa0 0x7f41240a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:21.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.651+0000 7f4136a00640 1 -- 172.21.15.31:0/1790405038 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f41300cf260 con 0x7f413c102150 2024-03-21T10:34:21.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.652+0000 7f4141400640 1 --2- 172.21.15.31:0/1790405038 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f41240a2fa0 0x7f41240a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:21.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.652+0000 7f4141400640 1 --2- 172.21.15.31:0/1790405038 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f41240a2fa0 0x7f41240a5460 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f412802c170 tx=0x7f412802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:21.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.655+0000 7f4136a00640 1 -- 172.21.15.31:0/1790405038 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f413009be40 con 0x7f413c102150 2024-03-21T10:34:21.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.902+0000 7f4144800640 1 -- 172.21.15.31:0/1790405038 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f413c02d070 con 0x7f413c102150 2024-03-21T10:34:21.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.903+0000 7f4136a00640 1 -- 172.21.15.31:0/1790405038 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7f413009fee0 con 0x7f413c102150 2024-03-21T10:34:21.901 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:34:21.901 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-21T10:34:21.901 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 1 2024-03-21T10:34:21.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.906+0000 7f4144800640 1 -- 172.21.15.31:0/1790405038 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f41240a2fa0 msgr2=0x7f41240a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:21.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.906+0000 7f4144800640 1 --2- 172.21.15.31:0/1790405038 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f41240a2fa0 0x7f41240a5460 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f412802c170 tx=0x7f412802df90 comp rx=0 tx=0).stop 2024-03-21T10:34:21.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.906+0000 7f4144800640 1 -- 172.21.15.31:0/1790405038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f413c102150 msgr2=0x7f413c101ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:21.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.906+0000 7f4144800640 1 --2- 172.21.15.31:0/1790405038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f413c102150 0x7f413c101ac0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f413000e9f0 tx=0x7f413000eeb0 comp rx=0 tx=0).stop 2024-03-21T10:34:21.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.906+0000 7f4144800640 1 -- 172.21.15.31:0/1790405038 shutdown_connections 2024-03-21T10:34:21.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.906+0000 7f4144800640 1 --2- 172.21.15.31:0/1790405038 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f41240a2fa0 0x7f41240a5460 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:21.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.906+0000 7f4144800640 1 --2- 172.21.15.31:0/1790405038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f413c1073b0 0x7f413c100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:21.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.906+0000 7f4144800640 1 --2- 172.21.15.31:0/1790405038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f413c104a80 0x7f413c100100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:21.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.906+0000 7f4144800640 1 --2- 172.21.15.31:0/1790405038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f413c102150 0x7f413c101ac0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:21.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.906+0000 7f4144800640 1 -- 172.21.15.31:0/1790405038 >> 172.21.15.31:0/1790405038 conn(0x7f413c0fbb40 msgr2=0x7f413c102ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:21.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.906+0000 7f4144800640 1 -- 172.21.15.31:0/1790405038 shutdown_connections 2024-03-21T10:34:21.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:21.906+0000 7f4144800640 1 -- 172.21.15.31:0/1790405038 wait complete. 2024-03-21T10:34:22.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:21 smithi040 ceph-mon[36154]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:22.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:21 smithi040 ceph-mon[36154]: Deploying daemon mds.l on smithi070 2024-03-21T10:34:22.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:21 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/625029889' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:34:22.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:21 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2345105023' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:34:22.192 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:21 smithi070 ceph-mon[36330]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:22.192 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:21 smithi070 ceph-mon[36330]: Deploying daemon mds.l on smithi070 2024-03-21T10:34:22.192 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:21 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/625029889' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:34:22.192 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:21 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2345105023' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:34:22.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:21 smithi031 ceph-mon[31673]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:22.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:21 smithi031 ceph-mon[31673]: Deploying daemon mds.l on smithi070 2024-03-21T10:34:22.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:21 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/625029889' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:34:22.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:21 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2345105023' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:34:23.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:22 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1790405038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:23.074 INFO:journalctl@ceph.mds.l.smithi070.stdout:Mar 21 10:34:23 smithi070 podman[66400]: 2024-03-21 10:34:23.059903077 +0000 UTC m=+0.135780140 container create 0f729b3e58ba0e4aaa0e36ea461dedc41a5215d250eb1d786f074ef07801da03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-l, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-21T10:34:23.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:22 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1790405038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:23.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:22 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1790405038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:23.574 INFO:journalctl@ceph.mds.l.smithi070.stdout:Mar 21 10:34:23 smithi070 podman[66400]: 2024-03-21 10:34:23.296218011 +0000 UTC m=+0.372095073 container init 0f729b3e58ba0e4aaa0e36ea461dedc41a5215d250eb1d786f074ef07801da03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-l, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T10:34:23.574 INFO:journalctl@ceph.mds.l.smithi070.stdout:Mar 21 10:34:23 smithi070 podman[66400]: 2024-03-21 10:34:23.298701251 +0000 UTC m=+0.374578295 container start 0f729b3e58ba0e4aaa0e36ea461dedc41a5215d250eb1d786f074ef07801da03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-l, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.license=GPLv2) 2024-03-21T10:34:23.574 INFO:journalctl@ceph.mds.l.smithi070.stdout:Mar 21 10:34:23 smithi070 ceph-mds[66419]: -- 172.21.15.70:0/4064466661 <== mon.1 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556704df7a40 con 0x556704dcb000 2024-03-21T10:34:23.574 INFO:journalctl@ceph.mds.l.smithi070.stdout:Mar 21 10:34:23 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-l[66415]: starting mds.l at 2024-03-21T10:34:23.575 INFO:journalctl@ceph.mds.l.smithi070.stdout:Mar 21 10:34:23 smithi070 bash[66400]: 0f729b3e58ba0e4aaa0e36ea461dedc41a5215d250eb1d786f074ef07801da03 2024-03-21T10:34:23.575 INFO:journalctl@ceph.mds.l.smithi070.stdout:Mar 21 10:34:23 smithi070 systemd[1]: Started Ceph mds.l for 3847408c-e76d-11ee-95cd-87774f69a715. 2024-03-21T10:34:23.926 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:34:24.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.161+0000 7fb266e00640 1 -- 172.21.15.31:0/2060747179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb260104b50 msgr2=0x7fb260104fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:24.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.161+0000 7fb266e00640 1 --2- 172.21.15.31:0/2060747179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb260104b50 0x7fb260104fd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb248009900 tx=0x7fb24802bc30 comp rx=0 tx=0).stop 2024-03-21T10:34:24.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.162+0000 7fb266e00640 1 -- 172.21.15.31:0/2060747179 shutdown_connections 2024-03-21T10:34:24.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.162+0000 7fb266e00640 1 --2- 172.21.15.31:0/2060747179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb260105510 0x7fb26010bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:24.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.162+0000 7fb266e00640 1 --2- 172.21.15.31:0/2060747179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb260104b50 0x7fb260104fd0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:24.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.162+0000 7fb266e00640 1 --2- 172.21.15.31:0/2060747179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb260103960 0x7fb260103d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:24.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.162+0000 7fb266e00640 1 -- 172.21.15.31:0/2060747179 >> 172.21.15.31:0/2060747179 conn(0x7fb2600feeb0 msgr2=0x7fb260101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:24.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.162+0000 7fb266e00640 1 -- 172.21.15.31:0/2060747179 shutdown_connections 2024-03-21T10:34:24.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.162+0000 7fb266e00640 1 -- 172.21.15.31:0/2060747179 wait complete. 2024-03-21T10:34:24.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.163+0000 7fb266e00640 1 Processor -- start 2024-03-21T10:34:24.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.163+0000 7fb266e00640 1 -- start start 2024-03-21T10:34:24.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.164+0000 7fb266e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb260103960 0x7fb260199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:24.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.164+0000 7fb266e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb260104b50 0x7fb26019a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:24.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.164+0000 7fb266e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb260105510 0x7fb2601a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:24.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.164+0000 7fb266e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26010f570 con 0x7fb260104b50 2024-03-21T10:34:24.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.164+0000 7fb266e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26010f3f0 con 0x7fb260105510 2024-03-21T10:34:24.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.164+0000 7fb266e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26010f6f0 con 0x7fb260103960 2024-03-21T10:34:24.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.164+0000 7fb25fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb260103960 0x7fb260199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:24.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.164+0000 7fb25fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb260103960 0x7fb260199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:35884/0 (socket says 172.21.15.31:35884) 2024-03-21T10:34:24.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.164+0000 7fb25fe00640 1 -- 172.21.15.31:0/2366547726 learned_addr learned my addr 172.21.15.31:0/2366547726 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:34:24.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.164+0000 7fb25f400640 1 --2- 172.21.15.31:0/2366547726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb260104b50 0x7fb26019a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:24.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.164+0000 7fb264e00640 1 --2- 172.21.15.31:0/2366547726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb260105510 0x7fb2601a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:24.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.164+0000 7fb25fe00640 1 -- 172.21.15.31:0/2366547726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb260105510 msgr2=0x7fb2601a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:24.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.164+0000 7fb25fe00640 1 --2- 172.21.15.31:0/2366547726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb260105510 0x7fb2601a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:24.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.165+0000 7fb25fe00640 1 -- 172.21.15.31:0/2366547726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb260104b50 msgr2=0x7fb26019a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:24.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.165+0000 7fb25fe00640 1 --2- 172.21.15.31:0/2366547726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb260104b50 0x7fb26019a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:24.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.165+0000 7fb25fe00640 1 -- 172.21.15.31:0/2366547726 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2601a17e0 con 0x7fb260103960 2024-03-21T10:34:24.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.165+0000 7fb25fe00640 1 --2- 172.21.15.31:0/2366547726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb260103960 0x7fb260199bc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb25000eb70 tx=0x7fb25000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:24.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.165+0000 7fb25cc00640 1 -- 172.21.15.31:0/2366547726 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb250003b70 con 0x7fb260103960 2024-03-21T10:34:24.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.165+0000 7fb25cc00640 1 -- 172.21.15.31:0/2366547726 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb25000ee70 con 0x7fb260103960 2024-03-21T10:34:24.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.165+0000 7fb266e00640 1 -- 172.21.15.31:0/2366547726 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2601a1ad0 con 0x7fb260103960 2024-03-21T10:34:24.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.165+0000 7fb25cc00640 1 -- 172.21.15.31:0/2366547726 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb250010830 con 0x7fb260103960 2024-03-21T10:34:24.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.165+0000 7fb266e00640 1 -- 172.21.15.31:0/2366547726 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2601a1fa0 con 0x7fb260103960 2024-03-21T10:34:24.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.166+0000 7fb266e00640 1 -- 172.21.15.31:0/2366547726 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb26010ac80 con 0x7fb260103960 2024-03-21T10:34:24.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.168+0000 7fb25cc00640 1 -- 172.21.15.31:0/2366547726 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fb250004020 con 0x7fb260103960 2024-03-21T10:34:24.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.169+0000 7fb25cc00640 1 --2- 172.21.15.31:0/2366547726 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb2440a2f20 0x7fb2440a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:24.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.169+0000 7fb25cc00640 1 -- 172.21.15.31:0/2366547726 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb2500cf660 con 0x7fb260103960 2024-03-21T10:34:24.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.170+0000 7fb25f400640 1 --2- 172.21.15.31:0/2366547726 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb2440a2f20 0x7fb2440a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:24.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.171+0000 7fb25f400640 1 --2- 172.21.15.31:0/2366547726 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb2440a2f20 0x7fb2440a53e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb24802c7b0 tx=0x7fb248005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:24.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.174+0000 7fb25cc00640 1 -- 172.21.15.31:0/2366547726 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb25009c890 con 0x7fb260103960 2024-03-21T10:34:24.172 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:23 smithi040 ceph-mon[36154]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:24.172 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:23 smithi040 ceph-mon[36154]: mds.? [v2:172.21.15.70:6832/4158670810,v1:172.21.15.70:6833/4158670810] up:boot 2024-03-21T10:34:24.173 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:23 smithi040 ceph-mon[36154]: fsmap 1 up:standby 2024-03-21T10:34:24.173 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:23 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-21T10:34:24.173 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:23 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:24.173 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:23 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:24.173 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:23 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:24.173 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:23 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:24.173 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:23 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:24.173 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:23 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:24.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:23 smithi031 ceph-mon[31673]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:24.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:23 smithi031 ceph-mon[31673]: mds.? [v2:172.21.15.70:6832/4158670810,v1:172.21.15.70:6833/4158670810] up:boot 2024-03-21T10:34:24.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:23 smithi031 ceph-mon[31673]: fsmap 1 up:standby 2024-03-21T10:34:24.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:23 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-21T10:34:24.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:23 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:24.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:23 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:24.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:23 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:24.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:23 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:24.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:23 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:24.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:23 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:24.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:23 smithi070 ceph-mon[36330]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:24.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:23 smithi070 ceph-mon[36330]: mds.? [v2:172.21.15.70:6832/4158670810,v1:172.21.15.70:6833/4158670810] up:boot 2024-03-21T10:34:24.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:23 smithi070 ceph-mon[36330]: fsmap 1 up:standby 2024-03-21T10:34:24.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:23 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-21T10:34:24.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:23 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:24.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:23 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:24.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:23 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:24.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:23 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:24.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:23 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:24.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:23 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:24.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.418+0000 7fb266e00640 1 -- 172.21.15.31:0/2366547726 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2600622a0 con 0x7fb260103960 2024-03-21T10:34:24.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.419+0000 7fb25cc00640 1 -- 172.21.15.31:0/2366547726 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1325 (secure 0 0 0) 0x7fb25009c190 con 0x7fb260103960 2024-03-21T10:34:24.417 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:34:24.417 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-21T10:34:24.418 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 2 2024-03-21T10:34:24.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.421+0000 7fb266e00640 1 -- 172.21.15.31:0/2366547726 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb2440a2f20 msgr2=0x7fb2440a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:24.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.421+0000 7fb266e00640 1 --2- 172.21.15.31:0/2366547726 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb2440a2f20 0x7fb2440a53e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb24802c7b0 tx=0x7fb248005fd0 comp rx=0 tx=0).stop 2024-03-21T10:34:24.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.421+0000 7fb266e00640 1 -- 172.21.15.31:0/2366547726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb260103960 msgr2=0x7fb260199bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:24.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.421+0000 7fb266e00640 1 --2- 172.21.15.31:0/2366547726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb260103960 0x7fb260199bc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb25000eb70 tx=0x7fb25000c6a0 comp rx=0 tx=0).stop 2024-03-21T10:34:24.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.422+0000 7fb266e00640 1 -- 172.21.15.31:0/2366547726 shutdown_connections 2024-03-21T10:34:24.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.422+0000 7fb266e00640 1 --2- 172.21.15.31:0/2366547726 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb2440a2f20 0x7fb2440a53e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:24.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.422+0000 7fb266e00640 1 --2- 172.21.15.31:0/2366547726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb260105510 0x7fb2601a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:24.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.422+0000 7fb266e00640 1 --2- 172.21.15.31:0/2366547726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb260104b50 0x7fb26019a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:24.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.422+0000 7fb266e00640 1 --2- 172.21.15.31:0/2366547726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb260103960 0x7fb260199bc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:24.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.422+0000 7fb266e00640 1 -- 172.21.15.31:0/2366547726 >> 172.21.15.31:0/2366547726 conn(0x7fb2600feeb0 msgr2=0x7fb260107e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:24.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.422+0000 7fb266e00640 1 -- 172.21.15.31:0/2366547726 shutdown_connections 2024-03-21T10:34:24.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:24.422+0000 7fb266e00640 1 -- 172.21.15.31:0/2366547726 wait complete. 2024-03-21T10:34:25.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:24 smithi040 ceph-mon[36154]: Deploying daemon mds.k on smithi040 2024-03-21T10:34:25.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:24 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2366547726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:25.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:24 smithi031 ceph-mon[31673]: Deploying daemon mds.k on smithi040 2024-03-21T10:34:25.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:24 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2366547726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:25.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:24 smithi070 ceph-mon[36330]: Deploying daemon mds.k on smithi040 2024-03-21T10:34:25.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:24 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2366547726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:26.177 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:25 smithi040 ceph-mon[36154]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:26.177 INFO:journalctl@ceph.mds.k.smithi040.stdout:Mar 21 10:34:25 smithi040 systemd[1]: Starting Ceph mds.k for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:34:26.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:25 smithi031 ceph-mon[31673]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:26.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:25 smithi070 ceph-mon[36330]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:26.429 INFO:journalctl@ceph.mds.k.smithi040.stdout:Mar 21 10:34:26 smithi040 podman[65493]: 2024-03-21 10:34:26.081567984 +0000 UTC m=+0.018873014 image pull 237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232 2024-03-21T10:34:26.429 INFO:journalctl@ceph.mds.k.smithi040.stdout:Mar 21 10:34:26 smithi040 podman[65493]: 2024-03-21 10:34:26.200553542 +0000 UTC m=+0.137858581 container create 2858ae883013fbfe5841c3e00e41102dd97ec8068d2b02cc6ebb79b2e49f96db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-k, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-21T10:34:26.429 INFO:journalctl@ceph.mds.k.smithi040.stdout:Mar 21 10:34:26 smithi040 podman[65493]: 2024-03-21 10:34:26.42831254 +0000 UTC m=+0.365617565 container init 2858ae883013fbfe5841c3e00e41102dd97ec8068d2b02cc6ebb79b2e49f96db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-k, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-21T10:34:26.429 INFO:journalctl@ceph.mds.k.smithi040.stdout:Mar 21 10:34:26 smithi040 podman[65493]: 2024-03-21 10:34:26.431438313 +0000 UTC m=+0.368743330 container start 2858ae883013fbfe5841c3e00e41102dd97ec8068d2b02cc6ebb79b2e49f96db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-k, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-21T10:34:26.443 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:34:26.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.679+0000 7f89eec00640 1 -- 172.21.15.31:0/1089648886 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89e8101730 msgr2=0x7f89e8101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:26.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.679+0000 7f89eec00640 1 --2- 172.21.15.31:0/1089648886 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89e8101730 0x7f89e8101b50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f89d8009900 tx=0x7f89d802bc30 comp rx=0 tx=0).stop 2024-03-21T10:34:26.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.679+0000 7f89eec00640 1 -- 172.21.15.31:0/1089648886 shutdown_connections 2024-03-21T10:34:26.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.679+0000 7f89eec00640 1 --2- 172.21.15.31:0/1089648886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89e81032e0 0x7f89e8109b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:26.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.679+0000 7f89eec00640 1 --2- 172.21.15.31:0/1089648886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89e8102920 0x7f89e8102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:26.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.679+0000 7f89eec00640 1 --2- 172.21.15.31:0/1089648886 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89e8101730 0x7f89e8101b50 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:26.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.679+0000 7f89eec00640 1 -- 172.21.15.31:0/1089648886 >> 172.21.15.31:0/1089648886 conn(0x7f89e80fccc0 msgr2=0x7f89e80ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:26.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.679+0000 7f89eec00640 1 -- 172.21.15.31:0/1089648886 shutdown_connections 2024-03-21T10:34:26.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.679+0000 7f89eec00640 1 -- 172.21.15.31:0/1089648886 wait complete. 2024-03-21T10:34:26.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.680+0000 7f89eec00640 1 Processor -- start 2024-03-21T10:34:26.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.681+0000 7f89eec00640 1 -- start start 2024-03-21T10:34:26.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.681+0000 7f89eec00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89e8101730 0x7f89e8197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:26.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.681+0000 7f89eec00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89e8102920 0x7f89e8197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:26.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.681+0000 7f89eec00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89e81032e0 0x7f89e819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:26.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.681+0000 7f89eec00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e810d360 con 0x7f89e8101730 2024-03-21T10:34:26.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.681+0000 7f89eec00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e810d1e0 con 0x7f89e8102920 2024-03-21T10:34:26.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.681+0000 7f89eec00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e810d4e0 con 0x7f89e81032e0 2024-03-21T10:34:26.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.681+0000 7f89e7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89e8101730 0x7f89e8197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:26.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.681+0000 7f89e7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89e8102920 0x7f89e8197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:26.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.681+0000 7f89e7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89e8102920 0x7f89e8197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:45504/0 (socket says 172.21.15.31:45504) 2024-03-21T10:34:26.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.681+0000 7f89e7400640 1 -- 172.21.15.31:0/15063569 learned_addr learned my addr 172.21.15.31:0/15063569 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:34:26.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.681+0000 7f89e7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89e8101730 0x7f89e8197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40104/0 (socket says 172.21.15.31:40104) 2024-03-21T10:34:26.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.681+0000 7f89ecc00640 1 --2- 172.21.15.31:0/15063569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89e81032e0 0x7f89e819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:26.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.682+0000 7f89e7400640 1 -- 172.21.15.31:0/15063569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89e81032e0 msgr2=0x7f89e819efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:26.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.682+0000 7f89e7400640 1 --2- 172.21.15.31:0/15063569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89e81032e0 0x7f89e819efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:26.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.682+0000 7f89e7400640 1 -- 172.21.15.31:0/15063569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89e8101730 msgr2=0x7f89e8197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:26.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.682+0000 7f89e7400640 1 --2- 172.21.15.31:0/15063569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89e8101730 0x7f89e8197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:26.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.682+0000 7f89e7400640 1 -- 172.21.15.31:0/15063569 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e819f620 con 0x7f89e8102920 2024-03-21T10:34:26.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.682+0000 7f89e7e00640 1 --2- 172.21.15.31:0/15063569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89e8101730 0x7f89e8197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:34:26.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.682+0000 7f89e7400640 1 --2- 172.21.15.31:0/15063569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89e8102920 0x7f89e8197f40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f89d400e9f0 tx=0x7f89d400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:26.682 INFO:journalctl@ceph.mds.k.smithi040.stdout:Mar 21 10:34:26 smithi040 ceph-mds[65513]: -- 172.21.15.40:0/4153304723 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a032b25a40 con 0x55a032af9000 2024-03-21T10:34:26.682 INFO:journalctl@ceph.mds.k.smithi040.stdout:Mar 21 10:34:26 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-k[65509]: starting mds.k at 2024-03-21T10:34:26.682 INFO:journalctl@ceph.mds.k.smithi040.stdout:Mar 21 10:34:26 smithi040 bash[65493]: 2858ae883013fbfe5841c3e00e41102dd97ec8068d2b02cc6ebb79b2e49f96db 2024-03-21T10:34:26.682 INFO:journalctl@ceph.mds.k.smithi040.stdout:Mar 21 10:34:26 smithi040 systemd[1]: Started Ceph mds.k for 3847408c-e76d-11ee-95cd-87774f69a715. 2024-03-21T10:34:26.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.682+0000 7f89e4c00640 1 -- 172.21.15.31:0/15063569 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d400cd80 con 0x7f89e8102920 2024-03-21T10:34:26.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.682+0000 7f89e4c00640 1 -- 172.21.15.31:0/15063569 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f89d4010910 con 0x7f89e8102920 2024-03-21T10:34:26.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.682+0000 7f89eec00640 1 -- 172.21.15.31:0/15063569 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e819f910 con 0x7f89e8102920 2024-03-21T10:34:26.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.682+0000 7f89e4c00640 1 -- 172.21.15.31:0/15063569 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d4018990 con 0x7f89e8102920 2024-03-21T10:34:26.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.683+0000 7f89eec00640 1 -- 172.21.15.31:0/15063569 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e819feb0 con 0x7f89e8102920 2024-03-21T10:34:26.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.684+0000 7f89eec00640 1 -- 172.21.15.31:0/15063569 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89e8108a50 con 0x7f89e8102920 2024-03-21T10:34:26.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.686+0000 7f89e4c00640 1 -- 172.21.15.31:0/15063569 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f89d4010ac0 con 0x7f89e8102920 2024-03-21T10:34:26.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.688+0000 7f89e4c00640 1 --2- 172.21.15.31:0/15063569 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f89cc0a2fa0 0x7f89cc0a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:26.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.688+0000 7f89e4c00640 1 -- 172.21.15.31:0/15063569 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f89d40cf250 con 0x7f89e8102920 2024-03-21T10:34:26.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.688+0000 7f89e7e00640 1 --2- 172.21.15.31:0/15063569 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f89cc0a2fa0 0x7f89cc0a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:26.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.689+0000 7f89e7e00640 1 --2- 172.21.15.31:0/15063569 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f89cc0a2fa0 0x7f89cc0a5460 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f89d802c170 tx=0x7f89d802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:26.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.692+0000 7f89e4c00640 1 -- 172.21.15.31:0/15063569 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f89d409be30 con 0x7f89e8102920 2024-03-21T10:34:26.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.954+0000 7f89eec00640 1 -- 172.21.15.31:0/15063569 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89e802d070 con 0x7f89e8102920 2024-03-21T10:34:26.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.955+0000 7f89e4c00640 1 -- 172.21.15.31:0/15063569 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2116 (secure 0 0 0) 0x7f89d409fed0 con 0x7f89e8102920 2024-03-21T10:34:26.954 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:34:26.954 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-21T10:34:26.954 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 3 2024-03-21T10:34:26.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.958+0000 7f89eec00640 1 -- 172.21.15.31:0/15063569 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f89cc0a2fa0 msgr2=0x7f89cc0a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:26.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.958+0000 7f89eec00640 1 --2- 172.21.15.31:0/15063569 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f89cc0a2fa0 0x7f89cc0a5460 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f89d802c170 tx=0x7f89d802df90 comp rx=0 tx=0).stop 2024-03-21T10:34:26.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.958+0000 7f89eec00640 1 -- 172.21.15.31:0/15063569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89e8102920 msgr2=0x7f89e8197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:26.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.958+0000 7f89eec00640 1 --2- 172.21.15.31:0/15063569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89e8102920 0x7f89e8197f40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f89d400e9f0 tx=0x7f89d400eeb0 comp rx=0 tx=0).stop 2024-03-21T10:34:26.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.958+0000 7f89eec00640 1 -- 172.21.15.31:0/15063569 shutdown_connections 2024-03-21T10:34:26.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.958+0000 7f89eec00640 1 --2- 172.21.15.31:0/15063569 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f89cc0a2fa0 0x7f89cc0a5460 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:26.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.958+0000 7f89eec00640 1 --2- 172.21.15.31:0/15063569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89e81032e0 0x7f89e819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:26.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.958+0000 7f89eec00640 1 --2- 172.21.15.31:0/15063569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89e8102920 0x7f89e8197f40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:26.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.958+0000 7f89eec00640 1 --2- 172.21.15.31:0/15063569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89e8101730 0x7f89e8197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:26.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.958+0000 7f89eec00640 1 -- 172.21.15.31:0/15063569 >> 172.21.15.31:0/15063569 conn(0x7f89e80fccc0 msgr2=0x7f89e8105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:26.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.958+0000 7f89eec00640 1 -- 172.21.15.31:0/15063569 shutdown_connections 2024-03-21T10:34:26.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:26.959+0000 7f89eec00640 1 -- 172.21.15.31:0/15063569 wait complete. 2024-03-21T10:34:27.873 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:27 smithi040 ceph-mon[36154]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:27.873 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:27 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:27.873 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:27 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:27.873 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:27 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:27.874 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:27 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:27.874 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:27 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:27.874 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:27 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:27.874 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:27 smithi040 ceph-mon[36154]: Deploying daemon mds.j on smithi031 2024-03-21T10:34:27.874 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:27 smithi040 ceph-mon[36154]: mds.? [v2:172.21.15.40:6832/2806510034,v1:172.21.15.40:6833/2806510034] up:boot 2024-03-21T10:34:27.874 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:27 smithi040 ceph-mon[36154]: fsmap 2 up:standby 2024-03-21T10:34:27.874 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:27 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-21T10:34:27.874 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:27 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/15063569' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:27.889 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:27 smithi031 ceph-mon[31673]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:27.889 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:27 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:27.889 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:27 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:27.889 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:27 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:27.890 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:27 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:27.890 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:27 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:27.890 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:27 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:27.890 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:27 smithi031 ceph-mon[31673]: Deploying daemon mds.j on smithi031 2024-03-21T10:34:27.890 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:27 smithi031 ceph-mon[31673]: mds.? [v2:172.21.15.40:6832/2806510034,v1:172.21.15.40:6833/2806510034] up:boot 2024-03-21T10:34:27.890 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:27 smithi031 ceph-mon[31673]: fsmap 2 up:standby 2024-03-21T10:34:27.890 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:27 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-21T10:34:27.890 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:27 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/15063569' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:28.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:27 smithi070 ceph-mon[36330]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:28.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:27 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:28.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:27 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:28.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:27 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:28.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:27 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:28.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:27 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:28.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:27 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:28.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:27 smithi070 ceph-mon[36330]: Deploying daemon mds.j on smithi031 2024-03-21T10:34:28.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:27 smithi070 ceph-mon[36330]: mds.? [v2:172.21.15.40:6832/2806510034,v1:172.21.15.40:6833/2806510034] up:boot 2024-03-21T10:34:28.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:27 smithi070 ceph-mon[36330]: fsmap 2 up:standby 2024-03-21T10:34:28.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:27 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-21T10:34:28.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:27 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/15063569' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:28.978 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:34:29.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.197+0000 7f9344e00640 1 -- 172.21.15.31:0/1534185138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93401020f0 msgr2=0x7f93401044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:29.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.197+0000 7f9344e00640 1 --2- 172.21.15.31:0/1534185138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93401020f0 0x7f93401044e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9330009900 tx=0x7f933002bc30 comp rx=0 tx=0).stop 2024-03-21T10:34:29.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.197+0000 7f9344e00640 1 -- 172.21.15.31:0/1534185138 shutdown_connections 2024-03-21T10:34:29.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.197+0000 7f9344e00640 1 --2- 172.21.15.31:0/1534185138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9340107350 0x7f9340109740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:29.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.197+0000 7f9344e00640 1 --2- 172.21.15.31:0/1534185138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9340104a20 0x7f9340106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:29.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.197+0000 7f9344e00640 1 --2- 172.21.15.31:0/1534185138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93401020f0 0x7f93401044e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:29.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.197+0000 7f9344e00640 1 -- 172.21.15.31:0/1534185138 >> 172.21.15.31:0/1534185138 conn(0x7f93400fba20 msgr2=0x7f93400fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:29.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.197+0000 7f9344e00640 1 -- 172.21.15.31:0/1534185138 shutdown_connections 2024-03-21T10:34:29.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.197+0000 7f9344e00640 1 -- 172.21.15.31:0/1534185138 wait complete. 2024-03-21T10:34:29.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.198+0000 7f9344e00640 1 Processor -- start 2024-03-21T10:34:29.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.198+0000 7f9344e00640 1 -- start start 2024-03-21T10:34:29.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.199+0000 7f9344e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93401020f0 0x7f9340199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:29.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.199+0000 7f9344e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9340104a20 0x7f934019a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:29.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.199+0000 7f9344e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9340107350 0x7f93401a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:29.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.199+0000 7f9344e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934010d2a0 con 0x7f9340104a20 2024-03-21T10:34:29.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.199+0000 7f9344e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934010d120 con 0x7f93401020f0 2024-03-21T10:34:29.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.199+0000 7f9344e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934010d420 con 0x7f9340107350 2024-03-21T10:34:29.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.199+0000 7f933ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9340104a20 0x7f934019a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:29.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.199+0000 7f933ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9340104a20 0x7f934019a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56344/0 (socket says 172.21.15.31:56344) 2024-03-21T10:34:29.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.199+0000 7f933ea00640 1 -- 172.21.15.31:0/3709337561 learned_addr learned my addr 172.21.15.31:0/3709337561 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:34:29.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.199+0000 7f933f400640 1 --2- 172.21.15.31:0/3709337561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93401020f0 0x7f9340199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:29.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.199+0000 7f933fe00640 1 --2- 172.21.15.31:0/3709337561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9340107350 0x7f93401a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:29.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.199+0000 7f933f400640 1 -- 172.21.15.31:0/3709337561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9340107350 msgr2=0x7f93401a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:29.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.199+0000 7f933f400640 1 --2- 172.21.15.31:0/3709337561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9340107350 0x7f93401a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:29.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.199+0000 7f933f400640 1 -- 172.21.15.31:0/3709337561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9340104a20 msgr2=0x7f934019a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:29.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.199+0000 7f933f400640 1 --2- 172.21.15.31:0/3709337561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9340104a20 0x7f934019a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:29.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.200+0000 7f933f400640 1 -- 172.21.15.31:0/3709337561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93401a1730 con 0x7f93401020f0 2024-03-21T10:34:29.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.200+0000 7f933fe00640 1 --2- 172.21.15.31:0/3709337561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9340107350 0x7f93401a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:34:29.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.200+0000 7f933ea00640 1 --2- 172.21.15.31:0/3709337561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9340104a20 0x7f934019a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:34:29.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.200+0000 7f933f400640 1 --2- 172.21.15.31:0/3709337561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93401020f0 0x7f9340199b10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9330036040 tx=0x7f933002c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:29.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.200+0000 7f9337e00640 1 -- 172.21.15.31:0/3709337561 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9330043020 con 0x7f93401020f0 2024-03-21T10:34:29.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.200+0000 7f9344e00640 1 -- 172.21.15.31:0/3709337561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93401a19c0 con 0x7f93401020f0 2024-03-21T10:34:29.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.200+0000 7f9337e00640 1 -- 172.21.15.31:0/3709337561 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93300054e0 con 0x7f93401020f0 2024-03-21T10:34:29.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.200+0000 7f9337e00640 1 -- 172.21.15.31:0/3709337561 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9330003cb0 con 0x7f93401020f0 2024-03-21T10:34:29.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.200+0000 7f9344e00640 1 -- 172.21.15.31:0/3709337561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93401a1f30 con 0x7f93401020f0 2024-03-21T10:34:29.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.201+0000 7f9344e00640 1 -- 172.21.15.31:0/3709337561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f934004f470 con 0x7f93401020f0 2024-03-21T10:34:29.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.204+0000 7f9337e00640 1 -- 172.21.15.31:0/3709337561 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f9330046020 con 0x7f93401020f0 2024-03-21T10:34:29.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.205+0000 7f9337e00640 1 --2- 172.21.15.31:0/3709337561 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f93240a2f10 0x7f93240a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:29.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.205+0000 7f9337e00640 1 -- 172.21.15.31:0/3709337561 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f93300f1660 con 0x7f93401020f0 2024-03-21T10:34:29.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.205+0000 7f933ea00640 1 --2- 172.21.15.31:0/3709337561 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f93240a2f10 0x7f93240a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:29.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.206+0000 7f933ea00640 1 --2- 172.21.15.31:0/3709337561 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f93240a2f10 0x7f93240a53d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f934019b020 tx=0x7f932c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:29.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.207+0000 7f9337e00640 1 -- 172.21.15.31:0/3709337561 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f933004d020 con 0x7f93401020f0 2024-03-21T10:34:29.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.497+0000 7f9344e00640 1 -- 172.21.15.31:0/3709337561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9340108e40 con 0x7f93401020f0 2024-03-21T10:34:29.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.498+0000 7f9337e00640 1 -- 172.21.15.31:0/3709337561 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2116 (secure 0 0 0) 0x7f93300be890 con 0x7f93401020f0 2024-03-21T10:34:29.496 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:34:29.496 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-21T10:34:29.496 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 3 2024-03-21T10:34:29.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.500+0000 7f9344e00640 1 -- 172.21.15.31:0/3709337561 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f93240a2f10 msgr2=0x7f93240a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:29.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.500+0000 7f9344e00640 1 --2- 172.21.15.31:0/3709337561 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f93240a2f10 0x7f93240a53d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f934019b020 tx=0x7f932c00b410 comp rx=0 tx=0).stop 2024-03-21T10:34:29.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.500+0000 7f9344e00640 1 -- 172.21.15.31:0/3709337561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93401020f0 msgr2=0x7f9340199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:29.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.500+0000 7f9344e00640 1 --2- 172.21.15.31:0/3709337561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93401020f0 0x7f9340199b10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9330036040 tx=0x7f933002c6c0 comp rx=0 tx=0).stop 2024-03-21T10:34:29.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.500+0000 7f9344e00640 1 -- 172.21.15.31:0/3709337561 shutdown_connections 2024-03-21T10:34:29.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.500+0000 7f9344e00640 1 --2- 172.21.15.31:0/3709337561 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f93240a2f10 0x7f93240a53d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:29.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.500+0000 7f9344e00640 1 --2- 172.21.15.31:0/3709337561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9340107350 0x7f93401a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:29.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.500+0000 7f9344e00640 1 --2- 172.21.15.31:0/3709337561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9340104a20 0x7f934019a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:29.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.500+0000 7f9344e00640 1 --2- 172.21.15.31:0/3709337561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93401020f0 0x7f9340199b10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:29.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.501+0000 7f9344e00640 1 -- 172.21.15.31:0/3709337561 >> 172.21.15.31:0/3709337561 conn(0x7f93400fba20 msgr2=0x7f9340101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:29.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.501+0000 7f9344e00640 1 -- 172.21.15.31:0/3709337561 shutdown_connections 2024-03-21T10:34:29.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:29.501+0000 7f9344e00640 1 -- 172.21.15.31:0/3709337561 wait complete. 2024-03-21T10:34:29.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:29 smithi031 ceph-mon[31673]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:29.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:29 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3709337561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:29.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:29 smithi040 ceph-mon[36154]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:29.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:29 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3709337561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:30.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:29 smithi070 ceph-mon[36330]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:30.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:29 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3709337561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:30.248 INFO:journalctl@ceph.mds.j.smithi031.stdout:Mar 21 10:34:29 smithi031 systemd[1]: Starting Ceph mds.j for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:34:30.556 INFO:journalctl@ceph.mds.j.smithi031.stdout:Mar 21 10:34:30 smithi031 podman[71358]: 2024-03-21 10:34:30.16147946 +0000 UTC m=+0.044795504 image pull 237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232 2024-03-21T10:34:30.556 INFO:journalctl@ceph.mds.j.smithi031.stdout:Mar 21 10:34:30 smithi031 podman[71358]: 2024-03-21 10:34:30.32385652 +0000 UTC m=+0.207172593 container create 64416c35f4077144393ca8cb51fe82295bbfa5b2c1ae0cd73521021fb35a3172 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-j, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-21T10:34:31.056 INFO:journalctl@ceph.mds.j.smithi031.stdout:Mar 21 10:34:30 smithi031 podman[71358]: 2024-03-21 10:34:30.632896349 +0000 UTC m=+0.516212424 container init 64416c35f4077144393ca8cb51fe82295bbfa5b2c1ae0cd73521021fb35a3172 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-j, org.label-schema.build-date=20240319, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-21T10:34:31.056 INFO:journalctl@ceph.mds.j.smithi031.stdout:Mar 21 10:34:30 smithi031 podman[71358]: 2024-03-21 10:34:30.643158981 +0000 UTC m=+0.526475035 container start 64416c35f4077144393ca8cb51fe82295bbfa5b2c1ae0cd73521021fb35a3172 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-j, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T10:34:31.056 INFO:journalctl@ceph.mds.j.smithi031.stdout:Mar 21 10:34:30 smithi031 ceph-mds[71377]: -- 172.21.15.31:0/1361885105 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5576a0ae5a40 con 0x5576a0ab9000 2024-03-21T10:34:31.056 INFO:journalctl@ceph.mds.j.smithi031.stdout:Mar 21 10:34:30 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-j[71373]: starting mds.j at 2024-03-21T10:34:31.056 INFO:journalctl@ceph.mds.j.smithi031.stdout:Mar 21 10:34:30 smithi031 bash[71358]: 64416c35f4077144393ca8cb51fe82295bbfa5b2c1ae0cd73521021fb35a3172 2024-03-21T10:34:31.056 INFO:journalctl@ceph.mds.j.smithi031.stdout:Mar 21 10:34:30 smithi031 systemd[1]: Started Ceph mds.j for 3847408c-e76d-11ee-95cd-87774f69a715. 2024-03-21T10:34:31.519 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:34:31.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.747+0000 7f365aa00640 1 -- 172.21.15.31:0/476002811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3654104b40 msgr2=0x7f3654104fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:31.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.747+0000 7f365aa00640 1 --2- 172.21.15.31:0/476002811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3654104b40 0x7f3654104fc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3640009900 tx=0x7f364002bc40 comp rx=0 tx=0).stop 2024-03-21T10:34:31.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.747+0000 7f365aa00640 1 -- 172.21.15.31:0/476002811 shutdown_connections 2024-03-21T10:34:31.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.747+0000 7f365aa00640 1 --2- 172.21.15.31:0/476002811 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3654105500 0x7f365410bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:31.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.747+0000 7f365aa00640 1 --2- 172.21.15.31:0/476002811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3654104b40 0x7f3654104fc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:31.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.747+0000 7f365aa00640 1 --2- 172.21.15.31:0/476002811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3654103950 0x7f3654103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:31.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.747+0000 7f365aa00640 1 -- 172.21.15.31:0/476002811 >> 172.21.15.31:0/476002811 conn(0x7f36540fef00 msgr2=0x7f3654101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:31.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.747+0000 7f365aa00640 1 -- 172.21.15.31:0/476002811 shutdown_connections 2024-03-21T10:34:31.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.747+0000 7f365aa00640 1 -- 172.21.15.31:0/476002811 wait complete. 2024-03-21T10:34:31.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.748+0000 7f365aa00640 1 Processor -- start 2024-03-21T10:34:31.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.748+0000 7f365aa00640 1 -- start start 2024-03-21T10:34:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.749+0000 7f365aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3654103950 0x7f3654199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.749+0000 7f365aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3654104b40 0x7f365419a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.749+0000 7f365aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3654105500 0x7f36541a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.749+0000 7f365aa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365410f590 con 0x7f3654105500 2024-03-21T10:34:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.749+0000 7f365aa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365410f410 con 0x7f3654104b40 2024-03-21T10:34:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.749+0000 7f365aa00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365410f710 con 0x7f3654103950 2024-03-21T10:34:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.749+0000 7f3653400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3654104b40 0x7f365419a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.749+0000 7f3658a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3654105500 0x7f36541a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.749+0000 7f3653400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3654104b40 0x7f365419a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:43298/0 (socket says 172.21.15.31:43298) 2024-03-21T10:34:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.749+0000 7f3653400640 1 -- 172.21.15.31:0/3631229130 learned_addr learned my addr 172.21.15.31:0/3631229130 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:34:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.749+0000 7f3653e00640 1 --2- 172.21.15.31:0/3631229130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3654103950 0x7f3654199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.750+0000 7f3653400640 1 -- 172.21.15.31:0/3631229130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3654103950 msgr2=0x7f3654199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.750+0000 7f3653400640 1 --2- 172.21.15.31:0/3631229130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3654103950 0x7f3654199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.750+0000 7f3653400640 1 -- 172.21.15.31:0/3631229130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3654105500 msgr2=0x7f36541a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.750+0000 7f3653400640 1 --2- 172.21.15.31:0/3631229130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3654105500 0x7f36541a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:31.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.750+0000 7f3653400640 1 -- 172.21.15.31:0/3631229130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36541a1800 con 0x7f3654104b40 2024-03-21T10:34:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.750+0000 7f3653400640 1 --2- 172.21.15.31:0/3631229130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3654104b40 0x7f365419a120 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f364002c8f0 tx=0x7f364002c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.750+0000 7f3650c00640 1 -- 172.21.15.31:0/3631229130 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3640041070 con 0x7f3654104b40 2024-03-21T10:34:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.750+0000 7f3650c00640 1 -- 172.21.15.31:0/3631229130 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3640004450 con 0x7f3654104b40 2024-03-21T10:34:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.750+0000 7f3650c00640 1 -- 172.21.15.31:0/3631229130 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3640003d90 con 0x7f3654104b40 2024-03-21T10:34:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.750+0000 7f365aa00640 1 -- 172.21.15.31:0/3631229130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36541a1a90 con 0x7f3654104b40 2024-03-21T10:34:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.751+0000 7f365aa00640 1 -- 172.21.15.31:0/3631229130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36541a1f80 con 0x7f3654104b40 2024-03-21T10:34:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.752+0000 7f365aa00640 1 -- 172.21.15.31:0/3631229130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f365410ac70 con 0x7f3654104b40 2024-03-21T10:34:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.754+0000 7f3650c00640 1 -- 172.21.15.31:0/3631229130 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f3640044070 con 0x7f3654104b40 2024-03-21T10:34:31.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.755+0000 7f3650c00640 1 --2- 172.21.15.31:0/3631229130 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f36380a2fa0 0x7f36380a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:31.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.755+0000 7f3650c00640 1 -- 172.21.15.31:0/3631229130 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f36400f0bb0 con 0x7f3654104b40 2024-03-21T10:34:31.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.755+0000 7f3653e00640 1 --2- 172.21.15.31:0/3631229130 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f36380a2fa0 0x7f36380a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:31.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.756+0000 7f3653e00640 1 --2- 172.21.15.31:0/3631229130 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f36380a2fa0 0x7f36380a5460 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f36541049a0 tx=0x7f3644009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:31.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:31.757+0000 7f3650c00640 1 -- 172.21.15.31:0/3631229130 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f36400bdde0 con 0x7f3654104b40 2024-03-21T10:34:31.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:31 smithi040 ceph-mon[36154]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:31.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:31 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:31.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:31 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:31.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:31 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:31.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:31 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:31.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:31 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:31.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:31 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:32.020 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:31 smithi031 ceph-mon[31673]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:32.020 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:31 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:32.020 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:31 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:32.021 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:31 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:32.021 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:31 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:32.021 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:31 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:32.021 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:31 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:32.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:32.037+0000 7f365aa00640 1 -- 172.21.15.31:0/3631229130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f365402d230 con 0x7f3654104b40 2024-03-21T10:34:32.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:32.037+0000 7f3650c00640 1 -- 172.21.15.31:0/3631229130 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2907 (secure 0 0 0) 0x7f36400bd6e0 con 0x7f3654104b40 2024-03-21T10:34:32.036 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:34:32.036 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-21T10:34:32.036 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 4 2024-03-21T10:34:32.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:32.040+0000 7f365aa00640 1 -- 172.21.15.31:0/3631229130 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f36380a2fa0 msgr2=0x7f36380a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:32.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:32.040+0000 7f365aa00640 1 --2- 172.21.15.31:0/3631229130 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f36380a2fa0 0x7f36380a5460 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f36541049a0 tx=0x7f3644009500 comp rx=0 tx=0).stop 2024-03-21T10:34:32.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:32.040+0000 7f365aa00640 1 -- 172.21.15.31:0/3631229130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3654104b40 msgr2=0x7f365419a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:32.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:32.040+0000 7f365aa00640 1 --2- 172.21.15.31:0/3631229130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3654104b40 0x7f365419a120 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f364002c8f0 tx=0x7f364002c660 comp rx=0 tx=0).stop 2024-03-21T10:34:32.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:32.040+0000 7f365aa00640 1 -- 172.21.15.31:0/3631229130 shutdown_connections 2024-03-21T10:34:32.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:32.040+0000 7f365aa00640 1 --2- 172.21.15.31:0/3631229130 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f36380a2fa0 0x7f36380a5460 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:32.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:32.040+0000 7f365aa00640 1 --2- 172.21.15.31:0/3631229130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3654105500 0x7f36541a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:32.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:32.040+0000 7f365aa00640 1 --2- 172.21.15.31:0/3631229130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3654104b40 0x7f365419a120 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:32.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:32.040+0000 7f365aa00640 1 --2- 172.21.15.31:0/3631229130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3654103950 0x7f3654199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:32.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:32.040+0000 7f365aa00640 1 -- 172.21.15.31:0/3631229130 >> 172.21.15.31:0/3631229130 conn(0x7f36540fef00 msgr2=0x7f3654107e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:32.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:32.041+0000 7f365aa00640 1 -- 172.21.15.31:0/3631229130 shutdown_connections 2024-03-21T10:34:32.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:32.041+0000 7f365aa00640 1 -- 172.21.15.31:0/3631229130 wait complete. 2024-03-21T10:34:32.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:31 smithi070 ceph-mon[36330]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:32.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:31 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:32.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:31 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:32.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:31 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:32.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:31 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:32.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:31 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:32.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:31 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:32.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:32 smithi040 ceph-mon[36154]: Deploying daemon mds.d on smithi031 2024-03-21T10:34:32.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:32 smithi040 ceph-mon[36154]: mds.? [v2:172.21.15.31:6834/2099524212,v1:172.21.15.31:6835/2099524212] up:boot 2024-03-21T10:34:32.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:32 smithi040 ceph-mon[36154]: fsmap 3 up:standby 2024-03-21T10:34:32.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:32 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-21T10:34:32.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:32 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3631229130' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:32.998 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:32 smithi031 ceph-mon[31673]: Deploying daemon mds.d on smithi031 2024-03-21T10:34:32.998 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:32 smithi031 ceph-mon[31673]: mds.? [v2:172.21.15.31:6834/2099524212,v1:172.21.15.31:6835/2099524212] up:boot 2024-03-21T10:34:32.998 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:32 smithi031 ceph-mon[31673]: fsmap 3 up:standby 2024-03-21T10:34:32.998 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:32 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-21T10:34:32.999 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:32 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3631229130' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:33.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:32 smithi070 ceph-mon[36330]: Deploying daemon mds.d on smithi031 2024-03-21T10:34:33.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:32 smithi070 ceph-mon[36330]: mds.? [v2:172.21.15.31:6834/2099524212,v1:172.21.15.31:6835/2099524212] up:boot 2024-03-21T10:34:33.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:32 smithi070 ceph-mon[36330]: fsmap 3 up:standby 2024-03-21T10:34:33.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:32 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-21T10:34:33.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:32 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3631229130' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:33.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:33 smithi031 ceph-mon[31673]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:33.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:33 smithi040 ceph-mon[36154]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:34.060 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:34:34.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:33 smithi070 ceph-mon[36330]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:34.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.438+0000 7f1c0de00640 1 -- 172.21.15.31:0/3303822643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c081028e0 msgr2=0x7f1c08102d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:34.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.438+0000 7f1c0de00640 1 --2- 172.21.15.31:0/3303822643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c081028e0 0x7f1c08102d60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1bf0009900 tx=0x7f1bf002bc30 comp rx=0 tx=0).stop 2024-03-21T10:34:34.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.438+0000 7f1c0de00640 1 -- 172.21.15.31:0/3303822643 shutdown_connections 2024-03-21T10:34:34.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.438+0000 7f1c0de00640 1 --2- 172.21.15.31:0/3303822643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c081032a0 0x7f1c08109b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:34.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.438+0000 7f1c0de00640 1 --2- 172.21.15.31:0/3303822643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c081028e0 0x7f1c08102d60 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:34.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.438+0000 7f1c0de00640 1 --2- 172.21.15.31:0/3303822643 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c081016f0 0x7f1c08101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:34.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.438+0000 7f1c0de00640 1 -- 172.21.15.31:0/3303822643 >> 172.21.15.31:0/3303822643 conn(0x7f1c080fcc60 msgr2=0x7f1c080ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:34.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.438+0000 7f1c0de00640 1 -- 172.21.15.31:0/3303822643 shutdown_connections 2024-03-21T10:34:34.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.438+0000 7f1c0de00640 1 -- 172.21.15.31:0/3303822643 wait complete. 2024-03-21T10:34:34.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.439+0000 7f1c0de00640 1 Processor -- start 2024-03-21T10:34:34.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.440+0000 7f1c0de00640 1 -- start start 2024-03-21T10:34:34.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.440+0000 7f1c0de00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c081016f0 0x7f1c08197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:34.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.440+0000 7f1c0de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c081028e0 0x7f1c08197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:34.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.440+0000 7f1c0de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c081032a0 0x7f1c0819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:34.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.440+0000 7f1c0de00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0810d370 con 0x7f1c081032a0 2024-03-21T10:34:34.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.440+0000 7f1c0de00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0810d1f0 con 0x7f1c081028e0 2024-03-21T10:34:34.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.440+0000 7f1c0de00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0810d4f0 con 0x7f1c081016f0 2024-03-21T10:34:34.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.440+0000 7f1c07400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c081032a0 0x7f1c0819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:34.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.440+0000 7f1bfe000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c081028e0 0x7f1c08197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:34.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.440+0000 7f1c07400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c081032a0 0x7f1c0819efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56414/0 (socket says 172.21.15.31:56414) 2024-03-21T10:34:34.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.440+0000 7f1c07400640 1 -- 172.21.15.31:0/2422470209 learned_addr learned my addr 172.21.15.31:0/2422470209 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:34:34.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.440+0000 7f1c06a00640 1 --2- 172.21.15.31:0/2422470209 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c081016f0 0x7f1c08197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:34.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.441+0000 7f1c07400640 1 -- 172.21.15.31:0/2422470209 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c081016f0 msgr2=0x7f1c08197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:34.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.441+0000 7f1c07400640 1 --2- 172.21.15.31:0/2422470209 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c081016f0 0x7f1c08197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:34.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.441+0000 7f1c07400640 1 -- 172.21.15.31:0/2422470209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c081028e0 msgr2=0x7f1c08197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:34.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.441+0000 7f1c07400640 1 --2- 172.21.15.31:0/2422470209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c081028e0 0x7f1c08197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:34.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.441+0000 7f1c07400640 1 -- 172.21.15.31:0/2422470209 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0819f630 con 0x7f1c081032a0 2024-03-21T10:34:34.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.441+0000 7f1c07400640 1 --2- 172.21.15.31:0/2422470209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c081032a0 0x7f1c0819efc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1c00009fd0 tx=0x7f1c0000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:34.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.441+0000 7f1c0ca00640 1 -- 172.21.15.31:0/2422470209 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c00017070 con 0x7f1c081032a0 2024-03-21T10:34:34.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.441+0000 7f1c0ca00640 1 -- 172.21.15.31:0/2422470209 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c00003e50 con 0x7f1c081032a0 2024-03-21T10:34:34.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.441+0000 7f1c0de00640 1 -- 172.21.15.31:0/2422470209 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c0819f920 con 0x7f1c081032a0 2024-03-21T10:34:34.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.441+0000 7f1c0ca00640 1 -- 172.21.15.31:0/2422470209 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c000106c0 con 0x7f1c081032a0 2024-03-21T10:34:34.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.442+0000 7f1c0de00640 1 -- 172.21.15.31:0/2422470209 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c0819fdf0 con 0x7f1c081032a0 2024-03-21T10:34:34.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.443+0000 7f1c0de00640 1 -- 172.21.15.31:0/2422470209 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c08108a30 con 0x7f1c081032a0 2024-03-21T10:34:34.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.445+0000 7f1c0ca00640 1 -- 172.21.15.31:0/2422470209 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f1c00003970 con 0x7f1c081032a0 2024-03-21T10:34:34.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.446+0000 7f1c0ca00640 1 --2- 172.21.15.31:0/2422470209 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1bec0a2f20 0x7f1bec0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:34.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.447+0000 7f1c0ca00640 1 -- 172.21.15.31:0/2422470209 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1c000ced00 con 0x7f1c081032a0 2024-03-21T10:34:34.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.447+0000 7f1c06a00640 1 --2- 172.21.15.31:0/2422470209 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1bec0a2f20 0x7f1bec0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:34.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.448+0000 7f1c06a00640 1 --2- 172.21.15.31:0/2422470209 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1bec0a2f20 0x7f1bec0a53e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1c08102740 tx=0x7f1bf8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:34.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.451+0000 7f1c0ca00640 1 -- 172.21.15.31:0/2422470209 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1c0009b8e0 con 0x7f1c081032a0 2024-03-21T10:34:34.557 INFO:journalctl@ceph.mds.d.smithi031.stdout:Mar 21 10:34:34 smithi031 systemd[1]: Starting Ceph mds.d for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:34:34.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.721+0000 7f1c0de00640 1 -- 172.21.15.31:0/2422470209 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c080622a0 con 0x7f1c081032a0 2024-03-21T10:34:34.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.722+0000 7f1c0ca00640 1 -- 172.21.15.31:0/2422470209 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2907 (secure 0 0 0) 0x7f1c0009f980 con 0x7f1c081032a0 2024-03-21T10:34:34.720 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:34:34.720 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-21T10:34:34.721 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 4 2024-03-21T10:34:34.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.724+0000 7f1c0de00640 1 -- 172.21.15.31:0/2422470209 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1bec0a2f20 msgr2=0x7f1bec0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:34.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.724+0000 7f1c0de00640 1 --2- 172.21.15.31:0/2422470209 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1bec0a2f20 0x7f1bec0a53e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1c08102740 tx=0x7f1bf8005f90 comp rx=0 tx=0).stop 2024-03-21T10:34:34.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.724+0000 7f1c0de00640 1 -- 172.21.15.31:0/2422470209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c081032a0 msgr2=0x7f1c0819efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:34.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.724+0000 7f1c0de00640 1 --2- 172.21.15.31:0/2422470209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c081032a0 0x7f1c0819efc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1c00009fd0 tx=0x7f1c0000b260 comp rx=0 tx=0).stop 2024-03-21T10:34:34.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.724+0000 7f1c0de00640 1 -- 172.21.15.31:0/2422470209 shutdown_connections 2024-03-21T10:34:34.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.725+0000 7f1c0de00640 1 --2- 172.21.15.31:0/2422470209 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1bec0a2f20 0x7f1bec0a53e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:34.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.725+0000 7f1c0de00640 1 --2- 172.21.15.31:0/2422470209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c081032a0 0x7f1c0819efc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:34.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.725+0000 7f1c0de00640 1 --2- 172.21.15.31:0/2422470209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c081028e0 0x7f1c08197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:34.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.725+0000 7f1c0de00640 1 --2- 172.21.15.31:0/2422470209 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c081016f0 0x7f1c08197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:34.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.725+0000 7f1c0de00640 1 -- 172.21.15.31:0/2422470209 >> 172.21.15.31:0/2422470209 conn(0x7f1c080fcc60 msgr2=0x7f1c08105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:34.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.725+0000 7f1c0de00640 1 -- 172.21.15.31:0/2422470209 shutdown_connections 2024-03-21T10:34:34.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:34.725+0000 7f1c0de00640 1 -- 172.21.15.31:0/2422470209 wait complete. 2024-03-21T10:34:34.999 INFO:journalctl@ceph.mds.d.smithi031.stdout:Mar 21 10:34:34 smithi031 podman[71735]: 2024-03-21 10:34:34.615826952 +0000 UTC m=+0.038206796 image pull 237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232 2024-03-21T10:34:35.000 INFO:journalctl@ceph.mds.d.smithi031.stdout:Mar 21 10:34:34 smithi031 podman[71735]: 2024-03-21 10:34:34.738275318 +0000 UTC m=+0.160655141 container create 283c28155b7b50696faf1f76bd1e40b8c400850ae75187485d33059a95ddb84f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-d, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T10:34:35.000 INFO:journalctl@ceph.mds.d.smithi031.stdout:Mar 21 10:34:35 smithi031 podman[71735]: 2024-03-21 10:34:35.000669589 +0000 UTC m=+0.423049482 container init 283c28155b7b50696faf1f76bd1e40b8c400850ae75187485d33059a95ddb84f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-d, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T10:34:35.306 INFO:journalctl@ceph.mds.d.smithi031.stdout:Mar 21 10:34:35 smithi031 podman[71735]: 2024-03-21 10:34:35.011667806 +0000 UTC m=+0.434047628 container start 283c28155b7b50696faf1f76bd1e40b8c400850ae75187485d33059a95ddb84f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-d, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-21T10:34:35.306 INFO:journalctl@ceph.mds.d.smithi031.stdout:Mar 21 10:34:35 smithi031 ceph-mds[71756]: -- 172.21.15.31:0/2813940479 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56552eb03a40 con 0x56552ead6800 2024-03-21T10:34:35.306 INFO:journalctl@ceph.mds.d.smithi031.stdout:Mar 21 10:34:35 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-d[71752]: starting mds.d at 2024-03-21T10:34:35.306 INFO:journalctl@ceph.mds.d.smithi031.stdout:Mar 21 10:34:35 smithi031 bash[71735]: 283c28155b7b50696faf1f76bd1e40b8c400850ae75187485d33059a95ddb84f 2024-03-21T10:34:35.306 INFO:journalctl@ceph.mds.d.smithi031.stdout:Mar 21 10:34:35 smithi031 systemd[1]: Started Ceph mds.d for 3847408c-e76d-11ee-95cd-87774f69a715. 2024-03-21T10:34:35.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:35 smithi040 ceph-mon[36154]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:35.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:35 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2422470209' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:35.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:35 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:35.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:35 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:35.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:35 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:35.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:35 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:35.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:35 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:35.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:35 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:36.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:35 smithi031 ceph-mon[31673]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:36.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:35 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2422470209' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:36.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:35 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:36.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:35 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:36.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:35 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:36.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:35 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:36.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:35 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:36.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:35 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:36.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:35 smithi070 ceph-mon[36330]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:36.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:35 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2422470209' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:36.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:35 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:36.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:35 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:36.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:35 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:36.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:35 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:36.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:35 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:36.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:35 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:36.743 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:34:36.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:36 smithi040 ceph-mon[36154]: Deploying daemon mds.e on smithi040 2024-03-21T10:34:36.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:36 smithi040 ceph-mon[36154]: mds.? [v2:172.21.15.31:6836/1193249174,v1:172.21.15.31:6837/1193249174] up:boot 2024-03-21T10:34:36.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:36 smithi040 ceph-mon[36154]: fsmap 4 up:standby 2024-03-21T10:34:36.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:36 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-21T10:34:36.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.981+0000 7fdd38800640 1 -- 172.21.15.31:0/3881781536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd300756e0 msgr2=0x7fdd30073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:36.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.981+0000 7fdd38800640 1 --2- 172.21.15.31:0/3881781536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd300756e0 0x7fdd30073b30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009cd0 tx=0x7fdd2802c0f0 comp rx=0 tx=0).stop 2024-03-21T10:34:36.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.982+0000 7fdd38800640 1 -- 172.21.15.31:0/3881781536 shutdown_connections 2024-03-21T10:34:36.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.982+0000 7fdd38800640 1 --2- 172.21.15.31:0/3881781536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd3010dd00 0x7fdd301100f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:36.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.982+0000 7fdd38800640 1 --2- 172.21.15.31:0/3881781536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd30074100 0x7fdd30074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:36.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.982+0000 7fdd38800640 1 --2- 172.21.15.31:0/3881781536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd300756e0 0x7fdd30073b30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:36.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.982+0000 7fdd38800640 1 -- 172.21.15.31:0/3881781536 >> 172.21.15.31:0/3881781536 conn(0x7fdd300fccc0 msgr2=0x7fdd300ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:36.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.982+0000 7fdd38800640 1 -- 172.21.15.31:0/3881781536 shutdown_connections 2024-03-21T10:34:36.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.982+0000 7fdd38800640 1 -- 172.21.15.31:0/3881781536 wait complete. 2024-03-21T10:34:36.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.983+0000 7fdd38800640 1 Processor -- start 2024-03-21T10:34:36.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.983+0000 7fdd38800640 1 -- start start 2024-03-21T10:34:36.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.983+0000 7fdd38800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd30074100 0x7fdd3019e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:36.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.983+0000 7fdd38800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd300756e0 0x7fdd3019e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:36.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.983+0000 7fdd38800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd3010dd00 0x7fdd301a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:36.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.983+0000 7fdd38800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd30113910 con 0x7fdd3010dd00 2024-03-21T10:34:36.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.983+0000 7fdd38800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd30113790 con 0x7fdd30074100 2024-03-21T10:34:36.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.983+0000 7fdd38800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd30113a90 con 0x7fdd300756e0 2024-03-21T10:34:36.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.984+0000 7fdd35e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd30074100 0x7fdd3019e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:36.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.984+0000 7fdd35e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd30074100 0x7fdd3019e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60882/0 (socket says 172.21.15.31:60882) 2024-03-21T10:34:36.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.984+0000 7fdd35e00640 1 -- 172.21.15.31:0/3380107464 learned_addr learned my addr 172.21.15.31:0/3380107464 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:34:36.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.984+0000 7fdd36800640 1 --2- 172.21.15.31:0/3380107464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd3010dd00 0x7fdd301a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:36.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.984+0000 7fdd35400640 1 --2- 172.21.15.31:0/3380107464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd300756e0 0x7fdd3019e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:36.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.984+0000 7fdd35e00640 1 -- 172.21.15.31:0/3380107464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd300756e0 msgr2=0x7fdd3019e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:36.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.984+0000 7fdd35e00640 1 --2- 172.21.15.31:0/3380107464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd300756e0 0x7fdd3019e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:36.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.984+0000 7fdd35e00640 1 -- 172.21.15.31:0/3380107464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd3010dd00 msgr2=0x7fdd301a55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:36.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.984+0000 7fdd35e00640 1 --2- 172.21.15.31:0/3380107464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd3010dd00 0x7fdd301a55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:36.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.984+0000 7fdd35e00640 1 -- 172.21.15.31:0/3380107464 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd301a5c20 con 0x7fdd30074100 2024-03-21T10:34:36.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.984+0000 7fdd35e00640 1 --2- 172.21.15.31:0/3380107464 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd30074100 0x7fdd3019e000 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdd28000f80 tx=0x7fdd28032ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:36.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.985+0000 7fdd26a00640 1 -- 172.21.15.31:0/3380107464 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd28041020 con 0x7fdd30074100 2024-03-21T10:34:36.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.985+0000 7fdd26a00640 1 -- 172.21.15.31:0/3380107464 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd28034cd0 con 0x7fdd30074100 2024-03-21T10:34:36.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.985+0000 7fdd38800640 1 -- 172.21.15.31:0/3380107464 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd301a5eb0 con 0x7fdd30074100 2024-03-21T10:34:36.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.985+0000 7fdd26a00640 1 -- 172.21.15.31:0/3380107464 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd28039040 con 0x7fdd30074100 2024-03-21T10:34:36.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.985+0000 7fdd38800640 1 -- 172.21.15.31:0/3380107464 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd301a62b0 con 0x7fdd30074100 2024-03-21T10:34:36.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.986+0000 7fdd38800640 1 -- 172.21.15.31:0/3380107464 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd30103ed0 con 0x7fdd30074100 2024-03-21T10:34:36.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.988+0000 7fdd26a00640 1 -- 172.21.15.31:0/3380107464 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fdd28044020 con 0x7fdd30074100 2024-03-21T10:34:36.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.989+0000 7fdd26a00640 1 --2- 172.21.15.31:0/3380107464 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fdd1c0a2f20 0x7fdd1c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:36.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.990+0000 7fdd26a00640 1 -- 172.21.15.31:0/3380107464 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdd280f10d0 con 0x7fdd30074100 2024-03-21T10:34:36.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.990+0000 7fdd35400640 1 --2- 172.21.15.31:0/3380107464 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fdd1c0a2f20 0x7fdd1c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:36.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.990+0000 7fdd35400640 1 --2- 172.21.15.31:0/3380107464 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fdd1c0a2f20 0x7fdd1c0a53e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdd3019f510 tx=0x7fdd20005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:36.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:36.992+0000 7fdd26a00640 1 -- 172.21.15.31:0/3380107464 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdd2804b020 con 0x7fdd30074100 2024-03-21T10:34:37.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:36 smithi031 ceph-mon[31673]: Deploying daemon mds.e on smithi040 2024-03-21T10:34:37.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:36 smithi031 ceph-mon[31673]: mds.? [v2:172.21.15.31:6836/1193249174,v1:172.21.15.31:6837/1193249174] up:boot 2024-03-21T10:34:37.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:36 smithi031 ceph-mon[31673]: fsmap 4 up:standby 2024-03-21T10:34:37.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:36 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-21T10:34:37.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:36 smithi070 ceph-mon[36330]: Deploying daemon mds.e on smithi040 2024-03-21T10:34:37.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:36 smithi070 ceph-mon[36330]: mds.? [v2:172.21.15.31:6836/1193249174,v1:172.21.15.31:6837/1193249174] up:boot 2024-03-21T10:34:37.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:36 smithi070 ceph-mon[36330]: fsmap 4 up:standby 2024-03-21T10:34:37.075 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:36 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-21T10:34:37.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:37.234+0000 7fdd38800640 1 -- 172.21.15.31:0/3380107464 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd3002d070 con 0x7fdd30074100 2024-03-21T10:34:37.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:37.235+0000 7fdd26a00640 1 -- 172.21.15.31:0/3380107464 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3698 (secure 0 0 0) 0x7fdd280be390 con 0x7fdd30074100 2024-03-21T10:34:37.233 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:34:37.234 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-21T10:34:37.234 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 5 2024-03-21T10:34:37.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:37.237+0000 7fdd38800640 1 -- 172.21.15.31:0/3380107464 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fdd1c0a2f20 msgr2=0x7fdd1c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:37.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:37.237+0000 7fdd38800640 1 --2- 172.21.15.31:0/3380107464 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fdd1c0a2f20 0x7fdd1c0a53e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdd3019f510 tx=0x7fdd20005f90 comp rx=0 tx=0).stop 2024-03-21T10:34:37.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:37.237+0000 7fdd38800640 1 -- 172.21.15.31:0/3380107464 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd30074100 msgr2=0x7fdd3019e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:37.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:37.237+0000 7fdd38800640 1 --2- 172.21.15.31:0/3380107464 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd30074100 0x7fdd3019e000 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdd28000f80 tx=0x7fdd28032ec0 comp rx=0 tx=0).stop 2024-03-21T10:34:37.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:37.238+0000 7fdd38800640 1 -- 172.21.15.31:0/3380107464 shutdown_connections 2024-03-21T10:34:37.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:37.238+0000 7fdd38800640 1 --2- 172.21.15.31:0/3380107464 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fdd1c0a2f20 0x7fdd1c0a53e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:37.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:37.238+0000 7fdd38800640 1 --2- 172.21.15.31:0/3380107464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd3010dd00 0x7fdd301a55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:37.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:37.238+0000 7fdd38800640 1 --2- 172.21.15.31:0/3380107464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd300756e0 0x7fdd3019e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:37.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:37.238+0000 7fdd38800640 1 --2- 172.21.15.31:0/3380107464 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd30074100 0x7fdd3019e000 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:37.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:37.238+0000 7fdd38800640 1 -- 172.21.15.31:0/3380107464 >> 172.21.15.31:0/3380107464 conn(0x7fdd300fccc0 msgr2=0x7fdd301010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:37.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:37.238+0000 7fdd38800640 1 -- 172.21.15.31:0/3380107464 shutdown_connections 2024-03-21T10:34:37.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:37.238+0000 7fdd38800640 1 -- 172.21.15.31:0/3380107464 wait complete. 2024-03-21T10:34:37.650 INFO:journalctl@ceph.mds.e.smithi040.stdout:Mar 21 10:34:37 smithi040 systemd[1]: Starting Ceph mds.e for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:34:37.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:37 smithi040 ceph-mon[36154]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:37.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:37 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3380107464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:37.932 INFO:journalctl@ceph.mds.e.smithi040.stdout:Mar 21 10:34:37 smithi040 podman[65815]: 2024-03-21 10:34:37.705067262 +0000 UTC m=+0.018088879 image pull 237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232 2024-03-21T10:34:37.932 INFO:journalctl@ceph.mds.e.smithi040.stdout:Mar 21 10:34:37 smithi040 podman[65815]: 2024-03-21 10:34:37.814113526 +0000 UTC m=+0.127135121 container create 4a091e9b018ebe5a412245e78c446eb19b2cbded9bef23e677e11b519645ac74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-e, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-21T10:34:38.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:37 smithi031 ceph-mon[31673]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:38.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:37 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3380107464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:38.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:37 smithi070 ceph-mon[36330]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:38.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:37 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3380107464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:38.312 INFO:journalctl@ceph.mds.e.smithi040.stdout:Mar 21 10:34:38 smithi040 podman[65815]: 2024-03-21 10:34:38.058455915 +0000 UTC m=+0.371477508 container init 4a091e9b018ebe5a412245e78c446eb19b2cbded9bef23e677e11b519645ac74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-e, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T10:34:38.312 INFO:journalctl@ceph.mds.e.smithi040.stdout:Mar 21 10:34:38 smithi040 podman[65815]: 2024-03-21 10:34:38.061205873 +0000 UTC m=+0.374227468 container start 4a091e9b018ebe5a412245e78c446eb19b2cbded9bef23e677e11b519645ac74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-21T10:34:38.312 INFO:journalctl@ceph.mds.e.smithi040.stdout:Mar 21 10:34:38 smithi040 ceph-mds[65834]: -- 172.21.15.40:0/1916496532 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f8fa3e3a40 con 0x55f8fa3b6c00 2024-03-21T10:34:38.313 INFO:journalctl@ceph.mds.e.smithi040.stdout:Mar 21 10:34:38 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-e[65830]: starting mds.e at 2024-03-21T10:34:38.313 INFO:journalctl@ceph.mds.e.smithi040.stdout:Mar 21 10:34:38 smithi040 bash[65815]: 4a091e9b018ebe5a412245e78c446eb19b2cbded9bef23e677e11b519645ac74 2024-03-21T10:34:38.313 INFO:journalctl@ceph.mds.e.smithi040.stdout:Mar 21 10:34:38 smithi040 systemd[1]: Started Ceph mds.e for 3847408c-e76d-11ee-95cd-87774f69a715. 2024-03-21T10:34:39.256 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:34:39.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:39 smithi031 ceph-mon[31673]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:39.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:39 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:39.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:39 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:39.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:39 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:39.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:39 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:39.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:39 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:39.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:39 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:39.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:39 smithi031 ceph-mon[31673]: Deploying daemon mds.f on smithi070 2024-03-21T10:34:39.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:39 smithi031 ceph-mon[31673]: mds.? [v2:172.21.15.40:6834/2081017532,v1:172.21.15.40:6835/2081017532] up:boot 2024-03-21T10:34:39.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:39 smithi031 ceph-mon[31673]: fsmap 5 up:standby 2024-03-21T10:34:39.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:39 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-21T10:34:39.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.497+0000 7f63e7200640 1 -- 172.21.15.31:0/3186494519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63e8108db0 msgr2=0x7f63e8109230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:39.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.497+0000 7f63e7200640 1 --2- 172.21.15.31:0/3186494519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63e8108db0 0x7f63e8109230 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f63d0009ca0 tx=0x7f63d002c100 comp rx=0 tx=0).stop 2024-03-21T10:34:39.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.497+0000 7f63e7200640 1 -- 172.21.15.31:0/3186494519 shutdown_connections 2024-03-21T10:34:39.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.497+0000 7f63e7200640 1 --2- 172.21.15.31:0/3186494519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63e8109770 0x7f63e8110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:39.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.497+0000 7f63e7200640 1 --2- 172.21.15.31:0/3186494519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63e8108db0 0x7f63e8109230 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:39.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.497+0000 7f63e7200640 1 --2- 172.21.15.31:0/3186494519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63e8107bc0 0x7f63e8107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:39.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.497+0000 7f63e7200640 1 -- 172.21.15.31:0/3186494519 >> 172.21.15.31:0/3186494519 conn(0x7f63e8076b60 msgr2=0x7f63e8078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:39.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.497+0000 7f63e7200640 1 -- 172.21.15.31:0/3186494519 shutdown_connections 2024-03-21T10:34:39.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.497+0000 7f63e7200640 1 -- 172.21.15.31:0/3186494519 wait complete. 2024-03-21T10:34:39.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.498+0000 7f63e7200640 1 Processor -- start 2024-03-21T10:34:39.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.498+0000 7f63e7200640 1 -- start start 2024-03-21T10:34:39.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.499+0000 7f63e7200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63e8107bc0 0x7f63e819def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:39.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.499+0000 7f63e7200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63e8108db0 0x7f63e819e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:39.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.499+0000 7f63e7200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63e8109770 0x7f63e81a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:39.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.499+0000 7f63e7200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e8113820 con 0x7f63e8107bc0 2024-03-21T10:34:39.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.499+0000 7f63e7200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e81136a0 con 0x7f63e8108db0 2024-03-21T10:34:39.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.499+0000 7f63e7200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e81139a0 con 0x7f63e8109770 2024-03-21T10:34:39.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.499+0000 7f63e5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63e8107bc0 0x7f63e819def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:39.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.499+0000 7f63e5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63e8107bc0 0x7f63e819def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58458/0 (socket says 172.21.15.31:58458) 2024-03-21T10:34:39.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.499+0000 7f63e5e00640 1 -- 172.21.15.31:0/885762214 learned_addr learned my addr 172.21.15.31:0/885762214 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:34:39.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.499+0000 7f63e6800640 1 --2- 172.21.15.31:0/885762214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63e8109770 0x7f63e81a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:39.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.499+0000 7f63e5400640 1 --2- 172.21.15.31:0/885762214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63e8108db0 0x7f63e819e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:39.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.500+0000 7f63e5e00640 1 -- 172.21.15.31:0/885762214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63e8109770 msgr2=0x7f63e81a5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:39.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.500+0000 7f63e5e00640 1 --2- 172.21.15.31:0/885762214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63e8109770 0x7f63e81a5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:39.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.500+0000 7f63e5e00640 1 -- 172.21.15.31:0/885762214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63e8108db0 msgr2=0x7f63e819e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:39.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.500+0000 7f63e5e00640 1 --2- 172.21.15.31:0/885762214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63e8108db0 0x7f63e819e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:39.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.500+0000 7f63e5e00640 1 -- 172.21.15.31:0/885762214 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e81a5a70 con 0x7f63e8107bc0 2024-03-21T10:34:39.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.500+0000 7f63e6800640 1 --2- 172.21.15.31:0/885762214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63e8109770 0x7f63e81a5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:34:39.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.500+0000 7f63e5e00640 1 --2- 172.21.15.31:0/885762214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63e8107bc0 0x7f63e819def0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f63d800e9c0 tx=0x7f63d800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:39.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.500+0000 7f63dea00640 1 -- 172.21.15.31:0/885762214 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d800ccc0 con 0x7f63e8107bc0 2024-03-21T10:34:39.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.500+0000 7f63dea00640 1 -- 172.21.15.31:0/885762214 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f63d80035f0 con 0x7f63e8107bc0 2024-03-21T10:34:39.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.500+0000 7f63dea00640 1 -- 172.21.15.31:0/885762214 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d800ce60 con 0x7f63e8107bc0 2024-03-21T10:34:39.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.500+0000 7f63e7200640 1 -- 172.21.15.31:0/885762214 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63e81a5d60 con 0x7f63e8107bc0 2024-03-21T10:34:39.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.500+0000 7f63e7200640 1 -- 172.21.15.31:0/885762214 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63e81a61d0 con 0x7f63e8107bc0 2024-03-21T10:34:39.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.502+0000 7f63e7200640 1 -- 172.21.15.31:0/885762214 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63e810eee0 con 0x7f63e8107bc0 2024-03-21T10:34:39.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.504+0000 7f63dea00640 1 -- 172.21.15.31:0/885762214 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f63d80037a0 con 0x7f63e8107bc0 2024-03-21T10:34:39.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.505+0000 7f63dea00640 1 --2- 172.21.15.31:0/885762214 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f63cc0a2f90 0x7f63cc0a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:39.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.505+0000 7f63dea00640 1 -- 172.21.15.31:0/885762214 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f63d80cf390 con 0x7f63e8107bc0 2024-03-21T10:34:39.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.506+0000 7f63e5400640 1 --2- 172.21.15.31:0/885762214 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f63cc0a2f90 0x7f63cc0a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:39.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.506+0000 7f63e5400640 1 --2- 172.21.15.31:0/885762214 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f63cc0a2f90 0x7f63cc0a5450 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f63d002c710 tx=0x7f63d00323c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:39.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.508+0000 7f63dea00640 1 -- 172.21.15.31:0/885762214 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f63d809c5c0 con 0x7f63e8107bc0 2024-03-21T10:34:39.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:39 smithi070 ceph-mon[36330]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:39.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:39 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:39.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:39 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:39.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:39 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:39.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:39 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:39.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:39 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:39.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:39 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:39.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:39 smithi070 ceph-mon[36330]: Deploying daemon mds.f on smithi070 2024-03-21T10:34:39.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:39 smithi070 ceph-mon[36330]: mds.? [v2:172.21.15.40:6834/2081017532,v1:172.21.15.40:6835/2081017532] up:boot 2024-03-21T10:34:39.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:39 smithi070 ceph-mon[36330]: fsmap 5 up:standby 2024-03-21T10:34:39.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:39 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-21T10:34:39.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:39 smithi040 ceph-mon[36154]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:39.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:39 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:39.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:39 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:39.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:39 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:39.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:39 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:39.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:39 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:39.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:39 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:39.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:39 smithi040 ceph-mon[36154]: Deploying daemon mds.f on smithi070 2024-03-21T10:34:39.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:39 smithi040 ceph-mon[36154]: mds.? [v2:172.21.15.40:6834/2081017532,v1:172.21.15.40:6835/2081017532] up:boot 2024-03-21T10:34:39.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:39 smithi040 ceph-mon[36154]: fsmap 5 up:standby 2024-03-21T10:34:39.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:39 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-21T10:34:39.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.755+0000 7f63e7200640 1 -- 172.21.15.31:0/885762214 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63e802d3c0 con 0x7f63e8107bc0 2024-03-21T10:34:39.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.756+0000 7f63dea00640 1 -- 172.21.15.31:0/885762214 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4489 (secure 0 0 0) 0x7f63e802d3c0 con 0x7f63e8107bc0 2024-03-21T10:34:39.755 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:34:39.755 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-21T10:34:39.755 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 6 2024-03-21T10:34:39.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.759+0000 7f63e7200640 1 -- 172.21.15.31:0/885762214 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f63cc0a2f90 msgr2=0x7f63cc0a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:39.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.759+0000 7f63e7200640 1 --2- 172.21.15.31:0/885762214 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f63cc0a2f90 0x7f63cc0a5450 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f63d002c710 tx=0x7f63d00323c0 comp rx=0 tx=0).stop 2024-03-21T10:34:39.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.759+0000 7f63e7200640 1 -- 172.21.15.31:0/885762214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63e8107bc0 msgr2=0x7f63e819def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:39.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.759+0000 7f63e7200640 1 --2- 172.21.15.31:0/885762214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63e8107bc0 0x7f63e819def0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f63d800e9c0 tx=0x7f63d800edc0 comp rx=0 tx=0).stop 2024-03-21T10:34:39.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.759+0000 7f63e7200640 1 -- 172.21.15.31:0/885762214 shutdown_connections 2024-03-21T10:34:39.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.759+0000 7f63e7200640 1 --2- 172.21.15.31:0/885762214 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f63cc0a2f90 0x7f63cc0a5450 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:39.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.759+0000 7f63e7200640 1 --2- 172.21.15.31:0/885762214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63e8109770 0x7f63e81a5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:39.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.759+0000 7f63e7200640 1 --2- 172.21.15.31:0/885762214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63e8108db0 0x7f63e819e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:39.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.759+0000 7f63e7200640 1 --2- 172.21.15.31:0/885762214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63e8107bc0 0x7f63e819def0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:39.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.759+0000 7f63e7200640 1 -- 172.21.15.31:0/885762214 >> 172.21.15.31:0/885762214 conn(0x7f63e8076b60 msgr2=0x7f63e810c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:39.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.759+0000 7f63e7200640 1 -- 172.21.15.31:0/885762214 shutdown_connections 2024-03-21T10:34:39.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:39.759+0000 7f63e7200640 1 -- 172.21.15.31:0/885762214 wait complete. 2024-03-21T10:34:40.229 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:40 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/885762214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:40.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:40 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/885762214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:40.574 INFO:journalctl@ceph.mds.f.smithi070.stdout:Mar 21 10:34:40 smithi070 systemd[1]: Starting Ceph mds.f for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:34:40.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:40 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/885762214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:40.939 INFO:journalctl@ceph.mds.f.smithi070.stdout:Mar 21 10:34:40 smithi070 podman[66719]: 2024-03-21 10:34:40.564506913 +0000 UTC m=+0.017892176 image pull 237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232 2024-03-21T10:34:40.940 INFO:journalctl@ceph.mds.f.smithi070.stdout:Mar 21 10:34:40 smithi070 podman[66719]: 2024-03-21 10:34:40.680674979 +0000 UTC m=+0.134060235 container create 33d6518933b04d0140568e257e4cca6c1a51418966afb818e6655c4986e02b4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-f, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T10:34:40.940 INFO:journalctl@ceph.mds.f.smithi070.stdout:Mar 21 10:34:40 smithi070 podman[66719]: 2024-03-21 10:34:40.941703833 +0000 UTC m=+0.395089110 container init 33d6518933b04d0140568e257e4cca6c1a51418966afb818e6655c4986e02b4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-f, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-21T10:34:41.217 INFO:journalctl@ceph.mds.f.smithi070.stdout:Mar 21 10:34:40 smithi070 podman[66719]: 2024-03-21 10:34:40.944555821 +0000 UTC m=+0.397941090 container start 33d6518933b04d0140568e257e4cca6c1a51418966afb818e6655c4986e02b4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-f, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-21T10:34:41.217 INFO:journalctl@ceph.mds.f.smithi070.stdout:Mar 21 10:34:40 smithi070 ceph-mds[66738]: -- 172.21.15.70:0/4180008221 <== mon.1 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5572b8627a40 con 0x5572b85fac00 2024-03-21T10:34:41.217 INFO:journalctl@ceph.mds.f.smithi070.stdout:Mar 21 10:34:40 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-f[66734]: starting mds.f at 2024-03-21T10:34:41.217 INFO:journalctl@ceph.mds.f.smithi070.stdout:Mar 21 10:34:41 smithi070 bash[66719]: 33d6518933b04d0140568e257e4cca6c1a51418966afb818e6655c4986e02b4f 2024-03-21T10:34:41.217 INFO:journalctl@ceph.mds.f.smithi070.stdout:Mar 21 10:34:41 smithi070 systemd[1]: Started Ceph mds.f for 3847408c-e76d-11ee-95cd-87774f69a715. 2024-03-21T10:34:41.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:41 smithi031 ceph-mon[31673]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:41.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:41 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:41.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:41 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:41.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:41 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:41.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:41 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:41.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:41 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:41.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:41 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:41.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:41 smithi070 ceph-mon[36330]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:41.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:41 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:41.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:41 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:41.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:41 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:41.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:41 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:41.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:41 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:41.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:41 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:41.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:41 smithi040 ceph-mon[36154]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:41.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:41 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:41.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:41 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:41.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:41 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:41.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:41 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:41.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:41 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:41.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:41 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:41.780 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:34:42.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.015+0000 7f6418a00640 1 -- 172.21.15.31:0/2238557455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6414109770 msgr2=0x7f6414110000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:42.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.015+0000 7f6418a00640 1 --2- 172.21.15.31:0/2238557455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6414109770 0x7f6414110000 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f640c009ca0 tx=0x7f640c02c100 comp rx=0 tx=0).stop 2024-03-21T10:34:42.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.015+0000 7f6418a00640 1 -- 172.21.15.31:0/2238557455 shutdown_connections 2024-03-21T10:34:42.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.015+0000 7f6418a00640 1 --2- 172.21.15.31:0/2238557455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6414109770 0x7f6414110000 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:42.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.015+0000 7f6418a00640 1 --2- 172.21.15.31:0/2238557455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6414108db0 0x7f6414109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:42.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.015+0000 7f6418a00640 1 --2- 172.21.15.31:0/2238557455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6414107bc0 0x7f6414107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:42.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.015+0000 7f6418a00640 1 -- 172.21.15.31:0/2238557455 >> 172.21.15.31:0/2238557455 conn(0x7f6414076b60 msgr2=0x7f6414078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:42.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.015+0000 7f6418a00640 1 -- 172.21.15.31:0/2238557455 shutdown_connections 2024-03-21T10:34:42.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.015+0000 7f6418a00640 1 -- 172.21.15.31:0/2238557455 wait complete. 2024-03-21T10:34:42.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.016+0000 7f6418a00640 1 Processor -- start 2024-03-21T10:34:42.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.016+0000 7f6418a00640 1 -- start start 2024-03-21T10:34:42.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.016+0000 7f6418a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6414107bc0 0x7f641419de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:42.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.016+0000 7f6418a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6414108db0 0x7f641419e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:42.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.016+0000 7f6418a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6414109770 0x7f64141a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:42.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.016+0000 7f6418a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6414113840 con 0x7f6414107bc0 2024-03-21T10:34:42.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.016+0000 7f6418a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64141136c0 con 0x7f6414109770 2024-03-21T10:34:42.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.016+0000 7f6418a00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64141139c0 con 0x7f6414108db0 2024-03-21T10:34:42.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.017+0000 7f6413400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6414107bc0 0x7f641419de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:42.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.017+0000 7f6412a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6414108db0 0x7f641419e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:42.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.017+0000 7f6413400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6414107bc0 0x7f641419de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58472/0 (socket says 172.21.15.31:58472) 2024-03-21T10:34:42.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.017+0000 7f6413400640 1 -- 172.21.15.31:0/3229249901 learned_addr learned my addr 172.21.15.31:0/3229249901 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:34:42.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.017+0000 7f6413e00640 1 --2- 172.21.15.31:0/3229249901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6414109770 0x7f64141a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:42.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.017+0000 7f6413400640 1 -- 172.21.15.31:0/3229249901 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6414108db0 msgr2=0x7f641419e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:42.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.017+0000 7f6413400640 1 --2- 172.21.15.31:0/3229249901 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6414108db0 0x7f641419e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:42.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.017+0000 7f6413400640 1 -- 172.21.15.31:0/3229249901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6414109770 msgr2=0x7f64141a53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:42.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.017+0000 7f6413400640 1 --2- 172.21.15.31:0/3229249901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6414109770 0x7f64141a53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:42.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.017+0000 7f6413400640 1 -- 172.21.15.31:0/3229249901 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64141a5a20 con 0x7f6414107bc0 2024-03-21T10:34:42.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.017+0000 7f6412a00640 1 --2- 172.21.15.31:0/3229249901 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6414108db0 0x7f641419e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:34:42.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.017+0000 7f6413e00640 1 --2- 172.21.15.31:0/3229249901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6414109770 0x7f64141a53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:34:42.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.017+0000 7f6413400640 1 --2- 172.21.15.31:0/3229249901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6414107bc0 0x7f641419de00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f640400e9c0 tx=0x7f640400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:42.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.018+0000 7f640be00640 1 -- 172.21.15.31:0/3229249901 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640400cd80 con 0x7f6414107bc0 2024-03-21T10:34:42.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.018+0000 7f640be00640 1 -- 172.21.15.31:0/3229249901 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64040035f0 con 0x7f6414107bc0 2024-03-21T10:34:42.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.018+0000 7f6418a00640 1 -- 172.21.15.31:0/3229249901 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64141a5d10 con 0x7f6414107bc0 2024-03-21T10:34:42.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.018+0000 7f640be00640 1 -- 172.21.15.31:0/3229249901 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6404010690 con 0x7f6414107bc0 2024-03-21T10:34:42.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.018+0000 7f6418a00640 1 -- 172.21.15.31:0/3229249901 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64141a61b0 con 0x7f6414107bc0 2024-03-21T10:34:42.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.019+0000 7f6418a00640 1 -- 172.21.15.31:0/3229249901 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f641410ee80 con 0x7f6414107bc0 2024-03-21T10:34:42.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.021+0000 7f640be00640 1 -- 172.21.15.31:0/3229249901 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f64040037a0 con 0x7f6414107bc0 2024-03-21T10:34:42.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.023+0000 7f640be00640 1 --2- 172.21.15.31:0/3229249901 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f63f80a2f10 0x7f63f80a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:42.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.023+0000 7f640be00640 1 -- 172.21.15.31:0/3229249901 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f64040cf3e0 con 0x7f6414107bc0 2024-03-21T10:34:42.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.023+0000 7f6412a00640 1 --2- 172.21.15.31:0/3229249901 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f63f80a2f10 0x7f63f80a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:42.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.024+0000 7f6412a00640 1 --2- 172.21.15.31:0/3229249901 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f63f80a2f10 0x7f63f80a53d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f641419f310 tx=0x7f6400005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:42.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.025+0000 7f640be00640 1 -- 172.21.15.31:0/3229249901 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f640409c740 con 0x7f6414107bc0 2024-03-21T10:34:42.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.271+0000 7f6418a00640 1 -- 172.21.15.31:0/3229249901 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f641402d070 con 0x7f6414107bc0 2024-03-21T10:34:42.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.272+0000 7f640be00640 1 -- 172.21.15.31:0/3229249901 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5277 (secure 0 0 0) 0x7f640409c040 con 0x7f6414107bc0 2024-03-21T10:34:42.272 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:34:42.272 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7}],"filesystems":[]} 2024-03-21T10:34:42.272 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 7 2024-03-21T10:34:42.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.275+0000 7f6418a00640 1 -- 172.21.15.31:0/3229249901 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f63f80a2f10 msgr2=0x7f63f80a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:42.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.275+0000 7f6418a00640 1 --2- 172.21.15.31:0/3229249901 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f63f80a2f10 0x7f63f80a53d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f641419f310 tx=0x7f6400005f90 comp rx=0 tx=0).stop 2024-03-21T10:34:42.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.275+0000 7f6418a00640 1 -- 172.21.15.31:0/3229249901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6414107bc0 msgr2=0x7f641419de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:42.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.275+0000 7f6418a00640 1 --2- 172.21.15.31:0/3229249901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6414107bc0 0x7f641419de00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f640400e9c0 tx=0x7f640400ee80 comp rx=0 tx=0).stop 2024-03-21T10:34:42.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.275+0000 7f6418a00640 1 -- 172.21.15.31:0/3229249901 shutdown_connections 2024-03-21T10:34:42.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.275+0000 7f6418a00640 1 --2- 172.21.15.31:0/3229249901 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f63f80a2f10 0x7f63f80a53d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:42.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.275+0000 7f6418a00640 1 --2- 172.21.15.31:0/3229249901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6414109770 0x7f64141a53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:42.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.275+0000 7f6418a00640 1 --2- 172.21.15.31:0/3229249901 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6414108db0 0x7f641419e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:42.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.275+0000 7f6418a00640 1 --2- 172.21.15.31:0/3229249901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6414107bc0 0x7f641419de00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:42.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.275+0000 7f6418a00640 1 -- 172.21.15.31:0/3229249901 >> 172.21.15.31:0/3229249901 conn(0x7f6414076b60 msgr2=0x7f641410c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:42.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.275+0000 7f6418a00640 1 -- 172.21.15.31:0/3229249901 shutdown_connections 2024-03-21T10:34:42.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:42.276+0000 7f6418a00640 1 -- 172.21.15.31:0/3229249901 wait complete. 2024-03-21T10:34:42.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:42 smithi031 ceph-mon[31673]: Deploying daemon mds.c on smithi070 2024-03-21T10:34:42.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:42 smithi031 ceph-mon[31673]: mds.? [v2:172.21.15.70:6834/729664889,v1:172.21.15.70:6835/729664889] up:boot 2024-03-21T10:34:42.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:42 smithi031 ceph-mon[31673]: fsmap 6 up:standby 2024-03-21T10:34:42.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:42 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-21T10:34:42.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:42 smithi070 ceph-mon[36330]: Deploying daemon mds.c on smithi070 2024-03-21T10:34:42.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:42 smithi070 ceph-mon[36330]: mds.? [v2:172.21.15.70:6834/729664889,v1:172.21.15.70:6835/729664889] up:boot 2024-03-21T10:34:42.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:42 smithi070 ceph-mon[36330]: fsmap 6 up:standby 2024-03-21T10:34:42.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:42 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-21T10:34:42.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:42 smithi040 ceph-mon[36154]: Deploying daemon mds.c on smithi070 2024-03-21T10:34:42.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:42 smithi040 ceph-mon[36154]: mds.? [v2:172.21.15.70:6834/729664889,v1:172.21.15.70:6835/729664889] up:boot 2024-03-21T10:34:42.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:42 smithi040 ceph-mon[36154]: fsmap 6 up:standby 2024-03-21T10:34:42.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:42 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-21T10:34:43.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:43 smithi031 ceph-mon[31673]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:43.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:43 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3229249901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:43.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:43 smithi070 ceph-mon[36330]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:43.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:43 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3229249901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:43.574 INFO:journalctl@ceph.mds.c.smithi070.stdout:Mar 21 10:34:43 smithi070 systemd[1]: Starting Ceph mds.c for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:34:43.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:43 smithi040 ceph-mon[36154]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:43.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:43 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3229249901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:43.872 INFO:journalctl@ceph.mds.c.smithi070.stdout:Mar 21 10:34:43 smithi070 podman[67028]: 2024-03-21 10:34:43.512948129 +0000 UTC m=+0.017901642 image pull 237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232 2024-03-21T10:34:43.872 INFO:journalctl@ceph.mds.c.smithi070.stdout:Mar 21 10:34:43 smithi070 podman[67028]: 2024-03-21 10:34:43.743766462 +0000 UTC m=+0.248719965 container create 5b129a97f390ccc86b4010dc69cb0a2af7be775fdd4e579bb54d2124a45b12be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-21T10:34:44.296 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:34:44.324 INFO:journalctl@ceph.mds.c.smithi070.stdout:Mar 21 10:34:43 smithi070 podman[67028]: 2024-03-21 10:34:43.988618964 +0000 UTC m=+0.493572482 container init 5b129a97f390ccc86b4010dc69cb0a2af7be775fdd4e579bb54d2124a45b12be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-c, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=) 2024-03-21T10:34:44.324 INFO:journalctl@ceph.mds.c.smithi070.stdout:Mar 21 10:34:43 smithi070 podman[67028]: 2024-03-21 10:34:43.991159943 +0000 UTC m=+0.496113445 container start 5b129a97f390ccc86b4010dc69cb0a2af7be775fdd4e579bb54d2124a45b12be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-21T10:34:44.324 INFO:journalctl@ceph.mds.c.smithi070.stdout:Mar 21 10:34:44 smithi070 ceph-mds[67047]: -- 172.21.15.70:0/1686390520 <== mon.1 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e64aca1a40 con 0x55e64ac74c00 2024-03-21T10:34:44.324 INFO:journalctl@ceph.mds.c.smithi070.stdout:Mar 21 10:34:44 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-c[67043]: starting mds.c at 2024-03-21T10:34:44.324 INFO:journalctl@ceph.mds.c.smithi070.stdout:Mar 21 10:34:44 smithi070 bash[67028]: 5b129a97f390ccc86b4010dc69cb0a2af7be775fdd4e579bb54d2124a45b12be 2024-03-21T10:34:44.324 INFO:journalctl@ceph.mds.c.smithi070.stdout:Mar 21 10:34:44 smithi070 systemd[1]: Started Ceph mds.c for 3847408c-e76d-11ee-95cd-87774f69a715. 2024-03-21T10:34:44.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.525+0000 7faa6aa00640 1 -- 172.21.15.31:0/3845790368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa64077c40 msgr2=0x7faa640784e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:44.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.525+0000 7faa6aa00640 1 --2- 172.21.15.31:0/3845790368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa64077c40 0x7faa640784e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faa5c009ca0 tx=0x7faa5c02c100 comp rx=0 tx=0).stop 2024-03-21T10:34:44.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.525+0000 7faa6aa00640 1 -- 172.21.15.31:0/3845790368 shutdown_connections 2024-03-21T10:34:44.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.525+0000 7faa6aa00640 1 --2- 172.21.15.31:0/3845790368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa64077c40 0x7faa640784e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:44.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.525+0000 7faa6aa00640 1 --2- 172.21.15.31:0/3845790368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa640772a0 0x7faa64077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:44.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.525+0000 7faa6aa00640 1 --2- 172.21.15.31:0/3845790368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa64076060 0x7faa64076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:44.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.525+0000 7faa6aa00640 1 -- 172.21.15.31:0/3845790368 >> 172.21.15.31:0/3845790368 conn(0x7faa640fee60 msgr2=0x7faa64101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:44.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.525+0000 7faa6aa00640 1 -- 172.21.15.31:0/3845790368 shutdown_connections 2024-03-21T10:34:44.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.525+0000 7faa6aa00640 1 -- 172.21.15.31:0/3845790368 wait complete. 2024-03-21T10:34:44.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.526+0000 7faa6aa00640 1 Processor -- start 2024-03-21T10:34:44.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.526+0000 7faa6aa00640 1 -- start start 2024-03-21T10:34:44.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.527+0000 7faa6aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa64076060 0x7faa6419df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:44.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.527+0000 7faa6aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa640772a0 0x7faa6419e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:44.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.527+0000 7faa6aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa64077c40 0x7faa641a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:44.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.527+0000 7faa6aa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa64113960 con 0x7faa64077c40 2024-03-21T10:34:44.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.527+0000 7faa6aa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa641137e0 con 0x7faa64076060 2024-03-21T10:34:44.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.527+0000 7faa6aa00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa64113ae0 con 0x7faa640772a0 2024-03-21T10:34:44.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.527+0000 7faa63e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa64076060 0x7faa6419df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:44.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.527+0000 7faa68a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa64077c40 0x7faa641a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:44.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.527+0000 7faa63e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa64076060 0x7faa6419df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60926/0 (socket says 172.21.15.31:60926) 2024-03-21T10:34:44.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.527+0000 7faa63e00640 1 -- 172.21.15.31:0/2876322361 learned_addr learned my addr 172.21.15.31:0/2876322361 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:34:44.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.527+0000 7faa68a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa64077c40 0x7faa641a5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58480/0 (socket says 172.21.15.31:58480) 2024-03-21T10:34:44.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.527+0000 7faa63400640 1 --2- 172.21.15.31:0/2876322361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa640772a0 0x7faa6419e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:44.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.527+0000 7faa63e00640 1 -- 172.21.15.31:0/2876322361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa640772a0 msgr2=0x7faa6419e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:44.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.527+0000 7faa63e00640 1 --2- 172.21.15.31:0/2876322361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa640772a0 0x7faa6419e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:44.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.527+0000 7faa63e00640 1 -- 172.21.15.31:0/2876322361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa64077c40 msgr2=0x7faa641a5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:44.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.527+0000 7faa63e00640 1 --2- 172.21.15.31:0/2876322361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa64077c40 0x7faa641a5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:44.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.527+0000 7faa63e00640 1 -- 172.21.15.31:0/2876322361 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa641a5b70 con 0x7faa64076060 2024-03-21T10:34:44.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.527+0000 7faa68a00640 1 --2- 172.21.15.31:0/2876322361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa64077c40 0x7faa641a5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:34:44.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.528+0000 7faa63e00640 1 --2- 172.21.15.31:0/2876322361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa64076060 0x7faa6419df50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7faa5400e9f0 tx=0x7faa5400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:44.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.528+0000 7faa60c00640 1 -- 172.21.15.31:0/2876322361 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5400cd80 con 0x7faa64076060 2024-03-21T10:34:44.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.528+0000 7faa60c00640 1 -- 172.21.15.31:0/2876322361 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faa54005500 con 0x7faa64076060 2024-03-21T10:34:44.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.528+0000 7faa60c00640 1 -- 172.21.15.31:0/2876322361 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa54010690 con 0x7faa64076060 2024-03-21T10:34:44.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.528+0000 7faa6aa00640 1 -- 172.21.15.31:0/2876322361 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa641a5e60 con 0x7faa64076060 2024-03-21T10:34:44.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.529+0000 7faa6aa00640 1 -- 172.21.15.31:0/2876322361 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa641a6210 con 0x7faa64076060 2024-03-21T10:34:44.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.530+0000 7faa6aa00640 1 -- 172.21.15.31:0/2876322361 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa64110380 con 0x7faa64076060 2024-03-21T10:34:44.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.531+0000 7faa60c00640 1 -- 172.21.15.31:0/2876322361 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7faa540046c0 con 0x7faa64076060 2024-03-21T10:34:44.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.533+0000 7faa60c00640 1 --2- 172.21.15.31:0/2876322361 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7faa480a2f20 0x7faa480a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:44.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.533+0000 7faa60c00640 1 -- 172.21.15.31:0/2876322361 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7faa540cf3b0 con 0x7faa64076060 2024-03-21T10:34:44.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.533+0000 7faa63400640 1 --2- 172.21.15.31:0/2876322361 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7faa480a2f20 0x7faa480a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:44.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.534+0000 7faa63400640 1 --2- 172.21.15.31:0/2876322361 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7faa480a2f20 0x7faa480a53e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7faa500095f0 tx=0x7faa50005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:44.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.537+0000 7faa60c00640 1 -- 172.21.15.31:0/2876322361 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faa5409c5e0 con 0x7faa64076060 2024-03-21T10:34:44.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.784+0000 7faa6aa00640 1 -- 172.21.15.31:0/2876322361 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa6402d070 con 0x7faa64076060 2024-03-21T10:34:44.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.785+0000 7faa60c00640 1 -- 172.21.15.31:0/2876322361 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6068 (secure 0 0 0) 0x7faa540bc0d0 con 0x7faa64076060 2024-03-21T10:34:44.785 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:34:44.785 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-21T10:34:44.785 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 8 2024-03-21T10:34:44.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.789+0000 7faa6aa00640 1 -- 172.21.15.31:0/2876322361 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7faa480a2f20 msgr2=0x7faa480a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:44.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.789+0000 7faa6aa00640 1 --2- 172.21.15.31:0/2876322361 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7faa480a2f20 0x7faa480a53e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7faa500095f0 tx=0x7faa50005f90 comp rx=0 tx=0).stop 2024-03-21T10:34:44.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.789+0000 7faa6aa00640 1 -- 172.21.15.31:0/2876322361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa64076060 msgr2=0x7faa6419df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:44.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.789+0000 7faa6aa00640 1 --2- 172.21.15.31:0/2876322361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa64076060 0x7faa6419df50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7faa5400e9f0 tx=0x7faa5400edf0 comp rx=0 tx=0).stop 2024-03-21T10:34:44.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.789+0000 7faa6aa00640 1 -- 172.21.15.31:0/2876322361 shutdown_connections 2024-03-21T10:34:44.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.789+0000 7faa6aa00640 1 --2- 172.21.15.31:0/2876322361 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7faa480a2f20 0x7faa480a53e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:44.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.789+0000 7faa6aa00640 1 --2- 172.21.15.31:0/2876322361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa64077c40 0x7faa641a5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:44.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.789+0000 7faa6aa00640 1 --2- 172.21.15.31:0/2876322361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa640772a0 0x7faa6419e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:44.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.789+0000 7faa6aa00640 1 --2- 172.21.15.31:0/2876322361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa64076060 0x7faa6419df50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:44.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.789+0000 7faa6aa00640 1 -- 172.21.15.31:0/2876322361 >> 172.21.15.31:0/2876322361 conn(0x7faa640fee60 msgr2=0x7faa64100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:44.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.789+0000 7faa6aa00640 1 -- 172.21.15.31:0/2876322361 shutdown_connections 2024-03-21T10:34:44.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:44.789+0000 7faa6aa00640 1 -- 172.21.15.31:0/2876322361 wait complete. 2024-03-21T10:34:45.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:45 smithi040 ceph-mon[36154]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:45.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:45 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:45.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:45 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:45.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:45 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:45.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:45 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:45.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:45 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:45.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:45 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:45.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:45 smithi040 ceph-mon[36154]: Deploying daemon mds.h on smithi040 2024-03-21T10:34:45.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:45 smithi040 ceph-mon[36154]: mds.? [v2:172.21.15.70:6836/3650805915,v1:172.21.15.70:6837/3650805915] up:boot 2024-03-21T10:34:45.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:45 smithi040 ceph-mon[36154]: fsmap 7 up:standby 2024-03-21T10:34:45.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:45 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-21T10:34:45.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:45 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2876322361' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:45.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:45 smithi031 ceph-mon[31673]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:45.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:45 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:45.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:45 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:45.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:45 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:45.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:45 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:45.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:45 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:45.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:45 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:45.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:45 smithi031 ceph-mon[31673]: Deploying daemon mds.h on smithi040 2024-03-21T10:34:45.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:45 smithi031 ceph-mon[31673]: mds.? [v2:172.21.15.70:6836/3650805915,v1:172.21.15.70:6837/3650805915] up:boot 2024-03-21T10:34:45.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:45 smithi031 ceph-mon[31673]: fsmap 7 up:standby 2024-03-21T10:34:45.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:45 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-21T10:34:45.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:45 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2876322361' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:45.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:45 smithi070 ceph-mon[36330]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:45.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:45 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:45.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:45 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:45.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:45 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:45.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:45 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:45.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:45 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:45.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:45 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:45.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:45 smithi070 ceph-mon[36330]: Deploying daemon mds.h on smithi040 2024-03-21T10:34:45.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:45 smithi070 ceph-mon[36330]: mds.? [v2:172.21.15.70:6836/3650805915,v1:172.21.15.70:6837/3650805915] up:boot 2024-03-21T10:34:45.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:45 smithi070 ceph-mon[36330]: fsmap 7 up:standby 2024-03-21T10:34:45.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:45 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-21T10:34:45.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:45 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2876322361' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:46.653 INFO:journalctl@ceph.mds.h.smithi040.stdout:Mar 21 10:34:46 smithi040 systemd[1]: Starting Ceph mds.h for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:34:46.809 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:34:46.932 INFO:journalctl@ceph.mds.h.smithi040.stdout:Mar 21 10:34:46 smithi040 podman[66121]: 2024-03-21 10:34:46.558492213 +0000 UTC m=+0.018343468 image pull 237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232 2024-03-21T10:34:46.932 INFO:journalctl@ceph.mds.h.smithi040.stdout:Mar 21 10:34:46 smithi040 podman[66121]: 2024-03-21 10:34:46.677783252 +0000 UTC m=+0.137634534 container create 101a42563d4bad5bc0ab4fbc3519b5f8d062e4b74bcbdc7253d8ee39afc0da1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-h, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-21T10:34:47.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.044+0000 7f6805a00640 1 -- 172.21.15.31:0/1762893900 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6800076060 msgr2=0x7f6800076480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:47.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.044+0000 7f6805a00640 1 --2- 172.21.15.31:0/1762893900 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6800076060 0x7f6800076480 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f67f0009be0 tx=0x7f67f002bef0 comp rx=0 tx=0).stop 2024-03-21T10:34:47.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.045+0000 7f6805a00640 1 -- 172.21.15.31:0/1762893900 shutdown_connections 2024-03-21T10:34:47.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.045+0000 7f6805a00640 1 --2- 172.21.15.31:0/1762893900 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6800077c40 0x7f68000784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:47.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.045+0000 7f6805a00640 1 --2- 172.21.15.31:0/1762893900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68000772a0 0x7f6800077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:47.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.045+0000 7f6805a00640 1 --2- 172.21.15.31:0/1762893900 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6800076060 0x7f6800076480 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:47.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.045+0000 7f6805a00640 1 -- 172.21.15.31:0/1762893900 >> 172.21.15.31:0/1762893900 conn(0x7f68000fef00 msgr2=0x7f6800101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:47.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.045+0000 7f6805a00640 1 -- 172.21.15.31:0/1762893900 shutdown_connections 2024-03-21T10:34:47.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.045+0000 7f6805a00640 1 -- 172.21.15.31:0/1762893900 wait complete. 2024-03-21T10:34:47.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.046+0000 7f6805a00640 1 Processor -- start 2024-03-21T10:34:47.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.046+0000 7f6805a00640 1 -- start start 2024-03-21T10:34:47.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.046+0000 7f6805a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6800076060 0x7f680019dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:47.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.046+0000 7f6805a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68000772a0 0x7f680019e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:47.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.046+0000 7f6805a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6800077c40 0x7f68001a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:47.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.046+0000 7f6805a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6800113970 con 0x7f68000772a0 2024-03-21T10:34:47.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.046+0000 7f6805a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68001137f0 con 0x7f6800076060 2024-03-21T10:34:47.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.046+0000 7f6805a00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6800113af0 con 0x7f6800077c40 2024-03-21T10:34:47.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.047+0000 7f67fe000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68000772a0 0x7f680019e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:47.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.047+0000 7f67ff400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6800077c40 0x7f68001a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:47.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.047+0000 7f67fea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6800076060 0x7f680019dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:47.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.047+0000 7f67ff400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6800077c40 0x7f68001a5580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:43100/0 (socket says 172.21.15.31:43100) 2024-03-21T10:34:47.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.047+0000 7f67ff400640 1 -- 172.21.15.31:0/3852474099 learned_addr learned my addr 172.21.15.31:0/3852474099 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:34:47.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.047+0000 7f67fe000640 1 -- 172.21.15.31:0/3852474099 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6800077c40 msgr2=0x7f68001a5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:47.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.047+0000 7f67fe000640 1 --2- 172.21.15.31:0/3852474099 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6800077c40 0x7f68001a5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:47.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.047+0000 7f67fe000640 1 -- 172.21.15.31:0/3852474099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6800076060 msgr2=0x7f680019dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:47.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.047+0000 7f67fe000640 1 --2- 172.21.15.31:0/3852474099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6800076060 0x7f680019dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:47.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.047+0000 7f67fe000640 1 -- 172.21.15.31:0/3852474099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68001a5bf0 con 0x7f68000772a0 2024-03-21T10:34:47.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.047+0000 7f67fea00640 1 --2- 172.21.15.31:0/3852474099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6800076060 0x7f680019dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:34:47.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.048+0000 7f67fe000640 1 --2- 172.21.15.31:0/3852474099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68000772a0 0x7f680019e510 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f67e800e9f0 tx=0x7f67e800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:47.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.048+0000 7f67f7400640 1 -- 172.21.15.31:0/3852474099 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e800ccc0 con 0x7f68000772a0 2024-03-21T10:34:47.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.048+0000 7f67f7400640 1 -- 172.21.15.31:0/3852474099 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67e8010910 con 0x7f68000772a0 2024-03-21T10:34:47.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.048+0000 7f6805a00640 1 -- 172.21.15.31:0/3852474099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68001a5ee0 con 0x7f68000772a0 2024-03-21T10:34:47.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.048+0000 7f67f7400640 1 -- 172.21.15.31:0/3852474099 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e8018960 con 0x7f68000772a0 2024-03-21T10:34:47.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.048+0000 7f6805a00640 1 -- 172.21.15.31:0/3852474099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68001a63a0 con 0x7f68000772a0 2024-03-21T10:34:47.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.050+0000 7f6805a00640 1 -- 172.21.15.31:0/3852474099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6800110470 con 0x7f68000772a0 2024-03-21T10:34:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.052+0000 7f67f7400640 1 -- 172.21.15.31:0/3852474099 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f67e8010ac0 con 0x7f68000772a0 2024-03-21T10:34:47.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.053+0000 7f67f7400640 1 --2- 172.21.15.31:0/3852474099 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f67e40a2f20 0x7f67e40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:47.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.053+0000 7f67f7400640 1 -- 172.21.15.31:0/3852474099 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f67e80cf400 con 0x7f68000772a0 2024-03-21T10:34:47.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.053+0000 7f67fea00640 1 --2- 172.21.15.31:0/3852474099 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f67e40a2f20 0x7f67e40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.054+0000 7f67fea00640 1 --2- 172.21.15.31:0/3852474099 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f67e40a2f20 0x7f67e40a53e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f67f0006010 tx=0x7f67f0000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:47.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.058+0000 7f67f7400640 1 -- 172.21.15.31:0/3852474099 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f67e809c630 con 0x7f68000772a0 2024-03-21T10:34:47.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.308+0000 7f6805a00640 1 -- 172.21.15.31:0/3852474099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6800062340 con 0x7f68000772a0 2024-03-21T10:34:47.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.309+0000 7f67f7400640 1 -- 172.21.15.31:0/3852474099 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6859 (secure 0 0 0) 0x7f67e809bf30 con 0x7f68000772a0 2024-03-21T10:34:47.307 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:34:47.307 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24539,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-21T10:34:47.308 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 9 2024-03-21T10:34:47.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.311+0000 7f6805a00640 1 -- 172.21.15.31:0/3852474099 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f67e40a2f20 msgr2=0x7f67e40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:47.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.311+0000 7f6805a00640 1 --2- 172.21.15.31:0/3852474099 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f67e40a2f20 0x7f67e40a53e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f67f0006010 tx=0x7f67f0000f40 comp rx=0 tx=0).stop 2024-03-21T10:34:47.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.311+0000 7f6805a00640 1 -- 172.21.15.31:0/3852474099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68000772a0 msgr2=0x7f680019e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:47.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.311+0000 7f6805a00640 1 --2- 172.21.15.31:0/3852474099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68000772a0 0x7f680019e510 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f67e800e9f0 tx=0x7f67e800eeb0 comp rx=0 tx=0).stop 2024-03-21T10:34:47.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.312+0000 7f6805a00640 1 -- 172.21.15.31:0/3852474099 shutdown_connections 2024-03-21T10:34:47.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.312+0000 7f6805a00640 1 --2- 172.21.15.31:0/3852474099 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f67e40a2f20 0x7f67e40a53e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:47.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.312+0000 7f6805a00640 1 --2- 172.21.15.31:0/3852474099 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6800077c40 0x7f68001a5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:47.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.312+0000 7f6805a00640 1 --2- 172.21.15.31:0/3852474099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68000772a0 0x7f680019e510 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:47.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.312+0000 7f6805a00640 1 --2- 172.21.15.31:0/3852474099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6800076060 0x7f680019dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:47.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.312+0000 7f6805a00640 1 -- 172.21.15.31:0/3852474099 >> 172.21.15.31:0/3852474099 conn(0x7f68000fef00 msgr2=0x7f68001005e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:47.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.312+0000 7f6805a00640 1 -- 172.21.15.31:0/3852474099 shutdown_connections 2024-03-21T10:34:47.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:47.312+0000 7f6805a00640 1 -- 172.21.15.31:0/3852474099 wait complete. 2024-03-21T10:34:47.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:47 smithi040 ceph-mon[36154]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:47.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:47 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:47.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:47 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:47.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:47 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:47.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:47 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:47.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:47 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:47.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:47 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:47.432 INFO:journalctl@ceph.mds.h.smithi040.stdout:Mar 21 10:34:46 smithi040 podman[66121]: 2024-03-21 10:34:46.939107654 +0000 UTC m=+0.398958908 container init 101a42563d4bad5bc0ab4fbc3519b5f8d062e4b74bcbdc7253d8ee39afc0da1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-h, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_BRANCH=HEAD) 2024-03-21T10:34:47.433 INFO:journalctl@ceph.mds.h.smithi040.stdout:Mar 21 10:34:46 smithi040 podman[66121]: 2024-03-21 10:34:46.941868866 +0000 UTC m=+0.401720104 container start 101a42563d4bad5bc0ab4fbc3519b5f8d062e4b74bcbdc7253d8ee39afc0da1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-h, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T10:34:47.433 INFO:journalctl@ceph.mds.h.smithi040.stdout:Mar 21 10:34:46 smithi040 ceph-mds[66140]: -- 172.21.15.40:0/60263208 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560e92d13a40 con 0x560e92ce7000 2024-03-21T10:34:47.433 INFO:journalctl@ceph.mds.h.smithi040.stdout:Mar 21 10:34:46 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-h[66136]: starting mds.h at 2024-03-21T10:34:47.433 INFO:journalctl@ceph.mds.h.smithi040.stdout:Mar 21 10:34:47 smithi040 bash[66121]: 101a42563d4bad5bc0ab4fbc3519b5f8d062e4b74bcbdc7253d8ee39afc0da1f 2024-03-21T10:34:47.433 INFO:journalctl@ceph.mds.h.smithi040.stdout:Mar 21 10:34:47 smithi040 systemd[1]: Started Ceph mds.h for 3847408c-e76d-11ee-95cd-87774f69a715. 2024-03-21T10:34:47.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:47 smithi031 ceph-mon[31673]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:47.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:47 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:47.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:47 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:47.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:47 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:47.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:47 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:47.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:47 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:47.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:47 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:47.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:47 smithi070 ceph-mon[36330]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:47.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:47 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:47.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:47 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:47.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:47 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:47.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:47 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:47.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:47 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:47.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:47 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:48.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:48 smithi031 ceph-mon[31673]: Deploying daemon mds.i on smithi070 2024-03-21T10:34:48.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:48 smithi031 ceph-mon[31673]: mds.? [v2:172.21.15.40:6836/4121707657,v1:172.21.15.40:6837/4121707657] up:boot 2024-03-21T10:34:48.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:48 smithi031 ceph-mon[31673]: fsmap 8 up:standby 2024-03-21T10:34:48.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:48 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-21T10:34:48.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:48 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3852474099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:48.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:48 smithi070 ceph-mon[36330]: Deploying daemon mds.i on smithi070 2024-03-21T10:34:48.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:48 smithi070 ceph-mon[36330]: mds.? [v2:172.21.15.40:6836/4121707657,v1:172.21.15.40:6837/4121707657] up:boot 2024-03-21T10:34:48.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:48 smithi070 ceph-mon[36330]: fsmap 8 up:standby 2024-03-21T10:34:48.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:48 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-21T10:34:48.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:48 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3852474099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:48.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:48 smithi040 ceph-mon[36154]: Deploying daemon mds.i on smithi070 2024-03-21T10:34:48.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:48 smithi040 ceph-mon[36154]: mds.? [v2:172.21.15.40:6836/4121707657,v1:172.21.15.40:6837/4121707657] up:boot 2024-03-21T10:34:48.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:48 smithi040 ceph-mon[36154]: fsmap 8 up:standby 2024-03-21T10:34:48.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:48 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-21T10:34:48.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:48 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3852474099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:49.331 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:34:49.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:49 smithi031 ceph-mon[31673]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:49.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.564+0000 7f0668800640 1 -- 172.21.15.31:0/1723755528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0660076060 msgr2=0x7f0660076480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:49.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.564+0000 7f0668800640 1 --2- 172.21.15.31:0/1723755528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0660076060 0x7f0660076480 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0658009be0 tx=0x7f065802bef0 comp rx=0 tx=0).stop 2024-03-21T10:34:49.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.564+0000 7f0668800640 1 -- 172.21.15.31:0/1723755528 shutdown_connections 2024-03-21T10:34:49.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.564+0000 7f0668800640 1 --2- 172.21.15.31:0/1723755528 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0660077c40 0x7f06600784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:49.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.564+0000 7f0668800640 1 --2- 172.21.15.31:0/1723755528 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06600772a0 0x7f0660077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:49.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.564+0000 7f0668800640 1 --2- 172.21.15.31:0/1723755528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0660076060 0x7f0660076480 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:49.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.564+0000 7f0668800640 1 -- 172.21.15.31:0/1723755528 >> 172.21.15.31:0/1723755528 conn(0x7f06600feeb0 msgr2=0x7f0660101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:49.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.564+0000 7f0668800640 1 -- 172.21.15.31:0/1723755528 shutdown_connections 2024-03-21T10:34:49.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.564+0000 7f0668800640 1 -- 172.21.15.31:0/1723755528 wait complete. 2024-03-21T10:34:49.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.565+0000 7f0668800640 1 Processor -- start 2024-03-21T10:34:49.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.566+0000 7f0668800640 1 -- start start 2024-03-21T10:34:49.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.566+0000 7f0668800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0660076060 0x7f066019e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:49.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.566+0000 7f0668800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06600772a0 0x7f066019e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:49.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.566+0000 7f0668800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0660077c40 0x7f06601a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:49.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.566+0000 7f0668800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0660113980 con 0x7f0660076060 2024-03-21T10:34:49.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.566+0000 7f0668800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0660113800 con 0x7f06600772a0 2024-03-21T10:34:49.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.566+0000 7f0668800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0660113b00 con 0x7f0660077c40 2024-03-21T10:34:49.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.566+0000 7f0665e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0660076060 0x7f066019e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:49.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.566+0000 7f0666800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0660077c40 0x7f06601a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:49.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.567+0000 7f0665e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0660076060 0x7f066019e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33358/0 (socket says 172.21.15.31:33358) 2024-03-21T10:34:49.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.567+0000 7f0665e00640 1 -- 172.21.15.31:0/3883189624 learned_addr learned my addr 172.21.15.31:0/3883189624 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:34:49.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.567+0000 7f0665400640 1 --2- 172.21.15.31:0/3883189624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06600772a0 0x7f066019e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:49.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.567+0000 7f0666800640 1 -- 172.21.15.31:0/3883189624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06600772a0 msgr2=0x7f066019e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:49.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.567+0000 7f0666800640 1 --2- 172.21.15.31:0/3883189624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06600772a0 0x7f066019e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:49.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.567+0000 7f0666800640 1 -- 172.21.15.31:0/3883189624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0660076060 msgr2=0x7f066019e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:49.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.567+0000 7f0666800640 1 --2- 172.21.15.31:0/3883189624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0660076060 0x7f066019e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:49.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.567+0000 7f0666800640 1 -- 172.21.15.31:0/3883189624 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06601a5c20 con 0x7f0660077c40 2024-03-21T10:34:49.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.567+0000 7f0665400640 1 --2- 172.21.15.31:0/3883189624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06600772a0 0x7f066019e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:34:49.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.567+0000 7f0666800640 1 --2- 172.21.15.31:0/3883189624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0660077c40 0x7f06601a55b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f065c005fd0 tx=0x7f065c00ed20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:49.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.567+0000 7f0656a00640 1 -- 172.21.15.31:0/3883189624 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065c00ccc0 con 0x7f0660077c40 2024-03-21T10:34:49.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.567+0000 7f0656a00640 1 -- 172.21.15.31:0/3883189624 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f065c0035f0 con 0x7f0660077c40 2024-03-21T10:34:49.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.567+0000 7f0656a00640 1 -- 172.21.15.31:0/3883189624 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065c00ce60 con 0x7f0660077c40 2024-03-21T10:34:49.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.568+0000 7f0668800640 1 -- 172.21.15.31:0/3883189624 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06601a5f10 con 0x7f0660077c40 2024-03-21T10:34:49.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.568+0000 7f0668800640 1 -- 172.21.15.31:0/3883189624 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06601a63d0 con 0x7f0660077c40 2024-03-21T10:34:49.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.569+0000 7f0668800640 1 -- 172.21.15.31:0/3883189624 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06601104c0 con 0x7f0660077c40 2024-03-21T10:34:49.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.571+0000 7f0656a00640 1 -- 172.21.15.31:0/3883189624 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f065c0037a0 con 0x7f0660077c40 2024-03-21T10:34:49.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.572+0000 7f0656a00640 1 --2- 172.21.15.31:0/3883189624 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f064c0a2fa0 0x7f064c0a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:49.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.572+0000 7f0656a00640 1 -- 172.21.15.31:0/3883189624 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f065c0cf270 con 0x7f0660077c40 2024-03-21T10:34:49.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.572+0000 7f0665e00640 1 --2- 172.21.15.31:0/3883189624 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f064c0a2fa0 0x7f064c0a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:49.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.573+0000 7f0665e00640 1 --2- 172.21.15.31:0/3883189624 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f064c0a2fa0 0x7f064c0a5460 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0658004af0 tx=0x7f0658004a80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:49.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:49 smithi070 ceph-mon[36330]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:49.574 INFO:journalctl@ceph.mds.i.smithi070.stdout:Mar 21 10:34:49 smithi070 systemd[1]: Starting Ceph mds.i for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:34:49.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.576+0000 7f0656a00640 1 -- 172.21.15.31:0/3883189624 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f065c09c4a0 con 0x7f0660077c40 2024-03-21T10:34:49.583 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:49 smithi040 ceph-mon[36154]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:49.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.823+0000 7f0668800640 1 -- 172.21.15.31:0/3883189624 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0660062340 con 0x7f0660077c40 2024-03-21T10:34:49.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.824+0000 7f0656a00640 1 -- 172.21.15.31:0/3883189624 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6859 (secure 0 0 0) 0x7f065c09bda0 con 0x7f0660077c40 2024-03-21T10:34:49.823 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:34:49.823 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24539,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-21T10:34:49.823 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 9 2024-03-21T10:34:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.827+0000 7f0668800640 1 -- 172.21.15.31:0/3883189624 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f064c0a2fa0 msgr2=0x7f064c0a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.827+0000 7f0668800640 1 --2- 172.21.15.31:0/3883189624 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f064c0a2fa0 0x7f064c0a5460 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0658004af0 tx=0x7f0658004a80 comp rx=0 tx=0).stop 2024-03-21T10:34:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.827+0000 7f0668800640 1 -- 172.21.15.31:0/3883189624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0660077c40 msgr2=0x7f06601a55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.827+0000 7f0668800640 1 --2- 172.21.15.31:0/3883189624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0660077c40 0x7f06601a55b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f065c005fd0 tx=0x7f065c00ed20 comp rx=0 tx=0).stop 2024-03-21T10:34:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.827+0000 7f0668800640 1 -- 172.21.15.31:0/3883189624 shutdown_connections 2024-03-21T10:34:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.827+0000 7f0668800640 1 --2- 172.21.15.31:0/3883189624 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f064c0a2fa0 0x7f064c0a5460 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.827+0000 7f0668800640 1 --2- 172.21.15.31:0/3883189624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0660077c40 0x7f06601a55b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.827+0000 7f0668800640 1 --2- 172.21.15.31:0/3883189624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06600772a0 0x7f066019e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.827+0000 7f0668800640 1 --2- 172.21.15.31:0/3883189624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0660076060 0x7f066019e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.827+0000 7f0668800640 1 -- 172.21.15.31:0/3883189624 >> 172.21.15.31:0/3883189624 conn(0x7f06600feeb0 msgr2=0x7f0660100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.827+0000 7f0668800640 1 -- 172.21.15.31:0/3883189624 shutdown_connections 2024-03-21T10:34:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:49.827+0000 7f0668800640 1 -- 172.21.15.31:0/3883189624 wait complete. 2024-03-21T10:34:49.866 INFO:journalctl@ceph.mds.i.smithi070.stdout:Mar 21 10:34:49 smithi070 podman[67339]: 2024-03-21 10:34:49.485779773 +0000 UTC m=+0.017926662 image pull 237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232 2024-03-21T10:34:49.867 INFO:journalctl@ceph.mds.i.smithi070.stdout:Mar 21 10:34:49 smithi070 podman[67339]: 2024-03-21 10:34:49.607799116 +0000 UTC m=+0.139945981 container create e44ffbadd2e579bbcb9f8ff915329f86170c2d03232233ac461891ecb6c68c48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-i, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-21T10:34:49.867 INFO:journalctl@ceph.mds.i.smithi070.stdout:Mar 21 10:34:49 smithi070 podman[67339]: 2024-03-21 10:34:49.86929102 +0000 UTC m=+0.401437903 container init e44ffbadd2e579bbcb9f8ff915329f86170c2d03232233ac461891ecb6c68c48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-i, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-21T10:34:50.210 INFO:journalctl@ceph.mds.i.smithi070.stdout:Mar 21 10:34:49 smithi070 podman[67339]: 2024-03-21 10:34:49.872242056 +0000 UTC m=+0.404388944 container start e44ffbadd2e579bbcb9f8ff915329f86170c2d03232233ac461891ecb6c68c48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-i, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-21T10:34:50.210 INFO:journalctl@ceph.mds.i.smithi070.stdout:Mar 21 10:34:49 smithi070 ceph-mds[67357]: -- 172.21.15.70:0/3775824766 <== mon.1 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c096833a40 con 0x55c096806c00 2024-03-21T10:34:50.211 INFO:journalctl@ceph.mds.i.smithi070.stdout:Mar 21 10:34:49 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-i[67353]: starting mds.i at 2024-03-21T10:34:50.211 INFO:journalctl@ceph.mds.i.smithi070.stdout:Mar 21 10:34:49 smithi070 bash[67339]: e44ffbadd2e579bbcb9f8ff915329f86170c2d03232233ac461891ecb6c68c48 2024-03-21T10:34:50.211 INFO:journalctl@ceph.mds.i.smithi070.stdout:Mar 21 10:34:49 smithi070 systemd[1]: Started Ceph mds.i for 3847408c-e76d-11ee-95cd-87774f69a715. 2024-03-21T10:34:50.211 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:50 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3883189624' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:50.211 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:50 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:50.211 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:50 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:50.211 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:50 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:50.212 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:50 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:50.212 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:50 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:50.212 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:50 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:50.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:50 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3883189624' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:50.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:50 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:50.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:50 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:50.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:50 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:50.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:50 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:50.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:50 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:50.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:50 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:50.599 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:50 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3883189624' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:50.599 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:50 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:50.599 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:50 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:50.599 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:50 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:50.599 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:50 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:50.599 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:50 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:50.599 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:50 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:51.466 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:51 smithi031 ceph-mon[31673]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:51.466 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:51 smithi031 ceph-mon[31673]: Deploying daemon mds.g on smithi031 2024-03-21T10:34:51.466 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:51 smithi031 ceph-mon[31673]: mds.? [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] up:boot 2024-03-21T10:34:51.466 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:51 smithi031 ceph-mon[31673]: fsmap 9 up:standby 2024-03-21T10:34:51.466 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:51 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-21T10:34:51.559 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:51 smithi070 ceph-mon[36330]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:51.559 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:51 smithi070 ceph-mon[36330]: Deploying daemon mds.g on smithi031 2024-03-21T10:34:51.559 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:51 smithi070 ceph-mon[36330]: mds.? [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] up:boot 2024-03-21T10:34:51.559 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:51 smithi070 ceph-mon[36330]: fsmap 9 up:standby 2024-03-21T10:34:51.559 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:51 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-21T10:34:51.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:51 smithi040 ceph-mon[36154]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:51.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:51 smithi040 ceph-mon[36154]: Deploying daemon mds.g on smithi031 2024-03-21T10:34:51.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:51 smithi040 ceph-mon[36154]: mds.? [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] up:boot 2024-03-21T10:34:51.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:51 smithi040 ceph-mon[36154]: fsmap 9 up:standby 2024-03-21T10:34:51.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:51 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-21T10:34:51.845 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:34:52.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.081+0000 7f0649c00640 1 -- 172.21.15.31:0/3265409918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0644103960 msgr2=0x7f0644103d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:52.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.081+0000 7f0649c00640 1 --2- 172.21.15.31:0/3265409918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0644103960 0x7f0644103d80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0634009900 tx=0x7f063402bc50 comp rx=0 tx=0).stop 2024-03-21T10:34:52.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.081+0000 7f0649c00640 1 -- 172.21.15.31:0/3265409918 shutdown_connections 2024-03-21T10:34:52.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.081+0000 7f0649c00640 1 --2- 172.21.15.31:0/3265409918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0644105510 0x7f064410bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:52.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.081+0000 7f0649c00640 1 --2- 172.21.15.31:0/3265409918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0644104b50 0x7f0644104fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:52.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.081+0000 7f0649c00640 1 --2- 172.21.15.31:0/3265409918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0644103960 0x7f0644103d80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:52.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.081+0000 7f0649c00640 1 -- 172.21.15.31:0/3265409918 >> 172.21.15.31:0/3265409918 conn(0x7f06440feeb0 msgr2=0x7f0644101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:52.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.081+0000 7f0649c00640 1 -- 172.21.15.31:0/3265409918 shutdown_connections 2024-03-21T10:34:52.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.081+0000 7f0649c00640 1 -- 172.21.15.31:0/3265409918 wait complete. 2024-03-21T10:34:52.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.083+0000 7f0649c00640 1 Processor -- start 2024-03-21T10:34:52.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.083+0000 7f0649c00640 1 -- start start 2024-03-21T10:34:52.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.083+0000 7f0649c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0644103960 0x7f0644199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:52.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.083+0000 7f0649c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0644104b50 0x7f064419a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:52.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.083+0000 7f0649c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0644105510 0x7f06441a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:52.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.083+0000 7f0649c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064410f5b0 con 0x7f0644104b50 2024-03-21T10:34:52.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.083+0000 7f0649c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064410f430 con 0x7f0644103960 2024-03-21T10:34:52.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.083+0000 7f0649c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064410f730 con 0x7f0644105510 2024-03-21T10:34:52.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.083+0000 7f0642a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0644103960 0x7f0644199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:52.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.083+0000 7f0642000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0644104b50 0x7f064419a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:52.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.084+0000 7f0643400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0644105510 0x7f06441a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:52.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.084+0000 7f0642a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0644103960 0x7f0644199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:54506/0 (socket says 172.21.15.31:54506) 2024-03-21T10:34:52.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.084+0000 7f0642a00640 1 -- 172.21.15.31:0/1932756276 learned_addr learned my addr 172.21.15.31:0/1932756276 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:34:52.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.084+0000 7f0642a00640 1 -- 172.21.15.31:0/1932756276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0644105510 msgr2=0x7f06441a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:52.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.084+0000 7f0642a00640 1 --2- 172.21.15.31:0/1932756276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0644105510 0x7f06441a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:52.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.084+0000 7f0642a00640 1 -- 172.21.15.31:0/1932756276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0644104b50 msgr2=0x7f064419a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:52.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.084+0000 7f0642a00640 1 --2- 172.21.15.31:0/1932756276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0644104b50 0x7f064419a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:52.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.084+0000 7f0642a00640 1 -- 172.21.15.31:0/1932756276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06441a1810 con 0x7f0644103960 2024-03-21T10:34:52.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.084+0000 7f0643400640 1 --2- 172.21.15.31:0/1932756276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0644105510 0x7f06441a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:34:52.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.084+0000 7f0642a00640 1 --2- 172.21.15.31:0/1932756276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0644103960 0x7f0644199bf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f063402c260 tx=0x7f063402c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:52.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.085+0000 7f063b400640 1 -- 172.21.15.31:0/1932756276 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0634041070 con 0x7f0644103960 2024-03-21T10:34:52.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.085+0000 7f063b400640 1 -- 172.21.15.31:0/1932756276 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0634004450 con 0x7f0644103960 2024-03-21T10:34:52.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.085+0000 7f063b400640 1 -- 172.21.15.31:0/1932756276 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0634034440 con 0x7f0644103960 2024-03-21T10:34:52.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.085+0000 7f0649c00640 1 -- 172.21.15.31:0/1932756276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06441a1aa0 con 0x7f0644103960 2024-03-21T10:34:52.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.085+0000 7f0649c00640 1 -- 172.21.15.31:0/1932756276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06441a2040 con 0x7f0644103960 2024-03-21T10:34:52.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.086+0000 7f0649c00640 1 -- 172.21.15.31:0/1932756276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f064410ac20 con 0x7f0644103960 2024-03-21T10:34:52.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.088+0000 7f063b400640 1 -- 172.21.15.31:0/1932756276 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f0634044070 con 0x7f0644103960 2024-03-21T10:34:52.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.089+0000 7f063b400640 1 --2- 172.21.15.31:0/1932756276 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f06240a2fa0 0x7f06240a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:52.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.089+0000 7f063b400640 1 -- 172.21.15.31:0/1932756276 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f06340f09d0 con 0x7f0644103960 2024-03-21T10:34:52.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.089+0000 7f0642000640 1 --2- 172.21.15.31:0/1932756276 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f06240a2fa0 0x7f06240a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:52.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.090+0000 7f0642000640 1 --2- 172.21.15.31:0/1932756276 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f06240a2fa0 0x7f06240a5460 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f064419b100 tx=0x7f062c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:52.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.093+0000 7f063b400640 1 -- 172.21.15.31:0/1932756276 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f06340bd5b0 con 0x7f0644103960 2024-03-21T10:34:52.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.347+0000 7f0649c00640 1 -- 172.21.15.31:0/1932756276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06440622a0 con 0x7f0644103960 2024-03-21T10:34:52.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.348+0000 7f063b400640 1 -- 172.21.15.31:0/1932756276 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7652 (secure 0 0 0) 0x7f06340c1650 con 0x7f0644103960 2024-03-21T10:34:52.346 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:34:52.346 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24539,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-21T10:34:52.347 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 10 2024-03-21T10:34:52.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.350+0000 7f0649c00640 1 -- 172.21.15.31:0/1932756276 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f06240a2fa0 msgr2=0x7f06240a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:52.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.350+0000 7f0649c00640 1 --2- 172.21.15.31:0/1932756276 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f06240a2fa0 0x7f06240a5460 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f064419b100 tx=0x7f062c009500 comp rx=0 tx=0).stop 2024-03-21T10:34:52.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.350+0000 7f0649c00640 1 -- 172.21.15.31:0/1932756276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0644103960 msgr2=0x7f0644199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:52.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.350+0000 7f0649c00640 1 --2- 172.21.15.31:0/1932756276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0644103960 0x7f0644199bf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f063402c260 tx=0x7f063402c670 comp rx=0 tx=0).stop 2024-03-21T10:34:52.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.351+0000 7f0649c00640 1 -- 172.21.15.31:0/1932756276 shutdown_connections 2024-03-21T10:34:52.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.351+0000 7f0649c00640 1 --2- 172.21.15.31:0/1932756276 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f06240a2fa0 0x7f06240a5460 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:52.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.351+0000 7f0649c00640 1 --2- 172.21.15.31:0/1932756276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0644105510 0x7f06441a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:52.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.351+0000 7f0649c00640 1 --2- 172.21.15.31:0/1932756276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0644104b50 0x7f064419a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:52.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.351+0000 7f0649c00640 1 --2- 172.21.15.31:0/1932756276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0644103960 0x7f0644199bf0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:52.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.351+0000 7f0649c00640 1 -- 172.21.15.31:0/1932756276 >> 172.21.15.31:0/1932756276 conn(0x7f06440feeb0 msgr2=0x7f0644107df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:52.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.351+0000 7f0649c00640 1 -- 172.21.15.31:0/1932756276 shutdown_connections 2024-03-21T10:34:52.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:52.351+0000 7f0649c00640 1 -- 172.21.15.31:0/1932756276 wait complete. 2024-03-21T10:34:53.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:53 smithi031 ceph-mon[31673]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:53.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:53 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1932756276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:53.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:53 smithi070 ceph-mon[36330]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:53.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:53 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1932756276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:53.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:53 smithi040 ceph-mon[36154]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:53.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:53 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1932756276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:53.806 INFO:journalctl@ceph.mds.g.smithi031.stdout:Mar 21 10:34:53 smithi031 podman[72326]: 2024-03-21 10:34:53.643972521 +0000 UTC m=+0.204976486 container create cf9cdadc0cb09bcee953696058461416392348d9c278831c3d0d2923976241b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-g, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-21T10:34:54.306 INFO:journalctl@ceph.mds.g.smithi031.stdout:Mar 21 10:34:53 smithi031 podman[72326]: 2024-03-21 10:34:53.956570753 +0000 UTC m=+0.517574724 container init cf9cdadc0cb09bcee953696058461416392348d9c278831c3d0d2923976241b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-g, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-21T10:34:54.306 INFO:journalctl@ceph.mds.g.smithi031.stdout:Mar 21 10:34:53 smithi031 podman[72326]: 2024-03-21 10:34:53.965890713 +0000 UTC m=+0.526894660 container start cf9cdadc0cb09bcee953696058461416392348d9c278831c3d0d2923976241b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-g, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-21T10:34:54.306 INFO:journalctl@ceph.mds.g.smithi031.stdout:Mar 21 10:34:54 smithi031 ceph-mds[72353]: -- 172.21.15.31:0/3064659342 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d33d365a40 con 0x55d33d338c00 2024-03-21T10:34:54.306 INFO:journalctl@ceph.mds.g.smithi031.stdout:Mar 21 10:34:54 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-g[72344]: starting mds.g at 2024-03-21T10:34:54.306 INFO:journalctl@ceph.mds.g.smithi031.stdout:Mar 21 10:34:54 smithi031 bash[72326]: cf9cdadc0cb09bcee953696058461416392348d9c278831c3d0d2923976241b2 2024-03-21T10:34:54.306 INFO:journalctl@ceph.mds.g.smithi031.stdout:Mar 21 10:34:54 smithi031 systemd[1]: Started Ceph mds.g for 3847408c-e76d-11ee-95cd-87774f69a715. 2024-03-21T10:34:54.370 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:34:54.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.566+0000 7f5972e00640 1 -- 172.21.15.31:0/2767178210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f596c102920 msgr2=0x7f596c104d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:54.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.566+0000 7f5972e00640 1 --2- 172.21.15.31:0/2767178210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f596c102920 0x7f596c104d90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5958009900 tx=0x7f595802bc40 comp rx=0 tx=0).stop 2024-03-21T10:34:54.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.567+0000 7f5972e00640 1 -- 172.21.15.31:0/2767178210 shutdown_connections 2024-03-21T10:34:54.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.567+0000 7f5972e00640 1 --2- 172.21.15.31:0/2767178210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f596c1052d0 0x7f596c105750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:54.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.567+0000 7f5972e00640 1 --2- 172.21.15.31:0/2767178210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f596c102920 0x7f596c104d90 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:54.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.567+0000 7f5972e00640 1 --2- 172.21.15.31:0/2767178210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f596c0fffb0 0x7f596c1023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:54.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.567+0000 7f5972e00640 1 -- 172.21.15.31:0/2767178210 >> 172.21.15.31:0/2767178210 conn(0x7f596c0fbbd0 msgr2=0x7f596c0fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:54.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.567+0000 7f5972e00640 1 -- 172.21.15.31:0/2767178210 shutdown_connections 2024-03-21T10:34:54.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.567+0000 7f5972e00640 1 -- 172.21.15.31:0/2767178210 wait complete. 2024-03-21T10:34:54.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.568+0000 7f5972e00640 1 Processor -- start 2024-03-21T10:34:54.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.568+0000 7f5972e00640 1 -- start start 2024-03-21T10:34:54.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.568+0000 7f5972e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f596c0fffb0 0x7f596c19e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:54.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.568+0000 7f5972e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f596c102920 0x7f596c19e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:54.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.568+0000 7f5972e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f596c1052d0 0x7f596c1a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:54.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.568+0000 7f5972e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c113970 con 0x7f596c1052d0 2024-03-21T10:34:54.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.568+0000 7f5972e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c1137f0 con 0x7f596c102920 2024-03-21T10:34:54.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.568+0000 7f5972e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c113af0 con 0x7f596c0fffb0 2024-03-21T10:34:54.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.569+0000 7f5970e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f596c1052d0 0x7f596c1a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:54.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.569+0000 7f5970e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f596c1052d0 0x7f596c1a5550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33428/0 (socket says 172.21.15.31:33428) 2024-03-21T10:34:54.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.569+0000 7f5970e00640 1 -- 172.21.15.31:0/772344234 learned_addr learned my addr 172.21.15.31:0/772344234 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:34:54.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.569+0000 7f596be00640 1 --2- 172.21.15.31:0/772344234 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f596c0fffb0 0x7f596c19e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:54.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.569+0000 7f596b400640 1 --2- 172.21.15.31:0/772344234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f596c102920 0x7f596c19e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:54.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.569+0000 7f5970e00640 1 -- 172.21.15.31:0/772344234 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f596c0fffb0 msgr2=0x7f596c19e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:54.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.569+0000 7f5970e00640 1 --2- 172.21.15.31:0/772344234 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f596c0fffb0 0x7f596c19e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:54.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.569+0000 7f5970e00640 1 -- 172.21.15.31:0/772344234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f596c102920 msgr2=0x7f596c19e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:54.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.569+0000 7f5970e00640 1 --2- 172.21.15.31:0/772344234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f596c102920 0x7f596c19e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:54.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.569+0000 7f5970e00640 1 -- 172.21.15.31:0/772344234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596c1a5bc0 con 0x7f596c1052d0 2024-03-21T10:34:54.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.569+0000 7f596be00640 1 --2- 172.21.15.31:0/772344234 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f596c0fffb0 0x7f596c19e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:34:54.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.569+0000 7f5970e00640 1 --2- 172.21.15.31:0/772344234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f596c1052d0 0x7f596c1a5550 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f596400eb10 tx=0x7f596400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:54.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.570+0000 7f5968c00640 1 -- 172.21.15.31:0/772344234 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5964004b20 con 0x7f596c1052d0 2024-03-21T10:34:54.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.570+0000 7f5968c00640 1 -- 172.21.15.31:0/772344234 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5964004cc0 con 0x7f596c1052d0 2024-03-21T10:34:54.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.570+0000 7f5972e00640 1 -- 172.21.15.31:0/772344234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f596c1a5eb0 con 0x7f596c1052d0 2024-03-21T10:34:54.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.570+0000 7f5968c00640 1 -- 172.21.15.31:0/772344234 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5964010760 con 0x7f596c1052d0 2024-03-21T10:34:54.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.570+0000 7f5972e00640 1 -- 172.21.15.31:0/772344234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f596c1a6370 con 0x7f596c1052d0 2024-03-21T10:34:54.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.572+0000 7f5972e00640 1 -- 172.21.15.31:0/772344234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f596c0fd6c0 con 0x7f596c1052d0 2024-03-21T10:34:54.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.573+0000 7f5968c00640 1 -- 172.21.15.31:0/772344234 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f5964005020 con 0x7f596c1052d0 2024-03-21T10:34:54.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.575+0000 7f5968c00640 1 --2- 172.21.15.31:0/772344234 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f59500a2fa0 0x7f59500a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:54.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.575+0000 7f5968c00640 1 -- 172.21.15.31:0/772344234 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f59640cf440 con 0x7f596c1052d0 2024-03-21T10:34:54.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.575+0000 7f596be00640 1 --2- 172.21.15.31:0/772344234 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f59500a2fa0 0x7f59500a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:54.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.576+0000 7f596be00640 1 --2- 172.21.15.31:0/772344234 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f59500a2fa0 0x7f59500a5460 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f595c009fc0 tx=0x7f595c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:54.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.579+0000 7f5968c00640 1 -- 172.21.15.31:0/772344234 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f596409c710 con 0x7f596c1052d0 2024-03-21T10:34:54.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.818+0000 7f5972e00640 1 -- 172.21.15.31:0/772344234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f596c02d270 con 0x7f596c1052d0 2024-03-21T10:34:54.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.819+0000 7f5968c00640 1 -- 172.21.15.31:0/772344234 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8444 (secure 0 0 0) 0x7f596409c010 con 0x7f596c1052d0 2024-03-21T10:34:54.818 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:34:54.818 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24539,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-21T10:34:54.818 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 11 2024-03-21T10:34:54.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.822+0000 7f5972e00640 1 -- 172.21.15.31:0/772344234 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f59500a2fa0 msgr2=0x7f59500a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:54.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.822+0000 7f5972e00640 1 --2- 172.21.15.31:0/772344234 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f59500a2fa0 0x7f59500a5460 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f595c009fc0 tx=0x7f595c005f90 comp rx=0 tx=0).stop 2024-03-21T10:34:54.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.822+0000 7f5972e00640 1 -- 172.21.15.31:0/772344234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f596c1052d0 msgr2=0x7f596c1a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:54.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.822+0000 7f5972e00640 1 --2- 172.21.15.31:0/772344234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f596c1052d0 0x7f596c1a5550 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f596400eb10 tx=0x7f596400c2d0 comp rx=0 tx=0).stop 2024-03-21T10:34:54.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.823+0000 7f5972e00640 1 -- 172.21.15.31:0/772344234 shutdown_connections 2024-03-21T10:34:54.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.823+0000 7f5972e00640 1 --2- 172.21.15.31:0/772344234 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f59500a2fa0 0x7f59500a5460 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:54.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.823+0000 7f5972e00640 1 --2- 172.21.15.31:0/772344234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f596c1052d0 0x7f596c1a5550 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:54.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.823+0000 7f5972e00640 1 --2- 172.21.15.31:0/772344234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f596c102920 0x7f596c19e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:54.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.823+0000 7f5972e00640 1 --2- 172.21.15.31:0/772344234 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f596c0fffb0 0x7f596c19e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:54.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.823+0000 7f5972e00640 1 -- 172.21.15.31:0/772344234 >> 172.21.15.31:0/772344234 conn(0x7f596c0fbbd0 msgr2=0x7f596c100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:54.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.823+0000 7f5972e00640 1 -- 172.21.15.31:0/772344234 shutdown_connections 2024-03-21T10:34:54.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:54.823+0000 7f5972e00640 1 -- 172.21.15.31:0/772344234 wait complete. 2024-03-21T10:34:55.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:55 smithi031 ceph-mon[31673]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:55.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:55 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:55.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:55 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:55.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:55 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:55.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:55 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:55.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:55 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:55.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:55 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:55.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:55 smithi031 ceph-mon[31673]: mds.? [v2:172.21.15.31:6838/4261892640,v1:172.21.15.31:6839/4261892640] up:boot 2024-03-21T10:34:55.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:55 smithi031 ceph-mon[31673]: fsmap 10 up:standby 2024-03-21T10:34:55.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:55 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-21T10:34:55.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:55 smithi031 ceph-mon[31673]: Deploying daemon mds.a on smithi031 2024-03-21T10:34:55.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:55 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/772344234' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:55.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:55 smithi070 ceph-mon[36330]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:55.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:55 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:55.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:55 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:55.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:55 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:55.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:55 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:55.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:55 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:55.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:55 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:55.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:55 smithi070 ceph-mon[36330]: mds.? [v2:172.21.15.31:6838/4261892640,v1:172.21.15.31:6839/4261892640] up:boot 2024-03-21T10:34:55.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:55 smithi070 ceph-mon[36330]: fsmap 10 up:standby 2024-03-21T10:34:55.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:55 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-21T10:34:55.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:55 smithi070 ceph-mon[36330]: Deploying daemon mds.a on smithi031 2024-03-21T10:34:55.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:55 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/772344234' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:55.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:55 smithi040 ceph-mon[36154]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:55.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:55 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:55.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:55 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:55.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:55 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:55.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:55 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:55.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:55 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:55.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:55 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:55.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:55 smithi040 ceph-mon[36154]: mds.? [v2:172.21.15.31:6838/4261892640,v1:172.21.15.31:6839/4261892640] up:boot 2024-03-21T10:34:55.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:55 smithi040 ceph-mon[36154]: fsmap 10 up:standby 2024-03-21T10:34:55.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:55 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-21T10:34:55.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:55 smithi040 ceph-mon[36154]: Deploying daemon mds.a on smithi031 2024-03-21T10:34:55.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:55 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/772344234' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:56.841 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:34:57.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.188+0000 7f662a000640 1 -- 172.21.15.31:0/161896918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f661c0a5470 msgr2=0x7f661c0ac470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:57.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.188+0000 7f662a000640 1 --2- 172.21.15.31:0/161896918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f661c0a5470 0x7f661c0ac470 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6618009900 tx=0x7f661802bc80 comp rx=0 tx=0).stop 2024-03-21T10:34:57.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.188+0000 7f662a000640 1 -- 172.21.15.31:0/161896918 shutdown_connections 2024-03-21T10:34:57.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.188+0000 7f662a000640 1 --2- 172.21.15.31:0/161896918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f661c0aca00 0x7f661c0aeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:57.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.188+0000 7f662a000640 1 --2- 172.21.15.31:0/161896918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f661c0a5470 0x7f661c0ac470 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:57.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.188+0000 7f662a000640 1 --2- 172.21.15.31:0/161896918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f661c0a4b10 0x7f661c0a4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:57.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.188+0000 7f662a000640 1 -- 172.21.15.31:0/161896918 >> 172.21.15.31:0/161896918 conn(0x7f661c0a00d0 msgr2=0x7f661c0a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:57.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.188+0000 7f662a000640 1 -- 172.21.15.31:0/161896918 shutdown_connections 2024-03-21T10:34:57.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.188+0000 7f662a000640 1 -- 172.21.15.31:0/161896918 wait complete. 2024-03-21T10:34:57.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.189+0000 7f662a000640 1 Processor -- start 2024-03-21T10:34:57.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.189+0000 7f662a000640 1 -- start start 2024-03-21T10:34:57.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.190+0000 7f662a000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f661c0a4b10 0x7f661c13d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:57.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.190+0000 7f662a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f661c0a5470 0x7f661c13d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:57.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.190+0000 7f662a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f661c0aca00 0x7f661c144790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:57.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.190+0000 7f662a000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c0b2b70 con 0x7f661c0a5470 2024-03-21T10:34:57.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.190+0000 7f662a000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c0b29f0 con 0x7f661c0aca00 2024-03-21T10:34:57.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.190+0000 7f662a000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c0b2cf0 con 0x7f661c0a4b10 2024-03-21T10:34:57.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.190+0000 7f6628c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f661c0a4b10 0x7f661c13d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:57.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.190+0000 7f6628c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f661c0a4b10 0x7f661c13d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:55172/0 (socket says 172.21.15.31:55172) 2024-03-21T10:34:57.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.190+0000 7f6628c00640 1 -- 172.21.15.31:0/622046920 learned_addr learned my addr 172.21.15.31:0/622046920 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:34:57.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.190+0000 7f6629600640 1 --2- 172.21.15.31:0/622046920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f661c0aca00 0x7f661c144790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:57.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.190+0000 7f6623e00640 1 --2- 172.21.15.31:0/622046920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f661c0a5470 0x7f661c13d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:57.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.191+0000 7f6629600640 1 -- 172.21.15.31:0/622046920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f661c0a4b10 msgr2=0x7f661c13d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:57.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.191+0000 7f6629600640 1 --2- 172.21.15.31:0/622046920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f661c0a4b10 0x7f661c13d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:57.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.191+0000 7f6629600640 1 -- 172.21.15.31:0/622046920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f661c0a5470 msgr2=0x7f661c13d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:57.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.191+0000 7f6629600640 1 --2- 172.21.15.31:0/622046920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f661c0a5470 0x7f661c13d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:57.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.191+0000 7f6629600640 1 -- 172.21.15.31:0/622046920 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c0a62c0 con 0x7f661c0aca00 2024-03-21T10:34:57.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.191+0000 7f6628c00640 1 --2- 172.21.15.31:0/622046920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f661c0a4b10 0x7f661c13d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:34:57.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.191+0000 7f6629600640 1 --2- 172.21.15.31:0/622046920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f661c0aca00 0x7f661c144790 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6624050d00 tx=0x7f662406d430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:57.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.194+0000 7f6621600640 1 -- 172.21.15.31:0/622046920 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6624079020 con 0x7f661c0aca00 2024-03-21T10:34:57.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.194+0000 7f6621600640 1 -- 172.21.15.31:0/622046920 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f662406dd10 con 0x7f661c0aca00 2024-03-21T10:34:57.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.194+0000 7f662a000640 1 -- 172.21.15.31:0/622046920 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f661c0a65b0 con 0x7f661c0aca00 2024-03-21T10:34:57.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.194+0000 7f6621600640 1 -- 172.21.15.31:0/622046920 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6624074720 con 0x7f661c0aca00 2024-03-21T10:34:57.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.194+0000 7f662a000640 1 -- 172.21.15.31:0/622046920 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f661c0a6a80 con 0x7f661c0aca00 2024-03-21T10:34:57.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.196+0000 7f662a000640 1 -- 172.21.15.31:0/622046920 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661c002890 con 0x7f661c0aca00 2024-03-21T10:34:57.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.197+0000 7f6621600640 1 -- 172.21.15.31:0/622046920 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f662407c020 con 0x7f661c0aca00 2024-03-21T10:34:57.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.199+0000 7f6621600640 1 --2- 172.21.15.31:0/622046920 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f660c0a3070 0x7f660c0a5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:57.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.199+0000 7f6621600640 1 -- 172.21.15.31:0/622046920 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6624131b70 con 0x7f661c0aca00 2024-03-21T10:34:57.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.199+0000 7f6628c00640 1 --2- 172.21.15.31:0/622046920 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f660c0a3070 0x7f660c0a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:57.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.200+0000 7f6628c00640 1 --2- 172.21.15.31:0/622046920 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f660c0a3070 0x7f660c0a5530 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6610009e00 tx=0x7f6610009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:57.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.203+0000 7f6621600640 1 -- 172.21.15.31:0/622046920 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6624083020 con 0x7f661c0aca00 2024-03-21T10:34:57.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.455+0000 7f662a000640 1 -- 172.21.15.31:0/622046920 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f661c003020 con 0x7f661c0aca00 2024-03-21T10:34:57.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.456+0000 7f6621600640 1 -- 172.21.15.31:0/622046920 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8444 (secure 0 0 0) 0x7f66240feda0 con 0x7f661c0aca00 2024-03-21T10:34:57.455 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:34:57.455 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24539,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-21T10:34:57.455 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 11 2024-03-21T10:34:57.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.459+0000 7f662a000640 1 -- 172.21.15.31:0/622046920 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f660c0a3070 msgr2=0x7f660c0a5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:57.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.459+0000 7f662a000640 1 --2- 172.21.15.31:0/622046920 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f660c0a3070 0x7f660c0a5530 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6610009e00 tx=0x7f6610009190 comp rx=0 tx=0).stop 2024-03-21T10:34:57.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.459+0000 7f662a000640 1 -- 172.21.15.31:0/622046920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f661c0aca00 msgr2=0x7f661c144790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:57.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.459+0000 7f662a000640 1 --2- 172.21.15.31:0/622046920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f661c0aca00 0x7f661c144790 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6624050d00 tx=0x7f662406d430 comp rx=0 tx=0).stop 2024-03-21T10:34:57.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.459+0000 7f662a000640 1 -- 172.21.15.31:0/622046920 shutdown_connections 2024-03-21T10:34:57.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.459+0000 7f662a000640 1 --2- 172.21.15.31:0/622046920 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f660c0a3070 0x7f660c0a5530 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:57.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.459+0000 7f662a000640 1 --2- 172.21.15.31:0/622046920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f661c0aca00 0x7f661c144790 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:57.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.459+0000 7f662a000640 1 --2- 172.21.15.31:0/622046920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f661c0a5470 0x7f661c13d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:57.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.459+0000 7f662a000640 1 --2- 172.21.15.31:0/622046920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f661c0a4b10 0x7f661c13d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:57.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.459+0000 7f662a000640 1 -- 172.21.15.31:0/622046920 >> 172.21.15.31:0/622046920 conn(0x7f661c0a00d0 msgr2=0x7f661c0a0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:57.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.459+0000 7f662a000640 1 -- 172.21.15.31:0/622046920 shutdown_connections 2024-03-21T10:34:57.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:57.459+0000 7f662a000640 1 -- 172.21.15.31:0/622046920 wait complete. 2024-03-21T10:34:57.477 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:57 smithi031 ceph-mon[31673]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:57.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:57 smithi070 ceph-mon[36330]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:57.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:57 smithi040 ceph-mon[36154]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:58.011 INFO:journalctl@ceph.mds.a.smithi031.stdout:Mar 21 10:34:57 smithi031 podman[72724]: 2024-03-21 10:34:57.816732685 +0000 UTC m=+0.184654197 container create 0700393bf215ae26e97f128e64ea04d5dfeb35f6a5640c6b70227e6d9ca3cfa8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-a, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-21T10:34:58.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:58 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/622046920' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:58.556 INFO:journalctl@ceph.mds.a.smithi031.stdout:Mar 21 10:34:58 smithi031 podman[72724]: 2024-03-21 10:34:58.178899286 +0000 UTC m=+0.546820804 container init 0700393bf215ae26e97f128e64ea04d5dfeb35f6a5640c6b70227e6d9ca3cfa8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T10:34:58.556 INFO:journalctl@ceph.mds.a.smithi031.stdout:Mar 21 10:34:58 smithi031 podman[72724]: 2024-03-21 10:34:58.189350849 +0000 UTC m=+0.557272368 container start 0700393bf215ae26e97f128e64ea04d5dfeb35f6a5640c6b70227e6d9ca3cfa8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-21T10:34:58.556 INFO:journalctl@ceph.mds.a.smithi031.stdout:Mar 21 10:34:58 smithi031 ceph-mds[72743]: -- 172.21.15.31:0/1535530525 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5575c1617a40 con 0x5575c15eb000 2024-03-21T10:34:58.556 INFO:journalctl@ceph.mds.a.smithi031.stdout:Mar 21 10:34:58 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-a[72739]: starting mds.a at 2024-03-21T10:34:58.556 INFO:journalctl@ceph.mds.a.smithi031.stdout:Mar 21 10:34:58 smithi031 bash[72724]: 0700393bf215ae26e97f128e64ea04d5dfeb35f6a5640c6b70227e6d9ca3cfa8 2024-03-21T10:34:58.556 INFO:journalctl@ceph.mds.a.smithi031.stdout:Mar 21 10:34:58 smithi031 systemd[1]: Started Ceph mds.a for 3847408c-e76d-11ee-95cd-87774f69a715. 2024-03-21T10:34:58.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:58 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/622046920' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:58.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:58 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/622046920' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:34:59.480 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:34:59.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:59 smithi040 ceph-mon[36154]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:59.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:59 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:59.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:59 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:59.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:59 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:59.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:59 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:59.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:59 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:59.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:59 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:59.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:59 smithi040 ceph-mon[36154]: Deploying daemon mds.b on smithi040 2024-03-21T10:34:59.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:59 smithi040 ceph-mon[36154]: mds.? [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] up:boot 2024-03-21T10:34:59.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:59 smithi040 ceph-mon[36154]: fsmap 11 up:standby 2024-03-21T10:34:59.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:34:59 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-21T10:34:59.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.720+0000 7fe8afa00640 1 -- 172.21.15.31:0/3289857823 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8a81028b0 msgr2=0x7fe8a8104d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:59.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.720+0000 7fe8afa00640 1 --2- 172.21.15.31:0/3289857823 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8a81028b0 0x7fe8a8104d20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe894009ca0 tx=0x7fe89402c0f0 comp rx=0 tx=0).stop 2024-03-21T10:34:59.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.721+0000 7fe8afa00640 1 -- 172.21.15.31:0/3289857823 shutdown_connections 2024-03-21T10:34:59.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.721+0000 7fe8afa00640 1 --2- 172.21.15.31:0/3289857823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8a8105260 0x7fe8a81056e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:59.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.721+0000 7fe8afa00640 1 --2- 172.21.15.31:0/3289857823 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8a81028b0 0x7fe8a8104d20 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:59.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.721+0000 7fe8afa00640 1 --2- 172.21.15.31:0/3289857823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8a80fff40 0x7fe8a8102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:59.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.721+0000 7fe8afa00640 1 -- 172.21.15.31:0/3289857823 >> 172.21.15.31:0/3289857823 conn(0x7fe8a80fbb40 msgr2=0x7fe8a80fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:59.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.721+0000 7fe8afa00640 1 -- 172.21.15.31:0/3289857823 shutdown_connections 2024-03-21T10:34:59.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.721+0000 7fe8afa00640 1 -- 172.21.15.31:0/3289857823 wait complete. 2024-03-21T10:34:59.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.722+0000 7fe8afa00640 1 Processor -- start 2024-03-21T10:34:59.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.722+0000 7fe8afa00640 1 -- start start 2024-03-21T10:34:59.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.723+0000 7fe8afa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8a80fff40 0x7fe8a819df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:59.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.723+0000 7fe8afa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8a81028b0 0x7fe8a819e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:59.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.723+0000 7fe8afa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8a8105260 0x7fe8a81a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:59.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.723+0000 7fe8afa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a8113920 con 0x7fe8a80fff40 2024-03-21T10:34:59.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.723+0000 7fe8afa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a81137a0 con 0x7fe8a81028b0 2024-03-21T10:34:59.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.723+0000 7fe8afa00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a8113aa0 con 0x7fe8a8105260 2024-03-21T10:34:59.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.723+0000 7fe8ad000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8a80fff40 0x7fe8a819df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:59.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.723+0000 7fe8ad000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8a80fff40 0x7fe8a819df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54538/0 (socket says 172.21.15.31:54538) 2024-03-21T10:34:59.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.723+0000 7fe8ad000640 1 -- 172.21.15.31:0/2963058997 learned_addr learned my addr 172.21.15.31:0/2963058997 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:34:59.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.723+0000 7fe8a7e00640 1 --2- 172.21.15.31:0/2963058997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8a81028b0 0x7fe8a819e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:59.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.723+0000 7fe8ada00640 1 --2- 172.21.15.31:0/2963058997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8a8105260 0x7fe8a81a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:59.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.724+0000 7fe8ad000640 1 -- 172.21.15.31:0/2963058997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8a8105260 msgr2=0x7fe8a81a5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:59.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.724+0000 7fe8ad000640 1 --2- 172.21.15.31:0/2963058997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8a8105260 0x7fe8a81a5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:59.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.724+0000 7fe8ad000640 1 -- 172.21.15.31:0/2963058997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8a81028b0 msgr2=0x7fe8a819e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:59.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.724+0000 7fe8ad000640 1 --2- 172.21.15.31:0/2963058997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8a81028b0 0x7fe8a819e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:59.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.724+0000 7fe8ad000640 1 -- 172.21.15.31:0/2963058997 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a81a5bb0 con 0x7fe8a80fff40 2024-03-21T10:34:59.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.724+0000 7fe8ad000640 1 --2- 172.21.15.31:0/2963058997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8a80fff40 0x7fe8a819df90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe89800e9f0 tx=0x7fe89800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:59.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.724+0000 7fe8a5600640 1 -- 172.21.15.31:0/2963058997 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89800ccc0 con 0x7fe8a80fff40 2024-03-21T10:34:59.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.724+0000 7fe8a5600640 1 -- 172.21.15.31:0/2963058997 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe898010910 con 0x7fe8a80fff40 2024-03-21T10:34:59.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.724+0000 7fe8afa00640 1 -- 172.21.15.31:0/2963058997 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a81a5ea0 con 0x7fe8a80fff40 2024-03-21T10:34:59.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.724+0000 7fe8a5600640 1 -- 172.21.15.31:0/2963058997 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8980188a0 con 0x7fe8a80fff40 2024-03-21T10:34:59.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.724+0000 7fe8afa00640 1 -- 172.21.15.31:0/2963058997 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a81a6310 con 0x7fe8a80fff40 2024-03-21T10:34:59.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.726+0000 7fe8afa00640 1 -- 172.21.15.31:0/2963058997 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a80fd630 con 0x7fe8a80fff40 2024-03-21T10:34:59.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.728+0000 7fe8a5600640 1 -- 172.21.15.31:0/2963058997 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fe898010ac0 con 0x7fe8a80fff40 2024-03-21T10:34:59.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.729+0000 7fe8a5600640 1 --2- 172.21.15.31:0/2963058997 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe88c0a2fa0 0x7fe88c0a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:34:59.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.730+0000 7fe8a5600640 1 -- 172.21.15.31:0/2963058997 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe8980cf380 con 0x7fe8a80fff40 2024-03-21T10:34:59.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.730+0000 7fe8a7e00640 1 --2- 172.21.15.31:0/2963058997 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe88c0a2fa0 0x7fe88c0a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:34:59.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.731+0000 7fe8a7e00640 1 --2- 172.21.15.31:0/2963058997 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe88c0a2fa0 0x7fe88c0a5460 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe894006010 tx=0x7fe894032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:34:59.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.734+0000 7fe8a5600640 1 -- 172.21.15.31:0/2963058997 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe89809c5b0 con 0x7fe8a80fff40 2024-03-21T10:34:59.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:59 smithi031 ceph-mon[31673]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:59.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:59 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:59.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:59 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:59.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:59 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:59.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:59 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:59.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:59 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:59.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:59 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:59.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:59 smithi031 ceph-mon[31673]: Deploying daemon mds.b on smithi040 2024-03-21T10:34:59.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:59 smithi031 ceph-mon[31673]: mds.? [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] up:boot 2024-03-21T10:34:59.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:59 smithi031 ceph-mon[31673]: fsmap 11 up:standby 2024-03-21T10:34:59.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:34:59 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-21T10:34:59.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:59 smithi070 ceph-mon[36330]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:34:59.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:59 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:59.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:59 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:59.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:59 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:34:59.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:59 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-21T10:34:59.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:59 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-21T10:34:59.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:59 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:34:59.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:59 smithi070 ceph-mon[36330]: Deploying daemon mds.b on smithi040 2024-03-21T10:34:59.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:59 smithi070 ceph-mon[36330]: mds.? [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] up:boot 2024-03-21T10:34:59.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:59 smithi070 ceph-mon[36330]: fsmap 11 up:standby 2024-03-21T10:34:59.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:34:59 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-21T10:34:59.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.981+0000 7fe8afa00640 1 -- 172.21.15.31:0/2963058997 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8a802d070 con 0x7fe8a80fff40 2024-03-21T10:34:59.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.983+0000 7fe8a5600640 1 -- 172.21.15.31:0/2963058997 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9236 (secure 0 0 0) 0x7fe89809beb0 con 0x7fe8a80fff40 2024-03-21T10:34:59.981 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:34:59.981 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24539,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24584,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[]} 2024-03-21T10:34:59.981 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 12 2024-03-21T10:34:59.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.986+0000 7fe8afa00640 1 -- 172.21.15.31:0/2963058997 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe88c0a2fa0 msgr2=0x7fe88c0a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:59.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.986+0000 7fe8afa00640 1 --2- 172.21.15.31:0/2963058997 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe88c0a2fa0 0x7fe88c0a5460 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe894006010 tx=0x7fe894032590 comp rx=0 tx=0).stop 2024-03-21T10:34:59.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.986+0000 7fe8afa00640 1 -- 172.21.15.31:0/2963058997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8a80fff40 msgr2=0x7fe8a819df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:34:59.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.986+0000 7fe8afa00640 1 --2- 172.21.15.31:0/2963058997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8a80fff40 0x7fe8a819df90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe89800e9f0 tx=0x7fe89800edf0 comp rx=0 tx=0).stop 2024-03-21T10:34:59.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.986+0000 7fe8afa00640 1 -- 172.21.15.31:0/2963058997 shutdown_connections 2024-03-21T10:34:59.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.986+0000 7fe8afa00640 1 --2- 172.21.15.31:0/2963058997 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe88c0a2fa0 0x7fe88c0a5460 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:59.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.986+0000 7fe8afa00640 1 --2- 172.21.15.31:0/2963058997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8a8105260 0x7fe8a81a5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:59.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.986+0000 7fe8afa00640 1 --2- 172.21.15.31:0/2963058997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8a81028b0 0x7fe8a819e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:59.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.986+0000 7fe8afa00640 1 --2- 172.21.15.31:0/2963058997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8a80fff40 0x7fe8a819df90 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:34:59.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.986+0000 7fe8afa00640 1 -- 172.21.15.31:0/2963058997 >> 172.21.15.31:0/2963058997 conn(0x7fe8a80fbb40 msgr2=0x7fe8a81009a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:34:59.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.986+0000 7fe8afa00640 1 -- 172.21.15.31:0/2963058997 shutdown_connections 2024-03-21T10:34:59.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:34:59.986+0000 7fe8afa00640 1 -- 172.21.15.31:0/2963058997 wait complete. 2024-03-21T10:35:00.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:00 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2963058997' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:35:00.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:00 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2963058997' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:35:00.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:00 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2963058997' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:35:01.314 INFO:journalctl@ceph.mds.b.smithi040.stdout:Mar 21 10:35:01 smithi040 systemd[1]: Started Ceph mds.b for 3847408c-e76d-11ee-95cd-87774f69a715. 2024-03-21T10:35:01.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:01 smithi040 ceph-mon[36154]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:35:01.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:01 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:01.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:01 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:01.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:01 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:01.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:01 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:01.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:01 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:01.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:01 smithi031 ceph-mon[31673]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:35:01.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:01 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:01.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:01 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:01.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:01 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:01.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:01 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:01.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:01 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:01.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:01 smithi070 ceph-mon[36330]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:35:01.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:01 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:01.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:01 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:01.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:01 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:01.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:01 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:01.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:01 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:02.003 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:35:02.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.228+0000 7fb444c00640 1 -- 172.21.15.31:0/245170811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb440076060 msgr2=0x7fb440076480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:02.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.228+0000 7fb444c00640 1 --2- 172.21.15.31:0/245170811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb440076060 0x7fb440076480 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb430009900 tx=0x7fb43002bc50 comp rx=0 tx=0).stop 2024-03-21T10:35:02.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.228+0000 7fb444c00640 1 -- 172.21.15.31:0/245170811 shutdown_connections 2024-03-21T10:35:02.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.228+0000 7fb444c00640 1 --2- 172.21.15.31:0/245170811 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb440077c40 0x7fb4400784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:02.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.228+0000 7fb444c00640 1 --2- 172.21.15.31:0/245170811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4400772a0 0x7fb440077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:02.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.228+0000 7fb444c00640 1 --2- 172.21.15.31:0/245170811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb440076060 0x7fb440076480 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:02.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.228+0000 7fb444c00640 1 -- 172.21.15.31:0/245170811 >> 172.21.15.31:0/245170811 conn(0x7fb4400fed70 msgr2=0x7fb440101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:02.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.228+0000 7fb444c00640 1 -- 172.21.15.31:0/245170811 shutdown_connections 2024-03-21T10:35:02.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.228+0000 7fb444c00640 1 -- 172.21.15.31:0/245170811 wait complete. 2024-03-21T10:35:02.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.229+0000 7fb444c00640 1 Processor -- start 2024-03-21T10:35:02.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.230+0000 7fb444c00640 1 -- start start 2024-03-21T10:35:02.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.230+0000 7fb444c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb440076060 0x7fb44019deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:02.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.230+0000 7fb444c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4400772a0 0x7fb44019e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:02.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.230+0000 7fb444c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb440077c40 0x7fb4401a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:02.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.230+0000 7fb444c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4401138b0 con 0x7fb440076060 2024-03-21T10:35:02.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.230+0000 7fb444c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb440113730 con 0x7fb4400772a0 2024-03-21T10:35:02.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.230+0000 7fb444c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb440113a30 con 0x7fb440077c40 2024-03-21T10:35:02.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.230+0000 7fb43f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb440076060 0x7fb44019deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:02.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.230+0000 7fb43ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4400772a0 0x7fb44019e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:02.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.230+0000 7fb43fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb440077c40 0x7fb4401a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:02.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.230+0000 7fb43ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4400772a0 0x7fb44019e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56350/0 (socket says 172.21.15.31:56350) 2024-03-21T10:35:02.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.230+0000 7fb43ea00640 1 -- 172.21.15.31:0/176994272 learned_addr learned my addr 172.21.15.31:0/176994272 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:02.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.231+0000 7fb43ea00640 1 -- 172.21.15.31:0/176994272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb440077c40 msgr2=0x7fb4401a5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:02.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.231+0000 7fb43ea00640 1 --2- 172.21.15.31:0/176994272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb440077c40 0x7fb4401a5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:02.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.231+0000 7fb43ea00640 1 -- 172.21.15.31:0/176994272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb440076060 msgr2=0x7fb44019deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:02.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.231+0000 7fb43ea00640 1 --2- 172.21.15.31:0/176994272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb440076060 0x7fb44019deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:02.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.231+0000 7fb43ea00640 1 -- 172.21.15.31:0/176994272 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4401a5ad0 con 0x7fb4400772a0 2024-03-21T10:35:02.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.231+0000 7fb43fe00640 1 --2- 172.21.15.31:0/176994272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb440077c40 0x7fb4401a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:35:02.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.231+0000 7fb43ea00640 1 --2- 172.21.15.31:0/176994272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4400772a0 0x7fb44019e3f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb42c00ea60 tx=0x7fb42c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:02.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.232+0000 7fb437e00640 1 -- 172.21.15.31:0/176994272 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c00cde0 con 0x7fb4400772a0 2024-03-21T10:35:02.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.232+0000 7fb437e00640 1 -- 172.21.15.31:0/176994272 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb42c0035f0 con 0x7fb4400772a0 2024-03-21T10:35:02.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.232+0000 7fb444c00640 1 -- 172.21.15.31:0/176994272 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4401a5dc0 con 0x7fb4400772a0 2024-03-21T10:35:02.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.232+0000 7fb437e00640 1 -- 172.21.15.31:0/176994272 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c010670 con 0x7fb4400772a0 2024-03-21T10:35:02.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.232+0000 7fb444c00640 1 -- 172.21.15.31:0/176994272 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4401a6290 con 0x7fb4400772a0 2024-03-21T10:35:02.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.233+0000 7fb444c00640 1 -- 172.21.15.31:0/176994272 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4401102e0 con 0x7fb4400772a0 2024-03-21T10:35:02.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.238+0000 7fb437e00640 1 -- 172.21.15.31:0/176994272 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fb42c0037a0 con 0x7fb4400772a0 2024-03-21T10:35:02.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.240+0000 7fb437e00640 1 --2- 172.21.15.31:0/176994272 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb4240a2f90 0x7fb4240a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:02.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.240+0000 7fb437e00640 1 -- 172.21.15.31:0/176994272 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb42c0cf200 con 0x7fb4400772a0 2024-03-21T10:35:02.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.240+0000 7fb43f400640 1 --2- 172.21.15.31:0/176994272 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb4240a2f90 0x7fb4240a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:02.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.240+0000 7fb437e00640 1 -- 172.21.15.31:0/176994272 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb42c0cf680 con 0x7fb4400772a0 2024-03-21T10:35:02.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.241+0000 7fb43f400640 1 --2- 172.21.15.31:0/176994272 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb4240a2f90 0x7fb4240a5450 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb43002c260 tx=0x7fb430000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:02.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.495+0000 7fb444c00640 1 -- 172.21.15.31:0/176994272 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb44002d2c0 con 0x7fb4400772a0 2024-03-21T10:35:02.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.496+0000 7fb437e00640 1 -- 172.21.15.31:0/176994272 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10028 (secure 0 0 0) 0x7fb42c09c430 con 0x7fb4400772a0 2024-03-21T10:35:02.494 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:35:02.494 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24539,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24584,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24596,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[]} 2024-03-21T10:35:02.495 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 13 2024-03-21T10:35:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.498+0000 7fb444c00640 1 -- 172.21.15.31:0/176994272 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb4240a2f90 msgr2=0x7fb4240a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.498+0000 7fb444c00640 1 --2- 172.21.15.31:0/176994272 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb4240a2f90 0x7fb4240a5450 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb43002c260 tx=0x7fb430000bc0 comp rx=0 tx=0).stop 2024-03-21T10:35:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.498+0000 7fb444c00640 1 -- 172.21.15.31:0/176994272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4400772a0 msgr2=0x7fb44019e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.498+0000 7fb444c00640 1 --2- 172.21.15.31:0/176994272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4400772a0 0x7fb44019e3f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb42c00ea60 tx=0x7fb42c00ef20 comp rx=0 tx=0).stop 2024-03-21T10:35:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.498+0000 7fb444c00640 1 -- 172.21.15.31:0/176994272 shutdown_connections 2024-03-21T10:35:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.498+0000 7fb444c00640 1 --2- 172.21.15.31:0/176994272 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb4240a2f90 0x7fb4240a5450 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.498+0000 7fb444c00640 1 --2- 172.21.15.31:0/176994272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb440077c40 0x7fb4401a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.498+0000 7fb444c00640 1 --2- 172.21.15.31:0/176994272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4400772a0 0x7fb44019e3f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.498+0000 7fb444c00640 1 --2- 172.21.15.31:0/176994272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb440076060 0x7fb44019deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.498+0000 7fb444c00640 1 -- 172.21.15.31:0/176994272 >> 172.21.15.31:0/176994272 conn(0x7fb4400fed70 msgr2=0x7fb440100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.499+0000 7fb444c00640 1 -- 172.21.15.31:0/176994272 shutdown_connections 2024-03-21T10:35:02.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.499+0000 7fb444c00640 1 -- 172.21.15.31:0/176994272 wait complete. 2024-03-21T10:35:02.516 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T10:35:02.685 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:02 smithi040 ceph-mon[36154]: mds.? [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] up:boot 2024-03-21T10:35:02.685 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:02 smithi040 ceph-mon[36154]: fsmap 12 up:standby 2024-03-21T10:35:02.685 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:02 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-21T10:35:02.685 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:02 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:35:02.697 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:02 smithi070 ceph-mon[36330]: mds.? [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] up:boot 2024-03-21T10:35:02.697 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:02 smithi070 ceph-mon[36330]: fsmap 12 up:standby 2024-03-21T10:35:02.697 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:02 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-21T10:35:02.697 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:02 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:35:02.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.734+0000 7effee200640 1 -- 172.21.15.31:0/4147120479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effe81027b0 msgr2=0x7effe8102c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:02.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.734+0000 7effee200640 1 --2- 172.21.15.31:0/4147120479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effe81027b0 0x7effe8102c30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7effd4009860 tx=0x7effd402bb70 comp rx=0 tx=0).stop 2024-03-21T10:35:02.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.735+0000 7effee200640 1 -- 172.21.15.31:0/4147120479 shutdown_connections 2024-03-21T10:35:02.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.735+0000 7effee200640 1 --2- 172.21.15.31:0/4147120479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effe8103170 0x7effe8109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:02.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.735+0000 7effee200640 1 --2- 172.21.15.31:0/4147120479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effe81027b0 0x7effe8102c30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:02.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.735+0000 7effee200640 1 --2- 172.21.15.31:0/4147120479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effe81015c0 0x7effe81019e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:02.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.735+0000 7effee200640 1 -- 172.21.15.31:0/4147120479 >> 172.21.15.31:0/4147120479 conn(0x7effe80fcb30 msgr2=0x7effe80fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:02.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.735+0000 7effee200640 1 -- 172.21.15.31:0/4147120479 shutdown_connections 2024-03-21T10:35:02.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.735+0000 7effee200640 1 -- 172.21.15.31:0/4147120479 wait complete. 2024-03-21T10:35:02.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.736+0000 7effee200640 1 Processor -- start 2024-03-21T10:35:02.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.736+0000 7effee200640 1 -- start start 2024-03-21T10:35:02.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.736+0000 7effee200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effe81015c0 0x7effe8199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:02.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.736+0000 7effee200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effe81027b0 0x7effe819a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:02.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.736+0000 7effee200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effe8103170 0x7effe81a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:02.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.736+0000 7effee200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe810d220 con 0x7effe8103170 2024-03-21T10:35:02.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.736+0000 7effee200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe810d0a0 con 0x7effe81015c0 2024-03-21T10:35:02.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.736+0000 7effee200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe810d3a0 con 0x7effe81027b0 2024-03-21T10:35:02.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.736+0000 7effe7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effe81027b0 0x7effe819a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:02.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.736+0000 7effed800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effe8103170 0x7effe81a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:02.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.736+0000 7effe7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effe81027b0 0x7effe819a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:55242/0 (socket says 172.21.15.31:55242) 2024-03-21T10:35:02.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.736+0000 7effe7e00640 1 -- 172.21.15.31:0/3891884812 learned_addr learned my addr 172.21.15.31:0/3891884812 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:02.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.736+0000 7effece00640 1 --2- 172.21.15.31:0/3891884812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effe81015c0 0x7effe8199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:02.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.737+0000 7effe7e00640 1 -- 172.21.15.31:0/3891884812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effe81015c0 msgr2=0x7effe8199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:02.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.737+0000 7effe7e00640 1 --2- 172.21.15.31:0/3891884812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effe81015c0 0x7effe8199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:02.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.737+0000 7effe7e00640 1 -- 172.21.15.31:0/3891884812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effe8103170 msgr2=0x7effe81a1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:02.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.737+0000 7effe7e00640 1 --2- 172.21.15.31:0/3891884812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effe8103170 0x7effe81a1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:02.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.737+0000 7effe7e00640 1 -- 172.21.15.31:0/3891884812 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe81a16f0 con 0x7effe81027b0 2024-03-21T10:35:02.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.737+0000 7effe7e00640 1 --2- 172.21.15.31:0/3891884812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effe81027b0 0x7effe819a010 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7effd4005940 tx=0x7effd4032470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:02.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.738+0000 7effe5600640 1 -- 172.21.15.31:0/3891884812 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4041020 con 0x7effe81027b0 2024-03-21T10:35:02.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.738+0000 7effe5600640 1 -- 172.21.15.31:0/3891884812 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effd402c9f0 con 0x7effe81027b0 2024-03-21T10:35:02.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.738+0000 7effee200640 1 -- 172.21.15.31:0/3891884812 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe81a1980 con 0x7effe81027b0 2024-03-21T10:35:02.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.738+0000 7effe5600640 1 -- 172.21.15.31:0/3891884812 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4039040 con 0x7effe81027b0 2024-03-21T10:35:02.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.738+0000 7effee200640 1 -- 172.21.15.31:0/3891884812 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe81a1ec0 con 0x7effe81027b0 2024-03-21T10:35:02.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.739+0000 7effee200640 1 -- 172.21.15.31:0/3891884812 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe81a20f0 con 0x7effe81027b0 2024-03-21T10:35:02.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.741+0000 7effe5600640 1 -- 172.21.15.31:0/3891884812 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7effd4044020 con 0x7effe81027b0 2024-03-21T10:35:02.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.742+0000 7effe5600640 1 --2- 172.21.15.31:0/3891884812 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7effd00a2f90 0x7effd00a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:02.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.743+0000 7effe5600640 1 -- 172.21.15.31:0/3891884812 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7effd40f08e0 con 0x7effe81027b0 2024-03-21T10:35:02.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.743+0000 7effece00640 1 --2- 172.21.15.31:0/3891884812 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7effd00a2f90 0x7effd00a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:02.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.744+0000 7effece00640 1 --2- 172.21.15.31:0/3891884812 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7effd00a2f90 0x7effd00a5450 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7effe8102610 tx=0x7effdc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:02.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.746+0000 7effe5600640 1 -- 172.21.15.31:0/3891884812 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7effd404b020 con 0x7effe81027b0 2024-03-21T10:35:02.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:02 smithi031 ceph-mon[31673]: mds.? [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] up:boot 2024-03-21T10:35:02.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:02 smithi031 ceph-mon[31673]: fsmap 12 up:standby 2024-03-21T10:35:02.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:02 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-21T10:35:02.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:02 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:35:02.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.958+0000 7effee200640 1 -- 172.21.15.31:0/3891884812 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7effe81a22c0 con 0x7effe81027b0 2024-03-21T10:35:02.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.959+0000 7effe5600640 1 -- 172.21.15.31:0/3891884812 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18337 (secure 0 0 0) 0x7effd40bdb10 con 0x7effe81027b0 2024-03-21T10:35:02.958 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:35:02.958 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":81,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","created":"2024-03-21T10:25:41.971117+0000","modified":"2024-03-21T10:34:09.260664+0000","last_up_change":"2024-03-21T10:34:07.757196+0000","last_in_change":"2024-03-21T10:33:47.460406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T10:29:44.553968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0244d33-00e2-47d5-9f66-34efc138c462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6803","nonce":3089885125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6805","nonce":3089885125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6809","nonce":3089885125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6807","nonce":3089885125}]},"public_addr":"172.21.15.31:6803/3089885125","cluster_addr":"172.21.15.31:6805/3089885125","heartbeat_back_addr":"172.21.15.31:6809/3089885125","heartbeat_front_addr":"172.21.15.31:6807/3089885125","state":["exists","up"]},{"osd":1,"uuid":"76a63504-beab-4354-a6c3-eebc3b16ad03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6801","nonce":618840037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6803","nonce":618840037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6807","nonce":618840037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6805","nonce":618840037}]},"public_addr":"172.21.15.40:6801/618840037","cluster_addr":"172.21.15.40:6803/618840037","heartbeat_back_addr":"172.21.15.40:6807/618840037","heartbeat_front_addr":"172.21.15.40:6805/618840037","state":["exists","up"]},{"osd":2,"uuid":"fa191bd6-9a21-4ee2-8adf-8a63c9cfd821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6801","nonce":2524184117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6803","nonce":2524184117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6807","nonce":2524184117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6805","nonce":2524184117}]},"public_addr":"172.21.15.70:6801/2524184117","cluster_addr":"172.21.15.70:6803/2524184117","heartbeat_back_addr":"172.21.15.70:6807/2524184117","heartbeat_front_addr":"172.21.15.70:6805/2524184117","state":["exists","up"]},{"osd":3,"uuid":"edb44507-ffb0-42a4-827b-08461e1bf952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6811","nonce":2823036296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6813","nonce":2823036296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6817","nonce":2823036296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6815","nonce":2823036296}]},"public_addr":"172.21.15.31:6811/2823036296","cluster_addr":"172.21.15.31:6813/2823036296","heartbeat_back_addr":"172.21.15.31:6817/2823036296","heartbeat_front_addr":"172.21.15.31:6815/2823036296","state":["exists","up"]},{"osd":4,"uuid":"8cb89153-8c3b-431f-b57c-8863fada76dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6809","nonce":4215186309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6811","nonce":4215186309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6815","nonce":4215186309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6813","nonce":4215186309}]},"public_addr":"172.21.15.40:6809/4215186309","cluster_addr":"172.21.15.40:6811/4215186309","heartbeat_back_addr":"172.21.15.40:6815/4215186309","heartbeat_front_addr":"172.21.15.40:6813/4215186309","state":["exists","up"]},{"osd":5,"uuid":"9409505a-d463-408c-a3cd-218380c04d68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6809","nonce":3402830452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6811","nonce":3402830452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6815","nonce":3402830452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6813","nonce":3402830452}]},"public_addr":"172.21.15.70:6809/3402830452","cluster_addr":"172.21.15.70:6811/3402830452","heartbeat_back_addr":"172.21.15.70:6815/3402830452","heartbeat_front_addr":"172.21.15.70:6813/3402830452","state":["exists","up"]},{"osd":6,"uuid":"f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6819","nonce":1193497058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6821","nonce":1193497058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6825","nonce":1193497058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6823","nonce":1193497058}]},"public_addr":"172.21.15.31:6819/1193497058","cluster_addr":"172.21.15.31:6821/1193497058","heartbeat_back_addr":"172.21.15.31:6825/1193497058","heartbeat_front_addr":"172.21.15.31:6823/1193497058","state":["exists","up"]},{"osd":7,"uuid":"db228a87-29a5-45d4-8fe6-a0a554262eb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6817","nonce":3825628119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6819","nonce":3825628119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6823","nonce":3825628119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6821","nonce":3825628119}]},"public_addr":"172.21.15.40:6817/3825628119","cluster_addr":"172.21.15.40:6819/3825628119","heartbeat_back_addr":"172.21.15.40:6823/3825628119","heartbeat_front_addr":"172.21.15.40:6821/3825628119","state":["exists","up"]},{"osd":8,"uuid":"59b9fde0-095d-4e0b-87da-ead37bd8ff1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6817","nonce":534880586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6819","nonce":534880586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6823","nonce":534880586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6821","nonce":534880586}]},"public_addr":"172.21.15.70:6817/534880586","cluster_addr":"172.21.15.70:6819/534880586","heartbeat_back_addr":"172.21.15.70:6823/534880586","heartbeat_front_addr":"172.21.15.70:6821/534880586","state":["exists","up"]},{"osd":9,"uuid":"77f47061-0c5f-4159-b276-e814935d6c1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6827","nonce":587047469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6829","nonce":587047469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6833","nonce":587047469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6831","nonce":587047469}]},"public_addr":"172.21.15.31:6827/587047469","cluster_addr":"172.21.15.31:6829/587047469","heartbeat_back_addr":"172.21.15.31:6833/587047469","heartbeat_front_addr":"172.21.15.31:6831/587047469","state":["exists","up"]},{"osd":10,"uuid":"8853dc43-7bd1-4c93-8063-f7982fd7da38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6825","nonce":512108711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6827","nonce":512108711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6831","nonce":512108711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6829","nonce":512108711}]},"public_addr":"172.21.15.40:6825/512108711","cluster_addr":"172.21.15.40:6827/512108711","heartbeat_back_addr":"172.21.15.40:6831/512108711","heartbeat_front_addr":"172.21.15.40:6829/512108711","state":["exists","up"]},{"osd":11,"uuid":"84b7c2de-c424-45a6-9583-60097e372a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6825","nonce":540305718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6827","nonce":540305718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6831","nonce":540305718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6829","nonce":540305718}]},"public_addr":"172.21.15.70:6825/540305718","cluster_addr":"172.21.15.70:6827/540305718","heartbeat_back_addr":"172.21.15.70:6831/540305718","heartbeat_front_addr":"172.21.15.70:6829/540305718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:28:51.477358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:17.405045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:41.626268+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:15.339568+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:42.676996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:07.516999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:42.412273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:10.365041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:35.699120+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:11.733639+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:39.505442+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:34:05.449301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451930554":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/3342588843":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6801/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/1626448389":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/1910059351":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/344822770":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/3811368512":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/1981437651":"2024-03-22T10:26:16.959165+0000","172.21.15.31:6801/1981437651":"2024-03-22T10:26:16.959165+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T10:35:02.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.962+0000 7effee200640 1 -- 172.21.15.31:0/3891884812 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7effd00a2f90 msgr2=0x7effd00a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:02.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.962+0000 7effee200640 1 --2- 172.21.15.31:0/3891884812 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7effd00a2f90 0x7effd00a5450 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7effe8102610 tx=0x7effdc005f90 comp rx=0 tx=0).stop 2024-03-21T10:35:02.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.962+0000 7effee200640 1 -- 172.21.15.31:0/3891884812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effe81027b0 msgr2=0x7effe819a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:02.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.962+0000 7effee200640 1 --2- 172.21.15.31:0/3891884812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effe81027b0 0x7effe819a010 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7effd4005940 tx=0x7effd4032470 comp rx=0 tx=0).stop 2024-03-21T10:35:02.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.962+0000 7effee200640 1 -- 172.21.15.31:0/3891884812 shutdown_connections 2024-03-21T10:35:02.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.963+0000 7effee200640 1 --2- 172.21.15.31:0/3891884812 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7effd00a2f90 0x7effd00a5450 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:02.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.963+0000 7effee200640 1 --2- 172.21.15.31:0/3891884812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effe8103170 0x7effe81a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:02.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.963+0000 7effee200640 1 --2- 172.21.15.31:0/3891884812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effe81027b0 0x7effe819a010 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:02.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.963+0000 7effee200640 1 --2- 172.21.15.31:0/3891884812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effe81015c0 0x7effe8199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:02.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.963+0000 7effee200640 1 -- 172.21.15.31:0/3891884812 >> 172.21.15.31:0/3891884812 conn(0x7effe80fcb30 msgr2=0x7effe80738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:02.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.963+0000 7effee200640 1 -- 172.21.15.31:0/3891884812 shutdown_connections 2024-03-21T10:35:02.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:02.963+0000 7effee200640 1 -- 172.21.15.31:0/3891884812 wait complete. 2024-03-21T10:35:02.979 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T10:29:44.553968+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-21T10:35:02.980 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T10:35:03.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.197+0000 7f4b5b600640 1 -- 172.21.15.31:0/1879903164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b54104ad0 msgr2=0x7f4b54104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:03.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.197+0000 7f4b5b600640 1 --2- 172.21.15.31:0/1879903164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b54104ad0 0x7f4b54104f50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4b44009900 tx=0x7f4b4402bc30 comp rx=0 tx=0).stop 2024-03-21T10:35:03.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.197+0000 7f4b5b600640 1 -- 172.21.15.31:0/1879903164 shutdown_connections 2024-03-21T10:35:03.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.197+0000 7f4b5b600640 1 --2- 172.21.15.31:0/1879903164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b54105490 0x7f4b5410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:03.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.197+0000 7f4b5b600640 1 --2- 172.21.15.31:0/1879903164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b54104ad0 0x7f4b54104f50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:03.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.197+0000 7f4b5b600640 1 --2- 172.21.15.31:0/1879903164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b541038e0 0x7f4b54103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:03.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.197+0000 7f4b5b600640 1 -- 172.21.15.31:0/1879903164 >> 172.21.15.31:0/1879903164 conn(0x7f4b540fee50 msgr2=0x7f4b54101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:03.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.197+0000 7f4b5b600640 1 -- 172.21.15.31:0/1879903164 shutdown_connections 2024-03-21T10:35:03.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.197+0000 7f4b5b600640 1 -- 172.21.15.31:0/1879903164 wait complete. 2024-03-21T10:35:03.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.198+0000 7f4b5b600640 1 Processor -- start 2024-03-21T10:35:03.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.199+0000 7f4b5b600640 1 -- start start 2024-03-21T10:35:03.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.199+0000 7f4b5b600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b541038e0 0x7f4b54199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:03.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.199+0000 7f4b5b600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b54104ad0 0x7f4b5419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:03.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.199+0000 7f4b5b600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b54105490 0x7f4b541a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:03.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.199+0000 7f4b5b600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5410f540 con 0x7f4b54104ad0 2024-03-21T10:35:03.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.199+0000 7f4b5b600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5410f3c0 con 0x7f4b54105490 2024-03-21T10:35:03.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.199+0000 7f4b5b600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5410f6c0 con 0x7f4b541038e0 2024-03-21T10:35:03.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.199+0000 7f4b58c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b541038e0 0x7f4b54199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:03.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.199+0000 7f4b59600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b54105490 0x7f4b541a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:03.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.199+0000 7f4b58c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b541038e0 0x7f4b54199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:55264/0 (socket says 172.21.15.31:55264) 2024-03-21T10:35:03.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.199+0000 7f4b58c00640 1 -- 172.21.15.31:0/754213177 learned_addr learned my addr 172.21.15.31:0/754213177 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:03.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.199+0000 7f4b53e00640 1 --2- 172.21.15.31:0/754213177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b54104ad0 0x7f4b5419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:03.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.200+0000 7f4b58c00640 1 -- 172.21.15.31:0/754213177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b54105490 msgr2=0x7f4b541a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:03.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.200+0000 7f4b58c00640 1 --2- 172.21.15.31:0/754213177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b54105490 0x7f4b541a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:03.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.200+0000 7f4b58c00640 1 -- 172.21.15.31:0/754213177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b54104ad0 msgr2=0x7f4b5419a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:03.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.200+0000 7f4b58c00640 1 --2- 172.21.15.31:0/754213177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b54104ad0 0x7f4b5419a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:03.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.200+0000 7f4b58c00640 1 -- 172.21.15.31:0/754213177 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b541a1830 con 0x7f4b541038e0 2024-03-21T10:35:03.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.200+0000 7f4b53e00640 1 --2- 172.21.15.31:0/754213177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b54104ad0 0x7f4b5419a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:35:03.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.200+0000 7f4b58c00640 1 --2- 172.21.15.31:0/754213177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b541038e0 0x7f4b54199c20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4b4800e9c0 tx=0x7f4b4800edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:03.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.200+0000 7f4b51600640 1 -- 172.21.15.31:0/754213177 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4800ccc0 con 0x7f4b541038e0 2024-03-21T10:35:03.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.200+0000 7f4b51600640 1 -- 172.21.15.31:0/754213177 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b480035f0 con 0x7f4b541038e0 2024-03-21T10:35:03.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.200+0000 7f4b5b600640 1 -- 172.21.15.31:0/754213177 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b541a1b20 con 0x7f4b541038e0 2024-03-21T10:35:03.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.200+0000 7f4b51600640 1 -- 172.21.15.31:0/754213177 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4800ce60 con 0x7f4b541038e0 2024-03-21T10:35:03.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.200+0000 7f4b5b600640 1 -- 172.21.15.31:0/754213177 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b541a2070 con 0x7f4b541038e0 2024-03-21T10:35:03.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.202+0000 7f4b5b600640 1 -- 172.21.15.31:0/754213177 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b5410ac00 con 0x7f4b541038e0 2024-03-21T10:35:03.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.204+0000 7f4b51600640 1 -- 172.21.15.31:0/754213177 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f4b480037a0 con 0x7f4b541038e0 2024-03-21T10:35:03.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.205+0000 7f4b51600640 1 --2- 172.21.15.31:0/754213177 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4b3c0a2f20 0x7f4b3c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:03.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.205+0000 7f4b51600640 1 -- 172.21.15.31:0/754213177 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4b480cf4d0 con 0x7f4b541038e0 2024-03-21T10:35:03.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.205+0000 7f4b53e00640 1 --2- 172.21.15.31:0/754213177 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4b3c0a2f20 0x7f4b3c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:03.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.206+0000 7f4b53e00640 1 --2- 172.21.15.31:0/754213177 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4b3c0a2f20 0x7f4b3c0a53e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4b44006010 tx=0x7f4b4402c8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:03.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.213+0000 7f4b51600640 1 -- 172.21.15.31:0/754213177 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4b4809c700 con 0x7f4b541038e0 2024-03-21T10:35:03.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.421+0000 7f4b5b600640 1 -- 172.21.15.31:0/754213177 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4b54062340 con 0x7f4b541038e0 2024-03-21T10:35:03.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.422+0000 7f4b51600640 1 -- 172.21.15.31:0/754213177 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4b4809c000 con 0x7f4b541038e0 2024-03-21T10:35:03.420 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-03-21T10:35:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.425+0000 7f4b5b600640 1 -- 172.21.15.31:0/754213177 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4b3c0a2f20 msgr2=0x7f4b3c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.425+0000 7f4b5b600640 1 --2- 172.21.15.31:0/754213177 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4b3c0a2f20 0x7f4b3c0a53e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4b44006010 tx=0x7f4b4402c8a0 comp rx=0 tx=0).stop 2024-03-21T10:35:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.425+0000 7f4b5b600640 1 -- 172.21.15.31:0/754213177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b541038e0 msgr2=0x7f4b54199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.425+0000 7f4b5b600640 1 --2- 172.21.15.31:0/754213177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b541038e0 0x7f4b54199c20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4b4800e9c0 tx=0x7f4b4800edc0 comp rx=0 tx=0).stop 2024-03-21T10:35:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.425+0000 7f4b5b600640 1 -- 172.21.15.31:0/754213177 shutdown_connections 2024-03-21T10:35:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.425+0000 7f4b5b600640 1 --2- 172.21.15.31:0/754213177 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4b3c0a2f20 0x7f4b3c0a53e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.425+0000 7f4b5b600640 1 --2- 172.21.15.31:0/754213177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b54105490 0x7f4b541a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.425+0000 7f4b5b600640 1 --2- 172.21.15.31:0/754213177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b54104ad0 0x7f4b5419a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.425+0000 7f4b5b600640 1 --2- 172.21.15.31:0/754213177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b541038e0 0x7f4b54199c20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.425+0000 7f4b5b600640 1 -- 172.21.15.31:0/754213177 >> 172.21.15.31:0/754213177 conn(0x7f4b540fee50 msgr2=0x7f4b54107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:03.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.426+0000 7f4b5b600640 1 -- 172.21.15.31:0/754213177 shutdown_connections 2024-03-21T10:35:03.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.426+0000 7f4b5b600640 1 -- 172.21.15.31:0/754213177 wait complete. 2024-03-21T10:35:03.446 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-21T10:35:03.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.679+0000 7f6eb3c00640 1 -- 172.21.15.31:0/2376787496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6eac104af0 msgr2=0x7f6eac106ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:03.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.679+0000 7f6eb3c00640 1 --2- 172.21.15.31:0/2376787496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6eac104af0 0x7f6eac106ee0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6e98009900 tx=0x7f6e9802bc40 comp rx=0 tx=0).stop 2024-03-21T10:35:03.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.679+0000 7f6eb3c00640 1 -- 172.21.15.31:0/2376787496 shutdown_connections 2024-03-21T10:35:03.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.679+0000 7f6eb3c00640 1 --2- 172.21.15.31:0/2376787496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6eac107420 0x7f6eac109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:03.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.679+0000 7f6eb3c00640 1 --2- 172.21.15.31:0/2376787496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6eac104af0 0x7f6eac106ee0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:03.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.679+0000 7f6eb3c00640 1 --2- 172.21.15.31:0/2376787496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6eac1021c0 0x7f6eac1045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:03.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.679+0000 7f6eb3c00640 1 -- 172.21.15.31:0/2376787496 >> 172.21.15.31:0/2376787496 conn(0x7f6eac0fbbd0 msgr2=0x7f6eac0fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:03.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.679+0000 7f6eb3c00640 1 -- 172.21.15.31:0/2376787496 shutdown_connections 2024-03-21T10:35:03.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.679+0000 7f6eb3c00640 1 -- 172.21.15.31:0/2376787496 wait complete. 2024-03-21T10:35:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.680+0000 7f6eb3c00640 1 Processor -- start 2024-03-21T10:35:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.680+0000 7f6eb3c00640 1 -- start start 2024-03-21T10:35:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.681+0000 7f6eb3c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6eac1021c0 0x7f6eac1979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.681+0000 7f6eb3c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6eac104af0 0x7f6eac197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.681+0000 7f6eb3c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6eac107420 0x7f6eac19ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.681+0000 7f6eb3c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eac10d3a0 con 0x7f6eac107420 2024-03-21T10:35:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.681+0000 7f6eb3c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eac10d220 con 0x7f6eac1021c0 2024-03-21T10:35:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.681+0000 7f6eb3c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eac10d520 con 0x7f6eac104af0 2024-03-21T10:35:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.681+0000 7f6eb1200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6eac1021c0 0x7f6eac1979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.681+0000 7f6eb1200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6eac1021c0 0x7f6eac1979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56412/0 (socket says 172.21.15.31:56412) 2024-03-21T10:35:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.681+0000 7f6eb1200640 1 -- 172.21.15.31:0/2417490511 learned_addr learned my addr 172.21.15.31:0/2417490511 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.681+0000 7f6eabe00640 1 --2- 172.21.15.31:0/2417490511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6eac104af0 0x7f6eac197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.681+0000 7f6eb1c00640 1 --2- 172.21.15.31:0/2417490511 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6eac107420 0x7f6eac19ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.681+0000 7f6eb1200640 1 -- 172.21.15.31:0/2417490511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6eac104af0 msgr2=0x7f6eac197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.681+0000 7f6eb1200640 1 --2- 172.21.15.31:0/2417490511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6eac104af0 0x7f6eac197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.681+0000 7f6eb1200640 1 -- 172.21.15.31:0/2417490511 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6eac107420 msgr2=0x7f6eac19ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.681+0000 7f6eb1200640 1 --2- 172.21.15.31:0/2417490511 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6eac107420 0x7f6eac19ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.681+0000 7f6eb1200640 1 -- 172.21.15.31:0/2417490511 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eac19f5c0 con 0x7f6eac1021c0 2024-03-21T10:35:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.682+0000 7f6eb1200640 1 --2- 172.21.15.31:0/2417490511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6eac1021c0 0x7f6eac1979a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6ea000d950 tx=0x7f6ea000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.682+0000 7f6ea9600640 1 -- 172.21.15.31:0/2417490511 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea0017030 con 0x7f6eac1021c0 2024-03-21T10:35:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.682+0000 7f6ea9600640 1 -- 172.21.15.31:0/2417490511 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6ea0004020 con 0x7f6eac1021c0 2024-03-21T10:35:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.682+0000 7f6eb3c00640 1 -- 172.21.15.31:0/2417490511 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eac19f8b0 con 0x7f6eac1021c0 2024-03-21T10:35:03.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.682+0000 7f6ea9600640 1 -- 172.21.15.31:0/2417490511 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea00047a0 con 0x7f6eac1021c0 2024-03-21T10:35:03.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.682+0000 7f6eb3c00640 1 -- 172.21.15.31:0/2417490511 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eac19fe20 con 0x7f6eac1021c0 2024-03-21T10:35:03.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:03 smithi040 ceph-mon[36154]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:35:03.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:03 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/176994272' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:35:03.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:03 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3891884812' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:03.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:03 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/754213177' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:35:03.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.683+0000 7f6eb3c00640 1 -- 172.21.15.31:0/2417490511 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eac0fd6c0 con 0x7f6eac1021c0 2024-03-21T10:35:03.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.685+0000 7f6ea9600640 1 -- 172.21.15.31:0/2417490511 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f6ea00041d0 con 0x7f6eac1021c0 2024-03-21T10:35:03.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.687+0000 7f6ea9600640 1 --2- 172.21.15.31:0/2417490511 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f6e940a2fa0 0x7f6e940a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:03.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.687+0000 7f6ea9600640 1 -- 172.21.15.31:0/2417490511 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6ea00ceaa0 con 0x7f6eac1021c0 2024-03-21T10:35:03.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.687+0000 7f6eabe00640 1 --2- 172.21.15.31:0/2417490511 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f6e940a2fa0 0x7f6e940a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:03.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.688+0000 7f6eabe00640 1 --2- 172.21.15.31:0/2417490511 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f6e940a2fa0 0x7f6e940a5460 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6e9802c180 tx=0x7f6e98000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:03.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.695+0000 7f6ea9600640 1 -- 172.21.15.31:0/2417490511 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6ea009bcd0 con 0x7f6eac1021c0 2024-03-21T10:35:03.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:03 smithi031 ceph-mon[31673]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:35:03.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:03 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/176994272' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:35:03.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:03 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3891884812' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:03.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:03 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/754213177' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:35:03.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:03 smithi070 ceph-mon[36330]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:35:03.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:03 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/176994272' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:35:03.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:03 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3891884812' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:03.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:03 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/754213177' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:35:03.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.921+0000 7f6eb3c00640 1 -- 172.21.15.31:0/2417490511 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f6eac0622d0 con 0x7f6eac1021c0 2024-03-21T10:35:03.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.922+0000 7f6ea9600640 1 -- 172.21.15.31:0/2417490511 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) v1 ==== 84+0+65 (secure 0 0 0) 0x7f6ea009b5d0 con 0x7f6eac1021c0 2024-03-21T10:35:03.920 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:35:03.920 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-03-21T10:35:03.920 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:35:03.921 INFO:teuthology.orchestra.run.smithi031.stdout: "poolnum": 1, 2024-03-21T10:35:03.921 INFO:teuthology.orchestra.run.smithi031.stdout: "poolname": ".mgr" 2024-03-21T10:35:03.921 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:35:03.921 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-03-21T10:35:03.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.926+0000 7f6eb3c00640 1 -- 172.21.15.31:0/2417490511 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f6e940a2fa0 msgr2=0x7f6e940a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:03.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.926+0000 7f6eb3c00640 1 --2- 172.21.15.31:0/2417490511 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f6e940a2fa0 0x7f6e940a5460 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6e9802c180 tx=0x7f6e98000bc0 comp rx=0 tx=0).stop 2024-03-21T10:35:03.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.926+0000 7f6eb3c00640 1 -- 172.21.15.31:0/2417490511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6eac1021c0 msgr2=0x7f6eac1979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:03.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.926+0000 7f6eb3c00640 1 --2- 172.21.15.31:0/2417490511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6eac1021c0 0x7f6eac1979a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6ea000d950 tx=0x7f6ea000de10 comp rx=0 tx=0).stop 2024-03-21T10:35:03.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.926+0000 7f6eb3c00640 1 -- 172.21.15.31:0/2417490511 shutdown_connections 2024-03-21T10:35:03.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.926+0000 7f6eb3c00640 1 --2- 172.21.15.31:0/2417490511 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f6e940a2fa0 0x7f6e940a5460 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:03.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.926+0000 7f6eb3c00640 1 --2- 172.21.15.31:0/2417490511 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6eac107420 0x7f6eac19ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:03.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.926+0000 7f6eb3c00640 1 --2- 172.21.15.31:0/2417490511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6eac104af0 0x7f6eac197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:03.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.926+0000 7f6eb3c00640 1 --2- 172.21.15.31:0/2417490511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6eac1021c0 0x7f6eac1979a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:03.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.926+0000 7f6eb3c00640 1 -- 172.21.15.31:0/2417490511 >> 172.21.15.31:0/2417490511 conn(0x7f6eac0fbbd0 msgr2=0x7f6eac102b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:03.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.926+0000 7f6eb3c00640 1 -- 172.21.15.31:0/2417490511 shutdown_connections 2024-03-21T10:35:03.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:03.926+0000 7f6eb3c00640 1 -- 172.21.15.31:0/2417490511 wait complete. 2024-03-21T10:35:03.952 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-21T10:35:03.953 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-21T10:35:04.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.176+0000 7f5750200640 1 -- 172.21.15.31:0/1517575569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f574810ff20 msgr2=0x7f5748112310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:04.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.176+0000 7f5750200640 1 --2- 172.21.15.31:0/1517575569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f574810ff20 0x7f5748112310 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5740009be0 tx=0x7f574002bef0 comp rx=0 tx=0).stop 2024-03-21T10:35:04.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.176+0000 7f5750200640 1 -- 172.21.15.31:0/1517575569 shutdown_connections 2024-03-21T10:35:04.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.176+0000 7f5750200640 1 --2- 172.21.15.31:0/1517575569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f574810ff20 0x7f5748112310 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:04.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.176+0000 7f5750200640 1 --2- 172.21.15.31:0/1517575569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5748109000 0x7f574810f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:04.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.176+0000 7f5750200640 1 --2- 172.21.15.31:0/1517575569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57481086a0 0x7f5748108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:04.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.176+0000 7f5750200640 1 -- 172.21.15.31:0/1517575569 >> 172.21.15.31:0/1517575569 conn(0x7f574806e640 msgr2=0x7f5748070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:04.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.176+0000 7f5750200640 1 -- 172.21.15.31:0/1517575569 shutdown_connections 2024-03-21T10:35:04.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.176+0000 7f5750200640 1 -- 172.21.15.31:0/1517575569 wait complete. 2024-03-21T10:35:04.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.177+0000 7f5750200640 1 Processor -- start 2024-03-21T10:35:04.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.177+0000 7f5750200640 1 -- start start 2024-03-21T10:35:04.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.178+0000 7f5750200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57481086a0 0x7f574810b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:04.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.178+0000 7f5750200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5748109000 0x7f5748109970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:04.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.178+0000 7f5750200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f574810ff20 0x7f5748109eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:04.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.178+0000 7f5750200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5748115b10 con 0x7f574810ff20 2024-03-21T10:35:04.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.178+0000 7f5750200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5748115990 con 0x7f5748109000 2024-03-21T10:35:04.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.178+0000 7f5750200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5748115c90 con 0x7f57481086a0 2024-03-21T10:35:04.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.178+0000 7f574ce00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5748109000 0x7f5748109970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:04.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.178+0000 7f574ce00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5748109000 0x7f5748109970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56432/0 (socket says 172.21.15.31:56432) 2024-03-21T10:35:04.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.178+0000 7f574e200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f574810ff20 0x7f5748109eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:04.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.178+0000 7f574ce00640 1 -- 172.21.15.31:0/1079625750 learned_addr learned my addr 172.21.15.31:0/1079625750 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:04.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.178+0000 7f574d800640 1 --2- 172.21.15.31:0/1079625750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57481086a0 0x7f574810b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:04.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.178+0000 7f574ce00640 1 -- 172.21.15.31:0/1079625750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57481086a0 msgr2=0x7f574810b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:04.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.178+0000 7f574ce00640 1 --2- 172.21.15.31:0/1079625750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57481086a0 0x7f574810b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:04.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.178+0000 7f574ce00640 1 -- 172.21.15.31:0/1079625750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f574810ff20 msgr2=0x7f5748109eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:04.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.178+0000 7f574ce00640 1 --2- 172.21.15.31:0/1079625750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f574810ff20 0x7f5748109eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:04.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.178+0000 7f574ce00640 1 -- 172.21.15.31:0/1079625750 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574810a740 con 0x7f5748109000 2024-03-21T10:35:04.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.178+0000 7f574d800640 1 --2- 172.21.15.31:0/1079625750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57481086a0 0x7f574810b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:35:04.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.179+0000 7f574ce00640 1 --2- 172.21.15.31:0/1079625750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5748109000 0x7f5748109970 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f573400e9f0 tx=0x7f573400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:04.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.179+0000 7f5746000640 1 -- 172.21.15.31:0/1079625750 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573400ccc0 con 0x7f5748109000 2024-03-21T10:35:04.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.179+0000 7f5746000640 1 -- 172.21.15.31:0/1079625750 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f573400ce60 con 0x7f5748109000 2024-03-21T10:35:04.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.179+0000 7f5750200640 1 -- 172.21.15.31:0/1079625750 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57481aa250 con 0x7f5748109000 2024-03-21T10:35:04.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.179+0000 7f5746000640 1 -- 172.21.15.31:0/1079625750 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5734010690 con 0x7f5748109000 2024-03-21T10:35:04.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.179+0000 7f5750200640 1 -- 172.21.15.31:0/1079625750 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57481aa7a0 con 0x7f5748109000 2024-03-21T10:35:04.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.180+0000 7f5750200640 1 -- 172.21.15.31:0/1079625750 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57481aaac0 con 0x7f5748109000 2024-03-21T10:35:04.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.183+0000 7f5746000640 1 -- 172.21.15.31:0/1079625750 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f57340046c0 con 0x7f5748109000 2024-03-21T10:35:04.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.184+0000 7f5746000640 1 --2- 172.21.15.31:0/1079625750 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f57300a2fa0 0x7f57300a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:04.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.185+0000 7f5746000640 1 -- 172.21.15.31:0/1079625750 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f57340cf3c0 con 0x7f5748109000 2024-03-21T10:35:04.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.185+0000 7f574d800640 1 --2- 172.21.15.31:0/1079625750 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f57300a2fa0 0x7f57300a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:04.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.186+0000 7f574d800640 1 --2- 172.21.15.31:0/1079625750 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f57300a2fa0 0x7f57300a5460 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f573c005fd0 tx=0x7f573c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:04.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.188+0000 7f5746000640 1 -- 172.21.15.31:0/1079625750 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f573409c5f0 con 0x7f5748109000 2024-03-21T10:35:04.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.400+0000 7f5750200640 1 -- 172.21.15.31:0/1079625750 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f57481aac90 con 0x7f5748109000 2024-03-21T10:35:04.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.985+0000 7f5746000640 1 -- 172.21.15.31:0/1079625750 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) v1 ==== 138+0+0 (secure 0 0 0) 0x7f573409bef0 con 0x7f5748109000 2024-03-21T10:35:04.985 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' created 2024-03-21T10:35:04.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.990+0000 7f5750200640 1 -- 172.21.15.31:0/1079625750 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f57300a2fa0 msgr2=0x7f57300a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:04.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.990+0000 7f5750200640 1 --2- 172.21.15.31:0/1079625750 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f57300a2fa0 0x7f57300a5460 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f573c005fd0 tx=0x7f573c009260 comp rx=0 tx=0).stop 2024-03-21T10:35:04.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.990+0000 7f5750200640 1 -- 172.21.15.31:0/1079625750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5748109000 msgr2=0x7f5748109970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:04.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.990+0000 7f5750200640 1 --2- 172.21.15.31:0/1079625750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5748109000 0x7f5748109970 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f573400e9f0 tx=0x7f573400edf0 comp rx=0 tx=0).stop 2024-03-21T10:35:04.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.991+0000 7f5750200640 1 -- 172.21.15.31:0/1079625750 shutdown_connections 2024-03-21T10:35:04.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.991+0000 7f5750200640 1 --2- 172.21.15.31:0/1079625750 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f57300a2fa0 0x7f57300a5460 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:04.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.991+0000 7f5750200640 1 --2- 172.21.15.31:0/1079625750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f574810ff20 0x7f5748109eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:04.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.991+0000 7f5750200640 1 --2- 172.21.15.31:0/1079625750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5748109000 0x7f5748109970 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:04.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.991+0000 7f5750200640 1 --2- 172.21.15.31:0/1079625750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57481086a0 0x7f574810b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:04.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.991+0000 7f5750200640 1 -- 172.21.15.31:0/1079625750 >> 172.21.15.31:0/1079625750 conn(0x7f574806e640 msgr2=0x7f5748070290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:04.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.992+0000 7f5750200640 1 -- 172.21.15.31:0/1079625750 shutdown_connections 2024-03-21T10:35:04.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:04.992+0000 7f5750200640 1 -- 172.21.15.31:0/1079625750 wait complete. 2024-03-21T10:35:05.014 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-21T10:35:05.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.204+0000 7f01c6c00640 1 -- 172.21.15.31:0/2833418549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01c01048f0 msgr2=0x7f01c0104d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:05.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.204+0000 7f01c6c00640 1 --2- 172.21.15.31:0/2833418549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01c01048f0 0x7f01c0104d70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f01ac009900 tx=0x7f01ac02bc30 comp rx=0 tx=0).stop 2024-03-21T10:35:05.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.204+0000 7f01c6c00640 1 -- 172.21.15.31:0/2833418549 shutdown_connections 2024-03-21T10:35:05.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.204+0000 7f01c6c00640 1 --2- 172.21.15.31:0/2833418549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01c01052b0 0x7f01c010bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:05.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.204+0000 7f01c6c00640 1 --2- 172.21.15.31:0/2833418549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01c01048f0 0x7f01c0104d70 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:05.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.204+0000 7f01c6c00640 1 --2- 172.21.15.31:0/2833418549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01c0103700 0x7f01c0103b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:05.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.204+0000 7f01c6c00640 1 -- 172.21.15.31:0/2833418549 >> 172.21.15.31:0/2833418549 conn(0x7f01c00fec70 msgr2=0x7f01c01010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:05.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.204+0000 7f01c6c00640 1 -- 172.21.15.31:0/2833418549 shutdown_connections 2024-03-21T10:35:05.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.204+0000 7f01c6c00640 1 -- 172.21.15.31:0/2833418549 wait complete. 2024-03-21T10:35:05.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.205+0000 7f01c6c00640 1 Processor -- start 2024-03-21T10:35:05.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.205+0000 7f01c6c00640 1 -- start start 2024-03-21T10:35:05.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.206+0000 7f01c6c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01c0103700 0x7f01c0079450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:05.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.206+0000 7f01c6c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01c01048f0 0x7f01c0079990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:05.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.206+0000 7f01c6c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01c01052b0 0x7f01c0075fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:05.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.206+0000 7f01c6c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c010f3d0 con 0x7f01c01048f0 2024-03-21T10:35:05.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.206+0000 7f01c6c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c010f250 con 0x7f01c0103700 2024-03-21T10:35:05.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.206+0000 7f01c6c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c010f550 con 0x7f01c01052b0 2024-03-21T10:35:05.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.206+0000 7f01c4e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01c01048f0 0x7f01c0079990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:05.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.206+0000 7f01c5800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01c0103700 0x7f01c0079450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:05.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.206+0000 7f01c5800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01c0103700 0x7f01c0079450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56442/0 (socket says 172.21.15.31:56442) 2024-03-21T10:35:05.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.206+0000 7f01c5800640 1 -- 172.21.15.31:0/208083896 learned_addr learned my addr 172.21.15.31:0/208083896 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:05.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.206+0000 7f01c6200640 1 --2- 172.21.15.31:0/208083896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01c01052b0 0x7f01c0075fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:05.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.206+0000 7f01c5800640 1 -- 172.21.15.31:0/208083896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01c01052b0 msgr2=0x7f01c0075fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:05.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.206+0000 7f01c5800640 1 --2- 172.21.15.31:0/208083896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01c01052b0 0x7f01c0075fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:05.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.206+0000 7f01c5800640 1 -- 172.21.15.31:0/208083896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01c01048f0 msgr2=0x7f01c0079990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:05.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.206+0000 7f01c5800640 1 --2- 172.21.15.31:0/208083896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01c01048f0 0x7f01c0079990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:05.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.206+0000 7f01c5800640 1 -- 172.21.15.31:0/208083896 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c0076880 con 0x7f01c0103700 2024-03-21T10:35:05.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.207+0000 7f01c5800640 1 --2- 172.21.15.31:0/208083896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01c0103700 0x7f01c0079450 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f01b400d770 tx=0x7f01b400dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:05.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.207+0000 7f01be000640 1 -- 172.21.15.31:0/208083896 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b4017070 con 0x7f01c0103700 2024-03-21T10:35:05.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.207+0000 7f01be000640 1 -- 172.21.15.31:0/208083896 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f01b4009c30 con 0x7f01c0103700 2024-03-21T10:35:05.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.207+0000 7f01c6c00640 1 -- 172.21.15.31:0/208083896 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01c01a5e30 con 0x7f01c0103700 2024-03-21T10:35:05.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.207+0000 7f01be000640 1 -- 172.21.15.31:0/208083896 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b4003dd0 con 0x7f01c0103700 2024-03-21T10:35:05.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.207+0000 7f01c6c00640 1 -- 172.21.15.31:0/208083896 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c01a61e0 con 0x7f01c0103700 2024-03-21T10:35:05.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.208+0000 7f01c6c00640 1 -- 172.21.15.31:0/208083896 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01c004f400 con 0x7f01c0103700 2024-03-21T10:35:05.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.210+0000 7f01be000640 1 -- 172.21.15.31:0/208083896 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f01b4003f70 con 0x7f01c0103700 2024-03-21T10:35:05.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.211+0000 7f01be000640 1 --2- 172.21.15.31:0/208083896 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f01a80a2fa0 0x7f01a80a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:05.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.212+0000 7f01be000640 1 -- 172.21.15.31:0/208083896 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f01b40ceb60 con 0x7f01c0103700 2024-03-21T10:35:05.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.212+0000 7f01c4e00640 1 --2- 172.21.15.31:0/208083896 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f01a80a2fa0 0x7f01a80a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:05.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.212+0000 7f01c4e00640 1 --2- 172.21.15.31:0/208083896 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f01a80a2fa0 0x7f01a80a5460 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f01ac02c240 tx=0x7f01ac02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:05.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.214+0000 7f01be000640 1 -- 172.21.15.31:0/208083896 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f01b4009de0 con 0x7f01c0103700 2024-03-21T10:35:05.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:04 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:05.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:04 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:05.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:04 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:05.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:04 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:05.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:04 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2417490511' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-21T10:35:05.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:04 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:05.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:04 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1079625750' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-21T10:35:05.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:04 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-21T10:35:05.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:04 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:05.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:04 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:05.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:04 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:05.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:04 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:05.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:04 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2417490511' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-21T10:35:05.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:04 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:05.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:04 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1079625750' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-21T10:35:05.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:04 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-21T10:35:05.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.419+0000 7f01c6c00640 1 -- 172.21.15.31:0/208083896 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f01c010aa20 con 0x7f01c0103700 2024-03-21T10:35:05.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:04 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:05.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:04 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:05.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:04 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:05.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:04 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:05.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:04 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2417490511' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-21T10:35:05.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:04 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:05.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:04 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1079625750' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-21T10:35:05.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:04 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-21T10:35:05.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:05.997+0000 7f01be000640 1 -- 172.21.15.31:0/208083896 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) v1 ==== 171+0+0 (secure 0 0 0) 0x7f01b409b650 con 0x7f01c0103700 2024-03-21T10:35:05.999 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' created 2024-03-21T10:35:06.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.010+0000 7f01c6c00640 1 -- 172.21.15.31:0/208083896 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f01a80a2fa0 msgr2=0x7f01a80a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:06.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.010+0000 7f01c6c00640 1 --2- 172.21.15.31:0/208083896 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f01a80a2fa0 0x7f01a80a5460 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f01ac02c240 tx=0x7f01ac02df90 comp rx=0 tx=0).stop 2024-03-21T10:35:06.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.010+0000 7f01c6c00640 1 -- 172.21.15.31:0/208083896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01c0103700 msgr2=0x7f01c0079450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:06.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.010+0000 7f01c6c00640 1 --2- 172.21.15.31:0/208083896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01c0103700 0x7f01c0079450 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f01b400d770 tx=0x7f01b400dc30 comp rx=0 tx=0).stop 2024-03-21T10:35:06.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.010+0000 7f01c6c00640 1 -- 172.21.15.31:0/208083896 shutdown_connections 2024-03-21T10:35:06.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.010+0000 7f01c6c00640 1 --2- 172.21.15.31:0/208083896 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f01a80a2fa0 0x7f01a80a5460 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:06.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.010+0000 7f01c6c00640 1 --2- 172.21.15.31:0/208083896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01c01052b0 0x7f01c0075fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:06.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.010+0000 7f01c6c00640 1 --2- 172.21.15.31:0/208083896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01c01048f0 0x7f01c0079990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:06.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.010+0000 7f01c6c00640 1 --2- 172.21.15.31:0/208083896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01c0103700 0x7f01c0079450 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:06.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.010+0000 7f01c6c00640 1 -- 172.21.15.31:0/208083896 >> 172.21.15.31:0/208083896 conn(0x7f01c00fec70 msgr2=0x7f01c0107bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:06.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.011+0000 7f01c6c00640 1 -- 172.21.15.31:0/208083896 shutdown_connections 2024-03-21T10:35:06.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.011+0000 7f01c6c00640 1 -- 172.21.15.31:0/208083896 wait complete. 2024-03-21T10:35:06.038 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-21T10:35:06.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.229+0000 7fddc9600640 1 -- 172.21.15.31:0/2059092923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddc41028b0 msgr2=0x7fddc4102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:06.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.229+0000 7fddc9600640 1 --2- 172.21.15.31:0/2059092923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddc41028b0 0x7fddc4102d30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fddb0009860 tx=0x7fddb002bb70 comp rx=0 tx=0).stop 2024-03-21T10:35:06.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.229+0000 7fddc9600640 1 -- 172.21.15.31:0/2059092923 shutdown_connections 2024-03-21T10:35:06.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.229+0000 7fddc9600640 1 --2- 172.21.15.31:0/2059092923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddc4103270 0x7fddc4109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:06.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.229+0000 7fddc9600640 1 --2- 172.21.15.31:0/2059092923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddc41028b0 0x7fddc4102d30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:06.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.229+0000 7fddc9600640 1 --2- 172.21.15.31:0/2059092923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddc41016c0 0x7fddc4101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:06.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.229+0000 7fddc9600640 1 -- 172.21.15.31:0/2059092923 >> 172.21.15.31:0/2059092923 conn(0x7fddc40fcc30 msgr2=0x7fddc40ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:06.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.229+0000 7fddc9600640 1 -- 172.21.15.31:0/2059092923 shutdown_connections 2024-03-21T10:35:06.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.229+0000 7fddc9600640 1 -- 172.21.15.31:0/2059092923 wait complete. 2024-03-21T10:35:06.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.230+0000 7fddc9600640 1 Processor -- start 2024-03-21T10:35:06.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.230+0000 7fddc9600640 1 -- start start 2024-03-21T10:35:06.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.230+0000 7fddc9600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddc41016c0 0x7fddc4075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:06.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.230+0000 7fddc9600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddc41028b0 0x7fddc4073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:06.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.231+0000 7fddc9600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddc4103270 0x7fddc4073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:06.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.231+0000 7fddc9600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc410d320 con 0x7fddc4103270 2024-03-21T10:35:06.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.231+0000 7fddc9600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc410d1a0 con 0x7fddc41016c0 2024-03-21T10:35:06.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.231+0000 7fddc9600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc410d4a0 con 0x7fddc41028b0 2024-03-21T10:35:06.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.231+0000 7fddc3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddc4103270 0x7fddc4073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:06.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.231+0000 7fddc2000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddc41028b0 0x7fddc4073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:06.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.231+0000 7fddc2000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddc41028b0 0x7fddc4073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:55338/0 (socket says 172.21.15.31:55338) 2024-03-21T10:35:06.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.231+0000 7fddc2000640 1 -- 172.21.15.31:0/4154206699 learned_addr learned my addr 172.21.15.31:0/4154206699 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:06.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.231+0000 7fddc2a00640 1 --2- 172.21.15.31:0/4154206699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddc41016c0 0x7fddc4075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:06.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.231+0000 7fddc2000640 1 -- 172.21.15.31:0/4154206699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddc41016c0 msgr2=0x7fddc4075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:06.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.231+0000 7fddc2000640 1 --2- 172.21.15.31:0/4154206699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddc41016c0 0x7fddc4075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:06.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.231+0000 7fddc2000640 1 -- 172.21.15.31:0/4154206699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddc4103270 msgr2=0x7fddc4073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:06.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.231+0000 7fddc2000640 1 --2- 172.21.15.31:0/4154206699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddc4103270 0x7fddc4073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:06.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.231+0000 7fddc2000640 1 -- 172.21.15.31:0/4154206699 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc4074860 con 0x7fddc41028b0 2024-03-21T10:35:06.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.232+0000 7fddc2000640 1 --2- 172.21.15.31:0/4154206699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddc41028b0 0x7fddc4073a90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fddb0005940 tx=0x7fddb0032470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:06.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.232+0000 7fddbb400640 1 -- 172.21.15.31:0/4154206699 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb0041020 con 0x7fddc41028b0 2024-03-21T10:35:06.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.232+0000 7fddc9600640 1 -- 172.21.15.31:0/4154206699 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddc41a1a80 con 0x7fddc41028b0 2024-03-21T10:35:06.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.232+0000 7fddbb400640 1 -- 172.21.15.31:0/4154206699 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fddb002c9f0 con 0x7fddc41028b0 2024-03-21T10:35:06.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.232+0000 7fddbb400640 1 -- 172.21.15.31:0/4154206699 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb0039040 con 0x7fddc41028b0 2024-03-21T10:35:06.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.232+0000 7fddc9600640 1 -- 172.21.15.31:0/4154206699 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddc41a1fd0 con 0x7fddc41028b0 2024-03-21T10:35:06.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.234+0000 7fddc9600640 1 -- 172.21.15.31:0/4154206699 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddc4108a30 con 0x7fddc41028b0 2024-03-21T10:35:06.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.238+0000 7fddbb400640 1 -- 172.21.15.31:0/4154206699 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fddb0044020 con 0x7fddc41028b0 2024-03-21T10:35:06.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.240+0000 7fddbb400640 1 --2- 172.21.15.31:0/4154206699 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fdda80a2f20 0x7fdda80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:06.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.240+0000 7fddbb400640 1 -- 172.21.15.31:0/4154206699 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7879+0+0 (secure 0 0 0) 0x7fddb00f0c50 con 0x7fddc41028b0 2024-03-21T10:35:06.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.240+0000 7fddc2a00640 1 --2- 172.21.15.31:0/4154206699 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fdda80a2f20 0x7fdda80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:06.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.241+0000 7fddc2a00640 1 --2- 172.21.15.31:0/4154206699 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fdda80a2f20 0x7fdda80a53e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fddc4102710 tx=0x7fddb4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:06.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.243+0000 7fddbb400640 1 -- 172.21.15.31:0/4154206699 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fddb004b020 con 0x7fddc41028b0 2024-03-21T10:35:06.250 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:05 smithi040 ceph-mon[36154]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:35:06.250 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:05 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:06.251 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:05 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:06.251 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:05 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:35:06.251 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:05 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:35:06.251 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:05 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-21T10:35:06.251 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:05 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:06.251 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:05 smithi040 ceph-mon[36154]: osdmap e82: 12 total, 12 up, 12 in 2024-03-21T10:35:06.251 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:05 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:35:06.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:05 smithi031 ceph-mon[31673]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:35:06.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:05 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:06.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:05 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:06.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:05 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:35:06.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:05 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:35:06.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:05 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-21T10:35:06.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:05 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:06.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:05 smithi031 ceph-mon[31673]: osdmap e82: 12 total, 12 up, 12 in 2024-03-21T10:35:06.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:05 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:35:06.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:05 smithi070 ceph-mon[36330]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:35:06.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:05 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:06.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:05 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:06.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:05 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:35:06.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:05 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:35:06.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:05 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-21T10:35:06.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:05 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:06.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:05 smithi070 ceph-mon[36330]: osdmap e82: 12 total, 12 up, 12 in 2024-03-21T10:35:06.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:05 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:35:06.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.489+0000 7fddc9600640 1 -- 172.21.15.31:0/4154206699 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fddc402d070 con 0x7fddc41028b0 2024-03-21T10:35:06.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:06.996+0000 7fddbb400640 1 -- 172.21.15.31:0/4154206699 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7fddb00bdb90 con 0x7fddc41028b0 2024-03-21T10:35:06.995 INFO:teuthology.orchestra.run.smithi031.stderr:new fs with metadata pool 2 and data pool 3 2024-03-21T10:35:06.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.000+0000 7fddc9600640 1 -- 172.21.15.31:0/4154206699 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fdda80a2f20 msgr2=0x7fdda80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:06.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.000+0000 7fddc9600640 1 --2- 172.21.15.31:0/4154206699 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fdda80a2f20 0x7fdda80a53e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fddc4102710 tx=0x7fddb4005f90 comp rx=0 tx=0).stop 2024-03-21T10:35:06.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.000+0000 7fddc9600640 1 -- 172.21.15.31:0/4154206699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddc41028b0 msgr2=0x7fddc4073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:06.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.000+0000 7fddc9600640 1 --2- 172.21.15.31:0/4154206699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddc41028b0 0x7fddc4073a90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fddb0005940 tx=0x7fddb0032470 comp rx=0 tx=0).stop 2024-03-21T10:35:06.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.000+0000 7fddc9600640 1 -- 172.21.15.31:0/4154206699 shutdown_connections 2024-03-21T10:35:06.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.000+0000 7fddc9600640 1 --2- 172.21.15.31:0/4154206699 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fdda80a2f20 0x7fdda80a53e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:06.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.000+0000 7fddc9600640 1 --2- 172.21.15.31:0/4154206699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddc4103270 0x7fddc4073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:06.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.000+0000 7fddc9600640 1 --2- 172.21.15.31:0/4154206699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddc41028b0 0x7fddc4073a90 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fddb0005940 tx=0x7fddb0032470 comp rx=0 tx=0).stop 2024-03-21T10:35:06.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.000+0000 7fddc9600640 1 --2- 172.21.15.31:0/4154206699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddc41016c0 0x7fddc4075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:06.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.000+0000 7fddc9600640 1 -- 172.21.15.31:0/4154206699 >> 172.21.15.31:0/4154206699 conn(0x7fddc40fcc30 msgr2=0x7fddc4105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:06.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.001+0000 7fddc9600640 1 -- 172.21.15.31:0/4154206699 shutdown_connections 2024-03-21T10:35:06.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.001+0000 7fddc9600640 1 -- 172.21.15.31:0/4154206699 wait complete. 2024-03-21T10:35:07.022 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-21T10:35:07.022 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-21T10:35:07.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.230+0000 7f7154600640 1 -- 172.21.15.31:0/3144993593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f714c1028c0 msgr2=0x7f714c102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:07.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.230+0000 7f7154600640 1 --2- 172.21.15.31:0/3144993593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f714c1028c0 0x7f714c102d40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7138009900 tx=0x7f713802bc30 comp rx=0 tx=0).stop 2024-03-21T10:35:07.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.230+0000 7f7154600640 1 -- 172.21.15.31:0/3144993593 shutdown_connections 2024-03-21T10:35:07.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.230+0000 7f7154600640 1 --2- 172.21.15.31:0/3144993593 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f714c103280 0x7f714c109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:07.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.230+0000 7f7154600640 1 --2- 172.21.15.31:0/3144993593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f714c1028c0 0x7f714c102d40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:07.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.230+0000 7f7154600640 1 --2- 172.21.15.31:0/3144993593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f714c1016d0 0x7f714c101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:07.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.230+0000 7f7154600640 1 -- 172.21.15.31:0/3144993593 >> 172.21.15.31:0/3144993593 conn(0x7f714c0fcc60 msgr2=0x7f714c0ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:07.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.230+0000 7f7154600640 1 -- 172.21.15.31:0/3144993593 shutdown_connections 2024-03-21T10:35:07.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.230+0000 7f7154600640 1 -- 172.21.15.31:0/3144993593 wait complete. 2024-03-21T10:35:07.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.231+0000 7f7154600640 1 Processor -- start 2024-03-21T10:35:07.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.231+0000 7f7154600640 1 -- start start 2024-03-21T10:35:07.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.231+0000 7f7154600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f714c1016d0 0x7f714c199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:07.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.232+0000 7f7154600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f714c1028c0 0x7f714c19a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:07.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.232+0000 7f7154600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f714c103280 0x7f714c1a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:07.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.232+0000 7f7154600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714c10d330 con 0x7f714c1016d0 2024-03-21T10:35:07.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.232+0000 7f7154600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714c10d1b0 con 0x7f714c103280 2024-03-21T10:35:07.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.232+0000 7f7154600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714c10d4b0 con 0x7f714c1028c0 2024-03-21T10:35:07.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.232+0000 7f7151200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f714c1028c0 0x7f714c19a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:07.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.232+0000 7f7152600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f714c103280 0x7f714c1a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:07.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.232+0000 7f7151200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f714c1028c0 0x7f714c19a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:50682/0 (socket says 172.21.15.31:50682) 2024-03-21T10:35:07.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.232+0000 7f7151200640 1 -- 172.21.15.31:0/1730872613 learned_addr learned my addr 172.21.15.31:0/1730872613 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:07.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.232+0000 7f7151c00640 1 --2- 172.21.15.31:0/1730872613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f714c1016d0 0x7f714c199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:07.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.232+0000 7f7151200640 1 -- 172.21.15.31:0/1730872613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f714c103280 msgr2=0x7f714c1a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:07.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.232+0000 7f7151200640 1 --2- 172.21.15.31:0/1730872613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f714c103280 0x7f714c1a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:07.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.232+0000 7f7151200640 1 -- 172.21.15.31:0/1730872613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f714c1016d0 msgr2=0x7f714c199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:07.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.232+0000 7f7151200640 1 --2- 172.21.15.31:0/1730872613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f714c1016d0 0x7f714c199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:07.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.232+0000 7f7151200640 1 -- 172.21.15.31:0/1730872613 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714c1a1840 con 0x7f714c1028c0 2024-03-21T10:35:07.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.233+0000 7f7151200640 1 --2- 172.21.15.31:0/1730872613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f714c1028c0 0x7f714c19a170 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7138009230 tx=0x7f713802c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:07.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.233+0000 7f714a000640 1 -- 172.21.15.31:0/1730872613 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7138041070 con 0x7f714c1028c0 2024-03-21T10:35:07.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.233+0000 7f714a000640 1 -- 172.21.15.31:0/1730872613 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7138004450 con 0x7f714c1028c0 2024-03-21T10:35:07.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.233+0000 7f7154600640 1 -- 172.21.15.31:0/1730872613 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f714c1a1ad0 con 0x7f714c1028c0 2024-03-21T10:35:07.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.233+0000 7f714a000640 1 -- 172.21.15.31:0/1730872613 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7138003e20 con 0x7f714c1028c0 2024-03-21T10:35:07.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.233+0000 7f7154600640 1 -- 172.21.15.31:0/1730872613 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f714c1a2010 con 0x7f714c1028c0 2024-03-21T10:35:07.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.235+0000 7f7154600640 1 -- 172.21.15.31:0/1730872613 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f714c1a21e0 con 0x7f714c1028c0 2024-03-21T10:35:07.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.236+0000 7f714a000640 1 -- 172.21.15.31:0/1730872613 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f7138044070 con 0x7f714c1028c0 2024-03-21T10:35:07.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.237+0000 7f714a000640 1 --2- 172.21.15.31:0/1730872613 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f71340a2fa0 0x7f71340a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:07.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.238+0000 7f714a000640 1 -- 172.21.15.31:0/1730872613 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f71380f0ec0 con 0x7f714c1028c0 2024-03-21T10:35:07.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.238+0000 7f7151c00640 1 --2- 172.21.15.31:0/1730872613 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f71340a2fa0 0x7f71340a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:07.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.238+0000 7f7151c00640 1 --2- 172.21.15.31:0/1730872613 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f71340a2fa0 0x7f71340a5460 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f714c19b100 tx=0x7f7140005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:07.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.241+0000 7f714a000640 1 -- 172.21.15.31:0/1730872613 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f71380bd750 con 0x7f714c1028c0 2024-03-21T10:35:07.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:06 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/208083896' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-21T10:35:07.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:06 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-21T10:35:07.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:06 smithi031 ceph-mon[31673]: pgmap v318: 33 pgs: 7 creating+peering, 1 active+clean, 25 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:35:07.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:06 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-21T10:35:07.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:06 smithi031 ceph-mon[31673]: osdmap e83: 12 total, 12 up, 12 in 2024-03-21T10:35:07.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:06 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/4154206699' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-21T10:35:07.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:06 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-21T10:35:07.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:06 smithi031 ceph-mon[31673]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-21T10:35:07.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:06 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:07.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:06 smithi031 ceph-mon[31673]: osdmap e84: 12 total, 12 up, 12 in 2024-03-21T10:35:07.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:06 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:07.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:06 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/208083896' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-21T10:35:07.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:06 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-21T10:35:07.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:06 smithi070 ceph-mon[36330]: pgmap v318: 33 pgs: 7 creating+peering, 1 active+clean, 25 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:35:07.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:06 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-21T10:35:07.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:06 smithi070 ceph-mon[36330]: osdmap e83: 12 total, 12 up, 12 in 2024-03-21T10:35:07.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:06 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/4154206699' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-21T10:35:07.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:06 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-21T10:35:07.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:06 smithi070 ceph-mon[36330]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-21T10:35:07.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:06 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:07.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:06 smithi070 ceph-mon[36330]: osdmap e84: 12 total, 12 up, 12 in 2024-03-21T10:35:07.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:06 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:07.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:06 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/208083896' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-21T10:35:07.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:06 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-21T10:35:07.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:06 smithi040 ceph-mon[36154]: pgmap v318: 33 pgs: 7 creating+peering, 1 active+clean, 25 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:35:07.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:06 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-21T10:35:07.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:06 smithi040 ceph-mon[36154]: osdmap e83: 12 total, 12 up, 12 in 2024-03-21T10:35:07.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:06 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/4154206699' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-21T10:35:07.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:06 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-21T10:35:07.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:06 smithi040 ceph-mon[36154]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-21T10:35:07.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:06 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:07.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:06 smithi040 ceph-mon[36154]: osdmap e84: 12 total, 12 up, 12 in 2024-03-21T10:35:07.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:06 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:07.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.462+0000 7f7154600640 1 -- 172.21.15.31:0/1730872613 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f714c1a23b0 con 0x7f714c1028c0 2024-03-21T10:35:07.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.891+0000 7f714a000640 1 -- 172.21.15.31:0/1730872613 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) v1 ==== 157+0+0 (secure 0 0 0) 0x7f71380c17f0 con 0x7f714c1028c0 2024-03-21T10:35:07.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.896+0000 7f7154600640 1 -- 172.21.15.31:0/1730872613 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f71340a2fa0 msgr2=0x7f71340a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:07.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.896+0000 7f7154600640 1 --2- 172.21.15.31:0/1730872613 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f71340a2fa0 0x7f71340a5460 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f714c19b100 tx=0x7f7140005f90 comp rx=0 tx=0).stop 2024-03-21T10:35:07.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.896+0000 7f7154600640 1 -- 172.21.15.31:0/1730872613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f714c1028c0 msgr2=0x7f714c19a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:07.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.896+0000 7f7154600640 1 --2- 172.21.15.31:0/1730872613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f714c1028c0 0x7f714c19a170 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7138009230 tx=0x7f713802c650 comp rx=0 tx=0).stop 2024-03-21T10:35:07.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.896+0000 7f7154600640 1 -- 172.21.15.31:0/1730872613 shutdown_connections 2024-03-21T10:35:07.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.896+0000 7f7154600640 1 --2- 172.21.15.31:0/1730872613 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f71340a2fa0 0x7f71340a5460 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:07.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.896+0000 7f7154600640 1 --2- 172.21.15.31:0/1730872613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f714c103280 0x7f714c1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:07.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.896+0000 7f7154600640 1 --2- 172.21.15.31:0/1730872613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f714c1028c0 0x7f714c19a170 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:07.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.896+0000 7f7154600640 1 --2- 172.21.15.31:0/1730872613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f714c1016d0 0x7f714c199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:07.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.896+0000 7f7154600640 1 -- 172.21.15.31:0/1730872613 >> 172.21.15.31:0/1730872613 conn(0x7f714c0fcc60 msgr2=0x7f714c0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:07.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.896+0000 7f7154600640 1 -- 172.21.15.31:0/1730872613 shutdown_connections 2024-03-21T10:35:07.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:07.896+0000 7f7154600640 1 -- 172.21.15.31:0/1730872613 wait complete. 2024-03-21T10:35:07.915 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-21T10:35:08.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.140+0000 7f5793200640 1 -- 172.21.15.31:0/1940192522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57941071a0 msgr2=0x7f57941095f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:08.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.140+0000 7f5793200640 1 --2- 172.21.15.31:0/1940192522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57941071a0 0x7f57941095f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f578c009900 tx=0x7f578c02bc40 comp rx=0 tx=0).stop 2024-03-21T10:35:08.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.141+0000 7f5793200640 1 -- 172.21.15.31:0/1940192522 shutdown_connections 2024-03-21T10:35:08.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.141+0000 7f5793200640 1 --2- 172.21.15.31:0/1940192522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57941071a0 0x7f57941095f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:08.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.141+0000 7f5793200640 1 --2- 172.21.15.31:0/1940192522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5794104870 0x7f5794106c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:08.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.141+0000 7f5793200640 1 --2- 172.21.15.31:0/1940192522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5794101f40 0x7f5794104330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:08.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.141+0000 7f5793200640 1 -- 172.21.15.31:0/1940192522 >> 172.21.15.31:0/1940192522 conn(0x7f57940fb930 msgr2=0x7f57940fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:08.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.141+0000 7f5793200640 1 -- 172.21.15.31:0/1940192522 shutdown_connections 2024-03-21T10:35:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.141+0000 7f5793200640 1 -- 172.21.15.31:0/1940192522 wait complete. 2024-03-21T10:35:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.142+0000 7f5793200640 1 Processor -- start 2024-03-21T10:35:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.142+0000 7f5793200640 1 -- start start 2024-03-21T10:35:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.142+0000 7f5793200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5794101f40 0x7f5794199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.142+0000 7f5793200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5794104870 0x7f5794199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.142+0000 7f5793200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57941071a0 0x7f57941a0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.142+0000 7f5793200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579410d150 con 0x7f57941071a0 2024-03-21T10:35:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.142+0000 7f5793200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579410cfd0 con 0x7f5794101f40 2024-03-21T10:35:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.142+0000 7f5793200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579410d2d0 con 0x7f5794104870 2024-03-21T10:35:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.143+0000 7f5791e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5794101f40 0x7f5794199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.143+0000 7f5791e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5794101f40 0x7f5794199940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:36948/0 (socket says 172.21.15.31:36948) 2024-03-21T10:35:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.143+0000 7f5791e00640 1 -- 172.21.15.31:0/2566228896 learned_addr learned my addr 172.21.15.31:0/2566228896 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.143+0000 7f5791400640 1 --2- 172.21.15.31:0/2566228896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5794104870 0x7f5794199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.143+0000 7f5792800640 1 --2- 172.21.15.31:0/2566228896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57941071a0 0x7f57941a0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:08.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.143+0000 7f5791e00640 1 -- 172.21.15.31:0/2566228896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5794104870 msgr2=0x7f5794199e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:08.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.143+0000 7f5791e00640 1 --2- 172.21.15.31:0/2566228896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5794104870 0x7f5794199e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:08.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.143+0000 7f5791e00640 1 -- 172.21.15.31:0/2566228896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57941071a0 msgr2=0x7f57941a0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:08.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.143+0000 7f5791e00640 1 --2- 172.21.15.31:0/2566228896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57941071a0 0x7f57941a0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:08.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.143+0000 7f5791e00640 1 -- 172.21.15.31:0/2566228896 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57941a1560 con 0x7f5794101f40 2024-03-21T10:35:08.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.143+0000 7f5791e00640 1 --2- 172.21.15.31:0/2566228896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5794101f40 0x7f5794199940 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f578400db80 tx=0x7f578400b630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:08.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.144+0000 7f578aa00640 1 -- 172.21.15.31:0/2566228896 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5784017070 con 0x7f5794101f40 2024-03-21T10:35:08.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.144+0000 7f578aa00640 1 -- 172.21.15.31:0/2566228896 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5784003b70 con 0x7f5794101f40 2024-03-21T10:35:08.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.144+0000 7f578aa00640 1 -- 172.21.15.31:0/2566228896 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57840106c0 con 0x7f5794101f40 2024-03-21T10:35:08.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.144+0000 7f5793200640 1 -- 172.21.15.31:0/2566228896 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57941a1850 con 0x7f5794101f40 2024-03-21T10:35:08.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.144+0000 7f5793200640 1 -- 172.21.15.31:0/2566228896 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57941a1d50 con 0x7f5794101f40 2024-03-21T10:35:08.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.145+0000 7f5793200640 1 -- 172.21.15.31:0/2566228896 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5794108c90 con 0x7f5794101f40 2024-03-21T10:35:08.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.147+0000 7f578aa00640 1 -- 172.21.15.31:0/2566228896 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f5784004020 con 0x7f5794101f40 2024-03-21T10:35:08.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.148+0000 7f578aa00640 1 --2- 172.21.15.31:0/2566228896 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f57780a2f90 0x7f57780a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:08.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.148+0000 7f578aa00640 1 -- 172.21.15.31:0/2566228896 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f57840cf1a0 con 0x7f5794101f40 2024-03-21T10:35:08.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.148+0000 7f5791400640 1 --2- 172.21.15.31:0/2566228896 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f57780a2f90 0x7f57780a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:08.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.149+0000 7f5791400640 1 --2- 172.21.15.31:0/2566228896 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f57780a2f90 0x7f57780a5450 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f577c0095f0 tx=0x7f577c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:08.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.152+0000 7f578aa00640 1 -- 172.21.15.31:0/2566228896 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5784020070 con 0x7f5794101f40 2024-03-21T10:35:08.245 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:07 smithi070 ceph-mon[36330]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T10:35:08.245 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:07 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:08.245 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:07 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-21T10:35:08.245 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:07 smithi070 ceph-mon[36330]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-21T10:35:08.245 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:07 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:08.245 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:07 smithi070 ceph-mon[36330]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-21T10:35:08.245 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:07 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1730872613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T10:35:08.245 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:07 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T10:35:08.245 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:07 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-21T10:35:08.246 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:07 smithi070 ceph-mon[36330]: osdmap e85: 12 total, 12 up, 12 in 2024-03-21T10:35:08.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:07 smithi031 ceph-mon[31673]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T10:35:08.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:07 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:08.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:07 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-21T10:35:08.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:07 smithi031 ceph-mon[31673]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-21T10:35:08.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:07 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:08.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:07 smithi031 ceph-mon[31673]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-21T10:35:08.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:07 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1730872613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T10:35:08.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:07 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T10:35:08.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:07 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-21T10:35:08.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:07 smithi031 ceph-mon[31673]: osdmap e85: 12 total, 12 up, 12 in 2024-03-21T10:35:08.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:08.359+0000 7f5793200640 1 -- 172.21.15.31:0/2566228896 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f579402d070 con 0x7f5794101f40 2024-03-21T10:35:08.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:07 smithi040 ceph-mon[36154]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T10:35:08.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:07 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:08.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:07 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-21T10:35:08.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:07 smithi040 ceph-mon[36154]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-21T10:35:08.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:07 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:08.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:07 smithi040 ceph-mon[36154]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-21T10:35:08.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:07 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1730872613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T10:35:08.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:07 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-21T10:35:08.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:07 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-21T10:35:08.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:07 smithi040 ceph-mon[36154]: osdmap e85: 12 total, 12 up, 12 in 2024-03-21T10:35:09.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:09 smithi031 ceph-mon[31673]: pgmap v322: 97 pgs: 15 creating+peering, 24 active+clean, 58 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 4 op/s 2024-03-21T10:35:09.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:09 smithi031 ceph-mon[31673]: mds.? [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] up:active 2024-03-21T10:35:09.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:09 smithi031 ceph-mon[31673]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-21T10:35:09.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:09 smithi031 ceph-mon[31673]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-21T10:35:09.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:09.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:09.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:35:09.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:35:09.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:09 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:09.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:09 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2566228896' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-21T10:35:09.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:09 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-21T10:35:09.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:09 smithi031 ceph-mon[31673]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T10:35:09.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:09 smithi031 ceph-mon[31673]: Cluster is now healthy 2024-03-21T10:35:09.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:09 smithi070 ceph-mon[36330]: pgmap v322: 97 pgs: 15 creating+peering, 24 active+clean, 58 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 4 op/s 2024-03-21T10:35:09.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:09 smithi070 ceph-mon[36330]: mds.? [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] up:active 2024-03-21T10:35:09.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:09 smithi070 ceph-mon[36330]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-21T10:35:09.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:09 smithi070 ceph-mon[36330]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-21T10:35:09.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:09.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:09.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:35:09.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:35:09.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:09 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:09.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:09 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2566228896' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-21T10:35:09.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:09 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-21T10:35:09.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:09 smithi070 ceph-mon[36330]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T10:35:09.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:09 smithi070 ceph-mon[36330]: Cluster is now healthy 2024-03-21T10:35:09.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:09 smithi040 ceph-mon[36154]: pgmap v322: 97 pgs: 15 creating+peering, 24 active+clean, 58 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 4 op/s 2024-03-21T10:35:09.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:09 smithi040 ceph-mon[36154]: mds.? [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] up:active 2024-03-21T10:35:09.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:09 smithi040 ceph-mon[36154]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-21T10:35:09.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:09 smithi040 ceph-mon[36154]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-21T10:35:09.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:09.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:09.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:35:09.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:35:09.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:09 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:35:09.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:09 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2566228896' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-21T10:35:09.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:09 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-21T10:35:09.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:09 smithi040 ceph-mon[36154]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T10:35:09.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:09 smithi040 ceph-mon[36154]: Cluster is now healthy 2024-03-21T10:35:10.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.153+0000 7f578aa00640 1 -- 172.21.15.31:0/2566228896 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) v1 ==== 228+0+0 (secure 0 0 0) 0x7f579402d070 con 0x7f5794101f40 2024-03-21T10:35:10.152 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data_ec' created 2024-03-21T10:35:10.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.160+0000 7f5793200640 1 -- 172.21.15.31:0/2566228896 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f57780a2f90 msgr2=0x7f57780a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:10.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.160+0000 7f5793200640 1 --2- 172.21.15.31:0/2566228896 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f57780a2f90 0x7f57780a5450 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f577c0095f0 tx=0x7f577c005f90 comp rx=0 tx=0).stop 2024-03-21T10:35:10.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.160+0000 7f5793200640 1 -- 172.21.15.31:0/2566228896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5794101f40 msgr2=0x7f5794199940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:10.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.160+0000 7f5793200640 1 --2- 172.21.15.31:0/2566228896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5794101f40 0x7f5794199940 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f578400db80 tx=0x7f578400b630 comp rx=0 tx=0).stop 2024-03-21T10:35:10.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.160+0000 7f5793200640 1 -- 172.21.15.31:0/2566228896 shutdown_connections 2024-03-21T10:35:10.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.160+0000 7f5793200640 1 --2- 172.21.15.31:0/2566228896 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f57780a2f90 0x7f57780a5450 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:10.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.160+0000 7f5793200640 1 --2- 172.21.15.31:0/2566228896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57941071a0 0x7f57941a0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:10.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.160+0000 7f5793200640 1 --2- 172.21.15.31:0/2566228896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5794104870 0x7f5794199e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:10.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.160+0000 7f5793200640 1 --2- 172.21.15.31:0/2566228896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5794101f40 0x7f5794199940 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:10.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.160+0000 7f5793200640 1 -- 172.21.15.31:0/2566228896 >> 172.21.15.31:0/2566228896 conn(0x7f57940fb930 msgr2=0x7f5794101600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:10.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.161+0000 7f5793200640 1 -- 172.21.15.31:0/2566228896 shutdown_connections 2024-03-21T10:35:10.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.161+0000 7f5793200640 1 -- 172.21.15.31:0/2566228896 wait complete. 2024-03-21T10:35:10.185 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-21T10:35:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.381+0000 7fa8b9c00640 1 -- 172.21.15.31:0/790362702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8b4107250 msgr2=0x7fa8b4109640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.381+0000 7fa8b9c00640 1 --2- 172.21.15.31:0/790362702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8b4107250 0x7fa8b4109640 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac009ca0 tx=0x7fa8ac02c100 comp rx=0 tx=0).stop 2024-03-21T10:35:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.381+0000 7fa8b9c00640 1 -- 172.21.15.31:0/790362702 shutdown_connections 2024-03-21T10:35:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.381+0000 7fa8b9c00640 1 --2- 172.21.15.31:0/790362702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8b4107250 0x7fa8b4109640 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.381+0000 7fa8b9c00640 1 --2- 172.21.15.31:0/790362702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8b4104920 0x7fa8b4106d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.381+0000 7fa8b9c00640 1 --2- 172.21.15.31:0/790362702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8b4101ff0 0x7fa8b41043e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.381+0000 7fa8b9c00640 1 -- 172.21.15.31:0/790362702 >> 172.21.15.31:0/790362702 conn(0x7fa8b40fb980 msgr2=0x7fa8b40fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.381+0000 7fa8b9c00640 1 -- 172.21.15.31:0/790362702 shutdown_connections 2024-03-21T10:35:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.381+0000 7fa8b9c00640 1 -- 172.21.15.31:0/790362702 wait complete. 2024-03-21T10:35:10.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.383+0000 7fa8b9c00640 1 Processor -- start 2024-03-21T10:35:10.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.383+0000 7fa8b9c00640 1 -- start start 2024-03-21T10:35:10.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.383+0000 7fa8b9c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8b4101ff0 0x7fa8b4101960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:10.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.383+0000 7fa8b9c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8b4104920 0x7fa8b40fffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:10.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.383+0000 7fa8b9c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8b4107250 0x7fa8b41004e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:10.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.383+0000 7fa8b9c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b410d180 con 0x7fa8b4104920 2024-03-21T10:35:10.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.383+0000 7fa8b9c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b410d000 con 0x7fa8b4101ff0 2024-03-21T10:35:10.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.383+0000 7fa8b9c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b410d300 con 0x7fa8b4107250 2024-03-21T10:35:10.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.384+0000 7fa8b3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8b4104920 0x7fa8b40fffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:10.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.384+0000 7fa8b3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8b4101ff0 0x7fa8b4101960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:10.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.384+0000 7fa8b3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8b4104920 0x7fa8b40fffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59182/0 (socket says 172.21.15.31:59182) 2024-03-21T10:35:10.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.384+0000 7fa8b3400640 1 -- 172.21.15.31:0/1255067593 learned_addr learned my addr 172.21.15.31:0/1255067593 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:10.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.384+0000 7fa8b9200640 1 --2- 172.21.15.31:0/1255067593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8b4107250 0x7fa8b41004e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:10.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.384+0000 7fa8b3e00640 1 -- 172.21.15.31:0/1255067593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8b4107250 msgr2=0x7fa8b41004e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:10.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.384+0000 7fa8b3e00640 1 --2- 172.21.15.31:0/1255067593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8b4107250 0x7fa8b41004e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:10.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.384+0000 7fa8b3e00640 1 -- 172.21.15.31:0/1255067593 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8b4104920 msgr2=0x7fa8b40fffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:10.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.384+0000 7fa8b3e00640 1 --2- 172.21.15.31:0/1255067593 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8b4104920 0x7fa8b40fffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:10.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.384+0000 7fa8b3e00640 1 -- 172.21.15.31:0/1255067593 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b4100d70 con 0x7fa8b4101ff0 2024-03-21T10:35:10.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.384+0000 7fa8b3e00640 1 --2- 172.21.15.31:0/1255067593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8b4101ff0 0x7fa8b4101960 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa8a400c970 tx=0x7fa8a400ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:10.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.385+0000 7fa8b0c00640 1 -- 172.21.15.31:0/1255067593 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a4007bb0 con 0x7fa8b4101ff0 2024-03-21T10:35:10.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.385+0000 7fa8b0c00640 1 -- 172.21.15.31:0/1255067593 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8a4007d50 con 0x7fa8b4101ff0 2024-03-21T10:35:10.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.385+0000 7fa8b0c00640 1 -- 172.21.15.31:0/1255067593 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a400f450 con 0x7fa8b4101ff0 2024-03-21T10:35:10.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.385+0000 7fa8b9c00640 1 -- 172.21.15.31:0/1255067593 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b41a18e0 con 0x7fa8b4101ff0 2024-03-21T10:35:10.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.385+0000 7fa8b9c00640 1 -- 172.21.15.31:0/1255067593 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b41a1e80 con 0x7fa8b4101ff0 2024-03-21T10:35:10.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.386+0000 7fa8b9c00640 1 -- 172.21.15.31:0/1255067593 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b40fd470 con 0x7fa8b4101ff0 2024-03-21T10:35:10.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.388+0000 7fa8b0c00640 1 -- 172.21.15.31:0/1255067593 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fa8a4004ef0 con 0x7fa8b4101ff0 2024-03-21T10:35:10.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.389+0000 7fa8b0c00640 1 --2- 172.21.15.31:0/1255067593 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa8980a2fa0 0x7fa8980a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:10.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.389+0000 7fa8b0c00640 1 -- 172.21.15.31:0/1255067593 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8770+0+0 (secure 0 0 0) 0x7fa8a40cf230 con 0x7fa8b4101ff0 2024-03-21T10:35:10.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.389+0000 7fa8b3400640 1 --2- 172.21.15.31:0/1255067593 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa8980a2fa0 0x7fa8980a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:10.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.390+0000 7fa8b3400640 1 --2- 172.21.15.31:0/1255067593 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa8980a2fa0 0x7fa8980a5460 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa8b41016d0 tx=0x7fa8a0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:10.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.393+0000 7fa8b0c00640 1 -- 172.21.15.31:0/1255067593 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa8a4014030 con 0x7fa8b4101ff0 2024-03-21T10:35:10.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:10 smithi040 ceph-mon[36154]: osdmap e86: 12 total, 12 up, 12 in 2024-03-21T10:35:10.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:10 smithi031 ceph-mon[31673]: osdmap e86: 12 total, 12 up, 12 in 2024-03-21T10:35:10.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:10 smithi070 ceph-mon[36330]: osdmap e86: 12 total, 12 up, 12 in 2024-03-21T10:35:10.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:10.595+0000 7fa8b9c00640 1 -- 172.21.15.31:0/1255067593 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7fa8b402d070 con 0x7fa8b4101ff0 2024-03-21T10:35:11.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.164+0000 7fa8b0c00640 1 -- 172.21.15.31:0/1255067593 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa8a409b790 con 0x7fa8b4101ff0 2024-03-21T10:35:11.163 INFO:teuthology.orchestra.run.smithi031.stderr:set pool 4 allow_ec_overwrites to true 2024-03-21T10:35:11.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.169+0000 7fa8b9c00640 1 -- 172.21.15.31:0/1255067593 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa8980a2fa0 msgr2=0x7fa8980a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:11.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.169+0000 7fa8b9c00640 1 --2- 172.21.15.31:0/1255067593 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa8980a2fa0 0x7fa8980a5460 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa8b41016d0 tx=0x7fa8a0005f90 comp rx=0 tx=0).stop 2024-03-21T10:35:11.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.169+0000 7fa8b9c00640 1 -- 172.21.15.31:0/1255067593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8b4101ff0 msgr2=0x7fa8b4101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:11.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.169+0000 7fa8b9c00640 1 --2- 172.21.15.31:0/1255067593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8b4101ff0 0x7fa8b4101960 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa8a400c970 tx=0x7fa8a400ce30 comp rx=0 tx=0).stop 2024-03-21T10:35:11.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.171+0000 7fa8b9c00640 1 -- 172.21.15.31:0/1255067593 shutdown_connections 2024-03-21T10:35:11.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.171+0000 7fa8b9c00640 1 --2- 172.21.15.31:0/1255067593 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa8980a2fa0 0x7fa8980a5460 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:11.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.171+0000 7fa8b9c00640 1 --2- 172.21.15.31:0/1255067593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8b4107250 0x7fa8b41004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:11.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.171+0000 7fa8b9c00640 1 --2- 172.21.15.31:0/1255067593 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8b4104920 0x7fa8b40fffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:11.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.171+0000 7fa8b9c00640 1 --2- 172.21.15.31:0/1255067593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8b4101ff0 0x7fa8b4101960 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:11.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.171+0000 7fa8b9c00640 1 -- 172.21.15.31:0/1255067593 >> 172.21.15.31:0/1255067593 conn(0x7fa8b40fb980 msgr2=0x7fa8b4102950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:11.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.172+0000 7fa8b9c00640 1 -- 172.21.15.31:0/1255067593 shutdown_connections 2024-03-21T10:35:11.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.172+0000 7fa8b9c00640 1 -- 172.21.15.31:0/1255067593 wait complete. 2024-03-21T10:35:11.205 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-21T10:35:11.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.398+0000 7f2b25600640 1 -- 172.21.15.31:0/3827911506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b200756e0 msgr2=0x7f2b20073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:11.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.398+0000 7f2b25600640 1 --2- 172.21.15.31:0/3827911506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b200756e0 0x7f2b20073b30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2b10009930 tx=0x7f2b1002bc30 comp rx=0 tx=0).stop 2024-03-21T10:35:11.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.398+0000 7f2b25600640 1 -- 172.21.15.31:0/3827911506 shutdown_connections 2024-03-21T10:35:11.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.398+0000 7f2b25600640 1 --2- 172.21.15.31:0/3827911506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b2010dc80 0x7f2b20110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:11.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.398+0000 7f2b25600640 1 --2- 172.21.15.31:0/3827911506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b20074100 0x7f2b20074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:11.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.398+0000 7f2b25600640 1 --2- 172.21.15.31:0/3827911506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b200756e0 0x7f2b20073b30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:11.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.398+0000 7f2b25600640 1 -- 172.21.15.31:0/3827911506 >> 172.21.15.31:0/3827911506 conn(0x7f2b200fcc20 msgr2=0x7f2b200ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:11.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.398+0000 7f2b25600640 1 -- 172.21.15.31:0/3827911506 shutdown_connections 2024-03-21T10:35:11.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.398+0000 7f2b25600640 1 -- 172.21.15.31:0/3827911506 wait complete. 2024-03-21T10:35:11.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.399+0000 7f2b25600640 1 Processor -- start 2024-03-21T10:35:11.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.399+0000 7f2b25600640 1 -- start start 2024-03-21T10:35:11.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.399+0000 7f2b25600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b20074100 0x7f2b2019e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:11.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.399+0000 7f2b25600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b200756e0 0x7f2b2019e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:11.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.399+0000 7f2b25600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b2010dc80 0x7f2b201a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:11.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.399+0000 7f2b25600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b20113900 con 0x7f2b20074100 2024-03-21T10:35:11.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.399+0000 7f2b25600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b20113780 con 0x7f2b200756e0 2024-03-21T10:35:11.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.399+0000 7f2b25600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b20113a80 con 0x7f2b2010dc80 2024-03-21T10:35:11.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.400+0000 7f2b16000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b200756e0 0x7f2b2019e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:11.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.400+0000 7f2b1ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b20074100 0x7f2b2019e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:11.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.400+0000 7f2b16000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b200756e0 0x7f2b2019e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:36980/0 (socket says 172.21.15.31:36980) 2024-03-21T10:35:11.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.400+0000 7f2b1f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b2010dc80 0x7f2b201a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:11.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.400+0000 7f2b16000640 1 -- 172.21.15.31:0/1150981691 learned_addr learned my addr 172.21.15.31:0/1150981691 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:11.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.400+0000 7f2b16000640 1 -- 172.21.15.31:0/1150981691 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b2010dc80 msgr2=0x7f2b201a55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:11.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.400+0000 7f2b16000640 1 --2- 172.21.15.31:0/1150981691 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b2010dc80 0x7f2b201a55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:11.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.400+0000 7f2b16000640 1 -- 172.21.15.31:0/1150981691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b20074100 msgr2=0x7f2b2019e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:11.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.400+0000 7f2b16000640 1 --2- 172.21.15.31:0/1150981691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b20074100 0x7f2b2019e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:11.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.400+0000 7f2b16000640 1 -- 172.21.15.31:0/1150981691 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b201a5c30 con 0x7f2b200756e0 2024-03-21T10:35:11.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.400+0000 7f2b16000640 1 --2- 172.21.15.31:0/1150981691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b200756e0 0x7f2b2019e550 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2b08009d50 tx=0x7f2b0800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:11.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.401+0000 7f2b17e00640 1 -- 172.21.15.31:0/1150981691 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b08003b70 con 0x7f2b200756e0 2024-03-21T10:35:11.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.401+0000 7f2b17e00640 1 -- 172.21.15.31:0/1150981691 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2b08003d10 con 0x7f2b200756e0 2024-03-21T10:35:11.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.401+0000 7f2b17e00640 1 -- 172.21.15.31:0/1150981691 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b080108f0 con 0x7f2b200756e0 2024-03-21T10:35:11.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.401+0000 7f2b25600640 1 -- 172.21.15.31:0/1150981691 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b201a5f20 con 0x7f2b200756e0 2024-03-21T10:35:11.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.401+0000 7f2b25600640 1 -- 172.21.15.31:0/1150981691 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b201a6390 con 0x7f2b200756e0 2024-03-21T10:35:11.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.402+0000 7f2b25600640 1 -- 172.21.15.31:0/1150981691 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b20103e50 con 0x7f2b200756e0 2024-03-21T10:35:11.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.404+0000 7f2b17e00640 1 -- 172.21.15.31:0/1150981691 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f2b08004000 con 0x7f2b200756e0 2024-03-21T10:35:11.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.405+0000 7f2b17e00640 1 --2- 172.21.15.31:0/1150981691 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2b040a2fa0 0x7f2b040a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:11.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.405+0000 7f2b17e00640 1 -- 172.21.15.31:0/1150981691 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8770+0+0 (secure 0 0 0) 0x7f2b080cfbe0 con 0x7f2b200756e0 2024-03-21T10:35:11.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.406+0000 7f2b1ea00640 1 --2- 172.21.15.31:0/1150981691 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2b040a2fa0 0x7f2b040a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:11.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.406+0000 7f2b1ea00640 1 --2- 172.21.15.31:0/1150981691 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2b040a2fa0 0x7f2b040a5460 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2b10000c00 tx=0x7f2b10005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:11.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.409+0000 7f2b17e00640 1 -- 172.21.15.31:0/1150981691 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2b0809c790 con 0x7f2b200756e0 2024-03-21T10:35:11.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:11 smithi040 ceph-mon[36154]: pgmap v324: 97 pgs: 15 creating+peering, 70 active+clean, 12 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-03-21T10:35:11.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:11 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-21T10:35:11.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:11 smithi040 ceph-mon[36154]: osdmap e87: 12 total, 12 up, 12 in 2024-03-21T10:35:11.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:11 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1255067593' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-21T10:35:11.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:11 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-21T10:35:11.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:11 smithi031 ceph-mon[31673]: pgmap v324: 97 pgs: 15 creating+peering, 70 active+clean, 12 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-03-21T10:35:11.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:11 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-21T10:35:11.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:11 smithi031 ceph-mon[31673]: osdmap e87: 12 total, 12 up, 12 in 2024-03-21T10:35:11.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:11 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1255067593' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-21T10:35:11.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:11 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-21T10:35:11.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:11 smithi070 ceph-mon[36330]: pgmap v324: 97 pgs: 15 creating+peering, 70 active+clean, 12 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-03-21T10:35:11.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:11 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-21T10:35:11.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:11 smithi070 ceph-mon[36330]: osdmap e87: 12 total, 12 up, 12 in 2024-03-21T10:35:11.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:11 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1255067593' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-21T10:35:11.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:11 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-21T10:35:11.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:11.654+0000 7f2b25600640 1 -- 172.21.15.31:0/1150981691 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f2b20062340 con 0x7f2b200756e0 2024-03-21T10:35:11.931 INFO:journalctl@ceph.mds.b.smithi040.stdout:Mar 21 10:35:11 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-b[66460]: 2024-03-21T10:35:11.437+0000 7fe6c5734700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-21T10:35:12.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.179+0000 7f2b17e00640 1 -- 172.21.15.31:0/1150981691 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7f2b0809c090 con 0x7f2b200756e0 2024-03-21T10:35:12.178 INFO:teuthology.orchestra.run.smithi031.stderr:added data pool 4 to fsmap 2024-03-21T10:35:12.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.185+0000 7f2b25600640 1 -- 172.21.15.31:0/1150981691 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2b040a2fa0 msgr2=0x7f2b040a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:12.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.185+0000 7f2b25600640 1 --2- 172.21.15.31:0/1150981691 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2b040a2fa0 0x7f2b040a5460 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2b10000c00 tx=0x7f2b10005fd0 comp rx=0 tx=0).stop 2024-03-21T10:35:12.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.185+0000 7f2b25600640 1 -- 172.21.15.31:0/1150981691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b200756e0 msgr2=0x7f2b2019e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:12.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.185+0000 7f2b25600640 1 --2- 172.21.15.31:0/1150981691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b200756e0 0x7f2b2019e550 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2b08009d50 tx=0x7f2b0800c6a0 comp rx=0 tx=0).stop 2024-03-21T10:35:12.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.185+0000 7f2b25600640 1 -- 172.21.15.31:0/1150981691 shutdown_connections 2024-03-21T10:35:12.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.185+0000 7f2b25600640 1 --2- 172.21.15.31:0/1150981691 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2b040a2fa0 0x7f2b040a5460 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:12.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.185+0000 7f2b25600640 1 --2- 172.21.15.31:0/1150981691 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b2010dc80 0x7f2b201a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:12.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.185+0000 7f2b25600640 1 --2- 172.21.15.31:0/1150981691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b200756e0 0x7f2b2019e550 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:12.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.185+0000 7f2b25600640 1 --2- 172.21.15.31:0/1150981691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b20074100 0x7f2b2019e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:12.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.185+0000 7f2b25600640 1 -- 172.21.15.31:0/1150981691 >> 172.21.15.31:0/1150981691 conn(0x7f2b200fcc20 msgr2=0x7f2b20101020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:12.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.185+0000 7f2b25600640 1 -- 172.21.15.31:0/1150981691 shutdown_connections 2024-03-21T10:35:12.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.185+0000 7f2b25600640 1 -- 172.21.15.31:0/1150981691 wait complete. 2024-03-21T10:35:12.203 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:35:12.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.410+0000 7f4623200640 1 -- 172.21.15.31:0/2363264642 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4624108db0 msgr2=0x7f4624109230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:12.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.410+0000 7f4623200640 1 --2- 172.21.15.31:0/2363264642 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4624108db0 0x7f4624109230 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f460c009ca0 tx=0x7f460c02c100 comp rx=0 tx=0).stop 2024-03-21T10:35:12.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.410+0000 7f4623200640 1 -- 172.21.15.31:0/2363264642 shutdown_connections 2024-03-21T10:35:12.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.410+0000 7f4623200640 1 --2- 172.21.15.31:0/2363264642 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4624109770 0x7f4624110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:12.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.410+0000 7f4623200640 1 --2- 172.21.15.31:0/2363264642 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4624108db0 0x7f4624109230 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:12.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.410+0000 7f4623200640 1 --2- 172.21.15.31:0/2363264642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4624107bc0 0x7f4624107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:12.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.410+0000 7f4623200640 1 -- 172.21.15.31:0/2363264642 >> 172.21.15.31:0/2363264642 conn(0x7f4624076b60 msgr2=0x7f4624078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:12.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.410+0000 7f4623200640 1 -- 172.21.15.31:0/2363264642 shutdown_connections 2024-03-21T10:35:12.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.410+0000 7f4623200640 1 -- 172.21.15.31:0/2363264642 wait complete. 2024-03-21T10:35:12.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.411+0000 7f4623200640 1 Processor -- start 2024-03-21T10:35:12.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.412+0000 7f4623200640 1 -- start start 2024-03-21T10:35:12.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.412+0000 7f4623200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4624107bc0 0x7f462419de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:12.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.412+0000 7f4623200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4624108db0 0x7f462419e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:12.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.412+0000 7f4623200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4624109770 0x7f46241a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:12.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.412+0000 7f4623200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4624113820 con 0x7f4624107bc0 2024-03-21T10:35:12.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.412+0000 7f4623200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46241136a0 con 0x7f4624109770 2024-03-21T10:35:12.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.412+0000 7f4623200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46241139a0 con 0x7f4624108db0 2024-03-21T10:35:12.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.412+0000 7f4621e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4624107bc0 0x7f462419de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:12.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.412+0000 7f4621e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4624107bc0 0x7f462419de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59200/0 (socket says 172.21.15.31:59200) 2024-03-21T10:35:12.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.412+0000 7f4621e00640 1 -- 172.21.15.31:0/2064381619 learned_addr learned my addr 172.21.15.31:0/2064381619 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:12.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.413+0000 7f4622800640 1 --2- 172.21.15.31:0/2064381619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4624109770 0x7f46241a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:12.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.413+0000 7f4619400640 1 --2- 172.21.15.31:0/2064381619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4624108db0 0x7f462419e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:12.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.413+0000 7f4621e00640 1 -- 172.21.15.31:0/2064381619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4624108db0 msgr2=0x7f462419e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:12.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.413+0000 7f4621e00640 1 --2- 172.21.15.31:0/2064381619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4624108db0 0x7f462419e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:12.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.413+0000 7f4621e00640 1 -- 172.21.15.31:0/2064381619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4624109770 msgr2=0x7f46241a53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:12.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.413+0000 7f4621e00640 1 --2- 172.21.15.31:0/2064381619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4624109770 0x7f46241a53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:12.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.413+0000 7f4621e00640 1 -- 172.21.15.31:0/2064381619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46241a5a60 con 0x7f4624107bc0 2024-03-21T10:35:12.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.413+0000 7f4621e00640 1 --2- 172.21.15.31:0/2064381619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4624107bc0 0x7f462419de40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f461400d770 tx=0x7f461400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:12.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.413+0000 7f461b400640 1 -- 172.21.15.31:0/2064381619 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4614017070 con 0x7f4624107bc0 2024-03-21T10:35:12.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.413+0000 7f461b400640 1 -- 172.21.15.31:0/2064381619 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4614009c30 con 0x7f4624107bc0 2024-03-21T10:35:12.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.413+0000 7f4623200640 1 -- 172.21.15.31:0/2064381619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46241a5d50 con 0x7f4624107bc0 2024-03-21T10:35:12.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.413+0000 7f461b400640 1 -- 172.21.15.31:0/2064381619 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4614003dd0 con 0x7f4624107bc0 2024-03-21T10:35:12.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.414+0000 7f4623200640 1 -- 172.21.15.31:0/2064381619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46241a61c0 con 0x7f4624107bc0 2024-03-21T10:35:12.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.415+0000 7f4623200640 1 -- 172.21.15.31:0/2064381619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f462410eee0 con 0x7f4624107bc0 2024-03-21T10:35:12.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.418+0000 7f461b400640 1 -- 172.21.15.31:0/2064381619 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f4614004000 con 0x7f4624107bc0 2024-03-21T10:35:12.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.419+0000 7f461b400640 1 --2- 172.21.15.31:0/2064381619 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f46080a2f10 0x7f46080a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:12.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.419+0000 7f461b400640 1 -- 172.21.15.31:0/2064381619 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f46140cf450 con 0x7f4624107bc0 2024-03-21T10:35:12.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.419+0000 7f4619400640 1 --2- 172.21.15.31:0/2064381619 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f46080a2f10 0x7f46080a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:12.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.420+0000 7f4619400640 1 --2- 172.21.15.31:0/2064381619 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f46080a2f10 0x7f46080a53d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f460c02c710 tx=0x7f460c009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:12.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.421+0000 7f461b400640 1 -- 172.21.15.31:0/2064381619 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4614009de0 con 0x7f4624107bc0 2024-03-21T10:35:12.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:12 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-21T10:35:12.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:12 smithi040 ceph-mon[36154]: osdmap e88: 12 total, 12 up, 12 in 2024-03-21T10:35:12.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:12 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1150981691' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-21T10:35:12.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:12 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-21T10:35:12.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:12 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-21T10:35:12.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:12 smithi031 ceph-mon[31673]: osdmap e88: 12 total, 12 up, 12 in 2024-03-21T10:35:12.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:12 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1150981691' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-21T10:35:12.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:12 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-21T10:35:12.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:12 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-21T10:35:12.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:12 smithi070 ceph-mon[36330]: osdmap e88: 12 total, 12 up, 12 in 2024-03-21T10:35:12.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:12 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1150981691' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-21T10:35:12.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:12 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-21T10:35:12.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.654+0000 7f4623200640 1 -- 172.21.15.31:0/2064381619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f462402d070 con 0x7f4624107bc0 2024-03-21T10:35:12.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.656+0000 7f461b400640 1 -- 172.21.15.31:0/2064381619 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+11312 (secure 0 0 0) 0x7f461409bfe0 con 0x7f4624107bc0 2024-03-21T10:35:12.654 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:35:12.654 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24539,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24584,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:35:11.657448+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24596},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24596,"qdb_cluster":[24596]},"id":1}]} 2024-03-21T10:35:12.654 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 17 2024-03-21T10:35:12.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.658+0000 7f4623200640 1 -- 172.21.15.31:0/2064381619 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f46080a2f10 msgr2=0x7f46080a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:12.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.658+0000 7f4623200640 1 --2- 172.21.15.31:0/2064381619 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f46080a2f10 0x7f46080a53d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f460c02c710 tx=0x7f460c009920 comp rx=0 tx=0).stop 2024-03-21T10:35:12.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.658+0000 7f4623200640 1 -- 172.21.15.31:0/2064381619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4624107bc0 msgr2=0x7f462419de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:12.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.658+0000 7f4623200640 1 --2- 172.21.15.31:0/2064381619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4624107bc0 0x7f462419de40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f461400d770 tx=0x7f461400dc30 comp rx=0 tx=0).stop 2024-03-21T10:35:12.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.659+0000 7f4623200640 1 -- 172.21.15.31:0/2064381619 shutdown_connections 2024-03-21T10:35:12.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.659+0000 7f4623200640 1 --2- 172.21.15.31:0/2064381619 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f46080a2f10 0x7f46080a53d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:12.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.659+0000 7f4623200640 1 --2- 172.21.15.31:0/2064381619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4624109770 0x7f46241a53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:12.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.659+0000 7f4623200640 1 --2- 172.21.15.31:0/2064381619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4624108db0 0x7f462419e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:12.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.659+0000 7f4623200640 1 --2- 172.21.15.31:0/2064381619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4624107bc0 0x7f462419de40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:12.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.659+0000 7f4623200640 1 -- 172.21.15.31:0/2064381619 >> 172.21.15.31:0/2064381619 conn(0x7f4624076b60 msgr2=0x7f462410c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:12.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.659+0000 7f4623200640 1 -- 172.21.15.31:0/2064381619 shutdown_connections 2024-03-21T10:35:12.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.659+0000 7f4623200640 1 -- 172.21.15.31:0/2064381619 wait complete. 2024-03-21T10:35:12.676 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T10:35:12.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.895+0000 7fba6e400640 1 -- 172.21.15.31:0/2979538500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba68105370 msgr2=0x7fba6810bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:12.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.895+0000 7fba6e400640 1 --2- 172.21.15.31:0/2979538500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba68105370 0x7fba6810bc00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fba60009be0 tx=0x7fba6002bef0 comp rx=0 tx=0).stop 2024-03-21T10:35:12.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.896+0000 7fba6e400640 1 -- 172.21.15.31:0/2979538500 shutdown_connections 2024-03-21T10:35:12.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.896+0000 7fba6e400640 1 --2- 172.21.15.31:0/2979538500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba68105370 0x7fba6810bc00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:12.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.896+0000 7fba6e400640 1 --2- 172.21.15.31:0/2979538500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba681049b0 0x7fba68104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:12.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.896+0000 7fba6e400640 1 --2- 172.21.15.31:0/2979538500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba681037c0 0x7fba68103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:12.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.896+0000 7fba6e400640 1 -- 172.21.15.31:0/2979538500 >> 172.21.15.31:0/2979538500 conn(0x7fba680fed50 msgr2=0x7fba68101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:12.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.896+0000 7fba6e400640 1 -- 172.21.15.31:0/2979538500 shutdown_connections 2024-03-21T10:35:12.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.896+0000 7fba6e400640 1 -- 172.21.15.31:0/2979538500 wait complete. 2024-03-21T10:35:12.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.897+0000 7fba6e400640 1 Processor -- start 2024-03-21T10:35:12.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.897+0000 7fba6e400640 1 -- start start 2024-03-21T10:35:12.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.897+0000 7fba6e400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba681037c0 0x7fba68199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:12.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.897+0000 7fba6e400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba681049b0 0x7fba68199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:12.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.897+0000 7fba6e400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba68105370 0x7fba681a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:12.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.897+0000 7fba6e400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6810f470 con 0x7fba681049b0 2024-03-21T10:35:12.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.897+0000 7fba6e400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6810f2f0 con 0x7fba681037c0 2024-03-21T10:35:12.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.897+0000 7fba6e400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6810f5f0 con 0x7fba68105370 2024-03-21T10:35:12.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.898+0000 7fba6d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba681037c0 0x7fba68199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:12.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.898+0000 7fba6d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba681037c0 0x7fba68199a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37016/0 (socket says 172.21.15.31:37016) 2024-03-21T10:35:12.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.898+0000 7fba6d000640 1 -- 172.21.15.31:0/4285880770 learned_addr learned my addr 172.21.15.31:0/4285880770 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:12.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.898+0000 7fba6da00640 1 --2- 172.21.15.31:0/4285880770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba68105370 0x7fba681a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:12.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.898+0000 7fba67e00640 1 --2- 172.21.15.31:0/4285880770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba681049b0 0x7fba68199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:12.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.898+0000 7fba6d000640 1 -- 172.21.15.31:0/4285880770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba68105370 msgr2=0x7fba681a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:12.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.898+0000 7fba6d000640 1 --2- 172.21.15.31:0/4285880770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba68105370 0x7fba681a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:12.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.898+0000 7fba6d000640 1 -- 172.21.15.31:0/4285880770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba681049b0 msgr2=0x7fba68199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:12.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.898+0000 7fba6d000640 1 --2- 172.21.15.31:0/4285880770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba681049b0 0x7fba68199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:12.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.898+0000 7fba6d000640 1 -- 172.21.15.31:0/4285880770 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba681a1670 con 0x7fba681037c0 2024-03-21T10:35:12.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.898+0000 7fba67e00640 1 --2- 172.21.15.31:0/4285880770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba681049b0 0x7fba68199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:35:12.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.898+0000 7fba6d000640 1 --2- 172.21.15.31:0/4285880770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba681037c0 0x7fba68199a50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fba5400e9a0 tx=0x7fba5400ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:12.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.899+0000 7fba65600640 1 -- 172.21.15.31:0/4285880770 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5400cd80 con 0x7fba681037c0 2024-03-21T10:35:12.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.899+0000 7fba65600640 1 -- 172.21.15.31:0/4285880770 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba54010910 con 0x7fba681037c0 2024-03-21T10:35:12.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.899+0000 7fba6e400640 1 -- 172.21.15.31:0/4285880770 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba681a1960 con 0x7fba681037c0 2024-03-21T10:35:12.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.899+0000 7fba65600640 1 -- 172.21.15.31:0/4285880770 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba54018a30 con 0x7fba681037c0 2024-03-21T10:35:12.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.899+0000 7fba6e400640 1 -- 172.21.15.31:0/4285880770 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba681a1eb0 con 0x7fba681037c0 2024-03-21T10:35:12.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.900+0000 7fba6e400640 1 -- 172.21.15.31:0/4285880770 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba6804f470 con 0x7fba681037c0 2024-03-21T10:35:12.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.902+0000 7fba65600640 1 -- 172.21.15.31:0/4285880770 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fba54010ac0 con 0x7fba681037c0 2024-03-21T10:35:12.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.903+0000 7fba65600640 1 --2- 172.21.15.31:0/4285880770 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fba500a2f10 0x7fba500a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:12.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.903+0000 7fba65600640 1 -- 172.21.15.31:0/4285880770 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fba540cff50 con 0x7fba681037c0 2024-03-21T10:35:12.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.903+0000 7fba67e00640 1 --2- 172.21.15.31:0/4285880770 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fba500a2f10 0x7fba500a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:12.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.904+0000 7fba67e00640 1 --2- 172.21.15.31:0/4285880770 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fba500a2f10 0x7fba500a53d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fba5c0058c0 tx=0x7fba5c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:12.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:12.910+0000 7fba65600640 1 -- 172.21.15.31:0/4285880770 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fba5409cae0 con 0x7fba681037c0 2024-03-21T10:35:13.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.108+0000 7fba6e400640 1 -- 172.21.15.31:0/4285880770 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba680622a0 con 0x7fba681037c0 2024-03-21T10:35:13.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.111+0000 7fba65600640 1 -- 172.21.15.31:0/4285880770 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24009 (secure 0 0 0) 0x7fba5409c3e0 con 0x7fba681037c0 2024-03-21T10:35:13.109 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:35:13.109 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","created":"2024-03-21T10:25:41.971117+0000","modified":"2024-03-21T10:35:11.657050+0000","last_up_change":"2024-03-21T10:34:07.757196+0000","last_in_change":"2024-03-21T10:33:47.460406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T10:29:44.553968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T10:35:04.945449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T10:35:05.970589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T10:35:09.688711+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e0244d33-00e2-47d5-9f66-34efc138c462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6803","nonce":3089885125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6805","nonce":3089885125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6809","nonce":3089885125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6807","nonce":3089885125}]},"public_addr":"172.21.15.31:6803/3089885125","cluster_addr":"172.21.15.31:6805/3089885125","heartbeat_back_addr":"172.21.15.31:6809/3089885125","heartbeat_front_addr":"172.21.15.31:6807/3089885125","state":["exists","up"]},{"osd":1,"uuid":"76a63504-beab-4354-a6c3-eebc3b16ad03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6801","nonce":618840037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6803","nonce":618840037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6807","nonce":618840037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6805","nonce":618840037}]},"public_addr":"172.21.15.40:6801/618840037","cluster_addr":"172.21.15.40:6803/618840037","heartbeat_back_addr":"172.21.15.40:6807/618840037","heartbeat_front_addr":"172.21.15.40:6805/618840037","state":["exists","up"]},{"osd":2,"uuid":"fa191bd6-9a21-4ee2-8adf-8a63c9cfd821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6801","nonce":2524184117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6803","nonce":2524184117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6807","nonce":2524184117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6805","nonce":2524184117}]},"public_addr":"172.21.15.70:6801/2524184117","cluster_addr":"172.21.15.70:6803/2524184117","heartbeat_back_addr":"172.21.15.70:6807/2524184117","heartbeat_front_addr":"172.21.15.70:6805/2524184117","state":["exists","up"]},{"osd":3,"uuid":"edb44507-ffb0-42a4-827b-08461e1bf952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6811","nonce":2823036296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6813","nonce":2823036296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6817","nonce":2823036296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6815","nonce":2823036296}]},"public_addr":"172.21.15.31:6811/2823036296","cluster_addr":"172.21.15.31:6813/2823036296","heartbeat_back_addr":"172.21.15.31:6817/2823036296","heartbeat_front_addr":"172.21.15.31:6815/2823036296","state":["exists","up"]},{"osd":4,"uuid":"8cb89153-8c3b-431f-b57c-8863fada76dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6809","nonce":4215186309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6811","nonce":4215186309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6815","nonce":4215186309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6813","nonce":4215186309}]},"public_addr":"172.21.15.40:6809/4215186309","cluster_addr":"172.21.15.40:6811/4215186309","heartbeat_back_addr":"172.21.15.40:6815/4215186309","heartbeat_front_addr":"172.21.15.40:6813/4215186309","state":["exists","up"]},{"osd":5,"uuid":"9409505a-d463-408c-a3cd-218380c04d68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6809","nonce":3402830452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6811","nonce":3402830452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6815","nonce":3402830452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6813","nonce":3402830452}]},"public_addr":"172.21.15.70:6809/3402830452","cluster_addr":"172.21.15.70:6811/3402830452","heartbeat_back_addr":"172.21.15.70:6815/3402830452","heartbeat_front_addr":"172.21.15.70:6813/3402830452","state":["exists","up"]},{"osd":6,"uuid":"f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6819","nonce":1193497058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6821","nonce":1193497058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6825","nonce":1193497058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6823","nonce":1193497058}]},"public_addr":"172.21.15.31:6819/1193497058","cluster_addr":"172.21.15.31:6821/1193497058","heartbeat_back_addr":"172.21.15.31:6825/1193497058","heartbeat_front_addr":"172.21.15.31:6823/1193497058","state":["exists","up"]},{"osd":7,"uuid":"db228a87-29a5-45d4-8fe6-a0a554262eb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6817","nonce":3825628119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6819","nonce":3825628119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6823","nonce":3825628119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6821","nonce":3825628119}]},"public_addr":"172.21.15.40:6817/3825628119","cluster_addr":"172.21.15.40:6819/3825628119","heartbeat_back_addr":"172.21.15.40:6823/3825628119","heartbeat_front_addr":"172.21.15.40:6821/3825628119","state":["exists","up"]},{"osd":8,"uuid":"59b9fde0-095d-4e0b-87da-ead37bd8ff1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6817","nonce":534880586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6819","nonce":534880586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6823","nonce":534880586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6821","nonce":534880586}]},"public_addr":"172.21.15.70:6817/534880586","cluster_addr":"172.21.15.70:6819/534880586","heartbeat_back_addr":"172.21.15.70:6823/534880586","heartbeat_front_addr":"172.21.15.70:6821/534880586","state":["exists","up"]},{"osd":9,"uuid":"77f47061-0c5f-4159-b276-e814935d6c1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6827","nonce":587047469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6829","nonce":587047469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6833","nonce":587047469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6831","nonce":587047469}]},"public_addr":"172.21.15.31:6827/587047469","cluster_addr":"172.21.15.31:6829/587047469","heartbeat_back_addr":"172.21.15.31:6833/587047469","heartbeat_front_addr":"172.21.15.31:6831/587047469","state":["exists","up"]},{"osd":10,"uuid":"8853dc43-7bd1-4c93-8063-f7982fd7da38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6825","nonce":512108711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6827","nonce":512108711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6831","nonce":512108711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6829","nonce":512108711}]},"public_addr":"172.21.15.40:6825/512108711","cluster_addr":"172.21.15.40:6827/512108711","heartbeat_back_addr":"172.21.15.40:6831/512108711","heartbeat_front_addr":"172.21.15.40:6829/512108711","state":["exists","up"]},{"osd":11,"uuid":"84b7c2de-c424-45a6-9583-60097e372a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6825","nonce":540305718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6827","nonce":540305718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6831","nonce":540305718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6829","nonce":540305718}]},"public_addr":"172.21.15.70:6825/540305718","cluster_addr":"172.21.15.70:6827/540305718","heartbeat_back_addr":"172.21.15.70:6831/540305718","heartbeat_front_addr":"172.21.15.70:6829/540305718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:28:51.477358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:17.405045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:41.626268+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:15.339568+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:42.676996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:07.516999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:42.412273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:10.365041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:35.699120+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:11.733639+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:39.505442+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:34:05.449301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451930554":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/3342588843":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6801/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/1626448389":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/1910059351":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/344822770":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/3811368512":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/1981437651":"2024-03-22T10:26:16.959165+0000","172.21.15.31:6801/1981437651":"2024-03-22T10:26:16.959165+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T10:35:13.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.113+0000 7fba6e400640 1 -- 172.21.15.31:0/4285880770 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fba500a2f10 msgr2=0x7fba500a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:13.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.113+0000 7fba6e400640 1 --2- 172.21.15.31:0/4285880770 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fba500a2f10 0x7fba500a53d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fba5c0058c0 tx=0x7fba5c005850 comp rx=0 tx=0).stop 2024-03-21T10:35:13.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.113+0000 7fba6e400640 1 -- 172.21.15.31:0/4285880770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba681037c0 msgr2=0x7fba68199a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:13.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.113+0000 7fba6e400640 1 --2- 172.21.15.31:0/4285880770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba681037c0 0x7fba68199a50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fba5400e9a0 tx=0x7fba5400ee60 comp rx=0 tx=0).stop 2024-03-21T10:35:13.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.113+0000 7fba6e400640 1 -- 172.21.15.31:0/4285880770 shutdown_connections 2024-03-21T10:35:13.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.113+0000 7fba6e400640 1 --2- 172.21.15.31:0/4285880770 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fba500a2f10 0x7fba500a53d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:13.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.113+0000 7fba6e400640 1 --2- 172.21.15.31:0/4285880770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba68105370 0x7fba681a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:13.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.113+0000 7fba6e400640 1 --2- 172.21.15.31:0/4285880770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba681049b0 0x7fba68199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:13.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.113+0000 7fba6e400640 1 --2- 172.21.15.31:0/4285880770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba681037c0 0x7fba68199a50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:13.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.113+0000 7fba6e400640 1 -- 172.21.15.31:0/4285880770 >> 172.21.15.31:0/4285880770 conn(0x7fba680fed50 msgr2=0x7fba68107c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:13.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.114+0000 7fba6e400640 1 -- 172.21.15.31:0/4285880770 shutdown_connections 2024-03-21T10:35:13.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.114+0000 7fba6e400640 1 -- 172.21.15.31:0/4285880770 wait complete. 2024-03-21T10:35:13.131 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T10:35:13.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.340+0000 7fae5aa00640 1 -- 172.21.15.31:0/134455758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae54102790 msgr2=0x7fae54104c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:13.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.340+0000 7fae5aa00640 1 --2- 172.21.15.31:0/134455758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae54102790 0x7fae54104c00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fae44009900 tx=0x7fae4402bc30 comp rx=0 tx=0).stop 2024-03-21T10:35:13.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.340+0000 7fae5aa00640 1 -- 172.21.15.31:0/134455758 shutdown_connections 2024-03-21T10:35:13.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.340+0000 7fae5aa00640 1 --2- 172.21.15.31:0/134455758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae54105140 0x7fae541055c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:13.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.340+0000 7fae5aa00640 1 --2- 172.21.15.31:0/134455758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae54102790 0x7fae54104c00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:13.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.340+0000 7fae5aa00640 1 --2- 172.21.15.31:0/134455758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae540ffe20 0x7fae54102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:13.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.340+0000 7fae5aa00640 1 -- 172.21.15.31:0/134455758 >> 172.21.15.31:0/134455758 conn(0x7fae540fba00 msgr2=0x7fae540fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:13.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.341+0000 7fae5aa00640 1 -- 172.21.15.31:0/134455758 shutdown_connections 2024-03-21T10:35:13.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.341+0000 7fae5aa00640 1 -- 172.21.15.31:0/134455758 wait complete. 2024-03-21T10:35:13.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.342+0000 7fae5aa00640 1 Processor -- start 2024-03-21T10:35:13.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.342+0000 7fae5aa00640 1 -- start start 2024-03-21T10:35:13.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.342+0000 7fae5aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae540ffe20 0x7fae5419de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:13.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.342+0000 7fae5aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae54102790 0x7fae5419e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:13.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.342+0000 7fae5aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae54105140 0x7fae541a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:13.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.342+0000 7fae5aa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae54113870 con 0x7fae54102790 2024-03-21T10:35:13.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.342+0000 7fae5aa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae541136f0 con 0x7fae540ffe20 2024-03-21T10:35:13.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.342+0000 7fae5aa00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae541139f0 con 0x7fae54105140 2024-03-21T10:35:13.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.343+0000 7fae59600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae540ffe20 0x7fae5419de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:13.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.343+0000 7fae59600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae540ffe20 0x7fae5419de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37036/0 (socket says 172.21.15.31:37036) 2024-03-21T10:35:13.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.343+0000 7fae59600640 1 -- 172.21.15.31:0/300586972 learned_addr learned my addr 172.21.15.31:0/300586972 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:13.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.343+0000 7fae5a000640 1 --2- 172.21.15.31:0/300586972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae54105140 0x7fae541a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:13.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.343+0000 7fae58c00640 1 --2- 172.21.15.31:0/300586972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae54102790 0x7fae5419e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:13.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.343+0000 7fae59600640 1 -- 172.21.15.31:0/300586972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae54105140 msgr2=0x7fae541a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:13.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.343+0000 7fae59600640 1 --2- 172.21.15.31:0/300586972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae54105140 0x7fae541a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:13.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.343+0000 7fae59600640 1 -- 172.21.15.31:0/300586972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae54102790 msgr2=0x7fae5419e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:13.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.343+0000 7fae59600640 1 --2- 172.21.15.31:0/300586972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae54102790 0x7fae5419e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:13.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.343+0000 7fae59600640 1 -- 172.21.15.31:0/300586972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae541a5aa0 con 0x7fae540ffe20 2024-03-21T10:35:13.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.343+0000 7fae5a000640 1 --2- 172.21.15.31:0/300586972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae54105140 0x7fae541a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:35:13.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.343+0000 7fae59600640 1 --2- 172.21.15.31:0/300586972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae540ffe20 0x7fae5419de80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fae4800e9f0 tx=0x7fae4800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:13.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.344+0000 7fae52000640 1 -- 172.21.15.31:0/300586972 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4800cd80 con 0x7fae540ffe20 2024-03-21T10:35:13.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.344+0000 7fae52000640 1 -- 172.21.15.31:0/300586972 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae48005500 con 0x7fae540ffe20 2024-03-21T10:35:13.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.344+0000 7fae5aa00640 1 -- 172.21.15.31:0/300586972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae541a5d90 con 0x7fae540ffe20 2024-03-21T10:35:13.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.344+0000 7fae52000640 1 -- 172.21.15.31:0/300586972 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae48010690 con 0x7fae540ffe20 2024-03-21T10:35:13.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.344+0000 7fae5aa00640 1 -- 172.21.15.31:0/300586972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae541a61f0 con 0x7fae540ffe20 2024-03-21T10:35:13.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.345+0000 7fae5aa00640 1 -- 172.21.15.31:0/300586972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae540fd4f0 con 0x7fae540ffe20 2024-03-21T10:35:13.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.347+0000 7fae52000640 1 -- 172.21.15.31:0/300586972 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fae480046c0 con 0x7fae540ffe20 2024-03-21T10:35:13.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.348+0000 7fae52000640 1 --2- 172.21.15.31:0/300586972 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fae3c0a2f90 0x7fae3c0a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:13.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.348+0000 7fae52000640 1 -- 172.21.15.31:0/300586972 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fae480cf9f0 con 0x7fae540ffe20 2024-03-21T10:35:13.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.348+0000 7fae58c00640 1 --2- 172.21.15.31:0/300586972 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fae3c0a2f90 0x7fae3c0a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:13.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.349+0000 7fae58c00640 1 --2- 172.21.15.31:0/300586972 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fae3c0a2f90 0x7fae3c0a5450 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fae4402c240 tx=0x7fae4402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:13.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.352+0000 7fae52000640 1 -- 172.21.15.31:0/300586972 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fae4809c580 con 0x7fae540ffe20 2024-03-21T10:35:13.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:13 smithi040 ceph-mon[36154]: pgmap v327: 129 pgs: 1 creating+peering, 99 active+clean, 29 unknown; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 6 op/s 2024-03-21T10:35:13.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:13 smithi040 ceph-mon[36154]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T10:35:13.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:13 smithi040 ceph-mon[36154]: osdmap e89: 12 total, 12 up, 12 in 2024-03-21T10:35:13.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:13 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-21T10:35:13.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:13 smithi040 ceph-mon[36154]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-21T10:35:13.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:13 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2064381619' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:35:13.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:13 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/4285880770' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:13.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.555+0000 7fae5aa00640 1 -- 172.21.15.31:0/300586972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae540622a0 con 0x7fae540ffe20 2024-03-21T10:35:13.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:13 smithi031 ceph-mon[31673]: pgmap v327: 129 pgs: 1 creating+peering, 99 active+clean, 29 unknown; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 6 op/s 2024-03-21T10:35:13.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:13 smithi031 ceph-mon[31673]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T10:35:13.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:13 smithi031 ceph-mon[31673]: osdmap e89: 12 total, 12 up, 12 in 2024-03-21T10:35:13.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:13 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-21T10:35:13.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:13 smithi031 ceph-mon[31673]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-21T10:35:13.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:13 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2064381619' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:35:13.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:13 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/4285880770' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:13.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.558+0000 7fae52000640 1 -- 172.21.15.31:0/300586972 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24009 (secure 0 0 0) 0x7fae4809be80 con 0x7fae540ffe20 2024-03-21T10:35:13.556 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:35:13.556 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","created":"2024-03-21T10:25:41.971117+0000","modified":"2024-03-21T10:35:11.657050+0000","last_up_change":"2024-03-21T10:34:07.757196+0000","last_in_change":"2024-03-21T10:33:47.460406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T10:29:44.553968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T10:35:04.945449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T10:35:05.970589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T10:35:09.688711+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e0244d33-00e2-47d5-9f66-34efc138c462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6803","nonce":3089885125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6805","nonce":3089885125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6809","nonce":3089885125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6807","nonce":3089885125}]},"public_addr":"172.21.15.31:6803/3089885125","cluster_addr":"172.21.15.31:6805/3089885125","heartbeat_back_addr":"172.21.15.31:6809/3089885125","heartbeat_front_addr":"172.21.15.31:6807/3089885125","state":["exists","up"]},{"osd":1,"uuid":"76a63504-beab-4354-a6c3-eebc3b16ad03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6801","nonce":618840037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6803","nonce":618840037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6807","nonce":618840037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6805","nonce":618840037}]},"public_addr":"172.21.15.40:6801/618840037","cluster_addr":"172.21.15.40:6803/618840037","heartbeat_back_addr":"172.21.15.40:6807/618840037","heartbeat_front_addr":"172.21.15.40:6805/618840037","state":["exists","up"]},{"osd":2,"uuid":"fa191bd6-9a21-4ee2-8adf-8a63c9cfd821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6801","nonce":2524184117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6803","nonce":2524184117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6807","nonce":2524184117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6805","nonce":2524184117}]},"public_addr":"172.21.15.70:6801/2524184117","cluster_addr":"172.21.15.70:6803/2524184117","heartbeat_back_addr":"172.21.15.70:6807/2524184117","heartbeat_front_addr":"172.21.15.70:6805/2524184117","state":["exists","up"]},{"osd":3,"uuid":"edb44507-ffb0-42a4-827b-08461e1bf952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6811","nonce":2823036296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6813","nonce":2823036296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6817","nonce":2823036296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6815","nonce":2823036296}]},"public_addr":"172.21.15.31:6811/2823036296","cluster_addr":"172.21.15.31:6813/2823036296","heartbeat_back_addr":"172.21.15.31:6817/2823036296","heartbeat_front_addr":"172.21.15.31:6815/2823036296","state":["exists","up"]},{"osd":4,"uuid":"8cb89153-8c3b-431f-b57c-8863fada76dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6809","nonce":4215186309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6811","nonce":4215186309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6815","nonce":4215186309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6813","nonce":4215186309}]},"public_addr":"172.21.15.40:6809/4215186309","cluster_addr":"172.21.15.40:6811/4215186309","heartbeat_back_addr":"172.21.15.40:6815/4215186309","heartbeat_front_addr":"172.21.15.40:6813/4215186309","state":["exists","up"]},{"osd":5,"uuid":"9409505a-d463-408c-a3cd-218380c04d68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6809","nonce":3402830452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6811","nonce":3402830452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6815","nonce":3402830452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6813","nonce":3402830452}]},"public_addr":"172.21.15.70:6809/3402830452","cluster_addr":"172.21.15.70:6811/3402830452","heartbeat_back_addr":"172.21.15.70:6815/3402830452","heartbeat_front_addr":"172.21.15.70:6813/3402830452","state":["exists","up"]},{"osd":6,"uuid":"f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6819","nonce":1193497058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6821","nonce":1193497058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6825","nonce":1193497058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6823","nonce":1193497058}]},"public_addr":"172.21.15.31:6819/1193497058","cluster_addr":"172.21.15.31:6821/1193497058","heartbeat_back_addr":"172.21.15.31:6825/1193497058","heartbeat_front_addr":"172.21.15.31:6823/1193497058","state":["exists","up"]},{"osd":7,"uuid":"db228a87-29a5-45d4-8fe6-a0a554262eb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6817","nonce":3825628119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6819","nonce":3825628119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6823","nonce":3825628119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6821","nonce":3825628119}]},"public_addr":"172.21.15.40:6817/3825628119","cluster_addr":"172.21.15.40:6819/3825628119","heartbeat_back_addr":"172.21.15.40:6823/3825628119","heartbeat_front_addr":"172.21.15.40:6821/3825628119","state":["exists","up"]},{"osd":8,"uuid":"59b9fde0-095d-4e0b-87da-ead37bd8ff1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6817","nonce":534880586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6819","nonce":534880586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6823","nonce":534880586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6821","nonce":534880586}]},"public_addr":"172.21.15.70:6817/534880586","cluster_addr":"172.21.15.70:6819/534880586","heartbeat_back_addr":"172.21.15.70:6823/534880586","heartbeat_front_addr":"172.21.15.70:6821/534880586","state":["exists","up"]},{"osd":9,"uuid":"77f47061-0c5f-4159-b276-e814935d6c1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6827","nonce":587047469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6829","nonce":587047469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6833","nonce":587047469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6831","nonce":587047469}]},"public_addr":"172.21.15.31:6827/587047469","cluster_addr":"172.21.15.31:6829/587047469","heartbeat_back_addr":"172.21.15.31:6833/587047469","heartbeat_front_addr":"172.21.15.31:6831/587047469","state":["exists","up"]},{"osd":10,"uuid":"8853dc43-7bd1-4c93-8063-f7982fd7da38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6825","nonce":512108711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6827","nonce":512108711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6831","nonce":512108711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6829","nonce":512108711}]},"public_addr":"172.21.15.40:6825/512108711","cluster_addr":"172.21.15.40:6827/512108711","heartbeat_back_addr":"172.21.15.40:6831/512108711","heartbeat_front_addr":"172.21.15.40:6829/512108711","state":["exists","up"]},{"osd":11,"uuid":"84b7c2de-c424-45a6-9583-60097e372a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6825","nonce":540305718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6827","nonce":540305718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6831","nonce":540305718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6829","nonce":540305718}]},"public_addr":"172.21.15.70:6825/540305718","cluster_addr":"172.21.15.70:6827/540305718","heartbeat_back_addr":"172.21.15.70:6831/540305718","heartbeat_front_addr":"172.21.15.70:6829/540305718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:28:51.477358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:17.405045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:41.626268+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:15.339568+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:42.676996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:07.516999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:42.412273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:10.365041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:35.699120+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:11.733639+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:39.505442+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:34:05.449301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451930554":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/3342588843":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6801/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/1626448389":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/1910059351":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/344822770":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/3811368512":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/1981437651":"2024-03-22T10:26:16.959165+0000","172.21.15.31:6801/1981437651":"2024-03-22T10:26:16.959165+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T10:35:13.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.561+0000 7fae5aa00640 1 -- 172.21.15.31:0/300586972 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fae3c0a2f90 msgr2=0x7fae3c0a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:13.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.561+0000 7fae5aa00640 1 --2- 172.21.15.31:0/300586972 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fae3c0a2f90 0x7fae3c0a5450 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fae4402c240 tx=0x7fae4402df90 comp rx=0 tx=0).stop 2024-03-21T10:35:13.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.561+0000 7fae5aa00640 1 -- 172.21.15.31:0/300586972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae540ffe20 msgr2=0x7fae5419de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:13.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.561+0000 7fae5aa00640 1 --2- 172.21.15.31:0/300586972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae540ffe20 0x7fae5419de80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fae4800e9f0 tx=0x7fae4800edf0 comp rx=0 tx=0).stop 2024-03-21T10:35:13.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.561+0000 7fae5aa00640 1 -- 172.21.15.31:0/300586972 shutdown_connections 2024-03-21T10:35:13.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.561+0000 7fae5aa00640 1 --2- 172.21.15.31:0/300586972 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fae3c0a2f90 0x7fae3c0a5450 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:13.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.561+0000 7fae5aa00640 1 --2- 172.21.15.31:0/300586972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae54105140 0x7fae541a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:13.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.561+0000 7fae5aa00640 1 --2- 172.21.15.31:0/300586972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae54102790 0x7fae5419e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:13.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.561+0000 7fae5aa00640 1 --2- 172.21.15.31:0/300586972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae540ffe20 0x7fae5419de80 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:13.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.561+0000 7fae5aa00640 1 -- 172.21.15.31:0/300586972 >> 172.21.15.31:0/300586972 conn(0x7fae540fba00 msgr2=0x7fae541007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:13.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.561+0000 7fae5aa00640 1 -- 172.21.15.31:0/300586972 shutdown_connections 2024-03-21T10:35:13.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.561+0000 7fae5aa00640 1 -- 172.21.15.31:0/300586972 wait complete. 2024-03-21T10:35:13.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:13 smithi070 ceph-mon[36330]: pgmap v327: 129 pgs: 1 creating+peering, 99 active+clean, 29 unknown; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 6 op/s 2024-03-21T10:35:13.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:13 smithi070 ceph-mon[36330]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T10:35:13.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:13 smithi070 ceph-mon[36330]: osdmap e89: 12 total, 12 up, 12 in 2024-03-21T10:35:13.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:13 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-21T10:35:13.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:13 smithi070 ceph-mon[36330]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-21T10:35:13.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:13 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2064381619' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:35:13.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:13 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/4285880770' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:13.593 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:35:13.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.823+0000 7f38a2a00640 1 -- 172.21.15.31:0/273501170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f389c108db0 msgr2=0x7f389c109230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:13.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.823+0000 7f38a2a00640 1 --2- 172.21.15.31:0/273501170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f389c108db0 0x7f389c109230 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3888009900 tx=0x7f388802bc40 comp rx=0 tx=0).stop 2024-03-21T10:35:13.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.823+0000 7f38a2a00640 1 -- 172.21.15.31:0/273501170 shutdown_connections 2024-03-21T10:35:13.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.823+0000 7f38a2a00640 1 --2- 172.21.15.31:0/273501170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f389c109770 0x7f389c110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:13.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.823+0000 7f38a2a00640 1 --2- 172.21.15.31:0/273501170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f389c108db0 0x7f389c109230 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:13.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.823+0000 7f38a2a00640 1 --2- 172.21.15.31:0/273501170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f389c107bc0 0x7f389c107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:13.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.823+0000 7f38a2a00640 1 -- 172.21.15.31:0/273501170 >> 172.21.15.31:0/273501170 conn(0x7f389c076b60 msgr2=0x7f389c078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:13.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.823+0000 7f38a2a00640 1 -- 172.21.15.31:0/273501170 shutdown_connections 2024-03-21T10:35:13.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.823+0000 7f38a2a00640 1 -- 172.21.15.31:0/273501170 wait complete. 2024-03-21T10:35:13.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.824+0000 7f38a2a00640 1 Processor -- start 2024-03-21T10:35:13.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.824+0000 7f38a2a00640 1 -- start start 2024-03-21T10:35:13.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.825+0000 7f38a2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f389c107bc0 0x7f389c19ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:13.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.825+0000 7f38a2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f389c108db0 0x7f389c19e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:13.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.825+0000 7f38a2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f389c109770 0x7f389c1a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:13.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.825+0000 7f38a2a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389c113870 con 0x7f389c109770 2024-03-21T10:35:13.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.825+0000 7f38a2a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389c1136f0 con 0x7f389c108db0 2024-03-21T10:35:13.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.825+0000 7f38a2a00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389c1139f0 con 0x7f389c107bc0 2024-03-21T10:35:13.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.825+0000 7f38a0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f389c108db0 0x7f389c19e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:13.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.825+0000 7f38a0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f389c108db0 0x7f389c19e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37056/0 (socket says 172.21.15.31:37056) 2024-03-21T10:35:13.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.825+0000 7f38a0c00640 1 -- 172.21.15.31:0/311249033 learned_addr learned my addr 172.21.15.31:0/311249033 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:13.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.825+0000 7f38a2000640 1 --2- 172.21.15.31:0/311249033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f389c109770 0x7f389c1a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:13.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.825+0000 7f38a1600640 1 --2- 172.21.15.31:0/311249033 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f389c107bc0 0x7f389c19ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:13.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.825+0000 7f38a0c00640 1 -- 172.21.15.31:0/311249033 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f389c107bc0 msgr2=0x7f389c19ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:13.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.825+0000 7f38a0c00640 1 --2- 172.21.15.31:0/311249033 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f389c107bc0 0x7f389c19ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:13.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.825+0000 7f38a0c00640 1 -- 172.21.15.31:0/311249033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f389c109770 msgr2=0x7f389c1a5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:13.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.825+0000 7f38a0c00640 1 --2- 172.21.15.31:0/311249033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f389c109770 0x7f389c1a5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:13.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.825+0000 7f38a0c00640 1 -- 172.21.15.31:0/311249033 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389c1a5af0 con 0x7f389c108db0 2024-03-21T10:35:13.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.825+0000 7f38a1600640 1 --2- 172.21.15.31:0/311249033 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f389c107bc0 0x7f389c19ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:35:13.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.826+0000 7f38a0c00640 1 --2- 172.21.15.31:0/311249033 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f389c108db0 0x7f389c19e410 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3888005ab0 tx=0x7f388802c6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:13.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.826+0000 7f389a000640 1 -- 172.21.15.31:0/311249033 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3888041070 con 0x7f389c108db0 2024-03-21T10:35:13.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.826+0000 7f389a000640 1 -- 172.21.15.31:0/311249033 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3888004450 con 0x7f389c108db0 2024-03-21T10:35:13.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.826+0000 7f38a2a00640 1 -- 172.21.15.31:0/311249033 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f389c1a5d80 con 0x7f389c108db0 2024-03-21T10:35:13.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.826+0000 7f389a000640 1 -- 172.21.15.31:0/311249033 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3888034440 con 0x7f389c108db0 2024-03-21T10:35:13.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.826+0000 7f38a2a00640 1 -- 172.21.15.31:0/311249033 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f389c1a6210 con 0x7f389c108db0 2024-03-21T10:35:13.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.827+0000 7f38a2a00640 1 -- 172.21.15.31:0/311249033 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f389c10ee80 con 0x7f389c108db0 2024-03-21T10:35:13.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.829+0000 7f389a000640 1 -- 172.21.15.31:0/311249033 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f3888044070 con 0x7f389c108db0 2024-03-21T10:35:13.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.831+0000 7f389a000640 1 --2- 172.21.15.31:0/311249033 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f38840a3060 0x7f38840a5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:13.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.831+0000 7f389a000640 1 -- 172.21.15.31:0/311249033 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f38880f11a0 con 0x7f389c108db0 2024-03-21T10:35:13.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.831+0000 7f38a1600640 1 --2- 172.21.15.31:0/311249033 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f38840a3060 0x7f38840a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:13.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.832+0000 7f38a1600640 1 --2- 172.21.15.31:0/311249033 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f38840a3060 0x7f38840a5520 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f389c108c10 tx=0x7f3890005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:13.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:13.834+0000 7f389a000640 1 -- 172.21.15.31:0/311249033 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f38880bdd30 con 0x7f389c108db0 2024-03-21T10:35:14.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.084+0000 7f38a2a00640 1 -- 172.21.15.31:0/311249033 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f389c02d2b0 con 0x7f389c108db0 2024-03-21T10:35:14.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.085+0000 7f389a000640 1 -- 172.21.15.31:0/311249033 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+11312 (secure 0 0 0) 0x7f38880bd630 con 0x7f389c108db0 2024-03-21T10:35:14.084 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:35:14.084 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24539,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24584,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:35:11.657448+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24596},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24596,"qdb_cluster":[24596]},"id":1}]} 2024-03-21T10:35:14.084 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 17 2024-03-21T10:35:14.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.089+0000 7f38a2a00640 1 -- 172.21.15.31:0/311249033 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f38840a3060 msgr2=0x7f38840a5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:14.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.089+0000 7f38a2a00640 1 --2- 172.21.15.31:0/311249033 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f38840a3060 0x7f38840a5520 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f389c108c10 tx=0x7f3890005f90 comp rx=0 tx=0).stop 2024-03-21T10:35:14.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.089+0000 7f38a2a00640 1 -- 172.21.15.31:0/311249033 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f389c108db0 msgr2=0x7f389c19e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:14.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.089+0000 7f38a2a00640 1 --2- 172.21.15.31:0/311249033 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f389c108db0 0x7f389c19e410 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3888005ab0 tx=0x7f388802c6d0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.089+0000 7f38a2a00640 1 -- 172.21.15.31:0/311249033 shutdown_connections 2024-03-21T10:35:14.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.089+0000 7f38a2a00640 1 --2- 172.21.15.31:0/311249033 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f38840a3060 0x7f38840a5520 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.089+0000 7f38a2a00640 1 --2- 172.21.15.31:0/311249033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f389c109770 0x7f389c1a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.089+0000 7f38a2a00640 1 --2- 172.21.15.31:0/311249033 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f389c108db0 0x7f389c19e410 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.089+0000 7f38a2a00640 1 --2- 172.21.15.31:0/311249033 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f389c107bc0 0x7f389c19ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.089+0000 7f38a2a00640 1 -- 172.21.15.31:0/311249033 >> 172.21.15.31:0/311249033 conn(0x7f389c076b60 msgr2=0x7f389c10c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:14.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.089+0000 7f38a2a00640 1 -- 172.21.15.31:0/311249033 shutdown_connections 2024-03-21T10:35:14.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.089+0000 7f38a2a00640 1 -- 172.21.15.31:0/311249033 wait complete. 2024-03-21T10:35:14.106 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T10:35:14.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.336+0000 7f61c4a00640 1 -- 172.21.15.31:0/1886020197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61c0076060 msgr2=0x7f61c0076480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:14.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.336+0000 7f61c4a00640 1 --2- 172.21.15.31:0/1886020197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61c0076060 0x7f61c0076480 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f61b0009900 tx=0x7f61b002bc30 comp rx=0 tx=0).stop 2024-03-21T10:35:14.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.336+0000 7f61c4a00640 1 -- 172.21.15.31:0/1886020197 shutdown_connections 2024-03-21T10:35:14.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.336+0000 7f61c4a00640 1 --2- 172.21.15.31:0/1886020197 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61c0077c40 0x7f61c00784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.336+0000 7f61c4a00640 1 --2- 172.21.15.31:0/1886020197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61c00772a0 0x7f61c0077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.336+0000 7f61c4a00640 1 --2- 172.21.15.31:0/1886020197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61c0076060 0x7f61c0076480 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.336+0000 7f61c4a00640 1 -- 172.21.15.31:0/1886020197 >> 172.21.15.31:0/1886020197 conn(0x7f61c00feeb0 msgr2=0x7f61c0101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:14.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.336+0000 7f61c4a00640 1 -- 172.21.15.31:0/1886020197 shutdown_connections 2024-03-21T10:35:14.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.336+0000 7f61c4a00640 1 -- 172.21.15.31:0/1886020197 wait complete. 2024-03-21T10:35:14.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.337+0000 7f61c4a00640 1 Processor -- start 2024-03-21T10:35:14.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.337+0000 7f61c4a00640 1 -- start start 2024-03-21T10:35:14.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.338+0000 7f61c4a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61c0076060 0x7f61c019dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:14.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.338+0000 7f61c4a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61c00772a0 0x7f61c019e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:14.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.338+0000 7f61c4a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61c0077c40 0x7f61c01a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:14.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.338+0000 7f61c4a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c01139c0 con 0x7f61c00772a0 2024-03-21T10:35:14.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.338+0000 7f61c4a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c0113840 con 0x7f61c0076060 2024-03-21T10:35:14.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.338+0000 7f61c4a00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c0113b40 con 0x7f61c0077c40 2024-03-21T10:35:14.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.338+0000 7f61bde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61c0076060 0x7f61c019dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:14.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.338+0000 7f61bde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61c0076060 0x7f61c019dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37084/0 (socket says 172.21.15.31:37084) 2024-03-21T10:35:14.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.338+0000 7f61bde00640 1 -- 172.21.15.31:0/2382234525 learned_addr learned my addr 172.21.15.31:0/2382234525 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:14.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.338+0000 7f61bd400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61c00772a0 0x7f61c019e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:14.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.338+0000 7f61be800640 1 --2- 172.21.15.31:0/2382234525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61c0077c40 0x7f61c01a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:14.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.338+0000 7f61bde00640 1 -- 172.21.15.31:0/2382234525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61c0077c40 msgr2=0x7f61c01a5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:14.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.338+0000 7f61bde00640 1 --2- 172.21.15.31:0/2382234525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61c0077c40 0x7f61c01a5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.338+0000 7f61bde00640 1 -- 172.21.15.31:0/2382234525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61c00772a0 msgr2=0x7f61c019e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:14.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.338+0000 7f61bde00640 1 --2- 172.21.15.31:0/2382234525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61c00772a0 0x7f61c019e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.338+0000 7f61bde00640 1 -- 172.21.15.31:0/2382234525 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c01a5be0 con 0x7f61c0076060 2024-03-21T10:35:14.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.338+0000 7f61be800640 1 --2- 172.21.15.31:0/2382234525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61c0077c40 0x7f61c01a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:35:14.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.339+0000 7f61bde00640 1 --2- 172.21.15.31:0/2382234525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61c0076060 0x7f61c019dfc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f61b0000c00 tx=0x7f61b002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:14.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.339+0000 7f61b6a00640 1 -- 172.21.15.31:0/2382234525 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b0043020 con 0x7f61c0076060 2024-03-21T10:35:14.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.339+0000 7f61b6a00640 1 -- 172.21.15.31:0/2382234525 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61b00054e0 con 0x7f61c0076060 2024-03-21T10:35:14.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.339+0000 7f61b6a00640 1 -- 172.21.15.31:0/2382234525 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b0003cd0 con 0x7f61c0076060 2024-03-21T10:35:14.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.339+0000 7f61c4a00640 1 -- 172.21.15.31:0/2382234525 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61c01a5e70 con 0x7f61c0076060 2024-03-21T10:35:14.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.339+0000 7f61c4a00640 1 -- 172.21.15.31:0/2382234525 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61c01a6330 con 0x7f61c0076060 2024-03-21T10:35:14.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.340+0000 7f61c4a00640 1 -- 172.21.15.31:0/2382234525 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61c0110420 con 0x7f61c0076060 2024-03-21T10:35:14.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.342+0000 7f61b6a00640 1 -- 172.21.15.31:0/2382234525 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f61b0046020 con 0x7f61c0076060 2024-03-21T10:35:14.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.343+0000 7f61b6a00640 1 --2- 172.21.15.31:0/2382234525 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f61a40a2fa0 0x7f61a40a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:14.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.343+0000 7f61b6a00640 1 -- 172.21.15.31:0/2382234525 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f61b00f1700 con 0x7f61c0076060 2024-03-21T10:35:14.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.344+0000 7f61bd400640 1 --2- 172.21.15.31:0/2382234525 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f61a40a2fa0 0x7f61a40a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:14.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.344+0000 7f61bd400640 1 --2- 172.21.15.31:0/2382234525 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f61a40a2fa0 0x7f61a40a5460 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f61c019f4d0 tx=0x7f61ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:14.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.347+0000 7f61b6a00640 1 -- 172.21.15.31:0/2382234525 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f61b004d020 con 0x7f61c0076060 2024-03-21T10:35:14.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.550+0000 7f61c4a00640 1 -- 172.21.15.31:0/2382234525 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61c0062340 con 0x7f61c0076060 2024-03-21T10:35:14.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.553+0000 7f61b6a00640 1 -- 172.21.15.31:0/2382234525 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24009 (secure 0 0 0) 0x7f61b00be3b0 con 0x7f61c0076060 2024-03-21T10:35:14.551 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:35:14.552 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","created":"2024-03-21T10:25:41.971117+0000","modified":"2024-03-21T10:35:11.657050+0000","last_up_change":"2024-03-21T10:34:07.757196+0000","last_in_change":"2024-03-21T10:33:47.460406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T10:29:44.553968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T10:35:04.945449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T10:35:05.970589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T10:35:09.688711+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e0244d33-00e2-47d5-9f66-34efc138c462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6803","nonce":3089885125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6805","nonce":3089885125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6809","nonce":3089885125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6807","nonce":3089885125}]},"public_addr":"172.21.15.31:6803/3089885125","cluster_addr":"172.21.15.31:6805/3089885125","heartbeat_back_addr":"172.21.15.31:6809/3089885125","heartbeat_front_addr":"172.21.15.31:6807/3089885125","state":["exists","up"]},{"osd":1,"uuid":"76a63504-beab-4354-a6c3-eebc3b16ad03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6801","nonce":618840037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6803","nonce":618840037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6807","nonce":618840037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6805","nonce":618840037}]},"public_addr":"172.21.15.40:6801/618840037","cluster_addr":"172.21.15.40:6803/618840037","heartbeat_back_addr":"172.21.15.40:6807/618840037","heartbeat_front_addr":"172.21.15.40:6805/618840037","state":["exists","up"]},{"osd":2,"uuid":"fa191bd6-9a21-4ee2-8adf-8a63c9cfd821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6801","nonce":2524184117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6803","nonce":2524184117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6807","nonce":2524184117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6805","nonce":2524184117}]},"public_addr":"172.21.15.70:6801/2524184117","cluster_addr":"172.21.15.70:6803/2524184117","heartbeat_back_addr":"172.21.15.70:6807/2524184117","heartbeat_front_addr":"172.21.15.70:6805/2524184117","state":["exists","up"]},{"osd":3,"uuid":"edb44507-ffb0-42a4-827b-08461e1bf952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6811","nonce":2823036296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6813","nonce":2823036296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6817","nonce":2823036296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6815","nonce":2823036296}]},"public_addr":"172.21.15.31:6811/2823036296","cluster_addr":"172.21.15.31:6813/2823036296","heartbeat_back_addr":"172.21.15.31:6817/2823036296","heartbeat_front_addr":"172.21.15.31:6815/2823036296","state":["exists","up"]},{"osd":4,"uuid":"8cb89153-8c3b-431f-b57c-8863fada76dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6809","nonce":4215186309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6811","nonce":4215186309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6815","nonce":4215186309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6813","nonce":4215186309}]},"public_addr":"172.21.15.40:6809/4215186309","cluster_addr":"172.21.15.40:6811/4215186309","heartbeat_back_addr":"172.21.15.40:6815/4215186309","heartbeat_front_addr":"172.21.15.40:6813/4215186309","state":["exists","up"]},{"osd":5,"uuid":"9409505a-d463-408c-a3cd-218380c04d68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6809","nonce":3402830452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6811","nonce":3402830452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6815","nonce":3402830452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6813","nonce":3402830452}]},"public_addr":"172.21.15.70:6809/3402830452","cluster_addr":"172.21.15.70:6811/3402830452","heartbeat_back_addr":"172.21.15.70:6815/3402830452","heartbeat_front_addr":"172.21.15.70:6813/3402830452","state":["exists","up"]},{"osd":6,"uuid":"f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6819","nonce":1193497058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6821","nonce":1193497058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6825","nonce":1193497058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6823","nonce":1193497058}]},"public_addr":"172.21.15.31:6819/1193497058","cluster_addr":"172.21.15.31:6821/1193497058","heartbeat_back_addr":"172.21.15.31:6825/1193497058","heartbeat_front_addr":"172.21.15.31:6823/1193497058","state":["exists","up"]},{"osd":7,"uuid":"db228a87-29a5-45d4-8fe6-a0a554262eb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6817","nonce":3825628119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6819","nonce":3825628119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6823","nonce":3825628119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6821","nonce":3825628119}]},"public_addr":"172.21.15.40:6817/3825628119","cluster_addr":"172.21.15.40:6819/3825628119","heartbeat_back_addr":"172.21.15.40:6823/3825628119","heartbeat_front_addr":"172.21.15.40:6821/3825628119","state":["exists","up"]},{"osd":8,"uuid":"59b9fde0-095d-4e0b-87da-ead37bd8ff1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6817","nonce":534880586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6819","nonce":534880586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6823","nonce":534880586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6821","nonce":534880586}]},"public_addr":"172.21.15.70:6817/534880586","cluster_addr":"172.21.15.70:6819/534880586","heartbeat_back_addr":"172.21.15.70:6823/534880586","heartbeat_front_addr":"172.21.15.70:6821/534880586","state":["exists","up"]},{"osd":9,"uuid":"77f47061-0c5f-4159-b276-e814935d6c1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6827","nonce":587047469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6829","nonce":587047469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6833","nonce":587047469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6831","nonce":587047469}]},"public_addr":"172.21.15.31:6827/587047469","cluster_addr":"172.21.15.31:6829/587047469","heartbeat_back_addr":"172.21.15.31:6833/587047469","heartbeat_front_addr":"172.21.15.31:6831/587047469","state":["exists","up"]},{"osd":10,"uuid":"8853dc43-7bd1-4c93-8063-f7982fd7da38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6825","nonce":512108711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6827","nonce":512108711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6831","nonce":512108711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6829","nonce":512108711}]},"public_addr":"172.21.15.40:6825/512108711","cluster_addr":"172.21.15.40:6827/512108711","heartbeat_back_addr":"172.21.15.40:6831/512108711","heartbeat_front_addr":"172.21.15.40:6829/512108711","state":["exists","up"]},{"osd":11,"uuid":"84b7c2de-c424-45a6-9583-60097e372a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6825","nonce":540305718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6827","nonce":540305718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6831","nonce":540305718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6829","nonce":540305718}]},"public_addr":"172.21.15.70:6825/540305718","cluster_addr":"172.21.15.70:6827/540305718","heartbeat_back_addr":"172.21.15.70:6831/540305718","heartbeat_front_addr":"172.21.15.70:6829/540305718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:28:51.477358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:17.405045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:41.626268+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:15.339568+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:42.676996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:07.516999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:42.412273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:10.365041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:35.699120+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:11.733639+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:39.505442+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:34:05.449301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451930554":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/3342588843":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6801/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/1626448389":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/1910059351":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/344822770":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/3811368512":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/1981437651":"2024-03-22T10:26:16.959165+0000","172.21.15.31:6801/1981437651":"2024-03-22T10:26:16.959165+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T10:35:14.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.555+0000 7f61c4a00640 1 -- 172.21.15.31:0/2382234525 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f61a40a2fa0 msgr2=0x7f61a40a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:14.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.555+0000 7f61c4a00640 1 --2- 172.21.15.31:0/2382234525 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f61a40a2fa0 0x7f61a40a5460 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f61c019f4d0 tx=0x7f61ac005f90 comp rx=0 tx=0).stop 2024-03-21T10:35:14.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.555+0000 7f61c4a00640 1 -- 172.21.15.31:0/2382234525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61c0076060 msgr2=0x7f61c019dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:14.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.555+0000 7f61c4a00640 1 --2- 172.21.15.31:0/2382234525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61c0076060 0x7f61c019dfc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f61b0000c00 tx=0x7f61b002c650 comp rx=0 tx=0).stop 2024-03-21T10:35:14.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.556+0000 7f61c4a00640 1 -- 172.21.15.31:0/2382234525 shutdown_connections 2024-03-21T10:35:14.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.556+0000 7f61c4a00640 1 --2- 172.21.15.31:0/2382234525 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f61a40a2fa0 0x7f61a40a5460 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.556+0000 7f61c4a00640 1 --2- 172.21.15.31:0/2382234525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61c0077c40 0x7f61c01a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.556+0000 7f61c4a00640 1 --2- 172.21.15.31:0/2382234525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61c00772a0 0x7f61c019e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.556+0000 7f61c4a00640 1 --2- 172.21.15.31:0/2382234525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61c0076060 0x7f61c019dfc0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.556+0000 7f61c4a00640 1 -- 172.21.15.31:0/2382234525 >> 172.21.15.31:0/2382234525 conn(0x7f61c00feeb0 msgr2=0x7f61c01005b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:14.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.556+0000 7f61c4a00640 1 -- 172.21.15.31:0/2382234525 shutdown_connections 2024-03-21T10:35:14.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.556+0000 7f61c4a00640 1 -- 172.21.15.31:0/2382234525 wait complete. 2024-03-21T10:35:14.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:14 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/300586972' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:14.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:14 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/311249033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:35:14.574 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-21T10:35:06.492115+0000', 'modified': '2024-03-21T10:35:11.657448+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24596}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24596': {'gid': 24596, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6839/2539201148', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6838', 'nonce': 2539201148}, {'type': 'v1', 'addr': '172.21.15.40:6839', 'nonce': 2539201148}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24596, 'qdb_cluster': [24596]} 2024-03-21T10:35:14.574 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-21T10:35:14.574 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-21T10:35:14.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:14 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/300586972' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:14.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:14 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/311249033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:35:14.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:14 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/300586972' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:14.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:14 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/311249033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:35:14.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.801+0000 7fb35b200640 1 -- 172.21.15.31:0/309158710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb35c1048f0 msgr2=0x7fb35c104d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:14.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.801+0000 7fb35b200640 1 --2- 172.21.15.31:0/309158710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb35c1048f0 0x7fb35c104d70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb348009900 tx=0x7fb34802bc30 comp rx=0 tx=0).stop 2024-03-21T10:35:14.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.801+0000 7fb35b200640 1 -- 172.21.15.31:0/309158710 shutdown_connections 2024-03-21T10:35:14.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.801+0000 7fb35b200640 1 --2- 172.21.15.31:0/309158710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb35c1052b0 0x7fb35c10bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.801+0000 7fb35b200640 1 --2- 172.21.15.31:0/309158710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb35c1048f0 0x7fb35c104d70 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.801+0000 7fb35b200640 1 --2- 172.21.15.31:0/309158710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb35c103700 0x7fb35c103b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.801+0000 7fb35b200640 1 -- 172.21.15.31:0/309158710 >> 172.21.15.31:0/309158710 conn(0x7fb35c0fec70 msgr2=0x7fb35c1010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:14.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.801+0000 7fb35b200640 1 -- 172.21.15.31:0/309158710 shutdown_connections 2024-03-21T10:35:14.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.801+0000 7fb35b200640 1 -- 172.21.15.31:0/309158710 wait complete. 2024-03-21T10:35:14.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.802+0000 7fb35b200640 1 Processor -- start 2024-03-21T10:35:14.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.802+0000 7fb35b200640 1 -- start start 2024-03-21T10:35:14.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.803+0000 7fb35b200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb35c103700 0x7fb35c199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:14.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.803+0000 7fb35b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb35c1048f0 0x7fb35c199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:14.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.803+0000 7fb35b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb35c1052b0 0x7fb35c1a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:14.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.803+0000 7fb35b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c10f3d0 con 0x7fb35c1048f0 2024-03-21T10:35:14.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.803+0000 7fb35b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c10f250 con 0x7fb35c1052b0 2024-03-21T10:35:14.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.803+0000 7fb35b200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c10f550 con 0x7fb35c103700 2024-03-21T10:35:14.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.803+0000 7fb359400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb35c1048f0 0x7fb35c199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:14.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.803+0000 7fb35a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb35c1052b0 0x7fb35c1a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:14.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.803+0000 7fb359e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb35c103700 0x7fb35c199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:14.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.803+0000 7fb359400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb35c1048f0 0x7fb35c199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59288/0 (socket says 172.21.15.31:59288) 2024-03-21T10:35:14.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.803+0000 7fb35a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb35c1052b0 0x7fb35c1a0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37108/0 (socket says 172.21.15.31:37108) 2024-03-21T10:35:14.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.803+0000 7fb359400640 1 -- 172.21.15.31:0/1934720061 learned_addr learned my addr 172.21.15.31:0/1934720061 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:14.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.804+0000 7fb35a800640 1 -- 172.21.15.31:0/1934720061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb35c103700 msgr2=0x7fb35c199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:14.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.804+0000 7fb35a800640 1 --2- 172.21.15.31:0/1934720061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb35c103700 0x7fb35c199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.804+0000 7fb35a800640 1 -- 172.21.15.31:0/1934720061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb35c1048f0 msgr2=0x7fb35c199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:14.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.804+0000 7fb35a800640 1 --2- 172.21.15.31:0/1934720061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb35c1048f0 0x7fb35c199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.804+0000 7fb35a800640 1 -- 172.21.15.31:0/1934720061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35c1a1640 con 0x7fb35c1052b0 2024-03-21T10:35:14.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.804+0000 7fb359400640 1 --2- 172.21.15.31:0/1934720061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb35c1048f0 0x7fb35c199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:35:14.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.804+0000 7fb35a800640 1 --2- 172.21.15.31:0/1934720061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb35c1052b0 0x7fb35c1a0fd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb35400d680 tx=0x7fb35400db40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:14.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.804+0000 7fb352a00640 1 -- 172.21.15.31:0/1934720061 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb354017070 con 0x7fb35c1052b0 2024-03-21T10:35:14.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.804+0000 7fb352a00640 1 -- 172.21.15.31:0/1934720061 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb354009820 con 0x7fb35c1052b0 2024-03-21T10:35:14.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.804+0000 7fb352a00640 1 -- 172.21.15.31:0/1934720061 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb354004e00 con 0x7fb35c1052b0 2024-03-21T10:35:14.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.804+0000 7fb35b200640 1 -- 172.21.15.31:0/1934720061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb35c1a1930 con 0x7fb35c1052b0 2024-03-21T10:35:14.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.804+0000 7fb35b200640 1 -- 172.21.15.31:0/1934720061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb35c1a1e00 con 0x7fb35c1052b0 2024-03-21T10:35:14.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.807+0000 7fb352a00640 1 -- 172.21.15.31:0/1934720061 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fb3540099d0 con 0x7fb35c1052b0 2024-03-21T10:35:14.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.808+0000 7fb36382c740 1 Processor -- start 2024-03-21T10:35:14.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.808+0000 7fb36382c740 1 -- start start 2024-03-21T10:35:14.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.808+0000 7fb36382c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621272b9d80 0x5621272ba1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:14.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.808+0000 7fb36382c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5621272ba770 0x5621272bf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:14.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.808+0000 7fb36382c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621272bf8e0 0x5621272c1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:14.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.808+0000 7fb36382c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56212721ec90 con 0x5621272b9d80 2024-03-21T10:35:14.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.808+0000 7fb36382c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621271c7980 con 0x5621272bf8e0 2024-03-21T10:35:14.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.808+0000 7fb36382c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56212721ad30 con 0x5621272ba770 2024-03-21T10:35:14.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.808+0000 7fb359400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621272b9d80 0x5621272ba1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:14.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.808+0000 7fb359400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621272b9d80 0x5621272ba1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59304/0 (socket says 172.21.15.31:59304) 2024-03-21T10:35:14.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.808+0000 7fb359e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5621272ba770 0x5621272bf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:14.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.808+0000 7fb359400640 1 -- 172.21.15.31:0/3871735638 learned_addr learned my addr 172.21.15.31:0/3871735638 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:14.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.809+0000 7fb359400640 1 --2- 172.21.15.31:0/3871735638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621272bf8e0 0x5621272c1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:14.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.809+0000 7fb359e00640 1 -- 172.21.15.31:0/3871735638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621272bf8e0 msgr2=0x5621272c1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:35:14.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.809+0000 7fb359e00640 1 --2- 172.21.15.31:0/3871735638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621272bf8e0 0x5621272c1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.809+0000 7fb359e00640 1 -- 172.21.15.31:0/3871735638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621272b9d80 msgr2=0x5621272ba1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:35:14.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.809+0000 7fb359e00640 1 --2- 172.21.15.31:0/3871735638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621272b9d80 0x5621272ba1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.809+0000 7fb359e00640 1 -- 172.21.15.31:0/3871735638 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5621271df510 con 0x5621272ba770 2024-03-21T10:35:14.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.809+0000 7fb352a00640 1 --2- 172.21.15.31:0/1934720061 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb3400a2fa0 0x7fb3400a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.809+0000 7fb352a00640 1 -- 172.21.15.31:0/1934720061 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb3540cee10 con 0x7fb35c1052b0 2024-03-21T10:35:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.809+0000 7fb359e00640 1 --2- 172.21.15.31:0/3871735638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5621272ba770 0x5621272bf270 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb34c005fd0 tx=0x7fb34c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df1231304a717ed3 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.809+0000 7fb359400640 1 --2- 172.21.15.31:0/1934720061 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb3400a2fa0 0x7fb3400a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.809+0000 7fb347000640 1 -- 172.21.15.31:0/3871735638 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c004b20 con 0x5621272ba770 2024-03-21T10:35:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.809+0000 7fb347000640 1 -- 172.21.15.31:0/3871735638 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb34c003e50 con 0x5621272ba770 2024-03-21T10:35:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.810+0000 7fb347000640 1 -- 172.21.15.31:0/3871735638 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c00f540 con 0x5621272ba770 2024-03-21T10:35:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.810+0000 7fb36382c740 1 -- 172.21.15.31:0/3871735638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5621272ba770 msgr2=0x5621272bf270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.810+0000 7fb359400640 1 --2- 172.21.15.31:0/1934720061 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb3400a2fa0 0x7fb3400a5460 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb348005ec0 tx=0x7fb348005bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.810+0000 7fb36382c740 1 --2- 172.21.15.31:0/3871735638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5621272ba770 0x5621272bf270 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb34c005fd0 tx=0x7fb34c0118d0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.810+0000 7fb36382c740 1 -- 172.21.15.31:0/3871735638 shutdown_connections 2024-03-21T10:35:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.810+0000 7fb36382c740 1 --2- 172.21.15.31:0/3871735638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621272bf8e0 0x5621272c1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.810+0000 7fb36382c740 1 --2- 172.21.15.31:0/3871735638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5621272ba770 0x5621272bf270 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.810+0000 7fb36382c740 1 --2- 172.21.15.31:0/3871735638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621272b9d80 0x5621272ba1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.810+0000 7fb36382c740 1 -- 172.21.15.31:0/3871735638 >> 172.21.15.31:0/3871735638 conn(0x5621272b52f0 msgr2=0x5621272b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:14.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.810+0000 7fb36382c740 1 -- 172.21.15.31:0/3871735638 shutdown_connections 2024-03-21T10:35:14.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.810+0000 7fb36382c740 1 -- 172.21.15.31:0/3871735638 wait complete. 2024-03-21T10:35:14.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.811+0000 7fb36382c740 1 -- start start 2024-03-21T10:35:14.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.811+0000 7fb36382c740 1 Processor -- start 2024-03-21T10:35:14.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.811+0000 7fb36382c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5621272b9d80 0x5621272c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:14.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.811+0000 7fb36382c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621272ba770 0x5621272c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:14.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.811+0000 7fb36382c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621272bf8e0 0x5621272cb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:14.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.811+0000 7fb36382c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621272c2890 con 0x5621272ba770 2024-03-21T10:35:14.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.811+0000 7fb36382c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621271dadd0 con 0x5621272bf8e0 2024-03-21T10:35:14.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.811+0000 7fb36382c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621272c2a10 con 0x5621272b9d80 2024-03-21T10:35:14.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.812+0000 7fb36382c740 10 client.0 _ll_register_callbacks cb 0x562126f32ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:35:14.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.812+0000 7fb359400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621272ba770 0x5621272c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:14.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.812+0000 7fb359e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5621272b9d80 0x5621272c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:14.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.812+0000 7fb359400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621272ba770 0x5621272c47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59318/0 (socket says 172.21.15.31:59318) 2024-03-21T10:35:14.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.812+0000 7fb359400640 1 -- 172.21.15.31:0/247599245 learned_addr learned my addr 172.21.15.31:0/247599245 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:14.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.812+0000 7fb35a800640 1 --2- 172.21.15.31:0/247599245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621272bf8e0 0x5621272cb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:14.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.812+0000 7fb35a800640 1 -- 172.21.15.31:0/247599245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5621272b9d80 msgr2=0x5621272c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:35:14.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.812+0000 7fb35a800640 1 --2- 172.21.15.31:0/247599245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5621272b9d80 0x5621272c4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.812+0000 7fb35a800640 1 -- 172.21.15.31:0/247599245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621272ba770 msgr2=0x5621272c47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:35:14.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.812+0000 7fb35a800640 1 --2- 172.21.15.31:0/247599245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621272ba770 0x5621272c47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.812+0000 7fb35a800640 1 -- 172.21.15.31:0/247599245 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562126f332d0 con 0x5621272bf8e0 2024-03-21T10:35:14.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.812+0000 7fb359e00640 1 --2- 172.21.15.31:0/247599245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5621272b9d80 0x5621272c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:35:14.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.812+0000 7fb35a800640 1 --2- 172.21.15.31:0/247599245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621272bf8e0 0x5621272cb810 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb35409b2e0 tx=0x7fb35409b490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f5143f6e346d53d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:14.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.812+0000 7fb33b400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:35:14.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.813+0000 7fb33b400640 1 -- 172.21.15.31:0/247599245 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb354020070 con 0x5621272bf8e0 2024-03-21T10:35:14.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.813+0000 7fb33b400640 1 -- 172.21.15.31:0/247599245 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb354009d60 con 0x5621272bf8e0 2024-03-21T10:35:14.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.813+0000 7fb36382c740 1 -- 172.21.15.31:0/247599245 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb34c003e50 con 0x5621272bf8e0 2024-03-21T10:35:14.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.813+0000 7fb33b400640 1 -- 172.21.15.31:0/247599245 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3540979c0 con 0x5621272bf8e0 2024-03-21T10:35:14.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.813+0000 7fb36382c740 10 client.24704 fetch_fsmap learned FSMap version 17 2024-03-21T10:35:14.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.813+0000 7fb36382c740 1 -- 172.21.15.31:0/247599245 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x562126ffd6b0 con 0x5621272bf8e0 2024-03-21T10:35:14.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.813+0000 7fb33b400640 1 -- 172.21.15.31:0/247599245 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb34c003e50 con 0x5621272bf8e0 2024-03-21T10:35:14.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.813+0000 7fb33b400640 1 -- 172.21.15.31:0/247599245 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 17) v1 ==== 7922+0+0 (secure 0 0 0) 0x7fb35409a7a0 con 0x5621272bf8e0 2024-03-21T10:35:14.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.813+0000 7fb36382c740 10 client.24704 fetch_fsmap finished waiting for FSMap version 17 2024-03-21T10:35:14.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.813+0000 7fb36382c740 10 client.24704 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-21T10:35:14.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.813+0000 7fb36382c740 20 client.24704 populate_metadata read hostname 'smithi031' 2024-03-21T10:35:14.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.813+0000 7fb36382c740 1 --2- 172.21.15.31:0/247599245 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5621272d0aa0 0x5621272cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:14.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.813+0000 7fb36382c740 4 client.24704 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:35:14.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.813+0000 7fb36382c740 1 -- 172.21.15.31:0/247599245 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5621271c0520 con 0x5621272d0aa0 2024-03-21T10:35:14.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.814+0000 7fb359e00640 1 --2- 172.21.15.31:0/247599245 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5621272d0aa0 0x5621272cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:14.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.814+0000 7fb359e00640 1 --2- 172.21.15.31:0/247599245 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5621272d0aa0 0x5621272cfb40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d129599c9ee2086 server_cookie=b4d94313f423667a in_seq=0 out_seq=0 2024-03-21T10:35:14.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.814+0000 7fb33b400640 10 client.24704 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:35:14.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.815+0000 7fb33b400640 1 -- 172.21.15.31:0/247599245 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5621271c0520 con 0x5621272d0aa0 2024-03-21T10:35:14.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.815+0000 7fb33b400640 10 client.24704 handle_command_reply: tid=0 2024-03-21T10:35:14.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.815+0000 7fb36382c740 1 client.24704 shutdown 2024-03-21T10:35:14.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.815+0000 7fb36382c740 1 -- 172.21.15.31:0/247599245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621272bf8e0 msgr2=0x5621272cb810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:14.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.815+0000 7fb36382c740 1 --2- 172.21.15.31:0/247599245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621272bf8e0 0x5621272cb810 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb35409b2e0 tx=0x7fb35409b490 comp rx=0 tx=0).stop 2024-03-21T10:35:14.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.815+0000 7fb36382c740 1 -- 172.21.15.31:0/247599245 shutdown_connections 2024-03-21T10:35:14.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.815+0000 7fb36382c740 1 --2- 172.21.15.31:0/247599245 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5621272d0aa0 0x5621272cfb40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.815+0000 7fb36382c740 1 --2- 172.21.15.31:0/247599245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621272bf8e0 0x5621272cb810 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.815+0000 7fb33b400640 0 client.24704 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:35:14.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.815+0000 7fb36382c740 1 --2- 172.21.15.31:0/247599245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621272ba770 0x5621272c47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.815+0000 7fb36382c740 1 --2- 172.21.15.31:0/247599245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5621272b9d80 0x5621272c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.815+0000 7fb36382c740 1 -- 172.21.15.31:0/247599245 >> 172.21.15.31:0/247599245 conn(0x5621272b52f0 msgr2=0x5621272b44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:14.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.815+0000 7fb36382c740 1 -- 172.21.15.31:0/247599245 shutdown_connections 2024-03-21T10:35:14.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.815+0000 7fb36382c740 1 -- 172.21.15.31:0/247599245 wait complete. 2024-03-21T10:35:14.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.816+0000 7fb36382c740 20 client.24704 trim_cache size 0 max 16384 2024-03-21T10:35:14.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.834+0000 7fb36382c740 1 Processor -- start 2024-03-21T10:35:14.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.834+0000 7fb36382c740 1 -- start start 2024-03-21T10:35:14.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.835+0000 7fb36382c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5621272b9d80 0x5621272b56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:14.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.835+0000 7fb36382c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621272ba770 0x5621272cb730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:14.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.835+0000 7fb36382c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621272bf8e0 0x5621272c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:14.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.835+0000 7fb36382c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621272d02c0 con 0x5621272ba770 2024-03-21T10:35:14.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.835+0000 7fb36382c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621272d0920 con 0x5621272bf8e0 2024-03-21T10:35:14.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.835+0000 7fb36382c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621272d0f80 con 0x5621272b9d80 2024-03-21T10:35:14.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.835+0000 7fb35a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621272bf8e0 0x5621272c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:14.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.835+0000 7fb359400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621272ba770 0x5621272cb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:14.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.835+0000 7fb35a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621272bf8e0 0x5621272c4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37132/0 (socket says 172.21.15.31:37132) 2024-03-21T10:35:14.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.835+0000 7fb35a800640 1 -- 172.21.15.31:0/3158095427 learned_addr learned my addr 172.21.15.31:0/3158095427 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:14.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.835+0000 7fb359e00640 1 --2- 172.21.15.31:0/3158095427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5621272b9d80 0x5621272b56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:14.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.835+0000 7fb35a800640 1 -- 172.21.15.31:0/3158095427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5621272b9d80 msgr2=0x5621272b56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:35:14.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.835+0000 7fb35a800640 1 --2- 172.21.15.31:0/3158095427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5621272b9d80 0x5621272b56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.835+0000 7fb35a800640 1 -- 172.21.15.31:0/3158095427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621272ba770 msgr2=0x5621272cb730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:35:14.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.835+0000 7fb35a800640 1 --2- 172.21.15.31:0/3158095427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621272ba770 0x5621272cb730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.835+0000 7fb35a800640 1 -- 172.21.15.31:0/3158095427 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5621272c36a0 con 0x5621272bf8e0 2024-03-21T10:35:14.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.835+0000 7fb35a800640 1 --2- 172.21.15.31:0/3158095427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621272bf8e0 0x5621272c4290 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb354097d20 tx=0x7fb354099860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29192a3641e08310 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:14.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.836+0000 7fb346600640 1 -- 172.21.15.31:0/3158095427 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb354020070 con 0x5621272bf8e0 2024-03-21T10:35:14.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.836+0000 7fb346600640 1 -- 172.21.15.31:0/3158095427 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb354009d60 con 0x5621272bf8e0 2024-03-21T10:35:14.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.836+0000 7fb346600640 1 -- 172.21.15.31:0/3158095427 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3540ccb30 con 0x5621272bf8e0 2024-03-21T10:35:14.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.836+0000 7fb36382c740 1 -- 172.21.15.31:0/3158095427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621272bf8e0 msgr2=0x5621272c4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:14.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.836+0000 7fb36382c740 1 --2- 172.21.15.31:0/3158095427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621272bf8e0 0x5621272c4290 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb354097d20 tx=0x7fb354099860 comp rx=0 tx=0).stop 2024-03-21T10:35:14.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.836+0000 7fb36382c740 1 -- 172.21.15.31:0/3158095427 shutdown_connections 2024-03-21T10:35:14.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.836+0000 7fb36382c740 1 --2- 172.21.15.31:0/3158095427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621272bf8e0 0x5621272c4290 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.836+0000 7fb36382c740 1 --2- 172.21.15.31:0/3158095427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621272ba770 0x5621272cb730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.836+0000 7fb36382c740 1 --2- 172.21.15.31:0/3158095427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5621272b9d80 0x5621272b56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.836+0000 7fb36382c740 1 -- 172.21.15.31:0/3158095427 >> 172.21.15.31:0/3158095427 conn(0x5621272b52f0 msgr2=0x5621272cfb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:14.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.836+0000 7fb36382c740 1 -- 172.21.15.31:0/3158095427 shutdown_connections 2024-03-21T10:35:14.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.836+0000 7fb36382c740 1 -- 172.21.15.31:0/3158095427 wait complete. 2024-03-21T10:35:14.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.837+0000 7fb36382c740 1 -- start start 2024-03-21T10:35:14.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.837+0000 7fb36382c740 1 Processor -- start 2024-03-21T10:35:14.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.837+0000 7fb36382c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621272b9d80 0x5621272c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:14.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.837+0000 7fb36382c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621272ba770 0x5621272c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:14.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.837+0000 7fb36382c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5621272bf8e0 0x5621272cad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:14.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.837+0000 7fb36382c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621272d1c40 con 0x5621272ba770 2024-03-21T10:35:14.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.837+0000 7fb36382c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621272d15e0 con 0x5621272b9d80 2024-03-21T10:35:14.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.838+0000 7fb36382c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621272d22a0 con 0x5621272bf8e0 2024-03-21T10:35:14.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.838+0000 7fb36382c740 10 client.0 _ll_register_callbacks cb 0x7fb354097670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:35:14.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.838+0000 7fb359e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621272b9d80 0x5621272c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:14.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.838+0000 7fb359400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621272ba770 0x5621272c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:14.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.838+0000 7fb359e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621272b9d80 0x5621272c77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37140/0 (socket says 172.21.15.31:37140) 2024-03-21T10:35:14.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.838+0000 7fb359e00640 1 -- 172.21.15.31:0/2951029233 learned_addr learned my addr 172.21.15.31:0/2951029233 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:14.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.838+0000 7fb35a800640 1 --2- 172.21.15.31:0/2951029233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5621272bf8e0 0x5621272cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:14.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.838+0000 7fb359e00640 1 -- 172.21.15.31:0/2951029233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5621272bf8e0 msgr2=0x5621272cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:35:14.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.838+0000 7fb359e00640 1 --2- 172.21.15.31:0/2951029233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5621272bf8e0 0x5621272cad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.838+0000 7fb359e00640 1 -- 172.21.15.31:0/2951029233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621272ba770 msgr2=0x5621272c7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:35:14.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.838+0000 7fb359e00640 1 --2- 172.21.15.31:0/2951029233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621272ba770 0x5621272c7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.838+0000 7fb359e00640 1 -- 172.21.15.31:0/2951029233 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5621272cb300 con 0x5621272b9d80 2024-03-21T10:35:14.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.838+0000 7fb359e00640 1 --2- 172.21.15.31:0/2951029233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621272b9d80 0x5621272c77b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb34c011dd0 tx=0x7fb34c0096c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e707dd05ce670a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:14.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.838+0000 7fb33b400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:35:14.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.839+0000 7fb33b400640 1 -- 172.21.15.31:0/2951029233 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c004940 con 0x5621272b9d80 2024-03-21T10:35:14.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.839+0000 7fb33b400640 1 -- 172.21.15.31:0/2951029233 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb34c016040 con 0x5621272b9d80 2024-03-21T10:35:14.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.839+0000 7fb36382c740 1 -- 172.21.15.31:0/2951029233 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb354009d60 con 0x5621272b9d80 2024-03-21T10:35:14.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.839+0000 7fb33b400640 1 -- 172.21.15.31:0/2951029233 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb34c038050 con 0x5621272b9d80 2024-03-21T10:35:14.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.839+0000 7fb36382c740 10 client.24710 fetch_fsmap learned FSMap version 17 2024-03-21T10:35:14.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.839+0000 7fb36382c740 1 -- 172.21.15.31:0/2951029233 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x56212732c4a0 con 0x5621272b9d80 2024-03-21T10:35:14.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.839+0000 7fb33b400640 1 -- 172.21.15.31:0/2951029233 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb354009d60 con 0x5621272b9d80 2024-03-21T10:35:14.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.839+0000 7fb33b400640 1 -- 172.21.15.31:0/2951029233 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 17) v1 ==== 7922+0+0 (secure 0 0 0) 0x7fb34c0050d0 con 0x5621272b9d80 2024-03-21T10:35:14.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.839+0000 7fb36382c740 10 client.24710 fetch_fsmap finished waiting for FSMap version 17 2024-03-21T10:35:14.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.839+0000 7fb36382c740 10 client.24710 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-21T10:35:14.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.839+0000 7fb36382c740 20 client.24710 populate_metadata read hostname 'smithi031' 2024-03-21T10:35:14.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.839+0000 7fb36382c740 1 --2- 172.21.15.31:0/2951029233 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562127330990 0x56212732c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:14.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.839+0000 7fb36382c740 4 client.24710 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-21T10:35:14.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.839+0000 7fb36382c740 1 -- 172.21.15.31:0/2951029233 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56212722c1d0 con 0x562127330990 2024-03-21T10:35:14.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.840+0000 7fb359400640 1 --2- 172.21.15.31:0/2951029233 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562127330990 0x56212732c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:14.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.840+0000 7fb359400640 1 --2- 172.21.15.31:0/2951029233 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562127330990 0x56212732c730 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4840c8880958e006 server_cookie=c40c74790b36ef90 in_seq=0 out_seq=0 2024-03-21T10:35:14.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.840+0000 7fb33b400640 10 client.24710 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:35:14.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.840+0000 7fb33b400640 1 -- 172.21.15.31:0/2951029233 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x56212722c1d0 con 0x562127330990 2024-03-21T10:35:14.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.840+0000 7fb33b400640 10 client.24710 handle_command_reply: tid=0 2024-03-21T10:35:14.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.840+0000 7fb36382c740 1 client.24710 shutdown 2024-03-21T10:35:14.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.840+0000 7fb36382c740 1 -- 172.21.15.31:0/2951029233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621272b9d80 msgr2=0x5621272c77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:14.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.840+0000 7fb36382c740 1 --2- 172.21.15.31:0/2951029233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621272b9d80 0x5621272c77b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb34c011dd0 tx=0x7fb34c0096c0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.841+0000 7fb36382c740 1 -- 172.21.15.31:0/2951029233 shutdown_connections 2024-03-21T10:35:14.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.841+0000 7fb36382c740 1 --2- 172.21.15.31:0/2951029233 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562127330990 0x56212732c730 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.841+0000 7fb36382c740 1 --2- 172.21.15.31:0/2951029233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5621272bf8e0 0x5621272cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.841+0000 7fb36382c740 1 --2- 172.21.15.31:0/2951029233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621272ba770 0x5621272c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.841+0000 7fb36382c740 1 --2- 172.21.15.31:0/2951029233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621272b9d80 0x5621272c77b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.841+0000 7fb33b400640 0 client.24710 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:35:14.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.841+0000 7fb36382c740 1 -- 172.21.15.31:0/2951029233 >> 172.21.15.31:0/2951029233 conn(0x5621272b52f0 msgr2=0x5621272b56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:14.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.841+0000 7fb36382c740 1 -- 172.21.15.31:0/2951029233 shutdown_connections 2024-03-21T10:35:14.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.841+0000 7fb36382c740 1 -- 172.21.15.31:0/2951029233 wait complete. 2024-03-21T10:35:14.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.841+0000 7fb36382c740 20 client.24710 trim_cache size 0 max 16384 2024-03-21T10:35:14.844 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:35:14.845 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "3847408c-e76d-11ee-95cd-87774f69a715", 2024-03-21T10:35:14.845 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2024-03-21T10:35:14.845 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24596, 2024-03-21T10:35:14.845 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-03-21T10:35:14.845 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-03-21T10:35:14.845 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-03-21T10:35:14.845 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 7.8441513220000001, 2024-03-21T10:35:14.845 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 17, 2024-03-21T10:35:14.845 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 89, 2024-03-21T10:35:14.845 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 89, 2024-03-21T10:35:14.845 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 13.667808202 2024-03-21T10:35:14.845 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:35:14.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.841+0000 7fb35b200640 1 -- 172.21.15.31:0/1934720061 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb3400a2fa0 msgr2=0x7fb3400a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:14.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.841+0000 7fb35b200640 1 --2- 172.21.15.31:0/1934720061 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb3400a2fa0 0x7fb3400a5460 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb348005ec0 tx=0x7fb348005bd0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.841+0000 7fb35b200640 1 -- 172.21.15.31:0/1934720061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb35c1052b0 msgr2=0x7fb35c1a0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:14.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.841+0000 7fb35b200640 1 --2- 172.21.15.31:0/1934720061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb35c1052b0 0x7fb35c1a0fd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb35400d680 tx=0x7fb35400db40 comp rx=0 tx=0).stop 2024-03-21T10:35:14.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.842+0000 7fb35b200640 1 -- 172.21.15.31:0/1934720061 shutdown_connections 2024-03-21T10:35:14.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.842+0000 7fb35b200640 1 --2- 172.21.15.31:0/1934720061 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb3400a2fa0 0x7fb3400a5460 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.842+0000 7fb35b200640 1 --2- 172.21.15.31:0/1934720061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb35c1052b0 0x7fb35c1a0fd0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.842+0000 7fb35b200640 1 --2- 172.21.15.31:0/1934720061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb35c1048f0 0x7fb35c199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.842+0000 7fb35b200640 1 --2- 172.21.15.31:0/1934720061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb35c103700 0x7fb35c199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:14.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.842+0000 7fb35b200640 1 -- 172.21.15.31:0/1934720061 >> 172.21.15.31:0/1934720061 conn(0x7fb35c0fec70 msgr2=0x7fb35c107bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:14.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.842+0000 7fb35b200640 1 -- 172.21.15.31:0/1934720061 shutdown_connections 2024-03-21T10:35:14.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:14.842+0000 7fb35b200640 1 -- 172.21.15.31:0/1934720061 wait complete. 2024-03-21T10:35:14.857 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-21T10:35:14.857 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-21T10:35:14.857 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph fs ls 2024-03-21T10:35:15.314 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:35:15.469 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:15 smithi031 ceph-mon[31673]: pgmap v329: 129 pgs: 1 creating+peering, 104 active+clean, 24 unknown; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 5 op/s 2024-03-21T10:35:15.469 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:15 smithi031 ceph-mon[31673]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T10:35:15.469 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:15 smithi031 ceph-mon[31673]: Cluster is now healthy 2024-03-21T10:35:15.469 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:15 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2382234525' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:15.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:15 smithi070 ceph-mon[36330]: pgmap v329: 129 pgs: 1 creating+peering, 104 active+clean, 24 unknown; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 5 op/s 2024-03-21T10:35:15.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:15 smithi070 ceph-mon[36330]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T10:35:15.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:15 smithi070 ceph-mon[36330]: Cluster is now healthy 2024-03-21T10:35:15.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:15 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2382234525' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:15.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:15 smithi040 ceph-mon[36154]: pgmap v329: 129 pgs: 1 creating+peering, 104 active+clean, 24 unknown; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 5 op/s 2024-03-21T10:35:15.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:15 smithi040 ceph-mon[36154]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T10:35:15.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:15 smithi040 ceph-mon[36154]: Cluster is now healthy 2024-03-21T10:35:15.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:15 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2382234525' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:17.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:17 smithi031 ceph-mon[31673]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-21T10:35:17.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:17 smithi070 ceph-mon[36330]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-21T10:35:17.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:17 smithi040 ceph-mon[36154]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-21T10:35:17.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.807+0000 7f34eaa00700 1 -- 172.21.15.31:0/921307743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e406a380 msgr2=0x7f34e406a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:17.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.807+0000 7f34eaa00700 1 --2- 172.21.15.31:0/921307743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e406a380 0x7f34e406a7e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f34cc009990 tx=0x7f34cc009ca0 comp rx=0 tx=0).stop 2024-03-21T10:35:17.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.807+0000 7f34eaa00700 1 -- 172.21.15.31:0/921307743 shutdown_connections 2024-03-21T10:35:17.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.807+0000 7f34eaa00700 1 --2- 172.21.15.31:0/921307743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34e4104b90 0x7f34e4105020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:17.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.807+0000 7f34eaa00700 1 --2- 172.21.15.31:0/921307743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34e4107080 0x7f34e4069e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:17.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.807+0000 7f34eaa00700 1 --2- 172.21.15.31:0/921307743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e406a380 0x7f34e406a7e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:17.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.807+0000 7f34eaa00700 1 -- 172.21.15.31:0/921307743 >> 172.21.15.31:0/921307743 conn(0x7f34e4077fc0 msgr2=0x7f34e40783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:17.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.807+0000 7f34eaa00700 1 -- 172.21.15.31:0/921307743 shutdown_connections 2024-03-21T10:35:17.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.807+0000 7f34eaa00700 1 -- 172.21.15.31:0/921307743 wait complete. 2024-03-21T10:35:17.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.808+0000 7f34eaa00700 1 Processor -- start 2024-03-21T10:35:17.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.808+0000 7f34eaa00700 1 -- start start 2024-03-21T10:35:17.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.809+0000 7f34eaa00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e406a380 0x7f34e419ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:17.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.809+0000 7f34eaa00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34e4104b90 0x7f34e419f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:17.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.809+0000 7f34eaa00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34e4107080 0x7f34e41a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:17.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.809+0000 7f34eaa00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e4118a80 con 0x7f34e4104b90 2024-03-21T10:35:17.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.809+0000 7f34eaa00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e4118900 con 0x7f34e406a380 2024-03-21T10:35:17.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.809+0000 7f34eaa00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e4118c00 con 0x7f34e4107080 2024-03-21T10:35:17.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.809+0000 7f34e3e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e406a380 0x7f34e419ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:17.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.809+0000 7f34e3e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e406a380 0x7f34e419ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:45438/0 (socket says 172.21.15.31:45438) 2024-03-21T10:35:17.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.809+0000 7f34e3e00700 1 -- 172.21.15.31:0/3701140618 learned_addr learned my addr 172.21.15.31:0/3701140618 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:17.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.809+0000 7f34db400700 1 --2- 172.21.15.31:0/3701140618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34e4104b90 0x7f34e419f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:17.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.809+0000 7f34e8a00700 1 --2- 172.21.15.31:0/3701140618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34e4107080 0x7f34e41a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:17.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.809+0000 7f34e3e00700 1 -- 172.21.15.31:0/3701140618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34e4107080 msgr2=0x7f34e41a37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:17.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.809+0000 7f34e3e00700 1 --2- 172.21.15.31:0/3701140618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34e4107080 0x7f34e41a37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:17.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.809+0000 7f34e3e00700 1 -- 172.21.15.31:0/3701140618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34e4104b90 msgr2=0x7f34e419f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:17.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.809+0000 7f34e3e00700 1 --2- 172.21.15.31:0/3701140618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34e4104b90 0x7f34e419f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:17.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.809+0000 7f34e3e00700 1 -- 172.21.15.31:0/3701140618 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e41a3f20 con 0x7f34e406a380 2024-03-21T10:35:17.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.809+0000 7f34db400700 1 --2- 172.21.15.31:0/3701140618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34e4104b90 0x7f34e419f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:35:17.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.809+0000 7f34e8a00700 1 --2- 172.21.15.31:0/3701140618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34e4107080 0x7f34e41a37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:35:17.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.810+0000 7f34e3e00700 1 --2- 172.21.15.31:0/3701140618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e406a380 0x7f34e419ef10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f34d400e9a0 tx=0x7f34d400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:17.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.810+0000 7f34e1600700 1 -- 172.21.15.31:0/3701140618 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d400cbd0 con 0x7f34e406a380 2024-03-21T10:35:17.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.810+0000 7f34e1600700 1 -- 172.21.15.31:0/3701140618 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f34d400cd70 con 0x7f34e406a380 2024-03-21T10:35:17.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.810+0000 7f34e1600700 1 -- 172.21.15.31:0/3701140618 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d40187c0 con 0x7f34e406a380 2024-03-21T10:35:17.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.810+0000 7f34eaa00700 1 -- 172.21.15.31:0/3701140618 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e41a4210 con 0x7f34e406a380 2024-03-21T10:35:17.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.810+0000 7f34eaa00700 1 -- 172.21.15.31:0/3701140618 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e41aba90 con 0x7f34e406a380 2024-03-21T10:35:17.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.811+0000 7f34eaa00700 1 -- 172.21.15.31:0/3701140618 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34e4068e50 con 0x7f34e406a380 2024-03-21T10:35:17.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.813+0000 7f34e1600700 1 -- 172.21.15.31:0/3701140618 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f34d4010430 con 0x7f34e406a380 2024-03-21T10:35:17.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.814+0000 7f34e1600700 1 --2- 172.21.15.31:0/3701140618 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f34c80a3030 0x7f34c80a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:17.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.815+0000 7f34e1600700 1 -- 172.21.15.31:0/3701140618 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f34d40cf880 con 0x7f34e406a380 2024-03-21T10:35:17.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.815+0000 7f34db400700 1 --2- 172.21.15.31:0/3701140618 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f34c80a3030 0x7f34c80a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:17.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.815+0000 7f34db400700 1 --2- 172.21.15.31:0/3701140618 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f34c80a3030 0x7f34c80a54f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f34cc00b2e0 tx=0x7f34cc01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:17.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:17.818+0000 7f34e1600700 1 -- 172.21.15.31:0/3701140618 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f34d409c410 con 0x7f34e406a380 2024-03-21T10:35:18.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:18.099+0000 7f34eaa00700 1 -- 172.21.15.31:0/3701140618 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f34e41abd30 con 0x7f34e406a380 2024-03-21T10:35:18.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:18.099+0000 7f34e1600700 1 -- 172.21.15.31:0/3701140618 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7f34d409bd10 con 0x7f34e406a380 2024-03-21T10:35:18.098 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-21T10:35:18.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:18.103+0000 7f34eaa00700 1 -- 172.21.15.31:0/3701140618 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f34c80a3030 msgr2=0x7f34c80a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:18.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:18.103+0000 7f34eaa00700 1 --2- 172.21.15.31:0/3701140618 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f34c80a3030 0x7f34c80a54f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f34cc00b2e0 tx=0x7f34cc01a040 comp rx=0 tx=0).stop 2024-03-21T10:35:18.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:18.103+0000 7f34eaa00700 1 -- 172.21.15.31:0/3701140618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e406a380 msgr2=0x7f34e419ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:18.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:18.103+0000 7f34eaa00700 1 --2- 172.21.15.31:0/3701140618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e406a380 0x7f34e419ef10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f34d400e9a0 tx=0x7f34d400ed60 comp rx=0 tx=0).stop 2024-03-21T10:35:18.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:18.103+0000 7f34eaa00700 1 -- 172.21.15.31:0/3701140618 shutdown_connections 2024-03-21T10:35:18.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:18.103+0000 7f34eaa00700 1 --2- 172.21.15.31:0/3701140618 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f34c80a3030 0x7f34c80a54f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:18.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:18.103+0000 7f34eaa00700 1 --2- 172.21.15.31:0/3701140618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34e4107080 0x7f34e41a37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:18.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:18.103+0000 7f34eaa00700 1 --2- 172.21.15.31:0/3701140618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e406a380 0x7f34e419ef10 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:18.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:18.103+0000 7f34eaa00700 1 --2- 172.21.15.31:0/3701140618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34e4104b90 0x7f34e419f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:18.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:18.103+0000 7f34eaa00700 1 -- 172.21.15.31:0/3701140618 >> 172.21.15.31:0/3701140618 conn(0x7f34e4077fc0 msgr2=0x7f34e4103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:18.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:18.103+0000 7f34eaa00700 1 -- 172.21.15.31:0/3701140618 shutdown_connections 2024-03-21T10:35:18.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:18.103+0000 7f34eaa00700 1 -- 172.21.15.31:0/3701140618 wait complete. 2024-03-21T10:35:18.501 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:18 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3701140618' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-21T10:35:18.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:18 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3701140618' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-21T10:35:18.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:18 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3701140618' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-21T10:35:19.068 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-21T10:35:19.068 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-21T10:35:19.068 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi031.front.sepia.ceph.com 2024-03-21T10:35:19.068 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-21T10:35:19.068 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-21T10:35:19.068 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-21T10:35:19.068 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-21T10:35:19.068 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-21T10:35:19.068 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-21T10:35:19.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:35:19.069 DEBUG:teuthology.orchestra.run.smithi031:> ip addr 2024-03-21T10:35:19.128 INFO:teuthology.orchestra.run.smithi031.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-21T10:35:19.128 INFO:teuthology.orchestra.run.smithi031.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-21T10:35:19.128 INFO:teuthology.orchestra.run.smithi031.stdout: inet 127.0.0.1/8 scope host lo 2024-03-21T10:35:19.128 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-03-21T10:35:19.128 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 ::1/128 scope host 2024-03-21T10:35:19.128 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-03-21T10:35:19.128 INFO:teuthology.orchestra.run.smithi031.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-21T10:35:19.129 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cc brd ff:ff:ff:ff:ff:ff 2024-03-21T10:35:19.129 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f0 2024-03-21T10:35:19.129 INFO:teuthology.orchestra.run.smithi031.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-21T10:35:19.129 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cd brd ff:ff:ff:ff:ff:ff 2024-03-21T10:35:19.129 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f1 2024-03-21T10:35:19.129 INFO:teuthology.orchestra.run.smithi031.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-21T10:35:19.129 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:ba brd ff:ff:ff:ff:ff:ff 2024-03-21T10:35:19.129 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f0 2024-03-21T10:35:19.129 INFO:teuthology.orchestra.run.smithi031.stdout: inet 172.21.15.31/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-21T10:35:19.129 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft 42015sec preferred_lft 42015sec 2024-03-21T10:35:19.129 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::ae1f:6bff:fea5:caba/64 scope link noprefixroute 2024-03-21T10:35:19.129 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-03-21T10:35:19.129 INFO:teuthology.orchestra.run.smithi031.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-21T10:35:19.129 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:bb brd ff:ff:ff:ff:ff:ff 2024-03-21T10:35:19.130 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f1 2024-03-21T10:35:19.131 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-21T10:35:19.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:35:19.131 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec bash -c ' 2024-03-21T10:35:19.131 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-03-21T10:35:19.131 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link add name ceph-brx type bridge 2024-03-21T10:35:19.131 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip addr flush dev ceph-brx 2024-03-21T10:35:19.131 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set ceph-brx up 2024-03-21T10:35:19.131 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-21T10:35:19.131 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-03-21T10:35:19.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:19 smithi031 ceph-mon[31673]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 391 B/s wr, 0 op/s 2024-03-21T10:35:19.498 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:19 smithi070 ceph-mon[36330]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 391 B/s wr, 0 op/s 2024-03-21T10:35:19.582 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:19 smithi040 ceph-mon[36154]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 391 B/s wr, 0 op/s 2024-03-21T10:35:19.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:35:19.594 DEBUG:teuthology.orchestra.run.smithi031:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-21T10:35:19.678 INFO:teuthology.orchestra.run.smithi031.stdout:1 2024-03-21T10:35:19.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:35:19.682 DEBUG:teuthology.orchestra.run.smithi031:> route 2024-03-21T10:35:19.742 INFO:teuthology.orchestra.run.smithi031.stdout:Kernel IP routing table 2024-03-21T10:35:19.743 INFO:teuthology.orchestra.run.smithi031.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-21T10:35:19.743 INFO:teuthology.orchestra.run.smithi031.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-21T10:35:19.743 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-21T10:35:19.743 INFO:teuthology.orchestra.run.smithi031.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-21T10:35:19.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:35:19.743 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec bash -c ' 2024-03-21T10:35:19.744 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-03-21T10:35:19.744 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-21T10:35:19.744 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-21T10:35:19.744 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-21T10:35:19.744 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-03-21T10:35:20.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:35:20.476 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-03-21T10:35:20.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:35:20.533 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list-id 2024-03-21T10:35:20.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:35:20.587 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec bash -c ' 2024-03-21T10:35:20.587 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-03-21T10:35:20.587 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-21T10:35:20.587 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-21T10:35:20.587 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-03-21T10:35:20.744 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-21T10:35:20.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:35:20.745 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec bash -c ' 2024-03-21T10:35:20.745 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-03-21T10:35:20.745 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-21T10:35:20.745 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-21T10:35:20.745 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-21T10:35:20.745 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-21T10:35:20.745 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-21T10:35:20.745 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-03-21T10:35:21.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:35:21.078 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec bash -c ' 2024-03-21T10:35:21.078 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-03-21T10:35:21.078 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set brx.0 up 2024-03-21T10:35:21.078 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set dev brx.0 master ceph-brx 2024-03-21T10:35:21.078 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-03-21T10:35:21.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:21 smithi031 ceph-mon[31673]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:35:21.256 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-21T10:35:21.256 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-21T10:35:21.256 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-21T10:35:21.305 INFO:teuthology.orchestra.run.smithi031.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-21T10:35:21.305 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-21T10:35:21.306 DEBUG:teuthology.orchestra.run.smithi031:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-21T10:35:21.403 DEBUG:teuthology.orchestra.run.smithi031:> sudo modprobe fuse 2024-03-21T10:35:21.504 DEBUG:teuthology.orchestra.run.smithi031:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T10:35:21.557 INFO:teuthology.orchestra.run.smithi031.stdout:/proc 2024-03-21T10:35:21.557 INFO:teuthology.orchestra.run.smithi031.stdout:/sys 2024-03-21T10:35:21.557 INFO:teuthology.orchestra.run.smithi031.stdout:/dev 2024-03-21T10:35:21.558 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/security 2024-03-21T10:35:21.558 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/shm 2024-03-21T10:35:21.558 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/pts 2024-03-21T10:35:21.558 INFO:teuthology.orchestra.run.smithi031.stdout:/run 2024-03-21T10:35:21.558 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/cgroup 2024-03-21T10:35:21.558 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/pstore 2024-03-21T10:35:21.558 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/bpf 2024-03-21T10:35:21.558 INFO:teuthology.orchestra.run.smithi031.stdout:/ 2024-03-21T10:35:21.558 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/selinux 2024-03-21T10:35:21.558 INFO:teuthology.orchestra.run.smithi031.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T10:35:21.558 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/mqueue 2024-03-21T10:35:21.558 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/hugepages 2024-03-21T10:35:21.559 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/debug 2024-03-21T10:35:21.559 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/tracing 2024-03-21T10:35:21.559 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/fuse/connections 2024-03-21T10:35:21.559 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/config 2024-03-21T10:35:21.559 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-sysctl.service 2024-03-21T10:35:21.559 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-21T10:35:21.559 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-21T10:35:21.559 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/nfs/rpc_pipefs 2024-03-21T10:35:21.559 INFO:teuthology.orchestra.run.smithi031.stdout:/run/user/1000 2024-03-21T10:35:21.559 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/ceph 2024-03-21T10:35:21.559 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay 2024-03-21T10:35:21.559 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/d9342e8164ea0a452b977b26a8e56bf21aeb53cc612745a6eb73d24f4ab2f769/merged 2024-03-21T10:35:21.559 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/9247b39d0001ff7e4dbf44c8d1966bd0e79fa73b34d55b886e6c4a796fedef66/merged 2024-03-21T10:35:21.559 INFO:teuthology.orchestra.run.smithi031.stdout:/run/user/0 2024-03-21T10:35:21.559 INFO:teuthology.orchestra.run.smithi031.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T10:35:21.560 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/975a59ac572327c093d339c1bf25ef324963c659ffefe6324088de5cf2efdb3b/merged 2024-03-21T10:35:21.560 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/a3884878edddb878b1479fa6630df4333b74eb73144b17e6ce084f0dfc4ea1ac/merged 2024-03-21T10:35:21.560 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/4aedf70b4ba6440dc92b8065dbbe13af01488fb62b9685b7c42174089cce5466/merged 2024-03-21T10:35:21.560 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/bc11052b2ad322052d9f19911abd4a67e23671f69e82647aadb0c15d9186ac3d/merged 2024-03-21T10:35:21.560 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/b85511635286942e590fece9c360ae71e3ea9dc879b911c5e08ad973fc680628/merged 2024-03-21T10:35:21.560 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/3246b7742b9b0345a83ddaacb54f083fda37995442363d76221d55ff216310b2/merged 2024-03-21T10:35:21.561 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/2427546b8b9f9c6440362b99bf3c61dafa15176625c0d895a2dca4f2fc30a9ad/merged 2024-03-21T10:35:21.561 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/e0bfd9558d815190e41b679c267e13159368747f35d66717cb07ca95b82ab9e3/merged 2024-03-21T10:35:21.561 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns 2024-03-21T10:35:21.561 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-21T10:35:21.561 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-21T10:35:21.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:35:21.562 DEBUG:teuthology.orchestra.run.smithi031:> ls /sys/fs/fuse/connections 2024-03-21T10:35:21.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:21 smithi070 ceph-mon[36330]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:35:21.615 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-21T10:35:21.615 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-21T10:35:21.620 DEBUG:teuthology.orchestra.run.smithi031:> sudo modprobe fuse 2024-03-21T10:35:21.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:21 smithi040 ceph-mon[36154]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:35:21.695 DEBUG:teuthology.orchestra.run.smithi031:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T10:35:21.738 INFO:teuthology.orchestra.run.smithi031.stdout:/proc 2024-03-21T10:35:21.739 INFO:teuthology.orchestra.run.smithi031.stdout:/sys 2024-03-21T10:35:21.739 INFO:teuthology.orchestra.run.smithi031.stdout:/dev 2024-03-21T10:35:21.739 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/security 2024-03-21T10:35:21.739 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/shm 2024-03-21T10:35:21.739 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/pts 2024-03-21T10:35:21.739 INFO:teuthology.orchestra.run.smithi031.stdout:/run 2024-03-21T10:35:21.739 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/cgroup 2024-03-21T10:35:21.739 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/pstore 2024-03-21T10:35:21.739 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/bpf 2024-03-21T10:35:21.739 INFO:teuthology.orchestra.run.smithi031.stdout:/ 2024-03-21T10:35:21.739 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/selinux 2024-03-21T10:35:21.739 INFO:teuthology.orchestra.run.smithi031.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T10:35:21.739 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/mqueue 2024-03-21T10:35:21.739 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/hugepages 2024-03-21T10:35:21.740 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/debug 2024-03-21T10:35:21.740 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/tracing 2024-03-21T10:35:21.740 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/fuse/connections 2024-03-21T10:35:21.740 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/config 2024-03-21T10:35:21.740 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-sysctl.service 2024-03-21T10:35:21.740 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-21T10:35:21.740 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-21T10:35:21.740 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/nfs/rpc_pipefs 2024-03-21T10:35:21.740 INFO:teuthology.orchestra.run.smithi031.stdout:/run/user/1000 2024-03-21T10:35:21.740 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/ceph 2024-03-21T10:35:21.740 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay 2024-03-21T10:35:21.741 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/d9342e8164ea0a452b977b26a8e56bf21aeb53cc612745a6eb73d24f4ab2f769/merged 2024-03-21T10:35:21.741 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/9247b39d0001ff7e4dbf44c8d1966bd0e79fa73b34d55b886e6c4a796fedef66/merged 2024-03-21T10:35:21.741 INFO:teuthology.orchestra.run.smithi031.stdout:/run/user/0 2024-03-21T10:35:21.741 INFO:teuthology.orchestra.run.smithi031.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T10:35:21.741 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/975a59ac572327c093d339c1bf25ef324963c659ffefe6324088de5cf2efdb3b/merged 2024-03-21T10:35:21.741 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/a3884878edddb878b1479fa6630df4333b74eb73144b17e6ce084f0dfc4ea1ac/merged 2024-03-21T10:35:21.741 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/4aedf70b4ba6440dc92b8065dbbe13af01488fb62b9685b7c42174089cce5466/merged 2024-03-21T10:35:21.741 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/bc11052b2ad322052d9f19911abd4a67e23671f69e82647aadb0c15d9186ac3d/merged 2024-03-21T10:35:21.741 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/b85511635286942e590fece9c360ae71e3ea9dc879b911c5e08ad973fc680628/merged 2024-03-21T10:35:21.741 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/3246b7742b9b0345a83ddaacb54f083fda37995442363d76221d55ff216310b2/merged 2024-03-21T10:35:21.741 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/2427546b8b9f9c6440362b99bf3c61dafa15176625c0d895a2dca4f2fc30a9ad/merged 2024-03-21T10:35:21.741 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/e0bfd9558d815190e41b679c267e13159368747f35d66717cb07ca95b82ab9e3/merged 2024-03-21T10:35:21.741 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns 2024-03-21T10:35:21.741 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-21T10:35:21.742 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-21T10:35:21.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:35:21.743 DEBUG:teuthology.orchestra.run.smithi031:> ls /sys/fs/fuse/connections 2024-03-21T10:35:21.827 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi031.stderr:2024-03-21T10:35:21.829+0000 7fba1ddfd480 -1 init, newargv = 0x55beb70bc560 newargc=15 2024-03-21T10:35:21.827 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi031.stderr:ceph-fuse[74852]: starting ceph client 2024-03-21T10:35:21.837 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi031.stderr:ceph-fuse[74852]: starting fuse 2024-03-21T10:35:22.936 DEBUG:teuthology.orchestra.run.smithi031:> sudo modprobe fuse 2024-03-21T10:35:23.043 DEBUG:teuthology.orchestra.run.smithi031:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T10:35:23.100 INFO:teuthology.orchestra.run.smithi031.stdout:/proc 2024-03-21T10:35:23.100 INFO:teuthology.orchestra.run.smithi031.stdout:/sys 2024-03-21T10:35:23.100 INFO:teuthology.orchestra.run.smithi031.stdout:/dev 2024-03-21T10:35:23.100 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/security 2024-03-21T10:35:23.100 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/shm 2024-03-21T10:35:23.100 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/pts 2024-03-21T10:35:23.100 INFO:teuthology.orchestra.run.smithi031.stdout:/run 2024-03-21T10:35:23.100 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/cgroup 2024-03-21T10:35:23.100 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/pstore 2024-03-21T10:35:23.101 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/bpf 2024-03-21T10:35:23.101 INFO:teuthology.orchestra.run.smithi031.stdout:/ 2024-03-21T10:35:23.101 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/selinux 2024-03-21T10:35:23.101 INFO:teuthology.orchestra.run.smithi031.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T10:35:23.101 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/mqueue 2024-03-21T10:35:23.101 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/hugepages 2024-03-21T10:35:23.101 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/debug 2024-03-21T10:35:23.101 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/tracing 2024-03-21T10:35:23.101 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/fuse/connections 2024-03-21T10:35:23.101 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/config 2024-03-21T10:35:23.101 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-sysctl.service 2024-03-21T10:35:23.101 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-21T10:35:23.101 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-21T10:35:23.101 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/nfs/rpc_pipefs 2024-03-21T10:35:23.101 INFO:teuthology.orchestra.run.smithi031.stdout:/run/user/1000 2024-03-21T10:35:23.102 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/ceph 2024-03-21T10:35:23.102 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay 2024-03-21T10:35:23.102 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/d9342e8164ea0a452b977b26a8e56bf21aeb53cc612745a6eb73d24f4ab2f769/merged 2024-03-21T10:35:23.102 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/9247b39d0001ff7e4dbf44c8d1966bd0e79fa73b34d55b886e6c4a796fedef66/merged 2024-03-21T10:35:23.102 INFO:teuthology.orchestra.run.smithi031.stdout:/run/user/0 2024-03-21T10:35:23.102 INFO:teuthology.orchestra.run.smithi031.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T10:35:23.102 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/975a59ac572327c093d339c1bf25ef324963c659ffefe6324088de5cf2efdb3b/merged 2024-03-21T10:35:23.102 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/a3884878edddb878b1479fa6630df4333b74eb73144b17e6ce084f0dfc4ea1ac/merged 2024-03-21T10:35:23.102 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/4aedf70b4ba6440dc92b8065dbbe13af01488fb62b9685b7c42174089cce5466/merged 2024-03-21T10:35:23.102 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/bc11052b2ad322052d9f19911abd4a67e23671f69e82647aadb0c15d9186ac3d/merged 2024-03-21T10:35:23.102 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/b85511635286942e590fece9c360ae71e3ea9dc879b911c5e08ad973fc680628/merged 2024-03-21T10:35:23.102 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/3246b7742b9b0345a83ddaacb54f083fda37995442363d76221d55ff216310b2/merged 2024-03-21T10:35:23.102 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/2427546b8b9f9c6440362b99bf3c61dafa15176625c0d895a2dca4f2fc30a9ad/merged 2024-03-21T10:35:23.102 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/e0bfd9558d815190e41b679c267e13159368747f35d66717cb07ca95b82ab9e3/merged 2024-03-21T10:35:23.103 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns 2024-03-21T10:35:23.103 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-21T10:35:23.103 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-21T10:35:23.103 INFO:teuthology.orchestra.run.smithi031.stdout:/home/ubuntu/cephtest/mnt.admin 2024-03-21T10:35:23.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:35:23.104 DEBUG:teuthology.orchestra.run.smithi031:> ls /sys/fs/fuse/connections 2024-03-21T10:35:23.160 INFO:teuthology.orchestra.run.smithi031.stdout:118 2024-03-21T10:35:23.160 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [118] 2024-03-21T10:35:23.160 DEBUG:teuthology.orchestra.run.smithi031:> sudo stdin-killer -- python3 -c ' 2024-03-21T10:35:23.160 DEBUG:teuthology.orchestra.run.smithi031:> import glob 2024-03-21T10:35:23.160 DEBUG:teuthology.orchestra.run.smithi031:> import re 2024-03-21T10:35:23.160 DEBUG:teuthology.orchestra.run.smithi031:> import os 2024-03-21T10:35:23.160 DEBUG:teuthology.orchestra.run.smithi031:> import subprocess 2024-03-21T10:35:23.160 DEBUG:teuthology.orchestra.run.smithi031:> 2024-03-21T10:35:23.160 DEBUG:teuthology.orchestra.run.smithi031:> def _find_admin_socket(client_name): 2024-03-21T10:35:23.161 DEBUG:teuthology.orchestra.run.smithi031:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-21T10:35:23.161 DEBUG:teuthology.orchestra.run.smithi031:> files = glob.glob(asok_path) 2024-03-21T10:35:23.161 DEBUG:teuthology.orchestra.run.smithi031:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-21T10:35:23.161 DEBUG:teuthology.orchestra.run.smithi031:> 2024-03-21T10:35:23.161 DEBUG:teuthology.orchestra.run.smithi031:> # Given a non-glob path, it better be there 2024-03-21T10:35:23.161 DEBUG:teuthology.orchestra.run.smithi031:> if "*" not in asok_path: 2024-03-21T10:35:23.161 DEBUG:teuthology.orchestra.run.smithi031:> assert(len(files) == 1) 2024-03-21T10:35:23.161 DEBUG:teuthology.orchestra.run.smithi031:> return files[0] 2024-03-21T10:35:23.161 DEBUG:teuthology.orchestra.run.smithi031:> 2024-03-21T10:35:23.161 DEBUG:teuthology.orchestra.run.smithi031:> for f in files: 2024-03-21T10:35:23.161 DEBUG:teuthology.orchestra.run.smithi031:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-21T10:35:23.161 DEBUG:teuthology.orchestra.run.smithi031:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-21T10:35:23.161 DEBUG:teuthology.orchestra.run.smithi031:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-21T10:35:23.161 DEBUG:teuthology.orchestra.run.smithi031:> contents = proc_f.read() 2024-03-21T10:35:23.161 DEBUG:teuthology.orchestra.run.smithi031:> if mountpoint in contents: 2024-03-21T10:35:23.161 DEBUG:teuthology.orchestra.run.smithi031:> return f 2024-03-21T10:35:23.162 DEBUG:teuthology.orchestra.run.smithi031:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-21T10:35:23.162 DEBUG:teuthology.orchestra.run.smithi031:> 2024-03-21T10:35:23.162 DEBUG:teuthology.orchestra.run.smithi031:> print(_find_admin_socket("client.admin")) 2024-03-21T10:35:23.162 DEBUG:teuthology.orchestra.run.smithi031:> ' 2024-03-21T10:35:23.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:23 smithi031 ceph-mon[31673]: pgmap v333: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:35:23.433 INFO:teuthology.orchestra.run.smithi031.stdout:/var/run/ceph/ceph-client.admin.74852.asok 2024-03-21T10:35:23.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-21T10:35:23.436 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.74852.asok 2024-03-21T10:35:23.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:35:23.437 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.74852.asok status 2024-03-21T10:35:23.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:23 smithi070 ceph-mon[36330]: pgmap v333: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:35:23.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:23 smithi040 ceph-mon[36154]: pgmap v333: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:35:23.689 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:35:23.689 INFO:teuthology.orchestra.run.smithi031.stdout: "metadata": { 2024-03-21T10:35:23.689 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_sha1": "2edfee68ad7243bed422a049b6f5f58619300f11", 2024-03-21T10:35:23.689 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_version": "ceph version 19.0.0-2231-g2edfee68 (2edfee68ad7243bed422a049b6f5f58619300f11) squid (dev)", 2024-03-21T10:35:23.689 INFO:teuthology.orchestra.run.smithi031.stdout: "entity_id": "admin", 2024-03-21T10:35:23.689 INFO:teuthology.orchestra.run.smithi031.stdout: "hostname": "smithi031", 2024-03-21T10:35:23.690 INFO:teuthology.orchestra.run.smithi031.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-21T10:35:23.690 INFO:teuthology.orchestra.run.smithi031.stdout: "pid": "74852", 2024-03-21T10:35:23.690 INFO:teuthology.orchestra.run.smithi031.stdout: "root": "/" 2024-03-21T10:35:23.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:35:23.690 INFO:teuthology.orchestra.run.smithi031.stdout: "dentry_count": 0, 2024-03-21T10:35:23.690 INFO:teuthology.orchestra.run.smithi031.stdout: "dentry_pinned_count": 0, 2024-03-21T10:35:23.690 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24722, 2024-03-21T10:35:23.690 INFO:teuthology.orchestra.run.smithi031.stdout: "inst": { 2024-03-21T10:35:23.690 INFO:teuthology.orchestra.run.smithi031.stdout: "name": { 2024-03-21T10:35:23.690 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client", 2024-03-21T10:35:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "num": 24722 2024-03-21T10:35:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:35:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": { 2024-03-21T10:35:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-03-21T10:35:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2024-03-21T10:35:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3050101777 2024-03-21T10:35:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:35:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:35:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": { 2024-03-21T10:35:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-03-21T10:35:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2024-03-21T10:35:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3050101777 2024-03-21T10:35:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:35:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "inst_str": "client.24722 172.21.15.31:0/3050101777", 2024-03-21T10:35:23.692 INFO:teuthology.orchestra.run.smithi031.stdout: "addr_str": "172.21.15.31:0/3050101777", 2024-03-21T10:35:23.692 INFO:teuthology.orchestra.run.smithi031.stdout: "inode_count": 1, 2024-03-21T10:35:23.692 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_epoch": 17, 2024-03-21T10:35:23.692 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_epoch": 89, 2024-03-21T10:35:23.692 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_epoch_barrier": 0, 2024-03-21T10:35:23.692 INFO:teuthology.orchestra.run.smithi031.stdout: "blocklisted": false, 2024-03-21T10:35:23.692 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs" 2024-03-21T10:35:23.692 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:35:23.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:35:23.707 DEBUG:teuthology.orchestra.run.smithi031:> stat --file-system '--printf=%T 2024-03-21T10:35:23.707 DEBUG:teuthology.orchestra.run.smithi031:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-21T10:35:23.762 INFO:teuthology.orchestra.run.smithi031.stdout:fuseblk 2024-03-21T10:35:23.762 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-21T10:35:23.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:35:23.762 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-21T10:35:23.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:35:23.862 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-21T10:35:23.959 INFO:teuthology.orchestra.run.smithi031.stdout:# file: . 2024-03-21T10:35:23.960 INFO:teuthology.orchestra.run.smithi031.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-21T10:35:23.960 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:35:23.961 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-21T10:35:23.961 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T10:35:24.018 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T10:35:24.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.255+0000 7fb02d800640 1 -- 172.21.15.31:0/1963004300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb028077c40 msgr2=0x7fb0280784e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:24.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.255+0000 7fb02d800640 1 --2- 172.21.15.31:0/1963004300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb028077c40 0x7fb0280784e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb020009ca0 tx=0x7fb02002c100 comp rx=0 tx=0).stop 2024-03-21T10:35:24.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.255+0000 7fb02d800640 1 -- 172.21.15.31:0/1963004300 shutdown_connections 2024-03-21T10:35:24.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.255+0000 7fb02d800640 1 --2- 172.21.15.31:0/1963004300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb028077c40 0x7fb0280784e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:24.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.255+0000 7fb02d800640 1 --2- 172.21.15.31:0/1963004300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0280772a0 0x7fb028077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:24.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.255+0000 7fb02d800640 1 --2- 172.21.15.31:0/1963004300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb028076060 0x7fb028076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:24.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.255+0000 7fb02d800640 1 -- 172.21.15.31:0/1963004300 >> 172.21.15.31:0/1963004300 conn(0x7fb0280fed50 msgr2=0x7fb028101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:24.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.255+0000 7fb02d800640 1 -- 172.21.15.31:0/1963004300 shutdown_connections 2024-03-21T10:35:24.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.255+0000 7fb02d800640 1 -- 172.21.15.31:0/1963004300 wait complete. 2024-03-21T10:35:24.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.256+0000 7fb02d800640 1 Processor -- start 2024-03-21T10:35:24.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.256+0000 7fb02d800640 1 -- start start 2024-03-21T10:35:24.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.256+0000 7fb02d800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb028076060 0x7fb02819de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:24.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.256+0000 7fb02d800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0280772a0 0x7fb02819e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:24.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.256+0000 7fb02d800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb028077c40 0x7fb0281a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:24.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.256+0000 7fb02d800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb028113830 con 0x7fb028077c40 2024-03-21T10:35:24.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.256+0000 7fb02d800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0281136b0 con 0x7fb028076060 2024-03-21T10:35:24.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.256+0000 7fb02d800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0281139b0 con 0x7fb0280772a0 2024-03-21T10:35:24.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.257+0000 7fb027e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb028076060 0x7fb02819de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:24.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.257+0000 7fb027e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb028076060 0x7fb02819de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:45446/0 (socket says 172.21.15.31:45446) 2024-03-21T10:35:24.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.257+0000 7fb027e00640 1 -- 172.21.15.31:0/558511371 learned_addr learned my addr 172.21.15.31:0/558511371 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:24.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.257+0000 7fb02ce00640 1 --2- 172.21.15.31:0/558511371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb028077c40 0x7fb0281a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:24.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.257+0000 7fb01fe00640 1 --2- 172.21.15.31:0/558511371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0280772a0 0x7fb02819e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:24.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.257+0000 7fb027e00640 1 -- 172.21.15.31:0/558511371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0280772a0 msgr2=0x7fb02819e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:24.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.257+0000 7fb027e00640 1 --2- 172.21.15.31:0/558511371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0280772a0 0x7fb02819e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:24.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.257+0000 7fb027e00640 1 -- 172.21.15.31:0/558511371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb028077c40 msgr2=0x7fb0281a53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:24.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.257+0000 7fb027e00640 1 --2- 172.21.15.31:0/558511371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb028077c40 0x7fb0281a53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:24.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.257+0000 7fb027e00640 1 -- 172.21.15.31:0/558511371 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0281a5a50 con 0x7fb028076060 2024-03-21T10:35:24.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.257+0000 7fb027e00640 1 --2- 172.21.15.31:0/558511371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb028076060 0x7fb02819de30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb01800e9f0 tx=0x7fb01800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:24.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.257+0000 7fb025600640 1 -- 172.21.15.31:0/558511371 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01800cd80 con 0x7fb028076060 2024-03-21T10:35:24.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.258+0000 7fb025600640 1 -- 172.21.15.31:0/558511371 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb018010910 con 0x7fb028076060 2024-03-21T10:35:24.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.258+0000 7fb02d800640 1 -- 172.21.15.31:0/558511371 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0281a5d40 con 0x7fb028076060 2024-03-21T10:35:24.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.258+0000 7fb025600640 1 -- 172.21.15.31:0/558511371 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb018018990 con 0x7fb028076060 2024-03-21T10:35:24.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.258+0000 7fb02d800640 1 -- 172.21.15.31:0/558511371 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0281a6220 con 0x7fb028076060 2024-03-21T10:35:24.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.259+0000 7fb02d800640 1 -- 172.21.15.31:0/558511371 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb02804f470 con 0x7fb028076060 2024-03-21T10:35:24.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.260+0000 7fb025600640 1 -- 172.21.15.31:0/558511371 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fb018010ac0 con 0x7fb028076060 2024-03-21T10:35:24.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.262+0000 7fb025600640 1 --2- 172.21.15.31:0/558511371 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb0080a2f90 0x7fb0080a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:24.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.262+0000 7fb025600640 1 -- 172.21.15.31:0/558511371 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb0180cf8f0 con 0x7fb028076060 2024-03-21T10:35:24.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.262+0000 7fb01fe00640 1 --2- 172.21.15.31:0/558511371 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb0080a2f90 0x7fb0080a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:24.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.263+0000 7fb01fe00640 1 --2- 172.21.15.31:0/558511371 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb0080a2f90 0x7fb0080a5450 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb02819f340 tx=0x7fb010005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:24.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.266+0000 7fb025600640 1 -- 172.21.15.31:0/558511371 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb01809be30 con 0x7fb028076060 2024-03-21T10:35:24.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.469+0000 7fb02d800640 1 -- 172.21.15.31:0/558511371 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb028110110 con 0x7fb028076060 2024-03-21T10:35:24.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.471+0000 7fb025600640 1 -- 172.21.15.31:0/558511371 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24009 (secure 0 0 0) 0x7fb01809fed0 con 0x7fb028076060 2024-03-21T10:35:24.470 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:35:24.470 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","created":"2024-03-21T10:25:41.971117+0000","modified":"2024-03-21T10:35:11.657050+0000","last_up_change":"2024-03-21T10:34:07.757196+0000","last_in_change":"2024-03-21T10:33:47.460406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T10:29:44.553968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T10:35:04.945449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T10:35:05.970589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T10:35:09.688711+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e0244d33-00e2-47d5-9f66-34efc138c462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6803","nonce":3089885125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6805","nonce":3089885125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6809","nonce":3089885125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6807","nonce":3089885125}]},"public_addr":"172.21.15.31:6803/3089885125","cluster_addr":"172.21.15.31:6805/3089885125","heartbeat_back_addr":"172.21.15.31:6809/3089885125","heartbeat_front_addr":"172.21.15.31:6807/3089885125","state":["exists","up"]},{"osd":1,"uuid":"76a63504-beab-4354-a6c3-eebc3b16ad03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6801","nonce":618840037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6803","nonce":618840037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6807","nonce":618840037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6805","nonce":618840037}]},"public_addr":"172.21.15.40:6801/618840037","cluster_addr":"172.21.15.40:6803/618840037","heartbeat_back_addr":"172.21.15.40:6807/618840037","heartbeat_front_addr":"172.21.15.40:6805/618840037","state":["exists","up"]},{"osd":2,"uuid":"fa191bd6-9a21-4ee2-8adf-8a63c9cfd821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6801","nonce":2524184117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6803","nonce":2524184117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6807","nonce":2524184117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6805","nonce":2524184117}]},"public_addr":"172.21.15.70:6801/2524184117","cluster_addr":"172.21.15.70:6803/2524184117","heartbeat_back_addr":"172.21.15.70:6807/2524184117","heartbeat_front_addr":"172.21.15.70:6805/2524184117","state":["exists","up"]},{"osd":3,"uuid":"edb44507-ffb0-42a4-827b-08461e1bf952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6811","nonce":2823036296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6813","nonce":2823036296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6817","nonce":2823036296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6815","nonce":2823036296}]},"public_addr":"172.21.15.31:6811/2823036296","cluster_addr":"172.21.15.31:6813/2823036296","heartbeat_back_addr":"172.21.15.31:6817/2823036296","heartbeat_front_addr":"172.21.15.31:6815/2823036296","state":["exists","up"]},{"osd":4,"uuid":"8cb89153-8c3b-431f-b57c-8863fada76dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6809","nonce":4215186309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6811","nonce":4215186309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6815","nonce":4215186309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6813","nonce":4215186309}]},"public_addr":"172.21.15.40:6809/4215186309","cluster_addr":"172.21.15.40:6811/4215186309","heartbeat_back_addr":"172.21.15.40:6815/4215186309","heartbeat_front_addr":"172.21.15.40:6813/4215186309","state":["exists","up"]},{"osd":5,"uuid":"9409505a-d463-408c-a3cd-218380c04d68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6809","nonce":3402830452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6811","nonce":3402830452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6815","nonce":3402830452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6813","nonce":3402830452}]},"public_addr":"172.21.15.70:6809/3402830452","cluster_addr":"172.21.15.70:6811/3402830452","heartbeat_back_addr":"172.21.15.70:6815/3402830452","heartbeat_front_addr":"172.21.15.70:6813/3402830452","state":["exists","up"]},{"osd":6,"uuid":"f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6819","nonce":1193497058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6821","nonce":1193497058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6825","nonce":1193497058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6823","nonce":1193497058}]},"public_addr":"172.21.15.31:6819/1193497058","cluster_addr":"172.21.15.31:6821/1193497058","heartbeat_back_addr":"172.21.15.31:6825/1193497058","heartbeat_front_addr":"172.21.15.31:6823/1193497058","state":["exists","up"]},{"osd":7,"uuid":"db228a87-29a5-45d4-8fe6-a0a554262eb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6817","nonce":3825628119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6819","nonce":3825628119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6823","nonce":3825628119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6821","nonce":3825628119}]},"public_addr":"172.21.15.40:6817/3825628119","cluster_addr":"172.21.15.40:6819/3825628119","heartbeat_back_addr":"172.21.15.40:6823/3825628119","heartbeat_front_addr":"172.21.15.40:6821/3825628119","state":["exists","up"]},{"osd":8,"uuid":"59b9fde0-095d-4e0b-87da-ead37bd8ff1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6817","nonce":534880586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6819","nonce":534880586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6823","nonce":534880586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6821","nonce":534880586}]},"public_addr":"172.21.15.70:6817/534880586","cluster_addr":"172.21.15.70:6819/534880586","heartbeat_back_addr":"172.21.15.70:6823/534880586","heartbeat_front_addr":"172.21.15.70:6821/534880586","state":["exists","up"]},{"osd":9,"uuid":"77f47061-0c5f-4159-b276-e814935d6c1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6827","nonce":587047469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6829","nonce":587047469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6833","nonce":587047469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6831","nonce":587047469}]},"public_addr":"172.21.15.31:6827/587047469","cluster_addr":"172.21.15.31:6829/587047469","heartbeat_back_addr":"172.21.15.31:6833/587047469","heartbeat_front_addr":"172.21.15.31:6831/587047469","state":["exists","up"]},{"osd":10,"uuid":"8853dc43-7bd1-4c93-8063-f7982fd7da38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6825","nonce":512108711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6827","nonce":512108711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6831","nonce":512108711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6829","nonce":512108711}]},"public_addr":"172.21.15.40:6825/512108711","cluster_addr":"172.21.15.40:6827/512108711","heartbeat_back_addr":"172.21.15.40:6831/512108711","heartbeat_front_addr":"172.21.15.40:6829/512108711","state":["exists","up"]},{"osd":11,"uuid":"84b7c2de-c424-45a6-9583-60097e372a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6825","nonce":540305718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6827","nonce":540305718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6831","nonce":540305718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6829","nonce":540305718}]},"public_addr":"172.21.15.70:6825/540305718","cluster_addr":"172.21.15.70:6827/540305718","heartbeat_back_addr":"172.21.15.70:6831/540305718","heartbeat_front_addr":"172.21.15.70:6829/540305718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:28:51.477358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:17.405045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:41.626268+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:15.339568+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:42.676996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:07.516999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:42.412273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:10.365041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:35.699120+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:11.733639+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:39.505442+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:34:05.449301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451930554":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/3342588843":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6801/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/1626448389":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/1910059351":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/344822770":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/3811368512":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/1981437651":"2024-03-22T10:26:16.959165+0000","172.21.15.31:6801/1981437651":"2024-03-22T10:26:16.959165+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T10:35:24.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.474+0000 7fb02d800640 1 -- 172.21.15.31:0/558511371 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb0080a2f90 msgr2=0x7fb0080a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:24.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.474+0000 7fb02d800640 1 --2- 172.21.15.31:0/558511371 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb0080a2f90 0x7fb0080a5450 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb02819f340 tx=0x7fb010005f90 comp rx=0 tx=0).stop 2024-03-21T10:35:24.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.474+0000 7fb02d800640 1 -- 172.21.15.31:0/558511371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb028076060 msgr2=0x7fb02819de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:24.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.474+0000 7fb02d800640 1 --2- 172.21.15.31:0/558511371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb028076060 0x7fb02819de30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb01800e9f0 tx=0x7fb01800eeb0 comp rx=0 tx=0).stop 2024-03-21T10:35:24.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.474+0000 7fb02d800640 1 -- 172.21.15.31:0/558511371 shutdown_connections 2024-03-21T10:35:24.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.474+0000 7fb02d800640 1 --2- 172.21.15.31:0/558511371 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb0080a2f90 0x7fb0080a5450 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:24.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.474+0000 7fb02d800640 1 --2- 172.21.15.31:0/558511371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb028077c40 0x7fb0281a53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:24.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.474+0000 7fb02d800640 1 --2- 172.21.15.31:0/558511371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0280772a0 0x7fb02819e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:24.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.474+0000 7fb02d800640 1 --2- 172.21.15.31:0/558511371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb028076060 0x7fb02819de30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:24.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.474+0000 7fb02d800640 1 -- 172.21.15.31:0/558511371 >> 172.21.15.31:0/558511371 conn(0x7fb0280fed50 msgr2=0x7fb028100850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:24.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.474+0000 7fb02d800640 1 -- 172.21.15.31:0/558511371 shutdown_connections 2024-03-21T10:35:24.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.474+0000 7fb02d800640 1 -- 172.21.15.31:0/558511371 wait complete. 2024-03-21T10:35:24.493 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T10:29:44.553968+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T10:35:04.945449+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T10:35:05.970589+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T10:35:09.688711+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T10:35:24.494 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T10:35:24.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.729+0000 7f7983600640 1 -- 172.21.15.31:0/669418606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f797c1038c0 msgr2=0x7f797c103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:24.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.729+0000 7f7983600640 1 --2- 172.21.15.31:0/669418606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f797c1038c0 0x7f797c103ce0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7970009900 tx=0x7f797002bc30 comp rx=0 tx=0).stop 2024-03-21T10:35:24.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.729+0000 7f7983600640 1 -- 172.21.15.31:0/669418606 shutdown_connections 2024-03-21T10:35:24.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.729+0000 7f7983600640 1 --2- 172.21.15.31:0/669418606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f797c105470 0x7f797c10bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:24.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.729+0000 7f7983600640 1 --2- 172.21.15.31:0/669418606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f797c104ab0 0x7f797c104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:24.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.729+0000 7f7983600640 1 --2- 172.21.15.31:0/669418606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f797c1038c0 0x7f797c103ce0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:24.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.729+0000 7f7983600640 1 -- 172.21.15.31:0/669418606 >> 172.21.15.31:0/669418606 conn(0x7f797c0fee50 msgr2=0x7f797c101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:24.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.729+0000 7f7983600640 1 -- 172.21.15.31:0/669418606 shutdown_connections 2024-03-21T10:35:24.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.730+0000 7f7983600640 1 -- 172.21.15.31:0/669418606 wait complete. 2024-03-21T10:35:24.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.731+0000 7f7983600640 1 Processor -- start 2024-03-21T10:35:24.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.731+0000 7f7983600640 1 -- start start 2024-03-21T10:35:24.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.731+0000 7f7983600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f797c1038c0 0x7f797c199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:24.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.731+0000 7f7983600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f797c104ab0 0x7f797c19a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:24.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.731+0000 7f7983600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f797c105470 0x7f797c1a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:24.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.731+0000 7f7983600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797c10f560 con 0x7f797c1038c0 2024-03-21T10:35:24.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.731+0000 7f7983600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797c10f3e0 con 0x7f797c104ab0 2024-03-21T10:35:24.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.731+0000 7f7983600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797c10f6e0 con 0x7f797c105470 2024-03-21T10:35:24.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.732+0000 7f797be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f797c104ab0 0x7f797c19a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.732+0000 7f7980c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f797c1038c0 0x7f797c199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.732+0000 7f797be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f797c104ab0 0x7f797c19a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:45454/0 (socket says 172.21.15.31:45454) 2024-03-21T10:35:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.732+0000 7f797be00640 1 -- 172.21.15.31:0/339992753 learned_addr learned my addr 172.21.15.31:0/339992753 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.732+0000 7f7981600640 1 --2- 172.21.15.31:0/339992753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f797c105470 0x7f797c1a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.732+0000 7f797be00640 1 -- 172.21.15.31:0/339992753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f797c105470 msgr2=0x7f797c1a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.732+0000 7f797be00640 1 --2- 172.21.15.31:0/339992753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f797c105470 0x7f797c1a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.732+0000 7f797be00640 1 -- 172.21.15.31:0/339992753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f797c1038c0 msgr2=0x7f797c199bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.732+0000 7f797be00640 1 --2- 172.21.15.31:0/339992753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f797c1038c0 0x7f797c199bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.732+0000 7f797be00640 1 -- 172.21.15.31:0/339992753 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f797c1a17f0 con 0x7f797c104ab0 2024-03-21T10:35:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.732+0000 7f797be00640 1 --2- 172.21.15.31:0/339992753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f797c104ab0 0x7f797c19a110 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f796800e9f0 tx=0x7f796800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.732+0000 7f7979600640 1 -- 172.21.15.31:0/339992753 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796800cd80 con 0x7f797c104ab0 2024-03-21T10:35:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.733+0000 7f7979600640 1 -- 172.21.15.31:0/339992753 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7968010910 con 0x7f797c104ab0 2024-03-21T10:35:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.733+0000 7f7983600640 1 -- 172.21.15.31:0/339992753 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f797c1a1ae0 con 0x7f797c104ab0 2024-03-21T10:35:24.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.733+0000 7f7979600640 1 -- 172.21.15.31:0/339992753 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7968018990 con 0x7f797c104ab0 2024-03-21T10:35:24.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.733+0000 7f7983600640 1 -- 172.21.15.31:0/339992753 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f797c1a2080 con 0x7f797c104ab0 2024-03-21T10:35:24.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.734+0000 7f7983600640 1 -- 172.21.15.31:0/339992753 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f797c10abe0 con 0x7f797c104ab0 2024-03-21T10:35:24.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.736+0000 7f7979600640 1 -- 172.21.15.31:0/339992753 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f7968010ac0 con 0x7f797c104ab0 2024-03-21T10:35:24.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.737+0000 7f7979600640 1 --2- 172.21.15.31:0/339992753 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f79640a2fa0 0x7f79640a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:24.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.738+0000 7f7979600640 1 -- 172.21.15.31:0/339992753 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f79680cf9a0 con 0x7f797c104ab0 2024-03-21T10:35:24.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.738+0000 7f7980c00640 1 --2- 172.21.15.31:0/339992753 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f79640a2fa0 0x7f79640a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:24.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.739+0000 7f7980c00640 1 --2- 172.21.15.31:0/339992753 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f79640a2fa0 0x7f79640a5460 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f797002c170 tx=0x7f797002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:24.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.741+0000 7f7979600640 1 -- 172.21.15.31:0/339992753 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f796809c530 con 0x7f797c104ab0 2024-03-21T10:35:24.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.945+0000 7f7983600640 1 -- 172.21.15.31:0/339992753 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f797c0622a0 con 0x7f797c104ab0 2024-03-21T10:35:24.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.946+0000 7f7979600640 1 -- 172.21.15.31:0/339992753 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f796809be30 con 0x7f797c104ab0 2024-03-21T10:35:24.944 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-03-21T10:35:24.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.949+0000 7f7983600640 1 -- 172.21.15.31:0/339992753 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f79640a2fa0 msgr2=0x7f79640a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:24.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.949+0000 7f7983600640 1 --2- 172.21.15.31:0/339992753 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f79640a2fa0 0x7f79640a5460 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f797002c170 tx=0x7f797002df90 comp rx=0 tx=0).stop 2024-03-21T10:35:24.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.949+0000 7f7983600640 1 -- 172.21.15.31:0/339992753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f797c104ab0 msgr2=0x7f797c19a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:24.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.949+0000 7f7983600640 1 --2- 172.21.15.31:0/339992753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f797c104ab0 0x7f797c19a110 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f796800e9f0 tx=0x7f796800eeb0 comp rx=0 tx=0).stop 2024-03-21T10:35:24.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.949+0000 7f7983600640 1 -- 172.21.15.31:0/339992753 shutdown_connections 2024-03-21T10:35:24.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.949+0000 7f7983600640 1 --2- 172.21.15.31:0/339992753 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f79640a2fa0 0x7f79640a5460 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:24.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.949+0000 7f7983600640 1 --2- 172.21.15.31:0/339992753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f797c105470 0x7f797c1a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:24.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.949+0000 7f7983600640 1 --2- 172.21.15.31:0/339992753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f797c104ab0 0x7f797c19a110 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:24.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.949+0000 7f7983600640 1 --2- 172.21.15.31:0/339992753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f797c1038c0 0x7f797c199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:24.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.949+0000 7f7983600640 1 -- 172.21.15.31:0/339992753 >> 172.21.15.31:0/339992753 conn(0x7f797c0fee50 msgr2=0x7f797c107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:24.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.949+0000 7f7983600640 1 -- 172.21.15.31:0/339992753 shutdown_connections 2024-03-21T10:35:24.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:24.949+0000 7f7983600640 1 -- 172.21.15.31:0/339992753 wait complete. 2024-03-21T10:35:24.968 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T10:35:25.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.202+0000 7f2948e00640 1 -- 172.21.15.31:0/3174711242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29441038d0 msgr2=0x7f2944103cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:25.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.202+0000 7f2948e00640 1 --2- 172.21.15.31:0/3174711242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29441038d0 0x7f2944103cf0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f292c009900 tx=0x7f292c02bc40 comp rx=0 tx=0).stop 2024-03-21T10:35:25.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.203+0000 7f2948e00640 1 -- 172.21.15.31:0/3174711242 shutdown_connections 2024-03-21T10:35:25.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.203+0000 7f2948e00640 1 --2- 172.21.15.31:0/3174711242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2944105480 0x7f294410bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:25.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.203+0000 7f2948e00640 1 --2- 172.21.15.31:0/3174711242 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2944104ac0 0x7f2944104f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:25.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.203+0000 7f2948e00640 1 --2- 172.21.15.31:0/3174711242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29441038d0 0x7f2944103cf0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:25.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.203+0000 7f2948e00640 1 -- 172.21.15.31:0/3174711242 >> 172.21.15.31:0/3174711242 conn(0x7f29440fee60 msgr2=0x7f2944101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:25.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.203+0000 7f2948e00640 1 -- 172.21.15.31:0/3174711242 shutdown_connections 2024-03-21T10:35:25.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.203+0000 7f2948e00640 1 -- 172.21.15.31:0/3174711242 wait complete. 2024-03-21T10:35:25.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.204+0000 7f2948e00640 1 Processor -- start 2024-03-21T10:35:25.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.204+0000 7f2948e00640 1 -- start start 2024-03-21T10:35:25.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.204+0000 7f2948e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29441038d0 0x7f2944199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:25.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.204+0000 7f2948e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2944104ac0 0x7f294419a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:25.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.205+0000 7f2948e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2944105480 0x7f29441a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:25.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.205+0000 7f2948e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294410f570 con 0x7f2944104ac0 2024-03-21T10:35:25.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.205+0000 7f2948e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294410f3f0 con 0x7f2944105480 2024-03-21T10:35:25.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.205+0000 7f2948e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294410f6f0 con 0x7f29441038d0 2024-03-21T10:35:25.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.205+0000 7f2941400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2944104ac0 0x7f294419a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:25.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.205+0000 7f2941e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29441038d0 0x7f2944199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:25.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.205+0000 7f2942800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2944105480 0x7f29441a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:25.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.205+0000 7f2941e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29441038d0 0x7f2944199b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42444/0 (socket says 172.21.15.31:42444) 2024-03-21T10:35:25.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.205+0000 7f2941e00640 1 -- 172.21.15.31:0/1333132378 learned_addr learned my addr 172.21.15.31:0/1333132378 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:25.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.205+0000 7f2941e00640 1 -- 172.21.15.31:0/1333132378 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2944105480 msgr2=0x7f29441a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:25.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.205+0000 7f2941e00640 1 --2- 172.21.15.31:0/1333132378 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2944105480 0x7f29441a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:25.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.205+0000 7f2941e00640 1 -- 172.21.15.31:0/1333132378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2944104ac0 msgr2=0x7f294419a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:25.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.205+0000 7f2941e00640 1 --2- 172.21.15.31:0/1333132378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2944104ac0 0x7f294419a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:25.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.205+0000 7f2941e00640 1 -- 172.21.15.31:0/1333132378 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29441a17a0 con 0x7f29441038d0 2024-03-21T10:35:25.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.206+0000 7f2941e00640 1 --2- 172.21.15.31:0/1333132378 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29441038d0 0x7f2944199b80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f292c02efd0 tx=0x7f292c02c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:25.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.206+0000 7f293aa00640 1 -- 172.21.15.31:0/1333132378 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292c041070 con 0x7f29441038d0 2024-03-21T10:35:25.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.206+0000 7f2948e00640 1 -- 172.21.15.31:0/1333132378 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29441a1a30 con 0x7f29441038d0 2024-03-21T10:35:25.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.206+0000 7f293aa00640 1 -- 172.21.15.31:0/1333132378 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f292c004450 con 0x7f29441038d0 2024-03-21T10:35:25.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.206+0000 7f293aa00640 1 -- 172.21.15.31:0/1333132378 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292c003e50 con 0x7f29441038d0 2024-03-21T10:35:25.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.206+0000 7f2948e00640 1 -- 172.21.15.31:0/1333132378 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29441a1f40 con 0x7f29441038d0 2024-03-21T10:35:25.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.207+0000 7f2948e00640 1 -- 172.21.15.31:0/1333132378 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f294410abf0 con 0x7f29441038d0 2024-03-21T10:35:25.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.209+0000 7f293aa00640 1 -- 172.21.15.31:0/1333132378 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f292c044070 con 0x7f29441038d0 2024-03-21T10:35:25.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.211+0000 7f293aa00640 1 --2- 172.21.15.31:0/1333132378 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f29280a30c0 0x7f29280a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:25.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.211+0000 7f293aa00640 1 -- 172.21.15.31:0/1333132378 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f292c0f1580 con 0x7f29441038d0 2024-03-21T10:35:25.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.211+0000 7f2941400640 1 --2- 172.21.15.31:0/1333132378 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f29280a30c0 0x7f29280a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:25.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.212+0000 7f2941400640 1 --2- 172.21.15.31:0/1333132378 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f29280a30c0 0x7f29280a5580 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f294419b090 tx=0x7f2934009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:25.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.213+0000 7f293aa00640 1 -- 172.21.15.31:0/1333132378 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f292c0bdac0 con 0x7f29441038d0 2024-03-21T10:35:25.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.417+0000 7f2948e00640 1 -- 172.21.15.31:0/1333132378 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f294402d270 con 0x7f29441038d0 2024-03-21T10:35:25.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.418+0000 7f293aa00640 1 -- 172.21.15.31:0/1333132378 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f292c0c1b60 con 0x7f29441038d0 2024-03-21T10:35:25.416 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-03-21T10:35:25.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.422+0000 7f2948e00640 1 -- 172.21.15.31:0/1333132378 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f29280a30c0 msgr2=0x7f29280a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:25.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.422+0000 7f2948e00640 1 --2- 172.21.15.31:0/1333132378 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f29280a30c0 0x7f29280a5580 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f294419b090 tx=0x7f2934009500 comp rx=0 tx=0).stop 2024-03-21T10:35:25.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.422+0000 7f2948e00640 1 -- 172.21.15.31:0/1333132378 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29441038d0 msgr2=0x7f2944199b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:25.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.422+0000 7f2948e00640 1 --2- 172.21.15.31:0/1333132378 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29441038d0 0x7f2944199b80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f292c02efd0 tx=0x7f292c02c660 comp rx=0 tx=0).stop 2024-03-21T10:35:25.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.422+0000 7f2948e00640 1 -- 172.21.15.31:0/1333132378 shutdown_connections 2024-03-21T10:35:25.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.422+0000 7f2948e00640 1 --2- 172.21.15.31:0/1333132378 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f29280a30c0 0x7f29280a5580 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:25.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.422+0000 7f2948e00640 1 --2- 172.21.15.31:0/1333132378 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2944105480 0x7f29441a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:25.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.422+0000 7f2948e00640 1 --2- 172.21.15.31:0/1333132378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2944104ac0 0x7f294419a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:25.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.422+0000 7f2948e00640 1 --2- 172.21.15.31:0/1333132378 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29441038d0 0x7f2944199b80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:25.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.422+0000 7f2948e00640 1 -- 172.21.15.31:0/1333132378 >> 172.21.15.31:0/1333132378 conn(0x7f29440fee60 msgr2=0x7f2944107dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:25.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.422+0000 7f2948e00640 1 -- 172.21.15.31:0/1333132378 shutdown_connections 2024-03-21T10:35:25.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.422+0000 7f2948e00640 1 -- 172.21.15.31:0/1333132378 wait complete. 2024-03-21T10:35:25.438 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T10:35:25.522 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:25 smithi031 ceph-mon[31673]: pgmap v334: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:35:25.522 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:25 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/558511371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:25.522 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:25 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/339992753' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:35:25.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:25 smithi070 ceph-mon[36330]: pgmap v334: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:35:25.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:25 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/558511371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:25.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:25 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/339992753' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:35:25.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.672+0000 7f6824200640 1 -- 172.21.15.31:0/120538645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f681c1038c0 msgr2=0x7f681c103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:25.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.672+0000 7f6824200640 1 --2- 172.21.15.31:0/120538645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f681c1038c0 0x7f681c103ce0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6810009ca0 tx=0x7f681002c100 comp rx=0 tx=0).stop 2024-03-21T10:35:25.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.672+0000 7f6824200640 1 -- 172.21.15.31:0/120538645 shutdown_connections 2024-03-21T10:35:25.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.672+0000 7f6824200640 1 --2- 172.21.15.31:0/120538645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f681c105470 0x7f681c10bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:25.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.672+0000 7f6824200640 1 --2- 172.21.15.31:0/120538645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f681c104ab0 0x7f681c104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:25.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.672+0000 7f6824200640 1 --2- 172.21.15.31:0/120538645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f681c1038c0 0x7f681c103ce0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:25.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.672+0000 7f6824200640 1 -- 172.21.15.31:0/120538645 >> 172.21.15.31:0/120538645 conn(0x7f681c0fee50 msgr2=0x7f681c101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:25.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.673+0000 7f6824200640 1 -- 172.21.15.31:0/120538645 shutdown_connections 2024-03-21T10:35:25.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.673+0000 7f6824200640 1 -- 172.21.15.31:0/120538645 wait complete. 2024-03-21T10:35:25.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.673+0000 7f6824200640 1 Processor -- start 2024-03-21T10:35:25.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.674+0000 7f6824200640 1 -- start start 2024-03-21T10:35:25.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.674+0000 7f6824200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f681c1038c0 0x7f681c199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:25.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.674+0000 7f6824200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f681c104ab0 0x7f681c19a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:25.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.674+0000 7f6824200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f681c105470 0x7f681c1a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:25.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.674+0000 7f6824200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c10f510 con 0x7f681c105470 2024-03-21T10:35:25.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.674+0000 7f6824200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c10f390 con 0x7f681c1038c0 2024-03-21T10:35:25.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.674+0000 7f6824200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c10f690 con 0x7f681c104ab0 2024-03-21T10:35:25.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.674+0000 7f6821800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f681c1038c0 0x7f681c199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:25.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.674+0000 7f6822200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f681c105470 0x7f681c1a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:25.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.674+0000 7f6821800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f681c1038c0 0x7f681c199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:45502/0 (socket says 172.21.15.31:45502) 2024-03-21T10:35:25.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.674+0000 7f6821800640 1 -- 172.21.15.31:0/1866825889 learned_addr learned my addr 172.21.15.31:0/1866825889 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:25.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.674+0000 7f6820e00640 1 --2- 172.21.15.31:0/1866825889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f681c104ab0 0x7f681c19a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:25.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.675+0000 7f6821800640 1 -- 172.21.15.31:0/1866825889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f681c104ab0 msgr2=0x7f681c19a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:25.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.675+0000 7f6821800640 1 --2- 172.21.15.31:0/1866825889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f681c104ab0 0x7f681c19a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:25.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.675+0000 7f6821800640 1 -- 172.21.15.31:0/1866825889 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f681c105470 msgr2=0x7f681c1a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:25.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.675+0000 7f6821800640 1 --2- 172.21.15.31:0/1866825889 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f681c105470 0x7f681c1a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:25.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.675+0000 7f6821800640 1 -- 172.21.15.31:0/1866825889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681c1a1810 con 0x7f681c1038c0 2024-03-21T10:35:25.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.675+0000 7f6821800640 1 --2- 172.21.15.31:0/1866825889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f681c1038c0 0x7f681c199bf0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f68100324c0 tx=0x7f6810035d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:25.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.675+0000 7f681a000640 1 -- 172.21.15.31:0/1866825889 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6810039040 con 0x7f681c1038c0 2024-03-21T10:35:25.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.675+0000 7f681a000640 1 -- 172.21.15.31:0/1866825889 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6810044070 con 0x7f681c1038c0 2024-03-21T10:35:25.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.675+0000 7f6824200640 1 -- 172.21.15.31:0/1866825889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f681c1a1aa0 con 0x7f681c1038c0 2024-03-21T10:35:25.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.675+0000 7f681a000640 1 -- 172.21.15.31:0/1866825889 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68100044a0 con 0x7f681c1038c0 2024-03-21T10:35:25.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.675+0000 7f6824200640 1 -- 172.21.15.31:0/1866825889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681c1a1f90 con 0x7f681c1038c0 2024-03-21T10:35:25.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.677+0000 7f6824200640 1 -- 172.21.15.31:0/1866825889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681c10ac60 con 0x7f681c1038c0 2024-03-21T10:35:25.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.678+0000 7f681a000640 1 -- 172.21.15.31:0/1866825889 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f6810004640 con 0x7f681c1038c0 2024-03-21T10:35:25.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.680+0000 7f681a000640 1 --2- 172.21.15.31:0/1866825889 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f68040a2f20 0x7f68040a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:25.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.680+0000 7f681a000640 1 -- 172.21.15.31:0/1866825889 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f68100f1d10 con 0x7f681c1038c0 2024-03-21T10:35:25.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.680+0000 7f6820e00640 1 --2- 172.21.15.31:0/1866825889 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f68040a2f20 0x7f68040a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:25.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.681+0000 7f6820e00640 1 --2- 172.21.15.31:0/1866825889 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f68040a2f20 0x7f68040a53e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f681c19b100 tx=0x7f680800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:25.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:25 smithi040 ceph-mon[36154]: pgmap v334: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:35:25.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:25 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/558511371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:25.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:25 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/339992753' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:35:25.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.683+0000 7f681a000640 1 -- 172.21.15.31:0/1866825889 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f68100f7020 con 0x7f681c1038c0 2024-03-21T10:35:25.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.890+0000 7f6824200640 1 -- 172.21.15.31:0/1866825889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f681c0622a0 con 0x7f681c1038c0 2024-03-21T10:35:25.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.891+0000 7f681a000640 1 -- 172.21.15.31:0/1866825889 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f68100be8a0 con 0x7f681c1038c0 2024-03-21T10:35:25.889 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-03-21T10:35:25.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.893+0000 7f6824200640 1 -- 172.21.15.31:0/1866825889 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f68040a2f20 msgr2=0x7f68040a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:25.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.893+0000 7f6824200640 1 --2- 172.21.15.31:0/1866825889 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f68040a2f20 0x7f68040a53e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f681c19b100 tx=0x7f680800b410 comp rx=0 tx=0).stop 2024-03-21T10:35:25.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.894+0000 7f6824200640 1 -- 172.21.15.31:0/1866825889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f681c1038c0 msgr2=0x7f681c199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:25.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.894+0000 7f6824200640 1 --2- 172.21.15.31:0/1866825889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f681c1038c0 0x7f681c199bf0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f68100324c0 tx=0x7f6810035d90 comp rx=0 tx=0).stop 2024-03-21T10:35:25.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.894+0000 7f6824200640 1 -- 172.21.15.31:0/1866825889 shutdown_connections 2024-03-21T10:35:25.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.894+0000 7f6824200640 1 --2- 172.21.15.31:0/1866825889 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f68040a2f20 0x7f68040a53e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:25.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.894+0000 7f6824200640 1 --2- 172.21.15.31:0/1866825889 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f681c105470 0x7f681c1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:25.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.894+0000 7f6824200640 1 --2- 172.21.15.31:0/1866825889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f681c104ab0 0x7f681c19a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:25.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.894+0000 7f6824200640 1 --2- 172.21.15.31:0/1866825889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f681c1038c0 0x7f681c199bf0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:25.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.894+0000 7f6824200640 1 -- 172.21.15.31:0/1866825889 >> 172.21.15.31:0/1866825889 conn(0x7f681c0fee50 msgr2=0x7f681c107e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:25.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.894+0000 7f6824200640 1 -- 172.21.15.31:0/1866825889 shutdown_connections 2024-03-21T10:35:25.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:25.894+0000 7f6824200640 1 -- 172.21.15.31:0/1866825889 wait complete. 2024-03-21T10:35:25.910 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T10:35:26.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.128+0000 7fbfb2800640 1 -- 172.21.15.31:0/1294799973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfac1038e0 msgr2=0x7fbfac103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:26.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.128+0000 7fbfb2800640 1 --2- 172.21.15.31:0/1294799973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfac1038e0 0x7fbfac103d00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c009be0 tx=0x7fbf9c02bef0 comp rx=0 tx=0).stop 2024-03-21T10:35:26.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.128+0000 7fbfb2800640 1 -- 172.21.15.31:0/1294799973 shutdown_connections 2024-03-21T10:35:26.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.128+0000 7fbfb2800640 1 --2- 172.21.15.31:0/1294799973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfac105490 0x7fbfac10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:26.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.128+0000 7fbfb2800640 1 --2- 172.21.15.31:0/1294799973 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfac104ad0 0x7fbfac104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:26.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.128+0000 7fbfb2800640 1 --2- 172.21.15.31:0/1294799973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfac1038e0 0x7fbfac103d00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:26.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.128+0000 7fbfb2800640 1 -- 172.21.15.31:0/1294799973 >> 172.21.15.31:0/1294799973 conn(0x7fbfac0fee50 msgr2=0x7fbfac101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:26.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.129+0000 7fbfb2800640 1 -- 172.21.15.31:0/1294799973 shutdown_connections 2024-03-21T10:35:26.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.129+0000 7fbfb2800640 1 -- 172.21.15.31:0/1294799973 wait complete. 2024-03-21T10:35:26.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.130+0000 7fbfb2800640 1 Processor -- start 2024-03-21T10:35:26.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.130+0000 7fbfb2800640 1 -- start start 2024-03-21T10:35:26.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.131+0000 7fbfb2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfac1038e0 0x7fbfac199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:26.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.131+0000 7fbfb2800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfac104ad0 0x7fbfac19a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:26.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.131+0000 7fbfb2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfac105490 0x7fbfac1a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:26.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.131+0000 7fbfb2800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac10f5b0 con 0x7fbfac105490 2024-03-21T10:35:26.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.131+0000 7fbfb2800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac10f430 con 0x7fbfac1038e0 2024-03-21T10:35:26.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.131+0000 7fbfb2800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac10f730 con 0x7fbfac104ad0 2024-03-21T10:35:26.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.131+0000 7fbfab400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfac1038e0 0x7fbfac199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:26.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.131+0000 7fbfab400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfac1038e0 0x7fbfac199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:45524/0 (socket says 172.21.15.31:45524) 2024-03-21T10:35:26.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.131+0000 7fbfab400640 1 -- 172.21.15.31:0/425869237 learned_addr learned my addr 172.21.15.31:0/425869237 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:26.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.131+0000 7fbfabe00640 1 --2- 172.21.15.31:0/425869237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfac105490 0x7fbfac1a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:26.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.131+0000 7fbfaaa00640 1 --2- 172.21.15.31:0/425869237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfac104ad0 0x7fbfac19a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:26.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.131+0000 7fbfab400640 1 -- 172.21.15.31:0/425869237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfac104ad0 msgr2=0x7fbfac19a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:26.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.131+0000 7fbfab400640 1 --2- 172.21.15.31:0/425869237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfac104ad0 0x7fbfac19a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:26.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.131+0000 7fbfab400640 1 -- 172.21.15.31:0/425869237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfac105490 msgr2=0x7fbfac1a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:26.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.131+0000 7fbfab400640 1 --2- 172.21.15.31:0/425869237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfac105490 0x7fbfac1a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:26.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.131+0000 7fbfab400640 1 -- 172.21.15.31:0/425869237 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfac1a1810 con 0x7fbfac1038e0 2024-03-21T10:35:26.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.131+0000 7fbfabe00640 1 --2- 172.21.15.31:0/425869237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfac105490 0x7fbfac1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:35:26.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.132+0000 7fbfab400640 1 --2- 172.21.15.31:0/425869237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfac1038e0 0x7fbfac199bf0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c02c500 tx=0x7fbf9c033580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:26.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.132+0000 7fbfa3e00640 1 -- 172.21.15.31:0/425869237 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c043020 con 0x7fbfac1038e0 2024-03-21T10:35:26.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.132+0000 7fbfa3e00640 1 -- 172.21.15.31:0/425869237 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf9c003e50 con 0x7fbfac1038e0 2024-03-21T10:35:26.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.132+0000 7fbfa3e00640 1 -- 172.21.15.31:0/425869237 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c034690 con 0x7fbfac1038e0 2024-03-21T10:35:26.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.132+0000 7fbfb2800640 1 -- 172.21.15.31:0/425869237 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfac1a1aa0 con 0x7fbfac1038e0 2024-03-21T10:35:26.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.132+0000 7fbfb2800640 1 -- 172.21.15.31:0/425869237 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfac1a1fa0 con 0x7fbfac1038e0 2024-03-21T10:35:26.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.134+0000 7fbfb2800640 1 -- 172.21.15.31:0/425869237 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfac10aba0 con 0x7fbfac1038e0 2024-03-21T10:35:26.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.135+0000 7fbfa3e00640 1 -- 172.21.15.31:0/425869237 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fbf9c046020 con 0x7fbfac1038e0 2024-03-21T10:35:26.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.137+0000 7fbfa3e00640 1 --2- 172.21.15.31:0/425869237 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fbf900a2f20 0x7fbf900a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:26.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.137+0000 7fbfa3e00640 1 -- 172.21.15.31:0/425869237 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fbf9c0f1f10 con 0x7fbfac1038e0 2024-03-21T10:35:26.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.137+0000 7fbfaaa00640 1 --2- 172.21.15.31:0/425869237 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fbf900a2f20 0x7fbf900a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:26.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.138+0000 7fbfaaa00640 1 --2- 172.21.15.31:0/425869237 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fbf900a2f20 0x7fbf900a53e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbfac19b100 tx=0x7fbf94009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:26.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.141+0000 7fbfa3e00640 1 -- 172.21.15.31:0/425869237 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbf9c04d020 con 0x7fbfac1038e0 2024-03-21T10:35:26.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.349+0000 7fbfb2800640 1 -- 172.21.15.31:0/425869237 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fbfac0622a0 con 0x7fbfac1038e0 2024-03-21T10:35:26.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.350+0000 7fbfa3e00640 1 -- 172.21.15.31:0/425869237 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7fbf9c0beaa0 con 0x7fbfac1038e0 2024-03-21T10:35:26.349 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-03-21T10:35:26.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.353+0000 7fbfb2800640 1 -- 172.21.15.31:0/425869237 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fbf900a2f20 msgr2=0x7fbf900a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:26.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.353+0000 7fbfb2800640 1 --2- 172.21.15.31:0/425869237 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fbf900a2f20 0x7fbf900a53e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbfac19b100 tx=0x7fbf94009260 comp rx=0 tx=0).stop 2024-03-21T10:35:26.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.353+0000 7fbfb2800640 1 -- 172.21.15.31:0/425869237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfac1038e0 msgr2=0x7fbfac199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:26.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.353+0000 7fbfb2800640 1 --2- 172.21.15.31:0/425869237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfac1038e0 0x7fbfac199bf0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c02c500 tx=0x7fbf9c033580 comp rx=0 tx=0).stop 2024-03-21T10:35:26.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.353+0000 7fbfb2800640 1 -- 172.21.15.31:0/425869237 shutdown_connections 2024-03-21T10:35:26.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.353+0000 7fbfb2800640 1 --2- 172.21.15.31:0/425869237 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fbf900a2f20 0x7fbf900a53e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:26.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.353+0000 7fbfb2800640 1 --2- 172.21.15.31:0/425869237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfac105490 0x7fbfac1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:26.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.353+0000 7fbfb2800640 1 --2- 172.21.15.31:0/425869237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfac104ad0 0x7fbfac19a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:26.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.353+0000 7fbfb2800640 1 --2- 172.21.15.31:0/425869237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfac1038e0 0x7fbfac199bf0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:26.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.353+0000 7fbfb2800640 1 -- 172.21.15.31:0/425869237 >> 172.21.15.31:0/425869237 conn(0x7fbfac0fee50 msgr2=0x7fbfac107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:26.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.353+0000 7fbfb2800640 1 -- 172.21.15.31:0/425869237 shutdown_connections 2024-03-21T10:35:26.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.354+0000 7fbfb2800640 1 -- 172.21.15.31:0/425869237 wait complete. 2024-03-21T10:35:26.372 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-21T10:35:26.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:26 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1333132378' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:35:26.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:26 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1866825889' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:35:26.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:26 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1333132378' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:35:26.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:26 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1866825889' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:35:26.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.601+0000 7f8dc7200640 1 -- 172.21.15.31:0/3601583541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dc80756e0 msgr2=0x7f8dc8073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:26.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.602+0000 7f8dc7200640 1 --2- 172.21.15.31:0/3601583541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dc80756e0 0x7f8dc8073b30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8db8009cd0 tx=0x7f8db802c0f0 comp rx=0 tx=0).stop 2024-03-21T10:35:26.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.602+0000 7f8dc7200640 1 -- 172.21.15.31:0/3601583541 shutdown_connections 2024-03-21T10:35:26.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.602+0000 7f8dc7200640 1 --2- 172.21.15.31:0/3601583541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dc810dab0 0x7f8dc810fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:26.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.602+0000 7f8dc7200640 1 --2- 172.21.15.31:0/3601583541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8dc8074100 0x7f8dc8074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:26.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.602+0000 7f8dc7200640 1 --2- 172.21.15.31:0/3601583541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dc80756e0 0x7f8dc8073b30 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:26.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.602+0000 7f8dc7200640 1 -- 172.21.15.31:0/3601583541 >> 172.21.15.31:0/3601583541 conn(0x7f8dc80fca50 msgr2=0x7f8dc80fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:26.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.602+0000 7f8dc7200640 1 -- 172.21.15.31:0/3601583541 shutdown_connections 2024-03-21T10:35:26.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.602+0000 7f8dc7200640 1 -- 172.21.15.31:0/3601583541 wait complete. 2024-03-21T10:35:26.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.603+0000 7f8dc7200640 1 Processor -- start 2024-03-21T10:35:26.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.603+0000 7f8dc7200640 1 -- start start 2024-03-21T10:35:26.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.603+0000 7f8dc7200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dc8074100 0x7f8dc819de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:26.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.604+0000 7f8dc7200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8dc80756e0 0x7f8dc819e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:26.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.604+0000 7f8dc7200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dc810dab0 0x7f8dc81a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:26.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.604+0000 7f8dc7200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc8113730 con 0x7f8dc8074100 2024-03-21T10:35:26.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.604+0000 7f8dc7200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc81135b0 con 0x7f8dc810dab0 2024-03-21T10:35:26.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.604+0000 7f8dc7200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc81138b0 con 0x7f8dc80756e0 2024-03-21T10:35:26.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.604+0000 7f8dc5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dc8074100 0x7f8dc819de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:26.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.604+0000 7f8dc6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dc810dab0 0x7f8dc81a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:26.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.604+0000 7f8dc5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dc8074100 0x7f8dc819de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57366/0 (socket says 172.21.15.31:57366) 2024-03-21T10:35:26.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.604+0000 7f8dc5e00640 1 -- 172.21.15.31:0/1857124725 learned_addr learned my addr 172.21.15.31:0/1857124725 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:26.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.604+0000 7f8dc5400640 1 --2- 172.21.15.31:0/1857124725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8dc80756e0 0x7f8dc819e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:26.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.604+0000 7f8dc6800640 1 -- 172.21.15.31:0/1857124725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8dc80756e0 msgr2=0x7f8dc819e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:26.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.604+0000 7f8dc6800640 1 --2- 172.21.15.31:0/1857124725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8dc80756e0 0x7f8dc819e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:26.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.604+0000 7f8dc6800640 1 -- 172.21.15.31:0/1857124725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dc8074100 msgr2=0x7f8dc819de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:26.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.604+0000 7f8dc6800640 1 --2- 172.21.15.31:0/1857124725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dc8074100 0x7f8dc819de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:26.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.604+0000 7f8dc6800640 1 -- 172.21.15.31:0/1857124725 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc81a5a50 con 0x7f8dc810dab0 2024-03-21T10:35:26.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.604+0000 7f8dc5400640 1 --2- 172.21.15.31:0/1857124725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8dc80756e0 0x7f8dc819e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:35:26.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.605+0000 7f8dc6800640 1 --2- 172.21.15.31:0/1857124725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dc810dab0 0x7f8dc81a53e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8dc000d6b0 tx=0x7f8dc000db70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:26.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.605+0000 7f8dbea00640 1 -- 172.21.15.31:0/1857124725 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc0017070 con 0x7f8dc810dab0 2024-03-21T10:35:26.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.605+0000 7f8dbea00640 1 -- 172.21.15.31:0/1857124725 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8dc0009d00 con 0x7f8dc810dab0 2024-03-21T10:35:26.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.605+0000 7f8dbea00640 1 -- 172.21.15.31:0/1857124725 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc0003d60 con 0x7f8dc810dab0 2024-03-21T10:35:26.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.605+0000 7f8dc7200640 1 -- 172.21.15.31:0/1857124725 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc81a5d40 con 0x7f8dc810dab0 2024-03-21T10:35:26.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.605+0000 7f8dc7200640 1 -- 172.21.15.31:0/1857124725 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc81a6230 con 0x7f8dc810dab0 2024-03-21T10:35:26.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.606+0000 7f8dc7200640 1 -- 172.21.15.31:0/1857124725 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc8103c80 con 0x7f8dc810dab0 2024-03-21T10:35:26.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.608+0000 7f8dbea00640 1 -- 172.21.15.31:0/1857124725 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f8dc0009820 con 0x7f8dc810dab0 2024-03-21T10:35:26.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.609+0000 7f8dbea00640 1 --2- 172.21.15.31:0/1857124725 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f8dac0a2f20 0x7f8dac0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:26.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.609+0000 7f8dbea00640 1 -- 172.21.15.31:0/1857124725 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f8dc00ceed0 con 0x7f8dc810dab0 2024-03-21T10:35:26.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.609+0000 7f8dc5e00640 1 --2- 172.21.15.31:0/1857124725 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f8dac0a2f20 0x7f8dac0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:26.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.610+0000 7f8dc5e00640 1 --2- 172.21.15.31:0/1857124725 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f8dac0a2f20 0x7f8dac0a53e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8db8005e30 tx=0x7f8db8004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:26.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.613+0000 7f8dbea00640 1 -- 172.21.15.31:0/1857124725 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8dc009ba60 con 0x7f8dc810dab0 2024-03-21T10:35:26.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:26 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1333132378' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:35:26.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:26 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1866825889' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:35:26.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.833+0000 7f8dc7200640 1 -- 172.21.15.31:0/1857124725 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8dc802d070 con 0x7f8dc810dab0 2024-03-21T10:35:26.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.833+0000 7f8dbea00640 1 -- 172.21.15.31:0/1857124725 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+591 (secure 0 0 0) 0x7f8dc009b360 con 0x7f8dc810dab0 2024-03-21T10:35:26.832 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/1451930554 2024-03-22T10:26:53.842623+0000 2024-03-21T10:35:26.832 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/3342588843 2024-03-22T10:26:53.842623+0000 2024-03-21T10:35:26.832 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6801/4085461895 2024-03-22T10:26:53.842623+0000 2024-03-21T10:35:26.832 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6800/4085461895 2024-03-22T10:26:53.842623+0000 2024-03-21T10:35:26.832 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/1626448389 2024-03-22T10:26:16.959165+0000 2024-03-21T10:35:26.832 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/1910059351 2024-03-22T10:26:16.959165+0000 2024-03-21T10:35:26.833 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/344822770 2024-03-22T10:26:16.959165+0000 2024-03-21T10:35:26.833 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/3811368512 2024-03-22T10:26:53.842623+0000 2024-03-21T10:35:26.833 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6800/1981437651 2024-03-22T10:26:16.959165+0000 2024-03-21T10:35:26.833 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6801/1981437651 2024-03-22T10:26:16.959165+0000 2024-03-21T10:35:26.833 INFO:teuthology.orchestra.run.smithi031.stderr:listed 10 entries 2024-03-21T10:35:26.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.836+0000 7f8dc7200640 1 -- 172.21.15.31:0/1857124725 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f8dac0a2f20 msgr2=0x7f8dac0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:26.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.836+0000 7f8dc7200640 1 --2- 172.21.15.31:0/1857124725 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f8dac0a2f20 0x7f8dac0a53e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8db8005e30 tx=0x7f8db8004a40 comp rx=0 tx=0).stop 2024-03-21T10:35:26.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.836+0000 7f8dc7200640 1 -- 172.21.15.31:0/1857124725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dc810dab0 msgr2=0x7f8dc81a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:26.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.836+0000 7f8dc7200640 1 --2- 172.21.15.31:0/1857124725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dc810dab0 0x7f8dc81a53e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8dc000d6b0 tx=0x7f8dc000db70 comp rx=0 tx=0).stop 2024-03-21T10:35:26.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.836+0000 7f8dc7200640 1 -- 172.21.15.31:0/1857124725 shutdown_connections 2024-03-21T10:35:26.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.836+0000 7f8dc7200640 1 --2- 172.21.15.31:0/1857124725 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f8dac0a2f20 0x7f8dac0a53e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:26.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.836+0000 7f8dc7200640 1 --2- 172.21.15.31:0/1857124725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dc810dab0 0x7f8dc81a53e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:26.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.836+0000 7f8dc7200640 1 --2- 172.21.15.31:0/1857124725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8dc80756e0 0x7f8dc819e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:26.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.836+0000 7f8dc7200640 1 --2- 172.21.15.31:0/1857124725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dc8074100 0x7f8dc819de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:26.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.836+0000 7f8dc7200640 1 -- 172.21.15.31:0/1857124725 >> 172.21.15.31:0/1857124725 conn(0x7f8dc80fca50 msgr2=0x7f8dc8100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:26.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.837+0000 7f8dc7200640 1 -- 172.21.15.31:0/1857124725 shutdown_connections 2024-03-21T10:35:26.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:26.837+0000 7f8dc7200640 1 -- 172.21.15.31:0/1857124725 wait complete. 2024-03-21T10:35:26.855 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-21T10:35:26.855 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T10:35:26.910 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T10:35:27.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.188+0000 7fc453200640 1 -- 172.21.15.31:0/1854213496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc44c076060 msgr2=0x7fc44c076480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:27.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.188+0000 7fc453200640 1 --2- 172.21.15.31:0/1854213496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc44c076060 0x7fc44c076480 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc440009900 tx=0x7fc44002bc30 comp rx=0 tx=0).stop 2024-03-21T10:35:27.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.188+0000 7fc453200640 1 -- 172.21.15.31:0/1854213496 shutdown_connections 2024-03-21T10:35:27.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.188+0000 7fc453200640 1 --2- 172.21.15.31:0/1854213496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc44c077c40 0x7fc44c0784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:27.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.188+0000 7fc453200640 1 --2- 172.21.15.31:0/1854213496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc44c0772a0 0x7fc44c077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:27.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.188+0000 7fc453200640 1 --2- 172.21.15.31:0/1854213496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc44c076060 0x7fc44c076480 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:27.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.188+0000 7fc453200640 1 -- 172.21.15.31:0/1854213496 >> 172.21.15.31:0/1854213496 conn(0x7fc44c0fed50 msgr2=0x7fc44c101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:27.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.188+0000 7fc453200640 1 -- 172.21.15.31:0/1854213496 shutdown_connections 2024-03-21T10:35:27.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.188+0000 7fc453200640 1 -- 172.21.15.31:0/1854213496 wait complete. 2024-03-21T10:35:27.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.189+0000 7fc453200640 1 Processor -- start 2024-03-21T10:35:27.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.189+0000 7fc453200640 1 -- start start 2024-03-21T10:35:27.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.189+0000 7fc453200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc44c076060 0x7fc44c19de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:27.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.189+0000 7fc453200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc44c0772a0 0x7fc44c19e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:27.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.189+0000 7fc453200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc44c077c40 0x7fc44c1a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:27.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.189+0000 7fc453200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c113870 con 0x7fc44c076060 2024-03-21T10:35:27.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.189+0000 7fc453200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c1136f0 con 0x7fc44c0772a0 2024-03-21T10:35:27.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.189+0000 7fc453200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c1139f0 con 0x7fc44c077c40 2024-03-21T10:35:27.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.190+0000 7fc451400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc44c0772a0 0x7fc44c19e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:27.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.190+0000 7fc451e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc44c076060 0x7fc44c19de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:27.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.190+0000 7fc451400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc44c0772a0 0x7fc44c19e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49314/0 (socket says 172.21.15.31:49314) 2024-03-21T10:35:27.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.190+0000 7fc451400640 1 -- 172.21.15.31:0/1185944224 learned_addr learned my addr 172.21.15.31:0/1185944224 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:27.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.190+0000 7fc452800640 1 --2- 172.21.15.31:0/1185944224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc44c077c40 0x7fc44c1a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:27.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.190+0000 7fc451400640 1 -- 172.21.15.31:0/1185944224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc44c077c40 msgr2=0x7fc44c1a5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:27.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.190+0000 7fc451400640 1 --2- 172.21.15.31:0/1185944224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc44c077c40 0x7fc44c1a5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:27.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.190+0000 7fc451400640 1 -- 172.21.15.31:0/1185944224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc44c076060 msgr2=0x7fc44c19de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:27.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.190+0000 7fc451400640 1 --2- 172.21.15.31:0/1185944224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc44c076060 0x7fc44c19de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:27.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.190+0000 7fc451400640 1 -- 172.21.15.31:0/1185944224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44c1a5ab0 con 0x7fc44c0772a0 2024-03-21T10:35:27.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.190+0000 7fc452800640 1 --2- 172.21.15.31:0/1185944224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc44c077c40 0x7fc44c1a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:35:27.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.190+0000 7fc451400640 1 --2- 172.21.15.31:0/1185944224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc44c0772a0 0x7fc44c19e3d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc43c00e9f0 tx=0x7fc43c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:27.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.191+0000 7fc446a00640 1 -- 172.21.15.31:0/1185944224 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43c00cd80 con 0x7fc44c0772a0 2024-03-21T10:35:27.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.191+0000 7fc453200640 1 -- 172.21.15.31:0/1185944224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc44c1a5da0 con 0x7fc44c0772a0 2024-03-21T10:35:27.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.191+0000 7fc446a00640 1 -- 172.21.15.31:0/1185944224 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc43c010910 con 0x7fc44c0772a0 2024-03-21T10:35:27.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.191+0000 7fc446a00640 1 -- 172.21.15.31:0/1185944224 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43c0188f0 con 0x7fc44c0772a0 2024-03-21T10:35:27.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.191+0000 7fc453200640 1 -- 172.21.15.31:0/1185944224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc44c1a6270 con 0x7fc44c0772a0 2024-03-21T10:35:27.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.192+0000 7fc453200640 1 -- 172.21.15.31:0/1185944224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc44c04f470 con 0x7fc44c0772a0 2024-03-21T10:35:27.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.196+0000 7fc446a00640 1 -- 172.21.15.31:0/1185944224 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fc43c010ac0 con 0x7fc44c0772a0 2024-03-21T10:35:27.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.198+0000 7fc446a00640 1 --2- 172.21.15.31:0/1185944224 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc4340a2f10 0x7fc4340a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:27.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.198+0000 7fc446a00640 1 -- 172.21.15.31:0/1185944224 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc43c0cff50 con 0x7fc44c0772a0 2024-03-21T10:35:27.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.198+0000 7fc451e00640 1 --2- 172.21.15.31:0/1185944224 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc4340a2f10 0x7fc4340a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:27.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.199+0000 7fc446a00640 1 -- 172.21.15.31:0/1185944224 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc43c09cb40 con 0x7fc44c0772a0 2024-03-21T10:35:27.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.200+0000 7fc451e00640 1 --2- 172.21.15.31:0/1185944224 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc4340a2f10 0x7fc4340a53d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc44002c240 tx=0x7fc44002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:27.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.407+0000 7fc453200640 1 -- 172.21.15.31:0/1185944224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc44c110110 con 0x7fc44c0772a0 2024-03-21T10:35:27.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.410+0000 7fc446a00640 1 -- 172.21.15.31:0/1185944224 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24009 (secure 0 0 0) 0x7fc44c110110 con 0x7fc44c0772a0 2024-03-21T10:35:27.408 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:35:27.408 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","created":"2024-03-21T10:25:41.971117+0000","modified":"2024-03-21T10:35:11.657050+0000","last_up_change":"2024-03-21T10:34:07.757196+0000","last_in_change":"2024-03-21T10:33:47.460406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T10:29:44.553968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T10:35:04.945449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T10:35:05.970589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T10:35:09.688711+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e0244d33-00e2-47d5-9f66-34efc138c462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6803","nonce":3089885125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6805","nonce":3089885125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6809","nonce":3089885125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6807","nonce":3089885125}]},"public_addr":"172.21.15.31:6803/3089885125","cluster_addr":"172.21.15.31:6805/3089885125","heartbeat_back_addr":"172.21.15.31:6809/3089885125","heartbeat_front_addr":"172.21.15.31:6807/3089885125","state":["exists","up"]},{"osd":1,"uuid":"76a63504-beab-4354-a6c3-eebc3b16ad03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6801","nonce":618840037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6803","nonce":618840037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6807","nonce":618840037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6805","nonce":618840037}]},"public_addr":"172.21.15.40:6801/618840037","cluster_addr":"172.21.15.40:6803/618840037","heartbeat_back_addr":"172.21.15.40:6807/618840037","heartbeat_front_addr":"172.21.15.40:6805/618840037","state":["exists","up"]},{"osd":2,"uuid":"fa191bd6-9a21-4ee2-8adf-8a63c9cfd821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6801","nonce":2524184117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6803","nonce":2524184117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6807","nonce":2524184117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6805","nonce":2524184117}]},"public_addr":"172.21.15.70:6801/2524184117","cluster_addr":"172.21.15.70:6803/2524184117","heartbeat_back_addr":"172.21.15.70:6807/2524184117","heartbeat_front_addr":"172.21.15.70:6805/2524184117","state":["exists","up"]},{"osd":3,"uuid":"edb44507-ffb0-42a4-827b-08461e1bf952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6811","nonce":2823036296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6813","nonce":2823036296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6817","nonce":2823036296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6815","nonce":2823036296}]},"public_addr":"172.21.15.31:6811/2823036296","cluster_addr":"172.21.15.31:6813/2823036296","heartbeat_back_addr":"172.21.15.31:6817/2823036296","heartbeat_front_addr":"172.21.15.31:6815/2823036296","state":["exists","up"]},{"osd":4,"uuid":"8cb89153-8c3b-431f-b57c-8863fada76dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6809","nonce":4215186309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6811","nonce":4215186309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6815","nonce":4215186309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6813","nonce":4215186309}]},"public_addr":"172.21.15.40:6809/4215186309","cluster_addr":"172.21.15.40:6811/4215186309","heartbeat_back_addr":"172.21.15.40:6815/4215186309","heartbeat_front_addr":"172.21.15.40:6813/4215186309","state":["exists","up"]},{"osd":5,"uuid":"9409505a-d463-408c-a3cd-218380c04d68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6809","nonce":3402830452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6811","nonce":3402830452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6815","nonce":3402830452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6813","nonce":3402830452}]},"public_addr":"172.21.15.70:6809/3402830452","cluster_addr":"172.21.15.70:6811/3402830452","heartbeat_back_addr":"172.21.15.70:6815/3402830452","heartbeat_front_addr":"172.21.15.70:6813/3402830452","state":["exists","up"]},{"osd":6,"uuid":"f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6819","nonce":1193497058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6821","nonce":1193497058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6825","nonce":1193497058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6823","nonce":1193497058}]},"public_addr":"172.21.15.31:6819/1193497058","cluster_addr":"172.21.15.31:6821/1193497058","heartbeat_back_addr":"172.21.15.31:6825/1193497058","heartbeat_front_addr":"172.21.15.31:6823/1193497058","state":["exists","up"]},{"osd":7,"uuid":"db228a87-29a5-45d4-8fe6-a0a554262eb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6817","nonce":3825628119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6819","nonce":3825628119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6823","nonce":3825628119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6821","nonce":3825628119}]},"public_addr":"172.21.15.40:6817/3825628119","cluster_addr":"172.21.15.40:6819/3825628119","heartbeat_back_addr":"172.21.15.40:6823/3825628119","heartbeat_front_addr":"172.21.15.40:6821/3825628119","state":["exists","up"]},{"osd":8,"uuid":"59b9fde0-095d-4e0b-87da-ead37bd8ff1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6817","nonce":534880586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6819","nonce":534880586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6823","nonce":534880586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6821","nonce":534880586}]},"public_addr":"172.21.15.70:6817/534880586","cluster_addr":"172.21.15.70:6819/534880586","heartbeat_back_addr":"172.21.15.70:6823/534880586","heartbeat_front_addr":"172.21.15.70:6821/534880586","state":["exists","up"]},{"osd":9,"uuid":"77f47061-0c5f-4159-b276-e814935d6c1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6827","nonce":587047469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6829","nonce":587047469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6833","nonce":587047469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6831","nonce":587047469}]},"public_addr":"172.21.15.31:6827/587047469","cluster_addr":"172.21.15.31:6829/587047469","heartbeat_back_addr":"172.21.15.31:6833/587047469","heartbeat_front_addr":"172.21.15.31:6831/587047469","state":["exists","up"]},{"osd":10,"uuid":"8853dc43-7bd1-4c93-8063-f7982fd7da38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6825","nonce":512108711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6827","nonce":512108711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6831","nonce":512108711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6829","nonce":512108711}]},"public_addr":"172.21.15.40:6825/512108711","cluster_addr":"172.21.15.40:6827/512108711","heartbeat_back_addr":"172.21.15.40:6831/512108711","heartbeat_front_addr":"172.21.15.40:6829/512108711","state":["exists","up"]},{"osd":11,"uuid":"84b7c2de-c424-45a6-9583-60097e372a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6825","nonce":540305718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6827","nonce":540305718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6831","nonce":540305718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6829","nonce":540305718}]},"public_addr":"172.21.15.70:6825/540305718","cluster_addr":"172.21.15.70:6827/540305718","heartbeat_back_addr":"172.21.15.70:6831/540305718","heartbeat_front_addr":"172.21.15.70:6829/540305718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:28:51.477358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:17.405045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:41.626268+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:15.339568+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:42.676996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:07.516999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:42.412273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:10.365041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:35.699120+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:11.733639+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:39.505442+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:34:05.449301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451930554":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/3342588843":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6801/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/1626448389":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/1910059351":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/344822770":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/3811368512":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/1981437651":"2024-03-22T10:26:16.959165+0000","172.21.15.31:6801/1981437651":"2024-03-22T10:26:16.959165+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T10:35:27.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.413+0000 7fc453200640 1 -- 172.21.15.31:0/1185944224 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc4340a2f10 msgr2=0x7fc4340a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:27.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.413+0000 7fc453200640 1 --2- 172.21.15.31:0/1185944224 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc4340a2f10 0x7fc4340a53d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc44002c240 tx=0x7fc44002df90 comp rx=0 tx=0).stop 2024-03-21T10:35:27.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.413+0000 7fc453200640 1 -- 172.21.15.31:0/1185944224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc44c0772a0 msgr2=0x7fc44c19e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:27.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.413+0000 7fc453200640 1 --2- 172.21.15.31:0/1185944224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc44c0772a0 0x7fc44c19e3d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc43c00e9f0 tx=0x7fc43c00eeb0 comp rx=0 tx=0).stop 2024-03-21T10:35:27.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.413+0000 7fc453200640 1 -- 172.21.15.31:0/1185944224 shutdown_connections 2024-03-21T10:35:27.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.413+0000 7fc453200640 1 --2- 172.21.15.31:0/1185944224 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc4340a2f10 0x7fc4340a53d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:27.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.413+0000 7fc453200640 1 --2- 172.21.15.31:0/1185944224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc44c077c40 0x7fc44c1a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:27.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.413+0000 7fc453200640 1 --2- 172.21.15.31:0/1185944224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc44c0772a0 0x7fc44c19e3d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:27.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.413+0000 7fc453200640 1 --2- 172.21.15.31:0/1185944224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc44c076060 0x7fc44c19de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:27.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.413+0000 7fc453200640 1 -- 172.21.15.31:0/1185944224 >> 172.21.15.31:0/1185944224 conn(0x7fc44c0fed50 msgr2=0x7fc44c100850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:27.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.413+0000 7fc453200640 1 -- 172.21.15.31:0/1185944224 shutdown_connections 2024-03-21T10:35:27.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.413+0000 7fc453200640 1 -- 172.21.15.31:0/1185944224 wait complete. 2024-03-21T10:35:27.432 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T10:29:44.553968+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T10:35:04.945449+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T10:35:05.970589+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T10:35:09.688711+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T10:35:27.432 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T10:35:27.516 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:27 smithi031 ceph-mon[31673]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-21T10:35:27.516 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:27 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/425869237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:35:27.516 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:27 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1857124725' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T10:35:27.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:27 smithi070 ceph-mon[36330]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-21T10:35:27.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:27 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/425869237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:35:27.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:27 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1857124725' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T10:35:27.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.661+0000 7f4e94e00640 1 -- 172.21.15.31:0/2189141358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e90103260 msgr2=0x7f4e90109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:27.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.661+0000 7f4e94e00640 1 --2- 172.21.15.31:0/2189141358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e90103260 0x7f4e90109af0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4e88009ca0 tx=0x7f4e8802c0f0 comp rx=0 tx=0).stop 2024-03-21T10:35:27.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.661+0000 7f4e94e00640 1 -- 172.21.15.31:0/2189141358 shutdown_connections 2024-03-21T10:35:27.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.661+0000 7f4e94e00640 1 --2- 172.21.15.31:0/2189141358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e90103260 0x7f4e90109af0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:27.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.661+0000 7f4e94e00640 1 --2- 172.21.15.31:0/2189141358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e901028a0 0x7f4e90102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:27.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.661+0000 7f4e94e00640 1 --2- 172.21.15.31:0/2189141358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e901016b0 0x7f4e90101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:27.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.661+0000 7f4e94e00640 1 -- 172.21.15.31:0/2189141358 >> 172.21.15.31:0/2189141358 conn(0x7f4e900fcc20 msgr2=0x7f4e900ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:27.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.661+0000 7f4e94e00640 1 -- 172.21.15.31:0/2189141358 shutdown_connections 2024-03-21T10:35:27.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.661+0000 7f4e94e00640 1 -- 172.21.15.31:0/2189141358 wait complete. 2024-03-21T10:35:27.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.662+0000 7f4e94e00640 1 Processor -- start 2024-03-21T10:35:27.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.662+0000 7f4e94e00640 1 -- start start 2024-03-21T10:35:27.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.662+0000 7f4e94e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e901016b0 0x7f4e90199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:27.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.663+0000 7f4e94e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e901028a0 0x7f4e9019a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:27.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.663+0000 7f4e94e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e90103260 0x7f4e901a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:27.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.663+0000 7f4e94e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9010d310 con 0x7f4e90103260 2024-03-21T10:35:27.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.663+0000 7f4e94e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9010d190 con 0x7f4e901016b0 2024-03-21T10:35:27.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.663+0000 7f4e94e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9010d490 con 0x7f4e901028a0 2024-03-21T10:35:27.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.663+0000 7f4e8de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e901016b0 0x7f4e90199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:27.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.663+0000 7f4e8de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e901016b0 0x7f4e90199b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49328/0 (socket says 172.21.15.31:49328) 2024-03-21T10:35:27.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.663+0000 7f4e8de00640 1 -- 172.21.15.31:0/3467919744 learned_addr learned my addr 172.21.15.31:0/3467919744 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:27.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.663+0000 7f4e8d400640 1 --2- 172.21.15.31:0/3467919744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e901028a0 0x7f4e9019a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:27.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.663+0000 7f4e8e800640 1 --2- 172.21.15.31:0/3467919744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e90103260 0x7f4e901a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:27.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.663+0000 7f4e8de00640 1 -- 172.21.15.31:0/3467919744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e901028a0 msgr2=0x7f4e9019a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:27.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.663+0000 7f4e8de00640 1 --2- 172.21.15.31:0/3467919744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e901028a0 0x7f4e9019a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:27.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.663+0000 7f4e8de00640 1 -- 172.21.15.31:0/3467919744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e90103260 msgr2=0x7f4e901a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:27.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.663+0000 7f4e8de00640 1 --2- 172.21.15.31:0/3467919744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e90103260 0x7f4e901a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:27.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.663+0000 7f4e8de00640 1 -- 172.21.15.31:0/3467919744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e901a17b0 con 0x7f4e901016b0 2024-03-21T10:35:27.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.664+0000 7f4e8de00640 1 --2- 172.21.15.31:0/3467919744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e901016b0 0x7f4e90199b90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4e8000e9c0 tx=0x7f4e8000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:27.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.664+0000 7f4e86a00640 1 -- 172.21.15.31:0/3467919744 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8000cd80 con 0x7f4e901016b0 2024-03-21T10:35:27.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.664+0000 7f4e86a00640 1 -- 172.21.15.31:0/3467919744 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e800035f0 con 0x7f4e901016b0 2024-03-21T10:35:27.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.664+0000 7f4e86a00640 1 -- 172.21.15.31:0/3467919744 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e80010690 con 0x7f4e901016b0 2024-03-21T10:35:27.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.664+0000 7f4e94e00640 1 -- 172.21.15.31:0/3467919744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e901a1aa0 con 0x7f4e901016b0 2024-03-21T10:35:27.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.664+0000 7f4e94e00640 1 -- 172.21.15.31:0/3467919744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e901a1fa0 con 0x7f4e901016b0 2024-03-21T10:35:27.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.665+0000 7f4e94e00640 1 -- 172.21.15.31:0/3467919744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e901a2230 con 0x7f4e901016b0 2024-03-21T10:35:27.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.667+0000 7f4e86a00640 1 -- 172.21.15.31:0/3467919744 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f4e800037a0 con 0x7f4e901016b0 2024-03-21T10:35:27.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.668+0000 7f4e86a00640 1 --2- 172.21.15.31:0/3467919744 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4e740a2fa0 0x7f4e740a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:27.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.668+0000 7f4e86a00640 1 -- 172.21.15.31:0/3467919744 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f4e800cf9f0 con 0x7f4e901016b0 2024-03-21T10:35:27.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.669+0000 7f4e8d400640 1 --2- 172.21.15.31:0/3467919744 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4e740a2fa0 0x7f4e740a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:27.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.669+0000 7f4e8d400640 1 --2- 172.21.15.31:0/3467919744 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4e740a2fa0 0x7f4e740a5460 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4e9019b0a0 tx=0x7f4e7c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:27.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.672+0000 7f4e86a00640 1 -- 172.21.15.31:0/3467919744 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4e8009c580 con 0x7f4e901016b0 2024-03-21T10:35:27.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:27 smithi040 ceph-mon[36154]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-21T10:35:27.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:27 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/425869237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:35:27.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:27 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1857124725' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T10:35:27.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.877+0000 7f4e94e00640 1 -- 172.21.15.31:0/3467919744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4e901a2400 con 0x7f4e901016b0 2024-03-21T10:35:27.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.877+0000 7f4e86a00640 1 -- 172.21.15.31:0/3467919744 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4e8009be80 con 0x7f4e901016b0 2024-03-21T10:35:27.876 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-03-21T10:35:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.880+0000 7f4e94e00640 1 -- 172.21.15.31:0/3467919744 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4e740a2fa0 msgr2=0x7f4e740a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.880+0000 7f4e94e00640 1 --2- 172.21.15.31:0/3467919744 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4e740a2fa0 0x7f4e740a5460 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4e9019b0a0 tx=0x7f4e7c005f90 comp rx=0 tx=0).stop 2024-03-21T10:35:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.880+0000 7f4e94e00640 1 -- 172.21.15.31:0/3467919744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e901016b0 msgr2=0x7f4e90199b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.880+0000 7f4e94e00640 1 --2- 172.21.15.31:0/3467919744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e901016b0 0x7f4e90199b90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4e8000e9c0 tx=0x7f4e8000ee80 comp rx=0 tx=0).stop 2024-03-21T10:35:27.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.880+0000 7f4e94e00640 1 -- 172.21.15.31:0/3467919744 shutdown_connections 2024-03-21T10:35:27.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.880+0000 7f4e94e00640 1 --2- 172.21.15.31:0/3467919744 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4e740a2fa0 0x7f4e740a5460 secure :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4e9019b0a0 tx=0x7f4e7c005f90 comp rx=0 tx=0).stop 2024-03-21T10:35:27.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.880+0000 7f4e94e00640 1 --2- 172.21.15.31:0/3467919744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e90103260 0x7f4e901a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:27.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.880+0000 7f4e94e00640 1 --2- 172.21.15.31:0/3467919744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e901028a0 0x7f4e9019a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:27.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.880+0000 7f4e94e00640 1 --2- 172.21.15.31:0/3467919744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e901016b0 0x7f4e90199b90 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:27.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.880+0000 7f4e94e00640 1 -- 172.21.15.31:0/3467919744 >> 172.21.15.31:0/3467919744 conn(0x7f4e900fcc20 msgr2=0x7f4e900738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:27.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.880+0000 7f4e94e00640 1 -- 172.21.15.31:0/3467919744 shutdown_connections 2024-03-21T10:35:27.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:27.881+0000 7f4e94e00640 1 -- 172.21.15.31:0/3467919744 wait complete. 2024-03-21T10:35:27.897 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T10:35:28.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.111+0000 7fec5ec00640 1 -- 172.21.15.31:0/1513242478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec580fffb0 msgr2=0x7fec581003d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:28.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.111+0000 7fec5ec00640 1 --2- 172.21.15.31:0/1513242478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec580fffb0 0x7fec581003d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fec48009900 tx=0x7fec4802bcc0 comp rx=0 tx=0).stop 2024-03-21T10:35:28.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.111+0000 7fec5ec00640 1 -- 172.21.15.31:0/1513242478 shutdown_connections 2024-03-21T10:35:28.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.111+0000 7fec5ec00640 1 --2- 172.21.15.31:0/1513242478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec581012b0 0x7fec58101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:28.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.111+0000 7fec5ec00640 1 --2- 172.21.15.31:0/1513242478 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec58100910 0x7fec58100d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:28.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.111+0000 7fec5ec00640 1 --2- 172.21.15.31:0/1513242478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec580fffb0 0x7fec581003d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:28.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.111+0000 7fec5ec00640 1 -- 172.21.15.31:0/1513242478 >> 172.21.15.31:0/1513242478 conn(0x7fec580fbb30 msgr2=0x7fec580fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:28.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.111+0000 7fec5ec00640 1 -- 172.21.15.31:0/1513242478 shutdown_connections 2024-03-21T10:35:28.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.111+0000 7fec5ec00640 1 -- 172.21.15.31:0/1513242478 wait complete. 2024-03-21T10:35:28.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.112+0000 7fec5ec00640 1 Processor -- start 2024-03-21T10:35:28.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.112+0000 7fec5ec00640 1 -- start start 2024-03-21T10:35:28.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.112+0000 7fec5ec00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec580fffb0 0x7fec58199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:28.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.112+0000 7fec5ec00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec58100910 0x7fec5819a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:28.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.112+0000 7fec5ec00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec581012b0 0x7fec581a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:28.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.112+0000 7fec5ec00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5810f590 con 0x7fec58100910 2024-03-21T10:35:28.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.112+0000 7fec5ec00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5810f410 con 0x7fec581012b0 2024-03-21T10:35:28.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.112+0000 7fec5ec00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5810f710 con 0x7fec580fffb0 2024-03-21T10:35:28.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.113+0000 7fec57e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec580fffb0 0x7fec58199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:28.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.113+0000 7fec4fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec58100910 0x7fec5819a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:28.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.113+0000 7fec5cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec581012b0 0x7fec581a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:28.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.113+0000 7fec5cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec581012b0 0x7fec581a11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49334/0 (socket says 172.21.15.31:49334) 2024-03-21T10:35:28.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.113+0000 7fec4fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec58100910 0x7fec5819a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33692/0 (socket says 172.21.15.31:33692) 2024-03-21T10:35:28.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.113+0000 7fec57e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec580fffb0 0x7fec58199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:58672/0 (socket says 172.21.15.31:58672) 2024-03-21T10:35:28.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.113+0000 7fec5cc00640 1 -- 172.21.15.31:0/3992522633 learned_addr learned my addr 172.21.15.31:0/3992522633 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:28.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.113+0000 7fec57e00640 1 -- 172.21.15.31:0/3992522633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec581012b0 msgr2=0x7fec581a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:28.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.113+0000 7fec57e00640 1 --2- 172.21.15.31:0/3992522633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec581012b0 0x7fec581a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:28.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.113+0000 7fec57e00640 1 -- 172.21.15.31:0/3992522633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec58100910 msgr2=0x7fec5819a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:28.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.113+0000 7fec57e00640 1 --2- 172.21.15.31:0/3992522633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec58100910 0x7fec5819a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:28.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.113+0000 7fec57e00640 1 -- 172.21.15.31:0/3992522633 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec581a1860 con 0x7fec580fffb0 2024-03-21T10:35:28.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.113+0000 7fec5cc00640 1 --2- 172.21.15.31:0/3992522633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec581012b0 0x7fec581a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:35:28.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.114+0000 7fec57e00640 1 --2- 172.21.15.31:0/3992522633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec580fffb0 0x7fec58199c50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fec48036040 tx=0x7fec48005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:28.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.114+0000 7fec55600640 1 -- 172.21.15.31:0/3992522633 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec48043020 con 0x7fec580fffb0 2024-03-21T10:35:28.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.114+0000 7fec55600640 1 -- 172.21.15.31:0/3992522633 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec480054e0 con 0x7fec580fffb0 2024-03-21T10:35:28.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.114+0000 7fec5ec00640 1 -- 172.21.15.31:0/3992522633 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec581a1af0 con 0x7fec580fffb0 2024-03-21T10:35:28.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.114+0000 7fec55600640 1 -- 172.21.15.31:0/3992522633 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec48003c30 con 0x7fec580fffb0 2024-03-21T10:35:28.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.114+0000 7fec5ec00640 1 -- 172.21.15.31:0/3992522633 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec581a2010 con 0x7fec580fffb0 2024-03-21T10:35:28.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.116+0000 7fec5ec00640 1 -- 172.21.15.31:0/3992522633 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec5810ad00 con 0x7fec580fffb0 2024-03-21T10:35:28.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.117+0000 7fec55600640 1 -- 172.21.15.31:0/3992522633 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fec48046020 con 0x7fec580fffb0 2024-03-21T10:35:28.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.119+0000 7fec55600640 1 --2- 172.21.15.31:0/3992522633 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fec3c0a2f20 0x7fec3c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:28.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.119+0000 7fec55600640 1 -- 172.21.15.31:0/3992522633 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fec480f1780 con 0x7fec580fffb0 2024-03-21T10:35:28.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.119+0000 7fec4fe00640 1 --2- 172.21.15.31:0/3992522633 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fec3c0a2f20 0x7fec3c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:28.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.120+0000 7fec4fe00640 1 --2- 172.21.15.31:0/3992522633 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fec3c0a2f20 0x7fec3c0a53e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fec40009c50 tx=0x7fec40005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:28.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.123+0000 7fec55600640 1 -- 172.21.15.31:0/3992522633 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fec4804d020 con 0x7fec580fffb0 2024-03-21T10:35:28.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.369+0000 7fec5ec00640 1 -- 172.21.15.31:0/3992522633 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fec580622d0 con 0x7fec580fffb0 2024-03-21T10:35:28.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.370+0000 7fec55600640 1 -- 172.21.15.31:0/3992522633 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7fec480be310 con 0x7fec580fffb0 2024-03-21T10:35:28.368 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-03-21T10:35:28.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.373+0000 7fec5ec00640 1 -- 172.21.15.31:0/3992522633 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fec3c0a2f20 msgr2=0x7fec3c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:28.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.373+0000 7fec5ec00640 1 --2- 172.21.15.31:0/3992522633 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fec3c0a2f20 0x7fec3c0a53e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fec40009c50 tx=0x7fec40005f90 comp rx=0 tx=0).stop 2024-03-21T10:35:28.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.373+0000 7fec5ec00640 1 -- 172.21.15.31:0/3992522633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec580fffb0 msgr2=0x7fec58199c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:28.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.373+0000 7fec5ec00640 1 --2- 172.21.15.31:0/3992522633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec580fffb0 0x7fec58199c50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fec48036040 tx=0x7fec48005d50 comp rx=0 tx=0).stop 2024-03-21T10:35:28.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.373+0000 7fec5ec00640 1 -- 172.21.15.31:0/3992522633 shutdown_connections 2024-03-21T10:35:28.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.373+0000 7fec5ec00640 1 --2- 172.21.15.31:0/3992522633 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fec3c0a2f20 0x7fec3c0a53e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:28.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.373+0000 7fec5ec00640 1 --2- 172.21.15.31:0/3992522633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec581012b0 0x7fec581a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:28.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.373+0000 7fec5ec00640 1 --2- 172.21.15.31:0/3992522633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec58100910 0x7fec5819a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:28.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.373+0000 7fec5ec00640 1 --2- 172.21.15.31:0/3992522633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec580fffb0 0x7fec58199c50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:28.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.373+0000 7fec5ec00640 1 -- 172.21.15.31:0/3992522633 >> 172.21.15.31:0/3992522633 conn(0x7fec580fbb30 msgr2=0x7fec580fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:28.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.373+0000 7fec5ec00640 1 -- 172.21.15.31:0/3992522633 shutdown_connections 2024-03-21T10:35:28.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.373+0000 7fec5ec00640 1 -- 172.21.15.31:0/3992522633 wait complete. 2024-03-21T10:35:28.388 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T10:35:28.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:28 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1185944224' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:28.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:28 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3467919744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:35:28.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:28 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1185944224' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:28.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:28 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3467919744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:35:28.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.590+0000 7fae14400640 1 -- 172.21.15.31:0/1239018011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae0c103260 msgr2=0x7fae0c109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:28.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.590+0000 7fae14400640 1 --2- 172.21.15.31:0/1239018011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae0c103260 0x7fae0c109af0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fae04009860 tx=0x7fae0402bb70 comp rx=0 tx=0).stop 2024-03-21T10:35:28.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.590+0000 7fae14400640 1 -- 172.21.15.31:0/1239018011 shutdown_connections 2024-03-21T10:35:28.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.590+0000 7fae14400640 1 --2- 172.21.15.31:0/1239018011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae0c103260 0x7fae0c109af0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:28.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.590+0000 7fae14400640 1 --2- 172.21.15.31:0/1239018011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae0c1028a0 0x7fae0c102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:28.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.590+0000 7fae14400640 1 --2- 172.21.15.31:0/1239018011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae0c1016b0 0x7fae0c101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:28.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.590+0000 7fae14400640 1 -- 172.21.15.31:0/1239018011 >> 172.21.15.31:0/1239018011 conn(0x7fae0c0fcc20 msgr2=0x7fae0c0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:28.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.590+0000 7fae14400640 1 -- 172.21.15.31:0/1239018011 shutdown_connections 2024-03-21T10:35:28.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.590+0000 7fae14400640 1 -- 172.21.15.31:0/1239018011 wait complete. 2024-03-21T10:35:28.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.591+0000 7fae14400640 1 Processor -- start 2024-03-21T10:35:28.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.591+0000 7fae14400640 1 -- start start 2024-03-21T10:35:28.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.591+0000 7fae14400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae0c1016b0 0x7fae0c197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:28.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.591+0000 7fae14400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae0c1028a0 0x7fae0c197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:28.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.592+0000 7fae14400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae0c103260 0x7fae0c19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:28.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.592+0000 7fae14400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0c10d330 con 0x7fae0c1016b0 2024-03-21T10:35:28.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.592+0000 7fae14400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0c10d1b0 con 0x7fae0c103260 2024-03-21T10:35:28.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.592+0000 7fae14400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0c10d4b0 con 0x7fae0c1028a0 2024-03-21T10:35:28.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.592+0000 7fae11000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae0c1028a0 0x7fae0c197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:28.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.592+0000 7fae11a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae0c1016b0 0x7fae0c197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:28.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.592+0000 7fae11000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae0c1028a0 0x7fae0c197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:58698/0 (socket says 172.21.15.31:58698) 2024-03-21T10:35:28.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.592+0000 7fae11000640 1 -- 172.21.15.31:0/1066694474 learned_addr learned my addr 172.21.15.31:0/1066694474 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:28.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.592+0000 7fae12400640 1 --2- 172.21.15.31:0/1066694474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae0c103260 0x7fae0c19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:28.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.592+0000 7fae11000640 1 -- 172.21.15.31:0/1066694474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae0c103260 msgr2=0x7fae0c19ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:28.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.592+0000 7fae11000640 1 --2- 172.21.15.31:0/1066694474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae0c103260 0x7fae0c19ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:28.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.592+0000 7fae11000640 1 -- 172.21.15.31:0/1066694474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae0c1016b0 msgr2=0x7fae0c197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:28.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.592+0000 7fae11000640 1 --2- 172.21.15.31:0/1066694474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae0c1016b0 0x7fae0c197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:28.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.592+0000 7fae11000640 1 -- 172.21.15.31:0/1066694474 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae0c19f5a0 con 0x7fae0c1028a0 2024-03-21T10:35:28.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.593+0000 7fae11000640 1 --2- 172.21.15.31:0/1066694474 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae0c1028a0 0x7fae0c197ec0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fadf800e9f0 tx=0x7fadf800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:28.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.593+0000 7fae0a000640 1 -- 172.21.15.31:0/1066694474 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf800cd80 con 0x7fae0c1028a0 2024-03-21T10:35:28.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.593+0000 7fae14400640 1 -- 172.21.15.31:0/1066694474 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae0c19f890 con 0x7fae0c1028a0 2024-03-21T10:35:28.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.593+0000 7fae0a000640 1 -- 172.21.15.31:0/1066694474 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fadf8005500 con 0x7fae0c1028a0 2024-03-21T10:35:28.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.593+0000 7fae0a000640 1 -- 172.21.15.31:0/1066694474 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf8010690 con 0x7fae0c1028a0 2024-03-21T10:35:28.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.593+0000 7fae14400640 1 -- 172.21.15.31:0/1066694474 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae0c19fd90 con 0x7fae0c1028a0 2024-03-21T10:35:28.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.594+0000 7fae14400640 1 -- 172.21.15.31:0/1066694474 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae0c1089d0 con 0x7fae0c1028a0 2024-03-21T10:35:28.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.596+0000 7fae0a000640 1 -- 172.21.15.31:0/1066694474 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fadf80046c0 con 0x7fae0c1028a0 2024-03-21T10:35:28.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.597+0000 7fae0a000640 1 --2- 172.21.15.31:0/1066694474 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fadf40a2f20 0x7fadf40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:28.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.597+0000 7fae0a000640 1 -- 172.21.15.31:0/1066694474 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fadf80cfa80 con 0x7fae0c1028a0 2024-03-21T10:35:28.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.598+0000 7fae11a00640 1 --2- 172.21.15.31:0/1066694474 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fadf40a2f20 0x7fadf40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:28.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.598+0000 7fae11a00640 1 --2- 172.21.15.31:0/1066694474 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fadf40a2f20 0x7fadf40a53e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fae0c102700 tx=0x7fae00010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:28.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.601+0000 7fae0a000640 1 -- 172.21.15.31:0/1066694474 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fadf80d1e20 con 0x7fae0c1028a0 2024-03-21T10:35:28.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:28 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1185944224' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:28.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:28 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3467919744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:35:28.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.801+0000 7fae14400640 1 -- 172.21.15.31:0/1066694474 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fae0c0622a0 con 0x7fae0c1028a0 2024-03-21T10:35:28.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.801+0000 7fae0a000640 1 -- 172.21.15.31:0/1066694474 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7fadf809c610 con 0x7fae0c1028a0 2024-03-21T10:35:28.800 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-03-21T10:35:28.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.804+0000 7fae14400640 1 -- 172.21.15.31:0/1066694474 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fadf40a2f20 msgr2=0x7fadf40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:28.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.804+0000 7fae14400640 1 --2- 172.21.15.31:0/1066694474 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fadf40a2f20 0x7fadf40a53e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fae0c102700 tx=0x7fae00010040 comp rx=0 tx=0).stop 2024-03-21T10:35:28.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.804+0000 7fae14400640 1 -- 172.21.15.31:0/1066694474 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae0c1028a0 msgr2=0x7fae0c197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:28.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.804+0000 7fae14400640 1 --2- 172.21.15.31:0/1066694474 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae0c1028a0 0x7fae0c197ec0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fadf800e9f0 tx=0x7fadf800edf0 comp rx=0 tx=0).stop 2024-03-21T10:35:28.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.804+0000 7fae14400640 1 -- 172.21.15.31:0/1066694474 shutdown_connections 2024-03-21T10:35:28.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.804+0000 7fae14400640 1 --2- 172.21.15.31:0/1066694474 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fadf40a2f20 0x7fadf40a53e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:28.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.804+0000 7fae14400640 1 --2- 172.21.15.31:0/1066694474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae0c103260 0x7fae0c19ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:28.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.804+0000 7fae14400640 1 --2- 172.21.15.31:0/1066694474 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae0c1028a0 0x7fae0c197ec0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:28.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.804+0000 7fae14400640 1 --2- 172.21.15.31:0/1066694474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae0c1016b0 0x7fae0c197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:28.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.804+0000 7fae14400640 1 -- 172.21.15.31:0/1066694474 >> 172.21.15.31:0/1066694474 conn(0x7fae0c0fcc20 msgr2=0x7fae0c105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:28.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.804+0000 7fae14400640 1 -- 172.21.15.31:0/1066694474 shutdown_connections 2024-03-21T10:35:28.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:28.804+0000 7fae14400640 1 -- 172.21.15.31:0/1066694474 wait complete. 2024-03-21T10:35:28.820 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T10:35:29.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.036+0000 7fcc82e00640 1 -- 172.21.15.31:0/1723783662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc7c107b00 msgr2=0x7fcc7c107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:29.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.036+0000 7fcc82e00640 1 --2- 172.21.15.31:0/1723783662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc7c107b00 0x7fcc7c107f20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcc70009900 tx=0x7fcc7002bc30 comp rx=0 tx=0).stop 2024-03-21T10:35:29.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.036+0000 7fcc82e00640 1 -- 172.21.15.31:0/1723783662 shutdown_connections 2024-03-21T10:35:29.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.036+0000 7fcc82e00640 1 --2- 172.21.15.31:0/1723783662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc7c1096b0 0x7fcc7c10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:29.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.036+0000 7fcc82e00640 1 --2- 172.21.15.31:0/1723783662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c108cf0 0x7fcc7c109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:29.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.036+0000 7fcc82e00640 1 --2- 172.21.15.31:0/1723783662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc7c107b00 0x7fcc7c107f20 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:29.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.036+0000 7fcc82e00640 1 -- 172.21.15.31:0/1723783662 >> 172.21.15.31:0/1723783662 conn(0x7fcc7c076a90 msgr2=0x7fcc7c078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:29.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.036+0000 7fcc82e00640 1 -- 172.21.15.31:0/1723783662 shutdown_connections 2024-03-21T10:35:29.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.036+0000 7fcc82e00640 1 -- 172.21.15.31:0/1723783662 wait complete. 2024-03-21T10:35:29.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.037+0000 7fcc82e00640 1 Processor -- start 2024-03-21T10:35:29.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.037+0000 7fcc82e00640 1 -- start start 2024-03-21T10:35:29.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.038+0000 7fcc82e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c107b00 0x7fcc7c19de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:29.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.038+0000 7fcc82e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc7c108cf0 0x7fcc7c19e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:29.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.038+0000 7fcc82e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc7c1096b0 0x7fcc7c1a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:29.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.038+0000 7fcc82e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c113780 con 0x7fcc7c1096b0 2024-03-21T10:35:29.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.038+0000 7fcc82e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c113600 con 0x7fcc7c107b00 2024-03-21T10:35:29.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.038+0000 7fcc82e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c113900 con 0x7fcc7c108cf0 2024-03-21T10:35:29.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.038+0000 7fcc81a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c107b00 0x7fcc7c19de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:29.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.038+0000 7fcc82400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc7c1096b0 0x7fcc7c1a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:29.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.038+0000 7fcc81a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c107b00 0x7fcc7c19de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49378/0 (socket says 172.21.15.31:49378) 2024-03-21T10:35:29.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.038+0000 7fcc81a00640 1 -- 172.21.15.31:0/3739992560 learned_addr learned my addr 172.21.15.31:0/3739992560 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:29.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.038+0000 7fcc81000640 1 --2- 172.21.15.31:0/3739992560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc7c108cf0 0x7fcc7c19e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:29.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.038+0000 7fcc81a00640 1 -- 172.21.15.31:0/3739992560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc7c108cf0 msgr2=0x7fcc7c19e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:29.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.038+0000 7fcc81a00640 1 --2- 172.21.15.31:0/3739992560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc7c108cf0 0x7fcc7c19e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:29.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.038+0000 7fcc81a00640 1 -- 172.21.15.31:0/3739992560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc7c1096b0 msgr2=0x7fcc7c1a53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:29.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.038+0000 7fcc81a00640 1 --2- 172.21.15.31:0/3739992560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc7c1096b0 0x7fcc7c1a53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:29.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.038+0000 7fcc81a00640 1 -- 172.21.15.31:0/3739992560 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7c1a5a40 con 0x7fcc7c107b00 2024-03-21T10:35:29.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.039+0000 7fcc81a00640 1 --2- 172.21.15.31:0/3739992560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c107b00 0x7fcc7c19de70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcc70000c00 tx=0x7fcc7002c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:29.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.039+0000 7fcc7a000640 1 -- 172.21.15.31:0/3739992560 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc70041070 con 0x7fcc7c107b00 2024-03-21T10:35:29.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.039+0000 7fcc7a000640 1 -- 172.21.15.31:0/3739992560 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc70004450 con 0x7fcc7c107b00 2024-03-21T10:35:29.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.039+0000 7fcc7a000640 1 -- 172.21.15.31:0/3739992560 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc70034440 con 0x7fcc7c107b00 2024-03-21T10:35:29.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.039+0000 7fcc82e00640 1 -- 172.21.15.31:0/3739992560 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc7c1a5cd0 con 0x7fcc7c107b00 2024-03-21T10:35:29.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.039+0000 7fcc82e00640 1 -- 172.21.15.31:0/3739992560 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc7c1a61c0 con 0x7fcc7c107b00 2024-03-21T10:35:29.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.040+0000 7fcc82e00640 1 -- 172.21.15.31:0/3739992560 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc7c10ee20 con 0x7fcc7c107b00 2024-03-21T10:35:29.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.042+0000 7fcc7a000640 1 -- 172.21.15.31:0/3739992560 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fcc70044070 con 0x7fcc7c107b00 2024-03-21T10:35:29.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.043+0000 7fcc7a000640 1 --2- 172.21.15.31:0/3739992560 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcc640a2fa0 0x7fcc640a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:29.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.043+0000 7fcc7a000640 1 -- 172.21.15.31:0/3739992560 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fcc700f10c0 con 0x7fcc7c107b00 2024-03-21T10:35:29.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.043+0000 7fcc81000640 1 --2- 172.21.15.31:0/3739992560 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcc640a2fa0 0x7fcc640a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:29.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.044+0000 7fcc81000640 1 --2- 172.21.15.31:0/3739992560 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcc640a2fa0 0x7fcc640a5460 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c0095f0 tx=0x7fcc6c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:29.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.047+0000 7fcc7a000640 1 -- 172.21.15.31:0/3739992560 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcc700bd690 con 0x7fcc7c107b00 2024-03-21T10:35:29.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.245+0000 7fcc82e00640 1 -- 172.21.15.31:0/3739992560 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fcc7c062340 con 0x7fcc7c107b00 2024-03-21T10:35:29.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.253+0000 7fcc7a000640 1 -- 172.21.15.31:0/3739992560 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7fcc700c1730 con 0x7fcc7c107b00 2024-03-21T10:35:29.252 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-03-21T10:35:29.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.256+0000 7fcc82e00640 1 -- 172.21.15.31:0/3739992560 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcc640a2fa0 msgr2=0x7fcc640a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:29.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.256+0000 7fcc82e00640 1 --2- 172.21.15.31:0/3739992560 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcc640a2fa0 0x7fcc640a5460 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c0095f0 tx=0x7fcc6c005f90 comp rx=0 tx=0).stop 2024-03-21T10:35:29.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.256+0000 7fcc82e00640 1 -- 172.21.15.31:0/3739992560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c107b00 msgr2=0x7fcc7c19de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:29.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.256+0000 7fcc82e00640 1 --2- 172.21.15.31:0/3739992560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c107b00 0x7fcc7c19de70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcc70000c00 tx=0x7fcc7002c640 comp rx=0 tx=0).stop 2024-03-21T10:35:29.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.256+0000 7fcc82e00640 1 -- 172.21.15.31:0/3739992560 shutdown_connections 2024-03-21T10:35:29.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.256+0000 7fcc82e00640 1 --2- 172.21.15.31:0/3739992560 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcc640a2fa0 0x7fcc640a5460 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:29.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.256+0000 7fcc82e00640 1 --2- 172.21.15.31:0/3739992560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc7c1096b0 0x7fcc7c1a53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:29.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.256+0000 7fcc82e00640 1 --2- 172.21.15.31:0/3739992560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc7c108cf0 0x7fcc7c19e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:29.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.256+0000 7fcc82e00640 1 --2- 172.21.15.31:0/3739992560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c107b00 0x7fcc7c19de70 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:29.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.256+0000 7fcc82e00640 1 -- 172.21.15.31:0/3739992560 >> 172.21.15.31:0/3739992560 conn(0x7fcc7c076a90 msgr2=0x7fcc7c10bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:29.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.256+0000 7fcc82e00640 1 -- 172.21.15.31:0/3739992560 shutdown_connections 2024-03-21T10:35:29.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.256+0000 7fcc82e00640 1 -- 172.21.15.31:0/3739992560 wait complete. 2024-03-21T10:35:29.273 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-21T10:35:29.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.496+0000 7f4512200640 1 -- 172.21.15.31:0/228910669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f450c1048a0 msgr2=0x7f450c106c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:29.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.496+0000 7f4512200640 1 --2- 172.21.15.31:0/228910669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f450c1048a0 0x7f450c106c90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f44f8009900 tx=0x7f44f802bc50 comp rx=0 tx=0).stop 2024-03-21T10:35:29.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.496+0000 7f4512200640 1 -- 172.21.15.31:0/228910669 shutdown_connections 2024-03-21T10:35:29.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.496+0000 7f4512200640 1 --2- 172.21.15.31:0/228910669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f450c1071d0 0x7f450c109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:29.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.496+0000 7f4512200640 1 --2- 172.21.15.31:0/228910669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f450c1048a0 0x7f450c106c90 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:29.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.496+0000 7f4512200640 1 --2- 172.21.15.31:0/228910669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f450c101f70 0x7f450c104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:29.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.496+0000 7f4512200640 1 -- 172.21.15.31:0/228910669 >> 172.21.15.31:0/228910669 conn(0x7f450c0fb980 msgr2=0x7f450c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:29.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.496+0000 7f4512200640 1 -- 172.21.15.31:0/228910669 shutdown_connections 2024-03-21T10:35:29.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.496+0000 7f4512200640 1 -- 172.21.15.31:0/228910669 wait complete. 2024-03-21T10:35:29.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.497+0000 7f4512200640 1 Processor -- start 2024-03-21T10:35:29.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.497+0000 7f4512200640 1 -- start start 2024-03-21T10:35:29.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.498+0000 7f4512200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f450c101f70 0x7f450c195670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:29.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.498+0000 7f4512200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f450c1048a0 0x7f450c195bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:29.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.498+0000 7f4512200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f450c1071d0 0x7f450c1a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:29.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.498+0000 7f4512200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f450c10d1d0 con 0x7f450c1048a0 2024-03-21T10:35:29.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.498+0000 7f4512200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f450c10d050 con 0x7f450c101f70 2024-03-21T10:35:29.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.498+0000 7f4512200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f450c10d350 con 0x7f450c1071d0 2024-03-21T10:35:29.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.498+0000 7f4510e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f450c101f70 0x7f450c195670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:29.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.498+0000 7f450be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f450c1048a0 0x7f450c195bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:29.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.498+0000 7f4510e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f450c101f70 0x7f450c195670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49390/0 (socket says 172.21.15.31:49390) 2024-03-21T10:35:29.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.498+0000 7f4510e00640 1 -- 172.21.15.31:0/2679141297 learned_addr learned my addr 172.21.15.31:0/2679141297 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:29.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.498+0000 7f4511800640 1 --2- 172.21.15.31:0/2679141297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f450c1071d0 0x7f450c1a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:29.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.498+0000 7f4510e00640 1 -- 172.21.15.31:0/2679141297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f450c1071d0 msgr2=0x7f450c1a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:29.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.498+0000 7f4510e00640 1 --2- 172.21.15.31:0/2679141297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f450c1071d0 0x7f450c1a54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:29.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.498+0000 7f4510e00640 1 -- 172.21.15.31:0/2679141297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f450c1048a0 msgr2=0x7f450c195bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:29.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.498+0000 7f4510e00640 1 --2- 172.21.15.31:0/2679141297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f450c1048a0 0x7f450c195bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:29.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.498+0000 7f4510e00640 1 -- 172.21.15.31:0/2679141297 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f450c1a5ad0 con 0x7f450c101f70 2024-03-21T10:35:29.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.499+0000 7f4510e00640 1 --2- 172.21.15.31:0/2679141297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f450c101f70 0x7f450c195670 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f450000d770 tx=0x7f450000dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:29.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.499+0000 7f4509600640 1 -- 172.21.15.31:0/2679141297 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4500017070 con 0x7f450c101f70 2024-03-21T10:35:29.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.499+0000 7f4509600640 1 -- 172.21.15.31:0/2679141297 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4500005500 con 0x7f450c101f70 2024-03-21T10:35:29.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.499+0000 7f4512200640 1 -- 172.21.15.31:0/2679141297 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f450c1a5dc0 con 0x7f450c101f70 2024-03-21T10:35:29.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.499+0000 7f4509600640 1 -- 172.21.15.31:0/2679141297 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4500004e00 con 0x7f450c101f70 2024-03-21T10:35:29.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.499+0000 7f4512200640 1 -- 172.21.15.31:0/2679141297 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f450c1a6220 con 0x7f450c101f70 2024-03-21T10:35:29.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.500+0000 7f4512200640 1 -- 172.21.15.31:0/2679141297 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f450c108d20 con 0x7f450c101f70 2024-03-21T10:35:29.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.502+0000 7f4509600640 1 -- 172.21.15.31:0/2679141297 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f4500004fa0 con 0x7f450c101f70 2024-03-21T10:35:29.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.503+0000 7f4509600640 1 --2- 172.21.15.31:0/2679141297 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f44f40a2fa0 0x7f44f40a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:29.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.504+0000 7f4509600640 1 -- 172.21.15.31:0/2679141297 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f45000cefb0 con 0x7f450c101f70 2024-03-21T10:35:29.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.504+0000 7f450be00640 1 --2- 172.21.15.31:0/2679141297 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f44f40a2fa0 0x7f44f40a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:29.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.504+0000 7f450be00640 1 --2- 172.21.15.31:0/2679141297 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f44f40a2fa0 0x7f44f40a5460 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f44f802c190 tx=0x7f44f8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:29.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.507+0000 7f4509600640 1 -- 172.21.15.31:0/2679141297 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f450009b4f0 con 0x7f450c101f70 2024-03-21T10:35:29.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:29 smithi031 ceph-mon[31673]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-21T10:35:29.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:29 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3992522633' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:35:29.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:29 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1066694474' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:35:29.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:29 smithi070 ceph-mon[36330]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-21T10:35:29.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:29 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3992522633' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:35:29.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:29 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1066694474' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:35:29.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:29 smithi040 ceph-mon[36154]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-21T10:35:29.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:29 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3992522633' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:35:29.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:29 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1066694474' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:35:29.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.726+0000 7f4512200640 1 -- 172.21.15.31:0/2679141297 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f450c0622a0 con 0x7f450c101f70 2024-03-21T10:35:29.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.727+0000 7f4509600640 1 -- 172.21.15.31:0/2679141297 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+591 (secure 0 0 0) 0x7f45000bb030 con 0x7f450c101f70 2024-03-21T10:35:29.725 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/1451930554 2024-03-22T10:26:53.842623+0000 2024-03-21T10:35:29.725 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/3342588843 2024-03-22T10:26:53.842623+0000 2024-03-21T10:35:29.726 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6801/4085461895 2024-03-22T10:26:53.842623+0000 2024-03-21T10:35:29.726 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6800/4085461895 2024-03-22T10:26:53.842623+0000 2024-03-21T10:35:29.726 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/1626448389 2024-03-22T10:26:16.959165+0000 2024-03-21T10:35:29.726 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/1910059351 2024-03-22T10:26:16.959165+0000 2024-03-21T10:35:29.726 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/344822770 2024-03-22T10:26:16.959165+0000 2024-03-21T10:35:29.726 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/3811368512 2024-03-22T10:26:53.842623+0000 2024-03-21T10:35:29.726 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6800/1981437651 2024-03-22T10:26:16.959165+0000 2024-03-21T10:35:29.726 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6801/1981437651 2024-03-22T10:26:16.959165+0000 2024-03-21T10:35:29.726 INFO:teuthology.orchestra.run.smithi031.stderr:listed 10 entries 2024-03-21T10:35:29.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.730+0000 7f4512200640 1 -- 172.21.15.31:0/2679141297 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f44f40a2fa0 msgr2=0x7f44f40a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:29.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.730+0000 7f4512200640 1 --2- 172.21.15.31:0/2679141297 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f44f40a2fa0 0x7f44f40a5460 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f44f802c190 tx=0x7f44f8000bc0 comp rx=0 tx=0).stop 2024-03-21T10:35:29.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.730+0000 7f4512200640 1 -- 172.21.15.31:0/2679141297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f450c101f70 msgr2=0x7f450c195670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:29.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.730+0000 7f4512200640 1 --2- 172.21.15.31:0/2679141297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f450c101f70 0x7f450c195670 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f450000d770 tx=0x7f450000dc30 comp rx=0 tx=0).stop 2024-03-21T10:35:29.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.730+0000 7f4512200640 1 -- 172.21.15.31:0/2679141297 shutdown_connections 2024-03-21T10:35:29.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.730+0000 7f4512200640 1 --2- 172.21.15.31:0/2679141297 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f44f40a2fa0 0x7f44f40a5460 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:29.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.730+0000 7f4512200640 1 --2- 172.21.15.31:0/2679141297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f450c1071d0 0x7f450c1a54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:29.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.730+0000 7f4512200640 1 --2- 172.21.15.31:0/2679141297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f450c1048a0 0x7f450c195bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:29.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.730+0000 7f4512200640 1 --2- 172.21.15.31:0/2679141297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f450c101f70 0x7f450c195670 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:29.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.730+0000 7f4512200640 1 -- 172.21.15.31:0/2679141297 >> 172.21.15.31:0/2679141297 conn(0x7f450c0fb980 msgr2=0x7f450c101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:29.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.730+0000 7f4512200640 1 -- 172.21.15.31:0/2679141297 shutdown_connections 2024-03-21T10:35:29.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:29.730+0000 7f4512200640 1 -- 172.21.15.31:0/2679141297 wait complete. 2024-03-21T10:35:29.744 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi031.front.sepia.ceph.com... 2024-03-21T10:35:29.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:35:29.745 DEBUG:teuthology.orchestra.run.smithi031:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-21T10:35:29.835 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi031.stderr:ceph-fuse[74852]: fuse finished with error 0 and tester_r 0 2024-03-21T10:35:29.840 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T10:35:30.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:30 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3739992560' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:35:30.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:30 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2679141297' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T10:35:30.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:30 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3739992560' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:35:30.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:30 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2679141297' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T10:35:30.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:30 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3739992560' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:35:30.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:30 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2679141297' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T10:35:31.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:31 smithi031 ceph-mon[31673]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-21T10:35:31.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:31 smithi070 ceph-mon[36330]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-21T10:35:31.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:31 smithi040 ceph-mon[36154]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-21T10:35:33.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:33 smithi031 ceph-mon[31673]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-21T10:35:33.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:33 smithi070 ceph-mon[36330]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-21T10:35:33.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:33 smithi040 ceph-mon[36154]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-21T10:35:35.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:35 smithi031 ceph-mon[31673]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-21T10:35:35.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:35 smithi070 ceph-mon[36330]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-21T10:35:35.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:35 smithi040 ceph-mon[36154]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-21T10:35:35.942 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi031.front.sepia.ceph.com 2024-03-21T10:35:35.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:35:35.942 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-21T10:35:36.030 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T10:35:36.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.263+0000 7f991c800640 1 -- 172.21.15.31:0/4234497098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99141032e0 msgr2=0x7f9914109b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:36.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.263+0000 7f991c800640 1 --2- 172.21.15.31:0/4234497098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99141032e0 0x7f9914109b70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9910009ca0 tx=0x7f991002c100 comp rx=0 tx=0).stop 2024-03-21T10:35:36.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.264+0000 7f991c800640 1 -- 172.21.15.31:0/4234497098 shutdown_connections 2024-03-21T10:35:36.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.264+0000 7f991c800640 1 --2- 172.21.15.31:0/4234497098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99141032e0 0x7f9914109b70 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:36.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.264+0000 7f991c800640 1 --2- 172.21.15.31:0/4234497098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9914102920 0x7f9914102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:36.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.264+0000 7f991c800640 1 --2- 172.21.15.31:0/4234497098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9914101730 0x7f9914101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:36.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.264+0000 7f991c800640 1 -- 172.21.15.31:0/4234497098 >> 172.21.15.31:0/4234497098 conn(0x7f99140fcc80 msgr2=0x7f99140ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:36.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.264+0000 7f991c800640 1 -- 172.21.15.31:0/4234497098 shutdown_connections 2024-03-21T10:35:36.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.264+0000 7f991c800640 1 -- 172.21.15.31:0/4234497098 wait complete. 2024-03-21T10:35:36.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.265+0000 7f991c800640 1 Processor -- start 2024-03-21T10:35:36.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.265+0000 7f991c800640 1 -- start start 2024-03-21T10:35:36.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.265+0000 7f991c800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9914101730 0x7f9914199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:36.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.265+0000 7f991c800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9914102920 0x7f991419a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:36.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.265+0000 7f991c800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99141032e0 0x7f99141a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:36.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.265+0000 7f991c800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991410d340 con 0x7f99141032e0 2024-03-21T10:35:36.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.265+0000 7f991c800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991410d1c0 con 0x7f9914101730 2024-03-21T10:35:36.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.265+0000 7f991c800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991410d4c0 con 0x7f9914102920 2024-03-21T10:35:36.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.265+0000 7f9919e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9914101730 0x7f9914199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:36.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.265+0000 7f9919e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9914101730 0x7f9914199c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49404/0 (socket says 172.21.15.31:49404) 2024-03-21T10:35:36.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.265+0000 7f9919e00640 1 -- 172.21.15.31:0/1721008909 learned_addr learned my addr 172.21.15.31:0/1721008909 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:36.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.265+0000 7f991a800640 1 --2- 172.21.15.31:0/1721008909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99141032e0 0x7f99141a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:36.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.266+0000 7f9919400640 1 --2- 172.21.15.31:0/1721008909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9914102920 0x7f991419a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:36.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.266+0000 7f9919e00640 1 -- 172.21.15.31:0/1721008909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9914102920 msgr2=0x7f991419a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:36.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.266+0000 7f9919e00640 1 --2- 172.21.15.31:0/1721008909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9914102920 0x7f991419a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:36.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.266+0000 7f9919e00640 1 -- 172.21.15.31:0/1721008909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99141032e0 msgr2=0x7f99141a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:36.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.266+0000 7f9919e00640 1 --2- 172.21.15.31:0/1721008909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99141032e0 0x7f99141a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:36.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.266+0000 7f9919e00640 1 -- 172.21.15.31:0/1721008909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99141a1840 con 0x7f9914101730 2024-03-21T10:35:36.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.266+0000 7f9919e00640 1 --2- 172.21.15.31:0/1721008909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9914101730 0x7f9914199c30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f990800e9c0 tx=0x7f990800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:36.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.266+0000 7f990ea00640 1 -- 172.21.15.31:0/1721008909 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990800cde0 con 0x7f9914101730 2024-03-21T10:35:36.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.266+0000 7f991c800640 1 -- 172.21.15.31:0/1721008909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99141a1b30 con 0x7f9914101730 2024-03-21T10:35:36.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.267+0000 7f990ea00640 1 -- 172.21.15.31:0/1721008909 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9908010910 con 0x7f9914101730 2024-03-21T10:35:36.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.267+0000 7f990ea00640 1 -- 172.21.15.31:0/1721008909 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9908018a10 con 0x7f9914101730 2024-03-21T10:35:36.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.267+0000 7f991c800640 1 -- 172.21.15.31:0/1721008909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99141a20d0 con 0x7f9914101730 2024-03-21T10:35:36.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.268+0000 7f991c800640 1 -- 172.21.15.31:0/1721008909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99141a22a0 con 0x7f9914101730 2024-03-21T10:35:36.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.270+0000 7f990ea00640 1 -- 172.21.15.31:0/1721008909 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f990801f020 con 0x7f9914101730 2024-03-21T10:35:36.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.271+0000 7f990ea00640 1 --2- 172.21.15.31:0/1721008909 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f98fc0a2f20 0x7f98fc0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:36.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.271+0000 7f990ea00640 1 -- 172.21.15.31:0/1721008909 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f99080cfa20 con 0x7f9914101730 2024-03-21T10:35:36.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.271+0000 7f9919400640 1 --2- 172.21.15.31:0/1721008909 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f98fc0a2f20 0x7f98fc0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:36.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.272+0000 7f9919400640 1 --2- 172.21.15.31:0/1721008909 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f98fc0a2f20 0x7f98fc0a53e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9904009b50 tx=0x7f9904005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:36.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.278+0000 7f990ea00640 1 -- 172.21.15.31:0/1721008909 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f990809c5b0 con 0x7f9914101730 2024-03-21T10:35:36.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.482+0000 7f991c800640 1 -- 172.21.15.31:0/1721008909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99141a2970 con 0x7f9914101730 2024-03-21T10:35:36.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.483+0000 7f990ea00640 1 -- 172.21.15.31:0/1721008909 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24009 (secure 0 0 0) 0x7f990809beb0 con 0x7f9914101730 2024-03-21T10:35:36.483 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:35:36.484 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","created":"2024-03-21T10:25:41.971117+0000","modified":"2024-03-21T10:35:11.657050+0000","last_up_change":"2024-03-21T10:34:07.757196+0000","last_in_change":"2024-03-21T10:33:47.460406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T10:29:44.553968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T10:35:04.945449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T10:35:05.970589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T10:35:09.688711+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e0244d33-00e2-47d5-9f66-34efc138c462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6803","nonce":3089885125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6805","nonce":3089885125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6809","nonce":3089885125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6807","nonce":3089885125}]},"public_addr":"172.21.15.31:6803/3089885125","cluster_addr":"172.21.15.31:6805/3089885125","heartbeat_back_addr":"172.21.15.31:6809/3089885125","heartbeat_front_addr":"172.21.15.31:6807/3089885125","state":["exists","up"]},{"osd":1,"uuid":"76a63504-beab-4354-a6c3-eebc3b16ad03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6801","nonce":618840037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6803","nonce":618840037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6807","nonce":618840037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6805","nonce":618840037}]},"public_addr":"172.21.15.40:6801/618840037","cluster_addr":"172.21.15.40:6803/618840037","heartbeat_back_addr":"172.21.15.40:6807/618840037","heartbeat_front_addr":"172.21.15.40:6805/618840037","state":["exists","up"]},{"osd":2,"uuid":"fa191bd6-9a21-4ee2-8adf-8a63c9cfd821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6801","nonce":2524184117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6803","nonce":2524184117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6807","nonce":2524184117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6805","nonce":2524184117}]},"public_addr":"172.21.15.70:6801/2524184117","cluster_addr":"172.21.15.70:6803/2524184117","heartbeat_back_addr":"172.21.15.70:6807/2524184117","heartbeat_front_addr":"172.21.15.70:6805/2524184117","state":["exists","up"]},{"osd":3,"uuid":"edb44507-ffb0-42a4-827b-08461e1bf952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6811","nonce":2823036296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6813","nonce":2823036296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6817","nonce":2823036296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6815","nonce":2823036296}]},"public_addr":"172.21.15.31:6811/2823036296","cluster_addr":"172.21.15.31:6813/2823036296","heartbeat_back_addr":"172.21.15.31:6817/2823036296","heartbeat_front_addr":"172.21.15.31:6815/2823036296","state":["exists","up"]},{"osd":4,"uuid":"8cb89153-8c3b-431f-b57c-8863fada76dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6809","nonce":4215186309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6811","nonce":4215186309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6815","nonce":4215186309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6813","nonce":4215186309}]},"public_addr":"172.21.15.40:6809/4215186309","cluster_addr":"172.21.15.40:6811/4215186309","heartbeat_back_addr":"172.21.15.40:6815/4215186309","heartbeat_front_addr":"172.21.15.40:6813/4215186309","state":["exists","up"]},{"osd":5,"uuid":"9409505a-d463-408c-a3cd-218380c04d68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6809","nonce":3402830452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6811","nonce":3402830452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6815","nonce":3402830452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6813","nonce":3402830452}]},"public_addr":"172.21.15.70:6809/3402830452","cluster_addr":"172.21.15.70:6811/3402830452","heartbeat_back_addr":"172.21.15.70:6815/3402830452","heartbeat_front_addr":"172.21.15.70:6813/3402830452","state":["exists","up"]},{"osd":6,"uuid":"f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6819","nonce":1193497058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6821","nonce":1193497058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6825","nonce":1193497058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6823","nonce":1193497058}]},"public_addr":"172.21.15.31:6819/1193497058","cluster_addr":"172.21.15.31:6821/1193497058","heartbeat_back_addr":"172.21.15.31:6825/1193497058","heartbeat_front_addr":"172.21.15.31:6823/1193497058","state":["exists","up"]},{"osd":7,"uuid":"db228a87-29a5-45d4-8fe6-a0a554262eb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6817","nonce":3825628119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6819","nonce":3825628119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6823","nonce":3825628119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6821","nonce":3825628119}]},"public_addr":"172.21.15.40:6817/3825628119","cluster_addr":"172.21.15.40:6819/3825628119","heartbeat_back_addr":"172.21.15.40:6823/3825628119","heartbeat_front_addr":"172.21.15.40:6821/3825628119","state":["exists","up"]},{"osd":8,"uuid":"59b9fde0-095d-4e0b-87da-ead37bd8ff1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6817","nonce":534880586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6819","nonce":534880586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6823","nonce":534880586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6821","nonce":534880586}]},"public_addr":"172.21.15.70:6817/534880586","cluster_addr":"172.21.15.70:6819/534880586","heartbeat_back_addr":"172.21.15.70:6823/534880586","heartbeat_front_addr":"172.21.15.70:6821/534880586","state":["exists","up"]},{"osd":9,"uuid":"77f47061-0c5f-4159-b276-e814935d6c1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6827","nonce":587047469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6829","nonce":587047469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6833","nonce":587047469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6831","nonce":587047469}]},"public_addr":"172.21.15.31:6827/587047469","cluster_addr":"172.21.15.31:6829/587047469","heartbeat_back_addr":"172.21.15.31:6833/587047469","heartbeat_front_addr":"172.21.15.31:6831/587047469","state":["exists","up"]},{"osd":10,"uuid":"8853dc43-7bd1-4c93-8063-f7982fd7da38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6825","nonce":512108711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6827","nonce":512108711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6831","nonce":512108711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6829","nonce":512108711}]},"public_addr":"172.21.15.40:6825/512108711","cluster_addr":"172.21.15.40:6827/512108711","heartbeat_back_addr":"172.21.15.40:6831/512108711","heartbeat_front_addr":"172.21.15.40:6829/512108711","state":["exists","up"]},{"osd":11,"uuid":"84b7c2de-c424-45a6-9583-60097e372a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6825","nonce":540305718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6827","nonce":540305718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6831","nonce":540305718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6829","nonce":540305718}]},"public_addr":"172.21.15.70:6825/540305718","cluster_addr":"172.21.15.70:6827/540305718","heartbeat_back_addr":"172.21.15.70:6831/540305718","heartbeat_front_addr":"172.21.15.70:6829/540305718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:28:51.477358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:17.405045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:41.626268+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:15.339568+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:42.676996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:07.516999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:42.412273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:10.365041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:35.699120+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:11.733639+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:39.505442+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:34:05.449301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451930554":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/3342588843":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6801/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/1626448389":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/1910059351":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/344822770":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/3811368512":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/1981437651":"2024-03-22T10:26:16.959165+0000","172.21.15.31:6801/1981437651":"2024-03-22T10:26:16.959165+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T10:35:36.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.486+0000 7f991c800640 1 -- 172.21.15.31:0/1721008909 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f98fc0a2f20 msgr2=0x7f98fc0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:36.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.486+0000 7f991c800640 1 --2- 172.21.15.31:0/1721008909 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f98fc0a2f20 0x7f98fc0a53e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9904009b50 tx=0x7f9904005f90 comp rx=0 tx=0).stop 2024-03-21T10:35:36.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.486+0000 7f991c800640 1 -- 172.21.15.31:0/1721008909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9914101730 msgr2=0x7f9914199c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:36.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.486+0000 7f991c800640 1 --2- 172.21.15.31:0/1721008909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9914101730 0x7f9914199c30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f990800e9c0 tx=0x7f990800ee80 comp rx=0 tx=0).stop 2024-03-21T10:35:36.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.487+0000 7f991c800640 1 -- 172.21.15.31:0/1721008909 shutdown_connections 2024-03-21T10:35:36.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.487+0000 7f991c800640 1 --2- 172.21.15.31:0/1721008909 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f98fc0a2f20 0x7f98fc0a53e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:36.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.487+0000 7f991c800640 1 --2- 172.21.15.31:0/1721008909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99141032e0 0x7f99141a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:36.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.487+0000 7f991c800640 1 --2- 172.21.15.31:0/1721008909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9914102920 0x7f991419a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:36.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.487+0000 7f991c800640 1 --2- 172.21.15.31:0/1721008909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9914101730 0x7f9914199c30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:36.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.487+0000 7f991c800640 1 -- 172.21.15.31:0/1721008909 >> 172.21.15.31:0/1721008909 conn(0x7f99140fcc80 msgr2=0x7f99140738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:36.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.487+0000 7f991c800640 1 -- 172.21.15.31:0/1721008909 shutdown_connections 2024-03-21T10:35:36.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.487+0000 7f991c800640 1 -- 172.21.15.31:0/1721008909 wait complete. 2024-03-21T10:35:36.503 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T10:35:36.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.715+0000 7fb503200640 1 -- 172.21.15.31:0/3790960076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb504109450 msgr2=0x7fb50410b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:36.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.715+0000 7fb503200640 1 --2- 172.21.15.31:0/3790960076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb504109450 0x7fb50410b8f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc009ca0 tx=0x7fb4fc02c180 comp rx=0 tx=0).stop 2024-03-21T10:35:36.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.715+0000 7fb503200640 1 -- 172.21.15.31:0/3790960076 shutdown_connections 2024-03-21T10:35:36.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.715+0000 7fb503200640 1 --2- 172.21.15.31:0/3790960076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb504109450 0x7fb50410b8f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:36.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.715+0000 7fb503200640 1 --2- 172.21.15.31:0/3790960076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5040740a0 0x7fb504074500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:36.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.715+0000 7fb503200640 1 --2- 172.21.15.31:0/3790960076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb504075680 0x7fb504073ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:36.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.715+0000 7fb503200640 1 -- 172.21.15.31:0/3790960076 >> 172.21.15.31:0/3790960076 conn(0x7fb5040fcb30 msgr2=0x7fb5040fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:36.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.715+0000 7fb503200640 1 -- 172.21.15.31:0/3790960076 shutdown_connections 2024-03-21T10:35:36.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.715+0000 7fb503200640 1 -- 172.21.15.31:0/3790960076 wait complete. 2024-03-21T10:35:36.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.716+0000 7fb503200640 1 Processor -- start 2024-03-21T10:35:36.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.716+0000 7fb503200640 1 -- start start 2024-03-21T10:35:36.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.716+0000 7fb503200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5040740a0 0x7fb504199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:36.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.716+0000 7fb503200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb504075680 0x7fb504199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:36.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.716+0000 7fb503200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb504109450 0x7fb5041a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:36.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.716+0000 7fb503200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50410f420 con 0x7fb5040740a0 2024-03-21T10:35:36.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.716+0000 7fb503200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50410f2a0 con 0x7fb504075680 2024-03-21T10:35:36.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.716+0000 7fb503200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50410f5a0 con 0x7fb504109450 2024-03-21T10:35:36.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.717+0000 7fb4f9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb504075680 0x7fb504199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:36.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.717+0000 7fb501e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5040740a0 0x7fb504199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:36.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.717+0000 7fb4f9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb504075680 0x7fb504199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49414/0 (socket says 172.21.15.31:49414) 2024-03-21T10:35:36.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.717+0000 7fb4f9400640 1 -- 172.21.15.31:0/736219909 learned_addr learned my addr 172.21.15.31:0/736219909 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:36.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.717+0000 7fb502800640 1 --2- 172.21.15.31:0/736219909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb504109450 0x7fb5041a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:36.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.717+0000 7fb4f9400640 1 -- 172.21.15.31:0/736219909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb504109450 msgr2=0x7fb5041a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:36.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.717+0000 7fb4f9400640 1 --2- 172.21.15.31:0/736219909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb504109450 0x7fb5041a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:36.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.717+0000 7fb4f9400640 1 -- 172.21.15.31:0/736219909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5040740a0 msgr2=0x7fb504199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:36.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.717+0000 7fb4f9400640 1 --2- 172.21.15.31:0/736219909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5040740a0 0x7fb504199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:36.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.717+0000 7fb4f9400640 1 -- 172.21.15.31:0/736219909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5041a1630 con 0x7fb504075680 2024-03-21T10:35:36.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.717+0000 7fb4f9400640 1 --2- 172.21.15.31:0/736219909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb504075680 0x7fb504199f50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec00d950 tx=0x7fb4ec00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:36.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.718+0000 7fb4fb400640 1 -- 172.21.15.31:0/736219909 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec017030 con 0x7fb504075680 2024-03-21T10:35:36.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.718+0000 7fb503200640 1 -- 172.21.15.31:0/736219909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5041a1920 con 0x7fb504075680 2024-03-21T10:35:36.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.718+0000 7fb4fb400640 1 -- 172.21.15.31:0/736219909 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb4ec004020 con 0x7fb504075680 2024-03-21T10:35:36.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.718+0000 7fb4fb400640 1 -- 172.21.15.31:0/736219909 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec0047a0 con 0x7fb504075680 2024-03-21T10:35:36.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.718+0000 7fb503200640 1 -- 172.21.15.31:0/736219909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5041a1e70 con 0x7fb504075680 2024-03-21T10:35:36.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.719+0000 7fb503200640 1 -- 172.21.15.31:0/736219909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb504108210 con 0x7fb504075680 2024-03-21T10:35:36.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.721+0000 7fb4fb400640 1 -- 172.21.15.31:0/736219909 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fb4ec0041d0 con 0x7fb504075680 2024-03-21T10:35:36.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.722+0000 7fb4fb400640 1 --2- 172.21.15.31:0/736219909 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb4e80a2f10 0x7fb4e80a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:36.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.723+0000 7fb4fb400640 1 -- 172.21.15.31:0/736219909 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb4ec0cf6a0 con 0x7fb504075680 2024-03-21T10:35:36.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.723+0000 7fb501e00640 1 --2- 172.21.15.31:0/736219909 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb4e80a2f10 0x7fb4e80a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:36.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.723+0000 7fb501e00640 1 --2- 172.21.15.31:0/736219909 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb4e80a2f10 0x7fb4e80a53d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4005d00 tx=0x7fb4f400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:36.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.725+0000 7fb4fb400640 1 -- 172.21.15.31:0/736219909 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb4ec09c230 con 0x7fb504075680 2024-03-21T10:35:36.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.920+0000 7fb503200640 1 -- 172.21.15.31:0/736219909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb50402d070 con 0x7fb504075680 2024-03-21T10:35:36.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.922+0000 7fb4fb400640 1 -- 172.21.15.31:0/736219909 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24009 (secure 0 0 0) 0x7fb4ec09bb30 con 0x7fb504075680 2024-03-21T10:35:36.921 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:35:36.921 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","created":"2024-03-21T10:25:41.971117+0000","modified":"2024-03-21T10:35:11.657050+0000","last_up_change":"2024-03-21T10:34:07.757196+0000","last_in_change":"2024-03-21T10:33:47.460406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T10:29:44.553968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T10:35:04.945449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T10:35:05.970589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T10:35:09.688711+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e0244d33-00e2-47d5-9f66-34efc138c462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6803","nonce":3089885125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6805","nonce":3089885125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6809","nonce":3089885125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6807","nonce":3089885125}]},"public_addr":"172.21.15.31:6803/3089885125","cluster_addr":"172.21.15.31:6805/3089885125","heartbeat_back_addr":"172.21.15.31:6809/3089885125","heartbeat_front_addr":"172.21.15.31:6807/3089885125","state":["exists","up"]},{"osd":1,"uuid":"76a63504-beab-4354-a6c3-eebc3b16ad03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6801","nonce":618840037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6803","nonce":618840037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6807","nonce":618840037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6805","nonce":618840037}]},"public_addr":"172.21.15.40:6801/618840037","cluster_addr":"172.21.15.40:6803/618840037","heartbeat_back_addr":"172.21.15.40:6807/618840037","heartbeat_front_addr":"172.21.15.40:6805/618840037","state":["exists","up"]},{"osd":2,"uuid":"fa191bd6-9a21-4ee2-8adf-8a63c9cfd821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6801","nonce":2524184117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6803","nonce":2524184117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6807","nonce":2524184117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6805","nonce":2524184117}]},"public_addr":"172.21.15.70:6801/2524184117","cluster_addr":"172.21.15.70:6803/2524184117","heartbeat_back_addr":"172.21.15.70:6807/2524184117","heartbeat_front_addr":"172.21.15.70:6805/2524184117","state":["exists","up"]},{"osd":3,"uuid":"edb44507-ffb0-42a4-827b-08461e1bf952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6811","nonce":2823036296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6813","nonce":2823036296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6817","nonce":2823036296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6815","nonce":2823036296}]},"public_addr":"172.21.15.31:6811/2823036296","cluster_addr":"172.21.15.31:6813/2823036296","heartbeat_back_addr":"172.21.15.31:6817/2823036296","heartbeat_front_addr":"172.21.15.31:6815/2823036296","state":["exists","up"]},{"osd":4,"uuid":"8cb89153-8c3b-431f-b57c-8863fada76dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6809","nonce":4215186309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6811","nonce":4215186309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6815","nonce":4215186309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6813","nonce":4215186309}]},"public_addr":"172.21.15.40:6809/4215186309","cluster_addr":"172.21.15.40:6811/4215186309","heartbeat_back_addr":"172.21.15.40:6815/4215186309","heartbeat_front_addr":"172.21.15.40:6813/4215186309","state":["exists","up"]},{"osd":5,"uuid":"9409505a-d463-408c-a3cd-218380c04d68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6809","nonce":3402830452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6811","nonce":3402830452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6815","nonce":3402830452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6813","nonce":3402830452}]},"public_addr":"172.21.15.70:6809/3402830452","cluster_addr":"172.21.15.70:6811/3402830452","heartbeat_back_addr":"172.21.15.70:6815/3402830452","heartbeat_front_addr":"172.21.15.70:6813/3402830452","state":["exists","up"]},{"osd":6,"uuid":"f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6819","nonce":1193497058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6821","nonce":1193497058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6825","nonce":1193497058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6823","nonce":1193497058}]},"public_addr":"172.21.15.31:6819/1193497058","cluster_addr":"172.21.15.31:6821/1193497058","heartbeat_back_addr":"172.21.15.31:6825/1193497058","heartbeat_front_addr":"172.21.15.31:6823/1193497058","state":["exists","up"]},{"osd":7,"uuid":"db228a87-29a5-45d4-8fe6-a0a554262eb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6817","nonce":3825628119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6819","nonce":3825628119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6823","nonce":3825628119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6821","nonce":3825628119}]},"public_addr":"172.21.15.40:6817/3825628119","cluster_addr":"172.21.15.40:6819/3825628119","heartbeat_back_addr":"172.21.15.40:6823/3825628119","heartbeat_front_addr":"172.21.15.40:6821/3825628119","state":["exists","up"]},{"osd":8,"uuid":"59b9fde0-095d-4e0b-87da-ead37bd8ff1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6817","nonce":534880586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6819","nonce":534880586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6823","nonce":534880586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6821","nonce":534880586}]},"public_addr":"172.21.15.70:6817/534880586","cluster_addr":"172.21.15.70:6819/534880586","heartbeat_back_addr":"172.21.15.70:6823/534880586","heartbeat_front_addr":"172.21.15.70:6821/534880586","state":["exists","up"]},{"osd":9,"uuid":"77f47061-0c5f-4159-b276-e814935d6c1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6827","nonce":587047469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6829","nonce":587047469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6833","nonce":587047469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6831","nonce":587047469}]},"public_addr":"172.21.15.31:6827/587047469","cluster_addr":"172.21.15.31:6829/587047469","heartbeat_back_addr":"172.21.15.31:6833/587047469","heartbeat_front_addr":"172.21.15.31:6831/587047469","state":["exists","up"]},{"osd":10,"uuid":"8853dc43-7bd1-4c93-8063-f7982fd7da38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6825","nonce":512108711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6827","nonce":512108711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6831","nonce":512108711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6829","nonce":512108711}]},"public_addr":"172.21.15.40:6825/512108711","cluster_addr":"172.21.15.40:6827/512108711","heartbeat_back_addr":"172.21.15.40:6831/512108711","heartbeat_front_addr":"172.21.15.40:6829/512108711","state":["exists","up"]},{"osd":11,"uuid":"84b7c2de-c424-45a6-9583-60097e372a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6825","nonce":540305718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6827","nonce":540305718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6831","nonce":540305718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6829","nonce":540305718}]},"public_addr":"172.21.15.70:6825/540305718","cluster_addr":"172.21.15.70:6827/540305718","heartbeat_back_addr":"172.21.15.70:6831/540305718","heartbeat_front_addr":"172.21.15.70:6829/540305718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:28:51.477358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:17.405045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:41.626268+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:15.339568+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:42.676996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:07.516999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:42.412273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:10.365041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:35.699120+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:11.733639+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:39.505442+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:34:05.449301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451930554":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/3342588843":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6801/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/1626448389":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/1910059351":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/344822770":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/3811368512":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/1981437651":"2024-03-22T10:26:16.959165+0000","172.21.15.31:6801/1981437651":"2024-03-22T10:26:16.959165+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T10:35:36.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.924+0000 7fb503200640 1 -- 172.21.15.31:0/736219909 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb4e80a2f10 msgr2=0x7fb4e80a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:36.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.924+0000 7fb503200640 1 --2- 172.21.15.31:0/736219909 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb4e80a2f10 0x7fb4e80a53d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4005d00 tx=0x7fb4f400b410 comp rx=0 tx=0).stop 2024-03-21T10:35:36.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.924+0000 7fb503200640 1 -- 172.21.15.31:0/736219909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb504075680 msgr2=0x7fb504199f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:36.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.924+0000 7fb503200640 1 --2- 172.21.15.31:0/736219909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb504075680 0x7fb504199f50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec00d950 tx=0x7fb4ec00de10 comp rx=0 tx=0).stop 2024-03-21T10:35:36.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.924+0000 7fb503200640 1 -- 172.21.15.31:0/736219909 shutdown_connections 2024-03-21T10:35:36.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.924+0000 7fb503200640 1 --2- 172.21.15.31:0/736219909 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb4e80a2f10 0x7fb4e80a53d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:36.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.924+0000 7fb503200640 1 --2- 172.21.15.31:0/736219909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb504109450 0x7fb5041a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:36.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.924+0000 7fb503200640 1 --2- 172.21.15.31:0/736219909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb504075680 0x7fb504199f50 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:36.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.924+0000 7fb503200640 1 --2- 172.21.15.31:0/736219909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5040740a0 0x7fb504199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:36.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.924+0000 7fb503200640 1 -- 172.21.15.31:0/736219909 >> 172.21.15.31:0/736219909 conn(0x7fb5040fcb30 msgr2=0x7fb5041053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:36.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.925+0000 7fb503200640 1 -- 172.21.15.31:0/736219909 shutdown_connections 2024-03-21T10:35:36.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:36.925+0000 7fb503200640 1 -- 172.21.15.31:0/736219909 wait complete. 2024-03-21T10:35:36.940 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-21T10:35:37.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.156+0000 7f45f3e00640 1 -- 172.21.15.31:0/3766305728 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45f41048f0 msgr2=0x7f45f4104d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:37.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.156+0000 7f45f3e00640 1 --2- 172.21.15.31:0/3766305728 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45f41048f0 0x7f45f4104d70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f45e0009900 tx=0x7f45e002bc30 comp rx=0 tx=0).stop 2024-03-21T10:35:37.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.156+0000 7f45f3e00640 1 -- 172.21.15.31:0/3766305728 shutdown_connections 2024-03-21T10:35:37.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.156+0000 7f45f3e00640 1 --2- 172.21.15.31:0/3766305728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45f41052b0 0x7f45f410bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:37.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.156+0000 7f45f3e00640 1 --2- 172.21.15.31:0/3766305728 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45f41048f0 0x7f45f4104d70 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:37.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.156+0000 7f45f3e00640 1 --2- 172.21.15.31:0/3766305728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45f4103700 0x7f45f4103b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:37.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.156+0000 7f45f3e00640 1 -- 172.21.15.31:0/3766305728 >> 172.21.15.31:0/3766305728 conn(0x7f45f40fec70 msgr2=0x7f45f41010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:37.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.156+0000 7f45f3e00640 1 -- 172.21.15.31:0/3766305728 shutdown_connections 2024-03-21T10:35:37.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.156+0000 7f45f3e00640 1 -- 172.21.15.31:0/3766305728 wait complete. 2024-03-21T10:35:37.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.157+0000 7f45f3e00640 1 Processor -- start 2024-03-21T10:35:37.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.158+0000 7f45f3e00640 1 -- start start 2024-03-21T10:35:37.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.158+0000 7f45f3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45f4103700 0x7f45f4199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:37.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.158+0000 7f45f3e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45f41048f0 0x7f45f4199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:37.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.158+0000 7f45f3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45f41052b0 0x7f45f41a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:37.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.158+0000 7f45f3e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f410f3b0 con 0x7f45f4103700 2024-03-21T10:35:37.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.158+0000 7f45f3e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f410f230 con 0x7f45f41052b0 2024-03-21T10:35:37.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.158+0000 7f45f3e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f410f530 con 0x7f45f41048f0 2024-03-21T10:35:37.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.158+0000 7f45f2000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45f41048f0 0x7f45f4199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:37.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.158+0000 7f45f2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45f4103700 0x7f45f4199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:37.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.158+0000 7f45f2000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45f41048f0 0x7f45f4199f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53560/0 (socket says 172.21.15.31:53560) 2024-03-21T10:35:37.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.158+0000 7f45f2000640 1 -- 172.21.15.31:0/3246919402 learned_addr learned my addr 172.21.15.31:0/3246919402 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:37.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.158+0000 7f45f3400640 1 --2- 172.21.15.31:0/3246919402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45f41052b0 0x7f45f41a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:37.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.158+0000 7f45f2000640 1 -- 172.21.15.31:0/3246919402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45f41052b0 msgr2=0x7f45f41a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:37.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.158+0000 7f45f2000640 1 --2- 172.21.15.31:0/3246919402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45f41052b0 0x7f45f41a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:37.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.158+0000 7f45f2000640 1 -- 172.21.15.31:0/3246919402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45f4103700 msgr2=0x7f45f4199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:37.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.158+0000 7f45f2000640 1 --2- 172.21.15.31:0/3246919402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45f4103700 0x7f45f4199a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:37.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.159+0000 7f45f2000640 1 -- 172.21.15.31:0/3246919402 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f41a1670 con 0x7f45f41048f0 2024-03-21T10:35:37.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.159+0000 7f45f2000640 1 --2- 172.21.15.31:0/3246919402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45f41048f0 0x7f45f4199f90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f45e0005ab0 tx=0x7f45e002c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:37.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.159+0000 7f45f8a00640 1 -- 172.21.15.31:0/3246919402 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e0043020 con 0x7f45f41048f0 2024-03-21T10:35:37.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.159+0000 7f45f8a00640 1 -- 172.21.15.31:0/3246919402 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45e00054e0 con 0x7f45f41048f0 2024-03-21T10:35:37.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.159+0000 7f45f3e00640 1 -- 172.21.15.31:0/3246919402 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45f41a1900 con 0x7f45f41048f0 2024-03-21T10:35:37.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.159+0000 7f45f8a00640 1 -- 172.21.15.31:0/3246919402 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e0003ce0 con 0x7f45f41048f0 2024-03-21T10:35:37.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.159+0000 7f45f3e00640 1 -- 172.21.15.31:0/3246919402 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45f41a1e50 con 0x7f45f41048f0 2024-03-21T10:35:37.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.160+0000 7f45f3e00640 1 -- 172.21.15.31:0/3246919402 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45f410aaf0 con 0x7f45f41048f0 2024-03-21T10:35:37.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.163+0000 7f45f8a00640 1 -- 172.21.15.31:0/3246919402 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f45e0046020 con 0x7f45f41048f0 2024-03-21T10:35:37.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.164+0000 7f45f8a00640 1 --2- 172.21.15.31:0/3246919402 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f45d80a2f20 0x7f45d80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:37.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.165+0000 7f45f8a00640 1 -- 172.21.15.31:0/3246919402 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f45e00f1b80 con 0x7f45f41048f0 2024-03-21T10:35:37.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.165+0000 7f45f2a00640 1 --2- 172.21.15.31:0/3246919402 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f45d80a2f20 0x7f45d80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:37.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.165+0000 7f45f2a00640 1 --2- 172.21.15.31:0/3246919402 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f45d80a2f20 0x7f45d80a53e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f45f4104750 tx=0x7f45e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:37.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.166+0000 7f45f8a00640 1 -- 172.21.15.31:0/3246919402 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f45e004d020 con 0x7f45f41048f0 2024-03-21T10:35:37.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:37.416+0000 7f45f3e00640 1 -- 172.21.15.31:0/3246919402 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f45f402d280 con 0x7f45f41048f0 2024-03-21T10:35:37.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:37 smithi031 ceph-mon[31673]: pgmap v340: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-21T10:35:37.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:37 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1721008909' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:37.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:37 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/736219909' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:37.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:37 smithi070 ceph-mon[36330]: pgmap v340: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-21T10:35:37.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:37 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1721008909' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:37.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:37 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/736219909' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:37.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:37 smithi040 ceph-mon[36154]: pgmap v340: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-21T10:35:37.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:37 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1721008909' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:37.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:37 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/736219909' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:38.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.288+0000 7f45f8a00640 1 -- 172.21.15.31:0/3246919402 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7f45e00be710 con 0x7f45f41048f0 2024-03-21T10:35:38.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.292+0000 7f45f3e00640 1 -- 172.21.15.31:0/3246919402 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f45d80a2f20 msgr2=0x7f45d80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:38.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.292+0000 7f45f3e00640 1 --2- 172.21.15.31:0/3246919402 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f45d80a2f20 0x7f45d80a53e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f45f4104750 tx=0x7f45e4005f90 comp rx=0 tx=0).stop 2024-03-21T10:35:38.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.292+0000 7f45f3e00640 1 -- 172.21.15.31:0/3246919402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45f41048f0 msgr2=0x7f45f4199f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:38.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.292+0000 7f45f3e00640 1 --2- 172.21.15.31:0/3246919402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45f41048f0 0x7f45f4199f90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f45e0005ab0 tx=0x7f45e002c6c0 comp rx=0 tx=0).stop 2024-03-21T10:35:38.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.293+0000 7f45f3e00640 1 -- 172.21.15.31:0/3246919402 shutdown_connections 2024-03-21T10:35:38.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.293+0000 7f45f3e00640 1 --2- 172.21.15.31:0/3246919402 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f45d80a2f20 0x7f45d80a53e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:38.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.293+0000 7f45f3e00640 1 --2- 172.21.15.31:0/3246919402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45f41052b0 0x7f45f41a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:38.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.293+0000 7f45f3e00640 1 --2- 172.21.15.31:0/3246919402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45f41048f0 0x7f45f4199f90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:38.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.293+0000 7f45f3e00640 1 --2- 172.21.15.31:0/3246919402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45f4103700 0x7f45f4199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:38.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.293+0000 7f45f3e00640 1 -- 172.21.15.31:0/3246919402 >> 172.21.15.31:0/3246919402 conn(0x7f45f40fec70 msgr2=0x7f45f4107cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:38.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.293+0000 7f45f3e00640 1 -- 172.21.15.31:0/3246919402 shutdown_connections 2024-03-21T10:35:38.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.293+0000 7f45f3e00640 1 -- 172.21.15.31:0/3246919402 wait complete. 2024-03-21T10:35:38.314 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-21T10:35:38.314 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-21T10:35:38.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.520+0000 7f3a6e400640 1 -- 172.21.15.31:0/856759656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a681026e0 msgr2=0x7f3a68102b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:38.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.520+0000 7f3a6e400640 1 --2- 172.21.15.31:0/856759656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a681026e0 0x7f3a68102b60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3a54009900 tx=0x7f3a5402bc30 comp rx=0 tx=0).stop 2024-03-21T10:35:38.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.520+0000 7f3a6e400640 1 -- 172.21.15.31:0/856759656 shutdown_connections 2024-03-21T10:35:38.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.521+0000 7f3a6e400640 1 --2- 172.21.15.31:0/856759656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a681030a0 0x7f3a68109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:38.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.521+0000 7f3a6e400640 1 --2- 172.21.15.31:0/856759656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a681026e0 0x7f3a68102b60 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:38.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.521+0000 7f3a6e400640 1 --2- 172.21.15.31:0/856759656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a681014f0 0x7f3a68101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:38.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.521+0000 7f3a6e400640 1 -- 172.21.15.31:0/856759656 >> 172.21.15.31:0/856759656 conn(0x7f3a680fca60 msgr2=0x7f3a680feea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:38.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.521+0000 7f3a6e400640 1 -- 172.21.15.31:0/856759656 shutdown_connections 2024-03-21T10:35:38.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.521+0000 7f3a6e400640 1 -- 172.21.15.31:0/856759656 wait complete. 2024-03-21T10:35:38.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.522+0000 7f3a6e400640 1 Processor -- start 2024-03-21T10:35:38.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.522+0000 7f3a6e400640 1 -- start start 2024-03-21T10:35:38.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.523+0000 7f3a6e400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a681014f0 0x7f3a68107220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:38.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.523+0000 7f3a6e400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a681026e0 0x7f3a68107760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:38.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.523+0000 7f3a6e400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a681030a0 0x7f3a68103d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:38.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.523+0000 7f3a6e400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6810d1d0 con 0x7f3a681026e0 2024-03-21T10:35:38.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.523+0000 7f3a6e400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6810d050 con 0x7f3a681030a0 2024-03-21T10:35:38.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.523+0000 7f3a6e400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6810d350 con 0x7f3a681014f0 2024-03-21T10:35:38.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.523+0000 7f3a67e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a681026e0 0x7f3a68107760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:38.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.523+0000 7f3a6d000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a681014f0 0x7f3a68107220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:38.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.523+0000 7f3a6d000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a681014f0 0x7f3a68107220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53566/0 (socket says 172.21.15.31:53566) 2024-03-21T10:35:38.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.523+0000 7f3a6d000640 1 -- 172.21.15.31:0/2871424706 learned_addr learned my addr 172.21.15.31:0/2871424706 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:38.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.523+0000 7f3a6da00640 1 --2- 172.21.15.31:0/2871424706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a681030a0 0x7f3a68103d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:38.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.523+0000 7f3a6d000640 1 -- 172.21.15.31:0/2871424706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a681030a0 msgr2=0x7f3a68103d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:38.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.523+0000 7f3a6d000640 1 --2- 172.21.15.31:0/2871424706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a681030a0 0x7f3a68103d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:38.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.523+0000 7f3a6d000640 1 -- 172.21.15.31:0/2871424706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a681026e0 msgr2=0x7f3a68107760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:38.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.523+0000 7f3a6d000640 1 --2- 172.21.15.31:0/2871424706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a681026e0 0x7f3a68107760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:38.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.523+0000 7f3a6d000640 1 -- 172.21.15.31:0/2871424706 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a68104650 con 0x7f3a681014f0 2024-03-21T10:35:38.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.524+0000 7f3a6d000640 1 --2- 172.21.15.31:0/2871424706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a681014f0 0x7f3a68107220 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c00eb70 tx=0x7f3a5c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:38.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.524+0000 7f3a65600640 1 -- 172.21.15.31:0/2871424706 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c003b70 con 0x7f3a681014f0 2024-03-21T10:35:38.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.524+0000 7f3a6e400640 1 -- 172.21.15.31:0/2871424706 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a681a5e20 con 0x7f3a681014f0 2024-03-21T10:35:38.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.524+0000 7f3a65600640 1 -- 172.21.15.31:0/2871424706 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a5c00ee70 con 0x7f3a681014f0 2024-03-21T10:35:38.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.524+0000 7f3a65600640 1 -- 172.21.15.31:0/2871424706 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c010830 con 0x7f3a681014f0 2024-03-21T10:35:38.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.524+0000 7f3a6e400640 1 -- 172.21.15.31:0/2871424706 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a681a61d0 con 0x7f3a681014f0 2024-03-21T10:35:38.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.526+0000 7f3a6e400640 1 -- 172.21.15.31:0/2871424706 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a681a63a0 con 0x7f3a681014f0 2024-03-21T10:35:38.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.527+0000 7f3a65600640 1 -- 172.21.15.31:0/2871424706 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f3a5c004020 con 0x7f3a681014f0 2024-03-21T10:35:38.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.529+0000 7f3a65600640 1 --2- 172.21.15.31:0/2871424706 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f3a500a2f20 0x7f3a500a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:38.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.529+0000 7f3a65600640 1 -- 172.21.15.31:0/2871424706 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f3a5c0cfcf0 con 0x7f3a681014f0 2024-03-21T10:35:38.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.529+0000 7f3a67e00640 1 --2- 172.21.15.31:0/2871424706 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f3a500a2f20 0x7f3a500a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:38.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.530+0000 7f3a67e00640 1 --2- 172.21.15.31:0/2871424706 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f3a500a2f20 0x7f3a500a53e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3a5402c240 tx=0x7f3a5402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:38.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.533+0000 7f3a65600640 1 -- 172.21.15.31:0/2871424706 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3a5c09c880 con 0x7f3a681014f0 2024-03-21T10:35:38.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:38 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3246919402' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-21T10:35:38.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:38 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-21T10:35:38.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:38 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3246919402' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-21T10:35:38.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:38 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-21T10:35:38.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:38 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3246919402' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-21T10:35:38.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:38 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-21T10:35:38.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:38.777+0000 7f3a6e400640 1 -- 172.21.15.31:0/2871424706 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f3a681a6610 con 0x7f3a681014f0 2024-03-21T10:35:39.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.301+0000 7f3a65600640 1 -- 172.21.15.31:0/2871424706 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) v1 ==== 105+0+0 (secure 0 0 0) 0x7f3a5c09c180 con 0x7f3a681014f0 2024-03-21T10:35:39.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.305+0000 7f3a6e400640 1 -- 172.21.15.31:0/2871424706 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f3a500a2f20 msgr2=0x7f3a500a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:39.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.305+0000 7f3a6e400640 1 --2- 172.21.15.31:0/2871424706 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f3a500a2f20 0x7f3a500a53e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3a5402c240 tx=0x7f3a5402df90 comp rx=0 tx=0).stop 2024-03-21T10:35:39.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.305+0000 7f3a6e400640 1 -- 172.21.15.31:0/2871424706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a681014f0 msgr2=0x7f3a68107220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:39.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.305+0000 7f3a6e400640 1 --2- 172.21.15.31:0/2871424706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a681014f0 0x7f3a68107220 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c00eb70 tx=0x7f3a5c00c6a0 comp rx=0 tx=0).stop 2024-03-21T10:35:39.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.305+0000 7f3a6e400640 1 -- 172.21.15.31:0/2871424706 shutdown_connections 2024-03-21T10:35:39.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.305+0000 7f3a6e400640 1 --2- 172.21.15.31:0/2871424706 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f3a500a2f20 0x7f3a500a53e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:39.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.305+0000 7f3a6e400640 1 --2- 172.21.15.31:0/2871424706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a681030a0 0x7f3a68103d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:39.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.305+0000 7f3a6e400640 1 --2- 172.21.15.31:0/2871424706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a681026e0 0x7f3a68107760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:39.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.305+0000 7f3a6e400640 1 --2- 172.21.15.31:0/2871424706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a681014f0 0x7f3a68107220 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:39.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.305+0000 7f3a6e400640 1 -- 172.21.15.31:0/2871424706 >> 172.21.15.31:0/2871424706 conn(0x7f3a680fca60 msgr2=0x7f3a680738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:39.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.306+0000 7f3a6e400640 1 -- 172.21.15.31:0/2871424706 shutdown_connections 2024-03-21T10:35:39.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.306+0000 7f3a6e400640 1 -- 172.21.15.31:0/2871424706 wait complete. 2024-03-21T10:35:39.330 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-21T10:35:39.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.531+0000 7fa907e00640 1 -- 172.21.15.31:0/2468456908 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9081096b0 msgr2=0x7fa90810ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:39.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.531+0000 7fa907e00640 1 --2- 172.21.15.31:0/2468456908 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9081096b0 0x7fa90810ff40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa900009ca0 tx=0x7fa90002c100 comp rx=0 tx=0).stop 2024-03-21T10:35:39.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.531+0000 7fa907e00640 1 -- 172.21.15.31:0/2468456908 shutdown_connections 2024-03-21T10:35:39.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.531+0000 7fa907e00640 1 --2- 172.21.15.31:0/2468456908 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9081096b0 0x7fa90810ff40 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:39.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.531+0000 7fa907e00640 1 --2- 172.21.15.31:0/2468456908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa908108cf0 0x7fa908109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:39.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.531+0000 7fa907e00640 1 --2- 172.21.15.31:0/2468456908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa908107b00 0x7fa908107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:39.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.531+0000 7fa907e00640 1 -- 172.21.15.31:0/2468456908 >> 172.21.15.31:0/2468456908 conn(0x7fa908076a90 msgr2=0x7fa908078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:39.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.531+0000 7fa907e00640 1 -- 172.21.15.31:0/2468456908 shutdown_connections 2024-03-21T10:35:39.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.531+0000 7fa907e00640 1 -- 172.21.15.31:0/2468456908 wait complete. 2024-03-21T10:35:39.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.532+0000 7fa907e00640 1 Processor -- start 2024-03-21T10:35:39.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.532+0000 7fa907e00640 1 -- start start 2024-03-21T10:35:39.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.533+0000 7fa907e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa908107b00 0x7fa90819ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:39.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.533+0000 7fa907e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa908108cf0 0x7fa90819e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:39.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.533+0000 7fa907e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9081096b0 0x7fa9081a5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:39.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.533+0000 7fa907e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa908113780 con 0x7fa908108cf0 2024-03-21T10:35:39.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.533+0000 7fa907e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa908113600 con 0x7fa908107b00 2024-03-21T10:35:39.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.533+0000 7fa907e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa908113900 con 0x7fa9081096b0 2024-03-21T10:35:39.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.533+0000 7fa906a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa908107b00 0x7fa90819ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:39.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.533+0000 7fa906a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa908107b00 0x7fa90819ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:43922/0 (socket says 172.21.15.31:43922) 2024-03-21T10:35:39.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.533+0000 7fa906a00640 1 -- 172.21.15.31:0/4256821448 learned_addr learned my addr 172.21.15.31:0/4256821448 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:39.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.533+0000 7fa906000640 1 --2- 172.21.15.31:0/4256821448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa908108cf0 0x7fa90819e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:39.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.533+0000 7fa907400640 1 --2- 172.21.15.31:0/4256821448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9081096b0 0x7fa9081a5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:39.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.534+0000 7fa906a00640 1 -- 172.21.15.31:0/4256821448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9081096b0 msgr2=0x7fa9081a5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:39.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.534+0000 7fa906a00640 1 --2- 172.21.15.31:0/4256821448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9081096b0 0x7fa9081a5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:39.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.534+0000 7fa906a00640 1 -- 172.21.15.31:0/4256821448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa908108cf0 msgr2=0x7fa90819e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:39.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.534+0000 7fa906a00640 1 --2- 172.21.15.31:0/4256821448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa908108cf0 0x7fa90819e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:39.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.534+0000 7fa906a00640 1 -- 172.21.15.31:0/4256821448 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9081a59c0 con 0x7fa908107b00 2024-03-21T10:35:39.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.534+0000 7fa906000640 1 --2- 172.21.15.31:0/4256821448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa908108cf0 0x7fa90819e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:35:39.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.534+0000 7fa906a00640 1 --2- 172.21.15.31:0/4256821448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa908107b00 0x7fa90819ddb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa8f800e9c0 tx=0x7fa8f800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:39.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.534+0000 7fa8ff400640 1 -- 172.21.15.31:0/4256821448 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f800cd80 con 0x7fa908107b00 2024-03-21T10:35:39.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.534+0000 7fa907e00640 1 -- 172.21.15.31:0/4256821448 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9081a5cb0 con 0x7fa908107b00 2024-03-21T10:35:39.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.534+0000 7fa8ff400640 1 -- 172.21.15.31:0/4256821448 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8f80035f0 con 0x7fa908107b00 2024-03-21T10:35:39.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.534+0000 7fa8ff400640 1 -- 172.21.15.31:0/4256821448 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8010690 con 0x7fa908107b00 2024-03-21T10:35:39.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.534+0000 7fa907e00640 1 -- 172.21.15.31:0/4256821448 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9081a61a0 con 0x7fa908107b00 2024-03-21T10:35:39.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.535+0000 7fa907e00640 1 -- 172.21.15.31:0/4256821448 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa90810eea0 con 0x7fa908107b00 2024-03-21T10:35:39.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.537+0000 7fa8ff400640 1 -- 172.21.15.31:0/4256821448 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fa8f80037a0 con 0x7fa908107b00 2024-03-21T10:35:39.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.539+0000 7fa8ff400640 1 --2- 172.21.15.31:0/4256821448 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa8ec0a2fa0 0x7fa8ec0a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:39.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.539+0000 7fa8ff400640 1 -- 172.21.15.31:0/4256821448 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fa8f80cfb30 con 0x7fa908107b00 2024-03-21T10:35:39.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.539+0000 7fa906000640 1 --2- 172.21.15.31:0/4256821448 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa8ec0a2fa0 0x7fa8ec0a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:39.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.540+0000 7fa906000640 1 --2- 172.21.15.31:0/4256821448 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa8ec0a2fa0 0x7fa8ec0a5460 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4009c20 tx=0x7fa8f4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:39.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.548+0000 7fa8ff400640 1 -- 172.21.15.31:0/4256821448 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa8f809c6c0 con 0x7fa908107b00 2024-03-21T10:35:39.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:39 smithi031 ceph-mon[31673]: pgmap v341: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-21T10:35:39.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:39 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-21T10:35:39.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:39 smithi031 ceph-mon[31673]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-21T10:35:39.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:39 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2871424706' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-21T10:35:39.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:39 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-21T10:35:39.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:39 smithi070 ceph-mon[36330]: pgmap v341: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-21T10:35:39.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:39 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-21T10:35:39.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:39 smithi070 ceph-mon[36330]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-21T10:35:39.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:39 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2871424706' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-21T10:35:39.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:39 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-21T10:35:39.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:39 smithi040 ceph-mon[36154]: pgmap v341: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-21T10:35:39.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:39 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-21T10:35:39.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:39 smithi040 ceph-mon[36154]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-21T10:35:39.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:39 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2871424706' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-21T10:35:39.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:39 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-21T10:35:39.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:39.792+0000 7fa907e00640 1 -- 172.21.15.31:0/4256821448 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fa90802d070 con 0x7fa908107b00 2024-03-21T10:35:40.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.312+0000 7fa8ff400640 1 -- 172.21.15.31:0/4256821448 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) v1 ==== 121+0+0 (secure 0 0 0) 0x7fa8f809bfc0 con 0x7fa908107b00 2024-03-21T10:35:40.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.315+0000 7fa907e00640 1 -- 172.21.15.31:0/4256821448 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa8ec0a2fa0 msgr2=0x7fa8ec0a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:40.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.315+0000 7fa907e00640 1 --2- 172.21.15.31:0/4256821448 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa8ec0a2fa0 0x7fa8ec0a5460 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4009c20 tx=0x7fa8f4005f90 comp rx=0 tx=0).stop 2024-03-21T10:35:40.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.315+0000 7fa907e00640 1 -- 172.21.15.31:0/4256821448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa908107b00 msgr2=0x7fa90819ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:40.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.315+0000 7fa907e00640 1 --2- 172.21.15.31:0/4256821448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa908107b00 0x7fa90819ddb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa8f800e9c0 tx=0x7fa8f800ee80 comp rx=0 tx=0).stop 2024-03-21T10:35:40.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.316+0000 7fa907e00640 1 -- 172.21.15.31:0/4256821448 shutdown_connections 2024-03-21T10:35:40.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.316+0000 7fa907e00640 1 --2- 172.21.15.31:0/4256821448 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa8ec0a2fa0 0x7fa8ec0a5460 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:40.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.316+0000 7fa907e00640 1 --2- 172.21.15.31:0/4256821448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9081096b0 0x7fa9081a5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:40.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.316+0000 7fa907e00640 1 --2- 172.21.15.31:0/4256821448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa908108cf0 0x7fa90819e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:40.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.316+0000 7fa907e00640 1 --2- 172.21.15.31:0/4256821448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa908107b00 0x7fa90819ddb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:40.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.316+0000 7fa907e00640 1 -- 172.21.15.31:0/4256821448 >> 172.21.15.31:0/4256821448 conn(0x7fa908076a90 msgr2=0x7fa90810c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:40.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.316+0000 7fa907e00640 1 -- 172.21.15.31:0/4256821448 shutdown_connections 2024-03-21T10:35:40.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.316+0000 7fa907e00640 1 -- 172.21.15.31:0/4256821448 wait complete. 2024-03-21T10:35:40.343 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-21T10:35:40.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.536+0000 7f1b76400640 1 -- 172.21.15.31:0/2022182344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b701048f0 msgr2=0x7f1b70104d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:40.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.536+0000 7f1b76400640 1 --2- 172.21.15.31:0/2022182344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b701048f0 0x7f1b70104d70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1b58009900 tx=0x7f1b5802bc40 comp rx=0 tx=0).stop 2024-03-21T10:35:40.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.537+0000 7f1b76400640 1 -- 172.21.15.31:0/2022182344 shutdown_connections 2024-03-21T10:35:40.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.537+0000 7f1b76400640 1 --2- 172.21.15.31:0/2022182344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b701052b0 0x7f1b7010bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:40.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.537+0000 7f1b76400640 1 --2- 172.21.15.31:0/2022182344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b701048f0 0x7f1b70104d70 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:40.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.537+0000 7f1b76400640 1 --2- 172.21.15.31:0/2022182344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b70103700 0x7f1b70103b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:40.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.537+0000 7f1b76400640 1 -- 172.21.15.31:0/2022182344 >> 172.21.15.31:0/2022182344 conn(0x7f1b700fec70 msgr2=0x7f1b701010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:40.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.537+0000 7f1b76400640 1 -- 172.21.15.31:0/2022182344 shutdown_connections 2024-03-21T10:35:40.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.537+0000 7f1b76400640 1 -- 172.21.15.31:0/2022182344 wait complete. 2024-03-21T10:35:40.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.538+0000 7f1b76400640 1 Processor -- start 2024-03-21T10:35:40.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.538+0000 7f1b76400640 1 -- start start 2024-03-21T10:35:40.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.539+0000 7f1b76400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b70103700 0x7f1b70199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:40.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.539+0000 7f1b76400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b701048f0 0x7f1b70199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:40.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.539+0000 7f1b76400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b701052b0 0x7f1b701a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:40.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.539+0000 7f1b76400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7010f3b0 con 0x7f1b701052b0 2024-03-21T10:35:40.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.539+0000 7f1b76400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7010f230 con 0x7f1b70103700 2024-03-21T10:35:40.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.539+0000 7f1b76400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7010f530 con 0x7f1b701048f0 2024-03-21T10:35:40.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.539+0000 7f1b75000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b70103700 0x7f1b70199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:40.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.539+0000 7f1b75000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b70103700 0x7f1b70199a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:43948/0 (socket says 172.21.15.31:43948) 2024-03-21T10:35:40.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.539+0000 7f1b75000640 1 -- 172.21.15.31:0/517860720 learned_addr learned my addr 172.21.15.31:0/517860720 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:40.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.539+0000 7f1b67e00640 1 --2- 172.21.15.31:0/517860720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b701048f0 0x7f1b70199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:40.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.539+0000 7f1b75a00640 1 --2- 172.21.15.31:0/517860720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b701052b0 0x7f1b701a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:40.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.539+0000 7f1b75000640 1 -- 172.21.15.31:0/517860720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b701048f0 msgr2=0x7f1b70199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:40.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.539+0000 7f1b75000640 1 --2- 172.21.15.31:0/517860720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b701048f0 0x7f1b70199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:40.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.539+0000 7f1b75000640 1 -- 172.21.15.31:0/517860720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b701052b0 msgr2=0x7f1b701a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:40.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.539+0000 7f1b75000640 1 --2- 172.21.15.31:0/517860720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b701052b0 0x7f1b701a0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:40.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.540+0000 7f1b75000640 1 -- 172.21.15.31:0/517860720 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b701a1640 con 0x7f1b70103700 2024-03-21T10:35:40.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.540+0000 7f1b75000640 1 --2- 172.21.15.31:0/517860720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b70103700 0x7f1b70199a20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1b6000e9c0 tx=0x7f1b6000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:40.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.540+0000 7f1b6e000640 1 -- 172.21.15.31:0/517860720 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6000cd80 con 0x7f1b70103700 2024-03-21T10:35:40.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.540+0000 7f1b6e000640 1 -- 172.21.15.31:0/517860720 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1b600035f0 con 0x7f1b70103700 2024-03-21T10:35:40.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.540+0000 7f1b76400640 1 -- 172.21.15.31:0/517860720 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b701a1930 con 0x7f1b70103700 2024-03-21T10:35:40.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.540+0000 7f1b6e000640 1 -- 172.21.15.31:0/517860720 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b60010690 con 0x7f1b70103700 2024-03-21T10:35:40.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.540+0000 7f1b76400640 1 -- 172.21.15.31:0/517860720 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b701a1e50 con 0x7f1b70103700 2024-03-21T10:35:40.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.541+0000 7f1b76400640 1 -- 172.21.15.31:0/517860720 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b7004f400 con 0x7f1b70103700 2024-03-21T10:35:40.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.543+0000 7f1b6e000640 1 -- 172.21.15.31:0/517860720 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f1b600037a0 con 0x7f1b70103700 2024-03-21T10:35:40.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.544+0000 7f1b6e000640 1 --2- 172.21.15.31:0/517860720 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1b540a2f20 0x7f1b540a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:40.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.545+0000 7f1b6e000640 1 -- 172.21.15.31:0/517860720 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f1b600cfb10 con 0x7f1b70103700 2024-03-21T10:35:40.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.545+0000 7f1b67e00640 1 --2- 172.21.15.31:0/517860720 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1b540a2f20 0x7f1b540a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:40.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.546+0000 7f1b67e00640 1 --2- 172.21.15.31:0/517860720 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1b540a2f20 0x7f1b540a53e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1b5802c250 tx=0x7f1b5802cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:40.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.549+0000 7f1b6e000640 1 -- 172.21.15.31:0/517860720 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1b6009c6a0 con 0x7f1b70103700 2024-03-21T10:35:40.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:40 smithi031 ceph-mon[31673]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-21T10:35:40.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:40 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-21T10:35:40.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:40 smithi031 ceph-mon[31673]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-21T10:35:40.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:40 smithi031 ceph-mon[31673]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-21T10:35:40.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:40 smithi031 ceph-mon[31673]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-03-21T10:35:40.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:40 smithi031 ceph-mon[31673]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-21T10:35:40.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:40 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/4256821448' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-21T10:35:40.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:40 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-21T10:35:40.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:40 smithi070 ceph-mon[36330]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-21T10:35:40.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:40 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-21T10:35:40.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:40 smithi070 ceph-mon[36330]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-21T10:35:40.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:40 smithi070 ceph-mon[36330]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-21T10:35:40.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:40 smithi070 ceph-mon[36330]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-03-21T10:35:40.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:40 smithi070 ceph-mon[36330]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-21T10:35:40.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:40 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/4256821448' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-21T10:35:40.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:40 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-21T10:35:40.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:40 smithi040 ceph-mon[36154]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-21T10:35:40.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:40 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-21T10:35:40.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:40 smithi040 ceph-mon[36154]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-21T10:35:40.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:40 smithi040 ceph-mon[36154]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-21T10:35:40.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:40 smithi040 ceph-mon[36154]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-03-21T10:35:40.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:40 smithi040 ceph-mon[36154]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-21T10:35:40.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:40 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/4256821448' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-21T10:35:40.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:40 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-21T10:35:40.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.947+0000 7f1b76400640 1 -- 172.21.15.31:0/517860720 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f1b7010aa20 con 0x7f1b70103700 2024-03-21T10:35:40.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.994+0000 7f1b6e000640 1 -- 172.21.15.31:0/517860720 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) v1 ==== 115+0+0 (secure 0 0 0) 0x7f1b6009bfa0 con 0x7f1b70103700 2024-03-21T10:35:40.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.997+0000 7f1b76400640 1 -- 172.21.15.31:0/517860720 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1b540a2f20 msgr2=0x7f1b540a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:40.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.997+0000 7f1b76400640 1 --2- 172.21.15.31:0/517860720 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1b540a2f20 0x7f1b540a53e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1b5802c250 tx=0x7f1b5802cf90 comp rx=0 tx=0).stop 2024-03-21T10:35:40.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.997+0000 7f1b76400640 1 -- 172.21.15.31:0/517860720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b70103700 msgr2=0x7f1b70199a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:40.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.997+0000 7f1b76400640 1 --2- 172.21.15.31:0/517860720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b70103700 0x7f1b70199a20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1b6000e9c0 tx=0x7f1b6000ee80 comp rx=0 tx=0).stop 2024-03-21T10:35:40.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.997+0000 7f1b76400640 1 -- 172.21.15.31:0/517860720 shutdown_connections 2024-03-21T10:35:40.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.997+0000 7f1b76400640 1 --2- 172.21.15.31:0/517860720 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1b540a2f20 0x7f1b540a53e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:40.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.997+0000 7f1b76400640 1 --2- 172.21.15.31:0/517860720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b701052b0 0x7f1b701a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:40.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.997+0000 7f1b76400640 1 --2- 172.21.15.31:0/517860720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b701048f0 0x7f1b70199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:40.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.997+0000 7f1b76400640 1 --2- 172.21.15.31:0/517860720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b70103700 0x7f1b70199a20 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:40.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.997+0000 7f1b76400640 1 -- 172.21.15.31:0/517860720 >> 172.21.15.31:0/517860720 conn(0x7f1b700fec70 msgr2=0x7f1b70107bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:40.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.998+0000 7f1b76400640 1 -- 172.21.15.31:0/517860720 shutdown_connections 2024-03-21T10:35:40.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:40.998+0000 7f1b76400640 1 -- 172.21.15.31:0/517860720 wait complete. 2024-03-21T10:35:41.016 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-03-21T10:35:41.016 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-21T10:35:41.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.231+0000 7fccd3a00640 1 -- 172.21.15.31:0/3949632850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcccc102890 msgr2=0x7fcccc102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:41.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.231+0000 7fccd3a00640 1 --2- 172.21.15.31:0/3949632850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcccc102890 0x7fcccc102d10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fccb8009900 tx=0x7fccb802bc30 comp rx=0 tx=0).stop 2024-03-21T10:35:41.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.231+0000 7fccd3a00640 1 -- 172.21.15.31:0/3949632850 shutdown_connections 2024-03-21T10:35:41.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.232+0000 7fccd3a00640 1 --2- 172.21.15.31:0/3949632850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcccc103250 0x7fcccc109ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:41.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.232+0000 7fccd3a00640 1 --2- 172.21.15.31:0/3949632850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcccc102890 0x7fcccc102d10 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:41.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.232+0000 7fccd3a00640 1 --2- 172.21.15.31:0/3949632850 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcccc1016a0 0x7fcccc101ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:41.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.232+0000 7fccd3a00640 1 -- 172.21.15.31:0/3949632850 >> 172.21.15.31:0/3949632850 conn(0x7fcccc0fcbd0 msgr2=0x7fcccc0ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:41.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.232+0000 7fccd3a00640 1 -- 172.21.15.31:0/3949632850 shutdown_connections 2024-03-21T10:35:41.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.232+0000 7fccd3a00640 1 -- 172.21.15.31:0/3949632850 wait complete. 2024-03-21T10:35:41.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.232+0000 7fccd3a00640 1 Processor -- start 2024-03-21T10:35:41.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.233+0000 7fccd3a00640 1 -- start start 2024-03-21T10:35:41.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.233+0000 7fccd3a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcccc1016a0 0x7fcccc197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:41.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.233+0000 7fccd3a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcccc102890 0x7fcccc197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:41.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.233+0000 7fccd3a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcccc103250 0x7fcccc19ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:41.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.233+0000 7fccd3a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc10d350 con 0x7fcccc102890 2024-03-21T10:35:41.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.233+0000 7fccd3a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc10d1d0 con 0x7fcccc1016a0 2024-03-21T10:35:41.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.233+0000 7fccd3a00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc10d4d0 con 0x7fcccc103250 2024-03-21T10:35:41.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.233+0000 7fccd1000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcccc1016a0 0x7fcccc197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:41.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.233+0000 7fcccbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcccc102890 0x7fcccc197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:41.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.233+0000 7fccd1000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcccc1016a0 0x7fcccc197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:43970/0 (socket says 172.21.15.31:43970) 2024-03-21T10:35:41.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.233+0000 7fccd1000640 1 -- 172.21.15.31:0/234233513 learned_addr learned my addr 172.21.15.31:0/234233513 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:41.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.233+0000 7fccd1a00640 1 --2- 172.21.15.31:0/234233513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcccc103250 0x7fcccc19ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:41.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.234+0000 7fccd1000640 1 -- 172.21.15.31:0/234233513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcccc103250 msgr2=0x7fcccc19ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:41.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.234+0000 7fccd1000640 1 --2- 172.21.15.31:0/234233513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcccc103250 0x7fcccc19ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:41.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.234+0000 7fccd1000640 1 -- 172.21.15.31:0/234233513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcccc102890 msgr2=0x7fcccc197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:41.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.234+0000 7fccd1000640 1 --2- 172.21.15.31:0/234233513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcccc102890 0x7fcccc197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:41.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.234+0000 7fccd1000640 1 -- 172.21.15.31:0/234233513 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc19f5b0 con 0x7fcccc1016a0 2024-03-21T10:35:41.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.234+0000 7fccd1000640 1 --2- 172.21.15.31:0/234233513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcccc1016a0 0x7fcccc197990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fccc000e9c0 tx=0x7fccc000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:41.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.234+0000 7fccc9600640 1 -- 172.21.15.31:0/234233513 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc000cd80 con 0x7fcccc1016a0 2024-03-21T10:35:41.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.234+0000 7fccc9600640 1 -- 172.21.15.31:0/234233513 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fccc00035f0 con 0x7fcccc1016a0 2024-03-21T10:35:41.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.234+0000 7fccd3a00640 1 -- 172.21.15.31:0/234233513 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcccc19f8a0 con 0x7fcccc1016a0 2024-03-21T10:35:41.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.234+0000 7fccc9600640 1 -- 172.21.15.31:0/234233513 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc0010690 con 0x7fcccc1016a0 2024-03-21T10:35:41.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.234+0000 7fccd3a00640 1 -- 172.21.15.31:0/234233513 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcccc19fe10 con 0x7fcccc1016a0 2024-03-21T10:35:41.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.235+0000 7fccd3a00640 1 -- 172.21.15.31:0/234233513 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcccc108a40 con 0x7fcccc1016a0 2024-03-21T10:35:41.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.237+0000 7fccc9600640 1 -- 172.21.15.31:0/234233513 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fccc00037a0 con 0x7fcccc1016a0 2024-03-21T10:35:41.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.238+0000 7fccc9600640 1 --2- 172.21.15.31:0/234233513 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fccb40a2fa0 0x7fccb40a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:41.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.239+0000 7fccc9600640 1 -- 172.21.15.31:0/234233513 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fccc00cf9f0 con 0x7fcccc1016a0 2024-03-21T10:35:41.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.239+0000 7fcccbe00640 1 --2- 172.21.15.31:0/234233513 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fccb40a2fa0 0x7fccb40a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:41.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.240+0000 7fcccbe00640 1 --2- 172.21.15.31:0/234233513 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fccb40a2fa0 0x7fccb40a5460 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fccb802c240 tx=0x7fccb802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:41.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.242+0000 7fccc9600640 1 -- 172.21.15.31:0/234233513 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fccc009c580 con 0x7fcccc1016a0 2024-03-21T10:35:41.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.502+0000 7fccd3a00640 1 -- 172.21.15.31:0/234233513 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fcccc061ae0 con 0x7fccb40a2fa0 2024-03-21T10:35:41.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.532+0000 7fccc9600640 1 -- 172.21.15.31:0/234233513 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fcccc061ae0 con 0x7fccb40a2fa0 2024-03-21T10:35:41.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.535+0000 7fccd3a00640 1 -- 172.21.15.31:0/234233513 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fccb40a2fa0 msgr2=0x7fccb40a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:41.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.535+0000 7fccd3a00640 1 --2- 172.21.15.31:0/234233513 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fccb40a2fa0 0x7fccb40a5460 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fccb802c240 tx=0x7fccb802df90 comp rx=0 tx=0).stop 2024-03-21T10:35:41.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.535+0000 7fccd3a00640 1 -- 172.21.15.31:0/234233513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcccc1016a0 msgr2=0x7fcccc197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:41.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.535+0000 7fccd3a00640 1 --2- 172.21.15.31:0/234233513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcccc1016a0 0x7fcccc197990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fccc000e9c0 tx=0x7fccc000ee80 comp rx=0 tx=0).stop 2024-03-21T10:35:41.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.536+0000 7fccd3a00640 1 -- 172.21.15.31:0/234233513 shutdown_connections 2024-03-21T10:35:41.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.536+0000 7fccd3a00640 1 --2- 172.21.15.31:0/234233513 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fccb40a2fa0 0x7fccb40a5460 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:41.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.536+0000 7fccd3a00640 1 --2- 172.21.15.31:0/234233513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcccc103250 0x7fcccc19ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:41.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.536+0000 7fccd3a00640 1 --2- 172.21.15.31:0/234233513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcccc102890 0x7fcccc197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:41.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.536+0000 7fccd3a00640 1 --2- 172.21.15.31:0/234233513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcccc1016a0 0x7fcccc197990 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:41.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.536+0000 7fccd3a00640 1 -- 172.21.15.31:0/234233513 >> 172.21.15.31:0/234233513 conn(0x7fcccc0fcbd0 msgr2=0x7fcccc105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:41.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.536+0000 7fccd3a00640 1 -- 172.21.15.31:0/234233513 shutdown_connections 2024-03-21T10:35:41.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.536+0000 7fccd3a00640 1 -- 172.21.15.31:0/234233513 wait complete. 2024-03-21T10:35:41.555 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-21T10:35:41.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:41 smithi070 ceph-mon[36330]: pgmap v342: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-21T10:35:41.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:41 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-21T10:35:41.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:41 smithi070 ceph-mon[36330]: mds.? [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] up:active 2024-03-21T10:35:41.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:41 smithi070 ceph-mon[36330]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-21T10:35:41.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:41 smithi070 ceph-mon[36330]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-21T10:35:41.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:41 smithi070 ceph-mon[36330]: Cluster is now healthy 2024-03-21T10:35:41.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:41 smithi070 ceph-mon[36330]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-03-21T10:35:41.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:41 smithi070 ceph-mon[36330]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:creating} 9 up:standby 2024-03-21T10:35:41.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:41 smithi070 ceph-mon[36330]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-21T10:35:41.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:41 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/517860720' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-21T10:35:41.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:41 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-21T10:35:41.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:41 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-21T10:35:41.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:41 smithi070 ceph-mon[36330]: mds.? [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] up:active 2024-03-21T10:35:41.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:41 smithi070 ceph-mon[36330]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:35:41.622 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:35:41 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:35:41.512+0000 7f89ed400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-21T10:35:41.622 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:35:41 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:35:41.512+0000 7f89ed400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-21T10:35:41.622 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:35:41 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:35:41.512+0000 7f89ed400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-21T10:35:41.622 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:35:41 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:35:41.512+0000 7f89ed400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-21T10:35:41.622 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:35:41 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:35:41.512+0000 7f89ed400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-21T10:35:41.623 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:41 smithi031 ceph-mon[31673]: pgmap v342: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-21T10:35:41.623 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:41 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-21T10:35:41.623 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:41 smithi031 ceph-mon[31673]: mds.? [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] up:active 2024-03-21T10:35:41.623 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:41 smithi031 ceph-mon[31673]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-21T10:35:41.623 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:41 smithi031 ceph-mon[31673]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-21T10:35:41.623 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:41 smithi031 ceph-mon[31673]: Cluster is now healthy 2024-03-21T10:35:41.623 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:41 smithi031 ceph-mon[31673]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-03-21T10:35:41.623 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:41 smithi031 ceph-mon[31673]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:creating} 9 up:standby 2024-03-21T10:35:41.623 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:41 smithi031 ceph-mon[31673]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-21T10:35:41.623 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:41 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/517860720' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-21T10:35:41.623 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:41 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-21T10:35:41.623 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:41 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-21T10:35:41.624 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:41 smithi031 ceph-mon[31673]: mds.? [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] up:active 2024-03-21T10:35:41.624 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:41 smithi031 ceph-mon[31673]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:35:41.681 INFO:journalctl@ceph.mds.b.smithi040.stdout:Mar 21 10:35:41 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-b[66460]: 2024-03-21T10:35:41.438+0000 7fe6c5734700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-21T10:35:41.681 INFO:journalctl@ceph.mds.b.smithi040.stdout:Mar 21 10:35:41 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-b[66460]: 2024-03-21T10:35:41.438+0000 7fe6c5734700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-21T10:35:41.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:41 smithi040 ceph-mon[36154]: pgmap v342: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-21T10:35:41.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:41 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-21T10:35:41.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:41 smithi040 ceph-mon[36154]: mds.? [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] up:active 2024-03-21T10:35:41.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:41 smithi040 ceph-mon[36154]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-21T10:35:41.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:41 smithi040 ceph-mon[36154]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-21T10:35:41.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:41 smithi040 ceph-mon[36154]: Cluster is now healthy 2024-03-21T10:35:41.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:41 smithi040 ceph-mon[36154]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-03-21T10:35:41.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:41 smithi040 ceph-mon[36154]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:creating} 9 up:standby 2024-03-21T10:35:41.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:41 smithi040 ceph-mon[36154]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-21T10:35:41.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:41 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/517860720' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-21T10:35:41.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:41 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-21T10:35:41.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:41 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-21T10:35:41.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:41 smithi040 ceph-mon[36154]: mds.? [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] up:active 2024-03-21T10:35:41.683 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:41 smithi040 ceph-mon[36154]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:35:41.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.771+0000 7f5224600640 1 -- 172.21.15.31:0/2940626623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f521c102140 msgr2=0x7f521c104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:41.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.771+0000 7f5224600640 1 --2- 172.21.15.31:0/2940626623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f521c102140 0x7f521c104530 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5210009ca0 tx=0x7f521002c100 comp rx=0 tx=0).stop 2024-03-21T10:35:41.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.771+0000 7f5224600640 1 -- 172.21.15.31:0/2940626623 shutdown_connections 2024-03-21T10:35:41.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.771+0000 7f5224600640 1 --2- 172.21.15.31:0/2940626623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f521c1073a0 0x7f521c1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:41.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.771+0000 7f5224600640 1 --2- 172.21.15.31:0/2940626623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f521c104a70 0x7f521c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:41.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.771+0000 7f5224600640 1 --2- 172.21.15.31:0/2940626623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f521c102140 0x7f521c104530 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:41.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.771+0000 7f5224600640 1 -- 172.21.15.31:0/2940626623 >> 172.21.15.31:0/2940626623 conn(0x7f521c0fbb30 msgr2=0x7f521c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:41.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.771+0000 7f5224600640 1 -- 172.21.15.31:0/2940626623 shutdown_connections 2024-03-21T10:35:41.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.771+0000 7f5224600640 1 -- 172.21.15.31:0/2940626623 wait complete. 2024-03-21T10:35:41.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.772+0000 7f5224600640 1 Processor -- start 2024-03-21T10:35:41.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.772+0000 7f5224600640 1 -- start start 2024-03-21T10:35:41.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.773+0000 7f5224600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f521c102140 0x7f521c199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:41.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.773+0000 7f5224600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f521c104a70 0x7f521c19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:41.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.773+0000 7f5224600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f521c1073a0 0x7f521c1a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:41.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.773+0000 7f5224600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521c10d360 con 0x7f521c1073a0 2024-03-21T10:35:41.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.773+0000 7f5224600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521c10d1e0 con 0x7f521c104a70 2024-03-21T10:35:41.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.773+0000 7f5224600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521c10d4e0 con 0x7f521c102140 2024-03-21T10:35:41.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.773+0000 7f5221200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f521c104a70 0x7f521c19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:41.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.773+0000 7f5221c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f521c102140 0x7f521c199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:41.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.773+0000 7f5221200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f521c104a70 0x7f521c19a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:43982/0 (socket says 172.21.15.31:43982) 2024-03-21T10:35:41.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.773+0000 7f5221200640 1 -- 172.21.15.31:0/3517514335 learned_addr learned my addr 172.21.15.31:0/3517514335 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:41.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.773+0000 7f5221c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f521c102140 0x7f521c199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53638/0 (socket says 172.21.15.31:53638) 2024-03-21T10:35:41.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.773+0000 7f5222600640 1 --2- 172.21.15.31:0/3517514335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f521c1073a0 0x7f521c1a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:41.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.773+0000 7f5221200640 1 -- 172.21.15.31:0/3517514335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f521c102140 msgr2=0x7f521c199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:41.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.773+0000 7f5221200640 1 --2- 172.21.15.31:0/3517514335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f521c102140 0x7f521c199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:41.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.773+0000 7f5221200640 1 -- 172.21.15.31:0/3517514335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f521c1073a0 msgr2=0x7f521c1a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:41.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.773+0000 7f5221200640 1 --2- 172.21.15.31:0/3517514335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f521c1073a0 0x7f521c1a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:41.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.773+0000 7f5221200640 1 -- 172.21.15.31:0/3517514335 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521c1a18b0 con 0x7f521c104a70 2024-03-21T10:35:41.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.774+0000 7f5221c00640 1 --2- 172.21.15.31:0/3517514335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f521c102140 0x7f521c199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:35:41.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.774+0000 7f5221200640 1 --2- 172.21.15.31:0/3517514335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f521c104a70 0x7f521c19a1b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f520800e9f0 tx=0x7f520800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:41.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.774+0000 7f521a000640 1 -- 172.21.15.31:0/3517514335 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520800ccc0 con 0x7f521c104a70 2024-03-21T10:35:41.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.774+0000 7f521a000640 1 -- 172.21.15.31:0/3517514335 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5208005500 con 0x7f521c104a70 2024-03-21T10:35:41.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.774+0000 7f5224600640 1 -- 172.21.15.31:0/3517514335 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f521c1a1ba0 con 0x7f521c104a70 2024-03-21T10:35:41.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.774+0000 7f521a000640 1 -- 172.21.15.31:0/3517514335 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5208010690 con 0x7f521c104a70 2024-03-21T10:35:41.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.774+0000 7f5224600640 1 -- 172.21.15.31:0/3517514335 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f521c1a2070 con 0x7f521c104a70 2024-03-21T10:35:41.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.775+0000 7f5224600640 1 -- 172.21.15.31:0/3517514335 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f521c108ef0 con 0x7f521c104a70 2024-03-21T10:35:41.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.777+0000 7f521a000640 1 -- 172.21.15.31:0/3517514335 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f52080046c0 con 0x7f521c104a70 2024-03-21T10:35:41.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.778+0000 7f521a000640 1 --2- 172.21.15.31:0/3517514335 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f52040a2f20 0x7f52040a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:41.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.779+0000 7f521a000640 1 -- 172.21.15.31:0/3517514335 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f52080cf9d0 con 0x7f521c104a70 2024-03-21T10:35:41.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.779+0000 7f5221c00640 1 --2- 172.21.15.31:0/3517514335 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f52040a2f20 0x7f52040a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:41.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.780+0000 7f5221c00640 1 --2- 172.21.15.31:0/3517514335 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f52040a2f20 0x7f52040a53e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f521002c710 tx=0x7f5210005ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:41.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:41.782+0000 7f521a000640 1 -- 172.21.15.31:0/3517514335 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f520809c560 con 0x7f521c104a70 2024-03-21T10:35:42.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.016+0000 7f5224600640 1 -- 172.21.15.31:0/3517514335 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f521c061ae0 con 0x7f52040a2f20 2024-03-21T10:35:42.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.047+0000 7f521a000640 1 -- 172.21.15.31:0/3517514335 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f521c061ae0 con 0x7f52040a2f20 2024-03-21T10:35:42.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.050+0000 7f5224600640 1 -- 172.21.15.31:0/3517514335 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f52040a2f20 msgr2=0x7f52040a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:42.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.050+0000 7f5224600640 1 --2- 172.21.15.31:0/3517514335 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f52040a2f20 0x7f52040a53e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f521002c710 tx=0x7f5210005ef0 comp rx=0 tx=0).stop 2024-03-21T10:35:42.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.050+0000 7f5224600640 1 -- 172.21.15.31:0/3517514335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f521c104a70 msgr2=0x7f521c19a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:42.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.050+0000 7f5224600640 1 --2- 172.21.15.31:0/3517514335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f521c104a70 0x7f521c19a1b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f520800e9f0 tx=0x7f520800eeb0 comp rx=0 tx=0).stop 2024-03-21T10:35:42.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.050+0000 7f5224600640 1 -- 172.21.15.31:0/3517514335 shutdown_connections 2024-03-21T10:35:42.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.050+0000 7f5224600640 1 --2- 172.21.15.31:0/3517514335 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f52040a2f20 0x7f52040a53e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:42.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.050+0000 7f5224600640 1 --2- 172.21.15.31:0/3517514335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f521c1073a0 0x7f521c1a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:42.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.050+0000 7f5224600640 1 --2- 172.21.15.31:0/3517514335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f521c104a70 0x7f521c19a1b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:42.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.050+0000 7f5224600640 1 --2- 172.21.15.31:0/3517514335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f521c102140 0x7f521c199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:42.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.050+0000 7f5224600640 1 -- 172.21.15.31:0/3517514335 >> 172.21.15.31:0/3517514335 conn(0x7f521c0fbb30 msgr2=0x7f521c1018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:42.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.051+0000 7f5224600640 1 -- 172.21.15.31:0/3517514335 shutdown_connections 2024-03-21T10:35:42.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.051+0000 7f5224600640 1 -- 172.21.15.31:0/3517514335 wait complete. 2024-03-21T10:35:42.077 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-21T10:35:42.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.306+0000 7f146fe00640 1 -- 172.21.15.31:0/515284683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1470103700 msgr2=0x7f1470103b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:42.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.306+0000 7f146fe00640 1 --2- 172.21.15.31:0/515284683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1470103700 0x7f1470103b20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1460009900 tx=0x7f146002bc40 comp rx=0 tx=0).stop 2024-03-21T10:35:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.306+0000 7f146fe00640 1 -- 172.21.15.31:0/515284683 shutdown_connections 2024-03-21T10:35:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.306+0000 7f146fe00640 1 --2- 172.21.15.31:0/515284683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14701052b0 0x7f147010bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.306+0000 7f146fe00640 1 --2- 172.21.15.31:0/515284683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14701048f0 0x7f1470104d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.306+0000 7f146fe00640 1 --2- 172.21.15.31:0/515284683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1470103700 0x7f1470103b20 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.306+0000 7f146fe00640 1 -- 172.21.15.31:0/515284683 >> 172.21.15.31:0/515284683 conn(0x7f14700fec70 msgr2=0x7f14701010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.306+0000 7f146fe00640 1 -- 172.21.15.31:0/515284683 shutdown_connections 2024-03-21T10:35:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.306+0000 7f146fe00640 1 -- 172.21.15.31:0/515284683 wait complete. 2024-03-21T10:35:42.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.307+0000 7f146fe00640 1 Processor -- start 2024-03-21T10:35:42.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.308+0000 7f146fe00640 1 -- start start 2024-03-21T10:35:42.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.308+0000 7f146fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1470103700 0x7f1470199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:42.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.308+0000 7f146fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14701048f0 0x7f1470199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:42.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.308+0000 7f146fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14701052b0 0x7f14701a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:42.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.308+0000 7f146fe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147010f3a0 con 0x7f14701052b0 2024-03-21T10:35:42.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.308+0000 7f146fe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147010f220 con 0x7f1470103700 2024-03-21T10:35:42.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.308+0000 7f146fe00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147010f520 con 0x7f14701048f0 2024-03-21T10:35:42.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.308+0000 7f146ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1470103700 0x7f1470199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:42.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.308+0000 7f146ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1470103700 0x7f1470199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:43990/0 (socket says 172.21.15.31:43990) 2024-03-21T10:35:42.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.308+0000 7f146ea00640 1 -- 172.21.15.31:0/2356887730 learned_addr learned my addr 172.21.15.31:0/2356887730 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:42.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.308+0000 7f1467e00640 1 --2- 172.21.15.31:0/2356887730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14701048f0 0x7f1470199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:42.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.309+0000 7f146f400640 1 --2- 172.21.15.31:0/2356887730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14701052b0 0x7f14701a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:42.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.309+0000 7f146ea00640 1 -- 172.21.15.31:0/2356887730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14701048f0 msgr2=0x7f1470199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:42.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.309+0000 7f146ea00640 1 --2- 172.21.15.31:0/2356887730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14701048f0 0x7f1470199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:42.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.309+0000 7f146ea00640 1 -- 172.21.15.31:0/2356887730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14701052b0 msgr2=0x7f14701a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:42.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.309+0000 7f146ea00640 1 --2- 172.21.15.31:0/2356887730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14701052b0 0x7f14701a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:42.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.309+0000 7f146ea00640 1 -- 172.21.15.31:0/2356887730 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14701a1630 con 0x7f1470103700 2024-03-21T10:35:42.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.309+0000 7f1467e00640 1 --2- 172.21.15.31:0/2356887730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14701048f0 0x7f1470199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:35:42.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.309+0000 7f146ea00640 1 --2- 172.21.15.31:0/2356887730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1470103700 0x7f1470199a10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f146002efd0 tx=0x7f1460005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:42.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.310+0000 7f1467400640 1 -- 172.21.15.31:0/2356887730 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1460041070 con 0x7f1470103700 2024-03-21T10:35:42.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.310+0000 7f1467400640 1 -- 172.21.15.31:0/2356887730 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14600044d0 con 0x7f1470103700 2024-03-21T10:35:42.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.310+0000 7f146fe00640 1 -- 172.21.15.31:0/2356887730 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14701a18c0 con 0x7f1470103700 2024-03-21T10:35:42.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.310+0000 7f1467400640 1 -- 172.21.15.31:0/2356887730 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1460034440 con 0x7f1470103700 2024-03-21T10:35:42.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.310+0000 7f146fe00640 1 -- 172.21.15.31:0/2356887730 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14701a1dd0 con 0x7f1470103700 2024-03-21T10:35:42.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.311+0000 7f146fe00640 1 -- 172.21.15.31:0/2356887730 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f147010aa20 con 0x7f1470103700 2024-03-21T10:35:42.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.313+0000 7f1467400640 1 -- 172.21.15.31:0/2356887730 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f1460044070 con 0x7f1470103700 2024-03-21T10:35:42.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.315+0000 7f1467400640 1 --2- 172.21.15.31:0/2356887730 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f14540a2fa0 0x7f14540a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:42.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.315+0000 7f1467400640 1 -- 172.21.15.31:0/2356887730 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f14600f11d0 con 0x7f1470103700 2024-03-21T10:35:42.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.315+0000 7f1467e00640 1 --2- 172.21.15.31:0/2356887730 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f14540a2fa0 0x7f14540a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:42.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.316+0000 7f1467e00640 1 --2- 172.21.15.31:0/2356887730 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f14540a2fa0 0x7f14540a5460 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f147019af20 tx=0x7f1458005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:42.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.318+0000 7f1467400640 1 -- 172.21.15.31:0/2356887730 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f14600bdd60 con 0x7f1470103700 2024-03-21T10:35:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.332+0000 7f1467400640 1 -- 172.21.15.31:0/2356887730 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7f14600b9cd0 con 0x7f1470103700 2024-03-21T10:35:42.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:42 smithi031 ceph-mon[31673]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:35:42.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.564+0000 7f146fe00640 1 -- 172.21.15.31:0/2356887730 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f1470061b70 con 0x7f14540a2fa0 2024-03-21T10:35:42.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.570+0000 7f1467400640 1 -- 172.21.15.31:0/2356887730 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f1470061b70 con 0x7f14540a2fa0 2024-03-21T10:35:42.569 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/qa/sv_0/dfc970d3-3ac6-4cf7-ae09-4e7eff6b7d44 2024-03-21T10:35:42.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.572+0000 7f146fe00640 1 -- 172.21.15.31:0/2356887730 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f14540a2fa0 msgr2=0x7f14540a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:42.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.572+0000 7f146fe00640 1 --2- 172.21.15.31:0/2356887730 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f14540a2fa0 0x7f14540a5460 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f147019af20 tx=0x7f1458005f90 comp rx=0 tx=0).stop 2024-03-21T10:35:42.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.572+0000 7f146fe00640 1 -- 172.21.15.31:0/2356887730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1470103700 msgr2=0x7f1470199a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:42.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.572+0000 7f146fe00640 1 --2- 172.21.15.31:0/2356887730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1470103700 0x7f1470199a10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f146002efd0 tx=0x7f1460005d50 comp rx=0 tx=0).stop 2024-03-21T10:35:42.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.572+0000 7f146fe00640 1 -- 172.21.15.31:0/2356887730 shutdown_connections 2024-03-21T10:35:42.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.572+0000 7f146fe00640 1 --2- 172.21.15.31:0/2356887730 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f14540a2fa0 0x7f14540a5460 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:42.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.572+0000 7f146fe00640 1 --2- 172.21.15.31:0/2356887730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14701052b0 0x7f14701a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:42.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.572+0000 7f146fe00640 1 --2- 172.21.15.31:0/2356887730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14701048f0 0x7f1470199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:42.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.572+0000 7f146fe00640 1 --2- 172.21.15.31:0/2356887730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1470103700 0x7f1470199a10 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:42.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.572+0000 7f146fe00640 1 -- 172.21.15.31:0/2356887730 >> 172.21.15.31:0/2356887730 conn(0x7f14700fec70 msgr2=0x7f1470107bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:42.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.573+0000 7f146fe00640 1 -- 172.21.15.31:0/2356887730 shutdown_connections 2024-03-21T10:35:42.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.573+0000 7f146fe00640 1 -- 172.21.15.31:0/2356887730 wait complete. 2024-03-21T10:35:42.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:42 smithi070 ceph-mon[36330]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:35:42.592 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-21T10:35:42.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:42 smithi040 ceph-mon[36154]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:35:42.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.824+0000 7ffa39000640 1 -- 172.21.15.31:0/2893645855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa341026d0 msgr2=0x7ffa34104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:42.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.824+0000 7ffa39000640 1 --2- 172.21.15.31:0/2893645855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa341026d0 0x7ffa34104b40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ffa20009900 tx=0x7ffa2002bc30 comp rx=0 tx=0).stop 2024-03-21T10:35:42.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.824+0000 7ffa39000640 1 -- 172.21.15.31:0/2893645855 shutdown_connections 2024-03-21T10:35:42.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.824+0000 7ffa39000640 1 --2- 172.21.15.31:0/2893645855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa34105080 0x7ffa34105500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:42.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.824+0000 7ffa39000640 1 --2- 172.21.15.31:0/2893645855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa341026d0 0x7ffa34104b40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:42.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.824+0000 7ffa39000640 1 --2- 172.21.15.31:0/2893645855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa340ffd60 0x7ffa34102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:42.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.824+0000 7ffa39000640 1 -- 172.21.15.31:0/2893645855 >> 172.21.15.31:0/2893645855 conn(0x7ffa340fb980 msgr2=0x7ffa340fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:42.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.824+0000 7ffa39000640 1 -- 172.21.15.31:0/2893645855 shutdown_connections 2024-03-21T10:35:42.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.824+0000 7ffa39000640 1 -- 172.21.15.31:0/2893645855 wait complete. 2024-03-21T10:35:42.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.825+0000 7ffa39000640 1 Processor -- start 2024-03-21T10:35:42.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.826+0000 7ffa39000640 1 -- start start 2024-03-21T10:35:42.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.826+0000 7ffa39000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa340ffd60 0x7ffa3419dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:42.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.826+0000 7ffa39000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa341026d0 0x7ffa3419e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:42.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.826+0000 7ffa39000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa34105080 0x7ffa341a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:42.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.826+0000 7ffa39000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa34113790 con 0x7ffa340ffd60 2024-03-21T10:35:42.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.826+0000 7ffa39000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa34113610 con 0x7ffa34105080 2024-03-21T10:35:42.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.826+0000 7ffa39000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa34113910 con 0x7ffa341026d0 2024-03-21T10:35:42.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.826+0000 7ffa32a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa341026d0 0x7ffa3419e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:42.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.826+0000 7ffa33400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa340ffd60 0x7ffa3419dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:42.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.827+0000 7ffa32a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa341026d0 0x7ffa3419e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53668/0 (socket says 172.21.15.31:53668) 2024-03-21T10:35:42.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.827+0000 7ffa32a00640 1 -- 172.21.15.31:0/1064963440 learned_addr learned my addr 172.21.15.31:0/1064963440 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:42.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.827+0000 7ffa33e00640 1 --2- 172.21.15.31:0/1064963440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa34105080 0x7ffa341a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:42.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.827+0000 7ffa32a00640 1 -- 172.21.15.31:0/1064963440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa34105080 msgr2=0x7ffa341a5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:42.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.827+0000 7ffa32a00640 1 --2- 172.21.15.31:0/1064963440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa34105080 0x7ffa341a5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:42.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.827+0000 7ffa32a00640 1 -- 172.21.15.31:0/1064963440 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa340ffd60 msgr2=0x7ffa3419dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:42.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.827+0000 7ffa32a00640 1 --2- 172.21.15.31:0/1064963440 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa340ffd60 0x7ffa3419dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:42.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.827+0000 7ffa32a00640 1 -- 172.21.15.31:0/1064963440 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa341a5a00 con 0x7ffa341026d0 2024-03-21T10:35:42.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.827+0000 7ffa32a00640 1 --2- 172.21.15.31:0/1064963440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa341026d0 0x7ffa3419e320 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ffa20005ab0 tx=0x7ffa2002c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:42.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.828+0000 7ffa2be00640 1 -- 172.21.15.31:0/1064963440 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa20043020 con 0x7ffa341026d0 2024-03-21T10:35:42.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.828+0000 7ffa2be00640 1 -- 172.21.15.31:0/1064963440 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa200054e0 con 0x7ffa341026d0 2024-03-21T10:35:42.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.828+0000 7ffa39000640 1 -- 172.21.15.31:0/1064963440 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa341a5c90 con 0x7ffa341026d0 2024-03-21T10:35:42.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.828+0000 7ffa2be00640 1 -- 172.21.15.31:0/1064963440 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa20003ce0 con 0x7ffa341026d0 2024-03-21T10:35:42.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.828+0000 7ffa39000640 1 -- 172.21.15.31:0/1064963440 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa341a61d0 con 0x7ffa341026d0 2024-03-21T10:35:42.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.829+0000 7ffa39000640 1 -- 172.21.15.31:0/1064963440 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa340fd470 con 0x7ffa341026d0 2024-03-21T10:35:42.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.831+0000 7ffa2be00640 1 -- 172.21.15.31:0/1064963440 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7ffa20046020 con 0x7ffa341026d0 2024-03-21T10:35:42.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.833+0000 7ffa2be00640 1 --2- 172.21.15.31:0/1064963440 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ffa180a3030 0x7ffa180a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:42.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.833+0000 7ffa2be00640 1 -- 172.21.15.31:0/1064963440 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7ffa200f1780 con 0x7ffa341026d0 2024-03-21T10:35:42.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.833+0000 7ffa33400640 1 --2- 172.21.15.31:0/1064963440 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ffa180a3030 0x7ffa180a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:42.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.834+0000 7ffa33400640 1 --2- 172.21.15.31:0/1064963440 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ffa180a3030 0x7ffa180a54f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009bd0 tx=0x7ffa24005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:42.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:42.836+0000 7ffa2be00640 1 -- 172.21.15.31:0/1064963440 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ffa2004d020 con 0x7ffa341026d0 2024-03-21T10:35:43.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.073+0000 7ffa39000640 1 -- 172.21.15.31:0/1064963440 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa34061b70 con 0x7ffa180a3030 2024-03-21T10:35:43.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.096+0000 7ffa2be00640 1 -- 172.21.15.31:0/1064963440 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ffa34061b70 con 0x7ffa180a3030 2024-03-21T10:35:43.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.099+0000 7ffa39000640 1 -- 172.21.15.31:0/1064963440 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ffa180a3030 msgr2=0x7ffa180a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:43.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.099+0000 7ffa39000640 1 --2- 172.21.15.31:0/1064963440 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ffa180a3030 0x7ffa180a54f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009bd0 tx=0x7ffa24005f90 comp rx=0 tx=0).stop 2024-03-21T10:35:43.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.099+0000 7ffa39000640 1 -- 172.21.15.31:0/1064963440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa341026d0 msgr2=0x7ffa3419e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:43.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.099+0000 7ffa39000640 1 --2- 172.21.15.31:0/1064963440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa341026d0 0x7ffa3419e320 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ffa20005ab0 tx=0x7ffa2002c6c0 comp rx=0 tx=0).stop 2024-03-21T10:35:43.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.099+0000 7ffa39000640 1 -- 172.21.15.31:0/1064963440 shutdown_connections 2024-03-21T10:35:43.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.099+0000 7ffa39000640 1 --2- 172.21.15.31:0/1064963440 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ffa180a3030 0x7ffa180a54f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:43.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.099+0000 7ffa39000640 1 --2- 172.21.15.31:0/1064963440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa34105080 0x7ffa341a5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:43.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.099+0000 7ffa39000640 1 --2- 172.21.15.31:0/1064963440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa341026d0 0x7ffa3419e320 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:43.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.099+0000 7ffa39000640 1 --2- 172.21.15.31:0/1064963440 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa340ffd60 0x7ffa3419dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:43.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.099+0000 7ffa39000640 1 -- 172.21.15.31:0/1064963440 >> 172.21.15.31:0/1064963440 conn(0x7ffa340fb980 msgr2=0x7ffa34100ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:43.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.100+0000 7ffa39000640 1 -- 172.21.15.31:0/1064963440 shutdown_connections 2024-03-21T10:35:43.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.100+0000 7ffa39000640 1 -- 172.21.15.31:0/1064963440 wait complete. 2024-03-21T10:35:43.119 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-21T10:35:43.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.348+0000 7f1685400640 1 -- 172.21.15.31:0/474292765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16801014e0 msgr2=0x7f1680101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:43.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.348+0000 7f1685400640 1 --2- 172.21.15.31:0/474292765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16801014e0 0x7f1680101900 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1670009900 tx=0x7f167002bc40 comp rx=0 tx=0).stop 2024-03-21T10:35:43.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.348+0000 7f1685400640 1 -- 172.21.15.31:0/474292765 shutdown_connections 2024-03-21T10:35:43.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.348+0000 7f1685400640 1 --2- 172.21.15.31:0/474292765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1680103090 0x7f1680109920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:43.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.348+0000 7f1685400640 1 --2- 172.21.15.31:0/474292765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16801026d0 0x7f1680102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:43.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.348+0000 7f1685400640 1 --2- 172.21.15.31:0/474292765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16801014e0 0x7f1680101900 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:43.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.348+0000 7f1685400640 1 -- 172.21.15.31:0/474292765 >> 172.21.15.31:0/474292765 conn(0x7f16800fca50 msgr2=0x7f16800fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:43.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.348+0000 7f1685400640 1 -- 172.21.15.31:0/474292765 shutdown_connections 2024-03-21T10:35:43.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.348+0000 7f1685400640 1 -- 172.21.15.31:0/474292765 wait complete. 2024-03-21T10:35:43.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.349+0000 7f1685400640 1 Processor -- start 2024-03-21T10:35:43.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.349+0000 7f1685400640 1 -- start start 2024-03-21T10:35:43.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.349+0000 7f1685400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16801014e0 0x7f1680199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:43.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.349+0000 7f1685400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16801026d0 0x7f1680199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:43.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.349+0000 7f1685400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1680103090 0x7f16801a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:43.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.350+0000 7f1685400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168010d1c0 con 0x7f1680103090 2024-03-21T10:35:43.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.350+0000 7f1685400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168010d040 con 0x7f16801014e0 2024-03-21T10:35:43.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.350+0000 7f1685400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168010d340 con 0x7f16801026d0 2024-03-21T10:35:43.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.350+0000 7f167f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16801026d0 0x7f1680199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:43.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.350+0000 7f167fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16801014e0 0x7f1680199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:43.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.350+0000 7f167fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16801014e0 0x7f1680199a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44024/0 (socket says 172.21.15.31:44024) 2024-03-21T10:35:43.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.350+0000 7f1684a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1680103090 0x7f16801a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:43.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.350+0000 7f167fe00640 1 -- 172.21.15.31:0/1529331087 learned_addr learned my addr 172.21.15.31:0/1529331087 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:43.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.350+0000 7f167f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16801026d0 0x7f1680199f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53692/0 (socket says 172.21.15.31:53692) 2024-03-21T10:35:43.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.350+0000 7f167f400640 1 -- 172.21.15.31:0/1529331087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16801014e0 msgr2=0x7f1680199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:43.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.350+0000 7f167f400640 1 --2- 172.21.15.31:0/1529331087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16801014e0 0x7f1680199a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:43.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.350+0000 7f167f400640 1 -- 172.21.15.31:0/1529331087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1680103090 msgr2=0x7f16801a0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:43.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.350+0000 7f167f400640 1 --2- 172.21.15.31:0/1529331087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1680103090 0x7f16801a0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:43.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.350+0000 7f167f400640 1 -- 172.21.15.31:0/1529331087 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16801a1650 con 0x7f16801026d0 2024-03-21T10:35:43.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.350+0000 7f167fe00640 1 --2- 172.21.15.31:0/1529331087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16801014e0 0x7f1680199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:35:43.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.351+0000 7f167f400640 1 --2- 172.21.15.31:0/1529331087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16801026d0 0x7f1680199f70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f166c00d6d0 tx=0x7f166c00db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:43.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.351+0000 7f167cc00640 1 -- 172.21.15.31:0/1529331087 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f166c017070 con 0x7f16801026d0 2024-03-21T10:35:43.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.351+0000 7f167cc00640 1 -- 172.21.15.31:0/1529331087 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f166c009820 con 0x7f16801026d0 2024-03-21T10:35:43.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.351+0000 7f1685400640 1 -- 172.21.15.31:0/1529331087 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16801a1940 con 0x7f16801026d0 2024-03-21T10:35:43.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.351+0000 7f167cc00640 1 -- 172.21.15.31:0/1529331087 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f166c004d00 con 0x7f16801026d0 2024-03-21T10:35:43.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.351+0000 7f1685400640 1 -- 172.21.15.31:0/1529331087 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16801a1e90 con 0x7f16801026d0 2024-03-21T10:35:43.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.352+0000 7f1685400640 1 -- 172.21.15.31:0/1529331087 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16801a20c0 con 0x7f16801026d0 2024-03-21T10:35:43.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.355+0000 7f167cc00640 1 -- 172.21.15.31:0/1529331087 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7f166c0099d0 con 0x7f16801026d0 2024-03-21T10:35:43.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.357+0000 7f167cc00640 1 --2- 172.21.15.31:0/1529331087 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f16640a2fb0 0x7f16640a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:43.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.357+0000 7f167cc00640 1 -- 172.21.15.31:0/1529331087 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f166c0cf320 con 0x7f16801026d0 2024-03-21T10:35:43.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.357+0000 7f167fe00640 1 --2- 172.21.15.31:0/1529331087 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f16640a2fb0 0x7f16640a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:43.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.358+0000 7f167fe00640 1 --2- 172.21.15.31:0/1529331087 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f16640a2fb0 0x7f16640a5470 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1670005ab0 tx=0x7f167002cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:43.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.361+0000 7f167cc00640 1 -- 172.21.15.31:0/1529331087 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f166c09b700 con 0x7f16801026d0 2024-03-21T10:35:43.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:43 smithi031 ceph-mon[31673]: pgmap v343: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 2 op/s 2024-03-21T10:35:43.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:43 smithi031 ceph-mon[31673]: from='client.24848 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:35:43.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:43 smithi031 ceph-mon[31673]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-21T10:35:43.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:43 smithi031 ceph-mon[31673]: from='client.24854 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:35:43.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:43 smithi070 ceph-mon[36330]: pgmap v343: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 2 op/s 2024-03-21T10:35:43.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:43 smithi070 ceph-mon[36330]: from='client.24848 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:35:43.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:43 smithi070 ceph-mon[36330]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-21T10:35:43.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:43 smithi070 ceph-mon[36330]: from='client.24854 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:35:43.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:43 smithi040 ceph-mon[36154]: pgmap v343: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 2 op/s 2024-03-21T10:35:43.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:43 smithi040 ceph-mon[36154]: from='client.24848 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:35:43.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:43 smithi040 ceph-mon[36154]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-21T10:35:43.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:43 smithi040 ceph-mon[36154]: from='client.24854 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:35:43.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.691+0000 7f1685400640 1 -- 172.21.15.31:0/1529331087 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1680074ff0 con 0x7f16640a2fb0 2024-03-21T10:35:43.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.697+0000 7f167cc00640 1 -- 172.21.15.31:0/1529331087 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f1680074ff0 con 0x7f16640a2fb0 2024-03-21T10:35:43.696 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/qa/sv_1/b825424c-c202-4d75-aea8-4547df630897 2024-03-21T10:35:43.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.699+0000 7f1685400640 1 -- 172.21.15.31:0/1529331087 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f16640a2fb0 msgr2=0x7f16640a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:43.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.699+0000 7f1685400640 1 --2- 172.21.15.31:0/1529331087 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f16640a2fb0 0x7f16640a5470 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1670005ab0 tx=0x7f167002cf90 comp rx=0 tx=0).stop 2024-03-21T10:35:43.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.699+0000 7f1685400640 1 -- 172.21.15.31:0/1529331087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16801026d0 msgr2=0x7f1680199f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:43.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.699+0000 7f1685400640 1 --2- 172.21.15.31:0/1529331087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16801026d0 0x7f1680199f70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f166c00d6d0 tx=0x7f166c00db90 comp rx=0 tx=0).stop 2024-03-21T10:35:43.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.699+0000 7f1685400640 1 -- 172.21.15.31:0/1529331087 shutdown_connections 2024-03-21T10:35:43.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.699+0000 7f1685400640 1 --2- 172.21.15.31:0/1529331087 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f16640a2fb0 0x7f16640a5470 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:43.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.699+0000 7f1685400640 1 --2- 172.21.15.31:0/1529331087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1680103090 0x7f16801a0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:43.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.699+0000 7f1685400640 1 --2- 172.21.15.31:0/1529331087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16801026d0 0x7f1680199f70 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:43.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.699+0000 7f1685400640 1 --2- 172.21.15.31:0/1529331087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16801014e0 0x7f1680199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:43.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.699+0000 7f1685400640 1 -- 172.21.15.31:0/1529331087 >> 172.21.15.31:0/1529331087 conn(0x7f16800fca50 msgr2=0x7f16800738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:43.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.700+0000 7f1685400640 1 -- 172.21.15.31:0/1529331087 shutdown_connections 2024-03-21T10:35:43.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.700+0000 7f1685400640 1 -- 172.21.15.31:0/1529331087 wait complete. 2024-03-21T10:35:43.720 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:35:43.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.937+0000 7f5270c00640 1 -- 172.21.15.31:0/679313069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f526c1020d0 msgr2=0x7f526c1044c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:43.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.937+0000 7f5270c00640 1 --2- 172.21.15.31:0/679313069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f526c1020d0 0x7f526c1044c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f525c009900 tx=0x7f525c02bc40 comp rx=0 tx=0).stop 2024-03-21T10:35:43.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.937+0000 7f5270c00640 1 -- 172.21.15.31:0/679313069 shutdown_connections 2024-03-21T10:35:43.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.937+0000 7f5270c00640 1 --2- 172.21.15.31:0/679313069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f526c107330 0x7f526c109720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:43.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.937+0000 7f5270c00640 1 --2- 172.21.15.31:0/679313069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f526c104a00 0x7f526c106df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:43.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.937+0000 7f5270c00640 1 --2- 172.21.15.31:0/679313069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f526c1020d0 0x7f526c1044c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:43.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.937+0000 7f5270c00640 1 -- 172.21.15.31:0/679313069 >> 172.21.15.31:0/679313069 conn(0x7f526c0fba20 msgr2=0x7f526c0fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:43.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.937+0000 7f5270c00640 1 -- 172.21.15.31:0/679313069 shutdown_connections 2024-03-21T10:35:43.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.937+0000 7f5270c00640 1 -- 172.21.15.31:0/679313069 wait complete. 2024-03-21T10:35:43.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.938+0000 7f5270c00640 1 Processor -- start 2024-03-21T10:35:43.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.938+0000 7f5270c00640 1 -- start start 2024-03-21T10:35:43.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.939+0000 7f5270c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f526c1020d0 0x7f526c1978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:43.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.939+0000 7f5270c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f526c104a00 0x7f526c197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:43.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.939+0000 7f5270c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f526c107330 0x7f526c19ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:43.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.939+0000 7f5270c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526c10d270 con 0x7f526c107330 2024-03-21T10:35:43.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.939+0000 7f5270c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526c10d0f0 con 0x7f526c104a00 2024-03-21T10:35:43.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.939+0000 7f5270c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526c10d3f0 con 0x7f526c1020d0 2024-03-21T10:35:43.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.939+0000 7f526b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f526c1020d0 0x7f526c1978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:43.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.939+0000 7f526b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f526c1020d0 0x7f526c1978e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53708/0 (socket says 172.21.15.31:53708) 2024-03-21T10:35:43.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.939+0000 7f526b400640 1 -- 172.21.15.31:0/1096957372 learned_addr learned my addr 172.21.15.31:0/1096957372 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:43.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.939+0000 7f526be00640 1 --2- 172.21.15.31:0/1096957372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f526c107330 0x7f526c19ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:43.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.939+0000 7f5263e00640 1 --2- 172.21.15.31:0/1096957372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f526c104a00 0x7f526c197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:43.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.939+0000 7f526b400640 1 -- 172.21.15.31:0/1096957372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f526c104a00 msgr2=0x7f526c197e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:43.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.939+0000 7f526b400640 1 --2- 172.21.15.31:0/1096957372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f526c104a00 0x7f526c197e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:43.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.939+0000 7f526b400640 1 -- 172.21.15.31:0/1096957372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f526c107330 msgr2=0x7f526c19ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:43.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.939+0000 7f526b400640 1 --2- 172.21.15.31:0/1096957372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f526c107330 0x7f526c19ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:43.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.939+0000 7f526b400640 1 -- 172.21.15.31:0/1096957372 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f526c19f500 con 0x7f526c1020d0 2024-03-21T10:35:43.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.940+0000 7f526b400640 1 --2- 172.21.15.31:0/1096957372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f526c1020d0 0x7f526c1978e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f525c02c8f0 tx=0x7f525c005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:43.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.940+0000 7f5268c00640 1 -- 172.21.15.31:0/1096957372 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525c041070 con 0x7f526c1020d0 2024-03-21T10:35:43.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.940+0000 7f5268c00640 1 -- 172.21.15.31:0/1096957372 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f525c0044d0 con 0x7f526c1020d0 2024-03-21T10:35:43.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.940+0000 7f5268c00640 1 -- 172.21.15.31:0/1096957372 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525c034440 con 0x7f526c1020d0 2024-03-21T10:35:43.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.940+0000 7f5270c00640 1 -- 172.21.15.31:0/1096957372 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f526c19f790 con 0x7f526c1020d0 2024-03-21T10:35:43.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.940+0000 7f5270c00640 1 -- 172.21.15.31:0/1096957372 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f526c19fc30 con 0x7f526c1020d0 2024-03-21T10:35:43.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.941+0000 7f5270c00640 1 -- 172.21.15.31:0/1096957372 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f526c0fd510 con 0x7f526c1020d0 2024-03-21T10:35:43.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.943+0000 7f5268c00640 1 -- 172.21.15.31:0/1096957372 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7f525c044070 con 0x7f526c1020d0 2024-03-21T10:35:43.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.945+0000 7f5268c00640 1 --2- 172.21.15.31:0/1096957372 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f52500a2fa0 0x7f52500a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:43.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.945+0000 7f5268c00640 1 -- 172.21.15.31:0/1096957372 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f525c0f1290 con 0x7f526c1020d0 2024-03-21T10:35:43.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.945+0000 7f5263e00640 1 --2- 172.21.15.31:0/1096957372 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f52500a2fa0 0x7f52500a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:43.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.946+0000 7f5263e00640 1 --2- 172.21.15.31:0/1096957372 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f52500a2fa0 0x7f52500a5460 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f526c198df0 tx=0x7f5254005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:43.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:43.948+0000 7f5268c00640 1 -- 172.21.15.31:0/1096957372 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f525c0bd780 con 0x7f526c1020d0 2024-03-21T10:35:44.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.196+0000 7f5270c00640 1 -- 172.21.15.31:0/1096957372 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f526c02d240 con 0x7f526c1020d0 2024-03-21T10:35:44.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.197+0000 7f5268c00640 1 -- 172.21.15.31:0/1096957372 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11384 (secure 0 0 0) 0x7f525c0c1820 con 0x7f526c1020d0 2024-03-21T10:35:44.196 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:35:44.196 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:35:40.990205+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24596,24547]},"id":1}]} 2024-03-21T10:35:44.196 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2024-03-21T10:35:44.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.200+0000 7f5270c00640 1 -- 172.21.15.31:0/1096957372 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f52500a2fa0 msgr2=0x7f52500a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:44.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.200+0000 7f5270c00640 1 --2- 172.21.15.31:0/1096957372 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f52500a2fa0 0x7f52500a5460 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f526c198df0 tx=0x7f5254005f90 comp rx=0 tx=0).stop 2024-03-21T10:35:44.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.200+0000 7f5270c00640 1 -- 172.21.15.31:0/1096957372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f526c1020d0 msgr2=0x7f526c1978e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:44.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.200+0000 7f5270c00640 1 --2- 172.21.15.31:0/1096957372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f526c1020d0 0x7f526c1978e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f525c02c8f0 tx=0x7f525c005d50 comp rx=0 tx=0).stop 2024-03-21T10:35:44.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.200+0000 7f5270c00640 1 -- 172.21.15.31:0/1096957372 shutdown_connections 2024-03-21T10:35:44.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.200+0000 7f5270c00640 1 --2- 172.21.15.31:0/1096957372 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f52500a2fa0 0x7f52500a5460 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:44.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.200+0000 7f5270c00640 1 --2- 172.21.15.31:0/1096957372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f526c107330 0x7f526c19ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:44.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.200+0000 7f5270c00640 1 --2- 172.21.15.31:0/1096957372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f526c104a00 0x7f526c197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:44.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.200+0000 7f5270c00640 1 --2- 172.21.15.31:0/1096957372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f526c1020d0 0x7f526c1978e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:44.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.200+0000 7f5270c00640 1 -- 172.21.15.31:0/1096957372 >> 172.21.15.31:0/1096957372 conn(0x7f526c0fba20 msgr2=0x7f526c102bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:44.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.200+0000 7f5270c00640 1 -- 172.21.15.31:0/1096957372 shutdown_connections 2024-03-21T10:35:44.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.200+0000 7f5270c00640 1 -- 172.21.15.31:0/1096957372 wait complete. 2024-03-21T10:35:44.222 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T10:35:44.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.479+0000 7f6ae3200640 1 -- 172.21.15.31:0/2496746452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ae41037c0 msgr2=0x7f6ae4103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:44.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.479+0000 7f6ae3200640 1 --2- 172.21.15.31:0/2496746452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ae41037c0 0x7f6ae4103be0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4009900 tx=0x7f6ad402bc50 comp rx=0 tx=0).stop 2024-03-21T10:35:44.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.480+0000 7f6ae3200640 1 -- 172.21.15.31:0/2496746452 shutdown_connections 2024-03-21T10:35:44.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.480+0000 7f6ae3200640 1 --2- 172.21.15.31:0/2496746452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ae4105370 0x7f6ae410bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:44.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.480+0000 7f6ae3200640 1 --2- 172.21.15.31:0/2496746452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ae41049b0 0x7f6ae4104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:44.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.480+0000 7f6ae3200640 1 --2- 172.21.15.31:0/2496746452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ae41037c0 0x7f6ae4103be0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:44.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.480+0000 7f6ae3200640 1 -- 172.21.15.31:0/2496746452 >> 172.21.15.31:0/2496746452 conn(0x7f6ae40fed50 msgr2=0x7f6ae4101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:44.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.480+0000 7f6ae3200640 1 -- 172.21.15.31:0/2496746452 shutdown_connections 2024-03-21T10:35:44.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.480+0000 7f6ae3200640 1 -- 172.21.15.31:0/2496746452 wait complete. 2024-03-21T10:35:44.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.481+0000 7f6ae3200640 1 Processor -- start 2024-03-21T10:35:44.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.481+0000 7f6ae3200640 1 -- start start 2024-03-21T10:35:44.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.481+0000 7f6ae3200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ae41037c0 0x7f6ae4199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:44.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.481+0000 7f6ae3200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ae41049b0 0x7f6ae4199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:44.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.481+0000 7f6ae3200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ae4105370 0x7f6ae41a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:44.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.481+0000 7f6ae3200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae410f460 con 0x7f6ae41037c0 2024-03-21T10:35:44.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.481+0000 7f6ae3200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae410f2e0 con 0x7f6ae4105370 2024-03-21T10:35:44.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.481+0000 7f6ae3200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae410f5e0 con 0x7f6ae41049b0 2024-03-21T10:35:44.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.482+0000 7f6ae1e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ae41037c0 0x7f6ae4199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:44.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.482+0000 7f6ae1400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ae41049b0 0x7f6ae4199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:44.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.482+0000 7f6ae1e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ae41037c0 0x7f6ae4199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41182/0 (socket says 172.21.15.31:41182) 2024-03-21T10:35:44.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.482+0000 7f6ae1400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ae41049b0 0x7f6ae4199fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53712/0 (socket says 172.21.15.31:53712) 2024-03-21T10:35:44.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.482+0000 7f6ae1e00640 1 -- 172.21.15.31:0/3799186618 learned_addr learned my addr 172.21.15.31:0/3799186618 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:44.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.482+0000 7f6ae2800640 1 --2- 172.21.15.31:0/3799186618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ae4105370 0x7f6ae41a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:44.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.482+0000 7f6ae1400640 1 -- 172.21.15.31:0/3799186618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ae4105370 msgr2=0x7f6ae41a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:44.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.482+0000 7f6ae1400640 1 --2- 172.21.15.31:0/3799186618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ae4105370 0x7f6ae41a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:44.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.482+0000 7f6ae1400640 1 -- 172.21.15.31:0/3799186618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ae41037c0 msgr2=0x7f6ae4199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:44.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.482+0000 7f6ae1400640 1 --2- 172.21.15.31:0/3799186618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ae41037c0 0x7f6ae4199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:44.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.482+0000 7f6ae1400640 1 -- 172.21.15.31:0/3799186618 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae41a16a0 con 0x7f6ae41049b0 2024-03-21T10:35:44.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.482+0000 7f6ae2800640 1 --2- 172.21.15.31:0/3799186618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ae4105370 0x7f6ae41a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:35:44.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.482+0000 7f6ae1e00640 1 --2- 172.21.15.31:0/3799186618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ae41037c0 0x7f6ae4199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:35:44.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.482+0000 7f6ae1400640 1 --2- 172.21.15.31:0/3799186618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ae41049b0 0x7f6ae4199fc0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0005fd0 tx=0x7f6ad000dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:44.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.483+0000 7f6adaa00640 1 -- 172.21.15.31:0/3799186618 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad0009c30 con 0x7f6ae41049b0 2024-03-21T10:35:44.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.483+0000 7f6ae3200640 1 -- 172.21.15.31:0/3799186618 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ae41a1990 con 0x7f6ae41049b0 2024-03-21T10:35:44.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.483+0000 7f6adaa00640 1 -- 172.21.15.31:0/3799186618 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6ad0009dd0 con 0x7f6ae41049b0 2024-03-21T10:35:44.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.483+0000 7f6adaa00640 1 -- 172.21.15.31:0/3799186618 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad0010460 con 0x7f6ae41049b0 2024-03-21T10:35:44.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.483+0000 7f6ae3200640 1 -- 172.21.15.31:0/3799186618 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ae41a1ee0 con 0x7f6ae41049b0 2024-03-21T10:35:44.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.484+0000 7f6ae3200640 1 -- 172.21.15.31:0/3799186618 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ae410aae0 con 0x7f6ae41049b0 2024-03-21T10:35:44.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.488+0000 7f6adaa00640 1 -- 172.21.15.31:0/3799186618 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7f6ad0003ef0 con 0x7f6ae41049b0 2024-03-21T10:35:44.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.489+0000 7f6adaa00640 1 --2- 172.21.15.31:0/3799186618 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f6ac80a2fa0 0x7f6ac80a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:44.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.489+0000 7f6adaa00640 1 -- 172.21.15.31:0/3799186618 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f6ad00cf440 con 0x7f6ae41049b0 2024-03-21T10:35:44.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.489+0000 7f6ae1e00640 1 --2- 172.21.15.31:0/3799186618 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f6ac80a2fa0 0x7f6ac80a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:44.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.489+0000 7f6ae1e00640 1 --2- 172.21.15.31:0/3799186618 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f6ac80a2fa0 0x7f6ac80a5460 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6ad402c900 tx=0x7f6ad4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:44.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.493+0000 7f6adaa00640 1 -- 172.21.15.31:0/3799186618 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6ad0017020 con 0x7f6ae41049b0 2024-03-21T10:35:44.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:44 smithi031 ceph-mon[31673]: from='client.24847 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:35:44.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:44 smithi031 ceph-mon[31673]: from='client.24853 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:35:44.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:44 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1096957372' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:35:44.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:44 smithi040 ceph-mon[36154]: from='client.24847 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:35:44.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:44 smithi040 ceph-mon[36154]: from='client.24853 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:35:44.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:44 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1096957372' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:35:44.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.705+0000 7f6ae3200640 1 -- 172.21.15.31:0/3799186618 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ae402d270 con 0x7f6ae41049b0 2024-03-21T10:35:44.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.707+0000 7f6adaa00640 1 -- 172.21.15.31:0/3799186618 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24009 (secure 0 0 0) 0x7f6ad009be70 con 0x7f6ae41049b0 2024-03-21T10:35:44.707 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:35:44.707 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","created":"2024-03-21T10:25:41.971117+0000","modified":"2024-03-21T10:35:11.657050+0000","last_up_change":"2024-03-21T10:34:07.757196+0000","last_in_change":"2024-03-21T10:33:47.460406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T10:29:44.553968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T10:35:04.945449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T10:35:05.970589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T10:35:09.688711+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e0244d33-00e2-47d5-9f66-34efc138c462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6803","nonce":3089885125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6805","nonce":3089885125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6809","nonce":3089885125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6807","nonce":3089885125}]},"public_addr":"172.21.15.31:6803/3089885125","cluster_addr":"172.21.15.31:6805/3089885125","heartbeat_back_addr":"172.21.15.31:6809/3089885125","heartbeat_front_addr":"172.21.15.31:6807/3089885125","state":["exists","up"]},{"osd":1,"uuid":"76a63504-beab-4354-a6c3-eebc3b16ad03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6801","nonce":618840037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6803","nonce":618840037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6807","nonce":618840037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6805","nonce":618840037}]},"public_addr":"172.21.15.40:6801/618840037","cluster_addr":"172.21.15.40:6803/618840037","heartbeat_back_addr":"172.21.15.40:6807/618840037","heartbeat_front_addr":"172.21.15.40:6805/618840037","state":["exists","up"]},{"osd":2,"uuid":"fa191bd6-9a21-4ee2-8adf-8a63c9cfd821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6801","nonce":2524184117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6803","nonce":2524184117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6807","nonce":2524184117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6805","nonce":2524184117}]},"public_addr":"172.21.15.70:6801/2524184117","cluster_addr":"172.21.15.70:6803/2524184117","heartbeat_back_addr":"172.21.15.70:6807/2524184117","heartbeat_front_addr":"172.21.15.70:6805/2524184117","state":["exists","up"]},{"osd":3,"uuid":"edb44507-ffb0-42a4-827b-08461e1bf952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6811","nonce":2823036296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6813","nonce":2823036296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6817","nonce":2823036296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6815","nonce":2823036296}]},"public_addr":"172.21.15.31:6811/2823036296","cluster_addr":"172.21.15.31:6813/2823036296","heartbeat_back_addr":"172.21.15.31:6817/2823036296","heartbeat_front_addr":"172.21.15.31:6815/2823036296","state":["exists","up"]},{"osd":4,"uuid":"8cb89153-8c3b-431f-b57c-8863fada76dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6809","nonce":4215186309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6811","nonce":4215186309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6815","nonce":4215186309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6813","nonce":4215186309}]},"public_addr":"172.21.15.40:6809/4215186309","cluster_addr":"172.21.15.40:6811/4215186309","heartbeat_back_addr":"172.21.15.40:6815/4215186309","heartbeat_front_addr":"172.21.15.40:6813/4215186309","state":["exists","up"]},{"osd":5,"uuid":"9409505a-d463-408c-a3cd-218380c04d68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6809","nonce":3402830452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6811","nonce":3402830452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6815","nonce":3402830452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6813","nonce":3402830452}]},"public_addr":"172.21.15.70:6809/3402830452","cluster_addr":"172.21.15.70:6811/3402830452","heartbeat_back_addr":"172.21.15.70:6815/3402830452","heartbeat_front_addr":"172.21.15.70:6813/3402830452","state":["exists","up"]},{"osd":6,"uuid":"f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6819","nonce":1193497058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6821","nonce":1193497058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6825","nonce":1193497058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6823","nonce":1193497058}]},"public_addr":"172.21.15.31:6819/1193497058","cluster_addr":"172.21.15.31:6821/1193497058","heartbeat_back_addr":"172.21.15.31:6825/1193497058","heartbeat_front_addr":"172.21.15.31:6823/1193497058","state":["exists","up"]},{"osd":7,"uuid":"db228a87-29a5-45d4-8fe6-a0a554262eb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6817","nonce":3825628119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6819","nonce":3825628119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6823","nonce":3825628119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6821","nonce":3825628119}]},"public_addr":"172.21.15.40:6817/3825628119","cluster_addr":"172.21.15.40:6819/3825628119","heartbeat_back_addr":"172.21.15.40:6823/3825628119","heartbeat_front_addr":"172.21.15.40:6821/3825628119","state":["exists","up"]},{"osd":8,"uuid":"59b9fde0-095d-4e0b-87da-ead37bd8ff1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6817","nonce":534880586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6819","nonce":534880586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6823","nonce":534880586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6821","nonce":534880586}]},"public_addr":"172.21.15.70:6817/534880586","cluster_addr":"172.21.15.70:6819/534880586","heartbeat_back_addr":"172.21.15.70:6823/534880586","heartbeat_front_addr":"172.21.15.70:6821/534880586","state":["exists","up"]},{"osd":9,"uuid":"77f47061-0c5f-4159-b276-e814935d6c1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6827","nonce":587047469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6829","nonce":587047469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6833","nonce":587047469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6831","nonce":587047469}]},"public_addr":"172.21.15.31:6827/587047469","cluster_addr":"172.21.15.31:6829/587047469","heartbeat_back_addr":"172.21.15.31:6833/587047469","heartbeat_front_addr":"172.21.15.31:6831/587047469","state":["exists","up"]},{"osd":10,"uuid":"8853dc43-7bd1-4c93-8063-f7982fd7da38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6825","nonce":512108711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6827","nonce":512108711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6831","nonce":512108711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6829","nonce":512108711}]},"public_addr":"172.21.15.40:6825/512108711","cluster_addr":"172.21.15.40:6827/512108711","heartbeat_back_addr":"172.21.15.40:6831/512108711","heartbeat_front_addr":"172.21.15.40:6829/512108711","state":["exists","up"]},{"osd":11,"uuid":"84b7c2de-c424-45a6-9583-60097e372a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6825","nonce":540305718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6827","nonce":540305718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6831","nonce":540305718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6829","nonce":540305718}]},"public_addr":"172.21.15.70:6825/540305718","cluster_addr":"172.21.15.70:6827/540305718","heartbeat_back_addr":"172.21.15.70:6831/540305718","heartbeat_front_addr":"172.21.15.70:6829/540305718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:28:51.477358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:17.405045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:41.626268+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:15.339568+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:42.676996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:07.516999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:42.412273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:10.365041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:35.699120+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:11.733639+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:39.505442+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:34:05.449301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451930554":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/3342588843":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6801/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/1626448389":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/1910059351":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/344822770":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/3811368512":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/1981437651":"2024-03-22T10:26:16.959165+0000","172.21.15.31:6801/1981437651":"2024-03-22T10:26:16.959165+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T10:35:44.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.710+0000 7f6ae3200640 1 -- 172.21.15.31:0/3799186618 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f6ac80a2fa0 msgr2=0x7f6ac80a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:44.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.710+0000 7f6ae3200640 1 --2- 172.21.15.31:0/3799186618 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f6ac80a2fa0 0x7f6ac80a5460 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6ad402c900 tx=0x7f6ad4000bc0 comp rx=0 tx=0).stop 2024-03-21T10:35:44.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.710+0000 7f6ae3200640 1 -- 172.21.15.31:0/3799186618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ae41049b0 msgr2=0x7f6ae4199fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:44.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.710+0000 7f6ae3200640 1 --2- 172.21.15.31:0/3799186618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ae41049b0 0x7f6ae4199fc0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0005fd0 tx=0x7f6ad000dde0 comp rx=0 tx=0).stop 2024-03-21T10:35:44.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.710+0000 7f6ae3200640 1 -- 172.21.15.31:0/3799186618 shutdown_connections 2024-03-21T10:35:44.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.710+0000 7f6ae3200640 1 --2- 172.21.15.31:0/3799186618 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f6ac80a2fa0 0x7f6ac80a5460 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:44.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.710+0000 7f6ae3200640 1 --2- 172.21.15.31:0/3799186618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ae4105370 0x7f6ae41a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:44.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.710+0000 7f6ae3200640 1 --2- 172.21.15.31:0/3799186618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ae41049b0 0x7f6ae4199fc0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:44.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.710+0000 7f6ae3200640 1 --2- 172.21.15.31:0/3799186618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ae41037c0 0x7f6ae4199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:44.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.710+0000 7f6ae3200640 1 -- 172.21.15.31:0/3799186618 >> 172.21.15.31:0/3799186618 conn(0x7f6ae40fed50 msgr2=0x7f6ae4107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:44.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.710+0000 7f6ae3200640 1 -- 172.21.15.31:0/3799186618 shutdown_connections 2024-03-21T10:35:44.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.710+0000 7f6ae3200640 1 -- 172.21.15.31:0/3799186618 wait complete. 2024-03-21T10:35:44.733 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-21T10:35:44.733 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-21T10:35:44.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:44 smithi070 ceph-mon[36330]: from='client.24847 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:35:44.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:44 smithi070 ceph-mon[36330]: from='client.24853 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:35:44.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:44 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1096957372' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:35:44.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.956+0000 7fc88ca00640 1 -- 172.21.15.31:0/2815602311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc888102090 msgr2=0x7fc888104480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:44.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.956+0000 7fc88ca00640 1 --2- 172.21.15.31:0/2815602311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc888102090 0x7fc888104480 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc878009900 tx=0x7fc87802bc40 comp rx=0 tx=0).stop 2024-03-21T10:35:44.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.957+0000 7fc88ca00640 1 -- 172.21.15.31:0/2815602311 shutdown_connections 2024-03-21T10:35:44.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.957+0000 7fc88ca00640 1 --2- 172.21.15.31:0/2815602311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8881072f0 0x7fc8881096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:44.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.957+0000 7fc88ca00640 1 --2- 172.21.15.31:0/2815602311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8881049c0 0x7fc888106db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:44.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.957+0000 7fc88ca00640 1 --2- 172.21.15.31:0/2815602311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc888102090 0x7fc888104480 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:44.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.957+0000 7fc88ca00640 1 -- 172.21.15.31:0/2815602311 >> 172.21.15.31:0/2815602311 conn(0x7fc8880fba40 msgr2=0x7fc8880fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:44.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.957+0000 7fc88ca00640 1 -- 172.21.15.31:0/2815602311 shutdown_connections 2024-03-21T10:35:44.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.957+0000 7fc88ca00640 1 -- 172.21.15.31:0/2815602311 wait complete. 2024-03-21T10:35:44.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.958+0000 7fc88ca00640 1 Processor -- start 2024-03-21T10:35:44.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.958+0000 7fc88ca00640 1 -- start start 2024-03-21T10:35:44.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.959+0000 7fc88ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc888102090 0x7fc888199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:44.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.959+0000 7fc88ca00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8881049c0 0x7fc888199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:44.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.959+0000 7fc88ca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8881072f0 0x7fc8881a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:44.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.959+0000 7fc88ca00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88810d240 con 0x7fc8881072f0 2024-03-21T10:35:44.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.959+0000 7fc88ca00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88810d0c0 con 0x7fc888102090 2024-03-21T10:35:44.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.959+0000 7fc88ca00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88810d3c0 con 0x7fc8881049c0 2024-03-21T10:35:44.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.959+0000 7fc887400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc888102090 0x7fc888199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:44.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.959+0000 7fc887400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc888102090 0x7fc888199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44076/0 (socket says 172.21.15.31:44076) 2024-03-21T10:35:44.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.959+0000 7fc887400640 1 -- 172.21.15.31:0/2859143779 learned_addr learned my addr 172.21.15.31:0/2859143779 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:44.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.959+0000 7fc886a00640 1 --2- 172.21.15.31:0/2859143779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8881049c0 0x7fc888199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:44.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.959+0000 7fc887e00640 1 --2- 172.21.15.31:0/2859143779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8881072f0 0x7fc8881a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:44.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.960+0000 7fc887400640 1 -- 172.21.15.31:0/2859143779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8881049c0 msgr2=0x7fc888199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:44.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.960+0000 7fc887400640 1 --2- 172.21.15.31:0/2859143779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8881049c0 0x7fc888199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:44.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.960+0000 7fc887400640 1 -- 172.21.15.31:0/2859143779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8881072f0 msgr2=0x7fc8881a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:44.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.960+0000 7fc887400640 1 --2- 172.21.15.31:0/2859143779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8881072f0 0x7fc8881a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:44.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.960+0000 7fc887400640 1 -- 172.21.15.31:0/2859143779 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8881a16d0 con 0x7fc888102090 2024-03-21T10:35:44.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.960+0000 7fc887400640 1 --2- 172.21.15.31:0/2859143779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc888102090 0x7fc888199ab0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc878009230 tx=0x7fc878005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:44.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.960+0000 7fc87fe00640 1 -- 172.21.15.31:0/2859143779 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc878041070 con 0x7fc888102090 2024-03-21T10:35:44.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.960+0000 7fc88ca00640 1 -- 172.21.15.31:0/2859143779 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8881a1960 con 0x7fc888102090 2024-03-21T10:35:44.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.960+0000 7fc87fe00640 1 -- 172.21.15.31:0/2859143779 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc8780044d0 con 0x7fc888102090 2024-03-21T10:35:44.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.960+0000 7fc87fe00640 1 -- 172.21.15.31:0/2859143779 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc878034440 con 0x7fc888102090 2024-03-21T10:35:44.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.960+0000 7fc88ca00640 1 -- 172.21.15.31:0/2859143779 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8881a1e50 con 0x7fc888102090 2024-03-21T10:35:44.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.962+0000 7fc88ca00640 1 -- 172.21.15.31:0/2859143779 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc888108de0 con 0x7fc888102090 2024-03-21T10:35:44.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.965+0000 7fc87fe00640 1 -- 172.21.15.31:0/2859143779 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7fc878044070 con 0x7fc888102090 2024-03-21T10:35:44.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.967+0000 7fc87fe00640 1 --2- 172.21.15.31:0/2859143779 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc86c0a2fa0 0x7fc86c0a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:44.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.967+0000 7fc87fe00640 1 -- 172.21.15.31:0/2859143779 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc8780f14b0 con 0x7fc888102090 2024-03-21T10:35:44.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.967+0000 7fc886a00640 1 --2- 172.21.15.31:0/2859143779 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc86c0a2fa0 0x7fc86c0a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:44.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.968+0000 7fc886a00640 1 --2- 172.21.15.31:0/2859143779 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc86c0a2fa0 0x7fc86c0a5460 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc88819afc0 tx=0x7fc874005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:44.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:44.972+0000 7fc87fe00640 1 -- 172.21.15.31:0/2859143779 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc8780bdff0 con 0x7fc888102090 2024-03-21T10:35:45.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.173+0000 7fc88ca00640 1 -- 172.21.15.31:0/2859143779 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc888061ae0 con 0x7fc86c0a2fa0 2024-03-21T10:35:45.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.210+0000 7fc87fe00640 1 -- 172.21.15.31:0/2859143779 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340562 (secure 0 0 0) 0x7fc888061ae0 con 0x7fc86c0a2fa0 2024-03-21T10:35:45.214 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:35:45.216 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-03-21T10:35:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.216+0000 7fc88ca00640 1 -- 172.21.15.31:0/2859143779 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc86c0a2fa0 msgr2=0x7fc86c0a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.216+0000 7fc88ca00640 1 --2- 172.21.15.31:0/2859143779 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc86c0a2fa0 0x7fc86c0a5460 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc88819afc0 tx=0x7fc874005f90 comp rx=0 tx=0).stop 2024-03-21T10:35:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.216+0000 7fc88ca00640 1 -- 172.21.15.31:0/2859143779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc888102090 msgr2=0x7fc888199ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.216+0000 7fc88ca00640 1 --2- 172.21.15.31:0/2859143779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc888102090 0x7fc888199ab0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc878009230 tx=0x7fc878005d50 comp rx=0 tx=0).stop 2024-03-21T10:35:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.216+0000 7fc88ca00640 1 -- 172.21.15.31:0/2859143779 shutdown_connections 2024-03-21T10:35:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.216+0000 7fc88ca00640 1 --2- 172.21.15.31:0/2859143779 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc86c0a2fa0 0x7fc86c0a5460 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.216+0000 7fc88ca00640 1 --2- 172.21.15.31:0/2859143779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8881072f0 0x7fc8881a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.216+0000 7fc88ca00640 1 --2- 172.21.15.31:0/2859143779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8881049c0 0x7fc888199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.216+0000 7fc88ca00640 1 --2- 172.21.15.31:0/2859143779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc888102090 0x7fc888199ab0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.216+0000 7fc88ca00640 1 -- 172.21.15.31:0/2859143779 >> 172.21.15.31:0/2859143779 conn(0x7fc8880fba40 msgr2=0x7fc888101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.216+0000 7fc88ca00640 1 -- 172.21.15.31:0/2859143779 shutdown_connections 2024-03-21T10:35:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.216+0000 7fc88ca00640 1 -- 172.21.15.31:0/2859143779 wait complete. 2024-03-21T10:35:45.239 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-03-21T10:35:44.003900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595439,"num_objects":46,"num_object_clones":0,"num_object_copies":139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":141,"num_read_kb":124,"num_write":278,"num_write_kb":2975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":220,"ondisk_log_size":220,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":364928,"kb_used_data":11596,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576896,"statfs":{"total":1151940427776,"available":1151566741504,"internally_reserved":0,"allocated":11874304,"data_stored":7523879,"data_compressed":284274,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":431,"num_objects":22,"num_object_clones":0,"num_object_copies":67,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006428"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171420+0000","last_change":"2024-03-21T10:35:06.923544+0000","last_active":"2024-03-21T10:35:12.171420+0000","last_peered":"2024-03-21T10:35:12.171420+0000","last_clean":"2024-03-21T10:35:12.171420+0000","last_became_active":"2024-03-21T10:35:06.923254+0000","last_became_peered":"2024-03-21T10:35:06.923254+0000","last_unstale":"2024-03-21T10:35:12.171420+0000","last_undegraded":"2024-03-21T10:35:12.171420+0000","last_fullsized":"2024-03-21T10:35:12.171420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:38:14.573590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.183497+0000","last_change":"2024-03-21T10:35:06.929242+0000","last_active":"2024-03-21T10:35:12.183497+0000","last_peered":"2024-03-21T10:35:12.183497+0000","last_clean":"2024-03-21T10:35:12.183497+0000","last_became_active":"2024-03-21T10:35:06.927078+0000","last_became_peered":"2024-03-21T10:35:06.927078+0000","last_unstale":"2024-03-21T10:35:12.183497+0000","last_undegraded":"2024-03-21T10:35:12.183497+0000","last_fullsized":"2024-03-21T10:35:12.183497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:27:10.347945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.184632+0000","last_change":"2024-03-21T10:35:06.927792+0000","last_active":"2024-03-21T10:35:12.184632+0000","last_peered":"2024-03-21T10:35:12.184632+0000","last_clean":"2024-03-21T10:35:12.184632+0000","last_became_active":"2024-03-21T10:35:06.927386+0000","last_became_peered":"2024-03-21T10:35:06.927386+0000","last_unstale":"2024-03-21T10:35:12.184632+0000","last_undegraded":"2024-03-21T10:35:12.184632+0000","last_fullsized":"2024-03-21T10:35:12.184632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:53:51.310283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171374+0000","last_change":"2024-03-21T10:35:06.908901+0000","last_active":"2024-03-21T10:35:12.171374+0000","last_peered":"2024-03-21T10:35:12.171374+0000","last_clean":"2024-03-21T10:35:12.171374+0000","last_became_active":"2024-03-21T10:35:06.908586+0000","last_became_peered":"2024-03-21T10:35:06.908586+0000","last_unstale":"2024-03-21T10:35:12.171374+0000","last_undegraded":"2024-03-21T10:35:12.171374+0000","last_fullsized":"2024-03-21T10:35:12.171374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:25:57.387769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170261+0000","last_change":"2024-03-21T10:35:06.932537+0000","last_active":"2024-03-21T10:35:12.170261+0000","last_peered":"2024-03-21T10:35:12.170261+0000","last_clean":"2024-03-21T10:35:12.170261+0000","last_became_active":"2024-03-21T10:35:06.932185+0000","last_became_peered":"2024-03-21T10:35:06.932185+0000","last_unstale":"2024-03-21T10:35:12.170261+0000","last_undegraded":"2024-03-21T10:35:12.170261+0000","last_fullsized":"2024-03-21T10:35:12.170261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:30:25.127073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.166158+0000","last_change":"2024-03-21T10:35:06.922765+0000","last_active":"2024-03-21T10:35:12.166158+0000","last_peered":"2024-03-21T10:35:12.166158+0000","last_clean":"2024-03-21T10:35:12.166158+0000","last_became_active":"2024-03-21T10:35:06.922447+0000","last_became_peered":"2024-03-21T10:35:06.922447+0000","last_unstale":"2024-03-21T10:35:12.166158+0000","last_undegraded":"2024-03-21T10:35:12.166158+0000","last_fullsized":"2024-03-21T10:35:12.166158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:37:33.310735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170172+0000","last_change":"2024-03-21T10:35:06.896777+0000","last_active":"2024-03-21T10:35:12.170172+0000","last_peered":"2024-03-21T10:35:12.170172+0000","last_clean":"2024-03-21T10:35:12.170172+0000","last_became_active":"2024-03-21T10:35:06.896544+0000","last_became_peered":"2024-03-21T10:35:06.896544+0000","last_unstale":"2024-03-21T10:35:12.170172+0000","last_undegraded":"2024-03-21T10:35:12.170172+0000","last_fullsized":"2024-03-21T10:35:12.170172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:34:05.000661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.174554+0000","last_change":"2024-03-21T10:35:06.923294+0000","last_active":"2024-03-21T10:35:12.174554+0000","last_peered":"2024-03-21T10:35:12.174554+0000","last_clean":"2024-03-21T10:35:12.174554+0000","last_became_active":"2024-03-21T10:35:06.921320+0000","last_became_peered":"2024-03-21T10:35:06.921320+0000","last_unstale":"2024-03-21T10:35:12.174554+0000","last_undegraded":"2024-03-21T10:35:12.174554+0000","last_fullsized":"2024-03-21T10:35:12.174554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:33:53.855908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171265+0000","last_change":"2024-03-21T10:35:06.912977+0000","last_active":"2024-03-21T10:35:12.171265+0000","last_peered":"2024-03-21T10:35:12.171265+0000","last_clean":"2024-03-21T10:35:12.171265+0000","last_became_active":"2024-03-21T10:35:06.912626+0000","last_became_peered":"2024-03-21T10:35:06.912626+0000","last_unstale":"2024-03-21T10:35:12.171265+0000","last_undegraded":"2024-03-21T10:35:12.171265+0000","last_fullsized":"2024-03-21T10:35:12.171265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:42:13.472454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.166534+0000","last_change":"2024-03-21T10:35:06.910252+0000","last_active":"2024-03-21T10:35:12.166534+0000","last_peered":"2024-03-21T10:35:12.166534+0000","last_clean":"2024-03-21T10:35:12.166534+0000","last_became_active":"2024-03-21T10:35:06.909924+0000","last_became_peered":"2024-03-21T10:35:06.909924+0000","last_unstale":"2024-03-21T10:35:12.166534+0000","last_undegraded":"2024-03-21T10:35:12.166534+0000","last_fullsized":"2024-03-21T10:35:12.166534+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:43:15.527349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171192+0000","last_change":"2024-03-21T10:35:06.909569+0000","last_active":"2024-03-21T10:35:12.171192+0000","last_peered":"2024-03-21T10:35:12.171192+0000","last_clean":"2024-03-21T10:35:12.171192+0000","last_became_active":"2024-03-21T10:35:06.908982+0000","last_became_peered":"2024-03-21T10:35:06.908982+0000","last_unstale":"2024-03-21T10:35:12.171192+0000","last_undegraded":"2024-03-21T10:35:12.171192+0000","last_fullsized":"2024-03-21T10:35:12.171192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:13:56.020281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171353+0000","last_change":"2024-03-21T10:35:06.913861+0000","last_active":"2024-03-21T10:35:12.171353+0000","last_peered":"2024-03-21T10:35:12.171353+0000","last_clean":"2024-03-21T10:35:12.171353+0000","last_became_active":"2024-03-21T10:35:06.913490+0000","last_became_peered":"2024-03-21T10:35:06.913490+0000","last_unstale":"2024-03-21T10:35:12.171353+0000","last_undegraded":"2024-03-21T10:35:12.171353+0000","last_fullsized":"2024-03-21T10:35:12.171353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:24:18.573735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.180710+0000","last_change":"2024-03-21T10:35:06.934537+0000","last_active":"2024-03-21T10:35:12.180710+0000","last_peered":"2024-03-21T10:35:12.180710+0000","last_clean":"2024-03-21T10:35:12.180710+0000","last_became_active":"2024-03-21T10:35:06.932951+0000","last_became_peered":"2024-03-21T10:35:06.932951+0000","last_unstale":"2024-03-21T10:35:12.180710+0000","last_undegraded":"2024-03-21T10:35:12.180710+0000","last_fullsized":"2024-03-21T10:35:12.180710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:43:17.748434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.176882+0000","last_change":"2024-03-21T10:35:06.925495+0000","last_active":"2024-03-21T10:35:12.176882+0000","last_peered":"2024-03-21T10:35:12.176882+0000","last_clean":"2024-03-21T10:35:12.176882+0000","last_became_active":"2024-03-21T10:35:06.924326+0000","last_became_peered":"2024-03-21T10:35:06.924326+0000","last_unstale":"2024-03-21T10:35:12.176882+0000","last_undegraded":"2024-03-21T10:35:12.176882+0000","last_fullsized":"2024-03-21T10:35:12.176882+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:27:19.598213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.177431+0000","last_change":"2024-03-21T10:35:06.923695+0000","last_active":"2024-03-21T10:35:12.177431+0000","last_peered":"2024-03-21T10:35:12.177431+0000","last_clean":"2024-03-21T10:35:12.177431+0000","last_became_active":"2024-03-21T10:35:06.919114+0000","last_became_peered":"2024-03-21T10:35:06.919114+0000","last_unstale":"2024-03-21T10:35:12.177431+0000","last_undegraded":"2024-03-21T10:35:12.177431+0000","last_fullsized":"2024-03-21T10:35:12.177431+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:42:49.179653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.169783+0000","last_change":"2024-03-21T10:35:06.930225+0000","last_active":"2024-03-21T10:35:12.169783+0000","last_peered":"2024-03-21T10:35:12.169783+0000","last_clean":"2024-03-21T10:35:12.169783+0000","last_became_active":"2024-03-21T10:35:06.929905+0000","last_became_peered":"2024-03-21T10:35:06.929905+0000","last_unstale":"2024-03-21T10:35:12.169783+0000","last_undegraded":"2024-03-21T10:35:12.169783+0000","last_fullsized":"2024-03-21T10:35:12.169783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:21:07.231703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171177+0000","last_change":"2024-03-21T10:35:06.923498+0000","last_active":"2024-03-21T10:35:12.171177+0000","last_peered":"2024-03-21T10:35:12.171177+0000","last_clean":"2024-03-21T10:35:12.171177+0000","last_became_active":"2024-03-21T10:35:06.923242+0000","last_became_peered":"2024-03-21T10:35:06.923242+0000","last_unstale":"2024-03-21T10:35:12.171177+0000","last_undegraded":"2024-03-21T10:35:12.171177+0000","last_fullsized":"2024-03-21T10:35:12.171177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:42:04.309701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.173784+0000","last_change":"2024-03-21T10:35:06.918314+0000","last_active":"2024-03-21T10:35:12.173784+0000","last_peered":"2024-03-21T10:35:12.173784+0000","last_clean":"2024-03-21T10:35:12.173784+0000","last_became_active":"2024-03-21T10:35:06.917865+0000","last_became_peered":"2024-03-21T10:35:06.917865+0000","last_unstale":"2024-03-21T10:35:12.173784+0000","last_undegraded":"2024-03-21T10:35:12.173784+0000","last_fullsized":"2024-03-21T10:35:12.173784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:53:48.536341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.167081+0000","last_change":"2024-03-21T10:35:06.936234+0000","last_active":"2024-03-21T10:35:12.167081+0000","last_peered":"2024-03-21T10:35:12.167081+0000","last_clean":"2024-03-21T10:35:12.167081+0000","last_became_active":"2024-03-21T10:35:06.935958+0000","last_became_peered":"2024-03-21T10:35:06.935958+0000","last_unstale":"2024-03-21T10:35:12.167081+0000","last_undegraded":"2024-03-21T10:35:12.167081+0000","last_fullsized":"2024-03-21T10:35:12.167081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:52:43.509671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.166489+0000","last_change":"2024-03-21T10:35:11.198276+0000","last_active":"2024-03-21T10:35:12.166489+0000","last_peered":"2024-03-21T10:35:12.166489+0000","last_clean":"2024-03-21T10:35:12.166489+0000","last_became_active":"2024-03-21T10:35:11.197899+0000","last_became_peered":"2024-03-21T10:35:11.197899+0000","last_unstale":"2024-03-21T10:35:12.166489+0000","last_undegraded":"2024-03-21T10:35:12.166489+0000","last_fullsized":"2024-03-21T10:35:12.166489+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:17:30.499932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171995+0000","last_change":"2024-03-21T10:35:06.010647+0000","last_active":"2024-03-21T10:35:12.171995+0000","last_peered":"2024-03-21T10:35:12.171995+0000","last_clean":"2024-03-21T10:35:12.171995+0000","last_became_active":"2024-03-21T10:35:06.010239+0000","last_became_peered":"2024-03-21T10:35:06.010239+0000","last_unstale":"2024-03-21T10:35:12.171995+0000","last_undegraded":"2024-03-21T10:35:12.171995+0000","last_fullsized":"2024-03-21T10:35:12.171995+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:35:53.324044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.184015+0000","last_change":"2024-03-21T10:35:06.926398+0000","last_active":"2024-03-21T10:35:12.184015+0000","last_peered":"2024-03-21T10:35:12.184015+0000","last_clean":"2024-03-21T10:35:12.184015+0000","last_became_active":"2024-03-21T10:35:06.925904+0000","last_became_peered":"2024-03-21T10:35:06.925904+0000","last_unstale":"2024-03-21T10:35:12.184015+0000","last_undegraded":"2024-03-21T10:35:12.184015+0000","last_fullsized":"2024-03-21T10:35:12.184015+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:37:52.811604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171680+0000","last_change":"2024-03-21T10:35:11.195922+0000","last_active":"2024-03-21T10:35:12.171680+0000","last_peered":"2024-03-21T10:35:12.171680+0000","last_clean":"2024-03-21T10:35:12.171680+0000","last_became_active":"2024-03-21T10:35:11.195487+0000","last_became_peered":"2024-03-21T10:35:11.195487+0000","last_unstale":"2024-03-21T10:35:12.171680+0000","last_undegraded":"2024-03-21T10:35:12.171680+0000","last_fullsized":"2024-03-21T10:35:12.171680+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:34:16.346282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.184218+0000","last_change":"2024-03-21T10:35:06.022750+0000","last_active":"2024-03-21T10:35:12.184218+0000","last_peered":"2024-03-21T10:35:12.184218+0000","last_clean":"2024-03-21T10:35:12.184218+0000","last_became_active":"2024-03-21T10:35:06.022093+0000","last_became_peered":"2024-03-21T10:35:06.022093+0000","last_unstale":"2024-03-21T10:35:12.184218+0000","last_undegraded":"2024-03-21T10:35:12.184218+0000","last_fullsized":"2024-03-21T10:35:12.184218+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:53:28.458486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.182592+0000","last_change":"2024-03-21T10:35:06.927594+0000","last_active":"2024-03-21T10:35:12.182592+0000","last_peered":"2024-03-21T10:35:12.182592+0000","last_clean":"2024-03-21T10:35:12.182592+0000","last_became_active":"2024-03-21T10:35:06.926868+0000","last_became_peered":"2024-03-21T10:35:06.926868+0000","last_unstale":"2024-03-21T10:35:12.182592+0000","last_undegraded":"2024-03-21T10:35:12.182592+0000","last_fullsized":"2024-03-21T10:35:12.182592+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:07:33.127465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170428+0000","last_change":"2024-03-21T10:35:11.193475+0000","last_active":"2024-03-21T10:35:12.170428+0000","last_peered":"2024-03-21T10:35:12.170428+0000","last_clean":"2024-03-21T10:35:12.170428+0000","last_became_active":"2024-03-21T10:35:11.193054+0000","last_became_peered":"2024-03-21T10:35:11.193054+0000","last_unstale":"2024-03-21T10:35:12.170428+0000","last_undegraded":"2024-03-21T10:35:12.170428+0000","last_fullsized":"2024-03-21T10:35:12.170428+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:32:36.564147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:40.328623+0000","last_change":"2024-03-21T10:35:06.004257+0000","last_active":"2024-03-21T10:35:40.328623+0000","last_peered":"2024-03-21T10:35:40.328623+0000","last_clean":"2024-03-21T10:35:40.328623+0000","last_became_active":"2024-03-21T10:35:06.003936+0000","last_became_peered":"2024-03-21T10:35:06.003936+0000","last_unstale":"2024-03-21T10:35:40.328623+0000","last_undegraded":"2024-03-21T10:35:40.328623+0000","last_fullsized":"2024-03-21T10:35:40.328623+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:48:48.608979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170958+0000","last_change":"2024-03-21T10:35:06.941681+0000","last_active":"2024-03-21T10:35:12.170958+0000","last_peered":"2024-03-21T10:35:12.170958+0000","last_clean":"2024-03-21T10:35:12.170958+0000","last_became_active":"2024-03-21T10:35:06.941314+0000","last_became_peered":"2024-03-21T10:35:06.941314+0000","last_unstale":"2024-03-21T10:35:12.170958+0000","last_undegraded":"2024-03-21T10:35:12.170958+0000","last_fullsized":"2024-03-21T10:35:12.170958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:46:09.948922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170860+0000","last_change":"2024-03-21T10:35:11.188472+0000","last_active":"2024-03-21T10:35:12.170860+0000","last_peered":"2024-03-21T10:35:12.170860+0000","last_clean":"2024-03-21T10:35:12.170860+0000","last_became_active":"2024-03-21T10:35:11.188072+0000","last_became_peered":"2024-03-21T10:35:11.188072+0000","last_unstale":"2024-03-21T10:35:12.170860+0000","last_undegraded":"2024-03-21T10:35:12.170860+0000","last_fullsized":"2024-03-21T10:35:12.170860+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:04:16.911334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170556+0000","last_change":"2024-03-21T10:35:05.998493+0000","last_active":"2024-03-21T10:35:12.170556+0000","last_peered":"2024-03-21T10:35:12.170556+0000","last_clean":"2024-03-21T10:35:12.170556+0000","last_became_active":"2024-03-21T10:35:05.998134+0000","last_became_peered":"2024-03-21T10:35:05.998134+0000","last_unstale":"2024-03-21T10:35:12.170556+0000","last_undegraded":"2024-03-21T10:35:12.170556+0000","last_fullsized":"2024-03-21T10:35:12.170556+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:05:32.033328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171417+0000","last_change":"2024-03-21T10:35:06.912973+0000","last_active":"2024-03-21T10:35:12.171417+0000","last_peered":"2024-03-21T10:35:12.171417+0000","last_clean":"2024-03-21T10:35:12.171417+0000","last_became_active":"2024-03-21T10:35:06.912382+0000","last_became_peered":"2024-03-21T10:35:06.912382+0000","last_unstale":"2024-03-21T10:35:12.171417+0000","last_undegraded":"2024-03-21T10:35:12.171417+0000","last_fullsized":"2024-03-21T10:35:12.171417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:13:23.757805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.166991+0000","last_change":"2024-03-21T10:35:11.180806+0000","last_active":"2024-03-21T10:35:12.166991+0000","last_peered":"2024-03-21T10:35:12.166991+0000","last_clean":"2024-03-21T10:35:12.166991+0000","last_became_active":"2024-03-21T10:35:11.180303+0000","last_became_peered":"2024-03-21T10:35:11.180303+0000","last_unstale":"2024-03-21T10:35:12.166991+0000","last_undegraded":"2024-03-21T10:35:12.166991+0000","last_fullsized":"2024-03-21T10:35:12.166991+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:27:11.719654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170854+0000","last_change":"2024-03-21T10:35:06.004555+0000","last_active":"2024-03-21T10:35:12.170854+0000","last_peered":"2024-03-21T10:35:12.170854+0000","last_clean":"2024-03-21T10:35:12.170854+0000","last_became_active":"2024-03-21T10:35:06.003926+0000","last_became_peered":"2024-03-21T10:35:06.003926+0000","last_unstale":"2024-03-21T10:35:12.170854+0000","last_undegraded":"2024-03-21T10:35:12.170854+0000","last_fullsized":"2024-03-21T10:35:12.170854+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:05:49.721000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170924+0000","last_change":"2024-03-21T10:35:06.913556+0000","last_active":"2024-03-21T10:35:12.170924+0000","last_peered":"2024-03-21T10:35:12.170924+0000","last_clean":"2024-03-21T10:35:12.170924+0000","last_became_active":"2024-03-21T10:35:06.913213+0000","last_became_peered":"2024-03-21T10:35:06.913213+0000","last_unstale":"2024-03-21T10:35:12.170924+0000","last_undegraded":"2024-03-21T10:35:12.170924+0000","last_fullsized":"2024-03-21T10:35:12.170924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:11:09.235927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.166899+0000","last_change":"2024-03-21T10:35:11.195036+0000","last_active":"2024-03-21T10:35:12.166899+0000","last_peered":"2024-03-21T10:35:12.166899+0000","last_clean":"2024-03-21T10:35:12.166899+0000","last_became_active":"2024-03-21T10:35:11.194670+0000","last_became_peered":"2024-03-21T10:35:11.194670+0000","last_unstale":"2024-03-21T10:35:12.166899+0000","last_undegraded":"2024-03-21T10:35:12.166899+0000","last_fullsized":"2024-03-21T10:35:12.166899+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:37:02.955167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:40.335649+0000","last_change":"2024-03-21T10:35:06.012527+0000","last_active":"2024-03-21T10:35:40.335649+0000","last_peered":"2024-03-21T10:35:40.335649+0000","last_clean":"2024-03-21T10:35:40.335649+0000","last_became_active":"2024-03-21T10:35:06.012228+0000","last_became_peered":"2024-03-21T10:35:06.012228+0000","last_unstale":"2024-03-21T10:35:40.335649+0000","last_undegraded":"2024-03-21T10:35:40.335649+0000","last_fullsized":"2024-03-21T10:35:40.335649+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:52:34.622306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170989+0000","last_change":"2024-03-21T10:35:06.925810+0000","last_active":"2024-03-21T10:35:12.170989+0000","last_peered":"2024-03-21T10:35:12.170989+0000","last_clean":"2024-03-21T10:35:12.170989+0000","last_became_active":"2024-03-21T10:35:06.925540+0000","last_became_peered":"2024-03-21T10:35:06.925540+0000","last_unstale":"2024-03-21T10:35:12.170989+0000","last_undegraded":"2024-03-21T10:35:12.170989+0000","last_fullsized":"2024-03-21T10:35:12.170989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:26:10.949653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171101+0000","last_change":"2024-03-21T10:35:11.196543+0000","last_active":"2024-03-21T10:35:12.171101+0000","last_peered":"2024-03-21T10:35:12.171101+0000","last_clean":"2024-03-21T10:35:12.171101+0000","last_became_active":"2024-03-21T10:35:11.196151+0000","last_became_peered":"2024-03-21T10:35:11.196151+0000","last_unstale":"2024-03-21T10:35:12.171101+0000","last_undegraded":"2024-03-21T10:35:12.171101+0000","last_fullsized":"2024-03-21T10:35:12.171101+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:15:39.193647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.180661+0000","last_change":"2024-03-21T10:35:06.004588+0000","last_active":"2024-03-21T10:35:12.180661+0000","last_peered":"2024-03-21T10:35:12.180661+0000","last_clean":"2024-03-21T10:35:12.180661+0000","last_became_active":"2024-03-21T10:35:06.004190+0000","last_became_peered":"2024-03-21T10:35:06.004190+0000","last_unstale":"2024-03-21T10:35:12.180661+0000","last_undegraded":"2024-03-21T10:35:12.180661+0000","last_fullsized":"2024-03-21T10:35:12.180661+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:32:18.563692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.175870+0000","last_change":"2024-03-21T10:35:06.941738+0000","last_active":"2024-03-21T10:35:12.175870+0000","last_peered":"2024-03-21T10:35:12.175870+0000","last_clean":"2024-03-21T10:35:12.175870+0000","last_became_active":"2024-03-21T10:35:06.941386+0000","last_became_peered":"2024-03-21T10:35:06.941386+0000","last_unstale":"2024-03-21T10:35:12.175870+0000","last_undegraded":"2024-03-21T10:35:12.175870+0000","last_fullsized":"2024-03-21T10:35:12.175870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:52:55.519095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.184480+0000","last_change":"2024-03-21T10:35:11.199370+0000","last_active":"2024-03-21T10:35:12.184480+0000","last_peered":"2024-03-21T10:35:12.184480+0000","last_clean":"2024-03-21T10:35:12.184480+0000","last_became_active":"2024-03-21T10:35:11.198598+0000","last_became_peered":"2024-03-21T10:35:11.198598+0000","last_unstale":"2024-03-21T10:35:12.184480+0000","last_undegraded":"2024-03-21T10:35:12.184480+0000","last_fullsized":"2024-03-21T10:35:12.184480+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:07:19.419405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171940+0000","last_change":"2024-03-21T10:35:06.004893+0000","last_active":"2024-03-21T10:35:12.171940+0000","last_peered":"2024-03-21T10:35:12.171940+0000","last_clean":"2024-03-21T10:35:12.171940+0000","last_became_active":"2024-03-21T10:35:06.004586+0000","last_became_peered":"2024-03-21T10:35:06.004586+0000","last_unstale":"2024-03-21T10:35:12.171940+0000","last_undegraded":"2024-03-21T10:35:12.171940+0000","last_fullsized":"2024-03-21T10:35:12.171940+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:05:29.106575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.169378+0000","last_change":"2024-03-21T10:35:06.924448+0000","last_active":"2024-03-21T10:35:12.169378+0000","last_peered":"2024-03-21T10:35:12.169378+0000","last_clean":"2024-03-21T10:35:12.169378+0000","last_became_active":"2024-03-21T10:35:06.924049+0000","last_became_peered":"2024-03-21T10:35:06.924049+0000","last_unstale":"2024-03-21T10:35:12.169378+0000","last_undegraded":"2024-03-21T10:35:12.169378+0000","last_fullsized":"2024-03-21T10:35:12.169378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:23:12.070723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.176865+0000","last_change":"2024-03-21T10:35:11.188691+0000","last_active":"2024-03-21T10:35:12.176865+0000","last_peered":"2024-03-21T10:35:12.176865+0000","last_clean":"2024-03-21T10:35:12.176865+0000","last_became_active":"2024-03-21T10:35:11.188138+0000","last_became_peered":"2024-03-21T10:35:11.188138+0000","last_unstale":"2024-03-21T10:35:12.176865+0000","last_undegraded":"2024-03-21T10:35:12.176865+0000","last_fullsized":"2024-03-21T10:35:12.176865+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:51:50.920920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:39.325104+0000","last_change":"2024-03-21T10:35:06.015276+0000","last_active":"2024-03-21T10:35:39.325104+0000","last_peered":"2024-03-21T10:35:39.325104+0000","last_clean":"2024-03-21T10:35:39.325104+0000","last_became_active":"2024-03-21T10:35:06.014734+0000","last_became_peered":"2024-03-21T10:35:06.014734+0000","last_unstale":"2024-03-21T10:35:39.325104+0000","last_undegraded":"2024-03-21T10:35:39.325104+0000","last_fullsized":"2024-03-21T10:35:39.325104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:18:45.905451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170963+0000","last_change":"2024-03-21T10:35:06.926689+0000","last_active":"2024-03-21T10:35:12.170963+0000","last_peered":"2024-03-21T10:35:12.170963+0000","last_clean":"2024-03-21T10:35:12.170963+0000","last_became_active":"2024-03-21T10:35:06.926319+0000","last_became_peered":"2024-03-21T10:35:06.926319+0000","last_unstale":"2024-03-21T10:35:12.170963+0000","last_undegraded":"2024-03-21T10:35:12.170963+0000","last_fullsized":"2024-03-21T10:35:12.170963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:06:22.371608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.169159+0000","last_change":"2024-03-21T10:35:11.203528+0000","last_active":"2024-03-21T10:35:12.169159+0000","last_peered":"2024-03-21T10:35:12.169159+0000","last_clean":"2024-03-21T10:35:12.169159+0000","last_became_active":"2024-03-21T10:35:11.202174+0000","last_became_peered":"2024-03-21T10:35:11.202174+0000","last_unstale":"2024-03-21T10:35:12.169159+0000","last_undegraded":"2024-03-21T10:35:12.169159+0000","last_fullsized":"2024-03-21T10:35:12.169159+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:21:07.038401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:39.324186+0000","last_change":"2024-03-21T10:35:06.007231+0000","last_active":"2024-03-21T10:35:39.324186+0000","last_peered":"2024-03-21T10:35:39.324186+0000","last_clean":"2024-03-21T10:35:39.324186+0000","last_became_active":"2024-03-21T10:35:06.006875+0000","last_became_peered":"2024-03-21T10:35:06.006875+0000","last_unstale":"2024-03-21T10:35:39.324186+0000","last_undegraded":"2024-03-21T10:35:39.324186+0000","last_fullsized":"2024-03-21T10:35:39.324186+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:08:02.489741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.183859+0000","last_change":"2024-03-21T10:35:06.925940+0000","last_active":"2024-03-21T10:35:12.183859+0000","last_peered":"2024-03-21T10:35:12.183859+0000","last_clean":"2024-03-21T10:35:12.183859+0000","last_became_active":"2024-03-21T10:35:06.924515+0000","last_became_peered":"2024-03-21T10:35:06.924515+0000","last_unstale":"2024-03-21T10:35:12.183859+0000","last_undegraded":"2024-03-21T10:35:12.183859+0000","last_fullsized":"2024-03-21T10:35:12.183859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:19:20.527047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170638+0000","last_change":"2024-03-21T10:35:11.193382+0000","last_active":"2024-03-21T10:35:12.170638+0000","last_peered":"2024-03-21T10:35:12.170638+0000","last_clean":"2024-03-21T10:35:12.170638+0000","last_became_active":"2024-03-21T10:35:11.192959+0000","last_became_peered":"2024-03-21T10:35:11.192959+0000","last_unstale":"2024-03-21T10:35:12.170638+0000","last_undegraded":"2024-03-21T10:35:12.170638+0000","last_fullsized":"2024-03-21T10:35:12.170638+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:02:57.187370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171296+0000","last_change":"2024-03-21T10:35:06.019969+0000","last_active":"2024-03-21T10:35:12.171296+0000","last_peered":"2024-03-21T10:35:12.171296+0000","last_clean":"2024-03-21T10:35:12.171296+0000","last_became_active":"2024-03-21T10:35:06.019648+0000","last_became_peered":"2024-03-21T10:35:06.019648+0000","last_unstale":"2024-03-21T10:35:12.171296+0000","last_undegraded":"2024-03-21T10:35:12.171296+0000","last_fullsized":"2024-03-21T10:35:12.171296+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:24:02.227205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171469+0000","last_change":"2024-03-21T10:35:06.910294+0000","last_active":"2024-03-21T10:35:12.171469+0000","last_peered":"2024-03-21T10:35:12.171469+0000","last_clean":"2024-03-21T10:35:12.171469+0000","last_became_active":"2024-03-21T10:35:06.909862+0000","last_became_peered":"2024-03-21T10:35:06.909862+0000","last_unstale":"2024-03-21T10:35:12.171469+0000","last_undegraded":"2024-03-21T10:35:12.171469+0000","last_fullsized":"2024-03-21T10:35:12.171469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:37:13.396003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.169593+0000","last_change":"2024-03-21T10:35:11.196398+0000","last_active":"2024-03-21T10:35:12.169593+0000","last_peered":"2024-03-21T10:35:12.169593+0000","last_clean":"2024-03-21T10:35:12.169593+0000","last_became_active":"2024-03-21T10:35:11.194965+0000","last_became_peered":"2024-03-21T10:35:11.194965+0000","last_unstale":"2024-03-21T10:35:12.169593+0000","last_undegraded":"2024-03-21T10:35:12.169593+0000","last_fullsized":"2024-03-21T10:35:12.169593+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:48:22.024137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:39.323326+0000","last_change":"2024-03-21T10:35:06.005037+0000","last_active":"2024-03-21T10:35:39.323326+0000","last_peered":"2024-03-21T10:35:39.323326+0000","last_clean":"2024-03-21T10:35:39.323326+0000","last_became_active":"2024-03-21T10:35:06.004686+0000","last_became_peered":"2024-03-21T10:35:06.004686+0000","last_unstale":"2024-03-21T10:35:39.323326+0000","last_undegraded":"2024-03-21T10:35:39.323326+0000","last_fullsized":"2024-03-21T10:35:39.323326+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:24:13.312112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170837+0000","last_change":"2024-03-21T10:35:06.906525+0000","last_active":"2024-03-21T10:35:12.170837+0000","last_peered":"2024-03-21T10:35:12.170837+0000","last_clean":"2024-03-21T10:35:12.170837+0000","last_became_active":"2024-03-21T10:35:06.905150+0000","last_became_peered":"2024-03-21T10:35:06.905150+0000","last_unstale":"2024-03-21T10:35:12.170837+0000","last_undegraded":"2024-03-21T10:35:12.170837+0000","last_fullsized":"2024-03-21T10:35:12.170837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:46:27.958779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171957+0000","last_change":"2024-03-21T10:35:11.193519+0000","last_active":"2024-03-21T10:35:12.171957+0000","last_peered":"2024-03-21T10:35:12.171957+0000","last_clean":"2024-03-21T10:35:12.171957+0000","last_became_active":"2024-03-21T10:35:11.193043+0000","last_became_peered":"2024-03-21T10:35:11.193043+0000","last_unstale":"2024-03-21T10:35:12.171957+0000","last_undegraded":"2024-03-21T10:35:12.171957+0000","last_fullsized":"2024-03-21T10:35:12.171957+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:23:20.984472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:39.323912+0000","last_change":"2024-03-21T10:35:05.999220+0000","last_active":"2024-03-21T10:35:39.323912+0000","last_peered":"2024-03-21T10:35:39.323912+0000","last_clean":"2024-03-21T10:35:39.323912+0000","last_became_active":"2024-03-21T10:35:05.998918+0000","last_became_peered":"2024-03-21T10:35:05.998918+0000","last_unstale":"2024-03-21T10:35:39.323912+0000","last_undegraded":"2024-03-21T10:35:39.323912+0000","last_fullsized":"2024-03-21T10:35:39.323912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:01:09.816758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.181059+0000","last_change":"2024-03-21T10:35:06.941234+0000","last_active":"2024-03-21T10:35:12.181059+0000","last_peered":"2024-03-21T10:35:12.181059+0000","last_clean":"2024-03-21T10:35:12.181059+0000","last_became_active":"2024-03-21T10:35:06.940828+0000","last_became_peered":"2024-03-21T10:35:06.940828+0000","last_unstale":"2024-03-21T10:35:12.181059+0000","last_undegraded":"2024-03-21T10:35:12.181059+0000","last_fullsized":"2024-03-21T10:35:12.181059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:50:34.398822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171713+0000","last_change":"2024-03-21T10:35:11.196007+0000","last_active":"2024-03-21T10:35:12.171713+0000","last_peered":"2024-03-21T10:35:12.171713+0000","last_clean":"2024-03-21T10:35:12.171713+0000","last_became_active":"2024-03-21T10:35:11.195609+0000","last_became_peered":"2024-03-21T10:35:11.195609+0000","last_unstale":"2024-03-21T10:35:12.171713+0000","last_undegraded":"2024-03-21T10:35:12.171713+0000","last_fullsized":"2024-03-21T10:35:12.171713+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:21:38.803390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170877+0000","last_change":"2024-03-21T10:35:06.011594+0000","last_active":"2024-03-21T10:35:12.170877+0000","last_peered":"2024-03-21T10:35:12.170877+0000","last_clean":"2024-03-21T10:35:12.170877+0000","last_became_active":"2024-03-21T10:35:06.011254+0000","last_became_peered":"2024-03-21T10:35:06.011254+0000","last_unstale":"2024-03-21T10:35:12.170877+0000","last_undegraded":"2024-03-21T10:35:12.170877+0000","last_fullsized":"2024-03-21T10:35:12.170877+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:52:31.045659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170534+0000","last_change":"2024-03-21T10:35:06.929170+0000","last_active":"2024-03-21T10:35:12.170534+0000","last_peered":"2024-03-21T10:35:12.170534+0000","last_clean":"2024-03-21T10:35:12.170534+0000","last_became_active":"2024-03-21T10:35:06.928822+0000","last_became_peered":"2024-03-21T10:35:06.928822+0000","last_unstale":"2024-03-21T10:35:12.170534+0000","last_undegraded":"2024-03-21T10:35:12.170534+0000","last_fullsized":"2024-03-21T10:35:12.170534+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:37:40.328453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171663+0000","last_change":"2024-03-21T10:35:11.204285+0000","last_active":"2024-03-21T10:35:12.171663+0000","last_peered":"2024-03-21T10:35:12.171663+0000","last_clean":"2024-03-21T10:35:12.171663+0000","last_became_active":"2024-03-21T10:35:11.203698+0000","last_became_peered":"2024-03-21T10:35:11.203698+0000","last_unstale":"2024-03-21T10:35:12.171663+0000","last_undegraded":"2024-03-21T10:35:12.171663+0000","last_fullsized":"2024-03-21T10:35:12.171663+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:03:32.732488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170547+0000","last_change":"2024-03-21T10:35:06.005028+0000","last_active":"2024-03-21T10:35:12.170547+0000","last_peered":"2024-03-21T10:35:12.170547+0000","last_clean":"2024-03-21T10:35:12.170547+0000","last_became_active":"2024-03-21T10:35:06.004670+0000","last_became_peered":"2024-03-21T10:35:06.004670+0000","last_unstale":"2024-03-21T10:35:12.170547+0000","last_undegraded":"2024-03-21T10:35:12.170547+0000","last_fullsized":"2024-03-21T10:35:12.170547+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:00:18.481051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170455+0000","last_change":"2024-03-21T10:35:06.911696+0000","last_active":"2024-03-21T10:35:12.170455+0000","last_peered":"2024-03-21T10:35:12.170455+0000","last_clean":"2024-03-21T10:35:12.170455+0000","last_became_active":"2024-03-21T10:35:06.911231+0000","last_became_peered":"2024-03-21T10:35:06.911231+0000","last_unstale":"2024-03-21T10:35:12.170455+0000","last_undegraded":"2024-03-21T10:35:12.170455+0000","last_fullsized":"2024-03-21T10:35:12.170455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:21:51.540993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.181675+0000","last_change":"2024-03-21T10:35:11.204422+0000","last_active":"2024-03-21T10:35:12.181675+0000","last_peered":"2024-03-21T10:35:12.181675+0000","last_clean":"2024-03-21T10:35:12.181675+0000","last_became_active":"2024-03-21T10:35:11.203904+0000","last_became_peered":"2024-03-21T10:35:11.203904+0000","last_unstale":"2024-03-21T10:35:12.181675+0000","last_undegraded":"2024-03-21T10:35:12.181675+0000","last_fullsized":"2024-03-21T10:35:12.181675+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:38:14.486012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:40.336487+0000","last_change":"2024-03-21T10:35:06.019823+0000","last_active":"2024-03-21T10:35:40.336487+0000","last_peered":"2024-03-21T10:35:40.336487+0000","last_clean":"2024-03-21T10:35:40.336487+0000","last_became_active":"2024-03-21T10:35:06.019502+0000","last_became_peered":"2024-03-21T10:35:06.019502+0000","last_unstale":"2024-03-21T10:35:40.336487+0000","last_undegraded":"2024-03-21T10:35:40.336487+0000","last_fullsized":"2024-03-21T10:35:40.336487+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:33:22.438868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170828+0000","last_change":"2024-03-21T10:35:06.906671+0000","last_active":"2024-03-21T10:35:12.170828+0000","last_peered":"2024-03-21T10:35:12.170828+0000","last_clean":"2024-03-21T10:35:12.170828+0000","last_became_active":"2024-03-21T10:35:06.905873+0000","last_became_peered":"2024-03-21T10:35:06.905873+0000","last_unstale":"2024-03-21T10:35:12.170828+0000","last_undegraded":"2024-03-21T10:35:12.170828+0000","last_fullsized":"2024-03-21T10:35:12.170828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:14:59.304255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170758+0000","last_change":"2024-03-21T10:35:11.197698+0000","last_active":"2024-03-21T10:35:12.170758+0000","last_peered":"2024-03-21T10:35:12.170758+0000","last_clean":"2024-03-21T10:35:12.170758+0000","last_became_active":"2024-03-21T10:35:11.197295+0000","last_became_peered":"2024-03-21T10:35:11.197295+0000","last_unstale":"2024-03-21T10:35:12.170758+0000","last_undegraded":"2024-03-21T10:35:12.170758+0000","last_fullsized":"2024-03-21T10:35:12.170758+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:02:24.338317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:40.997992+0000","last_change":"2024-03-21T10:35:06.000951+0000","last_active":"2024-03-21T10:35:40.997992+0000","last_peered":"2024-03-21T10:35:40.997992+0000","last_clean":"2024-03-21T10:35:40.997992+0000","last_became_active":"2024-03-21T10:35:05.999169+0000","last_became_peered":"2024-03-21T10:35:05.999169+0000","last_unstale":"2024-03-21T10:35:40.997992+0000","last_undegraded":"2024-03-21T10:35:40.997992+0000","last_fullsized":"2024-03-21T10:35:40.997992+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:31:25.602621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170766+0000","last_change":"2024-03-21T10:35:06.908067+0000","last_active":"2024-03-21T10:35:12.170766+0000","last_peered":"2024-03-21T10:35:12.170766+0000","last_clean":"2024-03-21T10:35:12.170766+0000","last_became_active":"2024-03-21T10:35:06.907720+0000","last_became_peered":"2024-03-21T10:35:06.907720+0000","last_unstale":"2024-03-21T10:35:12.170766+0000","last_undegraded":"2024-03-21T10:35:12.170766+0000","last_fullsized":"2024-03-21T10:35:12.170766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:16:00.502637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170863+0000","last_change":"2024-03-21T10:35:11.229470+0000","last_active":"2024-03-21T10:35:12.170863+0000","last_peered":"2024-03-21T10:35:12.170863+0000","last_clean":"2024-03-21T10:35:12.170863+0000","last_became_active":"2024-03-21T10:35:11.228977+0000","last_became_peered":"2024-03-21T10:35:11.228977+0000","last_unstale":"2024-03-21T10:35:12.170863+0000","last_undegraded":"2024-03-21T10:35:12.170863+0000","last_fullsized":"2024-03-21T10:35:12.170863+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:05:52.269712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171362+0000","last_change":"2024-03-21T10:35:06.019866+0000","last_active":"2024-03-21T10:35:12.171362+0000","last_peered":"2024-03-21T10:35:12.171362+0000","last_clean":"2024-03-21T10:35:12.171362+0000","last_became_active":"2024-03-21T10:35:06.019550+0000","last_became_peered":"2024-03-21T10:35:06.019550+0000","last_unstale":"2024-03-21T10:35:12.171362+0000","last_undegraded":"2024-03-21T10:35:12.171362+0000","last_fullsized":"2024-03-21T10:35:12.171362+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:51:46.989647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.166384+0000","last_change":"2024-03-21T10:35:06.928051+0000","last_active":"2024-03-21T10:35:12.166384+0000","last_peered":"2024-03-21T10:35:12.166384+0000","last_clean":"2024-03-21T10:35:12.166384+0000","last_became_active":"2024-03-21T10:35:06.927707+0000","last_became_peered":"2024-03-21T10:35:06.927707+0000","last_unstale":"2024-03-21T10:35:12.166384+0000","last_undegraded":"2024-03-21T10:35:12.166384+0000","last_fullsized":"2024-03-21T10:35:12.166384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:01:39.502891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.182897+0000","last_change":"2024-03-21T10:35:11.199714+0000","last_active":"2024-03-21T10:35:12.182897+0000","last_peered":"2024-03-21T10:35:12.182897+0000","last_clean":"2024-03-21T10:35:12.182897+0000","last_became_active":"2024-03-21T10:35:11.199277+0000","last_became_peered":"2024-03-21T10:35:11.199277+0000","last_unstale":"2024-03-21T10:35:12.182897+0000","last_undegraded":"2024-03-21T10:35:12.182897+0000","last_fullsized":"2024-03-21T10:35:12.182897+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:58:01.471162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170722+0000","last_change":"2024-03-21T10:35:06.015090+0000","last_active":"2024-03-21T10:35:12.170722+0000","last_peered":"2024-03-21T10:35:12.170722+0000","last_clean":"2024-03-21T10:35:12.170722+0000","last_became_active":"2024-03-21T10:35:06.014666+0000","last_became_peered":"2024-03-21T10:35:06.014666+0000","last_unstale":"2024-03-21T10:35:12.170722+0000","last_undegraded":"2024-03-21T10:35:12.170722+0000","last_fullsized":"2024-03-21T10:35:12.170722+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:41:17.608069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.166085+0000","last_change":"2024-03-21T10:35:06.923729+0000","last_active":"2024-03-21T10:35:12.166085+0000","last_peered":"2024-03-21T10:35:12.166085+0000","last_clean":"2024-03-21T10:35:12.166085+0000","last_became_active":"2024-03-21T10:35:06.923361+0000","last_became_peered":"2024-03-21T10:35:06.923361+0000","last_unstale":"2024-03-21T10:35:12.166085+0000","last_undegraded":"2024-03-21T10:35:12.166085+0000","last_fullsized":"2024-03-21T10:35:12.166085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:20:40.819207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.183026+0000","last_change":"2024-03-21T10:35:11.230677+0000","last_active":"2024-03-21T10:35:12.183026+0000","last_peered":"2024-03-21T10:35:12.183026+0000","last_clean":"2024-03-21T10:35:12.183026+0000","last_became_active":"2024-03-21T10:35:11.230087+0000","last_became_peered":"2024-03-21T10:35:11.230087+0000","last_unstale":"2024-03-21T10:35:12.183026+0000","last_undegraded":"2024-03-21T10:35:12.183026+0000","last_fullsized":"2024-03-21T10:35:12.183026+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:39:47.663950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171306+0000","last_change":"2024-03-21T10:35:06.904287+0000","last_active":"2024-03-21T10:35:12.171306+0000","last_peered":"2024-03-21T10:35:12.171306+0000","last_clean":"2024-03-21T10:35:12.171306+0000","last_became_active":"2024-03-21T10:35:06.903821+0000","last_became_peered":"2024-03-21T10:35:06.903821+0000","last_unstale":"2024-03-21T10:35:12.171306+0000","last_undegraded":"2024-03-21T10:35:12.171306+0000","last_fullsized":"2024-03-21T10:35:12.171306+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:36:25.362203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.180503+0000","last_change":"2024-03-21T10:35:06.014432+0000","last_active":"2024-03-21T10:35:12.180503+0000","last_peered":"2024-03-21T10:35:12.180503+0000","last_clean":"2024-03-21T10:35:12.180503+0000","last_became_active":"2024-03-21T10:35:06.013448+0000","last_became_peered":"2024-03-21T10:35:06.013448+0000","last_unstale":"2024-03-21T10:35:12.180503+0000","last_undegraded":"2024-03-21T10:35:12.180503+0000","last_fullsized":"2024-03-21T10:35:12.180503+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:29:10.600794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.404581+0000","last_change":"2024-03-21T10:35:11.193577+0000","last_active":"2024-03-21T10:35:12.404581+0000","last_peered":"2024-03-21T10:35:12.404581+0000","last_clean":"2024-03-21T10:35:12.404581+0000","last_became_active":"2024-03-21T10:35:11.193132+0000","last_became_peered":"2024-03-21T10:35:11.193132+0000","last_unstale":"2024-03-21T10:35:12.404581+0000","last_undegraded":"2024-03-21T10:35:12.404581+0000","last_fullsized":"2024-03-21T10:35:12.404581+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:40:31.100489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.178899+0000","last_change":"2024-03-21T10:35:06.924376+0000","last_active":"2024-03-21T10:35:12.178899+0000","last_peered":"2024-03-21T10:35:12.178899+0000","last_clean":"2024-03-21T10:35:12.178899+0000","last_became_active":"2024-03-21T10:35:06.923874+0000","last_became_peered":"2024-03-21T10:35:06.923874+0000","last_unstale":"2024-03-21T10:35:12.178899+0000","last_undegraded":"2024-03-21T10:35:12.178899+0000","last_fullsized":"2024-03-21T10:35:12.178899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:56:12.620667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:39.323907+0000","last_change":"2024-03-21T10:35:06.015222+0000","last_active":"2024-03-21T10:35:39.323907+0000","last_peered":"2024-03-21T10:35:39.323907+0000","last_clean":"2024-03-21T10:35:39.323907+0000","last_became_active":"2024-03-21T10:35:06.014858+0000","last_became_peered":"2024-03-21T10:35:06.014858+0000","last_unstale":"2024-03-21T10:35:39.323907+0000","last_undegraded":"2024-03-21T10:35:39.323907+0000","last_fullsized":"2024-03-21T10:35:39.323907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:45:12.528851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":357,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170173+0000","last_change":"2024-03-21T10:33:44.536069+0000","last_active":"2024-03-21T10:35:12.170173+0000","last_peered":"2024-03-21T10:35:12.170173+0000","last_clean":"2024-03-21T10:35:12.170173+0000","last_became_active":"2024-03-21T10:33:44.535623+0000","last_became_peered":"2024-03-21T10:33:44.535623+0000","last_unstale":"2024-03-21T10:35:12.170173+0000","last_undegraded":"2024-03-21T10:35:12.170173+0000","last_fullsized":"2024-03-21T10:35:12.170173+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:29:44.554765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:29:44.554765+0000","last_clean_scrub_stamp":"2024-03-21T10:29:44.554765+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:43:23.440837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171426+0000","last_change":"2024-03-21T10:35:11.229580+0000","last_active":"2024-03-21T10:35:12.171426+0000","last_peered":"2024-03-21T10:35:12.171426+0000","last_clean":"2024-03-21T10:35:12.171426+0000","last_became_active":"2024-03-21T10:35:11.229044+0000","last_became_peered":"2024-03-21T10:35:11.229044+0000","last_unstale":"2024-03-21T10:35:12.171426+0000","last_undegraded":"2024-03-21T10:35:12.171426+0000","last_fullsized":"2024-03-21T10:35:12.171426+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:54:50.535253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.180069+0000","last_change":"2024-03-21T10:35:06.927690+0000","last_active":"2024-03-21T10:35:12.180069+0000","last_peered":"2024-03-21T10:35:12.180069+0000","last_clean":"2024-03-21T10:35:12.180069+0000","last_became_active":"2024-03-21T10:35:06.927304+0000","last_became_peered":"2024-03-21T10:35:06.927304+0000","last_unstale":"2024-03-21T10:35:12.180069+0000","last_undegraded":"2024-03-21T10:35:12.180069+0000","last_fullsized":"2024-03-21T10:35:12.180069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:30:48.950459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:40.328991+0000","last_change":"2024-03-21T10:35:06.004952+0000","last_active":"2024-03-21T10:35:40.328991+0000","last_peered":"2024-03-21T10:35:40.328991+0000","last_clean":"2024-03-21T10:35:40.328991+0000","last_became_active":"2024-03-21T10:35:06.004206+0000","last_became_peered":"2024-03-21T10:35:06.004206+0000","last_unstale":"2024-03-21T10:35:40.328991+0000","last_undegraded":"2024-03-21T10:35:40.328991+0000","last_fullsized":"2024-03-21T10:35:40.328991+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:20:29.198715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171390+0000","last_change":"2024-03-21T10:35:11.179125+0000","last_active":"2024-03-21T10:35:12.171390+0000","last_peered":"2024-03-21T10:35:12.171390+0000","last_clean":"2024-03-21T10:35:12.171390+0000","last_became_active":"2024-03-21T10:35:11.178658+0000","last_became_peered":"2024-03-21T10:35:11.178658+0000","last_unstale":"2024-03-21T10:35:12.171390+0000","last_undegraded":"2024-03-21T10:35:12.171390+0000","last_fullsized":"2024-03-21T10:35:12.171390+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:33:25.281721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:39.326145+0000","last_change":"2024-03-21T10:35:06.019996+0000","last_active":"2024-03-21T10:35:39.326145+0000","last_peered":"2024-03-21T10:35:39.326145+0000","last_clean":"2024-03-21T10:35:39.326145+0000","last_became_active":"2024-03-21T10:35:06.019624+0000","last_became_peered":"2024-03-21T10:35:06.019624+0000","last_unstale":"2024-03-21T10:35:39.326145+0000","last_undegraded":"2024-03-21T10:35:39.326145+0000","last_fullsized":"2024-03-21T10:35:39.326145+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:31:09.551433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.183595+0000","last_change":"2024-03-21T10:35:06.939324+0000","last_active":"2024-03-21T10:35:12.183595+0000","last_peered":"2024-03-21T10:35:12.183595+0000","last_clean":"2024-03-21T10:35:12.183595+0000","last_became_active":"2024-03-21T10:35:06.937750+0000","last_became_peered":"2024-03-21T10:35:06.937750+0000","last_unstale":"2024-03-21T10:35:12.183595+0000","last_undegraded":"2024-03-21T10:35:12.183595+0000","last_fullsized":"2024-03-21T10:35:12.183595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:45:14.600863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.175220+0000","last_change":"2024-03-21T10:35:11.229479+0000","last_active":"2024-03-21T10:35:12.175220+0000","last_peered":"2024-03-21T10:35:12.175220+0000","last_clean":"2024-03-21T10:35:12.175220+0000","last_became_active":"2024-03-21T10:35:11.229005+0000","last_became_peered":"2024-03-21T10:35:11.229005+0000","last_unstale":"2024-03-21T10:35:12.175220+0000","last_undegraded":"2024-03-21T10:35:12.175220+0000","last_fullsized":"2024-03-21T10:35:12.175220+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:39:19.859581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.173461+0000","last_change":"2024-03-21T10:35:06.013744+0000","last_active":"2024-03-21T10:35:12.173461+0000","last_peered":"2024-03-21T10:35:12.173461+0000","last_clean":"2024-03-21T10:35:12.173461+0000","last_became_active":"2024-03-21T10:35:06.013509+0000","last_became_peered":"2024-03-21T10:35:06.013509+0000","last_unstale":"2024-03-21T10:35:12.173461+0000","last_undegraded":"2024-03-21T10:35:12.173461+0000","last_fullsized":"2024-03-21T10:35:12.173461+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:51:45.000933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.184553+0000","last_change":"2024-03-21T10:35:06.930685+0000","last_active":"2024-03-21T10:35:12.184553+0000","last_peered":"2024-03-21T10:35:12.184553+0000","last_clean":"2024-03-21T10:35:12.184553+0000","last_became_active":"2024-03-21T10:35:06.929775+0000","last_became_peered":"2024-03-21T10:35:06.929775+0000","last_unstale":"2024-03-21T10:35:12.184553+0000","last_undegraded":"2024-03-21T10:35:12.184553+0000","last_fullsized":"2024-03-21T10:35:12.184553+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:13:18.058752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.166440+0000","last_change":"2024-03-21T10:35:11.177061+0000","last_active":"2024-03-21T10:35:12.166440+0000","last_peered":"2024-03-21T10:35:12.166440+0000","last_clean":"2024-03-21T10:35:12.166440+0000","last_became_active":"2024-03-21T10:35:11.176603+0000","last_became_peered":"2024-03-21T10:35:11.176603+0000","last_unstale":"2024-03-21T10:35:12.166440+0000","last_undegraded":"2024-03-21T10:35:12.166440+0000","last_fullsized":"2024-03-21T10:35:12.166440+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:54.532780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:39.321741+0000","last_change":"2024-03-21T10:35:06.009018+0000","last_active":"2024-03-21T10:35:39.321741+0000","last_peered":"2024-03-21T10:35:39.321741+0000","last_clean":"2024-03-21T10:35:39.321741+0000","last_became_active":"2024-03-21T10:35:06.008638+0000","last_became_peered":"2024-03-21T10:35:06.008638+0000","last_unstale":"2024-03-21T10:35:39.321741+0000","last_undegraded":"2024-03-21T10:35:39.321741+0000","last_fullsized":"2024-03-21T10:35:39.321741+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:42:19.527898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.183376+0000","last_change":"2024-03-21T10:35:06.931588+0000","last_active":"2024-03-21T10:35:12.183376+0000","last_peered":"2024-03-21T10:35:12.183376+0000","last_clean":"2024-03-21T10:35:12.183376+0000","last_became_active":"2024-03-21T10:35:06.930707+0000","last_became_peered":"2024-03-21T10:35:06.930707+0000","last_unstale":"2024-03-21T10:35:12.183376+0000","last_undegraded":"2024-03-21T10:35:12.183376+0000","last_fullsized":"2024-03-21T10:35:12.183376+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:44:55.862799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171662+0000","last_change":"2024-03-21T10:35:11.229408+0000","last_active":"2024-03-21T10:35:12.171662+0000","last_peered":"2024-03-21T10:35:12.171662+0000","last_clean":"2024-03-21T10:35:12.171662+0000","last_became_active":"2024-03-21T10:35:11.228997+0000","last_became_peered":"2024-03-21T10:35:11.228997+0000","last_unstale":"2024-03-21T10:35:12.171662+0000","last_undegraded":"2024-03-21T10:35:12.171662+0000","last_fullsized":"2024-03-21T10:35:12.171662+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:56:11.517597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:39.325411+0000","last_change":"2024-03-21T10:35:06.022332+0000","last_active":"2024-03-21T10:35:39.325411+0000","last_peered":"2024-03-21T10:35:39.325411+0000","last_clean":"2024-03-21T10:35:39.325411+0000","last_became_active":"2024-03-21T10:35:06.022015+0000","last_became_peered":"2024-03-21T10:35:06.022015+0000","last_unstale":"2024-03-21T10:35:39.325411+0000","last_undegraded":"2024-03-21T10:35:39.325411+0000","last_fullsized":"2024-03-21T10:35:39.325411+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:01:39.676516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170119+0000","last_change":"2024-03-21T10:35:06.908585+0000","last_active":"2024-03-21T10:35:12.170119+0000","last_peered":"2024-03-21T10:35:12.170119+0000","last_clean":"2024-03-21T10:35:12.170119+0000","last_became_active":"2024-03-21T10:35:06.908253+0000","last_became_peered":"2024-03-21T10:35:06.908253+0000","last_unstale":"2024-03-21T10:35:12.170119+0000","last_undegraded":"2024-03-21T10:35:12.170119+0000","last_fullsized":"2024-03-21T10:35:12.170119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:46:15.984450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171723+0000","last_change":"2024-03-21T10:35:11.199455+0000","last_active":"2024-03-21T10:35:12.171723+0000","last_peered":"2024-03-21T10:35:12.171723+0000","last_clean":"2024-03-21T10:35:12.171723+0000","last_became_active":"2024-03-21T10:35:11.197705+0000","last_became_peered":"2024-03-21T10:35:11.197705+0000","last_unstale":"2024-03-21T10:35:12.171723+0000","last_undegraded":"2024-03-21T10:35:12.171723+0000","last_fullsized":"2024-03-21T10:35:12.171723+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:59:42.792764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:39.328081+0000","last_change":"2024-03-21T10:35:06.016152+0000","last_active":"2024-03-21T10:35:39.328081+0000","last_peered":"2024-03-21T10:35:39.328081+0000","last_clean":"2024-03-21T10:35:39.328081+0000","last_became_active":"2024-03-21T10:35:06.015439+0000","last_became_peered":"2024-03-21T10:35:06.015439+0000","last_unstale":"2024-03-21T10:35:39.328081+0000","last_undegraded":"2024-03-21T10:35:39.328081+0000","last_fullsized":"2024-03-21T10:35:39.328081+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:05:23.221826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171138+0000","last_change":"2024-03-21T10:35:06.929241+0000","last_active":"2024-03-21T10:35:12.171138+0000","last_peered":"2024-03-21T10:35:12.171138+0000","last_clean":"2024-03-21T10:35:12.171138+0000","last_became_active":"2024-03-21T10:35:06.928943+0000","last_became_peered":"2024-03-21T10:35:06.928943+0000","last_unstale":"2024-03-21T10:35:12.171138+0000","last_undegraded":"2024-03-21T10:35:12.171138+0000","last_fullsized":"2024-03-21T10:35:12.171138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:24:10.484540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171251+0000","last_change":"2024-03-21T10:35:11.195929+0000","last_active":"2024-03-21T10:35:12.171251+0000","last_peered":"2024-03-21T10:35:12.171251+0000","last_clean":"2024-03-21T10:35:12.171251+0000","last_became_active":"2024-03-21T10:35:11.195489+0000","last_became_peered":"2024-03-21T10:35:11.195489+0000","last_unstale":"2024-03-21T10:35:12.171251+0000","last_undegraded":"2024-03-21T10:35:12.171251+0000","last_fullsized":"2024-03-21T10:35:12.171251+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:41:15.137917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.166385+0000","last_change":"2024-03-21T10:35:06.002127+0000","last_active":"2024-03-21T10:35:12.166385+0000","last_peered":"2024-03-21T10:35:12.166385+0000","last_clean":"2024-03-21T10:35:12.166385+0000","last_became_active":"2024-03-21T10:35:06.001706+0000","last_became_peered":"2024-03-21T10:35:06.001706+0000","last_unstale":"2024-03-21T10:35:12.166385+0000","last_undegraded":"2024-03-21T10:35:12.166385+0000","last_fullsized":"2024-03-21T10:35:12.166385+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:26:32.844991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.182261+0000","last_change":"2024-03-21T10:35:06.925945+0000","last_active":"2024-03-21T10:35:12.182261+0000","last_peered":"2024-03-21T10:35:12.182261+0000","last_clean":"2024-03-21T10:35:12.182261+0000","last_became_active":"2024-03-21T10:35:06.925551+0000","last_became_peered":"2024-03-21T10:35:06.925551+0000","last_unstale":"2024-03-21T10:35:12.182261+0000","last_undegraded":"2024-03-21T10:35:12.182261+0000","last_fullsized":"2024-03-21T10:35:12.182261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:24:10.212676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171393+0000","last_change":"2024-03-21T10:35:11.229067+0000","last_active":"2024-03-21T10:35:12.171393+0000","last_peered":"2024-03-21T10:35:12.171393+0000","last_clean":"2024-03-21T10:35:12.171393+0000","last_became_active":"2024-03-21T10:35:11.228628+0000","last_became_peered":"2024-03-21T10:35:11.228628+0000","last_unstale":"2024-03-21T10:35:12.171393+0000","last_undegraded":"2024-03-21T10:35:12.171393+0000","last_fullsized":"2024-03-21T10:35:12.171393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:45:05.473277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:39.323065+0000","last_change":"2024-03-21T10:35:06.004310+0000","last_active":"2024-03-21T10:35:39.323065+0000","last_peered":"2024-03-21T10:35:39.323065+0000","last_clean":"2024-03-21T10:35:39.323065+0000","last_became_active":"2024-03-21T10:35:06.003939+0000","last_became_peered":"2024-03-21T10:35:06.003939+0000","last_unstale":"2024-03-21T10:35:39.323065+0000","last_undegraded":"2024-03-21T10:35:39.323065+0000","last_fullsized":"2024-03-21T10:35:39.323065+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:08:46.008208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170782+0000","last_change":"2024-03-21T10:35:06.928049+0000","last_active":"2024-03-21T10:35:12.170782+0000","last_peered":"2024-03-21T10:35:12.170782+0000","last_clean":"2024-03-21T10:35:12.170782+0000","last_became_active":"2024-03-21T10:35:06.927717+0000","last_became_peered":"2024-03-21T10:35:06.927717+0000","last_unstale":"2024-03-21T10:35:12.170782+0000","last_undegraded":"2024-03-21T10:35:12.170782+0000","last_fullsized":"2024-03-21T10:35:12.170782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:27:47.554668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170095+0000","last_change":"2024-03-21T10:35:11.181090+0000","last_active":"2024-03-21T10:35:12.170095+0000","last_peered":"2024-03-21T10:35:12.170095+0000","last_clean":"2024-03-21T10:35:12.170095+0000","last_became_active":"2024-03-21T10:35:11.180511+0000","last_became_peered":"2024-03-21T10:35:11.180511+0000","last_unstale":"2024-03-21T10:35:12.170095+0000","last_undegraded":"2024-03-21T10:35:12.170095+0000","last_fullsized":"2024-03-21T10:35:12.170095+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:21:51.517845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.166972+0000","last_change":"2024-03-21T10:35:06.007828+0000","last_active":"2024-03-21T10:35:12.166972+0000","last_peered":"2024-03-21T10:35:12.166972+0000","last_clean":"2024-03-21T10:35:12.166972+0000","last_became_active":"2024-03-21T10:35:06.007490+0000","last_became_peered":"2024-03-21T10:35:06.007490+0000","last_unstale":"2024-03-21T10:35:12.166972+0000","last_undegraded":"2024-03-21T10:35:12.166972+0000","last_fullsized":"2024-03-21T10:35:12.166972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:08:18.579308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171294+0000","last_change":"2024-03-21T10:35:06.909313+0000","last_active":"2024-03-21T10:35:12.171294+0000","last_peered":"2024-03-21T10:35:12.171294+0000","last_clean":"2024-03-21T10:35:12.171294+0000","last_became_active":"2024-03-21T10:35:06.908555+0000","last_became_peered":"2024-03-21T10:35:06.908555+0000","last_unstale":"2024-03-21T10:35:12.171294+0000","last_undegraded":"2024-03-21T10:35:12.171294+0000","last_fullsized":"2024-03-21T10:35:12.171294+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:09:57.961971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.177393+0000","last_change":"2024-03-21T10:35:11.195996+0000","last_active":"2024-03-21T10:35:12.177393+0000","last_peered":"2024-03-21T10:35:12.177393+0000","last_clean":"2024-03-21T10:35:12.177393+0000","last_became_active":"2024-03-21T10:35:11.195506+0000","last_became_peered":"2024-03-21T10:35:11.195506+0000","last_unstale":"2024-03-21T10:35:12.177393+0000","last_undegraded":"2024-03-21T10:35:12.177393+0000","last_fullsized":"2024-03-21T10:35:12.177393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:49:33.010899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171522+0000","last_change":"2024-03-21T10:35:06.011498+0000","last_active":"2024-03-21T10:35:12.171522+0000","last_peered":"2024-03-21T10:35:12.171522+0000","last_clean":"2024-03-21T10:35:12.171522+0000","last_became_active":"2024-03-21T10:35:06.011203+0000","last_became_peered":"2024-03-21T10:35:06.011203+0000","last_unstale":"2024-03-21T10:35:12.171522+0000","last_undegraded":"2024-03-21T10:35:12.171522+0000","last_fullsized":"2024-03-21T10:35:12.171522+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:28:11.369740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170782+0000","last_change":"2024-03-21T10:35:06.935514+0000","last_active":"2024-03-21T10:35:12.170782+0000","last_peered":"2024-03-21T10:35:12.170782+0000","last_clean":"2024-03-21T10:35:12.170782+0000","last_became_active":"2024-03-21T10:35:06.935174+0000","last_became_peered":"2024-03-21T10:35:06.935174+0000","last_unstale":"2024-03-21T10:35:12.170782+0000","last_undegraded":"2024-03-21T10:35:12.170782+0000","last_fullsized":"2024-03-21T10:35:12.170782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:35:42.884471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170944+0000","last_change":"2024-03-21T10:35:11.180123+0000","last_active":"2024-03-21T10:35:12.170944+0000","last_peered":"2024-03-21T10:35:12.170944+0000","last_clean":"2024-03-21T10:35:12.170944+0000","last_became_active":"2024-03-21T10:35:11.179721+0000","last_became_peered":"2024-03-21T10:35:11.179721+0000","last_unstale":"2024-03-21T10:35:12.170944+0000","last_undegraded":"2024-03-21T10:35:12.170944+0000","last_fullsized":"2024-03-21T10:35:12.170944+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:02:09.287601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.182281+0000","last_change":"2024-03-21T10:35:06.007216+0000","last_active":"2024-03-21T10:35:12.182281+0000","last_peered":"2024-03-21T10:35:12.182281+0000","last_clean":"2024-03-21T10:35:12.182281+0000","last_became_active":"2024-03-21T10:35:06.006883+0000","last_became_peered":"2024-03-21T10:35:06.006883+0000","last_unstale":"2024-03-21T10:35:12.182281+0000","last_undegraded":"2024-03-21T10:35:12.182281+0000","last_fullsized":"2024-03-21T10:35:12.182281+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:46:45.310919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171319+0000","last_change":"2024-03-21T10:35:06.906017+0000","last_active":"2024-03-21T10:35:12.171319+0000","last_peered":"2024-03-21T10:35:12.171319+0000","last_clean":"2024-03-21T10:35:12.171319+0000","last_became_active":"2024-03-21T10:35:06.905648+0000","last_became_peered":"2024-03-21T10:35:06.905648+0000","last_unstale":"2024-03-21T10:35:12.171319+0000","last_undegraded":"2024-03-21T10:35:12.171319+0000","last_fullsized":"2024-03-21T10:35:12.171319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:56:50.380146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171189+0000","last_change":"2024-03-21T10:35:06.905186+0000","last_active":"2024-03-21T10:35:12.171189+0000","last_peered":"2024-03-21T10:35:12.171189+0000","last_clean":"2024-03-21T10:35:12.171189+0000","last_became_active":"2024-03-21T10:35:06.904835+0000","last_became_peered":"2024-03-21T10:35:06.904835+0000","last_unstale":"2024-03-21T10:35:12.171189+0000","last_undegraded":"2024-03-21T10:35:12.171189+0000","last_fullsized":"2024-03-21T10:35:12.171189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:53:19.293437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.166693+0000","last_change":"2024-03-21T10:35:06.903318+0000","last_active":"2024-03-21T10:35:12.166693+0000","last_peered":"2024-03-21T10:35:12.166693+0000","last_clean":"2024-03-21T10:35:12.166693+0000","last_became_active":"2024-03-21T10:35:06.902994+0000","last_became_peered":"2024-03-21T10:35:06.902994+0000","last_unstale":"2024-03-21T10:35:12.166693+0000","last_undegraded":"2024-03-21T10:35:12.166693+0000","last_fullsized":"2024-03-21T10:35:12.166693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:21:27.412565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.166337+0000","last_change":"2024-03-21T10:35:06.934582+0000","last_active":"2024-03-21T10:35:12.166337+0000","last_peered":"2024-03-21T10:35:12.166337+0000","last_clean":"2024-03-21T10:35:12.166337+0000","last_became_active":"2024-03-21T10:35:06.934311+0000","last_became_peered":"2024-03-21T10:35:06.934311+0000","last_unstale":"2024-03-21T10:35:12.166337+0000","last_undegraded":"2024-03-21T10:35:12.166337+0000","last_fullsized":"2024-03-21T10:35:12.166337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:05:30.785830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.177868+0000","last_change":"2024-03-21T10:35:06.944221+0000","last_active":"2024-03-21T10:35:12.177868+0000","last_peered":"2024-03-21T10:35:12.177868+0000","last_clean":"2024-03-21T10:35:12.177868+0000","last_became_active":"2024-03-21T10:35:06.943732+0000","last_became_peered":"2024-03-21T10:35:06.943732+0000","last_unstale":"2024-03-21T10:35:12.177868+0000","last_undegraded":"2024-03-21T10:35:12.177868+0000","last_fullsized":"2024-03-21T10:35:12.177868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:33:03.653160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.183234+0000","last_change":"2024-03-21T10:35:06.940832+0000","last_active":"2024-03-21T10:35:12.183234+0000","last_peered":"2024-03-21T10:35:12.183234+0000","last_clean":"2024-03-21T10:35:12.183234+0000","last_became_active":"2024-03-21T10:35:06.939565+0000","last_became_peered":"2024-03-21T10:35:06.939565+0000","last_unstale":"2024-03-21T10:35:12.183234+0000","last_undegraded":"2024-03-21T10:35:12.183234+0000","last_fullsized":"2024-03-21T10:35:12.183234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:57:27.743704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171828+0000","last_change":"2024-03-21T10:35:06.907753+0000","last_active":"2024-03-21T10:35:12.171828+0000","last_peered":"2024-03-21T10:35:12.171828+0000","last_clean":"2024-03-21T10:35:12.171828+0000","last_became_active":"2024-03-21T10:35:06.907432+0000","last_became_peered":"2024-03-21T10:35:06.907432+0000","last_unstale":"2024-03-21T10:35:12.171828+0000","last_undegraded":"2024-03-21T10:35:12.171828+0000","last_fullsized":"2024-03-21T10:35:12.171828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:31:02.528855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.167065+0000","last_change":"2024-03-21T10:35:06.908597+0000","last_active":"2024-03-21T10:35:12.167065+0000","last_peered":"2024-03-21T10:35:12.167065+0000","last_clean":"2024-03-21T10:35:12.167065+0000","last_became_active":"2024-03-21T10:35:06.908282+0000","last_became_peered":"2024-03-21T10:35:06.908282+0000","last_unstale":"2024-03-21T10:35:12.167065+0000","last_undegraded":"2024-03-21T10:35:12.167065+0000","last_fullsized":"2024-03-21T10:35:12.167065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:42:18.221045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.184741+0000","last_change":"2024-03-21T10:35:06.931015+0000","last_active":"2024-03-21T10:35:12.184741+0000","last_peered":"2024-03-21T10:35:12.184741+0000","last_clean":"2024-03-21T10:35:12.184741+0000","last_became_active":"2024-03-21T10:35:06.930209+0000","last_became_peered":"2024-03-21T10:35:06.930209+0000","last_unstale":"2024-03-21T10:35:12.184741+0000","last_undegraded":"2024-03-21T10:35:12.184741+0000","last_fullsized":"2024-03-21T10:35:12.184741+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:42:15.277874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171191+0000","last_change":"2024-03-21T10:35:06.930070+0000","last_active":"2024-03-21T10:35:12.171191+0000","last_peered":"2024-03-21T10:35:12.171191+0000","last_clean":"2024-03-21T10:35:12.171191+0000","last_became_active":"2024-03-21T10:35:06.929773+0000","last_became_peered":"2024-03-21T10:35:06.929773+0000","last_unstale":"2024-03-21T10:35:12.171191+0000","last_undegraded":"2024-03-21T10:35:12.171191+0000","last_fullsized":"2024-03-21T10:35:12.171191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:17:25.176666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.177677+0000","last_change":"2024-03-21T10:35:06.929575+0000","last_active":"2024-03-21T10:35:12.177677+0000","last_peered":"2024-03-21T10:35:12.177677+0000","last_clean":"2024-03-21T10:35:12.177677+0000","last_became_active":"2024-03-21T10:35:06.929240+0000","last_became_peered":"2024-03-21T10:35:06.929240+0000","last_unstale":"2024-03-21T10:35:12.177677+0000","last_undegraded":"2024-03-21T10:35:12.177677+0000","last_fullsized":"2024-03-21T10:35:12.177677+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:03:40.697640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170351+0000","last_change":"2024-03-21T10:35:06.934693+0000","last_active":"2024-03-21T10:35:12.170351+0000","last_peered":"2024-03-21T10:35:12.170351+0000","last_clean":"2024-03-21T10:35:12.170351+0000","last_became_active":"2024-03-21T10:35:06.934361+0000","last_became_peered":"2024-03-21T10:35:06.934361+0000","last_unstale":"2024-03-21T10:35:12.170351+0000","last_undegraded":"2024-03-21T10:35:12.170351+0000","last_fullsized":"2024-03-21T10:35:12.170351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:14:00.954370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.181074+0000","last_change":"2024-03-21T10:35:06.941095+0000","last_active":"2024-03-21T10:35:12.181074+0000","last_peered":"2024-03-21T10:35:12.181074+0000","last_clean":"2024-03-21T10:35:12.181074+0000","last_became_active":"2024-03-21T10:35:06.939348+0000","last_became_peered":"2024-03-21T10:35:06.939348+0000","last_unstale":"2024-03-21T10:35:12.181074+0000","last_undegraded":"2024-03-21T10:35:12.181074+0000","last_fullsized":"2024-03-21T10:35:12.181074+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:31:43.783592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171853+0000","last_change":"2024-03-21T10:35:06.908340+0000","last_active":"2024-03-21T10:35:12.171853+0000","last_peered":"2024-03-21T10:35:12.171853+0000","last_clean":"2024-03-21T10:35:12.171853+0000","last_became_active":"2024-03-21T10:35:06.908043+0000","last_became_peered":"2024-03-21T10:35:06.908043+0000","last_unstale":"2024-03-21T10:35:12.171853+0000","last_undegraded":"2024-03-21T10:35:12.171853+0000","last_fullsized":"2024-03-21T10:35:12.171853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:50:33.167396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282863,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4954,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":96,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":19229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383702,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":970752,"data_stored":481893,"data_compressed":14696,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645338,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":1073003,"data_compressed":50670,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808863,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":477999,"data_compressed":14696,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070503,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":983040,"data_stored":477953,"data_compressed":14696,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332140,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1032192,"data_stored":1073862,"data_compressed":50670,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495666,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14696,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":481057,"data_compressed":14696,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738452,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":966656,"data_stored":477305,"data_compressed":14696,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":481091,"data_compressed":14696,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182466,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":477499,"data_compressed":14696,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920828,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":477237,"data_compressed":14696,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757303,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1024000,"data_stored":1067765,"data_compressed":50670,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34331,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34331,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34331,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T10:35:45.244 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-21T10:35:45.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.500+0000 7f7b1fa00640 1 -- 172.21.15.31:0/4257352438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b181038c0 msgr2=0x7f7b18103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:45.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.500+0000 7f7b1fa00640 1 --2- 172.21.15.31:0/4257352438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b181038c0 0x7f7b18103ce0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c009900 tx=0x7f7b0c02bc30 comp rx=0 tx=0).stop 2024-03-21T10:35:45.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.500+0000 7f7b1fa00640 1 -- 172.21.15.31:0/4257352438 shutdown_connections 2024-03-21T10:35:45.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.500+0000 7f7b1fa00640 1 --2- 172.21.15.31:0/4257352438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b18105470 0x7f7b1810bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:45.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.500+0000 7f7b1fa00640 1 --2- 172.21.15.31:0/4257352438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b18104ab0 0x7f7b18104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:45.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.500+0000 7f7b1fa00640 1 --2- 172.21.15.31:0/4257352438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b181038c0 0x7f7b18103ce0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:45.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.500+0000 7f7b1fa00640 1 -- 172.21.15.31:0/4257352438 >> 172.21.15.31:0/4257352438 conn(0x7f7b180fee90 msgr2=0x7f7b181012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:45.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.500+0000 7f7b1fa00640 1 -- 172.21.15.31:0/4257352438 shutdown_connections 2024-03-21T10:35:45.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.500+0000 7f7b1fa00640 1 -- 172.21.15.31:0/4257352438 wait complete. 2024-03-21T10:35:45.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.501+0000 7f7b1fa00640 1 Processor -- start 2024-03-21T10:35:45.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.501+0000 7f7b1fa00640 1 -- start start 2024-03-21T10:35:45.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.501+0000 7f7b1fa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b181038c0 0x7f7b18199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:45.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.501+0000 7f7b1fa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b18104ab0 0x7f7b1819a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:45.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.502+0000 7f7b1fa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b18105470 0x7f7b181a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:45.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.502+0000 7f7b1fa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1810f560 con 0x7f7b18104ab0 2024-03-21T10:35:45.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.502+0000 7f7b1fa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1810f3e0 con 0x7f7b181038c0 2024-03-21T10:35:45.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.502+0000 7f7b1fa00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1810f6e0 con 0x7f7b18105470 2024-03-21T10:35:45.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.502+0000 7f7b1d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b181038c0 0x7f7b18199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:45.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.502+0000 7f7b1da00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b18105470 0x7f7b181a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:45.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.502+0000 7f7b17e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b18104ab0 0x7f7b1819a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:45.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.502+0000 7f7b1d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b181038c0 0x7f7b18199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44104/0 (socket says 172.21.15.31:44104) 2024-03-21T10:35:45.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.502+0000 7f7b1d000640 1 -- 172.21.15.31:0/1123381061 learned_addr learned my addr 172.21.15.31:0/1123381061 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:45.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.502+0000 7f7b1da00640 1 -- 172.21.15.31:0/1123381061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b181038c0 msgr2=0x7f7b18199bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:45.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.502+0000 7f7b1da00640 1 --2- 172.21.15.31:0/1123381061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b181038c0 0x7f7b18199bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:45.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.502+0000 7f7b1da00640 1 -- 172.21.15.31:0/1123381061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b18104ab0 msgr2=0x7f7b1819a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:45.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.502+0000 7f7b1da00640 1 --2- 172.21.15.31:0/1123381061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b18104ab0 0x7f7b1819a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:45.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.502+0000 7f7b1da00640 1 -- 172.21.15.31:0/1123381061 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b181a17f0 con 0x7f7b18105470 2024-03-21T10:35:45.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.502+0000 7f7b1d000640 1 --2- 172.21.15.31:0/1123381061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b181038c0 0x7f7b18199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:35:45.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.503+0000 7f7b1da00640 1 --2- 172.21.15.31:0/1123381061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b18105470 0x7f7b181a1180 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7b1000da10 tx=0x7f7b1000ded0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:45.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.503+0000 7f7b15600640 1 -- 172.21.15.31:0/1123381061 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1000bd70 con 0x7f7b18105470 2024-03-21T10:35:45.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.503+0000 7f7b15600640 1 -- 172.21.15.31:0/1123381061 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b10003e50 con 0x7f7b18105470 2024-03-21T10:35:45.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.503+0000 7f7b1fa00640 1 -- 172.21.15.31:0/1123381061 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b181a1ae0 con 0x7f7b18105470 2024-03-21T10:35:45.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.503+0000 7f7b15600640 1 -- 172.21.15.31:0/1123381061 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b100106a0 con 0x7f7b18105470 2024-03-21T10:35:45.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.503+0000 7f7b1fa00640 1 -- 172.21.15.31:0/1123381061 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b181a1fe0 con 0x7f7b18105470 2024-03-21T10:35:45.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.504+0000 7f7b1fa00640 1 -- 172.21.15.31:0/1123381061 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b1810abe0 con 0x7f7b18105470 2024-03-21T10:35:45.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.506+0000 7f7b15600640 1 -- 172.21.15.31:0/1123381061 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7f7b10003970 con 0x7f7b18105470 2024-03-21T10:35:45.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.507+0000 7f7b15600640 1 --2- 172.21.15.31:0/1123381061 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f7b000a2fb0 0x7f7b000a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:45.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.507+0000 7f7b15600640 1 -- 172.21.15.31:0/1123381061 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f7b100cfc10 con 0x7f7b18105470 2024-03-21T10:35:45.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.507+0000 7f7b1d000640 1 --2- 172.21.15.31:0/1123381061 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f7b000a2fb0 0x7f7b000a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:45.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.508+0000 7f7b1d000640 1 --2- 172.21.15.31:0/1123381061 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f7b000a2fb0 0x7f7b000a5470 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c006010 tx=0x7f7b0c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:45.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.511+0000 7f7b15600640 1 -- 172.21.15.31:0/1123381061 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7b1009c640 con 0x7f7b18105470 2024-03-21T10:35:45.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:45 smithi031 ceph-mon[31673]: pgmap v344: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-21T10:35:45.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:45 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3799186618' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:45.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:45 smithi040 ceph-mon[36154]: pgmap v344: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-21T10:35:45.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:45 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3799186618' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:45.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.797+0000 7f7b1fa00640 1 -- 172.21.15.31:0/1123381061 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7b18061ae0 con 0x7f7b000a2fb0 2024-03-21T10:35:45.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.825+0000 7f7b15600640 1 -- 172.21.15.31:0/1123381061 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340562 (secure 0 0 0) 0x7f7b18061ae0 con 0x7f7b000a2fb0 2024-03-21T10:35:45.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:45 smithi070 ceph-mon[36330]: pgmap v344: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-21T10:35:45.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:45 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3799186618' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:45.828 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:35:45.830 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-03-21T10:35:45.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.832+0000 7f7b1fa00640 1 -- 172.21.15.31:0/1123381061 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f7b000a2fb0 msgr2=0x7f7b000a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:45.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.832+0000 7f7b1fa00640 1 --2- 172.21.15.31:0/1123381061 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f7b000a2fb0 0x7f7b000a5470 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c006010 tx=0x7f7b0c02df90 comp rx=0 tx=0).stop 2024-03-21T10:35:45.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.832+0000 7f7b1fa00640 1 -- 172.21.15.31:0/1123381061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b18105470 msgr2=0x7f7b181a1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:45.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.832+0000 7f7b1fa00640 1 --2- 172.21.15.31:0/1123381061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b18105470 0x7f7b181a1180 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7b1000da10 tx=0x7f7b1000ded0 comp rx=0 tx=0).stop 2024-03-21T10:35:45.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.833+0000 7f7b1fa00640 1 -- 172.21.15.31:0/1123381061 shutdown_connections 2024-03-21T10:35:45.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.833+0000 7f7b1fa00640 1 --2- 172.21.15.31:0/1123381061 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f7b000a2fb0 0x7f7b000a5470 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:45.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.833+0000 7f7b1fa00640 1 --2- 172.21.15.31:0/1123381061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b18105470 0x7f7b181a1180 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:45.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.833+0000 7f7b1fa00640 1 --2- 172.21.15.31:0/1123381061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b18104ab0 0x7f7b1819a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:45.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.833+0000 7f7b1fa00640 1 --2- 172.21.15.31:0/1123381061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b181038c0 0x7f7b18199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:45.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.833+0000 7f7b1fa00640 1 -- 172.21.15.31:0/1123381061 >> 172.21.15.31:0/1123381061 conn(0x7f7b180fee90 msgr2=0x7f7b18107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:45.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.834+0000 7f7b1fa00640 1 -- 172.21.15.31:0/1123381061 shutdown_connections 2024-03-21T10:35:45.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:45.834+0000 7f7b1fa00640 1 -- 172.21.15.31:0/1123381061 wait complete. 2024-03-21T10:35:45.854 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-03-21T10:35:44.003900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595439,"num_objects":46,"num_object_clones":0,"num_object_copies":139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":141,"num_read_kb":124,"num_write":278,"num_write_kb":2975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":220,"ondisk_log_size":220,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":364928,"kb_used_data":11596,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576896,"statfs":{"total":1151940427776,"available":1151566741504,"internally_reserved":0,"allocated":11874304,"data_stored":7523879,"data_compressed":284274,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":431,"num_objects":22,"num_object_clones":0,"num_object_copies":67,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006428"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171420+0000","last_change":"2024-03-21T10:35:06.923544+0000","last_active":"2024-03-21T10:35:12.171420+0000","last_peered":"2024-03-21T10:35:12.171420+0000","last_clean":"2024-03-21T10:35:12.171420+0000","last_became_active":"2024-03-21T10:35:06.923254+0000","last_became_peered":"2024-03-21T10:35:06.923254+0000","last_unstale":"2024-03-21T10:35:12.171420+0000","last_undegraded":"2024-03-21T10:35:12.171420+0000","last_fullsized":"2024-03-21T10:35:12.171420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:38:14.573590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.183497+0000","last_change":"2024-03-21T10:35:06.929242+0000","last_active":"2024-03-21T10:35:12.183497+0000","last_peered":"2024-03-21T10:35:12.183497+0000","last_clean":"2024-03-21T10:35:12.183497+0000","last_became_active":"2024-03-21T10:35:06.927078+0000","last_became_peered":"2024-03-21T10:35:06.927078+0000","last_unstale":"2024-03-21T10:35:12.183497+0000","last_undegraded":"2024-03-21T10:35:12.183497+0000","last_fullsized":"2024-03-21T10:35:12.183497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:27:10.347945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.184632+0000","last_change":"2024-03-21T10:35:06.927792+0000","last_active":"2024-03-21T10:35:12.184632+0000","last_peered":"2024-03-21T10:35:12.184632+0000","last_clean":"2024-03-21T10:35:12.184632+0000","last_became_active":"2024-03-21T10:35:06.927386+0000","last_became_peered":"2024-03-21T10:35:06.927386+0000","last_unstale":"2024-03-21T10:35:12.184632+0000","last_undegraded":"2024-03-21T10:35:12.184632+0000","last_fullsized":"2024-03-21T10:35:12.184632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:53:51.310283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171374+0000","last_change":"2024-03-21T10:35:06.908901+0000","last_active":"2024-03-21T10:35:12.171374+0000","last_peered":"2024-03-21T10:35:12.171374+0000","last_clean":"2024-03-21T10:35:12.171374+0000","last_became_active":"2024-03-21T10:35:06.908586+0000","last_became_peered":"2024-03-21T10:35:06.908586+0000","last_unstale":"2024-03-21T10:35:12.171374+0000","last_undegraded":"2024-03-21T10:35:12.171374+0000","last_fullsized":"2024-03-21T10:35:12.171374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:25:57.387769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170261+0000","last_change":"2024-03-21T10:35:06.932537+0000","last_active":"2024-03-21T10:35:12.170261+0000","last_peered":"2024-03-21T10:35:12.170261+0000","last_clean":"2024-03-21T10:35:12.170261+0000","last_became_active":"2024-03-21T10:35:06.932185+0000","last_became_peered":"2024-03-21T10:35:06.932185+0000","last_unstale":"2024-03-21T10:35:12.170261+0000","last_undegraded":"2024-03-21T10:35:12.170261+0000","last_fullsized":"2024-03-21T10:35:12.170261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:30:25.127073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.166158+0000","last_change":"2024-03-21T10:35:06.922765+0000","last_active":"2024-03-21T10:35:12.166158+0000","last_peered":"2024-03-21T10:35:12.166158+0000","last_clean":"2024-03-21T10:35:12.166158+0000","last_became_active":"2024-03-21T10:35:06.922447+0000","last_became_peered":"2024-03-21T10:35:06.922447+0000","last_unstale":"2024-03-21T10:35:12.166158+0000","last_undegraded":"2024-03-21T10:35:12.166158+0000","last_fullsized":"2024-03-21T10:35:12.166158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:37:33.310735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170172+0000","last_change":"2024-03-21T10:35:06.896777+0000","last_active":"2024-03-21T10:35:12.170172+0000","last_peered":"2024-03-21T10:35:12.170172+0000","last_clean":"2024-03-21T10:35:12.170172+0000","last_became_active":"2024-03-21T10:35:06.896544+0000","last_became_peered":"2024-03-21T10:35:06.896544+0000","last_unstale":"2024-03-21T10:35:12.170172+0000","last_undegraded":"2024-03-21T10:35:12.170172+0000","last_fullsized":"2024-03-21T10:35:12.170172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:34:05.000661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.174554+0000","last_change":"2024-03-21T10:35:06.923294+0000","last_active":"2024-03-21T10:35:12.174554+0000","last_peered":"2024-03-21T10:35:12.174554+0000","last_clean":"2024-03-21T10:35:12.174554+0000","last_became_active":"2024-03-21T10:35:06.921320+0000","last_became_peered":"2024-03-21T10:35:06.921320+0000","last_unstale":"2024-03-21T10:35:12.174554+0000","last_undegraded":"2024-03-21T10:35:12.174554+0000","last_fullsized":"2024-03-21T10:35:12.174554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:33:53.855908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171265+0000","last_change":"2024-03-21T10:35:06.912977+0000","last_active":"2024-03-21T10:35:12.171265+0000","last_peered":"2024-03-21T10:35:12.171265+0000","last_clean":"2024-03-21T10:35:12.171265+0000","last_became_active":"2024-03-21T10:35:06.912626+0000","last_became_peered":"2024-03-21T10:35:06.912626+0000","last_unstale":"2024-03-21T10:35:12.171265+0000","last_undegraded":"2024-03-21T10:35:12.171265+0000","last_fullsized":"2024-03-21T10:35:12.171265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:42:13.472454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.166534+0000","last_change":"2024-03-21T10:35:06.910252+0000","last_active":"2024-03-21T10:35:12.166534+0000","last_peered":"2024-03-21T10:35:12.166534+0000","last_clean":"2024-03-21T10:35:12.166534+0000","last_became_active":"2024-03-21T10:35:06.909924+0000","last_became_peered":"2024-03-21T10:35:06.909924+0000","last_unstale":"2024-03-21T10:35:12.166534+0000","last_undegraded":"2024-03-21T10:35:12.166534+0000","last_fullsized":"2024-03-21T10:35:12.166534+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:43:15.527349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171192+0000","last_change":"2024-03-21T10:35:06.909569+0000","last_active":"2024-03-21T10:35:12.171192+0000","last_peered":"2024-03-21T10:35:12.171192+0000","last_clean":"2024-03-21T10:35:12.171192+0000","last_became_active":"2024-03-21T10:35:06.908982+0000","last_became_peered":"2024-03-21T10:35:06.908982+0000","last_unstale":"2024-03-21T10:35:12.171192+0000","last_undegraded":"2024-03-21T10:35:12.171192+0000","last_fullsized":"2024-03-21T10:35:12.171192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:13:56.020281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171353+0000","last_change":"2024-03-21T10:35:06.913861+0000","last_active":"2024-03-21T10:35:12.171353+0000","last_peered":"2024-03-21T10:35:12.171353+0000","last_clean":"2024-03-21T10:35:12.171353+0000","last_became_active":"2024-03-21T10:35:06.913490+0000","last_became_peered":"2024-03-21T10:35:06.913490+0000","last_unstale":"2024-03-21T10:35:12.171353+0000","last_undegraded":"2024-03-21T10:35:12.171353+0000","last_fullsized":"2024-03-21T10:35:12.171353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:24:18.573735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.180710+0000","last_change":"2024-03-21T10:35:06.934537+0000","last_active":"2024-03-21T10:35:12.180710+0000","last_peered":"2024-03-21T10:35:12.180710+0000","last_clean":"2024-03-21T10:35:12.180710+0000","last_became_active":"2024-03-21T10:35:06.932951+0000","last_became_peered":"2024-03-21T10:35:06.932951+0000","last_unstale":"2024-03-21T10:35:12.180710+0000","last_undegraded":"2024-03-21T10:35:12.180710+0000","last_fullsized":"2024-03-21T10:35:12.180710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:43:17.748434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.176882+0000","last_change":"2024-03-21T10:35:06.925495+0000","last_active":"2024-03-21T10:35:12.176882+0000","last_peered":"2024-03-21T10:35:12.176882+0000","last_clean":"2024-03-21T10:35:12.176882+0000","last_became_active":"2024-03-21T10:35:06.924326+0000","last_became_peered":"2024-03-21T10:35:06.924326+0000","last_unstale":"2024-03-21T10:35:12.176882+0000","last_undegraded":"2024-03-21T10:35:12.176882+0000","last_fullsized":"2024-03-21T10:35:12.176882+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:27:19.598213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.177431+0000","last_change":"2024-03-21T10:35:06.923695+0000","last_active":"2024-03-21T10:35:12.177431+0000","last_peered":"2024-03-21T10:35:12.177431+0000","last_clean":"2024-03-21T10:35:12.177431+0000","last_became_active":"2024-03-21T10:35:06.919114+0000","last_became_peered":"2024-03-21T10:35:06.919114+0000","last_unstale":"2024-03-21T10:35:12.177431+0000","last_undegraded":"2024-03-21T10:35:12.177431+0000","last_fullsized":"2024-03-21T10:35:12.177431+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:42:49.179653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.169783+0000","last_change":"2024-03-21T10:35:06.930225+0000","last_active":"2024-03-21T10:35:12.169783+0000","last_peered":"2024-03-21T10:35:12.169783+0000","last_clean":"2024-03-21T10:35:12.169783+0000","last_became_active":"2024-03-21T10:35:06.929905+0000","last_became_peered":"2024-03-21T10:35:06.929905+0000","last_unstale":"2024-03-21T10:35:12.169783+0000","last_undegraded":"2024-03-21T10:35:12.169783+0000","last_fullsized":"2024-03-21T10:35:12.169783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:21:07.231703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171177+0000","last_change":"2024-03-21T10:35:06.923498+0000","last_active":"2024-03-21T10:35:12.171177+0000","last_peered":"2024-03-21T10:35:12.171177+0000","last_clean":"2024-03-21T10:35:12.171177+0000","last_became_active":"2024-03-21T10:35:06.923242+0000","last_became_peered":"2024-03-21T10:35:06.923242+0000","last_unstale":"2024-03-21T10:35:12.171177+0000","last_undegraded":"2024-03-21T10:35:12.171177+0000","last_fullsized":"2024-03-21T10:35:12.171177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:42:04.309701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.173784+0000","last_change":"2024-03-21T10:35:06.918314+0000","last_active":"2024-03-21T10:35:12.173784+0000","last_peered":"2024-03-21T10:35:12.173784+0000","last_clean":"2024-03-21T10:35:12.173784+0000","last_became_active":"2024-03-21T10:35:06.917865+0000","last_became_peered":"2024-03-21T10:35:06.917865+0000","last_unstale":"2024-03-21T10:35:12.173784+0000","last_undegraded":"2024-03-21T10:35:12.173784+0000","last_fullsized":"2024-03-21T10:35:12.173784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:53:48.536341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.167081+0000","last_change":"2024-03-21T10:35:06.936234+0000","last_active":"2024-03-21T10:35:12.167081+0000","last_peered":"2024-03-21T10:35:12.167081+0000","last_clean":"2024-03-21T10:35:12.167081+0000","last_became_active":"2024-03-21T10:35:06.935958+0000","last_became_peered":"2024-03-21T10:35:06.935958+0000","last_unstale":"2024-03-21T10:35:12.167081+0000","last_undegraded":"2024-03-21T10:35:12.167081+0000","last_fullsized":"2024-03-21T10:35:12.167081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:52:43.509671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.166489+0000","last_change":"2024-03-21T10:35:11.198276+0000","last_active":"2024-03-21T10:35:12.166489+0000","last_peered":"2024-03-21T10:35:12.166489+0000","last_clean":"2024-03-21T10:35:12.166489+0000","last_became_active":"2024-03-21T10:35:11.197899+0000","last_became_peered":"2024-03-21T10:35:11.197899+0000","last_unstale":"2024-03-21T10:35:12.166489+0000","last_undegraded":"2024-03-21T10:35:12.166489+0000","last_fullsized":"2024-03-21T10:35:12.166489+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:17:30.499932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171995+0000","last_change":"2024-03-21T10:35:06.010647+0000","last_active":"2024-03-21T10:35:12.171995+0000","last_peered":"2024-03-21T10:35:12.171995+0000","last_clean":"2024-03-21T10:35:12.171995+0000","last_became_active":"2024-03-21T10:35:06.010239+0000","last_became_peered":"2024-03-21T10:35:06.010239+0000","last_unstale":"2024-03-21T10:35:12.171995+0000","last_undegraded":"2024-03-21T10:35:12.171995+0000","last_fullsized":"2024-03-21T10:35:12.171995+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:35:53.324044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.184015+0000","last_change":"2024-03-21T10:35:06.926398+0000","last_active":"2024-03-21T10:35:12.184015+0000","last_peered":"2024-03-21T10:35:12.184015+0000","last_clean":"2024-03-21T10:35:12.184015+0000","last_became_active":"2024-03-21T10:35:06.925904+0000","last_became_peered":"2024-03-21T10:35:06.925904+0000","last_unstale":"2024-03-21T10:35:12.184015+0000","last_undegraded":"2024-03-21T10:35:12.184015+0000","last_fullsized":"2024-03-21T10:35:12.184015+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:37:52.811604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171680+0000","last_change":"2024-03-21T10:35:11.195922+0000","last_active":"2024-03-21T10:35:12.171680+0000","last_peered":"2024-03-21T10:35:12.171680+0000","last_clean":"2024-03-21T10:35:12.171680+0000","last_became_active":"2024-03-21T10:35:11.195487+0000","last_became_peered":"2024-03-21T10:35:11.195487+0000","last_unstale":"2024-03-21T10:35:12.171680+0000","last_undegraded":"2024-03-21T10:35:12.171680+0000","last_fullsized":"2024-03-21T10:35:12.171680+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:34:16.346282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.184218+0000","last_change":"2024-03-21T10:35:06.022750+0000","last_active":"2024-03-21T10:35:12.184218+0000","last_peered":"2024-03-21T10:35:12.184218+0000","last_clean":"2024-03-21T10:35:12.184218+0000","last_became_active":"2024-03-21T10:35:06.022093+0000","last_became_peered":"2024-03-21T10:35:06.022093+0000","last_unstale":"2024-03-21T10:35:12.184218+0000","last_undegraded":"2024-03-21T10:35:12.184218+0000","last_fullsized":"2024-03-21T10:35:12.184218+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:53:28.458486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.182592+0000","last_change":"2024-03-21T10:35:06.927594+0000","last_active":"2024-03-21T10:35:12.182592+0000","last_peered":"2024-03-21T10:35:12.182592+0000","last_clean":"2024-03-21T10:35:12.182592+0000","last_became_active":"2024-03-21T10:35:06.926868+0000","last_became_peered":"2024-03-21T10:35:06.926868+0000","last_unstale":"2024-03-21T10:35:12.182592+0000","last_undegraded":"2024-03-21T10:35:12.182592+0000","last_fullsized":"2024-03-21T10:35:12.182592+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:07:33.127465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170428+0000","last_change":"2024-03-21T10:35:11.193475+0000","last_active":"2024-03-21T10:35:12.170428+0000","last_peered":"2024-03-21T10:35:12.170428+0000","last_clean":"2024-03-21T10:35:12.170428+0000","last_became_active":"2024-03-21T10:35:11.193054+0000","last_became_peered":"2024-03-21T10:35:11.193054+0000","last_unstale":"2024-03-21T10:35:12.170428+0000","last_undegraded":"2024-03-21T10:35:12.170428+0000","last_fullsized":"2024-03-21T10:35:12.170428+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:32:36.564147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:40.328623+0000","last_change":"2024-03-21T10:35:06.004257+0000","last_active":"2024-03-21T10:35:40.328623+0000","last_peered":"2024-03-21T10:35:40.328623+0000","last_clean":"2024-03-21T10:35:40.328623+0000","last_became_active":"2024-03-21T10:35:06.003936+0000","last_became_peered":"2024-03-21T10:35:06.003936+0000","last_unstale":"2024-03-21T10:35:40.328623+0000","last_undegraded":"2024-03-21T10:35:40.328623+0000","last_fullsized":"2024-03-21T10:35:40.328623+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:48:48.608979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170958+0000","last_change":"2024-03-21T10:35:06.941681+0000","last_active":"2024-03-21T10:35:12.170958+0000","last_peered":"2024-03-21T10:35:12.170958+0000","last_clean":"2024-03-21T10:35:12.170958+0000","last_became_active":"2024-03-21T10:35:06.941314+0000","last_became_peered":"2024-03-21T10:35:06.941314+0000","last_unstale":"2024-03-21T10:35:12.170958+0000","last_undegraded":"2024-03-21T10:35:12.170958+0000","last_fullsized":"2024-03-21T10:35:12.170958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:46:09.948922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170860+0000","last_change":"2024-03-21T10:35:11.188472+0000","last_active":"2024-03-21T10:35:12.170860+0000","last_peered":"2024-03-21T10:35:12.170860+0000","last_clean":"2024-03-21T10:35:12.170860+0000","last_became_active":"2024-03-21T10:35:11.188072+0000","last_became_peered":"2024-03-21T10:35:11.188072+0000","last_unstale":"2024-03-21T10:35:12.170860+0000","last_undegraded":"2024-03-21T10:35:12.170860+0000","last_fullsized":"2024-03-21T10:35:12.170860+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:04:16.911334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170556+0000","last_change":"2024-03-21T10:35:05.998493+0000","last_active":"2024-03-21T10:35:12.170556+0000","last_peered":"2024-03-21T10:35:12.170556+0000","last_clean":"2024-03-21T10:35:12.170556+0000","last_became_active":"2024-03-21T10:35:05.998134+0000","last_became_peered":"2024-03-21T10:35:05.998134+0000","last_unstale":"2024-03-21T10:35:12.170556+0000","last_undegraded":"2024-03-21T10:35:12.170556+0000","last_fullsized":"2024-03-21T10:35:12.170556+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:05:32.033328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171417+0000","last_change":"2024-03-21T10:35:06.912973+0000","last_active":"2024-03-21T10:35:12.171417+0000","last_peered":"2024-03-21T10:35:12.171417+0000","last_clean":"2024-03-21T10:35:12.171417+0000","last_became_active":"2024-03-21T10:35:06.912382+0000","last_became_peered":"2024-03-21T10:35:06.912382+0000","last_unstale":"2024-03-21T10:35:12.171417+0000","last_undegraded":"2024-03-21T10:35:12.171417+0000","last_fullsized":"2024-03-21T10:35:12.171417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:13:23.757805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.166991+0000","last_change":"2024-03-21T10:35:11.180806+0000","last_active":"2024-03-21T10:35:12.166991+0000","last_peered":"2024-03-21T10:35:12.166991+0000","last_clean":"2024-03-21T10:35:12.166991+0000","last_became_active":"2024-03-21T10:35:11.180303+0000","last_became_peered":"2024-03-21T10:35:11.180303+0000","last_unstale":"2024-03-21T10:35:12.166991+0000","last_undegraded":"2024-03-21T10:35:12.166991+0000","last_fullsized":"2024-03-21T10:35:12.166991+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:27:11.719654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170854+0000","last_change":"2024-03-21T10:35:06.004555+0000","last_active":"2024-03-21T10:35:12.170854+0000","last_peered":"2024-03-21T10:35:12.170854+0000","last_clean":"2024-03-21T10:35:12.170854+0000","last_became_active":"2024-03-21T10:35:06.003926+0000","last_became_peered":"2024-03-21T10:35:06.003926+0000","last_unstale":"2024-03-21T10:35:12.170854+0000","last_undegraded":"2024-03-21T10:35:12.170854+0000","last_fullsized":"2024-03-21T10:35:12.170854+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:05:49.721000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170924+0000","last_change":"2024-03-21T10:35:06.913556+0000","last_active":"2024-03-21T10:35:12.170924+0000","last_peered":"2024-03-21T10:35:12.170924+0000","last_clean":"2024-03-21T10:35:12.170924+0000","last_became_active":"2024-03-21T10:35:06.913213+0000","last_became_peered":"2024-03-21T10:35:06.913213+0000","last_unstale":"2024-03-21T10:35:12.170924+0000","last_undegraded":"2024-03-21T10:35:12.170924+0000","last_fullsized":"2024-03-21T10:35:12.170924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:11:09.235927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.166899+0000","last_change":"2024-03-21T10:35:11.195036+0000","last_active":"2024-03-21T10:35:12.166899+0000","last_peered":"2024-03-21T10:35:12.166899+0000","last_clean":"2024-03-21T10:35:12.166899+0000","last_became_active":"2024-03-21T10:35:11.194670+0000","last_became_peered":"2024-03-21T10:35:11.194670+0000","last_unstale":"2024-03-21T10:35:12.166899+0000","last_undegraded":"2024-03-21T10:35:12.166899+0000","last_fullsized":"2024-03-21T10:35:12.166899+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:37:02.955167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:40.335649+0000","last_change":"2024-03-21T10:35:06.012527+0000","last_active":"2024-03-21T10:35:40.335649+0000","last_peered":"2024-03-21T10:35:40.335649+0000","last_clean":"2024-03-21T10:35:40.335649+0000","last_became_active":"2024-03-21T10:35:06.012228+0000","last_became_peered":"2024-03-21T10:35:06.012228+0000","last_unstale":"2024-03-21T10:35:40.335649+0000","last_undegraded":"2024-03-21T10:35:40.335649+0000","last_fullsized":"2024-03-21T10:35:40.335649+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:52:34.622306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170989+0000","last_change":"2024-03-21T10:35:06.925810+0000","last_active":"2024-03-21T10:35:12.170989+0000","last_peered":"2024-03-21T10:35:12.170989+0000","last_clean":"2024-03-21T10:35:12.170989+0000","last_became_active":"2024-03-21T10:35:06.925540+0000","last_became_peered":"2024-03-21T10:35:06.925540+0000","last_unstale":"2024-03-21T10:35:12.170989+0000","last_undegraded":"2024-03-21T10:35:12.170989+0000","last_fullsized":"2024-03-21T10:35:12.170989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:26:10.949653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171101+0000","last_change":"2024-03-21T10:35:11.196543+0000","last_active":"2024-03-21T10:35:12.171101+0000","last_peered":"2024-03-21T10:35:12.171101+0000","last_clean":"2024-03-21T10:35:12.171101+0000","last_became_active":"2024-03-21T10:35:11.196151+0000","last_became_peered":"2024-03-21T10:35:11.196151+0000","last_unstale":"2024-03-21T10:35:12.171101+0000","last_undegraded":"2024-03-21T10:35:12.171101+0000","last_fullsized":"2024-03-21T10:35:12.171101+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:15:39.193647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.180661+0000","last_change":"2024-03-21T10:35:06.004588+0000","last_active":"2024-03-21T10:35:12.180661+0000","last_peered":"2024-03-21T10:35:12.180661+0000","last_clean":"2024-03-21T10:35:12.180661+0000","last_became_active":"2024-03-21T10:35:06.004190+0000","last_became_peered":"2024-03-21T10:35:06.004190+0000","last_unstale":"2024-03-21T10:35:12.180661+0000","last_undegraded":"2024-03-21T10:35:12.180661+0000","last_fullsized":"2024-03-21T10:35:12.180661+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:32:18.563692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.175870+0000","last_change":"2024-03-21T10:35:06.941738+0000","last_active":"2024-03-21T10:35:12.175870+0000","last_peered":"2024-03-21T10:35:12.175870+0000","last_clean":"2024-03-21T10:35:12.175870+0000","last_became_active":"2024-03-21T10:35:06.941386+0000","last_became_peered":"2024-03-21T10:35:06.941386+0000","last_unstale":"2024-03-21T10:35:12.175870+0000","last_undegraded":"2024-03-21T10:35:12.175870+0000","last_fullsized":"2024-03-21T10:35:12.175870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:52:55.519095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.184480+0000","last_change":"2024-03-21T10:35:11.199370+0000","last_active":"2024-03-21T10:35:12.184480+0000","last_peered":"2024-03-21T10:35:12.184480+0000","last_clean":"2024-03-21T10:35:12.184480+0000","last_became_active":"2024-03-21T10:35:11.198598+0000","last_became_peered":"2024-03-21T10:35:11.198598+0000","last_unstale":"2024-03-21T10:35:12.184480+0000","last_undegraded":"2024-03-21T10:35:12.184480+0000","last_fullsized":"2024-03-21T10:35:12.184480+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:07:19.419405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171940+0000","last_change":"2024-03-21T10:35:06.004893+0000","last_active":"2024-03-21T10:35:12.171940+0000","last_peered":"2024-03-21T10:35:12.171940+0000","last_clean":"2024-03-21T10:35:12.171940+0000","last_became_active":"2024-03-21T10:35:06.004586+0000","last_became_peered":"2024-03-21T10:35:06.004586+0000","last_unstale":"2024-03-21T10:35:12.171940+0000","last_undegraded":"2024-03-21T10:35:12.171940+0000","last_fullsized":"2024-03-21T10:35:12.171940+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:05:29.106575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.169378+0000","last_change":"2024-03-21T10:35:06.924448+0000","last_active":"2024-03-21T10:35:12.169378+0000","last_peered":"2024-03-21T10:35:12.169378+0000","last_clean":"2024-03-21T10:35:12.169378+0000","last_became_active":"2024-03-21T10:35:06.924049+0000","last_became_peered":"2024-03-21T10:35:06.924049+0000","last_unstale":"2024-03-21T10:35:12.169378+0000","last_undegraded":"2024-03-21T10:35:12.169378+0000","last_fullsized":"2024-03-21T10:35:12.169378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:23:12.070723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.176865+0000","last_change":"2024-03-21T10:35:11.188691+0000","last_active":"2024-03-21T10:35:12.176865+0000","last_peered":"2024-03-21T10:35:12.176865+0000","last_clean":"2024-03-21T10:35:12.176865+0000","last_became_active":"2024-03-21T10:35:11.188138+0000","last_became_peered":"2024-03-21T10:35:11.188138+0000","last_unstale":"2024-03-21T10:35:12.176865+0000","last_undegraded":"2024-03-21T10:35:12.176865+0000","last_fullsized":"2024-03-21T10:35:12.176865+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:51:50.920920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:39.325104+0000","last_change":"2024-03-21T10:35:06.015276+0000","last_active":"2024-03-21T10:35:39.325104+0000","last_peered":"2024-03-21T10:35:39.325104+0000","last_clean":"2024-03-21T10:35:39.325104+0000","last_became_active":"2024-03-21T10:35:06.014734+0000","last_became_peered":"2024-03-21T10:35:06.014734+0000","last_unstale":"2024-03-21T10:35:39.325104+0000","last_undegraded":"2024-03-21T10:35:39.325104+0000","last_fullsized":"2024-03-21T10:35:39.325104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:18:45.905451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170963+0000","last_change":"2024-03-21T10:35:06.926689+0000","last_active":"2024-03-21T10:35:12.170963+0000","last_peered":"2024-03-21T10:35:12.170963+0000","last_clean":"2024-03-21T10:35:12.170963+0000","last_became_active":"2024-03-21T10:35:06.926319+0000","last_became_peered":"2024-03-21T10:35:06.926319+0000","last_unstale":"2024-03-21T10:35:12.170963+0000","last_undegraded":"2024-03-21T10:35:12.170963+0000","last_fullsized":"2024-03-21T10:35:12.170963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:06:22.371608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.169159+0000","last_change":"2024-03-21T10:35:11.203528+0000","last_active":"2024-03-21T10:35:12.169159+0000","last_peered":"2024-03-21T10:35:12.169159+0000","last_clean":"2024-03-21T10:35:12.169159+0000","last_became_active":"2024-03-21T10:35:11.202174+0000","last_became_peered":"2024-03-21T10:35:11.202174+0000","last_unstale":"2024-03-21T10:35:12.169159+0000","last_undegraded":"2024-03-21T10:35:12.169159+0000","last_fullsized":"2024-03-21T10:35:12.169159+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:21:07.038401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:39.324186+0000","last_change":"2024-03-21T10:35:06.007231+0000","last_active":"2024-03-21T10:35:39.324186+0000","last_peered":"2024-03-21T10:35:39.324186+0000","last_clean":"2024-03-21T10:35:39.324186+0000","last_became_active":"2024-03-21T10:35:06.006875+0000","last_became_peered":"2024-03-21T10:35:06.006875+0000","last_unstale":"2024-03-21T10:35:39.324186+0000","last_undegraded":"2024-03-21T10:35:39.324186+0000","last_fullsized":"2024-03-21T10:35:39.324186+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:08:02.489741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.183859+0000","last_change":"2024-03-21T10:35:06.925940+0000","last_active":"2024-03-21T10:35:12.183859+0000","last_peered":"2024-03-21T10:35:12.183859+0000","last_clean":"2024-03-21T10:35:12.183859+0000","last_became_active":"2024-03-21T10:35:06.924515+0000","last_became_peered":"2024-03-21T10:35:06.924515+0000","last_unstale":"2024-03-21T10:35:12.183859+0000","last_undegraded":"2024-03-21T10:35:12.183859+0000","last_fullsized":"2024-03-21T10:35:12.183859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:19:20.527047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170638+0000","last_change":"2024-03-21T10:35:11.193382+0000","last_active":"2024-03-21T10:35:12.170638+0000","last_peered":"2024-03-21T10:35:12.170638+0000","last_clean":"2024-03-21T10:35:12.170638+0000","last_became_active":"2024-03-21T10:35:11.192959+0000","last_became_peered":"2024-03-21T10:35:11.192959+0000","last_unstale":"2024-03-21T10:35:12.170638+0000","last_undegraded":"2024-03-21T10:35:12.170638+0000","last_fullsized":"2024-03-21T10:35:12.170638+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:02:57.187370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171296+0000","last_change":"2024-03-21T10:35:06.019969+0000","last_active":"2024-03-21T10:35:12.171296+0000","last_peered":"2024-03-21T10:35:12.171296+0000","last_clean":"2024-03-21T10:35:12.171296+0000","last_became_active":"2024-03-21T10:35:06.019648+0000","last_became_peered":"2024-03-21T10:35:06.019648+0000","last_unstale":"2024-03-21T10:35:12.171296+0000","last_undegraded":"2024-03-21T10:35:12.171296+0000","last_fullsized":"2024-03-21T10:35:12.171296+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:24:02.227205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171469+0000","last_change":"2024-03-21T10:35:06.910294+0000","last_active":"2024-03-21T10:35:12.171469+0000","last_peered":"2024-03-21T10:35:12.171469+0000","last_clean":"2024-03-21T10:35:12.171469+0000","last_became_active":"2024-03-21T10:35:06.909862+0000","last_became_peered":"2024-03-21T10:35:06.909862+0000","last_unstale":"2024-03-21T10:35:12.171469+0000","last_undegraded":"2024-03-21T10:35:12.171469+0000","last_fullsized":"2024-03-21T10:35:12.171469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:37:13.396003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.169593+0000","last_change":"2024-03-21T10:35:11.196398+0000","last_active":"2024-03-21T10:35:12.169593+0000","last_peered":"2024-03-21T10:35:12.169593+0000","last_clean":"2024-03-21T10:35:12.169593+0000","last_became_active":"2024-03-21T10:35:11.194965+0000","last_became_peered":"2024-03-21T10:35:11.194965+0000","last_unstale":"2024-03-21T10:35:12.169593+0000","last_undegraded":"2024-03-21T10:35:12.169593+0000","last_fullsized":"2024-03-21T10:35:12.169593+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:48:22.024137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:39.323326+0000","last_change":"2024-03-21T10:35:06.005037+0000","last_active":"2024-03-21T10:35:39.323326+0000","last_peered":"2024-03-21T10:35:39.323326+0000","last_clean":"2024-03-21T10:35:39.323326+0000","last_became_active":"2024-03-21T10:35:06.004686+0000","last_became_peered":"2024-03-21T10:35:06.004686+0000","last_unstale":"2024-03-21T10:35:39.323326+0000","last_undegraded":"2024-03-21T10:35:39.323326+0000","last_fullsized":"2024-03-21T10:35:39.323326+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:24:13.312112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170837+0000","last_change":"2024-03-21T10:35:06.906525+0000","last_active":"2024-03-21T10:35:12.170837+0000","last_peered":"2024-03-21T10:35:12.170837+0000","last_clean":"2024-03-21T10:35:12.170837+0000","last_became_active":"2024-03-21T10:35:06.905150+0000","last_became_peered":"2024-03-21T10:35:06.905150+0000","last_unstale":"2024-03-21T10:35:12.170837+0000","last_undegraded":"2024-03-21T10:35:12.170837+0000","last_fullsized":"2024-03-21T10:35:12.170837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:46:27.958779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171957+0000","last_change":"2024-03-21T10:35:11.193519+0000","last_active":"2024-03-21T10:35:12.171957+0000","last_peered":"2024-03-21T10:35:12.171957+0000","last_clean":"2024-03-21T10:35:12.171957+0000","last_became_active":"2024-03-21T10:35:11.193043+0000","last_became_peered":"2024-03-21T10:35:11.193043+0000","last_unstale":"2024-03-21T10:35:12.171957+0000","last_undegraded":"2024-03-21T10:35:12.171957+0000","last_fullsized":"2024-03-21T10:35:12.171957+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:23:20.984472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:39.323912+0000","last_change":"2024-03-21T10:35:05.999220+0000","last_active":"2024-03-21T10:35:39.323912+0000","last_peered":"2024-03-21T10:35:39.323912+0000","last_clean":"2024-03-21T10:35:39.323912+0000","last_became_active":"2024-03-21T10:35:05.998918+0000","last_became_peered":"2024-03-21T10:35:05.998918+0000","last_unstale":"2024-03-21T10:35:39.323912+0000","last_undegraded":"2024-03-21T10:35:39.323912+0000","last_fullsized":"2024-03-21T10:35:39.323912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:01:09.816758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.181059+0000","last_change":"2024-03-21T10:35:06.941234+0000","last_active":"2024-03-21T10:35:12.181059+0000","last_peered":"2024-03-21T10:35:12.181059+0000","last_clean":"2024-03-21T10:35:12.181059+0000","last_became_active":"2024-03-21T10:35:06.940828+0000","last_became_peered":"2024-03-21T10:35:06.940828+0000","last_unstale":"2024-03-21T10:35:12.181059+0000","last_undegraded":"2024-03-21T10:35:12.181059+0000","last_fullsized":"2024-03-21T10:35:12.181059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:50:34.398822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171713+0000","last_change":"2024-03-21T10:35:11.196007+0000","last_active":"2024-03-21T10:35:12.171713+0000","last_peered":"2024-03-21T10:35:12.171713+0000","last_clean":"2024-03-21T10:35:12.171713+0000","last_became_active":"2024-03-21T10:35:11.195609+0000","last_became_peered":"2024-03-21T10:35:11.195609+0000","last_unstale":"2024-03-21T10:35:12.171713+0000","last_undegraded":"2024-03-21T10:35:12.171713+0000","last_fullsized":"2024-03-21T10:35:12.171713+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:21:38.803390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170877+0000","last_change":"2024-03-21T10:35:06.011594+0000","last_active":"2024-03-21T10:35:12.170877+0000","last_peered":"2024-03-21T10:35:12.170877+0000","last_clean":"2024-03-21T10:35:12.170877+0000","last_became_active":"2024-03-21T10:35:06.011254+0000","last_became_peered":"2024-03-21T10:35:06.011254+0000","last_unstale":"2024-03-21T10:35:12.170877+0000","last_undegraded":"2024-03-21T10:35:12.170877+0000","last_fullsized":"2024-03-21T10:35:12.170877+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:52:31.045659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170534+0000","last_change":"2024-03-21T10:35:06.929170+0000","last_active":"2024-03-21T10:35:12.170534+0000","last_peered":"2024-03-21T10:35:12.170534+0000","last_clean":"2024-03-21T10:35:12.170534+0000","last_became_active":"2024-03-21T10:35:06.928822+0000","last_became_peered":"2024-03-21T10:35:06.928822+0000","last_unstale":"2024-03-21T10:35:12.170534+0000","last_undegraded":"2024-03-21T10:35:12.170534+0000","last_fullsized":"2024-03-21T10:35:12.170534+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:37:40.328453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171663+0000","last_change":"2024-03-21T10:35:11.204285+0000","last_active":"2024-03-21T10:35:12.171663+0000","last_peered":"2024-03-21T10:35:12.171663+0000","last_clean":"2024-03-21T10:35:12.171663+0000","last_became_active":"2024-03-21T10:35:11.203698+0000","last_became_peered":"2024-03-21T10:35:11.203698+0000","last_unstale":"2024-03-21T10:35:12.171663+0000","last_undegraded":"2024-03-21T10:35:12.171663+0000","last_fullsized":"2024-03-21T10:35:12.171663+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:03:32.732488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170547+0000","last_change":"2024-03-21T10:35:06.005028+0000","last_active":"2024-03-21T10:35:12.170547+0000","last_peered":"2024-03-21T10:35:12.170547+0000","last_clean":"2024-03-21T10:35:12.170547+0000","last_became_active":"2024-03-21T10:35:06.004670+0000","last_became_peered":"2024-03-21T10:35:06.004670+0000","last_unstale":"2024-03-21T10:35:12.170547+0000","last_undegraded":"2024-03-21T10:35:12.170547+0000","last_fullsized":"2024-03-21T10:35:12.170547+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:00:18.481051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170455+0000","last_change":"2024-03-21T10:35:06.911696+0000","last_active":"2024-03-21T10:35:12.170455+0000","last_peered":"2024-03-21T10:35:12.170455+0000","last_clean":"2024-03-21T10:35:12.170455+0000","last_became_active":"2024-03-21T10:35:06.911231+0000","last_became_peered":"2024-03-21T10:35:06.911231+0000","last_unstale":"2024-03-21T10:35:12.170455+0000","last_undegraded":"2024-03-21T10:35:12.170455+0000","last_fullsized":"2024-03-21T10:35:12.170455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:21:51.540993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.181675+0000","last_change":"2024-03-21T10:35:11.204422+0000","last_active":"2024-03-21T10:35:12.181675+0000","last_peered":"2024-03-21T10:35:12.181675+0000","last_clean":"2024-03-21T10:35:12.181675+0000","last_became_active":"2024-03-21T10:35:11.203904+0000","last_became_peered":"2024-03-21T10:35:11.203904+0000","last_unstale":"2024-03-21T10:35:12.181675+0000","last_undegraded":"2024-03-21T10:35:12.181675+0000","last_fullsized":"2024-03-21T10:35:12.181675+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:38:14.486012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:40.336487+0000","last_change":"2024-03-21T10:35:06.019823+0000","last_active":"2024-03-21T10:35:40.336487+0000","last_peered":"2024-03-21T10:35:40.336487+0000","last_clean":"2024-03-21T10:35:40.336487+0000","last_became_active":"2024-03-21T10:35:06.019502+0000","last_became_peered":"2024-03-21T10:35:06.019502+0000","last_unstale":"2024-03-21T10:35:40.336487+0000","last_undegraded":"2024-03-21T10:35:40.336487+0000","last_fullsized":"2024-03-21T10:35:40.336487+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:33:22.438868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170828+0000","last_change":"2024-03-21T10:35:06.906671+0000","last_active":"2024-03-21T10:35:12.170828+0000","last_peered":"2024-03-21T10:35:12.170828+0000","last_clean":"2024-03-21T10:35:12.170828+0000","last_became_active":"2024-03-21T10:35:06.905873+0000","last_became_peered":"2024-03-21T10:35:06.905873+0000","last_unstale":"2024-03-21T10:35:12.170828+0000","last_undegraded":"2024-03-21T10:35:12.170828+0000","last_fullsized":"2024-03-21T10:35:12.170828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:14:59.304255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170758+0000","last_change":"2024-03-21T10:35:11.197698+0000","last_active":"2024-03-21T10:35:12.170758+0000","last_peered":"2024-03-21T10:35:12.170758+0000","last_clean":"2024-03-21T10:35:12.170758+0000","last_became_active":"2024-03-21T10:35:11.197295+0000","last_became_peered":"2024-03-21T10:35:11.197295+0000","last_unstale":"2024-03-21T10:35:12.170758+0000","last_undegraded":"2024-03-21T10:35:12.170758+0000","last_fullsized":"2024-03-21T10:35:12.170758+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:02:24.338317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:40.997992+0000","last_change":"2024-03-21T10:35:06.000951+0000","last_active":"2024-03-21T10:35:40.997992+0000","last_peered":"2024-03-21T10:35:40.997992+0000","last_clean":"2024-03-21T10:35:40.997992+0000","last_became_active":"2024-03-21T10:35:05.999169+0000","last_became_peered":"2024-03-21T10:35:05.999169+0000","last_unstale":"2024-03-21T10:35:40.997992+0000","last_undegraded":"2024-03-21T10:35:40.997992+0000","last_fullsized":"2024-03-21T10:35:40.997992+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:31:25.602621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170766+0000","last_change":"2024-03-21T10:35:06.908067+0000","last_active":"2024-03-21T10:35:12.170766+0000","last_peered":"2024-03-21T10:35:12.170766+0000","last_clean":"2024-03-21T10:35:12.170766+0000","last_became_active":"2024-03-21T10:35:06.907720+0000","last_became_peered":"2024-03-21T10:35:06.907720+0000","last_unstale":"2024-03-21T10:35:12.170766+0000","last_undegraded":"2024-03-21T10:35:12.170766+0000","last_fullsized":"2024-03-21T10:35:12.170766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:16:00.502637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170863+0000","last_change":"2024-03-21T10:35:11.229470+0000","last_active":"2024-03-21T10:35:12.170863+0000","last_peered":"2024-03-21T10:35:12.170863+0000","last_clean":"2024-03-21T10:35:12.170863+0000","last_became_active":"2024-03-21T10:35:11.228977+0000","last_became_peered":"2024-03-21T10:35:11.228977+0000","last_unstale":"2024-03-21T10:35:12.170863+0000","last_undegraded":"2024-03-21T10:35:12.170863+0000","last_fullsized":"2024-03-21T10:35:12.170863+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:05:52.269712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171362+0000","last_change":"2024-03-21T10:35:06.019866+0000","last_active":"2024-03-21T10:35:12.171362+0000","last_peered":"2024-03-21T10:35:12.171362+0000","last_clean":"2024-03-21T10:35:12.171362+0000","last_became_active":"2024-03-21T10:35:06.019550+0000","last_became_peered":"2024-03-21T10:35:06.019550+0000","last_unstale":"2024-03-21T10:35:12.171362+0000","last_undegraded":"2024-03-21T10:35:12.171362+0000","last_fullsized":"2024-03-21T10:35:12.171362+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:51:46.989647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.166384+0000","last_change":"2024-03-21T10:35:06.928051+0000","last_active":"2024-03-21T10:35:12.166384+0000","last_peered":"2024-03-21T10:35:12.166384+0000","last_clean":"2024-03-21T10:35:12.166384+0000","last_became_active":"2024-03-21T10:35:06.927707+0000","last_became_peered":"2024-03-21T10:35:06.927707+0000","last_unstale":"2024-03-21T10:35:12.166384+0000","last_undegraded":"2024-03-21T10:35:12.166384+0000","last_fullsized":"2024-03-21T10:35:12.166384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:01:39.502891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.182897+0000","last_change":"2024-03-21T10:35:11.199714+0000","last_active":"2024-03-21T10:35:12.182897+0000","last_peered":"2024-03-21T10:35:12.182897+0000","last_clean":"2024-03-21T10:35:12.182897+0000","last_became_active":"2024-03-21T10:35:11.199277+0000","last_became_peered":"2024-03-21T10:35:11.199277+0000","last_unstale":"2024-03-21T10:35:12.182897+0000","last_undegraded":"2024-03-21T10:35:12.182897+0000","last_fullsized":"2024-03-21T10:35:12.182897+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:58:01.471162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170722+0000","last_change":"2024-03-21T10:35:06.015090+0000","last_active":"2024-03-21T10:35:12.170722+0000","last_peered":"2024-03-21T10:35:12.170722+0000","last_clean":"2024-03-21T10:35:12.170722+0000","last_became_active":"2024-03-21T10:35:06.014666+0000","last_became_peered":"2024-03-21T10:35:06.014666+0000","last_unstale":"2024-03-21T10:35:12.170722+0000","last_undegraded":"2024-03-21T10:35:12.170722+0000","last_fullsized":"2024-03-21T10:35:12.170722+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:41:17.608069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.166085+0000","last_change":"2024-03-21T10:35:06.923729+0000","last_active":"2024-03-21T10:35:12.166085+0000","last_peered":"2024-03-21T10:35:12.166085+0000","last_clean":"2024-03-21T10:35:12.166085+0000","last_became_active":"2024-03-21T10:35:06.923361+0000","last_became_peered":"2024-03-21T10:35:06.923361+0000","last_unstale":"2024-03-21T10:35:12.166085+0000","last_undegraded":"2024-03-21T10:35:12.166085+0000","last_fullsized":"2024-03-21T10:35:12.166085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:20:40.819207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.183026+0000","last_change":"2024-03-21T10:35:11.230677+0000","last_active":"2024-03-21T10:35:12.183026+0000","last_peered":"2024-03-21T10:35:12.183026+0000","last_clean":"2024-03-21T10:35:12.183026+0000","last_became_active":"2024-03-21T10:35:11.230087+0000","last_became_peered":"2024-03-21T10:35:11.230087+0000","last_unstale":"2024-03-21T10:35:12.183026+0000","last_undegraded":"2024-03-21T10:35:12.183026+0000","last_fullsized":"2024-03-21T10:35:12.183026+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:39:47.663950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171306+0000","last_change":"2024-03-21T10:35:06.904287+0000","last_active":"2024-03-21T10:35:12.171306+0000","last_peered":"2024-03-21T10:35:12.171306+0000","last_clean":"2024-03-21T10:35:12.171306+0000","last_became_active":"2024-03-21T10:35:06.903821+0000","last_became_peered":"2024-03-21T10:35:06.903821+0000","last_unstale":"2024-03-21T10:35:12.171306+0000","last_undegraded":"2024-03-21T10:35:12.171306+0000","last_fullsized":"2024-03-21T10:35:12.171306+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:36:25.362203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.180503+0000","last_change":"2024-03-21T10:35:06.014432+0000","last_active":"2024-03-21T10:35:12.180503+0000","last_peered":"2024-03-21T10:35:12.180503+0000","last_clean":"2024-03-21T10:35:12.180503+0000","last_became_active":"2024-03-21T10:35:06.013448+0000","last_became_peered":"2024-03-21T10:35:06.013448+0000","last_unstale":"2024-03-21T10:35:12.180503+0000","last_undegraded":"2024-03-21T10:35:12.180503+0000","last_fullsized":"2024-03-21T10:35:12.180503+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:29:10.600794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.404581+0000","last_change":"2024-03-21T10:35:11.193577+0000","last_active":"2024-03-21T10:35:12.404581+0000","last_peered":"2024-03-21T10:35:12.404581+0000","last_clean":"2024-03-21T10:35:12.404581+0000","last_became_active":"2024-03-21T10:35:11.193132+0000","last_became_peered":"2024-03-21T10:35:11.193132+0000","last_unstale":"2024-03-21T10:35:12.404581+0000","last_undegraded":"2024-03-21T10:35:12.404581+0000","last_fullsized":"2024-03-21T10:35:12.404581+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:40:31.100489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.178899+0000","last_change":"2024-03-21T10:35:06.924376+0000","last_active":"2024-03-21T10:35:12.178899+0000","last_peered":"2024-03-21T10:35:12.178899+0000","last_clean":"2024-03-21T10:35:12.178899+0000","last_became_active":"2024-03-21T10:35:06.923874+0000","last_became_peered":"2024-03-21T10:35:06.923874+0000","last_unstale":"2024-03-21T10:35:12.178899+0000","last_undegraded":"2024-03-21T10:35:12.178899+0000","last_fullsized":"2024-03-21T10:35:12.178899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:56:12.620667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:39.323907+0000","last_change":"2024-03-21T10:35:06.015222+0000","last_active":"2024-03-21T10:35:39.323907+0000","last_peered":"2024-03-21T10:35:39.323907+0000","last_clean":"2024-03-21T10:35:39.323907+0000","last_became_active":"2024-03-21T10:35:06.014858+0000","last_became_peered":"2024-03-21T10:35:06.014858+0000","last_unstale":"2024-03-21T10:35:39.323907+0000","last_undegraded":"2024-03-21T10:35:39.323907+0000","last_fullsized":"2024-03-21T10:35:39.323907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:45:12.528851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":357,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170173+0000","last_change":"2024-03-21T10:33:44.536069+0000","last_active":"2024-03-21T10:35:12.170173+0000","last_peered":"2024-03-21T10:35:12.170173+0000","last_clean":"2024-03-21T10:35:12.170173+0000","last_became_active":"2024-03-21T10:33:44.535623+0000","last_became_peered":"2024-03-21T10:33:44.535623+0000","last_unstale":"2024-03-21T10:35:12.170173+0000","last_undegraded":"2024-03-21T10:35:12.170173+0000","last_fullsized":"2024-03-21T10:35:12.170173+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:29:44.554765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:29:44.554765+0000","last_clean_scrub_stamp":"2024-03-21T10:29:44.554765+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:43:23.440837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171426+0000","last_change":"2024-03-21T10:35:11.229580+0000","last_active":"2024-03-21T10:35:12.171426+0000","last_peered":"2024-03-21T10:35:12.171426+0000","last_clean":"2024-03-21T10:35:12.171426+0000","last_became_active":"2024-03-21T10:35:11.229044+0000","last_became_peered":"2024-03-21T10:35:11.229044+0000","last_unstale":"2024-03-21T10:35:12.171426+0000","last_undegraded":"2024-03-21T10:35:12.171426+0000","last_fullsized":"2024-03-21T10:35:12.171426+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:54:50.535253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.180069+0000","last_change":"2024-03-21T10:35:06.927690+0000","last_active":"2024-03-21T10:35:12.180069+0000","last_peered":"2024-03-21T10:35:12.180069+0000","last_clean":"2024-03-21T10:35:12.180069+0000","last_became_active":"2024-03-21T10:35:06.927304+0000","last_became_peered":"2024-03-21T10:35:06.927304+0000","last_unstale":"2024-03-21T10:35:12.180069+0000","last_undegraded":"2024-03-21T10:35:12.180069+0000","last_fullsized":"2024-03-21T10:35:12.180069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:30:48.950459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:40.328991+0000","last_change":"2024-03-21T10:35:06.004952+0000","last_active":"2024-03-21T10:35:40.328991+0000","last_peered":"2024-03-21T10:35:40.328991+0000","last_clean":"2024-03-21T10:35:40.328991+0000","last_became_active":"2024-03-21T10:35:06.004206+0000","last_became_peered":"2024-03-21T10:35:06.004206+0000","last_unstale":"2024-03-21T10:35:40.328991+0000","last_undegraded":"2024-03-21T10:35:40.328991+0000","last_fullsized":"2024-03-21T10:35:40.328991+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:20:29.198715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171390+0000","last_change":"2024-03-21T10:35:11.179125+0000","last_active":"2024-03-21T10:35:12.171390+0000","last_peered":"2024-03-21T10:35:12.171390+0000","last_clean":"2024-03-21T10:35:12.171390+0000","last_became_active":"2024-03-21T10:35:11.178658+0000","last_became_peered":"2024-03-21T10:35:11.178658+0000","last_unstale":"2024-03-21T10:35:12.171390+0000","last_undegraded":"2024-03-21T10:35:12.171390+0000","last_fullsized":"2024-03-21T10:35:12.171390+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:33:25.281721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:39.326145+0000","last_change":"2024-03-21T10:35:06.019996+0000","last_active":"2024-03-21T10:35:39.326145+0000","last_peered":"2024-03-21T10:35:39.326145+0000","last_clean":"2024-03-21T10:35:39.326145+0000","last_became_active":"2024-03-21T10:35:06.019624+0000","last_became_peered":"2024-03-21T10:35:06.019624+0000","last_unstale":"2024-03-21T10:35:39.326145+0000","last_undegraded":"2024-03-21T10:35:39.326145+0000","last_fullsized":"2024-03-21T10:35:39.326145+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:31:09.551433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.183595+0000","last_change":"2024-03-21T10:35:06.939324+0000","last_active":"2024-03-21T10:35:12.183595+0000","last_peered":"2024-03-21T10:35:12.183595+0000","last_clean":"2024-03-21T10:35:12.183595+0000","last_became_active":"2024-03-21T10:35:06.937750+0000","last_became_peered":"2024-03-21T10:35:06.937750+0000","last_unstale":"2024-03-21T10:35:12.183595+0000","last_undegraded":"2024-03-21T10:35:12.183595+0000","last_fullsized":"2024-03-21T10:35:12.183595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:45:14.600863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.175220+0000","last_change":"2024-03-21T10:35:11.229479+0000","last_active":"2024-03-21T10:35:12.175220+0000","last_peered":"2024-03-21T10:35:12.175220+0000","last_clean":"2024-03-21T10:35:12.175220+0000","last_became_active":"2024-03-21T10:35:11.229005+0000","last_became_peered":"2024-03-21T10:35:11.229005+0000","last_unstale":"2024-03-21T10:35:12.175220+0000","last_undegraded":"2024-03-21T10:35:12.175220+0000","last_fullsized":"2024-03-21T10:35:12.175220+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:39:19.859581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.173461+0000","last_change":"2024-03-21T10:35:06.013744+0000","last_active":"2024-03-21T10:35:12.173461+0000","last_peered":"2024-03-21T10:35:12.173461+0000","last_clean":"2024-03-21T10:35:12.173461+0000","last_became_active":"2024-03-21T10:35:06.013509+0000","last_became_peered":"2024-03-21T10:35:06.013509+0000","last_unstale":"2024-03-21T10:35:12.173461+0000","last_undegraded":"2024-03-21T10:35:12.173461+0000","last_fullsized":"2024-03-21T10:35:12.173461+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:51:45.000933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.184553+0000","last_change":"2024-03-21T10:35:06.930685+0000","last_active":"2024-03-21T10:35:12.184553+0000","last_peered":"2024-03-21T10:35:12.184553+0000","last_clean":"2024-03-21T10:35:12.184553+0000","last_became_active":"2024-03-21T10:35:06.929775+0000","last_became_peered":"2024-03-21T10:35:06.929775+0000","last_unstale":"2024-03-21T10:35:12.184553+0000","last_undegraded":"2024-03-21T10:35:12.184553+0000","last_fullsized":"2024-03-21T10:35:12.184553+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:13:18.058752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.166440+0000","last_change":"2024-03-21T10:35:11.177061+0000","last_active":"2024-03-21T10:35:12.166440+0000","last_peered":"2024-03-21T10:35:12.166440+0000","last_clean":"2024-03-21T10:35:12.166440+0000","last_became_active":"2024-03-21T10:35:11.176603+0000","last_became_peered":"2024-03-21T10:35:11.176603+0000","last_unstale":"2024-03-21T10:35:12.166440+0000","last_undegraded":"2024-03-21T10:35:12.166440+0000","last_fullsized":"2024-03-21T10:35:12.166440+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:54.532780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:39.321741+0000","last_change":"2024-03-21T10:35:06.009018+0000","last_active":"2024-03-21T10:35:39.321741+0000","last_peered":"2024-03-21T10:35:39.321741+0000","last_clean":"2024-03-21T10:35:39.321741+0000","last_became_active":"2024-03-21T10:35:06.008638+0000","last_became_peered":"2024-03-21T10:35:06.008638+0000","last_unstale":"2024-03-21T10:35:39.321741+0000","last_undegraded":"2024-03-21T10:35:39.321741+0000","last_fullsized":"2024-03-21T10:35:39.321741+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:42:19.527898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.183376+0000","last_change":"2024-03-21T10:35:06.931588+0000","last_active":"2024-03-21T10:35:12.183376+0000","last_peered":"2024-03-21T10:35:12.183376+0000","last_clean":"2024-03-21T10:35:12.183376+0000","last_became_active":"2024-03-21T10:35:06.930707+0000","last_became_peered":"2024-03-21T10:35:06.930707+0000","last_unstale":"2024-03-21T10:35:12.183376+0000","last_undegraded":"2024-03-21T10:35:12.183376+0000","last_fullsized":"2024-03-21T10:35:12.183376+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:44:55.862799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171662+0000","last_change":"2024-03-21T10:35:11.229408+0000","last_active":"2024-03-21T10:35:12.171662+0000","last_peered":"2024-03-21T10:35:12.171662+0000","last_clean":"2024-03-21T10:35:12.171662+0000","last_became_active":"2024-03-21T10:35:11.228997+0000","last_became_peered":"2024-03-21T10:35:11.228997+0000","last_unstale":"2024-03-21T10:35:12.171662+0000","last_undegraded":"2024-03-21T10:35:12.171662+0000","last_fullsized":"2024-03-21T10:35:12.171662+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:56:11.517597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:39.325411+0000","last_change":"2024-03-21T10:35:06.022332+0000","last_active":"2024-03-21T10:35:39.325411+0000","last_peered":"2024-03-21T10:35:39.325411+0000","last_clean":"2024-03-21T10:35:39.325411+0000","last_became_active":"2024-03-21T10:35:06.022015+0000","last_became_peered":"2024-03-21T10:35:06.022015+0000","last_unstale":"2024-03-21T10:35:39.325411+0000","last_undegraded":"2024-03-21T10:35:39.325411+0000","last_fullsized":"2024-03-21T10:35:39.325411+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:01:39.676516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170119+0000","last_change":"2024-03-21T10:35:06.908585+0000","last_active":"2024-03-21T10:35:12.170119+0000","last_peered":"2024-03-21T10:35:12.170119+0000","last_clean":"2024-03-21T10:35:12.170119+0000","last_became_active":"2024-03-21T10:35:06.908253+0000","last_became_peered":"2024-03-21T10:35:06.908253+0000","last_unstale":"2024-03-21T10:35:12.170119+0000","last_undegraded":"2024-03-21T10:35:12.170119+0000","last_fullsized":"2024-03-21T10:35:12.170119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:46:15.984450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171723+0000","last_change":"2024-03-21T10:35:11.199455+0000","last_active":"2024-03-21T10:35:12.171723+0000","last_peered":"2024-03-21T10:35:12.171723+0000","last_clean":"2024-03-21T10:35:12.171723+0000","last_became_active":"2024-03-21T10:35:11.197705+0000","last_became_peered":"2024-03-21T10:35:11.197705+0000","last_unstale":"2024-03-21T10:35:12.171723+0000","last_undegraded":"2024-03-21T10:35:12.171723+0000","last_fullsized":"2024-03-21T10:35:12.171723+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:59:42.792764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:39.328081+0000","last_change":"2024-03-21T10:35:06.016152+0000","last_active":"2024-03-21T10:35:39.328081+0000","last_peered":"2024-03-21T10:35:39.328081+0000","last_clean":"2024-03-21T10:35:39.328081+0000","last_became_active":"2024-03-21T10:35:06.015439+0000","last_became_peered":"2024-03-21T10:35:06.015439+0000","last_unstale":"2024-03-21T10:35:39.328081+0000","last_undegraded":"2024-03-21T10:35:39.328081+0000","last_fullsized":"2024-03-21T10:35:39.328081+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:05:23.221826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171138+0000","last_change":"2024-03-21T10:35:06.929241+0000","last_active":"2024-03-21T10:35:12.171138+0000","last_peered":"2024-03-21T10:35:12.171138+0000","last_clean":"2024-03-21T10:35:12.171138+0000","last_became_active":"2024-03-21T10:35:06.928943+0000","last_became_peered":"2024-03-21T10:35:06.928943+0000","last_unstale":"2024-03-21T10:35:12.171138+0000","last_undegraded":"2024-03-21T10:35:12.171138+0000","last_fullsized":"2024-03-21T10:35:12.171138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:24:10.484540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171251+0000","last_change":"2024-03-21T10:35:11.195929+0000","last_active":"2024-03-21T10:35:12.171251+0000","last_peered":"2024-03-21T10:35:12.171251+0000","last_clean":"2024-03-21T10:35:12.171251+0000","last_became_active":"2024-03-21T10:35:11.195489+0000","last_became_peered":"2024-03-21T10:35:11.195489+0000","last_unstale":"2024-03-21T10:35:12.171251+0000","last_undegraded":"2024-03-21T10:35:12.171251+0000","last_fullsized":"2024-03-21T10:35:12.171251+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:41:15.137917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.166385+0000","last_change":"2024-03-21T10:35:06.002127+0000","last_active":"2024-03-21T10:35:12.166385+0000","last_peered":"2024-03-21T10:35:12.166385+0000","last_clean":"2024-03-21T10:35:12.166385+0000","last_became_active":"2024-03-21T10:35:06.001706+0000","last_became_peered":"2024-03-21T10:35:06.001706+0000","last_unstale":"2024-03-21T10:35:12.166385+0000","last_undegraded":"2024-03-21T10:35:12.166385+0000","last_fullsized":"2024-03-21T10:35:12.166385+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:26:32.844991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.182261+0000","last_change":"2024-03-21T10:35:06.925945+0000","last_active":"2024-03-21T10:35:12.182261+0000","last_peered":"2024-03-21T10:35:12.182261+0000","last_clean":"2024-03-21T10:35:12.182261+0000","last_became_active":"2024-03-21T10:35:06.925551+0000","last_became_peered":"2024-03-21T10:35:06.925551+0000","last_unstale":"2024-03-21T10:35:12.182261+0000","last_undegraded":"2024-03-21T10:35:12.182261+0000","last_fullsized":"2024-03-21T10:35:12.182261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:24:10.212676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171393+0000","last_change":"2024-03-21T10:35:11.229067+0000","last_active":"2024-03-21T10:35:12.171393+0000","last_peered":"2024-03-21T10:35:12.171393+0000","last_clean":"2024-03-21T10:35:12.171393+0000","last_became_active":"2024-03-21T10:35:11.228628+0000","last_became_peered":"2024-03-21T10:35:11.228628+0000","last_unstale":"2024-03-21T10:35:12.171393+0000","last_undegraded":"2024-03-21T10:35:12.171393+0000","last_fullsized":"2024-03-21T10:35:12.171393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:45:05.473277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:39.323065+0000","last_change":"2024-03-21T10:35:06.004310+0000","last_active":"2024-03-21T10:35:39.323065+0000","last_peered":"2024-03-21T10:35:39.323065+0000","last_clean":"2024-03-21T10:35:39.323065+0000","last_became_active":"2024-03-21T10:35:06.003939+0000","last_became_peered":"2024-03-21T10:35:06.003939+0000","last_unstale":"2024-03-21T10:35:39.323065+0000","last_undegraded":"2024-03-21T10:35:39.323065+0000","last_fullsized":"2024-03-21T10:35:39.323065+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:08:46.008208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170782+0000","last_change":"2024-03-21T10:35:06.928049+0000","last_active":"2024-03-21T10:35:12.170782+0000","last_peered":"2024-03-21T10:35:12.170782+0000","last_clean":"2024-03-21T10:35:12.170782+0000","last_became_active":"2024-03-21T10:35:06.927717+0000","last_became_peered":"2024-03-21T10:35:06.927717+0000","last_unstale":"2024-03-21T10:35:12.170782+0000","last_undegraded":"2024-03-21T10:35:12.170782+0000","last_fullsized":"2024-03-21T10:35:12.170782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:27:47.554668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170095+0000","last_change":"2024-03-21T10:35:11.181090+0000","last_active":"2024-03-21T10:35:12.170095+0000","last_peered":"2024-03-21T10:35:12.170095+0000","last_clean":"2024-03-21T10:35:12.170095+0000","last_became_active":"2024-03-21T10:35:11.180511+0000","last_became_peered":"2024-03-21T10:35:11.180511+0000","last_unstale":"2024-03-21T10:35:12.170095+0000","last_undegraded":"2024-03-21T10:35:12.170095+0000","last_fullsized":"2024-03-21T10:35:12.170095+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:21:51.517845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.166972+0000","last_change":"2024-03-21T10:35:06.007828+0000","last_active":"2024-03-21T10:35:12.166972+0000","last_peered":"2024-03-21T10:35:12.166972+0000","last_clean":"2024-03-21T10:35:12.166972+0000","last_became_active":"2024-03-21T10:35:06.007490+0000","last_became_peered":"2024-03-21T10:35:06.007490+0000","last_unstale":"2024-03-21T10:35:12.166972+0000","last_undegraded":"2024-03-21T10:35:12.166972+0000","last_fullsized":"2024-03-21T10:35:12.166972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:08:18.579308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171294+0000","last_change":"2024-03-21T10:35:06.909313+0000","last_active":"2024-03-21T10:35:12.171294+0000","last_peered":"2024-03-21T10:35:12.171294+0000","last_clean":"2024-03-21T10:35:12.171294+0000","last_became_active":"2024-03-21T10:35:06.908555+0000","last_became_peered":"2024-03-21T10:35:06.908555+0000","last_unstale":"2024-03-21T10:35:12.171294+0000","last_undegraded":"2024-03-21T10:35:12.171294+0000","last_fullsized":"2024-03-21T10:35:12.171294+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:09:57.961971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.177393+0000","last_change":"2024-03-21T10:35:11.195996+0000","last_active":"2024-03-21T10:35:12.177393+0000","last_peered":"2024-03-21T10:35:12.177393+0000","last_clean":"2024-03-21T10:35:12.177393+0000","last_became_active":"2024-03-21T10:35:11.195506+0000","last_became_peered":"2024-03-21T10:35:11.195506+0000","last_unstale":"2024-03-21T10:35:12.177393+0000","last_undegraded":"2024-03-21T10:35:12.177393+0000","last_fullsized":"2024-03-21T10:35:12.177393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:49:33.010899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171522+0000","last_change":"2024-03-21T10:35:06.011498+0000","last_active":"2024-03-21T10:35:12.171522+0000","last_peered":"2024-03-21T10:35:12.171522+0000","last_clean":"2024-03-21T10:35:12.171522+0000","last_became_active":"2024-03-21T10:35:06.011203+0000","last_became_peered":"2024-03-21T10:35:06.011203+0000","last_unstale":"2024-03-21T10:35:12.171522+0000","last_undegraded":"2024-03-21T10:35:12.171522+0000","last_fullsized":"2024-03-21T10:35:12.171522+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:28:11.369740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170782+0000","last_change":"2024-03-21T10:35:06.935514+0000","last_active":"2024-03-21T10:35:12.170782+0000","last_peered":"2024-03-21T10:35:12.170782+0000","last_clean":"2024-03-21T10:35:12.170782+0000","last_became_active":"2024-03-21T10:35:06.935174+0000","last_became_peered":"2024-03-21T10:35:06.935174+0000","last_unstale":"2024-03-21T10:35:12.170782+0000","last_undegraded":"2024-03-21T10:35:12.170782+0000","last_fullsized":"2024-03-21T10:35:12.170782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:35:42.884471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170944+0000","last_change":"2024-03-21T10:35:11.180123+0000","last_active":"2024-03-21T10:35:12.170944+0000","last_peered":"2024-03-21T10:35:12.170944+0000","last_clean":"2024-03-21T10:35:12.170944+0000","last_became_active":"2024-03-21T10:35:11.179721+0000","last_became_peered":"2024-03-21T10:35:11.179721+0000","last_unstale":"2024-03-21T10:35:12.170944+0000","last_undegraded":"2024-03-21T10:35:12.170944+0000","last_fullsized":"2024-03-21T10:35:12.170944+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:02:09.287601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.182281+0000","last_change":"2024-03-21T10:35:06.007216+0000","last_active":"2024-03-21T10:35:12.182281+0000","last_peered":"2024-03-21T10:35:12.182281+0000","last_clean":"2024-03-21T10:35:12.182281+0000","last_became_active":"2024-03-21T10:35:06.006883+0000","last_became_peered":"2024-03-21T10:35:06.006883+0000","last_unstale":"2024-03-21T10:35:12.182281+0000","last_undegraded":"2024-03-21T10:35:12.182281+0000","last_fullsized":"2024-03-21T10:35:12.182281+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:46:45.310919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171319+0000","last_change":"2024-03-21T10:35:06.906017+0000","last_active":"2024-03-21T10:35:12.171319+0000","last_peered":"2024-03-21T10:35:12.171319+0000","last_clean":"2024-03-21T10:35:12.171319+0000","last_became_active":"2024-03-21T10:35:06.905648+0000","last_became_peered":"2024-03-21T10:35:06.905648+0000","last_unstale":"2024-03-21T10:35:12.171319+0000","last_undegraded":"2024-03-21T10:35:12.171319+0000","last_fullsized":"2024-03-21T10:35:12.171319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:56:50.380146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171189+0000","last_change":"2024-03-21T10:35:06.905186+0000","last_active":"2024-03-21T10:35:12.171189+0000","last_peered":"2024-03-21T10:35:12.171189+0000","last_clean":"2024-03-21T10:35:12.171189+0000","last_became_active":"2024-03-21T10:35:06.904835+0000","last_became_peered":"2024-03-21T10:35:06.904835+0000","last_unstale":"2024-03-21T10:35:12.171189+0000","last_undegraded":"2024-03-21T10:35:12.171189+0000","last_fullsized":"2024-03-21T10:35:12.171189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:53:19.293437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.166693+0000","last_change":"2024-03-21T10:35:06.903318+0000","last_active":"2024-03-21T10:35:12.166693+0000","last_peered":"2024-03-21T10:35:12.166693+0000","last_clean":"2024-03-21T10:35:12.166693+0000","last_became_active":"2024-03-21T10:35:06.902994+0000","last_became_peered":"2024-03-21T10:35:06.902994+0000","last_unstale":"2024-03-21T10:35:12.166693+0000","last_undegraded":"2024-03-21T10:35:12.166693+0000","last_fullsized":"2024-03-21T10:35:12.166693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:21:27.412565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.166337+0000","last_change":"2024-03-21T10:35:06.934582+0000","last_active":"2024-03-21T10:35:12.166337+0000","last_peered":"2024-03-21T10:35:12.166337+0000","last_clean":"2024-03-21T10:35:12.166337+0000","last_became_active":"2024-03-21T10:35:06.934311+0000","last_became_peered":"2024-03-21T10:35:06.934311+0000","last_unstale":"2024-03-21T10:35:12.166337+0000","last_undegraded":"2024-03-21T10:35:12.166337+0000","last_fullsized":"2024-03-21T10:35:12.166337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:05:30.785830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.177868+0000","last_change":"2024-03-21T10:35:06.944221+0000","last_active":"2024-03-21T10:35:12.177868+0000","last_peered":"2024-03-21T10:35:12.177868+0000","last_clean":"2024-03-21T10:35:12.177868+0000","last_became_active":"2024-03-21T10:35:06.943732+0000","last_became_peered":"2024-03-21T10:35:06.943732+0000","last_unstale":"2024-03-21T10:35:12.177868+0000","last_undegraded":"2024-03-21T10:35:12.177868+0000","last_fullsized":"2024-03-21T10:35:12.177868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:33:03.653160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.183234+0000","last_change":"2024-03-21T10:35:06.940832+0000","last_active":"2024-03-21T10:35:12.183234+0000","last_peered":"2024-03-21T10:35:12.183234+0000","last_clean":"2024-03-21T10:35:12.183234+0000","last_became_active":"2024-03-21T10:35:06.939565+0000","last_became_peered":"2024-03-21T10:35:06.939565+0000","last_unstale":"2024-03-21T10:35:12.183234+0000","last_undegraded":"2024-03-21T10:35:12.183234+0000","last_fullsized":"2024-03-21T10:35:12.183234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:57:27.743704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171828+0000","last_change":"2024-03-21T10:35:06.907753+0000","last_active":"2024-03-21T10:35:12.171828+0000","last_peered":"2024-03-21T10:35:12.171828+0000","last_clean":"2024-03-21T10:35:12.171828+0000","last_became_active":"2024-03-21T10:35:06.907432+0000","last_became_peered":"2024-03-21T10:35:06.907432+0000","last_unstale":"2024-03-21T10:35:12.171828+0000","last_undegraded":"2024-03-21T10:35:12.171828+0000","last_fullsized":"2024-03-21T10:35:12.171828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:31:02.528855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.167065+0000","last_change":"2024-03-21T10:35:06.908597+0000","last_active":"2024-03-21T10:35:12.167065+0000","last_peered":"2024-03-21T10:35:12.167065+0000","last_clean":"2024-03-21T10:35:12.167065+0000","last_became_active":"2024-03-21T10:35:06.908282+0000","last_became_peered":"2024-03-21T10:35:06.908282+0000","last_unstale":"2024-03-21T10:35:12.167065+0000","last_undegraded":"2024-03-21T10:35:12.167065+0000","last_fullsized":"2024-03-21T10:35:12.167065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:42:18.221045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.184741+0000","last_change":"2024-03-21T10:35:06.931015+0000","last_active":"2024-03-21T10:35:12.184741+0000","last_peered":"2024-03-21T10:35:12.184741+0000","last_clean":"2024-03-21T10:35:12.184741+0000","last_became_active":"2024-03-21T10:35:06.930209+0000","last_became_peered":"2024-03-21T10:35:06.930209+0000","last_unstale":"2024-03-21T10:35:12.184741+0000","last_undegraded":"2024-03-21T10:35:12.184741+0000","last_fullsized":"2024-03-21T10:35:12.184741+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:42:15.277874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171191+0000","last_change":"2024-03-21T10:35:06.930070+0000","last_active":"2024-03-21T10:35:12.171191+0000","last_peered":"2024-03-21T10:35:12.171191+0000","last_clean":"2024-03-21T10:35:12.171191+0000","last_became_active":"2024-03-21T10:35:06.929773+0000","last_became_peered":"2024-03-21T10:35:06.929773+0000","last_unstale":"2024-03-21T10:35:12.171191+0000","last_undegraded":"2024-03-21T10:35:12.171191+0000","last_fullsized":"2024-03-21T10:35:12.171191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:17:25.176666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.177677+0000","last_change":"2024-03-21T10:35:06.929575+0000","last_active":"2024-03-21T10:35:12.177677+0000","last_peered":"2024-03-21T10:35:12.177677+0000","last_clean":"2024-03-21T10:35:12.177677+0000","last_became_active":"2024-03-21T10:35:06.929240+0000","last_became_peered":"2024-03-21T10:35:06.929240+0000","last_unstale":"2024-03-21T10:35:12.177677+0000","last_undegraded":"2024-03-21T10:35:12.177677+0000","last_fullsized":"2024-03-21T10:35:12.177677+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:03:40.697640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.170351+0000","last_change":"2024-03-21T10:35:06.934693+0000","last_active":"2024-03-21T10:35:12.170351+0000","last_peered":"2024-03-21T10:35:12.170351+0000","last_clean":"2024-03-21T10:35:12.170351+0000","last_became_active":"2024-03-21T10:35:06.934361+0000","last_became_peered":"2024-03-21T10:35:06.934361+0000","last_unstale":"2024-03-21T10:35:12.170351+0000","last_undegraded":"2024-03-21T10:35:12.170351+0000","last_fullsized":"2024-03-21T10:35:12.170351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:14:00.954370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.181074+0000","last_change":"2024-03-21T10:35:06.941095+0000","last_active":"2024-03-21T10:35:12.181074+0000","last_peered":"2024-03-21T10:35:12.181074+0000","last_clean":"2024-03-21T10:35:12.181074+0000","last_became_active":"2024-03-21T10:35:06.939348+0000","last_became_peered":"2024-03-21T10:35:06.939348+0000","last_unstale":"2024-03-21T10:35:12.181074+0000","last_undegraded":"2024-03-21T10:35:12.181074+0000","last_fullsized":"2024-03-21T10:35:12.181074+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:31:43.783592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-21T10:35:12.171853+0000","last_change":"2024-03-21T10:35:06.908340+0000","last_active":"2024-03-21T10:35:12.171853+0000","last_peered":"2024-03-21T10:35:12.171853+0000","last_clean":"2024-03-21T10:35:12.171853+0000","last_became_active":"2024-03-21T10:35:06.908043+0000","last_became_peered":"2024-03-21T10:35:06.908043+0000","last_unstale":"2024-03-21T10:35:12.171853+0000","last_undegraded":"2024-03-21T10:35:12.171853+0000","last_fullsized":"2024-03-21T10:35:12.171853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:50:33.167396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282863,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4954,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":96,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":19229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383702,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":970752,"data_stored":481893,"data_compressed":14696,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645338,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":1073003,"data_compressed":50670,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808863,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":477999,"data_compressed":14696,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070503,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":983040,"data_stored":477953,"data_compressed":14696,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332140,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1032192,"data_stored":1073862,"data_compressed":50670,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495666,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14696,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":481057,"data_compressed":14696,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738452,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":966656,"data_stored":477305,"data_compressed":14696,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":481091,"data_compressed":14696,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182466,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":477499,"data_compressed":14696,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920828,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":477237,"data_compressed":14696,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757303,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1024000,"data_stored":1067765,"data_compressed":50670,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34331,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34331,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34331,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T10:35:45.859 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-21T10:35:45.860 INFO:tasks.cephadm:Setting up client nodes... 2024-03-21T10:35:45.860 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-21T10:35:46.393 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:35:46.590 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:46 smithi031 ceph-mon[31673]: from='client.24881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T10:35:46.590 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:46 smithi031 ceph-mon[31673]: from='client.24877 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T10:35:46.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:46 smithi040 ceph-mon[36154]: from='client.24881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T10:35:46.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:46 smithi040 ceph-mon[36154]: from='client.24877 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T10:35:46.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:46 smithi070 ceph-mon[36330]: from='client.24881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T10:35:46.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:46 smithi070 ceph-mon[36330]: from='client.24877 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T10:35:47.601 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:47 smithi031 ceph-mon[31673]: pgmap v345: 129 pgs: 129 active+clean; 622 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 6.1 KiB/s wr, 12 op/s 2024-03-21T10:35:47.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:47 smithi040 ceph-mon[36154]: pgmap v345: 129 pgs: 129 active+clean; 622 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 6.1 KiB/s wr, 12 op/s 2024-03-21T10:35:47.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:47 smithi070 ceph-mon[36330]: pgmap v345: 129 pgs: 129 active+clean; 622 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 6.1 KiB/s wr, 12 op/s 2024-03-21T10:35:48.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.728+0000 7f06f1a00700 1 -- 172.21.15.31:0/3009408050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06ec10a0c0 msgr2=0x7f06ec10c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:48.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.728+0000 7f06f1a00700 1 --2- 172.21.15.31:0/3009408050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06ec10a0c0 0x7f06ec10c550 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f06e400b0a0 tx=0x7f06e400b3b0 comp rx=0 tx=0).stop 2024-03-21T10:35:48.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.728+0000 7f06f1a00700 1 -- 172.21.15.31:0/3009408050 shutdown_connections 2024-03-21T10:35:48.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.728+0000 7f06f1a00700 1 --2- 172.21.15.31:0/3009408050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06ec10a0c0 0x7f06ec10c550 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:48.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.728+0000 7f06f1a00700 1 --2- 172.21.15.31:0/3009408050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06ec0761c0 0x7f06ec074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:48.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.728+0000 7f06f1a00700 1 --2- 172.21.15.31:0/3009408050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06ec074bf0 0x7f06ec075070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:48.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.728+0000 7f06f1a00700 1 -- 172.21.15.31:0/3009408050 >> 172.21.15.31:0/3009408050 conn(0x7f06ec0fd770 msgr2=0x7f06ec0ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:48.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.728+0000 7f06f1a00700 1 -- 172.21.15.31:0/3009408050 shutdown_connections 2024-03-21T10:35:48.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.729+0000 7f06f1a00700 1 -- 172.21.15.31:0/3009408050 wait complete. 2024-03-21T10:35:48.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.730+0000 7f06f1a00700 1 Processor -- start 2024-03-21T10:35:48.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.730+0000 7f06f1a00700 1 -- start start 2024-03-21T10:35:48.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.730+0000 7f06f1a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06ec074bf0 0x7f06ec19a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:48.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.730+0000 7f06f1a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06ec0761c0 0x7f06ec19aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:48.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.730+0000 7f06f1a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06ec10a0c0 0x7f06ec1a1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:48.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.730+0000 7f06f1a00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec110020 con 0x7f06ec074bf0 2024-03-21T10:35:48.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.730+0000 7f06f1a00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec10fea0 con 0x7f06ec0761c0 2024-03-21T10:35:48.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.730+0000 7f06f1a00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec1101a0 con 0x7f06ec10a0c0 2024-03-21T10:35:48.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.731+0000 7f06eb400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06ec10a0c0 0x7f06ec1a1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:48.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.731+0000 7f06eb400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06ec10a0c0 0x7f06ec1a1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:46932/0 (socket says 172.21.15.31:46932) 2024-03-21T10:35:48.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.731+0000 7f06eb400700 1 -- 172.21.15.31:0/2462930775 learned_addr learned my addr 172.21.15.31:0/2462930775 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:48.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.731+0000 7f06ea000700 1 --2- 172.21.15.31:0/2462930775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06ec0761c0 0x7f06ec19aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:48.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.731+0000 7f06eaa00700 1 --2- 172.21.15.31:0/2462930775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06ec074bf0 0x7f06ec19a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:48.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.731+0000 7f06ea000700 1 -- 172.21.15.31:0/2462930775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06ec10a0c0 msgr2=0x7f06ec1a1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:48.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.731+0000 7f06ea000700 1 --2- 172.21.15.31:0/2462930775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06ec10a0c0 0x7f06ec1a1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:48.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.731+0000 7f06ea000700 1 -- 172.21.15.31:0/2462930775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06ec074bf0 msgr2=0x7f06ec19a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:48.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.731+0000 7f06ea000700 1 --2- 172.21.15.31:0/2462930775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06ec074bf0 0x7f06ec19a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:48.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.731+0000 7f06ea000700 1 -- 172.21.15.31:0/2462930775 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ec1a2170 con 0x7f06ec0761c0 2024-03-21T10:35:48.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.732+0000 7f06eaa00700 1 --2- 172.21.15.31:0/2462930775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06ec074bf0 0x7f06ec19a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:35:48.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.732+0000 7f06ea000700 1 --2- 172.21.15.31:0/2462930775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06ec0761c0 0x7f06ec19aa50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f06d800e9c0 tx=0x7f06d800ed80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:48.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.732+0000 7f06e3400700 1 -- 172.21.15.31:0/2462930775 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d800cbd0 con 0x7f06ec0761c0 2024-03-21T10:35:48.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.732+0000 7f06e3400700 1 -- 172.21.15.31:0/2462930775 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f06d8004500 con 0x7f06ec0761c0 2024-03-21T10:35:48.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.732+0000 7f06e3400700 1 -- 172.21.15.31:0/2462930775 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d8010430 con 0x7f06ec0761c0 2024-03-21T10:35:48.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.732+0000 7f06f1a00700 1 -- 172.21.15.31:0/2462930775 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06ec1a2460 con 0x7f06ec0761c0 2024-03-21T10:35:48.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.732+0000 7f06f1a00700 1 -- 172.21.15.31:0/2462930775 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06ec1a2950 con 0x7f06ec0761c0 2024-03-21T10:35:48.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.734+0000 7f06f1a00700 1 -- 172.21.15.31:0/2462930775 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06ec0506a0 con 0x7f06ec0761c0 2024-03-21T10:35:48.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.736+0000 7f06e3400700 1 -- 172.21.15.31:0/2462930775 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7f06d8004020 con 0x7f06ec0761c0 2024-03-21T10:35:48.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.738+0000 7f06e3400700 1 --2- 172.21.15.31:0/2462930775 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f06d00a3170 0x7f06d00a5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:48.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.738+0000 7f06e3400700 1 -- 172.21.15.31:0/2462930775 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f06d80cfaa0 con 0x7f06ec0761c0 2024-03-21T10:35:48.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.739+0000 7f06eaa00700 1 --2- 172.21.15.31:0/2462930775 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f06d00a3170 0x7f06d00a5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:48.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.739+0000 7f06eaa00700 1 --2- 172.21.15.31:0/2462930775 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f06d00a3170 0x7f06d00a5630 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f06ec0765a0 tx=0x7f06dc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:48.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:48.746+0000 7f06e3400700 1 -- 172.21.15.31:0/2462930775 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f06d809c4d0 con 0x7f06ec0761c0 2024-03-21T10:35:49.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:49.082+0000 7f06f1a00700 1 -- 172.21.15.31:0/2462930775 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f06ec02dad0 con 0x7f06ec0761c0 2024-03-21T10:35:49.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:49.089+0000 7f06e3400700 1 -- 172.21.15.31:0/2462930775 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7f06ec02dad0 con 0x7f06ec0761c0 2024-03-21T10:35:49.088 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2024-03-21T10:35:49.088 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCFDfxlvqYOBRAAvvswJERHxKt1UQAEYk/xNg== 2024-03-21T10:35:49.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:49.092+0000 7f06f1a00700 1 -- 172.21.15.31:0/2462930775 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f06d00a3170 msgr2=0x7f06d00a5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:49.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:49.092+0000 7f06f1a00700 1 --2- 172.21.15.31:0/2462930775 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f06d00a3170 0x7f06d00a5630 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f06ec0765a0 tx=0x7f06dc009260 comp rx=0 tx=0).stop 2024-03-21T10:35:49.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:49.092+0000 7f06f1a00700 1 -- 172.21.15.31:0/2462930775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06ec0761c0 msgr2=0x7f06ec19aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:49.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:49.092+0000 7f06f1a00700 1 --2- 172.21.15.31:0/2462930775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06ec0761c0 0x7f06ec19aa50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f06d800e9c0 tx=0x7f06d800ed80 comp rx=0 tx=0).stop 2024-03-21T10:35:49.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:49.093+0000 7f06f1a00700 1 -- 172.21.15.31:0/2462930775 shutdown_connections 2024-03-21T10:35:49.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:49.093+0000 7f06f1a00700 1 --2- 172.21.15.31:0/2462930775 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f06d00a3170 0x7f06d00a5630 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:49.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:49.093+0000 7f06f1a00700 1 --2- 172.21.15.31:0/2462930775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06ec10a0c0 0x7f06ec1a1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:49.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:49.093+0000 7f06f1a00700 1 --2- 172.21.15.31:0/2462930775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06ec074bf0 0x7f06ec19a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:49.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:49.093+0000 7f06f1a00700 1 --2- 172.21.15.31:0/2462930775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06ec0761c0 0x7f06ec19aa50 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:49.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:49.093+0000 7f06f1a00700 1 -- 172.21.15.31:0/2462930775 >> 172.21.15.31:0/2462930775 conn(0x7f06ec0fd770 msgr2=0x7f06ec105fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:49.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:49.093+0000 7f06f1a00700 1 -- 172.21.15.31:0/2462930775 shutdown_connections 2024-03-21T10:35:49.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:49.093+0000 7f06f1a00700 1 -- 172.21.15.31:0/2462930775 wait complete. 2024-03-21T10:35:49.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:49 smithi040 ceph-mon[36154]: pgmap v346: 129 pgs: 129 active+clean; 622 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 6.1 KiB/s wr, 12 op/s 2024-03-21T10:35:49.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:49 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2462930775' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T10:35:49.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:49 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T10:35:49.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:49 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T10:35:49.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:49 smithi031 ceph-mon[31673]: pgmap v346: 129 pgs: 129 active+clean; 622 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 6.1 KiB/s wr, 12 op/s 2024-03-21T10:35:49.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:49 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2462930775' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T10:35:49.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:49 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T10:35:49.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:49 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T10:35:49.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:49 smithi070 ceph-mon[36330]: pgmap v346: 129 pgs: 129 active+clean; 622 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 6.1 KiB/s wr, 12 op/s 2024-03-21T10:35:49.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:49 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2462930775' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T10:35:49.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:49 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T10:35:49.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:49 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T10:35:49.896 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-21T10:35:49.896 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-21T10:35:49.896 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-21T10:35:50.038 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-21T10:35:50.039 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-21T10:35:50.039 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-21T10:35:50.525 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:35:51.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:51 smithi031 ceph-mon[31673]: pgmap v347: 129 pgs: 129 active+clean; 622 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.2 KiB/s wr, 14 op/s 2024-03-21T10:35:51.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:51 smithi040 ceph-mon[36154]: pgmap v347: 129 pgs: 129 active+clean; 622 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.2 KiB/s wr, 14 op/s 2024-03-21T10:35:51.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:51 smithi070 ceph-mon[36330]: pgmap v347: 129 pgs: 129 active+clean; 622 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.2 KiB/s wr, 14 op/s 2024-03-21T10:35:52.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.889+0000 7f53dd000700 1 -- 172.21.15.31:0/1000759752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53d8104f90 msgr2=0x7f53d8108fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:52.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.889+0000 7f53dd000700 1 --2- 172.21.15.31:0/1000759752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53d8104f90 0x7f53d8108fe0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f53c4009990 tx=0x7f53c4009ca0 comp rx=0 tx=0).stop 2024-03-21T10:35:52.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.890+0000 7f53dd000700 1 -- 172.21.15.31:0/1000759752 shutdown_connections 2024-03-21T10:35:52.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.890+0000 7f53dd000700 1 --2- 172.21.15.31:0/1000759752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53d81096c0 0x7f53d8111290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:52.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.890+0000 7f53dd000700 1 --2- 172.21.15.31:0/1000759752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53d81045e0 0x7f53d81049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:52.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.890+0000 7f53dd000700 1 --2- 172.21.15.31:0/1000759752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53d8104f90 0x7f53d8108fe0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:52.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.890+0000 7f53dd000700 1 -- 172.21.15.31:0/1000759752 >> 172.21.15.31:0/1000759752 conn(0x7f53d80ffe90 msgr2=0x7f53d81022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:52.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.890+0000 7f53dd000700 1 -- 172.21.15.31:0/1000759752 shutdown_connections 2024-03-21T10:35:52.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.890+0000 7f53dd000700 1 -- 172.21.15.31:0/1000759752 wait complete. 2024-03-21T10:35:52.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.892+0000 7f53dd000700 1 Processor -- start 2024-03-21T10:35:52.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.892+0000 7f53dd000700 1 -- start start 2024-03-21T10:35:52.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.893+0000 7f53dd000700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53d81045e0 0x7f53d81a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:52.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.893+0000 7f53dd000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53d8104f90 0x7f53d81a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:52.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.893+0000 7f53dd000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53d81096c0 0x7f53d819a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:52.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.893+0000 7f53dd000700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d81146d0 con 0x7f53d81045e0 2024-03-21T10:35:52.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.893+0000 7f53dd000700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d8114550 con 0x7f53d81096c0 2024-03-21T10:35:52.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.893+0000 7f53dd000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d8114850 con 0x7f53d8104f90 2024-03-21T10:35:52.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.894+0000 7f53d5400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53d8104f90 0x7f53d81a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:52.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.894+0000 7f53d5e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53d81045e0 0x7f53d81a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:52.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.894+0000 7f53d5400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53d8104f90 0x7f53d81a0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:46956/0 (socket says 172.21.15.31:46956) 2024-03-21T10:35:52.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.894+0000 7f53d5400700 1 -- 172.21.15.31:0/630776778 learned_addr learned my addr 172.21.15.31:0/630776778 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:52.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.894+0000 7f53d6800700 1 --2- 172.21.15.31:0/630776778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53d81096c0 0x7f53d819a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:52.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.894+0000 7f53d5400700 1 -- 172.21.15.31:0/630776778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53d81096c0 msgr2=0x7f53d819a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:52.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.894+0000 7f53d5400700 1 --2- 172.21.15.31:0/630776778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53d81096c0 0x7f53d819a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:52.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.894+0000 7f53d5400700 1 -- 172.21.15.31:0/630776778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53d81045e0 msgr2=0x7f53d81a0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:52.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.894+0000 7f53d5400700 1 --2- 172.21.15.31:0/630776778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53d81045e0 0x7f53d81a0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:52.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.894+0000 7f53d5400700 1 -- 172.21.15.31:0/630776778 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d819b130 con 0x7f53d8104f90 2024-03-21T10:35:52.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.894+0000 7f53d6800700 1 --2- 172.21.15.31:0/630776778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53d81096c0 0x7f53d819a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:35:52.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.894+0000 7f53d5400700 1 --2- 172.21.15.31:0/630776778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53d8104f90 0x7f53d81a0d50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f53c4000c00 tx=0x7f53c4005590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:52.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.895+0000 7f53cea00700 1 -- 172.21.15.31:0/630776778 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c4027070 con 0x7f53d8104f90 2024-03-21T10:35:52.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.895+0000 7f53cea00700 1 -- 172.21.15.31:0/630776778 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f53c40056b0 con 0x7f53d8104f90 2024-03-21T10:35:52.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.895+0000 7f53dd000700 1 -- 172.21.15.31:0/630776778 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53d819b3c0 con 0x7f53d8104f90 2024-03-21T10:35:52.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.895+0000 7f53cea00700 1 -- 172.21.15.31:0/630776778 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c40173f0 con 0x7f53d8104f90 2024-03-21T10:35:52.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.896+0000 7f53dd000700 1 -- 172.21.15.31:0/630776778 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53d81a76e0 con 0x7f53d8104f90 2024-03-21T10:35:52.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.897+0000 7f53dd000700 1 -- 172.21.15.31:0/630776778 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53d8068e50 con 0x7f53d8104f90 2024-03-21T10:35:52.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.900+0000 7f53cea00700 1 -- 172.21.15.31:0/630776778 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7f53c402a070 con 0x7f53d8104f90 2024-03-21T10:35:52.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.901+0000 7f53cea00700 1 --2- 172.21.15.31:0/630776778 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f53bc0a30c0 0x7f53bc0a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:52.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.902+0000 7f53cea00700 1 -- 172.21.15.31:0/630776778 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f53c40d0070 con 0x7f53d8104f90 2024-03-21T10:35:52.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.902+0000 7f53d5e00700 1 --2- 172.21.15.31:0/630776778 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f53bc0a30c0 0x7f53bc0a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:52.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.903+0000 7f53d5e00700 1 --2- 172.21.15.31:0/630776778 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f53bc0a30c0 0x7f53bc0a5580 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f53d819bb10 tx=0x7f53c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:52.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:52.905+0000 7f53cea00700 1 -- 172.21.15.31:0/630776778 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f53c4013070 con 0x7f53d8104f90 2024-03-21T10:35:53.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:53.240+0000 7f53dd000700 1 -- 172.21.15.31:0/630776778 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f53d802da30 con 0x7f53d8104f90 2024-03-21T10:35:53.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:53.245+0000 7f53cea00700 1 -- 172.21.15.31:0/630776778 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+263552 (secure 0 0 0) 0x7f53c409cb30 con 0x7f53d8104f90 2024-03-21T10:35:53.248 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:35:53.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:53.251+0000 7f53dd000700 1 -- 172.21.15.31:0/630776778 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f53bc0a30c0 msgr2=0x7f53bc0a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:53.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:53.251+0000 7f53dd000700 1 --2- 172.21.15.31:0/630776778 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f53bc0a30c0 0x7f53bc0a5580 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f53d819bb10 tx=0x7f53c8009260 comp rx=0 tx=0).stop 2024-03-21T10:35:53.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:53.251+0000 7f53dd000700 1 -- 172.21.15.31:0/630776778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53d8104f90 msgr2=0x7f53d81a0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:53.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:53.251+0000 7f53dd000700 1 --2- 172.21.15.31:0/630776778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53d8104f90 0x7f53d81a0d50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f53c4000c00 tx=0x7f53c4005590 comp rx=0 tx=0).stop 2024-03-21T10:35:53.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:53.252+0000 7f53dd000700 1 -- 172.21.15.31:0/630776778 shutdown_connections 2024-03-21T10:35:53.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:53.252+0000 7f53dd000700 1 --2- 172.21.15.31:0/630776778 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f53bc0a30c0 0x7f53bc0a5580 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:53.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:53.252+0000 7f53dd000700 1 --2- 172.21.15.31:0/630776778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53d81096c0 0x7f53d819a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:53.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:53.252+0000 7f53dd000700 1 --2- 172.21.15.31:0/630776778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53d81045e0 0x7f53d81a0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:53.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:53.252+0000 7f53dd000700 1 --2- 172.21.15.31:0/630776778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53d8104f90 0x7f53d81a0d50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:53.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:53.252+0000 7f53dd000700 1 -- 172.21.15.31:0/630776778 >> 172.21.15.31:0/630776778 conn(0x7f53d80ffe90 msgr2=0x7f53d8102180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:53.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:53.252+0000 7f53dd000700 1 -- 172.21.15.31:0/630776778 shutdown_connections 2024-03-21T10:35:53.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:53.252+0000 7f53dd000700 1 -- 172.21.15.31:0/630776778 wait complete. 2024-03-21T10:35:53.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:53 smithi031 ceph-mon[31673]: pgmap v348: 129 pgs: 129 active+clean; 626 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 6.6 KiB/s wr, 15 op/s 2024-03-21T10:35:53.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:53 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/630776778' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T10:35:53.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:53 smithi040 ceph-mon[36154]: pgmap v348: 129 pgs: 129 active+clean; 626 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 6.6 KiB/s wr, 15 op/s 2024-03-21T10:35:53.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:53 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/630776778' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T10:35:53.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:53 smithi070 ceph-mon[36330]: pgmap v348: 129 pgs: 129 active+clean; 626 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 6.6 KiB/s wr, 15 op/s 2024-03-21T10:35:53.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:53 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/630776778' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T10:35:54.155 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3875791078},{"type":"v1","addr":"172.21.15.31:6801","nonce":3875791078}]},"active_addr":"172.21.15.31:6801/3875791078","active_change":"2024-03-21T10:26:53.842854+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.31:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":3633519331}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":3488063077}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":2171929076}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":1729503305}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":1088356945}]}]} 2024-03-21T10:35:54.160 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-21T10:35:54.160 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-21T10:35:54.160 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-21T10:35:54.667 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:35:54.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:54 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-21T10:35:54.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:54 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-03-21T10:35:54.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:54 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-21T10:35:54.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:54 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]: dispatch 2024-03-21T10:35:54.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:54 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-03-21T10:35:54.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:54 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9, 4, 6]}]: dispatch 2024-03-21T10:35:54.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:54 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 10]}]: dispatch 2024-03-21T10:35:54.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:54 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-03-21T10:35:54.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:54 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-03-21T10:35:54.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:54 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-21T10:35:54.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:54 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-03-21T10:35:54.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:54 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-21T10:35:54.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:54 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]: dispatch 2024-03-21T10:35:54.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:54 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-03-21T10:35:54.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:54 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9, 4, 6]}]: dispatch 2024-03-21T10:35:54.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:54 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 10]}]: dispatch 2024-03-21T10:35:54.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:54 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-03-21T10:35:54.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:54 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-03-21T10:35:54.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:54 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-21T10:35:54.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:54 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-03-21T10:35:54.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:54 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-21T10:35:54.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:54 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]: dispatch 2024-03-21T10:35:54.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:54 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-03-21T10:35:54.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:54 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9, 4, 6]}]: dispatch 2024-03-21T10:35:54.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:54 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 10]}]: dispatch 2024-03-21T10:35:54.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:54 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-03-21T10:35:54.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:54 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-03-21T10:35:55.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:55 smithi040 ceph-mon[36154]: pgmap v349: 129 pgs: 129 active+clean; 626 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 5.9 KiB/s wr, 13 op/s 2024-03-21T10:35:55.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:55 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-21T10:35:55.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:55 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-03-21T10:35:55.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:55 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-21T10:35:55.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:55 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]': finished 2024-03-21T10:35:55.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:55 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-03-21T10:35:55.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:55 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9, 4, 6]}]': finished 2024-03-21T10:35:55.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:55 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 10]}]': finished 2024-03-21T10:35:55.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:55 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-03-21T10:35:55.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:55 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-03-21T10:35:55.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:55 smithi040 ceph-mon[36154]: osdmap e90: 12 total, 12 up, 12 in 2024-03-21T10:35:55.768 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:55 smithi031 ceph-mon[31673]: pgmap v349: 129 pgs: 129 active+clean; 626 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 5.9 KiB/s wr, 13 op/s 2024-03-21T10:35:55.768 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:55 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-21T10:35:55.768 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:55 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-03-21T10:35:55.768 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:55 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-21T10:35:55.768 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:55 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]': finished 2024-03-21T10:35:55.768 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:55 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-03-21T10:35:55.768 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:55 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9, 4, 6]}]': finished 2024-03-21T10:35:55.768 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:55 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 10]}]': finished 2024-03-21T10:35:55.768 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:55 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-03-21T10:35:55.769 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:55 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-03-21T10:35:55.769 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:55 smithi031 ceph-mon[31673]: osdmap e90: 12 total, 12 up, 12 in 2024-03-21T10:35:55.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:55 smithi070 ceph-mon[36330]: pgmap v349: 129 pgs: 129 active+clean; 626 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 5.9 KiB/s wr, 13 op/s 2024-03-21T10:35:55.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:55 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-21T10:35:55.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:55 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-03-21T10:35:55.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:55 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-21T10:35:55.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:55 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]': finished 2024-03-21T10:35:55.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:55 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-03-21T10:35:55.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:55 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9, 4, 6]}]': finished 2024-03-21T10:35:55.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:55 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 10]}]': finished 2024-03-21T10:35:55.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:55 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-03-21T10:35:55.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:55 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-03-21T10:35:55.825 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:55 smithi070 ceph-mon[36330]: osdmap e90: 12 total, 12 up, 12 in 2024-03-21T10:35:56.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:56 smithi031 ceph-mon[31673]: osdmap e91: 12 total, 12 up, 12 in 2024-03-21T10:35:56.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:56 smithi040 ceph-mon[36154]: osdmap e91: 12 total, 12 up, 12 in 2024-03-21T10:35:56.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:56 smithi070 ceph-mon[36330]: osdmap e91: 12 total, 12 up, 12 in 2024-03-21T10:35:57.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.049+0000 7fab4be00700 1 -- 172.21.15.31:0/762178720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab4c1073b0 msgr2=0x7fab4c10ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:57.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.049+0000 7fab4be00700 1 --2- 172.21.15.31:0/762178720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab4c1073b0 0x7fab4c10ef80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fab44009fc0 tx=0x7fab4400b330 comp rx=0 tx=0).stop 2024-03-21T10:35:57.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.049+0000 7fab4be00700 1 -- 172.21.15.31:0/762178720 shutdown_connections 2024-03-21T10:35:57.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.049+0000 7fab4be00700 1 --2- 172.21.15.31:0/762178720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab4c1073b0 0x7fab4c10ef80 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:57.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.049+0000 7fab4be00700 1 --2- 172.21.15.31:0/762178720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab4c1022d0 0x7fab4c1026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:57.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.049+0000 7fab4be00700 1 --2- 172.21.15.31:0/762178720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab4c102c80 0x7fab4c106cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:57.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.049+0000 7fab4be00700 1 -- 172.21.15.31:0/762178720 >> 172.21.15.31:0/762178720 conn(0x7fab4c0fdb60 msgr2=0x7fab4c0fff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:57.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.050+0000 7fab4be00700 1 -- 172.21.15.31:0/762178720 shutdown_connections 2024-03-21T10:35:57.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.050+0000 7fab4be00700 1 -- 172.21.15.31:0/762178720 wait complete. 2024-03-21T10:35:57.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.051+0000 7fab4be00700 1 Processor -- start 2024-03-21T10:35:57.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.051+0000 7fab4be00700 1 -- start start 2024-03-21T10:35:57.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.052+0000 7fab4be00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab4c1022d0 0x7fab4c10e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:57.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.052+0000 7fab4be00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab4c102c80 0x7fab4c10e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:57.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.052+0000 7fab4be00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab4c1073b0 0x7fab4c1a36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:57.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.052+0000 7fab4be00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c1123f0 con 0x7fab4c1073b0 2024-03-21T10:35:57.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.052+0000 7fab4be00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c112270 con 0x7fab4c102c80 2024-03-21T10:35:57.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.052+0000 7fab4be00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c112570 con 0x7fab4c1022d0 2024-03-21T10:35:57.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.052+0000 7fab4aa00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab4c1022d0 0x7fab4c10e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:57.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.052+0000 7fab4aa00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab4c1022d0 0x7fab4c10e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:39530/0 (socket says 172.21.15.31:39530) 2024-03-21T10:35:57.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.052+0000 7fab4aa00700 1 -- 172.21.15.31:0/1627080230 learned_addr learned my addr 172.21.15.31:0/1627080230 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:35:57.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.052+0000 7fab43e00700 1 --2- 172.21.15.31:0/1627080230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab4c102c80 0x7fab4c10e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:57.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.052+0000 7fab4b400700 1 --2- 172.21.15.31:0/1627080230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab4c1073b0 0x7fab4c1a36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:57.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.052+0000 7fab4aa00700 1 -- 172.21.15.31:0/1627080230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab4c102c80 msgr2=0x7fab4c10e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:57.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.052+0000 7fab4aa00700 1 --2- 172.21.15.31:0/1627080230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab4c102c80 0x7fab4c10e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:57.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.052+0000 7fab4aa00700 1 -- 172.21.15.31:0/1627080230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab4c1073b0 msgr2=0x7fab4c1a36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:57.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.052+0000 7fab4aa00700 1 --2- 172.21.15.31:0/1627080230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab4c1073b0 0x7fab4c1a36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:57.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.052+0000 7fab4aa00700 1 -- 172.21.15.31:0/1627080230 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4c1a3df0 con 0x7fab4c1022d0 2024-03-21T10:35:57.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.052+0000 7fab4b400700 1 --2- 172.21.15.31:0/1627080230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab4c1073b0 0x7fab4c1a36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:35:57.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.053+0000 7fab4aa00700 1 --2- 172.21.15.31:0/1627080230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab4c1022d0 0x7fab4c10e220 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fab3c00e9a0 tx=0x7fab3c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:57.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.053+0000 7fab43400700 1 -- 172.21.15.31:0/1627080230 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c00cbd0 con 0x7fab4c1022d0 2024-03-21T10:35:57.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.053+0000 7fab43400700 1 -- 172.21.15.31:0/1627080230 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab3c00cd70 con 0x7fab4c1022d0 2024-03-21T10:35:57.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.053+0000 7fab4be00700 1 -- 172.21.15.31:0/1627080230 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab4c1a40e0 con 0x7fab4c1022d0 2024-03-21T10:35:57.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.053+0000 7fab43400700 1 -- 172.21.15.31:0/1627080230 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c018750 con 0x7fab4c1022d0 2024-03-21T10:35:57.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.053+0000 7fab4be00700 1 -- 172.21.15.31:0/1627080230 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab4c1ab960 con 0x7fab4c1022d0 2024-03-21T10:35:57.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.055+0000 7fab4be00700 1 -- 172.21.15.31:0/1627080230 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab4c068e50 con 0x7fab4c1022d0 2024-03-21T10:35:57.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.056+0000 7fab43400700 1 -- 172.21.15.31:0/1627080230 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7fab3c010430 con 0x7fab4c1022d0 2024-03-21T10:35:57.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.057+0000 7fab43400700 1 --2- 172.21.15.31:0/1627080230 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fab300a30c0 0x7fab300a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:35:57.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.057+0000 7fab43400700 1 -- 172.21.15.31:0/1627080230 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fab3c0cfb70 con 0x7fab4c1022d0 2024-03-21T10:35:57.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.058+0000 7fab43e00700 1 --2- 172.21.15.31:0/1627080230 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fab300a30c0 0x7fab300a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:35:57.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.058+0000 7fab43e00700 1 --2- 172.21.15.31:0/1627080230 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fab300a30c0 0x7fab300a5580 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fab34009750 tx=0x7fab34009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:35:57.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.061+0000 7fab43400700 1 -- 172.21.15.31:0/1627080230 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fab3c09c4a0 con 0x7fab4c1022d0 2024-03-21T10:35:57.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.286+0000 7fab4be00700 1 -- 172.21.15.31:0/1627080230 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab4c1abb90 con 0x7fab4c1022d0 2024-03-21T10:35:57.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.288+0000 7fab43400700 1 -- 172.21.15.31:0/1627080230 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24507 (secure 0 0 0) 0x7fab3c09bda0 con 0x7fab4c1022d0 2024-03-21T10:35:57.287 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:35:57.287 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":91,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","created":"2024-03-21T10:25:41.971117+0000","modified":"2024-03-21T10:35:55.370062+0000","last_up_change":"2024-03-21T10:34:07.757196+0000","last_in_change":"2024-03-21T10:33:47.460406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T10:29:44.553968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T10:35:04.945449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T10:35:05.970589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T10:35:09.688711+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e0244d33-00e2-47d5-9f66-34efc138c462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6803","nonce":3089885125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6805","nonce":3089885125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6809","nonce":3089885125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6807","nonce":3089885125}]},"public_addr":"172.21.15.31:6803/3089885125","cluster_addr":"172.21.15.31:6805/3089885125","heartbeat_back_addr":"172.21.15.31:6809/3089885125","heartbeat_front_addr":"172.21.15.31:6807/3089885125","state":["exists","up"]},{"osd":1,"uuid":"76a63504-beab-4354-a6c3-eebc3b16ad03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6801","nonce":618840037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6803","nonce":618840037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6807","nonce":618840037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6805","nonce":618840037}]},"public_addr":"172.21.15.40:6801/618840037","cluster_addr":"172.21.15.40:6803/618840037","heartbeat_back_addr":"172.21.15.40:6807/618840037","heartbeat_front_addr":"172.21.15.40:6805/618840037","state":["exists","up"]},{"osd":2,"uuid":"fa191bd6-9a21-4ee2-8adf-8a63c9cfd821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6801","nonce":2524184117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6803","nonce":2524184117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6807","nonce":2524184117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6805","nonce":2524184117}]},"public_addr":"172.21.15.70:6801/2524184117","cluster_addr":"172.21.15.70:6803/2524184117","heartbeat_back_addr":"172.21.15.70:6807/2524184117","heartbeat_front_addr":"172.21.15.70:6805/2524184117","state":["exists","up"]},{"osd":3,"uuid":"edb44507-ffb0-42a4-827b-08461e1bf952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6811","nonce":2823036296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6813","nonce":2823036296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6817","nonce":2823036296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6815","nonce":2823036296}]},"public_addr":"172.21.15.31:6811/2823036296","cluster_addr":"172.21.15.31:6813/2823036296","heartbeat_back_addr":"172.21.15.31:6817/2823036296","heartbeat_front_addr":"172.21.15.31:6815/2823036296","state":["exists","up"]},{"osd":4,"uuid":"8cb89153-8c3b-431f-b57c-8863fada76dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6809","nonce":4215186309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6811","nonce":4215186309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6815","nonce":4215186309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6813","nonce":4215186309}]},"public_addr":"172.21.15.40:6809/4215186309","cluster_addr":"172.21.15.40:6811/4215186309","heartbeat_back_addr":"172.21.15.40:6815/4215186309","heartbeat_front_addr":"172.21.15.40:6813/4215186309","state":["exists","up"]},{"osd":5,"uuid":"9409505a-d463-408c-a3cd-218380c04d68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6809","nonce":3402830452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6811","nonce":3402830452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6815","nonce":3402830452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6813","nonce":3402830452}]},"public_addr":"172.21.15.70:6809/3402830452","cluster_addr":"172.21.15.70:6811/3402830452","heartbeat_back_addr":"172.21.15.70:6815/3402830452","heartbeat_front_addr":"172.21.15.70:6813/3402830452","state":["exists","up"]},{"osd":6,"uuid":"f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6819","nonce":1193497058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6821","nonce":1193497058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6825","nonce":1193497058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6823","nonce":1193497058}]},"public_addr":"172.21.15.31:6819/1193497058","cluster_addr":"172.21.15.31:6821/1193497058","heartbeat_back_addr":"172.21.15.31:6825/1193497058","heartbeat_front_addr":"172.21.15.31:6823/1193497058","state":["exists","up"]},{"osd":7,"uuid":"db228a87-29a5-45d4-8fe6-a0a554262eb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6817","nonce":3825628119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6819","nonce":3825628119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6823","nonce":3825628119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6821","nonce":3825628119}]},"public_addr":"172.21.15.40:6817/3825628119","cluster_addr":"172.21.15.40:6819/3825628119","heartbeat_back_addr":"172.21.15.40:6823/3825628119","heartbeat_front_addr":"172.21.15.40:6821/3825628119","state":["exists","up"]},{"osd":8,"uuid":"59b9fde0-095d-4e0b-87da-ead37bd8ff1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6817","nonce":534880586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6819","nonce":534880586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6823","nonce":534880586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6821","nonce":534880586}]},"public_addr":"172.21.15.70:6817/534880586","cluster_addr":"172.21.15.70:6819/534880586","heartbeat_back_addr":"172.21.15.70:6823/534880586","heartbeat_front_addr":"172.21.15.70:6821/534880586","state":["exists","up"]},{"osd":9,"uuid":"77f47061-0c5f-4159-b276-e814935d6c1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6827","nonce":587047469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6829","nonce":587047469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6833","nonce":587047469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6831","nonce":587047469}]},"public_addr":"172.21.15.31:6827/587047469","cluster_addr":"172.21.15.31:6829/587047469","heartbeat_back_addr":"172.21.15.31:6833/587047469","heartbeat_front_addr":"172.21.15.31:6831/587047469","state":["exists","up"]},{"osd":10,"uuid":"8853dc43-7bd1-4c93-8063-f7982fd7da38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6825","nonce":512108711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6827","nonce":512108711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6831","nonce":512108711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6829","nonce":512108711}]},"public_addr":"172.21.15.40:6825/512108711","cluster_addr":"172.21.15.40:6827/512108711","heartbeat_back_addr":"172.21.15.40:6831/512108711","heartbeat_front_addr":"172.21.15.40:6829/512108711","state":["exists","up"]},{"osd":11,"uuid":"84b7c2de-c424-45a6-9583-60097e372a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6825","nonce":540305718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6827","nonce":540305718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6831","nonce":540305718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6829","nonce":540305718}]},"public_addr":"172.21.15.70:6825/540305718","cluster_addr":"172.21.15.70:6827/540305718","heartbeat_back_addr":"172.21.15.70:6831/540305718","heartbeat_front_addr":"172.21.15.70:6829/540305718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:28:51.477358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:17.405045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:41.626268+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:15.339568+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:42.676996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:07.516999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:42.412273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:10.365041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:35.699120+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:11.733639+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:39.505442+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:34:05.449301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451930554":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/3342588843":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6801/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/1626448389":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/1910059351":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/344822770":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/3811368512":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/1981437651":"2024-03-22T10:26:16.959165+0000","172.21.15.31:6801/1981437651":"2024-03-22T10:26:16.959165+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T10:35:57.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.292+0000 7fab4be00700 1 -- 172.21.15.31:0/1627080230 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fab300a30c0 msgr2=0x7fab300a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:57.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.292+0000 7fab4be00700 1 --2- 172.21.15.31:0/1627080230 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fab300a30c0 0x7fab300a5580 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fab34009750 tx=0x7fab34009310 comp rx=0 tx=0).stop 2024-03-21T10:35:57.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.292+0000 7fab4be00700 1 -- 172.21.15.31:0/1627080230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab4c1022d0 msgr2=0x7fab4c10e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:35:57.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.292+0000 7fab4be00700 1 --2- 172.21.15.31:0/1627080230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab4c1022d0 0x7fab4c10e220 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fab3c00e9a0 tx=0x7fab3c00ecb0 comp rx=0 tx=0).stop 2024-03-21T10:35:57.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.292+0000 7fab4be00700 1 -- 172.21.15.31:0/1627080230 shutdown_connections 2024-03-21T10:35:57.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.292+0000 7fab4be00700 1 --2- 172.21.15.31:0/1627080230 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fab300a30c0 0x7fab300a5580 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:57.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.292+0000 7fab4be00700 1 --2- 172.21.15.31:0/1627080230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab4c1073b0 0x7fab4c1a36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:57.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.292+0000 7fab4be00700 1 --2- 172.21.15.31:0/1627080230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab4c1022d0 0x7fab4c10e220 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:57.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.292+0000 7fab4be00700 1 --2- 172.21.15.31:0/1627080230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab4c102c80 0x7fab4c10e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:35:57.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.292+0000 7fab4be00700 1 -- 172.21.15.31:0/1627080230 >> 172.21.15.31:0/1627080230 conn(0x7fab4c0fdb60 msgr2=0x7fab4c0fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:35:57.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.292+0000 7fab4be00700 1 -- 172.21.15.31:0/1627080230 shutdown_connections 2024-03-21T10:35:57.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:35:57.292+0000 7fab4be00700 1 -- 172.21.15.31:0/1627080230 wait complete. 2024-03-21T10:35:57.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:57 smithi040 ceph-mon[36154]: pgmap v352: 129 pgs: 4 peering, 1 active+recovery_wait+degraded, 124 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.6 KiB/s wr, 9 op/s; 1/194 objects degraded (0.515%) 2024-03-21T10:35:57.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:57 smithi040 ceph-mon[36154]: Health check failed: Degraded data redundancy: 1/194 objects degraded (0.515%), 1 pg degraded (PG_DEGRADED) 2024-03-21T10:35:57.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:57 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1627080230' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:57.786 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:57 smithi031 ceph-mon[31673]: pgmap v352: 129 pgs: 4 peering, 1 active+recovery_wait+degraded, 124 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.6 KiB/s wr, 9 op/s; 1/194 objects degraded (0.515%) 2024-03-21T10:35:57.786 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:57 smithi031 ceph-mon[31673]: Health check failed: Degraded data redundancy: 1/194 objects degraded (0.515%), 1 pg degraded (PG_DEGRADED) 2024-03-21T10:35:57.786 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:57 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1627080230' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:57.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:57 smithi070 ceph-mon[36330]: pgmap v352: 129 pgs: 4 peering, 1 active+recovery_wait+degraded, 124 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.6 KiB/s wr, 9 op/s; 1/194 objects degraded (0.515%) 2024-03-21T10:35:57.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:57 smithi070 ceph-mon[36330]: Health check failed: Degraded data redundancy: 1/194 objects degraded (0.515%), 1 pg degraded (PG_DEGRADED) 2024-03-21T10:35:57.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:57 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1627080230' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:35:58.259 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-21T10:35:58.259 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-21T10:35:58.811 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:35:59.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:35:59 smithi040 ceph-mon[36154]: pgmap v353: 129 pgs: 4 peering, 1 active+recovery_wait+degraded, 124 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.5 KiB/s wr, 7 op/s; 1/194 objects degraded (0.515%) 2024-03-21T10:35:59.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:35:59 smithi031 ceph-mon[31673]: pgmap v353: 129 pgs: 4 peering, 1 active+recovery_wait+degraded, 124 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.5 KiB/s wr, 7 op/s; 1/194 objects degraded (0.515%) 2024-03-21T10:35:59.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:35:59 smithi070 ceph-mon[36330]: pgmap v353: 129 pgs: 4 peering, 1 active+recovery_wait+degraded, 124 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.5 KiB/s wr, 7 op/s; 1/194 objects degraded (0.515%) 2024-03-21T10:36:01.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.154+0000 7f4090e00700 1 -- 172.21.15.31:0/1204251993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f408c10c660 msgr2=0x7f408c10eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:01.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.154+0000 7f4090e00700 1 --2- 172.21.15.31:0/1204251993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f408c10c660 0x7f408c10eb20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4084009990 tx=0x7f4084009ca0 comp rx=0 tx=0).stop 2024-03-21T10:36:01.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.154+0000 7f4090e00700 1 -- 172.21.15.31:0/1204251993 shutdown_connections 2024-03-21T10:36:01.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.154+0000 7f4090e00700 1 --2- 172.21.15.31:0/1204251993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f408c10c660 0x7f408c10eb20 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:01.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.154+0000 7f4090e00700 1 --2- 172.21.15.31:0/1204251993 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f408c103230 0x7f408c103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:01.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.154+0000 7f4090e00700 1 --2- 172.21.15.31:0/1204251993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f408c103b50 0x7f408c10c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:01.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.154+0000 7f4090e00700 1 -- 172.21.15.31:0/1204251993 >> 172.21.15.31:0/1204251993 conn(0x7f408c0fca60 msgr2=0x7f408c0fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:01.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.154+0000 7f4090e00700 1 -- 172.21.15.31:0/1204251993 shutdown_connections 2024-03-21T10:36:01.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.154+0000 7f4090e00700 1 -- 172.21.15.31:0/1204251993 wait complete. 2024-03-21T10:36:01.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.155+0000 7f4090e00700 1 Processor -- start 2024-03-21T10:36:01.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.155+0000 7f4090e00700 1 -- start start 2024-03-21T10:36:01.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.156+0000 7f4090e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f408c103230 0x7f408c1a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:01.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.156+0000 7f4090e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f408c103b50 0x7f408c1a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:01.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.156+0000 7f4090e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f408c10c660 0x7f408c19a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:01.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.156+0000 7f4090e00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408c1123d0 con 0x7f408c10c660 2024-03-21T10:36:01.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.156+0000 7f4090e00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408c112250 con 0x7f408c103230 2024-03-21T10:36:01.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.156+0000 7f4090e00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408c112550 con 0x7f408c103b50 2024-03-21T10:36:01.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.156+0000 7f408b400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f408c103230 0x7f408c1a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:01.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.156+0000 7f408aa00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f408c103b50 0x7f408c1a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:01.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.156+0000 7f408b400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f408c103230 0x7f408c1a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56942/0 (socket says 172.21.15.31:56942) 2024-03-21T10:36:01.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.156+0000 7f408b400700 1 -- 172.21.15.31:0/3047645608 learned_addr learned my addr 172.21.15.31:0/3047645608 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:36:01.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.156+0000 7f408be00700 1 --2- 172.21.15.31:0/3047645608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f408c10c660 0x7f408c19a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:01.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.157+0000 7f408b400700 1 -- 172.21.15.31:0/3047645608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f408c103b50 msgr2=0x7f408c1a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:01.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.157+0000 7f408b400700 1 --2- 172.21.15.31:0/3047645608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f408c103b50 0x7f408c1a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:01.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.157+0000 7f408b400700 1 -- 172.21.15.31:0/3047645608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f408c10c660 msgr2=0x7f408c19a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:01.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.157+0000 7f408b400700 1 --2- 172.21.15.31:0/3047645608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f408c10c660 0x7f408c19a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:01.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.157+0000 7f408b400700 1 -- 172.21.15.31:0/3047645608 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408c19afd0 con 0x7f408c103230 2024-03-21T10:36:01.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.157+0000 7f408aa00700 1 --2- 172.21.15.31:0/3047645608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f408c103b50 0x7f408c1a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:36:01.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.157+0000 7f408be00700 1 --2- 172.21.15.31:0/3047645608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f408c10c660 0x7f408c19a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:36:01.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.157+0000 7f408b400700 1 --2- 172.21.15.31:0/3047645608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f408c103230 0x7f408c1a06b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f407c00ef10 tx=0x7f407c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:01.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.157+0000 7f4083e00700 1 -- 172.21.15.31:0/3047645608 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407c00ca70 con 0x7f408c103230 2024-03-21T10:36:01.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.157+0000 7f4083e00700 1 -- 172.21.15.31:0/3047645608 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f407c019070 con 0x7f408c103230 2024-03-21T10:36:01.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.157+0000 7f4090e00700 1 -- 172.21.15.31:0/3047645608 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f408c19b260 con 0x7f408c103230 2024-03-21T10:36:01.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.157+0000 7f4083e00700 1 -- 172.21.15.31:0/3047645608 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407c0146e0 con 0x7f408c103230 2024-03-21T10:36:01.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.158+0000 7f4090e00700 1 -- 172.21.15.31:0/3047645608 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f408c1a7580 con 0x7f408c103230 2024-03-21T10:36:01.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.159+0000 7f4090e00700 1 -- 172.21.15.31:0/3047645608 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f408c068e50 con 0x7f408c103230 2024-03-21T10:36:01.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.161+0000 7f4083e00700 1 -- 172.21.15.31:0/3047645608 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7f407c012050 con 0x7f408c103230 2024-03-21T10:36:01.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.162+0000 7f4083e00700 1 --2- 172.21.15.31:0/3047645608 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f40700a30c0 0x7f40700a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:01.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.162+0000 7f4083e00700 1 -- 172.21.15.31:0/3047645608 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f407c0cfbb0 con 0x7f408c103230 2024-03-21T10:36:01.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.162+0000 7f408aa00700 1 --2- 172.21.15.31:0/3047645608 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f40700a30c0 0x7f40700a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:01.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.163+0000 7f408aa00700 1 --2- 172.21.15.31:0/3047645608 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f40700a30c0 0x7f40700a5580 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f408c10d910 tx=0x7f4078009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:01.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.164+0000 7f4083e00700 1 -- 172.21.15.31:0/3047645608 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f407c010040 con 0x7f408c103230 2024-03-21T10:36:01.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.392+0000 7f4090e00700 1 -- 172.21.15.31:0/3047645608 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f408c1a7750 con 0x7f408c103230 2024-03-21T10:36:01.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.394+0000 7f4083e00700 1 -- 172.21.15.31:0/3047645608 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24507 (secure 0 0 0) 0x7f407c09c4e0 con 0x7f408c103230 2024-03-21T10:36:01.393 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:36:01.393 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":91,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","created":"2024-03-21T10:25:41.971117+0000","modified":"2024-03-21T10:35:55.370062+0000","last_up_change":"2024-03-21T10:34:07.757196+0000","last_in_change":"2024-03-21T10:33:47.460406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T10:29:44.553968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T10:35:04.945449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T10:35:05.970589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T10:35:09.688711+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e0244d33-00e2-47d5-9f66-34efc138c462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6803","nonce":3089885125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6805","nonce":3089885125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6809","nonce":3089885125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6807","nonce":3089885125}]},"public_addr":"172.21.15.31:6803/3089885125","cluster_addr":"172.21.15.31:6805/3089885125","heartbeat_back_addr":"172.21.15.31:6809/3089885125","heartbeat_front_addr":"172.21.15.31:6807/3089885125","state":["exists","up"]},{"osd":1,"uuid":"76a63504-beab-4354-a6c3-eebc3b16ad03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6801","nonce":618840037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6803","nonce":618840037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6807","nonce":618840037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6805","nonce":618840037}]},"public_addr":"172.21.15.40:6801/618840037","cluster_addr":"172.21.15.40:6803/618840037","heartbeat_back_addr":"172.21.15.40:6807/618840037","heartbeat_front_addr":"172.21.15.40:6805/618840037","state":["exists","up"]},{"osd":2,"uuid":"fa191bd6-9a21-4ee2-8adf-8a63c9cfd821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6801","nonce":2524184117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6803","nonce":2524184117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6807","nonce":2524184117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6805","nonce":2524184117}]},"public_addr":"172.21.15.70:6801/2524184117","cluster_addr":"172.21.15.70:6803/2524184117","heartbeat_back_addr":"172.21.15.70:6807/2524184117","heartbeat_front_addr":"172.21.15.70:6805/2524184117","state":["exists","up"]},{"osd":3,"uuid":"edb44507-ffb0-42a4-827b-08461e1bf952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6811","nonce":2823036296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6813","nonce":2823036296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6817","nonce":2823036296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6815","nonce":2823036296}]},"public_addr":"172.21.15.31:6811/2823036296","cluster_addr":"172.21.15.31:6813/2823036296","heartbeat_back_addr":"172.21.15.31:6817/2823036296","heartbeat_front_addr":"172.21.15.31:6815/2823036296","state":["exists","up"]},{"osd":4,"uuid":"8cb89153-8c3b-431f-b57c-8863fada76dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6809","nonce":4215186309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6811","nonce":4215186309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6815","nonce":4215186309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6813","nonce":4215186309}]},"public_addr":"172.21.15.40:6809/4215186309","cluster_addr":"172.21.15.40:6811/4215186309","heartbeat_back_addr":"172.21.15.40:6815/4215186309","heartbeat_front_addr":"172.21.15.40:6813/4215186309","state":["exists","up"]},{"osd":5,"uuid":"9409505a-d463-408c-a3cd-218380c04d68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6809","nonce":3402830452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6811","nonce":3402830452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6815","nonce":3402830452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6813","nonce":3402830452}]},"public_addr":"172.21.15.70:6809/3402830452","cluster_addr":"172.21.15.70:6811/3402830452","heartbeat_back_addr":"172.21.15.70:6815/3402830452","heartbeat_front_addr":"172.21.15.70:6813/3402830452","state":["exists","up"]},{"osd":6,"uuid":"f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6819","nonce":1193497058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6821","nonce":1193497058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6825","nonce":1193497058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6823","nonce":1193497058}]},"public_addr":"172.21.15.31:6819/1193497058","cluster_addr":"172.21.15.31:6821/1193497058","heartbeat_back_addr":"172.21.15.31:6825/1193497058","heartbeat_front_addr":"172.21.15.31:6823/1193497058","state":["exists","up"]},{"osd":7,"uuid":"db228a87-29a5-45d4-8fe6-a0a554262eb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6817","nonce":3825628119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6819","nonce":3825628119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6823","nonce":3825628119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6821","nonce":3825628119}]},"public_addr":"172.21.15.40:6817/3825628119","cluster_addr":"172.21.15.40:6819/3825628119","heartbeat_back_addr":"172.21.15.40:6823/3825628119","heartbeat_front_addr":"172.21.15.40:6821/3825628119","state":["exists","up"]},{"osd":8,"uuid":"59b9fde0-095d-4e0b-87da-ead37bd8ff1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6817","nonce":534880586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6819","nonce":534880586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6823","nonce":534880586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6821","nonce":534880586}]},"public_addr":"172.21.15.70:6817/534880586","cluster_addr":"172.21.15.70:6819/534880586","heartbeat_back_addr":"172.21.15.70:6823/534880586","heartbeat_front_addr":"172.21.15.70:6821/534880586","state":["exists","up"]},{"osd":9,"uuid":"77f47061-0c5f-4159-b276-e814935d6c1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6827","nonce":587047469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6829","nonce":587047469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6833","nonce":587047469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6831","nonce":587047469}]},"public_addr":"172.21.15.31:6827/587047469","cluster_addr":"172.21.15.31:6829/587047469","heartbeat_back_addr":"172.21.15.31:6833/587047469","heartbeat_front_addr":"172.21.15.31:6831/587047469","state":["exists","up"]},{"osd":10,"uuid":"8853dc43-7bd1-4c93-8063-f7982fd7da38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6825","nonce":512108711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6827","nonce":512108711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6831","nonce":512108711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6829","nonce":512108711}]},"public_addr":"172.21.15.40:6825/512108711","cluster_addr":"172.21.15.40:6827/512108711","heartbeat_back_addr":"172.21.15.40:6831/512108711","heartbeat_front_addr":"172.21.15.40:6829/512108711","state":["exists","up"]},{"osd":11,"uuid":"84b7c2de-c424-45a6-9583-60097e372a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6825","nonce":540305718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6827","nonce":540305718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6831","nonce":540305718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6829","nonce":540305718}]},"public_addr":"172.21.15.70:6825/540305718","cluster_addr":"172.21.15.70:6827/540305718","heartbeat_back_addr":"172.21.15.70:6831/540305718","heartbeat_front_addr":"172.21.15.70:6829/540305718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:28:51.477358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:17.405045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:41.626268+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:15.339568+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:42.676996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:07.516999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:42.412273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:10.365041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:35.699120+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:11.733639+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:39.505442+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:34:05.449301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451930554":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/3342588843":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6801/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/1626448389":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/1910059351":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/344822770":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/3811368512":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/1981437651":"2024-03-22T10:26:16.959165+0000","172.21.15.31:6801/1981437651":"2024-03-22T10:26:16.959165+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T10:36:01.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.398+0000 7f4090e00700 1 -- 172.21.15.31:0/3047645608 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f40700a30c0 msgr2=0x7f40700a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:01.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.398+0000 7f4090e00700 1 --2- 172.21.15.31:0/3047645608 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f40700a30c0 0x7f40700a5580 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f408c10d910 tx=0x7f4078009260 comp rx=0 tx=0).stop 2024-03-21T10:36:01.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.398+0000 7f4090e00700 1 -- 172.21.15.31:0/3047645608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f408c103230 msgr2=0x7f408c1a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:01.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.398+0000 7f4090e00700 1 --2- 172.21.15.31:0/3047645608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f408c103230 0x7f408c1a06b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f407c00ef10 tx=0x7f407c00c3a0 comp rx=0 tx=0).stop 2024-03-21T10:36:01.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.398+0000 7f4090e00700 1 -- 172.21.15.31:0/3047645608 shutdown_connections 2024-03-21T10:36:01.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.398+0000 7f4090e00700 1 --2- 172.21.15.31:0/3047645608 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f40700a30c0 0x7f40700a5580 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:01.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.398+0000 7f4090e00700 1 --2- 172.21.15.31:0/3047645608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f408c10c660 0x7f408c19a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:01.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.398+0000 7f4090e00700 1 --2- 172.21.15.31:0/3047645608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f408c103230 0x7f408c1a06b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:01.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.398+0000 7f4090e00700 1 --2- 172.21.15.31:0/3047645608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f408c103b50 0x7f408c1a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:01.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.398+0000 7f4090e00700 1 -- 172.21.15.31:0/3047645608 >> 172.21.15.31:0/3047645608 conn(0x7f408c0fca60 msgr2=0x7f408c10e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:01.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.398+0000 7f4090e00700 1 -- 172.21.15.31:0/3047645608 shutdown_connections 2024-03-21T10:36:01.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:01.399+0000 7f4090e00700 1 -- 172.21.15.31:0/3047645608 wait complete. 2024-03-21T10:36:01.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:01 smithi040 ceph-mon[36154]: pgmap v354: 129 pgs: 1 peering, 1 active+recovery_wait+degraded, 127 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.9 KiB/s wr, 5 op/s; 1/194 objects degraded (0.515%); 0 B/s, 0 objects/s recovering 2024-03-21T10:36:01.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:01 smithi031 ceph-mon[31673]: pgmap v354: 129 pgs: 1 peering, 1 active+recovery_wait+degraded, 127 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.9 KiB/s wr, 5 op/s; 1/194 objects degraded (0.515%); 0 B/s, 0 objects/s recovering 2024-03-21T10:36:01.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:01 smithi070 ceph-mon[36330]: pgmap v354: 129 pgs: 1 peering, 1 active+recovery_wait+degraded, 127 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.9 KiB/s wr, 5 op/s; 1/194 objects degraded (0.515%); 0 B/s, 0 objects/s recovering 2024-03-21T10:36:02.288 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-21T10:36:02.289 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-21T10:36:02.289 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-21T10:36:02.289 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-21T10:36:02.290 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-21T10:36:02.290 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-21T10:36:02.290 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-21T10:36:02.291 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-21T10:36:02.291 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-21T10:36:02.292 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-21T10:36:02.292 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-21T10:36:02.292 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-21T10:36:02.475 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:02 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3047645608' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:36:02.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:02 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3047645608' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:36:02.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:02 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3047645608' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:36:03.444 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:36:03.449 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:36:03.478 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:36:03.500 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:36:03.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:03 smithi040 ceph-mon[36154]: pgmap v355: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.9 KiB/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T10:36:03.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:03 smithi040 ceph-mon[36154]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/194 objects degraded (0.515%), 1 pg degraded) 2024-03-21T10:36:03.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:03 smithi040 ceph-mon[36154]: Cluster is now healthy 2024-03-21T10:36:03.697 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:03 smithi031 ceph-mon[31673]: pgmap v355: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.9 KiB/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T10:36:03.697 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:03 smithi031 ceph-mon[31673]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/194 objects degraded (0.515%), 1 pg degraded) 2024-03-21T10:36:03.697 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:03 smithi031 ceph-mon[31673]: Cluster is now healthy 2024-03-21T10:36:03.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:03 smithi070 ceph-mon[36330]: pgmap v355: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.9 KiB/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T10:36:03.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:03 smithi070 ceph-mon[36330]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/194 objects degraded (0.515%), 1 pg degraded) 2024-03-21T10:36:03.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:03 smithi070 ceph-mon[36330]: Cluster is now healthy 2024-03-21T10:36:03.968 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:36:03.969 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:36:03.973 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:36:03.974 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:36:04.106 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:36:05.442 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:05 smithi031 ceph-mon[31673]: pgmap v356: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.6 KiB/s wr, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T10:36:05.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:05 smithi040 ceph-mon[36154]: pgmap v356: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.6 KiB/s wr, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T10:36:05.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:05 smithi070 ceph-mon[36330]: pgmap v356: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.6 KiB/s wr, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T10:36:07.643 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:07 smithi031 ceph-mon[31673]: pgmap v357: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.9 KiB/s wr, 7 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T10:36:07.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:07 smithi040 ceph-mon[36154]: pgmap v357: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.9 KiB/s wr, 7 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T10:36:07.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:07 smithi070 ceph-mon[36330]: pgmap v357: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.9 KiB/s wr, 7 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T10:36:07.867 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:36:07.868 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:36:08.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:08 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:36:08.725 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:08 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:36:08.789 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:08 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:36:09.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:09 smithi070 ceph-mon[36330]: pgmap v358: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T10:36:09.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:09 smithi040 ceph-mon[36154]: pgmap v358: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T10:36:09.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:09 smithi031 ceph-mon[31673]: pgmap v358: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T10:36:11.281 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:11 smithi031 ceph-mon[31673]: pgmap v359: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T10:36:11.281 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:11 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:36:11.281 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:11 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:36:11.281 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:11 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:36:11.281 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:11 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:36:11.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:11 smithi040 ceph-mon[36154]: pgmap v359: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T10:36:11.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:11 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:36:11.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:11 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:36:11.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:11 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:36:11.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:11 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:36:11.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:11 smithi070 ceph-mon[36330]: pgmap v359: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T10:36:11.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:11 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:36:11.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:11 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:36:11.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:11 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:36:11.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:11 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:36:13.417 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:13 smithi031 ceph-mon[31673]: pgmap v360: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T10:36:13.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:13 smithi040 ceph-mon[36154]: pgmap v360: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T10:36:13.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:13 smithi070 ceph-mon[36330]: pgmap v360: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T10:36:13.690 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:36:15.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:15 smithi031 ceph-mon[31673]: pgmap v361: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-21T10:36:15.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:15 smithi040 ceph-mon[36154]: pgmap v361: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-21T10:36:15.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:15 smithi070 ceph-mon[36330]: pgmap v361: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-21T10:36:17.499 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:17 smithi031 ceph-mon[31673]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 596 B/s wr, 7 op/s 2024-03-21T10:36:17.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:17 smithi070 ceph-mon[36330]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 596 B/s wr, 7 op/s 2024-03-21T10:36:17.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:17 smithi040 ceph-mon[36154]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 596 B/s wr, 7 op/s 2024-03-21T10:36:17.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.891+0000 7feb17800700 1 -- 172.21.15.31:0/433884831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feb10104600 msgr2=0x7feb101049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:17.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.891+0000 7feb17800700 1 --2- 172.21.15.31:0/433884831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feb10104600 0x7feb101049e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7feb040099c0 tx=0x7feb04009cd0 comp rx=0 tx=0).stop 2024-03-21T10:36:17.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.891+0000 7feb17800700 1 -- 172.21.15.31:0/433884831 shutdown_connections 2024-03-21T10:36:17.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.891+0000 7feb17800700 1 --2- 172.21.15.31:0/433884831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb101096e0 0x7feb101112b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:17.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.891+0000 7feb17800700 1 --2- 172.21.15.31:0/433884831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feb10104600 0x7feb101049e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:17.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.891+0000 7feb17800700 1 --2- 172.21.15.31:0/433884831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb10104fb0 0x7feb10109000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:17.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.891+0000 7feb17800700 1 -- 172.21.15.31:0/433884831 >> 172.21.15.31:0/433884831 conn(0x7feb100ffe70 msgr2=0x7feb10102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:17.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.891+0000 7feb17800700 1 -- 172.21.15.31:0/433884831 shutdown_connections 2024-03-21T10:36:17.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.891+0000 7feb17800700 1 -- 172.21.15.31:0/433884831 wait complete. 2024-03-21T10:36:17.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.893+0000 7feb17800700 1 Processor -- start 2024-03-21T10:36:17.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.893+0000 7feb17800700 1 -- start start 2024-03-21T10:36:17.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.893+0000 7feb17800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb10104600 0x7feb101a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:17.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.893+0000 7feb17800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feb10104fb0 0x7feb101a0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:17.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.893+0000 7feb17800700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb101096e0 0x7feb1019a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:17.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.893+0000 7feb17800700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb10114730 con 0x7feb101096e0 2024-03-21T10:36:17.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.893+0000 7feb17800700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb101145b0 con 0x7feb10104600 2024-03-21T10:36:17.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.893+0000 7feb17800700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb101148b0 con 0x7feb10104fb0 2024-03-21T10:36:17.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.894+0000 7feb14e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb10104600 0x7feb101a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:17.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.894+0000 7feb0fe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feb10104fb0 0x7feb101a0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:17.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.894+0000 7feb14e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb10104600 0x7feb101a07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:45484/0 (socket says 172.21.15.31:45484) 2024-03-21T10:36:17.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.894+0000 7feb14e00700 1 -- 172.21.15.31:0/3796079398 learned_addr learned my addr 172.21.15.31:0/3796079398 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:36:17.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.894+0000 7feb15800700 1 --2- 172.21.15.31:0/3796079398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb101096e0 0x7feb1019a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:17.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.894+0000 7feb0fe00700 1 -- 172.21.15.31:0/3796079398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb10104600 msgr2=0x7feb101a07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:17.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.894+0000 7feb0fe00700 1 --2- 172.21.15.31:0/3796079398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb10104600 0x7feb101a07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:17.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.894+0000 7feb0fe00700 1 -- 172.21.15.31:0/3796079398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb101096e0 msgr2=0x7feb1019a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:17.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.894+0000 7feb0fe00700 1 --2- 172.21.15.31:0/3796079398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb101096e0 0x7feb1019a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:17.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.894+0000 7feb0fe00700 1 -- 172.21.15.31:0/3796079398 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb1019b110 con 0x7feb10104fb0 2024-03-21T10:36:17.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.894+0000 7feb15800700 1 --2- 172.21.15.31:0/3796079398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb101096e0 0x7feb1019a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:36:17.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.894+0000 7feb0fe00700 1 --2- 172.21.15.31:0/3796079398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feb10104fb0 0x7feb101a0d30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7feafc00e9c0 tx=0x7feafc00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:17.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.895+0000 7feb0d600700 1 -- 172.21.15.31:0/3796079398 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc00cb70 con 0x7feb10104fb0 2024-03-21T10:36:17.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.895+0000 7feb0d600700 1 -- 172.21.15.31:0/3796079398 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feafc00cd10 con 0x7feb10104fb0 2024-03-21T10:36:17.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.895+0000 7feb0d600700 1 -- 172.21.15.31:0/3796079398 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc018750 con 0x7feb10104fb0 2024-03-21T10:36:17.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.895+0000 7feb17800700 1 -- 172.21.15.31:0/3796079398 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb1019b3a0 con 0x7feb10104fb0 2024-03-21T10:36:17.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.895+0000 7feb17800700 1 -- 172.21.15.31:0/3796079398 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb101a76c0 con 0x7feb10104fb0 2024-03-21T10:36:17.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.896+0000 7feb17800700 1 -- 172.21.15.31:0/3796079398 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feb100628e0 con 0x7feb10104fb0 2024-03-21T10:36:17.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.898+0000 7feb0d600700 1 -- 172.21.15.31:0/3796079398 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7feafc010430 con 0x7feb10104fb0 2024-03-21T10:36:17.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.901+0000 7feb0d600700 1 --2- 172.21.15.31:0/3796079398 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7feaf80a30c0 0x7feaf80a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:17.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.901+0000 7feb0d600700 1 -- 172.21.15.31:0/3796079398 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7feafc0d0380 con 0x7feb10104fb0 2024-03-21T10:36:17.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.901+0000 7feb14e00700 1 --2- 172.21.15.31:0/3796079398 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7feaf80a30c0 0x7feaf80a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:17.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.901+0000 7feb0d600700 1 --2- 172.21.15.31:0/3796079398 >> [v2:172.21.15.40:6824/512108711,v1:172.21.15.40:6825/512108711] conn(0x7feaf80ae1a0 0x7feaf80b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:17.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.901+0000 7feb0d600700 1 -- 172.21.15.31:0/3796079398 --> [v2:172.21.15.40:6824/512108711,v1:172.21.15.40:6825/512108711] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feaf80b0d10 con 0x7feaf80ae1a0 2024-03-21T10:36:17.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.901+0000 7feb0d600700 1 -- 172.21.15.31:0/3796079398 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7feafc0d07d0 con 0x7feb10104fb0 2024-03-21T10:36:17.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.902+0000 7feb15800700 1 --2- 172.21.15.31:0/3796079398 >> [v2:172.21.15.40:6824/512108711,v1:172.21.15.40:6825/512108711] conn(0x7feaf80ae1a0 0x7feaf80b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:17.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.902+0000 7feb14e00700 1 --2- 172.21.15.31:0/3796079398 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7feaf80a30c0 0x7feaf80a5580 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7feb04000c00 tx=0x7feb04005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:17.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.902+0000 7feb15800700 1 --2- 172.21.15.31:0/3796079398 >> [v2:172.21.15.40:6824/512108711,v1:172.21.15.40:6825/512108711] conn(0x7feaf80ae1a0 0x7feaf80b0640 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:17.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.904+0000 7feb0d600700 1 -- 172.21.15.31:0/3796079398 <== osd.10 v2:172.21.15.40:6824/512108711 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7feaf80b0d10 con 0x7feaf80ae1a0 2024-03-21T10:36:17.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.922+0000 7feb17800700 1 -- 172.21.15.31:0/3796079398 --> [v2:172.21.15.40:6824/512108711,v1:172.21.15.40:6825/512108711] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7feb10062d60 con 0x7feaf80ae1a0 2024-03-21T10:36:17.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.923+0000 7feb0d600700 1 -- 172.21.15.31:0/3796079398 <== osd.10 v2:172.21.15.40:6824/512108711 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7feb10062d60 con 0x7feaf80ae1a0 2024-03-21T10:36:17.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.923+0000 7feb17800700 1 -- 172.21.15.31:0/3796079398 >> [v2:172.21.15.40:6824/512108711,v1:172.21.15.40:6825/512108711] conn(0x7feaf80ae1a0 msgr2=0x7feaf80b0640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:17.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.923+0000 7feb17800700 1 --2- 172.21.15.31:0/3796079398 >> [v2:172.21.15.40:6824/512108711,v1:172.21.15.40:6825/512108711] conn(0x7feaf80ae1a0 0x7feaf80b0640 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:17.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.924+0000 7feb17800700 1 -- 172.21.15.31:0/3796079398 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7feaf80a30c0 msgr2=0x7feaf80a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:17.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.924+0000 7feb17800700 1 --2- 172.21.15.31:0/3796079398 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7feaf80a30c0 0x7feaf80a5580 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7feb04000c00 tx=0x7feb04005fb0 comp rx=0 tx=0).stop 2024-03-21T10:36:17.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.924+0000 7feb17800700 1 -- 172.21.15.31:0/3796079398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feb10104fb0 msgr2=0x7feb101a0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:17.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.924+0000 7feb17800700 1 --2- 172.21.15.31:0/3796079398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feb10104fb0 0x7feb101a0d30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7feafc00e9c0 tx=0x7feafc00ecd0 comp rx=0 tx=0).stop 2024-03-21T10:36:17.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.924+0000 7feb15800700 1 -- 172.21.15.31:0/3796079398 reap_dead start 2024-03-21T10:36:17.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.924+0000 7feb17800700 1 -- 172.21.15.31:0/3796079398 shutdown_connections 2024-03-21T10:36:17.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.924+0000 7feb17800700 1 -- 172.21.15.31:0/3796079398 >> 172.21.15.31:0/3796079398 conn(0x7feb100ffe70 msgr2=0x7feb10108870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:17.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.924+0000 7feb17800700 1 -- 172.21.15.31:0/3796079398 shutdown_connections 2024-03-21T10:36:17.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.924+0000 7feb17800700 1 -- 172.21.15.31:0/3796079398 wait complete. 2024-03-21T10:36:18.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.978+0000 7f6785000700 1 -- 172.21.15.31:0/945698018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67801081e0 msgr2=0x7f67801085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:18.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.978+0000 7f6785000700 1 --2- 172.21.15.31:0/945698018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67801081e0 0x7f67801085c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f67700099c0 tx=0x7f6770009cd0 comp rx=0 tx=0).stop 2024-03-21T10:36:18.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.978+0000 7f6785000700 1 -- 172.21.15.31:0/945698018 shutdown_connections 2024-03-21T10:36:18.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.978+0000 7f6785000700 1 --2- 172.21.15.31:0/945698018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f678006a7e0 0x7f678010f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:18.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.978+0000 7f6785000700 1 --2- 172.21.15.31:0/945698018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67801081e0 0x7f67801085c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:18.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.978+0000 7f6785000700 1 --2- 172.21.15.31:0/945698018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6780069e40 0x7f678006a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:18.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.978+0000 7f6785000700 1 -- 172.21.15.31:0/945698018 >> 172.21.15.31:0/945698018 conn(0x7f6780077fc0 msgr2=0x7f67800783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:18.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.978+0000 7f6785000700 1 -- 172.21.15.31:0/945698018 shutdown_connections 2024-03-21T10:36:18.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.978+0000 7f6785000700 1 -- 172.21.15.31:0/945698018 wait complete. 2024-03-21T10:36:18.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.980+0000 7f6785000700 1 Processor -- start 2024-03-21T10:36:18.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.980+0000 7f6785000700 1 -- start start 2024-03-21T10:36:18.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.981+0000 7f6785000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6780069e40 0x7f678010e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:18.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.981+0000 7f6785000700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f678006a7e0 0x7f678010e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:18.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.981+0000 7f6785000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67801081e0 0x7f67801a3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:18.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.981+0000 7f6785000700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6780112490 con 0x7f678006a7e0 2024-03-21T10:36:18.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.981+0000 7f6785000700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6780112310 con 0x7f6780069e40 2024-03-21T10:36:18.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.981+0000 7f6785000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6780112610 con 0x7f67801081e0 2024-03-21T10:36:18.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.981+0000 7f677de00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6780069e40 0x7f678010e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:18.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.981+0000 7f677de00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6780069e40 0x7f678010e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:45496/0 (socket says 172.21.15.31:45496) 2024-03-21T10:36:18.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.981+0000 7f677de00700 1 -- 172.21.15.31:0/3750909231 learned_addr learned my addr 172.21.15.31:0/3750909231 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:36:18.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.981+0000 7f677e800700 1 --2- 172.21.15.31:0/3750909231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67801081e0 0x7f67801a3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:18.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.981+0000 7f6777e00700 1 --2- 172.21.15.31:0/3750909231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f678006a7e0 0x7f678010e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:18.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.982+0000 7f677de00700 1 -- 172.21.15.31:0/3750909231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67801081e0 msgr2=0x7f67801a3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:18.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.982+0000 7f677de00700 1 --2- 172.21.15.31:0/3750909231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67801081e0 0x7f67801a3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:18.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.982+0000 7f677de00700 1 -- 172.21.15.31:0/3750909231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f678006a7e0 msgr2=0x7f678010e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:18.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.982+0000 7f677de00700 1 --2- 172.21.15.31:0/3750909231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f678006a7e0 0x7f678010e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:18.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.982+0000 7f677de00700 1 -- 172.21.15.31:0/3750909231 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67801a3f50 con 0x7f6780069e40 2024-03-21T10:36:18.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.982+0000 7f6777e00700 1 --2- 172.21.15.31:0/3750909231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f678006a7e0 0x7f678010e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:36:18.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.982+0000 7f677de00700 1 --2- 172.21.15.31:0/3750909231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6780069e40 0x7f678010e360 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6770006010 tx=0x7f6770003e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:18.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.982+0000 7f6776a00700 1 -- 172.21.15.31:0/3750909231 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6770009e60 con 0x7f6780069e40 2024-03-21T10:36:18.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.982+0000 7f6776a00700 1 -- 172.21.15.31:0/3750909231 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6770003680 con 0x7f6780069e40 2024-03-21T10:36:18.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.982+0000 7f6776a00700 1 -- 172.21.15.31:0/3750909231 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67700173f0 con 0x7f6780069e40 2024-03-21T10:36:18.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.982+0000 7f6785000700 1 -- 172.21.15.31:0/3750909231 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67801a41e0 con 0x7f6780069e40 2024-03-21T10:36:18.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.983+0000 7f6785000700 1 -- 172.21.15.31:0/3750909231 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67801a4670 con 0x7f6780069e40 2024-03-21T10:36:18.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.984+0000 7f6785000700 1 -- 172.21.15.31:0/3750909231 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6780101e20 con 0x7f6780069e40 2024-03-21T10:36:18.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.986+0000 7f6776a00700 1 -- 172.21.15.31:0/3750909231 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7f677000f680 con 0x7f6780069e40 2024-03-21T10:36:18.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.988+0000 7f6776a00700 1 --2- 172.21.15.31:0/3750909231 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f67640a30c0 0x7f67640a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:18.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.989+0000 7f6776a00700 1 -- 172.21.15.31:0/3750909231 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f67700d0680 con 0x7f6780069e40 2024-03-21T10:36:18.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.989+0000 7f6777e00700 1 --2- 172.21.15.31:0/3750909231 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f67640a30c0 0x7f67640a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:18.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.989+0000 7f6776a00700 1 --2- 172.21.15.31:0/3750909231 >> [v2:172.21.15.31:6826/587047469,v1:172.21.15.31:6827/587047469] conn(0x7f67640ae1a0 0x7f67640b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:18.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.989+0000 7f6776a00700 1 -- 172.21.15.31:0/3750909231 --> [v2:172.21.15.31:6826/587047469,v1:172.21.15.31:6827/587047469] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f67640b0d10 con 0x7f67640ae1a0 2024-03-21T10:36:18.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.989+0000 7f6776a00700 1 -- 172.21.15.31:0/3750909231 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67700d7050 con 0x7f6780069e40 2024-03-21T10:36:18.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.990+0000 7f677e800700 1 --2- 172.21.15.31:0/3750909231 >> [v2:172.21.15.31:6826/587047469,v1:172.21.15.31:6827/587047469] conn(0x7f67640ae1a0 0x7f67640b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:18.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.990+0000 7f6777e00700 1 --2- 172.21.15.31:0/3750909231 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f67640a30c0 0x7f67640a5580 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f67800ff850 tx=0x7f6768009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:18.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.990+0000 7f677e800700 1 --2- 172.21.15.31:0/3750909231 >> [v2:172.21.15.31:6826/587047469,v1:172.21.15.31:6827/587047469] conn(0x7f67640ae1a0 0x7f67640b0640 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:18.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:17.992+0000 7f6776a00700 1 -- 172.21.15.31:0/3750909231 <== osd.9 v2:172.21.15.31:6826/587047469 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f67640b0d10 con 0x7f67640ae1a0 2024-03-21T10:36:18.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.015+0000 7f6785000700 1 -- 172.21.15.31:0/3750909231 --> [v2:172.21.15.31:6826/587047469,v1:172.21.15.31:6827/587047469] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6780103890 con 0x7f67640ae1a0 2024-03-21T10:36:18.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.016+0000 7f6776a00700 1 -- 172.21.15.31:0/3750909231 <== osd.9 v2:172.21.15.31:6826/587047469 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f6780103890 con 0x7f67640ae1a0 2024-03-21T10:36:18.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.016+0000 7f6785000700 1 -- 172.21.15.31:0/3750909231 >> [v2:172.21.15.31:6826/587047469,v1:172.21.15.31:6827/587047469] conn(0x7f67640ae1a0 msgr2=0x7f67640b0640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:18.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.016+0000 7f6785000700 1 --2- 172.21.15.31:0/3750909231 >> [v2:172.21.15.31:6826/587047469,v1:172.21.15.31:6827/587047469] conn(0x7f67640ae1a0 0x7f67640b0640 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:18.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.017+0000 7f6785000700 1 -- 172.21.15.31:0/3750909231 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f67640a30c0 msgr2=0x7f67640a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:18.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.017+0000 7f6785000700 1 --2- 172.21.15.31:0/3750909231 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f67640a30c0 0x7f67640a5580 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f67800ff850 tx=0x7f6768009260 comp rx=0 tx=0).stop 2024-03-21T10:36:18.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.017+0000 7f6785000700 1 -- 172.21.15.31:0/3750909231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6780069e40 msgr2=0x7f678010e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:18.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.017+0000 7f6785000700 1 --2- 172.21.15.31:0/3750909231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6780069e40 0x7f678010e360 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6770006010 tx=0x7f6770003e00 comp rx=0 tx=0).stop 2024-03-21T10:36:18.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.017+0000 7f677e800700 1 -- 172.21.15.31:0/3750909231 reap_dead start 2024-03-21T10:36:18.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.017+0000 7f6785000700 1 -- 172.21.15.31:0/3750909231 shutdown_connections 2024-03-21T10:36:18.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.017+0000 7f6785000700 1 -- 172.21.15.31:0/3750909231 >> 172.21.15.31:0/3750909231 conn(0x7f6780077fc0 msgr2=0x7f67801013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:18.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.017+0000 7f6785000700 1 -- 172.21.15.31:0/3750909231 shutdown_connections 2024-03-21T10:36:18.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.017+0000 7f6785000700 1 -- 172.21.15.31:0/3750909231 wait complete. 2024-03-21T10:36:18.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.880+0000 7fb429600700 1 -- 172.21.15.31:0/330261865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb424107510 msgr2=0x7fb42410f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:18.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.880+0000 7fb429600700 1 --2- 172.21.15.31:0/330261865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb424107510 0x7fb42410f090 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb41c00b0a0 tx=0x7fb41c00b3b0 comp rx=0 tx=0).stop 2024-03-21T10:36:18.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.880+0000 7fb429600700 1 -- 172.21.15.31:0/330261865 shutdown_connections 2024-03-21T10:36:18.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.880+0000 7fb429600700 1 --2- 172.21.15.31:0/330261865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb424107510 0x7fb42410f090 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:18.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.880+0000 7fb429600700 1 --2- 172.21.15.31:0/330261865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb42406a930 0x7fb42406ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:18.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.881+0000 7fb429600700 1 --2- 172.21.15.31:0/330261865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb42406b2e0 0x7fb424106de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:18.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.881+0000 7fb429600700 1 -- 172.21.15.31:0/330261865 >> 172.21.15.31:0/330261865 conn(0x7fb4240770e0 msgr2=0x7fb4240774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:18.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.881+0000 7fb429600700 1 -- 172.21.15.31:0/330261865 shutdown_connections 2024-03-21T10:36:18.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.881+0000 7fb429600700 1 -- 172.21.15.31:0/330261865 wait complete. 2024-03-21T10:36:18.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.882+0000 7fb429600700 1 Processor -- start 2024-03-21T10:36:18.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.882+0000 7fb429600700 1 -- start start 2024-03-21T10:36:18.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.883+0000 7fb429600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb42406a930 0x7fb4241a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:18.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.883+0000 7fb429600700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb42406b2e0 0x7fb4241a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:18.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.883+0000 7fb429600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb424107510 0x7fb42419a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:18.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.883+0000 7fb429600700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb424112500 con 0x7fb42406b2e0 2024-03-21T10:36:18.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.883+0000 7fb429600700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb424112380 con 0x7fb424107510 2024-03-21T10:36:18.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.883+0000 7fb429600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb424112680 con 0x7fb42406a930 2024-03-21T10:36:18.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.883+0000 7fb422a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb42406a930 0x7fb4241a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:18.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.883+0000 7fb423400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb424107510 0x7fb42419a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:18.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.883+0000 7fb422a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb42406a930 0x7fb4241a07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:59482/0 (socket says 172.21.15.31:59482) 2024-03-21T10:36:18.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.883+0000 7fb422a00700 1 -- 172.21.15.31:0/239679949 learned_addr learned my addr 172.21.15.31:0/239679949 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:36:18.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.883+0000 7fb422000700 1 --2- 172.21.15.31:0/239679949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb42406b2e0 0x7fb4241a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:18.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.883+0000 7fb422a00700 1 -- 172.21.15.31:0/239679949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb424107510 msgr2=0x7fb42419a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:18.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.883+0000 7fb422a00700 1 --2- 172.21.15.31:0/239679949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb424107510 0x7fb42419a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:18.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.883+0000 7fb422a00700 1 -- 172.21.15.31:0/239679949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb42406b2e0 msgr2=0x7fb4241a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:18.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.883+0000 7fb422a00700 1 --2- 172.21.15.31:0/239679949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb42406b2e0 0x7fb4241a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:18.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.884+0000 7fb422a00700 1 -- 172.21.15.31:0/239679949 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42419b100 con 0x7fb42406a930 2024-03-21T10:36:18.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.884+0000 7fb423400700 1 --2- 172.21.15.31:0/239679949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb424107510 0x7fb42419a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:36:18.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.884+0000 7fb422000700 1 --2- 172.21.15.31:0/239679949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb42406b2e0 0x7fb4241a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:36:18.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.884+0000 7fb422a00700 1 --2- 172.21.15.31:0/239679949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb42406a930 0x7fb4241a07e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb41400eb30 tx=0x7fb41400ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:18.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.884+0000 7fb41b400700 1 -- 172.21.15.31:0/239679949 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41400cc90 con 0x7fb42406a930 2024-03-21T10:36:18.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.884+0000 7fb429600700 1 -- 172.21.15.31:0/239679949 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb42419b3f0 con 0x7fb42406a930 2024-03-21T10:36:18.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.884+0000 7fb41b400700 1 -- 172.21.15.31:0/239679949 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb414004500 con 0x7fb42406a930 2024-03-21T10:36:18.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.884+0000 7fb41b400700 1 -- 172.21.15.31:0/239679949 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb414010430 con 0x7fb42406a930 2024-03-21T10:36:18.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.884+0000 7fb429600700 1 -- 172.21.15.31:0/239679949 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4241a76b0 con 0x7fb42406a930 2024-03-21T10:36:18.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.885+0000 7fb429600700 1 -- 172.21.15.31:0/239679949 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb4240ffa50 con 0x7fb42406a930 2024-03-21T10:36:18.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.887+0000 7fb41b400700 1 -- 172.21.15.31:0/239679949 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7fb414003f70 con 0x7fb42406a930 2024-03-21T10:36:18.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.889+0000 7fb41b400700 1 --2- 172.21.15.31:0/239679949 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb4080a3040 0x7fb4080a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:18.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.889+0000 7fb41b400700 1 -- 172.21.15.31:0/239679949 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fb4140cfc70 con 0x7fb42406a930 2024-03-21T10:36:18.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.889+0000 7fb422000700 1 --2- 172.21.15.31:0/239679949 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb4080a3040 0x7fb4080a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:18.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.889+0000 7fb41b400700 1 --2- 172.21.15.31:0/239679949 >> [v2:172.21.15.31:6810/2823036296,v1:172.21.15.31:6811/2823036296] conn(0x7fb4080ae120 0x7fb4080b05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:18.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.889+0000 7fb41b400700 1 -- 172.21.15.31:0/239679949 --> [v2:172.21.15.31:6810/2823036296,v1:172.21.15.31:6811/2823036296] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb41400be50 con 0x7fb4080ae120 2024-03-21T10:36:18.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.889+0000 7fb41b400700 1 -- 172.21.15.31:0/239679949 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb4140d2290 con 0x7fb42406a930 2024-03-21T10:36:18.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.890+0000 7fb423400700 1 --2- 172.21.15.31:0/239679949 >> [v2:172.21.15.31:6810/2823036296,v1:172.21.15.31:6811/2823036296] conn(0x7fb4080ae120 0x7fb4080b05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:18.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.890+0000 7fb422000700 1 --2- 172.21.15.31:0/239679949 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb4080a3040 0x7fb4080a5500 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb424102a20 tx=0x7fb410009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:18.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.890+0000 7fb423400700 1 --2- 172.21.15.31:0/239679949 >> [v2:172.21.15.31:6810/2823036296,v1:172.21.15.31:6811/2823036296] conn(0x7fb4080ae120 0x7fb4080b05c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:18.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.891+0000 7fb41b400700 1 -- 172.21.15.31:0/239679949 <== osd.3 v2:172.21.15.31:6810/2823036296 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb41400be50 con 0x7fb4080ae120 2024-03-21T10:36:18.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.910+0000 7fb429600700 1 -- 172.21.15.31:0/239679949 --> [v2:172.21.15.31:6810/2823036296,v1:172.21.15.31:6811/2823036296] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb4241038c0 con 0x7fb4080ae120 2024-03-21T10:36:18.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.910+0000 7fb41b400700 1 -- 172.21.15.31:0/239679949 <== osd.3 v2:172.21.15.31:6810/2823036296 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb4241038c0 con 0x7fb4080ae120 2024-03-21T10:36:18.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.911+0000 7fb429600700 1 -- 172.21.15.31:0/239679949 >> [v2:172.21.15.31:6810/2823036296,v1:172.21.15.31:6811/2823036296] conn(0x7fb4080ae120 msgr2=0x7fb4080b05c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:18.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.911+0000 7fb429600700 1 --2- 172.21.15.31:0/239679949 >> [v2:172.21.15.31:6810/2823036296,v1:172.21.15.31:6811/2823036296] conn(0x7fb4080ae120 0x7fb4080b05c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:18.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.911+0000 7fb429600700 1 -- 172.21.15.31:0/239679949 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb4080a3040 msgr2=0x7fb4080a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:18.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.911+0000 7fb429600700 1 --2- 172.21.15.31:0/239679949 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb4080a3040 0x7fb4080a5500 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb424102a20 tx=0x7fb410009260 comp rx=0 tx=0).stop 2024-03-21T10:36:18.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.911+0000 7fb429600700 1 -- 172.21.15.31:0/239679949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb42406a930 msgr2=0x7fb4241a07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:18.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.911+0000 7fb429600700 1 --2- 172.21.15.31:0/239679949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb42406a930 0x7fb4241a07e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb41400eb30 tx=0x7fb41400ee40 comp rx=0 tx=0).stop 2024-03-21T10:36:18.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.911+0000 7fb423400700 1 -- 172.21.15.31:0/239679949 reap_dead start 2024-03-21T10:36:18.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.911+0000 7fb429600700 1 -- 172.21.15.31:0/239679949 shutdown_connections 2024-03-21T10:36:18.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.911+0000 7fb429600700 1 -- 172.21.15.31:0/239679949 >> 172.21.15.31:0/239679949 conn(0x7fb4240770e0 msgr2=0x7fb4240ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:18.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.911+0000 7fb429600700 1 -- 172.21.15.31:0/239679949 shutdown_connections 2024-03-21T10:36:18.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:18.912+0000 7fb429600700 1 -- 172.21.15.31:0/239679949 wait complete. 2024-03-21T10:36:19.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.601+0000 7fe6ed400700 1 -- 172.21.15.31:0/1109088047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6e8104e90 msgr2=0x7fe6e8108ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:19.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.601+0000 7fe6ed400700 1 --2- 172.21.15.31:0/1109088047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6e8104e90 0x7fe6e8108ee0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe6d4009990 tx=0x7fe6d4009ca0 comp rx=0 tx=0).stop 2024-03-21T10:36:19.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.601+0000 7fe6ed400700 1 -- 172.21.15.31:0/1109088047 shutdown_connections 2024-03-21T10:36:19.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.601+0000 7fe6ed400700 1 --2- 172.21.15.31:0/1109088047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6e81095c0 0x7fe6e8111190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:19.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.601+0000 7fe6ed400700 1 --2- 172.21.15.31:0/1109088047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6e81044e0 0x7fe6e81048c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:19.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.601+0000 7fe6ed400700 1 --2- 172.21.15.31:0/1109088047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6e8104e90 0x7fe6e8108ee0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:19.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.601+0000 7fe6ed400700 1 -- 172.21.15.31:0/1109088047 >> 172.21.15.31:0/1109088047 conn(0x7fe6e80ffdb0 msgr2=0x7fe6e81021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:19.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.602+0000 7fe6ed400700 1 -- 172.21.15.31:0/1109088047 shutdown_connections 2024-03-21T10:36:19.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.602+0000 7fe6ed400700 1 -- 172.21.15.31:0/1109088047 wait complete. 2024-03-21T10:36:19.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.603+0000 7fe6ed400700 1 Processor -- start 2024-03-21T10:36:19.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.603+0000 7fe6ed400700 1 -- start start 2024-03-21T10:36:19.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.604+0000 7fe6ed400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6e81044e0 0x7fe6e81a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:19.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.604+0000 7fe6ed400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6e8104e90 0x7fe6e81a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:19.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.604+0000 7fe6ed400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6e81095c0 0x7fe6e819a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:19.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.604+0000 7fe6ed400700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e81145f0 con 0x7fe6e81095c0 2024-03-21T10:36:19.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.604+0000 7fe6ed400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e8114470 con 0x7fe6e8104e90 2024-03-21T10:36:19.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.604+0000 7fe6ed400700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e8114770 con 0x7fe6e81044e0 2024-03-21T10:36:19.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.604+0000 7fe6e7e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6e81044e0 0x7fe6e81a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:19.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.604+0000 7fe6e7e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6e81044e0 0x7fe6e81a06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:35570/0 (socket says 172.21.15.31:35570) 2024-03-21T10:36:19.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.604+0000 7fe6e7e00700 1 -- 172.21.15.31:0/1207880576 learned_addr learned my addr 172.21.15.31:0/1207880576 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:36:19.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.604+0000 7fe6e7400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6e8104e90 0x7fe6e81a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:19.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.604+0000 7fe6eca00700 1 --2- 172.21.15.31:0/1207880576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6e81095c0 0x7fe6e819a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:19.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.604+0000 7fe6e7e00700 1 -- 172.21.15.31:0/1207880576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6e8104e90 msgr2=0x7fe6e81a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:19.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.604+0000 7fe6e7e00700 1 --2- 172.21.15.31:0/1207880576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6e8104e90 0x7fe6e81a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:19.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.604+0000 7fe6e7e00700 1 -- 172.21.15.31:0/1207880576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6e81095c0 msgr2=0x7fe6e819a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:19.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.604+0000 7fe6e7e00700 1 --2- 172.21.15.31:0/1207880576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6e81095c0 0x7fe6e819a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:19.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.604+0000 7fe6e7e00700 1 -- 172.21.15.31:0/1207880576 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e819b030 con 0x7fe6e81044e0 2024-03-21T10:36:19.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.605+0000 7fe6e7e00700 1 --2- 172.21.15.31:0/1207880576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6e81044e0 0x7fe6e81a06d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe6d800d950 tx=0x7fe6d800dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:19.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.605+0000 7fe6e4c00700 1 -- 172.21.15.31:0/1207880576 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d8016070 con 0x7fe6e81044e0 2024-03-21T10:36:19.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.605+0000 7fe6e4c00700 1 -- 172.21.15.31:0/1207880576 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6d8009e20 con 0x7fe6e81044e0 2024-03-21T10:36:19.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.605+0000 7fe6ed400700 1 -- 172.21.15.31:0/1207880576 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e819b320 con 0x7fe6e81044e0 2024-03-21T10:36:19.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.605+0000 7fe6e4c00700 1 -- 172.21.15.31:0/1207880576 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d800f620 con 0x7fe6e81044e0 2024-03-21T10:36:19.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.605+0000 7fe6ed400700 1 -- 172.21.15.31:0/1207880576 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e81a7530 con 0x7fe6e81044e0 2024-03-21T10:36:19.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.606+0000 7fe6ed400700 1 -- 172.21.15.31:0/1207880576 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe6e8062920 con 0x7fe6e81044e0 2024-03-21T10:36:19.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.608+0000 7fe6e4c00700 1 -- 172.21.15.31:0/1207880576 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7fe6d8019070 con 0x7fe6e81044e0 2024-03-21T10:36:19.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.609+0000 7fe6e4c00700 1 --2- 172.21.15.31:0/1207880576 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe6cc0a30c0 0x7fe6cc0a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:19.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.609+0000 7fe6e4c00700 1 -- 172.21.15.31:0/1207880576 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fe6d80cf3c0 con 0x7fe6e81044e0 2024-03-21T10:36:19.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.609+0000 7fe6e7400700 1 --2- 172.21.15.31:0/1207880576 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe6cc0a30c0 0x7fe6cc0a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:19.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.610+0000 7fe6e4c00700 1 --2- 172.21.15.31:0/1207880576 >> [v2:172.21.15.70:6816/534880586,v1:172.21.15.70:6817/534880586] conn(0x7fe6cc0ae1a0 0x7fe6cc0b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:19.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.610+0000 7fe6e4c00700 1 -- 172.21.15.31:0/1207880576 --> [v2:172.21.15.70:6816/534880586,v1:172.21.15.70:6817/534880586] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6d8010e50 con 0x7fe6cc0ae1a0 2024-03-21T10:36:19.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.610+0000 7fe6e4c00700 1 -- 172.21.15.31:0/1207880576 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6d80d6050 con 0x7fe6e81044e0 2024-03-21T10:36:19.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.610+0000 7fe6eca00700 1 --2- 172.21.15.31:0/1207880576 >> [v2:172.21.15.70:6816/534880586,v1:172.21.15.70:6817/534880586] conn(0x7fe6cc0ae1a0 0x7fe6cc0b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:19.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.610+0000 7fe6e7400700 1 --2- 172.21.15.31:0/1207880576 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe6cc0a30c0 0x7fe6cc0a5580 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe6e81034b0 tx=0x7fe6d400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:19.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.611+0000 7fe6eca00700 1 --2- 172.21.15.31:0/1207880576 >> [v2:172.21.15.70:6816/534880586,v1:172.21.15.70:6817/534880586] conn(0x7fe6cc0ae1a0 0x7fe6cc0b0640 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:19.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.611+0000 7fe6e4c00700 1 -- 172.21.15.31:0/1207880576 <== osd.8 v2:172.21.15.70:6816/534880586 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe6d8010e50 con 0x7fe6cc0ae1a0 2024-03-21T10:36:19.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.633+0000 7fe6ed400700 1 -- 172.21.15.31:0/1207880576 --> [v2:172.21.15.70:6816/534880586,v1:172.21.15.70:6817/534880586] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe6e8062d50 con 0x7fe6cc0ae1a0 2024-03-21T10:36:19.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.633+0000 7fe6e4c00700 1 -- 172.21.15.31:0/1207880576 <== osd.8 v2:172.21.15.70:6816/534880586 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe6e8062d50 con 0x7fe6cc0ae1a0 2024-03-21T10:36:19.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.633+0000 7fe6ed400700 1 -- 172.21.15.31:0/1207880576 >> [v2:172.21.15.70:6816/534880586,v1:172.21.15.70:6817/534880586] conn(0x7fe6cc0ae1a0 msgr2=0x7fe6cc0b0640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:19.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.633+0000 7fe6ed400700 1 --2- 172.21.15.31:0/1207880576 >> [v2:172.21.15.70:6816/534880586,v1:172.21.15.70:6817/534880586] conn(0x7fe6cc0ae1a0 0x7fe6cc0b0640 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:19.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.634+0000 7fe6ed400700 1 -- 172.21.15.31:0/1207880576 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe6cc0a30c0 msgr2=0x7fe6cc0a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:19.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.634+0000 7fe6ed400700 1 --2- 172.21.15.31:0/1207880576 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe6cc0a30c0 0x7fe6cc0a5580 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe6e81034b0 tx=0x7fe6d400b260 comp rx=0 tx=0).stop 2024-03-21T10:36:19.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.634+0000 7fe6ed400700 1 -- 172.21.15.31:0/1207880576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6e81044e0 msgr2=0x7fe6e81a06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:19.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.634+0000 7fe6ed400700 1 --2- 172.21.15.31:0/1207880576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6e81044e0 0x7fe6e81a06d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe6d800d950 tx=0x7fe6d800dc60 comp rx=0 tx=0).stop 2024-03-21T10:36:19.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.634+0000 7fe6eca00700 1 -- 172.21.15.31:0/1207880576 reap_dead start 2024-03-21T10:36:19.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.634+0000 7fe6ed400700 1 -- 172.21.15.31:0/1207880576 shutdown_connections 2024-03-21T10:36:19.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.634+0000 7fe6ed400700 1 -- 172.21.15.31:0/1207880576 >> 172.21.15.31:0/1207880576 conn(0x7fe6e80ffdb0 msgr2=0x7fe6e81008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:19.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.634+0000 7fe6ed400700 1 -- 172.21.15.31:0/1207880576 shutdown_connections 2024-03-21T10:36:19.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:14.634+0000 7fe6ed400700 1 -- 172.21.15.31:0/1207880576 wait complete. 2024-03-21T10:36:19.483 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:19 smithi040 ceph-mon[36154]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-21T10:36:19.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:19 smithi031 ceph-mon[31673]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-21T10:36:19.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:19 smithi070 ceph-mon[36330]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-21T10:36:20.058 INFO:teuthology.orchestra.run.smithi031.stdout:253403070510 2024-03-21T10:36:20.058 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-21T10:36:21.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:21 smithi031 ceph-mon[31673]: pgmap v364: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-21T10:36:21.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:21 smithi070 ceph-mon[36330]: pgmap v364: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-21T10:36:21.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:21 smithi040 ceph-mon[36154]: pgmap v364: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-21T10:36:22.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.678+0000 7efe14200700 1 -- 172.21.15.31:0/2111808566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe0c101090 msgr2=0x7efe0c1014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:22.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.678+0000 7efe14200700 1 --2- 172.21.15.31:0/2111808566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe0c101090 0x7efe0c1014f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efdf8009990 tx=0x7efdf8009ca0 comp rx=0 tx=0).stop 2024-03-21T10:36:22.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.678+0000 7efe14200700 1 -- 172.21.15.31:0/2111808566 shutdown_connections 2024-03-21T10:36:22.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.678+0000 7efe14200700 1 --2- 172.21.15.31:0/2111808566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c101a30 0x7efe0c10df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:22.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.678+0000 7efe14200700 1 --2- 172.21.15.31:0/2111808566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe0c1070d0 0x7efe0c1074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:22.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.678+0000 7efe14200700 1 --2- 172.21.15.31:0/2111808566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe0c101090 0x7efe0c1014f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:22.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.678+0000 7efe14200700 1 -- 172.21.15.31:0/2111808566 >> 172.21.15.31:0/2111808566 conn(0x7efe0c077fc0 msgr2=0x7efe0c0783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:22.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.678+0000 7efe14200700 1 -- 172.21.15.31:0/2111808566 shutdown_connections 2024-03-21T10:36:22.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.678+0000 7efe14200700 1 -- 172.21.15.31:0/2111808566 wait complete. 2024-03-21T10:36:22.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.679+0000 7efe14200700 1 Processor -- start 2024-03-21T10:36:22.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.680+0000 7efe14200700 1 -- start start 2024-03-21T10:36:22.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.680+0000 7efe14200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe0c101090 0x7efe0c19e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:22.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.680+0000 7efe14200700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe0c101a30 0x7efe0c19ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:22.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.680+0000 7efe14200700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c1070d0 0x7efe0c198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:22.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.680+0000 7efe14200700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c1124a0 con 0x7efe0c101a30 2024-03-21T10:36:22.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.680+0000 7efe14200700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c112320 con 0x7efe0c1070d0 2024-03-21T10:36:22.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.680+0000 7efe14200700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c112620 con 0x7efe0c101090 2024-03-21T10:36:22.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.680+0000 7efe11800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe0c101090 0x7efe0c19e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:22.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.680+0000 7efe11800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe0c101090 0x7efe0c19e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:59496/0 (socket says 172.21.15.31:59496) 2024-03-21T10:36:22.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.680+0000 7efe11800700 1 -- 172.21.15.31:0/413247385 learned_addr learned my addr 172.21.15.31:0/413247385 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:36:22.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.680+0000 7efe12200700 1 --2- 172.21.15.31:0/413247385 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c1070d0 0x7efe0c198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:22.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.681+0000 7efe10e00700 1 --2- 172.21.15.31:0/413247385 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe0c101a30 0x7efe0c19ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:22.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.681+0000 7efe11800700 1 -- 172.21.15.31:0/413247385 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c1070d0 msgr2=0x7efe0c198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:22.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.681+0000 7efe11800700 1 --2- 172.21.15.31:0/413247385 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c1070d0 0x7efe0c198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:22.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.681+0000 7efe11800700 1 -- 172.21.15.31:0/413247385 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe0c101a30 msgr2=0x7efe0c19ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:22.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.681+0000 7efe11800700 1 --2- 172.21.15.31:0/413247385 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe0c101a30 0x7efe0c19ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:22.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.681+0000 7efe11800700 1 -- 172.21.15.31:0/413247385 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe0c198fe0 con 0x7efe0c101090 2024-03-21T10:36:22.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.681+0000 7efe11800700 1 --2- 172.21.15.31:0/413247385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe0c101090 0x7efe0c19e680 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7efe0000d950 tx=0x7efe0000dd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:22.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.681+0000 7efe0a000700 1 -- 172.21.15.31:0/413247385 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe00016070 con 0x7efe0c101090 2024-03-21T10:36:22.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.682+0000 7efe0a000700 1 -- 172.21.15.31:0/413247385 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe00009e10 con 0x7efe0c101090 2024-03-21T10:36:22.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.682+0000 7efe0a000700 1 -- 172.21.15.31:0/413247385 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0000f620 con 0x7efe0c101090 2024-03-21T10:36:22.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.682+0000 7efe14200700 1 -- 172.21.15.31:0/413247385 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe0c1992d0 con 0x7efe0c101090 2024-03-21T10:36:22.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.682+0000 7efe14200700 1 -- 172.21.15.31:0/413247385 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe0c1a54e0 con 0x7efe0c101090 2024-03-21T10:36:22.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.683+0000 7efe14200700 1 -- 172.21.15.31:0/413247385 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efe0c0628a0 con 0x7efe0c101090 2024-03-21T10:36:22.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.685+0000 7efe0a000700 1 -- 172.21.15.31:0/413247385 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7efe00019070 con 0x7efe0c101090 2024-03-21T10:36:22.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.686+0000 7efe0a000700 1 --2- 172.21.15.31:0/413247385 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7efdf40a30c0 0x7efdf40a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:22.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.686+0000 7efe0a000700 1 -- 172.21.15.31:0/413247385 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7efe000cf650 con 0x7efe0c101090 2024-03-21T10:36:22.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.686+0000 7efe10e00700 1 --2- 172.21.15.31:0/413247385 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7efdf40a30c0 0x7efdf40a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:22.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.686+0000 7efe0a000700 1 --2- 172.21.15.31:0/413247385 >> [v2:172.21.15.70:6808/3402830452,v1:172.21.15.70:6809/3402830452] conn(0x7efdf40ae1a0 0x7efdf40b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:22.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.686+0000 7efe0a000700 1 -- 172.21.15.31:0/413247385 --> [v2:172.21.15.70:6808/3402830452,v1:172.21.15.70:6809/3402830452] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efe0000ee50 con 0x7efdf40ae1a0 2024-03-21T10:36:22.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.686+0000 7efe0a000700 1 -- 172.21.15.31:0/413247385 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7efe000cfa00 con 0x7efe0c101090 2024-03-21T10:36:22.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.687+0000 7efe12200700 1 --2- 172.21.15.31:0/413247385 >> [v2:172.21.15.70:6808/3402830452,v1:172.21.15.70:6809/3402830452] conn(0x7efdf40ae1a0 0x7efdf40b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:22.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.687+0000 7efe10e00700 1 --2- 172.21.15.31:0/413247385 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7efdf40a30c0 0x7efdf40a5580 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7efdf800b2e0 tx=0x7efdf801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:22.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.687+0000 7efe12200700 1 --2- 172.21.15.31:0/413247385 >> [v2:172.21.15.70:6808/3402830452,v1:172.21.15.70:6809/3402830452] conn(0x7efdf40ae1a0 0x7efdf40b0640 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:22.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.688+0000 7efe0a000700 1 -- 172.21.15.31:0/413247385 <== osd.5 v2:172.21.15.70:6808/3402830452 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7efe0000ee50 con 0x7efdf40ae1a0 2024-03-21T10:36:22.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.706+0000 7efe14200700 1 -- 172.21.15.31:0/413247385 --> [v2:172.21.15.70:6808/3402830452,v1:172.21.15.70:6809/3402830452] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efe0c062d70 con 0x7efdf40ae1a0 2024-03-21T10:36:22.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.706+0000 7efe0a000700 1 -- 172.21.15.31:0/413247385 <== osd.5 v2:172.21.15.70:6808/3402830452 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7efe0c062d70 con 0x7efdf40ae1a0 2024-03-21T10:36:22.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.707+0000 7efe14200700 1 -- 172.21.15.31:0/413247385 >> [v2:172.21.15.70:6808/3402830452,v1:172.21.15.70:6809/3402830452] conn(0x7efdf40ae1a0 msgr2=0x7efdf40b0640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:22.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.707+0000 7efe14200700 1 --2- 172.21.15.31:0/413247385 >> [v2:172.21.15.70:6808/3402830452,v1:172.21.15.70:6809/3402830452] conn(0x7efdf40ae1a0 0x7efdf40b0640 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:22.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.707+0000 7efe14200700 1 -- 172.21.15.31:0/413247385 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7efdf40a30c0 msgr2=0x7efdf40a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:22.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.707+0000 7efe14200700 1 --2- 172.21.15.31:0/413247385 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7efdf40a30c0 0x7efdf40a5580 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7efdf800b2e0 tx=0x7efdf801a040 comp rx=0 tx=0).stop 2024-03-21T10:36:22.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.707+0000 7efe14200700 1 -- 172.21.15.31:0/413247385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe0c101090 msgr2=0x7efe0c19e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:22.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.707+0000 7efe14200700 1 --2- 172.21.15.31:0/413247385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe0c101090 0x7efe0c19e680 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7efe0000d950 tx=0x7efe0000dd10 comp rx=0 tx=0).stop 2024-03-21T10:36:22.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.707+0000 7efe12200700 1 -- 172.21.15.31:0/413247385 reap_dead start 2024-03-21T10:36:22.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.707+0000 7efe14200700 1 -- 172.21.15.31:0/413247385 shutdown_connections 2024-03-21T10:36:22.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.707+0000 7efe14200700 1 -- 172.21.15.31:0/413247385 >> 172.21.15.31:0/413247385 conn(0x7efe0c077fc0 msgr2=0x7efe0c0ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:22.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.707+0000 7efe14200700 1 -- 172.21.15.31:0/413247385 shutdown_connections 2024-03-21T10:36:22.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:21.707+0000 7efe14200700 1 -- 172.21.15.31:0/413247385 wait complete. 2024-03-21T10:36:23.060 INFO:teuthology.orchestra.run.smithi031.stdout:309237645346 2024-03-21T10:36:23.060 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-21T10:36:23.286 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:23 smithi031 ceph-mon[31673]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-21T10:36:23.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:23 smithi070 ceph-mon[36330]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-21T10:36:23.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:23 smithi040 ceph-mon[36154]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-21T10:36:24.077 INFO:teuthology.orchestra.run.smithi031.stdout:287762808871 2024-03-21T10:36:24.077 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-21T10:36:25.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.286+0000 7fe5b9600700 1 -- 172.21.15.31:0/224932107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b4074dd0 msgr2=0x7fe5b410e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:25.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.286+0000 7fe5b9600700 1 --2- 172.21.15.31:0/224932107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b4074dd0 0x7fe5b410e4f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe5a00098f0 tx=0x7fe5a0009c00 comp rx=0 tx=0).stop 2024-03-21T10:36:25.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.286+0000 7fe5b9600700 1 -- 172.21.15.31:0/224932107 shutdown_connections 2024-03-21T10:36:25.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.286+0000 7fe5b9600700 1 --2- 172.21.15.31:0/224932107 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5b410ea30 0x7fe5b4110e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:25.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.286+0000 7fe5b9600700 1 --2- 172.21.15.31:0/224932107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5b40744b0 0x7fe5b4074890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:25.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.286+0000 7fe5b9600700 1 --2- 172.21.15.31:0/224932107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b4074dd0 0x7fe5b410e4f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:25.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.286+0000 7fe5b9600700 1 -- 172.21.15.31:0/224932107 >> 172.21.15.31:0/224932107 conn(0x7fe5b40fdc40 msgr2=0x7fe5b4100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:25.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.286+0000 7fe5b9600700 1 -- 172.21.15.31:0/224932107 shutdown_connections 2024-03-21T10:36:25.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.286+0000 7fe5b9600700 1 -- 172.21.15.31:0/224932107 wait complete. 2024-03-21T10:36:25.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.288+0000 7fe5b9600700 1 Processor -- start 2024-03-21T10:36:25.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.288+0000 7fe5b9600700 1 -- start start 2024-03-21T10:36:25.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.289+0000 7fe5b9600700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5b40744b0 0x7fe5b41a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:25.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.289+0000 7fe5b9600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b4074dd0 0x7fe5b41a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:25.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.289+0000 7fe5b9600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5b410ea30 0x7fe5b419aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:25.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.289+0000 7fe5b9600700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b41146d0 con 0x7fe5b40744b0 2024-03-21T10:36:25.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.289+0000 7fe5b9600700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b4114550 con 0x7fe5b4074dd0 2024-03-21T10:36:25.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.289+0000 7fe5b9600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b4114850 con 0x7fe5b410ea30 2024-03-21T10:36:25.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.289+0000 7fe5b2000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b4074dd0 0x7fe5b41a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:25.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.289+0000 7fe5b2000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b4074dd0 0x7fe5b41a0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:45518/0 (socket says 172.21.15.31:45518) 2024-03-21T10:36:25.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.289+0000 7fe5b2000700 1 -- 172.21.15.31:0/2265777851 learned_addr learned my addr 172.21.15.31:0/2265777851 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:36:25.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.289+0000 7fe5b2a00700 1 --2- 172.21.15.31:0/2265777851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5b40744b0 0x7fe5b41a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:25.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.289+0000 7fe5b3400700 1 --2- 172.21.15.31:0/2265777851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5b410ea30 0x7fe5b419aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:25.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.289+0000 7fe5b2000700 1 -- 172.21.15.31:0/2265777851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5b410ea30 msgr2=0x7fe5b419aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:25.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.289+0000 7fe5b2000700 1 --2- 172.21.15.31:0/2265777851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5b410ea30 0x7fe5b419aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:25.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.289+0000 7fe5b2000700 1 -- 172.21.15.31:0/2265777851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5b40744b0 msgr2=0x7fe5b41a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:25.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.289+0000 7fe5b2000700 1 --2- 172.21.15.31:0/2265777851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5b40744b0 0x7fe5b41a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:25.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.289+0000 7fe5b2000700 1 -- 172.21.15.31:0/2265777851 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b419b190 con 0x7fe5b4074dd0 2024-03-21T10:36:25.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.289+0000 7fe5b3400700 1 --2- 172.21.15.31:0/2265777851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5b410ea30 0x7fe5b419aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:36:25.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.289+0000 7fe5b2a00700 1 --2- 172.21.15.31:0/2265777851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5b40744b0 0x7fe5b41a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:36:25.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.290+0000 7fe5b2000700 1 --2- 172.21.15.31:0/2265777851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b4074dd0 0x7fe5b41a0d60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0005950 tx=0x7fe5a00050f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:25.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.290+0000 7fe5ab400700 1 -- 172.21.15.31:0/2265777851 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a0005390 con 0x7fe5b4074dd0 2024-03-21T10:36:25.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.290+0000 7fe5ab400700 1 -- 172.21.15.31:0/2265777851 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe5a0005530 con 0x7fe5b4074dd0 2024-03-21T10:36:25.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.290+0000 7fe5ab400700 1 -- 172.21.15.31:0/2265777851 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a0003d40 con 0x7fe5b4074dd0 2024-03-21T10:36:25.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.290+0000 7fe5b9600700 1 -- 172.21.15.31:0/2265777851 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5b419b420 con 0x7fe5b4074dd0 2024-03-21T10:36:25.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.290+0000 7fe5b9600700 1 -- 172.21.15.31:0/2265777851 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5b41a76f0 con 0x7fe5b4074dd0 2024-03-21T10:36:25.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.292+0000 7fe5b9600700 1 -- 172.21.15.31:0/2265777851 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe5b4062920 con 0x7fe5b4074dd0 2024-03-21T10:36:25.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.296+0000 7fe5ab400700 1 -- 172.21.15.31:0/2265777851 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7fe5a000b560 con 0x7fe5b4074dd0 2024-03-21T10:36:25.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.297+0000 7fe5ab400700 1 --2- 172.21.15.31:0/2265777851 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe5980a30c0 0x7fe5980a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:25.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.298+0000 7fe5ab400700 1 -- 172.21.15.31:0/2265777851 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fe5a00d12a0 con 0x7fe5b4074dd0 2024-03-21T10:36:25.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.298+0000 7fe5b2a00700 1 --2- 172.21.15.31:0/2265777851 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe5980a30c0 0x7fe5980a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:25.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.298+0000 7fe5ab400700 1 --2- 172.21.15.31:0/2265777851 >> [v2:172.21.15.31:6818/1193497058,v1:172.21.15.31:6819/1193497058] conn(0x7fe5980ae1a0 0x7fe5980b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:25.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.298+0000 7fe5ab400700 1 -- 172.21.15.31:0/2265777851 --> [v2:172.21.15.31:6818/1193497058,v1:172.21.15.31:6819/1193497058] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe5a0024e70 con 0x7fe5980ae1a0 2024-03-21T10:36:25.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.298+0000 7fe5ab400700 1 -- 172.21.15.31:0/2265777851 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5a00d1650 con 0x7fe5b4074dd0 2024-03-21T10:36:25.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.298+0000 7fe5b3400700 1 --2- 172.21.15.31:0/2265777851 >> [v2:172.21.15.31:6818/1193497058,v1:172.21.15.31:6819/1193497058] conn(0x7fe5980ae1a0 0x7fe5980b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:25.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.299+0000 7fe5b2a00700 1 --2- 172.21.15.31:0/2265777851 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe5980a30c0 0x7fe5980a5580 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe5b4101810 tx=0x7fe5a4009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:25.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.299+0000 7fe5b3400700 1 --2- 172.21.15.31:0/2265777851 >> [v2:172.21.15.31:6818/1193497058,v1:172.21.15.31:6819/1193497058] conn(0x7fe5980ae1a0 0x7fe5980b0640 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:25.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.301+0000 7fe5ab400700 1 -- 172.21.15.31:0/2265777851 <== osd.6 v2:172.21.15.31:6818/1193497058 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe5a0024e70 con 0x7fe5980ae1a0 2024-03-21T10:36:25.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.319+0000 7fe5b9600700 1 -- 172.21.15.31:0/2265777851 --> [v2:172.21.15.31:6818/1193497058,v1:172.21.15.31:6819/1193497058] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe5b4062d20 con 0x7fe5980ae1a0 2024-03-21T10:36:25.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.320+0000 7fe5ab400700 1 -- 172.21.15.31:0/2265777851 <== osd.6 v2:172.21.15.31:6818/1193497058 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe5b4062d20 con 0x7fe5980ae1a0 2024-03-21T10:36:25.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.320+0000 7fe5b9600700 1 -- 172.21.15.31:0/2265777851 >> [v2:172.21.15.31:6818/1193497058,v1:172.21.15.31:6819/1193497058] conn(0x7fe5980ae1a0 msgr2=0x7fe5980b0640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:25.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.320+0000 7fe5b9600700 1 --2- 172.21.15.31:0/2265777851 >> [v2:172.21.15.31:6818/1193497058,v1:172.21.15.31:6819/1193497058] conn(0x7fe5980ae1a0 0x7fe5980b0640 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:25.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.320+0000 7fe5b9600700 1 -- 172.21.15.31:0/2265777851 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe5980a30c0 msgr2=0x7fe5980a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:25.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.320+0000 7fe5b9600700 1 --2- 172.21.15.31:0/2265777851 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe5980a30c0 0x7fe5980a5580 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe5b4101810 tx=0x7fe5a4009310 comp rx=0 tx=0).stop 2024-03-21T10:36:25.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.320+0000 7fe5b9600700 1 -- 172.21.15.31:0/2265777851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b4074dd0 msgr2=0x7fe5b41a0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:25.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.320+0000 7fe5b9600700 1 --2- 172.21.15.31:0/2265777851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5b4074dd0 0x7fe5b41a0d60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0005950 tx=0x7fe5a00050f0 comp rx=0 tx=0).stop 2024-03-21T10:36:25.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.320+0000 7fe5b3400700 1 -- 172.21.15.31:0/2265777851 reap_dead start 2024-03-21T10:36:25.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.321+0000 7fe5b9600700 1 -- 172.21.15.31:0/2265777851 shutdown_connections 2024-03-21T10:36:25.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.321+0000 7fe5b9600700 1 -- 172.21.15.31:0/2265777851 >> 172.21.15.31:0/2265777851 conn(0x7fe5b40fdc40 msgr2=0x7fe5b40fe950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:25.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.321+0000 7fe5b9600700 1 -- 172.21.15.31:0/2265777851 shutdown_connections 2024-03-21T10:36:25.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:23.321+0000 7fe5b9600700 1 -- 172.21.15.31:0/2265777851 wait complete. 2024-03-21T10:36:25.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:25 smithi031 ceph-mon[31673]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-21T10:36:25.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:25 smithi070 ceph-mon[36330]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-21T10:36:25.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:25 smithi040 ceph-mon[36154]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-21T10:36:25.931 INFO:teuthology.orchestra.run.smithi031.stdout:163208757312 2024-03-21T10:36:25.931 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-21T10:36:26.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.068+0000 7f93e3000700 1 -- 172.21.15.31:0/1544509176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93dc1081e0 msgr2=0x7f93dc1085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:26.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.068+0000 7f93e3000700 1 --2- 172.21.15.31:0/1544509176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93dc1081e0 0x7f93dc1085c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f93cc009920 tx=0x7f93cc009c30 comp rx=0 tx=0).stop 2024-03-21T10:36:26.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.069+0000 7f93e3000700 1 -- 172.21.15.31:0/1544509176 shutdown_connections 2024-03-21T10:36:26.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.069+0000 7f93e3000700 1 --2- 172.21.15.31:0/1544509176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93dc06a7e0 0x7f93dc10f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:26.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.069+0000 7f93e3000700 1 --2- 172.21.15.31:0/1544509176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93dc1081e0 0x7f93dc1085c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:26.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.069+0000 7f93e3000700 1 --2- 172.21.15.31:0/1544509176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93dc069e40 0x7f93dc06a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:26.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.069+0000 7f93e3000700 1 -- 172.21.15.31:0/1544509176 >> 172.21.15.31:0/1544509176 conn(0x7f93dc077fc0 msgr2=0x7f93dc0783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:26.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.069+0000 7f93e3000700 1 -- 172.21.15.31:0/1544509176 shutdown_connections 2024-03-21T10:36:26.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.069+0000 7f93e3000700 1 -- 172.21.15.31:0/1544509176 wait complete. 2024-03-21T10:36:26.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.070+0000 7f93e3000700 1 Processor -- start 2024-03-21T10:36:26.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.071+0000 7f93e3000700 1 -- start start 2024-03-21T10:36:26.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.071+0000 7f93e3000700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93dc069e40 0x7f93dc19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:26.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.071+0000 7f93e3000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93dc06a7e0 0x7f93dc19ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:26.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.071+0000 7f93e3000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93dc1081e0 0x7f93dc1987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:26.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.071+0000 7f93e3000700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc112490 con 0x7f93dc069e40 2024-03-21T10:36:26.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.071+0000 7f93e3000700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc112310 con 0x7f93dc1081e0 2024-03-21T10:36:26.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.071+0000 7f93e3000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc112610 con 0x7f93dc06a7e0 2024-03-21T10:36:26.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.072+0000 7f93db400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93dc06a7e0 0x7f93dc19ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:26.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.072+0000 7f93dbe00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93dc069e40 0x7f93dc19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:26.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.072+0000 7f93dbe00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93dc069e40 0x7f93dc19e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47246/0 (socket says 172.21.15.31:47246) 2024-03-21T10:36:26.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.072+0000 7f93dbe00700 1 -- 172.21.15.31:0/2214127256 learned_addr learned my addr 172.21.15.31:0/2214127256 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:36:26.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.072+0000 7f93db400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93dc06a7e0 0x7f93dc19ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:59546/0 (socket says 172.21.15.31:59546) 2024-03-21T10:36:26.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.072+0000 7f93e1000700 1 --2- 172.21.15.31:0/2214127256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93dc1081e0 0x7f93dc1987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:26.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.072+0000 7f93db400700 1 -- 172.21.15.31:0/2214127256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93dc1081e0 msgr2=0x7f93dc1987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:26.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.072+0000 7f93db400700 1 --2- 172.21.15.31:0/2214127256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93dc1081e0 0x7f93dc1987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:26.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.072+0000 7f93db400700 1 -- 172.21.15.31:0/2214127256 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93dc069e40 msgr2=0x7f93dc19e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:26.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.072+0000 7f93db400700 1 --2- 172.21.15.31:0/2214127256 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93dc069e40 0x7f93dc19e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:26.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.072+0000 7f93db400700 1 -- 172.21.15.31:0/2214127256 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93dc198fd0 con 0x7f93dc06a7e0 2024-03-21T10:36:26.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.072+0000 7f93db400700 1 --2- 172.21.15.31:0/2214127256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93dc06a7e0 0x7f93dc19ebb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f93c800e9c0 tx=0x7f93c800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:26.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.073+0000 7f93d8c00700 1 -- 172.21.15.31:0/2214127256 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c8004220 con 0x7f93dc06a7e0 2024-03-21T10:36:26.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.073+0000 7f93d8c00700 1 -- 172.21.15.31:0/2214127256 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93c8004d10 con 0x7f93dc06a7e0 2024-03-21T10:36:26.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.073+0000 7f93d8c00700 1 -- 172.21.15.31:0/2214127256 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c800ce60 con 0x7f93dc06a7e0 2024-03-21T10:36:26.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.073+0000 7f93e3000700 1 -- 172.21.15.31:0/2214127256 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93dc1992c0 con 0x7f93dc06a7e0 2024-03-21T10:36:26.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.073+0000 7f93e3000700 1 -- 172.21.15.31:0/2214127256 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93dc1a54d0 con 0x7f93dc06a7e0 2024-03-21T10:36:26.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.074+0000 7f93e3000700 1 -- 172.21.15.31:0/2214127256 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f93dc062920 con 0x7f93dc06a7e0 2024-03-21T10:36:26.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.077+0000 7f93d8c00700 1 -- 172.21.15.31:0/2214127256 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7f93c8004750 con 0x7f93dc06a7e0 2024-03-21T10:36:26.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.078+0000 7f93d8c00700 1 --2- 172.21.15.31:0/2214127256 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f93c00a30c0 0x7f93c00a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:26.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.078+0000 7f93d8c00700 1 -- 172.21.15.31:0/2214127256 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f93c80cf900 con 0x7f93dc06a7e0 2024-03-21T10:36:26.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.079+0000 7f93dbe00700 1 --2- 172.21.15.31:0/2214127256 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f93c00a30c0 0x7f93c00a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:26.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.079+0000 7f93d8c00700 1 --2- 172.21.15.31:0/2214127256 >> [v2:172.21.15.70:6800/2524184117,v1:172.21.15.70:6801/2524184117] conn(0x7f93c00ae1a0 0x7f93c00b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:26.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.079+0000 7f93d8c00700 1 -- 172.21.15.31:0/2214127256 --> [v2:172.21.15.70:6800/2524184117,v1:172.21.15.70:6801/2524184117] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f93c00b0d10 con 0x7f93c00ae1a0 2024-03-21T10:36:26.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.079+0000 7f93d8c00700 1 -- 172.21.15.31:0/2214127256 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93c80cfcb0 con 0x7f93dc06a7e0 2024-03-21T10:36:26.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.079+0000 7f93e1000700 1 --2- 172.21.15.31:0/2214127256 >> [v2:172.21.15.70:6800/2524184117,v1:172.21.15.70:6801/2524184117] conn(0x7f93c00ae1a0 0x7f93c00b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:26.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.079+0000 7f93dbe00700 1 --2- 172.21.15.31:0/2214127256 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f93c00a30c0 0x7f93c00a5580 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f93cc009680 tx=0x7f93cc019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:26.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.080+0000 7f93e1000700 1 --2- 172.21.15.31:0/2214127256 >> [v2:172.21.15.70:6800/2524184117,v1:172.21.15.70:6801/2524184117] conn(0x7f93c00ae1a0 0x7f93c00b0640 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:26.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.081+0000 7f93d8c00700 1 -- 172.21.15.31:0/2214127256 <== osd.2 v2:172.21.15.70:6800/2524184117 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f93c00b0d10 con 0x7f93c00ae1a0 2024-03-21T10:36:26.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.101+0000 7f93e3000700 1 -- 172.21.15.31:0/2214127256 --> [v2:172.21.15.70:6800/2524184117,v1:172.21.15.70:6801/2524184117] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f93dc062da0 con 0x7f93c00ae1a0 2024-03-21T10:36:26.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.102+0000 7f93d8c00700 1 -- 172.21.15.31:0/2214127256 <== osd.2 v2:172.21.15.70:6800/2524184117 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f93dc062da0 con 0x7f93c00ae1a0 2024-03-21T10:36:26.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.102+0000 7f93e3000700 1 -- 172.21.15.31:0/2214127256 >> [v2:172.21.15.70:6800/2524184117,v1:172.21.15.70:6801/2524184117] conn(0x7f93c00ae1a0 msgr2=0x7f93c00b0640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:26.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.102+0000 7f93e3000700 1 --2- 172.21.15.31:0/2214127256 >> [v2:172.21.15.70:6800/2524184117,v1:172.21.15.70:6801/2524184117] conn(0x7f93c00ae1a0 0x7f93c00b0640 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:26.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.103+0000 7f93e3000700 1 -- 172.21.15.31:0/2214127256 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f93c00a30c0 msgr2=0x7f93c00a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:26.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.103+0000 7f93e3000700 1 --2- 172.21.15.31:0/2214127256 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f93c00a30c0 0x7f93c00a5580 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f93cc009680 tx=0x7f93cc019040 comp rx=0 tx=0).stop 2024-03-21T10:36:26.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.103+0000 7f93e3000700 1 -- 172.21.15.31:0/2214127256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93dc06a7e0 msgr2=0x7f93dc19ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:26.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.103+0000 7f93e3000700 1 --2- 172.21.15.31:0/2214127256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93dc06a7e0 0x7f93dc19ebb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f93c800e9c0 tx=0x7f93c800ecd0 comp rx=0 tx=0).stop 2024-03-21T10:36:26.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.103+0000 7f93e1000700 1 -- 172.21.15.31:0/2214127256 reap_dead start 2024-03-21T10:36:26.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.103+0000 7f93e3000700 1 -- 172.21.15.31:0/2214127256 shutdown_connections 2024-03-21T10:36:26.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.103+0000 7f93e3000700 1 -- 172.21.15.31:0/2214127256 >> 172.21.15.31:0/2214127256 conn(0x7f93dc077fc0 msgr2=0x7f93dc0ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:26.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.103+0000 7f93e3000700 1 -- 172.21.15.31:0/2214127256 shutdown_connections 2024-03-21T10:36:26.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:24.103+0000 7f93e3000700 1 -- 172.21.15.31:0/2214127256 wait complete. 2024-03-21T10:36:27.184 INFO:teuthology.orchestra.run.smithi031.stdout:197568495675 2024-03-21T10:36:27.185 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-21T10:36:27.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:27 smithi031 ceph-mon[31673]: pgmap v367: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-21T10:36:27.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:27 smithi070 ceph-mon[36330]: pgmap v367: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-21T10:36:27.632 INFO:teuthology.orchestra.run.smithi031.stdout:77309411409 2024-03-21T10:36:27.633 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-21T10:36:27.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:27 smithi040 ceph-mon[36154]: pgmap v367: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-21T10:36:29.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.173+0000 7ff811600700 1 -- 172.21.15.31:0/401295981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff80c107510 msgr2=0x7ff80c10f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:29.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.173+0000 7ff811600700 1 --2- 172.21.15.31:0/401295981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff80c107510 0x7ff80c10f0e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff80400b0d0 tx=0x7ff80400b3e0 comp rx=0 tx=0).stop 2024-03-21T10:36:29.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.173+0000 7ff811600700 1 -- 172.21.15.31:0/401295981 shutdown_connections 2024-03-21T10:36:29.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.173+0000 7ff811600700 1 --2- 172.21.15.31:0/401295981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff80c107510 0x7ff80c10f0e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:29.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.173+0000 7ff811600700 1 --2- 172.21.15.31:0/401295981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff80c102440 0x7ff80c102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:29.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.173+0000 7ff811600700 1 --2- 172.21.15.31:0/401295981 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff80c102df0 0x7ff80c106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:29.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.173+0000 7ff811600700 1 -- 172.21.15.31:0/401295981 >> 172.21.15.31:0/401295981 conn(0x7ff80c0792f0 msgr2=0x7ff80c079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:29.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.173+0000 7ff811600700 1 -- 172.21.15.31:0/401295981 shutdown_connections 2024-03-21T10:36:29.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.173+0000 7ff811600700 1 -- 172.21.15.31:0/401295981 wait complete. 2024-03-21T10:36:29.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.174+0000 7ff811600700 1 Processor -- start 2024-03-21T10:36:29.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.174+0000 7ff811600700 1 -- start start 2024-03-21T10:36:29.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.174+0000 7ff811600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff80c102440 0x7ff80c074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:29.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.174+0000 7ff811600700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff80c102df0 0x7ff80c0749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:29.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.174+0000 7ff811600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff80c107510 0x7ff80c074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:29.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.174+0000 7ff811600700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80c112580 con 0x7ff80c102df0 2024-03-21T10:36:29.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.174+0000 7ff811600700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80c112400 con 0x7ff80c107510 2024-03-21T10:36:29.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.174+0000 7ff811600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80c112700 con 0x7ff80c102440 2024-03-21T10:36:29.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.175+0000 7ff80aa00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff80c102440 0x7ff80c074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:29.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.175+0000 7ff80aa00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff80c102440 0x7ff80c074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53766/0 (socket says 172.21.15.31:53766) 2024-03-21T10:36:29.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.175+0000 7ff80aa00700 1 -- 172.21.15.31:0/994335856 learned_addr learned my addr 172.21.15.31:0/994335856 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:36:29.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.175+0000 7ff803e00700 1 --2- 172.21.15.31:0/994335856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff80c102df0 0x7ff80c0749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:29.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.175+0000 7ff80b400700 1 --2- 172.21.15.31:0/994335856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff80c107510 0x7ff80c074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:29.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.175+0000 7ff80aa00700 1 -- 172.21.15.31:0/994335856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff80c107510 msgr2=0x7ff80c074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:29.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.175+0000 7ff80aa00700 1 --2- 172.21.15.31:0/994335856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff80c107510 0x7ff80c074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:29.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.175+0000 7ff80aa00700 1 -- 172.21.15.31:0/994335856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff80c102df0 msgr2=0x7ff80c0749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:29.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.175+0000 7ff80aa00700 1 --2- 172.21.15.31:0/994335856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff80c102df0 0x7ff80c0749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:29.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.175+0000 7ff80aa00700 1 -- 172.21.15.31:0/994335856 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80c1a6ee0 con 0x7ff80c102440 2024-03-21T10:36:29.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.175+0000 7ff80b400700 1 --2- 172.21.15.31:0/994335856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff80c107510 0x7ff80c074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:36:29.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.176+0000 7ff80aa00700 1 --2- 172.21.15.31:0/994335856 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff80c102440 0x7ff80c074490 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc00e9c0 tx=0x7ff7fc00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:29.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.176+0000 7ff803400700 1 -- 172.21.15.31:0/994335856 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc00cbd0 con 0x7ff80c102440 2024-03-21T10:36:29.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.176+0000 7ff803400700 1 -- 172.21.15.31:0/994335856 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff7fc00ee60 con 0x7ff80c102440 2024-03-21T10:36:29.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.176+0000 7ff803400700 1 -- 172.21.15.31:0/994335856 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc005630 con 0x7ff80c102440 2024-03-21T10:36:29.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.176+0000 7ff811600700 1 -- 172.21.15.31:0/994335856 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff80c1a71d0 con 0x7ff80c102440 2024-03-21T10:36:29.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.176+0000 7ff811600700 1 -- 172.21.15.31:0/994335856 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff80c1a76c0 con 0x7ff80c102440 2024-03-21T10:36:29.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.177+0000 7ff811600700 1 -- 172.21.15.31:0/994335856 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff80c062860 con 0x7ff80c102440 2024-03-21T10:36:29.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.179+0000 7ff803400700 1 -- 172.21.15.31:0/994335856 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7ff7fc004750 con 0x7ff80c102440 2024-03-21T10:36:29.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.181+0000 7ff803400700 1 --2- 172.21.15.31:0/994335856 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff7f00a30c0 0x7ff7f00a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:29.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.181+0000 7ff803400700 1 -- 172.21.15.31:0/994335856 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7ff7fc0cff60 con 0x7ff80c102440 2024-03-21T10:36:29.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.181+0000 7ff803e00700 1 --2- 172.21.15.31:0/994335856 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff7f00a30c0 0x7ff7f00a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:29.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.181+0000 7ff803400700 1 --2- 172.21.15.31:0/994335856 >> [v2:172.21.15.40:6808/4215186309,v1:172.21.15.40:6809/4215186309] conn(0x7ff7f00ae1a0 0x7ff7f00b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:29.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.181+0000 7ff803400700 1 -- 172.21.15.31:0/994335856 --> [v2:172.21.15.40:6808/4215186309,v1:172.21.15.40:6809/4215186309] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff7fc01ee50 con 0x7ff7f00ae1a0 2024-03-21T10:36:29.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.181+0000 7ff803400700 1 -- 172.21.15.31:0/994335856 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff7fc0d0310 con 0x7ff80c102440 2024-03-21T10:36:29.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.181+0000 7ff80b400700 1 --2- 172.21.15.31:0/994335856 >> [v2:172.21.15.40:6808/4215186309,v1:172.21.15.40:6809/4215186309] conn(0x7ff7f00ae1a0 0x7ff7f00b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:29.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.182+0000 7ff803e00700 1 --2- 172.21.15.31:0/994335856 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff7f00a30c0 0x7ff7f00a5580 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff80c108970 tx=0x7ff7f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:29.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.182+0000 7ff80b400700 1 --2- 172.21.15.31:0/994335856 >> [v2:172.21.15.40:6808/4215186309,v1:172.21.15.40:6809/4215186309] conn(0x7ff7f00ae1a0 0x7ff7f00b0640 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:29.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.183+0000 7ff803400700 1 -- 172.21.15.31:0/994335856 <== osd.4 v2:172.21.15.40:6808/4215186309 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff7fc01ee50 con 0x7ff7f00ae1a0 2024-03-21T10:36:29.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.202+0000 7ff811600700 1 -- 172.21.15.31:0/994335856 --> [v2:172.21.15.40:6808/4215186309,v1:172.21.15.40:6809/4215186309] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff80c062c60 con 0x7ff7f00ae1a0 2024-03-21T10:36:29.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.203+0000 7ff803400700 1 -- 172.21.15.31:0/994335856 <== osd.4 v2:172.21.15.40:6808/4215186309 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff80c062c60 con 0x7ff7f00ae1a0 2024-03-21T10:36:29.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.203+0000 7ff811600700 1 -- 172.21.15.31:0/994335856 >> [v2:172.21.15.40:6808/4215186309,v1:172.21.15.40:6809/4215186309] conn(0x7ff7f00ae1a0 msgr2=0x7ff7f00b0640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:29.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.203+0000 7ff811600700 1 --2- 172.21.15.31:0/994335856 >> [v2:172.21.15.40:6808/4215186309,v1:172.21.15.40:6809/4215186309] conn(0x7ff7f00ae1a0 0x7ff7f00b0640 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:29.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.203+0000 7ff811600700 1 -- 172.21.15.31:0/994335856 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff7f00a30c0 msgr2=0x7ff7f00a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:29.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.203+0000 7ff811600700 1 --2- 172.21.15.31:0/994335856 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff7f00a30c0 0x7ff7f00a5580 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff80c108970 tx=0x7ff7f4009260 comp rx=0 tx=0).stop 2024-03-21T10:36:29.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.203+0000 7ff811600700 1 -- 172.21.15.31:0/994335856 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff80c102440 msgr2=0x7ff80c074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:29.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.203+0000 7ff811600700 1 --2- 172.21.15.31:0/994335856 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff80c102440 0x7ff80c074490 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc00e9c0 tx=0x7ff7fc00ecd0 comp rx=0 tx=0).stop 2024-03-21T10:36:29.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.203+0000 7ff80b400700 1 -- 172.21.15.31:0/994335856 reap_dead start 2024-03-21T10:36:29.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.204+0000 7ff811600700 1 -- 172.21.15.31:0/994335856 shutdown_connections 2024-03-21T10:36:29.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.204+0000 7ff811600700 1 -- 172.21.15.31:0/994335856 >> 172.21.15.31:0/994335856 conn(0x7ff80c0792f0 msgr2=0x7ff80c1001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:29.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.204+0000 7ff811600700 1 -- 172.21.15.31:0/994335856 shutdown_connections 2024-03-21T10:36:29.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:29.204+0000 7ff811600700 1 -- 172.21.15.31:0/994335856 wait complete. 2024-03-21T10:36:29.300 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:29 smithi031 ceph-mon[31673]: pgmap v368: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:36:29.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:29 smithi070 ceph-mon[36330]: pgmap v368: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:36:29.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:29 smithi040 ceph-mon[36154]: pgmap v368: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:36:30.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.003+0000 7fcca8800700 1 -- 172.21.15.31:0/1510576593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcca010ea50 msgr2=0x7fcca0110e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:30.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.003+0000 7fcca8800700 1 --2- 172.21.15.31:0/1510576593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcca010ea50 0x7fcca0110e60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c00b0a0 tx=0x7fcc9c00b3b0 comp rx=0 tx=0).stop 2024-03-21T10:36:30.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.003+0000 7fcca8800700 1 -- 172.21.15.31:0/1510576593 shutdown_connections 2024-03-21T10:36:30.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.003+0000 7fcca8800700 1 --2- 172.21.15.31:0/1510576593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcca010ea50 0x7fcca0110e60 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:30.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.003+0000 7fcca8800700 1 --2- 172.21.15.31:0/1510576593 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcca0074560 0x7fcca0074940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:30.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.003+0000 7fcca8800700 1 --2- 172.21.15.31:0/1510576593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcca0074e80 0x7fcca010e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:30.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.003+0000 7fcca8800700 1 -- 172.21.15.31:0/1510576593 >> 172.21.15.31:0/1510576593 conn(0x7fcca00fdc60 msgr2=0x7fcca0100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:30.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.003+0000 7fcca8800700 1 -- 172.21.15.31:0/1510576593 shutdown_connections 2024-03-21T10:36:30.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.003+0000 7fcca8800700 1 -- 172.21.15.31:0/1510576593 wait complete. 2024-03-21T10:36:30.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.004+0000 7fcca8800700 1 Processor -- start 2024-03-21T10:36:30.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.004+0000 7fcca8800700 1 -- start start 2024-03-21T10:36:30.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.005+0000 7fcca8800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcca0074560 0x7fcca019ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:30.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.005+0000 7fcca8800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcca0074e80 0x7fcca019b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:30.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.005+0000 7fcca8800700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcca010ea50 0x7fcca019f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:30.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.005+0000 7fcca8800700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca0114740 con 0x7fcca010ea50 2024-03-21T10:36:30.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.005+0000 7fcca8800700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca01145c0 con 0x7fcca0074e80 2024-03-21T10:36:30.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.005+0000 7fcca8800700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca01148c0 con 0x7fcca0074560 2024-03-21T10:36:30.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.005+0000 7fcca5e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcca0074560 0x7fcca019ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:30.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.005+0000 7fcca5400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcca0074e80 0x7fcca019b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:30.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.005+0000 7fcca5e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcca0074560 0x7fcca019ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53788/0 (socket says 172.21.15.31:53788) 2024-03-21T10:36:30.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.005+0000 7fcca5e00700 1 -- 172.21.15.31:0/4171233656 learned_addr learned my addr 172.21.15.31:0/4171233656 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:36:30.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.005+0000 7fcca6800700 1 --2- 172.21.15.31:0/4171233656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcca010ea50 0x7fcca019f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:30.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.006+0000 7fcca5e00700 1 -- 172.21.15.31:0/4171233656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcca0074e80 msgr2=0x7fcca019b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:30.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.006+0000 7fcca5e00700 1 --2- 172.21.15.31:0/4171233656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcca0074e80 0x7fcca019b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:30.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.006+0000 7fcca5e00700 1 -- 172.21.15.31:0/4171233656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcca010ea50 msgr2=0x7fcca019f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:30.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.006+0000 7fcca5e00700 1 --2- 172.21.15.31:0/4171233656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcca010ea50 0x7fcca019f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:30.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.006+0000 7fcca5e00700 1 -- 172.21.15.31:0/4171233656 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca019fc10 con 0x7fcca0074560 2024-03-21T10:36:30.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.006+0000 7fcca6800700 1 --2- 172.21.15.31:0/4171233656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcca010ea50 0x7fcca019f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:36:30.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.006+0000 7fcca5e00700 1 --2- 172.21.15.31:0/4171233656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcca0074560 0x7fcca019ac10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcc9400e9a0 tx=0x7fcc9400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:30.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.006+0000 7fcc9aa00700 1 -- 172.21.15.31:0/4171233656 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9400cbd0 con 0x7fcca0074560 2024-03-21T10:36:30.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.006+0000 7fcc9aa00700 1 -- 172.21.15.31:0/4171233656 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc9400cd70 con 0x7fcca0074560 2024-03-21T10:36:30.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.006+0000 7fcc9aa00700 1 -- 172.21.15.31:0/4171233656 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc94018750 con 0x7fcca0074560 2024-03-21T10:36:30.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.006+0000 7fcca8800700 1 -- 172.21.15.31:0/4171233656 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcca019ff00 con 0x7fcca0074560 2024-03-21T10:36:30.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.007+0000 7fcca8800700 1 -- 172.21.15.31:0/4171233656 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcca01a77c0 con 0x7fcca0074560 2024-03-21T10:36:30.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.007+0000 7fcca8800700 1 -- 172.21.15.31:0/4171233656 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcca0110db0 con 0x7fcca0074560 2024-03-21T10:36:30.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.010+0000 7fcc9aa00700 1 -- 172.21.15.31:0/4171233656 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7fcc94010430 con 0x7fcca0074560 2024-03-21T10:36:30.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.011+0000 7fcc9aa00700 1 --2- 172.21.15.31:0/4171233656 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcc880a30c0 0x7fcc880a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:30.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.011+0000 7fcc9aa00700 1 -- 172.21.15.31:0/4171233656 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fcc940d0350 con 0x7fcca0074560 2024-03-21T10:36:30.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.011+0000 7fcca5400700 1 --2- 172.21.15.31:0/4171233656 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcc880a30c0 0x7fcc880a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:30.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.011+0000 7fcc9aa00700 1 --2- 172.21.15.31:0/4171233656 >> [v2:172.21.15.70:6824/540305718,v1:172.21.15.70:6825/540305718] conn(0x7fcc880ae1a0 0x7fcc880b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:30.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.011+0000 7fcc9aa00700 1 -- 172.21.15.31:0/4171233656 --> [v2:172.21.15.70:6824/540305718,v1:172.21.15.70:6825/540305718] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcc880b0d10 con 0x7fcc880ae1a0 2024-03-21T10:36:30.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.011+0000 7fcc9aa00700 1 -- 172.21.15.31:0/4171233656 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc940d0700 con 0x7fcca0074560 2024-03-21T10:36:30.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.012+0000 7fcca6800700 1 --2- 172.21.15.31:0/4171233656 >> [v2:172.21.15.70:6824/540305718,v1:172.21.15.70:6825/540305718] conn(0x7fcc880ae1a0 0x7fcc880b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:30.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.012+0000 7fcca5400700 1 --2- 172.21.15.31:0/4171233656 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcc880a30c0 0x7fcc880a5580 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fcca01100d0 tx=0x7fcc90009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:30.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.012+0000 7fcca6800700 1 --2- 172.21.15.31:0/4171233656 >> [v2:172.21.15.70:6824/540305718,v1:172.21.15.70:6825/540305718] conn(0x7fcc880ae1a0 0x7fcc880b0640 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:30.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.013+0000 7fcc9aa00700 1 -- 172.21.15.31:0/4171233656 <== osd.11 v2:172.21.15.70:6824/540305718 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fcc880b0d10 con 0x7fcc880ae1a0 2024-03-21T10:36:30.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.038+0000 7fcca8800700 1 -- 172.21.15.31:0/4171233656 --> [v2:172.21.15.70:6824/540305718,v1:172.21.15.70:6825/540305718] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcca0075c10 con 0x7fcc880ae1a0 2024-03-21T10:36:30.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.039+0000 7fcc9aa00700 1 -- 172.21.15.31:0/4171233656 <== osd.11 v2:172.21.15.70:6824/540305718 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fcca0075c10 con 0x7fcc880ae1a0 2024-03-21T10:36:30.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.039+0000 7fcca8800700 1 -- 172.21.15.31:0/4171233656 >> [v2:172.21.15.70:6824/540305718,v1:172.21.15.70:6825/540305718] conn(0x7fcc880ae1a0 msgr2=0x7fcc880b0640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:30.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.039+0000 7fcca8800700 1 --2- 172.21.15.31:0/4171233656 >> [v2:172.21.15.70:6824/540305718,v1:172.21.15.70:6825/540305718] conn(0x7fcc880ae1a0 0x7fcc880b0640 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:30.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.040+0000 7fcca8800700 1 -- 172.21.15.31:0/4171233656 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcc880a30c0 msgr2=0x7fcc880a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:30.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.040+0000 7fcca8800700 1 --2- 172.21.15.31:0/4171233656 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcc880a30c0 0x7fcc880a5580 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fcca01100d0 tx=0x7fcc90009260 comp rx=0 tx=0).stop 2024-03-21T10:36:30.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.040+0000 7fcca8800700 1 -- 172.21.15.31:0/4171233656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcca0074560 msgr2=0x7fcca019ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:30.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.040+0000 7fcca8800700 1 --2- 172.21.15.31:0/4171233656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcca0074560 0x7fcca019ac10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcc9400e9a0 tx=0x7fcc9400ecb0 comp rx=0 tx=0).stop 2024-03-21T10:36:30.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.040+0000 7fcca6800700 1 -- 172.21.15.31:0/4171233656 reap_dead start 2024-03-21T10:36:30.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.040+0000 7fcca8800700 1 -- 172.21.15.31:0/4171233656 shutdown_connections 2024-03-21T10:36:30.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.040+0000 7fcca8800700 1 -- 172.21.15.31:0/4171233656 >> 172.21.15.31:0/4171233656 conn(0x7fcca00fdc60 msgr2=0x7fcca0110410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:30.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.040+0000 7fcca8800700 1 -- 172.21.15.31:0/4171233656 shutdown_connections 2024-03-21T10:36:30.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:30.040+0000 7fcca8800700 1 -- 172.21.15.31:0/4171233656 wait complete. 2024-03-21T10:36:30.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.298+0000 7f3927a00700 1 -- 172.21.15.31:0/362356644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3920074ce0 msgr2=0x7f392010e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:30.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.298+0000 7f3927a00700 1 --2- 172.21.15.31:0/362356644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3920074ce0 0x7f392010e650 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f390c0098f0 tx=0x7f390c009c00 comp rx=0 tx=0).stop 2024-03-21T10:36:30.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.299+0000 7f3927a00700 1 -- 172.21.15.31:0/362356644 shutdown_connections 2024-03-21T10:36:30.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.299+0000 7f3927a00700 1 --2- 172.21.15.31:0/362356644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f392010eb90 0x7f3920110f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:30.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.299+0000 7f3927a00700 1 --2- 172.21.15.31:0/362356644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39200743c0 0x7f39200747a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:30.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.299+0000 7f3927a00700 1 --2- 172.21.15.31:0/362356644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3920074ce0 0x7f392010e650 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:30.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.299+0000 7f3927a00700 1 -- 172.21.15.31:0/362356644 >> 172.21.15.31:0/362356644 conn(0x7f39200792f0 msgr2=0x7f3920079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:30.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.299+0000 7f3927a00700 1 -- 172.21.15.31:0/362356644 shutdown_connections 2024-03-21T10:36:30.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.299+0000 7f3927a00700 1 -- 172.21.15.31:0/362356644 wait complete. 2024-03-21T10:36:30.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.300+0000 7f3927a00700 1 Processor -- start 2024-03-21T10:36:30.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.300+0000 7f3927a00700 1 -- start start 2024-03-21T10:36:30.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.300+0000 7f3927a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39200743c0 0x7f39201a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:30.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.300+0000 7f3927a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3920074ce0 0x7f39201a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:30.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.300+0000 7f3927a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f392010eb90 0x7f392019aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:30.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.300+0000 7f3927a00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39201147a0 con 0x7f39200743c0 2024-03-21T10:36:30.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.300+0000 7f3927a00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3920114620 con 0x7f392010eb90 2024-03-21T10:36:30.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.300+0000 7f3927a00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3920114920 con 0x7f3920074ce0 2024-03-21T10:36:30.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.301+0000 7f391fe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3920074ce0 0x7f39201a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:30.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.301+0000 7f3925a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f392010eb90 0x7f392019aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:30.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.301+0000 7f391fe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3920074ce0 0x7f39201a0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:59554/0 (socket says 172.21.15.31:59554) 2024-03-21T10:36:30.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.301+0000 7f391fe00700 1 -- 172.21.15.31:0/2397215401 learned_addr learned my addr 172.21.15.31:0/2397215401 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:36:30.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.301+0000 7f3925000700 1 --2- 172.21.15.31:0/2397215401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39200743c0 0x7f39201a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:30.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.301+0000 7f391fe00700 1 -- 172.21.15.31:0/2397215401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f392010eb90 msgr2=0x7f392019aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:30.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.301+0000 7f391fe00700 1 --2- 172.21.15.31:0/2397215401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f392010eb90 0x7f392019aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:30.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.301+0000 7f391fe00700 1 -- 172.21.15.31:0/2397215401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39200743c0 msgr2=0x7f39201a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:30.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.301+0000 7f391fe00700 1 --2- 172.21.15.31:0/2397215401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39200743c0 0x7f39201a08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:30.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.301+0000 7f391fe00700 1 -- 172.21.15.31:0/2397215401 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392019b220 con 0x7f3920074ce0 2024-03-21T10:36:30.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.301+0000 7f391fe00700 1 --2- 172.21.15.31:0/2397215401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3920074ce0 0x7f39201a0df0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f390c005950 tx=0x7f390c00fb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:30.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.301+0000 7f391d600700 1 -- 172.21.15.31:0/2397215401 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390c0051f0 con 0x7f3920074ce0 2024-03-21T10:36:30.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.301+0000 7f391d600700 1 -- 172.21.15.31:0/2397215401 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f390c00b560 con 0x7f3920074ce0 2024-03-21T10:36:30.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.302+0000 7f3927a00700 1 -- 172.21.15.31:0/2397215401 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f392019b4b0 con 0x7f3920074ce0 2024-03-21T10:36:30.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.302+0000 7f391d600700 1 -- 172.21.15.31:0/2397215401 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390c003ba0 con 0x7f3920074ce0 2024-03-21T10:36:30.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.302+0000 7f3927a00700 1 -- 172.21.15.31:0/2397215401 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39201a7780 con 0x7f3920074ce0 2024-03-21T10:36:30.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.303+0000 7f3927a00700 1 -- 172.21.15.31:0/2397215401 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f39200628a0 con 0x7f3920074ce0 2024-03-21T10:36:30.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.305+0000 7f391d600700 1 -- 172.21.15.31:0/2397215401 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7f390c00ba80 con 0x7f3920074ce0 2024-03-21T10:36:30.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.306+0000 7f391d600700 1 --2- 172.21.15.31:0/2397215401 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f39080a30c0 0x7f39080a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:30.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.306+0000 7f391d600700 1 -- 172.21.15.31:0/2397215401 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f390c0d0a10 con 0x7f3920074ce0 2024-03-21T10:36:30.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.306+0000 7f3925000700 1 --2- 172.21.15.31:0/2397215401 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f39080a30c0 0x7f39080a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:30.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.307+0000 7f391d600700 1 --2- 172.21.15.31:0/2397215401 >> [v2:172.21.15.31:6802/3089885125,v1:172.21.15.31:6803/3089885125] conn(0x7f39080ae1a0 0x7f39080b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:30.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.307+0000 7f391d600700 1 -- 172.21.15.31:0/2397215401 --> [v2:172.21.15.31:6802/3089885125,v1:172.21.15.31:6803/3089885125] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f390c00bde0 con 0x7f39080ae1a0 2024-03-21T10:36:30.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.307+0000 7f391d600700 1 -- 172.21.15.31:0/2397215401 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f390c0d0dc0 con 0x7f3920074ce0 2024-03-21T10:36:30.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.307+0000 7f3925a00700 1 --2- 172.21.15.31:0/2397215401 >> [v2:172.21.15.31:6802/3089885125,v1:172.21.15.31:6803/3089885125] conn(0x7f39080ae1a0 0x7f39080b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:30.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.307+0000 7f3925000700 1 --2- 172.21.15.31:0/2397215401 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f39080a30c0 0x7f39080a5580 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f39201016b0 tx=0x7f3914009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:30.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.308+0000 7f3925a00700 1 --2- 172.21.15.31:0/2397215401 >> [v2:172.21.15.31:6802/3089885125,v1:172.21.15.31:6803/3089885125] conn(0x7f39080ae1a0 0x7f39080b0640 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:30.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.309+0000 7f391d600700 1 -- 172.21.15.31:0/2397215401 <== osd.0 v2:172.21.15.31:6802/3089885125 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f390c00bde0 con 0x7f39080ae1a0 2024-03-21T10:36:30.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.326+0000 7f3927a00700 1 -- 172.21.15.31:0/2397215401 --> [v2:172.21.15.31:6802/3089885125,v1:172.21.15.31:6803/3089885125] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3920062ca0 con 0x7f39080ae1a0 2024-03-21T10:36:30.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.327+0000 7f391d600700 1 -- 172.21.15.31:0/2397215401 <== osd.0 v2:172.21.15.31:6802/3089885125 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3920062ca0 con 0x7f39080ae1a0 2024-03-21T10:36:30.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.327+0000 7f3927a00700 1 -- 172.21.15.31:0/2397215401 >> [v2:172.21.15.31:6802/3089885125,v1:172.21.15.31:6803/3089885125] conn(0x7f39080ae1a0 msgr2=0x7f39080b0640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:30.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.327+0000 7f3927a00700 1 --2- 172.21.15.31:0/2397215401 >> [v2:172.21.15.31:6802/3089885125,v1:172.21.15.31:6803/3089885125] conn(0x7f39080ae1a0 0x7f39080b0640 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:30.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.328+0000 7f3927a00700 1 -- 172.21.15.31:0/2397215401 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f39080a30c0 msgr2=0x7f39080a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:30.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.328+0000 7f3927a00700 1 --2- 172.21.15.31:0/2397215401 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f39080a30c0 0x7f39080a5580 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f39201016b0 tx=0x7f3914009260 comp rx=0 tx=0).stop 2024-03-21T10:36:30.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.328+0000 7f3927a00700 1 -- 172.21.15.31:0/2397215401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3920074ce0 msgr2=0x7f39201a0df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:30.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.328+0000 7f3927a00700 1 --2- 172.21.15.31:0/2397215401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3920074ce0 0x7f39201a0df0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f390c005950 tx=0x7f390c00fb20 comp rx=0 tx=0).stop 2024-03-21T10:36:30.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.328+0000 7f3925a00700 1 -- 172.21.15.31:0/2397215401 reap_dead start 2024-03-21T10:36:30.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.328+0000 7f3927a00700 1 -- 172.21.15.31:0/2397215401 shutdown_connections 2024-03-21T10:36:30.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.328+0000 7f3927a00700 1 -- 172.21.15.31:0/2397215401 >> 172.21.15.31:0/2397215401 conn(0x7f39200792f0 msgr2=0x7f39200ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:30.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.328+0000 7f3927a00700 1 -- 172.21.15.31:0/2397215401 shutdown_connections 2024-03-21T10:36:30.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:26.328+0000 7f3927a00700 1 -- 172.21.15.31:0/2397215401 wait complete. 2024-03-21T10:36:31.316 INFO:teuthology.orchestra.run.smithi031.stdout:34359738462 2024-03-21T10:36:31.316 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-21T10:36:31.470 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:31 smithi031 ceph-mon[31673]: pgmap v369: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-21T10:36:31.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:31 smithi070 ceph-mon[36330]: pgmap v369: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-21T10:36:31.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:31 smithi040 ceph-mon[36154]: pgmap v369: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-21T10:36:31.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.762+0000 7fb372800700 1 -- 172.21.15.31:0/2067230358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb36c101140 msgr2=0x7fb36c10f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:31.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.762+0000 7fb372800700 1 --2- 172.21.15.31:0/2067230358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb36c101140 0x7fb36c10f0a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb36400b0a0 tx=0x7fb36400b3b0 comp rx=0 tx=0).stop 2024-03-21T10:36:31.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.762+0000 7fb372800700 1 -- 172.21.15.31:0/2067230358 shutdown_connections 2024-03-21T10:36:31.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.762+0000 7fb372800700 1 --2- 172.21.15.31:0/2067230358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb36c101140 0x7fb36c10f0a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:31.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.762+0000 7fb372800700 1 --2- 172.21.15.31:0/2067230358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb36c107200 0x7fb36c1075e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:31.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.762+0000 7fb372800700 1 --2- 172.21.15.31:0/2067230358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb36c100780 0x7fb36c100c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:31.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.762+0000 7fb372800700 1 -- 172.21.15.31:0/2067230358 >> 172.21.15.31:0/2067230358 conn(0x7fb36c077690 msgr2=0x7fb36c0feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:31.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.763+0000 7fb372800700 1 -- 172.21.15.31:0/2067230358 shutdown_connections 2024-03-21T10:36:31.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.763+0000 7fb372800700 1 -- 172.21.15.31:0/2067230358 wait complete. 2024-03-21T10:36:31.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.764+0000 7fb372800700 1 Processor -- start 2024-03-21T10:36:31.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.764+0000 7fb372800700 1 -- start start 2024-03-21T10:36:31.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.764+0000 7fb372800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb36c100780 0x7fb36c1a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:31.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.764+0000 7fb372800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb36c101140 0x7fb36c1a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:31.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.764+0000 7fb372800700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb36c107200 0x7fb36c19aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:31.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.764+0000 7fb372800700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c1125c0 con 0x7fb36c107200 2024-03-21T10:36:31.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.764+0000 7fb372800700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c112440 con 0x7fb36c100780 2024-03-21T10:36:31.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.764+0000 7fb372800700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c112740 con 0x7fb36c101140 2024-03-21T10:36:31.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.765+0000 7fb36aa00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb36c101140 0x7fb36c1a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:31.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.765+0000 7fb36b400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb36c100780 0x7fb36c1a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:31.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.765+0000 7fb36aa00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb36c101140 0x7fb36c1a0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53804/0 (socket says 172.21.15.31:53804) 2024-03-21T10:36:31.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.765+0000 7fb36aa00700 1 -- 172.21.15.31:0/326182726 learned_addr learned my addr 172.21.15.31:0/326182726 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:36:31.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.765+0000 7fb36be00700 1 --2- 172.21.15.31:0/326182726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb36c107200 0x7fb36c19aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:31.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.765+0000 7fb36b400700 1 -- 172.21.15.31:0/326182726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb36c101140 msgr2=0x7fb36c1a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:31.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.765+0000 7fb36b400700 1 --2- 172.21.15.31:0/326182726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb36c101140 0x7fb36c1a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:31.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.765+0000 7fb36b400700 1 -- 172.21.15.31:0/326182726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb36c107200 msgr2=0x7fb36c19aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:31.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.765+0000 7fb36b400700 1 --2- 172.21.15.31:0/326182726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb36c107200 0x7fb36c19aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:31.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.765+0000 7fb36b400700 1 -- 172.21.15.31:0/326182726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36c19b160 con 0x7fb36c100780 2024-03-21T10:36:31.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.765+0000 7fb36aa00700 1 --2- 172.21.15.31:0/326182726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb36c101140 0x7fb36c1a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:36:31.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.765+0000 7fb36b400700 1 --2- 172.21.15.31:0/326182726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb36c100780 0x7fb36c1a0840 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb35c00e9f0 tx=0x7fb35c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:31.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.766+0000 7fb363e00700 1 -- 172.21.15.31:0/326182726 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c00cbd0 con 0x7fb36c100780 2024-03-21T10:36:31.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.766+0000 7fb363e00700 1 -- 172.21.15.31:0/326182726 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb35c00cd70 con 0x7fb36c100780 2024-03-21T10:36:31.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.766+0000 7fb372800700 1 -- 172.21.15.31:0/326182726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb36c19b450 con 0x7fb36c100780 2024-03-21T10:36:31.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.766+0000 7fb363e00700 1 -- 172.21.15.31:0/326182726 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c018710 con 0x7fb36c100780 2024-03-21T10:36:31.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.766+0000 7fb372800700 1 -- 172.21.15.31:0/326182726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb36c1a7710 con 0x7fb36c100780 2024-03-21T10:36:31.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.767+0000 7fb372800700 1 -- 172.21.15.31:0/326182726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb36c10e740 con 0x7fb36c100780 2024-03-21T10:36:31.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.769+0000 7fb363e00700 1 -- 172.21.15.31:0/326182726 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7fb35c010430 con 0x7fb36c100780 2024-03-21T10:36:31.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.770+0000 7fb363e00700 1 --2- 172.21.15.31:0/326182726 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb3500a30c0 0x7fb3500a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:31.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.770+0000 7fb363e00700 1 -- 172.21.15.31:0/326182726 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fb35c0d0290 con 0x7fb36c100780 2024-03-21T10:36:31.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.770+0000 7fb36aa00700 1 --2- 172.21.15.31:0/326182726 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb3500a30c0 0x7fb3500a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:31.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.771+0000 7fb363e00700 1 --2- 172.21.15.31:0/326182726 >> [v2:172.21.15.40:6800/618840037,v1:172.21.15.40:6801/618840037] conn(0x7fb3500ae1a0 0x7fb3500b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:31.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.771+0000 7fb363e00700 1 -- 172.21.15.31:0/326182726 --> [v2:172.21.15.40:6800/618840037,v1:172.21.15.40:6801/618840037] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb3500b0d10 con 0x7fb3500ae1a0 2024-03-21T10:36:31.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.771+0000 7fb363e00700 1 -- 172.21.15.31:0/326182726 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb35c0d0640 con 0x7fb36c100780 2024-03-21T10:36:31.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.771+0000 7fb36be00700 1 --2- 172.21.15.31:0/326182726 >> [v2:172.21.15.40:6800/618840037,v1:172.21.15.40:6801/618840037] conn(0x7fb3500ae1a0 0x7fb3500b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:31.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.771+0000 7fb36aa00700 1 --2- 172.21.15.31:0/326182726 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb3500a30c0 0x7fb3500a5580 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb36c1038f0 tx=0x7fb358009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:31.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.772+0000 7fb36be00700 1 --2- 172.21.15.31:0/326182726 >> [v2:172.21.15.40:6800/618840037,v1:172.21.15.40:6801/618840037] conn(0x7fb3500ae1a0 0x7fb3500b0640 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:31.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.773+0000 7fb363e00700 1 -- 172.21.15.31:0/326182726 <== osd.1 v2:172.21.15.40:6800/618840037 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb3500b0d10 con 0x7fb3500ae1a0 2024-03-21T10:36:31.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.791+0000 7fb372800700 1 -- 172.21.15.31:0/326182726 --> [v2:172.21.15.40:6800/618840037,v1:172.21.15.40:6801/618840037] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb36c103aa0 con 0x7fb3500ae1a0 2024-03-21T10:36:31.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.792+0000 7fb363e00700 1 -- 172.21.15.31:0/326182726 <== osd.1 v2:172.21.15.40:6800/618840037 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb36c103aa0 con 0x7fb3500ae1a0 2024-03-21T10:36:31.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.792+0000 7fb372800700 1 -- 172.21.15.31:0/326182726 >> [v2:172.21.15.40:6800/618840037,v1:172.21.15.40:6801/618840037] conn(0x7fb3500ae1a0 msgr2=0x7fb3500b0640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:31.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.792+0000 7fb372800700 1 --2- 172.21.15.31:0/326182726 >> [v2:172.21.15.40:6800/618840037,v1:172.21.15.40:6801/618840037] conn(0x7fb3500ae1a0 0x7fb3500b0640 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:31.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.792+0000 7fb372800700 1 -- 172.21.15.31:0/326182726 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb3500a30c0 msgr2=0x7fb3500a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:31.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.792+0000 7fb372800700 1 --2- 172.21.15.31:0/326182726 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb3500a30c0 0x7fb3500a5580 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb36c1038f0 tx=0x7fb358009260 comp rx=0 tx=0).stop 2024-03-21T10:36:31.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.792+0000 7fb372800700 1 -- 172.21.15.31:0/326182726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb36c100780 msgr2=0x7fb36c1a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:31.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.792+0000 7fb372800700 1 --2- 172.21.15.31:0/326182726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb36c100780 0x7fb36c1a0840 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb35c00e9f0 tx=0x7fb35c00ed00 comp rx=0 tx=0).stop 2024-03-21T10:36:31.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.792+0000 7fb36be00700 1 -- 172.21.15.31:0/326182726 reap_dead start 2024-03-21T10:36:31.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.793+0000 7fb372800700 1 -- 172.21.15.31:0/326182726 shutdown_connections 2024-03-21T10:36:31.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.793+0000 7fb372800700 1 -- 172.21.15.31:0/326182726 >> 172.21.15.31:0/326182726 conn(0x7fb36c077690 msgr2=0x7fb36c1053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:31.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.793+0000 7fb372800700 1 -- 172.21.15.31:0/326182726 shutdown_connections 2024-03-21T10:36:31.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:31.793+0000 7fb372800700 1 -- 172.21.15.31:0/326182726 wait complete. 2024-03-21T10:36:32.535 INFO:teuthology.orchestra.run.smithi031.stdout:107374182474 2024-03-21T10:36:32.535 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-21T10:36:32.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.792+0000 7f519f200700 1 -- 172.21.15.31:0/3897000981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5198104f70 msgr2=0x7f5198108fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:32.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.792+0000 7f519f200700 1 --2- 172.21.15.31:0/3897000981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5198104f70 0x7f5198108fc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5184009990 tx=0x7f5184009ca0 comp rx=0 tx=0).stop 2024-03-21T10:36:32.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.793+0000 7f519f200700 1 -- 172.21.15.31:0/3897000981 shutdown_connections 2024-03-21T10:36:32.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.793+0000 7f519f200700 1 --2- 172.21.15.31:0/3897000981 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51981096a0 0x7f5198111270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:32.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.793+0000 7f519f200700 1 --2- 172.21.15.31:0/3897000981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51981045c0 0x7f51981049a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:32.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.793+0000 7f519f200700 1 --2- 172.21.15.31:0/3897000981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5198104f70 0x7f5198108fc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:32.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.793+0000 7f519f200700 1 -- 172.21.15.31:0/3897000981 >> 172.21.15.31:0/3897000981 conn(0x7f51980ffe70 msgr2=0x7f5198102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:32.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.793+0000 7f519f200700 1 -- 172.21.15.31:0/3897000981 shutdown_connections 2024-03-21T10:36:32.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.793+0000 7f519f200700 1 -- 172.21.15.31:0/3897000981 wait complete. 2024-03-21T10:36:32.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.794+0000 7f519f200700 1 Processor -- start 2024-03-21T10:36:32.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.794+0000 7f519f200700 1 -- start start 2024-03-21T10:36:32.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.795+0000 7f519f200700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51981045c0 0x7f519810f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:32.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.795+0000 7f519f200700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5198104f70 0x7f519810fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:32.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.795+0000 7f519f200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51981096a0 0x7f51981aaed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:32.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.795+0000 7f519f200700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5198077390 con 0x7f5198104f70 2024-03-21T10:36:32.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.795+0000 7f519f200700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5198077210 con 0x7f51981045c0 2024-03-21T10:36:32.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.795+0000 7f519f200700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5198077510 con 0x7f51981096a0 2024-03-21T10:36:32.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.795+0000 7f5197e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51981045c0 0x7f519810f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:32.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.795+0000 7f5197e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51981045c0 0x7f519810f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51864/0 (socket says 172.21.15.31:51864) 2024-03-21T10:36:32.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.795+0000 7f5197e00700 1 -- 172.21.15.31:0/2103339431 learned_addr learned my addr 172.21.15.31:0/2103339431 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:36:32.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.795+0000 7f5197400700 1 --2- 172.21.15.31:0/2103339431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5198104f70 0x7f519810fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:32.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.795+0000 7f5197e00700 1 -- 172.21.15.31:0/2103339431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51981096a0 msgr2=0x7f51981aaed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:32.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.795+0000 7f5197e00700 1 --2- 172.21.15.31:0/2103339431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f51981096a0 0x7f51981aaed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:32.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.795+0000 7f5197e00700 1 -- 172.21.15.31:0/2103339431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5198104f70 msgr2=0x7f519810fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:32.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.795+0000 7f5197e00700 1 --2- 172.21.15.31:0/2103339431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5198104f70 0x7f519810fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:32.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.796+0000 7f5197e00700 1 -- 172.21.15.31:0/2103339431 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51981ab410 con 0x7f51981045c0 2024-03-21T10:36:32.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.796+0000 7f5197e00700 1 --2- 172.21.15.31:0/2103339431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51981045c0 0x7f519810f6e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f518800e9a0 tx=0x7f518800ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:32.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.796+0000 7f5194c00700 1 -- 172.21.15.31:0/2103339431 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518800cbd0 con 0x7f51981045c0 2024-03-21T10:36:32.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.796+0000 7f5194c00700 1 -- 172.21.15.31:0/2103339431 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f518800cd70 con 0x7f51981045c0 2024-03-21T10:36:32.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.796+0000 7f519f200700 1 -- 172.21.15.31:0/2103339431 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51981ab5e0 con 0x7f51981045c0 2024-03-21T10:36:32.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.796+0000 7f5194c00700 1 -- 172.21.15.31:0/2103339431 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5188018780 con 0x7f51981045c0 2024-03-21T10:36:32.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.796+0000 7f519f200700 1 -- 172.21.15.31:0/2103339431 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51981abb20 con 0x7f51981045c0 2024-03-21T10:36:32.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.798+0000 7f519f200700 1 -- 172.21.15.31:0/2103339431 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5198079e30 con 0x7f51981045c0 2024-03-21T10:36:32.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.801+0000 7f5194c00700 1 -- 172.21.15.31:0/2103339431 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7f5188010430 con 0x7f51981045c0 2024-03-21T10:36:32.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.803+0000 7f5194c00700 1 --2- 172.21.15.31:0/2103339431 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f517c0a3100 0x7f517c0a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:32.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.804+0000 7f5194c00700 1 -- 172.21.15.31:0/2103339431 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f51880cfbe0 con 0x7f51981045c0 2024-03-21T10:36:32.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.804+0000 7f5197400700 1 --2- 172.21.15.31:0/2103339431 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f517c0a3100 0x7f517c0a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:32.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.804+0000 7f5194c00700 1 --2- 172.21.15.31:0/2103339431 >> [v2:172.21.15.40:6816/3825628119,v1:172.21.15.40:6817/3825628119] conn(0x7f517c0ae1e0 0x7f517c0b0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:32.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.804+0000 7f5194c00700 1 -- 172.21.15.31:0/2103339431 --> [v2:172.21.15.40:6816/3825628119,v1:172.21.15.40:6817/3825628119] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f518801ee50 con 0x7f517c0ae1e0 2024-03-21T10:36:32.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.804+0000 7f5194c00700 1 -- 172.21.15.31:0/2103339431 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51880d4220 con 0x7f51981045c0 2024-03-21T10:36:32.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.804+0000 7f519d200700 1 --2- 172.21.15.31:0/2103339431 >> [v2:172.21.15.40:6816/3825628119,v1:172.21.15.40:6817/3825628119] conn(0x7f517c0ae1e0 0x7f517c0b0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:32.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.805+0000 7f5197400700 1 --2- 172.21.15.31:0/2103339431 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f517c0a3100 0x7f517c0a55c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5198109000 tx=0x7f518401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:32.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.805+0000 7f519d200700 1 --2- 172.21.15.31:0/2103339431 >> [v2:172.21.15.40:6816/3825628119,v1:172.21.15.40:6817/3825628119] conn(0x7f517c0ae1e0 0x7f517c0b0680 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:32.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.806+0000 7f5194c00700 1 -- 172.21.15.31:0/2103339431 <== osd.7 v2:172.21.15.40:6816/3825628119 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f518801ee50 con 0x7f517c0ae1e0 2024-03-21T10:36:32.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.836+0000 7f519f200700 1 -- 172.21.15.31:0/2103339431 --> [v2:172.21.15.40:6816/3825628119,v1:172.21.15.40:6817/3825628119] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5198105aa0 con 0x7f517c0ae1e0 2024-03-21T10:36:32.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.836+0000 7f5194c00700 1 -- 172.21.15.31:0/2103339431 <== osd.7 v2:172.21.15.40:6816/3825628119 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f5198105aa0 con 0x7f517c0ae1e0 2024-03-21T10:36:32.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.837+0000 7f519f200700 1 -- 172.21.15.31:0/2103339431 >> [v2:172.21.15.40:6816/3825628119,v1:172.21.15.40:6817/3825628119] conn(0x7f517c0ae1e0 msgr2=0x7f517c0b0680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:32.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.837+0000 7f519f200700 1 --2- 172.21.15.31:0/2103339431 >> [v2:172.21.15.40:6816/3825628119,v1:172.21.15.40:6817/3825628119] conn(0x7f517c0ae1e0 0x7f517c0b0680 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:32.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.837+0000 7f519f200700 1 -- 172.21.15.31:0/2103339431 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f517c0a3100 msgr2=0x7f517c0a55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:32.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.837+0000 7f519f200700 1 --2- 172.21.15.31:0/2103339431 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f517c0a3100 0x7f517c0a55c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5198109000 tx=0x7f518401a040 comp rx=0 tx=0).stop 2024-03-21T10:36:32.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.837+0000 7f519f200700 1 -- 172.21.15.31:0/2103339431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51981045c0 msgr2=0x7f519810f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:32.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.837+0000 7f519f200700 1 --2- 172.21.15.31:0/2103339431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51981045c0 0x7f519810f6e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f518800e9a0 tx=0x7f518800ed60 comp rx=0 tx=0).stop 2024-03-21T10:36:32.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.837+0000 7f519d200700 1 -- 172.21.15.31:0/2103339431 reap_dead start 2024-03-21T10:36:32.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.837+0000 7f519f200700 1 -- 172.21.15.31:0/2103339431 shutdown_connections 2024-03-21T10:36:32.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.837+0000 7f519f200700 1 -- 172.21.15.31:0/2103339431 >> 172.21.15.31:0/2103339431 conn(0x7f51980ffe70 msgr2=0x7f5198102170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:32.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.838+0000 7f519f200700 1 -- 172.21.15.31:0/2103339431 shutdown_connections 2024-03-21T10:36:32.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:32.838+0000 7f519f200700 1 -- 172.21.15.31:0/2103339431 wait complete. 2024-03-21T10:36:33.507 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:33 smithi031 ceph-mon[31673]: pgmap v370: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-21T10:36:33.573 INFO:teuthology.orchestra.run.smithi031.stdout:55834574938 2024-03-21T10:36:33.573 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-21T10:36:33.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:33 smithi070 ceph-mon[36330]: pgmap v370: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-21T10:36:33.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:33 smithi040 ceph-mon[36154]: pgmap v370: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-21T10:36:34.688 INFO:teuthology.orchestra.run.smithi031.stdout:343597383712 2024-03-21T10:36:34.689 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-21T10:36:34.797 INFO:teuthology.orchestra.run.smithi031.stdout:141733920838 2024-03-21T10:36:34.797 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-21T10:36:35.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:35 smithi031 ceph-mon[31673]: pgmap v371: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:36:35.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:35 smithi070 ceph-mon[36330]: pgmap v371: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:36:35.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:35 smithi040 ceph-mon[36154]: pgmap v371: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:36:35.696 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:36:35.718 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:36:35.718 INFO:teuthology.orchestra.run.smithi031.stdout:219043332151 2024-03-21T10:36:35.719 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:36:35.719 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-21T10:36:35.730 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:36:35.870 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:36:36.873 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:36:36.873 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:36:36.879 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:36:37.478 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:37 smithi031 ceph-mon[31673]: pgmap v372: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-21T10:36:37.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:37 smithi070 ceph-mon[36330]: pgmap v372: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-21T10:36:37.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:37 smithi040 ceph-mon[36154]: pgmap v372: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-21T10:36:38.197 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:36:38.221 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:36:39.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:39 smithi031 ceph-mon[31673]: pgmap v373: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:36:39.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:39 smithi070 ceph-mon[36330]: pgmap v373: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:36:39.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:39 smithi040 ceph-mon[36154]: pgmap v373: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:36:41.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:41 smithi031 ceph-mon[31673]: pgmap v374: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:36:41.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:41 smithi070 ceph-mon[36330]: pgmap v374: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:36:41.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:41 smithi040 ceph-mon[36154]: pgmap v374: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:36:43.513 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:43 smithi031 ceph-mon[31673]: pgmap v375: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:36:43.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:43 smithi070 ceph-mon[36330]: pgmap v375: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:36:43.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:43 smithi040 ceph-mon[36154]: pgmap v375: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:36:45.397 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:45 smithi031 ceph-mon[31673]: pgmap v376: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:36:45.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:45 smithi070 ceph-mon[36330]: pgmap v376: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:36:45.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:45 smithi040 ceph-mon[36154]: pgmap v376: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:36:45.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.397+0000 7f275ee00700 1 -- 172.21.15.31:0/1781937065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2758108210 msgr2=0x7f27581085f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:45.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.397+0000 7f275ee00700 1 --2- 172.21.15.31:0/1781937065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2758108210 0x7f27581085f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f27480099c0 tx=0x7f2748009cd0 comp rx=0 tx=0).stop 2024-03-21T10:36:45.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.397+0000 7f275ee00700 1 -- 172.21.15.31:0/1781937065 shutdown_connections 2024-03-21T10:36:45.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.397+0000 7f275ee00700 1 --2- 172.21.15.31:0/1781937065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f275806b210 0x7f275810f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:45.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.397+0000 7f275ee00700 1 --2- 172.21.15.31:0/1781937065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2758108210 0x7f27581085f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:45.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.397+0000 7f275ee00700 1 --2- 172.21.15.31:0/1781937065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f275806a870 0x7f275806acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:45.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.397+0000 7f275ee00700 1 -- 172.21.15.31:0/1781937065 >> 172.21.15.31:0/1781937065 conn(0x7f27580770e0 msgr2=0x7f27580774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:45.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.397+0000 7f275ee00700 1 -- 172.21.15.31:0/1781937065 shutdown_connections 2024-03-21T10:36:45.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.397+0000 7f275ee00700 1 -- 172.21.15.31:0/1781937065 wait complete. 2024-03-21T10:36:45.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.398+0000 7f275ee00700 1 Processor -- start 2024-03-21T10:36:45.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.398+0000 7f275ee00700 1 -- start start 2024-03-21T10:36:45.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.399+0000 7f275ee00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f275806a870 0x7f275819abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:45.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.399+0000 7f275ee00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f275806b210 0x7f275819b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:45.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.399+0000 7f275ee00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2758108210 0x7f275819f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:45.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.399+0000 7f275ee00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27581124a0 con 0x7f2758108210 2024-03-21T10:36:45.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.399+0000 7f275ee00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2758112320 con 0x7f275806b210 2024-03-21T10:36:45.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.399+0000 7f275ee00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2758112620 con 0x7f275806a870 2024-03-21T10:36:45.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.399+0000 7f2757e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f275806a870 0x7f275819abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:45.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.399+0000 7f2757e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f275806a870 0x7f275819abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:47214/0 (socket says 172.21.15.31:47214) 2024-03-21T10:36:45.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.399+0000 7f2757e00700 1 -- 172.21.15.31:0/1781620685 learned_addr learned my addr 172.21.15.31:0/1781620685 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:36:45.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.399+0000 7f275ce00700 1 --2- 172.21.15.31:0/1781620685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2758108210 0x7f275819f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:45.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.399+0000 7f2757400700 1 --2- 172.21.15.31:0/1781620685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f275806b210 0x7f275819b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:45.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.400+0000 7f2757e00700 1 -- 172.21.15.31:0/1781620685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f275806b210 msgr2=0x7f275819b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:45.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.400+0000 7f2757e00700 1 --2- 172.21.15.31:0/1781620685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f275806b210 0x7f275819b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:45.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.400+0000 7f2757e00700 1 -- 172.21.15.31:0/1781620685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2758108210 msgr2=0x7f275819f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:45.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.400+0000 7f2757e00700 1 --2- 172.21.15.31:0/1781620685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2758108210 0x7f275819f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:45.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.400+0000 7f2757e00700 1 -- 172.21.15.31:0/1781620685 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f275819fbc0 con 0x7f275806a870 2024-03-21T10:36:45.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.400+0000 7f2757e00700 1 --2- 172.21.15.31:0/1781620685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f275806a870 0x7f275819abc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2748006010 tx=0x7f2748003a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:45.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.400+0000 7f2754c00700 1 -- 172.21.15.31:0/1781620685 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2748003e50 con 0x7f275806a870 2024-03-21T10:36:45.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.400+0000 7f2754c00700 1 -- 172.21.15.31:0/1781620685 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f27480055a0 con 0x7f275806a870 2024-03-21T10:36:45.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.400+0000 7f2754c00700 1 -- 172.21.15.31:0/1781620685 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274800fe30 con 0x7f275806a870 2024-03-21T10:36:45.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.400+0000 7f275ee00700 1 -- 172.21.15.31:0/1781620685 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f275819fe50 con 0x7f275806a870 2024-03-21T10:36:45.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.401+0000 7f275ee00700 1 -- 172.21.15.31:0/1781620685 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27581a02e0 con 0x7f275806a870 2024-03-21T10:36:45.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.402+0000 7f275ee00700 1 -- 172.21.15.31:0/1781620685 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2758068e50 con 0x7f275806a870 2024-03-21T10:36:45.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.403+0000 7f2754c00700 1 -- 172.21.15.31:0/1781620685 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7f274800f4f0 con 0x7f275806a870 2024-03-21T10:36:45.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.406+0000 7f2754c00700 1 --2- 172.21.15.31:0/1781620685 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f273c0a3260 0x7f273c0a5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:45.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.406+0000 7f2754c00700 1 -- 172.21.15.31:0/1781620685 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f27480d07b0 con 0x7f275806a870 2024-03-21T10:36:45.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.406+0000 7f2757400700 1 --2- 172.21.15.31:0/1781620685 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f273c0a3260 0x7f273c0a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:45.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.407+0000 7f2757400700 1 --2- 172.21.15.31:0/1781620685 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f273c0a3260 0x7f273c0a5720 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2744009680 tx=0x7f2744009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:45.792 INFO:teuthology.orchestra.run.smithi031.stdout:287762808876 2024-03-21T10:36:45.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.408+0000 7f2754c00700 1 -- 172.21.15.31:0/1781620685 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2748013070 con 0x7f275806a870 2024-03-21T10:36:45.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.687+0000 7f275ee00700 1 -- 172.21.15.31:0/1781620685 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f27581a7770 con 0x7f275806a870 2024-03-21T10:36:45.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.687+0000 7f2754c00700 1 -- 172.21.15.31:0/1781620685 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f274809d170 con 0x7f275806a870 2024-03-21T10:36:45.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.690+0000 7f275ee00700 1 -- 172.21.15.31:0/1781620685 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f273c0a3260 msgr2=0x7f273c0a5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:45.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.690+0000 7f275ee00700 1 --2- 172.21.15.31:0/1781620685 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f273c0a3260 0x7f273c0a5720 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2744009680 tx=0x7f2744009260 comp rx=0 tx=0).stop 2024-03-21T10:36:45.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.690+0000 7f275ee00700 1 -- 172.21.15.31:0/1781620685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f275806a870 msgr2=0x7f275819abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:45.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.690+0000 7f275ee00700 1 --2- 172.21.15.31:0/1781620685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f275806a870 0x7f275819abc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2748006010 tx=0x7f2748003a90 comp rx=0 tx=0).stop 2024-03-21T10:36:45.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.691+0000 7f275ee00700 1 -- 172.21.15.31:0/1781620685 shutdown_connections 2024-03-21T10:36:45.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.691+0000 7f275ee00700 1 --2- 172.21.15.31:0/1781620685 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f273c0a3260 0x7f273c0a5720 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:45.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.691+0000 7f275ee00700 1 --2- 172.21.15.31:0/1781620685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2758108210 0x7f275819f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:45.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.691+0000 7f275ee00700 1 --2- 172.21.15.31:0/1781620685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f275806a870 0x7f275819abc0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:45.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.691+0000 7f275ee00700 1 --2- 172.21.15.31:0/1781620685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f275806b210 0x7f275819b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:45.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.691+0000 7f275ee00700 1 -- 172.21.15.31:0/1781620685 >> 172.21.15.31:0/1781620685 conn(0x7f27580770e0 msgr2=0x7f2758103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:45.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.691+0000 7f275ee00700 1 -- 172.21.15.31:0/1781620685 shutdown_connections 2024-03-21T10:36:45.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:45.691+0000 7f275ee00700 1 -- 172.21.15.31:0/1781620685 wait complete. 2024-03-21T10:36:46.461 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:46 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1781620685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-21T10:36:46.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:46 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1781620685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-21T10:36:46.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:46 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1781620685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-21T10:36:47.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:47 smithi031 ceph-mon[31673]: pgmap v377: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:36:47.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:47 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1074504880' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-21T10:36:47.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:47 smithi070 ceph-mon[36330]: pgmap v377: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:36:47.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:47 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1074504880' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-21T10:36:47.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:47 smithi040 ceph-mon[36154]: pgmap v377: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:36:47.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:47 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1074504880' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-21T10:36:47.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.657+0000 7f3fae600700 1 -- 172.21.15.31:0/2075597650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fa8104600 msgr2=0x7f3fa81049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:47.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.657+0000 7f3fae600700 1 --2- 172.21.15.31:0/2075597650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fa8104600 0x7f3fa81049e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3f98009990 tx=0x7f3f98009ca0 comp rx=0 tx=0).stop 2024-03-21T10:36:47.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.657+0000 7f3fae600700 1 -- 172.21.15.31:0/2075597650 shutdown_connections 2024-03-21T10:36:47.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.657+0000 7f3fae600700 1 --2- 172.21.15.31:0/2075597650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fa81096e0 0x7f3fa81112b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:47.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.657+0000 7f3fae600700 1 --2- 172.21.15.31:0/2075597650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fa8104600 0x7f3fa81049e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:47.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.657+0000 7f3fae600700 1 --2- 172.21.15.31:0/2075597650 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fa8104fb0 0x7f3fa8109000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:47.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.657+0000 7f3fae600700 1 -- 172.21.15.31:0/2075597650 >> 172.21.15.31:0/2075597650 conn(0x7f3fa80ffe90 msgr2=0x7f3fa81022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:47.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.657+0000 7f3fae600700 1 -- 172.21.15.31:0/2075597650 shutdown_connections 2024-03-21T10:36:47.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.657+0000 7f3fae600700 1 -- 172.21.15.31:0/2075597650 wait complete. 2024-03-21T10:36:47.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.658+0000 7f3fae600700 1 Processor -- start 2024-03-21T10:36:47.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.658+0000 7f3fae600700 1 -- start start 2024-03-21T10:36:47.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.659+0000 7f3fae600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fa8104600 0x7f3fa81a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:47.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.659+0000 7f3fae600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fa8104fb0 0x7f3fa81a0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:47.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.659+0000 7f3fae600700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fa81096e0 0x7f3fa819a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:47.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.659+0000 7f3fae600700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa8114710 con 0x7f3fa81096e0 2024-03-21T10:36:47.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.659+0000 7f3fae600700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa8114590 con 0x7f3fa8104fb0 2024-03-21T10:36:47.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.659+0000 7f3fae600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa8114890 con 0x7f3fa8104600 2024-03-21T10:36:47.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.659+0000 7f3fa6a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fa8104fb0 0x7f3fa81a0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:47.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.659+0000 7f3fa6a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fa8104fb0 0x7f3fa81a0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42534/0 (socket says 172.21.15.31:42534) 2024-03-21T10:36:47.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.659+0000 7f3fa6a00700 1 -- 172.21.15.31:0/3262995297 learned_addr learned my addr 172.21.15.31:0/3262995297 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:36:47.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.659+0000 7f3fa7400700 1 --2- 172.21.15.31:0/3262995297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fa8104600 0x7f3fa81a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:47.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.660+0000 7f3fa7e00700 1 --2- 172.21.15.31:0/3262995297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fa81096e0 0x7f3fa819a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:47.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.660+0000 7f3fa6a00700 1 -- 172.21.15.31:0/3262995297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fa8104600 msgr2=0x7f3fa81a07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:47.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.660+0000 7f3fa6a00700 1 --2- 172.21.15.31:0/3262995297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fa8104600 0x7f3fa81a07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:47.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.660+0000 7f3fa6a00700 1 -- 172.21.15.31:0/3262995297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fa81096e0 msgr2=0x7f3fa819a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:47.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.660+0000 7f3fa6a00700 1 --2- 172.21.15.31:0/3262995297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fa81096e0 0x7f3fa819a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:47.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.660+0000 7f3fa6a00700 1 -- 172.21.15.31:0/3262995297 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa819b110 con 0x7f3fa8104fb0 2024-03-21T10:36:47.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.660+0000 7f3fa6a00700 1 --2- 172.21.15.31:0/3262995297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fa8104fb0 0x7f3fa81a0d30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3f9400ea30 tx=0x7f3f9400ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:47.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.660+0000 7f3f9fe00700 1 -- 172.21.15.31:0/3262995297 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f9400cbd0 con 0x7f3fa8104fb0 2024-03-21T10:36:47.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.660+0000 7f3f9fe00700 1 -- 172.21.15.31:0/3262995297 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3f94004500 con 0x7f3fa8104fb0 2024-03-21T10:36:47.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.660+0000 7f3fae600700 1 -- 172.21.15.31:0/3262995297 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fa819b400 con 0x7f3fa8104fb0 2024-03-21T10:36:47.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.661+0000 7f3f9fe00700 1 -- 172.21.15.31:0/3262995297 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f94010430 con 0x7f3fa8104fb0 2024-03-21T10:36:47.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.661+0000 7f3fae600700 1 -- 172.21.15.31:0/3262995297 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fa81a76c0 con 0x7f3fa8104fb0 2024-03-21T10:36:47.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.662+0000 7f3fae600700 1 -- 172.21.15.31:0/3262995297 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fa8068e50 con 0x7f3fa8104fb0 2024-03-21T10:36:47.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.665+0000 7f3f9fe00700 1 -- 172.21.15.31:0/3262995297 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7f3f94003f50 con 0x7f3fa8104fb0 2024-03-21T10:36:47.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.667+0000 7f3f9fe00700 1 --2- 172.21.15.31:0/3262995297 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f3f8c0a30c0 0x7f3f8c0a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:47.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.667+0000 7f3f9fe00700 1 -- 172.21.15.31:0/3262995297 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f3f940cfae0 con 0x7f3fa8104fb0 2024-03-21T10:36:47.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.667+0000 7f3fa7400700 1 --2- 172.21.15.31:0/3262995297 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f3f8c0a30c0 0x7f3f8c0a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:47.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.668+0000 7f3fa7400700 1 --2- 172.21.15.31:0/3262995297 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f3f8c0a30c0 0x7f3f8c0a5580 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3fa819baf0 tx=0x7f3f9800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:47.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.671+0000 7f3f9fe00700 1 -- 172.21.15.31:0/3262995297 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3f9409c410 con 0x7f3fa8104fb0 2024-03-21T10:36:47.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.344+0000 7f1e0e200700 1 -- 172.21.15.31:0/3330359730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e08102440 msgr2=0x7f1e08102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:47.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.344+0000 7f1e0e200700 1 --2- 172.21.15.31:0/3330359730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e08102440 0x7f1e08102820 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1df8009920 tx=0x7f1df8009c30 comp rx=0 tx=0).stop 2024-03-21T10:36:47.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.344+0000 7f1e0e200700 1 -- 172.21.15.31:0/3330359730 shutdown_connections 2024-03-21T10:36:47.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.344+0000 7f1e0e200700 1 --2- 172.21.15.31:0/3330359730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e08107510 0x7f1e0810f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:47.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.344+0000 7f1e0e200700 1 --2- 172.21.15.31:0/3330359730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e08102440 0x7f1e08102820 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:47.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.344+0000 7f1e0e200700 1 --2- 172.21.15.31:0/3330359730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e08102df0 0x7f1e08106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:47.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.344+0000 7f1e0e200700 1 -- 172.21.15.31:0/3330359730 >> 172.21.15.31:0/3330359730 conn(0x7f1e080792f0 msgr2=0x7f1e08079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:47.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.344+0000 7f1e0e200700 1 -- 172.21.15.31:0/3330359730 shutdown_connections 2024-03-21T10:36:47.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.344+0000 7f1e0e200700 1 -- 172.21.15.31:0/3330359730 wait complete. 2024-03-21T10:36:47.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.345+0000 7f1e0e200700 1 Processor -- start 2024-03-21T10:36:47.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.345+0000 7f1e0e200700 1 -- start start 2024-03-21T10:36:47.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.345+0000 7f1e0e200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e08102440 0x7f1e081a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:47.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.345+0000 7f1e0e200700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e08102df0 0x7f1e081a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:47.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.345+0000 7f1e0e200700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e08107510 0x7f1e0819aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:47.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.346+0000 7f1e0e200700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e08112540 con 0x7f1e08107510 2024-03-21T10:36:47.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.346+0000 7f1e0e200700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e081123c0 con 0x7f1e08102df0 2024-03-21T10:36:47.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.346+0000 7f1e0e200700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e081126c0 con 0x7f1e08102440 2024-03-21T10:36:47.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.346+0000 7f1e06a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e08102df0 0x7f1e081a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:47.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.346+0000 7f1e07400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e08102440 0x7f1e081a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:47.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.346+0000 7f1e06a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e08102df0 0x7f1e081a0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49718/0 (socket says 172.21.15.31:49718) 2024-03-21T10:36:47.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.346+0000 7f1e07400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e08102440 0x7f1e081a08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:47232/0 (socket says 172.21.15.31:47232) 2024-03-21T10:36:47.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.346+0000 7f1e06a00700 1 -- 172.21.15.31:0/1074504880 learned_addr learned my addr 172.21.15.31:0/1074504880 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:36:47.840 INFO:teuthology.orchestra.run.smithi031.stdout:77309411414 2024-03-21T10:36:47.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.346+0000 7f1e07e00700 1 --2- 172.21.15.31:0/1074504880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e08107510 0x7f1e0819aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:47.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.346+0000 7f1e06a00700 1 -- 172.21.15.31:0/1074504880 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e08102440 msgr2=0x7f1e081a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:47.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.346+0000 7f1e06a00700 1 --2- 172.21.15.31:0/1074504880 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e08102440 0x7f1e081a08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:47.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.346+0000 7f1e06a00700 1 -- 172.21.15.31:0/1074504880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e08107510 msgr2=0x7f1e0819aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:47.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.346+0000 7f1e06a00700 1 --2- 172.21.15.31:0/1074504880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e08107510 0x7f1e0819aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:47.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.346+0000 7f1e06a00700 1 -- 172.21.15.31:0/1074504880 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e0819b1f0 con 0x7f1e08102df0 2024-03-21T10:36:47.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.347+0000 7f1e06a00700 1 --2- 172.21.15.31:0/1074504880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e08102df0 0x7f1e081a0e10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1df400ec70 tx=0x7f1df400ef80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:47.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.347+0000 7f1dffe00700 1 -- 172.21.15.31:0/1074504880 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df400cc90 con 0x7f1e08102df0 2024-03-21T10:36:47.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.347+0000 7f1dffe00700 1 -- 172.21.15.31:0/1074504880 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1df400ce30 con 0x7f1e08102df0 2024-03-21T10:36:47.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.347+0000 7f1e0e200700 1 -- 172.21.15.31:0/1074504880 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e0819b4e0 con 0x7f1e08102df0 2024-03-21T10:36:47.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.347+0000 7f1dffe00700 1 -- 172.21.15.31:0/1074504880 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df4010470 con 0x7f1e08102df0 2024-03-21T10:36:47.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.347+0000 7f1e0e200700 1 -- 172.21.15.31:0/1074504880 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e081a77a0 con 0x7f1e08102df0 2024-03-21T10:36:47.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.348+0000 7f1e0e200700 1 -- 172.21.15.31:0/1074504880 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e08068e50 con 0x7f1e08102df0 2024-03-21T10:36:47.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.350+0000 7f1dffe00700 1 -- 172.21.15.31:0/1074504880 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7f1df4004000 con 0x7f1e08102df0 2024-03-21T10:36:47.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.351+0000 7f1dffe00700 1 --2- 172.21.15.31:0/1074504880 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1dec0a3040 0x7f1dec0a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:47.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.351+0000 7f1dffe00700 1 -- 172.21.15.31:0/1074504880 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f1df40cfc40 con 0x7f1e08102df0 2024-03-21T10:36:47.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.351+0000 7f1e07400700 1 --2- 172.21.15.31:0/1074504880 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1dec0a3040 0x7f1dec0a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:47.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.352+0000 7f1e07400700 1 --2- 172.21.15.31:0/1074504880 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1dec0a3040 0x7f1dec0a5500 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1df8009680 tx=0x7f1df8019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:47.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.355+0000 7f1dffe00700 1 -- 172.21.15.31:0/1074504880 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1df409c570 con 0x7f1e08102df0 2024-03-21T10:36:47.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.598+0000 7f1e0e200700 1 -- 172.21.15.31:0/1074504880 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f1e081a7a30 con 0x7f1e08102df0 2024-03-21T10:36:47.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.599+0000 7f1dffe00700 1 -- 172.21.15.31:0/1074504880 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1df409be70 con 0x7f1e08102df0 2024-03-21T10:36:47.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.603+0000 7f1e0e200700 1 -- 172.21.15.31:0/1074504880 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1dec0a3040 msgr2=0x7f1dec0a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:47.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.603+0000 7f1e0e200700 1 --2- 172.21.15.31:0/1074504880 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1dec0a3040 0x7f1dec0a5500 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1df8009680 tx=0x7f1df8019040 comp rx=0 tx=0).stop 2024-03-21T10:36:47.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.603+0000 7f1e0e200700 1 -- 172.21.15.31:0/1074504880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e08102df0 msgr2=0x7f1e081a0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:47.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.603+0000 7f1e0e200700 1 --2- 172.21.15.31:0/1074504880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e08102df0 0x7f1e081a0e10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1df400ec70 tx=0x7f1df400ef80 comp rx=0 tx=0).stop 2024-03-21T10:36:47.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.603+0000 7f1e0e200700 1 -- 172.21.15.31:0/1074504880 shutdown_connections 2024-03-21T10:36:47.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.603+0000 7f1e0e200700 1 --2- 172.21.15.31:0/1074504880 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1dec0a3040 0x7f1dec0a5500 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:47.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.603+0000 7f1e0e200700 1 --2- 172.21.15.31:0/1074504880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e08107510 0x7f1e0819aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:47.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.603+0000 7f1e0e200700 1 --2- 172.21.15.31:0/1074504880 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e08102440 0x7f1e081a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:47.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.603+0000 7f1e0e200700 1 --2- 172.21.15.31:0/1074504880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e08102df0 0x7f1e081a0e10 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:47.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.603+0000 7f1e0e200700 1 -- 172.21.15.31:0/1074504880 >> 172.21.15.31:0/1074504880 conn(0x7f1e080792f0 msgr2=0x7f1e081001b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:47.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.603+0000 7f1e0e200700 1 -- 172.21.15.31:0/1074504880 shutdown_connections 2024-03-21T10:36:47.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:46.603+0000 7f1e0e200700 1 -- 172.21.15.31:0/1074504880 wait complete. 2024-03-21T10:36:47.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.922+0000 7f3fae600700 1 -- 172.21.15.31:0/3262995297 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f3fa819bee0 con 0x7f3fa8104fb0 2024-03-21T10:36:47.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.922+0000 7f3f9fe00700 1 -- 172.21.15.31:0/3262995297 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f3f9409bd10 con 0x7f3fa8104fb0 2024-03-21T10:36:47.920 INFO:teuthology.orchestra.run.smithi031.stdout:197568495680 2024-03-21T10:36:47.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.926+0000 7f3fae600700 1 -- 172.21.15.31:0/3262995297 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f3f8c0a30c0 msgr2=0x7f3f8c0a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:47.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.926+0000 7f3fae600700 1 --2- 172.21.15.31:0/3262995297 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f3f8c0a30c0 0x7f3f8c0a5580 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3fa819baf0 tx=0x7f3f9800b260 comp rx=0 tx=0).stop 2024-03-21T10:36:47.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.926+0000 7f3fae600700 1 -- 172.21.15.31:0/3262995297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fa8104fb0 msgr2=0x7f3fa81a0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:47.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.926+0000 7f3fae600700 1 --2- 172.21.15.31:0/3262995297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fa8104fb0 0x7f3fa81a0d30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3f9400ea30 tx=0x7f3f9400ed40 comp rx=0 tx=0).stop 2024-03-21T10:36:47.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.926+0000 7f3fae600700 1 -- 172.21.15.31:0/3262995297 shutdown_connections 2024-03-21T10:36:47.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.926+0000 7f3fae600700 1 --2- 172.21.15.31:0/3262995297 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f3f8c0a30c0 0x7f3f8c0a5580 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:47.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.926+0000 7f3fae600700 1 --2- 172.21.15.31:0/3262995297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fa81096e0 0x7f3fa819a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:47.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.926+0000 7f3fae600700 1 --2- 172.21.15.31:0/3262995297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fa8104600 0x7f3fa81a07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:47.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.926+0000 7f3fae600700 1 --2- 172.21.15.31:0/3262995297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fa8104fb0 0x7f3fa81a0d30 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:47.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.926+0000 7f3fae600700 1 -- 172.21.15.31:0/3262995297 >> 172.21.15.31:0/3262995297 conn(0x7f3fa80ffe90 msgr2=0x7f3fa8102160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:47.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.926+0000 7f3fae600700 1 -- 172.21.15.31:0/3262995297 shutdown_connections 2024-03-21T10:36:47.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:47.926+0000 7f3fae600700 1 -- 172.21.15.31:0/3262995297 wait complete. 2024-03-21T10:36:48.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:48 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3262995297' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-21T10:36:48.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:48 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3262995297' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-21T10:36:48.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:48 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3262995297' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-21T10:36:48.894 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808871 got 287762808876 for osd.9 2024-03-21T10:36:48.894 DEBUG:teuthology.parallel:result is None 2024-03-21T10:36:49.415 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:49 smithi031 ceph-mon[31673]: pgmap v378: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:36:49.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:49 smithi070 ceph-mon[36330]: pgmap v378: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:36:49.582 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:49 smithi040 ceph-mon[36154]: pgmap v378: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:36:51.438 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:51 smithi031 ceph-mon[31673]: pgmap v379: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:36:51.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.348+0000 7f3d79e00700 1 -- 172.21.15.31:0/3844958500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d74102410 msgr2=0x7f3d741027f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:51.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.348+0000 7f3d79e00700 1 --2- 172.21.15.31:0/3844958500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d74102410 0x7f3d741027f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3d64009990 tx=0x7f3d64009ca0 comp rx=0 tx=0).stop 2024-03-21T10:36:51.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.348+0000 7f3d79e00700 1 -- 172.21.15.31:0/3844958500 shutdown_connections 2024-03-21T10:36:51.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.348+0000 7f3d79e00700 1 --2- 172.21.15.31:0/3844958500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d741074f0 0x7f3d7410f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:51.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.348+0000 7f3d79e00700 1 --2- 172.21.15.31:0/3844958500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d74102410 0x7f3d741027f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:51.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.348+0000 7f3d79e00700 1 --2- 172.21.15.31:0/3844958500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d74102dc0 0x7f3d74106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:51.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.348+0000 7f3d79e00700 1 -- 172.21.15.31:0/3844958500 >> 172.21.15.31:0/3844958500 conn(0x7f3d740fdc60 msgr2=0x7f3d74100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:51.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.349+0000 7f3d79e00700 1 -- 172.21.15.31:0/3844958500 shutdown_connections 2024-03-21T10:36:51.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.349+0000 7f3d79e00700 1 -- 172.21.15.31:0/3844958500 wait complete. 2024-03-21T10:36:51.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.350+0000 7f3d79e00700 1 Processor -- start 2024-03-21T10:36:51.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.350+0000 7f3d79e00700 1 -- start start 2024-03-21T10:36:51.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.350+0000 7f3d79e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d74102410 0x7f3d741a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:51.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.350+0000 7f3d79e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d74102dc0 0x7f3d741a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:51.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.351+0000 7f3d79e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d741074f0 0x7f3d7419a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:51.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.351+0000 7f3d79e00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d74112520 con 0x7f3d74102410 2024-03-21T10:36:51.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.351+0000 7f3d79e00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d741123a0 con 0x7f3d74102dc0 2024-03-21T10:36:51.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.351+0000 7f3d79e00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d741126a0 con 0x7f3d741074f0 2024-03-21T10:36:51.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.351+0000 7f3d72000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d74102dc0 0x7f3d741a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:51.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.351+0000 7f3d72a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d74102410 0x7f3d741a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:51.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.351+0000 7f3d72000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d74102dc0 0x7f3d741a0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42552/0 (socket says 172.21.15.31:42552) 2024-03-21T10:36:51.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.351+0000 7f3d72000700 1 -- 172.21.15.31:0/1996718304 learned_addr learned my addr 172.21.15.31:0/1996718304 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:36:51.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.351+0000 7f3d73400700 1 --2- 172.21.15.31:0/1996718304 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d741074f0 0x7f3d7419a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:51.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.351+0000 7f3d72000700 1 -- 172.21.15.31:0/1996718304 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d741074f0 msgr2=0x7f3d7419a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:51.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.351+0000 7f3d72000700 1 --2- 172.21.15.31:0/1996718304 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d741074f0 0x7f3d7419a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:51.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.351+0000 7f3d72000700 1 -- 172.21.15.31:0/1996718304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d74102410 msgr2=0x7f3d741a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:51.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.351+0000 7f3d72000700 1 --2- 172.21.15.31:0/1996718304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d74102410 0x7f3d741a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:51.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.351+0000 7f3d72000700 1 -- 172.21.15.31:0/1996718304 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7419b120 con 0x7f3d74102dc0 2024-03-21T10:36:51.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.352+0000 7f3d72000700 1 --2- 172.21.15.31:0/1996718304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d74102dc0 0x7f3d741a0d40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3d6000d980 tx=0x7f3d6000dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:51.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.352+0000 7f3d78a00700 1 -- 172.21.15.31:0/1996718304 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d60017070 con 0x7f3d74102dc0 2024-03-21T10:36:51.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.352+0000 7f3d78a00700 1 -- 172.21.15.31:0/1996718304 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d6000fd70 con 0x7f3d74102dc0 2024-03-21T10:36:51.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.352+0000 7f3d79e00700 1 -- 172.21.15.31:0/1996718304 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d7419b410 con 0x7f3d74102dc0 2024-03-21T10:36:51.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.352+0000 7f3d78a00700 1 -- 172.21.15.31:0/1996718304 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6000e3b0 con 0x7f3d74102dc0 2024-03-21T10:36:51.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.352+0000 7f3d79e00700 1 -- 172.21.15.31:0/1996718304 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d741a76d0 con 0x7f3d74102dc0 2024-03-21T10:36:51.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.354+0000 7f3d79e00700 1 -- 172.21.15.31:0/1996718304 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d74068e50 con 0x7f3d74102dc0 2024-03-21T10:36:51.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.356+0000 7f3d78a00700 1 -- 172.21.15.31:0/1996718304 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7f3d6000f890 con 0x7f3d74102dc0 2024-03-21T10:36:51.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.358+0000 7f3d78a00700 1 --2- 172.21.15.31:0/1996718304 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f3d580a30c0 0x7f3d580a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:51.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.358+0000 7f3d78a00700 1 -- 172.21.15.31:0/1996718304 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f3d600cf340 con 0x7f3d74102dc0 2024-03-21T10:36:51.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.359+0000 7f3d72a00700 1 --2- 172.21.15.31:0/1996718304 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f3d580a30c0 0x7f3d580a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:51.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.359+0000 7f3d72a00700 1 --2- 172.21.15.31:0/1996718304 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f3d580a30c0 0x7f3d580a5580 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3d7419bb00 tx=0x7f3d6401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:51.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.362+0000 7f3d78a00700 1 -- 172.21.15.31:0/1996718304 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3d6009bc70 con 0x7f3d74102dc0 2024-03-21T10:36:51.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:51 smithi070 ceph-mon[36330]: pgmap v379: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:36:51.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:51 smithi040 ceph-mon[36154]: pgmap v379: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:36:51.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.814+0000 7f3d79e00700 1 -- 172.21.15.31:0/1996718304 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f3d741a7900 con 0x7f3d74102dc0 2024-03-21T10:36:51.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.814+0000 7f3d78a00700 1 -- 172.21.15.31:0/1996718304 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3d6009b570 con 0x7f3d74102dc0 2024-03-21T10:36:51.813 INFO:teuthology.orchestra.run.smithi031.stdout:34359738467 2024-03-21T10:36:51.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.817+0000 7f3d79e00700 1 -- 172.21.15.31:0/1996718304 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f3d580a30c0 msgr2=0x7f3d580a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:51.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.817+0000 7f3d79e00700 1 --2- 172.21.15.31:0/1996718304 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f3d580a30c0 0x7f3d580a5580 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3d7419bb00 tx=0x7f3d6401a040 comp rx=0 tx=0).stop 2024-03-21T10:36:51.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.817+0000 7f3d79e00700 1 -- 172.21.15.31:0/1996718304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d74102dc0 msgr2=0x7f3d741a0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:51.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.817+0000 7f3d79e00700 1 --2- 172.21.15.31:0/1996718304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d74102dc0 0x7f3d741a0d40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3d6000d980 tx=0x7f3d6000dc90 comp rx=0 tx=0).stop 2024-03-21T10:36:51.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.818+0000 7f3d79e00700 1 -- 172.21.15.31:0/1996718304 shutdown_connections 2024-03-21T10:36:51.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.818+0000 7f3d79e00700 1 --2- 172.21.15.31:0/1996718304 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f3d580a30c0 0x7f3d580a5580 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:51.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.818+0000 7f3d79e00700 1 --2- 172.21.15.31:0/1996718304 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d741074f0 0x7f3d7419a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:51.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.818+0000 7f3d79e00700 1 --2- 172.21.15.31:0/1996718304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d74102410 0x7f3d741a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:51.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.818+0000 7f3d79e00700 1 --2- 172.21.15.31:0/1996718304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d74102dc0 0x7f3d741a0d40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:51.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.818+0000 7f3d79e00700 1 -- 172.21.15.31:0/1996718304 >> 172.21.15.31:0/1996718304 conn(0x7f3d740fdc60 msgr2=0x7f3d7410e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:51.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.818+0000 7f3d79e00700 1 -- 172.21.15.31:0/1996718304 shutdown_connections 2024-03-21T10:36:51.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:51.818+0000 7f3d79e00700 1 -- 172.21.15.31:0/1996718304 wait complete. 2024-03-21T10:36:52.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:52 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1996718304' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-21T10:36:52.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:52 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1996718304' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-21T10:36:52.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:52 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1996718304' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-21T10:36:52.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.356+0000 7f13f7600700 1 -- 172.21.15.31:0/1611881731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13f01021e0 msgr2=0x7f13f0102660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:52.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.356+0000 7f13f7600700 1 --2- 172.21.15.31:0/1611881731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13f01021e0 0x7f13f0102660 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f13dc0099c0 tx=0x7f13dc009cd0 comp rx=0 tx=0).stop 2024-03-21T10:36:52.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.356+0000 7f13f7600700 1 -- 172.21.15.31:0/1611881731 shutdown_connections 2024-03-21T10:36:52.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.356+0000 7f13f7600700 1 --2- 172.21.15.31:0/1611881731 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13f0102ba0 0x7f13f010efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:52.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.356+0000 7f13f7600700 1 --2- 172.21.15.31:0/1611881731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13f01081c0 0x7f13f01085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:52.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.356+0000 7f13f7600700 1 --2- 172.21.15.31:0/1611881731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13f01021e0 0x7f13f0102660 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:52.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.356+0000 7f13f7600700 1 -- 172.21.15.31:0/1611881731 >> 172.21.15.31:0/1611881731 conn(0x7f13f00792c0 msgr2=0x7f13f00796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:52.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.356+0000 7f13f7600700 1 -- 172.21.15.31:0/1611881731 shutdown_connections 2024-03-21T10:36:52.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.356+0000 7f13f7600700 1 -- 172.21.15.31:0/1611881731 wait complete. 2024-03-21T10:36:52.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.357+0000 7f13f7600700 1 Processor -- start 2024-03-21T10:36:52.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.357+0000 7f13f7600700 1 -- start start 2024-03-21T10:36:52.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.358+0000 7f13f7600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13f01021e0 0x7f13f01a0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.358+0000 7f13f7600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13f0102ba0 0x7f13f01a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.358+0000 7f13f7600700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13f01081c0 0x7f13f019a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.358+0000 7f13f7600700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f01124c0 con 0x7f13f01081c0 2024-03-21T10:36:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.358+0000 7f13f7600700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f0112340 con 0x7f13f0102ba0 2024-03-21T10:36:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.358+0000 7f13f7600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f0112640 con 0x7f13f01021e0 2024-03-21T10:36:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.358+0000 7f13efe00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13f0102ba0 0x7f13f01a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.358+0000 7f13f4c00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13f01021e0 0x7f13f01a0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.358+0000 7f13efe00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13f0102ba0 0x7f13f01a0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42562/0 (socket says 172.21.15.31:42562) 2024-03-21T10:36:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.358+0000 7f13efe00700 1 -- 172.21.15.31:0/3853618208 learned_addr learned my addr 172.21.15.31:0/3853618208 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:36:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.358+0000 7f13f5600700 1 --2- 172.21.15.31:0/3853618208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13f01081c0 0x7f13f019a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.358+0000 7f13efe00700 1 -- 172.21.15.31:0/3853618208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13f01021e0 msgr2=0x7f13f01a0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.358+0000 7f13efe00700 1 --2- 172.21.15.31:0/3853618208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13f01021e0 0x7f13f01a0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.358+0000 7f13efe00700 1 -- 172.21.15.31:0/3853618208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13f01081c0 msgr2=0x7f13f019a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:52.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.358+0000 7f13efe00700 1 --2- 172.21.15.31:0/3853618208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13f01081c0 0x7f13f019a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:52.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.358+0000 7f13efe00700 1 -- 172.21.15.31:0/3853618208 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f019b060 con 0x7f13f0102ba0 2024-03-21T10:36:52.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.358+0000 7f13f5600700 1 --2- 172.21.15.31:0/3853618208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13f01081c0 0x7f13f019a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:36:52.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.359+0000 7f13efe00700 1 --2- 172.21.15.31:0/3853618208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13f0102ba0 0x7f13f01a0c80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f13dc006010 tx=0x7f13dc003e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:52.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.359+0000 7f13ed600700 1 -- 172.21.15.31:0/3853618208 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13dc009e60 con 0x7f13f0102ba0 2024-03-21T10:36:52.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.359+0000 7f13ed600700 1 -- 172.21.15.31:0/3853618208 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f13dc003680 con 0x7f13f0102ba0 2024-03-21T10:36:52.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.359+0000 7f13f7600700 1 -- 172.21.15.31:0/3853618208 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13f019b2f0 con 0x7f13f0102ba0 2024-03-21T10:36:52.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.359+0000 7f13ed600700 1 -- 172.21.15.31:0/3853618208 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13dc0173f0 con 0x7f13f0102ba0 2024-03-21T10:36:52.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.359+0000 7f13f7600700 1 -- 172.21.15.31:0/3853618208 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13f01a7610 con 0x7f13f0102ba0 2024-03-21T10:36:52.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.360+0000 7f13f7600700 1 -- 172.21.15.31:0/3853618208 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13f0068e50 con 0x7f13f0102ba0 2024-03-21T10:36:52.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.362+0000 7f13ed600700 1 -- 172.21.15.31:0/3853618208 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7f13dc00f680 con 0x7f13f0102ba0 2024-03-21T10:36:52.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.363+0000 7f13ed600700 1 --2- 172.21.15.31:0/3853618208 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f13d80a30c0 0x7f13d80a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:52.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.363+0000 7f13ed600700 1 -- 172.21.15.31:0/3853618208 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f13dc0d0680 con 0x7f13f0102ba0 2024-03-21T10:36:52.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.364+0000 7f13f4c00700 1 --2- 172.21.15.31:0/3853618208 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f13d80a30c0 0x7f13d80a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:52.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.364+0000 7f13f4c00700 1 --2- 172.21.15.31:0/3853618208 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f13d80a30c0 0x7f13d80a5580 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f13f019ba40 tx=0x7f13e400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:52.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.367+0000 7f13ed600700 1 -- 172.21.15.31:0/3853618208 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f13dc013070 con 0x7f13f0102ba0 2024-03-21T10:36:52.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.642+0000 7f13f7600700 1 -- 172.21.15.31:0/3853618208 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f13f01a77e0 con 0x7f13f0102ba0 2024-03-21T10:36:52.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.642+0000 7f13ed600700 1 -- 172.21.15.31:0/3853618208 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f13dc09d040 con 0x7f13f0102ba0 2024-03-21T10:36:52.777 INFO:teuthology.orchestra.run.smithi031.stdout:253403070517 2024-03-21T10:36:52.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.645+0000 7f13f7600700 1 -- 172.21.15.31:0/3853618208 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f13d80a30c0 msgr2=0x7f13d80a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:52.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.645+0000 7f13f7600700 1 --2- 172.21.15.31:0/3853618208 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f13d80a30c0 0x7f13d80a5580 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f13f019ba40 tx=0x7f13e400b410 comp rx=0 tx=0).stop 2024-03-21T10:36:52.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.645+0000 7f13f7600700 1 -- 172.21.15.31:0/3853618208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13f0102ba0 msgr2=0x7f13f01a0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:52.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.645+0000 7f13f7600700 1 --2- 172.21.15.31:0/3853618208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13f0102ba0 0x7f13f01a0c80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f13dc006010 tx=0x7f13dc003e00 comp rx=0 tx=0).stop 2024-03-21T10:36:52.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.645+0000 7f13f7600700 1 -- 172.21.15.31:0/3853618208 shutdown_connections 2024-03-21T10:36:52.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.645+0000 7f13f7600700 1 --2- 172.21.15.31:0/3853618208 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f13d80a30c0 0x7f13d80a5580 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:52.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.645+0000 7f13f7600700 1 --2- 172.21.15.31:0/3853618208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13f01081c0 0x7f13f019a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:52.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.645+0000 7f13f7600700 1 --2- 172.21.15.31:0/3853618208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13f01021e0 0x7f13f01a0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:52.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.645+0000 7f13f7600700 1 --2- 172.21.15.31:0/3853618208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13f0102ba0 0x7f13f01a0c80 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:52.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.645+0000 7f13f7600700 1 -- 172.21.15.31:0/3853618208 >> 172.21.15.31:0/3853618208 conn(0x7f13f00792c0 msgr2=0x7f13f00745f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:52.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.645+0000 7f13f7600700 1 -- 172.21.15.31:0/3853618208 shutdown_connections 2024-03-21T10:36:52.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:52.645+0000 7f13f7600700 1 -- 172.21.15.31:0/3853618208 wait complete. 2024-03-21T10:36:53.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:53 smithi031 ceph-mon[31673]: pgmap v380: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:36:53.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:53 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3853618208' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-21T10:36:53.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:53 smithi070 ceph-mon[36330]: pgmap v380: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:36:53.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:53 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3853618208' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-21T10:36:53.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:53 smithi040 ceph-mon[36154]: pgmap v380: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:36:53.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:53 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3853618208' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-21T10:36:54.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.161+0000 7fd111600700 1 -- 172.21.15.31:0/3918896418 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd10c06a7e0 msgr2=0x7fd10c10f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:54.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.161+0000 7fd111600700 1 --2- 172.21.15.31:0/3918896418 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd10c06a7e0 0x7fd10c10f020 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd10400b0a0 tx=0x7fd10400b3b0 comp rx=0 tx=0).stop 2024-03-21T10:36:54.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.161+0000 7fd111600700 1 -- 172.21.15.31:0/3918896418 shutdown_connections 2024-03-21T10:36:54.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.161+0000 7fd111600700 1 --2- 172.21.15.31:0/3918896418 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd10c06a7e0 0x7fd10c10f020 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:54.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.161+0000 7fd111600700 1 --2- 172.21.15.31:0/3918896418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd10c1081e0 0x7fd10c1085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:54.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.161+0000 7fd111600700 1 --2- 172.21.15.31:0/3918896418 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd10c069e40 0x7fd10c06a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:54.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.161+0000 7fd111600700 1 -- 172.21.15.31:0/3918896418 >> 172.21.15.31:0/3918896418 conn(0x7fd10c077fc0 msgr2=0x7fd10c0783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:54.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.161+0000 7fd111600700 1 -- 172.21.15.31:0/3918896418 shutdown_connections 2024-03-21T10:36:54.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.161+0000 7fd111600700 1 -- 172.21.15.31:0/3918896418 wait complete. 2024-03-21T10:36:54.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.163+0000 7fd111600700 1 Processor -- start 2024-03-21T10:36:54.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.163+0000 7fd111600700 1 -- start start 2024-03-21T10:36:54.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.163+0000 7fd111600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd10c069e40 0x7fd10c1a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:54.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.163+0000 7fd111600700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd10c06a7e0 0x7fd10c1a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:54.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.163+0000 7fd111600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd10c1081e0 0x7fd10c19a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:54.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.163+0000 7fd111600700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c1124b0 con 0x7fd10c06a7e0 2024-03-21T10:36:54.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.163+0000 7fd111600700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c112330 con 0x7fd10c1081e0 2024-03-21T10:36:54.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.163+0000 7fd111600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c112630 con 0x7fd10c069e40 2024-03-21T10:36:54.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.163+0000 7fd10aa00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd10c069e40 0x7fd10c1a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:54.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.164+0000 7fd10aa00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd10c069e40 0x7fd10c1a07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:51290/0 (socket says 172.21.15.31:51290) 2024-03-21T10:36:54.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.164+0000 7fd10aa00700 1 -- 172.21.15.31:0/3139333208 learned_addr learned my addr 172.21.15.31:0/3139333208 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:36:54.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.164+0000 7fd10a000700 1 --2- 172.21.15.31:0/3139333208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd10c06a7e0 0x7fd10c1a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:54.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.164+0000 7fd10b400700 1 --2- 172.21.15.31:0/3139333208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd10c1081e0 0x7fd10c19a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:54.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.164+0000 7fd10aa00700 1 -- 172.21.15.31:0/3139333208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd10c1081e0 msgr2=0x7fd10c19a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:54.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.164+0000 7fd10aa00700 1 --2- 172.21.15.31:0/3139333208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd10c1081e0 0x7fd10c19a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:54.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.164+0000 7fd10aa00700 1 -- 172.21.15.31:0/3139333208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd10c06a7e0 msgr2=0x7fd10c1a0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:54.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.164+0000 7fd10aa00700 1 --2- 172.21.15.31:0/3139333208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd10c06a7e0 0x7fd10c1a0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:54.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.164+0000 7fd10aa00700 1 -- 172.21.15.31:0/3139333208 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd10c19b0e0 con 0x7fd10c069e40 2024-03-21T10:36:54.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.164+0000 7fd10a000700 1 --2- 172.21.15.31:0/3139333208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd10c06a7e0 0x7fd10c1a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:36:54.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.164+0000 7fd10aa00700 1 --2- 172.21.15.31:0/3139333208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd10c069e40 0x7fd10c1a07c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc00e9a0 tx=0x7fd0fc00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:54.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.165+0000 7fd103400700 1 -- 172.21.15.31:0/3139333208 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0fc00cbd0 con 0x7fd10c069e40 2024-03-21T10:36:54.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.165+0000 7fd103400700 1 -- 172.21.15.31:0/3139333208 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd0fc00cd70 con 0x7fd10c069e40 2024-03-21T10:36:54.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.165+0000 7fd111600700 1 -- 172.21.15.31:0/3139333208 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd10c19b3d0 con 0x7fd10c069e40 2024-03-21T10:36:54.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.165+0000 7fd103400700 1 -- 172.21.15.31:0/3139333208 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0fc018750 con 0x7fd10c069e40 2024-03-21T10:36:54.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.165+0000 7fd111600700 1 -- 172.21.15.31:0/3139333208 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd10c1a7690 con 0x7fd10c069e40 2024-03-21T10:36:54.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.166+0000 7fd111600700 1 -- 172.21.15.31:0/3139333208 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd10c068e50 con 0x7fd10c069e40 2024-03-21T10:36:54.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.174+0000 7fd103400700 1 -- 172.21.15.31:0/3139333208 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) v1 ==== 137905+0+0 (secure 0 0 0) 0x7fd0fc010430 con 0x7fd10c069e40 2024-03-21T10:36:54.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.176+0000 7fd103400700 1 --2- 172.21.15.31:0/3139333208 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd0f00a3040 0x7fd0f00a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:54.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.177+0000 7fd103400700 1 -- 172.21.15.31:0/3139333208 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fd0fc0d0360 con 0x7fd10c069e40 2024-03-21T10:36:54.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.177+0000 7fd10a000700 1 --2- 172.21.15.31:0/3139333208 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd0f00a3040 0x7fd0f00a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:54.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.177+0000 7fd103400700 1 -- 172.21.15.31:0/3139333208 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd0fc0d07e0 con 0x7fd10c069e40 2024-03-21T10:36:54.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.178+0000 7fd10a000700 1 --2- 172.21.15.31:0/3139333208 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd0f00a3040 0x7fd0f00a5500 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd10c103040 tx=0x7fd0f8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:54.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.295+0000 7fd103400700 1 -- 172.21.15.31:0/3139333208 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fd0fc098c00 con 0x7fd10c069e40 2024-03-21T10:36:54.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.476+0000 7fd111600700 1 -- 172.21.15.31:0/3139333208 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fd10c1a78c0 con 0x7fd10c069e40 2024-03-21T10:36:54.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.476+0000 7fd103400700 1 -- 172.21.15.31:0/3139333208 <== mon.1 v2:172.21.15.70:3300/0 8 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd0fc09cc90 con 0x7fd10c069e40 2024-03-21T10:36:54.475 INFO:teuthology.orchestra.run.smithi031.stdout:163208757318 2024-03-21T10:36:54.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.480+0000 7fd111600700 1 -- 172.21.15.31:0/3139333208 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd0f00a3040 msgr2=0x7fd0f00a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:54.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.480+0000 7fd111600700 1 --2- 172.21.15.31:0/3139333208 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd0f00a3040 0x7fd0f00a5500 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd10c103040 tx=0x7fd0f8009260 comp rx=0 tx=0).stop 2024-03-21T10:36:54.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.480+0000 7fd111600700 1 -- 172.21.15.31:0/3139333208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd10c069e40 msgr2=0x7fd10c1a07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:54.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.480+0000 7fd111600700 1 --2- 172.21.15.31:0/3139333208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd10c069e40 0x7fd10c1a07c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc00e9a0 tx=0x7fd0fc00ecb0 comp rx=0 tx=0).stop 2024-03-21T10:36:54.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.481+0000 7fd111600700 1 -- 172.21.15.31:0/3139333208 shutdown_connections 2024-03-21T10:36:54.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.481+0000 7fd111600700 1 --2- 172.21.15.31:0/3139333208 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd0f00a3040 0x7fd0f00a5500 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:54.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.481+0000 7fd111600700 1 --2- 172.21.15.31:0/3139333208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd10c1081e0 0x7fd10c19a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:54.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.481+0000 7fd111600700 1 --2- 172.21.15.31:0/3139333208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd10c069e40 0x7fd10c1a07c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:54.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.481+0000 7fd111600700 1 --2- 172.21.15.31:0/3139333208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd10c06a7e0 0x7fd10c1a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:54.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.481+0000 7fd111600700 1 -- 172.21.15.31:0/3139333208 >> 172.21.15.31:0/3139333208 conn(0x7fd10c077fc0 msgr2=0x7fd10c10d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:54.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.481+0000 7fd111600700 1 -- 172.21.15.31:0/3139333208 shutdown_connections 2024-03-21T10:36:54.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.481+0000 7fd111600700 1 -- 172.21.15.31:0/3139333208 wait complete. 2024-03-21T10:36:54.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.515+0000 7efcfa200700 1 -- 172.21.15.31:0/418025824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcf4069e40 msgr2=0x7efcf406a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:54.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.515+0000 7efcfa200700 1 --2- 172.21.15.31:0/418025824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcf4069e40 0x7efcf406a2a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7efce0009990 tx=0x7efce0009ca0 comp rx=0 tx=0).stop 2024-03-21T10:36:54.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.515+0000 7efcfa200700 1 -- 172.21.15.31:0/418025824 shutdown_connections 2024-03-21T10:36:54.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.515+0000 7efcfa200700 1 --2- 172.21.15.31:0/418025824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcf406a7e0 0x7efcf410f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:54.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.515+0000 7efcfa200700 1 --2- 172.21.15.31:0/418025824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcf41081e0 0x7efcf41085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:54.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.515+0000 7efcfa200700 1 --2- 172.21.15.31:0/418025824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcf4069e40 0x7efcf406a2a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:54.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.515+0000 7efcfa200700 1 -- 172.21.15.31:0/418025824 >> 172.21.15.31:0/418025824 conn(0x7efcf4077fc0 msgr2=0x7efcf40783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:54.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.516+0000 7efcfa200700 1 -- 172.21.15.31:0/418025824 shutdown_connections 2024-03-21T10:36:54.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.516+0000 7efcfa200700 1 -- 172.21.15.31:0/418025824 wait complete. 2024-03-21T10:36:54.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.517+0000 7efcfa200700 1 Processor -- start 2024-03-21T10:36:54.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.517+0000 7efcfa200700 1 -- start start 2024-03-21T10:36:54.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.518+0000 7efcfa200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcf4069e40 0x7efcf41a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:54.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.518+0000 7efcfa200700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcf406a7e0 0x7efcf41a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:54.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.518+0000 7efcfa200700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcf41081e0 0x7efcf419a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:54.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.518+0000 7efcfa200700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf4112490 con 0x7efcf41081e0 2024-03-21T10:36:54.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.518+0000 7efcfa200700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf4112310 con 0x7efcf406a7e0 2024-03-21T10:36:54.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.518+0000 7efcfa200700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf4112610 con 0x7efcf4069e40 2024-03-21T10:36:54.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.518+0000 7efcf3400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcf4069e40 0x7efcf41a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:54.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.518+0000 7efcf3400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcf4069e40 0x7efcf41a07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:51300/0 (socket says 172.21.15.31:51300) 2024-03-21T10:36:54.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.518+0000 7efcf3400700 1 -- 172.21.15.31:0/3594369937 learned_addr learned my addr 172.21.15.31:0/3594369937 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:36:54.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.518+0000 7efcf2a00700 1 --2- 172.21.15.31:0/3594369937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcf406a7e0 0x7efcf41a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:54.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.518+0000 7efcf3400700 1 -- 172.21.15.31:0/3594369937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcf406a7e0 msgr2=0x7efcf41a0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:54.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.518+0000 7efcf3400700 1 --2- 172.21.15.31:0/3594369937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcf406a7e0 0x7efcf41a0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:54.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.519+0000 7efcf3400700 1 -- 172.21.15.31:0/3594369937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcf41081e0 msgr2=0x7efcf419a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:54.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.519+0000 7efcf3400700 1 --2- 172.21.15.31:0/3594369937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcf41081e0 0x7efcf419a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:54.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.519+0000 7efcf3400700 1 -- 172.21.15.31:0/3594369937 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf419b0f0 con 0x7efcf4069e40 2024-03-21T10:36:54.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.519+0000 7efcf3400700 1 --2- 172.21.15.31:0/3594369937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcf4069e40 0x7efcf41a07d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7efce400e9a0 tx=0x7efce400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:54.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.519+0000 7efcebe00700 1 -- 172.21.15.31:0/3594369937 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce400cb50 con 0x7efcf4069e40 2024-03-21T10:36:54.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.519+0000 7efcebe00700 1 -- 172.21.15.31:0/3594369937 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efce400ccf0 con 0x7efcf4069e40 2024-03-21T10:36:54.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.519+0000 7efcebe00700 1 -- 172.21.15.31:0/3594369937 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce4018680 con 0x7efcf4069e40 2024-03-21T10:36:54.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.520+0000 7efcfa200700 1 -- 172.21.15.31:0/3594369937 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf419b3e0 con 0x7efcf4069e40 2024-03-21T10:36:54.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.520+0000 7efcfa200700 1 -- 172.21.15.31:0/3594369937 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf41a76a0 con 0x7efcf4069e40 2024-03-21T10:36:54.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.521+0000 7efcfa200700 1 -- 172.21.15.31:0/3594369937 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf4068e50 con 0x7efcf4069e40 2024-03-21T10:36:54.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.523+0000 7efcebe00700 1 -- 172.21.15.31:0/3594369937 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7efce401f020 con 0x7efcf4069e40 2024-03-21T10:36:54.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.524+0000 7efcebe00700 1 --2- 172.21.15.31:0/3594369937 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7efcd80a3030 0x7efcd80a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:54.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.524+0000 7efcebe00700 1 -- 172.21.15.31:0/3594369937 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7efce40cf5c0 con 0x7efcf4069e40 2024-03-21T10:36:54.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.524+0000 7efcf2a00700 1 --2- 172.21.15.31:0/3594369937 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7efcd80a3030 0x7efcd80a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:54.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.525+0000 7efcf2a00700 1 --2- 172.21.15.31:0/3594369937 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7efcd80a3030 0x7efcd80a54f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7efcf4103180 tx=0x7efce000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:54.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.528+0000 7efcebe00700 1 -- 172.21.15.31:0/3594369937 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efce409c050 con 0x7efcf4069e40 2024-03-21T10:36:54.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.783+0000 7efcfa200700 1 -- 172.21.15.31:0/3594369937 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7efcf41a7910 con 0x7efcf4069e40 2024-03-21T10:36:54.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.784+0000 7efcebe00700 1 -- 172.21.15.31:0/3594369937 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7efce409b950 con 0x7efcf4069e40 2024-03-21T10:36:54.782 INFO:teuthology.orchestra.run.smithi031.stdout:309237645353 2024-03-21T10:36:54.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.787+0000 7efcfa200700 1 -- 172.21.15.31:0/3594369937 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7efcd80a3030 msgr2=0x7efcd80a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:54.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.787+0000 7efcfa200700 1 --2- 172.21.15.31:0/3594369937 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7efcd80a3030 0x7efcd80a54f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7efcf4103180 tx=0x7efce000b260 comp rx=0 tx=0).stop 2024-03-21T10:36:54.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.787+0000 7efcfa200700 1 -- 172.21.15.31:0/3594369937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcf4069e40 msgr2=0x7efcf41a07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:54.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.787+0000 7efcfa200700 1 --2- 172.21.15.31:0/3594369937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcf4069e40 0x7efcf41a07d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7efce400e9a0 tx=0x7efce400ecb0 comp rx=0 tx=0).stop 2024-03-21T10:36:54.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.788+0000 7efcfa200700 1 -- 172.21.15.31:0/3594369937 shutdown_connections 2024-03-21T10:36:54.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.788+0000 7efcfa200700 1 --2- 172.21.15.31:0/3594369937 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7efcd80a3030 0x7efcd80a54f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:54.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.788+0000 7efcfa200700 1 --2- 172.21.15.31:0/3594369937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcf41081e0 0x7efcf419a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:54.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.788+0000 7efcfa200700 1 --2- 172.21.15.31:0/3594369937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcf4069e40 0x7efcf41a07d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:54.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.788+0000 7efcfa200700 1 --2- 172.21.15.31:0/3594369937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcf406a7e0 0x7efcf41a0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:54.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.788+0000 7efcfa200700 1 -- 172.21.15.31:0/3594369937 >> 172.21.15.31:0/3594369937 conn(0x7efcf4077fc0 msgr2=0x7efcf410d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:54.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.788+0000 7efcfa200700 1 -- 172.21.15.31:0/3594369937 shutdown_connections 2024-03-21T10:36:54.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:54.788+0000 7efcfa200700 1 -- 172.21.15.31:0/3594369937 wait complete. 2024-03-21T10:36:55.542 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:55 smithi031 ceph-mon[31673]: pgmap v381: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:36:55.542 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:55 smithi031 ceph-mon[31673]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-21T10:36:55.542 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:55 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]: dispatch 2024-03-21T10:36:55.542 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:55 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3139333208' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-21T10:36:55.542 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:55 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3594369937' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-21T10:36:55.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:55 smithi070 ceph-mon[36330]: pgmap v381: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:36:55.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:55 smithi070 ceph-mon[36330]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-21T10:36:55.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:55 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]: dispatch 2024-03-21T10:36:55.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:55 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3139333208' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-21T10:36:55.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:55 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3594369937' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-21T10:36:55.607 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070510 got 253403070517 for osd.8 2024-03-21T10:36:55.607 DEBUG:teuthology.parallel:result is None 2024-03-21T10:36:55.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:55 smithi040 ceph-mon[36154]: pgmap v381: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:36:55.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:55 smithi040 ceph-mon[36154]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-21T10:36:55.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:55 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]: dispatch 2024-03-21T10:36:55.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:55 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3139333208' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-21T10:36:55.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:55 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3594369937' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-21T10:36:56.084 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738462 got 34359738467 for osd.0 2024-03-21T10:36:56.085 DEBUG:teuthology.parallel:result is None 2024-03-21T10:36:56.302 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:56 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]': finished 2024-03-21T10:36:56.302 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:56 smithi031 ceph-mon[31673]: osdmap e92: 12 total, 12 up, 12 in 2024-03-21T10:36:56.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:56 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]': finished 2024-03-21T10:36:56.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:56 smithi070 ceph-mon[36330]: osdmap e92: 12 total, 12 up, 12 in 2024-03-21T10:36:56.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:56 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]': finished 2024-03-21T10:36:56.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:56 smithi040 ceph-mon[36154]: osdmap e92: 12 total, 12 up, 12 in 2024-03-21T10:36:56.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.936+0000 7fd6b9400700 1 -- 172.21.15.31:0/1830221180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6b4074530 msgr2=0x7fd6b4074910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:56.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.936+0000 7fd6b9400700 1 --2- 172.21.15.31:0/1830221180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6b4074530 0x7fd6b4074910 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4009990 tx=0x7fd6a4009ca0 comp rx=0 tx=0).stop 2024-03-21T10:36:56.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.937+0000 7fd6b9400700 1 -- 172.21.15.31:0/1830221180 shutdown_connections 2024-03-21T10:36:56.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.937+0000 7fd6b9400700 1 --2- 172.21.15.31:0/1830221180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6b4113180 0x7fd6b4115570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:56.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.937+0000 7fd6b9400700 1 --2- 172.21.15.31:0/1830221180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6b4074530 0x7fd6b4074910 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:56.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.937+0000 7fd6b9400700 1 --2- 172.21.15.31:0/1830221180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6b4074e50 0x7fd6b4112c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:56.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.937+0000 7fd6b9400700 1 -- 172.21.15.31:0/1830221180 >> 172.21.15.31:0/1830221180 conn(0x7fd6b40fdc40 msgr2=0x7fd6b4100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:56.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.937+0000 7fd6b9400700 1 -- 172.21.15.31:0/1830221180 shutdown_connections 2024-03-21T10:36:56.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.937+0000 7fd6b9400700 1 -- 172.21.15.31:0/1830221180 wait complete. 2024-03-21T10:36:56.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.939+0000 7fd6b9400700 1 Processor -- start 2024-03-21T10:36:56.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.939+0000 7fd6b9400700 1 -- start start 2024-03-21T10:36:56.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.939+0000 7fd6b9400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6b4074530 0x7fd6b419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:56.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.939+0000 7fd6b9400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6b4074e50 0x7fd6b419f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:56.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.939+0000 7fd6b9400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6b4113180 0x7fd6b41a3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:56.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.939+0000 7fd6b9400700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b4118ae0 con 0x7fd6b4113180 2024-03-21T10:36:56.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.939+0000 7fd6b9400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b4118960 con 0x7fd6b4074e50 2024-03-21T10:36:56.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.939+0000 7fd6b9400700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b4118c60 con 0x7fd6b4074530 2024-03-21T10:36:56.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.940+0000 7fd6b1e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6b4074e50 0x7fd6b419f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:56.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.940+0000 7fd6b2800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6b4074530 0x7fd6b419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:56.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.940+0000 7fd6b2800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6b4074530 0x7fd6b419efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42096/0 (socket says 172.21.15.31:42096) 2024-03-21T10:36:56.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.940+0000 7fd6b1e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6b4074e50 0x7fd6b419f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33668/0 (socket says 172.21.15.31:33668) 2024-03-21T10:36:56.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.940+0000 7fd6b2800700 1 -- 172.21.15.31:0/1864964939 learned_addr learned my addr 172.21.15.31:0/1864964939 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:36:56.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.940+0000 7fd6b3200700 1 --2- 172.21.15.31:0/1864964939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6b4113180 0x7fd6b41a3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:56.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.940+0000 7fd6b1e00700 1 -- 172.21.15.31:0/1864964939 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6b4074530 msgr2=0x7fd6b419efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:56.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.940+0000 7fd6b1e00700 1 --2- 172.21.15.31:0/1864964939 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6b4074530 0x7fd6b419efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:56.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.940+0000 7fd6b1e00700 1 -- 172.21.15.31:0/1864964939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6b4113180 msgr2=0x7fd6b41a3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:56.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.940+0000 7fd6b1e00700 1 --2- 172.21.15.31:0/1864964939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6b4113180 0x7fd6b41a3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:56.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.940+0000 7fd6b1e00700 1 -- 172.21.15.31:0/1864964939 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b41a3f90 con 0x7fd6b4074e50 2024-03-21T10:36:56.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.940+0000 7fd6b3200700 1 --2- 172.21.15.31:0/1864964939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6b4113180 0x7fd6b41a3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:36:56.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.940+0000 7fd6b2800700 1 --2- 172.21.15.31:0/1864964939 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6b4074530 0x7fd6b419efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:36:56.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.941+0000 7fd6b1e00700 1 --2- 172.21.15.31:0/1864964939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6b4074e50 0x7fd6b419f510 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd6a000d950 tx=0x7fd6a000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:56.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.941+0000 7fd6ab400700 1 -- 172.21.15.31:0/1864964939 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a0017070 con 0x7fd6b4074e50 2024-03-21T10:36:56.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.941+0000 7fd6ab400700 1 -- 172.21.15.31:0/1864964939 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6a0009e10 con 0x7fd6b4074e50 2024-03-21T10:36:56.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.941+0000 7fd6b9400700 1 -- 172.21.15.31:0/1864964939 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6b41a4280 con 0x7fd6b4074e50 2024-03-21T10:36:56.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.941+0000 7fd6ab400700 1 -- 172.21.15.31:0/1864964939 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a000f3c0 con 0x7fd6b4074e50 2024-03-21T10:36:56.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.941+0000 7fd6b9400700 1 -- 172.21.15.31:0/1864964939 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6b41abb00 con 0x7fd6b4074e50 2024-03-21T10:36:56.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.943+0000 7fd6b9400700 1 -- 172.21.15.31:0/1864964939 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6b4068e50 con 0x7fd6b4074e50 2024-03-21T10:36:56.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.944+0000 7fd6ab400700 1 -- 172.21.15.31:0/1864964939 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fd6a001a070 con 0x7fd6b4074e50 2024-03-21T10:36:56.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.946+0000 7fd6ab400700 1 --2- 172.21.15.31:0/1864964939 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd6980a3030 0x7fd6980a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:56.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.946+0000 7fd6ab400700 1 -- 172.21.15.31:0/1864964939 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fd6a00cf100 con 0x7fd6b4074e50 2024-03-21T10:36:56.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.946+0000 7fd6b2800700 1 --2- 172.21.15.31:0/1864964939 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd6980a3030 0x7fd6980a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:56.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.947+0000 7fd6b2800700 1 --2- 172.21.15.31:0/1864964939 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd6980a3030 0x7fd6980a54f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4006010 tx=0x7fd6a400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:56.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:56.950+0000 7fd6ab400700 1 -- 172.21.15.31:0/1864964939 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd6a009bb70 con 0x7fd6b4074e50 2024-03-21T10:36:57.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.190+0000 7fd6b9400700 1 -- 172.21.15.31:0/1864964939 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fd6b402da30 con 0x7fd6b4074e50 2024-03-21T10:36:57.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.190+0000 7fd6ab400700 1 -- 172.21.15.31:0/1864964939 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd6a009b470 con 0x7fd6b4074e50 2024-03-21T10:36:57.189 INFO:teuthology.orchestra.run.smithi031.stdout:107374182481 2024-03-21T10:36:57.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.195+0000 7fd6b9400700 1 -- 172.21.15.31:0/1864964939 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd6980a3030 msgr2=0x7fd6980a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:57.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.195+0000 7fd6b9400700 1 --2- 172.21.15.31:0/1864964939 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd6980a3030 0x7fd6980a54f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4006010 tx=0x7fd6a400b260 comp rx=0 tx=0).stop 2024-03-21T10:36:57.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.195+0000 7fd6b9400700 1 -- 172.21.15.31:0/1864964939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6b4074e50 msgr2=0x7fd6b419f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:57.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.195+0000 7fd6b9400700 1 --2- 172.21.15.31:0/1864964939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6b4074e50 0x7fd6b419f510 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd6a000d950 tx=0x7fd6a000dc60 comp rx=0 tx=0).stop 2024-03-21T10:36:57.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.195+0000 7fd6b9400700 1 -- 172.21.15.31:0/1864964939 shutdown_connections 2024-03-21T10:36:57.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.195+0000 7fd6b9400700 1 --2- 172.21.15.31:0/1864964939 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd6980a3030 0x7fd6980a54f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:57.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.195+0000 7fd6b9400700 1 --2- 172.21.15.31:0/1864964939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6b4113180 0x7fd6b41a3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:57.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.195+0000 7fd6b9400700 1 --2- 172.21.15.31:0/1864964939 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6b4074530 0x7fd6b419efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:57.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.195+0000 7fd6b9400700 1 --2- 172.21.15.31:0/1864964939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6b4074e50 0x7fd6b419f510 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:57.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.195+0000 7fd6b9400700 1 -- 172.21.15.31:0/1864964939 >> 172.21.15.31:0/1864964939 conn(0x7fd6b40fdc40 msgr2=0x7fd6b40fe950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:57.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.196+0000 7fd6b9400700 1 -- 172.21.15.31:0/1864964939 shutdown_connections 2024-03-21T10:36:57.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.196+0000 7fd6b9400700 1 -- 172.21.15.31:0/1864964939 wait complete. 2024-03-21T10:36:57.223 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645346 got 309237645353 for osd.10 2024-03-21T10:36:57.223 DEBUG:teuthology.parallel:result is None 2024-03-21T10:36:57.453 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:57 smithi031 ceph-mon[31673]: pgmap v383: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-21T10:36:57.453 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:57 smithi031 ceph-mon[31673]: osdmap e93: 12 total, 12 up, 12 in 2024-03-21T10:36:57.453 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:57 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1864964939' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-21T10:36:57.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:57 smithi070 ceph-mon[36330]: pgmap v383: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-21T10:36:57.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:57 smithi070 ceph-mon[36330]: osdmap e93: 12 total, 12 up, 12 in 2024-03-21T10:36:57.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:57 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1864964939' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-21T10:36:57.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:57 smithi040 ceph-mon[36154]: pgmap v383: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-21T10:36:57.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:57 smithi040 ceph-mon[36154]: osdmap e93: 12 total, 12 up, 12 in 2024-03-21T10:36:57.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:57 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1864964939' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-21T10:36:58.465 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:58 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2932259841' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-21T10:36:58.465 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:58 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:36:58.465 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:58 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:36:58.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:58 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2932259841' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-21T10:36:58.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:58 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:36:58.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:58 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:36:58.600 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182474 got 107374182481 for osd.3 2024-03-21T10:36:58.600 DEBUG:teuthology.parallel:result is None 2024-03-21T10:36:58.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:58 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2932259841' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-21T10:36:58.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:58 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:36:58.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:58 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:36:58.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.916+0000 7fa982600700 1 -- 172.21.15.31:0/3956121587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa97c103240 msgr2=0x7fa97c103620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:58.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.916+0000 7fa982600700 1 --2- 172.21.15.31:0/3956121587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa97c103240 0x7fa97c103620 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa9700098f0 tx=0x7fa970009c00 comp rx=0 tx=0).stop 2024-03-21T10:36:58.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.916+0000 7fa982600700 1 -- 172.21.15.31:0/3956121587 shutdown_connections 2024-03-21T10:36:58.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.916+0000 7fa982600700 1 --2- 172.21.15.31:0/3956121587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa97c10c670 0x7fa97c10eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:58.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.916+0000 7fa982600700 1 --2- 172.21.15.31:0/3956121587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa97c103240 0x7fa97c103620 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:58.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.916+0000 7fa982600700 1 --2- 172.21.15.31:0/3956121587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa97c103b60 0x7fa97c10c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:58.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.916+0000 7fa982600700 1 -- 172.21.15.31:0/3956121587 >> 172.21.15.31:0/3956121587 conn(0x7fa97c0fca50 msgr2=0x7fa97c0fee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:58.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.916+0000 7fa982600700 1 -- 172.21.15.31:0/3956121587 shutdown_connections 2024-03-21T10:36:58.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.917+0000 7fa982600700 1 -- 172.21.15.31:0/3956121587 wait complete. 2024-03-21T10:36:58.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.918+0000 7fa982600700 1 Processor -- start 2024-03-21T10:36:58.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.918+0000 7fa982600700 1 -- start start 2024-03-21T10:36:58.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.918+0000 7fa982600700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa97c103240 0x7fa97c1a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:58.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.918+0000 7fa982600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa97c103b60 0x7fa97c1a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:58.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.918+0000 7fa982600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa97c10c670 0x7fa97c19a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:58.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.918+0000 7fa982600700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97c112400 con 0x7fa97c103240 2024-03-21T10:36:58.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.918+0000 7fa982600700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97c112280 con 0x7fa97c10c670 2024-03-21T10:36:58.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.918+0000 7fa982600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97c112580 con 0x7fa97c103b60 2024-03-21T10:36:58.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.919+0000 7fa981200700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa97c103240 0x7fa97c1a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:58.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.919+0000 7fa97be00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa97c103b60 0x7fa97c1a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:58.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.919+0000 7fa981c00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa97c10c670 0x7fa97c19a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:58.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.919+0000 7fa97be00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa97c103b60 0x7fa97c1a0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42114/0 (socket says 172.21.15.31:42114) 2024-03-21T10:36:58.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.919+0000 7fa97be00700 1 -- 172.21.15.31:0/2932259841 learned_addr learned my addr 172.21.15.31:0/2932259841 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:36:58.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.919+0000 7fa981200700 1 --2- 172.21.15.31:0/2932259841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa97c103240 0x7fa97c1a07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58776/0 (socket says 172.21.15.31:58776) 2024-03-21T10:36:58.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.919+0000 7fa97be00700 1 -- 172.21.15.31:0/2932259841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa97c10c670 msgr2=0x7fa97c19a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:58.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.919+0000 7fa97be00700 1 --2- 172.21.15.31:0/2932259841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa97c10c670 0x7fa97c19a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:58.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.919+0000 7fa97be00700 1 -- 172.21.15.31:0/2932259841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa97c103240 msgr2=0x7fa97c1a07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:58.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.919+0000 7fa97be00700 1 --2- 172.21.15.31:0/2932259841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa97c103240 0x7fa97c1a07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:58.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.919+0000 7fa97be00700 1 -- 172.21.15.31:0/2932259841 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa97c19b0e0 con 0x7fa97c103b60 2024-03-21T10:36:58.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.919+0000 7fa97be00700 1 --2- 172.21.15.31:0/2932259841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa97c103b60 0x7fa97c1a0d00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa96800e9c0 tx=0x7fa96800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:58.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.920+0000 7fa979600700 1 -- 172.21.15.31:0/2932259841 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa968004a30 con 0x7fa97c103b60 2024-03-21T10:36:58.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.920+0000 7fa979600700 1 -- 172.21.15.31:0/2932259841 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa96800cca0 con 0x7fa97c103b60 2024-03-21T10:36:58.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.920+0000 7fa982600700 1 -- 172.21.15.31:0/2932259841 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa97c19b3d0 con 0x7fa97c103b60 2024-03-21T10:36:58.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.920+0000 7fa979600700 1 -- 172.21.15.31:0/2932259841 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa968005610 con 0x7fa97c103b60 2024-03-21T10:36:58.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.920+0000 7fa982600700 1 -- 172.21.15.31:0/2932259841 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa97c1a7690 con 0x7fa97c103b60 2024-03-21T10:36:58.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.921+0000 7fa982600700 1 -- 172.21.15.31:0/2932259841 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa97c068e50 con 0x7fa97c103b60 2024-03-21T10:36:58.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.923+0000 7fa979600700 1 -- 172.21.15.31:0/2932259841 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fa96800c7c0 con 0x7fa97c103b60 2024-03-21T10:36:58.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.924+0000 7fa979600700 1 --2- 172.21.15.31:0/2932259841 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa9640a3030 0x7fa9640a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:36:58.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.924+0000 7fa979600700 1 -- 172.21.15.31:0/2932259841 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fa9680cf820 con 0x7fa97c103b60 2024-03-21T10:36:58.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.924+0000 7fa981200700 1 --2- 172.21.15.31:0/2932259841 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa9640a3030 0x7fa9640a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:36:58.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.925+0000 7fa981200700 1 --2- 172.21.15.31:0/2932259841 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa9640a3030 0x7fa9640a54f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa9700098c0 tx=0x7fa970019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:36:58.707 INFO:teuthology.orchestra.run.smithi031.stdout:55834574943 2024-03-21T10:36:58.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:57.932+0000 7fa979600700 1 -- 172.21.15.31:0/2932259841 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa96809c290 con 0x7fa97c103b60 2024-03-21T10:36:58.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:58.196+0000 7fa982600700 1 -- 172.21.15.31:0/2932259841 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fa97c19beb0 con 0x7fa97c103b60 2024-03-21T10:36:58.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:58.197+0000 7fa979600700 1 -- 172.21.15.31:0/2932259841 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa97c19beb0 con 0x7fa97c103b60 2024-03-21T10:36:58.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:58.200+0000 7fa982600700 1 -- 172.21.15.31:0/2932259841 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa9640a3030 msgr2=0x7fa9640a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:58.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:58.200+0000 7fa982600700 1 --2- 172.21.15.31:0/2932259841 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa9640a3030 0x7fa9640a54f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa9700098c0 tx=0x7fa970019040 comp rx=0 tx=0).stop 2024-03-21T10:36:58.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:58.200+0000 7fa982600700 1 -- 172.21.15.31:0/2932259841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa97c103b60 msgr2=0x7fa97c1a0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:36:58.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:58.200+0000 7fa982600700 1 --2- 172.21.15.31:0/2932259841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa97c103b60 0x7fa97c1a0d00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa96800e9c0 tx=0x7fa96800ecd0 comp rx=0 tx=0).stop 2024-03-21T10:36:58.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:58.200+0000 7fa982600700 1 -- 172.21.15.31:0/2932259841 shutdown_connections 2024-03-21T10:36:58.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:58.200+0000 7fa982600700 1 --2- 172.21.15.31:0/2932259841 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa9640a3030 0x7fa9640a54f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:58.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:58.200+0000 7fa982600700 1 --2- 172.21.15.31:0/2932259841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa97c10c670 0x7fa97c19a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:58.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:58.200+0000 7fa982600700 1 --2- 172.21.15.31:0/2932259841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa97c103240 0x7fa97c1a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:58.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:58.200+0000 7fa982600700 1 --2- 172.21.15.31:0/2932259841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa97c103b60 0x7fa97c1a0d00 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:36:58.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:58.200+0000 7fa982600700 1 -- 172.21.15.31:0/2932259841 >> 172.21.15.31:0/2932259841 conn(0x7fa97c0fca50 msgr2=0x7fa97c0fee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:36:58.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:58.201+0000 7fa982600700 1 -- 172.21.15.31:0/2932259841 shutdown_connections 2024-03-21T10:36:58.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:36:58.201+0000 7fa982600700 1 -- 172.21.15.31:0/2932259841 wait complete. 2024-03-21T10:36:59.555 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757312 got 163208757318 for osd.5 2024-03-21T10:36:59.555 DEBUG:teuthology.parallel:result is None 2024-03-21T10:36:59.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:36:59 smithi070 ceph-mon[36330]: pgmap v385: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 639 B/s wr, 7 op/s 2024-03-21T10:36:59.620 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:36:59 smithi031 ceph-mon[31673]: pgmap v385: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 639 B/s wr, 7 op/s 2024-03-21T10:36:59.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:36:59 smithi040 ceph-mon[36154]: pgmap v385: 129 pgs: 129 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 639 B/s wr, 7 op/s 2024-03-21T10:37:00.059 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495675 got 197568495680 for osd.6 2024-03-21T10:37:00.059 DEBUG:teuthology.parallel:result is None 2024-03-21T10:37:00.592 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:00 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:37:00.592 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:00 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:37:00.592 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:00 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:37:00.595 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411409 got 77309411414 for osd.2 2024-03-21T10:37:00.595 DEBUG:teuthology.parallel:result is None 2024-03-21T10:37:00.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:00 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:37:00.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:00 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:37:00.682 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:00 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:37:00.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:00 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:37:00.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:00 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:37:00.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:00 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:37:00.878 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:37:00.881 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:37:01.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.166+0000 7ffa53400700 1 -- 172.21.15.31:0/2457045993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c102bd0 msgr2=0x7ffa4c10f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:01.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.166+0000 7ffa53400700 1 --2- 172.21.15.31:0/2457045993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c102bd0 0x7ffa4c10f060 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ffa4400b0d0 tx=0x7ffa4400b3e0 comp rx=0 tx=0).stop 2024-03-21T10:37:01.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.166+0000 7ffa53400700 1 -- 172.21.15.31:0/2457045993 shutdown_connections 2024-03-21T10:37:01.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.166+0000 7ffa53400700 1 --2- 172.21.15.31:0/2457045993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c102bd0 0x7ffa4c10f060 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:01.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.166+0000 7ffa53400700 1 --2- 172.21.15.31:0/2457045993 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa4c108260 0x7ffa4c108640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:01.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.166+0000 7ffa53400700 1 --2- 172.21.15.31:0/2457045993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa4c102210 0x7ffa4c102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:01.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.166+0000 7ffa53400700 1 -- 172.21.15.31:0/2457045993 >> 172.21.15.31:0/2457045993 conn(0x7ffa4c0792e0 msgr2=0x7ffa4c0796f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:01.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.166+0000 7ffa53400700 1 -- 172.21.15.31:0/2457045993 shutdown_connections 2024-03-21T10:37:01.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.166+0000 7ffa53400700 1 -- 172.21.15.31:0/2457045993 wait complete. 2024-03-21T10:37:01.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.167+0000 7ffa53400700 1 Processor -- start 2024-03-21T10:37:01.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.168+0000 7ffa53400700 1 -- start start 2024-03-21T10:37:01.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.168+0000 7ffa53400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa4c102210 0x7ffa4c074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:01.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.168+0000 7ffa53400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa4c102bd0 0x7ffa4c0749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:01.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.168+0000 7ffa53400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c108260 0x7ffa4c074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:01.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.168+0000 7ffa53400700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c1124f0 con 0x7ffa4c102210 2024-03-21T10:37:01.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.168+0000 7ffa53400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c112370 con 0x7ffa4c108260 2024-03-21T10:37:01.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.168+0000 7ffa53400700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c112670 con 0x7ffa4c102bd0 2024-03-21T10:37:01.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.168+0000 7ffa50a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa4c102210 0x7ffa4c074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:01.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.168+0000 7ffa4be00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa4c102bd0 0x7ffa4c0749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:01.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.168+0000 7ffa50a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa4c102210 0x7ffa4c074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58782/0 (socket says 172.21.15.31:58782) 2024-03-21T10:37:01.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.168+0000 7ffa50a00700 1 -- 172.21.15.31:0/4127387102 learned_addr learned my addr 172.21.15.31:0/4127387102 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:01.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.168+0000 7ffa51400700 1 --2- 172.21.15.31:0/4127387102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c108260 0x7ffa4c074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:01.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.169+0000 7ffa50a00700 1 -- 172.21.15.31:0/4127387102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa4c102bd0 msgr2=0x7ffa4c0749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:01.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.169+0000 7ffa50a00700 1 --2- 172.21.15.31:0/4127387102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa4c102bd0 0x7ffa4c0749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:01.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.169+0000 7ffa50a00700 1 -- 172.21.15.31:0/4127387102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c108260 msgr2=0x7ffa4c074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:01.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.169+0000 7ffa50a00700 1 --2- 172.21.15.31:0/4127387102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c108260 0x7ffa4c074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:01.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.169+0000 7ffa50a00700 1 -- 172.21.15.31:0/4127387102 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4c1a6e00 con 0x7ffa4c102210 2024-03-21T10:37:01.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.169+0000 7ffa4be00700 1 --2- 172.21.15.31:0/4127387102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa4c102bd0 0x7ffa4c0749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:37:01.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.169+0000 7ffa51400700 1 --2- 172.21.15.31:0/4127387102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c108260 0x7ffa4c074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:37:01.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.169+0000 7ffa50a00700 1 --2- 172.21.15.31:0/4127387102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa4c102210 0x7ffa4c074490 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ffa4000e9a0 tx=0x7ffa4000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:01.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.170+0000 7ffa49600700 1 -- 172.21.15.31:0/4127387102 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4000ca10 con 0x7ffa4c102210 2024-03-21T10:37:01.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.170+0000 7ffa53400700 1 -- 172.21.15.31:0/4127387102 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa4c1a70f0 con 0x7ffa4c102210 2024-03-21T10:37:01.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.170+0000 7ffa49600700 1 -- 172.21.15.31:0/4127387102 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa4000cbb0 con 0x7ffa4c102210 2024-03-21T10:37:01.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.170+0000 7ffa49600700 1 -- 172.21.15.31:0/4127387102 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa40018680 con 0x7ffa4c102210 2024-03-21T10:37:01.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.170+0000 7ffa53400700 1 -- 172.21.15.31:0/4127387102 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa4c1a7630 con 0x7ffa4c102210 2024-03-21T10:37:01.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.171+0000 7ffa53400700 1 -- 172.21.15.31:0/4127387102 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa4c068e50 con 0x7ffa4c102210 2024-03-21T10:37:01.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.174+0000 7ffa49600700 1 -- 172.21.15.31:0/4127387102 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7ffa40010430 con 0x7ffa4c102210 2024-03-21T10:37:01.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.175+0000 7ffa49600700 1 --2- 172.21.15.31:0/4127387102 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ffa340a2fb0 0x7ffa340a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:01.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.175+0000 7ffa49600700 1 -- 172.21.15.31:0/4127387102 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7ffa400cfe10 con 0x7ffa4c102210 2024-03-21T10:37:01.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.175+0000 7ffa4be00700 1 --2- 172.21.15.31:0/4127387102 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ffa340a2fb0 0x7ffa340a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:01.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.176+0000 7ffa4be00700 1 --2- 172.21.15.31:0/4127387102 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ffa340a2fb0 0x7ffa340a5470 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c0798e0 tx=0x7ffa38009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:01.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.179+0000 7ffa49600700 1 -- 172.21.15.31:0/4127387102 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ffa4009c880 con 0x7ffa4c102210 2024-03-21T10:37:01.388 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:01 smithi031 ceph-mon[31673]: pgmap v386: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 639 B/s wr, 6 op/s 2024-03-21T10:37:01.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.432+0000 7ffa53400700 1 -- 172.21.15.31:0/4127387102 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7ffa4c02da30 con 0x7ffa4c102210 2024-03-21T10:37:01.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.432+0000 7ffa49600700 1 -- 172.21.15.31:0/4127387102 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7ffa4009c180 con 0x7ffa4c102210 2024-03-21T10:37:01.430 INFO:teuthology.orchestra.run.smithi031.stdout:343597383718 2024-03-21T10:37:01.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.436+0000 7ffa53400700 1 -- 172.21.15.31:0/4127387102 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ffa340a2fb0 msgr2=0x7ffa340a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:01.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.436+0000 7ffa53400700 1 --2- 172.21.15.31:0/4127387102 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ffa340a2fb0 0x7ffa340a5470 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c0798e0 tx=0x7ffa38009260 comp rx=0 tx=0).stop 2024-03-21T10:37:01.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.436+0000 7ffa53400700 1 -- 172.21.15.31:0/4127387102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa4c102210 msgr2=0x7ffa4c074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:01.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.436+0000 7ffa53400700 1 --2- 172.21.15.31:0/4127387102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa4c102210 0x7ffa4c074490 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ffa4000e9a0 tx=0x7ffa4000ecb0 comp rx=0 tx=0).stop 2024-03-21T10:37:01.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.436+0000 7ffa53400700 1 -- 172.21.15.31:0/4127387102 shutdown_connections 2024-03-21T10:37:01.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.436+0000 7ffa53400700 1 --2- 172.21.15.31:0/4127387102 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ffa340a2fb0 0x7ffa340a5470 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:01.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.436+0000 7ffa53400700 1 --2- 172.21.15.31:0/4127387102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c108260 0x7ffa4c074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:01.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.436+0000 7ffa53400700 1 --2- 172.21.15.31:0/4127387102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa4c102210 0x7ffa4c074490 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:01.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.436+0000 7ffa53400700 1 --2- 172.21.15.31:0/4127387102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa4c102bd0 0x7ffa4c0749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:01.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.436+0000 7ffa53400700 1 -- 172.21.15.31:0/4127387102 >> 172.21.15.31:0/4127387102 conn(0x7ffa4c0792e0 msgr2=0x7ffa4c100f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:01.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.436+0000 7ffa53400700 1 -- 172.21.15.31:0/4127387102 shutdown_connections 2024-03-21T10:37:01.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:01.436+0000 7ffa53400700 1 -- 172.21.15.31:0/4127387102 wait complete. 2024-03-21T10:37:01.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:01 smithi040 ceph-mon[36154]: pgmap v386: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 639 B/s wr, 6 op/s 2024-03-21T10:37:01.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:01 smithi070 ceph-mon[36330]: pgmap v386: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 639 B/s wr, 6 op/s 2024-03-21T10:37:02.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:02 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/4127387102' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-21T10:37:02.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:02 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/4127387102' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-21T10:37:02.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:02 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/4127387102' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-21T10:37:03.544 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383712 got 343597383718 for osd.11 2024-03-21T10:37:03.544 DEBUG:teuthology.parallel:result is None 2024-03-21T10:37:03.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:03 smithi040 ceph-mon[36154]: pgmap v387: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 639 B/s wr, 8 op/s 2024-03-21T10:37:03.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:03 smithi031 ceph-mon[31673]: pgmap v387: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 639 B/s wr, 8 op/s 2024-03-21T10:37:03.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:03 smithi070 ceph-mon[36330]: pgmap v387: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 639 B/s wr, 8 op/s 2024-03-21T10:37:03.969 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574938 got 55834574943 for osd.1 2024-03-21T10:37:03.969 DEBUG:teuthology.parallel:result is None 2024-03-21T10:37:05.600 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:05 smithi031 ceph-mon[31673]: pgmap v388: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 468 B/s wr, 5 op/s 2024-03-21T10:37:05.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:05 smithi040 ceph-mon[36154]: pgmap v388: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 468 B/s wr, 5 op/s 2024-03-21T10:37:05.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:05 smithi070 ceph-mon[36330]: pgmap v388: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 468 B/s wr, 5 op/s 2024-03-21T10:37:05.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.886+0000 7fea35400700 1 -- 172.21.15.31:0/3779982602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea30069e40 msgr2=0x7fea3006a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:05.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.886+0000 7fea35400700 1 --2- 172.21.15.31:0/3779982602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea30069e40 0x7fea3006a2a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fea1c0099c0 tx=0x7fea1c009cd0 comp rx=0 tx=0).stop 2024-03-21T10:37:05.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.886+0000 7fea35400700 1 -- 172.21.15.31:0/3779982602 shutdown_connections 2024-03-21T10:37:05.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.886+0000 7fea35400700 1 --2- 172.21.15.31:0/3779982602 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea3006a7e0 0x7fea3010f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:05.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.886+0000 7fea35400700 1 --2- 172.21.15.31:0/3779982602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea301081e0 0x7fea301085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:05.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.886+0000 7fea35400700 1 --2- 172.21.15.31:0/3779982602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea30069e40 0x7fea3006a2a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:05.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.886+0000 7fea35400700 1 -- 172.21.15.31:0/3779982602 >> 172.21.15.31:0/3779982602 conn(0x7fea30077fc0 msgr2=0x7fea300783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:05.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.886+0000 7fea35400700 1 -- 172.21.15.31:0/3779982602 shutdown_connections 2024-03-21T10:37:05.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.886+0000 7fea35400700 1 -- 172.21.15.31:0/3779982602 wait complete. 2024-03-21T10:37:05.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.887+0000 7fea35400700 1 Processor -- start 2024-03-21T10:37:05.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.888+0000 7fea35400700 1 -- start start 2024-03-21T10:37:05.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.888+0000 7fea35400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea30069e40 0x7fea30101160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:05.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.888+0000 7fea35400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea3006a7e0 0x7fea301016a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:05.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.888+0000 7fea35400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea301081e0 0x7fea30101be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:05.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.888+0000 7fea35400700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea30112490 con 0x7fea301081e0 2024-03-21T10:37:05.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.888+0000 7fea35400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea30112310 con 0x7fea30069e40 2024-03-21T10:37:05.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.888+0000 7fea35400700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea30112610 con 0x7fea3006a7e0 2024-03-21T10:37:05.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.889+0000 7fea2e800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea30069e40 0x7fea30101160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:05.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.889+0000 7fea2e800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea30069e40 0x7fea30101160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33712/0 (socket says 172.21.15.31:33712) 2024-03-21T10:37:05.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.889+0000 7fea2e800700 1 -- 172.21.15.31:0/734642552 learned_addr learned my addr 172.21.15.31:0/734642552 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:05.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.889+0000 7fea2de00700 1 --2- 172.21.15.31:0/734642552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea3006a7e0 0x7fea301016a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:05.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.889+0000 7fea2f200700 1 --2- 172.21.15.31:0/734642552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea301081e0 0x7fea30101be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:05.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.889+0000 7fea2e800700 1 -- 172.21.15.31:0/734642552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea3006a7e0 msgr2=0x7fea301016a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:05.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.889+0000 7fea2e800700 1 --2- 172.21.15.31:0/734642552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea3006a7e0 0x7fea301016a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:05.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.889+0000 7fea2e800700 1 -- 172.21.15.31:0/734642552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea301081e0 msgr2=0x7fea30101be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:05.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.889+0000 7fea2e800700 1 --2- 172.21.15.31:0/734642552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea301081e0 0x7fea30101be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:05.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.889+0000 7fea2e800700 1 -- 172.21.15.31:0/734642552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea301a6e80 con 0x7fea30069e40 2024-03-21T10:37:05.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.889+0000 7fea2de00700 1 --2- 172.21.15.31:0/734642552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea3006a7e0 0x7fea301016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:37:05.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.890+0000 7fea2e800700 1 --2- 172.21.15.31:0/734642552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea30069e40 0x7fea30101160 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fea2000e9a0 tx=0x7fea2000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:05.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.890+0000 7fea27400700 1 -- 172.21.15.31:0/734642552 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2000cbd0 con 0x7fea30069e40 2024-03-21T10:37:05.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.890+0000 7fea35400700 1 -- 172.21.15.31:0/734642552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea301a7170 con 0x7fea30069e40 2024-03-21T10:37:05.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.890+0000 7fea27400700 1 -- 172.21.15.31:0/734642552 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fea2000cd70 con 0x7fea30069e40 2024-03-21T10:37:05.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.890+0000 7fea27400700 1 -- 172.21.15.31:0/734642552 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea20018750 con 0x7fea30069e40 2024-03-21T10:37:05.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.890+0000 7fea35400700 1 -- 172.21.15.31:0/734642552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea301a7680 con 0x7fea30069e40 2024-03-21T10:37:05.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.891+0000 7fea35400700 1 -- 172.21.15.31:0/734642552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea30068e50 con 0x7fea30069e40 2024-03-21T10:37:05.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.893+0000 7fea27400700 1 -- 172.21.15.31:0/734642552 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fea20010430 con 0x7fea30069e40 2024-03-21T10:37:05.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.895+0000 7fea27400700 1 --2- 172.21.15.31:0/734642552 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fea140a2fb0 0x7fea140a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:05.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.895+0000 7fea27400700 1 -- 172.21.15.31:0/734642552 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fea200cf9f0 con 0x7fea30069e40 2024-03-21T10:37:05.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.895+0000 7fea2de00700 1 --2- 172.21.15.31:0/734642552 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fea140a2fb0 0x7fea140a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:05.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.896+0000 7fea2de00700 1 --2- 172.21.15.31:0/734642552 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fea140a2fb0 0x7fea140a5470 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fea30102ba0 tx=0x7fea1c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:05.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:05.902+0000 7fea27400700 1 -- 172.21.15.31:0/734642552 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fea2009c460 con 0x7fea30069e40 2024-03-21T10:37:06.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.001+0000 7ffa86a00700 1 -- 172.21.15.31:0/637782101 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa801074f0 msgr2=0x7ffa8010f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:06.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.001+0000 7ffa86a00700 1 --2- 172.21.15.31:0/637782101 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa801074f0 0x7ffa8010f0c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ffa78009fc0 tx=0x7ffa7800b330 comp rx=0 tx=0).stop 2024-03-21T10:37:06.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.001+0000 7ffa86a00700 1 -- 172.21.15.31:0/637782101 shutdown_connections 2024-03-21T10:37:06.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.001+0000 7ffa86a00700 1 --2- 172.21.15.31:0/637782101 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa801074f0 0x7ffa8010f0c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:06.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.001+0000 7ffa86a00700 1 --2- 172.21.15.31:0/637782101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa80102410 0x7ffa801027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:06.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.001+0000 7ffa86a00700 1 --2- 172.21.15.31:0/637782101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa80102dc0 0x7ffa80106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:06.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.001+0000 7ffa86a00700 1 -- 172.21.15.31:0/637782101 >> 172.21.15.31:0/637782101 conn(0x7ffa800fdc60 msgr2=0x7ffa80100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:06.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.001+0000 7ffa86a00700 1 -- 172.21.15.31:0/637782101 shutdown_connections 2024-03-21T10:37:06.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.002+0000 7ffa86a00700 1 -- 172.21.15.31:0/637782101 wait complete. 2024-03-21T10:37:06.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.003+0000 7ffa86a00700 1 Processor -- start 2024-03-21T10:37:06.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.003+0000 7ffa86a00700 1 -- start start 2024-03-21T10:37:06.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.003+0000 7ffa86a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa80102410 0x7ffa801a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:06.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.004+0000 7ffa86a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa80102dc0 0x7ffa801a0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:06.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.004+0000 7ffa86a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa801074f0 0x7ffa8019a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:06.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.004+0000 7ffa86a00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa80112530 con 0x7ffa801074f0 2024-03-21T10:37:06.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.004+0000 7ffa86a00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa801123b0 con 0x7ffa80102dc0 2024-03-21T10:37:06.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.004+0000 7ffa86a00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa801126b0 con 0x7ffa80102410 2024-03-21T10:37:06.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.004+0000 7ffa7fe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa80102410 0x7ffa801a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:06.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.004+0000 7ffa7fe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa80102410 0x7ffa801a0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42172/0 (socket says 172.21.15.31:42172) 2024-03-21T10:37:06.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.004+0000 7ffa7fe00700 1 -- 172.21.15.31:0/1047825673 learned_addr learned my addr 172.21.15.31:0/1047825673 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:06.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.004+0000 7ffa84a00700 1 --2- 172.21.15.31:0/1047825673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa801074f0 0x7ffa8019a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:06.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.004+0000 7ffa7f400700 1 --2- 172.21.15.31:0/1047825673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa80102dc0 0x7ffa801a0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:06.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.004+0000 7ffa7fe00700 1 -- 172.21.15.31:0/1047825673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa80102dc0 msgr2=0x7ffa801a0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:06.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.004+0000 7ffa7fe00700 1 --2- 172.21.15.31:0/1047825673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa80102dc0 0x7ffa801a0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:06.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.004+0000 7ffa7fe00700 1 -- 172.21.15.31:0/1047825673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa801074f0 msgr2=0x7ffa8019a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:06.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.004+0000 7ffa7fe00700 1 --2- 172.21.15.31:0/1047825673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa801074f0 0x7ffa8019a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:06.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.004+0000 7ffa7fe00700 1 -- 172.21.15.31:0/1047825673 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa8019b090 con 0x7ffa80102410 2024-03-21T10:37:06.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.004+0000 7ffa84a00700 1 --2- 172.21.15.31:0/1047825673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa801074f0 0x7ffa8019a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:37:06.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.005+0000 7ffa7fe00700 1 --2- 172.21.15.31:0/1047825673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa80102410 0x7ffa801a0770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ffa7000e9a0 tx=0x7ffa7000ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:06.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.005+0000 7ffa7cc00700 1 -- 172.21.15.31:0/1047825673 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7000cbd0 con 0x7ffa80102410 2024-03-21T10:37:06.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.005+0000 7ffa7cc00700 1 -- 172.21.15.31:0/1047825673 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa7000cd70 con 0x7ffa80102410 2024-03-21T10:37:06.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.005+0000 7ffa7cc00700 1 -- 172.21.15.31:0/1047825673 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa70018710 con 0x7ffa80102410 2024-03-21T10:37:06.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.005+0000 7ffa86a00700 1 -- 172.21.15.31:0/1047825673 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa8019b380 con 0x7ffa80102410 2024-03-21T10:37:06.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.005+0000 7ffa86a00700 1 -- 172.21.15.31:0/1047825673 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa801a7640 con 0x7ffa80102410 2024-03-21T10:37:06.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.006+0000 7ffa86a00700 1 -- 172.21.15.31:0/1047825673 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa80068e50 con 0x7ffa80102410 2024-03-21T10:37:06.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.009+0000 7ffa7cc00700 1 -- 172.21.15.31:0/1047825673 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7ffa70010430 con 0x7ffa80102410 2024-03-21T10:37:06.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.011+0000 7ffa7cc00700 1 --2- 172.21.15.31:0/1047825673 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ffa640a30b0 0x7ffa640a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:06.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.011+0000 7ffa7cc00700 1 -- 172.21.15.31:0/1047825673 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7ffa700cfd30 con 0x7ffa80102410 2024-03-21T10:37:06.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.011+0000 7ffa7f400700 1 --2- 172.21.15.31:0/1047825673 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ffa640a30b0 0x7ffa640a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:06.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.012+0000 7ffa7cc00700 1 -- 172.21.15.31:0/1047825673 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ffa7009c7a0 con 0x7ffa80102410 2024-03-21T10:37:06.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.012+0000 7ffa7f400700 1 --2- 172.21.15.31:0/1047825673 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ffa640a30b0 0x7ffa640a5570 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ffa800fe2c0 tx=0x7ffa6c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:06.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.151+0000 7fea35400700 1 -- 172.21.15.31:0/734642552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fea3002da30 con 0x7fea30069e40 2024-03-21T10:37:06.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.151+0000 7fea27400700 1 -- 172.21.15.31:0/734642552 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fea2009bd60 con 0x7fea30069e40 2024-03-21T10:37:06.150 INFO:teuthology.orchestra.run.smithi031.stdout:219043332157 2024-03-21T10:37:06.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.156+0000 7fea35400700 1 -- 172.21.15.31:0/734642552 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fea140a2fb0 msgr2=0x7fea140a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:06.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.156+0000 7fea35400700 1 --2- 172.21.15.31:0/734642552 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fea140a2fb0 0x7fea140a5470 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fea30102ba0 tx=0x7fea1c005fb0 comp rx=0 tx=0).stop 2024-03-21T10:37:06.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.156+0000 7fea35400700 1 -- 172.21.15.31:0/734642552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea30069e40 msgr2=0x7fea30101160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:06.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.156+0000 7fea35400700 1 --2- 172.21.15.31:0/734642552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea30069e40 0x7fea30101160 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fea2000e9a0 tx=0x7fea2000ecb0 comp rx=0 tx=0).stop 2024-03-21T10:37:06.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.156+0000 7fea35400700 1 -- 172.21.15.31:0/734642552 shutdown_connections 2024-03-21T10:37:06.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.156+0000 7fea35400700 1 --2- 172.21.15.31:0/734642552 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fea140a2fb0 0x7fea140a5470 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:06.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.156+0000 7fea35400700 1 --2- 172.21.15.31:0/734642552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea301081e0 0x7fea30101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:06.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.156+0000 7fea35400700 1 --2- 172.21.15.31:0/734642552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea30069e40 0x7fea30101160 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:06.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.156+0000 7fea35400700 1 --2- 172.21.15.31:0/734642552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea3006a7e0 0x7fea301016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:06.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.156+0000 7fea35400700 1 -- 172.21.15.31:0/734642552 >> 172.21.15.31:0/734642552 conn(0x7fea30077fc0 msgr2=0x7fea300ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:06.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.158+0000 7fea35400700 1 -- 172.21.15.31:0/734642552 shutdown_connections 2024-03-21T10:37:06.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.158+0000 7fea35400700 1 -- 172.21.15.31:0/734642552 wait complete. 2024-03-21T10:37:06.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.331+0000 7ffa86a00700 1 -- 172.21.15.31:0/1047825673 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ffa801a78b0 con 0x7ffa80102410 2024-03-21T10:37:06.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.331+0000 7ffa7cc00700 1 -- 172.21.15.31:0/1047825673 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ffa7009c0a0 con 0x7ffa80102410 2024-03-21T10:37:06.329 INFO:teuthology.orchestra.run.smithi031.stdout:141733920845 2024-03-21T10:37:06.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.334+0000 7ffa86a00700 1 -- 172.21.15.31:0/1047825673 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ffa640a30b0 msgr2=0x7ffa640a5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:06.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.334+0000 7ffa86a00700 1 --2- 172.21.15.31:0/1047825673 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ffa640a30b0 0x7ffa640a5570 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ffa800fe2c0 tx=0x7ffa6c009310 comp rx=0 tx=0).stop 2024-03-21T10:37:06.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.334+0000 7ffa86a00700 1 -- 172.21.15.31:0/1047825673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa80102410 msgr2=0x7ffa801a0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:06.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.334+0000 7ffa86a00700 1 --2- 172.21.15.31:0/1047825673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa80102410 0x7ffa801a0770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ffa7000e9a0 tx=0x7ffa7000ed60 comp rx=0 tx=0).stop 2024-03-21T10:37:06.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.334+0000 7ffa86a00700 1 -- 172.21.15.31:0/1047825673 shutdown_connections 2024-03-21T10:37:06.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.334+0000 7ffa86a00700 1 --2- 172.21.15.31:0/1047825673 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ffa640a30b0 0x7ffa640a5570 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:06.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.334+0000 7ffa86a00700 1 --2- 172.21.15.31:0/1047825673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa801074f0 0x7ffa8019a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:06.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.334+0000 7ffa86a00700 1 --2- 172.21.15.31:0/1047825673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa80102410 0x7ffa801a0770 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:06.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.334+0000 7ffa86a00700 1 --2- 172.21.15.31:0/1047825673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa80102dc0 0x7ffa801a0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:06.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.334+0000 7ffa86a00700 1 -- 172.21.15.31:0/1047825673 >> 172.21.15.31:0/1047825673 conn(0x7ffa800fdc60 msgr2=0x7ffa8010d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:06.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.334+0000 7ffa86a00700 1 -- 172.21.15.31:0/1047825673 shutdown_connections 2024-03-21T10:37:06.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:06.334+0000 7ffa86a00700 1 -- 172.21.15.31:0/1047825673 wait complete. 2024-03-21T10:37:06.455 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:06 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/734642552' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-21T10:37:06.455 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:06 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1047825673' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-21T10:37:06.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:06 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/734642552' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-21T10:37:06.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:06 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1047825673' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-21T10:37:06.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:06 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/734642552' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-21T10:37:06.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:06 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1047825673' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-21T10:37:07.513 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332151 got 219043332157 for osd.7 2024-03-21T10:37:07.513 DEBUG:teuthology.parallel:result is None 2024-03-21T10:37:07.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:07 smithi040 ceph-mon[36154]: pgmap v389: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-21T10:37:07.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:07 smithi031 ceph-mon[31673]: pgmap v389: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-21T10:37:07.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:07 smithi070 ceph-mon[36330]: pgmap v389: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-21T10:37:07.983 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920838 got 141733920845 for osd.4 2024-03-21T10:37:07.983 DEBUG:teuthology.parallel:result is None 2024-03-21T10:37:07.983 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-21T10:37:07.984 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T10:37:08.464 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:37:09.625 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:09 smithi070 ceph-mon[36330]: pgmap v390: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-21T10:37:09.666 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:09 smithi031 ceph-mon[31673]: pgmap v390: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-21T10:37:09.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:09 smithi040 ceph-mon[36154]: pgmap v390: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-21T10:37:10.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.830+0000 7f6627e00700 1 -- 172.21.15.31:0/3441603336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6628103b60 msgr2=0x7f662810c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:10.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.830+0000 7f6627e00700 1 --2- 172.21.15.31:0/3441603336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6628103b60 0x7f662810c050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f661000b0a0 tx=0x7f661000b3b0 comp rx=0 tx=0).stop 2024-03-21T10:37:10.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.830+0000 7f6627e00700 1 -- 172.21.15.31:0/3441603336 shutdown_connections 2024-03-21T10:37:10.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.830+0000 7f6627e00700 1 --2- 172.21.15.31:0/3441603336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f662810c670 0x7f662810eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:10.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.830+0000 7f6627e00700 1 --2- 172.21.15.31:0/3441603336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6628103240 0x7f6628103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:10.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.830+0000 7f6627e00700 1 --2- 172.21.15.31:0/3441603336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6628103b60 0x7f662810c050 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:10.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.830+0000 7f6627e00700 1 -- 172.21.15.31:0/3441603336 >> 172.21.15.31:0/3441603336 conn(0x7f66280fca70 msgr2=0x7f66280fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:10.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.830+0000 7f6627e00700 1 -- 172.21.15.31:0/3441603336 shutdown_connections 2024-03-21T10:37:10.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.830+0000 7f6627e00700 1 -- 172.21.15.31:0/3441603336 wait complete. 2024-03-21T10:37:10.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.831+0000 7f6627e00700 1 Processor -- start 2024-03-21T10:37:10.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.831+0000 7f6627e00700 1 -- start start 2024-03-21T10:37:10.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.832+0000 7f6627e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6628103240 0x7f662819e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:10.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.832+0000 7f6627e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6628103b60 0x7f662819eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:10.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.832+0000 7f6627e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f662810c670 0x7f6628198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:10.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.832+0000 7f6627e00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66281123e0 con 0x7f6628103240 2024-03-21T10:37:10.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.832+0000 7f6627e00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6628112260 con 0x7f662810c670 2024-03-21T10:37:10.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.832+0000 7f6627e00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6628112560 con 0x7f6628103b60 2024-03-21T10:37:10.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.832+0000 7f6627400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f662810c670 0x7f6628198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:10.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.832+0000 7f6627400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f662810c670 0x7f6628198800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:54874/0 (socket says 172.21.15.31:54874) 2024-03-21T10:37:10.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.832+0000 7f6627400700 1 -- 172.21.15.31:0/497574901 learned_addr learned my addr 172.21.15.31:0/497574901 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:10.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.832+0000 7f661e000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6628103b60 0x7f662819eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:10.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.832+0000 7f6626a00700 1 --2- 172.21.15.31:0/497574901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6628103240 0x7f662819e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:10.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.833+0000 7f661e000700 1 -- 172.21.15.31:0/497574901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f662810c670 msgr2=0x7f6628198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:10.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.833+0000 7f661e000700 1 --2- 172.21.15.31:0/497574901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f662810c670 0x7f6628198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:10.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.833+0000 7f661e000700 1 -- 172.21.15.31:0/497574901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6628103240 msgr2=0x7f662819e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:10.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.833+0000 7f661e000700 1 --2- 172.21.15.31:0/497574901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6628103240 0x7f662819e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:10.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.833+0000 7f661e000700 1 -- 172.21.15.31:0/497574901 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6628198fe0 con 0x7f6628103b60 2024-03-21T10:37:10.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.833+0000 7f6627400700 1 --2- 172.21.15.31:0/497574901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f662810c670 0x7f6628198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:37:10.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.833+0000 7f661e000700 1 --2- 172.21.15.31:0/497574901 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6628103b60 0x7f662819eb70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f661000b070 tx=0x7f6610004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:10.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.833+0000 7f661fe00700 1 -- 172.21.15.31:0/497574901 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6610021070 con 0x7f6628103b60 2024-03-21T10:37:10.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.833+0000 7f661fe00700 1 -- 172.21.15.31:0/497574901 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6610007b80 con 0x7f6628103b60 2024-03-21T10:37:10.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.833+0000 7f6627e00700 1 -- 172.21.15.31:0/497574901 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6628199270 con 0x7f6628103b60 2024-03-21T10:37:10.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.833+0000 7f661fe00700 1 -- 172.21.15.31:0/497574901 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661001de60 con 0x7f6628103b60 2024-03-21T10:37:10.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.833+0000 7f6627e00700 1 -- 172.21.15.31:0/497574901 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66281a5490 con 0x7f6628103b60 2024-03-21T10:37:10.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.835+0000 7f6627e00700 1 -- 172.21.15.31:0/497574901 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f662806afb0 con 0x7f6628103b60 2024-03-21T10:37:10.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.836+0000 7f661fe00700 1 -- 172.21.15.31:0/497574901 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f6610024070 con 0x7f6628103b60 2024-03-21T10:37:10.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.838+0000 7f661fe00700 1 --2- 172.21.15.31:0/497574901 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f660c0a3030 0x7f660c0a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:10.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.838+0000 7f661fe00700 1 -- 172.21.15.31:0/497574901 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f661000f020 con 0x7f6628103b60 2024-03-21T10:37:10.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.838+0000 7f6626a00700 1 --2- 172.21.15.31:0/497574901 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f660c0a3030 0x7f660c0a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:10.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.839+0000 7f6626a00700 1 --2- 172.21.15.31:0/497574901 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f660c0a3030 0x7f660c0a54f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6618009bf0 tx=0x7f6618009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:10.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:10.841+0000 7f661fe00700 1 -- 172.21.15.31:0/497574901 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f661002b020 con 0x7f6628103b60 2024-03-21T10:37:11.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:11.064+0000 7f6627e00700 1 -- 172.21.15.31:0/497574901 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6628199ce0 con 0x7f660c0a3030 2024-03-21T10:37:11.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:11.080+0000 7f661fe00700 1 -- 172.21.15.31:0/497574901 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340782 (secure 0 0 0) 0x7f6628199ce0 con 0x7f660c0a3030 2024-03-21T10:37:11.086 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:37:11.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:11.087+0000 7f6627e00700 1 -- 172.21.15.31:0/497574901 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f660c0a3030 msgr2=0x7f660c0a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:11.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:11.087+0000 7f6627e00700 1 --2- 172.21.15.31:0/497574901 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f660c0a3030 0x7f660c0a54f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6618009bf0 tx=0x7f6618009260 comp rx=0 tx=0).stop 2024-03-21T10:37:11.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:11.087+0000 7f6627e00700 1 -- 172.21.15.31:0/497574901 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6628103b60 msgr2=0x7f662819eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:11.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:11.087+0000 7f6627e00700 1 --2- 172.21.15.31:0/497574901 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6628103b60 0x7f662819eb70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f661000b070 tx=0x7f6610004d00 comp rx=0 tx=0).stop 2024-03-21T10:37:11.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:11.087+0000 7f6627e00700 1 -- 172.21.15.31:0/497574901 shutdown_connections 2024-03-21T10:37:11.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:11.087+0000 7f6627e00700 1 --2- 172.21.15.31:0/497574901 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f660c0a3030 0x7f660c0a54f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:11.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:11.087+0000 7f6627e00700 1 --2- 172.21.15.31:0/497574901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f662810c670 0x7f6628198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:11.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:11.087+0000 7f6627e00700 1 --2- 172.21.15.31:0/497574901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6628103240 0x7f662819e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:11.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:11.087+0000 7f6627e00700 1 --2- 172.21.15.31:0/497574901 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6628103b60 0x7f662819eb70 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:11.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:11.087+0000 7f6627e00700 1 -- 172.21.15.31:0/497574901 >> 172.21.15.31:0/497574901 conn(0x7f66280fca70 msgr2=0x7f66280fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:11.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:11.087+0000 7f6627e00700 1 -- 172.21.15.31:0/497574901 shutdown_connections 2024-03-21T10:37:11.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:11.087+0000 7f6627e00700 1 -- 172.21.15.31:0/497574901 wait complete. 2024-03-21T10:37:11.089 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-03-21T10:37:11.365 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:11 smithi031 ceph-mon[31673]: pgmap v391: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:37:11.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:11 smithi040 ceph-mon[36154]: pgmap v391: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:37:11.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:11 smithi070 ceph-mon[36330]: pgmap v391: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:37:11.998 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":391,"stamp":"2024-03-21T10:37:10.046220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659199,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":681,"num_read_kb":465,"num_write":355,"num_write_kb":3063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":787,"ondisk_log_size":787,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369996,"kb_used_data":12440,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571828,"statfs":{"total":1151940427776,"available":1151561551872,"internally_reserved":0,"allocated":12738560,"data_stored":8186291,"data_compressed":433329,"data_compressed_allocated":589824,"data_compressed_original":2678784,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005242"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619622+0000","last_change":"2024-03-21T10:35:06.923544+0000","last_active":"2024-03-21T10:36:56.619622+0000","last_peered":"2024-03-21T10:36:56.619622+0000","last_clean":"2024-03-21T10:36:56.619622+0000","last_became_active":"2024-03-21T10:35:06.923254+0000","last_became_peered":"2024-03-21T10:35:06.923254+0000","last_unstale":"2024-03-21T10:36:56.619622+0000","last_undegraded":"2024-03-21T10:36:56.619622+0000","last_fullsized":"2024-03-21T10:36:56.619622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:38:14.573590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.622113+0000","last_change":"2024-03-21T10:35:06.929242+0000","last_active":"2024-03-21T10:36:56.622113+0000","last_peered":"2024-03-21T10:36:56.622113+0000","last_clean":"2024-03-21T10:36:56.622113+0000","last_became_active":"2024-03-21T10:35:06.927078+0000","last_became_peered":"2024-03-21T10:35:06.927078+0000","last_unstale":"2024-03-21T10:36:56.622113+0000","last_undegraded":"2024-03-21T10:36:56.622113+0000","last_fullsized":"2024-03-21T10:36:56.622113+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:27:10.347945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621754+0000","last_change":"2024-03-21T10:35:06.927792+0000","last_active":"2024-03-21T10:36:56.621754+0000","last_peered":"2024-03-21T10:36:56.621754+0000","last_clean":"2024-03-21T10:36:56.621754+0000","last_became_active":"2024-03-21T10:35:06.927386+0000","last_became_peered":"2024-03-21T10:35:06.927386+0000","last_unstale":"2024-03-21T10:36:56.621754+0000","last_undegraded":"2024-03-21T10:36:56.621754+0000","last_fullsized":"2024-03-21T10:36:56.621754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:53:51.310283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540833+0000","last_change":"2024-03-21T10:35:06.908901+0000","last_active":"2024-03-21T10:36:56.540833+0000","last_peered":"2024-03-21T10:36:56.540833+0000","last_clean":"2024-03-21T10:36:56.540833+0000","last_became_active":"2024-03-21T10:35:06.908586+0000","last_became_peered":"2024-03-21T10:35:06.908586+0000","last_unstale":"2024-03-21T10:36:56.540833+0000","last_undegraded":"2024-03-21T10:36:56.540833+0000","last_fullsized":"2024-03-21T10:36:56.540833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:25:57.387769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312668+0000","last_change":"2024-03-21T10:35:06.932537+0000","last_active":"2024-03-21T10:36:56.312668+0000","last_peered":"2024-03-21T10:36:56.312668+0000","last_clean":"2024-03-21T10:36:56.312668+0000","last_became_active":"2024-03-21T10:35:06.932185+0000","last_became_peered":"2024-03-21T10:35:06.932185+0000","last_unstale":"2024-03-21T10:36:56.312668+0000","last_undegraded":"2024-03-21T10:36:56.312668+0000","last_fullsized":"2024-03-21T10:36:56.312668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:30:25.127073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311395+0000","last_change":"2024-03-21T10:35:06.922765+0000","last_active":"2024-03-21T10:36:56.311395+0000","last_peered":"2024-03-21T10:36:56.311395+0000","last_clean":"2024-03-21T10:36:56.311395+0000","last_became_active":"2024-03-21T10:35:06.922447+0000","last_became_peered":"2024-03-21T10:35:06.922447+0000","last_unstale":"2024-03-21T10:36:56.311395+0000","last_undegraded":"2024-03-21T10:36:56.311395+0000","last_fullsized":"2024-03-21T10:36:56.311395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:37:33.310735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618972+0000","last_change":"2024-03-21T10:35:06.896777+0000","last_active":"2024-03-21T10:36:56.618972+0000","last_peered":"2024-03-21T10:36:56.618972+0000","last_clean":"2024-03-21T10:36:56.618972+0000","last_became_active":"2024-03-21T10:35:06.896544+0000","last_became_peered":"2024-03-21T10:35:06.896544+0000","last_unstale":"2024-03-21T10:36:56.618972+0000","last_undegraded":"2024-03-21T10:36:56.618972+0000","last_fullsized":"2024-03-21T10:36:56.618972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:34:05.000661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.324782+0000","last_change":"2024-03-21T10:35:06.923294+0000","last_active":"2024-03-21T10:36:56.324782+0000","last_peered":"2024-03-21T10:36:56.324782+0000","last_clean":"2024-03-21T10:36:56.324782+0000","last_became_active":"2024-03-21T10:35:06.921320+0000","last_became_peered":"2024-03-21T10:35:06.921320+0000","last_unstale":"2024-03-21T10:36:56.324782+0000","last_undegraded":"2024-03-21T10:36:56.324782+0000","last_fullsized":"2024-03-21T10:36:56.324782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:33:53.855908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619791+0000","last_change":"2024-03-21T10:35:06.912977+0000","last_active":"2024-03-21T10:36:56.619791+0000","last_peered":"2024-03-21T10:36:56.619791+0000","last_clean":"2024-03-21T10:36:56.619791+0000","last_became_active":"2024-03-21T10:35:06.912626+0000","last_became_peered":"2024-03-21T10:35:06.912626+0000","last_unstale":"2024-03-21T10:36:56.619791+0000","last_undegraded":"2024-03-21T10:36:56.619791+0000","last_fullsized":"2024-03-21T10:36:56.619791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:42:13.472454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311863+0000","last_change":"2024-03-21T10:35:06.910252+0000","last_active":"2024-03-21T10:36:56.311863+0000","last_peered":"2024-03-21T10:36:56.311863+0000","last_clean":"2024-03-21T10:36:56.311863+0000","last_became_active":"2024-03-21T10:35:06.909924+0000","last_became_peered":"2024-03-21T10:35:06.909924+0000","last_unstale":"2024-03-21T10:36:56.311863+0000","last_undegraded":"2024-03-21T10:36:56.311863+0000","last_fullsized":"2024-03-21T10:36:56.311863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:43:15.527349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619675+0000","last_change":"2024-03-21T10:35:06.909569+0000","last_active":"2024-03-21T10:36:56.619675+0000","last_peered":"2024-03-21T10:36:56.619675+0000","last_clean":"2024-03-21T10:36:56.619675+0000","last_became_active":"2024-03-21T10:35:06.908982+0000","last_became_peered":"2024-03-21T10:35:06.908982+0000","last_unstale":"2024-03-21T10:36:56.619675+0000","last_undegraded":"2024-03-21T10:36:56.619675+0000","last_fullsized":"2024-03-21T10:36:56.619675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:13:56.020281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540766+0000","last_change":"2024-03-21T10:35:06.913861+0000","last_active":"2024-03-21T10:36:56.540766+0000","last_peered":"2024-03-21T10:36:56.540766+0000","last_clean":"2024-03-21T10:36:56.540766+0000","last_became_active":"2024-03-21T10:35:06.913490+0000","last_became_peered":"2024-03-21T10:35:06.913490+0000","last_unstale":"2024-03-21T10:36:56.540766+0000","last_undegraded":"2024-03-21T10:36:56.540766+0000","last_fullsized":"2024-03-21T10:36:56.540766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:24:18.573735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312522+0000","last_change":"2024-03-21T10:35:06.934537+0000","last_active":"2024-03-21T10:36:56.312522+0000","last_peered":"2024-03-21T10:36:56.312522+0000","last_clean":"2024-03-21T10:36:56.312522+0000","last_became_active":"2024-03-21T10:35:06.932951+0000","last_became_peered":"2024-03-21T10:35:06.932951+0000","last_unstale":"2024-03-21T10:36:56.312522+0000","last_undegraded":"2024-03-21T10:36:56.312522+0000","last_fullsized":"2024-03-21T10:36:56.312522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:43:17.748434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621126+0000","last_change":"2024-03-21T10:35:06.925495+0000","last_active":"2024-03-21T10:36:56.621126+0000","last_peered":"2024-03-21T10:36:56.621126+0000","last_clean":"2024-03-21T10:36:56.621126+0000","last_became_active":"2024-03-21T10:35:06.924326+0000","last_became_peered":"2024-03-21T10:35:06.924326+0000","last_unstale":"2024-03-21T10:36:56.621126+0000","last_undegraded":"2024-03-21T10:36:56.621126+0000","last_fullsized":"2024-03-21T10:36:56.621126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:27:19.598213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.622171+0000","last_change":"2024-03-21T10:35:06.923695+0000","last_active":"2024-03-21T10:36:56.622171+0000","last_peered":"2024-03-21T10:36:56.622171+0000","last_clean":"2024-03-21T10:36:56.622171+0000","last_became_active":"2024-03-21T10:35:06.919114+0000","last_became_peered":"2024-03-21T10:35:06.919114+0000","last_unstale":"2024-03-21T10:36:56.622171+0000","last_undegraded":"2024-03-21T10:36:56.622171+0000","last_fullsized":"2024-03-21T10:36:56.622171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:42:49.179653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619077+0000","last_change":"2024-03-21T10:35:06.930225+0000","last_active":"2024-03-21T10:36:56.619077+0000","last_peered":"2024-03-21T10:36:56.619077+0000","last_clean":"2024-03-21T10:36:56.619077+0000","last_became_active":"2024-03-21T10:35:06.929905+0000","last_became_peered":"2024-03-21T10:35:06.929905+0000","last_unstale":"2024-03-21T10:36:56.619077+0000","last_undegraded":"2024-03-21T10:36:56.619077+0000","last_fullsized":"2024-03-21T10:36:56.619077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:21:07.231703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619469+0000","last_change":"2024-03-21T10:35:06.923498+0000","last_active":"2024-03-21T10:36:56.619469+0000","last_peered":"2024-03-21T10:36:56.619469+0000","last_clean":"2024-03-21T10:36:56.619469+0000","last_became_active":"2024-03-21T10:35:06.923242+0000","last_became_peered":"2024-03-21T10:35:06.923242+0000","last_unstale":"2024-03-21T10:36:56.619469+0000","last_undegraded":"2024-03-21T10:36:56.619469+0000","last_fullsized":"2024-03-21T10:36:56.619469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:42:04.309701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623751+0000","last_change":"2024-03-21T10:35:06.918314+0000","last_active":"2024-03-21T10:36:56.623751+0000","last_peered":"2024-03-21T10:36:56.623751+0000","last_clean":"2024-03-21T10:36:56.623751+0000","last_became_active":"2024-03-21T10:35:06.917865+0000","last_became_peered":"2024-03-21T10:35:06.917865+0000","last_unstale":"2024-03-21T10:36:56.623751+0000","last_undegraded":"2024-03-21T10:36:56.623751+0000","last_fullsized":"2024-03-21T10:36:56.623751+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:53:48.536341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619252+0000","last_change":"2024-03-21T10:35:06.936234+0000","last_active":"2024-03-21T10:36:56.619252+0000","last_peered":"2024-03-21T10:36:56.619252+0000","last_clean":"2024-03-21T10:36:56.619252+0000","last_became_active":"2024-03-21T10:35:06.935958+0000","last_became_peered":"2024-03-21T10:35:06.935958+0000","last_unstale":"2024-03-21T10:36:56.619252+0000","last_undegraded":"2024-03-21T10:36:56.619252+0000","last_fullsized":"2024-03-21T10:36:56.619252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:52:43.509671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618568+0000","last_change":"2024-03-21T10:35:11.198276+0000","last_active":"2024-03-21T10:36:56.618568+0000","last_peered":"2024-03-21T10:36:56.618568+0000","last_clean":"2024-03-21T10:36:56.618568+0000","last_became_active":"2024-03-21T10:35:11.197899+0000","last_became_peered":"2024-03-21T10:35:11.197899+0000","last_unstale":"2024-03-21T10:36:56.618568+0000","last_undegraded":"2024-03-21T10:36:56.618568+0000","last_fullsized":"2024-03-21T10:36:56.618568+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:17:30.499932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623436+0000","last_change":"2024-03-21T10:35:06.010647+0000","last_active":"2024-03-21T10:36:56.623436+0000","last_peered":"2024-03-21T10:36:56.623436+0000","last_clean":"2024-03-21T10:36:56.623436+0000","last_became_active":"2024-03-21T10:35:06.010239+0000","last_became_peered":"2024-03-21T10:35:06.010239+0000","last_unstale":"2024-03-21T10:36:56.623436+0000","last_undegraded":"2024-03-21T10:36:56.623436+0000","last_fullsized":"2024-03-21T10:36:56.623436+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:35:53.324044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325857+0000","last_change":"2024-03-21T10:35:06.926398+0000","last_active":"2024-03-21T10:36:56.325857+0000","last_peered":"2024-03-21T10:36:56.325857+0000","last_clean":"2024-03-21T10:36:56.325857+0000","last_became_active":"2024-03-21T10:35:06.925904+0000","last_became_peered":"2024-03-21T10:35:06.925904+0000","last_unstale":"2024-03-21T10:36:56.325857+0000","last_undegraded":"2024-03-21T10:36:56.325857+0000","last_fullsized":"2024-03-21T10:36:56.325857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:37:52.811604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540440+0000","last_change":"2024-03-21T10:35:55.503266+0000","last_active":"2024-03-21T10:36:56.540440+0000","last_peered":"2024-03-21T10:36:56.540440+0000","last_clean":"2024-03-21T10:36:56.540440+0000","last_became_active":"2024-03-21T10:35:55.502694+0000","last_became_peered":"2024-03-21T10:35:55.502694+0000","last_unstale":"2024-03-21T10:36:56.540440+0000","last_undegraded":"2024-03-21T10:36:56.540440+0000","last_fullsized":"2024-03-21T10:36:56.540440+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:56:44.480667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325780+0000","last_change":"2024-03-21T10:35:06.022750+0000","last_active":"2024-03-21T10:36:56.325780+0000","last_peered":"2024-03-21T10:36:56.325780+0000","last_clean":"2024-03-21T10:36:56.325780+0000","last_became_active":"2024-03-21T10:35:06.022093+0000","last_became_peered":"2024-03-21T10:35:06.022093+0000","last_unstale":"2024-03-21T10:36:56.325780+0000","last_undegraded":"2024-03-21T10:36:56.325780+0000","last_fullsized":"2024-03-21T10:36:56.325780+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:53:28.458486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.624041+0000","last_change":"2024-03-21T10:35:06.927594+0000","last_active":"2024-03-21T10:36:56.624041+0000","last_peered":"2024-03-21T10:36:56.624041+0000","last_clean":"2024-03-21T10:36:56.624041+0000","last_became_active":"2024-03-21T10:35:06.926868+0000","last_became_peered":"2024-03-21T10:35:06.926868+0000","last_unstale":"2024-03-21T10:36:56.624041+0000","last_undegraded":"2024-03-21T10:36:56.624041+0000","last_fullsized":"2024-03-21T10:36:56.624041+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:07:33.127465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540535+0000","last_change":"2024-03-21T10:35:11.193475+0000","last_active":"2024-03-21T10:36:56.540535+0000","last_peered":"2024-03-21T10:36:56.540535+0000","last_clean":"2024-03-21T10:36:56.540535+0000","last_became_active":"2024-03-21T10:35:11.193054+0000","last_became_peered":"2024-03-21T10:35:11.193054+0000","last_unstale":"2024-03-21T10:36:56.540535+0000","last_undegraded":"2024-03-21T10:36:56.540535+0000","last_fullsized":"2024-03-21T10:36:56.540535+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:32:36.564147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313082+0000","last_change":"2024-03-21T10:35:06.004257+0000","last_active":"2024-03-21T10:36:56.313082+0000","last_peered":"2024-03-21T10:36:56.313082+0000","last_clean":"2024-03-21T10:36:56.313082+0000","last_became_active":"2024-03-21T10:35:06.003936+0000","last_became_peered":"2024-03-21T10:35:06.003936+0000","last_unstale":"2024-03-21T10:36:56.313082+0000","last_undegraded":"2024-03-21T10:36:56.313082+0000","last_fullsized":"2024-03-21T10:36:56.313082+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:48:48.608979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.620100+0000","last_change":"2024-03-21T10:35:06.941681+0000","last_active":"2024-03-21T10:36:56.620100+0000","last_peered":"2024-03-21T10:36:56.620100+0000","last_clean":"2024-03-21T10:36:56.620100+0000","last_became_active":"2024-03-21T10:35:06.941314+0000","last_became_peered":"2024-03-21T10:35:06.941314+0000","last_unstale":"2024-03-21T10:36:56.620100+0000","last_undegraded":"2024-03-21T10:36:56.620100+0000","last_fullsized":"2024-03-21T10:36:56.620100+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:46:09.948922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540433+0000","last_change":"2024-03-21T10:35:11.188472+0000","last_active":"2024-03-21T10:36:56.540433+0000","last_peered":"2024-03-21T10:36:56.540433+0000","last_clean":"2024-03-21T10:36:56.540433+0000","last_became_active":"2024-03-21T10:35:11.188072+0000","last_became_peered":"2024-03-21T10:35:11.188072+0000","last_unstale":"2024-03-21T10:36:56.540433+0000","last_undegraded":"2024-03-21T10:36:56.540433+0000","last_fullsized":"2024-03-21T10:36:56.540433+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:04:16.911334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312976+0000","last_change":"2024-03-21T10:35:05.998493+0000","last_active":"2024-03-21T10:36:56.312976+0000","last_peered":"2024-03-21T10:36:56.312976+0000","last_clean":"2024-03-21T10:36:56.312976+0000","last_became_active":"2024-03-21T10:35:05.998134+0000","last_became_peered":"2024-03-21T10:35:05.998134+0000","last_unstale":"2024-03-21T10:36:56.312976+0000","last_undegraded":"2024-03-21T10:36:56.312976+0000","last_fullsized":"2024-03-21T10:36:56.312976+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:05:32.033328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311421+0000","last_change":"2024-03-21T10:35:06.912973+0000","last_active":"2024-03-21T10:36:56.311421+0000","last_peered":"2024-03-21T10:36:56.311421+0000","last_clean":"2024-03-21T10:36:56.311421+0000","last_became_active":"2024-03-21T10:35:06.912382+0000","last_became_peered":"2024-03-21T10:35:06.912382+0000","last_unstale":"2024-03-21T10:36:56.311421+0000","last_undegraded":"2024-03-21T10:36:56.311421+0000","last_fullsized":"2024-03-21T10:36:56.311421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:13:23.757805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619093+0000","last_change":"2024-03-21T10:35:11.180806+0000","last_active":"2024-03-21T10:36:56.619093+0000","last_peered":"2024-03-21T10:36:56.619093+0000","last_clean":"2024-03-21T10:36:56.619093+0000","last_became_active":"2024-03-21T10:35:11.180303+0000","last_became_peered":"2024-03-21T10:35:11.180303+0000","last_unstale":"2024-03-21T10:36:56.619093+0000","last_undegraded":"2024-03-21T10:36:56.619093+0000","last_fullsized":"2024-03-21T10:36:56.619093+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:27:11.719654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":130,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:37:08.092912+0000","last_change":"2024-03-21T10:35:06.004555+0000","last_active":"2024-03-21T10:37:08.092912+0000","last_peered":"2024-03-21T10:37:08.092912+0000","last_clean":"2024-03-21T10:37:08.092912+0000","last_became_active":"2024-03-21T10:35:06.003926+0000","last_became_peered":"2024-03-21T10:35:06.003926+0000","last_unstale":"2024-03-21T10:37:08.092912+0000","last_undegraded":"2024-03-21T10:37:08.092912+0000","last_fullsized":"2024-03-21T10:37:08.092912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:05:49.721000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":93,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312684+0000","last_change":"2024-03-21T10:35:06.913556+0000","last_active":"2024-03-21T10:36:56.312684+0000","last_peered":"2024-03-21T10:36:56.312684+0000","last_clean":"2024-03-21T10:36:56.312684+0000","last_became_active":"2024-03-21T10:35:06.913213+0000","last_became_peered":"2024-03-21T10:35:06.913213+0000","last_unstale":"2024-03-21T10:36:56.312684+0000","last_undegraded":"2024-03-21T10:36:56.312684+0000","last_fullsized":"2024-03-21T10:36:56.312684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:11:09.235927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619326+0000","last_change":"2024-03-21T10:35:55.523819+0000","last_active":"2024-03-21T10:36:56.619326+0000","last_peered":"2024-03-21T10:36:56.619326+0000","last_clean":"2024-03-21T10:36:56.619326+0000","last_became_active":"2024-03-21T10:35:55.523333+0000","last_became_peered":"2024-03-21T10:35:55.523333+0000","last_unstale":"2024-03-21T10:36:56.619326+0000","last_undegraded":"2024-03-21T10:36:56.619326+0000","last_fullsized":"2024-03-21T10:36:56.619326+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:25:36.277481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311293+0000","last_change":"2024-03-21T10:35:06.012527+0000","last_active":"2024-03-21T10:36:56.311293+0000","last_peered":"2024-03-21T10:36:56.311293+0000","last_clean":"2024-03-21T10:36:56.311293+0000","last_became_active":"2024-03-21T10:35:06.012228+0000","last_became_peered":"2024-03-21T10:35:06.012228+0000","last_unstale":"2024-03-21T10:36:56.311293+0000","last_undegraded":"2024-03-21T10:36:56.311293+0000","last_fullsized":"2024-03-21T10:36:56.311293+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:52:34.622306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.622381+0000","last_change":"2024-03-21T10:35:55.503531+0000","last_active":"2024-03-21T10:36:56.622381+0000","last_peered":"2024-03-21T10:36:56.622381+0000","last_clean":"2024-03-21T10:36:56.622381+0000","last_became_active":"2024-03-21T10:35:55.502793+0000","last_became_peered":"2024-03-21T10:35:55.502793+0000","last_unstale":"2024-03-21T10:36:56.622381+0000","last_undegraded":"2024-03-21T10:36:56.622381+0000","last_fullsized":"2024-03-21T10:36:56.622381+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:08:39.599084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,3],"acting":[9,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619462+0000","last_change":"2024-03-21T10:35:11.196543+0000","last_active":"2024-03-21T10:36:56.619462+0000","last_peered":"2024-03-21T10:36:56.619462+0000","last_clean":"2024-03-21T10:36:56.619462+0000","last_became_active":"2024-03-21T10:35:11.196151+0000","last_became_peered":"2024-03-21T10:35:11.196151+0000","last_unstale":"2024-03-21T10:36:56.619462+0000","last_undegraded":"2024-03-21T10:36:56.619462+0000","last_fullsized":"2024-03-21T10:36:56.619462+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:15:39.193647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312140+0000","last_change":"2024-03-21T10:35:06.004588+0000","last_active":"2024-03-21T10:36:56.312140+0000","last_peered":"2024-03-21T10:36:56.312140+0000","last_clean":"2024-03-21T10:36:56.312140+0000","last_became_active":"2024-03-21T10:35:06.004190+0000","last_became_peered":"2024-03-21T10:35:06.004190+0000","last_unstale":"2024-03-21T10:36:56.312140+0000","last_undegraded":"2024-03-21T10:36:56.312140+0000","last_fullsized":"2024-03-21T10:36:56.312140+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:32:18.563692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325018+0000","last_change":"2024-03-21T10:35:06.941738+0000","last_active":"2024-03-21T10:36:56.325018+0000","last_peered":"2024-03-21T10:36:56.325018+0000","last_clean":"2024-03-21T10:36:56.325018+0000","last_became_active":"2024-03-21T10:35:06.941386+0000","last_became_peered":"2024-03-21T10:35:06.941386+0000","last_unstale":"2024-03-21T10:36:56.325018+0000","last_undegraded":"2024-03-21T10:36:56.325018+0000","last_fullsized":"2024-03-21T10:36:56.325018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:52:55.519095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621671+0000","last_change":"2024-03-21T10:35:11.199370+0000","last_active":"2024-03-21T10:36:56.621671+0000","last_peered":"2024-03-21T10:36:56.621671+0000","last_clean":"2024-03-21T10:36:56.621671+0000","last_became_active":"2024-03-21T10:35:11.198598+0000","last_became_peered":"2024-03-21T10:35:11.198598+0000","last_unstale":"2024-03-21T10:36:56.621671+0000","last_undegraded":"2024-03-21T10:36:56.621671+0000","last_fullsized":"2024-03-21T10:36:56.621671+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:07:19.419405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311161+0000","last_change":"2024-03-21T10:35:06.004893+0000","last_active":"2024-03-21T10:36:56.311161+0000","last_peered":"2024-03-21T10:36:56.311161+0000","last_clean":"2024-03-21T10:36:56.311161+0000","last_became_active":"2024-03-21T10:35:06.004586+0000","last_became_peered":"2024-03-21T10:35:06.004586+0000","last_unstale":"2024-03-21T10:36:56.311161+0000","last_undegraded":"2024-03-21T10:36:56.311161+0000","last_fullsized":"2024-03-21T10:36:56.311161+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:05:29.106575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621244+0000","last_change":"2024-03-21T10:35:06.924448+0000","last_active":"2024-03-21T10:36:56.621244+0000","last_peered":"2024-03-21T10:36:56.621244+0000","last_clean":"2024-03-21T10:36:56.621244+0000","last_became_active":"2024-03-21T10:35:06.924049+0000","last_became_peered":"2024-03-21T10:35:06.924049+0000","last_unstale":"2024-03-21T10:36:56.621244+0000","last_undegraded":"2024-03-21T10:36:56.621244+0000","last_fullsized":"2024-03-21T10:36:56.621244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:23:12.070723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.324956+0000","last_change":"2024-03-21T10:35:11.188691+0000","last_active":"2024-03-21T10:36:56.324956+0000","last_peered":"2024-03-21T10:36:56.324956+0000","last_clean":"2024-03-21T10:36:56.324956+0000","last_became_active":"2024-03-21T10:35:11.188138+0000","last_became_peered":"2024-03-21T10:35:11.188138+0000","last_unstale":"2024-03-21T10:36:56.324956+0000","last_undegraded":"2024-03-21T10:36:56.324956+0000","last_fullsized":"2024-03-21T10:36:56.324956+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:51:50.920920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":120,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:37:04.088825+0000","last_change":"2024-03-21T10:35:06.015276+0000","last_active":"2024-03-21T10:37:04.088825+0000","last_peered":"2024-03-21T10:37:04.088825+0000","last_clean":"2024-03-21T10:37:04.088825+0000","last_became_active":"2024-03-21T10:35:06.014734+0000","last_became_peered":"2024-03-21T10:35:06.014734+0000","last_unstale":"2024-03-21T10:37:04.088825+0000","last_undegraded":"2024-03-21T10:37:04.088825+0000","last_fullsized":"2024-03-21T10:37:04.088825+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:18:45.905451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":85,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.620294+0000","last_change":"2024-03-21T10:35:06.926689+0000","last_active":"2024-03-21T10:36:56.620294+0000","last_peered":"2024-03-21T10:36:56.620294+0000","last_clean":"2024-03-21T10:36:56.620294+0000","last_became_active":"2024-03-21T10:35:06.926319+0000","last_became_peered":"2024-03-21T10:35:06.926319+0000","last_unstale":"2024-03-21T10:36:56.620294+0000","last_undegraded":"2024-03-21T10:36:56.620294+0000","last_fullsized":"2024-03-21T10:36:56.620294+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:06:22.371608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311047+0000","last_change":"2024-03-21T10:35:55.383776+0000","last_active":"2024-03-21T10:36:56.311047+0000","last_peered":"2024-03-21T10:36:56.311047+0000","last_clean":"2024-03-21T10:36:56.311047+0000","last_became_active":"2024-03-21T10:35:55.383300+0000","last_became_peered":"2024-03-21T10:35:55.383300+0000","last_unstale":"2024-03-21T10:36:56.311047+0000","last_undegraded":"2024-03-21T10:36:56.311047+0000","last_fullsized":"2024-03-21T10:36:56.311047+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:55:56.244325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2,4],"acting":[10,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619425+0000","last_change":"2024-03-21T10:35:06.007231+0000","last_active":"2024-03-21T10:36:56.619425+0000","last_peered":"2024-03-21T10:36:56.619425+0000","last_clean":"2024-03-21T10:36:56.619425+0000","last_became_active":"2024-03-21T10:35:06.006875+0000","last_became_peered":"2024-03-21T10:35:06.006875+0000","last_unstale":"2024-03-21T10:36:56.619425+0000","last_undegraded":"2024-03-21T10:36:56.619425+0000","last_fullsized":"2024-03-21T10:36:56.619425+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:08:02.489741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.624205+0000","last_change":"2024-03-21T10:35:55.524107+0000","last_active":"2024-03-21T10:36:56.624205+0000","last_peered":"2024-03-21T10:36:56.624205+0000","last_clean":"2024-03-21T10:36:56.624205+0000","last_became_active":"2024-03-21T10:35:55.523523+0000","last_became_peered":"2024-03-21T10:35:55.523523+0000","last_unstale":"2024-03-21T10:36:56.624205+0000","last_undegraded":"2024-03-21T10:36:56.624205+0000","last_fullsized":"2024-03-21T10:36:56.624205+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:49:29.121279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540676+0000","last_change":"2024-03-21T10:35:11.193382+0000","last_active":"2024-03-21T10:36:56.540676+0000","last_peered":"2024-03-21T10:36:56.540676+0000","last_clean":"2024-03-21T10:36:56.540676+0000","last_became_active":"2024-03-21T10:35:11.192959+0000","last_became_peered":"2024-03-21T10:35:11.192959+0000","last_unstale":"2024-03-21T10:36:56.540676+0000","last_undegraded":"2024-03-21T10:36:56.540676+0000","last_fullsized":"2024-03-21T10:36:56.540676+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:02:57.187370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311322+0000","last_change":"2024-03-21T10:35:06.019969+0000","last_active":"2024-03-21T10:36:56.311322+0000","last_peered":"2024-03-21T10:36:56.311322+0000","last_clean":"2024-03-21T10:36:56.311322+0000","last_became_active":"2024-03-21T10:35:06.019648+0000","last_became_peered":"2024-03-21T10:35:06.019648+0000","last_unstale":"2024-03-21T10:36:56.311322+0000","last_undegraded":"2024-03-21T10:36:56.311322+0000","last_fullsized":"2024-03-21T10:36:56.311322+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:24:02.227205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.620236+0000","last_change":"2024-03-21T10:35:06.910294+0000","last_active":"2024-03-21T10:36:56.620236+0000","last_peered":"2024-03-21T10:36:56.620236+0000","last_clean":"2024-03-21T10:36:56.620236+0000","last_became_active":"2024-03-21T10:35:06.909862+0000","last_became_peered":"2024-03-21T10:35:06.909862+0000","last_unstale":"2024-03-21T10:36:56.620236+0000","last_undegraded":"2024-03-21T10:36:56.620236+0000","last_fullsized":"2024-03-21T10:36:56.620236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:37:13.396003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621461+0000","last_change":"2024-03-21T10:35:11.196398+0000","last_active":"2024-03-21T10:36:56.621461+0000","last_peered":"2024-03-21T10:36:56.621461+0000","last_clean":"2024-03-21T10:36:56.621461+0000","last_became_active":"2024-03-21T10:35:11.194965+0000","last_became_peered":"2024-03-21T10:35:11.194965+0000","last_unstale":"2024-03-21T10:36:56.621461+0000","last_undegraded":"2024-03-21T10:36:56.621461+0000","last_fullsized":"2024-03-21T10:36:56.621461+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:48:22.024137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'36","reported_seq":247,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:37:05.089442+0000","last_change":"2024-03-21T10:35:06.005037+0000","last_active":"2024-03-21T10:37:05.089442+0000","last_peered":"2024-03-21T10:37:05.089442+0000","last_clean":"2024-03-21T10:37:05.089442+0000","last_became_active":"2024-03-21T10:35:06.004686+0000","last_became_peered":"2024-03-21T10:35:06.004686+0000","last_unstale":"2024-03-21T10:37:05.089442+0000","last_undegraded":"2024-03-21T10:37:05.089442+0000","last_fullsized":"2024-03-21T10:37:05.089442+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:24:13.312112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64455,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":180,"num_read_kb":154,"num_write":37,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312911+0000","last_change":"2024-03-21T10:35:06.906525+0000","last_active":"2024-03-21T10:36:56.312911+0000","last_peered":"2024-03-21T10:36:56.312911+0000","last_clean":"2024-03-21T10:36:56.312911+0000","last_became_active":"2024-03-21T10:35:06.905150+0000","last_became_peered":"2024-03-21T10:35:06.905150+0000","last_unstale":"2024-03-21T10:36:56.312911+0000","last_undegraded":"2024-03-21T10:36:56.312911+0000","last_fullsized":"2024-03-21T10:36:56.312911+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:46:27.958779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311678+0000","last_change":"2024-03-21T10:35:11.193519+0000","last_active":"2024-03-21T10:36:56.311678+0000","last_peered":"2024-03-21T10:36:56.311678+0000","last_clean":"2024-03-21T10:36:56.311678+0000","last_became_active":"2024-03-21T10:35:11.193043+0000","last_became_peered":"2024-03-21T10:35:11.193043+0000","last_unstale":"2024-03-21T10:36:56.311678+0000","last_undegraded":"2024-03-21T10:36:56.311678+0000","last_fullsized":"2024-03-21T10:36:56.311678+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:23:20.984472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618921+0000","last_change":"2024-03-21T10:35:05.999220+0000","last_active":"2024-03-21T10:36:56.618921+0000","last_peered":"2024-03-21T10:36:56.618921+0000","last_clean":"2024-03-21T10:36:56.618921+0000","last_became_active":"2024-03-21T10:35:05.998918+0000","last_became_peered":"2024-03-21T10:35:05.998918+0000","last_unstale":"2024-03-21T10:36:56.618921+0000","last_undegraded":"2024-03-21T10:36:56.618921+0000","last_fullsized":"2024-03-21T10:36:56.618921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:01:09.816758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312420+0000","last_change":"2024-03-21T10:35:06.941234+0000","last_active":"2024-03-21T10:36:56.312420+0000","last_peered":"2024-03-21T10:36:56.312420+0000","last_clean":"2024-03-21T10:36:56.312420+0000","last_became_active":"2024-03-21T10:35:06.940828+0000","last_became_peered":"2024-03-21T10:35:06.940828+0000","last_unstale":"2024-03-21T10:36:56.312420+0000","last_undegraded":"2024-03-21T10:36:56.312420+0000","last_fullsized":"2024-03-21T10:36:56.312420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:50:34.398822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618764+0000","last_change":"2024-03-21T10:35:11.196007+0000","last_active":"2024-03-21T10:36:56.618764+0000","last_peered":"2024-03-21T10:36:56.618764+0000","last_clean":"2024-03-21T10:36:56.618764+0000","last_became_active":"2024-03-21T10:35:11.195609+0000","last_became_peered":"2024-03-21T10:35:11.195609+0000","last_unstale":"2024-03-21T10:36:56.618764+0000","last_undegraded":"2024-03-21T10:36:56.618764+0000","last_fullsized":"2024-03-21T10:36:56.618764+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:21:38.803390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311467+0000","last_change":"2024-03-21T10:35:06.011594+0000","last_active":"2024-03-21T10:36:56.311467+0000","last_peered":"2024-03-21T10:36:56.311467+0000","last_clean":"2024-03-21T10:36:56.311467+0000","last_became_active":"2024-03-21T10:35:06.011254+0000","last_became_peered":"2024-03-21T10:35:06.011254+0000","last_unstale":"2024-03-21T10:36:56.311467+0000","last_undegraded":"2024-03-21T10:36:56.311467+0000","last_fullsized":"2024-03-21T10:36:56.311467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:52:31.045659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619892+0000","last_change":"2024-03-21T10:35:06.929170+0000","last_active":"2024-03-21T10:36:56.619892+0000","last_peered":"2024-03-21T10:36:56.619892+0000","last_clean":"2024-03-21T10:36:56.619892+0000","last_became_active":"2024-03-21T10:35:06.928822+0000","last_became_peered":"2024-03-21T10:35:06.928822+0000","last_unstale":"2024-03-21T10:36:56.619892+0000","last_undegraded":"2024-03-21T10:36:56.619892+0000","last_fullsized":"2024-03-21T10:36:56.619892+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:37:40.328453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.622940+0000","last_change":"2024-03-21T10:35:11.204285+0000","last_active":"2024-03-21T10:36:56.622940+0000","last_peered":"2024-03-21T10:36:56.622940+0000","last_clean":"2024-03-21T10:36:56.622940+0000","last_became_active":"2024-03-21T10:35:11.203698+0000","last_became_peered":"2024-03-21T10:35:11.203698+0000","last_unstale":"2024-03-21T10:36:56.622940+0000","last_undegraded":"2024-03-21T10:36:56.622940+0000","last_fullsized":"2024-03-21T10:36:56.622940+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:03:32.732488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'3","reported_seq":119,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:37:03.123726+0000","last_change":"2024-03-21T10:35:06.005028+0000","last_active":"2024-03-21T10:37:03.123726+0000","last_peered":"2024-03-21T10:37:03.123726+0000","last_clean":"2024-03-21T10:37:03.123726+0000","last_became_active":"2024-03-21T10:35:06.004670+0000","last_became_peered":"2024-03-21T10:35:06.004670+0000","last_unstale":"2024-03-21T10:37:03.123726+0000","last_undegraded":"2024-03-21T10:37:03.123726+0000","last_fullsized":"2024-03-21T10:37:03.123726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:00:18.481051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619489+0000","last_change":"2024-03-21T10:35:06.911696+0000","last_active":"2024-03-21T10:36:56.619489+0000","last_peered":"2024-03-21T10:36:56.619489+0000","last_clean":"2024-03-21T10:36:56.619489+0000","last_became_active":"2024-03-21T10:35:06.911231+0000","last_became_peered":"2024-03-21T10:35:06.911231+0000","last_unstale":"2024-03-21T10:36:56.619489+0000","last_undegraded":"2024-03-21T10:36:56.619489+0000","last_fullsized":"2024-03-21T10:36:56.619489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:21:51.540993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312860+0000","last_change":"2024-03-21T10:35:11.204422+0000","last_active":"2024-03-21T10:36:56.312860+0000","last_peered":"2024-03-21T10:36:56.312860+0000","last_clean":"2024-03-21T10:36:56.312860+0000","last_became_active":"2024-03-21T10:35:11.203904+0000","last_became_peered":"2024-03-21T10:35:11.203904+0000","last_unstale":"2024-03-21T10:36:56.312860+0000","last_undegraded":"2024-03-21T10:36:56.312860+0000","last_fullsized":"2024-03-21T10:36:56.312860+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:38:14.486012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313109+0000","last_change":"2024-03-21T10:35:06.019823+0000","last_active":"2024-03-21T10:36:56.313109+0000","last_peered":"2024-03-21T10:36:56.313109+0000","last_clean":"2024-03-21T10:36:56.313109+0000","last_became_active":"2024-03-21T10:35:06.019502+0000","last_became_peered":"2024-03-21T10:35:06.019502+0000","last_unstale":"2024-03-21T10:36:56.313109+0000","last_undegraded":"2024-03-21T10:36:56.313109+0000","last_fullsized":"2024-03-21T10:36:56.313109+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:33:22.438868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311184+0000","last_change":"2024-03-21T10:35:06.906671+0000","last_active":"2024-03-21T10:36:56.311184+0000","last_peered":"2024-03-21T10:36:56.311184+0000","last_clean":"2024-03-21T10:36:56.311184+0000","last_became_active":"2024-03-21T10:35:06.905873+0000","last_became_peered":"2024-03-21T10:35:06.905873+0000","last_unstale":"2024-03-21T10:36:56.311184+0000","last_undegraded":"2024-03-21T10:36:56.311184+0000","last_fullsized":"2024-03-21T10:36:56.311184+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:14:59.304255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311141+0000","last_change":"2024-03-21T10:35:11.197698+0000","last_active":"2024-03-21T10:36:56.311141+0000","last_peered":"2024-03-21T10:36:56.311141+0000","last_clean":"2024-03-21T10:36:56.311141+0000","last_became_active":"2024-03-21T10:35:11.197295+0000","last_became_peered":"2024-03-21T10:35:11.197295+0000","last_unstale":"2024-03-21T10:36:56.311141+0000","last_undegraded":"2024-03-21T10:36:56.311141+0000","last_fullsized":"2024-03-21T10:36:56.311141+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:02:24.338317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313163+0000","last_change":"2024-03-21T10:35:06.000951+0000","last_active":"2024-03-21T10:36:56.313163+0000","last_peered":"2024-03-21T10:36:56.313163+0000","last_clean":"2024-03-21T10:36:56.313163+0000","last_became_active":"2024-03-21T10:35:05.999169+0000","last_became_peered":"2024-03-21T10:35:05.999169+0000","last_unstale":"2024-03-21T10:36:56.313163+0000","last_undegraded":"2024-03-21T10:36:56.313163+0000","last_fullsized":"2024-03-21T10:36:56.313163+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:31:25.602621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619292+0000","last_change":"2024-03-21T10:35:06.908067+0000","last_active":"2024-03-21T10:36:56.619292+0000","last_peered":"2024-03-21T10:36:56.619292+0000","last_clean":"2024-03-21T10:36:56.619292+0000","last_became_active":"2024-03-21T10:35:06.907720+0000","last_became_peered":"2024-03-21T10:35:06.907720+0000","last_unstale":"2024-03-21T10:36:56.619292+0000","last_undegraded":"2024-03-21T10:36:56.619292+0000","last_fullsized":"2024-03-21T10:36:56.619292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:16:00.502637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312681+0000","last_change":"2024-03-21T10:35:11.229470+0000","last_active":"2024-03-21T10:36:56.312681+0000","last_peered":"2024-03-21T10:36:56.312681+0000","last_clean":"2024-03-21T10:36:56.312681+0000","last_became_active":"2024-03-21T10:35:11.228977+0000","last_became_peered":"2024-03-21T10:35:11.228977+0000","last_unstale":"2024-03-21T10:36:56.312681+0000","last_undegraded":"2024-03-21T10:36:56.312681+0000","last_fullsized":"2024-03-21T10:36:56.312681+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:05:52.269712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623076+0000","last_change":"2024-03-21T10:35:06.019866+0000","last_active":"2024-03-21T10:36:56.623076+0000","last_peered":"2024-03-21T10:36:56.623076+0000","last_clean":"2024-03-21T10:36:56.623076+0000","last_became_active":"2024-03-21T10:35:06.019550+0000","last_became_peered":"2024-03-21T10:35:06.019550+0000","last_unstale":"2024-03-21T10:36:56.623076+0000","last_undegraded":"2024-03-21T10:36:56.623076+0000","last_fullsized":"2024-03-21T10:36:56.623076+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:51:46.989647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311661+0000","last_change":"2024-03-21T10:35:55.385617+0000","last_active":"2024-03-21T10:36:56.311661+0000","last_peered":"2024-03-21T10:36:56.311661+0000","last_clean":"2024-03-21T10:36:56.311661+0000","last_became_active":"2024-03-21T10:35:55.385293+0000","last_became_peered":"2024-03-21T10:35:55.385293+0000","last_unstale":"2024-03-21T10:36:56.311661+0000","last_undegraded":"2024-03-21T10:36:56.311661+0000","last_fullsized":"2024-03-21T10:36:56.311661+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:18:08.200169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313017+0000","last_change":"2024-03-21T10:35:11.199714+0000","last_active":"2024-03-21T10:36:56.313017+0000","last_peered":"2024-03-21T10:36:56.313017+0000","last_clean":"2024-03-21T10:36:56.313017+0000","last_became_active":"2024-03-21T10:35:11.199277+0000","last_became_peered":"2024-03-21T10:35:11.199277+0000","last_unstale":"2024-03-21T10:36:56.313017+0000","last_undegraded":"2024-03-21T10:36:56.313017+0000","last_fullsized":"2024-03-21T10:36:56.313017+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:58:01.471162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619747+0000","last_change":"2024-03-21T10:35:06.015090+0000","last_active":"2024-03-21T10:36:56.619747+0000","last_peered":"2024-03-21T10:36:56.619747+0000","last_clean":"2024-03-21T10:36:56.619747+0000","last_became_active":"2024-03-21T10:35:06.014666+0000","last_became_peered":"2024-03-21T10:35:06.014666+0000","last_unstale":"2024-03-21T10:36:56.619747+0000","last_undegraded":"2024-03-21T10:36:56.619747+0000","last_fullsized":"2024-03-21T10:36:56.619747+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:41:17.608069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311329+0000","last_change":"2024-03-21T10:35:55.388548+0000","last_active":"2024-03-21T10:36:56.311329+0000","last_peered":"2024-03-21T10:36:56.311329+0000","last_clean":"2024-03-21T10:36:56.311329+0000","last_became_active":"2024-03-21T10:35:55.388253+0000","last_became_peered":"2024-03-21T10:35:55.388253+0000","last_unstale":"2024-03-21T10:36:56.311329+0000","last_undegraded":"2024-03-21T10:36:56.311329+0000","last_fullsized":"2024-03-21T10:36:56.311329+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:56.678224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312617+0000","last_change":"2024-03-21T10:35:11.230677+0000","last_active":"2024-03-21T10:36:56.312617+0000","last_peered":"2024-03-21T10:36:56.312617+0000","last_clean":"2024-03-21T10:36:56.312617+0000","last_became_active":"2024-03-21T10:35:11.230087+0000","last_became_peered":"2024-03-21T10:35:11.230087+0000","last_unstale":"2024-03-21T10:36:56.312617+0000","last_undegraded":"2024-03-21T10:36:56.312617+0000","last_fullsized":"2024-03-21T10:36:56.312617+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:39:47.663950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311339+0000","last_change":"2024-03-21T10:35:06.904287+0000","last_active":"2024-03-21T10:36:56.311339+0000","last_peered":"2024-03-21T10:36:56.311339+0000","last_clean":"2024-03-21T10:36:56.311339+0000","last_became_active":"2024-03-21T10:35:06.903821+0000","last_became_peered":"2024-03-21T10:35:06.903821+0000","last_unstale":"2024-03-21T10:36:56.311339+0000","last_undegraded":"2024-03-21T10:36:56.311339+0000","last_fullsized":"2024-03-21T10:36:56.311339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:36:25.362203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":124,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:37:08.130733+0000","last_change":"2024-03-21T10:35:06.014432+0000","last_active":"2024-03-21T10:37:08.130733+0000","last_peered":"2024-03-21T10:37:08.130733+0000","last_clean":"2024-03-21T10:37:08.130733+0000","last_became_active":"2024-03-21T10:35:06.013448+0000","last_became_peered":"2024-03-21T10:35:06.013448+0000","last_unstale":"2024-03-21T10:37:08.130733+0000","last_undegraded":"2024-03-21T10:37:08.130733+0000","last_fullsized":"2024-03-21T10:37:08.130733+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:29:10.600794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":89,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619058+0000","last_change":"2024-03-21T10:35:11.193577+0000","last_active":"2024-03-21T10:36:56.619058+0000","last_peered":"2024-03-21T10:36:56.619058+0000","last_clean":"2024-03-21T10:36:56.619058+0000","last_became_active":"2024-03-21T10:35:11.193132+0000","last_became_peered":"2024-03-21T10:35:11.193132+0000","last_unstale":"2024-03-21T10:36:56.619058+0000","last_undegraded":"2024-03-21T10:36:56.619058+0000","last_fullsized":"2024-03-21T10:36:56.619058+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:40:31.100489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325044+0000","last_change":"2024-03-21T10:35:06.924376+0000","last_active":"2024-03-21T10:36:56.325044+0000","last_peered":"2024-03-21T10:36:56.325044+0000","last_clean":"2024-03-21T10:36:56.325044+0000","last_became_active":"2024-03-21T10:35:06.923874+0000","last_became_peered":"2024-03-21T10:35:06.923874+0000","last_unstale":"2024-03-21T10:36:56.325044+0000","last_undegraded":"2024-03-21T10:36:56.325044+0000","last_fullsized":"2024-03-21T10:36:56.325044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:56:12.620667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540544+0000","last_change":"2024-03-21T10:35:06.015222+0000","last_active":"2024-03-21T10:36:56.540544+0000","last_peered":"2024-03-21T10:36:56.540544+0000","last_clean":"2024-03-21T10:36:56.540544+0000","last_became_active":"2024-03-21T10:35:06.014858+0000","last_became_peered":"2024-03-21T10:35:06.014858+0000","last_unstale":"2024-03-21T10:36:56.540544+0000","last_undegraded":"2024-03-21T10:36:56.540544+0000","last_fullsized":"2024-03-21T10:36:56.540544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:45:12.528851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":366,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618752+0000","last_change":"2024-03-21T10:33:44.536069+0000","last_active":"2024-03-21T10:36:56.618752+0000","last_peered":"2024-03-21T10:36:56.618752+0000","last_clean":"2024-03-21T10:36:56.618752+0000","last_became_active":"2024-03-21T10:33:44.535623+0000","last_became_peered":"2024-03-21T10:33:44.535623+0000","last_unstale":"2024-03-21T10:36:56.618752+0000","last_undegraded":"2024-03-21T10:36:56.618752+0000","last_fullsized":"2024-03-21T10:36:56.618752+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:29:44.554765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:29:44.554765+0000","last_clean_scrub_stamp":"2024-03-21T10:29:44.554765+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:43:23.440837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311600+0000","last_change":"2024-03-21T10:35:11.229580+0000","last_active":"2024-03-21T10:36:56.311600+0000","last_peered":"2024-03-21T10:36:56.311600+0000","last_clean":"2024-03-21T10:36:56.311600+0000","last_became_active":"2024-03-21T10:35:11.229044+0000","last_became_peered":"2024-03-21T10:35:11.229044+0000","last_unstale":"2024-03-21T10:36:56.311600+0000","last_undegraded":"2024-03-21T10:36:56.311600+0000","last_fullsized":"2024-03-21T10:36:56.311600+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:54:50.535253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311965+0000","last_change":"2024-03-21T10:35:06.927690+0000","last_active":"2024-03-21T10:36:56.311965+0000","last_peered":"2024-03-21T10:36:56.311965+0000","last_clean":"2024-03-21T10:36:56.311965+0000","last_became_active":"2024-03-21T10:35:06.927304+0000","last_became_peered":"2024-03-21T10:35:06.927304+0000","last_unstale":"2024-03-21T10:36:56.311965+0000","last_undegraded":"2024-03-21T10:36:56.311965+0000","last_fullsized":"2024-03-21T10:36:56.311965+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:30:48.950459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619515+0000","last_change":"2024-03-21T10:35:06.004952+0000","last_active":"2024-03-21T10:36:56.619515+0000","last_peered":"2024-03-21T10:36:56.619515+0000","last_clean":"2024-03-21T10:36:56.619515+0000","last_became_active":"2024-03-21T10:35:06.004206+0000","last_became_peered":"2024-03-21T10:35:06.004206+0000","last_unstale":"2024-03-21T10:36:56.619515+0000","last_undegraded":"2024-03-21T10:36:56.619515+0000","last_fullsized":"2024-03-21T10:36:56.619515+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:20:29.198715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619011+0000","last_change":"2024-03-21T10:35:11.179125+0000","last_active":"2024-03-21T10:36:56.619011+0000","last_peered":"2024-03-21T10:36:56.619011+0000","last_clean":"2024-03-21T10:36:56.619011+0000","last_became_active":"2024-03-21T10:35:11.178658+0000","last_became_peered":"2024-03-21T10:35:11.178658+0000","last_unstale":"2024-03-21T10:36:56.619011+0000","last_undegraded":"2024-03-21T10:36:56.619011+0000","last_fullsized":"2024-03-21T10:36:56.619011+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:33:25.281721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312967+0000","last_change":"2024-03-21T10:35:06.019996+0000","last_active":"2024-03-21T10:36:56.312967+0000","last_peered":"2024-03-21T10:36:56.312967+0000","last_clean":"2024-03-21T10:36:56.312967+0000","last_became_active":"2024-03-21T10:35:06.019624+0000","last_became_peered":"2024-03-21T10:35:06.019624+0000","last_unstale":"2024-03-21T10:36:56.312967+0000","last_undegraded":"2024-03-21T10:36:56.312967+0000","last_fullsized":"2024-03-21T10:36:56.312967+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:31:09.551433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313425+0000","last_change":"2024-03-21T10:35:06.939324+0000","last_active":"2024-03-21T10:36:56.313425+0000","last_peered":"2024-03-21T10:36:56.313425+0000","last_clean":"2024-03-21T10:36:56.313425+0000","last_became_active":"2024-03-21T10:35:06.937750+0000","last_became_peered":"2024-03-21T10:35:06.937750+0000","last_unstale":"2024-03-21T10:36:56.313425+0000","last_undegraded":"2024-03-21T10:36:56.313425+0000","last_fullsized":"2024-03-21T10:36:56.313425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:45:14.600863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.324871+0000","last_change":"2024-03-21T10:35:11.229479+0000","last_active":"2024-03-21T10:36:56.324871+0000","last_peered":"2024-03-21T10:36:56.324871+0000","last_clean":"2024-03-21T10:36:56.324871+0000","last_became_active":"2024-03-21T10:35:11.229005+0000","last_became_peered":"2024-03-21T10:35:11.229005+0000","last_unstale":"2024-03-21T10:36:56.324871+0000","last_undegraded":"2024-03-21T10:36:56.324871+0000","last_fullsized":"2024-03-21T10:36:56.324871+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:39:19.859581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'11","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623558+0000","last_change":"2024-03-21T10:35:06.013744+0000","last_active":"2024-03-21T10:36:56.623558+0000","last_peered":"2024-03-21T10:36:56.623558+0000","last_clean":"2024-03-21T10:36:56.623558+0000","last_became_active":"2024-03-21T10:35:06.013509+0000","last_became_peered":"2024-03-21T10:35:06.013509+0000","last_unstale":"2024-03-21T10:36:56.623558+0000","last_undegraded":"2024-03-21T10:36:56.623558+0000","last_fullsized":"2024-03-21T10:36:56.623558+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:51:45.000933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325882+0000","last_change":"2024-03-21T10:35:06.930685+0000","last_active":"2024-03-21T10:36:56.325882+0000","last_peered":"2024-03-21T10:36:56.325882+0000","last_clean":"2024-03-21T10:36:56.325882+0000","last_became_active":"2024-03-21T10:35:06.929775+0000","last_became_peered":"2024-03-21T10:35:06.929775+0000","last_unstale":"2024-03-21T10:36:56.325882+0000","last_undegraded":"2024-03-21T10:36:56.325882+0000","last_fullsized":"2024-03-21T10:36:56.325882+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:13:18.058752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618559+0000","last_change":"2024-03-21T10:35:11.177061+0000","last_active":"2024-03-21T10:36:56.618559+0000","last_peered":"2024-03-21T10:36:56.618559+0000","last_clean":"2024-03-21T10:36:56.618559+0000","last_became_active":"2024-03-21T10:35:11.176603+0000","last_became_peered":"2024-03-21T10:35:11.176603+0000","last_unstale":"2024-03-21T10:36:56.618559+0000","last_undegraded":"2024-03-21T10:36:56.618559+0000","last_fullsized":"2024-03-21T10:36:56.618559+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:54.532780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540908+0000","last_change":"2024-03-21T10:35:06.009018+0000","last_active":"2024-03-21T10:36:56.540908+0000","last_peered":"2024-03-21T10:36:56.540908+0000","last_clean":"2024-03-21T10:36:56.540908+0000","last_became_active":"2024-03-21T10:35:06.008638+0000","last_became_peered":"2024-03-21T10:35:06.008638+0000","last_unstale":"2024-03-21T10:36:56.540908+0000","last_undegraded":"2024-03-21T10:36:56.540908+0000","last_fullsized":"2024-03-21T10:36:56.540908+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:42:19.527898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621871+0000","last_change":"2024-03-21T10:35:06.931588+0000","last_active":"2024-03-21T10:36:56.621871+0000","last_peered":"2024-03-21T10:36:56.621871+0000","last_clean":"2024-03-21T10:36:56.621871+0000","last_became_active":"2024-03-21T10:35:06.930707+0000","last_became_peered":"2024-03-21T10:35:06.930707+0000","last_unstale":"2024-03-21T10:36:56.621871+0000","last_undegraded":"2024-03-21T10:36:56.621871+0000","last_fullsized":"2024-03-21T10:36:56.621871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:44:55.862799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540757+0000","last_change":"2024-03-21T10:35:11.229408+0000","last_active":"2024-03-21T10:36:56.540757+0000","last_peered":"2024-03-21T10:36:56.540757+0000","last_clean":"2024-03-21T10:36:56.540757+0000","last_became_active":"2024-03-21T10:35:11.228997+0000","last_became_peered":"2024-03-21T10:35:11.228997+0000","last_unstale":"2024-03-21T10:36:56.540757+0000","last_undegraded":"2024-03-21T10:36:56.540757+0000","last_fullsized":"2024-03-21T10:36:56.540757+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:56:11.517597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619202+0000","last_change":"2024-03-21T10:35:55.388978+0000","last_active":"2024-03-21T10:36:56.619202+0000","last_peered":"2024-03-21T10:36:56.619202+0000","last_clean":"2024-03-21T10:36:56.619202+0000","last_became_active":"2024-03-21T10:35:55.383048+0000","last_became_peered":"2024-03-21T10:35:55.383048+0000","last_unstale":"2024-03-21T10:36:56.619202+0000","last_undegraded":"2024-03-21T10:36:56.619202+0000","last_fullsized":"2024-03-21T10:36:56.619202+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:24:01.933115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312584+0000","last_change":"2024-03-21T10:35:06.908585+0000","last_active":"2024-03-21T10:36:56.312584+0000","last_peered":"2024-03-21T10:36:56.312584+0000","last_clean":"2024-03-21T10:36:56.312584+0000","last_became_active":"2024-03-21T10:35:06.908253+0000","last_became_peered":"2024-03-21T10:35:06.908253+0000","last_unstale":"2024-03-21T10:36:56.312584+0000","last_undegraded":"2024-03-21T10:36:56.312584+0000","last_fullsized":"2024-03-21T10:36:56.312584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:46:15.984450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623210+0000","last_change":"2024-03-21T10:35:11.199455+0000","last_active":"2024-03-21T10:36:56.623210+0000","last_peered":"2024-03-21T10:36:56.623210+0000","last_clean":"2024-03-21T10:36:56.623210+0000","last_became_active":"2024-03-21T10:35:11.197705+0000","last_became_peered":"2024-03-21T10:35:11.197705+0000","last_unstale":"2024-03-21T10:36:56.623210+0000","last_undegraded":"2024-03-21T10:36:56.623210+0000","last_fullsized":"2024-03-21T10:36:56.623210+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:59:42.792764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325091+0000","last_change":"2024-03-21T10:35:06.016152+0000","last_active":"2024-03-21T10:36:56.325091+0000","last_peered":"2024-03-21T10:36:56.325091+0000","last_clean":"2024-03-21T10:36:56.325091+0000","last_became_active":"2024-03-21T10:35:06.015439+0000","last_became_peered":"2024-03-21T10:35:06.015439+0000","last_unstale":"2024-03-21T10:36:56.325091+0000","last_undegraded":"2024-03-21T10:36:56.325091+0000","last_fullsized":"2024-03-21T10:36:56.325091+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:05:23.221826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618788+0000","last_change":"2024-03-21T10:35:06.929241+0000","last_active":"2024-03-21T10:36:56.618788+0000","last_peered":"2024-03-21T10:36:56.618788+0000","last_clean":"2024-03-21T10:36:56.618788+0000","last_became_active":"2024-03-21T10:35:06.928943+0000","last_became_peered":"2024-03-21T10:35:06.928943+0000","last_unstale":"2024-03-21T10:36:56.618788+0000","last_undegraded":"2024-03-21T10:36:56.618788+0000","last_fullsized":"2024-03-21T10:36:56.618788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:24:10.484540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540647+0000","last_change":"2024-03-21T10:35:11.195929+0000","last_active":"2024-03-21T10:36:56.540647+0000","last_peered":"2024-03-21T10:36:56.540647+0000","last_clean":"2024-03-21T10:36:56.540647+0000","last_became_active":"2024-03-21T10:35:11.195489+0000","last_became_peered":"2024-03-21T10:35:11.195489+0000","last_unstale":"2024-03-21T10:36:56.540647+0000","last_undegraded":"2024-03-21T10:36:56.540647+0000","last_fullsized":"2024-03-21T10:36:56.540647+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:41:15.137917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618664+0000","last_change":"2024-03-21T10:35:06.002127+0000","last_active":"2024-03-21T10:36:56.618664+0000","last_peered":"2024-03-21T10:36:56.618664+0000","last_clean":"2024-03-21T10:36:56.618664+0000","last_became_active":"2024-03-21T10:35:06.001706+0000","last_became_peered":"2024-03-21T10:35:06.001706+0000","last_unstale":"2024-03-21T10:36:56.618664+0000","last_undegraded":"2024-03-21T10:36:56.618664+0000","last_fullsized":"2024-03-21T10:36:56.618664+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:26:32.844991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325124+0000","last_change":"2024-03-21T10:35:06.925945+0000","last_active":"2024-03-21T10:36:56.325124+0000","last_peered":"2024-03-21T10:36:56.325124+0000","last_clean":"2024-03-21T10:36:56.325124+0000","last_became_active":"2024-03-21T10:35:06.925551+0000","last_became_peered":"2024-03-21T10:35:06.925551+0000","last_unstale":"2024-03-21T10:36:56.325124+0000","last_undegraded":"2024-03-21T10:36:56.325124+0000","last_fullsized":"2024-03-21T10:36:56.325124+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:24:10.212676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540869+0000","last_change":"2024-03-21T10:35:11.229067+0000","last_active":"2024-03-21T10:36:56.540869+0000","last_peered":"2024-03-21T10:36:56.540869+0000","last_clean":"2024-03-21T10:36:56.540869+0000","last_became_active":"2024-03-21T10:35:11.228628+0000","last_became_peered":"2024-03-21T10:35:11.228628+0000","last_unstale":"2024-03-21T10:36:56.540869+0000","last_undegraded":"2024-03-21T10:36:56.540869+0000","last_fullsized":"2024-03-21T10:36:56.540869+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:45:05.473277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618908+0000","last_change":"2024-03-21T10:35:55.384922+0000","last_active":"2024-03-21T10:36:56.618908+0000","last_peered":"2024-03-21T10:36:56.618908+0000","last_clean":"2024-03-21T10:36:56.618908+0000","last_became_active":"2024-03-21T10:35:55.380952+0000","last_became_peered":"2024-03-21T10:35:55.380952+0000","last_unstale":"2024-03-21T10:36:56.618908+0000","last_undegraded":"2024-03-21T10:36:56.618908+0000","last_fullsized":"2024-03-21T10:36:56.618908+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:29:37.873699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619590+0000","last_change":"2024-03-21T10:35:06.928049+0000","last_active":"2024-03-21T10:36:56.619590+0000","last_peered":"2024-03-21T10:36:56.619590+0000","last_clean":"2024-03-21T10:36:56.619590+0000","last_became_active":"2024-03-21T10:35:06.927717+0000","last_became_peered":"2024-03-21T10:35:06.927717+0000","last_unstale":"2024-03-21T10:36:56.619590+0000","last_undegraded":"2024-03-21T10:36:56.619590+0000","last_fullsized":"2024-03-21T10:36:56.619590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:27:47.554668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619331+0000","last_change":"2024-03-21T10:35:11.181090+0000","last_active":"2024-03-21T10:36:56.619331+0000","last_peered":"2024-03-21T10:36:56.619331+0000","last_clean":"2024-03-21T10:36:56.619331+0000","last_became_active":"2024-03-21T10:35:11.180511+0000","last_became_peered":"2024-03-21T10:35:11.180511+0000","last_unstale":"2024-03-21T10:36:56.619331+0000","last_undegraded":"2024-03-21T10:36:56.619331+0000","last_fullsized":"2024-03-21T10:36:56.619331+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:21:51.517845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619547+0000","last_change":"2024-03-21T10:35:06.007828+0000","last_active":"2024-03-21T10:36:56.619547+0000","last_peered":"2024-03-21T10:36:56.619547+0000","last_clean":"2024-03-21T10:36:56.619547+0000","last_became_active":"2024-03-21T10:35:06.007490+0000","last_became_peered":"2024-03-21T10:35:06.007490+0000","last_unstale":"2024-03-21T10:36:56.619547+0000","last_undegraded":"2024-03-21T10:36:56.619547+0000","last_fullsized":"2024-03-21T10:36:56.619547+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:08:18.579308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619789+0000","last_change":"2024-03-21T10:35:06.909313+0000","last_active":"2024-03-21T10:36:56.619789+0000","last_peered":"2024-03-21T10:36:56.619789+0000","last_clean":"2024-03-21T10:36:56.619789+0000","last_became_active":"2024-03-21T10:35:06.908555+0000","last_became_peered":"2024-03-21T10:35:06.908555+0000","last_unstale":"2024-03-21T10:36:56.619789+0000","last_undegraded":"2024-03-21T10:36:56.619789+0000","last_fullsized":"2024-03-21T10:36:56.619789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:09:57.961971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.622209+0000","last_change":"2024-03-21T10:35:11.195996+0000","last_active":"2024-03-21T10:36:56.622209+0000","last_peered":"2024-03-21T10:36:56.622209+0000","last_clean":"2024-03-21T10:36:56.622209+0000","last_became_active":"2024-03-21T10:35:11.195506+0000","last_became_peered":"2024-03-21T10:35:11.195506+0000","last_unstale":"2024-03-21T10:36:56.622209+0000","last_undegraded":"2024-03-21T10:36:56.622209+0000","last_fullsized":"2024-03-21T10:36:56.622209+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:49:33.010899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311536+0000","last_change":"2024-03-21T10:35:06.011498+0000","last_active":"2024-03-21T10:36:56.311536+0000","last_peered":"2024-03-21T10:36:56.311536+0000","last_clean":"2024-03-21T10:36:56.311536+0000","last_became_active":"2024-03-21T10:35:06.011203+0000","last_became_peered":"2024-03-21T10:35:06.011203+0000","last_unstale":"2024-03-21T10:36:56.311536+0000","last_undegraded":"2024-03-21T10:36:56.311536+0000","last_fullsized":"2024-03-21T10:36:56.311536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:28:11.369740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619070+0000","last_change":"2024-03-21T10:35:06.935514+0000","last_active":"2024-03-21T10:36:56.619070+0000","last_peered":"2024-03-21T10:36:56.619070+0000","last_clean":"2024-03-21T10:36:56.619070+0000","last_became_active":"2024-03-21T10:35:06.935174+0000","last_became_peered":"2024-03-21T10:35:06.935174+0000","last_unstale":"2024-03-21T10:36:56.619070+0000","last_undegraded":"2024-03-21T10:36:56.619070+0000","last_fullsized":"2024-03-21T10:36:56.619070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:35:42.884471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621386+0000","last_change":"2024-03-21T10:36:56.621053+0000","last_active":"2024-03-21T10:36:56.621386+0000","last_peered":"2024-03-21T10:36:56.621386+0000","last_clean":"2024-03-21T10:36:56.621386+0000","last_became_active":"2024-03-21T10:36:56.620688+0000","last_became_peered":"2024-03-21T10:36:56.620688+0000","last_unstale":"2024-03-21T10:36:56.621386+0000","last_undegraded":"2024-03-21T10:36:56.621386+0000","last_fullsized":"2024-03-21T10:36:56.621386+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:50:02.121787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312981+0000","last_change":"2024-03-21T10:35:06.007216+0000","last_active":"2024-03-21T10:36:56.312981+0000","last_peered":"2024-03-21T10:36:56.312981+0000","last_clean":"2024-03-21T10:36:56.312981+0000","last_became_active":"2024-03-21T10:35:06.006883+0000","last_became_peered":"2024-03-21T10:35:06.006883+0000","last_unstale":"2024-03-21T10:36:56.312981+0000","last_undegraded":"2024-03-21T10:36:56.312981+0000","last_fullsized":"2024-03-21T10:36:56.312981+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:46:45.310919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619751+0000","last_change":"2024-03-21T10:35:06.906017+0000","last_active":"2024-03-21T10:36:56.619751+0000","last_peered":"2024-03-21T10:36:56.619751+0000","last_clean":"2024-03-21T10:36:56.619751+0000","last_became_active":"2024-03-21T10:35:06.905648+0000","last_became_peered":"2024-03-21T10:35:06.905648+0000","last_unstale":"2024-03-21T10:36:56.619751+0000","last_undegraded":"2024-03-21T10:36:56.619751+0000","last_fullsized":"2024-03-21T10:36:56.619751+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:56:50.380146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618905+0000","last_change":"2024-03-21T10:35:06.905186+0000","last_active":"2024-03-21T10:36:56.618905+0000","last_peered":"2024-03-21T10:36:56.618905+0000","last_clean":"2024-03-21T10:36:56.618905+0000","last_became_active":"2024-03-21T10:35:06.904835+0000","last_became_peered":"2024-03-21T10:35:06.904835+0000","last_unstale":"2024-03-21T10:36:56.618905+0000","last_undegraded":"2024-03-21T10:36:56.618905+0000","last_fullsized":"2024-03-21T10:36:56.618905+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:53:19.293437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618890+0000","last_change":"2024-03-21T10:35:06.903318+0000","last_active":"2024-03-21T10:36:56.618890+0000","last_peered":"2024-03-21T10:36:56.618890+0000","last_clean":"2024-03-21T10:36:56.618890+0000","last_became_active":"2024-03-21T10:35:06.902994+0000","last_became_peered":"2024-03-21T10:35:06.902994+0000","last_unstale":"2024-03-21T10:36:56.618890+0000","last_undegraded":"2024-03-21T10:36:56.618890+0000","last_fullsized":"2024-03-21T10:36:56.618890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:21:27.412565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618616+0000","last_change":"2024-03-21T10:35:06.934582+0000","last_active":"2024-03-21T10:36:56.618616+0000","last_peered":"2024-03-21T10:36:56.618616+0000","last_clean":"2024-03-21T10:36:56.618616+0000","last_became_active":"2024-03-21T10:35:06.934311+0000","last_became_peered":"2024-03-21T10:35:06.934311+0000","last_unstale":"2024-03-21T10:36:56.618616+0000","last_undegraded":"2024-03-21T10:36:56.618616+0000","last_fullsized":"2024-03-21T10:36:56.618616+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:05:30.785830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.622771+0000","last_change":"2024-03-21T10:35:06.944221+0000","last_active":"2024-03-21T10:36:56.622771+0000","last_peered":"2024-03-21T10:36:56.622771+0000","last_clean":"2024-03-21T10:36:56.622771+0000","last_became_active":"2024-03-21T10:35:06.943732+0000","last_became_peered":"2024-03-21T10:35:06.943732+0000","last_unstale":"2024-03-21T10:36:56.622771+0000","last_undegraded":"2024-03-21T10:36:56.622771+0000","last_fullsized":"2024-03-21T10:36:56.622771+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:33:03.653160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313274+0000","last_change":"2024-03-21T10:35:06.940832+0000","last_active":"2024-03-21T10:36:56.313274+0000","last_peered":"2024-03-21T10:36:56.313274+0000","last_clean":"2024-03-21T10:36:56.313274+0000","last_became_active":"2024-03-21T10:35:06.939565+0000","last_became_peered":"2024-03-21T10:35:06.939565+0000","last_unstale":"2024-03-21T10:36:56.313274+0000","last_undegraded":"2024-03-21T10:36:56.313274+0000","last_fullsized":"2024-03-21T10:36:56.313274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:57:27.743704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619538+0000","last_change":"2024-03-21T10:35:06.907753+0000","last_active":"2024-03-21T10:36:56.619538+0000","last_peered":"2024-03-21T10:36:56.619538+0000","last_clean":"2024-03-21T10:36:56.619538+0000","last_became_active":"2024-03-21T10:35:06.907432+0000","last_became_peered":"2024-03-21T10:35:06.907432+0000","last_unstale":"2024-03-21T10:36:56.619538+0000","last_undegraded":"2024-03-21T10:36:56.619538+0000","last_fullsized":"2024-03-21T10:36:56.619538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:31:02.528855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619622+0000","last_change":"2024-03-21T10:35:06.908597+0000","last_active":"2024-03-21T10:36:56.619622+0000","last_peered":"2024-03-21T10:36:56.619622+0000","last_clean":"2024-03-21T10:36:56.619622+0000","last_became_active":"2024-03-21T10:35:06.908282+0000","last_became_peered":"2024-03-21T10:35:06.908282+0000","last_unstale":"2024-03-21T10:36:56.619622+0000","last_undegraded":"2024-03-21T10:36:56.619622+0000","last_fullsized":"2024-03-21T10:36:56.619622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:42:18.221045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325969+0000","last_change":"2024-03-21T10:35:06.931015+0000","last_active":"2024-03-21T10:36:56.325969+0000","last_peered":"2024-03-21T10:36:56.325969+0000","last_clean":"2024-03-21T10:36:56.325969+0000","last_became_active":"2024-03-21T10:35:06.930209+0000","last_became_peered":"2024-03-21T10:35:06.930209+0000","last_unstale":"2024-03-21T10:36:56.325969+0000","last_undegraded":"2024-03-21T10:36:56.325969+0000","last_fullsized":"2024-03-21T10:36:56.325969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:42:15.277874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311194+0000","last_change":"2024-03-21T10:35:06.930070+0000","last_active":"2024-03-21T10:36:56.311194+0000","last_peered":"2024-03-21T10:36:56.311194+0000","last_clean":"2024-03-21T10:36:56.311194+0000","last_became_active":"2024-03-21T10:35:06.929773+0000","last_became_peered":"2024-03-21T10:35:06.929773+0000","last_unstale":"2024-03-21T10:36:56.311194+0000","last_undegraded":"2024-03-21T10:36:56.311194+0000","last_fullsized":"2024-03-21T10:36:56.311194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:17:25.176666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325300+0000","last_change":"2024-03-21T10:35:06.929575+0000","last_active":"2024-03-21T10:36:56.325300+0000","last_peered":"2024-03-21T10:36:56.325300+0000","last_clean":"2024-03-21T10:36:56.325300+0000","last_became_active":"2024-03-21T10:35:06.929240+0000","last_became_peered":"2024-03-21T10:35:06.929240+0000","last_unstale":"2024-03-21T10:36:56.325300+0000","last_undegraded":"2024-03-21T10:36:56.325300+0000","last_fullsized":"2024-03-21T10:36:56.325300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:03:40.697640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619574+0000","last_change":"2024-03-21T10:35:06.934693+0000","last_active":"2024-03-21T10:36:56.619574+0000","last_peered":"2024-03-21T10:36:56.619574+0000","last_clean":"2024-03-21T10:36:56.619574+0000","last_became_active":"2024-03-21T10:35:06.934361+0000","last_became_peered":"2024-03-21T10:35:06.934361+0000","last_unstale":"2024-03-21T10:36:56.619574+0000","last_undegraded":"2024-03-21T10:36:56.619574+0000","last_fullsized":"2024-03-21T10:36:56.619574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:14:00.954370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312448+0000","last_change":"2024-03-21T10:35:06.941095+0000","last_active":"2024-03-21T10:36:56.312448+0000","last_peered":"2024-03-21T10:36:56.312448+0000","last_clean":"2024-03-21T10:36:56.312448+0000","last_became_active":"2024-03-21T10:35:06.939348+0000","last_became_peered":"2024-03-21T10:35:06.939348+0000","last_unstale":"2024-03-21T10:36:56.312448+0000","last_undegraded":"2024-03-21T10:36:56.312448+0000","last_fullsized":"2024-03-21T10:36:56.312448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:31:43.783592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619147+0000","last_change":"2024-03-21T10:35:06.908340+0000","last_active":"2024-03-21T10:36:56.619147+0000","last_peered":"2024-03-21T10:36:56.619147+0000","last_clean":"2024-03-21T10:36:56.619147+0000","last_became_active":"2024-03-21T10:35:06.908043+0000","last_became_peered":"2024-03-21T10:35:06.908043+0000","last_unstale":"2024-03-21T10:36:56.619147+0000","last_undegraded":"2024-03-21T10:36:56.619147+0000","last_fullsized":"2024-03-21T10:36:56.619147+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:50:33.167396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282863,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68597,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":545,"num_read_kb":346,"num_write":172,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":205791,"data_compressed":5487,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383720,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1019904,"data_stored":519397,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645356,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32320,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712832,"statfs":{"total":95995035648,"available":95961939968,"internally_reserved":0,"allocated":1163264,"data_stored":1171120,"data_compressed":64463,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808882,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":523729,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070521,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1036288,"data_stored":519553,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332158,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1085440,"data_stored":1115603,"data_compressed":62634,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495685,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":514809,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574945,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1081344,"data_stored":579174,"data_compressed":28489,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738471,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":516197,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1105920,"data_stored":581056,"data_compressed":28489,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182484,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":515003,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920846,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":519459,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757322,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1093632,"data_stored":1111191,"data_compressed":62634,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34331,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34331,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34331,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":64455,"data_compressed":1829,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":66337,"data_compressed":1829,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":66033,"data_compressed":1829,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T10:37:12.005 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T10:37:12.486 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:37:12.631 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:12 smithi031 ceph-mon[31673]: from='client.25048 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T10:37:12.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:12 smithi040 ceph-mon[36154]: from='client.25048 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T10:37:12.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:12 smithi070 ceph-mon[36330]: from='client.25048 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T10:37:13.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:13 smithi040 ceph-mon[36154]: pgmap v392: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:37:13.720 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:13 smithi031 ceph-mon[31673]: pgmap v392: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:37:13.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:13 smithi070 ceph-mon[36330]: pgmap v392: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:37:14.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.899+0000 7f5908000700 1 -- 172.21.15.31:0/1468549688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5900102df0 msgr2=0x7f5900106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:14.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.899+0000 7f5908000700 1 --2- 172.21.15.31:0/1468549688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5900102df0 0x7f5900106e30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f58ec0099c0 tx=0x7f58ec009cd0 comp rx=0 tx=0).stop 2024-03-21T10:37:14.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.899+0000 7f5908000700 1 -- 172.21.15.31:0/1468549688 shutdown_connections 2024-03-21T10:37:14.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.899+0000 7f5908000700 1 --2- 172.21.15.31:0/1468549688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5900107510 0x7f590010f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:14.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.899+0000 7f5908000700 1 --2- 172.21.15.31:0/1468549688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5900102440 0x7f5900102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:14.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.899+0000 7f5908000700 1 --2- 172.21.15.31:0/1468549688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5900102df0 0x7f5900106e30 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:14.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.899+0000 7f5908000700 1 -- 172.21.15.31:0/1468549688 >> 172.21.15.31:0/1468549688 conn(0x7f59000792f0 msgr2=0x7f5900079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:14.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.899+0000 7f5908000700 1 -- 172.21.15.31:0/1468549688 shutdown_connections 2024-03-21T10:37:14.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.899+0000 7f5908000700 1 -- 172.21.15.31:0/1468549688 wait complete. 2024-03-21T10:37:14.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.900+0000 7f5908000700 1 Processor -- start 2024-03-21T10:37:14.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.901+0000 7f5908000700 1 -- start start 2024-03-21T10:37:14.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.901+0000 7f5908000700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5900102440 0x7f590019e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:14.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.901+0000 7f5908000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5900102df0 0x7f590019ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:14.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.901+0000 7f5908000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5900107510 0x7f5900198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:14.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.901+0000 7f5908000700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5900112520 con 0x7f5900102440 2024-03-21T10:37:14.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.901+0000 7f5908000700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59001123a0 con 0x7f5900102df0 2024-03-21T10:37:14.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.901+0000 7f5908000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59001126a0 con 0x7f5900107510 2024-03-21T10:37:14.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.902+0000 7f5904c00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5900102df0 0x7f590019ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:14.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.902+0000 7f5906000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5900107510 0x7f5900198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:14.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.902+0000 7f5905600700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5900102440 0x7f590019e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:14.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.902+0000 7f5904c00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5900102df0 0x7f590019ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:54894/0 (socket says 172.21.15.31:54894) 2024-03-21T10:37:14.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.902+0000 7f5904c00700 1 -- 172.21.15.31:0/1919505841 learned_addr learned my addr 172.21.15.31:0/1919505841 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:14.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.902+0000 7f5906000700 1 -- 172.21.15.31:0/1919505841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5900102df0 msgr2=0x7f590019ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:14.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.902+0000 7f5906000700 1 --2- 172.21.15.31:0/1919505841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5900102df0 0x7f590019ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:14.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.902+0000 7f5906000700 1 -- 172.21.15.31:0/1919505841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5900102440 msgr2=0x7f590019e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:14.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.902+0000 7f5906000700 1 --2- 172.21.15.31:0/1919505841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5900102440 0x7f590019e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:14.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.902+0000 7f5906000700 1 -- 172.21.15.31:0/1919505841 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5900198fe0 con 0x7f5900107510 2024-03-21T10:37:14.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.902+0000 7f5904c00700 1 --2- 172.21.15.31:0/1919505841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5900102df0 0x7f590019ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:37:14.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.903+0000 7f5906000700 1 --2- 172.21.15.31:0/1919505841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5900107510 0x7f5900198880 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f58f800efc0 tx=0x7f58f800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:14.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.903+0000 7f58fe000700 1 -- 172.21.15.31:0/1919505841 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f800cb40 con 0x7f5900107510 2024-03-21T10:37:14.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.903+0000 7f5908000700 1 -- 172.21.15.31:0/1919505841 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5900199270 con 0x7f5900107510 2024-03-21T10:37:14.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.903+0000 7f58fe000700 1 -- 172.21.15.31:0/1919505841 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f58f8019070 con 0x7f5900107510 2024-03-21T10:37:14.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.903+0000 7f58fe000700 1 -- 172.21.15.31:0/1919505841 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f80146e0 con 0x7f5900107510 2024-03-21T10:37:14.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.903+0000 7f5908000700 1 -- 172.21.15.31:0/1919505841 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59001a5590 con 0x7f5900107510 2024-03-21T10:37:14.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.905+0000 7f5908000700 1 -- 172.21.15.31:0/1919505841 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5900068e50 con 0x7f5900107510 2024-03-21T10:37:14.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.906+0000 7f58fe000700 1 -- 172.21.15.31:0/1919505841 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f58f8012050 con 0x7f5900107510 2024-03-21T10:37:14.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.907+0000 7f58fe000700 1 --2- 172.21.15.31:0/1919505841 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f58e80a3030 0x7f58e80a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:14.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.908+0000 7f58fe000700 1 -- 172.21.15.31:0/1919505841 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f58f80cf220 con 0x7f5900107510 2024-03-21T10:37:14.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.908+0000 7f5905600700 1 --2- 172.21.15.31:0/1919505841 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f58e80a3030 0x7f58e80a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:14.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.908+0000 7f5905600700 1 --2- 172.21.15.31:0/1919505841 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f58e80a3030 0x7f58e80a54f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f59001999c0 tx=0x7f58f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:14.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:14.914+0000 7f58fe000700 1 -- 172.21.15.31:0/1919505841 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f58f8022020 con 0x7f5900107510 2024-03-21T10:37:15.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:15.147+0000 7f5908000700 1 -- 172.21.15.31:0/1919505841 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5900199db0 con 0x7f58e80a3030 2024-03-21T10:37:15.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:15.158+0000 7f58fe000700 1 -- 172.21.15.31:0/1919505841 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340782 (secure 0 0 0) 0x7f5900199db0 con 0x7f58e80a3030 2024-03-21T10:37:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:37:15.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:15.165+0000 7f5908000700 1 -- 172.21.15.31:0/1919505841 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f58e80a3030 msgr2=0x7f58e80a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:15.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:15.165+0000 7f5908000700 1 --2- 172.21.15.31:0/1919505841 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f58e80a3030 0x7f58e80a54f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f59001999c0 tx=0x7f58f4009260 comp rx=0 tx=0).stop 2024-03-21T10:37:15.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:15.165+0000 7f5908000700 1 -- 172.21.15.31:0/1919505841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5900107510 msgr2=0x7f5900198880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:15.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:15.165+0000 7f5908000700 1 --2- 172.21.15.31:0/1919505841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5900107510 0x7f5900198880 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f58f800efc0 tx=0x7f58f800c3a0 comp rx=0 tx=0).stop 2024-03-21T10:37:15.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:15.165+0000 7f5908000700 1 -- 172.21.15.31:0/1919505841 shutdown_connections 2024-03-21T10:37:15.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:15.165+0000 7f5908000700 1 --2- 172.21.15.31:0/1919505841 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f58e80a3030 0x7f58e80a54f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:15.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:15.165+0000 7f5908000700 1 --2- 172.21.15.31:0/1919505841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5900107510 0x7f5900198880 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:15.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:15.165+0000 7f5908000700 1 --2- 172.21.15.31:0/1919505841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5900102440 0x7f590019e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:15.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:15.165+0000 7f5908000700 1 --2- 172.21.15.31:0/1919505841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5900102df0 0x7f590019ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:15.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:15.165+0000 7f5908000700 1 -- 172.21.15.31:0/1919505841 >> 172.21.15.31:0/1919505841 conn(0x7f59000792f0 msgr2=0x7f5900100440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:15.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:15.166+0000 7f5908000700 1 -- 172.21.15.31:0/1919505841 shutdown_connections 2024-03-21T10:37:15.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:15.166+0000 7f5908000700 1 -- 172.21.15.31:0/1919505841 wait complete. 2024-03-21T10:37:15.165 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-03-21T10:37:15.537 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:15 smithi031 ceph-mon[31673]: pgmap v393: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:15.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:15 smithi040 ceph-mon[36154]: pgmap v393: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:15.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:15 smithi070 ceph-mon[36330]: pgmap v393: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:15.989 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":393,"stamp":"2024-03-21T10:37:14.048321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659199,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":701,"num_read_kb":475,"num_write":355,"num_write_kb":3063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":787,"ondisk_log_size":787,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369996,"kb_used_data":12440,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571828,"statfs":{"total":1151940427776,"available":1151561551872,"internally_reserved":0,"allocated":12738560,"data_stored":8186291,"data_compressed":433329,"data_compressed_allocated":589824,"data_compressed_original":2678784,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006031"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619666+0000","last_change":"2024-03-21T10:35:06.923544+0000","last_active":"2024-03-21T10:36:56.619666+0000","last_peered":"2024-03-21T10:36:56.619666+0000","last_clean":"2024-03-21T10:36:56.619666+0000","last_became_active":"2024-03-21T10:35:06.923254+0000","last_became_peered":"2024-03-21T10:35:06.923254+0000","last_unstale":"2024-03-21T10:36:56.619666+0000","last_undegraded":"2024-03-21T10:36:56.619666+0000","last_fullsized":"2024-03-21T10:36:56.619666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:38:14.573590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.622113+0000","last_change":"2024-03-21T10:35:06.929242+0000","last_active":"2024-03-21T10:36:56.622113+0000","last_peered":"2024-03-21T10:36:56.622113+0000","last_clean":"2024-03-21T10:36:56.622113+0000","last_became_active":"2024-03-21T10:35:06.927078+0000","last_became_peered":"2024-03-21T10:35:06.927078+0000","last_unstale":"2024-03-21T10:36:56.622113+0000","last_undegraded":"2024-03-21T10:36:56.622113+0000","last_fullsized":"2024-03-21T10:36:56.622113+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:27:10.347945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621754+0000","last_change":"2024-03-21T10:35:06.927792+0000","last_active":"2024-03-21T10:36:56.621754+0000","last_peered":"2024-03-21T10:36:56.621754+0000","last_clean":"2024-03-21T10:36:56.621754+0000","last_became_active":"2024-03-21T10:35:06.927386+0000","last_became_peered":"2024-03-21T10:35:06.927386+0000","last_unstale":"2024-03-21T10:36:56.621754+0000","last_undegraded":"2024-03-21T10:36:56.621754+0000","last_fullsized":"2024-03-21T10:36:56.621754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:53:51.310283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540864+0000","last_change":"2024-03-21T10:35:06.908901+0000","last_active":"2024-03-21T10:36:56.540864+0000","last_peered":"2024-03-21T10:36:56.540864+0000","last_clean":"2024-03-21T10:36:56.540864+0000","last_became_active":"2024-03-21T10:35:06.908586+0000","last_became_peered":"2024-03-21T10:35:06.908586+0000","last_unstale":"2024-03-21T10:36:56.540864+0000","last_undegraded":"2024-03-21T10:36:56.540864+0000","last_fullsized":"2024-03-21T10:36:56.540864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:25:57.387769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312668+0000","last_change":"2024-03-21T10:35:06.932537+0000","last_active":"2024-03-21T10:36:56.312668+0000","last_peered":"2024-03-21T10:36:56.312668+0000","last_clean":"2024-03-21T10:36:56.312668+0000","last_became_active":"2024-03-21T10:35:06.932185+0000","last_became_peered":"2024-03-21T10:35:06.932185+0000","last_unstale":"2024-03-21T10:36:56.312668+0000","last_undegraded":"2024-03-21T10:36:56.312668+0000","last_fullsized":"2024-03-21T10:36:56.312668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:30:25.127073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311395+0000","last_change":"2024-03-21T10:35:06.922765+0000","last_active":"2024-03-21T10:36:56.311395+0000","last_peered":"2024-03-21T10:36:56.311395+0000","last_clean":"2024-03-21T10:36:56.311395+0000","last_became_active":"2024-03-21T10:35:06.922447+0000","last_became_peered":"2024-03-21T10:35:06.922447+0000","last_unstale":"2024-03-21T10:36:56.311395+0000","last_undegraded":"2024-03-21T10:36:56.311395+0000","last_fullsized":"2024-03-21T10:36:56.311395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:37:33.310735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618972+0000","last_change":"2024-03-21T10:35:06.896777+0000","last_active":"2024-03-21T10:36:56.618972+0000","last_peered":"2024-03-21T10:36:56.618972+0000","last_clean":"2024-03-21T10:36:56.618972+0000","last_became_active":"2024-03-21T10:35:06.896544+0000","last_became_peered":"2024-03-21T10:35:06.896544+0000","last_unstale":"2024-03-21T10:36:56.618972+0000","last_undegraded":"2024-03-21T10:36:56.618972+0000","last_fullsized":"2024-03-21T10:36:56.618972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:34:05.000661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.324782+0000","last_change":"2024-03-21T10:35:06.923294+0000","last_active":"2024-03-21T10:36:56.324782+0000","last_peered":"2024-03-21T10:36:56.324782+0000","last_clean":"2024-03-21T10:36:56.324782+0000","last_became_active":"2024-03-21T10:35:06.921320+0000","last_became_peered":"2024-03-21T10:35:06.921320+0000","last_unstale":"2024-03-21T10:36:56.324782+0000","last_undegraded":"2024-03-21T10:36:56.324782+0000","last_fullsized":"2024-03-21T10:36:56.324782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:33:53.855908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619791+0000","last_change":"2024-03-21T10:35:06.912977+0000","last_active":"2024-03-21T10:36:56.619791+0000","last_peered":"2024-03-21T10:36:56.619791+0000","last_clean":"2024-03-21T10:36:56.619791+0000","last_became_active":"2024-03-21T10:35:06.912626+0000","last_became_peered":"2024-03-21T10:35:06.912626+0000","last_unstale":"2024-03-21T10:36:56.619791+0000","last_undegraded":"2024-03-21T10:36:56.619791+0000","last_fullsized":"2024-03-21T10:36:56.619791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:42:13.472454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311863+0000","last_change":"2024-03-21T10:35:06.910252+0000","last_active":"2024-03-21T10:36:56.311863+0000","last_peered":"2024-03-21T10:36:56.311863+0000","last_clean":"2024-03-21T10:36:56.311863+0000","last_became_active":"2024-03-21T10:35:06.909924+0000","last_became_peered":"2024-03-21T10:35:06.909924+0000","last_unstale":"2024-03-21T10:36:56.311863+0000","last_undegraded":"2024-03-21T10:36:56.311863+0000","last_fullsized":"2024-03-21T10:36:56.311863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:43:15.527349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619675+0000","last_change":"2024-03-21T10:35:06.909569+0000","last_active":"2024-03-21T10:36:56.619675+0000","last_peered":"2024-03-21T10:36:56.619675+0000","last_clean":"2024-03-21T10:36:56.619675+0000","last_became_active":"2024-03-21T10:35:06.908982+0000","last_became_peered":"2024-03-21T10:35:06.908982+0000","last_unstale":"2024-03-21T10:36:56.619675+0000","last_undegraded":"2024-03-21T10:36:56.619675+0000","last_fullsized":"2024-03-21T10:36:56.619675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:13:56.020281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540766+0000","last_change":"2024-03-21T10:35:06.913861+0000","last_active":"2024-03-21T10:36:56.540766+0000","last_peered":"2024-03-21T10:36:56.540766+0000","last_clean":"2024-03-21T10:36:56.540766+0000","last_became_active":"2024-03-21T10:35:06.913490+0000","last_became_peered":"2024-03-21T10:35:06.913490+0000","last_unstale":"2024-03-21T10:36:56.540766+0000","last_undegraded":"2024-03-21T10:36:56.540766+0000","last_fullsized":"2024-03-21T10:36:56.540766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:24:18.573735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312522+0000","last_change":"2024-03-21T10:35:06.934537+0000","last_active":"2024-03-21T10:36:56.312522+0000","last_peered":"2024-03-21T10:36:56.312522+0000","last_clean":"2024-03-21T10:36:56.312522+0000","last_became_active":"2024-03-21T10:35:06.932951+0000","last_became_peered":"2024-03-21T10:35:06.932951+0000","last_unstale":"2024-03-21T10:36:56.312522+0000","last_undegraded":"2024-03-21T10:36:56.312522+0000","last_fullsized":"2024-03-21T10:36:56.312522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:43:17.748434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621126+0000","last_change":"2024-03-21T10:35:06.925495+0000","last_active":"2024-03-21T10:36:56.621126+0000","last_peered":"2024-03-21T10:36:56.621126+0000","last_clean":"2024-03-21T10:36:56.621126+0000","last_became_active":"2024-03-21T10:35:06.924326+0000","last_became_peered":"2024-03-21T10:35:06.924326+0000","last_unstale":"2024-03-21T10:36:56.621126+0000","last_undegraded":"2024-03-21T10:36:56.621126+0000","last_fullsized":"2024-03-21T10:36:56.621126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:27:19.598213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.622171+0000","last_change":"2024-03-21T10:35:06.923695+0000","last_active":"2024-03-21T10:36:56.622171+0000","last_peered":"2024-03-21T10:36:56.622171+0000","last_clean":"2024-03-21T10:36:56.622171+0000","last_became_active":"2024-03-21T10:35:06.919114+0000","last_became_peered":"2024-03-21T10:35:06.919114+0000","last_unstale":"2024-03-21T10:36:56.622171+0000","last_undegraded":"2024-03-21T10:36:56.622171+0000","last_fullsized":"2024-03-21T10:36:56.622171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:42:49.179653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619077+0000","last_change":"2024-03-21T10:35:06.930225+0000","last_active":"2024-03-21T10:36:56.619077+0000","last_peered":"2024-03-21T10:36:56.619077+0000","last_clean":"2024-03-21T10:36:56.619077+0000","last_became_active":"2024-03-21T10:35:06.929905+0000","last_became_peered":"2024-03-21T10:35:06.929905+0000","last_unstale":"2024-03-21T10:36:56.619077+0000","last_undegraded":"2024-03-21T10:36:56.619077+0000","last_fullsized":"2024-03-21T10:36:56.619077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:21:07.231703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619469+0000","last_change":"2024-03-21T10:35:06.923498+0000","last_active":"2024-03-21T10:36:56.619469+0000","last_peered":"2024-03-21T10:36:56.619469+0000","last_clean":"2024-03-21T10:36:56.619469+0000","last_became_active":"2024-03-21T10:35:06.923242+0000","last_became_peered":"2024-03-21T10:35:06.923242+0000","last_unstale":"2024-03-21T10:36:56.619469+0000","last_undegraded":"2024-03-21T10:36:56.619469+0000","last_fullsized":"2024-03-21T10:36:56.619469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:42:04.309701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623751+0000","last_change":"2024-03-21T10:35:06.918314+0000","last_active":"2024-03-21T10:36:56.623751+0000","last_peered":"2024-03-21T10:36:56.623751+0000","last_clean":"2024-03-21T10:36:56.623751+0000","last_became_active":"2024-03-21T10:35:06.917865+0000","last_became_peered":"2024-03-21T10:35:06.917865+0000","last_unstale":"2024-03-21T10:36:56.623751+0000","last_undegraded":"2024-03-21T10:36:56.623751+0000","last_fullsized":"2024-03-21T10:36:56.623751+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:53:48.536341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619252+0000","last_change":"2024-03-21T10:35:06.936234+0000","last_active":"2024-03-21T10:36:56.619252+0000","last_peered":"2024-03-21T10:36:56.619252+0000","last_clean":"2024-03-21T10:36:56.619252+0000","last_became_active":"2024-03-21T10:35:06.935958+0000","last_became_peered":"2024-03-21T10:35:06.935958+0000","last_unstale":"2024-03-21T10:36:56.619252+0000","last_undegraded":"2024-03-21T10:36:56.619252+0000","last_fullsized":"2024-03-21T10:36:56.619252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:52:43.509671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618568+0000","last_change":"2024-03-21T10:35:11.198276+0000","last_active":"2024-03-21T10:36:56.618568+0000","last_peered":"2024-03-21T10:36:56.618568+0000","last_clean":"2024-03-21T10:36:56.618568+0000","last_became_active":"2024-03-21T10:35:11.197899+0000","last_became_peered":"2024-03-21T10:35:11.197899+0000","last_unstale":"2024-03-21T10:36:56.618568+0000","last_undegraded":"2024-03-21T10:36:56.618568+0000","last_fullsized":"2024-03-21T10:36:56.618568+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:17:30.499932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623436+0000","last_change":"2024-03-21T10:35:06.010647+0000","last_active":"2024-03-21T10:36:56.623436+0000","last_peered":"2024-03-21T10:36:56.623436+0000","last_clean":"2024-03-21T10:36:56.623436+0000","last_became_active":"2024-03-21T10:35:06.010239+0000","last_became_peered":"2024-03-21T10:35:06.010239+0000","last_unstale":"2024-03-21T10:36:56.623436+0000","last_undegraded":"2024-03-21T10:36:56.623436+0000","last_fullsized":"2024-03-21T10:36:56.623436+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:35:53.324044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325857+0000","last_change":"2024-03-21T10:35:06.926398+0000","last_active":"2024-03-21T10:36:56.325857+0000","last_peered":"2024-03-21T10:36:56.325857+0000","last_clean":"2024-03-21T10:36:56.325857+0000","last_became_active":"2024-03-21T10:35:06.925904+0000","last_became_peered":"2024-03-21T10:35:06.925904+0000","last_unstale":"2024-03-21T10:36:56.325857+0000","last_undegraded":"2024-03-21T10:36:56.325857+0000","last_fullsized":"2024-03-21T10:36:56.325857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:37:52.811604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540440+0000","last_change":"2024-03-21T10:35:55.503266+0000","last_active":"2024-03-21T10:36:56.540440+0000","last_peered":"2024-03-21T10:36:56.540440+0000","last_clean":"2024-03-21T10:36:56.540440+0000","last_became_active":"2024-03-21T10:35:55.502694+0000","last_became_peered":"2024-03-21T10:35:55.502694+0000","last_unstale":"2024-03-21T10:36:56.540440+0000","last_undegraded":"2024-03-21T10:36:56.540440+0000","last_fullsized":"2024-03-21T10:36:56.540440+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:56:44.480667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325780+0000","last_change":"2024-03-21T10:35:06.022750+0000","last_active":"2024-03-21T10:36:56.325780+0000","last_peered":"2024-03-21T10:36:56.325780+0000","last_clean":"2024-03-21T10:36:56.325780+0000","last_became_active":"2024-03-21T10:35:06.022093+0000","last_became_peered":"2024-03-21T10:35:06.022093+0000","last_unstale":"2024-03-21T10:36:56.325780+0000","last_undegraded":"2024-03-21T10:36:56.325780+0000","last_fullsized":"2024-03-21T10:36:56.325780+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:53:28.458486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.624041+0000","last_change":"2024-03-21T10:35:06.927594+0000","last_active":"2024-03-21T10:36:56.624041+0000","last_peered":"2024-03-21T10:36:56.624041+0000","last_clean":"2024-03-21T10:36:56.624041+0000","last_became_active":"2024-03-21T10:35:06.926868+0000","last_became_peered":"2024-03-21T10:35:06.926868+0000","last_unstale":"2024-03-21T10:36:56.624041+0000","last_undegraded":"2024-03-21T10:36:56.624041+0000","last_fullsized":"2024-03-21T10:36:56.624041+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:07:33.127465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540535+0000","last_change":"2024-03-21T10:35:11.193475+0000","last_active":"2024-03-21T10:36:56.540535+0000","last_peered":"2024-03-21T10:36:56.540535+0000","last_clean":"2024-03-21T10:36:56.540535+0000","last_became_active":"2024-03-21T10:35:11.193054+0000","last_became_peered":"2024-03-21T10:35:11.193054+0000","last_unstale":"2024-03-21T10:36:56.540535+0000","last_undegraded":"2024-03-21T10:36:56.540535+0000","last_fullsized":"2024-03-21T10:36:56.540535+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:32:36.564147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313082+0000","last_change":"2024-03-21T10:35:06.004257+0000","last_active":"2024-03-21T10:36:56.313082+0000","last_peered":"2024-03-21T10:36:56.313082+0000","last_clean":"2024-03-21T10:36:56.313082+0000","last_became_active":"2024-03-21T10:35:06.003936+0000","last_became_peered":"2024-03-21T10:35:06.003936+0000","last_unstale":"2024-03-21T10:36:56.313082+0000","last_undegraded":"2024-03-21T10:36:56.313082+0000","last_fullsized":"2024-03-21T10:36:56.313082+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:48:48.608979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.620100+0000","last_change":"2024-03-21T10:35:06.941681+0000","last_active":"2024-03-21T10:36:56.620100+0000","last_peered":"2024-03-21T10:36:56.620100+0000","last_clean":"2024-03-21T10:36:56.620100+0000","last_became_active":"2024-03-21T10:35:06.941314+0000","last_became_peered":"2024-03-21T10:35:06.941314+0000","last_unstale":"2024-03-21T10:36:56.620100+0000","last_undegraded":"2024-03-21T10:36:56.620100+0000","last_fullsized":"2024-03-21T10:36:56.620100+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:46:09.948922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540433+0000","last_change":"2024-03-21T10:35:11.188472+0000","last_active":"2024-03-21T10:36:56.540433+0000","last_peered":"2024-03-21T10:36:56.540433+0000","last_clean":"2024-03-21T10:36:56.540433+0000","last_became_active":"2024-03-21T10:35:11.188072+0000","last_became_peered":"2024-03-21T10:35:11.188072+0000","last_unstale":"2024-03-21T10:36:56.540433+0000","last_undegraded":"2024-03-21T10:36:56.540433+0000","last_fullsized":"2024-03-21T10:36:56.540433+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:04:16.911334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312976+0000","last_change":"2024-03-21T10:35:05.998493+0000","last_active":"2024-03-21T10:36:56.312976+0000","last_peered":"2024-03-21T10:36:56.312976+0000","last_clean":"2024-03-21T10:36:56.312976+0000","last_became_active":"2024-03-21T10:35:05.998134+0000","last_became_peered":"2024-03-21T10:35:05.998134+0000","last_unstale":"2024-03-21T10:36:56.312976+0000","last_undegraded":"2024-03-21T10:36:56.312976+0000","last_fullsized":"2024-03-21T10:36:56.312976+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:05:32.033328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311421+0000","last_change":"2024-03-21T10:35:06.912973+0000","last_active":"2024-03-21T10:36:56.311421+0000","last_peered":"2024-03-21T10:36:56.311421+0000","last_clean":"2024-03-21T10:36:56.311421+0000","last_became_active":"2024-03-21T10:35:06.912382+0000","last_became_peered":"2024-03-21T10:35:06.912382+0000","last_unstale":"2024-03-21T10:36:56.311421+0000","last_undegraded":"2024-03-21T10:36:56.311421+0000","last_fullsized":"2024-03-21T10:36:56.311421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:13:23.757805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619093+0000","last_change":"2024-03-21T10:35:11.180806+0000","last_active":"2024-03-21T10:36:56.619093+0000","last_peered":"2024-03-21T10:36:56.619093+0000","last_clean":"2024-03-21T10:36:56.619093+0000","last_became_active":"2024-03-21T10:35:11.180303+0000","last_became_peered":"2024-03-21T10:35:11.180303+0000","last_unstale":"2024-03-21T10:36:56.619093+0000","last_undegraded":"2024-03-21T10:36:56.619093+0000","last_fullsized":"2024-03-21T10:36:56.619093+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:27:11.719654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":130,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:37:08.092912+0000","last_change":"2024-03-21T10:35:06.004555+0000","last_active":"2024-03-21T10:37:08.092912+0000","last_peered":"2024-03-21T10:37:08.092912+0000","last_clean":"2024-03-21T10:37:08.092912+0000","last_became_active":"2024-03-21T10:35:06.003926+0000","last_became_peered":"2024-03-21T10:35:06.003926+0000","last_unstale":"2024-03-21T10:37:08.092912+0000","last_undegraded":"2024-03-21T10:37:08.092912+0000","last_fullsized":"2024-03-21T10:37:08.092912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:05:49.721000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":93,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312684+0000","last_change":"2024-03-21T10:35:06.913556+0000","last_active":"2024-03-21T10:36:56.312684+0000","last_peered":"2024-03-21T10:36:56.312684+0000","last_clean":"2024-03-21T10:36:56.312684+0000","last_became_active":"2024-03-21T10:35:06.913213+0000","last_became_peered":"2024-03-21T10:35:06.913213+0000","last_unstale":"2024-03-21T10:36:56.312684+0000","last_undegraded":"2024-03-21T10:36:56.312684+0000","last_fullsized":"2024-03-21T10:36:56.312684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:11:09.235927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619326+0000","last_change":"2024-03-21T10:35:55.523819+0000","last_active":"2024-03-21T10:36:56.619326+0000","last_peered":"2024-03-21T10:36:56.619326+0000","last_clean":"2024-03-21T10:36:56.619326+0000","last_became_active":"2024-03-21T10:35:55.523333+0000","last_became_peered":"2024-03-21T10:35:55.523333+0000","last_unstale":"2024-03-21T10:36:56.619326+0000","last_undegraded":"2024-03-21T10:36:56.619326+0000","last_fullsized":"2024-03-21T10:36:56.619326+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:25:36.277481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311293+0000","last_change":"2024-03-21T10:35:06.012527+0000","last_active":"2024-03-21T10:36:56.311293+0000","last_peered":"2024-03-21T10:36:56.311293+0000","last_clean":"2024-03-21T10:36:56.311293+0000","last_became_active":"2024-03-21T10:35:06.012228+0000","last_became_peered":"2024-03-21T10:35:06.012228+0000","last_unstale":"2024-03-21T10:36:56.311293+0000","last_undegraded":"2024-03-21T10:36:56.311293+0000","last_fullsized":"2024-03-21T10:36:56.311293+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:52:34.622306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.622381+0000","last_change":"2024-03-21T10:35:55.503531+0000","last_active":"2024-03-21T10:36:56.622381+0000","last_peered":"2024-03-21T10:36:56.622381+0000","last_clean":"2024-03-21T10:36:56.622381+0000","last_became_active":"2024-03-21T10:35:55.502793+0000","last_became_peered":"2024-03-21T10:35:55.502793+0000","last_unstale":"2024-03-21T10:36:56.622381+0000","last_undegraded":"2024-03-21T10:36:56.622381+0000","last_fullsized":"2024-03-21T10:36:56.622381+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:08:39.599084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,3],"acting":[9,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619462+0000","last_change":"2024-03-21T10:35:11.196543+0000","last_active":"2024-03-21T10:36:56.619462+0000","last_peered":"2024-03-21T10:36:56.619462+0000","last_clean":"2024-03-21T10:36:56.619462+0000","last_became_active":"2024-03-21T10:35:11.196151+0000","last_became_peered":"2024-03-21T10:35:11.196151+0000","last_unstale":"2024-03-21T10:36:56.619462+0000","last_undegraded":"2024-03-21T10:36:56.619462+0000","last_fullsized":"2024-03-21T10:36:56.619462+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:15:39.193647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312140+0000","last_change":"2024-03-21T10:35:06.004588+0000","last_active":"2024-03-21T10:36:56.312140+0000","last_peered":"2024-03-21T10:36:56.312140+0000","last_clean":"2024-03-21T10:36:56.312140+0000","last_became_active":"2024-03-21T10:35:06.004190+0000","last_became_peered":"2024-03-21T10:35:06.004190+0000","last_unstale":"2024-03-21T10:36:56.312140+0000","last_undegraded":"2024-03-21T10:36:56.312140+0000","last_fullsized":"2024-03-21T10:36:56.312140+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:32:18.563692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325018+0000","last_change":"2024-03-21T10:35:06.941738+0000","last_active":"2024-03-21T10:36:56.325018+0000","last_peered":"2024-03-21T10:36:56.325018+0000","last_clean":"2024-03-21T10:36:56.325018+0000","last_became_active":"2024-03-21T10:35:06.941386+0000","last_became_peered":"2024-03-21T10:35:06.941386+0000","last_unstale":"2024-03-21T10:36:56.325018+0000","last_undegraded":"2024-03-21T10:36:56.325018+0000","last_fullsized":"2024-03-21T10:36:56.325018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:52:55.519095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621671+0000","last_change":"2024-03-21T10:35:11.199370+0000","last_active":"2024-03-21T10:36:56.621671+0000","last_peered":"2024-03-21T10:36:56.621671+0000","last_clean":"2024-03-21T10:36:56.621671+0000","last_became_active":"2024-03-21T10:35:11.198598+0000","last_became_peered":"2024-03-21T10:35:11.198598+0000","last_unstale":"2024-03-21T10:36:56.621671+0000","last_undegraded":"2024-03-21T10:36:56.621671+0000","last_fullsized":"2024-03-21T10:36:56.621671+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:07:19.419405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311161+0000","last_change":"2024-03-21T10:35:06.004893+0000","last_active":"2024-03-21T10:36:56.311161+0000","last_peered":"2024-03-21T10:36:56.311161+0000","last_clean":"2024-03-21T10:36:56.311161+0000","last_became_active":"2024-03-21T10:35:06.004586+0000","last_became_peered":"2024-03-21T10:35:06.004586+0000","last_unstale":"2024-03-21T10:36:56.311161+0000","last_undegraded":"2024-03-21T10:36:56.311161+0000","last_fullsized":"2024-03-21T10:36:56.311161+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:05:29.106575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621244+0000","last_change":"2024-03-21T10:35:06.924448+0000","last_active":"2024-03-21T10:36:56.621244+0000","last_peered":"2024-03-21T10:36:56.621244+0000","last_clean":"2024-03-21T10:36:56.621244+0000","last_became_active":"2024-03-21T10:35:06.924049+0000","last_became_peered":"2024-03-21T10:35:06.924049+0000","last_unstale":"2024-03-21T10:36:56.621244+0000","last_undegraded":"2024-03-21T10:36:56.621244+0000","last_fullsized":"2024-03-21T10:36:56.621244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:23:12.070723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.324956+0000","last_change":"2024-03-21T10:35:11.188691+0000","last_active":"2024-03-21T10:36:56.324956+0000","last_peered":"2024-03-21T10:36:56.324956+0000","last_clean":"2024-03-21T10:36:56.324956+0000","last_became_active":"2024-03-21T10:35:11.188138+0000","last_became_peered":"2024-03-21T10:35:11.188138+0000","last_unstale":"2024-03-21T10:36:56.324956+0000","last_undegraded":"2024-03-21T10:36:56.324956+0000","last_fullsized":"2024-03-21T10:36:56.324956+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:51:50.920920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":125,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:37:09.093539+0000","last_change":"2024-03-21T10:35:06.015276+0000","last_active":"2024-03-21T10:37:09.093539+0000","last_peered":"2024-03-21T10:37:09.093539+0000","last_clean":"2024-03-21T10:37:09.093539+0000","last_became_active":"2024-03-21T10:35:06.014734+0000","last_became_peered":"2024-03-21T10:35:06.014734+0000","last_unstale":"2024-03-21T10:37:09.093539+0000","last_undegraded":"2024-03-21T10:37:09.093539+0000","last_fullsized":"2024-03-21T10:37:09.093539+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:18:45.905451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":90,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.620294+0000","last_change":"2024-03-21T10:35:06.926689+0000","last_active":"2024-03-21T10:36:56.620294+0000","last_peered":"2024-03-21T10:36:56.620294+0000","last_clean":"2024-03-21T10:36:56.620294+0000","last_became_active":"2024-03-21T10:35:06.926319+0000","last_became_peered":"2024-03-21T10:35:06.926319+0000","last_unstale":"2024-03-21T10:36:56.620294+0000","last_undegraded":"2024-03-21T10:36:56.620294+0000","last_fullsized":"2024-03-21T10:36:56.620294+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:06:22.371608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311047+0000","last_change":"2024-03-21T10:35:55.383776+0000","last_active":"2024-03-21T10:36:56.311047+0000","last_peered":"2024-03-21T10:36:56.311047+0000","last_clean":"2024-03-21T10:36:56.311047+0000","last_became_active":"2024-03-21T10:35:55.383300+0000","last_became_peered":"2024-03-21T10:35:55.383300+0000","last_unstale":"2024-03-21T10:36:56.311047+0000","last_undegraded":"2024-03-21T10:36:56.311047+0000","last_fullsized":"2024-03-21T10:36:56.311047+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:55:56.244325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2,4],"acting":[10,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619425+0000","last_change":"2024-03-21T10:35:06.007231+0000","last_active":"2024-03-21T10:36:56.619425+0000","last_peered":"2024-03-21T10:36:56.619425+0000","last_clean":"2024-03-21T10:36:56.619425+0000","last_became_active":"2024-03-21T10:35:06.006875+0000","last_became_peered":"2024-03-21T10:35:06.006875+0000","last_unstale":"2024-03-21T10:36:56.619425+0000","last_undegraded":"2024-03-21T10:36:56.619425+0000","last_fullsized":"2024-03-21T10:36:56.619425+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:08:02.489741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.624205+0000","last_change":"2024-03-21T10:35:55.524107+0000","last_active":"2024-03-21T10:36:56.624205+0000","last_peered":"2024-03-21T10:36:56.624205+0000","last_clean":"2024-03-21T10:36:56.624205+0000","last_became_active":"2024-03-21T10:35:55.523523+0000","last_became_peered":"2024-03-21T10:35:55.523523+0000","last_unstale":"2024-03-21T10:36:56.624205+0000","last_undegraded":"2024-03-21T10:36:56.624205+0000","last_fullsized":"2024-03-21T10:36:56.624205+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:49:29.121279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540676+0000","last_change":"2024-03-21T10:35:11.193382+0000","last_active":"2024-03-21T10:36:56.540676+0000","last_peered":"2024-03-21T10:36:56.540676+0000","last_clean":"2024-03-21T10:36:56.540676+0000","last_became_active":"2024-03-21T10:35:11.192959+0000","last_became_peered":"2024-03-21T10:35:11.192959+0000","last_unstale":"2024-03-21T10:36:56.540676+0000","last_undegraded":"2024-03-21T10:36:56.540676+0000","last_fullsized":"2024-03-21T10:36:56.540676+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:02:57.187370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311322+0000","last_change":"2024-03-21T10:35:06.019969+0000","last_active":"2024-03-21T10:36:56.311322+0000","last_peered":"2024-03-21T10:36:56.311322+0000","last_clean":"2024-03-21T10:36:56.311322+0000","last_became_active":"2024-03-21T10:35:06.019648+0000","last_became_peered":"2024-03-21T10:35:06.019648+0000","last_unstale":"2024-03-21T10:36:56.311322+0000","last_undegraded":"2024-03-21T10:36:56.311322+0000","last_fullsized":"2024-03-21T10:36:56.311322+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:24:02.227205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.620236+0000","last_change":"2024-03-21T10:35:06.910294+0000","last_active":"2024-03-21T10:36:56.620236+0000","last_peered":"2024-03-21T10:36:56.620236+0000","last_clean":"2024-03-21T10:36:56.620236+0000","last_became_active":"2024-03-21T10:35:06.909862+0000","last_became_peered":"2024-03-21T10:35:06.909862+0000","last_unstale":"2024-03-21T10:36:56.620236+0000","last_undegraded":"2024-03-21T10:36:56.620236+0000","last_fullsized":"2024-03-21T10:36:56.620236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:37:13.396003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621461+0000","last_change":"2024-03-21T10:35:11.196398+0000","last_active":"2024-03-21T10:36:56.621461+0000","last_peered":"2024-03-21T10:36:56.621461+0000","last_clean":"2024-03-21T10:36:56.621461+0000","last_became_active":"2024-03-21T10:35:11.194965+0000","last_became_peered":"2024-03-21T10:35:11.194965+0000","last_unstale":"2024-03-21T10:36:56.621461+0000","last_undegraded":"2024-03-21T10:36:56.621461+0000","last_fullsized":"2024-03-21T10:36:56.621461+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:48:22.024137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'36","reported_seq":257,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:37:10.093962+0000","last_change":"2024-03-21T10:35:06.005037+0000","last_active":"2024-03-21T10:37:10.093962+0000","last_peered":"2024-03-21T10:37:10.093962+0000","last_clean":"2024-03-21T10:37:10.093962+0000","last_became_active":"2024-03-21T10:35:06.004686+0000","last_became_peered":"2024-03-21T10:35:06.004686+0000","last_unstale":"2024-03-21T10:37:10.093962+0000","last_undegraded":"2024-03-21T10:37:10.093962+0000","last_fullsized":"2024-03-21T10:37:10.093962+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:24:13.312112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64455,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":190,"num_read_kb":159,"num_write":37,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312911+0000","last_change":"2024-03-21T10:35:06.906525+0000","last_active":"2024-03-21T10:36:56.312911+0000","last_peered":"2024-03-21T10:36:56.312911+0000","last_clean":"2024-03-21T10:36:56.312911+0000","last_became_active":"2024-03-21T10:35:06.905150+0000","last_became_peered":"2024-03-21T10:35:06.905150+0000","last_unstale":"2024-03-21T10:36:56.312911+0000","last_undegraded":"2024-03-21T10:36:56.312911+0000","last_fullsized":"2024-03-21T10:36:56.312911+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:46:27.958779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311678+0000","last_change":"2024-03-21T10:35:11.193519+0000","last_active":"2024-03-21T10:36:56.311678+0000","last_peered":"2024-03-21T10:36:56.311678+0000","last_clean":"2024-03-21T10:36:56.311678+0000","last_became_active":"2024-03-21T10:35:11.193043+0000","last_became_peered":"2024-03-21T10:35:11.193043+0000","last_unstale":"2024-03-21T10:36:56.311678+0000","last_undegraded":"2024-03-21T10:36:56.311678+0000","last_fullsized":"2024-03-21T10:36:56.311678+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:23:20.984472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618921+0000","last_change":"2024-03-21T10:35:05.999220+0000","last_active":"2024-03-21T10:36:56.618921+0000","last_peered":"2024-03-21T10:36:56.618921+0000","last_clean":"2024-03-21T10:36:56.618921+0000","last_became_active":"2024-03-21T10:35:05.998918+0000","last_became_peered":"2024-03-21T10:35:05.998918+0000","last_unstale":"2024-03-21T10:36:56.618921+0000","last_undegraded":"2024-03-21T10:36:56.618921+0000","last_fullsized":"2024-03-21T10:36:56.618921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:01:09.816758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312420+0000","last_change":"2024-03-21T10:35:06.941234+0000","last_active":"2024-03-21T10:36:56.312420+0000","last_peered":"2024-03-21T10:36:56.312420+0000","last_clean":"2024-03-21T10:36:56.312420+0000","last_became_active":"2024-03-21T10:35:06.940828+0000","last_became_peered":"2024-03-21T10:35:06.940828+0000","last_unstale":"2024-03-21T10:36:56.312420+0000","last_undegraded":"2024-03-21T10:36:56.312420+0000","last_fullsized":"2024-03-21T10:36:56.312420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:50:34.398822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618764+0000","last_change":"2024-03-21T10:35:11.196007+0000","last_active":"2024-03-21T10:36:56.618764+0000","last_peered":"2024-03-21T10:36:56.618764+0000","last_clean":"2024-03-21T10:36:56.618764+0000","last_became_active":"2024-03-21T10:35:11.195609+0000","last_became_peered":"2024-03-21T10:35:11.195609+0000","last_unstale":"2024-03-21T10:36:56.618764+0000","last_undegraded":"2024-03-21T10:36:56.618764+0000","last_fullsized":"2024-03-21T10:36:56.618764+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:21:38.803390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311467+0000","last_change":"2024-03-21T10:35:06.011594+0000","last_active":"2024-03-21T10:36:56.311467+0000","last_peered":"2024-03-21T10:36:56.311467+0000","last_clean":"2024-03-21T10:36:56.311467+0000","last_became_active":"2024-03-21T10:35:06.011254+0000","last_became_peered":"2024-03-21T10:35:06.011254+0000","last_unstale":"2024-03-21T10:36:56.311467+0000","last_undegraded":"2024-03-21T10:36:56.311467+0000","last_fullsized":"2024-03-21T10:36:56.311467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:52:31.045659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619892+0000","last_change":"2024-03-21T10:35:06.929170+0000","last_active":"2024-03-21T10:36:56.619892+0000","last_peered":"2024-03-21T10:36:56.619892+0000","last_clean":"2024-03-21T10:36:56.619892+0000","last_became_active":"2024-03-21T10:35:06.928822+0000","last_became_peered":"2024-03-21T10:35:06.928822+0000","last_unstale":"2024-03-21T10:36:56.619892+0000","last_undegraded":"2024-03-21T10:36:56.619892+0000","last_fullsized":"2024-03-21T10:36:56.619892+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:37:40.328453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.622940+0000","last_change":"2024-03-21T10:35:11.204285+0000","last_active":"2024-03-21T10:36:56.622940+0000","last_peered":"2024-03-21T10:36:56.622940+0000","last_clean":"2024-03-21T10:36:56.622940+0000","last_became_active":"2024-03-21T10:35:11.203698+0000","last_became_peered":"2024-03-21T10:35:11.203698+0000","last_unstale":"2024-03-21T10:36:56.622940+0000","last_undegraded":"2024-03-21T10:36:56.622940+0000","last_fullsized":"2024-03-21T10:36:56.622940+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:03:32.732488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'3","reported_seq":124,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:37:08.131351+0000","last_change":"2024-03-21T10:35:06.005028+0000","last_active":"2024-03-21T10:37:08.131351+0000","last_peered":"2024-03-21T10:37:08.131351+0000","last_clean":"2024-03-21T10:37:08.131351+0000","last_became_active":"2024-03-21T10:35:06.004670+0000","last_became_peered":"2024-03-21T10:35:06.004670+0000","last_unstale":"2024-03-21T10:37:08.131351+0000","last_undegraded":"2024-03-21T10:37:08.131351+0000","last_fullsized":"2024-03-21T10:37:08.131351+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:00:18.481051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619489+0000","last_change":"2024-03-21T10:35:06.911696+0000","last_active":"2024-03-21T10:36:56.619489+0000","last_peered":"2024-03-21T10:36:56.619489+0000","last_clean":"2024-03-21T10:36:56.619489+0000","last_became_active":"2024-03-21T10:35:06.911231+0000","last_became_peered":"2024-03-21T10:35:06.911231+0000","last_unstale":"2024-03-21T10:36:56.619489+0000","last_undegraded":"2024-03-21T10:36:56.619489+0000","last_fullsized":"2024-03-21T10:36:56.619489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:21:51.540993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312860+0000","last_change":"2024-03-21T10:35:11.204422+0000","last_active":"2024-03-21T10:36:56.312860+0000","last_peered":"2024-03-21T10:36:56.312860+0000","last_clean":"2024-03-21T10:36:56.312860+0000","last_became_active":"2024-03-21T10:35:11.203904+0000","last_became_peered":"2024-03-21T10:35:11.203904+0000","last_unstale":"2024-03-21T10:36:56.312860+0000","last_undegraded":"2024-03-21T10:36:56.312860+0000","last_fullsized":"2024-03-21T10:36:56.312860+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:38:14.486012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313109+0000","last_change":"2024-03-21T10:35:06.019823+0000","last_active":"2024-03-21T10:36:56.313109+0000","last_peered":"2024-03-21T10:36:56.313109+0000","last_clean":"2024-03-21T10:36:56.313109+0000","last_became_active":"2024-03-21T10:35:06.019502+0000","last_became_peered":"2024-03-21T10:35:06.019502+0000","last_unstale":"2024-03-21T10:36:56.313109+0000","last_undegraded":"2024-03-21T10:36:56.313109+0000","last_fullsized":"2024-03-21T10:36:56.313109+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:33:22.438868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311184+0000","last_change":"2024-03-21T10:35:06.906671+0000","last_active":"2024-03-21T10:36:56.311184+0000","last_peered":"2024-03-21T10:36:56.311184+0000","last_clean":"2024-03-21T10:36:56.311184+0000","last_became_active":"2024-03-21T10:35:06.905873+0000","last_became_peered":"2024-03-21T10:35:06.905873+0000","last_unstale":"2024-03-21T10:36:56.311184+0000","last_undegraded":"2024-03-21T10:36:56.311184+0000","last_fullsized":"2024-03-21T10:36:56.311184+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:14:59.304255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311141+0000","last_change":"2024-03-21T10:35:11.197698+0000","last_active":"2024-03-21T10:36:56.311141+0000","last_peered":"2024-03-21T10:36:56.311141+0000","last_clean":"2024-03-21T10:36:56.311141+0000","last_became_active":"2024-03-21T10:35:11.197295+0000","last_became_peered":"2024-03-21T10:35:11.197295+0000","last_unstale":"2024-03-21T10:36:56.311141+0000","last_undegraded":"2024-03-21T10:36:56.311141+0000","last_fullsized":"2024-03-21T10:36:56.311141+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:02:24.338317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313163+0000","last_change":"2024-03-21T10:35:06.000951+0000","last_active":"2024-03-21T10:36:56.313163+0000","last_peered":"2024-03-21T10:36:56.313163+0000","last_clean":"2024-03-21T10:36:56.313163+0000","last_became_active":"2024-03-21T10:35:05.999169+0000","last_became_peered":"2024-03-21T10:35:05.999169+0000","last_unstale":"2024-03-21T10:36:56.313163+0000","last_undegraded":"2024-03-21T10:36:56.313163+0000","last_fullsized":"2024-03-21T10:36:56.313163+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:31:25.602621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619292+0000","last_change":"2024-03-21T10:35:06.908067+0000","last_active":"2024-03-21T10:36:56.619292+0000","last_peered":"2024-03-21T10:36:56.619292+0000","last_clean":"2024-03-21T10:36:56.619292+0000","last_became_active":"2024-03-21T10:35:06.907720+0000","last_became_peered":"2024-03-21T10:35:06.907720+0000","last_unstale":"2024-03-21T10:36:56.619292+0000","last_undegraded":"2024-03-21T10:36:56.619292+0000","last_fullsized":"2024-03-21T10:36:56.619292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:16:00.502637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312681+0000","last_change":"2024-03-21T10:35:11.229470+0000","last_active":"2024-03-21T10:36:56.312681+0000","last_peered":"2024-03-21T10:36:56.312681+0000","last_clean":"2024-03-21T10:36:56.312681+0000","last_became_active":"2024-03-21T10:35:11.228977+0000","last_became_peered":"2024-03-21T10:35:11.228977+0000","last_unstale":"2024-03-21T10:36:56.312681+0000","last_undegraded":"2024-03-21T10:36:56.312681+0000","last_fullsized":"2024-03-21T10:36:56.312681+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:05:52.269712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623076+0000","last_change":"2024-03-21T10:35:06.019866+0000","last_active":"2024-03-21T10:36:56.623076+0000","last_peered":"2024-03-21T10:36:56.623076+0000","last_clean":"2024-03-21T10:36:56.623076+0000","last_became_active":"2024-03-21T10:35:06.019550+0000","last_became_peered":"2024-03-21T10:35:06.019550+0000","last_unstale":"2024-03-21T10:36:56.623076+0000","last_undegraded":"2024-03-21T10:36:56.623076+0000","last_fullsized":"2024-03-21T10:36:56.623076+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:51:46.989647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311661+0000","last_change":"2024-03-21T10:35:55.385617+0000","last_active":"2024-03-21T10:36:56.311661+0000","last_peered":"2024-03-21T10:36:56.311661+0000","last_clean":"2024-03-21T10:36:56.311661+0000","last_became_active":"2024-03-21T10:35:55.385293+0000","last_became_peered":"2024-03-21T10:35:55.385293+0000","last_unstale":"2024-03-21T10:36:56.311661+0000","last_undegraded":"2024-03-21T10:36:56.311661+0000","last_fullsized":"2024-03-21T10:36:56.311661+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:18:08.200169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313017+0000","last_change":"2024-03-21T10:35:11.199714+0000","last_active":"2024-03-21T10:36:56.313017+0000","last_peered":"2024-03-21T10:36:56.313017+0000","last_clean":"2024-03-21T10:36:56.313017+0000","last_became_active":"2024-03-21T10:35:11.199277+0000","last_became_peered":"2024-03-21T10:35:11.199277+0000","last_unstale":"2024-03-21T10:36:56.313017+0000","last_undegraded":"2024-03-21T10:36:56.313017+0000","last_fullsized":"2024-03-21T10:36:56.313017+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:58:01.471162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619799+0000","last_change":"2024-03-21T10:35:06.015090+0000","last_active":"2024-03-21T10:36:56.619799+0000","last_peered":"2024-03-21T10:36:56.619799+0000","last_clean":"2024-03-21T10:36:56.619799+0000","last_became_active":"2024-03-21T10:35:06.014666+0000","last_became_peered":"2024-03-21T10:35:06.014666+0000","last_unstale":"2024-03-21T10:36:56.619799+0000","last_undegraded":"2024-03-21T10:36:56.619799+0000","last_fullsized":"2024-03-21T10:36:56.619799+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:41:17.608069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311329+0000","last_change":"2024-03-21T10:35:55.388548+0000","last_active":"2024-03-21T10:36:56.311329+0000","last_peered":"2024-03-21T10:36:56.311329+0000","last_clean":"2024-03-21T10:36:56.311329+0000","last_became_active":"2024-03-21T10:35:55.388253+0000","last_became_peered":"2024-03-21T10:35:55.388253+0000","last_unstale":"2024-03-21T10:36:56.311329+0000","last_undegraded":"2024-03-21T10:36:56.311329+0000","last_fullsized":"2024-03-21T10:36:56.311329+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:56.678224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312617+0000","last_change":"2024-03-21T10:35:11.230677+0000","last_active":"2024-03-21T10:36:56.312617+0000","last_peered":"2024-03-21T10:36:56.312617+0000","last_clean":"2024-03-21T10:36:56.312617+0000","last_became_active":"2024-03-21T10:35:11.230087+0000","last_became_peered":"2024-03-21T10:35:11.230087+0000","last_unstale":"2024-03-21T10:36:56.312617+0000","last_undegraded":"2024-03-21T10:36:56.312617+0000","last_fullsized":"2024-03-21T10:36:56.312617+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:39:47.663950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311339+0000","last_change":"2024-03-21T10:35:06.904287+0000","last_active":"2024-03-21T10:36:56.311339+0000","last_peered":"2024-03-21T10:36:56.311339+0000","last_clean":"2024-03-21T10:36:56.311339+0000","last_became_active":"2024-03-21T10:35:06.903821+0000","last_became_peered":"2024-03-21T10:35:06.903821+0000","last_unstale":"2024-03-21T10:36:56.311339+0000","last_undegraded":"2024-03-21T10:36:56.311339+0000","last_fullsized":"2024-03-21T10:36:56.311339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:36:25.362203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":124,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:37:08.130733+0000","last_change":"2024-03-21T10:35:06.014432+0000","last_active":"2024-03-21T10:37:08.130733+0000","last_peered":"2024-03-21T10:37:08.130733+0000","last_clean":"2024-03-21T10:37:08.130733+0000","last_became_active":"2024-03-21T10:35:06.013448+0000","last_became_peered":"2024-03-21T10:35:06.013448+0000","last_unstale":"2024-03-21T10:37:08.130733+0000","last_undegraded":"2024-03-21T10:37:08.130733+0000","last_fullsized":"2024-03-21T10:37:08.130733+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:29:10.600794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":89,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619058+0000","last_change":"2024-03-21T10:35:11.193577+0000","last_active":"2024-03-21T10:36:56.619058+0000","last_peered":"2024-03-21T10:36:56.619058+0000","last_clean":"2024-03-21T10:36:56.619058+0000","last_became_active":"2024-03-21T10:35:11.193132+0000","last_became_peered":"2024-03-21T10:35:11.193132+0000","last_unstale":"2024-03-21T10:36:56.619058+0000","last_undegraded":"2024-03-21T10:36:56.619058+0000","last_fullsized":"2024-03-21T10:36:56.619058+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:40:31.100489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325105+0000","last_change":"2024-03-21T10:35:06.924376+0000","last_active":"2024-03-21T10:36:56.325105+0000","last_peered":"2024-03-21T10:36:56.325105+0000","last_clean":"2024-03-21T10:36:56.325105+0000","last_became_active":"2024-03-21T10:35:06.923874+0000","last_became_peered":"2024-03-21T10:35:06.923874+0000","last_unstale":"2024-03-21T10:36:56.325105+0000","last_undegraded":"2024-03-21T10:36:56.325105+0000","last_fullsized":"2024-03-21T10:36:56.325105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:56:12.620667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540608+0000","last_change":"2024-03-21T10:35:06.015222+0000","last_active":"2024-03-21T10:36:56.540608+0000","last_peered":"2024-03-21T10:36:56.540608+0000","last_clean":"2024-03-21T10:36:56.540608+0000","last_became_active":"2024-03-21T10:35:06.014858+0000","last_became_peered":"2024-03-21T10:35:06.014858+0000","last_unstale":"2024-03-21T10:36:56.540608+0000","last_undegraded":"2024-03-21T10:36:56.540608+0000","last_fullsized":"2024-03-21T10:36:56.540608+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:45:12.528851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":366,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618752+0000","last_change":"2024-03-21T10:33:44.536069+0000","last_active":"2024-03-21T10:36:56.618752+0000","last_peered":"2024-03-21T10:36:56.618752+0000","last_clean":"2024-03-21T10:36:56.618752+0000","last_became_active":"2024-03-21T10:33:44.535623+0000","last_became_peered":"2024-03-21T10:33:44.535623+0000","last_unstale":"2024-03-21T10:36:56.618752+0000","last_undegraded":"2024-03-21T10:36:56.618752+0000","last_fullsized":"2024-03-21T10:36:56.618752+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:29:44.554765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:29:44.554765+0000","last_clean_scrub_stamp":"2024-03-21T10:29:44.554765+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:43:23.440837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311600+0000","last_change":"2024-03-21T10:35:11.229580+0000","last_active":"2024-03-21T10:36:56.311600+0000","last_peered":"2024-03-21T10:36:56.311600+0000","last_clean":"2024-03-21T10:36:56.311600+0000","last_became_active":"2024-03-21T10:35:11.229044+0000","last_became_peered":"2024-03-21T10:35:11.229044+0000","last_unstale":"2024-03-21T10:36:56.311600+0000","last_undegraded":"2024-03-21T10:36:56.311600+0000","last_fullsized":"2024-03-21T10:36:56.311600+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:54:50.535253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311965+0000","last_change":"2024-03-21T10:35:06.927690+0000","last_active":"2024-03-21T10:36:56.311965+0000","last_peered":"2024-03-21T10:36:56.311965+0000","last_clean":"2024-03-21T10:36:56.311965+0000","last_became_active":"2024-03-21T10:35:06.927304+0000","last_became_peered":"2024-03-21T10:35:06.927304+0000","last_unstale":"2024-03-21T10:36:56.311965+0000","last_undegraded":"2024-03-21T10:36:56.311965+0000","last_fullsized":"2024-03-21T10:36:56.311965+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:30:48.950459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619515+0000","last_change":"2024-03-21T10:35:06.004952+0000","last_active":"2024-03-21T10:36:56.619515+0000","last_peered":"2024-03-21T10:36:56.619515+0000","last_clean":"2024-03-21T10:36:56.619515+0000","last_became_active":"2024-03-21T10:35:06.004206+0000","last_became_peered":"2024-03-21T10:35:06.004206+0000","last_unstale":"2024-03-21T10:36:56.619515+0000","last_undegraded":"2024-03-21T10:36:56.619515+0000","last_fullsized":"2024-03-21T10:36:56.619515+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:20:29.198715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619011+0000","last_change":"2024-03-21T10:35:11.179125+0000","last_active":"2024-03-21T10:36:56.619011+0000","last_peered":"2024-03-21T10:36:56.619011+0000","last_clean":"2024-03-21T10:36:56.619011+0000","last_became_active":"2024-03-21T10:35:11.178658+0000","last_became_peered":"2024-03-21T10:35:11.178658+0000","last_unstale":"2024-03-21T10:36:56.619011+0000","last_undegraded":"2024-03-21T10:36:56.619011+0000","last_fullsized":"2024-03-21T10:36:56.619011+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:33:25.281721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313030+0000","last_change":"2024-03-21T10:35:06.019996+0000","last_active":"2024-03-21T10:36:56.313030+0000","last_peered":"2024-03-21T10:36:56.313030+0000","last_clean":"2024-03-21T10:36:56.313030+0000","last_became_active":"2024-03-21T10:35:06.019624+0000","last_became_peered":"2024-03-21T10:35:06.019624+0000","last_unstale":"2024-03-21T10:36:56.313030+0000","last_undegraded":"2024-03-21T10:36:56.313030+0000","last_fullsized":"2024-03-21T10:36:56.313030+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:31:09.551433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313425+0000","last_change":"2024-03-21T10:35:06.939324+0000","last_active":"2024-03-21T10:36:56.313425+0000","last_peered":"2024-03-21T10:36:56.313425+0000","last_clean":"2024-03-21T10:36:56.313425+0000","last_became_active":"2024-03-21T10:35:06.937750+0000","last_became_peered":"2024-03-21T10:35:06.937750+0000","last_unstale":"2024-03-21T10:36:56.313425+0000","last_undegraded":"2024-03-21T10:36:56.313425+0000","last_fullsized":"2024-03-21T10:36:56.313425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:45:14.600863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.324947+0000","last_change":"2024-03-21T10:35:11.229479+0000","last_active":"2024-03-21T10:36:56.324947+0000","last_peered":"2024-03-21T10:36:56.324947+0000","last_clean":"2024-03-21T10:36:56.324947+0000","last_became_active":"2024-03-21T10:35:11.229005+0000","last_became_peered":"2024-03-21T10:35:11.229005+0000","last_unstale":"2024-03-21T10:36:56.324947+0000","last_undegraded":"2024-03-21T10:36:56.324947+0000","last_fullsized":"2024-03-21T10:36:56.324947+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:39:19.859581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'11","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623558+0000","last_change":"2024-03-21T10:35:06.013744+0000","last_active":"2024-03-21T10:36:56.623558+0000","last_peered":"2024-03-21T10:36:56.623558+0000","last_clean":"2024-03-21T10:36:56.623558+0000","last_became_active":"2024-03-21T10:35:06.013509+0000","last_became_peered":"2024-03-21T10:35:06.013509+0000","last_unstale":"2024-03-21T10:36:56.623558+0000","last_undegraded":"2024-03-21T10:36:56.623558+0000","last_fullsized":"2024-03-21T10:36:56.623558+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:51:45.000933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325882+0000","last_change":"2024-03-21T10:35:06.930685+0000","last_active":"2024-03-21T10:36:56.325882+0000","last_peered":"2024-03-21T10:36:56.325882+0000","last_clean":"2024-03-21T10:36:56.325882+0000","last_became_active":"2024-03-21T10:35:06.929775+0000","last_became_peered":"2024-03-21T10:35:06.929775+0000","last_unstale":"2024-03-21T10:36:56.325882+0000","last_undegraded":"2024-03-21T10:36:56.325882+0000","last_fullsized":"2024-03-21T10:36:56.325882+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:13:18.058752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618559+0000","last_change":"2024-03-21T10:35:11.177061+0000","last_active":"2024-03-21T10:36:56.618559+0000","last_peered":"2024-03-21T10:36:56.618559+0000","last_clean":"2024-03-21T10:36:56.618559+0000","last_became_active":"2024-03-21T10:35:11.176603+0000","last_became_peered":"2024-03-21T10:35:11.176603+0000","last_unstale":"2024-03-21T10:36:56.618559+0000","last_undegraded":"2024-03-21T10:36:56.618559+0000","last_fullsized":"2024-03-21T10:36:56.618559+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:54.532780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540908+0000","last_change":"2024-03-21T10:35:06.009018+0000","last_active":"2024-03-21T10:36:56.540908+0000","last_peered":"2024-03-21T10:36:56.540908+0000","last_clean":"2024-03-21T10:36:56.540908+0000","last_became_active":"2024-03-21T10:35:06.008638+0000","last_became_peered":"2024-03-21T10:35:06.008638+0000","last_unstale":"2024-03-21T10:36:56.540908+0000","last_undegraded":"2024-03-21T10:36:56.540908+0000","last_fullsized":"2024-03-21T10:36:56.540908+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:42:19.527898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621871+0000","last_change":"2024-03-21T10:35:06.931588+0000","last_active":"2024-03-21T10:36:56.621871+0000","last_peered":"2024-03-21T10:36:56.621871+0000","last_clean":"2024-03-21T10:36:56.621871+0000","last_became_active":"2024-03-21T10:35:06.930707+0000","last_became_peered":"2024-03-21T10:35:06.930707+0000","last_unstale":"2024-03-21T10:36:56.621871+0000","last_undegraded":"2024-03-21T10:36:56.621871+0000","last_fullsized":"2024-03-21T10:36:56.621871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:44:55.862799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540757+0000","last_change":"2024-03-21T10:35:11.229408+0000","last_active":"2024-03-21T10:36:56.540757+0000","last_peered":"2024-03-21T10:36:56.540757+0000","last_clean":"2024-03-21T10:36:56.540757+0000","last_became_active":"2024-03-21T10:35:11.228997+0000","last_became_peered":"2024-03-21T10:35:11.228997+0000","last_unstale":"2024-03-21T10:36:56.540757+0000","last_undegraded":"2024-03-21T10:36:56.540757+0000","last_fullsized":"2024-03-21T10:36:56.540757+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:56:11.517597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619202+0000","last_change":"2024-03-21T10:35:55.388978+0000","last_active":"2024-03-21T10:36:56.619202+0000","last_peered":"2024-03-21T10:36:56.619202+0000","last_clean":"2024-03-21T10:36:56.619202+0000","last_became_active":"2024-03-21T10:35:55.383048+0000","last_became_peered":"2024-03-21T10:35:55.383048+0000","last_unstale":"2024-03-21T10:36:56.619202+0000","last_undegraded":"2024-03-21T10:36:56.619202+0000","last_fullsized":"2024-03-21T10:36:56.619202+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:24:01.933115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312584+0000","last_change":"2024-03-21T10:35:06.908585+0000","last_active":"2024-03-21T10:36:56.312584+0000","last_peered":"2024-03-21T10:36:56.312584+0000","last_clean":"2024-03-21T10:36:56.312584+0000","last_became_active":"2024-03-21T10:35:06.908253+0000","last_became_peered":"2024-03-21T10:35:06.908253+0000","last_unstale":"2024-03-21T10:36:56.312584+0000","last_undegraded":"2024-03-21T10:36:56.312584+0000","last_fullsized":"2024-03-21T10:36:56.312584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:46:15.984450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623210+0000","last_change":"2024-03-21T10:35:11.199455+0000","last_active":"2024-03-21T10:36:56.623210+0000","last_peered":"2024-03-21T10:36:56.623210+0000","last_clean":"2024-03-21T10:36:56.623210+0000","last_became_active":"2024-03-21T10:35:11.197705+0000","last_became_peered":"2024-03-21T10:35:11.197705+0000","last_unstale":"2024-03-21T10:36:56.623210+0000","last_undegraded":"2024-03-21T10:36:56.623210+0000","last_fullsized":"2024-03-21T10:36:56.623210+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:59:42.792764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325091+0000","last_change":"2024-03-21T10:35:06.016152+0000","last_active":"2024-03-21T10:36:56.325091+0000","last_peered":"2024-03-21T10:36:56.325091+0000","last_clean":"2024-03-21T10:36:56.325091+0000","last_became_active":"2024-03-21T10:35:06.015439+0000","last_became_peered":"2024-03-21T10:35:06.015439+0000","last_unstale":"2024-03-21T10:36:56.325091+0000","last_undegraded":"2024-03-21T10:36:56.325091+0000","last_fullsized":"2024-03-21T10:36:56.325091+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:05:23.221826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618788+0000","last_change":"2024-03-21T10:35:06.929241+0000","last_active":"2024-03-21T10:36:56.618788+0000","last_peered":"2024-03-21T10:36:56.618788+0000","last_clean":"2024-03-21T10:36:56.618788+0000","last_became_active":"2024-03-21T10:35:06.928943+0000","last_became_peered":"2024-03-21T10:35:06.928943+0000","last_unstale":"2024-03-21T10:36:56.618788+0000","last_undegraded":"2024-03-21T10:36:56.618788+0000","last_fullsized":"2024-03-21T10:36:56.618788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:24:10.484540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540647+0000","last_change":"2024-03-21T10:35:11.195929+0000","last_active":"2024-03-21T10:36:56.540647+0000","last_peered":"2024-03-21T10:36:56.540647+0000","last_clean":"2024-03-21T10:36:56.540647+0000","last_became_active":"2024-03-21T10:35:11.195489+0000","last_became_peered":"2024-03-21T10:35:11.195489+0000","last_unstale":"2024-03-21T10:36:56.540647+0000","last_undegraded":"2024-03-21T10:36:56.540647+0000","last_fullsized":"2024-03-21T10:36:56.540647+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:41:15.137917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618664+0000","last_change":"2024-03-21T10:35:06.002127+0000","last_active":"2024-03-21T10:36:56.618664+0000","last_peered":"2024-03-21T10:36:56.618664+0000","last_clean":"2024-03-21T10:36:56.618664+0000","last_became_active":"2024-03-21T10:35:06.001706+0000","last_became_peered":"2024-03-21T10:35:06.001706+0000","last_unstale":"2024-03-21T10:36:56.618664+0000","last_undegraded":"2024-03-21T10:36:56.618664+0000","last_fullsized":"2024-03-21T10:36:56.618664+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:26:32.844991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325124+0000","last_change":"2024-03-21T10:35:06.925945+0000","last_active":"2024-03-21T10:36:56.325124+0000","last_peered":"2024-03-21T10:36:56.325124+0000","last_clean":"2024-03-21T10:36:56.325124+0000","last_became_active":"2024-03-21T10:35:06.925551+0000","last_became_peered":"2024-03-21T10:35:06.925551+0000","last_unstale":"2024-03-21T10:36:56.325124+0000","last_undegraded":"2024-03-21T10:36:56.325124+0000","last_fullsized":"2024-03-21T10:36:56.325124+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:24:10.212676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540869+0000","last_change":"2024-03-21T10:35:11.229067+0000","last_active":"2024-03-21T10:36:56.540869+0000","last_peered":"2024-03-21T10:36:56.540869+0000","last_clean":"2024-03-21T10:36:56.540869+0000","last_became_active":"2024-03-21T10:35:11.228628+0000","last_became_peered":"2024-03-21T10:35:11.228628+0000","last_unstale":"2024-03-21T10:36:56.540869+0000","last_undegraded":"2024-03-21T10:36:56.540869+0000","last_fullsized":"2024-03-21T10:36:56.540869+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:45:05.473277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618908+0000","last_change":"2024-03-21T10:35:55.384922+0000","last_active":"2024-03-21T10:36:56.618908+0000","last_peered":"2024-03-21T10:36:56.618908+0000","last_clean":"2024-03-21T10:36:56.618908+0000","last_became_active":"2024-03-21T10:35:55.380952+0000","last_became_peered":"2024-03-21T10:35:55.380952+0000","last_unstale":"2024-03-21T10:36:56.618908+0000","last_undegraded":"2024-03-21T10:36:56.618908+0000","last_fullsized":"2024-03-21T10:36:56.618908+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:29:37.873699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619590+0000","last_change":"2024-03-21T10:35:06.928049+0000","last_active":"2024-03-21T10:36:56.619590+0000","last_peered":"2024-03-21T10:36:56.619590+0000","last_clean":"2024-03-21T10:36:56.619590+0000","last_became_active":"2024-03-21T10:35:06.927717+0000","last_became_peered":"2024-03-21T10:35:06.927717+0000","last_unstale":"2024-03-21T10:36:56.619590+0000","last_undegraded":"2024-03-21T10:36:56.619590+0000","last_fullsized":"2024-03-21T10:36:56.619590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:27:47.554668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619331+0000","last_change":"2024-03-21T10:35:11.181090+0000","last_active":"2024-03-21T10:36:56.619331+0000","last_peered":"2024-03-21T10:36:56.619331+0000","last_clean":"2024-03-21T10:36:56.619331+0000","last_became_active":"2024-03-21T10:35:11.180511+0000","last_became_peered":"2024-03-21T10:35:11.180511+0000","last_unstale":"2024-03-21T10:36:56.619331+0000","last_undegraded":"2024-03-21T10:36:56.619331+0000","last_fullsized":"2024-03-21T10:36:56.619331+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:21:51.517845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619547+0000","last_change":"2024-03-21T10:35:06.007828+0000","last_active":"2024-03-21T10:36:56.619547+0000","last_peered":"2024-03-21T10:36:56.619547+0000","last_clean":"2024-03-21T10:36:56.619547+0000","last_became_active":"2024-03-21T10:35:06.007490+0000","last_became_peered":"2024-03-21T10:35:06.007490+0000","last_unstale":"2024-03-21T10:36:56.619547+0000","last_undegraded":"2024-03-21T10:36:56.619547+0000","last_fullsized":"2024-03-21T10:36:56.619547+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:08:18.579308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619789+0000","last_change":"2024-03-21T10:35:06.909313+0000","last_active":"2024-03-21T10:36:56.619789+0000","last_peered":"2024-03-21T10:36:56.619789+0000","last_clean":"2024-03-21T10:36:56.619789+0000","last_became_active":"2024-03-21T10:35:06.908555+0000","last_became_peered":"2024-03-21T10:35:06.908555+0000","last_unstale":"2024-03-21T10:36:56.619789+0000","last_undegraded":"2024-03-21T10:36:56.619789+0000","last_fullsized":"2024-03-21T10:36:56.619789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:09:57.961971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.622209+0000","last_change":"2024-03-21T10:35:11.195996+0000","last_active":"2024-03-21T10:36:56.622209+0000","last_peered":"2024-03-21T10:36:56.622209+0000","last_clean":"2024-03-21T10:36:56.622209+0000","last_became_active":"2024-03-21T10:35:11.195506+0000","last_became_peered":"2024-03-21T10:35:11.195506+0000","last_unstale":"2024-03-21T10:36:56.622209+0000","last_undegraded":"2024-03-21T10:36:56.622209+0000","last_fullsized":"2024-03-21T10:36:56.622209+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:49:33.010899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311536+0000","last_change":"2024-03-21T10:35:06.011498+0000","last_active":"2024-03-21T10:36:56.311536+0000","last_peered":"2024-03-21T10:36:56.311536+0000","last_clean":"2024-03-21T10:36:56.311536+0000","last_became_active":"2024-03-21T10:35:06.011203+0000","last_became_peered":"2024-03-21T10:35:06.011203+0000","last_unstale":"2024-03-21T10:36:56.311536+0000","last_undegraded":"2024-03-21T10:36:56.311536+0000","last_fullsized":"2024-03-21T10:36:56.311536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:28:11.369740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619070+0000","last_change":"2024-03-21T10:35:06.935514+0000","last_active":"2024-03-21T10:36:56.619070+0000","last_peered":"2024-03-21T10:36:56.619070+0000","last_clean":"2024-03-21T10:36:56.619070+0000","last_became_active":"2024-03-21T10:35:06.935174+0000","last_became_peered":"2024-03-21T10:35:06.935174+0000","last_unstale":"2024-03-21T10:36:56.619070+0000","last_undegraded":"2024-03-21T10:36:56.619070+0000","last_fullsized":"2024-03-21T10:36:56.619070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:35:42.884471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621386+0000","last_change":"2024-03-21T10:36:56.621053+0000","last_active":"2024-03-21T10:36:56.621386+0000","last_peered":"2024-03-21T10:36:56.621386+0000","last_clean":"2024-03-21T10:36:56.621386+0000","last_became_active":"2024-03-21T10:36:56.620688+0000","last_became_peered":"2024-03-21T10:36:56.620688+0000","last_unstale":"2024-03-21T10:36:56.621386+0000","last_undegraded":"2024-03-21T10:36:56.621386+0000","last_fullsized":"2024-03-21T10:36:56.621386+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:50:02.121787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312981+0000","last_change":"2024-03-21T10:35:06.007216+0000","last_active":"2024-03-21T10:36:56.312981+0000","last_peered":"2024-03-21T10:36:56.312981+0000","last_clean":"2024-03-21T10:36:56.312981+0000","last_became_active":"2024-03-21T10:35:06.006883+0000","last_became_peered":"2024-03-21T10:35:06.006883+0000","last_unstale":"2024-03-21T10:36:56.312981+0000","last_undegraded":"2024-03-21T10:36:56.312981+0000","last_fullsized":"2024-03-21T10:36:56.312981+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:46:45.310919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619751+0000","last_change":"2024-03-21T10:35:06.906017+0000","last_active":"2024-03-21T10:36:56.619751+0000","last_peered":"2024-03-21T10:36:56.619751+0000","last_clean":"2024-03-21T10:36:56.619751+0000","last_became_active":"2024-03-21T10:35:06.905648+0000","last_became_peered":"2024-03-21T10:35:06.905648+0000","last_unstale":"2024-03-21T10:36:56.619751+0000","last_undegraded":"2024-03-21T10:36:56.619751+0000","last_fullsized":"2024-03-21T10:36:56.619751+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:56:50.380146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618954+0000","last_change":"2024-03-21T10:35:06.905186+0000","last_active":"2024-03-21T10:36:56.618954+0000","last_peered":"2024-03-21T10:36:56.618954+0000","last_clean":"2024-03-21T10:36:56.618954+0000","last_became_active":"2024-03-21T10:35:06.904835+0000","last_became_peered":"2024-03-21T10:35:06.904835+0000","last_unstale":"2024-03-21T10:36:56.618954+0000","last_undegraded":"2024-03-21T10:36:56.618954+0000","last_fullsized":"2024-03-21T10:36:56.618954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:53:19.293437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618890+0000","last_change":"2024-03-21T10:35:06.903318+0000","last_active":"2024-03-21T10:36:56.618890+0000","last_peered":"2024-03-21T10:36:56.618890+0000","last_clean":"2024-03-21T10:36:56.618890+0000","last_became_active":"2024-03-21T10:35:06.902994+0000","last_became_peered":"2024-03-21T10:35:06.902994+0000","last_unstale":"2024-03-21T10:36:56.618890+0000","last_undegraded":"2024-03-21T10:36:56.618890+0000","last_fullsized":"2024-03-21T10:36:56.618890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:21:27.412565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618616+0000","last_change":"2024-03-21T10:35:06.934582+0000","last_active":"2024-03-21T10:36:56.618616+0000","last_peered":"2024-03-21T10:36:56.618616+0000","last_clean":"2024-03-21T10:36:56.618616+0000","last_became_active":"2024-03-21T10:35:06.934311+0000","last_became_peered":"2024-03-21T10:35:06.934311+0000","last_unstale":"2024-03-21T10:36:56.618616+0000","last_undegraded":"2024-03-21T10:36:56.618616+0000","last_fullsized":"2024-03-21T10:36:56.618616+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:05:30.785830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.622771+0000","last_change":"2024-03-21T10:35:06.944221+0000","last_active":"2024-03-21T10:36:56.622771+0000","last_peered":"2024-03-21T10:36:56.622771+0000","last_clean":"2024-03-21T10:36:56.622771+0000","last_became_active":"2024-03-21T10:35:06.943732+0000","last_became_peered":"2024-03-21T10:35:06.943732+0000","last_unstale":"2024-03-21T10:36:56.622771+0000","last_undegraded":"2024-03-21T10:36:56.622771+0000","last_fullsized":"2024-03-21T10:36:56.622771+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:33:03.653160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313274+0000","last_change":"2024-03-21T10:35:06.940832+0000","last_active":"2024-03-21T10:36:56.313274+0000","last_peered":"2024-03-21T10:36:56.313274+0000","last_clean":"2024-03-21T10:36:56.313274+0000","last_became_active":"2024-03-21T10:35:06.939565+0000","last_became_peered":"2024-03-21T10:35:06.939565+0000","last_unstale":"2024-03-21T10:36:56.313274+0000","last_undegraded":"2024-03-21T10:36:56.313274+0000","last_fullsized":"2024-03-21T10:36:56.313274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:57:27.743704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619538+0000","last_change":"2024-03-21T10:35:06.907753+0000","last_active":"2024-03-21T10:36:56.619538+0000","last_peered":"2024-03-21T10:36:56.619538+0000","last_clean":"2024-03-21T10:36:56.619538+0000","last_became_active":"2024-03-21T10:35:06.907432+0000","last_became_peered":"2024-03-21T10:35:06.907432+0000","last_unstale":"2024-03-21T10:36:56.619538+0000","last_undegraded":"2024-03-21T10:36:56.619538+0000","last_fullsized":"2024-03-21T10:36:56.619538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:31:02.528855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619622+0000","last_change":"2024-03-21T10:35:06.908597+0000","last_active":"2024-03-21T10:36:56.619622+0000","last_peered":"2024-03-21T10:36:56.619622+0000","last_clean":"2024-03-21T10:36:56.619622+0000","last_became_active":"2024-03-21T10:35:06.908282+0000","last_became_peered":"2024-03-21T10:35:06.908282+0000","last_unstale":"2024-03-21T10:36:56.619622+0000","last_undegraded":"2024-03-21T10:36:56.619622+0000","last_fullsized":"2024-03-21T10:36:56.619622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:42:18.221045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325969+0000","last_change":"2024-03-21T10:35:06.931015+0000","last_active":"2024-03-21T10:36:56.325969+0000","last_peered":"2024-03-21T10:36:56.325969+0000","last_clean":"2024-03-21T10:36:56.325969+0000","last_became_active":"2024-03-21T10:35:06.930209+0000","last_became_peered":"2024-03-21T10:35:06.930209+0000","last_unstale":"2024-03-21T10:36:56.325969+0000","last_undegraded":"2024-03-21T10:36:56.325969+0000","last_fullsized":"2024-03-21T10:36:56.325969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:42:15.277874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311194+0000","last_change":"2024-03-21T10:35:06.930070+0000","last_active":"2024-03-21T10:36:56.311194+0000","last_peered":"2024-03-21T10:36:56.311194+0000","last_clean":"2024-03-21T10:36:56.311194+0000","last_became_active":"2024-03-21T10:35:06.929773+0000","last_became_peered":"2024-03-21T10:35:06.929773+0000","last_unstale":"2024-03-21T10:36:56.311194+0000","last_undegraded":"2024-03-21T10:36:56.311194+0000","last_fullsized":"2024-03-21T10:36:56.311194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:17:25.176666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325300+0000","last_change":"2024-03-21T10:35:06.929575+0000","last_active":"2024-03-21T10:36:56.325300+0000","last_peered":"2024-03-21T10:36:56.325300+0000","last_clean":"2024-03-21T10:36:56.325300+0000","last_became_active":"2024-03-21T10:35:06.929240+0000","last_became_peered":"2024-03-21T10:35:06.929240+0000","last_unstale":"2024-03-21T10:36:56.325300+0000","last_undegraded":"2024-03-21T10:36:56.325300+0000","last_fullsized":"2024-03-21T10:36:56.325300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:03:40.697640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619574+0000","last_change":"2024-03-21T10:35:06.934693+0000","last_active":"2024-03-21T10:36:56.619574+0000","last_peered":"2024-03-21T10:36:56.619574+0000","last_clean":"2024-03-21T10:36:56.619574+0000","last_became_active":"2024-03-21T10:35:06.934361+0000","last_became_peered":"2024-03-21T10:35:06.934361+0000","last_unstale":"2024-03-21T10:36:56.619574+0000","last_undegraded":"2024-03-21T10:36:56.619574+0000","last_fullsized":"2024-03-21T10:36:56.619574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:14:00.954370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312448+0000","last_change":"2024-03-21T10:35:06.941095+0000","last_active":"2024-03-21T10:36:56.312448+0000","last_peered":"2024-03-21T10:36:56.312448+0000","last_clean":"2024-03-21T10:36:56.312448+0000","last_became_active":"2024-03-21T10:35:06.939348+0000","last_became_peered":"2024-03-21T10:35:06.939348+0000","last_unstale":"2024-03-21T10:36:56.312448+0000","last_undegraded":"2024-03-21T10:36:56.312448+0000","last_fullsized":"2024-03-21T10:36:56.312448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:31:43.783592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619147+0000","last_change":"2024-03-21T10:35:06.908340+0000","last_active":"2024-03-21T10:36:56.619147+0000","last_peered":"2024-03-21T10:36:56.619147+0000","last_clean":"2024-03-21T10:36:56.619147+0000","last_became_active":"2024-03-21T10:35:06.908043+0000","last_became_peered":"2024-03-21T10:35:06.908043+0000","last_unstale":"2024-03-21T10:36:56.619147+0000","last_undegraded":"2024-03-21T10:36:56.619147+0000","last_fullsized":"2024-03-21T10:36:56.619147+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:50:33.167396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282863,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68597,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":565,"num_read_kb":356,"num_write":172,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":205791,"data_compressed":5487,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383721,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1019904,"data_stored":519397,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645357,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32320,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712832,"statfs":{"total":95995035648,"available":95961939968,"internally_reserved":0,"allocated":1163264,"data_stored":1171120,"data_compressed":64463,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808882,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":523729,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1036288,"data_stored":519553,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332159,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1085440,"data_stored":1115603,"data_compressed":62634,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495685,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":514809,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574946,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1081344,"data_stored":579174,"data_compressed":28489,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738471,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":516197,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1105920,"data_stored":581056,"data_compressed":28489,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182485,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":515003,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":519459,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757322,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1093632,"data_stored":1111191,"data_compressed":62634,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34331,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34331,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34331,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":64455,"data_compressed":1829,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":66337,"data_compressed":1829,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":66033,"data_compressed":1829,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T10:37:15.994 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-21T10:37:15.995 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-21T10:37:15.995 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-21T10:37:15.995 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-21T10:37:16.497 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:37:16.645 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:16 smithi031 ceph-mon[31673]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T10:37:16.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:16 smithi040 ceph-mon[36154]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T10:37:16.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:16 smithi070 ceph-mon[36330]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T10:37:17.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:17 smithi040 ceph-mon[36154]: pgmap v394: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:37:17.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:17 smithi031 ceph-mon[31673]: pgmap v394: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:37:17.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:17 smithi070 ceph-mon[36330]: pgmap v394: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:37:18.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.861+0000 7f1ae6c00700 1 -- 172.21.15.31:0/1655367325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ae01081e0 msgr2=0x7f1ae01085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:18.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.861+0000 7f1ae6c00700 1 --2- 172.21.15.31:0/1655367325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ae01081e0 0x7f1ae01085c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0009990 tx=0x7f1ad0009ca0 comp rx=0 tx=0).stop 2024-03-21T10:37:18.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.862+0000 7f1ae6c00700 1 -- 172.21.15.31:0/1655367325 shutdown_connections 2024-03-21T10:37:18.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.862+0000 7f1ae6c00700 1 --2- 172.21.15.31:0/1655367325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ae006a7e0 0x7f1ae010f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:18.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.862+0000 7f1ae6c00700 1 --2- 172.21.15.31:0/1655367325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ae01081e0 0x7f1ae01085c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:18.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.862+0000 7f1ae6c00700 1 --2- 172.21.15.31:0/1655367325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ae0069e40 0x7f1ae006a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:18.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.862+0000 7f1ae6c00700 1 -- 172.21.15.31:0/1655367325 >> 172.21.15.31:0/1655367325 conn(0x7f1ae0077fc0 msgr2=0x7f1ae00783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:18.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.862+0000 7f1ae6c00700 1 -- 172.21.15.31:0/1655367325 shutdown_connections 2024-03-21T10:37:18.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.862+0000 7f1ae6c00700 1 -- 172.21.15.31:0/1655367325 wait complete. 2024-03-21T10:37:18.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.863+0000 7f1ae6c00700 1 Processor -- start 2024-03-21T10:37:18.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.863+0000 7f1ae6c00700 1 -- start start 2024-03-21T10:37:18.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.864+0000 7f1ae6c00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ae0069e40 0x7f1ae01a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:18.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.864+0000 7f1ae6c00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ae006a7e0 0x7f1ae01a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:18.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.864+0000 7f1ae6c00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ae01081e0 0x7f1ae019aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:18.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.864+0000 7f1ae6c00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae0112470 con 0x7f1ae01081e0 2024-03-21T10:37:18.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.864+0000 7f1ae6c00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae01122f0 con 0x7f1ae0069e40 2024-03-21T10:37:18.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.864+0000 7f1ae6c00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae01125f0 con 0x7f1ae006a7e0 2024-03-21T10:37:18.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.864+0000 7f1adfe00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ae0069e40 0x7f1ae01a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:18.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.864+0000 7f1adfe00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ae0069e40 0x7f1ae01a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38710/0 (socket says 172.21.15.31:38710) 2024-03-21T10:37:18.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.864+0000 7f1adfe00700 1 -- 172.21.15.31:0/862648606 learned_addr learned my addr 172.21.15.31:0/862648606 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:18.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.864+0000 7f1ad7400700 1 --2- 172.21.15.31:0/862648606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ae006a7e0 0x7f1ae01a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:18.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.864+0000 7f1ae4c00700 1 --2- 172.21.15.31:0/862648606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ae01081e0 0x7f1ae019aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:18.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.865+0000 7f1adfe00700 1 -- 172.21.15.31:0/862648606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ae006a7e0 msgr2=0x7f1ae01a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:18.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.865+0000 7f1adfe00700 1 --2- 172.21.15.31:0/862648606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ae006a7e0 0x7f1ae01a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:18.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.865+0000 7f1adfe00700 1 -- 172.21.15.31:0/862648606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ae01081e0 msgr2=0x7f1ae019aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:18.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.865+0000 7f1adfe00700 1 --2- 172.21.15.31:0/862648606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ae01081e0 0x7f1ae019aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:18.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.865+0000 7f1adfe00700 1 -- 172.21.15.31:0/862648606 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae019b160 con 0x7f1ae0069e40 2024-03-21T10:37:18.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.865+0000 7f1adfe00700 1 --2- 172.21.15.31:0/862648606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ae0069e40 0x7f1ae01a0840 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0000c00 tx=0x7f1ad00057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:18.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.865+0000 7f1add600700 1 -- 172.21.15.31:0/862648606 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0027070 con 0x7f1ae0069e40 2024-03-21T10:37:18.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.865+0000 7f1add600700 1 -- 172.21.15.31:0/862648606 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ad000be70 con 0x7f1ae0069e40 2024-03-21T10:37:18.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.865+0000 7f1add600700 1 -- 172.21.15.31:0/862648606 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0017650 con 0x7f1ae0069e40 2024-03-21T10:37:18.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.865+0000 7f1ae6c00700 1 -- 172.21.15.31:0/862648606 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae019b3f0 con 0x7f1ae0069e40 2024-03-21T10:37:18.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.866+0000 7f1ae6c00700 1 -- 172.21.15.31:0/862648606 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae01a7710 con 0x7f1ae0069e40 2024-03-21T10:37:18.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.867+0000 7f1ae6c00700 1 -- 172.21.15.31:0/862648606 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae0068e50 con 0x7f1ae0069e40 2024-03-21T10:37:18.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.868+0000 7f1add600700 1 -- 172.21.15.31:0/862648606 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f1ad002a070 con 0x7f1ae0069e40 2024-03-21T10:37:18.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.871+0000 7f1add600700 1 --2- 172.21.15.31:0/862648606 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1ac40a3030 0x7f1ac40a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:18.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.871+0000 7f1add600700 1 -- 172.21.15.31:0/862648606 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f1ad00d0720 con 0x7f1ae0069e40 2024-03-21T10:37:18.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.871+0000 7f1ad7400700 1 --2- 172.21.15.31:0/862648606 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1ac40a3030 0x7f1ac40a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:18.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.872+0000 7f1ad7400700 1 --2- 172.21.15.31:0/862648606 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1ac40a3030 0x7f1ac40a54f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1ae01030d0 tx=0x7f1ac8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:18.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:18.874+0000 7f1add600700 1 -- 172.21.15.31:0/862648606 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1ad005e740 con 0x7f1ae0069e40 2024-03-21T10:37:19.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:19.151+0000 7f1ae6c00700 1 -- 172.21.15.31:0/862648606 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1ae01a78e0 con 0x7f1ae0069e40 2024-03-21T10:37:19.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:19.152+0000 7f1add600700 1 -- 172.21.15.31:0/862648606 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f1ad0013070 con 0x7f1ae0069e40 2024-03-21T10:37:19.150 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:37:19.150 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-21T10:37:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:19.155+0000 7f1ae6c00700 1 -- 172.21.15.31:0/862648606 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1ac40a3030 msgr2=0x7f1ac40a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:19.155+0000 7f1ae6c00700 1 --2- 172.21.15.31:0/862648606 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1ac40a3030 0x7f1ac40a54f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1ae01030d0 tx=0x7f1ac8009260 comp rx=0 tx=0).stop 2024-03-21T10:37:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:19.155+0000 7f1ae6c00700 1 -- 172.21.15.31:0/862648606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ae0069e40 msgr2=0x7f1ae01a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:19.155+0000 7f1ae6c00700 1 --2- 172.21.15.31:0/862648606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ae0069e40 0x7f1ae01a0840 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0000c00 tx=0x7f1ad00057c0 comp rx=0 tx=0).stop 2024-03-21T10:37:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:19.155+0000 7f1ae6c00700 1 -- 172.21.15.31:0/862648606 shutdown_connections 2024-03-21T10:37:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:19.155+0000 7f1ae6c00700 1 --2- 172.21.15.31:0/862648606 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1ac40a3030 0x7f1ac40a54f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:19.155+0000 7f1ae6c00700 1 --2- 172.21.15.31:0/862648606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ae01081e0 0x7f1ae019aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:19.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:19.155+0000 7f1ae6c00700 1 --2- 172.21.15.31:0/862648606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ae0069e40 0x7f1ae01a0840 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:19.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:19.155+0000 7f1ae6c00700 1 --2- 172.21.15.31:0/862648606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ae006a7e0 0x7f1ae01a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:19.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:19.155+0000 7f1ae6c00700 1 -- 172.21.15.31:0/862648606 >> 172.21.15.31:0/862648606 conn(0x7f1ae0077fc0 msgr2=0x7f1ae010d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:19.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:19.155+0000 7f1ae6c00700 1 -- 172.21.15.31:0/862648606 shutdown_connections 2024-03-21T10:37:19.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:19.155+0000 7f1ae6c00700 1 -- 172.21.15.31:0/862648606 wait complete. 2024-03-21T10:37:19.547 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:19 smithi031 ceph-mon[31673]: pgmap v395: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:19.547 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:19 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/862648606' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-21T10:37:19.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:19 smithi040 ceph-mon[36154]: pgmap v395: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:19.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:19 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/862648606' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-21T10:37:19.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:19 smithi070 ceph-mon[36330]: pgmap v395: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:19.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:19 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/862648606' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-21T10:37:20.016 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-21T10:37:20.016 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T10:37:20.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.248+0000 7f8693e00640 1 -- 172.21.15.31:0/1450714052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8694074100 msgr2=0x7f8694074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:20.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.248+0000 7f8693e00640 1 --2- 172.21.15.31:0/1450714052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8694074100 0x7f8694074580 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8680009900 tx=0x7f868002bc30 comp rx=0 tx=0).stop 2024-03-21T10:37:20.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.248+0000 7f8693e00640 1 -- 172.21.15.31:0/1450714052 shutdown_connections 2024-03-21T10:37:20.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.248+0000 7f8693e00640 1 --2- 172.21.15.31:0/1450714052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f869410db70 0x7f869410ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:20.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.248+0000 7f8693e00640 1 --2- 172.21.15.31:0/1450714052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8694074100 0x7f8694074580 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:20.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.248+0000 7f8693e00640 1 --2- 172.21.15.31:0/1450714052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86940756e0 0x7f8694073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:20.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.248+0000 7f8693e00640 1 -- 172.21.15.31:0/1450714052 >> 172.21.15.31:0/1450714052 conn(0x7f86940fcb30 msgr2=0x7f86940fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:20.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.248+0000 7f8693e00640 1 -- 172.21.15.31:0/1450714052 shutdown_connections 2024-03-21T10:37:20.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.248+0000 7f8693e00640 1 -- 172.21.15.31:0/1450714052 wait complete. 2024-03-21T10:37:20.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.249+0000 7f8693e00640 1 Processor -- start 2024-03-21T10:37:20.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.249+0000 7f8693e00640 1 -- start start 2024-03-21T10:37:20.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.250+0000 7f8693e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8694074100 0x7f869419dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:20.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.250+0000 7f8693e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86940756e0 0x7f869419e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:20.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.250+0000 7f8693e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f869410db70 0x7f86941a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:20.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.250+0000 7f8693e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8694113840 con 0x7f86940756e0 2024-03-21T10:37:20.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.250+0000 7f8693e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86941136c0 con 0x7f869410db70 2024-03-21T10:37:20.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.250+0000 7f8693e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86941139c0 con 0x7f8694074100 2024-03-21T10:37:20.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.250+0000 7f8692a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8694074100 0x7f869419dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:20.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.250+0000 7f8692a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8694074100 0x7f869419dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:47284/0 (socket says 172.21.15.31:47284) 2024-03-21T10:37:20.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.250+0000 7f8692a00640 1 -- 172.21.15.31:0/826963604 learned_addr learned my addr 172.21.15.31:0/826963604 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:20.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.250+0000 7f8692000640 1 --2- 172.21.15.31:0/826963604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86940756e0 0x7f869419e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:20.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.250+0000 7f8693400640 1 --2- 172.21.15.31:0/826963604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f869410db70 0x7f86941a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:20.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.251+0000 7f8692a00640 1 -- 172.21.15.31:0/826963604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f869410db70 msgr2=0x7f86941a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:20.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.251+0000 7f8692a00640 1 --2- 172.21.15.31:0/826963604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f869410db70 0x7f86941a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:20.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.251+0000 7f8692a00640 1 -- 172.21.15.31:0/826963604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86940756e0 msgr2=0x7f869419e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:20.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.251+0000 7f8692a00640 1 --2- 172.21.15.31:0/826963604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86940756e0 0x7f869419e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:20.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.251+0000 7f8692a00640 1 -- 172.21.15.31:0/826963604 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86941a5b00 con 0x7f8694074100 2024-03-21T10:37:20.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.251+0000 7f8693400640 1 --2- 172.21.15.31:0/826963604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f869410db70 0x7f86941a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:37:20.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.251+0000 7f8692000640 1 --2- 172.21.15.31:0/826963604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86940756e0 0x7f869419e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:37:20.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.251+0000 7f8692a00640 1 --2- 172.21.15.31:0/826963604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8694074100 0x7f869419dee0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f868400e9f0 tx=0x7f868400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:20.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.251+0000 7f868b400640 1 -- 172.21.15.31:0/826963604 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868400cd80 con 0x7f8694074100 2024-03-21T10:37:20.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.251+0000 7f868b400640 1 -- 172.21.15.31:0/826963604 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8684010910 con 0x7f8694074100 2024-03-21T10:37:20.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.251+0000 7f868b400640 1 -- 172.21.15.31:0/826963604 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8684018990 con 0x7f8694074100 2024-03-21T10:37:20.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.251+0000 7f8693e00640 1 -- 172.21.15.31:0/826963604 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86941a5df0 con 0x7f8694074100 2024-03-21T10:37:20.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.251+0000 7f8693e00640 1 -- 172.21.15.31:0/826963604 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86941a6260 con 0x7f8694074100 2024-03-21T10:37:20.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.253+0000 7f8693e00640 1 -- 172.21.15.31:0/826963604 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8694103d40 con 0x7f8694074100 2024-03-21T10:37:20.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.256+0000 7f868b400640 1 -- 172.21.15.31:0/826963604 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f8684010ac0 con 0x7f8694074100 2024-03-21T10:37:20.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.257+0000 7f868b400640 1 --2- 172.21.15.31:0/826963604 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f86780a2f90 0x7f86780a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:20.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.258+0000 7f868b400640 1 -- 172.21.15.31:0/826963604 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f86840cfb70 con 0x7f8694074100 2024-03-21T10:37:20.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.258+0000 7f8692000640 1 --2- 172.21.15.31:0/826963604 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f86780a2f90 0x7f86780a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:20.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.259+0000 7f8692000640 1 --2- 172.21.15.31:0/826963604 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f86780a2f90 0x7f86780a5450 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f868002c240 tx=0x7f8680005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:20.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.262+0000 7f868b400640 1 -- 172.21.15.31:0/826963604 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f868409c5e0 con 0x7f8694074100 2024-03-21T10:37:20.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.468+0000 7f8693e00640 1 -- 172.21.15.31:0/826963604 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86940622d0 con 0x7f8694074100 2024-03-21T10:37:20.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.470+0000 7f868b400640 1 -- 172.21.15.31:0/826963604 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24553 (secure 0 0 0) 0x7f868409bee0 con 0x7f8694074100 2024-03-21T10:37:20.468 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:37:20.468 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":93,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","created":"2024-03-21T10:25:41.971117+0000","modified":"2024-03-21T10:36:56.301695+0000","last_up_change":"2024-03-21T10:34:07.757196+0000","last_in_change":"2024-03-21T10:33:47.460406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T10:29:44.553968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T10:35:04.945449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T10:35:05.970589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T10:35:09.688711+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e0244d33-00e2-47d5-9f66-34efc138c462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6803","nonce":3089885125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6805","nonce":3089885125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6809","nonce":3089885125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6807","nonce":3089885125}]},"public_addr":"172.21.15.31:6803/3089885125","cluster_addr":"172.21.15.31:6805/3089885125","heartbeat_back_addr":"172.21.15.31:6809/3089885125","heartbeat_front_addr":"172.21.15.31:6807/3089885125","state":["exists","up"]},{"osd":1,"uuid":"76a63504-beab-4354-a6c3-eebc3b16ad03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6801","nonce":618840037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6803","nonce":618840037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6807","nonce":618840037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6805","nonce":618840037}]},"public_addr":"172.21.15.40:6801/618840037","cluster_addr":"172.21.15.40:6803/618840037","heartbeat_back_addr":"172.21.15.40:6807/618840037","heartbeat_front_addr":"172.21.15.40:6805/618840037","state":["exists","up"]},{"osd":2,"uuid":"fa191bd6-9a21-4ee2-8adf-8a63c9cfd821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6801","nonce":2524184117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6803","nonce":2524184117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6807","nonce":2524184117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6805","nonce":2524184117}]},"public_addr":"172.21.15.70:6801/2524184117","cluster_addr":"172.21.15.70:6803/2524184117","heartbeat_back_addr":"172.21.15.70:6807/2524184117","heartbeat_front_addr":"172.21.15.70:6805/2524184117","state":["exists","up"]},{"osd":3,"uuid":"edb44507-ffb0-42a4-827b-08461e1bf952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6811","nonce":2823036296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6813","nonce":2823036296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6817","nonce":2823036296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6815","nonce":2823036296}]},"public_addr":"172.21.15.31:6811/2823036296","cluster_addr":"172.21.15.31:6813/2823036296","heartbeat_back_addr":"172.21.15.31:6817/2823036296","heartbeat_front_addr":"172.21.15.31:6815/2823036296","state":["exists","up"]},{"osd":4,"uuid":"8cb89153-8c3b-431f-b57c-8863fada76dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6809","nonce":4215186309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6811","nonce":4215186309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6815","nonce":4215186309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6813","nonce":4215186309}]},"public_addr":"172.21.15.40:6809/4215186309","cluster_addr":"172.21.15.40:6811/4215186309","heartbeat_back_addr":"172.21.15.40:6815/4215186309","heartbeat_front_addr":"172.21.15.40:6813/4215186309","state":["exists","up"]},{"osd":5,"uuid":"9409505a-d463-408c-a3cd-218380c04d68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6809","nonce":3402830452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6811","nonce":3402830452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6815","nonce":3402830452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6813","nonce":3402830452}]},"public_addr":"172.21.15.70:6809/3402830452","cluster_addr":"172.21.15.70:6811/3402830452","heartbeat_back_addr":"172.21.15.70:6815/3402830452","heartbeat_front_addr":"172.21.15.70:6813/3402830452","state":["exists","up"]},{"osd":6,"uuid":"f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6819","nonce":1193497058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6821","nonce":1193497058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6825","nonce":1193497058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6823","nonce":1193497058}]},"public_addr":"172.21.15.31:6819/1193497058","cluster_addr":"172.21.15.31:6821/1193497058","heartbeat_back_addr":"172.21.15.31:6825/1193497058","heartbeat_front_addr":"172.21.15.31:6823/1193497058","state":["exists","up"]},{"osd":7,"uuid":"db228a87-29a5-45d4-8fe6-a0a554262eb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6817","nonce":3825628119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6819","nonce":3825628119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6823","nonce":3825628119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6821","nonce":3825628119}]},"public_addr":"172.21.15.40:6817/3825628119","cluster_addr":"172.21.15.40:6819/3825628119","heartbeat_back_addr":"172.21.15.40:6823/3825628119","heartbeat_front_addr":"172.21.15.40:6821/3825628119","state":["exists","up"]},{"osd":8,"uuid":"59b9fde0-095d-4e0b-87da-ead37bd8ff1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6817","nonce":534880586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6819","nonce":534880586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6823","nonce":534880586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6821","nonce":534880586}]},"public_addr":"172.21.15.70:6817/534880586","cluster_addr":"172.21.15.70:6819/534880586","heartbeat_back_addr":"172.21.15.70:6823/534880586","heartbeat_front_addr":"172.21.15.70:6821/534880586","state":["exists","up"]},{"osd":9,"uuid":"77f47061-0c5f-4159-b276-e814935d6c1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6827","nonce":587047469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6829","nonce":587047469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6833","nonce":587047469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6831","nonce":587047469}]},"public_addr":"172.21.15.31:6827/587047469","cluster_addr":"172.21.15.31:6829/587047469","heartbeat_back_addr":"172.21.15.31:6833/587047469","heartbeat_front_addr":"172.21.15.31:6831/587047469","state":["exists","up"]},{"osd":10,"uuid":"8853dc43-7bd1-4c93-8063-f7982fd7da38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6825","nonce":512108711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6827","nonce":512108711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6831","nonce":512108711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6829","nonce":512108711}]},"public_addr":"172.21.15.40:6825/512108711","cluster_addr":"172.21.15.40:6827/512108711","heartbeat_back_addr":"172.21.15.40:6831/512108711","heartbeat_front_addr":"172.21.15.40:6829/512108711","state":["exists","up"]},{"osd":11,"uuid":"84b7c2de-c424-45a6-9583-60097e372a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6825","nonce":540305718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6827","nonce":540305718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6831","nonce":540305718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6829","nonce":540305718}]},"public_addr":"172.21.15.70:6825/540305718","cluster_addr":"172.21.15.70:6827/540305718","heartbeat_back_addr":"172.21.15.70:6831/540305718","heartbeat_front_addr":"172.21.15.70:6829/540305718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:28:51.477358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:17.405045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:41.626268+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:15.339568+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:42.676996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:07.516999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:42.412273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:10.365041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:35.699120+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:11.733639+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:39.505442+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:34:05.449301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451930554":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/3342588843":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6801/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/1626448389":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/1910059351":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/344822770":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/3811368512":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/1981437651":"2024-03-22T10:26:16.959165+0000","172.21.15.31:6801/1981437651":"2024-03-22T10:26:16.959165+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T10:37:20.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.473+0000 7f8693e00640 1 -- 172.21.15.31:0/826963604 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f86780a2f90 msgr2=0x7f86780a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:20.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.473+0000 7f8693e00640 1 --2- 172.21.15.31:0/826963604 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f86780a2f90 0x7f86780a5450 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f868002c240 tx=0x7f8680005fd0 comp rx=0 tx=0).stop 2024-03-21T10:37:20.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.473+0000 7f8693e00640 1 -- 172.21.15.31:0/826963604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8694074100 msgr2=0x7f869419dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:20.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.473+0000 7f8693e00640 1 --2- 172.21.15.31:0/826963604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8694074100 0x7f869419dee0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f868400e9f0 tx=0x7f868400eeb0 comp rx=0 tx=0).stop 2024-03-21T10:37:20.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.473+0000 7f8693e00640 1 -- 172.21.15.31:0/826963604 shutdown_connections 2024-03-21T10:37:20.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.473+0000 7f8693e00640 1 --2- 172.21.15.31:0/826963604 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f86780a2f90 0x7f86780a5450 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:20.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.473+0000 7f8693e00640 1 --2- 172.21.15.31:0/826963604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f869410db70 0x7f86941a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:20.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.473+0000 7f8693e00640 1 --2- 172.21.15.31:0/826963604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86940756e0 0x7f869419e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:20.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.473+0000 7f8693e00640 1 --2- 172.21.15.31:0/826963604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8694074100 0x7f869419dee0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:20.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.473+0000 7f8693e00640 1 -- 172.21.15.31:0/826963604 >> 172.21.15.31:0/826963604 conn(0x7f86940fcb30 msgr2=0x7f8694100f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:20.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.473+0000 7f8693e00640 1 -- 172.21.15.31:0/826963604 shutdown_connections 2024-03-21T10:37:20.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.473+0000 7f8693e00640 1 -- 172.21.15.31:0/826963604 wait complete. 2024-03-21T10:37:20.493 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T10:29:44.553968+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T10:35:04.945449+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T10:35:05.970589+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T10:35:09.688711+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T10:37:20.493 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T10:37:20.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.702+0000 7fa9b6e00640 1 -- 172.21.15.31:0/2222475782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9b01048a0 msgr2=0x7fa9b0106c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:20.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.702+0000 7fa9b6e00640 1 --2- 172.21.15.31:0/2222475782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9b01048a0 0x7fa9b0106c90 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa99c009900 tx=0x7fa99c02bc40 comp rx=0 tx=0).stop 2024-03-21T10:37:20.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.702+0000 7fa9b6e00640 1 -- 172.21.15.31:0/2222475782 shutdown_connections 2024-03-21T10:37:20.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.702+0000 7fa9b6e00640 1 --2- 172.21.15.31:0/2222475782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9b01071d0 0x7fa9b0109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:20.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.702+0000 7fa9b6e00640 1 --2- 172.21.15.31:0/2222475782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9b01048a0 0x7fa9b0106c90 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:20.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.702+0000 7fa9b6e00640 1 --2- 172.21.15.31:0/2222475782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9b0101f70 0x7fa9b0104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:20.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.702+0000 7fa9b6e00640 1 -- 172.21.15.31:0/2222475782 >> 172.21.15.31:0/2222475782 conn(0x7fa9b00fb980 msgr2=0x7fa9b00fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:20.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.702+0000 7fa9b6e00640 1 -- 172.21.15.31:0/2222475782 shutdown_connections 2024-03-21T10:37:20.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.702+0000 7fa9b6e00640 1 -- 172.21.15.31:0/2222475782 wait complete. 2024-03-21T10:37:20.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.703+0000 7fa9b6e00640 1 Processor -- start 2024-03-21T10:37:20.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.703+0000 7fa9b6e00640 1 -- start start 2024-03-21T10:37:20.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.704+0000 7fa9b6e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9b0101f70 0x7fa9b0199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:20.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.704+0000 7fa9b6e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9b01048a0 0x7fa9b0199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:20.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.704+0000 7fa9b6e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9b01071d0 0x7fa9b01a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:20.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.704+0000 7fa9b6e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b010d140 con 0x7fa9b01071d0 2024-03-21T10:37:20.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.704+0000 7fa9b6e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b010cfc0 con 0x7fa9b0101f70 2024-03-21T10:37:20.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.704+0000 7fa9b6e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b010d2c0 con 0x7fa9b01048a0 2024-03-21T10:37:20.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.704+0000 7fa9b5a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9b0101f70 0x7fa9b0199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:20.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.704+0000 7fa9b5a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9b0101f70 0x7fa9b0199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38740/0 (socket says 172.21.15.31:38740) 2024-03-21T10:37:20.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.704+0000 7fa9b5a00640 1 -- 172.21.15.31:0/2136077495 learned_addr learned my addr 172.21.15.31:0/2136077495 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:20.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.704+0000 7fa9b6400640 1 --2- 172.21.15.31:0/2136077495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9b01071d0 0x7fa9b01a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:20.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.704+0000 7fa9b5000640 1 --2- 172.21.15.31:0/2136077495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9b01048a0 0x7fa9b0199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:20.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.704+0000 7fa9b5a00640 1 -- 172.21.15.31:0/2136077495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9b01048a0 msgr2=0x7fa9b0199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:20.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.704+0000 7fa9b5a00640 1 --2- 172.21.15.31:0/2136077495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9b01048a0 0x7fa9b0199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:20.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.704+0000 7fa9b5a00640 1 -- 172.21.15.31:0/2136077495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9b01071d0 msgr2=0x7fa9b01a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:20.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.704+0000 7fa9b5a00640 1 --2- 172.21.15.31:0/2136077495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9b01071d0 0x7fa9b01a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:20.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.704+0000 7fa9b5a00640 1 -- 172.21.15.31:0/2136077495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b01a1630 con 0x7fa9b0101f70 2024-03-21T10:37:20.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.705+0000 7fa9b5a00640 1 --2- 172.21.15.31:0/2136077495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9b0101f70 0x7fa9b0199a10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa9a400e9f0 tx=0x7fa9a400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:20.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.705+0000 7fa9ae000640 1 -- 172.21.15.31:0/2136077495 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a400cd80 con 0x7fa9b0101f70 2024-03-21T10:37:20.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.705+0000 7fa9ae000640 1 -- 172.21.15.31:0/2136077495 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9a4010910 con 0x7fa9b0101f70 2024-03-21T10:37:20.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.705+0000 7fa9b6e00640 1 -- 172.21.15.31:0/2136077495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9b01a1920 con 0x7fa9b0101f70 2024-03-21T10:37:20.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.705+0000 7fa9ae000640 1 -- 172.21.15.31:0/2136077495 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4018990 con 0x7fa9b0101f70 2024-03-21T10:37:20.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.705+0000 7fa9b6e00640 1 -- 172.21.15.31:0/2136077495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9b01a1e90 con 0x7fa9b0101f70 2024-03-21T10:37:20.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.706+0000 7fa9b6e00640 1 -- 172.21.15.31:0/2136077495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9b004f400 con 0x7fa9b0101f70 2024-03-21T10:37:20.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.709+0000 7fa9ae000640 1 -- 172.21.15.31:0/2136077495 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fa9a4010ac0 con 0x7fa9b0101f70 2024-03-21T10:37:20.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.710+0000 7fa9ae000640 1 --2- 172.21.15.31:0/2136077495 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa9980a2fa0 0x7fa9980a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:20.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.710+0000 7fa9ae000640 1 -- 172.21.15.31:0/2136077495 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fa9a40cfa10 con 0x7fa9b0101f70 2024-03-21T10:37:20.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.710+0000 7fa9b5000640 1 --2- 172.21.15.31:0/2136077495 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa9980a2fa0 0x7fa9980a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:20.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.711+0000 7fa9b5000640 1 --2- 172.21.15.31:0/2136077495 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa9980a2fa0 0x7fa9980a5460 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa99c000c00 tx=0x7fa99c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:20.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.714+0000 7fa9ae000640 1 -- 172.21.15.31:0/2136077495 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa9a409be30 con 0x7fa9b0101f70 2024-03-21T10:37:20.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.919+0000 7fa9b6e00640 1 -- 172.21.15.31:0/2136077495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa9b00622a0 con 0x7fa9b0101f70 2024-03-21T10:37:20.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.920+0000 7fa9ae000640 1 -- 172.21.15.31:0/2136077495 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa9a409fed0 con 0x7fa9b0101f70 2024-03-21T10:37:20.917 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-03-21T10:37:20.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.922+0000 7fa9b6e00640 1 -- 172.21.15.31:0/2136077495 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa9980a2fa0 msgr2=0x7fa9980a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:20.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.922+0000 7fa9b6e00640 1 --2- 172.21.15.31:0/2136077495 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa9980a2fa0 0x7fa9980a5460 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa99c000c00 tx=0x7fa99c005fd0 comp rx=0 tx=0).stop 2024-03-21T10:37:20.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.922+0000 7fa9b6e00640 1 -- 172.21.15.31:0/2136077495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9b0101f70 msgr2=0x7fa9b0199a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:20.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.922+0000 7fa9b6e00640 1 --2- 172.21.15.31:0/2136077495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9b0101f70 0x7fa9b0199a10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa9a400e9f0 tx=0x7fa9a400eeb0 comp rx=0 tx=0).stop 2024-03-21T10:37:20.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.922+0000 7fa9b6e00640 1 -- 172.21.15.31:0/2136077495 shutdown_connections 2024-03-21T10:37:20.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.922+0000 7fa9b6e00640 1 --2- 172.21.15.31:0/2136077495 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa9980a2fa0 0x7fa9980a5460 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:20.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.922+0000 7fa9b6e00640 1 --2- 172.21.15.31:0/2136077495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9b01071d0 0x7fa9b01a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:20.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.922+0000 7fa9b6e00640 1 --2- 172.21.15.31:0/2136077495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9b01048a0 0x7fa9b0199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:20.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.922+0000 7fa9b6e00640 1 --2- 172.21.15.31:0/2136077495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9b0101f70 0x7fa9b0199a10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:20.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.922+0000 7fa9b6e00640 1 -- 172.21.15.31:0/2136077495 >> 172.21.15.31:0/2136077495 conn(0x7fa9b00fb980 msgr2=0x7fa9b0101620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:20.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.923+0000 7fa9b6e00640 1 -- 172.21.15.31:0/2136077495 shutdown_connections 2024-03-21T10:37:20.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:20.923+0000 7fa9b6e00640 1 -- 172.21.15.31:0/2136077495 wait complete. 2024-03-21T10:37:20.942 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T10:37:21.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.155+0000 7f2f37200640 1 -- 172.21.15.31:0/521176915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f381071d0 msgr2=0x7f2f38109620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:21.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.155+0000 7f2f37200640 1 --2- 172.21.15.31:0/521176915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f381071d0 0x7f2f38109620 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2f30009be0 tx=0x7f2f3002bef0 comp rx=0 tx=0).stop 2024-03-21T10:37:21.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.155+0000 7f2f37200640 1 -- 172.21.15.31:0/521176915 shutdown_connections 2024-03-21T10:37:21.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.155+0000 7f2f37200640 1 --2- 172.21.15.31:0/521176915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f381071d0 0x7f2f38109620 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:21.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.155+0000 7f2f37200640 1 --2- 172.21.15.31:0/521176915 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f381048a0 0x7f2f38106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:21.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.155+0000 7f2f37200640 1 --2- 172.21.15.31:0/521176915 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f38101f70 0x7f2f38104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:21.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.155+0000 7f2f37200640 1 -- 172.21.15.31:0/521176915 >> 172.21.15.31:0/521176915 conn(0x7f2f380fb980 msgr2=0x7f2f380fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:21.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.155+0000 7f2f37200640 1 -- 172.21.15.31:0/521176915 shutdown_connections 2024-03-21T10:37:21.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.155+0000 7f2f37200640 1 -- 172.21.15.31:0/521176915 wait complete. 2024-03-21T10:37:21.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.156+0000 7f2f37200640 1 Processor -- start 2024-03-21T10:37:21.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.157+0000 7f2f37200640 1 -- start start 2024-03-21T10:37:21.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.157+0000 7f2f37200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f38101f70 0x7f2f38199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:21.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.157+0000 7f2f37200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f381048a0 0x7f2f38199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:21.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.157+0000 7f2f37200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f381071d0 0x7f2f381a0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:21.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.157+0000 7f2f37200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3810d1d0 con 0x7f2f381048a0 2024-03-21T10:37:21.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.157+0000 7f2f37200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3810d050 con 0x7f2f381071d0 2024-03-21T10:37:21.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.157+0000 7f2f37200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3810d350 con 0x7f2f38101f70 2024-03-21T10:37:21.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.157+0000 7f2f36800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f381071d0 0x7f2f381a0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:21.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.157+0000 7f2f35e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f38101f70 0x7f2f38199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:21.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.157+0000 7f2f2fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f381048a0 0x7f2f38199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:21.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.157+0000 7f2f36800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f381071d0 0x7f2f381a0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38754/0 (socket says 172.21.15.31:38754) 2024-03-21T10:37:21.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.157+0000 7f2f36800640 1 -- 172.21.15.31:0/479179250 learned_addr learned my addr 172.21.15.31:0/479179250 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:21.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.158+0000 7f2f36800640 1 -- 172.21.15.31:0/479179250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f38101f70 msgr2=0x7f2f38199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:21.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.158+0000 7f2f36800640 1 --2- 172.21.15.31:0/479179250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f38101f70 0x7f2f38199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:21.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.158+0000 7f2f36800640 1 -- 172.21.15.31:0/479179250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f381048a0 msgr2=0x7f2f38199ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:21.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.158+0000 7f2f36800640 1 --2- 172.21.15.31:0/479179250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f381048a0 0x7f2f38199ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:21.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.158+0000 7f2f36800640 1 -- 172.21.15.31:0/479179250 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f381a15a0 con 0x7f2f381071d0 2024-03-21T10:37:21.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.158+0000 7f2f35e00640 1 --2- 172.21.15.31:0/479179250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f38101f70 0x7f2f38199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:37:21.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.158+0000 7f2f36800640 1 --2- 172.21.15.31:0/479179250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f381071d0 0x7f2f381a0f30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2f30036040 tx=0x7f2f300337b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:21.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.158+0000 7f2f2ea00640 1 -- 172.21.15.31:0/479179250 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f30043020 con 0x7f2f381071d0 2024-03-21T10:37:21.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.158+0000 7f2f2ea00640 1 -- 172.21.15.31:0/479179250 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2f30003b80 con 0x7f2f381071d0 2024-03-21T10:37:21.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.158+0000 7f2f2ea00640 1 -- 172.21.15.31:0/479179250 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f30034690 con 0x7f2f381071d0 2024-03-21T10:37:21.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.158+0000 7f2f37200640 1 -- 172.21.15.31:0/479179250 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f381a1830 con 0x7f2f381071d0 2024-03-21T10:37:21.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.159+0000 7f2f37200640 1 -- 172.21.15.31:0/479179250 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f381a1d30 con 0x7f2f381071d0 2024-03-21T10:37:21.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.160+0000 7f2f37200640 1 -- 172.21.15.31:0/479179250 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f38108cc0 con 0x7f2f381071d0 2024-03-21T10:37:21.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.162+0000 7f2f2ea00640 1 -- 172.21.15.31:0/479179250 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f2f30046020 con 0x7f2f381071d0 2024-03-21T10:37:21.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.163+0000 7f2f2ea00640 1 --2- 172.21.15.31:0/479179250 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2f1c0a2fa0 0x7f2f1c0a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:21.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.163+0000 7f2f2ea00640 1 -- 172.21.15.31:0/479179250 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f2f300f1e50 con 0x7f2f381071d0 2024-03-21T10:37:21.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.163+0000 7f2f35e00640 1 --2- 172.21.15.31:0/479179250 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2f1c0a2fa0 0x7f2f1c0a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:21.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.164+0000 7f2f35e00640 1 --2- 172.21.15.31:0/479179250 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2f1c0a2fa0 0x7f2f1c0a5460 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2f28005fd0 tx=0x7f2f28009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:21.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.167+0000 7f2f2ea00640 1 -- 172.21.15.31:0/479179250 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2f3004d020 con 0x7f2f381071d0 2024-03-21T10:37:21.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.368+0000 7f2f37200640 1 -- 172.21.15.31:0/479179250 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2f38062340 con 0x7f2f381071d0 2024-03-21T10:37:21.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.369+0000 7f2f2ea00640 1 -- 172.21.15.31:0/479179250 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2f300be8c0 con 0x7f2f381071d0 2024-03-21T10:37:21.367 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-03-21T10:37:21.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.371+0000 7f2f37200640 1 -- 172.21.15.31:0/479179250 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2f1c0a2fa0 msgr2=0x7f2f1c0a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:21.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.371+0000 7f2f37200640 1 --2- 172.21.15.31:0/479179250 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2f1c0a2fa0 0x7f2f1c0a5460 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2f28005fd0 tx=0x7f2f28009260 comp rx=0 tx=0).stop 2024-03-21T10:37:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.371+0000 7f2f37200640 1 -- 172.21.15.31:0/479179250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f381071d0 msgr2=0x7f2f381a0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.371+0000 7f2f37200640 1 --2- 172.21.15.31:0/479179250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f381071d0 0x7f2f381a0f30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2f30036040 tx=0x7f2f300337b0 comp rx=0 tx=0).stop 2024-03-21T10:37:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.372+0000 7f2f37200640 1 -- 172.21.15.31:0/479179250 shutdown_connections 2024-03-21T10:37:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.372+0000 7f2f37200640 1 --2- 172.21.15.31:0/479179250 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2f1c0a2fa0 0x7f2f1c0a5460 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.372+0000 7f2f37200640 1 --2- 172.21.15.31:0/479179250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f381071d0 0x7f2f381a0f30 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.372+0000 7f2f37200640 1 --2- 172.21.15.31:0/479179250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f381048a0 0x7f2f38199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.372+0000 7f2f37200640 1 --2- 172.21.15.31:0/479179250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f38101f70 0x7f2f38199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.372+0000 7f2f37200640 1 -- 172.21.15.31:0/479179250 >> 172.21.15.31:0/479179250 conn(0x7f2f380fb980 msgr2=0x7f2f38101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.372+0000 7f2f37200640 1 -- 172.21.15.31:0/479179250 shutdown_connections 2024-03-21T10:37:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.372+0000 7f2f37200640 1 -- 172.21.15.31:0/479179250 wait complete. 2024-03-21T10:37:21.388 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T10:37:21.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.623+0000 7f1efa400640 1 -- 172.21.15.31:0/4242664022 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ef41026d0 msgr2=0x7f1ef4104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:21.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.623+0000 7f1efa400640 1 --2- 172.21.15.31:0/4242664022 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ef41026d0 0x7f1ef4104b40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1edc009900 tx=0x7f1edc02bc30 comp rx=0 tx=0).stop 2024-03-21T10:37:21.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.623+0000 7f1efa400640 1 -- 172.21.15.31:0/4242664022 shutdown_connections 2024-03-21T10:37:21.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.623+0000 7f1efa400640 1 --2- 172.21.15.31:0/4242664022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ef4105080 0x7f1ef4105500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:21.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.623+0000 7f1efa400640 1 --2- 172.21.15.31:0/4242664022 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ef41026d0 0x7f1ef4104b40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:21.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.623+0000 7f1efa400640 1 --2- 172.21.15.31:0/4242664022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef40ffd60 0x7f1ef4102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:21.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.623+0000 7f1efa400640 1 -- 172.21.15.31:0/4242664022 >> 172.21.15.31:0/4242664022 conn(0x7f1ef40fb980 msgr2=0x7f1ef40fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:21.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.623+0000 7f1efa400640 1 -- 172.21.15.31:0/4242664022 shutdown_connections 2024-03-21T10:37:21.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.623+0000 7f1efa400640 1 -- 172.21.15.31:0/4242664022 wait complete. 2024-03-21T10:37:21.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.625+0000 7f1efa400640 1 Processor -- start 2024-03-21T10:37:21.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.625+0000 7f1efa400640 1 -- start start 2024-03-21T10:37:21.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.625+0000 7f1efa400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ef40ffd60 0x7f1ef419de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:21.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.625+0000 7f1efa400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef41026d0 0x7f1ef419e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:21.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.625+0000 7f1efa400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ef4105080 0x7f1ef41a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:21.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.625+0000 7f1efa400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef4113740 con 0x7f1ef40ffd60 2024-03-21T10:37:21.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.625+0000 7f1efa400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef41135c0 con 0x7f1ef41026d0 2024-03-21T10:37:21.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.625+0000 7f1efa400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef41138c0 con 0x7f1ef4105080 2024-03-21T10:37:21.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.626+0000 7f1eebe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef41026d0 0x7f1ef419e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:21.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.626+0000 7f1eebe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef41026d0 0x7f1ef419e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38766/0 (socket says 172.21.15.31:38766) 2024-03-21T10:37:21.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.626+0000 7f1eebe00640 1 -- 172.21.15.31:0/566383705 learned_addr learned my addr 172.21.15.31:0/566383705 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:21.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.626+0000 7f1ef9a00640 1 --2- 172.21.15.31:0/566383705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ef4105080 0x7f1ef41a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:21.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.626+0000 7f1ef9000640 1 --2- 172.21.15.31:0/566383705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ef40ffd60 0x7f1ef419de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:21.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.626+0000 7f1eebe00640 1 -- 172.21.15.31:0/566383705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ef4105080 msgr2=0x7f1ef41a5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:21.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.626+0000 7f1eebe00640 1 --2- 172.21.15.31:0/566383705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ef4105080 0x7f1ef41a5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:21.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.626+0000 7f1eebe00640 1 -- 172.21.15.31:0/566383705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ef40ffd60 msgr2=0x7f1ef419de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:21.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.626+0000 7f1eebe00640 1 --2- 172.21.15.31:0/566383705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ef40ffd60 0x7f1ef419de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:21.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.626+0000 7f1eebe00640 1 -- 172.21.15.31:0/566383705 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef41a59f0 con 0x7f1ef41026d0 2024-03-21T10:37:21.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.626+0000 7f1eebe00640 1 --2- 172.21.15.31:0/566383705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef41026d0 0x7f1ef419e360 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1edc009230 tx=0x7f1edc005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:21.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.627+0000 7f1ef2000640 1 -- 172.21.15.31:0/566383705 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1edc041070 con 0x7f1ef41026d0 2024-03-21T10:37:21.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.627+0000 7f1ef2000640 1 -- 172.21.15.31:0/566383705 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1edc0044d0 con 0x7f1ef41026d0 2024-03-21T10:37:21.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.627+0000 7f1ef2000640 1 -- 172.21.15.31:0/566383705 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1edc034440 con 0x7f1ef41026d0 2024-03-21T10:37:21.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.627+0000 7f1efa400640 1 -- 172.21.15.31:0/566383705 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ef41a5c80 con 0x7f1ef41026d0 2024-03-21T10:37:21.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.627+0000 7f1efa400640 1 -- 172.21.15.31:0/566383705 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ef41a6120 con 0x7f1ef41026d0 2024-03-21T10:37:21.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.628+0000 7f1efa400640 1 -- 172.21.15.31:0/566383705 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ef40fd470 con 0x7f1ef41026d0 2024-03-21T10:37:21.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.630+0000 7f1ef2000640 1 -- 172.21.15.31:0/566383705 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f1edc044070 con 0x7f1ef41026d0 2024-03-21T10:37:21.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.632+0000 7f1ef2000640 1 --2- 172.21.15.31:0/566383705 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1ed80a2fa0 0x7f1ed80a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:21.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.632+0000 7f1ef2000640 1 -- 172.21.15.31:0/566383705 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f1edc0f12e0 con 0x7f1ef41026d0 2024-03-21T10:37:21.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.632+0000 7f1ef9000640 1 --2- 172.21.15.31:0/566383705 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1ed80a2fa0 0x7f1ed80a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:21.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.633+0000 7f1ef9000640 1 --2- 172.21.15.31:0/566383705 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1ed80a2fa0 0x7f1ed80a5460 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4005fd0 tx=0x7f1ee4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:21.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.635+0000 7f1ef2000640 1 -- 172.21.15.31:0/566383705 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1edc0bd700 con 0x7f1ef41026d0 2024-03-21T10:37:21.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:21 smithi040 ceph-mon[36154]: pgmap v396: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:37:21.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:21 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/826963604' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:37:21.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:21 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2136077495' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:37:21.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:21 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/479179250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:37:21.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:21 smithi031 ceph-mon[31673]: pgmap v396: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:37:21.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:21 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/826963604' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:37:21.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:21 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2136077495' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:37:21.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:21 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/479179250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:37:21.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:21 smithi070 ceph-mon[36330]: pgmap v396: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:37:21.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:21 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/826963604' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:37:21.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:21 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2136077495' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:37:21.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:21 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/479179250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:37:21.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.837+0000 7f1efa400640 1 -- 172.21.15.31:0/566383705 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1ef40622a0 con 0x7f1ef41026d0 2024-03-21T10:37:21.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.838+0000 7f1ef2000640 1 -- 172.21.15.31:0/566383705 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1edc0c17a0 con 0x7f1ef41026d0 2024-03-21T10:37:21.836 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-03-21T10:37:21.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.842+0000 7f1efa400640 1 -- 172.21.15.31:0/566383705 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1ed80a2fa0 msgr2=0x7f1ed80a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:21.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.842+0000 7f1efa400640 1 --2- 172.21.15.31:0/566383705 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1ed80a2fa0 0x7f1ed80a5460 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4005fd0 tx=0x7f1ee4009500 comp rx=0 tx=0).stop 2024-03-21T10:37:21.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.842+0000 7f1efa400640 1 -- 172.21.15.31:0/566383705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef41026d0 msgr2=0x7f1ef419e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:21.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.842+0000 7f1efa400640 1 --2- 172.21.15.31:0/566383705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef41026d0 0x7f1ef419e360 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1edc009230 tx=0x7f1edc005d50 comp rx=0 tx=0).stop 2024-03-21T10:37:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.842+0000 7f1efa400640 1 -- 172.21.15.31:0/566383705 shutdown_connections 2024-03-21T10:37:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.842+0000 7f1efa400640 1 --2- 172.21.15.31:0/566383705 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1ed80a2fa0 0x7f1ed80a5460 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.842+0000 7f1efa400640 1 --2- 172.21.15.31:0/566383705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ef4105080 0x7f1ef41a5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.842+0000 7f1efa400640 1 --2- 172.21.15.31:0/566383705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef41026d0 0x7f1ef419e360 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.842+0000 7f1efa400640 1 --2- 172.21.15.31:0/566383705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ef40ffd60 0x7f1ef419de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.842+0000 7f1efa400640 1 -- 172.21.15.31:0/566383705 >> 172.21.15.31:0/566383705 conn(0x7f1ef40fb980 msgr2=0x7f1ef4100ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.842+0000 7f1efa400640 1 -- 172.21.15.31:0/566383705 shutdown_connections 2024-03-21T10:37:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:21.842+0000 7f1efa400640 1 -- 172.21.15.31:0/566383705 wait complete. 2024-03-21T10:37:21.858 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T10:37:22.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.096+0000 7f2bbfe00640 1 -- 172.21.15.31:0/3522597375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bc0107b00 msgr2=0x7f2bc0107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:22.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.096+0000 7f2bbfe00640 1 --2- 172.21.15.31:0/3522597375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bc0107b00 0x7f2bc0107f20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0009900 tx=0x7f2bb002bc50 comp rx=0 tx=0).stop 2024-03-21T10:37:22.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.096+0000 7f2bbfe00640 1 -- 172.21.15.31:0/3522597375 shutdown_connections 2024-03-21T10:37:22.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.096+0000 7f2bbfe00640 1 --2- 172.21.15.31:0/3522597375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bc01096b0 0x7f2bc010ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:22.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.096+0000 7f2bbfe00640 1 --2- 172.21.15.31:0/3522597375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bc0108cf0 0x7f2bc0109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:22.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.096+0000 7f2bbfe00640 1 --2- 172.21.15.31:0/3522597375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bc0107b00 0x7f2bc0107f20 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:22.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.096+0000 7f2bbfe00640 1 -- 172.21.15.31:0/3522597375 >> 172.21.15.31:0/3522597375 conn(0x7f2bc0076a90 msgr2=0x7f2bc0078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:22.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.096+0000 7f2bbfe00640 1 -- 172.21.15.31:0/3522597375 shutdown_connections 2024-03-21T10:37:22.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.096+0000 7f2bbfe00640 1 -- 172.21.15.31:0/3522597375 wait complete. 2024-03-21T10:37:22.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.097+0000 7f2bbfe00640 1 Processor -- start 2024-03-21T10:37:22.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.097+0000 7f2bbfe00640 1 -- start start 2024-03-21T10:37:22.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.097+0000 7f2bbfe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bc0107b00 0x7f2bc019de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:22.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.097+0000 7f2bbfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bc0108cf0 0x7f2bc019e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:22.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.097+0000 7f2bbfe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bc01096b0 0x7f2bc01a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:22.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.097+0000 7f2bbfe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc01137a0 con 0x7f2bc01096b0 2024-03-21T10:37:22.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.097+0000 7f2bbfe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc0113620 con 0x7f2bc0108cf0 2024-03-21T10:37:22.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.097+0000 7f2bbfe00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc0113920 con 0x7f2bc0107b00 2024-03-21T10:37:22.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.098+0000 7f2bbe000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bc0108cf0 0x7f2bc019e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:22.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.098+0000 7f2bbea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bc0107b00 0x7f2bc019de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:22.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.098+0000 7f2bbe000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bc0108cf0 0x7f2bc019e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38792/0 (socket says 172.21.15.31:38792) 2024-03-21T10:37:22.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.098+0000 7f2bbea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bc0107b00 0x7f2bc019de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:47354/0 (socket says 172.21.15.31:47354) 2024-03-21T10:37:22.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.098+0000 7f2bbe000640 1 -- 172.21.15.31:0/2504338426 learned_addr learned my addr 172.21.15.31:0/2504338426 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:22.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.098+0000 7f2bbf400640 1 --2- 172.21.15.31:0/2504338426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bc01096b0 0x7f2bc01a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:22.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.098+0000 7f2bbe000640 1 -- 172.21.15.31:0/2504338426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bc0107b00 msgr2=0x7f2bc019de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:22.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.098+0000 7f2bbe000640 1 --2- 172.21.15.31:0/2504338426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bc0107b00 0x7f2bc019de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:22.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.098+0000 7f2bbe000640 1 -- 172.21.15.31:0/2504338426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bc01096b0 msgr2=0x7f2bc01a5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:22.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.098+0000 7f2bbe000640 1 --2- 172.21.15.31:0/2504338426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bc01096b0 0x7f2bc01a5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:22.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.098+0000 7f2bbe000640 1 -- 172.21.15.31:0/2504338426 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc01a5a70 con 0x7f2bc0108cf0 2024-03-21T10:37:22.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.098+0000 7f2bbe000640 1 --2- 172.21.15.31:0/2504338426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bc0108cf0 0x7f2bc019e390 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2bac00ea60 tx=0x7f2bac00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:22.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.099+0000 7f2bb7400640 1 -- 172.21.15.31:0/2504338426 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bac00cde0 con 0x7f2bc0108cf0 2024-03-21T10:37:22.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.099+0000 7f2bb7400640 1 -- 172.21.15.31:0/2504338426 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2bac0035f0 con 0x7f2bc0108cf0 2024-03-21T10:37:22.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.099+0000 7f2bbfe00640 1 -- 172.21.15.31:0/2504338426 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc01a5d60 con 0x7f2bc0108cf0 2024-03-21T10:37:22.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.099+0000 7f2bb7400640 1 -- 172.21.15.31:0/2504338426 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bac010670 con 0x7f2bc0108cf0 2024-03-21T10:37:22.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.099+0000 7f2bbfe00640 1 -- 172.21.15.31:0/2504338426 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc01a6270 con 0x7f2bc0108cf0 2024-03-21T10:37:22.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.100+0000 7f2bbfe00640 1 -- 172.21.15.31:0/2504338426 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc010eea0 con 0x7f2bc0108cf0 2024-03-21T10:37:22.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.102+0000 7f2bb7400640 1 -- 172.21.15.31:0/2504338426 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f2bac0037a0 con 0x7f2bc0108cf0 2024-03-21T10:37:22.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.104+0000 7f2bb7400640 1 --2- 172.21.15.31:0/2504338426 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2ba40a2fa0 0x7f2ba40a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:22.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.104+0000 7f2bb7400640 1 -- 172.21.15.31:0/2504338426 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f2bac0cf990 con 0x7f2bc0108cf0 2024-03-21T10:37:22.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.104+0000 7f2bbea00640 1 --2- 172.21.15.31:0/2504338426 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2ba40a2fa0 0x7f2ba40a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:22.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.105+0000 7f2bbea00640 1 --2- 172.21.15.31:0/2504338426 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2ba40a2fa0 0x7f2ba40a5460 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2bb002c190 tx=0x7f2bb0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:22.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.108+0000 7f2bb7400640 1 -- 172.21.15.31:0/2504338426 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2bac09c400 con 0x7f2bc0108cf0 2024-03-21T10:37:22.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.309+0000 7f2bbfe00640 1 -- 172.21.15.31:0/2504338426 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f2bc00623a0 con 0x7f2bc0108cf0 2024-03-21T10:37:22.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.310+0000 7f2bb7400640 1 -- 172.21.15.31:0/2504338426 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7f2bac09bd00 con 0x7f2bc0108cf0 2024-03-21T10:37:22.308 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-03-21T10:37:22.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.312+0000 7f2bbfe00640 1 -- 172.21.15.31:0/2504338426 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2ba40a2fa0 msgr2=0x7f2ba40a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:22.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.312+0000 7f2bbfe00640 1 --2- 172.21.15.31:0/2504338426 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2ba40a2fa0 0x7f2ba40a5460 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2bb002c190 tx=0x7f2bb0000bc0 comp rx=0 tx=0).stop 2024-03-21T10:37:22.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.312+0000 7f2bbfe00640 1 -- 172.21.15.31:0/2504338426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bc0108cf0 msgr2=0x7f2bc019e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:22.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.312+0000 7f2bbfe00640 1 --2- 172.21.15.31:0/2504338426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bc0108cf0 0x7f2bc019e390 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2bac00ea60 tx=0x7f2bac00ef20 comp rx=0 tx=0).stop 2024-03-21T10:37:22.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.313+0000 7f2bbfe00640 1 -- 172.21.15.31:0/2504338426 shutdown_connections 2024-03-21T10:37:22.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.313+0000 7f2bbfe00640 1 --2- 172.21.15.31:0/2504338426 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2ba40a2fa0 0x7f2ba40a5460 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:22.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.313+0000 7f2bbfe00640 1 --2- 172.21.15.31:0/2504338426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bc01096b0 0x7f2bc01a5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:22.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.313+0000 7f2bbfe00640 1 --2- 172.21.15.31:0/2504338426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bc0108cf0 0x7f2bc019e390 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:22.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.313+0000 7f2bbfe00640 1 --2- 172.21.15.31:0/2504338426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bc0107b00 0x7f2bc019de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:22.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.313+0000 7f2bbfe00640 1 -- 172.21.15.31:0/2504338426 >> 172.21.15.31:0/2504338426 conn(0x7f2bc0076a90 msgr2=0x7f2bc010c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:22.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.313+0000 7f2bbfe00640 1 -- 172.21.15.31:0/2504338426 shutdown_connections 2024-03-21T10:37:22.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.313+0000 7f2bbfe00640 1 -- 172.21.15.31:0/2504338426 wait complete. 2024-03-21T10:37:22.331 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:37:22.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.544+0000 7f9509800640 1 -- 172.21.15.31:0/1445033717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9504103940 msgr2=0x7f9504103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:22.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.544+0000 7f9509800640 1 --2- 172.21.15.31:0/1445033717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9504103940 0x7f9504103d60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f94f4009900 tx=0x7f94f402bc30 comp rx=0 tx=0).stop 2024-03-21T10:37:22.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.544+0000 7f9509800640 1 -- 172.21.15.31:0/1445033717 shutdown_connections 2024-03-21T10:37:22.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.544+0000 7f9509800640 1 --2- 172.21.15.31:0/1445033717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95041054f0 0x7f950410bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:22.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.544+0000 7f9509800640 1 --2- 172.21.15.31:0/1445033717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9504104b30 0x7f9504104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:22.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.544+0000 7f9509800640 1 --2- 172.21.15.31:0/1445033717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9504103940 0x7f9504103d60 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:22.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.544+0000 7f9509800640 1 -- 172.21.15.31:0/1445033717 >> 172.21.15.31:0/1445033717 conn(0x7f95040feeb0 msgr2=0x7f95041012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:22.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.544+0000 7f9509800640 1 -- 172.21.15.31:0/1445033717 shutdown_connections 2024-03-21T10:37:22.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.545+0000 7f9509800640 1 -- 172.21.15.31:0/1445033717 wait complete. 2024-03-21T10:37:22.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.546+0000 7f9509800640 1 Processor -- start 2024-03-21T10:37:22.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.546+0000 7f9509800640 1 -- start start 2024-03-21T10:37:22.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.546+0000 7f9509800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9504103940 0x7f9504199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:22.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.546+0000 7f9509800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9504104b30 0x7f950419a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:22.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.546+0000 7f9509800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95041054f0 0x7f95041a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:22.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.546+0000 7f9509800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950410f570 con 0x7f95041054f0 2024-03-21T10:37:22.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.546+0000 7f9509800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950410f3f0 con 0x7f9504104b30 2024-03-21T10:37:22.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.546+0000 7f9509800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950410f6f0 con 0x7f9504103940 2024-03-21T10:37:22.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.547+0000 7f9502a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9504103940 0x7f9504199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:22.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.547+0000 7f94fbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9504104b30 0x7f950419a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:22.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.547+0000 7f9502a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9504103940 0x7f9504199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:47360/0 (socket says 172.21.15.31:47360) 2024-03-21T10:37:22.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.547+0000 7f9502a00640 1 -- 172.21.15.31:0/1584836396 learned_addr learned my addr 172.21.15.31:0/1584836396 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:22.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.547+0000 7f9503400640 1 --2- 172.21.15.31:0/1584836396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95041054f0 0x7f95041a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:22.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.547+0000 7f9502a00640 1 -- 172.21.15.31:0/1584836396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9504104b30 msgr2=0x7f950419a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:22.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.547+0000 7f9502a00640 1 --2- 172.21.15.31:0/1584836396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9504104b30 0x7f950419a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:22.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.547+0000 7f9502a00640 1 -- 172.21.15.31:0/1584836396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95041054f0 msgr2=0x7f95041a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:22.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.547+0000 7f9502a00640 1 --2- 172.21.15.31:0/1584836396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95041054f0 0x7f95041a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:22.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.547+0000 7f9502a00640 1 -- 172.21.15.31:0/1584836396 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95041a17e0 con 0x7f9504103940 2024-03-21T10:37:22.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.547+0000 7f9502a00640 1 --2- 172.21.15.31:0/1584836396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9504103940 0x7f9504199bc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f94f4000c00 tx=0x7f94f402c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:22.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.548+0000 7f94fb400640 1 -- 172.21.15.31:0/1584836396 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f4043020 con 0x7f9504103940 2024-03-21T10:37:22.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.548+0000 7f94fb400640 1 -- 172.21.15.31:0/1584836396 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f94f40054e0 con 0x7f9504103940 2024-03-21T10:37:22.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.548+0000 7f94fb400640 1 -- 172.21.15.31:0/1584836396 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f4003ce0 con 0x7f9504103940 2024-03-21T10:37:22.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.548+0000 7f9509800640 1 -- 172.21.15.31:0/1584836396 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95041a1a70 con 0x7f9504103940 2024-03-21T10:37:22.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.548+0000 7f9509800640 1 -- 172.21.15.31:0/1584836396 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95041a1f40 con 0x7f9504103940 2024-03-21T10:37:22.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.549+0000 7f9509800640 1 -- 172.21.15.31:0/1584836396 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f950410ac60 con 0x7f9504103940 2024-03-21T10:37:22.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.551+0000 7f94fb400640 1 -- 172.21.15.31:0/1584836396 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f94f4046020 con 0x7f9504103940 2024-03-21T10:37:22.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.552+0000 7f94fb400640 1 --2- 172.21.15.31:0/1584836396 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f94e80a2fa0 0x7f94e80a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:22.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.552+0000 7f94fb400640 1 -- 172.21.15.31:0/1584836396 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f94f40f18a0 con 0x7f9504103940 2024-03-21T10:37:22.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.552+0000 7f94fbe00640 1 --2- 172.21.15.31:0/1584836396 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f94e80a2fa0 0x7f94e80a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:22.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.553+0000 7f94fbe00640 1 --2- 172.21.15.31:0/1584836396 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f94e80a2fa0 0x7f94e80a5460 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f950419b0d0 tx=0x7f94ec00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:22.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.556+0000 7f94fb400640 1 -- 172.21.15.31:0/1584836396 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f94f404d020 con 0x7f9504103940 2024-03-21T10:37:22.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:22 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/566383705' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:37:22.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:22 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2504338426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:37:22.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:22 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/566383705' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:37:22.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:22 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2504338426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:37:22.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:22 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/566383705' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:37:22.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:22 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2504338426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:37:22.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.829+0000 7f9509800640 1 -- 172.21.15.31:0/1584836396 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95040622d0 con 0x7f9504103940 2024-03-21T10:37:22.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.830+0000 7f94fb400640 1 -- 172.21.15.31:0/1584836396 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11384 (secure 0 0 0) 0x7f94f40be310 con 0x7f9504103940 2024-03-21T10:37:22.827 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:37:22.828 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:35:40.990205+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24596,24547]},"id":1}]} 2024-03-21T10:37:22.828 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2024-03-21T10:37:22.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.832+0000 7f9509800640 1 -- 172.21.15.31:0/1584836396 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f94e80a2fa0 msgr2=0x7f94e80a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:22.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.832+0000 7f9509800640 1 --2- 172.21.15.31:0/1584836396 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f94e80a2fa0 0x7f94e80a5460 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f950419b0d0 tx=0x7f94ec00b390 comp rx=0 tx=0).stop 2024-03-21T10:37:22.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.832+0000 7f9509800640 1 -- 172.21.15.31:0/1584836396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9504103940 msgr2=0x7f9504199bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:22.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.832+0000 7f9509800640 1 --2- 172.21.15.31:0/1584836396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9504103940 0x7f9504199bc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f94f4000c00 tx=0x7f94f402c6c0 comp rx=0 tx=0).stop 2024-03-21T10:37:22.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.833+0000 7f9509800640 1 -- 172.21.15.31:0/1584836396 shutdown_connections 2024-03-21T10:37:22.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.833+0000 7f9509800640 1 --2- 172.21.15.31:0/1584836396 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f94e80a2fa0 0x7f94e80a5460 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:22.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.833+0000 7f9509800640 1 --2- 172.21.15.31:0/1584836396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95041054f0 0x7f95041a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:22.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.833+0000 7f9509800640 1 --2- 172.21.15.31:0/1584836396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9504104b30 0x7f950419a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:22.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.833+0000 7f9509800640 1 --2- 172.21.15.31:0/1584836396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9504103940 0x7f9504199bc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:22.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.833+0000 7f9509800640 1 -- 172.21.15.31:0/1584836396 >> 172.21.15.31:0/1584836396 conn(0x7f95040feeb0 msgr2=0x7f9504107e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:22.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.833+0000 7f9509800640 1 -- 172.21.15.31:0/1584836396 shutdown_connections 2024-03-21T10:37:22.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:22.833+0000 7f9509800640 1 -- 172.21.15.31:0/1584836396 wait complete. 2024-03-21T10:37:22.851 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T10:37:23.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.067+0000 7f062be00640 1 -- 172.21.15.31:0/853701073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f062c109470 msgr2=0x7f062c10b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:23.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.067+0000 7f062be00640 1 --2- 172.21.15.31:0/853701073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f062c109470 0x7f062c10b900 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0624009be0 tx=0x7f062402bf80 comp rx=0 tx=0).stop 2024-03-21T10:37:23.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.067+0000 7f062be00640 1 -- 172.21.15.31:0/853701073 shutdown_connections 2024-03-21T10:37:23.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.067+0000 7f062be00640 1 --2- 172.21.15.31:0/853701073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f062c109470 0x7f062c10b900 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:23.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.067+0000 7f062be00640 1 --2- 172.21.15.31:0/853701073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f062c074080 0x7f062c0744e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:23.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.067+0000 7f062be00640 1 --2- 172.21.15.31:0/853701073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f062c075660 0x7f062c073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:23.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.067+0000 7f062be00640 1 -- 172.21.15.31:0/853701073 >> 172.21.15.31:0/853701073 conn(0x7f062c0fcb30 msgr2=0x7f062c0fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:23.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.067+0000 7f062be00640 1 -- 172.21.15.31:0/853701073 shutdown_connections 2024-03-21T10:37:23.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.067+0000 7f062be00640 1 -- 172.21.15.31:0/853701073 wait complete. 2024-03-21T10:37:23.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.068+0000 7f062be00640 1 Processor -- start 2024-03-21T10:37:23.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.068+0000 7f062be00640 1 -- start start 2024-03-21T10:37:23.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.069+0000 7f062be00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f062c074080 0x7f062c199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:23.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.069+0000 7f062be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f062c075660 0x7f062c199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:23.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.069+0000 7f062be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f062c109470 0x7f062c1a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:23.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.069+0000 7f062be00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f062c10f480 con 0x7f062c109470 2024-03-21T10:37:23.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.069+0000 7f062be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f062c10f300 con 0x7f062c075660 2024-03-21T10:37:23.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.069+0000 7f062be00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f062c10f600 con 0x7f062c074080 2024-03-21T10:37:23.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.069+0000 7f062aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f062c074080 0x7f062c199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:23.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.069+0000 7f0623e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f062c075660 0x7f062c199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:23.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.069+0000 7f062b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f062c109470 0x7f062c1a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:23.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.069+0000 7f0623e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f062c075660 0x7f062c199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38822/0 (socket says 172.21.15.31:38822) 2024-03-21T10:37:23.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.069+0000 7f0623e00640 1 -- 172.21.15.31:0/3841372905 learned_addr learned my addr 172.21.15.31:0/3841372905 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:23.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.069+0000 7f062aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f062c074080 0x7f062c199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:47382/0 (socket says 172.21.15.31:47382) 2024-03-21T10:37:23.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.069+0000 7f062aa00640 1 -- 172.21.15.31:0/3841372905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f062c075660 msgr2=0x7f062c199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:23.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.069+0000 7f062aa00640 1 --2- 172.21.15.31:0/3841372905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f062c075660 0x7f062c199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:23.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.069+0000 7f062aa00640 1 -- 172.21.15.31:0/3841372905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f062c109470 msgr2=0x7f062c1a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:23.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.069+0000 7f062aa00640 1 --2- 172.21.15.31:0/3841372905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f062c109470 0x7f062c1a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:23.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.069+0000 7f062aa00640 1 -- 172.21.15.31:0/3841372905 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f062c1a1630 con 0x7f062c074080 2024-03-21T10:37:23.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.070+0000 7f062aa00640 1 --2- 172.21.15.31:0/3841372905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f062c074080 0x7f062c199a10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f061c00e790 tx=0x7f061c00ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:23.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.070+0000 7f0623400640 1 -- 172.21.15.31:0/3841372905 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061c00eea0 con 0x7f062c074080 2024-03-21T10:37:23.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.070+0000 7f0623400640 1 -- 172.21.15.31:0/3841372905 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f061c00ccc0 con 0x7f062c074080 2024-03-21T10:37:23.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.070+0000 7f062be00640 1 -- 172.21.15.31:0/3841372905 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f062c1a1920 con 0x7f062c074080 2024-03-21T10:37:23.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.070+0000 7f0623400640 1 -- 172.21.15.31:0/3841372905 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061c018760 con 0x7f062c074080 2024-03-21T10:37:23.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.070+0000 7f062be00640 1 -- 172.21.15.31:0/3841372905 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f062c1a1ec0 con 0x7f062c074080 2024-03-21T10:37:23.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.071+0000 7f062be00640 1 -- 172.21.15.31:0/3841372905 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f062c04f470 con 0x7f062c074080 2024-03-21T10:37:23.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.073+0000 7f0623400640 1 -- 172.21.15.31:0/3841372905 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f061c010430 con 0x7f062c074080 2024-03-21T10:37:23.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.074+0000 7f0623400640 1 --2- 172.21.15.31:0/3841372905 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f06100a2f90 0x7f06100a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:23.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.075+0000 7f0623400640 1 -- 172.21.15.31:0/3841372905 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f061c0cf8b0 con 0x7f062c074080 2024-03-21T10:37:23.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.075+0000 7f0623e00640 1 --2- 172.21.15.31:0/3841372905 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f06100a2f90 0x7f06100a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:23.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.075+0000 7f0623e00640 1 --2- 172.21.15.31:0/3841372905 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f06100a2f90 0x7f06100a5450 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f062c19af20 tx=0x7f0614005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:23.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.078+0000 7f0623400640 1 -- 172.21.15.31:0/3841372905 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f061c09bcd0 con 0x7f062c074080 2024-03-21T10:37:23.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.282+0000 7f062be00640 1 -- 172.21.15.31:0/3841372905 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f062c0622a0 con 0x7f062c074080 2024-03-21T10:37:23.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.284+0000 7f0623400640 1 -- 172.21.15.31:0/3841372905 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24553 (secure 0 0 0) 0x7f061c09fd70 con 0x7f062c074080 2024-03-21T10:37:23.282 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:37:23.283 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":93,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","created":"2024-03-21T10:25:41.971117+0000","modified":"2024-03-21T10:36:56.301695+0000","last_up_change":"2024-03-21T10:34:07.757196+0000","last_in_change":"2024-03-21T10:33:47.460406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T10:29:44.553968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T10:35:04.945449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T10:35:05.970589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T10:35:09.688711+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e0244d33-00e2-47d5-9f66-34efc138c462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6803","nonce":3089885125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6805","nonce":3089885125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6809","nonce":3089885125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6807","nonce":3089885125}]},"public_addr":"172.21.15.31:6803/3089885125","cluster_addr":"172.21.15.31:6805/3089885125","heartbeat_back_addr":"172.21.15.31:6809/3089885125","heartbeat_front_addr":"172.21.15.31:6807/3089885125","state":["exists","up"]},{"osd":1,"uuid":"76a63504-beab-4354-a6c3-eebc3b16ad03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6801","nonce":618840037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6803","nonce":618840037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6807","nonce":618840037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6805","nonce":618840037}]},"public_addr":"172.21.15.40:6801/618840037","cluster_addr":"172.21.15.40:6803/618840037","heartbeat_back_addr":"172.21.15.40:6807/618840037","heartbeat_front_addr":"172.21.15.40:6805/618840037","state":["exists","up"]},{"osd":2,"uuid":"fa191bd6-9a21-4ee2-8adf-8a63c9cfd821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6801","nonce":2524184117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6803","nonce":2524184117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6807","nonce":2524184117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6805","nonce":2524184117}]},"public_addr":"172.21.15.70:6801/2524184117","cluster_addr":"172.21.15.70:6803/2524184117","heartbeat_back_addr":"172.21.15.70:6807/2524184117","heartbeat_front_addr":"172.21.15.70:6805/2524184117","state":["exists","up"]},{"osd":3,"uuid":"edb44507-ffb0-42a4-827b-08461e1bf952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6811","nonce":2823036296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6813","nonce":2823036296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6817","nonce":2823036296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6815","nonce":2823036296}]},"public_addr":"172.21.15.31:6811/2823036296","cluster_addr":"172.21.15.31:6813/2823036296","heartbeat_back_addr":"172.21.15.31:6817/2823036296","heartbeat_front_addr":"172.21.15.31:6815/2823036296","state":["exists","up"]},{"osd":4,"uuid":"8cb89153-8c3b-431f-b57c-8863fada76dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6809","nonce":4215186309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6811","nonce":4215186309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6815","nonce":4215186309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6813","nonce":4215186309}]},"public_addr":"172.21.15.40:6809/4215186309","cluster_addr":"172.21.15.40:6811/4215186309","heartbeat_back_addr":"172.21.15.40:6815/4215186309","heartbeat_front_addr":"172.21.15.40:6813/4215186309","state":["exists","up"]},{"osd":5,"uuid":"9409505a-d463-408c-a3cd-218380c04d68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6809","nonce":3402830452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6811","nonce":3402830452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6815","nonce":3402830452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6813","nonce":3402830452}]},"public_addr":"172.21.15.70:6809/3402830452","cluster_addr":"172.21.15.70:6811/3402830452","heartbeat_back_addr":"172.21.15.70:6815/3402830452","heartbeat_front_addr":"172.21.15.70:6813/3402830452","state":["exists","up"]},{"osd":6,"uuid":"f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6819","nonce":1193497058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6821","nonce":1193497058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6825","nonce":1193497058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6823","nonce":1193497058}]},"public_addr":"172.21.15.31:6819/1193497058","cluster_addr":"172.21.15.31:6821/1193497058","heartbeat_back_addr":"172.21.15.31:6825/1193497058","heartbeat_front_addr":"172.21.15.31:6823/1193497058","state":["exists","up"]},{"osd":7,"uuid":"db228a87-29a5-45d4-8fe6-a0a554262eb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6817","nonce":3825628119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6819","nonce":3825628119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6823","nonce":3825628119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6821","nonce":3825628119}]},"public_addr":"172.21.15.40:6817/3825628119","cluster_addr":"172.21.15.40:6819/3825628119","heartbeat_back_addr":"172.21.15.40:6823/3825628119","heartbeat_front_addr":"172.21.15.40:6821/3825628119","state":["exists","up"]},{"osd":8,"uuid":"59b9fde0-095d-4e0b-87da-ead37bd8ff1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6817","nonce":534880586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6819","nonce":534880586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6823","nonce":534880586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6821","nonce":534880586}]},"public_addr":"172.21.15.70:6817/534880586","cluster_addr":"172.21.15.70:6819/534880586","heartbeat_back_addr":"172.21.15.70:6823/534880586","heartbeat_front_addr":"172.21.15.70:6821/534880586","state":["exists","up"]},{"osd":9,"uuid":"77f47061-0c5f-4159-b276-e814935d6c1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6827","nonce":587047469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6829","nonce":587047469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6833","nonce":587047469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6831","nonce":587047469}]},"public_addr":"172.21.15.31:6827/587047469","cluster_addr":"172.21.15.31:6829/587047469","heartbeat_back_addr":"172.21.15.31:6833/587047469","heartbeat_front_addr":"172.21.15.31:6831/587047469","state":["exists","up"]},{"osd":10,"uuid":"8853dc43-7bd1-4c93-8063-f7982fd7da38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6825","nonce":512108711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6827","nonce":512108711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6831","nonce":512108711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6829","nonce":512108711}]},"public_addr":"172.21.15.40:6825/512108711","cluster_addr":"172.21.15.40:6827/512108711","heartbeat_back_addr":"172.21.15.40:6831/512108711","heartbeat_front_addr":"172.21.15.40:6829/512108711","state":["exists","up"]},{"osd":11,"uuid":"84b7c2de-c424-45a6-9583-60097e372a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6825","nonce":540305718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6827","nonce":540305718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6831","nonce":540305718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6829","nonce":540305718}]},"public_addr":"172.21.15.70:6825/540305718","cluster_addr":"172.21.15.70:6827/540305718","heartbeat_back_addr":"172.21.15.70:6831/540305718","heartbeat_front_addr":"172.21.15.70:6829/540305718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:28:51.477358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:17.405045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:41.626268+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:15.339568+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:42.676996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:07.516999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:42.412273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:10.365041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:35.699120+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:11.733639+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:39.505442+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:34:05.449301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451930554":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/3342588843":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6801/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/1626448389":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/1910059351":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/344822770":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/3811368512":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/1981437651":"2024-03-22T10:26:16.959165+0000","172.21.15.31:6801/1981437651":"2024-03-22T10:26:16.959165+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T10:37:23.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.287+0000 7f062be00640 1 -- 172.21.15.31:0/3841372905 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f06100a2f90 msgr2=0x7f06100a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:23.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.287+0000 7f062be00640 1 --2- 172.21.15.31:0/3841372905 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f06100a2f90 0x7f06100a5450 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f062c19af20 tx=0x7f0614005f90 comp rx=0 tx=0).stop 2024-03-21T10:37:23.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.287+0000 7f062be00640 1 -- 172.21.15.31:0/3841372905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f062c074080 msgr2=0x7f062c199a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:23.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.287+0000 7f062be00640 1 --2- 172.21.15.31:0/3841372905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f062c074080 0x7f062c199a10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f061c00e790 tx=0x7f061c00ec50 comp rx=0 tx=0).stop 2024-03-21T10:37:23.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.287+0000 7f062be00640 1 -- 172.21.15.31:0/3841372905 shutdown_connections 2024-03-21T10:37:23.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.287+0000 7f062be00640 1 --2- 172.21.15.31:0/3841372905 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f06100a2f90 0x7f06100a5450 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:23.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.287+0000 7f062be00640 1 --2- 172.21.15.31:0/3841372905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f062c109470 0x7f062c1a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:23.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.287+0000 7f062be00640 1 --2- 172.21.15.31:0/3841372905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f062c075660 0x7f062c199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:23.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.287+0000 7f062be00640 1 --2- 172.21.15.31:0/3841372905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f062c074080 0x7f062c199a10 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:23.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.287+0000 7f062be00640 1 -- 172.21.15.31:0/3841372905 >> 172.21.15.31:0/3841372905 conn(0x7f062c0fcb30 msgr2=0x7f062c105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:23.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.287+0000 7f062be00640 1 -- 172.21.15.31:0/3841372905 shutdown_connections 2024-03-21T10:37:23.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.287+0000 7f062be00640 1 -- 172.21.15.31:0/3841372905 wait complete. 2024-03-21T10:37:23.305 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T10:29:44.553968+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T10:35:04.945449+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T10:35:05.970589+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T10:35:09.688711+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T10:37:23.305 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T10:37:23.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.525+0000 7fed70a00640 1 -- 172.21.15.31:0/370432111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed6c1071b0 msgr2=0x7fed6c109600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:23.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.525+0000 7fed70a00640 1 --2- 172.21.15.31:0/370432111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed6c1071b0 0x7fed6c109600 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fed64009ca0 tx=0x7fed6402c100 comp rx=0 tx=0).stop 2024-03-21T10:37:23.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.525+0000 7fed70a00640 1 -- 172.21.15.31:0/370432111 shutdown_connections 2024-03-21T10:37:23.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.525+0000 7fed70a00640 1 --2- 172.21.15.31:0/370432111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed6c1071b0 0x7fed6c109600 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.525+0000 7fed70a00640 1 --2- 172.21.15.31:0/370432111 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed6c104880 0x7fed6c106c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.525+0000 7fed70a00640 1 --2- 172.21.15.31:0/370432111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed6c101f50 0x7fed6c104340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.525+0000 7fed70a00640 1 -- 172.21.15.31:0/370432111 >> 172.21.15.31:0/370432111 conn(0x7fed6c0fb980 msgr2=0x7fed6c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.525+0000 7fed70a00640 1 -- 172.21.15.31:0/370432111 shutdown_connections 2024-03-21T10:37:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.525+0000 7fed70a00640 1 -- 172.21.15.31:0/370432111 wait complete. 2024-03-21T10:37:23.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.526+0000 7fed70a00640 1 Processor -- start 2024-03-21T10:37:23.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.526+0000 7fed70a00640 1 -- start start 2024-03-21T10:37:23.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.527+0000 7fed70a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed6c101f50 0x7fed6c197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:23.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.527+0000 7fed70a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed6c104880 0x7fed6c197cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:23.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.527+0000 7fed70a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed6c1071b0 0x7fed6c19ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.527+0000 7fed70a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c10d160 con 0x7fed6c101f50 2024-03-21T10:37:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.527+0000 7fed70a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c10cfe0 con 0x7fed6c104880 2024-03-21T10:37:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.527+0000 7fed70a00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c10d2e0 con 0x7fed6c1071b0 2024-03-21T10:37:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.527+0000 7fed6a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed6c101f50 0x7fed6c197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.527+0000 7fed63e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed6c104880 0x7fed6c197cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.527+0000 7fed6a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed6c101f50 0x7fed6c197780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46432/0 (socket says 172.21.15.31:46432) 2024-03-21T10:37:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.527+0000 7fed63e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed6c104880 0x7fed6c197cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38842/0 (socket says 172.21.15.31:38842) 2024-03-21T10:37:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.527+0000 7fed6a800640 1 -- 172.21.15.31:0/1681680303 learned_addr learned my addr 172.21.15.31:0/1681680303 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.527+0000 7fed6b200640 1 --2- 172.21.15.31:0/1681680303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed6c1071b0 0x7fed6c19ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.528+0000 7fed63e00640 1 -- 172.21.15.31:0/1681680303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed6c1071b0 msgr2=0x7fed6c19ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.528+0000 7fed63e00640 1 --2- 172.21.15.31:0/1681680303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed6c1071b0 0x7fed6c19ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.528+0000 7fed63e00640 1 -- 172.21.15.31:0/1681680303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed6c101f50 msgr2=0x7fed6c197780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.528+0000 7fed63e00640 1 --2- 172.21.15.31:0/1681680303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed6c101f50 0x7fed6c197780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.528+0000 7fed63e00640 1 -- 172.21.15.31:0/1681680303 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed6c19f3a0 con 0x7fed6c104880 2024-03-21T10:37:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.528+0000 7fed6b200640 1 --2- 172.21.15.31:0/1681680303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed6c1071b0 0x7fed6c19ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:37:23.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.528+0000 7fed63e00640 1 --2- 172.21.15.31:0/1681680303 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed6c104880 0x7fed6c197cc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fed5400e9c0 tx=0x7fed5400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:23.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.528+0000 7fed63400640 1 -- 172.21.15.31:0/1681680303 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5400ccc0 con 0x7fed6c104880 2024-03-21T10:37:23.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.528+0000 7fed63400640 1 -- 172.21.15.31:0/1681680303 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed54010910 con 0x7fed6c104880 2024-03-21T10:37:23.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.528+0000 7fed70a00640 1 -- 172.21.15.31:0/1681680303 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed6c19f690 con 0x7fed6c104880 2024-03-21T10:37:23.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.528+0000 7fed63400640 1 -- 172.21.15.31:0/1681680303 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed540188c0 con 0x7fed6c104880 2024-03-21T10:37:23.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.528+0000 7fed70a00640 1 -- 172.21.15.31:0/1681680303 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed6c19fc30 con 0x7fed6c104880 2024-03-21T10:37:23.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.529+0000 7fed70a00640 1 -- 172.21.15.31:0/1681680303 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed6c0fd470 con 0x7fed6c104880 2024-03-21T10:37:23.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.531+0000 7fed63400640 1 -- 172.21.15.31:0/1681680303 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fed54010ac0 con 0x7fed6c104880 2024-03-21T10:37:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.532+0000 7fed63400640 1 --2- 172.21.15.31:0/1681680303 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fed500a2fa0 0x7fed500a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.533+0000 7fed63400640 1 -- 172.21.15.31:0/1681680303 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fed540cfa90 con 0x7fed6c104880 2024-03-21T10:37:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.533+0000 7fed6a800640 1 --2- 172.21.15.31:0/1681680303 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fed500a2fa0 0x7fed500a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.533+0000 7fed6a800640 1 --2- 172.21.15.31:0/1681680303 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fed500a2fa0 0x7fed500a5460 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fed5c009c90 tx=0x7fed5c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:23.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.541+0000 7fed63400640 1 -- 172.21.15.31:0/1681680303 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fed5409beb0 con 0x7fed6c104880 2024-03-21T10:37:23.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:23 smithi031 ceph-mon[31673]: pgmap v397: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:37:23.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:23 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1584836396' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:37:23.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:23 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3841372905' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:37:23.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:23 smithi040 ceph-mon[36154]: pgmap v397: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:37:23.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:23 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1584836396' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:37:23.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:23 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3841372905' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:37:23.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.742+0000 7fed70a00640 1 -- 172.21.15.31:0/1681680303 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fed6c0622a0 con 0x7fed6c104880 2024-03-21T10:37:23.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.743+0000 7fed63400640 1 -- 172.21.15.31:0/1681680303 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7fed5409ff50 con 0x7fed6c104880 2024-03-21T10:37:23.740 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-03-21T10:37:23.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.746+0000 7fed70a00640 1 -- 172.21.15.31:0/1681680303 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fed500a2fa0 msgr2=0x7fed500a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:23.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.746+0000 7fed70a00640 1 --2- 172.21.15.31:0/1681680303 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fed500a2fa0 0x7fed500a5460 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fed5c009c90 tx=0x7fed5c005f90 comp rx=0 tx=0).stop 2024-03-21T10:37:23.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.746+0000 7fed70a00640 1 -- 172.21.15.31:0/1681680303 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed6c104880 msgr2=0x7fed6c197cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:23.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.746+0000 7fed70a00640 1 --2- 172.21.15.31:0/1681680303 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed6c104880 0x7fed6c197cc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fed5400e9c0 tx=0x7fed5400ee80 comp rx=0 tx=0).stop 2024-03-21T10:37:23.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.746+0000 7fed70a00640 1 -- 172.21.15.31:0/1681680303 shutdown_connections 2024-03-21T10:37:23.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.746+0000 7fed70a00640 1 --2- 172.21.15.31:0/1681680303 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fed500a2fa0 0x7fed500a5460 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:23.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.746+0000 7fed70a00640 1 --2- 172.21.15.31:0/1681680303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed6c1071b0 0x7fed6c19ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:23.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.746+0000 7fed70a00640 1 --2- 172.21.15.31:0/1681680303 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed6c104880 0x7fed6c197cc0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:23.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.746+0000 7fed70a00640 1 --2- 172.21.15.31:0/1681680303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed6c101f50 0x7fed6c197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:23.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.746+0000 7fed70a00640 1 -- 172.21.15.31:0/1681680303 >> 172.21.15.31:0/1681680303 conn(0x7fed6c0fb980 msgr2=0x7fed6c102890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:23.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.746+0000 7fed70a00640 1 -- 172.21.15.31:0/1681680303 shutdown_connections 2024-03-21T10:37:23.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.746+0000 7fed70a00640 1 -- 172.21.15.31:0/1681680303 wait complete. 2024-03-21T10:37:23.764 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T10:37:23.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:23 smithi070 ceph-mon[36330]: pgmap v397: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:37:23.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:23 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1584836396' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:37:23.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:23 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3841372905' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:37:23.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.986+0000 7f30a7e00640 1 -- 172.21.15.31:0/2790077789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30a8101f70 msgr2=0x7f30a8104360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:23.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.987+0000 7f30a7e00640 1 --2- 172.21.15.31:0/2790077789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30a8101f70 0x7f30a8104360 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3098009900 tx=0x7f309802bc30 comp rx=0 tx=0).stop 2024-03-21T10:37:23.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.987+0000 7f30a7e00640 1 -- 172.21.15.31:0/2790077789 shutdown_connections 2024-03-21T10:37:23.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.987+0000 7f30a7e00640 1 --2- 172.21.15.31:0/2790077789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30a81071d0 0x7f30a8109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:23.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.987+0000 7f30a7e00640 1 --2- 172.21.15.31:0/2790077789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30a81048a0 0x7f30a8106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:23.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.987+0000 7f30a7e00640 1 --2- 172.21.15.31:0/2790077789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30a8101f70 0x7f30a8104360 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:23.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.987+0000 7f30a7e00640 1 -- 172.21.15.31:0/2790077789 >> 172.21.15.31:0/2790077789 conn(0x7f30a80fb980 msgr2=0x7f30a80fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:23.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.987+0000 7f30a7e00640 1 -- 172.21.15.31:0/2790077789 shutdown_connections 2024-03-21T10:37:23.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.987+0000 7f30a7e00640 1 -- 172.21.15.31:0/2790077789 wait complete. 2024-03-21T10:37:23.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.988+0000 7f30a7e00640 1 Processor -- start 2024-03-21T10:37:23.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.988+0000 7f30a7e00640 1 -- start start 2024-03-21T10:37:23.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.989+0000 7f30a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30a8101f70 0x7f30a81018e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:23.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.989+0000 7f30a7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30a81048a0 0x7f30a80fff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:23.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.989+0000 7f30a7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30a81071d0 0x7f30a8100460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:23.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.989+0000 7f30a7e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a810d1d0 con 0x7f30a81071d0 2024-03-21T10:37:23.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.989+0000 7f30a7e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a810d050 con 0x7f30a8101f70 2024-03-21T10:37:23.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.989+0000 7f30a7e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a810d350 con 0x7f30a81048a0 2024-03-21T10:37:23.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.989+0000 7f30a6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30a8101f70 0x7f30a81018e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:23.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.989+0000 7f30a6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30a8101f70 0x7f30a81018e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38852/0 (socket says 172.21.15.31:38852) 2024-03-21T10:37:23.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.989+0000 7f30a6a00640 1 -- 172.21.15.31:0/3091807301 learned_addr learned my addr 172.21.15.31:0/3091807301 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:23.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.989+0000 7f309fe00640 1 --2- 172.21.15.31:0/3091807301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30a81048a0 0x7f30a80fff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:23.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.989+0000 7f30a7400640 1 --2- 172.21.15.31:0/3091807301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30a81071d0 0x7f30a8100460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:23.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.989+0000 7f30a6a00640 1 -- 172.21.15.31:0/3091807301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30a81048a0 msgr2=0x7f30a80fff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:23.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.989+0000 7f30a6a00640 1 --2- 172.21.15.31:0/3091807301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30a81048a0 0x7f30a80fff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:23.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.989+0000 7f30a6a00640 1 -- 172.21.15.31:0/3091807301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30a81071d0 msgr2=0x7f30a8100460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:23.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.989+0000 7f30a6a00640 1 --2- 172.21.15.31:0/3091807301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30a81071d0 0x7f30a8100460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:23.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.989+0000 7f30a6a00640 1 -- 172.21.15.31:0/3091807301 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a8100cf0 con 0x7f30a8101f70 2024-03-21T10:37:23.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.989+0000 7f309fe00640 1 --2- 172.21.15.31:0/3091807301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30a81048a0 0x7f30a80fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:37:23.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.990+0000 7f30a6a00640 1 --2- 172.21.15.31:0/3091807301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30a8101f70 0x7f30a81018e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3098005ab0 tx=0x7f309802c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:23.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.990+0000 7f309f400640 1 -- 172.21.15.31:0/3091807301 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3098043020 con 0x7f30a8101f70 2024-03-21T10:37:23.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.990+0000 7f309f400640 1 -- 172.21.15.31:0/3091807301 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30980054e0 con 0x7f30a8101f70 2024-03-21T10:37:23.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.990+0000 7f30a7e00640 1 -- 172.21.15.31:0/3091807301 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a81a1930 con 0x7f30a8101f70 2024-03-21T10:37:23.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.990+0000 7f309f400640 1 -- 172.21.15.31:0/3091807301 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3098003ce0 con 0x7f30a8101f70 2024-03-21T10:37:23.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.990+0000 7f30a7e00640 1 -- 172.21.15.31:0/3091807301 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a81a1e80 con 0x7f30a8101f70 2024-03-21T10:37:23.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.992+0000 7f30a7e00640 1 -- 172.21.15.31:0/3091807301 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a80fd470 con 0x7f30a8101f70 2024-03-21T10:37:23.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.993+0000 7f309f400640 1 -- 172.21.15.31:0/3091807301 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f3098046020 con 0x7f30a8101f70 2024-03-21T10:37:23.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.995+0000 7f309f400640 1 --2- 172.21.15.31:0/3091807301 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f308c0a2f20 0x7f308c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:23.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.995+0000 7f309f400640 1 -- 172.21.15.31:0/3091807301 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f30980f1830 con 0x7f30a8101f70 2024-03-21T10:37:23.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.995+0000 7f309fe00640 1 --2- 172.21.15.31:0/3091807301 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f308c0a2f20 0x7f308c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:23.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.996+0000 7f309fe00640 1 --2- 172.21.15.31:0/3091807301 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f308c0a2f20 0x7f308c0a53e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f30a8101650 tx=0x7f309000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:23.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:23.998+0000 7f309f400640 1 -- 172.21.15.31:0/3091807301 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f309804d020 con 0x7f30a8101f70 2024-03-21T10:37:24.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.205+0000 7f30a7e00640 1 -- 172.21.15.31:0/3091807301 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f30a80622a0 con 0x7f30a8101f70 2024-03-21T10:37:24.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.206+0000 7f309f400640 1 -- 172.21.15.31:0/3091807301 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f30980be300 con 0x7f30a8101f70 2024-03-21T10:37:24.203 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-03-21T10:37:24.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.208+0000 7f30a7e00640 1 -- 172.21.15.31:0/3091807301 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f308c0a2f20 msgr2=0x7f308c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:24.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.208+0000 7f30a7e00640 1 --2- 172.21.15.31:0/3091807301 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f308c0a2f20 0x7f308c0a53e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f30a8101650 tx=0x7f309000b390 comp rx=0 tx=0).stop 2024-03-21T10:37:24.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.208+0000 7f30a7e00640 1 -- 172.21.15.31:0/3091807301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30a8101f70 msgr2=0x7f30a81018e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:24.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.208+0000 7f30a7e00640 1 --2- 172.21.15.31:0/3091807301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30a8101f70 0x7f30a81018e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3098005ab0 tx=0x7f309802c6c0 comp rx=0 tx=0).stop 2024-03-21T10:37:24.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.208+0000 7f30a7e00640 1 -- 172.21.15.31:0/3091807301 shutdown_connections 2024-03-21T10:37:24.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.208+0000 7f30a7e00640 1 --2- 172.21.15.31:0/3091807301 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f308c0a2f20 0x7f308c0a53e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:24.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.208+0000 7f30a7e00640 1 --2- 172.21.15.31:0/3091807301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30a81071d0 0x7f30a8100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:24.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.208+0000 7f30a7e00640 1 --2- 172.21.15.31:0/3091807301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30a81048a0 0x7f30a80fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:24.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.208+0000 7f30a7e00640 1 --2- 172.21.15.31:0/3091807301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30a8101f70 0x7f30a81018e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:24.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.209+0000 7f30a7e00640 1 -- 172.21.15.31:0/3091807301 >> 172.21.15.31:0/3091807301 conn(0x7f30a80fb980 msgr2=0x7f30a81028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:24.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.209+0000 7f30a7e00640 1 -- 172.21.15.31:0/3091807301 shutdown_connections 2024-03-21T10:37:24.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.209+0000 7f30a7e00640 1 -- 172.21.15.31:0/3091807301 wait complete. 2024-03-21T10:37:24.227 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T10:37:24.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.455+0000 7fe844800640 1 -- 172.21.15.31:0/2501699579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe83c074100 msgr2=0x7fe83c074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:24.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.455+0000 7fe844800640 1 --2- 172.21.15.31:0/2501699579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe83c074100 0x7fe83c074580 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe82c009900 tx=0x7fe82c02bc40 comp rx=0 tx=0).stop 2024-03-21T10:37:24.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.455+0000 7fe844800640 1 -- 172.21.15.31:0/2501699579 shutdown_connections 2024-03-21T10:37:24.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.455+0000 7fe844800640 1 --2- 172.21.15.31:0/2501699579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe83c109570 0x7fe83c10ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:24.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.455+0000 7fe844800640 1 --2- 172.21.15.31:0/2501699579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe83c074100 0x7fe83c074580 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:24.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.455+0000 7fe844800640 1 --2- 172.21.15.31:0/2501699579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe83c0756e0 0x7fe83c073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:24.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.455+0000 7fe844800640 1 -- 172.21.15.31:0/2501699579 >> 172.21.15.31:0/2501699579 conn(0x7fe83c0fcc20 msgr2=0x7fe83c0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:24.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.455+0000 7fe844800640 1 -- 172.21.15.31:0/2501699579 shutdown_connections 2024-03-21T10:37:24.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.456+0000 7fe844800640 1 -- 172.21.15.31:0/2501699579 wait complete. 2024-03-21T10:37:24.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.457+0000 7fe844800640 1 Processor -- start 2024-03-21T10:37:24.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.457+0000 7fe844800640 1 -- start start 2024-03-21T10:37:24.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.457+0000 7fe844800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe83c074100 0x7fe83c199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:24.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.457+0000 7fe844800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe83c0756e0 0x7fe83c19a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:24.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.457+0000 7fe844800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe83c109570 0x7fe83c1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:24.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.457+0000 7fe844800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe83c10f540 con 0x7fe83c109570 2024-03-21T10:37:24.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.457+0000 7fe844800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe83c10f3c0 con 0x7fe83c0756e0 2024-03-21T10:37:24.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.457+0000 7fe844800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe83c10f6c0 con 0x7fe83c074100 2024-03-21T10:37:24.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.458+0000 7fe841e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe83c074100 0x7fe83c199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:24.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.458+0000 7fe841400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe83c0756e0 0x7fe83c19a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:24.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.458+0000 7fe841400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe83c0756e0 0x7fe83c19a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38874/0 (socket says 172.21.15.31:38874) 2024-03-21T10:37:24.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.458+0000 7fe841400640 1 -- 172.21.15.31:0/244416798 learned_addr learned my addr 172.21.15.31:0/244416798 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:24.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.458+0000 7fe841e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe83c074100 0x7fe83c199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:47428/0 (socket says 172.21.15.31:47428) 2024-03-21T10:37:24.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.458+0000 7fe842800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe83c109570 0x7fe83c1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:24.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.458+0000 7fe841400640 1 -- 172.21.15.31:0/244416798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe83c074100 msgr2=0x7fe83c199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:24.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.458+0000 7fe841400640 1 --2- 172.21.15.31:0/244416798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe83c074100 0x7fe83c199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:24.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.458+0000 7fe841400640 1 -- 172.21.15.31:0/244416798 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe83c109570 msgr2=0x7fe83c1a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:24.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.458+0000 7fe841400640 1 --2- 172.21.15.31:0/244416798 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe83c109570 0x7fe83c1a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:24.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.458+0000 7fe841400640 1 -- 172.21.15.31:0/244416798 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe83c1a1830 con 0x7fe83c0756e0 2024-03-21T10:37:24.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.458+0000 7fe841e00640 1 --2- 172.21.15.31:0/244416798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe83c074100 0x7fe83c199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:37:24.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.458+0000 7fe841400640 1 --2- 172.21.15.31:0/244416798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe83c0756e0 0x7fe83c19a160 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe82c02c8f0 tx=0x7fe82c02c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:24.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.459+0000 7fe836a00640 1 -- 172.21.15.31:0/244416798 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82c041070 con 0x7fe83c0756e0 2024-03-21T10:37:24.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.459+0000 7fe836a00640 1 -- 172.21.15.31:0/244416798 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe82c004450 con 0x7fe83c0756e0 2024-03-21T10:37:24.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.459+0000 7fe844800640 1 -- 172.21.15.31:0/244416798 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe83c1a1ac0 con 0x7fe83c0756e0 2024-03-21T10:37:24.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.459+0000 7fe836a00640 1 -- 172.21.15.31:0/244416798 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82c034440 con 0x7fe83c0756e0 2024-03-21T10:37:24.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.459+0000 7fe844800640 1 -- 172.21.15.31:0/244416798 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe83c1a1fb0 con 0x7fe83c0756e0 2024-03-21T10:37:24.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.460+0000 7fe844800640 1 -- 172.21.15.31:0/244416798 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe83c1082a0 con 0x7fe83c0756e0 2024-03-21T10:37:24.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.462+0000 7fe836a00640 1 -- 172.21.15.31:0/244416798 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fe82c044070 con 0x7fe83c0756e0 2024-03-21T10:37:24.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.463+0000 7fe836a00640 1 --2- 172.21.15.31:0/244416798 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe8240a2f20 0x7fe8240a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:24.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.463+0000 7fe836a00640 1 -- 172.21.15.31:0/244416798 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fe82c0f1330 con 0x7fe83c0756e0 2024-03-21T10:37:24.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.463+0000 7fe841e00640 1 --2- 172.21.15.31:0/244416798 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe8240a2f20 0x7fe8240a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:24.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.464+0000 7fe841e00640 1 --2- 172.21.15.31:0/244416798 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe8240a2f20 0x7fe8240a53e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe83c19b0f0 tx=0x7fe830005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:24.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.467+0000 7fe836a00640 1 -- 172.21.15.31:0/244416798 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe82c0bd750 con 0x7fe83c0756e0 2024-03-21T10:37:24.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:24 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1681680303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:37:24.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:24 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3091807301' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:37:24.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.668+0000 7fe844800640 1 -- 172.21.15.31:0/244416798 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe83c0622a0 con 0x7fe83c0756e0 2024-03-21T10:37:24.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.668+0000 7fe836a00640 1 -- 172.21.15.31:0/244416798 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe82c0c17f0 con 0x7fe83c0756e0 2024-03-21T10:37:24.667 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-03-21T10:37:24.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.671+0000 7fe844800640 1 -- 172.21.15.31:0/244416798 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe8240a2f20 msgr2=0x7fe8240a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:24.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.671+0000 7fe844800640 1 --2- 172.21.15.31:0/244416798 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe8240a2f20 0x7fe8240a53e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe83c19b0f0 tx=0x7fe830005f90 comp rx=0 tx=0).stop 2024-03-21T10:37:24.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.671+0000 7fe844800640 1 -- 172.21.15.31:0/244416798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe83c0756e0 msgr2=0x7fe83c19a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:24.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.671+0000 7fe844800640 1 --2- 172.21.15.31:0/244416798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe83c0756e0 0x7fe83c19a160 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe82c02c8f0 tx=0x7fe82c02c660 comp rx=0 tx=0).stop 2024-03-21T10:37:24.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.671+0000 7fe844800640 1 -- 172.21.15.31:0/244416798 shutdown_connections 2024-03-21T10:37:24.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.671+0000 7fe844800640 1 --2- 172.21.15.31:0/244416798 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe8240a2f20 0x7fe8240a53e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:24.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.671+0000 7fe844800640 1 --2- 172.21.15.31:0/244416798 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe83c109570 0x7fe83c1a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:24.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.671+0000 7fe844800640 1 --2- 172.21.15.31:0/244416798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe83c0756e0 0x7fe83c19a160 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:24.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.671+0000 7fe844800640 1 --2- 172.21.15.31:0/244416798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe83c074100 0x7fe83c199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:24.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.671+0000 7fe844800640 1 -- 172.21.15.31:0/244416798 >> 172.21.15.31:0/244416798 conn(0x7fe83c0fcc20 msgr2=0x7fe83c105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:24.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.671+0000 7fe844800640 1 -- 172.21.15.31:0/244416798 shutdown_connections 2024-03-21T10:37:24.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.671+0000 7fe844800640 1 -- 172.21.15.31:0/244416798 wait complete. 2024-03-21T10:37:24.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:24 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1681680303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:37:24.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:24 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3091807301' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:37:24.688 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T10:37:24.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:24 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1681680303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:37:24.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:24 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3091807301' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:37:24.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.924+0000 7fe53b200640 1 -- 172.21.15.31:0/264065189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe53c101f60 msgr2=0x7fe53c104350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:24.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.924+0000 7fe53b200640 1 --2- 172.21.15.31:0/264065189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe53c101f60 0x7fe53c104350 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009900 tx=0x7fe52c02bc30 comp rx=0 tx=0).stop 2024-03-21T10:37:24.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.924+0000 7fe53b200640 1 -- 172.21.15.31:0/264065189 shutdown_connections 2024-03-21T10:37:24.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.924+0000 7fe53b200640 1 --2- 172.21.15.31:0/264065189 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe53c1071c0 0x7fe53c109610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:24.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.924+0000 7fe53b200640 1 --2- 172.21.15.31:0/264065189 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe53c104890 0x7fe53c106c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:24.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.924+0000 7fe53b200640 1 --2- 172.21.15.31:0/264065189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe53c101f60 0x7fe53c104350 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:24.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.924+0000 7fe53b200640 1 -- 172.21.15.31:0/264065189 >> 172.21.15.31:0/264065189 conn(0x7fe53c0fb990 msgr2=0x7fe53c0fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:24.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.924+0000 7fe53b200640 1 -- 172.21.15.31:0/264065189 shutdown_connections 2024-03-21T10:37:24.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.924+0000 7fe53b200640 1 -- 172.21.15.31:0/264065189 wait complete. 2024-03-21T10:37:24.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.925+0000 7fe53b200640 1 Processor -- start 2024-03-21T10:37:24.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.926+0000 7fe53b200640 1 -- start start 2024-03-21T10:37:24.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.926+0000 7fe53b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe53c101f60 0x7fe53c199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:24.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.926+0000 7fe53b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe53c104890 0x7fe53c199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:24.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.926+0000 7fe53b200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe53c1071c0 0x7fe53c1a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:24.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.926+0000 7fe53b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c10d170 con 0x7fe53c104890 2024-03-21T10:37:24.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.926+0000 7fe53b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c10cff0 con 0x7fe53c101f60 2024-03-21T10:37:24.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.926+0000 7fe53b200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c10d2f0 con 0x7fe53c1071c0 2024-03-21T10:37:24.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.926+0000 7fe539e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe53c101f60 0x7fe53c199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:24.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.926+0000 7fe539e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe53c101f60 0x7fe53c199a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38880/0 (socket says 172.21.15.31:38880) 2024-03-21T10:37:24.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.926+0000 7fe539e00640 1 -- 172.21.15.31:0/3037378204 learned_addr learned my addr 172.21.15.31:0/3037378204 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:24.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.926+0000 7fe539400640 1 --2- 172.21.15.31:0/3037378204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe53c104890 0x7fe53c199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:24.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.927+0000 7fe53a800640 1 --2- 172.21.15.31:0/3037378204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe53c1071c0 0x7fe53c1a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:24.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.927+0000 7fe539e00640 1 -- 172.21.15.31:0/3037378204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe53c1071c0 msgr2=0x7fe53c1a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:24.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.927+0000 7fe539e00640 1 --2- 172.21.15.31:0/3037378204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe53c1071c0 0x7fe53c1a0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:24.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.927+0000 7fe539e00640 1 -- 172.21.15.31:0/3037378204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe53c104890 msgr2=0x7fe53c199f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:24.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.927+0000 7fe539e00640 1 --2- 172.21.15.31:0/3037378204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe53c104890 0x7fe53c199f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:24.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.927+0000 7fe539e00640 1 -- 172.21.15.31:0/3037378204 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53c1a1660 con 0x7fe53c101f60 2024-03-21T10:37:24.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.927+0000 7fe539e00640 1 --2- 172.21.15.31:0/3037378204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe53c101f60 0x7fe53c199a40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009230 tx=0x7fe52c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:24.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.927+0000 7fe532a00640 1 -- 172.21.15.31:0/3037378204 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52c041070 con 0x7fe53c101f60 2024-03-21T10:37:24.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.928+0000 7fe532a00640 1 -- 172.21.15.31:0/3037378204 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe52c004450 con 0x7fe53c101f60 2024-03-21T10:37:24.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.928+0000 7fe53b200640 1 -- 172.21.15.31:0/3037378204 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe53c1a18f0 con 0x7fe53c101f60 2024-03-21T10:37:24.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.928+0000 7fe532a00640 1 -- 172.21.15.31:0/3037378204 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52c003db0 con 0x7fe53c101f60 2024-03-21T10:37:24.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.928+0000 7fe53b200640 1 -- 172.21.15.31:0/3037378204 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe53c1a1de0 con 0x7fe53c101f60 2024-03-21T10:37:24.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.929+0000 7fe53b200640 1 -- 172.21.15.31:0/3037378204 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe53c108d10 con 0x7fe53c101f60 2024-03-21T10:37:24.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.931+0000 7fe532a00640 1 -- 172.21.15.31:0/3037378204 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fe52c044070 con 0x7fe53c101f60 2024-03-21T10:37:24.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.932+0000 7fe532a00640 1 --2- 172.21.15.31:0/3037378204 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe5200a2f20 0x7fe5200a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:24.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.932+0000 7fe532a00640 1 -- 172.21.15.31:0/3037378204 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fe52c0f1370 con 0x7fe53c101f60 2024-03-21T10:37:24.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.932+0000 7fe539400640 1 --2- 172.21.15.31:0/3037378204 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe5200a2f20 0x7fe5200a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:24.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.933+0000 7fe539400640 1 --2- 172.21.15.31:0/3037378204 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe5200a2f20 0x7fe5200a53e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe53c19af50 tx=0x7fe528005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:24.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:24.936+0000 7fe532a00640 1 -- 172.21.15.31:0/3037378204 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe52c0bdde0 con 0x7fe53c101f60 2024-03-21T10:37:25.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.138+0000 7fe53b200640 1 -- 172.21.15.31:0/3037378204 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fe53c02d070 con 0x7fe53c101f60 2024-03-21T10:37:25.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.139+0000 7fe532a00640 1 -- 172.21.15.31:0/3037378204 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe52c0bd6e0 con 0x7fe53c101f60 2024-03-21T10:37:25.137 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-03-21T10:37:25.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.141+0000 7fe53b200640 1 -- 172.21.15.31:0/3037378204 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe5200a2f20 msgr2=0x7fe5200a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:25.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.141+0000 7fe53b200640 1 --2- 172.21.15.31:0/3037378204 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe5200a2f20 0x7fe5200a53e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe53c19af50 tx=0x7fe528005f90 comp rx=0 tx=0).stop 2024-03-21T10:37:25.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.141+0000 7fe53b200640 1 -- 172.21.15.31:0/3037378204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe53c101f60 msgr2=0x7fe53c199a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:25.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.141+0000 7fe53b200640 1 --2- 172.21.15.31:0/3037378204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe53c101f60 0x7fe53c199a40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009230 tx=0x7fe52c02c650 comp rx=0 tx=0).stop 2024-03-21T10:37:25.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.141+0000 7fe53b200640 1 -- 172.21.15.31:0/3037378204 shutdown_connections 2024-03-21T10:37:25.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.141+0000 7fe53b200640 1 --2- 172.21.15.31:0/3037378204 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe5200a2f20 0x7fe5200a53e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:25.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.141+0000 7fe53b200640 1 --2- 172.21.15.31:0/3037378204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe53c1071c0 0x7fe53c1a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:25.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.141+0000 7fe53b200640 1 --2- 172.21.15.31:0/3037378204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe53c104890 0x7fe53c199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:25.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.142+0000 7fe53b200640 1 --2- 172.21.15.31:0/3037378204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe53c101f60 0x7fe53c199a40 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:25.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.142+0000 7fe53b200640 1 -- 172.21.15.31:0/3037378204 >> 172.21.15.31:0/3037378204 conn(0x7fe53c0fb990 msgr2=0x7fe53c1015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:25.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.142+0000 7fe53b200640 1 -- 172.21.15.31:0/3037378204 shutdown_connections 2024-03-21T10:37:25.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.142+0000 7fe53b200640 1 -- 172.21.15.31:0/3037378204 wait complete. 2024-03-21T10:37:25.159 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:37:25.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.372+0000 7fccba200640 1 -- 172.21.15.31:0/725956590 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccb4103170 msgr2=0x7fccb4109a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:25.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.372+0000 7fccba200640 1 --2- 172.21.15.31:0/725956590 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccb4103170 0x7fccb4109a00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fccac009ca0 tx=0x7fccac02c100 comp rx=0 tx=0).stop 2024-03-21T10:37:25.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.372+0000 7fccba200640 1 -- 172.21.15.31:0/725956590 shutdown_connections 2024-03-21T10:37:25.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.372+0000 7fccba200640 1 --2- 172.21.15.31:0/725956590 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccb4103170 0x7fccb4109a00 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:25.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.372+0000 7fccba200640 1 --2- 172.21.15.31:0/725956590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccb41027b0 0x7fccb4102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:25.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.372+0000 7fccba200640 1 --2- 172.21.15.31:0/725956590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fccb41015c0 0x7fccb41019e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:25.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.372+0000 7fccba200640 1 -- 172.21.15.31:0/725956590 >> 172.21.15.31:0/725956590 conn(0x7fccb40fcb30 msgr2=0x7fccb40fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:25.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.372+0000 7fccba200640 1 -- 172.21.15.31:0/725956590 shutdown_connections 2024-03-21T10:37:25.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.372+0000 7fccba200640 1 -- 172.21.15.31:0/725956590 wait complete. 2024-03-21T10:37:25.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.373+0000 7fccba200640 1 Processor -- start 2024-03-21T10:37:25.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.373+0000 7fccba200640 1 -- start start 2024-03-21T10:37:25.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.374+0000 7fccba200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccb41015c0 0x7fccb41999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:25.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.374+0000 7fccba200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccb41027b0 0x7fccb4199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:25.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.374+0000 7fccba200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fccb4103170 0x7fccb41a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:25.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.374+0000 7fccba200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb410d220 con 0x7fccb4103170 2024-03-21T10:37:25.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.374+0000 7fccba200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb410d0a0 con 0x7fccb41027b0 2024-03-21T10:37:25.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.374+0000 7fccba200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb410d3a0 con 0x7fccb41015c0 2024-03-21T10:37:25.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.374+0000 7fccb3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccb41027b0 0x7fccb4199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:25.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.374+0000 7fccb8e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccb41015c0 0x7fccb41999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:25.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.374+0000 7fccb9800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fccb4103170 0x7fccb41a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:25.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.374+0000 7fccb8e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccb41015c0 0x7fccb41999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:47470/0 (socket says 172.21.15.31:47470) 2024-03-21T10:37:25.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.374+0000 7fccb3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccb41027b0 0x7fccb4199f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38892/0 (socket says 172.21.15.31:38892) 2024-03-21T10:37:25.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.374+0000 7fccb8e00640 1 -- 172.21.15.31:0/710714697 learned_addr learned my addr 172.21.15.31:0/710714697 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:25.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.374+0000 7fccb8e00640 1 -- 172.21.15.31:0/710714697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccb41027b0 msgr2=0x7fccb4199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:25.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.374+0000 7fccb8e00640 1 --2- 172.21.15.31:0/710714697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccb41027b0 0x7fccb4199f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:25.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.374+0000 7fccb8e00640 1 -- 172.21.15.31:0/710714697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fccb4103170 msgr2=0x7fccb41a0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:25.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.374+0000 7fccb8e00640 1 --2- 172.21.15.31:0/710714697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fccb4103170 0x7fccb41a0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:25.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.374+0000 7fccb8e00640 1 -- 172.21.15.31:0/710714697 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb41a1610 con 0x7fccb41015c0 2024-03-21T10:37:25.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.374+0000 7fccb3e00640 1 --2- 172.21.15.31:0/710714697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccb41027b0 0x7fccb4199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:37:25.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.375+0000 7fccb9800640 1 --2- 172.21.15.31:0/710714697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fccb4103170 0x7fccb41a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:37:25.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.375+0000 7fccb8e00640 1 --2- 172.21.15.31:0/710714697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccb41015c0 0x7fccb41999f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcca800e9c0 tx=0x7fcca800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:25.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.375+0000 7fccb1600640 1 -- 172.21.15.31:0/710714697 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca800cde0 con 0x7fccb41015c0 2024-03-21T10:37:25.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.375+0000 7fccb1600640 1 -- 172.21.15.31:0/710714697 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcca8010910 con 0x7fccb41015c0 2024-03-21T10:37:25.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.375+0000 7fccba200640 1 -- 172.21.15.31:0/710714697 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb41a1900 con 0x7fccb41015c0 2024-03-21T10:37:25.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.375+0000 7fccb1600640 1 -- 172.21.15.31:0/710714697 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca8018a00 con 0x7fccb41015c0 2024-03-21T10:37:25.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.375+0000 7fccba200640 1 -- 172.21.15.31:0/710714697 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb41a1ea0 con 0x7fccb41015c0 2024-03-21T10:37:25.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.376+0000 7fccba200640 1 -- 172.21.15.31:0/710714697 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb404f470 con 0x7fccb41015c0 2024-03-21T10:37:25.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.378+0000 7fccb1600640 1 -- 172.21.15.31:0/710714697 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fcca801f020 con 0x7fccb41015c0 2024-03-21T10:37:25.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.379+0000 7fccb1600640 1 --2- 172.21.15.31:0/710714697 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcc9c0a2f90 0x7fcc9c0a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:25.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.379+0000 7fccb1600640 1 -- 172.21.15.31:0/710714697 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fcca80cf900 con 0x7fccb41015c0 2024-03-21T10:37:25.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.380+0000 7fccb3e00640 1 --2- 172.21.15.31:0/710714697 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcc9c0a2f90 0x7fcc9c0a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:25.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.380+0000 7fccb3e00640 1 --2- 172.21.15.31:0/710714697 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcc9c0a2f90 0x7fcc9c0a5450 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fccb419af00 tx=0x7fcca0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:25.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.383+0000 7fccb1600640 1 -- 172.21.15.31:0/710714697 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcca809c370 con 0x7fccb41015c0 2024-03-21T10:37:25.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.652+0000 7fccba200640 1 -- 172.21.15.31:0/710714697 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccb41a27a0 con 0x7fccb41015c0 2024-03-21T10:37:25.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.653+0000 7fccb1600640 1 -- 172.21.15.31:0/710714697 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11384 (secure 0 0 0) 0x7fcca809bc70 con 0x7fccb41015c0 2024-03-21T10:37:25.651 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:37:25.651 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:35:40.990205+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24596,24547]},"id":1}]} 2024-03-21T10:37:25.651 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2024-03-21T10:37:25.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.655+0000 7fccba200640 1 -- 172.21.15.31:0/710714697 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcc9c0a2f90 msgr2=0x7fcc9c0a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:25.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.655+0000 7fccba200640 1 --2- 172.21.15.31:0/710714697 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcc9c0a2f90 0x7fcc9c0a5450 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fccb419af00 tx=0x7fcca0005f90 comp rx=0 tx=0).stop 2024-03-21T10:37:25.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.655+0000 7fccba200640 1 -- 172.21.15.31:0/710714697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccb41015c0 msgr2=0x7fccb41999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:25.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.655+0000 7fccba200640 1 --2- 172.21.15.31:0/710714697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccb41015c0 0x7fccb41999f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcca800e9c0 tx=0x7fcca800ee80 comp rx=0 tx=0).stop 2024-03-21T10:37:25.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.656+0000 7fccba200640 1 -- 172.21.15.31:0/710714697 shutdown_connections 2024-03-21T10:37:25.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.656+0000 7fccba200640 1 --2- 172.21.15.31:0/710714697 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcc9c0a2f90 0x7fcc9c0a5450 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:25.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.656+0000 7fccba200640 1 --2- 172.21.15.31:0/710714697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fccb4103170 0x7fccb41a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:25.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.656+0000 7fccba200640 1 --2- 172.21.15.31:0/710714697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccb41027b0 0x7fccb4199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:25.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.656+0000 7fccba200640 1 --2- 172.21.15.31:0/710714697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccb41015c0 0x7fccb41999f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:25.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.656+0000 7fccba200640 1 -- 172.21.15.31:0/710714697 >> 172.21.15.31:0/710714697 conn(0x7fccb40fcb30 msgr2=0x7fccb40738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:25.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.656+0000 7fccba200640 1 -- 172.21.15.31:0/710714697 shutdown_connections 2024-03-21T10:37:25.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.656+0000 7fccba200640 1 -- 172.21.15.31:0/710714697 wait complete. 2024-03-21T10:37:25.670 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:25 smithi031 ceph-mon[31673]: pgmap v398: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:25.670 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:25 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/244416798' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:37:25.670 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:25 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3037378204' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:37:25.673 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T10:37:25.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:25 smithi040 ceph-mon[36154]: pgmap v398: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:25.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:25 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/244416798' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:37:25.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:25 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3037378204' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:37:25.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:25 smithi070 ceph-mon[36330]: pgmap v398: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:25.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:25 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/244416798' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:37:25.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:25 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3037378204' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:37:25.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.894+0000 7fb94fa00640 1 -- 172.21.15.31:0/349377069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb948103960 msgr2=0x7fb948103d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:25.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.894+0000 7fb94fa00640 1 --2- 172.21.15.31:0/349377069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb948103960 0x7fb948103d80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb938009900 tx=0x7fb93802bc30 comp rx=0 tx=0).stop 2024-03-21T10:37:25.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.894+0000 7fb94fa00640 1 -- 172.21.15.31:0/349377069 shutdown_connections 2024-03-21T10:37:25.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.894+0000 7fb94fa00640 1 --2- 172.21.15.31:0/349377069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb948105510 0x7fb94810bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:25.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.894+0000 7fb94fa00640 1 --2- 172.21.15.31:0/349377069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb948104b50 0x7fb948104fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:25.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.894+0000 7fb94fa00640 1 --2- 172.21.15.31:0/349377069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb948103960 0x7fb948103d80 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:25.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.894+0000 7fb94fa00640 1 -- 172.21.15.31:0/349377069 >> 172.21.15.31:0/349377069 conn(0x7fb9480feeb0 msgr2=0x7fb948101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:25.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.894+0000 7fb94fa00640 1 -- 172.21.15.31:0/349377069 shutdown_connections 2024-03-21T10:37:25.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.894+0000 7fb94fa00640 1 -- 172.21.15.31:0/349377069 wait complete. 2024-03-21T10:37:25.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.895+0000 7fb94fa00640 1 Processor -- start 2024-03-21T10:37:25.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.895+0000 7fb94fa00640 1 -- start start 2024-03-21T10:37:25.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.895+0000 7fb94fa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb948103960 0x7fb948199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:25.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.895+0000 7fb94fa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb948104b50 0x7fb94819a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:25.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.895+0000 7fb94fa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb948105510 0x7fb9481a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:25.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.896+0000 7fb94fa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94810f590 con 0x7fb948103960 2024-03-21T10:37:25.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.896+0000 7fb94fa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94810f410 con 0x7fb948105510 2024-03-21T10:37:25.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.896+0000 7fb94fa00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94810f710 con 0x7fb948104b50 2024-03-21T10:37:25.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.896+0000 7fb93fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb948104b50 0x7fb94819a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:25.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.896+0000 7fb94da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb948105510 0x7fb9481a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:25.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.896+0000 7fb94d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb948103960 0x7fb948199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:25.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.896+0000 7fb93fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb948104b50 0x7fb94819a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:47492/0 (socket says 172.21.15.31:47492) 2024-03-21T10:37:25.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.896+0000 7fb93fe00640 1 -- 172.21.15.31:0/2744678904 learned_addr learned my addr 172.21.15.31:0/2744678904 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.896+0000 7fb94da00640 1 -- 172.21.15.31:0/2744678904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb948104b50 msgr2=0x7fb94819a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.896+0000 7fb94da00640 1 --2- 172.21.15.31:0/2744678904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb948104b50 0x7fb94819a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.896+0000 7fb94da00640 1 -- 172.21.15.31:0/2744678904 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb948103960 msgr2=0x7fb948199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.896+0000 7fb94da00640 1 --2- 172.21.15.31:0/2744678904 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb948103960 0x7fb948199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.896+0000 7fb94da00640 1 -- 172.21.15.31:0/2744678904 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9481a1830 con 0x7fb948105510 2024-03-21T10:37:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.896+0000 7fb93fe00640 1 --2- 172.21.15.31:0/2744678904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb948104b50 0x7fb94819a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:37:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.897+0000 7fb94da00640 1 --2- 172.21.15.31:0/2744678904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb948105510 0x7fb9481a1190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb940009fd0 tx=0x7fb94000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.897+0000 7fb946000640 1 -- 172.21.15.31:0/2744678904 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94000ce20 con 0x7fb948105510 2024-03-21T10:37:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.897+0000 7fb946000640 1 -- 172.21.15.31:0/2744678904 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb940005020 con 0x7fb948105510 2024-03-21T10:37:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.897+0000 7fb94fa00640 1 -- 172.21.15.31:0/2744678904 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9481a1b20 con 0x7fb948105510 2024-03-21T10:37:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.897+0000 7fb946000640 1 -- 172.21.15.31:0/2744678904 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb940010720 con 0x7fb948105510 2024-03-21T10:37:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.897+0000 7fb94fa00640 1 -- 172.21.15.31:0/2744678904 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9481a2040 con 0x7fb948105510 2024-03-21T10:37:25.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.898+0000 7fb94fa00640 1 -- 172.21.15.31:0/2744678904 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb94810ac80 con 0x7fb948105510 2024-03-21T10:37:25.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.901+0000 7fb946000640 1 -- 172.21.15.31:0/2744678904 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fb940005540 con 0x7fb948105510 2024-03-21T10:37:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.904+0000 7fb946000640 1 --2- 172.21.15.31:0/2744678904 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb92c0a2fa0 0x7fb92c0a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:25.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.904+0000 7fb94d000640 1 --2- 172.21.15.31:0/2744678904 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb92c0a2fa0 0x7fb92c0a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:25.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.904+0000 7fb946000640 1 -- 172.21.15.31:0/2744678904 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fb9400cfeb0 con 0x7fb948105510 2024-03-21T10:37:25.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.905+0000 7fb94d000640 1 --2- 172.21.15.31:0/2744678904 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb92c0a2fa0 0x7fb92c0a5460 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb93802c170 tx=0x7fb93802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:25.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:25.907+0000 7fb946000640 1 -- 172.21.15.31:0/2744678904 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb94009c920 con 0x7fb948105510 2024-03-21T10:37:26.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.113+0000 7fb94fa00640 1 -- 172.21.15.31:0/2744678904 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb948062340 con 0x7fb948105510 2024-03-21T10:37:26.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.114+0000 7fb946000640 1 -- 172.21.15.31:0/2744678904 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24553 (secure 0 0 0) 0x7fb94009c220 con 0x7fb948105510 2024-03-21T10:37:26.113 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:37:26.113 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":93,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","created":"2024-03-21T10:25:41.971117+0000","modified":"2024-03-21T10:36:56.301695+0000","last_up_change":"2024-03-21T10:34:07.757196+0000","last_in_change":"2024-03-21T10:33:47.460406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T10:29:44.553968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T10:35:04.945449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T10:35:05.970589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T10:35:09.688711+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e0244d33-00e2-47d5-9f66-34efc138c462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6803","nonce":3089885125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6805","nonce":3089885125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6809","nonce":3089885125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6807","nonce":3089885125}]},"public_addr":"172.21.15.31:6803/3089885125","cluster_addr":"172.21.15.31:6805/3089885125","heartbeat_back_addr":"172.21.15.31:6809/3089885125","heartbeat_front_addr":"172.21.15.31:6807/3089885125","state":["exists","up"]},{"osd":1,"uuid":"76a63504-beab-4354-a6c3-eebc3b16ad03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6801","nonce":618840037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6803","nonce":618840037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6807","nonce":618840037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6805","nonce":618840037}]},"public_addr":"172.21.15.40:6801/618840037","cluster_addr":"172.21.15.40:6803/618840037","heartbeat_back_addr":"172.21.15.40:6807/618840037","heartbeat_front_addr":"172.21.15.40:6805/618840037","state":["exists","up"]},{"osd":2,"uuid":"fa191bd6-9a21-4ee2-8adf-8a63c9cfd821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6801","nonce":2524184117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6803","nonce":2524184117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6807","nonce":2524184117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6805","nonce":2524184117}]},"public_addr":"172.21.15.70:6801/2524184117","cluster_addr":"172.21.15.70:6803/2524184117","heartbeat_back_addr":"172.21.15.70:6807/2524184117","heartbeat_front_addr":"172.21.15.70:6805/2524184117","state":["exists","up"]},{"osd":3,"uuid":"edb44507-ffb0-42a4-827b-08461e1bf952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6811","nonce":2823036296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6813","nonce":2823036296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6817","nonce":2823036296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6815","nonce":2823036296}]},"public_addr":"172.21.15.31:6811/2823036296","cluster_addr":"172.21.15.31:6813/2823036296","heartbeat_back_addr":"172.21.15.31:6817/2823036296","heartbeat_front_addr":"172.21.15.31:6815/2823036296","state":["exists","up"]},{"osd":4,"uuid":"8cb89153-8c3b-431f-b57c-8863fada76dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6809","nonce":4215186309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6811","nonce":4215186309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6815","nonce":4215186309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6813","nonce":4215186309}]},"public_addr":"172.21.15.40:6809/4215186309","cluster_addr":"172.21.15.40:6811/4215186309","heartbeat_back_addr":"172.21.15.40:6815/4215186309","heartbeat_front_addr":"172.21.15.40:6813/4215186309","state":["exists","up"]},{"osd":5,"uuid":"9409505a-d463-408c-a3cd-218380c04d68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6809","nonce":3402830452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6811","nonce":3402830452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6815","nonce":3402830452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6813","nonce":3402830452}]},"public_addr":"172.21.15.70:6809/3402830452","cluster_addr":"172.21.15.70:6811/3402830452","heartbeat_back_addr":"172.21.15.70:6815/3402830452","heartbeat_front_addr":"172.21.15.70:6813/3402830452","state":["exists","up"]},{"osd":6,"uuid":"f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6819","nonce":1193497058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6821","nonce":1193497058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6825","nonce":1193497058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6823","nonce":1193497058}]},"public_addr":"172.21.15.31:6819/1193497058","cluster_addr":"172.21.15.31:6821/1193497058","heartbeat_back_addr":"172.21.15.31:6825/1193497058","heartbeat_front_addr":"172.21.15.31:6823/1193497058","state":["exists","up"]},{"osd":7,"uuid":"db228a87-29a5-45d4-8fe6-a0a554262eb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6817","nonce":3825628119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6819","nonce":3825628119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6823","nonce":3825628119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6821","nonce":3825628119}]},"public_addr":"172.21.15.40:6817/3825628119","cluster_addr":"172.21.15.40:6819/3825628119","heartbeat_back_addr":"172.21.15.40:6823/3825628119","heartbeat_front_addr":"172.21.15.40:6821/3825628119","state":["exists","up"]},{"osd":8,"uuid":"59b9fde0-095d-4e0b-87da-ead37bd8ff1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6817","nonce":534880586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6819","nonce":534880586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6823","nonce":534880586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6821","nonce":534880586}]},"public_addr":"172.21.15.70:6817/534880586","cluster_addr":"172.21.15.70:6819/534880586","heartbeat_back_addr":"172.21.15.70:6823/534880586","heartbeat_front_addr":"172.21.15.70:6821/534880586","state":["exists","up"]},{"osd":9,"uuid":"77f47061-0c5f-4159-b276-e814935d6c1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6827","nonce":587047469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6829","nonce":587047469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6833","nonce":587047469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6831","nonce":587047469}]},"public_addr":"172.21.15.31:6827/587047469","cluster_addr":"172.21.15.31:6829/587047469","heartbeat_back_addr":"172.21.15.31:6833/587047469","heartbeat_front_addr":"172.21.15.31:6831/587047469","state":["exists","up"]},{"osd":10,"uuid":"8853dc43-7bd1-4c93-8063-f7982fd7da38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6825","nonce":512108711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6827","nonce":512108711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6831","nonce":512108711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6829","nonce":512108711}]},"public_addr":"172.21.15.40:6825/512108711","cluster_addr":"172.21.15.40:6827/512108711","heartbeat_back_addr":"172.21.15.40:6831/512108711","heartbeat_front_addr":"172.21.15.40:6829/512108711","state":["exists","up"]},{"osd":11,"uuid":"84b7c2de-c424-45a6-9583-60097e372a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6825","nonce":540305718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6827","nonce":540305718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6831","nonce":540305718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6829","nonce":540305718}]},"public_addr":"172.21.15.70:6825/540305718","cluster_addr":"172.21.15.70:6827/540305718","heartbeat_back_addr":"172.21.15.70:6831/540305718","heartbeat_front_addr":"172.21.15.70:6829/540305718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:28:51.477358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:17.405045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:41.626268+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:15.339568+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:42.676996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:07.516999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:42.412273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:10.365041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:35.699120+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:11.733639+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:39.505442+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:34:05.449301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451930554":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/3342588843":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6801/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/1626448389":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/1910059351":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/344822770":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/3811368512":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/1981437651":"2024-03-22T10:26:16.959165+0000","172.21.15.31:6801/1981437651":"2024-03-22T10:26:16.959165+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T10:37:26.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.118+0000 7fb94fa00640 1 -- 172.21.15.31:0/2744678904 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb92c0a2fa0 msgr2=0x7fb92c0a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:26.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.118+0000 7fb94fa00640 1 --2- 172.21.15.31:0/2744678904 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb92c0a2fa0 0x7fb92c0a5460 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb93802c170 tx=0x7fb93802df90 comp rx=0 tx=0).stop 2024-03-21T10:37:26.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.118+0000 7fb94fa00640 1 -- 172.21.15.31:0/2744678904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb948105510 msgr2=0x7fb9481a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:26.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.118+0000 7fb94fa00640 1 --2- 172.21.15.31:0/2744678904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb948105510 0x7fb9481a1190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb940009fd0 tx=0x7fb94000ef10 comp rx=0 tx=0).stop 2024-03-21T10:37:26.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.118+0000 7fb94fa00640 1 -- 172.21.15.31:0/2744678904 shutdown_connections 2024-03-21T10:37:26.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.118+0000 7fb94fa00640 1 --2- 172.21.15.31:0/2744678904 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb92c0a2fa0 0x7fb92c0a5460 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:26.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.118+0000 7fb94fa00640 1 --2- 172.21.15.31:0/2744678904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb948105510 0x7fb9481a1190 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:26.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.118+0000 7fb94fa00640 1 --2- 172.21.15.31:0/2744678904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb948104b50 0x7fb94819a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:26.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.118+0000 7fb94fa00640 1 --2- 172.21.15.31:0/2744678904 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb948103960 0x7fb948199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:26.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.118+0000 7fb94fa00640 1 -- 172.21.15.31:0/2744678904 >> 172.21.15.31:0/2744678904 conn(0x7fb9480feeb0 msgr2=0x7fb948107e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:26.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.118+0000 7fb94fa00640 1 -- 172.21.15.31:0/2744678904 shutdown_connections 2024-03-21T10:37:26.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.118+0000 7fb94fa00640 1 -- 172.21.15.31:0/2744678904 wait complete. 2024-03-21T10:37:26.138 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:37:26.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.368+0000 7fecf9600640 1 -- 172.21.15.31:0/3385206354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecf41049b0 msgr2=0x7fecf4104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:26.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.368+0000 7fecf9600640 1 --2- 172.21.15.31:0/3385206354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecf41049b0 0x7fecf4104e30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fece0009900 tx=0x7fece002bc30 comp rx=0 tx=0).stop 2024-03-21T10:37:26.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.368+0000 7fecf9600640 1 -- 172.21.15.31:0/3385206354 shutdown_connections 2024-03-21T10:37:26.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.368+0000 7fecf9600640 1 --2- 172.21.15.31:0/3385206354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fecf4105370 0x7fecf410bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:26.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.368+0000 7fecf9600640 1 --2- 172.21.15.31:0/3385206354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecf41049b0 0x7fecf4104e30 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:26.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.368+0000 7fecf9600640 1 --2- 172.21.15.31:0/3385206354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fecf41037c0 0x7fecf4103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:26.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.368+0000 7fecf9600640 1 -- 172.21.15.31:0/3385206354 >> 172.21.15.31:0/3385206354 conn(0x7fecf40fed50 msgr2=0x7fecf4101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:26.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.368+0000 7fecf9600640 1 -- 172.21.15.31:0/3385206354 shutdown_connections 2024-03-21T10:37:26.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.368+0000 7fecf9600640 1 -- 172.21.15.31:0/3385206354 wait complete. 2024-03-21T10:37:26.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.370+0000 7fecf9600640 1 Processor -- start 2024-03-21T10:37:26.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.370+0000 7fecf9600640 1 -- start start 2024-03-21T10:37:26.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.370+0000 7fecf9600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fecf41037c0 0x7fecf4199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:26.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.370+0000 7fecf9600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fecf41049b0 0x7fecf4199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:26.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.370+0000 7fecf9600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecf4105370 0x7fecf41a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:26.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.370+0000 7fecf9600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf410f490 con 0x7fecf41037c0 2024-03-21T10:37:26.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.370+0000 7fecf9600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf410f310 con 0x7fecf4105370 2024-03-21T10:37:26.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.370+0000 7fecf9600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf410f610 con 0x7fecf41049b0 2024-03-21T10:37:26.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.371+0000 7fecf3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fecf41049b0 0x7fecf4199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:26.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.371+0000 7fecf3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fecf41049b0 0x7fecf4199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:47516/0 (socket says 172.21.15.31:47516) 2024-03-21T10:37:26.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.371+0000 7fecf3400640 1 -- 172.21.15.31:0/1486917795 learned_addr learned my addr 172.21.15.31:0/1486917795 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:26.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.371+0000 7fecf8c00640 1 --2- 172.21.15.31:0/1486917795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecf4105370 0x7fecf41a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:26.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.371+0000 7fecf3e00640 1 --2- 172.21.15.31:0/1486917795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fecf41037c0 0x7fecf4199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:26.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.371+0000 7fecf3400640 1 -- 172.21.15.31:0/1486917795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecf4105370 msgr2=0x7fecf41a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:26.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.371+0000 7fecf3400640 1 --2- 172.21.15.31:0/1486917795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecf4105370 0x7fecf41a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:26.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.371+0000 7fecf3400640 1 -- 172.21.15.31:0/1486917795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fecf41037c0 msgr2=0x7fecf4199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:26.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.371+0000 7fecf3400640 1 --2- 172.21.15.31:0/1486917795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fecf41037c0 0x7fecf4199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:26.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.371+0000 7fecf3400640 1 -- 172.21.15.31:0/1486917795 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf41a16d0 con 0x7fecf41049b0 2024-03-21T10:37:26.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.371+0000 7fecf3e00640 1 --2- 172.21.15.31:0/1486917795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fecf41037c0 0x7fecf4199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:37:26.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.371+0000 7fecf3400640 1 --2- 172.21.15.31:0/1486917795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fecf41049b0 0x7fecf4199ff0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fece0005ab0 tx=0x7fece002c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:26.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.372+0000 7fecf0c00640 1 -- 172.21.15.31:0/1486917795 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece0043020 con 0x7fecf41049b0 2024-03-21T10:37:26.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.372+0000 7fecf0c00640 1 -- 172.21.15.31:0/1486917795 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fece00054e0 con 0x7fecf41049b0 2024-03-21T10:37:26.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.372+0000 7fecf0c00640 1 -- 172.21.15.31:0/1486917795 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece0003ce0 con 0x7fecf41049b0 2024-03-21T10:37:26.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.372+0000 7fecf9600640 1 -- 172.21.15.31:0/1486917795 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf41a1960 con 0x7fecf41049b0 2024-03-21T10:37:26.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.372+0000 7fecf9600640 1 -- 172.21.15.31:0/1486917795 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf41a1e60 con 0x7fecf41049b0 2024-03-21T10:37:26.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.373+0000 7fecf9600640 1 -- 172.21.15.31:0/1486917795 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf410aae0 con 0x7fecf41049b0 2024-03-21T10:37:26.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.378+0000 7fecf0c00640 1 -- 172.21.15.31:0/1486917795 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fece0046020 con 0x7fecf41049b0 2024-03-21T10:37:26.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.380+0000 7fecf0c00640 1 --2- 172.21.15.31:0/1486917795 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fecd80a2f90 0x7fecd80a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:26.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.380+0000 7fecf0c00640 1 -- 172.21.15.31:0/1486917795 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fece00f19a0 con 0x7fecf41049b0 2024-03-21T10:37:26.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.380+0000 7fecf3e00640 1 --2- 172.21.15.31:0/1486917795 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fecd80a2f90 0x7fecd80a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:26.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.380+0000 7fecf0c00640 1 -- 172.21.15.31:0/1486917795 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fece004d020 con 0x7fecf41049b0 2024-03-21T10:37:26.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.381+0000 7fecf3e00640 1 --2- 172.21.15.31:0/1486917795 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fecd80a2f90 0x7fecd80a5450 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fecf4104810 tx=0x7fece4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:26.418 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:26 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/710714697' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:37:26.418 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:26 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2744678904' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:37:26.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:26 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/710714697' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:37:26.858 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:26 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2744678904' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:37:26.859 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:26 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/710714697' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:37:26.859 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:26 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2744678904' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:37:26.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.724+0000 7fecf9600640 1 -- 172.21.15.31:0/1486917795 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fecf402d280 con 0x7fecf41049b0 2024-03-21T10:37:26.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.725+0000 7fecf0c00640 1 -- 172.21.15.31:0/1486917795 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11384 (secure 0 0 0) 0x7fece00be410 con 0x7fecf41049b0 2024-03-21T10:37:26.860 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:37:26.860 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:35:40.990205+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24596,24547]},"id":1}]} 2024-03-21T10:37:26.862 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2024-03-21T10:37:26.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.728+0000 7fecf9600640 1 -- 172.21.15.31:0/1486917795 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fecd80a2f90 msgr2=0x7fecd80a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:26.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.728+0000 7fecf9600640 1 --2- 172.21.15.31:0/1486917795 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fecd80a2f90 0x7fecd80a5450 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fecf4104810 tx=0x7fece4005f90 comp rx=0 tx=0).stop 2024-03-21T10:37:26.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.728+0000 7fecf9600640 1 -- 172.21.15.31:0/1486917795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fecf41049b0 msgr2=0x7fecf4199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:26.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.728+0000 7fecf9600640 1 --2- 172.21.15.31:0/1486917795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fecf41049b0 0x7fecf4199ff0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fece0005ab0 tx=0x7fece002c6c0 comp rx=0 tx=0).stop 2024-03-21T10:37:26.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.728+0000 7fecf9600640 1 -- 172.21.15.31:0/1486917795 shutdown_connections 2024-03-21T10:37:26.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.728+0000 7fecf9600640 1 --2- 172.21.15.31:0/1486917795 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fecd80a2f90 0x7fecd80a5450 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:26.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.728+0000 7fecf9600640 1 --2- 172.21.15.31:0/1486917795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecf4105370 0x7fecf41a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:26.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.728+0000 7fecf9600640 1 --2- 172.21.15.31:0/1486917795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fecf41049b0 0x7fecf4199ff0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:26.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.728+0000 7fecf9600640 1 --2- 172.21.15.31:0/1486917795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fecf41037c0 0x7fecf4199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:26.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.728+0000 7fecf9600640 1 -- 172.21.15.31:0/1486917795 >> 172.21.15.31:0/1486917795 conn(0x7fecf40fed50 msgr2=0x7fecf4107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:26.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.728+0000 7fecf9600640 1 -- 172.21.15.31:0/1486917795 shutdown_connections 2024-03-21T10:37:26.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:26.728+0000 7fecf9600640 1 -- 172.21.15.31:0/1486917795 wait complete. 2024-03-21T10:37:26.864 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-21T10:35:06.492115+0000', 'modified': '2024-03-21T10:35:40.990205+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24596, 'mds_1': 24584, 'mds_2': 24547}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24511': {'gid': 24511, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.70:6835/729664889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6834', 'nonce': 729664889}, {'type': 'v1', 'addr': '172.21.15.70:6835', 'nonce': 729664889}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24523': {'gid': 24523, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.70:6837/3650805915', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6836', 'nonce': 3650805915}, {'type': 'v1', 'addr': '172.21.15.70:6837', 'nonce': 3650805915}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24539': {'gid': 24539, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.40:6837/4121707657', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6836', 'nonce': 4121707657}, {'type': 'v1', 'addr': '172.21.15.40:6837', 'nonce': 4121707657}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24547': {'gid': 24547, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.70:6839/3204327153', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6838', 'nonce': 3204327153}, {'type': 'v1', 'addr': '172.21.15.70:6839', 'nonce': 3204327153}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24584': {'gid': 24584, 'name': 'a', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.31:6841/1752599123', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6840', 'nonce': 1752599123}, {'type': 'v1', 'addr': '172.21.15.31:6841', 'nonce': 1752599123}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24596': {'gid': 24596, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6839/2539201148', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6838', 'nonce': 2539201148}, {'type': 'v1', 'addr': '172.21.15.40:6839', 'nonce': 2539201148}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24596, 'qdb_cluster': [24584, 24596, 24547]} 2024-03-21T10:37:26.864 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-21T10:37:26.864 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-21T10:37:27.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.087+0000 7fada7e00640 1 -- 172.21.15.31:0/2771619258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fada8105080 msgr2=0x7fada8105500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.087+0000 7fada7e00640 1 --2- 172.21.15.31:0/2771619258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fada8105080 0x7fada8105500 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fada0009ca0 tx=0x7fada002c100 comp rx=0 tx=0).stop 2024-03-21T10:37:27.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.087+0000 7fada7e00640 1 -- 172.21.15.31:0/2771619258 shutdown_connections 2024-03-21T10:37:27.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.087+0000 7fada7e00640 1 --2- 172.21.15.31:0/2771619258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fada8105080 0x7fada8105500 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.087+0000 7fada7e00640 1 --2- 172.21.15.31:0/2771619258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fada81026d0 0x7fada8104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.087+0000 7fada7e00640 1 --2- 172.21.15.31:0/2771619258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fada80ffd60 0x7fada8102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.087+0000 7fada7e00640 1 -- 172.21.15.31:0/2771619258 >> 172.21.15.31:0/2771619258 conn(0x7fada80fb980 msgr2=0x7fada80fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:27.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.087+0000 7fada7e00640 1 -- 172.21.15.31:0/2771619258 shutdown_connections 2024-03-21T10:37:27.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.087+0000 7fada7e00640 1 -- 172.21.15.31:0/2771619258 wait complete. 2024-03-21T10:37:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.088+0000 7fada7e00640 1 Processor -- start 2024-03-21T10:37:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.088+0000 7fada7e00640 1 -- start start 2024-03-21T10:37:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.089+0000 7fada7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fada80ffd60 0x7fada819dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.089+0000 7fada7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fada81026d0 0x7fada819e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.089+0000 7fada7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fada8105080 0x7fada81a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.089+0000 7fada7e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada8113740 con 0x7fada80ffd60 2024-03-21T10:37:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.089+0000 7fada7e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada81135c0 con 0x7fada81026d0 2024-03-21T10:37:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.089+0000 7fada7e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada81138c0 con 0x7fada8105080 2024-03-21T10:37:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.089+0000 7fad9fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fada81026d0 0x7fada819e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.089+0000 7fada6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fada80ffd60 0x7fada819dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.089+0000 7fada6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fada80ffd60 0x7fada819dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52628/0 (socket says 172.21.15.31:52628) 2024-03-21T10:37:27.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.089+0000 7fada6a00640 1 -- 172.21.15.31:0/1856659426 learned_addr learned my addr 172.21.15.31:0/1856659426 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:27.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.089+0000 7fada7400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fada8105080 0x7fada81a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.089+0000 7fada7400640 1 -- 172.21.15.31:0/1856659426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fada81026d0 msgr2=0x7fada819e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.089+0000 7fada7400640 1 --2- 172.21.15.31:0/1856659426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fada81026d0 0x7fada819e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.089+0000 7fada7400640 1 -- 172.21.15.31:0/1856659426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fada80ffd60 msgr2=0x7fada819dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.089+0000 7fada7400640 1 --2- 172.21.15.31:0/1856659426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fada80ffd60 0x7fada819dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.089+0000 7fada7400640 1 -- 172.21.15.31:0/1856659426 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada81a59b0 con 0x7fada8105080 2024-03-21T10:37:27.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.089+0000 7fada6a00640 1 --2- 172.21.15.31:0/1856659426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fada80ffd60 0x7fada819dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:37:27.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.090+0000 7fada7400640 1 --2- 172.21.15.31:0/1856659426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fada8105080 0x7fada81a5310 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fada002c640 tx=0x7fada0035d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:27.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.090+0000 7fad9f400640 1 -- 172.21.15.31:0/1856659426 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada0039040 con 0x7fada8105080 2024-03-21T10:37:27.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.090+0000 7fad9f400640 1 -- 172.21.15.31:0/1856659426 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fada0044070 con 0x7fada8105080 2024-03-21T10:37:27.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.090+0000 7fad9f400640 1 -- 172.21.15.31:0/1856659426 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada00043e0 con 0x7fada8105080 2024-03-21T10:37:27.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.090+0000 7fada7e00640 1 -- 172.21.15.31:0/1856659426 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada81a5c40 con 0x7fada8105080 2024-03-21T10:37:27.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.090+0000 7fada7e00640 1 -- 172.21.15.31:0/1856659426 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada81a60b0 con 0x7fada8105080 2024-03-21T10:37:27.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.093+0000 7fad9f400640 1 -- 172.21.15.31:0/1856659426 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fada0004580 con 0x7fada8105080 2024-03-21T10:37:27.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.093+0000 7fadb0814740 1 Processor -- start 2024-03-21T10:37:27.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.094+0000 7fadb0814740 1 -- start start 2024-03-21T10:37:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.094+0000 7fadb0814740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f54082ed80 0x55f54082f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.094+0000 7fadb0814740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f54082f770 0x55f540834270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.094+0000 7fadb0814740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5408348e0 0x55f540836d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.094+0000 7fadb0814740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f540793c90 con 0x55f54082f770 2024-03-21T10:37:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.094+0000 7fadb0814740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f54073c980 con 0x55f54082ed80 2024-03-21T10:37:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.094+0000 7fadb0814740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f54078fd30 con 0x55f5408348e0 2024-03-21T10:37:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.094+0000 7fad9fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f54082ed80 0x55f54082f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.094+0000 7fada6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f54082f770 0x55f540834270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.094+0000 7fad9fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f54082ed80 0x55f54082f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:59296/0 (socket says 172.21.15.31:59296) 2024-03-21T10:37:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.094+0000 7fad9fe00640 1 -- 172.21.15.31:0/2787984101 learned_addr learned my addr 172.21.15.31:0/2787984101 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.094+0000 7fad9fe00640 1 --2- 172.21.15.31:0/2787984101 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5408348e0 0x55f540836d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.095+0000 7fada6a00640 1 -- 172.21.15.31:0/2787984101 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5408348e0 msgr2=0x55f540836d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:37:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.095+0000 7fada6a00640 1 --2- 172.21.15.31:0/2787984101 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5408348e0 0x55f540836d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.095+0000 7fada6a00640 1 -- 172.21.15.31:0/2787984101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f54082ed80 msgr2=0x55f54082f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:37:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.095+0000 7fada6a00640 1 --2- 172.21.15.31:0/2787984101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f54082ed80 0x55f54082f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.095+0000 7fada6a00640 1 -- 172.21.15.31:0/2787984101 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f540754510 con 0x55f54082f770 2024-03-21T10:37:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.095+0000 7fad9fe00640 1 --2- 172.21.15.31:0/2787984101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f54082ed80 0x55f54082f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:37:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.095+0000 7fad9f400640 1 --2- 172.21.15.31:0/1856659426 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fad8c0a2fa0 0x7fad8c0a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.095+0000 7fada6a00640 1 --2- 172.21.15.31:0/2787984101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f54082f770 0x55f540834270 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fada819f270 tx=0x7fad980119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3027aefdba050278 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.095+0000 7fad9f400640 1 -- 172.21.15.31:0/1856659426 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fada00f1e10 con 0x7fada8105080 2024-03-21T10:37:27.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.096+0000 7fad9fe00640 1 --2- 172.21.15.31:0/1856659426 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fad8c0a2fa0 0x7fad8c0a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.096+0000 7fad97e00640 1 -- 172.21.15.31:0/2787984101 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad98004030 con 0x55f54082f770 2024-03-21T10:37:27.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.096+0000 7fad97e00640 1 -- 172.21.15.31:0/2787984101 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad98005500 con 0x55f54082f770 2024-03-21T10:37:27.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.096+0000 7fad97e00640 1 -- 172.21.15.31:0/2787984101 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad98004e00 con 0x55f54082f770 2024-03-21T10:37:27.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.096+0000 7fadb0814740 1 -- 172.21.15.31:0/2787984101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f54082f770 msgr2=0x55f540834270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.096+0000 7fadb0814740 1 --2- 172.21.15.31:0/2787984101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f54082f770 0x55f540834270 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fada819f270 tx=0x7fad980119a0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.096+0000 7fad9fe00640 1 --2- 172.21.15.31:0/1856659426 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fad8c0a2fa0 0x7fad8c0a5460 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x55f54082ffd0 tx=0x7fad90009370 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:27.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.097+0000 7fadb0814740 1 -- 172.21.15.31:0/2787984101 shutdown_connections 2024-03-21T10:37:27.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.097+0000 7fadb0814740 1 --2- 172.21.15.31:0/2787984101 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5408348e0 0x55f540836d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.097+0000 7fadb0814740 1 --2- 172.21.15.31:0/2787984101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f54082f770 0x55f540834270 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.097+0000 7fadb0814740 1 --2- 172.21.15.31:0/2787984101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f54082ed80 0x55f54082f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.097+0000 7fadb0814740 1 -- 172.21.15.31:0/2787984101 >> 172.21.15.31:0/2787984101 conn(0x55f54082a2f0 msgr2=0x55f54082a6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:27.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.097+0000 7fadb0814740 1 -- 172.21.15.31:0/2787984101 shutdown_connections 2024-03-21T10:37:27.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.097+0000 7fadb0814740 1 -- 172.21.15.31:0/2787984101 wait complete. 2024-03-21T10:37:27.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.097+0000 7fadb0814740 1 -- start start 2024-03-21T10:37:27.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.097+0000 7fadb0814740 1 Processor -- start 2024-03-21T10:37:27.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.098+0000 7fadb0814740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f54082ed80 0x55f540839230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.098+0000 7fadb0814740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f54082f770 0x55f540839770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.098+0000 7fadb0814740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5408348e0 0x55f5408407b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.098+0000 7fadb0814740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f540837890 con 0x55f5408348e0 2024-03-21T10:37:27.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.098+0000 7fadb0814740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f54074fdd0 con 0x55f54082f770 2024-03-21T10:37:27.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.098+0000 7fadb0814740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f540837a10 con 0x55f54082ed80 2024-03-21T10:37:27.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.098+0000 7fadb0814740 10 client.0 _ll_register_callbacks cb 0x55f5404a7ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:37:27.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.098+0000 7fada6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f54082ed80 0x55f540839230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.098+0000 7fad9fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f54082f770 0x55f540839770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.098+0000 7fada6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f54082ed80 0x55f540839230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:55122/0 (socket says 172.21.15.31:55122) 2024-03-21T10:37:27.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.098+0000 7fada6a00640 1 -- 172.21.15.31:0/713524107 learned_addr learned my addr 172.21.15.31:0/713524107 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:27.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.098+0000 7fad9fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f54082f770 0x55f540839770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:59302/0 (socket says 172.21.15.31:59302) 2024-03-21T10:37:27.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.098+0000 7fada7400640 1 --2- 172.21.15.31:0/713524107 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5408348e0 0x55f5408407b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.099+0000 7fada6a00640 1 -- 172.21.15.31:0/713524107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f54082f770 msgr2=0x55f540839770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:37:27.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.099+0000 7fada6a00640 1 --2- 172.21.15.31:0/713524107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f54082f770 0x55f540839770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.099+0000 7fada6a00640 1 -- 172.21.15.31:0/713524107 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5408348e0 msgr2=0x55f5408407b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:37:27.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.099+0000 7fada6a00640 1 --2- 172.21.15.31:0/713524107 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5408348e0 0x55f5408407b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.099+0000 7fada6a00640 1 -- 172.21.15.31:0/713524107 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f5404a82d0 con 0x55f54082ed80 2024-03-21T10:37:27.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.099+0000 7fad9fe00640 1 --2- 172.21.15.31:0/713524107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f54082f770 0x55f540839770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:37:27.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.099+0000 7fada6a00640 1 --2- 172.21.15.31:0/713524107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f54082ed80 0x55f540839230 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fad9801b040 tx=0x7fad98005220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2487b2457f73c15c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:27.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.099+0000 7fad94c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:37:27.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.099+0000 7fad94c00640 1 -- 172.21.15.31:0/713524107 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad98028070 con 0x55f54082ed80 2024-03-21T10:37:27.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.099+0000 7fad94c00640 1 -- 172.21.15.31:0/713524107 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fad9800f5c0 con 0x55f54082ed80 2024-03-21T10:37:27.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.099+0000 7fadb0814740 1 -- 172.21.15.31:0/713524107 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fad98005500 con 0x55f54082ed80 2024-03-21T10:37:27.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.100+0000 7fad94c00640 1 -- 172.21.15.31:0/713524107 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad98034050 con 0x55f54082ed80 2024-03-21T10:37:27.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.100+0000 7fadb0814740 10 client.25156 fetch_fsmap learned FSMap version 23 2024-03-21T10:37:27.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.100+0000 7fadb0814740 1 -- 172.21.15.31:0/713524107 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f5405726b0 con 0x55f54082ed80 2024-03-21T10:37:27.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.100+0000 7fad94c00640 1 -- 172.21.15.31:0/713524107 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad98005500 con 0x55f54082ed80 2024-03-21T10:37:27.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.100+0000 7fad94c00640 1 -- 172.21.15.31:0/713524107 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 23) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fad98019610 con 0x55f54082ed80 2024-03-21T10:37:27.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.100+0000 7fadb0814740 10 client.25156 fetch_fsmap finished waiting for FSMap version 23 2024-03-21T10:37:27.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.100+0000 7fadb0814740 10 client.25156 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-21T10:37:27.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.100+0000 7fadb0814740 20 client.25156 populate_metadata read hostname 'smithi031' 2024-03-21T10:37:27.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.100+0000 7fadb0814740 1 --2- 172.21.15.31:0/713524107 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55f540845750 0x55f5408449d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.100+0000 7fadb0814740 4 client.25156 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:37:27.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.100+0000 7fadb0814740 1 -- 172.21.15.31:0/713524107 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f540735520 con 0x55f540845750 2024-03-21T10:37:27.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.101+0000 7fad9fe00640 1 --2- 172.21.15.31:0/713524107 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55f540845750 0x55f5408449d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.101+0000 7fad9fe00640 1 --2- 172.21.15.31:0/713524107 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55f540845750 0x55f5408449d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=46b3d7a0f75506a9 server_cookie=906507d6f039a8ce in_seq=0 out_seq=0 2024-03-21T10:37:27.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.101+0000 7fad94c00640 10 client.25156 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:37:27.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.102+0000 7fad94c00640 1 -- 172.21.15.31:0/713524107 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f540735520 con 0x55f540845750 2024-03-21T10:37:27.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.102+0000 7fad94c00640 10 client.25156 handle_command_reply: tid=0 2024-03-21T10:37:27.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.102+0000 7fadb0814740 1 client.25156 shutdown 2024-03-21T10:37:27.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.102+0000 7fadb0814740 1 -- 172.21.15.31:0/713524107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f54082ed80 msgr2=0x55f540839230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.102+0000 7fadb0814740 1 --2- 172.21.15.31:0/713524107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f54082ed80 0x55f540839230 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fad9801b040 tx=0x7fad98005220 comp rx=0 tx=0).stop 2024-03-21T10:37:27.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.102+0000 7fadb0814740 1 -- 172.21.15.31:0/713524107 shutdown_connections 2024-03-21T10:37:27.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.102+0000 7fadb0814740 1 --2- 172.21.15.31:0/713524107 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55f540845750 0x55f5408449d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.102+0000 7fadb0814740 1 --2- 172.21.15.31:0/713524107 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5408348e0 0x55f5408407b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.102+0000 7fadb0814740 1 --2- 172.21.15.31:0/713524107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f54082f770 0x55f540839770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.102+0000 7fadb0814740 1 --2- 172.21.15.31:0/713524107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f54082ed80 0x55f540839230 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.102+0000 7fad94c00640 0 client.25156 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:37:27.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.102+0000 7fadb0814740 1 -- 172.21.15.31:0/713524107 >> 172.21.15.31:0/713524107 conn(0x55f54082a2f0 msgr2=0x55f5408294f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:27.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.103+0000 7fadb0814740 1 -- 172.21.15.31:0/713524107 shutdown_connections 2024-03-21T10:37:27.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.103+0000 7fadb0814740 1 -- 172.21.15.31:0/713524107 wait complete. 2024-03-21T10:37:27.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.103+0000 7fadb0814740 20 client.25156 trim_cache size 0 max 16384 2024-03-21T10:37:27.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.133+0000 7fadb0814740 1 Processor -- start 2024-03-21T10:37:27.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.133+0000 7fadb0814740 1 -- start start 2024-03-21T10:37:27.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.133+0000 7fadb0814740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f54082ed80 0x55f540837de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.133+0000 7fadb0814740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f54082f770 0x55f5408406d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.133+0000 7fadb0814740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5408348e0 0x55f540828c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.133+0000 7fadb0814740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f540844f70 con 0x55f5408348e0 2024-03-21T10:37:27.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.133+0000 7fadb0814740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5408455d0 con 0x55f54082ed80 2024-03-21T10:37:27.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.133+0000 7fadb0814740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f540845c30 con 0x55f54082f770 2024-03-21T10:37:27.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.134+0000 7fad9fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f54082f770 0x55f5408406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.134+0000 7fada7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5408348e0 0x55f540828c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.134+0000 7fada6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f54082ed80 0x55f540837de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.134+0000 7fad9fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f54082f770 0x55f5408406d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:55136/0 (socket says 172.21.15.31:55136) 2024-03-21T10:37:27.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.134+0000 7fada6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f54082ed80 0x55f540837de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:59308/0 (socket says 172.21.15.31:59308) 2024-03-21T10:37:27.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.134+0000 7fad9fe00640 1 -- 172.21.15.31:0/2545162957 learned_addr learned my addr 172.21.15.31:0/2545162957 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:27.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.134+0000 7fad9fe00640 1 -- 172.21.15.31:0/2545162957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f54082ed80 msgr2=0x55f540837de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:37:27.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.134+0000 7fad9fe00640 1 --2- 172.21.15.31:0/2545162957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f54082ed80 0x55f540837de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.134+0000 7fad9fe00640 1 -- 172.21.15.31:0/2545162957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5408348e0 msgr2=0x55f540828c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:37:27.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.134+0000 7fad9fe00640 1 --2- 172.21.15.31:0/2545162957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5408348e0 0x55f540828c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.134+0000 7fad9fe00640 1 -- 172.21.15.31:0/2545162957 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f54082b060 con 0x55f54082f770 2024-03-21T10:37:27.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.134+0000 7fad9fe00640 1 --2- 172.21.15.31:0/2545162957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f54082f770 0x55f5408406d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fad90009ae0 tx=0x7fad900046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d36e65c41c002005 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:27.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.134+0000 7fad97400640 1 -- 172.21.15.31:0/2545162957 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad90020c00 con 0x55f54082f770 2024-03-21T10:37:27.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.134+0000 7fad97400640 1 -- 172.21.15.31:0/2545162957 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad90020da0 con 0x55f54082f770 2024-03-21T10:37:27.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.134+0000 7fad97400640 1 -- 172.21.15.31:0/2545162957 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad90004d90 con 0x55f54082f770 2024-03-21T10:37:27.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.135+0000 7fadb0814740 1 -- 172.21.15.31:0/2545162957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f54082f770 msgr2=0x55f5408406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.135+0000 7fadb0814740 1 --2- 172.21.15.31:0/2545162957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f54082f770 0x55f5408406d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fad90009ae0 tx=0x7fad900046c0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.135+0000 7fadb0814740 1 -- 172.21.15.31:0/2545162957 shutdown_connections 2024-03-21T10:37:27.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.135+0000 7fadb0814740 1 --2- 172.21.15.31:0/2545162957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5408348e0 0x55f540828c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.135+0000 7fadb0814740 1 --2- 172.21.15.31:0/2545162957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f54082f770 0x55f5408406d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.135+0000 7fadb0814740 1 --2- 172.21.15.31:0/2545162957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f54082ed80 0x55f540837de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.135+0000 7fadb0814740 1 -- 172.21.15.31:0/2545162957 >> 172.21.15.31:0/2545162957 conn(0x55f54082a2f0 msgr2=0x55f54082a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:27.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.135+0000 7fadb0814740 1 -- 172.21.15.31:0/2545162957 shutdown_connections 2024-03-21T10:37:27.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.135+0000 7fadb0814740 1 -- 172.21.15.31:0/2545162957 wait complete. 2024-03-21T10:37:27.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.136+0000 7fadb0814740 1 -- start start 2024-03-21T10:37:27.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.136+0000 7fadb0814740 1 Processor -- start 2024-03-21T10:37:27.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.136+0000 7fadb0814740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f54082ed80 0x55f54083c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.136+0000 7fadb0814740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f54082f770 0x55f54083ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.136+0000 7fadb0814740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5408348e0 0x55f54083faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.136+0000 7fadb0814740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5408468f0 con 0x55f5408348e0 2024-03-21T10:37:27.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.136+0000 7fadb0814740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f540846290 con 0x55f54082f770 2024-03-21T10:37:27.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.136+0000 7fadb0814740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f540846f50 con 0x55f54082ed80 2024-03-21T10:37:27.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.136+0000 7fadb0814740 10 client.0 _ll_register_callbacks cb 0x7fad90009b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:37:27.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.136+0000 7fada6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f54082ed80 0x55f54083c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.137+0000 7fada7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5408348e0 0x55f54083faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.137+0000 7fada6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f54082ed80 0x55f54083c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:55152/0 (socket says 172.21.15.31:55152) 2024-03-21T10:37:27.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.137+0000 7fada6a00640 1 -- 172.21.15.31:0/1779767161 learned_addr learned my addr 172.21.15.31:0/1779767161 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:27.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.137+0000 7fad9fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f54082f770 0x55f54083ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.137+0000 7fada6a00640 1 -- 172.21.15.31:0/1779767161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f54082f770 msgr2=0x55f54083ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:37:27.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.137+0000 7fada6a00640 1 --2- 172.21.15.31:0/1779767161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f54082f770 0x55f54083ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.137+0000 7fada6a00640 1 -- 172.21.15.31:0/1779767161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5408348e0 msgr2=0x55f54083faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:37:27.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.137+0000 7fada6a00640 1 --2- 172.21.15.31:0/1779767161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5408348e0 0x55f54083faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.137+0000 7fada6a00640 1 -- 172.21.15.31:0/1779767161 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f5408400e0 con 0x55f54082ed80 2024-03-21T10:37:27.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.137+0000 7fad9fe00640 1 --2- 172.21.15.31:0/1779767161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f54082f770 0x55f54083ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:37:27.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.137+0000 7fada6a00640 1 --2- 172.21.15.31:0/1779767161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f54082ed80 0x55f54083c900 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fad98009380 tx=0x7fad98005720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cad7e4371c565db server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:27.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.137+0000 7fad94c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:37:27.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.137+0000 7fad94c00640 1 -- 172.21.15.31:0/1779767161 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad98028040 con 0x55f54082ed80 2024-03-21T10:37:27.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.137+0000 7fad94c00640 1 -- 172.21.15.31:0/1779767161 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fad98005290 con 0x55f54082ed80 2024-03-21T10:37:27.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.137+0000 7fadb0814740 1 -- 172.21.15.31:0/1779767161 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fad90020da0 con 0x55f54082ed80 2024-03-21T10:37:27.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.138+0000 7fad94c00640 1 -- 172.21.15.31:0/1779767161 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad98041050 con 0x55f54082ed80 2024-03-21T10:37:27.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.138+0000 7fadb0814740 10 client.25162 fetch_fsmap learned FSMap version 23 2024-03-21T10:37:27.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.138+0000 7fadb0814740 1 -- 172.21.15.31:0/1779767161 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f5408a0f30 con 0x55f54082ed80 2024-03-21T10:37:27.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.138+0000 7fad94c00640 1 -- 172.21.15.31:0/1779767161 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad90020da0 con 0x55f54082ed80 2024-03-21T10:37:27.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.138+0000 7fad94c00640 1 -- 172.21.15.31:0/1779767161 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 23) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fad980035f0 con 0x55f54082ed80 2024-03-21T10:37:27.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.138+0000 7fadb0814740 10 client.25162 fetch_fsmap finished waiting for FSMap version 23 2024-03-21T10:37:27.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.138+0000 7fadb0814740 10 client.25162 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-21T10:37:27.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.138+0000 7fadb0814740 20 client.25162 populate_metadata read hostname 'smithi031' 2024-03-21T10:37:27.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.138+0000 7fadb0814740 1 --2- 172.21.15.31:0/1779767161 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55f5408a5460 0x55f5408a1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.138+0000 7fadb0814740 4 client.25162 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-21T10:37:27.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.138+0000 7fadb0814740 1 -- 172.21.15.31:0/1779767161 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f5407a11d0 con 0x55f5408a5460 2024-03-21T10:37:27.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.139+0000 7fad9fe00640 1 --2- 172.21.15.31:0/1779767161 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55f5408a5460 0x55f5408a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.139+0000 7fad9fe00640 1 --2- 172.21.15.31:0/1779767161 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55f5408a5460 0x55f5408a1100 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7dee27d7c53cd5db server_cookie=8136ef84f67459e2 in_seq=0 out_seq=0 2024-03-21T10:37:27.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.139+0000 7fad94c00640 10 client.25162 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:37:27.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.139+0000 7fad94c00640 1 -- 172.21.15.31:0/1779767161 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55f5407a11d0 con 0x55f5408a5460 2024-03-21T10:37:27.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.139+0000 7fad94c00640 10 client.25162 handle_command_reply: tid=0 2024-03-21T10:37:27.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.139+0000 7fadb0814740 1 client.25162 shutdown 2024-03-21T10:37:27.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.139+0000 7fadb0814740 1 -- 172.21.15.31:0/1779767161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f54082ed80 msgr2=0x55f54083c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.139+0000 7fadb0814740 1 --2- 172.21.15.31:0/1779767161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f54082ed80 0x55f54083c900 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fad98009380 tx=0x7fad98005720 comp rx=0 tx=0).stop 2024-03-21T10:37:27.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.140+0000 7fadb0814740 1 -- 172.21.15.31:0/1779767161 shutdown_connections 2024-03-21T10:37:27.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.140+0000 7fadb0814740 1 --2- 172.21.15.31:0/1779767161 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55f5408a5460 0x55f5408a1100 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.140+0000 7fadb0814740 1 --2- 172.21.15.31:0/1779767161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5408348e0 0x55f54083faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.140+0000 7fadb0814740 1 --2- 172.21.15.31:0/1779767161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f54082f770 0x55f54083ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.140+0000 7fadb0814740 1 --2- 172.21.15.31:0/1779767161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f54082ed80 0x55f54083c900 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.140+0000 7fadb0814740 1 -- 172.21.15.31:0/1779767161 >> 172.21.15.31:0/1779767161 conn(0x55f54082a2f0 msgr2=0x55f54082d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:27.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.140+0000 7fad94c00640 0 client.25162 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:37:27.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.140+0000 7fadb0814740 1 -- 172.21.15.31:0/1779767161 shutdown_connections 2024-03-21T10:37:27.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.140+0000 7fadb0814740 1 -- 172.21.15.31:0/1779767161 wait complete. 2024-03-21T10:37:27.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.140+0000 7fadb0814740 20 client.25162 trim_cache size 0 max 16384 2024-03-21T10:37:27.143 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:37:27.143 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "3847408c-e76d-11ee-95cd-87774f69a715", 2024-03-21T10:37:27.143 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 2, 2024-03-21T10:37:27.143 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24547, 2024-03-21T10:37:27.143 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-03-21T10:37:27.143 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-03-21T10:37:27.143 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-03-21T10:37:27.143 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 106.817625857, 2024-03-21T10:37:27.144 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 23, 2024-03-21T10:37:27.144 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 93, 2024-03-21T10:37:27.144 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 89, 2024-03-21T10:37:27.144 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 157.24254223899999 2024-03-21T10:37:27.144 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:37:27.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.140+0000 7fada7e00640 1 -- 172.21.15.31:0/1856659426 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fad8c0a2fa0 msgr2=0x7fad8c0a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.140+0000 7fada7e00640 1 --2- 172.21.15.31:0/1856659426 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fad8c0a2fa0 0x7fad8c0a5460 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x55f54082ffd0 tx=0x7fad90009370 comp rx=0 tx=0).stop 2024-03-21T10:37:27.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.140+0000 7fada7e00640 1 -- 172.21.15.31:0/1856659426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fada8105080 msgr2=0x7fada81a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.140+0000 7fada7e00640 1 --2- 172.21.15.31:0/1856659426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fada8105080 0x7fada81a5310 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fada002c640 tx=0x7fada0035d90 comp rx=0 tx=0).stop 2024-03-21T10:37:27.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.141+0000 7fada7e00640 1 -- 172.21.15.31:0/1856659426 shutdown_connections 2024-03-21T10:37:27.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.141+0000 7fada7e00640 1 --2- 172.21.15.31:0/1856659426 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fad8c0a2fa0 0x7fad8c0a5460 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.141+0000 7fada7e00640 1 --2- 172.21.15.31:0/1856659426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fada8105080 0x7fada81a5310 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.141+0000 7fada7e00640 1 --2- 172.21.15.31:0/1856659426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fada81026d0 0x7fada819e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.141+0000 7fada7e00640 1 --2- 172.21.15.31:0/1856659426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fada80ffd60 0x7fada819dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.141+0000 7fada7e00640 1 -- 172.21.15.31:0/1856659426 >> 172.21.15.31:0/1856659426 conn(0x7fada80fb980 msgr2=0x7fada8100a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:27.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.141+0000 7fada7e00640 1 -- 172.21.15.31:0/1856659426 shutdown_connections 2024-03-21T10:37:27.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.141+0000 7fada7e00640 1 -- 172.21.15.31:0/1856659426 wait complete. 2024-03-21T10:37:27.160 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-21T10:37:27.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.366+0000 7fcac8a00640 1 -- 172.21.15.31:0/255593750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcac41028a0 msgr2=0x7fcac4104d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.366+0000 7fcac8a00640 1 --2- 172.21.15.31:0/255593750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcac41028a0 0x7fcac4104d10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcab0009900 tx=0x7fcab002bc30 comp rx=0 tx=0).stop 2024-03-21T10:37:27.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.367+0000 7fcac8a00640 1 -- 172.21.15.31:0/255593750 shutdown_connections 2024-03-21T10:37:27.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.367+0000 7fcac8a00640 1 --2- 172.21.15.31:0/255593750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcac4105250 0x7fcac41056d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.367+0000 7fcac8a00640 1 --2- 172.21.15.31:0/255593750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcac41028a0 0x7fcac4104d10 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.367+0000 7fcac8a00640 1 --2- 172.21.15.31:0/255593750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcac40fff30 0x7fcac4102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.367+0000 7fcac8a00640 1 -- 172.21.15.31:0/255593750 >> 172.21.15.31:0/255593750 conn(0x7fcac40fbb30 msgr2=0x7fcac40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:27.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.367+0000 7fcac8a00640 1 -- 172.21.15.31:0/255593750 shutdown_connections 2024-03-21T10:37:27.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.367+0000 7fcac8a00640 1 -- 172.21.15.31:0/255593750 wait complete. 2024-03-21T10:37:27.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.368+0000 7fcac8a00640 1 Processor -- start 2024-03-21T10:37:27.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.368+0000 7fcac8a00640 1 -- start start 2024-03-21T10:37:27.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.368+0000 7fcac8a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcac40fff30 0x7fcac419dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.368+0000 7fcac8a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcac41028a0 0x7fcac419e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.368+0000 7fcac8a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcac4105250 0x7fcac41a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.368+0000 7fcac8a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac4113960 con 0x7fcac41028a0 2024-03-21T10:37:27.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.368+0000 7fcac8a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac41137e0 con 0x7fcac4105250 2024-03-21T10:37:27.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.368+0000 7fcac8a00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac4113ae0 con 0x7fcac40fff30 2024-03-21T10:37:27.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.369+0000 7fcac1e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcac40fff30 0x7fcac419dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.369+0000 7fcac1e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcac40fff30 0x7fcac419dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:55168/0 (socket says 172.21.15.31:55168) 2024-03-21T10:37:27.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.369+0000 7fcac1e00640 1 -- 172.21.15.31:0/2237162928 learned_addr learned my addr 172.21.15.31:0/2237162928 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:27.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.369+0000 7fcac1400640 1 --2- 172.21.15.31:0/2237162928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcac41028a0 0x7fcac419e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.369+0000 7fcac2800640 1 --2- 172.21.15.31:0/2237162928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcac4105250 0x7fcac41a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.369+0000 7fcac1e00640 1 -- 172.21.15.31:0/2237162928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcac4105250 msgr2=0x7fcac41a5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.369+0000 7fcac1e00640 1 --2- 172.21.15.31:0/2237162928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcac4105250 0x7fcac41a5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.369+0000 7fcac1e00640 1 -- 172.21.15.31:0/2237162928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcac41028a0 msgr2=0x7fcac419e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.369+0000 7fcac1e00640 1 --2- 172.21.15.31:0/2237162928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcac41028a0 0x7fcac419e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.369+0000 7fcac1e00640 1 -- 172.21.15.31:0/2237162928 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac41a5bc0 con 0x7fcac40fff30 2024-03-21T10:37:27.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.369+0000 7fcac2800640 1 --2- 172.21.15.31:0/2237162928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcac4105250 0x7fcac41a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:37:27.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.369+0000 7fcac1e00640 1 --2- 172.21.15.31:0/2237162928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcac40fff30 0x7fcac419dfa0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcab400e9f0 tx=0x7fcab400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:27.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.370+0000 7fcabaa00640 1 -- 172.21.15.31:0/2237162928 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab400cd80 con 0x7fcac40fff30 2024-03-21T10:37:27.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.370+0000 7fcabaa00640 1 -- 172.21.15.31:0/2237162928 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcab4010910 con 0x7fcac40fff30 2024-03-21T10:37:27.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.370+0000 7fcac8a00640 1 -- 172.21.15.31:0/2237162928 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac41a5eb0 con 0x7fcac40fff30 2024-03-21T10:37:27.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.370+0000 7fcabaa00640 1 -- 172.21.15.31:0/2237162928 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab4018990 con 0x7fcac40fff30 2024-03-21T10:37:27.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.370+0000 7fcac8a00640 1 -- 172.21.15.31:0/2237162928 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac41a6320 con 0x7fcac40fff30 2024-03-21T10:37:27.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.374+0000 7fcabaa00640 1 -- 172.21.15.31:0/2237162928 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fcab4010ac0 con 0x7fcac40fff30 2024-03-21T10:37:27.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.374+0000 7fcacb7fd740 1 Processor -- start 2024-03-21T10:37:27.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.374+0000 7fcacb7fd740 1 -- start start 2024-03-21T10:37:27.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.374+0000 7fcacb7fd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c030ba0 0x55f64c030fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.374+0000 7fcacb7fd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f64c031590 0x55f64c036090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.374+0000 7fcacb7fd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f64c036700 0x55f64c038b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.374+0000 7fcacb7fd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64bf95c90 con 0x55f64c036700 2024-03-21T10:37:27.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.374+0000 7fcacb7fd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64bf3e980 con 0x55f64c030ba0 2024-03-21T10:37:27.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.374+0000 7fcacb7fd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64bf91d30 con 0x55f64c031590 2024-03-21T10:37:27.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.374+0000 7fcac1e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f64c031590 0x55f64c036090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.374+0000 7fcac2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c030ba0 0x55f64c030fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.375+0000 7fcac2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c030ba0 0x55f64c030fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:59354/0 (socket says 172.21.15.31:59354) 2024-03-21T10:37:27.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.375+0000 7fcac2800640 1 -- 172.21.15.31:0/37533575 learned_addr learned my addr 172.21.15.31:0/37533575 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:27.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.375+0000 7fcac1400640 1 --2- 172.21.15.31:0/37533575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f64c036700 0x55f64c038b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.375+0000 7fcac2800640 1 -- 172.21.15.31:0/37533575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f64c031590 msgr2=0x55f64c036090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:37:27.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.375+0000 7fcac2800640 1 --2- 172.21.15.31:0/37533575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f64c031590 0x55f64c036090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.375+0000 7fcac2800640 1 -- 172.21.15.31:0/37533575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f64c036700 msgr2=0x55f64c038b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:37:27.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.375+0000 7fcac2800640 1 --2- 172.21.15.31:0/37533575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f64c036700 0x55f64c038b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.375+0000 7fcac2800640 1 -- 172.21.15.31:0/37533575 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f64bf56510 con 0x55f64c030ba0 2024-03-21T10:37:27.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.375+0000 7fcac2800640 1 --2- 172.21.15.31:0/37533575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c030ba0 0x55f64c030fc0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fcabc010750 tx=0x7fcabc010c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9be6f01c3da26933 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:27.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.376+0000 7fcaaf000640 1 -- 172.21.15.31:0/37533575 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcabc00ead0 con 0x55f64c030ba0 2024-03-21T10:37:27.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.376+0000 7fcaaf000640 1 -- 172.21.15.31:0/37533575 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcabc010e60 con 0x55f64c030ba0 2024-03-21T10:37:27.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.376+0000 7fcaaf000640 1 -- 172.21.15.31:0/37533575 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcabc011640 con 0x55f64c030ba0 2024-03-21T10:37:27.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.376+0000 7fcabaa00640 1 --2- 172.21.15.31:0/2237162928 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcaa80a2fa0 0x7fcaa80a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.376+0000 7fcabaa00640 1 -- 172.21.15.31:0/2237162928 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fcab40cfa10 con 0x7fcac40fff30 2024-03-21T10:37:27.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.376+0000 7fcac1e00640 1 --2- 172.21.15.31:0/2237162928 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcaa80a2fa0 0x7fcaa80a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.376+0000 7fcacb7fd740 1 -- 172.21.15.31:0/37533575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c030ba0 msgr2=0x55f64c030fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.376+0000 7fcacb7fd740 1 --2- 172.21.15.31:0/37533575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c030ba0 0x55f64c030fc0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fcabc010750 tx=0x7fcabc010c10 comp rx=0 tx=0).stop 2024-03-21T10:37:27.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.377+0000 7fcacb7fd740 1 -- 172.21.15.31:0/37533575 shutdown_connections 2024-03-21T10:37:27.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.377+0000 7fcacb7fd740 1 --2- 172.21.15.31:0/37533575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f64c036700 0x55f64c038b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.377+0000 7fcacb7fd740 1 --2- 172.21.15.31:0/37533575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f64c031590 0x55f64c036090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.377+0000 7fcacb7fd740 1 --2- 172.21.15.31:0/37533575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c030ba0 0x55f64c030fc0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.377+0000 7fcacb7fd740 1 -- 172.21.15.31:0/37533575 >> 172.21.15.31:0/37533575 conn(0x55f64c02c110 msgr2=0x55f64c02c510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:27.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.377+0000 7fcacb7fd740 1 -- 172.21.15.31:0/37533575 shutdown_connections 2024-03-21T10:37:27.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.377+0000 7fcacb7fd740 1 -- 172.21.15.31:0/37533575 wait complete. 2024-03-21T10:37:27.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.377+0000 7fcac1e00640 1 --2- 172.21.15.31:0/2237162928 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcaa80a2fa0 0x7fcaa80a5460 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcab409bdc0 tx=0x7fcab40d54b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:27.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.377+0000 7fcacb7fd740 1 -- start start 2024-03-21T10:37:27.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.378+0000 7fcacb7fd740 1 Processor -- start 2024-03-21T10:37:27.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.378+0000 7fcacb7fd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c030ba0 0x55f64c03b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.378+0000 7fcacb7fd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f64c031590 0x55f64c03b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.378+0000 7fcacb7fd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f64c036700 0x55f64c042640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.378+0000 7fcacb7fd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64c0396b0 con 0x55f64c036700 2024-03-21T10:37:27.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.378+0000 7fcacb7fd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64bf51dd0 con 0x55f64c030ba0 2024-03-21T10:37:27.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.378+0000 7fcacb7fd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64c039830 con 0x55f64c031590 2024-03-21T10:37:27.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.378+0000 7fcacb7fd740 10 client.0 _ll_register_callbacks cb 0x55f64bca9ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:37:27.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.379+0000 7fcac1400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f64c031590 0x55f64c03b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.379+0000 7fcac1400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f64c031590 0x55f64c03b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:55194/0 (socket says 172.21.15.31:55194) 2024-03-21T10:37:27.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.379+0000 7fcac1400640 1 -- 172.21.15.31:0/1019954292 learned_addr learned my addr 172.21.15.31:0/1019954292 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:27.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.379+0000 7fcac2800640 1 --2- 172.21.15.31:0/1019954292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c030ba0 0x55f64c03b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.379+0000 7fcac2800640 1 --2- 172.21.15.31:0/1019954292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f64c036700 0x55f64c042640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.379+0000 7fcac1400640 1 -- 172.21.15.31:0/1019954292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c030ba0 msgr2=0x55f64c03b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:37:27.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.379+0000 7fcac1400640 1 --2- 172.21.15.31:0/1019954292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c030ba0 0x55f64c03b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.379+0000 7fcac1400640 1 -- 172.21.15.31:0/1019954292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f64c036700 msgr2=0x55f64c042640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:37:27.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.379+0000 7fcac1400640 1 --2- 172.21.15.31:0/1019954292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f64c036700 0x55f64c042640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.379+0000 7fcac1400640 1 -- 172.21.15.31:0/1019954292 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f64bcaa2d0 con 0x55f64c031590 2024-03-21T10:37:27.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.379+0000 7fcac2800640 1 --2- 172.21.15.31:0/1019954292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c030ba0 0x55f64c03b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:37:27.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.380+0000 7fcaa3400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:37:27.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.379+0000 7fcac1400640 1 --2- 172.21.15.31:0/1019954292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f64c031590 0x55f64c03b600 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcab0000c00 tx=0x7fcab0004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=283e24e5eed90910 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:27.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.380+0000 7fcaa3400640 1 -- 172.21.15.31:0/1019954292 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab0005d50 con 0x55f64c031590 2024-03-21T10:37:27.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.380+0000 7fcaa3400640 1 -- 172.21.15.31:0/1019954292 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fcab00046c0 con 0x55f64c031590 2024-03-21T10:37:27.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.380+0000 7fcacb7fd740 1 -- 172.21.15.31:0/1019954292 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcabc010e60 con 0x55f64c031590 2024-03-21T10:37:27.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.380+0000 7fcaa3400640 1 -- 172.21.15.31:0/1019954292 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcab0053050 con 0x55f64c031590 2024-03-21T10:37:27.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.380+0000 7fcacb7fd740 10 client.25174 fetch_fsmap learned FSMap version 23 2024-03-21T10:37:27.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.380+0000 7fcacb7fd740 1 -- 172.21.15.31:0/1019954292 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f64bd746b0 con 0x55f64c031590 2024-03-21T10:37:27.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.381+0000 7fcaa3400640 1 -- 172.21.15.31:0/1019954292 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcabc010e60 con 0x55f64c031590 2024-03-21T10:37:27.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.381+0000 7fcaa3400640 1 -- 172.21.15.31:0/1019954292 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 23) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fcab0003680 con 0x55f64c031590 2024-03-21T10:37:27.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.381+0000 7fcacb7fd740 10 client.25174 fetch_fsmap finished waiting for FSMap version 23 2024-03-21T10:37:27.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.381+0000 7fcacb7fd740 10 client.25174 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-21T10:37:27.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.381+0000 7fcacb7fd740 20 client.25174 populate_metadata read hostname 'smithi031' 2024-03-21T10:37:27.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.381+0000 7fcacb7fd740 1 --2- 172.21.15.31:0/1019954292 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55f64c0478c0 0x55f64c046980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.381+0000 7fcacb7fd740 4 client.25174 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:37:27.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.381+0000 7fcacb7fd740 1 -- 172.21.15.31:0/1019954292 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f64bf37520 con 0x55f64c0478c0 2024-03-21T10:37:27.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.381+0000 7fcac2800640 1 --2- 172.21.15.31:0/1019954292 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55f64c0478c0 0x55f64c046980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.382+0000 7fcac2800640 1 --2- 172.21.15.31:0/1019954292 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55f64c0478c0 0x55f64c046980 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=acc3f5e5b77c433e server_cookie=87d87dd28cf96d3c in_seq=0 out_seq=0 2024-03-21T10:37:27.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.382+0000 7fcaa3400640 10 client.25174 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:37:27.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.383+0000 7fcaa3400640 1 -- 172.21.15.31:0/1019954292 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f64bf37520 con 0x55f64c0478c0 2024-03-21T10:37:27.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.383+0000 7fcaa3400640 10 client.25174 handle_command_reply: tid=0 2024-03-21T10:37:27.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.383+0000 7fcacb7fd740 1 client.25174 shutdown 2024-03-21T10:37:27.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.383+0000 7fcacb7fd740 1 -- 172.21.15.31:0/1019954292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f64c031590 msgr2=0x55f64c03b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.383+0000 7fcacb7fd740 1 --2- 172.21.15.31:0/1019954292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f64c031590 0x55f64c03b600 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcab0000c00 tx=0x7fcab0004270 comp rx=0 tx=0).stop 2024-03-21T10:37:27.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.384+0000 7fcacb7fd740 1 -- 172.21.15.31:0/1019954292 shutdown_connections 2024-03-21T10:37:27.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.384+0000 7fcacb7fd740 1 --2- 172.21.15.31:0/1019954292 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55f64c0478c0 0x55f64c046980 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.384+0000 7fcacb7fd740 1 --2- 172.21.15.31:0/1019954292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f64c036700 0x55f64c042640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.384+0000 7fcacb7fd740 1 --2- 172.21.15.31:0/1019954292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f64c031590 0x55f64c03b600 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.384+0000 7fcacb7fd740 1 --2- 172.21.15.31:0/1019954292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c030ba0 0x55f64c03b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.384+0000 7fcaa3400640 0 client.25174 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:37:27.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.384+0000 7fcacb7fd740 1 -- 172.21.15.31:0/1019954292 >> 172.21.15.31:0/1019954292 conn(0x55f64c02c110 msgr2=0x55f64c02b310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:27.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.384+0000 7fcacb7fd740 1 -- 172.21.15.31:0/1019954292 shutdown_connections 2024-03-21T10:37:27.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.384+0000 7fcacb7fd740 1 -- 172.21.15.31:0/1019954292 wait complete. 2024-03-21T10:37:27.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.384+0000 7fcacb7fd740 20 client.25174 trim_cache size 0 max 16384 2024-03-21T10:37:27.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.421+0000 7fcacb7fd740 1 Processor -- start 2024-03-21T10:37:27.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.422+0000 7fcacb7fd740 1 -- start start 2024-03-21T10:37:27.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.422+0000 7fcacb7fd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f64c030ba0 0x55f64c026e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.422+0000 7fcacb7fd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c031590 0x55f64c02c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.422+0000 7fcacb7fd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f64c036700 0x55f64c02aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.422+0000 7fcacb7fd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64c0470e0 con 0x55f64c030ba0 2024-03-21T10:37:27.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.422+0000 7fcacb7fd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64c047740 con 0x55f64c031590 2024-03-21T10:37:27.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.422+0000 7fcacb7fd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64c047da0 con 0x55f64c036700 2024-03-21T10:37:27.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.422+0000 7fcac1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c031590 0x55f64c02c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.422+0000 7fcac2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f64c030ba0 0x55f64c026e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.422+0000 7fcac1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c031590 0x55f64c02c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:59376/0 (socket says 172.21.15.31:59376) 2024-03-21T10:37:27.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.422+0000 7fcac1400640 1 -- 172.21.15.31:0/1830498937 learned_addr learned my addr 172.21.15.31:0/1830498937 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:27.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.423+0000 7fcac2800640 1 --2- 172.21.15.31:0/1830498937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f64c036700 0x55f64c02aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.423+0000 7fcac1400640 1 -- 172.21.15.31:0/1830498937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f64c036700 msgr2=0x55f64c02aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:37:27.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.423+0000 7fcac1400640 1 --2- 172.21.15.31:0/1830498937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f64c036700 0x55f64c02aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.423+0000 7fcac1400640 1 -- 172.21.15.31:0/1830498937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f64c030ba0 msgr2=0x55f64c026e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:37:27.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.423+0000 7fcac1400640 1 --2- 172.21.15.31:0/1830498937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f64c030ba0 0x55f64c026e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.423+0000 7fcac1400640 1 -- 172.21.15.31:0/1830498937 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f64c03c0b0 con 0x55f64c031590 2024-03-21T10:37:27.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.423+0000 7fcac2800640 1 --2- 172.21.15.31:0/1830498937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f64c036700 0x55f64c02aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:37:27.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.423+0000 7fcac1400640 1 --2- 172.21.15.31:0/1830498937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c031590 0x55f64c02c4f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcab0005ab0 tx=0x7fcab0005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86334a3889d0b308 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:27.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.424+0000 7fcaae600640 1 -- 172.21.15.31:0/1830498937 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab0005d50 con 0x55f64c031590 2024-03-21T10:37:27.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.424+0000 7fcaae600640 1 -- 172.21.15.31:0/1830498937 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcab0003c40 con 0x55f64c031590 2024-03-21T10:37:27.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.424+0000 7fcaae600640 1 -- 172.21.15.31:0/1830498937 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab00055f0 con 0x55f64c031590 2024-03-21T10:37:27.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.425+0000 7fcacb7fd740 1 -- 172.21.15.31:0/1830498937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c031590 msgr2=0x55f64c02c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.425+0000 7fcacb7fd740 1 --2- 172.21.15.31:0/1830498937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c031590 0x55f64c02c4f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcab0005ab0 tx=0x7fcab0005b50 comp rx=0 tx=0).stop 2024-03-21T10:37:27.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.425+0000 7fcacb7fd740 1 -- 172.21.15.31:0/1830498937 shutdown_connections 2024-03-21T10:37:27.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.425+0000 7fcacb7fd740 1 --2- 172.21.15.31:0/1830498937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f64c036700 0x55f64c02aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.425+0000 7fcacb7fd740 1 --2- 172.21.15.31:0/1830498937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c031590 0x55f64c02c4f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.425+0000 7fcacb7fd740 1 --2- 172.21.15.31:0/1830498937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f64c030ba0 0x55f64c026e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.425+0000 7fcacb7fd740 1 -- 172.21.15.31:0/1830498937 >> 172.21.15.31:0/1830498937 conn(0x55f64c02c110 msgr2=0x55f64c046980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:27.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.425+0000 7fcacb7fd740 1 -- 172.21.15.31:0/1830498937 shutdown_connections 2024-03-21T10:37:27.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.425+0000 7fcacb7fd740 1 -- 172.21.15.31:0/1830498937 wait complete. 2024-03-21T10:37:27.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.426+0000 7fcacb7fd740 1 -- start start 2024-03-21T10:37:27.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.426+0000 7fcacb7fd740 1 Processor -- start 2024-03-21T10:37:27.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.426+0000 7fcacb7fd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f64c030ba0 0x55f64c0a1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.426+0000 7fcacb7fd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c031590 0x55f64c0a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.426+0000 7fcacb7fd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f64c036700 0x55f64c0a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.426+0000 7fcacb7fd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64c048a60 con 0x55f64c036700 2024-03-21T10:37:27.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.426+0000 7fcacb7fd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64c048400 con 0x55f64c031590 2024-03-21T10:37:27.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.427+0000 7fcacb7fd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f64c0490c0 con 0x55f64c030ba0 2024-03-21T10:37:27.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.427+0000 7fcacb7fd740 10 client.0 _ll_register_callbacks cb 0x7fcab0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:37:27.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.427+0000 7fcac1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c031590 0x55f64c0a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.427+0000 7fcac1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c031590 0x55f64c0a05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:59380/0 (socket says 172.21.15.31:59380) 2024-03-21T10:37:27.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.427+0000 7fcac2800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f64c030ba0 0x55f64c0a1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.427+0000 7fcac1400640 1 -- 172.21.15.31:0/3871343644 learned_addr learned my addr 172.21.15.31:0/3871343644 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:27.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.427+0000 7fcac2800640 1 --2- 172.21.15.31:0/3871343644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f64c036700 0x55f64c0a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.427+0000 7fcac1400640 1 -- 172.21.15.31:0/3871343644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f64c030ba0 msgr2=0x55f64c0a1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:37:27.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.427+0000 7fcac1400640 1 --2- 172.21.15.31:0/3871343644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f64c030ba0 0x55f64c0a1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.427+0000 7fcac1400640 1 -- 172.21.15.31:0/3871343644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f64c036700 msgr2=0x55f64c0a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:37:27.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.427+0000 7fcac1400640 1 --2- 172.21.15.31:0/3871343644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f64c036700 0x55f64c0a0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.427+0000 7fcac1400640 1 -- 172.21.15.31:0/3871343644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f64c0a1060 con 0x55f64c031590 2024-03-21T10:37:27.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.428+0000 7fcac1400640 1 --2- 172.21.15.31:0/3871343644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c031590 0x55f64c0a05e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcab0003b70 tx=0x7fcab0004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f465ba28b3c4e72 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:27.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.428+0000 7fcaa3400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:37:27.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.428+0000 7fcaa3400640 1 -- 172.21.15.31:0/3871343644 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab0004250 con 0x55f64c031590 2024-03-21T10:37:27.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.428+0000 7fcaa3400640 1 -- 172.21.15.31:0/3871343644 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fcab0004eb0 con 0x55f64c031590 2024-03-21T10:37:27.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.428+0000 7fcacb7fd740 1 -- 172.21.15.31:0/3871343644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcab0003c40 con 0x55f64c031590 2024-03-21T10:37:27.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.428+0000 7fcaa3400640 1 -- 172.21.15.31:0/3871343644 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcab00050b0 con 0x55f64c031590 2024-03-21T10:37:27.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.428+0000 7fcacb7fd740 10 client.25190 fetch_fsmap learned FSMap version 23 2024-03-21T10:37:27.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.428+0000 7fcacb7fd740 1 -- 172.21.15.31:0/3871343644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f64c041200 con 0x55f64c031590 2024-03-21T10:37:27.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.428+0000 7fcaa3400640 1 -- 172.21.15.31:0/3871343644 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcab0003c40 con 0x55f64c031590 2024-03-21T10:37:27.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.428+0000 7fcaa3400640 1 -- 172.21.15.31:0/3871343644 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 23) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fcab004aa10 con 0x55f64c031590 2024-03-21T10:37:27.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.429+0000 7fcacb7fd740 10 client.25190 fetch_fsmap finished waiting for FSMap version 23 2024-03-21T10:37:27.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.429+0000 7fcacb7fd740 10 client.25190 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-21T10:37:27.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.429+0000 7fcacb7fd740 20 client.25190 populate_metadata read hostname 'smithi031' 2024-03-21T10:37:27.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.429+0000 7fcacb7fd740 1 --2- 172.21.15.31:0/3871343644 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55f64c0b8a60 0x55f64c041d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.429+0000 7fcacb7fd740 4 client.25190 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-21T10:37:27.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.429+0000 7fcacb7fd740 1 -- 172.21.15.31:0/3871343644 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f64bfa31d0 con 0x55f64c0b8a60 2024-03-21T10:37:27.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.429+0000 7fcac2800640 1 --2- 172.21.15.31:0/3871343644 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55f64c0b8a60 0x55f64c041d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.430+0000 7fcac2800640 1 --2- 172.21.15.31:0/3871343644 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55f64c0b8a60 0x55f64c041d90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=348a1a97db883121 server_cookie=75adbfb8ed56d76e in_seq=0 out_seq=0 2024-03-21T10:37:27.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.430+0000 7fcaa3400640 10 client.25190 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:37:27.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.430+0000 7fcaa3400640 1 -- 172.21.15.31:0/3871343644 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x55f64bfa31d0 con 0x55f64c0b8a60 2024-03-21T10:37:27.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.430+0000 7fcaa3400640 10 client.25190 handle_command_reply: tid=0 2024-03-21T10:37:27.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.430+0000 7fcacb7fd740 1 client.25190 shutdown 2024-03-21T10:37:27.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.430+0000 7fcacb7fd740 1 -- 172.21.15.31:0/3871343644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c031590 msgr2=0x55f64c0a05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.430+0000 7fcacb7fd740 1 --2- 172.21.15.31:0/3871343644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c031590 0x55f64c0a05e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcab0003b70 tx=0x7fcab0004ad0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.430+0000 7fcacb7fd740 1 -- 172.21.15.31:0/3871343644 shutdown_connections 2024-03-21T10:37:27.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.430+0000 7fcacb7fd740 1 --2- 172.21.15.31:0/3871343644 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55f64c0b8a60 0x55f64c041d90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.430+0000 7fcacb7fd740 1 --2- 172.21.15.31:0/3871343644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f64c036700 0x55f64c0a0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.430+0000 7fcacb7fd740 1 --2- 172.21.15.31:0/3871343644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f64c031590 0x55f64c0a05e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.430+0000 7fcaa3400640 0 client.25190 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:37:27.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.430+0000 7fcacb7fd740 1 --2- 172.21.15.31:0/3871343644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f64c030ba0 0x55f64c0a1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.430+0000 7fcacb7fd740 1 -- 172.21.15.31:0/3871343644 >> 172.21.15.31:0/3871343644 conn(0x55f64c02c110 msgr2=0x55f64c026e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:27.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.430+0000 7fcacb7fd740 1 -- 172.21.15.31:0/3871343644 shutdown_connections 2024-03-21T10:37:27.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.431+0000 7fcacb7fd740 1 -- 172.21.15.31:0/3871343644 wait complete. 2024-03-21T10:37:27.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.431+0000 7fcacb7fd740 20 client.25190 trim_cache size 0 max 16384 2024-03-21T10:37:27.433 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:37:27.433 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "3847408c-e76d-11ee-95cd-87774f69a715", 2024-03-21T10:37:27.433 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 1, 2024-03-21T10:37:27.433 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24584, 2024-03-21T10:37:27.434 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-03-21T10:37:27.434 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-03-21T10:37:27.434 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-03-21T10:37:27.434 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 108.118845346, 2024-03-21T10:37:27.434 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 23, 2024-03-21T10:37:27.434 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 93, 2024-03-21T10:37:27.434 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 89, 2024-03-21T10:37:27.434 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 149.186541613 2024-03-21T10:37:27.434 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:37:27.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.431+0000 7fcac8a00640 1 -- 172.21.15.31:0/2237162928 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcaa80a2fa0 msgr2=0x7fcaa80a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.431+0000 7fcac8a00640 1 --2- 172.21.15.31:0/2237162928 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcaa80a2fa0 0x7fcaa80a5460 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcab409bdc0 tx=0x7fcab40d54b0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.431+0000 7fcac8a00640 1 -- 172.21.15.31:0/2237162928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcac40fff30 msgr2=0x7fcac419dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.431+0000 7fcac8a00640 1 --2- 172.21.15.31:0/2237162928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcac40fff30 0x7fcac419dfa0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcab400e9f0 tx=0x7fcab400eeb0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.431+0000 7fcac8a00640 1 -- 172.21.15.31:0/2237162928 shutdown_connections 2024-03-21T10:37:27.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.431+0000 7fcac8a00640 1 --2- 172.21.15.31:0/2237162928 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcaa80a2fa0 0x7fcaa80a5460 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.431+0000 7fcac8a00640 1 --2- 172.21.15.31:0/2237162928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcac4105250 0x7fcac41a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.431+0000 7fcac8a00640 1 --2- 172.21.15.31:0/2237162928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcac41028a0 0x7fcac419e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.431+0000 7fcac8a00640 1 --2- 172.21.15.31:0/2237162928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcac40fff30 0x7fcac419dfa0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.431+0000 7fcac8a00640 1 -- 172.21.15.31:0/2237162928 >> 172.21.15.31:0/2237162928 conn(0x7fcac40fbb30 msgr2=0x7fcac4100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:27.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.432+0000 7fcac8a00640 1 -- 172.21.15.31:0/2237162928 shutdown_connections 2024-03-21T10:37:27.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.432+0000 7fcac8a00640 1 -- 172.21.15.31:0/2237162928 wait complete. 2024-03-21T10:37:27.448 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-21T10:37:27.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.646+0000 7fe8ddc00640 1 -- 172.21.15.31:0/3617259039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8d8101f70 msgr2=0x7fe8d8104360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.646+0000 7fe8ddc00640 1 --2- 172.21.15.31:0/3617259039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8d8101f70 0x7fe8d8104360 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8009900 tx=0x7fe8c802bc50 comp rx=0 tx=0).stop 2024-03-21T10:37:27.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.646+0000 7fe8ddc00640 1 -- 172.21.15.31:0/3617259039 shutdown_connections 2024-03-21T10:37:27.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.646+0000 7fe8ddc00640 1 --2- 172.21.15.31:0/3617259039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8d81071d0 0x7fe8d8109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.646+0000 7fe8ddc00640 1 --2- 172.21.15.31:0/3617259039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8d81048a0 0x7fe8d8106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.646+0000 7fe8ddc00640 1 --2- 172.21.15.31:0/3617259039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8d8101f70 0x7fe8d8104360 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.646+0000 7fe8ddc00640 1 -- 172.21.15.31:0/3617259039 >> 172.21.15.31:0/3617259039 conn(0x7fe8d80fb980 msgr2=0x7fe8d80fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:27.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.646+0000 7fe8ddc00640 1 -- 172.21.15.31:0/3617259039 shutdown_connections 2024-03-21T10:37:27.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.646+0000 7fe8ddc00640 1 -- 172.21.15.31:0/3617259039 wait complete. 2024-03-21T10:37:27.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.647+0000 7fe8ddc00640 1 Processor -- start 2024-03-21T10:37:27.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.647+0000 7fe8ddc00640 1 -- start start 2024-03-21T10:37:27.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.648+0000 7fe8ddc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8d8101f70 0x7fe8d8199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.648+0000 7fe8ddc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8d81048a0 0x7fe8d8199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.648+0000 7fe8ddc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8d81071d0 0x7fe8d81a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.648+0000 7fe8ddc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d810d1d0 con 0x7fe8d81048a0 2024-03-21T10:37:27.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.648+0000 7fe8ddc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d810d050 con 0x7fe8d81071d0 2024-03-21T10:37:27.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.648+0000 7fe8ddc00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d810d350 con 0x7fe8d8101f70 2024-03-21T10:37:27.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.648+0000 7fe8d7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8d8101f70 0x7fe8d8199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.648+0000 7fe8dd200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8d81071d0 0x7fe8d81a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.648+0000 7fe8d7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8d8101f70 0x7fe8d8199a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:55226/0 (socket says 172.21.15.31:55226) 2024-03-21T10:37:27.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.648+0000 7fe8d7e00640 1 -- 172.21.15.31:0/2871731126 learned_addr learned my addr 172.21.15.31:0/2871731126 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:27.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.648+0000 7fe8d7400640 1 --2- 172.21.15.31:0/2871731126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8d81048a0 0x7fe8d8199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.649+0000 7fe8d7e00640 1 -- 172.21.15.31:0/2871731126 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8d81071d0 msgr2=0x7fe8d81a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.649+0000 7fe8d7e00640 1 --2- 172.21.15.31:0/2871731126 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8d81071d0 0x7fe8d81a0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.649+0000 7fe8d7e00640 1 -- 172.21.15.31:0/2871731126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8d81048a0 msgr2=0x7fe8d8199f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.649+0000 7fe8d7e00640 1 --2- 172.21.15.31:0/2871731126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8d81048a0 0x7fe8d8199f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.649+0000 7fe8d7e00640 1 -- 172.21.15.31:0/2871731126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d81a1660 con 0x7fe8d8101f70 2024-03-21T10:37:27.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.649+0000 7fe8d7400640 1 --2- 172.21.15.31:0/2871731126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8d81048a0 0x7fe8d8199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:37:27.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.649+0000 7fe8d7e00640 1 --2- 172.21.15.31:0/2871731126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8d8101f70 0x7fe8d8199a40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8005ab0 tx=0x7fe8c802c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:27.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.649+0000 7fe8d4c00640 1 -- 172.21.15.31:0/2871731126 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c8041070 con 0x7fe8d8101f70 2024-03-21T10:37:27.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.649+0000 7fe8d4c00640 1 -- 172.21.15.31:0/2871731126 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe8c8004450 con 0x7fe8d8101f70 2024-03-21T10:37:27.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.649+0000 7fe8d4c00640 1 -- 172.21.15.31:0/2871731126 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c8034440 con 0x7fe8d8101f70 2024-03-21T10:37:27.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.649+0000 7fe8ddc00640 1 -- 172.21.15.31:0/2871731126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8d81a18f0 con 0x7fe8d8101f70 2024-03-21T10:37:27.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.649+0000 7fe8ddc00640 1 -- 172.21.15.31:0/2871731126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8d81a1e40 con 0x7fe8d8101f70 2024-03-21T10:37:27.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.652+0000 7fe8d4c00640 1 -- 172.21.15.31:0/2871731126 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fe8c8044070 con 0x7fe8d8101f70 2024-03-21T10:37:27.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.653+0000 7fe8e2004740 1 Processor -- start 2024-03-21T10:37:27.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.653+0000 7fe8e2004740 1 -- start start 2024-03-21T10:37:27.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.653+0000 7fe8e2004740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5e5ff4d80 0x55c5e5ff51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.653+0000 7fe8e2004740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5e5ff5770 0x55c5e5ffa270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.653+0000 7fe8e2004740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5e5ffa8e0 0x55c5e5ffcd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.653+0000 7fe8e2004740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e5f59c90 con 0x55c5e5ffa8e0 2024-03-21T10:37:27.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.653+0000 7fe8e2004740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e5f02980 con 0x55c5e5ff4d80 2024-03-21T10:37:27.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.653+0000 7fe8e2004740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e5f55d30 con 0x55c5e5ff5770 2024-03-21T10:37:27.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.654+0000 7fe8dd200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5e5ff4d80 0x55c5e5ff51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.654+0000 7fe8d7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5e5ff5770 0x55c5e5ffa270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.654+0000 7fe8dd200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5e5ff4d80 0x55c5e5ff51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:59410/0 (socket says 172.21.15.31:59410) 2024-03-21T10:37:27.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.654+0000 7fe8dd200640 1 -- 172.21.15.31:0/1778712584 learned_addr learned my addr 172.21.15.31:0/1778712584 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:27.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.654+0000 7fe8d7400640 1 --2- 172.21.15.31:0/1778712584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5e5ffa8e0 0x55c5e5ffcd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.654+0000 7fe8dd200640 1 -- 172.21.15.31:0/1778712584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5e5ff5770 msgr2=0x55c5e5ffa270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:37:27.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.654+0000 7fe8d4c00640 1 --2- 172.21.15.31:0/2871731126 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe8bc0a2f20 0x7fe8bc0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.654+0000 7fe8dd200640 1 --2- 172.21.15.31:0/1778712584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5e5ff5770 0x55c5e5ffa270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.654+0000 7fe8dd200640 1 -- 172.21.15.31:0/1778712584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5e5ffa8e0 msgr2=0x55c5e5ffcd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:37:27.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.654+0000 7fe8dd200640 1 --2- 172.21.15.31:0/1778712584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5e5ffa8e0 0x55c5e5ffcd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.654+0000 7fe8dd200640 1 -- 172.21.15.31:0/1778712584 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5e5f1a510 con 0x55c5e5ff4d80 2024-03-21T10:37:27.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.654+0000 7fe8d7e00640 1 --2- 172.21.15.31:0/1778712584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5e5ff5770 0x55c5e5ffa270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:37:27.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.654+0000 7fe8d4c00640 1 -- 172.21.15.31:0/2871731126 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fe8c80f13d0 con 0x7fe8d8101f70 2024-03-21T10:37:27.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.655+0000 7fe8dd200640 1 --2- 172.21.15.31:0/2871731126 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe8bc0a2f20 0x7fe8bc0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.655+0000 7fe8dd200640 1 --2- 172.21.15.31:0/1778712584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5e5ff4d80 0x55c5e5ff51a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe8d0009fd0 tx=0x7fe8d00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ec3c2a3f1328d64 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:27.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.655+0000 7fe8cd600640 1 -- 172.21.15.31:0/1778712584 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0004250 con 0x55c5e5ff4d80 2024-03-21T10:37:27.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.655+0000 7fe8cd600640 1 -- 172.21.15.31:0/1778712584 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe8d0003db0 con 0x55c5e5ff4d80 2024-03-21T10:37:27.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.655+0000 7fe8cd600640 1 -- 172.21.15.31:0/1778712584 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d001bc20 con 0x55c5e5ff4d80 2024-03-21T10:37:27.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.656+0000 7fe8e2004740 1 -- 172.21.15.31:0/1778712584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5e5ff4d80 msgr2=0x55c5e5ff51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.656+0000 7fe8dd200640 1 --2- 172.21.15.31:0/2871731126 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe8bc0a2f20 0x7fe8bc0a53e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe8d0020000 tx=0x7fe8d0021000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:27.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.656+0000 7fe8e2004740 1 --2- 172.21.15.31:0/1778712584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5e5ff4d80 0x55c5e5ff51a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe8d0009fd0 tx=0x7fe8d00118d0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.656+0000 7fe8e2004740 1 -- 172.21.15.31:0/1778712584 shutdown_connections 2024-03-21T10:37:27.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.656+0000 7fe8e2004740 1 --2- 172.21.15.31:0/1778712584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5e5ffa8e0 0x55c5e5ffcd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.656+0000 7fe8e2004740 1 --2- 172.21.15.31:0/1778712584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5e5ff5770 0x55c5e5ffa270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.656+0000 7fe8e2004740 1 --2- 172.21.15.31:0/1778712584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5e5ff4d80 0x55c5e5ff51a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.656+0000 7fe8e2004740 1 -- 172.21.15.31:0/1778712584 >> 172.21.15.31:0/1778712584 conn(0x55c5e5ff02f0 msgr2=0x55c5e5ff06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:27.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.656+0000 7fe8e2004740 1 -- 172.21.15.31:0/1778712584 shutdown_connections 2024-03-21T10:37:27.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.656+0000 7fe8e2004740 1 -- 172.21.15.31:0/1778712584 wait complete. 2024-03-21T10:37:27.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.657+0000 7fe8e2004740 1 -- start start 2024-03-21T10:37:27.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.657+0000 7fe8e2004740 1 Processor -- start 2024-03-21T10:37:27.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.657+0000 7fe8e2004740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5e5ff4d80 0x55c5e5fff250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.657+0000 7fe8e2004740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5e5ff5770 0x55c5e5fff790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.657+0000 7fe8e2004740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5e5ffa8e0 0x55c5e60067d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.657+0000 7fe8e2004740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e5ffd890 con 0x55c5e5ff5770 2024-03-21T10:37:27.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.657+0000 7fe8e2004740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e5f15dd0 con 0x55c5e5ff4d80 2024-03-21T10:37:27.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.657+0000 7fe8e2004740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e5ffda10 con 0x55c5e5ffa8e0 2024-03-21T10:37:27.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.658+0000 7fe8e2004740 10 client.0 _ll_register_callbacks cb 0x55c5e5c6dac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:37:27.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.658+0000 7fe8d7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5e5ff4d80 0x55c5e5fff250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.658+0000 7fe8dd200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5e5ffa8e0 0x55c5e60067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.658+0000 7fe8d7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5e5ff4d80 0x55c5e5fff250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:59418/0 (socket says 172.21.15.31:59418) 2024-03-21T10:37:27.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.658+0000 7fe8d7e00640 1 -- 172.21.15.31:0/3895581641 learned_addr learned my addr 172.21.15.31:0/3895581641 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:27.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.658+0000 7fe8d7400640 1 --2- 172.21.15.31:0/3895581641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5e5ff5770 0x55c5e5fff790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.658+0000 7fe8dd200640 1 -- 172.21.15.31:0/3895581641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5e5ff4d80 msgr2=0x55c5e5fff250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:37:27.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.658+0000 7fe8dd200640 1 --2- 172.21.15.31:0/3895581641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5e5ff4d80 0x55c5e5fff250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.658+0000 7fe8dd200640 1 -- 172.21.15.31:0/3895581641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5e5ff5770 msgr2=0x55c5e5fff790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:37:27.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.658+0000 7fe8dd200640 1 --2- 172.21.15.31:0/3895581641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5e5ff5770 0x55c5e5fff790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.658+0000 7fe8dd200640 1 -- 172.21.15.31:0/3895581641 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c5e5c6e2d0 con 0x55c5e5ffa8e0 2024-03-21T10:37:27.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.658+0000 7fe8d7e00640 1 --2- 172.21.15.31:0/3895581641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5e5ff4d80 0x55c5e5fff250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:37:27.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.659+0000 7fe8dd200640 1 --2- 172.21.15.31:0/3895581641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5e5ffa8e0 0x55c5e60067d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe8d0006010 tx=0x7fe8d001b540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd381d93a95e38f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:27.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.659+0000 7fe8c2000640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:37:27.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.659+0000 7fe8c2000640 1 -- 172.21.15.31:0/3895581641 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0034030 con 0x55c5e5ffa8e0 2024-03-21T10:37:27.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.659+0000 7fe8c2000640 1 -- 172.21.15.31:0/3895581641 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fe8d001b9a0 con 0x55c5e5ffa8e0 2024-03-21T10:37:27.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.659+0000 7fe8e2004740 1 -- 172.21.15.31:0/3895581641 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe8d0003db0 con 0x55c5e5ffa8e0 2024-03-21T10:37:27.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.659+0000 7fe8c2000640 1 -- 172.21.15.31:0/3895581641 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe8d003e050 con 0x55c5e5ffa8e0 2024-03-21T10:37:27.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.659+0000 7fe8e2004740 10 client.25192 fetch_fsmap learned FSMap version 23 2024-03-21T10:37:27.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.659+0000 7fe8e2004740 1 -- 172.21.15.31:0/3895581641 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c5e5d386b0 con 0x55c5e5ffa8e0 2024-03-21T10:37:27.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.659+0000 7fe8c2000640 1 -- 172.21.15.31:0/3895581641 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe8d0003db0 con 0x55c5e5ffa8e0 2024-03-21T10:37:27.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.660+0000 7fe8c2000640 1 -- 172.21.15.31:0/3895581641 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 23) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fe8d002f950 con 0x55c5e5ffa8e0 2024-03-21T10:37:27.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.660+0000 7fe8e2004740 10 client.25192 fetch_fsmap finished waiting for FSMap version 23 2024-03-21T10:37:27.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.660+0000 7fe8e2004740 10 client.25192 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-21T10:37:27.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.660+0000 7fe8e2004740 20 client.25192 populate_metadata read hostname 'smithi031' 2024-03-21T10:37:27.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.660+0000 7fe8e2004740 1 --2- 172.21.15.31:0/3895581641 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c5e600b790 0x55c5e600a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.660+0000 7fe8e2004740 4 client.25192 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:37:27.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.660+0000 7fe8e2004740 1 -- 172.21.15.31:0/3895581641 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c5e5efb520 con 0x55c5e600b790 2024-03-21T10:37:27.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.660+0000 7fe8d7e00640 1 --2- 172.21.15.31:0/3895581641 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c5e600b790 0x55c5e600a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.661+0000 7fe8d7e00640 1 --2- 172.21.15.31:0/3895581641 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c5e600b790 0x55c5e600a990 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39a741d8410b9085 server_cookie=b90cc27985372ea0 in_seq=0 out_seq=0 2024-03-21T10:37:27.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.661+0000 7fe8c2000640 10 client.25192 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:37:27.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.661+0000 7fe8c2000640 1 -- 172.21.15.31:0/3895581641 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c5e5efb520 con 0x55c5e600b790 2024-03-21T10:37:27.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.661+0000 7fe8c2000640 10 client.25192 handle_command_reply: tid=0 2024-03-21T10:37:27.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.661+0000 7fe8e2004740 1 client.25192 shutdown 2024-03-21T10:37:27.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.661+0000 7fe8e2004740 1 -- 172.21.15.31:0/3895581641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5e5ffa8e0 msgr2=0x55c5e60067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.662+0000 7fe8e2004740 1 --2- 172.21.15.31:0/3895581641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5e5ffa8e0 0x55c5e60067d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe8d0006010 tx=0x7fe8d001b540 comp rx=0 tx=0).stop 2024-03-21T10:37:27.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.662+0000 7fe8e2004740 1 -- 172.21.15.31:0/3895581641 shutdown_connections 2024-03-21T10:37:27.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.662+0000 7fe8e2004740 1 --2- 172.21.15.31:0/3895581641 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c5e600b790 0x55c5e600a990 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.662+0000 7fe8e2004740 1 --2- 172.21.15.31:0/3895581641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5e5ffa8e0 0x55c5e60067d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.662+0000 7fe8e2004740 1 --2- 172.21.15.31:0/3895581641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5e5ff5770 0x55c5e5fff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.662+0000 7fe8e2004740 1 --2- 172.21.15.31:0/3895581641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5e5ff4d80 0x55c5e5fff250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.662+0000 7fe8e2004740 1 -- 172.21.15.31:0/3895581641 >> 172.21.15.31:0/3895581641 conn(0x55c5e5ff02f0 msgr2=0x55c5e5fef4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:27.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.662+0000 7fe8c2000640 0 client.25192 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:37:27.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.662+0000 7fe8e2004740 1 -- 172.21.15.31:0/3895581641 shutdown_connections 2024-03-21T10:37:27.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.662+0000 7fe8e2004740 1 -- 172.21.15.31:0/3895581641 wait complete. 2024-03-21T10:37:27.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.662+0000 7fe8e2004740 20 client.25192 trim_cache size 0 max 16384 2024-03-21T10:37:27.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:27 smithi040 ceph-mon[36154]: pgmap v399: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:37:27.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:27 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1486917795' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:37:27.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.689+0000 7fe8e2004740 1 Processor -- start 2024-03-21T10:37:27.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.690+0000 7fe8e2004740 1 -- start start 2024-03-21T10:37:27.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.690+0000 7fe8e2004740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5e5ff4d80 0x55c5e5ff06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.690+0000 7fe8e2004740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5e5ff5770 0x55c5e6006790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.690+0000 7fe8e2004740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5e5ffa8e0 0x55c5e5ffddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.690+0000 7fe8e2004740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e600afb0 con 0x55c5e5ffa8e0 2024-03-21T10:37:27.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.690+0000 7fe8e2004740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e600b610 con 0x55c5e5ff5770 2024-03-21T10:37:27.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.690+0000 7fe8e2004740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e600bc70 con 0x55c5e5ff4d80 2024-03-21T10:37:27.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.690+0000 7fe8d7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5e5ff4d80 0x55c5e5ff06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.690+0000 7fe8d7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5e5ff5770 0x55c5e6006790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.690+0000 7fe8d7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5e5ff4d80 0x55c5e5ff06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:55266/0 (socket says 172.21.15.31:55266) 2024-03-21T10:37:27.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.690+0000 7fe8d7e00640 1 -- 172.21.15.31:0/1493841183 learned_addr learned my addr 172.21.15.31:0/1493841183 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:27.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.690+0000 7fe8dd200640 1 --2- 172.21.15.31:0/1493841183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5e5ffa8e0 0x55c5e5ffddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.691+0000 7fe8d7e00640 1 -- 172.21.15.31:0/1493841183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5e5ff5770 msgr2=0x55c5e6006790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:37:27.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.691+0000 7fe8d7e00640 1 --2- 172.21.15.31:0/1493841183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5e5ff5770 0x55c5e6006790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.691+0000 7fe8d7e00640 1 -- 172.21.15.31:0/1493841183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5e5ffa8e0 msgr2=0x55c5e5ffddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:37:27.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.691+0000 7fe8d7e00640 1 --2- 172.21.15.31:0/1493841183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5e5ffa8e0 0x55c5e5ffddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.691+0000 7fe8d7e00640 1 -- 172.21.15.31:0/1493841183 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5e600ad00 con 0x55c5e5ff4d80 2024-03-21T10:37:27.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.691+0000 7fe8d7e00640 1 --2- 172.21.15.31:0/1493841183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5e5ff4d80 0x55c5e5ff06d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe8c80b9ba0 tx=0x7fe8c80a8040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a7755b2b75ac223 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:27.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.691+0000 7fe8ccc00640 1 -- 172.21.15.31:0/1493841183 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c804b040 con 0x55c5e5ff4d80 2024-03-21T10:37:27.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.691+0000 7fe8ccc00640 1 -- 172.21.15.31:0/1493841183 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe8c80bde40 con 0x55c5e5ff4d80 2024-03-21T10:37:27.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.691+0000 7fe8ccc00640 1 -- 172.21.15.31:0/1493841183 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c80c6150 con 0x55c5e5ff4d80 2024-03-21T10:37:27.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.692+0000 7fe8e2004740 1 -- 172.21.15.31:0/1493841183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5e5ff4d80 msgr2=0x55c5e5ff06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.692+0000 7fe8e2004740 1 --2- 172.21.15.31:0/1493841183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5e5ff4d80 0x55c5e5ff06d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe8c80b9ba0 tx=0x7fe8c80a8040 comp rx=0 tx=0).stop 2024-03-21T10:37:27.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.692+0000 7fe8e2004740 1 -- 172.21.15.31:0/1493841183 shutdown_connections 2024-03-21T10:37:27.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.692+0000 7fe8e2004740 1 --2- 172.21.15.31:0/1493841183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5e5ffa8e0 0x55c5e5ffddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.692+0000 7fe8e2004740 1 --2- 172.21.15.31:0/1493841183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5e5ff5770 0x55c5e6006790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.692+0000 7fe8e2004740 1 --2- 172.21.15.31:0/1493841183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5e5ff4d80 0x55c5e5ff06d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.693+0000 7fe8e2004740 1 -- 172.21.15.31:0/1493841183 >> 172.21.15.31:0/1493841183 conn(0x55c5e5ff02f0 msgr2=0x55c5e5ff6170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:27.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.693+0000 7fe8e2004740 1 -- 172.21.15.31:0/1493841183 shutdown_connections 2024-03-21T10:37:27.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.693+0000 7fe8e2004740 1 -- 172.21.15.31:0/1493841183 wait complete. 2024-03-21T10:37:27.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.693+0000 7fe8e2004740 1 -- start start 2024-03-21T10:37:27.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.693+0000 7fe8e2004740 1 Processor -- start 2024-03-21T10:37:27.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.694+0000 7fe8e2004740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5e5ff4d80 0x55c5e6004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.694+0000 7fe8e2004740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5e5ff5770 0x55c5e60627d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.694+0000 7fe8e2004740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5e5ffa8e0 0x55c5e6064d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.694+0000 7fe8e2004740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e600c930 con 0x55c5e5ff5770 2024-03-21T10:37:27.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.694+0000 7fe8e2004740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e600c2d0 con 0x55c5e5ff4d80 2024-03-21T10:37:27.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.694+0000 7fe8e2004740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e600cf90 con 0x55c5e5ffa8e0 2024-03-21T10:37:27.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.694+0000 7fe8e2004740 10 client.0 _ll_register_callbacks cb 0x7fe8c80f4ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:37:27.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.694+0000 7fe8d7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5e5ff4d80 0x55c5e6004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.694+0000 7fe8d7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5e5ff4d80 0x55c5e6004780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:59438/0 (socket says 172.21.15.31:59438) 2024-03-21T10:37:27.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.694+0000 7fe8d7e00640 1 -- 172.21.15.31:0/3984210663 learned_addr learned my addr 172.21.15.31:0/3984210663 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:27.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.694+0000 7fe8dd200640 1 --2- 172.21.15.31:0/3984210663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5e5ffa8e0 0x55c5e6064d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.694+0000 7fe8d7400640 1 --2- 172.21.15.31:0/3984210663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5e5ff5770 0x55c5e60627d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.695+0000 7fe8d7e00640 1 -- 172.21.15.31:0/3984210663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5e5ffa8e0 msgr2=0x55c5e6064d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:37:27.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.695+0000 7fe8d7e00640 1 --2- 172.21.15.31:0/3984210663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5e5ffa8e0 0x55c5e6064d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.695+0000 7fe8d7e00640 1 -- 172.21.15.31:0/3984210663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5e5ff5770 msgr2=0x55c5e60627d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:37:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.695+0000 7fe8d7e00640 1 --2- 172.21.15.31:0/3984210663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5e5ff5770 0x55c5e60627d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.695+0000 7fe8d7e00640 1 -- 172.21.15.31:0/3984210663 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c5e6065500 con 0x55c5e5ff4d80 2024-03-21T10:37:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.695+0000 7fe8d7400640 1 --2- 172.21.15.31:0/3984210663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5e5ff5770 0x55c5e60627d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:37:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.695+0000 7fe8d7e00640 1 --2- 172.21.15.31:0/3984210663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5e5ff4d80 0x55c5e6004780 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe8c80ba110 tx=0x7fe8c8009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b5f293dbb9e1aa8 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.695+0000 7fe8c2000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:37:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.695+0000 7fe8c2000640 1 -- 172.21.15.31:0/3984210663 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c804b070 con 0x55c5e5ff4d80 2024-03-21T10:37:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.695+0000 7fe8c2000640 1 -- 172.21.15.31:0/3984210663 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fe8c80a80b0 con 0x55c5e5ff4d80 2024-03-21T10:37:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.695+0000 7fe8e2004740 1 -- 172.21.15.31:0/3984210663 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe8c80bde40 con 0x55c5e5ff4d80 2024-03-21T10:37:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.695+0000 7fe8c2000640 1 -- 172.21.15.31:0/3984210663 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe8c80bd510 con 0x55c5e5ff4d80 2024-03-21T10:37:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.695+0000 7fe8e2004740 10 client.25208 fetch_fsmap learned FSMap version 23 2024-03-21T10:37:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.695+0000 7fe8e2004740 1 -- 172.21.15.31:0/3984210663 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c5e60667e0 con 0x55c5e5ff4d80 2024-03-21T10:37:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.696+0000 7fe8c2000640 1 -- 172.21.15.31:0/3984210663 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe8c80bde40 con 0x55c5e5ff4d80 2024-03-21T10:37:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.696+0000 7fe8c2000640 1 -- 172.21.15.31:0/3984210663 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 23) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fe8c80c5cc0 con 0x55c5e5ff4d80 2024-03-21T10:37:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.696+0000 7fe8e2004740 10 client.25208 fetch_fsmap finished waiting for FSMap version 23 2024-03-21T10:37:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.696+0000 7fe8e2004740 10 client.25208 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-21T10:37:27.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.696+0000 7fe8e2004740 20 client.25208 populate_metadata read hostname 'smithi031' 2024-03-21T10:37:27.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.696+0000 7fe8e2004740 1 --2- 172.21.15.31:0/3984210663 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c5e606aa30 0x55c5e60667e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:27.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.696+0000 7fe8e2004740 4 client.25208 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-21T10:37:27.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.696+0000 7fe8e2004740 1 -- 172.21.15.31:0/3984210663 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c5e5f671d0 con 0x55c5e606aa30 2024-03-21T10:37:27.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.696+0000 7fe8d7400640 1 --2- 172.21.15.31:0/3984210663 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c5e606aa30 0x55c5e60667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:27.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.697+0000 7fe8d7400640 1 --2- 172.21.15.31:0/3984210663 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c5e606aa30 0x55c5e60667e0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=776df4c6ca0c5946 server_cookie=9206ca5d2f867736 in_seq=0 out_seq=0 2024-03-21T10:37:27.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.697+0000 7fe8c2000640 10 client.25208 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:37:27.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.697+0000 7fe8c2000640 1 -- 172.21.15.31:0/3984210663 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55c5e5f671d0 con 0x55c5e606aa30 2024-03-21T10:37:27.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.697+0000 7fe8c2000640 10 client.25208 handle_command_reply: tid=0 2024-03-21T10:37:27.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.697+0000 7fe8e2004740 1 client.25208 shutdown 2024-03-21T10:37:27.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.697+0000 7fe8e2004740 1 -- 172.21.15.31:0/3984210663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5e5ff4d80 msgr2=0x55c5e6004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.697+0000 7fe8e2004740 1 --2- 172.21.15.31:0/3984210663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5e5ff4d80 0x55c5e6004780 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe8c80ba110 tx=0x7fe8c8009680 comp rx=0 tx=0).stop 2024-03-21T10:37:27.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.697+0000 7fe8e2004740 1 -- 172.21.15.31:0/3984210663 shutdown_connections 2024-03-21T10:37:27.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.697+0000 7fe8e2004740 1 --2- 172.21.15.31:0/3984210663 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c5e606aa30 0x55c5e60667e0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.697+0000 7fe8e2004740 1 --2- 172.21.15.31:0/3984210663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5e5ffa8e0 0x55c5e6064d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.697+0000 7fe8e2004740 1 --2- 172.21.15.31:0/3984210663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5e5ff5770 0x55c5e60627d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.697+0000 7fe8e2004740 1 --2- 172.21.15.31:0/3984210663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5e5ff4d80 0x55c5e6004780 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.697+0000 7fe8c2000640 0 client.25208 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:37:27.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.697+0000 7fe8e2004740 1 -- 172.21.15.31:0/3984210663 >> 172.21.15.31:0/3984210663 conn(0x55c5e5ff02f0 msgr2=0x55c5e5ff06d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:27.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.698+0000 7fe8e2004740 1 -- 172.21.15.31:0/3984210663 shutdown_connections 2024-03-21T10:37:27.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.698+0000 7fe8e2004740 1 -- 172.21.15.31:0/3984210663 wait complete. 2024-03-21T10:37:27.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.698+0000 7fe8e2004740 20 client.25208 trim_cache size 0 max 16384 2024-03-21T10:37:27.699 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:37:27.700 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "3847408c-e76d-11ee-95cd-87774f69a715", 2024-03-21T10:37:27.700 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2024-03-21T10:37:27.700 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24596, 2024-03-21T10:37:27.700 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-03-21T10:37:27.700 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-03-21T10:37:27.700 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-03-21T10:37:27.700 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 140.70090421500001, 2024-03-21T10:37:27.700 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 23, 2024-03-21T10:37:27.700 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 93, 2024-03-21T10:37:27.700 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 89, 2024-03-21T10:37:27.700 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 146.52456634200001 2024-03-21T10:37:27.700 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:37:27.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.698+0000 7fe8ddc00640 1 -- 172.21.15.31:0/2871731126 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe8bc0a2f20 msgr2=0x7fe8bc0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.698+0000 7fe8ddc00640 1 --2- 172.21.15.31:0/2871731126 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe8bc0a2f20 0x7fe8bc0a53e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe8d0020000 tx=0x7fe8d0021000 comp rx=0 tx=0).stop 2024-03-21T10:37:27.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.698+0000 7fe8ddc00640 1 -- 172.21.15.31:0/2871731126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8d8101f70 msgr2=0x7fe8d8199a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:27.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.698+0000 7fe8ddc00640 1 --2- 172.21.15.31:0/2871731126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8d8101f70 0x7fe8d8199a40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8005ab0 tx=0x7fe8c802c6e0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.698+0000 7fe8ddc00640 1 -- 172.21.15.31:0/2871731126 shutdown_connections 2024-03-21T10:37:27.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.698+0000 7fe8ddc00640 1 --2- 172.21.15.31:0/2871731126 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fe8bc0a2f20 0x7fe8bc0a53e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.698+0000 7fe8ddc00640 1 --2- 172.21.15.31:0/2871731126 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8d81071d0 0x7fe8d81a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.698+0000 7fe8ddc00640 1 --2- 172.21.15.31:0/2871731126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8d81048a0 0x7fe8d8199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.698+0000 7fe8ddc00640 1 --2- 172.21.15.31:0/2871731126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8d8101f70 0x7fe8d8199a40 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:27.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.698+0000 7fe8ddc00640 1 -- 172.21.15.31:0/2871731126 >> 172.21.15.31:0/2871731126 conn(0x7fe8d80fb980 msgr2=0x7fe8d8101700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:27.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.699+0000 7fe8ddc00640 1 -- 172.21.15.31:0/2871731126 shutdown_connections 2024-03-21T10:37:27.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:27.699+0000 7fe8ddc00640 1 -- 172.21.15.31:0/2871731126 wait complete. 2024-03-21T10:37:27.711 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:27 smithi031 ceph-mon[31673]: pgmap v399: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:37:27.711 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:27 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1486917795' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:37:27.714 INFO:tasks.cephadm:Setup complete, yielding 2024-03-21T10:37:27.715 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-21T10:37:27.725 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2024-03-21T10:37:27.725 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- bash -c 'ceph orch status' 2024-03-21T10:37:27.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:27 smithi070 ceph-mon[36330]: pgmap v399: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:37:27.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:27 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1486917795' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:37:28.228 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:37:29.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:29 smithi040 ceph-mon[36154]: pgmap v400: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:29.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:29 smithi031 ceph-mon[31673]: pgmap v400: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:29.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:29 smithi070 ceph-mon[36330]: pgmap v400: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:30.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.758+0000 7f3269000700 1 -- 172.21.15.31:0/802169567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3264104410 msgr2=0x7f3264104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:30.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.758+0000 7f3269000700 1 --2- 172.21.15.31:0/802169567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3264104410 0x7f3264104830 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3254009990 tx=0x7f3254009ca0 comp rx=0 tx=0).stop 2024-03-21T10:37:30.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.759+0000 7f3269000700 1 -- 172.21.15.31:0/802169567 shutdown_connections 2024-03-21T10:37:30.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.759+0000 7f3269000700 1 --2- 172.21.15.31:0/802169567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3264105fb0 0x7f326410c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:30.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.759+0000 7f3269000700 1 --2- 172.21.15.31:0/802169567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3264104410 0x7f3264104830 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:30.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.759+0000 7f3269000700 1 --2- 172.21.15.31:0/802169567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3264105610 0x7f3264105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:30.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.759+0000 7f3269000700 1 -- 172.21.15.31:0/802169567 >> 172.21.15.31:0/802169567 conn(0x7f32640ff9b0 msgr2=0x7f3264101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:30.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.759+0000 7f3269000700 1 -- 172.21.15.31:0/802169567 shutdown_connections 2024-03-21T10:37:30.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.759+0000 7f3269000700 1 -- 172.21.15.31:0/802169567 wait complete. 2024-03-21T10:37:30.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.760+0000 7f3269000700 1 Processor -- start 2024-03-21T10:37:30.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.760+0000 7f3269000700 1 -- start start 2024-03-21T10:37:30.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.761+0000 7f3269000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3264104410 0x7f326419a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:30.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.761+0000 7f3269000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3264105610 0x7f326419abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:30.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.761+0000 7f3269000700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3264105fb0 0x7f32641a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:30.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.761+0000 7f3269000700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3264110050 con 0x7f3264105fb0 2024-03-21T10:37:30.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.761+0000 7f3269000700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326410fed0 con 0x7f3264104410 2024-03-21T10:37:30.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.761+0000 7f3269000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32641101d0 con 0x7f3264105610 2024-03-21T10:37:30.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.761+0000 7f3261e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3264104410 0x7f326419a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:30.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.761+0000 7f3261e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3264104410 0x7f326419a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:59454/0 (socket says 172.21.15.31:59454) 2024-03-21T10:37:30.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.761+0000 7f3261e00700 1 -- 172.21.15.31:0/1276578881 learned_addr learned my addr 172.21.15.31:0/1276578881 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:30.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.761+0000 7f3262800700 1 --2- 172.21.15.31:0/1276578881 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3264105fb0 0x7f32641a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:30.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.761+0000 7f3261400700 1 --2- 172.21.15.31:0/1276578881 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3264105610 0x7f326419abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:30.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.761+0000 7f3261e00700 1 -- 172.21.15.31:0/1276578881 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3264105610 msgr2=0x7f326419abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:30.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.761+0000 7f3261e00700 1 --2- 172.21.15.31:0/1276578881 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3264105610 0x7f326419abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:30.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.761+0000 7f3261e00700 1 -- 172.21.15.31:0/1276578881 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3264105fb0 msgr2=0x7f32641a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:30.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.761+0000 7f3261e00700 1 --2- 172.21.15.31:0/1276578881 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3264105fb0 0x7f32641a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:30.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.761+0000 7f3261e00700 1 -- 172.21.15.31:0/1276578881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32641a22f0 con 0x7f3264104410 2024-03-21T10:37:30.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.761+0000 7f3262800700 1 --2- 172.21.15.31:0/1276578881 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3264105fb0 0x7f32641a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:37:30.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.762+0000 7f3261e00700 1 --2- 172.21.15.31:0/1276578881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3264104410 0x7f326419a690 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f325400b820 tx=0x7f325400f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:30.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.762+0000 7f325aa00700 1 -- 172.21.15.31:0/1276578881 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3254026070 con 0x7f3264104410 2024-03-21T10:37:30.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.762+0000 7f325aa00700 1 -- 172.21.15.31:0/1276578881 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32540218f0 con 0x7f3264104410 2024-03-21T10:37:30.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.762+0000 7f3269000700 1 -- 172.21.15.31:0/1276578881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32641a2580 con 0x7f3264104410 2024-03-21T10:37:30.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.762+0000 7f325aa00700 1 -- 172.21.15.31:0/1276578881 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3254017650 con 0x7f3264104410 2024-03-21T10:37:30.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.762+0000 7f3269000700 1 -- 172.21.15.31:0/1276578881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32641a29d0 con 0x7f3264104410 2024-03-21T10:37:30.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.763+0000 7f3269000700 1 -- 172.21.15.31:0/1276578881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3264068e50 con 0x7f3264104410 2024-03-21T10:37:30.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.765+0000 7f325aa00700 1 -- 172.21.15.31:0/1276578881 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f3254029070 con 0x7f3264104410 2024-03-21T10:37:30.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.766+0000 7f325aa00700 1 --2- 172.21.15.31:0/1276578881 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f32480a2fb0 0x7f32480a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:30.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.766+0000 7f325aa00700 1 -- 172.21.15.31:0/1276578881 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f32540d0650 con 0x7f3264104410 2024-03-21T10:37:30.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.767+0000 7f3261400700 1 --2- 172.21.15.31:0/1276578881 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f32480a2fb0 0x7f32480a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:30.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.767+0000 7f3261400700 1 --2- 172.21.15.31:0/1276578881 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f32480a2fb0 0x7f32480a5470 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3250009c60 tx=0x7f3250009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:30.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:30.770+0000 7f325aa00700 1 -- 172.21.15.31:0/1276578881 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3254013070 con 0x7f3264104410 2024-03-21T10:37:31.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:31.028+0000 7f3269000700 1 -- 172.21.15.31:0/1276578881 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f32640625d0 con 0x7f32480a2fb0 2024-03-21T10:37:31.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:31.031+0000 7f325aa00700 1 -- 172.21.15.31:0/1276578881 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f32640625d0 con 0x7f32480a2fb0 2024-03-21T10:37:31.029 INFO:teuthology.orchestra.run.smithi031.stdout:Backend: cephadm 2024-03-21T10:37:31.029 INFO:teuthology.orchestra.run.smithi031.stdout:Available: Yes 2024-03-21T10:37:31.029 INFO:teuthology.orchestra.run.smithi031.stdout:Paused: No 2024-03-21T10:37:31.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:31.035+0000 7f3269000700 1 -- 172.21.15.31:0/1276578881 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f32480a2fb0 msgr2=0x7f32480a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:31.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:31.035+0000 7f3269000700 1 --2- 172.21.15.31:0/1276578881 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f32480a2fb0 0x7f32480a5470 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3250009c60 tx=0x7f3250009260 comp rx=0 tx=0).stop 2024-03-21T10:37:31.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:31.035+0000 7f3269000700 1 -- 172.21.15.31:0/1276578881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3264104410 msgr2=0x7f326419a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:31.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:31.035+0000 7f3269000700 1 --2- 172.21.15.31:0/1276578881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3264104410 0x7f326419a690 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f325400b820 tx=0x7f325400f690 comp rx=0 tx=0).stop 2024-03-21T10:37:31.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:31.035+0000 7f3269000700 1 -- 172.21.15.31:0/1276578881 shutdown_connections 2024-03-21T10:37:31.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:31.035+0000 7f3269000700 1 --2- 172.21.15.31:0/1276578881 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f32480a2fb0 0x7f32480a5470 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:31.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:31.035+0000 7f3269000700 1 --2- 172.21.15.31:0/1276578881 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3264105fb0 0x7f32641a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:31.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:31.035+0000 7f3269000700 1 --2- 172.21.15.31:0/1276578881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3264104410 0x7f326419a690 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:31.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:31.035+0000 7f3269000700 1 --2- 172.21.15.31:0/1276578881 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3264105610 0x7f326419abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:31.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:31.035+0000 7f3269000700 1 -- 172.21.15.31:0/1276578881 >> 172.21.15.31:0/1276578881 conn(0x7f32640ff9b0 msgr2=0x7f3264108910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:31.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:31.035+0000 7f3269000700 1 -- 172.21.15.31:0/1276578881 shutdown_connections 2024-03-21T10:37:31.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:31.035+0000 7f3269000700 1 -- 172.21.15.31:0/1276578881 wait complete. 2024-03-21T10:37:31.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:31 smithi040 ceph-mon[36154]: pgmap v401: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:37:31.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:31 smithi031 ceph-mon[31673]: pgmap v401: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:37:31.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:31 smithi070 ceph-mon[36330]: pgmap v401: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:37:31.983 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-21T10:37:32.446 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:37:32.789 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:32 smithi031 ceph-mon[31673]: from='client.25214 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:37:32.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:32 smithi070 ceph-mon[36330]: from='client.25214 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:37:32.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:32 smithi040 ceph-mon[36154]: from='client.25214 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:37:33.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:33 smithi031 ceph-mon[31673]: pgmap v402: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:37:33.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:33 smithi070 ceph-mon[36330]: pgmap v402: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:37:33.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:33 smithi040 ceph-mon[36154]: pgmap v402: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:37:34.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.848+0000 7fea80800700 1 -- 172.21.15.31:0/2100804943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea78074b90 msgr2=0x7fea78075010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:34.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.848+0000 7fea80800700 1 --2- 172.21.15.31:0/2100804943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea78074b90 0x7fea78075010 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fea68009990 tx=0x7fea68009ca0 comp rx=0 tx=0).stop 2024-03-21T10:37:34.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.848+0000 7fea80800700 1 -- 172.21.15.31:0/2100804943 shutdown_connections 2024-03-21T10:37:34.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.848+0000 7fea80800700 1 --2- 172.21.15.31:0/2100804943 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea7810a0f0 0x7fea7810c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:34.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.848+0000 7fea80800700 1 --2- 172.21.15.31:0/2100804943 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea78076160 0x7fea780745c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:34.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.848+0000 7fea80800700 1 --2- 172.21.15.31:0/2100804943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea78074b90 0x7fea78075010 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:34.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.848+0000 7fea80800700 1 -- 172.21.15.31:0/2100804943 >> 172.21.15.31:0/2100804943 conn(0x7fea780fd7b0 msgr2=0x7fea780ffbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:34.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.848+0000 7fea80800700 1 -- 172.21.15.31:0/2100804943 shutdown_connections 2024-03-21T10:37:34.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.848+0000 7fea80800700 1 -- 172.21.15.31:0/2100804943 wait complete. 2024-03-21T10:37:34.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.849+0000 7fea80800700 1 Processor -- start 2024-03-21T10:37:34.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.849+0000 7fea80800700 1 -- start start 2024-03-21T10:37:34.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.850+0000 7fea80800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea78074b90 0x7fea7819a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:34.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.850+0000 7fea80800700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea78076160 0x7fea7819ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:34.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.850+0000 7fea80800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea7810a0f0 0x7fea781a1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:34.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.850+0000 7fea80800700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea78110010 con 0x7fea78076160 2024-03-21T10:37:34.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.850+0000 7fea80800700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea7810fe90 con 0x7fea78074b90 2024-03-21T10:37:34.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.850+0000 7fea80800700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea78110190 con 0x7fea7810a0f0 2024-03-21T10:37:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.850+0000 7fea7d400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea78076160 0x7fea7819ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.850+0000 7fea7de00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea78074b90 0x7fea7819a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.850+0000 7fea7e800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea7810a0f0 0x7fea781a1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.850+0000 7fea7e800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea7810a0f0 0x7fea781a1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:55312/0 (socket says 172.21.15.31:55312) 2024-03-21T10:37:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.850+0000 7fea7d400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea78076160 0x7fea7819ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52812/0 (socket says 172.21.15.31:52812) 2024-03-21T10:37:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.850+0000 7fea7de00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea78074b90 0x7fea7819a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:59468/0 (socket says 172.21.15.31:59468) 2024-03-21T10:37:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.850+0000 7fea7e800700 1 -- 172.21.15.31:0/3344957999 learned_addr learned my addr 172.21.15.31:0/3344957999 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:34.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.851+0000 7fea7de00700 1 -- 172.21.15.31:0/3344957999 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea7810a0f0 msgr2=0x7fea781a1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:34.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.851+0000 7fea7de00700 1 --2- 172.21.15.31:0/3344957999 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea7810a0f0 0x7fea781a1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:34.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.851+0000 7fea7de00700 1 -- 172.21.15.31:0/3344957999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea78076160 msgr2=0x7fea7819ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:34.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.851+0000 7fea7de00700 1 --2- 172.21.15.31:0/3344957999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea78076160 0x7fea7819ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:34.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.851+0000 7fea7de00700 1 -- 172.21.15.31:0/3344957999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea781a2230 con 0x7fea78074b90 2024-03-21T10:37:34.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.851+0000 7fea7de00700 1 --2- 172.21.15.31:0/3344957999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea78074b90 0x7fea7819a5d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fea6c00e9f0 tx=0x7fea6c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:34.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.851+0000 7fea72a00700 1 -- 172.21.15.31:0/3344957999 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6c00cb70 con 0x7fea78074b90 2024-03-21T10:37:34.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.851+0000 7fea72a00700 1 -- 172.21.15.31:0/3344957999 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fea6c00ee90 con 0x7fea78074b90 2024-03-21T10:37:34.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.851+0000 7fea72a00700 1 -- 172.21.15.31:0/3344957999 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6c003df0 con 0x7fea78074b90 2024-03-21T10:37:34.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.851+0000 7fea80800700 1 -- 172.21.15.31:0/3344957999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea781a2520 con 0x7fea78074b90 2024-03-21T10:37:34.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.852+0000 7fea80800700 1 -- 172.21.15.31:0/3344957999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea781a2ab0 con 0x7fea78074b90 2024-03-21T10:37:34.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.853+0000 7fea80800700 1 -- 172.21.15.31:0/3344957999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea78068e50 con 0x7fea78074b90 2024-03-21T10:37:34.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.855+0000 7fea72a00700 1 -- 172.21.15.31:0/3344957999 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fea6c003f90 con 0x7fea78074b90 2024-03-21T10:37:34.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.856+0000 7fea72a00700 1 --2- 172.21.15.31:0/3344957999 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fea600a3030 0x7fea600a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:34.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.856+0000 7fea72a00700 1 -- 172.21.15.31:0/3344957999 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fea6c0cf830 con 0x7fea78074b90 2024-03-21T10:37:34.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.857+0000 7fea7d400700 1 --2- 172.21.15.31:0/3344957999 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fea600a3030 0x7fea600a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:34.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.857+0000 7fea7d400700 1 --2- 172.21.15.31:0/3344957999 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fea600a3030 0x7fea600a54f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fea6800b2e0 tx=0x7fea68005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:34.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:34.865+0000 7fea72a00700 1 -- 172.21.15.31:0/3344957999 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fea6c09c2a0 con 0x7fea78074b90 2024-03-21T10:37:35.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:35.140+0000 7fea80800700 1 -- 172.21.15.31:0/3344957999 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fea780625d0 con 0x7fea600a3030 2024-03-21T10:37:35.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:35.179+0000 7fea72a00700 1 -- 172.21.15.31:0/3344957999 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4123 (secure 0 0 0) 0x7fea780625d0 con 0x7fea600a3030 2024-03-21T10:37:35.178 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T10:37:35.178 INFO:teuthology.orchestra.run.smithi031.stdout:mds.a smithi031 running (2m) 36s ago 2m 16.1M - 19.0.0-2231-g2edfee68 237e43c7a65f 0700393bf215 2024-03-21T10:37:35.178 INFO:teuthology.orchestra.run.smithi031.stdout:mds.b smithi040 running (2m) 84s ago 2m 25.6M - 19.0.0-2231-g2edfee68 237e43c7a65f 68b79ae43eaf 2024-03-21T10:37:35.178 INFO:teuthology.orchestra.run.smithi031.stdout:mds.c smithi070 running (2m) 84s ago 2m 17.3M - 19.0.0-2231-g2edfee68 237e43c7a65f 5b129a97f390 2024-03-21T10:37:35.178 INFO:teuthology.orchestra.run.smithi031.stdout:mds.d smithi031 running (3m) 36s ago 3m 12.5M - 19.0.0-2231-g2edfee68 237e43c7a65f 283c28155b7b 2024-03-21T10:37:35.178 INFO:teuthology.orchestra.run.smithi031.stdout:mds.e smithi040 running (2m) 84s ago 2m 16.3M - 19.0.0-2231-g2edfee68 237e43c7a65f 4a091e9b018e 2024-03-21T10:37:35.178 INFO:teuthology.orchestra.run.smithi031.stdout:mds.f smithi070 running (2m) 84s ago 2m 18.2M - 19.0.0-2231-g2edfee68 237e43c7a65f 33d6518933b0 2024-03-21T10:37:35.179 INFO:teuthology.orchestra.run.smithi031.stdout:mds.g smithi031 running (2m) 36s ago 2m 12.4M - 19.0.0-2231-g2edfee68 237e43c7a65f cf9cdadc0cb0 2024-03-21T10:37:35.179 INFO:teuthology.orchestra.run.smithi031.stdout:mds.h smithi040 running (2m) 84s ago 2m 15.8M - 19.0.0-2231-g2edfee68 237e43c7a65f 101a42563d4b 2024-03-21T10:37:35.179 INFO:teuthology.orchestra.run.smithi031.stdout:mds.i smithi070 running (2m) 84s ago 2m 16.5M - 19.0.0-2231-g2edfee68 237e43c7a65f e44ffbadd2e5 2024-03-21T10:37:35.179 INFO:teuthology.orchestra.run.smithi031.stdout:mds.j smithi031 running (3m) 36s ago 3m 12.5M - 19.0.0-2231-g2edfee68 237e43c7a65f 64416c35f407 2024-03-21T10:37:35.179 INFO:teuthology.orchestra.run.smithi031.stdout:mds.k smithi040 running (3m) 84s ago 3m 13.2M - 19.0.0-2231-g2edfee68 237e43c7a65f 2858ae883013 2024-03-21T10:37:35.179 INFO:teuthology.orchestra.run.smithi031.stdout:mds.l smithi070 running (3m) 84s ago 3m 16.4M - 19.0.0-2231-g2edfee68 237e43c7a65f 0f729b3e58ba 2024-03-21T10:37:35.179 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.x smithi031 *:9283,8765 running (11m) 36s ago 11m 513M - 19.0.0-2231-g2edfee68 237e43c7a65f d2fad9ce65f9 2024-03-21T10:37:35.179 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.y smithi040 *:8443,8765 running (9m) 84s ago 9m 451M - 19.0.0-2231-g2edfee68 237e43c7a65f 37bcf80653ae 2024-03-21T10:37:35.179 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.z smithi070 *:8443,8765 running (9m) 84s ago 9m 454M - 19.0.0-2231-g2edfee68 237e43c7a65f 7aeba3458430 2024-03-21T10:37:35.179 INFO:teuthology.orchestra.run.smithi031.stdout:mon.a smithi031 running (11m) 36s ago 11m 48.2M 2048M 19.0.0-2231-g2edfee68 237e43c7a65f aeb8a37c34e6 2024-03-21T10:37:35.179 INFO:teuthology.orchestra.run.smithi031.stdout:mon.b smithi040 running (9m) 84s ago 9m 41.5M 2048M 19.0.0-2231-g2edfee68 237e43c7a65f 5897a61780c0 2024-03-21T10:37:35.179 INFO:teuthology.orchestra.run.smithi031.stdout:mon.c smithi070 running (9m) 84s ago 9m 43.8M 2048M 19.0.0-2231-g2edfee68 237e43c7a65f 0db0cc412609 2024-03-21T10:37:35.179 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (8m) 36s ago 8m 56.5M 4273M 19.0.0-2231-g2edfee68 237e43c7a65f 5a69b9fd1361 2024-03-21T10:37:35.179 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi040 running (8m) 84s ago 8m 57.1M 4271M 19.0.0-2231-g2edfee68 237e43c7a65f e878d4a7bd93 2024-03-21T10:37:35.179 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi070 running (7m) 84s ago 7m 45.3M 4270M 19.0.0-2231-g2edfee68 237e43c7a65f cee1732e04fc 2024-03-21T10:37:35.180 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (7m) 36s ago 7m 43.2M 4273M 19.0.0-2231-g2edfee68 237e43c7a65f 1c9db06e79bc 2024-03-21T10:37:35.180 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi040 running (6m) 84s ago 6m 42.9M 4271M 19.0.0-2231-g2edfee68 237e43c7a65f 4aff6cfa99dc 2024-03-21T10:37:35.180 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi070 running (6m) 84s ago 6m 44.8M 4270M 19.0.0-2231-g2edfee68 237e43c7a65f 8b449ee2c4de 2024-03-21T10:37:35.180 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi031 running (5m) 36s ago 5m 42.0M 4273M 19.0.0-2231-g2edfee68 237e43c7a65f d0481594607d 2024-03-21T10:37:35.180 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi040 running (5m) 84s ago 5m 55.9M 4271M 19.0.0-2231-g2edfee68 237e43c7a65f 72f416a9c71f 2024-03-21T10:37:35.180 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8 smithi070 running (5m) 84s ago 5m 56.3M 4270M 19.0.0-2231-g2edfee68 237e43c7a65f 8e799f5de332 2024-03-21T10:37:35.180 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9 smithi031 running (4m) 36s ago 4m 42.3M 4273M 19.0.0-2231-g2edfee68 237e43c7a65f 78e358f696b2 2024-03-21T10:37:35.180 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10 smithi040 running (4m) 84s ago 4m 54.2M 4271M 19.0.0-2231-g2edfee68 237e43c7a65f ef51dcba7b29 2024-03-21T10:37:35.180 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11 smithi070 running (3m) 84s ago 3m 50.3M 4270M 19.0.0-2231-g2edfee68 237e43c7a65f 2553e8cac7c5 2024-03-21T10:37:35.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:35.182+0000 7fea80800700 1 -- 172.21.15.31:0/3344957999 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fea600a3030 msgr2=0x7fea600a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:35.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:35.182+0000 7fea80800700 1 --2- 172.21.15.31:0/3344957999 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fea600a3030 0x7fea600a54f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fea6800b2e0 tx=0x7fea68005fd0 comp rx=0 tx=0).stop 2024-03-21T10:37:35.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:35.182+0000 7fea80800700 1 -- 172.21.15.31:0/3344957999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea78074b90 msgr2=0x7fea7819a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:35.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:35.182+0000 7fea80800700 1 --2- 172.21.15.31:0/3344957999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea78074b90 0x7fea7819a5d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fea6c00e9f0 tx=0x7fea6c00ed00 comp rx=0 tx=0).stop 2024-03-21T10:37:35.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:35.182+0000 7fea80800700 1 -- 172.21.15.31:0/3344957999 shutdown_connections 2024-03-21T10:37:35.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:35.182+0000 7fea80800700 1 --2- 172.21.15.31:0/3344957999 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fea600a3030 0x7fea600a54f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:35.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:35.182+0000 7fea80800700 1 --2- 172.21.15.31:0/3344957999 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea7810a0f0 0x7fea781a1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:35.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:35.182+0000 7fea80800700 1 --2- 172.21.15.31:0/3344957999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea78074b90 0x7fea7819a5d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:35.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:35.182+0000 7fea80800700 1 --2- 172.21.15.31:0/3344957999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea78076160 0x7fea7819ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:35.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:35.182+0000 7fea80800700 1 -- 172.21.15.31:0/3344957999 >> 172.21.15.31:0/3344957999 conn(0x7fea780fd7b0 msgr2=0x7fea78106040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:35.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:35.183+0000 7fea80800700 1 -- 172.21.15.31:0/3344957999 shutdown_connections 2024-03-21T10:37:35.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:35.183+0000 7fea80800700 1 -- 172.21.15.31:0/3344957999 wait complete. 2024-03-21T10:37:35.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:35 smithi031 ceph-mon[31673]: pgmap v403: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:35.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:35 smithi070 ceph-mon[36330]: pgmap v403: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:35.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:35 smithi040 ceph-mon[36154]: pgmap v403: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:36.101 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-21T10:37:36.590 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:37:36.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:36 smithi031 ceph-mon[31673]: from='client.25220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:37:36.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:36 smithi070 ceph-mon[36330]: from='client.25220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:37:36.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:36 smithi040 ceph-mon[36154]: from='client.25220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:37:37.792 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:37 smithi031 ceph-mon[31673]: pgmap v404: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:37:37.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:37 smithi070 ceph-mon[36330]: pgmap v404: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:37:37.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:37 smithi040 ceph-mon[36154]: pgmap v404: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:37:39.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.096+0000 7f12c5800700 1 -- 172.21.15.31:0/1373728086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12c0108810 msgr2=0x7f12c0108c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:39.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.096+0000 7f12c5800700 1 --2- 172.21.15.31:0/1373728086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12c0108810 0x7f12c0108c30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f12b00098f0 tx=0x7f12b0009c00 comp rx=0 tx=0).stop 2024-03-21T10:37:39.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.097+0000 7f12c5800700 1 -- 172.21.15.31:0/1373728086 shutdown_connections 2024-03-21T10:37:39.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.097+0000 7f12c5800700 1 --2- 172.21.15.31:0/1373728086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12c010a3b0 0x7f12c0110c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:39.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.097+0000 7f12c5800700 1 --2- 172.21.15.31:0/1373728086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12c0108810 0x7f12c0108c30 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:39.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.097+0000 7f12c5800700 1 --2- 172.21.15.31:0/1373728086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12c0109a10 0x7f12c0109e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:39.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.097+0000 7f12c5800700 1 -- 172.21.15.31:0/1373728086 >> 172.21.15.31:0/1373728086 conn(0x7f12c00777e0 msgr2=0x7f12c0079c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:39.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.097+0000 7f12c5800700 1 -- 172.21.15.31:0/1373728086 shutdown_connections 2024-03-21T10:37:39.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.097+0000 7f12c5800700 1 -- 172.21.15.31:0/1373728086 wait complete. 2024-03-21T10:37:39.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.098+0000 7f12c5800700 1 Processor -- start 2024-03-21T10:37:39.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.098+0000 7f12c5800700 1 -- start start 2024-03-21T10:37:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.098+0000 7f12c5800700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12c0108810 0x7f12c019ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.099+0000 7f12c5800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12c0109a10 0x7f12c019efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.099+0000 7f12c5800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12c010a3b0 0x7f12c01a6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.099+0000 7f12c5800700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c0114420 con 0x7f12c0108810 2024-03-21T10:37:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.099+0000 7f12c5800700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c01142a0 con 0x7f12c010a3b0 2024-03-21T10:37:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.099+0000 7f12c5800700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c01145a0 con 0x7f12c0109a10 2024-03-21T10:37:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.099+0000 7f12bea00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12c0108810 0x7f12c019ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.099+0000 7f12bea00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12c0108810 0x7f12c019ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40560/0 (socket says 172.21.15.31:40560) 2024-03-21T10:37:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.099+0000 7f12bea00700 1 -- 172.21.15.31:0/3117978344 learned_addr learned my addr 172.21.15.31:0/3117978344 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.099+0000 7f12be000700 1 --2- 172.21.15.31:0/3117978344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12c0109a10 0x7f12c019efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:39.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.099+0000 7f12bf400700 1 --2- 172.21.15.31:0/3117978344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12c010a3b0 0x7f12c01a6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:39.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.099+0000 7f12bea00700 1 -- 172.21.15.31:0/3117978344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12c0109a10 msgr2=0x7f12c019efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:39.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.099+0000 7f12bea00700 1 --2- 172.21.15.31:0/3117978344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12c0109a10 0x7f12c019efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:39.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.099+0000 7f12bea00700 1 -- 172.21.15.31:0/3117978344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12c010a3b0 msgr2=0x7f12c01a6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:39.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.099+0000 7f12bea00700 1 --2- 172.21.15.31:0/3117978344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12c010a3b0 0x7f12c01a6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:39.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.099+0000 7f12bea00700 1 -- 172.21.15.31:0/3117978344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c01a66c0 con 0x7f12c0108810 2024-03-21T10:37:39.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.099+0000 7f12be000700 1 --2- 172.21.15.31:0/3117978344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12c0109a10 0x7f12c019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:37:39.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.100+0000 7f12bea00700 1 --2- 172.21.15.31:0/3117978344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12c0108810 0x7f12c019ea60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f12b0000c00 tx=0x7f12b00052f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:39.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.100+0000 7f12b7400700 1 -- 172.21.15.31:0/3117978344 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b000b560 con 0x7f12c0108810 2024-03-21T10:37:39.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.100+0000 7f12b7400700 1 -- 172.21.15.31:0/3117978344 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12b000bbe0 con 0x7f12c0108810 2024-03-21T10:37:39.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.100+0000 7f12b7400700 1 -- 172.21.15.31:0/3117978344 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b0017400 con 0x7f12c0108810 2024-03-21T10:37:39.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.100+0000 7f12c5800700 1 -- 172.21.15.31:0/3117978344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c01a6950 con 0x7f12c0108810 2024-03-21T10:37:39.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.100+0000 7f12c5800700 1 -- 172.21.15.31:0/3117978344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c01a6d50 con 0x7f12c0108810 2024-03-21T10:37:39.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.102+0000 7f12c5800700 1 -- 172.21.15.31:0/3117978344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c0068e90 con 0x7f12c0108810 2024-03-21T10:37:39.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.104+0000 7f12b7400700 1 -- 172.21.15.31:0/3117978344 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f12b0003680 con 0x7f12c0108810 2024-03-21T10:37:39.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.105+0000 7f12b7400700 1 --2- 172.21.15.31:0/3117978344 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f12a40a3030 0x7f12a40a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:39.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.105+0000 7f12b7400700 1 -- 172.21.15.31:0/3117978344 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f12b00d0980 con 0x7f12c0108810 2024-03-21T10:37:39.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.105+0000 7f12be000700 1 --2- 172.21.15.31:0/3117978344 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f12a40a3030 0x7f12a40a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:39.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.106+0000 7f12be000700 1 --2- 172.21.15.31:0/3117978344 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f12a40a3030 0x7f12a40a54f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f12ac009750 tx=0x7f12ac009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:39.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.109+0000 7f12b7400700 1 -- 172.21.15.31:0/3117978344 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f12b0013030 con 0x7f12c0108810 2024-03-21T10:37:39.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.358+0000 7f12c5800700 1 -- 172.21.15.31:0/3117978344 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f12c00625d0 con 0x7f12a40a3030 2024-03-21T10:37:39.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.376+0000 7f12b7400700 1 -- 172.21.15.31:0/3117978344 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7f12c00625d0 con 0x7f12a40a3030 2024-03-21T10:37:39.375 INFO:teuthology.orchestra.run.smithi031.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-21T10:37:39.375 INFO:teuthology.orchestra.run.smithi031.stdout:mds.all 12/12 89s ago 3m smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi070=c;smithi070=f;smithi070=i;smithi070=l;count:12 2024-03-21T10:37:39.375 INFO:teuthology.orchestra.run.smithi031.stdout:mgr 3/3 89s ago 9m smithi031=x;smithi040=y;smithi070=z;count:3 2024-03-21T10:37:39.375 INFO:teuthology.orchestra.run.smithi031.stdout:mon 3/3 89s ago 9m smithi031:172.21.15.31=a;smithi040:172.21.15.40=b;smithi070:172.21.15.70=c;count:3 2024-03-21T10:37:39.375 INFO:teuthology.orchestra.run.smithi031.stdout:osd 12 89s ago - 2024-03-21T10:37:39.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.379+0000 7f12c5800700 1 -- 172.21.15.31:0/3117978344 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f12a40a3030 msgr2=0x7f12a40a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:39.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.379+0000 7f12c5800700 1 --2- 172.21.15.31:0/3117978344 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f12a40a3030 0x7f12a40a54f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f12ac009750 tx=0x7f12ac009310 comp rx=0 tx=0).stop 2024-03-21T10:37:39.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.379+0000 7f12c5800700 1 -- 172.21.15.31:0/3117978344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12c0108810 msgr2=0x7f12c019ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:39.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.379+0000 7f12c5800700 1 --2- 172.21.15.31:0/3117978344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12c0108810 0x7f12c019ea60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f12b0000c00 tx=0x7f12b00052f0 comp rx=0 tx=0).stop 2024-03-21T10:37:39.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.379+0000 7f12c5800700 1 -- 172.21.15.31:0/3117978344 shutdown_connections 2024-03-21T10:37:39.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.379+0000 7f12c5800700 1 --2- 172.21.15.31:0/3117978344 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f12a40a3030 0x7f12a40a54f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:39.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.379+0000 7f12c5800700 1 --2- 172.21.15.31:0/3117978344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12c010a3b0 0x7f12c01a6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:39.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.379+0000 7f12c5800700 1 --2- 172.21.15.31:0/3117978344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12c0108810 0x7f12c019ea60 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:39.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.379+0000 7f12c5800700 1 --2- 172.21.15.31:0/3117978344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12c0109a10 0x7f12c019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:39.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.379+0000 7f12c5800700 1 -- 172.21.15.31:0/3117978344 >> 172.21.15.31:0/3117978344 conn(0x7f12c00777e0 msgr2=0x7f12c010cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:39.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.379+0000 7f12c5800700 1 -- 172.21.15.31:0/3117978344 shutdown_connections 2024-03-21T10:37:39.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:39.380+0000 7f12c5800700 1 -- 172.21.15.31:0/3117978344 wait complete. 2024-03-21T10:37:39.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:39 smithi031 ceph-mon[31673]: pgmap v405: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:39.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:39 smithi070 ceph-mon[36330]: pgmap v405: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:39.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:39 smithi040 ceph-mon[36154]: pgmap v405: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:40.431 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-21T10:37:40.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:40 smithi031 ceph-mon[31673]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:37:40.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:40 smithi070 ceph-mon[36330]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:37:40.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:40 smithi040 ceph-mon[36154]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:37:40.940 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:37:41.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:41 smithi031 ceph-mon[31673]: pgmap v406: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:37:41.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:41 smithi070 ceph-mon[36330]: pgmap v406: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:37:41.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:41 smithi040 ceph-mon[36154]: pgmap v406: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:37:43.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.428+0000 7f10cf200700 1 -- 172.21.15.31:0/3937998909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10c8107eb0 msgr2=0x7f10c810a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:43.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.428+0000 7f10cf200700 1 --2- 172.21.15.31:0/3937998909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10c8107eb0 0x7f10c810a320 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f10c000b0a0 tx=0x7f10c000b3b0 comp rx=0 tx=0).stop 2024-03-21T10:37:43.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.428+0000 7f10cf200700 1 -- 172.21.15.31:0/3937998909 shutdown_connections 2024-03-21T10:37:43.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.428+0000 7f10cf200700 1 --2- 172.21.15.31:0/3937998909 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10c8107eb0 0x7f10c810a320 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:43.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.428+0000 7f10cf200700 1 --2- 172.21.15.31:0/3937998909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10c8102c50 0x7f10c8105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:43.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.428+0000 7f10cf200700 1 --2- 172.21.15.31:0/3937998909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10c8105580 0x7f10c8107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:43.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.428+0000 7f10cf200700 1 -- 172.21.15.31:0/3937998909 >> 172.21.15.31:0/3937998909 conn(0x7f10c80fc660 msgr2=0x7f10c80feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:43.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.428+0000 7f10cf200700 1 -- 172.21.15.31:0/3937998909 shutdown_connections 2024-03-21T10:37:43.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.428+0000 7f10cf200700 1 -- 172.21.15.31:0/3937998909 wait complete. 2024-03-21T10:37:43.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.429+0000 7f10cf200700 1 Processor -- start 2024-03-21T10:37:43.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.430+0000 7f10cf200700 1 -- start start 2024-03-21T10:37:43.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.430+0000 7f10cf200700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10c8102c50 0x7f10c819a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:43.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.430+0000 7f10cf200700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10c8105580 0x7f10c819abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:43.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.430+0000 7f10cf200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10c8107eb0 0x7f10c81a1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:43.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.430+0000 7f10cf200700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c810ddf0 con 0x7f10c8102c50 2024-03-21T10:37:43.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.430+0000 7f10cf200700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c810dc70 con 0x7f10c8105580 2024-03-21T10:37:43.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.430+0000 7f10cf200700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c810df70 con 0x7f10c8107eb0 2024-03-21T10:37:43.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.430+0000 7f10c7400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10c8105580 0x7f10c819abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:43.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.430+0000 7f10c7400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10c8105580 0x7f10c819abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:47132/0 (socket says 172.21.15.31:47132) 2024-03-21T10:37:43.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.430+0000 7f10c7400700 1 -- 172.21.15.31:0/2420162090 learned_addr learned my addr 172.21.15.31:0/2420162090 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:43.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.430+0000 7f10c7e00700 1 --2- 172.21.15.31:0/2420162090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10c8102c50 0x7f10c819a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:43.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.431+0000 7f10cd200700 1 --2- 172.21.15.31:0/2420162090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10c8107eb0 0x7f10c81a1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:43.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.431+0000 7f10c7400700 1 -- 172.21.15.31:0/2420162090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10c8107eb0 msgr2=0x7f10c81a1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:43.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.431+0000 7f10c7400700 1 --2- 172.21.15.31:0/2420162090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10c8107eb0 0x7f10c81a1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:43.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.431+0000 7f10c7400700 1 -- 172.21.15.31:0/2420162090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10c8102c50 msgr2=0x7f10c819a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:43.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.431+0000 7f10c7400700 1 --2- 172.21.15.31:0/2420162090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10c8102c50 0x7f10c819a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:43.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.431+0000 7f10c7400700 1 -- 172.21.15.31:0/2420162090 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c81a22d0 con 0x7f10c8105580 2024-03-21T10:37:43.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.431+0000 7f10cd200700 1 --2- 172.21.15.31:0/2420162090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10c8107eb0 0x7f10c81a1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:37:43.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.431+0000 7f10c7e00700 1 --2- 172.21.15.31:0/2420162090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10c8102c50 0x7f10c819a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:37:43.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.431+0000 7f10c7400700 1 --2- 172.21.15.31:0/2420162090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10c8105580 0x7f10c819abb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f10b400e9c0 tx=0x7f10b400ed80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:43.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.431+0000 7f10c4c00700 1 -- 172.21.15.31:0/2420162090 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b400cbd0 con 0x7f10c8105580 2024-03-21T10:37:43.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.432+0000 7f10cf200700 1 -- 172.21.15.31:0/2420162090 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10c81a25c0 con 0x7f10c8105580 2024-03-21T10:37:43.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.432+0000 7f10c4c00700 1 -- 172.21.15.31:0/2420162090 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f10b400cd70 con 0x7f10c8105580 2024-03-21T10:37:43.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.432+0000 7f10c4c00700 1 -- 172.21.15.31:0/2420162090 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b4010430 con 0x7f10c8105580 2024-03-21T10:37:43.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.432+0000 7f10cf200700 1 -- 172.21.15.31:0/2420162090 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10c81a2b00 con 0x7f10c8105580 2024-03-21T10:37:43.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.433+0000 7f10cf200700 1 -- 172.21.15.31:0/2420162090 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10c8068e50 con 0x7f10c8105580 2024-03-21T10:37:43.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.435+0000 7f10c4c00700 1 -- 172.21.15.31:0/2420162090 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f10b4004750 con 0x7f10c8105580 2024-03-21T10:37:43.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.436+0000 7f10c4c00700 1 --2- 172.21.15.31:0/2420162090 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f10ac0a2fb0 0x7f10ac0a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:43.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.436+0000 7f10c4c00700 1 -- 172.21.15.31:0/2420162090 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f10b40cfc10 con 0x7f10c8105580 2024-03-21T10:37:43.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.437+0000 7f10c7e00700 1 --2- 172.21.15.31:0/2420162090 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f10ac0a2fb0 0x7f10ac0a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:43.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.437+0000 7f10c7e00700 1 --2- 172.21.15.31:0/2420162090 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f10ac0a2fb0 0x7f10ac0a5470 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f10b8005fd0 tx=0x7f10b8005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:43.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.441+0000 7f10c4c00700 1 -- 172.21.15.31:0/2420162090 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f10b409c680 con 0x7f10c8105580 2024-03-21T10:37:43.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.675+0000 7f10cf200700 1 -- 172.21.15.31:0/2420162090 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f10c80625d0 con 0x7f10ac0a2fb0 2024-03-21T10:37:43.674 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:43 smithi031 ceph-mon[31673]: pgmap v407: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:37:43.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.678+0000 7f10c4c00700 1 -- 172.21.15.31:0/2420162090 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+187 (secure 0 0 0) 0x7f10c80625d0 con 0x7f10ac0a2fb0 2024-03-21T10:37:43.677 INFO:teuthology.orchestra.run.smithi031.stdout:HOST ADDR LABELS STATUS 2024-03-21T10:37:43.677 INFO:teuthology.orchestra.run.smithi031.stdout:smithi031 172.21.15.31 2024-03-21T10:37:43.677 INFO:teuthology.orchestra.run.smithi031.stdout:smithi040 172.21.15.40 2024-03-21T10:37:43.677 INFO:teuthology.orchestra.run.smithi031.stdout:smithi070 172.21.15.70 2024-03-21T10:37:43.677 INFO:teuthology.orchestra.run.smithi031.stdout:3 hosts in cluster 2024-03-21T10:37:43.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.681+0000 7f10cf200700 1 -- 172.21.15.31:0/2420162090 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f10ac0a2fb0 msgr2=0x7f10ac0a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:43.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.681+0000 7f10cf200700 1 --2- 172.21.15.31:0/2420162090 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f10ac0a2fb0 0x7f10ac0a5470 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f10b8005fd0 tx=0x7f10b8005f60 comp rx=0 tx=0).stop 2024-03-21T10:37:43.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.681+0000 7f10cf200700 1 -- 172.21.15.31:0/2420162090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10c8105580 msgr2=0x7f10c819abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:43.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.681+0000 7f10cf200700 1 --2- 172.21.15.31:0/2420162090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10c8105580 0x7f10c819abb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f10b400e9c0 tx=0x7f10b400ed80 comp rx=0 tx=0).stop 2024-03-21T10:37:43.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.681+0000 7f10cf200700 1 -- 172.21.15.31:0/2420162090 shutdown_connections 2024-03-21T10:37:43.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.681+0000 7f10cf200700 1 --2- 172.21.15.31:0/2420162090 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f10ac0a2fb0 0x7f10ac0a5470 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:43.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.681+0000 7f10cf200700 1 --2- 172.21.15.31:0/2420162090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10c8107eb0 0x7f10c81a1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:43.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.681+0000 7f10cf200700 1 --2- 172.21.15.31:0/2420162090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10c8102c50 0x7f10c819a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:43.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.681+0000 7f10cf200700 1 --2- 172.21.15.31:0/2420162090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10c8105580 0x7f10c819abb0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:43.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.681+0000 7f10cf200700 1 -- 172.21.15.31:0/2420162090 >> 172.21.15.31:0/2420162090 conn(0x7f10c80fc660 msgr2=0x7f10c8102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:43.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.681+0000 7f10cf200700 1 -- 172.21.15.31:0/2420162090 shutdown_connections 2024-03-21T10:37:43.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:43.681+0000 7f10cf200700 1 -- 172.21.15.31:0/2420162090 wait complete. 2024-03-21T10:37:43.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:43 smithi070 ceph-mon[36330]: pgmap v407: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:37:43.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:43 smithi040 ceph-mon[36154]: pgmap v407: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:37:44.564 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-21T10:37:44.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:44 smithi031 ceph-mon[31673]: from='client.25232 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:37:44.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:44 smithi070 ceph-mon[36330]: from='client.25232 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:37:44.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:44 smithi040 ceph-mon[36154]: from='client.25232 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:37:45.053 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:37:45.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:45 smithi031 ceph-mon[31673]: pgmap v408: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:45.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:45 smithi070 ceph-mon[36330]: pgmap v408: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:45.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:45 smithi040 ceph-mon[36154]: pgmap v408: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:47.247 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:47 smithi031 ceph-mon[31673]: pgmap v409: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:37:47.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:47 smithi070 ceph-mon[36330]: pgmap v409: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:37:47.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.421+0000 7fc47e400700 1 -- 172.21.15.31:0/3946605241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc478102b80 msgr2=0x7fc478104f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.421+0000 7fc47e400700 1 --2- 172.21.15.31:0/3946605241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc478102b80 0x7fc478104f70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc46c009990 tx=0x7fc46c009ca0 comp rx=0 tx=0).stop 2024-03-21T10:37:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.421+0000 7fc47e400700 1 -- 172.21.15.31:0/3946605241 shutdown_connections 2024-03-21T10:37:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.421+0000 7fc47e400700 1 --2- 172.21.15.31:0/3946605241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc478107de0 0x7fc47810a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.421+0000 7fc47e400700 1 --2- 172.21.15.31:0/3946605241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc478102b80 0x7fc478104f70 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.421+0000 7fc47e400700 1 --2- 172.21.15.31:0/3946605241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4781054b0 0x7fc4781078a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.421+0000 7fc47e400700 1 -- 172.21.15.31:0/3946605241 >> 172.21.15.31:0/3946605241 conn(0x7fc4780fc5b0 msgr2=0x7fc4780fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.421+0000 7fc47e400700 1 -- 172.21.15.31:0/3946605241 shutdown_connections 2024-03-21T10:37:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.421+0000 7fc47e400700 1 -- 172.21.15.31:0/3946605241 wait complete. 2024-03-21T10:37:47.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.423+0000 7fc47e400700 1 Processor -- start 2024-03-21T10:37:47.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.424+0000 7fc47e400700 1 -- start start 2024-03-21T10:37:47.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.424+0000 7fc47e400700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc478102b80 0x7fc4781024e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:47.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.424+0000 7fc47e400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4781054b0 0x7fc478100b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:47.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.424+0000 7fc47e400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc478107de0 0x7fc478101070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:47.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.424+0000 7fc47e400700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47810dd50 con 0x7fc478102b80 2024-03-21T10:37:47.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.424+0000 7fc47e400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47810dbd0 con 0x7fc4781054b0 2024-03-21T10:37:47.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.424+0000 7fc47e400700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47810ded0 con 0x7fc478107de0 2024-03-21T10:37:47.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.424+0000 7fc477e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4781054b0 0x7fc478100b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:47.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.425+0000 7fc477e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4781054b0 0x7fc478100b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:40052/0 (socket says 172.21.15.31:40052) 2024-03-21T10:37:47.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.425+0000 7fc477e00700 1 -- 172.21.15.31:0/3463304015 learned_addr learned my addr 172.21.15.31:0/3463304015 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:47.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.425+0000 7fc47d000700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc478102b80 0x7fc4781024e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:47.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.425+0000 7fc47da00700 1 --2- 172.21.15.31:0/3463304015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc478107de0 0x7fc478101070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:47.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.425+0000 7fc477e00700 1 -- 172.21.15.31:0/3463304015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc478107de0 msgr2=0x7fc478101070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:47.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.425+0000 7fc477e00700 1 --2- 172.21.15.31:0/3463304015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc478107de0 0x7fc478101070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:47.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.425+0000 7fc477e00700 1 -- 172.21.15.31:0/3463304015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc478102b80 msgr2=0x7fc4781024e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:47.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.425+0000 7fc477e00700 1 --2- 172.21.15.31:0/3463304015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc478102b80 0x7fc4781024e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:47.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.425+0000 7fc477e00700 1 -- 172.21.15.31:0/3463304015 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc478101920 con 0x7fc4781054b0 2024-03-21T10:37:47.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.425+0000 7fc477e00700 1 --2- 172.21.15.31:0/3463304015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4781054b0 0x7fc478100b30 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc46400e9a0 tx=0x7fc46400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:47.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.426+0000 7fc475600700 1 -- 172.21.15.31:0/3463304015 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46400ca10 con 0x7fc4781054b0 2024-03-21T10:37:47.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.426+0000 7fc475600700 1 -- 172.21.15.31:0/3463304015 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc46400cbb0 con 0x7fc4781054b0 2024-03-21T10:37:47.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.426+0000 7fc475600700 1 -- 172.21.15.31:0/3463304015 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc464018680 con 0x7fc4781054b0 2024-03-21T10:37:47.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.426+0000 7fc47e400700 1 -- 172.21.15.31:0/3463304015 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4781a2490 con 0x7fc4781054b0 2024-03-21T10:37:47.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.426+0000 7fc47e400700 1 -- 172.21.15.31:0/3463304015 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4781a2950 con 0x7fc4781054b0 2024-03-21T10:37:47.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.427+0000 7fc47e400700 1 -- 172.21.15.31:0/3463304015 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc478068e50 con 0x7fc4781054b0 2024-03-21T10:37:47.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.430+0000 7fc475600700 1 -- 172.21.15.31:0/3463304015 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fc464010430 con 0x7fc4781054b0 2024-03-21T10:37:47.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.432+0000 7fc475600700 1 --2- 172.21.15.31:0/3463304015 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc4600a3030 0x7fc4600a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:47.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.432+0000 7fc475600700 1 -- 172.21.15.31:0/3463304015 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fc4640cf930 con 0x7fc4781054b0 2024-03-21T10:37:47.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.432+0000 7fc47d000700 1 --2- 172.21.15.31:0/3463304015 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc4600a3030 0x7fc4600a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:47.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.433+0000 7fc47d000700 1 --2- 172.21.15.31:0/3463304015 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc4600a3030 0x7fc4600a54f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc46c00b2e0 tx=0x7fc46c00b880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:47.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:47 smithi040 ceph-mon[36154]: pgmap v409: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:37:47.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.436+0000 7fc475600700 1 -- 172.21.15.31:0/3463304015 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc46409c3a0 con 0x7fc4781054b0 2024-03-21T10:37:47.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.690+0000 7fc47e400700 1 -- 172.21.15.31:0/3463304015 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc47802de40 con 0x7fc4600a3030 2024-03-21T10:37:47.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.693+0000 7fc475600700 1 -- 172.21.15.31:0/3463304015 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7fc47802de40 con 0x7fc4600a3030 2024-03-21T10:37:47.691 INFO:teuthology.orchestra.run.smithi031.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-21T10:37:47.691 INFO:teuthology.orchestra.run.smithi031.stdout:smithi031 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT53310008400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-03-21T10:37:47.691 INFO:teuthology.orchestra.run.smithi031.stdout:smithi040 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330009J400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-03-21T10:37:47.691 INFO:teuthology.orchestra.run.smithi031.stdout:smithi070 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513002P480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-03-21T10:37:47.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.696+0000 7fc47e400700 1 -- 172.21.15.31:0/3463304015 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc4600a3030 msgr2=0x7fc4600a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:47.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.696+0000 7fc47e400700 1 --2- 172.21.15.31:0/3463304015 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc4600a3030 0x7fc4600a54f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc46c00b2e0 tx=0x7fc46c00b880 comp rx=0 tx=0).stop 2024-03-21T10:37:47.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.696+0000 7fc47e400700 1 -- 172.21.15.31:0/3463304015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4781054b0 msgr2=0x7fc478100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:47.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.696+0000 7fc47e400700 1 --2- 172.21.15.31:0/3463304015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4781054b0 0x7fc478100b30 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc46400e9a0 tx=0x7fc46400ecb0 comp rx=0 tx=0).stop 2024-03-21T10:37:47.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.696+0000 7fc47e400700 1 -- 172.21.15.31:0/3463304015 shutdown_connections 2024-03-21T10:37:47.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.696+0000 7fc47e400700 1 --2- 172.21.15.31:0/3463304015 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc4600a3030 0x7fc4600a54f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:47.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.696+0000 7fc47e400700 1 --2- 172.21.15.31:0/3463304015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc478107de0 0x7fc478101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:47.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.696+0000 7fc47e400700 1 --2- 172.21.15.31:0/3463304015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc478102b80 0x7fc4781024e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:47.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.696+0000 7fc47e400700 1 --2- 172.21.15.31:0/3463304015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4781054b0 0x7fc478100b30 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:47.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.696+0000 7fc47e400700 1 -- 172.21.15.31:0/3463304015 >> 172.21.15.31:0/3463304015 conn(0x7fc4780fc5b0 msgr2=0x7fc4781034c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:47.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.697+0000 7fc47e400700 1 -- 172.21.15.31:0/3463304015 shutdown_connections 2024-03-21T10:37:47.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:47.697+0000 7fc47e400700 1 -- 172.21.15.31:0/3463304015 wait complete. 2024-03-21T10:37:48.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:48 smithi031 ceph-mon[31673]: from='client.25238 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:37:48.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:48 smithi070 ceph-mon[36330]: from='client.25238 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:37:48.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:48 smithi040 ceph-mon[36154]: from='client.25238 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:37:48.560 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-21T10:37:48.573 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2024-03-21T10:37:48.573 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-21T10:37:49.058 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:37:49.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:49 smithi070 ceph-mon[36330]: pgmap v410: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:49.331 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:49 smithi040 ceph-mon[36154]: pgmap v410: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:49.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:49 smithi031 ceph-mon[31673]: pgmap v410: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:51.217 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:51 smithi031 ceph-mon[31673]: pgmap v411: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:37:51.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:51 smithi040 ceph-mon[36154]: pgmap v411: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:37:51.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:51 smithi070 ceph-mon[36330]: pgmap v411: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:37:51.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.586+0000 7fc910e00700 1 -- 172.21.15.31:0/1947984288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc90c105610 msgr2=0x7fc90c105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:51.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.586+0000 7fc910e00700 1 --2- 172.21.15.31:0/1947984288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc90c105610 0x7fc90c105a70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc8f80099c0 tx=0x7fc8f8009cd0 comp rx=0 tx=0).stop 2024-03-21T10:37:51.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.587+0000 7fc910e00700 1 -- 172.21.15.31:0/1947984288 shutdown_connections 2024-03-21T10:37:51.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.587+0000 7fc910e00700 1 --2- 172.21.15.31:0/1947984288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc90c105fb0 0x7fc90c10c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:51.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.587+0000 7fc910e00700 1 --2- 172.21.15.31:0/1947984288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc90c104410 0x7fc90c104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:51.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.587+0000 7fc910e00700 1 --2- 172.21.15.31:0/1947984288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc90c105610 0x7fc90c105a70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:51.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.587+0000 7fc910e00700 1 -- 172.21.15.31:0/1947984288 >> 172.21.15.31:0/1947984288 conn(0x7fc90c0ff9b0 msgr2=0x7fc90c101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:51.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.587+0000 7fc910e00700 1 -- 172.21.15.31:0/1947984288 shutdown_connections 2024-03-21T10:37:51.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.587+0000 7fc910e00700 1 -- 172.21.15.31:0/1947984288 wait complete. 2024-03-21T10:37:51.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.588+0000 7fc910e00700 1 Processor -- start 2024-03-21T10:37:51.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.588+0000 7fc910e00700 1 -- start start 2024-03-21T10:37:51.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.588+0000 7fc910e00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc90c104410 0x7fc90c19a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:51.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.588+0000 7fc910e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc90c105610 0x7fc90c19ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:51.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.588+0000 7fc910e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc90c105fb0 0x7fc90c1a1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:51.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.588+0000 7fc910e00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c110030 con 0x7fc90c104410 2024-03-21T10:37:51.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.588+0000 7fc910e00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c10feb0 con 0x7fc90c105610 2024-03-21T10:37:51.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.588+0000 7fc910e00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c1101b0 con 0x7fc90c105fb0 2024-03-21T10:37:51.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.589+0000 7fc909400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc90c105610 0x7fc90c19ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:51.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.589+0000 7fc909400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc90c105610 0x7fc90c19ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:40074/0 (socket says 172.21.15.31:40074) 2024-03-21T10:37:51.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.589+0000 7fc909400700 1 -- 172.21.15.31:0/2691750181 learned_addr learned my addr 172.21.15.31:0/2691750181 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:51.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.589+0000 7fc90a800700 1 --2- 172.21.15.31:0/2691750181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc90c105fb0 0x7fc90c1a1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:51.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.589+0000 7fc909e00700 1 --2- 172.21.15.31:0/2691750181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc90c104410 0x7fc90c19a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:51.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.589+0000 7fc90a800700 1 -- 172.21.15.31:0/2691750181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc90c105610 msgr2=0x7fc90c19ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:51.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.589+0000 7fc90a800700 1 --2- 172.21.15.31:0/2691750181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc90c105610 0x7fc90c19ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:51.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.589+0000 7fc90a800700 1 -- 172.21.15.31:0/2691750181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc90c104410 msgr2=0x7fc90c19a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:51.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.589+0000 7fc90a800700 1 --2- 172.21.15.31:0/2691750181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc90c104410 0x7fc90c19a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:51.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.589+0000 7fc90a800700 1 -- 172.21.15.31:0/2691750181 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc90c1a2240 con 0x7fc90c105fb0 2024-03-21T10:37:51.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.589+0000 7fc909e00700 1 --2- 172.21.15.31:0/2691750181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc90c104410 0x7fc90c19a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:37:51.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.589+0000 7fc909400700 1 --2- 172.21.15.31:0/2691750181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc90c105610 0x7fc90c19ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:37:51.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.589+0000 7fc90a800700 1 --2- 172.21.15.31:0/2691750181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc90c105fb0 0x7fc90c1a1ba0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc904009f60 tx=0x7fc904009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:51.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.590+0000 7fc902a00700 1 -- 172.21.15.31:0/2691750181 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9040092e0 con 0x7fc90c105fb0 2024-03-21T10:37:51.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.590+0000 7fc910e00700 1 -- 172.21.15.31:0/2691750181 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc90c1a24d0 con 0x7fc90c105fb0 2024-03-21T10:37:51.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.590+0000 7fc902a00700 1 -- 172.21.15.31:0/2691750181 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc904004bb0 con 0x7fc90c105fb0 2024-03-21T10:37:51.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.590+0000 7fc902a00700 1 -- 172.21.15.31:0/2691750181 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc904004280 con 0x7fc90c105fb0 2024-03-21T10:37:51.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.590+0000 7fc910e00700 1 -- 172.21.15.31:0/2691750181 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc90c1a2a10 con 0x7fc90c105fb0 2024-03-21T10:37:51.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.591+0000 7fc910e00700 1 -- 172.21.15.31:0/2691750181 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc90c068e50 con 0x7fc90c105fb0 2024-03-21T10:37:51.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.594+0000 7fc902a00700 1 -- 172.21.15.31:0/2691750181 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fc90401b030 con 0x7fc90c105fb0 2024-03-21T10:37:51.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.596+0000 7fc902a00700 1 --2- 172.21.15.31:0/2691750181 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc8f00a2fb0 0x7fc8f00a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:51.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.596+0000 7fc902a00700 1 -- 172.21.15.31:0/2691750181 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fc9040cf270 con 0x7fc90c105fb0 2024-03-21T10:37:51.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.596+0000 7fc909e00700 1 --2- 172.21.15.31:0/2691750181 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc8f00a2fb0 0x7fc8f00a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:51.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.597+0000 7fc909e00700 1 --2- 172.21.15.31:0/2691750181 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc8f00a2fb0 0x7fc8f00a5470 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc90c105470 tx=0x7fc8fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:51.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.600+0000 7fc902a00700 1 -- 172.21.15.31:0/2691750181 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc904022020 con 0x7fc90c105fb0 2024-03-21T10:37:51.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.877+0000 7fc910e00700 1 -- 172.21.15.31:0/2691750181 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc90c062da0 con 0x7fc90c105fb0 2024-03-21T10:37:51.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.878+0000 7fc902a00700 1 -- 172.21.15.31:0/2691750181 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) v1 ==== 76+0+2988 (secure 0 0 0) 0x7fc904010040 con 0x7fc90c105fb0 2024-03-21T10:37:51.877 INFO:teuthology.orchestra.run.smithi031.stdout:e23 2024-03-21T10:37:51.877 INFO:teuthology.orchestra.run.smithi031.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-21T10:37:51.877 INFO:teuthology.orchestra.run.smithi031.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-21T10:37:51.877 INFO:teuthology.orchestra.run.smithi031.stdout:legacy client fscid: 1 2024-03-21T10:37:51.877 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:37:51.877 INFO:teuthology.orchestra.run.smithi031.stdout:Filesystem 'cephfs' (1) 2024-03-21T10:37:51.877 INFO:teuthology.orchestra.run.smithi031.stdout:fs_name cephfs 2024-03-21T10:37:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 23 2024-03-21T10:37:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-21T10:37:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:created 2024-03-21T10:35:06.492115+0000 2024-03-21T10:37:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:modified 2024-03-21T10:35:40.990205+0000 2024-03-21T10:37:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:tableserver 0 2024-03-21T10:37:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:root 0 2024-03-21T10:37:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:session_timeout 300 2024-03-21T10:37:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:session_autoclose 300 2024-03-21T10:37:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:max_file_size 1099511627776 2024-03-21T10:37:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:max_xattr_size 65536 2024-03-21T10:37:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:required_client_features {} 2024-03-21T10:37:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:last_failure 0 2024-03-21T10:37:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:last_failure_osd_epoch 0 2024-03-21T10:37:51.878 INFO:teuthology.orchestra.run.smithi031.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-21T10:37:51.879 INFO:teuthology.orchestra.run.smithi031.stdout:max_mds 3 2024-03-21T10:37:51.879 INFO:teuthology.orchestra.run.smithi031.stdout:in 0,1,2 2024-03-21T10:37:51.879 INFO:teuthology.orchestra.run.smithi031.stdout:up {0=24596,1=24584,2=24547} 2024-03-21T10:37:51.879 INFO:teuthology.orchestra.run.smithi031.stdout:failed 2024-03-21T10:37:51.879 INFO:teuthology.orchestra.run.smithi031.stdout:damaged 2024-03-21T10:37:51.879 INFO:teuthology.orchestra.run.smithi031.stdout:stopped 2024-03-21T10:37:51.879 INFO:teuthology.orchestra.run.smithi031.stdout:data_pools [3,4] 2024-03-21T10:37:51.879 INFO:teuthology.orchestra.run.smithi031.stdout:metadata_pool 2 2024-03-21T10:37:51.879 INFO:teuthology.orchestra.run.smithi031.stdout:inline_data disabled 2024-03-21T10:37:51.879 INFO:teuthology.orchestra.run.smithi031.stdout:balancer 2024-03-21T10:37:51.879 INFO:teuthology.orchestra.run.smithi031.stdout:bal_rank_mask -1 2024-03-21T10:37:51.879 INFO:teuthology.orchestra.run.smithi031.stdout:standby_count_wanted 0 2024-03-21T10:37:51.879 INFO:teuthology.orchestra.run.smithi031.stdout:qdb_cluster leader: 24596 members: 24584,24596,24547 2024-03-21T10:37:51.879 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.b{0:24596} state up:active seq 3 addr [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] compat {c=[1],r=[1],i=[fff]}] 2024-03-21T10:37:51.879 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.h{0:24539} state up:standby-replay seq 1 addr [v2:172.21.15.40:6836/4121707657,v1:172.21.15.40:6837/4121707657] compat {c=[1],r=[1],i=[fff]}] 2024-03-21T10:37:51.880 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.a{1:24584} state up:active seq 12 addr [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] compat {c=[1],r=[1],i=[fff]}] 2024-03-21T10:37:51.880 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.c{1:24523} state up:standby-replay seq 1 addr [v2:172.21.15.70:6836/3650805915,v1:172.21.15.70:6837/3650805915] compat {c=[1],r=[1],i=[fff]}] 2024-03-21T10:37:51.880 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.i{2:24547} state up:active seq 14 addr [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] compat {c=[1],r=[1],i=[fff]}] 2024-03-21T10:37:51.880 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.f{2:24511} state up:standby-replay seq 1 addr [v2:172.21.15.70:6834/729664889,v1:172.21.15.70:6835/729664889] compat {c=[1],r=[1],i=[fff]}] 2024-03-21T10:37:51.880 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:37:51.880 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:37:51.880 INFO:teuthology.orchestra.run.smithi031.stdout:Standby daemons: 2024-03-21T10:37:51.880 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:37:51.880 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.j{-1:14385} state up:standby seq 1 addr [v2:172.21.15.31:6834/2099524212,v1:172.21.15.31:6835/2099524212] compat {c=[1],r=[1],i=[fff]}] 2024-03-21T10:37:51.880 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.d{-1:14403} state up:standby seq 1 addr [v2:172.21.15.31:6836/1193249174,v1:172.21.15.31:6837/1193249174] compat {c=[1],r=[1],i=[fff]}] 2024-03-21T10:37:51.880 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.g{-1:14451} state up:standby seq 1 addr [v2:172.21.15.31:6838/4261892640,v1:172.21.15.31:6839/4261892640] compat {c=[1],r=[1],i=[fff]}] 2024-03-21T10:37:51.880 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.l{-1:24442} state up:standby seq 1 addr [v2:172.21.15.70:6832/4158670810,v1:172.21.15.70:6833/4158670810] compat {c=[1],r=[1],i=[fff]}] 2024-03-21T10:37:51.880 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.k{-1:24458} state up:standby seq 1 addr [v2:172.21.15.40:6832/2806510034,v1:172.21.15.40:6833/2806510034] compat {c=[1],r=[1],i=[fff]}] 2024-03-21T10:37:51.880 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.e{-1:24506} state up:standby seq 1 addr [v2:172.21.15.40:6834/2081017532,v1:172.21.15.40:6835/2081017532] compat {c=[1],r=[1],i=[fff]}] 2024-03-21T10:37:51.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.881+0000 7fc910e00700 1 -- 172.21.15.31:0/2691750181 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc8f00a2fb0 msgr2=0x7fc8f00a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:51.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.881+0000 7fc910e00700 1 --2- 172.21.15.31:0/2691750181 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc8f00a2fb0 0x7fc8f00a5470 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc90c105470 tx=0x7fc8fc009260 comp rx=0 tx=0).stop 2024-03-21T10:37:51.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.881+0000 7fc910e00700 1 -- 172.21.15.31:0/2691750181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc90c105fb0 msgr2=0x7fc90c1a1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:51.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.881+0000 7fc910e00700 1 --2- 172.21.15.31:0/2691750181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc90c105fb0 0x7fc90c1a1ba0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc904009f60 tx=0x7fc904009f90 comp rx=0 tx=0).stop 2024-03-21T10:37:51.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.882+0000 7fc910e00700 1 -- 172.21.15.31:0/2691750181 shutdown_connections 2024-03-21T10:37:51.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.882+0000 7fc910e00700 1 --2- 172.21.15.31:0/2691750181 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fc8f00a2fb0 0x7fc8f00a5470 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:51.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.882+0000 7fc910e00700 1 --2- 172.21.15.31:0/2691750181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc90c105fb0 0x7fc90c1a1ba0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:51.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.882+0000 7fc910e00700 1 --2- 172.21.15.31:0/2691750181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc90c104410 0x7fc90c19a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:51.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.882+0000 7fc910e00700 1 --2- 172.21.15.31:0/2691750181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc90c105610 0x7fc90c19ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:51.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.882+0000 7fc910e00700 1 -- 172.21.15.31:0/2691750181 >> 172.21.15.31:0/2691750181 conn(0x7fc90c0ff9b0 msgr2=0x7fc90c108910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:51.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.882+0000 7fc910e00700 1 -- 172.21.15.31:0/2691750181 shutdown_connections 2024-03-21T10:37:51.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:51.882+0000 7fc910e00700 1 -- 172.21.15.31:0/2691750181 wait complete. 2024-03-21T10:37:51.882 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2024-03-21T10:37:52.272 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:52 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2691750181' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-21T10:37:52.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:52 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2691750181' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-21T10:37:52.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:52 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2691750181' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-21T10:37:53.013 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-21T10:37:53.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:53 smithi031 ceph-mon[31673]: pgmap v412: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:37:53.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:53 smithi040 ceph-mon[36154]: pgmap v412: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:37:53.480 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:37:53.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:53 smithi070 ceph-mon[36330]: pgmap v412: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:37:55.421 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:55 smithi031 ceph-mon[31673]: pgmap v413: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:55.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:55 smithi040 ceph-mon[36154]: pgmap v413: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:55.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:55 smithi070 ceph-mon[36330]: pgmap v413: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:56.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.077+0000 7f1df0c00700 1 -- 172.21.15.31:0/193823740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dec1033e0 msgr2=0x7f1dec103840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:56.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.077+0000 7f1df0c00700 1 --2- 172.21.15.31:0/193823740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dec1033e0 0x7f1dec103840 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1dd80099c0 tx=0x7f1dd8009cd0 comp rx=0 tx=0).stop 2024-03-21T10:37:56.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.077+0000 7f1df0c00700 1 -- 172.21.15.31:0/193823740 shutdown_connections 2024-03-21T10:37:56.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.077+0000 7f1df0c00700 1 --2- 172.21.15.31:0/193823740 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dec103d80 0x7f1dec10a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:56.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.077+0000 7f1df0c00700 1 --2- 172.21.15.31:0/193823740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dec1021e0 0x7f1dec102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:56.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.077+0000 7f1df0c00700 1 --2- 172.21.15.31:0/193823740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dec1033e0 0x7f1dec103840 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:56.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.077+0000 7f1df0c00700 1 -- 172.21.15.31:0/193823740 >> 172.21.15.31:0/193823740 conn(0x7f1dec0fd740 msgr2=0x7f1dec0ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:56.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.077+0000 7f1df0c00700 1 -- 172.21.15.31:0/193823740 shutdown_connections 2024-03-21T10:37:56.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.077+0000 7f1df0c00700 1 -- 172.21.15.31:0/193823740 wait complete. 2024-03-21T10:37:56.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.078+0000 7f1df0c00700 1 Processor -- start 2024-03-21T10:37:56.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.078+0000 7f1df0c00700 1 -- start start 2024-03-21T10:37:56.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.079+0000 7f1df0c00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dec1021e0 0x7f1dec19a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:56.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.079+0000 7f1df0c00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dec1033e0 0x7f1dec19ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:56.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.079+0000 7f1df0c00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dec103d80 0x7f1dec1a1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:56.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.079+0000 7f1df0c00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec10de00 con 0x7f1dec103d80 2024-03-21T10:37:56.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.079+0000 7f1df0c00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec10dc80 con 0x7f1dec1033e0 2024-03-21T10:37:56.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.079+0000 7f1df0c00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec10df80 con 0x7f1dec1021e0 2024-03-21T10:37:56.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.079+0000 7f1de9e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dec1021e0 0x7f1dec19a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:56.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.079+0000 7f1de9e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dec1021e0 0x7f1dec19a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:46932/0 (socket says 172.21.15.31:46932) 2024-03-21T10:37:56.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.079+0000 7f1de9e00700 1 -- 172.21.15.31:0/1524261185 learned_addr learned my addr 172.21.15.31:0/1524261185 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:56.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.079+0000 7f1de9400700 1 --2- 172.21.15.31:0/1524261185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dec1033e0 0x7f1dec19ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:56.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.079+0000 7f1dea800700 1 --2- 172.21.15.31:0/1524261185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dec103d80 0x7f1dec1a1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:56.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.080+0000 7f1de9e00700 1 -- 172.21.15.31:0/1524261185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dec1033e0 msgr2=0x7f1dec19ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:56.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.080+0000 7f1de9e00700 1 --2- 172.21.15.31:0/1524261185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dec1033e0 0x7f1dec19ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:56.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.080+0000 7f1de9e00700 1 -- 172.21.15.31:0/1524261185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dec103d80 msgr2=0x7f1dec1a1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:56.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.080+0000 7f1de9e00700 1 --2- 172.21.15.31:0/1524261185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dec103d80 0x7f1dec1a1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:56.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.080+0000 7f1de9e00700 1 -- 172.21.15.31:0/1524261185 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dec1a2280 con 0x7f1dec1021e0 2024-03-21T10:37:56.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.080+0000 7f1dea800700 1 --2- 172.21.15.31:0/1524261185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dec103d80 0x7f1dec1a1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:37:56.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.080+0000 7f1de9e00700 1 --2- 172.21.15.31:0/1524261185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dec1021e0 0x7f1dec19a620 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc00e9a0 tx=0x7f1ddc00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:56.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.080+0000 7f1de2a00700 1 -- 172.21.15.31:0/1524261185 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc00cbd0 con 0x7f1dec1021e0 2024-03-21T10:37:56.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.080+0000 7f1de2a00700 1 -- 172.21.15.31:0/1524261185 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ddc004500 con 0x7f1dec1021e0 2024-03-21T10:37:56.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.080+0000 7f1df0c00700 1 -- 172.21.15.31:0/1524261185 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dec1a2570 con 0x7f1dec1021e0 2024-03-21T10:37:56.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.080+0000 7f1de2a00700 1 -- 172.21.15.31:0/1524261185 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc003e00 con 0x7f1dec1021e0 2024-03-21T10:37:56.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.080+0000 7f1df0c00700 1 -- 172.21.15.31:0/1524261185 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dec1a2b00 con 0x7f1dec1021e0 2024-03-21T10:37:56.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.082+0000 7f1df0c00700 1 -- 172.21.15.31:0/1524261185 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dec068e50 con 0x7f1dec1021e0 2024-03-21T10:37:56.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.085+0000 7f1de2a00700 1 -- 172.21.15.31:0/1524261185 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f1ddc003fe0 con 0x7f1dec1021e0 2024-03-21T10:37:56.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.086+0000 7f1de2a00700 1 --2- 172.21.15.31:0/1524261185 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1dd00a3030 0x7f1dd00a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:56.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.087+0000 7f1de2a00700 1 -- 172.21.15.31:0/1524261185 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f1ddc0cfa00 con 0x7f1dec1021e0 2024-03-21T10:37:56.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.087+0000 7f1de9400700 1 --2- 172.21.15.31:0/1524261185 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1dd00a3030 0x7f1dd00a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:56.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.087+0000 7f1de9400700 1 --2- 172.21.15.31:0/1524261185 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1dd00a3030 0x7f1dd00a54f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8000c00 tx=0x7f1dd8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:56.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.090+0000 7f1de2a00700 1 -- 172.21.15.31:0/1524261185 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1ddc09c470 con 0x7f1dec1021e0 2024-03-21T10:37:56.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.325+0000 7f1df0c00700 1 -- 172.21.15.31:0/1524261185 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f1dec1a3540 con 0x7f1dec1021e0 2024-03-21T10:37:56.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.327+0000 7f1de2a00700 1 -- 172.21.15.31:0/1524261185 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) v1 ==== 56+0+5566 (secure 0 0 0) 0x7f1ddc09bd70 con 0x7f1dec1021e0 2024-03-21T10:37:56.325 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 93 2024-03-21T10:37:56.325 INFO:teuthology.orchestra.run.smithi031.stdout:fsid 3847408c-e76d-11ee-95cd-87774f69a715 2024-03-21T10:37:56.325 INFO:teuthology.orchestra.run.smithi031.stdout:created 2024-03-21T10:25:41.971117+0000 2024-03-21T10:37:56.325 INFO:teuthology.orchestra.run.smithi031.stdout:modified 2024-03-21T10:36:56.301695+0000 2024-03-21T10:37:56.326 INFO:teuthology.orchestra.run.smithi031.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-21T10:37:56.326 INFO:teuthology.orchestra.run.smithi031.stdout:crush_version 27 2024-03-21T10:37:56.326 INFO:teuthology.orchestra.run.smithi031.stdout:full_ratio 0.95 2024-03-21T10:37:56.326 INFO:teuthology.orchestra.run.smithi031.stdout:backfillfull_ratio 0.9 2024-03-21T10:37:56.326 INFO:teuthology.orchestra.run.smithi031.stdout:nearfull_ratio 0.85 2024-03-21T10:37:56.326 INFO:teuthology.orchestra.run.smithi031.stdout:require_min_compat_client luminous 2024-03-21T10:37:56.326 INFO:teuthology.orchestra.run.smithi031.stdout:min_compat_client luminous 2024-03-21T10:37:56.326 INFO:teuthology.orchestra.run.smithi031.stdout:require_osd_release squid 2024-03-21T10:37:56.326 INFO:teuthology.orchestra.run.smithi031.stdout:stretch_mode_enabled false 2024-03-21T10:37:56.326 INFO:teuthology.orchestra.run.smithi031.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-21T10:37:56.326 INFO:teuthology.orchestra.run.smithi031.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-21T10:37:56.326 INFO:teuthology.orchestra.run.smithi031.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-21T10:37:56.326 INFO:teuthology.orchestra.run.smithi031.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-21T10:37:56.327 INFO:teuthology.orchestra.run.smithi031.stdout:max_osd 12 2024-03-21T10:37:56.327 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 up in weight 1 up_from 8 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6802/3089885125,v1:172.21.15.31:6803/3089885125] [v2:172.21.15.31:6804/3089885125,v1:172.21.15.31:6805/3089885125] exists,up e0244d33-00e2-47d5-9f66-34efc138c462 2024-03-21T10:37:56.327 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 up in weight 1 up_from 13 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6800/618840037,v1:172.21.15.40:6801/618840037] [v2:172.21.15.40:6802/618840037,v1:172.21.15.40:6803/618840037] exists,up 76a63504-beab-4354-a6c3-eebc3b16ad03 2024-03-21T10:37:56.327 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 up in weight 1 up_from 18 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.70:6800/2524184117,v1:172.21.15.70:6801/2524184117] [v2:172.21.15.70:6802/2524184117,v1:172.21.15.70:6803/2524184117] exists,up fa191bd6-9a21-4ee2-8adf-8a63c9cfd821 2024-03-21T10:37:56.327 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6810/2823036296,v1:172.21.15.31:6811/2823036296] [v2:172.21.15.31:6812/2823036296,v1:172.21.15.31:6813/2823036296] exists,up edb44507-ffb0-42a4-827b-08461e1bf952 2024-03-21T10:37:56.327 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 up in weight 1 up_from 33 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6808/4215186309,v1:172.21.15.40:6809/4215186309] [v2:172.21.15.40:6810/4215186309,v1:172.21.15.40:6811/4215186309] exists,up 8cb89153-8c3b-431f-b57c-8863fada76dc 2024-03-21T10:37:56.327 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 up in weight 1 up_from 38 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.70:6808/3402830452,v1:172.21.15.70:6809/3402830452] [v2:172.21.15.70:6810/3402830452,v1:172.21.15.70:6811/3402830452] exists,up 9409505a-d463-408c-a3cd-218380c04d68 2024-03-21T10:37:56.327 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 up in weight 1 up_from 46 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6818/1193497058,v1:172.21.15.31:6819/1193497058] [v2:172.21.15.31:6820/1193497058,v1:172.21.15.31:6821/1193497058] exists,up f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50 2024-03-21T10:37:56.327 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 up in weight 1 up_from 51 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6816/3825628119,v1:172.21.15.40:6817/3825628119] [v2:172.21.15.40:6818/3825628119,v1:172.21.15.40:6819/3825628119] exists,up db228a87-29a5-45d4-8fe6-a0a554262eb7 2024-03-21T10:37:56.327 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8 up in weight 1 up_from 59 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.70:6816/534880586,v1:172.21.15.70:6817/534880586] [v2:172.21.15.70:6818/534880586,v1:172.21.15.70:6819/534880586] exists,up 59b9fde0-095d-4e0b-87da-ead37bd8ff1b 2024-03-21T10:37:56.327 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9 up in weight 1 up_from 67 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6826/587047469,v1:172.21.15.31:6827/587047469] [v2:172.21.15.31:6828/587047469,v1:172.21.15.31:6829/587047469] exists,up 77f47061-0c5f-4159-b276-e814935d6c1c 2024-03-21T10:37:56.327 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10 up in weight 1 up_from 72 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6824/512108711,v1:172.21.15.40:6825/512108711] [v2:172.21.15.40:6826/512108711,v1:172.21.15.40:6827/512108711] exists,up 8853dc43-7bd1-4c93-8063-f7982fd7da38 2024-03-21T10:37:56.327 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11 up in weight 1 up_from 80 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.70:6824/540305718,v1:172.21.15.70:6825/540305718] [v2:172.21.15.70:6826/540305718,v1:172.21.15.70:6827/540305718] exists,up 84b7c2de-c424-45a6-9583-60097e372a86 2024-03-21T10:37:56.327 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 2.8 [2,11] 2024-03-21T10:37:56.328 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 2.b [8,11] 2024-03-21T10:37:56.328 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 3.3 [4,9] 2024-03-21T10:37:56.328 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 3.5 [3,9] 2024-03-21T10:37:56.328 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 3.17 [4,8] 2024-03-21T10:37:56.328 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 3.1b [5,9,4,6] 2024-03-21T10:37:56.328 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 4.8 [5,8] 2024-03-21T10:37:56.328 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 4.10 [9,10] 2024-03-21T10:37:56.328 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 4.18 [9,8] 2024-03-21T10:37:56.328 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 4.1c [9,8] 2024-03-21T10:37:56.328 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/1451930554 expires 2024-03-22T10:26:53.842623+0000 2024-03-21T10:37:56.328 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/3342588843 expires 2024-03-22T10:26:53.842623+0000 2024-03-21T10:37:56.328 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:6801/4085461895 expires 2024-03-22T10:26:53.842623+0000 2024-03-21T10:37:56.328 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:6800/4085461895 expires 2024-03-22T10:26:53.842623+0000 2024-03-21T10:37:56.328 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/1626448389 expires 2024-03-22T10:26:16.959165+0000 2024-03-21T10:37:56.328 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/1910059351 expires 2024-03-22T10:26:16.959165+0000 2024-03-21T10:37:56.329 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/344822770 expires 2024-03-22T10:26:16.959165+0000 2024-03-21T10:37:56.329 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/3811368512 expires 2024-03-22T10:26:53.842623+0000 2024-03-21T10:37:56.329 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:6800/1981437651 expires 2024-03-22T10:26:16.959165+0000 2024-03-21T10:37:56.329 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:6801/1981437651 expires 2024-03-22T10:26:16.959165+0000 2024-03-21T10:37:56.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.330+0000 7f1df0c00700 1 -- 172.21.15.31:0/1524261185 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1dd00a3030 msgr2=0x7f1dd00a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:56.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.330+0000 7f1df0c00700 1 --2- 172.21.15.31:0/1524261185 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1dd00a3030 0x7f1dd00a54f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8000c00 tx=0x7f1dd8005fb0 comp rx=0 tx=0).stop 2024-03-21T10:37:56.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.330+0000 7f1df0c00700 1 -- 172.21.15.31:0/1524261185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dec1021e0 msgr2=0x7f1dec19a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:56.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.330+0000 7f1df0c00700 1 --2- 172.21.15.31:0/1524261185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dec1021e0 0x7f1dec19a620 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc00e9a0 tx=0x7f1ddc00ecb0 comp rx=0 tx=0).stop 2024-03-21T10:37:56.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.330+0000 7f1df0c00700 1 -- 172.21.15.31:0/1524261185 shutdown_connections 2024-03-21T10:37:56.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.330+0000 7f1df0c00700 1 --2- 172.21.15.31:0/1524261185 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1dd00a3030 0x7f1dd00a54f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:56.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.330+0000 7f1df0c00700 1 --2- 172.21.15.31:0/1524261185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dec103d80 0x7f1dec1a1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:56.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.330+0000 7f1df0c00700 1 --2- 172.21.15.31:0/1524261185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dec1021e0 0x7f1dec19a620 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:56.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.330+0000 7f1df0c00700 1 --2- 172.21.15.31:0/1524261185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dec1033e0 0x7f1dec19ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:56.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.330+0000 7f1df0c00700 1 -- 172.21.15.31:0/1524261185 >> 172.21.15.31:0/1524261185 conn(0x7f1dec0fd740 msgr2=0x7f1dec0743c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:56.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.330+0000 7f1df0c00700 1 -- 172.21.15.31:0/1524261185 shutdown_connections 2024-03-21T10:37:56.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:56.330+0000 7f1df0c00700 1 -- 172.21.15.31:0/1524261185 wait complete. 2024-03-21T10:37:57.366 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-21T10:37:57.378 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T10:37:57.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:57 smithi040 ceph-mon[36154]: pgmap v414: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:37:57.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:57 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1524261185' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-21T10:37:57.468 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:57 smithi031 ceph-mon[31673]: pgmap v414: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:37:57.468 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:57 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1524261185' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-21T10:37:57.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:57 smithi070 ceph-mon[36330]: pgmap v414: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:37:57.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:57 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1524261185' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-21T10:37:57.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.614+0000 7f6ee9800640 1 -- 172.21.15.31:0/506627895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ee4075660 msgr2=0x7f6ee4073ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:57.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.614+0000 7f6ee9800640 1 --2- 172.21.15.31:0/506627895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ee4075660 0x7f6ee4073ab0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4009900 tx=0x7f6ed402bcd0 comp rx=0 tx=0).stop 2024-03-21T10:37:57.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.614+0000 7f6ee9800640 1 -- 172.21.15.31:0/506627895 shutdown_connections 2024-03-21T10:37:57.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.614+0000 7f6ee9800640 1 --2- 172.21.15.31:0/506627895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ee41095e0 0x7f6ee410ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:57.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.614+0000 7f6ee9800640 1 --2- 172.21.15.31:0/506627895 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ee4074080 0x7f6ee40744e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:57.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.614+0000 7f6ee9800640 1 --2- 172.21.15.31:0/506627895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ee4075660 0x7f6ee4073ab0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:57.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.614+0000 7f6ee9800640 1 -- 172.21.15.31:0/506627895 >> 172.21.15.31:0/506627895 conn(0x7f6ee40fccc0 msgr2=0x7f6ee40ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:57.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.614+0000 7f6ee9800640 1 -- 172.21.15.31:0/506627895 shutdown_connections 2024-03-21T10:37:57.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.614+0000 7f6ee9800640 1 -- 172.21.15.31:0/506627895 wait complete. 2024-03-21T10:37:57.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.615+0000 7f6ee9800640 1 Processor -- start 2024-03-21T10:37:57.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.616+0000 7f6ee9800640 1 -- start start 2024-03-21T10:37:57.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.616+0000 7f6ee9800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ee4074080 0x7f6ee4199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:57.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.616+0000 7f6ee9800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ee4075660 0x7f6ee419a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:57.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.616+0000 7f6ee9800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ee41095e0 0x7f6ee41a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:57.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.616+0000 7f6ee9800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee410f590 con 0x7f6ee41095e0 2024-03-21T10:37:57.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.616+0000 7f6ee9800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee410f410 con 0x7f6ee4075660 2024-03-21T10:37:57.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.616+0000 7f6ee9800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee410f710 con 0x7f6ee4074080 2024-03-21T10:37:57.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.616+0000 7f6ee2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ee4074080 0x7f6ee4199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:57.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.616+0000 7f6ee2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ee4074080 0x7f6ee4199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:40192/0 (socket says 172.21.15.31:40192) 2024-03-21T10:37:57.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.616+0000 7f6ee2a00640 1 -- 172.21.15.31:0/754317288 learned_addr learned my addr 172.21.15.31:0/754317288 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:57.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.616+0000 7f6edbe00640 1 --2- 172.21.15.31:0/754317288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ee4075660 0x7f6ee419a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:57.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.616+0000 7f6ee3400640 1 --2- 172.21.15.31:0/754317288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ee41095e0 0x7f6ee41a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:57.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.617+0000 7f6ee2a00640 1 -- 172.21.15.31:0/754317288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ee4075660 msgr2=0x7f6ee419a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:57.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.617+0000 7f6ee2a00640 1 --2- 172.21.15.31:0/754317288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ee4075660 0x7f6ee419a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:57.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.617+0000 7f6ee2a00640 1 -- 172.21.15.31:0/754317288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ee41095e0 msgr2=0x7f6ee41a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:57.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.617+0000 7f6ee2a00640 1 --2- 172.21.15.31:0/754317288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ee41095e0 0x7f6ee41a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:57.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.617+0000 7f6ee2a00640 1 -- 172.21.15.31:0/754317288 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee41a17e0 con 0x7f6ee4074080 2024-03-21T10:37:57.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.617+0000 7f6ee3400640 1 --2- 172.21.15.31:0/754317288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ee41095e0 0x7f6ee41a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:37:57.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.617+0000 7f6edbe00640 1 --2- 172.21.15.31:0/754317288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ee4075660 0x7f6ee419a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:37:57.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.617+0000 7f6ee2a00640 1 --2- 172.21.15.31:0/754317288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ee4074080 0x7f6ee4199bc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4000c00 tx=0x7f6ed402c6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:57.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.618+0000 7f6edb400640 1 -- 172.21.15.31:0/754317288 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4043020 con 0x7f6ee4074080 2024-03-21T10:37:57.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.618+0000 7f6edb400640 1 -- 172.21.15.31:0/754317288 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6ed40054e0 con 0x7f6ee4074080 2024-03-21T10:37:57.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.618+0000 7f6edb400640 1 -- 172.21.15.31:0/754317288 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4003c20 con 0x7f6ee4074080 2024-03-21T10:37:57.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.618+0000 7f6ee9800640 1 -- 172.21.15.31:0/754317288 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee41a1a70 con 0x7f6ee4074080 2024-03-21T10:37:57.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.618+0000 7f6ee9800640 1 -- 172.21.15.31:0/754317288 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee41a2010 con 0x7f6ee4074080 2024-03-21T10:37:57.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.619+0000 7f6ee9800640 1 -- 172.21.15.31:0/754317288 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee41083a0 con 0x7f6ee4074080 2024-03-21T10:37:57.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.621+0000 7f6edb400640 1 -- 172.21.15.31:0/754317288 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f6ed4046020 con 0x7f6ee4074080 2024-03-21T10:37:57.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.622+0000 7f6edb400640 1 --2- 172.21.15.31:0/754317288 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f6ec80a2fa0 0x7f6ec80a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:57.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.623+0000 7f6edb400640 1 -- 172.21.15.31:0/754317288 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f6ed40f1ab0 con 0x7f6ee4074080 2024-03-21T10:37:57.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.623+0000 7f6edbe00640 1 --2- 172.21.15.31:0/754317288 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f6ec80a2fa0 0x7f6ec80a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:57.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.624+0000 7f6edbe00640 1 --2- 172.21.15.31:0/754317288 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f6ec80a2fa0 0x7f6ec80a5460 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6ee419b0d0 tx=0x7f6ecc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:57.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.625+0000 7f6edb400640 1 -- 172.21.15.31:0/754317288 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6ed404d020 con 0x7f6ee4074080 2024-03-21T10:37:57.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.829+0000 7f6ee9800640 1 -- 172.21.15.31:0/754317288 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ee40622a0 con 0x7f6ee4074080 2024-03-21T10:37:57.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.831+0000 7f6edb400640 1 -- 172.21.15.31:0/754317288 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24553 (secure 0 0 0) 0x7f6ed40be520 con 0x7f6ee4074080 2024-03-21T10:37:57.829 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:37:57.829 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":93,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","created":"2024-03-21T10:25:41.971117+0000","modified":"2024-03-21T10:36:56.301695+0000","last_up_change":"2024-03-21T10:34:07.757196+0000","last_in_change":"2024-03-21T10:33:47.460406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T10:29:44.553968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T10:35:04.945449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T10:35:05.970589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T10:35:09.688711+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e0244d33-00e2-47d5-9f66-34efc138c462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6803","nonce":3089885125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6805","nonce":3089885125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6809","nonce":3089885125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6807","nonce":3089885125}]},"public_addr":"172.21.15.31:6803/3089885125","cluster_addr":"172.21.15.31:6805/3089885125","heartbeat_back_addr":"172.21.15.31:6809/3089885125","heartbeat_front_addr":"172.21.15.31:6807/3089885125","state":["exists","up"]},{"osd":1,"uuid":"76a63504-beab-4354-a6c3-eebc3b16ad03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6801","nonce":618840037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6803","nonce":618840037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6807","nonce":618840037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6805","nonce":618840037}]},"public_addr":"172.21.15.40:6801/618840037","cluster_addr":"172.21.15.40:6803/618840037","heartbeat_back_addr":"172.21.15.40:6807/618840037","heartbeat_front_addr":"172.21.15.40:6805/618840037","state":["exists","up"]},{"osd":2,"uuid":"fa191bd6-9a21-4ee2-8adf-8a63c9cfd821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6801","nonce":2524184117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6803","nonce":2524184117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6807","nonce":2524184117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6805","nonce":2524184117}]},"public_addr":"172.21.15.70:6801/2524184117","cluster_addr":"172.21.15.70:6803/2524184117","heartbeat_back_addr":"172.21.15.70:6807/2524184117","heartbeat_front_addr":"172.21.15.70:6805/2524184117","state":["exists","up"]},{"osd":3,"uuid":"edb44507-ffb0-42a4-827b-08461e1bf952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6811","nonce":2823036296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6813","nonce":2823036296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6817","nonce":2823036296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6815","nonce":2823036296}]},"public_addr":"172.21.15.31:6811/2823036296","cluster_addr":"172.21.15.31:6813/2823036296","heartbeat_back_addr":"172.21.15.31:6817/2823036296","heartbeat_front_addr":"172.21.15.31:6815/2823036296","state":["exists","up"]},{"osd":4,"uuid":"8cb89153-8c3b-431f-b57c-8863fada76dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6809","nonce":4215186309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6811","nonce":4215186309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6815","nonce":4215186309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6813","nonce":4215186309}]},"public_addr":"172.21.15.40:6809/4215186309","cluster_addr":"172.21.15.40:6811/4215186309","heartbeat_back_addr":"172.21.15.40:6815/4215186309","heartbeat_front_addr":"172.21.15.40:6813/4215186309","state":["exists","up"]},{"osd":5,"uuid":"9409505a-d463-408c-a3cd-218380c04d68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6809","nonce":3402830452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6811","nonce":3402830452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6815","nonce":3402830452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6813","nonce":3402830452}]},"public_addr":"172.21.15.70:6809/3402830452","cluster_addr":"172.21.15.70:6811/3402830452","heartbeat_back_addr":"172.21.15.70:6815/3402830452","heartbeat_front_addr":"172.21.15.70:6813/3402830452","state":["exists","up"]},{"osd":6,"uuid":"f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6819","nonce":1193497058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6821","nonce":1193497058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6825","nonce":1193497058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6823","nonce":1193497058}]},"public_addr":"172.21.15.31:6819/1193497058","cluster_addr":"172.21.15.31:6821/1193497058","heartbeat_back_addr":"172.21.15.31:6825/1193497058","heartbeat_front_addr":"172.21.15.31:6823/1193497058","state":["exists","up"]},{"osd":7,"uuid":"db228a87-29a5-45d4-8fe6-a0a554262eb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6817","nonce":3825628119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6819","nonce":3825628119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6823","nonce":3825628119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6821","nonce":3825628119}]},"public_addr":"172.21.15.40:6817/3825628119","cluster_addr":"172.21.15.40:6819/3825628119","heartbeat_back_addr":"172.21.15.40:6823/3825628119","heartbeat_front_addr":"172.21.15.40:6821/3825628119","state":["exists","up"]},{"osd":8,"uuid":"59b9fde0-095d-4e0b-87da-ead37bd8ff1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6817","nonce":534880586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6819","nonce":534880586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6823","nonce":534880586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6821","nonce":534880586}]},"public_addr":"172.21.15.70:6817/534880586","cluster_addr":"172.21.15.70:6819/534880586","heartbeat_back_addr":"172.21.15.70:6823/534880586","heartbeat_front_addr":"172.21.15.70:6821/534880586","state":["exists","up"]},{"osd":9,"uuid":"77f47061-0c5f-4159-b276-e814935d6c1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6827","nonce":587047469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6829","nonce":587047469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6833","nonce":587047469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6831","nonce":587047469}]},"public_addr":"172.21.15.31:6827/587047469","cluster_addr":"172.21.15.31:6829/587047469","heartbeat_back_addr":"172.21.15.31:6833/587047469","heartbeat_front_addr":"172.21.15.31:6831/587047469","state":["exists","up"]},{"osd":10,"uuid":"8853dc43-7bd1-4c93-8063-f7982fd7da38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6825","nonce":512108711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6827","nonce":512108711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6831","nonce":512108711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6829","nonce":512108711}]},"public_addr":"172.21.15.40:6825/512108711","cluster_addr":"172.21.15.40:6827/512108711","heartbeat_back_addr":"172.21.15.40:6831/512108711","heartbeat_front_addr":"172.21.15.40:6829/512108711","state":["exists","up"]},{"osd":11,"uuid":"84b7c2de-c424-45a6-9583-60097e372a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6825","nonce":540305718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6827","nonce":540305718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6831","nonce":540305718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6829","nonce":540305718}]},"public_addr":"172.21.15.70:6825/540305718","cluster_addr":"172.21.15.70:6827/540305718","heartbeat_back_addr":"172.21.15.70:6831/540305718","heartbeat_front_addr":"172.21.15.70:6829/540305718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:28:51.477358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:17.405045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:41.626268+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:15.339568+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:42.676996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:07.516999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:42.412273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:10.365041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:35.699120+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:11.733639+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:39.505442+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:34:05.449301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451930554":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/3342588843":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6801/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/1626448389":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/1910059351":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/344822770":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/3811368512":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/1981437651":"2024-03-22T10:26:16.959165+0000","172.21.15.31:6801/1981437651":"2024-03-22T10:26:16.959165+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T10:37:57.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.834+0000 7f6ee9800640 1 -- 172.21.15.31:0/754317288 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f6ec80a2fa0 msgr2=0x7f6ec80a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:57.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.834+0000 7f6ee9800640 1 --2- 172.21.15.31:0/754317288 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f6ec80a2fa0 0x7f6ec80a5460 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6ee419b0d0 tx=0x7f6ecc005f90 comp rx=0 tx=0).stop 2024-03-21T10:37:57.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.834+0000 7f6ee9800640 1 -- 172.21.15.31:0/754317288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ee4074080 msgr2=0x7f6ee4199bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:57.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.834+0000 7f6ee9800640 1 --2- 172.21.15.31:0/754317288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ee4074080 0x7f6ee4199bc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4000c00 tx=0x7f6ed402c6f0 comp rx=0 tx=0).stop 2024-03-21T10:37:57.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.835+0000 7f6ee9800640 1 -- 172.21.15.31:0/754317288 shutdown_connections 2024-03-21T10:37:57.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.835+0000 7f6ee9800640 1 --2- 172.21.15.31:0/754317288 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f6ec80a2fa0 0x7f6ec80a5460 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:57.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.835+0000 7f6ee9800640 1 --2- 172.21.15.31:0/754317288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ee41095e0 0x7f6ee41a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:57.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.835+0000 7f6ee9800640 1 --2- 172.21.15.31:0/754317288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ee4075660 0x7f6ee419a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:57.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.835+0000 7f6ee9800640 1 --2- 172.21.15.31:0/754317288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ee4074080 0x7f6ee4199bc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:57.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.835+0000 7f6ee9800640 1 -- 172.21.15.31:0/754317288 >> 172.21.15.31:0/754317288 conn(0x7f6ee40fccc0 msgr2=0x7f6ee4105570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:57.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.835+0000 7f6ee9800640 1 -- 172.21.15.31:0/754317288 shutdown_connections 2024-03-21T10:37:57.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:57.835+0000 7f6ee9800640 1 -- 172.21.15.31:0/754317288 wait complete. 2024-03-21T10:37:57.852 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T10:29:44.553968+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T10:35:04.945449+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T10:35:05.970589+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T10:35:09.688711+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T10:37:57.852 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T10:37:58.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.083+0000 7f1253800640 1 -- 172.21.15.31:0/51874445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f124c107440 msgr2=0x7f124c109830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:58.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.083+0000 7f1253800640 1 --2- 172.21.15.31:0/51874445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f124c107440 0x7f124c109830 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1244009be0 tx=0x7f124402bef0 comp rx=0 tx=0).stop 2024-03-21T10:37:58.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.083+0000 7f1253800640 1 -- 172.21.15.31:0/51874445 shutdown_connections 2024-03-21T10:37:58.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.083+0000 7f1253800640 1 --2- 172.21.15.31:0/51874445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f124c107440 0x7f124c109830 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:58.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.083+0000 7f1253800640 1 --2- 172.21.15.31:0/51874445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f124c104b10 0x7f124c106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:58.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.083+0000 7f1253800640 1 --2- 172.21.15.31:0/51874445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f124c1021e0 0x7f124c1045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:58.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.083+0000 7f1253800640 1 -- 172.21.15.31:0/51874445 >> 172.21.15.31:0/51874445 conn(0x7f124c0fbb30 msgr2=0x7f124c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:58.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.083+0000 7f1253800640 1 -- 172.21.15.31:0/51874445 shutdown_connections 2024-03-21T10:37:58.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.083+0000 7f1253800640 1 -- 172.21.15.31:0/51874445 wait complete. 2024-03-21T10:37:58.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.085+0000 7f1253800640 1 Processor -- start 2024-03-21T10:37:58.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.085+0000 7f1253800640 1 -- start start 2024-03-21T10:37:58.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.085+0000 7f1253800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f124c1021e0 0x7f124c101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:58.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.085+0000 7f1253800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f124c104b10 0x7f124c100190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:58.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.085+0000 7f1253800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f124c107440 0x7f124c1006d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:58.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.085+0000 7f1253800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124c10d3b0 con 0x7f124c104b10 2024-03-21T10:37:58.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.085+0000 7f1253800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124c10d230 con 0x7f124c107440 2024-03-21T10:37:58.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.085+0000 7f1253800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124c10d530 con 0x7f124c1021e0 2024-03-21T10:37:58.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.086+0000 7f1250e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f124c1021e0 0x7f124c101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:58.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.086+0000 7f1251800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f124c107440 0x7f124c1006d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:58.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.086+0000 7f1250e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f124c1021e0 0x7f124c101b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:40202/0 (socket says 172.21.15.31:40202) 2024-03-21T10:37:58.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.086+0000 7f1250e00640 1 -- 172.21.15.31:0/1480486051 learned_addr learned my addr 172.21.15.31:0/1480486051 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:58.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.086+0000 7f124be00640 1 --2- 172.21.15.31:0/1480486051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f124c104b10 0x7f124c100190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:58.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.086+0000 7f1251800640 1 -- 172.21.15.31:0/1480486051 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f124c1021e0 msgr2=0x7f124c101b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:58.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.086+0000 7f1251800640 1 --2- 172.21.15.31:0/1480486051 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f124c1021e0 0x7f124c101b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:58.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.086+0000 7f1251800640 1 -- 172.21.15.31:0/1480486051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f124c104b10 msgr2=0x7f124c100190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:58.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.086+0000 7f1251800640 1 --2- 172.21.15.31:0/1480486051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f124c104b10 0x7f124c100190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:58.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.086+0000 7f1251800640 1 -- 172.21.15.31:0/1480486051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f124c100f60 con 0x7f124c107440 2024-03-21T10:37:58.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.086+0000 7f1251800640 1 --2- 172.21.15.31:0/1480486051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f124c107440 0x7f124c1006d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f124402c430 tx=0x7f12440337b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:58.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.086+0000 7f1249600640 1 -- 172.21.15.31:0/1480486051 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1244043020 con 0x7f124c107440 2024-03-21T10:37:58.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.087+0000 7f1253800640 1 -- 172.21.15.31:0/1480486051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f124c1a1ac0 con 0x7f124c107440 2024-03-21T10:37:58.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.087+0000 7f1249600640 1 -- 172.21.15.31:0/1480486051 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f124402ca60 con 0x7f124c107440 2024-03-21T10:37:58.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.087+0000 7f1249600640 1 -- 172.21.15.31:0/1480486051 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1244034690 con 0x7f124c107440 2024-03-21T10:37:58.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.087+0000 7f1253800640 1 -- 172.21.15.31:0/1480486051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f124c1a2060 con 0x7f124c107440 2024-03-21T10:37:58.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.088+0000 7f1253800640 1 -- 172.21.15.31:0/1480486051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f124c0fd620 con 0x7f124c107440 2024-03-21T10:37:58.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.090+0000 7f1249600640 1 -- 172.21.15.31:0/1480486051 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f1244046020 con 0x7f124c107440 2024-03-21T10:37:58.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.091+0000 7f1249600640 1 --2- 172.21.15.31:0/1480486051 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f12340a2f20 0x7f12340a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:58.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.091+0000 7f1249600640 1 -- 172.21.15.31:0/1480486051 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f12440f2170 con 0x7f124c107440 2024-03-21T10:37:58.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.092+0000 7f1250e00640 1 --2- 172.21.15.31:0/1480486051 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f12340a2f20 0x7f12340a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:58.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.092+0000 7f1250e00640 1 --2- 172.21.15.31:0/1480486051 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f12340a2f20 0x7f12340a53e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1240009ca0 tx=0x7f1240009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:58.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.094+0000 7f1249600640 1 -- 172.21.15.31:0/1480486051 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1244071e20 con 0x7f124c107440 2024-03-21T10:37:58.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.313+0000 7f1253800640 1 -- 172.21.15.31:0/1480486051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f124c02d070 con 0x7f124c107440 2024-03-21T10:37:58.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.314+0000 7f1249600640 1 -- 172.21.15.31:0/1480486051 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7f124404d030 con 0x7f124c107440 2024-03-21T10:37:58.312 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-03-21T10:37:58.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.316+0000 7f1253800640 1 -- 172.21.15.31:0/1480486051 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f12340a2f20 msgr2=0x7f12340a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:58.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.316+0000 7f1253800640 1 --2- 172.21.15.31:0/1480486051 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f12340a2f20 0x7f12340a53e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1240009ca0 tx=0x7f1240009260 comp rx=0 tx=0).stop 2024-03-21T10:37:58.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.317+0000 7f1253800640 1 -- 172.21.15.31:0/1480486051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f124c107440 msgr2=0x7f124c1006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:58.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.317+0000 7f1253800640 1 --2- 172.21.15.31:0/1480486051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f124c107440 0x7f124c1006d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f124402c430 tx=0x7f12440337b0 comp rx=0 tx=0).stop 2024-03-21T10:37:58.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.317+0000 7f1253800640 1 -- 172.21.15.31:0/1480486051 shutdown_connections 2024-03-21T10:37:58.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.317+0000 7f1253800640 1 --2- 172.21.15.31:0/1480486051 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f12340a2f20 0x7f12340a53e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:58.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.317+0000 7f1253800640 1 --2- 172.21.15.31:0/1480486051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f124c107440 0x7f124c1006d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:58.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.317+0000 7f1253800640 1 --2- 172.21.15.31:0/1480486051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f124c104b10 0x7f124c100190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:58.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.317+0000 7f1253800640 1 --2- 172.21.15.31:0/1480486051 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f124c1021e0 0x7f124c101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:58.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.317+0000 7f1253800640 1 -- 172.21.15.31:0/1480486051 >> 172.21.15.31:0/1480486051 conn(0x7f124c0fbb30 msgr2=0x7f124c102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:58.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.317+0000 7f1253800640 1 -- 172.21.15.31:0/1480486051 shutdown_connections 2024-03-21T10:37:58.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.317+0000 7f1253800640 1 -- 172.21.15.31:0/1480486051 wait complete. 2024-03-21T10:37:58.336 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T10:37:58.405 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:58 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/754317288' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:37:58.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:58 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/754317288' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:37:58.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.537+0000 7f764f200640 1 -- 172.21.15.31:0/1033720162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76501030b0 msgr2=0x7f7650109940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:58.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.537+0000 7f764f200640 1 --2- 172.21.15.31:0/1033720162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76501030b0 0x7f7650109940 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7648009ca0 tx=0x7f764802c100 comp rx=0 tx=0).stop 2024-03-21T10:37:58.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.538+0000 7f764f200640 1 -- 172.21.15.31:0/1033720162 shutdown_connections 2024-03-21T10:37:58.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.538+0000 7f764f200640 1 --2- 172.21.15.31:0/1033720162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76501030b0 0x7f7650109940 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:58.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.538+0000 7f764f200640 1 --2- 172.21.15.31:0/1033720162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76501026f0 0x7f7650102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:58.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.538+0000 7f764f200640 1 --2- 172.21.15.31:0/1033720162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7650101500 0x7f7650101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:58.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.538+0000 7f764f200640 1 -- 172.21.15.31:0/1033720162 >> 172.21.15.31:0/1033720162 conn(0x7f76500fcab0 msgr2=0x7f76500feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:58.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.538+0000 7f764f200640 1 -- 172.21.15.31:0/1033720162 shutdown_connections 2024-03-21T10:37:58.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.538+0000 7f764f200640 1 -- 172.21.15.31:0/1033720162 wait complete. 2024-03-21T10:37:58.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.539+0000 7f764f200640 1 Processor -- start 2024-03-21T10:37:58.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.539+0000 7f764f200640 1 -- start start 2024-03-21T10:37:58.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.539+0000 7f764f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7650101500 0x7f7650107280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:58.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.539+0000 7f764f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76501026f0 0x7f76501077c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:58.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.539+0000 7f764f200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f76501030b0 0x7f7650103df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:58.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.539+0000 7f764f200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765010d190 con 0x7f7650101500 2024-03-21T10:37:58.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.539+0000 7f764f200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765010d010 con 0x7f76501026f0 2024-03-21T10:37:58.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.539+0000 7f764f200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765010d310 con 0x7f76501030b0 2024-03-21T10:37:58.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.540+0000 7f764d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76501026f0 0x7f76501077c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:58.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.540+0000 7f764e800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f76501030b0 0x7f7650103df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:58.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.540+0000 7f764d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76501026f0 0x7f76501077c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:57316/0 (socket says 172.21.15.31:57316) 2024-03-21T10:37:58.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.540+0000 7f764d400640 1 -- 172.21.15.31:0/2678817951 learned_addr learned my addr 172.21.15.31:0/2678817951 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:58.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.540+0000 7f764de00640 1 --2- 172.21.15.31:0/2678817951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7650101500 0x7f7650107280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:58.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.540+0000 7f764e800640 1 -- 172.21.15.31:0/2678817951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76501026f0 msgr2=0x7f76501077c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:58.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.540+0000 7f764e800640 1 --2- 172.21.15.31:0/2678817951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76501026f0 0x7f76501077c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:58.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.540+0000 7f764e800640 1 -- 172.21.15.31:0/2678817951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7650101500 msgr2=0x7f7650107280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:58.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.540+0000 7f764e800640 1 --2- 172.21.15.31:0/2678817951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7650101500 0x7f7650107280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:58.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.540+0000 7f764e800640 1 -- 172.21.15.31:0/2678817951 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76501046b0 con 0x7f76501030b0 2024-03-21T10:37:58.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.540+0000 7f764d400640 1 --2- 172.21.15.31:0/2678817951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76501026f0 0x7f76501077c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:37:58.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.540+0000 7f764e800640 1 --2- 172.21.15.31:0/2678817951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f76501030b0 0x7f7650103df0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7648037040 tx=0x7f7648035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:58.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.541+0000 7f7646a00640 1 -- 172.21.15.31:0/2678817951 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7648039040 con 0x7f76501030b0 2024-03-21T10:37:58.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.541+0000 7f7646a00640 1 -- 172.21.15.31:0/2678817951 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7648044070 con 0x7f76501030b0 2024-03-21T10:37:58.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.541+0000 7f7646a00640 1 -- 172.21.15.31:0/2678817951 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7648004350 con 0x7f76501030b0 2024-03-21T10:37:58.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.541+0000 7f764f200640 1 -- 172.21.15.31:0/2678817951 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76501a5df0 con 0x7f76501030b0 2024-03-21T10:37:58.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.541+0000 7f764f200640 1 -- 172.21.15.31:0/2678817951 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76501a61a0 con 0x7f76501030b0 2024-03-21T10:37:58.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.542+0000 7f764f200640 1 -- 172.21.15.31:0/2678817951 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f765004f430 con 0x7f76501030b0 2024-03-21T10:37:58.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.544+0000 7f7646a00640 1 -- 172.21.15.31:0/2678817951 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f76480044f0 con 0x7f76501030b0 2024-03-21T10:37:58.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.545+0000 7f7646a00640 1 --2- 172.21.15.31:0/2678817951 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f76340a2fa0 0x7f76340a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:58.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.545+0000 7f7646a00640 1 -- 172.21.15.31:0/2678817951 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f76480f1c10 con 0x7f76501030b0 2024-03-21T10:37:58.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.545+0000 7f764de00640 1 --2- 172.21.15.31:0/2678817951 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f76340a2fa0 0x7f76340a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:58.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.546+0000 7f764de00640 1 --2- 172.21.15.31:0/2678817951 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f76340a2fa0 0x7f76340a5460 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7650102550 tx=0x7f7640005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:58.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.549+0000 7f7646a00640 1 -- 172.21.15.31:0/2678817951 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f765004f430 con 0x7f76501030b0 2024-03-21T10:37:58.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:58 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/754317288' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:37:58.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.750+0000 7f764f200640 1 -- 172.21.15.31:0/2678817951 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f76501a6b30 con 0x7f76501030b0 2024-03-21T10:37:58.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.751+0000 7f7646a00640 1 -- 172.21.15.31:0/2678817951 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f76480be680 con 0x7f76501030b0 2024-03-21T10:37:58.749 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-03-21T10:37:58.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.753+0000 7f764f200640 1 -- 172.21.15.31:0/2678817951 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f76340a2fa0 msgr2=0x7f76340a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:58.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.753+0000 7f764f200640 1 --2- 172.21.15.31:0/2678817951 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f76340a2fa0 0x7f76340a5460 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7650102550 tx=0x7f7640005f90 comp rx=0 tx=0).stop 2024-03-21T10:37:58.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.753+0000 7f764f200640 1 -- 172.21.15.31:0/2678817951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f76501030b0 msgr2=0x7f7650103df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:58.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.753+0000 7f764f200640 1 --2- 172.21.15.31:0/2678817951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f76501030b0 0x7f7650103df0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7648037040 tx=0x7f7648035d10 comp rx=0 tx=0).stop 2024-03-21T10:37:58.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.754+0000 7f764f200640 1 -- 172.21.15.31:0/2678817951 shutdown_connections 2024-03-21T10:37:58.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.754+0000 7f764f200640 1 --2- 172.21.15.31:0/2678817951 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f76340a2fa0 0x7f76340a5460 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:58.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.754+0000 7f764f200640 1 --2- 172.21.15.31:0/2678817951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f76501030b0 0x7f7650103df0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:58.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.754+0000 7f764f200640 1 --2- 172.21.15.31:0/2678817951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76501026f0 0x7f76501077c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:58.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.754+0000 7f764f200640 1 --2- 172.21.15.31:0/2678817951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7650101500 0x7f7650107280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:58.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.754+0000 7f764f200640 1 -- 172.21.15.31:0/2678817951 >> 172.21.15.31:0/2678817951 conn(0x7f76500fcab0 msgr2=0x7f76500738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:58.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.754+0000 7f764f200640 1 -- 172.21.15.31:0/2678817951 shutdown_connections 2024-03-21T10:37:58.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.754+0000 7f764f200640 1 -- 172.21.15.31:0/2678817951 wait complete. 2024-03-21T10:37:58.770 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T10:37:58.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.987+0000 7f44e0000640 1 -- 172.21.15.31:0/2236107780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d81028c0 msgr2=0x7f44d8102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:58.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.987+0000 7f44e0000640 1 --2- 172.21.15.31:0/2236107780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d81028c0 0x7f44d8102d40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f44c4009900 tx=0x7f44c402bc30 comp rx=0 tx=0).stop 2024-03-21T10:37:58.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.987+0000 7f44e0000640 1 -- 172.21.15.31:0/2236107780 shutdown_connections 2024-03-21T10:37:58.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.987+0000 7f44e0000640 1 --2- 172.21.15.31:0/2236107780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44d8103280 0x7f44d8109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:58.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.987+0000 7f44e0000640 1 --2- 172.21.15.31:0/2236107780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d81028c0 0x7f44d8102d40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:58.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.987+0000 7f44e0000640 1 --2- 172.21.15.31:0/2236107780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44d81016d0 0x7f44d8101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:58.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.987+0000 7f44e0000640 1 -- 172.21.15.31:0/2236107780 >> 172.21.15.31:0/2236107780 conn(0x7f44d80fcc60 msgr2=0x7f44d80ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:58.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.987+0000 7f44e0000640 1 -- 172.21.15.31:0/2236107780 shutdown_connections 2024-03-21T10:37:58.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.987+0000 7f44e0000640 1 -- 172.21.15.31:0/2236107780 wait complete. 2024-03-21T10:37:58.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.988+0000 7f44e0000640 1 Processor -- start 2024-03-21T10:37:58.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.988+0000 7f44e0000640 1 -- start start 2024-03-21T10:37:58.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.989+0000 7f44e0000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44d81016d0 0x7f44d8199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:58.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.989+0000 7f44e0000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44d81028c0 0x7f44d819a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:58.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.989+0000 7f44e0000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d8103280 0x7f44d81a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:58.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.989+0000 7f44e0000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d810d3a0 con 0x7f44d81016d0 2024-03-21T10:37:58.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.989+0000 7f44e0000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d810d220 con 0x7f44d8103280 2024-03-21T10:37:58.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.989+0000 7f44e0000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d810d520 con 0x7f44d81028c0 2024-03-21T10:37:58.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.989+0000 7f44dd600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44d81016d0 0x7f44d8199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:58.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.989+0000 7f44dcc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44d81028c0 0x7f44d819a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:58.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.989+0000 7f44dd600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44d81016d0 0x7f44d8199c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39728/0 (socket says 172.21.15.31:39728) 2024-03-21T10:37:58.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.989+0000 7f44dd600640 1 -- 172.21.15.31:0/29350216 learned_addr learned my addr 172.21.15.31:0/29350216 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:58.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.989+0000 7f44de000640 1 --2- 172.21.15.31:0/29350216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d8103280 0x7f44d81a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:58.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.989+0000 7f44dcc00640 1 -- 172.21.15.31:0/29350216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d8103280 msgr2=0x7f44d81a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:58.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.989+0000 7f44dcc00640 1 --2- 172.21.15.31:0/29350216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d8103280 0x7f44d81a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:58.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.989+0000 7f44dcc00640 1 -- 172.21.15.31:0/29350216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44d81016d0 msgr2=0x7f44d8199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:58.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.989+0000 7f44dcc00640 1 --2- 172.21.15.31:0/29350216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44d81016d0 0x7f44d8199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:58.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.989+0000 7f44dcc00640 1 -- 172.21.15.31:0/29350216 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d81a1840 con 0x7f44d81028c0 2024-03-21T10:37:58.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.990+0000 7f44dcc00640 1 --2- 172.21.15.31:0/29350216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44d81028c0 0x7f44d819a170 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f44c4036040 tx=0x7f44c402c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:58.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.990+0000 7f44d6000640 1 -- 172.21.15.31:0/29350216 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c4043020 con 0x7f44d81028c0 2024-03-21T10:37:58.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.990+0000 7f44d6000640 1 -- 172.21.15.31:0/29350216 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44c40054e0 con 0x7f44d81028c0 2024-03-21T10:37:58.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.990+0000 7f44e0000640 1 -- 172.21.15.31:0/29350216 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d81a1ad0 con 0x7f44d81028c0 2024-03-21T10:37:58.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.990+0000 7f44d6000640 1 -- 172.21.15.31:0/29350216 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c4003ca0 con 0x7f44d81028c0 2024-03-21T10:37:58.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.990+0000 7f44e0000640 1 -- 172.21.15.31:0/29350216 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d81a2040 con 0x7f44d81028c0 2024-03-21T10:37:58.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.991+0000 7f44e0000640 1 -- 172.21.15.31:0/29350216 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d81a2210 con 0x7f44d81028c0 2024-03-21T10:37:58.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.993+0000 7f44d6000640 1 -- 172.21.15.31:0/29350216 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f44c4046020 con 0x7f44d81028c0 2024-03-21T10:37:58.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.995+0000 7f44d6000640 1 --2- 172.21.15.31:0/29350216 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f44c00a2fa0 0x7f44c00a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:58.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.996+0000 7f44d6000640 1 -- 172.21.15.31:0/29350216 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f44c40f1850 con 0x7f44d81028c0 2024-03-21T10:37:58.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.996+0000 7f44dd600640 1 --2- 172.21.15.31:0/29350216 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f44c00a2fa0 0x7f44c00a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:58.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.996+0000 7f44dd600640 1 --2- 172.21.15.31:0/29350216 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f44c00a2fa0 0x7f44c00a5460 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f44d819b100 tx=0x7f44cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:58.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:58.998+0000 7f44d6000640 1 -- 172.21.15.31:0/29350216 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f44c404d020 con 0x7f44d81028c0 2024-03-21T10:37:59.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.217+0000 7f44e0000640 1 -- 172.21.15.31:0/29350216 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f44d81a23e0 con 0x7f44d81028c0 2024-03-21T10:37:59.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.218+0000 7f44d6000640 1 -- 172.21.15.31:0/29350216 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f44c40be2c0 con 0x7f44d81028c0 2024-03-21T10:37:59.215 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-03-21T10:37:59.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.220+0000 7f44e0000640 1 -- 172.21.15.31:0/29350216 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f44c00a2fa0 msgr2=0x7f44c00a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:59.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.220+0000 7f44e0000640 1 --2- 172.21.15.31:0/29350216 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f44c00a2fa0 0x7f44c00a5460 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f44d819b100 tx=0x7f44cc005f90 comp rx=0 tx=0).stop 2024-03-21T10:37:59.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.220+0000 7f44e0000640 1 -- 172.21.15.31:0/29350216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44d81028c0 msgr2=0x7f44d819a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:59.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.220+0000 7f44e0000640 1 --2- 172.21.15.31:0/29350216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44d81028c0 0x7f44d819a170 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f44c4036040 tx=0x7f44c402c6c0 comp rx=0 tx=0).stop 2024-03-21T10:37:59.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.220+0000 7f44e0000640 1 -- 172.21.15.31:0/29350216 shutdown_connections 2024-03-21T10:37:59.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.220+0000 7f44e0000640 1 --2- 172.21.15.31:0/29350216 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f44c00a2fa0 0x7f44c00a5460 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:59.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.220+0000 7f44e0000640 1 --2- 172.21.15.31:0/29350216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d8103280 0x7f44d81a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:59.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.220+0000 7f44e0000640 1 --2- 172.21.15.31:0/29350216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44d81028c0 0x7f44d819a170 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:59.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.220+0000 7f44e0000640 1 --2- 172.21.15.31:0/29350216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44d81016d0 0x7f44d8199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:59.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.220+0000 7f44e0000640 1 -- 172.21.15.31:0/29350216 >> 172.21.15.31:0/29350216 conn(0x7f44d80fcc60 msgr2=0x7f44d80738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:59.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.221+0000 7f44e0000640 1 -- 172.21.15.31:0/29350216 shutdown_connections 2024-03-21T10:37:59.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.221+0000 7f44e0000640 1 -- 172.21.15.31:0/29350216 wait complete. 2024-03-21T10:37:59.238 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T10:37:59.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:59 smithi040 ceph-mon[36154]: pgmap v415: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:59.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:59 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1480486051' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:37:59.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:37:59 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2678817951' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:37:59.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.456+0000 7fa9b0200640 1 -- 172.21.15.31:0/4155880420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9a80fff30 msgr2=0x7fa9a8102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:59.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.456+0000 7fa9b0200640 1 --2- 172.21.15.31:0/4155880420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9a80fff30 0x7fa9a8102360 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa99c009900 tx=0x7fa99c02bc40 comp rx=0 tx=0).stop 2024-03-21T10:37:59.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.456+0000 7fa9b0200640 1 -- 172.21.15.31:0/4155880420 shutdown_connections 2024-03-21T10:37:59.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.456+0000 7fa9b0200640 1 --2- 172.21.15.31:0/4155880420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9a8105250 0x7fa9a81056d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:59.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.456+0000 7fa9b0200640 1 --2- 172.21.15.31:0/4155880420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9a81028a0 0x7fa9a8104d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:59.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.456+0000 7fa9b0200640 1 --2- 172.21.15.31:0/4155880420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9a80fff30 0x7fa9a8102360 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:59.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.456+0000 7fa9b0200640 1 -- 172.21.15.31:0/4155880420 >> 172.21.15.31:0/4155880420 conn(0x7fa9a80fbb30 msgr2=0x7fa9a80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:59.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.456+0000 7fa9b0200640 1 -- 172.21.15.31:0/4155880420 shutdown_connections 2024-03-21T10:37:59.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.456+0000 7fa9b0200640 1 -- 172.21.15.31:0/4155880420 wait complete. 2024-03-21T10:37:59.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.457+0000 7fa9b0200640 1 Processor -- start 2024-03-21T10:37:59.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.457+0000 7fa9b0200640 1 -- start start 2024-03-21T10:37:59.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.458+0000 7fa9b0200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9a80fff30 0x7fa9a819dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:59.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.458+0000 7fa9b0200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9a81028a0 0x7fa9a819e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:59.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.458+0000 7fa9b0200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9a8105250 0x7fa9a81a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:59.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.458+0000 7fa9b0200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a8113960 con 0x7fa9a80fff30 2024-03-21T10:37:59.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.458+0000 7fa9b0200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a81137e0 con 0x7fa9a81028a0 2024-03-21T10:37:59.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.458+0000 7fa9b0200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a8113ae0 con 0x7fa9a8105250 2024-03-21T10:37:59.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.458+0000 7fa9ace00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9a81028a0 0x7fa9a819e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:59.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.458+0000 7fa9ace00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9a81028a0 0x7fa9a819e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:57348/0 (socket says 172.21.15.31:57348) 2024-03-21T10:37:59.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.458+0000 7fa9ace00640 1 -- 172.21.15.31:0/1055088885 learned_addr learned my addr 172.21.15.31:0/1055088885 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:59.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.458+0000 7fa9ad800640 1 --2- 172.21.15.31:0/1055088885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9a80fff30 0x7fa9a819dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:59.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.458+0000 7fa9ae200640 1 --2- 172.21.15.31:0/1055088885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9a8105250 0x7fa9a81a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:59.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.458+0000 7fa9ace00640 1 -- 172.21.15.31:0/1055088885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9a8105250 msgr2=0x7fa9a81a5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:59.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.458+0000 7fa9ace00640 1 --2- 172.21.15.31:0/1055088885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9a8105250 0x7fa9a81a5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:59.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.458+0000 7fa9ace00640 1 -- 172.21.15.31:0/1055088885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9a80fff30 msgr2=0x7fa9a819dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:59.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.458+0000 7fa9ace00640 1 --2- 172.21.15.31:0/1055088885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9a80fff30 0x7fa9a819dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:59.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.459+0000 7fa9ace00640 1 -- 172.21.15.31:0/1055088885 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a81a5bb0 con 0x7fa9a81028a0 2024-03-21T10:37:59.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.459+0000 7fa9ace00640 1 --2- 172.21.15.31:0/1055088885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9a81028a0 0x7fa9a819e520 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa99400e9f0 tx=0x7fa99400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:59.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.459+0000 7fa9a6000640 1 -- 172.21.15.31:0/1055088885 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99400cd80 con 0x7fa9a81028a0 2024-03-21T10:37:59.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.459+0000 7fa9a6000640 1 -- 172.21.15.31:0/1055088885 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa994010910 con 0x7fa9a81028a0 2024-03-21T10:37:59.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.459+0000 7fa9a6000640 1 -- 172.21.15.31:0/1055088885 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa994018990 con 0x7fa9a81028a0 2024-03-21T10:37:59.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.459+0000 7fa9b0200640 1 -- 172.21.15.31:0/1055088885 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a81a5ea0 con 0x7fa9a81028a0 2024-03-21T10:37:59.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.459+0000 7fa9b0200640 1 -- 172.21.15.31:0/1055088885 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9a81a6360 con 0x7fa9a81028a0 2024-03-21T10:37:59.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.460+0000 7fa9b0200640 1 -- 172.21.15.31:0/1055088885 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9a80fd620 con 0x7fa9a81028a0 2024-03-21T10:37:59.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.463+0000 7fa9a6000640 1 -- 172.21.15.31:0/1055088885 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fa994010ac0 con 0x7fa9a81028a0 2024-03-21T10:37:59.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.464+0000 7fa9a6000640 1 --2- 172.21.15.31:0/1055088885 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa9900a2f20 0x7fa9900a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:59.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.465+0000 7fa9a6000640 1 -- 172.21.15.31:0/1055088885 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fa9940cfa90 con 0x7fa9a81028a0 2024-03-21T10:37:59.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.465+0000 7fa9ad800640 1 --2- 172.21.15.31:0/1055088885 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa9900a2f20 0x7fa9900a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:59.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.465+0000 7fa9ad800640 1 --2- 172.21.15.31:0/1055088885 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa9900a2f20 0x7fa9900a53e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa99c02c250 tx=0x7fa99c02cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:59.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.468+0000 7fa9a6000640 1 -- 172.21.15.31:0/1055088885 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa99409beb0 con 0x7fa9a81028a0 2024-03-21T10:37:59.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:59 smithi031 ceph-mon[31673]: pgmap v415: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:59.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:59 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1480486051' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:37:59.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:37:59 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2678817951' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:37:59.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:59 smithi070 ceph-mon[36330]: pgmap v415: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:37:59.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:59 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1480486051' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:37:59.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:37:59 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2678817951' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:37:59.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.739+0000 7fa9b0200640 1 -- 172.21.15.31:0/1055088885 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fa9a8062340 con 0x7fa9a81028a0 2024-03-21T10:37:59.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.740+0000 7fa9a6000640 1 -- 172.21.15.31:0/1055088885 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7fa99409ff50 con 0x7fa9a81028a0 2024-03-21T10:37:59.738 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-03-21T10:37:59.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.742+0000 7fa9b0200640 1 -- 172.21.15.31:0/1055088885 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa9900a2f20 msgr2=0x7fa9900a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:59.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.742+0000 7fa9b0200640 1 --2- 172.21.15.31:0/1055088885 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa9900a2f20 0x7fa9900a53e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa99c02c250 tx=0x7fa99c02cf90 comp rx=0 tx=0).stop 2024-03-21T10:37:59.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.742+0000 7fa9b0200640 1 -- 172.21.15.31:0/1055088885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9a81028a0 msgr2=0x7fa9a819e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:59.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.742+0000 7fa9b0200640 1 --2- 172.21.15.31:0/1055088885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9a81028a0 0x7fa9a819e520 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa99400e9f0 tx=0x7fa99400eeb0 comp rx=0 tx=0).stop 2024-03-21T10:37:59.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.742+0000 7fa9b0200640 1 -- 172.21.15.31:0/1055088885 shutdown_connections 2024-03-21T10:37:59.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.742+0000 7fa9b0200640 1 --2- 172.21.15.31:0/1055088885 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa9900a2f20 0x7fa9900a53e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:59.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.742+0000 7fa9b0200640 1 --2- 172.21.15.31:0/1055088885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9a8105250 0x7fa9a81a5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:59.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.742+0000 7fa9b0200640 1 --2- 172.21.15.31:0/1055088885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9a81028a0 0x7fa9a819e520 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:59.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.742+0000 7fa9b0200640 1 --2- 172.21.15.31:0/1055088885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9a80fff30 0x7fa9a819dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:59.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.743+0000 7fa9b0200640 1 -- 172.21.15.31:0/1055088885 >> 172.21.15.31:0/1055088885 conn(0x7fa9a80fbb30 msgr2=0x7fa9a8103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:59.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.743+0000 7fa9b0200640 1 -- 172.21.15.31:0/1055088885 shutdown_connections 2024-03-21T10:37:59.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.743+0000 7fa9b0200640 1 -- 172.21.15.31:0/1055088885 wait complete. 2024-03-21T10:37:59.764 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:37:59.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.992+0000 7f2eea600640 1 -- 172.21.15.31:0/2260676151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ee4104a20 msgr2=0x7f2ee4106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:59.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.992+0000 7f2eea600640 1 --2- 172.21.15.31:0/2260676151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ee4104a20 0x7f2ee4106e10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0009900 tx=0x7f2ed002bc40 comp rx=0 tx=0).stop 2024-03-21T10:37:59.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.992+0000 7f2eea600640 1 -- 172.21.15.31:0/2260676151 shutdown_connections 2024-03-21T10:37:59.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.992+0000 7f2eea600640 1 --2- 172.21.15.31:0/2260676151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ee4107350 0x7f2ee4109740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:59.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.992+0000 7f2eea600640 1 --2- 172.21.15.31:0/2260676151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ee4104a20 0x7f2ee4106e10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:59.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.992+0000 7f2eea600640 1 --2- 172.21.15.31:0/2260676151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ee41020f0 0x7f2ee41044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:59.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.992+0000 7f2eea600640 1 -- 172.21.15.31:0/2260676151 >> 172.21.15.31:0/2260676151 conn(0x7f2ee40fba20 msgr2=0x7f2ee40fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:37:59.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.992+0000 7f2eea600640 1 -- 172.21.15.31:0/2260676151 shutdown_connections 2024-03-21T10:37:59.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.992+0000 7f2eea600640 1 -- 172.21.15.31:0/2260676151 wait complete. 2024-03-21T10:37:59.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.993+0000 7f2eea600640 1 Processor -- start 2024-03-21T10:37:59.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.994+0000 7f2eea600640 1 -- start start 2024-03-21T10:37:59.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.994+0000 7f2eea600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ee41020f0 0x7f2ee4101a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:59.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.994+0000 7f2eea600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ee4104a20 0x7f2ee41000a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:59.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.994+0000 7f2eea600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ee4107350 0x7f2ee41005e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:59.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.994+0000 7f2eea600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee410d290 con 0x7f2ee41020f0 2024-03-21T10:37:59.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.994+0000 7f2eea600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee410d110 con 0x7f2ee4107350 2024-03-21T10:37:59.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.994+0000 7f2eea600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee410d410 con 0x7f2ee4104a20 2024-03-21T10:37:59.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.994+0000 7f2ee3e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ee4104a20 0x7f2ee41000a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:59.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.994+0000 7f2ee9200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ee41020f0 0x7f2ee4101a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:59.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.994+0000 7f2ee3e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ee4104a20 0x7f2ee41000a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:40284/0 (socket says 172.21.15.31:40284) 2024-03-21T10:37:59.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.994+0000 7f2ee9c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ee4107350 0x7f2ee41005e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:59.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.994+0000 7f2ee3e00640 1 -- 172.21.15.31:0/4077919767 learned_addr learned my addr 172.21.15.31:0/4077919767 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:37:59.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.995+0000 7f2ee3e00640 1 -- 172.21.15.31:0/4077919767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ee4107350 msgr2=0x7f2ee41005e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:59.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.995+0000 7f2ee3e00640 1 --2- 172.21.15.31:0/4077919767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ee4107350 0x7f2ee41005e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:59.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.995+0000 7f2ee3e00640 1 -- 172.21.15.31:0/4077919767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ee41020f0 msgr2=0x7f2ee4101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:37:59.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.995+0000 7f2ee3e00640 1 --2- 172.21.15.31:0/4077919767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ee41020f0 0x7f2ee4101a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:37:59.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.995+0000 7f2ee3e00640 1 -- 172.21.15.31:0/4077919767 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee4100e70 con 0x7f2ee4104a20 2024-03-21T10:37:59.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.995+0000 7f2ee9c00640 1 --2- 172.21.15.31:0/4077919767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ee4107350 0x7f2ee41005e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:37:59.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.995+0000 7f2ee3e00640 1 --2- 172.21.15.31:0/4077919767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ee4104a20 0x7f2ee41000a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2ed002efd0 tx=0x7f2ed002c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:37:59.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.995+0000 7f2ee1600640 1 -- 172.21.15.31:0/4077919767 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0041070 con 0x7f2ee4104a20 2024-03-21T10:37:59.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.995+0000 7f2ee1600640 1 -- 172.21.15.31:0/4077919767 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ed0004450 con 0x7f2ee4104a20 2024-03-21T10:37:59.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.995+0000 7f2eea600640 1 -- 172.21.15.31:0/4077919767 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee41a1990 con 0x7f2ee4104a20 2024-03-21T10:37:59.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.995+0000 7f2ee1600640 1 -- 172.21.15.31:0/4077919767 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0003e50 con 0x7f2ee4104a20 2024-03-21T10:37:59.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.995+0000 7f2eea600640 1 -- 172.21.15.31:0/4077919767 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee41a1f00 con 0x7f2ee4104a20 2024-03-21T10:37:59.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.996+0000 7f2eea600640 1 -- 172.21.15.31:0/4077919767 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee40fd510 con 0x7f2ee4104a20 2024-03-21T10:37:59.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:37:59.998+0000 7f2ee1600640 1 -- 172.21.15.31:0/4077919767 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f2ed0044070 con 0x7f2ee4104a20 2024-03-21T10:37:59.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.000+0000 7f2ee1600640 1 --2- 172.21.15.31:0/4077919767 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2ecc0a3130 0x7f2ecc0a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:37:59.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.000+0000 7f2ee1600640 1 -- 172.21.15.31:0/4077919767 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f2ed00f1250 con 0x7f2ee4104a20 2024-03-21T10:37:59.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.000+0000 7f2ee9200640 1 --2- 172.21.15.31:0/4077919767 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2ecc0a3130 0x7f2ecc0a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:37:59.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.000+0000 7f2ee9200640 1 --2- 172.21.15.31:0/4077919767 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2ecc0a3130 0x7f2ecc0a55f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8009bd0 tx=0x7f2ed8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:00.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.004+0000 7f2ee1600640 1 -- 172.21.15.31:0/4077919767 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2ed00bd670 con 0x7f2ee4104a20 2024-03-21T10:38:00.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.246+0000 7f2eea600640 1 -- 172.21.15.31:0/4077919767 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ee402d070 con 0x7f2ee4104a20 2024-03-21T10:38:00.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.247+0000 7f2ee1600640 1 -- 172.21.15.31:0/4077919767 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11384 (secure 0 0 0) 0x7f2ed00c1710 con 0x7f2ee4104a20 2024-03-21T10:38:00.245 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:38:00.245 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:35:40.990205+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24596,24547]},"id":1}]} 2024-03-21T10:38:00.245 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2024-03-21T10:38:00.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.249+0000 7f2eea600640 1 -- 172.21.15.31:0/4077919767 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2ecc0a3130 msgr2=0x7f2ecc0a55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:00.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.249+0000 7f2eea600640 1 --2- 172.21.15.31:0/4077919767 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2ecc0a3130 0x7f2ecc0a55f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8009bd0 tx=0x7f2ed8005f90 comp rx=0 tx=0).stop 2024-03-21T10:38:00.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.249+0000 7f2eea600640 1 -- 172.21.15.31:0/4077919767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ee4104a20 msgr2=0x7f2ee41000a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:00.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.249+0000 7f2eea600640 1 --2- 172.21.15.31:0/4077919767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ee4104a20 0x7f2ee41000a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2ed002efd0 tx=0x7f2ed002c660 comp rx=0 tx=0).stop 2024-03-21T10:38:00.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.250+0000 7f2eea600640 1 -- 172.21.15.31:0/4077919767 shutdown_connections 2024-03-21T10:38:00.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.250+0000 7f2eea600640 1 --2- 172.21.15.31:0/4077919767 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f2ecc0a3130 0x7f2ecc0a55f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:00.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.250+0000 7f2eea600640 1 --2- 172.21.15.31:0/4077919767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ee4107350 0x7f2ee41005e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:00.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.250+0000 7f2eea600640 1 --2- 172.21.15.31:0/4077919767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ee4104a20 0x7f2ee41000a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:00.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.250+0000 7f2eea600640 1 --2- 172.21.15.31:0/4077919767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ee41020f0 0x7f2ee4101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:00.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.250+0000 7f2eea600640 1 -- 172.21.15.31:0/4077919767 >> 172.21.15.31:0/4077919767 conn(0x7f2ee40fba20 msgr2=0x7f2ee4102a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:00.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.250+0000 7f2eea600640 1 -- 172.21.15.31:0/4077919767 shutdown_connections 2024-03-21T10:38:00.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.250+0000 7f2eea600640 1 -- 172.21.15.31:0/4077919767 wait complete. 2024-03-21T10:38:00.266 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T10:38:00.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:00 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/29350216' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:38:00.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:00 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1055088885' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:38:00.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.468+0000 7f18faa00640 1 -- 172.21.15.31:0/1268496048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18f40ffe20 msgr2=0x7f18f4100240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:00.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.468+0000 7f18faa00640 1 --2- 172.21.15.31:0/1268496048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18f40ffe20 0x7f18f4100240 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f18e8009860 tx=0x7f18e802bb70 comp rx=0 tx=0).stop 2024-03-21T10:38:00.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.468+0000 7f18faa00640 1 -- 172.21.15.31:0/1268496048 shutdown_connections 2024-03-21T10:38:00.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.468+0000 7f18faa00640 1 --2- 172.21.15.31:0/1268496048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f4101120 0x7f18f41015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:00.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.468+0000 7f18faa00640 1 --2- 172.21.15.31:0/1268496048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18f4100780 0x7f18f4100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:00.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.468+0000 7f18faa00640 1 --2- 172.21.15.31:0/1268496048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18f40ffe20 0x7f18f4100240 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:00.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.468+0000 7f18faa00640 1 -- 172.21.15.31:0/1268496048 >> 172.21.15.31:0/1268496048 conn(0x7f18f40fba20 msgr2=0x7f18f40fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:00.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.469+0000 7f18faa00640 1 -- 172.21.15.31:0/1268496048 shutdown_connections 2024-03-21T10:38:00.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.469+0000 7f18faa00640 1 -- 172.21.15.31:0/1268496048 wait complete. 2024-03-21T10:38:00.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.469+0000 7f18faa00640 1 Processor -- start 2024-03-21T10:38:00.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.470+0000 7f18faa00640 1 -- start start 2024-03-21T10:38:00.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.470+0000 7f18faa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f40ffe20 0x7f18f4199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:00.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.470+0000 7f18faa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18f4100780 0x7f18f4199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:00.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.470+0000 7f18faa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18f4101120 0x7f18f41a1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:00.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.470+0000 7f18faa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f410f3f0 con 0x7f18f40ffe20 2024-03-21T10:38:00.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.470+0000 7f18faa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f410f270 con 0x7f18f4101120 2024-03-21T10:38:00.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.470+0000 7f18faa00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f410f570 con 0x7f18f4100780 2024-03-21T10:38:00.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.470+0000 7f18f9600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f40ffe20 0x7f18f4199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:00.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.470+0000 7f18f9600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f40ffe20 0x7f18f4199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39768/0 (socket says 172.21.15.31:39768) 2024-03-21T10:38:00.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.470+0000 7f18f9600640 1 -- 172.21.15.31:0/3309177159 learned_addr learned my addr 172.21.15.31:0/3309177159 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:00.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.470+0000 7f18fa000640 1 --2- 172.21.15.31:0/3309177159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18f4101120 0x7f18f41a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:00.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.470+0000 7f18f8c00640 1 --2- 172.21.15.31:0/3309177159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18f4100780 0x7f18f4199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:00.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.471+0000 7f18f9600640 1 -- 172.21.15.31:0/3309177159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18f4100780 msgr2=0x7f18f4199fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:00.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.471+0000 7f18f9600640 1 --2- 172.21.15.31:0/3309177159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18f4100780 0x7f18f4199fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:00.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.471+0000 7f18f9600640 1 -- 172.21.15.31:0/3309177159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18f4101120 msgr2=0x7f18f41a1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:00.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.471+0000 7f18f9600640 1 --2- 172.21.15.31:0/3309177159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18f4101120 0x7f18f41a1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:00.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.471+0000 7f18f9600640 1 -- 172.21.15.31:0/3309177159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f41a16b0 con 0x7f18f40ffe20 2024-03-21T10:38:00.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.471+0000 7f18fa000640 1 --2- 172.21.15.31:0/3309177159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18f4101120 0x7f18f41a1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:00.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.471+0000 7f18f9600640 1 --2- 172.21.15.31:0/3309177159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f40ffe20 0x7f18f4199a90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f18e80349e0 tx=0x7f18e8032450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:00.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.471+0000 7f18f2000640 1 -- 172.21.15.31:0/3309177159 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e8041030 con 0x7f18f40ffe20 2024-03-21T10:38:00.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.471+0000 7f18f2000640 1 -- 172.21.15.31:0/3309177159 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f18e802c510 con 0x7f18f40ffe20 2024-03-21T10:38:00.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.471+0000 7f18faa00640 1 -- 172.21.15.31:0/3309177159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f41a1940 con 0x7f18f40ffe20 2024-03-21T10:38:00.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.471+0000 7f18f2000640 1 -- 172.21.15.31:0/3309177159 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e8039040 con 0x7f18f40ffe20 2024-03-21T10:38:00.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.471+0000 7f18faa00640 1 -- 172.21.15.31:0/3309177159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f41a1e30 con 0x7f18f40ffe20 2024-03-21T10:38:00.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.473+0000 7f18faa00640 1 -- 172.21.15.31:0/3309177159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18f410ab70 con 0x7f18f40ffe20 2024-03-21T10:38:00.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.476+0000 7f18f2000640 1 -- 172.21.15.31:0/3309177159 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f18e8044070 con 0x7f18f40ffe20 2024-03-21T10:38:00.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.478+0000 7f18f2000640 1 --2- 172.21.15.31:0/3309177159 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f18dc0a2f10 0x7f18dc0a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:00.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.479+0000 7f18f2000640 1 -- 172.21.15.31:0/3309177159 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f18e80f0cb0 con 0x7f18f40ffe20 2024-03-21T10:38:00.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.479+0000 7f18f8c00640 1 --2- 172.21.15.31:0/3309177159 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f18dc0a2f10 0x7f18dc0a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:00.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.479+0000 7f18f8c00640 1 --2- 172.21.15.31:0/3309177159 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f18dc0a2f10 0x7f18dc0a53d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f18f419afa0 tx=0x7f18e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:00.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.484+0000 7f18f2000640 1 -- 172.21.15.31:0/3309177159 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f18e80bd720 con 0x7f18f40ffe20 2024-03-21T10:38:00.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:00 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/29350216' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:38:00.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:00 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1055088885' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:38:00.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:00 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/29350216' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:38:00.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:00 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1055088885' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:38:00.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.774+0000 7f18faa00640 1 -- 172.21.15.31:0/3309177159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18f40622a0 con 0x7f18f40ffe20 2024-03-21T10:38:00.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.777+0000 7f18f2000640 1 -- 172.21.15.31:0/3309177159 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24553 (secure 0 0 0) 0x7f18e80bd020 con 0x7f18f40ffe20 2024-03-21T10:38:00.776 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:38:00.776 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":93,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","created":"2024-03-21T10:25:41.971117+0000","modified":"2024-03-21T10:36:56.301695+0000","last_up_change":"2024-03-21T10:34:07.757196+0000","last_in_change":"2024-03-21T10:33:47.460406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T10:29:44.553968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T10:35:04.945449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T10:35:05.970589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T10:35:09.688711+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e0244d33-00e2-47d5-9f66-34efc138c462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6803","nonce":3089885125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6805","nonce":3089885125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6809","nonce":3089885125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6807","nonce":3089885125}]},"public_addr":"172.21.15.31:6803/3089885125","cluster_addr":"172.21.15.31:6805/3089885125","heartbeat_back_addr":"172.21.15.31:6809/3089885125","heartbeat_front_addr":"172.21.15.31:6807/3089885125","state":["exists","up"]},{"osd":1,"uuid":"76a63504-beab-4354-a6c3-eebc3b16ad03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6801","nonce":618840037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6803","nonce":618840037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6807","nonce":618840037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6805","nonce":618840037}]},"public_addr":"172.21.15.40:6801/618840037","cluster_addr":"172.21.15.40:6803/618840037","heartbeat_back_addr":"172.21.15.40:6807/618840037","heartbeat_front_addr":"172.21.15.40:6805/618840037","state":["exists","up"]},{"osd":2,"uuid":"fa191bd6-9a21-4ee2-8adf-8a63c9cfd821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6801","nonce":2524184117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6803","nonce":2524184117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6807","nonce":2524184117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6805","nonce":2524184117}]},"public_addr":"172.21.15.70:6801/2524184117","cluster_addr":"172.21.15.70:6803/2524184117","heartbeat_back_addr":"172.21.15.70:6807/2524184117","heartbeat_front_addr":"172.21.15.70:6805/2524184117","state":["exists","up"]},{"osd":3,"uuid":"edb44507-ffb0-42a4-827b-08461e1bf952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6811","nonce":2823036296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6813","nonce":2823036296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6817","nonce":2823036296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6815","nonce":2823036296}]},"public_addr":"172.21.15.31:6811/2823036296","cluster_addr":"172.21.15.31:6813/2823036296","heartbeat_back_addr":"172.21.15.31:6817/2823036296","heartbeat_front_addr":"172.21.15.31:6815/2823036296","state":["exists","up"]},{"osd":4,"uuid":"8cb89153-8c3b-431f-b57c-8863fada76dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6809","nonce":4215186309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6811","nonce":4215186309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6815","nonce":4215186309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6813","nonce":4215186309}]},"public_addr":"172.21.15.40:6809/4215186309","cluster_addr":"172.21.15.40:6811/4215186309","heartbeat_back_addr":"172.21.15.40:6815/4215186309","heartbeat_front_addr":"172.21.15.40:6813/4215186309","state":["exists","up"]},{"osd":5,"uuid":"9409505a-d463-408c-a3cd-218380c04d68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6809","nonce":3402830452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6811","nonce":3402830452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6815","nonce":3402830452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6813","nonce":3402830452}]},"public_addr":"172.21.15.70:6809/3402830452","cluster_addr":"172.21.15.70:6811/3402830452","heartbeat_back_addr":"172.21.15.70:6815/3402830452","heartbeat_front_addr":"172.21.15.70:6813/3402830452","state":["exists","up"]},{"osd":6,"uuid":"f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6819","nonce":1193497058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6821","nonce":1193497058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6825","nonce":1193497058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6823","nonce":1193497058}]},"public_addr":"172.21.15.31:6819/1193497058","cluster_addr":"172.21.15.31:6821/1193497058","heartbeat_back_addr":"172.21.15.31:6825/1193497058","heartbeat_front_addr":"172.21.15.31:6823/1193497058","state":["exists","up"]},{"osd":7,"uuid":"db228a87-29a5-45d4-8fe6-a0a554262eb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6817","nonce":3825628119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6819","nonce":3825628119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6823","nonce":3825628119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6821","nonce":3825628119}]},"public_addr":"172.21.15.40:6817/3825628119","cluster_addr":"172.21.15.40:6819/3825628119","heartbeat_back_addr":"172.21.15.40:6823/3825628119","heartbeat_front_addr":"172.21.15.40:6821/3825628119","state":["exists","up"]},{"osd":8,"uuid":"59b9fde0-095d-4e0b-87da-ead37bd8ff1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6817","nonce":534880586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6819","nonce":534880586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6823","nonce":534880586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6821","nonce":534880586}]},"public_addr":"172.21.15.70:6817/534880586","cluster_addr":"172.21.15.70:6819/534880586","heartbeat_back_addr":"172.21.15.70:6823/534880586","heartbeat_front_addr":"172.21.15.70:6821/534880586","state":["exists","up"]},{"osd":9,"uuid":"77f47061-0c5f-4159-b276-e814935d6c1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6827","nonce":587047469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6829","nonce":587047469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6833","nonce":587047469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6831","nonce":587047469}]},"public_addr":"172.21.15.31:6827/587047469","cluster_addr":"172.21.15.31:6829/587047469","heartbeat_back_addr":"172.21.15.31:6833/587047469","heartbeat_front_addr":"172.21.15.31:6831/587047469","state":["exists","up"]},{"osd":10,"uuid":"8853dc43-7bd1-4c93-8063-f7982fd7da38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6825","nonce":512108711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6827","nonce":512108711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6831","nonce":512108711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6829","nonce":512108711}]},"public_addr":"172.21.15.40:6825/512108711","cluster_addr":"172.21.15.40:6827/512108711","heartbeat_back_addr":"172.21.15.40:6831/512108711","heartbeat_front_addr":"172.21.15.40:6829/512108711","state":["exists","up"]},{"osd":11,"uuid":"84b7c2de-c424-45a6-9583-60097e372a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6825","nonce":540305718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6827","nonce":540305718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6831","nonce":540305718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6829","nonce":540305718}]},"public_addr":"172.21.15.70:6825/540305718","cluster_addr":"172.21.15.70:6827/540305718","heartbeat_back_addr":"172.21.15.70:6831/540305718","heartbeat_front_addr":"172.21.15.70:6829/540305718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:28:51.477358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:17.405045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:41.626268+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:15.339568+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:42.676996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:07.516999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:42.412273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:10.365041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:35.699120+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:11.733639+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:39.505442+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:34:05.449301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451930554":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/3342588843":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6801/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/1626448389":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/1910059351":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/344822770":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/3811368512":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/1981437651":"2024-03-22T10:26:16.959165+0000","172.21.15.31:6801/1981437651":"2024-03-22T10:26:16.959165+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T10:38:00.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.781+0000 7f18faa00640 1 -- 172.21.15.31:0/3309177159 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f18dc0a2f10 msgr2=0x7f18dc0a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.781+0000 7f18faa00640 1 --2- 172.21.15.31:0/3309177159 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f18dc0a2f10 0x7f18dc0a53d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f18f419afa0 tx=0x7f18e0009260 comp rx=0 tx=0).stop 2024-03-21T10:38:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.781+0000 7f18faa00640 1 -- 172.21.15.31:0/3309177159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f40ffe20 msgr2=0x7f18f4199a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.781+0000 7f18faa00640 1 --2- 172.21.15.31:0/3309177159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f40ffe20 0x7f18f4199a90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f18e80349e0 tx=0x7f18e8032450 comp rx=0 tx=0).stop 2024-03-21T10:38:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.781+0000 7f18faa00640 1 -- 172.21.15.31:0/3309177159 shutdown_connections 2024-03-21T10:38:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.781+0000 7f18faa00640 1 --2- 172.21.15.31:0/3309177159 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f18dc0a2f10 0x7f18dc0a53d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.781+0000 7f18faa00640 1 --2- 172.21.15.31:0/3309177159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18f4101120 0x7f18f41a1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.781+0000 7f18faa00640 1 --2- 172.21.15.31:0/3309177159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18f4100780 0x7f18f4199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.781+0000 7f18faa00640 1 --2- 172.21.15.31:0/3309177159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f40ffe20 0x7f18f4199a90 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.781+0000 7f18faa00640 1 -- 172.21.15.31:0/3309177159 >> 172.21.15.31:0/3309177159 conn(0x7f18f40fba20 msgr2=0x7f18f40fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.781+0000 7f18faa00640 1 -- 172.21.15.31:0/3309177159 shutdown_connections 2024-03-21T10:38:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:00.781+0000 7f18faa00640 1 -- 172.21.15.31:0/3309177159 wait complete. 2024-03-21T10:38:00.806 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T10:29:44.553968+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T10:35:04.945449+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T10:35:05.970589+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T10:35:09.688711+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T10:38:00.807 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T10:38:01.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.001+0000 7fb93ba00640 1 -- 172.21.15.31:0/169186469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb934075660 msgr2=0x7fb934073ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:01.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.001+0000 7fb93ba00640 1 --2- 172.21.15.31:0/169186469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb934075660 0x7fb934073ab0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb928009900 tx=0x7fb92802bcd0 comp rx=0 tx=0).stop 2024-03-21T10:38:01.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.002+0000 7fb93ba00640 1 -- 172.21.15.31:0/169186469 shutdown_connections 2024-03-21T10:38:01.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.002+0000 7fb93ba00640 1 --2- 172.21.15.31:0/169186469 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb934109560 0x7fb93410b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:01.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.002+0000 7fb93ba00640 1 --2- 172.21.15.31:0/169186469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb934074080 0x7fb9340744e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:01.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.002+0000 7fb93ba00640 1 --2- 172.21.15.31:0/169186469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb934075660 0x7fb934073ab0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:01.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.002+0000 7fb93ba00640 1 -- 172.21.15.31:0/169186469 >> 172.21.15.31:0/169186469 conn(0x7fb9340fcc20 msgr2=0x7fb9340ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:01.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.002+0000 7fb93ba00640 1 -- 172.21.15.31:0/169186469 shutdown_connections 2024-03-21T10:38:01.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.002+0000 7fb93ba00640 1 -- 172.21.15.31:0/169186469 wait complete. 2024-03-21T10:38:01.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.003+0000 7fb93ba00640 1 Processor -- start 2024-03-21T10:38:01.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.003+0000 7fb93ba00640 1 -- start start 2024-03-21T10:38:01.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.004+0000 7fb93ba00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb934074080 0x7fb934199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:01.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.004+0000 7fb93ba00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb934075660 0x7fb93419a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:01.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.004+0000 7fb93ba00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb934109560 0x7fb9341a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:01.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.004+0000 7fb93ba00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93410f530 con 0x7fb934075660 2024-03-21T10:38:01.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.004+0000 7fb93ba00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93410f3b0 con 0x7fb934074080 2024-03-21T10:38:01.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.004+0000 7fb93ba00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93410f6b0 con 0x7fb934109560 2024-03-21T10:38:01.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.004+0000 7fb939000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb934074080 0x7fb934199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:01.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.004+0000 7fb933e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb934075660 0x7fb93419a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:01.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.004+0000 7fb939000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb934074080 0x7fb934199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:57402/0 (socket says 172.21.15.31:57402) 2024-03-21T10:38:01.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.004+0000 7fb939000640 1 -- 172.21.15.31:0/4002333534 learned_addr learned my addr 172.21.15.31:0/4002333534 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:01.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.004+0000 7fb939a00640 1 --2- 172.21.15.31:0/4002333534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb934109560 0x7fb9341a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:01.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.005+0000 7fb939000640 1 -- 172.21.15.31:0/4002333534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb934109560 msgr2=0x7fb9341a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:01.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.005+0000 7fb939000640 1 --2- 172.21.15.31:0/4002333534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb934109560 0x7fb9341a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:01.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.005+0000 7fb939000640 1 -- 172.21.15.31:0/4002333534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb934075660 msgr2=0x7fb93419a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:01.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.005+0000 7fb939000640 1 --2- 172.21.15.31:0/4002333534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb934075660 0x7fb93419a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:01.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.005+0000 7fb939000640 1 -- 172.21.15.31:0/4002333534 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9341a17f0 con 0x7fb934074080 2024-03-21T10:38:01.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.005+0000 7fb939000640 1 --2- 172.21.15.31:0/4002333534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb934074080 0x7fb934199bd0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb928005ab0 tx=0x7fb92802c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:01.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.005+0000 7fb931600640 1 -- 172.21.15.31:0/4002333534 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb928041070 con 0x7fb934074080 2024-03-21T10:38:01.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.005+0000 7fb931600640 1 -- 172.21.15.31:0/4002333534 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb928004450 con 0x7fb934074080 2024-03-21T10:38:01.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.005+0000 7fb931600640 1 -- 172.21.15.31:0/4002333534 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb928034440 con 0x7fb934074080 2024-03-21T10:38:01.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.005+0000 7fb93ba00640 1 -- 172.21.15.31:0/4002333534 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9341a1a80 con 0x7fb934074080 2024-03-21T10:38:01.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.005+0000 7fb93ba00640 1 -- 172.21.15.31:0/4002333534 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9341a1f20 con 0x7fb934074080 2024-03-21T10:38:01.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.007+0000 7fb93ba00640 1 -- 172.21.15.31:0/4002333534 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb934108320 con 0x7fb934074080 2024-03-21T10:38:01.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.008+0000 7fb931600640 1 -- 172.21.15.31:0/4002333534 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fb928044070 con 0x7fb934074080 2024-03-21T10:38:01.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.009+0000 7fb931600640 1 --2- 172.21.15.31:0/4002333534 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb91c0a2f20 0x7fb91c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:01.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.010+0000 7fb931600640 1 -- 172.21.15.31:0/4002333534 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fb9280f1240 con 0x7fb934074080 2024-03-21T10:38:01.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.010+0000 7fb933e00640 1 --2- 172.21.15.31:0/4002333534 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb91c0a2f20 0x7fb91c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:01.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.011+0000 7fb933e00640 1 --2- 172.21.15.31:0/4002333534 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb91c0a2f20 0x7fb91c0a53e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb93419b0e0 tx=0x7fb920005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:01.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.014+0000 7fb931600640 1 -- 172.21.15.31:0/4002333534 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb9280bd6f0 con 0x7fb934074080 2024-03-21T10:38:01.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.289+0000 7fb93ba00640 1 -- 172.21.15.31:0/4002333534 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb93402d290 con 0x7fb934074080 2024-03-21T10:38:01.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.290+0000 7fb931600640 1 -- 172.21.15.31:0/4002333534 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb9280c1790 con 0x7fb934074080 2024-03-21T10:38:01.288 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-03-21T10:38:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.293+0000 7fb93ba00640 1 -- 172.21.15.31:0/4002333534 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb91c0a2f20 msgr2=0x7fb91c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.293+0000 7fb93ba00640 1 --2- 172.21.15.31:0/4002333534 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb91c0a2f20 0x7fb91c0a53e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb93419b0e0 tx=0x7fb920005f90 comp rx=0 tx=0).stop 2024-03-21T10:38:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.293+0000 7fb93ba00640 1 -- 172.21.15.31:0/4002333534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb934074080 msgr2=0x7fb934199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.293+0000 7fb93ba00640 1 --2- 172.21.15.31:0/4002333534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb934074080 0x7fb934199bd0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb928005ab0 tx=0x7fb92802c6e0 comp rx=0 tx=0).stop 2024-03-21T10:38:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.293+0000 7fb93ba00640 1 -- 172.21.15.31:0/4002333534 shutdown_connections 2024-03-21T10:38:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.293+0000 7fb93ba00640 1 --2- 172.21.15.31:0/4002333534 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb91c0a2f20 0x7fb91c0a53e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.293+0000 7fb93ba00640 1 --2- 172.21.15.31:0/4002333534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb934109560 0x7fb9341a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.293+0000 7fb93ba00640 1 --2- 172.21.15.31:0/4002333534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb934075660 0x7fb93419a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.293+0000 7fb93ba00640 1 --2- 172.21.15.31:0/4002333534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb934074080 0x7fb934199bd0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.293+0000 7fb93ba00640 1 -- 172.21.15.31:0/4002333534 >> 172.21.15.31:0/4002333534 conn(0x7fb9340fcc20 msgr2=0x7fb9341054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.293+0000 7fb93ba00640 1 -- 172.21.15.31:0/4002333534 shutdown_connections 2024-03-21T10:38:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.293+0000 7fb93ba00640 1 -- 172.21.15.31:0/4002333534 wait complete. 2024-03-21T10:38:01.313 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T10:38:01.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:01 smithi040 ceph-mon[36154]: pgmap v416: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:38:01.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:01 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/4077919767' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:38:01.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:01 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:38:01.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:01 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3309177159' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:38:01.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:01 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:01.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:01 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:01.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:01 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:01.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:01 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:01.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.552+0000 7fdf6ae00640 1 -- 172.21.15.31:0/1854915702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf64104ad0 msgr2=0x7fdf64104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:01.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.552+0000 7fdf6ae00640 1 --2- 172.21.15.31:0/1854915702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf64104ad0 0x7fdf64104f50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fdf50009900 tx=0x7fdf5002bc30 comp rx=0 tx=0).stop 2024-03-21T10:38:01.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.552+0000 7fdf6ae00640 1 -- 172.21.15.31:0/1854915702 shutdown_connections 2024-03-21T10:38:01.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.552+0000 7fdf6ae00640 1 --2- 172.21.15.31:0/1854915702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf64105490 0x7fdf6410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:01.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.552+0000 7fdf6ae00640 1 --2- 172.21.15.31:0/1854915702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf64104ad0 0x7fdf64104f50 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:01.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.552+0000 7fdf6ae00640 1 --2- 172.21.15.31:0/1854915702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf641038e0 0x7fdf64103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:01.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.552+0000 7fdf6ae00640 1 -- 172.21.15.31:0/1854915702 >> 172.21.15.31:0/1854915702 conn(0x7fdf640fee50 msgr2=0x7fdf64101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:01.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.552+0000 7fdf6ae00640 1 -- 172.21.15.31:0/1854915702 shutdown_connections 2024-03-21T10:38:01.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.552+0000 7fdf6ae00640 1 -- 172.21.15.31:0/1854915702 wait complete. 2024-03-21T10:38:01.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.553+0000 7fdf6ae00640 1 Processor -- start 2024-03-21T10:38:01.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.553+0000 7fdf6ae00640 1 -- start start 2024-03-21T10:38:01.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.553+0000 7fdf6ae00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf641038e0 0x7fdf64199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:01.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.553+0000 7fdf6ae00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf64104ad0 0x7fdf6419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:01.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.554+0000 7fdf6ae00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf64105490 0x7fdf641a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:01.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.554+0000 7fdf6ae00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6410f560 con 0x7fdf64105490 2024-03-21T10:38:01.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.554+0000 7fdf6ae00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6410f3e0 con 0x7fdf64104ad0 2024-03-21T10:38:01.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.554+0000 7fdf6ae00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6410f6e0 con 0x7fdf641038e0 2024-03-21T10:38:01.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.554+0000 7fdf63e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf641038e0 0x7fdf64199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:01.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.554+0000 7fdf63e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf641038e0 0x7fdf64199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:40316/0 (socket says 172.21.15.31:40316) 2024-03-21T10:38:01.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.554+0000 7fdf63e00640 1 -- 172.21.15.31:0/3130223435 learned_addr learned my addr 172.21.15.31:0/3130223435 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:01.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.554+0000 7fdf63400640 1 --2- 172.21.15.31:0/3130223435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf64104ad0 0x7fdf6419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:01.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.554+0000 7fdf68e00640 1 --2- 172.21.15.31:0/3130223435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf64105490 0x7fdf641a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:01.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.554+0000 7fdf63e00640 1 -- 172.21.15.31:0/3130223435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf64104ad0 msgr2=0x7fdf6419a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:01.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.554+0000 7fdf63e00640 1 --2- 172.21.15.31:0/3130223435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf64104ad0 0x7fdf6419a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:01.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.554+0000 7fdf63e00640 1 -- 172.21.15.31:0/3130223435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf64105490 msgr2=0x7fdf641a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:01.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.554+0000 7fdf63e00640 1 --2- 172.21.15.31:0/3130223435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf64105490 0x7fdf641a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:01.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.554+0000 7fdf63e00640 1 -- 172.21.15.31:0/3130223435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf641a1830 con 0x7fdf641038e0 2024-03-21T10:38:01.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.554+0000 7fdf68e00640 1 --2- 172.21.15.31:0/3130223435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf64105490 0x7fdf641a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:01.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.554+0000 7fdf63400640 1 --2- 172.21.15.31:0/3130223435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf64104ad0 0x7fdf6419a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:01.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.555+0000 7fdf63e00640 1 --2- 172.21.15.31:0/3130223435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf641038e0 0x7fdf64199c20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdf5400e9c0 tx=0x7fdf5400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:01.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.555+0000 7fdf60c00640 1 -- 172.21.15.31:0/3130223435 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5400cd80 con 0x7fdf641038e0 2024-03-21T10:38:01.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.555+0000 7fdf60c00640 1 -- 172.21.15.31:0/3130223435 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdf540035f0 con 0x7fdf641038e0 2024-03-21T10:38:01.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.555+0000 7fdf6ae00640 1 -- 172.21.15.31:0/3130223435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf641a1b20 con 0x7fdf641038e0 2024-03-21T10:38:01.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.555+0000 7fdf60c00640 1 -- 172.21.15.31:0/3130223435 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf54010690 con 0x7fdf641038e0 2024-03-21T10:38:01.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.555+0000 7fdf6ae00640 1 -- 172.21.15.31:0/3130223435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf641a2020 con 0x7fdf641038e0 2024-03-21T10:38:01.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.557+0000 7fdf6ae00640 1 -- 172.21.15.31:0/3130223435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf6410ac00 con 0x7fdf641038e0 2024-03-21T10:38:01.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:01 smithi031 ceph-mon[31673]: pgmap v416: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:38:01.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:01 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/4077919767' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:38:01.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:01 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:38:01.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:01 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3309177159' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:38:01.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:01 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:01.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:01 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:01.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:01 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:01.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:01 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:01.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.559+0000 7fdf60c00640 1 -- 172.21.15.31:0/3130223435 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fdf540037a0 con 0x7fdf641038e0 2024-03-21T10:38:01.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.561+0000 7fdf60c00640 1 --2- 172.21.15.31:0/3130223435 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fdf480a2f20 0x7fdf480a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:01.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.562+0000 7fdf60c00640 1 -- 172.21.15.31:0/3130223435 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fdf540cfc50 con 0x7fdf641038e0 2024-03-21T10:38:01.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.562+0000 7fdf63400640 1 --2- 172.21.15.31:0/3130223435 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fdf480a2f20 0x7fdf480a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:01.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.563+0000 7fdf63400640 1 --2- 172.21.15.31:0/3130223435 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fdf480a2f20 0x7fdf480a53e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fdf5002c7b0 tx=0x7fdf50005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:01.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.564+0000 7fdf60c00640 1 -- 172.21.15.31:0/3130223435 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdf5409c6c0 con 0x7fdf641038e0 2024-03-21T10:38:01.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:01 smithi070 ceph-mon[36330]: pgmap v416: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:38:01.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:01 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/4077919767' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:38:01.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:01 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:38:01.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:01 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3309177159' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:38:01.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:01 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:01.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:01 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:01.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:01 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:01.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:01 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:01.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.770+0000 7fdf6ae00640 1 -- 172.21.15.31:0/3130223435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdf64062340 con 0x7fdf641038e0 2024-03-21T10:38:01.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.770+0000 7fdf60c00640 1 -- 172.21.15.31:0/3130223435 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdf5409bfc0 con 0x7fdf641038e0 2024-03-21T10:38:01.768 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-03-21T10:38:01.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.773+0000 7fdf6ae00640 1 -- 172.21.15.31:0/3130223435 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fdf480a2f20 msgr2=0x7fdf480a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:01.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.773+0000 7fdf6ae00640 1 --2- 172.21.15.31:0/3130223435 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fdf480a2f20 0x7fdf480a53e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fdf5002c7b0 tx=0x7fdf50005fd0 comp rx=0 tx=0).stop 2024-03-21T10:38:01.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.773+0000 7fdf6ae00640 1 -- 172.21.15.31:0/3130223435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf641038e0 msgr2=0x7fdf64199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:01.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.773+0000 7fdf6ae00640 1 --2- 172.21.15.31:0/3130223435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf641038e0 0x7fdf64199c20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdf5400e9c0 tx=0x7fdf5400ee80 comp rx=0 tx=0).stop 2024-03-21T10:38:01.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.773+0000 7fdf6ae00640 1 -- 172.21.15.31:0/3130223435 shutdown_connections 2024-03-21T10:38:01.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.773+0000 7fdf6ae00640 1 --2- 172.21.15.31:0/3130223435 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fdf480a2f20 0x7fdf480a53e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:01.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.773+0000 7fdf6ae00640 1 --2- 172.21.15.31:0/3130223435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf64105490 0x7fdf641a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:01.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.773+0000 7fdf6ae00640 1 --2- 172.21.15.31:0/3130223435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf64104ad0 0x7fdf6419a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:01.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.773+0000 7fdf6ae00640 1 --2- 172.21.15.31:0/3130223435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf641038e0 0x7fdf64199c20 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:01.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.773+0000 7fdf6ae00640 1 -- 172.21.15.31:0/3130223435 >> 172.21.15.31:0/3130223435 conn(0x7fdf640fee50 msgr2=0x7fdf64107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:01.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.774+0000 7fdf6ae00640 1 -- 172.21.15.31:0/3130223435 shutdown_connections 2024-03-21T10:38:01.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:01.774+0000 7fdf6ae00640 1 -- 172.21.15.31:0/3130223435 wait complete. 2024-03-21T10:38:01.791 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T10:38:02.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.007+0000 7fb68b200640 1 -- 172.21.15.31:0/2435605342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb68c1026e0 msgr2=0x7fb68c102b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:02.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.007+0000 7fb68b200640 1 --2- 172.21.15.31:0/2435605342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb68c1026e0 0x7fb68c102b60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb678009900 tx=0x7fb67802bc30 comp rx=0 tx=0).stop 2024-03-21T10:38:02.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.008+0000 7fb68b200640 1 -- 172.21.15.31:0/2435605342 shutdown_connections 2024-03-21T10:38:02.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.008+0000 7fb68b200640 1 --2- 172.21.15.31:0/2435605342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb68c1030a0 0x7fb68c109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:02.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.008+0000 7fb68b200640 1 --2- 172.21.15.31:0/2435605342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb68c1026e0 0x7fb68c102b60 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:02.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.008+0000 7fb68b200640 1 --2- 172.21.15.31:0/2435605342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb68c1014f0 0x7fb68c101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:02.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.008+0000 7fb68b200640 1 -- 172.21.15.31:0/2435605342 >> 172.21.15.31:0/2435605342 conn(0x7fb68c0fca60 msgr2=0x7fb68c0feea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:02.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.008+0000 7fb68b200640 1 -- 172.21.15.31:0/2435605342 shutdown_connections 2024-03-21T10:38:02.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.008+0000 7fb68b200640 1 -- 172.21.15.31:0/2435605342 wait complete. 2024-03-21T10:38:02.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.009+0000 7fb68b200640 1 Processor -- start 2024-03-21T10:38:02.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.009+0000 7fb68b200640 1 -- start start 2024-03-21T10:38:02.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.009+0000 7fb68b200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb68c1014f0 0x7fb68c199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:02.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.009+0000 7fb68b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb68c1026e0 0x7fb68c199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:02.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.009+0000 7fb68b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb68c1030a0 0x7fb68c1a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:02.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.009+0000 7fb68b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c10d180 con 0x7fb68c1026e0 2024-03-21T10:38:02.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.009+0000 7fb68b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c10d000 con 0x7fb68c1030a0 2024-03-21T10:38:02.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.009+0000 7fb68b200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c10d300 con 0x7fb68c1014f0 2024-03-21T10:38:02.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.009+0000 7fb689e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb68c1014f0 0x7fb68c199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:02.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.009+0000 7fb689400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb68c1026e0 0x7fb68c199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:02.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.009+0000 7fb689e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb68c1014f0 0x7fb68c199a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:40332/0 (socket says 172.21.15.31:40332) 2024-03-21T10:38:02.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.009+0000 7fb689e00640 1 -- 172.21.15.31:0/3968065017 learned_addr learned my addr 172.21.15.31:0/3968065017 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:02.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.010+0000 7fb68a800640 1 --2- 172.21.15.31:0/3968065017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb68c1030a0 0x7fb68c1a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:02.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.010+0000 7fb689e00640 1 -- 172.21.15.31:0/3968065017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb68c1030a0 msgr2=0x7fb68c1a0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:02.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.010+0000 7fb689e00640 1 --2- 172.21.15.31:0/3968065017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb68c1030a0 0x7fb68c1a0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:02.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.010+0000 7fb689e00640 1 -- 172.21.15.31:0/3968065017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb68c1026e0 msgr2=0x7fb68c199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:02.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.010+0000 7fb689e00640 1 --2- 172.21.15.31:0/3968065017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb68c1026e0 0x7fb68c199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:02.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.010+0000 7fb689e00640 1 -- 172.21.15.31:0/3968065017 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68c1a15d0 con 0x7fb68c1014f0 2024-03-21T10:38:02.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.010+0000 7fb68a800640 1 --2- 172.21.15.31:0/3968065017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb68c1030a0 0x7fb68c1a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:02.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.010+0000 7fb689e00640 1 --2- 172.21.15.31:0/3968065017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb68c1014f0 0x7fb68c199a50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb67c00e9c0 tx=0x7fb67c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:02.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.010+0000 7fb682a00640 1 -- 172.21.15.31:0/3968065017 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c00cd80 con 0x7fb68c1014f0 2024-03-21T10:38:02.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.010+0000 7fb682a00640 1 -- 172.21.15.31:0/3968065017 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb67c0035f0 con 0x7fb68c1014f0 2024-03-21T10:38:02.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.011+0000 7fb68b200640 1 -- 172.21.15.31:0/3968065017 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb68c1a18c0 con 0x7fb68c1014f0 2024-03-21T10:38:02.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.011+0000 7fb682a00640 1 -- 172.21.15.31:0/3968065017 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c010690 con 0x7fb68c1014f0 2024-03-21T10:38:02.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.011+0000 7fb68b200640 1 -- 172.21.15.31:0/3968065017 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb68c1a1d90 con 0x7fb68c1014f0 2024-03-21T10:38:02.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.012+0000 7fb68b200640 1 -- 172.21.15.31:0/3968065017 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb68c1a1ff0 con 0x7fb68c1014f0 2024-03-21T10:38:02.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.014+0000 7fb682a00640 1 -- 172.21.15.31:0/3968065017 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fb67c0037a0 con 0x7fb68c1014f0 2024-03-21T10:38:02.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.016+0000 7fb682a00640 1 --2- 172.21.15.31:0/3968065017 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb6700a2f20 0x7fb6700a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:02.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.016+0000 7fb682a00640 1 -- 172.21.15.31:0/3968065017 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fb67c0cfbc0 con 0x7fb68c1014f0 2024-03-21T10:38:02.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.016+0000 7fb689400640 1 --2- 172.21.15.31:0/3968065017 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb6700a2f20 0x7fb6700a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:02.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.017+0000 7fb689400640 1 --2- 172.21.15.31:0/3968065017 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb6700a2f20 0x7fb6700a53e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb67802c7b0 tx=0x7fb678005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:02.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.020+0000 7fb682a00640 1 -- 172.21.15.31:0/3968065017 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb67c09c630 con 0x7fb68c1014f0 2024-03-21T10:38:02.326 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:02 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/4002333534' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:38:02.326 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:02 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:02.326 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:02 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:02.326 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:02 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3130223435' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:38:02.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.328+0000 7fb68b200640 1 -- 172.21.15.31:0/3968065017 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb68c1a21c0 con 0x7fb68c1014f0 2024-03-21T10:38:02.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.328+0000 7fb682a00640 1 -- 172.21.15.31:0/3968065017 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb67c09bf30 con 0x7fb68c1014f0 2024-03-21T10:38:02.327 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-03-21T10:38:02.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.332+0000 7fb68b200640 1 -- 172.21.15.31:0/3968065017 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb6700a2f20 msgr2=0x7fb6700a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:02.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.332+0000 7fb68b200640 1 --2- 172.21.15.31:0/3968065017 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb6700a2f20 0x7fb6700a53e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb67802c7b0 tx=0x7fb678005fd0 comp rx=0 tx=0).stop 2024-03-21T10:38:02.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.332+0000 7fb68b200640 1 -- 172.21.15.31:0/3968065017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb68c1014f0 msgr2=0x7fb68c199a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:02.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.332+0000 7fb68b200640 1 --2- 172.21.15.31:0/3968065017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb68c1014f0 0x7fb68c199a50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb67c00e9c0 tx=0x7fb67c00ee80 comp rx=0 tx=0).stop 2024-03-21T10:38:02.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.332+0000 7fb68b200640 1 -- 172.21.15.31:0/3968065017 shutdown_connections 2024-03-21T10:38:02.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.332+0000 7fb68b200640 1 --2- 172.21.15.31:0/3968065017 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb6700a2f20 0x7fb6700a53e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:02.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.332+0000 7fb68b200640 1 --2- 172.21.15.31:0/3968065017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb68c1030a0 0x7fb68c1a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:02.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.332+0000 7fb68b200640 1 --2- 172.21.15.31:0/3968065017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb68c1026e0 0x7fb68c199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:02.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.332+0000 7fb68b200640 1 --2- 172.21.15.31:0/3968065017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb68c1014f0 0x7fb68c199a50 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:02.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.332+0000 7fb68b200640 1 -- 172.21.15.31:0/3968065017 >> 172.21.15.31:0/3968065017 conn(0x7fb68c0fca60 msgr2=0x7fb68c0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:02.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.332+0000 7fb68b200640 1 -- 172.21.15.31:0/3968065017 shutdown_connections 2024-03-21T10:38:02.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.332+0000 7fb68b200640 1 -- 172.21.15.31:0/3968065017 wait complete. 2024-03-21T10:38:02.350 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T10:38:02.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:02 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/4002333534' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:38:02.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:02 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:02.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:02 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:02.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:02 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3130223435' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:38:02.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.543+0000 7ff57f000640 1 -- 172.21.15.31:0/1158131001 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5780fff90 msgr2=0x7ff5781003b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:02.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.543+0000 7ff57f000640 1 --2- 172.21.15.31:0/1158131001 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5780fff90 0x7ff5781003b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff568009900 tx=0x7ff56802bc40 comp rx=0 tx=0).stop 2024-03-21T10:38:02.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.543+0000 7ff57f000640 1 -- 172.21.15.31:0/1158131001 shutdown_connections 2024-03-21T10:38:02.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.543+0000 7ff57f000640 1 --2- 172.21.15.31:0/1158131001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff578101290 0x7ff578101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:02.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.543+0000 7ff57f000640 1 --2- 172.21.15.31:0/1158131001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5781008f0 0x7ff578100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:02.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.543+0000 7ff57f000640 1 --2- 172.21.15.31:0/1158131001 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5780fff90 0x7ff5781003b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:02.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.543+0000 7ff57f000640 1 -- 172.21.15.31:0/1158131001 >> 172.21.15.31:0/1158131001 conn(0x7ff5780fbb30 msgr2=0x7ff5780fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:02.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.543+0000 7ff57f000640 1 -- 172.21.15.31:0/1158131001 shutdown_connections 2024-03-21T10:38:02.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.543+0000 7ff57f000640 1 -- 172.21.15.31:0/1158131001 wait complete. 2024-03-21T10:38:02.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.544+0000 7ff57f000640 1 Processor -- start 2024-03-21T10:38:02.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.544+0000 7ff57f000640 1 -- start start 2024-03-21T10:38:02.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.545+0000 7ff57f000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5780fff90 0x7ff578199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:02.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.545+0000 7ff57f000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5781008f0 0x7ff57819a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:02.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.545+0000 7ff57f000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff578101290 0x7ff5781a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:02.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.545+0000 7ff57f000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57810f500 con 0x7ff5780fff90 2024-03-21T10:38:02.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.545+0000 7ff57f000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57810f380 con 0x7ff578101290 2024-03-21T10:38:02.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.545+0000 7ff57f000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57810f680 con 0x7ff5781008f0 2024-03-21T10:38:02.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.545+0000 7ff57d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff578101290 0x7ff5781a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:02.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.545+0000 7ff577e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5780fff90 0x7ff578199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:02.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.545+0000 7ff577400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5781008f0 0x7ff57819a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:02.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.545+0000 7ff57d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff578101290 0x7ff5781a1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:57434/0 (socket says 172.21.15.31:57434) 2024-03-21T10:38:02.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.545+0000 7ff57d000640 1 -- 172.21.15.31:0/1906633441 learned_addr learned my addr 172.21.15.31:0/1906633441 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:02.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.545+0000 7ff57d000640 1 -- 172.21.15.31:0/1906633441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5781008f0 msgr2=0x7ff57819a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:02.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.545+0000 7ff57d000640 1 --2- 172.21.15.31:0/1906633441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5781008f0 0x7ff57819a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:02.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.545+0000 7ff57d000640 1 -- 172.21.15.31:0/1906633441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5780fff90 msgr2=0x7ff578199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:02.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.545+0000 7ff57d000640 1 --2- 172.21.15.31:0/1906633441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5780fff90 0x7ff578199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:02.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.545+0000 7ff57d000640 1 -- 172.21.15.31:0/1906633441 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5781a1790 con 0x7ff578101290 2024-03-21T10:38:02.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.546+0000 7ff57d000640 1 --2- 172.21.15.31:0/1906633441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff578101290 0x7ff5781a1120 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff57000eb10 tx=0x7ff57000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:02.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.546+0000 7ff574c00640 1 -- 172.21.15.31:0/1906633441 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff570004b20 con 0x7ff578101290 2024-03-21T10:38:02.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.546+0000 7ff574c00640 1 -- 172.21.15.31:0/1906633441 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff570004cc0 con 0x7ff578101290 2024-03-21T10:38:02.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.546+0000 7ff57f000640 1 -- 172.21.15.31:0/1906633441 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5781a1a80 con 0x7ff578101290 2024-03-21T10:38:02.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.546+0000 7ff574c00640 1 -- 172.21.15.31:0/1906633441 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff570010760 con 0x7ff578101290 2024-03-21T10:38:02.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.546+0000 7ff57f000640 1 -- 172.21.15.31:0/1906633441 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5781a1f80 con 0x7ff578101290 2024-03-21T10:38:02.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.547+0000 7ff57f000640 1 -- 172.21.15.31:0/1906633441 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff57810ace0 con 0x7ff578101290 2024-03-21T10:38:02.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.549+0000 7ff574c00640 1 -- 172.21.15.31:0/1906633441 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7ff570044b40 con 0x7ff578101290 2024-03-21T10:38:02.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.550+0000 7ff574c00640 1 --2- 172.21.15.31:0/1906633441 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff55c0a2f20 0x7ff55c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:02.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.550+0000 7ff574c00640 1 -- 172.21.15.31:0/1906633441 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7ff5700cfd00 con 0x7ff578101290 2024-03-21T10:38:02.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.551+0000 7ff577e00640 1 --2- 172.21.15.31:0/1906633441 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff55c0a2f20 0x7ff55c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:02.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.551+0000 7ff577e00640 1 --2- 172.21.15.31:0/1906633441 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff55c0a2f20 0x7ff55c0a53e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff56802c7c0 tx=0x7ff568005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:02.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.553+0000 7ff574c00640 1 -- 172.21.15.31:0/1906633441 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff57009c770 con 0x7ff578101290 2024-03-21T10:38:02.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:02 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/4002333534' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:38:02.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:02 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:02.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:02 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:02.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:02 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3130223435' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:38:02.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.770+0000 7ff57f000640 1 -- 172.21.15.31:0/1906633441 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff57802d2c0 con 0x7ff578101290 2024-03-21T10:38:02.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.771+0000 7ff574c00640 1 -- 172.21.15.31:0/1906633441 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff57009c070 con 0x7ff578101290 2024-03-21T10:38:02.769 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-03-21T10:38:02.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.774+0000 7ff57f000640 1 -- 172.21.15.31:0/1906633441 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff55c0a2f20 msgr2=0x7ff55c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:02.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.774+0000 7ff57f000640 1 --2- 172.21.15.31:0/1906633441 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff55c0a2f20 0x7ff55c0a53e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff56802c7c0 tx=0x7ff568005fd0 comp rx=0 tx=0).stop 2024-03-21T10:38:02.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.774+0000 7ff57f000640 1 -- 172.21.15.31:0/1906633441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff578101290 msgr2=0x7ff5781a1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:02.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.774+0000 7ff57f000640 1 --2- 172.21.15.31:0/1906633441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff578101290 0x7ff5781a1120 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff57000eb10 tx=0x7ff57000c2d0 comp rx=0 tx=0).stop 2024-03-21T10:38:02.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.774+0000 7ff57f000640 1 -- 172.21.15.31:0/1906633441 shutdown_connections 2024-03-21T10:38:02.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.774+0000 7ff57f000640 1 --2- 172.21.15.31:0/1906633441 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff55c0a2f20 0x7ff55c0a53e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:02.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.774+0000 7ff57f000640 1 --2- 172.21.15.31:0/1906633441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff578101290 0x7ff5781a1120 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:02.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.774+0000 7ff57f000640 1 --2- 172.21.15.31:0/1906633441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5781008f0 0x7ff57819a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:02.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.774+0000 7ff57f000640 1 --2- 172.21.15.31:0/1906633441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5780fff90 0x7ff578199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:02.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.774+0000 7ff57f000640 1 -- 172.21.15.31:0/1906633441 >> 172.21.15.31:0/1906633441 conn(0x7ff5780fbb30 msgr2=0x7ff5780fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:02.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.774+0000 7ff57f000640 1 -- 172.21.15.31:0/1906633441 shutdown_connections 2024-03-21T10:38:02.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.774+0000 7ff57f000640 1 -- 172.21.15.31:0/1906633441 wait complete. 2024-03-21T10:38:02.792 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:38:02.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.992+0000 7f3b31e00640 1 -- 172.21.15.31:0/1891244854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b2c103960 msgr2=0x7f3b2c103d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:02.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.992+0000 7f3b31e00640 1 --2- 172.21.15.31:0/1891244854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b2c103960 0x7f3b2c103d80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c009900 tx=0x7f3b1c02bc30 comp rx=0 tx=0).stop 2024-03-21T10:38:02.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.992+0000 7f3b31e00640 1 -- 172.21.15.31:0/1891244854 shutdown_connections 2024-03-21T10:38:02.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.992+0000 7f3b31e00640 1 --2- 172.21.15.31:0/1891244854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b2c105510 0x7f3b2c10bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:02.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.992+0000 7f3b31e00640 1 --2- 172.21.15.31:0/1891244854 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b2c104b50 0x7f3b2c104fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:02.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.992+0000 7f3b31e00640 1 --2- 172.21.15.31:0/1891244854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b2c103960 0x7f3b2c103d80 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:02.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.992+0000 7f3b31e00640 1 -- 172.21.15.31:0/1891244854 >> 172.21.15.31:0/1891244854 conn(0x7f3b2c0feeb0 msgr2=0x7f3b2c101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:02.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.992+0000 7f3b31e00640 1 -- 172.21.15.31:0/1891244854 shutdown_connections 2024-03-21T10:38:02.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.992+0000 7f3b31e00640 1 -- 172.21.15.31:0/1891244854 wait complete. 2024-03-21T10:38:02.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.993+0000 7f3b31e00640 1 Processor -- start 2024-03-21T10:38:02.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.994+0000 7f3b31e00640 1 -- start start 2024-03-21T10:38:02.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.994+0000 7f3b31e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b2c103960 0x7f3b2c199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:02.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.994+0000 7f3b31e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b2c104b50 0x7f3b2c19a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:02.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.994+0000 7f3b31e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b2c105510 0x7f3b2c1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:02.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.994+0000 7f3b31e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c10f5c0 con 0x7f3b2c103960 2024-03-21T10:38:02.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.994+0000 7f3b31e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c10f440 con 0x7f3b2c105510 2024-03-21T10:38:02.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.994+0000 7f3b31e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c10f740 con 0x7f3b2c104b50 2024-03-21T10:38:02.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.994+0000 7f3b2aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b2c103960 0x7f3b2c199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:02.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.994+0000 7f3b2b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b2c105510 0x7f3b2c1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:02.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.995+0000 7f3b2a000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b2c104b50 0x7f3b2c19a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:02.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.995+0000 7f3b2aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b2c103960 0x7f3b2c199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39866/0 (socket says 172.21.15.31:39866) 2024-03-21T10:38:02.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.995+0000 7f3b2aa00640 1 -- 172.21.15.31:0/3603250068 learned_addr learned my addr 172.21.15.31:0/3603250068 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:02.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.995+0000 7f3b2b400640 1 -- 172.21.15.31:0/3603250068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b2c104b50 msgr2=0x7f3b2c19a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:02.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.995+0000 7f3b2b400640 1 --2- 172.21.15.31:0/3603250068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b2c104b50 0x7f3b2c19a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:02.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.995+0000 7f3b2b400640 1 -- 172.21.15.31:0/3603250068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b2c103960 msgr2=0x7f3b2c199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:02.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.995+0000 7f3b2b400640 1 --2- 172.21.15.31:0/3603250068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b2c103960 0x7f3b2c199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:02.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.995+0000 7f3b2b400640 1 -- 172.21.15.31:0/3603250068 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c1a1830 con 0x7f3b2c105510 2024-03-21T10:38:02.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.995+0000 7f3b2a000640 1 --2- 172.21.15.31:0/3603250068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b2c104b50 0x7f3b2c19a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:02.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.995+0000 7f3b2b400640 1 --2- 172.21.15.31:0/3603250068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b2c105510 0x7f3b2c1a1190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009fd0 tx=0x7f3b2400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:02.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.995+0000 7f3b30a00640 1 -- 172.21.15.31:0/3603250068 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2400ce20 con 0x7f3b2c105510 2024-03-21T10:38:02.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.995+0000 7f3b30a00640 1 -- 172.21.15.31:0/3603250068 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3b24005020 con 0x7f3b2c105510 2024-03-21T10:38:02.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.995+0000 7f3b30a00640 1 -- 172.21.15.31:0/3603250068 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b24010720 con 0x7f3b2c105510 2024-03-21T10:38:02.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.995+0000 7f3b31e00640 1 -- 172.21.15.31:0/3603250068 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b2c1a1b20 con 0x7f3b2c105510 2024-03-21T10:38:02.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.996+0000 7f3b31e00640 1 -- 172.21.15.31:0/3603250068 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b2c1a2020 con 0x7f3b2c105510 2024-03-21T10:38:02.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.997+0000 7f3b31e00640 1 -- 172.21.15.31:0/3603250068 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b2c10ac20 con 0x7f3b2c105510 2024-03-21T10:38:02.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:02.998+0000 7f3b30a00640 1 -- 172.21.15.31:0/3603250068 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f3b24005540 con 0x7f3b2c105510 2024-03-21T10:38:02.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.000+0000 7f3b30a00640 1 --2- 172.21.15.31:0/3603250068 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f3b100a2fa0 0x7f3b100a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:02.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.000+0000 7f3b30a00640 1 -- 172.21.15.31:0/3603250068 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f3b240cfd90 con 0x7f3b2c105510 2024-03-21T10:38:02.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.000+0000 7f3b2aa00640 1 --2- 172.21.15.31:0/3603250068 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f3b100a2fa0 0x7f3b100a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:02.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.001+0000 7f3b2aa00640 1 --2- 172.21.15.31:0/3603250068 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f3b100a2fa0 0x7f3b100a5460 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c19b0f0 tx=0x7f3b1c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:03.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.004+0000 7f3b30a00640 1 -- 172.21.15.31:0/3603250068 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3b2409c800 con 0x7f3b2c105510 2024-03-21T10:38:03.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.255+0000 7f3b31e00640 1 -- 172.21.15.31:0/3603250068 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b2c02d070 con 0x7f3b2c105510 2024-03-21T10:38:03.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.256+0000 7f3b30a00640 1 -- 172.21.15.31:0/3603250068 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11384 (secure 0 0 0) 0x7f3b2409c100 con 0x7f3b2c105510 2024-03-21T10:38:03.254 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:38:03.254 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:35:40.990205+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24596,24547]},"id":1}]} 2024-03-21T10:38:03.255 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2024-03-21T10:38:03.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.260+0000 7f3b31e00640 1 -- 172.21.15.31:0/3603250068 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f3b100a2fa0 msgr2=0x7f3b100a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:03.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.260+0000 7f3b31e00640 1 --2- 172.21.15.31:0/3603250068 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f3b100a2fa0 0x7f3b100a5460 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c19b0f0 tx=0x7f3b1c02df90 comp rx=0 tx=0).stop 2024-03-21T10:38:03.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.260+0000 7f3b31e00640 1 -- 172.21.15.31:0/3603250068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b2c105510 msgr2=0x7f3b2c1a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:03.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.260+0000 7f3b31e00640 1 --2- 172.21.15.31:0/3603250068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b2c105510 0x7f3b2c1a1190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009fd0 tx=0x7f3b2400ef10 comp rx=0 tx=0).stop 2024-03-21T10:38:03.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.260+0000 7f3b31e00640 1 -- 172.21.15.31:0/3603250068 shutdown_connections 2024-03-21T10:38:03.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.260+0000 7f3b31e00640 1 --2- 172.21.15.31:0/3603250068 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f3b100a2fa0 0x7f3b100a5460 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.260+0000 7f3b31e00640 1 --2- 172.21.15.31:0/3603250068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b2c105510 0x7f3b2c1a1190 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.260+0000 7f3b31e00640 1 --2- 172.21.15.31:0/3603250068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b2c104b50 0x7f3b2c19a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.260+0000 7f3b31e00640 1 --2- 172.21.15.31:0/3603250068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b2c103960 0x7f3b2c199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.260+0000 7f3b31e00640 1 -- 172.21.15.31:0/3603250068 >> 172.21.15.31:0/3603250068 conn(0x7f3b2c0feeb0 msgr2=0x7f3b2c107df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:03.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.260+0000 7f3b31e00640 1 -- 172.21.15.31:0/3603250068 shutdown_connections 2024-03-21T10:38:03.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.260+0000 7f3b31e00640 1 -- 172.21.15.31:0/3603250068 wait complete. 2024-03-21T10:38:03.272 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:03 smithi031 ceph-mon[31673]: pgmap v417: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:38:03.272 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:03 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3968065017' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:38:03.272 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:03 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1906633441' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:38:03.272 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:03 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:03.272 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:03 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:03.272 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:03 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:03.272 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:03 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:03.276 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T10:38:03.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:03 smithi040 ceph-mon[36154]: pgmap v417: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:38:03.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:03 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3968065017' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:38:03.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:03 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1906633441' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:38:03.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:03 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:03.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:03 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:03.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:03 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:03.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:03 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:03.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.489+0000 7f16fe200640 1 -- 172.21.15.31:0/2899551564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16f8103950 msgr2=0x7f16f8103d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:03.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.489+0000 7f16fe200640 1 --2- 172.21.15.31:0/2899551564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16f8103950 0x7f16f8103d70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f16e8009900 tx=0x7f16e802bc30 comp rx=0 tx=0).stop 2024-03-21T10:38:03.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.489+0000 7f16fe200640 1 -- 172.21.15.31:0/2899551564 shutdown_connections 2024-03-21T10:38:03.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.489+0000 7f16fe200640 1 --2- 172.21.15.31:0/2899551564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16f8105500 0x7f16f810bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.489+0000 7f16fe200640 1 --2- 172.21.15.31:0/2899551564 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16f8104b40 0x7f16f8104fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.489+0000 7f16fe200640 1 --2- 172.21.15.31:0/2899551564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16f8103950 0x7f16f8103d70 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.489+0000 7f16fe200640 1 -- 172.21.15.31:0/2899551564 >> 172.21.15.31:0/2899551564 conn(0x7f16f80fef00 msgr2=0x7f16f8101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:03.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.489+0000 7f16fe200640 1 -- 172.21.15.31:0/2899551564 shutdown_connections 2024-03-21T10:38:03.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.489+0000 7f16fe200640 1 -- 172.21.15.31:0/2899551564 wait complete. 2024-03-21T10:38:03.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.490+0000 7f16fe200640 1 Processor -- start 2024-03-21T10:38:03.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.490+0000 7f16fe200640 1 -- start start 2024-03-21T10:38:03.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.491+0000 7f16fe200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16f8103950 0x7f16f8199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:03.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.491+0000 7f16fe200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16f8104b40 0x7f16f819a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:03.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.491+0000 7f16fe200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16f8105500 0x7f16f81a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:03.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.491+0000 7f16fe200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f810f5a0 con 0x7f16f8105500 2024-03-21T10:38:03.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.491+0000 7f16fe200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f810f420 con 0x7f16f8104b40 2024-03-21T10:38:03.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.491+0000 7f16fe200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f810f720 con 0x7f16f8103950 2024-03-21T10:38:03.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.491+0000 7f16f7400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16f8103950 0x7f16f8199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:03.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.491+0000 7f16f6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16f8104b40 0x7f16f819a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:03.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.491+0000 7f16f7400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16f8103950 0x7f16f8199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:40394/0 (socket says 172.21.15.31:40394) 2024-03-21T10:38:03.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.491+0000 7f16f6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16f8104b40 0x7f16f819a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:57480/0 (socket says 172.21.15.31:57480) 2024-03-21T10:38:03.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.491+0000 7f16f7400640 1 -- 172.21.15.31:0/341144963 learned_addr learned my addr 172.21.15.31:0/341144963 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:03.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.491+0000 7f16f7e00640 1 --2- 172.21.15.31:0/341144963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16f8105500 0x7f16f81a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:03.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.492+0000 7f16f7400640 1 -- 172.21.15.31:0/341144963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16f8104b40 msgr2=0x7f16f819a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:03.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.492+0000 7f16f7400640 1 --2- 172.21.15.31:0/341144963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16f8104b40 0x7f16f819a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.492+0000 7f16f7400640 1 -- 172.21.15.31:0/341144963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16f8105500 msgr2=0x7f16f81a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:03.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.492+0000 7f16f7400640 1 --2- 172.21.15.31:0/341144963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16f8105500 0x7f16f81a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.492+0000 7f16f7400640 1 -- 172.21.15.31:0/341144963 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f81a1810 con 0x7f16f8103950 2024-03-21T10:38:03.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.492+0000 7f16f6a00640 1 --2- 172.21.15.31:0/341144963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16f8104b40 0x7f16f819a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:03.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.492+0000 7f16f7400640 1 --2- 172.21.15.31:0/341144963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16f8103950 0x7f16f8199bf0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f16e8000c00 tx=0x7f16e802c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:03.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.492+0000 7f16efe00640 1 -- 172.21.15.31:0/341144963 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e8043020 con 0x7f16f8103950 2024-03-21T10:38:03.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.492+0000 7f16fe200640 1 -- 172.21.15.31:0/341144963 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f81a1aa0 con 0x7f16f8103950 2024-03-21T10:38:03.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.492+0000 7f16efe00640 1 -- 172.21.15.31:0/341144963 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f16e80054e0 con 0x7f16f8103950 2024-03-21T10:38:03.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.492+0000 7f16efe00640 1 -- 172.21.15.31:0/341144963 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e8003ce0 con 0x7f16f8103950 2024-03-21T10:38:03.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.492+0000 7f16fe200640 1 -- 172.21.15.31:0/341144963 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f81a2010 con 0x7f16f8103950 2024-03-21T10:38:03.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.494+0000 7f16fe200640 1 -- 172.21.15.31:0/341144963 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f810ac70 con 0x7f16f8103950 2024-03-21T10:38:03.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.496+0000 7f16efe00640 1 -- 172.21.15.31:0/341144963 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f16e8046020 con 0x7f16f8103950 2024-03-21T10:38:03.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.497+0000 7f16efe00640 1 --2- 172.21.15.31:0/341144963 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f16dc0a2fa0 0x7f16dc0a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:03.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.497+0000 7f16efe00640 1 -- 172.21.15.31:0/341144963 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f16e80f17f0 con 0x7f16f8103950 2024-03-21T10:38:03.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.497+0000 7f16f6a00640 1 --2- 172.21.15.31:0/341144963 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f16dc0a2fa0 0x7f16dc0a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:03.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.498+0000 7f16f6a00640 1 --2- 172.21.15.31:0/341144963 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f16dc0a2fa0 0x7f16dc0a5460 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f16f819b100 tx=0x7f16e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:03.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.505+0000 7f16efe00640 1 -- 172.21.15.31:0/341144963 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f16e804d020 con 0x7f16f8103950 2024-03-21T10:38:03.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:03 smithi070 ceph-mon[36330]: pgmap v417: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:38:03.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:03 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3968065017' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:38:03.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:03 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1906633441' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:38:03.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:03 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:03.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:03 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:03.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:03 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:03.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:03 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:03.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.725+0000 7f16fe200640 1 -- 172.21.15.31:0/341144963 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16f80622a0 con 0x7f16f8103950 2024-03-21T10:38:03.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.728+0000 7f16efe00640 1 -- 172.21.15.31:0/341144963 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24553 (secure 0 0 0) 0x7f16e80be260 con 0x7f16f8103950 2024-03-21T10:38:03.726 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:38:03.726 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":93,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","created":"2024-03-21T10:25:41.971117+0000","modified":"2024-03-21T10:36:56.301695+0000","last_up_change":"2024-03-21T10:34:07.757196+0000","last_in_change":"2024-03-21T10:33:47.460406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T10:29:44.553968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T10:35:04.945449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T10:35:05.970589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T10:35:09.688711+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e0244d33-00e2-47d5-9f66-34efc138c462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6803","nonce":3089885125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6805","nonce":3089885125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6809","nonce":3089885125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6807","nonce":3089885125}]},"public_addr":"172.21.15.31:6803/3089885125","cluster_addr":"172.21.15.31:6805/3089885125","heartbeat_back_addr":"172.21.15.31:6809/3089885125","heartbeat_front_addr":"172.21.15.31:6807/3089885125","state":["exists","up"]},{"osd":1,"uuid":"76a63504-beab-4354-a6c3-eebc3b16ad03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6801","nonce":618840037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6803","nonce":618840037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6807","nonce":618840037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6805","nonce":618840037}]},"public_addr":"172.21.15.40:6801/618840037","cluster_addr":"172.21.15.40:6803/618840037","heartbeat_back_addr":"172.21.15.40:6807/618840037","heartbeat_front_addr":"172.21.15.40:6805/618840037","state":["exists","up"]},{"osd":2,"uuid":"fa191bd6-9a21-4ee2-8adf-8a63c9cfd821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6801","nonce":2524184117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6803","nonce":2524184117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6807","nonce":2524184117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6805","nonce":2524184117}]},"public_addr":"172.21.15.70:6801/2524184117","cluster_addr":"172.21.15.70:6803/2524184117","heartbeat_back_addr":"172.21.15.70:6807/2524184117","heartbeat_front_addr":"172.21.15.70:6805/2524184117","state":["exists","up"]},{"osd":3,"uuid":"edb44507-ffb0-42a4-827b-08461e1bf952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6811","nonce":2823036296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6813","nonce":2823036296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6817","nonce":2823036296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6815","nonce":2823036296}]},"public_addr":"172.21.15.31:6811/2823036296","cluster_addr":"172.21.15.31:6813/2823036296","heartbeat_back_addr":"172.21.15.31:6817/2823036296","heartbeat_front_addr":"172.21.15.31:6815/2823036296","state":["exists","up"]},{"osd":4,"uuid":"8cb89153-8c3b-431f-b57c-8863fada76dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6809","nonce":4215186309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6811","nonce":4215186309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6815","nonce":4215186309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6813","nonce":4215186309}]},"public_addr":"172.21.15.40:6809/4215186309","cluster_addr":"172.21.15.40:6811/4215186309","heartbeat_back_addr":"172.21.15.40:6815/4215186309","heartbeat_front_addr":"172.21.15.40:6813/4215186309","state":["exists","up"]},{"osd":5,"uuid":"9409505a-d463-408c-a3cd-218380c04d68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6809","nonce":3402830452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6811","nonce":3402830452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6815","nonce":3402830452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6813","nonce":3402830452}]},"public_addr":"172.21.15.70:6809/3402830452","cluster_addr":"172.21.15.70:6811/3402830452","heartbeat_back_addr":"172.21.15.70:6815/3402830452","heartbeat_front_addr":"172.21.15.70:6813/3402830452","state":["exists","up"]},{"osd":6,"uuid":"f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6819","nonce":1193497058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6821","nonce":1193497058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6825","nonce":1193497058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6823","nonce":1193497058}]},"public_addr":"172.21.15.31:6819/1193497058","cluster_addr":"172.21.15.31:6821/1193497058","heartbeat_back_addr":"172.21.15.31:6825/1193497058","heartbeat_front_addr":"172.21.15.31:6823/1193497058","state":["exists","up"]},{"osd":7,"uuid":"db228a87-29a5-45d4-8fe6-a0a554262eb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6817","nonce":3825628119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6819","nonce":3825628119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6823","nonce":3825628119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6821","nonce":3825628119}]},"public_addr":"172.21.15.40:6817/3825628119","cluster_addr":"172.21.15.40:6819/3825628119","heartbeat_back_addr":"172.21.15.40:6823/3825628119","heartbeat_front_addr":"172.21.15.40:6821/3825628119","state":["exists","up"]},{"osd":8,"uuid":"59b9fde0-095d-4e0b-87da-ead37bd8ff1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6817","nonce":534880586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6819","nonce":534880586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6823","nonce":534880586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6821","nonce":534880586}]},"public_addr":"172.21.15.70:6817/534880586","cluster_addr":"172.21.15.70:6819/534880586","heartbeat_back_addr":"172.21.15.70:6823/534880586","heartbeat_front_addr":"172.21.15.70:6821/534880586","state":["exists","up"]},{"osd":9,"uuid":"77f47061-0c5f-4159-b276-e814935d6c1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6827","nonce":587047469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6829","nonce":587047469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6833","nonce":587047469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6831","nonce":587047469}]},"public_addr":"172.21.15.31:6827/587047469","cluster_addr":"172.21.15.31:6829/587047469","heartbeat_back_addr":"172.21.15.31:6833/587047469","heartbeat_front_addr":"172.21.15.31:6831/587047469","state":["exists","up"]},{"osd":10,"uuid":"8853dc43-7bd1-4c93-8063-f7982fd7da38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6825","nonce":512108711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6827","nonce":512108711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6831","nonce":512108711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6829","nonce":512108711}]},"public_addr":"172.21.15.40:6825/512108711","cluster_addr":"172.21.15.40:6827/512108711","heartbeat_back_addr":"172.21.15.40:6831/512108711","heartbeat_front_addr":"172.21.15.40:6829/512108711","state":["exists","up"]},{"osd":11,"uuid":"84b7c2de-c424-45a6-9583-60097e372a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6825","nonce":540305718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6827","nonce":540305718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6831","nonce":540305718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6829","nonce":540305718}]},"public_addr":"172.21.15.70:6825/540305718","cluster_addr":"172.21.15.70:6827/540305718","heartbeat_back_addr":"172.21.15.70:6831/540305718","heartbeat_front_addr":"172.21.15.70:6829/540305718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:28:51.477358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:17.405045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:41.626268+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:15.339568+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:42.676996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:07.516999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:42.412273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:10.365041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:35.699120+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:11.733639+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:39.505442+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:34:05.449301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451930554":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/3342588843":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6801/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/1626448389":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/1910059351":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/344822770":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/3811368512":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/1981437651":"2024-03-22T10:26:16.959165+0000","172.21.15.31:6801/1981437651":"2024-03-22T10:26:16.959165+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T10:38:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.731+0000 7f16fe200640 1 -- 172.21.15.31:0/341144963 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f16dc0a2fa0 msgr2=0x7f16dc0a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.731+0000 7f16fe200640 1 --2- 172.21.15.31:0/341144963 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f16dc0a2fa0 0x7f16dc0a5460 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f16f819b100 tx=0x7f16e4005f90 comp rx=0 tx=0).stop 2024-03-21T10:38:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.731+0000 7f16fe200640 1 -- 172.21.15.31:0/341144963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16f8103950 msgr2=0x7f16f8199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.731+0000 7f16fe200640 1 --2- 172.21.15.31:0/341144963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16f8103950 0x7f16f8199bf0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f16e8000c00 tx=0x7f16e802c6c0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.731+0000 7f16fe200640 1 -- 172.21.15.31:0/341144963 shutdown_connections 2024-03-21T10:38:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.731+0000 7f16fe200640 1 --2- 172.21.15.31:0/341144963 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f16dc0a2fa0 0x7f16dc0a5460 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.731+0000 7f16fe200640 1 --2- 172.21.15.31:0/341144963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16f8105500 0x7f16f81a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.731+0000 7f16fe200640 1 --2- 172.21.15.31:0/341144963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16f8104b40 0x7f16f819a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.731+0000 7f16fe200640 1 --2- 172.21.15.31:0/341144963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16f8103950 0x7f16f8199bf0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.731+0000 7f16fe200640 1 -- 172.21.15.31:0/341144963 >> 172.21.15.31:0/341144963 conn(0x7f16f80fef00 msgr2=0x7f16f8107e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.731+0000 7f16fe200640 1 -- 172.21.15.31:0/341144963 shutdown_connections 2024-03-21T10:38:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.731+0000 7f16fe200640 1 -- 172.21.15.31:0/341144963 wait complete. 2024-03-21T10:38:03.754 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-21T10:35:06.492115+0000', 'modified': '2024-03-21T10:35:40.990205+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24596, 'mds_1': 24584, 'mds_2': 24547}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24511': {'gid': 24511, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.70:6835/729664889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6834', 'nonce': 729664889}, {'type': 'v1', 'addr': '172.21.15.70:6835', 'nonce': 729664889}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24523': {'gid': 24523, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.70:6837/3650805915', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6836', 'nonce': 3650805915}, {'type': 'v1', 'addr': '172.21.15.70:6837', 'nonce': 3650805915}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24539': {'gid': 24539, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.40:6837/4121707657', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6836', 'nonce': 4121707657}, {'type': 'v1', 'addr': '172.21.15.40:6837', 'nonce': 4121707657}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24547': {'gid': 24547, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.70:6839/3204327153', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6838', 'nonce': 3204327153}, {'type': 'v1', 'addr': '172.21.15.70:6839', 'nonce': 3204327153}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24584': {'gid': 24584, 'name': 'a', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.31:6841/1752599123', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6840', 'nonce': 1752599123}, {'type': 'v1', 'addr': '172.21.15.31:6841', 'nonce': 1752599123}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24596': {'gid': 24596, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6839/2539201148', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6838', 'nonce': 2539201148}, {'type': 'v1', 'addr': '172.21.15.40:6839', 'nonce': 2539201148}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24596, 'qdb_cluster': [24584, 24596, 24547]} 2024-03-21T10:38:03.754 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-21T10:38:03.754 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-21T10:38:03.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.940+0000 7fcb1e200640 1 -- 172.21.15.31:0/3483637562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb181048a0 msgr2=0x7fcb18106c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:03.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.940+0000 7fcb1e200640 1 --2- 172.21.15.31:0/3483637562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb181048a0 0x7fcb18106c90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fcb04009900 tx=0x7fcb0402bc30 comp rx=0 tx=0).stop 2024-03-21T10:38:03.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.940+0000 7fcb1e200640 1 -- 172.21.15.31:0/3483637562 shutdown_connections 2024-03-21T10:38:03.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.940+0000 7fcb1e200640 1 --2- 172.21.15.31:0/3483637562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb181071d0 0x7fcb18109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.940+0000 7fcb1e200640 1 --2- 172.21.15.31:0/3483637562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb181048a0 0x7fcb18106c90 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.940+0000 7fcb1e200640 1 --2- 172.21.15.31:0/3483637562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb18101f70 0x7fcb18104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.940+0000 7fcb1e200640 1 -- 172.21.15.31:0/3483637562 >> 172.21.15.31:0/3483637562 conn(0x7fcb180fb980 msgr2=0x7fcb180fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:03.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.941+0000 7fcb1e200640 1 -- 172.21.15.31:0/3483637562 shutdown_connections 2024-03-21T10:38:03.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.941+0000 7fcb1e200640 1 -- 172.21.15.31:0/3483637562 wait complete. 2024-03-21T10:38:03.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.942+0000 7fcb1e200640 1 Processor -- start 2024-03-21T10:38:03.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.942+0000 7fcb1e200640 1 -- start start 2024-03-21T10:38:03.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.942+0000 7fcb1e200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb18101f70 0x7fcb18199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:03.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.942+0000 7fcb1e200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb181048a0 0x7fcb18199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:03.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.942+0000 7fcb1e200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb181071d0 0x7fcb181a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:03.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.942+0000 7fcb1e200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1810d1d0 con 0x7fcb181048a0 2024-03-21T10:38:03.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.942+0000 7fcb1e200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1810d050 con 0x7fcb18101f70 2024-03-21T10:38:03.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.942+0000 7fcb1e200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1810d350 con 0x7fcb181071d0 2024-03-21T10:38:03.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.942+0000 7fcb1ce00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb18101f70 0x7fcb18199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:03.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.942+0000 7fcb1ce00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb18101f70 0x7fcb18199a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:57498/0 (socket says 172.21.15.31:57498) 2024-03-21T10:38:03.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.942+0000 7fcb1ce00640 1 -- 172.21.15.31:0/3056532275 learned_addr learned my addr 172.21.15.31:0/3056532275 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:03.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.942+0000 7fcb17e00640 1 --2- 172.21.15.31:0/3056532275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb181048a0 0x7fcb18199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:03.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.943+0000 7fcb1d800640 1 --2- 172.21.15.31:0/3056532275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb181071d0 0x7fcb181a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:03.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.943+0000 7fcb1ce00640 1 -- 172.21.15.31:0/3056532275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb181071d0 msgr2=0x7fcb181a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:03.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.943+0000 7fcb1ce00640 1 --2- 172.21.15.31:0/3056532275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb181071d0 0x7fcb181a0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.943+0000 7fcb1ce00640 1 -- 172.21.15.31:0/3056532275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb181048a0 msgr2=0x7fcb18199f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:03.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.943+0000 7fcb1ce00640 1 --2- 172.21.15.31:0/3056532275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb181048a0 0x7fcb18199f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.943+0000 7fcb1ce00640 1 -- 172.21.15.31:0/3056532275 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb181a1660 con 0x7fcb18101f70 2024-03-21T10:38:03.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.943+0000 7fcb1d800640 1 --2- 172.21.15.31:0/3056532275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb181071d0 0x7fcb181a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:03.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.943+0000 7fcb1ce00640 1 --2- 172.21.15.31:0/3056532275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb18101f70 0x7fcb18199a40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcb0800e9f0 tx=0x7fcb0800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:03.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.943+0000 7fcb15600640 1 -- 172.21.15.31:0/3056532275 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0800cd80 con 0x7fcb18101f70 2024-03-21T10:38:03.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.943+0000 7fcb1e200640 1 -- 172.21.15.31:0/3056532275 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb181a1950 con 0x7fcb18101f70 2024-03-21T10:38:03.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.943+0000 7fcb15600640 1 -- 172.21.15.31:0/3056532275 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcb08010910 con 0x7fcb18101f70 2024-03-21T10:38:03.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.943+0000 7fcb15600640 1 -- 172.21.15.31:0/3056532275 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb080188f0 con 0x7fcb18101f70 2024-03-21T10:38:03.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.944+0000 7fcb1e200640 1 -- 172.21.15.31:0/3056532275 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb181a1e70 con 0x7fcb18101f70 2024-03-21T10:38:03.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.947+0000 7fcb22579740 1 Processor -- start 2024-03-21T10:38:03.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.947+0000 7fcb22579740 1 -- start start 2024-03-21T10:38:03.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.948+0000 7fcb22579740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56371cc36d80 0x56371cc371a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:03.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.948+0000 7fcb22579740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371cc37770 0x56371cc3c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:03.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.948+0000 7fcb22579740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371cc3c8e0 0x56371cc3ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:03.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.948+0000 7fcb22579740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56371cb9bc90 con 0x56371cc37770 2024-03-21T10:38:03.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.948+0000 7fcb22579740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56371cb44980 con 0x56371cc3c8e0 2024-03-21T10:38:03.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.948+0000 7fcb22579740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56371cb97d30 con 0x56371cc36d80 2024-03-21T10:38:03.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.948+0000 7fcb15600640 1 -- 172.21.15.31:0/3056532275 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fcb08010ac0 con 0x7fcb18101f70 2024-03-21T10:38:03.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.948+0000 7fcb1d800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56371cc36d80 0x56371cc371a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:03.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.948+0000 7fcb1d800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56371cc36d80 0x56371cc371a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:40400/0 (socket says 172.21.15.31:40400) 2024-03-21T10:38:03.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.948+0000 7fcb1d800640 1 -- 172.21.15.31:0/1548779833 learned_addr learned my addr 172.21.15.31:0/1548779833 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:03.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.948+0000 7fcb1d800640 1 -- 172.21.15.31:0/1548779833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371cc3c8e0 msgr2=0x56371cc3ed70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:38:03.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.948+0000 7fcb1d800640 1 --2- 172.21.15.31:0/1548779833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371cc3c8e0 0x56371cc3ed70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.948+0000 7fcb1d800640 1 -- 172.21.15.31:0/1548779833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371cc37770 msgr2=0x56371cc3c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:03.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.948+0000 7fcb1ce00640 1 --2- 172.21.15.31:0/1548779833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371cc37770 0x56371cc3c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:03.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.949+0000 7fcb1d800640 1 --2- 172.21.15.31:0/1548779833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371cc37770 0x56371cc3c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.949+0000 7fcb1d800640 1 -- 172.21.15.31:0/1548779833 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56371cb5c510 con 0x56371cc36d80 2024-03-21T10:38:03.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.949+0000 7fcb1ce00640 1 --2- 172.21.15.31:0/1548779833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371cc37770 0x56371cc3c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:38:03.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.949+0000 7fcb1d800640 1 --2- 172.21.15.31:0/1548779833 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56371cc36d80 0x56371cc371a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fcb10010450 tx=0x7fcb10010910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fcaa95e6541b9ac server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:03.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.949+0000 7fcb0e000640 1 -- 172.21.15.31:0/1548779833 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb10017070 con 0x56371cc36d80 2024-03-21T10:38:03.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.949+0000 7fcb0e000640 1 -- 172.21.15.31:0/1548779833 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcb10003b70 con 0x56371cc36d80 2024-03-21T10:38:03.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.949+0000 7fcb0e000640 1 -- 172.21.15.31:0/1548779833 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb10011940 con 0x56371cc36d80 2024-03-21T10:38:03.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.950+0000 7fcb15600640 1 --2- 172.21.15.31:0/3056532275 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcafc0a2f20 0x7fcafc0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:03.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.950+0000 7fcb15600640 1 -- 172.21.15.31:0/3056532275 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fcb080cfc40 con 0x7fcb18101f70 2024-03-21T10:38:03.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.950+0000 7fcb1ce00640 1 --2- 172.21.15.31:0/3056532275 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcafc0a2f20 0x7fcafc0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:03.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.950+0000 7fcb22579740 1 -- 172.21.15.31:0/1548779833 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56371cc36d80 msgr2=0x56371cc371a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:03.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.950+0000 7fcb22579740 1 --2- 172.21.15.31:0/1548779833 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56371cc36d80 0x56371cc371a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fcb10010450 tx=0x7fcb10010910 comp rx=0 tx=0).stop 2024-03-21T10:38:03.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.950+0000 7fcb22579740 1 -- 172.21.15.31:0/1548779833 shutdown_connections 2024-03-21T10:38:03.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.950+0000 7fcb22579740 1 --2- 172.21.15.31:0/1548779833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371cc3c8e0 0x56371cc3ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.950+0000 7fcb22579740 1 --2- 172.21.15.31:0/1548779833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371cc37770 0x56371cc3c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.950+0000 7fcb22579740 1 --2- 172.21.15.31:0/1548779833 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56371cc36d80 0x56371cc371a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.950+0000 7fcb22579740 1 -- 172.21.15.31:0/1548779833 >> 172.21.15.31:0/1548779833 conn(0x56371cc322f0 msgr2=0x56371cc326f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:03.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.950+0000 7fcb22579740 1 -- 172.21.15.31:0/1548779833 shutdown_connections 2024-03-21T10:38:03.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.951+0000 7fcb22579740 1 -- 172.21.15.31:0/1548779833 wait complete. 2024-03-21T10:38:03.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.951+0000 7fcb1ce00640 1 --2- 172.21.15.31:0/3056532275 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcafc0a2f20 0x7fcafc0a53e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcb0809bf40 tx=0x7fcb0801f000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:03.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.951+0000 7fcb22579740 1 -- start start 2024-03-21T10:38:03.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.951+0000 7fcb22579740 1 Processor -- start 2024-03-21T10:38:03.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.952+0000 7fcb22579740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371cc36d80 0x56371cc41270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:03.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.952+0000 7fcb22579740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56371cc37770 0x56371cc417b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:03.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.952+0000 7fcb22579740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371cc3c8e0 0x56371cc487f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:03.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.952+0000 7fcb22579740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56371cc3f890 con 0x56371cc36d80 2024-03-21T10:38:03.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.952+0000 7fcb22579740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56371cb57dd0 con 0x56371cc3c8e0 2024-03-21T10:38:03.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.952+0000 7fcb22579740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56371cc3fa10 con 0x56371cc37770 2024-03-21T10:38:03.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.952+0000 7fcb22579740 10 client.0 _ll_register_callbacks cb 0x56371c8afac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:38:03.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.952+0000 7fcb1d800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371cc36d80 0x56371cc41270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:03.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.952+0000 7fcb1d800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371cc36d80 0x56371cc41270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39900/0 (socket says 172.21.15.31:39900) 2024-03-21T10:38:03.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.952+0000 7fcb1d800640 1 -- 172.21.15.31:0/3282105510 learned_addr learned my addr 172.21.15.31:0/3282105510 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:03.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.952+0000 7fcb17e00640 1 --2- 172.21.15.31:0/3282105510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56371cc37770 0x56371cc417b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:03.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.952+0000 7fcb1d800640 1 --2- 172.21.15.31:0/3282105510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371cc3c8e0 0x56371cc487f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:03.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.953+0000 7fcb17e00640 1 -- 172.21.15.31:0/3282105510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371cc3c8e0 msgr2=0x56371cc487f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:03.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.953+0000 7fcb17e00640 1 --2- 172.21.15.31:0/3282105510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371cc3c8e0 0x56371cc487f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.953+0000 7fcb17e00640 1 -- 172.21.15.31:0/3282105510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371cc36d80 msgr2=0x56371cc41270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:03.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.953+0000 7fcb17e00640 1 --2- 172.21.15.31:0/3282105510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371cc36d80 0x56371cc41270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.953+0000 7fcb17e00640 1 -- 172.21.15.31:0/3282105510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56371c8b02d0 con 0x56371cc37770 2024-03-21T10:38:03.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.953+0000 7fcb17e00640 1 --2- 172.21.15.31:0/3282105510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56371cc37770 0x56371cc417b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fcb04009230 tx=0x7fcb04004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=769fa294fa58fffe server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:03.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.953+0000 7fcb02a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:38:03.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.953+0000 7fcb02a00640 1 -- 172.21.15.31:0/3282105510 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb04005d50 con 0x56371cc37770 2024-03-21T10:38:03.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.953+0000 7fcb02a00640 1 -- 172.21.15.31:0/3282105510 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fcb04005030 con 0x56371cc37770 2024-03-21T10:38:03.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.953+0000 7fcb22579740 1 -- 172.21.15.31:0/3282105510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcb10003b70 con 0x56371cc37770 2024-03-21T10:38:03.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.954+0000 7fcb02a00640 1 -- 172.21.15.31:0/3282105510 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb04005230 con 0x56371cc37770 2024-03-21T10:38:03.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.954+0000 7fcb22579740 10 client.25330 fetch_fsmap learned FSMap version 23 2024-03-21T10:38:03.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.954+0000 7fcb22579740 1 -- 172.21.15.31:0/3282105510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56371c97a6b0 con 0x56371cc37770 2024-03-21T10:38:03.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.954+0000 7fcb02a00640 1 -- 172.21.15.31:0/3282105510 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcb10003b70 con 0x56371cc37770 2024-03-21T10:38:03.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.954+0000 7fcb02a00640 1 -- 172.21.15.31:0/3282105510 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 23) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fcb04004cc0 con 0x56371cc37770 2024-03-21T10:38:03.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.954+0000 7fcb22579740 10 client.25330 fetch_fsmap finished waiting for FSMap version 23 2024-03-21T10:38:03.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.954+0000 7fcb22579740 10 client.25330 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-21T10:38:03.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.954+0000 7fcb22579740 20 client.25330 populate_metadata read hostname 'smithi031' 2024-03-21T10:38:03.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.954+0000 7fcb22579740 1 --2- 172.21.15.31:0/3282105510 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x56371cc4d9d0 0x56371cc4caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:03.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.954+0000 7fcb22579740 4 client.25330 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:38:03.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.954+0000 7fcb22579740 1 -- 172.21.15.31:0/3282105510 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56371cb3d520 con 0x56371cc4d9d0 2024-03-21T10:38:03.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.955+0000 7fcb1d800640 1 --2- 172.21.15.31:0/3282105510 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x56371cc4d9d0 0x56371cc4caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:03.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.955+0000 7fcb1d800640 1 --2- 172.21.15.31:0/3282105510 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x56371cc4d9d0 0x56371cc4caa0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b9ceb3a57bb06a1f server_cookie=8c5b08f1a7e685d2 in_seq=0 out_seq=0 2024-03-21T10:38:03.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.955+0000 7fcb02a00640 10 client.25330 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:38:03.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.956+0000 7fcb02a00640 1 -- 172.21.15.31:0/3282105510 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56371cb3d520 con 0x56371cc4d9d0 2024-03-21T10:38:03.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.956+0000 7fcb02a00640 10 client.25330 handle_command_reply: tid=0 2024-03-21T10:38:03.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.956+0000 7fcb22579740 1 client.25330 shutdown 2024-03-21T10:38:03.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.956+0000 7fcb22579740 1 -- 172.21.15.31:0/3282105510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56371cc37770 msgr2=0x56371cc417b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:03.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.956+0000 7fcb22579740 1 --2- 172.21.15.31:0/3282105510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56371cc37770 0x56371cc417b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fcb04009230 tx=0x7fcb04004270 comp rx=0 tx=0).stop 2024-03-21T10:38:03.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.956+0000 7fcb22579740 1 -- 172.21.15.31:0/3282105510 shutdown_connections 2024-03-21T10:38:03.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.956+0000 7fcb22579740 1 --2- 172.21.15.31:0/3282105510 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x56371cc4d9d0 0x56371cc4caa0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.956+0000 7fcb22579740 1 --2- 172.21.15.31:0/3282105510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371cc3c8e0 0x56371cc487f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.956+0000 7fcb22579740 1 --2- 172.21.15.31:0/3282105510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56371cc37770 0x56371cc417b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.956+0000 7fcb22579740 1 --2- 172.21.15.31:0/3282105510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371cc36d80 0x56371cc41270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.956+0000 7fcb02a00640 0 client.25330 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:38:03.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.956+0000 7fcb22579740 1 -- 172.21.15.31:0/3282105510 >> 172.21.15.31:0/3282105510 conn(0x56371cc322f0 msgr2=0x56371cc314f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:03.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.957+0000 7fcb22579740 1 -- 172.21.15.31:0/3282105510 shutdown_connections 2024-03-21T10:38:03.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.957+0000 7fcb22579740 1 -- 172.21.15.31:0/3282105510 wait complete. 2024-03-21T10:38:03.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.957+0000 7fcb22579740 20 client.25330 trim_cache size 0 max 16384 2024-03-21T10:38:03.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.982+0000 7fcb22579740 1 Processor -- start 2024-03-21T10:38:03.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.982+0000 7fcb22579740 1 -- start start 2024-03-21T10:38:03.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.982+0000 7fcb22579740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371cc36d80 0x56371cc3c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:03.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.982+0000 7fcb22579740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56371cc37770 0x56371cc487b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:03.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.982+0000 7fcb22579740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371cc3c8e0 0x56371cc3ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:03.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.982+0000 7fcb22579740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56371cc4d1f0 con 0x56371cc36d80 2024-03-21T10:38:03.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.982+0000 7fcb22579740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56371cc4d850 con 0x56371cc3c8e0 2024-03-21T10:38:03.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.982+0000 7fcb22579740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56371cc4deb0 con 0x56371cc37770 2024-03-21T10:38:03.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.983+0000 7fcb17e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56371cc37770 0x56371cc487b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:03.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.983+0000 7fcb1d800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371cc36d80 0x56371cc3c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:03.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.983+0000 7fcb17e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56371cc37770 0x56371cc487b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:40424/0 (socket says 172.21.15.31:40424) 2024-03-21T10:38:03.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.983+0000 7fcb17e00640 1 -- 172.21.15.31:0/2827809213 learned_addr learned my addr 172.21.15.31:0/2827809213 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:03.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.983+0000 7fcb1d800640 1 --2- 172.21.15.31:0/2827809213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371cc3c8e0 0x56371cc3ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:03.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.983+0000 7fcb17e00640 1 -- 172.21.15.31:0/2827809213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371cc3c8e0 msgr2=0x56371cc3ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:03.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.983+0000 7fcb17e00640 1 --2- 172.21.15.31:0/2827809213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371cc3c8e0 0x56371cc3ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.983+0000 7fcb17e00640 1 -- 172.21.15.31:0/2827809213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371cc36d80 msgr2=0x56371cc3c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:03.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.983+0000 7fcb17e00640 1 --2- 172.21.15.31:0/2827809213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371cc36d80 0x56371cc3c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.983+0000 7fcb17e00640 1 -- 172.21.15.31:0/2827809213 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56371cc24170 con 0x56371cc37770 2024-03-21T10:38:03.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.983+0000 7fcb17e00640 1 --2- 172.21.15.31:0/2827809213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56371cc37770 0x56371cc487b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcb04003670 tx=0x7fcb04005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b627a9e5736a54e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:03.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.984+0000 7fcb0d600640 1 -- 172.21.15.31:0/2827809213 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb04058070 con 0x56371cc37770 2024-03-21T10:38:03.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.984+0000 7fcb0d600640 1 -- 172.21.15.31:0/2827809213 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcb04003a60 con 0x56371cc37770 2024-03-21T10:38:03.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.984+0000 7fcb0d600640 1 -- 172.21.15.31:0/2827809213 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb04004af0 con 0x56371cc37770 2024-03-21T10:38:03.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.984+0000 7fcb22579740 1 -- 172.21.15.31:0/2827809213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56371cc37770 msgr2=0x56371cc487b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:03.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.984+0000 7fcb22579740 1 --2- 172.21.15.31:0/2827809213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56371cc37770 0x56371cc487b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcb04003670 tx=0x7fcb04005fd0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.984+0000 7fcb22579740 1 -- 172.21.15.31:0/2827809213 shutdown_connections 2024-03-21T10:38:03.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.984+0000 7fcb22579740 1 --2- 172.21.15.31:0/2827809213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371cc3c8e0 0x56371cc3ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.984+0000 7fcb22579740 1 --2- 172.21.15.31:0/2827809213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56371cc37770 0x56371cc487b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.984+0000 7fcb22579740 1 --2- 172.21.15.31:0/2827809213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371cc36d80 0x56371cc3c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.984+0000 7fcb22579740 1 -- 172.21.15.31:0/2827809213 >> 172.21.15.31:0/2827809213 conn(0x56371cc322f0 msgr2=0x56371cc38210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:03.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.985+0000 7fcb22579740 1 -- 172.21.15.31:0/2827809213 shutdown_connections 2024-03-21T10:38:03.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.985+0000 7fcb22579740 1 -- 172.21.15.31:0/2827809213 wait complete. 2024-03-21T10:38:03.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.985+0000 7fcb22579740 1 -- start start 2024-03-21T10:38:03.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.985+0000 7fcb22579740 1 Processor -- start 2024-03-21T10:38:03.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.986+0000 7fcb22579740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56371cc36d80 0x56371cc48150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:03.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.986+0000 7fcb22579740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371cc37770 0x56371cc48690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:03.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.986+0000 7fcb22579740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371cc3c8e0 0x56371cc44d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:03.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.986+0000 7fcb22579740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56371cc4eb70 con 0x56371cc3c8e0 2024-03-21T10:38:03.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.986+0000 7fcb22579740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56371cc4e510 con 0x56371cc37770 2024-03-21T10:38:03.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.986+0000 7fcb22579740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56371cc4f1d0 con 0x56371cc36d80 2024-03-21T10:38:03.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.986+0000 7fcb22579740 10 client.0 _ll_register_callbacks cb 0x7fcb100093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:38:03.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.986+0000 7fcb17e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371cc37770 0x56371cc48690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:03.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.986+0000 7fcb1d800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56371cc36d80 0x56371cc48150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:03.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.986+0000 7fcb17e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371cc37770 0x56371cc48690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:57530/0 (socket says 172.21.15.31:57530) 2024-03-21T10:38:03.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.986+0000 7fcb17e00640 1 -- 172.21.15.31:0/2492946151 learned_addr learned my addr 172.21.15.31:0/2492946151 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:03.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.986+0000 7fcb1d800640 1 --2- 172.21.15.31:0/2492946151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371cc3c8e0 0x56371cc44d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:03.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.986+0000 7fcb17e00640 1 -- 172.21.15.31:0/2492946151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56371cc36d80 msgr2=0x56371cc48150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:03.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.986+0000 7fcb17e00640 1 --2- 172.21.15.31:0/2492946151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56371cc36d80 0x56371cc48150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.986+0000 7fcb17e00640 1 -- 172.21.15.31:0/2492946151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371cc3c8e0 msgr2=0x56371cc44d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:03.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.986+0000 7fcb17e00640 1 --2- 172.21.15.31:0/2492946151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371cc3c8e0 0x56371cc44d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.986+0000 7fcb17e00640 1 -- 172.21.15.31:0/2492946151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56371cc45480 con 0x56371cc37770 2024-03-21T10:38:03.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.986+0000 7fcb1d800640 1 --2- 172.21.15.31:0/2492946151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56371cc36d80 0x56371cc48150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:03.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.987+0000 7fcb17e00640 1 --2- 172.21.15.31:0/2492946151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371cc37770 0x56371cc48690 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcb0402c7b0 tx=0x7fcb04005a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30903cafd8c3e72 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:03.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.987+0000 7fcb02a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:38:03.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.987+0000 7fcb02a00640 1 -- 172.21.15.31:0/2492946151 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb04058070 con 0x56371cc37770 2024-03-21T10:38:03.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.987+0000 7fcb22579740 1 -- 172.21.15.31:0/2492946151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56371cba91d0 con 0x56371cc37770 2024-03-21T10:38:03.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.987+0000 7fcb02a00640 1 -- 172.21.15.31:0/2492946151 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fcb04004a40 con 0x56371cc37770 2024-03-21T10:38:03.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.987+0000 7fcb02a00640 1 -- 172.21.15.31:0/2492946151 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb04062050 con 0x56371cc37770 2024-03-21T10:38:03.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.987+0000 7fcb22579740 10 client.25343 fetch_fsmap learned FSMap version 23 2024-03-21T10:38:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.987+0000 7fcb22579740 1 -- 172.21.15.31:0/2492946151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56371cca9690 con 0x56371cc37770 2024-03-21T10:38:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.988+0000 7fcb02a00640 1 -- 172.21.15.31:0/2492946151 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56371cba91d0 con 0x56371cc37770 2024-03-21T10:38:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.988+0000 7fcb02a00640 1 -- 172.21.15.31:0/2492946151 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 23) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fcb04053980 con 0x56371cc37770 2024-03-21T10:38:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.988+0000 7fcb22579740 10 client.25343 fetch_fsmap finished waiting for FSMap version 23 2024-03-21T10:38:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.988+0000 7fcb22579740 10 client.25343 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-21T10:38:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.988+0000 7fcb22579740 20 client.25343 populate_metadata read hostname 'smithi031' 2024-03-21T10:38:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.988+0000 7fcb22579740 1 --2- 172.21.15.31:0/2492946151 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x56371ccad970 0x56371cca9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.988+0000 7fcb22579740 4 client.25343 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-21T10:38:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.988+0000 7fcb22579740 1 -- 172.21.15.31:0/2492946151 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56371cba3520 con 0x56371ccad970 2024-03-21T10:38:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.988+0000 7fcb1d800640 1 --2- 172.21.15.31:0/2492946151 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x56371ccad970 0x56371cca9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:03.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.989+0000 7fcb1d800640 1 --2- 172.21.15.31:0/2492946151 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x56371ccad970 0x56371cca9920 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d31879728b3c5d5 server_cookie=945fe20ff467ec5d in_seq=0 out_seq=0 2024-03-21T10:38:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.989+0000 7fcb02a00640 10 client.25343 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:38:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.989+0000 7fcb02a00640 1 -- 172.21.15.31:0/2492946151 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x56371cba3520 con 0x56371ccad970 2024-03-21T10:38:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.989+0000 7fcb02a00640 10 client.25343 handle_command_reply: tid=0 2024-03-21T10:38:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.989+0000 7fcb22579740 1 client.25343 shutdown 2024-03-21T10:38:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.989+0000 7fcb22579740 1 -- 172.21.15.31:0/2492946151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371cc37770 msgr2=0x56371cc48690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.989+0000 7fcb22579740 1 --2- 172.21.15.31:0/2492946151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371cc37770 0x56371cc48690 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcb0402c7b0 tx=0x7fcb04005a40 comp rx=0 tx=0).stop 2024-03-21T10:38:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.989+0000 7fcb22579740 1 -- 172.21.15.31:0/2492946151 shutdown_connections 2024-03-21T10:38:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.989+0000 7fcb22579740 1 --2- 172.21.15.31:0/2492946151 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x56371ccad970 0x56371cca9920 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.989+0000 7fcb22579740 1 --2- 172.21.15.31:0/2492946151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371cc3c8e0 0x56371cc44d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.989+0000 7fcb22579740 1 --2- 172.21.15.31:0/2492946151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371cc37770 0x56371cc48690 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.989+0000 7fcb02a00640 0 client.25343 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:38:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.989+0000 7fcb22579740 1 --2- 172.21.15.31:0/2492946151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56371cc36d80 0x56371cc48150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.989+0000 7fcb22579740 1 -- 172.21.15.31:0/2492946151 >> 172.21.15.31:0/2492946151 conn(0x56371cc322f0 msgr2=0x56371cc4caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.989+0000 7fcb22579740 1 -- 172.21.15.31:0/2492946151 shutdown_connections 2024-03-21T10:38:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.990+0000 7fcb22579740 1 -- 172.21.15.31:0/2492946151 wait complete. 2024-03-21T10:38:03.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.990+0000 7fcb22579740 20 client.25343 trim_cache size 0 max 16384 2024-03-21T10:38:03.992 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:38:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "3847408c-e76d-11ee-95cd-87774f69a715", 2024-03-21T10:38:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 2, 2024-03-21T10:38:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24547, 2024-03-21T10:38:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-03-21T10:38:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-03-21T10:38:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-03-21T10:38:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 143.66724520899999, 2024-03-21T10:38:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 23, 2024-03-21T10:38:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 93, 2024-03-21T10:38:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 89, 2024-03-21T10:38:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 194.092157947 2024-03-21T10:38:03.993 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:38:03.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.990+0000 7fcb1e200640 1 -- 172.21.15.31:0/3056532275 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcafc0a2f20 msgr2=0x7fcafc0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:03.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.990+0000 7fcb1e200640 1 --2- 172.21.15.31:0/3056532275 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcafc0a2f20 0x7fcafc0a53e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcb0809bf40 tx=0x7fcb0801f000 comp rx=0 tx=0).stop 2024-03-21T10:38:03.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.990+0000 7fcb1e200640 1 -- 172.21.15.31:0/3056532275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb18101f70 msgr2=0x7fcb18199a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:03.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.990+0000 7fcb1e200640 1 --2- 172.21.15.31:0/3056532275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb18101f70 0x7fcb18199a40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcb0800e9f0 tx=0x7fcb0800eeb0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.991+0000 7fcb1e200640 1 -- 172.21.15.31:0/3056532275 shutdown_connections 2024-03-21T10:38:03.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.991+0000 7fcb1e200640 1 --2- 172.21.15.31:0/3056532275 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcafc0a2f20 0x7fcafc0a53e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.991+0000 7fcb1e200640 1 --2- 172.21.15.31:0/3056532275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb181071d0 0x7fcb181a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.991+0000 7fcb1e200640 1 --2- 172.21.15.31:0/3056532275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb181048a0 0x7fcb18199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.991+0000 7fcb1e200640 1 --2- 172.21.15.31:0/3056532275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb18101f70 0x7fcb18199a40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:03.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.991+0000 7fcb1e200640 1 -- 172.21.15.31:0/3056532275 >> 172.21.15.31:0/3056532275 conn(0x7fcb180fb980 msgr2=0x7fcb18101650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:03.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.991+0000 7fcb1e200640 1 -- 172.21.15.31:0/3056532275 shutdown_connections 2024-03-21T10:38:03.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:03.991+0000 7fcb1e200640 1 -- 172.21.15.31:0/3056532275 wait complete. 2024-03-21T10:38:04.011 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-21T10:38:04.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.222+0000 7f1d2f000640 1 -- 172.21.15.31:0/3820531474 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d281026e0 msgr2=0x7f1d28102b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:04.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.222+0000 7f1d2f000640 1 --2- 172.21.15.31:0/3820531474 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d281026e0 0x7f1d28102b60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1d14009900 tx=0x7f1d1402bc30 comp rx=0 tx=0).stop 2024-03-21T10:38:04.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.222+0000 7f1d2f000640 1 -- 172.21.15.31:0/3820531474 shutdown_connections 2024-03-21T10:38:04.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.222+0000 7f1d2f000640 1 --2- 172.21.15.31:0/3820531474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d281030a0 0x7f1d28109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.222+0000 7f1d2f000640 1 --2- 172.21.15.31:0/3820531474 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d281026e0 0x7f1d28102b60 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.222+0000 7f1d2f000640 1 --2- 172.21.15.31:0/3820531474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d281014f0 0x7f1d28101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.222+0000 7f1d2f000640 1 -- 172.21.15.31:0/3820531474 >> 172.21.15.31:0/3820531474 conn(0x7f1d280fca60 msgr2=0x7f1d280feea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:04.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.222+0000 7f1d2f000640 1 -- 172.21.15.31:0/3820531474 shutdown_connections 2024-03-21T10:38:04.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.222+0000 7f1d2f000640 1 -- 172.21.15.31:0/3820531474 wait complete. 2024-03-21T10:38:04.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.223+0000 7f1d2f000640 1 Processor -- start 2024-03-21T10:38:04.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.223+0000 7f1d2f000640 1 -- start start 2024-03-21T10:38:04.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.224+0000 7f1d2f000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d281014f0 0x7f1d281999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.224+0000 7f1d2f000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d281026e0 0x7f1d28199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.224+0000 7f1d2f000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d281030a0 0x7f1d281a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.224+0000 7f1d2f000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2810d1d0 con 0x7f1d281014f0 2024-03-21T10:38:04.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.224+0000 7f1d2f000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2810d050 con 0x7f1d281030a0 2024-03-21T10:38:04.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.224+0000 7f1d2f000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2810d350 con 0x7f1d281026e0 2024-03-21T10:38:04.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.224+0000 7f1d2dc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d281014f0 0x7f1d281999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.224+0000 7f1d2dc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d281014f0 0x7f1d281999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39942/0 (socket says 172.21.15.31:39942) 2024-03-21T10:38:04.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.224+0000 7f1d2e600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d281030a0 0x7f1d281a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.224+0000 7f1d2dc00640 1 -- 172.21.15.31:0/742200257 learned_addr learned my addr 172.21.15.31:0/742200257 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:04.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.224+0000 7f1d2d200640 1 --2- 172.21.15.31:0/742200257 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d281026e0 0x7f1d28199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.224+0000 7f1d2e600640 1 -- 172.21.15.31:0/742200257 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d281026e0 msgr2=0x7f1d28199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:04.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.225+0000 7f1d2e600640 1 --2- 172.21.15.31:0/742200257 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d281026e0 0x7f1d28199f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.225+0000 7f1d2e600640 1 -- 172.21.15.31:0/742200257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d281014f0 msgr2=0x7f1d281999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:04.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.225+0000 7f1d2e600640 1 --2- 172.21.15.31:0/742200257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d281014f0 0x7f1d281999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.225+0000 7f1d2e600640 1 -- 172.21.15.31:0/742200257 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d281a1610 con 0x7f1d281030a0 2024-03-21T10:38:04.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.225+0000 7f1d2dc00640 1 --2- 172.21.15.31:0/742200257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d281014f0 0x7f1d281999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:04.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.225+0000 7f1d2e600640 1 --2- 172.21.15.31:0/742200257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d281030a0 0x7f1d281a0fa0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1d20009fd0 tx=0x7f1d2000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:04.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.225+0000 7f1d26000640 1 -- 172.21.15.31:0/742200257 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2000ce20 con 0x7f1d281030a0 2024-03-21T10:38:04.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.225+0000 7f1d26000640 1 -- 172.21.15.31:0/742200257 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d20005020 con 0x7f1d281030a0 2024-03-21T10:38:04.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.225+0000 7f1d26000640 1 -- 172.21.15.31:0/742200257 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d20010720 con 0x7f1d281030a0 2024-03-21T10:38:04.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.225+0000 7f1d2f000640 1 -- 172.21.15.31:0/742200257 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d281a1900 con 0x7f1d281030a0 2024-03-21T10:38:04.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.225+0000 7f1d2f000640 1 -- 172.21.15.31:0/742200257 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d281a1e20 con 0x7f1d281030a0 2024-03-21T10:38:04.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.229+0000 7f1d333ec740 1 Processor -- start 2024-03-21T10:38:04.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.230+0000 7f1d26000640 1 -- 172.21.15.31:0/742200257 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f1d20005540 con 0x7f1d281030a0 2024-03-21T10:38:04.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.230+0000 7f1d333ec740 1 -- start start 2024-03-21T10:38:04.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.230+0000 7f1d333ec740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b2cb1fd80 0x556b2cb201a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.230+0000 7f1d333ec740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b2cb20770 0x556b2cb25270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.230+0000 7f1d333ec740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2cb258e0 0x556b2cb27d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.230+0000 7f1d333ec740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b2ca84c90 con 0x556b2cb20770 2024-03-21T10:38:04.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.230+0000 7f1d333ec740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b2ca2d980 con 0x556b2cb1fd80 2024-03-21T10:38:04.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.230+0000 7f1d333ec740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b2ca80d30 con 0x556b2cb258e0 2024-03-21T10:38:04.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.230+0000 7f1d2d200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b2cb1fd80 0x556b2cb201a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.230+0000 7f1d2d200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b2cb1fd80 0x556b2cb201a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:57566/0 (socket says 172.21.15.31:57566) 2024-03-21T10:38:04.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.230+0000 7f1d2d200640 1 -- 172.21.15.31:0/1056558619 learned_addr learned my addr 172.21.15.31:0/1056558619 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:04.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.230+0000 7f1d2dc00640 1 --2- 172.21.15.31:0/1056558619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b2cb20770 0x556b2cb25270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.230+0000 7f1d2d200640 1 --2- 172.21.15.31:0/1056558619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2cb258e0 0x556b2cb27d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.231+0000 7f1d2d200640 1 -- 172.21.15.31:0/1056558619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2cb258e0 msgr2=0x556b2cb27d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:04.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.231+0000 7f1d2d200640 1 --2- 172.21.15.31:0/1056558619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2cb258e0 0x556b2cb27d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.231+0000 7f1d2d200640 1 -- 172.21.15.31:0/1056558619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b2cb20770 msgr2=0x556b2cb25270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:04.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.231+0000 7f1d2d200640 1 --2- 172.21.15.31:0/1056558619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b2cb20770 0x556b2cb25270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.231+0000 7f1d2d200640 1 -- 172.21.15.31:0/1056558619 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b2ca45510 con 0x556b2cb1fd80 2024-03-21T10:38:04.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.231+0000 7f1d2d200640 1 --2- 172.21.15.31:0/1056558619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b2cb1fd80 0x556b2cb201a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1d1402c8e0 tx=0x7f1d14005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24101e71ac3d8999 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:04.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.231+0000 7f1d26000640 1 --2- 172.21.15.31:0/742200257 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1d100a2fa0 0x7f1d100a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.231+0000 7f1d26000640 1 -- 172.21.15.31:0/742200257 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f1d200cfdc0 con 0x7f1d281030a0 2024-03-21T10:38:04.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.231+0000 7f1d1aa00640 1 -- 172.21.15.31:0/1056558619 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d14004030 con 0x556b2cb1fd80 2024-03-21T10:38:04.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.232+0000 7f1d1aa00640 1 -- 172.21.15.31:0/1056558619 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d14004ba0 con 0x556b2cb1fd80 2024-03-21T10:38:04.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.232+0000 7f1d1aa00640 1 -- 172.21.15.31:0/1056558619 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d140038e0 con 0x556b2cb1fd80 2024-03-21T10:38:04.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.232+0000 7f1d2dc00640 1 --2- 172.21.15.31:0/742200257 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1d100a2fa0 0x7f1d100a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.232+0000 7f1d333ec740 1 -- 172.21.15.31:0/1056558619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b2cb1fd80 msgr2=0x556b2cb201a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:04.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.232+0000 7f1d2dc00640 1 --2- 172.21.15.31:0/742200257 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1d100a2fa0 0x7f1d100a5460 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x556b2cb20b70 tx=0x7f1d1c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:04.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.232+0000 7f1d333ec740 1 --2- 172.21.15.31:0/1056558619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b2cb1fd80 0x556b2cb201a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1d1402c8e0 tx=0x7f1d14005fd0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.233+0000 7f1d333ec740 1 -- 172.21.15.31:0/1056558619 shutdown_connections 2024-03-21T10:38:04.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.233+0000 7f1d333ec740 1 --2- 172.21.15.31:0/1056558619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2cb258e0 0x556b2cb27d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.233+0000 7f1d333ec740 1 --2- 172.21.15.31:0/1056558619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b2cb20770 0x556b2cb25270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.233+0000 7f1d333ec740 1 --2- 172.21.15.31:0/1056558619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b2cb1fd80 0x556b2cb201a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.233+0000 7f1d333ec740 1 -- 172.21.15.31:0/1056558619 >> 172.21.15.31:0/1056558619 conn(0x556b2cb1b2f0 msgr2=0x556b2cb1b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:04.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.233+0000 7f1d333ec740 1 -- 172.21.15.31:0/1056558619 shutdown_connections 2024-03-21T10:38:04.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.233+0000 7f1d333ec740 1 -- 172.21.15.31:0/1056558619 wait complete. 2024-03-21T10:38:04.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.233+0000 7f1d333ec740 1 -- start start 2024-03-21T10:38:04.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.234+0000 7f1d333ec740 1 Processor -- start 2024-03-21T10:38:04.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.234+0000 7f1d333ec740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2cb1fd80 0x556b2cb2a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.234+0000 7f1d333ec740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b2cb20770 0x556b2cb2a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.234+0000 7f1d333ec740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b2cb258e0 0x556b2cb31820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.234+0000 7f1d333ec740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b2cb28830 con 0x556b2cb258e0 2024-03-21T10:38:04.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.234+0000 7f1d333ec740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b2ca40dd0 con 0x556b2cb20770 2024-03-21T10:38:04.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.234+0000 7f1d333ec740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b2cb289b0 con 0x556b2cb1fd80 2024-03-21T10:38:04.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.234+0000 7f1d333ec740 10 client.0 _ll_register_callbacks cb 0x556b2c798ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:38:04.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.234+0000 7f1d2e600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b2cb258e0 0x556b2cb31820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.234+0000 7f1d2d200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b2cb20770 0x556b2cb2a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.234+0000 7f1d2e600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b2cb258e0 0x556b2cb31820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39964/0 (socket says 172.21.15.31:39964) 2024-03-21T10:38:04.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.234+0000 7f1d2e600640 1 -- 172.21.15.31:0/2515521831 learned_addr learned my addr 172.21.15.31:0/2515521831 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:04.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.235+0000 7f1d2dc00640 1 --2- 172.21.15.31:0/2515521831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2cb1fd80 0x556b2cb2a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.235+0000 7f1d2d200640 1 -- 172.21.15.31:0/2515521831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2cb1fd80 msgr2=0x556b2cb2a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:04.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.235+0000 7f1d2d200640 1 --2- 172.21.15.31:0/2515521831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2cb1fd80 0x556b2cb2a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.235+0000 7f1d2d200640 1 -- 172.21.15.31:0/2515521831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b2cb258e0 msgr2=0x556b2cb31820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:04.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.235+0000 7f1d2d200640 1 --2- 172.21.15.31:0/2515521831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b2cb258e0 0x556b2cb31820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.235+0000 7f1d2d200640 1 -- 172.21.15.31:0/2515521831 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556b2c7992d0 con 0x556b2cb20770 2024-03-21T10:38:04.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.235+0000 7f1d2dc00640 1 --2- 172.21.15.31:0/2515521831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2cb1fd80 0x556b2cb2a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:04.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.235+0000 7f1d2d200640 1 --2- 172.21.15.31:0/2515521831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b2cb20770 0x556b2cb2a7e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1d14005ab0 tx=0x7f1d14004950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5e22496cc1aa068 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:04.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.235+0000 7f1d0f400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:38:04.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.235+0000 7f1d0f400640 1 -- 172.21.15.31:0/2515521831 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d140052c0 con 0x556b2cb20770 2024-03-21T10:38:04.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.236+0000 7f1d333ec740 1 -- 172.21.15.31:0/2515521831 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1d14004ba0 con 0x556b2cb20770 2024-03-21T10:38:04.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.236+0000 7f1d0f400640 1 -- 172.21.15.31:0/2515521831 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f1d14005460 con 0x556b2cb20770 2024-03-21T10:38:04.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.236+0000 7f1d0f400640 1 -- 172.21.15.31:0/2515521831 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d14005660 con 0x556b2cb20770 2024-03-21T10:38:04.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.236+0000 7f1d333ec740 10 client.25355 fetch_fsmap learned FSMap version 23 2024-03-21T10:38:04.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.236+0000 7f1d333ec740 1 -- 172.21.15.31:0/2515521831 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556b2c8636b0 con 0x556b2cb20770 2024-03-21T10:38:04.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.236+0000 7f1d0f400640 1 -- 172.21.15.31:0/2515521831 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d14004ba0 con 0x556b2cb20770 2024-03-21T10:38:04.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.236+0000 7f1d0f400640 1 -- 172.21.15.31:0/2515521831 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 23) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f1d1404d470 con 0x556b2cb20770 2024-03-21T10:38:04.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.236+0000 7f1d333ec740 10 client.25355 fetch_fsmap finished waiting for FSMap version 23 2024-03-21T10:38:04.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.236+0000 7f1d333ec740 10 client.25355 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-21T10:38:04.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.236+0000 7f1d333ec740 20 client.25355 populate_metadata read hostname 'smithi031' 2024-03-21T10:38:04.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.237+0000 7f1d333ec740 1 --2- 172.21.15.31:0/2515521831 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x556b2cb36490 0x556b2cb3a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.237+0000 7f1d333ec740 4 client.25355 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:38:04.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.237+0000 7f1d333ec740 1 -- 172.21.15.31:0/2515521831 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556b2ca26520 con 0x556b2cb36490 2024-03-21T10:38:04.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.237+0000 7f1d2dc00640 1 --2- 172.21.15.31:0/2515521831 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x556b2cb36490 0x556b2cb3a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.237+0000 7f1d2dc00640 1 --2- 172.21.15.31:0/2515521831 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x556b2cb36490 0x556b2cb3a910 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7b3e2b8df04407b5 server_cookie=fe976aaa935e2db7 in_seq=0 out_seq=0 2024-03-21T10:38:04.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.237+0000 7f1d0f400640 10 client.25355 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:38:04.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.238+0000 7f1d0f400640 1 -- 172.21.15.31:0/2515521831 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556b2ca26520 con 0x556b2cb36490 2024-03-21T10:38:04.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.238+0000 7f1d0f400640 10 client.25355 handle_command_reply: tid=0 2024-03-21T10:38:04.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.238+0000 7f1d333ec740 1 client.25355 shutdown 2024-03-21T10:38:04.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.238+0000 7f1d333ec740 1 -- 172.21.15.31:0/2515521831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b2cb20770 msgr2=0x556b2cb2a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:04.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.239+0000 7f1d333ec740 1 --2- 172.21.15.31:0/2515521831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b2cb20770 0x556b2cb2a7e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1d14005ab0 tx=0x7f1d14004950 comp rx=0 tx=0).stop 2024-03-21T10:38:04.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.239+0000 7f1d333ec740 1 -- 172.21.15.31:0/2515521831 shutdown_connections 2024-03-21T10:38:04.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.239+0000 7f1d333ec740 1 --2- 172.21.15.31:0/2515521831 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x556b2cb36490 0x556b2cb3a910 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.239+0000 7f1d333ec740 1 --2- 172.21.15.31:0/2515521831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b2cb258e0 0x556b2cb31820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.239+0000 7f1d333ec740 1 --2- 172.21.15.31:0/2515521831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b2cb20770 0x556b2cb2a7e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.239+0000 7f1d333ec740 1 --2- 172.21.15.31:0/2515521831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2cb1fd80 0x556b2cb2a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.239+0000 7f1d0f400640 0 client.25355 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:38:04.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.239+0000 7f1d333ec740 1 -- 172.21.15.31:0/2515521831 >> 172.21.15.31:0/2515521831 conn(0x556b2cb1b2f0 msgr2=0x556b2cb1a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:04.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.239+0000 7f1d333ec740 1 -- 172.21.15.31:0/2515521831 shutdown_connections 2024-03-21T10:38:04.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.239+0000 7f1d333ec740 1 -- 172.21.15.31:0/2515521831 wait complete. 2024-03-21T10:38:04.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.239+0000 7f1d333ec740 20 client.25355 trim_cache size 0 max 16384 2024-03-21T10:38:04.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.258+0000 7f1d333ec740 1 Processor -- start 2024-03-21T10:38:04.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.258+0000 7f1d333ec740 1 -- start start 2024-03-21T10:38:04.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.258+0000 7f1d333ec740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b2cb1fd80 0x556b2cb2a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.258+0000 7f1d333ec740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b2cb20770 0x556b2cb317e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.258+0000 7f1d333ec740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2cb258e0 0x556b2cb19c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.258+0000 7f1d333ec740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b2cb35cb0 con 0x556b2cb20770 2024-03-21T10:38:04.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.258+0000 7f1d333ec740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b2cb36310 con 0x556b2cb1fd80 2024-03-21T10:38:04.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.258+0000 7f1d333ec740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b2cb36970 con 0x556b2cb258e0 2024-03-21T10:38:04.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.259+0000 7f1d2e600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2cb258e0 0x556b2cb19c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.259+0000 7f1d2dc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b2cb1fd80 0x556b2cb2a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.259+0000 7f1d2e600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2cb258e0 0x556b2cb19c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:40490/0 (socket says 172.21.15.31:40490) 2024-03-21T10:38:04.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.259+0000 7f1d2e600640 1 -- 172.21.15.31:0/718502851 learned_addr learned my addr 172.21.15.31:0/718502851 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:04.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.259+0000 7f1d2dc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b2cb1fd80 0x556b2cb2a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:57586/0 (socket says 172.21.15.31:57586) 2024-03-21T10:38:04.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.259+0000 7f1d2d200640 1 --2- 172.21.15.31:0/718502851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b2cb20770 0x556b2cb317e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.259+0000 7f1d2dc00640 1 -- 172.21.15.31:0/718502851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2cb258e0 msgr2=0x556b2cb19c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:04.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.259+0000 7f1d2dc00640 1 --2- 172.21.15.31:0/718502851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2cb258e0 0x556b2cb19c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.259+0000 7f1d2dc00640 1 -- 172.21.15.31:0/718502851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b2cb20770 msgr2=0x556b2cb317e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:04.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.259+0000 7f1d2dc00640 1 --2- 172.21.15.31:0/718502851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b2cb20770 0x556b2cb317e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.259+0000 7f1d2dc00640 1 -- 172.21.15.31:0/718502851 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b2cb1c060 con 0x556b2cb1fd80 2024-03-21T10:38:04.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.259+0000 7f1d2d200640 1 --2- 172.21.15.31:0/718502851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b2cb20770 0x556b2cb317e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:04.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.259+0000 7f1d2e600640 1 --2- 172.21.15.31:0/718502851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2cb258e0 0x556b2cb19c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:04.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.260+0000 7f1d2dc00640 1 --2- 172.21.15.31:0/718502851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b2cb1fd80 0x556b2cb2a5c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x556b2cb1c260 tx=0x7f1d1c01f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=280a48fb565235ec server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:04.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.260+0000 7f1d1a000640 1 -- 172.21.15.31:0/718502851 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c033d70 con 0x556b2cb1fd80 2024-03-21T10:38:04.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.260+0000 7f1d1a000640 1 -- 172.21.15.31:0/718502851 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d1c039070 con 0x556b2cb1fd80 2024-03-21T10:38:04.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.260+0000 7f1d1a000640 1 -- 172.21.15.31:0/718502851 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c00e9e0 con 0x556b2cb1fd80 2024-03-21T10:38:04.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.261+0000 7f1d333ec740 1 -- 172.21.15.31:0/718502851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b2cb1fd80 msgr2=0x556b2cb2a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:04.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.261+0000 7f1d333ec740 1 --2- 172.21.15.31:0/718502851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b2cb1fd80 0x556b2cb2a5c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x556b2cb1c260 tx=0x7f1d1c01f000 comp rx=0 tx=0).stop 2024-03-21T10:38:04.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.261+0000 7f1d333ec740 1 -- 172.21.15.31:0/718502851 shutdown_connections 2024-03-21T10:38:04.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.261+0000 7f1d333ec740 1 --2- 172.21.15.31:0/718502851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2cb258e0 0x556b2cb19c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.261+0000 7f1d333ec740 1 --2- 172.21.15.31:0/718502851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b2cb20770 0x556b2cb317e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.261+0000 7f1d333ec740 1 --2- 172.21.15.31:0/718502851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b2cb1fd80 0x556b2cb2a5c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.261+0000 7f1d333ec740 1 -- 172.21.15.31:0/718502851 >> 172.21.15.31:0/718502851 conn(0x556b2cb1b2f0 msgr2=0x556b2cb29eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:04.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.261+0000 7f1d333ec740 1 -- 172.21.15.31:0/718502851 shutdown_connections 2024-03-21T10:38:04.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.261+0000 7f1d333ec740 1 -- 172.21.15.31:0/718502851 wait complete. 2024-03-21T10:38:04.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.261+0000 7f1d333ec740 1 -- start start 2024-03-21T10:38:04.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.262+0000 7f1d333ec740 1 Processor -- start 2024-03-21T10:38:04.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.262+0000 7f1d333ec740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2cb1fd80 0x556b2cb31000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.262+0000 7f1d333ec740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b2cb20770 0x556b2cb31540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.262+0000 7f1d333ec740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b2cb258e0 0x556b2cb2dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.262+0000 7f1d333ec740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b2cb37630 con 0x556b2cb20770 2024-03-21T10:38:04.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.262+0000 7f1d333ec740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b2cb36fd0 con 0x556b2cb258e0 2024-03-21T10:38:04.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.262+0000 7f1d333ec740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b2cb37c90 con 0x556b2cb1fd80 2024-03-21T10:38:04.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.262+0000 7f1d333ec740 10 client.0 _ll_register_callbacks cb 0x7f1d1c009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:38:04.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.263+0000 7f1d2dc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2cb1fd80 0x556b2cb31000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.263+0000 7f1d2e600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b2cb258e0 0x556b2cb2dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.263+0000 7f1d2d200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b2cb20770 0x556b2cb31540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.263+0000 7f1d2dc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2cb1fd80 0x556b2cb31000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:40492/0 (socket says 172.21.15.31:40492) 2024-03-21T10:38:04.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.263+0000 7f1d2dc00640 1 -- 172.21.15.31:0/3542124906 learned_addr learned my addr 172.21.15.31:0/3542124906 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:04.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.263+0000 7f1d2dc00640 1 -- 172.21.15.31:0/3542124906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b2cb258e0 msgr2=0x556b2cb2dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:04.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.263+0000 7f1d2dc00640 1 --2- 172.21.15.31:0/3542124906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b2cb258e0 0x556b2cb2dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.263+0000 7f1d2dc00640 1 -- 172.21.15.31:0/3542124906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b2cb20770 msgr2=0x556b2cb31540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:04.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.263+0000 7f1d2dc00640 1 --2- 172.21.15.31:0/3542124906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b2cb20770 0x556b2cb31540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.263+0000 7f1d2dc00640 1 -- 172.21.15.31:0/3542124906 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556b2cb2e170 con 0x556b2cb1fd80 2024-03-21T10:38:04.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.263+0000 7f1d2e600640 1 --2- 172.21.15.31:0/3542124906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b2cb258e0 0x556b2cb2dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:04.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.263+0000 7f1d2dc00640 1 --2- 172.21.15.31:0/3542124906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2cb1fd80 0x556b2cb31000 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c036040 tx=0x7f1d1c009fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8dc01bbc47f2347 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:04.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.263+0000 7f1d0f400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:38:04.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.264+0000 7f1d0f400640 1 -- 172.21.15.31:0/3542124906 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c043070 con 0x556b2cb1fd80 2024-03-21T10:38:04.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.264+0000 7f1d0f400640 1 -- 172.21.15.31:0/3542124906 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f1d1c00e960 con 0x556b2cb1fd80 2024-03-21T10:38:04.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.264+0000 7f1d333ec740 1 -- 172.21.15.31:0/3542124906 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1d1c039070 con 0x556b2cb1fd80 2024-03-21T10:38:04.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.264+0000 7f1d0f400640 1 -- 172.21.15.31:0/3542124906 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d1c04d050 con 0x556b2cb1fd80 2024-03-21T10:38:04.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.264+0000 7f1d333ec740 10 client.25354 fetch_fsmap learned FSMap version 23 2024-03-21T10:38:04.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.264+0000 7f1d333ec740 1 -- 172.21.15.31:0/3542124906 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556b2cb925f0 con 0x556b2cb1fd80 2024-03-21T10:38:04.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.264+0000 7f1d0f400640 1 -- 172.21.15.31:0/3542124906 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d1c039070 con 0x556b2cb1fd80 2024-03-21T10:38:04.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.265+0000 7f1d0f400640 1 -- 172.21.15.31:0/3542124906 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 23) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f1d1c048440 con 0x556b2cb1fd80 2024-03-21T10:38:04.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.265+0000 7f1d333ec740 10 client.25354 fetch_fsmap finished waiting for FSMap version 23 2024-03-21T10:38:04.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.265+0000 7f1d333ec740 10 client.25354 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-21T10:38:04.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.265+0000 7f1d333ec740 20 client.25354 populate_metadata read hostname 'smithi031' 2024-03-21T10:38:04.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.265+0000 7f1d333ec740 1 --2- 172.21.15.31:0/3542124906 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x556b2cb968f0 0x556b2cb928d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.265+0000 7f1d333ec740 4 client.25354 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-21T10:38:04.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.265+0000 7f1d333ec740 1 -- 172.21.15.31:0/3542124906 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556b2ca921d0 con 0x556b2cb968f0 2024-03-21T10:38:04.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.266+0000 7f1d2d200640 1 --2- 172.21.15.31:0/3542124906 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x556b2cb968f0 0x556b2cb928d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.266+0000 7f1d2d200640 1 --2- 172.21.15.31:0/3542124906 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x556b2cb968f0 0x556b2cb928d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=634dcc92042d864f server_cookie=7e9269e4828453a0 in_seq=0 out_seq=0 2024-03-21T10:38:04.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.266+0000 7f1d0f400640 10 client.25354 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:38:04.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.266+0000 7f1d0f400640 1 -- 172.21.15.31:0/3542124906 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x556b2ca921d0 con 0x556b2cb968f0 2024-03-21T10:38:04.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.266+0000 7f1d0f400640 10 client.25354 handle_command_reply: tid=0 2024-03-21T10:38:04.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.266+0000 7f1d333ec740 1 client.25354 shutdown 2024-03-21T10:38:04.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.267+0000 7f1d333ec740 1 -- 172.21.15.31:0/3542124906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2cb1fd80 msgr2=0x556b2cb31000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:04.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.267+0000 7f1d333ec740 1 --2- 172.21.15.31:0/3542124906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2cb1fd80 0x556b2cb31000 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c036040 tx=0x7f1d1c009fa0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.267+0000 7f1d333ec740 1 -- 172.21.15.31:0/3542124906 shutdown_connections 2024-03-21T10:38:04.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.267+0000 7f1d333ec740 1 --2- 172.21.15.31:0/3542124906 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x556b2cb968f0 0x556b2cb928d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.267+0000 7f1d333ec740 1 --2- 172.21.15.31:0/3542124906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b2cb258e0 0x556b2cb2dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.267+0000 7f1d333ec740 1 --2- 172.21.15.31:0/3542124906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b2cb20770 0x556b2cb31540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.267+0000 7f1d333ec740 1 --2- 172.21.15.31:0/3542124906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2cb1fd80 0x556b2cb31000 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.267+0000 7f1d0f400640 0 client.25354 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:38:04.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.267+0000 7f1d333ec740 1 -- 172.21.15.31:0/3542124906 >> 172.21.15.31:0/3542124906 conn(0x556b2cb1b2f0 msgr2=0x556b2cb1e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:04.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.267+0000 7f1d333ec740 1 -- 172.21.15.31:0/3542124906 shutdown_connections 2024-03-21T10:38:04.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.267+0000 7f1d333ec740 1 -- 172.21.15.31:0/3542124906 wait complete. 2024-03-21T10:38:04.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.267+0000 7f1d333ec740 20 client.25354 trim_cache size 0 max 16384 2024-03-21T10:38:04.269 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:38:04.269 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "3847408c-e76d-11ee-95cd-87774f69a715", 2024-03-21T10:38:04.269 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 1, 2024-03-21T10:38:04.269 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24584, 2024-03-21T10:38:04.269 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-03-21T10:38:04.269 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-03-21T10:38:04.269 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-03-21T10:38:04.269 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 144.955369023, 2024-03-21T10:38:04.269 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 23, 2024-03-21T10:38:04.269 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 93, 2024-03-21T10:38:04.269 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 89, 2024-03-21T10:38:04.270 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 186.02306560700001 2024-03-21T10:38:04.270 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:38:04.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.268+0000 7f1d2f000640 1 -- 172.21.15.31:0/742200257 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1d100a2fa0 msgr2=0x7f1d100a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:04.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.268+0000 7f1d2f000640 1 --2- 172.21.15.31:0/742200257 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1d100a2fa0 0x7f1d100a5460 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x556b2cb20b70 tx=0x7f1d1c005f90 comp rx=0 tx=0).stop 2024-03-21T10:38:04.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.268+0000 7f1d2f000640 1 -- 172.21.15.31:0/742200257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d281030a0 msgr2=0x7f1d281a0fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:04.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.268+0000 7f1d2f000640 1 --2- 172.21.15.31:0/742200257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d281030a0 0x7f1d281a0fa0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1d20009fd0 tx=0x7f1d2000ef10 comp rx=0 tx=0).stop 2024-03-21T10:38:04.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.268+0000 7f1d2f000640 1 -- 172.21.15.31:0/742200257 shutdown_connections 2024-03-21T10:38:04.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.268+0000 7f1d2f000640 1 --2- 172.21.15.31:0/742200257 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1d100a2fa0 0x7f1d100a5460 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.268+0000 7f1d2f000640 1 --2- 172.21.15.31:0/742200257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d281030a0 0x7f1d281a0fa0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.268+0000 7f1d2f000640 1 --2- 172.21.15.31:0/742200257 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d281026e0 0x7f1d28199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.268+0000 7f1d2f000640 1 --2- 172.21.15.31:0/742200257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d281014f0 0x7f1d281999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.268+0000 7f1d2f000640 1 -- 172.21.15.31:0/742200257 >> 172.21.15.31:0/742200257 conn(0x7f1d280fca60 msgr2=0x7f1d280738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:04.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.268+0000 7f1d2f000640 1 -- 172.21.15.31:0/742200257 shutdown_connections 2024-03-21T10:38:04.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.268+0000 7f1d2f000640 1 -- 172.21.15.31:0/742200257 wait complete. 2024-03-21T10:38:04.282 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:04 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3603250068' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:38:04.282 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:04 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/341144963' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:38:04.284 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-21T10:38:04.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:04 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3603250068' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:38:04.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:04 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/341144963' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:38:04.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.476+0000 7f74d3400640 1 -- 172.21.15.31:0/2466148870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74cc103280 msgr2=0x7f74cc109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:04.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.476+0000 7f74d3400640 1 --2- 172.21.15.31:0/2466148870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74cc103280 0x7f74cc109b10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f74c4009be0 tx=0x7f74c402bef0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.477+0000 7f74d3400640 1 -- 172.21.15.31:0/2466148870 shutdown_connections 2024-03-21T10:38:04.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.477+0000 7f74d3400640 1 --2- 172.21.15.31:0/2466148870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74cc103280 0x7f74cc109b10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.477+0000 7f74d3400640 1 --2- 172.21.15.31:0/2466148870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74cc1028c0 0x7f74cc102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.477+0000 7f74d3400640 1 --2- 172.21.15.31:0/2466148870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74cc1016d0 0x7f74cc101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.477+0000 7f74d3400640 1 -- 172.21.15.31:0/2466148870 >> 172.21.15.31:0/2466148870 conn(0x7f74cc0fcc60 msgr2=0x7f74cc0ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:04.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.477+0000 7f74d3400640 1 -- 172.21.15.31:0/2466148870 shutdown_connections 2024-03-21T10:38:04.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.477+0000 7f74d3400640 1 -- 172.21.15.31:0/2466148870 wait complete. 2024-03-21T10:38:04.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.478+0000 7f74d3400640 1 Processor -- start 2024-03-21T10:38:04.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.478+0000 7f74d3400640 1 -- start start 2024-03-21T10:38:04.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.478+0000 7f74d3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74cc1016d0 0x7f74cc199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.478+0000 7f74d3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74cc1028c0 0x7f74cc19a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.478+0000 7f74d3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74cc103280 0x7f74cc1a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.478+0000 7f74d3400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc10d380 con 0x7f74cc103280 2024-03-21T10:38:04.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.478+0000 7f74d3400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc10d200 con 0x7f74cc1016d0 2024-03-21T10:38:04.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.478+0000 7f74d3400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc10d500 con 0x7f74cc1028c0 2024-03-21T10:38:04.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.478+0000 7f74cbe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74cc1028c0 0x7f74cc19a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.478+0000 7f74d0a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74cc1016d0 0x7f74cc199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.478+0000 7f74cbe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74cc1028c0 0x7f74cc19a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:40514/0 (socket says 172.21.15.31:40514) 2024-03-21T10:38:04.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.478+0000 7f74cbe00640 1 -- 172.21.15.31:0/688138427 learned_addr learned my addr 172.21.15.31:0/688138427 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:04.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.478+0000 7f74d1400640 1 --2- 172.21.15.31:0/688138427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74cc103280 0x7f74cc1a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.479+0000 7f74cbe00640 1 -- 172.21.15.31:0/688138427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74cc1016d0 msgr2=0x7f74cc199b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:04.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.479+0000 7f74cbe00640 1 --2- 172.21.15.31:0/688138427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74cc1016d0 0x7f74cc199b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.479+0000 7f74cbe00640 1 -- 172.21.15.31:0/688138427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74cc103280 msgr2=0x7f74cc1a1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:04.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.479+0000 7f74cbe00640 1 --2- 172.21.15.31:0/688138427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74cc103280 0x7f74cc1a1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.479+0000 7f74cbe00640 1 -- 172.21.15.31:0/688138427 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74cc1a1780 con 0x7f74cc1028c0 2024-03-21T10:38:04.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.479+0000 7f74cbe00640 1 --2- 172.21.15.31:0/688138427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74cc1028c0 0x7f74cc19a0a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f74b800e9c0 tx=0x7f74b800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:04.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.479+0000 7f74c9600640 1 -- 172.21.15.31:0/688138427 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b800ccc0 con 0x7f74cc1028c0 2024-03-21T10:38:04.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.479+0000 7f74c9600640 1 -- 172.21.15.31:0/688138427 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f74b8005500 con 0x7f74cc1028c0 2024-03-21T10:38:04.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.480+0000 7f74c9600640 1 -- 172.21.15.31:0/688138427 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b8010430 con 0x7f74cc1028c0 2024-03-21T10:38:04.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.480+0000 7f74d3400640 1 -- 172.21.15.31:0/688138427 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74cc1a1a70 con 0x7f74cc1028c0 2024-03-21T10:38:04.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.480+0000 7f74d3400640 1 -- 172.21.15.31:0/688138427 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74cc1a1f70 con 0x7f74cc1028c0 2024-03-21T10:38:04.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.483+0000 7f74c9600640 1 -- 172.21.15.31:0/688138427 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f74b80046c0 con 0x7f74cc1028c0 2024-03-21T10:38:04.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.483+0000 7f74d5dfd740 1 Processor -- start 2024-03-21T10:38:04.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.483+0000 7f74d5dfd740 1 -- start start 2024-03-21T10:38:04.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.484+0000 7f74d5dfd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7b39c4ba0 0x55a7b39c4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.484+0000 7f74d5dfd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7b39c5590 0x55a7b39ca090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.484+0000 7f74d5dfd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7b39ca700 0x55a7b39ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.484+0000 7f74d5dfd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7b3929c90 con 0x55a7b39c4ba0 2024-03-21T10:38:04.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.484+0000 7f74d5dfd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7b38d2980 con 0x55a7b39c5590 2024-03-21T10:38:04.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.484+0000 7f74d5dfd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7b3925d30 con 0x55a7b39ca700 2024-03-21T10:38:04.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.484+0000 7f74d0a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7b39c5590 0x55a7b39ca090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.484+0000 7f74d0a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7b39c5590 0x55a7b39ca090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:57632/0 (socket says 172.21.15.31:57632) 2024-03-21T10:38:04.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.484+0000 7f74d0a00640 1 -- 172.21.15.31:0/2920260414 learned_addr learned my addr 172.21.15.31:0/2920260414 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:04.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.484+0000 7f74cbe00640 1 --2- 172.21.15.31:0/2920260414 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7b39ca700 0x55a7b39ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.484+0000 7f74d1400640 1 --2- 172.21.15.31:0/2920260414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7b39c4ba0 0x55a7b39c4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.485+0000 7f74d0a00640 1 -- 172.21.15.31:0/2920260414 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7b39ca700 msgr2=0x55a7b39ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:04.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.485+0000 7f74d0a00640 1 --2- 172.21.15.31:0/2920260414 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7b39ca700 0x55a7b39ccb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.485+0000 7f74d0a00640 1 -- 172.21.15.31:0/2920260414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7b39c4ba0 msgr2=0x55a7b39c4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:04.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.485+0000 7f74d0a00640 1 --2- 172.21.15.31:0/2920260414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7b39c4ba0 0x55a7b39c4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.485+0000 7f74d0a00640 1 -- 172.21.15.31:0/2920260414 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a7b38ea510 con 0x55a7b39c5590 2024-03-21T10:38:04.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.485+0000 7f74d1400640 1 --2- 172.21.15.31:0/2920260414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7b39c4ba0 0x55a7b39c4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:04.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.485+0000 7f74d0a00640 1 --2- 172.21.15.31:0/2920260414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7b39c5590 0x55a7b39ca090 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f74c0009ca0 tx=0x7f74c0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=608bad52e212fc9f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:04.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.485+0000 7f74c9600640 1 --2- 172.21.15.31:0/688138427 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f74b40a2fa0 0x7f74b40a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.485+0000 7f74bdc00640 1 -- 172.21.15.31:0/2920260414 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c0004220 con 0x55a7b39c5590 2024-03-21T10:38:04.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.485+0000 7f74bdc00640 1 -- 172.21.15.31:0/2920260414 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f74c00043c0 con 0x55a7b39c5590 2024-03-21T10:38:04.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.485+0000 7f74bdc00640 1 -- 172.21.15.31:0/2920260414 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c0019930 con 0x55a7b39c5590 2024-03-21T10:38:04.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.485+0000 7f74c9600640 1 -- 172.21.15.31:0/688138427 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f74b80cf9f0 con 0x7f74cc1028c0 2024-03-21T10:38:04.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.486+0000 7f74d1400640 1 --2- 172.21.15.31:0/688138427 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f74b40a2fa0 0x7f74b40a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.486+0000 7f74d5dfd740 1 -- 172.21.15.31:0/2920260414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7b39c5590 msgr2=0x55a7b39ca090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:04.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.486+0000 7f74d5dfd740 1 --2- 172.21.15.31:0/2920260414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7b39c5590 0x55a7b39ca090 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f74c0009ca0 tx=0x7f74c0011810 comp rx=0 tx=0).stop 2024-03-21T10:38:04.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.487+0000 7f74d5dfd740 1 -- 172.21.15.31:0/2920260414 shutdown_connections 2024-03-21T10:38:04.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.487+0000 7f74d5dfd740 1 --2- 172.21.15.31:0/2920260414 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7b39ca700 0x55a7b39ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.487+0000 7f74d5dfd740 1 --2- 172.21.15.31:0/2920260414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7b39c5590 0x55a7b39ca090 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.487+0000 7f74d5dfd740 1 --2- 172.21.15.31:0/2920260414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7b39c4ba0 0x55a7b39c4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.487+0000 7f74d5dfd740 1 -- 172.21.15.31:0/2920260414 >> 172.21.15.31:0/2920260414 conn(0x55a7b39c0110 msgr2=0x55a7b39c0510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:04.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.487+0000 7f74d1400640 1 --2- 172.21.15.31:0/688138427 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f74b40a2fa0 0x7f74b40a5460 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f74c4006010 tx=0x7f74c4004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:04.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.487+0000 7f74d5dfd740 1 -- 172.21.15.31:0/2920260414 shutdown_connections 2024-03-21T10:38:04.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.487+0000 7f74d5dfd740 1 -- 172.21.15.31:0/2920260414 wait complete. 2024-03-21T10:38:04.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.487+0000 7f74d5dfd740 1 -- start start 2024-03-21T10:38:04.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.488+0000 7f74d5dfd740 1 Processor -- start 2024-03-21T10:38:04.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.488+0000 7f74d5dfd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7b39c4ba0 0x55a7b39cf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.488+0000 7f74d5dfd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7b39c5590 0x55a7b39cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.488+0000 7f74d5dfd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7b39ca700 0x55a7b39d6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.488+0000 7f74d5dfd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7b39cd6b0 con 0x55a7b39c4ba0 2024-03-21T10:38:04.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.488+0000 7f74d5dfd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7b38e5dd0 con 0x55a7b39ca700 2024-03-21T10:38:04.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.488+0000 7f74d5dfd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7b39cd830 con 0x55a7b39c5590 2024-03-21T10:38:04.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.488+0000 7f74d5dfd740 10 client.0 _ll_register_callbacks cb 0x55a7b363dac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:38:04.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.489+0000 7f74cbe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7b39c5590 0x55a7b39cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.489+0000 7f74cbe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7b39c5590 0x55a7b39cf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:40538/0 (socket says 172.21.15.31:40538) 2024-03-21T10:38:04.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.489+0000 7f74cbe00640 1 -- 172.21.15.31:0/4248891488 learned_addr learned my addr 172.21.15.31:0/4248891488 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:04.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.489+0000 7f74d0a00640 1 --2- 172.21.15.31:0/4248891488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7b39c4ba0 0x55a7b39cf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.489+0000 7f74d1400640 1 --2- 172.21.15.31:0/4248891488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7b39ca700 0x55a7b39d6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.489+0000 7f74cbe00640 1 -- 172.21.15.31:0/4248891488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7b39ca700 msgr2=0x55a7b39d6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:04.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.489+0000 7f74cbe00640 1 --2- 172.21.15.31:0/4248891488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7b39ca700 0x55a7b39d6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.489+0000 7f74cbe00640 1 -- 172.21.15.31:0/4248891488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7b39c4ba0 msgr2=0x55a7b39cf0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:04.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.489+0000 7f74cbe00640 1 --2- 172.21.15.31:0/4248891488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7b39c4ba0 0x55a7b39cf0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.489+0000 7f74cbe00640 1 -- 172.21.15.31:0/4248891488 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a7b363e2d0 con 0x55a7b39c5590 2024-03-21T10:38:04.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.489+0000 7f74cbe00640 1 --2- 172.21.15.31:0/4248891488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7b39c5590 0x55a7b39cf5e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f74b8098730 tx=0x7f74b8009400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e527dc83a36b8af server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:04.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.490+0000 7f74b2000640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:38:04.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.490+0000 7f74b2000640 1 -- 172.21.15.31:0/4248891488 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b809b8a0 con 0x55a7b39c5590 2024-03-21T10:38:04.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.490+0000 7f74b2000640 1 -- 172.21.15.31:0/4248891488 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f74b80a44e0 con 0x55a7b39c5590 2024-03-21T10:38:04.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.490+0000 7f74d5dfd740 1 -- 172.21.15.31:0/4248891488 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f74c00043c0 con 0x55a7b39c5590 2024-03-21T10:38:04.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.490+0000 7f74b2000640 1 -- 172.21.15.31:0/4248891488 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74b809b6c0 con 0x55a7b39c5590 2024-03-21T10:38:04.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.490+0000 7f74d5dfd740 10 client.25366 fetch_fsmap learned FSMap version 23 2024-03-21T10:38:04.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.490+0000 7f74d5dfd740 1 -- 172.21.15.31:0/4248891488 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a7b37086b0 con 0x55a7b39c5590 2024-03-21T10:38:04.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.490+0000 7f74b2000640 1 -- 172.21.15.31:0/4248891488 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f74c00043c0 con 0x55a7b39c5590 2024-03-21T10:38:04.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.490+0000 7f74b2000640 1 -- 172.21.15.31:0/4248891488 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 23) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f74b8086040 con 0x55a7b39c5590 2024-03-21T10:38:04.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.491+0000 7f74d5dfd740 10 client.25366 fetch_fsmap finished waiting for FSMap version 23 2024-03-21T10:38:04.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.491+0000 7f74d5dfd740 10 client.25366 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-21T10:38:04.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.491+0000 7f74d5dfd740 20 client.25366 populate_metadata read hostname 'smithi031' 2024-03-21T10:38:04.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.491+0000 7f74d5dfd740 1 --2- 172.21.15.31:0/4248891488 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a7b39db940 0x55a7b39da9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.491+0000 7f74d5dfd740 4 client.25366 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:38:04.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.491+0000 7f74d5dfd740 1 -- 172.21.15.31:0/4248891488 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a7b38cb520 con 0x55a7b39db940 2024-03-21T10:38:04.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.491+0000 7f74d0a00640 1 --2- 172.21.15.31:0/4248891488 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a7b39db940 0x55a7b39da9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.492+0000 7f74d0a00640 1 --2- 172.21.15.31:0/4248891488 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a7b39db940 0x55a7b39da9c0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fdb85e0eaf82e8d server_cookie=210c847af569a01b in_seq=0 out_seq=0 2024-03-21T10:38:04.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.492+0000 7f74b2000640 10 client.25366 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:04.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.493+0000 7f74b2000640 1 -- 172.21.15.31:0/4248891488 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a7b38cb520 con 0x55a7b39db940 2024-03-21T10:38:04.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.493+0000 7f74b2000640 10 client.25366 handle_command_reply: tid=0 2024-03-21T10:38:04.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.493+0000 7f74d5dfd740 1 client.25366 shutdown 2024-03-21T10:38:04.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.493+0000 7f74d5dfd740 1 -- 172.21.15.31:0/4248891488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7b39c5590 msgr2=0x55a7b39cf5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:04.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.493+0000 7f74d5dfd740 1 --2- 172.21.15.31:0/4248891488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7b39c5590 0x55a7b39cf5e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f74b8098730 tx=0x7f74b8009400 comp rx=0 tx=0).stop 2024-03-21T10:38:04.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.493+0000 7f74d5dfd740 1 -- 172.21.15.31:0/4248891488 shutdown_connections 2024-03-21T10:38:04.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.493+0000 7f74d5dfd740 1 --2- 172.21.15.31:0/4248891488 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a7b39db940 0x55a7b39da9c0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.493+0000 7f74d5dfd740 1 --2- 172.21.15.31:0/4248891488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7b39ca700 0x55a7b39d6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.493+0000 7f74d5dfd740 1 --2- 172.21.15.31:0/4248891488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7b39c5590 0x55a7b39cf5e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.493+0000 7f74d5dfd740 1 --2- 172.21.15.31:0/4248891488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7b39c4ba0 0x55a7b39cf0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.493+0000 7f74b2000640 0 client.25366 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:04.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.493+0000 7f74d5dfd740 1 -- 172.21.15.31:0/4248891488 >> 172.21.15.31:0/4248891488 conn(0x55a7b39c0110 msgr2=0x55a7b39bf310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:04.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.493+0000 7f74d5dfd740 1 -- 172.21.15.31:0/4248891488 shutdown_connections 2024-03-21T10:38:04.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.493+0000 7f74d5dfd740 1 -- 172.21.15.31:0/4248891488 wait complete. 2024-03-21T10:38:04.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.493+0000 7f74d5dfd740 20 client.25366 trim_cache size 0 max 16384 2024-03-21T10:38:04.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.512+0000 7f74d5dfd740 1 Processor -- start 2024-03-21T10:38:04.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.513+0000 7f74d5dfd740 1 -- start start 2024-03-21T10:38:04.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.513+0000 7f74d5dfd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7b39c4ba0 0x55a7b39bae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.513+0000 7f74d5dfd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7b39c5590 0x55a7b39d65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.513+0000 7f74d5dfd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7b39ca700 0x55a7b39c04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.513+0000 7f74d5dfd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7b39db160 con 0x55a7b39c5590 2024-03-21T10:38:04.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.513+0000 7f74d5dfd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7b39db7c0 con 0x55a7b39c4ba0 2024-03-21T10:38:04.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.513+0000 7f74d5dfd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7b39dbe20 con 0x55a7b39ca700 2024-03-21T10:38:04.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.513+0000 7f74d0a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7b39c4ba0 0x55a7b39bae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.513+0000 7f74cbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7b39c5590 0x55a7b39d65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.513+0000 7f74cbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7b39c5590 0x55a7b39d65e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40026/0 (socket says 172.21.15.31:40026) 2024-03-21T10:38:04.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.513+0000 7f74cbe00640 1 -- 172.21.15.31:0/1772579153 learned_addr learned my addr 172.21.15.31:0/1772579153 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:04.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.513+0000 7f74d1400640 1 --2- 172.21.15.31:0/1772579153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7b39ca700 0x55a7b39c04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.513+0000 7f74d0a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7b39c4ba0 0x55a7b39bae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:57660/0 (socket says 172.21.15.31:57660) 2024-03-21T10:38:04.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.514+0000 7f74d1400640 1 -- 172.21.15.31:0/1772579153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7b39c4ba0 msgr2=0x55a7b39bae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.514+0000 7f74d1400640 1 --2- 172.21.15.31:0/1772579153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7b39c4ba0 0x55a7b39bae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.514+0000 7f74d1400640 1 -- 172.21.15.31:0/1772579153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7b39c5590 msgr2=0x55a7b39d65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.514+0000 7f74d1400640 1 --2- 172.21.15.31:0/1772579153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7b39c5590 0x55a7b39d65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.514+0000 7f74d1400640 1 -- 172.21.15.31:0/1772579153 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a7b39c73d0 con 0x55a7b39ca700 2024-03-21T10:38:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.514+0000 7f74d0a00640 1 --2- 172.21.15.31:0/1772579153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7b39c4ba0 0x55a7b39bae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.514+0000 7f74d1400640 1 --2- 172.21.15.31:0/1772579153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7b39ca700 0x55a7b39c04f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f74c402c430 tx=0x7f74c4041000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cafb98b3ebe1a0bb server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.514+0000 7f74bd200640 1 -- 172.21.15.31:0/1772579153 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c4060070 con 0x55a7b39ca700 2024-03-21T10:38:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.515+0000 7f74bd200640 1 -- 172.21.15.31:0/1772579153 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f74c4003e50 con 0x55a7b39ca700 2024-03-21T10:38:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.515+0000 7f74bd200640 1 -- 172.21.15.31:0/1772579153 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c4004240 con 0x55a7b39ca700 2024-03-21T10:38:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.515+0000 7f74d5dfd740 1 -- 172.21.15.31:0/1772579153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7b39ca700 msgr2=0x55a7b39c04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.515+0000 7f74d5dfd740 1 --2- 172.21.15.31:0/1772579153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7b39ca700 0x55a7b39c04f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f74c402c430 tx=0x7f74c4041000 comp rx=0 tx=0).stop 2024-03-21T10:38:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.515+0000 7f74d5dfd740 1 -- 172.21.15.31:0/1772579153 shutdown_connections 2024-03-21T10:38:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.515+0000 7f74d5dfd740 1 --2- 172.21.15.31:0/1772579153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7b39ca700 0x55a7b39c04f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.515+0000 7f74d5dfd740 1 --2- 172.21.15.31:0/1772579153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7b39c5590 0x55a7b39d65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.515+0000 7f74d5dfd740 1 --2- 172.21.15.31:0/1772579153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7b39c4ba0 0x55a7b39bae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.515+0000 7f74d5dfd740 1 -- 172.21.15.31:0/1772579153 >> 172.21.15.31:0/1772579153 conn(0x55a7b39c0110 msgr2=0x55a7b39c6030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.515+0000 7f74d5dfd740 1 -- 172.21.15.31:0/1772579153 shutdown_connections 2024-03-21T10:38:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.516+0000 7f74d5dfd740 1 -- 172.21.15.31:0/1772579153 wait complete. 2024-03-21T10:38:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.516+0000 7f74d5dfd740 1 -- start start 2024-03-21T10:38:04.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.516+0000 7f74d5dfd740 1 Processor -- start 2024-03-21T10:38:04.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.517+0000 7f74d5dfd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7b39c4ba0 0x55a7b39d5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.517+0000 7f74d5dfd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7b39c5590 0x55a7b39d6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.517+0000 7f74d5dfd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7b39ca700 0x55a7b39d2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.517+0000 7f74d5dfd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7b39dcae0 con 0x55a7b39ca700 2024-03-21T10:38:04.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.517+0000 7f74d5dfd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7b39dc480 con 0x55a7b39c4ba0 2024-03-21T10:38:04.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.517+0000 7f74d5dfd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7b39dd140 con 0x55a7b39c5590 2024-03-21T10:38:04.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.517+0000 7f74d5dfd740 10 client.0 _ll_register_callbacks cb 0x7f74b80d5220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:38:04.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.517+0000 7f74d0a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7b39c4ba0 0x55a7b39d5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.517+0000 7f74d0a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7b39c4ba0 0x55a7b39d5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:57676/0 (socket says 172.21.15.31:57676) 2024-03-21T10:38:04.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.517+0000 7f74d0a00640 1 -- 172.21.15.31:0/2664463479 learned_addr learned my addr 172.21.15.31:0/2664463479 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:04.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.517+0000 7f74cbe00640 1 --2- 172.21.15.31:0/2664463479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7b39c5590 0x55a7b39d6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.518+0000 7f74d1400640 1 --2- 172.21.15.31:0/2664463479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7b39ca700 0x55a7b39d2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.518+0000 7f74d0a00640 1 -- 172.21.15.31:0/2664463479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7b39c5590 msgr2=0x55a7b39d6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:04.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.518+0000 7f74d0a00640 1 --2- 172.21.15.31:0/2664463479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7b39c5590 0x55a7b39d6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.518+0000 7f74d0a00640 1 -- 172.21.15.31:0/2664463479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7b39ca700 msgr2=0x55a7b39d2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:04.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.518+0000 7f74d0a00640 1 --2- 172.21.15.31:0/2664463479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7b39ca700 0x55a7b39d2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.518+0000 7f74d0a00640 1 -- 172.21.15.31:0/2664463479 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a7b39d3100 con 0x55a7b39c4ba0 2024-03-21T10:38:04.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.518+0000 7f74cbe00640 1 --2- 172.21.15.31:0/2664463479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7b39c5590 0x55a7b39d6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:04.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.518+0000 7f74d0a00640 1 --2- 172.21.15.31:0/2664463479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7b39c4ba0 0x55a7b39d5ef0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f74c0003e40 tx=0x7f74c0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2c0ed94055ac10a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:04.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.518+0000 7f74b2000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:38:04.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.518+0000 7f74b2000640 1 -- 172.21.15.31:0/2664463479 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c00050b0 con 0x55a7b39c4ba0 2024-03-21T10:38:04.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.518+0000 7f74b2000640 1 -- 172.21.15.31:0/2664463479 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f74c001f040 con 0x55a7b39c4ba0 2024-03-21T10:38:04.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.518+0000 7f74d5dfd740 1 -- 172.21.15.31:0/2664463479 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f74c4003e50 con 0x55a7b39c4ba0 2024-03-21T10:38:04.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.519+0000 7f74b2000640 1 -- 172.21.15.31:0/2664463479 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74c003a050 con 0x55a7b39c4ba0 2024-03-21T10:38:04.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.519+0000 7f74d5dfd740 10 client.25379 fetch_fsmap learned FSMap version 23 2024-03-21T10:38:04.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.519+0000 7f74d5dfd740 1 -- 172.21.15.31:0/2664463479 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a7b3a37650 con 0x55a7b39c4ba0 2024-03-21T10:38:04.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.519+0000 7f74b2000640 1 -- 172.21.15.31:0/2664463479 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f74c4003e50 con 0x55a7b39c4ba0 2024-03-21T10:38:04.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.519+0000 7f74b2000640 1 -- 172.21.15.31:0/2664463479 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 23) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f74c002b520 con 0x55a7b39c4ba0 2024-03-21T10:38:04.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.519+0000 7f74d5dfd740 10 client.25379 fetch_fsmap finished waiting for FSMap version 23 2024-03-21T10:38:04.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.519+0000 7f74d5dfd740 10 client.25379 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-21T10:38:04.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.519+0000 7f74d5dfd740 20 client.25379 populate_metadata read hostname 'smithi031' 2024-03-21T10:38:04.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.519+0000 7f74d5dfd740 1 --2- 172.21.15.31:0/2664463479 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a7b3a3bb40 0x55a7b3a37940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:04.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.519+0000 7f74d5dfd740 4 client.25379 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-21T10:38:04.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.519+0000 7f74d5dfd740 1 -- 172.21.15.31:0/2664463479 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a7b39371d0 con 0x55a7b3a3bb40 2024-03-21T10:38:04.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.520+0000 7f74cbe00640 1 --2- 172.21.15.31:0/2664463479 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a7b3a3bb40 0x55a7b3a37940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:04.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.520+0000 7f74b2000640 10 client.25379 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:04.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.520+0000 7f74cbe00640 1 --2- 172.21.15.31:0/2664463479 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a7b3a3bb40 0x55a7b3a37940 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6493e582c161741 server_cookie=c9e0968c47f88808 in_seq=0 out_seq=0 2024-03-21T10:38:04.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.521+0000 7f74b2000640 1 -- 172.21.15.31:0/2664463479 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55a7b39371d0 con 0x55a7b3a3bb40 2024-03-21T10:38:04.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.521+0000 7f74b2000640 10 client.25379 handle_command_reply: tid=0 2024-03-21T10:38:04.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.521+0000 7f74d5dfd740 1 client.25379 shutdown 2024-03-21T10:38:04.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.521+0000 7f74d5dfd740 1 -- 172.21.15.31:0/2664463479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7b39c4ba0 msgr2=0x55a7b39d5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:04.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.521+0000 7f74d5dfd740 1 --2- 172.21.15.31:0/2664463479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7b39c4ba0 0x55a7b39d5ef0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f74c0003e40 tx=0x7f74c0011810 comp rx=0 tx=0).stop 2024-03-21T10:38:04.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.521+0000 7f74d5dfd740 1 -- 172.21.15.31:0/2664463479 shutdown_connections 2024-03-21T10:38:04.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.521+0000 7f74d5dfd740 1 --2- 172.21.15.31:0/2664463479 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a7b3a3bb40 0x55a7b3a37940 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.521+0000 7f74d5dfd740 1 --2- 172.21.15.31:0/2664463479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7b39ca700 0x55a7b39d2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.521+0000 7f74d5dfd740 1 --2- 172.21.15.31:0/2664463479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7b39c5590 0x55a7b39d6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.521+0000 7f74d5dfd740 1 --2- 172.21.15.31:0/2664463479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7b39c4ba0 0x55a7b39d5ef0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.521+0000 7f74b2000640 0 client.25379 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:04.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.521+0000 7f74d5dfd740 1 -- 172.21.15.31:0/2664463479 >> 172.21.15.31:0/2664463479 conn(0x55a7b39c0110 msgr2=0x55a7b39bae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:04.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.521+0000 7f74d5dfd740 1 -- 172.21.15.31:0/2664463479 shutdown_connections 2024-03-21T10:38:04.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.521+0000 7f74d5dfd740 1 -- 172.21.15.31:0/2664463479 wait complete. 2024-03-21T10:38:04.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.521+0000 7f74d5dfd740 20 client.25379 trim_cache size 0 max 16384 2024-03-21T10:38:04.523 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:38:04.523 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "3847408c-e76d-11ee-95cd-87774f69a715", 2024-03-21T10:38:04.523 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2024-03-21T10:38:04.523 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24596, 2024-03-21T10:38:04.523 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-03-21T10:38:04.523 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-03-21T10:38:04.523 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-03-21T10:38:04.523 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 177.524354942, 2024-03-21T10:38:04.523 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 23, 2024-03-21T10:38:04.523 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 93, 2024-03-21T10:38:04.524 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 89, 2024-03-21T10:38:04.524 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 183.34801405499999 2024-03-21T10:38:04.524 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:38:04.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.522+0000 7f74d3400640 1 -- 172.21.15.31:0/688138427 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f74b40a2fa0 msgr2=0x7f74b40a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:04.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.522+0000 7f74d3400640 1 --2- 172.21.15.31:0/688138427 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f74b40a2fa0 0x7f74b40a5460 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f74c4006010 tx=0x7f74c4004a40 comp rx=0 tx=0).stop 2024-03-21T10:38:04.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.522+0000 7f74d3400640 1 -- 172.21.15.31:0/688138427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74cc1028c0 msgr2=0x7f74cc19a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:04.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.522+0000 7f74d3400640 1 --2- 172.21.15.31:0/688138427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74cc1028c0 0x7f74cc19a0a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f74b800e9c0 tx=0x7f74b800ee80 comp rx=0 tx=0).stop 2024-03-21T10:38:04.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.522+0000 7f74d3400640 1 -- 172.21.15.31:0/688138427 shutdown_connections 2024-03-21T10:38:04.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.522+0000 7f74d3400640 1 --2- 172.21.15.31:0/688138427 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f74b40a2fa0 0x7f74b40a5460 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.522+0000 7f74d3400640 1 --2- 172.21.15.31:0/688138427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74cc103280 0x7f74cc1a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.522+0000 7f74d3400640 1 --2- 172.21.15.31:0/688138427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74cc1028c0 0x7f74cc19a0a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.522+0000 7f74d3400640 1 --2- 172.21.15.31:0/688138427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74cc1016d0 0x7f74cc199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:04.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.522+0000 7f74d3400640 1 -- 172.21.15.31:0/688138427 >> 172.21.15.31:0/688138427 conn(0x7f74cc0fcc60 msgr2=0x7f74cc0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:04.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.522+0000 7f74d3400640 1 -- 172.21.15.31:0/688138427 shutdown_connections 2024-03-21T10:38:04.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:04.522+0000 7f74d3400640 1 -- 172.21.15.31:0/688138427 wait complete. 2024-03-21T10:38:04.539 INFO:teuthology.run_tasks:Running task kclient... 2024-03-21T10:38:04.553 INFO:tasks.kclient:Mounting kernel clients... 2024-03-21T10:38:04.553 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-21T10:38:04.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:38:04.553 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-03-21T10:38:04.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:04 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3603250068' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:38:04.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:04 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/341144963' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:38:04.594 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-21T10:38:04.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:38:04.595 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-21T10:38:04.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:38:04.732 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link delete ceph-brx 2024-03-21T10:38:04.855 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-03-21T10:38:04.855 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-21T10:38:04.856 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-21T10:38:04.856 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph fs ls 2024-03-21T10:38:05.286 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:38:05.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:05 smithi040 ceph-mon[36154]: pgmap v418: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:38:05.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:05 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:05.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:05 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:05.782 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:05 smithi031 ceph-mon[31673]: pgmap v418: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:38:05.783 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:05 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:05.783 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:05 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:05.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:05 smithi070 ceph-mon[36330]: pgmap v418: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:38:05.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:05 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:05.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:05 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:06.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:06 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:38:06.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:06 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:38:06.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:06 smithi040 ceph-mon[36154]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:06.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:06 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:38:06.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:06 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:38:06.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:06 smithi031 ceph-mon[31673]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:06.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:06 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:38:06.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:06 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:38:06.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:06 smithi070 ceph-mon[36330]: from='mgr.14152 172.21.15.31:0/595200071' entity='mgr.x' 2024-03-21T10:38:07.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:07 smithi031 ceph-mon[31673]: pgmap v419: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:38:07.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.784+0000 7fda07a00700 1 -- 172.21.15.31:0/247421432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda001081e0 msgr2=0x7fda001085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:07.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.784+0000 7fda07a00700 1 --2- 172.21.15.31:0/247421432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda001081e0 0x7fda001085c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4009990 tx=0x7fd9f4009ca0 comp rx=0 tx=0).stop 2024-03-21T10:38:07.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.784+0000 7fda07a00700 1 -- 172.21.15.31:0/247421432 shutdown_connections 2024-03-21T10:38:07.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.784+0000 7fda07a00700 1 --2- 172.21.15.31:0/247421432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda0006a7e0 0x7fda0010f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:07.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.784+0000 7fda07a00700 1 --2- 172.21.15.31:0/247421432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda001081e0 0x7fda001085c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:07.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.784+0000 7fda07a00700 1 --2- 172.21.15.31:0/247421432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fda00069e40 0x7fda0006a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:07.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.784+0000 7fda07a00700 1 -- 172.21.15.31:0/247421432 >> 172.21.15.31:0/247421432 conn(0x7fda00077fc0 msgr2=0x7fda000783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:07.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.784+0000 7fda07a00700 1 -- 172.21.15.31:0/247421432 shutdown_connections 2024-03-21T10:38:07.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.784+0000 7fda07a00700 1 -- 172.21.15.31:0/247421432 wait complete. 2024-03-21T10:38:07.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.785+0000 7fda07a00700 1 Processor -- start 2024-03-21T10:38:07.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.785+0000 7fda07a00700 1 -- start start 2024-03-21T10:38:07.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.786+0000 7fda07a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fda00069e40 0x7fda00101160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:07.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.786+0000 7fda07a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda0006a7e0 0x7fda001016a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:07.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.786+0000 7fda07a00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda001081e0 0x7fda00101be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:07.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.786+0000 7fda07a00700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda00112470 con 0x7fda001081e0 2024-03-21T10:38:07.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.786+0000 7fda07a00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda001122f0 con 0x7fda0006a7e0 2024-03-21T10:38:07.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.786+0000 7fda07a00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda001125f0 con 0x7fda00069e40 2024-03-21T10:38:07.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.786+0000 7fd9ffe00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda0006a7e0 0x7fda001016a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:07.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.786+0000 7fda05000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fda00069e40 0x7fda00101160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:07.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.786+0000 7fd9ffe00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda0006a7e0 0x7fda001016a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51256/0 (socket says 172.21.15.31:51256) 2024-03-21T10:38:07.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.786+0000 7fd9ffe00700 1 -- 172.21.15.31:0/715339400 learned_addr learned my addr 172.21.15.31:0/715339400 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:07.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.786+0000 7fda05a00700 1 --2- 172.21.15.31:0/715339400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda001081e0 0x7fda00101be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:07.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.787+0000 7fda05000700 1 -- 172.21.15.31:0/715339400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda0006a7e0 msgr2=0x7fda001016a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:07.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.787+0000 7fda05000700 1 --2- 172.21.15.31:0/715339400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda0006a7e0 0x7fda001016a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:07.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.787+0000 7fda05000700 1 -- 172.21.15.31:0/715339400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda001081e0 msgr2=0x7fda00101be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:07.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.787+0000 7fda05000700 1 --2- 172.21.15.31:0/715339400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda001081e0 0x7fda00101be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:07.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.787+0000 7fda05000700 1 -- 172.21.15.31:0/715339400 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda001a6e10 con 0x7fda00069e40 2024-03-21T10:38:07.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.787+0000 7fda05a00700 1 --2- 172.21.15.31:0/715339400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda001081e0 0x7fda00101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:07.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.787+0000 7fda05000700 1 --2- 172.21.15.31:0/715339400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fda00069e40 0x7fda00101160 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd9f40098e0 tx=0x7fd9f400b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:07.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.787+0000 7fd9fd600700 1 -- 172.21.15.31:0/715339400 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f4027070 con 0x7fda00069e40 2024-03-21T10:38:07.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.787+0000 7fd9fd600700 1 -- 172.21.15.31:0/715339400 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9f400fde0 con 0x7fda00069e40 2024-03-21T10:38:07.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.787+0000 7fda07a00700 1 -- 172.21.15.31:0/715339400 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda001a70a0 con 0x7fda00069e40 2024-03-21T10:38:07.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.787+0000 7fd9fd600700 1 -- 172.21.15.31:0/715339400 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f40173f0 con 0x7fda00069e40 2024-03-21T10:38:07.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.788+0000 7fda07a00700 1 -- 172.21.15.31:0/715339400 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda001a7550 con 0x7fda00069e40 2024-03-21T10:38:07.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.789+0000 7fda07a00700 1 -- 172.21.15.31:0/715339400 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda00068e50 con 0x7fda00069e40 2024-03-21T10:38:07.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.791+0000 7fd9fd600700 1 -- 172.21.15.31:0/715339400 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fd9f402a070 con 0x7fda00069e40 2024-03-21T10:38:07.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.792+0000 7fd9fd600700 1 --2- 172.21.15.31:0/715339400 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd9e80a3030 0x7fd9e80a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:07.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.792+0000 7fd9fd600700 1 -- 172.21.15.31:0/715339400 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fd9f40d02c0 con 0x7fda00069e40 2024-03-21T10:38:07.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.792+0000 7fd9ffe00700 1 --2- 172.21.15.31:0/715339400 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd9e80a3030 0x7fd9e80a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:07.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.793+0000 7fd9ffe00700 1 --2- 172.21.15.31:0/715339400 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd9e80a3030 0x7fd9e80a54f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec009680 tx=0x7fd9ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:07.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:07.796+0000 7fd9fd600700 1 -- 172.21.15.31:0/715339400 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd9f4013070 con 0x7fda00069e40 2024-03-21T10:38:08.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:07 smithi070 ceph-mon[36330]: pgmap v419: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:38:08.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:08.080+0000 7fda07a00700 1 -- 172.21.15.31:0/715339400 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fda0002da30 con 0x7fda00069e40 2024-03-21T10:38:08.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:08.081+0000 7fd9fd600700 1 -- 172.21.15.31:0/715339400 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) v1 ==== 53+0+88 (secure 0 0 0) 0x7fd9f409ce50 con 0x7fda00069e40 2024-03-21T10:38:08.079 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-21T10:38:08.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:08.085+0000 7fda07a00700 1 -- 172.21.15.31:0/715339400 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd9e80a3030 msgr2=0x7fd9e80a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:08.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:08.085+0000 7fda07a00700 1 --2- 172.21.15.31:0/715339400 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd9e80a3030 0x7fd9e80a54f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec009680 tx=0x7fd9ec009260 comp rx=0 tx=0).stop 2024-03-21T10:38:08.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:08.085+0000 7fda07a00700 1 -- 172.21.15.31:0/715339400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fda00069e40 msgr2=0x7fda00101160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:08.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:08.085+0000 7fda07a00700 1 --2- 172.21.15.31:0/715339400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fda00069e40 0x7fda00101160 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd9f40098e0 tx=0x7fd9f400b840 comp rx=0 tx=0).stop 2024-03-21T10:38:08.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:08.085+0000 7fda07a00700 1 -- 172.21.15.31:0/715339400 shutdown_connections 2024-03-21T10:38:08.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:08.086+0000 7fda07a00700 1 --2- 172.21.15.31:0/715339400 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd9e80a3030 0x7fd9e80a54f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:08.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:08.086+0000 7fda07a00700 1 --2- 172.21.15.31:0/715339400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda001081e0 0x7fda00101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:08.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:08.086+0000 7fda07a00700 1 --2- 172.21.15.31:0/715339400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fda00069e40 0x7fda00101160 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:08.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:08.086+0000 7fda07a00700 1 --2- 172.21.15.31:0/715339400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda0006a7e0 0x7fda001016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:08.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:08.086+0000 7fda07a00700 1 -- 172.21.15.31:0/715339400 >> 172.21.15.31:0/715339400 conn(0x7fda00077fc0 msgr2=0x7fda000ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:08.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:08.086+0000 7fda07a00700 1 -- 172.21.15.31:0/715339400 shutdown_connections 2024-03-21T10:38:08.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:08.086+0000 7fda07a00700 1 -- 172.21.15.31:0/715339400 wait complete. 2024-03-21T10:38:08.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:07 smithi040 ceph-mon[36154]: pgmap v419: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:38:08.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:08 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/715339400' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-21T10:38:09.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:08 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/715339400' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-21T10:38:09.088 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-21T10:38:09.088 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-21T10:38:09.088 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi031.front.sepia.ceph.com 2024-03-21T10:38:09.088 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-21T10:38:09.088 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-21T10:38:09.088 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-21T10:38:09.088 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-21T10:38:09.088 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-21T10:38:09.088 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-21T10:38:09.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:38:09.089 DEBUG:teuthology.orchestra.run.smithi031:> ip addr 2024-03-21T10:38:09.151 INFO:teuthology.orchestra.run.smithi031.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-21T10:38:09.151 INFO:teuthology.orchestra.run.smithi031.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-21T10:38:09.151 INFO:teuthology.orchestra.run.smithi031.stdout: inet 127.0.0.1/8 scope host lo 2024-03-21T10:38:09.151 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-03-21T10:38:09.151 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 ::1/128 scope host 2024-03-21T10:38:09.151 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-03-21T10:38:09.151 INFO:teuthology.orchestra.run.smithi031.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-21T10:38:09.151 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cc brd ff:ff:ff:ff:ff:ff 2024-03-21T10:38:09.151 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f0 2024-03-21T10:38:09.151 INFO:teuthology.orchestra.run.smithi031.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-21T10:38:09.152 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cd brd ff:ff:ff:ff:ff:ff 2024-03-21T10:38:09.152 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f1 2024-03-21T10:38:09.152 INFO:teuthology.orchestra.run.smithi031.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-21T10:38:09.152 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:ba brd ff:ff:ff:ff:ff:ff 2024-03-21T10:38:09.152 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f0 2024-03-21T10:38:09.152 INFO:teuthology.orchestra.run.smithi031.stdout: inet 172.21.15.31/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-21T10:38:09.152 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft 41845sec preferred_lft 41845sec 2024-03-21T10:38:09.152 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::ae1f:6bff:fea5:caba/64 scope link noprefixroute 2024-03-21T10:38:09.152 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-03-21T10:38:09.152 INFO:teuthology.orchestra.run.smithi031.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-21T10:38:09.152 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:bb brd ff:ff:ff:ff:ff:ff 2024-03-21T10:38:09.152 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f1 2024-03-21T10:38:09.153 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-21T10:38:09.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:38:09.153 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec bash -c ' 2024-03-21T10:38:09.153 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-03-21T10:38:09.153 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link add name ceph-brx type bridge 2024-03-21T10:38:09.154 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip addr flush dev ceph-brx 2024-03-21T10:38:09.154 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set ceph-brx up 2024-03-21T10:38:09.154 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-21T10:38:09.154 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-03-21T10:38:09.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:08 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/715339400' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-21T10:38:09.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:38:09.373 DEBUG:teuthology.orchestra.run.smithi031:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-21T10:38:09.456 INFO:teuthology.orchestra.run.smithi031.stdout:1 2024-03-21T10:38:09.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:38:09.460 DEBUG:teuthology.orchestra.run.smithi031:> route 2024-03-21T10:38:09.508 INFO:teuthology.orchestra.run.smithi031.stdout:Kernel IP routing table 2024-03-21T10:38:09.509 INFO:teuthology.orchestra.run.smithi031.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-21T10:38:09.509 INFO:teuthology.orchestra.run.smithi031.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-21T10:38:09.509 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-21T10:38:09.509 INFO:teuthology.orchestra.run.smithi031.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-21T10:38:09.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:38:09.509 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec bash -c ' 2024-03-21T10:38:09.509 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-03-21T10:38:09.509 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-21T10:38:09.510 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-21T10:38:09.510 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-21T10:38:09.510 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-03-21T10:38:09.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:38:09.690 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-03-21T10:38:09.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:38:09.739 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list-id 2024-03-21T10:38:09.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:38:09.790 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec bash -c ' 2024-03-21T10:38:09.790 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-03-21T10:38:09.790 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T10:38:09.790 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-21T10:38:09.790 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-03-21T10:38:09.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:09 smithi031 ceph-mon[31673]: pgmap v420: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:38:09.933 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-21T10:38:09.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:38:09.934 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec bash -c ' 2024-03-21T10:38:09.934 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-03-21T10:38:09.934 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-21T10:38:09.934 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-21T10:38:09.934 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-21T10:38:09.934 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-21T10:38:09.934 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-21T10:38:09.934 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-03-21T10:38:09.974 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:09 smithi070 ceph-mon[36330]: pgmap v420: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:38:10.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:09 smithi040 ceph-mon[36154]: pgmap v420: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:38:10.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:38:10.205 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec bash -c ' 2024-03-21T10:38:10.205 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-03-21T10:38:10.205 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set brx.0 up 2024-03-21T10:38:10.205 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set dev brx.0 master ceph-brx 2024-03-21T10:38:10.205 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-03-21T10:38:10.372 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-21T10:38:10.373 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-21T10:38:10.419 INFO:teuthology.orchestra.run.smithi031.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-21T10:38:10.419 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-21T10:38:10.419 DEBUG:teuthology.orchestra.run.smithi031:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-21T10:38:10.514 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-21T10:38:10.515 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-21T10:38:10.515 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/b825424c-c202-4d75-aea8-4547df630897 2024-03-21T10:38:10.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:38:10.515 DEBUG:teuthology.orchestra.run.smithi031:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/b825424c-c202-4d75-aea8-4547df630897 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-03-21T10:38:10.651 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-03-21T10:38:10.651 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2024-03-21T10:38:10.652 INFO:teuthology.orchestra.run.smithi031.stdout:invalid new device string format 2024-03-21T10:38:11.334 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-03-21T10:38:11.335 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2024-03-21T10:38:11.335 INFO:teuthology.orchestra.run.smithi031.stdout:invalid new device string format 2024-03-21T10:38:11.335 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: resolved to: "172.21.15.31:6789,172.21.15.40:6789,172.21.15.70:6789" 2024-03-21T10:38:11.335 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.31:6789,172.21.15.40:6789,172.21.15.70:6789:/volumes/qa/sv_1/b825424c-c202-4d75-aea8-4547df630897 2024-03-21T10:38:11.335 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync,key=0,fsid=3847408c-e76d-11ee-95cd-87774f69a715" will pass to kernel 2024-03-21T10:38:11.338 INFO:teuthology.orchestra.run.smithi031.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-21T10:38:11.338 INFO:teuthology.orchestra.run.smithi031.stdout: You just mounted a file system that supports labels which does not 2024-03-21T10:38:11.338 INFO:teuthology.orchestra.run.smithi031.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-21T10:38:11.338 INFO:teuthology.orchestra.run.smithi031.stdout: applications will generate AVC messages and not be allowed access to 2024-03-21T10:38:11.338 INFO:teuthology.orchestra.run.smithi031.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-21T10:38:11.343 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-21T10:38:11.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:38:11.343 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-21T10:38:11.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:38:11.451 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-21T10:38:11.508 INFO:teuthology.orchestra.run.smithi031.stdout:client25388 2024-03-21T10:38:11.508 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-21T10:38:11.508 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T10:38:11.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:38:11.556 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-21T10:38:11.655 INFO:teuthology.orchestra.run.smithi031.stdout:client25388 2024-03-21T10:38:11.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:38:11.656 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-21T10:38:11.717 INFO:teuthology.orchestra.run.smithi031.stdout:client25388 2024-03-21T10:38:11.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:38:11.717 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/3847408c-e76d-11ee-95cd-87774f69a715.client25388/status') 2024-03-21T10:38:11.804 INFO:teuthology.orchestra.run.smithi031.stdout:instance: client.25388 (3)172.21.15.31:0/653233213 2024-03-21T10:38:11.805 INFO:teuthology.orchestra.run.smithi031.stdout:blocklisted: false 2024-03-21T10:38:11.805 INFO:teuthology.orchestra.run.smithi031.stderr:0+1 records in 2024-03-21T10:38:11.805 INFO:teuthology.orchestra.run.smithi031.stderr:0+1 records out 2024-03-21T10:38:11.805 INFO:teuthology.orchestra.run.smithi031.stderr:70 bytes copied, 6.9372e-05 s, 1.0 MB/s 2024-03-21T10:38:11.809 INFO:teuthology.run_tasks:Running task exec... 2024-03-21T10:38:11.819 INFO:teuthology.task.exec:Executing custom commands... 2024-03-21T10:38:11.819 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2024-03-21T10:38:11.819 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-03-21T10:38:12.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:11 smithi031 ceph-mon[31673]: pgmap v421: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:38:12.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.058+0000 7f8799600640 1 -- 172.21.15.31:0/573825048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8794102700 msgr2=0x7f8794102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:12.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.058+0000 7f8799600640 1 --2- 172.21.15.31:0/573825048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8794102700 0x7f8794102b80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8780009900 tx=0x7f878002bc30 comp rx=0 tx=0).stop 2024-03-21T10:38:12.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.058+0000 7f8799600640 1 -- 172.21.15.31:0/573825048 shutdown_connections 2024-03-21T10:38:12.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.058+0000 7f8799600640 1 --2- 172.21.15.31:0/573825048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87941030c0 0x7f8794109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:12.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.058+0000 7f8799600640 1 --2- 172.21.15.31:0/573825048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8794102700 0x7f8794102b80 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:12.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.058+0000 7f8799600640 1 --2- 172.21.15.31:0/573825048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8794101510 0x7f8794101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:12.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.058+0000 7f8799600640 1 -- 172.21.15.31:0/573825048 >> 172.21.15.31:0/573825048 conn(0x7f87940fca80 msgr2=0x7f87940feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:12.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.058+0000 7f8799600640 1 -- 172.21.15.31:0/573825048 shutdown_connections 2024-03-21T10:38:12.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.058+0000 7f8799600640 1 -- 172.21.15.31:0/573825048 wait complete. 2024-03-21T10:38:12.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.059+0000 7f8799600640 1 Processor -- start 2024-03-21T10:38:12.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.059+0000 7f8799600640 1 -- start start 2024-03-21T10:38:12.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.059+0000 7f8799600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8794101510 0x7f87941999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.059+0000 7f8799600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8794102700 0x7f8794199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.059+0000 7f8799600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87941030c0 0x7f87941a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.059+0000 7f8799600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879410d1f0 con 0x7f87941030c0 2024-03-21T10:38:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.059+0000 7f8799600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879410d070 con 0x7f8794101510 2024-03-21T10:38:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.059+0000 7f8799600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879410d370 con 0x7f8794102700 2024-03-21T10:38:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.060+0000 7f8793e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8794101510 0x7f87941999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.060+0000 7f8793e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8794101510 0x7f87941999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51288/0 (socket says 172.21.15.31:51288) 2024-03-21T10:38:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.060+0000 7f8793e00640 1 -- 172.21.15.31:0/1645852435 learned_addr learned my addr 172.21.15.31:0/1645852435 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.060+0000 7f8793400640 1 --2- 172.21.15.31:0/1645852435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8794102700 0x7f8794199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.060+0000 7f8798c00640 1 --2- 172.21.15.31:0/1645852435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87941030c0 0x7f87941a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.060+0000 7f8793e00640 1 -- 172.21.15.31:0/1645852435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8794102700 msgr2=0x7f8794199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.060+0000 7f8793e00640 1 --2- 172.21.15.31:0/1645852435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8794102700 0x7f8794199f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.060+0000 7f8793e00640 1 -- 172.21.15.31:0/1645852435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87941030c0 msgr2=0x7f87941a0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.060+0000 7f8793e00640 1 --2- 172.21.15.31:0/1645852435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87941030c0 0x7f87941a0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:12.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.060+0000 7f8793e00640 1 -- 172.21.15.31:0/1645852435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87941a1610 con 0x7f8794101510 2024-03-21T10:38:12.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.060+0000 7f8798c00640 1 --2- 172.21.15.31:0/1645852435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87941030c0 0x7f87941a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:12.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.060+0000 7f8793400640 1 --2- 172.21.15.31:0/1645852435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8794102700 0x7f8794199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:12.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.060+0000 7f8793e00640 1 --2- 172.21.15.31:0/1645852435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8794101510 0x7f87941999f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f878400e9c0 tx=0x7f878400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:12.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.061+0000 7f8790c00640 1 -- 172.21.15.31:0/1645852435 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878400cd80 con 0x7f8794101510 2024-03-21T10:38:12.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.061+0000 7f8790c00640 1 -- 172.21.15.31:0/1645852435 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87840035f0 con 0x7f8794101510 2024-03-21T10:38:12.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.061+0000 7f8799600640 1 -- 172.21.15.31:0/1645852435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87941a1900 con 0x7f8794101510 2024-03-21T10:38:12.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.061+0000 7f8790c00640 1 -- 172.21.15.31:0/1645852435 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8784010690 con 0x7f8794101510 2024-03-21T10:38:12.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.061+0000 7f8799600640 1 -- 172.21.15.31:0/1645852435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87941a1e70 con 0x7f8794101510 2024-03-21T10:38:12.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.062+0000 7f8799600640 1 -- 172.21.15.31:0/1645852435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87941a2100 con 0x7f8794101510 2024-03-21T10:38:12.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.066+0000 7f8790c00640 1 -- 172.21.15.31:0/1645852435 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f87840037a0 con 0x7f8794101510 2024-03-21T10:38:12.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.067+0000 7f8790c00640 1 --2- 172.21.15.31:0/1645852435 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f87780a2f80 0x7f87780a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:12.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.067+0000 7f8790c00640 1 -- 172.21.15.31:0/1645852435 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f87840cfbc0 con 0x7f8794101510 2024-03-21T10:38:12.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.067+0000 7f8793400640 1 --2- 172.21.15.31:0/1645852435 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f87780a2f80 0x7f87780a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:12.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.068+0000 7f8793400640 1 --2- 172.21.15.31:0/1645852435 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f87780a2f80 0x7f87780a5440 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f878002c240 tx=0x7f878002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:12.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.071+0000 7f8790c00640 1 -- 172.21.15.31:0/1645852435 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f878409c630 con 0x7f8794101510 2024-03-21T10:38:12.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:11 smithi070 ceph-mon[36330]: pgmap v421: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:38:12.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:11 smithi040 ceph-mon[36154]: pgmap v421: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-21T10:38:12.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.317+0000 7f8799600640 1 -- 172.21.15.31:0/1645852435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) v1 -- 0x7f87941a2870 con 0x7f8794101510 2024-03-21T10:38:12.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.720+0000 7f8790c00640 1 -- 172.21.15.31:0/1645852435 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v24) v1 ==== 117+0+0 (secure 0 0 0) 0x7f878409bf30 con 0x7f8794101510 2024-03-21T10:38:12.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.724+0000 7f8799600640 1 -- 172.21.15.31:0/1645852435 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f87780a2f80 msgr2=0x7f87780a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:12.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.724+0000 7f8799600640 1 --2- 172.21.15.31:0/1645852435 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f87780a2f80 0x7f87780a5440 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f878002c240 tx=0x7f878002df90 comp rx=0 tx=0).stop 2024-03-21T10:38:12.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.724+0000 7f8799600640 1 -- 172.21.15.31:0/1645852435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8794101510 msgr2=0x7f87941999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:12.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.724+0000 7f8799600640 1 --2- 172.21.15.31:0/1645852435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8794101510 0x7f87941999f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f878400e9c0 tx=0x7f878400ee80 comp rx=0 tx=0).stop 2024-03-21T10:38:12.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.724+0000 7f8799600640 1 -- 172.21.15.31:0/1645852435 shutdown_connections 2024-03-21T10:38:12.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.724+0000 7f8799600640 1 --2- 172.21.15.31:0/1645852435 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f87780a2f80 0x7f87780a5440 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:12.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.724+0000 7f8799600640 1 --2- 172.21.15.31:0/1645852435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87941030c0 0x7f87941a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:12.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.724+0000 7f8799600640 1 --2- 172.21.15.31:0/1645852435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8794102700 0x7f8794199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:12.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.724+0000 7f8799600640 1 --2- 172.21.15.31:0/1645852435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8794101510 0x7f87941999f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:12.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.724+0000 7f8799600640 1 -- 172.21.15.31:0/1645852435 >> 172.21.15.31:0/1645852435 conn(0x7f87940fca80 msgr2=0x7f87940738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:12.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.725+0000 7f8799600640 1 -- 172.21.15.31:0/1645852435 shutdown_connections 2024-03-21T10:38:12.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:12.725+0000 7f8799600640 1 -- 172.21.15.31:0/1645852435 wait complete. 2024-03-21T10:38:12.746 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-21T10:38:12.759 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-21T10:38:12.778 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T10:38:12.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.000+0000 7f5b73200640 1 -- 172.21.15.31:0/2175326257 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5b741049b0 msgr2=0x7f5b74104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:12.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.000+0000 7f5b73200640 1 --2- 172.21.15.31:0/2175326257 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5b741049b0 0x7f5b74104e30 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5b60009900 tx=0x7f5b6002bc50 comp rx=0 tx=0).stop 2024-03-21T10:38:12.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.000+0000 7f5b73200640 1 -- 172.21.15.31:0/2175326257 shutdown_connections 2024-03-21T10:38:12.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.000+0000 7f5b73200640 1 --2- 172.21.15.31:0/2175326257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b74105370 0x7f5b7410bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:12.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.000+0000 7f5b73200640 1 --2- 172.21.15.31:0/2175326257 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5b741049b0 0x7f5b74104e30 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:12.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.000+0000 7f5b73200640 1 --2- 172.21.15.31:0/2175326257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b741037c0 0x7f5b74103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:12.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.000+0000 7f5b73200640 1 -- 172.21.15.31:0/2175326257 >> 172.21.15.31:0/2175326257 conn(0x7f5b740fed50 msgr2=0x7f5b74101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:12.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.000+0000 7f5b73200640 1 -- 172.21.15.31:0/2175326257 shutdown_connections 2024-03-21T10:38:12.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.000+0000 7f5b73200640 1 -- 172.21.15.31:0/2175326257 wait complete. 2024-03-21T10:38:12.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.001+0000 7f5b73200640 1 Processor -- start 2024-03-21T10:38:12.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.002+0000 7f5b73200640 1 -- start start 2024-03-21T10:38:13.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.002+0000 7f5b73200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5b741037c0 0x7f5b74199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:13.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.002+0000 7f5b73200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b741049b0 0x7f5b74199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:13.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.002+0000 7f5b73200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b74105370 0x7f5b741a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:13.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.002+0000 7f5b73200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7410f470 con 0x7f5b74105370 2024-03-21T10:38:13.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.002+0000 7f5b73200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7410f2f0 con 0x7f5b741049b0 2024-03-21T10:38:13.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.002+0000 7f5b73200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7410f5f0 con 0x7f5b741037c0 2024-03-21T10:38:13.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.002+0000 7f5b71e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5b741037c0 0x7f5b74199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:13.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.003+0000 7f5b71400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b741049b0 0x7f5b74199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:13.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.002+0000 7f5b71e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5b741037c0 0x7f5b74199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:55272/0 (socket says 172.21.15.31:55272) 2024-03-21T10:38:13.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.003+0000 7f5b71e00640 1 -- 172.21.15.31:0/181187631 learned_addr learned my addr 172.21.15.31:0/181187631 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:13.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.003+0000 7f5b72800640 1 --2- 172.21.15.31:0/181187631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b74105370 0x7f5b741a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:13.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.003+0000 7f5b71e00640 1 -- 172.21.15.31:0/181187631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b741049b0 msgr2=0x7f5b74199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:13.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.003+0000 7f5b71e00640 1 --2- 172.21.15.31:0/181187631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b741049b0 0x7f5b74199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:13.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.003+0000 7f5b71e00640 1 -- 172.21.15.31:0/181187631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b74105370 msgr2=0x7f5b741a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:13.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.003+0000 7f5b71e00640 1 --2- 172.21.15.31:0/181187631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b74105370 0x7f5b741a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:13.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.003+0000 7f5b71e00640 1 -- 172.21.15.31:0/181187631 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b741a16a0 con 0x7f5b741037c0 2024-03-21T10:38:13.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.003+0000 7f5b71400640 1 --2- 172.21.15.31:0/181187631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b741049b0 0x7f5b74199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:13.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.003+0000 7f5b71e00640 1 --2- 172.21.15.31:0/181187631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5b741037c0 0x7f5b74199a80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5b6400e9c0 tx=0x7f5b6400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:13.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.003+0000 7f5b6aa00640 1 -- 172.21.15.31:0/181187631 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6400cd80 con 0x7f5b741037c0 2024-03-21T10:38:13.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.004+0000 7f5b6aa00640 1 -- 172.21.15.31:0/181187631 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b640035f0 con 0x7f5b741037c0 2024-03-21T10:38:13.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.004+0000 7f5b73200640 1 -- 172.21.15.31:0/181187631 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b741a1990 con 0x7f5b741037c0 2024-03-21T10:38:13.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.004+0000 7f5b6aa00640 1 -- 172.21.15.31:0/181187631 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b64010690 con 0x7f5b741037c0 2024-03-21T10:38:13.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.004+0000 7f5b73200640 1 -- 172.21.15.31:0/181187631 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b741a1f00 con 0x7f5b741037c0 2024-03-21T10:38:13.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.005+0000 7f5b73200640 1 -- 172.21.15.31:0/181187631 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b7404f470 con 0x7f5b741037c0 2024-03-21T10:38:13.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.008+0000 7f5b6aa00640 1 -- 172.21.15.31:0/181187631 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f5b640037a0 con 0x7f5b741037c0 2024-03-21T10:38:13.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.009+0000 7f5b6aa00640 1 --2- 172.21.15.31:0/181187631 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5b580a2f90 0x7f5b580a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:13.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.010+0000 7f5b6aa00640 1 -- 172.21.15.31:0/181187631 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f5b640cfab0 con 0x7f5b741037c0 2024-03-21T10:38:13.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.010+0000 7f5b71400640 1 --2- 172.21.15.31:0/181187631 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5b580a2f90 0x7f5b580a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:13.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.011+0000 7f5b71400640 1 --2- 172.21.15.31:0/181187631 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5b580a2f90 0x7f5b580a5450 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5b6002c260 tx=0x7f5b60000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:13.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.015+0000 7f5b6aa00640 1 -- 172.21.15.31:0/181187631 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5b6409c580 con 0x7f5b741037c0 2024-03-21T10:38:13.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:12 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1645852435' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-21T10:38:13.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:12 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-21T10:38:13.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:12 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1645852435' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-21T10:38:13.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:12 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-21T10:38:13.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:12 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1645852435' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-21T10:38:13.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:12 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-21T10:38:13.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.221+0000 7f5b73200640 1 -- 172.21.15.31:0/181187631 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b740622a0 con 0x7f5b741037c0 2024-03-21T10:38:13.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.222+0000 7f5b6aa00640 1 -- 172.21.15.31:0/181187631 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24553 (secure 0 0 0) 0x7f5b6409be80 con 0x7f5b741037c0 2024-03-21T10:38:13.221 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:38:13.221 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":93,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","created":"2024-03-21T10:25:41.971117+0000","modified":"2024-03-21T10:36:56.301695+0000","last_up_change":"2024-03-21T10:34:07.757196+0000","last_in_change":"2024-03-21T10:33:47.460406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T10:29:44.553968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T10:35:04.945449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T10:35:05.970589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T10:35:09.688711+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e0244d33-00e2-47d5-9f66-34efc138c462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6803","nonce":3089885125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6805","nonce":3089885125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6809","nonce":3089885125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6807","nonce":3089885125}]},"public_addr":"172.21.15.31:6803/3089885125","cluster_addr":"172.21.15.31:6805/3089885125","heartbeat_back_addr":"172.21.15.31:6809/3089885125","heartbeat_front_addr":"172.21.15.31:6807/3089885125","state":["exists","up"]},{"osd":1,"uuid":"76a63504-beab-4354-a6c3-eebc3b16ad03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6801","nonce":618840037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6803","nonce":618840037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6807","nonce":618840037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6805","nonce":618840037}]},"public_addr":"172.21.15.40:6801/618840037","cluster_addr":"172.21.15.40:6803/618840037","heartbeat_back_addr":"172.21.15.40:6807/618840037","heartbeat_front_addr":"172.21.15.40:6805/618840037","state":["exists","up"]},{"osd":2,"uuid":"fa191bd6-9a21-4ee2-8adf-8a63c9cfd821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6801","nonce":2524184117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6803","nonce":2524184117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6807","nonce":2524184117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6805","nonce":2524184117}]},"public_addr":"172.21.15.70:6801/2524184117","cluster_addr":"172.21.15.70:6803/2524184117","heartbeat_back_addr":"172.21.15.70:6807/2524184117","heartbeat_front_addr":"172.21.15.70:6805/2524184117","state":["exists","up"]},{"osd":3,"uuid":"edb44507-ffb0-42a4-827b-08461e1bf952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6811","nonce":2823036296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6813","nonce":2823036296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6817","nonce":2823036296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6815","nonce":2823036296}]},"public_addr":"172.21.15.31:6811/2823036296","cluster_addr":"172.21.15.31:6813/2823036296","heartbeat_back_addr":"172.21.15.31:6817/2823036296","heartbeat_front_addr":"172.21.15.31:6815/2823036296","state":["exists","up"]},{"osd":4,"uuid":"8cb89153-8c3b-431f-b57c-8863fada76dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6809","nonce":4215186309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6811","nonce":4215186309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6815","nonce":4215186309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6813","nonce":4215186309}]},"public_addr":"172.21.15.40:6809/4215186309","cluster_addr":"172.21.15.40:6811/4215186309","heartbeat_back_addr":"172.21.15.40:6815/4215186309","heartbeat_front_addr":"172.21.15.40:6813/4215186309","state":["exists","up"]},{"osd":5,"uuid":"9409505a-d463-408c-a3cd-218380c04d68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6809","nonce":3402830452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6811","nonce":3402830452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6815","nonce":3402830452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6813","nonce":3402830452}]},"public_addr":"172.21.15.70:6809/3402830452","cluster_addr":"172.21.15.70:6811/3402830452","heartbeat_back_addr":"172.21.15.70:6815/3402830452","heartbeat_front_addr":"172.21.15.70:6813/3402830452","state":["exists","up"]},{"osd":6,"uuid":"f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6819","nonce":1193497058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6821","nonce":1193497058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6825","nonce":1193497058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6823","nonce":1193497058}]},"public_addr":"172.21.15.31:6819/1193497058","cluster_addr":"172.21.15.31:6821/1193497058","heartbeat_back_addr":"172.21.15.31:6825/1193497058","heartbeat_front_addr":"172.21.15.31:6823/1193497058","state":["exists","up"]},{"osd":7,"uuid":"db228a87-29a5-45d4-8fe6-a0a554262eb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6817","nonce":3825628119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6819","nonce":3825628119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6823","nonce":3825628119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6821","nonce":3825628119}]},"public_addr":"172.21.15.40:6817/3825628119","cluster_addr":"172.21.15.40:6819/3825628119","heartbeat_back_addr":"172.21.15.40:6823/3825628119","heartbeat_front_addr":"172.21.15.40:6821/3825628119","state":["exists","up"]},{"osd":8,"uuid":"59b9fde0-095d-4e0b-87da-ead37bd8ff1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6817","nonce":534880586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6819","nonce":534880586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6823","nonce":534880586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6821","nonce":534880586}]},"public_addr":"172.21.15.70:6817/534880586","cluster_addr":"172.21.15.70:6819/534880586","heartbeat_back_addr":"172.21.15.70:6823/534880586","heartbeat_front_addr":"172.21.15.70:6821/534880586","state":["exists","up"]},{"osd":9,"uuid":"77f47061-0c5f-4159-b276-e814935d6c1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6827","nonce":587047469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6829","nonce":587047469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6833","nonce":587047469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6831","nonce":587047469}]},"public_addr":"172.21.15.31:6827/587047469","cluster_addr":"172.21.15.31:6829/587047469","heartbeat_back_addr":"172.21.15.31:6833/587047469","heartbeat_front_addr":"172.21.15.31:6831/587047469","state":["exists","up"]},{"osd":10,"uuid":"8853dc43-7bd1-4c93-8063-f7982fd7da38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6825","nonce":512108711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6827","nonce":512108711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6831","nonce":512108711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6829","nonce":512108711}]},"public_addr":"172.21.15.40:6825/512108711","cluster_addr":"172.21.15.40:6827/512108711","heartbeat_back_addr":"172.21.15.40:6831/512108711","heartbeat_front_addr":"172.21.15.40:6829/512108711","state":["exists","up"]},{"osd":11,"uuid":"84b7c2de-c424-45a6-9583-60097e372a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6825","nonce":540305718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6827","nonce":540305718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6831","nonce":540305718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6829","nonce":540305718}]},"public_addr":"172.21.15.70:6825/540305718","cluster_addr":"172.21.15.70:6827/540305718","heartbeat_back_addr":"172.21.15.70:6831/540305718","heartbeat_front_addr":"172.21.15.70:6829/540305718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:28:51.477358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:17.405045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:41.626268+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:15.339568+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:42.676996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:07.516999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:42.412273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:10.365041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:35.699120+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:11.733639+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:39.505442+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:34:05.449301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451930554":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/3342588843":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6801/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/1626448389":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/1910059351":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/344822770":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/3811368512":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/1981437651":"2024-03-22T10:26:16.959165+0000","172.21.15.31:6801/1981437651":"2024-03-22T10:26:16.959165+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T10:38:13.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.225+0000 7f5b73200640 1 -- 172.21.15.31:0/181187631 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5b580a2f90 msgr2=0x7f5b580a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:13.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.225+0000 7f5b73200640 1 --2- 172.21.15.31:0/181187631 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5b580a2f90 0x7f5b580a5450 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5b6002c260 tx=0x7f5b60000bc0 comp rx=0 tx=0).stop 2024-03-21T10:38:13.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.225+0000 7f5b73200640 1 -- 172.21.15.31:0/181187631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5b741037c0 msgr2=0x7f5b74199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:13.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.225+0000 7f5b73200640 1 --2- 172.21.15.31:0/181187631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5b741037c0 0x7f5b74199a80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5b6400e9c0 tx=0x7f5b6400ee80 comp rx=0 tx=0).stop 2024-03-21T10:38:13.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.226+0000 7f5b73200640 1 -- 172.21.15.31:0/181187631 shutdown_connections 2024-03-21T10:38:13.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.226+0000 7f5b73200640 1 --2- 172.21.15.31:0/181187631 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5b580a2f90 0x7f5b580a5450 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:13.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.226+0000 7f5b73200640 1 --2- 172.21.15.31:0/181187631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b74105370 0x7f5b741a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:13.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.226+0000 7f5b73200640 1 --2- 172.21.15.31:0/181187631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b741049b0 0x7f5b74199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:13.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.226+0000 7f5b73200640 1 --2- 172.21.15.31:0/181187631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5b741037c0 0x7f5b74199a80 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:13.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.226+0000 7f5b73200640 1 -- 172.21.15.31:0/181187631 >> 172.21.15.31:0/181187631 conn(0x7f5b740fed50 msgr2=0x7f5b74107c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:13.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.226+0000 7f5b73200640 1 -- 172.21.15.31:0/181187631 shutdown_connections 2024-03-21T10:38:13.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.226+0000 7f5b73200640 1 -- 172.21.15.31:0/181187631 wait complete. 2024-03-21T10:38:13.251 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T10:29:44.553968+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T10:35:04.945449+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T10:35:05.970589+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T10:35:09.688711+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T10:38:13.251 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T10:38:13.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.498+0000 7f4ff8e00640 1 -- 172.21.15.31:0/744907201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ff41048f0 msgr2=0x7f4ff4104d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:13.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.498+0000 7f4ff8e00640 1 --2- 172.21.15.31:0/744907201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ff41048f0 0x7f4ff4104d70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0009900 tx=0x7f4fe002bc40 comp rx=0 tx=0).stop 2024-03-21T10:38:13.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.498+0000 7f4ff8e00640 1 -- 172.21.15.31:0/744907201 shutdown_connections 2024-03-21T10:38:13.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.498+0000 7f4ff8e00640 1 --2- 172.21.15.31:0/744907201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ff41052b0 0x7f4ff410bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:13.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.498+0000 7f4ff8e00640 1 --2- 172.21.15.31:0/744907201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ff41048f0 0x7f4ff4104d70 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:13.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.498+0000 7f4ff8e00640 1 --2- 172.21.15.31:0/744907201 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ff4103700 0x7f4ff4103b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:13.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.498+0000 7f4ff8e00640 1 -- 172.21.15.31:0/744907201 >> 172.21.15.31:0/744907201 conn(0x7f4ff40fec70 msgr2=0x7f4ff41010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:13.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.498+0000 7f4ff8e00640 1 -- 172.21.15.31:0/744907201 shutdown_connections 2024-03-21T10:38:13.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.498+0000 7f4ff8e00640 1 -- 172.21.15.31:0/744907201 wait complete. 2024-03-21T10:38:13.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.499+0000 7f4ff8e00640 1 Processor -- start 2024-03-21T10:38:13.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.500+0000 7f4ff8e00640 1 -- start start 2024-03-21T10:38:13.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.500+0000 7f4ff8e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ff4103700 0x7f4ff4199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:13.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.500+0000 7f4ff8e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ff41048f0 0x7f4ff4199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:13.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.500+0000 7f4ff8e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ff41052b0 0x7f4ff41a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:13.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.500+0000 7f4ff8e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff410f3d0 con 0x7f4ff4103700 2024-03-21T10:38:13.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.500+0000 7f4ff8e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff410f250 con 0x7f4ff41052b0 2024-03-21T10:38:13.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.500+0000 7f4ff8e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff410f550 con 0x7f4ff41048f0 2024-03-21T10:38:13.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.500+0000 7f4ff2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ff41048f0 0x7f4ff4199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:13.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.500+0000 7f4ff2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ff41048f0 0x7f4ff4199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:55296/0 (socket says 172.21.15.31:55296) 2024-03-21T10:38:13.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.500+0000 7f4ff2a00640 1 -- 172.21.15.31:0/3901027485 learned_addr learned my addr 172.21.15.31:0/3901027485 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:13.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.500+0000 7f4ff3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ff4103700 0x7f4ff4199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:13.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.501+0000 7f4ff3e00640 1 --2- 172.21.15.31:0/3901027485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ff41052b0 0x7f4ff41a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:13.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.501+0000 7f4ff2a00640 1 -- 172.21.15.31:0/3901027485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ff41052b0 msgr2=0x7f4ff41a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:13.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.501+0000 7f4ff2a00640 1 --2- 172.21.15.31:0/3901027485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ff41052b0 0x7f4ff41a0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:13.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.501+0000 7f4ff2a00640 1 -- 172.21.15.31:0/3901027485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ff4103700 msgr2=0x7f4ff4199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:13.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.501+0000 7f4ff2a00640 1 --2- 172.21.15.31:0/3901027485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ff4103700 0x7f4ff4199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:13.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.501+0000 7f4ff2a00640 1 -- 172.21.15.31:0/3901027485 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff41a1640 con 0x7f4ff41048f0 2024-03-21T10:38:13.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.501+0000 7f4ff2a00640 1 --2- 172.21.15.31:0/3901027485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ff41048f0 0x7f4ff4199f60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4fe002c8f0 tx=0x7f4fe0005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:13.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.501+0000 7f4febe00640 1 -- 172.21.15.31:0/3901027485 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe0041070 con 0x7f4ff41048f0 2024-03-21T10:38:13.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.502+0000 7f4ff8e00640 1 -- 172.21.15.31:0/3901027485 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff41a18d0 con 0x7f4ff41048f0 2024-03-21T10:38:13.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.502+0000 7f4febe00640 1 -- 172.21.15.31:0/3901027485 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4fe00044d0 con 0x7f4ff41048f0 2024-03-21T10:38:13.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.502+0000 7f4febe00640 1 -- 172.21.15.31:0/3901027485 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe0034440 con 0x7f4ff41048f0 2024-03-21T10:38:13.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.502+0000 7f4ff8e00640 1 -- 172.21.15.31:0/3901027485 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff41a1e10 con 0x7f4ff41048f0 2024-03-21T10:38:13.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.503+0000 7f4ff8e00640 1 -- 172.21.15.31:0/3901027485 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ff410aa20 con 0x7f4ff41048f0 2024-03-21T10:38:13.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.505+0000 7f4febe00640 1 -- 172.21.15.31:0/3901027485 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f4fe0044070 con 0x7f4ff41048f0 2024-03-21T10:38:13.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.508+0000 7f4febe00640 1 --2- 172.21.15.31:0/3901027485 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4fd80a2f20 0x7f4fd80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:13.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.508+0000 7f4febe00640 1 -- 172.21.15.31:0/3901027485 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f4fe00f16c0 con 0x7f4ff41048f0 2024-03-21T10:38:13.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.508+0000 7f4ff3400640 1 --2- 172.21.15.31:0/3901027485 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4fd80a2f20 0x7f4fd80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:13.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.509+0000 7f4ff3400640 1 --2- 172.21.15.31:0/3901027485 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4fd80a2f20 0x7f4fd80a53e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4104750 tx=0x7f4fe4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:13.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.512+0000 7f4febe00640 1 -- 172.21.15.31:0/3901027485 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4fe00bdae0 con 0x7f4ff41048f0 2024-03-21T10:38:13.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.717+0000 7f4ff8e00640 1 -- 172.21.15.31:0/3901027485 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4ff402d230 con 0x7f4ff41048f0 2024-03-21T10:38:13.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.727+0000 7f4febe00640 1 -- 172.21.15.31:0/3901027485 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4fe00c1b80 con 0x7f4ff41048f0 2024-03-21T10:38:13.725 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-03-21T10:38:13.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.730+0000 7f4ff8e00640 1 -- 172.21.15.31:0/3901027485 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4fd80a2f20 msgr2=0x7f4fd80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:13.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.730+0000 7f4ff8e00640 1 --2- 172.21.15.31:0/3901027485 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4fd80a2f20 0x7f4fd80a53e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4104750 tx=0x7f4fe4005f90 comp rx=0 tx=0).stop 2024-03-21T10:38:13.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.730+0000 7f4ff8e00640 1 -- 172.21.15.31:0/3901027485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ff41048f0 msgr2=0x7f4ff4199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:13.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.730+0000 7f4ff8e00640 1 --2- 172.21.15.31:0/3901027485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ff41048f0 0x7f4ff4199f60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4fe002c8f0 tx=0x7f4fe0005d50 comp rx=0 tx=0).stop 2024-03-21T10:38:13.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.730+0000 7f4ff8e00640 1 -- 172.21.15.31:0/3901027485 shutdown_connections 2024-03-21T10:38:13.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.730+0000 7f4ff8e00640 1 --2- 172.21.15.31:0/3901027485 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4fd80a2f20 0x7f4fd80a53e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:13.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.730+0000 7f4ff8e00640 1 --2- 172.21.15.31:0/3901027485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ff41052b0 0x7f4ff41a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:13.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.730+0000 7f4ff8e00640 1 --2- 172.21.15.31:0/3901027485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ff41048f0 0x7f4ff4199f60 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:13.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.730+0000 7f4ff8e00640 1 --2- 172.21.15.31:0/3901027485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ff4103700 0x7f4ff4199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:13.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.730+0000 7f4ff8e00640 1 -- 172.21.15.31:0/3901027485 >> 172.21.15.31:0/3901027485 conn(0x7f4ff40fec70 msgr2=0x7f4ff4107bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:13.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.730+0000 7f4ff8e00640 1 -- 172.21.15.31:0/3901027485 shutdown_connections 2024-03-21T10:38:13.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.730+0000 7f4ff8e00640 1 -- 172.21.15.31:0/3901027485 wait complete. 2024-03-21T10:38:13.747 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T10:38:13.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.973+0000 7f0d21800640 1 -- 172.21.15.31:0/1257596048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d1c1048a0 msgr2=0x7f0d1c106c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:13.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.973+0000 7f0d21800640 1 --2- 172.21.15.31:0/1257596048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d1c1048a0 0x7f0d1c106c90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0d04009900 tx=0x7f0d0402bc40 comp rx=0 tx=0).stop 2024-03-21T10:38:13.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.973+0000 7f0d21800640 1 -- 172.21.15.31:0/1257596048 shutdown_connections 2024-03-21T10:38:13.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.973+0000 7f0d21800640 1 --2- 172.21.15.31:0/1257596048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d1c1071d0 0x7f0d1c109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:13.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.973+0000 7f0d21800640 1 --2- 172.21.15.31:0/1257596048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d1c1048a0 0x7f0d1c106c90 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:13.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.973+0000 7f0d21800640 1 --2- 172.21.15.31:0/1257596048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d1c101f70 0x7f0d1c104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:13.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.973+0000 7f0d21800640 1 -- 172.21.15.31:0/1257596048 >> 172.21.15.31:0/1257596048 conn(0x7f0d1c0fb980 msgr2=0x7f0d1c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:13.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.973+0000 7f0d21800640 1 -- 172.21.15.31:0/1257596048 shutdown_connections 2024-03-21T10:38:13.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.973+0000 7f0d21800640 1 -- 172.21.15.31:0/1257596048 wait complete. 2024-03-21T10:38:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.974+0000 7f0d21800640 1 Processor -- start 2024-03-21T10:38:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.974+0000 7f0d21800640 1 -- start start 2024-03-21T10:38:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.974+0000 7f0d21800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d1c101f70 0x7f0d1c1961e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.974+0000 7f0d21800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d1c1048a0 0x7f0d1c196720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.974+0000 7f0d21800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d1c1071d0 0x7f0d1c192d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.975+0000 7f0d21800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c10d1d0 con 0x7f0d1c101f70 2024-03-21T10:38:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.975+0000 7f0d21800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c10d050 con 0x7f0d1c1071d0 2024-03-21T10:38:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.975+0000 7f0d21800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c10d350 con 0x7f0d1c1048a0 2024-03-21T10:38:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.975+0000 7f0d1be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d1c101f70 0x7f0d1c1961e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.975+0000 7f0d20e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d1c1071d0 0x7f0d1c192d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:13.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.975+0000 7f0d13e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d1c1048a0 0x7f0d1c196720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:13.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.975+0000 7f0d13e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d1c1048a0 0x7f0d1c196720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:55324/0 (socket says 172.21.15.31:55324) 2024-03-21T10:38:13.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.975+0000 7f0d20e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d1c1071d0 0x7f0d1c192d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51354/0 (socket says 172.21.15.31:51354) 2024-03-21T10:38:13.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.975+0000 7f0d13e00640 1 -- 172.21.15.31:0/1564617790 learned_addr learned my addr 172.21.15.31:0/1564617790 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:13.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.975+0000 7f0d1be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d1c101f70 0x7f0d1c1961e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34100/0 (socket says 172.21.15.31:34100) 2024-03-21T10:38:13.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.975+0000 7f0d13e00640 1 -- 172.21.15.31:0/1564617790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d1c1071d0 msgr2=0x7f0d1c192d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:13.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.975+0000 7f0d13e00640 1 --2- 172.21.15.31:0/1564617790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d1c1071d0 0x7f0d1c192d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:13.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.975+0000 7f0d13e00640 1 -- 172.21.15.31:0/1564617790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d1c101f70 msgr2=0x7f0d1c1961e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:13.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.975+0000 7f0d13e00640 1 --2- 172.21.15.31:0/1564617790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d1c101f70 0x7f0d1c1961e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:13.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.975+0000 7f0d13e00640 1 -- 172.21.15.31:0/1564617790 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d1c193610 con 0x7f0d1c1048a0 2024-03-21T10:38:13.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.976+0000 7f0d13e00640 1 --2- 172.21.15.31:0/1564617790 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d1c1048a0 0x7f0d1c196720 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0d0402efd0 tx=0x7f0d04005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:13.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.976+0000 7f0d19600640 1 -- 172.21.15.31:0/1564617790 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d04041070 con 0x7f0d1c1048a0 2024-03-21T10:38:13.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.976+0000 7f0d19600640 1 -- 172.21.15.31:0/1564617790 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d040044d0 con 0x7f0d1c1048a0 2024-03-21T10:38:13.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.976+0000 7f0d19600640 1 -- 172.21.15.31:0/1564617790 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d04034440 con 0x7f0d1c1048a0 2024-03-21T10:38:13.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.976+0000 7f0d21800640 1 -- 172.21.15.31:0/1564617790 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d1c1a5e50 con 0x7f0d1c1048a0 2024-03-21T10:38:13.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.976+0000 7f0d21800640 1 -- 172.21.15.31:0/1564617790 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d1c1a6200 con 0x7f0d1c1048a0 2024-03-21T10:38:13.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.977+0000 7f0d21800640 1 -- 172.21.15.31:0/1564617790 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d1c108d20 con 0x7f0d1c1048a0 2024-03-21T10:38:13.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.979+0000 7f0d19600640 1 -- 172.21.15.31:0/1564617790 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f0d04044070 con 0x7f0d1c1048a0 2024-03-21T10:38:13.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.980+0000 7f0d19600640 1 --2- 172.21.15.31:0/1564617790 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0d000a2fa0 0x7f0d000a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:13.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.980+0000 7f0d19600640 1 -- 172.21.15.31:0/1564617790 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f0d040f1240 con 0x7f0d1c1048a0 2024-03-21T10:38:13.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.981+0000 7f0d1be00640 1 --2- 172.21.15.31:0/1564617790 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0d000a2fa0 0x7f0d000a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:13.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.981+0000 7f0d1be00640 1 --2- 172.21.15.31:0/1564617790 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0d000a2fa0 0x7f0d000a5460 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c0095f0 tx=0x7f0d0c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:13.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:13.984+0000 7f0d19600640 1 -- 172.21.15.31:0/1564617790 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0d040bd660 con 0x7f0d1c1048a0 2024-03-21T10:38:14.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:13 smithi031 ceph-mon[31673]: pgmap v422: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:38:14.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:13 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-21T10:38:14.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:13 smithi031 ceph-mon[31673]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:38:14.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:13 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/181187631' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:38:14.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:13 smithi070 ceph-mon[36330]: pgmap v422: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:38:14.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:13 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-21T10:38:14.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:13 smithi070 ceph-mon[36330]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:38:14.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:13 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/181187631' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:38:14.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:13 smithi040 ceph-mon[36154]: pgmap v422: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-21T10:38:14.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:13 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-21T10:38:14.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:13 smithi040 ceph-mon[36154]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:38:14.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:13 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/181187631' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:38:14.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.188+0000 7f0d21800640 1 -- 172.21.15.31:0/1564617790 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0d1c0622a0 con 0x7f0d1c1048a0 2024-03-21T10:38:14.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.189+0000 7f0d19600640 1 -- 172.21.15.31:0/1564617790 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0d040c1700 con 0x7f0d1c1048a0 2024-03-21T10:38:14.187 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-03-21T10:38:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.192+0000 7f0d21800640 1 -- 172.21.15.31:0/1564617790 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0d000a2fa0 msgr2=0x7f0d000a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.192+0000 7f0d21800640 1 --2- 172.21.15.31:0/1564617790 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0d000a2fa0 0x7f0d000a5460 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c0095f0 tx=0x7f0d0c005f90 comp rx=0 tx=0).stop 2024-03-21T10:38:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.192+0000 7f0d21800640 1 -- 172.21.15.31:0/1564617790 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d1c1048a0 msgr2=0x7f0d1c196720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.192+0000 7f0d21800640 1 --2- 172.21.15.31:0/1564617790 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d1c1048a0 0x7f0d1c196720 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0d0402efd0 tx=0x7f0d04005d50 comp rx=0 tx=0).stop 2024-03-21T10:38:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.192+0000 7f0d21800640 1 -- 172.21.15.31:0/1564617790 shutdown_connections 2024-03-21T10:38:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.192+0000 7f0d21800640 1 --2- 172.21.15.31:0/1564617790 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0d000a2fa0 0x7f0d000a5460 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.192+0000 7f0d21800640 1 --2- 172.21.15.31:0/1564617790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d1c1071d0 0x7f0d1c192d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.192+0000 7f0d21800640 1 --2- 172.21.15.31:0/1564617790 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d1c1048a0 0x7f0d1c196720 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.192+0000 7f0d21800640 1 --2- 172.21.15.31:0/1564617790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d1c101f70 0x7f0d1c1961e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.192+0000 7f0d21800640 1 -- 172.21.15.31:0/1564617790 >> 172.21.15.31:0/1564617790 conn(0x7f0d1c0fb980 msgr2=0x7f0d1c101670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.192+0000 7f0d21800640 1 -- 172.21.15.31:0/1564617790 shutdown_connections 2024-03-21T10:38:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.192+0000 7f0d21800640 1 -- 172.21.15.31:0/1564617790 wait complete. 2024-03-21T10:38:14.210 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T10:38:14.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.433+0000 7f9326800640 1 -- 172.21.15.31:0/200836232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93201038e0 msgr2=0x7f9320103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:14.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.433+0000 7f9326800640 1 --2- 172.21.15.31:0/200836232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93201038e0 0x7f9320103d00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9310009900 tx=0x7f931002bc40 comp rx=0 tx=0).stop 2024-03-21T10:38:14.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.433+0000 7f9326800640 1 -- 172.21.15.31:0/200836232 shutdown_connections 2024-03-21T10:38:14.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.433+0000 7f9326800640 1 --2- 172.21.15.31:0/200836232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9320105490 0x7f932010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:14.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.433+0000 7f9326800640 1 --2- 172.21.15.31:0/200836232 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9320104ad0 0x7f9320104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:14.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.433+0000 7f9326800640 1 --2- 172.21.15.31:0/200836232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93201038e0 0x7f9320103d00 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:14.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.433+0000 7f9326800640 1 -- 172.21.15.31:0/200836232 >> 172.21.15.31:0/200836232 conn(0x7f93200fee50 msgr2=0x7f9320101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:14.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.433+0000 7f9326800640 1 -- 172.21.15.31:0/200836232 shutdown_connections 2024-03-21T10:38:14.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.433+0000 7f9326800640 1 -- 172.21.15.31:0/200836232 wait complete. 2024-03-21T10:38:14.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.434+0000 7f9326800640 1 Processor -- start 2024-03-21T10:38:14.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.435+0000 7f9326800640 1 -- start start 2024-03-21T10:38:14.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.435+0000 7f9326800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93201038e0 0x7f9320199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:14.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.435+0000 7f9326800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9320104ad0 0x7f932019a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:14.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.435+0000 7f9326800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9320105490 0x7f93201a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:14.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.435+0000 7f9326800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932010f580 con 0x7f9320105490 2024-03-21T10:38:14.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.435+0000 7f9326800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932010f400 con 0x7f93201038e0 2024-03-21T10:38:14.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.435+0000 7f9326800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932010f700 con 0x7f9320104ad0 2024-03-21T10:38:14.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.435+0000 7f931fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9320105490 0x7f93201a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:14.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.436+0000 7f931f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93201038e0 0x7f9320199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:14.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.436+0000 7f931fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9320105490 0x7f93201a1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34124/0 (socket says 172.21.15.31:34124) 2024-03-21T10:38:14.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.436+0000 7f931fe00640 1 -- 172.21.15.31:0/2170611686 learned_addr learned my addr 172.21.15.31:0/2170611686 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:14.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.436+0000 7f931ea00640 1 --2- 172.21.15.31:0/2170611686 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9320104ad0 0x7f932019a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:14.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.436+0000 7f931f400640 1 -- 172.21.15.31:0/2170611686 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9320104ad0 msgr2=0x7f932019a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:14.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.436+0000 7f931f400640 1 --2- 172.21.15.31:0/2170611686 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9320104ad0 0x7f932019a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:14.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.436+0000 7f931f400640 1 -- 172.21.15.31:0/2170611686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9320105490 msgr2=0x7f93201a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:14.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.436+0000 7f931f400640 1 --2- 172.21.15.31:0/2170611686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9320105490 0x7f93201a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:14.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.436+0000 7f931f400640 1 -- 172.21.15.31:0/2170611686 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93201a17e0 con 0x7f93201038e0 2024-03-21T10:38:14.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.436+0000 7f931ea00640 1 --2- 172.21.15.31:0/2170611686 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9320104ad0 0x7f932019a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:14.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.436+0000 7f931f400640 1 --2- 172.21.15.31:0/2170611686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93201038e0 0x7f9320199bc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9310009230 tx=0x7f9310005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:14.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.437+0000 7f9317e00640 1 -- 172.21.15.31:0/2170611686 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9310041070 con 0x7f93201038e0 2024-03-21T10:38:14.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.437+0000 7f9317e00640 1 -- 172.21.15.31:0/2170611686 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93100054e0 con 0x7f93201038e0 2024-03-21T10:38:14.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.437+0000 7f9317e00640 1 -- 172.21.15.31:0/2170611686 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9310034440 con 0x7f93201038e0 2024-03-21T10:38:14.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.437+0000 7f9326800640 1 -- 172.21.15.31:0/2170611686 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93201a1a70 con 0x7f93201038e0 2024-03-21T10:38:14.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.437+0000 7f9326800640 1 -- 172.21.15.31:0/2170611686 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93201a1f80 con 0x7f93201038e0 2024-03-21T10:38:14.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.438+0000 7f9326800640 1 -- 172.21.15.31:0/2170611686 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f932010ac00 con 0x7f93201038e0 2024-03-21T10:38:14.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.441+0000 7f9317e00640 1 -- 172.21.15.31:0/2170611686 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f9310044070 con 0x7f93201038e0 2024-03-21T10:38:14.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.442+0000 7f9317e00640 1 --2- 172.21.15.31:0/2170611686 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f93040a2fa0 0x7f93040a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:14.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.442+0000 7f9317e00640 1 -- 172.21.15.31:0/2170611686 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f93100f1190 con 0x7f93201038e0 2024-03-21T10:38:14.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.442+0000 7f931ea00640 1 --2- 172.21.15.31:0/2170611686 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f93040a2fa0 0x7f93040a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:14.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.443+0000 7f931ea00640 1 --2- 172.21.15.31:0/2170611686 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f93040a2fa0 0x7f93040a5460 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f932019b0d0 tx=0x7f930c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:14.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.446+0000 7f9317e00640 1 -- 172.21.15.31:0/2170611686 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f93100bd610 con 0x7f93201038e0 2024-03-21T10:38:14.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.649+0000 7f9326800640 1 -- 172.21.15.31:0/2170611686 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f93200622a0 con 0x7f93201038e0 2024-03-21T10:38:14.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.650+0000 7f9317e00640 1 -- 172.21.15.31:0/2170611686 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f93100c16b0 con 0x7f93201038e0 2024-03-21T10:38:14.648 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-03-21T10:38:14.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.653+0000 7f9326800640 1 -- 172.21.15.31:0/2170611686 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f93040a2fa0 msgr2=0x7f93040a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:14.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.653+0000 7f9326800640 1 --2- 172.21.15.31:0/2170611686 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f93040a2fa0 0x7f93040a5460 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f932019b0d0 tx=0x7f930c005f90 comp rx=0 tx=0).stop 2024-03-21T10:38:14.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.653+0000 7f9326800640 1 -- 172.21.15.31:0/2170611686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93201038e0 msgr2=0x7f9320199bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:14.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.653+0000 7f9326800640 1 --2- 172.21.15.31:0/2170611686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93201038e0 0x7f9320199bc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9310009230 tx=0x7f9310005d50 comp rx=0 tx=0).stop 2024-03-21T10:38:14.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.653+0000 7f9326800640 1 -- 172.21.15.31:0/2170611686 shutdown_connections 2024-03-21T10:38:14.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.653+0000 7f9326800640 1 --2- 172.21.15.31:0/2170611686 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f93040a2fa0 0x7f93040a5460 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:14.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.653+0000 7f9326800640 1 --2- 172.21.15.31:0/2170611686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9320105490 0x7f93201a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:14.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.653+0000 7f9326800640 1 --2- 172.21.15.31:0/2170611686 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9320104ad0 0x7f932019a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:14.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.653+0000 7f9326800640 1 --2- 172.21.15.31:0/2170611686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93201038e0 0x7f9320199bc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:14.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.653+0000 7f9326800640 1 -- 172.21.15.31:0/2170611686 >> 172.21.15.31:0/2170611686 conn(0x7f93200fee50 msgr2=0x7f9320107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:14.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.653+0000 7f9326800640 1 -- 172.21.15.31:0/2170611686 shutdown_connections 2024-03-21T10:38:14.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.653+0000 7f9326800640 1 -- 172.21.15.31:0/2170611686 wait complete. 2024-03-21T10:38:14.672 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T10:38:14.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.890+0000 7f4b8be00640 1 -- 172.21.15.31:0/1566774606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b8c1048f0 msgr2=0x7f4b8c104d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:14.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.890+0000 7f4b8be00640 1 --2- 172.21.15.31:0/1566774606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b8c1048f0 0x7f4b8c104d70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4b74009900 tx=0x7f4b7402bc30 comp rx=0 tx=0).stop 2024-03-21T10:38:14.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.890+0000 7f4b8be00640 1 -- 172.21.15.31:0/1566774606 shutdown_connections 2024-03-21T10:38:14.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.890+0000 7f4b8be00640 1 --2- 172.21.15.31:0/1566774606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b8c1052b0 0x7f4b8c10bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:14.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.890+0000 7f4b8be00640 1 --2- 172.21.15.31:0/1566774606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b8c1048f0 0x7f4b8c104d70 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:14.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.890+0000 7f4b8be00640 1 --2- 172.21.15.31:0/1566774606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b8c103700 0x7f4b8c103b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:14.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.890+0000 7f4b8be00640 1 -- 172.21.15.31:0/1566774606 >> 172.21.15.31:0/1566774606 conn(0x7f4b8c0fec70 msgr2=0x7f4b8c1010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:14.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.890+0000 7f4b8be00640 1 -- 172.21.15.31:0/1566774606 shutdown_connections 2024-03-21T10:38:14.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.890+0000 7f4b8be00640 1 -- 172.21.15.31:0/1566774606 wait complete. 2024-03-21T10:38:14.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.892+0000 7f4b8be00640 1 Processor -- start 2024-03-21T10:38:14.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.892+0000 7f4b8be00640 1 -- start start 2024-03-21T10:38:14.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.892+0000 7f4b8be00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b8c103700 0x7f4b8c199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:14.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.892+0000 7f4b8be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b8c1048f0 0x7f4b8c199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:14.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.892+0000 7f4b8be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b8c1052b0 0x7f4b8c1a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:14.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.892+0000 7f4b8be00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c10f3b0 con 0x7f4b8c1048f0 2024-03-21T10:38:14.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.892+0000 7f4b8be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c10f230 con 0x7f4b8c1052b0 2024-03-21T10:38:14.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.892+0000 7f4b8be00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c10f530 con 0x7f4b8c103700 2024-03-21T10:38:14.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.892+0000 7f4b8aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b8c103700 0x7f4b8c199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:14.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.893+0000 7f4b82000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b8c1048f0 0x7f4b8c199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:14.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.893+0000 7f4b8aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b8c103700 0x7f4b8c199a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:55358/0 (socket says 172.21.15.31:55358) 2024-03-21T10:38:14.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.893+0000 7f4b8aa00640 1 -- 172.21.15.31:0/3209739294 learned_addr learned my addr 172.21.15.31:0/3209739294 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:14.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.893+0000 7f4b8b400640 1 --2- 172.21.15.31:0/3209739294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b8c1052b0 0x7f4b8c1a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:14.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.893+0000 7f4b8aa00640 1 -- 172.21.15.31:0/3209739294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b8c1052b0 msgr2=0x7f4b8c1a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:14.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.893+0000 7f4b8aa00640 1 --2- 172.21.15.31:0/3209739294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b8c1052b0 0x7f4b8c1a0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:14.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.893+0000 7f4b8aa00640 1 -- 172.21.15.31:0/3209739294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b8c1048f0 msgr2=0x7f4b8c199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:14.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.893+0000 7f4b8aa00640 1 --2- 172.21.15.31:0/3209739294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b8c1048f0 0x7f4b8c199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:14.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.893+0000 7f4b8aa00640 1 -- 172.21.15.31:0/3209739294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b8c1a1640 con 0x7f4b8c103700 2024-03-21T10:38:14.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.893+0000 7f4b8aa00640 1 --2- 172.21.15.31:0/3209739294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b8c103700 0x7f4b8c199a20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00ebc0 tx=0x7f4b7c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:14.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.893+0000 7f4b83e00640 1 -- 172.21.15.31:0/3209739294 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c003b70 con 0x7f4b8c103700 2024-03-21T10:38:14.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.894+0000 7f4b83e00640 1 -- 172.21.15.31:0/3209739294 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b7c003d10 con 0x7f4b8c103700 2024-03-21T10:38:14.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.894+0000 7f4b8be00640 1 -- 172.21.15.31:0/3209739294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b8c1a1930 con 0x7f4b8c103700 2024-03-21T10:38:14.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.894+0000 7f4b83e00640 1 -- 172.21.15.31:0/3209739294 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c019980 con 0x7f4b8c103700 2024-03-21T10:38:14.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.894+0000 7f4b8be00640 1 -- 172.21.15.31:0/3209739294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b8c1a1ea0 con 0x7f4b8c103700 2024-03-21T10:38:14.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.895+0000 7f4b8be00640 1 -- 172.21.15.31:0/3209739294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b8c10aa20 con 0x7f4b8c103700 2024-03-21T10:38:14.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.896+0000 7f4b83e00640 1 -- 172.21.15.31:0/3209739294 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f4b7c019b20 con 0x7f4b8c103700 2024-03-21T10:38:14.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.898+0000 7f4b83e00640 1 --2- 172.21.15.31:0/3209739294 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4b700a2fa0 0x7f4b700a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:14.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.898+0000 7f4b83e00640 1 -- 172.21.15.31:0/3209739294 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f4b7c0cff30 con 0x7f4b8c103700 2024-03-21T10:38:14.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.898+0000 7f4b82000640 1 --2- 172.21.15.31:0/3209739294 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4b700a2fa0 0x7f4b700a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:14.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.899+0000 7f4b82000640 1 --2- 172.21.15.31:0/3209739294 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4b700a2fa0 0x7f4b700a5460 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4b7402c240 tx=0x7f4b7402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:14.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:14.902+0000 7f4b83e00640 1 -- 172.21.15.31:0/3209739294 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4b7c09c9a0 con 0x7f4b8c103700 2024-03-21T10:38:15.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:14 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3901027485' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:38:15.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:14 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1564617790' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:38:15.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:14 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2170611686' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:38:15.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:14 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3901027485' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:38:15.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:14 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1564617790' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:38:15.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:14 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2170611686' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:38:15.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.107+0000 7f4b8be00640 1 -- 172.21.15.31:0/3209739294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f4b8c02d070 con 0x7f4b8c103700 2024-03-21T10:38:15.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.108+0000 7f4b83e00640 1 -- 172.21.15.31:0/3209739294 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7f4b7c09c2a0 con 0x7f4b8c103700 2024-03-21T10:38:15.106 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-03-21T10:38:15.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.111+0000 7f4b8be00640 1 -- 172.21.15.31:0/3209739294 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4b700a2fa0 msgr2=0x7f4b700a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:15.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.111+0000 7f4b8be00640 1 --2- 172.21.15.31:0/3209739294 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4b700a2fa0 0x7f4b700a5460 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4b7402c240 tx=0x7f4b7402df90 comp rx=0 tx=0).stop 2024-03-21T10:38:15.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.111+0000 7f4b8be00640 1 -- 172.21.15.31:0/3209739294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b8c103700 msgr2=0x7f4b8c199a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:15.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.111+0000 7f4b8be00640 1 --2- 172.21.15.31:0/3209739294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b8c103700 0x7f4b8c199a20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00ebc0 tx=0x7f4b7c00c6a0 comp rx=0 tx=0).stop 2024-03-21T10:38:15.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.111+0000 7f4b8be00640 1 -- 172.21.15.31:0/3209739294 shutdown_connections 2024-03-21T10:38:15.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.111+0000 7f4b8be00640 1 --2- 172.21.15.31:0/3209739294 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f4b700a2fa0 0x7f4b700a5460 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:15.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.111+0000 7f4b8be00640 1 --2- 172.21.15.31:0/3209739294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b8c1052b0 0x7f4b8c1a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:15.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.111+0000 7f4b8be00640 1 --2- 172.21.15.31:0/3209739294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b8c1048f0 0x7f4b8c199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:15.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.111+0000 7f4b8be00640 1 --2- 172.21.15.31:0/3209739294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b8c103700 0x7f4b8c199a20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:15.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.111+0000 7f4b8be00640 1 -- 172.21.15.31:0/3209739294 >> 172.21.15.31:0/3209739294 conn(0x7f4b8c0fec70 msgr2=0x7f4b8c107bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:15.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.111+0000 7f4b8be00640 1 -- 172.21.15.31:0/3209739294 shutdown_connections 2024-03-21T10:38:15.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.111+0000 7f4b8be00640 1 -- 172.21.15.31:0/3209739294 wait complete. 2024-03-21T10:38:15.128 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T10:38:15.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:14 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3901027485' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:38:15.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:14 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1564617790' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:38:15.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:14 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2170611686' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:38:15.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.342+0000 7f1fde600640 1 -- 172.21.15.31:0/3019138431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fd8108ed0 msgr2=0x7f1fd8109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:15.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.342+0000 7f1fde600640 1 --2- 172.21.15.31:0/3019138431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fd8108ed0 0x7f1fd8109350 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009900 tx=0x7f1fc402bc30 comp rx=0 tx=0).stop 2024-03-21T10:38:15.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.342+0000 7f1fde600640 1 -- 172.21.15.31:0/3019138431 shutdown_connections 2024-03-21T10:38:15.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.342+0000 7f1fde600640 1 --2- 172.21.15.31:0/3019138431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1fd8109890 0x7f1fd8110120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:15.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.342+0000 7f1fde600640 1 --2- 172.21.15.31:0/3019138431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fd8108ed0 0x7f1fd8109350 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:15.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.342+0000 7f1fde600640 1 --2- 172.21.15.31:0/3019138431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd8107ce0 0x7f1fd8108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:15.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.342+0000 7f1fde600640 1 -- 172.21.15.31:0/3019138431 >> 172.21.15.31:0/3019138431 conn(0x7f1fd8076ca0 msgr2=0x7f1fd80790c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:15.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.342+0000 7f1fde600640 1 -- 172.21.15.31:0/3019138431 shutdown_connections 2024-03-21T10:38:15.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.343+0000 7f1fde600640 1 -- 172.21.15.31:0/3019138431 wait complete. 2024-03-21T10:38:15.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.344+0000 7f1fde600640 1 Processor -- start 2024-03-21T10:38:15.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.344+0000 7f1fde600640 1 -- start start 2024-03-21T10:38:15.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.344+0000 7f1fde600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1fd8107ce0 0x7f1fd819dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:15.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.344+0000 7f1fde600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fd8108ed0 0x7f1fd819e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:15.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.344+0000 7f1fde600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd8109890 0x7f1fd81a55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:15.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.344+0000 7f1fde600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd81139b0 con 0x7f1fd8109890 2024-03-21T10:38:15.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.344+0000 7f1fde600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd8113830 con 0x7f1fd8108ed0 2024-03-21T10:38:15.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.344+0000 7f1fde600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd8113b30 con 0x7f1fd8107ce0 2024-03-21T10:38:15.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.345+0000 7f1fd7400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1fd8107ce0 0x7f1fd819dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:15.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.345+0000 7f1fd7400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1fd8107ce0 0x7f1fd819dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:55386/0 (socket says 172.21.15.31:55386) 2024-03-21T10:38:15.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.345+0000 7f1fd6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fd8108ed0 0x7f1fd819e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:15.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.345+0000 7f1fd7400640 1 -- 172.21.15.31:0/1396893822 learned_addr learned my addr 172.21.15.31:0/1396893822 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:15.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.345+0000 7f1fd7e00640 1 --2- 172.21.15.31:0/1396893822 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd8109890 0x7f1fd81a55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:15.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.345+0000 7f1fd7400640 1 -- 172.21.15.31:0/1396893822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fd8108ed0 msgr2=0x7f1fd819e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:15.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.345+0000 7f1fd7400640 1 --2- 172.21.15.31:0/1396893822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fd8108ed0 0x7f1fd819e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:15.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.345+0000 7f1fd7400640 1 -- 172.21.15.31:0/1396893822 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd8109890 msgr2=0x7f1fd81a55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:15.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.345+0000 7f1fd7400640 1 --2- 172.21.15.31:0/1396893822 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd8109890 0x7f1fd81a55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:15.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.345+0000 7f1fd7400640 1 -- 172.21.15.31:0/1396893822 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd81a5c10 con 0x7f1fd8107ce0 2024-03-21T10:38:15.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.345+0000 7f1fd7e00640 1 --2- 172.21.15.31:0/1396893822 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd8109890 0x7f1fd81a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:15.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.345+0000 7f1fd7400640 1 --2- 172.21.15.31:0/1396893822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1fd8107ce0 0x7f1fd819dff0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1fc800d950 tx=0x7f1fc800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:15.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.346+0000 7f1fcfe00640 1 -- 172.21.15.31:0/1396893822 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc8017020 con 0x7f1fd8107ce0 2024-03-21T10:38:15.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.346+0000 7f1fcfe00640 1 -- 172.21.15.31:0/1396893822 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1fc8009c30 con 0x7f1fd8107ce0 2024-03-21T10:38:15.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.346+0000 7f1fde600640 1 -- 172.21.15.31:0/1396893822 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd81a5f00 con 0x7f1fd8107ce0 2024-03-21T10:38:15.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.346+0000 7f1fcfe00640 1 -- 172.21.15.31:0/1396893822 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc800f720 con 0x7f1fd8107ce0 2024-03-21T10:38:15.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.346+0000 7f1fde600640 1 -- 172.21.15.31:0/1396893822 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd81a6360 con 0x7f1fd8107ce0 2024-03-21T10:38:15.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.347+0000 7f1fde600640 1 -- 172.21.15.31:0/1396893822 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fd810efa0 con 0x7f1fd8107ce0 2024-03-21T10:38:15.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.349+0000 7f1fcfe00640 1 -- 172.21.15.31:0/1396893822 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f1fc801a020 con 0x7f1fd8107ce0 2024-03-21T10:38:15.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.350+0000 7f1fcfe00640 1 --2- 172.21.15.31:0/1396893822 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1fbc0a2f20 0x7f1fbc0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:15.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.350+0000 7f1fcfe00640 1 -- 172.21.15.31:0/1396893822 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f1fc80cf8b0 con 0x7f1fd8107ce0 2024-03-21T10:38:15.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.350+0000 7f1fd6a00640 1 --2- 172.21.15.31:0/1396893822 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1fbc0a2f20 0x7f1fbc0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:15.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.351+0000 7f1fd6a00640 1 --2- 172.21.15.31:0/1396893822 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1fbc0a2f20 0x7f1fbc0a53e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1fc402c240 tx=0x7f1fc402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:15.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.352+0000 7f1fcfe00640 1 -- 172.21.15.31:0/1396893822 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1fc8021020 con 0x7f1fd8107ce0 2024-03-21T10:38:15.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.559+0000 7f1fde600640 1 -- 172.21.15.31:0/1396893822 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fd802d320 con 0x7f1fd8107ce0 2024-03-21T10:38:15.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.561+0000 7f1fcfe00640 1 -- 172.21.15.31:0/1396893822 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24553 (secure 0 0 0) 0x7f1fc8009de0 con 0x7f1fd8107ce0 2024-03-21T10:38:15.560 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:38:15.560 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":93,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","created":"2024-03-21T10:25:41.971117+0000","modified":"2024-03-21T10:36:56.301695+0000","last_up_change":"2024-03-21T10:34:07.757196+0000","last_in_change":"2024-03-21T10:33:47.460406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T10:29:44.553968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T10:35:04.945449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T10:35:05.970589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T10:35:09.688711+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e0244d33-00e2-47d5-9f66-34efc138c462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6803","nonce":3089885125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6805","nonce":3089885125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6809","nonce":3089885125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6807","nonce":3089885125}]},"public_addr":"172.21.15.31:6803/3089885125","cluster_addr":"172.21.15.31:6805/3089885125","heartbeat_back_addr":"172.21.15.31:6809/3089885125","heartbeat_front_addr":"172.21.15.31:6807/3089885125","state":["exists","up"]},{"osd":1,"uuid":"76a63504-beab-4354-a6c3-eebc3b16ad03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6801","nonce":618840037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6803","nonce":618840037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6807","nonce":618840037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6805","nonce":618840037}]},"public_addr":"172.21.15.40:6801/618840037","cluster_addr":"172.21.15.40:6803/618840037","heartbeat_back_addr":"172.21.15.40:6807/618840037","heartbeat_front_addr":"172.21.15.40:6805/618840037","state":["exists","up"]},{"osd":2,"uuid":"fa191bd6-9a21-4ee2-8adf-8a63c9cfd821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6801","nonce":2524184117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6803","nonce":2524184117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6807","nonce":2524184117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6805","nonce":2524184117}]},"public_addr":"172.21.15.70:6801/2524184117","cluster_addr":"172.21.15.70:6803/2524184117","heartbeat_back_addr":"172.21.15.70:6807/2524184117","heartbeat_front_addr":"172.21.15.70:6805/2524184117","state":["exists","up"]},{"osd":3,"uuid":"edb44507-ffb0-42a4-827b-08461e1bf952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6811","nonce":2823036296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6813","nonce":2823036296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6817","nonce":2823036296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6815","nonce":2823036296}]},"public_addr":"172.21.15.31:6811/2823036296","cluster_addr":"172.21.15.31:6813/2823036296","heartbeat_back_addr":"172.21.15.31:6817/2823036296","heartbeat_front_addr":"172.21.15.31:6815/2823036296","state":["exists","up"]},{"osd":4,"uuid":"8cb89153-8c3b-431f-b57c-8863fada76dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6809","nonce":4215186309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6811","nonce":4215186309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6815","nonce":4215186309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6813","nonce":4215186309}]},"public_addr":"172.21.15.40:6809/4215186309","cluster_addr":"172.21.15.40:6811/4215186309","heartbeat_back_addr":"172.21.15.40:6815/4215186309","heartbeat_front_addr":"172.21.15.40:6813/4215186309","state":["exists","up"]},{"osd":5,"uuid":"9409505a-d463-408c-a3cd-218380c04d68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6809","nonce":3402830452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6811","nonce":3402830452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6815","nonce":3402830452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6813","nonce":3402830452}]},"public_addr":"172.21.15.70:6809/3402830452","cluster_addr":"172.21.15.70:6811/3402830452","heartbeat_back_addr":"172.21.15.70:6815/3402830452","heartbeat_front_addr":"172.21.15.70:6813/3402830452","state":["exists","up"]},{"osd":6,"uuid":"f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6819","nonce":1193497058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6821","nonce":1193497058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6825","nonce":1193497058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6823","nonce":1193497058}]},"public_addr":"172.21.15.31:6819/1193497058","cluster_addr":"172.21.15.31:6821/1193497058","heartbeat_back_addr":"172.21.15.31:6825/1193497058","heartbeat_front_addr":"172.21.15.31:6823/1193497058","state":["exists","up"]},{"osd":7,"uuid":"db228a87-29a5-45d4-8fe6-a0a554262eb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6817","nonce":3825628119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6819","nonce":3825628119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6823","nonce":3825628119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6821","nonce":3825628119}]},"public_addr":"172.21.15.40:6817/3825628119","cluster_addr":"172.21.15.40:6819/3825628119","heartbeat_back_addr":"172.21.15.40:6823/3825628119","heartbeat_front_addr":"172.21.15.40:6821/3825628119","state":["exists","up"]},{"osd":8,"uuid":"59b9fde0-095d-4e0b-87da-ead37bd8ff1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6817","nonce":534880586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6819","nonce":534880586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6823","nonce":534880586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6821","nonce":534880586}]},"public_addr":"172.21.15.70:6817/534880586","cluster_addr":"172.21.15.70:6819/534880586","heartbeat_back_addr":"172.21.15.70:6823/534880586","heartbeat_front_addr":"172.21.15.70:6821/534880586","state":["exists","up"]},{"osd":9,"uuid":"77f47061-0c5f-4159-b276-e814935d6c1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6827","nonce":587047469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6829","nonce":587047469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6833","nonce":587047469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6831","nonce":587047469}]},"public_addr":"172.21.15.31:6827/587047469","cluster_addr":"172.21.15.31:6829/587047469","heartbeat_back_addr":"172.21.15.31:6833/587047469","heartbeat_front_addr":"172.21.15.31:6831/587047469","state":["exists","up"]},{"osd":10,"uuid":"8853dc43-7bd1-4c93-8063-f7982fd7da38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6825","nonce":512108711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6827","nonce":512108711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6831","nonce":512108711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6829","nonce":512108711}]},"public_addr":"172.21.15.40:6825/512108711","cluster_addr":"172.21.15.40:6827/512108711","heartbeat_back_addr":"172.21.15.40:6831/512108711","heartbeat_front_addr":"172.21.15.40:6829/512108711","state":["exists","up"]},{"osd":11,"uuid":"84b7c2de-c424-45a6-9583-60097e372a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6825","nonce":540305718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6827","nonce":540305718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6831","nonce":540305718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6829","nonce":540305718}]},"public_addr":"172.21.15.70:6825/540305718","cluster_addr":"172.21.15.70:6827/540305718","heartbeat_back_addr":"172.21.15.70:6831/540305718","heartbeat_front_addr":"172.21.15.70:6829/540305718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:28:51.477358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:17.405045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:41.626268+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:15.339568+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:42.676996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:07.516999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:42.412273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:10.365041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:35.699120+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:11.733639+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:39.505442+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:34:05.449301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451930554":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/3342588843":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6801/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/1626448389":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/1910059351":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/344822770":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/3811368512":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/1981437651":"2024-03-22T10:26:16.959165+0000","172.21.15.31:6801/1981437651":"2024-03-22T10:26:16.959165+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T10:38:15.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.564+0000 7f1fde600640 1 -- 172.21.15.31:0/1396893822 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1fbc0a2f20 msgr2=0x7f1fbc0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:15.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.564+0000 7f1fde600640 1 --2- 172.21.15.31:0/1396893822 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1fbc0a2f20 0x7f1fbc0a53e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1fc402c240 tx=0x7f1fc402df90 comp rx=0 tx=0).stop 2024-03-21T10:38:15.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.564+0000 7f1fde600640 1 -- 172.21.15.31:0/1396893822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1fd8107ce0 msgr2=0x7f1fd819dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:15.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.564+0000 7f1fde600640 1 --2- 172.21.15.31:0/1396893822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1fd8107ce0 0x7f1fd819dff0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1fc800d950 tx=0x7f1fc800de10 comp rx=0 tx=0).stop 2024-03-21T10:38:15.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.565+0000 7f1fde600640 1 -- 172.21.15.31:0/1396893822 shutdown_connections 2024-03-21T10:38:15.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.565+0000 7f1fde600640 1 --2- 172.21.15.31:0/1396893822 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f1fbc0a2f20 0x7f1fbc0a53e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:15.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.565+0000 7f1fde600640 1 --2- 172.21.15.31:0/1396893822 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fd8109890 0x7f1fd81a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:15.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.565+0000 7f1fde600640 1 --2- 172.21.15.31:0/1396893822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fd8108ed0 0x7f1fd819e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:15.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.565+0000 7f1fde600640 1 --2- 172.21.15.31:0/1396893822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1fd8107ce0 0x7f1fd819dff0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:15.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.565+0000 7f1fde600640 1 -- 172.21.15.31:0/1396893822 >> 172.21.15.31:0/1396893822 conn(0x7f1fd8076ca0 msgr2=0x7f1fd810c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:15.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.565+0000 7f1fde600640 1 -- 172.21.15.31:0/1396893822 shutdown_connections 2024-03-21T10:38:15.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.565+0000 7f1fde600640 1 -- 172.21.15.31:0/1396893822 wait complete. 2024-03-21T10:38:15.582 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T10:29:44.553968+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T10:35:04.945449+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T10:35:05.970589+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T10:35:09.688711+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T10:38:15.582 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T10:38:15.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.821+0000 7fa5c3000640 1 -- 172.21.15.31:0/2708408624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5bc103700 msgr2=0x7fa5bc103b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:15.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.821+0000 7fa5c3000640 1 --2- 172.21.15.31:0/2708408624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5bc103700 0x7fa5bc103b20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0009900 tx=0x7fa5b002bc50 comp rx=0 tx=0).stop 2024-03-21T10:38:15.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.822+0000 7fa5c3000640 1 -- 172.21.15.31:0/2708408624 shutdown_connections 2024-03-21T10:38:15.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.822+0000 7fa5c3000640 1 --2- 172.21.15.31:0/2708408624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5bc1052b0 0x7fa5bc10bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:15.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.822+0000 7fa5c3000640 1 --2- 172.21.15.31:0/2708408624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5bc1048f0 0x7fa5bc104d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:15.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.822+0000 7fa5c3000640 1 --2- 172.21.15.31:0/2708408624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5bc103700 0x7fa5bc103b20 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:15.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.822+0000 7fa5c3000640 1 -- 172.21.15.31:0/2708408624 >> 172.21.15.31:0/2708408624 conn(0x7fa5bc0fec70 msgr2=0x7fa5bc1010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:15.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.822+0000 7fa5c3000640 1 -- 172.21.15.31:0/2708408624 shutdown_connections 2024-03-21T10:38:15.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.822+0000 7fa5c3000640 1 -- 172.21.15.31:0/2708408624 wait complete. 2024-03-21T10:38:15.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.823+0000 7fa5c3000640 1 Processor -- start 2024-03-21T10:38:15.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.823+0000 7fa5c3000640 1 -- start start 2024-03-21T10:38:15.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.823+0000 7fa5c3000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5bc103700 0x7fa5bc199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:15.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.823+0000 7fa5c3000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5bc1048f0 0x7fa5bc199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.823+0000 7fa5c3000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5bc1052b0 0x7fa5bc1a1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.823+0000 7fa5c3000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc10f3c0 con 0x7fa5bc1052b0 2024-03-21T10:38:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.823+0000 7fa5c3000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc10f240 con 0x7fa5bc103700 2024-03-21T10:38:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.823+0000 7fa5c3000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc10f540 con 0x7fa5bc1048f0 2024-03-21T10:38:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.823+0000 7fa5c1c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5bc103700 0x7fa5bc199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.823+0000 7fa5c1200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5bc1048f0 0x7fa5bc199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.823+0000 7fa5c1c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5bc103700 0x7fa5bc199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51418/0 (socket says 172.21.15.31:51418) 2024-03-21T10:38:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.823+0000 7fa5c1c00640 1 -- 172.21.15.31:0/847938423 learned_addr learned my addr 172.21.15.31:0/847938423 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.823+0000 7fa5c1200640 1 --2- 172.21.15.31:0/847938423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5bc1048f0 0x7fa5bc199fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:55400/0 (socket says 172.21.15.31:55400) 2024-03-21T10:38:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.824+0000 7fa5c2600640 1 --2- 172.21.15.31:0/847938423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5bc1052b0 0x7fa5bc1a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.824+0000 7fa5c1200640 1 -- 172.21.15.31:0/847938423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5bc103700 msgr2=0x7fa5bc199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.824+0000 7fa5c1200640 1 --2- 172.21.15.31:0/847938423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5bc103700 0x7fa5bc199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.824+0000 7fa5c1200640 1 -- 172.21.15.31:0/847938423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5bc1052b0 msgr2=0x7fa5bc1a1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.824+0000 7fa5c1200640 1 --2- 172.21.15.31:0/847938423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5bc1052b0 0x7fa5bc1a1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:15.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.824+0000 7fa5c1200640 1 -- 172.21.15.31:0/847938423 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5bc1a16b0 con 0x7fa5bc1048f0 2024-03-21T10:38:15.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.824+0000 7fa5c1200640 1 --2- 172.21.15.31:0/847938423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5bc1048f0 0x7fa5bc199fd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa5a800e9f0 tx=0x7fa5a800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:15.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.824+0000 7fa5ba000640 1 -- 172.21.15.31:0/847938423 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a800ccc0 con 0x7fa5bc1048f0 2024-03-21T10:38:15.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.824+0000 7fa5ba000640 1 -- 172.21.15.31:0/847938423 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5a8005020 con 0x7fa5bc1048f0 2024-03-21T10:38:15.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.824+0000 7fa5c3000640 1 -- 172.21.15.31:0/847938423 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5bc1a19a0 con 0x7fa5bc1048f0 2024-03-21T10:38:15.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.824+0000 7fa5ba000640 1 -- 172.21.15.31:0/847938423 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a80049e0 con 0x7fa5bc1048f0 2024-03-21T10:38:15.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.825+0000 7fa5c3000640 1 -- 172.21.15.31:0/847938423 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5bc1a1ea0 con 0x7fa5bc1048f0 2024-03-21T10:38:15.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.826+0000 7fa5c3000640 1 -- 172.21.15.31:0/847938423 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5bc10aa70 con 0x7fa5bc1048f0 2024-03-21T10:38:15.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.830+0000 7fa5ba000640 1 -- 172.21.15.31:0/847938423 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fa5a8005560 con 0x7fa5bc1048f0 2024-03-21T10:38:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.832+0000 7fa5ba000640 1 --2- 172.21.15.31:0/847938423 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa5a40a2f20 0x7fa5a40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.832+0000 7fa5ba000640 1 -- 172.21.15.31:0/847938423 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fa5a80cf7c0 con 0x7fa5bc1048f0 2024-03-21T10:38:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.832+0000 7fa5c1c00640 1 --2- 172.21.15.31:0/847938423 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa5a40a2f20 0x7fa5a40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.833+0000 7fa5c1c00640 1 --2- 172.21.15.31:0/847938423 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa5a40a2f20 0x7fa5a40a53e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa5b002c900 tx=0x7fa5b0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:15.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:15.835+0000 7fa5ba000640 1 -- 172.21.15.31:0/847938423 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa5a80051d0 con 0x7fa5bc1048f0 2024-03-21T10:38:16.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.039+0000 7fa5c3000640 1 -- 172.21.15.31:0/847938423 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa5bc0622a0 con 0x7fa5bc1048f0 2024-03-21T10:38:16.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.040+0000 7fa5ba000640 1 -- 172.21.15.31:0/847938423 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa5a809c230 con 0x7fa5bc1048f0 2024-03-21T10:38:16.037 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-03-21T10:38:16.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.043+0000 7fa5c3000640 1 -- 172.21.15.31:0/847938423 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa5a40a2f20 msgr2=0x7fa5a40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:16.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.043+0000 7fa5c3000640 1 --2- 172.21.15.31:0/847938423 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa5a40a2f20 0x7fa5a40a53e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa5b002c900 tx=0x7fa5b0000bc0 comp rx=0 tx=0).stop 2024-03-21T10:38:16.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.043+0000 7fa5c3000640 1 -- 172.21.15.31:0/847938423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5bc1048f0 msgr2=0x7fa5bc199fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:16.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.043+0000 7fa5c3000640 1 --2- 172.21.15.31:0/847938423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5bc1048f0 0x7fa5bc199fd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa5a800e9f0 tx=0x7fa5a800eeb0 comp rx=0 tx=0).stop 2024-03-21T10:38:16.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.043+0000 7fa5c3000640 1 -- 172.21.15.31:0/847938423 shutdown_connections 2024-03-21T10:38:16.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.043+0000 7fa5c3000640 1 --2- 172.21.15.31:0/847938423 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fa5a40a2f20 0x7fa5a40a53e0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:16.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.043+0000 7fa5c3000640 1 --2- 172.21.15.31:0/847938423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5bc1052b0 0x7fa5bc1a1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:16.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.043+0000 7fa5c3000640 1 --2- 172.21.15.31:0/847938423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5bc1048f0 0x7fa5bc199fd0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:16.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.043+0000 7fa5c3000640 1 --2- 172.21.15.31:0/847938423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5bc103700 0x7fa5bc199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:16.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.043+0000 7fa5c3000640 1 -- 172.21.15.31:0/847938423 >> 172.21.15.31:0/847938423 conn(0x7fa5bc0fec70 msgr2=0x7fa5bc107c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:16.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.043+0000 7fa5c3000640 1 -- 172.21.15.31:0/847938423 shutdown_connections 2024-03-21T10:38:16.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.043+0000 7fa5c3000640 1 -- 172.21.15.31:0/847938423 wait complete. 2024-03-21T10:38:16.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:15 smithi031 ceph-mon[31673]: pgmap v423: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-21T10:38:16.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:15 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3209739294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:38:16.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:15 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1396893822' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:38:16.063 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T10:38:16.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:15 smithi070 ceph-mon[36330]: pgmap v423: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-21T10:38:16.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:15 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3209739294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:38:16.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:15 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1396893822' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:38:16.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:15 smithi040 ceph-mon[36154]: pgmap v423: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-21T10:38:16.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:15 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3209739294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:38:16.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:15 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1396893822' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:38:16.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.284+0000 7fcc82200640 1 -- 172.21.15.31:0/1839469778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c1014e0 msgr2=0x7fcc7c101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:16.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.284+0000 7fcc82200640 1 --2- 172.21.15.31:0/1839469778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c1014e0 0x7fcc7c101900 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fcc70009900 tx=0x7fcc7002bc40 comp rx=0 tx=0).stop 2024-03-21T10:38:16.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.284+0000 7fcc82200640 1 -- 172.21.15.31:0/1839469778 shutdown_connections 2024-03-21T10:38:16.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.284+0000 7fcc82200640 1 --2- 172.21.15.31:0/1839469778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc7c103090 0x7fcc7c109920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:16.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.284+0000 7fcc82200640 1 --2- 172.21.15.31:0/1839469778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc7c1026d0 0x7fcc7c102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:16.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.284+0000 7fcc82200640 1 --2- 172.21.15.31:0/1839469778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c1014e0 0x7fcc7c101900 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:16.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.284+0000 7fcc82200640 1 -- 172.21.15.31:0/1839469778 >> 172.21.15.31:0/1839469778 conn(0x7fcc7c0fca50 msgr2=0x7fcc7c0fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:16.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.284+0000 7fcc82200640 1 -- 172.21.15.31:0/1839469778 shutdown_connections 2024-03-21T10:38:16.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.284+0000 7fcc82200640 1 -- 172.21.15.31:0/1839469778 wait complete. 2024-03-21T10:38:16.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.285+0000 7fcc82200640 1 Processor -- start 2024-03-21T10:38:16.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.286+0000 7fcc82200640 1 -- start start 2024-03-21T10:38:16.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.286+0000 7fcc82200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc7c1014e0 0x7fcc7c199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:16.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.286+0000 7fcc82200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc7c1026d0 0x7fcc7c199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:16.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.286+0000 7fcc82200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c103090 0x7fcc7c1a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:16.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.286+0000 7fcc82200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c10d1c0 con 0x7fcc7c1026d0 2024-03-21T10:38:16.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.286+0000 7fcc82200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c10d040 con 0x7fcc7c103090 2024-03-21T10:38:16.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.286+0000 7fcc82200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c10d340 con 0x7fcc7c1014e0 2024-03-21T10:38:16.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.286+0000 7fcc80e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc7c1014e0 0x7fcc7c199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:16.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.286+0000 7fcc80e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc7c1014e0 0x7fcc7c199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:55432/0 (socket says 172.21.15.31:55432) 2024-03-21T10:38:16.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.286+0000 7fcc80e00640 1 -- 172.21.15.31:0/968186684 learned_addr learned my addr 172.21.15.31:0/968186684 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:16.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.286+0000 7fcc7be00640 1 --2- 172.21.15.31:0/968186684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc7c1026d0 0x7fcc7c199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:16.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.286+0000 7fcc81800640 1 --2- 172.21.15.31:0/968186684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c103090 0x7fcc7c1a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:16.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.287+0000 7fcc80e00640 1 -- 172.21.15.31:0/968186684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c103090 msgr2=0x7fcc7c1a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:16.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.287+0000 7fcc80e00640 1 --2- 172.21.15.31:0/968186684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c103090 0x7fcc7c1a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:16.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.287+0000 7fcc80e00640 1 -- 172.21.15.31:0/968186684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc7c1026d0 msgr2=0x7fcc7c199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:16.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.287+0000 7fcc80e00640 1 --2- 172.21.15.31:0/968186684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc7c1026d0 0x7fcc7c199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:16.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.287+0000 7fcc80e00640 1 -- 172.21.15.31:0/968186684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7c1a16c0 con 0x7fcc7c1014e0 2024-03-21T10:38:16.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.287+0000 7fcc80e00640 1 --2- 172.21.15.31:0/968186684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc7c1014e0 0x7fcc7c199aa0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fcc7002c8f0 tx=0x7fcc70005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:16.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.287+0000 7fcc79600640 1 -- 172.21.15.31:0/968186684 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc70041070 con 0x7fcc7c1014e0 2024-03-21T10:38:16.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.287+0000 7fcc79600640 1 -- 172.21.15.31:0/968186684 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc700044d0 con 0x7fcc7c1014e0 2024-03-21T10:38:16.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.287+0000 7fcc79600640 1 -- 172.21.15.31:0/968186684 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc70034440 con 0x7fcc7c1014e0 2024-03-21T10:38:16.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.287+0000 7fcc82200640 1 -- 172.21.15.31:0/968186684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc7c1a1950 con 0x7fcc7c1014e0 2024-03-21T10:38:16.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.288+0000 7fcc82200640 1 -- 172.21.15.31:0/968186684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc7c1a1df0 con 0x7fcc7c1014e0 2024-03-21T10:38:16.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.289+0000 7fcc82200640 1 -- 172.21.15.31:0/968186684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc7c1a1ff0 con 0x7fcc7c1014e0 2024-03-21T10:38:16.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.290+0000 7fcc79600640 1 -- 172.21.15.31:0/968186684 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fcc70044070 con 0x7fcc7c1014e0 2024-03-21T10:38:16.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.292+0000 7fcc79600640 1 --2- 172.21.15.31:0/968186684 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcc640a2fa0 0x7fcc640a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:16.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.292+0000 7fcc79600640 1 -- 172.21.15.31:0/968186684 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fcc700f12e0 con 0x7fcc7c1014e0 2024-03-21T10:38:16.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.292+0000 7fcc7be00640 1 --2- 172.21.15.31:0/968186684 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcc640a2fa0 0x7fcc640a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:16.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.293+0000 7fcc7be00640 1 --2- 172.21.15.31:0/968186684 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcc640a2fa0 0x7fcc640a5460 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c19afb0 tx=0x7fcc6800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:16.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.296+0000 7fcc79600640 1 -- 172.21.15.31:0/968186684 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcc700bd700 con 0x7fcc7c1014e0 2024-03-21T10:38:16.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.499+0000 7fcc82200640 1 -- 172.21.15.31:0/968186684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcc7c1a21c0 con 0x7fcc7c1014e0 2024-03-21T10:38:16.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.500+0000 7fcc79600640 1 -- 172.21.15.31:0/968186684 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcc700c17a0 con 0x7fcc7c1014e0 2024-03-21T10:38:16.498 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-03-21T10:38:16.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.503+0000 7fcc82200640 1 -- 172.21.15.31:0/968186684 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcc640a2fa0 msgr2=0x7fcc640a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:16.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.503+0000 7fcc82200640 1 --2- 172.21.15.31:0/968186684 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcc640a2fa0 0x7fcc640a5460 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c19afb0 tx=0x7fcc6800b410 comp rx=0 tx=0).stop 2024-03-21T10:38:16.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.503+0000 7fcc82200640 1 -- 172.21.15.31:0/968186684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc7c1014e0 msgr2=0x7fcc7c199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:16.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.503+0000 7fcc82200640 1 --2- 172.21.15.31:0/968186684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc7c1014e0 0x7fcc7c199aa0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fcc7002c8f0 tx=0x7fcc70005d50 comp rx=0 tx=0).stop 2024-03-21T10:38:16.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.503+0000 7fcc82200640 1 -- 172.21.15.31:0/968186684 shutdown_connections 2024-03-21T10:38:16.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.503+0000 7fcc82200640 1 --2- 172.21.15.31:0/968186684 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcc640a2fa0 0x7fcc640a5460 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:16.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.503+0000 7fcc82200640 1 --2- 172.21.15.31:0/968186684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c103090 0x7fcc7c1a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:16.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.503+0000 7fcc82200640 1 --2- 172.21.15.31:0/968186684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc7c1026d0 0x7fcc7c199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:16.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.503+0000 7fcc82200640 1 --2- 172.21.15.31:0/968186684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc7c1014e0 0x7fcc7c199aa0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:16.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.503+0000 7fcc82200640 1 -- 172.21.15.31:0/968186684 >> 172.21.15.31:0/968186684 conn(0x7fcc7c0fca50 msgr2=0x7fcc7c0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:16.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.503+0000 7fcc82200640 1 -- 172.21.15.31:0/968186684 shutdown_connections 2024-03-21T10:38:16.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.503+0000 7fcc82200640 1 -- 172.21.15.31:0/968186684 wait complete. 2024-03-21T10:38:16.523 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T10:38:16.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.744+0000 7fd1bea00640 1 -- 172.21.15.31:0/3519660634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1b8074100 msgr2=0x7fd1b8074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:16.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.744+0000 7fd1bea00640 1 --2- 172.21.15.31:0/3519660634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1b8074100 0x7fd1b8074580 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4009900 tx=0x7fd1a402bc40 comp rx=0 tx=0).stop 2024-03-21T10:38:16.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.744+0000 7fd1bea00640 1 -- 172.21.15.31:0/3519660634 shutdown_connections 2024-03-21T10:38:16.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.744+0000 7fd1bea00640 1 --2- 172.21.15.31:0/3519660634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1b81093c0 0x7fd1b810b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:16.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.744+0000 7fd1bea00640 1 --2- 172.21.15.31:0/3519660634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1b8074100 0x7fd1b8074580 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:16.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.744+0000 7fd1bea00640 1 --2- 172.21.15.31:0/3519660634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1b80756e0 0x7fd1b8073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:16.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.744+0000 7fd1bea00640 1 -- 172.21.15.31:0/3519660634 >> 172.21.15.31:0/3519660634 conn(0x7fd1b80fcab0 msgr2=0x7fd1b80feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:16.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.744+0000 7fd1bea00640 1 -- 172.21.15.31:0/3519660634 shutdown_connections 2024-03-21T10:38:16.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.744+0000 7fd1bea00640 1 -- 172.21.15.31:0/3519660634 wait complete. 2024-03-21T10:38:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.745+0000 7fd1bea00640 1 Processor -- start 2024-03-21T10:38:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.745+0000 7fd1bea00640 1 -- start start 2024-03-21T10:38:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.746+0000 7fd1bea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1b8074100 0x7fd1b8199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.746+0000 7fd1bea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1b80756e0 0x7fd1b8199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.746+0000 7fd1bea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1b81093c0 0x7fd1b81a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.746+0000 7fd1bea00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b810f3b0 con 0x7fd1b80756e0 2024-03-21T10:38:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.746+0000 7fd1bea00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b810f230 con 0x7fd1b8074100 2024-03-21T10:38:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.746+0000 7fd1bea00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b810f530 con 0x7fd1b81093c0 2024-03-21T10:38:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.746+0000 7fd1bcc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1b80756e0 0x7fd1b8199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.746+0000 7fd1bcc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1b80756e0 0x7fd1b8199f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34214/0 (socket says 172.21.15.31:34214) 2024-03-21T10:38:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.746+0000 7fd1bcc00640 1 -- 172.21.15.31:0/651720607 learned_addr learned my addr 172.21.15.31:0/651720607 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:16.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.746+0000 7fd1bd600640 1 --2- 172.21.15.31:0/651720607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1b8074100 0x7fd1b8199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.747+0000 7fd1bcc00640 1 -- 172.21.15.31:0/651720607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1b81093c0 msgr2=0x7fd1b81a0f90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-21T10:38:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.747+0000 7fd1bcc00640 1 --2- 172.21.15.31:0/651720607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1b81093c0 0x7fd1b81a0f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.747+0000 7fd1bcc00640 1 -- 172.21.15.31:0/651720607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1b8074100 msgr2=0x7fd1b8199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.747+0000 7fd1bcc00640 1 --2- 172.21.15.31:0/651720607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1b8074100 0x7fd1b8199a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.747+0000 7fd1bcc00640 1 -- 172.21.15.31:0/651720607 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b81a1600 con 0x7fd1b80756e0 2024-03-21T10:38:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.747+0000 7fd1bd600640 1 --2- 172.21.15.31:0/651720607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1b8074100 0x7fd1b8199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.747+0000 7fd1bcc00640 1 --2- 172.21.15.31:0/651720607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1b80756e0 0x7fd1b8199f90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd1a402c8f0 tx=0x7fd1a4005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.747+0000 7fd1b6000640 1 -- 172.21.15.31:0/651720607 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4041070 con 0x7fd1b80756e0 2024-03-21T10:38:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.747+0000 7fd1b6000640 1 -- 172.21.15.31:0/651720607 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd1a40044d0 con 0x7fd1b80756e0 2024-03-21T10:38:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.747+0000 7fd1bea00640 1 -- 172.21.15.31:0/651720607 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1b81a1890 con 0x7fd1b80756e0 2024-03-21T10:38:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.748+0000 7fd1b6000640 1 -- 172.21.15.31:0/651720607 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4034440 con 0x7fd1b80756e0 2024-03-21T10:38:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.748+0000 7fd1bea00640 1 -- 172.21.15.31:0/651720607 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1b81a1d80 con 0x7fd1b80756e0 2024-03-21T10:38:16.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.749+0000 7fd1bea00640 1 -- 172.21.15.31:0/651720607 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b81080f0 con 0x7fd1b80756e0 2024-03-21T10:38:16.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.752+0000 7fd1b6000640 1 -- 172.21.15.31:0/651720607 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fd1a4044070 con 0x7fd1b80756e0 2024-03-21T10:38:16.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.754+0000 7fd1b6000640 1 --2- 172.21.15.31:0/651720607 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd1a00a2f20 0x7fd1a00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:16.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.754+0000 7fd1b6000640 1 -- 172.21.15.31:0/651720607 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fd1a40f13c0 con 0x7fd1b80756e0 2024-03-21T10:38:16.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.754+0000 7fd1bd600640 1 --2- 172.21.15.31:0/651720607 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd1a00a2f20 0x7fd1a00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:16.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.755+0000 7fd1bd600640 1 --2- 172.21.15.31:0/651720607 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd1a00a2f20 0x7fd1a00a53e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac009fc0 tx=0x7fd1ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:16.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.758+0000 7fd1b6000640 1 -- 172.21.15.31:0/651720607 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd1a40bde30 con 0x7fd1b80756e0 2024-03-21T10:38:16.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:16 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/847938423' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:38:16.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:16 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/968186684' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:38:16.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.982+0000 7fd1bea00640 1 -- 172.21.15.31:0/651720607 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd1b80622a0 con 0x7fd1b80756e0 2024-03-21T10:38:16.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.983+0000 7fd1b6000640 1 -- 172.21.15.31:0/651720607 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd1a40bd730 con 0x7fd1b80756e0 2024-03-21T10:38:16.981 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-03-21T10:38:16.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.986+0000 7fd1bea00640 1 -- 172.21.15.31:0/651720607 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd1a00a2f20 msgr2=0x7fd1a00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:16.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.986+0000 7fd1bea00640 1 --2- 172.21.15.31:0/651720607 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd1a00a2f20 0x7fd1a00a53e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac009fc0 tx=0x7fd1ac005f90 comp rx=0 tx=0).stop 2024-03-21T10:38:16.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.986+0000 7fd1bea00640 1 -- 172.21.15.31:0/651720607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1b80756e0 msgr2=0x7fd1b8199f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:16.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.986+0000 7fd1bea00640 1 --2- 172.21.15.31:0/651720607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1b80756e0 0x7fd1b8199f90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd1a402c8f0 tx=0x7fd1a4005d50 comp rx=0 tx=0).stop 2024-03-21T10:38:16.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.986+0000 7fd1bea00640 1 -- 172.21.15.31:0/651720607 shutdown_connections 2024-03-21T10:38:16.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.986+0000 7fd1bea00640 1 --2- 172.21.15.31:0/651720607 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd1a00a2f20 0x7fd1a00a53e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:16.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.986+0000 7fd1bea00640 1 --2- 172.21.15.31:0/651720607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1b81093c0 0x7fd1b81a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:16.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.986+0000 7fd1bea00640 1 --2- 172.21.15.31:0/651720607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1b80756e0 0x7fd1b8199f90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:16.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.986+0000 7fd1bea00640 1 --2- 172.21.15.31:0/651720607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1b8074100 0x7fd1b8199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:16.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.986+0000 7fd1bea00640 1 -- 172.21.15.31:0/651720607 >> 172.21.15.31:0/651720607 conn(0x7fd1b80fcab0 msgr2=0x7fd1b81052c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:16.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.986+0000 7fd1bea00640 1 -- 172.21.15.31:0/651720607 shutdown_connections 2024-03-21T10:38:16.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:16.986+0000 7fd1bea00640 1 -- 172.21.15.31:0/651720607 wait complete. 2024-03-21T10:38:17.009 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T10:38:17.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:16 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/847938423' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:38:17.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:16 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/968186684' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:38:17.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:16 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/847938423' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:38:17.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:16 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/968186684' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:38:17.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.225+0000 7f8e43200640 1 -- 172.21.15.31:0/2118437081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e441030c0 msgr2=0x7f8e44109950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:17.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.225+0000 7f8e43200640 1 --2- 172.21.15.31:0/2118437081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e441030c0 0x7f8e44109950 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c009ca0 tx=0x7f8e3c02c100 comp rx=0 tx=0).stop 2024-03-21T10:38:17.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.225+0000 7f8e43200640 1 -- 172.21.15.31:0/2118437081 shutdown_connections 2024-03-21T10:38:17.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.225+0000 7f8e43200640 1 --2- 172.21.15.31:0/2118437081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e441030c0 0x7f8e44109950 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:17.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.225+0000 7f8e43200640 1 --2- 172.21.15.31:0/2118437081 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e44102700 0x7f8e44102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:17.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.225+0000 7f8e43200640 1 --2- 172.21.15.31:0/2118437081 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e44101510 0x7f8e44101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:17.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.225+0000 7f8e43200640 1 -- 172.21.15.31:0/2118437081 >> 172.21.15.31:0/2118437081 conn(0x7f8e440fca80 msgr2=0x7f8e440feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:17.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.225+0000 7f8e43200640 1 -- 172.21.15.31:0/2118437081 shutdown_connections 2024-03-21T10:38:17.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.225+0000 7f8e43200640 1 -- 172.21.15.31:0/2118437081 wait complete. 2024-03-21T10:38:17.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.226+0000 7f8e43200640 1 Processor -- start 2024-03-21T10:38:17.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.226+0000 7f8e43200640 1 -- start start 2024-03-21T10:38:17.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.227+0000 7f8e43200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e44101510 0x7f8e44075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:17.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.227+0000 7f8e43200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e44102700 0x7f8e44073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:17.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.227+0000 7f8e43200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e441030c0 0x7f8e44073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:17.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.227+0000 7f8e43200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4410d1a0 con 0x7f8e44101510 2024-03-21T10:38:17.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.227+0000 7f8e43200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4410d020 con 0x7f8e441030c0 2024-03-21T10:38:17.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.227+0000 7f8e43200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4410d320 con 0x7f8e44102700 2024-03-21T10:38:17.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.227+0000 7f8e42800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e441030c0 0x7f8e44073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:17.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.227+0000 7f8e41400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e44102700 0x7f8e44073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:17.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.227+0000 7f8e42800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e441030c0 0x7f8e44073fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:45994/0 (socket says 172.21.15.31:45994) 2024-03-21T10:38:17.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.227+0000 7f8e42800640 1 -- 172.21.15.31:0/3875844793 learned_addr learned my addr 172.21.15.31:0/3875844793 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:17.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.227+0000 7f8e41e00640 1 --2- 172.21.15.31:0/3875844793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e44101510 0x7f8e44075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:17.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.227+0000 7f8e42800640 1 -- 172.21.15.31:0/3875844793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e44102700 msgr2=0x7f8e44073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:17.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.227+0000 7f8e42800640 1 --2- 172.21.15.31:0/3875844793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e44102700 0x7f8e44073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:17.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.227+0000 7f8e42800640 1 -- 172.21.15.31:0/3875844793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e44101510 msgr2=0x7f8e44075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:17.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.227+0000 7f8e42800640 1 --2- 172.21.15.31:0/3875844793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e44101510 0x7f8e44075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:17.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.227+0000 7f8e42800640 1 -- 172.21.15.31:0/3875844793 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e44074510 con 0x7f8e441030c0 2024-03-21T10:38:17.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.227+0000 7f8e41e00640 1 --2- 172.21.15.31:0/3875844793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e44101510 0x7f8e44075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:17.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.228+0000 7f8e42800640 1 --2- 172.21.15.31:0/3875844793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e441030c0 0x7f8e44073fd0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c02c640 tx=0x7f8e3c035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:17.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.228+0000 7f8e3aa00640 1 -- 172.21.15.31:0/3875844793 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c039040 con 0x7f8e441030c0 2024-03-21T10:38:17.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.228+0000 7f8e3aa00640 1 -- 172.21.15.31:0/3875844793 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e3c044070 con 0x7f8e441030c0 2024-03-21T10:38:17.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.228+0000 7f8e3aa00640 1 -- 172.21.15.31:0/3875844793 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c0043c0 con 0x7f8e441030c0 2024-03-21T10:38:17.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.228+0000 7f8e43200640 1 -- 172.21.15.31:0/3875844793 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e440747a0 con 0x7f8e441030c0 2024-03-21T10:38:17.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.228+0000 7f8e43200640 1 -- 172.21.15.31:0/3875844793 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e441a61d0 con 0x7f8e441030c0 2024-03-21T10:38:17.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.229+0000 7f8e43200640 1 -- 172.21.15.31:0/3875844793 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e441087d0 con 0x7f8e441030c0 2024-03-21T10:38:17.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.231+0000 7f8e3aa00640 1 -- 172.21.15.31:0/3875844793 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f8e3c004560 con 0x7f8e441030c0 2024-03-21T10:38:17.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.232+0000 7f8e3aa00640 1 --2- 172.21.15.31:0/3875844793 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f8e280a2fa0 0x7f8e280a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:17.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.233+0000 7f8e3aa00640 1 -- 172.21.15.31:0/3875844793 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f8e3c0f1fe0 con 0x7f8e441030c0 2024-03-21T10:38:17.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.233+0000 7f8e41e00640 1 --2- 172.21.15.31:0/3875844793 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f8e280a2fa0 0x7f8e280a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:17.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.234+0000 7f8e41e00640 1 --2- 172.21.15.31:0/3875844793 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f8e280a2fa0 0x7f8e280a5460 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8e44102560 tx=0x7f8e34005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:17.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.235+0000 7f8e3aa00640 1 -- 172.21.15.31:0/3875844793 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8e3c0bea70 con 0x7f8e441030c0 2024-03-21T10:38:17.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.444+0000 7f8e43200640 1 -- 172.21.15.31:0/3875844793 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f8e4402d2c0 con 0x7f8e441030c0 2024-03-21T10:38:17.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.444+0000 7f8e3aa00640 1 -- 172.21.15.31:0/3875844793 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7f8e3c0098b0 con 0x7f8e441030c0 2024-03-21T10:38:17.442 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-03-21T10:38:17.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.447+0000 7f8e43200640 1 -- 172.21.15.31:0/3875844793 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f8e280a2fa0 msgr2=0x7f8e280a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:17.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.447+0000 7f8e43200640 1 --2- 172.21.15.31:0/3875844793 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f8e280a2fa0 0x7f8e280a5460 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8e44102560 tx=0x7f8e34005f90 comp rx=0 tx=0).stop 2024-03-21T10:38:17.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.447+0000 7f8e43200640 1 -- 172.21.15.31:0/3875844793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e441030c0 msgr2=0x7f8e44073fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:17.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.447+0000 7f8e43200640 1 --2- 172.21.15.31:0/3875844793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e441030c0 0x7f8e44073fd0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c02c640 tx=0x7f8e3c035d10 comp rx=0 tx=0).stop 2024-03-21T10:38:17.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.447+0000 7f8e43200640 1 -- 172.21.15.31:0/3875844793 shutdown_connections 2024-03-21T10:38:17.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.447+0000 7f8e43200640 1 --2- 172.21.15.31:0/3875844793 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f8e280a2fa0 0x7f8e280a5460 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:17.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.447+0000 7f8e43200640 1 --2- 172.21.15.31:0/3875844793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e441030c0 0x7f8e44073fd0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:17.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.447+0000 7f8e43200640 1 --2- 172.21.15.31:0/3875844793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e44102700 0x7f8e44073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:17.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.447+0000 7f8e43200640 1 --2- 172.21.15.31:0/3875844793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e44101510 0x7f8e44075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:17.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.447+0000 7f8e43200640 1 -- 172.21.15.31:0/3875844793 >> 172.21.15.31:0/3875844793 conn(0x7f8e440fca80 msgr2=0x7f8e441059a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:17.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.448+0000 7f8e43200640 1 -- 172.21.15.31:0/3875844793 shutdown_connections 2024-03-21T10:38:17.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.448+0000 7f8e43200640 1 -- 172.21.15.31:0/3875844793 wait complete. 2024-03-21T10:38:17.465 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-21T10:38:17.465 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:38:17.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.686+0000 7f5ca9800640 1 -- 172.21.15.31:0/4063749288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ca41027b0 msgr2=0x7f5ca4102c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:17.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.686+0000 7f5ca9800640 1 --2- 172.21.15.31:0/4063749288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ca41027b0 0x7f5ca4102c30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5c90009900 tx=0x7f5c9002bc40 comp rx=0 tx=0).stop 2024-03-21T10:38:17.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.686+0000 7f5ca9800640 1 -- 172.21.15.31:0/4063749288 shutdown_connections 2024-03-21T10:38:17.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.686+0000 7f5ca9800640 1 --2- 172.21.15.31:0/4063749288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ca4103170 0x7f5ca4109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:17.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.686+0000 7f5ca9800640 1 --2- 172.21.15.31:0/4063749288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ca41027b0 0x7f5ca4102c30 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:17.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.686+0000 7f5ca9800640 1 --2- 172.21.15.31:0/4063749288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ca41015c0 0x7f5ca41019e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:17.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.686+0000 7f5ca9800640 1 -- 172.21.15.31:0/4063749288 >> 172.21.15.31:0/4063749288 conn(0x7f5ca40fcb30 msgr2=0x7f5ca40fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:17.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.686+0000 7f5ca9800640 1 -- 172.21.15.31:0/4063749288 shutdown_connections 2024-03-21T10:38:17.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.686+0000 7f5ca9800640 1 -- 172.21.15.31:0/4063749288 wait complete. 2024-03-21T10:38:17.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.687+0000 7f5ca9800640 1 Processor -- start 2024-03-21T10:38:17.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.688+0000 7f5ca9800640 1 -- start start 2024-03-21T10:38:17.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.688+0000 7f5ca9800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ca41015c0 0x7f5ca4199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:17.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.688+0000 7f5ca9800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ca41027b0 0x7f5ca4199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:17.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.688+0000 7f5ca9800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ca4103170 0x7f5ca41a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:17.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.688+0000 7f5ca9800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca410d2a0 con 0x7f5ca41015c0 2024-03-21T10:38:17.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.688+0000 7f5ca9800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca410d120 con 0x7f5ca41027b0 2024-03-21T10:38:17.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.688+0000 7f5ca9800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca410d420 con 0x7f5ca4103170 2024-03-21T10:38:17.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.688+0000 7f5ca3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ca41027b0 0x7f5ca4199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:17.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.688+0000 7f5ca8e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ca4103170 0x7f5ca41a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:17.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.688+0000 7f5ca3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ca41015c0 0x7f5ca4199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:17.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.688+0000 7f5ca3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ca41027b0 0x7f5ca4199fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46010/0 (socket says 172.21.15.31:46010) 2024-03-21T10:38:17.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.688+0000 7f5ca3400640 1 -- 172.21.15.31:0/1744317026 learned_addr learned my addr 172.21.15.31:0/1744317026 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:17.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.689+0000 7f5ca8e00640 1 -- 172.21.15.31:0/1744317026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ca41027b0 msgr2=0x7f5ca4199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:17.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.689+0000 7f5ca8e00640 1 --2- 172.21.15.31:0/1744317026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ca41027b0 0x7f5ca4199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:17.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.689+0000 7f5ca8e00640 1 -- 172.21.15.31:0/1744317026 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ca41015c0 msgr2=0x7f5ca4199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:17.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.689+0000 7f5ca8e00640 1 --2- 172.21.15.31:0/1744317026 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ca41015c0 0x7f5ca4199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:17.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.689+0000 7f5ca8e00640 1 -- 172.21.15.31:0/1744317026 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca41a16c0 con 0x7f5ca4103170 2024-03-21T10:38:17.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.689+0000 7f5ca3400640 1 --2- 172.21.15.31:0/1744317026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ca41027b0 0x7f5ca4199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:17.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.689+0000 7f5ca8e00640 1 --2- 172.21.15.31:0/1744317026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ca4103170 0x7f5ca41a1050 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c009fd0 tx=0x7f5c9c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:17.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.689+0000 7f5ca0c00640 1 -- 172.21.15.31:0/1744317026 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9c00ce20 con 0x7f5ca4103170 2024-03-21T10:38:17.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.689+0000 7f5ca0c00640 1 -- 172.21.15.31:0/1744317026 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5c9c005020 con 0x7f5ca4103170 2024-03-21T10:38:17.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.689+0000 7f5ca0c00640 1 -- 172.21.15.31:0/1744317026 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9c010720 con 0x7f5ca4103170 2024-03-21T10:38:17.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.689+0000 7f5ca9800640 1 -- 172.21.15.31:0/1744317026 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ca41a19b0 con 0x7f5ca4103170 2024-03-21T10:38:17.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.689+0000 7f5ca9800640 1 -- 172.21.15.31:0/1744317026 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ca41a1f20 con 0x7f5ca4103170 2024-03-21T10:38:17.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.691+0000 7f5ca9800640 1 -- 172.21.15.31:0/1744317026 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca41a21b0 con 0x7f5ca4103170 2024-03-21T10:38:17.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.694+0000 7f5ca0c00640 1 -- 172.21.15.31:0/1744317026 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f5c9c005540 con 0x7f5ca4103170 2024-03-21T10:38:17.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.695+0000 7f5ca0c00640 1 --2- 172.21.15.31:0/1744317026 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5c880a2f90 0x7f5c880a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:17.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.695+0000 7f5ca0c00640 1 -- 172.21.15.31:0/1744317026 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f5c9c0cfd00 con 0x7f5ca4103170 2024-03-21T10:38:17.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.696+0000 7f5ca3e00640 1 --2- 172.21.15.31:0/1744317026 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5c880a2f90 0x7f5c880a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:17.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.696+0000 7f5ca3e00640 1 --2- 172.21.15.31:0/1744317026 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5c880a2f90 0x7f5c880a5450 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5ca4102610 tx=0x7f5c94005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:17.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:17.699+0000 7f5ca0c00640 1 -- 172.21.15.31:0/1744317026 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5c9c09c770 con 0x7f5ca4103170 2024-03-21T10:38:18.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.016+0000 7f5ca9800640 1 -- 172.21.15.31:0/1744317026 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ca41a2380 con 0x7f5ca4103170 2024-03-21T10:38:18.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.016+0000 7f5ca0c00640 1 -- 172.21.15.31:0/1744317026 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11384 (secure 0 0 0) 0x7f5c9c09c070 con 0x7f5ca4103170 2024-03-21T10:38:18.014 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:38:18.014 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:38:12.319122+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24547,24596]},"id":1}]} 2024-03-21T10:38:18.015 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 24 2024-03-21T10:38:18.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.019+0000 7f5ca9800640 1 -- 172.21.15.31:0/1744317026 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5c880a2f90 msgr2=0x7f5c880a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:18.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.019+0000 7f5ca9800640 1 --2- 172.21.15.31:0/1744317026 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5c880a2f90 0x7f5c880a5450 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5ca4102610 tx=0x7f5c94005f90 comp rx=0 tx=0).stop 2024-03-21T10:38:18.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.019+0000 7f5ca9800640 1 -- 172.21.15.31:0/1744317026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ca4103170 msgr2=0x7f5ca41a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:18.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.019+0000 7f5ca9800640 1 --2- 172.21.15.31:0/1744317026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ca4103170 0x7f5ca41a1050 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c009fd0 tx=0x7f5c9c00ef10 comp rx=0 tx=0).stop 2024-03-21T10:38:18.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.019+0000 7f5ca9800640 1 -- 172.21.15.31:0/1744317026 shutdown_connections 2024-03-21T10:38:18.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.019+0000 7f5ca9800640 1 --2- 172.21.15.31:0/1744317026 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f5c880a2f90 0x7f5c880a5450 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:18.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.019+0000 7f5ca9800640 1 --2- 172.21.15.31:0/1744317026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ca4103170 0x7f5ca41a1050 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:18.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.019+0000 7f5ca9800640 1 --2- 172.21.15.31:0/1744317026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ca41027b0 0x7f5ca4199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:18.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.019+0000 7f5ca9800640 1 --2- 172.21.15.31:0/1744317026 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ca41015c0 0x7f5ca4199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:18.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.019+0000 7f5ca9800640 1 -- 172.21.15.31:0/1744317026 >> 172.21.15.31:0/1744317026 conn(0x7f5ca40fcb30 msgr2=0x7f5ca40738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:18.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.020+0000 7f5ca9800640 1 -- 172.21.15.31:0/1744317026 shutdown_connections 2024-03-21T10:38:18.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.020+0000 7f5ca9800640 1 -- 172.21.15.31:0/1744317026 wait complete. 2024-03-21T10:38:18.041 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:17 smithi031 ceph-mon[31673]: pgmap v424: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-21T10:38:18.041 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:17 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/651720607' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:38:18.041 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:17 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3875844793' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:38:18.045 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-21T10:38:18.045 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-21T10:38:18.046 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-21T10:38:18.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:17 smithi070 ceph-mon[36330]: pgmap v424: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-21T10:38:18.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:17 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/651720607' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:38:18.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:17 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3875844793' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:38:18.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:17 smithi040 ceph-mon[36154]: pgmap v424: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-21T10:38:18.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:17 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/651720607' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:38:18.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:17 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3875844793' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:38:18.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.257+0000 7ff8cee00640 1 -- 172.21.15.31:0/2142069346 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8c8102920 msgr2=0x7ff8c8102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:18.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.257+0000 7ff8cee00640 1 --2- 172.21.15.31:0/2142069346 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8c8102920 0x7ff8c8102da0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0009900 tx=0x7ff8b002bc30 comp rx=0 tx=0).stop 2024-03-21T10:38:18.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.258+0000 7ff8cee00640 1 -- 172.21.15.31:0/2142069346 shutdown_connections 2024-03-21T10:38:18.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.258+0000 7ff8cee00640 1 --2- 172.21.15.31:0/2142069346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8c81032e0 0x7ff8c8109b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:18.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.258+0000 7ff8cee00640 1 --2- 172.21.15.31:0/2142069346 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8c8102920 0x7ff8c8102da0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:18.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.258+0000 7ff8cee00640 1 --2- 172.21.15.31:0/2142069346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8c8101730 0x7ff8c8101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:18.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.258+0000 7ff8cee00640 1 -- 172.21.15.31:0/2142069346 >> 172.21.15.31:0/2142069346 conn(0x7ff8c80fcc80 msgr2=0x7ff8c80ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:18.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.258+0000 7ff8cee00640 1 -- 172.21.15.31:0/2142069346 shutdown_connections 2024-03-21T10:38:18.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.258+0000 7ff8cee00640 1 -- 172.21.15.31:0/2142069346 wait complete. 2024-03-21T10:38:18.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.259+0000 7ff8cee00640 1 Processor -- start 2024-03-21T10:38:18.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.259+0000 7ff8cee00640 1 -- start start 2024-03-21T10:38:18.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.260+0000 7ff8cee00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8c8101730 0x7ff8c8199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:18.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.260+0000 7ff8cee00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8c8102920 0x7ff8c819a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:18.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.260+0000 7ff8cee00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8c81032e0 0x7ff8c81a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:18.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.260+0000 7ff8cee00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c810d390 con 0x7ff8c8101730 2024-03-21T10:38:18.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.260+0000 7ff8cee00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c810d210 con 0x7ff8c81032e0 2024-03-21T10:38:18.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.260+0000 7ff8cee00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c810d510 con 0x7ff8c8102920 2024-03-21T10:38:18.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.260+0000 7ff8bf400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8c8102920 0x7ff8c819a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:18.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.260+0000 7ff8c7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8c8101730 0x7ff8c8199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:18.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.260+0000 7ff8bf400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8c8102920 0x7ff8c819a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53496/0 (socket says 172.21.15.31:53496) 2024-03-21T10:38:18.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.260+0000 7ff8bf400640 1 -- 172.21.15.31:0/3849108656 learned_addr learned my addr 172.21.15.31:0/3849108656 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:18.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.260+0000 7ff8cce00640 1 --2- 172.21.15.31:0/3849108656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8c81032e0 0x7ff8c81a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:18.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.261+0000 7ff8bf400640 1 -- 172.21.15.31:0/3849108656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8c81032e0 msgr2=0x7ff8c81a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:18.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.261+0000 7ff8bf400640 1 --2- 172.21.15.31:0/3849108656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8c81032e0 0x7ff8c81a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:18.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.261+0000 7ff8bf400640 1 -- 172.21.15.31:0/3849108656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8c8101730 msgr2=0x7ff8c8199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:18.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.261+0000 7ff8bf400640 1 --2- 172.21.15.31:0/3849108656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8c8101730 0x7ff8c8199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:18.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.261+0000 7ff8bf400640 1 -- 172.21.15.31:0/3849108656 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c81a1810 con 0x7ff8c8102920 2024-03-21T10:38:18.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.261+0000 7ff8bf400640 1 --2- 172.21.15.31:0/3849108656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8c8102920 0x7ff8c819a130 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0005ab0 tx=0x7ff8b002c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:18.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.261+0000 7ff8c5600640 1 -- 172.21.15.31:0/3849108656 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0043020 con 0x7ff8c8102920 2024-03-21T10:38:18.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.261+0000 7ff8c5600640 1 -- 172.21.15.31:0/3849108656 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff8b00054e0 con 0x7ff8c8102920 2024-03-21T10:38:18.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.261+0000 7ff8c5600640 1 -- 172.21.15.31:0/3849108656 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0003ce0 con 0x7ff8c8102920 2024-03-21T10:38:18.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.261+0000 7ff8cee00640 1 -- 172.21.15.31:0/3849108656 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8c81a1aa0 con 0x7ff8c8102920 2024-03-21T10:38:18.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.262+0000 7ff8cee00640 1 -- 172.21.15.31:0/3849108656 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8c81a2040 con 0x7ff8c8102920 2024-03-21T10:38:18.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.263+0000 7ff8cee00640 1 -- 172.21.15.31:0/3849108656 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8c81a2210 con 0x7ff8c8102920 2024-03-21T10:38:18.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.264+0000 7ff8c5600640 1 -- 172.21.15.31:0/3849108656 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7ff8b0046020 con 0x7ff8c8102920 2024-03-21T10:38:18.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.266+0000 7ff8c5600640 1 --2- 172.21.15.31:0/3849108656 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff8ac0a2fa0 0x7ff8ac0a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:18.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.266+0000 7ff8c5600640 1 -- 172.21.15.31:0/3849108656 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7ff8b00f17f0 con 0x7ff8c8102920 2024-03-21T10:38:18.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.266+0000 7ff8c7e00640 1 --2- 172.21.15.31:0/3849108656 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff8ac0a2fa0 0x7ff8ac0a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:18.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.267+0000 7ff8c7e00640 1 --2- 172.21.15.31:0/3849108656 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff8ac0a2fa0 0x7ff8ac0a5460 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8102780 tx=0x7ff8b8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:18.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.269+0000 7ff8c5600640 1 -- 172.21.15.31:0/3849108656 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff8b004d020 con 0x7ff8c8102920 2024-03-21T10:38:18.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.465+0000 7ff8cee00640 1 -- 172.21.15.31:0/3849108656 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff8c8074ff0 con 0x7ff8ac0a2fa0 2024-03-21T10:38:18.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.497+0000 7ff8c5600640 1 -- 172.21.15.31:0/3849108656 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340815 (secure 0 0 0) 0x7ff8c8074ff0 con 0x7ff8ac0a2fa0 2024-03-21T10:38:18.498 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:38:18.502 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-03-21T10:38:18.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.505+0000 7ff8cee00640 1 -- 172.21.15.31:0/3849108656 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff8ac0a2fa0 msgr2=0x7ff8ac0a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:18.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.505+0000 7ff8cee00640 1 --2- 172.21.15.31:0/3849108656 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff8ac0a2fa0 0x7ff8ac0a5460 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8102780 tx=0x7ff8b8009500 comp rx=0 tx=0).stop 2024-03-21T10:38:18.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.505+0000 7ff8cee00640 1 -- 172.21.15.31:0/3849108656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8c8102920 msgr2=0x7ff8c819a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:18.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.505+0000 7ff8cee00640 1 --2- 172.21.15.31:0/3849108656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8c8102920 0x7ff8c819a130 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0005ab0 tx=0x7ff8b002c6c0 comp rx=0 tx=0).stop 2024-03-21T10:38:18.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.505+0000 7ff8cee00640 1 -- 172.21.15.31:0/3849108656 shutdown_connections 2024-03-21T10:38:18.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.505+0000 7ff8cee00640 1 --2- 172.21.15.31:0/3849108656 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff8ac0a2fa0 0x7ff8ac0a5460 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:18.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.505+0000 7ff8cee00640 1 --2- 172.21.15.31:0/3849108656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8c81032e0 0x7ff8c81a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:18.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.505+0000 7ff8cee00640 1 --2- 172.21.15.31:0/3849108656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8c8102920 0x7ff8c819a130 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:18.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.505+0000 7ff8cee00640 1 --2- 172.21.15.31:0/3849108656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8c8101730 0x7ff8c8199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:18.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.505+0000 7ff8cee00640 1 -- 172.21.15.31:0/3849108656 >> 172.21.15.31:0/3849108656 conn(0x7ff8c80fcc80 msgr2=0x7ff8c80738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:18.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.505+0000 7ff8cee00640 1 -- 172.21.15.31:0/3849108656 shutdown_connections 2024-03-21T10:38:18.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.506+0000 7ff8cee00640 1 -- 172.21.15.31:0/3849108656 wait complete. 2024-03-21T10:38:18.525 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2024-03-21T10:38:18.076535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659811,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":1093,"num_read_kb":672,"num_write":363,"num_write_kb":3069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":793,"ondisk_log_size":793,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369956,"kb_used_data":12440,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571868,"statfs":{"total":1151940427776,"available":1151561592832,"internally_reserved":0,"allocated":12738560,"data_stored":8188127,"data_compressed":433329,"data_compressed_allocated":589824,"data_compressed_original":2678784,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":612,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":32,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005529"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619666+0000","last_change":"2024-03-21T10:35:06.923544+0000","last_active":"2024-03-21T10:36:56.619666+0000","last_peered":"2024-03-21T10:36:56.619666+0000","last_clean":"2024-03-21T10:36:56.619666+0000","last_became_active":"2024-03-21T10:35:06.923254+0000","last_became_peered":"2024-03-21T10:35:06.923254+0000","last_unstale":"2024-03-21T10:36:56.619666+0000","last_undegraded":"2024-03-21T10:36:56.619666+0000","last_fullsized":"2024-03-21T10:36:56.619666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:38:14.573590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.622736+0000","last_change":"2024-03-21T10:35:06.929242+0000","last_active":"2024-03-21T10:36:56.622736+0000","last_peered":"2024-03-21T10:36:56.622736+0000","last_clean":"2024-03-21T10:36:56.622736+0000","last_became_active":"2024-03-21T10:35:06.927078+0000","last_became_peered":"2024-03-21T10:35:06.927078+0000","last_unstale":"2024-03-21T10:36:56.622736+0000","last_undegraded":"2024-03-21T10:36:56.622736+0000","last_fullsized":"2024-03-21T10:36:56.622736+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:27:10.347945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621857+0000","last_change":"2024-03-21T10:35:06.927792+0000","last_active":"2024-03-21T10:36:56.621857+0000","last_peered":"2024-03-21T10:36:56.621857+0000","last_clean":"2024-03-21T10:36:56.621857+0000","last_became_active":"2024-03-21T10:35:06.927386+0000","last_became_peered":"2024-03-21T10:35:06.927386+0000","last_unstale":"2024-03-21T10:36:56.621857+0000","last_undegraded":"2024-03-21T10:36:56.621857+0000","last_fullsized":"2024-03-21T10:36:56.621857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:53:51.310283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540864+0000","last_change":"2024-03-21T10:35:06.908901+0000","last_active":"2024-03-21T10:36:56.540864+0000","last_peered":"2024-03-21T10:36:56.540864+0000","last_clean":"2024-03-21T10:36:56.540864+0000","last_became_active":"2024-03-21T10:35:06.908586+0000","last_became_peered":"2024-03-21T10:35:06.908586+0000","last_unstale":"2024-03-21T10:36:56.540864+0000","last_undegraded":"2024-03-21T10:36:56.540864+0000","last_fullsized":"2024-03-21T10:36:56.540864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:25:57.387769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312725+0000","last_change":"2024-03-21T10:35:06.932537+0000","last_active":"2024-03-21T10:36:56.312725+0000","last_peered":"2024-03-21T10:36:56.312725+0000","last_clean":"2024-03-21T10:36:56.312725+0000","last_became_active":"2024-03-21T10:35:06.932185+0000","last_became_peered":"2024-03-21T10:35:06.932185+0000","last_unstale":"2024-03-21T10:36:56.312725+0000","last_undegraded":"2024-03-21T10:36:56.312725+0000","last_fullsized":"2024-03-21T10:36:56.312725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:30:25.127073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311445+0000","last_change":"2024-03-21T10:35:06.922765+0000","last_active":"2024-03-21T10:36:56.311445+0000","last_peered":"2024-03-21T10:36:56.311445+0000","last_clean":"2024-03-21T10:36:56.311445+0000","last_became_active":"2024-03-21T10:35:06.922447+0000","last_became_peered":"2024-03-21T10:35:06.922447+0000","last_unstale":"2024-03-21T10:36:56.311445+0000","last_undegraded":"2024-03-21T10:36:56.311445+0000","last_fullsized":"2024-03-21T10:36:56.311445+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:37:33.310735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619036+0000","last_change":"2024-03-21T10:35:06.896777+0000","last_active":"2024-03-21T10:36:56.619036+0000","last_peered":"2024-03-21T10:36:56.619036+0000","last_clean":"2024-03-21T10:36:56.619036+0000","last_became_active":"2024-03-21T10:35:06.896544+0000","last_became_peered":"2024-03-21T10:35:06.896544+0000","last_unstale":"2024-03-21T10:36:56.619036+0000","last_undegraded":"2024-03-21T10:36:56.619036+0000","last_fullsized":"2024-03-21T10:36:56.619036+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:34:05.000661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325012+0000","last_change":"2024-03-21T10:35:06.923294+0000","last_active":"2024-03-21T10:36:56.325012+0000","last_peered":"2024-03-21T10:36:56.325012+0000","last_clean":"2024-03-21T10:36:56.325012+0000","last_became_active":"2024-03-21T10:35:06.921320+0000","last_became_peered":"2024-03-21T10:35:06.921320+0000","last_unstale":"2024-03-21T10:36:56.325012+0000","last_undegraded":"2024-03-21T10:36:56.325012+0000","last_fullsized":"2024-03-21T10:36:56.325012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:33:53.855908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619843+0000","last_change":"2024-03-21T10:35:06.912977+0000","last_active":"2024-03-21T10:36:56.619843+0000","last_peered":"2024-03-21T10:36:56.619843+0000","last_clean":"2024-03-21T10:36:56.619843+0000","last_became_active":"2024-03-21T10:35:06.912626+0000","last_became_peered":"2024-03-21T10:35:06.912626+0000","last_unstale":"2024-03-21T10:36:56.619843+0000","last_undegraded":"2024-03-21T10:36:56.619843+0000","last_fullsized":"2024-03-21T10:36:56.619843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:42:13.472454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311923+0000","last_change":"2024-03-21T10:35:06.910252+0000","last_active":"2024-03-21T10:36:56.311923+0000","last_peered":"2024-03-21T10:36:56.311923+0000","last_clean":"2024-03-21T10:36:56.311923+0000","last_became_active":"2024-03-21T10:35:06.909924+0000","last_became_peered":"2024-03-21T10:35:06.909924+0000","last_unstale":"2024-03-21T10:36:56.311923+0000","last_undegraded":"2024-03-21T10:36:56.311923+0000","last_fullsized":"2024-03-21T10:36:56.311923+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:43:15.527349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619732+0000","last_change":"2024-03-21T10:35:06.909569+0000","last_active":"2024-03-21T10:36:56.619732+0000","last_peered":"2024-03-21T10:36:56.619732+0000","last_clean":"2024-03-21T10:36:56.619732+0000","last_became_active":"2024-03-21T10:35:06.908982+0000","last_became_peered":"2024-03-21T10:35:06.908982+0000","last_unstale":"2024-03-21T10:36:56.619732+0000","last_undegraded":"2024-03-21T10:36:56.619732+0000","last_fullsized":"2024-03-21T10:36:56.619732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:13:56.020281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540823+0000","last_change":"2024-03-21T10:35:06.913861+0000","last_active":"2024-03-21T10:36:56.540823+0000","last_peered":"2024-03-21T10:36:56.540823+0000","last_clean":"2024-03-21T10:36:56.540823+0000","last_became_active":"2024-03-21T10:35:06.913490+0000","last_became_peered":"2024-03-21T10:35:06.913490+0000","last_unstale":"2024-03-21T10:36:56.540823+0000","last_undegraded":"2024-03-21T10:36:56.540823+0000","last_fullsized":"2024-03-21T10:36:56.540823+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:24:18.573735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312584+0000","last_change":"2024-03-21T10:35:06.934537+0000","last_active":"2024-03-21T10:36:56.312584+0000","last_peered":"2024-03-21T10:36:56.312584+0000","last_clean":"2024-03-21T10:36:56.312584+0000","last_became_active":"2024-03-21T10:35:06.932951+0000","last_became_peered":"2024-03-21T10:35:06.932951+0000","last_unstale":"2024-03-21T10:36:56.312584+0000","last_undegraded":"2024-03-21T10:36:56.312584+0000","last_fullsized":"2024-03-21T10:36:56.312584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:43:17.748434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621260+0000","last_change":"2024-03-21T10:35:06.925495+0000","last_active":"2024-03-21T10:36:56.621260+0000","last_peered":"2024-03-21T10:36:56.621260+0000","last_clean":"2024-03-21T10:36:56.621260+0000","last_became_active":"2024-03-21T10:35:06.924326+0000","last_became_peered":"2024-03-21T10:35:06.924326+0000","last_unstale":"2024-03-21T10:36:56.621260+0000","last_undegraded":"2024-03-21T10:36:56.621260+0000","last_fullsized":"2024-03-21T10:36:56.621260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:27:19.598213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623885+0000","last_change":"2024-03-21T10:35:06.923695+0000","last_active":"2024-03-21T10:36:56.623885+0000","last_peered":"2024-03-21T10:36:56.623885+0000","last_clean":"2024-03-21T10:36:56.623885+0000","last_became_active":"2024-03-21T10:35:06.919114+0000","last_became_peered":"2024-03-21T10:35:06.919114+0000","last_unstale":"2024-03-21T10:36:56.623885+0000","last_undegraded":"2024-03-21T10:36:56.623885+0000","last_fullsized":"2024-03-21T10:36:56.623885+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:42:49.179653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619136+0000","last_change":"2024-03-21T10:35:06.930225+0000","last_active":"2024-03-21T10:36:56.619136+0000","last_peered":"2024-03-21T10:36:56.619136+0000","last_clean":"2024-03-21T10:36:56.619136+0000","last_became_active":"2024-03-21T10:35:06.929905+0000","last_became_peered":"2024-03-21T10:35:06.929905+0000","last_unstale":"2024-03-21T10:36:56.619136+0000","last_undegraded":"2024-03-21T10:36:56.619136+0000","last_fullsized":"2024-03-21T10:36:56.619136+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:21:07.231703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619516+0000","last_change":"2024-03-21T10:35:06.923498+0000","last_active":"2024-03-21T10:36:56.619516+0000","last_peered":"2024-03-21T10:36:56.619516+0000","last_clean":"2024-03-21T10:36:56.619516+0000","last_became_active":"2024-03-21T10:35:06.923242+0000","last_became_peered":"2024-03-21T10:35:06.923242+0000","last_unstale":"2024-03-21T10:36:56.619516+0000","last_undegraded":"2024-03-21T10:36:56.619516+0000","last_fullsized":"2024-03-21T10:36:56.619516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:42:04.309701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623901+0000","last_change":"2024-03-21T10:35:06.918314+0000","last_active":"2024-03-21T10:36:56.623901+0000","last_peered":"2024-03-21T10:36:56.623901+0000","last_clean":"2024-03-21T10:36:56.623901+0000","last_became_active":"2024-03-21T10:35:06.917865+0000","last_became_peered":"2024-03-21T10:35:06.917865+0000","last_unstale":"2024-03-21T10:36:56.623901+0000","last_undegraded":"2024-03-21T10:36:56.623901+0000","last_fullsized":"2024-03-21T10:36:56.623901+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:53:48.536341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619301+0000","last_change":"2024-03-21T10:35:06.936234+0000","last_active":"2024-03-21T10:36:56.619301+0000","last_peered":"2024-03-21T10:36:56.619301+0000","last_clean":"2024-03-21T10:36:56.619301+0000","last_became_active":"2024-03-21T10:35:06.935958+0000","last_became_peered":"2024-03-21T10:35:06.935958+0000","last_unstale":"2024-03-21T10:36:56.619301+0000","last_undegraded":"2024-03-21T10:36:56.619301+0000","last_fullsized":"2024-03-21T10:36:56.619301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:52:43.509671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618623+0000","last_change":"2024-03-21T10:35:11.198276+0000","last_active":"2024-03-21T10:36:56.618623+0000","last_peered":"2024-03-21T10:36:56.618623+0000","last_clean":"2024-03-21T10:36:56.618623+0000","last_became_active":"2024-03-21T10:35:11.197899+0000","last_became_peered":"2024-03-21T10:35:11.197899+0000","last_unstale":"2024-03-21T10:36:56.618623+0000","last_undegraded":"2024-03-21T10:36:56.618623+0000","last_fullsized":"2024-03-21T10:36:56.618623+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:17:30.499932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623544+0000","last_change":"2024-03-21T10:35:06.010647+0000","last_active":"2024-03-21T10:36:56.623544+0000","last_peered":"2024-03-21T10:36:56.623544+0000","last_clean":"2024-03-21T10:36:56.623544+0000","last_became_active":"2024-03-21T10:35:06.010239+0000","last_became_peered":"2024-03-21T10:35:06.010239+0000","last_unstale":"2024-03-21T10:36:56.623544+0000","last_undegraded":"2024-03-21T10:36:56.623544+0000","last_fullsized":"2024-03-21T10:36:56.623544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:35:53.324044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325857+0000","last_change":"2024-03-21T10:35:06.926398+0000","last_active":"2024-03-21T10:36:56.325857+0000","last_peered":"2024-03-21T10:36:56.325857+0000","last_clean":"2024-03-21T10:36:56.325857+0000","last_became_active":"2024-03-21T10:35:06.925904+0000","last_became_peered":"2024-03-21T10:35:06.925904+0000","last_unstale":"2024-03-21T10:36:56.325857+0000","last_undegraded":"2024-03-21T10:36:56.325857+0000","last_fullsized":"2024-03-21T10:36:56.325857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:37:52.811604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540507+0000","last_change":"2024-03-21T10:35:55.503266+0000","last_active":"2024-03-21T10:36:56.540507+0000","last_peered":"2024-03-21T10:36:56.540507+0000","last_clean":"2024-03-21T10:36:56.540507+0000","last_became_active":"2024-03-21T10:35:55.502694+0000","last_became_peered":"2024-03-21T10:35:55.502694+0000","last_unstale":"2024-03-21T10:36:56.540507+0000","last_undegraded":"2024-03-21T10:36:56.540507+0000","last_fullsized":"2024-03-21T10:36:56.540507+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:56:44.480667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325780+0000","last_change":"2024-03-21T10:35:06.022750+0000","last_active":"2024-03-21T10:36:56.325780+0000","last_peered":"2024-03-21T10:36:56.325780+0000","last_clean":"2024-03-21T10:36:56.325780+0000","last_became_active":"2024-03-21T10:35:06.022093+0000","last_became_peered":"2024-03-21T10:35:06.022093+0000","last_unstale":"2024-03-21T10:36:56.325780+0000","last_undegraded":"2024-03-21T10:36:56.325780+0000","last_fullsized":"2024-03-21T10:36:56.325780+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:53:28.458486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.624188+0000","last_change":"2024-03-21T10:35:06.927594+0000","last_active":"2024-03-21T10:36:56.624188+0000","last_peered":"2024-03-21T10:36:56.624188+0000","last_clean":"2024-03-21T10:36:56.624188+0000","last_became_active":"2024-03-21T10:35:06.926868+0000","last_became_peered":"2024-03-21T10:35:06.926868+0000","last_unstale":"2024-03-21T10:36:56.624188+0000","last_undegraded":"2024-03-21T10:36:56.624188+0000","last_fullsized":"2024-03-21T10:36:56.624188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:07:33.127465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540607+0000","last_change":"2024-03-21T10:35:11.193475+0000","last_active":"2024-03-21T10:36:56.540607+0000","last_peered":"2024-03-21T10:36:56.540607+0000","last_clean":"2024-03-21T10:36:56.540607+0000","last_became_active":"2024-03-21T10:35:11.193054+0000","last_became_peered":"2024-03-21T10:35:11.193054+0000","last_unstale":"2024-03-21T10:36:56.540607+0000","last_undegraded":"2024-03-21T10:36:56.540607+0000","last_fullsized":"2024-03-21T10:36:56.540607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:32:36.564147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313127+0000","last_change":"2024-03-21T10:35:06.004257+0000","last_active":"2024-03-21T10:36:56.313127+0000","last_peered":"2024-03-21T10:36:56.313127+0000","last_clean":"2024-03-21T10:36:56.313127+0000","last_became_active":"2024-03-21T10:35:06.003936+0000","last_became_peered":"2024-03-21T10:35:06.003936+0000","last_unstale":"2024-03-21T10:36:56.313127+0000","last_undegraded":"2024-03-21T10:36:56.313127+0000","last_fullsized":"2024-03-21T10:36:56.313127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:48:48.608979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.620162+0000","last_change":"2024-03-21T10:35:06.941681+0000","last_active":"2024-03-21T10:36:56.620162+0000","last_peered":"2024-03-21T10:36:56.620162+0000","last_clean":"2024-03-21T10:36:56.620162+0000","last_became_active":"2024-03-21T10:35:06.941314+0000","last_became_peered":"2024-03-21T10:35:06.941314+0000","last_unstale":"2024-03-21T10:36:56.620162+0000","last_undegraded":"2024-03-21T10:36:56.620162+0000","last_fullsized":"2024-03-21T10:36:56.620162+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:46:09.948922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540509+0000","last_change":"2024-03-21T10:35:11.188472+0000","last_active":"2024-03-21T10:36:56.540509+0000","last_peered":"2024-03-21T10:36:56.540509+0000","last_clean":"2024-03-21T10:36:56.540509+0000","last_became_active":"2024-03-21T10:35:11.188072+0000","last_became_peered":"2024-03-21T10:35:11.188072+0000","last_unstale":"2024-03-21T10:36:56.540509+0000","last_undegraded":"2024-03-21T10:36:56.540509+0000","last_fullsized":"2024-03-21T10:36:56.540509+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:04:16.911334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313102+0000","last_change":"2024-03-21T10:35:05.998493+0000","last_active":"2024-03-21T10:36:56.313102+0000","last_peered":"2024-03-21T10:36:56.313102+0000","last_clean":"2024-03-21T10:36:56.313102+0000","last_became_active":"2024-03-21T10:35:05.998134+0000","last_became_peered":"2024-03-21T10:35:05.998134+0000","last_unstale":"2024-03-21T10:36:56.313102+0000","last_undegraded":"2024-03-21T10:36:56.313102+0000","last_fullsized":"2024-03-21T10:36:56.313102+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:05:32.033328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311467+0000","last_change":"2024-03-21T10:35:06.912973+0000","last_active":"2024-03-21T10:36:56.311467+0000","last_peered":"2024-03-21T10:36:56.311467+0000","last_clean":"2024-03-21T10:36:56.311467+0000","last_became_active":"2024-03-21T10:35:06.912382+0000","last_became_peered":"2024-03-21T10:35:06.912382+0000","last_unstale":"2024-03-21T10:36:56.311467+0000","last_undegraded":"2024-03-21T10:36:56.311467+0000","last_fullsized":"2024-03-21T10:36:56.311467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:13:23.757805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619140+0000","last_change":"2024-03-21T10:35:11.180806+0000","last_active":"2024-03-21T10:36:56.619140+0000","last_peered":"2024-03-21T10:36:56.619140+0000","last_clean":"2024-03-21T10:36:56.619140+0000","last_became_active":"2024-03-21T10:35:11.180303+0000","last_became_peered":"2024-03-21T10:35:11.180303+0000","last_unstale":"2024-03-21T10:36:56.619140+0000","last_undegraded":"2024-03-21T10:36:56.619140+0000","last_fullsized":"2024-03-21T10:36:56.619140+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:27:11.719654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":196,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:38:13.155079+0000","last_change":"2024-03-21T10:35:06.004555+0000","last_active":"2024-03-21T10:38:13.155079+0000","last_peered":"2024-03-21T10:38:13.155079+0000","last_clean":"2024-03-21T10:38:13.155079+0000","last_became_active":"2024-03-21T10:35:06.003926+0000","last_became_peered":"2024-03-21T10:35:06.003926+0000","last_unstale":"2024-03-21T10:38:13.155079+0000","last_undegraded":"2024-03-21T10:38:13.155079+0000","last_fullsized":"2024-03-21T10:38:13.155079+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:05:49.721000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":158,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312743+0000","last_change":"2024-03-21T10:35:06.913556+0000","last_active":"2024-03-21T10:36:56.312743+0000","last_peered":"2024-03-21T10:36:56.312743+0000","last_clean":"2024-03-21T10:36:56.312743+0000","last_became_active":"2024-03-21T10:35:06.913213+0000","last_became_peered":"2024-03-21T10:35:06.913213+0000","last_unstale":"2024-03-21T10:36:56.312743+0000","last_undegraded":"2024-03-21T10:36:56.312743+0000","last_fullsized":"2024-03-21T10:36:56.312743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:11:09.235927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619374+0000","last_change":"2024-03-21T10:35:55.523819+0000","last_active":"2024-03-21T10:36:56.619374+0000","last_peered":"2024-03-21T10:36:56.619374+0000","last_clean":"2024-03-21T10:36:56.619374+0000","last_became_active":"2024-03-21T10:35:55.523333+0000","last_became_peered":"2024-03-21T10:35:55.523333+0000","last_unstale":"2024-03-21T10:36:56.619374+0000","last_undegraded":"2024-03-21T10:36:56.619374+0000","last_fullsized":"2024-03-21T10:36:56.619374+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:25:36.277481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311347+0000","last_change":"2024-03-21T10:35:06.012527+0000","last_active":"2024-03-21T10:36:56.311347+0000","last_peered":"2024-03-21T10:36:56.311347+0000","last_clean":"2024-03-21T10:36:56.311347+0000","last_became_active":"2024-03-21T10:35:06.012228+0000","last_became_peered":"2024-03-21T10:35:06.012228+0000","last_unstale":"2024-03-21T10:36:56.311347+0000","last_undegraded":"2024-03-21T10:36:56.311347+0000","last_fullsized":"2024-03-21T10:36:56.311347+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:52:34.622306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.622381+0000","last_change":"2024-03-21T10:35:55.503531+0000","last_active":"2024-03-21T10:36:56.622381+0000","last_peered":"2024-03-21T10:36:56.622381+0000","last_clean":"2024-03-21T10:36:56.622381+0000","last_became_active":"2024-03-21T10:35:55.502793+0000","last_became_peered":"2024-03-21T10:35:55.502793+0000","last_unstale":"2024-03-21T10:36:56.622381+0000","last_undegraded":"2024-03-21T10:36:56.622381+0000","last_fullsized":"2024-03-21T10:36:56.622381+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:08:39.599084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,3],"acting":[9,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619509+0000","last_change":"2024-03-21T10:35:11.196543+0000","last_active":"2024-03-21T10:36:56.619509+0000","last_peered":"2024-03-21T10:36:56.619509+0000","last_clean":"2024-03-21T10:36:56.619509+0000","last_became_active":"2024-03-21T10:35:11.196151+0000","last_became_peered":"2024-03-21T10:35:11.196151+0000","last_unstale":"2024-03-21T10:36:56.619509+0000","last_undegraded":"2024-03-21T10:36:56.619509+0000","last_fullsized":"2024-03-21T10:36:56.619509+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:15:39.193647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312200+0000","last_change":"2024-03-21T10:35:06.004588+0000","last_active":"2024-03-21T10:36:56.312200+0000","last_peered":"2024-03-21T10:36:56.312200+0000","last_clean":"2024-03-21T10:36:56.312200+0000","last_became_active":"2024-03-21T10:35:06.004190+0000","last_became_peered":"2024-03-21T10:35:06.004190+0000","last_unstale":"2024-03-21T10:36:56.312200+0000","last_undegraded":"2024-03-21T10:36:56.312200+0000","last_fullsized":"2024-03-21T10:36:56.312200+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:32:18.563692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325080+0000","last_change":"2024-03-21T10:35:06.941738+0000","last_active":"2024-03-21T10:36:56.325080+0000","last_peered":"2024-03-21T10:36:56.325080+0000","last_clean":"2024-03-21T10:36:56.325080+0000","last_became_active":"2024-03-21T10:35:06.941386+0000","last_became_peered":"2024-03-21T10:35:06.941386+0000","last_unstale":"2024-03-21T10:36:56.325080+0000","last_undegraded":"2024-03-21T10:36:56.325080+0000","last_fullsized":"2024-03-21T10:36:56.325080+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:52:55.519095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621767+0000","last_change":"2024-03-21T10:35:11.199370+0000","last_active":"2024-03-21T10:36:56.621767+0000","last_peered":"2024-03-21T10:36:56.621767+0000","last_clean":"2024-03-21T10:36:56.621767+0000","last_became_active":"2024-03-21T10:35:11.198598+0000","last_became_peered":"2024-03-21T10:35:11.198598+0000","last_unstale":"2024-03-21T10:36:56.621767+0000","last_undegraded":"2024-03-21T10:36:56.621767+0000","last_fullsized":"2024-03-21T10:36:56.621767+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:07:19.419405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311217+0000","last_change":"2024-03-21T10:35:06.004893+0000","last_active":"2024-03-21T10:36:56.311217+0000","last_peered":"2024-03-21T10:36:56.311217+0000","last_clean":"2024-03-21T10:36:56.311217+0000","last_became_active":"2024-03-21T10:35:06.004586+0000","last_became_peered":"2024-03-21T10:35:06.004586+0000","last_unstale":"2024-03-21T10:36:56.311217+0000","last_undegraded":"2024-03-21T10:36:56.311217+0000","last_fullsized":"2024-03-21T10:36:56.311217+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:05:29.106575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621403+0000","last_change":"2024-03-21T10:35:06.924448+0000","last_active":"2024-03-21T10:36:56.621403+0000","last_peered":"2024-03-21T10:36:56.621403+0000","last_clean":"2024-03-21T10:36:56.621403+0000","last_became_active":"2024-03-21T10:35:06.924049+0000","last_became_peered":"2024-03-21T10:35:06.924049+0000","last_unstale":"2024-03-21T10:36:56.621403+0000","last_undegraded":"2024-03-21T10:36:56.621403+0000","last_fullsized":"2024-03-21T10:36:56.621403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:23:12.070723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325020+0000","last_change":"2024-03-21T10:35:11.188691+0000","last_active":"2024-03-21T10:36:56.325020+0000","last_peered":"2024-03-21T10:36:56.325020+0000","last_clean":"2024-03-21T10:36:56.325020+0000","last_became_active":"2024-03-21T10:35:11.188138+0000","last_became_peered":"2024-03-21T10:35:11.188138+0000","last_unstale":"2024-03-21T10:36:56.325020+0000","last_undegraded":"2024-03-21T10:36:56.325020+0000","last_fullsized":"2024-03-21T10:36:56.325020+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:51:50.920920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":191,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:38:14.155614+0000","last_change":"2024-03-21T10:35:06.015276+0000","last_active":"2024-03-21T10:38:14.155614+0000","last_peered":"2024-03-21T10:38:14.155614+0000","last_clean":"2024-03-21T10:38:14.155614+0000","last_became_active":"2024-03-21T10:35:06.014734+0000","last_became_peered":"2024-03-21T10:35:06.014734+0000","last_unstale":"2024-03-21T10:38:14.155614+0000","last_undegraded":"2024-03-21T10:38:14.155614+0000","last_fullsized":"2024-03-21T10:38:14.155614+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:18:45.905451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":155,"num_read_kb":155,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.620346+0000","last_change":"2024-03-21T10:35:06.926689+0000","last_active":"2024-03-21T10:36:56.620346+0000","last_peered":"2024-03-21T10:36:56.620346+0000","last_clean":"2024-03-21T10:36:56.620346+0000","last_became_active":"2024-03-21T10:35:06.926319+0000","last_became_peered":"2024-03-21T10:35:06.926319+0000","last_unstale":"2024-03-21T10:36:56.620346+0000","last_undegraded":"2024-03-21T10:36:56.620346+0000","last_fullsized":"2024-03-21T10:36:56.620346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:06:22.371608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311104+0000","last_change":"2024-03-21T10:35:55.383776+0000","last_active":"2024-03-21T10:36:56.311104+0000","last_peered":"2024-03-21T10:36:56.311104+0000","last_clean":"2024-03-21T10:36:56.311104+0000","last_became_active":"2024-03-21T10:35:55.383300+0000","last_became_peered":"2024-03-21T10:35:55.383300+0000","last_unstale":"2024-03-21T10:36:56.311104+0000","last_undegraded":"2024-03-21T10:36:56.311104+0000","last_fullsized":"2024-03-21T10:36:56.311104+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:55:56.244325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2,4],"acting":[10,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619479+0000","last_change":"2024-03-21T10:35:06.007231+0000","last_active":"2024-03-21T10:36:56.619479+0000","last_peered":"2024-03-21T10:36:56.619479+0000","last_clean":"2024-03-21T10:36:56.619479+0000","last_became_active":"2024-03-21T10:35:06.006875+0000","last_became_peered":"2024-03-21T10:35:06.006875+0000","last_unstale":"2024-03-21T10:36:56.619479+0000","last_undegraded":"2024-03-21T10:36:56.619479+0000","last_fullsized":"2024-03-21T10:36:56.619479+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:08:02.489741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.624356+0000","last_change":"2024-03-21T10:35:55.524107+0000","last_active":"2024-03-21T10:36:56.624356+0000","last_peered":"2024-03-21T10:36:56.624356+0000","last_clean":"2024-03-21T10:36:56.624356+0000","last_became_active":"2024-03-21T10:35:55.523523+0000","last_became_peered":"2024-03-21T10:35:55.523523+0000","last_unstale":"2024-03-21T10:36:56.624356+0000","last_undegraded":"2024-03-21T10:36:56.624356+0000","last_fullsized":"2024-03-21T10:36:56.624356+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:49:29.121279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540722+0000","last_change":"2024-03-21T10:35:11.193382+0000","last_active":"2024-03-21T10:36:56.540722+0000","last_peered":"2024-03-21T10:36:56.540722+0000","last_clean":"2024-03-21T10:36:56.540722+0000","last_became_active":"2024-03-21T10:35:11.192959+0000","last_became_peered":"2024-03-21T10:35:11.192959+0000","last_unstale":"2024-03-21T10:36:56.540722+0000","last_undegraded":"2024-03-21T10:36:56.540722+0000","last_fullsized":"2024-03-21T10:36:56.540722+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:02:57.187370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311370+0000","last_change":"2024-03-21T10:35:06.019969+0000","last_active":"2024-03-21T10:36:56.311370+0000","last_peered":"2024-03-21T10:36:56.311370+0000","last_clean":"2024-03-21T10:36:56.311370+0000","last_became_active":"2024-03-21T10:35:06.019648+0000","last_became_peered":"2024-03-21T10:35:06.019648+0000","last_unstale":"2024-03-21T10:36:56.311370+0000","last_undegraded":"2024-03-21T10:36:56.311370+0000","last_fullsized":"2024-03-21T10:36:56.311370+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:24:02.227205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.620292+0000","last_change":"2024-03-21T10:35:06.910294+0000","last_active":"2024-03-21T10:36:56.620292+0000","last_peered":"2024-03-21T10:36:56.620292+0000","last_clean":"2024-03-21T10:36:56.620292+0000","last_became_active":"2024-03-21T10:35:06.909862+0000","last_became_peered":"2024-03-21T10:35:06.909862+0000","last_unstale":"2024-03-21T10:36:56.620292+0000","last_undegraded":"2024-03-21T10:36:56.620292+0000","last_fullsized":"2024-03-21T10:36:56.620292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:37:13.396003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621610+0000","last_change":"2024-03-21T10:35:11.196398+0000","last_active":"2024-03-21T10:36:56.621610+0000","last_peered":"2024-03-21T10:36:56.621610+0000","last_clean":"2024-03-21T10:36:56.621610+0000","last_became_active":"2024-03-21T10:35:11.194965+0000","last_became_peered":"2024-03-21T10:35:11.194965+0000","last_unstale":"2024-03-21T10:36:56.621610+0000","last_undegraded":"2024-03-21T10:36:56.621610+0000","last_fullsized":"2024-03-21T10:36:56.621610+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:48:22.024137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'38","reported_seq":391,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:38:15.156609+0000","last_change":"2024-03-21T10:35:06.005037+0000","last_active":"2024-03-21T10:38:15.156609+0000","last_peered":"2024-03-21T10:38:15.156609+0000","last_clean":"2024-03-21T10:38:15.156609+0000","last_became_active":"2024-03-21T10:35:06.004686+0000","last_became_peered":"2024-03-21T10:35:06.004686+0000","last_unstale":"2024-03-21T10:38:15.156609+0000","last_undegraded":"2024-03-21T10:38:15.156609+0000","last_fullsized":"2024-03-21T10:38:15.156609+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:24:13.312112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64761,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":321,"num_read_kb":225,"num_write":39,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313007+0000","last_change":"2024-03-21T10:35:06.906525+0000","last_active":"2024-03-21T10:36:56.313007+0000","last_peered":"2024-03-21T10:36:56.313007+0000","last_clean":"2024-03-21T10:36:56.313007+0000","last_became_active":"2024-03-21T10:35:06.905150+0000","last_became_peered":"2024-03-21T10:35:06.905150+0000","last_unstale":"2024-03-21T10:36:56.313007+0000","last_undegraded":"2024-03-21T10:36:56.313007+0000","last_fullsized":"2024-03-21T10:36:56.313007+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:46:27.958779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311725+0000","last_change":"2024-03-21T10:35:11.193519+0000","last_active":"2024-03-21T10:36:56.311725+0000","last_peered":"2024-03-21T10:36:56.311725+0000","last_clean":"2024-03-21T10:36:56.311725+0000","last_became_active":"2024-03-21T10:35:11.193043+0000","last_became_peered":"2024-03-21T10:35:11.193043+0000","last_unstale":"2024-03-21T10:36:56.311725+0000","last_undegraded":"2024-03-21T10:36:56.311725+0000","last_fullsized":"2024-03-21T10:36:56.311725+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:23:20.984472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618979+0000","last_change":"2024-03-21T10:35:05.999220+0000","last_active":"2024-03-21T10:36:56.618979+0000","last_peered":"2024-03-21T10:36:56.618979+0000","last_clean":"2024-03-21T10:36:56.618979+0000","last_became_active":"2024-03-21T10:35:05.998918+0000","last_became_peered":"2024-03-21T10:35:05.998918+0000","last_unstale":"2024-03-21T10:36:56.618979+0000","last_undegraded":"2024-03-21T10:36:56.618979+0000","last_fullsized":"2024-03-21T10:36:56.618979+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:01:09.816758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312512+0000","last_change":"2024-03-21T10:35:06.941234+0000","last_active":"2024-03-21T10:36:56.312512+0000","last_peered":"2024-03-21T10:36:56.312512+0000","last_clean":"2024-03-21T10:36:56.312512+0000","last_became_active":"2024-03-21T10:35:06.940828+0000","last_became_peered":"2024-03-21T10:35:06.940828+0000","last_unstale":"2024-03-21T10:36:56.312512+0000","last_undegraded":"2024-03-21T10:36:56.312512+0000","last_fullsized":"2024-03-21T10:36:56.312512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:50:34.398822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618827+0000","last_change":"2024-03-21T10:35:11.196007+0000","last_active":"2024-03-21T10:36:56.618827+0000","last_peered":"2024-03-21T10:36:56.618827+0000","last_clean":"2024-03-21T10:36:56.618827+0000","last_became_active":"2024-03-21T10:35:11.195609+0000","last_became_peered":"2024-03-21T10:35:11.195609+0000","last_unstale":"2024-03-21T10:36:56.618827+0000","last_undegraded":"2024-03-21T10:36:56.618827+0000","last_fullsized":"2024-03-21T10:36:56.618827+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:21:38.803390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311522+0000","last_change":"2024-03-21T10:35:06.011594+0000","last_active":"2024-03-21T10:36:56.311522+0000","last_peered":"2024-03-21T10:36:56.311522+0000","last_clean":"2024-03-21T10:36:56.311522+0000","last_became_active":"2024-03-21T10:35:06.011254+0000","last_became_peered":"2024-03-21T10:35:06.011254+0000","last_unstale":"2024-03-21T10:36:56.311522+0000","last_undegraded":"2024-03-21T10:36:56.311522+0000","last_fullsized":"2024-03-21T10:36:56.311522+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:52:31.045659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619993+0000","last_change":"2024-03-21T10:35:06.929170+0000","last_active":"2024-03-21T10:36:56.619993+0000","last_peered":"2024-03-21T10:36:56.619993+0000","last_clean":"2024-03-21T10:36:56.619993+0000","last_became_active":"2024-03-21T10:35:06.928822+0000","last_became_peered":"2024-03-21T10:35:06.928822+0000","last_unstale":"2024-03-21T10:36:56.619993+0000","last_undegraded":"2024-03-21T10:36:56.619993+0000","last_fullsized":"2024-03-21T10:36:56.619993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:37:40.328453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623051+0000","last_change":"2024-03-21T10:35:11.204285+0000","last_active":"2024-03-21T10:36:56.623051+0000","last_peered":"2024-03-21T10:36:56.623051+0000","last_clean":"2024-03-21T10:36:56.623051+0000","last_became_active":"2024-03-21T10:35:11.203698+0000","last_became_peered":"2024-03-21T10:35:11.203698+0000","last_unstale":"2024-03-21T10:36:56.623051+0000","last_undegraded":"2024-03-21T10:36:56.623051+0000","last_fullsized":"2024-03-21T10:36:56.623051+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:03:32.732488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"93'4","reported_seq":192,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:38:13.235271+0000","last_change":"2024-03-21T10:35:06.005028+0000","last_active":"2024-03-21T10:38:13.235271+0000","last_peered":"2024-03-21T10:38:13.235271+0000","last_clean":"2024-03-21T10:38:13.235271+0000","last_became_active":"2024-03-21T10:35:06.004670+0000","last_became_peered":"2024-03-21T10:35:06.004670+0000","last_unstale":"2024-03-21T10:38:13.235271+0000","last_undegraded":"2024-03-21T10:38:13.235271+0000","last_fullsized":"2024-03-21T10:38:13.235271+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:00:18.481051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":894,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":158,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619542+0000","last_change":"2024-03-21T10:35:06.911696+0000","last_active":"2024-03-21T10:36:56.619542+0000","last_peered":"2024-03-21T10:36:56.619542+0000","last_clean":"2024-03-21T10:36:56.619542+0000","last_became_active":"2024-03-21T10:35:06.911231+0000","last_became_peered":"2024-03-21T10:35:06.911231+0000","last_unstale":"2024-03-21T10:36:56.619542+0000","last_undegraded":"2024-03-21T10:36:56.619542+0000","last_fullsized":"2024-03-21T10:36:56.619542+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:21:51.540993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312937+0000","last_change":"2024-03-21T10:35:11.204422+0000","last_active":"2024-03-21T10:36:56.312937+0000","last_peered":"2024-03-21T10:36:56.312937+0000","last_clean":"2024-03-21T10:36:56.312937+0000","last_became_active":"2024-03-21T10:35:11.203904+0000","last_became_peered":"2024-03-21T10:35:11.203904+0000","last_unstale":"2024-03-21T10:36:56.312937+0000","last_undegraded":"2024-03-21T10:36:56.312937+0000","last_fullsized":"2024-03-21T10:36:56.312937+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:38:14.486012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313161+0000","last_change":"2024-03-21T10:35:06.019823+0000","last_active":"2024-03-21T10:36:56.313161+0000","last_peered":"2024-03-21T10:36:56.313161+0000","last_clean":"2024-03-21T10:36:56.313161+0000","last_became_active":"2024-03-21T10:35:06.019502+0000","last_became_peered":"2024-03-21T10:35:06.019502+0000","last_unstale":"2024-03-21T10:36:56.313161+0000","last_undegraded":"2024-03-21T10:36:56.313161+0000","last_fullsized":"2024-03-21T10:36:56.313161+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:33:22.438868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311234+0000","last_change":"2024-03-21T10:35:06.906671+0000","last_active":"2024-03-21T10:36:56.311234+0000","last_peered":"2024-03-21T10:36:56.311234+0000","last_clean":"2024-03-21T10:36:56.311234+0000","last_became_active":"2024-03-21T10:35:06.905873+0000","last_became_peered":"2024-03-21T10:35:06.905873+0000","last_unstale":"2024-03-21T10:36:56.311234+0000","last_undegraded":"2024-03-21T10:36:56.311234+0000","last_fullsized":"2024-03-21T10:36:56.311234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:14:59.304255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311196+0000","last_change":"2024-03-21T10:35:11.197698+0000","last_active":"2024-03-21T10:36:56.311196+0000","last_peered":"2024-03-21T10:36:56.311196+0000","last_clean":"2024-03-21T10:36:56.311196+0000","last_became_active":"2024-03-21T10:35:11.197295+0000","last_became_peered":"2024-03-21T10:35:11.197295+0000","last_unstale":"2024-03-21T10:36:56.311196+0000","last_undegraded":"2024-03-21T10:36:56.311196+0000","last_fullsized":"2024-03-21T10:36:56.311196+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:02:24.338317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"93'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:37:22.250209+0000","last_change":"2024-03-21T10:35:06.000951+0000","last_active":"2024-03-21T10:37:22.250209+0000","last_peered":"2024-03-21T10:37:22.250209+0000","last_clean":"2024-03-21T10:37:22.250209+0000","last_became_active":"2024-03-21T10:35:05.999169+0000","last_became_peered":"2024-03-21T10:35:05.999169+0000","last_unstale":"2024-03-21T10:37:22.250209+0000","last_undegraded":"2024-03-21T10:37:22.250209+0000","last_fullsized":"2024-03-21T10:37:22.250209+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:31:25.602621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619339+0000","last_change":"2024-03-21T10:35:06.908067+0000","last_active":"2024-03-21T10:36:56.619339+0000","last_peered":"2024-03-21T10:36:56.619339+0000","last_clean":"2024-03-21T10:36:56.619339+0000","last_became_active":"2024-03-21T10:35:06.907720+0000","last_became_peered":"2024-03-21T10:35:06.907720+0000","last_unstale":"2024-03-21T10:36:56.619339+0000","last_undegraded":"2024-03-21T10:36:56.619339+0000","last_fullsized":"2024-03-21T10:36:56.619339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:16:00.502637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312728+0000","last_change":"2024-03-21T10:35:11.229470+0000","last_active":"2024-03-21T10:36:56.312728+0000","last_peered":"2024-03-21T10:36:56.312728+0000","last_clean":"2024-03-21T10:36:56.312728+0000","last_became_active":"2024-03-21T10:35:11.228977+0000","last_became_peered":"2024-03-21T10:35:11.228977+0000","last_unstale":"2024-03-21T10:36:56.312728+0000","last_undegraded":"2024-03-21T10:36:56.312728+0000","last_fullsized":"2024-03-21T10:36:56.312728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:05:52.269712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623236+0000","last_change":"2024-03-21T10:35:06.019866+0000","last_active":"2024-03-21T10:36:56.623236+0000","last_peered":"2024-03-21T10:36:56.623236+0000","last_clean":"2024-03-21T10:36:56.623236+0000","last_became_active":"2024-03-21T10:35:06.019550+0000","last_became_peered":"2024-03-21T10:35:06.019550+0000","last_unstale":"2024-03-21T10:36:56.623236+0000","last_undegraded":"2024-03-21T10:36:56.623236+0000","last_fullsized":"2024-03-21T10:36:56.623236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:51:46.989647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311710+0000","last_change":"2024-03-21T10:35:55.385617+0000","last_active":"2024-03-21T10:36:56.311710+0000","last_peered":"2024-03-21T10:36:56.311710+0000","last_clean":"2024-03-21T10:36:56.311710+0000","last_became_active":"2024-03-21T10:35:55.385293+0000","last_became_peered":"2024-03-21T10:35:55.385293+0000","last_unstale":"2024-03-21T10:36:56.311710+0000","last_undegraded":"2024-03-21T10:36:56.311710+0000","last_fullsized":"2024-03-21T10:36:56.311710+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:18:08.200169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313097+0000","last_change":"2024-03-21T10:35:11.199714+0000","last_active":"2024-03-21T10:36:56.313097+0000","last_peered":"2024-03-21T10:36:56.313097+0000","last_clean":"2024-03-21T10:36:56.313097+0000","last_became_active":"2024-03-21T10:35:11.199277+0000","last_became_peered":"2024-03-21T10:35:11.199277+0000","last_unstale":"2024-03-21T10:36:56.313097+0000","last_undegraded":"2024-03-21T10:36:56.313097+0000","last_fullsized":"2024-03-21T10:36:56.313097+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:58:01.471162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619799+0000","last_change":"2024-03-21T10:35:06.015090+0000","last_active":"2024-03-21T10:36:56.619799+0000","last_peered":"2024-03-21T10:36:56.619799+0000","last_clean":"2024-03-21T10:36:56.619799+0000","last_became_active":"2024-03-21T10:35:06.014666+0000","last_became_peered":"2024-03-21T10:35:06.014666+0000","last_unstale":"2024-03-21T10:36:56.619799+0000","last_undegraded":"2024-03-21T10:36:56.619799+0000","last_fullsized":"2024-03-21T10:36:56.619799+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:41:17.608069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311382+0000","last_change":"2024-03-21T10:35:55.388548+0000","last_active":"2024-03-21T10:36:56.311382+0000","last_peered":"2024-03-21T10:36:56.311382+0000","last_clean":"2024-03-21T10:36:56.311382+0000","last_became_active":"2024-03-21T10:35:55.388253+0000","last_became_peered":"2024-03-21T10:35:55.388253+0000","last_unstale":"2024-03-21T10:36:56.311382+0000","last_undegraded":"2024-03-21T10:36:56.311382+0000","last_fullsized":"2024-03-21T10:36:56.311382+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:56.678224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312691+0000","last_change":"2024-03-21T10:35:11.230677+0000","last_active":"2024-03-21T10:36:56.312691+0000","last_peered":"2024-03-21T10:36:56.312691+0000","last_clean":"2024-03-21T10:36:56.312691+0000","last_became_active":"2024-03-21T10:35:11.230087+0000","last_became_peered":"2024-03-21T10:35:11.230087+0000","last_unstale":"2024-03-21T10:36:56.312691+0000","last_undegraded":"2024-03-21T10:36:56.312691+0000","last_fullsized":"2024-03-21T10:36:56.312691+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:39:47.663950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311339+0000","last_change":"2024-03-21T10:35:06.904287+0000","last_active":"2024-03-21T10:36:56.311339+0000","last_peered":"2024-03-21T10:36:56.311339+0000","last_clean":"2024-03-21T10:36:56.311339+0000","last_became_active":"2024-03-21T10:35:06.903821+0000","last_became_peered":"2024-03-21T10:35:06.903821+0000","last_unstale":"2024-03-21T10:36:56.311339+0000","last_undegraded":"2024-03-21T10:36:56.311339+0000","last_fullsized":"2024-03-21T10:36:56.311339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:36:25.362203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'5","reported_seq":191,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:38:13.234869+0000","last_change":"2024-03-21T10:35:06.014432+0000","last_active":"2024-03-21T10:38:13.234869+0000","last_peered":"2024-03-21T10:38:13.234869+0000","last_clean":"2024-03-21T10:38:13.234869+0000","last_became_active":"2024-03-21T10:35:06.013448+0000","last_became_peered":"2024-03-21T10:35:06.013448+0000","last_unstale":"2024-03-21T10:38:13.234869+0000","last_undegraded":"2024-03-21T10:38:13.234869+0000","last_fullsized":"2024-03-21T10:38:13.234869+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:29:10.600794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":154,"num_read_kb":154,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619114+0000","last_change":"2024-03-21T10:35:11.193577+0000","last_active":"2024-03-21T10:36:56.619114+0000","last_peered":"2024-03-21T10:36:56.619114+0000","last_clean":"2024-03-21T10:36:56.619114+0000","last_became_active":"2024-03-21T10:35:11.193132+0000","last_became_peered":"2024-03-21T10:35:11.193132+0000","last_unstale":"2024-03-21T10:36:56.619114+0000","last_undegraded":"2024-03-21T10:36:56.619114+0000","last_fullsized":"2024-03-21T10:36:56.619114+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:40:31.100489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325105+0000","last_change":"2024-03-21T10:35:06.924376+0000","last_active":"2024-03-21T10:36:56.325105+0000","last_peered":"2024-03-21T10:36:56.325105+0000","last_clean":"2024-03-21T10:36:56.325105+0000","last_became_active":"2024-03-21T10:35:06.923874+0000","last_became_peered":"2024-03-21T10:35:06.923874+0000","last_unstale":"2024-03-21T10:36:56.325105+0000","last_undegraded":"2024-03-21T10:36:56.325105+0000","last_fullsized":"2024-03-21T10:36:56.325105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:56:12.620667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540608+0000","last_change":"2024-03-21T10:35:06.015222+0000","last_active":"2024-03-21T10:36:56.540608+0000","last_peered":"2024-03-21T10:36:56.540608+0000","last_clean":"2024-03-21T10:36:56.540608+0000","last_became_active":"2024-03-21T10:35:06.014858+0000","last_became_peered":"2024-03-21T10:35:06.014858+0000","last_unstale":"2024-03-21T10:36:56.540608+0000","last_undegraded":"2024-03-21T10:36:56.540608+0000","last_fullsized":"2024-03-21T10:36:56.540608+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:45:12.528851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":367,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618815+0000","last_change":"2024-03-21T10:33:44.536069+0000","last_active":"2024-03-21T10:36:56.618815+0000","last_peered":"2024-03-21T10:36:56.618815+0000","last_clean":"2024-03-21T10:36:56.618815+0000","last_became_active":"2024-03-21T10:33:44.535623+0000","last_became_peered":"2024-03-21T10:33:44.535623+0000","last_unstale":"2024-03-21T10:36:56.618815+0000","last_undegraded":"2024-03-21T10:36:56.618815+0000","last_fullsized":"2024-03-21T10:36:56.618815+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:29:44.554765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:29:44.554765+0000","last_clean_scrub_stamp":"2024-03-21T10:29:44.554765+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:43:23.440837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311654+0000","last_change":"2024-03-21T10:35:11.229580+0000","last_active":"2024-03-21T10:36:56.311654+0000","last_peered":"2024-03-21T10:36:56.311654+0000","last_clean":"2024-03-21T10:36:56.311654+0000","last_became_active":"2024-03-21T10:35:11.229044+0000","last_became_peered":"2024-03-21T10:35:11.229044+0000","last_unstale":"2024-03-21T10:36:56.311654+0000","last_undegraded":"2024-03-21T10:36:56.311654+0000","last_fullsized":"2024-03-21T10:36:56.311654+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:54:50.535253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312060+0000","last_change":"2024-03-21T10:35:06.927690+0000","last_active":"2024-03-21T10:36:56.312060+0000","last_peered":"2024-03-21T10:36:56.312060+0000","last_clean":"2024-03-21T10:36:56.312060+0000","last_became_active":"2024-03-21T10:35:06.927304+0000","last_became_peered":"2024-03-21T10:35:06.927304+0000","last_unstale":"2024-03-21T10:36:56.312060+0000","last_undegraded":"2024-03-21T10:36:56.312060+0000","last_fullsized":"2024-03-21T10:36:56.312060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:30:48.950459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619515+0000","last_change":"2024-03-21T10:35:06.004952+0000","last_active":"2024-03-21T10:36:56.619515+0000","last_peered":"2024-03-21T10:36:56.619515+0000","last_clean":"2024-03-21T10:36:56.619515+0000","last_became_active":"2024-03-21T10:35:06.004206+0000","last_became_peered":"2024-03-21T10:35:06.004206+0000","last_unstale":"2024-03-21T10:36:56.619515+0000","last_undegraded":"2024-03-21T10:36:56.619515+0000","last_fullsized":"2024-03-21T10:36:56.619515+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:20:29.198715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619052+0000","last_change":"2024-03-21T10:35:11.179125+0000","last_active":"2024-03-21T10:36:56.619052+0000","last_peered":"2024-03-21T10:36:56.619052+0000","last_clean":"2024-03-21T10:36:56.619052+0000","last_became_active":"2024-03-21T10:35:11.178658+0000","last_became_peered":"2024-03-21T10:35:11.178658+0000","last_unstale":"2024-03-21T10:36:56.619052+0000","last_undegraded":"2024-03-21T10:36:56.619052+0000","last_fullsized":"2024-03-21T10:36:56.619052+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:33:25.281721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313030+0000","last_change":"2024-03-21T10:35:06.019996+0000","last_active":"2024-03-21T10:36:56.313030+0000","last_peered":"2024-03-21T10:36:56.313030+0000","last_clean":"2024-03-21T10:36:56.313030+0000","last_became_active":"2024-03-21T10:35:06.019624+0000","last_became_peered":"2024-03-21T10:35:06.019624+0000","last_unstale":"2024-03-21T10:36:56.313030+0000","last_undegraded":"2024-03-21T10:36:56.313030+0000","last_fullsized":"2024-03-21T10:36:56.313030+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:31:09.551433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313495+0000","last_change":"2024-03-21T10:35:06.939324+0000","last_active":"2024-03-21T10:36:56.313495+0000","last_peered":"2024-03-21T10:36:56.313495+0000","last_clean":"2024-03-21T10:36:56.313495+0000","last_became_active":"2024-03-21T10:35:06.937750+0000","last_became_peered":"2024-03-21T10:35:06.937750+0000","last_unstale":"2024-03-21T10:36:56.313495+0000","last_undegraded":"2024-03-21T10:36:56.313495+0000","last_fullsized":"2024-03-21T10:36:56.313495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:45:14.600863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.324947+0000","last_change":"2024-03-21T10:35:11.229479+0000","last_active":"2024-03-21T10:36:56.324947+0000","last_peered":"2024-03-21T10:36:56.324947+0000","last_clean":"2024-03-21T10:36:56.324947+0000","last_became_active":"2024-03-21T10:35:11.229005+0000","last_became_peered":"2024-03-21T10:35:11.229005+0000","last_unstale":"2024-03-21T10:36:56.324947+0000","last_undegraded":"2024-03-21T10:36:56.324947+0000","last_fullsized":"2024-03-21T10:36:56.324947+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:39:19.859581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'12","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:37:19.740273+0000","last_change":"2024-03-21T10:35:06.013744+0000","last_active":"2024-03-21T10:37:19.740273+0000","last_peered":"2024-03-21T10:37:19.740273+0000","last_clean":"2024-03-21T10:37:19.740273+0000","last_became_active":"2024-03-21T10:35:06.013509+0000","last_became_peered":"2024-03-21T10:35:06.013509+0000","last_unstale":"2024-03-21T10:37:19.740273+0000","last_undegraded":"2024-03-21T10:37:19.740273+0000","last_fullsized":"2024-03-21T10:37:19.740273+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:51:45.000933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325968+0000","last_change":"2024-03-21T10:35:06.930685+0000","last_active":"2024-03-21T10:36:56.325968+0000","last_peered":"2024-03-21T10:36:56.325968+0000","last_clean":"2024-03-21T10:36:56.325968+0000","last_became_active":"2024-03-21T10:35:06.929775+0000","last_became_peered":"2024-03-21T10:35:06.929775+0000","last_unstale":"2024-03-21T10:36:56.325968+0000","last_undegraded":"2024-03-21T10:36:56.325968+0000","last_fullsized":"2024-03-21T10:36:56.325968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:13:18.058752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618619+0000","last_change":"2024-03-21T10:35:11.177061+0000","last_active":"2024-03-21T10:36:56.618619+0000","last_peered":"2024-03-21T10:36:56.618619+0000","last_clean":"2024-03-21T10:36:56.618619+0000","last_became_active":"2024-03-21T10:35:11.176603+0000","last_became_peered":"2024-03-21T10:35:11.176603+0000","last_unstale":"2024-03-21T10:36:56.618619+0000","last_undegraded":"2024-03-21T10:36:56.618619+0000","last_fullsized":"2024-03-21T10:36:56.618619+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:54.532780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540962+0000","last_change":"2024-03-21T10:35:06.009018+0000","last_active":"2024-03-21T10:36:56.540962+0000","last_peered":"2024-03-21T10:36:56.540962+0000","last_clean":"2024-03-21T10:36:56.540962+0000","last_became_active":"2024-03-21T10:35:06.008638+0000","last_became_peered":"2024-03-21T10:35:06.008638+0000","last_unstale":"2024-03-21T10:36:56.540962+0000","last_undegraded":"2024-03-21T10:36:56.540962+0000","last_fullsized":"2024-03-21T10:36:56.540962+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:42:19.527898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621997+0000","last_change":"2024-03-21T10:35:06.931588+0000","last_active":"2024-03-21T10:36:56.621997+0000","last_peered":"2024-03-21T10:36:56.621997+0000","last_clean":"2024-03-21T10:36:56.621997+0000","last_became_active":"2024-03-21T10:35:06.930707+0000","last_became_peered":"2024-03-21T10:35:06.930707+0000","last_unstale":"2024-03-21T10:36:56.621997+0000","last_undegraded":"2024-03-21T10:36:56.621997+0000","last_fullsized":"2024-03-21T10:36:56.621997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:44:55.862799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540818+0000","last_change":"2024-03-21T10:35:11.229408+0000","last_active":"2024-03-21T10:36:56.540818+0000","last_peered":"2024-03-21T10:36:56.540818+0000","last_clean":"2024-03-21T10:36:56.540818+0000","last_became_active":"2024-03-21T10:35:11.228997+0000","last_became_peered":"2024-03-21T10:35:11.228997+0000","last_unstale":"2024-03-21T10:36:56.540818+0000","last_undegraded":"2024-03-21T10:36:56.540818+0000","last_fullsized":"2024-03-21T10:36:56.540818+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:56:11.517597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619256+0000","last_change":"2024-03-21T10:35:55.388978+0000","last_active":"2024-03-21T10:36:56.619256+0000","last_peered":"2024-03-21T10:36:56.619256+0000","last_clean":"2024-03-21T10:36:56.619256+0000","last_became_active":"2024-03-21T10:35:55.383048+0000","last_became_peered":"2024-03-21T10:35:55.383048+0000","last_unstale":"2024-03-21T10:36:56.619256+0000","last_undegraded":"2024-03-21T10:36:56.619256+0000","last_fullsized":"2024-03-21T10:36:56.619256+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:24:01.933115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312642+0000","last_change":"2024-03-21T10:35:06.908585+0000","last_active":"2024-03-21T10:36:56.312642+0000","last_peered":"2024-03-21T10:36:56.312642+0000","last_clean":"2024-03-21T10:36:56.312642+0000","last_became_active":"2024-03-21T10:35:06.908253+0000","last_became_peered":"2024-03-21T10:35:06.908253+0000","last_unstale":"2024-03-21T10:36:56.312642+0000","last_undegraded":"2024-03-21T10:36:56.312642+0000","last_fullsized":"2024-03-21T10:36:56.312642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:46:15.984450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623323+0000","last_change":"2024-03-21T10:35:11.199455+0000","last_active":"2024-03-21T10:36:56.623323+0000","last_peered":"2024-03-21T10:36:56.623323+0000","last_clean":"2024-03-21T10:36:56.623323+0000","last_became_active":"2024-03-21T10:35:11.197705+0000","last_became_peered":"2024-03-21T10:35:11.197705+0000","last_unstale":"2024-03-21T10:36:56.623323+0000","last_undegraded":"2024-03-21T10:36:56.623323+0000","last_fullsized":"2024-03-21T10:36:56.623323+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:59:42.792764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325149+0000","last_change":"2024-03-21T10:35:06.016152+0000","last_active":"2024-03-21T10:36:56.325149+0000","last_peered":"2024-03-21T10:36:56.325149+0000","last_clean":"2024-03-21T10:36:56.325149+0000","last_became_active":"2024-03-21T10:35:06.015439+0000","last_became_peered":"2024-03-21T10:35:06.015439+0000","last_unstale":"2024-03-21T10:36:56.325149+0000","last_undegraded":"2024-03-21T10:36:56.325149+0000","last_fullsized":"2024-03-21T10:36:56.325149+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:05:23.221826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618846+0000","last_change":"2024-03-21T10:35:06.929241+0000","last_active":"2024-03-21T10:36:56.618846+0000","last_peered":"2024-03-21T10:36:56.618846+0000","last_clean":"2024-03-21T10:36:56.618846+0000","last_became_active":"2024-03-21T10:35:06.928943+0000","last_became_peered":"2024-03-21T10:35:06.928943+0000","last_unstale":"2024-03-21T10:36:56.618846+0000","last_undegraded":"2024-03-21T10:36:56.618846+0000","last_fullsized":"2024-03-21T10:36:56.618846+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:24:10.484540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540706+0000","last_change":"2024-03-21T10:35:11.195929+0000","last_active":"2024-03-21T10:36:56.540706+0000","last_peered":"2024-03-21T10:36:56.540706+0000","last_clean":"2024-03-21T10:36:56.540706+0000","last_became_active":"2024-03-21T10:35:11.195489+0000","last_became_peered":"2024-03-21T10:35:11.195489+0000","last_unstale":"2024-03-21T10:36:56.540706+0000","last_undegraded":"2024-03-21T10:36:56.540706+0000","last_fullsized":"2024-03-21T10:36:56.540706+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:41:15.137917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618718+0000","last_change":"2024-03-21T10:35:06.002127+0000","last_active":"2024-03-21T10:36:56.618718+0000","last_peered":"2024-03-21T10:36:56.618718+0000","last_clean":"2024-03-21T10:36:56.618718+0000","last_became_active":"2024-03-21T10:35:06.001706+0000","last_became_peered":"2024-03-21T10:35:06.001706+0000","last_unstale":"2024-03-21T10:36:56.618718+0000","last_undegraded":"2024-03-21T10:36:56.618718+0000","last_fullsized":"2024-03-21T10:36:56.618718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:26:32.844991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325187+0000","last_change":"2024-03-21T10:35:06.925945+0000","last_active":"2024-03-21T10:36:56.325187+0000","last_peered":"2024-03-21T10:36:56.325187+0000","last_clean":"2024-03-21T10:36:56.325187+0000","last_became_active":"2024-03-21T10:35:06.925551+0000","last_became_peered":"2024-03-21T10:35:06.925551+0000","last_unstale":"2024-03-21T10:36:56.325187+0000","last_undegraded":"2024-03-21T10:36:56.325187+0000","last_fullsized":"2024-03-21T10:36:56.325187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:24:10.212676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540914+0000","last_change":"2024-03-21T10:35:11.229067+0000","last_active":"2024-03-21T10:36:56.540914+0000","last_peered":"2024-03-21T10:36:56.540914+0000","last_clean":"2024-03-21T10:36:56.540914+0000","last_became_active":"2024-03-21T10:35:11.228628+0000","last_became_peered":"2024-03-21T10:35:11.228628+0000","last_unstale":"2024-03-21T10:36:56.540914+0000","last_undegraded":"2024-03-21T10:36:56.540914+0000","last_fullsized":"2024-03-21T10:36:56.540914+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:45:05.473277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618958+0000","last_change":"2024-03-21T10:35:55.384922+0000","last_active":"2024-03-21T10:36:56.618958+0000","last_peered":"2024-03-21T10:36:56.618958+0000","last_clean":"2024-03-21T10:36:56.618958+0000","last_became_active":"2024-03-21T10:35:55.380952+0000","last_became_peered":"2024-03-21T10:35:55.380952+0000","last_unstale":"2024-03-21T10:36:56.618958+0000","last_undegraded":"2024-03-21T10:36:56.618958+0000","last_fullsized":"2024-03-21T10:36:56.618958+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:29:37.873699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619644+0000","last_change":"2024-03-21T10:35:06.928049+0000","last_active":"2024-03-21T10:36:56.619644+0000","last_peered":"2024-03-21T10:36:56.619644+0000","last_clean":"2024-03-21T10:36:56.619644+0000","last_became_active":"2024-03-21T10:35:06.927717+0000","last_became_peered":"2024-03-21T10:35:06.927717+0000","last_unstale":"2024-03-21T10:36:56.619644+0000","last_undegraded":"2024-03-21T10:36:56.619644+0000","last_fullsized":"2024-03-21T10:36:56.619644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:27:47.554668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619381+0000","last_change":"2024-03-21T10:35:11.181090+0000","last_active":"2024-03-21T10:36:56.619381+0000","last_peered":"2024-03-21T10:36:56.619381+0000","last_clean":"2024-03-21T10:36:56.619381+0000","last_became_active":"2024-03-21T10:35:11.180511+0000","last_became_peered":"2024-03-21T10:35:11.180511+0000","last_unstale":"2024-03-21T10:36:56.619381+0000","last_undegraded":"2024-03-21T10:36:56.619381+0000","last_fullsized":"2024-03-21T10:36:56.619381+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:21:51.517845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619595+0000","last_change":"2024-03-21T10:35:06.007828+0000","last_active":"2024-03-21T10:36:56.619595+0000","last_peered":"2024-03-21T10:36:56.619595+0000","last_clean":"2024-03-21T10:36:56.619595+0000","last_became_active":"2024-03-21T10:35:06.007490+0000","last_became_peered":"2024-03-21T10:35:06.007490+0000","last_unstale":"2024-03-21T10:36:56.619595+0000","last_undegraded":"2024-03-21T10:36:56.619595+0000","last_fullsized":"2024-03-21T10:36:56.619595+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:08:18.579308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619840+0000","last_change":"2024-03-21T10:35:06.909313+0000","last_active":"2024-03-21T10:36:56.619840+0000","last_peered":"2024-03-21T10:36:56.619840+0000","last_clean":"2024-03-21T10:36:56.619840+0000","last_became_active":"2024-03-21T10:35:06.908555+0000","last_became_peered":"2024-03-21T10:35:06.908555+0000","last_unstale":"2024-03-21T10:36:56.619840+0000","last_undegraded":"2024-03-21T10:36:56.619840+0000","last_fullsized":"2024-03-21T10:36:56.619840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:09:57.961971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.622374+0000","last_change":"2024-03-21T10:35:11.195996+0000","last_active":"2024-03-21T10:36:56.622374+0000","last_peered":"2024-03-21T10:36:56.622374+0000","last_clean":"2024-03-21T10:36:56.622374+0000","last_became_active":"2024-03-21T10:35:11.195506+0000","last_became_peered":"2024-03-21T10:35:11.195506+0000","last_unstale":"2024-03-21T10:36:56.622374+0000","last_undegraded":"2024-03-21T10:36:56.622374+0000","last_fullsized":"2024-03-21T10:36:56.622374+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:49:33.010899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311586+0000","last_change":"2024-03-21T10:35:06.011498+0000","last_active":"2024-03-21T10:36:56.311586+0000","last_peered":"2024-03-21T10:36:56.311586+0000","last_clean":"2024-03-21T10:36:56.311586+0000","last_became_active":"2024-03-21T10:35:06.011203+0000","last_became_peered":"2024-03-21T10:35:06.011203+0000","last_unstale":"2024-03-21T10:36:56.311586+0000","last_undegraded":"2024-03-21T10:36:56.311586+0000","last_fullsized":"2024-03-21T10:36:56.311586+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:28:11.369740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619130+0000","last_change":"2024-03-21T10:35:06.935514+0000","last_active":"2024-03-21T10:36:56.619130+0000","last_peered":"2024-03-21T10:36:56.619130+0000","last_clean":"2024-03-21T10:36:56.619130+0000","last_became_active":"2024-03-21T10:35:06.935174+0000","last_became_peered":"2024-03-21T10:35:06.935174+0000","last_unstale":"2024-03-21T10:36:56.619130+0000","last_undegraded":"2024-03-21T10:36:56.619130+0000","last_fullsized":"2024-03-21T10:36:56.619130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:35:42.884471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621386+0000","last_change":"2024-03-21T10:36:56.621053+0000","last_active":"2024-03-21T10:36:56.621386+0000","last_peered":"2024-03-21T10:36:56.621386+0000","last_clean":"2024-03-21T10:36:56.621386+0000","last_became_active":"2024-03-21T10:36:56.620688+0000","last_became_peered":"2024-03-21T10:36:56.620688+0000","last_unstale":"2024-03-21T10:36:56.621386+0000","last_undegraded":"2024-03-21T10:36:56.621386+0000","last_fullsized":"2024-03-21T10:36:56.621386+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:50:02.121787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313022+0000","last_change":"2024-03-21T10:35:06.007216+0000","last_active":"2024-03-21T10:36:56.313022+0000","last_peered":"2024-03-21T10:36:56.313022+0000","last_clean":"2024-03-21T10:36:56.313022+0000","last_became_active":"2024-03-21T10:35:06.006883+0000","last_became_peered":"2024-03-21T10:35:06.006883+0000","last_unstale":"2024-03-21T10:36:56.313022+0000","last_undegraded":"2024-03-21T10:36:56.313022+0000","last_fullsized":"2024-03-21T10:36:56.313022+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:46:45.310919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619803+0000","last_change":"2024-03-21T10:35:06.906017+0000","last_active":"2024-03-21T10:36:56.619803+0000","last_peered":"2024-03-21T10:36:56.619803+0000","last_clean":"2024-03-21T10:36:56.619803+0000","last_became_active":"2024-03-21T10:35:06.905648+0000","last_became_peered":"2024-03-21T10:35:06.905648+0000","last_unstale":"2024-03-21T10:36:56.619803+0000","last_undegraded":"2024-03-21T10:36:56.619803+0000","last_fullsized":"2024-03-21T10:36:56.619803+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:56:50.380146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618954+0000","last_change":"2024-03-21T10:35:06.905186+0000","last_active":"2024-03-21T10:36:56.618954+0000","last_peered":"2024-03-21T10:36:56.618954+0000","last_clean":"2024-03-21T10:36:56.618954+0000","last_became_active":"2024-03-21T10:35:06.904835+0000","last_became_peered":"2024-03-21T10:35:06.904835+0000","last_unstale":"2024-03-21T10:36:56.618954+0000","last_undegraded":"2024-03-21T10:36:56.618954+0000","last_fullsized":"2024-03-21T10:36:56.618954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:53:19.293437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619036+0000","last_change":"2024-03-21T10:35:06.903318+0000","last_active":"2024-03-21T10:36:56.619036+0000","last_peered":"2024-03-21T10:36:56.619036+0000","last_clean":"2024-03-21T10:36:56.619036+0000","last_became_active":"2024-03-21T10:35:06.902994+0000","last_became_peered":"2024-03-21T10:35:06.902994+0000","last_unstale":"2024-03-21T10:36:56.619036+0000","last_undegraded":"2024-03-21T10:36:56.619036+0000","last_fullsized":"2024-03-21T10:36:56.619036+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:21:27.412565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618679+0000","last_change":"2024-03-21T10:35:06.934582+0000","last_active":"2024-03-21T10:36:56.618679+0000","last_peered":"2024-03-21T10:36:56.618679+0000","last_clean":"2024-03-21T10:36:56.618679+0000","last_became_active":"2024-03-21T10:35:06.934311+0000","last_became_peered":"2024-03-21T10:35:06.934311+0000","last_unstale":"2024-03-21T10:36:56.618679+0000","last_undegraded":"2024-03-21T10:36:56.618679+0000","last_fullsized":"2024-03-21T10:36:56.618679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:05:30.785830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.622973+0000","last_change":"2024-03-21T10:35:06.944221+0000","last_active":"2024-03-21T10:36:56.622973+0000","last_peered":"2024-03-21T10:36:56.622973+0000","last_clean":"2024-03-21T10:36:56.622973+0000","last_became_active":"2024-03-21T10:35:06.943732+0000","last_became_peered":"2024-03-21T10:35:06.943732+0000","last_unstale":"2024-03-21T10:36:56.622973+0000","last_undegraded":"2024-03-21T10:36:56.622973+0000","last_fullsized":"2024-03-21T10:36:56.622973+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:33:03.653160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313346+0000","last_change":"2024-03-21T10:35:06.940832+0000","last_active":"2024-03-21T10:36:56.313346+0000","last_peered":"2024-03-21T10:36:56.313346+0000","last_clean":"2024-03-21T10:36:56.313346+0000","last_became_active":"2024-03-21T10:35:06.939565+0000","last_became_peered":"2024-03-21T10:35:06.939565+0000","last_unstale":"2024-03-21T10:36:56.313346+0000","last_undegraded":"2024-03-21T10:36:56.313346+0000","last_fullsized":"2024-03-21T10:36:56.313346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:57:27.743704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619590+0000","last_change":"2024-03-21T10:35:06.907753+0000","last_active":"2024-03-21T10:36:56.619590+0000","last_peered":"2024-03-21T10:36:56.619590+0000","last_clean":"2024-03-21T10:36:56.619590+0000","last_became_active":"2024-03-21T10:35:06.907432+0000","last_became_peered":"2024-03-21T10:35:06.907432+0000","last_unstale":"2024-03-21T10:36:56.619590+0000","last_undegraded":"2024-03-21T10:36:56.619590+0000","last_fullsized":"2024-03-21T10:36:56.619590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:31:02.528855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619670+0000","last_change":"2024-03-21T10:35:06.908597+0000","last_active":"2024-03-21T10:36:56.619670+0000","last_peered":"2024-03-21T10:36:56.619670+0000","last_clean":"2024-03-21T10:36:56.619670+0000","last_became_active":"2024-03-21T10:35:06.908282+0000","last_became_peered":"2024-03-21T10:35:06.908282+0000","last_unstale":"2024-03-21T10:36:56.619670+0000","last_undegraded":"2024-03-21T10:36:56.619670+0000","last_fullsized":"2024-03-21T10:36:56.619670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:42:18.221045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.326006+0000","last_change":"2024-03-21T10:35:06.931015+0000","last_active":"2024-03-21T10:36:56.326006+0000","last_peered":"2024-03-21T10:36:56.326006+0000","last_clean":"2024-03-21T10:36:56.326006+0000","last_became_active":"2024-03-21T10:35:06.930209+0000","last_became_peered":"2024-03-21T10:35:06.930209+0000","last_unstale":"2024-03-21T10:36:56.326006+0000","last_undegraded":"2024-03-21T10:36:56.326006+0000","last_fullsized":"2024-03-21T10:36:56.326006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:42:15.277874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311263+0000","last_change":"2024-03-21T10:35:06.930070+0000","last_active":"2024-03-21T10:36:56.311263+0000","last_peered":"2024-03-21T10:36:56.311263+0000","last_clean":"2024-03-21T10:36:56.311263+0000","last_became_active":"2024-03-21T10:35:06.929773+0000","last_became_peered":"2024-03-21T10:35:06.929773+0000","last_unstale":"2024-03-21T10:36:56.311263+0000","last_undegraded":"2024-03-21T10:36:56.311263+0000","last_fullsized":"2024-03-21T10:36:56.311263+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:17:25.176666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325363+0000","last_change":"2024-03-21T10:35:06.929575+0000","last_active":"2024-03-21T10:36:56.325363+0000","last_peered":"2024-03-21T10:36:56.325363+0000","last_clean":"2024-03-21T10:36:56.325363+0000","last_became_active":"2024-03-21T10:35:06.929240+0000","last_became_peered":"2024-03-21T10:35:06.929240+0000","last_unstale":"2024-03-21T10:36:56.325363+0000","last_undegraded":"2024-03-21T10:36:56.325363+0000","last_fullsized":"2024-03-21T10:36:56.325363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:03:40.697640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619625+0000","last_change":"2024-03-21T10:35:06.934693+0000","last_active":"2024-03-21T10:36:56.619625+0000","last_peered":"2024-03-21T10:36:56.619625+0000","last_clean":"2024-03-21T10:36:56.619625+0000","last_became_active":"2024-03-21T10:35:06.934361+0000","last_became_peered":"2024-03-21T10:35:06.934361+0000","last_unstale":"2024-03-21T10:36:56.619625+0000","last_undegraded":"2024-03-21T10:36:56.619625+0000","last_fullsized":"2024-03-21T10:36:56.619625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:14:00.954370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312500+0000","last_change":"2024-03-21T10:35:06.941095+0000","last_active":"2024-03-21T10:36:56.312500+0000","last_peered":"2024-03-21T10:36:56.312500+0000","last_clean":"2024-03-21T10:36:56.312500+0000","last_became_active":"2024-03-21T10:35:06.939348+0000","last_became_peered":"2024-03-21T10:35:06.939348+0000","last_unstale":"2024-03-21T10:36:56.312500+0000","last_undegraded":"2024-03-21T10:36:56.312500+0000","last_fullsized":"2024-03-21T10:36:56.312500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:31:43.783592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619193+0000","last_change":"2024-03-21T10:35:06.908340+0000","last_active":"2024-03-21T10:36:56.619193+0000","last_peered":"2024-03-21T10:36:56.619193+0000","last_clean":"2024-03-21T10:36:56.619193+0000","last_became_active":"2024-03-21T10:35:06.908043+0000","last_became_peered":"2024-03-21T10:35:06.908043+0000","last_unstale":"2024-03-21T10:36:56.619193+0000","last_undegraded":"2024-03-21T10:36:56.619193+0000","last_fullsized":"2024-03-21T10:36:56.619193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:50:33.167396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282863,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69209,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":957,"num_read_kb":553,"num_write":180,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":207627,"data_compressed":5487,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":154,"ondisk_log_size":154,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383733,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1019904,"data_stored":519397,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645370,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32312,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712840,"statfs":{"total":95995035648,"available":95961948160,"internally_reserved":0,"allocated":1163264,"data_stored":1171732,"data_compressed":64463,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808895,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":523729,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070535,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1036288,"data_stored":519553,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332172,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1085440,"data_stored":1115909,"data_compressed":62634,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495698,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":514809,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574959,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1081344,"data_stored":579480,"data_compressed":28489,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738484,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":516197,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411432,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1105920,"data_stored":581362,"data_compressed":28489,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182498,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":515003,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920860,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":519765,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757335,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1093632,"data_stored":1111191,"data_compressed":62634,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34331,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34331,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34331,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":64761,"data_compressed":1829,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":66643,"data_compressed":1829,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":66645,"data_compressed":1829,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T10:38:18.531 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-21T10:38:18.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.793+0000 7f3073200640 1 -- 172.21.15.31:0/2810052993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3074077c40 msgr2=0x7f30740784e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:18.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.793+0000 7f3073200640 1 --2- 172.21.15.31:0/2810052993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3074077c40 0x7f30740784e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f306c009be0 tx=0x7f306c02bef0 comp rx=0 tx=0).stop 2024-03-21T10:38:18.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.793+0000 7f3073200640 1 -- 172.21.15.31:0/2810052993 shutdown_connections 2024-03-21T10:38:18.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.793+0000 7f3073200640 1 --2- 172.21.15.31:0/2810052993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3074077c40 0x7f30740784e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:18.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.793+0000 7f3073200640 1 --2- 172.21.15.31:0/2810052993 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30740772a0 0x7f3074077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:18.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.793+0000 7f3073200640 1 --2- 172.21.15.31:0/2810052993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3074076060 0x7f3074076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:18.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.793+0000 7f3073200640 1 -- 172.21.15.31:0/2810052993 >> 172.21.15.31:0/2810052993 conn(0x7f30740fed50 msgr2=0x7f3074101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:18.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.793+0000 7f3073200640 1 -- 172.21.15.31:0/2810052993 shutdown_connections 2024-03-21T10:38:18.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.793+0000 7f3073200640 1 -- 172.21.15.31:0/2810052993 wait complete. 2024-03-21T10:38:18.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.794+0000 7f3073200640 1 Processor -- start 2024-03-21T10:38:18.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.794+0000 7f3073200640 1 -- start start 2024-03-21T10:38:18.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.795+0000 7f3073200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3074076060 0x7f307419de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:18.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.795+0000 7f3073200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30740772a0 0x7f307419e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:18.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.795+0000 7f3073200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3074077c40 0x7f30741a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:18.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.795+0000 7f3073200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3074113890 con 0x7f30740772a0 2024-03-21T10:38:18.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.795+0000 7f3073200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3074113710 con 0x7f3074077c40 2024-03-21T10:38:18.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.795+0000 7f3073200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3074113a10 con 0x7f3074076060 2024-03-21T10:38:18.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.795+0000 7f3071e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3074076060 0x7f307419de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:18.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.795+0000 7f3072800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3074077c40 0x7f30741a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:18.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.795+0000 7f3072800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3074077c40 0x7f30741a53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46050/0 (socket says 172.21.15.31:46050) 2024-03-21T10:38:18.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.795+0000 7f3071e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3074076060 0x7f307419de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53510/0 (socket says 172.21.15.31:53510) 2024-03-21T10:38:18.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.795+0000 7f3072800640 1 -- 172.21.15.31:0/3923028762 learned_addr learned my addr 172.21.15.31:0/3923028762 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:18.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.795+0000 7f3069400640 1 --2- 172.21.15.31:0/3923028762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30740772a0 0x7f307419e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:18.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.795+0000 7f3072800640 1 -- 172.21.15.31:0/3923028762 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3074076060 msgr2=0x7f307419de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:18.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.796+0000 7f3072800640 1 --2- 172.21.15.31:0/3923028762 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3074076060 0x7f307419de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:18.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.796+0000 7f3072800640 1 -- 172.21.15.31:0/3923028762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30740772a0 msgr2=0x7f307419e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:18.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.796+0000 7f3072800640 1 --2- 172.21.15.31:0/3923028762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30740772a0 0x7f307419e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:18.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.796+0000 7f3072800640 1 -- 172.21.15.31:0/3923028762 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30741a5a20 con 0x7f3074077c40 2024-03-21T10:38:18.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.796+0000 7f3071e00640 1 --2- 172.21.15.31:0/3923028762 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3074076060 0x7f307419de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:18.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.796+0000 7f3072800640 1 --2- 172.21.15.31:0/3923028762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3074077c40 0x7f30741a53b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f306c036040 tx=0x7f306c0337b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:18.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.796+0000 7f306b400640 1 -- 172.21.15.31:0/3923028762 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306c043020 con 0x7f3074077c40 2024-03-21T10:38:18.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.796+0000 7f306b400640 1 -- 172.21.15.31:0/3923028762 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f306c003b80 con 0x7f3074077c40 2024-03-21T10:38:18.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.796+0000 7f306b400640 1 -- 172.21.15.31:0/3923028762 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306c034690 con 0x7f3074077c40 2024-03-21T10:38:18.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.796+0000 7f3073200640 1 -- 172.21.15.31:0/3923028762 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30741a5cb0 con 0x7f3074077c40 2024-03-21T10:38:18.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.796+0000 7f3073200640 1 -- 172.21.15.31:0/3923028762 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30741a61f0 con 0x7f3074077c40 2024-03-21T10:38:18.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.797+0000 7f3073200640 1 -- 172.21.15.31:0/3923028762 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3074110130 con 0x7f3074077c40 2024-03-21T10:38:18.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.799+0000 7f306b400640 1 -- 172.21.15.31:0/3923028762 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f306c046020 con 0x7f3074077c40 2024-03-21T10:38:18.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.800+0000 7f306b400640 1 --2- 172.21.15.31:0/3923028762 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f30580a2f90 0x7f30580a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:18.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.800+0000 7f306b400640 1 -- 172.21.15.31:0/3923028762 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f306c0f1db0 con 0x7f3074077c40 2024-03-21T10:38:18.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.801+0000 7f3071e00640 1 --2- 172.21.15.31:0/3923028762 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f30580a2f90 0x7f30580a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:18.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.801+0000 7f3071e00640 1 --2- 172.21.15.31:0/3923028762 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f30580a2f90 0x7f30580a5450 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3064005fd0 tx=0x7f3064009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:18.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:18.804+0000 7f306b400640 1 -- 172.21.15.31:0/3923028762 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f306c04d020 con 0x7f3074077c40 2024-03-21T10:38:18.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:18 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1744317026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:38:19.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.016+0000 7f3073200640 1 -- 172.21.15.31:0/3923028762 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3074061ae0 con 0x7f30580a2f90 2024-03-21T10:38:19.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.037+0000 7f306b400640 1 -- 172.21.15.31:0/3923028762 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340815 (secure 0 0 0) 0x7f3074061ae0 con 0x7f30580a2f90 2024-03-21T10:38:19.037 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:38:19.043 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-03-21T10:38:19.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.047+0000 7f3073200640 1 -- 172.21.15.31:0/3923028762 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f30580a2f90 msgr2=0x7f30580a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:19.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.047+0000 7f3073200640 1 --2- 172.21.15.31:0/3923028762 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f30580a2f90 0x7f30580a5450 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3064005fd0 tx=0x7f3064009260 comp rx=0 tx=0).stop 2024-03-21T10:38:19.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.047+0000 7f3073200640 1 -- 172.21.15.31:0/3923028762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3074077c40 msgr2=0x7f30741a53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:19.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.047+0000 7f3073200640 1 --2- 172.21.15.31:0/3923028762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3074077c40 0x7f30741a53b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f306c036040 tx=0x7f306c0337b0 comp rx=0 tx=0).stop 2024-03-21T10:38:19.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.047+0000 7f3073200640 1 -- 172.21.15.31:0/3923028762 shutdown_connections 2024-03-21T10:38:19.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.047+0000 7f3073200640 1 --2- 172.21.15.31:0/3923028762 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f30580a2f90 0x7f30580a5450 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:19.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.047+0000 7f3073200640 1 --2- 172.21.15.31:0/3923028762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3074077c40 0x7f30741a53b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:19.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.047+0000 7f3073200640 1 --2- 172.21.15.31:0/3923028762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30740772a0 0x7f307419e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:19.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.047+0000 7f3073200640 1 --2- 172.21.15.31:0/3923028762 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3074076060 0x7f307419de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:19.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.047+0000 7f3073200640 1 -- 172.21.15.31:0/3923028762 >> 172.21.15.31:0/3923028762 conn(0x7f30740fed50 msgr2=0x7f3074100850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:19.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.047+0000 7f3073200640 1 -- 172.21.15.31:0/3923028762 shutdown_connections 2024-03-21T10:38:19.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.047+0000 7f3073200640 1 -- 172.21.15.31:0/3923028762 wait complete. 2024-03-21T10:38:19.066 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2024-03-21T10:38:18.076535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659811,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":1093,"num_read_kb":672,"num_write":363,"num_write_kb":3069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":793,"ondisk_log_size":793,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369956,"kb_used_data":12440,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571868,"statfs":{"total":1151940427776,"available":1151561592832,"internally_reserved":0,"allocated":12738560,"data_stored":8188127,"data_compressed":433329,"data_compressed_allocated":589824,"data_compressed_original":2678784,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":612,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":32,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005529"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619666+0000","last_change":"2024-03-21T10:35:06.923544+0000","last_active":"2024-03-21T10:36:56.619666+0000","last_peered":"2024-03-21T10:36:56.619666+0000","last_clean":"2024-03-21T10:36:56.619666+0000","last_became_active":"2024-03-21T10:35:06.923254+0000","last_became_peered":"2024-03-21T10:35:06.923254+0000","last_unstale":"2024-03-21T10:36:56.619666+0000","last_undegraded":"2024-03-21T10:36:56.619666+0000","last_fullsized":"2024-03-21T10:36:56.619666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:38:14.573590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.622736+0000","last_change":"2024-03-21T10:35:06.929242+0000","last_active":"2024-03-21T10:36:56.622736+0000","last_peered":"2024-03-21T10:36:56.622736+0000","last_clean":"2024-03-21T10:36:56.622736+0000","last_became_active":"2024-03-21T10:35:06.927078+0000","last_became_peered":"2024-03-21T10:35:06.927078+0000","last_unstale":"2024-03-21T10:36:56.622736+0000","last_undegraded":"2024-03-21T10:36:56.622736+0000","last_fullsized":"2024-03-21T10:36:56.622736+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:27:10.347945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621857+0000","last_change":"2024-03-21T10:35:06.927792+0000","last_active":"2024-03-21T10:36:56.621857+0000","last_peered":"2024-03-21T10:36:56.621857+0000","last_clean":"2024-03-21T10:36:56.621857+0000","last_became_active":"2024-03-21T10:35:06.927386+0000","last_became_peered":"2024-03-21T10:35:06.927386+0000","last_unstale":"2024-03-21T10:36:56.621857+0000","last_undegraded":"2024-03-21T10:36:56.621857+0000","last_fullsized":"2024-03-21T10:36:56.621857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:53:51.310283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540864+0000","last_change":"2024-03-21T10:35:06.908901+0000","last_active":"2024-03-21T10:36:56.540864+0000","last_peered":"2024-03-21T10:36:56.540864+0000","last_clean":"2024-03-21T10:36:56.540864+0000","last_became_active":"2024-03-21T10:35:06.908586+0000","last_became_peered":"2024-03-21T10:35:06.908586+0000","last_unstale":"2024-03-21T10:36:56.540864+0000","last_undegraded":"2024-03-21T10:36:56.540864+0000","last_fullsized":"2024-03-21T10:36:56.540864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:25:57.387769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312725+0000","last_change":"2024-03-21T10:35:06.932537+0000","last_active":"2024-03-21T10:36:56.312725+0000","last_peered":"2024-03-21T10:36:56.312725+0000","last_clean":"2024-03-21T10:36:56.312725+0000","last_became_active":"2024-03-21T10:35:06.932185+0000","last_became_peered":"2024-03-21T10:35:06.932185+0000","last_unstale":"2024-03-21T10:36:56.312725+0000","last_undegraded":"2024-03-21T10:36:56.312725+0000","last_fullsized":"2024-03-21T10:36:56.312725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:30:25.127073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311445+0000","last_change":"2024-03-21T10:35:06.922765+0000","last_active":"2024-03-21T10:36:56.311445+0000","last_peered":"2024-03-21T10:36:56.311445+0000","last_clean":"2024-03-21T10:36:56.311445+0000","last_became_active":"2024-03-21T10:35:06.922447+0000","last_became_peered":"2024-03-21T10:35:06.922447+0000","last_unstale":"2024-03-21T10:36:56.311445+0000","last_undegraded":"2024-03-21T10:36:56.311445+0000","last_fullsized":"2024-03-21T10:36:56.311445+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:37:33.310735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619036+0000","last_change":"2024-03-21T10:35:06.896777+0000","last_active":"2024-03-21T10:36:56.619036+0000","last_peered":"2024-03-21T10:36:56.619036+0000","last_clean":"2024-03-21T10:36:56.619036+0000","last_became_active":"2024-03-21T10:35:06.896544+0000","last_became_peered":"2024-03-21T10:35:06.896544+0000","last_unstale":"2024-03-21T10:36:56.619036+0000","last_undegraded":"2024-03-21T10:36:56.619036+0000","last_fullsized":"2024-03-21T10:36:56.619036+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:34:05.000661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325012+0000","last_change":"2024-03-21T10:35:06.923294+0000","last_active":"2024-03-21T10:36:56.325012+0000","last_peered":"2024-03-21T10:36:56.325012+0000","last_clean":"2024-03-21T10:36:56.325012+0000","last_became_active":"2024-03-21T10:35:06.921320+0000","last_became_peered":"2024-03-21T10:35:06.921320+0000","last_unstale":"2024-03-21T10:36:56.325012+0000","last_undegraded":"2024-03-21T10:36:56.325012+0000","last_fullsized":"2024-03-21T10:36:56.325012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:33:53.855908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619843+0000","last_change":"2024-03-21T10:35:06.912977+0000","last_active":"2024-03-21T10:36:56.619843+0000","last_peered":"2024-03-21T10:36:56.619843+0000","last_clean":"2024-03-21T10:36:56.619843+0000","last_became_active":"2024-03-21T10:35:06.912626+0000","last_became_peered":"2024-03-21T10:35:06.912626+0000","last_unstale":"2024-03-21T10:36:56.619843+0000","last_undegraded":"2024-03-21T10:36:56.619843+0000","last_fullsized":"2024-03-21T10:36:56.619843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:42:13.472454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311923+0000","last_change":"2024-03-21T10:35:06.910252+0000","last_active":"2024-03-21T10:36:56.311923+0000","last_peered":"2024-03-21T10:36:56.311923+0000","last_clean":"2024-03-21T10:36:56.311923+0000","last_became_active":"2024-03-21T10:35:06.909924+0000","last_became_peered":"2024-03-21T10:35:06.909924+0000","last_unstale":"2024-03-21T10:36:56.311923+0000","last_undegraded":"2024-03-21T10:36:56.311923+0000","last_fullsized":"2024-03-21T10:36:56.311923+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:43:15.527349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619732+0000","last_change":"2024-03-21T10:35:06.909569+0000","last_active":"2024-03-21T10:36:56.619732+0000","last_peered":"2024-03-21T10:36:56.619732+0000","last_clean":"2024-03-21T10:36:56.619732+0000","last_became_active":"2024-03-21T10:35:06.908982+0000","last_became_peered":"2024-03-21T10:35:06.908982+0000","last_unstale":"2024-03-21T10:36:56.619732+0000","last_undegraded":"2024-03-21T10:36:56.619732+0000","last_fullsized":"2024-03-21T10:36:56.619732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:13:56.020281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540823+0000","last_change":"2024-03-21T10:35:06.913861+0000","last_active":"2024-03-21T10:36:56.540823+0000","last_peered":"2024-03-21T10:36:56.540823+0000","last_clean":"2024-03-21T10:36:56.540823+0000","last_became_active":"2024-03-21T10:35:06.913490+0000","last_became_peered":"2024-03-21T10:35:06.913490+0000","last_unstale":"2024-03-21T10:36:56.540823+0000","last_undegraded":"2024-03-21T10:36:56.540823+0000","last_fullsized":"2024-03-21T10:36:56.540823+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:24:18.573735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312584+0000","last_change":"2024-03-21T10:35:06.934537+0000","last_active":"2024-03-21T10:36:56.312584+0000","last_peered":"2024-03-21T10:36:56.312584+0000","last_clean":"2024-03-21T10:36:56.312584+0000","last_became_active":"2024-03-21T10:35:06.932951+0000","last_became_peered":"2024-03-21T10:35:06.932951+0000","last_unstale":"2024-03-21T10:36:56.312584+0000","last_undegraded":"2024-03-21T10:36:56.312584+0000","last_fullsized":"2024-03-21T10:36:56.312584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:43:17.748434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621260+0000","last_change":"2024-03-21T10:35:06.925495+0000","last_active":"2024-03-21T10:36:56.621260+0000","last_peered":"2024-03-21T10:36:56.621260+0000","last_clean":"2024-03-21T10:36:56.621260+0000","last_became_active":"2024-03-21T10:35:06.924326+0000","last_became_peered":"2024-03-21T10:35:06.924326+0000","last_unstale":"2024-03-21T10:36:56.621260+0000","last_undegraded":"2024-03-21T10:36:56.621260+0000","last_fullsized":"2024-03-21T10:36:56.621260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:27:19.598213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623885+0000","last_change":"2024-03-21T10:35:06.923695+0000","last_active":"2024-03-21T10:36:56.623885+0000","last_peered":"2024-03-21T10:36:56.623885+0000","last_clean":"2024-03-21T10:36:56.623885+0000","last_became_active":"2024-03-21T10:35:06.919114+0000","last_became_peered":"2024-03-21T10:35:06.919114+0000","last_unstale":"2024-03-21T10:36:56.623885+0000","last_undegraded":"2024-03-21T10:36:56.623885+0000","last_fullsized":"2024-03-21T10:36:56.623885+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:42:49.179653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619136+0000","last_change":"2024-03-21T10:35:06.930225+0000","last_active":"2024-03-21T10:36:56.619136+0000","last_peered":"2024-03-21T10:36:56.619136+0000","last_clean":"2024-03-21T10:36:56.619136+0000","last_became_active":"2024-03-21T10:35:06.929905+0000","last_became_peered":"2024-03-21T10:35:06.929905+0000","last_unstale":"2024-03-21T10:36:56.619136+0000","last_undegraded":"2024-03-21T10:36:56.619136+0000","last_fullsized":"2024-03-21T10:36:56.619136+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:21:07.231703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619516+0000","last_change":"2024-03-21T10:35:06.923498+0000","last_active":"2024-03-21T10:36:56.619516+0000","last_peered":"2024-03-21T10:36:56.619516+0000","last_clean":"2024-03-21T10:36:56.619516+0000","last_became_active":"2024-03-21T10:35:06.923242+0000","last_became_peered":"2024-03-21T10:35:06.923242+0000","last_unstale":"2024-03-21T10:36:56.619516+0000","last_undegraded":"2024-03-21T10:36:56.619516+0000","last_fullsized":"2024-03-21T10:36:56.619516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:42:04.309701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623901+0000","last_change":"2024-03-21T10:35:06.918314+0000","last_active":"2024-03-21T10:36:56.623901+0000","last_peered":"2024-03-21T10:36:56.623901+0000","last_clean":"2024-03-21T10:36:56.623901+0000","last_became_active":"2024-03-21T10:35:06.917865+0000","last_became_peered":"2024-03-21T10:35:06.917865+0000","last_unstale":"2024-03-21T10:36:56.623901+0000","last_undegraded":"2024-03-21T10:36:56.623901+0000","last_fullsized":"2024-03-21T10:36:56.623901+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:53:48.536341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619301+0000","last_change":"2024-03-21T10:35:06.936234+0000","last_active":"2024-03-21T10:36:56.619301+0000","last_peered":"2024-03-21T10:36:56.619301+0000","last_clean":"2024-03-21T10:36:56.619301+0000","last_became_active":"2024-03-21T10:35:06.935958+0000","last_became_peered":"2024-03-21T10:35:06.935958+0000","last_unstale":"2024-03-21T10:36:56.619301+0000","last_undegraded":"2024-03-21T10:36:56.619301+0000","last_fullsized":"2024-03-21T10:36:56.619301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:52:43.509671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618623+0000","last_change":"2024-03-21T10:35:11.198276+0000","last_active":"2024-03-21T10:36:56.618623+0000","last_peered":"2024-03-21T10:36:56.618623+0000","last_clean":"2024-03-21T10:36:56.618623+0000","last_became_active":"2024-03-21T10:35:11.197899+0000","last_became_peered":"2024-03-21T10:35:11.197899+0000","last_unstale":"2024-03-21T10:36:56.618623+0000","last_undegraded":"2024-03-21T10:36:56.618623+0000","last_fullsized":"2024-03-21T10:36:56.618623+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:17:30.499932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623544+0000","last_change":"2024-03-21T10:35:06.010647+0000","last_active":"2024-03-21T10:36:56.623544+0000","last_peered":"2024-03-21T10:36:56.623544+0000","last_clean":"2024-03-21T10:36:56.623544+0000","last_became_active":"2024-03-21T10:35:06.010239+0000","last_became_peered":"2024-03-21T10:35:06.010239+0000","last_unstale":"2024-03-21T10:36:56.623544+0000","last_undegraded":"2024-03-21T10:36:56.623544+0000","last_fullsized":"2024-03-21T10:36:56.623544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:35:53.324044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325857+0000","last_change":"2024-03-21T10:35:06.926398+0000","last_active":"2024-03-21T10:36:56.325857+0000","last_peered":"2024-03-21T10:36:56.325857+0000","last_clean":"2024-03-21T10:36:56.325857+0000","last_became_active":"2024-03-21T10:35:06.925904+0000","last_became_peered":"2024-03-21T10:35:06.925904+0000","last_unstale":"2024-03-21T10:36:56.325857+0000","last_undegraded":"2024-03-21T10:36:56.325857+0000","last_fullsized":"2024-03-21T10:36:56.325857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:37:52.811604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540507+0000","last_change":"2024-03-21T10:35:55.503266+0000","last_active":"2024-03-21T10:36:56.540507+0000","last_peered":"2024-03-21T10:36:56.540507+0000","last_clean":"2024-03-21T10:36:56.540507+0000","last_became_active":"2024-03-21T10:35:55.502694+0000","last_became_peered":"2024-03-21T10:35:55.502694+0000","last_unstale":"2024-03-21T10:36:56.540507+0000","last_undegraded":"2024-03-21T10:36:56.540507+0000","last_fullsized":"2024-03-21T10:36:56.540507+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:56:44.480667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325780+0000","last_change":"2024-03-21T10:35:06.022750+0000","last_active":"2024-03-21T10:36:56.325780+0000","last_peered":"2024-03-21T10:36:56.325780+0000","last_clean":"2024-03-21T10:36:56.325780+0000","last_became_active":"2024-03-21T10:35:06.022093+0000","last_became_peered":"2024-03-21T10:35:06.022093+0000","last_unstale":"2024-03-21T10:36:56.325780+0000","last_undegraded":"2024-03-21T10:36:56.325780+0000","last_fullsized":"2024-03-21T10:36:56.325780+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:53:28.458486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.624188+0000","last_change":"2024-03-21T10:35:06.927594+0000","last_active":"2024-03-21T10:36:56.624188+0000","last_peered":"2024-03-21T10:36:56.624188+0000","last_clean":"2024-03-21T10:36:56.624188+0000","last_became_active":"2024-03-21T10:35:06.926868+0000","last_became_peered":"2024-03-21T10:35:06.926868+0000","last_unstale":"2024-03-21T10:36:56.624188+0000","last_undegraded":"2024-03-21T10:36:56.624188+0000","last_fullsized":"2024-03-21T10:36:56.624188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:07:33.127465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540607+0000","last_change":"2024-03-21T10:35:11.193475+0000","last_active":"2024-03-21T10:36:56.540607+0000","last_peered":"2024-03-21T10:36:56.540607+0000","last_clean":"2024-03-21T10:36:56.540607+0000","last_became_active":"2024-03-21T10:35:11.193054+0000","last_became_peered":"2024-03-21T10:35:11.193054+0000","last_unstale":"2024-03-21T10:36:56.540607+0000","last_undegraded":"2024-03-21T10:36:56.540607+0000","last_fullsized":"2024-03-21T10:36:56.540607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:32:36.564147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313127+0000","last_change":"2024-03-21T10:35:06.004257+0000","last_active":"2024-03-21T10:36:56.313127+0000","last_peered":"2024-03-21T10:36:56.313127+0000","last_clean":"2024-03-21T10:36:56.313127+0000","last_became_active":"2024-03-21T10:35:06.003936+0000","last_became_peered":"2024-03-21T10:35:06.003936+0000","last_unstale":"2024-03-21T10:36:56.313127+0000","last_undegraded":"2024-03-21T10:36:56.313127+0000","last_fullsized":"2024-03-21T10:36:56.313127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:48:48.608979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.620162+0000","last_change":"2024-03-21T10:35:06.941681+0000","last_active":"2024-03-21T10:36:56.620162+0000","last_peered":"2024-03-21T10:36:56.620162+0000","last_clean":"2024-03-21T10:36:56.620162+0000","last_became_active":"2024-03-21T10:35:06.941314+0000","last_became_peered":"2024-03-21T10:35:06.941314+0000","last_unstale":"2024-03-21T10:36:56.620162+0000","last_undegraded":"2024-03-21T10:36:56.620162+0000","last_fullsized":"2024-03-21T10:36:56.620162+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:46:09.948922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540509+0000","last_change":"2024-03-21T10:35:11.188472+0000","last_active":"2024-03-21T10:36:56.540509+0000","last_peered":"2024-03-21T10:36:56.540509+0000","last_clean":"2024-03-21T10:36:56.540509+0000","last_became_active":"2024-03-21T10:35:11.188072+0000","last_became_peered":"2024-03-21T10:35:11.188072+0000","last_unstale":"2024-03-21T10:36:56.540509+0000","last_undegraded":"2024-03-21T10:36:56.540509+0000","last_fullsized":"2024-03-21T10:36:56.540509+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:04:16.911334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313102+0000","last_change":"2024-03-21T10:35:05.998493+0000","last_active":"2024-03-21T10:36:56.313102+0000","last_peered":"2024-03-21T10:36:56.313102+0000","last_clean":"2024-03-21T10:36:56.313102+0000","last_became_active":"2024-03-21T10:35:05.998134+0000","last_became_peered":"2024-03-21T10:35:05.998134+0000","last_unstale":"2024-03-21T10:36:56.313102+0000","last_undegraded":"2024-03-21T10:36:56.313102+0000","last_fullsized":"2024-03-21T10:36:56.313102+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:05:32.033328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311467+0000","last_change":"2024-03-21T10:35:06.912973+0000","last_active":"2024-03-21T10:36:56.311467+0000","last_peered":"2024-03-21T10:36:56.311467+0000","last_clean":"2024-03-21T10:36:56.311467+0000","last_became_active":"2024-03-21T10:35:06.912382+0000","last_became_peered":"2024-03-21T10:35:06.912382+0000","last_unstale":"2024-03-21T10:36:56.311467+0000","last_undegraded":"2024-03-21T10:36:56.311467+0000","last_fullsized":"2024-03-21T10:36:56.311467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:13:23.757805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619140+0000","last_change":"2024-03-21T10:35:11.180806+0000","last_active":"2024-03-21T10:36:56.619140+0000","last_peered":"2024-03-21T10:36:56.619140+0000","last_clean":"2024-03-21T10:36:56.619140+0000","last_became_active":"2024-03-21T10:35:11.180303+0000","last_became_peered":"2024-03-21T10:35:11.180303+0000","last_unstale":"2024-03-21T10:36:56.619140+0000","last_undegraded":"2024-03-21T10:36:56.619140+0000","last_fullsized":"2024-03-21T10:36:56.619140+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:27:11.719654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":196,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:38:13.155079+0000","last_change":"2024-03-21T10:35:06.004555+0000","last_active":"2024-03-21T10:38:13.155079+0000","last_peered":"2024-03-21T10:38:13.155079+0000","last_clean":"2024-03-21T10:38:13.155079+0000","last_became_active":"2024-03-21T10:35:06.003926+0000","last_became_peered":"2024-03-21T10:35:06.003926+0000","last_unstale":"2024-03-21T10:38:13.155079+0000","last_undegraded":"2024-03-21T10:38:13.155079+0000","last_fullsized":"2024-03-21T10:38:13.155079+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:05:49.721000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":158,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312743+0000","last_change":"2024-03-21T10:35:06.913556+0000","last_active":"2024-03-21T10:36:56.312743+0000","last_peered":"2024-03-21T10:36:56.312743+0000","last_clean":"2024-03-21T10:36:56.312743+0000","last_became_active":"2024-03-21T10:35:06.913213+0000","last_became_peered":"2024-03-21T10:35:06.913213+0000","last_unstale":"2024-03-21T10:36:56.312743+0000","last_undegraded":"2024-03-21T10:36:56.312743+0000","last_fullsized":"2024-03-21T10:36:56.312743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:11:09.235927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619374+0000","last_change":"2024-03-21T10:35:55.523819+0000","last_active":"2024-03-21T10:36:56.619374+0000","last_peered":"2024-03-21T10:36:56.619374+0000","last_clean":"2024-03-21T10:36:56.619374+0000","last_became_active":"2024-03-21T10:35:55.523333+0000","last_became_peered":"2024-03-21T10:35:55.523333+0000","last_unstale":"2024-03-21T10:36:56.619374+0000","last_undegraded":"2024-03-21T10:36:56.619374+0000","last_fullsized":"2024-03-21T10:36:56.619374+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:25:36.277481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311347+0000","last_change":"2024-03-21T10:35:06.012527+0000","last_active":"2024-03-21T10:36:56.311347+0000","last_peered":"2024-03-21T10:36:56.311347+0000","last_clean":"2024-03-21T10:36:56.311347+0000","last_became_active":"2024-03-21T10:35:06.012228+0000","last_became_peered":"2024-03-21T10:35:06.012228+0000","last_unstale":"2024-03-21T10:36:56.311347+0000","last_undegraded":"2024-03-21T10:36:56.311347+0000","last_fullsized":"2024-03-21T10:36:56.311347+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:52:34.622306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.622381+0000","last_change":"2024-03-21T10:35:55.503531+0000","last_active":"2024-03-21T10:36:56.622381+0000","last_peered":"2024-03-21T10:36:56.622381+0000","last_clean":"2024-03-21T10:36:56.622381+0000","last_became_active":"2024-03-21T10:35:55.502793+0000","last_became_peered":"2024-03-21T10:35:55.502793+0000","last_unstale":"2024-03-21T10:36:56.622381+0000","last_undegraded":"2024-03-21T10:36:56.622381+0000","last_fullsized":"2024-03-21T10:36:56.622381+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:08:39.599084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,3],"acting":[9,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619509+0000","last_change":"2024-03-21T10:35:11.196543+0000","last_active":"2024-03-21T10:36:56.619509+0000","last_peered":"2024-03-21T10:36:56.619509+0000","last_clean":"2024-03-21T10:36:56.619509+0000","last_became_active":"2024-03-21T10:35:11.196151+0000","last_became_peered":"2024-03-21T10:35:11.196151+0000","last_unstale":"2024-03-21T10:36:56.619509+0000","last_undegraded":"2024-03-21T10:36:56.619509+0000","last_fullsized":"2024-03-21T10:36:56.619509+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:15:39.193647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312200+0000","last_change":"2024-03-21T10:35:06.004588+0000","last_active":"2024-03-21T10:36:56.312200+0000","last_peered":"2024-03-21T10:36:56.312200+0000","last_clean":"2024-03-21T10:36:56.312200+0000","last_became_active":"2024-03-21T10:35:06.004190+0000","last_became_peered":"2024-03-21T10:35:06.004190+0000","last_unstale":"2024-03-21T10:36:56.312200+0000","last_undegraded":"2024-03-21T10:36:56.312200+0000","last_fullsized":"2024-03-21T10:36:56.312200+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:32:18.563692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325080+0000","last_change":"2024-03-21T10:35:06.941738+0000","last_active":"2024-03-21T10:36:56.325080+0000","last_peered":"2024-03-21T10:36:56.325080+0000","last_clean":"2024-03-21T10:36:56.325080+0000","last_became_active":"2024-03-21T10:35:06.941386+0000","last_became_peered":"2024-03-21T10:35:06.941386+0000","last_unstale":"2024-03-21T10:36:56.325080+0000","last_undegraded":"2024-03-21T10:36:56.325080+0000","last_fullsized":"2024-03-21T10:36:56.325080+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:52:55.519095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621767+0000","last_change":"2024-03-21T10:35:11.199370+0000","last_active":"2024-03-21T10:36:56.621767+0000","last_peered":"2024-03-21T10:36:56.621767+0000","last_clean":"2024-03-21T10:36:56.621767+0000","last_became_active":"2024-03-21T10:35:11.198598+0000","last_became_peered":"2024-03-21T10:35:11.198598+0000","last_unstale":"2024-03-21T10:36:56.621767+0000","last_undegraded":"2024-03-21T10:36:56.621767+0000","last_fullsized":"2024-03-21T10:36:56.621767+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:07:19.419405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311217+0000","last_change":"2024-03-21T10:35:06.004893+0000","last_active":"2024-03-21T10:36:56.311217+0000","last_peered":"2024-03-21T10:36:56.311217+0000","last_clean":"2024-03-21T10:36:56.311217+0000","last_became_active":"2024-03-21T10:35:06.004586+0000","last_became_peered":"2024-03-21T10:35:06.004586+0000","last_unstale":"2024-03-21T10:36:56.311217+0000","last_undegraded":"2024-03-21T10:36:56.311217+0000","last_fullsized":"2024-03-21T10:36:56.311217+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:05:29.106575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621403+0000","last_change":"2024-03-21T10:35:06.924448+0000","last_active":"2024-03-21T10:36:56.621403+0000","last_peered":"2024-03-21T10:36:56.621403+0000","last_clean":"2024-03-21T10:36:56.621403+0000","last_became_active":"2024-03-21T10:35:06.924049+0000","last_became_peered":"2024-03-21T10:35:06.924049+0000","last_unstale":"2024-03-21T10:36:56.621403+0000","last_undegraded":"2024-03-21T10:36:56.621403+0000","last_fullsized":"2024-03-21T10:36:56.621403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:23:12.070723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325020+0000","last_change":"2024-03-21T10:35:11.188691+0000","last_active":"2024-03-21T10:36:56.325020+0000","last_peered":"2024-03-21T10:36:56.325020+0000","last_clean":"2024-03-21T10:36:56.325020+0000","last_became_active":"2024-03-21T10:35:11.188138+0000","last_became_peered":"2024-03-21T10:35:11.188138+0000","last_unstale":"2024-03-21T10:36:56.325020+0000","last_undegraded":"2024-03-21T10:36:56.325020+0000","last_fullsized":"2024-03-21T10:36:56.325020+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:51:50.920920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":191,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:38:14.155614+0000","last_change":"2024-03-21T10:35:06.015276+0000","last_active":"2024-03-21T10:38:14.155614+0000","last_peered":"2024-03-21T10:38:14.155614+0000","last_clean":"2024-03-21T10:38:14.155614+0000","last_became_active":"2024-03-21T10:35:06.014734+0000","last_became_peered":"2024-03-21T10:35:06.014734+0000","last_unstale":"2024-03-21T10:38:14.155614+0000","last_undegraded":"2024-03-21T10:38:14.155614+0000","last_fullsized":"2024-03-21T10:38:14.155614+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:18:45.905451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":155,"num_read_kb":155,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.620346+0000","last_change":"2024-03-21T10:35:06.926689+0000","last_active":"2024-03-21T10:36:56.620346+0000","last_peered":"2024-03-21T10:36:56.620346+0000","last_clean":"2024-03-21T10:36:56.620346+0000","last_became_active":"2024-03-21T10:35:06.926319+0000","last_became_peered":"2024-03-21T10:35:06.926319+0000","last_unstale":"2024-03-21T10:36:56.620346+0000","last_undegraded":"2024-03-21T10:36:56.620346+0000","last_fullsized":"2024-03-21T10:36:56.620346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:06:22.371608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311104+0000","last_change":"2024-03-21T10:35:55.383776+0000","last_active":"2024-03-21T10:36:56.311104+0000","last_peered":"2024-03-21T10:36:56.311104+0000","last_clean":"2024-03-21T10:36:56.311104+0000","last_became_active":"2024-03-21T10:35:55.383300+0000","last_became_peered":"2024-03-21T10:35:55.383300+0000","last_unstale":"2024-03-21T10:36:56.311104+0000","last_undegraded":"2024-03-21T10:36:56.311104+0000","last_fullsized":"2024-03-21T10:36:56.311104+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:55:56.244325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2,4],"acting":[10,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619479+0000","last_change":"2024-03-21T10:35:06.007231+0000","last_active":"2024-03-21T10:36:56.619479+0000","last_peered":"2024-03-21T10:36:56.619479+0000","last_clean":"2024-03-21T10:36:56.619479+0000","last_became_active":"2024-03-21T10:35:06.006875+0000","last_became_peered":"2024-03-21T10:35:06.006875+0000","last_unstale":"2024-03-21T10:36:56.619479+0000","last_undegraded":"2024-03-21T10:36:56.619479+0000","last_fullsized":"2024-03-21T10:36:56.619479+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:08:02.489741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.624356+0000","last_change":"2024-03-21T10:35:55.524107+0000","last_active":"2024-03-21T10:36:56.624356+0000","last_peered":"2024-03-21T10:36:56.624356+0000","last_clean":"2024-03-21T10:36:56.624356+0000","last_became_active":"2024-03-21T10:35:55.523523+0000","last_became_peered":"2024-03-21T10:35:55.523523+0000","last_unstale":"2024-03-21T10:36:56.624356+0000","last_undegraded":"2024-03-21T10:36:56.624356+0000","last_fullsized":"2024-03-21T10:36:56.624356+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:49:29.121279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540722+0000","last_change":"2024-03-21T10:35:11.193382+0000","last_active":"2024-03-21T10:36:56.540722+0000","last_peered":"2024-03-21T10:36:56.540722+0000","last_clean":"2024-03-21T10:36:56.540722+0000","last_became_active":"2024-03-21T10:35:11.192959+0000","last_became_peered":"2024-03-21T10:35:11.192959+0000","last_unstale":"2024-03-21T10:36:56.540722+0000","last_undegraded":"2024-03-21T10:36:56.540722+0000","last_fullsized":"2024-03-21T10:36:56.540722+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:02:57.187370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311370+0000","last_change":"2024-03-21T10:35:06.019969+0000","last_active":"2024-03-21T10:36:56.311370+0000","last_peered":"2024-03-21T10:36:56.311370+0000","last_clean":"2024-03-21T10:36:56.311370+0000","last_became_active":"2024-03-21T10:35:06.019648+0000","last_became_peered":"2024-03-21T10:35:06.019648+0000","last_unstale":"2024-03-21T10:36:56.311370+0000","last_undegraded":"2024-03-21T10:36:56.311370+0000","last_fullsized":"2024-03-21T10:36:56.311370+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:24:02.227205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.620292+0000","last_change":"2024-03-21T10:35:06.910294+0000","last_active":"2024-03-21T10:36:56.620292+0000","last_peered":"2024-03-21T10:36:56.620292+0000","last_clean":"2024-03-21T10:36:56.620292+0000","last_became_active":"2024-03-21T10:35:06.909862+0000","last_became_peered":"2024-03-21T10:35:06.909862+0000","last_unstale":"2024-03-21T10:36:56.620292+0000","last_undegraded":"2024-03-21T10:36:56.620292+0000","last_fullsized":"2024-03-21T10:36:56.620292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:37:13.396003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621610+0000","last_change":"2024-03-21T10:35:11.196398+0000","last_active":"2024-03-21T10:36:56.621610+0000","last_peered":"2024-03-21T10:36:56.621610+0000","last_clean":"2024-03-21T10:36:56.621610+0000","last_became_active":"2024-03-21T10:35:11.194965+0000","last_became_peered":"2024-03-21T10:35:11.194965+0000","last_unstale":"2024-03-21T10:36:56.621610+0000","last_undegraded":"2024-03-21T10:36:56.621610+0000","last_fullsized":"2024-03-21T10:36:56.621610+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:48:22.024137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'38","reported_seq":391,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:38:15.156609+0000","last_change":"2024-03-21T10:35:06.005037+0000","last_active":"2024-03-21T10:38:15.156609+0000","last_peered":"2024-03-21T10:38:15.156609+0000","last_clean":"2024-03-21T10:38:15.156609+0000","last_became_active":"2024-03-21T10:35:06.004686+0000","last_became_peered":"2024-03-21T10:35:06.004686+0000","last_unstale":"2024-03-21T10:38:15.156609+0000","last_undegraded":"2024-03-21T10:38:15.156609+0000","last_fullsized":"2024-03-21T10:38:15.156609+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:24:13.312112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64761,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":321,"num_read_kb":225,"num_write":39,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313007+0000","last_change":"2024-03-21T10:35:06.906525+0000","last_active":"2024-03-21T10:36:56.313007+0000","last_peered":"2024-03-21T10:36:56.313007+0000","last_clean":"2024-03-21T10:36:56.313007+0000","last_became_active":"2024-03-21T10:35:06.905150+0000","last_became_peered":"2024-03-21T10:35:06.905150+0000","last_unstale":"2024-03-21T10:36:56.313007+0000","last_undegraded":"2024-03-21T10:36:56.313007+0000","last_fullsized":"2024-03-21T10:36:56.313007+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:46:27.958779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311725+0000","last_change":"2024-03-21T10:35:11.193519+0000","last_active":"2024-03-21T10:36:56.311725+0000","last_peered":"2024-03-21T10:36:56.311725+0000","last_clean":"2024-03-21T10:36:56.311725+0000","last_became_active":"2024-03-21T10:35:11.193043+0000","last_became_peered":"2024-03-21T10:35:11.193043+0000","last_unstale":"2024-03-21T10:36:56.311725+0000","last_undegraded":"2024-03-21T10:36:56.311725+0000","last_fullsized":"2024-03-21T10:36:56.311725+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:23:20.984472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618979+0000","last_change":"2024-03-21T10:35:05.999220+0000","last_active":"2024-03-21T10:36:56.618979+0000","last_peered":"2024-03-21T10:36:56.618979+0000","last_clean":"2024-03-21T10:36:56.618979+0000","last_became_active":"2024-03-21T10:35:05.998918+0000","last_became_peered":"2024-03-21T10:35:05.998918+0000","last_unstale":"2024-03-21T10:36:56.618979+0000","last_undegraded":"2024-03-21T10:36:56.618979+0000","last_fullsized":"2024-03-21T10:36:56.618979+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:01:09.816758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312512+0000","last_change":"2024-03-21T10:35:06.941234+0000","last_active":"2024-03-21T10:36:56.312512+0000","last_peered":"2024-03-21T10:36:56.312512+0000","last_clean":"2024-03-21T10:36:56.312512+0000","last_became_active":"2024-03-21T10:35:06.940828+0000","last_became_peered":"2024-03-21T10:35:06.940828+0000","last_unstale":"2024-03-21T10:36:56.312512+0000","last_undegraded":"2024-03-21T10:36:56.312512+0000","last_fullsized":"2024-03-21T10:36:56.312512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:50:34.398822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618827+0000","last_change":"2024-03-21T10:35:11.196007+0000","last_active":"2024-03-21T10:36:56.618827+0000","last_peered":"2024-03-21T10:36:56.618827+0000","last_clean":"2024-03-21T10:36:56.618827+0000","last_became_active":"2024-03-21T10:35:11.195609+0000","last_became_peered":"2024-03-21T10:35:11.195609+0000","last_unstale":"2024-03-21T10:36:56.618827+0000","last_undegraded":"2024-03-21T10:36:56.618827+0000","last_fullsized":"2024-03-21T10:36:56.618827+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:21:38.803390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311522+0000","last_change":"2024-03-21T10:35:06.011594+0000","last_active":"2024-03-21T10:36:56.311522+0000","last_peered":"2024-03-21T10:36:56.311522+0000","last_clean":"2024-03-21T10:36:56.311522+0000","last_became_active":"2024-03-21T10:35:06.011254+0000","last_became_peered":"2024-03-21T10:35:06.011254+0000","last_unstale":"2024-03-21T10:36:56.311522+0000","last_undegraded":"2024-03-21T10:36:56.311522+0000","last_fullsized":"2024-03-21T10:36:56.311522+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:52:31.045659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619993+0000","last_change":"2024-03-21T10:35:06.929170+0000","last_active":"2024-03-21T10:36:56.619993+0000","last_peered":"2024-03-21T10:36:56.619993+0000","last_clean":"2024-03-21T10:36:56.619993+0000","last_became_active":"2024-03-21T10:35:06.928822+0000","last_became_peered":"2024-03-21T10:35:06.928822+0000","last_unstale":"2024-03-21T10:36:56.619993+0000","last_undegraded":"2024-03-21T10:36:56.619993+0000","last_fullsized":"2024-03-21T10:36:56.619993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:37:40.328453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623051+0000","last_change":"2024-03-21T10:35:11.204285+0000","last_active":"2024-03-21T10:36:56.623051+0000","last_peered":"2024-03-21T10:36:56.623051+0000","last_clean":"2024-03-21T10:36:56.623051+0000","last_became_active":"2024-03-21T10:35:11.203698+0000","last_became_peered":"2024-03-21T10:35:11.203698+0000","last_unstale":"2024-03-21T10:36:56.623051+0000","last_undegraded":"2024-03-21T10:36:56.623051+0000","last_fullsized":"2024-03-21T10:36:56.623051+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:03:32.732488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"93'4","reported_seq":192,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:38:13.235271+0000","last_change":"2024-03-21T10:35:06.005028+0000","last_active":"2024-03-21T10:38:13.235271+0000","last_peered":"2024-03-21T10:38:13.235271+0000","last_clean":"2024-03-21T10:38:13.235271+0000","last_became_active":"2024-03-21T10:35:06.004670+0000","last_became_peered":"2024-03-21T10:35:06.004670+0000","last_unstale":"2024-03-21T10:38:13.235271+0000","last_undegraded":"2024-03-21T10:38:13.235271+0000","last_fullsized":"2024-03-21T10:38:13.235271+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:00:18.481051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":894,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":158,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619542+0000","last_change":"2024-03-21T10:35:06.911696+0000","last_active":"2024-03-21T10:36:56.619542+0000","last_peered":"2024-03-21T10:36:56.619542+0000","last_clean":"2024-03-21T10:36:56.619542+0000","last_became_active":"2024-03-21T10:35:06.911231+0000","last_became_peered":"2024-03-21T10:35:06.911231+0000","last_unstale":"2024-03-21T10:36:56.619542+0000","last_undegraded":"2024-03-21T10:36:56.619542+0000","last_fullsized":"2024-03-21T10:36:56.619542+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:21:51.540993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312937+0000","last_change":"2024-03-21T10:35:11.204422+0000","last_active":"2024-03-21T10:36:56.312937+0000","last_peered":"2024-03-21T10:36:56.312937+0000","last_clean":"2024-03-21T10:36:56.312937+0000","last_became_active":"2024-03-21T10:35:11.203904+0000","last_became_peered":"2024-03-21T10:35:11.203904+0000","last_unstale":"2024-03-21T10:36:56.312937+0000","last_undegraded":"2024-03-21T10:36:56.312937+0000","last_fullsized":"2024-03-21T10:36:56.312937+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:38:14.486012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313161+0000","last_change":"2024-03-21T10:35:06.019823+0000","last_active":"2024-03-21T10:36:56.313161+0000","last_peered":"2024-03-21T10:36:56.313161+0000","last_clean":"2024-03-21T10:36:56.313161+0000","last_became_active":"2024-03-21T10:35:06.019502+0000","last_became_peered":"2024-03-21T10:35:06.019502+0000","last_unstale":"2024-03-21T10:36:56.313161+0000","last_undegraded":"2024-03-21T10:36:56.313161+0000","last_fullsized":"2024-03-21T10:36:56.313161+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:33:22.438868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311234+0000","last_change":"2024-03-21T10:35:06.906671+0000","last_active":"2024-03-21T10:36:56.311234+0000","last_peered":"2024-03-21T10:36:56.311234+0000","last_clean":"2024-03-21T10:36:56.311234+0000","last_became_active":"2024-03-21T10:35:06.905873+0000","last_became_peered":"2024-03-21T10:35:06.905873+0000","last_unstale":"2024-03-21T10:36:56.311234+0000","last_undegraded":"2024-03-21T10:36:56.311234+0000","last_fullsized":"2024-03-21T10:36:56.311234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:14:59.304255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311196+0000","last_change":"2024-03-21T10:35:11.197698+0000","last_active":"2024-03-21T10:36:56.311196+0000","last_peered":"2024-03-21T10:36:56.311196+0000","last_clean":"2024-03-21T10:36:56.311196+0000","last_became_active":"2024-03-21T10:35:11.197295+0000","last_became_peered":"2024-03-21T10:35:11.197295+0000","last_unstale":"2024-03-21T10:36:56.311196+0000","last_undegraded":"2024-03-21T10:36:56.311196+0000","last_fullsized":"2024-03-21T10:36:56.311196+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:02:24.338317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"93'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:37:22.250209+0000","last_change":"2024-03-21T10:35:06.000951+0000","last_active":"2024-03-21T10:37:22.250209+0000","last_peered":"2024-03-21T10:37:22.250209+0000","last_clean":"2024-03-21T10:37:22.250209+0000","last_became_active":"2024-03-21T10:35:05.999169+0000","last_became_peered":"2024-03-21T10:35:05.999169+0000","last_unstale":"2024-03-21T10:37:22.250209+0000","last_undegraded":"2024-03-21T10:37:22.250209+0000","last_fullsized":"2024-03-21T10:37:22.250209+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:31:25.602621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619339+0000","last_change":"2024-03-21T10:35:06.908067+0000","last_active":"2024-03-21T10:36:56.619339+0000","last_peered":"2024-03-21T10:36:56.619339+0000","last_clean":"2024-03-21T10:36:56.619339+0000","last_became_active":"2024-03-21T10:35:06.907720+0000","last_became_peered":"2024-03-21T10:35:06.907720+0000","last_unstale":"2024-03-21T10:36:56.619339+0000","last_undegraded":"2024-03-21T10:36:56.619339+0000","last_fullsized":"2024-03-21T10:36:56.619339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:16:00.502637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312728+0000","last_change":"2024-03-21T10:35:11.229470+0000","last_active":"2024-03-21T10:36:56.312728+0000","last_peered":"2024-03-21T10:36:56.312728+0000","last_clean":"2024-03-21T10:36:56.312728+0000","last_became_active":"2024-03-21T10:35:11.228977+0000","last_became_peered":"2024-03-21T10:35:11.228977+0000","last_unstale":"2024-03-21T10:36:56.312728+0000","last_undegraded":"2024-03-21T10:36:56.312728+0000","last_fullsized":"2024-03-21T10:36:56.312728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:05:52.269712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623236+0000","last_change":"2024-03-21T10:35:06.019866+0000","last_active":"2024-03-21T10:36:56.623236+0000","last_peered":"2024-03-21T10:36:56.623236+0000","last_clean":"2024-03-21T10:36:56.623236+0000","last_became_active":"2024-03-21T10:35:06.019550+0000","last_became_peered":"2024-03-21T10:35:06.019550+0000","last_unstale":"2024-03-21T10:36:56.623236+0000","last_undegraded":"2024-03-21T10:36:56.623236+0000","last_fullsized":"2024-03-21T10:36:56.623236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:51:46.989647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311710+0000","last_change":"2024-03-21T10:35:55.385617+0000","last_active":"2024-03-21T10:36:56.311710+0000","last_peered":"2024-03-21T10:36:56.311710+0000","last_clean":"2024-03-21T10:36:56.311710+0000","last_became_active":"2024-03-21T10:35:55.385293+0000","last_became_peered":"2024-03-21T10:35:55.385293+0000","last_unstale":"2024-03-21T10:36:56.311710+0000","last_undegraded":"2024-03-21T10:36:56.311710+0000","last_fullsized":"2024-03-21T10:36:56.311710+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:18:08.200169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313097+0000","last_change":"2024-03-21T10:35:11.199714+0000","last_active":"2024-03-21T10:36:56.313097+0000","last_peered":"2024-03-21T10:36:56.313097+0000","last_clean":"2024-03-21T10:36:56.313097+0000","last_became_active":"2024-03-21T10:35:11.199277+0000","last_became_peered":"2024-03-21T10:35:11.199277+0000","last_unstale":"2024-03-21T10:36:56.313097+0000","last_undegraded":"2024-03-21T10:36:56.313097+0000","last_fullsized":"2024-03-21T10:36:56.313097+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:58:01.471162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619799+0000","last_change":"2024-03-21T10:35:06.015090+0000","last_active":"2024-03-21T10:36:56.619799+0000","last_peered":"2024-03-21T10:36:56.619799+0000","last_clean":"2024-03-21T10:36:56.619799+0000","last_became_active":"2024-03-21T10:35:06.014666+0000","last_became_peered":"2024-03-21T10:35:06.014666+0000","last_unstale":"2024-03-21T10:36:56.619799+0000","last_undegraded":"2024-03-21T10:36:56.619799+0000","last_fullsized":"2024-03-21T10:36:56.619799+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:41:17.608069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311382+0000","last_change":"2024-03-21T10:35:55.388548+0000","last_active":"2024-03-21T10:36:56.311382+0000","last_peered":"2024-03-21T10:36:56.311382+0000","last_clean":"2024-03-21T10:36:56.311382+0000","last_became_active":"2024-03-21T10:35:55.388253+0000","last_became_peered":"2024-03-21T10:35:55.388253+0000","last_unstale":"2024-03-21T10:36:56.311382+0000","last_undegraded":"2024-03-21T10:36:56.311382+0000","last_fullsized":"2024-03-21T10:36:56.311382+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:56.678224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312691+0000","last_change":"2024-03-21T10:35:11.230677+0000","last_active":"2024-03-21T10:36:56.312691+0000","last_peered":"2024-03-21T10:36:56.312691+0000","last_clean":"2024-03-21T10:36:56.312691+0000","last_became_active":"2024-03-21T10:35:11.230087+0000","last_became_peered":"2024-03-21T10:35:11.230087+0000","last_unstale":"2024-03-21T10:36:56.312691+0000","last_undegraded":"2024-03-21T10:36:56.312691+0000","last_fullsized":"2024-03-21T10:36:56.312691+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:39:47.663950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311339+0000","last_change":"2024-03-21T10:35:06.904287+0000","last_active":"2024-03-21T10:36:56.311339+0000","last_peered":"2024-03-21T10:36:56.311339+0000","last_clean":"2024-03-21T10:36:56.311339+0000","last_became_active":"2024-03-21T10:35:06.903821+0000","last_became_peered":"2024-03-21T10:35:06.903821+0000","last_unstale":"2024-03-21T10:36:56.311339+0000","last_undegraded":"2024-03-21T10:36:56.311339+0000","last_fullsized":"2024-03-21T10:36:56.311339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:36:25.362203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'5","reported_seq":191,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:38:13.234869+0000","last_change":"2024-03-21T10:35:06.014432+0000","last_active":"2024-03-21T10:38:13.234869+0000","last_peered":"2024-03-21T10:38:13.234869+0000","last_clean":"2024-03-21T10:38:13.234869+0000","last_became_active":"2024-03-21T10:35:06.013448+0000","last_became_peered":"2024-03-21T10:35:06.013448+0000","last_unstale":"2024-03-21T10:38:13.234869+0000","last_undegraded":"2024-03-21T10:38:13.234869+0000","last_fullsized":"2024-03-21T10:38:13.234869+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:29:10.600794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":154,"num_read_kb":154,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619114+0000","last_change":"2024-03-21T10:35:11.193577+0000","last_active":"2024-03-21T10:36:56.619114+0000","last_peered":"2024-03-21T10:36:56.619114+0000","last_clean":"2024-03-21T10:36:56.619114+0000","last_became_active":"2024-03-21T10:35:11.193132+0000","last_became_peered":"2024-03-21T10:35:11.193132+0000","last_unstale":"2024-03-21T10:36:56.619114+0000","last_undegraded":"2024-03-21T10:36:56.619114+0000","last_fullsized":"2024-03-21T10:36:56.619114+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:40:31.100489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325105+0000","last_change":"2024-03-21T10:35:06.924376+0000","last_active":"2024-03-21T10:36:56.325105+0000","last_peered":"2024-03-21T10:36:56.325105+0000","last_clean":"2024-03-21T10:36:56.325105+0000","last_became_active":"2024-03-21T10:35:06.923874+0000","last_became_peered":"2024-03-21T10:35:06.923874+0000","last_unstale":"2024-03-21T10:36:56.325105+0000","last_undegraded":"2024-03-21T10:36:56.325105+0000","last_fullsized":"2024-03-21T10:36:56.325105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:56:12.620667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540608+0000","last_change":"2024-03-21T10:35:06.015222+0000","last_active":"2024-03-21T10:36:56.540608+0000","last_peered":"2024-03-21T10:36:56.540608+0000","last_clean":"2024-03-21T10:36:56.540608+0000","last_became_active":"2024-03-21T10:35:06.014858+0000","last_became_peered":"2024-03-21T10:35:06.014858+0000","last_unstale":"2024-03-21T10:36:56.540608+0000","last_undegraded":"2024-03-21T10:36:56.540608+0000","last_fullsized":"2024-03-21T10:36:56.540608+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:45:12.528851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":367,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618815+0000","last_change":"2024-03-21T10:33:44.536069+0000","last_active":"2024-03-21T10:36:56.618815+0000","last_peered":"2024-03-21T10:36:56.618815+0000","last_clean":"2024-03-21T10:36:56.618815+0000","last_became_active":"2024-03-21T10:33:44.535623+0000","last_became_peered":"2024-03-21T10:33:44.535623+0000","last_unstale":"2024-03-21T10:36:56.618815+0000","last_undegraded":"2024-03-21T10:36:56.618815+0000","last_fullsized":"2024-03-21T10:36:56.618815+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:29:44.554765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:29:44.554765+0000","last_clean_scrub_stamp":"2024-03-21T10:29:44.554765+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:43:23.440837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311654+0000","last_change":"2024-03-21T10:35:11.229580+0000","last_active":"2024-03-21T10:36:56.311654+0000","last_peered":"2024-03-21T10:36:56.311654+0000","last_clean":"2024-03-21T10:36:56.311654+0000","last_became_active":"2024-03-21T10:35:11.229044+0000","last_became_peered":"2024-03-21T10:35:11.229044+0000","last_unstale":"2024-03-21T10:36:56.311654+0000","last_undegraded":"2024-03-21T10:36:56.311654+0000","last_fullsized":"2024-03-21T10:36:56.311654+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:54:50.535253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312060+0000","last_change":"2024-03-21T10:35:06.927690+0000","last_active":"2024-03-21T10:36:56.312060+0000","last_peered":"2024-03-21T10:36:56.312060+0000","last_clean":"2024-03-21T10:36:56.312060+0000","last_became_active":"2024-03-21T10:35:06.927304+0000","last_became_peered":"2024-03-21T10:35:06.927304+0000","last_unstale":"2024-03-21T10:36:56.312060+0000","last_undegraded":"2024-03-21T10:36:56.312060+0000","last_fullsized":"2024-03-21T10:36:56.312060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:30:48.950459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619515+0000","last_change":"2024-03-21T10:35:06.004952+0000","last_active":"2024-03-21T10:36:56.619515+0000","last_peered":"2024-03-21T10:36:56.619515+0000","last_clean":"2024-03-21T10:36:56.619515+0000","last_became_active":"2024-03-21T10:35:06.004206+0000","last_became_peered":"2024-03-21T10:35:06.004206+0000","last_unstale":"2024-03-21T10:36:56.619515+0000","last_undegraded":"2024-03-21T10:36:56.619515+0000","last_fullsized":"2024-03-21T10:36:56.619515+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:20:29.198715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619052+0000","last_change":"2024-03-21T10:35:11.179125+0000","last_active":"2024-03-21T10:36:56.619052+0000","last_peered":"2024-03-21T10:36:56.619052+0000","last_clean":"2024-03-21T10:36:56.619052+0000","last_became_active":"2024-03-21T10:35:11.178658+0000","last_became_peered":"2024-03-21T10:35:11.178658+0000","last_unstale":"2024-03-21T10:36:56.619052+0000","last_undegraded":"2024-03-21T10:36:56.619052+0000","last_fullsized":"2024-03-21T10:36:56.619052+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:33:25.281721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313030+0000","last_change":"2024-03-21T10:35:06.019996+0000","last_active":"2024-03-21T10:36:56.313030+0000","last_peered":"2024-03-21T10:36:56.313030+0000","last_clean":"2024-03-21T10:36:56.313030+0000","last_became_active":"2024-03-21T10:35:06.019624+0000","last_became_peered":"2024-03-21T10:35:06.019624+0000","last_unstale":"2024-03-21T10:36:56.313030+0000","last_undegraded":"2024-03-21T10:36:56.313030+0000","last_fullsized":"2024-03-21T10:36:56.313030+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:31:09.551433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313495+0000","last_change":"2024-03-21T10:35:06.939324+0000","last_active":"2024-03-21T10:36:56.313495+0000","last_peered":"2024-03-21T10:36:56.313495+0000","last_clean":"2024-03-21T10:36:56.313495+0000","last_became_active":"2024-03-21T10:35:06.937750+0000","last_became_peered":"2024-03-21T10:35:06.937750+0000","last_unstale":"2024-03-21T10:36:56.313495+0000","last_undegraded":"2024-03-21T10:36:56.313495+0000","last_fullsized":"2024-03-21T10:36:56.313495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:45:14.600863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.324947+0000","last_change":"2024-03-21T10:35:11.229479+0000","last_active":"2024-03-21T10:36:56.324947+0000","last_peered":"2024-03-21T10:36:56.324947+0000","last_clean":"2024-03-21T10:36:56.324947+0000","last_became_active":"2024-03-21T10:35:11.229005+0000","last_became_peered":"2024-03-21T10:35:11.229005+0000","last_unstale":"2024-03-21T10:36:56.324947+0000","last_undegraded":"2024-03-21T10:36:56.324947+0000","last_fullsized":"2024-03-21T10:36:56.324947+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:39:19.859581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'12","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:37:19.740273+0000","last_change":"2024-03-21T10:35:06.013744+0000","last_active":"2024-03-21T10:37:19.740273+0000","last_peered":"2024-03-21T10:37:19.740273+0000","last_clean":"2024-03-21T10:37:19.740273+0000","last_became_active":"2024-03-21T10:35:06.013509+0000","last_became_peered":"2024-03-21T10:35:06.013509+0000","last_unstale":"2024-03-21T10:37:19.740273+0000","last_undegraded":"2024-03-21T10:37:19.740273+0000","last_fullsized":"2024-03-21T10:37:19.740273+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:51:45.000933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325968+0000","last_change":"2024-03-21T10:35:06.930685+0000","last_active":"2024-03-21T10:36:56.325968+0000","last_peered":"2024-03-21T10:36:56.325968+0000","last_clean":"2024-03-21T10:36:56.325968+0000","last_became_active":"2024-03-21T10:35:06.929775+0000","last_became_peered":"2024-03-21T10:35:06.929775+0000","last_unstale":"2024-03-21T10:36:56.325968+0000","last_undegraded":"2024-03-21T10:36:56.325968+0000","last_fullsized":"2024-03-21T10:36:56.325968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:13:18.058752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618619+0000","last_change":"2024-03-21T10:35:11.177061+0000","last_active":"2024-03-21T10:36:56.618619+0000","last_peered":"2024-03-21T10:36:56.618619+0000","last_clean":"2024-03-21T10:36:56.618619+0000","last_became_active":"2024-03-21T10:35:11.176603+0000","last_became_peered":"2024-03-21T10:35:11.176603+0000","last_unstale":"2024-03-21T10:36:56.618619+0000","last_undegraded":"2024-03-21T10:36:56.618619+0000","last_fullsized":"2024-03-21T10:36:56.618619+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:54.532780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540962+0000","last_change":"2024-03-21T10:35:06.009018+0000","last_active":"2024-03-21T10:36:56.540962+0000","last_peered":"2024-03-21T10:36:56.540962+0000","last_clean":"2024-03-21T10:36:56.540962+0000","last_became_active":"2024-03-21T10:35:06.008638+0000","last_became_peered":"2024-03-21T10:35:06.008638+0000","last_unstale":"2024-03-21T10:36:56.540962+0000","last_undegraded":"2024-03-21T10:36:56.540962+0000","last_fullsized":"2024-03-21T10:36:56.540962+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:42:19.527898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621997+0000","last_change":"2024-03-21T10:35:06.931588+0000","last_active":"2024-03-21T10:36:56.621997+0000","last_peered":"2024-03-21T10:36:56.621997+0000","last_clean":"2024-03-21T10:36:56.621997+0000","last_became_active":"2024-03-21T10:35:06.930707+0000","last_became_peered":"2024-03-21T10:35:06.930707+0000","last_unstale":"2024-03-21T10:36:56.621997+0000","last_undegraded":"2024-03-21T10:36:56.621997+0000","last_fullsized":"2024-03-21T10:36:56.621997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:44:55.862799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540818+0000","last_change":"2024-03-21T10:35:11.229408+0000","last_active":"2024-03-21T10:36:56.540818+0000","last_peered":"2024-03-21T10:36:56.540818+0000","last_clean":"2024-03-21T10:36:56.540818+0000","last_became_active":"2024-03-21T10:35:11.228997+0000","last_became_peered":"2024-03-21T10:35:11.228997+0000","last_unstale":"2024-03-21T10:36:56.540818+0000","last_undegraded":"2024-03-21T10:36:56.540818+0000","last_fullsized":"2024-03-21T10:36:56.540818+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:56:11.517597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619256+0000","last_change":"2024-03-21T10:35:55.388978+0000","last_active":"2024-03-21T10:36:56.619256+0000","last_peered":"2024-03-21T10:36:56.619256+0000","last_clean":"2024-03-21T10:36:56.619256+0000","last_became_active":"2024-03-21T10:35:55.383048+0000","last_became_peered":"2024-03-21T10:35:55.383048+0000","last_unstale":"2024-03-21T10:36:56.619256+0000","last_undegraded":"2024-03-21T10:36:56.619256+0000","last_fullsized":"2024-03-21T10:36:56.619256+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:24:01.933115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312642+0000","last_change":"2024-03-21T10:35:06.908585+0000","last_active":"2024-03-21T10:36:56.312642+0000","last_peered":"2024-03-21T10:36:56.312642+0000","last_clean":"2024-03-21T10:36:56.312642+0000","last_became_active":"2024-03-21T10:35:06.908253+0000","last_became_peered":"2024-03-21T10:35:06.908253+0000","last_unstale":"2024-03-21T10:36:56.312642+0000","last_undegraded":"2024-03-21T10:36:56.312642+0000","last_fullsized":"2024-03-21T10:36:56.312642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:46:15.984450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623323+0000","last_change":"2024-03-21T10:35:11.199455+0000","last_active":"2024-03-21T10:36:56.623323+0000","last_peered":"2024-03-21T10:36:56.623323+0000","last_clean":"2024-03-21T10:36:56.623323+0000","last_became_active":"2024-03-21T10:35:11.197705+0000","last_became_peered":"2024-03-21T10:35:11.197705+0000","last_unstale":"2024-03-21T10:36:56.623323+0000","last_undegraded":"2024-03-21T10:36:56.623323+0000","last_fullsized":"2024-03-21T10:36:56.623323+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:59:42.792764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325149+0000","last_change":"2024-03-21T10:35:06.016152+0000","last_active":"2024-03-21T10:36:56.325149+0000","last_peered":"2024-03-21T10:36:56.325149+0000","last_clean":"2024-03-21T10:36:56.325149+0000","last_became_active":"2024-03-21T10:35:06.015439+0000","last_became_peered":"2024-03-21T10:35:06.015439+0000","last_unstale":"2024-03-21T10:36:56.325149+0000","last_undegraded":"2024-03-21T10:36:56.325149+0000","last_fullsized":"2024-03-21T10:36:56.325149+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:05:23.221826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618846+0000","last_change":"2024-03-21T10:35:06.929241+0000","last_active":"2024-03-21T10:36:56.618846+0000","last_peered":"2024-03-21T10:36:56.618846+0000","last_clean":"2024-03-21T10:36:56.618846+0000","last_became_active":"2024-03-21T10:35:06.928943+0000","last_became_peered":"2024-03-21T10:35:06.928943+0000","last_unstale":"2024-03-21T10:36:56.618846+0000","last_undegraded":"2024-03-21T10:36:56.618846+0000","last_fullsized":"2024-03-21T10:36:56.618846+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:24:10.484540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540706+0000","last_change":"2024-03-21T10:35:11.195929+0000","last_active":"2024-03-21T10:36:56.540706+0000","last_peered":"2024-03-21T10:36:56.540706+0000","last_clean":"2024-03-21T10:36:56.540706+0000","last_became_active":"2024-03-21T10:35:11.195489+0000","last_became_peered":"2024-03-21T10:35:11.195489+0000","last_unstale":"2024-03-21T10:36:56.540706+0000","last_undegraded":"2024-03-21T10:36:56.540706+0000","last_fullsized":"2024-03-21T10:36:56.540706+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:41:15.137917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618718+0000","last_change":"2024-03-21T10:35:06.002127+0000","last_active":"2024-03-21T10:36:56.618718+0000","last_peered":"2024-03-21T10:36:56.618718+0000","last_clean":"2024-03-21T10:36:56.618718+0000","last_became_active":"2024-03-21T10:35:06.001706+0000","last_became_peered":"2024-03-21T10:35:06.001706+0000","last_unstale":"2024-03-21T10:36:56.618718+0000","last_undegraded":"2024-03-21T10:36:56.618718+0000","last_fullsized":"2024-03-21T10:36:56.618718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:26:32.844991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325187+0000","last_change":"2024-03-21T10:35:06.925945+0000","last_active":"2024-03-21T10:36:56.325187+0000","last_peered":"2024-03-21T10:36:56.325187+0000","last_clean":"2024-03-21T10:36:56.325187+0000","last_became_active":"2024-03-21T10:35:06.925551+0000","last_became_peered":"2024-03-21T10:35:06.925551+0000","last_unstale":"2024-03-21T10:36:56.325187+0000","last_undegraded":"2024-03-21T10:36:56.325187+0000","last_fullsized":"2024-03-21T10:36:56.325187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:24:10.212676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540914+0000","last_change":"2024-03-21T10:35:11.229067+0000","last_active":"2024-03-21T10:36:56.540914+0000","last_peered":"2024-03-21T10:36:56.540914+0000","last_clean":"2024-03-21T10:36:56.540914+0000","last_became_active":"2024-03-21T10:35:11.228628+0000","last_became_peered":"2024-03-21T10:35:11.228628+0000","last_unstale":"2024-03-21T10:36:56.540914+0000","last_undegraded":"2024-03-21T10:36:56.540914+0000","last_fullsized":"2024-03-21T10:36:56.540914+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:45:05.473277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618958+0000","last_change":"2024-03-21T10:35:55.384922+0000","last_active":"2024-03-21T10:36:56.618958+0000","last_peered":"2024-03-21T10:36:56.618958+0000","last_clean":"2024-03-21T10:36:56.618958+0000","last_became_active":"2024-03-21T10:35:55.380952+0000","last_became_peered":"2024-03-21T10:35:55.380952+0000","last_unstale":"2024-03-21T10:36:56.618958+0000","last_undegraded":"2024-03-21T10:36:56.618958+0000","last_fullsized":"2024-03-21T10:36:56.618958+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:29:37.873699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619644+0000","last_change":"2024-03-21T10:35:06.928049+0000","last_active":"2024-03-21T10:36:56.619644+0000","last_peered":"2024-03-21T10:36:56.619644+0000","last_clean":"2024-03-21T10:36:56.619644+0000","last_became_active":"2024-03-21T10:35:06.927717+0000","last_became_peered":"2024-03-21T10:35:06.927717+0000","last_unstale":"2024-03-21T10:36:56.619644+0000","last_undegraded":"2024-03-21T10:36:56.619644+0000","last_fullsized":"2024-03-21T10:36:56.619644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:27:47.554668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619381+0000","last_change":"2024-03-21T10:35:11.181090+0000","last_active":"2024-03-21T10:36:56.619381+0000","last_peered":"2024-03-21T10:36:56.619381+0000","last_clean":"2024-03-21T10:36:56.619381+0000","last_became_active":"2024-03-21T10:35:11.180511+0000","last_became_peered":"2024-03-21T10:35:11.180511+0000","last_unstale":"2024-03-21T10:36:56.619381+0000","last_undegraded":"2024-03-21T10:36:56.619381+0000","last_fullsized":"2024-03-21T10:36:56.619381+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:21:51.517845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619595+0000","last_change":"2024-03-21T10:35:06.007828+0000","last_active":"2024-03-21T10:36:56.619595+0000","last_peered":"2024-03-21T10:36:56.619595+0000","last_clean":"2024-03-21T10:36:56.619595+0000","last_became_active":"2024-03-21T10:35:06.007490+0000","last_became_peered":"2024-03-21T10:35:06.007490+0000","last_unstale":"2024-03-21T10:36:56.619595+0000","last_undegraded":"2024-03-21T10:36:56.619595+0000","last_fullsized":"2024-03-21T10:36:56.619595+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:08:18.579308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619840+0000","last_change":"2024-03-21T10:35:06.909313+0000","last_active":"2024-03-21T10:36:56.619840+0000","last_peered":"2024-03-21T10:36:56.619840+0000","last_clean":"2024-03-21T10:36:56.619840+0000","last_became_active":"2024-03-21T10:35:06.908555+0000","last_became_peered":"2024-03-21T10:35:06.908555+0000","last_unstale":"2024-03-21T10:36:56.619840+0000","last_undegraded":"2024-03-21T10:36:56.619840+0000","last_fullsized":"2024-03-21T10:36:56.619840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:09:57.961971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.622374+0000","last_change":"2024-03-21T10:35:11.195996+0000","last_active":"2024-03-21T10:36:56.622374+0000","last_peered":"2024-03-21T10:36:56.622374+0000","last_clean":"2024-03-21T10:36:56.622374+0000","last_became_active":"2024-03-21T10:35:11.195506+0000","last_became_peered":"2024-03-21T10:35:11.195506+0000","last_unstale":"2024-03-21T10:36:56.622374+0000","last_undegraded":"2024-03-21T10:36:56.622374+0000","last_fullsized":"2024-03-21T10:36:56.622374+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:49:33.010899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311586+0000","last_change":"2024-03-21T10:35:06.011498+0000","last_active":"2024-03-21T10:36:56.311586+0000","last_peered":"2024-03-21T10:36:56.311586+0000","last_clean":"2024-03-21T10:36:56.311586+0000","last_became_active":"2024-03-21T10:35:06.011203+0000","last_became_peered":"2024-03-21T10:35:06.011203+0000","last_unstale":"2024-03-21T10:36:56.311586+0000","last_undegraded":"2024-03-21T10:36:56.311586+0000","last_fullsized":"2024-03-21T10:36:56.311586+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:28:11.369740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619130+0000","last_change":"2024-03-21T10:35:06.935514+0000","last_active":"2024-03-21T10:36:56.619130+0000","last_peered":"2024-03-21T10:36:56.619130+0000","last_clean":"2024-03-21T10:36:56.619130+0000","last_became_active":"2024-03-21T10:35:06.935174+0000","last_became_peered":"2024-03-21T10:35:06.935174+0000","last_unstale":"2024-03-21T10:36:56.619130+0000","last_undegraded":"2024-03-21T10:36:56.619130+0000","last_fullsized":"2024-03-21T10:36:56.619130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:35:42.884471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621386+0000","last_change":"2024-03-21T10:36:56.621053+0000","last_active":"2024-03-21T10:36:56.621386+0000","last_peered":"2024-03-21T10:36:56.621386+0000","last_clean":"2024-03-21T10:36:56.621386+0000","last_became_active":"2024-03-21T10:36:56.620688+0000","last_became_peered":"2024-03-21T10:36:56.620688+0000","last_unstale":"2024-03-21T10:36:56.621386+0000","last_undegraded":"2024-03-21T10:36:56.621386+0000","last_fullsized":"2024-03-21T10:36:56.621386+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:50:02.121787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313022+0000","last_change":"2024-03-21T10:35:06.007216+0000","last_active":"2024-03-21T10:36:56.313022+0000","last_peered":"2024-03-21T10:36:56.313022+0000","last_clean":"2024-03-21T10:36:56.313022+0000","last_became_active":"2024-03-21T10:35:06.006883+0000","last_became_peered":"2024-03-21T10:35:06.006883+0000","last_unstale":"2024-03-21T10:36:56.313022+0000","last_undegraded":"2024-03-21T10:36:56.313022+0000","last_fullsized":"2024-03-21T10:36:56.313022+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:46:45.310919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619803+0000","last_change":"2024-03-21T10:35:06.906017+0000","last_active":"2024-03-21T10:36:56.619803+0000","last_peered":"2024-03-21T10:36:56.619803+0000","last_clean":"2024-03-21T10:36:56.619803+0000","last_became_active":"2024-03-21T10:35:06.905648+0000","last_became_peered":"2024-03-21T10:35:06.905648+0000","last_unstale":"2024-03-21T10:36:56.619803+0000","last_undegraded":"2024-03-21T10:36:56.619803+0000","last_fullsized":"2024-03-21T10:36:56.619803+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:56:50.380146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618954+0000","last_change":"2024-03-21T10:35:06.905186+0000","last_active":"2024-03-21T10:36:56.618954+0000","last_peered":"2024-03-21T10:36:56.618954+0000","last_clean":"2024-03-21T10:36:56.618954+0000","last_became_active":"2024-03-21T10:35:06.904835+0000","last_became_peered":"2024-03-21T10:35:06.904835+0000","last_unstale":"2024-03-21T10:36:56.618954+0000","last_undegraded":"2024-03-21T10:36:56.618954+0000","last_fullsized":"2024-03-21T10:36:56.618954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:53:19.293437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619036+0000","last_change":"2024-03-21T10:35:06.903318+0000","last_active":"2024-03-21T10:36:56.619036+0000","last_peered":"2024-03-21T10:36:56.619036+0000","last_clean":"2024-03-21T10:36:56.619036+0000","last_became_active":"2024-03-21T10:35:06.902994+0000","last_became_peered":"2024-03-21T10:35:06.902994+0000","last_unstale":"2024-03-21T10:36:56.619036+0000","last_undegraded":"2024-03-21T10:36:56.619036+0000","last_fullsized":"2024-03-21T10:36:56.619036+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:21:27.412565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618679+0000","last_change":"2024-03-21T10:35:06.934582+0000","last_active":"2024-03-21T10:36:56.618679+0000","last_peered":"2024-03-21T10:36:56.618679+0000","last_clean":"2024-03-21T10:36:56.618679+0000","last_became_active":"2024-03-21T10:35:06.934311+0000","last_became_peered":"2024-03-21T10:35:06.934311+0000","last_unstale":"2024-03-21T10:36:56.618679+0000","last_undegraded":"2024-03-21T10:36:56.618679+0000","last_fullsized":"2024-03-21T10:36:56.618679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:05:30.785830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.622973+0000","last_change":"2024-03-21T10:35:06.944221+0000","last_active":"2024-03-21T10:36:56.622973+0000","last_peered":"2024-03-21T10:36:56.622973+0000","last_clean":"2024-03-21T10:36:56.622973+0000","last_became_active":"2024-03-21T10:35:06.943732+0000","last_became_peered":"2024-03-21T10:35:06.943732+0000","last_unstale":"2024-03-21T10:36:56.622973+0000","last_undegraded":"2024-03-21T10:36:56.622973+0000","last_fullsized":"2024-03-21T10:36:56.622973+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:33:03.653160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313346+0000","last_change":"2024-03-21T10:35:06.940832+0000","last_active":"2024-03-21T10:36:56.313346+0000","last_peered":"2024-03-21T10:36:56.313346+0000","last_clean":"2024-03-21T10:36:56.313346+0000","last_became_active":"2024-03-21T10:35:06.939565+0000","last_became_peered":"2024-03-21T10:35:06.939565+0000","last_unstale":"2024-03-21T10:36:56.313346+0000","last_undegraded":"2024-03-21T10:36:56.313346+0000","last_fullsized":"2024-03-21T10:36:56.313346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:57:27.743704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619590+0000","last_change":"2024-03-21T10:35:06.907753+0000","last_active":"2024-03-21T10:36:56.619590+0000","last_peered":"2024-03-21T10:36:56.619590+0000","last_clean":"2024-03-21T10:36:56.619590+0000","last_became_active":"2024-03-21T10:35:06.907432+0000","last_became_peered":"2024-03-21T10:35:06.907432+0000","last_unstale":"2024-03-21T10:36:56.619590+0000","last_undegraded":"2024-03-21T10:36:56.619590+0000","last_fullsized":"2024-03-21T10:36:56.619590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:31:02.528855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619670+0000","last_change":"2024-03-21T10:35:06.908597+0000","last_active":"2024-03-21T10:36:56.619670+0000","last_peered":"2024-03-21T10:36:56.619670+0000","last_clean":"2024-03-21T10:36:56.619670+0000","last_became_active":"2024-03-21T10:35:06.908282+0000","last_became_peered":"2024-03-21T10:35:06.908282+0000","last_unstale":"2024-03-21T10:36:56.619670+0000","last_undegraded":"2024-03-21T10:36:56.619670+0000","last_fullsized":"2024-03-21T10:36:56.619670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:42:18.221045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.326006+0000","last_change":"2024-03-21T10:35:06.931015+0000","last_active":"2024-03-21T10:36:56.326006+0000","last_peered":"2024-03-21T10:36:56.326006+0000","last_clean":"2024-03-21T10:36:56.326006+0000","last_became_active":"2024-03-21T10:35:06.930209+0000","last_became_peered":"2024-03-21T10:35:06.930209+0000","last_unstale":"2024-03-21T10:36:56.326006+0000","last_undegraded":"2024-03-21T10:36:56.326006+0000","last_fullsized":"2024-03-21T10:36:56.326006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:42:15.277874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311263+0000","last_change":"2024-03-21T10:35:06.930070+0000","last_active":"2024-03-21T10:36:56.311263+0000","last_peered":"2024-03-21T10:36:56.311263+0000","last_clean":"2024-03-21T10:36:56.311263+0000","last_became_active":"2024-03-21T10:35:06.929773+0000","last_became_peered":"2024-03-21T10:35:06.929773+0000","last_unstale":"2024-03-21T10:36:56.311263+0000","last_undegraded":"2024-03-21T10:36:56.311263+0000","last_fullsized":"2024-03-21T10:36:56.311263+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:17:25.176666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325363+0000","last_change":"2024-03-21T10:35:06.929575+0000","last_active":"2024-03-21T10:36:56.325363+0000","last_peered":"2024-03-21T10:36:56.325363+0000","last_clean":"2024-03-21T10:36:56.325363+0000","last_became_active":"2024-03-21T10:35:06.929240+0000","last_became_peered":"2024-03-21T10:35:06.929240+0000","last_unstale":"2024-03-21T10:36:56.325363+0000","last_undegraded":"2024-03-21T10:36:56.325363+0000","last_fullsized":"2024-03-21T10:36:56.325363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:03:40.697640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619625+0000","last_change":"2024-03-21T10:35:06.934693+0000","last_active":"2024-03-21T10:36:56.619625+0000","last_peered":"2024-03-21T10:36:56.619625+0000","last_clean":"2024-03-21T10:36:56.619625+0000","last_became_active":"2024-03-21T10:35:06.934361+0000","last_became_peered":"2024-03-21T10:35:06.934361+0000","last_unstale":"2024-03-21T10:36:56.619625+0000","last_undegraded":"2024-03-21T10:36:56.619625+0000","last_fullsized":"2024-03-21T10:36:56.619625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:14:00.954370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312500+0000","last_change":"2024-03-21T10:35:06.941095+0000","last_active":"2024-03-21T10:36:56.312500+0000","last_peered":"2024-03-21T10:36:56.312500+0000","last_clean":"2024-03-21T10:36:56.312500+0000","last_became_active":"2024-03-21T10:35:06.939348+0000","last_became_peered":"2024-03-21T10:35:06.939348+0000","last_unstale":"2024-03-21T10:36:56.312500+0000","last_undegraded":"2024-03-21T10:36:56.312500+0000","last_fullsized":"2024-03-21T10:36:56.312500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:31:43.783592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619193+0000","last_change":"2024-03-21T10:35:06.908340+0000","last_active":"2024-03-21T10:36:56.619193+0000","last_peered":"2024-03-21T10:36:56.619193+0000","last_clean":"2024-03-21T10:36:56.619193+0000","last_became_active":"2024-03-21T10:35:06.908043+0000","last_became_peered":"2024-03-21T10:35:06.908043+0000","last_unstale":"2024-03-21T10:36:56.619193+0000","last_undegraded":"2024-03-21T10:36:56.619193+0000","last_fullsized":"2024-03-21T10:36:56.619193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:50:33.167396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282863,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69209,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":957,"num_read_kb":553,"num_write":180,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":207627,"data_compressed":5487,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":154,"ondisk_log_size":154,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383733,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1019904,"data_stored":519397,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645370,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32312,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712840,"statfs":{"total":95995035648,"available":95961948160,"internally_reserved":0,"allocated":1163264,"data_stored":1171732,"data_compressed":64463,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808895,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":523729,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070535,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1036288,"data_stored":519553,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332172,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1085440,"data_stored":1115909,"data_compressed":62634,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495698,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":514809,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574959,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1081344,"data_stored":579480,"data_compressed":28489,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738484,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":516197,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411432,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1105920,"data_stored":581362,"data_compressed":28489,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182498,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":515003,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920860,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":519765,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757335,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1093632,"data_stored":1111191,"data_compressed":62634,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34331,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34331,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34331,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":64761,"data_compressed":1829,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":66643,"data_compressed":1829,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":66645,"data_compressed":1829,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T10:38:19.071 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-21T10:38:19.071 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-21T10:38:19.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:18 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1744317026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:38:19.081 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:18 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1744317026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:38:19.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.327+0000 7f2585c00640 1 -- 172.21.15.31:0/370798021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2580102790 msgr2=0x7f2580104c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:19.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.327+0000 7f2585c00640 1 --2- 172.21.15.31:0/370798021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2580102790 0x7f2580104c00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f256c009900 tx=0x7f256c02bc50 comp rx=0 tx=0).stop 2024-03-21T10:38:19.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.327+0000 7f2585c00640 1 -- 172.21.15.31:0/370798021 shutdown_connections 2024-03-21T10:38:19.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.327+0000 7f2585c00640 1 --2- 172.21.15.31:0/370798021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2580105140 0x7f25801055c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:19.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.327+0000 7f2585c00640 1 --2- 172.21.15.31:0/370798021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2580102790 0x7f2580104c00 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:19.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.327+0000 7f2585c00640 1 --2- 172.21.15.31:0/370798021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25800ffe20 0x7f2580102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:19.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.327+0000 7f2585c00640 1 -- 172.21.15.31:0/370798021 >> 172.21.15.31:0/370798021 conn(0x7f25800fba20 msgr2=0x7f25800fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:19.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.328+0000 7f2585c00640 1 -- 172.21.15.31:0/370798021 shutdown_connections 2024-03-21T10:38:19.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.328+0000 7f2585c00640 1 -- 172.21.15.31:0/370798021 wait complete. 2024-03-21T10:38:19.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.328+0000 7f2585c00640 1 Processor -- start 2024-03-21T10:38:19.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.329+0000 7f2585c00640 1 -- start start 2024-03-21T10:38:19.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.329+0000 7f2585c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25800ffe20 0x7f258019de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:19.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.329+0000 7f2585c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2580102790 0x7f258019e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:19.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.329+0000 7f2585c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2580105140 0x7f25801a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:19.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.329+0000 7f2585c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2580113850 con 0x7f2580102790 2024-03-21T10:38:19.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.329+0000 7f2585c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25801136d0 con 0x7f2580105140 2024-03-21T10:38:19.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.329+0000 7f2585c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25801139d0 con 0x7f25800ffe20 2024-03-21T10:38:19.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.329+0000 7f257fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25800ffe20 0x7f258019de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:19.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.329+0000 7f257f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2580102790 0x7f258019e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:19.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.329+0000 7f2585200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2580105140 0x7f25801a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:19.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.329+0000 7f257fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25800ffe20 0x7f258019de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53522/0 (socket says 172.21.15.31:53522) 2024-03-21T10:38:19.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.329+0000 7f2585200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2580105140 0x7f25801a5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46082/0 (socket says 172.21.15.31:46082) 2024-03-21T10:38:19.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.329+0000 7f257fe00640 1 -- 172.21.15.31:0/3325402456 learned_addr learned my addr 172.21.15.31:0/3325402456 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:19.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.330+0000 7f257fe00640 1 -- 172.21.15.31:0/3325402456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2580105140 msgr2=0x7f25801a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:19.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.330+0000 7f257fe00640 1 --2- 172.21.15.31:0/3325402456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2580105140 0x7f25801a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:19.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.330+0000 7f257fe00640 1 -- 172.21.15.31:0/3325402456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2580102790 msgr2=0x7f258019e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:19.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.330+0000 7f257fe00640 1 --2- 172.21.15.31:0/3325402456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2580102790 0x7f258019e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:19.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.330+0000 7f257fe00640 1 -- 172.21.15.31:0/3325402456 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25801a5aa0 con 0x7f25800ffe20 2024-03-21T10:38:19.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.330+0000 7f2585200640 1 --2- 172.21.15.31:0/3325402456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2580105140 0x7f25801a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:19.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.330+0000 7f257fe00640 1 --2- 172.21.15.31:0/3325402456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25800ffe20 0x7f258019de80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f257000e9f0 tx=0x7f257000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:19.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.330+0000 7f257cc00640 1 -- 172.21.15.31:0/3325402456 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257000cd80 con 0x7f25800ffe20 2024-03-21T10:38:19.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.330+0000 7f257cc00640 1 -- 172.21.15.31:0/3325402456 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2570010910 con 0x7f25800ffe20 2024-03-21T10:38:19.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.330+0000 7f257cc00640 1 -- 172.21.15.31:0/3325402456 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2570018990 con 0x7f25800ffe20 2024-03-21T10:38:19.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.331+0000 7f2585c00640 1 -- 172.21.15.31:0/3325402456 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25801a5d90 con 0x7f25800ffe20 2024-03-21T10:38:19.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.331+0000 7f2585c00640 1 -- 172.21.15.31:0/3325402456 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25801a61a0 con 0x7f25800ffe20 2024-03-21T10:38:19.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.332+0000 7f2585c00640 1 -- 172.21.15.31:0/3325402456 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25800fd510 con 0x7f25800ffe20 2024-03-21T10:38:19.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.334+0000 7f257cc00640 1 -- 172.21.15.31:0/3325402456 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f2570010ac0 con 0x7f25800ffe20 2024-03-21T10:38:19.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.335+0000 7f257cc00640 1 --2- 172.21.15.31:0/3325402456 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f25640a2f90 0x7f25640a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:19.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.335+0000 7f257cc00640 1 -- 172.21.15.31:0/3325402456 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f25700cfa10 con 0x7f25800ffe20 2024-03-21T10:38:19.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.336+0000 7f257f400640 1 --2- 172.21.15.31:0/3325402456 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f25640a2f90 0x7f25640a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:19.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.336+0000 7f257f400640 1 --2- 172.21.15.31:0/3325402456 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f25640a2f90 0x7f25640a5450 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f256c02c260 tx=0x7f256c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:19.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.339+0000 7f257cc00640 1 -- 172.21.15.31:0/3325402456 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f257009be30 con 0x7f25800ffe20 2024-03-21T10:38:19.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.550+0000 7f2585c00640 1 -- 172.21.15.31:0/3325402456 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2580061ae0 con 0x7f25640a2f90 2024-03-21T10:38:19.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.568+0000 7f257cc00640 1 -- 172.21.15.31:0/3325402456 <== mgr.14152 v2:172.21.15.31:6800/3875791078 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340815 (secure 0 0 0) 0x7f2580061ae0 con 0x7f25640a2f90 2024-03-21T10:38:19.570 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:38:19.572 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-03-21T10:38:19.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.574+0000 7f2585c00640 1 -- 172.21.15.31:0/3325402456 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f25640a2f90 msgr2=0x7f25640a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:19.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.574+0000 7f2585c00640 1 --2- 172.21.15.31:0/3325402456 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f25640a2f90 0x7f25640a5450 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f256c02c260 tx=0x7f256c000bc0 comp rx=0 tx=0).stop 2024-03-21T10:38:19.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.574+0000 7f2585c00640 1 -- 172.21.15.31:0/3325402456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25800ffe20 msgr2=0x7f258019de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:19.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.574+0000 7f2585c00640 1 --2- 172.21.15.31:0/3325402456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25800ffe20 0x7f258019de80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f257000e9f0 tx=0x7f257000eeb0 comp rx=0 tx=0).stop 2024-03-21T10:38:19.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.574+0000 7f2585c00640 1 -- 172.21.15.31:0/3325402456 shutdown_connections 2024-03-21T10:38:19.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.574+0000 7f2585c00640 1 --2- 172.21.15.31:0/3325402456 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f25640a2f90 0x7f25640a5450 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:19.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.574+0000 7f2585c00640 1 --2- 172.21.15.31:0/3325402456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2580105140 0x7f25801a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:19.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.574+0000 7f2585c00640 1 --2- 172.21.15.31:0/3325402456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2580102790 0x7f258019e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:19.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.574+0000 7f2585c00640 1 --2- 172.21.15.31:0/3325402456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25800ffe20 0x7f258019de80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:19.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.574+0000 7f2585c00640 1 -- 172.21.15.31:0/3325402456 >> 172.21.15.31:0/3325402456 conn(0x7f25800fba20 msgr2=0x7f25801007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:19.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.575+0000 7f2585c00640 1 -- 172.21.15.31:0/3325402456 shutdown_connections 2024-03-21T10:38:19.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.575+0000 7f2585c00640 1 -- 172.21.15.31:0/3325402456 wait complete. 2024-03-21T10:38:19.592 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2024-03-21T10:38:18.076535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659811,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":1093,"num_read_kb":672,"num_write":363,"num_write_kb":3069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":793,"ondisk_log_size":793,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369956,"kb_used_data":12440,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571868,"statfs":{"total":1151940427776,"available":1151561592832,"internally_reserved":0,"allocated":12738560,"data_stored":8188127,"data_compressed":433329,"data_compressed_allocated":589824,"data_compressed_original":2678784,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":612,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":32,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005529"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619666+0000","last_change":"2024-03-21T10:35:06.923544+0000","last_active":"2024-03-21T10:36:56.619666+0000","last_peered":"2024-03-21T10:36:56.619666+0000","last_clean":"2024-03-21T10:36:56.619666+0000","last_became_active":"2024-03-21T10:35:06.923254+0000","last_became_peered":"2024-03-21T10:35:06.923254+0000","last_unstale":"2024-03-21T10:36:56.619666+0000","last_undegraded":"2024-03-21T10:36:56.619666+0000","last_fullsized":"2024-03-21T10:36:56.619666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:38:14.573590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.622736+0000","last_change":"2024-03-21T10:35:06.929242+0000","last_active":"2024-03-21T10:36:56.622736+0000","last_peered":"2024-03-21T10:36:56.622736+0000","last_clean":"2024-03-21T10:36:56.622736+0000","last_became_active":"2024-03-21T10:35:06.927078+0000","last_became_peered":"2024-03-21T10:35:06.927078+0000","last_unstale":"2024-03-21T10:36:56.622736+0000","last_undegraded":"2024-03-21T10:36:56.622736+0000","last_fullsized":"2024-03-21T10:36:56.622736+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:27:10.347945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621857+0000","last_change":"2024-03-21T10:35:06.927792+0000","last_active":"2024-03-21T10:36:56.621857+0000","last_peered":"2024-03-21T10:36:56.621857+0000","last_clean":"2024-03-21T10:36:56.621857+0000","last_became_active":"2024-03-21T10:35:06.927386+0000","last_became_peered":"2024-03-21T10:35:06.927386+0000","last_unstale":"2024-03-21T10:36:56.621857+0000","last_undegraded":"2024-03-21T10:36:56.621857+0000","last_fullsized":"2024-03-21T10:36:56.621857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:53:51.310283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540864+0000","last_change":"2024-03-21T10:35:06.908901+0000","last_active":"2024-03-21T10:36:56.540864+0000","last_peered":"2024-03-21T10:36:56.540864+0000","last_clean":"2024-03-21T10:36:56.540864+0000","last_became_active":"2024-03-21T10:35:06.908586+0000","last_became_peered":"2024-03-21T10:35:06.908586+0000","last_unstale":"2024-03-21T10:36:56.540864+0000","last_undegraded":"2024-03-21T10:36:56.540864+0000","last_fullsized":"2024-03-21T10:36:56.540864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:25:57.387769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312725+0000","last_change":"2024-03-21T10:35:06.932537+0000","last_active":"2024-03-21T10:36:56.312725+0000","last_peered":"2024-03-21T10:36:56.312725+0000","last_clean":"2024-03-21T10:36:56.312725+0000","last_became_active":"2024-03-21T10:35:06.932185+0000","last_became_peered":"2024-03-21T10:35:06.932185+0000","last_unstale":"2024-03-21T10:36:56.312725+0000","last_undegraded":"2024-03-21T10:36:56.312725+0000","last_fullsized":"2024-03-21T10:36:56.312725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:30:25.127073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311445+0000","last_change":"2024-03-21T10:35:06.922765+0000","last_active":"2024-03-21T10:36:56.311445+0000","last_peered":"2024-03-21T10:36:56.311445+0000","last_clean":"2024-03-21T10:36:56.311445+0000","last_became_active":"2024-03-21T10:35:06.922447+0000","last_became_peered":"2024-03-21T10:35:06.922447+0000","last_unstale":"2024-03-21T10:36:56.311445+0000","last_undegraded":"2024-03-21T10:36:56.311445+0000","last_fullsized":"2024-03-21T10:36:56.311445+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:37:33.310735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619036+0000","last_change":"2024-03-21T10:35:06.896777+0000","last_active":"2024-03-21T10:36:56.619036+0000","last_peered":"2024-03-21T10:36:56.619036+0000","last_clean":"2024-03-21T10:36:56.619036+0000","last_became_active":"2024-03-21T10:35:06.896544+0000","last_became_peered":"2024-03-21T10:35:06.896544+0000","last_unstale":"2024-03-21T10:36:56.619036+0000","last_undegraded":"2024-03-21T10:36:56.619036+0000","last_fullsized":"2024-03-21T10:36:56.619036+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:34:05.000661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325012+0000","last_change":"2024-03-21T10:35:06.923294+0000","last_active":"2024-03-21T10:36:56.325012+0000","last_peered":"2024-03-21T10:36:56.325012+0000","last_clean":"2024-03-21T10:36:56.325012+0000","last_became_active":"2024-03-21T10:35:06.921320+0000","last_became_peered":"2024-03-21T10:35:06.921320+0000","last_unstale":"2024-03-21T10:36:56.325012+0000","last_undegraded":"2024-03-21T10:36:56.325012+0000","last_fullsized":"2024-03-21T10:36:56.325012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:33:53.855908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619843+0000","last_change":"2024-03-21T10:35:06.912977+0000","last_active":"2024-03-21T10:36:56.619843+0000","last_peered":"2024-03-21T10:36:56.619843+0000","last_clean":"2024-03-21T10:36:56.619843+0000","last_became_active":"2024-03-21T10:35:06.912626+0000","last_became_peered":"2024-03-21T10:35:06.912626+0000","last_unstale":"2024-03-21T10:36:56.619843+0000","last_undegraded":"2024-03-21T10:36:56.619843+0000","last_fullsized":"2024-03-21T10:36:56.619843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:42:13.472454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311923+0000","last_change":"2024-03-21T10:35:06.910252+0000","last_active":"2024-03-21T10:36:56.311923+0000","last_peered":"2024-03-21T10:36:56.311923+0000","last_clean":"2024-03-21T10:36:56.311923+0000","last_became_active":"2024-03-21T10:35:06.909924+0000","last_became_peered":"2024-03-21T10:35:06.909924+0000","last_unstale":"2024-03-21T10:36:56.311923+0000","last_undegraded":"2024-03-21T10:36:56.311923+0000","last_fullsized":"2024-03-21T10:36:56.311923+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:43:15.527349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619732+0000","last_change":"2024-03-21T10:35:06.909569+0000","last_active":"2024-03-21T10:36:56.619732+0000","last_peered":"2024-03-21T10:36:56.619732+0000","last_clean":"2024-03-21T10:36:56.619732+0000","last_became_active":"2024-03-21T10:35:06.908982+0000","last_became_peered":"2024-03-21T10:35:06.908982+0000","last_unstale":"2024-03-21T10:36:56.619732+0000","last_undegraded":"2024-03-21T10:36:56.619732+0000","last_fullsized":"2024-03-21T10:36:56.619732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:13:56.020281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540823+0000","last_change":"2024-03-21T10:35:06.913861+0000","last_active":"2024-03-21T10:36:56.540823+0000","last_peered":"2024-03-21T10:36:56.540823+0000","last_clean":"2024-03-21T10:36:56.540823+0000","last_became_active":"2024-03-21T10:35:06.913490+0000","last_became_peered":"2024-03-21T10:35:06.913490+0000","last_unstale":"2024-03-21T10:36:56.540823+0000","last_undegraded":"2024-03-21T10:36:56.540823+0000","last_fullsized":"2024-03-21T10:36:56.540823+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:24:18.573735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312584+0000","last_change":"2024-03-21T10:35:06.934537+0000","last_active":"2024-03-21T10:36:56.312584+0000","last_peered":"2024-03-21T10:36:56.312584+0000","last_clean":"2024-03-21T10:36:56.312584+0000","last_became_active":"2024-03-21T10:35:06.932951+0000","last_became_peered":"2024-03-21T10:35:06.932951+0000","last_unstale":"2024-03-21T10:36:56.312584+0000","last_undegraded":"2024-03-21T10:36:56.312584+0000","last_fullsized":"2024-03-21T10:36:56.312584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:43:17.748434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621260+0000","last_change":"2024-03-21T10:35:06.925495+0000","last_active":"2024-03-21T10:36:56.621260+0000","last_peered":"2024-03-21T10:36:56.621260+0000","last_clean":"2024-03-21T10:36:56.621260+0000","last_became_active":"2024-03-21T10:35:06.924326+0000","last_became_peered":"2024-03-21T10:35:06.924326+0000","last_unstale":"2024-03-21T10:36:56.621260+0000","last_undegraded":"2024-03-21T10:36:56.621260+0000","last_fullsized":"2024-03-21T10:36:56.621260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:27:19.598213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623885+0000","last_change":"2024-03-21T10:35:06.923695+0000","last_active":"2024-03-21T10:36:56.623885+0000","last_peered":"2024-03-21T10:36:56.623885+0000","last_clean":"2024-03-21T10:36:56.623885+0000","last_became_active":"2024-03-21T10:35:06.919114+0000","last_became_peered":"2024-03-21T10:35:06.919114+0000","last_unstale":"2024-03-21T10:36:56.623885+0000","last_undegraded":"2024-03-21T10:36:56.623885+0000","last_fullsized":"2024-03-21T10:36:56.623885+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:42:49.179653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619136+0000","last_change":"2024-03-21T10:35:06.930225+0000","last_active":"2024-03-21T10:36:56.619136+0000","last_peered":"2024-03-21T10:36:56.619136+0000","last_clean":"2024-03-21T10:36:56.619136+0000","last_became_active":"2024-03-21T10:35:06.929905+0000","last_became_peered":"2024-03-21T10:35:06.929905+0000","last_unstale":"2024-03-21T10:36:56.619136+0000","last_undegraded":"2024-03-21T10:36:56.619136+0000","last_fullsized":"2024-03-21T10:36:56.619136+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:21:07.231703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619516+0000","last_change":"2024-03-21T10:35:06.923498+0000","last_active":"2024-03-21T10:36:56.619516+0000","last_peered":"2024-03-21T10:36:56.619516+0000","last_clean":"2024-03-21T10:36:56.619516+0000","last_became_active":"2024-03-21T10:35:06.923242+0000","last_became_peered":"2024-03-21T10:35:06.923242+0000","last_unstale":"2024-03-21T10:36:56.619516+0000","last_undegraded":"2024-03-21T10:36:56.619516+0000","last_fullsized":"2024-03-21T10:36:56.619516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:42:04.309701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623901+0000","last_change":"2024-03-21T10:35:06.918314+0000","last_active":"2024-03-21T10:36:56.623901+0000","last_peered":"2024-03-21T10:36:56.623901+0000","last_clean":"2024-03-21T10:36:56.623901+0000","last_became_active":"2024-03-21T10:35:06.917865+0000","last_became_peered":"2024-03-21T10:35:06.917865+0000","last_unstale":"2024-03-21T10:36:56.623901+0000","last_undegraded":"2024-03-21T10:36:56.623901+0000","last_fullsized":"2024-03-21T10:36:56.623901+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:53:48.536341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619301+0000","last_change":"2024-03-21T10:35:06.936234+0000","last_active":"2024-03-21T10:36:56.619301+0000","last_peered":"2024-03-21T10:36:56.619301+0000","last_clean":"2024-03-21T10:36:56.619301+0000","last_became_active":"2024-03-21T10:35:06.935958+0000","last_became_peered":"2024-03-21T10:35:06.935958+0000","last_unstale":"2024-03-21T10:36:56.619301+0000","last_undegraded":"2024-03-21T10:36:56.619301+0000","last_fullsized":"2024-03-21T10:36:56.619301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:52:43.509671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618623+0000","last_change":"2024-03-21T10:35:11.198276+0000","last_active":"2024-03-21T10:36:56.618623+0000","last_peered":"2024-03-21T10:36:56.618623+0000","last_clean":"2024-03-21T10:36:56.618623+0000","last_became_active":"2024-03-21T10:35:11.197899+0000","last_became_peered":"2024-03-21T10:35:11.197899+0000","last_unstale":"2024-03-21T10:36:56.618623+0000","last_undegraded":"2024-03-21T10:36:56.618623+0000","last_fullsized":"2024-03-21T10:36:56.618623+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:17:30.499932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623544+0000","last_change":"2024-03-21T10:35:06.010647+0000","last_active":"2024-03-21T10:36:56.623544+0000","last_peered":"2024-03-21T10:36:56.623544+0000","last_clean":"2024-03-21T10:36:56.623544+0000","last_became_active":"2024-03-21T10:35:06.010239+0000","last_became_peered":"2024-03-21T10:35:06.010239+0000","last_unstale":"2024-03-21T10:36:56.623544+0000","last_undegraded":"2024-03-21T10:36:56.623544+0000","last_fullsized":"2024-03-21T10:36:56.623544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:35:53.324044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325857+0000","last_change":"2024-03-21T10:35:06.926398+0000","last_active":"2024-03-21T10:36:56.325857+0000","last_peered":"2024-03-21T10:36:56.325857+0000","last_clean":"2024-03-21T10:36:56.325857+0000","last_became_active":"2024-03-21T10:35:06.925904+0000","last_became_peered":"2024-03-21T10:35:06.925904+0000","last_unstale":"2024-03-21T10:36:56.325857+0000","last_undegraded":"2024-03-21T10:36:56.325857+0000","last_fullsized":"2024-03-21T10:36:56.325857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:37:52.811604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540507+0000","last_change":"2024-03-21T10:35:55.503266+0000","last_active":"2024-03-21T10:36:56.540507+0000","last_peered":"2024-03-21T10:36:56.540507+0000","last_clean":"2024-03-21T10:36:56.540507+0000","last_became_active":"2024-03-21T10:35:55.502694+0000","last_became_peered":"2024-03-21T10:35:55.502694+0000","last_unstale":"2024-03-21T10:36:56.540507+0000","last_undegraded":"2024-03-21T10:36:56.540507+0000","last_fullsized":"2024-03-21T10:36:56.540507+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:56:44.480667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325780+0000","last_change":"2024-03-21T10:35:06.022750+0000","last_active":"2024-03-21T10:36:56.325780+0000","last_peered":"2024-03-21T10:36:56.325780+0000","last_clean":"2024-03-21T10:36:56.325780+0000","last_became_active":"2024-03-21T10:35:06.022093+0000","last_became_peered":"2024-03-21T10:35:06.022093+0000","last_unstale":"2024-03-21T10:36:56.325780+0000","last_undegraded":"2024-03-21T10:36:56.325780+0000","last_fullsized":"2024-03-21T10:36:56.325780+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:53:28.458486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.624188+0000","last_change":"2024-03-21T10:35:06.927594+0000","last_active":"2024-03-21T10:36:56.624188+0000","last_peered":"2024-03-21T10:36:56.624188+0000","last_clean":"2024-03-21T10:36:56.624188+0000","last_became_active":"2024-03-21T10:35:06.926868+0000","last_became_peered":"2024-03-21T10:35:06.926868+0000","last_unstale":"2024-03-21T10:36:56.624188+0000","last_undegraded":"2024-03-21T10:36:56.624188+0000","last_fullsized":"2024-03-21T10:36:56.624188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:07:33.127465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540607+0000","last_change":"2024-03-21T10:35:11.193475+0000","last_active":"2024-03-21T10:36:56.540607+0000","last_peered":"2024-03-21T10:36:56.540607+0000","last_clean":"2024-03-21T10:36:56.540607+0000","last_became_active":"2024-03-21T10:35:11.193054+0000","last_became_peered":"2024-03-21T10:35:11.193054+0000","last_unstale":"2024-03-21T10:36:56.540607+0000","last_undegraded":"2024-03-21T10:36:56.540607+0000","last_fullsized":"2024-03-21T10:36:56.540607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:32:36.564147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313127+0000","last_change":"2024-03-21T10:35:06.004257+0000","last_active":"2024-03-21T10:36:56.313127+0000","last_peered":"2024-03-21T10:36:56.313127+0000","last_clean":"2024-03-21T10:36:56.313127+0000","last_became_active":"2024-03-21T10:35:06.003936+0000","last_became_peered":"2024-03-21T10:35:06.003936+0000","last_unstale":"2024-03-21T10:36:56.313127+0000","last_undegraded":"2024-03-21T10:36:56.313127+0000","last_fullsized":"2024-03-21T10:36:56.313127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:48:48.608979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.620162+0000","last_change":"2024-03-21T10:35:06.941681+0000","last_active":"2024-03-21T10:36:56.620162+0000","last_peered":"2024-03-21T10:36:56.620162+0000","last_clean":"2024-03-21T10:36:56.620162+0000","last_became_active":"2024-03-21T10:35:06.941314+0000","last_became_peered":"2024-03-21T10:35:06.941314+0000","last_unstale":"2024-03-21T10:36:56.620162+0000","last_undegraded":"2024-03-21T10:36:56.620162+0000","last_fullsized":"2024-03-21T10:36:56.620162+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:46:09.948922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540509+0000","last_change":"2024-03-21T10:35:11.188472+0000","last_active":"2024-03-21T10:36:56.540509+0000","last_peered":"2024-03-21T10:36:56.540509+0000","last_clean":"2024-03-21T10:36:56.540509+0000","last_became_active":"2024-03-21T10:35:11.188072+0000","last_became_peered":"2024-03-21T10:35:11.188072+0000","last_unstale":"2024-03-21T10:36:56.540509+0000","last_undegraded":"2024-03-21T10:36:56.540509+0000","last_fullsized":"2024-03-21T10:36:56.540509+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:04:16.911334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313102+0000","last_change":"2024-03-21T10:35:05.998493+0000","last_active":"2024-03-21T10:36:56.313102+0000","last_peered":"2024-03-21T10:36:56.313102+0000","last_clean":"2024-03-21T10:36:56.313102+0000","last_became_active":"2024-03-21T10:35:05.998134+0000","last_became_peered":"2024-03-21T10:35:05.998134+0000","last_unstale":"2024-03-21T10:36:56.313102+0000","last_undegraded":"2024-03-21T10:36:56.313102+0000","last_fullsized":"2024-03-21T10:36:56.313102+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:05:32.033328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311467+0000","last_change":"2024-03-21T10:35:06.912973+0000","last_active":"2024-03-21T10:36:56.311467+0000","last_peered":"2024-03-21T10:36:56.311467+0000","last_clean":"2024-03-21T10:36:56.311467+0000","last_became_active":"2024-03-21T10:35:06.912382+0000","last_became_peered":"2024-03-21T10:35:06.912382+0000","last_unstale":"2024-03-21T10:36:56.311467+0000","last_undegraded":"2024-03-21T10:36:56.311467+0000","last_fullsized":"2024-03-21T10:36:56.311467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:13:23.757805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619140+0000","last_change":"2024-03-21T10:35:11.180806+0000","last_active":"2024-03-21T10:36:56.619140+0000","last_peered":"2024-03-21T10:36:56.619140+0000","last_clean":"2024-03-21T10:36:56.619140+0000","last_became_active":"2024-03-21T10:35:11.180303+0000","last_became_peered":"2024-03-21T10:35:11.180303+0000","last_unstale":"2024-03-21T10:36:56.619140+0000","last_undegraded":"2024-03-21T10:36:56.619140+0000","last_fullsized":"2024-03-21T10:36:56.619140+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:27:11.719654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":196,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:38:13.155079+0000","last_change":"2024-03-21T10:35:06.004555+0000","last_active":"2024-03-21T10:38:13.155079+0000","last_peered":"2024-03-21T10:38:13.155079+0000","last_clean":"2024-03-21T10:38:13.155079+0000","last_became_active":"2024-03-21T10:35:06.003926+0000","last_became_peered":"2024-03-21T10:35:06.003926+0000","last_unstale":"2024-03-21T10:38:13.155079+0000","last_undegraded":"2024-03-21T10:38:13.155079+0000","last_fullsized":"2024-03-21T10:38:13.155079+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:05:49.721000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":158,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312743+0000","last_change":"2024-03-21T10:35:06.913556+0000","last_active":"2024-03-21T10:36:56.312743+0000","last_peered":"2024-03-21T10:36:56.312743+0000","last_clean":"2024-03-21T10:36:56.312743+0000","last_became_active":"2024-03-21T10:35:06.913213+0000","last_became_peered":"2024-03-21T10:35:06.913213+0000","last_unstale":"2024-03-21T10:36:56.312743+0000","last_undegraded":"2024-03-21T10:36:56.312743+0000","last_fullsized":"2024-03-21T10:36:56.312743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:11:09.235927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619374+0000","last_change":"2024-03-21T10:35:55.523819+0000","last_active":"2024-03-21T10:36:56.619374+0000","last_peered":"2024-03-21T10:36:56.619374+0000","last_clean":"2024-03-21T10:36:56.619374+0000","last_became_active":"2024-03-21T10:35:55.523333+0000","last_became_peered":"2024-03-21T10:35:55.523333+0000","last_unstale":"2024-03-21T10:36:56.619374+0000","last_undegraded":"2024-03-21T10:36:56.619374+0000","last_fullsized":"2024-03-21T10:36:56.619374+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:25:36.277481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311347+0000","last_change":"2024-03-21T10:35:06.012527+0000","last_active":"2024-03-21T10:36:56.311347+0000","last_peered":"2024-03-21T10:36:56.311347+0000","last_clean":"2024-03-21T10:36:56.311347+0000","last_became_active":"2024-03-21T10:35:06.012228+0000","last_became_peered":"2024-03-21T10:35:06.012228+0000","last_unstale":"2024-03-21T10:36:56.311347+0000","last_undegraded":"2024-03-21T10:36:56.311347+0000","last_fullsized":"2024-03-21T10:36:56.311347+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:52:34.622306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.622381+0000","last_change":"2024-03-21T10:35:55.503531+0000","last_active":"2024-03-21T10:36:56.622381+0000","last_peered":"2024-03-21T10:36:56.622381+0000","last_clean":"2024-03-21T10:36:56.622381+0000","last_became_active":"2024-03-21T10:35:55.502793+0000","last_became_peered":"2024-03-21T10:35:55.502793+0000","last_unstale":"2024-03-21T10:36:56.622381+0000","last_undegraded":"2024-03-21T10:36:56.622381+0000","last_fullsized":"2024-03-21T10:36:56.622381+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:08:39.599084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,3],"acting":[9,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619509+0000","last_change":"2024-03-21T10:35:11.196543+0000","last_active":"2024-03-21T10:36:56.619509+0000","last_peered":"2024-03-21T10:36:56.619509+0000","last_clean":"2024-03-21T10:36:56.619509+0000","last_became_active":"2024-03-21T10:35:11.196151+0000","last_became_peered":"2024-03-21T10:35:11.196151+0000","last_unstale":"2024-03-21T10:36:56.619509+0000","last_undegraded":"2024-03-21T10:36:56.619509+0000","last_fullsized":"2024-03-21T10:36:56.619509+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:15:39.193647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312200+0000","last_change":"2024-03-21T10:35:06.004588+0000","last_active":"2024-03-21T10:36:56.312200+0000","last_peered":"2024-03-21T10:36:56.312200+0000","last_clean":"2024-03-21T10:36:56.312200+0000","last_became_active":"2024-03-21T10:35:06.004190+0000","last_became_peered":"2024-03-21T10:35:06.004190+0000","last_unstale":"2024-03-21T10:36:56.312200+0000","last_undegraded":"2024-03-21T10:36:56.312200+0000","last_fullsized":"2024-03-21T10:36:56.312200+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:32:18.563692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325080+0000","last_change":"2024-03-21T10:35:06.941738+0000","last_active":"2024-03-21T10:36:56.325080+0000","last_peered":"2024-03-21T10:36:56.325080+0000","last_clean":"2024-03-21T10:36:56.325080+0000","last_became_active":"2024-03-21T10:35:06.941386+0000","last_became_peered":"2024-03-21T10:35:06.941386+0000","last_unstale":"2024-03-21T10:36:56.325080+0000","last_undegraded":"2024-03-21T10:36:56.325080+0000","last_fullsized":"2024-03-21T10:36:56.325080+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:52:55.519095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621767+0000","last_change":"2024-03-21T10:35:11.199370+0000","last_active":"2024-03-21T10:36:56.621767+0000","last_peered":"2024-03-21T10:36:56.621767+0000","last_clean":"2024-03-21T10:36:56.621767+0000","last_became_active":"2024-03-21T10:35:11.198598+0000","last_became_peered":"2024-03-21T10:35:11.198598+0000","last_unstale":"2024-03-21T10:36:56.621767+0000","last_undegraded":"2024-03-21T10:36:56.621767+0000","last_fullsized":"2024-03-21T10:36:56.621767+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:07:19.419405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311217+0000","last_change":"2024-03-21T10:35:06.004893+0000","last_active":"2024-03-21T10:36:56.311217+0000","last_peered":"2024-03-21T10:36:56.311217+0000","last_clean":"2024-03-21T10:36:56.311217+0000","last_became_active":"2024-03-21T10:35:06.004586+0000","last_became_peered":"2024-03-21T10:35:06.004586+0000","last_unstale":"2024-03-21T10:36:56.311217+0000","last_undegraded":"2024-03-21T10:36:56.311217+0000","last_fullsized":"2024-03-21T10:36:56.311217+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:05:29.106575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621403+0000","last_change":"2024-03-21T10:35:06.924448+0000","last_active":"2024-03-21T10:36:56.621403+0000","last_peered":"2024-03-21T10:36:56.621403+0000","last_clean":"2024-03-21T10:36:56.621403+0000","last_became_active":"2024-03-21T10:35:06.924049+0000","last_became_peered":"2024-03-21T10:35:06.924049+0000","last_unstale":"2024-03-21T10:36:56.621403+0000","last_undegraded":"2024-03-21T10:36:56.621403+0000","last_fullsized":"2024-03-21T10:36:56.621403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:23:12.070723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325020+0000","last_change":"2024-03-21T10:35:11.188691+0000","last_active":"2024-03-21T10:36:56.325020+0000","last_peered":"2024-03-21T10:36:56.325020+0000","last_clean":"2024-03-21T10:36:56.325020+0000","last_became_active":"2024-03-21T10:35:11.188138+0000","last_became_peered":"2024-03-21T10:35:11.188138+0000","last_unstale":"2024-03-21T10:36:56.325020+0000","last_undegraded":"2024-03-21T10:36:56.325020+0000","last_fullsized":"2024-03-21T10:36:56.325020+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:51:50.920920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":191,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:38:14.155614+0000","last_change":"2024-03-21T10:35:06.015276+0000","last_active":"2024-03-21T10:38:14.155614+0000","last_peered":"2024-03-21T10:38:14.155614+0000","last_clean":"2024-03-21T10:38:14.155614+0000","last_became_active":"2024-03-21T10:35:06.014734+0000","last_became_peered":"2024-03-21T10:35:06.014734+0000","last_unstale":"2024-03-21T10:38:14.155614+0000","last_undegraded":"2024-03-21T10:38:14.155614+0000","last_fullsized":"2024-03-21T10:38:14.155614+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:18:45.905451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":155,"num_read_kb":155,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.620346+0000","last_change":"2024-03-21T10:35:06.926689+0000","last_active":"2024-03-21T10:36:56.620346+0000","last_peered":"2024-03-21T10:36:56.620346+0000","last_clean":"2024-03-21T10:36:56.620346+0000","last_became_active":"2024-03-21T10:35:06.926319+0000","last_became_peered":"2024-03-21T10:35:06.926319+0000","last_unstale":"2024-03-21T10:36:56.620346+0000","last_undegraded":"2024-03-21T10:36:56.620346+0000","last_fullsized":"2024-03-21T10:36:56.620346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:06:22.371608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311104+0000","last_change":"2024-03-21T10:35:55.383776+0000","last_active":"2024-03-21T10:36:56.311104+0000","last_peered":"2024-03-21T10:36:56.311104+0000","last_clean":"2024-03-21T10:36:56.311104+0000","last_became_active":"2024-03-21T10:35:55.383300+0000","last_became_peered":"2024-03-21T10:35:55.383300+0000","last_unstale":"2024-03-21T10:36:56.311104+0000","last_undegraded":"2024-03-21T10:36:56.311104+0000","last_fullsized":"2024-03-21T10:36:56.311104+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:55:56.244325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2,4],"acting":[10,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619479+0000","last_change":"2024-03-21T10:35:06.007231+0000","last_active":"2024-03-21T10:36:56.619479+0000","last_peered":"2024-03-21T10:36:56.619479+0000","last_clean":"2024-03-21T10:36:56.619479+0000","last_became_active":"2024-03-21T10:35:06.006875+0000","last_became_peered":"2024-03-21T10:35:06.006875+0000","last_unstale":"2024-03-21T10:36:56.619479+0000","last_undegraded":"2024-03-21T10:36:56.619479+0000","last_fullsized":"2024-03-21T10:36:56.619479+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:08:02.489741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.624356+0000","last_change":"2024-03-21T10:35:55.524107+0000","last_active":"2024-03-21T10:36:56.624356+0000","last_peered":"2024-03-21T10:36:56.624356+0000","last_clean":"2024-03-21T10:36:56.624356+0000","last_became_active":"2024-03-21T10:35:55.523523+0000","last_became_peered":"2024-03-21T10:35:55.523523+0000","last_unstale":"2024-03-21T10:36:56.624356+0000","last_undegraded":"2024-03-21T10:36:56.624356+0000","last_fullsized":"2024-03-21T10:36:56.624356+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:49:29.121279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540722+0000","last_change":"2024-03-21T10:35:11.193382+0000","last_active":"2024-03-21T10:36:56.540722+0000","last_peered":"2024-03-21T10:36:56.540722+0000","last_clean":"2024-03-21T10:36:56.540722+0000","last_became_active":"2024-03-21T10:35:11.192959+0000","last_became_peered":"2024-03-21T10:35:11.192959+0000","last_unstale":"2024-03-21T10:36:56.540722+0000","last_undegraded":"2024-03-21T10:36:56.540722+0000","last_fullsized":"2024-03-21T10:36:56.540722+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:02:57.187370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311370+0000","last_change":"2024-03-21T10:35:06.019969+0000","last_active":"2024-03-21T10:36:56.311370+0000","last_peered":"2024-03-21T10:36:56.311370+0000","last_clean":"2024-03-21T10:36:56.311370+0000","last_became_active":"2024-03-21T10:35:06.019648+0000","last_became_peered":"2024-03-21T10:35:06.019648+0000","last_unstale":"2024-03-21T10:36:56.311370+0000","last_undegraded":"2024-03-21T10:36:56.311370+0000","last_fullsized":"2024-03-21T10:36:56.311370+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:24:02.227205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.620292+0000","last_change":"2024-03-21T10:35:06.910294+0000","last_active":"2024-03-21T10:36:56.620292+0000","last_peered":"2024-03-21T10:36:56.620292+0000","last_clean":"2024-03-21T10:36:56.620292+0000","last_became_active":"2024-03-21T10:35:06.909862+0000","last_became_peered":"2024-03-21T10:35:06.909862+0000","last_unstale":"2024-03-21T10:36:56.620292+0000","last_undegraded":"2024-03-21T10:36:56.620292+0000","last_fullsized":"2024-03-21T10:36:56.620292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:37:13.396003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621610+0000","last_change":"2024-03-21T10:35:11.196398+0000","last_active":"2024-03-21T10:36:56.621610+0000","last_peered":"2024-03-21T10:36:56.621610+0000","last_clean":"2024-03-21T10:36:56.621610+0000","last_became_active":"2024-03-21T10:35:11.194965+0000","last_became_peered":"2024-03-21T10:35:11.194965+0000","last_unstale":"2024-03-21T10:36:56.621610+0000","last_undegraded":"2024-03-21T10:36:56.621610+0000","last_fullsized":"2024-03-21T10:36:56.621610+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:48:22.024137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'38","reported_seq":391,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:38:15.156609+0000","last_change":"2024-03-21T10:35:06.005037+0000","last_active":"2024-03-21T10:38:15.156609+0000","last_peered":"2024-03-21T10:38:15.156609+0000","last_clean":"2024-03-21T10:38:15.156609+0000","last_became_active":"2024-03-21T10:35:06.004686+0000","last_became_peered":"2024-03-21T10:35:06.004686+0000","last_unstale":"2024-03-21T10:38:15.156609+0000","last_undegraded":"2024-03-21T10:38:15.156609+0000","last_fullsized":"2024-03-21T10:38:15.156609+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:24:13.312112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64761,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":321,"num_read_kb":225,"num_write":39,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313007+0000","last_change":"2024-03-21T10:35:06.906525+0000","last_active":"2024-03-21T10:36:56.313007+0000","last_peered":"2024-03-21T10:36:56.313007+0000","last_clean":"2024-03-21T10:36:56.313007+0000","last_became_active":"2024-03-21T10:35:06.905150+0000","last_became_peered":"2024-03-21T10:35:06.905150+0000","last_unstale":"2024-03-21T10:36:56.313007+0000","last_undegraded":"2024-03-21T10:36:56.313007+0000","last_fullsized":"2024-03-21T10:36:56.313007+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:46:27.958779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311725+0000","last_change":"2024-03-21T10:35:11.193519+0000","last_active":"2024-03-21T10:36:56.311725+0000","last_peered":"2024-03-21T10:36:56.311725+0000","last_clean":"2024-03-21T10:36:56.311725+0000","last_became_active":"2024-03-21T10:35:11.193043+0000","last_became_peered":"2024-03-21T10:35:11.193043+0000","last_unstale":"2024-03-21T10:36:56.311725+0000","last_undegraded":"2024-03-21T10:36:56.311725+0000","last_fullsized":"2024-03-21T10:36:56.311725+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:23:20.984472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618979+0000","last_change":"2024-03-21T10:35:05.999220+0000","last_active":"2024-03-21T10:36:56.618979+0000","last_peered":"2024-03-21T10:36:56.618979+0000","last_clean":"2024-03-21T10:36:56.618979+0000","last_became_active":"2024-03-21T10:35:05.998918+0000","last_became_peered":"2024-03-21T10:35:05.998918+0000","last_unstale":"2024-03-21T10:36:56.618979+0000","last_undegraded":"2024-03-21T10:36:56.618979+0000","last_fullsized":"2024-03-21T10:36:56.618979+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:01:09.816758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312512+0000","last_change":"2024-03-21T10:35:06.941234+0000","last_active":"2024-03-21T10:36:56.312512+0000","last_peered":"2024-03-21T10:36:56.312512+0000","last_clean":"2024-03-21T10:36:56.312512+0000","last_became_active":"2024-03-21T10:35:06.940828+0000","last_became_peered":"2024-03-21T10:35:06.940828+0000","last_unstale":"2024-03-21T10:36:56.312512+0000","last_undegraded":"2024-03-21T10:36:56.312512+0000","last_fullsized":"2024-03-21T10:36:56.312512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:50:34.398822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618827+0000","last_change":"2024-03-21T10:35:11.196007+0000","last_active":"2024-03-21T10:36:56.618827+0000","last_peered":"2024-03-21T10:36:56.618827+0000","last_clean":"2024-03-21T10:36:56.618827+0000","last_became_active":"2024-03-21T10:35:11.195609+0000","last_became_peered":"2024-03-21T10:35:11.195609+0000","last_unstale":"2024-03-21T10:36:56.618827+0000","last_undegraded":"2024-03-21T10:36:56.618827+0000","last_fullsized":"2024-03-21T10:36:56.618827+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:21:38.803390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311522+0000","last_change":"2024-03-21T10:35:06.011594+0000","last_active":"2024-03-21T10:36:56.311522+0000","last_peered":"2024-03-21T10:36:56.311522+0000","last_clean":"2024-03-21T10:36:56.311522+0000","last_became_active":"2024-03-21T10:35:06.011254+0000","last_became_peered":"2024-03-21T10:35:06.011254+0000","last_unstale":"2024-03-21T10:36:56.311522+0000","last_undegraded":"2024-03-21T10:36:56.311522+0000","last_fullsized":"2024-03-21T10:36:56.311522+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:52:31.045659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619993+0000","last_change":"2024-03-21T10:35:06.929170+0000","last_active":"2024-03-21T10:36:56.619993+0000","last_peered":"2024-03-21T10:36:56.619993+0000","last_clean":"2024-03-21T10:36:56.619993+0000","last_became_active":"2024-03-21T10:35:06.928822+0000","last_became_peered":"2024-03-21T10:35:06.928822+0000","last_unstale":"2024-03-21T10:36:56.619993+0000","last_undegraded":"2024-03-21T10:36:56.619993+0000","last_fullsized":"2024-03-21T10:36:56.619993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:37:40.328453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623051+0000","last_change":"2024-03-21T10:35:11.204285+0000","last_active":"2024-03-21T10:36:56.623051+0000","last_peered":"2024-03-21T10:36:56.623051+0000","last_clean":"2024-03-21T10:36:56.623051+0000","last_became_active":"2024-03-21T10:35:11.203698+0000","last_became_peered":"2024-03-21T10:35:11.203698+0000","last_unstale":"2024-03-21T10:36:56.623051+0000","last_undegraded":"2024-03-21T10:36:56.623051+0000","last_fullsized":"2024-03-21T10:36:56.623051+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:03:32.732488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"93'4","reported_seq":192,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:38:13.235271+0000","last_change":"2024-03-21T10:35:06.005028+0000","last_active":"2024-03-21T10:38:13.235271+0000","last_peered":"2024-03-21T10:38:13.235271+0000","last_clean":"2024-03-21T10:38:13.235271+0000","last_became_active":"2024-03-21T10:35:06.004670+0000","last_became_peered":"2024-03-21T10:35:06.004670+0000","last_unstale":"2024-03-21T10:38:13.235271+0000","last_undegraded":"2024-03-21T10:38:13.235271+0000","last_fullsized":"2024-03-21T10:38:13.235271+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:00:18.481051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":894,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":158,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619542+0000","last_change":"2024-03-21T10:35:06.911696+0000","last_active":"2024-03-21T10:36:56.619542+0000","last_peered":"2024-03-21T10:36:56.619542+0000","last_clean":"2024-03-21T10:36:56.619542+0000","last_became_active":"2024-03-21T10:35:06.911231+0000","last_became_peered":"2024-03-21T10:35:06.911231+0000","last_unstale":"2024-03-21T10:36:56.619542+0000","last_undegraded":"2024-03-21T10:36:56.619542+0000","last_fullsized":"2024-03-21T10:36:56.619542+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:21:51.540993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312937+0000","last_change":"2024-03-21T10:35:11.204422+0000","last_active":"2024-03-21T10:36:56.312937+0000","last_peered":"2024-03-21T10:36:56.312937+0000","last_clean":"2024-03-21T10:36:56.312937+0000","last_became_active":"2024-03-21T10:35:11.203904+0000","last_became_peered":"2024-03-21T10:35:11.203904+0000","last_unstale":"2024-03-21T10:36:56.312937+0000","last_undegraded":"2024-03-21T10:36:56.312937+0000","last_fullsized":"2024-03-21T10:36:56.312937+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:38:14.486012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313161+0000","last_change":"2024-03-21T10:35:06.019823+0000","last_active":"2024-03-21T10:36:56.313161+0000","last_peered":"2024-03-21T10:36:56.313161+0000","last_clean":"2024-03-21T10:36:56.313161+0000","last_became_active":"2024-03-21T10:35:06.019502+0000","last_became_peered":"2024-03-21T10:35:06.019502+0000","last_unstale":"2024-03-21T10:36:56.313161+0000","last_undegraded":"2024-03-21T10:36:56.313161+0000","last_fullsized":"2024-03-21T10:36:56.313161+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:33:22.438868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311234+0000","last_change":"2024-03-21T10:35:06.906671+0000","last_active":"2024-03-21T10:36:56.311234+0000","last_peered":"2024-03-21T10:36:56.311234+0000","last_clean":"2024-03-21T10:36:56.311234+0000","last_became_active":"2024-03-21T10:35:06.905873+0000","last_became_peered":"2024-03-21T10:35:06.905873+0000","last_unstale":"2024-03-21T10:36:56.311234+0000","last_undegraded":"2024-03-21T10:36:56.311234+0000","last_fullsized":"2024-03-21T10:36:56.311234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:14:59.304255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311196+0000","last_change":"2024-03-21T10:35:11.197698+0000","last_active":"2024-03-21T10:36:56.311196+0000","last_peered":"2024-03-21T10:36:56.311196+0000","last_clean":"2024-03-21T10:36:56.311196+0000","last_became_active":"2024-03-21T10:35:11.197295+0000","last_became_peered":"2024-03-21T10:35:11.197295+0000","last_unstale":"2024-03-21T10:36:56.311196+0000","last_undegraded":"2024-03-21T10:36:56.311196+0000","last_fullsized":"2024-03-21T10:36:56.311196+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:02:24.338317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"93'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:37:22.250209+0000","last_change":"2024-03-21T10:35:06.000951+0000","last_active":"2024-03-21T10:37:22.250209+0000","last_peered":"2024-03-21T10:37:22.250209+0000","last_clean":"2024-03-21T10:37:22.250209+0000","last_became_active":"2024-03-21T10:35:05.999169+0000","last_became_peered":"2024-03-21T10:35:05.999169+0000","last_unstale":"2024-03-21T10:37:22.250209+0000","last_undegraded":"2024-03-21T10:37:22.250209+0000","last_fullsized":"2024-03-21T10:37:22.250209+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:31:25.602621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619339+0000","last_change":"2024-03-21T10:35:06.908067+0000","last_active":"2024-03-21T10:36:56.619339+0000","last_peered":"2024-03-21T10:36:56.619339+0000","last_clean":"2024-03-21T10:36:56.619339+0000","last_became_active":"2024-03-21T10:35:06.907720+0000","last_became_peered":"2024-03-21T10:35:06.907720+0000","last_unstale":"2024-03-21T10:36:56.619339+0000","last_undegraded":"2024-03-21T10:36:56.619339+0000","last_fullsized":"2024-03-21T10:36:56.619339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:16:00.502637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312728+0000","last_change":"2024-03-21T10:35:11.229470+0000","last_active":"2024-03-21T10:36:56.312728+0000","last_peered":"2024-03-21T10:36:56.312728+0000","last_clean":"2024-03-21T10:36:56.312728+0000","last_became_active":"2024-03-21T10:35:11.228977+0000","last_became_peered":"2024-03-21T10:35:11.228977+0000","last_unstale":"2024-03-21T10:36:56.312728+0000","last_undegraded":"2024-03-21T10:36:56.312728+0000","last_fullsized":"2024-03-21T10:36:56.312728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:05:52.269712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623236+0000","last_change":"2024-03-21T10:35:06.019866+0000","last_active":"2024-03-21T10:36:56.623236+0000","last_peered":"2024-03-21T10:36:56.623236+0000","last_clean":"2024-03-21T10:36:56.623236+0000","last_became_active":"2024-03-21T10:35:06.019550+0000","last_became_peered":"2024-03-21T10:35:06.019550+0000","last_unstale":"2024-03-21T10:36:56.623236+0000","last_undegraded":"2024-03-21T10:36:56.623236+0000","last_fullsized":"2024-03-21T10:36:56.623236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:51:46.989647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311710+0000","last_change":"2024-03-21T10:35:55.385617+0000","last_active":"2024-03-21T10:36:56.311710+0000","last_peered":"2024-03-21T10:36:56.311710+0000","last_clean":"2024-03-21T10:36:56.311710+0000","last_became_active":"2024-03-21T10:35:55.385293+0000","last_became_peered":"2024-03-21T10:35:55.385293+0000","last_unstale":"2024-03-21T10:36:56.311710+0000","last_undegraded":"2024-03-21T10:36:56.311710+0000","last_fullsized":"2024-03-21T10:36:56.311710+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:18:08.200169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313097+0000","last_change":"2024-03-21T10:35:11.199714+0000","last_active":"2024-03-21T10:36:56.313097+0000","last_peered":"2024-03-21T10:36:56.313097+0000","last_clean":"2024-03-21T10:36:56.313097+0000","last_became_active":"2024-03-21T10:35:11.199277+0000","last_became_peered":"2024-03-21T10:35:11.199277+0000","last_unstale":"2024-03-21T10:36:56.313097+0000","last_undegraded":"2024-03-21T10:36:56.313097+0000","last_fullsized":"2024-03-21T10:36:56.313097+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:58:01.471162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619799+0000","last_change":"2024-03-21T10:35:06.015090+0000","last_active":"2024-03-21T10:36:56.619799+0000","last_peered":"2024-03-21T10:36:56.619799+0000","last_clean":"2024-03-21T10:36:56.619799+0000","last_became_active":"2024-03-21T10:35:06.014666+0000","last_became_peered":"2024-03-21T10:35:06.014666+0000","last_unstale":"2024-03-21T10:36:56.619799+0000","last_undegraded":"2024-03-21T10:36:56.619799+0000","last_fullsized":"2024-03-21T10:36:56.619799+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:41:17.608069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311382+0000","last_change":"2024-03-21T10:35:55.388548+0000","last_active":"2024-03-21T10:36:56.311382+0000","last_peered":"2024-03-21T10:36:56.311382+0000","last_clean":"2024-03-21T10:36:56.311382+0000","last_became_active":"2024-03-21T10:35:55.388253+0000","last_became_peered":"2024-03-21T10:35:55.388253+0000","last_unstale":"2024-03-21T10:36:56.311382+0000","last_undegraded":"2024-03-21T10:36:56.311382+0000","last_fullsized":"2024-03-21T10:36:56.311382+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:56.678224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312691+0000","last_change":"2024-03-21T10:35:11.230677+0000","last_active":"2024-03-21T10:36:56.312691+0000","last_peered":"2024-03-21T10:36:56.312691+0000","last_clean":"2024-03-21T10:36:56.312691+0000","last_became_active":"2024-03-21T10:35:11.230087+0000","last_became_peered":"2024-03-21T10:35:11.230087+0000","last_unstale":"2024-03-21T10:36:56.312691+0000","last_undegraded":"2024-03-21T10:36:56.312691+0000","last_fullsized":"2024-03-21T10:36:56.312691+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:39:47.663950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311339+0000","last_change":"2024-03-21T10:35:06.904287+0000","last_active":"2024-03-21T10:36:56.311339+0000","last_peered":"2024-03-21T10:36:56.311339+0000","last_clean":"2024-03-21T10:36:56.311339+0000","last_became_active":"2024-03-21T10:35:06.903821+0000","last_became_peered":"2024-03-21T10:35:06.903821+0000","last_unstale":"2024-03-21T10:36:56.311339+0000","last_undegraded":"2024-03-21T10:36:56.311339+0000","last_fullsized":"2024-03-21T10:36:56.311339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:36:25.362203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'5","reported_seq":191,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:38:13.234869+0000","last_change":"2024-03-21T10:35:06.014432+0000","last_active":"2024-03-21T10:38:13.234869+0000","last_peered":"2024-03-21T10:38:13.234869+0000","last_clean":"2024-03-21T10:38:13.234869+0000","last_became_active":"2024-03-21T10:35:06.013448+0000","last_became_peered":"2024-03-21T10:35:06.013448+0000","last_unstale":"2024-03-21T10:38:13.234869+0000","last_undegraded":"2024-03-21T10:38:13.234869+0000","last_fullsized":"2024-03-21T10:38:13.234869+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:29:10.600794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":154,"num_read_kb":154,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619114+0000","last_change":"2024-03-21T10:35:11.193577+0000","last_active":"2024-03-21T10:36:56.619114+0000","last_peered":"2024-03-21T10:36:56.619114+0000","last_clean":"2024-03-21T10:36:56.619114+0000","last_became_active":"2024-03-21T10:35:11.193132+0000","last_became_peered":"2024-03-21T10:35:11.193132+0000","last_unstale":"2024-03-21T10:36:56.619114+0000","last_undegraded":"2024-03-21T10:36:56.619114+0000","last_fullsized":"2024-03-21T10:36:56.619114+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:40:31.100489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325105+0000","last_change":"2024-03-21T10:35:06.924376+0000","last_active":"2024-03-21T10:36:56.325105+0000","last_peered":"2024-03-21T10:36:56.325105+0000","last_clean":"2024-03-21T10:36:56.325105+0000","last_became_active":"2024-03-21T10:35:06.923874+0000","last_became_peered":"2024-03-21T10:35:06.923874+0000","last_unstale":"2024-03-21T10:36:56.325105+0000","last_undegraded":"2024-03-21T10:36:56.325105+0000","last_fullsized":"2024-03-21T10:36:56.325105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:56:12.620667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540608+0000","last_change":"2024-03-21T10:35:06.015222+0000","last_active":"2024-03-21T10:36:56.540608+0000","last_peered":"2024-03-21T10:36:56.540608+0000","last_clean":"2024-03-21T10:36:56.540608+0000","last_became_active":"2024-03-21T10:35:06.014858+0000","last_became_peered":"2024-03-21T10:35:06.014858+0000","last_unstale":"2024-03-21T10:36:56.540608+0000","last_undegraded":"2024-03-21T10:36:56.540608+0000","last_fullsized":"2024-03-21T10:36:56.540608+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:45:12.528851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":367,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618815+0000","last_change":"2024-03-21T10:33:44.536069+0000","last_active":"2024-03-21T10:36:56.618815+0000","last_peered":"2024-03-21T10:36:56.618815+0000","last_clean":"2024-03-21T10:36:56.618815+0000","last_became_active":"2024-03-21T10:33:44.535623+0000","last_became_peered":"2024-03-21T10:33:44.535623+0000","last_unstale":"2024-03-21T10:36:56.618815+0000","last_undegraded":"2024-03-21T10:36:56.618815+0000","last_fullsized":"2024-03-21T10:36:56.618815+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:29:44.554765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:29:44.554765+0000","last_clean_scrub_stamp":"2024-03-21T10:29:44.554765+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:43:23.440837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311654+0000","last_change":"2024-03-21T10:35:11.229580+0000","last_active":"2024-03-21T10:36:56.311654+0000","last_peered":"2024-03-21T10:36:56.311654+0000","last_clean":"2024-03-21T10:36:56.311654+0000","last_became_active":"2024-03-21T10:35:11.229044+0000","last_became_peered":"2024-03-21T10:35:11.229044+0000","last_unstale":"2024-03-21T10:36:56.311654+0000","last_undegraded":"2024-03-21T10:36:56.311654+0000","last_fullsized":"2024-03-21T10:36:56.311654+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:54:50.535253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312060+0000","last_change":"2024-03-21T10:35:06.927690+0000","last_active":"2024-03-21T10:36:56.312060+0000","last_peered":"2024-03-21T10:36:56.312060+0000","last_clean":"2024-03-21T10:36:56.312060+0000","last_became_active":"2024-03-21T10:35:06.927304+0000","last_became_peered":"2024-03-21T10:35:06.927304+0000","last_unstale":"2024-03-21T10:36:56.312060+0000","last_undegraded":"2024-03-21T10:36:56.312060+0000","last_fullsized":"2024-03-21T10:36:56.312060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:30:48.950459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619515+0000","last_change":"2024-03-21T10:35:06.004952+0000","last_active":"2024-03-21T10:36:56.619515+0000","last_peered":"2024-03-21T10:36:56.619515+0000","last_clean":"2024-03-21T10:36:56.619515+0000","last_became_active":"2024-03-21T10:35:06.004206+0000","last_became_peered":"2024-03-21T10:35:06.004206+0000","last_unstale":"2024-03-21T10:36:56.619515+0000","last_undegraded":"2024-03-21T10:36:56.619515+0000","last_fullsized":"2024-03-21T10:36:56.619515+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:20:29.198715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619052+0000","last_change":"2024-03-21T10:35:11.179125+0000","last_active":"2024-03-21T10:36:56.619052+0000","last_peered":"2024-03-21T10:36:56.619052+0000","last_clean":"2024-03-21T10:36:56.619052+0000","last_became_active":"2024-03-21T10:35:11.178658+0000","last_became_peered":"2024-03-21T10:35:11.178658+0000","last_unstale":"2024-03-21T10:36:56.619052+0000","last_undegraded":"2024-03-21T10:36:56.619052+0000","last_fullsized":"2024-03-21T10:36:56.619052+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:33:25.281721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313030+0000","last_change":"2024-03-21T10:35:06.019996+0000","last_active":"2024-03-21T10:36:56.313030+0000","last_peered":"2024-03-21T10:36:56.313030+0000","last_clean":"2024-03-21T10:36:56.313030+0000","last_became_active":"2024-03-21T10:35:06.019624+0000","last_became_peered":"2024-03-21T10:35:06.019624+0000","last_unstale":"2024-03-21T10:36:56.313030+0000","last_undegraded":"2024-03-21T10:36:56.313030+0000","last_fullsized":"2024-03-21T10:36:56.313030+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:31:09.551433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313495+0000","last_change":"2024-03-21T10:35:06.939324+0000","last_active":"2024-03-21T10:36:56.313495+0000","last_peered":"2024-03-21T10:36:56.313495+0000","last_clean":"2024-03-21T10:36:56.313495+0000","last_became_active":"2024-03-21T10:35:06.937750+0000","last_became_peered":"2024-03-21T10:35:06.937750+0000","last_unstale":"2024-03-21T10:36:56.313495+0000","last_undegraded":"2024-03-21T10:36:56.313495+0000","last_fullsized":"2024-03-21T10:36:56.313495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:45:14.600863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.324947+0000","last_change":"2024-03-21T10:35:11.229479+0000","last_active":"2024-03-21T10:36:56.324947+0000","last_peered":"2024-03-21T10:36:56.324947+0000","last_clean":"2024-03-21T10:36:56.324947+0000","last_became_active":"2024-03-21T10:35:11.229005+0000","last_became_peered":"2024-03-21T10:35:11.229005+0000","last_unstale":"2024-03-21T10:36:56.324947+0000","last_undegraded":"2024-03-21T10:36:56.324947+0000","last_fullsized":"2024-03-21T10:36:56.324947+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:39:19.859581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'12","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:37:19.740273+0000","last_change":"2024-03-21T10:35:06.013744+0000","last_active":"2024-03-21T10:37:19.740273+0000","last_peered":"2024-03-21T10:37:19.740273+0000","last_clean":"2024-03-21T10:37:19.740273+0000","last_became_active":"2024-03-21T10:35:06.013509+0000","last_became_peered":"2024-03-21T10:35:06.013509+0000","last_unstale":"2024-03-21T10:37:19.740273+0000","last_undegraded":"2024-03-21T10:37:19.740273+0000","last_fullsized":"2024-03-21T10:37:19.740273+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:51:45.000933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325968+0000","last_change":"2024-03-21T10:35:06.930685+0000","last_active":"2024-03-21T10:36:56.325968+0000","last_peered":"2024-03-21T10:36:56.325968+0000","last_clean":"2024-03-21T10:36:56.325968+0000","last_became_active":"2024-03-21T10:35:06.929775+0000","last_became_peered":"2024-03-21T10:35:06.929775+0000","last_unstale":"2024-03-21T10:36:56.325968+0000","last_undegraded":"2024-03-21T10:36:56.325968+0000","last_fullsized":"2024-03-21T10:36:56.325968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:13:18.058752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618619+0000","last_change":"2024-03-21T10:35:11.177061+0000","last_active":"2024-03-21T10:36:56.618619+0000","last_peered":"2024-03-21T10:36:56.618619+0000","last_clean":"2024-03-21T10:36:56.618619+0000","last_became_active":"2024-03-21T10:35:11.176603+0000","last_became_peered":"2024-03-21T10:35:11.176603+0000","last_unstale":"2024-03-21T10:36:56.618619+0000","last_undegraded":"2024-03-21T10:36:56.618619+0000","last_fullsized":"2024-03-21T10:36:56.618619+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:54.532780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540962+0000","last_change":"2024-03-21T10:35:06.009018+0000","last_active":"2024-03-21T10:36:56.540962+0000","last_peered":"2024-03-21T10:36:56.540962+0000","last_clean":"2024-03-21T10:36:56.540962+0000","last_became_active":"2024-03-21T10:35:06.008638+0000","last_became_peered":"2024-03-21T10:35:06.008638+0000","last_unstale":"2024-03-21T10:36:56.540962+0000","last_undegraded":"2024-03-21T10:36:56.540962+0000","last_fullsized":"2024-03-21T10:36:56.540962+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:42:19.527898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621997+0000","last_change":"2024-03-21T10:35:06.931588+0000","last_active":"2024-03-21T10:36:56.621997+0000","last_peered":"2024-03-21T10:36:56.621997+0000","last_clean":"2024-03-21T10:36:56.621997+0000","last_became_active":"2024-03-21T10:35:06.930707+0000","last_became_peered":"2024-03-21T10:35:06.930707+0000","last_unstale":"2024-03-21T10:36:56.621997+0000","last_undegraded":"2024-03-21T10:36:56.621997+0000","last_fullsized":"2024-03-21T10:36:56.621997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:44:55.862799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540818+0000","last_change":"2024-03-21T10:35:11.229408+0000","last_active":"2024-03-21T10:36:56.540818+0000","last_peered":"2024-03-21T10:36:56.540818+0000","last_clean":"2024-03-21T10:36:56.540818+0000","last_became_active":"2024-03-21T10:35:11.228997+0000","last_became_peered":"2024-03-21T10:35:11.228997+0000","last_unstale":"2024-03-21T10:36:56.540818+0000","last_undegraded":"2024-03-21T10:36:56.540818+0000","last_fullsized":"2024-03-21T10:36:56.540818+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:56:11.517597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619256+0000","last_change":"2024-03-21T10:35:55.388978+0000","last_active":"2024-03-21T10:36:56.619256+0000","last_peered":"2024-03-21T10:36:56.619256+0000","last_clean":"2024-03-21T10:36:56.619256+0000","last_became_active":"2024-03-21T10:35:55.383048+0000","last_became_peered":"2024-03-21T10:35:55.383048+0000","last_unstale":"2024-03-21T10:36:56.619256+0000","last_undegraded":"2024-03-21T10:36:56.619256+0000","last_fullsized":"2024-03-21T10:36:56.619256+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:24:01.933115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312642+0000","last_change":"2024-03-21T10:35:06.908585+0000","last_active":"2024-03-21T10:36:56.312642+0000","last_peered":"2024-03-21T10:36:56.312642+0000","last_clean":"2024-03-21T10:36:56.312642+0000","last_became_active":"2024-03-21T10:35:06.908253+0000","last_became_peered":"2024-03-21T10:35:06.908253+0000","last_unstale":"2024-03-21T10:36:56.312642+0000","last_undegraded":"2024-03-21T10:36:56.312642+0000","last_fullsized":"2024-03-21T10:36:56.312642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:46:15.984450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.623323+0000","last_change":"2024-03-21T10:35:11.199455+0000","last_active":"2024-03-21T10:36:56.623323+0000","last_peered":"2024-03-21T10:36:56.623323+0000","last_clean":"2024-03-21T10:36:56.623323+0000","last_became_active":"2024-03-21T10:35:11.197705+0000","last_became_peered":"2024-03-21T10:35:11.197705+0000","last_unstale":"2024-03-21T10:36:56.623323+0000","last_undegraded":"2024-03-21T10:36:56.623323+0000","last_fullsized":"2024-03-21T10:36:56.623323+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:59:42.792764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325149+0000","last_change":"2024-03-21T10:35:06.016152+0000","last_active":"2024-03-21T10:36:56.325149+0000","last_peered":"2024-03-21T10:36:56.325149+0000","last_clean":"2024-03-21T10:36:56.325149+0000","last_became_active":"2024-03-21T10:35:06.015439+0000","last_became_peered":"2024-03-21T10:35:06.015439+0000","last_unstale":"2024-03-21T10:36:56.325149+0000","last_undegraded":"2024-03-21T10:36:56.325149+0000","last_fullsized":"2024-03-21T10:36:56.325149+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:05:23.221826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618846+0000","last_change":"2024-03-21T10:35:06.929241+0000","last_active":"2024-03-21T10:36:56.618846+0000","last_peered":"2024-03-21T10:36:56.618846+0000","last_clean":"2024-03-21T10:36:56.618846+0000","last_became_active":"2024-03-21T10:35:06.928943+0000","last_became_peered":"2024-03-21T10:35:06.928943+0000","last_unstale":"2024-03-21T10:36:56.618846+0000","last_undegraded":"2024-03-21T10:36:56.618846+0000","last_fullsized":"2024-03-21T10:36:56.618846+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:24:10.484540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540706+0000","last_change":"2024-03-21T10:35:11.195929+0000","last_active":"2024-03-21T10:36:56.540706+0000","last_peered":"2024-03-21T10:36:56.540706+0000","last_clean":"2024-03-21T10:36:56.540706+0000","last_became_active":"2024-03-21T10:35:11.195489+0000","last_became_peered":"2024-03-21T10:35:11.195489+0000","last_unstale":"2024-03-21T10:36:56.540706+0000","last_undegraded":"2024-03-21T10:36:56.540706+0000","last_fullsized":"2024-03-21T10:36:56.540706+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:41:15.137917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618718+0000","last_change":"2024-03-21T10:35:06.002127+0000","last_active":"2024-03-21T10:36:56.618718+0000","last_peered":"2024-03-21T10:36:56.618718+0000","last_clean":"2024-03-21T10:36:56.618718+0000","last_became_active":"2024-03-21T10:35:06.001706+0000","last_became_peered":"2024-03-21T10:35:06.001706+0000","last_unstale":"2024-03-21T10:36:56.618718+0000","last_undegraded":"2024-03-21T10:36:56.618718+0000","last_fullsized":"2024-03-21T10:36:56.618718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:26:32.844991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325187+0000","last_change":"2024-03-21T10:35:06.925945+0000","last_active":"2024-03-21T10:36:56.325187+0000","last_peered":"2024-03-21T10:36:56.325187+0000","last_clean":"2024-03-21T10:36:56.325187+0000","last_became_active":"2024-03-21T10:35:06.925551+0000","last_became_peered":"2024-03-21T10:35:06.925551+0000","last_unstale":"2024-03-21T10:36:56.325187+0000","last_undegraded":"2024-03-21T10:36:56.325187+0000","last_fullsized":"2024-03-21T10:36:56.325187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:24:10.212676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.540914+0000","last_change":"2024-03-21T10:35:11.229067+0000","last_active":"2024-03-21T10:36:56.540914+0000","last_peered":"2024-03-21T10:36:56.540914+0000","last_clean":"2024-03-21T10:36:56.540914+0000","last_became_active":"2024-03-21T10:35:11.228628+0000","last_became_peered":"2024-03-21T10:35:11.228628+0000","last_unstale":"2024-03-21T10:36:56.540914+0000","last_undegraded":"2024-03-21T10:36:56.540914+0000","last_fullsized":"2024-03-21T10:36:56.540914+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:45:05.473277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618958+0000","last_change":"2024-03-21T10:35:55.384922+0000","last_active":"2024-03-21T10:36:56.618958+0000","last_peered":"2024-03-21T10:36:56.618958+0000","last_clean":"2024-03-21T10:36:56.618958+0000","last_became_active":"2024-03-21T10:35:55.380952+0000","last_became_peered":"2024-03-21T10:35:55.380952+0000","last_unstale":"2024-03-21T10:36:56.618958+0000","last_undegraded":"2024-03-21T10:36:56.618958+0000","last_fullsized":"2024-03-21T10:36:56.618958+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:29:37.873699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619644+0000","last_change":"2024-03-21T10:35:06.928049+0000","last_active":"2024-03-21T10:36:56.619644+0000","last_peered":"2024-03-21T10:36:56.619644+0000","last_clean":"2024-03-21T10:36:56.619644+0000","last_became_active":"2024-03-21T10:35:06.927717+0000","last_became_peered":"2024-03-21T10:35:06.927717+0000","last_unstale":"2024-03-21T10:36:56.619644+0000","last_undegraded":"2024-03-21T10:36:56.619644+0000","last_fullsized":"2024-03-21T10:36:56.619644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:27:47.554668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619381+0000","last_change":"2024-03-21T10:35:11.181090+0000","last_active":"2024-03-21T10:36:56.619381+0000","last_peered":"2024-03-21T10:36:56.619381+0000","last_clean":"2024-03-21T10:36:56.619381+0000","last_became_active":"2024-03-21T10:35:11.180511+0000","last_became_peered":"2024-03-21T10:35:11.180511+0000","last_unstale":"2024-03-21T10:36:56.619381+0000","last_undegraded":"2024-03-21T10:36:56.619381+0000","last_fullsized":"2024-03-21T10:36:56.619381+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T22:21:51.517845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619595+0000","last_change":"2024-03-21T10:35:06.007828+0000","last_active":"2024-03-21T10:36:56.619595+0000","last_peered":"2024-03-21T10:36:56.619595+0000","last_clean":"2024-03-21T10:36:56.619595+0000","last_became_active":"2024-03-21T10:35:06.007490+0000","last_became_peered":"2024-03-21T10:35:06.007490+0000","last_unstale":"2024-03-21T10:36:56.619595+0000","last_undegraded":"2024-03-21T10:36:56.619595+0000","last_fullsized":"2024-03-21T10:36:56.619595+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:08:18.579308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619840+0000","last_change":"2024-03-21T10:35:06.909313+0000","last_active":"2024-03-21T10:36:56.619840+0000","last_peered":"2024-03-21T10:36:56.619840+0000","last_clean":"2024-03-21T10:36:56.619840+0000","last_became_active":"2024-03-21T10:35:06.908555+0000","last_became_peered":"2024-03-21T10:35:06.908555+0000","last_unstale":"2024-03-21T10:36:56.619840+0000","last_undegraded":"2024-03-21T10:36:56.619840+0000","last_fullsized":"2024-03-21T10:36:56.619840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:09:57.961971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.622374+0000","last_change":"2024-03-21T10:35:11.195996+0000","last_active":"2024-03-21T10:36:56.622374+0000","last_peered":"2024-03-21T10:36:56.622374+0000","last_clean":"2024-03-21T10:36:56.622374+0000","last_became_active":"2024-03-21T10:35:11.195506+0000","last_became_peered":"2024-03-21T10:35:11.195506+0000","last_unstale":"2024-03-21T10:36:56.622374+0000","last_undegraded":"2024-03-21T10:36:56.622374+0000","last_fullsized":"2024-03-21T10:36:56.622374+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:49:33.010899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311586+0000","last_change":"2024-03-21T10:35:06.011498+0000","last_active":"2024-03-21T10:36:56.311586+0000","last_peered":"2024-03-21T10:36:56.311586+0000","last_clean":"2024-03-21T10:36:56.311586+0000","last_became_active":"2024-03-21T10:35:06.011203+0000","last_became_peered":"2024-03-21T10:35:06.011203+0000","last_unstale":"2024-03-21T10:36:56.311586+0000","last_undegraded":"2024-03-21T10:36:56.311586+0000","last_fullsized":"2024-03-21T10:36:56.311586+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:28:11.369740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619130+0000","last_change":"2024-03-21T10:35:06.935514+0000","last_active":"2024-03-21T10:36:56.619130+0000","last_peered":"2024-03-21T10:36:56.619130+0000","last_clean":"2024-03-21T10:36:56.619130+0000","last_became_active":"2024-03-21T10:35:06.935174+0000","last_became_peered":"2024-03-21T10:35:06.935174+0000","last_unstale":"2024-03-21T10:36:56.619130+0000","last_undegraded":"2024-03-21T10:36:56.619130+0000","last_fullsized":"2024-03-21T10:36:56.619130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:35:42.884471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.621386+0000","last_change":"2024-03-21T10:36:56.621053+0000","last_active":"2024-03-21T10:36:56.621386+0000","last_peered":"2024-03-21T10:36:56.621386+0000","last_clean":"2024-03-21T10:36:56.621386+0000","last_became_active":"2024-03-21T10:36:56.620688+0000","last_became_peered":"2024-03-21T10:36:56.620688+0000","last_unstale":"2024-03-21T10:36:56.621386+0000","last_undegraded":"2024-03-21T10:36:56.621386+0000","last_fullsized":"2024-03-21T10:36:56.621386+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:10.128078+0000","last_clean_scrub_stamp":"2024-03-21T10:35:10.128078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:50:02.121787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313022+0000","last_change":"2024-03-21T10:35:06.007216+0000","last_active":"2024-03-21T10:36:56.313022+0000","last_peered":"2024-03-21T10:36:56.313022+0000","last_clean":"2024-03-21T10:36:56.313022+0000","last_became_active":"2024-03-21T10:35:06.006883+0000","last_became_peered":"2024-03-21T10:35:06.006883+0000","last_unstale":"2024-03-21T10:36:56.313022+0000","last_undegraded":"2024-03-21T10:36:56.313022+0000","last_fullsized":"2024-03-21T10:36:56.313022+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:04.971426+0000","last_clean_scrub_stamp":"2024-03-21T10:35:04.971426+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:46:45.310919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619803+0000","last_change":"2024-03-21T10:35:06.906017+0000","last_active":"2024-03-21T10:36:56.619803+0000","last_peered":"2024-03-21T10:36:56.619803+0000","last_clean":"2024-03-21T10:36:56.619803+0000","last_became_active":"2024-03-21T10:35:06.905648+0000","last_became_peered":"2024-03-21T10:35:06.905648+0000","last_unstale":"2024-03-21T10:36:56.619803+0000","last_undegraded":"2024-03-21T10:36:56.619803+0000","last_fullsized":"2024-03-21T10:36:56.619803+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:56:50.380146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618954+0000","last_change":"2024-03-21T10:35:06.905186+0000","last_active":"2024-03-21T10:36:56.618954+0000","last_peered":"2024-03-21T10:36:56.618954+0000","last_clean":"2024-03-21T10:36:56.618954+0000","last_became_active":"2024-03-21T10:35:06.904835+0000","last_became_peered":"2024-03-21T10:35:06.904835+0000","last_unstale":"2024-03-21T10:36:56.618954+0000","last_undegraded":"2024-03-21T10:36:56.618954+0000","last_fullsized":"2024-03-21T10:36:56.618954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:53:19.293437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619036+0000","last_change":"2024-03-21T10:35:06.903318+0000","last_active":"2024-03-21T10:36:56.619036+0000","last_peered":"2024-03-21T10:36:56.619036+0000","last_clean":"2024-03-21T10:36:56.619036+0000","last_became_active":"2024-03-21T10:35:06.902994+0000","last_became_peered":"2024-03-21T10:35:06.902994+0000","last_unstale":"2024-03-21T10:36:56.619036+0000","last_undegraded":"2024-03-21T10:36:56.619036+0000","last_fullsized":"2024-03-21T10:36:56.619036+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:21:27.412565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.618679+0000","last_change":"2024-03-21T10:35:06.934582+0000","last_active":"2024-03-21T10:36:56.618679+0000","last_peered":"2024-03-21T10:36:56.618679+0000","last_clean":"2024-03-21T10:36:56.618679+0000","last_became_active":"2024-03-21T10:35:06.934311+0000","last_became_peered":"2024-03-21T10:35:06.934311+0000","last_unstale":"2024-03-21T10:36:56.618679+0000","last_undegraded":"2024-03-21T10:36:56.618679+0000","last_fullsized":"2024-03-21T10:36:56.618679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:05:30.785830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.622973+0000","last_change":"2024-03-21T10:35:06.944221+0000","last_active":"2024-03-21T10:36:56.622973+0000","last_peered":"2024-03-21T10:36:56.622973+0000","last_clean":"2024-03-21T10:36:56.622973+0000","last_became_active":"2024-03-21T10:35:06.943732+0000","last_became_peered":"2024-03-21T10:35:06.943732+0000","last_unstale":"2024-03-21T10:36:56.622973+0000","last_undegraded":"2024-03-21T10:36:56.622973+0000","last_fullsized":"2024-03-21T10:36:56.622973+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:33:03.653160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.313346+0000","last_change":"2024-03-21T10:35:06.940832+0000","last_active":"2024-03-21T10:36:56.313346+0000","last_peered":"2024-03-21T10:36:56.313346+0000","last_clean":"2024-03-21T10:36:56.313346+0000","last_became_active":"2024-03-21T10:35:06.939565+0000","last_became_peered":"2024-03-21T10:35:06.939565+0000","last_unstale":"2024-03-21T10:36:56.313346+0000","last_undegraded":"2024-03-21T10:36:56.313346+0000","last_fullsized":"2024-03-21T10:36:56.313346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:57:27.743704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619590+0000","last_change":"2024-03-21T10:35:06.907753+0000","last_active":"2024-03-21T10:36:56.619590+0000","last_peered":"2024-03-21T10:36:56.619590+0000","last_clean":"2024-03-21T10:36:56.619590+0000","last_became_active":"2024-03-21T10:35:06.907432+0000","last_became_peered":"2024-03-21T10:35:06.907432+0000","last_unstale":"2024-03-21T10:36:56.619590+0000","last_undegraded":"2024-03-21T10:36:56.619590+0000","last_fullsized":"2024-03-21T10:36:56.619590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:31:02.528855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619670+0000","last_change":"2024-03-21T10:35:06.908597+0000","last_active":"2024-03-21T10:36:56.619670+0000","last_peered":"2024-03-21T10:36:56.619670+0000","last_clean":"2024-03-21T10:36:56.619670+0000","last_became_active":"2024-03-21T10:35:06.908282+0000","last_became_peered":"2024-03-21T10:35:06.908282+0000","last_unstale":"2024-03-21T10:36:56.619670+0000","last_undegraded":"2024-03-21T10:36:56.619670+0000","last_fullsized":"2024-03-21T10:36:56.619670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:42:18.221045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.326006+0000","last_change":"2024-03-21T10:35:06.931015+0000","last_active":"2024-03-21T10:36:56.326006+0000","last_peered":"2024-03-21T10:36:56.326006+0000","last_clean":"2024-03-21T10:36:56.326006+0000","last_became_active":"2024-03-21T10:35:06.930209+0000","last_became_peered":"2024-03-21T10:35:06.930209+0000","last_unstale":"2024-03-21T10:36:56.326006+0000","last_undegraded":"2024-03-21T10:36:56.326006+0000","last_fullsized":"2024-03-21T10:36:56.326006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:42:15.277874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.311263+0000","last_change":"2024-03-21T10:35:06.930070+0000","last_active":"2024-03-21T10:36:56.311263+0000","last_peered":"2024-03-21T10:36:56.311263+0000","last_clean":"2024-03-21T10:36:56.311263+0000","last_became_active":"2024-03-21T10:35:06.929773+0000","last_became_peered":"2024-03-21T10:35:06.929773+0000","last_unstale":"2024-03-21T10:36:56.311263+0000","last_undegraded":"2024-03-21T10:36:56.311263+0000","last_fullsized":"2024-03-21T10:36:56.311263+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T21:17:25.176666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.325363+0000","last_change":"2024-03-21T10:35:06.929575+0000","last_active":"2024-03-21T10:36:56.325363+0000","last_peered":"2024-03-21T10:36:56.325363+0000","last_clean":"2024-03-21T10:36:56.325363+0000","last_became_active":"2024-03-21T10:35:06.929240+0000","last_became_peered":"2024-03-21T10:35:06.929240+0000","last_unstale":"2024-03-21T10:36:56.325363+0000","last_undegraded":"2024-03-21T10:36:56.325363+0000","last_fullsized":"2024-03-21T10:36:56.325363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:03:40.697640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619625+0000","last_change":"2024-03-21T10:35:06.934693+0000","last_active":"2024-03-21T10:36:56.619625+0000","last_peered":"2024-03-21T10:36:56.619625+0000","last_clean":"2024-03-21T10:36:56.619625+0000","last_became_active":"2024-03-21T10:35:06.934361+0000","last_became_peered":"2024-03-21T10:35:06.934361+0000","last_unstale":"2024-03-21T10:36:56.619625+0000","last_undegraded":"2024-03-21T10:36:56.619625+0000","last_fullsized":"2024-03-21T10:36:56.619625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:14:00.954370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.312500+0000","last_change":"2024-03-21T10:35:06.941095+0000","last_active":"2024-03-21T10:36:56.312500+0000","last_peered":"2024-03-21T10:36:56.312500+0000","last_clean":"2024-03-21T10:36:56.312500+0000","last_became_active":"2024-03-21T10:35:06.939348+0000","last_became_peered":"2024-03-21T10:35:06.939348+0000","last_unstale":"2024-03-21T10:36:56.312500+0000","last_undegraded":"2024-03-21T10:36:56.312500+0000","last_fullsized":"2024-03-21T10:36:56.312500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:31:43.783592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-21T10:36:56.619193+0000","last_change":"2024-03-21T10:35:06.908340+0000","last_active":"2024-03-21T10:36:56.619193+0000","last_peered":"2024-03-21T10:36:56.619193+0000","last_clean":"2024-03-21T10:36:56.619193+0000","last_became_active":"2024-03-21T10:35:06.908043+0000","last_became_peered":"2024-03-21T10:35:06.908043+0000","last_unstale":"2024-03-21T10:36:56.619193+0000","last_undegraded":"2024-03-21T10:36:56.619193+0000","last_fullsized":"2024-03-21T10:36:56.619193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T10:35:05.985466+0000","last_clean_scrub_stamp":"2024-03-21T10:35:05.985466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:50:33.167396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282863,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69209,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":957,"num_read_kb":553,"num_write":180,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":207627,"data_compressed":5487,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":154,"ondisk_log_size":154,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383733,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1019904,"data_stored":519397,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645370,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32312,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712840,"statfs":{"total":95995035648,"available":95961948160,"internally_reserved":0,"allocated":1163264,"data_stored":1171732,"data_compressed":64463,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808895,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":523729,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070535,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1036288,"data_stored":519553,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332172,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1085440,"data_stored":1115909,"data_compressed":62634,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495698,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":514809,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574959,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1081344,"data_stored":579480,"data_compressed":28489,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738484,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":516197,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411432,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1105920,"data_stored":581362,"data_compressed":28489,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182498,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":515003,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920860,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":519765,"data_compressed":26660,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757335,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1093632,"data_stored":1111191,"data_compressed":62634,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34331,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34331,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34331,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35974,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":64761,"data_compressed":1829,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":66643,"data_compressed":1829,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":66645,"data_compressed":1829,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T10:38:19.598 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T10:38:19.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.887+0000 7fb68d200640 1 -- 172.21.15.31:0/3913381589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6880772a0 msgr2=0x7fb688077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:19.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.887+0000 7fb68d200640 1 --2- 172.21.15.31:0/3913381589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6880772a0 0x7fb688077700 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb674009900 tx=0x7fb67402bc40 comp rx=0 tx=0).stop 2024-03-21T10:38:19.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.887+0000 7fb68d200640 1 -- 172.21.15.31:0/3913381589 shutdown_connections 2024-03-21T10:38:19.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.887+0000 7fb68d200640 1 --2- 172.21.15.31:0/3913381589 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb688077c40 0x7fb6880784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:19.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.887+0000 7fb68d200640 1 --2- 172.21.15.31:0/3913381589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6880772a0 0x7fb688077700 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:19.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.887+0000 7fb68d200640 1 --2- 172.21.15.31:0/3913381589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb688076060 0x7fb688076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:19.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.887+0000 7fb68d200640 1 -- 172.21.15.31:0/3913381589 >> 172.21.15.31:0/3913381589 conn(0x7fb6880feeb0 msgr2=0x7fb688101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:19.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.887+0000 7fb68d200640 1 -- 172.21.15.31:0/3913381589 shutdown_connections 2024-03-21T10:38:19.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.887+0000 7fb68d200640 1 -- 172.21.15.31:0/3913381589 wait complete. 2024-03-21T10:38:19.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.888+0000 7fb68d200640 1 Processor -- start 2024-03-21T10:38:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.888+0000 7fb68d200640 1 -- start start 2024-03-21T10:38:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.889+0000 7fb68d200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb688076060 0x7fb68819dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.889+0000 7fb68d200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6880772a0 0x7fb68819e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.889+0000 7fb68d200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb688077c40 0x7fb6881a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.889+0000 7fb68d200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6881139c0 con 0x7fb6880772a0 2024-03-21T10:38:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.889+0000 7fb68d200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb688113840 con 0x7fb688076060 2024-03-21T10:38:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.889+0000 7fb68d200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb688113b40 con 0x7fb688077c40 2024-03-21T10:38:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.889+0000 7fb685e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb688076060 0x7fb68819dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.889+0000 7fb685e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb688076060 0x7fb68819dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46108/0 (socket says 172.21.15.31:46108) 2024-03-21T10:38:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.889+0000 7fb685e00640 1 -- 172.21.15.31:0/347119553 learned_addr learned my addr 172.21.15.31:0/347119553 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.889+0000 7fb685400640 1 --2- 172.21.15.31:0/347119553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6880772a0 0x7fb68819e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.889+0000 7fb686800640 1 --2- 172.21.15.31:0/347119553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb688077c40 0x7fb6881a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.889+0000 7fb685e00640 1 -- 172.21.15.31:0/347119553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb688077c40 msgr2=0x7fb6881a5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.889+0000 7fb685e00640 1 --2- 172.21.15.31:0/347119553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb688077c40 0x7fb6881a5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.889+0000 7fb685e00640 1 -- 172.21.15.31:0/347119553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6880772a0 msgr2=0x7fb68819e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.889+0000 7fb685e00640 1 --2- 172.21.15.31:0/347119553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6880772a0 0x7fb68819e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.889+0000 7fb685e00640 1 -- 172.21.15.31:0/347119553 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6881a5be0 con 0x7fb688076060 2024-03-21T10:38:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.890+0000 7fb685e00640 1 --2- 172.21.15.31:0/347119553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb688076060 0x7fb68819dfc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb67800e9f0 tx=0x7fb67800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.890+0000 7fb67ea00640 1 -- 172.21.15.31:0/347119553 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67800cd80 con 0x7fb688076060 2024-03-21T10:38:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.890+0000 7fb67ea00640 1 -- 172.21.15.31:0/347119553 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb678005500 con 0x7fb688076060 2024-03-21T10:38:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.890+0000 7fb68d200640 1 -- 172.21.15.31:0/347119553 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6881a5ed0 con 0x7fb688076060 2024-03-21T10:38:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.890+0000 7fb67ea00640 1 -- 172.21.15.31:0/347119553 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb678010690 con 0x7fb688076060 2024-03-21T10:38:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.890+0000 7fb68d200640 1 -- 172.21.15.31:0/347119553 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6881a6330 con 0x7fb688076060 2024-03-21T10:38:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.891+0000 7fb68d200640 1 -- 172.21.15.31:0/347119553 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb688110420 con 0x7fb688076060 2024-03-21T10:38:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.893+0000 7fb67ea00640 1 -- 172.21.15.31:0/347119553 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fb6780046c0 con 0x7fb688076060 2024-03-21T10:38:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.894+0000 7fb67ea00640 1 --2- 172.21.15.31:0/347119553 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb66c0a2fa0 0x7fb66c0a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.894+0000 7fb67ea00640 1 -- 172.21.15.31:0/347119553 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fb6780cfb10 con 0x7fb688076060 2024-03-21T10:38:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.894+0000 7fb685400640 1 --2- 172.21.15.31:0/347119553 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb66c0a2fa0 0x7fb66c0a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.895+0000 7fb685400640 1 --2- 172.21.15.31:0/347119553 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb66c0a2fa0 0x7fb66c0a5460 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb67402c250 tx=0x7fb67402cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:19.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:19.898+0000 7fb67ea00640 1 -- 172.21.15.31:0/347119553 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb67809c580 con 0x7fb688076060 2024-03-21T10:38:20.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:19 smithi031 ceph-mon[31673]: pgmap v425: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-21T10:38:20.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:19 smithi031 ceph-mon[31673]: from='client.25453 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T10:38:20.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:19 smithi070 ceph-mon[36330]: pgmap v425: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-21T10:38:20.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:19 smithi070 ceph-mon[36330]: from='client.25453 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T10:38:20.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.104+0000 7fb68d200640 1 -- 172.21.15.31:0/347119553 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6880622a0 con 0x7fb688076060 2024-03-21T10:38:20.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.106+0000 7fb67ea00640 1 -- 172.21.15.31:0/347119553 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24553 (secure 0 0 0) 0x7fb67809be80 con 0x7fb688076060 2024-03-21T10:38:20.104 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:38:20.105 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":93,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","created":"2024-03-21T10:25:41.971117+0000","modified":"2024-03-21T10:36:56.301695+0000","last_up_change":"2024-03-21T10:34:07.757196+0000","last_in_change":"2024-03-21T10:33:47.460406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T10:29:44.553968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T10:35:04.945449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T10:35:05.970589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T10:35:09.688711+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e0244d33-00e2-47d5-9f66-34efc138c462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6803","nonce":3089885125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6805","nonce":3089885125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6809","nonce":3089885125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6807","nonce":3089885125}]},"public_addr":"172.21.15.31:6803/3089885125","cluster_addr":"172.21.15.31:6805/3089885125","heartbeat_back_addr":"172.21.15.31:6809/3089885125","heartbeat_front_addr":"172.21.15.31:6807/3089885125","state":["exists","up"]},{"osd":1,"uuid":"76a63504-beab-4354-a6c3-eebc3b16ad03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6801","nonce":618840037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6803","nonce":618840037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6807","nonce":618840037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6805","nonce":618840037}]},"public_addr":"172.21.15.40:6801/618840037","cluster_addr":"172.21.15.40:6803/618840037","heartbeat_back_addr":"172.21.15.40:6807/618840037","heartbeat_front_addr":"172.21.15.40:6805/618840037","state":["exists","up"]},{"osd":2,"uuid":"fa191bd6-9a21-4ee2-8adf-8a63c9cfd821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6801","nonce":2524184117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6803","nonce":2524184117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6807","nonce":2524184117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6805","nonce":2524184117}]},"public_addr":"172.21.15.70:6801/2524184117","cluster_addr":"172.21.15.70:6803/2524184117","heartbeat_back_addr":"172.21.15.70:6807/2524184117","heartbeat_front_addr":"172.21.15.70:6805/2524184117","state":["exists","up"]},{"osd":3,"uuid":"edb44507-ffb0-42a4-827b-08461e1bf952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6811","nonce":2823036296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6813","nonce":2823036296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6817","nonce":2823036296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6815","nonce":2823036296}]},"public_addr":"172.21.15.31:6811/2823036296","cluster_addr":"172.21.15.31:6813/2823036296","heartbeat_back_addr":"172.21.15.31:6817/2823036296","heartbeat_front_addr":"172.21.15.31:6815/2823036296","state":["exists","up"]},{"osd":4,"uuid":"8cb89153-8c3b-431f-b57c-8863fada76dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6809","nonce":4215186309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6811","nonce":4215186309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6815","nonce":4215186309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6813","nonce":4215186309}]},"public_addr":"172.21.15.40:6809/4215186309","cluster_addr":"172.21.15.40:6811/4215186309","heartbeat_back_addr":"172.21.15.40:6815/4215186309","heartbeat_front_addr":"172.21.15.40:6813/4215186309","state":["exists","up"]},{"osd":5,"uuid":"9409505a-d463-408c-a3cd-218380c04d68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6809","nonce":3402830452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6811","nonce":3402830452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6815","nonce":3402830452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6813","nonce":3402830452}]},"public_addr":"172.21.15.70:6809/3402830452","cluster_addr":"172.21.15.70:6811/3402830452","heartbeat_back_addr":"172.21.15.70:6815/3402830452","heartbeat_front_addr":"172.21.15.70:6813/3402830452","state":["exists","up"]},{"osd":6,"uuid":"f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6819","nonce":1193497058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6821","nonce":1193497058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6825","nonce":1193497058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6823","nonce":1193497058}]},"public_addr":"172.21.15.31:6819/1193497058","cluster_addr":"172.21.15.31:6821/1193497058","heartbeat_back_addr":"172.21.15.31:6825/1193497058","heartbeat_front_addr":"172.21.15.31:6823/1193497058","state":["exists","up"]},{"osd":7,"uuid":"db228a87-29a5-45d4-8fe6-a0a554262eb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6817","nonce":3825628119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6819","nonce":3825628119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6823","nonce":3825628119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6821","nonce":3825628119}]},"public_addr":"172.21.15.40:6817/3825628119","cluster_addr":"172.21.15.40:6819/3825628119","heartbeat_back_addr":"172.21.15.40:6823/3825628119","heartbeat_front_addr":"172.21.15.40:6821/3825628119","state":["exists","up"]},{"osd":8,"uuid":"59b9fde0-095d-4e0b-87da-ead37bd8ff1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6817","nonce":534880586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6819","nonce":534880586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6823","nonce":534880586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6821","nonce":534880586}]},"public_addr":"172.21.15.70:6817/534880586","cluster_addr":"172.21.15.70:6819/534880586","heartbeat_back_addr":"172.21.15.70:6823/534880586","heartbeat_front_addr":"172.21.15.70:6821/534880586","state":["exists","up"]},{"osd":9,"uuid":"77f47061-0c5f-4159-b276-e814935d6c1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6827","nonce":587047469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6829","nonce":587047469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6833","nonce":587047469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6831","nonce":587047469}]},"public_addr":"172.21.15.31:6827/587047469","cluster_addr":"172.21.15.31:6829/587047469","heartbeat_back_addr":"172.21.15.31:6833/587047469","heartbeat_front_addr":"172.21.15.31:6831/587047469","state":["exists","up"]},{"osd":10,"uuid":"8853dc43-7bd1-4c93-8063-f7982fd7da38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6825","nonce":512108711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6827","nonce":512108711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6831","nonce":512108711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6829","nonce":512108711}]},"public_addr":"172.21.15.40:6825/512108711","cluster_addr":"172.21.15.40:6827/512108711","heartbeat_back_addr":"172.21.15.40:6831/512108711","heartbeat_front_addr":"172.21.15.40:6829/512108711","state":["exists","up"]},{"osd":11,"uuid":"84b7c2de-c424-45a6-9583-60097e372a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6825","nonce":540305718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6827","nonce":540305718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6831","nonce":540305718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6829","nonce":540305718}]},"public_addr":"172.21.15.70:6825/540305718","cluster_addr":"172.21.15.70:6827/540305718","heartbeat_back_addr":"172.21.15.70:6831/540305718","heartbeat_front_addr":"172.21.15.70:6829/540305718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:28:51.477358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:17.405045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:41.626268+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:15.339568+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:42.676996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:07.516999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:42.412273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:10.365041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:35.699120+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:11.733639+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:39.505442+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:34:05.449301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451930554":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/3342588843":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6801/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/1626448389":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/1910059351":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/344822770":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/3811368512":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/1981437651":"2024-03-22T10:26:16.959165+0000","172.21.15.31:6801/1981437651":"2024-03-22T10:26:16.959165+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T10:38:20.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.109+0000 7fb68d200640 1 -- 172.21.15.31:0/347119553 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb66c0a2fa0 msgr2=0x7fb66c0a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:20.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.109+0000 7fb68d200640 1 --2- 172.21.15.31:0/347119553 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb66c0a2fa0 0x7fb66c0a5460 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb67402c250 tx=0x7fb67402cf90 comp rx=0 tx=0).stop 2024-03-21T10:38:20.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.109+0000 7fb68d200640 1 -- 172.21.15.31:0/347119553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb688076060 msgr2=0x7fb68819dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:20.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.109+0000 7fb68d200640 1 --2- 172.21.15.31:0/347119553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb688076060 0x7fb68819dfc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb67800e9f0 tx=0x7fb67800edf0 comp rx=0 tx=0).stop 2024-03-21T10:38:20.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.109+0000 7fb68d200640 1 -- 172.21.15.31:0/347119553 shutdown_connections 2024-03-21T10:38:20.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.109+0000 7fb68d200640 1 --2- 172.21.15.31:0/347119553 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fb66c0a2fa0 0x7fb66c0a5460 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:20.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.110+0000 7fb68d200640 1 --2- 172.21.15.31:0/347119553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb688077c40 0x7fb6881a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:20.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.110+0000 7fb68d200640 1 --2- 172.21.15.31:0/347119553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6880772a0 0x7fb68819e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:20.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.110+0000 7fb68d200640 1 --2- 172.21.15.31:0/347119553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb688076060 0x7fb68819dfc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:20.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.110+0000 7fb68d200640 1 -- 172.21.15.31:0/347119553 >> 172.21.15.31:0/347119553 conn(0x7fb6880feeb0 msgr2=0x7fb6881005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:20.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.110+0000 7fb68d200640 1 -- 172.21.15.31:0/347119553 shutdown_connections 2024-03-21T10:38:20.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.110+0000 7fb68d200640 1 -- 172.21.15.31:0/347119553 wait complete. 2024-03-21T10:38:20.130 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T10:29:44.553968+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T10:35:04.945449+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T10:35:05.970589+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T10:35:09.688711+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T10:38:20.130 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T10:38:20.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:19 smithi040 ceph-mon[36154]: pgmap v425: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-21T10:38:20.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:19 smithi040 ceph-mon[36154]: from='client.25453 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T10:38:20.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.358+0000 7fcac3200640 1 -- 172.21.15.31:0/577518979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcac4104960 msgr2=0x7fcac4106d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:20.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.358+0000 7fcac3200640 1 --2- 172.21.15.31:0/577518979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcac4104960 0x7fcac4106d50 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcab0009900 tx=0x7fcab002bc30 comp rx=0 tx=0).stop 2024-03-21T10:38:20.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.358+0000 7fcac3200640 1 -- 172.21.15.31:0/577518979 shutdown_connections 2024-03-21T10:38:20.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.358+0000 7fcac3200640 1 --2- 172.21.15.31:0/577518979 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcac4107290 0x7fcac4109680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:20.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.358+0000 7fcac3200640 1 --2- 172.21.15.31:0/577518979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcac4104960 0x7fcac4106d50 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:20.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.358+0000 7fcac3200640 1 --2- 172.21.15.31:0/577518979 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcac4102030 0x7fcac4104420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:20.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.358+0000 7fcac3200640 1 -- 172.21.15.31:0/577518979 >> 172.21.15.31:0/577518979 conn(0x7fcac40fb980 msgr2=0x7fcac40fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:20.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.358+0000 7fcac3200640 1 -- 172.21.15.31:0/577518979 shutdown_connections 2024-03-21T10:38:20.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.358+0000 7fcac3200640 1 -- 172.21.15.31:0/577518979 wait complete. 2024-03-21T10:38:20.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.359+0000 7fcac3200640 1 Processor -- start 2024-03-21T10:38:20.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.359+0000 7fcac3200640 1 -- start start 2024-03-21T10:38:20.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.360+0000 7fcac3200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcac4102030 0x7fcac41019a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:20.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.360+0000 7fcac3200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcac4104960 0x7fcac40fffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:20.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.360+0000 7fcac3200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcac4107290 0x7fcac4100520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:20.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.360+0000 7fcac3200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac410d1e0 con 0x7fcac4107290 2024-03-21T10:38:20.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.360+0000 7fcac3200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac410d060 con 0x7fcac4104960 2024-03-21T10:38:20.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.360+0000 7fcac3200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac410d360 con 0x7fcac4102030 2024-03-21T10:38:20.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.360+0000 7fcac1e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcac4102030 0x7fcac41019a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:20.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.360+0000 7fcac1e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcac4102030 0x7fcac41019a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53552/0 (socket says 172.21.15.31:53552) 2024-03-21T10:38:20.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.360+0000 7fcac1e00640 1 -- 172.21.15.31:0/3055901490 learned_addr learned my addr 172.21.15.31:0/3055901490 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:20.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.360+0000 7fcac1400640 1 --2- 172.21.15.31:0/3055901490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcac4104960 0x7fcac40fffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:20.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.360+0000 7fcac2800640 1 --2- 172.21.15.31:0/3055901490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcac4107290 0x7fcac4100520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:20.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.360+0000 7fcac1e00640 1 -- 172.21.15.31:0/3055901490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcac4104960 msgr2=0x7fcac40fffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:20.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.360+0000 7fcac1e00640 1 --2- 172.21.15.31:0/3055901490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcac4104960 0x7fcac40fffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:20.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.360+0000 7fcac1e00640 1 -- 172.21.15.31:0/3055901490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcac4107290 msgr2=0x7fcac4100520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:20.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.360+0000 7fcac1e00640 1 --2- 172.21.15.31:0/3055901490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcac4107290 0x7fcac4100520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:20.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.361+0000 7fcac1e00640 1 -- 172.21.15.31:0/3055901490 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac4100db0 con 0x7fcac4102030 2024-03-21T10:38:20.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.361+0000 7fcac1e00640 1 --2- 172.21.15.31:0/3055901490 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcac4102030 0x7fcac41019a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcab400e9f0 tx=0x7fcab400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:20.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.361+0000 7fcabaa00640 1 -- 172.21.15.31:0/3055901490 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab400cd80 con 0x7fcac4102030 2024-03-21T10:38:20.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.361+0000 7fcabaa00640 1 -- 172.21.15.31:0/3055901490 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcab4005500 con 0x7fcac4102030 2024-03-21T10:38:20.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.361+0000 7fcac3200640 1 -- 172.21.15.31:0/3055901490 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac41a1980 con 0x7fcac4102030 2024-03-21T10:38:20.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.361+0000 7fcabaa00640 1 -- 172.21.15.31:0/3055901490 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab4010690 con 0x7fcac4102030 2024-03-21T10:38:20.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.361+0000 7fcac3200640 1 -- 172.21.15.31:0/3055901490 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac41a1e50 con 0x7fcac4102030 2024-03-21T10:38:20.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.363+0000 7fcac3200640 1 -- 172.21.15.31:0/3055901490 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcac404f400 con 0x7fcac4102030 2024-03-21T10:38:20.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.366+0000 7fcabaa00640 1 -- 172.21.15.31:0/3055901490 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fcab40046c0 con 0x7fcac4102030 2024-03-21T10:38:20.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.368+0000 7fcabaa00640 1 --2- 172.21.15.31:0/3055901490 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcaa80a2f20 0x7fcaa80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:20.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.368+0000 7fcabaa00640 1 -- 172.21.15.31:0/3055901490 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fcab40cfca0 con 0x7fcac4102030 2024-03-21T10:38:20.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.368+0000 7fcac1400640 1 --2- 172.21.15.31:0/3055901490 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcaa80a2f20 0x7fcaa80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:20.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.369+0000 7fcac1400640 1 --2- 172.21.15.31:0/3055901490 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcaa80a2f20 0x7fcaa80a53e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fcab002c170 tx=0x7fcab002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:20.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.376+0000 7fcabaa00640 1 -- 172.21.15.31:0/3055901490 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcab409c710 con 0x7fcac4102030 2024-03-21T10:38:20.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.577+0000 7fcac3200640 1 -- 172.21.15.31:0/3055901490 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcac402d070 con 0x7fcac4102030 2024-03-21T10:38:20.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.578+0000 7fcabaa00640 1 -- 172.21.15.31:0/3055901490 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcab409c010 con 0x7fcac4102030 2024-03-21T10:38:20.576 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-03-21T10:38:20.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.581+0000 7fcac3200640 1 -- 172.21.15.31:0/3055901490 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcaa80a2f20 msgr2=0x7fcaa80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:20.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.581+0000 7fcac3200640 1 --2- 172.21.15.31:0/3055901490 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcaa80a2f20 0x7fcaa80a53e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fcab002c170 tx=0x7fcab002df90 comp rx=0 tx=0).stop 2024-03-21T10:38:20.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.581+0000 7fcac3200640 1 -- 172.21.15.31:0/3055901490 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcac4102030 msgr2=0x7fcac41019a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:20.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.581+0000 7fcac3200640 1 --2- 172.21.15.31:0/3055901490 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcac4102030 0x7fcac41019a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcab400e9f0 tx=0x7fcab400edf0 comp rx=0 tx=0).stop 2024-03-21T10:38:20.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.581+0000 7fcac3200640 1 -- 172.21.15.31:0/3055901490 shutdown_connections 2024-03-21T10:38:20.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.581+0000 7fcac3200640 1 --2- 172.21.15.31:0/3055901490 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fcaa80a2f20 0x7fcaa80a53e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:20.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.581+0000 7fcac3200640 1 --2- 172.21.15.31:0/3055901490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcac4107290 0x7fcac4100520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:20.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.581+0000 7fcac3200640 1 --2- 172.21.15.31:0/3055901490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcac4104960 0x7fcac40fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:20.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.581+0000 7fcac3200640 1 --2- 172.21.15.31:0/3055901490 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcac4102030 0x7fcac41019a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:20.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.581+0000 7fcac3200640 1 -- 172.21.15.31:0/3055901490 >> 172.21.15.31:0/3055901490 conn(0x7fcac40fb980 msgr2=0x7fcac41028d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:20.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.581+0000 7fcac3200640 1 -- 172.21.15.31:0/3055901490 shutdown_connections 2024-03-21T10:38:20.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.581+0000 7fcac3200640 1 -- 172.21.15.31:0/3055901490 wait complete. 2024-03-21T10:38:20.600 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T10:38:20.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.823+0000 7f07f6e00640 1 -- 172.21.15.31:0/1508060552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07f0105470 msgr2=0x7f07f010bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:20.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.823+0000 7f07f6e00640 1 --2- 172.21.15.31:0/1508060552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07f0105470 0x7f07f010bd00 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f07e8009ca0 tx=0x7f07e802c0f0 comp rx=0 tx=0).stop 2024-03-21T10:38:20.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.823+0000 7f07f6e00640 1 -- 172.21.15.31:0/1508060552 shutdown_connections 2024-03-21T10:38:20.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.823+0000 7f07f6e00640 1 --2- 172.21.15.31:0/1508060552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07f0105470 0x7f07f010bd00 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:20.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.823+0000 7f07f6e00640 1 --2- 172.21.15.31:0/1508060552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f07f0104ab0 0x7f07f0104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:20.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.823+0000 7f07f6e00640 1 --2- 172.21.15.31:0/1508060552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07f01038c0 0x7f07f0103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:20.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.823+0000 7f07f6e00640 1 -- 172.21.15.31:0/1508060552 >> 172.21.15.31:0/1508060552 conn(0x7f07f00fee50 msgr2=0x7f07f0101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:20.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.824+0000 7f07f6e00640 1 -- 172.21.15.31:0/1508060552 shutdown_connections 2024-03-21T10:38:20.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.824+0000 7f07f6e00640 1 -- 172.21.15.31:0/1508060552 wait complete. 2024-03-21T10:38:20.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.825+0000 7f07f6e00640 1 Processor -- start 2024-03-21T10:38:20.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.825+0000 7f07f6e00640 1 -- start start 2024-03-21T10:38:20.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.825+0000 7f07f6e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07f01038c0 0x7f07f0199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:20.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.826+0000 7f07f6e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f07f0104ab0 0x7f07f019a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:20.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.826+0000 7f07f6e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07f0105470 0x7f07f01a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:20.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.826+0000 7f07f6e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f010f520 con 0x7f07f01038c0 2024-03-21T10:38:20.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.826+0000 7f07f6e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f010f3a0 con 0x7f07f0105470 2024-03-21T10:38:20.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.826+0000 7f07f6e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f010f6a0 con 0x7f07f0104ab0 2024-03-21T10:38:20.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.826+0000 7f07efe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07f01038c0 0x7f07f0199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:20.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.826+0000 7f07efe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07f01038c0 0x7f07f0199b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33524/0 (socket says 172.21.15.31:33524) 2024-03-21T10:38:20.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.826+0000 7f07efe00640 1 -- 172.21.15.31:0/3757984134 learned_addr learned my addr 172.21.15.31:0/3757984134 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:20.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.826+0000 7f07f4e00640 1 --2- 172.21.15.31:0/3757984134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07f0105470 0x7f07f01a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:20.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.826+0000 7f07ef400640 1 --2- 172.21.15.31:0/3757984134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f07f0104ab0 0x7f07f019a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:20.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.826+0000 7f07f4e00640 1 -- 172.21.15.31:0/3757984134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f07f0104ab0 msgr2=0x7f07f019a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:20.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.826+0000 7f07f4e00640 1 --2- 172.21.15.31:0/3757984134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f07f0104ab0 0x7f07f019a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:20.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.826+0000 7f07f4e00640 1 -- 172.21.15.31:0/3757984134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07f01038c0 msgr2=0x7f07f0199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:20.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.826+0000 7f07f4e00640 1 --2- 172.21.15.31:0/3757984134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07f01038c0 0x7f07f0199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:20.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.826+0000 7f07f4e00640 1 -- 172.21.15.31:0/3757984134 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f01a1750 con 0x7f07f0105470 2024-03-21T10:38:20.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.827+0000 7f07f4e00640 1 --2- 172.21.15.31:0/3757984134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07f0105470 0x7f07f01a10e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f07e802c630 tx=0x7f07e8032f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:20.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.827+0000 7f07ecc00640 1 -- 172.21.15.31:0/3757984134 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e8041020 con 0x7f07f0105470 2024-03-21T10:38:20.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.827+0000 7f07ecc00640 1 -- 172.21.15.31:0/3757984134 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f07e8034cd0 con 0x7f07f0105470 2024-03-21T10:38:20.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.827+0000 7f07ecc00640 1 -- 172.21.15.31:0/3757984134 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e8039040 con 0x7f07f0105470 2024-03-21T10:38:20.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.827+0000 7f07f6e00640 1 -- 172.21.15.31:0/3757984134 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07f01a19e0 con 0x7f07f0105470 2024-03-21T10:38:20.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.827+0000 7f07f6e00640 1 -- 172.21.15.31:0/3757984134 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07f01a1ea0 con 0x7f07f0105470 2024-03-21T10:38:20.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.828+0000 7f07f6e00640 1 -- 172.21.15.31:0/3757984134 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07f010abe0 con 0x7f07f0105470 2024-03-21T10:38:20.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.830+0000 7f07ecc00640 1 -- 172.21.15.31:0/3757984134 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f07e8044020 con 0x7f07f0105470 2024-03-21T10:38:20.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.831+0000 7f07ecc00640 1 --2- 172.21.15.31:0/3757984134 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f07d40a2f20 0x7f07d40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:20.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.831+0000 7f07ecc00640 1 -- 172.21.15.31:0/3757984134 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f07e80f1530 con 0x7f07f0105470 2024-03-21T10:38:20.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.832+0000 7f07efe00640 1 --2- 172.21.15.31:0/3757984134 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f07d40a2f20 0x7f07d40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:20.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.832+0000 7f07efe00640 1 --2- 172.21.15.31:0/3757984134 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f07d40a2f20 0x7f07d40a53e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f07e00058c0 tx=0x7f07e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:20.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:20.840+0000 7f07ecc00640 1 -- 172.21.15.31:0/3757984134 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f07e804b020 con 0x7f07f0105470 2024-03-21T10:38:21.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.041+0000 7f07f6e00640 1 -- 172.21.15.31:0/3757984134 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f07f002d230 con 0x7f07f0105470 2024-03-21T10:38:21.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.041+0000 7f07ecc00640 1 -- 172.21.15.31:0/3757984134 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f07e80be030 con 0x7f07f0105470 2024-03-21T10:38:21.039 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-03-21T10:38:21.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.044+0000 7f07f6e00640 1 -- 172.21.15.31:0/3757984134 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f07d40a2f20 msgr2=0x7f07d40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:21.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.044+0000 7f07f6e00640 1 --2- 172.21.15.31:0/3757984134 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f07d40a2f20 0x7f07d40a53e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f07e00058c0 tx=0x7f07e0005f90 comp rx=0 tx=0).stop 2024-03-21T10:38:21.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.044+0000 7f07f6e00640 1 -- 172.21.15.31:0/3757984134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07f0105470 msgr2=0x7f07f01a10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:21.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.044+0000 7f07f6e00640 1 --2- 172.21.15.31:0/3757984134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07f0105470 0x7f07f01a10e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f07e802c630 tx=0x7f07e8032f20 comp rx=0 tx=0).stop 2024-03-21T10:38:21.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.044+0000 7f07f6e00640 1 -- 172.21.15.31:0/3757984134 shutdown_connections 2024-03-21T10:38:21.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.044+0000 7f07f6e00640 1 --2- 172.21.15.31:0/3757984134 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f07d40a2f20 0x7f07d40a53e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:21.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.044+0000 7f07f6e00640 1 --2- 172.21.15.31:0/3757984134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07f0105470 0x7f07f01a10e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:21.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.044+0000 7f07f6e00640 1 --2- 172.21.15.31:0/3757984134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f07f0104ab0 0x7f07f019a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:21.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.044+0000 7f07f6e00640 1 --2- 172.21.15.31:0/3757984134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07f01038c0 0x7f07f0199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:21.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.044+0000 7f07f6e00640 1 -- 172.21.15.31:0/3757984134 >> 172.21.15.31:0/3757984134 conn(0x7f07f00fee50 msgr2=0x7f07f0107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:21.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.045+0000 7f07f6e00640 1 -- 172.21.15.31:0/3757984134 shutdown_connections 2024-03-21T10:38:21.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.045+0000 7f07f6e00640 1 -- 172.21.15.31:0/3757984134 wait complete. 2024-03-21T10:38:21.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:20 smithi031 ceph-mon[31673]: from='client.25472 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T10:38:21.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:20 smithi031 ceph-mon[31673]: from='client.25465 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T10:38:21.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:20 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/347119553' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:38:21.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:20 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3055901490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:38:21.061 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T10:38:21.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:20 smithi070 ceph-mon[36330]: from='client.25472 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T10:38:21.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:20 smithi070 ceph-mon[36330]: from='client.25465 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T10:38:21.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:20 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/347119553' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:38:21.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:20 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3055901490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:38:21.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:20 smithi040 ceph-mon[36154]: from='client.25472 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T10:38:21.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:20 smithi040 ceph-mon[36154]: from='client.25465 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T10:38:21.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:20 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/347119553' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:38:21.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:20 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3055901490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:38:21.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.268+0000 7f05e9800640 1 -- 172.21.15.31:0/1028520446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05e41017b0 msgr2=0x7f05e4101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:21.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.268+0000 7f05e9800640 1 --2- 172.21.15.31:0/1028520446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05e41017b0 0x7f05e4101c30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f05d0009900 tx=0x7f05d002bc30 comp rx=0 tx=0).stop 2024-03-21T10:38:21.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.268+0000 7f05e9800640 1 -- 172.21.15.31:0/1028520446 shutdown_connections 2024-03-21T10:38:21.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.268+0000 7f05e9800640 1 --2- 172.21.15.31:0/1028520446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05e4102170 0x7f05e4108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:21.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.268+0000 7f05e9800640 1 --2- 172.21.15.31:0/1028520446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05e41017b0 0x7f05e4101c30 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:21.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.268+0000 7f05e9800640 1 --2- 172.21.15.31:0/1028520446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05e41005c0 0x7f05e41009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:21.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.268+0000 7f05e9800640 1 -- 172.21.15.31:0/1028520446 >> 172.21.15.31:0/1028520446 conn(0x7f05e40fbb30 msgr2=0x7f05e40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:21.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.268+0000 7f05e9800640 1 -- 172.21.15.31:0/1028520446 shutdown_connections 2024-03-21T10:38:21.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.268+0000 7f05e9800640 1 -- 172.21.15.31:0/1028520446 wait complete. 2024-03-21T10:38:21.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.269+0000 7f05e9800640 1 Processor -- start 2024-03-21T10:38:21.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.269+0000 7f05e9800640 1 -- start start 2024-03-21T10:38:21.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.270+0000 7f05e9800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05e41005c0 0x7f05e4199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:21.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.270+0000 7f05e9800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05e41017b0 0x7f05e419a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:21.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.270+0000 7f05e9800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05e4102170 0x7f05e41a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:21.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.270+0000 7f05e9800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e410c220 con 0x7f05e41005c0 2024-03-21T10:38:21.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.270+0000 7f05e9800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e410c0a0 con 0x7f05e41017b0 2024-03-21T10:38:21.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.270+0000 7f05e9800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e410c3a0 con 0x7f05e4102170 2024-03-21T10:38:21.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.270+0000 7f05e2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05e41017b0 0x7f05e419a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:21.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.270+0000 7f05e2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05e41005c0 0x7f05e4199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:21.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.270+0000 7f05e2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05e41017b0 0x7f05e419a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46144/0 (socket says 172.21.15.31:46144) 2024-03-21T10:38:21.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.270+0000 7f05e2000640 1 -- 172.21.15.31:0/2541678704 learned_addr learned my addr 172.21.15.31:0/2541678704 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:21.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.270+0000 7f05e2a00640 1 --2- 172.21.15.31:0/2541678704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05e41005c0 0x7f05e4199c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33530/0 (socket says 172.21.15.31:33530) 2024-03-21T10:38:21.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.270+0000 7f05e3400640 1 --2- 172.21.15.31:0/2541678704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05e4102170 0x7f05e41a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:21.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.270+0000 7f05e2000640 1 -- 172.21.15.31:0/2541678704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05e4102170 msgr2=0x7f05e41a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:21.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.270+0000 7f05e2000640 1 --2- 172.21.15.31:0/2541678704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05e4102170 0x7f05e41a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:21.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.270+0000 7f05e2000640 1 -- 172.21.15.31:0/2541678704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05e41005c0 msgr2=0x7f05e4199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:21.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.271+0000 7f05e2000640 1 --2- 172.21.15.31:0/2541678704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05e41005c0 0x7f05e4199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:21.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.271+0000 7f05e2000640 1 -- 172.21.15.31:0/2541678704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e41a1840 con 0x7f05e41017b0 2024-03-21T10:38:21.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.271+0000 7f05e3400640 1 --2- 172.21.15.31:0/2541678704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05e4102170 0x7f05e41a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:21.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.271+0000 7f05e2a00640 1 --2- 172.21.15.31:0/2541678704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05e41005c0 0x7f05e4199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:21.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.271+0000 7f05e2000640 1 --2- 172.21.15.31:0/2541678704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05e41017b0 0x7f05e419a170 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f05d0009230 tx=0x7f05d002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:21.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.271+0000 7f05db400640 1 -- 172.21.15.31:0/2541678704 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d0041070 con 0x7f05e41017b0 2024-03-21T10:38:21.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.271+0000 7f05db400640 1 -- 172.21.15.31:0/2541678704 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05d0004450 con 0x7f05e41017b0 2024-03-21T10:38:21.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.271+0000 7f05db400640 1 -- 172.21.15.31:0/2541678704 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d0003db0 con 0x7f05e41017b0 2024-03-21T10:38:21.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.272+0000 7f05e9800640 1 -- 172.21.15.31:0/2541678704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05e41a1ad0 con 0x7f05e41017b0 2024-03-21T10:38:21.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.272+0000 7f05e9800640 1 -- 172.21.15.31:0/2541678704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05e41a1f70 con 0x7f05e41017b0 2024-03-21T10:38:21.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.273+0000 7f05e9800640 1 -- 172.21.15.31:0/2541678704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05e41a2140 con 0x7f05e41017b0 2024-03-21T10:38:21.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.275+0000 7f05db400640 1 -- 172.21.15.31:0/2541678704 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f05d0044070 con 0x7f05e41017b0 2024-03-21T10:38:21.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.277+0000 7f05db400640 1 --2- 172.21.15.31:0/2541678704 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f05c80a2fa0 0x7f05c80a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:21.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.277+0000 7f05db400640 1 -- 172.21.15.31:0/2541678704 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f05d00f13d0 con 0x7f05e41017b0 2024-03-21T10:38:21.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.277+0000 7f05e2a00640 1 --2- 172.21.15.31:0/2541678704 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f05c80a2fa0 0x7f05c80a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:21.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.278+0000 7f05e2a00640 1 --2- 172.21.15.31:0/2541678704 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f05c80a2fa0 0x7f05c80a5460 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f05e419b100 tx=0x7f05d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:21.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.283+0000 7f05db400640 1 -- 172.21.15.31:0/2541678704 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f05d00bdea0 con 0x7f05e41017b0 2024-03-21T10:38:21.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.486+0000 7f05e9800640 1 -- 172.21.15.31:0/2541678704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f05e41a2310 con 0x7f05e41017b0 2024-03-21T10:38:21.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.487+0000 7f05db400640 1 -- 172.21.15.31:0/2541678704 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f05d00bd7a0 con 0x7f05e41017b0 2024-03-21T10:38:21.485 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-03-21T10:38:21.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.490+0000 7f05e9800640 1 -- 172.21.15.31:0/2541678704 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f05c80a2fa0 msgr2=0x7f05c80a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:21.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.490+0000 7f05e9800640 1 --2- 172.21.15.31:0/2541678704 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f05c80a2fa0 0x7f05c80a5460 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f05e419b100 tx=0x7f05d4005f90 comp rx=0 tx=0).stop 2024-03-21T10:38:21.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.490+0000 7f05e9800640 1 -- 172.21.15.31:0/2541678704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05e41017b0 msgr2=0x7f05e419a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:21.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.490+0000 7f05e9800640 1 --2- 172.21.15.31:0/2541678704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05e41017b0 0x7f05e419a170 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f05d0009230 tx=0x7f05d002c650 comp rx=0 tx=0).stop 2024-03-21T10:38:21.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.491+0000 7f05e9800640 1 -- 172.21.15.31:0/2541678704 shutdown_connections 2024-03-21T10:38:21.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.491+0000 7f05e9800640 1 --2- 172.21.15.31:0/2541678704 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f05c80a2fa0 0x7f05c80a5460 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:21.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.491+0000 7f05e9800640 1 --2- 172.21.15.31:0/2541678704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05e4102170 0x7f05e41a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:21.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.491+0000 7f05e9800640 1 --2- 172.21.15.31:0/2541678704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05e41017b0 0x7f05e419a170 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:21.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.491+0000 7f05e9800640 1 --2- 172.21.15.31:0/2541678704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05e41005c0 0x7f05e4199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:21.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.491+0000 7f05e9800640 1 -- 172.21.15.31:0/2541678704 >> 172.21.15.31:0/2541678704 conn(0x7f05e40fbb30 msgr2=0x7f05e40fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:21.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.491+0000 7f05e9800640 1 -- 172.21.15.31:0/2541678704 shutdown_connections 2024-03-21T10:38:21.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.491+0000 7f05e9800640 1 -- 172.21.15.31:0/2541678704 wait complete. 2024-03-21T10:38:21.507 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T10:38:21.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.729+0000 7f9e2c000640 1 -- 172.21.15.31:0/3755647096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e24102140 msgr2=0x7f9e24104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:21.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.729+0000 7f9e2c000640 1 --2- 172.21.15.31:0/3755647096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e24102140 0x7f9e24104530 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009be0 tx=0x7f9e1802bef0 comp rx=0 tx=0).stop 2024-03-21T10:38:21.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.729+0000 7f9e2c000640 1 -- 172.21.15.31:0/3755647096 shutdown_connections 2024-03-21T10:38:21.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.729+0000 7f9e2c000640 1 --2- 172.21.15.31:0/3755647096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e241073a0 0x7f9e241097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:21.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.729+0000 7f9e2c000640 1 --2- 172.21.15.31:0/3755647096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e24104a70 0x7f9e24106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:21.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.729+0000 7f9e2c000640 1 --2- 172.21.15.31:0/3755647096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e24102140 0x7f9e24104530 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:21.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.729+0000 7f9e2c000640 1 -- 172.21.15.31:0/3755647096 >> 172.21.15.31:0/3755647096 conn(0x7f9e240fbb30 msgr2=0x7f9e240fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:21.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.729+0000 7f9e2c000640 1 -- 172.21.15.31:0/3755647096 shutdown_connections 2024-03-21T10:38:21.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.729+0000 7f9e2c000640 1 -- 172.21.15.31:0/3755647096 wait complete. 2024-03-21T10:38:21.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.730+0000 7f9e2c000640 1 Processor -- start 2024-03-21T10:38:21.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.731+0000 7f9e2c000640 1 -- start start 2024-03-21T10:38:21.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.731+0000 7f9e2c000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e24102140 0x7f9e24199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:21.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.731+0000 7f9e2c000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e24104a70 0x7f9e2419a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:21.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.731+0000 7f9e2c000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e241073a0 0x7f9e241a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:21.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.731+0000 7f9e2c000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2410d350 con 0x7f9e24102140 2024-03-21T10:38:21.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.731+0000 7f9e2c000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2410d1d0 con 0x7f9e24104a70 2024-03-21T10:38:21.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.731+0000 7f9e2c000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2410d4d0 con 0x7f9e241073a0 2024-03-21T10:38:21.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.731+0000 7f9e29600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e24102140 0x7f9e24199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:21.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.731+0000 7f9e28c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e24104a70 0x7f9e2419a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:21.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.731+0000 7f9e28c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e24104a70 0x7f9e2419a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46166/0 (socket says 172.21.15.31:46166) 2024-03-21T10:38:21.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.731+0000 7f9e28c00640 1 -- 172.21.15.31:0/1577721087 learned_addr learned my addr 172.21.15.31:0/1577721087 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:21.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.731+0000 7f9e2a000640 1 --2- 172.21.15.31:0/1577721087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e241073a0 0x7f9e241a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:21.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.732+0000 7f9e28c00640 1 -- 172.21.15.31:0/1577721087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e241073a0 msgr2=0x7f9e241a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:21.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.732+0000 7f9e28c00640 1 --2- 172.21.15.31:0/1577721087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e241073a0 0x7f9e241a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:21.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.732+0000 7f9e28c00640 1 -- 172.21.15.31:0/1577721087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e24102140 msgr2=0x7f9e24199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:21.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.732+0000 7f9e28c00640 1 --2- 172.21.15.31:0/1577721087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e24102140 0x7f9e24199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:21.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.732+0000 7f9e28c00640 1 -- 172.21.15.31:0/1577721087 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e241a1800 con 0x7f9e24104a70 2024-03-21T10:38:21.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.732+0000 7f9e29600640 1 --2- 172.21.15.31:0/1577721087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e24102140 0x7f9e24199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:21.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.732+0000 7f9e28c00640 1 --2- 172.21.15.31:0/1577721087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e24104a70 0x7f9e2419a120 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9e1000e9a0 tx=0x7f9e1000ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:21.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.732+0000 7f9e22000640 1 -- 172.21.15.31:0/1577721087 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1000cd80 con 0x7f9e24104a70 2024-03-21T10:38:21.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.732+0000 7f9e22000640 1 -- 172.21.15.31:0/1577721087 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e100035f0 con 0x7f9e24104a70 2024-03-21T10:38:21.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.732+0000 7f9e2c000640 1 -- 172.21.15.31:0/1577721087 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e241a1af0 con 0x7f9e24104a70 2024-03-21T10:38:21.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.732+0000 7f9e22000640 1 -- 172.21.15.31:0/1577721087 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e10010670 con 0x7f9e24104a70 2024-03-21T10:38:21.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.733+0000 7f9e2c000640 1 -- 172.21.15.31:0/1577721087 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e241a2040 con 0x7f9e24104a70 2024-03-21T10:38:21.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.734+0000 7f9e2c000640 1 -- 172.21.15.31:0/1577721087 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e24108ef0 con 0x7f9e24104a70 2024-03-21T10:38:21.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.736+0000 7f9e22000640 1 -- 172.21.15.31:0/1577721087 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f9e100037a0 con 0x7f9e24104a70 2024-03-21T10:38:21.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.737+0000 7f9e22000640 1 --2- 172.21.15.31:0/1577721087 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f9e0c0a3070 0x7f9e0c0a5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:21.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.738+0000 7f9e22000640 1 -- 172.21.15.31:0/1577721087 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f9e100cf9a0 con 0x7f9e24104a70 2024-03-21T10:38:21.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.738+0000 7f9e29600640 1 --2- 172.21.15.31:0/1577721087 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f9e0c0a3070 0x7f9e0c0a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:21.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.738+0000 7f9e29600640 1 --2- 172.21.15.31:0/1577721087 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f9e0c0a3070 0x7f9e0c0a5530 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9e180334e0 tx=0x7f9e18034f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:21.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.741+0000 7f9e22000640 1 -- 172.21.15.31:0/1577721087 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9e1009c410 con 0x7f9e24104a70 2024-03-21T10:38:21.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.966+0000 7f9e2c000640 1 -- 172.21.15.31:0/1577721087 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f9e240622a0 con 0x7f9e24104a70 2024-03-21T10:38:21.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.967+0000 7f9e22000640 1 -- 172.21.15.31:0/1577721087 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7f9e1009bd10 con 0x7f9e24104a70 2024-03-21T10:38:21.964 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-03-21T10:38:21.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.970+0000 7f9e2c000640 1 -- 172.21.15.31:0/1577721087 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f9e0c0a3070 msgr2=0x7f9e0c0a5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:21.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.970+0000 7f9e2c000640 1 --2- 172.21.15.31:0/1577721087 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f9e0c0a3070 0x7f9e0c0a5530 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9e180334e0 tx=0x7f9e18034f90 comp rx=0 tx=0).stop 2024-03-21T10:38:21.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.970+0000 7f9e2c000640 1 -- 172.21.15.31:0/1577721087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e24104a70 msgr2=0x7f9e2419a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:21.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.970+0000 7f9e2c000640 1 --2- 172.21.15.31:0/1577721087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e24104a70 0x7f9e2419a120 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9e1000e9a0 tx=0x7f9e1000ee60 comp rx=0 tx=0).stop 2024-03-21T10:38:21.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.970+0000 7f9e2c000640 1 -- 172.21.15.31:0/1577721087 shutdown_connections 2024-03-21T10:38:21.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.970+0000 7f9e2c000640 1 --2- 172.21.15.31:0/1577721087 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f9e0c0a3070 0x7f9e0c0a5530 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:21.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.970+0000 7f9e2c000640 1 --2- 172.21.15.31:0/1577721087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e241073a0 0x7f9e241a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:21.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.970+0000 7f9e2c000640 1 --2- 172.21.15.31:0/1577721087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e24104a70 0x7f9e2419a120 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:21.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.970+0000 7f9e2c000640 1 --2- 172.21.15.31:0/1577721087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e24102140 0x7f9e24199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:21.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.970+0000 7f9e2c000640 1 -- 172.21.15.31:0/1577721087 >> 172.21.15.31:0/1577721087 conn(0x7f9e240fbb30 msgr2=0x7f9e24101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:21.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.970+0000 7f9e2c000640 1 -- 172.21.15.31:0/1577721087 shutdown_connections 2024-03-21T10:38:21.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:21.970+0000 7f9e2c000640 1 -- 172.21.15.31:0/1577721087 wait complete. 2024-03-21T10:38:21.986 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:38:22.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:21 smithi031 ceph-mon[31673]: pgmap v426: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-21T10:38:22.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:21 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3757984134' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:38:22.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:21 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2541678704' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:38:22.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:21 smithi070 ceph-mon[36330]: pgmap v426: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-21T10:38:22.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:21 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3757984134' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:38:22.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:21 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2541678704' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:38:22.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:21 smithi040 ceph-mon[36154]: pgmap v426: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-21T10:38:22.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:21 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3757984134' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:38:22.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:21 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2541678704' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:38:22.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.199+0000 7fab68400640 1 -- 172.21.15.31:0/4202523585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab600756e0 msgr2=0x7fab60073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:22.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.199+0000 7fab68400640 1 --2- 172.21.15.31:0/4202523585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab600756e0 0x7fab60073b30 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fab54009930 tx=0x7fab5402bc30 comp rx=0 tx=0).stop 2024-03-21T10:38:22.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.199+0000 7fab68400640 1 -- 172.21.15.31:0/4202523585 shutdown_connections 2024-03-21T10:38:22.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.199+0000 7fab68400640 1 --2- 172.21.15.31:0/4202523585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab6010dd00 0x7fab601100f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:22.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.199+0000 7fab68400640 1 --2- 172.21.15.31:0/4202523585 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab60074100 0x7fab60074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:22.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.199+0000 7fab68400640 1 --2- 172.21.15.31:0/4202523585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab600756e0 0x7fab60073b30 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:22.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.199+0000 7fab68400640 1 -- 172.21.15.31:0/4202523585 >> 172.21.15.31:0/4202523585 conn(0x7fab600fccc0 msgr2=0x7fab600ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:22.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.199+0000 7fab68400640 1 -- 172.21.15.31:0/4202523585 shutdown_connections 2024-03-21T10:38:22.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.199+0000 7fab68400640 1 -- 172.21.15.31:0/4202523585 wait complete. 2024-03-21T10:38:22.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.200+0000 7fab68400640 1 Processor -- start 2024-03-21T10:38:22.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.200+0000 7fab68400640 1 -- start start 2024-03-21T10:38:22.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.201+0000 7fab68400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab60074100 0x7fab6019dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:22.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.201+0000 7fab68400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab600756e0 0x7fab6019e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:22.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.201+0000 7fab68400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab6010dd00 0x7fab601a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:22.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.201+0000 7fab68400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab60113910 con 0x7fab600756e0 2024-03-21T10:38:22.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.201+0000 7fab68400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab60113790 con 0x7fab60074100 2024-03-21T10:38:22.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.201+0000 7fab68400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab60113a90 con 0x7fab6010dd00 2024-03-21T10:38:22.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.201+0000 7fab65a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab60074100 0x7fab6019dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:22.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.201+0000 7fab65a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab60074100 0x7fab6019dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46188/0 (socket says 172.21.15.31:46188) 2024-03-21T10:38:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.201+0000 7fab65a00640 1 -- 172.21.15.31:0/2548878089 learned_addr learned my addr 172.21.15.31:0/2548878089 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.201+0000 7fab66400640 1 --2- 172.21.15.31:0/2548878089 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab6010dd00 0x7fab601a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.201+0000 7fab65000640 1 --2- 172.21.15.31:0/2548878089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab600756e0 0x7fab6019e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.201+0000 7fab65a00640 1 -- 172.21.15.31:0/2548878089 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab6010dd00 msgr2=0x7fab601a5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.201+0000 7fab65a00640 1 --2- 172.21.15.31:0/2548878089 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab6010dd00 0x7fab601a5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.201+0000 7fab65a00640 1 -- 172.21.15.31:0/2548878089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab600756e0 msgr2=0x7fab6019e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.201+0000 7fab65a00640 1 --2- 172.21.15.31:0/2548878089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab600756e0 0x7fab6019e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.201+0000 7fab65a00640 1 -- 172.21.15.31:0/2548878089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab601a5be0 con 0x7fab60074100 2024-03-21T10:38:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.201+0000 7fab65000640 1 --2- 172.21.15.31:0/2548878089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab600756e0 0x7fab6019e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.202+0000 7fab65a00640 1 --2- 172.21.15.31:0/2548878089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab60074100 0x7fab6019dfc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fab54000c00 tx=0x7fab54005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.202+0000 7fab5e000640 1 -- 172.21.15.31:0/2548878089 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab54041070 con 0x7fab60074100 2024-03-21T10:38:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.202+0000 7fab5e000640 1 -- 172.21.15.31:0/2548878089 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab540044d0 con 0x7fab60074100 2024-03-21T10:38:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.202+0000 7fab5e000640 1 -- 172.21.15.31:0/2548878089 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab54034440 con 0x7fab60074100 2024-03-21T10:38:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.202+0000 7fab68400640 1 -- 172.21.15.31:0/2548878089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab601a5e70 con 0x7fab60074100 2024-03-21T10:38:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.202+0000 7fab68400640 1 -- 172.21.15.31:0/2548878089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab601a62d0 con 0x7fab60074100 2024-03-21T10:38:22.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.203+0000 7fab68400640 1 -- 172.21.15.31:0/2548878089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab60103ed0 con 0x7fab60074100 2024-03-21T10:38:22.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.207+0000 7fab5e000640 1 -- 172.21.15.31:0/2548878089 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fab54044070 con 0x7fab60074100 2024-03-21T10:38:22.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.208+0000 7fab5e000640 1 --2- 172.21.15.31:0/2548878089 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fab480a2fa0 0x7fab480a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:22.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.208+0000 7fab5e000640 1 -- 172.21.15.31:0/2548878089 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fab540f12d0 con 0x7fab60074100 2024-03-21T10:38:22.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.208+0000 7fab65000640 1 --2- 172.21.15.31:0/2548878089 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fab480a2fa0 0x7fab480a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:22.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.209+0000 7fab65000640 1 --2- 172.21.15.31:0/2548878089 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fab480a2fa0 0x7fab480a5460 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fab6019f4d0 tx=0x7fab4c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:22.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.212+0000 7fab5e000640 1 -- 172.21.15.31:0/2548878089 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fab540bdd40 con 0x7fab60074100 2024-03-21T10:38:22.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.460+0000 7fab68400640 1 -- 172.21.15.31:0/2548878089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab6002d2b0 con 0x7fab60074100 2024-03-21T10:38:22.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.461+0000 7fab5e000640 1 -- 172.21.15.31:0/2548878089 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11384 (secure 0 0 0) 0x7fab540b9020 con 0x7fab60074100 2024-03-21T10:38:22.459 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:38:22.459 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:38:12.319122+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24547,24596]},"id":1}]} 2024-03-21T10:38:22.459 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 24 2024-03-21T10:38:22.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.464+0000 7fab68400640 1 -- 172.21.15.31:0/2548878089 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fab480a2fa0 msgr2=0x7fab480a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:22.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.464+0000 7fab68400640 1 --2- 172.21.15.31:0/2548878089 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fab480a2fa0 0x7fab480a5460 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fab6019f4d0 tx=0x7fab4c005f90 comp rx=0 tx=0).stop 2024-03-21T10:38:22.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.464+0000 7fab68400640 1 -- 172.21.15.31:0/2548878089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab60074100 msgr2=0x7fab6019dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:22.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.464+0000 7fab68400640 1 --2- 172.21.15.31:0/2548878089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab60074100 0x7fab6019dfc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fab54000c00 tx=0x7fab54005d50 comp rx=0 tx=0).stop 2024-03-21T10:38:22.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.464+0000 7fab68400640 1 -- 172.21.15.31:0/2548878089 shutdown_connections 2024-03-21T10:38:22.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.464+0000 7fab68400640 1 --2- 172.21.15.31:0/2548878089 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fab480a2fa0 0x7fab480a5460 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:22.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.464+0000 7fab68400640 1 --2- 172.21.15.31:0/2548878089 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab6010dd00 0x7fab601a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:22.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.464+0000 7fab68400640 1 --2- 172.21.15.31:0/2548878089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab600756e0 0x7fab6019e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:22.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.464+0000 7fab68400640 1 --2- 172.21.15.31:0/2548878089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab60074100 0x7fab6019dfc0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:22.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.464+0000 7fab68400640 1 -- 172.21.15.31:0/2548878089 >> 172.21.15.31:0/2548878089 conn(0x7fab600fccc0 msgr2=0x7fab601010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:22.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.464+0000 7fab68400640 1 -- 172.21.15.31:0/2548878089 shutdown_connections 2024-03-21T10:38:22.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.464+0000 7fab68400640 1 -- 172.21.15.31:0/2548878089 wait complete. 2024-03-21T10:38:22.485 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T10:38:22.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.705+0000 7f855ee00640 1 -- 172.21.15.31:0/2408868884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85581027b0 msgr2=0x7f8558102c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:22.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.705+0000 7f855ee00640 1 --2- 172.21.15.31:0/2408868884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85581027b0 0x7f8558102c30 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8544009900 tx=0x7f854402bc30 comp rx=0 tx=0).stop 2024-03-21T10:38:22.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.706+0000 7f855ee00640 1 -- 172.21.15.31:0/2408868884 shutdown_connections 2024-03-21T10:38:22.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.706+0000 7f855ee00640 1 --2- 172.21.15.31:0/2408868884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8558103170 0x7f8558109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:22.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.706+0000 7f855ee00640 1 --2- 172.21.15.31:0/2408868884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85581027b0 0x7f8558102c30 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:22.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.706+0000 7f855ee00640 1 --2- 172.21.15.31:0/2408868884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85581015c0 0x7f85581019e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:22.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.706+0000 7f855ee00640 1 -- 172.21.15.31:0/2408868884 >> 172.21.15.31:0/2408868884 conn(0x7f85580fcb30 msgr2=0x7f85580fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:22.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.706+0000 7f855ee00640 1 -- 172.21.15.31:0/2408868884 shutdown_connections 2024-03-21T10:38:22.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.706+0000 7f855ee00640 1 -- 172.21.15.31:0/2408868884 wait complete. 2024-03-21T10:38:22.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.707+0000 7f855ee00640 1 Processor -- start 2024-03-21T10:38:22.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.707+0000 7f855ee00640 1 -- start start 2024-03-21T10:38:22.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.708+0000 7f855ee00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85581015c0 0x7f8558075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:22.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.708+0000 7f855ee00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85581027b0 0x7f8558073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:22.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.708+0000 7f855ee00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8558103170 0x7f8558073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:22.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.708+0000 7f855ee00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855810d220 con 0x7f85581027b0 2024-03-21T10:38:22.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.708+0000 7f855ee00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855810d0a0 con 0x7f85581015c0 2024-03-21T10:38:22.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.708+0000 7f855ee00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855810d3a0 con 0x7f8558103170 2024-03-21T10:38:22.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.708+0000 7f855da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85581015c0 0x7f8558075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:22.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.708+0000 7f855da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85581015c0 0x7f8558075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46198/0 (socket says 172.21.15.31:46198) 2024-03-21T10:38:22.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.708+0000 7f855da00640 1 -- 172.21.15.31:0/3622328584 learned_addr learned my addr 172.21.15.31:0/3622328584 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:22.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.708+0000 7f855d000640 1 --2- 172.21.15.31:0/3622328584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85581027b0 0x7f8558073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:22.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.708+0000 7f855e400640 1 --2- 172.21.15.31:0/3622328584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8558103170 0x7f8558073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:22.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.709+0000 7f855da00640 1 -- 172.21.15.31:0/3622328584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8558103170 msgr2=0x7f8558073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:22.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.709+0000 7f855da00640 1 --2- 172.21.15.31:0/3622328584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8558103170 0x7f8558073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:22.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.709+0000 7f855da00640 1 -- 172.21.15.31:0/3622328584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85581027b0 msgr2=0x7f8558073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:22.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.709+0000 7f855da00640 1 --2- 172.21.15.31:0/3622328584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85581027b0 0x7f8558073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:22.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.709+0000 7f855da00640 1 -- 172.21.15.31:0/3622328584 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8558074860 con 0x7f85581015c0 2024-03-21T10:38:22.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.709+0000 7f855da00640 1 --2- 172.21.15.31:0/3622328584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85581015c0 0x7f8558075450 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f854c00e9c0 tx=0x7f854c00edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:22.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.709+0000 7f8556000640 1 -- 172.21.15.31:0/3622328584 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c00ccc0 con 0x7f85581015c0 2024-03-21T10:38:22.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.709+0000 7f8556000640 1 -- 172.21.15.31:0/3622328584 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f854c0035f0 con 0x7f85581015c0 2024-03-21T10:38:22.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.709+0000 7f8556000640 1 -- 172.21.15.31:0/3622328584 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c00ce60 con 0x7f85581015c0 2024-03-21T10:38:22.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.709+0000 7f855ee00640 1 -- 172.21.15.31:0/3622328584 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85581a1a10 con 0x7f85581015c0 2024-03-21T10:38:22.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.709+0000 7f855ee00640 1 -- 172.21.15.31:0/3622328584 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85581a1ee0 con 0x7f85581015c0 2024-03-21T10:38:22.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.710+0000 7f855ee00640 1 -- 172.21.15.31:0/3622328584 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f855804f470 con 0x7f85581015c0 2024-03-21T10:38:22.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.712+0000 7f8556000640 1 -- 172.21.15.31:0/3622328584 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f854c0037a0 con 0x7f85581015c0 2024-03-21T10:38:22.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.714+0000 7f8556000640 1 --2- 172.21.15.31:0/3622328584 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f85400a2f90 0x7f85400a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:22.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.714+0000 7f8556000640 1 -- 172.21.15.31:0/3622328584 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f854c0cfb10 con 0x7f85581015c0 2024-03-21T10:38:22.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.714+0000 7f855d000640 1 --2- 172.21.15.31:0/3622328584 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f85400a2f90 0x7f85400a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:22.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.715+0000 7f855d000640 1 --2- 172.21.15.31:0/3622328584 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f85400a2f90 0x7f85400a5450 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8544006010 tx=0x7f8544000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:22.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.717+0000 7f8556000640 1 -- 172.21.15.31:0/3622328584 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f854c09c5e0 con 0x7f85581015c0 2024-03-21T10:38:22.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.924+0000 7f855ee00640 1 -- 172.21.15.31:0/3622328584 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85580622a0 con 0x7f85581015c0 2024-03-21T10:38:22.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.926+0000 7f8556000640 1 -- 172.21.15.31:0/3622328584 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24553 (secure 0 0 0) 0x7f854c09bee0 con 0x7f85581015c0 2024-03-21T10:38:22.925 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:38:22.925 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":93,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","created":"2024-03-21T10:25:41.971117+0000","modified":"2024-03-21T10:36:56.301695+0000","last_up_change":"2024-03-21T10:34:07.757196+0000","last_in_change":"2024-03-21T10:33:47.460406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T10:29:44.553968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T10:35:04.945449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T10:35:05.970589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T10:35:09.688711+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e0244d33-00e2-47d5-9f66-34efc138c462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6803","nonce":3089885125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6805","nonce":3089885125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6809","nonce":3089885125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6807","nonce":3089885125}]},"public_addr":"172.21.15.31:6803/3089885125","cluster_addr":"172.21.15.31:6805/3089885125","heartbeat_back_addr":"172.21.15.31:6809/3089885125","heartbeat_front_addr":"172.21.15.31:6807/3089885125","state":["exists","up"]},{"osd":1,"uuid":"76a63504-beab-4354-a6c3-eebc3b16ad03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6801","nonce":618840037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6803","nonce":618840037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6807","nonce":618840037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6805","nonce":618840037}]},"public_addr":"172.21.15.40:6801/618840037","cluster_addr":"172.21.15.40:6803/618840037","heartbeat_back_addr":"172.21.15.40:6807/618840037","heartbeat_front_addr":"172.21.15.40:6805/618840037","state":["exists","up"]},{"osd":2,"uuid":"fa191bd6-9a21-4ee2-8adf-8a63c9cfd821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6801","nonce":2524184117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6803","nonce":2524184117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6807","nonce":2524184117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6805","nonce":2524184117}]},"public_addr":"172.21.15.70:6801/2524184117","cluster_addr":"172.21.15.70:6803/2524184117","heartbeat_back_addr":"172.21.15.70:6807/2524184117","heartbeat_front_addr":"172.21.15.70:6805/2524184117","state":["exists","up"]},{"osd":3,"uuid":"edb44507-ffb0-42a4-827b-08461e1bf952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6811","nonce":2823036296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6813","nonce":2823036296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6817","nonce":2823036296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6815","nonce":2823036296}]},"public_addr":"172.21.15.31:6811/2823036296","cluster_addr":"172.21.15.31:6813/2823036296","heartbeat_back_addr":"172.21.15.31:6817/2823036296","heartbeat_front_addr":"172.21.15.31:6815/2823036296","state":["exists","up"]},{"osd":4,"uuid":"8cb89153-8c3b-431f-b57c-8863fada76dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6809","nonce":4215186309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6811","nonce":4215186309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6815","nonce":4215186309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6813","nonce":4215186309}]},"public_addr":"172.21.15.40:6809/4215186309","cluster_addr":"172.21.15.40:6811/4215186309","heartbeat_back_addr":"172.21.15.40:6815/4215186309","heartbeat_front_addr":"172.21.15.40:6813/4215186309","state":["exists","up"]},{"osd":5,"uuid":"9409505a-d463-408c-a3cd-218380c04d68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6809","nonce":3402830452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6811","nonce":3402830452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6815","nonce":3402830452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6813","nonce":3402830452}]},"public_addr":"172.21.15.70:6809/3402830452","cluster_addr":"172.21.15.70:6811/3402830452","heartbeat_back_addr":"172.21.15.70:6815/3402830452","heartbeat_front_addr":"172.21.15.70:6813/3402830452","state":["exists","up"]},{"osd":6,"uuid":"f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6819","nonce":1193497058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6821","nonce":1193497058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6825","nonce":1193497058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6823","nonce":1193497058}]},"public_addr":"172.21.15.31:6819/1193497058","cluster_addr":"172.21.15.31:6821/1193497058","heartbeat_back_addr":"172.21.15.31:6825/1193497058","heartbeat_front_addr":"172.21.15.31:6823/1193497058","state":["exists","up"]},{"osd":7,"uuid":"db228a87-29a5-45d4-8fe6-a0a554262eb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6817","nonce":3825628119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6819","nonce":3825628119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6823","nonce":3825628119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6821","nonce":3825628119}]},"public_addr":"172.21.15.40:6817/3825628119","cluster_addr":"172.21.15.40:6819/3825628119","heartbeat_back_addr":"172.21.15.40:6823/3825628119","heartbeat_front_addr":"172.21.15.40:6821/3825628119","state":["exists","up"]},{"osd":8,"uuid":"59b9fde0-095d-4e0b-87da-ead37bd8ff1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6817","nonce":534880586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6819","nonce":534880586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6823","nonce":534880586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6821","nonce":534880586}]},"public_addr":"172.21.15.70:6817/534880586","cluster_addr":"172.21.15.70:6819/534880586","heartbeat_back_addr":"172.21.15.70:6823/534880586","heartbeat_front_addr":"172.21.15.70:6821/534880586","state":["exists","up"]},{"osd":9,"uuid":"77f47061-0c5f-4159-b276-e814935d6c1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6827","nonce":587047469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6829","nonce":587047469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6833","nonce":587047469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6831","nonce":587047469}]},"public_addr":"172.21.15.31:6827/587047469","cluster_addr":"172.21.15.31:6829/587047469","heartbeat_back_addr":"172.21.15.31:6833/587047469","heartbeat_front_addr":"172.21.15.31:6831/587047469","state":["exists","up"]},{"osd":10,"uuid":"8853dc43-7bd1-4c93-8063-f7982fd7da38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6825","nonce":512108711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6827","nonce":512108711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6831","nonce":512108711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6829","nonce":512108711}]},"public_addr":"172.21.15.40:6825/512108711","cluster_addr":"172.21.15.40:6827/512108711","heartbeat_back_addr":"172.21.15.40:6831/512108711","heartbeat_front_addr":"172.21.15.40:6829/512108711","state":["exists","up"]},{"osd":11,"uuid":"84b7c2de-c424-45a6-9583-60097e372a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6825","nonce":540305718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6827","nonce":540305718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6831","nonce":540305718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6829","nonce":540305718}]},"public_addr":"172.21.15.70:6825/540305718","cluster_addr":"172.21.15.70:6827/540305718","heartbeat_back_addr":"172.21.15.70:6831/540305718","heartbeat_front_addr":"172.21.15.70:6829/540305718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:28:51.477358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:17.405045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:41.626268+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:15.339568+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:42.676996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:07.516999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:42.412273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:10.365041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:35.699120+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:11.733639+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:39.505442+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:34:05.449301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/1451930554":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/3342588843":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6801/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/1626448389":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/1910059351":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/344822770":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/3811368512":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/1981437651":"2024-03-22T10:26:16.959165+0000","172.21.15.31:6801/1981437651":"2024-03-22T10:26:16.959165+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T10:38:22.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.929+0000 7f855ee00640 1 -- 172.21.15.31:0/3622328584 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f85400a2f90 msgr2=0x7f85400a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:22.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.929+0000 7f855ee00640 1 --2- 172.21.15.31:0/3622328584 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f85400a2f90 0x7f85400a5450 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8544006010 tx=0x7f8544000bc0 comp rx=0 tx=0).stop 2024-03-21T10:38:22.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.929+0000 7f855ee00640 1 -- 172.21.15.31:0/3622328584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85581015c0 msgr2=0x7f8558075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:22.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.929+0000 7f855ee00640 1 --2- 172.21.15.31:0/3622328584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85581015c0 0x7f8558075450 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f854c00e9c0 tx=0x7f854c00edc0 comp rx=0 tx=0).stop 2024-03-21T10:38:22.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.930+0000 7f855ee00640 1 -- 172.21.15.31:0/3622328584 shutdown_connections 2024-03-21T10:38:22.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.930+0000 7f855ee00640 1 --2- 172.21.15.31:0/3622328584 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f85400a2f90 0x7f85400a5450 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:22.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.930+0000 7f855ee00640 1 --2- 172.21.15.31:0/3622328584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8558103170 0x7f8558073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:22.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.930+0000 7f855ee00640 1 --2- 172.21.15.31:0/3622328584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85581027b0 0x7f8558073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:22.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.930+0000 7f855ee00640 1 --2- 172.21.15.31:0/3622328584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85581015c0 0x7f8558075450 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:22.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.930+0000 7f855ee00640 1 -- 172.21.15.31:0/3622328584 >> 172.21.15.31:0/3622328584 conn(0x7f85580fcb30 msgr2=0x7f8558105ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:22.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.930+0000 7f855ee00640 1 -- 172.21.15.31:0/3622328584 shutdown_connections 2024-03-21T10:38:22.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:22.930+0000 7f855ee00640 1 -- 172.21.15.31:0/3622328584 wait complete. 2024-03-21T10:38:22.949 DEBUG:tasks.fwd_scrub:Yielding 2024-03-21T10:38:22.949 INFO:teuthology.run_tasks:Running task exec... 2024-03-21T10:38:22.965 INFO:teuthology.task.exec:Executing custom commands... 2024-03-21T10:38:22.965 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2024-03-21T10:38:22.965 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-21T10:38:22.965 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-21T10:38:22.965 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-21T10:38:22.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:38:22.966 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-21T10:38:23.043 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:22 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1577721087' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:38:23.043 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:22 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2548878089' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:38:23.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:22 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1577721087' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:38:23.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:22 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2548878089' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:38:23.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.177+0000 7ff78cc00640 1 -- 172.21.15.31:0/83136633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff788105270 msgr2=0x7ff7881056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:23.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.177+0000 7ff78cc00640 1 --2- 172.21.15.31:0/83136633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff788105270 0x7ff7881056f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff780009ca0 tx=0x7ff78002c0f0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.177+0000 7ff78cc00640 1 -- 172.21.15.31:0/83136633 shutdown_connections 2024-03-21T10:38:23.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.177+0000 7ff78cc00640 1 --2- 172.21.15.31:0/83136633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff788105270 0x7ff7881056f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.177+0000 7ff78cc00640 1 --2- 172.21.15.31:0/83136633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7881028c0 0x7ff788104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.177+0000 7ff78cc00640 1 --2- 172.21.15.31:0/83136633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7880fff50 0x7ff788102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.177+0000 7ff78cc00640 1 -- 172.21.15.31:0/83136633 >> 172.21.15.31:0/83136633 conn(0x7ff7880fbb70 msgr2=0x7ff7880fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:23.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.177+0000 7ff78cc00640 1 -- 172.21.15.31:0/83136633 shutdown_connections 2024-03-21T10:38:23.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.177+0000 7ff78cc00640 1 -- 172.21.15.31:0/83136633 wait complete. 2024-03-21T10:38:23.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.178+0000 7ff78cc00640 1 Processor -- start 2024-03-21T10:38:23.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.178+0000 7ff78cc00640 1 -- start start 2024-03-21T10:38:23.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.179+0000 7ff78cc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7880fff50 0x7ff78819df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.179+0000 7ff78cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7881028c0 0x7ff78819e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.179+0000 7ff78cc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff788105270 0x7ff7881a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.179+0000 7ff78cc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff788113950 con 0x7ff7880fff50 2024-03-21T10:38:23.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.179+0000 7ff78cc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7881137d0 con 0x7ff7881028c0 2024-03-21T10:38:23.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.179+0000 7ff78cc00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff788113ad0 con 0x7ff788105270 2024-03-21T10:38:23.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.179+0000 7ff785400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7881028c0 0x7ff78819e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.179+0000 7ff785e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7880fff50 0x7ff78819df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.179+0000 7ff785400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7881028c0 0x7ff78819e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46218/0 (socket says 172.21.15.31:46218) 2024-03-21T10:38:23.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.179+0000 7ff785400640 1 -- 172.21.15.31:0/1347166313 learned_addr learned my addr 172.21.15.31:0/1347166313 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:23.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.179+0000 7ff786800640 1 --2- 172.21.15.31:0/1347166313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff788105270 0x7ff7881a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.179+0000 7ff786800640 1 -- 172.21.15.31:0/1347166313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7881028c0 msgr2=0x7ff78819e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:23.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.179+0000 7ff786800640 1 --2- 172.21.15.31:0/1347166313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7881028c0 0x7ff78819e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.179+0000 7ff786800640 1 -- 172.21.15.31:0/1347166313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7880fff50 msgr2=0x7ff78819df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:23.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.179+0000 7ff786800640 1 --2- 172.21.15.31:0/1347166313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7880fff50 0x7ff78819df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.180+0000 7ff786800640 1 -- 172.21.15.31:0/1347166313 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7881a5b60 con 0x7ff788105270 2024-03-21T10:38:23.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.180+0000 7ff786800640 1 --2- 172.21.15.31:0/1347166313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff788105270 0x7ff7881a54f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff78002c630 tx=0x7ff780032f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:23.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.180+0000 7ff77ea00640 1 -- 172.21.15.31:0/1347166313 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff780041020 con 0x7ff788105270 2024-03-21T10:38:23.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.180+0000 7ff77ea00640 1 -- 172.21.15.31:0/1347166313 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff780034cd0 con 0x7ff788105270 2024-03-21T10:38:23.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.180+0000 7ff78cc00640 1 -- 172.21.15.31:0/1347166313 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7881a5df0 con 0x7ff788105270 2024-03-21T10:38:23.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.180+0000 7ff77ea00640 1 -- 172.21.15.31:0/1347166313 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff780039040 con 0x7ff788105270 2024-03-21T10:38:23.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.180+0000 7ff78cc00640 1 -- 172.21.15.31:0/1347166313 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7881a6250 con 0x7ff788105270 2024-03-21T10:38:23.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.182+0000 7ff78cc00640 1 -- 172.21.15.31:0/1347166313 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7880fd660 con 0x7ff788105270 2024-03-21T10:38:23.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:22 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1577721087' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:38:23.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:22 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2548878089' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:38:23.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.184+0000 7ff77ea00640 1 -- 172.21.15.31:0/1347166313 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7ff780044020 con 0x7ff788105270 2024-03-21T10:38:23.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.185+0000 7ff77ea00640 1 --2- 172.21.15.31:0/1347166313 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff76c0a2f10 0x7ff76c0a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.185+0000 7ff77ea00640 1 -- 172.21.15.31:0/1347166313 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7ff7800f1530 con 0x7ff788105270 2024-03-21T10:38:23.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.185+0000 7ff785e00640 1 --2- 172.21.15.31:0/1347166313 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff76c0a2f10 0x7ff76c0a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.186+0000 7ff785e00640 1 --2- 172.21.15.31:0/1347166313 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff76c0a2f10 0x7ff76c0a53d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff778009bd0 tx=0x7ff778005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:23.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.189+0000 7ff77ea00640 1 -- 172.21.15.31:0/1347166313 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff78004b020 con 0x7ff788105270 2024-03-21T10:38:23.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.190+0000 7fd04ce00640 1 -- 172.21.15.31:0/4221225085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd048105080 msgr2=0x7fd048105500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:23.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.190+0000 7fd04ce00640 1 --2- 172.21.15.31:0/4221225085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd048105080 0x7fd048105500 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd040009be0 tx=0x7fd04002bef0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.190+0000 7fd04ce00640 1 -- 172.21.15.31:0/4221225085 shutdown_connections 2024-03-21T10:38:23.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.190+0000 7fd04ce00640 1 --2- 172.21.15.31:0/4221225085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd048105080 0x7fd048105500 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.190+0000 7fd04ce00640 1 --2- 172.21.15.31:0/4221225085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0481026d0 0x7fd048104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.190+0000 7fd04ce00640 1 --2- 172.21.15.31:0/4221225085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0480ffd60 0x7fd048102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.190+0000 7fd04ce00640 1 -- 172.21.15.31:0/4221225085 >> 172.21.15.31:0/4221225085 conn(0x7fd0480fb980 msgr2=0x7fd0480fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:23.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.190+0000 7fd04ce00640 1 -- 172.21.15.31:0/4221225085 shutdown_connections 2024-03-21T10:38:23.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.190+0000 7fd04ce00640 1 -- 172.21.15.31:0/4221225085 wait complete. 2024-03-21T10:38:23.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.191+0000 7fd04ce00640 1 Processor -- start 2024-03-21T10:38:23.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.191+0000 7fd04ce00640 1 -- start start 2024-03-21T10:38:23.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.192+0000 7fd04ce00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0480ffd60 0x7fd04819ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.192+0000 7fd04ce00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0481026d0 0x7fd04819e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.192+0000 7fd04ce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd048105080 0x7fd0481a5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.192+0000 7fd04ce00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0481137b0 con 0x7fd048105080 2024-03-21T10:38:23.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.192+0000 7fd04ce00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd048113630 con 0x7fd0481026d0 2024-03-21T10:38:23.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.192+0000 7fd04ce00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd048113930 con 0x7fd0480ffd60 2024-03-21T10:38:23.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.192+0000 7fd046a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0481026d0 0x7fd04819e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.192+0000 7fd047400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0480ffd60 0x7fd04819ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.192+0000 7fd046a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0481026d0 0x7fd04819e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46244/0 (socket says 172.21.15.31:46244) 2024-03-21T10:38:23.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.192+0000 7fd046a00640 1 -- 172.21.15.31:0/2308763301 learned_addr learned my addr 172.21.15.31:0/2308763301 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:23.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.192+0000 7fd047e00640 1 --2- 172.21.15.31:0/2308763301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd048105080 0x7fd0481a5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.192+0000 7fd047400640 1 -- 172.21.15.31:0/2308763301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0481026d0 msgr2=0x7fd04819e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:23.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.192+0000 7fd047400640 1 --2- 172.21.15.31:0/2308763301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0481026d0 0x7fd04819e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.192+0000 7fd047400640 1 -- 172.21.15.31:0/2308763301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd048105080 msgr2=0x7fd0481a5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:23.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.192+0000 7fd047400640 1 --2- 172.21.15.31:0/2308763301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd048105080 0x7fd0481a5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.192+0000 7fd047400640 1 -- 172.21.15.31:0/2308763301 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0481a59c0 con 0x7fd0480ffd60 2024-03-21T10:38:23.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.192+0000 7fd046a00640 1 --2- 172.21.15.31:0/2308763301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0481026d0 0x7fd04819e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:23.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.192+0000 7fd047e00640 1 --2- 172.21.15.31:0/2308763301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd048105080 0x7fd0481a5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:23.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.193+0000 7fd047400640 1 --2- 172.21.15.31:0/2308763301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0480ffd60 0x7fd04819ddb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd038009fd0 tx=0x7fd03800ec20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:23.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.193+0000 7fd03fe00640 1 -- 172.21.15.31:0/2308763301 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03800ccc0 con 0x7fd0480ffd60 2024-03-21T10:38:23.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.193+0000 7fd03fe00640 1 -- 172.21.15.31:0/2308763301 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd03800ee70 con 0x7fd0480ffd60 2024-03-21T10:38:23.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.193+0000 7fd04ce00640 1 -- 172.21.15.31:0/2308763301 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0481a5cb0 con 0x7fd0480ffd60 2024-03-21T10:38:23.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.193+0000 7fd03fe00640 1 -- 172.21.15.31:0/2308763301 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd038004a00 con 0x7fd0480ffd60 2024-03-21T10:38:23.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.193+0000 7fd04ce00640 1 -- 172.21.15.31:0/2308763301 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0481a61b0 con 0x7fd0480ffd60 2024-03-21T10:38:23.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.197+0000 7fd051214740 1 Processor -- start 2024-03-21T10:38:23.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.197+0000 7fd051214740 1 -- start start 2024-03-21T10:38:23.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.197+0000 7fd03fe00640 1 -- 172.21.15.31:0/2308763301 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7fd038004ba0 con 0x7fd0480ffd60 2024-03-21T10:38:23.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.197+0000 7fd051214740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56145dd1f2b0 0x56145dd1f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.197+0000 7fd051214740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56145dd1fca0 0x56145dd247a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.197+0000 7fd051214740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56145dd24e10 0x56145dd272a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.197+0000 7fd051214740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56145dc84eb0 con 0x56145dd1fca0 2024-03-21T10:38:23.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.197+0000 7fd051214740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56145dc2da60 con 0x56145dd1f2b0 2024-03-21T10:38:23.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.197+0000 7fd051214740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56145dc80f50 con 0x56145dd24e10 2024-03-21T10:38:23.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.198+0000 7fd047e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56145dd1f2b0 0x56145dd1f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.198+0000 7fd047e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56145dd1f2b0 0x56145dd1f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46254/0 (socket says 172.21.15.31:46254) 2024-03-21T10:38:23.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.198+0000 7fd047e00640 1 -- 172.21.15.31:0/3808985268 learned_addr learned my addr 172.21.15.31:0/3808985268 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:23.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.198+0000 7fd047400640 1 --2- 172.21.15.31:0/3808985268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56145dd1fca0 0x56145dd247a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.198+0000 7fd046a00640 1 --2- 172.21.15.31:0/3808985268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56145dd24e10 0x56145dd272a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.198+0000 7fd047e00640 1 -- 172.21.15.31:0/3808985268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56145dd24e10 msgr2=0x56145dd272a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:23.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.198+0000 7fd047e00640 1 --2- 172.21.15.31:0/3808985268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56145dd24e10 0x56145dd272a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.198+0000 7fd047e00640 1 -- 172.21.15.31:0/3808985268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56145dd1fca0 msgr2=0x56145dd247a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.199+0000 7fd047e00640 1 --2- 172.21.15.31:0/3808985268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56145dd1fca0 0x56145dd247a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.199+0000 7fd047e00640 1 -- 172.21.15.31:0/3808985268 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56145da617e0 con 0x56145dd1f2b0 2024-03-21T10:38:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.199+0000 7fd047400640 1 --2- 172.21.15.31:0/3808985268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56145dd1fca0 0x56145dd247a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.199+0000 7fd047e00640 1 --2- 172.21.15.31:0/3808985268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56145dd1f2b0 0x56145dd1f6d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd040034fd0 tx=0x7fd040009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d2fa6d05223495c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.199+0000 7fd03fe00640 1 --2- 172.21.15.31:0/2308763301 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd02c0a2fa0 0x7fd02c0a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.199+0000 7fd03cc00640 1 -- 172.21.15.31:0/3808985268 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd040043070 con 0x56145dd1f2b0 2024-03-21T10:38:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.199+0000 7fd03cc00640 1 -- 172.21.15.31:0/3808985268 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd040033bc0 con 0x56145dd1f2b0 2024-03-21T10:38:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.199+0000 7fd03cc00640 1 -- 172.21.15.31:0/3808985268 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04002cb80 con 0x56145dd1f2b0 2024-03-21T10:38:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.200+0000 7fd03fe00640 1 -- 172.21.15.31:0/2308763301 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fd0380cf770 con 0x7fd0480ffd60 2024-03-21T10:38:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.200+0000 7fd047400640 1 --2- 172.21.15.31:0/2308763301 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd02c0a2fa0 0x7fd02c0a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.200+0000 7fd051214740 1 -- 172.21.15.31:0/3808985268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56145dd1f2b0 msgr2=0x56145dd1f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.200+0000 7fd051214740 1 --2- 172.21.15.31:0/3808985268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56145dd1f2b0 0x56145dd1f6d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd040034fd0 tx=0x7fd040009840 comp rx=0 tx=0).stop 2024-03-21T10:38:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.201+0000 7fd051214740 1 -- 172.21.15.31:0/3808985268 shutdown_connections 2024-03-21T10:38:23.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.201+0000 7fd051214740 1 --2- 172.21.15.31:0/3808985268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56145dd24e10 0x56145dd272a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.201+0000 7fd051214740 1 --2- 172.21.15.31:0/3808985268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56145dd1fca0 0x56145dd247a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.201+0000 7fd051214740 1 --2- 172.21.15.31:0/3808985268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56145dd1f2b0 0x56145dd1f6d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.201+0000 7fd051214740 1 -- 172.21.15.31:0/3808985268 >> 172.21.15.31:0/3808985268 conn(0x56145dd1a820 msgr2=0x56145dd1ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:23.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.201+0000 7fd047400640 1 --2- 172.21.15.31:0/2308763301 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd02c0a2fa0 0x7fd02c0a5460 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd0380cd280 tx=0x7fd0380d5550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:23.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.201+0000 7fd051214740 1 -- 172.21.15.31:0/3808985268 shutdown_connections 2024-03-21T10:38:23.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.201+0000 7fd051214740 1 -- 172.21.15.31:0/3808985268 wait complete. 2024-03-21T10:38:23.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.202+0000 7fd051214740 1 -- start start 2024-03-21T10:38:23.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.202+0000 7fd051214740 1 Processor -- start 2024-03-21T10:38:23.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.203+0000 7fd051214740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56145dd1f2b0 0x56145dd29460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.203+0000 7fd051214740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56145dd1fca0 0x56145dd299a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.203+0000 7fd051214740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56145dd24e10 0x56145dd309e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.203+0000 7fd051214740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56145dd27d60 con 0x56145dd1fca0 2024-03-21T10:38:23.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.203+0000 7fd051214740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56145dc40eb0 con 0x56145dd24e10 2024-03-21T10:38:23.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.203+0000 7fd051214740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56145dd27ee0 con 0x56145dd1f2b0 2024-03-21T10:38:23.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.203+0000 7fd051214740 10 client.0 _ll_register_callbacks cb 0x56145dc35150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:38:23.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.203+0000 7fd047e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56145dd1f2b0 0x56145dd29460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.203+0000 7fd047e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56145dd1f2b0 0x56145dd29460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53688/0 (socket says 172.21.15.31:53688) 2024-03-21T10:38:23.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.203+0000 7fd047e00640 1 -- 172.21.15.31:0/1279924141 learned_addr learned my addr 172.21.15.31:0/1279924141 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:23.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.203+0000 7fd047e00640 1 --2- 172.21.15.31:0/1279924141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56145dd24e10 0x56145dd309e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.203+0000 7fd046a00640 1 --2- 172.21.15.31:0/1279924141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56145dd1fca0 0x56145dd299a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.203+0000 7fd047e00640 1 -- 172.21.15.31:0/1279924141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56145dd24e10 msgr2=0x56145dd309e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:23.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.203+0000 7fd047e00640 1 --2- 172.21.15.31:0/1279924141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56145dd24e10 0x56145dd309e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.203+0000 7fd047e00640 1 -- 172.21.15.31:0/1279924141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56145dd1fca0 msgr2=0x56145dd299a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:23.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.203+0000 7fd047e00640 1 --2- 172.21.15.31:0/1279924141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56145dd1fca0 0x56145dd299a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.203+0000 7fd047e00640 1 -- 172.21.15.31:0/1279924141 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56145dac1f70 con 0x56145dd1f2b0 2024-03-21T10:38:23.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.203+0000 7fd046a00640 1 --2- 172.21.15.31:0/1279924141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56145dd1fca0 0x56145dd299a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:23.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.204+0000 7fd047e00640 1 --2- 172.21.15.31:0/1279924141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56145dd1f2b0 0x56145dd29460 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd040009be0 tx=0x7fd040009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=854f7f00b6fa4bee server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:23.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.204+0000 7fd031600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:38:23.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.204+0000 7fd031600640 1 -- 172.21.15.31:0/1279924141 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd040043070 con 0x56145dd1f2b0 2024-03-21T10:38:23.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.204+0000 7fd031600640 1 -- 172.21.15.31:0/1279924141 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fd040004b40 con 0x56145dd1f2b0 2024-03-21T10:38:23.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.204+0000 7fd051214740 1 -- 172.21.15.31:0/1279924141 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd040033bc0 con 0x56145dd1f2b0 2024-03-21T10:38:23.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.204+0000 7fd031600640 1 -- 172.21.15.31:0/1279924141 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd04005c050 con 0x56145dd1f2b0 2024-03-21T10:38:23.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.205+0000 7fd051214740 10 client.25525 fetch_fsmap learned FSMap version 24 2024-03-21T10:38:23.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.205+0000 7fd051214740 1 -- 172.21.15.31:0/1279924141 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x56145d959170 con 0x56145dd1f2b0 2024-03-21T10:38:23.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.205+0000 7fd031600640 1 -- 172.21.15.31:0/1279924141 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd040033bc0 con 0x56145dd1f2b0 2024-03-21T10:38:23.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.205+0000 7fd031600640 1 -- 172.21.15.31:0/1279924141 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd040054020 con 0x56145dd1f2b0 2024-03-21T10:38:23.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.205+0000 7fd051214740 10 client.25525 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:38:23.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.205+0000 7fd051214740 10 client.25525 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:38:23.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.205+0000 7fd051214740 20 client.25525 populate_metadata read hostname 'smithi031' 2024-03-21T10:38:23.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.205+0000 7fd051214740 1 --2- 172.21.15.31:0/1279924141 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56145dd34ed0 0x56145dd372c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.205+0000 7fd051214740 4 client.25525 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:38:23.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.205+0000 7fd051214740 1 -- 172.21.15.31:0/1279924141 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56145dc923f0 con 0x56145dd34ed0 2024-03-21T10:38:23.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.206+0000 7fd046a00640 1 --2- 172.21.15.31:0/1279924141 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56145dd34ed0 0x56145dd372c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.206+0000 7fd046a00640 1 --2- 172.21.15.31:0/1279924141 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56145dd34ed0 0x56145dd372c0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8df0545382a177c6 server_cookie=e0f7392189b66114 in_seq=0 out_seq=0 2024-03-21T10:38:23.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.206+0000 7fd031600640 10 client.25525 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:23.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.207+0000 7fd031600640 1 -- 172.21.15.31:0/1279924141 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56145dc923f0 con 0x56145dd34ed0 2024-03-21T10:38:23.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.207+0000 7fd031600640 10 client.25525 handle_command_reply: tid=0 2024-03-21T10:38:23.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.207+0000 7fd051214740 1 client.25525 shutdown 2024-03-21T10:38:23.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.207+0000 7fd051214740 1 -- 172.21.15.31:0/1279924141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56145dd1f2b0 msgr2=0x56145dd29460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:23.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.207+0000 7fd051214740 1 --2- 172.21.15.31:0/1279924141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56145dd1f2b0 0x56145dd29460 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd040009be0 tx=0x7fd040009500 comp rx=0 tx=0).stop 2024-03-21T10:38:23.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.208+0000 7fd051214740 1 -- 172.21.15.31:0/1279924141 shutdown_connections 2024-03-21T10:38:23.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.208+0000 7fd051214740 1 --2- 172.21.15.31:0/1279924141 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56145dd34ed0 0x56145dd372c0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.208+0000 7fd051214740 1 --2- 172.21.15.31:0/1279924141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56145dd24e10 0x56145dd309e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.208+0000 7fd051214740 1 --2- 172.21.15.31:0/1279924141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56145dd1fca0 0x56145dd299a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.208+0000 7fd051214740 1 --2- 172.21.15.31:0/1279924141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56145dd1f2b0 0x56145dd29460 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.208+0000 7fd031600640 0 client.25525 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:23.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.208+0000 7fd051214740 1 -- 172.21.15.31:0/1279924141 >> 172.21.15.31:0/1279924141 conn(0x56145dd1a820 msgr2=0x56145dd19a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:23.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.208+0000 7fd051214740 1 -- 172.21.15.31:0/1279924141 shutdown_connections 2024-03-21T10:38:23.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.208+0000 7fd051214740 1 -- 172.21.15.31:0/1279924141 wait complete. 2024-03-21T10:38:23.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.208+0000 7fd051214740 20 client.25525 trim_cache size 0 max 16384 2024-03-21T10:38:23.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.226+0000 7fd051214740 1 Processor -- start 2024-03-21T10:38:23.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.226+0000 7fd051214740 1 -- start start 2024-03-21T10:38:23.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.227+0000 7fd051214740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56145dd1f2b0 0x56145dd3ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.227+0000 7fd051214740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56145dd1fca0 0x56145dd3bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.227+0000 7fd051214740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56145dd24e10 0x56145dd3c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.227+0000 7fd051214740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0481b2a00 con 0x56145dd24e10 2024-03-21T10:38:23.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.227+0000 7fd051214740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0481b0500 con 0x56145dd1fca0 2024-03-21T10:38:23.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.227+0000 7fd051214740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0481ae5a0 con 0x56145dd1f2b0 2024-03-21T10:38:23.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.227+0000 7fd046a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56145dd1fca0 0x56145dd3bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.227+0000 7fd047e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56145dd1f2b0 0x56145dd3ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.227+0000 7fd046a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56145dd1fca0 0x56145dd3bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46278/0 (socket says 172.21.15.31:46278) 2024-03-21T10:38:23.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.227+0000 7fd046a00640 1 -- 172.21.15.31:0/836343770 learned_addr learned my addr 172.21.15.31:0/836343770 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:23.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.227+0000 7fd047e00640 1 --2- 172.21.15.31:0/836343770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56145dd24e10 0x56145dd3c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.227+0000 7fd047e00640 1 -- 172.21.15.31:0/836343770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56145dd1fca0 msgr2=0x56145dd3bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:23.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.227+0000 7fd047e00640 1 --2- 172.21.15.31:0/836343770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56145dd1fca0 0x56145dd3bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.228+0000 7fd047e00640 1 -- 172.21.15.31:0/836343770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56145dd24e10 msgr2=0x56145dd3c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:23.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.228+0000 7fd047e00640 1 --2- 172.21.15.31:0/836343770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56145dd24e10 0x56145dd3c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.228+0000 7fd047e00640 1 -- 172.21.15.31:0/836343770 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56145dd37630 con 0x56145dd1f2b0 2024-03-21T10:38:23.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.228+0000 7fd046a00640 1 --2- 172.21.15.31:0/836343770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56145dd1fca0 0x56145dd3bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:23.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.228+0000 7fd047e00640 1 --2- 172.21.15.31:0/836343770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56145dd1f2b0 0x56145dd3ba90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd040033ff0 tx=0x7fd040034f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df8d2a77db8d247c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:23.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.228+0000 7fd033e00640 1 -- 172.21.15.31:0/836343770 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd040043040 con 0x56145dd1f2b0 2024-03-21T10:38:23.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.228+0000 7fd033e00640 1 -- 172.21.15.31:0/836343770 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd040009800 con 0x56145dd1f2b0 2024-03-21T10:38:23.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.228+0000 7fd033e00640 1 -- 172.21.15.31:0/836343770 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04002c890 con 0x56145dd1f2b0 2024-03-21T10:38:23.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.229+0000 7fd051214740 1 -- 172.21.15.31:0/836343770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56145dd1f2b0 msgr2=0x56145dd3ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:23.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.229+0000 7fd051214740 1 --2- 172.21.15.31:0/836343770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56145dd1f2b0 0x56145dd3ba90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd040033ff0 tx=0x7fd040034f90 comp rx=0 tx=0).stop 2024-03-21T10:38:23.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.229+0000 7fd051214740 1 -- 172.21.15.31:0/836343770 shutdown_connections 2024-03-21T10:38:23.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.229+0000 7fd051214740 1 --2- 172.21.15.31:0/836343770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56145dd24e10 0x56145dd3c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.229+0000 7fd051214740 1 --2- 172.21.15.31:0/836343770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56145dd1fca0 0x56145dd3bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.229+0000 7fd051214740 1 --2- 172.21.15.31:0/836343770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56145dd1f2b0 0x56145dd3ba90 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.229+0000 7fd051214740 1 -- 172.21.15.31:0/836343770 >> 172.21.15.31:0/836343770 conn(0x56145dd1a820 msgr2=0x56145dd1e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:23.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.229+0000 7fd051214740 1 -- 172.21.15.31:0/836343770 shutdown_connections 2024-03-21T10:38:23.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.229+0000 7fd051214740 1 -- 172.21.15.31:0/836343770 wait complete. 2024-03-21T10:38:23.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.230+0000 7fd051214740 1 -- start start 2024-03-21T10:38:23.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.230+0000 7fd051214740 1 Processor -- start 2024-03-21T10:38:23.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.231+0000 7fd051214740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56145dd1f2b0 0x56145dd2fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.231+0000 7fd051214740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56145dd1fca0 0x56145dd300c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.231+0000 7fd051214740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56145dd24e10 0x56145dd2e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.231+0000 7fd051214740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0481aeba0 con 0x56145dd1f2b0 2024-03-21T10:38:23.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.231+0000 7fd051214740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0481afea0 con 0x56145dd24e10 2024-03-21T10:38:23.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.231+0000 7fd051214740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0481af1e0 con 0x56145dd1fca0 2024-03-21T10:38:23.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.231+0000 7fd051214740 10 client.0 _ll_register_callbacks cb 0x7fd0380d5420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:38:23.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.231+0000 7fd047e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56145dd1f2b0 0x56145dd2fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.231+0000 7fd046a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56145dd1fca0 0x56145dd300c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.231+0000 7fd046a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56145dd1fca0 0x56145dd300c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53712/0 (socket says 172.21.15.31:53712) 2024-03-21T10:38:23.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.231+0000 7fd047e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56145dd1f2b0 0x56145dd2fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33662/0 (socket says 172.21.15.31:33662) 2024-03-21T10:38:23.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.231+0000 7fd046a00640 1 -- 172.21.15.31:0/2760478824 learned_addr learned my addr 172.21.15.31:0/2760478824 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:23.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.231+0000 7fd047e00640 1 --2- 172.21.15.31:0/2760478824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56145dd24e10 0x56145dd2e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.231+0000 7fd046a00640 1 -- 172.21.15.31:0/2760478824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56145dd24e10 msgr2=0x56145dd2e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:23.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.231+0000 7fd046a00640 1 --2- 172.21.15.31:0/2760478824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56145dd24e10 0x56145dd2e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.232+0000 7fd046a00640 1 -- 172.21.15.31:0/2760478824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56145dd1f2b0 msgr2=0x56145dd2fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:23.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.232+0000 7fd046a00640 1 --2- 172.21.15.31:0/2760478824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56145dd1f2b0 0x56145dd2fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.232+0000 7fd046a00640 1 -- 172.21.15.31:0/2760478824 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56145dd2ea80 con 0x56145dd1fca0 2024-03-21T10:38:23.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.232+0000 7fd047e00640 1 --2- 172.21.15.31:0/2760478824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56145dd24e10 0x56145dd2e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:23.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.232+0000 7fd046a00640 1 --2- 172.21.15.31:0/2760478824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56145dd1fca0 0x56145dd300c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd034009620 tx=0x7fd034010d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e622f15a67b562f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:23.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.232+0000 7fd031600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:38:23.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.232+0000 7fd031600640 1 -- 172.21.15.31:0/2760478824 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0340200f0 con 0x56145dd1fca0 2024-03-21T10:38:23.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.232+0000 7fd031600640 1 -- 172.21.15.31:0/2760478824 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fd034020290 con 0x56145dd1fca0 2024-03-21T10:38:23.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.233+0000 7fd051214740 1 -- 172.21.15.31:0/2760478824 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd040009800 con 0x56145dd1fca0 2024-03-21T10:38:23.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.233+0000 7fd031600640 1 -- 172.21.15.31:0/2760478824 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd034004030 con 0x56145dd1fca0 2024-03-21T10:38:23.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.233+0000 7fd051214740 10 client.25531 fetch_fsmap learned FSMap version 24 2024-03-21T10:38:23.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.233+0000 7fd051214740 1 -- 172.21.15.31:0/2760478824 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x56145dd91280 con 0x56145dd1fca0 2024-03-21T10:38:23.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.233+0000 7fd031600640 1 -- 172.21.15.31:0/2760478824 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd040009800 con 0x56145dd1fca0 2024-03-21T10:38:23.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.233+0000 7fd031600640 1 -- 172.21.15.31:0/2760478824 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd034029400 con 0x56145dd1fca0 2024-03-21T10:38:23.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.233+0000 7fd051214740 10 client.25531 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:38:23.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.233+0000 7fd051214740 10 client.25531 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:38:23.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.233+0000 7fd051214740 20 client.25531 populate_metadata read hostname 'smithi031' 2024-03-21T10:38:23.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.233+0000 7fd051214740 1 --2- 172.21.15.31:0/2760478824 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56145dd91c40 0x56145dd94060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.233+0000 7fd051214740 4 client.25531 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-21T10:38:23.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.233+0000 7fd051214740 1 -- 172.21.15.31:0/2760478824 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56145dc8c740 con 0x56145dd91c40 2024-03-21T10:38:23.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.234+0000 7fd047e00640 1 --2- 172.21.15.31:0/2760478824 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56145dd91c40 0x56145dd94060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.234+0000 7fd047e00640 1 --2- 172.21.15.31:0/2760478824 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56145dd91c40 0x56145dd94060 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56276d79946c3097 server_cookie=b0d2e22586b151e8 in_seq=0 out_seq=0 2024-03-21T10:38:23.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.234+0000 7fd031600640 10 client.25531 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:23.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.235+0000 7fd031600640 1 -- 172.21.15.31:0/2760478824 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56145dc8c740 con 0x56145dd91c40 2024-03-21T10:38:23.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.235+0000 7fd031600640 10 client.25531 handle_command_reply: tid=0 2024-03-21T10:38:23.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.235+0000 7fd051214740 1 client.25531 shutdown 2024-03-21T10:38:23.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.236+0000 7fd051214740 1 -- 172.21.15.31:0/2760478824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56145dd1fca0 msgr2=0x56145dd300c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:23.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.236+0000 7fd051214740 1 --2- 172.21.15.31:0/2760478824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56145dd1fca0 0x56145dd300c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd034009620 tx=0x7fd034010d90 comp rx=0 tx=0).stop 2024-03-21T10:38:23.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.236+0000 7fd051214740 1 -- 172.21.15.31:0/2760478824 shutdown_connections 2024-03-21T10:38:23.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.236+0000 7fd051214740 1 --2- 172.21.15.31:0/2760478824 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56145dd91c40 0x56145dd94060 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.236+0000 7fd051214740 1 --2- 172.21.15.31:0/2760478824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56145dd24e10 0x56145dd2e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.236+0000 7fd051214740 1 --2- 172.21.15.31:0/2760478824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56145dd1fca0 0x56145dd300c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.236+0000 7fd051214740 1 --2- 172.21.15.31:0/2760478824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56145dd1f2b0 0x56145dd2fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.236+0000 7fd031600640 0 client.25531 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:23.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.236+0000 7fd051214740 1 -- 172.21.15.31:0/2760478824 >> 172.21.15.31:0/2760478824 conn(0x56145dd1a820 msgr2=0x56145dd20740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:23.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.236+0000 7fd051214740 1 -- 172.21.15.31:0/2760478824 shutdown_connections 2024-03-21T10:38:23.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.236+0000 7fd051214740 1 -- 172.21.15.31:0/2760478824 wait complete. 2024-03-21T10:38:23.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.236+0000 7fd051214740 20 client.25531 trim_cache size 0 max 16384 2024-03-21T10:38:23.235 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:38:23.236 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-03-21T10:38:23.236 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "e6245416-4a98-4dd7-80f2-04e460d62b96", 2024-03-21T10:38:23.236 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-03-21T10:38:23.236 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:38:23.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.237+0000 7fd04ce00640 1 -- 172.21.15.31:0/2308763301 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd02c0a2fa0 msgr2=0x7fd02c0a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:23.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.237+0000 7fd04ce00640 1 --2- 172.21.15.31:0/2308763301 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd02c0a2fa0 0x7fd02c0a5460 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd0380cd280 tx=0x7fd0380d5550 comp rx=0 tx=0).stop 2024-03-21T10:38:23.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.237+0000 7fd04ce00640 1 -- 172.21.15.31:0/2308763301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0480ffd60 msgr2=0x7fd04819ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:23.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.237+0000 7fd04ce00640 1 --2- 172.21.15.31:0/2308763301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0480ffd60 0x7fd04819ddb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd038009fd0 tx=0x7fd03800ec20 comp rx=0 tx=0).stop 2024-03-21T10:38:23.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.237+0000 7fd04ce00640 1 -- 172.21.15.31:0/2308763301 shutdown_connections 2024-03-21T10:38:23.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.237+0000 7fd04ce00640 1 --2- 172.21.15.31:0/2308763301 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fd02c0a2fa0 0x7fd02c0a5460 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.237+0000 7fd04ce00640 1 --2- 172.21.15.31:0/2308763301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd048105080 0x7fd0481a5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.237+0000 7fd04ce00640 1 --2- 172.21.15.31:0/2308763301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0481026d0 0x7fd04819e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.237+0000 7fd04ce00640 1 --2- 172.21.15.31:0/2308763301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0480ffd60 0x7fd04819ddb0 secure :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd038009fd0 tx=0x7fd03800ec20 comp rx=0 tx=0).stop 2024-03-21T10:38:23.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.237+0000 7fd04ce00640 1 -- 172.21.15.31:0/2308763301 >> 172.21.15.31:0/2308763301 conn(0x7fd0480fb980 msgr2=0x7fd048100a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:23.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.238+0000 7fd04ce00640 1 -- 172.21.15.31:0/2308763301 shutdown_connections 2024-03-21T10:38:23.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.238+0000 7fd04ce00640 1 -- 172.21.15.31:0/2308763301 wait complete. 2024-03-21T10:38:23.253 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:38:23.253 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:38:23.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.478+0000 7ff78cc00640 1 -- 172.21.15.31:0/1347166313 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7ff7880622a0 con 0x7ff788105270 2024-03-21T10:38:23.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.517+0000 7f0ef3e00640 1 -- 172.21.15.31:0/2793311958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ef4108cf0 msgr2=0x7f0ef4109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:23.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.517+0000 7f0ef3e00640 1 --2- 172.21.15.31:0/2793311958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ef4108cf0 0x7f0ef4109170 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0ee0009900 tx=0x7f0ee002bc50 comp rx=0 tx=0).stop 2024-03-21T10:38:23.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.517+0000 7f0ef3e00640 1 -- 172.21.15.31:0/2793311958 shutdown_connections 2024-03-21T10:38:23.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.517+0000 7f0ef3e00640 1 --2- 172.21.15.31:0/2793311958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ef41096b0 0x7f0ef410ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.517+0000 7f0ef3e00640 1 --2- 172.21.15.31:0/2793311958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ef4108cf0 0x7f0ef4109170 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.517+0000 7f0ef3e00640 1 --2- 172.21.15.31:0/2793311958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ef4107b00 0x7f0ef4107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.517+0000 7f0ef3e00640 1 -- 172.21.15.31:0/2793311958 >> 172.21.15.31:0/2793311958 conn(0x7f0ef4076a90 msgr2=0x7f0ef4078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:23.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.517+0000 7f0ef3e00640 1 -- 172.21.15.31:0/2793311958 shutdown_connections 2024-03-21T10:38:23.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.517+0000 7f0ef3e00640 1 -- 172.21.15.31:0/2793311958 wait complete. 2024-03-21T10:38:23.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.519+0000 7f0ef3e00640 1 Processor -- start 2024-03-21T10:38:23.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.519+0000 7f0ef3e00640 1 -- start start 2024-03-21T10:38:23.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.520+0000 7f0ef3e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ef4107b00 0x7f0ef419ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.520+0000 7f0ef3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ef4108cf0 0x7f0ef419e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.520+0000 7f0ef3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ef41096b0 0x7f0ef41a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.520+0000 7f0ef3e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef41137b0 con 0x7f0ef4108cf0 2024-03-21T10:38:23.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.520+0000 7f0ef3e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef4113630 con 0x7f0ef41096b0 2024-03-21T10:38:23.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.520+0000 7f0ef3e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef4113930 con 0x7f0ef4107b00 2024-03-21T10:38:23.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.520+0000 7f0ef2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ef4108cf0 0x7f0ef419e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.520+0000 7f0ef3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ef41096b0 0x7f0ef41a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.520+0000 7f0ef2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ef4107b00 0x7f0ef419ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.520+0000 7f0ef3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ef41096b0 0x7f0ef41a5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46316/0 (socket says 172.21.15.31:46316) 2024-03-21T10:38:23.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.520+0000 7f0ef3400640 1 -- 172.21.15.31:0/3134844876 learned_addr learned my addr 172.21.15.31:0/3134844876 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:23.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.520+0000 7f0ef3400640 1 -- 172.21.15.31:0/3134844876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ef4107b00 msgr2=0x7f0ef419ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:23.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.520+0000 7f0ef3400640 1 --2- 172.21.15.31:0/3134844876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ef4107b00 0x7f0ef419ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.520+0000 7f0ef3400640 1 -- 172.21.15.31:0/3134844876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ef4108cf0 msgr2=0x7f0ef419e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:23.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.520+0000 7f0ef3400640 1 --2- 172.21.15.31:0/3134844876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ef4108cf0 0x7f0ef419e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.520+0000 7f0ef3400640 1 -- 172.21.15.31:0/3134844876 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef41a59f0 con 0x7f0ef41096b0 2024-03-21T10:38:23.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.520+0000 7f0ef2a00640 1 --2- 172.21.15.31:0/3134844876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ef4107b00 0x7f0ef419ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:23.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.521+0000 7f0ef3400640 1 --2- 172.21.15.31:0/3134844876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ef41096b0 0x7f0ef41a5380 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0eec00eb10 tx=0x7f0eec00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:23.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.521+0000 7f0eeb400640 1 -- 172.21.15.31:0/3134844876 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eec004b20 con 0x7f0ef41096b0 2024-03-21T10:38:23.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.521+0000 7f0eeb400640 1 -- 172.21.15.31:0/3134844876 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0eec004cc0 con 0x7f0ef41096b0 2024-03-21T10:38:23.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.521+0000 7f0ef3e00640 1 -- 172.21.15.31:0/3134844876 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ef41a5ce0 con 0x7f0ef41096b0 2024-03-21T10:38:23.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.521+0000 7f0eeb400640 1 -- 172.21.15.31:0/3134844876 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eec010760 con 0x7f0ef41096b0 2024-03-21T10:38:23.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.521+0000 7f0ef3e00640 1 -- 172.21.15.31:0/3134844876 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ef41a61b0 con 0x7f0ef41096b0 2024-03-21T10:38:23.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.524+0000 7f0efc7c3740 1 Processor -- start 2024-03-21T10:38:23.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.524+0000 7f0eeb400640 1 -- 172.21.15.31:0/3134844876 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 137854+0+0 (secure 0 0 0) 0x7f0eec005020 con 0x7f0ef41096b0 2024-03-21T10:38:23.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.525+0000 7f0efc7c3740 1 -- start start 2024-03-21T10:38:23.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.525+0000 7f0efc7c3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c4e28a050 0x560c4e28a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.525+0000 7f0efc7c3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c4e28aa40 0x560c4e28f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.525+0000 7f0efc7c3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c4e28fbb0 0x560c4e292040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.525+0000 7f0efc7c3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c4e1ef250 con 0x560c4e28a050 2024-03-21T10:38:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.525+0000 7f0efc7c3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c4e197a30 con 0x560c4e28fbb0 2024-03-21T10:38:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.525+0000 7f0efc7c3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c4e1eb2f0 con 0x560c4e28aa40 2024-03-21T10:38:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.525+0000 7f0ef2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c4e28aa40 0x560c4e28f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.525+0000 7f0ef2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c4e28a050 0x560c4e28a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.525+0000 7f0ef2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c4e28aa40 0x560c4e28f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53742/0 (socket says 172.21.15.31:53742) 2024-03-21T10:38:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.525+0000 7f0ef2a00640 1 -- 172.21.15.31:0/2661391424 learned_addr learned my addr 172.21.15.31:0/2661391424 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.525+0000 7f0ef2000640 1 --2- 172.21.15.31:0/2661391424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c4e28fbb0 0x560c4e292040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.526+0000 7f0ef2a00640 1 -- 172.21.15.31:0/2661391424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c4e28fbb0 msgr2=0x560c4e292040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.526+0000 7f0ef2a00640 1 --2- 172.21.15.31:0/2661391424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c4e28fbb0 0x560c4e292040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.526+0000 7f0ef2a00640 1 -- 172.21.15.31:0/2661391424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c4e28a050 msgr2=0x560c4e28a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.526+0000 7f0ef2a00640 1 --2- 172.21.15.31:0/2661391424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c4e28a050 0x560c4e28a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.526+0000 7f0ef2a00640 1 -- 172.21.15.31:0/2661391424 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c4e182690 con 0x560c4e28aa40 2024-03-21T10:38:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.526+0000 7f0ef2000640 1 --2- 172.21.15.31:0/2661391424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c4e28a050 0x560c4e28a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.526+0000 7f0ef2a00640 1 --2- 172.21.15.31:0/2661391424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c4e28aa40 0x560c4e28f540 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4010550 tx=0x7f0ee4010a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b8a2f012c00bf6e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:23.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.526+0000 7f0edfe00640 1 -- 172.21.15.31:0/2661391424 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4011760 con 0x560c4e28aa40 2024-03-21T10:38:23.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.526+0000 7f0edfe00640 1 -- 172.21.15.31:0/2661391424 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ee4011de0 con 0x560c4e28aa40 2024-03-21T10:38:23.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.526+0000 7f0edfe00640 1 -- 172.21.15.31:0/2661391424 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee400e430 con 0x560c4e28aa40 2024-03-21T10:38:23.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.526+0000 7f0eeb400640 1 --2- 172.21.15.31:0/3134844876 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0ed80a2f20 0x7f0ed80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.527+0000 7f0eeb400640 1 -- 172.21.15.31:0/3134844876 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f0eec0cfca0 con 0x7f0ef41096b0 2024-03-21T10:38:23.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.527+0000 7f0ef2000640 1 --2- 172.21.15.31:0/3134844876 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0ed80a2f20 0x7f0ed80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.527+0000 7f0efc7c3740 1 -- 172.21.15.31:0/2661391424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c4e28aa40 msgr2=0x560c4e28f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:23.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.527+0000 7f0efc7c3740 1 --2- 172.21.15.31:0/2661391424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c4e28aa40 0x560c4e28f540 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4010550 tx=0x7f0ee4010a10 comp rx=0 tx=0).stop 2024-03-21T10:38:23.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.527+0000 7f0efc7c3740 1 -- 172.21.15.31:0/2661391424 shutdown_connections 2024-03-21T10:38:23.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.527+0000 7f0efc7c3740 1 --2- 172.21.15.31:0/2661391424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c4e28fbb0 0x560c4e292040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.527+0000 7f0efc7c3740 1 --2- 172.21.15.31:0/2661391424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c4e28aa40 0x560c4e28f540 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.527+0000 7f0efc7c3740 1 --2- 172.21.15.31:0/2661391424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c4e28a050 0x560c4e28a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.527+0000 7f0efc7c3740 1 -- 172.21.15.31:0/2661391424 >> 172.21.15.31:0/2661391424 conn(0x560c4e252200 msgr2=0x560c4e2859c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.527+0000 7f0ef2000640 1 --2- 172.21.15.31:0/3134844876 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0ed80a2f20 0x7f0ed80a53e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0ee002c260 tx=0x7f0ee0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.527+0000 7f0efc7c3740 1 -- 172.21.15.31:0/2661391424 shutdown_connections 2024-03-21T10:38:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.528+0000 7f0efc7c3740 1 -- 172.21.15.31:0/2661391424 wait complete. 2024-03-21T10:38:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.528+0000 7f0efc7c3740 1 -- start start 2024-03-21T10:38:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.528+0000 7f0efc7c3740 1 Processor -- start 2024-03-21T10:38:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.529+0000 7f0efc7c3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c4e28a050 0x560c4e294020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.529+0000 7f0efc7c3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c4e28aa40 0x560c4e294560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.529+0000 7f0efc7c3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c4e28fbb0 0x560c4e29b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.529+0000 7f0efc7c3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c4e292b60 con 0x560c4e28aa40 2024-03-21T10:38:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.529+0000 7f0efc7c3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c4e1aae80 con 0x560c4e28a050 2024-03-21T10:38:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.529+0000 7f0efc7c3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c4e292ce0 con 0x560c4e28fbb0 2024-03-21T10:38:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.529+0000 7f0efc7c3740 10 client.0 _ll_register_callbacks cb 0x560c4e1bd130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:38:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.529+0000 7f0ef2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c4e28a050 0x560c4e294020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.529+0000 7f0ef3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c4e28fbb0 0x560c4e29b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.529+0000 7f0ef2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c4e28a050 0x560c4e294020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46336/0 (socket says 172.21.15.31:46336) 2024-03-21T10:38:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.529+0000 7f0ef2a00640 1 -- 172.21.15.31:0/229694302 learned_addr learned my addr 172.21.15.31:0/229694302 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.529+0000 7f0ef2000640 1 --2- 172.21.15.31:0/229694302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c4e28aa40 0x560c4e294560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.529+0000 7f0ef3400640 1 -- 172.21.15.31:0/229694302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c4e28a050 msgr2=0x560c4e294020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.529+0000 7f0ef3400640 1 --2- 172.21.15.31:0/229694302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c4e28a050 0x560c4e294020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.529+0000 7f0ef3400640 1 -- 172.21.15.31:0/229694302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c4e28aa40 msgr2=0x560c4e294560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.529+0000 7f0ef3400640 1 --2- 172.21.15.31:0/229694302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c4e28aa40 0x560c4e294560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.529+0000 7f0ef3400640 1 -- 172.21.15.31:0/229694302 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560c4dfcd7d0 con 0x560c4e28fbb0 2024-03-21T10:38:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.529+0000 7f0ef2a00640 1 --2- 172.21.15.31:0/229694302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c4e28a050 0x560c4e294020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.529+0000 7f0ef2000640 1 --2- 172.21.15.31:0/229694302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c4e28aa40 0x560c4e294560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.530+0000 7f0ef3400640 1 --2- 172.21.15.31:0/229694302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c4e28fbb0 0x560c4e29b5a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0cd3e0 tx=0x7f0eec09b970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7197a388ee861d5d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.530+0000 7f0edcc00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:38:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.530+0000 7f0edcc00640 1 -- 172.21.15.31:0/229694302 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eec0a0160 con 0x560c4e28fbb0 2024-03-21T10:38:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.530+0000 7f0edcc00640 1 -- 172.21.15.31:0/229694302 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f0eec0a4590 con 0x560c4e28fbb0 2024-03-21T10:38:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.530+0000 7f0efc7c3740 1 -- 172.21.15.31:0/229694302 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ee4011de0 con 0x560c4e28fbb0 2024-03-21T10:38:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.530+0000 7f0edcc00640 1 -- 172.21.15.31:0/229694302 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0eec0a4a20 con 0x560c4e28fbb0 2024-03-21T10:38:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.530+0000 7f0efc7c3740 10 client.25543 fetch_fsmap learned FSMap version 24 2024-03-21T10:38:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.530+0000 7f0efc7c3740 1 -- 172.21.15.31:0/229694302 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x560c4df033f0 con 0x560c4e28fbb0 2024-03-21T10:38:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.530+0000 7f0edcc00640 1 -- 172.21.15.31:0/229694302 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ee4011de0 con 0x560c4e28fbb0 2024-03-21T10:38:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.530+0000 7f0edcc00640 1 -- 172.21.15.31:0/229694302 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f0eec005310 con 0x560c4e28fbb0 2024-03-21T10:38:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.531+0000 7f0efc7c3740 10 client.25543 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:38:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.531+0000 7f0efc7c3740 10 client.25543 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:38:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.531+0000 7f0efc7c3740 20 client.25543 populate_metadata read hostname 'smithi031' 2024-03-21T10:38:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.531+0000 7f0efc7c3740 1 --2- 172.21.15.31:0/229694302 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560c4e29fdc0 0x560c4e2a2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.531+0000 7f0efc7c3740 4 client.25543 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:38:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.531+0000 7f0efc7c3740 1 -- 172.21.15.31:0/229694302 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560c4e1fc790 con 0x560c4e29fdc0 2024-03-21T10:38:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.531+0000 7f0ef2a00640 1 --2- 172.21.15.31:0/229694302 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560c4e29fdc0 0x560c4e2a2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.531+0000 7f0ef2a00640 1 --2- 172.21.15.31:0/229694302 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560c4e29fdc0 0x560c4e2a2230 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=801dd6f14d9fc39f server_cookie=38e8b9f40a6e5a34 in_seq=0 out_seq=0 2024-03-21T10:38:23.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.531+0000 7f0edcc00640 10 client.25543 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.532+0000 7f0edcc00640 1 -- 172.21.15.31:0/229694302 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560c4e1fc790 con 0x560c4e29fdc0 2024-03-21T10:38:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.532+0000 7f0edcc00640 10 client.25543 handle_command_reply: tid=0 2024-03-21T10:38:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.532+0000 7f0efc7c3740 1 client.25543 shutdown 2024-03-21T10:38:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.532+0000 7f0efc7c3740 1 -- 172.21.15.31:0/229694302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c4e28fbb0 msgr2=0x560c4e29b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.532+0000 7f0efc7c3740 1 --2- 172.21.15.31:0/229694302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c4e28fbb0 0x560c4e29b5a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0cd3e0 tx=0x7f0eec09b970 comp rx=0 tx=0).stop 2024-03-21T10:38:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.533+0000 7f0efc7c3740 1 -- 172.21.15.31:0/229694302 shutdown_connections 2024-03-21T10:38:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.533+0000 7f0efc7c3740 1 --2- 172.21.15.31:0/229694302 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560c4e29fdc0 0x560c4e2a2230 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.533+0000 7f0efc7c3740 1 --2- 172.21.15.31:0/229694302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c4e28fbb0 0x560c4e29b5a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.533+0000 7f0efc7c3740 1 --2- 172.21.15.31:0/229694302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c4e28aa40 0x560c4e294560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.533+0000 7f0efc7c3740 1 --2- 172.21.15.31:0/229694302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c4e28a050 0x560c4e294020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.533+0000 7f0edcc00640 0 client.25543 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.533+0000 7f0efc7c3740 1 -- 172.21.15.31:0/229694302 >> 172.21.15.31:0/229694302 conn(0x560c4e252200 msgr2=0x560c4e284bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.533+0000 7f0efc7c3740 1 -- 172.21.15.31:0/229694302 shutdown_connections 2024-03-21T10:38:23.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.533+0000 7f0efc7c3740 1 -- 172.21.15.31:0/229694302 wait complete. 2024-03-21T10:38:23.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.533+0000 7f0efc7c3740 20 client.25543 trim_cache size 0 max 16384 2024-03-21T10:38:23.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.551+0000 7f0efc7c3740 1 Processor -- start 2024-03-21T10:38:23.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.551+0000 7f0efc7c3740 1 -- start start 2024-03-21T10:38:23.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.551+0000 7f0efc7c3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c4e28a050 0x560c4e280920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.551+0000 7f0efc7c3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c4e28aa40 0x560c4e29b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.551+0000 7f0efc7c3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c4e28fbb0 0x560c4e284330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.551+0000 7f0efc7c3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef41b2920 con 0x560c4e28fbb0 2024-03-21T10:38:23.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.551+0000 7f0efc7c3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef41b0420 con 0x560c4e28a050 2024-03-21T10:38:23.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.551+0000 7f0efc7c3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef41ae4c0 con 0x560c4e28aa40 2024-03-21T10:38:23.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.552+0000 7f0ef2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c4e28a050 0x560c4e280920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.552+0000 7f0ef2000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c4e28aa40 0x560c4e29b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.552+0000 7f0ef2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c4e28a050 0x560c4e280920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46346/0 (socket says 172.21.15.31:46346) 2024-03-21T10:38:23.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.552+0000 7f0ef2a00640 1 -- 172.21.15.31:0/625017935 learned_addr learned my addr 172.21.15.31:0/625017935 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:23.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.552+0000 7f0ef3400640 1 --2- 172.21.15.31:0/625017935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c4e28fbb0 0x560c4e284330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.552+0000 7f0ef2a00640 1 -- 172.21.15.31:0/625017935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c4e28aa40 msgr2=0x560c4e29b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:23.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.552+0000 7f0ef2a00640 1 --2- 172.21.15.31:0/625017935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c4e28aa40 0x560c4e29b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.552+0000 7f0ef2a00640 1 -- 172.21.15.31:0/625017935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c4e28fbb0 msgr2=0x560c4e284330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:23.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.552+0000 7f0ef2a00640 1 --2- 172.21.15.31:0/625017935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c4e28fbb0 0x560c4e284330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.552+0000 7f0ef2a00640 1 -- 172.21.15.31:0/625017935 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c4dec30f0 con 0x560c4e28a050 2024-03-21T10:38:23.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.552+0000 7f0ef3400640 1 --2- 172.21.15.31:0/625017935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c4e28fbb0 0x560c4e284330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:23.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.552+0000 7f0ef2a00640 1 --2- 172.21.15.31:0/625017935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c4e28a050 0x560c4e280920 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4013010 tx=0x7f0ee400be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acd9aae1595f882d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:23.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.552+0000 7f0edf400640 1 -- 172.21.15.31:0/625017935 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4030070 con 0x560c4e28a050 2024-03-21T10:38:23.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.552+0000 7f0edf400640 1 -- 172.21.15.31:0/625017935 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ee40055c0 con 0x560c4e28a050 2024-03-21T10:38:23.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.552+0000 7f0edf400640 1 -- 172.21.15.31:0/625017935 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee40035f0 con 0x560c4e28a050 2024-03-21T10:38:23.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.553+0000 7f0efc7c3740 1 -- 172.21.15.31:0/625017935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c4e28a050 msgr2=0x560c4e280920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:23.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.553+0000 7f0efc7c3740 1 --2- 172.21.15.31:0/625017935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c4e28a050 0x560c4e280920 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4013010 tx=0x7f0ee400be20 comp rx=0 tx=0).stop 2024-03-21T10:38:23.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.553+0000 7f0efc7c3740 1 -- 172.21.15.31:0/625017935 shutdown_connections 2024-03-21T10:38:23.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.553+0000 7f0efc7c3740 1 --2- 172.21.15.31:0/625017935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c4e28fbb0 0x560c4e284330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.553+0000 7f0efc7c3740 1 --2- 172.21.15.31:0/625017935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c4e28aa40 0x560c4e29b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.553+0000 7f0efc7c3740 1 --2- 172.21.15.31:0/625017935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c4e28a050 0x560c4e280920 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.553+0000 7f0efc7c3740 1 -- 172.21.15.31:0/625017935 >> 172.21.15.31:0/625017935 conn(0x560c4e252200 msgr2=0x560c4e289490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:23.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.554+0000 7f0efc7c3740 1 -- 172.21.15.31:0/625017935 shutdown_connections 2024-03-21T10:38:23.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.554+0000 7f0efc7c3740 1 -- 172.21.15.31:0/625017935 wait complete. 2024-03-21T10:38:23.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.554+0000 7f0efc7c3740 1 -- start start 2024-03-21T10:38:23.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.554+0000 7f0efc7c3740 1 Processor -- start 2024-03-21T10:38:23.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.555+0000 7f0efc7c3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c4e28a050 0x560c4e298f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.555+0000 7f0efc7c3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c4e28aa40 0x560c4e2994c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.555+0000 7f0efc7c3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c4e28fbb0 0x560c4e29a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.555+0000 7f0efc7c3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef41aeac0 con 0x560c4e28fbb0 2024-03-21T10:38:23.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.555+0000 7f0efc7c3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef41afdc0 con 0x560c4e28aa40 2024-03-21T10:38:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.555+0000 7f0efc7c3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef41af100 con 0x560c4e28a050 2024-03-21T10:38:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.555+0000 7f0efc7c3740 10 client.0 _ll_register_callbacks cb 0x7f0eec09bde0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:38:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.555+0000 7f0ef2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c4e28a050 0x560c4e298f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.555+0000 7f0ef2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c4e28a050 0x560c4e298f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53766/0 (socket says 172.21.15.31:53766) 2024-03-21T10:38:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.555+0000 7f0ef2a00640 1 -- 172.21.15.31:0/805999421 learned_addr learned my addr 172.21.15.31:0/805999421 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.555+0000 7f0ef3400640 1 --2- 172.21.15.31:0/805999421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c4e28fbb0 0x560c4e29a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.555+0000 7f0ef2000640 1 --2- 172.21.15.31:0/805999421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c4e28aa40 0x560c4e2994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.556+0000 7f0ef2a00640 1 -- 172.21.15.31:0/805999421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c4e28aa40 msgr2=0x560c4e2994c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.556+0000 7f0ef2a00640 1 --2- 172.21.15.31:0/805999421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c4e28aa40 0x560c4e2994c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.556+0000 7f0ef2a00640 1 -- 172.21.15.31:0/805999421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c4e28fbb0 msgr2=0x560c4e29a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.556+0000 7f0ef2a00640 1 --2- 172.21.15.31:0/805999421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c4e28fbb0 0x560c4e29a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.556+0000 7f0ef2a00640 1 -- 172.21.15.31:0/805999421 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560c4e29aa50 con 0x560c4e28a050 2024-03-21T10:38:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.556+0000 7f0ef2000640 1 --2- 172.21.15.31:0/805999421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c4e28aa40 0x560c4e2994c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.556+0000 7f0ef2a00640 1 --2- 172.21.15.31:0/805999421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c4e28a050 0x560c4e298f80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4010550 tx=0x7f0ee400bbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6ced14c0ce622db server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:23.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.556+0000 7f0edcc00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:38:23.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.556+0000 7f0edcc00640 1 -- 172.21.15.31:0/805999421 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4039060 con 0x560c4e28a050 2024-03-21T10:38:23.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.556+0000 7f0edcc00640 1 -- 172.21.15.31:0/805999421 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f0ee402a780 con 0x560c4e28a050 2024-03-21T10:38:23.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.556+0000 7f0efc7c3740 1 -- 172.21.15.31:0/805999421 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ee40055c0 con 0x560c4e28a050 2024-03-21T10:38:23.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.556+0000 7f0edcc00640 1 -- 172.21.15.31:0/805999421 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ee4040050 con 0x560c4e28a050 2024-03-21T10:38:23.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.557+0000 7f0efc7c3740 10 client.25549 fetch_fsmap learned FSMap version 24 2024-03-21T10:38:23.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.557+0000 7f0efc7c3740 1 -- 172.21.15.31:0/805999421 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x560c4e2fb900 con 0x560c4e28a050 2024-03-21T10:38:23.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.557+0000 7f0edcc00640 1 -- 172.21.15.31:0/805999421 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ee40055c0 con 0x560c4e28a050 2024-03-21T10:38:23.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.557+0000 7f0edcc00640 1 -- 172.21.15.31:0/805999421 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f0ee4003ce0 con 0x560c4e28a050 2024-03-21T10:38:23.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.557+0000 7f0efc7c3740 10 client.25549 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:38:23.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.557+0000 7f0efc7c3740 10 client.25549 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:38:23.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.557+0000 7f0efc7c3740 20 client.25549 populate_metadata read hostname 'smithi031' 2024-03-21T10:38:23.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.557+0000 7f0efc7c3740 1 --2- 172.21.15.31:0/805999421 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560c4e2fbfc0 0x560c4e2fe3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:23.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.557+0000 7f0efc7c3740 4 client.25549 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:38:23.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.557+0000 7f0efc7c3740 1 -- 172.21.15.31:0/805999421 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560c4e1f6ae0 con 0x560c4e2fbfc0 2024-03-21T10:38:23.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.557+0000 7f0ef2000640 1 --2- 172.21.15.31:0/805999421 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560c4e2fbfc0 0x560c4e2fe3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:23.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.558+0000 7f0ef2000640 1 --2- 172.21.15.31:0/805999421 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560c4e2fbfc0 0x560c4e2fe3e0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=800841f6bc2683a0 server_cookie=535971b55d0c1b0f in_seq=0 out_seq=0 2024-03-21T10:38:23.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.558+0000 7f0edcc00640 10 client.25549 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:23.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.558+0000 7f0edcc00640 1 -- 172.21.15.31:0/805999421 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x560c4e1f6ae0 con 0x560c4e2fbfc0 2024-03-21T10:38:23.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.558+0000 7f0edcc00640 10 client.25549 handle_command_reply: tid=0 2024-03-21T10:38:23.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.558+0000 7f0efc7c3740 1 client.25549 shutdown 2024-03-21T10:38:23.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.558+0000 7f0efc7c3740 1 -- 172.21.15.31:0/805999421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c4e28a050 msgr2=0x560c4e298f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:23.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.558+0000 7f0efc7c3740 1 --2- 172.21.15.31:0/805999421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c4e28a050 0x560c4e298f80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4010550 tx=0x7f0ee400bbe0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.559+0000 7f0efc7c3740 1 -- 172.21.15.31:0/805999421 shutdown_connections 2024-03-21T10:38:23.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.559+0000 7f0efc7c3740 1 --2- 172.21.15.31:0/805999421 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560c4e2fbfc0 0x560c4e2fe3e0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.559+0000 7f0efc7c3740 1 --2- 172.21.15.31:0/805999421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c4e28fbb0 0x560c4e29a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.559+0000 7f0efc7c3740 1 --2- 172.21.15.31:0/805999421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c4e28aa40 0x560c4e2994c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.559+0000 7f0efc7c3740 1 --2- 172.21.15.31:0/805999421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c4e28a050 0x560c4e298f80 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.559+0000 7f0edcc00640 0 client.25549 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:23.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.559+0000 7f0efc7c3740 1 -- 172.21.15.31:0/805999421 >> 172.21.15.31:0/805999421 conn(0x560c4e252200 msgr2=0x560c4e288c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:23.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.559+0000 7f0efc7c3740 1 -- 172.21.15.31:0/805999421 shutdown_connections 2024-03-21T10:38:23.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.559+0000 7f0efc7c3740 1 -- 172.21.15.31:0/805999421 wait complete. 2024-03-21T10:38:23.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.559+0000 7f0efc7c3740 20 client.25549 trim_cache size 0 max 16384 2024-03-21T10:38:23.560 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:38:23.560 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-21T10:38:23.560 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-03-21T10:38:23.560 INFO:teuthology.orchestra.run.smithi031.stdout: "e6245416-4a98-4dd7-80f2-04e460d62b96": { 2024-03-21T10:38:23.560 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-03-21T10:38:23.560 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "e6245416-4a98-4dd7-80f2-04e460d62b96", 2024-03-21T10:38:23.560 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-03-21T10:38:23.561 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:38:23.561 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:38:23.561 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:38:23.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.559+0000 7f0ef3e00640 1 -- 172.21.15.31:0/3134844876 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0ed80a2f20 msgr2=0x7f0ed80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:23.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.559+0000 7f0ef3e00640 1 --2- 172.21.15.31:0/3134844876 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0ed80a2f20 0x7f0ed80a53e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0ee002c260 tx=0x7f0ee0005fd0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.559+0000 7f0ef3e00640 1 -- 172.21.15.31:0/3134844876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ef41096b0 msgr2=0x7f0ef41a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:23.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.559+0000 7f0ef3e00640 1 --2- 172.21.15.31:0/3134844876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ef41096b0 0x7f0ef41a5380 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0eec00eb10 tx=0x7f0eec00c2d0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.560+0000 7f0ef3e00640 1 -- 172.21.15.31:0/3134844876 shutdown_connections 2024-03-21T10:38:23.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.560+0000 7f0ef3e00640 1 --2- 172.21.15.31:0/3134844876 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0ed80a2f20 0x7f0ed80a53e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.560+0000 7f0ef3e00640 1 --2- 172.21.15.31:0/3134844876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ef41096b0 0x7f0ef41a5380 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.560+0000 7f0ef3e00640 1 --2- 172.21.15.31:0/3134844876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ef4108cf0 0x7f0ef419e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.560+0000 7f0ef3e00640 1 --2- 172.21.15.31:0/3134844876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ef4107b00 0x7f0ef419ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.560+0000 7f0ef3e00640 1 -- 172.21.15.31:0/3134844876 >> 172.21.15.31:0/3134844876 conn(0x7f0ef4076a90 msgr2=0x7f0ef410bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:23.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.560+0000 7f0ef3e00640 1 -- 172.21.15.31:0/3134844876 shutdown_connections 2024-03-21T10:38:23.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.560+0000 7f0ef3e00640 1 -- 172.21.15.31:0/3134844876 wait complete. 2024-03-21T10:38:23.581 INFO:tasks.cephfs.filesystem:scrub status for tag:e6245416-4a98-4dd7-80f2-04e460d62b96 - {'path': '/', 'tag': 'e6245416-4a98-4dd7-80f2-04e460d62b96', 'options': 'recursive,force'} 2024-03-21T10:38:23.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.803+0000 7ff77ea00640 1 -- 172.21.15.31:0/1347166313 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mgrmap(e 19) v1 ==== 137871+0+0 (secure 0 0 0) 0x7ff780003970 con 0x7ff788105270 2024-03-21T10:38:23.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.806+0000 7ff77ea00640 1 -- 172.21.15.31:0/1347166313 <== mon.1 v2:172.21.15.70:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7ff780098d30 con 0x7ff788105270 2024-03-21T10:38:23.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.812+0000 7ff78cc00640 1 -- 172.21.15.31:0/1347166313 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff76c0a2f10 msgr2=0x7ff76c0a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:23.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.812+0000 7ff78cc00640 1 --2- 172.21.15.31:0/1347166313 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff76c0a2f10 0x7ff76c0a53d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff778009bd0 tx=0x7ff778005f90 comp rx=0 tx=0).stop 2024-03-21T10:38:23.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.812+0000 7ff78cc00640 1 -- 172.21.15.31:0/1347166313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff788105270 msgr2=0x7ff7881a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:23.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.812+0000 7ff78cc00640 1 --2- 172.21.15.31:0/1347166313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff788105270 0x7ff7881a54f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff78002c630 tx=0x7ff780032f20 comp rx=0 tx=0).stop 2024-03-21T10:38:23.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.812+0000 7ff78cc00640 1 -- 172.21.15.31:0/1347166313 shutdown_connections 2024-03-21T10:38:23.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.812+0000 7ff78cc00640 1 --2- 172.21.15.31:0/1347166313 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7ff76c0a2f10 0x7ff76c0a53d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.812+0000 7ff78cc00640 1 --2- 172.21.15.31:0/1347166313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff788105270 0x7ff7881a54f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.812+0000 7ff78cc00640 1 --2- 172.21.15.31:0/1347166313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7881028c0 0x7ff78819e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.812+0000 7ff78cc00640 1 --2- 172.21.15.31:0/1347166313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7880fff50 0x7ff78819df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:23.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.812+0000 7ff78cc00640 1 -- 172.21.15.31:0/1347166313 >> 172.21.15.31:0/1347166313 conn(0x7ff7880fbb70 msgr2=0x7ff7881008d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:23.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.812+0000 7ff78cc00640 1 -- 172.21.15.31:0/1347166313 shutdown_connections 2024-03-21T10:38:23.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:23.812+0000 7ff78cc00640 1 -- 172.21.15.31:0/1347166313 wait complete. 2024-03-21T10:38:23.838 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-21T10:38:24.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:23 smithi031 ceph-mon[31673]: pgmap v427: 129 pgs: 129 active+clean; 649 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-21T10:38:24.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:23 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3622328584' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:38:24.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:23 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1347166313' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-21T10:38:24.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:23 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-21T10:38:24.056 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:23 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: ignoring --setuser ceph since I am not root 2024-03-21T10:38:24.056 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:23 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: ignoring --setgroup ceph since I am not root 2024-03-21T10:38:24.073 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:23 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: ignoring --setuser ceph since I am not root 2024-03-21T10:38:24.073 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:23 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: ignoring --setgroup ceph since I am not root 2024-03-21T10:38:24.073 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:23 smithi070 ceph-mgr[38200]: -- 172.21.15.70:0/2976927242 <== mon.1 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5581bd7065a0 con 0x5581bd6db000 2024-03-21T10:38:24.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:23 smithi070 ceph-mon[36330]: pgmap v427: 129 pgs: 129 active+clean; 649 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-21T10:38:24.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:23 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3622328584' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:38:24.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:23 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1347166313' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-21T10:38:24.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:23 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-21T10:38:24.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.078+0000 7fc003200640 1 -- 172.21.15.31:0/3336086544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0041030c0 msgr2=0x7fc004109950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:24.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.078+0000 7fc003200640 1 --2- 172.21.15.31:0/3336086544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0041030c0 0x7fc004109950 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbffc009be0 tx=0x7fbffc02bef0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.078+0000 7fc003200640 1 -- 172.21.15.31:0/3336086544 shutdown_connections 2024-03-21T10:38:24.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.078+0000 7fc003200640 1 --2- 172.21.15.31:0/3336086544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0041030c0 0x7fc004109950 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.078+0000 7fc003200640 1 --2- 172.21.15.31:0/3336086544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc004102700 0x7fc004102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.078+0000 7fc003200640 1 --2- 172.21.15.31:0/3336086544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc004101510 0x7fc004101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.078+0000 7fc003200640 1 -- 172.21.15.31:0/3336086544 >> 172.21.15.31:0/3336086544 conn(0x7fc0040fca80 msgr2=0x7fc0040feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:24.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.078+0000 7fc003200640 1 -- 172.21.15.31:0/3336086544 shutdown_connections 2024-03-21T10:38:24.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.078+0000 7fc003200640 1 -- 172.21.15.31:0/3336086544 wait complete. 2024-03-21T10:38:24.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.079+0000 7fc003200640 1 Processor -- start 2024-03-21T10:38:24.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.080+0000 7fc003200640 1 -- start start 2024-03-21T10:38:24.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.080+0000 7fc003200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc004101510 0x7fc0041999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:24.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.080+0000 7fc003200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc004102700 0x7fc004199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:24.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.080+0000 7fc003200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0041030c0 0x7fc0041a0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:24.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.080+0000 7fc003200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00410d1f0 con 0x7fc0041030c0 2024-03-21T10:38:24.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.080+0000 7fc003200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00410d070 con 0x7fc004101510 2024-03-21T10:38:24.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.080+0000 7fc003200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00410d370 con 0x7fc004102700 2024-03-21T10:38:24.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.080+0000 7fc001e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc004101510 0x7fc0041999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:24.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.080+0000 7fbffbe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc004102700 0x7fc004199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:24.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.080+0000 7fc001e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc004101510 0x7fc0041999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46376/0 (socket says 172.21.15.31:46376) 2024-03-21T10:38:24.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.080+0000 7fc001e00640 1 -- 172.21.15.31:0/1321894431 learned_addr learned my addr 172.21.15.31:0/1321894431 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:24.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.080+0000 7fc002800640 1 --2- 172.21.15.31:0/1321894431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0041030c0 0x7fc0041a0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:24.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.081+0000 7fc001e00640 1 -- 172.21.15.31:0/1321894431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc004102700 msgr2=0x7fc004199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:24.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.081+0000 7fc001e00640 1 --2- 172.21.15.31:0/1321894431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc004102700 0x7fc004199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.081+0000 7fc001e00640 1 -- 172.21.15.31:0/1321894431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0041030c0 msgr2=0x7fc0041a0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:24.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.081+0000 7fc001e00640 1 --2- 172.21.15.31:0/1321894431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0041030c0 0x7fc0041a0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.081+0000 7fc001e00640 1 -- 172.21.15.31:0/1321894431 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0041a15a0 con 0x7fc004101510 2024-03-21T10:38:24.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.081+0000 7fbffbe00640 1 --2- 172.21.15.31:0/1321894431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc004102700 0x7fc004199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:24.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.081+0000 7fc001e00640 1 --2- 172.21.15.31:0/1321894431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc004101510 0x7fc0041999d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbff400e9a0 tx=0x7fbff400ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:24.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.081+0000 7fbffaa00640 1 -- 172.21.15.31:0/1321894431 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff400cd80 con 0x7fc004101510 2024-03-21T10:38:24.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.081+0000 7fbffaa00640 1 -- 172.21.15.31:0/1321894431 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbff4010910 con 0x7fc004101510 2024-03-21T10:38:24.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.081+0000 7fbffaa00640 1 -- 172.21.15.31:0/1321894431 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff4018a30 con 0x7fc004101510 2024-03-21T10:38:24.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.081+0000 7fc003200640 1 -- 172.21.15.31:0/1321894431 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0041a1890 con 0x7fc004101510 2024-03-21T10:38:24.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.082+0000 7fc003200640 1 -- 172.21.15.31:0/1321894431 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0041a1d90 con 0x7fc004101510 2024-03-21T10:38:24.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.083+0000 7fc003200640 1 -- 172.21.15.31:0/1321894431 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0041a1ff0 con 0x7fc004101510 2024-03-21T10:38:24.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.085+0000 7fbffaa00640 1 -- 172.21.15.31:0/1321894431 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 19) v1 ==== 137871+0+0 (secure 0 0 0) 0x7fbff4010ac0 con 0x7fc004101510 2024-03-21T10:38:24.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.086+0000 7fbffaa00640 1 --2- 172.21.15.31:0/1321894431 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fbfe80a2f50 0x7fbfe80a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:24.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.086+0000 7fbffaa00640 1 -- 172.21.15.31:0/1321894431 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7fbff40cfb60 con 0x7fc004101510 2024-03-21T10:38:24.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.086+0000 7fbffbe00640 1 -- 172.21.15.31:0/1321894431 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fbfe80a2f50 msgr2=0x7fbfe80a5410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3875791078 2024-03-21T10:38:24.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.086+0000 7fbffbe00640 1 --2- 172.21.15.31:0/1321894431 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fbfe80a2f50 0x7fbfe80a5410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-21T10:38:24.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.090+0000 7fbffaa00640 1 -- 172.21.15.31:0/1321894431 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbff4053740 con 0x7fc004101510 2024-03-21T10:38:24.181 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:23 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: ignoring --setuser ceph since I am not root 2024-03-21T10:38:24.181 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:23 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: ignoring --setgroup ceph since I am not root 2024-03-21T10:38:24.182 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:23 smithi040 ceph-mgr[37543]: -- 172.21.15.40:0/743911697 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564bb7dea5a0 con 0x564bb7dbe800 2024-03-21T10:38:24.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:23 smithi040 ceph-mon[36154]: pgmap v427: 129 pgs: 129 active+clean; 649 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-21T10:38:24.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:23 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3622328584' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:38:24.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:23 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1347166313' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-21T10:38:24.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:23 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-21T10:38:24.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.287+0000 7fbffbe00640 1 -- 172.21.15.31:0/1321894431 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fbfe80a2f50 msgr2=0x7fbfe80a5410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3875791078 2024-03-21T10:38:24.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.287+0000 7fbffbe00640 1 --2- 172.21.15.31:0/1321894431 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fbfe80a2f50 0x7fbfe80a5410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-21T10:38:24.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.297+0000 7fc003200640 1 -- 172.21.15.31:0/1321894431 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fc0041a21c0 con 0x7fc004101510 2024-03-21T10:38:24.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.305+0000 7fbffaa00640 1 -- 172.21.15.31:0/1321894431 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) v1 ==== 138+0+0 (secure 0 0 0) 0x7fbff409c5d0 con 0x7fc004101510 2024-03-21T10:38:24.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.309+0000 7fc003200640 1 -- 172.21.15.31:0/1321894431 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fbfe80a2f50 msgr2=0x7fbfe80a5410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T10:38:24.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.309+0000 7fc003200640 1 --2- 172.21.15.31:0/1321894431 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fbfe80a2f50 0x7fbfe80a5410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.309+0000 7fc003200640 1 -- 172.21.15.31:0/1321894431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc004101510 msgr2=0x7fc0041999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:24.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.309+0000 7fc003200640 1 --2- 172.21.15.31:0/1321894431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc004101510 0x7fc0041999d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbff400e9a0 tx=0x7fbff400ee60 comp rx=0 tx=0).stop 2024-03-21T10:38:24.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.309+0000 7fc003200640 1 -- 172.21.15.31:0/1321894431 shutdown_connections 2024-03-21T10:38:24.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.309+0000 7fc003200640 1 --2- 172.21.15.31:0/1321894431 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7fbfe80a2f50 0x7fbfe80a5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.309+0000 7fc003200640 1 --2- 172.21.15.31:0/1321894431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0041030c0 0x7fc0041a0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.309+0000 7fc003200640 1 --2- 172.21.15.31:0/1321894431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc004102700 0x7fc004199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.309+0000 7fc003200640 1 --2- 172.21.15.31:0/1321894431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc004101510 0x7fc0041999d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.309+0000 7fc003200640 1 -- 172.21.15.31:0/1321894431 >> 172.21.15.31:0/1321894431 conn(0x7fc0040fca80 msgr2=0x7fc0040738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:24.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.309+0000 7fc003200640 1 -- 172.21.15.31:0/1321894431 shutdown_connections 2024-03-21T10:38:24.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.309+0000 7fc003200640 1 -- 172.21.15.31:0/1321894431 wait complete. 2024-03-21T10:38:24.335 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-21T10:38:24.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.539+0000 7f82e9200640 1 -- 172.21.15.31:0/3531732791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82e40772a0 msgr2=0x7f82e4077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:24.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.539+0000 7f82e9200640 1 --2- 172.21.15.31:0/3531732791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82e40772a0 0x7f82e4077700 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f82d0009860 tx=0x7f82d002bb70 comp rx=0 tx=0).stop 2024-03-21T10:38:24.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.539+0000 7f82e9200640 1 -- 172.21.15.31:0/3531732791 shutdown_connections 2024-03-21T10:38:24.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.539+0000 7f82e9200640 1 --2- 172.21.15.31:0/3531732791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82e4077c40 0x7f82e40784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.539+0000 7f82e9200640 1 --2- 172.21.15.31:0/3531732791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82e40772a0 0x7f82e4077700 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.539+0000 7f82e9200640 1 --2- 172.21.15.31:0/3531732791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82e4076060 0x7f82e4076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.539+0000 7f82e9200640 1 -- 172.21.15.31:0/3531732791 >> 172.21.15.31:0/3531732791 conn(0x7f82e40fee90 msgr2=0x7f82e41012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:24.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.539+0000 7f82e9200640 1 -- 172.21.15.31:0/3531732791 shutdown_connections 2024-03-21T10:38:24.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.539+0000 7f82e9200640 1 -- 172.21.15.31:0/3531732791 wait complete. 2024-03-21T10:38:24.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.540+0000 7f82e9200640 1 Processor -- start 2024-03-21T10:38:24.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.540+0000 7f82e9200640 1 -- start start 2024-03-21T10:38:24.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.541+0000 7f82e9200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82e4076060 0x7f82e419dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:24.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.541+0000 7f82e9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82e40772a0 0x7f82e419e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:24.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.541+0000 7f82e9200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82e4077c40 0x7f82e41a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:24.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.541+0000 7f82e9200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e4113990 con 0x7f82e4076060 2024-03-21T10:38:24.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.541+0000 7f82e9200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e4113810 con 0x7f82e40772a0 2024-03-21T10:38:24.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.541+0000 7f82e9200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e4113b10 con 0x7f82e4077c40 2024-03-21T10:38:24.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.541+0000 7f82e1e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82e4076060 0x7f82e419dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:24.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.541+0000 7f82e1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82e40772a0 0x7f82e419e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:24.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.541+0000 7f82e1e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82e4076060 0x7f82e419dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33744/0 (socket says 172.21.15.31:33744) 2024-03-21T10:38:24.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.541+0000 7f82e1e00640 1 -- 172.21.15.31:0/3456738624 learned_addr learned my addr 172.21.15.31:0/3456738624 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:24.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.541+0000 7f82e2800640 1 --2- 172.21.15.31:0/3456738624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82e4077c40 0x7f82e41a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:24.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.541+0000 7f82e1400640 1 -- 172.21.15.31:0/3456738624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82e4077c40 msgr2=0x7f82e41a5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:24.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.541+0000 7f82e1400640 1 --2- 172.21.15.31:0/3456738624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82e4077c40 0x7f82e41a5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.541+0000 7f82e1400640 1 -- 172.21.15.31:0/3456738624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82e4076060 msgr2=0x7f82e419dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:24.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.541+0000 7f82e1400640 1 --2- 172.21.15.31:0/3456738624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82e4076060 0x7f82e419dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.541+0000 7f82e1400640 1 -- 172.21.15.31:0/3456738624 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e41a5be0 con 0x7f82e40772a0 2024-03-21T10:38:24.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.542+0000 7f82e1400640 1 --2- 172.21.15.31:0/3456738624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82e40772a0 0x7f82e419e500 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f82d002c0b0 tx=0x7f82d0032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:24.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.542+0000 7f82daa00640 1 -- 172.21.15.31:0/3456738624 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d0041020 con 0x7f82e40772a0 2024-03-21T10:38:24.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.542+0000 7f82e9200640 1 -- 172.21.15.31:0/3456738624 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e41a5e70 con 0x7f82e40772a0 2024-03-21T10:38:24.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.542+0000 7f82daa00640 1 -- 172.21.15.31:0/3456738624 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f82d002c9f0 con 0x7f82e40772a0 2024-03-21T10:38:24.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.542+0000 7f82daa00640 1 -- 172.21.15.31:0/3456738624 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d0039040 con 0x7f82e40772a0 2024-03-21T10:38:24.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.542+0000 7f82e9200640 1 -- 172.21.15.31:0/3456738624 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e41a62e0 con 0x7f82e40772a0 2024-03-21T10:38:24.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.543+0000 7f82e9200640 1 -- 172.21.15.31:0/3456738624 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82e4110400 con 0x7f82e40772a0 2024-03-21T10:38:24.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.545+0000 7f82daa00640 1 -- 172.21.15.31:0/3456738624 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 19) v1 ==== 137871+0+0 (secure 0 0 0) 0x7f82d0044020 con 0x7f82e40772a0 2024-03-21T10:38:24.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.547+0000 7f82daa00640 1 --2- 172.21.15.31:0/3456738624 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f82c80a2fe0 0x7f82c80a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:24.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.547+0000 7f82daa00640 1 -- 172.21.15.31:0/3456738624 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f82d00f1300 con 0x7f82e40772a0 2024-03-21T10:38:24.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.547+0000 7f82e1e00640 1 -- 172.21.15.31:0/3456738624 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f82c80a2fe0 msgr2=0x7f82c80a54a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3875791078 2024-03-21T10:38:24.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.547+0000 7f82e1e00640 1 --2- 172.21.15.31:0/3456738624 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f82c80a2fe0 0x7f82c80a54a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-21T10:38:24.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.553+0000 7f82daa00640 1 -- 172.21.15.31:0/3456738624 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f82d004b020 con 0x7f82e40772a0 2024-03-21T10:38:24.681 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:24 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:38:24.379+0000 7f7a7bf18200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T10:38:24.681 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:24 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:38:24.476+0000 7f7a7bf18200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T10:38:24.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.748+0000 7f82e1e00640 1 -- 172.21.15.31:0/3456738624 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f82c80a2fe0 msgr2=0x7f82c80a54a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3875791078 2024-03-21T10:38:24.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.748+0000 7f82e1e00640 1 --2- 172.21.15.31:0/3456738624 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f82c80a2fe0 0x7f82c80a54a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-21T10:38:24.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.754+0000 7f82e9200640 1 -- 172.21.15.31:0/3456738624 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f82e402d070 con 0x7f82e40772a0 2024-03-21T10:38:24.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.762+0000 7f82daa00640 1 -- 172.21.15.31:0/3456738624 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) v1 ==== 133+0+0 (secure 0 0 0) 0x7f82d00bddf0 con 0x7f82e40772a0 2024-03-21T10:38:24.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.765+0000 7f82e9200640 1 -- 172.21.15.31:0/3456738624 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f82c80a2fe0 msgr2=0x7f82c80a54a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T10:38:24.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.765+0000 7f82e9200640 1 --2- 172.21.15.31:0/3456738624 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f82c80a2fe0 0x7f82c80a54a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.765+0000 7f82e9200640 1 -- 172.21.15.31:0/3456738624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82e40772a0 msgr2=0x7f82e419e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:24.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.765+0000 7f82e9200640 1 --2- 172.21.15.31:0/3456738624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82e40772a0 0x7f82e419e500 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f82d002c0b0 tx=0x7f82d0032430 comp rx=0 tx=0).stop 2024-03-21T10:38:24.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.765+0000 7f82e9200640 1 -- 172.21.15.31:0/3456738624 shutdown_connections 2024-03-21T10:38:24.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.765+0000 7f82e9200640 1 --2- 172.21.15.31:0/3456738624 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f82c80a2fe0 0x7f82c80a54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.765+0000 7f82e9200640 1 --2- 172.21.15.31:0/3456738624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82e4077c40 0x7f82e41a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.765+0000 7f82e9200640 1 --2- 172.21.15.31:0/3456738624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82e40772a0 0x7f82e419e500 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.765+0000 7f82e9200640 1 --2- 172.21.15.31:0/3456738624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82e4076060 0x7f82e419dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.765+0000 7f82e9200640 1 -- 172.21.15.31:0/3456738624 >> 172.21.15.31:0/3456738624 conn(0x7f82e40fee90 msgr2=0x7f82e4100590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:24.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.765+0000 7f82e9200640 1 -- 172.21.15.31:0/3456738624 shutdown_connections 2024-03-21T10:38:24.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.765+0000 7f82e9200640 1 -- 172.21.15.31:0/3456738624 wait complete. 2024-03-21T10:38:24.782 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-21T10:38:24.792 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:24 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:38:24.404+0000 7f37f45d9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T10:38:24.792 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:24 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:38:24.518+0000 7f37f45d9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T10:38:24.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.969+0000 7f0a9f200640 1 -- 172.21.15.31:0/179098471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a981032a0 msgr2=0x7f0a98109b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:24.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.969+0000 7f0a9f200640 1 --2- 172.21.15.31:0/179098471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a981032a0 0x7f0a98109b30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0a90009ca0 tx=0x7f0a9002c0f0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.969+0000 7f0a9f200640 1 -- 172.21.15.31:0/179098471 shutdown_connections 2024-03-21T10:38:24.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.969+0000 7f0a9f200640 1 --2- 172.21.15.31:0/179098471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a981032a0 0x7f0a98109b30 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.969+0000 7f0a9f200640 1 --2- 172.21.15.31:0/179098471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a981028e0 0x7f0a98102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.969+0000 7f0a9f200640 1 --2- 172.21.15.31:0/179098471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a981016f0 0x7f0a98101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.969+0000 7f0a9f200640 1 -- 172.21.15.31:0/179098471 >> 172.21.15.31:0/179098471 conn(0x7f0a980fcc60 msgr2=0x7f0a980ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:24.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.970+0000 7f0a9f200640 1 -- 172.21.15.31:0/179098471 shutdown_connections 2024-03-21T10:38:24.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.970+0000 7f0a9f200640 1 -- 172.21.15.31:0/179098471 wait complete. 2024-03-21T10:38:24.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.970+0000 7f0a9f200640 1 Processor -- start 2024-03-21T10:38:24.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.971+0000 7f0a9f200640 1 -- start start 2024-03-21T10:38:24.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.971+0000 7f0a9f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a981016f0 0x7f0a98075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:24.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.971+0000 7f0a9f200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a981028e0 0x7f0a98073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.971+0000 7f0a9f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a981032a0 0x7f0a98073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.971+0000 7f0a9f200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9810d350 con 0x7f0a981032a0 2024-03-21T10:38:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.971+0000 7f0a9f200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9810d1d0 con 0x7f0a981016f0 2024-03-21T10:38:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.971+0000 7f0a9f200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9810d4d0 con 0x7f0a981028e0 2024-03-21T10:38:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.971+0000 7f0a97e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a981016f0 0x7f0a98075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.971+0000 7f0a97400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a981028e0 0x7f0a98073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.971+0000 7f0a9d200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a981032a0 0x7f0a98073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.971+0000 7f0a97e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a981016f0 0x7f0a98075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46420/0 (socket says 172.21.15.31:46420) 2024-03-21T10:38:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.971+0000 7f0a97400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a981028e0 0x7f0a98073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53834/0 (socket says 172.21.15.31:53834) 2024-03-21T10:38:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.971+0000 7f0a97e00640 1 -- 172.21.15.31:0/705533408 learned_addr learned my addr 172.21.15.31:0/705533408 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.972+0000 7f0a97400640 1 -- 172.21.15.31:0/705533408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a981016f0 msgr2=0x7f0a98075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.972+0000 7f0a97400640 1 --2- 172.21.15.31:0/705533408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a981016f0 0x7f0a98075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.972+0000 7f0a97400640 1 -- 172.21.15.31:0/705533408 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a981032a0 msgr2=0x7f0a98073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.972+0000 7f0a97400640 1 --2- 172.21.15.31:0/705533408 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a981032a0 0x7f0a98073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:24.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.972+0000 7f0a97400640 1 -- 172.21.15.31:0/705533408 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a98074860 con 0x7f0a981028e0 2024-03-21T10:38:24.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.972+0000 7f0a97e00640 1 --2- 172.21.15.31:0/705533408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a981016f0 0x7f0a98075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:24.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.972+0000 7f0a97400640 1 --2- 172.21.15.31:0/705533408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a981028e0 0x7f0a98073a90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0a8400e9f0 tx=0x7f0a8400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:24.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.972+0000 7f0a94c00640 1 -- 172.21.15.31:0/705533408 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8400cd80 con 0x7f0a981028e0 2024-03-21T10:38:24.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.972+0000 7f0a94c00640 1 -- 172.21.15.31:0/705533408 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a840035f0 con 0x7f0a981028e0 2024-03-21T10:38:24.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.972+0000 7f0a94c00640 1 -- 172.21.15.31:0/705533408 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a84010690 con 0x7f0a981028e0 2024-03-21T10:38:24.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.972+0000 7f0a9f200640 1 -- 172.21.15.31:0/705533408 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a981a1a90 con 0x7f0a981028e0 2024-03-21T10:38:24.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.972+0000 7f0a9f200640 1 -- 172.21.15.31:0/705533408 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a981a1f90 con 0x7f0a981028e0 2024-03-21T10:38:24.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.974+0000 7f0a9f200640 1 -- 172.21.15.31:0/705533408 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a98108a10 con 0x7f0a981028e0 2024-03-21T10:38:24.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.976+0000 7f0a94c00640 1 -- 172.21.15.31:0/705533408 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) v1 ==== 137871+0+0 (secure 0 0 0) 0x7f0a84003b30 con 0x7f0a981028e0 2024-03-21T10:38:24.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.979+0000 7f0a94c00640 1 --2- 172.21.15.31:0/705533408 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0a7c0a2f60 0x7f0a7c0a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:24.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.979+0000 7f0a97e00640 1 -- 172.21.15.31:0/705533408 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0a7c0a2f60 msgr2=0x7f0a7c0a5420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3875791078 2024-03-21T10:38:24.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.979+0000 7f0a94c00640 1 -- 172.21.15.31:0/705533408 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9100+0+0 (secure 0 0 0) 0x7f0a840cfaa0 con 0x7f0a981028e0 2024-03-21T10:38:24.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.979+0000 7f0a97e00640 1 --2- 172.21.15.31:0/705533408 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0a7c0a2f60 0x7f0a7c0a5420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-21T10:38:24.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:24.982+0000 7f0a94c00640 1 -- 172.21.15.31:0/705533408 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0a8409c570 con 0x7f0a981028e0 2024-03-21T10:38:25.055 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:24 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:24.714+0000 7f462819b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T10:38:25.055 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:24 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:24.844+0000 7f462819b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T10:38:25.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:24 smithi031 ceph-mon[31673]: scrub queued for path: / 2024-03-21T10:38:25.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:24 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:38:25.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:24 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:38:25.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:24 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:38:25.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:24 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-21T10:38:25.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:24 smithi031 ceph-mon[31673]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-03-21T10:38:25.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:24 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' 2024-03-21T10:38:25.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:24 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' 2024-03-21T10:38:25.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:24 smithi070 ceph-mon[36330]: scrub queued for path: / 2024-03-21T10:38:25.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:24 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:38:25.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:24 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:38:25.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:24 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:38:25.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:24 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-21T10:38:25.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:24 smithi070 ceph-mon[36330]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-03-21T10:38:25.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:24 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' 2024-03-21T10:38:25.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:24 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' 2024-03-21T10:38:25.149 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:24 smithi040 ceph-mon[36154]: scrub queued for path: / 2024-03-21T10:38:25.149 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:24 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:38:25.149 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:24 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:38:25.149 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:24 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:38:25.149 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:24 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-21T10:38:25.149 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:24 smithi040 ceph-mon[36154]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-03-21T10:38:25.149 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:24 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' 2024-03-21T10:38:25.149 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:24 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' 2024-03-21T10:38:25.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:25.180+0000 7f0a97e00640 1 -- 172.21.15.31:0/705533408 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0a7c0a2f60 msgr2=0x7f0a7c0a5420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3875791078 2024-03-21T10:38:25.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:25.180+0000 7f0a97e00640 1 --2- 172.21.15.31:0/705533408 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0a7c0a2f60 0x7f0a7c0a5420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-21T10:38:25.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:25.240+0000 7f0a9f200640 1 -- 172.21.15.31:0/705533408 --> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a98061ae0 con 0x7f0a7c0a2f60 2024-03-21T10:38:25.431 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:25 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:38:25.151+0000 7f7a7bf18200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T10:38:25.431 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:25 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:38:25.247+0000 7f7a7bf18200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T10:38:25.431 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:25 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:38:25.393+0000 7f7a7bf18200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T10:38:25.443 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:25 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:38:25.186+0000 7f37f45d9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T10:38:25.443 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:25 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:38:25.280+0000 7f37f45d9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T10:38:25.443 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:25 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:38:25.445+0000 7f37f45d9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T10:38:25.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:25.581+0000 7f0a97e00640 1 -- 172.21.15.31:0/705533408 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0a7c0a2f60 msgr2=0x7f0a7c0a5420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3875791078 2024-03-21T10:38:25.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:25.581+0000 7f0a97e00640 1 --2- 172.21.15.31:0/705533408 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0a7c0a2f60 0x7f0a7c0a5420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-21T10:38:25.825 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:25 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:38:25.544+0000 7f37f45d9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T10:38:25.825 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:25 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:38:25.628+0000 7f37f45d9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T10:38:25.849 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:25 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:38:25.493+0000 7f7a7bf18200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T10:38:25.849 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:25 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:38:25.579+0000 7f7a7bf18200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T10:38:25.849 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:25 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:38:25.851+0000 7f7a7bf18200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T10:38:26.038 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:25 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:25.723+0000 7f462819b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T10:38:26.038 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:25 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:25.851+0000 7f462819b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T10:38:26.153 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:25 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:38:25.965+0000 7f7a7bf18200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T10:38:26.153 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:26 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:38:26.057+0000 7f7a7bf18200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T10:38:26.199 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:25 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:38:25.894+0000 7f37f45d9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T10:38:26.199 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:26 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:38:26.006+0000 7f37f45d9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T10:38:26.200 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:26 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:38:26.106+0000 7f37f45d9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T10:38:26.305 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:26 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:26.039+0000 7f462819b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T10:38:26.305 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:26 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:26.170+0000 7f462819b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T10:38:26.305 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:26 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:26.287+0000 7f462819b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T10:38:26.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:26.382+0000 7f0a97e00640 1 -- 172.21.15.31:0/705533408 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0a7c0a2f60 msgr2=0x7f0a7c0a5420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3875791078 2024-03-21T10:38:26.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:26.382+0000 7f0a97e00640 1 --2- 172.21.15.31:0/705533408 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0a7c0a2f60 0x7f0a7c0a5420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-21T10:38:26.431 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:26 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:38:26.155+0000 7f7a7bf18200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T10:38:26.431 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:26 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:38:26.337+0000 7f7a7bf18200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T10:38:26.573 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:26 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:38:26.202+0000 7f37f45d9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T10:38:26.573 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:26 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:38:26.380+0000 7f37f45d9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T10:38:26.897 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:26 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:26.635+0000 7f462819b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T10:38:26.897 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:26 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:26.782+0000 7f462819b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T10:38:26.897 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:26 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:26.898+0000 7f462819b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T10:38:27.264 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:27 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:27.035+0000 7f462819b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T10:38:27.555 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:27 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:27.266+0000 7f462819b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T10:38:27.823 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:27 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:38:27.359+0000 7f37f45d9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T10:38:27.823 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:27 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:38:27.510+0000 7f37f45d9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T10:38:27.931 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:27 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:38:27.500+0000 7f7a7bf18200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T10:38:28.077 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:27 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:38:27.652+0000 7f7a7bf18200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T10:38:28.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:27.984+0000 7f0a97e00640 1 -- 172.21.15.31:0/705533408 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0a7c0a2f60 msgr2=0x7f0a7c0a5420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3875791078 2024-03-21T10:38:28.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:27.984+0000 7f0a97e00640 1 --2- 172.21.15.31:0/705533408 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0a7c0a2f60 0x7f0a7c0a5420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-21T10:38:28.133 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:27 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:38:27.848+0000 7f37f45d9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T10:38:28.134 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:27 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:38:27.938+0000 7f37f45d9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T10:38:28.134 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:28 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:38:28.136+0000 7f37f45d9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T10:38:28.290 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:27 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:38:27.997+0000 7f7a7bf18200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T10:38:28.290 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:28 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:38:28.090+0000 7f7a7bf18200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T10:38:28.290 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:28 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:38:28.292+0000 7f7a7bf18200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T10:38:28.573 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:28 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:38:28.323+0000 7f37f45d9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T10:38:28.681 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:28 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:38:28.482+0000 7f7a7bf18200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T10:38:28.871 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:28 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:38:28.610+0000 7f37f45d9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T10:38:29.002 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:28 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:28.722+0000 7f462819b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T10:38:29.002 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:28 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:28.905+0000 7f462819b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T10:38:29.054 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:28 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:38:28.775+0000 7f7a7bf18200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T10:38:29.323 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:28 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:38:28.873+0000 7f37f45d9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T10:38:29.323 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:29 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:38:29.113+0000 7f37f45d9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T10:38:29.431 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:29 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:38:29.056+0000 7f7a7bf18200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T10:38:29.431 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:29 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:38:29.301+0000 7f7a7bf18200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T10:38:29.772 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:29 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:29.380+0000 7f462819b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T10:38:29.772 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:29 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:29.504+0000 7f462819b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T10:38:30.032 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:29 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:29.774+0000 7f462819b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T10:38:30.032 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:30 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:30.033+0000 7f462819b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T10:38:30.431 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:30 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:38:30.146+0000 7f7a7bf18200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T10:38:30.573 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:30 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:38:30.090+0000 7f37f45d9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T10:38:30.756 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:30 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:30.430+0000 7f462819b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T10:38:30.983 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:30 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:38:30.667+0000 7f37f45d9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T10:38:30.983 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:30 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:38:30.765+0000 7f37f45d9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T10:38:31.055 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:30 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:30.757+0000 7f462819b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T10:38:31.068 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:30 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:38:30.741+0000 7f7a7bf18200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T10:38:31.068 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:30 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:38:30.843+0000 7f7a7bf18200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T10:38:31.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:31.074+0000 7f0a94c00640 1 -- 172.21.15.31:0/705533408 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mgrmap(e 20) v1 ==== 137871+0+0 (secure 0 0 0) 0x7f0a840984e0 con 0x7f0a981028e0 2024-03-21T10:38:31.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:31.187+0000 7f0a97e00640 1 -- 172.21.15.31:0/705533408 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0a7c0a2f60 msgr2=0x7f0a7c0a5420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3875791078 2024-03-21T10:38:31.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:31.188+0000 7f0a97e00640 1 --2- 172.21.15.31:0/705533408 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0a7c0a2f60 0x7f0a7c0a5420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-21T10:38:31.323 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:38:30 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z[38196]: 2024-03-21T10:38:30.985+0000 7f37f45d9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T10:38:31.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:31 smithi070 ceph-mon[36330]: Standby manager daemon z restarted 2024-03-21T10:38:31.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:31 smithi070 ceph-mon[36330]: Standby manager daemon z started 2024-03-21T10:38:31.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:31 smithi070 ceph-mon[36330]: from='mgr.? 172.21.15.70:0/1616832927' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-21T10:38:31.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:31 smithi070 ceph-mon[36330]: from='mgr.? 172.21.15.70:0/1616832927' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T10:38:31.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:31 smithi070 ceph-mon[36330]: from='mgr.? 172.21.15.70:0/1616832927' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-21T10:38:31.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:31 smithi070 ceph-mon[36330]: from='mgr.? 172.21.15.70:0/1616832927' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T10:38:31.431 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:38:31 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y[37532]: 2024-03-21T10:38:31.071+0000 7f7a7bf18200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T10:38:31.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:31 smithi040 ceph-mon[36154]: Standby manager daemon z restarted 2024-03-21T10:38:31.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:31 smithi040 ceph-mon[36154]: Standby manager daemon z started 2024-03-21T10:38:31.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:31 smithi040 ceph-mon[36154]: from='mgr.? 172.21.15.70:0/1616832927' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-21T10:38:31.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:31 smithi040 ceph-mon[36154]: from='mgr.? 172.21.15.70:0/1616832927' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T10:38:31.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:31 smithi040 ceph-mon[36154]: from='mgr.? 172.21.15.70:0/1616832927' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-21T10:38:31.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:31 smithi040 ceph-mon[36154]: from='mgr.? 172.21.15.70:0/1616832927' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T10:38:31.554 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:31 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:31.086+0000 7f462819b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T10:38:31.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:31 smithi031 ceph-mon[31673]: Standby manager daemon z restarted 2024-03-21T10:38:31.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:31 smithi031 ceph-mon[31673]: Standby manager daemon z started 2024-03-21T10:38:31.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:31 smithi031 ceph-mon[31673]: from='mgr.? 172.21.15.70:0/1616832927' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-21T10:38:31.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:31 smithi031 ceph-mon[31673]: from='mgr.? 172.21.15.70:0/1616832927' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T10:38:31.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:31 smithi031 ceph-mon[31673]: from='mgr.? 172.21.15.70:0/1616832927' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-21T10:38:31.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:31 smithi031 ceph-mon[31673]: from='mgr.? 172.21.15.70:0/1616832927' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T10:38:32.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:32.099+0000 7f0a94c00640 1 -- 172.21.15.31:0/705533408 <== mon.1 v2:172.21.15.70:3300/0 8 ==== mgrmap(e 21) v1 ==== 137871+0+0 (secure 0 0 0) 0x7f0a84099b80 con 0x7f0a981028e0 2024-03-21T10:38:32.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:32 smithi040 ceph-mon[36154]: mgrmap e20: x(active, since 11m), standbys: y, z 2024-03-21T10:38:32.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:32 smithi040 ceph-mon[36154]: from='mgr.? 172.21.15.40:0/1971098852' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-21T10:38:32.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:32 smithi040 ceph-mon[36154]: from='mgr.? 172.21.15.40:0/1971098852' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T10:38:32.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:32 smithi040 ceph-mon[36154]: Standby manager daemon y restarted 2024-03-21T10:38:32.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:32 smithi040 ceph-mon[36154]: Standby manager daemon y started 2024-03-21T10:38:32.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:32 smithi040 ceph-mon[36154]: from='mgr.? 172.21.15.40:0/1971098852' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-21T10:38:32.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:32 smithi040 ceph-mon[36154]: from='mgr.? 172.21.15.40:0/1971098852' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T10:38:32.554 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:32 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:32.268+0000 7f462819b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T10:38:32.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:32 smithi031 ceph-mon[31673]: mgrmap e20: x(active, since 11m), standbys: y, z 2024-03-21T10:38:32.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:32 smithi031 ceph-mon[31673]: from='mgr.? 172.21.15.40:0/1971098852' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-21T10:38:32.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:32 smithi031 ceph-mon[31673]: from='mgr.? 172.21.15.40:0/1971098852' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T10:38:32.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:32 smithi031 ceph-mon[31673]: Standby manager daemon y restarted 2024-03-21T10:38:32.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:32 smithi031 ceph-mon[31673]: Standby manager daemon y started 2024-03-21T10:38:32.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:32 smithi031 ceph-mon[31673]: from='mgr.? 172.21.15.40:0/1971098852' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-21T10:38:32.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:32 smithi031 ceph-mon[31673]: from='mgr.? 172.21.15.40:0/1971098852' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T10:38:32.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:32 smithi070 ceph-mon[36330]: mgrmap e20: x(active, since 11m), standbys: y, z 2024-03-21T10:38:32.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:32 smithi070 ceph-mon[36330]: from='mgr.? 172.21.15.40:0/1971098852' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-21T10:38:32.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:32 smithi070 ceph-mon[36330]: from='mgr.? 172.21.15.40:0/1971098852' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T10:38:32.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:32 smithi070 ceph-mon[36330]: Standby manager daemon y restarted 2024-03-21T10:38:32.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:32 smithi070 ceph-mon[36330]: Standby manager daemon y started 2024-03-21T10:38:32.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:32 smithi070 ceph-mon[36330]: from='mgr.? 172.21.15.40:0/1971098852' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-21T10:38:32.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:32 smithi070 ceph-mon[36330]: from='mgr.? 172.21.15.40:0/1971098852' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T10:38:33.305 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:33 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:33.008+0000 7f462819b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T10:38:33.305 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:33 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:33.147+0000 7f462819b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T10:38:33.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:33 smithi031 ceph-mon[31673]: mgrmap e21: x(active, since 11m), standbys: y, z 2024-03-21T10:38:33.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:33 smithi040 ceph-mon[36154]: mgrmap e21: x(active, since 11m), standbys: y, z 2024-03-21T10:38:33.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:33.518+0000 7f0a94c00640 1 -- 172.21.15.31:0/705533408 <== mon.1 v2:172.21.15.70:3300/0 9 ==== mgrmap(e 22) v1 ==== 137539+0+0 (secure 0 0 0) 0x7f0a840cd290 con 0x7f0a981028e0 2024-03-21T10:38:33.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:33.519+0000 7f0a94c00640 1 -- 172.21.15.31:0/705533408 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0a7c0a2f60 msgr2=0x7f0a7c0a5420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T10:38:33.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:33.519+0000 7f0a94c00640 1 --2- 172.21.15.31:0/705533408 >> [v2:172.21.15.31:6800/3875791078,v1:172.21.15.31:6801/3875791078] conn(0x7f0a7c0a2f60 0x7f0a7c0a5420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:33.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:33 smithi070 ceph-mon[36330]: mgrmap e21: x(active, since 11m), standbys: y, z 2024-03-21T10:38:33.805 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:33 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:33.498+0000 7f462819b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T10:38:34.107 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:33 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:33.847+0000 7f457aa00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-21T10:38:34.107 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:33 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:33.847+0000 7f457aa00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-21T10:38:34.107 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:33 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:33.847+0000 7f457aa00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-21T10:38:34.107 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:33 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:33.847+0000 7f457aa00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-21T10:38:34.107 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:33 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:33.847+0000 7f457aa00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-21T10:38:34.107 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:33 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:33.849+0000 7f457dc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-21T10:38:34.108 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:33 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:33.849+0000 7f457dc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-21T10:38:34.108 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:33 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:33.849+0000 7f457dc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-21T10:38:34.108 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:33 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:33.849+0000 7f457dc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-21T10:38:34.108 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:38:33 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:38:33.849+0000 7f457dc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-21T10:38:34.108 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: Active manager daemon x restarted 2024-03-21T10:38:34.108 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: Activating manager daemon x 2024-03-21T10:38:34.108 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: osdmap e94: 12 total, 12 up, 12 in 2024-03-21T10:38:34.108 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: mgrmap e22: x(active, starting, since 0.0157707s), standbys: y, z 2024-03-21T10:38:34.108 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T10:38:34.108 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:38:34.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:38:34.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-21T10:38:34.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-21T10:38:34.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-21T10:38:34.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-21T10:38:34.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-21T10:38:34.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-21T10:38:34.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-21T10:38:34.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-21T10:38:34.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-21T10:38:34.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-21T10:38:34.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: Active manager daemon x restarted 2024-03-21T10:38:34.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: Activating manager daemon x 2024-03-21T10:38:34.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: osdmap e94: 12 total, 12 up, 12 in 2024-03-21T10:38:34.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: mgrmap e22: x(active, starting, since 0.0157707s), standbys: y, z 2024-03-21T10:38:34.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T10:38:34.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:38:34.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:38:34.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-21T10:38:34.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-21T10:38:34.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-21T10:38:34.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-21T10:38:34.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-21T10:38:34.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-21T10:38:34.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-21T10:38:34.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-21T10:38:34.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-21T10:38:34.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-21T10:38:34.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-21T10:38:34.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-21T10:38:34.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T10:38:34.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T10:38:34.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-21T10:38:34.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T10:38:34.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T10:38:34.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T10:38:34.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T10:38:34.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T10:38:34.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T10:38:34.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T10:38:34.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T10:38:34.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-21T10:38:34.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-21T10:38:34.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-21T10:38:34.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-21T10:38:34.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T10:38:34.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T10:38:34.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T10:38:34.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: Manager daemon x is now available 2024-03-21T10:38:34.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:38:34.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T10:38:34.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T10:38:34.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T10:38:34.434 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:34 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T10:38:34.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.528+0000 7f0a94c00640 1 -- 172.21.15.31:0/705533408 <== mon.1 v2:172.21.15.70:3300/0 10 ==== mgrmap(e 23) v1 ==== 137666+0+0 (secure 0 0 0) 0x7f0a840954a0 con 0x7f0a981028e0 2024-03-21T10:38:34.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.529+0000 7f0a94c00640 1 --2- 172.21.15.31:0/705533408 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f0a7c0687e0 0x7f0a7c068bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:34.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.529+0000 7f0a94c00640 1 -- 172.21.15.31:0/705533408 --> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a840056a0 con 0x7f0a7c0687e0 2024-03-21T10:38:34.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.536+0000 7f0a97e00640 1 --2- 172.21.15.31:0/705533408 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f0a7c0687e0 0x7f0a7c068bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:34.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.542+0000 7f0a97e00640 1 --2- 172.21.15.31:0/705533408 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f0a7c0687e0 0x7f0a7c068bc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0a880058c0 tx=0x7f0a88005f90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:34.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-21T10:38:34.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-21T10:38:34.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T10:38:34.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T10:38:34.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-21T10:38:34.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T10:38:34.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T10:38:34.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T10:38:34.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T10:38:34.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T10:38:34.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T10:38:34.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T10:38:34.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T10:38:34.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-21T10:38:34.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-21T10:38:34.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-21T10:38:34.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-21T10:38:34.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T10:38:34.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T10:38:34.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T10:38:34.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: Manager daemon x is now available 2024-03-21T10:38:34.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:38:34.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T10:38:34.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T10:38:34.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T10:38:34.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:34 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T10:38:34.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: Active manager daemon x restarted 2024-03-21T10:38:34.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: Activating manager daemon x 2024-03-21T10:38:34.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: osdmap e94: 12 total, 12 up, 12 in 2024-03-21T10:38:34.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: mgrmap e22: x(active, starting, since 0.0157707s), standbys: y, z 2024-03-21T10:38:34.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T10:38:34.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T10:38:34.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T10:38:34.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-21T10:38:34.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-21T10:38:34.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-21T10:38:34.575 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-21T10:38:34.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-21T10:38:34.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-21T10:38:34.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-21T10:38:34.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-21T10:38:34.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-21T10:38:34.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-21T10:38:34.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-21T10:38:34.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-21T10:38:34.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T10:38:34.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T10:38:34.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-21T10:38:34.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T10:38:34.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T10:38:34.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T10:38:34.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T10:38:34.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T10:38:34.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T10:38:34.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T10:38:34.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T10:38:34.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-21T10:38:34.577 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-21T10:38:34.578 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-21T10:38:34.578 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-21T10:38:34.578 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T10:38:34.578 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T10:38:34.578 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T10:38:34.578 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: Manager daemon x is now available 2024-03-21T10:38:34.578 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:38:34.578 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T10:38:34.578 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T10:38:34.578 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T10:38:34.578 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:34 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T10:38:34.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.625+0000 7f0a94c00640 1 -- 172.21.15.31:0/705533408 <== mgr.25564 v2:172.21.15.31:6800/3298970029 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f0a840056a0 con 0x7f0a7c0687e0 2024-03-21T10:38:34.623 INFO:teuthology.orchestra.run.smithi031.stdout:Schedule set for path / 2024-03-21T10:38:34.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.627+0000 7f0a9f200640 1 -- 172.21.15.31:0/705533408 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f0a7c0687e0 msgr2=0x7f0a7c068bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:34.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.627+0000 7f0a9f200640 1 --2- 172.21.15.31:0/705533408 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f0a7c0687e0 0x7f0a7c068bc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0a880058c0 tx=0x7f0a88005f90 comp rx=0 tx=0).stop 2024-03-21T10:38:34.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.627+0000 7f0a9f200640 1 -- 172.21.15.31:0/705533408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a981028e0 msgr2=0x7f0a98073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:34.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.627+0000 7f0a9f200640 1 --2- 172.21.15.31:0/705533408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a981028e0 0x7f0a98073a90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0a8400e9f0 tx=0x7f0a8400eeb0 comp rx=0 tx=0).stop 2024-03-21T10:38:34.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.627+0000 7f0a9d200640 1 -- 172.21.15.31:0/705533408 reap_dead start 2024-03-21T10:38:34.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.627+0000 7f0a9f200640 1 -- 172.21.15.31:0/705533408 shutdown_connections 2024-03-21T10:38:34.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.627+0000 7f0a9f200640 1 -- 172.21.15.31:0/705533408 >> 172.21.15.31:0/705533408 conn(0x7f0a980fcc60 msgr2=0x7f0a98105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:34.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.627+0000 7f0a9f200640 1 -- 172.21.15.31:0/705533408 shutdown_connections 2024-03-21T10:38:34.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.627+0000 7f0a9f200640 1 -- 172.21.15.31:0/705533408 wait complete. 2024-03-21T10:38:34.643 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-21T10:38:34.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.826+0000 7fd217200640 1 -- 172.21.15.31:0/127725569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd210101f90 msgr2=0x7fd210104380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:34.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.826+0000 7fd217200640 1 --2- 172.21.15.31:0/127725569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd210101f90 0x7fd210104380 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd204009900 tx=0x7fd20402bc30 comp rx=0 tx=0).stop 2024-03-21T10:38:34.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.826+0000 7fd217200640 1 -- 172.21.15.31:0/127725569 shutdown_connections 2024-03-21T10:38:34.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.826+0000 7fd217200640 1 --2- 172.21.15.31:0/127725569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2101071f0 0x7fd210109640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:34.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.826+0000 7fd217200640 1 --2- 172.21.15.31:0/127725569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2101048c0 0x7fd210106cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:34.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.826+0000 7fd217200640 1 --2- 172.21.15.31:0/127725569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd210101f90 0x7fd210104380 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:34.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.827+0000 7fd217200640 1 -- 172.21.15.31:0/127725569 >> 172.21.15.31:0/127725569 conn(0x7fd2100fb960 msgr2=0x7fd2100fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:34.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.827+0000 7fd217200640 1 -- 172.21.15.31:0/127725569 shutdown_connections 2024-03-21T10:38:34.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.827+0000 7fd217200640 1 -- 172.21.15.31:0/127725569 wait complete. 2024-03-21T10:38:34.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.828+0000 7fd217200640 1 Processor -- start 2024-03-21T10:38:34.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.828+0000 7fd217200640 1 -- start start 2024-03-21T10:38:34.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.828+0000 7fd217200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd210101f90 0x7fd2101999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:34.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.828+0000 7fd217200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2101048c0 0x7fd210199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:34.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.828+0000 7fd217200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2101071f0 0x7fd2101a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:34.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.828+0000 7fd217200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd21010d1a0 con 0x7fd2101048c0 2024-03-21T10:38:34.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.828+0000 7fd217200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd21010d020 con 0x7fd210101f90 2024-03-21T10:38:34.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.828+0000 7fd217200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd21010d320 con 0x7fd2101071f0 2024-03-21T10:38:34.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.828+0000 7fd215e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd210101f90 0x7fd2101999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:34.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.828+0000 7fd215e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd210101f90 0x7fd2101999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33536/0 (socket says 172.21.15.31:33536) 2024-03-21T10:38:34.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.828+0000 7fd215e00640 1 -- 172.21.15.31:0/393326695 learned_addr learned my addr 172.21.15.31:0/393326695 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:34.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.828+0000 7fd216800640 1 --2- 172.21.15.31:0/393326695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2101071f0 0x7fd2101a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:34.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.828+0000 7fd215400640 1 --2- 172.21.15.31:0/393326695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2101048c0 0x7fd210199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:34.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.829+0000 7fd215e00640 1 -- 172.21.15.31:0/393326695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2101071f0 msgr2=0x7fd2101a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:34.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.829+0000 7fd215e00640 1 --2- 172.21.15.31:0/393326695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2101071f0 0x7fd2101a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:34.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.829+0000 7fd215e00640 1 -- 172.21.15.31:0/393326695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2101048c0 msgr2=0x7fd210199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:34.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.829+0000 7fd215e00640 1 --2- 172.21.15.31:0/393326695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2101048c0 0x7fd210199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:34.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.829+0000 7fd215e00640 1 -- 172.21.15.31:0/393326695 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2101a15f0 con 0x7fd210101f90 2024-03-21T10:38:34.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.829+0000 7fd215e00640 1 --2- 172.21.15.31:0/393326695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd210101f90 0x7fd2101999d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd204009230 tx=0x7fd20402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:34.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.829+0000 7fd20aa00640 1 -- 172.21.15.31:0/393326695 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd204041070 con 0x7fd210101f90 2024-03-21T10:38:34.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.829+0000 7fd20aa00640 1 -- 172.21.15.31:0/393326695 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd204004450 con 0x7fd210101f90 2024-03-21T10:38:34.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.829+0000 7fd217200640 1 -- 172.21.15.31:0/393326695 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2101a1880 con 0x7fd210101f90 2024-03-21T10:38:34.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.829+0000 7fd20aa00640 1 -- 172.21.15.31:0/393326695 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd204003e20 con 0x7fd210101f90 2024-03-21T10:38:34.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.830+0000 7fd217200640 1 -- 172.21.15.31:0/393326695 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2101a1d70 con 0x7fd210101f90 2024-03-21T10:38:34.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.831+0000 7fd217200640 1 -- 172.21.15.31:0/393326695 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd210108d40 con 0x7fd210101f90 2024-03-21T10:38:34.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.833+0000 7fd20aa00640 1 -- 172.21.15.31:0/393326695 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 23) v1 ==== 137666+0+0 (secure 0 0 0) 0x7fd204044070 con 0x7fd210101f90 2024-03-21T10:38:34.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.834+0000 7fd20aa00640 1 --2- 172.21.15.31:0/393326695 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd1f80a2d00 0x7fd1f80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:34.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.834+0000 7fd20aa00640 1 -- 172.21.15.31:0/393326695 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fd2040f1090 con 0x7fd210101f90 2024-03-21T10:38:34.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.835+0000 7fd215400640 1 --2- 172.21.15.31:0/393326695 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd1f80a2d00 0x7fd1f80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:34.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.835+0000 7fd215400640 1 --2- 172.21.15.31:0/393326695 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd1f80a2d00 0x7fd1f80a51c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd21019aee0 tx=0x7fd200005f90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:34.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:34.838+0000 7fd20aa00640 1 -- 172.21.15.31:0/393326695 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd2040bdd10 con 0x7fd210101f90 2024-03-21T10:38:35.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.179+0000 7fd217200640 1 -- 172.21.15.31:0/393326695 --> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) v1 -- 0x7fd210061c40 con 0x7fd1f80a2d00 2024-03-21T10:38:35.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.260+0000 7fd20aa00640 1 -- 172.21.15.31:0/393326695 <== mgr.25564 v2:172.21.15.31:6800/3298970029 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fd210061c40 con 0x7fd1f80a2d00 2024-03-21T10:38:35.258 INFO:teuthology.orchestra.run.smithi031.stdout:Retention added to path / 2024-03-21T10:38:35.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.263+0000 7fd217200640 1 -- 172.21.15.31:0/393326695 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd1f80a2d00 msgr2=0x7fd1f80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.263+0000 7fd217200640 1 --2- 172.21.15.31:0/393326695 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd1f80a2d00 0x7fd1f80a51c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd21019aee0 tx=0x7fd200005f90 comp rx=0 tx=0).stop 2024-03-21T10:38:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.263+0000 7fd217200640 1 -- 172.21.15.31:0/393326695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd210101f90 msgr2=0x7fd2101999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.263+0000 7fd217200640 1 --2- 172.21.15.31:0/393326695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd210101f90 0x7fd2101999d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd204009230 tx=0x7fd20402c650 comp rx=0 tx=0).stop 2024-03-21T10:38:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.263+0000 7fd217200640 1 -- 172.21.15.31:0/393326695 shutdown_connections 2024-03-21T10:38:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.263+0000 7fd217200640 1 --2- 172.21.15.31:0/393326695 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd1f80a2d00 0x7fd1f80a51c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.263+0000 7fd217200640 1 --2- 172.21.15.31:0/393326695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2101071f0 0x7fd2101a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.263+0000 7fd217200640 1 --2- 172.21.15.31:0/393326695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2101048c0 0x7fd210199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.263+0000 7fd217200640 1 --2- 172.21.15.31:0/393326695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd210101f90 0x7fd2101999d0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.263+0000 7fd217200640 1 -- 172.21.15.31:0/393326695 >> 172.21.15.31:0/393326695 conn(0x7fd2100fb960 msgr2=0x7fd210101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.263+0000 7fd217200640 1 -- 172.21.15.31:0/393326695 shutdown_connections 2024-03-21T10:38:35.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.263+0000 7fd217200640 1 -- 172.21.15.31:0/393326695 wait complete. 2024-03-21T10:38:35.278 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-21T10:38:35.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.461+0000 7f652c200640 1 -- 172.21.15.31:0/2653578841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6524102250 msgr2=0x7f6524104640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:35.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.461+0000 7f652c200640 1 --2- 172.21.15.31:0/2653578841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6524102250 0x7f6524104640 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6518009900 tx=0x7f651802bc40 comp rx=0 tx=0).stop 2024-03-21T10:38:35.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.461+0000 7f652c200640 1 -- 172.21.15.31:0/2653578841 shutdown_connections 2024-03-21T10:38:35.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.461+0000 7f652c200640 1 --2- 172.21.15.31:0/2653578841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65241074b0 0x7f65241098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:35.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.461+0000 7f652c200640 1 --2- 172.21.15.31:0/2653578841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6524104b80 0x7f6524106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:35.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.461+0000 7f652c200640 1 --2- 172.21.15.31:0/2653578841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6524102250 0x7f6524104640 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:35.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.461+0000 7f652c200640 1 -- 172.21.15.31:0/2653578841 >> 172.21.15.31:0/2653578841 conn(0x7f65240fbb80 msgr2=0x7f65240fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:35.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.461+0000 7f652c200640 1 -- 172.21.15.31:0/2653578841 shutdown_connections 2024-03-21T10:38:35.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.461+0000 7f652c200640 1 -- 172.21.15.31:0/2653578841 wait complete. 2024-03-21T10:38:35.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.462+0000 7f652c200640 1 Processor -- start 2024-03-21T10:38:35.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.462+0000 7f652c200640 1 -- start start 2024-03-21T10:38:35.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.462+0000 7f652c200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6524102250 0x7f6524199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:35.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.462+0000 7f652c200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6524104b80 0x7f652419a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:35.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.462+0000 7f652c200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65241074b0 0x7f65241a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:35.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.462+0000 7f652c200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f652410d3e0 con 0x7f65241074b0 2024-03-21T10:38:35.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.462+0000 7f652c200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f652410d260 con 0x7f6524104b80 2024-03-21T10:38:35.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.462+0000 7f652c200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f652410d560 con 0x7f6524102250 2024-03-21T10:38:35.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.463+0000 7f6529800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6524102250 0x7f6524199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:35.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.463+0000 7f6528e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6524104b80 0x7f652419a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:35.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.463+0000 7f6529800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6524102250 0x7f6524199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60882/0 (socket says 172.21.15.31:60882) 2024-03-21T10:38:35.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.463+0000 7f6529800640 1 -- 172.21.15.31:0/3533867267 learned_addr learned my addr 172.21.15.31:0/3533867267 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:35.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.463+0000 7f652a200640 1 --2- 172.21.15.31:0/3533867267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65241074b0 0x7f65241a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:35.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.463+0000 7f6529800640 1 -- 172.21.15.31:0/3533867267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6524104b80 msgr2=0x7f652419a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:35.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.463+0000 7f6529800640 1 --2- 172.21.15.31:0/3533867267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6524104b80 0x7f652419a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:35.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.463+0000 7f6529800640 1 -- 172.21.15.31:0/3533867267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65241074b0 msgr2=0x7f65241a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:35.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.463+0000 7f6529800640 1 --2- 172.21.15.31:0/3533867267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65241074b0 0x7f65241a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:35.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.463+0000 7f6529800640 1 -- 172.21.15.31:0/3533867267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65241a1860 con 0x7f6524102250 2024-03-21T10:38:35.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.463+0000 7f6528e00640 1 --2- 172.21.15.31:0/3533867267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6524104b80 0x7f652419a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:35.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.464+0000 7f6529800640 1 --2- 172.21.15.31:0/3533867267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6524102250 0x7f6524199c50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f651802c8f0 tx=0x7f6518005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:35.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.464+0000 7f6522000640 1 -- 172.21.15.31:0/3533867267 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6518041070 con 0x7f6524102250 2024-03-21T10:38:35.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.464+0000 7f6522000640 1 -- 172.21.15.31:0/3533867267 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f65180044d0 con 0x7f6524102250 2024-03-21T10:38:35.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.464+0000 7f652c200640 1 -- 172.21.15.31:0/3533867267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65241a1af0 con 0x7f6524102250 2024-03-21T10:38:35.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.464+0000 7f6522000640 1 -- 172.21.15.31:0/3533867267 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6518034440 con 0x7f6524102250 2024-03-21T10:38:35.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.464+0000 7f652c200640 1 -- 172.21.15.31:0/3533867267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65241a1fe0 con 0x7f6524102250 2024-03-21T10:38:35.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.465+0000 7f652c200640 1 -- 172.21.15.31:0/3533867267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6524108fa0 con 0x7f6524102250 2024-03-21T10:38:35.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.467+0000 7f6522000640 1 -- 172.21.15.31:0/3533867267 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) v1 ==== 137666+0+0 (secure 0 0 0) 0x7f6518044070 con 0x7f6524102250 2024-03-21T10:38:35.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.468+0000 7f6522000640 1 --2- 172.21.15.31:0/3533867267 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f650c0a2d00 0x7f650c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:35.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.468+0000 7f6522000640 1 -- 172.21.15.31:0/3533867267 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f65180f1240 con 0x7f6524102250 2024-03-21T10:38:35.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.468+0000 7f6528e00640 1 --2- 172.21.15.31:0/3533867267 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f650c0a2d00 0x7f650c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:35.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.469+0000 7f6528e00640 1 --2- 172.21.15.31:0/3533867267 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f650c0a2d00 0x7f650c0a51c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f652406a750 tx=0x7f6510005f90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:35.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.471+0000 7f6522000640 1 -- 172.21.15.31:0/3533867267 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f65180bd870 con 0x7f6524102250 2024-03-21T10:38:35.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.536+0000 7f6522000640 1 -- 172.21.15.31:0/3533867267 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mgrmap(e 24) v1 ==== 137974+0+0 (secure 0 0 0) 0x7f65180b9e50 con 0x7f6524102250 2024-03-21T10:38:35.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:35 smithi031 ceph-mon[31673]: mgrmap e23: x(active, since 1.02612s), standbys: y, z 2024-03-21T10:38:35.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:35 smithi031 ceph-mon[31673]: from='client.25582 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:38:35.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:35 smithi031 ceph-mon[31673]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:38:35.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.725+0000 7f652c200640 1 -- 172.21.15.31:0/3533867267 --> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f6524061ae0 con 0x7f650c0a2d00 2024-03-21T10:38:35.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.735+0000 7f6522000640 1 -- 172.21.15.31:0/3533867267 <== mgr.25564 v2:172.21.15.31:6800/3298970029 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+298 (secure 0 0 0) 0x7f6524061ae0 con 0x7f650c0a2d00 2024-03-21T10:38:35.733 INFO:teuthology.orchestra.run.smithi031.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-21T00:00:00", "created": "2024-03-21T10:38:34", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-21T10:38:35.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.737+0000 7f652c200640 1 -- 172.21.15.31:0/3533867267 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f650c0a2d00 msgr2=0x7f650c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:35.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.737+0000 7f652c200640 1 --2- 172.21.15.31:0/3533867267 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f650c0a2d00 0x7f650c0a51c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f652406a750 tx=0x7f6510005f90 comp rx=0 tx=0).stop 2024-03-21T10:38:35.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.737+0000 7f652c200640 1 -- 172.21.15.31:0/3533867267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6524102250 msgr2=0x7f6524199c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:35.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.737+0000 7f652c200640 1 --2- 172.21.15.31:0/3533867267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6524102250 0x7f6524199c50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f651802c8f0 tx=0x7f6518005d50 comp rx=0 tx=0).stop 2024-03-21T10:38:35.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.737+0000 7f652c200640 1 -- 172.21.15.31:0/3533867267 shutdown_connections 2024-03-21T10:38:35.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.737+0000 7f652c200640 1 --2- 172.21.15.31:0/3533867267 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f650c0a2d00 0x7f650c0a51c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:35.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.737+0000 7f652c200640 1 --2- 172.21.15.31:0/3533867267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65241074b0 0x7f65241a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:35.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.737+0000 7f652c200640 1 --2- 172.21.15.31:0/3533867267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6524104b80 0x7f652419a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:35.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.737+0000 7f652c200640 1 --2- 172.21.15.31:0/3533867267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6524102250 0x7f6524199c50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:35.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.737+0000 7f652c200640 1 -- 172.21.15.31:0/3533867267 >> 172.21.15.31:0/3533867267 conn(0x7f65240fbb80 msgr2=0x7f6524101980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:35.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.737+0000 7f652c200640 1 -- 172.21.15.31:0/3533867267 shutdown_connections 2024-03-21T10:38:35.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.737+0000 7f652c200640 1 -- 172.21.15.31:0/3533867267 wait complete. 2024-03-21T10:38:35.752 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-21T10:38:35.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:35 smithi070 ceph-mon[36330]: mgrmap e23: x(active, since 1.02612s), standbys: y, z 2024-03-21T10:38:35.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:35 smithi070 ceph-mon[36330]: from='client.25582 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:38:35.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:35 smithi070 ceph-mon[36330]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:38:35.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:35 smithi040 ceph-mon[36154]: mgrmap e23: x(active, since 1.02612s), standbys: y, z 2024-03-21T10:38:35.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:35 smithi040 ceph-mon[36154]: from='client.25582 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:38:35.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:35 smithi040 ceph-mon[36154]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:38:35.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.952+0000 7f2a3dc00640 1 -- 172.21.15.31:0/3851611704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a38102700 msgr2=0x7f2a38102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:35.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.952+0000 7f2a3dc00640 1 --2- 172.21.15.31:0/3851611704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a38102700 0x7f2a38102b80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2a20009900 tx=0x7f2a2002bc50 comp rx=0 tx=0).stop 2024-03-21T10:38:35.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.952+0000 7f2a3dc00640 1 -- 172.21.15.31:0/3851611704 shutdown_connections 2024-03-21T10:38:35.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.952+0000 7f2a3dc00640 1 --2- 172.21.15.31:0/3851611704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a381030c0 0x7f2a38109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:35.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.952+0000 7f2a3dc00640 1 --2- 172.21.15.31:0/3851611704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a38102700 0x7f2a38102b80 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:35.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.952+0000 7f2a3dc00640 1 --2- 172.21.15.31:0/3851611704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a38101510 0x7f2a38101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:35.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.952+0000 7f2a3dc00640 1 -- 172.21.15.31:0/3851611704 >> 172.21.15.31:0/3851611704 conn(0x7f2a380fca80 msgr2=0x7f2a380feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:35.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.952+0000 7f2a3dc00640 1 -- 172.21.15.31:0/3851611704 shutdown_connections 2024-03-21T10:38:35.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.953+0000 7f2a3dc00640 1 -- 172.21.15.31:0/3851611704 wait complete. 2024-03-21T10:38:35.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.953+0000 7f2a3dc00640 1 Processor -- start 2024-03-21T10:38:35.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.954+0000 7f2a3dc00640 1 -- start start 2024-03-21T10:38:35.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.954+0000 7f2a3dc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a38101510 0x7f2a38075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:35.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.954+0000 7f2a3dc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a38102700 0x7f2a38073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:35.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.954+0000 7f2a3dc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a381030c0 0x7f2a38073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:35.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.954+0000 7f2a3dc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3810d1f0 con 0x7f2a38101510 2024-03-21T10:38:35.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.954+0000 7f2a3dc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3810d070 con 0x7f2a381030c0 2024-03-21T10:38:35.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.954+0000 7f2a3dc00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3810d370 con 0x7f2a38102700 2024-03-21T10:38:35.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.954+0000 7f2a37e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a38101510 0x7f2a38075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:35.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.954+0000 7f2a37400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a38102700 0x7f2a38073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:35.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.954+0000 7f2a37e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a38101510 0x7f2a38075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53466/0 (socket says 172.21.15.31:53466) 2024-03-21T10:38:35.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.954+0000 7f2a37e00640 1 -- 172.21.15.31:0/3277571946 learned_addr learned my addr 172.21.15.31:0/3277571946 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:35.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.954+0000 7f2a37400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a38102700 0x7f2a38073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60908/0 (socket says 172.21.15.31:60908) 2024-03-21T10:38:35.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.954+0000 7f2a3d200640 1 --2- 172.21.15.31:0/3277571946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a381030c0 0x7f2a38073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:35.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.955+0000 7f2a37400640 1 -- 172.21.15.31:0/3277571946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a381030c0 msgr2=0x7f2a38073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:35.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.955+0000 7f2a37400640 1 --2- 172.21.15.31:0/3277571946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a381030c0 0x7f2a38073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:35.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.955+0000 7f2a37400640 1 -- 172.21.15.31:0/3277571946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a38101510 msgr2=0x7f2a38075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:35.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.955+0000 7f2a37400640 1 --2- 172.21.15.31:0/3277571946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a38101510 0x7f2a38075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:35.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.955+0000 7f2a37400640 1 -- 172.21.15.31:0/3277571946 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a38074860 con 0x7f2a38102700 2024-03-21T10:38:35.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.955+0000 7f2a37e00640 1 --2- 172.21.15.31:0/3277571946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a38101510 0x7f2a38075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:35.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.955+0000 7f2a37400640 1 --2- 172.21.15.31:0/3277571946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a38102700 0x7f2a38073a90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2a20036040 tx=0x7f2a2002c670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:35.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.955+0000 7f2a34c00640 1 -- 172.21.15.31:0/3277571946 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a20041070 con 0x7f2a38102700 2024-03-21T10:38:35.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.955+0000 7f2a34c00640 1 -- 172.21.15.31:0/3277571946 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2a20004450 con 0x7f2a38102700 2024-03-21T10:38:35.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.955+0000 7f2a34c00640 1 -- 172.21.15.31:0/3277571946 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a20003e60 con 0x7f2a38102700 2024-03-21T10:38:35.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.956+0000 7f2a3dc00640 1 -- 172.21.15.31:0/3277571946 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a381a1920 con 0x7f2a38102700 2024-03-21T10:38:35.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.956+0000 7f2a3dc00640 1 -- 172.21.15.31:0/3277571946 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a381a1df0 con 0x7f2a38102700 2024-03-21T10:38:35.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.957+0000 7f2a3dc00640 1 -- 172.21.15.31:0/3277571946 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a38067f10 con 0x7f2a38102700 2024-03-21T10:38:35.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.960+0000 7f2a34c00640 1 -- 172.21.15.31:0/3277571946 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 24) v1 ==== 137974+0+0 (secure 0 0 0) 0x7f2a20044070 con 0x7f2a38102700 2024-03-21T10:38:35.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.961+0000 7f2a34c00640 1 --2- 172.21.15.31:0/3277571946 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f2a1c0a30f0 0x7f2a1c0a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:35.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.962+0000 7f2a34c00640 1 -- 172.21.15.31:0/3277571946 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f2a200f1440 con 0x7f2a38102700 2024-03-21T10:38:35.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.962+0000 7f2a37e00640 1 --2- 172.21.15.31:0/3277571946 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f2a1c0a30f0 0x7f2a1c0a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:35.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.962+0000 7f2a37e00640 1 --2- 172.21.15.31:0/3277571946 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f2a1c0a30f0 0x7f2a1c0a55b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2a280058c0 tx=0x7f2a28005f90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:35.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:35.970+0000 7f2a34c00640 1 -- 172.21.15.31:0/3277571946 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2a200bd650 con 0x7f2a38102700 2024-03-21T10:38:36.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:36.221+0000 7f2a3dc00640 1 -- 172.21.15.31:0/3277571946 --> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f2a38061ae0 con 0x7f2a1c0a30f0 2024-03-21T10:38:36.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:36.231+0000 7f2a34c00640 1 -- 172.21.15.31:0/3277571946 <== mgr.25564 v2:172.21.15.31:6800/3298970029 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f2a38061ae0 con 0x7f2a1c0a30f0 2024-03-21T10:38:36.229 INFO:teuthology.orchestra.run.smithi031.stdout:/ 1m 6m3h 2024-03-21T10:38:36.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:36.234+0000 7f2a3dc00640 1 -- 172.21.15.31:0/3277571946 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f2a1c0a30f0 msgr2=0x7f2a1c0a55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:36.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:36.234+0000 7f2a3dc00640 1 --2- 172.21.15.31:0/3277571946 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f2a1c0a30f0 0x7f2a1c0a55b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2a280058c0 tx=0x7f2a28005f90 comp rx=0 tx=0).stop 2024-03-21T10:38:36.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:36.234+0000 7f2a3dc00640 1 -- 172.21.15.31:0/3277571946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a38102700 msgr2=0x7f2a38073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:36.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:36.234+0000 7f2a3dc00640 1 --2- 172.21.15.31:0/3277571946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a38102700 0x7f2a38073a90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2a20036040 tx=0x7f2a2002c670 comp rx=0 tx=0).stop 2024-03-21T10:38:36.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:36.234+0000 7f2a3dc00640 1 -- 172.21.15.31:0/3277571946 shutdown_connections 2024-03-21T10:38:36.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:36.235+0000 7f2a3dc00640 1 --2- 172.21.15.31:0/3277571946 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f2a1c0a30f0 0x7f2a1c0a55b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:36.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:36.235+0000 7f2a3dc00640 1 --2- 172.21.15.31:0/3277571946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a381030c0 0x7f2a38073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:36.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:36.235+0000 7f2a3dc00640 1 --2- 172.21.15.31:0/3277571946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a38102700 0x7f2a38073a90 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:36.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:36.235+0000 7f2a3dc00640 1 --2- 172.21.15.31:0/3277571946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a38101510 0x7f2a38075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:36.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:36.235+0000 7f2a3dc00640 1 -- 172.21.15.31:0/3277571946 >> 172.21.15.31:0/3277571946 conn(0x7f2a380fca80 msgr2=0x7f2a38105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:36.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:36.235+0000 7f2a3dc00640 1 -- 172.21.15.31:0/3277571946 shutdown_connections 2024-03-21T10:38:36.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:36.235+0000 7f2a3dc00640 1 -- 172.21.15.31:0/3277571946 wait complete. 2024-03-21T10:38:36.256 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-21T10:38:36.336 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-21T10:38:36.348 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-21T10:38:36.373 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2024-03-21T10:38:36.373 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-21T10:38:36.378 INFO:teuthology.run_tasks:Running task workunit... 2024-03-21T10:38:36.392 INFO:tasks.workunit:Pulling workunits from ref 2edfee68ad7243bed422a049b6f5f58619300f11 2024-03-21T10:38:36.393 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-21T10:38:36.393 INFO:tasks.workunit:timeout=3h 2024-03-21T10:38:36.393 INFO:tasks.workunit:cleanup=True 2024-03-21T10:38:36.393 DEBUG:teuthology.orchestra.run.smithi031:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-21T10:38:36.433 INFO:teuthology.orchestra.run.smithi031.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-21T10:38:36.434 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-21T10:38:36.434 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 76h/118d Inode: 1099511628283 Links: 2 2024-03-21T10:38:36.434 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T10:38:36.434 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-21T10:38:36.434 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-03-21 10:35:43.079786104 +0000 2024-03-21T10:38:36.434 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-03-21 10:35:43.079786104 +0000 2024-03-21T10:38:36.434 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-03-21 10:38:11.443449622 +0000 2024-03-21T10:38:36.434 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-03-21 10:35:43.079786104 +0000 2024-03-21T10:38:36.434 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-21T10:38:36.435 DEBUG:teuthology.orchestra.run.smithi031:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-21T10:38:36.530 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:36 smithi031 ceph-mon[31673]: [21/Mar/2024:10:38:35] ENGINE Bus STARTING 2024-03-21T10:38:36.531 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:36 smithi031 ceph-mon[31673]: from='client.25637 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:38:36.531 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:36 smithi031 ceph-mon[31673]: [21/Mar/2024:10:38:35] ENGINE Serving on http://172.21.15.31:8765 2024-03-21T10:38:36.531 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:36 smithi031 ceph-mon[31673]: [21/Mar/2024:10:38:35] ENGINE Serving on https://172.21.15.31:7150 2024-03-21T10:38:36.531 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:36 smithi031 ceph-mon[31673]: [21/Mar/2024:10:38:35] ENGINE Bus STARTED 2024-03-21T10:38:36.531 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:36 smithi031 ceph-mon[31673]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:38:36.531 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:36 smithi031 ceph-mon[31673]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-03-21T10:38:36.569 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 2edfee68ad7243bed422a049b6f5f58619300f11 2024-03-21T10:38:36.611 INFO:tasks.workunit.client.0.smithi031.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-21T10:38:36.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:36 smithi031 ceph-mon[31673]: from='client.25630 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:38:36.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:36 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:36.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:36 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:36.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:36 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:36.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:36 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:36.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:36 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:38:36.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:36 smithi070 ceph-mon[36330]: [21/Mar/2024:10:38:35] ENGINE Bus STARTING 2024-03-21T10:38:36.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:36 smithi070 ceph-mon[36330]: from='client.25637 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:38:36.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:36 smithi070 ceph-mon[36330]: [21/Mar/2024:10:38:35] ENGINE Serving on http://172.21.15.31:8765 2024-03-21T10:38:36.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:36 smithi070 ceph-mon[36330]: [21/Mar/2024:10:38:35] ENGINE Serving on https://172.21.15.31:7150 2024-03-21T10:38:36.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:36 smithi070 ceph-mon[36330]: [21/Mar/2024:10:38:35] ENGINE Bus STARTED 2024-03-21T10:38:36.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:36 smithi070 ceph-mon[36330]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:38:36.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:36 smithi070 ceph-mon[36330]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-03-21T10:38:36.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:36 smithi070 ceph-mon[36330]: from='client.25630 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:38:36.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:36 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:36.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:36 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:36.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:36 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:36.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:36 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:36.824 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:36 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:38:36.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:36 smithi040 ceph-mon[36154]: [21/Mar/2024:10:38:35] ENGINE Bus STARTING 2024-03-21T10:38:36.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:36 smithi040 ceph-mon[36154]: from='client.25637 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:38:36.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:36 smithi040 ceph-mon[36154]: [21/Mar/2024:10:38:35] ENGINE Serving on http://172.21.15.31:8765 2024-03-21T10:38:36.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:36 smithi040 ceph-mon[36154]: [21/Mar/2024:10:38:35] ENGINE Serving on https://172.21.15.31:7150 2024-03-21T10:38:36.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:36 smithi040 ceph-mon[36154]: [21/Mar/2024:10:38:35] ENGINE Bus STARTED 2024-03-21T10:38:36.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:36 smithi040 ceph-mon[36154]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:38:36.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:36 smithi040 ceph-mon[36154]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-03-21T10:38:36.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:36 smithi040 ceph-mon[36154]: from='client.25630 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:38:36.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:36 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:36.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:36 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:36.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:36 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:36.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:36 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:36.932 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:36 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:38:38.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:37 smithi031 ceph-mon[31673]: from='client.25642 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:38:38.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:37 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:38.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:37 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:38.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:37 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:38.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:37 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:38.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:37 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:37 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:37 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:37 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:37 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:37 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:37 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:38.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:37 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:37 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:37 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:38.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:37 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:37 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:37 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:37 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:37 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:37 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:37 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:37 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:37 smithi070 ceph-mon[36330]: from='client.25642 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:38:38.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:37 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:38.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:37 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:38.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:37 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:38.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:37 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:38.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:37 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:37 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:37 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:37 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:37 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:37 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:37 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:38.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:37 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:37 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:37 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:38.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:37 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:37 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:37 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:37 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:37 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:37 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:37 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:37 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:37 smithi040 ceph-mon[36154]: from='client.25642 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:38:38.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:37 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:38.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:37 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:38.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:37 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:38.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:37 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:38.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:37 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:37 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:37 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:37 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:37 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:37 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:37 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:38.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:37 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:37 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:37 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:38.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:37 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:37 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:37 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:37 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:37 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:37 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:37 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:38.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:37 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:39.226 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:38 smithi070 ceph-mon[36330]: Adjusting osd_memory_target on smithi040 to 175.2M 2024-03-21T10:38:39.226 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:38 smithi070 ceph-mon[36330]: Unable to set osd_memory_target on smithi040 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-21T10:38:39.226 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:38 smithi070 ceph-mon[36330]: Adjusting osd_memory_target on smithi070 to 174.1M 2024-03-21T10:38:39.226 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:38 smithi070 ceph-mon[36330]: Unable to set osd_memory_target on smithi070 to 182619136: error parsing value: Value '182619136' is below minimum 939524096 2024-03-21T10:38:39.226 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:38 smithi070 ceph-mon[36330]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:38:39.226 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:38 smithi070 ceph-mon[36330]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-03-21T10:38:39.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:38 smithi031 ceph-mon[31673]: Adjusting osd_memory_target on smithi040 to 175.2M 2024-03-21T10:38:39.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:38 smithi031 ceph-mon[31673]: Unable to set osd_memory_target on smithi040 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-21T10:38:39.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:38 smithi031 ceph-mon[31673]: Adjusting osd_memory_target on smithi070 to 174.1M 2024-03-21T10:38:39.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:38 smithi031 ceph-mon[31673]: Unable to set osd_memory_target on smithi070 to 182619136: error parsing value: Value '182619136' is below minimum 939524096 2024-03-21T10:38:39.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:38 smithi031 ceph-mon[31673]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:38:39.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:38 smithi031 ceph-mon[31673]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-03-21T10:38:39.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:38 smithi040 ceph-mon[36154]: Adjusting osd_memory_target on smithi040 to 175.2M 2024-03-21T10:38:39.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:38 smithi040 ceph-mon[36154]: Unable to set osd_memory_target on smithi040 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-21T10:38:39.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:38 smithi040 ceph-mon[36154]: Adjusting osd_memory_target on smithi070 to 174.1M 2024-03-21T10:38:39.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:38 smithi040 ceph-mon[36154]: Unable to set osd_memory_target on smithi070 to 182619136: error parsing value: Value '182619136' is below minimum 939524096 2024-03-21T10:38:39.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:38 smithi040 ceph-mon[36154]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:38:39.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:38 smithi040 ceph-mon[36154]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-03-21T10:38:40.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:40 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:40.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:40 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:40.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:40 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:40.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:40 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:40.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:40 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:40 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:40 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:40 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:40 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:40 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:40 smithi031 ceph-mon[31673]: Adjusting osd_memory_target on smithi031 to 177.8M 2024-03-21T10:38:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:40 smithi031 ceph-mon[31673]: Unable to set osd_memory_target on smithi031 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-21T10:38:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:40 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:38:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:40 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:38:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:40 smithi031 ceph-mon[31673]: Updating smithi031:/etc/ceph/ceph.conf 2024-03-21T10:38:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:40 smithi031 ceph-mon[31673]: Updating smithi040:/etc/ceph/ceph.conf 2024-03-21T10:38:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:40 smithi031 ceph-mon[31673]: Updating smithi070:/etc/ceph/ceph.conf 2024-03-21T10:38:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:40 smithi031 ceph-mon[31673]: Updating smithi040:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T10:38:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:40 smithi031 ceph-mon[31673]: Updating smithi070:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T10:38:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:40 smithi031 ceph-mon[31673]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:38:40.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:40 smithi031 ceph-mon[31673]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-03-21T10:38:40.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:40 smithi031 ceph-mon[31673]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-03-21T10:38:40.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:40 smithi031 ceph-mon[31673]: Updating smithi031:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T10:38:40.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:40 smithi031 ceph-mon[31673]: Updating smithi070:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T10:38:40.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:40 smithi031 ceph-mon[31673]: Updating smithi040:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T10:38:40.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:40 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:40.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:40 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:40.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:40 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:40.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:40 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:40.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:40 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:40.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:40 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:40.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:40 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:40.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:40 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:40.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:40 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:40.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:40 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:40.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:40 smithi070 ceph-mon[36330]: Adjusting osd_memory_target on smithi031 to 177.8M 2024-03-21T10:38:40.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:40 smithi070 ceph-mon[36330]: Unable to set osd_memory_target on smithi031 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-21T10:38:40.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:40 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:38:40.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:40 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:38:40.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:40 smithi070 ceph-mon[36330]: Updating smithi031:/etc/ceph/ceph.conf 2024-03-21T10:38:40.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:40 smithi070 ceph-mon[36330]: Updating smithi040:/etc/ceph/ceph.conf 2024-03-21T10:38:40.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:40 smithi070 ceph-mon[36330]: Updating smithi070:/etc/ceph/ceph.conf 2024-03-21T10:38:40.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:40 smithi070 ceph-mon[36330]: Updating smithi040:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T10:38:40.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:40 smithi070 ceph-mon[36330]: Updating smithi070:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T10:38:40.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:40 smithi070 ceph-mon[36330]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:38:40.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:40 smithi070 ceph-mon[36330]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-03-21T10:38:40.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:40 smithi070 ceph-mon[36330]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-03-21T10:38:40.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:40 smithi070 ceph-mon[36330]: Updating smithi031:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T10:38:40.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:40 smithi070 ceph-mon[36330]: Updating smithi070:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T10:38:40.325 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:40 smithi070 ceph-mon[36330]: Updating smithi040:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T10:38:40.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:40 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:40.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:40 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:40.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:40 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:40.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:40 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:40.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:40 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:40.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:40 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:40.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:40 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:40.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:40 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:40.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:40 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:40.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:40 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:38:40.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:40 smithi040 ceph-mon[36154]: Adjusting osd_memory_target on smithi031 to 177.8M 2024-03-21T10:38:40.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:40 smithi040 ceph-mon[36154]: Unable to set osd_memory_target on smithi031 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-21T10:38:40.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:40 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:38:40.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:40 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:38:40.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:40 smithi040 ceph-mon[36154]: Updating smithi031:/etc/ceph/ceph.conf 2024-03-21T10:38:40.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:40 smithi040 ceph-mon[36154]: Updating smithi040:/etc/ceph/ceph.conf 2024-03-21T10:38:40.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:40 smithi040 ceph-mon[36154]: Updating smithi070:/etc/ceph/ceph.conf 2024-03-21T10:38:40.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:40 smithi040 ceph-mon[36154]: Updating smithi040:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T10:38:40.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:40 smithi040 ceph-mon[36154]: Updating smithi070:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T10:38:40.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:40 smithi040 ceph-mon[36154]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-21T10:38:40.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:40 smithi040 ceph-mon[36154]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-03-21T10:38:40.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:40 smithi040 ceph-mon[36154]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-03-21T10:38:40.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:40 smithi040 ceph-mon[36154]: Updating smithi031:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T10:38:40.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:40 smithi040 ceph-mon[36154]: Updating smithi070:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T10:38:40.433 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:40 smithi040 ceph-mon[36154]: Updating smithi040:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T10:38:41.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:41 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:41.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:41 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:41.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:41 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:41.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:41 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:41.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:41 smithi031 ceph-mon[31673]: Updating smithi031:/etc/ceph/ceph.client.admin.keyring 2024-03-21T10:38:41.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:41 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:41.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:41 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:41.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:41 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:41.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:41 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:41.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:41 smithi070 ceph-mon[36330]: Updating smithi031:/etc/ceph/ceph.client.admin.keyring 2024-03-21T10:38:41.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:41 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:41.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:41 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:41.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:41 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:41.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:41 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:41.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:41 smithi040 ceph-mon[36154]: Updating smithi031:/etc/ceph/ceph.client.admin.keyring 2024-03-21T10:38:43.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:42 smithi040 ceph-mon[36154]: Updating smithi031:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T10:38:43.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:42 smithi040 ceph-mon[36154]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 76 op/s 2024-03-21T10:38:43.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:42 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:43.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:42 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:43.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:42 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:43.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:42 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:38:43.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:42 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:38:43.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:42 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:38:43.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:42 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:43.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:42 smithi031 ceph-mon[31673]: Updating smithi031:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T10:38:43.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:42 smithi031 ceph-mon[31673]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 76 op/s 2024-03-21T10:38:43.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:42 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:43.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:42 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:43.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:42 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:43.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:42 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:38:43.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:42 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:38:43.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:42 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:38:43.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:42 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:43.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:42 smithi070 ceph-mon[36330]: Updating smithi031:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T10:38:43.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:42 smithi070 ceph-mon[36330]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 76 op/s 2024-03-21T10:38:43.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:42 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:43.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:42 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:43.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:42 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:43.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:42 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:38:43.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:42 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:38:43.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:42 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:38:43.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:42 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:38:45.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:44 smithi040 ceph-mon[36154]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 59 op/s 2024-03-21T10:38:45.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:44 smithi031 ceph-mon[31673]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 59 op/s 2024-03-21T10:38:45.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:44 smithi070 ceph-mon[36330]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 59 op/s 2024-03-21T10:38:47.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:46 smithi040 ceph-mon[36154]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-21T10:38:47.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:46 smithi031 ceph-mon[31673]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-21T10:38:47.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:46 smithi070 ceph-mon[36330]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-21T10:38:49.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:48 smithi040 ceph-mon[36154]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-21T10:38:49.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:48 smithi031 ceph-mon[31673]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-21T10:38:49.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:48 smithi070 ceph-mon[36330]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-21T10:38:49.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.638+0000 7f69ad600640 1 -- 172.21.15.31:0/3804192547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69a8075bd0 msgr2=0x7f69a8075fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:49.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.638+0000 7f69ad600640 1 --2- 172.21.15.31:0/3804192547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69a8075bd0 0x7f69a8075fb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6998009930 tx=0x7f699802bc20 comp rx=0 tx=0).stop 2024-03-21T10:38:49.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.638+0000 7f69ad600640 1 -- 172.21.15.31:0/3804192547 shutdown_connections 2024-03-21T10:38:49.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.638+0000 7f69ad600640 1 --2- 172.21.15.31:0/3804192547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69a8112520 0x7f69a8114910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.638+0000 7f69ad600640 1 --2- 172.21.15.31:0/3804192547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69a80764f0 0x7f69a8111fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.638+0000 7f69ad600640 1 --2- 172.21.15.31:0/3804192547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69a8075bd0 0x7f69a8075fb0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.638+0000 7f69ad600640 1 -- 172.21.15.31:0/3804192547 >> 172.21.15.31:0/3804192547 conn(0x7f69a80ff390 msgr2=0x7f69a81017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:49.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.638+0000 7f69ad600640 1 -- 172.21.15.31:0/3804192547 shutdown_connections 2024-03-21T10:38:49.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.638+0000 7f69ad600640 1 -- 172.21.15.31:0/3804192547 wait complete. 2024-03-21T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.639+0000 7f69ad600640 1 Processor -- start 2024-03-21T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.639+0000 7f69ad600640 1 -- start start 2024-03-21T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.640+0000 7f69ad600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69a8075bd0 0x7f69a819e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.640+0000 7f69ad600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69a80764f0 0x7f69a819eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.640+0000 7f69ad600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69a8112520 0x7f69a81a2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.640+0000 7f69ad600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a8118080 con 0x7f69a80764f0 2024-03-21T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.640+0000 7f69ad600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a8117f00 con 0x7f69a8112520 2024-03-21T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.640+0000 7f69ad600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a8118200 con 0x7f69a8075bd0 2024-03-21T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.640+0000 7f69a6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69a8075bd0 0x7f69a819e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.640+0000 7f69a6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69a8075bd0 0x7f69a819e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38278/0 (socket says 172.21.15.31:38278) 2024-03-21T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.640+0000 7f69a6a00640 1 -- 172.21.15.31:0/3986203265 learned_addr learned my addr 172.21.15.31:0/3986203265 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.640+0000 7f69a7400640 1 --2- 172.21.15.31:0/3986203265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69a8112520 0x7f69a81a2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.640+0000 7f69a6000640 1 --2- 172.21.15.31:0/3986203265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69a80764f0 0x7f69a819eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.640+0000 7f69a6a00640 1 -- 172.21.15.31:0/3986203265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69a8112520 msgr2=0x7f69a81a2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.640+0000 7f69a6a00640 1 --2- 172.21.15.31:0/3986203265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69a8112520 0x7f69a81a2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.640+0000 7f69a6a00640 1 -- 172.21.15.31:0/3986203265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69a80764f0 msgr2=0x7f69a819eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.640+0000 7f69a6a00640 1 --2- 172.21.15.31:0/3986203265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69a80764f0 0x7f69a819eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.641+0000 7f69a6a00640 1 -- 172.21.15.31:0/3986203265 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a81a3610 con 0x7f69a8075bd0 2024-03-21T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.641+0000 7f69a7400640 1 --2- 172.21.15.31:0/3986203265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69a8112520 0x7f69a81a2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.641+0000 7f69a6a00640 1 --2- 172.21.15.31:0/3986203265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69a8075bd0 0x7f69a819e620 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6998009430 tx=0x7f699802cb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.641+0000 7f699f400640 1 -- 172.21.15.31:0/3986203265 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6998041070 con 0x7f69a8075bd0 2024-03-21T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.641+0000 7f699f400640 1 -- 172.21.15.31:0/3986203265 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6998004450 con 0x7f69a8075bd0 2024-03-21T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.641+0000 7f69ad600640 1 -- 172.21.15.31:0/3986203265 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a81a38a0 con 0x7f69a8075bd0 2024-03-21T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.641+0000 7f699f400640 1 -- 172.21.15.31:0/3986203265 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6998003d00 con 0x7f69a8075bd0 2024-03-21T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.641+0000 7f69ad600640 1 -- 172.21.15.31:0/3986203265 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a81ab190 con 0x7f69a8075bd0 2024-03-21T10:38:49.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.646+0000 7f69b0247740 1 Processor -- start 2024-03-21T10:38:49.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.646+0000 7f69b0247740 1 -- start start 2024-03-21T10:38:49.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.646+0000 7f699f400640 1 -- 172.21.15.31:0/3986203265 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) v1 ==== 138030+0+0 (secure 0 0 0) 0x7f6998044070 con 0x7f69a8075bd0 2024-03-21T10:38:49.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.646+0000 7f69b0247740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f6625dfd0 0x560f6625e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:49.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.646+0000 7f69b0247740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f6625e8f0 0x560f66267180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.646+0000 7f69b0247740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f662676c0 0x560f66269ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.646+0000 7f69b0247740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f662203b0 con 0x560f6625dfd0 2024-03-21T10:38:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.646+0000 7f69b0247740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f661c2830 con 0x560f6625e8f0 2024-03-21T10:38:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.646+0000 7f69b0247740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f6616b5d0 con 0x560f662676c0 2024-03-21T10:38:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.647+0000 7f69a6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f6625e8f0 0x560f66267180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.647+0000 7f69a7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f6625dfd0 0x560f6625e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.647+0000 7f69a6000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f662676c0 0x560f66269ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.647+0000 7f69a7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f6625dfd0 0x560f6625e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33806/0 (socket says 172.21.15.31:33806) 2024-03-21T10:38:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.647+0000 7f69a6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f6625e8f0 0x560f66267180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42270/0 (socket says 172.21.15.31:42270) 2024-03-21T10:38:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.647+0000 7f69a7400640 1 -- 172.21.15.31:0/376358445 learned_addr learned my addr 172.21.15.31:0/376358445 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.647+0000 7f69a6000640 1 -- 172.21.15.31:0/376358445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f6625e8f0 msgr2=0x560f66267180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.647+0000 7f69a6000640 1 --2- 172.21.15.31:0/376358445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f6625e8f0 0x560f66267180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.647+0000 7f69a6000640 1 -- 172.21.15.31:0/376358445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f6625dfd0 msgr2=0x560f6625e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.647+0000 7f69a6000640 1 --2- 172.21.15.31:0/376358445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f6625dfd0 0x560f6625e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.647+0000 7f69a6000640 1 -- 172.21.15.31:0/376358445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f661832c0 con 0x560f662676c0 2024-03-21T10:38:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.647+0000 7f69a6000640 1 --2- 172.21.15.31:0/376358445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f662676c0 0x560f66269ab0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6994009b40 tx=0x7f69940113e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d9b0a36050c0cb server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:49.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.647+0000 7f6993400640 1 -- 172.21.15.31:0/376358445 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6994011e10 con 0x560f662676c0 2024-03-21T10:38:49.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.648+0000 7f6993400640 1 -- 172.21.15.31:0/376358445 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6994012930 con 0x560f662676c0 2024-03-21T10:38:49.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.648+0000 7f6993400640 1 -- 172.21.15.31:0/376358445 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699400fa50 con 0x560f662676c0 2024-03-21T10:38:49.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.648+0000 7f699f400640 1 --2- 172.21.15.31:0/3986203265 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f698c0a3190 0x7f698c0a5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:49.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.648+0000 7f699f400640 1 -- 172.21.15.31:0/3986203265 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f69980f1620 con 0x7f69a8075bd0 2024-03-21T10:38:49.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.648+0000 7f69a7400640 1 --2- 172.21.15.31:0/3986203265 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f698c0a3190 0x7f698c0a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:49.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.648+0000 7f69b0247740 1 -- 172.21.15.31:0/376358445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f662676c0 msgr2=0x560f66269ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:49.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.648+0000 7f69b0247740 1 --2- 172.21.15.31:0/376358445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f662676c0 0x560f66269ab0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6994009b40 tx=0x7f69940113e0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.648+0000 7f69b0247740 1 -- 172.21.15.31:0/376358445 shutdown_connections 2024-03-21T10:38:49.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.648+0000 7f69b0247740 1 --2- 172.21.15.31:0/376358445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f662676c0 0x560f66269ab0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.648+0000 7f69b0247740 1 --2- 172.21.15.31:0/376358445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f6625e8f0 0x560f66267180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.648+0000 7f69b0247740 1 --2- 172.21.15.31:0/376358445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f6625dfd0 0x560f6625e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.648+0000 7f69b0247740 1 -- 172.21.15.31:0/376358445 >> 172.21.15.31:0/376358445 conn(0x560f662588f0 msgr2=0x560f66258cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:49.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.649+0000 7f69b0247740 1 -- 172.21.15.31:0/376358445 shutdown_connections 2024-03-21T10:38:49.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.649+0000 7f69b0247740 1 -- 172.21.15.31:0/376358445 wait complete. 2024-03-21T10:38:49.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.649+0000 7f69a7400640 1 --2- 172.21.15.31:0/3986203265 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f698c0a3190 0x7f698c0a5650 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f69a00129a0 tx=0x7f69a0009500 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:49.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.649+0000 7f69b0247740 1 -- start start 2024-03-21T10:38:49.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.649+0000 7f69b0247740 1 Processor -- start 2024-03-21T10:38:49.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.650+0000 7f69b0247740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f6625dfd0 0x560f662646a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:49.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.650+0000 7f69b0247740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f6625e8f0 0x560f66264be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:49.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.650+0000 7f69b0247740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f662676c0 0x560f66265120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:49.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.650+0000 7f69b0247740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f6617ea20 con 0x560f662676c0 2024-03-21T10:38:49.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.650+0000 7f69b0247740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f661be8d0 con 0x560f6625e8f0 2024-03-21T10:38:49.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.650+0000 7f69b0247740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f6626a2f0 con 0x560f6625dfd0 2024-03-21T10:38:49.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.650+0000 7f69b0247740 10 client.0 _ll_register_callbacks cb 0x560f661bb1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:38:49.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.650+0000 7f69a6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f6625e8f0 0x560f66264be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:49.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.650+0000 7f69a6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f6625dfd0 0x560f662646a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:49.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.650+0000 7f69a6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f6625e8f0 0x560f66264be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42278/0 (socket says 172.21.15.31:42278) 2024-03-21T10:38:49.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.650+0000 7f69a6000640 1 -- 172.21.15.31:0/1109330327 learned_addr learned my addr 172.21.15.31:0/1109330327 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:49.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.650+0000 7f69a7400640 1 --2- 172.21.15.31:0/1109330327 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f662676c0 0x560f66265120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:49.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.651+0000 7f69a6a00640 1 -- 172.21.15.31:0/1109330327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f6625e8f0 msgr2=0x560f66264be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:49.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.651+0000 7f69a6a00640 1 --2- 172.21.15.31:0/1109330327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f6625e8f0 0x560f66264be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.651+0000 7f69a6a00640 1 -- 172.21.15.31:0/1109330327 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f662676c0 msgr2=0x560f66265120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:49.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.651+0000 7f69a6a00640 1 --2- 172.21.15.31:0/1109330327 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f662676c0 0x560f66265120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.651+0000 7f69a6a00640 1 -- 172.21.15.31:0/1109330327 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560f65ed7210 con 0x560f6625dfd0 2024-03-21T10:38:49.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.651+0000 7f69a6000640 1 --2- 172.21.15.31:0/1109330327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f6625e8f0 0x560f66264be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:49.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.651+0000 7f69a6a00640 1 --2- 172.21.15.31:0/1109330327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f6625dfd0 0x560f662646a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f69980eea20 tx=0x7f6998003ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=237af4d5acd4ade8 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:49.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.651+0000 7f6987e00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:38:49.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.651+0000 7f6987e00640 1 -- 172.21.15.31:0/1109330327 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699804b070 con 0x560f6625dfd0 2024-03-21T10:38:49.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.651+0000 7f6987e00640 1 -- 172.21.15.31:0/1109330327 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f69980c5b90 con 0x560f6625dfd0 2024-03-21T10:38:49.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.651+0000 7f69b0247740 1 -- 172.21.15.31:0/1109330327 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6994012930 con 0x560f6625dfd0 2024-03-21T10:38:49.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.652+0000 7f6987e00640 1 -- 172.21.15.31:0/1109330327 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69980bd150 con 0x560f6625dfd0 2024-03-21T10:38:49.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.652+0000 7f69b0247740 10 client.25654 fetch_fsmap learned FSMap version 24 2024-03-21T10:38:49.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.652+0000 7f69b0247740 1 -- 172.21.15.31:0/1109330327 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x560f65fa3620 con 0x560f6625dfd0 2024-03-21T10:38:49.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.652+0000 7f6987e00640 1 -- 172.21.15.31:0/1109330327 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6994012930 con 0x560f6625dfd0 2024-03-21T10:38:49.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.652+0000 7f6987e00640 1 -- 172.21.15.31:0/1109330327 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f69980eebc0 con 0x560f6625dfd0 2024-03-21T10:38:49.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.652+0000 7f69b0247740 10 client.25654 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:38:49.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.652+0000 7f69b0247740 10 client.25654 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:38:49.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.652+0000 7f69b0247740 20 client.25654 populate_metadata read hostname 'smithi031' 2024-03-21T10:38:49.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.652+0000 7f69b0247740 1 --2- 172.21.15.31:0/1109330327 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560f66277610 0x560f66279a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:49.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.652+0000 7f69b0247740 4 client.25654 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:38:49.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.652+0000 7f69b0247740 1 -- 172.21.15.31:0/1109330327 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560f661cfd70 con 0x560f66277610 2024-03-21T10:38:49.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.652+0000 7f69a6000640 1 --2- 172.21.15.31:0/1109330327 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560f66277610 0x560f66279a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:49.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.653+0000 7f69a6000640 1 --2- 172.21.15.31:0/1109330327 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560f66277610 0x560f66279a20 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41d23747190e99a3 server_cookie=2ca4220ce4df2fd9 in_seq=0 out_seq=0 2024-03-21T10:38:49.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.653+0000 7f6987e00640 10 client.25654 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:49.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.654+0000 7f6987e00640 1 -- 172.21.15.31:0/1109330327 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560f661cfd70 con 0x560f66277610 2024-03-21T10:38:49.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.654+0000 7f6987e00640 10 client.25654 handle_command_reply: tid=0 2024-03-21T10:38:49.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.654+0000 7f69b0247740 1 client.25654 shutdown 2024-03-21T10:38:49.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.654+0000 7f69b0247740 1 -- 172.21.15.31:0/1109330327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f6625dfd0 msgr2=0x560f662646a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:49.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.654+0000 7f69b0247740 1 --2- 172.21.15.31:0/1109330327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f6625dfd0 0x560f662646a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f69980eea20 tx=0x7f6998003ea0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.654+0000 7f69b0247740 1 -- 172.21.15.31:0/1109330327 shutdown_connections 2024-03-21T10:38:49.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.654+0000 7f69b0247740 1 --2- 172.21.15.31:0/1109330327 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560f66277610 0x560f66279a20 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.654+0000 7f69b0247740 1 --2- 172.21.15.31:0/1109330327 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f662676c0 0x560f66265120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.654+0000 7f69b0247740 1 --2- 172.21.15.31:0/1109330327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f6625e8f0 0x560f66264be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.654+0000 7f6987e00640 0 client.25654 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:49.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.654+0000 7f69b0247740 1 --2- 172.21.15.31:0/1109330327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f6625dfd0 0x560f662646a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.654+0000 7f69b0247740 1 -- 172.21.15.31:0/1109330327 >> 172.21.15.31:0/1109330327 conn(0x560f662588f0 msgr2=0x560f6625a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:49.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.654+0000 7f69b0247740 1 -- 172.21.15.31:0/1109330327 shutdown_connections 2024-03-21T10:38:49.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.654+0000 7f69b0247740 1 -- 172.21.15.31:0/1109330327 wait complete. 2024-03-21T10:38:49.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.654+0000 7f69b0247740 20 client.25654 trim_cache size 0 max 16384 2024-03-21T10:38:49.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.669+0000 7f69b0247740 1 Processor -- start 2024-03-21T10:38:49.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.669+0000 7f69b0247740 1 -- start start 2024-03-21T10:38:49.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.670+0000 7f69b0247740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f6625dfd0 0x560f66266dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:49.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.670+0000 7f69b0247740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f6625e8f0 0x560f662577d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:49.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.670+0000 7f69b0247740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f662676c0 0x560f66272a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:49.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.670+0000 7f69b0247740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a81b7460 con 0x560f6625e8f0 2024-03-21T10:38:49.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.670+0000 7f69b0247740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a81b4f60 con 0x560f6625dfd0 2024-03-21T10:38:49.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.670+0000 7f69b0247740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a81b3000 con 0x560f662676c0 2024-03-21T10:38:49.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.670+0000 7f69a6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f6625dfd0 0x560f66266dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:49.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.670+0000 7f69a6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f6625e8f0 0x560f662577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:49.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.670+0000 7f69a6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f6625dfd0 0x560f66266dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42288/0 (socket says 172.21.15.31:42288) 2024-03-21T10:38:49.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.670+0000 7f69a6a00640 1 -- 172.21.15.31:0/1759812893 learned_addr learned my addr 172.21.15.31:0/1759812893 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:49.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.670+0000 7f69a7400640 1 --2- 172.21.15.31:0/1759812893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f662676c0 0x560f66272a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:49.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.670+0000 7f69a6a00640 1 -- 172.21.15.31:0/1759812893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f662676c0 msgr2=0x560f66272a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:49.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.670+0000 7f69a6a00640 1 --2- 172.21.15.31:0/1759812893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f662676c0 0x560f66272a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.670+0000 7f69a6a00640 1 -- 172.21.15.31:0/1759812893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f6625e8f0 msgr2=0x560f662577d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:49.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.670+0000 7f69a6a00640 1 --2- 172.21.15.31:0/1759812893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f6625e8f0 0x560f662577d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.670+0000 7f69a6a00640 1 -- 172.21.15.31:0/1759812893 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f66262bf0 con 0x560f6625dfd0 2024-03-21T10:38:49.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.670+0000 7f69a6000640 1 --2- 172.21.15.31:0/1759812893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f6625e8f0 0x560f662577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:49.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.671+0000 7f69a6a00640 1 --2- 172.21.15.31:0/1759812893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f6625dfd0 0x560f66266dd0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6998000c00 tx=0x7f69980bbb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6759bc884b6bc991 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:49.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.671+0000 7f6992a00640 1 -- 172.21.15.31:0/1759812893 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699804b070 con 0x560f6625dfd0 2024-03-21T10:38:49.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.671+0000 7f6992a00640 1 -- 172.21.15.31:0/1759812893 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69980b9040 con 0x560f6625dfd0 2024-03-21T10:38:49.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.671+0000 7f6992a00640 1 -- 172.21.15.31:0/1759812893 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69980bcf70 con 0x560f6625dfd0 2024-03-21T10:38:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.671+0000 7f69b0247740 1 -- 172.21.15.31:0/1759812893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f6625dfd0 msgr2=0x560f66266dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.671+0000 7f69b0247740 1 --2- 172.21.15.31:0/1759812893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f6625dfd0 0x560f66266dd0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6998000c00 tx=0x7f69980bbb20 comp rx=0 tx=0).stop 2024-03-21T10:38:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.672+0000 7f69b0247740 1 -- 172.21.15.31:0/1759812893 shutdown_connections 2024-03-21T10:38:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.672+0000 7f69b0247740 1 --2- 172.21.15.31:0/1759812893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f662676c0 0x560f66272a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.672+0000 7f69b0247740 1 --2- 172.21.15.31:0/1759812893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f6625e8f0 0x560f662577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.672+0000 7f69b0247740 1 --2- 172.21.15.31:0/1759812893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f6625dfd0 0x560f66266dd0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.672+0000 7f69b0247740 1 -- 172.21.15.31:0/1759812893 >> 172.21.15.31:0/1759812893 conn(0x560f662588f0 msgr2=0x560f662570c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.672+0000 7f69b0247740 1 -- 172.21.15.31:0/1759812893 shutdown_connections 2024-03-21T10:38:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.672+0000 7f69b0247740 1 -- 172.21.15.31:0/1759812893 wait complete. 2024-03-21T10:38:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.672+0000 7f69b0247740 1 -- start start 2024-03-21T10:38:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.672+0000 7f69b0247740 1 Processor -- start 2024-03-21T10:38:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.673+0000 7f69b0247740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f6625dfd0 0x560f6626eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.673+0000 7f69b0247740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f6625e8f0 0x560f6626f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.673+0000 7f69b0247740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f662676c0 0x560f66261440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.673+0000 7f69b0247740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a81b3600 con 0x560f6625e8f0 2024-03-21T10:38:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.673+0000 7f69b0247740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a81b4900 con 0x560f662676c0 2024-03-21T10:38:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.673+0000 7f69b0247740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a81b3c40 con 0x560f6625dfd0 2024-03-21T10:38:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.673+0000 7f69b0247740 10 client.0 _ll_register_callbacks cb 0x7f6998009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:38:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.673+0000 7f69a6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f6625dfd0 0x560f6626eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.673+0000 7f69a6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f6625dfd0 0x560f6626eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38320/0 (socket says 172.21.15.31:38320) 2024-03-21T10:38:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.673+0000 7f69a6a00640 1 -- 172.21.15.31:0/3188168409 learned_addr learned my addr 172.21.15.31:0/3188168409 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.673+0000 7f69a7400640 1 --2- 172.21.15.31:0/3188168409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f662676c0 0x560f66261440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.673+0000 7f69a6000640 1 --2- 172.21.15.31:0/3188168409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f6625e8f0 0x560f6626f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.673+0000 7f69a6a00640 1 -- 172.21.15.31:0/3188168409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f662676c0 msgr2=0x560f66261440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.673+0000 7f69a6a00640 1 --2- 172.21.15.31:0/3188168409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f662676c0 0x560f66261440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.673+0000 7f69a6a00640 1 -- 172.21.15.31:0/3188168409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f6625e8f0 msgr2=0x560f6626f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.673+0000 7f69a6a00640 1 --2- 172.21.15.31:0/3188168409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f6625e8f0 0x560f6626f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.673+0000 7f69a6a00640 1 -- 172.21.15.31:0/3188168409 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560f66261c70 con 0x560f6625dfd0 2024-03-21T10:38:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.673+0000 7f69a6000640 1 --2- 172.21.15.31:0/3188168409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f6625e8f0 0x560f6626f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.674+0000 7f69a6a00640 1 --2- 172.21.15.31:0/3188168409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f6625dfd0 0x560f6626eb30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6998003fd0 tx=0x7f6998005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7b7a92df6ad311e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.674+0000 7f6987e00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:38:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.674+0000 7f6987e00640 1 -- 172.21.15.31:0/3188168409 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69980da060 con 0x560f6625dfd0 2024-03-21T10:38:49.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.674+0000 7f6987e00640 1 -- 172.21.15.31:0/3188168409 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f69980c5d90 con 0x560f6625dfd0 2024-03-21T10:38:49.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.674+0000 7f69b0247740 1 -- 172.21.15.31:0/3188168409 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f69980b9040 con 0x560f6625dfd0 2024-03-21T10:38:49.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.674+0000 7f6987e00640 1 -- 172.21.15.31:0/3188168409 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f699804b070 con 0x560f6625dfd0 2024-03-21T10:38:49.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.674+0000 7f69b0247740 10 client.25660 fetch_fsmap learned FSMap version 24 2024-03-21T10:38:49.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.674+0000 7f69b0247740 1 -- 172.21.15.31:0/3188168409 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x560f66261c70 con 0x560f6625dfd0 2024-03-21T10:38:49.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.674+0000 7f6987e00640 1 -- 172.21.15.31:0/3188168409 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69980b9040 con 0x560f6625dfd0 2024-03-21T10:38:49.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.675+0000 7f6987e00640 1 -- 172.21.15.31:0/3188168409 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f69980a6b50 con 0x560f6625dfd0 2024-03-21T10:38:49.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.675+0000 7f69b0247740 10 client.25660 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:38:49.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.675+0000 7f69b0247740 10 client.25660 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:38:49.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.675+0000 7f69b0247740 20 client.25660 populate_metadata read hostname 'smithi031' 2024-03-21T10:38:49.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.675+0000 7f69b0247740 1 --2- 172.21.15.31:0/3188168409 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560f662d3b20 0x560f662d5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:49.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.675+0000 7f69b0247740 4 client.25660 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:38:49.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.675+0000 7f69b0247740 1 -- 172.21.15.31:0/3188168409 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560f661ca0c0 con 0x560f662d3b20 2024-03-21T10:38:49.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.675+0000 7f69a6000640 1 --2- 172.21.15.31:0/3188168409 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560f662d3b20 0x560f662d5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:49.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.675+0000 7f69a6000640 1 --2- 172.21.15.31:0/3188168409 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560f662d3b20 0x560f662d5f10 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=456edd794d623dcb server_cookie=57bf6e56d2cdcac1 in_seq=0 out_seq=0 2024-03-21T10:38:49.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.675+0000 7f6987e00640 10 client.25660 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:49.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.687+0000 7f6987e00640 1 -- 172.21.15.31:0/3188168409 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560f661ca0c0 con 0x560f662d3b20 2024-03-21T10:38:49.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.687+0000 7f6987e00640 10 client.25660 handle_command_reply: tid=0 2024-03-21T10:38:49.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.687+0000 7f69b0247740 1 client.25660 shutdown 2024-03-21T10:38:49.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.688+0000 7f69b0247740 1 -- 172.21.15.31:0/3188168409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f6625dfd0 msgr2=0x560f6626eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:49.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.688+0000 7f69b0247740 1 --2- 172.21.15.31:0/3188168409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f6625dfd0 0x560f6626eb30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6998003fd0 tx=0x7f6998005fd0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.688+0000 7f69b0247740 1 -- 172.21.15.31:0/3188168409 shutdown_connections 2024-03-21T10:38:49.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.688+0000 7f69b0247740 1 --2- 172.21.15.31:0/3188168409 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560f662d3b20 0x560f662d5f10 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.688+0000 7f69b0247740 1 --2- 172.21.15.31:0/3188168409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f662676c0 0x560f66261440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.688+0000 7f69b0247740 1 --2- 172.21.15.31:0/3188168409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f6625e8f0 0x560f6626f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.688+0000 7f69b0247740 1 --2- 172.21.15.31:0/3188168409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f6625dfd0 0x560f6626eb30 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.688+0000 7f6987e00640 0 client.25660 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:49.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.688+0000 7f69b0247740 1 -- 172.21.15.31:0/3188168409 >> 172.21.15.31:0/3188168409 conn(0x560f662588f0 msgr2=0x560f66264ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:49.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.688+0000 7f69b0247740 1 -- 172.21.15.31:0/3188168409 shutdown_connections 2024-03-21T10:38:49.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.688+0000 7f69b0247740 1 -- 172.21.15.31:0/3188168409 wait complete. 2024-03-21T10:38:49.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.688+0000 7f69b0247740 20 client.25660 trim_cache size 0 max 16384 2024-03-21T10:38:49.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:38:49.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:38:49.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:38:49.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:38:49.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.689+0000 7f69ad600640 1 -- 172.21.15.31:0/3986203265 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f698c0a3190 msgr2=0x7f698c0a5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:49.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.689+0000 7f69ad600640 1 --2- 172.21.15.31:0/3986203265 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f698c0a3190 0x7f698c0a5650 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f69a00129a0 tx=0x7f69a0009500 comp rx=0 tx=0).stop 2024-03-21T10:38:49.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.689+0000 7f69ad600640 1 -- 172.21.15.31:0/3986203265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69a8075bd0 msgr2=0x7f69a819e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:49.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.689+0000 7f69ad600640 1 --2- 172.21.15.31:0/3986203265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69a8075bd0 0x7f69a819e620 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6998009430 tx=0x7f699802cb10 comp rx=0 tx=0).stop 2024-03-21T10:38:49.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.689+0000 7f69ad600640 1 -- 172.21.15.31:0/3986203265 shutdown_connections 2024-03-21T10:38:49.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.689+0000 7f69ad600640 1 --2- 172.21.15.31:0/3986203265 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f698c0a3190 0x7f698c0a5650 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.689+0000 7f69ad600640 1 --2- 172.21.15.31:0/3986203265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69a8112520 0x7f69a81a2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.689+0000 7f69ad600640 1 --2- 172.21.15.31:0/3986203265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69a80764f0 0x7f69a819eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.689+0000 7f69ad600640 1 --2- 172.21.15.31:0/3986203265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69a8075bd0 0x7f69a819e620 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:49.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.689+0000 7f69ad600640 1 -- 172.21.15.31:0/3986203265 >> 172.21.15.31:0/3986203265 conn(0x7f69a80ff390 msgr2=0x7f69a8113c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:49.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.689+0000 7f69ad600640 1 -- 172.21.15.31:0/3986203265 shutdown_connections 2024-03-21T10:38:49.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:38:49.689+0000 7f69ad600640 1 -- 172.21.15.31:0/3986203265 wait complete. 2024-03-21T10:38:51.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:50 smithi070 ceph-mon[36330]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-21T10:38:51.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:50 smithi040 ceph-mon[36154]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-21T10:38:51.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:50 smithi031 ceph-mon[31673]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-21T10:38:53.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:52 smithi040 ceph-mon[36154]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 49 op/s 2024-03-21T10:38:53.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:52 smithi031 ceph-mon[31673]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 49 op/s 2024-03-21T10:38:53.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:52 smithi070 ceph-mon[36330]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 49 op/s 2024-03-21T10:38:53.582 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:38:53.582 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:38:53.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.819+0000 7f684b000640 1 -- 172.21.15.31:0/1682105875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6844075bd0 msgr2=0x7f6844075fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:53.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.819+0000 7f684b000640 1 --2- 172.21.15.31:0/1682105875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6844075bd0 0x7f6844075fb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6838009cd0 tx=0x7f683802c0e0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.819+0000 7f684b000640 1 -- 172.21.15.31:0/1682105875 shutdown_connections 2024-03-21T10:38:53.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.819+0000 7f684b000640 1 --2- 172.21.15.31:0/1682105875 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6844112300 0x7f68441146f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.819+0000 7f684b000640 1 --2- 172.21.15.31:0/1682105875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68440764f0 0x7f6844111dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.819+0000 7f684b000640 1 --2- 172.21.15.31:0/1682105875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6844075bd0 0x7f6844075fb0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.819+0000 7f684b000640 1 -- 172.21.15.31:0/1682105875 >> 172.21.15.31:0/1682105875 conn(0x7f68440ff110 msgr2=0x7f6844101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:53.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.819+0000 7f684b000640 1 -- 172.21.15.31:0/1682105875 shutdown_connections 2024-03-21T10:38:53.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.819+0000 7f684b000640 1 -- 172.21.15.31:0/1682105875 wait complete. 2024-03-21T10:38:53.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.820+0000 7f684b000640 1 Processor -- start 2024-03-21T10:38:53.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.821+0000 7f684b000640 1 -- start start 2024-03-21T10:38:53.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.821+0000 7f684b000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6844075bd0 0x7f684419e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:53.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.821+0000 7f684b000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68440764f0 0x7f684419e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:53.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.821+0000 7f684b000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6844112300 0x7f68441a2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:53.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.821+0000 7f684b000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6844117e60 con 0x7f6844075bd0 2024-03-21T10:38:53.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.821+0000 7f684b000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6844117ce0 con 0x7f68440764f0 2024-03-21T10:38:53.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.821+0000 7f684b000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6844117fe0 con 0x7f6844112300 2024-03-21T10:38:53.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.821+0000 7f6849200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68440764f0 0x7f684419e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:53.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.822+0000 7f6849200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68440764f0 0x7f684419e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42316/0 (socket says 172.21.15.31:42316) 2024-03-21T10:38:53.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.822+0000 7f6849200640 1 -- 172.21.15.31:0/1814807061 learned_addr learned my addr 172.21.15.31:0/1814807061 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:53.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.822+0000 7f684a600640 1 --2- 172.21.15.31:0/1814807061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6844112300 0x7f68441a2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:53.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.822+0000 7f6849c00640 1 --2- 172.21.15.31:0/1814807061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6844075bd0 0x7f684419e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:53.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.822+0000 7f6849200640 1 -- 172.21.15.31:0/1814807061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6844112300 msgr2=0x7f68441a2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:53.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.822+0000 7f6849200640 1 --2- 172.21.15.31:0/1814807061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6844112300 0x7f68441a2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.822+0000 7f6849200640 1 -- 172.21.15.31:0/1814807061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6844075bd0 msgr2=0x7f684419e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:53.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.822+0000 7f6849200640 1 --2- 172.21.15.31:0/1814807061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6844075bd0 0x7f684419e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.822+0000 7f6849200640 1 -- 172.21.15.31:0/1814807061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68441a33b0 con 0x7f68440764f0 2024-03-21T10:38:53.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.822+0000 7f6849c00640 1 --2- 172.21.15.31:0/1814807061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6844075bd0 0x7f684419e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:53.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.822+0000 7f6849200640 1 --2- 172.21.15.31:0/1814807061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68440764f0 0x7f684419e900 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f683000d950 tx=0x7f683000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:53.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.823+0000 7f6842000640 1 -- 172.21.15.31:0/1814807061 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6830017070 con 0x7f68440764f0 2024-03-21T10:38:53.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.823+0000 7f6842000640 1 -- 172.21.15.31:0/1814807061 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6830009c30 con 0x7f68440764f0 2024-03-21T10:38:53.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.823+0000 7f684b000640 1 -- 172.21.15.31:0/1814807061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68441a36a0 con 0x7f68440764f0 2024-03-21T10:38:53.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.823+0000 7f6842000640 1 -- 172.21.15.31:0/1814807061 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683000f6b0 con 0x7f68440764f0 2024-03-21T10:38:53.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.823+0000 7f684b000640 1 -- 172.21.15.31:0/1814807061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68441aaf30 con 0x7f68440764f0 2024-03-21T10:38:53.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.826+0000 7f684f401740 1 Processor -- start 2024-03-21T10:38:53.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.826+0000 7f684f401740 1 -- start start 2024-03-21T10:38:53.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.827+0000 7f6842000640 1 -- 172.21.15.31:0/1814807061 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) v1 ==== 138030+0+0 (secure 0 0 0) 0x7f68300109d0 con 0x7f68440764f0 2024-03-21T10:38:53.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.827+0000 7f684f401740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c7de8edd0 0x563c7de8db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:53.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.827+0000 7f684f401740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c7de8e0f0 0x563c7de8e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:53.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.827+0000 7f684f401740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c7de97460 0x563c7de99850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:53.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.827+0000 7f684f401740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c7ddf2250 con 0x563c7de8e0f0 2024-03-21T10:38:53.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.827+0000 7f684f401740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c7dd9aa30 con 0x563c7de97460 2024-03-21T10:38:53.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.827+0000 7f684f401740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c7ddee2f0 con 0x563c7de8edd0 2024-03-21T10:38:53.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.827+0000 7f684a600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c7de8edd0 0x563c7de8db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:53.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.827+0000 7f684a600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c7de8edd0 0x563c7de8db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38360/0 (socket says 172.21.15.31:38360) 2024-03-21T10:38:53.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.827+0000 7f684a600640 1 -- 172.21.15.31:0/3962481638 learned_addr learned my addr 172.21.15.31:0/3962481638 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:53.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.827+0000 7f6849200640 1 --2- 172.21.15.31:0/3962481638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c7de97460 0x563c7de99850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:53.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.827+0000 7f6849c00640 1 --2- 172.21.15.31:0/3962481638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c7de8e0f0 0x563c7de8e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:53.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.827+0000 7f684a600640 1 -- 172.21.15.31:0/3962481638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c7de97460 msgr2=0x563c7de99850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:53.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.827+0000 7f684a600640 1 --2- 172.21.15.31:0/3962481638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c7de97460 0x563c7de99850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.827+0000 7f684a600640 1 -- 172.21.15.31:0/3962481638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c7de8e0f0 msgr2=0x563c7de8e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:53.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.827+0000 7f684a600640 1 --2- 172.21.15.31:0/3962481638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c7de8e0f0 0x563c7de8e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.827+0000 7f684a600640 1 -- 172.21.15.31:0/3962481638 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c7dd85690 con 0x563c7de8edd0 2024-03-21T10:38:53.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.828+0000 7f684a600640 1 --2- 172.21.15.31:0/3962481638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c7de8edd0 0x563c7de8db20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f683c009ba0 tx=0x7f683c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b557a08442914a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:53.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.828+0000 7f6836a00640 1 -- 172.21.15.31:0/3962481638 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c004230 con 0x563c7de8edd0 2024-03-21T10:38:53.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.828+0000 7f6836a00640 1 -- 172.21.15.31:0/3962481638 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f683c009e10 con 0x563c7de8edd0 2024-03-21T10:38:53.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.828+0000 7f6836a00640 1 -- 172.21.15.31:0/3962481638 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c004d10 con 0x563c7de8edd0 2024-03-21T10:38:53.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.828+0000 7f6842000640 1 --2- 172.21.15.31:0/1814807061 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f682c0a31a0 0x7f682c0a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:53.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.828+0000 7f6842000640 1 -- 172.21.15.31:0/1814807061 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f68300cf8b0 con 0x7f68440764f0 2024-03-21T10:38:53.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.828+0000 7f6849c00640 1 --2- 172.21.15.31:0/1814807061 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f682c0a31a0 0x7f682c0a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:53.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.829+0000 7f684f401740 1 -- 172.21.15.31:0/3962481638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c7de8edd0 msgr2=0x563c7de8db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:53.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.829+0000 7f684f401740 1 --2- 172.21.15.31:0/3962481638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c7de8edd0 0x563c7de8db20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f683c009ba0 tx=0x7f683c011810 comp rx=0 tx=0).stop 2024-03-21T10:38:53.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.829+0000 7f684f401740 1 -- 172.21.15.31:0/3962481638 shutdown_connections 2024-03-21T10:38:53.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.829+0000 7f684f401740 1 --2- 172.21.15.31:0/3962481638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c7de97460 0x563c7de99850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.829+0000 7f684f401740 1 --2- 172.21.15.31:0/3962481638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c7de8e0f0 0x563c7de8e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.829+0000 7f684f401740 1 --2- 172.21.15.31:0/3962481638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c7de8edd0 0x563c7de8db20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.829+0000 7f684f401740 1 -- 172.21.15.31:0/3962481638 >> 172.21.15.31:0/3962481638 conn(0x563c7de55200 msgr2=0x563c7de88b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:53.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.829+0000 7f684f401740 1 -- 172.21.15.31:0/3962481638 shutdown_connections 2024-03-21T10:38:53.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.829+0000 7f684f401740 1 -- 172.21.15.31:0/3962481638 wait complete. 2024-03-21T10:38:53.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.829+0000 7f6849c00640 1 --2- 172.21.15.31:0/1814807061 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f682c0a31a0 0x7f682c0a5660 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6838009ca0 tx=0x7f683802e040 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:53.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.830+0000 7f684f401740 1 -- start start 2024-03-21T10:38:53.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.830+0000 7f684f401740 1 Processor -- start 2024-03-21T10:38:53.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.830+0000 7f684f401740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c7de8e0f0 0x563c7de96a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:53.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.831+0000 7f684f401740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c7de8edd0 0x563c7de93df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:53.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.831+0000 7f684f401740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c7de97460 0x563c7de94330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:53.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.831+0000 7f684f401740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c7de9a2d0 con 0x563c7de8edd0 2024-03-21T10:38:53.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.831+0000 7f684f401740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c7ddade80 con 0x563c7de8e0f0 2024-03-21T10:38:53.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.831+0000 7f684f401740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c7de9a450 con 0x563c7de97460 2024-03-21T10:38:53.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.831+0000 7f684f401740 10 client.0 _ll_register_callbacks cb 0x563c7ddc0130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:38:53.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.831+0000 7f6849200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c7de8edd0 0x563c7de93df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:53.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.831+0000 7f684a600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c7de97460 0x563c7de94330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:53.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.831+0000 7f6849200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c7de8edd0 0x563c7de93df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33878/0 (socket says 172.21.15.31:33878) 2024-03-21T10:38:53.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.831+0000 7f6849200640 1 -- 172.21.15.31:0/3234275845 learned_addr learned my addr 172.21.15.31:0/3234275845 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:53.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.831+0000 7f6849c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c7de8e0f0 0x563c7de96a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:53.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.831+0000 7f6849200640 1 -- 172.21.15.31:0/3234275845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c7de97460 msgr2=0x563c7de94330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:53.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.831+0000 7f6849200640 1 --2- 172.21.15.31:0/3234275845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c7de97460 0x563c7de94330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.831+0000 7f6849200640 1 -- 172.21.15.31:0/3234275845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c7de8e0f0 msgr2=0x563c7de96a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:53.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.831+0000 7f6849200640 1 --2- 172.21.15.31:0/3234275845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c7de8e0f0 0x563c7de96a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.831+0000 7f6849200640 1 -- 172.21.15.31:0/3234275845 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563c7dbd07d0 con 0x563c7de8edd0 2024-03-21T10:38:53.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.831+0000 7f684a600640 1 --2- 172.21.15.31:0/3234275845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c7de97460 0x563c7de94330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:53.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.832+0000 7f6849200640 1 --2- 172.21.15.31:0/3234275845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c7de8edd0 0x563c7de93df0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6830009610 tx=0x7f6830097490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f93006ad9b64890 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:53.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.832+0000 7f682b400640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:38:53.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.832+0000 7f682b400640 1 -- 172.21.15.31:0/3234275845 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6830020070 con 0x563c7de8edd0 2024-03-21T10:38:53.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.832+0000 7f682b400640 1 -- 172.21.15.31:0/3234275845 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f68300cceb0 con 0x563c7de8edd0 2024-03-21T10:38:53.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.832+0000 7f684f401740 1 -- 172.21.15.31:0/3234275845 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f683c009e10 con 0x563c7de8edd0 2024-03-21T10:38:53.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.832+0000 7f682b400640 1 -- 172.21.15.31:0/3234275845 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f683009b330 con 0x563c7de8edd0 2024-03-21T10:38:53.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.832+0000 7f684f401740 10 client.15528 fetch_fsmap learned FSMap version 24 2024-03-21T10:38:53.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.832+0000 7f684f401740 1 -- 172.21.15.31:0/3234275845 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x563c7db063f0 con 0x563c7de8edd0 2024-03-21T10:38:53.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.832+0000 7f682b400640 1 -- 172.21.15.31:0/3234275845 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f683c009e10 con 0x563c7de8edd0 2024-03-21T10:38:53.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.833+0000 7f682b400640 1 -- 172.21.15.31:0/3234275845 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f683009ac40 con 0x563c7de8edd0 2024-03-21T10:38:53.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.833+0000 7f684f401740 10 client.15528 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:38:53.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.833+0000 7f684f401740 10 client.15528 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:38:53.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.833+0000 7f684f401740 20 client.15528 populate_metadata read hostname 'smithi031' 2024-03-21T10:38:53.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.833+0000 7f684f401740 1 --2- 172.21.15.31:0/3234275845 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563c7dea7270 0x563c7dea9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:53.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.833+0000 7f684f401740 4 client.15528 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:38:53.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.833+0000 7f684f401740 1 -- 172.21.15.31:0/3234275845 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563c7ddff790 con 0x563c7dea7270 2024-03-21T10:38:53.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.833+0000 7f6849c00640 1 --2- 172.21.15.31:0/3234275845 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563c7dea7270 0x563c7dea9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:53.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.834+0000 7f6849c00640 1 --2- 172.21.15.31:0/3234275845 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563c7dea7270 0x563c7dea9660 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0a2e6a3eb206060 server_cookie=4354d25fde2cf9c4 in_seq=0 out_seq=0 2024-03-21T10:38:53.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.834+0000 7f682b400640 10 client.15528 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:53.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.835+0000 7f682b400640 1 -- 172.21.15.31:0/3234275845 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563c7ddff790 con 0x563c7dea7270 2024-03-21T10:38:53.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.835+0000 7f682b400640 10 client.15528 handle_command_reply: tid=0 2024-03-21T10:38:53.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.835+0000 7f684f401740 1 client.15528 shutdown 2024-03-21T10:38:53.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.835+0000 7f684f401740 1 -- 172.21.15.31:0/3234275845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c7de8edd0 msgr2=0x563c7de93df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:53.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.835+0000 7f684f401740 1 --2- 172.21.15.31:0/3234275845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c7de8edd0 0x563c7de93df0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6830009610 tx=0x7f6830097490 comp rx=0 tx=0).stop 2024-03-21T10:38:53.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.835+0000 7f684f401740 1 -- 172.21.15.31:0/3234275845 shutdown_connections 2024-03-21T10:38:53.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.835+0000 7f684f401740 1 --2- 172.21.15.31:0/3234275845 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563c7dea7270 0x563c7dea9660 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.835+0000 7f684f401740 1 --2- 172.21.15.31:0/3234275845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c7de97460 0x563c7de94330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.835+0000 7f684f401740 1 --2- 172.21.15.31:0/3234275845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c7de8edd0 0x563c7de93df0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.835+0000 7f684f401740 1 --2- 172.21.15.31:0/3234275845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c7de8e0f0 0x563c7de96a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.835+0000 7f682b400640 0 client.15528 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:53.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.835+0000 7f684f401740 1 -- 172.21.15.31:0/3234275845 >> 172.21.15.31:0/3234275845 conn(0x563c7de55200 msgr2=0x563c7de8a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:53.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.835+0000 7f684f401740 1 -- 172.21.15.31:0/3234275845 shutdown_connections 2024-03-21T10:38:53.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.835+0000 7f684f401740 1 -- 172.21.15.31:0/3234275845 wait complete. 2024-03-21T10:38:53.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.835+0000 7f684f401740 20 client.15528 trim_cache size 0 max 16384 2024-03-21T10:38:53.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.854+0000 7f684f401740 1 Processor -- start 2024-03-21T10:38:53.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.854+0000 7f684f401740 1 -- start start 2024-03-21T10:38:53.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.854+0000 7f684f401740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c7de8e0f0 0x563c7de96520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:53.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.854+0000 7f684f401740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c7de8edd0 0x563c7de96a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:53.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.854+0000 7f684f401740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c7de97460 0x563c7de98850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:53.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.854+0000 7f684f401740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68441b7230 con 0x563c7de97460 2024-03-21T10:38:53.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.854+0000 7f684f401740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68441b4d30 con 0x563c7de8edd0 2024-03-21T10:38:53.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.854+0000 7f684f401740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68441b2dd0 con 0x563c7de8e0f0 2024-03-21T10:38:53.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.854+0000 7f6849200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c7de8edd0 0x563c7de96a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:53.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.854+0000 7f6849c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c7de8e0f0 0x563c7de96520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:53.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.854+0000 7f684a600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c7de97460 0x563c7de98850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:53.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.854+0000 7f6849200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c7de8edd0 0x563c7de96a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42346/0 (socket says 172.21.15.31:42346) 2024-03-21T10:38:53.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.854+0000 7f6849200640 1 -- 172.21.15.31:0/1373532885 learned_addr learned my addr 172.21.15.31:0/1373532885 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:53.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.855+0000 7f6849200640 1 -- 172.21.15.31:0/1373532885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c7de8e0f0 msgr2=0x563c7de96520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:53.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.855+0000 7f6849200640 1 --2- 172.21.15.31:0/1373532885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c7de8e0f0 0x563c7de96520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.855+0000 7f6849200640 1 -- 172.21.15.31:0/1373532885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c7de97460 msgr2=0x563c7de98850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:53.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.855+0000 7f6849200640 1 --2- 172.21.15.31:0/1373532885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c7de97460 0x563c7de98850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.855+0000 7f6849200640 1 -- 172.21.15.31:0/1373532885 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c7dac60f0 con 0x563c7de8edd0 2024-03-21T10:38:53.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.855+0000 7f6849c00640 1 --2- 172.21.15.31:0/1373532885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c7de8e0f0 0x563c7de96520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:53.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.855+0000 7f6849200640 1 --2- 172.21.15.31:0/1373532885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c7de8edd0 0x563c7de96a60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f68300058c0 tx=0x7f6830010d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1561f6bccf946138 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:53.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.855+0000 7f6836000640 1 -- 172.21.15.31:0/1373532885 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6830020070 con 0x563c7de8edd0 2024-03-21T10:38:53.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.855+0000 7f6836000640 1 -- 172.21.15.31:0/1373532885 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f68300d52d0 con 0x563c7de8edd0 2024-03-21T10:38:53.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.855+0000 7f6836000640 1 -- 172.21.15.31:0/1373532885 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68300d5080 con 0x563c7de8edd0 2024-03-21T10:38:53.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.856+0000 7f684f401740 1 -- 172.21.15.31:0/1373532885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c7de8edd0 msgr2=0x563c7de96a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:53.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.856+0000 7f684f401740 1 --2- 172.21.15.31:0/1373532885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c7de8edd0 0x563c7de96a60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f68300058c0 tx=0x7f6830010d70 comp rx=0 tx=0).stop 2024-03-21T10:38:53.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.856+0000 7f684f401740 1 -- 172.21.15.31:0/1373532885 shutdown_connections 2024-03-21T10:38:53.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.856+0000 7f684f401740 1 --2- 172.21.15.31:0/1373532885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c7de97460 0x563c7de98850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.856+0000 7f684f401740 1 --2- 172.21.15.31:0/1373532885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c7de8edd0 0x563c7de96a60 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.856+0000 7f684f401740 1 --2- 172.21.15.31:0/1373532885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c7de8e0f0 0x563c7de96520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.856+0000 7f684f401740 1 -- 172.21.15.31:0/1373532885 >> 172.21.15.31:0/1373532885 conn(0x563c7de55200 msgr2=0x563c7de92700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:53.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.856+0000 7f684f401740 1 -- 172.21.15.31:0/1373532885 shutdown_connections 2024-03-21T10:38:53.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.856+0000 7f684f401740 1 -- 172.21.15.31:0/1373532885 wait complete. 2024-03-21T10:38:53.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.857+0000 7f684f401740 1 -- start start 2024-03-21T10:38:53.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.857+0000 7f684f401740 1 Processor -- start 2024-03-21T10:38:53.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.857+0000 7f684f401740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c7de8e0f0 0x563c7de906c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:53.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.857+0000 7f684f401740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c7de8edd0 0x563c7de90c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:53.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.857+0000 7f684f401740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c7de97460 0x563c7de91140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:53.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.857+0000 7f684f401740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68441b33d0 con 0x563c7de8e0f0 2024-03-21T10:38:53.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.857+0000 7f684f401740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68441b46d0 con 0x563c7de8edd0 2024-03-21T10:38:53.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.857+0000 7f684f401740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68441b3a10 con 0x563c7de97460 2024-03-21T10:38:53.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.857+0000 7f684f401740 10 client.0 _ll_register_callbacks cb 0x7f68300d5220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:38:53.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.857+0000 7f6849200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c7de8edd0 0x563c7de90c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:53.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.857+0000 7f6849c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c7de8e0f0 0x563c7de906c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:53.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.857+0000 7f684a600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c7de97460 0x563c7de91140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:53.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.857+0000 7f6849200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c7de8edd0 0x563c7de90c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42356/0 (socket says 172.21.15.31:42356) 2024-03-21T10:38:53.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.857+0000 7f6849200640 1 -- 172.21.15.31:0/2228905520 learned_addr learned my addr 172.21.15.31:0/2228905520 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:53.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.858+0000 7f6849200640 1 -- 172.21.15.31:0/2228905520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c7de97460 msgr2=0x563c7de91140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:53.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.858+0000 7f6849200640 1 --2- 172.21.15.31:0/2228905520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c7de97460 0x563c7de91140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.858+0000 7f6849200640 1 -- 172.21.15.31:0/2228905520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c7de8e0f0 msgr2=0x563c7de906c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:53.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.858+0000 7f6849200640 1 --2- 172.21.15.31:0/2228905520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c7de8e0f0 0x563c7de906c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.858+0000 7f6849200640 1 -- 172.21.15.31:0/2228905520 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563c7df01c50 con 0x563c7de8edd0 2024-03-21T10:38:53.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.858+0000 7f6849200640 1 --2- 172.21.15.31:0/2228905520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c7de8edd0 0x563c7de90c00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6830009610 tx=0x7f6830098090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97e70e362512669d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:53.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.858+0000 7f682b400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:38:53.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.858+0000 7f682b400640 1 -- 172.21.15.31:0/2228905520 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6830020070 con 0x563c7de8edd0 2024-03-21T10:38:53.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.858+0000 7f682b400640 1 -- 172.21.15.31:0/2228905520 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f68300b4b20 con 0x563c7de8edd0 2024-03-21T10:38:53.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.858+0000 7f684f401740 1 -- 172.21.15.31:0/2228905520 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f68300d52d0 con 0x563c7de8edd0 2024-03-21T10:38:53.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.858+0000 7f682b400640 1 -- 172.21.15.31:0/2228905520 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6830081050 con 0x563c7de8edd0 2024-03-21T10:38:53.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.859+0000 7f684f401740 10 client.25691 fetch_fsmap learned FSMap version 24 2024-03-21T10:38:53.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.859+0000 7f684f401740 1 -- 172.21.15.31:0/2228905520 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x563c7df01c50 con 0x563c7de8edd0 2024-03-21T10:38:53.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.859+0000 7f682b400640 1 -- 172.21.15.31:0/2228905520 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f68300d52d0 con 0x563c7de8edd0 2024-03-21T10:38:53.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.859+0000 7f682b400640 1 -- 172.21.15.31:0/2228905520 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f6830097540 con 0x563c7de8edd0 2024-03-21T10:38:53.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.859+0000 7f684f401740 10 client.25691 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:38:53.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.859+0000 7f684f401740 10 client.25691 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:38:53.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.859+0000 7f684f401740 20 client.25691 populate_metadata read hostname 'smithi031' 2024-03-21T10:38:53.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.859+0000 7f684f401740 1 --2- 172.21.15.31:0/2228905520 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563c7df03190 0x563c7df05580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:53.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.859+0000 7f684f401740 4 client.25691 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:38:53.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.859+0000 7f684f401740 1 -- 172.21.15.31:0/2228905520 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563c7ddf9ae0 con 0x563c7df03190 2024-03-21T10:38:53.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.859+0000 7f6849c00640 1 --2- 172.21.15.31:0/2228905520 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563c7df03190 0x563c7df05580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:53.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.860+0000 7f6849c00640 1 --2- 172.21.15.31:0/2228905520 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563c7df03190 0x563c7df05580 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba076fcb145dfdb0 server_cookie=76d2929c99e638dd in_seq=0 out_seq=0 2024-03-21T10:38:53.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.860+0000 7f682b400640 10 client.25691 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:53.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.860+0000 7f682b400640 1 -- 172.21.15.31:0/2228905520 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x563c7ddf9ae0 con 0x563c7df03190 2024-03-21T10:38:53.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.860+0000 7f682b400640 10 client.25691 handle_command_reply: tid=0 2024-03-21T10:38:53.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.860+0000 7f684f401740 1 client.25691 shutdown 2024-03-21T10:38:53.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.860+0000 7f684f401740 1 -- 172.21.15.31:0/2228905520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c7de8edd0 msgr2=0x563c7de90c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:53.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.860+0000 7f684f401740 1 --2- 172.21.15.31:0/2228905520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c7de8edd0 0x563c7de90c00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6830009610 tx=0x7f6830098090 comp rx=0 tx=0).stop 2024-03-21T10:38:53.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.860+0000 7f684f401740 1 -- 172.21.15.31:0/2228905520 shutdown_connections 2024-03-21T10:38:53.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.860+0000 7f684f401740 1 --2- 172.21.15.31:0/2228905520 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563c7df03190 0x563c7df05580 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.860+0000 7f684f401740 1 --2- 172.21.15.31:0/2228905520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563c7de97460 0x563c7de91140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.860+0000 7f684f401740 1 --2- 172.21.15.31:0/2228905520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c7de8edd0 0x563c7de90c00 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.860+0000 7f684f401740 1 --2- 172.21.15.31:0/2228905520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c7de8e0f0 0x563c7de906c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.860+0000 7f684f401740 1 -- 172.21.15.31:0/2228905520 >> 172.21.15.31:0/2228905520 conn(0x563c7de55200 msgr2=0x563c7de92b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:53.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.860+0000 7f682b400640 0 client.25691 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:53.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.861+0000 7f684f401740 1 -- 172.21.15.31:0/2228905520 shutdown_connections 2024-03-21T10:38:53.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.861+0000 7f684f401740 1 -- 172.21.15.31:0/2228905520 wait complete. 2024-03-21T10:38:53.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.861+0000 7f684f401740 20 client.25691 trim_cache size 0 max 16384 2024-03-21T10:38:53.863 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:38:53.863 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-03-21T10:38:53.863 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-03-21T10:38:53.863 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:38:53.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.861+0000 7f684b000640 1 -- 172.21.15.31:0/1814807061 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f682c0a31a0 msgr2=0x7f682c0a5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:53.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.861+0000 7f684b000640 1 --2- 172.21.15.31:0/1814807061 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f682c0a31a0 0x7f682c0a5660 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6838009ca0 tx=0x7f683802e040 comp rx=0 tx=0).stop 2024-03-21T10:38:53.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.861+0000 7f684b000640 1 -- 172.21.15.31:0/1814807061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68440764f0 msgr2=0x7f684419e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:53.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.861+0000 7f684b000640 1 --2- 172.21.15.31:0/1814807061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68440764f0 0x7f684419e900 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f683000d950 tx=0x7f683000de10 comp rx=0 tx=0).stop 2024-03-21T10:38:53.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.861+0000 7f684b000640 1 -- 172.21.15.31:0/1814807061 shutdown_connections 2024-03-21T10:38:53.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.861+0000 7f684b000640 1 --2- 172.21.15.31:0/1814807061 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f682c0a31a0 0x7f682c0a5660 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.861+0000 7f684b000640 1 --2- 172.21.15.31:0/1814807061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6844112300 0x7f68441a2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.861+0000 7f684b000640 1 --2- 172.21.15.31:0/1814807061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68440764f0 0x7f684419e900 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.861+0000 7f684b000640 1 --2- 172.21.15.31:0/1814807061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6844075bd0 0x7f684419e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:53.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.861+0000 7f684b000640 1 -- 172.21.15.31:0/1814807061 >> 172.21.15.31:0/1814807061 conn(0x7f68440ff110 msgr2=0x7f6844113e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:53.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.862+0000 7f684b000640 1 -- 172.21.15.31:0/1814807061 shutdown_connections 2024-03-21T10:38:53.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:53.862+0000 7f684b000640 1 -- 172.21.15.31:0/1814807061 wait complete. 2024-03-21T10:38:53.878 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-21T10:38:53.878 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:38:54.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.098+0000 7f9d0b200640 1 -- 172.21.15.31:0/2897839571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d04109750 msgr2=0x7f9d04109b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:54.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.098+0000 7f9d0b200640 1 --2- 172.21.15.31:0/2897839571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d04109750 0x7f9d04109b30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8009900 tx=0x7f9cf802bc40 comp rx=0 tx=0).stop 2024-03-21T10:38:54.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.098+0000 7f9d0b200640 1 -- 172.21.15.31:0/2897839571 shutdown_connections 2024-03-21T10:38:54.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.098+0000 7f9d0b200640 1 --2- 172.21.15.31:0/2897839571 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d04104100 0x7f9d04110610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.098+0000 7f9d0b200640 1 --2- 172.21.15.31:0/2897839571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d04103740 0x7f9d04103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.098+0000 7f9d0b200640 1 --2- 172.21.15.31:0/2897839571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d04109750 0x7f9d04109b30 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.098+0000 7f9d0b200640 1 -- 172.21.15.31:0/2897839571 >> 172.21.15.31:0/2897839571 conn(0x7f9d040ff210 msgr2=0x7f9d04101630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:54.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.098+0000 7f9d0b200640 1 -- 172.21.15.31:0/2897839571 shutdown_connections 2024-03-21T10:38:54.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.098+0000 7f9d0b200640 1 -- 172.21.15.31:0/2897839571 wait complete. 2024-03-21T10:38:54.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.100+0000 7f9d0b200640 1 Processor -- start 2024-03-21T10:38:54.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.100+0000 7f9d0b200640 1 -- start start 2024-03-21T10:38:54.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.100+0000 7f9d0b200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d04103740 0x7f9d0419a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.100+0000 7f9d0b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d04104100 0x7f9d0419a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.100+0000 7f9d0b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d04109750 0x7f9d0419e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.100+0000 7f9d0b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d04113b70 con 0x7f9d04104100 2024-03-21T10:38:54.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.100+0000 7f9d0b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d041139f0 con 0x7f9d04109750 2024-03-21T10:38:54.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.100+0000 7f9d0b200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d04113cf0 con 0x7f9d04103740 2024-03-21T10:38:54.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.101+0000 7f9d09e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d04103740 0x7f9d0419a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.101+0000 7f9d09e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d04103740 0x7f9d0419a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38406/0 (socket says 172.21.15.31:38406) 2024-03-21T10:38:54.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.101+0000 7f9d09e00640 1 -- 172.21.15.31:0/914258114 learned_addr learned my addr 172.21.15.31:0/914258114 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:54.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.101+0000 7f9d0a800640 1 --2- 172.21.15.31:0/914258114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d04109750 0x7f9d0419e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.101+0000 7f9d09400640 1 --2- 172.21.15.31:0/914258114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d04104100 0x7f9d0419a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.101+0000 7f9d09e00640 1 -- 172.21.15.31:0/914258114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d04109750 msgr2=0x7f9d0419e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:54.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.101+0000 7f9d09e00640 1 --2- 172.21.15.31:0/914258114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d04109750 0x7f9d0419e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.101+0000 7f9d09e00640 1 -- 172.21.15.31:0/914258114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d04104100 msgr2=0x7f9d0419a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:54.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.101+0000 7f9d09e00640 1 --2- 172.21.15.31:0/914258114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d04104100 0x7f9d0419a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.101+0000 7f9d09e00640 1 -- 172.21.15.31:0/914258114 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0419f000 con 0x7f9d04103740 2024-03-21T10:38:54.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.101+0000 7f9d09400640 1 --2- 172.21.15.31:0/914258114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d04104100 0x7f9d0419a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:54.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.101+0000 7f9d09e00640 1 --2- 172.21.15.31:0/914258114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d04103740 0x7f9d0419a010 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8005ab0 tx=0x7f9cf80042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:54.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.102+0000 7f9cfea00640 1 -- 172.21.15.31:0/914258114 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8041070 con 0x7f9d04103740 2024-03-21T10:38:54.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.102+0000 7f9cfea00640 1 -- 172.21.15.31:0/914258114 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9cf802ccd0 con 0x7f9d04103740 2024-03-21T10:38:54.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.102+0000 7f9d0b200640 1 -- 172.21.15.31:0/914258114 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d0419f290 con 0x7f9d04103740 2024-03-21T10:38:54.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.102+0000 7f9cfea00640 1 -- 172.21.15.31:0/914258114 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8003ca0 con 0x7f9d04103740 2024-03-21T10:38:54.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.102+0000 7f9d0b200640 1 -- 172.21.15.31:0/914258114 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d041a6b80 con 0x7f9d04103740 2024-03-21T10:38:54.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.103+0000 7f9d0b200640 1 -- 172.21.15.31:0/914258114 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d0404f470 con 0x7f9d04103740 2024-03-21T10:38:54.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.105+0000 7f9cfea00640 1 -- 172.21.15.31:0/914258114 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) v1 ==== 138030+0+0 (secure 0 0 0) 0x7f9cf8044070 con 0x7f9d04103740 2024-03-21T10:38:54.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.106+0000 7f9cfea00640 1 --2- 172.21.15.31:0/914258114 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9cec0a3110 0x7f9cec0a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.106+0000 7f9cfea00640 1 -- 172.21.15.31:0/914258114 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f9cf80f16a0 con 0x7f9d04103740 2024-03-21T10:38:54.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.106+0000 7f9d09400640 1 --2- 172.21.15.31:0/914258114 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9cec0a3110 0x7f9cec0a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.107+0000 7f9d09400640 1 --2- 172.21.15.31:0/914258114 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9cec0a3110 0x7f9cec0a55d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9d04101820 tx=0x7f9cf400b5f0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:54.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.110+0000 7f9cfea00640 1 -- 172.21.15.31:0/914258114 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9cf80bdd90 con 0x7f9d04103740 2024-03-21T10:38:54.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.355+0000 7f9d0b200640 1 -- 172.21.15.31:0/914258114 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d0410bcb0 con 0x7f9d04103740 2024-03-21T10:38:54.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.356+0000 7f9cfea00640 1 -- 172.21.15.31:0/914258114 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11384 (secure 0 0 0) 0x7f9cf80bd690 con 0x7f9d04103740 2024-03-21T10:38:54.354 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:38:54.354 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:38:12.319122+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24547,24596]},"id":1}]} 2024-03-21T10:38:54.354 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 24 2024-03-21T10:38:54.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.360+0000 7f9d0b200640 1 -- 172.21.15.31:0/914258114 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9cec0a3110 msgr2=0x7f9cec0a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:54.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.360+0000 7f9d0b200640 1 --2- 172.21.15.31:0/914258114 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9cec0a3110 0x7f9cec0a55d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9d04101820 tx=0x7f9cf400b5f0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.360+0000 7f9d0b200640 1 -- 172.21.15.31:0/914258114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d04103740 msgr2=0x7f9d0419a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:54.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.360+0000 7f9d0b200640 1 --2- 172.21.15.31:0/914258114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d04103740 0x7f9d0419a010 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8005ab0 tx=0x7f9cf80042f0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.360+0000 7f9d0b200640 1 -- 172.21.15.31:0/914258114 shutdown_connections 2024-03-21T10:38:54.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.360+0000 7f9d0b200640 1 --2- 172.21.15.31:0/914258114 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9cec0a3110 0x7f9cec0a55d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.360+0000 7f9d0b200640 1 --2- 172.21.15.31:0/914258114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d04109750 0x7f9d0419e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.360+0000 7f9d0b200640 1 --2- 172.21.15.31:0/914258114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d04104100 0x7f9d0419a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.360+0000 7f9d0b200640 1 --2- 172.21.15.31:0/914258114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d04103740 0x7f9d0419a010 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.360+0000 7f9d0b200640 1 -- 172.21.15.31:0/914258114 >> 172.21.15.31:0/914258114 conn(0x7f9d040ff210 msgr2=0x7f9d040ff900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:54.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.361+0000 7f9d0b200640 1 -- 172.21.15.31:0/914258114 shutdown_connections 2024-03-21T10:38:54.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.361+0000 7f9d0b200640 1 -- 172.21.15.31:0/914258114 wait complete. 2024-03-21T10:38:54.378 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:38:54.378 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-21T10:38:54.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.624+0000 7fa1bfe00640 1 -- 172.21.15.31:0/2354761402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1c0104040 msgr2=0x7fa1c0110550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:54.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.624+0000 7fa1bfe00640 1 --2- 172.21.15.31:0/2354761402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1c0104040 0x7fa1c0110550 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8009930 tx=0x7fa1b802bc20 comp rx=0 tx=0).stop 2024-03-21T10:38:54.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.624+0000 7fa1bfe00640 1 -- 172.21.15.31:0/2354761402 shutdown_connections 2024-03-21T10:38:54.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.624+0000 7fa1bfe00640 1 --2- 172.21.15.31:0/2354761402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1c0104040 0x7fa1c0110550 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.624+0000 7fa1bfe00640 1 --2- 172.21.15.31:0/2354761402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c0103680 0x7fa1c0103b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.624+0000 7fa1bfe00640 1 --2- 172.21.15.31:0/2354761402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c0109690 0x7fa1c0109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.624+0000 7fa1bfe00640 1 -- 172.21.15.31:0/2354761402 >> 172.21.15.31:0/2354761402 conn(0x7fa1c00ff110 msgr2=0x7fa1c0101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:54.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.624+0000 7fa1bfe00640 1 -- 172.21.15.31:0/2354761402 shutdown_connections 2024-03-21T10:38:54.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.624+0000 7fa1bfe00640 1 -- 172.21.15.31:0/2354761402 wait complete. 2024-03-21T10:38:54.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.626+0000 7fa1bfe00640 1 Processor -- start 2024-03-21T10:38:54.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.626+0000 7fa1bfe00640 1 -- start start 2024-03-21T10:38:54.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.627+0000 7fa1bfe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1c0103680 0x7fa1c0199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.627+0000 7fa1bfe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c0104040 0x7fa1c019a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.627+0000 7fa1bfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c0109690 0x7fa1c019e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.627+0000 7fa1bfe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c0113af0 con 0x7fa1c0104040 2024-03-21T10:38:54.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.627+0000 7fa1bfe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c0113970 con 0x7fa1c0109690 2024-03-21T10:38:54.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.627+0000 7fa1bfe00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c0113c70 con 0x7fa1c0103680 2024-03-21T10:38:54.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.627+0000 7fa1bea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1c0103680 0x7fa1c0199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.627+0000 7fa1bea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1c0103680 0x7fa1c0199f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38426/0 (socket says 172.21.15.31:38426) 2024-03-21T10:38:54.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.627+0000 7fa1bea00640 1 -- 172.21.15.31:0/3256784209 learned_addr learned my addr 172.21.15.31:0/3256784209 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:54.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.627+0000 7fa1bf400640 1 --2- 172.21.15.31:0/3256784209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c0109690 0x7fa1c019e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.627+0000 7fa1b6000640 1 --2- 172.21.15.31:0/3256784209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c0104040 0x7fa1c019a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.627+0000 7fa1bea00640 1 -- 172.21.15.31:0/3256784209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c0109690 msgr2=0x7fa1c019e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:54.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.627+0000 7fa1bea00640 1 --2- 172.21.15.31:0/3256784209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c0109690 0x7fa1c019e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.627+0000 7fa1bea00640 1 -- 172.21.15.31:0/3256784209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c0104040 msgr2=0x7fa1c019a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:54.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.627+0000 7fa1bea00640 1 --2- 172.21.15.31:0/3256784209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c0104040 0x7fa1c019a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.627+0000 7fa1bea00640 1 -- 172.21.15.31:0/3256784209 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c019ef00 con 0x7fa1c0103680 2024-03-21T10:38:54.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.627+0000 7fa1b6000640 1 --2- 172.21.15.31:0/3256784209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c0104040 0x7fa1c019a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:54.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.628+0000 7fa1bea00640 1 --2- 172.21.15.31:0/3256784209 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1c0103680 0x7fa1c0199f10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa1b000ebc0 tx=0x7fa1b000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:54.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.628+0000 7fa1c4a00640 1 -- 172.21.15.31:0/3256784209 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b0010630 con 0x7fa1c0103680 2024-03-21T10:38:54.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.628+0000 7fa1c4a00640 1 -- 172.21.15.31:0/3256784209 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1b0010cb0 con 0x7fa1c0103680 2024-03-21T10:38:54.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.628+0000 7fa1c4a00640 1 -- 172.21.15.31:0/3256784209 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b0018bd0 con 0x7fa1c0103680 2024-03-21T10:38:54.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.628+0000 7fa1bfe00640 1 -- 172.21.15.31:0/3256784209 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1c019f1f0 con 0x7fa1c0103680 2024-03-21T10:38:54.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.628+0000 7fa1bfe00640 1 -- 172.21.15.31:0/3256784209 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1c01a6a80 con 0x7fa1c0103680 2024-03-21T10:38:54.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.631+0000 7fa1c4a00640 1 -- 172.21.15.31:0/3256784209 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) v1 ==== 138030+0+0 (secure 0 0 0) 0x7fa1b00107d0 con 0x7fa1c0103680 2024-03-21T10:38:54.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.632+0000 7fa1c8b7a740 1 Processor -- start 2024-03-21T10:38:54.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.632+0000 7fa1c8b7a740 1 -- start start 2024-03-21T10:38:54.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.632+0000 7fa1c8b7a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634de7498b0 0x5634de749c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.632+0000 7fa1c8b7a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634de74a260 0x5634de752810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.632+0000 7fa1c8b7a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634de752de0 0x5634de755230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.632+0000 7fa1c8b7a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634de6ad2c0 con 0x5634de74a260 2024-03-21T10:38:54.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.632+0000 7fa1c8b7a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634de655c90 con 0x5634de752de0 2024-03-21T10:38:54.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.632+0000 7fa1c8b7a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634de6a9360 con 0x5634de7498b0 2024-03-21T10:38:54.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.632+0000 7fa1bf400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634de7498b0 0x5634de749c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.632+0000 7fa1bea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634de74a260 0x5634de752810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.632+0000 7fa1bf400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634de7498b0 0x5634de749c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38442/0 (socket says 172.21.15.31:38442) 2024-03-21T10:38:54.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.632+0000 7fa1bf400640 1 -- 172.21.15.31:0/4229096780 learned_addr learned my addr 172.21.15.31:0/4229096780 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:54.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.633+0000 7fa1b6000640 1 --2- 172.21.15.31:0/4229096780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634de752de0 0x5634de755230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.633+0000 7fa1bf400640 1 -- 172.21.15.31:0/4229096780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634de752de0 msgr2=0x5634de755230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:54.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.633+0000 7fa1bf400640 1 --2- 172.21.15.31:0/4229096780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634de752de0 0x5634de755230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.633+0000 7fa1bf400640 1 -- 172.21.15.31:0/4229096780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634de74a260 msgr2=0x5634de752810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:54.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.633+0000 7fa1bf400640 1 --2- 172.21.15.31:0/4229096780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634de74a260 0x5634de752810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.633+0000 7fa1bf400640 1 -- 172.21.15.31:0/4229096780 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5634de67a080 con 0x5634de7498b0 2024-03-21T10:38:54.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.633+0000 7fa1c4a00640 1 --2- 172.21.15.31:0/3256784209 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa1a40a31a0 0x7fa1a40a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.633+0000 7fa1bf400640 1 --2- 172.21.15.31:0/4229096780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634de7498b0 0x5634de749c90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8009900 tx=0x7fa1b8005be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eaaa90591d4037 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:54.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.633+0000 7fa1c4a00640 1 -- 172.21.15.31:0/3256784209 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fa1b00d02b0 con 0x7fa1c0103680 2024-03-21T10:38:54.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.634+0000 7fa1bea00640 1 --2- 172.21.15.31:0/3256784209 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa1a40a31a0 0x7fa1a40a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.634+0000 7fa1afe00640 1 -- 172.21.15.31:0/4229096780 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b8045070 con 0x5634de7498b0 2024-03-21T10:38:54.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.634+0000 7fa1afe00640 1 -- 172.21.15.31:0/4229096780 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1b802c5c0 con 0x5634de7498b0 2024-03-21T10:38:54.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.634+0000 7fa1afe00640 1 -- 172.21.15.31:0/4229096780 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b80049b0 con 0x5634de7498b0 2024-03-21T10:38:54.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.634+0000 7fa1c8b7a740 1 -- 172.21.15.31:0/4229096780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634de7498b0 msgr2=0x5634de749c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:54.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.634+0000 7fa1bea00640 1 --2- 172.21.15.31:0/3256784209 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa1a40a31a0 0x7fa1a40a5660 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa1b00cd600 tx=0x7fa1b00d3cb0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:54.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.634+0000 7fa1c8b7a740 1 --2- 172.21.15.31:0/4229096780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634de7498b0 0x5634de749c90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8009900 tx=0x7fa1b8005be0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.634+0000 7fa1c8b7a740 1 -- 172.21.15.31:0/4229096780 shutdown_connections 2024-03-21T10:38:54.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.634+0000 7fa1c8b7a740 1 --2- 172.21.15.31:0/4229096780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634de752de0 0x5634de755230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.634+0000 7fa1c8b7a740 1 --2- 172.21.15.31:0/4229096780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634de74a260 0x5634de752810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.634+0000 7fa1c8b7a740 1 --2- 172.21.15.31:0/4229096780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634de7498b0 0x5634de749c90 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.634+0000 7fa1c8b7a740 1 -- 172.21.15.31:0/4229096780 >> 172.21.15.31:0/4229096780 conn(0x5634de743f80 msgr2=0x5634de744360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:54.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.635+0000 7fa1c8b7a740 1 -- 172.21.15.31:0/4229096780 shutdown_connections 2024-03-21T10:38:54.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.635+0000 7fa1c8b7a740 1 -- 172.21.15.31:0/4229096780 wait complete. 2024-03-21T10:38:54.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.635+0000 7fa1c8b7a740 1 -- start start 2024-03-21T10:38:54.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.635+0000 7fa1c8b7a740 1 Processor -- start 2024-03-21T10:38:54.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.636+0000 7fa1c8b7a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634de7498b0 0x5634de752700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.636+0000 7fa1c8b7a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634de74a260 0x5634de74fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.636+0000 7fa1c8b7a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634de752de0 0x5634de750030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.636+0000 7fa1c8b7a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634de755cf0 con 0x5634de74a260 2024-03-21T10:38:54.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.636+0000 7fa1c8b7a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634de6690d0 con 0x5634de7498b0 2024-03-21T10:38:54.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.636+0000 7fa1c8b7a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634de755e70 con 0x5634de752de0 2024-03-21T10:38:54.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.636+0000 7fa1c8b7a740 10 client.0 _ll_register_callbacks cb 0x5634de66de80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:38:54.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.636+0000 7fa1bf400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634de7498b0 0x5634de752700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.636+0000 7fa1b6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634de74a260 0x5634de74fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.636+0000 7fa1bf400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634de7498b0 0x5634de752700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42420/0 (socket says 172.21.15.31:42420) 2024-03-21T10:38:54.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.636+0000 7fa1bf400640 1 -- 172.21.15.31:0/3221074556 learned_addr learned my addr 172.21.15.31:0/3221074556 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:54.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.636+0000 7fa1bf400640 1 --2- 172.21.15.31:0/3221074556 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634de752de0 0x5634de750030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.636+0000 7fa1b6000640 1 -- 172.21.15.31:0/3221074556 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634de752de0 msgr2=0x5634de750030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:54.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.637+0000 7fa1b6000640 1 --2- 172.21.15.31:0/3221074556 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634de752de0 0x5634de750030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.637+0000 7fa1b6000640 1 -- 172.21.15.31:0/3221074556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634de7498b0 msgr2=0x5634de752700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:54.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.637+0000 7fa1b6000640 1 --2- 172.21.15.31:0/3221074556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634de7498b0 0x5634de752700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.637+0000 7fa1b6000640 1 -- 172.21.15.31:0/3221074556 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5634de3c13f0 con 0x5634de74a260 2024-03-21T10:38:54.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.637+0000 7fa1bf400640 1 --2- 172.21.15.31:0/3221074556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634de7498b0 0x5634de752700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:54.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.637+0000 7fa1b6000640 1 --2- 172.21.15.31:0/3221074556 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634de74a260 0x5634de74fa90 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa1a80126d0 tx=0x7fa1a8012b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c67725e856da0920 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:54.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.637+0000 7fa1acc00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:38:54.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.637+0000 7fa1acc00640 1 -- 172.21.15.31:0/3221074556 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a8004300 con 0x5634de74a260 2024-03-21T10:38:54.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.637+0000 7fa1acc00640 1 -- 172.21.15.31:0/3221074556 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fa1a80119a0 con 0x5634de74a260 2024-03-21T10:38:54.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.637+0000 7fa1c8b7a740 1 -- 172.21.15.31:0/3221074556 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa1b802c5c0 con 0x5634de74a260 2024-03-21T10:38:54.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.637+0000 7fa1acc00640 1 -- 172.21.15.31:0/3221074556 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1a8025050 con 0x5634de74a260 2024-03-21T10:38:54.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.637+0000 7fa1c8b7a740 10 client.15552 fetch_fsmap learned FSMap version 24 2024-03-21T10:38:54.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.637+0000 7fa1c8b7a740 1 -- 172.21.15.31:0/3221074556 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5634de48b7d0 con 0x5634de74a260 2024-03-21T10:38:54.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.638+0000 7fa1acc00640 1 -- 172.21.15.31:0/3221074556 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1b802c5c0 con 0x5634de74a260 2024-03-21T10:38:54.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.638+0000 7fa1acc00640 1 -- 172.21.15.31:0/3221074556 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa1a800f460 con 0x5634de74a260 2024-03-21T10:38:54.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.638+0000 7fa1c8b7a740 10 client.15552 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:38:54.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.638+0000 7fa1c8b7a740 10 client.15552 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:38:54.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.638+0000 7fa1c8b7a740 20 client.15552 populate_metadata read hostname 'smithi031' 2024-03-21T10:38:54.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.638+0000 7fa1c8b7a740 1 --2- 172.21.15.31:0/3221074556 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5634de762c60 0x5634de7650a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.638+0000 7fa1c8b7a740 4 client.15552 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:38:54.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.638+0000 7fa1c8b7a740 1 -- 172.21.15.31:0/3221074556 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5634de6ba800 con 0x5634de762c60 2024-03-21T10:38:54.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.638+0000 7fa1bf400640 1 --2- 172.21.15.31:0/3221074556 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5634de762c60 0x5634de7650a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.639+0000 7fa1bf400640 1 --2- 172.21.15.31:0/3221074556 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5634de762c60 0x5634de7650a0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b1e0107f1338855b server_cookie=a4b4e5f92dfb9b1d in_seq=0 out_seq=0 2024-03-21T10:38:54.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.639+0000 7fa1acc00640 10 client.15552 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:38:54.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.640+0000 7fa1acc00640 1 -- 172.21.15.31:0/3221074556 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5634de6ba800 con 0x5634de762c60 2024-03-21T10:38:54.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.640+0000 7fa1acc00640 10 client.15552 handle_command_reply: tid=0 2024-03-21T10:38:54.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.640+0000 7fa1c8b7a740 1 client.15552 shutdown 2024-03-21T10:38:54.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.640+0000 7fa1c8b7a740 1 -- 172.21.15.31:0/3221074556 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634de74a260 msgr2=0x5634de74fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:54.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.640+0000 7fa1c8b7a740 1 --2- 172.21.15.31:0/3221074556 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634de74a260 0x5634de74fa90 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa1a80126d0 tx=0x7fa1a8012b90 comp rx=0 tx=0).stop 2024-03-21T10:38:54.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.640+0000 7fa1c8b7a740 1 -- 172.21.15.31:0/3221074556 shutdown_connections 2024-03-21T10:38:54.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.640+0000 7fa1c8b7a740 1 --2- 172.21.15.31:0/3221074556 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5634de762c60 0x5634de7650a0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.640+0000 7fa1c8b7a740 1 --2- 172.21.15.31:0/3221074556 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634de752de0 0x5634de750030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.640+0000 7fa1c8b7a740 1 --2- 172.21.15.31:0/3221074556 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634de74a260 0x5634de74fa90 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.640+0000 7fa1c8b7a740 1 --2- 172.21.15.31:0/3221074556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634de7498b0 0x5634de752700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.640+0000 7fa1acc00640 0 client.15552 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:38:54.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.640+0000 7fa1c8b7a740 1 -- 172.21.15.31:0/3221074556 >> 172.21.15.31:0/3221074556 conn(0x5634de743f80 msgr2=0x5634de745cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:54.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.640+0000 7fa1c8b7a740 1 -- 172.21.15.31:0/3221074556 shutdown_connections 2024-03-21T10:38:54.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.640+0000 7fa1c8b7a740 1 -- 172.21.15.31:0/3221074556 wait complete. 2024-03-21T10:38:54.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.640+0000 7fa1c8b7a740 20 client.15552 trim_cache size 0 max 16384 2024-03-21T10:38:54.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.654+0000 7fa1c8b7a740 1 Processor -- start 2024-03-21T10:38:54.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.654+0000 7fa1c8b7a740 1 -- start start 2024-03-21T10:38:54.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.654+0000 7fa1c8b7a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634de7498b0 0x5634de743160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.654+0000 7fa1c8b7a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634de74a260 0x5634de7521c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.654+0000 7fa1c8b7a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634de752de0 0x5634de752700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.654+0000 7fa1c8b7a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c01b2da0 con 0x5634de74a260 2024-03-21T10:38:54.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.654+0000 7fa1c8b7a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c01b08a0 con 0x5634de7498b0 2024-03-21T10:38:54.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.654+0000 7fa1c8b7a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c01ae940 con 0x5634de752de0 2024-03-21T10:38:54.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.655+0000 7fa1bf400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634de7498b0 0x5634de743160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.655+0000 7fa1bf400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634de7498b0 0x5634de743160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42434/0 (socket says 172.21.15.31:42434) 2024-03-21T10:38:54.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.655+0000 7fa1bf400640 1 -- 172.21.15.31:0/883402498 learned_addr learned my addr 172.21.15.31:0/883402498 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:54.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.655+0000 7fa1b6000640 1 --2- 172.21.15.31:0/883402498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634de74a260 0x5634de7521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.655+0000 7fa1bf400640 1 --2- 172.21.15.31:0/883402498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634de752de0 0x5634de752700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.655+0000 7fa1bf400640 1 -- 172.21.15.31:0/883402498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634de752de0 msgr2=0x5634de752700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:54.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.655+0000 7fa1bf400640 1 --2- 172.21.15.31:0/883402498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634de752de0 0x5634de752700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.655+0000 7fa1bf400640 1 -- 172.21.15.31:0/883402498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634de74a260 msgr2=0x5634de7521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.655+0000 7fa1bf400640 1 --2- 172.21.15.31:0/883402498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634de74a260 0x5634de7521c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.655+0000 7fa1bf400640 1 -- 172.21.15.31:0/883402498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5634de74dfe0 con 0x5634de7498b0 2024-03-21T10:38:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.655+0000 7fa1bf400640 1 --2- 172.21.15.31:0/883402498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634de7498b0 0x5634de743160 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa1b802ce30 tx=0x7fa1b802bc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6320fe7a09c3ab31 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.655+0000 7fa1af400640 1 -- 172.21.15.31:0/883402498 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b8045040 con 0x5634de7498b0 2024-03-21T10:38:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.656+0000 7fa1af400640 1 -- 172.21.15.31:0/883402498 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1b802c5c0 con 0x5634de7498b0 2024-03-21T10:38:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.656+0000 7fa1af400640 1 -- 172.21.15.31:0/883402498 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b8005160 con 0x5634de7498b0 2024-03-21T10:38:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.656+0000 7fa1c8b7a740 1 -- 172.21.15.31:0/883402498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634de7498b0 msgr2=0x5634de743160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.656+0000 7fa1c8b7a740 1 --2- 172.21.15.31:0/883402498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634de7498b0 0x5634de743160 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa1b802ce30 tx=0x7fa1b802bc20 comp rx=0 tx=0).stop 2024-03-21T10:38:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.656+0000 7fa1c8b7a740 1 -- 172.21.15.31:0/883402498 shutdown_connections 2024-03-21T10:38:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.656+0000 7fa1c8b7a740 1 --2- 172.21.15.31:0/883402498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634de752de0 0x5634de752700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.656+0000 7fa1c8b7a740 1 --2- 172.21.15.31:0/883402498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634de74a260 0x5634de7521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.656+0000 7fa1c8b7a740 1 --2- 172.21.15.31:0/883402498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634de7498b0 0x5634de743160 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.656+0000 7fa1c8b7a740 1 -- 172.21.15.31:0/883402498 >> 172.21.15.31:0/883402498 conn(0x5634de743f80 msgr2=0x5634de744360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:54.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.656+0000 7fa1c8b7a740 1 -- 172.21.15.31:0/883402498 shutdown_connections 2024-03-21T10:38:54.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.656+0000 7fa1c8b7a740 1 -- 172.21.15.31:0/883402498 wait complete. 2024-03-21T10:38:54.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.657+0000 7fa1c8b7a740 1 -- start start 2024-03-21T10:38:54.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.657+0000 7fa1c8b7a740 1 Processor -- start 2024-03-21T10:38:54.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.657+0000 7fa1c8b7a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634de7498b0 0x5634de75aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.657+0000 7fa1c8b7a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634de74a260 0x5634de75afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.657+0000 7fa1c8b7a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634de752de0 0x5634de7590c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.657+0000 7fa1c8b7a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c01aef40 con 0x5634de74a260 2024-03-21T10:38:54.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.657+0000 7fa1c8b7a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c01b0240 con 0x5634de7498b0 2024-03-21T10:38:54.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.657+0000 7fa1c8b7a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c01af580 con 0x5634de752de0 2024-03-21T10:38:54.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.658+0000 7fa1c8b7a740 10 client.0 _ll_register_callbacks cb 0x7fa1a8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:38:54.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.658+0000 7fa1bf400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634de7498b0 0x5634de75aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.658+0000 7fa1b6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634de74a260 0x5634de75afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.658+0000 7fa1bf400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634de7498b0 0x5634de75aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42440/0 (socket says 172.21.15.31:42440) 2024-03-21T10:38:54.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.658+0000 7fa1bf400640 1 -- 172.21.15.31:0/754111628 learned_addr learned my addr 172.21.15.31:0/754111628 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:54.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.658+0000 7fa1bf400640 1 --2- 172.21.15.31:0/754111628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634de752de0 0x5634de7590c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.658+0000 7fa1b6000640 1 -- 172.21.15.31:0/754111628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634de752de0 msgr2=0x5634de7590c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:54.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.658+0000 7fa1b6000640 1 --2- 172.21.15.31:0/754111628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634de752de0 0x5634de7590c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.658+0000 7fa1b6000640 1 -- 172.21.15.31:0/754111628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634de7498b0 msgr2=0x5634de75aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:54.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.658+0000 7fa1b6000640 1 --2- 172.21.15.31:0/754111628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634de7498b0 0x5634de75aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.658+0000 7fa1b6000640 1 -- 172.21.15.31:0/754111628 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5634de7598c0 con 0x5634de74a260 2024-03-21T10:38:54.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.659+0000 7fa1b6000640 1 --2- 172.21.15.31:0/754111628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634de74a260 0x5634de75afc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa1a80095a0 tx=0x7fa1a800f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5ed8cf5a5d02ddc server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:54.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.659+0000 7fa1acc00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:38:54.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.659+0000 7fa1acc00640 1 -- 172.21.15.31:0/754111628 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a8022040 con 0x5634de74a260 2024-03-21T10:38:54.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.659+0000 7fa1c8b7a740 1 -- 172.21.15.31:0/754111628 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa1b802c5c0 con 0x5634de74a260 2024-03-21T10:38:54.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.659+0000 7fa1acc00640 1 -- 172.21.15.31:0/754111628 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fa1a800fa20 con 0x5634de74a260 2024-03-21T10:38:54.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.659+0000 7fa1acc00640 1 -- 172.21.15.31:0/754111628 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1a803b050 con 0x5634de74a260 2024-03-21T10:38:54.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.659+0000 7fa1c8b7a740 10 client.15558 fetch_fsmap learned FSMap version 24 2024-03-21T10:38:54.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.659+0000 7fa1c8b7a740 1 -- 172.21.15.31:0/754111628 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5634de7bf1a0 con 0x5634de74a260 2024-03-21T10:38:54.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.660+0000 7fa1acc00640 1 -- 172.21.15.31:0/754111628 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1b802c5c0 con 0x5634de74a260 2024-03-21T10:38:54.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.660+0000 7fa1acc00640 1 -- 172.21.15.31:0/754111628 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa1a80035f0 con 0x5634de74a260 2024-03-21T10:38:54.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.660+0000 7fa1c8b7a740 10 client.15558 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:38:54.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.660+0000 7fa1c8b7a740 10 client.15558 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:38:54.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.660+0000 7fa1c8b7a740 20 client.15558 populate_metadata read hostname 'smithi031' 2024-03-21T10:38:54.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.660+0000 7fa1c8b7a740 1 --2- 172.21.15.31:0/754111628 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5634de7bf8a0 0x5634de7c1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.660+0000 7fa1c8b7a740 4 client.15558 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:38:54.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.660+0000 7fa1c8b7a740 1 -- 172.21.15.31:0/754111628 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5634de6b4b50 con 0x5634de7bf8a0 2024-03-21T10:38:54.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.660+0000 7fa1bf400640 1 --2- 172.21.15.31:0/754111628 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5634de7bf8a0 0x5634de7c1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.661+0000 7fa1bf400640 1 --2- 172.21.15.31:0/754111628 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5634de7bf8a0 0x5634de7c1cc0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4be8272e3d2931c1 server_cookie=ec2bf4f7c3133a6e in_seq=0 out_seq=0 2024-03-21T10:38:54.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.661+0000 7fa1acc00640 10 client.15558 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:38:54.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.661+0000 7fa1acc00640 1 -- 172.21.15.31:0/754111628 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5634de6b4b50 con 0x5634de7bf8a0 2024-03-21T10:38:54.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.661+0000 7fa1acc00640 10 client.15558 handle_command_reply: tid=0 2024-03-21T10:38:54.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.661+0000 7fa1c8b7a740 1 client.15558 shutdown 2024-03-21T10:38:54.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.661+0000 7fa1c8b7a740 1 -- 172.21.15.31:0/754111628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634de74a260 msgr2=0x5634de75afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:54.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.661+0000 7fa1c8b7a740 1 --2- 172.21.15.31:0/754111628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634de74a260 0x5634de75afc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa1a80095a0 tx=0x7fa1a800f710 comp rx=0 tx=0).stop 2024-03-21T10:38:54.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.661+0000 7fa1c8b7a740 1 -- 172.21.15.31:0/754111628 shutdown_connections 2024-03-21T10:38:54.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.661+0000 7fa1c8b7a740 1 --2- 172.21.15.31:0/754111628 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5634de7bf8a0 0x5634de7c1cc0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.661+0000 7fa1c8b7a740 1 --2- 172.21.15.31:0/754111628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634de752de0 0x5634de7590c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.661+0000 7fa1c8b7a740 1 --2- 172.21.15.31:0/754111628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634de74a260 0x5634de75afc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.661+0000 7fa1c8b7a740 1 --2- 172.21.15.31:0/754111628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634de7498b0 0x5634de75aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.661+0000 7fa1c8b7a740 1 -- 172.21.15.31:0/754111628 >> 172.21.15.31:0/754111628 conn(0x5634de743f80 msgr2=0x5634de745350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:54.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.661+0000 7fa1acc00640 0 client.15558 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:38:54.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.661+0000 7fa1c8b7a740 1 -- 172.21.15.31:0/754111628 shutdown_connections 2024-03-21T10:38:54.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.661+0000 7fa1c8b7a740 1 -- 172.21.15.31:0/754111628 wait complete. 2024-03-21T10:38:54.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.661+0000 7fa1c8b7a740 20 client.15558 trim_cache size 0 max 16384 2024-03-21T10:38:54.663 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:38:54.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.662+0000 7fa1bfe00640 1 -- 172.21.15.31:0/3256784209 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa1a40a31a0 msgr2=0x7fa1a40a5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:54.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.662+0000 7fa1bfe00640 1 --2- 172.21.15.31:0/3256784209 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa1a40a31a0 0x7fa1a40a5660 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa1b00cd600 tx=0x7fa1b00d3cb0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.662+0000 7fa1bfe00640 1 -- 172.21.15.31:0/3256784209 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1c0103680 msgr2=0x7fa1c0199f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:54.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.662+0000 7fa1bfe00640 1 --2- 172.21.15.31:0/3256784209 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1c0103680 0x7fa1c0199f10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa1b000ebc0 tx=0x7fa1b000c6a0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.662+0000 7fa1bfe00640 1 -- 172.21.15.31:0/3256784209 shutdown_connections 2024-03-21T10:38:54.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.662+0000 7fa1bfe00640 1 --2- 172.21.15.31:0/3256784209 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa1a40a31a0 0x7fa1a40a5660 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.662+0000 7fa1bfe00640 1 --2- 172.21.15.31:0/3256784209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c0109690 0x7fa1c019e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.662+0000 7fa1bfe00640 1 --2- 172.21.15.31:0/3256784209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c0104040 0x7fa1c019a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.662+0000 7fa1bfe00640 1 --2- 172.21.15.31:0/3256784209 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1c0103680 0x7fa1c0199f10 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.662+0000 7fa1bfe00640 1 -- 172.21.15.31:0/3256784209 >> 172.21.15.31:0/3256784209 conn(0x7fa1c00ff110 msgr2=0x7fa1c00ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:54.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.662+0000 7fa1bfe00640 1 -- 172.21.15.31:0/3256784209 shutdown_connections 2024-03-21T10:38:54.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.662+0000 7fa1bfe00640 1 -- 172.21.15.31:0/3256784209 wait complete. 2024-03-21T10:38:54.677 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:38:54.678 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-21T10:38:54.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.884+0000 7f1133e00640 1 -- 172.21.15.31:0/3461680328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1134109670 msgr2=0x7f1134109a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:54.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.884+0000 7f1133e00640 1 --2- 172.21.15.31:0/3461680328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1134109670 0x7f1134109a50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1124009900 tx=0x7f112402bc30 comp rx=0 tx=0).stop 2024-03-21T10:38:54.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.884+0000 7f1133e00640 1 -- 172.21.15.31:0/3461680328 shutdown_connections 2024-03-21T10:38:54.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.884+0000 7f1133e00640 1 --2- 172.21.15.31:0/3461680328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1134104020 0x7f1134110530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.884+0000 7f1133e00640 1 --2- 172.21.15.31:0/3461680328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1134103660 0x7f1134103ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.884+0000 7f1133e00640 1 --2- 172.21.15.31:0/3461680328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1134109670 0x7f1134109a50 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.884+0000 7f1133e00640 1 -- 172.21.15.31:0/3461680328 >> 172.21.15.31:0/3461680328 conn(0x7f11340ff110 msgr2=0x7f1134101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:54.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.884+0000 7f1133e00640 1 -- 172.21.15.31:0/3461680328 shutdown_connections 2024-03-21T10:38:54.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.884+0000 7f1133e00640 1 -- 172.21.15.31:0/3461680328 wait complete. 2024-03-21T10:38:54.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.885+0000 7f1133e00640 1 Processor -- start 2024-03-21T10:38:54.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.885+0000 7f1133e00640 1 -- start start 2024-03-21T10:38:54.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.886+0000 7f1133e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1134103660 0x7f1134199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.886+0000 7f1133e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1134104020 0x7f113419a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.886+0000 7f1133e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1134109670 0x7f113419e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.886+0000 7f1133e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1134113a90 con 0x7f1134104020 2024-03-21T10:38:54.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.886+0000 7f1133e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1134113910 con 0x7f1134103660 2024-03-21T10:38:54.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.886+0000 7f1133e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1134113c10 con 0x7f1134109670 2024-03-21T10:38:54.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.886+0000 7f1132a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1134103660 0x7f1134199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.886+0000 7f1132a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1134103660 0x7f1134199f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42452/0 (socket says 172.21.15.31:42452) 2024-03-21T10:38:54.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.886+0000 7f1132a00640 1 -- 172.21.15.31:0/1137749134 learned_addr learned my addr 172.21.15.31:0/1137749134 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:54.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.886+0000 7f1133400640 1 --2- 172.21.15.31:0/1137749134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1134109670 0x7f113419e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.886+0000 7f1132000640 1 --2- 172.21.15.31:0/1137749134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1134104020 0x7f113419a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.886+0000 7f1132a00640 1 -- 172.21.15.31:0/1137749134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1134109670 msgr2=0x7f113419e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:54.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.886+0000 7f1132a00640 1 --2- 172.21.15.31:0/1137749134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1134109670 0x7f113419e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.886+0000 7f1132a00640 1 -- 172.21.15.31:0/1137749134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1134104020 msgr2=0x7f113419a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:54.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.886+0000 7f1132a00640 1 --2- 172.21.15.31:0/1137749134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1134104020 0x7f113419a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.887+0000 7f1132a00640 1 -- 172.21.15.31:0/1137749134 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113419ef80 con 0x7f1134103660 2024-03-21T10:38:54.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.887+0000 7f1133400640 1 --2- 172.21.15.31:0/1137749134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1134109670 0x7f113419e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:54.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.887+0000 7f1132a00640 1 --2- 172.21.15.31:0/1137749134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1134103660 0x7f1134199f90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f112402efd0 tx=0x7f11240042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:54.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.887+0000 7f1138a00640 1 -- 172.21.15.31:0/1137749134 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1124041070 con 0x7f1134103660 2024-03-21T10:38:54.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.887+0000 7f1138a00640 1 -- 172.21.15.31:0/1137749134 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f112402ccc0 con 0x7f1134103660 2024-03-21T10:38:54.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.887+0000 7f1138a00640 1 -- 172.21.15.31:0/1137749134 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1124003d10 con 0x7f1134103660 2024-03-21T10:38:54.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.887+0000 7f1133e00640 1 -- 172.21.15.31:0/1137749134 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113419f210 con 0x7f1134103660 2024-03-21T10:38:54.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.887+0000 7f1133e00640 1 -- 172.21.15.31:0/1137749134 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11341a6b00 con 0x7f1134103660 2024-03-21T10:38:54.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.892+0000 7f113cb86740 1 Processor -- start 2024-03-21T10:38:54.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.892+0000 7f113cb86740 1 -- start start 2024-03-21T10:38:54.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.892+0000 7f1138a00640 1 -- 172.21.15.31:0/1137749134 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) v1 ==== 138030+0+0 (secure 0 0 0) 0x7f1124044070 con 0x7f1134103660 2024-03-21T10:38:54.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.892+0000 7f113cb86740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574779e88b0 0x5574779e8c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.892+0000 7f113cb86740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574779e9260 0x5574779f1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.892+0000 7f113cb86740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5574779f1de0 0x5574779f4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.892+0000 7f113cb86740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55747794c2c0 con 0x5574779e9260 2024-03-21T10:38:54.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.892+0000 7f113cb86740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574778f4c90 con 0x5574779e88b0 2024-03-21T10:38:54.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.892+0000 7f113cb86740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x557477948360 con 0x5574779f1de0 2024-03-21T10:38:54.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.893+0000 7f1133400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574779e88b0 0x5574779e8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.893+0000 7f1133400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574779e88b0 0x5574779e8c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42468/0 (socket says 172.21.15.31:42468) 2024-03-21T10:38:54.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.893+0000 7f1132a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574779e9260 0x5574779f1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.893+0000 7f1133400640 1 -- 172.21.15.31:0/4138717030 learned_addr learned my addr 172.21.15.31:0/4138717030 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:54.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.893+0000 7f1132000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5574779f1de0 0x5574779f4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.893+0000 7f1133400640 1 -- 172.21.15.31:0/4138717030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5574779f1de0 msgr2=0x5574779f4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:54.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.893+0000 7f1133400640 1 --2- 172.21.15.31:0/4138717030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5574779f1de0 0x5574779f4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.893+0000 7f1133400640 1 -- 172.21.15.31:0/4138717030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574779e9260 msgr2=0x5574779f1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:54.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.893+0000 7f1133400640 1 --2- 172.21.15.31:0/4138717030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574779e9260 0x5574779f1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.893+0000 7f1133400640 1 -- 172.21.15.31:0/4138717030 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557477919080 con 0x5574779e88b0 2024-03-21T10:38:54.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.893+0000 7f1133400640 1 --2- 172.21.15.31:0/4138717030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574779e88b0 0x5574779e8c90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f112c005f40 tx=0x7f112c010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34a444e246bc8a45 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:54.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.894+0000 7f111fe00640 1 -- 172.21.15.31:0/4138717030 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c011700 con 0x5574779e88b0 2024-03-21T10:38:54.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.894+0000 7f111fe00640 1 -- 172.21.15.31:0/4138717030 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f112c011d80 con 0x5574779e88b0 2024-03-21T10:38:54.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.894+0000 7f111fe00640 1 -- 172.21.15.31:0/4138717030 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c00e3c0 con 0x5574779e88b0 2024-03-21T10:38:54.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.894+0000 7f1138a00640 1 --2- 172.21.15.31:0/1137749134 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f11180a3120 0x7f11180a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.894+0000 7f113cb86740 1 -- 172.21.15.31:0/4138717030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574779e88b0 msgr2=0x5574779e8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:54.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.894+0000 7f113cb86740 1 --2- 172.21.15.31:0/4138717030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574779e88b0 0x5574779e8c90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f112c005f40 tx=0x7f112c010860 comp rx=0 tx=0).stop 2024-03-21T10:38:54.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.895+0000 7f113cb86740 1 -- 172.21.15.31:0/4138717030 shutdown_connections 2024-03-21T10:38:54.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.895+0000 7f113cb86740 1 --2- 172.21.15.31:0/4138717030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5574779f1de0 0x5574779f4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.895+0000 7f113cb86740 1 --2- 172.21.15.31:0/4138717030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574779e9260 0x5574779f1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.895+0000 7f113cb86740 1 --2- 172.21.15.31:0/4138717030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574779e88b0 0x5574779e8c90 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.895+0000 7f113cb86740 1 -- 172.21.15.31:0/4138717030 >> 172.21.15.31:0/4138717030 conn(0x5574779e2f80 msgr2=0x5574779e3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:54.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.895+0000 7f1138a00640 1 -- 172.21.15.31:0/1137749134 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f11240f16c0 con 0x7f1134103660 2024-03-21T10:38:54.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.895+0000 7f1132a00640 1 --2- 172.21.15.31:0/1137749134 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f11180a3120 0x7f11180a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.895+0000 7f113cb86740 1 -- 172.21.15.31:0/4138717030 shutdown_connections 2024-03-21T10:38:54.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.895+0000 7f113cb86740 1 -- 172.21.15.31:0/4138717030 wait complete. 2024-03-21T10:38:54.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.895+0000 7f113cb86740 1 -- start start 2024-03-21T10:38:54.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.895+0000 7f1132a00640 1 --2- 172.21.15.31:0/1137749134 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f11180a3120 0x7f11180a55e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f11240eeb80 tx=0x7f11240f7480 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:54.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.896+0000 7f113cb86740 1 Processor -- start 2024-03-21T10:38:54.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.896+0000 7f113cb86740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574779e88b0 0x5574779f1700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.896+0000 7f113cb86740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5574779e9260 0x5574779eea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.896+0000 7f113cb86740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574779f1de0 0x5574779ef030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.896+0000 7f113cb86740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574779f4d50 con 0x5574779f1de0 2024-03-21T10:38:54.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.896+0000 7f113cb86740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574779080d0 con 0x5574779e88b0 2024-03-21T10:38:54.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.896+0000 7f113cb86740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574779f4ed0 con 0x5574779e9260 2024-03-21T10:38:54.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.896+0000 7f113cb86740 10 client.0 _ll_register_callbacks cb 0x55747790ce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:38:54.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.896+0000 7f1133400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574779e88b0 0x5574779f1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.896+0000 7f1132000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5574779e9260 0x5574779eea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.896+0000 7f1132000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5574779e9260 0x5574779eea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38494/0 (socket says 172.21.15.31:38494) 2024-03-21T10:38:54.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.896+0000 7f1133400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574779e88b0 0x5574779f1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42482/0 (socket says 172.21.15.31:42482) 2024-03-21T10:38:54.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.896+0000 7f1132000640 1 -- 172.21.15.31:0/3517543318 learned_addr learned my addr 172.21.15.31:0/3517543318 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:54.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.897+0000 7f1133400640 1 --2- 172.21.15.31:0/3517543318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574779f1de0 0x5574779ef030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.897+0000 7f1132000640 1 -- 172.21.15.31:0/3517543318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574779e88b0 msgr2=0x5574779f1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:54.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.897+0000 7f1132000640 1 --2- 172.21.15.31:0/3517543318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574779e88b0 0x5574779f1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.897+0000 7f1132000640 1 -- 172.21.15.31:0/3517543318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574779f1de0 msgr2=0x5574779ef030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:54.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.897+0000 7f1132000640 1 --2- 172.21.15.31:0/3517543318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574779f1de0 0x5574779ef030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.897+0000 7f1132000640 1 -- 172.21.15.31:0/3517543318 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5574776603f0 con 0x5574779e9260 2024-03-21T10:38:54.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.897+0000 7f1132000640 1 --2- 172.21.15.31:0/3517543318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5574779e9260 0x5574779eea90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1120013710 tx=0x7f1120013bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=578ac1aa4f379b2e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:54.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.897+0000 7f111cc00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:38:54.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.897+0000 7f111cc00640 1 -- 172.21.15.31:0/3517543318 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1120011e90 con 0x5574779e9260 2024-03-21T10:38:54.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.898+0000 7f111cc00640 1 -- 172.21.15.31:0/3517543318 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f1120004020 con 0x5574779e9260 2024-03-21T10:38:54.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.898+0000 7f113cb86740 1 -- 172.21.15.31:0/3517543318 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f112c011d80 con 0x5574779e9260 2024-03-21T10:38:54.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.898+0000 7f111cc00640 1 -- 172.21.15.31:0/3517543318 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1120004260 con 0x5574779e9260 2024-03-21T10:38:54.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.898+0000 7f113cb86740 10 client.25714 fetch_fsmap learned FSMap version 24 2024-03-21T10:38:54.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.898+0000 7f113cb86740 1 -- 172.21.15.31:0/3517543318 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55747772a7d0 con 0x5574779e9260 2024-03-21T10:38:54.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.898+0000 7f111cc00640 1 -- 172.21.15.31:0/3517543318 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f112c011d80 con 0x5574779e9260 2024-03-21T10:38:54.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.898+0000 7f111cc00640 1 -- 172.21.15.31:0/3517543318 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f112001a070 con 0x5574779e9260 2024-03-21T10:38:54.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.898+0000 7f113cb86740 10 client.25714 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:38:54.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.898+0000 7f113cb86740 10 client.25714 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:38:54.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.898+0000 7f113cb86740 20 client.25714 populate_metadata read hostname 'smithi031' 2024-03-21T10:38:54.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.898+0000 7f113cb86740 1 --2- 172.21.15.31:0/3517543318 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x557477a01cc0 0x557477a040e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.898+0000 7f113cb86740 4 client.25714 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:38:54.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.898+0000 7f113cb86740 1 -- 172.21.15.31:0/3517543318 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557477959800 con 0x557477a01cc0 2024-03-21T10:38:54.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.899+0000 7f1133400640 1 --2- 172.21.15.31:0/3517543318 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x557477a01cc0 0x557477a040e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.899+0000 7f1133400640 1 --2- 172.21.15.31:0/3517543318 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x557477a01cc0 0x557477a040e0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e458ad97627a2f02 server_cookie=f802f84635eba86a in_seq=0 out_seq=0 2024-03-21T10:38:54.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.899+0000 7f111cc00640 10 client.25714 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:38:54.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.900+0000 7f111cc00640 1 -- 172.21.15.31:0/3517543318 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557477959800 con 0x557477a01cc0 2024-03-21T10:38:54.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.900+0000 7f111cc00640 10 client.25714 handle_command_reply: tid=0 2024-03-21T10:38:54.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.900+0000 7f113cb86740 1 client.25714 shutdown 2024-03-21T10:38:54.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.901+0000 7f113cb86740 1 -- 172.21.15.31:0/3517543318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5574779e9260 msgr2=0x5574779eea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:54.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.901+0000 7f113cb86740 1 --2- 172.21.15.31:0/3517543318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5574779e9260 0x5574779eea90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1120013710 tx=0x7f1120013bd0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.901+0000 7f113cb86740 1 -- 172.21.15.31:0/3517543318 shutdown_connections 2024-03-21T10:38:54.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.901+0000 7f113cb86740 1 --2- 172.21.15.31:0/3517543318 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x557477a01cc0 0x557477a040e0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.901+0000 7f113cb86740 1 --2- 172.21.15.31:0/3517543318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574779f1de0 0x5574779ef030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.901+0000 7f113cb86740 1 --2- 172.21.15.31:0/3517543318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5574779e9260 0x5574779eea90 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.901+0000 7f113cb86740 1 --2- 172.21.15.31:0/3517543318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574779e88b0 0x5574779f1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.901+0000 7f113cb86740 1 -- 172.21.15.31:0/3517543318 >> 172.21.15.31:0/3517543318 conn(0x5574779e2f80 msgr2=0x5574779e4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:54.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.901+0000 7f111cc00640 0 client.25714 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:38:54.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.901+0000 7f113cb86740 1 -- 172.21.15.31:0/3517543318 shutdown_connections 2024-03-21T10:38:54.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.901+0000 7f113cb86740 1 -- 172.21.15.31:0/3517543318 wait complete. 2024-03-21T10:38:54.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.901+0000 7f113cb86740 20 client.25714 trim_cache size 0 max 16384 2024-03-21T10:38:54.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.921+0000 7f113cb86740 1 Processor -- start 2024-03-21T10:38:54.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.921+0000 7f113cb86740 1 -- start start 2024-03-21T10:38:54.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.921+0000 7f113cb86740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5574779e88b0 0x5574779e23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.921+0000 7f113cb86740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574779e9260 0x5574779f11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.921+0000 7f113cb86740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574779f1de0 0x5574779f1700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.921+0000 7f113cb86740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11341b2e60 con 0x5574779e9260 2024-03-21T10:38:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.921+0000 7f113cb86740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11341b0960 con 0x5574779f1de0 2024-03-21T10:38:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.921+0000 7f113cb86740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11341aea00 con 0x5574779e88b0 2024-03-21T10:38:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.922+0000 7f1133400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5574779e88b0 0x5574779e23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.922+0000 7f1133400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5574779e88b0 0x5574779e23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38506/0 (socket says 172.21.15.31:38506) 2024-03-21T10:38:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.922+0000 7f1133400640 1 -- 172.21.15.31:0/36479087 learned_addr learned my addr 172.21.15.31:0/36479087 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.922+0000 7f1132000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574779e9260 0x5574779f11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.922+0000 7f1133400640 1 --2- 172.21.15.31:0/36479087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574779f1de0 0x5574779f1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.922+0000 7f1133400640 1 -- 172.21.15.31:0/36479087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574779f1de0 msgr2=0x5574779f1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.922+0000 7f1133400640 1 --2- 172.21.15.31:0/36479087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574779f1de0 0x5574779f1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.922+0000 7f1133400640 1 -- 172.21.15.31:0/36479087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574779e9260 msgr2=0x5574779f11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:54.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.922+0000 7f1133400640 1 --2- 172.21.15.31:0/36479087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574779e9260 0x5574779f11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.922+0000 7f1133400640 1 -- 172.21.15.31:0/36479087 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574779ef3a0 con 0x5574779e88b0 2024-03-21T10:38:54.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.923+0000 7f1133400640 1 --2- 172.21.15.31:0/36479087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5574779e88b0 0x5574779e23a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f112c009dc0 tx=0x7f112c009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4c014bf2a271684 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:54.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.923+0000 7f111f400640 1 -- 172.21.15.31:0/36479087 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c02f0c0 con 0x5574779e88b0 2024-03-21T10:38:54.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.923+0000 7f111f400640 1 -- 172.21.15.31:0/36479087 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f112c02f260 con 0x5574779e88b0 2024-03-21T10:38:54.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.923+0000 7f111f400640 1 -- 172.21.15.31:0/36479087 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c01b430 con 0x5574779e88b0 2024-03-21T10:38:54.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.924+0000 7f113cb86740 1 -- 172.21.15.31:0/36479087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5574779e88b0 msgr2=0x5574779e23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:54.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.924+0000 7f113cb86740 1 --2- 172.21.15.31:0/36479087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5574779e88b0 0x5574779e23a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f112c009dc0 tx=0x7f112c009500 comp rx=0 tx=0).stop 2024-03-21T10:38:54.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.924+0000 7f113cb86740 1 -- 172.21.15.31:0/36479087 shutdown_connections 2024-03-21T10:38:54.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.924+0000 7f113cb86740 1 --2- 172.21.15.31:0/36479087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574779f1de0 0x5574779f1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.924+0000 7f113cb86740 1 --2- 172.21.15.31:0/36479087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574779e9260 0x5574779f11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.924+0000 7f113cb86740 1 --2- 172.21.15.31:0/36479087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5574779e88b0 0x5574779e23a0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.924+0000 7f113cb86740 1 -- 172.21.15.31:0/36479087 >> 172.21.15.31:0/36479087 conn(0x5574779e2f80 msgr2=0x5574779e4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:54.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.924+0000 7f113cb86740 1 -- 172.21.15.31:0/36479087 shutdown_connections 2024-03-21T10:38:54.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.924+0000 7f113cb86740 1 -- 172.21.15.31:0/36479087 wait complete. 2024-03-21T10:38:54.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.924+0000 7f113cb86740 1 -- start start 2024-03-21T10:38:54.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.924+0000 7f113cb86740 1 Processor -- start 2024-03-21T10:38:54.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.925+0000 7f113cb86740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574779e88b0 0x5574779f8890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.925+0000 7f113cb86740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5574779e9260 0x5574779f8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.925+0000 7f113cb86740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574779f1de0 0x557477a5d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.925+0000 7f113cb86740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11341af000 con 0x5574779e88b0 2024-03-21T10:38:54.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.925+0000 7f113cb86740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11341b0300 con 0x5574779f1de0 2024-03-21T10:38:54.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.925+0000 7f113cb86740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11341af640 con 0x5574779e9260 2024-03-21T10:38:54.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.925+0000 7f113cb86740 10 client.0 _ll_register_callbacks cb 0x7f11200096e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:38:54.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.925+0000 7f1132000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5574779e9260 0x5574779f8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.925+0000 7f1133400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574779e88b0 0x5574779f8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.925+0000 7f1132000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5574779e9260 0x5574779f8dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38514/0 (socket says 172.21.15.31:38514) 2024-03-21T10:38:54.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.925+0000 7f1132000640 1 -- 172.21.15.31:0/3142083106 learned_addr learned my addr 172.21.15.31:0/3142083106 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:54.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.925+0000 7f1133400640 1 --2- 172.21.15.31:0/3142083106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574779f1de0 0x557477a5d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.926+0000 7f1132000640 1 -- 172.21.15.31:0/3142083106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574779f1de0 msgr2=0x557477a5d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:54.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.926+0000 7f1132000640 1 --2- 172.21.15.31:0/3142083106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574779f1de0 0x557477a5d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.926+0000 7f1132000640 1 -- 172.21.15.31:0/3142083106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574779e88b0 msgr2=0x5574779f8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:54.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.926+0000 7f1132000640 1 --2- 172.21.15.31:0/3142083106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574779e88b0 0x5574779f8890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.926+0000 7f1132000640 1 -- 172.21.15.31:0/3142083106 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5574779f9690 con 0x5574779e9260 2024-03-21T10:38:54.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.926+0000 7f1133400640 1 --2- 172.21.15.31:0/3142083106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574779e88b0 0x5574779f8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:54.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.926+0000 7f1132000640 1 --2- 172.21.15.31:0/3142083106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5574779e9260 0x5574779f8dd0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1120000c00 tx=0x7f11200117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6846fcc2f7de8c93 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:54.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.926+0000 7f111cc00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:38:54.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.926+0000 7f111cc00640 1 -- 172.21.15.31:0/3142083106 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112002d070 con 0x5574779e9260 2024-03-21T10:38:54.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.926+0000 7f111cc00640 1 -- 172.21.15.31:0/3142083106 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f1120005250 con 0x5574779e9260 2024-03-21T10:38:54.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.926+0000 7f113cb86740 1 -- 172.21.15.31:0/3142083106 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f112c02f260 con 0x5574779e9260 2024-03-21T10:38:54.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.927+0000 7f111cc00640 1 -- 172.21.15.31:0/3142083106 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1120039050 con 0x5574779e9260 2024-03-21T10:38:54.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.927+0000 7f113cb86740 10 client.25720 fetch_fsmap learned FSMap version 24 2024-03-21T10:38:54.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.927+0000 7f113cb86740 1 -- 172.21.15.31:0/3142083106 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x557477a5ebc0 con 0x5574779e9260 2024-03-21T10:38:54.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.927+0000 7f111cc00640 1 -- 172.21.15.31:0/3142083106 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f112c02f260 con 0x5574779e9260 2024-03-21T10:38:54.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.927+0000 7f111cc00640 1 -- 172.21.15.31:0/3142083106 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f112001a040 con 0x5574779e9260 2024-03-21T10:38:54.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.927+0000 7f113cb86740 10 client.25720 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:38:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.927+0000 7f113cb86740 10 client.25720 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:38:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.927+0000 7f113cb86740 20 client.25720 populate_metadata read hostname 'smithi031' 2024-03-21T10:38:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.927+0000 7f113cb86740 1 --2- 172.21.15.31:0/3142083106 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x557477a5f2c0 0x557477a616e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.927+0000 7f113cb86740 4 client.25720 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:38:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.927+0000 7f113cb86740 1 -- 172.21.15.31:0/3142083106 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557477953b50 con 0x557477a5f2c0 2024-03-21T10:38:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.928+0000 7f1133400640 1 --2- 172.21.15.31:0/3142083106 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x557477a5f2c0 0x557477a616e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.928+0000 7f1133400640 1 --2- 172.21.15.31:0/3142083106 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x557477a5f2c0 0x557477a616e0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=96abc96675ba1139 server_cookie=7311a5ca4af24a50 in_seq=0 out_seq=0 2024-03-21T10:38:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.928+0000 7f111cc00640 10 client.25720 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:38:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.928+0000 7f111cc00640 1 -- 172.21.15.31:0/3142083106 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557477953b50 con 0x557477a5f2c0 2024-03-21T10:38:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.928+0000 7f111cc00640 10 client.25720 handle_command_reply: tid=0 2024-03-21T10:38:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.928+0000 7f113cb86740 1 client.25720 shutdown 2024-03-21T10:38:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.929+0000 7f113cb86740 1 -- 172.21.15.31:0/3142083106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5574779e9260 msgr2=0x5574779f8dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.929+0000 7f113cb86740 1 --2- 172.21.15.31:0/3142083106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5574779e9260 0x5574779f8dd0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1120000c00 tx=0x7f11200117b0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.929+0000 7f113cb86740 1 -- 172.21.15.31:0/3142083106 shutdown_connections 2024-03-21T10:38:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.929+0000 7f113cb86740 1 --2- 172.21.15.31:0/3142083106 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x557477a5f2c0 0x557477a616e0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.929+0000 7f113cb86740 1 --2- 172.21.15.31:0/3142083106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574779f1de0 0x557477a5d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.929+0000 7f113cb86740 1 --2- 172.21.15.31:0/3142083106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5574779e9260 0x5574779f8dd0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.929+0000 7f111cc00640 0 client.25720 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:38:54.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.929+0000 7f113cb86740 1 --2- 172.21.15.31:0/3142083106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574779e88b0 0x5574779f8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.929+0000 7f113cb86740 1 -- 172.21.15.31:0/3142083106 >> 172.21.15.31:0/3142083106 conn(0x5574779e2f80 msgr2=0x5574779e9640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:54.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.929+0000 7f113cb86740 1 -- 172.21.15.31:0/3142083106 shutdown_connections 2024-03-21T10:38:54.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.929+0000 7f113cb86740 1 -- 172.21.15.31:0/3142083106 wait complete. 2024-03-21T10:38:54.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.929+0000 7f113cb86740 20 client.25720 trim_cache size 0 max 16384 2024-03-21T10:38:54.930 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:38:54.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.930+0000 7f1133e00640 1 -- 172.21.15.31:0/1137749134 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f11180a3120 msgr2=0x7f11180a55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:54.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.930+0000 7f1133e00640 1 --2- 172.21.15.31:0/1137749134 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f11180a3120 0x7f11180a55e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f11240eeb80 tx=0x7f11240f7480 comp rx=0 tx=0).stop 2024-03-21T10:38:54.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.930+0000 7f1133e00640 1 -- 172.21.15.31:0/1137749134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1134103660 msgr2=0x7f1134199f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:54.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.930+0000 7f1133e00640 1 --2- 172.21.15.31:0/1137749134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1134103660 0x7f1134199f90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f112402efd0 tx=0x7f11240042f0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.930+0000 7f1133e00640 1 -- 172.21.15.31:0/1137749134 shutdown_connections 2024-03-21T10:38:54.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.930+0000 7f1133e00640 1 --2- 172.21.15.31:0/1137749134 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f11180a3120 0x7f11180a55e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.930+0000 7f1133e00640 1 --2- 172.21.15.31:0/1137749134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1134109670 0x7f113419e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.930+0000 7f1133e00640 1 --2- 172.21.15.31:0/1137749134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1134104020 0x7f113419a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.930+0000 7f1133e00640 1 --2- 172.21.15.31:0/1137749134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1134103660 0x7f1134199f90 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:54.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.930+0000 7f1133e00640 1 -- 172.21.15.31:0/1137749134 >> 172.21.15.31:0/1137749134 conn(0x7f11340ff110 msgr2=0x7f11340ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:54.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.930+0000 7f1133e00640 1 -- 172.21.15.31:0/1137749134 shutdown_connections 2024-03-21T10:38:54.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:54.930+0000 7f1133e00640 1 -- 172.21.15.31:0/1137749134 wait complete. 2024-03-21T10:38:54.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:54 smithi031 ceph-mon[31673]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:38:54.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:54 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/914258114' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:38:54.954 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:38:54.954 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-21T10:38:55.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.182+0000 7f2071a00640 1 -- 172.21.15.31:0/1345484341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f206c107500 msgr2=0x7f206c1078e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:55.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.182+0000 7f2071a00640 1 --2- 172.21.15.31:0/1345484341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f206c107500 0x7f206c1078e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f205c009900 tx=0x7f205c02bc30 comp rx=0 tx=0).stop 2024-03-21T10:38:55.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.182+0000 7f2071a00640 1 -- 172.21.15.31:0/1345484341 shutdown_connections 2024-03-21T10:38:55.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.182+0000 7f2071a00640 1 --2- 172.21.15.31:0/1345484341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f206c101e20 0x7f206c10e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.182+0000 7f2071a00640 1 --2- 172.21.15.31:0/1345484341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f206c101460 0x7f206c1018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.182+0000 7f2071a00640 1 --2- 172.21.15.31:0/1345484341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f206c107500 0x7f206c1078e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.182+0000 7f2071a00640 1 -- 172.21.15.31:0/1345484341 >> 172.21.15.31:0/1345484341 conn(0x7f206c0fcff0 msgr2=0x7f206c0ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:55.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.182+0000 7f2071a00640 1 -- 172.21.15.31:0/1345484341 shutdown_connections 2024-03-21T10:38:55.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.182+0000 7f2071a00640 1 -- 172.21.15.31:0/1345484341 wait complete. 2024-03-21T10:38:55.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:54 smithi040 ceph-mon[36154]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:38:55.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:54 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/914258114' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:38:55.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.183+0000 7f2071a00640 1 Processor -- start 2024-03-21T10:38:55.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.183+0000 7f2071a00640 1 -- start start 2024-03-21T10:38:55.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.184+0000 7f2071a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f206c101460 0x7f206c19a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:55.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.184+0000 7f2071a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f206c101e20 0x7f206c19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:55.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.184+0000 7f2071a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f206c107500 0x7f206c19e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:55.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.184+0000 7f2071a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206c0741d0 con 0x7f206c101e20 2024-03-21T10:38:55.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.184+0000 7f2071a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206c074050 con 0x7f206c101460 2024-03-21T10:38:55.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.184+0000 7f2071a00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206c074350 con 0x7f206c107500 2024-03-21T10:38:55.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.184+0000 7f206be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f206c101460 0x7f206c19a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:55.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.184+0000 7f206be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f206c101460 0x7f206c19a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42516/0 (socket says 172.21.15.31:42516) 2024-03-21T10:38:55.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.184+0000 7f206be00640 1 -- 172.21.15.31:0/3030534175 learned_addr learned my addr 172.21.15.31:0/3030534175 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:55.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.184+0000 7f2071000640 1 --2- 172.21.15.31:0/3030534175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f206c107500 0x7f206c19e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:55.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.184+0000 7f206b400640 1 --2- 172.21.15.31:0/3030534175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f206c101e20 0x7f206c19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:55.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.184+0000 7f206be00640 1 -- 172.21.15.31:0/3030534175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f206c107500 msgr2=0x7f206c19e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:55.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.184+0000 7f206be00640 1 --2- 172.21.15.31:0/3030534175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f206c107500 0x7f206c19e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.184+0000 7f206be00640 1 -- 172.21.15.31:0/3030534175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f206c101e20 msgr2=0x7f206c19a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:55.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.184+0000 7f206be00640 1 --2- 172.21.15.31:0/3030534175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f206c101e20 0x7f206c19a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.184+0000 7f206be00640 1 -- 172.21.15.31:0/3030534175 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f206c19efb0 con 0x7f206c101460 2024-03-21T10:38:55.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.184+0000 7f206b400640 1 --2- 172.21.15.31:0/3030534175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f206c101e20 0x7f206c19a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:55.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.185+0000 7f206be00640 1 --2- 172.21.15.31:0/3030534175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f206c101460 0x7f206c19a250 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f205c02efd0 tx=0x7f205c0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:55.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.185+0000 7f2068c00640 1 -- 172.21.15.31:0/3030534175 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205c041070 con 0x7f206c101460 2024-03-21T10:38:55.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.185+0000 7f2068c00640 1 -- 172.21.15.31:0/3030534175 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f205c02ccc0 con 0x7f206c101460 2024-03-21T10:38:55.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.185+0000 7f2068c00640 1 -- 172.21.15.31:0/3030534175 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205c003d40 con 0x7f206c101460 2024-03-21T10:38:55.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.185+0000 7f2071a00640 1 -- 172.21.15.31:0/3030534175 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f206c19f240 con 0x7f206c101460 2024-03-21T10:38:55.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.185+0000 7f2071a00640 1 -- 172.21.15.31:0/3030534175 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f206c1a6af0 con 0x7f206c101460 2024-03-21T10:38:55.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.189+0000 7f2075bf1740 1 Processor -- start 2024-03-21T10:38:55.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.189+0000 7f2068c00640 1 -- 172.21.15.31:0/3030534175 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) v1 ==== 138030+0+0 (secure 0 0 0) 0x7f205c044070 con 0x7f206c101460 2024-03-21T10:38:55.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.189+0000 7f2075bf1740 1 -- start start 2024-03-21T10:38:55.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.189+0000 7f2075bf1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e2db718b0 0x563e2db71c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:55.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.189+0000 7f2075bf1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e2db72260 0x563e2db7a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:55.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.189+0000 7f2075bf1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2db7ade0 0x563e2db7d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:55.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.189+0000 7f2075bf1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e2dad52c0 con 0x563e2db718b0 2024-03-21T10:38:55.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.189+0000 7f2075bf1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e2da7dc90 con 0x563e2db72260 2024-03-21T10:38:55.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.189+0000 7f2075bf1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e2dad1360 con 0x563e2db7ade0 2024-03-21T10:38:55.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.190+0000 7f206be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e2db72260 0x563e2db7a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:55.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.190+0000 7f2071000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e2db718b0 0x563e2db71c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:55.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.190+0000 7f206b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2db7ade0 0x563e2db7d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:55.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.190+0000 7f206be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e2db72260 0x563e2db7a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42524/0 (socket says 172.21.15.31:42524) 2024-03-21T10:38:55.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.190+0000 7f206b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2db7ade0 0x563e2db7d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38540/0 (socket says 172.21.15.31:38540) 2024-03-21T10:38:55.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.190+0000 7f206be00640 1 -- 172.21.15.31:0/3184487552 learned_addr learned my addr 172.21.15.31:0/3184487552 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:55.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.190+0000 7f206be00640 1 -- 172.21.15.31:0/3184487552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2db7ade0 msgr2=0x563e2db7d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:55.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.190+0000 7f206be00640 1 --2- 172.21.15.31:0/3184487552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2db7ade0 0x563e2db7d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.190+0000 7f206be00640 1 -- 172.21.15.31:0/3184487552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e2db718b0 msgr2=0x563e2db71c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:55.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.190+0000 7f206be00640 1 --2- 172.21.15.31:0/3184487552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e2db718b0 0x563e2db71c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.190+0000 7f206be00640 1 -- 172.21.15.31:0/3184487552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563e2daa2080 con 0x563e2db72260 2024-03-21T10:38:55.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.190+0000 7f206be00640 1 --2- 172.21.15.31:0/3184487552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e2db72260 0x563e2db7a810 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f205c0eea30 tx=0x7f205c0f7a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36739094e064c539 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:55.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.191+0000 7f2061600640 1 -- 172.21.15.31:0/3184487552 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205c100070 con 0x563e2db72260 2024-03-21T10:38:55.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.191+0000 7f2061600640 1 -- 172.21.15.31:0/3184487552 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f205c0f35b0 con 0x563e2db72260 2024-03-21T10:38:55.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.191+0000 7f2061600640 1 -- 172.21.15.31:0/3184487552 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205c0f9c80 con 0x563e2db72260 2024-03-21T10:38:55.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.191+0000 7f2068c00640 1 --2- 172.21.15.31:0/3030534175 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f20500a31a0 0x7f20500a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:55.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.191+0000 7f2068c00640 1 -- 172.21.15.31:0/3030534175 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f205c0f16c0 con 0x7f206c101460 2024-03-21T10:38:55.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.191+0000 7f2071000640 1 --2- 172.21.15.31:0/3030534175 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f20500a31a0 0x7f20500a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:55.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.191+0000 7f2075bf1740 1 -- 172.21.15.31:0/3184487552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e2db72260 msgr2=0x563e2db7a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:55.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.191+0000 7f2075bf1740 1 --2- 172.21.15.31:0/3184487552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e2db72260 0x563e2db7a810 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f205c0eea30 tx=0x7f205c0f7a70 comp rx=0 tx=0).stop 2024-03-21T10:38:55.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.192+0000 7f2075bf1740 1 -- 172.21.15.31:0/3184487552 shutdown_connections 2024-03-21T10:38:55.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.192+0000 7f2075bf1740 1 --2- 172.21.15.31:0/3184487552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2db7ade0 0x563e2db7d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.192+0000 7f2075bf1740 1 --2- 172.21.15.31:0/3184487552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e2db72260 0x563e2db7a810 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.192+0000 7f2075bf1740 1 --2- 172.21.15.31:0/3184487552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e2db718b0 0x563e2db71c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.192+0000 7f2075bf1740 1 -- 172.21.15.31:0/3184487552 >> 172.21.15.31:0/3184487552 conn(0x563e2db6bf80 msgr2=0x563e2db6c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:55.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.192+0000 7f2075bf1740 1 -- 172.21.15.31:0/3184487552 shutdown_connections 2024-03-21T10:38:55.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.192+0000 7f2071000640 1 --2- 172.21.15.31:0/3030534175 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f20500a31a0 0x7f20500a5660 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2064005fd0 tx=0x7f2064009500 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:55.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.192+0000 7f2075bf1740 1 -- 172.21.15.31:0/3184487552 wait complete. 2024-03-21T10:38:55.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.192+0000 7f2075bf1740 1 -- start start 2024-03-21T10:38:55.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.193+0000 7f2075bf1740 1 Processor -- start 2024-03-21T10:38:55.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.193+0000 7f2075bf1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e2db718b0 0x563e2db7a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:55.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.193+0000 7f2075bf1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2db72260 0x563e2db77a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:55.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.193+0000 7f2075bf1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e2db7ade0 0x563e2db77fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:55.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.193+0000 7f2075bf1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e2db7db10 con 0x563e2db7ade0 2024-03-21T10:38:55.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.193+0000 7f2075bf1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e2da910d0 con 0x563e2db718b0 2024-03-21T10:38:55.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.193+0000 7f2075bf1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e2db7dc90 con 0x563e2db72260 2024-03-21T10:38:55.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.193+0000 7f2075bf1740 10 client.0 _ll_register_callbacks cb 0x563e2da95e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:38:55.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.193+0000 7f206be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e2db718b0 0x563e2db7a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:55.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.193+0000 7f206b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2db72260 0x563e2db77a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:55.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.193+0000 7f206be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e2db718b0 0x563e2db7a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42526/0 (socket says 172.21.15.31:42526) 2024-03-21T10:38:55.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.193+0000 7f206be00640 1 -- 172.21.15.31:0/3147771603 learned_addr learned my addr 172.21.15.31:0/3147771603 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:55.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.193+0000 7f206b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2db72260 0x563e2db77a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38544/0 (socket says 172.21.15.31:38544) 2024-03-21T10:38:55.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.193+0000 7f2071000640 1 --2- 172.21.15.31:0/3147771603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e2db7ade0 0x563e2db77fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:55.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.194+0000 7f206be00640 1 -- 172.21.15.31:0/3147771603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2db72260 msgr2=0x563e2db77a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:55.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.194+0000 7f206be00640 1 --2- 172.21.15.31:0/3147771603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2db72260 0x563e2db77a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.194+0000 7f206be00640 1 -- 172.21.15.31:0/3147771603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e2db7ade0 msgr2=0x563e2db77fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:55.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.194+0000 7f206be00640 1 --2- 172.21.15.31:0/3147771603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e2db7ade0 0x563e2db77fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.194+0000 7f206be00640 1 -- 172.21.15.31:0/3147771603 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563e2d7e93f0 con 0x563e2db718b0 2024-03-21T10:38:55.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.194+0000 7f206be00640 1 --2- 172.21.15.31:0/3147771603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e2db718b0 0x563e2db7a6d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f205c0b9b10 tx=0x7f205c0f9520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5e3cef4a6c3b7f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:55.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.194+0000 7f2056000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:38:55.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.194+0000 7f2056000640 1 -- 172.21.15.31:0/3147771603 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205c04b070 con 0x563e2db718b0 2024-03-21T10:38:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.194+0000 7f2056000640 1 -- 172.21.15.31:0/3147771603 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f205c0bb9b0 con 0x563e2db718b0 2024-03-21T10:38:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.194+0000 7f2075bf1740 1 -- 172.21.15.31:0/3147771603 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f205c0f35b0 con 0x563e2db718b0 2024-03-21T10:38:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.195+0000 7f2056000640 1 -- 172.21.15.31:0/3147771603 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f205c100070 con 0x563e2db718b0 2024-03-21T10:38:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.195+0000 7f2075bf1740 10 client.25745 fetch_fsmap learned FSMap version 24 2024-03-21T10:38:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.195+0000 7f2075bf1740 1 -- 172.21.15.31:0/3147771603 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x563e2d8b37d0 con 0x563e2db718b0 2024-03-21T10:38:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.195+0000 7f2056000640 1 -- 172.21.15.31:0/3147771603 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f205c0f35b0 con 0x563e2db718b0 2024-03-21T10:38:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.195+0000 7f2056000640 1 -- 172.21.15.31:0/3147771603 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f205c0f1270 con 0x563e2db718b0 2024-03-21T10:38:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.195+0000 7f2075bf1740 10 client.25745 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:38:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.195+0000 7f2075bf1740 10 client.25745 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:38:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.195+0000 7f2075bf1740 20 client.25745 populate_metadata read hostname 'smithi031' 2024-03-21T10:38:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.195+0000 7f2075bf1740 1 --2- 172.21.15.31:0/3147771603 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563e2db8a800 0x563e2db8cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.195+0000 7f2075bf1740 4 client.25745 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:38:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.195+0000 7f2075bf1740 1 -- 172.21.15.31:0/3147771603 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563e2dae2800 con 0x563e2db8a800 2024-03-21T10:38:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.195+0000 7f206b400640 1 --2- 172.21.15.31:0/3147771603 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563e2db8a800 0x563e2db8cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.196+0000 7f206b400640 1 --2- 172.21.15.31:0/3147771603 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563e2db8a800 0x563e2db8cbf0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e0a6a21a3538b4d server_cookie=bc728c3b16e1ca31 in_seq=0 out_seq=0 2024-03-21T10:38:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.196+0000 7f2056000640 10 client.25745 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:55.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.197+0000 7f2056000640 1 -- 172.21.15.31:0/3147771603 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563e2dae2800 con 0x563e2db8a800 2024-03-21T10:38:55.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.197+0000 7f2056000640 10 client.25745 handle_command_reply: tid=0 2024-03-21T10:38:55.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.197+0000 7f2075bf1740 1 client.25745 shutdown 2024-03-21T10:38:55.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.197+0000 7f2075bf1740 1 -- 172.21.15.31:0/3147771603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e2db718b0 msgr2=0x563e2db7a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:55.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.197+0000 7f2075bf1740 1 --2- 172.21.15.31:0/3147771603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e2db718b0 0x563e2db7a6d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f205c0b9b10 tx=0x7f205c0f9520 comp rx=0 tx=0).stop 2024-03-21T10:38:55.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.197+0000 7f2075bf1740 1 -- 172.21.15.31:0/3147771603 shutdown_connections 2024-03-21T10:38:55.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.197+0000 7f2075bf1740 1 --2- 172.21.15.31:0/3147771603 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563e2db8a800 0x563e2db8cbf0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.198+0000 7f2075bf1740 1 --2- 172.21.15.31:0/3147771603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e2db7ade0 0x563e2db77fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.198+0000 7f2075bf1740 1 --2- 172.21.15.31:0/3147771603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2db72260 0x563e2db77a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.198+0000 7f2075bf1740 1 --2- 172.21.15.31:0/3147771603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e2db718b0 0x563e2db7a6d0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.198+0000 7f2075bf1740 1 -- 172.21.15.31:0/3147771603 >> 172.21.15.31:0/3147771603 conn(0x563e2db6bf80 msgr2=0x563e2db6dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:55.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.198+0000 7f2056000640 0 client.25745 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:55.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.198+0000 7f2075bf1740 1 -- 172.21.15.31:0/3147771603 shutdown_connections 2024-03-21T10:38:55.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.198+0000 7f2075bf1740 1 -- 172.21.15.31:0/3147771603 wait complete. 2024-03-21T10:38:55.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.198+0000 7f2075bf1740 20 client.25745 trim_cache size 0 max 16384 2024-03-21T10:38:55.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.214+0000 7f2075bf1740 1 Processor -- start 2024-03-21T10:38:55.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.214+0000 7f2075bf1740 1 -- start start 2024-03-21T10:38:55.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.214+0000 7f2075bf1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e2db718b0 0x563e2db7a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:55.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.215+0000 7f2075bf1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e2db72260 0x563e2db7a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:55.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.215+0000 7f2075bf1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2db7ade0 0x563e2db74d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:55.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.215+0000 7f2075bf1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206c1b2e70 con 0x563e2db72260 2024-03-21T10:38:55.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.215+0000 7f2075bf1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206c1b0970 con 0x563e2db718b0 2024-03-21T10:38:55.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.215+0000 7f2075bf1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206c1aea10 con 0x563e2db7ade0 2024-03-21T10:38:55.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.215+0000 7f206be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e2db718b0 0x563e2db7a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:55.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.215+0000 7f206be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e2db718b0 0x563e2db7a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42536/0 (socket says 172.21.15.31:42536) 2024-03-21T10:38:55.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.215+0000 7f206be00640 1 -- 172.21.15.31:0/1557221995 learned_addr learned my addr 172.21.15.31:0/1557221995 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:55.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.215+0000 7f2071000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2db7ade0 0x563e2db74d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:55.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.215+0000 7f206b400640 1 --2- 172.21.15.31:0/1557221995 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e2db72260 0x563e2db7a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:55.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.215+0000 7f206be00640 1 -- 172.21.15.31:0/1557221995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2db7ade0 msgr2=0x563e2db74d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:55.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.215+0000 7f206be00640 1 --2- 172.21.15.31:0/1557221995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2db7ade0 0x563e2db74d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.215+0000 7f206be00640 1 -- 172.21.15.31:0/1557221995 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e2db72260 msgr2=0x563e2db7a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:55.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.215+0000 7f206be00640 1 --2- 172.21.15.31:0/1557221995 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e2db72260 0x563e2db7a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.215+0000 7f206be00640 1 -- 172.21.15.31:0/1557221995 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563e2db76010 con 0x563e2db718b0 2024-03-21T10:38:55.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.215+0000 7f2071000640 1 --2- 172.21.15.31:0/1557221995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2db7ade0 0x563e2db74d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:55.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.216+0000 7f206be00640 1 --2- 172.21.15.31:0/1557221995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e2db718b0 0x563e2db7a190 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f205c0c5b90 tx=0x7f205c02bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8975ff241dad84f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:55.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.216+0000 7f2060c00640 1 -- 172.21.15.31:0/1557221995 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205c04b070 con 0x563e2db718b0 2024-03-21T10:38:55.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.216+0000 7f2060c00640 1 -- 172.21.15.31:0/1557221995 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f205c0bddb0 con 0x563e2db718b0 2024-03-21T10:38:55.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.216+0000 7f2060c00640 1 -- 172.21.15.31:0/1557221995 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205c0bd480 con 0x563e2db718b0 2024-03-21T10:38:55.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.217+0000 7f2075bf1740 1 -- 172.21.15.31:0/1557221995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e2db718b0 msgr2=0x563e2db7a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:55.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.217+0000 7f2075bf1740 1 --2- 172.21.15.31:0/1557221995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e2db718b0 0x563e2db7a190 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f205c0c5b90 tx=0x7f205c02bc30 comp rx=0 tx=0).stop 2024-03-21T10:38:55.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.217+0000 7f2075bf1740 1 -- 172.21.15.31:0/1557221995 shutdown_connections 2024-03-21T10:38:55.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.217+0000 7f2075bf1740 1 --2- 172.21.15.31:0/1557221995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2db7ade0 0x563e2db74d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.217+0000 7f2075bf1740 1 --2- 172.21.15.31:0/1557221995 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e2db72260 0x563e2db7a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.217+0000 7f2075bf1740 1 --2- 172.21.15.31:0/1557221995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e2db718b0 0x563e2db7a190 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.217+0000 7f2075bf1740 1 -- 172.21.15.31:0/1557221995 >> 172.21.15.31:0/1557221995 conn(0x563e2db6bf80 msgr2=0x563e2db6b160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:55.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.217+0000 7f2075bf1740 1 -- 172.21.15.31:0/1557221995 shutdown_connections 2024-03-21T10:38:55.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.217+0000 7f2075bf1740 1 -- 172.21.15.31:0/1557221995 wait complete. 2024-03-21T10:38:55.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.218+0000 7f2075bf1740 1 -- start start 2024-03-21T10:38:55.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.218+0000 7f2075bf1740 1 Processor -- start 2024-03-21T10:38:55.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.219+0000 7f2075bf1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2db718b0 0x563e2db80a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:55.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.219+0000 7f2075bf1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e2db72260 0x563e2db80f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:55.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.219+0000 7f2075bf1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e2db7ade0 0x563e2dbe4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:55.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.219+0000 7f2075bf1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206c1af010 con 0x563e2db7ade0 2024-03-21T10:38:55.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.219+0000 7f2075bf1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206c1b0310 con 0x563e2db72260 2024-03-21T10:38:55.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.219+0000 7f2075bf1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206c1af650 con 0x563e2db718b0 2024-03-21T10:38:55.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.219+0000 7f2075bf1740 10 client.0 _ll_register_callbacks cb 0x7f2064009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:38:55.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.219+0000 7f206be00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2db718b0 0x563e2db80a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:55.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.219+0000 7f206be00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2db718b0 0x563e2db80a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38566/0 (socket says 172.21.15.31:38566) 2024-03-21T10:38:55.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.219+0000 7f206be00640 1 -- 172.21.15.31:0/1610519927 learned_addr learned my addr 172.21.15.31:0/1610519927 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:55.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.219+0000 7f206b400640 1 --2- 172.21.15.31:0/1610519927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e2db72260 0x563e2db80f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:55.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.219+0000 7f2071000640 1 --2- 172.21.15.31:0/1610519927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e2db7ade0 0x563e2dbe4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:55.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.219+0000 7f206be00640 1 -- 172.21.15.31:0/1610519927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e2db72260 msgr2=0x563e2db80f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:55.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.219+0000 7f206be00640 1 --2- 172.21.15.31:0/1610519927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e2db72260 0x563e2db80f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.219+0000 7f206be00640 1 -- 172.21.15.31:0/1610519927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e2db7ade0 msgr2=0x563e2dbe4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:55.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.219+0000 7f206be00640 1 --2- 172.21.15.31:0/1610519927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e2db7ade0 0x563e2dbe4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.219+0000 7f206be00640 1 -- 172.21.15.31:0/1610519927 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563e2dbe5380 con 0x563e2db718b0 2024-03-21T10:38:55.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.219+0000 7f2071000640 1 --2- 172.21.15.31:0/1610519927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e2db7ade0 0x563e2dbe4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:55.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.220+0000 7f206be00640 1 --2- 172.21.15.31:0/1610519927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2db718b0 0x563e2db80a50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f205c0d3040 tx=0x7f205c005b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9b24bf0ca68c046 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:55.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.220+0000 7f2056000640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:38:55.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.220+0000 7f2056000640 1 -- 172.21.15.31:0/1610519927 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205c0da060 con 0x563e2db718b0 2024-03-21T10:38:55.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.220+0000 7f2075bf1740 1 -- 172.21.15.31:0/1610519927 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f205c0bddb0 con 0x563e2db718b0 2024-03-21T10:38:55.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.220+0000 7f2056000640 1 -- 172.21.15.31:0/1610519927 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f205c0f1270 con 0x563e2db718b0 2024-03-21T10:38:55.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.220+0000 7f2056000640 1 -- 172.21.15.31:0/1610519927 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f205c04b070 con 0x563e2db718b0 2024-03-21T10:38:55.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.220+0000 7f2075bf1740 10 client.25738 fetch_fsmap learned FSMap version 24 2024-03-21T10:38:55.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.220+0000 7f2075bf1740 1 -- 172.21.15.31:0/1610519927 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x563e2dbe5380 con 0x563e2db718b0 2024-03-21T10:38:55.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.220+0000 7f2056000640 1 -- 172.21.15.31:0/1610519927 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f205c0bddb0 con 0x563e2db718b0 2024-03-21T10:38:55.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.220+0000 7f2056000640 1 -- 172.21.15.31:0/1610519927 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f205c0c5c30 con 0x563e2db718b0 2024-03-21T10:38:55.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.220+0000 7f2075bf1740 10 client.25738 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:38:55.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.220+0000 7f2075bf1740 10 client.25738 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:38:55.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.220+0000 7f2075bf1740 20 client.25738 populate_metadata read hostname 'smithi031' 2024-03-21T10:38:55.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.220+0000 7f2075bf1740 1 --2- 172.21.15.31:0/1610519927 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563e2dbe68b0 0x563e2dbe8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:55.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.220+0000 7f2075bf1740 4 client.25738 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:38:55.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.220+0000 7f2075bf1740 1 -- 172.21.15.31:0/1610519927 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563e2dadcb50 con 0x563e2dbe68b0 2024-03-21T10:38:55.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.221+0000 7f206b400640 1 --2- 172.21.15.31:0/1610519927 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563e2dbe68b0 0x563e2dbe8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:55.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.221+0000 7f206b400640 1 --2- 172.21.15.31:0/1610519927 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563e2dbe68b0 0x563e2dbe8ca0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60f87a3eae62fd47 server_cookie=d31c5f3b8af262bc in_seq=0 out_seq=0 2024-03-21T10:38:55.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.221+0000 7f2056000640 10 client.25738 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:55.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.221+0000 7f2056000640 1 -- 172.21.15.31:0/1610519927 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563e2dadcb50 con 0x563e2dbe68b0 2024-03-21T10:38:55.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.221+0000 7f2056000640 10 client.25738 handle_command_reply: tid=0 2024-03-21T10:38:55.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.221+0000 7f2075bf1740 1 client.25738 shutdown 2024-03-21T10:38:55.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.222+0000 7f2075bf1740 1 -- 172.21.15.31:0/1610519927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2db718b0 msgr2=0x563e2db80a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:55.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.222+0000 7f2075bf1740 1 --2- 172.21.15.31:0/1610519927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2db718b0 0x563e2db80a50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f205c0d3040 tx=0x7f205c005b70 comp rx=0 tx=0).stop 2024-03-21T10:38:55.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.222+0000 7f2075bf1740 1 -- 172.21.15.31:0/1610519927 shutdown_connections 2024-03-21T10:38:55.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.222+0000 7f2075bf1740 1 --2- 172.21.15.31:0/1610519927 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563e2dbe68b0 0x563e2dbe8ca0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.222+0000 7f2075bf1740 1 --2- 172.21.15.31:0/1610519927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e2db7ade0 0x563e2dbe4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.222+0000 7f2075bf1740 1 --2- 172.21.15.31:0/1610519927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e2db72260 0x563e2db80f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.222+0000 7f2056000640 0 client.25738 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:55.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.222+0000 7f2075bf1740 1 --2- 172.21.15.31:0/1610519927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2db718b0 0x563e2db80a50 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.222+0000 7f2075bf1740 1 -- 172.21.15.31:0/1610519927 >> 172.21.15.31:0/1610519927 conn(0x563e2db6bf80 msgr2=0x563e2db72640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:55.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.222+0000 7f2075bf1740 1 -- 172.21.15.31:0/1610519927 shutdown_connections 2024-03-21T10:38:55.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.222+0000 7f2075bf1740 1 -- 172.21.15.31:0/1610519927 wait complete. 2024-03-21T10:38:55.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.222+0000 7f2075bf1740 20 client.25738 trim_cache size 0 max 16384 2024-03-21T10:38:55.223 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:38:55.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.223+0000 7f2071a00640 1 -- 172.21.15.31:0/3030534175 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f20500a31a0 msgr2=0x7f20500a5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:55.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.223+0000 7f2071a00640 1 --2- 172.21.15.31:0/3030534175 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f20500a31a0 0x7f20500a5660 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2064005fd0 tx=0x7f2064009500 comp rx=0 tx=0).stop 2024-03-21T10:38:55.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.223+0000 7f2071a00640 1 -- 172.21.15.31:0/3030534175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f206c101460 msgr2=0x7f206c19a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:55.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.223+0000 7f2071a00640 1 --2- 172.21.15.31:0/3030534175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f206c101460 0x7f206c19a250 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f205c02efd0 tx=0x7f205c0042f0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.223+0000 7f2071a00640 1 -- 172.21.15.31:0/3030534175 shutdown_connections 2024-03-21T10:38:55.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.223+0000 7f2071a00640 1 --2- 172.21.15.31:0/3030534175 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f20500a31a0 0x7f20500a5660 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.223+0000 7f2071a00640 1 --2- 172.21.15.31:0/3030534175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f206c107500 0x7f206c19e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.223+0000 7f2071a00640 1 --2- 172.21.15.31:0/3030534175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f206c101e20 0x7f206c19a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.223+0000 7f2071a00640 1 --2- 172.21.15.31:0/3030534175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f206c101460 0x7f206c19a250 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:55.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.223+0000 7f2071a00640 1 -- 172.21.15.31:0/3030534175 >> 172.21.15.31:0/3030534175 conn(0x7f206c0fcff0 msgr2=0x7f206c0fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:55.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.223+0000 7f2071a00640 1 -- 172.21.15.31:0/3030534175 shutdown_connections 2024-03-21T10:38:55.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:55.223+0000 7f2071a00640 1 -- 172.21.15.31:0/3030534175 wait complete. 2024-03-21T10:38:55.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:54 smithi070 ceph-mon[36330]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:38:55.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:54 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/914258114' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:38:56.243 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-21T10:38:56.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:38:56.244 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-21T10:38:56.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.471+0000 7fa766e00640 1 -- 172.21.15.31:0/470210204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa760112470 msgr2=0x7fa760114860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:56.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.471+0000 7fa766e00640 1 --2- 172.21.15.31:0/470210204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa760112470 0x7fa760114860 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa758009ca0 tx=0x7fa75802c190 comp rx=0 tx=0).stop 2024-03-21T10:38:56.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.471+0000 7fa766e00640 1 -- 172.21.15.31:0/470210204 shutdown_connections 2024-03-21T10:38:56.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.471+0000 7fa766e00640 1 --2- 172.21.15.31:0/470210204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa760112470 0x7fa760114860 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.471+0000 7fa766e00640 1 --2- 172.21.15.31:0/470210204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa760074360 0x7fa760111f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.471+0000 7fa766e00640 1 --2- 172.21.15.31:0/470210204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa760073a40 0x7fa760073e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.471+0000 7fa766e00640 1 -- 172.21.15.31:0/470210204 >> 172.21.15.31:0/470210204 conn(0x7fa7600fcff0 msgr2=0x7fa7600ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:56.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.472+0000 7fa766e00640 1 -- 172.21.15.31:0/470210204 shutdown_connections 2024-03-21T10:38:56.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.472+0000 7fa766e00640 1 -- 172.21.15.31:0/470210204 wait complete. 2024-03-21T10:38:56.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.473+0000 7fa766e00640 1 Processor -- start 2024-03-21T10:38:56.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.473+0000 7fa766e00640 1 -- start start 2024-03-21T10:38:56.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.474+0000 7fa766e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa760073a40 0x7fa76019e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.474+0000 7fa766e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa760074360 0x7fa76019e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.474+0000 7fa766e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa760112470 0x7fa7601a2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.474+0000 7fa766e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa760117e70 con 0x7fa760074360 2024-03-21T10:38:56.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.474+0000 7fa766e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa760117cf0 con 0x7fa760112470 2024-03-21T10:38:56.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.474+0000 7fa766e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa760117ff0 con 0x7fa760073a40 2024-03-21T10:38:56.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.474+0000 7fa765a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa760073a40 0x7fa76019e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.474+0000 7fa765a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa760073a40 0x7fa76019e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38586/0 (socket says 172.21.15.31:38586) 2024-03-21T10:38:56.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.474+0000 7fa765a00640 1 -- 172.21.15.31:0/2917855760 learned_addr learned my addr 172.21.15.31:0/2917855760 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:56.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.474+0000 7fa766400640 1 --2- 172.21.15.31:0/2917855760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa760112470 0x7fa7601a2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.474+0000 7fa765000640 1 --2- 172.21.15.31:0/2917855760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa760074360 0x7fa76019e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.475+0000 7fa765a00640 1 -- 172.21.15.31:0/2917855760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa760112470 msgr2=0x7fa7601a2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:56.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.475+0000 7fa765a00640 1 --2- 172.21.15.31:0/2917855760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa760112470 0x7fa7601a2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.475+0000 7fa765a00640 1 -- 172.21.15.31:0/2917855760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa760074360 msgr2=0x7fa76019e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:56.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.475+0000 7fa765a00640 1 --2- 172.21.15.31:0/2917855760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa760074360 0x7fa76019e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.475+0000 7fa765a00640 1 -- 172.21.15.31:0/2917855760 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7601a3330 con 0x7fa760073a40 2024-03-21T10:38:56.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.475+0000 7fa765a00640 1 --2- 172.21.15.31:0/2917855760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa760073a40 0x7fa76019e390 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa75400e9f0 tx=0x7fa75400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:56.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.475+0000 7fa75e000640 1 -- 172.21.15.31:0/2917855760 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75400cd80 con 0x7fa760073a40 2024-03-21T10:38:56.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.475+0000 7fa75e000640 1 -- 172.21.15.31:0/2917855760 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa754010910 con 0x7fa760073a40 2024-03-21T10:38:56.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.475+0000 7fa75e000640 1 -- 172.21.15.31:0/2917855760 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa754018990 con 0x7fa760073a40 2024-03-21T10:38:56.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.475+0000 7fa766e00640 1 -- 172.21.15.31:0/2917855760 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7601a3620 con 0x7fa760073a40 2024-03-21T10:38:56.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.475+0000 7fa766e00640 1 -- 172.21.15.31:0/2917855760 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7601aaeb0 con 0x7fa760073a40 2024-03-21T10:38:56.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.478+0000 7fa75e000640 1 -- 172.21.15.31:0/2917855760 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) v1 ==== 138030+0+0 (secure 0 0 0) 0x7fa754010ac0 con 0x7fa760073a40 2024-03-21T10:38:56.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.479+0000 7fa76b1d1740 1 Processor -- start 2024-03-21T10:38:56.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.479+0000 7fa76b1d1740 1 -- start start 2024-03-21T10:38:56.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.479+0000 7fa76b1d1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3408ad290 0x55a3408ad670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.480+0000 7fa76b1d1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3408adbb0 0x55a3408b6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.480+0000 7fa76b1d1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3408b6790 0x55a3408b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.480+0000 7fa76b1d1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a340811eb0 con 0x55a3408b6790 2024-03-21T10:38:56.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.480+0000 7fa76b1d1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3407baa60 con 0x55a3408adbb0 2024-03-21T10:38:56.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.480+0000 7fa76b1d1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34080df50 con 0x55a3408ad290 2024-03-21T10:38:56.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.480+0000 7fa766400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3408ad290 0x55a3408ad670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.480+0000 7fa765a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3408adbb0 0x55a3408b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.480+0000 7fa766400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3408ad290 0x55a3408ad670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38596/0 (socket says 172.21.15.31:38596) 2024-03-21T10:38:56.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.480+0000 7fa766400640 1 -- 172.21.15.31:0/3704195788 learned_addr learned my addr 172.21.15.31:0/3704195788 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:56.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.480+0000 7fa765000640 1 --2- 172.21.15.31:0/3704195788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3408b6790 0x55a3408b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.480+0000 7fa766400640 1 -- 172.21.15.31:0/3704195788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3408adbb0 msgr2=0x55a3408b6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:56.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.480+0000 7fa766400640 1 --2- 172.21.15.31:0/3704195788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3408adbb0 0x55a3408b6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.480+0000 7fa766400640 1 -- 172.21.15.31:0/3704195788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3408b6790 msgr2=0x55a3408b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:56.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.480+0000 7fa766400640 1 --2- 172.21.15.31:0/3704195788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3408b6790 0x55a3408b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.480+0000 7fa766400640 1 -- 172.21.15.31:0/3704195788 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a3405ee7e0 con 0x55a3408ad290 2024-03-21T10:38:56.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.481+0000 7fa766400640 1 --2- 172.21.15.31:0/3704195788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3408ad290 0x55a3408ad670 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa758004af0 tx=0x7fa758005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bf862b4704fa67d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:56.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.481+0000 7fa75e000640 1 --2- 172.21.15.31:0/2917855760 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa7480a31a0 0x7fa7480a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.481+0000 7fa75e000640 1 -- 172.21.15.31:0/2917855760 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fa7540cfff0 con 0x7fa760073a40 2024-03-21T10:38:56.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.481+0000 7fa752a00640 1 -- 172.21.15.31:0/3704195788 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa758043070 con 0x55a3408ad290 2024-03-21T10:38:56.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.481+0000 7fa752a00640 1 -- 172.21.15.31:0/3704195788 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa758003e50 con 0x55a3408ad290 2024-03-21T10:38:56.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.481+0000 7fa752a00640 1 -- 172.21.15.31:0/3704195788 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa758039040 con 0x55a3408ad290 2024-03-21T10:38:56.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.481+0000 7fa765a00640 1 --2- 172.21.15.31:0/2917855760 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa7480a31a0 0x7fa7480a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.482+0000 7fa76b1d1740 1 -- 172.21.15.31:0/3704195788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3408ad290 msgr2=0x55a3408ad670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:56.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.482+0000 7fa76b1d1740 1 --2- 172.21.15.31:0/3704195788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3408ad290 0x55a3408ad670 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa758004af0 tx=0x7fa758005f80 comp rx=0 tx=0).stop 2024-03-21T10:38:56.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.482+0000 7fa765a00640 1 --2- 172.21.15.31:0/2917855760 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa7480a31a0 0x7fa7480a5660 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa7540cd3a0 tx=0x7fa7540d3a70 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:56.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.482+0000 7fa76b1d1740 1 -- 172.21.15.31:0/3704195788 shutdown_connections 2024-03-21T10:38:56.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.482+0000 7fa76b1d1740 1 --2- 172.21.15.31:0/3704195788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3408b6790 0x55a3408b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.482+0000 7fa76b1d1740 1 --2- 172.21.15.31:0/3704195788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3408adbb0 0x55a3408b6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.482+0000 7fa76b1d1740 1 --2- 172.21.15.31:0/3704195788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3408ad290 0x55a3408ad670 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.482+0000 7fa76b1d1740 1 -- 172.21.15.31:0/3704195788 >> 172.21.15.31:0/3704195788 conn(0x55a3408a79c0 msgr2=0x55a3408a7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:56.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.482+0000 7fa76b1d1740 1 -- 172.21.15.31:0/3704195788 shutdown_connections 2024-03-21T10:38:56.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.482+0000 7fa76b1d1740 1 -- 172.21.15.31:0/3704195788 wait complete. 2024-03-21T10:38:56.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.483+0000 7fa76b1d1740 1 -- start start 2024-03-21T10:38:56.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.483+0000 7fa76b1d1740 1 Processor -- start 2024-03-21T10:38:56.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.483+0000 7fa76b1d1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3408ad290 0x55a3408b6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.483+0000 7fa76b1d1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3408adbb0 0x55a3408b34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.483+0000 7fa76b1d1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3408b6790 0x55a3408b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.483+0000 7fa76b1d1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3408b9520 con 0x55a3408b6790 2024-03-21T10:38:56.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.483+0000 7fa76b1d1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3407cdeb0 con 0x55a3408ad290 2024-03-21T10:38:56.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.483+0000 7fa76b1d1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3408b96a0 con 0x55a3408adbb0 2024-03-21T10:38:56.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.484+0000 7fa76b1d1740 10 client.0 _ll_register_callbacks cb 0x55a3407c2150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:38:56.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.484+0000 7fa765000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3408adbb0 0x55a3408b34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.484+0000 7fa766400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3408ad290 0x55a3408b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.484+0000 7fa765000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3408adbb0 0x55a3408b34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38604/0 (socket says 172.21.15.31:38604) 2024-03-21T10:38:56.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.484+0000 7fa765000640 1 -- 172.21.15.31:0/3336258562 learned_addr learned my addr 172.21.15.31:0/3336258562 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:56.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.484+0000 7fa766400640 1 --2- 172.21.15.31:0/3336258562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3408b6790 0x55a3408b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.484+0000 7fa765000640 1 -- 172.21.15.31:0/3336258562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3408ad290 msgr2=0x55a3408b6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:56.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.484+0000 7fa765000640 1 --2- 172.21.15.31:0/3336258562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3408ad290 0x55a3408b6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.484+0000 7fa765000640 1 -- 172.21.15.31:0/3336258562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3408b6790 msgr2=0x55a3408b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:56.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.484+0000 7fa765000640 1 --2- 172.21.15.31:0/3336258562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3408b6790 0x55a3408b3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.484+0000 7fa765000640 1 -- 172.21.15.31:0/3336258562 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a34064ef70 con 0x55a3408adbb0 2024-03-21T10:38:56.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.484+0000 7fa766400640 1 --2- 172.21.15.31:0/3336258562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3408b6790 0x55a3408b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:56.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.484+0000 7fa765000640 1 --2- 172.21.15.31:0/3336258562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3408adbb0 0x55a3408b34f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa74c012700 tx=0x7fa74c012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f05a80c3ca2d42 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:56.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.484+0000 7fa747400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:38:56.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.485+0000 7fa747400640 1 -- 172.21.15.31:0/3336258562 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c004240 con 0x55a3408adbb0 2024-03-21T10:38:56.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.485+0000 7fa747400640 1 -- 172.21.15.31:0/3336258562 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fa74c0043e0 con 0x55a3408adbb0 2024-03-21T10:38:56.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.485+0000 7fa76b1d1740 1 -- 172.21.15.31:0/3336258562 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa758003e50 con 0x55a3408adbb0 2024-03-21T10:38:56.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.485+0000 7fa747400640 1 -- 172.21.15.31:0/3336258562 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa74c0119a0 con 0x55a3408adbb0 2024-03-21T10:38:56.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.485+0000 7fa76b1d1740 10 client.25750 fetch_fsmap learned FSMap version 24 2024-03-21T10:38:56.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.485+0000 7fa76b1d1740 1 -- 172.21.15.31:0/3336258562 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55a3404e6170 con 0x55a3408adbb0 2024-03-21T10:38:56.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.485+0000 7fa747400640 1 -- 172.21.15.31:0/3336258562 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa758003e50 con 0x55a3408adbb0 2024-03-21T10:38:56.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.486+0000 7fa747400640 1 -- 172.21.15.31:0/3336258562 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa74c00f460 con 0x55a3408adbb0 2024-03-21T10:38:56.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.486+0000 7fa76b1d1740 10 client.25750 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:38:56.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.486+0000 7fa76b1d1740 10 client.25750 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:38:56.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.486+0000 7fa76b1d1740 20 client.25750 populate_metadata read hostname 'smithi031' 2024-03-21T10:38:56.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.486+0000 7fa76b1d1740 1 --2- 172.21.15.31:0/3336258562 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a3408c6840 0x55a3408c8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.486+0000 7fa76b1d1740 4 client.25750 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:38:56.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.486+0000 7fa76b1d1740 1 -- 172.21.15.31:0/3336258562 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a34081f3f0 con 0x55a3408c6840 2024-03-21T10:38:56.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.486+0000 7fa766400640 1 --2- 172.21.15.31:0/3336258562 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a3408c6840 0x55a3408c8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.486+0000 7fa766400640 1 --2- 172.21.15.31:0/3336258562 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a3408c6840 0x55a3408c8c60 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69fc02f3185ce833 server_cookie=e9abcabaf46a0ce1 in_seq=0 out_seq=0 2024-03-21T10:38:56.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.486+0000 7fa747400640 10 client.25750 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:56.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.487+0000 7fa747400640 1 -- 172.21.15.31:0/3336258562 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a34081f3f0 con 0x55a3408c6840 2024-03-21T10:38:56.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.487+0000 7fa747400640 10 client.25750 handle_command_reply: tid=0 2024-03-21T10:38:56.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.487+0000 7fa76b1d1740 1 client.25750 shutdown 2024-03-21T10:38:56.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.487+0000 7fa76b1d1740 1 -- 172.21.15.31:0/3336258562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3408adbb0 msgr2=0x55a3408b34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:56.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.487+0000 7fa76b1d1740 1 --2- 172.21.15.31:0/3336258562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3408adbb0 0x55a3408b34f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa74c012700 tx=0x7fa74c012bc0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.487+0000 7fa76b1d1740 1 -- 172.21.15.31:0/3336258562 shutdown_connections 2024-03-21T10:38:56.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.487+0000 7fa76b1d1740 1 --2- 172.21.15.31:0/3336258562 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a3408c6840 0x55a3408c8c60 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.488+0000 7fa76b1d1740 1 --2- 172.21.15.31:0/3336258562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3408b6790 0x55a3408b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.488+0000 7fa76b1d1740 1 --2- 172.21.15.31:0/3336258562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3408adbb0 0x55a3408b34f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.488+0000 7fa747400640 0 client.25750 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:56.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.488+0000 7fa76b1d1740 1 --2- 172.21.15.31:0/3336258562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3408ad290 0x55a3408b6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.488+0000 7fa76b1d1740 1 -- 172.21.15.31:0/3336258562 >> 172.21.15.31:0/3336258562 conn(0x55a3408a79c0 msgr2=0x55a3408a9710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:56.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.488+0000 7fa76b1d1740 1 -- 172.21.15.31:0/3336258562 shutdown_connections 2024-03-21T10:38:56.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.488+0000 7fa76b1d1740 1 -- 172.21.15.31:0/3336258562 wait complete. 2024-03-21T10:38:56.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.488+0000 7fa76b1d1740 20 client.25750 trim_cache size 0 max 16384 2024-03-21T10:38:56.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.513+0000 7fa76b1d1740 1 Processor -- start 2024-03-21T10:38:56.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.513+0000 7fa76b1d1740 1 -- start start 2024-03-21T10:38:56.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.514+0000 7fa76b1d1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3408ad290 0x55a3408b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.514+0000 7fa76b1d1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3408adbb0 0x55a3408b6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.514+0000 7fa76b1d1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3408b6790 0x55a3408ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.514+0000 7fa76b1d1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7601b71f0 con 0x55a3408adbb0 2024-03-21T10:38:56.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.514+0000 7fa76b1d1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7601b4cf0 con 0x55a3408b6790 2024-03-21T10:38:56.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.514+0000 7fa76b1d1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7601b2d90 con 0x55a3408ad290 2024-03-21T10:38:56.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.514+0000 7fa766400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3408ad290 0x55a3408b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.514+0000 7fa765000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3408adbb0 0x55a3408b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.514+0000 7fa766400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3408ad290 0x55a3408b5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38608/0 (socket says 172.21.15.31:38608) 2024-03-21T10:38:56.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.514+0000 7fa766400640 1 -- 172.21.15.31:0/2860165627 learned_addr learned my addr 172.21.15.31:0/2860165627 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:56.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.514+0000 7fa766400640 1 --2- 172.21.15.31:0/2860165627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3408b6790 0x55a3408ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.515+0000 7fa766400640 1 -- 172.21.15.31:0/2860165627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3408b6790 msgr2=0x55a3408ae7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:56.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.515+0000 7fa766400640 1 --2- 172.21.15.31:0/2860165627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3408b6790 0x55a3408ae7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.515+0000 7fa766400640 1 -- 172.21.15.31:0/2860165627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3408adbb0 msgr2=0x55a3408b6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:56.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.515+0000 7fa766400640 1 --2- 172.21.15.31:0/2860165627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3408adbb0 0x55a3408b6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.515+0000 7fa766400640 1 -- 172.21.15.31:0/2860165627 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a3408b3e70 con 0x55a3408ad290 2024-03-21T10:38:56.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.515+0000 7fa765000640 1 --2- 172.21.15.31:0/2860165627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3408adbb0 0x55a3408b6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:56.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.515+0000 7fa766400640 1 --2- 172.21.15.31:0/2860165627 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3408ad290 0x55a3408b5c20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa758004820 tx=0x7fa758041040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0482424ff5cb76b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:56.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.515+0000 7fa752000640 1 -- 172.21.15.31:0/2860165627 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa758039040 con 0x55a3408ad290 2024-03-21T10:38:56.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.515+0000 7fa752000640 1 -- 172.21.15.31:0/2860165627 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7580552d0 con 0x55a3408ad290 2024-03-21T10:38:56.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.515+0000 7fa752000640 1 -- 172.21.15.31:0/2860165627 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa758043040 con 0x55a3408ad290 2024-03-21T10:38:56.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.516+0000 7fa76b1d1740 1 -- 172.21.15.31:0/2860165627 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3408ad290 msgr2=0x55a3408b5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:56.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.516+0000 7fa76b1d1740 1 --2- 172.21.15.31:0/2860165627 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3408ad290 0x55a3408b5c20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa758004820 tx=0x7fa758041040 comp rx=0 tx=0).stop 2024-03-21T10:38:56.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.516+0000 7fa76b1d1740 1 -- 172.21.15.31:0/2860165627 shutdown_connections 2024-03-21T10:38:56.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.516+0000 7fa76b1d1740 1 --2- 172.21.15.31:0/2860165627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3408b6790 0x55a3408ae7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.516+0000 7fa76b1d1740 1 --2- 172.21.15.31:0/2860165627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3408adbb0 0x55a3408b6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.516+0000 7fa76b1d1740 1 --2- 172.21.15.31:0/2860165627 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3408ad290 0x55a3408b5c20 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.516+0000 7fa76b1d1740 1 -- 172.21.15.31:0/2860165627 >> 172.21.15.31:0/2860165627 conn(0x55a3408a79c0 msgr2=0x55a3408ae0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:56.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.516+0000 7fa76b1d1740 1 -- 172.21.15.31:0/2860165627 shutdown_connections 2024-03-21T10:38:56.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.516+0000 7fa76b1d1740 1 -- 172.21.15.31:0/2860165627 wait complete. 2024-03-21T10:38:56.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.516+0000 7fa76b1d1740 1 -- start start 2024-03-21T10:38:56.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.516+0000 7fa76b1d1740 1 Processor -- start 2024-03-21T10:38:56.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.517+0000 7fa76b1d1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3408ad290 0x55a3408bb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.517+0000 7fa76b1d1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3408adbb0 0x55a3408bb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.517+0000 7fa76b1d1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3408b6790 0x55a340922140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.517+0000 7fa76b1d1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7601b3390 con 0x55a3408adbb0 2024-03-21T10:38:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.517+0000 7fa76b1d1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7601b4690 con 0x55a3408ad290 2024-03-21T10:38:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.517+0000 7fa76b1d1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7601b39d0 con 0x55a3408b6790 2024-03-21T10:38:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.517+0000 7fa76b1d1740 10 client.0 _ll_register_callbacks cb 0x7fa74c009720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:38:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.517+0000 7fa766400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3408ad290 0x55a3408bb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.517+0000 7fa765000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3408adbb0 0x55a3408bb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.517+0000 7fa766400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3408ad290 0x55a3408bb0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42618/0 (socket says 172.21.15.31:42618) 2024-03-21T10:38:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.517+0000 7fa766400640 1 -- 172.21.15.31:0/1785110755 learned_addr learned my addr 172.21.15.31:0/1785110755 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.517+0000 7fa766400640 1 --2- 172.21.15.31:0/1785110755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3408b6790 0x55a340922140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.518+0000 7fa765000640 1 -- 172.21.15.31:0/1785110755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3408b6790 msgr2=0x55a340922140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.518+0000 7fa765000640 1 --2- 172.21.15.31:0/1785110755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3408b6790 0x55a340922140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.518+0000 7fa765000640 1 -- 172.21.15.31:0/1785110755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3408ad290 msgr2=0x55a3408bb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.518+0000 7fa765000640 1 --2- 172.21.15.31:0/1785110755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3408ad290 0x55a3408bb0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.518+0000 7fa765000640 1 -- 172.21.15.31:0/1785110755 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a340922680 con 0x55a3408adbb0 2024-03-21T10:38:56.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.518+0000 7fa766400640 1 --2- 172.21.15.31:0/1785110755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3408ad290 0x55a3408bb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:56.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.518+0000 7fa765000640 1 --2- 172.21.15.31:0/1785110755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3408adbb0 0x55a3408bb5f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa74c00f880 tx=0x7fa74c009d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de9ba42f7e05390f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:56.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.518+0000 7fa747400640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:38:56.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.518+0000 7fa747400640 1 -- 172.21.15.31:0/1785110755 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c011b40 con 0x55a3408adbb0 2024-03-21T10:38:56.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.519+0000 7fa747400640 1 -- 172.21.15.31:0/1785110755 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fa74c011ce0 con 0x55a3408adbb0 2024-03-21T10:38:56.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.519+0000 7fa76b1d1740 1 -- 172.21.15.31:0/1785110755 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa7580552d0 con 0x55a3408adbb0 2024-03-21T10:38:56.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.519+0000 7fa747400640 1 -- 172.21.15.31:0/1785110755 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa74c03b050 con 0x55a3408adbb0 2024-03-21T10:38:56.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.519+0000 7fa76b1d1740 10 client.15609 fetch_fsmap learned FSMap version 24 2024-03-21T10:38:56.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.519+0000 7fa76b1d1740 1 -- 172.21.15.31:0/1785110755 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55a340923840 con 0x55a3408adbb0 2024-03-21T10:38:56.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.519+0000 7fa747400640 1 -- 172.21.15.31:0/1785110755 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7580552d0 con 0x55a3408adbb0 2024-03-21T10:38:56.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.519+0000 7fa747400640 1 -- 172.21.15.31:0/1785110755 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa74c003c10 con 0x55a3408adbb0 2024-03-21T10:38:56.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.519+0000 7fa76b1d1740 10 client.15609 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:38:56.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.519+0000 7fa76b1d1740 10 client.15609 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:38:56.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.519+0000 7fa76b1d1740 20 client.15609 populate_metadata read hostname 'smithi031' 2024-03-21T10:38:56.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.519+0000 7fa76b1d1740 1 --2- 172.21.15.31:0/1785110755 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a340923f40 0x55a340926360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.520+0000 7fa76b1d1740 4 client.15609 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-21T10:38:56.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.520+0000 7fa76b1d1740 1 -- 172.21.15.31:0/1785110755 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55a340819740 con 0x55a340923f40 2024-03-21T10:38:56.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.520+0000 7fa766400640 1 --2- 172.21.15.31:0/1785110755 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a340923f40 0x55a340926360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.520+0000 7fa766400640 1 --2- 172.21.15.31:0/1785110755 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a340923f40 0x55a340926360 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=400f53ea5c62553 server_cookie=f03da644fdedf9b in_seq=0 out_seq=0 2024-03-21T10:38:56.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.520+0000 7fa747400640 10 client.15609 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:56.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.521+0000 7fa747400640 1 -- 172.21.15.31:0/1785110755 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55a340819740 con 0x55a340923f40 2024-03-21T10:38:56.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.521+0000 7fa747400640 10 client.15609 handle_command_reply: tid=0 2024-03-21T10:38:56.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.521+0000 7fa76b1d1740 1 client.15609 shutdown 2024-03-21T10:38:56.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.521+0000 7fa76b1d1740 1 -- 172.21.15.31:0/1785110755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3408adbb0 msgr2=0x55a3408bb5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:56.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.521+0000 7fa76b1d1740 1 --2- 172.21.15.31:0/1785110755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3408adbb0 0x55a3408bb5f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa74c00f880 tx=0x7fa74c009d20 comp rx=0 tx=0).stop 2024-03-21T10:38:56.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.522+0000 7fa76b1d1740 1 -- 172.21.15.31:0/1785110755 shutdown_connections 2024-03-21T10:38:56.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.522+0000 7fa76b1d1740 1 --2- 172.21.15.31:0/1785110755 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a340923f40 0x55a340926360 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.522+0000 7fa76b1d1740 1 --2- 172.21.15.31:0/1785110755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3408b6790 0x55a340922140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.522+0000 7fa76b1d1740 1 --2- 172.21.15.31:0/1785110755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3408adbb0 0x55a3408bb5f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.522+0000 7fa76b1d1740 1 --2- 172.21.15.31:0/1785110755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3408ad290 0x55a3408bb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.522+0000 7fa76b1d1740 1 -- 172.21.15.31:0/1785110755 >> 172.21.15.31:0/1785110755 conn(0x55a3408a79c0 msgr2=0x55a3408ae960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:56.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.522+0000 7fa747400640 0 client.15609 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:56.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.522+0000 7fa76b1d1740 1 -- 172.21.15.31:0/1785110755 shutdown_connections 2024-03-21T10:38:56.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.522+0000 7fa76b1d1740 1 -- 172.21.15.31:0/1785110755 wait complete. 2024-03-21T10:38:56.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.522+0000 7fa76b1d1740 20 client.15609 trim_cache size 0 max 16384 2024-03-21T10:38:56.523 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:38:56.523 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-03-21T10:38:56.523 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "d747a398-552a-4905-bce0-c1ceee57211f", 2024-03-21T10:38:56.523 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-03-21T10:38:56.523 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:38:56.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.523+0000 7fa766e00640 1 -- 172.21.15.31:0/2917855760 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa7480a31a0 msgr2=0x7fa7480a5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:56.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.523+0000 7fa766e00640 1 --2- 172.21.15.31:0/2917855760 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa7480a31a0 0x7fa7480a5660 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa7540cd3a0 tx=0x7fa7540d3a70 comp rx=0 tx=0).stop 2024-03-21T10:38:56.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.523+0000 7fa766e00640 1 -- 172.21.15.31:0/2917855760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa760073a40 msgr2=0x7fa76019e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:56.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.523+0000 7fa766e00640 1 --2- 172.21.15.31:0/2917855760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa760073a40 0x7fa76019e390 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa75400e9f0 tx=0x7fa75400eeb0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.523+0000 7fa766e00640 1 -- 172.21.15.31:0/2917855760 shutdown_connections 2024-03-21T10:38:56.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.523+0000 7fa766e00640 1 --2- 172.21.15.31:0/2917855760 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa7480a31a0 0x7fa7480a5660 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.523+0000 7fa766e00640 1 --2- 172.21.15.31:0/2917855760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa760112470 0x7fa7601a2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.523+0000 7fa766e00640 1 --2- 172.21.15.31:0/2917855760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa760074360 0x7fa76019e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.523+0000 7fa766e00640 1 --2- 172.21.15.31:0/2917855760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa760073a40 0x7fa76019e390 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.523+0000 7fa766e00640 1 -- 172.21.15.31:0/2917855760 >> 172.21.15.31:0/2917855760 conn(0x7fa7600fcff0 msgr2=0x7fa7600fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:56.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.523+0000 7fa766e00640 1 -- 172.21.15.31:0/2917855760 shutdown_connections 2024-03-21T10:38:56.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.523+0000 7fa766e00640 1 -- 172.21.15.31:0/2917855760 wait complete. 2024-03-21T10:38:56.545 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:38:56.545 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:38:56.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.775+0000 7f4395400640 1 -- 172.21.15.31:0/1873565983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4390100420 msgr2=0x7f4390100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:56.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.775+0000 7f4395400640 1 --2- 172.21.15.31:0/1873565983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4390100420 0x7f4390100800 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4380009ca0 tx=0x7f438002c190 comp rx=0 tx=0).stop 2024-03-21T10:38:56.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.775+0000 7f4395400640 1 -- 172.21.15.31:0/1873565983 shutdown_connections 2024-03-21T10:38:56.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.775+0000 7f4395400640 1 --2- 172.21.15.31:0/1873565983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4390105820 0x7f4390105c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.775+0000 7f4395400640 1 --2- 172.21.15.31:0/1873565983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4390100d40 0x7f4390105200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.775+0000 7f4395400640 1 --2- 172.21.15.31:0/1873565983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4390100420 0x7f4390100800 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.775+0000 7f4395400640 1 -- 172.21.15.31:0/1873565983 >> 172.21.15.31:0/1873565983 conn(0x7f4390077130 msgr2=0x7f43900fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:56.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.775+0000 7f4395400640 1 -- 172.21.15.31:0/1873565983 shutdown_connections 2024-03-21T10:38:56.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.775+0000 7f4395400640 1 -- 172.21.15.31:0/1873565983 wait complete. 2024-03-21T10:38:56.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.776+0000 7f4395400640 1 Processor -- start 2024-03-21T10:38:56.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.777+0000 7f4395400640 1 -- start start 2024-03-21T10:38:56.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.777+0000 7f4395400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4390100420 0x7f439019e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.777+0000 7f4395400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4390100d40 0x7f439019e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.777+0000 7f4395400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4390105820 0x7f43901a2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.777+0000 7f4395400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4390117e30 con 0x7f4390100420 2024-03-21T10:38:56.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.777+0000 7f4395400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4390117cb0 con 0x7f4390100d40 2024-03-21T10:38:56.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.777+0000 7f4395400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4390117fb0 con 0x7f4390105820 2024-03-21T10:38:56.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.777+0000 7f438f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4390100d40 0x7f439019e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.777+0000 7f438f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4390100d40 0x7f439019e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42642/0 (socket says 172.21.15.31:42642) 2024-03-21T10:38:56.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.778+0000 7f438f400640 1 -- 172.21.15.31:0/3598585125 learned_addr learned my addr 172.21.15.31:0/3598585125 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:56.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.778+0000 7f4394a00640 1 --2- 172.21.15.31:0/3598585125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4390105820 0x7f43901a2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.778+0000 7f438fe00640 1 --2- 172.21.15.31:0/3598585125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4390100420 0x7f439019e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.778+0000 7f438f400640 1 -- 172.21.15.31:0/3598585125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4390105820 msgr2=0x7f43901a2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:56.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.778+0000 7f438f400640 1 --2- 172.21.15.31:0/3598585125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4390105820 0x7f43901a2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.778+0000 7f438f400640 1 -- 172.21.15.31:0/3598585125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4390100420 msgr2=0x7f439019e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:56.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.778+0000 7f438f400640 1 --2- 172.21.15.31:0/3598585125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4390100420 0x7f439019e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.778+0000 7f438f400640 1 -- 172.21.15.31:0/3598585125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43901a3320 con 0x7f4390100d40 2024-03-21T10:38:56.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.778+0000 7f4394a00640 1 --2- 172.21.15.31:0/3598585125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4390105820 0x7f43901a2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:56.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.778+0000 7f438f400640 1 --2- 172.21.15.31:0/3598585125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4390100d40 0x7f439019e870 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f437c00e9f0 tx=0x7f437c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:56.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.779+0000 7f438cc00640 1 -- 172.21.15.31:0/3598585125 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437c00cd80 con 0x7f4390100d40 2024-03-21T10:38:56.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.779+0000 7f438cc00640 1 -- 172.21.15.31:0/3598585125 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f437c005500 con 0x7f4390100d40 2024-03-21T10:38:56.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.779+0000 7f438cc00640 1 -- 172.21.15.31:0/3598585125 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437c010690 con 0x7f4390100d40 2024-03-21T10:38:56.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.779+0000 7f4395400640 1 -- 172.21.15.31:0/3598585125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43901a3610 con 0x7f4390100d40 2024-03-21T10:38:56.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.779+0000 7f4395400640 1 -- 172.21.15.31:0/3598585125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43901aaea0 con 0x7f4390100d40 2024-03-21T10:38:56.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.782+0000 7f438cc00640 1 -- 172.21.15.31:0/3598585125 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) v1 ==== 138030+0+0 (secure 0 0 0) 0x7f437c0046c0 con 0x7f4390100d40 2024-03-21T10:38:56.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.783+0000 7f438cc00640 1 --2- 172.21.15.31:0/3598585125 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f43740a31a0 0x7f43740a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.784+0000 7f438cc00640 1 -- 172.21.15.31:0/3598585125 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f437c0d00a0 con 0x7f4390100d40 2024-03-21T10:38:56.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.784+0000 7f43997cf740 1 Processor -- start 2024-03-21T10:38:56.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.784+0000 7f438fe00640 1 --2- 172.21.15.31:0/3598585125 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f43740a31a0 0x7f43740a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.784+0000 7f43997cf740 1 -- start start 2024-03-21T10:38:56.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.784+0000 7f43997cf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56185c576dd0 0x56185c575b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.784+0000 7f43997cf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56185c5760f0 0x56185c576570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.784+0000 7f43997cf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56185c57f460 0x56185c581850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.784+0000 7f43997cf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56185c4da250 con 0x56185c57f460 2024-03-21T10:38:56.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.784+0000 7f43997cf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56185c482a30 con 0x56185c5760f0 2024-03-21T10:38:56.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.784+0000 7f43997cf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56185c4d62f0 con 0x56185c576dd0 2024-03-21T10:38:56.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.785+0000 7f438fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56185c576dd0 0x56185c575b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.785+0000 7f438f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56185c5760f0 0x56185c576570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.785+0000 7f438f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56185c5760f0 0x56185c576570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42656/0 (socket says 172.21.15.31:42656) 2024-03-21T10:38:56.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.785+0000 7f438f400640 1 -- 172.21.15.31:0/2258288949 learned_addr learned my addr 172.21.15.31:0/2258288949 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:56.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.785+0000 7f4394a00640 1 --2- 172.21.15.31:0/2258288949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56185c57f460 0x56185c581850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.785+0000 7f438fe00640 1 --2- 172.21.15.31:0/3598585125 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f43740a31a0 0x7f43740a5660 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4380000f80 tx=0x7f4380009360 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:56.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.785+0000 7f438f400640 1 -- 172.21.15.31:0/2258288949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56185c576dd0 msgr2=0x56185c575b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:56.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.785+0000 7f438f400640 1 --2- 172.21.15.31:0/2258288949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56185c576dd0 0x56185c575b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.785+0000 7f438f400640 1 -- 172.21.15.31:0/2258288949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56185c57f460 msgr2=0x56185c581850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:56.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.785+0000 7f438f400640 1 --2- 172.21.15.31:0/2258288949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56185c57f460 0x56185c581850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.785+0000 7f438f400640 1 -- 172.21.15.31:0/2258288949 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56185c46d690 con 0x56185c5760f0 2024-03-21T10:38:56.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.785+0000 7f438f400640 1 --2- 172.21.15.31:0/2258288949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56185c5760f0 0x56185c576570 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f437c09bc80 tx=0x7f437c09bbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51c2d595559e982e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:56.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.786+0000 7f4385200640 1 -- 172.21.15.31:0/2258288949 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437c09bf80 con 0x56185c5760f0 2024-03-21T10:38:56.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.786+0000 7f4385200640 1 -- 172.21.15.31:0/2258288949 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f437c0b5070 con 0x56185c5760f0 2024-03-21T10:38:56.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.786+0000 7f4385200640 1 -- 172.21.15.31:0/2258288949 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437c0a4500 con 0x56185c5760f0 2024-03-21T10:38:56.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.786+0000 7f43997cf740 1 -- 172.21.15.31:0/2258288949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56185c5760f0 msgr2=0x56185c576570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:56.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.786+0000 7f43997cf740 1 --2- 172.21.15.31:0/2258288949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56185c5760f0 0x56185c576570 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f437c09bc80 tx=0x7f437c09bbd0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.786+0000 7f43997cf740 1 -- 172.21.15.31:0/2258288949 shutdown_connections 2024-03-21T10:38:56.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.786+0000 7f43997cf740 1 --2- 172.21.15.31:0/2258288949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56185c57f460 0x56185c581850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.786+0000 7f43997cf740 1 --2- 172.21.15.31:0/2258288949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56185c5760f0 0x56185c576570 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.786+0000 7f43997cf740 1 --2- 172.21.15.31:0/2258288949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56185c576dd0 0x56185c575b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.786+0000 7f43997cf740 1 -- 172.21.15.31:0/2258288949 >> 172.21.15.31:0/2258288949 conn(0x56185c53d200 msgr2=0x56185c570b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:56.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.786+0000 7f43997cf740 1 -- 172.21.15.31:0/2258288949 shutdown_connections 2024-03-21T10:38:56.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.786+0000 7f43997cf740 1 -- 172.21.15.31:0/2258288949 wait complete. 2024-03-21T10:38:56.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.787+0000 7f43997cf740 1 -- start start 2024-03-21T10:38:56.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.787+0000 7f43997cf740 1 Processor -- start 2024-03-21T10:38:56.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.787+0000 7f43997cf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56185c5760f0 0x56185c57ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.787+0000 7f43997cf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56185c576dd0 0x56185c57bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.788+0000 7f43997cf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56185c57f460 0x56185c57c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.788+0000 7f43997cf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56185c5822d0 con 0x56185c57f460 2024-03-21T10:38:56.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.788+0000 7f43997cf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56185c495e80 con 0x56185c576dd0 2024-03-21T10:38:56.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.788+0000 7f43997cf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56185c582450 con 0x56185c5760f0 2024-03-21T10:38:56.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.788+0000 7f43997cf740 10 client.0 _ll_register_callbacks cb 0x56185c4a8130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:38:56.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.788+0000 7f438fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56185c5760f0 0x56185c57ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.788+0000 7f438fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56185c5760f0 0x56185c57ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38656/0 (socket says 172.21.15.31:38656) 2024-03-21T10:38:56.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.788+0000 7f438fe00640 1 -- 172.21.15.31:0/3022405671 learned_addr learned my addr 172.21.15.31:0/3022405671 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:56.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.788+0000 7f438f400640 1 --2- 172.21.15.31:0/3022405671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56185c576dd0 0x56185c57bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.788+0000 7f4394a00640 1 --2- 172.21.15.31:0/3022405671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56185c57f460 0x56185c57c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.788+0000 7f438fe00640 1 -- 172.21.15.31:0/3022405671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56185c576dd0 msgr2=0x56185c57bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:56.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.788+0000 7f438fe00640 1 --2- 172.21.15.31:0/3022405671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56185c576dd0 0x56185c57bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.788+0000 7f438fe00640 1 -- 172.21.15.31:0/3022405671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56185c57f460 msgr2=0x56185c57c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:56.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.788+0000 7f438fe00640 1 --2- 172.21.15.31:0/3022405671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56185c57f460 0x56185c57c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.788+0000 7f438fe00640 1 -- 172.21.15.31:0/3022405671 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56185c2b87d0 con 0x56185c5760f0 2024-03-21T10:38:56.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.788+0000 7f4394a00640 1 --2- 172.21.15.31:0/3022405671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56185c57f460 0x56185c57c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:56.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.788+0000 7f438fe00640 1 --2- 172.21.15.31:0/3022405671 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56185c5760f0 0x56185c57ebb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4380009ca0 tx=0x7f438002c190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a45b7f5054b933 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:56.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.788+0000 7f4379600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:38:56.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.789+0000 7f4379600640 1 -- 172.21.15.31:0/3022405671 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4380050070 con 0x56185c5760f0 2024-03-21T10:38:56.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.789+0000 7f43997cf740 1 -- 172.21.15.31:0/3022405671 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f437c0b5070 con 0x56185c5760f0 2024-03-21T10:38:56.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.789+0000 7f4379600640 1 -- 172.21.15.31:0/3022405671 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f4380032870 con 0x56185c5760f0 2024-03-21T10:38:56.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.789+0000 7f4379600640 1 -- 172.21.15.31:0/3022405671 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4380039040 con 0x56185c5760f0 2024-03-21T10:38:56.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.789+0000 7f43997cf740 10 client.25768 fetch_fsmap learned FSMap version 24 2024-03-21T10:38:56.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.789+0000 7f43997cf740 1 -- 172.21.15.31:0/3022405671 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x56185c1ee3f0 con 0x56185c5760f0 2024-03-21T10:38:56.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.789+0000 7f4379600640 1 -- 172.21.15.31:0/3022405671 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f437c0b5070 con 0x56185c5760f0 2024-03-21T10:38:56.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.789+0000 7f4379600640 1 -- 172.21.15.31:0/3022405671 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f4380034ae0 con 0x56185c5760f0 2024-03-21T10:38:56.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.789+0000 7f43997cf740 10 client.25768 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:38:56.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.789+0000 7f43997cf740 10 client.25768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:38:56.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.789+0000 7f43997cf740 20 client.25768 populate_metadata read hostname 'smithi031' 2024-03-21T10:38:56.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.790+0000 7f43997cf740 1 --2- 172.21.15.31:0/3022405671 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56185c58f390 0x56185c591780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.790+0000 7f43997cf740 4 client.25768 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:38:56.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.790+0000 7f43997cf740 1 -- 172.21.15.31:0/3022405671 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56185c4e7790 con 0x56185c58f390 2024-03-21T10:38:56.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.790+0000 7f438f400640 1 --2- 172.21.15.31:0/3022405671 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56185c58f390 0x56185c591780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.790+0000 7f438f400640 1 --2- 172.21.15.31:0/3022405671 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56185c58f390 0x56185c591780 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f8de0f359215b8f server_cookie=9a8b6b2d04276c13 in_seq=0 out_seq=0 2024-03-21T10:38:56.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.790+0000 7f4379600640 10 client.25768 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:56.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.791+0000 7f4379600640 1 -- 172.21.15.31:0/3022405671 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56185c4e7790 con 0x56185c58f390 2024-03-21T10:38:56.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.791+0000 7f4379600640 10 client.25768 handle_command_reply: tid=0 2024-03-21T10:38:56.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.791+0000 7f43997cf740 1 client.25768 shutdown 2024-03-21T10:38:56.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.791+0000 7f43997cf740 1 -- 172.21.15.31:0/3022405671 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56185c5760f0 msgr2=0x56185c57ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:56.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.791+0000 7f43997cf740 1 --2- 172.21.15.31:0/3022405671 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56185c5760f0 0x56185c57ebb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4380009ca0 tx=0x7f438002c190 comp rx=0 tx=0).stop 2024-03-21T10:38:56.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.791+0000 7f43997cf740 1 -- 172.21.15.31:0/3022405671 shutdown_connections 2024-03-21T10:38:56.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.791+0000 7f43997cf740 1 --2- 172.21.15.31:0/3022405671 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56185c58f390 0x56185c591780 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.791+0000 7f43997cf740 1 --2- 172.21.15.31:0/3022405671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56185c57f460 0x56185c57c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.791+0000 7f43997cf740 1 --2- 172.21.15.31:0/3022405671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56185c576dd0 0x56185c57bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.791+0000 7f43997cf740 1 --2- 172.21.15.31:0/3022405671 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56185c5760f0 0x56185c57ebb0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.791+0000 7f4379600640 0 client.25768 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:56.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.791+0000 7f43997cf740 1 -- 172.21.15.31:0/3022405671 >> 172.21.15.31:0/3022405671 conn(0x56185c53d200 msgr2=0x56185c572890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:56.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.791+0000 7f43997cf740 1 -- 172.21.15.31:0/3022405671 shutdown_connections 2024-03-21T10:38:56.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.791+0000 7f43997cf740 1 -- 172.21.15.31:0/3022405671 wait complete. 2024-03-21T10:38:56.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.791+0000 7f43997cf740 20 client.25768 trim_cache size 0 max 16384 2024-03-21T10:38:56.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.809+0000 7f43997cf740 1 Processor -- start 2024-03-21T10:38:56.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.809+0000 7f43997cf740 1 -- start start 2024-03-21T10:38:56.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.809+0000 7f43997cf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56185c5760f0 0x56185c57c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.809+0000 7f43997cf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56185c576dd0 0x56185c57e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.809+0000 7f43997cf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56185c57f460 0x56185c57ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.809+0000 7f43997cf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43901b71e0 con 0x56185c576dd0 2024-03-21T10:38:56.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.809+0000 7f43997cf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43901b4ce0 con 0x56185c5760f0 2024-03-21T10:38:56.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.809+0000 7f43997cf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43901b2d80 con 0x56185c57f460 2024-03-21T10:38:56.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.810+0000 7f438fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56185c5760f0 0x56185c57c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.810+0000 7f438fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56185c5760f0 0x56185c57c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42680/0 (socket says 172.21.15.31:42680) 2024-03-21T10:38:56.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.810+0000 7f438fe00640 1 -- 172.21.15.31:0/2853662077 learned_addr learned my addr 172.21.15.31:0/2853662077 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:56.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.810+0000 7f4394a00640 1 --2- 172.21.15.31:0/2853662077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56185c57f460 0x56185c57ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.810+0000 7f438f400640 1 --2- 172.21.15.31:0/2853662077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56185c576dd0 0x56185c57e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.810+0000 7f438fe00640 1 -- 172.21.15.31:0/2853662077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56185c57f460 msgr2=0x56185c57ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:56.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.810+0000 7f438fe00640 1 --2- 172.21.15.31:0/2853662077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56185c57f460 0x56185c57ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.810+0000 7f438fe00640 1 -- 172.21.15.31:0/2853662077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56185c576dd0 msgr2=0x56185c57e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:56.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.810+0000 7f438fe00640 1 --2- 172.21.15.31:0/2853662077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56185c576dd0 0x56185c57e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.810+0000 7f438fe00640 1 -- 172.21.15.31:0/2853662077 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56185c1ae0f0 con 0x56185c5760f0 2024-03-21T10:38:56.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.810+0000 7f4394a00640 1 --2- 172.21.15.31:0/2853662077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56185c57f460 0x56185c57ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:38:56.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.810+0000 7f438fe00640 1 --2- 172.21.15.31:0/2853662077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56185c5760f0 0x56185c57c610 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f438002f020 tx=0x7f4380005cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fb6b63aca13e13 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:56.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.810+0000 7f437be00640 1 -- 172.21.15.31:0/2853662077 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4380050070 con 0x56185c5760f0 2024-03-21T10:38:56.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.810+0000 7f437be00640 1 -- 172.21.15.31:0/2853662077 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4380044030 con 0x56185c5760f0 2024-03-21T10:38:56.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.810+0000 7f437be00640 1 -- 172.21.15.31:0/2853662077 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4380032c30 con 0x56185c5760f0 2024-03-21T10:38:56.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.811+0000 7f43997cf740 1 -- 172.21.15.31:0/2853662077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56185c5760f0 msgr2=0x56185c57c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:56.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.811+0000 7f43997cf740 1 --2- 172.21.15.31:0/2853662077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56185c5760f0 0x56185c57c610 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f438002f020 tx=0x7f4380005cf0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.811+0000 7f43997cf740 1 -- 172.21.15.31:0/2853662077 shutdown_connections 2024-03-21T10:38:56.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.811+0000 7f43997cf740 1 --2- 172.21.15.31:0/2853662077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56185c57f460 0x56185c57ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.811+0000 7f43997cf740 1 --2- 172.21.15.31:0/2853662077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56185c576dd0 0x56185c57e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.811+0000 7f43997cf740 1 --2- 172.21.15.31:0/2853662077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56185c5760f0 0x56185c57c610 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.811+0000 7f43997cf740 1 -- 172.21.15.31:0/2853662077 >> 172.21.15.31:0/2853662077 conn(0x56185c53d200 msgr2=0x56185c57bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:56.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.811+0000 7f43997cf740 1 -- 172.21.15.31:0/2853662077 shutdown_connections 2024-03-21T10:38:56.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.811+0000 7f43997cf740 1 -- 172.21.15.31:0/2853662077 wait complete. 2024-03-21T10:38:56.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.812+0000 7f43997cf740 1 -- start start 2024-03-21T10:38:56.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.812+0000 7f43997cf740 1 Processor -- start 2024-03-21T10:38:56.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.812+0000 7f43997cf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56185c5760f0 0x56185c580e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.812+0000 7f43997cf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56185c576dd0 0x56185c5791c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.812+0000 7f43997cf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56185c57f460 0x56185c579700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.812+0000 7f43997cf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43901b3380 con 0x56185c57f460 2024-03-21T10:38:56.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.812+0000 7f43997cf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43901b4680 con 0x56185c576dd0 2024-03-21T10:38:56.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.812+0000 7f43997cf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43901b39c0 con 0x56185c5760f0 2024-03-21T10:38:56.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.812+0000 7f43997cf740 10 client.0 _ll_register_callbacks cb 0x7f4380009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:38:56.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.812+0000 7f438f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56185c576dd0 0x56185c5791c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.813+0000 7f438f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56185c576dd0 0x56185c5791c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42686/0 (socket says 172.21.15.31:42686) 2024-03-21T10:38:56.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.813+0000 7f438f400640 1 -- 172.21.15.31:0/80847535 learned_addr learned my addr 172.21.15.31:0/80847535 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:38:56.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.813+0000 7f438fe00640 1 --2- 172.21.15.31:0/80847535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56185c5760f0 0x56185c580e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.813+0000 7f4394a00640 1 --2- 172.21.15.31:0/80847535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56185c57f460 0x56185c579700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.813+0000 7f438f400640 1 -- 172.21.15.31:0/80847535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56185c5760f0 msgr2=0x56185c580e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:56.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.813+0000 7f438f400640 1 --2- 172.21.15.31:0/80847535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56185c5760f0 0x56185c580e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.813+0000 7f438f400640 1 -- 172.21.15.31:0/80847535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56185c57f460 msgr2=0x56185c579700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:38:56.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.813+0000 7f438f400640 1 --2- 172.21.15.31:0/80847535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56185c57f460 0x56185c579700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.813+0000 7f438f400640 1 -- 172.21.15.31:0/80847535 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56185c579f30 con 0x56185c576dd0 2024-03-21T10:38:56.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.813+0000 7f438fe00640 1 --2- 172.21.15.31:0/80847535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56185c5760f0 0x56185c580e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:38:56.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.813+0000 7f438f400640 1 --2- 172.21.15.31:0/80847535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56185c576dd0 0x56185c5791c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f437c00c6a0 tx=0x7f437c09bf80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1776c9c311957516 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:38:56.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.813+0000 7f4379600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:38:56.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.813+0000 7f4379600640 1 -- 172.21.15.31:0/80847535 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437c0a00d0 con 0x56185c576dd0 2024-03-21T10:38:56.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.814+0000 7f4379600640 1 -- 172.21.15.31:0/80847535 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f437c0985f0 con 0x56185c576dd0 2024-03-21T10:38:56.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.814+0000 7f43997cf740 1 -- 172.21.15.31:0/80847535 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4380044030 con 0x56185c576dd0 2024-03-21T10:38:56.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.814+0000 7f4379600640 1 -- 172.21.15.31:0/80847535 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f437c0a4990 con 0x56185c576dd0 2024-03-21T10:38:56.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.814+0000 7f43997cf740 10 client.25787 fetch_fsmap learned FSMap version 24 2024-03-21T10:38:56.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.814+0000 7f43997cf740 1 -- 172.21.15.31:0/80847535 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x56185c5eabf0 con 0x56185c576dd0 2024-03-21T10:38:56.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.814+0000 7f4379600640 1 -- 172.21.15.31:0/80847535 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4380044030 con 0x56185c576dd0 2024-03-21T10:38:56.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.814+0000 7f4379600640 1 -- 172.21.15.31:0/80847535 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f437c0a4be0 con 0x56185c576dd0 2024-03-21T10:38:56.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.814+0000 7f43997cf740 10 client.25787 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:38:56.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.814+0000 7f43997cf740 10 client.25787 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:38:56.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.814+0000 7f43997cf740 20 client.25787 populate_metadata read hostname 'smithi031' 2024-03-21T10:38:56.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.814+0000 7f43997cf740 1 --2- 172.21.15.31:0/80847535 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56185c5eaed0 0x56185c5ed2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:38:56.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.814+0000 7f43997cf740 4 client.25787 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:38:56.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.814+0000 7f43997cf740 1 -- 172.21.15.31:0/80847535 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56185c4e1ae0 con 0x56185c5eaed0 2024-03-21T10:38:56.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.815+0000 7f438fe00640 1 --2- 172.21.15.31:0/80847535 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56185c5eaed0 0x56185c5ed2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:38:56.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.815+0000 7f438fe00640 1 --2- 172.21.15.31:0/80847535 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56185c5eaed0 0x56185c5ed2c0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3be6d4f86fea0c1 server_cookie=30a446851f9326a3 in_seq=0 out_seq=0 2024-03-21T10:38:56.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.815+0000 7f4379600640 10 client.25787 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:56.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.815+0000 7f4379600640 1 -- 172.21.15.31:0/80847535 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56185c4e1ae0 con 0x56185c5eaed0 2024-03-21T10:38:56.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.815+0000 7f4379600640 10 client.25787 handle_command_reply: tid=0 2024-03-21T10:38:56.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.815+0000 7f43997cf740 1 client.25787 shutdown 2024-03-21T10:38:56.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.815+0000 7f43997cf740 1 -- 172.21.15.31:0/80847535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56185c576dd0 msgr2=0x56185c5791c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:56.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.815+0000 7f43997cf740 1 --2- 172.21.15.31:0/80847535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56185c576dd0 0x56185c5791c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f437c00c6a0 tx=0x7f437c09bf80 comp rx=0 tx=0).stop 2024-03-21T10:38:56.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.815+0000 7f43997cf740 1 -- 172.21.15.31:0/80847535 shutdown_connections 2024-03-21T10:38:56.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.815+0000 7f43997cf740 1 --2- 172.21.15.31:0/80847535 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56185c5eaed0 0x56185c5ed2c0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.815+0000 7f43997cf740 1 --2- 172.21.15.31:0/80847535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56185c57f460 0x56185c579700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.815+0000 7f43997cf740 1 --2- 172.21.15.31:0/80847535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56185c576dd0 0x56185c5791c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.815+0000 7f43997cf740 1 --2- 172.21.15.31:0/80847535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56185c5760f0 0x56185c580e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.815+0000 7f43997cf740 1 -- 172.21.15.31:0/80847535 >> 172.21.15.31:0/80847535 conn(0x56185c53d200 msgr2=0x56185c57a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:56.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.815+0000 7f4379600640 0 client.25787 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:38:56.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.816+0000 7f43997cf740 1 -- 172.21.15.31:0/80847535 shutdown_connections 2024-03-21T10:38:56.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.816+0000 7f43997cf740 1 -- 172.21.15.31:0/80847535 wait complete. 2024-03-21T10:38:56.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.816+0000 7f43997cf740 20 client.25787 trim_cache size 0 max 16384 2024-03-21T10:38:56.818 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:38:56.818 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-21T10:38:56.818 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-03-21T10:38:56.818 INFO:teuthology.orchestra.run.smithi031.stdout: "d747a398-552a-4905-bce0-c1ceee57211f": { 2024-03-21T10:38:56.818 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-03-21T10:38:56.819 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "d747a398-552a-4905-bce0-c1ceee57211f", 2024-03-21T10:38:56.819 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-03-21T10:38:56.819 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:38:56.819 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:38:56.819 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:38:56.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.816+0000 7f4395400640 1 -- 172.21.15.31:0/3598585125 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f43740a31a0 msgr2=0x7f43740a5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:56.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.816+0000 7f4395400640 1 --2- 172.21.15.31:0/3598585125 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f43740a31a0 0x7f43740a5660 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4380000f80 tx=0x7f4380009360 comp rx=0 tx=0).stop 2024-03-21T10:38:56.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.816+0000 7f4395400640 1 -- 172.21.15.31:0/3598585125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4390100d40 msgr2=0x7f439019e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:38:56.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.816+0000 7f4395400640 1 --2- 172.21.15.31:0/3598585125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4390100d40 0x7f439019e870 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f437c00e9f0 tx=0x7f437c00edf0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.817+0000 7f4395400640 1 -- 172.21.15.31:0/3598585125 shutdown_connections 2024-03-21T10:38:56.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.817+0000 7f4395400640 1 --2- 172.21.15.31:0/3598585125 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f43740a31a0 0x7f43740a5660 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.817+0000 7f4395400640 1 --2- 172.21.15.31:0/3598585125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4390105820 0x7f43901a2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.817+0000 7f4395400640 1 --2- 172.21.15.31:0/3598585125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4390100d40 0x7f439019e870 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.817+0000 7f4395400640 1 --2- 172.21.15.31:0/3598585125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4390100420 0x7f439019e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:38:56.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.817+0000 7f4395400640 1 -- 172.21.15.31:0/3598585125 >> 172.21.15.31:0/3598585125 conn(0x7f4390077130 msgr2=0x7f43901031d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:38:56.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.817+0000 7f4395400640 1 -- 172.21.15.31:0/3598585125 shutdown_connections 2024-03-21T10:38:56.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:38:56.817+0000 7f4395400640 1 -- 172.21.15.31:0/3598585125 wait complete. 2024-03-21T10:38:56.832 INFO:tasks.cephfs.filesystem:scrub status for tag:d747a398-552a-4905-bce0-c1ceee57211f - {'path': '/', 'tag': 'd747a398-552a-4905-bce0-c1ceee57211f', 'options': 'recursive,force'} 2024-03-21T10:38:57.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:56 smithi040 ceph-mon[36154]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-03-21T10:38:57.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:56 smithi031 ceph-mon[31673]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-03-21T10:38:57.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:56 smithi070 ceph-mon[36330]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-03-21T10:38:58.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:57 smithi040 ceph-mon[36154]: scrub queued for path: / 2024-03-21T10:38:58.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:57 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:38:58.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:57 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:38:58.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:57 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:38:58.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:57 smithi031 ceph-mon[31673]: scrub queued for path: / 2024-03-21T10:38:58.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:57 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:38:58.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:57 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:38:58.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:57 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:38:58.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:57 smithi070 ceph-mon[36330]: scrub queued for path: / 2024-03-21T10:38:58.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:57 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:38:58.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:57 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:38:58.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:57 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:38:59.052 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:38:58 smithi031 ceph-mon[31673]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-03-21T10:38:59.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:38:58 smithi040 ceph-mon[36154]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-03-21T10:38:59.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:38:58 smithi070 ceph-mon[36330]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-03-21T10:39:00.555 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:39:00 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:39:00.286+0000 7f4553e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-21T10:39:00.556 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:39:00 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:39:00.286+0000 7f4553e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-21T10:39:00.556 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:39:00 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:39:00.286+0000 7f4553e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-21T10:39:00.556 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:39:00 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:39:00.286+0000 7f4553e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-21T10:39:00.556 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:39:00 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:39:00.286+0000 7f4553e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-21T10:39:01.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:00 smithi040 ceph-mon[36154]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-03-21T10:39:01.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:00 smithi031 ceph-mon[31673]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-03-21T10:39:01.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:00 smithi070 ceph-mon[36330]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-03-21T10:39:02.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:01 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:39:02.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:01 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:39:02.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:01 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:39:02.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.835+0000 7f95b7e00640 1 -- 172.21.15.31:0/1688151211 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95b0100de0 msgr2=0x7f95b010da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:02.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.835+0000 7f95b7e00640 1 --2- 172.21.15.31:0/1688151211 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95b0100de0 0x7f95b010da10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f95a4009ca0 tx=0x7f95a402c100 comp rx=0 tx=0).stop 2024-03-21T10:39:02.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.835+0000 7f95b7e00640 1 -- 172.21.15.31:0/1688151211 shutdown_connections 2024-03-21T10:39:02.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.835+0000 7f95b7e00640 1 --2- 172.21.15.31:0/1688151211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95b010dfe0 0x7f95b01103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.835+0000 7f95b7e00640 1 --2- 172.21.15.31:0/1688151211 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95b0100de0 0x7f95b010da10 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.835+0000 7f95b7e00640 1 --2- 172.21.15.31:0/1688151211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95b0100430 0x7f95b0100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.835+0000 7f95b7e00640 1 -- 172.21.15.31:0/1688151211 >> 172.21.15.31:0/1688151211 conn(0x7f95b00fc040 msgr2=0x7f95b00fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:02.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.835+0000 7f95b7e00640 1 -- 172.21.15.31:0/1688151211 shutdown_connections 2024-03-21T10:39:02.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.836+0000 7f95b7e00640 1 -- 172.21.15.31:0/1688151211 wait complete. 2024-03-21T10:39:02.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.836+0000 7f95b7e00640 1 Processor -- start 2024-03-21T10:39:02.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.837+0000 7f95b7e00640 1 -- start start 2024-03-21T10:39:02.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.837+0000 7f95b7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95b0100430 0x7f95b019a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:02.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.837+0000 7f95b7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95b0100de0 0x7f95b019a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:02.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.837+0000 7f95b7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95b010dfe0 0x7f95b019ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:02.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.837+0000 7f95b7e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b0113c70 con 0x7f95b0100430 2024-03-21T10:39:02.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.837+0000 7f95b7e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b0113af0 con 0x7f95b010dfe0 2024-03-21T10:39:02.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.837+0000 7f95b7e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b0113df0 con 0x7f95b0100de0 2024-03-21T10:39:02.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.837+0000 7f95b4a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95b0100de0 0x7f95b019a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:02.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.837+0000 7f95b4a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95b0100de0 0x7f95b019a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:40846/0 (socket says 172.21.15.31:40846) 2024-03-21T10:39:02.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.837+0000 7f95b4a00640 1 -- 172.21.15.31:0/1764962173 learned_addr learned my addr 172.21.15.31:0/1764962173 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:02.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.837+0000 7f95b5e00640 1 --2- 172.21.15.31:0/1764962173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95b010dfe0 0x7f95b019ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:02.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.838+0000 7f95b5400640 1 --2- 172.21.15.31:0/1764962173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95b0100430 0x7f95b019a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:02.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.838+0000 7f95b4a00640 1 -- 172.21.15.31:0/1764962173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95b010dfe0 msgr2=0x7f95b019ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:02.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.838+0000 7f95b4a00640 1 --2- 172.21.15.31:0/1764962173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95b010dfe0 0x7f95b019ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.838+0000 7f95b4a00640 1 -- 172.21.15.31:0/1764962173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95b0100430 msgr2=0x7f95b019a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:02.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.838+0000 7f95b4a00640 1 --2- 172.21.15.31:0/1764962173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95b0100430 0x7f95b019a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.838+0000 7f95b4a00640 1 -- 172.21.15.31:0/1764962173 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b019f190 con 0x7f95b0100de0 2024-03-21T10:39:02.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.838+0000 7f95b5400640 1 --2- 172.21.15.31:0/1764962173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95b0100430 0x7f95b019a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:39:02.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.838+0000 7f95b4a00640 1 --2- 172.21.15.31:0/1764962173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95b0100de0 0x7f95b019a730 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f95a4005e30 tx=0x7f95a4035ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:02.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.838+0000 7f95ae000640 1 -- 172.21.15.31:0/1764962173 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a4039040 con 0x7f95b0100de0 2024-03-21T10:39:02.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.838+0000 7f95ae000640 1 -- 172.21.15.31:0/1764962173 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95a4044070 con 0x7f95b0100de0 2024-03-21T10:39:02.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.838+0000 7f95b7e00640 1 -- 172.21.15.31:0/1764962173 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95b019f420 con 0x7f95b0100de0 2024-03-21T10:39:02.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.838+0000 7f95ae000640 1 -- 172.21.15.31:0/1764962173 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a4004310 con 0x7f95b0100de0 2024-03-21T10:39:02.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.838+0000 7f95b7e00640 1 -- 172.21.15.31:0/1764962173 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95b01a6d10 con 0x7f95b0100de0 2024-03-21T10:39:02.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.841+0000 7f95babca740 1 Processor -- start 2024-03-21T10:39:02.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.841+0000 7f95ae000640 1 -- 172.21.15.31:0/1764962173 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) v1 ==== 138087+0+0 (secure 0 0 0) 0x7f95a40044b0 con 0x7f95b0100de0 2024-03-21T10:39:02.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.842+0000 7f95babca740 1 -- start start 2024-03-21T10:39:02.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.842+0000 7f95babca740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f9a954fd0 0x563f9a9553b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:02.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.842+0000 7f95babca740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f9a9558f0 0x563f9a95e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:02.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.842+0000 7f95babca740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f9a95e6c0 0x563f9a960ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:02.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.842+0000 7f95babca740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f9a9173b0 con 0x563f9a954fd0 2024-03-21T10:39:02.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.842+0000 7f95babca740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f9a8b9830 con 0x563f9a9558f0 2024-03-21T10:39:02.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.842+0000 7f95babca740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f9a8625d0 con 0x563f9a95e6c0 2024-03-21T10:39:02.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.842+0000 7f95b5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f9a9558f0 0x563f9a95e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:02.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.842+0000 7f95b4a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f9a95e6c0 0x563f9a960ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:02.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.842+0000 7f95b5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f9a9558f0 0x563f9a95e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38914/0 (socket says 172.21.15.31:38914) 2024-03-21T10:39:02.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.842+0000 7f95b5400640 1 -- 172.21.15.31:0/2147968241 learned_addr learned my addr 172.21.15.31:0/2147968241 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:02.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.842+0000 7f95b5e00640 1 --2- 172.21.15.31:0/2147968241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f9a954fd0 0x563f9a9553b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:02.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.843+0000 7f95b5400640 1 -- 172.21.15.31:0/2147968241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f9a95e6c0 msgr2=0x563f9a960ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:02.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.843+0000 7f95b5400640 1 --2- 172.21.15.31:0/2147968241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f9a95e6c0 0x563f9a960ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.843+0000 7f95b5400640 1 -- 172.21.15.31:0/2147968241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f9a954fd0 msgr2=0x563f9a9553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:02.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.843+0000 7f95b5400640 1 --2- 172.21.15.31:0/2147968241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f9a954fd0 0x563f9a9553b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.843+0000 7f95b5400640 1 -- 172.21.15.31:0/2147968241 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f9a87a2c0 con 0x563f9a9558f0 2024-03-21T10:39:02.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.843+0000 7f95b5400640 1 --2- 172.21.15.31:0/2147968241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f9a9558f0 0x563f9a95e180 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f95a0009c40 tx=0x7f95a000e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=accf98dd2f3e0555 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:02.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.843+0000 7f959e600640 1 -- 172.21.15.31:0/2147968241 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a0016070 con 0x563f9a9558f0 2024-03-21T10:39:02.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.843+0000 7f95ae000640 1 --2- 172.21.15.31:0/1764962173 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f95980a3220 0x7f95980a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:02.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.843+0000 7f959e600640 1 -- 172.21.15.31:0/2147968241 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95a0011950 con 0x563f9a9558f0 2024-03-21T10:39:02.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.843+0000 7f959e600640 1 -- 172.21.15.31:0/2147968241 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a00109d0 con 0x563f9a9558f0 2024-03-21T10:39:02.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.844+0000 7f95ae000640 1 -- 172.21.15.31:0/1764962173 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f95a40f2370 con 0x7f95b0100de0 2024-03-21T10:39:02.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.844+0000 7f95b5e00640 1 --2- 172.21.15.31:0/1764962173 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f95980a3220 0x7f95980a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:02.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.844+0000 7f95babca740 1 -- 172.21.15.31:0/2147968241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f9a9558f0 msgr2=0x563f9a95e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:02.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.844+0000 7f95babca740 1 --2- 172.21.15.31:0/2147968241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f9a9558f0 0x563f9a95e180 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f95a0009c40 tx=0x7f95a000e6c0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.844+0000 7f95babca740 1 -- 172.21.15.31:0/2147968241 shutdown_connections 2024-03-21T10:39:02.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.844+0000 7f95babca740 1 --2- 172.21.15.31:0/2147968241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f9a95e6c0 0x563f9a960ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.844+0000 7f95babca740 1 --2- 172.21.15.31:0/2147968241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f9a9558f0 0x563f9a95e180 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.844+0000 7f95babca740 1 --2- 172.21.15.31:0/2147968241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f9a954fd0 0x563f9a9553b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.844+0000 7f95babca740 1 -- 172.21.15.31:0/2147968241 >> 172.21.15.31:0/2147968241 conn(0x563f9a94f8f0 msgr2=0x563f9a94fcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:02.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.844+0000 7f95babca740 1 -- 172.21.15.31:0/2147968241 shutdown_connections 2024-03-21T10:39:02.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.845+0000 7f95b5e00640 1 --2- 172.21.15.31:0/1764962173 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f95980a3220 0x7f95980a56e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f95b0069b10 tx=0x7f95a8005f90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:02.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.845+0000 7f95babca740 1 -- 172.21.15.31:0/2147968241 wait complete. 2024-03-21T10:39:02.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.845+0000 7f95babca740 1 -- start start 2024-03-21T10:39:02.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.845+0000 7f95babca740 1 Processor -- start 2024-03-21T10:39:02.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.846+0000 7f95babca740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f9a954fd0 0x563f9a95b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:02.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.846+0000 7f95babca740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f9a9558f0 0x563f9a95bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:02.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.846+0000 7f95babca740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f9a95e6c0 0x563f9a95c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:02.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.846+0000 7f95babca740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f9a875a20 con 0x563f9a954fd0 2024-03-21T10:39:02.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.846+0000 7f95babca740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f9a8b58d0 con 0x563f9a95e6c0 2024-03-21T10:39:02.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.846+0000 7f95babca740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f9a9612f0 con 0x563f9a9558f0 2024-03-21T10:39:02.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.846+0000 7f95babca740 10 client.0 _ll_register_callbacks cb 0x563f9a8b21c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:39:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.846+0000 7f95b4a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f9a9558f0 0x563f9a95bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.846+0000 7f95b5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f9a95e6c0 0x563f9a95c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.846+0000 7f95b5400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f9a954fd0 0x563f9a95b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.846+0000 7f95b4a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f9a9558f0 0x563f9a95bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:40874/0 (socket says 172.21.15.31:40874) 2024-03-21T10:39:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.846+0000 7f95b4a00640 1 -- 172.21.15.31:0/3256157574 learned_addr learned my addr 172.21.15.31:0/3256157574 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.846+0000 7f95b4a00640 1 -- 172.21.15.31:0/3256157574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f9a95e6c0 msgr2=0x563f9a95c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.846+0000 7f95b4a00640 1 --2- 172.21.15.31:0/3256157574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f9a95e6c0 0x563f9a95c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.846+0000 7f95b4a00640 1 -- 172.21.15.31:0/3256157574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f9a954fd0 msgr2=0x563f9a95b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.846+0000 7f95b4a00640 1 --2- 172.21.15.31:0/3256157574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f9a954fd0 0x563f9a95b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.846+0000 7f95b4a00640 1 -- 172.21.15.31:0/3256157574 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563f9a5ce210 con 0x563f9a9558f0 2024-03-21T10:39:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.847+0000 7f95b4a00640 1 --2- 172.21.15.31:0/3256157574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f9a9558f0 0x563f9a95bba0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f95a4033000 tx=0x7f95a4009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6631fd291f00d9ec server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.847+0000 7f9592a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:39:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.847+0000 7f9592a00640 1 -- 172.21.15.31:0/3256157574 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a40bddc0 con 0x563f9a9558f0 2024-03-21T10:39:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.847+0000 7f9592a00640 1 -- 172.21.15.31:0/3256157574 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f95a4003a00 con 0x563f9a9558f0 2024-03-21T10:39:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.847+0000 7f95babca740 1 -- 172.21.15.31:0/3256157574 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f95a0011950 con 0x563f9a9558f0 2024-03-21T10:39:02.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.847+0000 7f9592a00640 1 -- 172.21.15.31:0/3256157574 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95a40f2170 con 0x563f9a9558f0 2024-03-21T10:39:02.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.847+0000 7f95babca740 10 client.25792 fetch_fsmap learned FSMap version 24 2024-03-21T10:39:02.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.847+0000 7f95babca740 1 -- 172.21.15.31:0/3256157574 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x563f9a69a620 con 0x563f9a9558f0 2024-03-21T10:39:02.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.847+0000 7f9592a00640 1 -- 172.21.15.31:0/3256157574 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95a0011950 con 0x563f9a9558f0 2024-03-21T10:39:02.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.847+0000 7f9592a00640 1 -- 172.21.15.31:0/3256157574 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f95a403c030 con 0x563f9a9558f0 2024-03-21T10:39:02.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.847+0000 7f95babca740 10 client.25792 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:39:02.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.847+0000 7f95babca740 10 client.25792 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:39:02.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.847+0000 7f95babca740 20 client.25792 populate_metadata read hostname 'smithi031' 2024-03-21T10:39:02.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.847+0000 7f95babca740 1 --2- 172.21.15.31:0/3256157574 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563f9a96e660 0x563f9a970a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:02.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.848+0000 7f95babca740 4 client.25792 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:39:02.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.848+0000 7f95babca740 1 -- 172.21.15.31:0/3256157574 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563f9a8c6d70 con 0x563f9a96e660 2024-03-21T10:39:02.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.848+0000 7f95b5400640 1 --2- 172.21.15.31:0/3256157574 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563f9a96e660 0x563f9a970a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:02.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.848+0000 7f95b5400640 1 --2- 172.21.15.31:0/3256157574 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563f9a96e660 0x563f9a970a50 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=217bb452fb093ea3 server_cookie=d74421dd97098fd3 in_seq=0 out_seq=0 2024-03-21T10:39:02.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.848+0000 7f9592a00640 10 client.25792 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:02.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.849+0000 7f9592a00640 1 -- 172.21.15.31:0/3256157574 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563f9a8c6d70 con 0x563f9a96e660 2024-03-21T10:39:02.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.849+0000 7f9592a00640 10 client.25792 handle_command_reply: tid=0 2024-03-21T10:39:02.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.849+0000 7f95babca740 1 client.25792 shutdown 2024-03-21T10:39:02.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.849+0000 7f95babca740 1 -- 172.21.15.31:0/3256157574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f9a9558f0 msgr2=0x563f9a95bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:02.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.849+0000 7f95babca740 1 --2- 172.21.15.31:0/3256157574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f9a9558f0 0x563f9a95bba0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f95a4033000 tx=0x7f95a4009800 comp rx=0 tx=0).stop 2024-03-21T10:39:02.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.849+0000 7f95babca740 1 -- 172.21.15.31:0/3256157574 shutdown_connections 2024-03-21T10:39:02.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.849+0000 7f95babca740 1 --2- 172.21.15.31:0/3256157574 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563f9a96e660 0x563f9a970a50 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.849+0000 7f95babca740 1 --2- 172.21.15.31:0/3256157574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f9a95e6c0 0x563f9a95c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.849+0000 7f95babca740 1 --2- 172.21.15.31:0/3256157574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f9a9558f0 0x563f9a95bba0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.849+0000 7f95babca740 1 --2- 172.21.15.31:0/3256157574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f9a954fd0 0x563f9a95b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.849+0000 7f9592a00640 0 client.25792 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:02.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.849+0000 7f95babca740 1 -- 172.21.15.31:0/3256157574 >> 172.21.15.31:0/3256157574 conn(0x563f9a94f8f0 msgr2=0x563f9a951640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:02.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.850+0000 7f95babca740 1 -- 172.21.15.31:0/3256157574 shutdown_connections 2024-03-21T10:39:02.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.850+0000 7f95babca740 1 -- 172.21.15.31:0/3256157574 wait complete. 2024-03-21T10:39:02.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.850+0000 7f95babca740 20 client.25792 trim_cache size 0 max 16384 2024-03-21T10:39:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.865+0000 7f95babca740 1 Processor -- start 2024-03-21T10:39:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.865+0000 7f95babca740 1 -- start start 2024-03-21T10:39:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.865+0000 7f95babca740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f9a954fd0 0x563f9a95dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.866+0000 7f95babca740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f9a9558f0 0x563f9a969a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.866+0000 7f95babca740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f9a95e6c0 0x563f9a969f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.866+0000 7f95babca740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b01b3090 con 0x563f9a9558f0 2024-03-21T10:39:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.866+0000 7f95babca740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b01b0b90 con 0x563f9a954fd0 2024-03-21T10:39:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.866+0000 7f95babca740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b01aec30 con 0x563f9a95e6c0 2024-03-21T10:39:02.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.866+0000 7f95b5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f9a954fd0 0x563f9a95dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:02.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.866+0000 7f95b4a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f9a9558f0 0x563f9a969a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:02.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.866+0000 7f95b5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f9a954fd0 0x563f9a95dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38940/0 (socket says 172.21.15.31:38940) 2024-03-21T10:39:02.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.866+0000 7f95b5400640 1 -- 172.21.15.31:0/2183031097 learned_addr learned my addr 172.21.15.31:0/2183031097 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:02.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.866+0000 7f95b5e00640 1 --2- 172.21.15.31:0/2183031097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f9a95e6c0 0x563f9a969f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:02.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.866+0000 7f95b5400640 1 -- 172.21.15.31:0/2183031097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f9a95e6c0 msgr2=0x563f9a969f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:02.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.866+0000 7f95b5400640 1 --2- 172.21.15.31:0/2183031097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f9a95e6c0 0x563f9a969f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.866+0000 7f95b5400640 1 -- 172.21.15.31:0/2183031097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f9a9558f0 msgr2=0x563f9a969a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:02.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.866+0000 7f95b5400640 1 --2- 172.21.15.31:0/2183031097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f9a9558f0 0x563f9a969a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.866+0000 7f95b5400640 1 -- 172.21.15.31:0/2183031097 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f9a94f720 con 0x563f9a954fd0 2024-03-21T10:39:02.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.866+0000 7f95b4a00640 1 --2- 172.21.15.31:0/2183031097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f9a9558f0 0x563f9a969a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:39:02.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.867+0000 7f95b5400640 1 --2- 172.21.15.31:0/2183031097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f9a954fd0 0x563f9a95dd90 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f95a0009c40 tx=0x7f95a0005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=718144167536eead server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.867+0000 7f959dc00640 1 -- 172.21.15.31:0/2183031097 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a0016040 con 0x563f9a954fd0 2024-03-21T10:39:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.867+0000 7f959dc00640 1 -- 172.21.15.31:0/2183031097 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95a002a250 con 0x563f9a954fd0 2024-03-21T10:39:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.867+0000 7f959dc00640 1 -- 172.21.15.31:0/2183031097 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a0005240 con 0x563f9a954fd0 2024-03-21T10:39:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.867+0000 7f95babca740 1 -- 172.21.15.31:0/2183031097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f9a954fd0 msgr2=0x563f9a95dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.867+0000 7f95babca740 1 --2- 172.21.15.31:0/2183031097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f9a954fd0 0x563f9a95dd90 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f95a0009c40 tx=0x7f95a0005fd0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.868+0000 7f95babca740 1 -- 172.21.15.31:0/2183031097 shutdown_connections 2024-03-21T10:39:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.868+0000 7f95babca740 1 --2- 172.21.15.31:0/2183031097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f9a95e6c0 0x563f9a969f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.868+0000 7f95babca740 1 --2- 172.21.15.31:0/2183031097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f9a9558f0 0x563f9a969a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.868+0000 7f95babca740 1 --2- 172.21.15.31:0/2183031097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f9a954fd0 0x563f9a95dd90 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.868+0000 7f95babca740 1 -- 172.21.15.31:0/2183031097 >> 172.21.15.31:0/2183031097 conn(0x563f9a94f8f0 msgr2=0x563f9a9542b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.868+0000 7f95babca740 1 -- 172.21.15.31:0/2183031097 shutdown_connections 2024-03-21T10:39:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.868+0000 7f95babca740 1 -- 172.21.15.31:0/2183031097 wait complete. 2024-03-21T10:39:02.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.868+0000 7f95babca740 1 -- start start 2024-03-21T10:39:02.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.868+0000 7f95babca740 1 Processor -- start 2024-03-21T10:39:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.869+0000 7f95babca740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f9a954fd0 0x563f9a961b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.869+0000 7f95babca740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f9a9558f0 0x563f9a962060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.869+0000 7f95babca740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f9a95e6c0 0x563f9a9625a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.869+0000 7f95babca740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b01af230 con 0x563f9a9558f0 2024-03-21T10:39:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.869+0000 7f95babca740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b01b0530 con 0x563f9a95e6c0 2024-03-21T10:39:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.869+0000 7f95babca740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b01af870 con 0x563f9a954fd0 2024-03-21T10:39:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.869+0000 7f95babca740 10 client.0 _ll_register_callbacks cb 0x7f95a40f5950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:39:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.869+0000 7f95b5400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f9a954fd0 0x563f9a961b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.869+0000 7f95b5400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f9a954fd0 0x563f9a961b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:40886/0 (socket says 172.21.15.31:40886) 2024-03-21T10:39:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.869+0000 7f95b5400640 1 -- 172.21.15.31:0/2407811454 learned_addr learned my addr 172.21.15.31:0/2407811454 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.869+0000 7f95b5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f9a95e6c0 0x563f9a9625a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.869+0000 7f95b4a00640 1 --2- 172.21.15.31:0/2407811454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f9a9558f0 0x563f9a962060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.869+0000 7f95b5400640 1 -- 172.21.15.31:0/2407811454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f9a95e6c0 msgr2=0x563f9a9625a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.869+0000 7f95b5400640 1 --2- 172.21.15.31:0/2407811454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f9a95e6c0 0x563f9a9625a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.869+0000 7f95b5400640 1 -- 172.21.15.31:0/2407811454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f9a9558f0 msgr2=0x563f9a962060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:02.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.869+0000 7f95b5400640 1 --2- 172.21.15.31:0/2407811454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f9a9558f0 0x563f9a962060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.869+0000 7f95b5400640 1 -- 172.21.15.31:0/2407811454 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563f9a9c8f90 con 0x563f9a954fd0 2024-03-21T10:39:02.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.869+0000 7f95b4a00640 1 --2- 172.21.15.31:0/2407811454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f9a9558f0 0x563f9a962060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:39:02.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.870+0000 7f95b5400640 1 --2- 172.21.15.31:0/2407811454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f9a954fd0 0x563f9a961b20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f95a0005fa0 tx=0x7f95a00105b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c71165a8f2103a89 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:02.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.870+0000 7f9592a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:39:02.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.870+0000 7f9592a00640 1 -- 172.21.15.31:0/2407811454 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a0037060 con 0x563f9a954fd0 2024-03-21T10:39:02.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.870+0000 7f95babca740 1 -- 172.21.15.31:0/2407811454 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f95a002a250 con 0x563f9a954fd0 2024-03-21T10:39:02.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.870+0000 7f9592a00640 1 -- 172.21.15.31:0/2407811454 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f95a0004230 con 0x563f9a954fd0 2024-03-21T10:39:02.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.870+0000 7f9592a00640 1 -- 172.21.15.31:0/2407811454 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95a003e050 con 0x563f9a954fd0 2024-03-21T10:39:02.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.870+0000 7f95babca740 10 client.25798 fetch_fsmap learned FSMap version 24 2024-03-21T10:39:02.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.870+0000 7f95babca740 1 -- 172.21.15.31:0/2407811454 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x563f9a9ca280 con 0x563f9a954fd0 2024-03-21T10:39:02.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.870+0000 7f9592a00640 1 -- 172.21.15.31:0/2407811454 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95a002a250 con 0x563f9a954fd0 2024-03-21T10:39:02.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.871+0000 7f9592a00640 1 -- 172.21.15.31:0/2407811454 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f95a000f040 con 0x563f9a954fd0 2024-03-21T10:39:02.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.871+0000 7f95babca740 10 client.25798 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:39:02.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.871+0000 7f95babca740 10 client.25798 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:39:02.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.871+0000 7f95babca740 20 client.25798 populate_metadata read hostname 'smithi031' 2024-03-21T10:39:02.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.871+0000 7f95babca740 1 --2- 172.21.15.31:0/2407811454 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563f9a9ca700 0x563f9a9ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:02.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.871+0000 7f95babca740 4 client.25798 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:39:02.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.871+0000 7f95babca740 1 -- 172.21.15.31:0/2407811454 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563f9a8c10c0 con 0x563f9a9ca700 2024-03-21T10:39:02.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.871+0000 7f95b4a00640 1 --2- 172.21.15.31:0/2407811454 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563f9a9ca700 0x563f9a9ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:02.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.872+0000 7f9592a00640 10 client.25798 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:02.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.872+0000 7f95b4a00640 1 --2- 172.21.15.31:0/2407811454 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563f9a9ca700 0x563f9a9ccaf0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b00a3d5d205f15a5 server_cookie=cb16667c2aee74a6 in_seq=0 out_seq=0 2024-03-21T10:39:02.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.878+0000 7f9592a00640 1 -- 172.21.15.31:0/2407811454 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563f9a8c10c0 con 0x563f9a9ca700 2024-03-21T10:39:02.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.878+0000 7f9592a00640 10 client.25798 handle_command_reply: tid=0 2024-03-21T10:39:02.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.878+0000 7f95babca740 1 client.25798 shutdown 2024-03-21T10:39:02.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.878+0000 7f95babca740 1 -- 172.21.15.31:0/2407811454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f9a954fd0 msgr2=0x563f9a961b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:02.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.878+0000 7f95babca740 1 --2- 172.21.15.31:0/2407811454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f9a954fd0 0x563f9a961b20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f95a0005fa0 tx=0x7f95a00105b0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.879+0000 7f95babca740 1 -- 172.21.15.31:0/2407811454 shutdown_connections 2024-03-21T10:39:02.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.879+0000 7f95babca740 1 --2- 172.21.15.31:0/2407811454 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563f9a9ca700 0x563f9a9ccaf0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.879+0000 7f95babca740 1 --2- 172.21.15.31:0/2407811454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f9a95e6c0 0x563f9a9625a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.879+0000 7f95babca740 1 --2- 172.21.15.31:0/2407811454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f9a9558f0 0x563f9a962060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.879+0000 7f95babca740 1 --2- 172.21.15.31:0/2407811454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f9a954fd0 0x563f9a961b20 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.879+0000 7f9592a00640 0 client.25798 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:02.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.879+0000 7f95babca740 1 -- 172.21.15.31:0/2407811454 >> 172.21.15.31:0/2407811454 conn(0x563f9a94f8f0 msgr2=0x563f9a95b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:02.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.879+0000 7f95babca740 1 -- 172.21.15.31:0/2407811454 shutdown_connections 2024-03-21T10:39:02.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.879+0000 7f95babca740 1 -- 172.21.15.31:0/2407811454 wait complete. 2024-03-21T10:39:02.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.879+0000 7f95babca740 20 client.25798 trim_cache size 0 max 16384 2024-03-21T10:39:02.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:39:02.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:39:02.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:39:02.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:39:02.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.880+0000 7f95b7e00640 1 -- 172.21.15.31:0/1764962173 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f95980a3220 msgr2=0x7f95980a56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:02.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.880+0000 7f95b7e00640 1 --2- 172.21.15.31:0/1764962173 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f95980a3220 0x7f95980a56e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f95b0069b10 tx=0x7f95a8005f90 comp rx=0 tx=0).stop 2024-03-21T10:39:02.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.880+0000 7f95b7e00640 1 -- 172.21.15.31:0/1764962173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95b0100de0 msgr2=0x7f95b019a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:02.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.880+0000 7f95b7e00640 1 --2- 172.21.15.31:0/1764962173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95b0100de0 0x7f95b019a730 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f95a4005e30 tx=0x7f95a4035ae0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.880+0000 7f95b7e00640 1 -- 172.21.15.31:0/1764962173 shutdown_connections 2024-03-21T10:39:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.880+0000 7f95b7e00640 1 --2- 172.21.15.31:0/1764962173 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f95980a3220 0x7f95980a56e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.880+0000 7f95b7e00640 1 --2- 172.21.15.31:0/1764962173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95b010dfe0 0x7f95b019ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.880+0000 7f95b7e00640 1 --2- 172.21.15.31:0/1764962173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95b0100de0 0x7f95b019a730 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.880+0000 7f95b7e00640 1 --2- 172.21.15.31:0/1764962173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95b0100430 0x7f95b019a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.880+0000 7f95b7e00640 1 -- 172.21.15.31:0/1764962173 >> 172.21.15.31:0/1764962173 conn(0x7f95b00fc040 msgr2=0x7f95b00fe430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.880+0000 7f95b7e00640 1 -- 172.21.15.31:0/1764962173 shutdown_connections 2024-03-21T10:39:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:02.880+0000 7f95b7e00640 1 -- 172.21.15.31:0/1764962173 wait complete. 2024-03-21T10:39:03.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:02 smithi031 ceph-mon[31673]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-03-21T10:39:03.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:02 smithi031 ceph-mon[31673]: mgrmap e26: x(active, since 28s), standbys: y, z 2024-03-21T10:39:03.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:02 smithi070 ceph-mon[36330]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-03-21T10:39:03.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:02 smithi070 ceph-mon[36330]: mgrmap e26: x(active, since 28s), standbys: y, z 2024-03-21T10:39:03.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:02 smithi040 ceph-mon[36154]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-03-21T10:39:03.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:02 smithi040 ceph-mon[36154]: mgrmap e26: x(active, since 28s), standbys: y, z 2024-03-21T10:39:05.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:04 smithi031 ceph-mon[31673]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 10 op/s 2024-03-21T10:39:05.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:04 smithi070 ceph-mon[36330]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 10 op/s 2024-03-21T10:39:05.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:04 smithi040 ceph-mon[36154]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 10 op/s 2024-03-21T10:39:07.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:06 smithi031 ceph-mon[31673]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 KiB/s wr, 30 op/s 2024-03-21T10:39:07.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:06 smithi070 ceph-mon[36330]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 KiB/s wr, 30 op/s 2024-03-21T10:39:07.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:06 smithi040 ceph-mon[36154]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 KiB/s wr, 30 op/s 2024-03-21T10:39:09.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:08 smithi031 ceph-mon[31673]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-03-21T10:39:09.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:08 smithi070 ceph-mon[36330]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-03-21T10:39:09.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:08 smithi040 ceph-mon[36154]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-03-21T10:39:11.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:10 smithi031 ceph-mon[31673]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-03-21T10:39:11.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:10 smithi070 ceph-mon[36330]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-03-21T10:39:11.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:10 smithi040 ceph-mon[36154]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-03-21T10:39:13.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:12 smithi031 ceph-mon[31673]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-21T10:39:13.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:12 smithi070 ceph-mon[36330]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-21T10:39:13.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:12 smithi040 ceph-mon[36154]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-21T10:39:14.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:14 smithi031 ceph-mon[31673]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-03-21T10:39:14.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:14 smithi070 ceph-mon[36330]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-03-21T10:39:14.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:14 smithi040 ceph-mon[36154]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-03-21T10:39:16.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.034+0000 7f5f0d200640 1 -- 172.21.15.31:0/3818987289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f081014a0 msgr2=0x7f5f08101920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:16.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.034+0000 7f5f0d200640 1 --2- 172.21.15.31:0/3818987289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f081014a0 0x7f5f08101920 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4009930 tx=0x7f5ef402bc20 comp rx=0 tx=0).stop 2024-03-21T10:39:16.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.034+0000 7f5f0d200640 1 -- 172.21.15.31:0/3818987289 shutdown_connections 2024-03-21T10:39:16.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.034+0000 7f5f0d200640 1 --2- 172.21.15.31:0/3818987289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f08101e60 0x7f5f0810e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.034+0000 7f5f0d200640 1 --2- 172.21.15.31:0/3818987289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f081014a0 0x7f5f08101920 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.034+0000 7f5f0d200640 1 --2- 172.21.15.31:0/3818987289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f08107560 0x7f5f08107940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.034+0000 7f5f0d200640 1 -- 172.21.15.31:0/3818987289 >> 172.21.15.31:0/3818987289 conn(0x7f5f080fcfb0 msgr2=0x7f5f080ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:16.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.034+0000 7f5f0d200640 1 -- 172.21.15.31:0/3818987289 shutdown_connections 2024-03-21T10:39:16.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.034+0000 7f5f0d200640 1 -- 172.21.15.31:0/3818987289 wait complete. 2024-03-21T10:39:16.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.035+0000 7f5f0d200640 1 Processor -- start 2024-03-21T10:39:16.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.035+0000 7f5f0d200640 1 -- start start 2024-03-21T10:39:16.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.036+0000 7f5f0d200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f081014a0 0x7f5f08199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:16.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.036+0000 7f5f0d200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f08101e60 0x7f5f0819a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:16.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.036+0000 7f5f0d200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f08107560 0x7f5f0819e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:16.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.036+0000 7f5f0d200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f081118f0 con 0x7f5f08107560 2024-03-21T10:39:16.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.036+0000 7f5f0d200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f08111770 con 0x7f5f081014a0 2024-03-21T10:39:16.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.036+0000 7f5f0d200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f08111a70 con 0x7f5f08101e60 2024-03-21T10:39:16.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.036+0000 7f5f06a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f08101e60 0x7f5f0819a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:16.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.036+0000 7f5f07400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f081014a0 0x7f5f08199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:16.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.036+0000 7f5f06a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f08101e60 0x7f5f0819a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:55098/0 (socket says 172.21.15.31:55098) 2024-03-21T10:39:16.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.036+0000 7f5f06a00640 1 -- 172.21.15.31:0/4173309870 learned_addr learned my addr 172.21.15.31:0/4173309870 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:16.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.036+0000 7f5f07e00640 1 --2- 172.21.15.31:0/4173309870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f08107560 0x7f5f0819e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:16.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.037+0000 7f5f06a00640 1 -- 172.21.15.31:0/4173309870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f081014a0 msgr2=0x7f5f08199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:16.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.037+0000 7f5f06a00640 1 --2- 172.21.15.31:0/4173309870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f081014a0 0x7f5f08199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.037+0000 7f5f06a00640 1 -- 172.21.15.31:0/4173309870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f08107560 msgr2=0x7f5f0819e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:16.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.037+0000 7f5f06a00640 1 --2- 172.21.15.31:0/4173309870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f08107560 0x7f5f0819e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.037+0000 7f5f06a00640 1 -- 172.21.15.31:0/4173309870 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f0819ef30 con 0x7f5f08101e60 2024-03-21T10:39:16.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.037+0000 7f5f06a00640 1 --2- 172.21.15.31:0/4173309870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f08101e60 0x7f5f0819a480 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4005ab0 tx=0x7f5ef40042e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:16.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.037+0000 7f5effe00640 1 -- 172.21.15.31:0/4173309870 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef4043020 con 0x7f5f08101e60 2024-03-21T10:39:16.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.037+0000 7f5effe00640 1 -- 172.21.15.31:0/4173309870 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ef40044d0 con 0x7f5f08101e60 2024-03-21T10:39:16.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.037+0000 7f5f0d200640 1 -- 172.21.15.31:0/4173309870 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f0819f1c0 con 0x7f5f08101e60 2024-03-21T10:39:16.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.037+0000 7f5effe00640 1 -- 172.21.15.31:0/4173309870 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef4003bb0 con 0x7f5f08101e60 2024-03-21T10:39:16.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.037+0000 7f5f0d200640 1 -- 172.21.15.31:0/4173309870 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f081a6ab0 con 0x7f5f08101e60 2024-03-21T10:39:16.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.040+0000 7f5effe00640 1 -- 172.21.15.31:0/4173309870 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) v1 ==== 138087+0+0 (secure 0 0 0) 0x7f5ef4046020 con 0x7f5f08101e60 2024-03-21T10:39:16.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.041+0000 7f5f1158d740 1 Processor -- start 2024-03-21T10:39:16.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.041+0000 7f5f1158d740 1 -- start start 2024-03-21T10:39:16.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.041+0000 7f5f1158d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615272d01e0 0x5615272d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:16.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.041+0000 7f5f1158d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615272d0b00 0x5615272d9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:16.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.041+0000 7f5f1158d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615272d98d0 0x5615272dbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:16.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.041+0000 7f5f1158d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615272923b0 con 0x5615272d98d0 2024-03-21T10:39:16.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.041+0000 7f5f1158d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561527234830 con 0x5615272d0b00 2024-03-21T10:39:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.041+0000 7f5f1158d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615271dd5d0 con 0x5615272d01e0 2024-03-21T10:39:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.042+0000 7f5f07e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615272d01e0 0x5615272d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.042+0000 7f5f07e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615272d01e0 0x5615272d05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:55112/0 (socket says 172.21.15.31:55112) 2024-03-21T10:39:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.042+0000 7f5f07e00640 1 -- 172.21.15.31:0/3657907812 learned_addr learned my addr 172.21.15.31:0/3657907812 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.042+0000 7f5f07400640 1 --2- 172.21.15.31:0/3657907812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615272d0b00 0x5615272d9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.042+0000 7f5f06a00640 1 --2- 172.21.15.31:0/3657907812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615272d98d0 0x5615272dbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.042+0000 7f5f07e00640 1 -- 172.21.15.31:0/3657907812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615272d0b00 msgr2=0x5615272d9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.042+0000 7f5f07e00640 1 --2- 172.21.15.31:0/3657907812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615272d0b00 0x5615272d9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.042+0000 7f5f07e00640 1 -- 172.21.15.31:0/3657907812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615272d98d0 msgr2=0x5615272dbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.042+0000 7f5f07e00640 1 --2- 172.21.15.31:0/3657907812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615272d98d0 0x5615272dbcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.042+0000 7f5f07e00640 1 -- 172.21.15.31:0/3657907812 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5615271f52c0 con 0x5615272d01e0 2024-03-21T10:39:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.042+0000 7f5f07e00640 1 --2- 172.21.15.31:0/3657907812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615272d01e0 0x5615272d05c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5f00009f20 tx=0x7f5f0000c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66388c89fc44eefd server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.043+0000 7f5effe00640 1 --2- 172.21.15.31:0/4173309870 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f5eec0a3210 0x7f5eec0a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.043+0000 7f5efcc00640 1 -- 172.21.15.31:0/3657907812 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0001a070 con 0x5615272d01e0 2024-03-21T10:39:16.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.043+0000 7f5efcc00640 1 -- 172.21.15.31:0/3657907812 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f000158f0 con 0x5615272d01e0 2024-03-21T10:39:16.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.043+0000 7f5efcc00640 1 -- 172.21.15.31:0/3657907812 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0000a900 con 0x5615272d01e0 2024-03-21T10:39:16.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.043+0000 7f5effe00640 1 -- 172.21.15.31:0/4173309870 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f5ef40f1cc0 con 0x7f5f08101e60 2024-03-21T10:39:16.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.043+0000 7f5f07400640 1 --2- 172.21.15.31:0/4173309870 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f5eec0a3210 0x7f5eec0a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:16.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.043+0000 7f5f1158d740 1 -- 172.21.15.31:0/3657907812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615272d01e0 msgr2=0x5615272d05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:16.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.043+0000 7f5f1158d740 1 --2- 172.21.15.31:0/3657907812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615272d01e0 0x5615272d05c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5f00009f20 tx=0x7f5f0000c8b0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.044+0000 7f5f1158d740 1 -- 172.21.15.31:0/3657907812 shutdown_connections 2024-03-21T10:39:16.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.044+0000 7f5f1158d740 1 --2- 172.21.15.31:0/3657907812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615272d98d0 0x5615272dbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.044+0000 7f5f1158d740 1 --2- 172.21.15.31:0/3657907812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615272d0b00 0x5615272d9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.044+0000 7f5f1158d740 1 --2- 172.21.15.31:0/3657907812 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615272d01e0 0x5615272d05c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.044+0000 7f5f1158d740 1 -- 172.21.15.31:0/3657907812 >> 172.21.15.31:0/3657907812 conn(0x5615272cab00 msgr2=0x5615272caee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.044+0000 7f5f1158d740 1 -- 172.21.15.31:0/3657907812 shutdown_connections 2024-03-21T10:39:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.044+0000 7f5f07400640 1 --2- 172.21.15.31:0/4173309870 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f5eec0a3210 0x7f5eec0a56d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x5615272cfc90 tx=0x7f5ef8005f90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.044+0000 7f5f1158d740 1 -- 172.21.15.31:0/3657907812 wait complete. 2024-03-21T10:39:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.044+0000 7f5f1158d740 1 -- start start 2024-03-21T10:39:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.044+0000 7f5f1158d740 1 Processor -- start 2024-03-21T10:39:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.045+0000 7f5f1158d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615272d01e0 0x5615272d6970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.045+0000 7f5f1158d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615272d0b00 0x5615272d6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.045+0000 7f5f1158d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615272d98d0 0x5615272d73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.045+0000 7f5f1158d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615271f0a20 con 0x5615272d98d0 2024-03-21T10:39:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.045+0000 7f5f1158d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615272308d0 con 0x5615272d0b00 2024-03-21T10:39:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.045+0000 7f5f1158d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615272dc500 con 0x5615272d01e0 2024-03-21T10:39:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.045+0000 7f5f07400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615272d01e0 0x5615272d6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.045+0000 7f5f1158d740 10 client.0 _ll_register_callbacks cb 0x56152722d1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:39:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.045+0000 7f5f07400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615272d01e0 0x5615272d6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:55118/0 (socket says 172.21.15.31:55118) 2024-03-21T10:39:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.045+0000 7f5f07400640 1 -- 172.21.15.31:0/2818265405 learned_addr learned my addr 172.21.15.31:0/2818265405 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.045+0000 7f5f06a00640 1 --2- 172.21.15.31:0/2818265405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615272d0b00 0x5615272d6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.045+0000 7f5f07e00640 1 --2- 172.21.15.31:0/2818265405 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615272d98d0 0x5615272d73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.046+0000 7f5f06a00640 1 -- 172.21.15.31:0/2818265405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615272d01e0 msgr2=0x5615272d6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.046+0000 7f5f06a00640 1 --2- 172.21.15.31:0/2818265405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615272d01e0 0x5615272d6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.046+0000 7f5f06a00640 1 -- 172.21.15.31:0/2818265405 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615272d98d0 msgr2=0x5615272d73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.046+0000 7f5f06a00640 1 --2- 172.21.15.31:0/2818265405 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615272d98d0 0x5615272d73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.046+0000 7f5f06a00640 1 -- 172.21.15.31:0/2818265405 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561526f49210 con 0x5615272d0b00 2024-03-21T10:39:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.046+0000 7f5f06a00640 1 --2- 172.21.15.31:0/2818265405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615272d0b00 0x5615272d6eb0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5ef40ef060 tx=0x7f5ef40b9f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3284c00308a1c0fc server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.046+0000 7f5ef1600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:39:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.046+0000 7f5ef1600640 1 -- 172.21.15.31:0/2818265405 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef40f3040 con 0x5615272d0b00 2024-03-21T10:39:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.046+0000 7f5ef1600640 1 -- 172.21.15.31:0/2818265405 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f5ef40c61b0 con 0x5615272d0b00 2024-03-21T10:39:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.046+0000 7f5f1158d740 1 -- 172.21.15.31:0/2818265405 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5f000158f0 con 0x5615272d0b00 2024-03-21T10:39:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.046+0000 7f5ef1600640 1 -- 172.21.15.31:0/2818265405 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ef40db050 con 0x5615272d0b00 2024-03-21T10:39:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.046+0000 7f5f1158d740 10 client.25823 fetch_fsmap learned FSMap version 24 2024-03-21T10:39:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.046+0000 7f5f1158d740 1 -- 172.21.15.31:0/2818265405 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x561527015620 con 0x5615272d0b00 2024-03-21T10:39:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.047+0000 7f5ef1600640 1 -- 172.21.15.31:0/2818265405 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f000158f0 con 0x5615272d0b00 2024-03-21T10:39:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.047+0000 7f5ef1600640 1 -- 172.21.15.31:0/2818265405 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f5ef40ef230 con 0x5615272d0b00 2024-03-21T10:39:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.047+0000 7f5f1158d740 10 client.25823 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:39:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.047+0000 7f5f1158d740 10 client.25823 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:39:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.047+0000 7f5f1158d740 20 client.25823 populate_metadata read hostname 'smithi031' 2024-03-21T10:39:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.047+0000 7f5f1158d740 1 --2- 172.21.15.31:0/2818265405 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5615272e98d0 0x5615272ebce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.047+0000 7f5f1158d740 4 client.25823 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:39:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.047+0000 7f5f1158d740 1 -- 172.21.15.31:0/2818265405 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561527241d70 con 0x5615272e98d0 2024-03-21T10:39:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.047+0000 7f5f07400640 1 --2- 172.21.15.31:0/2818265405 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5615272e98d0 0x5615272ebce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.048+0000 7f5f07400640 1 --2- 172.21.15.31:0/2818265405 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5615272e98d0 0x5615272ebce0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f2d1bdcb84d9d85 server_cookie=b286c87a938f6739 in_seq=0 out_seq=0 2024-03-21T10:39:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.048+0000 7f5ef1600640 10 client.25823 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.048+0000 7f5ef1600640 1 -- 172.21.15.31:0/2818265405 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561527241d70 con 0x5615272e98d0 2024-03-21T10:39:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.048+0000 7f5ef1600640 10 client.25823 handle_command_reply: tid=0 2024-03-21T10:39:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.049+0000 7f5f1158d740 1 client.25823 shutdown 2024-03-21T10:39:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.049+0000 7f5f1158d740 1 -- 172.21.15.31:0/2818265405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615272d0b00 msgr2=0x5615272d6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.049+0000 7f5f1158d740 1 --2- 172.21.15.31:0/2818265405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615272d0b00 0x5615272d6eb0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5ef40ef060 tx=0x7f5ef40b9f90 comp rx=0 tx=0).stop 2024-03-21T10:39:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.049+0000 7f5f1158d740 1 -- 172.21.15.31:0/2818265405 shutdown_connections 2024-03-21T10:39:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.049+0000 7f5f1158d740 1 --2- 172.21.15.31:0/2818265405 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5615272e98d0 0x5615272ebce0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.049+0000 7f5f1158d740 1 --2- 172.21.15.31:0/2818265405 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615272d98d0 0x5615272d73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.049+0000 7f5f1158d740 1 --2- 172.21.15.31:0/2818265405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615272d0b00 0x5615272d6eb0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.049+0000 7f5f1158d740 1 --2- 172.21.15.31:0/2818265405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615272d01e0 0x5615272d6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.049+0000 7f5ef1600640 0 client.25823 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.049+0000 7f5f1158d740 1 -- 172.21.15.31:0/2818265405 >> 172.21.15.31:0/2818265405 conn(0x5615272cab00 msgr2=0x5615272cc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.049+0000 7f5f1158d740 1 -- 172.21.15.31:0/2818265405 shutdown_connections 2024-03-21T10:39:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.049+0000 7f5f1158d740 1 -- 172.21.15.31:0/2818265405 wait complete. 2024-03-21T10:39:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.049+0000 7f5f1158d740 20 client.25823 trim_cache size 0 max 16384 2024-03-21T10:39:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.065+0000 7f5f1158d740 1 Processor -- start 2024-03-21T10:39:16.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.066+0000 7f5f1158d740 1 -- start start 2024-03-21T10:39:16.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.066+0000 7f5f1158d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615272d01e0 0x5615272d90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:16.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.066+0000 7f5f1158d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615272d0b00 0x5615272cc510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:16.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.066+0000 7f5f1158d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615272d98d0 0x5615272cca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:16.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.066+0000 7f5f1158d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f081b2ce0 con 0x5615272d0b00 2024-03-21T10:39:16.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.066+0000 7f5f1158d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f081b07e0 con 0x5615272d01e0 2024-03-21T10:39:16.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.066+0000 7f5f1158d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f081ae880 con 0x5615272d98d0 2024-03-21T10:39:16.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.066+0000 7f5f07400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615272d01e0 0x5615272d90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:16.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.066+0000 7f5f07400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615272d01e0 0x5615272d90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49278/0 (socket says 172.21.15.31:49278) 2024-03-21T10:39:16.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.066+0000 7f5f07400640 1 -- 172.21.15.31:0/4004907031 learned_addr learned my addr 172.21.15.31:0/4004907031 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:16.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.066+0000 7f5f07e00640 1 --2- 172.21.15.31:0/4004907031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615272d98d0 0x5615272cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:16.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.066+0000 7f5f06a00640 1 --2- 172.21.15.31:0/4004907031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615272d0b00 0x5615272cc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.067+0000 7f5f07400640 1 -- 172.21.15.31:0/4004907031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615272d98d0 msgr2=0x5615272cca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.067+0000 7f5f07400640 1 --2- 172.21.15.31:0/4004907031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615272d98d0 0x5615272cca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.067+0000 7f5f07400640 1 -- 172.21.15.31:0/4004907031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615272d0b00 msgr2=0x5615272cc510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.067+0000 7f5f07400640 1 --2- 172.21.15.31:0/4004907031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615272d0b00 0x5615272cc510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.067+0000 7f5f07400640 1 -- 172.21.15.31:0/4004907031 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5615272ec360 con 0x5615272d01e0 2024-03-21T10:39:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.067+0000 7f5f07400640 1 --2- 172.21.15.31:0/4004907031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615272d01e0 0x5615272d90a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8009b80 tx=0x7f5ef801b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=288b7e739a0854c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.067+0000 7f5ef3e00640 1 -- 172.21.15.31:0/4004907031 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef802f020 con 0x5615272d01e0 2024-03-21T10:39:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.067+0000 7f5ef3e00640 1 -- 172.21.15.31:0/4004907031 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ef8004470 con 0x5615272d01e0 2024-03-21T10:39:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.067+0000 7f5ef3e00640 1 -- 172.21.15.31:0/4004907031 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef801f430 con 0x5615272d01e0 2024-03-21T10:39:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.068+0000 7f5f1158d740 1 -- 172.21.15.31:0/4004907031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615272d01e0 msgr2=0x5615272d90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.068+0000 7f5f1158d740 1 --2- 172.21.15.31:0/4004907031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615272d01e0 0x5615272d90a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8009b80 tx=0x7f5ef801b000 comp rx=0 tx=0).stop 2024-03-21T10:39:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.068+0000 7f5f1158d740 1 -- 172.21.15.31:0/4004907031 shutdown_connections 2024-03-21T10:39:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.068+0000 7f5f1158d740 1 --2- 172.21.15.31:0/4004907031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615272d98d0 0x5615272cca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.068+0000 7f5f1158d740 1 --2- 172.21.15.31:0/4004907031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615272d0b00 0x5615272cc510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.068+0000 7f5f1158d740 1 --2- 172.21.15.31:0/4004907031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615272d01e0 0x5615272d90a0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.068+0000 7f5f1158d740 1 -- 172.21.15.31:0/4004907031 >> 172.21.15.31:0/4004907031 conn(0x5615272cab00 msgr2=0x5615272cbe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.068+0000 7f5f1158d740 1 -- 172.21.15.31:0/4004907031 shutdown_connections 2024-03-21T10:39:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.068+0000 7f5f1158d740 1 -- 172.21.15.31:0/4004907031 wait complete. 2024-03-21T10:39:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.069+0000 7f5f1158d740 1 -- start start 2024-03-21T10:39:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.069+0000 7f5f1158d740 1 Processor -- start 2024-03-21T10:39:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.070+0000 7f5f1158d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615272d01e0 0x5615272d4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:16.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.070+0000 7f5f1158d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615272d0b00 0x5615272de720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:16.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.070+0000 7f5f1158d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615272d98d0 0x5615272dec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:16.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.070+0000 7f5f1158d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f081aee80 con 0x5615272d0b00 2024-03-21T10:39:16.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.070+0000 7f5f1158d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f081b0180 con 0x5615272d01e0 2024-03-21T10:39:16.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.070+0000 7f5f1158d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f081af4c0 con 0x5615272d98d0 2024-03-21T10:39:16.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.070+0000 7f5f1158d740 10 client.0 _ll_register_callbacks cb 0x7f5ef4009260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:39:16.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.070+0000 7f5f07e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615272d98d0 0x5615272dec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:16.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.070+0000 7f5f07400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615272d01e0 0x5615272d4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:16.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.070+0000 7f5f07400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615272d01e0 0x5615272d4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49286/0 (socket says 172.21.15.31:49286) 2024-03-21T10:39:16.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.070+0000 7f5f07400640 1 -- 172.21.15.31:0/3215830220 learned_addr learned my addr 172.21.15.31:0/3215830220 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:16.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.070+0000 7f5f06a00640 1 --2- 172.21.15.31:0/3215830220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615272d0b00 0x5615272de720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:16.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.070+0000 7f5f07e00640 1 --2- 172.21.15.31:0/3215830220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615272d98d0 0x5615272dec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:55136/0 (socket says 172.21.15.31:55136) 2024-03-21T10:39:16.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.070+0000 7f5f07400640 1 -- 172.21.15.31:0/3215830220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615272d98d0 msgr2=0x5615272dec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.070+0000 7f5f07400640 1 --2- 172.21.15.31:0/3215830220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615272d98d0 0x5615272dec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.070+0000 7f5f07400640 1 -- 172.21.15.31:0/3215830220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615272d0b00 msgr2=0x5615272de720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.070+0000 7f5f07400640 1 --2- 172.21.15.31:0/3215830220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615272d0b00 0x5615272de720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.071+0000 7f5f07400640 1 -- 172.21.15.31:0/3215830220 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5615272df440 con 0x5615272d01e0 2024-03-21T10:39:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.071+0000 7f5f06a00640 1 --2- 172.21.15.31:0/3215830220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615272d0b00 0x5615272de720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:39:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.071+0000 7f5f07e00640 1 --2- 172.21.15.31:0/3215830220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615272d98d0 0x5615272dec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:39:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.071+0000 7f5f07400640 1 --2- 172.21.15.31:0/3215830220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615272d01e0 0x5615272d4660 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8019040 tx=0x7f5ef8004ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3445af82c4f57dc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.071+0000 7f5ef1600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:39:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.071+0000 7f5ef1600640 1 -- 172.21.15.31:0/3215830220 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef8038070 con 0x5615272d01e0 2024-03-21T10:39:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.071+0000 7f5ef1600640 1 -- 172.21.15.31:0/3215830220 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f5ef8004980 con 0x5615272d01e0 2024-03-21T10:39:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.071+0000 7f5f1158d740 1 -- 172.21.15.31:0/3215830220 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5ef8004470 con 0x5615272d01e0 2024-03-21T10:39:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.071+0000 7f5ef1600640 1 -- 172.21.15.31:0/3215830220 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ef8044050 con 0x5615272d01e0 2024-03-21T10:39:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.071+0000 7f5f1158d740 10 client.25829 fetch_fsmap learned FSMap version 24 2024-03-21T10:39:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.072+0000 7f5f1158d740 1 -- 172.21.15.31:0/3215830220 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x561527346180 con 0x5615272d01e0 2024-03-21T10:39:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.072+0000 7f5ef1600640 1 -- 172.21.15.31:0/3215830220 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ef8004470 con 0x5615272d01e0 2024-03-21T10:39:16.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.072+0000 7f5ef1600640 1 -- 172.21.15.31:0/3215830220 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f5ef8005280 con 0x5615272d01e0 2024-03-21T10:39:16.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.072+0000 7f5f1158d740 10 client.25829 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:39:16.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.072+0000 7f5f1158d740 10 client.25829 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:39:16.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.072+0000 7f5f1158d740 20 client.25829 populate_metadata read hostname 'smithi031' 2024-03-21T10:39:16.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.072+0000 7f5f1158d740 1 --2- 172.21.15.31:0/3215830220 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561527346820 0x561527348c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:16.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.072+0000 7f5f1158d740 4 client.25829 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:39:16.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.072+0000 7f5f1158d740 1 -- 172.21.15.31:0/3215830220 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56152723c0c0 con 0x561527346820 2024-03-21T10:39:16.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.072+0000 7f5f06a00640 1 --2- 172.21.15.31:0/3215830220 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561527346820 0x561527348c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:16.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.073+0000 7f5f06a00640 1 --2- 172.21.15.31:0/3215830220 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561527346820 0x561527348c40 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88563e678adf57cf server_cookie=5b17afc8f3ce0f23 in_seq=0 out_seq=0 2024-03-21T10:39:16.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.073+0000 7f5ef1600640 10 client.25829 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.080+0000 7f5ef1600640 1 -- 172.21.15.31:0/3215830220 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56152723c0c0 con 0x561527346820 2024-03-21T10:39:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.080+0000 7f5ef1600640 10 client.25829 handle_command_reply: tid=0 2024-03-21T10:39:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.080+0000 7f5f1158d740 1 client.25829 shutdown 2024-03-21T10:39:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.080+0000 7f5f1158d740 1 -- 172.21.15.31:0/3215830220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615272d01e0 msgr2=0x5615272d4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.080+0000 7f5f1158d740 1 --2- 172.21.15.31:0/3215830220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615272d01e0 0x5615272d4660 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8019040 tx=0x7f5ef8004ce0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.080+0000 7f5f1158d740 1 -- 172.21.15.31:0/3215830220 shutdown_connections 2024-03-21T10:39:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.080+0000 7f5f1158d740 1 --2- 172.21.15.31:0/3215830220 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561527346820 0x561527348c40 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.080+0000 7f5f1158d740 1 --2- 172.21.15.31:0/3215830220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615272d98d0 0x5615272dec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.080+0000 7f5f1158d740 1 --2- 172.21.15.31:0/3215830220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615272d0b00 0x5615272de720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.080+0000 7f5ef1600640 0 client.25829 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.080+0000 7f5f1158d740 1 --2- 172.21.15.31:0/3215830220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615272d01e0 0x5615272d4660 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.080+0000 7f5f1158d740 1 -- 172.21.15.31:0/3215830220 >> 172.21.15.31:0/3215830220 conn(0x5615272cab00 msgr2=0x5615272cc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.080+0000 7f5f1158d740 1 -- 172.21.15.31:0/3215830220 shutdown_connections 2024-03-21T10:39:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.080+0000 7f5f1158d740 1 -- 172.21.15.31:0/3215830220 wait complete. 2024-03-21T10:39:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.081+0000 7f5f1158d740 20 client.25829 trim_cache size 0 max 16384 2024-03-21T10:39:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:39:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:39:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:39:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:39:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.081+0000 7f5f0d200640 1 -- 172.21.15.31:0/4173309870 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f5eec0a3210 msgr2=0x7f5eec0a56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.081+0000 7f5f0d200640 1 --2- 172.21.15.31:0/4173309870 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f5eec0a3210 0x7f5eec0a56d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x5615272cfc90 tx=0x7f5ef8005f90 comp rx=0 tx=0).stop 2024-03-21T10:39:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.081+0000 7f5f0d200640 1 -- 172.21.15.31:0/4173309870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f08101e60 msgr2=0x7f5f0819a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.081+0000 7f5f0d200640 1 --2- 172.21.15.31:0/4173309870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f08101e60 0x7f5f0819a480 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4005ab0 tx=0x7f5ef40042e0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.081+0000 7f5f0d200640 1 -- 172.21.15.31:0/4173309870 shutdown_connections 2024-03-21T10:39:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.081+0000 7f5f0d200640 1 --2- 172.21.15.31:0/4173309870 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f5eec0a3210 0x7f5eec0a56d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.081+0000 7f5f0d200640 1 --2- 172.21.15.31:0/4173309870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f08107560 0x7f5f0819e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.081+0000 7f5f0d200640 1 --2- 172.21.15.31:0/4173309870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f08101e60 0x7f5f0819a480 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.081+0000 7f5f0d200640 1 --2- 172.21.15.31:0/4173309870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f081014a0 0x7f5f08199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.081+0000 7f5f0d200640 1 -- 172.21.15.31:0/4173309870 >> 172.21.15.31:0/4173309870 conn(0x7f5f080fcfb0 msgr2=0x7f5f08074170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.082+0000 7f5f0d200640 1 -- 172.21.15.31:0/4173309870 shutdown_connections 2024-03-21T10:39:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:16.082+0000 7f5f0d200640 1 -- 172.21.15.31:0/4173309870 wait complete. 2024-03-21T10:39:16.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:16 smithi040 ceph-mon[36154]: pgmap v24: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-03-21T10:39:17.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:16 smithi031 ceph-mon[31673]: pgmap v24: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-03-21T10:39:17.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:16 smithi070 ceph-mon[36330]: pgmap v24: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-03-21T10:39:18.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:18 smithi040 ceph-mon[36154]: pgmap v25: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-21T10:39:19.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:18 smithi031 ceph-mon[31673]: pgmap v25: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-21T10:39:19.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:18 smithi070 ceph-mon[36330]: pgmap v25: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-21T10:39:20.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:20 smithi070 ceph-mon[36330]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-03-21T10:39:20.930 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:20 smithi040 ceph-mon[36154]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-03-21T10:39:21.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:20 smithi031 ceph-mon[31673]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-03-21T10:39:22.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:22 smithi040 ceph-mon[36154]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-03-21T10:39:23.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:22 smithi031 ceph-mon[31673]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-03-21T10:39:23.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:22 smithi070 ceph-mon[36330]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-03-21T10:39:24.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:24 smithi040 ceph-mon[36154]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-03-21T10:39:25.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:24 smithi031 ceph-mon[31673]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-03-21T10:39:25.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:24 smithi070 ceph-mon[36330]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-03-21T10:39:26.834 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:39:26.834 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:39:26.916 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:26 smithi031 ceph-mon[31673]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.5 KiB/s wr, 9 op/s 2024-03-21T10:39:26.930 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:26 smithi040 ceph-mon[36154]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.5 KiB/s wr, 9 op/s 2024-03-21T10:39:27.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.052+0000 7f4d8f200640 1 -- 172.21.15.31:0/2764962032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d90107aa0 msgr2=0x7f4d90107f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:27.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.052+0000 7f4d8f200640 1 --2- 172.21.15.31:0/2764962032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d90107aa0 0x7f4d90107f00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4d78009900 tx=0x7f4d7802bcd0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.052+0000 7f4d8f200640 1 -- 172.21.15.31:0/2764962032 shutdown_connections 2024-03-21T10:39:27.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.052+0000 7f4d8f200640 1 --2- 172.21.15.31:0/2764962032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d90108440 0x7f4d90114900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.052+0000 7f4d8f200640 1 --2- 172.21.15.31:0/2764962032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d90107aa0 0x7f4d90107f00 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.052+0000 7f4d8f200640 1 --2- 172.21.15.31:0/2764962032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d9010c980 0x7f4d9010cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.052+0000 7f4d8f200640 1 -- 172.21.15.31:0/2764962032 >> 172.21.15.31:0/2764962032 conn(0x7f4d900772a0 msgr2=0x7f4d900796c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:27.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.052+0000 7f4d8f200640 1 -- 172.21.15.31:0/2764962032 shutdown_connections 2024-03-21T10:39:27.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.052+0000 7f4d8f200640 1 -- 172.21.15.31:0/2764962032 wait complete. 2024-03-21T10:39:27.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.053+0000 7f4d8f200640 1 Processor -- start 2024-03-21T10:39:27.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.053+0000 7f4d8f200640 1 -- start start 2024-03-21T10:39:27.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.054+0000 7f4d8f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d90107aa0 0x7f4d9019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.054+0000 7f4d8f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d90108440 0x7f4d9019e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.054+0000 7f4d8f200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d9010c980 0x7f4d901a2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.054+0000 7f4d8f200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d90117ec0 con 0x7f4d90107aa0 2024-03-21T10:39:27.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.054+0000 7f4d8f200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d90117d40 con 0x7f4d90108440 2024-03-21T10:39:27.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.054+0000 7f4d8f200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d90118040 con 0x7f4d9010c980 2024-03-21T10:39:27.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.054+0000 7f4d85400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d90108440 0x7f4d9019e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.054+0000 7f4d8e800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d9010c980 0x7f4d901a2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.054+0000 7f4d85400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d90108440 0x7f4d9019e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49268/0 (socket says 172.21.15.31:49268) 2024-03-21T10:39:27.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.054+0000 7f4d85400640 1 -- 172.21.15.31:0/4036059170 learned_addr learned my addr 172.21.15.31:0/4036059170 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:27.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.054+0000 7f4d8e800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d9010c980 0x7f4d901a2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:46994/0 (socket says 172.21.15.31:46994) 2024-03-21T10:39:27.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.054+0000 7f4d8de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d90107aa0 0x7f4d9019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.054+0000 7f4d85400640 1 -- 172.21.15.31:0/4036059170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d9010c980 msgr2=0x7f4d901a2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:27.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.055+0000 7f4d85400640 1 --2- 172.21.15.31:0/4036059170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d9010c980 0x7f4d901a2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.055+0000 7f4d85400640 1 -- 172.21.15.31:0/4036059170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d90107aa0 msgr2=0x7f4d9019e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:27.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.055+0000 7f4d85400640 1 --2- 172.21.15.31:0/4036059170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d90107aa0 0x7f4d9019e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.055+0000 7f4d85400640 1 -- 172.21.15.31:0/4036059170 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d901a3380 con 0x7f4d90108440 2024-03-21T10:39:27.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.055+0000 7f4d8e800640 1 --2- 172.21.15.31:0/4036059170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d9010c980 0x7f4d901a2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:39:27.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.055+0000 7f4d85400640 1 --2- 172.21.15.31:0/4036059170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d90108440 0x7f4d9019e920 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f4d78000c00 tx=0x7f4d78004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:27.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.055+0000 7f4d87400640 1 -- 172.21.15.31:0/4036059170 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d78042020 con 0x7f4d90108440 2024-03-21T10:39:27.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.055+0000 7f4d87400640 1 -- 172.21.15.31:0/4036059170 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d780054e0 con 0x7f4d90108440 2024-03-21T10:39:27.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.055+0000 7f4d8f200640 1 -- 172.21.15.31:0/4036059170 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d901a3610 con 0x7f4d90108440 2024-03-21T10:39:27.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.055+0000 7f4d87400640 1 -- 172.21.15.31:0/4036059170 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d78003af0 con 0x7f4d90108440 2024-03-21T10:39:27.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.055+0000 7f4d8f200640 1 -- 172.21.15.31:0/4036059170 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d901aaf00 con 0x7f4d90108440 2024-03-21T10:39:27.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.058+0000 7f4d87400640 1 -- 172.21.15.31:0/4036059170 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) v1 ==== 138087+0+0 (secure 0 0 0) 0x7f4d78045020 con 0x7f4d90108440 2024-03-21T10:39:27.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.059+0000 7f4d977b3740 1 Processor -- start 2024-03-21T10:39:27.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.059+0000 7f4d977b3740 1 -- start start 2024-03-21T10:39:27.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.059+0000 7f4d977b3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4dec9cdd0 0x55a4dec9bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.059+0000 7f4d977b3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4dec9c0f0 0x55a4dec9c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.059+0000 7f4d977b3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4deca5460 0x55a4deca7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.059+0000 7f4d977b3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4dec00250 con 0x55a4dec9c0f0 2024-03-21T10:39:27.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.059+0000 7f4d977b3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4deba8a30 con 0x55a4dec9cdd0 2024-03-21T10:39:27.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.059+0000 7f4d977b3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4debfc2f0 con 0x55a4deca5460 2024-03-21T10:39:27.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.060+0000 7f4d8e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4dec9cdd0 0x55a4dec9bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.060+0000 7f4d85400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4deca5460 0x55a4deca7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.060+0000 7f4d8e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4dec9cdd0 0x55a4dec9bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49280/0 (socket says 172.21.15.31:49280) 2024-03-21T10:39:27.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.060+0000 7f4d8e800640 1 -- 172.21.15.31:0/3631344474 learned_addr learned my addr 172.21.15.31:0/3631344474 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:27.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.060+0000 7f4d8de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4dec9c0f0 0x55a4dec9c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.060+0000 7f4d8e800640 1 -- 172.21.15.31:0/3631344474 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4deca5460 msgr2=0x55a4deca7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:27.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.060+0000 7f4d8e800640 1 --2- 172.21.15.31:0/3631344474 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4deca5460 0x55a4deca7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.060+0000 7f4d8e800640 1 -- 172.21.15.31:0/3631344474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4dec9c0f0 msgr2=0x55a4dec9c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:27.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.060+0000 7f4d8e800640 1 --2- 172.21.15.31:0/3631344474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4dec9c0f0 0x55a4dec9c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.060+0000 7f4d8e800640 1 -- 172.21.15.31:0/3631344474 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a4deb93690 con 0x55a4dec9cdd0 2024-03-21T10:39:27.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.060+0000 7f4d8de00640 1 --2- 172.21.15.31:0/3631344474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4dec9c0f0 0x55a4dec9c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:39:27.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.060+0000 7f4d87400640 1 --2- 172.21.15.31:0/4036059170 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4d740a3230 0x7f4d740a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.060+0000 7f4d87400640 1 -- 172.21.15.31:0/4036059170 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f4d780f1870 con 0x7f4d90108440 2024-03-21T10:39:27.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.061+0000 7f4d8de00640 1 --2- 172.21.15.31:0/4036059170 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4d740a3230 0x7f4d740a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.061+0000 7f4d8e800640 1 --2- 172.21.15.31:0/3631344474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4dec9cdd0 0x55a4dec9bb20 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4d880117e0 tx=0x7f4d88011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a01d34803a515e6d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:27.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.061+0000 7f4d7f000640 1 -- 172.21.15.31:0/3631344474 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d88011310 con 0x55a4dec9cdd0 2024-03-21T10:39:27.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.061+0000 7f4d7f000640 1 -- 172.21.15.31:0/3631344474 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d88003e50 con 0x55a4dec9cdd0 2024-03-21T10:39:27.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.061+0000 7f4d7f000640 1 -- 172.21.15.31:0/3631344474 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d88004620 con 0x55a4dec9cdd0 2024-03-21T10:39:27.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.062+0000 7f4d977b3740 1 -- 172.21.15.31:0/3631344474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4dec9cdd0 msgr2=0x55a4dec9bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:27.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.062+0000 7f4d8de00640 1 --2- 172.21.15.31:0/4036059170 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4d740a3230 0x7f4d740a56f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x55a4dec9d320 tx=0x7f4d80005f90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:27.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.062+0000 7f4d977b3740 1 --2- 172.21.15.31:0/3631344474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4dec9cdd0 0x55a4dec9bb20 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4d880117e0 tx=0x7f4d88011ca0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.062+0000 7f4d977b3740 1 -- 172.21.15.31:0/3631344474 shutdown_connections 2024-03-21T10:39:27.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.062+0000 7f4d977b3740 1 --2- 172.21.15.31:0/3631344474 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4deca5460 0x55a4deca7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.062+0000 7f4d977b3740 1 --2- 172.21.15.31:0/3631344474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4dec9c0f0 0x55a4dec9c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.062+0000 7f4d977b3740 1 --2- 172.21.15.31:0/3631344474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4dec9cdd0 0x55a4dec9bb20 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.062+0000 7f4d977b3740 1 -- 172.21.15.31:0/3631344474 >> 172.21.15.31:0/3631344474 conn(0x55a4dec63200 msgr2=0x55a4dec96b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:27.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.062+0000 7f4d977b3740 1 -- 172.21.15.31:0/3631344474 shutdown_connections 2024-03-21T10:39:27.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.062+0000 7f4d977b3740 1 -- 172.21.15.31:0/3631344474 wait complete. 2024-03-21T10:39:27.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.063+0000 7f4d977b3740 1 -- start start 2024-03-21T10:39:27.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.063+0000 7f4d977b3740 1 Processor -- start 2024-03-21T10:39:27.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.064+0000 7f4d977b3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4dec9c0f0 0x55a4deca4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.064+0000 7f4d977b3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4dec9cdd0 0x55a4deca1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.064+0000 7f4d977b3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4deca5460 0x55a4deca23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.064+0000 7f4d977b3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4deca82d0 con 0x55a4deca5460 2024-03-21T10:39:27.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.064+0000 7f4d977b3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4debbbe80 con 0x55a4dec9cdd0 2024-03-21T10:39:27.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.064+0000 7f4d977b3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4deca8450 con 0x55a4dec9c0f0 2024-03-21T10:39:27.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.064+0000 7f4d977b3740 10 client.0 _ll_register_callbacks cb 0x55a4debce130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:39:27.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.064+0000 7f4d85400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4dec9cdd0 0x55a4deca1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.064+0000 7f4d8de00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4dec9c0f0 0x55a4deca4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.064+0000 7f4d85400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4dec9cdd0 0x55a4deca1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49288/0 (socket says 172.21.15.31:49288) 2024-03-21T10:39:27.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.064+0000 7f4d8de00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4dec9c0f0 0x55a4deca4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:47004/0 (socket says 172.21.15.31:47004) 2024-03-21T10:39:27.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.064+0000 7f4d85400640 1 -- 172.21.15.31:0/2123187471 learned_addr learned my addr 172.21.15.31:0/2123187471 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:27.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.064+0000 7f4d8e800640 1 --2- 172.21.15.31:0/2123187471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4deca5460 0x55a4deca23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.065+0000 7f4d85400640 1 -- 172.21.15.31:0/2123187471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4dec9c0f0 msgr2=0x55a4deca4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:27.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.065+0000 7f4d85400640 1 --2- 172.21.15.31:0/2123187471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4dec9c0f0 0x55a4deca4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.065+0000 7f4d85400640 1 -- 172.21.15.31:0/2123187471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4deca5460 msgr2=0x55a4deca23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:27.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.065+0000 7f4d85400640 1 --2- 172.21.15.31:0/2123187471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4deca5460 0x55a4deca23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.065+0000 7f4d85400640 1 -- 172.21.15.31:0/2123187471 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a4de9de7d0 con 0x55a4dec9cdd0 2024-03-21T10:39:27.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.065+0000 7f4d85400640 1 --2- 172.21.15.31:0/2123187471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4dec9cdd0 0x55a4deca1eb0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4d780bd480 tx=0x7f4d78005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=522c91f18f921f32 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:27.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.065+0000 7f4d73400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:39:27.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.065+0000 7f4d73400640 1 -- 172.21.15.31:0/2123187471 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d780bd550 con 0x55a4dec9cdd0 2024-03-21T10:39:27.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.065+0000 7f4d73400640 1 -- 172.21.15.31:0/2123187471 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f4d7802cc30 con 0x55a4dec9cdd0 2024-03-21T10:39:27.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.065+0000 7f4d977b3740 1 -- 172.21.15.31:0/2123187471 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4d88003e50 con 0x55a4dec9cdd0 2024-03-21T10:39:27.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.065+0000 7f4d73400640 1 -- 172.21.15.31:0/2123187471 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d780bd0e0 con 0x55a4dec9cdd0 2024-03-21T10:39:27.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.065+0000 7f4d977b3740 10 client.25841 fetch_fsmap learned FSMap version 24 2024-03-21T10:39:27.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.065+0000 7f4d977b3740 1 -- 172.21.15.31:0/2123187471 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55a4de9143f0 con 0x55a4dec9cdd0 2024-03-21T10:39:27.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.066+0000 7f4d73400640 1 -- 172.21.15.31:0/2123187471 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d88003e50 con 0x55a4dec9cdd0 2024-03-21T10:39:27.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.066+0000 7f4d73400640 1 -- 172.21.15.31:0/2123187471 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f4d780d6030 con 0x55a4dec9cdd0 2024-03-21T10:39:27.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.066+0000 7f4d977b3740 10 client.25841 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:39:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.066+0000 7f4d977b3740 10 client.25841 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:39:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.066+0000 7f4d977b3740 20 client.25841 populate_metadata read hostname 'smithi031' 2024-03-21T10:39:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.066+0000 7f4d977b3740 1 --2- 172.21.15.31:0/2123187471 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a4decb52c0 0x55a4decb76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.066+0000 7f4d977b3740 4 client.25841 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:39:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.066+0000 7f4d977b3740 1 -- 172.21.15.31:0/2123187471 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a4dec0d790 con 0x55a4decb52c0 2024-03-21T10:39:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.066+0000 7f4d8de00640 1 --2- 172.21.15.31:0/2123187471 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a4decb52c0 0x55a4decb76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.067+0000 7f4d8de00640 1 --2- 172.21.15.31:0/2123187471 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a4decb52c0 0x55a4decb76b0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e62a2a921bfcc26e server_cookie=7021e1c7c0565714 in_seq=0 out_seq=0 2024-03-21T10:39:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.067+0000 7f4d73400640 10 client.25841 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.067+0000 7f4d73400640 1 -- 172.21.15.31:0/2123187471 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a4dec0d790 con 0x55a4decb52c0 2024-03-21T10:39:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.067+0000 7f4d73400640 10 client.25841 handle_command_reply: tid=0 2024-03-21T10:39:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.067+0000 7f4d977b3740 1 client.25841 shutdown 2024-03-21T10:39:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.068+0000 7f4d977b3740 1 -- 172.21.15.31:0/2123187471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4dec9cdd0 msgr2=0x55a4deca1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.068+0000 7f4d977b3740 1 --2- 172.21.15.31:0/2123187471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4dec9cdd0 0x55a4deca1eb0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4d780bd480 tx=0x7f4d78005d50 comp rx=0 tx=0).stop 2024-03-21T10:39:27.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.068+0000 7f4d977b3740 1 -- 172.21.15.31:0/2123187471 shutdown_connections 2024-03-21T10:39:27.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.068+0000 7f4d977b3740 1 --2- 172.21.15.31:0/2123187471 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a4decb52c0 0x55a4decb76b0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.068+0000 7f4d977b3740 1 --2- 172.21.15.31:0/2123187471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4deca5460 0x55a4deca23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.068+0000 7f4d977b3740 1 --2- 172.21.15.31:0/2123187471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4dec9cdd0 0x55a4deca1eb0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.068+0000 7f4d977b3740 1 --2- 172.21.15.31:0/2123187471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4dec9c0f0 0x55a4deca4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.068+0000 7f4d73400640 0 client.25841 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:27.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.068+0000 7f4d977b3740 1 -- 172.21.15.31:0/2123187471 >> 172.21.15.31:0/2123187471 conn(0x55a4dec63200 msgr2=0x55a4dec98890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:27.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.068+0000 7f4d977b3740 1 -- 172.21.15.31:0/2123187471 shutdown_connections 2024-03-21T10:39:27.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.068+0000 7f4d977b3740 1 -- 172.21.15.31:0/2123187471 wait complete. 2024-03-21T10:39:27.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.068+0000 7f4d977b3740 20 client.25841 trim_cache size 0 max 16384 2024-03-21T10:39:27.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:26 smithi070 ceph-mon[36330]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.5 KiB/s wr, 9 op/s 2024-03-21T10:39:27.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.087+0000 7f4d977b3740 1 Processor -- start 2024-03-21T10:39:27.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.088+0000 7f4d977b3740 1 -- start start 2024-03-21T10:39:27.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.088+0000 7f4d977b3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4dec9c0f0 0x55a4deca45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.088+0000 7f4d977b3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4dec9cdd0 0x55a4deca4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.088+0000 7f4d977b3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4deca5460 0x55a4deca6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.088+0000 7f4d977b3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d901b7220 con 0x55a4dec9c0f0 2024-03-21T10:39:27.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.088+0000 7f4d977b3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d901b4d20 con 0x55a4dec9cdd0 2024-03-21T10:39:27.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.088+0000 7f4d977b3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d901b2dc0 con 0x55a4deca5460 2024-03-21T10:39:27.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.088+0000 7f4d85400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4dec9cdd0 0x55a4deca4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.088+0000 7f4d8e800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4deca5460 0x55a4deca6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.088+0000 7f4d8de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4dec9c0f0 0x55a4deca45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.088+0000 7f4d85400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4dec9cdd0 0x55a4deca4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49290/0 (socket says 172.21.15.31:49290) 2024-03-21T10:39:27.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.088+0000 7f4d85400640 1 -- 172.21.15.31:0/3095150129 learned_addr learned my addr 172.21.15.31:0/3095150129 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.089+0000 7f4d8e800640 1 -- 172.21.15.31:0/3095150129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4dec9cdd0 msgr2=0x55a4deca4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.089+0000 7f4d8e800640 1 --2- 172.21.15.31:0/3095150129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4dec9cdd0 0x55a4deca4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.089+0000 7f4d8e800640 1 -- 172.21.15.31:0/3095150129 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4dec9c0f0 msgr2=0x55a4deca45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.089+0000 7f4d8e800640 1 --2- 172.21.15.31:0/3095150129 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4dec9c0f0 0x55a4deca45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.089+0000 7f4d8e800640 1 -- 172.21.15.31:0/3095150129 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a4decb7a20 con 0x55a4deca5460 2024-03-21T10:39:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.089+0000 7f4d8e800640 1 --2- 172.21.15.31:0/3095150129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4deca5460 0x55a4deca6850 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4d8800a010 tx=0x7f4d88005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83b065a355021cbd server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.089+0000 7f4d7e600640 1 -- 172.21.15.31:0/3095150129 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8802a070 con 0x55a4deca5460 2024-03-21T10:39:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.089+0000 7f4d7e600640 1 -- 172.21.15.31:0/3095150129 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d88003e50 con 0x55a4deca5460 2024-03-21T10:39:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.089+0000 7f4d7e600640 1 -- 172.21.15.31:0/3095150129 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8800fea0 con 0x55a4deca5460 2024-03-21T10:39:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.090+0000 7f4d977b3740 1 -- 172.21.15.31:0/3095150129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4deca5460 msgr2=0x55a4deca6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.090+0000 7f4d977b3740 1 --2- 172.21.15.31:0/3095150129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4deca5460 0x55a4deca6850 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4d8800a010 tx=0x7f4d88005bd0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.090+0000 7f4d977b3740 1 -- 172.21.15.31:0/3095150129 shutdown_connections 2024-03-21T10:39:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.090+0000 7f4d977b3740 1 --2- 172.21.15.31:0/3095150129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4deca5460 0x55a4deca6850 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.090+0000 7f4d977b3740 1 --2- 172.21.15.31:0/3095150129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4dec9cdd0 0x55a4deca4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.090+0000 7f4d977b3740 1 --2- 172.21.15.31:0/3095150129 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4dec9c0f0 0x55a4deca45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.090+0000 7f4d977b3740 1 -- 172.21.15.31:0/3095150129 >> 172.21.15.31:0/3095150129 conn(0x55a4dec63200 msgr2=0x55a4deca07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:27.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.090+0000 7f4d977b3740 1 -- 172.21.15.31:0/3095150129 shutdown_connections 2024-03-21T10:39:27.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.090+0000 7f4d977b3740 1 -- 172.21.15.31:0/3095150129 wait complete. 2024-03-21T10:39:27.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.091+0000 7f4d977b3740 1 -- start start 2024-03-21T10:39:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.091+0000 7f4d977b3740 1 Processor -- start 2024-03-21T10:39:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.091+0000 7f4d977b3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4dec9c0f0 0x55a4dec9eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.091+0000 7f4d977b3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4dec9cdd0 0x55a4dec9f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.091+0000 7f4d977b3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4deca5460 0x55a4ded0fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.091+0000 7f4d977b3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d901b33c0 con 0x55a4dec9cdd0 2024-03-21T10:39:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.091+0000 7f4d977b3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d901b46c0 con 0x55a4deca5460 2024-03-21T10:39:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.091+0000 7f4d977b3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d901b3a00 con 0x55a4dec9c0f0 2024-03-21T10:39:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.091+0000 7f4d977b3740 10 client.0 _ll_register_callbacks cb 0x7f4d7802cb30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:39:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.092+0000 7f4d85400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4dec9cdd0 0x55a4dec9f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.092+0000 7f4d8de00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4dec9c0f0 0x55a4dec9eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.092+0000 7f4d85400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4dec9cdd0 0x55a4dec9f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33302/0 (socket says 172.21.15.31:33302) 2024-03-21T10:39:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.092+0000 7f4d85400640 1 -- 172.21.15.31:0/377541372 learned_addr learned my addr 172.21.15.31:0/377541372 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.092+0000 7f4d8e800640 1 --2- 172.21.15.31:0/377541372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4deca5460 0x55a4ded0fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.092+0000 7f4d8de00640 1 -- 172.21.15.31:0/377541372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4deca5460 msgr2=0x55a4ded0fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.092+0000 7f4d8de00640 1 --2- 172.21.15.31:0/377541372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4deca5460 0x55a4ded0fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.092+0000 7f4d8de00640 1 -- 172.21.15.31:0/377541372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4dec9cdd0 msgr2=0x55a4dec9f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.092+0000 7f4d8de00640 1 --2- 172.21.15.31:0/377541372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4dec9cdd0 0x55a4dec9f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.092+0000 7f4d8de00640 1 -- 172.21.15.31:0/377541372 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a4de8d40f0 con 0x55a4dec9c0f0 2024-03-21T10:39:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.092+0000 7f4d8de00640 1 --2- 172.21.15.31:0/377541372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4dec9c0f0 0x55a4dec9eda0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4d780bba30 tx=0x7f4d800046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f31d03de1f6183ae server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.092+0000 7f4d73400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:39:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.092+0000 7f4d73400640 1 -- 172.21.15.31:0/377541372 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d800044b0 con 0x55a4dec9c0f0 2024-03-21T10:39:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.093+0000 7f4d73400640 1 -- 172.21.15.31:0/377541372 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f4d80003ac0 con 0x55a4dec9c0f0 2024-03-21T10:39:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.093+0000 7f4d977b3740 1 -- 172.21.15.31:0/377541372 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4d88003e50 con 0x55a4dec9c0f0 2024-03-21T10:39:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.093+0000 7f4d73400640 1 -- 172.21.15.31:0/377541372 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d8003d050 con 0x55a4dec9c0f0 2024-03-21T10:39:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.093+0000 7f4d977b3740 10 client.25834 fetch_fsmap learned FSMap version 24 2024-03-21T10:39:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.093+0000 7f4d977b3740 1 -- 172.21.15.31:0/377541372 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55a4ded112b0 con 0x55a4dec9c0f0 2024-03-21T10:39:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.093+0000 7f4d73400640 1 -- 172.21.15.31:0/377541372 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d88003e50 con 0x55a4dec9c0f0 2024-03-21T10:39:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.093+0000 7f4d73400640 1 -- 172.21.15.31:0/377541372 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f4d80034490 con 0x55a4dec9c0f0 2024-03-21T10:39:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.093+0000 7f4d977b3740 10 client.25834 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:39:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.093+0000 7f4d977b3740 10 client.25834 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:39:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.093+0000 7f4d977b3740 20 client.25834 populate_metadata read hostname 'smithi031' 2024-03-21T10:39:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.093+0000 7f4d977b3740 1 --2- 172.21.15.31:0/377541372 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a4ded119e0 0x55a4ded13e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.093+0000 7f4d977b3740 4 client.25834 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:39:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.093+0000 7f4d977b3740 1 -- 172.21.15.31:0/377541372 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a4dec07ae0 con 0x55a4ded119e0 2024-03-21T10:39:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.094+0000 7f4d85400640 1 --2- 172.21.15.31:0/377541372 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a4ded119e0 0x55a4ded13e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.094+0000 7f4d73400640 10 client.25834 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.094+0000 7f4d85400640 1 --2- 172.21.15.31:0/377541372 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a4ded119e0 0x55a4ded13e00 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c70fce0ca10551ca server_cookie=fce843d3f5dcecd6 in_seq=0 out_seq=0 2024-03-21T10:39:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.094+0000 7f4d73400640 1 -- 172.21.15.31:0/377541372 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55a4dec07ae0 con 0x55a4ded119e0 2024-03-21T10:39:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.094+0000 7f4d73400640 10 client.25834 handle_command_reply: tid=0 2024-03-21T10:39:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.094+0000 7f4d977b3740 1 client.25834 shutdown 2024-03-21T10:39:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.095+0000 7f4d977b3740 1 -- 172.21.15.31:0/377541372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4dec9c0f0 msgr2=0x55a4dec9eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.095+0000 7f4d977b3740 1 --2- 172.21.15.31:0/377541372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4dec9c0f0 0x55a4dec9eda0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4d780bba30 tx=0x7f4d800046c0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.095+0000 7f4d977b3740 1 -- 172.21.15.31:0/377541372 shutdown_connections 2024-03-21T10:39:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.095+0000 7f4d977b3740 1 --2- 172.21.15.31:0/377541372 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a4ded119e0 0x55a4ded13e00 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.095+0000 7f4d977b3740 1 --2- 172.21.15.31:0/377541372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4deca5460 0x55a4ded0fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.095+0000 7f4d977b3740 1 --2- 172.21.15.31:0/377541372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4dec9cdd0 0x55a4dec9f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.095+0000 7f4d73400640 0 client.25834 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.095+0000 7f4d977b3740 1 --2- 172.21.15.31:0/377541372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4dec9c0f0 0x55a4dec9eda0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.095+0000 7f4d977b3740 1 -- 172.21.15.31:0/377541372 >> 172.21.15.31:0/377541372 conn(0x55a4dec63200 msgr2=0x55a4dec95330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.095+0000 7f4d977b3740 1 -- 172.21.15.31:0/377541372 shutdown_connections 2024-03-21T10:39:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.095+0000 7f4d977b3740 1 -- 172.21.15.31:0/377541372 wait complete. 2024-03-21T10:39:27.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.095+0000 7f4d977b3740 20 client.25834 trim_cache size 0 max 16384 2024-03-21T10:39:27.096 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:39:27.096 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-03-21T10:39:27.096 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-03-21T10:39:27.097 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:39:27.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.096+0000 7f4d8f200640 1 -- 172.21.15.31:0/4036059170 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4d740a3230 msgr2=0x7f4d740a56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:27.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.096+0000 7f4d8f200640 1 --2- 172.21.15.31:0/4036059170 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4d740a3230 0x7f4d740a56f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x55a4dec9d320 tx=0x7f4d80005f90 comp rx=0 tx=0).stop 2024-03-21T10:39:27.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.096+0000 7f4d8f200640 1 -- 172.21.15.31:0/4036059170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d90108440 msgr2=0x7f4d9019e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:27.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.096+0000 7f4d8f200640 1 --2- 172.21.15.31:0/4036059170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d90108440 0x7f4d9019e920 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f4d78000c00 tx=0x7f4d78004360 comp rx=0 tx=0).stop 2024-03-21T10:39:27.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.096+0000 7f4d8f200640 1 -- 172.21.15.31:0/4036059170 shutdown_connections 2024-03-21T10:39:27.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.096+0000 7f4d8f200640 1 --2- 172.21.15.31:0/4036059170 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4d740a3230 0x7f4d740a56f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.096+0000 7f4d8f200640 1 --2- 172.21.15.31:0/4036059170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d9010c980 0x7f4d901a2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.096+0000 7f4d8f200640 1 --2- 172.21.15.31:0/4036059170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d90108440 0x7f4d9019e920 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.096+0000 7f4d8f200640 1 --2- 172.21.15.31:0/4036059170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d90107aa0 0x7f4d9019e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.096+0000 7f4d8f200640 1 -- 172.21.15.31:0/4036059170 >> 172.21.15.31:0/4036059170 conn(0x7f4d900772a0 msgr2=0x7f4d90077ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:27.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.096+0000 7f4d8f200640 1 -- 172.21.15.31:0/4036059170 shutdown_connections 2024-03-21T10:39:27.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.096+0000 7f4d8f200640 1 -- 172.21.15.31:0/4036059170 wait complete. 2024-03-21T10:39:27.119 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-21T10:39:27.119 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:39:27.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.330+0000 7f5c35e00640 1 -- 172.21.15.31:0/1160797623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3010bb20 msgr2=0x7f5c3010dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:27.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.330+0000 7f5c35e00640 1 --2- 172.21.15.31:0/1160797623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3010bb20 0x7f5c3010dfc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5c28009ca0 tx=0x7f5c2802c180 comp rx=0 tx=0).stop 2024-03-21T10:39:27.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.330+0000 7f5c35e00640 1 -- 172.21.15.31:0/1160797623 shutdown_connections 2024-03-21T10:39:27.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.330+0000 7f5c35e00640 1 --2- 172.21.15.31:0/1160797623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3010bb20 0x7f5c3010dfc0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.330+0000 7f5c35e00640 1 --2- 172.21.15.31:0/1160797623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5c30103020 0x7f5c3010b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.330+0000 7f5c35e00640 1 --2- 172.21.15.31:0/1160797623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c30102700 0x7f5c30102ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.330+0000 7f5c35e00640 1 -- 172.21.15.31:0/1160797623 >> 172.21.15.31:0/1160797623 conn(0x7f5c300fbf00 msgr2=0x7f5c300fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:27.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.330+0000 7f5c35e00640 1 -- 172.21.15.31:0/1160797623 shutdown_connections 2024-03-21T10:39:27.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.330+0000 7f5c35e00640 1 -- 172.21.15.31:0/1160797623 wait complete. 2024-03-21T10:39:27.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.332+0000 7f5c35e00640 1 Processor -- start 2024-03-21T10:39:27.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.332+0000 7f5c35e00640 1 -- start start 2024-03-21T10:39:27.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.333+0000 7f5c35e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c30102700 0x7f5c30199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.333+0000 7f5c35e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5c30103020 0x7f5c3019a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.333+0000 7f5c35e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3010bb20 0x7f5c3019e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.333+0000 7f5c35e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c301118e0 con 0x7f5c30102700 2024-03-21T10:39:27.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.333+0000 7f5c35e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c30111760 con 0x7f5c3010bb20 2024-03-21T10:39:27.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.333+0000 7f5c35e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c30111a60 con 0x7f5c30103020 2024-03-21T10:39:27.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.333+0000 7f5c2fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5c30103020 0x7f5c3019a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.333+0000 7f5c2fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5c30103020 0x7f5c3019a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:47030/0 (socket says 172.21.15.31:47030) 2024-03-21T10:39:27.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.333+0000 7f5c34a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c30102700 0x7f5c30199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.333+0000 7f5c2fe00640 1 -- 172.21.15.31:0/755203861 learned_addr learned my addr 172.21.15.31:0/755203861 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:27.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.333+0000 7f5c35400640 1 --2- 172.21.15.31:0/755203861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3010bb20 0x7f5c3019e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.333+0000 7f5c2fe00640 1 -- 172.21.15.31:0/755203861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3010bb20 msgr2=0x7f5c3019e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:27.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.333+0000 7f5c2fe00640 1 --2- 172.21.15.31:0/755203861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3010bb20 0x7f5c3019e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.333+0000 7f5c2fe00640 1 -- 172.21.15.31:0/755203861 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c30102700 msgr2=0x7f5c30199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:27.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.333+0000 7f5c2fe00640 1 --2- 172.21.15.31:0/755203861 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c30102700 0x7f5c30199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.333+0000 7f5c2fe00640 1 -- 172.21.15.31:0/755203861 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3019efd0 con 0x7f5c30103020 2024-03-21T10:39:27.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.333+0000 7f5c35400640 1 --2- 172.21.15.31:0/755203861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3010bb20 0x7f5c3019e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:39:27.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.334+0000 7f5c2fe00640 1 --2- 172.21.15.31:0/755203861 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5c30103020 0x7f5c3019a520 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c00e9f0 tx=0x7f5c1c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:27.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.334+0000 7f5c2d600640 1 -- 172.21.15.31:0/755203861 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c00cd80 con 0x7f5c30103020 2024-03-21T10:39:27.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.334+0000 7f5c2d600640 1 -- 172.21.15.31:0/755203861 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5c1c0035f0 con 0x7f5c30103020 2024-03-21T10:39:27.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.334+0000 7f5c2d600640 1 -- 172.21.15.31:0/755203861 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c010670 con 0x7f5c30103020 2024-03-21T10:39:27.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.335+0000 7f5c35e00640 1 -- 172.21.15.31:0/755203861 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c3019f2c0 con 0x7f5c30103020 2024-03-21T10:39:27.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.335+0000 7f5c35e00640 1 -- 172.21.15.31:0/755203861 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c301a6b50 con 0x7f5c30103020 2024-03-21T10:39:27.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.336+0000 7f5c35e00640 1 -- 172.21.15.31:0/755203861 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c3004f470 con 0x7f5c30103020 2024-03-21T10:39:27.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.338+0000 7f5c2d600640 1 -- 172.21.15.31:0/755203861 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) v1 ==== 138087+0+0 (secure 0 0 0) 0x7f5c1c0037a0 con 0x7f5c30103020 2024-03-21T10:39:27.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.339+0000 7f5c2d600640 1 --2- 172.21.15.31:0/755203861 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f5c180a3220 0x7f5c180a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.339+0000 7f5c2d600640 1 -- 172.21.15.31:0/755203861 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f5c1c0cffe0 con 0x7f5c30103020 2024-03-21T10:39:27.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.340+0000 7f5c34a00640 1 --2- 172.21.15.31:0/755203861 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f5c180a3220 0x7f5c180a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.340+0000 7f5c34a00640 1 --2- 172.21.15.31:0/755203861 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f5c180a3220 0x7f5c180a56e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5c24009bd0 tx=0x7f5c24005f90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:27.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.343+0000 7f5c2d600640 1 -- 172.21.15.31:0/755203861 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5c1c09c540 con 0x7f5c30103020 2024-03-21T10:39:27.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.604+0000 7f5c35e00640 1 -- 172.21.15.31:0/755203861 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c301a6d20 con 0x7f5c30103020 2024-03-21T10:39:27.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.605+0000 7f5c2d600640 1 -- 172.21.15.31:0/755203861 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11384 (secure 0 0 0) 0x7f5c1c09be40 con 0x7f5c30103020 2024-03-21T10:39:27.602 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:39:27.603 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:38:12.319122+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24547,24596]},"id":1}]} 2024-03-21T10:39:27.603 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 24 2024-03-21T10:39:27.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.607+0000 7f5c35e00640 1 -- 172.21.15.31:0/755203861 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f5c180a3220 msgr2=0x7f5c180a56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:27.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.607+0000 7f5c35e00640 1 --2- 172.21.15.31:0/755203861 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f5c180a3220 0x7f5c180a56e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5c24009bd0 tx=0x7f5c24005f90 comp rx=0 tx=0).stop 2024-03-21T10:39:27.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.607+0000 7f5c35e00640 1 -- 172.21.15.31:0/755203861 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5c30103020 msgr2=0x7f5c3019a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:27.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.607+0000 7f5c35e00640 1 --2- 172.21.15.31:0/755203861 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5c30103020 0x7f5c3019a520 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c00e9f0 tx=0x7f5c1c00eeb0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.607+0000 7f5c35e00640 1 -- 172.21.15.31:0/755203861 shutdown_connections 2024-03-21T10:39:27.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.607+0000 7f5c35e00640 1 --2- 172.21.15.31:0/755203861 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f5c180a3220 0x7f5c180a56e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.607+0000 7f5c35e00640 1 --2- 172.21.15.31:0/755203861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3010bb20 0x7f5c3019e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.607+0000 7f5c35e00640 1 --2- 172.21.15.31:0/755203861 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5c30103020 0x7f5c3019a520 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.607+0000 7f5c35e00640 1 --2- 172.21.15.31:0/755203861 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c30102700 0x7f5c30199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.607+0000 7f5c35e00640 1 -- 172.21.15.31:0/755203861 >> 172.21.15.31:0/755203861 conn(0x7f5c300fbf00 msgr2=0x7f5c3010cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:27.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.608+0000 7f5c35e00640 1 -- 172.21.15.31:0/755203861 shutdown_connections 2024-03-21T10:39:27.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.608+0000 7f5c35e00640 1 -- 172.21.15.31:0/755203861 wait complete. 2024-03-21T10:39:27.622 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:39:27.622 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-21T10:39:27.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.856+0000 7f1942c00640 1 -- 172.21.15.31:0/2441432603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f193c10dd60 msgr2=0x7f193c110150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:27.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.856+0000 7f1942c00640 1 --2- 172.21.15.31:0/2441432603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f193c10dd60 0x7f193c110150 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1934009ca0 tx=0x7f193402c100 comp rx=0 tx=0).stop 2024-03-21T10:39:27.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.856+0000 7f1942c00640 1 -- 172.21.15.31:0/2441432603 shutdown_connections 2024-03-21T10:39:27.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.856+0000 7f1942c00640 1 --2- 172.21.15.31:0/2441432603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f193c10dd60 0x7f193c110150 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.856+0000 7f1942c00640 1 --2- 172.21.15.31:0/2441432603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f193c100b60 0x7f193c10d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.856+0000 7f1942c00640 1 --2- 172.21.15.31:0/2441432603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f193c1001b0 0x7f193c100590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.856+0000 7f1942c00640 1 -- 172.21.15.31:0/2441432603 >> 172.21.15.31:0/2441432603 conn(0x7f193c077130 msgr2=0x7f193c0fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:27.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.856+0000 7f1942c00640 1 -- 172.21.15.31:0/2441432603 shutdown_connections 2024-03-21T10:39:27.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.856+0000 7f1942c00640 1 -- 172.21.15.31:0/2441432603 wait complete. 2024-03-21T10:39:27.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.857+0000 7f1942c00640 1 Processor -- start 2024-03-21T10:39:27.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.857+0000 7f1942c00640 1 -- start start 2024-03-21T10:39:27.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.858+0000 7f1942c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f193c1001b0 0x7f193c199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.858+0000 7f1942c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f193c100b60 0x7f193c19a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.858+0000 7f1942c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f193c10dd60 0x7f193c19e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.858+0000 7f1942c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c113a50 con 0x7f193c1001b0 2024-03-21T10:39:27.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.858+0000 7f1942c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c1138d0 con 0x7f193c10dd60 2024-03-21T10:39:27.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.858+0000 7f1942c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c113bd0 con 0x7f193c100b60 2024-03-21T10:39:27.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.858+0000 7f1941800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f193c1001b0 0x7f193c199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.858+0000 7f1941800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f193c1001b0 0x7f193c199fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33348/0 (socket says 172.21.15.31:33348) 2024-03-21T10:39:27.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.858+0000 7f1941800640 1 -- 172.21.15.31:0/2830461600 learned_addr learned my addr 172.21.15.31:0/2830461600 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:27.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.858+0000 7f1942200640 1 --2- 172.21.15.31:0/2830461600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f193c10dd60 0x7f193c19e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.858+0000 7f1940e00640 1 --2- 172.21.15.31:0/2830461600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f193c100b60 0x7f193c19a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.858+0000 7f1942200640 1 -- 172.21.15.31:0/2830461600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f193c100b60 msgr2=0x7f193c19a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:27.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.858+0000 7f1942200640 1 --2- 172.21.15.31:0/2830461600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f193c100b60 0x7f193c19a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.858+0000 7f1942200640 1 -- 172.21.15.31:0/2830461600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f193c1001b0 msgr2=0x7f193c199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:27.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.858+0000 7f1942200640 1 --2- 172.21.15.31:0/2830461600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f193c1001b0 0x7f193c199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.858+0000 7f1942200640 1 -- 172.21.15.31:0/2830461600 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193c19efa0 con 0x7f193c10dd60 2024-03-21T10:39:27.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.859+0000 7f1942200640 1 --2- 172.21.15.31:0/2830461600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f193c10dd60 0x7f193c19e870 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f193402c710 tx=0x7f1934032f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:27.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.859+0000 7f193a000640 1 -- 172.21.15.31:0/2830461600 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1934039040 con 0x7f193c10dd60 2024-03-21T10:39:27.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.859+0000 7f193a000640 1 -- 172.21.15.31:0/2830461600 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1934044070 con 0x7f193c10dd60 2024-03-21T10:39:27.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.859+0000 7f1942c00640 1 -- 172.21.15.31:0/2830461600 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f193c19f230 con 0x7f193c10dd60 2024-03-21T10:39:27.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.859+0000 7f193a000640 1 -- 172.21.15.31:0/2830461600 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19340042a0 con 0x7f193c10dd60 2024-03-21T10:39:27.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.859+0000 7f1942c00640 1 -- 172.21.15.31:0/2830461600 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f193c1a6b20 con 0x7f193c10dd60 2024-03-21T10:39:27.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.863+0000 7f1946e2d740 1 Processor -- start 2024-03-21T10:39:27.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.864+0000 7f193a000640 1 -- 172.21.15.31:0/2830461600 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) v1 ==== 138087+0+0 (secure 0 0 0) 0x7f1934004440 con 0x7f193c10dd60 2024-03-21T10:39:27.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.864+0000 7f1946e2d740 1 -- start start 2024-03-21T10:39:27.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.864+0000 7f1946e2d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564bc337d8b0 0x564bc337dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.864+0000 7f1946e2d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564bc337e260 0x564bc3386810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.864+0000 7f1946e2d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bc3386de0 0x564bc3389230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.864+0000 7f1946e2d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bc32e12c0 con 0x564bc337d8b0 2024-03-21T10:39:27.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.864+0000 7f1946e2d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bc3289c90 con 0x564bc337e260 2024-03-21T10:39:27.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.864+0000 7f1946e2d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bc32dd360 con 0x564bc3386de0 2024-03-21T10:39:27.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.865+0000 7f1941800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564bc337e260 0x564bc3386810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.865+0000 7f1941800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564bc337e260 0x564bc3386810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49352/0 (socket says 172.21.15.31:49352) 2024-03-21T10:39:27.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.865+0000 7f1940e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564bc337d8b0 0x564bc337dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.865+0000 7f1941800640 1 -- 172.21.15.31:0/1351082718 learned_addr learned my addr 172.21.15.31:0/1351082718 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:27.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.865+0000 7f1940e00640 1 --2- 172.21.15.31:0/1351082718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bc3386de0 0x564bc3389230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.865+0000 7f1941800640 1 -- 172.21.15.31:0/1351082718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bc3386de0 msgr2=0x564bc3389230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:27.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.865+0000 7f1941800640 1 --2- 172.21.15.31:0/1351082718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bc3386de0 0x564bc3389230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.865+0000 7f1941800640 1 -- 172.21.15.31:0/1351082718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564bc337d8b0 msgr2=0x564bc337dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:27.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.865+0000 7f1941800640 1 --2- 172.21.15.31:0/1351082718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564bc337d8b0 0x564bc337dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.865+0000 7f1941800640 1 -- 172.21.15.31:0/1351082718 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564bc32ae080 con 0x564bc337e260 2024-03-21T10:39:27.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.865+0000 7f1940e00640 1 --2- 172.21.15.31:0/1351082718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bc3386de0 0x564bc3389230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:39:27.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.865+0000 7f1941800640 1 --2- 172.21.15.31:0/1351082718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564bc337e260 0x564bc3386810 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1930009e10 tx=0x7f1930011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8aab6c4ccdfacd65 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:27.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.866+0000 7f192ea00640 1 -- 172.21.15.31:0/1351082718 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193001e070 con 0x564bc337e260 2024-03-21T10:39:27.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.866+0000 7f192ea00640 1 -- 172.21.15.31:0/1351082718 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1930019b10 con 0x564bc337e260 2024-03-21T10:39:27.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.866+0000 7f192ea00640 1 -- 172.21.15.31:0/1351082718 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193000fbf0 con 0x564bc337e260 2024-03-21T10:39:27.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.866+0000 7f193a000640 1 --2- 172.21.15.31:0/2830461600 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f19240a3230 0x7f19240a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.866+0000 7f193a000640 1 -- 172.21.15.31:0/2830461600 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f19340f2110 con 0x7f193c10dd60 2024-03-21T10:39:27.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.866+0000 7f1940e00640 1 --2- 172.21.15.31:0/2830461600 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f19240a3230 0x7f19240a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.867+0000 7f1946e2d740 1 -- 172.21.15.31:0/1351082718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564bc337e260 msgr2=0x564bc3386810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:27.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.867+0000 7f1946e2d740 1 --2- 172.21.15.31:0/1351082718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564bc337e260 0x564bc3386810 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1930009e10 tx=0x7f1930011810 comp rx=0 tx=0).stop 2024-03-21T10:39:27.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.867+0000 7f1946e2d740 1 -- 172.21.15.31:0/1351082718 shutdown_connections 2024-03-21T10:39:27.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.867+0000 7f1946e2d740 1 --2- 172.21.15.31:0/1351082718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bc3386de0 0x564bc3389230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.867+0000 7f1946e2d740 1 --2- 172.21.15.31:0/1351082718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564bc337e260 0x564bc3386810 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.867+0000 7f1946e2d740 1 --2- 172.21.15.31:0/1351082718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564bc337d8b0 0x564bc337dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.867+0000 7f1946e2d740 1 -- 172.21.15.31:0/1351082718 >> 172.21.15.31:0/1351082718 conn(0x564bc3377f80 msgr2=0x564bc3378360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:27.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.867+0000 7f1946e2d740 1 -- 172.21.15.31:0/1351082718 shutdown_connections 2024-03-21T10:39:27.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.867+0000 7f1946e2d740 1 -- 172.21.15.31:0/1351082718 wait complete. 2024-03-21T10:39:27.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.867+0000 7f1940e00640 1 --2- 172.21.15.31:0/2830461600 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f19240a3230 0x7f19240a56f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f193c10f010 tx=0x7f1928005f90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:27.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.867+0000 7f1946e2d740 1 -- start start 2024-03-21T10:39:27.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.868+0000 7f1946e2d740 1 Processor -- start 2024-03-21T10:39:27.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.868+0000 7f1946e2d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564bc337d8b0 0x564bc3386750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.868+0000 7f1946e2d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bc337e260 0x564bc3383ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.868+0000 7f1946e2d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564bc3386de0 0x564bc33840f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.868+0000 7f1946e2d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bc3389d50 con 0x564bc3386de0 2024-03-21T10:39:27.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.868+0000 7f1946e2d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bc329d0d0 con 0x564bc337d8b0 2024-03-21T10:39:27.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.868+0000 7f1946e2d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bc3389ed0 con 0x564bc337e260 2024-03-21T10:39:27.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.869+0000 7f1946e2d740 10 client.0 _ll_register_callbacks cb 0x564bc32a1e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:39:27.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.869+0000 7f1940e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bc337e260 0x564bc3383ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.869+0000 7f1941800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564bc337d8b0 0x564bc3386750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.869+0000 7f1940e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bc337e260 0x564bc3383ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:47064/0 (socket says 172.21.15.31:47064) 2024-03-21T10:39:27.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.869+0000 7f1940e00640 1 -- 172.21.15.31:0/1912286950 learned_addr learned my addr 172.21.15.31:0/1912286950 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:27.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.869+0000 7f1942200640 1 --2- 172.21.15.31:0/1912286950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564bc3386de0 0x564bc33840f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.869+0000 7f1940e00640 1 -- 172.21.15.31:0/1912286950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564bc337d8b0 msgr2=0x564bc3386750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:27.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.869+0000 7f1940e00640 1 --2- 172.21.15.31:0/1912286950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564bc337d8b0 0x564bc3386750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.869+0000 7f1940e00640 1 -- 172.21.15.31:0/1912286950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564bc3386de0 msgr2=0x564bc33840f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:27.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.869+0000 7f1940e00640 1 --2- 172.21.15.31:0/1912286950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564bc3386de0 0x564bc33840f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.869+0000 7f1940e00640 1 -- 172.21.15.31:0/1912286950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564bc2ff53f0 con 0x564bc337e260 2024-03-21T10:39:27.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.869+0000 7f1941800640 1 --2- 172.21.15.31:0/1912286950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564bc337d8b0 0x564bc3386750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:39:27.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.869+0000 7f1942200640 1 --2- 172.21.15.31:0/1912286950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564bc3386de0 0x564bc33840f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:39:27.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.870+0000 7f1940e00640 1 --2- 172.21.15.31:0/1912286950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bc337e260 0x564bc3383ae0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1928009fd0 tx=0x7f1928004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5bdb1e73ae1fd18 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:27.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.870+0000 7f1923400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:39:27.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.870+0000 7f1923400640 1 -- 172.21.15.31:0/1912286950 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1928036060 con 0x564bc337e260 2024-03-21T10:39:27.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.870+0000 7f1946e2d740 1 -- 172.21.15.31:0/1912286950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1930019b10 con 0x564bc337e260 2024-03-21T10:39:27.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.870+0000 7f1923400640 1 -- 172.21.15.31:0/1912286950 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f1928019030 con 0x564bc337e260 2024-03-21T10:39:27.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.870+0000 7f1923400640 1 -- 172.21.15.31:0/1912286950 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f192803d050 con 0x564bc337e260 2024-03-21T10:39:27.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.870+0000 7f1946e2d740 10 client.25849 fetch_fsmap learned FSMap version 24 2024-03-21T10:39:27.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.870+0000 7f1946e2d740 1 -- 172.21.15.31:0/1912286950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x564bc30bf7d0 con 0x564bc337e260 2024-03-21T10:39:27.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.870+0000 7f1923400640 1 -- 172.21.15.31:0/1912286950 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1930019b10 con 0x564bc337e260 2024-03-21T10:39:27.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.871+0000 7f1923400640 1 -- 172.21.15.31:0/1912286950 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f192800d500 con 0x564bc337e260 2024-03-21T10:39:27.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.871+0000 7f1946e2d740 10 client.25849 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:39:27.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.871+0000 7f1946e2d740 10 client.25849 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:39:27.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.871+0000 7f1946e2d740 20 client.25849 populate_metadata read hostname 'smithi031' 2024-03-21T10:39:27.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.871+0000 7f1946e2d740 1 --2- 172.21.15.31:0/1912286950 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x564bc3396c10 0x564bc3399020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.871+0000 7f1946e2d740 4 client.25849 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:39:27.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.871+0000 7f1946e2d740 1 -- 172.21.15.31:0/1912286950 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564bc32ee800 con 0x564bc3396c10 2024-03-21T10:39:27.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.871+0000 7f1941800640 1 --2- 172.21.15.31:0/1912286950 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x564bc3396c10 0x564bc3399020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.872+0000 7f1941800640 1 --2- 172.21.15.31:0/1912286950 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x564bc3396c10 0x564bc3399020 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3f54474eeedace18 server_cookie=2d2b83f6531bfa63 in_seq=0 out_seq=0 2024-03-21T10:39:27.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.872+0000 7f1923400640 10 client.25849 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:39:27.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.872+0000 7f1923400640 1 -- 172.21.15.31:0/1912286950 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564bc32ee800 con 0x564bc3396c10 2024-03-21T10:39:27.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.872+0000 7f1923400640 10 client.25849 handle_command_reply: tid=0 2024-03-21T10:39:27.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.872+0000 7f1946e2d740 1 client.25849 shutdown 2024-03-21T10:39:27.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.873+0000 7f1946e2d740 1 -- 172.21.15.31:0/1912286950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bc337e260 msgr2=0x564bc3383ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:27.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.873+0000 7f1946e2d740 1 --2- 172.21.15.31:0/1912286950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bc337e260 0x564bc3383ae0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1928009fd0 tx=0x7f1928004730 comp rx=0 tx=0).stop 2024-03-21T10:39:27.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.873+0000 7f1946e2d740 1 -- 172.21.15.31:0/1912286950 shutdown_connections 2024-03-21T10:39:27.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.873+0000 7f1946e2d740 1 --2- 172.21.15.31:0/1912286950 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x564bc3396c10 0x564bc3399020 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.873+0000 7f1946e2d740 1 --2- 172.21.15.31:0/1912286950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564bc3386de0 0x564bc33840f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.873+0000 7f1946e2d740 1 --2- 172.21.15.31:0/1912286950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bc337e260 0x564bc3383ae0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.873+0000 7f1923400640 0 client.25849 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:39:27.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.873+0000 7f1946e2d740 1 --2- 172.21.15.31:0/1912286950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564bc337d8b0 0x564bc3386750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.873+0000 7f1946e2d740 1 -- 172.21.15.31:0/1912286950 >> 172.21.15.31:0/1912286950 conn(0x564bc3377f80 msgr2=0x564bc3379cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:27.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.873+0000 7f1946e2d740 1 -- 172.21.15.31:0/1912286950 shutdown_connections 2024-03-21T10:39:27.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.873+0000 7f1946e2d740 1 -- 172.21.15.31:0/1912286950 wait complete. 2024-03-21T10:39:27.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.873+0000 7f1946e2d740 20 client.25849 trim_cache size 0 max 16384 2024-03-21T10:39:27.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.887+0000 7f1946e2d740 1 Processor -- start 2024-03-21T10:39:27.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.887+0000 7f1946e2d740 1 -- start start 2024-03-21T10:39:27.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.887+0000 7f1946e2d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564bc337d8b0 0x564bc3386210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.887+0000 7f1946e2d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564bc337e260 0x564bc3386750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.887+0000 7f1946e2d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bc3386de0 0x564bc3380d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.888+0000 7f1946e2d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c1ae940 con 0x564bc337d8b0 2024-03-21T10:39:27.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.888+0000 7f1946e2d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c1ac440 con 0x564bc337e260 2024-03-21T10:39:27.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.888+0000 7f1946e2d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c1aa4e0 con 0x564bc3386de0 2024-03-21T10:39:27.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.888+0000 7f1940e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564bc337e260 0x564bc3386750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.888+0000 7f1941800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564bc337d8b0 0x564bc3386210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.888+0000 7f1942200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bc3386de0 0x564bc3380d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.888+0000 7f1940e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564bc337e260 0x564bc3386750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49382/0 (socket says 172.21.15.31:49382) 2024-03-21T10:39:27.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.888+0000 7f1940e00640 1 -- 172.21.15.31:0/871089989 learned_addr learned my addr 172.21.15.31:0/871089989 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:27.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.888+0000 7f1940e00640 1 -- 172.21.15.31:0/871089989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bc3386de0 msgr2=0x564bc3380d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:27.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.888+0000 7f1940e00640 1 --2- 172.21.15.31:0/871089989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bc3386de0 0x564bc3380d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.888+0000 7f1940e00640 1 -- 172.21.15.31:0/871089989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564bc337d8b0 msgr2=0x564bc3386210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:27.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.888+0000 7f1940e00640 1 --2- 172.21.15.31:0/871089989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564bc337d8b0 0x564bc3386210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.888+0000 7f1940e00640 1 -- 172.21.15.31:0/871089989 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564bc337cc00 con 0x564bc337e260 2024-03-21T10:39:27.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.888+0000 7f1942200640 1 --2- 172.21.15.31:0/871089989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bc3386de0 0x564bc3380d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:39:27.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.888+0000 7f1940e00640 1 --2- 172.21.15.31:0/871089989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564bc337e260 0x564bc3386750 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1928012d50 tx=0x7f192800f3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18d9f116b55ed0cb server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:27.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.889+0000 7f192e000640 1 -- 172.21.15.31:0/871089989 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1928046070 con 0x564bc337e260 2024-03-21T10:39:27.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.889+0000 7f192e000640 1 -- 172.21.15.31:0/871089989 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f192800dac0 con 0x564bc337e260 2024-03-21T10:39:27.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.889+0000 7f192e000640 1 -- 172.21.15.31:0/871089989 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1928031cc0 con 0x564bc337e260 2024-03-21T10:39:27.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.889+0000 7f1946e2d740 1 -- 172.21.15.31:0/871089989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564bc337e260 msgr2=0x564bc3386750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:27.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.889+0000 7f1946e2d740 1 --2- 172.21.15.31:0/871089989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564bc337e260 0x564bc3386750 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1928012d50 tx=0x7f192800f3c0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.890+0000 7f1946e2d740 1 -- 172.21.15.31:0/871089989 shutdown_connections 2024-03-21T10:39:27.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.890+0000 7f1946e2d740 1 --2- 172.21.15.31:0/871089989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bc3386de0 0x564bc3380d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.890+0000 7f1946e2d740 1 --2- 172.21.15.31:0/871089989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564bc337e260 0x564bc3386750 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.890+0000 7f1946e2d740 1 --2- 172.21.15.31:0/871089989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564bc337d8b0 0x564bc3386210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.890+0000 7f1946e2d740 1 -- 172.21.15.31:0/871089989 >> 172.21.15.31:0/871089989 conn(0x564bc3377f80 msgr2=0x564bc3377160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:27.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.890+0000 7f1946e2d740 1 -- 172.21.15.31:0/871089989 shutdown_connections 2024-03-21T10:39:27.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.890+0000 7f1946e2d740 1 -- 172.21.15.31:0/871089989 wait complete. 2024-03-21T10:39:27.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.890+0000 7f1946e2d740 1 -- start start 2024-03-21T10:39:27.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.890+0000 7f1946e2d740 1 Processor -- start 2024-03-21T10:39:27.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.891+0000 7f1946e2d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564bc337d8b0 0x564bc338eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.891+0000 7f1946e2d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bc337e260 0x564bc338f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.891+0000 7f1946e2d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564bc3386de0 0x564bc338d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.891+0000 7f1946e2d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c1aaae0 con 0x564bc337d8b0 2024-03-21T10:39:27.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.891+0000 7f1946e2d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c1abde0 con 0x564bc3386de0 2024-03-21T10:39:27.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.891+0000 7f1946e2d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c1ab120 con 0x564bc337e260 2024-03-21T10:39:27.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.891+0000 7f1946e2d740 10 client.0 _ll_register_callbacks cb 0x7f19280097e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:39:27.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.891+0000 7f1940e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bc337e260 0x564bc338f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.891+0000 7f1940e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bc337e260 0x564bc338f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:47088/0 (socket says 172.21.15.31:47088) 2024-03-21T10:39:27.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.891+0000 7f1940e00640 1 -- 172.21.15.31:0/3290797477 learned_addr learned my addr 172.21.15.31:0/3290797477 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:27.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.891+0000 7f1941800640 1 --2- 172.21.15.31:0/3290797477 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564bc337d8b0 0x564bc338eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.891+0000 7f1942200640 1 --2- 172.21.15.31:0/3290797477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564bc3386de0 0x564bc338d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.891+0000 7f1940e00640 1 -- 172.21.15.31:0/3290797477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564bc3386de0 msgr2=0x564bc338d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:27.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.891+0000 7f1940e00640 1 --2- 172.21.15.31:0/3290797477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564bc3386de0 0x564bc338d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.891+0000 7f1940e00640 1 -- 172.21.15.31:0/3290797477 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564bc337d8b0 msgr2=0x564bc338eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:27.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.891+0000 7f1940e00640 1 --2- 172.21.15.31:0/3290797477 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564bc337d8b0 0x564bc338eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.891+0000 7f1940e00640 1 -- 172.21.15.31:0/3290797477 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564bc338d8d0 con 0x564bc337e260 2024-03-21T10:39:27.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.892+0000 7f1940e00640 1 --2- 172.21.15.31:0/3290797477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bc337e260 0x564bc338f070 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f192800d950 tx=0x7f19280316d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf905ec4c03d05a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:27.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.892+0000 7f1923400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:39:27.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.893+0000 7f1923400640 1 -- 172.21.15.31:0/3290797477 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1928046050 con 0x564bc337e260 2024-03-21T10:39:27.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.893+0000 7f1923400640 1 -- 172.21.15.31:0/3290797477 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f192800d550 con 0x564bc337e260 2024-03-21T10:39:27.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.893+0000 7f1946e2d740 1 -- 172.21.15.31:0/3290797477 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f192800dac0 con 0x564bc337e260 2024-03-21T10:39:27.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.893+0000 7f1923400640 1 -- 172.21.15.31:0/3290797477 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1928031900 con 0x564bc337e260 2024-03-21T10:39:27.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.893+0000 7f1946e2d740 10 client.25855 fetch_fsmap learned FSMap version 24 2024-03-21T10:39:27.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.893+0000 7f1946e2d740 1 -- 172.21.15.31:0/3290797477 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x564bc33f3180 con 0x564bc337e260 2024-03-21T10:39:27.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.893+0000 7f1923400640 1 -- 172.21.15.31:0/3290797477 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f192800dac0 con 0x564bc337e260 2024-03-21T10:39:27.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.894+0000 7f1923400640 1 -- 172.21.15.31:0/3290797477 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f1928004680 con 0x564bc337e260 2024-03-21T10:39:27.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.894+0000 7f1946e2d740 10 client.25855 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:39:27.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.894+0000 7f1946e2d740 10 client.25855 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:39:27.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.894+0000 7f1946e2d740 20 client.25855 populate_metadata read hostname 'smithi031' 2024-03-21T10:39:27.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.894+0000 7f1946e2d740 1 --2- 172.21.15.31:0/3290797477 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x564bc33f3730 0x564bc33f5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:27.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.894+0000 7f1946e2d740 4 client.25855 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:39:27.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.894+0000 7f1946e2d740 1 -- 172.21.15.31:0/3290797477 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564bc32e8b50 con 0x564bc33f3730 2024-03-21T10:39:27.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.894+0000 7f1941800640 1 --2- 172.21.15.31:0/3290797477 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x564bc33f3730 0x564bc33f5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:27.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.895+0000 7f1923400640 10 client.25855 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:39:27.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.895+0000 7f1941800640 1 --2- 172.21.15.31:0/3290797477 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x564bc33f3730 0x564bc33f5b20 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=881b4b374f633a0d server_cookie=ce9b1af497a44c97 in_seq=0 out_seq=0 2024-03-21T10:39:27.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.895+0000 7f1923400640 1 -- 172.21.15.31:0/3290797477 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564bc32e8b50 con 0x564bc33f3730 2024-03-21T10:39:27.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.895+0000 7f1923400640 10 client.25855 handle_command_reply: tid=0 2024-03-21T10:39:27.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.895+0000 7f1946e2d740 1 client.25855 shutdown 2024-03-21T10:39:27.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.895+0000 7f1946e2d740 1 -- 172.21.15.31:0/3290797477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bc337e260 msgr2=0x564bc338f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:27.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.895+0000 7f1946e2d740 1 --2- 172.21.15.31:0/3290797477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bc337e260 0x564bc338f070 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f192800d950 tx=0x7f19280316d0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.895+0000 7f1946e2d740 1 -- 172.21.15.31:0/3290797477 shutdown_connections 2024-03-21T10:39:27.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.895+0000 7f1946e2d740 1 --2- 172.21.15.31:0/3290797477 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x564bc33f3730 0x564bc33f5b20 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.895+0000 7f1946e2d740 1 --2- 172.21.15.31:0/3290797477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564bc3386de0 0x564bc338d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.895+0000 7f1946e2d740 1 --2- 172.21.15.31:0/3290797477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bc337e260 0x564bc338f070 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.895+0000 7f1946e2d740 1 --2- 172.21.15.31:0/3290797477 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564bc337d8b0 0x564bc338eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.895+0000 7f1923400640 0 client.25855 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:39:27.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.895+0000 7f1946e2d740 1 -- 172.21.15.31:0/3290797477 >> 172.21.15.31:0/3290797477 conn(0x564bc3377f80 msgr2=0x564bc337e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:27.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.895+0000 7f1946e2d740 1 -- 172.21.15.31:0/3290797477 shutdown_connections 2024-03-21T10:39:27.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.895+0000 7f1946e2d740 1 -- 172.21.15.31:0/3290797477 wait complete. 2024-03-21T10:39:27.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.895+0000 7f1946e2d740 20 client.25855 trim_cache size 0 max 16384 2024-03-21T10:39:27.895 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:39:27.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.896+0000 7f1942c00640 1 -- 172.21.15.31:0/2830461600 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f19240a3230 msgr2=0x7f19240a56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:27.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.896+0000 7f1942c00640 1 --2- 172.21.15.31:0/2830461600 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f19240a3230 0x7f19240a56f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f193c10f010 tx=0x7f1928005f90 comp rx=0 tx=0).stop 2024-03-21T10:39:27.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.896+0000 7f1942c00640 1 -- 172.21.15.31:0/2830461600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f193c10dd60 msgr2=0x7f193c19e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:27.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.896+0000 7f1942c00640 1 --2- 172.21.15.31:0/2830461600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f193c10dd60 0x7f193c19e870 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f193402c710 tx=0x7f1934032f30 comp rx=0 tx=0).stop 2024-03-21T10:39:27.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.896+0000 7f1942c00640 1 -- 172.21.15.31:0/2830461600 shutdown_connections 2024-03-21T10:39:27.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.896+0000 7f1942c00640 1 --2- 172.21.15.31:0/2830461600 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f19240a3230 0x7f19240a56f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.896+0000 7f1942c00640 1 --2- 172.21.15.31:0/2830461600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f193c10dd60 0x7f193c19e870 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.896+0000 7f1942c00640 1 --2- 172.21.15.31:0/2830461600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f193c100b60 0x7f193c19a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.896+0000 7f1942c00640 1 --2- 172.21.15.31:0/2830461600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f193c1001b0 0x7f193c199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:27.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.896+0000 7f1942c00640 1 -- 172.21.15.31:0/2830461600 >> 172.21.15.31:0/2830461600 conn(0x7f193c077130 msgr2=0x7f193c0fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:27.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.897+0000 7f1942c00640 1 -- 172.21.15.31:0/2830461600 shutdown_connections 2024-03-21T10:39:27.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:27.897+0000 7f1942c00640 1 -- 172.21.15.31:0/2830461600 wait complete. 2024-03-21T10:39:27.909 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:27 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/755203861' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:39:27.911 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:39:27.911 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-21T10:39:27.930 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:27 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/755203861' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:39:28.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:27 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/755203861' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:39:28.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.135+0000 7fb337200640 1 -- 172.21.15.31:0/1120692050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3381014c0 msgr2=0x7fb338101940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:28.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.135+0000 7fb337200640 1 --2- 172.21.15.31:0/1120692050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3381014c0 0x7fb338101940 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb324009930 tx=0x7fb32402bc20 comp rx=0 tx=0).stop 2024-03-21T10:39:28.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.135+0000 7fb337200640 1 -- 172.21.15.31:0/1120692050 shutdown_connections 2024-03-21T10:39:28.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.135+0000 7fb337200640 1 --2- 172.21.15.31:0/1120692050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb338101e80 0x7fb33810e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.135+0000 7fb337200640 1 --2- 172.21.15.31:0/1120692050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3381014c0 0x7fb338101940 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.135+0000 7fb337200640 1 --2- 172.21.15.31:0/1120692050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb338107580 0x7fb338107960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.135+0000 7fb337200640 1 -- 172.21.15.31:0/1120692050 >> 172.21.15.31:0/1120692050 conn(0x7fb3380fcfb0 msgr2=0x7fb3380ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:28.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.135+0000 7fb337200640 1 -- 172.21.15.31:0/1120692050 shutdown_connections 2024-03-21T10:39:28.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.135+0000 7fb337200640 1 -- 172.21.15.31:0/1120692050 wait complete. 2024-03-21T10:39:28.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.136+0000 7fb337200640 1 Processor -- start 2024-03-21T10:39:28.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.136+0000 7fb337200640 1 -- start start 2024-03-21T10:39:28.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.137+0000 7fb337200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3381014c0 0x7fb3380739a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.137+0000 7fb337200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb338101e80 0x7fb338073ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.137+0000 7fb337200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb338107580 0x7fb338074420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.137+0000 7fb337200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3381118b0 con 0x7fb3381014c0 2024-03-21T10:39:28.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.137+0000 7fb337200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb338111730 con 0x7fb338107580 2024-03-21T10:39:28.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.137+0000 7fb337200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb338111a30 con 0x7fb338101e80 2024-03-21T10:39:28.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.137+0000 7fb335e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3381014c0 0x7fb3380739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.137+0000 7fb335400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb338101e80 0x7fb338073ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.137+0000 7fb336800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb338107580 0x7fb338074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.137+0000 7fb336800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb338107580 0x7fb338074420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49412/0 (socket says 172.21.15.31:49412) 2024-03-21T10:39:28.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.137+0000 7fb335400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb338101e80 0x7fb338073ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:47092/0 (socket says 172.21.15.31:47092) 2024-03-21T10:39:28.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.137+0000 7fb335e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3381014c0 0x7fb3380739a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33390/0 (socket says 172.21.15.31:33390) 2024-03-21T10:39:28.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.137+0000 7fb336800640 1 -- 172.21.15.31:0/2121779042 learned_addr learned my addr 172.21.15.31:0/2121779042 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:28.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.138+0000 7fb335400640 1 -- 172.21.15.31:0/2121779042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb338107580 msgr2=0x7fb338074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:28.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.138+0000 7fb335400640 1 --2- 172.21.15.31:0/2121779042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb338107580 0x7fb338074420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.138+0000 7fb335400640 1 -- 172.21.15.31:0/2121779042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3381014c0 msgr2=0x7fb3380739a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:28.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.138+0000 7fb335400640 1 --2- 172.21.15.31:0/2121779042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3381014c0 0x7fb3380739a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.138+0000 7fb335400640 1 -- 172.21.15.31:0/2121779042 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3381a6250 con 0x7fb338101e80 2024-03-21T10:39:28.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.138+0000 7fb336800640 1 --2- 172.21.15.31:0/2121779042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb338107580 0x7fb338074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:39:28.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.138+0000 7fb335400640 1 --2- 172.21.15.31:0/2121779042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb338101e80 0x7fb338073ee0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb324036040 tx=0x7fb324004340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:28.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.138+0000 7fb32ea00640 1 -- 172.21.15.31:0/2121779042 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb324043020 con 0x7fb338101e80 2024-03-21T10:39:28.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.138+0000 7fb32ea00640 1 -- 172.21.15.31:0/2121779042 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3240054e0 con 0x7fb338101e80 2024-03-21T10:39:28.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.138+0000 7fb337200640 1 -- 172.21.15.31:0/2121779042 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3381a64e0 con 0x7fb338101e80 2024-03-21T10:39:28.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.138+0000 7fb32ea00640 1 -- 172.21.15.31:0/2121779042 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb324003ab0 con 0x7fb338101e80 2024-03-21T10:39:28.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.138+0000 7fb337200640 1 -- 172.21.15.31:0/2121779042 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3381a6a00 con 0x7fb338101e80 2024-03-21T10:39:28.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.141+0000 7fb32ea00640 1 -- 172.21.15.31:0/2121779042 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) v1 ==== 138087+0+0 (secure 0 0 0) 0x7fb324046020 con 0x7fb338101e80 2024-03-21T10:39:28.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.142+0000 7fb33f83a740 1 Processor -- start 2024-03-21T10:39:28.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.142+0000 7fb33f83a740 1 -- start start 2024-03-21T10:39:28.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.142+0000 7fb33f83a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556fceba48b0 0x556fceba4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.142+0000 7fb33f83a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fceba5260 0x556fcebad810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.142+0000 7fb33f83a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556fcebadde0 0x556fcebb0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.142+0000 7fb33f83a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fceb082c0 con 0x556fceba48b0 2024-03-21T10:39:28.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.142+0000 7fb33f83a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fceab0c90 con 0x556fcebadde0 2024-03-21T10:39:28.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.142+0000 7fb33f83a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fceb04360 con 0x556fceba5260 2024-03-21T10:39:28.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.143+0000 7fb335e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fceba5260 0x556fcebad810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.142+0000 7fb336800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556fceba48b0 0x556fceba4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.143+0000 7fb335400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556fcebadde0 0x556fcebb0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.143+0000 7fb335e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fceba5260 0x556fcebad810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:47104/0 (socket says 172.21.15.31:47104) 2024-03-21T10:39:28.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.143+0000 7fb335e00640 1 -- 172.21.15.31:0/1324776213 learned_addr learned my addr 172.21.15.31:0/1324776213 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:28.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.143+0000 7fb335e00640 1 -- 172.21.15.31:0/1324776213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556fcebadde0 msgr2=0x556fcebb0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:28.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.143+0000 7fb335e00640 1 --2- 172.21.15.31:0/1324776213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556fcebadde0 0x556fcebb0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.143+0000 7fb335e00640 1 -- 172.21.15.31:0/1324776213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556fceba48b0 msgr2=0x556fceba4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:28.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.143+0000 7fb335e00640 1 --2- 172.21.15.31:0/1324776213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556fceba48b0 0x556fceba4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.143+0000 7fb335e00640 1 -- 172.21.15.31:0/1324776213 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556fcead5080 con 0x556fceba5260 2024-03-21T10:39:28.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.143+0000 7fb336800640 1 --2- 172.21.15.31:0/1324776213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556fceba48b0 0x556fceba4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:39:28.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.143+0000 7fb335400640 1 --2- 172.21.15.31:0/1324776213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556fcebadde0 0x556fcebb0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:39:28.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.144+0000 7fb335e00640 1 --2- 172.21.15.31:0/1324776213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fceba5260 0x556fcebad810 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb328009ca0 tx=0x7fb328011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b2c7f918025a844 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:28.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.144+0000 7fb323000640 1 -- 172.21.15.31:0/1324776213 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32801e070 con 0x556fceba5260 2024-03-21T10:39:28.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.144+0000 7fb323000640 1 -- 172.21.15.31:0/1324776213 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb328019b10 con 0x556fceba5260 2024-03-21T10:39:28.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.144+0000 7fb323000640 1 -- 172.21.15.31:0/1324776213 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32800eae0 con 0x556fceba5260 2024-03-21T10:39:28.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.144+0000 7fb32ea00640 1 --2- 172.21.15.31:0/2121779042 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb31c0a31e0 0x7fb31c0a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.144+0000 7fb32ea00640 1 -- 172.21.15.31:0/2121779042 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fb3240f1c20 con 0x7fb338101e80 2024-03-21T10:39:28.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.144+0000 7fb336800640 1 --2- 172.21.15.31:0/2121779042 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb31c0a31e0 0x7fb31c0a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.145+0000 7fb33f83a740 1 -- 172.21.15.31:0/1324776213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fceba5260 msgr2=0x556fcebad810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:28.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.145+0000 7fb33f83a740 1 --2- 172.21.15.31:0/1324776213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fceba5260 0x556fcebad810 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb328009ca0 tx=0x7fb328011810 comp rx=0 tx=0).stop 2024-03-21T10:39:28.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.145+0000 7fb33f83a740 1 -- 172.21.15.31:0/1324776213 shutdown_connections 2024-03-21T10:39:28.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.145+0000 7fb33f83a740 1 --2- 172.21.15.31:0/1324776213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556fcebadde0 0x556fcebb0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.145+0000 7fb33f83a740 1 --2- 172.21.15.31:0/1324776213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fceba5260 0x556fcebad810 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.145+0000 7fb33f83a740 1 --2- 172.21.15.31:0/1324776213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556fceba48b0 0x556fceba4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.145+0000 7fb33f83a740 1 -- 172.21.15.31:0/1324776213 >> 172.21.15.31:0/1324776213 conn(0x556fceb9ef80 msgr2=0x556fceb9f360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:28.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.145+0000 7fb33f83a740 1 -- 172.21.15.31:0/1324776213 shutdown_connections 2024-03-21T10:39:28.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.145+0000 7fb336800640 1 --2- 172.21.15.31:0/2121779042 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb31c0a31e0 0x7fb31c0a56a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb330005c40 tx=0x7fb330009760 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:28.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.145+0000 7fb33f83a740 1 -- 172.21.15.31:0/1324776213 wait complete. 2024-03-21T10:39:28.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.146+0000 7fb33f83a740 1 -- start start 2024-03-21T10:39:28.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.146+0000 7fb33f83a740 1 Processor -- start 2024-03-21T10:39:28.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.146+0000 7fb33f83a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556fceba48b0 0x556fcebad710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.146+0000 7fb33f83a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556fceba5260 0x556fcebaaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.146+0000 7fb33f83a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fcebadde0 0x556fcebab0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.146+0000 7fb33f83a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fcebb0d50 con 0x556fceba48b0 2024-03-21T10:39:28.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.146+0000 7fb33f83a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fceac40d0 con 0x556fceba5260 2024-03-21T10:39:28.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.146+0000 7fb33f83a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fcebb0ed0 con 0x556fcebadde0 2024-03-21T10:39:28.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.146+0000 7fb33f83a740 10 client.0 _ll_register_callbacks cb 0x556fceac8e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:39:28.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.147+0000 7fb336800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fcebadde0 0x556fcebab0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.147+0000 7fb335400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556fceba5260 0x556fcebaaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.147+0000 7fb335e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556fceba48b0 0x556fcebad710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.147+0000 7fb335400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556fceba5260 0x556fcebaaaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49432/0 (socket says 172.21.15.31:49432) 2024-03-21T10:39:28.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.147+0000 7fb336800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fcebadde0 0x556fcebab0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:47106/0 (socket says 172.21.15.31:47106) 2024-03-21T10:39:28.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.147+0000 7fb335400640 1 -- 172.21.15.31:0/4279423892 learned_addr learned my addr 172.21.15.31:0/4279423892 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:28.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.147+0000 7fb335400640 1 -- 172.21.15.31:0/4279423892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fcebadde0 msgr2=0x556fcebab0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:28.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.147+0000 7fb335400640 1 --2- 172.21.15.31:0/4279423892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fcebadde0 0x556fcebab0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.147+0000 7fb335400640 1 -- 172.21.15.31:0/4279423892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556fceba48b0 msgr2=0x556fcebad710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:28.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.147+0000 7fb335400640 1 --2- 172.21.15.31:0/4279423892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556fceba48b0 0x556fcebad710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.147+0000 7fb335400640 1 -- 172.21.15.31:0/4279423892 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556fce81c3f0 con 0x556fceba5260 2024-03-21T10:39:28.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.147+0000 7fb335e00640 1 --2- 172.21.15.31:0/4279423892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556fceba48b0 0x556fcebad710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:39:28.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.147+0000 7fb335400640 1 --2- 172.21.15.31:0/4279423892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556fceba5260 0x556fcebaaaa0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb324006010 tx=0x7fb324004530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6c0d3eae032001d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:28.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.147+0000 7fb317400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:39:28.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.148+0000 7fb317400640 1 -- 172.21.15.31:0/4279423892 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3240bee60 con 0x556fceba5260 2024-03-21T10:39:28.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.148+0000 7fb317400640 1 -- 172.21.15.31:0/4279423892 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fb324003c50 con 0x556fceba5260 2024-03-21T10:39:28.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.148+0000 7fb33f83a740 1 -- 172.21.15.31:0/4279423892 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb328019b10 con 0x556fceba5260 2024-03-21T10:39:28.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.148+0000 7fb317400640 1 -- 172.21.15.31:0/4279423892 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3240d6e60 con 0x556fceba5260 2024-03-21T10:39:28.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.148+0000 7fb33f83a740 10 client.25883 fetch_fsmap learned FSMap version 24 2024-03-21T10:39:28.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.148+0000 7fb33f83a740 1 -- 172.21.15.31:0/4279423892 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x556fce8e67d0 con 0x556fceba5260 2024-03-21T10:39:28.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.148+0000 7fb317400640 1 -- 172.21.15.31:0/4279423892 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3240d3020 con 0x556fceba5260 2024-03-21T10:39:28.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.148+0000 7fb317400640 1 -- 172.21.15.31:0/4279423892 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fb3240fb040 con 0x556fceba5260 2024-03-21T10:39:28.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.148+0000 7fb33f83a740 10 client.25883 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:39:28.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.148+0000 7fb33f83a740 10 client.25883 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:39:28.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.148+0000 7fb33f83a740 20 client.25883 populate_metadata read hostname 'smithi031' 2024-03-21T10:39:28.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.148+0000 7fb33f83a740 1 --2- 172.21.15.31:0/4279423892 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x556fcebbdc30 0x556fcebc0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.148+0000 7fb33f83a740 4 client.25883 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:39:28.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.148+0000 7fb33f83a740 1 -- 172.21.15.31:0/4279423892 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556fceb15800 con 0x556fcebbdc30 2024-03-21T10:39:28.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.149+0000 7fb335e00640 1 --2- 172.21.15.31:0/4279423892 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x556fcebbdc30 0x556fcebc0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.149+0000 7fb335e00640 1 --2- 172.21.15.31:0/4279423892 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x556fcebbdc30 0x556fcebc0070 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=459cad08cbd6abf5 server_cookie=338d4702322d8667 in_seq=0 out_seq=0 2024-03-21T10:39:28.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.149+0000 7fb317400640 10 client.25883 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:39:28.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.150+0000 7fb317400640 1 -- 172.21.15.31:0/4279423892 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556fceb15800 con 0x556fcebbdc30 2024-03-21T10:39:28.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.150+0000 7fb317400640 10 client.25883 handle_command_reply: tid=0 2024-03-21T10:39:28.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.151+0000 7fb33f83a740 1 client.25883 shutdown 2024-03-21T10:39:28.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.151+0000 7fb33f83a740 1 -- 172.21.15.31:0/4279423892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556fceba5260 msgr2=0x556fcebaaaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:28.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.151+0000 7fb33f83a740 1 --2- 172.21.15.31:0/4279423892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556fceba5260 0x556fcebaaaa0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb324006010 tx=0x7fb324004530 comp rx=0 tx=0).stop 2024-03-21T10:39:28.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.151+0000 7fb33f83a740 1 -- 172.21.15.31:0/4279423892 shutdown_connections 2024-03-21T10:39:28.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.151+0000 7fb33f83a740 1 --2- 172.21.15.31:0/4279423892 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x556fcebbdc30 0x556fcebc0070 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.151+0000 7fb33f83a740 1 --2- 172.21.15.31:0/4279423892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fcebadde0 0x556fcebab0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.151+0000 7fb33f83a740 1 --2- 172.21.15.31:0/4279423892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556fceba5260 0x556fcebaaaa0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.151+0000 7fb33f83a740 1 --2- 172.21.15.31:0/4279423892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556fceba48b0 0x556fcebad710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.151+0000 7fb33f83a740 1 -- 172.21.15.31:0/4279423892 >> 172.21.15.31:0/4279423892 conn(0x556fceb9ef80 msgr2=0x556fceba0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:28.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.151+0000 7fb317400640 0 client.25883 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:39:28.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.151+0000 7fb33f83a740 1 -- 172.21.15.31:0/4279423892 shutdown_connections 2024-03-21T10:39:28.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.151+0000 7fb33f83a740 1 -- 172.21.15.31:0/4279423892 wait complete. 2024-03-21T10:39:28.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.151+0000 7fb33f83a740 20 client.25883 trim_cache size 0 max 16384 2024-03-21T10:39:28.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.165+0000 7fb33f83a740 1 Processor -- start 2024-03-21T10:39:28.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.165+0000 7fb33f83a740 1 -- start start 2024-03-21T10:39:28.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.166+0000 7fb33f83a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fceba48b0 0x556fceb9e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.166+0000 7fb33f83a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556fceba5260 0x556fcebad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.166+0000 7fb33f83a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556fcebadde0 0x556fcebad710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.166+0000 7fb33f83a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3381b2db0 con 0x556fceba5260 2024-03-21T10:39:28.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.166+0000 7fb33f83a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3381b08b0 con 0x556fcebadde0 2024-03-21T10:39:28.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.166+0000 7fb33f83a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3381ae950 con 0x556fceba48b0 2024-03-21T10:39:28.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.166+0000 7fb335e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fceba48b0 0x556fceb9e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.166+0000 7fb335e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fceba48b0 0x556fceb9e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:47108/0 (socket says 172.21.15.31:47108) 2024-03-21T10:39:28.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.166+0000 7fb335e00640 1 -- 172.21.15.31:0/3605497280 learned_addr learned my addr 172.21.15.31:0/3605497280 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:28.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.166+0000 7fb335400640 1 --2- 172.21.15.31:0/3605497280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556fceba5260 0x556fcebad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.166+0000 7fb336800640 1 --2- 172.21.15.31:0/3605497280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556fcebadde0 0x556fcebad710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.166+0000 7fb335e00640 1 -- 172.21.15.31:0/3605497280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556fcebadde0 msgr2=0x556fcebad710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:28.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.166+0000 7fb335e00640 1 --2- 172.21.15.31:0/3605497280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556fcebadde0 0x556fcebad710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.166+0000 7fb335e00640 1 -- 172.21.15.31:0/3605497280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556fceba5260 msgr2=0x556fcebad1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:28.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.166+0000 7fb335e00640 1 --2- 172.21.15.31:0/3605497280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556fceba5260 0x556fcebad1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.166+0000 7fb335e00640 1 -- 172.21.15.31:0/3605497280 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556fceba39a0 con 0x556fceba48b0 2024-03-21T10:39:28.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.167+0000 7fb335e00640 1 --2- 172.21.15.31:0/3605497280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fceba48b0 0x556fceb9e160 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb32801c040 tx=0x7fb328011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d13f215b4a54de13 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:28.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.167+0000 7fb322600640 1 -- 172.21.15.31:0/3605497280 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32802b070 con 0x556fceba48b0 2024-03-21T10:39:28.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.167+0000 7fb322600640 1 -- 172.21.15.31:0/3605497280 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb328029070 con 0x556fceba48b0 2024-03-21T10:39:28.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.167+0000 7fb322600640 1 -- 172.21.15.31:0/3605497280 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32801e040 con 0x556fceba48b0 2024-03-21T10:39:28.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.168+0000 7fb33f83a740 1 -- 172.21.15.31:0/3605497280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fceba48b0 msgr2=0x556fceb9e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:28.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.168+0000 7fb33f83a740 1 --2- 172.21.15.31:0/3605497280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fceba48b0 0x556fceb9e160 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb32801c040 tx=0x7fb328011810 comp rx=0 tx=0).stop 2024-03-21T10:39:28.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.168+0000 7fb33f83a740 1 -- 172.21.15.31:0/3605497280 shutdown_connections 2024-03-21T10:39:28.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.168+0000 7fb33f83a740 1 --2- 172.21.15.31:0/3605497280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556fcebadde0 0x556fcebad710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.168+0000 7fb33f83a740 1 --2- 172.21.15.31:0/3605497280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556fceba5260 0x556fcebad1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.168+0000 7fb33f83a740 1 --2- 172.21.15.31:0/3605497280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fceba48b0 0x556fceb9e160 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.168+0000 7fb33f83a740 1 -- 172.21.15.31:0/3605497280 >> 172.21.15.31:0/3605497280 conn(0x556fceb9ef80 msgr2=0x556fceba5690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:28.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.168+0000 7fb33f83a740 1 -- 172.21.15.31:0/3605497280 shutdown_connections 2024-03-21T10:39:28.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.168+0000 7fb33f83a740 1 -- 172.21.15.31:0/3605497280 wait complete. 2024-03-21T10:39:28.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.168+0000 7fb33f83a740 1 -- start start 2024-03-21T10:39:28.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.168+0000 7fb33f83a740 1 Processor -- start 2024-03-21T10:39:28.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.169+0000 7fb33f83a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fceba48b0 0x556fcebb5910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.169+0000 7fb33f83a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556fceba5260 0x556fcebb5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.169+0000 7fb33f83a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556fcebadde0 0x556fcebb3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.169+0000 7fb33f83a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3381aef50 con 0x556fceba5260 2024-03-21T10:39:28.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.169+0000 7fb33f83a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3381b0250 con 0x556fcebadde0 2024-03-21T10:39:28.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.169+0000 7fb33f83a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3381af590 con 0x556fceba48b0 2024-03-21T10:39:28.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.169+0000 7fb33f83a740 10 client.0 _ll_register_callbacks cb 0x7fb324009290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:39:28.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.169+0000 7fb335e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fceba48b0 0x556fcebb5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.169+0000 7fb335400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556fceba5260 0x556fcebb5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.169+0000 7fb335e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fceba48b0 0x556fcebb5910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:47110/0 (socket says 172.21.15.31:47110) 2024-03-21T10:39:28.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.169+0000 7fb335e00640 1 -- 172.21.15.31:0/3422307466 learned_addr learned my addr 172.21.15.31:0/3422307466 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:28.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.169+0000 7fb336800640 1 --2- 172.21.15.31:0/3422307466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556fcebadde0 0x556fcebb3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.170+0000 7fb335e00640 1 -- 172.21.15.31:0/3422307466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556fcebadde0 msgr2=0x556fcebb3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:28.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.170+0000 7fb335e00640 1 --2- 172.21.15.31:0/3422307466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556fcebadde0 0x556fcebb3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.170+0000 7fb335e00640 1 -- 172.21.15.31:0/3422307466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556fceba5260 msgr2=0x556fcebb5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:28.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.170+0000 7fb335e00640 1 --2- 172.21.15.31:0/3422307466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556fceba5260 0x556fcebb5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.170+0000 7fb335e00640 1 -- 172.21.15.31:0/3422307466 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556fcebb45d0 con 0x556fceba48b0 2024-03-21T10:39:28.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.170+0000 7fb335e00640 1 --2- 172.21.15.31:0/3422307466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fceba48b0 0x556fcebb5910 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb328009bc0 tx=0x7fb328009ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fd244498bbb66c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:28.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.170+0000 7fb317400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:39:28.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.170+0000 7fb317400640 1 -- 172.21.15.31:0/3422307466 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32802b070 con 0x556fceba48b0 2024-03-21T10:39:28.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.170+0000 7fb33f83a740 1 -- 172.21.15.31:0/3422307466 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb328029070 con 0x556fceba48b0 2024-03-21T10:39:28.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.170+0000 7fb317400640 1 -- 172.21.15.31:0/3422307466 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fb328004220 con 0x556fceba48b0 2024-03-21T10:39:28.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.171+0000 7fb317400640 1 -- 172.21.15.31:0/3422307466 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb32803f050 con 0x556fceba48b0 2024-03-21T10:39:28.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.171+0000 7fb33f83a740 10 client.25873 fetch_fsmap learned FSMap version 24 2024-03-21T10:39:28.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.171+0000 7fb33f83a740 1 -- 172.21.15.31:0/3422307466 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x556fcec1a150 con 0x556fceba48b0 2024-03-21T10:39:28.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.171+0000 7fb317400640 1 -- 172.21.15.31:0/3422307466 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb328029070 con 0x556fceba48b0 2024-03-21T10:39:28.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.171+0000 7fb317400640 1 -- 172.21.15.31:0/3422307466 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fb328004680 con 0x556fceba48b0 2024-03-21T10:39:28.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.171+0000 7fb33f83a740 10 client.25873 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:39:28.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.171+0000 7fb33f83a740 10 client.25873 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:39:28.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.171+0000 7fb33f83a740 20 client.25873 populate_metadata read hostname 'smithi031' 2024-03-21T10:39:28.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.171+0000 7fb33f83a740 1 --2- 172.21.15.31:0/3422307466 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x556fcec1a5c0 0x556fcec1c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.171+0000 7fb33f83a740 4 client.25873 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:39:28.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.171+0000 7fb33f83a740 1 -- 172.21.15.31:0/3422307466 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556fceb0fb50 con 0x556fcec1a5c0 2024-03-21T10:39:28.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.172+0000 7fb335400640 1 --2- 172.21.15.31:0/3422307466 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x556fcec1a5c0 0x556fcec1c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.172+0000 7fb335400640 1 --2- 172.21.15.31:0/3422307466 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x556fcec1a5c0 0x556fcec1c9b0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c6e22bb3b9c0d8e1 server_cookie=98ebface75909398 in_seq=0 out_seq=0 2024-03-21T10:39:28.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.172+0000 7fb317400640 10 client.25873 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:39:28.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.172+0000 7fb317400640 1 -- 172.21.15.31:0/3422307466 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556fceb0fb50 con 0x556fcec1a5c0 2024-03-21T10:39:28.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.173+0000 7fb317400640 10 client.25873 handle_command_reply: tid=0 2024-03-21T10:39:28.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.173+0000 7fb33f83a740 1 client.25873 shutdown 2024-03-21T10:39:28.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.173+0000 7fb33f83a740 1 -- 172.21.15.31:0/3422307466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fceba48b0 msgr2=0x556fcebb5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:28.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.173+0000 7fb33f83a740 1 --2- 172.21.15.31:0/3422307466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fceba48b0 0x556fcebb5910 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb328009bc0 tx=0x7fb328009ee0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.173+0000 7fb33f83a740 1 -- 172.21.15.31:0/3422307466 shutdown_connections 2024-03-21T10:39:28.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.173+0000 7fb33f83a740 1 --2- 172.21.15.31:0/3422307466 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x556fcec1a5c0 0x556fcec1c9b0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.173+0000 7fb33f83a740 1 --2- 172.21.15.31:0/3422307466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556fcebadde0 0x556fcebb3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.173+0000 7fb33f83a740 1 --2- 172.21.15.31:0/3422307466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556fceba5260 0x556fcebb5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.173+0000 7fb317400640 0 client.25873 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:39:28.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.173+0000 7fb33f83a740 1 --2- 172.21.15.31:0/3422307466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556fceba48b0 0x556fcebb5910 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.173+0000 7fb33f83a740 1 -- 172.21.15.31:0/3422307466 >> 172.21.15.31:0/3422307466 conn(0x556fceb9ef80 msgr2=0x556fceba5ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:28.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.173+0000 7fb33f83a740 1 -- 172.21.15.31:0/3422307466 shutdown_connections 2024-03-21T10:39:28.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.173+0000 7fb33f83a740 1 -- 172.21.15.31:0/3422307466 wait complete. 2024-03-21T10:39:28.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.173+0000 7fb33f83a740 20 client.25873 trim_cache size 0 max 16384 2024-03-21T10:39:28.175 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:39:28.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.174+0000 7fb337200640 1 -- 172.21.15.31:0/2121779042 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb31c0a31e0 msgr2=0x7fb31c0a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:28.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.174+0000 7fb337200640 1 --2- 172.21.15.31:0/2121779042 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb31c0a31e0 0x7fb31c0a56a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb330005c40 tx=0x7fb330009760 comp rx=0 tx=0).stop 2024-03-21T10:39:28.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.174+0000 7fb337200640 1 -- 172.21.15.31:0/2121779042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb338101e80 msgr2=0x7fb338073ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:28.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.174+0000 7fb337200640 1 --2- 172.21.15.31:0/2121779042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb338101e80 0x7fb338073ee0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb324036040 tx=0x7fb324004340 comp rx=0 tx=0).stop 2024-03-21T10:39:28.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.174+0000 7fb337200640 1 -- 172.21.15.31:0/2121779042 shutdown_connections 2024-03-21T10:39:28.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.174+0000 7fb337200640 1 --2- 172.21.15.31:0/2121779042 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb31c0a31e0 0x7fb31c0a56a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.174+0000 7fb337200640 1 --2- 172.21.15.31:0/2121779042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb338107580 0x7fb338074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.174+0000 7fb337200640 1 --2- 172.21.15.31:0/2121779042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb338101e80 0x7fb338073ee0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.174+0000 7fb337200640 1 --2- 172.21.15.31:0/2121779042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3381014c0 0x7fb3380739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.174+0000 7fb337200640 1 -- 172.21.15.31:0/2121779042 >> 172.21.15.31:0/2121779042 conn(0x7fb3380fcfb0 msgr2=0x7fb3380fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:28.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.174+0000 7fb337200640 1 -- 172.21.15.31:0/2121779042 shutdown_connections 2024-03-21T10:39:28.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.174+0000 7fb337200640 1 -- 172.21.15.31:0/2121779042 wait complete. 2024-03-21T10:39:28.189 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:39:28.190 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-21T10:39:28.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.397+0000 7fcc0ae00640 1 -- 172.21.15.31:0/1458125547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc041096a0 msgr2=0x7fcc04109a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:28.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.397+0000 7fcc0ae00640 1 --2- 172.21.15.31:0/1458125547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc041096a0 0x7fcc04109a80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009ca0 tx=0x7fcbf802c100 comp rx=0 tx=0).stop 2024-03-21T10:39:28.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.398+0000 7fcc0ae00640 1 -- 172.21.15.31:0/1458125547 shutdown_connections 2024-03-21T10:39:28.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.398+0000 7fcc0ae00640 1 --2- 172.21.15.31:0/1458125547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc04104050 0x7fcc04110560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.398+0000 7fcc0ae00640 1 --2- 172.21.15.31:0/1458125547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc04103690 0x7fcc04103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.398+0000 7fcc0ae00640 1 --2- 172.21.15.31:0/1458125547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc041096a0 0x7fcc04109a80 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.398+0000 7fcc0ae00640 1 -- 172.21.15.31:0/1458125547 >> 172.21.15.31:0/1458125547 conn(0x7fcc040ff140 msgr2=0x7fcc04101560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:28.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.398+0000 7fcc0ae00640 1 -- 172.21.15.31:0/1458125547 shutdown_connections 2024-03-21T10:39:28.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.398+0000 7fcc0ae00640 1 -- 172.21.15.31:0/1458125547 wait complete. 2024-03-21T10:39:28.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.399+0000 7fcc0ae00640 1 Processor -- start 2024-03-21T10:39:28.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.399+0000 7fcc0ae00640 1 -- start start 2024-03-21T10:39:28.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.399+0000 7fcc0ae00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc04103690 0x7fcc04199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.399+0000 7fcc0ae00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc04104050 0x7fcc0419a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.399+0000 7fcc0ae00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc041096a0 0x7fcc0419e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.399+0000 7fcc0ae00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc04113a70 con 0x7fcc04103690 2024-03-21T10:39:28.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.399+0000 7fcc0ae00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc041138f0 con 0x7fcc041096a0 2024-03-21T10:39:28.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.399+0000 7fcc0ae00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc04113bf0 con 0x7fcc04104050 2024-03-21T10:39:28.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.400+0000 7fcc09000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc04104050 0x7fcc0419a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.400+0000 7fcc09a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc04103690 0x7fcc04199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.400+0000 7fcc09000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc04104050 0x7fcc0419a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:47138/0 (socket says 172.21.15.31:47138) 2024-03-21T10:39:28.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.400+0000 7fcc09000640 1 -- 172.21.15.31:0/1246763047 learned_addr learned my addr 172.21.15.31:0/1246763047 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:28.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.400+0000 7fcc0a400640 1 --2- 172.21.15.31:0/1246763047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc041096a0 0x7fcc0419e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.400+0000 7fcc09000640 1 -- 172.21.15.31:0/1246763047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc041096a0 msgr2=0x7fcc0419e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:28.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.400+0000 7fcc09000640 1 --2- 172.21.15.31:0/1246763047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc041096a0 0x7fcc0419e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.400+0000 7fcc09000640 1 -- 172.21.15.31:0/1246763047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc04103690 msgr2=0x7fcc04199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:28.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.400+0000 7fcc09000640 1 --2- 172.21.15.31:0/1246763047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc04103690 0x7fcc04199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.400+0000 7fcc09000640 1 -- 172.21.15.31:0/1246763047 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0419eef0 con 0x7fcc04104050 2024-03-21T10:39:28.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.400+0000 7fcc09000640 1 --2- 172.21.15.31:0/1246763047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc04104050 0x7fcc0419a490 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcbf000c920 tx=0x7fcbf000cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:28.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.401+0000 7fcc02000640 1 -- 172.21.15.31:0/1246763047 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf0007c70 con 0x7fcc04104050 2024-03-21T10:39:28.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.401+0000 7fcc0ae00640 1 -- 172.21.15.31:0/1246763047 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc0419f1e0 con 0x7fcc04104050 2024-03-21T10:39:28.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.401+0000 7fcc02000640 1 -- 172.21.15.31:0/1246763047 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbf0007e10 con 0x7fcc04104050 2024-03-21T10:39:28.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.401+0000 7fcc02000640 1 -- 172.21.15.31:0/1246763047 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf0018890 con 0x7fcc04104050 2024-03-21T10:39:28.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.401+0000 7fcc0ae00640 1 -- 172.21.15.31:0/1246763047 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc041a6a70 con 0x7fcc04104050 2024-03-21T10:39:28.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.405+0000 7fcc0f202740 1 Processor -- start 2024-03-21T10:39:28.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.405+0000 7fcc0f202740 1 -- start start 2024-03-21T10:39:28.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.406+0000 7fcc0f202740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4feac8b0 0x556c4feacc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.406+0000 7fcc0f202740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c4fead260 0x556c4feb5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.406+0000 7fcc0f202740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c4feb5de0 0x556c4feb8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.406+0000 7fcc0f202740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c4fe102c0 con 0x556c4feb5de0 2024-03-21T10:39:28.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.406+0000 7fcc0f202740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c4fdb8c90 con 0x556c4feac8b0 2024-03-21T10:39:28.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.406+0000 7fcc0f202740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c4fe0c360 con 0x556c4fead260 2024-03-21T10:39:28.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.406+0000 7fcc0a400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4feac8b0 0x556c4feacc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.406+0000 7fcc09a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c4fead260 0x556c4feb5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.406+0000 7fcc0a400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4feac8b0 0x556c4feacc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49464/0 (socket says 172.21.15.31:49464) 2024-03-21T10:39:28.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.406+0000 7fcc0a400640 1 -- 172.21.15.31:0/3935798621 learned_addr learned my addr 172.21.15.31:0/3935798621 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:28.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.406+0000 7fcc02000640 1 -- 172.21.15.31:0/1246763047 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) v1 ==== 138087+0+0 (secure 0 0 0) 0x7fcbf0018a70 con 0x7fcc04104050 2024-03-21T10:39:28.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.407+0000 7fcc09a00640 1 -- 172.21.15.31:0/3935798621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4feac8b0 msgr2=0x556c4feacc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:28.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.407+0000 7fcc09a00640 1 --2- 172.21.15.31:0/3935798621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4feac8b0 0x556c4feacc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.407+0000 7fcc09a00640 1 -- 172.21.15.31:0/3935798621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c4feb5de0 msgr2=0x556c4feb8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:28.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.407+0000 7fcc09a00640 1 --2- 172.21.15.31:0/3935798621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c4feb5de0 0x556c4feb8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.407+0000 7fcc09a00640 1 -- 172.21.15.31:0/3935798621 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556c4fddd080 con 0x556c4fead260 2024-03-21T10:39:28.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.407+0000 7fcc0a400640 1 --2- 172.21.15.31:0/3935798621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4feac8b0 0x556c4feacc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:39:28.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.407+0000 7fcc09a00640 1 --2- 172.21.15.31:0/3935798621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c4fead260 0x556c4feb5810 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcbf802c640 tx=0x7fcbf8009910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3f38c2d0523be2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:28.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.407+0000 7fcbf6a00640 1 -- 172.21.15.31:0/3935798621 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8045020 con 0x556c4fead260 2024-03-21T10:39:28.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.407+0000 7fcbf6a00640 1 -- 172.21.15.31:0/3935798621 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbf803a040 con 0x556c4fead260 2024-03-21T10:39:28.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.407+0000 7fcbf6a00640 1 -- 172.21.15.31:0/3935798621 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8039040 con 0x556c4fead260 2024-03-21T10:39:28.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.408+0000 7fcc02000640 1 --2- 172.21.15.31:0/1246763047 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fcbec0a3230 0x7fcbec0a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.408+0000 7fcc0f202740 1 -- 172.21.15.31:0/3935798621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c4fead260 msgr2=0x556c4feb5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:28.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.408+0000 7fcc0f202740 1 --2- 172.21.15.31:0/3935798621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c4fead260 0x556c4feb5810 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcbf802c640 tx=0x7fcbf8009910 comp rx=0 tx=0).stop 2024-03-21T10:39:28.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.408+0000 7fcc02000640 1 -- 172.21.15.31:0/1246763047 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fcbf00cfe30 con 0x7fcc04104050 2024-03-21T10:39:28.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.408+0000 7fcc0f202740 1 -- 172.21.15.31:0/3935798621 shutdown_connections 2024-03-21T10:39:28.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.408+0000 7fcc0f202740 1 --2- 172.21.15.31:0/3935798621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c4feb5de0 0x556c4feb8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.408+0000 7fcc0f202740 1 --2- 172.21.15.31:0/3935798621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c4fead260 0x556c4feb5810 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.408+0000 7fcc0f202740 1 --2- 172.21.15.31:0/3935798621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4feac8b0 0x556c4feacc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.408+0000 7fcc0f202740 1 -- 172.21.15.31:0/3935798621 >> 172.21.15.31:0/3935798621 conn(0x556c4fea6f80 msgr2=0x556c4fea7360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:28.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.408+0000 7fcc0a400640 1 --2- 172.21.15.31:0/1246763047 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fcbec0a3230 0x7fcbec0a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.409+0000 7fcc0f202740 1 -- 172.21.15.31:0/3935798621 shutdown_connections 2024-03-21T10:39:28.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.409+0000 7fcc0f202740 1 -- 172.21.15.31:0/3935798621 wait complete. 2024-03-21T10:39:28.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.409+0000 7fcc0f202740 1 -- start start 2024-03-21T10:39:28.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.409+0000 7fcc0a400640 1 --2- 172.21.15.31:0/1246763047 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fcbec0a3230 0x7fcbec0a56f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc009d20 tx=0x7fcbfc005f90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:28.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.409+0000 7fcc0f202740 1 Processor -- start 2024-03-21T10:39:28.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.410+0000 7fcc0f202740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c4feac8b0 0x556c4feb5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.410+0000 7fcc0f202740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4fead260 0x556c4feb2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.410+0000 7fcc0f202740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c4feb5de0 0x556c4feb30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.410+0000 7fcc0f202740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c4feb8cf0 con 0x556c4feb5de0 2024-03-21T10:39:28.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.410+0000 7fcc0f202740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c4fdcc0d0 con 0x556c4fead260 2024-03-21T10:39:28.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.410+0000 7fcc0f202740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c4feb8e70 con 0x556c4feac8b0 2024-03-21T10:39:28.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.410+0000 7fcc0f202740 10 client.0 _ll_register_callbacks cb 0x556c4fdd0e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:39:28.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.410+0000 7fcc09000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4fead260 0x556c4feb2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.410+0000 7fcc09000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4fead260 0x556c4feb2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49470/0 (socket says 172.21.15.31:49470) 2024-03-21T10:39:28.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.410+0000 7fcc09000640 1 -- 172.21.15.31:0/1822475974 learned_addr learned my addr 172.21.15.31:0/1822475974 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:28.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.410+0000 7fcc09a00640 1 --2- 172.21.15.31:0/1822475974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c4feac8b0 0x556c4feb5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.410+0000 7fcc0a400640 1 --2- 172.21.15.31:0/1822475974 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c4feb5de0 0x556c4feb30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.410+0000 7fcc09000640 1 -- 172.21.15.31:0/1822475974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c4feac8b0 msgr2=0x556c4feb5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:28.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.410+0000 7fcc09000640 1 --2- 172.21.15.31:0/1822475974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c4feac8b0 0x556c4feb5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.410+0000 7fcc09000640 1 -- 172.21.15.31:0/1822475974 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c4feb5de0 msgr2=0x556c4feb30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:28.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.410+0000 7fcc09000640 1 --2- 172.21.15.31:0/1822475974 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c4feb5de0 0x556c4feb30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.410+0000 7fcc09000640 1 -- 172.21.15.31:0/1822475974 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556c4fb243f0 con 0x556c4fead260 2024-03-21T10:39:28.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.410+0000 7fcc09a00640 1 --2- 172.21.15.31:0/1822475974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c4feac8b0 0x556c4feb5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:39:28.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.411+0000 7fcc09000640 1 --2- 172.21.15.31:0/1822475974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4fead260 0x556c4feb2ae0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0007590 tx=0x7fcbf009c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=947eca904e50e322 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:28.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.411+0000 7fcbeb400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:39:28.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.411+0000 7fcbeb400640 1 -- 172.21.15.31:0/1822475974 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf0086040 con 0x556c4fead260 2024-03-21T10:39:28.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.411+0000 7fcbeb400640 1 -- 172.21.15.31:0/1822475974 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fcbf00a41b0 con 0x556c4fead260 2024-03-21T10:39:28.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.411+0000 7fcc0f202740 1 -- 172.21.15.31:0/1822475974 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcbf803a040 con 0x556c4fead260 2024-03-21T10:39:28.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.411+0000 7fcbeb400640 1 -- 172.21.15.31:0/1822475974 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcbf00a4640 con 0x556c4fead260 2024-03-21T10:39:28.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.411+0000 7fcc0f202740 10 client.25901 fetch_fsmap learned FSMap version 24 2024-03-21T10:39:28.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.411+0000 7fcc0f202740 1 -- 172.21.15.31:0/1822475974 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x556c4fbee7d0 con 0x556c4fead260 2024-03-21T10:39:28.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.412+0000 7fcbeb400640 1 -- 172.21.15.31:0/1822475974 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbf803a040 con 0x556c4fead260 2024-03-21T10:39:28.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.412+0000 7fcbeb400640 1 -- 172.21.15.31:0/1822475974 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fcbf009b110 con 0x556c4fead260 2024-03-21T10:39:28.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.412+0000 7fcc0f202740 10 client.25901 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:39:28.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.412+0000 7fcc0f202740 10 client.25901 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:39:28.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.412+0000 7fcc0f202740 20 client.25901 populate_metadata read hostname 'smithi031' 2024-03-21T10:39:28.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.412+0000 7fcc0f202740 1 --2- 172.21.15.31:0/1822475974 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556c4fec5c40 0x556c4fec8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.412+0000 7fcc0f202740 4 client.25901 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:39:28.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.412+0000 7fcc0f202740 1 -- 172.21.15.31:0/1822475974 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556c4fe1d800 con 0x556c4fec5c40 2024-03-21T10:39:28.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.412+0000 7fcc09a00640 1 --2- 172.21.15.31:0/1822475974 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556c4fec5c40 0x556c4fec8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.413+0000 7fcbeb400640 10 client.25901 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:28.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.413+0000 7fcc09a00640 1 --2- 172.21.15.31:0/1822475974 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556c4fec5c40 0x556c4fec8030 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4dd4011dfc02fdb server_cookie=fe0b41d031f28e7e in_seq=0 out_seq=0 2024-03-21T10:39:28.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.413+0000 7fcbeb400640 1 -- 172.21.15.31:0/1822475974 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556c4fe1d800 con 0x556c4fec5c40 2024-03-21T10:39:28.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.413+0000 7fcbeb400640 10 client.25901 handle_command_reply: tid=0 2024-03-21T10:39:28.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.413+0000 7fcc0f202740 1 client.25901 shutdown 2024-03-21T10:39:28.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.414+0000 7fcc0f202740 1 -- 172.21.15.31:0/1822475974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4fead260 msgr2=0x556c4feb2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:28.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.414+0000 7fcc0f202740 1 --2- 172.21.15.31:0/1822475974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4fead260 0x556c4feb2ae0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0007590 tx=0x7fcbf009c3f0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.414+0000 7fcc0f202740 1 -- 172.21.15.31:0/1822475974 shutdown_connections 2024-03-21T10:39:28.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.414+0000 7fcc0f202740 1 --2- 172.21.15.31:0/1822475974 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556c4fec5c40 0x556c4fec8030 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.414+0000 7fcc0f202740 1 --2- 172.21.15.31:0/1822475974 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c4feb5de0 0x556c4feb30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.414+0000 7fcc0f202740 1 --2- 172.21.15.31:0/1822475974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4fead260 0x556c4feb2ae0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.414+0000 7fcc0f202740 1 --2- 172.21.15.31:0/1822475974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c4feac8b0 0x556c4feb5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.414+0000 7fcc0f202740 1 -- 172.21.15.31:0/1822475974 >> 172.21.15.31:0/1822475974 conn(0x556c4fea6f80 msgr2=0x556c4fea8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:28.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.414+0000 7fcbeb400640 0 client.25901 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:28.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.414+0000 7fcc0f202740 1 -- 172.21.15.31:0/1822475974 shutdown_connections 2024-03-21T10:39:28.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.414+0000 7fcc0f202740 1 -- 172.21.15.31:0/1822475974 wait complete. 2024-03-21T10:39:28.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.414+0000 7fcc0f202740 20 client.25901 trim_cache size 0 max 16384 2024-03-21T10:39:28.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.431+0000 7fcc0f202740 1 Processor -- start 2024-03-21T10:39:28.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.431+0000 7fcc0f202740 1 -- start start 2024-03-21T10:39:28.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.431+0000 7fcc0f202740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4feac8b0 0x556c4feb5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.431+0000 7fcc0f202740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c4fead260 0x556c4feb5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.431+0000 7fcc0f202740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c4feb5de0 0x556c4feafd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.431+0000 7fcc0f202740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc041b2cc0 con 0x556c4feb5de0 2024-03-21T10:39:28.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.431+0000 7fcc0f202740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc041b07c0 con 0x556c4feac8b0 2024-03-21T10:39:28.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.431+0000 7fcc0f202740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc041ae860 con 0x556c4fead260 2024-03-21T10:39:28.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.431+0000 7fcc09a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4feac8b0 0x556c4feb5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.431+0000 7fcc09a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4feac8b0 0x556c4feb5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49474/0 (socket says 172.21.15.31:49474) 2024-03-21T10:39:28.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.431+0000 7fcc09a00640 1 -- 172.21.15.31:0/4184377233 learned_addr learned my addr 172.21.15.31:0/4184377233 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:28.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.431+0000 7fcc09000640 1 --2- 172.21.15.31:0/4184377233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c4fead260 0x556c4feb5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.431+0000 7fcc0a400640 1 --2- 172.21.15.31:0/4184377233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c4feb5de0 0x556c4feafd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.432+0000 7fcc09a00640 1 -- 172.21.15.31:0/4184377233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c4fead260 msgr2=0x556c4feb5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:28.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.432+0000 7fcc09a00640 1 --2- 172.21.15.31:0/4184377233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c4fead260 0x556c4feb5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.432+0000 7fcc09a00640 1 -- 172.21.15.31:0/4184377233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c4feb5de0 msgr2=0x556c4feafd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:28.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.432+0000 7fcc09a00640 1 --2- 172.21.15.31:0/4184377233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c4feb5de0 0x556c4feafd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.432+0000 7fcc09a00640 1 -- 172.21.15.31:0/4184377233 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556c4fec8660 con 0x556c4feac8b0 2024-03-21T10:39:28.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.432+0000 7fcc09a00640 1 --2- 172.21.15.31:0/4184377233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4feac8b0 0x556c4feb5210 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fcbf802c140 tx=0x7fcbf8005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f40699f2eab3a5b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:28.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.432+0000 7fcbf6000640 1 -- 172.21.15.31:0/4184377233 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8045040 con 0x556c4feac8b0 2024-03-21T10:39:28.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.432+0000 7fcbf6000640 1 -- 172.21.15.31:0/4184377233 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbf8003970 con 0x556c4feac8b0 2024-03-21T10:39:28.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.432+0000 7fcbf6000640 1 -- 172.21.15.31:0/4184377233 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8039040 con 0x556c4feac8b0 2024-03-21T10:39:28.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.433+0000 7fcc0f202740 1 -- 172.21.15.31:0/4184377233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4feac8b0 msgr2=0x556c4feb5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:28.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.433+0000 7fcc0f202740 1 --2- 172.21.15.31:0/4184377233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4feac8b0 0x556c4feb5210 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fcbf802c140 tx=0x7fcbf8005ef0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.433+0000 7fcc0f202740 1 -- 172.21.15.31:0/4184377233 shutdown_connections 2024-03-21T10:39:28.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.433+0000 7fcc0f202740 1 --2- 172.21.15.31:0/4184377233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c4feb5de0 0x556c4feafd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.433+0000 7fcc0f202740 1 --2- 172.21.15.31:0/4184377233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c4fead260 0x556c4feb5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.433+0000 7fcc0f202740 1 --2- 172.21.15.31:0/4184377233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4feac8b0 0x556c4feb5210 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.433+0000 7fcc0f202740 1 -- 172.21.15.31:0/4184377233 >> 172.21.15.31:0/4184377233 conn(0x556c4fea6f80 msgr2=0x556c4fea8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:28.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.433+0000 7fcc0f202740 1 -- 172.21.15.31:0/4184377233 shutdown_connections 2024-03-21T10:39:28.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.433+0000 7fcc0f202740 1 -- 172.21.15.31:0/4184377233 wait complete. 2024-03-21T10:39:28.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.434+0000 7fcc0f202740 1 -- start start 2024-03-21T10:39:28.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.434+0000 7fcc0f202740 1 Processor -- start 2024-03-21T10:39:28.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.435+0000 7fcc0f202740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c4feac8b0 0x556c4febdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.435+0000 7fcc0f202740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4fead260 0x556c4febc000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.435+0000 7fcc0f202740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c4feb5de0 0x556c4febc540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.435+0000 7fcc0f202740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc041aee60 con 0x556c4feac8b0 2024-03-21T10:39:28.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.435+0000 7fcc0f202740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc041b0160 con 0x556c4fead260 2024-03-21T10:39:28.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.435+0000 7fcc0f202740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc041af4a0 con 0x556c4feb5de0 2024-03-21T10:39:28.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.435+0000 7fcc0f202740 10 client.0 _ll_register_callbacks cb 0x7fcbf000cf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:39:28.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.435+0000 7fcc09000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4fead260 0x556c4febc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.435+0000 7fcc09000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4fead260 0x556c4febc000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49490/0 (socket says 172.21.15.31:49490) 2024-03-21T10:39:28.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.435+0000 7fcc09000640 1 -- 172.21.15.31:0/653061231 learned_addr learned my addr 172.21.15.31:0/653061231 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:28.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.435+0000 7fcc09a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c4feac8b0 0x556c4febdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.435+0000 7fcc0a400640 1 --2- 172.21.15.31:0/653061231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c4feb5de0 0x556c4febc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.435+0000 7fcc09000640 1 -- 172.21.15.31:0/653061231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c4feb5de0 msgr2=0x556c4febc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:28.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.435+0000 7fcc09000640 1 --2- 172.21.15.31:0/653061231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c4feb5de0 0x556c4febc540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.435+0000 7fcc09000640 1 -- 172.21.15.31:0/653061231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c4feac8b0 msgr2=0x556c4febdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:28.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.435+0000 7fcc09000640 1 --2- 172.21.15.31:0/653061231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c4feac8b0 0x556c4febdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.435+0000 7fcc09000640 1 -- 172.21.15.31:0/653061231 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556c4febcb00 con 0x556c4fead260 2024-03-21T10:39:28.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.436+0000 7fcc09000640 1 --2- 172.21.15.31:0/653061231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4fead260 0x556c4febc000 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fcbf009b520 tx=0x7fcbf0000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bbb805c56e4cb1d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:28.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.436+0000 7fcbeb400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:39:28.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.436+0000 7fcbeb400640 1 -- 172.21.15.31:0/653061231 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf0086040 con 0x556c4fead260 2024-03-21T10:39:28.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.436+0000 7fcc0f202740 1 -- 172.21.15.31:0/653061231 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcbf8003970 con 0x556c4fead260 2024-03-21T10:39:28.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.436+0000 7fcbeb400640 1 -- 172.21.15.31:0/653061231 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fcbf00a41b0 con 0x556c4fead260 2024-03-21T10:39:28.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.436+0000 7fcbeb400640 1 -- 172.21.15.31:0/653061231 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcbf009b290 con 0x556c4fead260 2024-03-21T10:39:28.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.436+0000 7fcc0f202740 10 client.25907 fetch_fsmap learned FSMap version 24 2024-03-21T10:39:28.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.436+0000 7fcc0f202740 1 -- 172.21.15.31:0/653061231 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x556c4ff222a0 con 0x556c4fead260 2024-03-21T10:39:28.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.436+0000 7fcbeb400640 1 -- 172.21.15.31:0/653061231 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbf8003970 con 0x556c4fead260 2024-03-21T10:39:28.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.436+0000 7fcbeb400640 1 -- 172.21.15.31:0/653061231 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fcbf00a0f10 con 0x556c4fead260 2024-03-21T10:39:28.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.437+0000 7fcc0f202740 10 client.25907 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:39:28.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.437+0000 7fcc0f202740 10 client.25907 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:39:28.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.437+0000 7fcc0f202740 20 client.25907 populate_metadata read hostname 'smithi031' 2024-03-21T10:39:28.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.437+0000 7fcc0f202740 1 --2- 172.21.15.31:0/653061231 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556c4ff22970 0x556c4ff24d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:28.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.437+0000 7fcc0f202740 4 client.25907 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:39:28.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.437+0000 7fcc0f202740 1 -- 172.21.15.31:0/653061231 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556c4fe17b50 con 0x556c4ff22970 2024-03-21T10:39:28.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.437+0000 7fcc09a00640 1 --2- 172.21.15.31:0/653061231 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556c4ff22970 0x556c4ff24d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:28.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.437+0000 7fcc09a00640 1 --2- 172.21.15.31:0/653061231 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556c4ff22970 0x556c4ff24d90 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83876a365080a811 server_cookie=c7161025d6a7380a in_seq=0 out_seq=0 2024-03-21T10:39:28.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.437+0000 7fcbeb400640 10 client.25907 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:28.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.437+0000 7fcbeb400640 1 -- 172.21.15.31:0/653061231 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556c4fe17b50 con 0x556c4ff22970 2024-03-21T10:39:28.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.437+0000 7fcbeb400640 10 client.25907 handle_command_reply: tid=0 2024-03-21T10:39:28.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.437+0000 7fcc0f202740 1 client.25907 shutdown 2024-03-21T10:39:28.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.438+0000 7fcc0f202740 1 -- 172.21.15.31:0/653061231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4fead260 msgr2=0x556c4febc000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:28.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.438+0000 7fcc0f202740 1 --2- 172.21.15.31:0/653061231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4fead260 0x556c4febc000 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fcbf009b520 tx=0x7fcbf0000bc0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.438+0000 7fcc0f202740 1 -- 172.21.15.31:0/653061231 shutdown_connections 2024-03-21T10:39:28.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.438+0000 7fcc0f202740 1 --2- 172.21.15.31:0/653061231 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556c4ff22970 0x556c4ff24d90 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.438+0000 7fcc0f202740 1 --2- 172.21.15.31:0/653061231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556c4feb5de0 0x556c4febc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.438+0000 7fcc0f202740 1 --2- 172.21.15.31:0/653061231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c4fead260 0x556c4febc000 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.438+0000 7fcc0f202740 1 --2- 172.21.15.31:0/653061231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c4feac8b0 0x556c4febdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.438+0000 7fcbeb400640 0 client.25907 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:28.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.438+0000 7fcc0f202740 1 -- 172.21.15.31:0/653061231 >> 172.21.15.31:0/653061231 conn(0x556c4fea6f80 msgr2=0x556c4fea58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:28.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.438+0000 7fcc0f202740 1 -- 172.21.15.31:0/653061231 shutdown_connections 2024-03-21T10:39:28.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.438+0000 7fcc0f202740 1 -- 172.21.15.31:0/653061231 wait complete. 2024-03-21T10:39:28.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.438+0000 7fcc0f202740 20 client.25907 trim_cache size 0 max 16384 2024-03-21T10:39:28.439 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:39:28.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.439+0000 7fcc0ae00640 1 -- 172.21.15.31:0/1246763047 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fcbec0a3230 msgr2=0x7fcbec0a56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:28.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.439+0000 7fcc0ae00640 1 --2- 172.21.15.31:0/1246763047 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fcbec0a3230 0x7fcbec0a56f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc009d20 tx=0x7fcbfc005f90 comp rx=0 tx=0).stop 2024-03-21T10:39:28.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.439+0000 7fcc0ae00640 1 -- 172.21.15.31:0/1246763047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc04104050 msgr2=0x7fcc0419a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:28.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.439+0000 7fcc0ae00640 1 --2- 172.21.15.31:0/1246763047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc04104050 0x7fcc0419a490 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcbf000c920 tx=0x7fcbf000cde0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.439+0000 7fcc0ae00640 1 -- 172.21.15.31:0/1246763047 shutdown_connections 2024-03-21T10:39:28.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.439+0000 7fcc0ae00640 1 --2- 172.21.15.31:0/1246763047 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fcbec0a3230 0x7fcbec0a56f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.439+0000 7fcc0ae00640 1 --2- 172.21.15.31:0/1246763047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc041096a0 0x7fcc0419e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.439+0000 7fcc0ae00640 1 --2- 172.21.15.31:0/1246763047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc04104050 0x7fcc0419a490 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.439+0000 7fcc0ae00640 1 --2- 172.21.15.31:0/1246763047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc04103690 0x7fcc04199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:28.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.439+0000 7fcc0ae00640 1 -- 172.21.15.31:0/1246763047 >> 172.21.15.31:0/1246763047 conn(0x7fcc040ff140 msgr2=0x7fcc040ff7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:28.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.439+0000 7fcc0ae00640 1 -- 172.21.15.31:0/1246763047 shutdown_connections 2024-03-21T10:39:28.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:28.439+0000 7fcc0ae00640 1 -- 172.21.15.31:0/1246763047 wait complete. 2024-03-21T10:39:28.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:28 smithi031 ceph-mon[31673]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-21T10:39:28.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:28 smithi040 ceph-mon[36154]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-21T10:39:29.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:28 smithi070 ceph-mon[36330]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-21T10:39:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.227+0000 7f54bc800640 1 -- 172.21.15.31:0/2651084866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54b40764f0 msgr2=0x7f54b4111fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.227+0000 7f54bc800640 1 --2- 172.21.15.31:0/2651084866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54b40764f0 0x7f54b4111fd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f54a4009890 tx=0x7f54a402bb70 comp rx=0 tx=0).stop 2024-03-21T10:39:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.227+0000 7f54bc800640 1 -- 172.21.15.31:0/2651084866 shutdown_connections 2024-03-21T10:39:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.227+0000 7f54bc800640 1 --2- 172.21.15.31:0/2651084866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54b4112510 0x7f54b4114900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.227+0000 7f54bc800640 1 --2- 172.21.15.31:0/2651084866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54b40764f0 0x7f54b4111fd0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.227+0000 7f54bc800640 1 --2- 172.21.15.31:0/2651084866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54b4075bd0 0x7f54b4075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.227+0000 7f54bc800640 1 -- 172.21.15.31:0/2651084866 >> 172.21.15.31:0/2651084866 conn(0x7f54b40ff320 msgr2=0x7f54b4101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:29.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.227+0000 7f54bc800640 1 -- 172.21.15.31:0/2651084866 shutdown_connections 2024-03-21T10:39:29.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.227+0000 7f54bc800640 1 -- 172.21.15.31:0/2651084866 wait complete. 2024-03-21T10:39:29.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.229+0000 7f54bc800640 1 Processor -- start 2024-03-21T10:39:29.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.229+0000 7f54bc800640 1 -- start start 2024-03-21T10:39:29.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.229+0000 7f54bc800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54b4075bd0 0x7f54b419e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.229+0000 7f54bc800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54b40764f0 0x7f54b419eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.229+0000 7f54bc800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54b4112510 0x7f54b41a2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.229+0000 7f54bc800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b4118070 con 0x7f54b40764f0 2024-03-21T10:39:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.229+0000 7f54bc800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b4117ef0 con 0x7f54b4075bd0 2024-03-21T10:39:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.229+0000 7f54bc800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b41181f0 con 0x7f54b4112510 2024-03-21T10:39:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.230+0000 7f54b9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54b4075bd0 0x7f54b419e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.230+0000 7f54b9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54b4075bd0 0x7f54b419e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49496/0 (socket says 172.21.15.31:49496) 2024-03-21T10:39:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.230+0000 7f54ba800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54b4112510 0x7f54b41a2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.230+0000 7f54b9e00640 1 -- 172.21.15.31:0/109670354 learned_addr learned my addr 172.21.15.31:0/109670354 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.230+0000 7f54b9400640 1 --2- 172.21.15.31:0/109670354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54b40764f0 0x7f54b419eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.230+0000 7f54b9e00640 1 -- 172.21.15.31:0/109670354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54b4112510 msgr2=0x7f54b41a2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.230+0000 7f54b9e00640 1 --2- 172.21.15.31:0/109670354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54b4112510 0x7f54b41a2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.230+0000 7f54b9e00640 1 -- 172.21.15.31:0/109670354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54b40764f0 msgr2=0x7f54b419eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:29.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.230+0000 7f54b9e00640 1 --2- 172.21.15.31:0/109670354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54b40764f0 0x7f54b419eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.230+0000 7f54b9e00640 1 -- 172.21.15.31:0/109670354 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b41a35d0 con 0x7f54b4075bd0 2024-03-21T10:39:29.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.230+0000 7f54b9e00640 1 --2- 172.21.15.31:0/109670354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54b4075bd0 0x7f54b419e630 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f54a800e9f0 tx=0x7f54a800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.231+0000 7f54aea00640 1 -- 172.21.15.31:0/109670354 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a800ccc0 con 0x7f54b4075bd0 2024-03-21T10:39:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.231+0000 7f54aea00640 1 -- 172.21.15.31:0/109670354 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f54a8005500 con 0x7f54b4075bd0 2024-03-21T10:39:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.231+0000 7f54bc800640 1 -- 172.21.15.31:0/109670354 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54b41a38c0 con 0x7f54b4075bd0 2024-03-21T10:39:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.231+0000 7f54aea00640 1 -- 172.21.15.31:0/109670354 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a8010690 con 0x7f54b4075bd0 2024-03-21T10:39:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.231+0000 7f54bc800640 1 -- 172.21.15.31:0/109670354 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54b41ab150 con 0x7f54b4075bd0 2024-03-21T10:39:29.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.234+0000 7f54aea00640 1 -- 172.21.15.31:0/109670354 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) v1 ==== 138087+0+0 (secure 0 0 0) 0x7f54a80046c0 con 0x7f54b4075bd0 2024-03-21T10:39:29.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.235+0000 7f54aea00640 1 --2- 172.21.15.31:0/109670354 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f549c0a3220 0x7f549c0a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.236+0000 7f54aea00640 1 -- 172.21.15.31:0/109670354 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f54a80cfff0 con 0x7f54b4075bd0 2024-03-21T10:39:29.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.236+0000 7f54bf579740 1 Processor -- start 2024-03-21T10:39:29.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.236+0000 7f54b9400640 1 --2- 172.21.15.31:0/109670354 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f549c0a3220 0x7f549c0a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.236+0000 7f54bf579740 1 -- start start 2024-03-21T10:39:29.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.237+0000 7f54bf579740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590515bcfd0 0x5590515bd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.237+0000 7f54bf579740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5590515bd8f0 0x5590515c6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.237+0000 7f54bf579740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590515c66c0 0x5590515c8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.237+0000 7f54bf579740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55905157f3b0 con 0x5590515c66c0 2024-03-21T10:39:29.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.237+0000 7f54bf579740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x559051521830 con 0x5590515bd8f0 2024-03-21T10:39:29.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.237+0000 7f54bf579740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590514ca5d0 con 0x5590515bcfd0 2024-03-21T10:39:29.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.237+0000 7f54b9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5590515bd8f0 0x5590515c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.237+0000 7f54b9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5590515bd8f0 0x5590515c6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49504/0 (socket says 172.21.15.31:49504) 2024-03-21T10:39:29.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.237+0000 7f54b9400640 1 -- 172.21.15.31:0/1184836442 learned_addr learned my addr 172.21.15.31:0/1184836442 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:29.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.237+0000 7f54ba800640 1 --2- 172.21.15.31:0/1184836442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590515c66c0 0x5590515c8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.237+0000 7f54b9400640 1 --2- 172.21.15.31:0/109670354 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f549c0a3220 0x7f549c0a56e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f54a4037040 tx=0x7f54a40056d0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:29.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.237+0000 7f54b9e00640 1 --2- 172.21.15.31:0/1184836442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590515bcfd0 0x5590515bd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.238+0000 7f54b9400640 1 -- 172.21.15.31:0/1184836442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590515bcfd0 msgr2=0x5590515bd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:29.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.238+0000 7f54b9400640 1 --2- 172.21.15.31:0/1184836442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590515bcfd0 0x5590515bd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.238+0000 7f54b9400640 1 -- 172.21.15.31:0/1184836442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590515c66c0 msgr2=0x5590515c8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:29.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.238+0000 7f54b9400640 1 --2- 172.21.15.31:0/1184836442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590515c66c0 0x5590515c8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.238+0000 7f54b9400640 1 -- 172.21.15.31:0/1184836442 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5590514e22c0 con 0x5590515bd8f0 2024-03-21T10:39:29.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.238+0000 7f54b9400640 1 --2- 172.21.15.31:0/1184836442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5590515bd8f0 0x5590515c6180 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f54a4036000 tx=0x7f54a403e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=425274135d3c911e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.238+0000 7f54a3000640 1 -- 172.21.15.31:0/1184836442 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a4039040 con 0x5590515bd8f0 2024-03-21T10:39:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.238+0000 7f54a3000640 1 -- 172.21.15.31:0/1184836442 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f54a404f070 con 0x5590515bd8f0 2024-03-21T10:39:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.238+0000 7f54a3000640 1 -- 172.21.15.31:0/1184836442 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a4032640 con 0x5590515bd8f0 2024-03-21T10:39:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.239+0000 7f54bf579740 1 -- 172.21.15.31:0/1184836442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5590515bd8f0 msgr2=0x5590515c6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.239+0000 7f54bf579740 1 --2- 172.21.15.31:0/1184836442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5590515bd8f0 0x5590515c6180 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f54a4036000 tx=0x7f54a403e000 comp rx=0 tx=0).stop 2024-03-21T10:39:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.239+0000 7f54bf579740 1 -- 172.21.15.31:0/1184836442 shutdown_connections 2024-03-21T10:39:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.239+0000 7f54bf579740 1 --2- 172.21.15.31:0/1184836442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590515c66c0 0x5590515c8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.239+0000 7f54bf579740 1 --2- 172.21.15.31:0/1184836442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5590515bd8f0 0x5590515c6180 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.239+0000 7f54bf579740 1 --2- 172.21.15.31:0/1184836442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590515bcfd0 0x5590515bd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.239+0000 7f54bf579740 1 -- 172.21.15.31:0/1184836442 >> 172.21.15.31:0/1184836442 conn(0x5590515b78f0 msgr2=0x5590515b7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.239+0000 7f54bf579740 1 -- 172.21.15.31:0/1184836442 shutdown_connections 2024-03-21T10:39:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.239+0000 7f54bf579740 1 -- 172.21.15.31:0/1184836442 wait complete. 2024-03-21T10:39:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.240+0000 7f54bf579740 1 -- start start 2024-03-21T10:39:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.240+0000 7f54bf579740 1 Processor -- start 2024-03-21T10:39:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.240+0000 7f54bf579740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590515bcfd0 0x5590515c3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.240+0000 7f54bf579740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590515bd8f0 0x5590515c3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.240+0000 7f54bf579740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5590515c66c0 0x5590515c41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.240+0000 7f54bf579740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590514dda20 con 0x5590515bcfd0 2024-03-21T10:39:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.240+0000 7f54bf579740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55905151d8d0 con 0x5590515c66c0 2024-03-21T10:39:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.240+0000 7f54bf579740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590515c9290 con 0x5590515bd8f0 2024-03-21T10:39:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.241+0000 7f54bf579740 10 client.0 _ll_register_callbacks cb 0x55905151a1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:39:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.241+0000 7f54b9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590515bcfd0 0x5590515c3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.241+0000 7f54b9400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590515bd8f0 0x5590515c3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.241+0000 7f54b9400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590515bd8f0 0x5590515c3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:47208/0 (socket says 172.21.15.31:47208) 2024-03-21T10:39:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.241+0000 7f54b9400640 1 -- 172.21.15.31:0/3614123962 learned_addr learned my addr 172.21.15.31:0/3614123962 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.241+0000 7f54ba800640 1 --2- 172.21.15.31:0/3614123962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5590515c66c0 0x5590515c41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.241+0000 7f54b9400640 1 -- 172.21.15.31:0/3614123962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5590515c66c0 msgr2=0x5590515c41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.241+0000 7f54b9400640 1 --2- 172.21.15.31:0/3614123962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5590515c66c0 0x5590515c41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.241+0000 7f54b9400640 1 -- 172.21.15.31:0/3614123962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590515bcfd0 msgr2=0x5590515c3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.241+0000 7f54b9400640 1 --2- 172.21.15.31:0/3614123962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590515bcfd0 0x5590515c3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.241+0000 7f54b9400640 1 -- 172.21.15.31:0/3614123962 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559051236210 con 0x5590515bd8f0 2024-03-21T10:39:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.241+0000 7f54b9400640 1 --2- 172.21.15.31:0/3614123962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590515bd8f0 0x5590515c3c80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f54a4005e00 tx=0x7f54a4005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a48290385da08b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.241+0000 7f5497400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:39:29.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.242+0000 7f5497400640 1 -- 172.21.15.31:0/3614123962 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a4057070 con 0x5590515bd8f0 2024-03-21T10:39:29.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.242+0000 7f54bf579740 1 -- 172.21.15.31:0/3614123962 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f54a404f070 con 0x5590515bd8f0 2024-03-21T10:39:29.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.242+0000 7f5497400640 1 -- 172.21.15.31:0/3614123962 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f54a402c710 con 0x5590515bd8f0 2024-03-21T10:39:29.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.242+0000 7f5497400640 1 -- 172.21.15.31:0/3614123962 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54a4039040 con 0x5590515bd8f0 2024-03-21T10:39:29.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.242+0000 7f54bf579740 10 client.25903 fetch_fsmap learned FSMap version 24 2024-03-21T10:39:29.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.242+0000 7f54bf579740 1 -- 172.21.15.31:0/3614123962 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x559051302620 con 0x5590515bd8f0 2024-03-21T10:39:29.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.242+0000 7f5497400640 1 -- 172.21.15.31:0/3614123962 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f54a404f070 con 0x5590515bd8f0 2024-03-21T10:39:29.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.243+0000 7f5497400640 1 -- 172.21.15.31:0/3614123962 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f54a40035f0 con 0x5590515bd8f0 2024-03-21T10:39:29.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.243+0000 7f54bf579740 10 client.25903 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:39:29.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.243+0000 7f54bf579740 10 client.25903 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:39:29.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.243+0000 7f54bf579740 20 client.25903 populate_metadata read hostname 'smithi031' 2024-03-21T10:39:29.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.243+0000 7f54bf579740 1 --2- 172.21.15.31:0/3614123962 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5590515d6580 0x5590515d8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.243+0000 7f54bf579740 4 client.25903 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:39:29.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.243+0000 7f54bf579740 1 -- 172.21.15.31:0/3614123962 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55905152ed70 con 0x5590515d6580 2024-03-21T10:39:29.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.243+0000 7f54b9e00640 1 --2- 172.21.15.31:0/3614123962 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5590515d6580 0x5590515d8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.243+0000 7f54b9e00640 1 --2- 172.21.15.31:0/3614123962 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5590515d6580 0x5590515d8970 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9440d816097f729 server_cookie=e95ffc8293f986be in_seq=0 out_seq=0 2024-03-21T10:39:29.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.243+0000 7f5497400640 10 client.25903 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:29.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.244+0000 7f5497400640 1 -- 172.21.15.31:0/3614123962 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55905152ed70 con 0x5590515d6580 2024-03-21T10:39:29.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.244+0000 7f5497400640 10 client.25903 handle_command_reply: tid=0 2024-03-21T10:39:29.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.244+0000 7f54bf579740 1 client.25903 shutdown 2024-03-21T10:39:29.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.244+0000 7f54bf579740 1 -- 172.21.15.31:0/3614123962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590515bd8f0 msgr2=0x5590515c3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:29.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.244+0000 7f54bf579740 1 --2- 172.21.15.31:0/3614123962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590515bd8f0 0x5590515c3c80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f54a4005e00 tx=0x7f54a4005fd0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.245+0000 7f54bf579740 1 -- 172.21.15.31:0/3614123962 shutdown_connections 2024-03-21T10:39:29.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.245+0000 7f54bf579740 1 --2- 172.21.15.31:0/3614123962 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5590515d6580 0x5590515d8970 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.245+0000 7f54bf579740 1 --2- 172.21.15.31:0/3614123962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5590515c66c0 0x5590515c41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.245+0000 7f54bf579740 1 --2- 172.21.15.31:0/3614123962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590515bd8f0 0x5590515c3c80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.245+0000 7f54bf579740 1 --2- 172.21.15.31:0/3614123962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590515bcfd0 0x5590515c3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.245+0000 7f5497400640 0 client.25903 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:29.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.245+0000 7f54bf579740 1 -- 172.21.15.31:0/3614123962 >> 172.21.15.31:0/3614123962 conn(0x5590515b78f0 msgr2=0x5590515b9640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:29.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.245+0000 7f54bf579740 1 -- 172.21.15.31:0/3614123962 shutdown_connections 2024-03-21T10:39:29.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.245+0000 7f54bf579740 1 -- 172.21.15.31:0/3614123962 wait complete. 2024-03-21T10:39:29.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.245+0000 7f54bf579740 20 client.25903 trim_cache size 0 max 16384 2024-03-21T10:39:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.264+0000 7f54bf579740 1 Processor -- start 2024-03-21T10:39:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.264+0000 7f54bf579740 1 -- start start 2024-03-21T10:39:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.265+0000 7f54bf579740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590515bcfd0 0x5590515c09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.265+0000 7f54bf579740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590515bd8f0 0x5590515c0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.265+0000 7f54bf579740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5590515c66c0 0x5590515b8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.265+0000 7f54bf579740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b41b7410 con 0x5590515bcfd0 2024-03-21T10:39:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.265+0000 7f54bf579740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b41b4f10 con 0x5590515c66c0 2024-03-21T10:39:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.265+0000 7f54bf579740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b41b2fb0 con 0x5590515bd8f0 2024-03-21T10:39:29.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.265+0000 7f54b9400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590515bd8f0 0x5590515c0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.265+0000 7f54ba800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5590515c66c0 0x5590515b8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.265+0000 7f54ba800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5590515c66c0 0x5590515b8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49528/0 (socket says 172.21.15.31:49528) 2024-03-21T10:39:29.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.265+0000 7f54b9400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590515bd8f0 0x5590515c0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:47210/0 (socket says 172.21.15.31:47210) 2024-03-21T10:39:29.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.265+0000 7f54ba800640 1 -- 172.21.15.31:0/2105287160 learned_addr learned my addr 172.21.15.31:0/2105287160 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:29.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.265+0000 7f54b9e00640 1 --2- 172.21.15.31:0/2105287160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590515bcfd0 0x5590515c09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.265+0000 7f54b9400640 1 -- 172.21.15.31:0/2105287160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5590515c66c0 msgr2=0x5590515b8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:29.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.265+0000 7f54b9400640 1 --2- 172.21.15.31:0/2105287160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5590515c66c0 0x5590515b8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.265+0000 7f54b9400640 1 -- 172.21.15.31:0/2105287160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590515bcfd0 msgr2=0x5590515c09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:29.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.265+0000 7f54b9400640 1 --2- 172.21.15.31:0/2105287160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590515bcfd0 0x5590515c09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.265+0000 7f54b9400640 1 -- 172.21.15.31:0/2105287160 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5590515c8ab0 con 0x5590515bd8f0 2024-03-21T10:39:29.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.266+0000 7f54b9400640 1 --2- 172.21.15.31:0/2105287160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590515bd8f0 0x5590515c0ee0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f54a4009780 tx=0x7f54a4004200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df3459f04f2b8676 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:29.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.266+0000 7f54a2600640 1 -- 172.21.15.31:0/2105287160 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a4057070 con 0x5590515bd8f0 2024-03-21T10:39:29.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.266+0000 7f54a2600640 1 -- 172.21.15.31:0/2105287160 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f54a404f040 con 0x5590515bd8f0 2024-03-21T10:39:29.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.266+0000 7f54a2600640 1 -- 172.21.15.31:0/2105287160 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a402c800 con 0x5590515bd8f0 2024-03-21T10:39:29.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.267+0000 7f54bf579740 1 -- 172.21.15.31:0/2105287160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590515bd8f0 msgr2=0x5590515c0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:29.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.267+0000 7f54bf579740 1 --2- 172.21.15.31:0/2105287160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590515bd8f0 0x5590515c0ee0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f54a4009780 tx=0x7f54a4004200 comp rx=0 tx=0).stop 2024-03-21T10:39:29.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.267+0000 7f54bf579740 1 -- 172.21.15.31:0/2105287160 shutdown_connections 2024-03-21T10:39:29.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.267+0000 7f54bf579740 1 --2- 172.21.15.31:0/2105287160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5590515c66c0 0x5590515b8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.267+0000 7f54bf579740 1 --2- 172.21.15.31:0/2105287160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590515bd8f0 0x5590515c0ee0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.267+0000 7f54bf579740 1 --2- 172.21.15.31:0/2105287160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590515bcfd0 0x5590515c09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.267+0000 7f54bf579740 1 -- 172.21.15.31:0/2105287160 >> 172.21.15.31:0/2105287160 conn(0x5590515b78f0 msgr2=0x5590515c2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:29.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.267+0000 7f54bf579740 1 -- 172.21.15.31:0/2105287160 shutdown_connections 2024-03-21T10:39:29.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.267+0000 7f54bf579740 1 -- 172.21.15.31:0/2105287160 wait complete. 2024-03-21T10:39:29.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.267+0000 7f54bf579740 1 -- start start 2024-03-21T10:39:29.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.267+0000 7f54bf579740 1 Processor -- start 2024-03-21T10:39:29.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.268+0000 7f54bf579740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590515bcfd0 0x5590515d9550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.268+0000 7f54bf579740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590515bd8f0 0x5590515d9a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.268+0000 7f54bf579740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5590515c66c0 0x559051630ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.268+0000 7f54bf579740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b41b35b0 con 0x5590515bd8f0 2024-03-21T10:39:29.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.268+0000 7f54bf579740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b41b48b0 con 0x5590515c66c0 2024-03-21T10:39:29.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.268+0000 7f54bf579740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b41b3bf0 con 0x5590515bcfd0 2024-03-21T10:39:29.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.268+0000 7f54bf579740 10 client.0 _ll_register_callbacks cb 0x7f54b0009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:39:29.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.268+0000 7f54b9e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590515bcfd0 0x5590515d9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.268+0000 7f54b9e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590515bcfd0 0x5590515d9550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:47216/0 (socket says 172.21.15.31:47216) 2024-03-21T10:39:29.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.268+0000 7f54b9e00640 1 -- 172.21.15.31:0/1193204665 learned_addr learned my addr 172.21.15.31:0/1193204665 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:29.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.268+0000 7f54b9400640 1 --2- 172.21.15.31:0/1193204665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590515bd8f0 0x5590515d9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.268+0000 7f54ba800640 1 --2- 172.21.15.31:0/1193204665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5590515c66c0 0x559051630ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.269+0000 7f54b9e00640 1 -- 172.21.15.31:0/1193204665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5590515c66c0 msgr2=0x559051630ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:29.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.269+0000 7f54b9e00640 1 --2- 172.21.15.31:0/1193204665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5590515c66c0 0x559051630ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.269+0000 7f54b9e00640 1 -- 172.21.15.31:0/1193204665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590515bd8f0 msgr2=0x5590515d9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:29.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.269+0000 7f54b9e00640 1 --2- 172.21.15.31:0/1193204665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590515bd8f0 0x5590515d9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.269+0000 7f54b9e00640 1 -- 172.21.15.31:0/1193204665 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559051631020 con 0x5590515bcfd0 2024-03-21T10:39:29.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.269+0000 7f54ba800640 1 --2- 172.21.15.31:0/1193204665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5590515c66c0 0x559051630ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:39:29.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.269+0000 7f54b9e00640 1 --2- 172.21.15.31:0/1193204665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590515bcfd0 0x5590515d9550 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f54a809bde0 tx=0x7f54a8098220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7dd411bbe489dd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:29.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.269+0000 7f5497400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:39:29.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.269+0000 7f5497400640 1 -- 172.21.15.31:0/1193204665 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a80a4860 con 0x5590515bcfd0 2024-03-21T10:39:29.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.269+0000 7f5497400640 1 -- 172.21.15.31:0/1193204665 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f54a80a43d0 con 0x5590515bcfd0 2024-03-21T10:39:29.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.269+0000 7f54bf579740 1 -- 172.21.15.31:0/1193204665 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f54a404f040 con 0x5590515bcfd0 2024-03-21T10:39:29.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.269+0000 7f5497400640 1 -- 172.21.15.31:0/1193204665 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54a80a45d0 con 0x5590515bcfd0 2024-03-21T10:39:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.269+0000 7f54bf579740 10 client.25909 fetch_fsmap learned FSMap version 24 2024-03-21T10:39:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.269+0000 7f54bf579740 1 -- 172.21.15.31:0/1193204665 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x559051632330 con 0x5590515bcfd0 2024-03-21T10:39:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.269+0000 7f5497400640 1 -- 172.21.15.31:0/1193204665 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f54a404f040 con 0x5590515bcfd0 2024-03-21T10:39:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.270+0000 7f5497400640 1 -- 172.21.15.31:0/1193204665 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f54a80cd560 con 0x5590515bcfd0 2024-03-21T10:39:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.270+0000 7f54bf579740 10 client.25909 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:39:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.270+0000 7f54bf579740 10 client.25909 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:39:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.270+0000 7f54bf579740 20 client.25909 populate_metadata read hostname 'smithi031' 2024-03-21T10:39:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.270+0000 7f54bf579740 1 --2- 172.21.15.31:0/1193204665 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5590516329f0 0x559051634de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.270+0000 7f54bf579740 4 client.25909 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:39:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.270+0000 7f54bf579740 1 -- 172.21.15.31:0/1193204665 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5590515290c0 con 0x5590516329f0 2024-03-21T10:39:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.270+0000 7f54b9400640 1 --2- 172.21.15.31:0/1193204665 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5590516329f0 0x559051634de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.271+0000 7f5497400640 10 client.25909 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.271+0000 7f54b9400640 1 --2- 172.21.15.31:0/1193204665 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5590516329f0 0x559051634de0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc21be6966347949 server_cookie=c5974f088b318d20 in_seq=0 out_seq=0 2024-03-21T10:39:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.275+0000 7f5497400640 1 -- 172.21.15.31:0/1193204665 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5590515290c0 con 0x5590516329f0 2024-03-21T10:39:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.275+0000 7f5497400640 10 client.25909 handle_command_reply: tid=0 2024-03-21T10:39:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.275+0000 7f54bf579740 1 client.25909 shutdown 2024-03-21T10:39:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.276+0000 7f54bf579740 1 -- 172.21.15.31:0/1193204665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590515bcfd0 msgr2=0x5590515d9550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.276+0000 7f54bf579740 1 --2- 172.21.15.31:0/1193204665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590515bcfd0 0x5590515d9550 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f54a809bde0 tx=0x7f54a8098220 comp rx=0 tx=0).stop 2024-03-21T10:39:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.276+0000 7f54bf579740 1 -- 172.21.15.31:0/1193204665 shutdown_connections 2024-03-21T10:39:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.276+0000 7f54bf579740 1 --2- 172.21.15.31:0/1193204665 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5590516329f0 0x559051634de0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.276+0000 7f54bf579740 1 --2- 172.21.15.31:0/1193204665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5590515c66c0 0x559051630ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.276+0000 7f54bf579740 1 --2- 172.21.15.31:0/1193204665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590515bd8f0 0x5590515d9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.276+0000 7f5497400640 0 client.25909 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.276+0000 7f54bf579740 1 --2- 172.21.15.31:0/1193204665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5590515bcfd0 0x5590515d9550 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.276+0000 7f54bf579740 1 -- 172.21.15.31:0/1193204665 >> 172.21.15.31:0/1193204665 conn(0x5590515b78f0 msgr2=0x5590515b8d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.276+0000 7f54bf579740 1 -- 172.21.15.31:0/1193204665 shutdown_connections 2024-03-21T10:39:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.276+0000 7f54bf579740 1 -- 172.21.15.31:0/1193204665 wait complete. 2024-03-21T10:39:29.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.276+0000 7f54bf579740 20 client.25909 trim_cache size 0 max 16384 2024-03-21T10:39:29.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:39:29.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:39:29.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:39:29.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:39:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.277+0000 7f54bc800640 1 -- 172.21.15.31:0/109670354 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f549c0a3220 msgr2=0x7f549c0a56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.277+0000 7f54bc800640 1 --2- 172.21.15.31:0/109670354 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f549c0a3220 0x7f549c0a56e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f54a4037040 tx=0x7f54a40056d0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.277+0000 7f54bc800640 1 -- 172.21.15.31:0/109670354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54b4075bd0 msgr2=0x7f54b419e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.277+0000 7f54bc800640 1 --2- 172.21.15.31:0/109670354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54b4075bd0 0x7f54b419e630 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f54a800e9f0 tx=0x7f54a800eeb0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.277+0000 7f54bc800640 1 -- 172.21.15.31:0/109670354 shutdown_connections 2024-03-21T10:39:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.277+0000 7f54bc800640 1 --2- 172.21.15.31:0/109670354 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f549c0a3220 0x7f549c0a56e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.277+0000 7f54bc800640 1 --2- 172.21.15.31:0/109670354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54b4112510 0x7f54b41a2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.277+0000 7f54bc800640 1 --2- 172.21.15.31:0/109670354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54b40764f0 0x7f54b419eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.277+0000 7f54bc800640 1 --2- 172.21.15.31:0/109670354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54b4075bd0 0x7f54b419e630 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.277+0000 7f54bc800640 1 -- 172.21.15.31:0/109670354 >> 172.21.15.31:0/109670354 conn(0x7f54b40ff320 msgr2=0x7f54b4113ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.277+0000 7f54bc800640 1 -- 172.21.15.31:0/109670354 shutdown_connections 2024-03-21T10:39:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:29.277+0000 7f54bc800640 1 -- 172.21.15.31:0/109670354 wait complete. 2024-03-21T10:39:29.461 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-21T10:39:29.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:39:29.461 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-21T10:39:29.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.693+0000 7f454a200640 1 -- 172.21.15.31:0/3301574035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45441014a0 msgr2=0x7f4544101920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:29.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.693+0000 7f454a200640 1 --2- 172.21.15.31:0/3301574035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45441014a0 0x7f4544101920 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4530009930 tx=0x7f453002bc30 comp rx=0 tx=0).stop 2024-03-21T10:39:29.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.693+0000 7f454a200640 1 -- 172.21.15.31:0/3301574035 shutdown_connections 2024-03-21T10:39:29.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.693+0000 7f454a200640 1 --2- 172.21.15.31:0/3301574035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4544101e60 0x7f454410e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.693+0000 7f454a200640 1 --2- 172.21.15.31:0/3301574035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45441014a0 0x7f4544101920 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.693+0000 7f454a200640 1 --2- 172.21.15.31:0/3301574035 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4544107560 0x7f4544107940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.693+0000 7f454a200640 1 -- 172.21.15.31:0/3301574035 >> 172.21.15.31:0/3301574035 conn(0x7f45440fcfb0 msgr2=0x7f45440ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:29.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.694+0000 7f454a200640 1 -- 172.21.15.31:0/3301574035 shutdown_connections 2024-03-21T10:39:29.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.694+0000 7f454a200640 1 -- 172.21.15.31:0/3301574035 wait complete. 2024-03-21T10:39:29.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.695+0000 7f454a200640 1 Processor -- start 2024-03-21T10:39:29.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.695+0000 7f454a200640 1 -- start start 2024-03-21T10:39:29.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.695+0000 7f454a200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45441014a0 0x7f4544197d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.695+0000 7f454a200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4544101e60 0x7f4544198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.695+0000 7f454a200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4544107560 0x7f454419c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.695+0000 7f454a200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4544111870 con 0x7f4544107560 2024-03-21T10:39:29.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.695+0000 7f454a200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45441116f0 con 0x7f4544101e60 2024-03-21T10:39:29.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.695+0000 7f454a200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45441119f0 con 0x7f45441014a0 2024-03-21T10:39:29.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.696+0000 7f4543e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4544101e60 0x7f4544198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.696+0000 7f4548e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45441014a0 0x7f4544197d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.696+0000 7f4543e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4544101e60 0x7f4544198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49544/0 (socket says 172.21.15.31:49544) 2024-03-21T10:39:29.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.696+0000 7f4543e00640 1 -- 172.21.15.31:0/301499802 learned_addr learned my addr 172.21.15.31:0/301499802 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:29.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.696+0000 7f4548e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45441014a0 0x7f4544197d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:47242/0 (socket says 172.21.15.31:47242) 2024-03-21T10:39:29.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.696+0000 7f4549800640 1 --2- 172.21.15.31:0/301499802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4544107560 0x7f454419c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.696+0000 7f4543e00640 1 -- 172.21.15.31:0/301499802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45441014a0 msgr2=0x7f4544197d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:29.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.696+0000 7f4543e00640 1 --2- 172.21.15.31:0/301499802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45441014a0 0x7f4544197d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.696+0000 7f4543e00640 1 -- 172.21.15.31:0/301499802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4544107560 msgr2=0x7f454419c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:29.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.696+0000 7f4543e00640 1 --2- 172.21.15.31:0/301499802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4544107560 0x7f454419c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.696+0000 7f4543e00640 1 -- 172.21.15.31:0/301499802 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f454419cd40 con 0x7f4544101e60 2024-03-21T10:39:29.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.696+0000 7f4543e00640 1 --2- 172.21.15.31:0/301499802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4544101e60 0x7f4544198290 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f4530009900 tx=0x7f4530004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:29.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.697+0000 7f4541600640 1 -- 172.21.15.31:0/301499802 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4530043020 con 0x7f4544101e60 2024-03-21T10:39:29.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.697+0000 7f4541600640 1 -- 172.21.15.31:0/301499802 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45300054e0 con 0x7f4544101e60 2024-03-21T10:39:29.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.697+0000 7f4541600640 1 -- 172.21.15.31:0/301499802 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4530003bb0 con 0x7f4544101e60 2024-03-21T10:39:29.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.697+0000 7f454a200640 1 -- 172.21.15.31:0/301499802 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f454419cfd0 con 0x7f4544101e60 2024-03-21T10:39:29.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.697+0000 7f454a200640 1 -- 172.21.15.31:0/301499802 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45440738d0 con 0x7f4544101e60 2024-03-21T10:39:29.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.700+0000 7f454e5c6740 1 Processor -- start 2024-03-21T10:39:29.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.700+0000 7f454e5c6740 1 -- start start 2024-03-21T10:39:29.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.701+0000 7f454e5c6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154290 0x55ae45154670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.701+0000 7f454e5c6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae45154bb0 0x55ae4515d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.701+0000 7f454e5c6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae4515d790 0x55ae4515fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.701+0000 7f454e5c6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae450b8eb0 con 0x55ae4515d790 2024-03-21T10:39:29.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.701+0000 7f454e5c6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae45061a60 con 0x55ae45154290 2024-03-21T10:39:29.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.701+0000 7f454e5c6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae450b4f50 con 0x55ae45154bb0 2024-03-21T10:39:29.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.701+0000 7f4549800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154290 0x55ae45154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.701+0000 7f4548e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae45154bb0 0x55ae4515d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.701+0000 7f4549800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154290 0x55ae45154670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49552/0 (socket says 172.21.15.31:49552) 2024-03-21T10:39:29.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.701+0000 7f4549800640 1 -- 172.21.15.31:0/2996986968 learned_addr learned my addr 172.21.15.31:0/2996986968 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:29.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.701+0000 7f4541600640 1 -- 172.21.15.31:0/301499802 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) v1 ==== 138087+0+0 (secure 0 0 0) 0x7f4530046020 con 0x7f4544101e60 2024-03-21T10:39:29.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.701+0000 7f4549800640 1 -- 172.21.15.31:0/2996986968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae45154bb0 msgr2=0x55ae4515d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:29.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.701+0000 7f4549800640 1 --2- 172.21.15.31:0/2996986968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae45154bb0 0x55ae4515d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.701+0000 7f4549800640 1 -- 172.21.15.31:0/2996986968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae4515d790 msgr2=0x55ae4515fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:29.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.701+0000 7f4549800640 1 --2- 172.21.15.31:0/2996986968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae4515d790 0x55ae4515fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.701+0000 7f4549800640 1 -- 172.21.15.31:0/2996986968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ae44e957e0 con 0x55ae45154290 2024-03-21T10:39:29.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.702+0000 7f4549800640 1 --2- 172.21.15.31:0/2996986968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154290 0x55ae45154670 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f453c009fd0 tx=0x7f453c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bac1568beb2f7539 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:29.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.702+0000 7f4536000640 1 -- 172.21.15.31:0/2996986968 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c004d20 con 0x55ae45154290 2024-03-21T10:39:29.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.702+0000 7f4536000640 1 -- 172.21.15.31:0/2996986968 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f453c004050 con 0x55ae45154290 2024-03-21T10:39:29.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.702+0000 7f4536000640 1 -- 172.21.15.31:0/2996986968 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c005260 con 0x55ae45154290 2024-03-21T10:39:29.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.703+0000 7f454e5c6740 1 -- 172.21.15.31:0/2996986968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154290 msgr2=0x55ae45154670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:29.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.703+0000 7f454e5c6740 1 --2- 172.21.15.31:0/2996986968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154290 0x55ae45154670 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f453c009fd0 tx=0x7f453c0118d0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.703+0000 7f454e5c6740 1 -- 172.21.15.31:0/2996986968 shutdown_connections 2024-03-21T10:39:29.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.703+0000 7f454e5c6740 1 --2- 172.21.15.31:0/2996986968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae4515d790 0x55ae4515fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.703+0000 7f454e5c6740 1 --2- 172.21.15.31:0/2996986968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae45154bb0 0x55ae4515d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.703+0000 7f454e5c6740 1 --2- 172.21.15.31:0/2996986968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154290 0x55ae45154670 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.703+0000 7f4541600640 1 --2- 172.21.15.31:0/301499802 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f452c0a32b0 0x7f452c0a5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.703+0000 7f454e5c6740 1 -- 172.21.15.31:0/2996986968 >> 172.21.15.31:0/2996986968 conn(0x55ae4514e9c0 msgr2=0x55ae4514eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:29.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.703+0000 7f4541600640 1 -- 172.21.15.31:0/301499802 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f45300f1c90 con 0x7f4544101e60 2024-03-21T10:39:29.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.703+0000 7f454e5c6740 1 -- 172.21.15.31:0/2996986968 shutdown_connections 2024-03-21T10:39:29.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.704+0000 7f454e5c6740 1 -- 172.21.15.31:0/2996986968 wait complete. 2024-03-21T10:39:29.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.704+0000 7f4549800640 1 --2- 172.21.15.31:0/301499802 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f452c0a32b0 0x7f452c0a5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.704+0000 7f454e5c6740 1 -- start start 2024-03-21T10:39:29.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.704+0000 7f454e5c6740 1 Processor -- start 2024-03-21T10:39:29.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.705+0000 7f4549800640 1 --2- 172.21.15.31:0/301499802 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f452c0a32b0 0x7f452c0a5770 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f453c0049d0 tx=0x7f453c009be0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:29.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.705+0000 7f454e5c6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae45154290 0x55ae4515d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.705+0000 7f454e5c6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154bb0 0x55ae4515a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.705+0000 7f454e5c6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae4515d790 0x55ae4515ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.705+0000 7f454e5c6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae45160580 con 0x55ae45154290 2024-03-21T10:39:29.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.705+0000 7f454e5c6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae45074eb0 con 0x55ae45154bb0 2024-03-21T10:39:29.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.705+0000 7f454e5c6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae45160700 con 0x55ae4515d790 2024-03-21T10:39:29.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.705+0000 7f454e5c6740 10 client.0 _ll_register_callbacks cb 0x55ae45069150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:39:29.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.705+0000 7f4548e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae45154290 0x55ae4515d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.705+0000 7f4548e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae45154290 0x55ae4515d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33552/0 (socket says 172.21.15.31:33552) 2024-03-21T10:39:29.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.705+0000 7f4548e00640 1 -- 172.21.15.31:0/1031373224 learned_addr learned my addr 172.21.15.31:0/1031373224 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:29.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.705+0000 7f4549800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae4515d790 0x55ae4515ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.706+0000 7f4549800640 1 -- 172.21.15.31:0/1031373224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154bb0 msgr2=0x55ae4515a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:29.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.707+0000 7f4543e00640 1 --2- 172.21.15.31:0/1031373224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154bb0 0x55ae4515a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.707+0000 7f4549800640 1 --2- 172.21.15.31:0/1031373224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154bb0 0x55ae4515a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.707+0000 7f4549800640 1 -- 172.21.15.31:0/1031373224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae45154290 msgr2=0x55ae4515d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:29.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.707+0000 7f4549800640 1 --2- 172.21.15.31:0/1031373224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae45154290 0x55ae4515d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.707+0000 7f4549800640 1 -- 172.21.15.31:0/1031373224 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ae44ef5f70 con 0x55ae4515d790 2024-03-21T10:39:29.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.707+0000 7f4548e00640 1 --2- 172.21.15.31:0/1031373224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae45154290 0x55ae4515d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:39:29.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.707+0000 7f4543e00640 1 --2- 172.21.15.31:0/1031373224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154bb0 0x55ae4515a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:39:29.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.707+0000 7f4549800640 1 --2- 172.21.15.31:0/1031373224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae4515d790 0x55ae4515ab20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f453c011dd0 tx=0x7f453c005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b068bb36961fc3b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:29.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.707+0000 7f452aa00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:39:29.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.707+0000 7f452aa00640 1 -- 172.21.15.31:0/1031373224 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c034020 con 0x55ae4515d790 2024-03-21T10:39:29.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.707+0000 7f452aa00640 1 -- 172.21.15.31:0/1031373224 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f453c00f4e0 con 0x55ae4515d790 2024-03-21T10:39:29.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.707+0000 7f454e5c6740 1 -- 172.21.15.31:0/1031373224 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f453c004050 con 0x55ae4515d790 2024-03-21T10:39:29.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.708+0000 7f452aa00640 1 -- 172.21.15.31:0/1031373224 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f453c03f050 con 0x55ae4515d790 2024-03-21T10:39:29.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.708+0000 7f454e5c6740 10 client.25921 fetch_fsmap learned FSMap version 24 2024-03-21T10:39:29.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.708+0000 7f454e5c6740 1 -- 172.21.15.31:0/1031373224 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55ae44d8d170 con 0x55ae4515d790 2024-03-21T10:39:29.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.708+0000 7f452aa00640 1 -- 172.21.15.31:0/1031373224 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f453c004050 con 0x55ae4515d790 2024-03-21T10:39:29.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.708+0000 7f452aa00640 1 -- 172.21.15.31:0/1031373224 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f453c023550 con 0x55ae4515d790 2024-03-21T10:39:29.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.708+0000 7f454e5c6740 10 client.25921 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:39:29.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.708+0000 7f454e5c6740 10 client.25921 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:39:29.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.709+0000 7f454e5c6740 20 client.25921 populate_metadata read hostname 'smithi031' 2024-03-21T10:39:29.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.709+0000 7f454e5c6740 1 --2- 172.21.15.31:0/1031373224 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ae4516d800 0x55ae4516fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.709+0000 7f454e5c6740 4 client.25921 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:39:29.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.709+0000 7f454e5c6740 1 -- 172.21.15.31:0/1031373224 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ae450c63f0 con 0x55ae4516d800 2024-03-21T10:39:29.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.709+0000 7f4548e00640 1 --2- 172.21.15.31:0/1031373224 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ae4516d800 0x55ae4516fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.710+0000 7f4548e00640 1 --2- 172.21.15.31:0/1031373224 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ae4516d800 0x55ae4516fbf0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcec78739e02031b server_cookie=4e5298df681be60b in_seq=0 out_seq=0 2024-03-21T10:39:29.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.710+0000 7f452aa00640 10 client.25921 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:29.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.710+0000 7f452aa00640 1 -- 172.21.15.31:0/1031373224 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ae450c63f0 con 0x55ae4516d800 2024-03-21T10:39:29.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.710+0000 7f452aa00640 10 client.25921 handle_command_reply: tid=0 2024-03-21T10:39:29.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.710+0000 7f454e5c6740 1 client.25921 shutdown 2024-03-21T10:39:29.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.711+0000 7f454e5c6740 1 -- 172.21.15.31:0/1031373224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae4515d790 msgr2=0x55ae4515ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:29.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.711+0000 7f454e5c6740 1 --2- 172.21.15.31:0/1031373224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae4515d790 0x55ae4515ab20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f453c011dd0 tx=0x7f453c005fd0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.711+0000 7f454e5c6740 1 -- 172.21.15.31:0/1031373224 shutdown_connections 2024-03-21T10:39:29.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.711+0000 7f454e5c6740 1 --2- 172.21.15.31:0/1031373224 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ae4516d800 0x55ae4516fbf0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.711+0000 7f454e5c6740 1 --2- 172.21.15.31:0/1031373224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae4515d790 0x55ae4515ab20 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.711+0000 7f454e5c6740 1 --2- 172.21.15.31:0/1031373224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154bb0 0x55ae4515a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.711+0000 7f454e5c6740 1 --2- 172.21.15.31:0/1031373224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae45154290 0x55ae4515d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.711+0000 7f454e5c6740 1 -- 172.21.15.31:0/1031373224 >> 172.21.15.31:0/1031373224 conn(0x55ae4514e9c0 msgr2=0x55ae45150710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:29.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.711+0000 7f452aa00640 0 client.25921 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:29.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.711+0000 7f454e5c6740 1 -- 172.21.15.31:0/1031373224 shutdown_connections 2024-03-21T10:39:29.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.711+0000 7f454e5c6740 1 -- 172.21.15.31:0/1031373224 wait complete. 2024-03-21T10:39:29.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.711+0000 7f454e5c6740 20 client.25921 trim_cache size 0 max 16384 2024-03-21T10:39:29.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.736+0000 7f454e5c6740 1 Processor -- start 2024-03-21T10:39:29.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.736+0000 7f454e5c6740 1 -- start start 2024-03-21T10:39:29.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.736+0000 7f454e5c6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154290 0x55ae451508e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.736+0000 7f454e5c6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae45154bb0 0x55ae4515cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.736+0000 7f454e5c6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae4515d790 0x55ae4515d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.736+0000 7f454e5c6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45441b2e20 con 0x55ae45154bb0 2024-03-21T10:39:29.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.736+0000 7f454e5c6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45441b0920 con 0x55ae45154290 2024-03-21T10:39:29.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.736+0000 7f454e5c6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45441ae9c0 con 0x55ae4515d790 2024-03-21T10:39:29.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.737+0000 7f4548e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154290 0x55ae451508e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.737+0000 7f4548e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154290 0x55ae451508e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49576/0 (socket says 172.21.15.31:49576) 2024-03-21T10:39:29.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.737+0000 7f4548e00640 1 -- 172.21.15.31:0/957398830 learned_addr learned my addr 172.21.15.31:0/957398830 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:29.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.737+0000 7f4543e00640 1 --2- 172.21.15.31:0/957398830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae45154bb0 0x55ae4515cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.737+0000 7f4549800640 1 --2- 172.21.15.31:0/957398830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae4515d790 0x55ae4515d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.737+0000 7f4548e00640 1 -- 172.21.15.31:0/957398830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae4515d790 msgr2=0x55ae4515d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:29.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.737+0000 7f4548e00640 1 --2- 172.21.15.31:0/957398830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae4515d790 0x55ae4515d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.737+0000 7f4548e00640 1 -- 172.21.15.31:0/957398830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae45154bb0 msgr2=0x55ae4515cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:29.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.737+0000 7f4548e00640 1 --2- 172.21.15.31:0/957398830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae45154bb0 0x55ae4515cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.737+0000 7f4548e00640 1 -- 172.21.15.31:0/957398830 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ae45160460 con 0x55ae45154290 2024-03-21T10:39:29.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.737+0000 7f4548e00640 1 --2- 172.21.15.31:0/957398830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154290 0x55ae451508e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4538009710 tx=0x7f4538005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b8fdba7f4fdcaab server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:29.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.738+0000 7f4535600640 1 -- 172.21.15.31:0/957398830 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4538004050 con 0x55ae45154290 2024-03-21T10:39:29.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.738+0000 7f4535600640 1 -- 172.21.15.31:0/957398830 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45380041f0 con 0x55ae45154290 2024-03-21T10:39:29.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.738+0000 7f4535600640 1 -- 172.21.15.31:0/957398830 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453801fe40 con 0x55ae45154290 2024-03-21T10:39:29.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.739+0000 7f454e5c6740 1 -- 172.21.15.31:0/957398830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154290 msgr2=0x55ae451508e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:29.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.739+0000 7f454e5c6740 1 --2- 172.21.15.31:0/957398830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154290 0x55ae451508e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4538009710 tx=0x7f4538005f90 comp rx=0 tx=0).stop 2024-03-21T10:39:29.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.739+0000 7f454e5c6740 1 -- 172.21.15.31:0/957398830 shutdown_connections 2024-03-21T10:39:29.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.739+0000 7f454e5c6740 1 --2- 172.21.15.31:0/957398830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae4515d790 0x55ae4515d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.739+0000 7f454e5c6740 1 --2- 172.21.15.31:0/957398830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae45154bb0 0x55ae4515cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.739+0000 7f454e5c6740 1 --2- 172.21.15.31:0/957398830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154290 0x55ae451508e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.739+0000 7f454e5c6740 1 -- 172.21.15.31:0/957398830 >> 172.21.15.31:0/957398830 conn(0x55ae4514e9c0 msgr2=0x55ae4514da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:29.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.739+0000 7f454e5c6740 1 -- 172.21.15.31:0/957398830 shutdown_connections 2024-03-21T10:39:29.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.739+0000 7f454e5c6740 1 -- 172.21.15.31:0/957398830 wait complete. 2024-03-21T10:39:29.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.739+0000 7f454e5c6740 1 -- start start 2024-03-21T10:39:29.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.740+0000 7f454e5c6740 1 Processor -- start 2024-03-21T10:39:29.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.740+0000 7f454e5c6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154290 0x55ae45160ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.740+0000 7f454e5c6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae45154bb0 0x55ae45161430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.740+0000 7f454e5c6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae4515d790 0x55ae45161970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.740+0000 7f454e5c6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45441aefc0 con 0x55ae45154bb0 2024-03-21T10:39:29.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.740+0000 7f454e5c6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45441b02c0 con 0x55ae45154290 2024-03-21T10:39:29.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.740+0000 7f454e5c6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45441af600 con 0x55ae4515d790 2024-03-21T10:39:29.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.740+0000 7f454e5c6740 10 client.0 _ll_register_callbacks cb 0x7f45380097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:39:29.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.740+0000 7f4549800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae4515d790 0x55ae45161970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.740+0000 7f4548e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154290 0x55ae45160ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.740+0000 7f4543e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae45154bb0 0x55ae45161430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.741+0000 7f4548e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154290 0x55ae45160ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49592/0 (socket says 172.21.15.31:49592) 2024-03-21T10:39:29.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.741+0000 7f4548e00640 1 -- 172.21.15.31:0/4000597594 learned_addr learned my addr 172.21.15.31:0/4000597594 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:29.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.740+0000 7f4549800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae4515d790 0x55ae45161970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:47282/0 (socket says 172.21.15.31:47282) 2024-03-21T10:39:29.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.741+0000 7f4548e00640 1 -- 172.21.15.31:0/4000597594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae4515d790 msgr2=0x55ae45161970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:29.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.741+0000 7f4548e00640 1 --2- 172.21.15.31:0/4000597594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae4515d790 0x55ae45161970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.741+0000 7f4548e00640 1 -- 172.21.15.31:0/4000597594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae45154bb0 msgr2=0x55ae45161430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:29.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.741+0000 7f4548e00640 1 --2- 172.21.15.31:0/4000597594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae45154bb0 0x55ae45161430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.741+0000 7f4548e00640 1 -- 172.21.15.31:0/4000597594 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ae451c7fc0 con 0x55ae45154290 2024-03-21T10:39:29.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.741+0000 7f4548e00640 1 --2- 172.21.15.31:0/4000597594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154290 0x55ae45160ef0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f45380057c0 tx=0x7f4538009de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57f9e4bfce4bc057 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:29.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.741+0000 7f452aa00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:39:29.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.742+0000 7f452aa00640 1 -- 172.21.15.31:0/4000597594 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453801fac0 con 0x55ae45154290 2024-03-21T10:39:29.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.742+0000 7f454e5c6740 1 -- 172.21.15.31:0/4000597594 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f45380041f0 con 0x55ae45154290 2024-03-21T10:39:29.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.742+0000 7f452aa00640 1 -- 172.21.15.31:0/4000597594 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f453801fc60 con 0x55ae45154290 2024-03-21T10:39:29.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.742+0000 7f452aa00640 1 -- 172.21.15.31:0/4000597594 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f453803c050 con 0x55ae45154290 2024-03-21T10:39:29.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.742+0000 7f454e5c6740 10 client.25940 fetch_fsmap learned FSMap version 24 2024-03-21T10:39:29.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.742+0000 7f454e5c6740 1 -- 172.21.15.31:0/4000597594 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55ae451c92a0 con 0x55ae45154290 2024-03-21T10:39:29.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.742+0000 7f452aa00640 1 -- 172.21.15.31:0/4000597594 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45380041f0 con 0x55ae45154290 2024-03-21T10:39:29.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.743+0000 7f452aa00640 1 -- 172.21.15.31:0/4000597594 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f4538003950 con 0x55ae45154290 2024-03-21T10:39:29.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.743+0000 7f454e5c6740 10 client.25940 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:39:29.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.743+0000 7f454e5c6740 10 client.25940 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:39:29.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.743+0000 7f454e5c6740 20 client.25940 populate_metadata read hostname 'smithi031' 2024-03-21T10:39:29.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.743+0000 7f454e5c6740 1 --2- 172.21.15.31:0/4000597594 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ae451c9900 0x55ae451cbd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:29.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.743+0000 7f454e5c6740 4 client.25940 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-21T10:39:29.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.743+0000 7f454e5c6740 1 -- 172.21.15.31:0/4000597594 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55ae450c0740 con 0x55ae451c9900 2024-03-21T10:39:29.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.743+0000 7f4543e00640 1 --2- 172.21.15.31:0/4000597594 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ae451c9900 0x55ae451cbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:29.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.744+0000 7f4543e00640 1 --2- 172.21.15.31:0/4000597594 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ae451c9900 0x55ae451cbd20 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29eeb3d145049bb9 server_cookie=b71befa3b9b6fa6c in_seq=0 out_seq=0 2024-03-21T10:39:29.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.744+0000 7f452aa00640 10 client.25940 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:29.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.745+0000 7f452aa00640 1 -- 172.21.15.31:0/4000597594 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55ae450c0740 con 0x55ae451c9900 2024-03-21T10:39:29.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.745+0000 7f452aa00640 10 client.25940 handle_command_reply: tid=0 2024-03-21T10:39:29.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.745+0000 7f454e5c6740 1 client.25940 shutdown 2024-03-21T10:39:29.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.745+0000 7f454e5c6740 1 -- 172.21.15.31:0/4000597594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154290 msgr2=0x55ae45160ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:29.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.745+0000 7f454e5c6740 1 --2- 172.21.15.31:0/4000597594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154290 0x55ae45160ef0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f45380057c0 tx=0x7f4538009de0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.746+0000 7f454e5c6740 1 -- 172.21.15.31:0/4000597594 shutdown_connections 2024-03-21T10:39:29.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.746+0000 7f454e5c6740 1 --2- 172.21.15.31:0/4000597594 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ae451c9900 0x55ae451cbd20 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.746+0000 7f454e5c6740 1 --2- 172.21.15.31:0/4000597594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae4515d790 0x55ae45161970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.746+0000 7f454e5c6740 1 --2- 172.21.15.31:0/4000597594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae45154bb0 0x55ae45161430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.746+0000 7f454e5c6740 1 --2- 172.21.15.31:0/4000597594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae45154290 0x55ae45160ef0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.746+0000 7f454e5c6740 1 -- 172.21.15.31:0/4000597594 >> 172.21.15.31:0/4000597594 conn(0x55ae4514e9c0 msgr2=0x55ae45159dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:29.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.746+0000 7f452aa00640 0 client.25940 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:29.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.746+0000 7f454e5c6740 1 -- 172.21.15.31:0/4000597594 shutdown_connections 2024-03-21T10:39:29.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.746+0000 7f454e5c6740 1 -- 172.21.15.31:0/4000597594 wait complete. 2024-03-21T10:39:29.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.746+0000 7f454e5c6740 20 client.25940 trim_cache size 0 max 16384 2024-03-21T10:39:29.745 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:39:29.746 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-03-21T10:39:29.746 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "97323de0-3dac-4d64-805a-f7eaa0398908", 2024-03-21T10:39:29.746 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-03-21T10:39:29.746 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:39:29.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.747+0000 7f454a200640 1 -- 172.21.15.31:0/301499802 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f452c0a32b0 msgr2=0x7f452c0a5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:29.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.747+0000 7f454a200640 1 --2- 172.21.15.31:0/301499802 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f452c0a32b0 0x7f452c0a5770 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f453c0049d0 tx=0x7f453c009be0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.747+0000 7f454a200640 1 -- 172.21.15.31:0/301499802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4544101e60 msgr2=0x7f4544198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:29.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.747+0000 7f454a200640 1 --2- 172.21.15.31:0/301499802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4544101e60 0x7f4544198290 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f4530009900 tx=0x7f4530004360 comp rx=0 tx=0).stop 2024-03-21T10:39:29.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.748+0000 7f454a200640 1 -- 172.21.15.31:0/301499802 shutdown_connections 2024-03-21T10:39:29.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.748+0000 7f454a200640 1 --2- 172.21.15.31:0/301499802 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f452c0a32b0 0x7f452c0a5770 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.748+0000 7f454a200640 1 --2- 172.21.15.31:0/301499802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4544107560 0x7f454419c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.748+0000 7f454a200640 1 --2- 172.21.15.31:0/301499802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4544101e60 0x7f4544198290 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.748+0000 7f454a200640 1 --2- 172.21.15.31:0/301499802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45441014a0 0x7f4544197d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:29.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.748+0000 7f454a200640 1 -- 172.21.15.31:0/301499802 >> 172.21.15.31:0/301499802 conn(0x7f45440fcfb0 msgr2=0x7f45440fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:29.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.748+0000 7f454a200640 1 -- 172.21.15.31:0/301499802 shutdown_connections 2024-03-21T10:39:29.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:29.748+0000 7f454a200640 1 -- 172.21.15.31:0/301499802 wait complete. 2024-03-21T10:39:29.766 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:39:29.767 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:39:30.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.022+0000 7f2053e00640 1 -- 172.21.15.31:0/1426034637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f205410c980 msgr2=0x7f205410cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:30.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.022+0000 7f2053e00640 1 --2- 172.21.15.31:0/1426034637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f205410c980 0x7f205410cd60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2044009900 tx=0x7f204402bcc0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.022+0000 7f2053e00640 1 -- 172.21.15.31:0/1426034637 shutdown_connections 2024-03-21T10:39:30.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.022+0000 7f2053e00640 1 --2- 172.21.15.31:0/1426034637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2054108440 0x7f2054114900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.022+0000 7f2053e00640 1 --2- 172.21.15.31:0/1426034637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2054107aa0 0x7f2054107f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.022+0000 7f2053e00640 1 --2- 172.21.15.31:0/1426034637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f205410c980 0x7f205410cd60 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.022+0000 7f2053e00640 1 -- 172.21.15.31:0/1426034637 >> 172.21.15.31:0/1426034637 conn(0x7f20540772a0 msgr2=0x7f20540796c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:30.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.022+0000 7f2053e00640 1 -- 172.21.15.31:0/1426034637 shutdown_connections 2024-03-21T10:39:30.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.022+0000 7f2053e00640 1 -- 172.21.15.31:0/1426034637 wait complete. 2024-03-21T10:39:30.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.023+0000 7f2053e00640 1 Processor -- start 2024-03-21T10:39:30.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.024+0000 7f2053e00640 1 -- start start 2024-03-21T10:39:30.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.024+0000 7f2053e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2054107aa0 0x7f205419e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:30.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.024+0000 7f2053e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2054108440 0x7f205419e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:30.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.024+0000 7f2053e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f205410c980 0x7f20541a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:30.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.024+0000 7f2053e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2054117e90 con 0x7f205410c980 2024-03-21T10:39:30.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.024+0000 7f2053e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2054117d10 con 0x7f2054108440 2024-03-21T10:39:30.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.024+0000 7f2053e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2054118010 con 0x7f2054107aa0 2024-03-21T10:39:30.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.024+0000 7f2052000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2054108440 0x7f205419e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:30.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.024+0000 7f2052000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2054108440 0x7f205419e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49604/0 (socket says 172.21.15.31:49604) 2024-03-21T10:39:30.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.025+0000 7f2052a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2054107aa0 0x7f205419e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:30.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.025+0000 7f2052000640 1 -- 172.21.15.31:0/694006245 learned_addr learned my addr 172.21.15.31:0/694006245 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:30.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.025+0000 7f2052000640 1 -- 172.21.15.31:0/694006245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2054107aa0 msgr2=0x7f205419e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:30.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.025+0000 7f2052000640 1 --2- 172.21.15.31:0/694006245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2054107aa0 0x7f205419e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.025+0000 7f2052000640 1 -- 172.21.15.31:0/694006245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f205410c980 msgr2=0x7f20541a2c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T10:39:30.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.025+0000 7f2053400640 1 --2- 172.21.15.31:0/694006245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f205410c980 0x7f20541a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:30.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.025+0000 7f2052000640 1 --2- 172.21.15.31:0/694006245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f205410c980 0x7f20541a2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.025+0000 7f2052000640 1 -- 172.21.15.31:0/694006245 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20541a33c0 con 0x7f2054108440 2024-03-21T10:39:30.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.026+0000 7f2052a00640 1 --2- 172.21.15.31:0/694006245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2054107aa0 0x7f205419e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:39:30.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.026+0000 7f2053400640 1 --2- 172.21.15.31:0/694006245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f205410c980 0x7f20541a2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:39:30.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.026+0000 7f2052000640 1 --2- 172.21.15.31:0/694006245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2054108440 0x7f205419e910 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f204000d950 tx=0x7f204000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:30.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.026+0000 7f204b400640 1 -- 172.21.15.31:0/694006245 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2040017070 con 0x7f2054108440 2024-03-21T10:39:30.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.026+0000 7f204b400640 1 -- 172.21.15.31:0/694006245 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2040009c30 con 0x7f2054108440 2024-03-21T10:39:30.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.026+0000 7f204b400640 1 -- 172.21.15.31:0/694006245 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204000f6b0 con 0x7f2054108440 2024-03-21T10:39:30.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.026+0000 7f2053e00640 1 -- 172.21.15.31:0/694006245 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20541a36b0 con 0x7f2054108440 2024-03-21T10:39:30.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.026+0000 7f2053e00640 1 -- 172.21.15.31:0/694006245 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20541aaf40 con 0x7f2054108440 2024-03-21T10:39:30.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.032+0000 7f205ca21740 1 Processor -- start 2024-03-21T10:39:30.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.032+0000 7f204b400640 1 -- 172.21.15.31:0/694006245 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) v1 ==== 138087+0+0 (secure 0 0 0) 0x7f20400109d0 con 0x7f2054108440 2024-03-21T10:39:30.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.033+0000 7f205ca21740 1 -- start start 2024-03-21T10:39:30.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.033+0000 7f205ca21740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baa97addd0 0x55baa97acb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:30.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.033+0000 7f205ca21740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55baa97ad0f0 0x55baa97ad570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:30.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.033+0000 7f205ca21740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa97b6460 0x55baa97b8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:30.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.033+0000 7f205ca21740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baa9711250 con 0x55baa97b6460 2024-03-21T10:39:30.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.033+0000 7f205ca21740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baa96b9a30 con 0x55baa97addd0 2024-03-21T10:39:30.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.033+0000 7f205ca21740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baa970d2f0 con 0x55baa97ad0f0 2024-03-21T10:39:30.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.033+0000 7f2052000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa97b6460 0x55baa97b8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:30.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.033+0000 7f2053400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baa97addd0 0x55baa97acb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:30.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.034+0000 7f2052000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa97b6460 0x55baa97b8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33598/0 (socket says 172.21.15.31:33598) 2024-03-21T10:39:30.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.034+0000 7f2053400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baa97addd0 0x55baa97acb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49620/0 (socket says 172.21.15.31:49620) 2024-03-21T10:39:30.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.034+0000 7f2052000640 1 -- 172.21.15.31:0/1775122021 learned_addr learned my addr 172.21.15.31:0/1775122021 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:30.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.034+0000 7f2052a00640 1 --2- 172.21.15.31:0/1775122021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55baa97ad0f0 0x55baa97ad570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:30.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.034+0000 7f2052000640 1 -- 172.21.15.31:0/1775122021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55baa97ad0f0 msgr2=0x55baa97ad570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:30.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.034+0000 7f2052000640 1 --2- 172.21.15.31:0/1775122021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55baa97ad0f0 0x55baa97ad570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.034+0000 7f2052000640 1 -- 172.21.15.31:0/1775122021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baa97addd0 msgr2=0x55baa97acb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:30.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.034+0000 7f2052000640 1 --2- 172.21.15.31:0/1775122021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baa97addd0 0x55baa97acb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.034+0000 7f2052000640 1 -- 172.21.15.31:0/1775122021 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55baa96a4690 con 0x55baa97b6460 2024-03-21T10:39:30.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.034+0000 7f2053400640 1 --2- 172.21.15.31:0/1775122021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baa97addd0 0x55baa97acb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:39:30.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.034+0000 7f2052000640 1 --2- 172.21.15.31:0/1775122021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa97b6460 0x55baa97b8850 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f20400cc4f0 tx=0x7f20400d7990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14f3ba5246cc6a7e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:30.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.035+0000 7f204b400640 1 --2- 172.21.15.31:0/694006245 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f20380a3280 0x7f20380a5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:30.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.035+0000 7f204b400640 1 -- 172.21.15.31:0/694006245 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f20400cf2c0 con 0x7f2054108440 2024-03-21T10:39:30.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.035+0000 7f2053400640 1 --2- 172.21.15.31:0/694006245 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f20380a3280 0x7f20380a5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:30.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.036+0000 7f203fe00640 1 -- 172.21.15.31:0/1775122021 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20400e3070 con 0x55baa97b6460 2024-03-21T10:39:30.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.036+0000 7f203fe00640 1 -- 172.21.15.31:0/1775122021 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f204009f270 con 0x55baa97b6460 2024-03-21T10:39:30.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.036+0000 7f203fe00640 1 -- 172.21.15.31:0/1775122021 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204009b820 con 0x55baa97b6460 2024-03-21T10:39:30.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.036+0000 7f2053400640 1 --2- 172.21.15.31:0/694006245 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f20380a3280 0x7f20380a5740 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f204c009370 tx=0x7f204c009520 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:30.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.037+0000 7f205ca21740 1 -- 172.21.15.31:0/1775122021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa97b6460 msgr2=0x55baa97b8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:30.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.037+0000 7f205ca21740 1 --2- 172.21.15.31:0/1775122021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa97b6460 0x55baa97b8850 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f20400cc4f0 tx=0x7f20400d7990 comp rx=0 tx=0).stop 2024-03-21T10:39:30.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.037+0000 7f205ca21740 1 -- 172.21.15.31:0/1775122021 shutdown_connections 2024-03-21T10:39:30.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.037+0000 7f205ca21740 1 --2- 172.21.15.31:0/1775122021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa97b6460 0x55baa97b8850 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.037+0000 7f205ca21740 1 --2- 172.21.15.31:0/1775122021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55baa97ad0f0 0x55baa97ad570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.037+0000 7f205ca21740 1 --2- 172.21.15.31:0/1775122021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baa97addd0 0x55baa97acb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.037+0000 7f205ca21740 1 -- 172.21.15.31:0/1775122021 >> 172.21.15.31:0/1775122021 conn(0x55baa9774200 msgr2=0x55baa97a7b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:30.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.037+0000 7f205ca21740 1 -- 172.21.15.31:0/1775122021 shutdown_connections 2024-03-21T10:39:30.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.038+0000 7f205ca21740 1 -- 172.21.15.31:0/1775122021 wait complete. 2024-03-21T10:39:30.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.038+0000 7f205ca21740 1 -- start start 2024-03-21T10:39:30.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.038+0000 7f205ca21740 1 Processor -- start 2024-03-21T10:39:30.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.039+0000 7f205ca21740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55baa97ad0f0 0x55baa97b5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:30.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.039+0000 7f205ca21740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baa97addd0 0x55baa97b2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:30.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.039+0000 7f205ca21740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa97b6460 0x55baa97b33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:30.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.039+0000 7f205ca21740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baa97b92d0 con 0x55baa97b6460 2024-03-21T10:39:30.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.039+0000 7f205ca21740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baa96cce80 con 0x55baa97addd0 2024-03-21T10:39:30.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.039+0000 7f205ca21740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baa97b9450 con 0x55baa97ad0f0 2024-03-21T10:39:30.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.039+0000 7f205ca21740 10 client.0 _ll_register_callbacks cb 0x55baa96df130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:39:30.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.039+0000 7f2052000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baa97addd0 0x55baa97b2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:30.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.039+0000 7f2052a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55baa97ad0f0 0x55baa97b5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:30.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.039+0000 7f2052000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baa97addd0 0x55baa97b2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49628/0 (socket says 172.21.15.31:49628) 2024-03-21T10:39:30.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.039+0000 7f2052000640 1 -- 172.21.15.31:0/219028073 learned_addr learned my addr 172.21.15.31:0/219028073 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:30.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.040+0000 7f2053400640 1 --2- 172.21.15.31:0/219028073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa97b6460 0x55baa97b33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:30.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.040+0000 7f2052a00640 1 -- 172.21.15.31:0/219028073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baa97addd0 msgr2=0x55baa97b2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:30.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.040+0000 7f2052a00640 1 --2- 172.21.15.31:0/219028073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baa97addd0 0x55baa97b2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.040+0000 7f2052a00640 1 -- 172.21.15.31:0/219028073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa97b6460 msgr2=0x55baa97b33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:30.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.040+0000 7f2052a00640 1 --2- 172.21.15.31:0/219028073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa97b6460 0x55baa97b33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.040+0000 7f2052a00640 1 -- 172.21.15.31:0/219028073 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55baa94ef7d0 con 0x55baa97ad0f0 2024-03-21T10:39:30.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.040+0000 7f2053400640 1 --2- 172.21.15.31:0/219028073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa97b6460 0x55baa97b33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:39:30.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.040+0000 7f2052a00640 1 --2- 172.21.15.31:0/219028073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55baa97ad0f0 0x55baa97b5af0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2044009850 tx=0x7f2044004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d7e01ee616af087 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:30.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.040+0000 7f203cc00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:39:30.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.040+0000 7f203cc00640 1 -- 172.21.15.31:0/219028073 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2044043070 con 0x55baa97ad0f0 2024-03-21T10:39:30.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.041+0000 7f203cc00640 1 -- 172.21.15.31:0/219028073 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f204402c910 con 0x55baa97ad0f0 2024-03-21T10:39:30.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.041+0000 7f205ca21740 1 -- 172.21.15.31:0/219028073 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f204009f270 con 0x55baa97ad0f0 2024-03-21T10:39:30.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.041+0000 7f203cc00640 1 -- 172.21.15.31:0/219028073 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2044050050 con 0x55baa97ad0f0 2024-03-21T10:39:30.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.041+0000 7f205ca21740 10 client.25939 fetch_fsmap learned FSMap version 24 2024-03-21T10:39:30.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.041+0000 7f205ca21740 1 -- 172.21.15.31:0/219028073 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55baa94253f0 con 0x55baa97ad0f0 2024-03-21T10:39:30.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.041+0000 7f203cc00640 1 -- 172.21.15.31:0/219028073 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f204009f270 con 0x55baa97ad0f0 2024-03-21T10:39:30.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.042+0000 7f203cc00640 1 -- 172.21.15.31:0/219028073 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f20440035f0 con 0x55baa97ad0f0 2024-03-21T10:39:30.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.042+0000 7f205ca21740 10 client.25939 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:39:30.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.042+0000 7f205ca21740 10 client.25939 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:39:30.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.042+0000 7f205ca21740 20 client.25939 populate_metadata read hostname 'smithi031' 2024-03-21T10:39:30.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.042+0000 7f205ca21740 1 --2- 172.21.15.31:0/219028073 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55baa97c6370 0x55baa97c8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:30.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.042+0000 7f205ca21740 4 client.25939 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:39:30.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.042+0000 7f205ca21740 1 -- 172.21.15.31:0/219028073 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55baa971e790 con 0x55baa97c6370 2024-03-21T10:39:30.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.042+0000 7f2052000640 1 --2- 172.21.15.31:0/219028073 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55baa97c6370 0x55baa97c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:30.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.043+0000 7f2052000640 1 --2- 172.21.15.31:0/219028073 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55baa97c6370 0x55baa97c8760 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a19caed1e4c27005 server_cookie=9726ba57028ec8bc in_seq=0 out_seq=0 2024-03-21T10:39:30.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.043+0000 7f203cc00640 10 client.25939 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:30.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.043+0000 7f203cc00640 1 -- 172.21.15.31:0/219028073 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55baa971e790 con 0x55baa97c6370 2024-03-21T10:39:30.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.043+0000 7f203cc00640 10 client.25939 handle_command_reply: tid=0 2024-03-21T10:39:30.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.043+0000 7f205ca21740 1 client.25939 shutdown 2024-03-21T10:39:30.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.044+0000 7f205ca21740 1 -- 172.21.15.31:0/219028073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55baa97ad0f0 msgr2=0x55baa97b5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:30.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.044+0000 7f205ca21740 1 --2- 172.21.15.31:0/219028073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55baa97ad0f0 0x55baa97b5af0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2044009850 tx=0x7f2044004270 comp rx=0 tx=0).stop 2024-03-21T10:39:30.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.044+0000 7f205ca21740 1 -- 172.21.15.31:0/219028073 shutdown_connections 2024-03-21T10:39:30.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.044+0000 7f205ca21740 1 --2- 172.21.15.31:0/219028073 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55baa97c6370 0x55baa97c8760 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.044+0000 7f205ca21740 1 --2- 172.21.15.31:0/219028073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa97b6460 0x55baa97b33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.044+0000 7f205ca21740 1 --2- 172.21.15.31:0/219028073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baa97addd0 0x55baa97b2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.044+0000 7f205ca21740 1 --2- 172.21.15.31:0/219028073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55baa97ad0f0 0x55baa97b5af0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.044+0000 7f205ca21740 1 -- 172.21.15.31:0/219028073 >> 172.21.15.31:0/219028073 conn(0x55baa9774200 msgr2=0x55baa97a9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:30.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.044+0000 7f203cc00640 0 client.25939 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:30.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.044+0000 7f205ca21740 1 -- 172.21.15.31:0/219028073 shutdown_connections 2024-03-21T10:39:30.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.044+0000 7f205ca21740 1 -- 172.21.15.31:0/219028073 wait complete. 2024-03-21T10:39:30.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.044+0000 7f205ca21740 20 client.25939 trim_cache size 0 max 16384 2024-03-21T10:39:30.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.062+0000 7f205ca21740 1 Processor -- start 2024-03-21T10:39:30.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.062+0000 7f205ca21740 1 -- start start 2024-03-21T10:39:30.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.062+0000 7f205ca21740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa97ad0f0 0x55baa97b7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:30.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.062+0000 7f205ca21740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baa97addd0 0x55baa97b7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:30.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.062+0000 7f205ca21740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55baa97b6460 0x55baa97b5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:30.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.062+0000 7f205ca21740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20541b7240 con 0x55baa97ad0f0 2024-03-21T10:39:30.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.062+0000 7f205ca21740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20541b4d40 con 0x55baa97addd0 2024-03-21T10:39:30.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.062+0000 7f205ca21740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20541b2de0 con 0x55baa97b6460 2024-03-21T10:39:30.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.063+0000 7f2052a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa97ad0f0 0x55baa97b7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:30.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.063+0000 7f2052a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa97ad0f0 0x55baa97b7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33630/0 (socket says 172.21.15.31:33630) 2024-03-21T10:39:30.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.063+0000 7f2052a00640 1 -- 172.21.15.31:0/3940962562 learned_addr learned my addr 172.21.15.31:0/3940962562 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:30.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.063+0000 7f2053400640 1 --2- 172.21.15.31:0/3940962562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55baa97b6460 0x55baa97b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:30.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.063+0000 7f2052000640 1 --2- 172.21.15.31:0/3940962562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baa97addd0 0x55baa97b7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:30.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.063+0000 7f2053400640 1 -- 172.21.15.31:0/3940962562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baa97addd0 msgr2=0x55baa97b7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:30.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.063+0000 7f2053400640 1 --2- 172.21.15.31:0/3940962562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baa97addd0 0x55baa97b7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.063+0000 7f2053400640 1 -- 172.21.15.31:0/3940962562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa97ad0f0 msgr2=0x55baa97b7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:30.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.063+0000 7f2053400640 1 --2- 172.21.15.31:0/3940962562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa97ad0f0 0x55baa97b7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.063+0000 7f2053400640 1 -- 172.21.15.31:0/3940962562 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55baa93e50f0 con 0x55baa97b6460 2024-03-21T10:39:30.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.063+0000 7f2052a00640 1 --2- 172.21.15.31:0/3940962562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa97ad0f0 0x55baa97b7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:39:30.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.063+0000 7f2052000640 1 --2- 172.21.15.31:0/3940962562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baa97addd0 0x55baa97b7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:39:30.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.064+0000 7f2053400640 1 --2- 172.21.15.31:0/3940962562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55baa97b6460 0x55baa97b5540 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f204c011010 tx=0x7f204c009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68f28fe2f5c07483 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:30.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.064+0000 7f203f400640 1 -- 172.21.15.31:0/3940962562 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204c011820 con 0x55baa97b6460 2024-03-21T10:39:30.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.064+0000 7f203f400640 1 -- 172.21.15.31:0/3940962562 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f204c020430 con 0x55baa97b6460 2024-03-21T10:39:30.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.064+0000 7f203f400640 1 -- 172.21.15.31:0/3940962562 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204c028660 con 0x55baa97b6460 2024-03-21T10:39:30.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.065+0000 7f205ca21740 1 -- 172.21.15.31:0/3940962562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55baa97b6460 msgr2=0x55baa97b5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:30.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.065+0000 7f205ca21740 1 --2- 172.21.15.31:0/3940962562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55baa97b6460 0x55baa97b5540 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f204c011010 tx=0x7f204c009f90 comp rx=0 tx=0).stop 2024-03-21T10:39:30.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.065+0000 7f205ca21740 1 -- 172.21.15.31:0/3940962562 shutdown_connections 2024-03-21T10:39:30.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.065+0000 7f205ca21740 1 --2- 172.21.15.31:0/3940962562 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55baa97b6460 0x55baa97b5540 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.065+0000 7f205ca21740 1 --2- 172.21.15.31:0/3940962562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baa97addd0 0x55baa97b7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.065+0000 7f205ca21740 1 --2- 172.21.15.31:0/3940962562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa97ad0f0 0x55baa97b7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.065+0000 7f205ca21740 1 -- 172.21.15.31:0/3940962562 >> 172.21.15.31:0/3940962562 conn(0x55baa9774200 msgr2=0x55baa97a7670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:30.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.065+0000 7f205ca21740 1 -- 172.21.15.31:0/3940962562 shutdown_connections 2024-03-21T10:39:30.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.066+0000 7f205ca21740 1 -- 172.21.15.31:0/3940962562 wait complete. 2024-03-21T10:39:30.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.066+0000 7f205ca21740 1 -- start start 2024-03-21T10:39:30.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.066+0000 7f205ca21740 1 Processor -- start 2024-03-21T10:39:30.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.067+0000 7f205ca21740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55baa97ad0f0 0x55baa97af830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:30.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.067+0000 7f205ca21740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa97addd0 0x55baa97afd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:30.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.067+0000 7f205ca21740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baa97b6460 0x55baa9820bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:30.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.067+0000 7f205ca21740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20541b33e0 con 0x55baa97addd0 2024-03-21T10:39:30.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.067+0000 7f205ca21740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20541b46e0 con 0x55baa97b6460 2024-03-21T10:39:30.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.067+0000 7f205ca21740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20541b3a20 con 0x55baa97ad0f0 2024-03-21T10:39:30.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.067+0000 7f205ca21740 10 client.0 _ll_register_callbacks cb 0x7f2044009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:39:30.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.067+0000 7f2052a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55baa97ad0f0 0x55baa97af830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:30.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.067+0000 7f2052a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55baa97ad0f0 0x55baa97af830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:47352/0 (socket says 172.21.15.31:47352) 2024-03-21T10:39:30.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.067+0000 7f2052a00640 1 -- 172.21.15.31:0/181449009 learned_addr learned my addr 172.21.15.31:0/181449009 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:30.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.068+0000 7f2053400640 1 --2- 172.21.15.31:0/181449009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baa97b6460 0x55baa9820bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:30.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.068+0000 7f2052a00640 1 -- 172.21.15.31:0/181449009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baa97b6460 msgr2=0x55baa9820bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:30.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.068+0000 7f2052a00640 1 --2- 172.21.15.31:0/181449009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baa97b6460 0x55baa9820bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.068+0000 7f2052a00640 1 -- 172.21.15.31:0/181449009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa97addd0 msgr2=0x55baa97afd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:30.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.068+0000 7f2052000640 1 --2- 172.21.15.31:0/181449009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa97addd0 0x55baa97afd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:30.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.068+0000 7f2052a00640 1 --2- 172.21.15.31:0/181449009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa97addd0 0x55baa97afd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.068+0000 7f2052a00640 1 -- 172.21.15.31:0/181449009 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55baa97b0620 con 0x55baa97ad0f0 2024-03-21T10:39:30.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.068+0000 7f2052000640 1 --2- 172.21.15.31:0/181449009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa97addd0 0x55baa97afd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:39:30.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.069+0000 7f2052a00640 1 --2- 172.21.15.31:0/181449009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55baa97ad0f0 0x55baa97af830 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2044009850 tx=0x7f20440038a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41430c91acd16019 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:30.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.069+0000 7f203cc00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:39:30.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.069+0000 7f203cc00640 1 -- 172.21.15.31:0/181449009 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2044043040 con 0x55baa97ad0f0 2024-03-21T10:39:30.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.069+0000 7f203cc00640 1 -- 172.21.15.31:0/181449009 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f204402cd20 con 0x55baa97ad0f0 2024-03-21T10:39:30.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.069+0000 7f205ca21740 1 -- 172.21.15.31:0/181449009 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f204c020430 con 0x55baa97ad0f0 2024-03-21T10:39:30.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.069+0000 7f203cc00640 1 -- 172.21.15.31:0/181449009 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f204402c340 con 0x55baa97ad0f0 2024-03-21T10:39:30.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.069+0000 7f205ca21740 10 client.25945 fetch_fsmap learned FSMap version 24 2024-03-21T10:39:30.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.069+0000 7f205ca21740 1 -- 172.21.15.31:0/181449009 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55baa98221a0 con 0x55baa97ad0f0 2024-03-21T10:39:30.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.070+0000 7f203cc00640 1 -- 172.21.15.31:0/181449009 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f204c020430 con 0x55baa97ad0f0 2024-03-21T10:39:30.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.070+0000 7f203cc00640 1 -- 172.21.15.31:0/181449009 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f2044055440 con 0x55baa97ad0f0 2024-03-21T10:39:30.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.070+0000 7f205ca21740 10 client.25945 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:39:30.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.070+0000 7f205ca21740 10 client.25945 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:39:30.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.070+0000 7f205ca21740 20 client.25945 populate_metadata read hostname 'smithi031' 2024-03-21T10:39:30.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.070+0000 7f205ca21740 1 --2- 172.21.15.31:0/181449009 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55baa9822800 0x55baa9824bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:30.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.070+0000 7f205ca21740 4 client.25945 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:39:30.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.070+0000 7f205ca21740 1 -- 172.21.15.31:0/181449009 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55baa9718ae0 con 0x55baa9822800 2024-03-21T10:39:30.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.071+0000 7f2052000640 1 --2- 172.21.15.31:0/181449009 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55baa9822800 0x55baa9824bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:30.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.071+0000 7f2052000640 1 --2- 172.21.15.31:0/181449009 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55baa9822800 0x55baa9824bf0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e530acea289176de server_cookie=7df035f2530c7906 in_seq=0 out_seq=0 2024-03-21T10:39:30.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.071+0000 7f203cc00640 10 client.25945 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:30.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.071+0000 7f203cc00640 1 -- 172.21.15.31:0/181449009 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55baa9718ae0 con 0x55baa9822800 2024-03-21T10:39:30.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.071+0000 7f203cc00640 10 client.25945 handle_command_reply: tid=0 2024-03-21T10:39:30.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.071+0000 7f205ca21740 1 client.25945 shutdown 2024-03-21T10:39:30.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.072+0000 7f205ca21740 1 -- 172.21.15.31:0/181449009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55baa97ad0f0 msgr2=0x55baa97af830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:30.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.072+0000 7f205ca21740 1 --2- 172.21.15.31:0/181449009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55baa97ad0f0 0x55baa97af830 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2044009850 tx=0x7f20440038a0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.072+0000 7f205ca21740 1 -- 172.21.15.31:0/181449009 shutdown_connections 2024-03-21T10:39:30.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.072+0000 7f205ca21740 1 --2- 172.21.15.31:0/181449009 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55baa9822800 0x55baa9824bf0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.072+0000 7f205ca21740 1 --2- 172.21.15.31:0/181449009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baa97b6460 0x55baa9820bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.072+0000 7f205ca21740 1 --2- 172.21.15.31:0/181449009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa97addd0 0x55baa97afd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.072+0000 7f203cc00640 0 client.25945 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:30.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.072+0000 7f205ca21740 1 --2- 172.21.15.31:0/181449009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55baa97ad0f0 0x55baa97af830 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.072+0000 7f205ca21740 1 -- 172.21.15.31:0/181449009 >> 172.21.15.31:0/181449009 conn(0x55baa9774200 msgr2=0x55baa97b1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:30.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.072+0000 7f205ca21740 1 -- 172.21.15.31:0/181449009 shutdown_connections 2024-03-21T10:39:30.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.072+0000 7f205ca21740 1 -- 172.21.15.31:0/181449009 wait complete. 2024-03-21T10:39:30.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.072+0000 7f205ca21740 20 client.25945 trim_cache size 0 max 16384 2024-03-21T10:39:30.071 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:39:30.072 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-21T10:39:30.072 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-03-21T10:39:30.072 INFO:teuthology.orchestra.run.smithi031.stdout: "97323de0-3dac-4d64-805a-f7eaa0398908": { 2024-03-21T10:39:30.072 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-03-21T10:39:30.072 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "97323de0-3dac-4d64-805a-f7eaa0398908", 2024-03-21T10:39:30.072 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-03-21T10:39:30.072 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:39:30.072 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:39:30.072 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:39:30.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.073+0000 7f2053e00640 1 -- 172.21.15.31:0/694006245 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f20380a3280 msgr2=0x7f20380a5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:30.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.073+0000 7f2053e00640 1 --2- 172.21.15.31:0/694006245 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f20380a3280 0x7f20380a5740 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f204c009370 tx=0x7f204c009520 comp rx=0 tx=0).stop 2024-03-21T10:39:30.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.073+0000 7f2053e00640 1 -- 172.21.15.31:0/694006245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2054108440 msgr2=0x7f205419e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:30.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.073+0000 7f2053e00640 1 --2- 172.21.15.31:0/694006245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2054108440 0x7f205419e910 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f204000d950 tx=0x7f204000de10 comp rx=0 tx=0).stop 2024-03-21T10:39:30.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.073+0000 7f2053e00640 1 -- 172.21.15.31:0/694006245 shutdown_connections 2024-03-21T10:39:30.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.073+0000 7f2053e00640 1 --2- 172.21.15.31:0/694006245 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f20380a3280 0x7f20380a5740 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.073+0000 7f2053e00640 1 --2- 172.21.15.31:0/694006245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f205410c980 0x7f20541a2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.073+0000 7f2053e00640 1 --2- 172.21.15.31:0/694006245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2054108440 0x7f205419e910 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.073+0000 7f2053e00640 1 --2- 172.21.15.31:0/694006245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2054107aa0 0x7f205419e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:30.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.073+0000 7f2053e00640 1 -- 172.21.15.31:0/694006245 >> 172.21.15.31:0/694006245 conn(0x7f20540772a0 msgr2=0x7f2054077c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:30.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.074+0000 7f2053e00640 1 -- 172.21.15.31:0/694006245 shutdown_connections 2024-03-21T10:39:30.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:39:30.074+0000 7f2053e00640 1 -- 172.21.15.31:0/694006245 wait complete. 2024-03-21T10:39:30.095 INFO:tasks.cephfs.filesystem:scrub status for tag:97323de0-3dac-4d64-805a-f7eaa0398908 - {'path': '/', 'tag': '97323de0-3dac-4d64-805a-f7eaa0398908', 'options': 'recursive,force'} 2024-03-21T10:39:30.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:30 smithi040 ceph-mon[36154]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-21T10:39:30.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:30 smithi040 ceph-mon[36154]: scrub queued for path: / 2024-03-21T10:39:30.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:30 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:39:30.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:30 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:39:30.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:30 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:39:31.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:30 smithi031 ceph-mon[31673]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-21T10:39:31.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:30 smithi031 ceph-mon[31673]: scrub queued for path: / 2024-03-21T10:39:31.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:30 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:39:31.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:30 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:39:31.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:30 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:39:31.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:30 smithi070 ceph-mon[36330]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-21T10:39:31.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:30 smithi070 ceph-mon[36330]: scrub queued for path: / 2024-03-21T10:39:31.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:30 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:39:31.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:30 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:39:31.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:30 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:39:31.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:31 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:39:32.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:31 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:39:32.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:31 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:39:32.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:32 smithi040 ceph-mon[36154]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-21T10:39:33.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:32 smithi031 ceph-mon[31673]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-21T10:39:33.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:32 smithi070 ceph-mon[36330]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-21T10:39:34.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:34 smithi040 ceph-mon[36154]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-21T10:39:35.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:34 smithi031 ceph-mon[31673]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-21T10:39:35.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:34 smithi070 ceph-mon[36330]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-21T10:39:36.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:36 smithi040 ceph-mon[36154]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-21T10:39:37.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:36 smithi031 ceph-mon[31673]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-21T10:39:37.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:36 smithi070 ceph-mon[36330]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-21T10:39:38.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:38 smithi040 ceph-mon[36154]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-21T10:39:39.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:38 smithi031 ceph-mon[31673]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-21T10:39:39.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:38 smithi070 ceph-mon[36330]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-21T10:39:40.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:40 smithi040 ceph-mon[36154]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-21T10:39:41.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:40 smithi031 ceph-mon[31673]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-21T10:39:41.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:40 smithi070 ceph-mon[36330]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-21T10:39:42.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.467+0000 7f1024e00640 1 -- 172.21.15.31:0/210785175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10201096a0 msgr2=0x7f1020109a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:42.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.467+0000 7f1024e00640 1 --2- 172.21.15.31:0/210785175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10201096a0 0x7f1020109a80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1010009930 tx=0x7f101002bc20 comp rx=0 tx=0).stop 2024-03-21T10:39:42.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.467+0000 7f1024e00640 1 -- 172.21.15.31:0/210785175 shutdown_connections 2024-03-21T10:39:42.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.467+0000 7f1024e00640 1 --2- 172.21.15.31:0/210785175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1020104050 0x7f1020110560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.467+0000 7f1024e00640 1 --2- 172.21.15.31:0/210785175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1020103690 0x7f1020103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.467+0000 7f1024e00640 1 --2- 172.21.15.31:0/210785175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10201096a0 0x7f1020109a80 secure :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1010009930 tx=0x7f101002bc20 comp rx=0 tx=0).stop 2024-03-21T10:39:42.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.467+0000 7f1024e00640 1 -- 172.21.15.31:0/210785175 >> 172.21.15.31:0/210785175 conn(0x7f10200ff180 msgr2=0x7f10201015a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:42.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.467+0000 7f1024e00640 1 -- 172.21.15.31:0/210785175 shutdown_connections 2024-03-21T10:39:42.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.469+0000 7f1024e00640 1 -- 172.21.15.31:0/210785175 wait complete. 2024-03-21T10:39:42.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.470+0000 7f1024e00640 1 Processor -- start 2024-03-21T10:39:42.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.470+0000 7f1024e00640 1 -- start start 2024-03-21T10:39:42.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.471+0000 7f1024e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1020103690 0x7f102019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:42.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.471+0000 7f1024e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1020104050 0x7f102019a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:42.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.471+0000 7f1024e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f102019ea70 0x7f102019ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:42.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.471+0000 7f1024e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1020113ac0 con 0x7f1020104050 2024-03-21T10:39:42.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.471+0000 7f1024e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1020113940 con 0x7f102019ea70 2024-03-21T10:39:42.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.471+0000 7f1024e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1020113c40 con 0x7f1020103690 2024-03-21T10:39:42.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.471+0000 7f101fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f102019ea70 0x7f102019ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:42.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.471+0000 7f101f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1020103690 0x7f102019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:42.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.471+0000 7f101fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f102019ea70 0x7f102019ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:55914/0 (socket says 172.21.15.31:55914) 2024-03-21T10:39:42.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.471+0000 7f101fe00640 1 -- 172.21.15.31:0/1055138071 learned_addr learned my addr 172.21.15.31:0/1055138071 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:42.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.471+0000 7f101ea00640 1 --2- 172.21.15.31:0/1055138071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1020104050 0x7f102019a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:42.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.472+0000 7f101f400640 1 -- 172.21.15.31:0/1055138071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f102019ea70 msgr2=0x7f102019ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:42.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.472+0000 7f101f400640 1 --2- 172.21.15.31:0/1055138071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f102019ea70 0x7f102019ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.472+0000 7f101f400640 1 -- 172.21.15.31:0/1055138071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1020104050 msgr2=0x7f102019a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:42.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.472+0000 7f101f400640 1 --2- 172.21.15.31:0/1055138071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1020104050 0x7f102019a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.472+0000 7f101f400640 1 -- 172.21.15.31:0/1055138071 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102019f560 con 0x7f1020103690 2024-03-21T10:39:42.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.472+0000 7f101fe00640 1 --2- 172.21.15.31:0/1055138071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f102019ea70 0x7f102019ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:39:42.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.472+0000 7f101f400640 1 --2- 172.21.15.31:0/1055138071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1020103690 0x7f102019a1b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1010008000 tx=0x7f10100042e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:42.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.473+0000 7f1017e00640 1 -- 172.21.15.31:0/1055138071 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1010043020 con 0x7f1020103690 2024-03-21T10:39:42.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.473+0000 7f1017e00640 1 -- 172.21.15.31:0/1055138071 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f10100044d0 con 0x7f1020103690 2024-03-21T10:39:42.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.473+0000 7f1024e00640 1 -- 172.21.15.31:0/1055138071 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f102019f7f0 con 0x7f1020103690 2024-03-21T10:39:42.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.473+0000 7f1017e00640 1 -- 172.21.15.31:0/1055138071 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1010003b50 con 0x7f1020103690 2024-03-21T10:39:42.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.473+0000 7f1024e00640 1 -- 172.21.15.31:0/1055138071 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10201a6fb0 con 0x7f1020103690 2024-03-21T10:39:42.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.478+0000 7f102916e740 1 Processor -- start 2024-03-21T10:39:42.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.478+0000 7f102916e740 1 -- start start 2024-03-21T10:39:42.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.478+0000 7f1017e00640 1 -- 172.21.15.31:0/1055138071 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) v1 ==== 138087+0+0 (secure 0 0 0) 0x7f1010046020 con 0x7f1020103690 2024-03-21T10:39:42.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.479+0000 7f102916e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8a51e0 0x55bfdc8a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:42.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.479+0000 7f102916e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfdc8a5b00 0x55bfdc8ae390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:42.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.479+0000 7f102916e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfdc8ae8d0 0x55bfdc8b0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:42.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.479+0000 7f102916e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfdc8673b0 con 0x55bfdc8ae8d0 2024-03-21T10:39:42.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.479+0000 7f102916e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfdc809830 con 0x55bfdc8a51e0 2024-03-21T10:39:42.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.479+0000 7f102916e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfdc7b25d0 con 0x55bfdc8a5b00 2024-03-21T10:39:42.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.479+0000 7f101fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8a51e0 0x55bfdc8a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:42.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.479+0000 7f101fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8a51e0 0x55bfdc8a55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:55928/0 (socket says 172.21.15.31:55928) 2024-03-21T10:39:42.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.479+0000 7f101fe00640 1 -- 172.21.15.31:0/374886718 learned_addr learned my addr 172.21.15.31:0/374886718 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:42.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.479+0000 7f101fe00640 1 -- 172.21.15.31:0/374886718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfdc8a5b00 msgr2=0x55bfdc8ae390 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:39:42.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.479+0000 7f101fe00640 1 --2- 172.21.15.31:0/374886718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfdc8a5b00 0x55bfdc8ae390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.479+0000 7f101fe00640 1 -- 172.21.15.31:0/374886718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfdc8ae8d0 msgr2=0x55bfdc8b0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:42.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.479+0000 7f101ea00640 1 --2- 172.21.15.31:0/374886718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfdc8ae8d0 0x55bfdc8b0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:42.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.480+0000 7f101fe00640 1 --2- 172.21.15.31:0/374886718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfdc8ae8d0 0x55bfdc8b0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.480+0000 7f101fe00640 1 -- 172.21.15.31:0/374886718 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bfdc7ca2c0 con 0x55bfdc8a51e0 2024-03-21T10:39:42.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.480+0000 7f101fe00640 1 --2- 172.21.15.31:0/374886718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8a51e0 0x55bfdc8a55c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f1018011850 tx=0x7f1018011d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c29b6909c7eff0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:42.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.480+0000 7f1014c00640 1 -- 172.21.15.31:0/374886718 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1018011330 con 0x55bfdc8a51e0 2024-03-21T10:39:42.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.480+0000 7f1014c00640 1 -- 172.21.15.31:0/374886718 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f101800fe80 con 0x55bfdc8a51e0 2024-03-21T10:39:42.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.480+0000 7f1014c00640 1 -- 172.21.15.31:0/374886718 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10180193e0 con 0x55bfdc8a51e0 2024-03-21T10:39:42.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.481+0000 7f1017e00640 1 --2- 172.21.15.31:0/1055138071 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f10040a3290 0x7f10040a5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:42.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.482+0000 7f1017e00640 1 -- 172.21.15.31:0/1055138071 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f10100f1aa0 con 0x7f1020103690 2024-03-21T10:39:42.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.482+0000 7f101f400640 1 --2- 172.21.15.31:0/1055138071 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f10040a3290 0x7f10040a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:42.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.483+0000 7f101f400640 1 --2- 172.21.15.31:0/1055138071 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f10040a3290 0x7f10040a5750 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f10100eee40 tx=0x7f10100035f0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:42.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.492+0000 7f102916e740 1 -- 172.21.15.31:0/374886718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8a51e0 msgr2=0x55bfdc8a55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:42.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.492+0000 7f102916e740 1 --2- 172.21.15.31:0/374886718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8a51e0 0x55bfdc8a55c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f1018011850 tx=0x7f1018011d10 comp rx=0 tx=0).stop 2024-03-21T10:39:42.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.492+0000 7f102916e740 1 -- 172.21.15.31:0/374886718 shutdown_connections 2024-03-21T10:39:42.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.492+0000 7f102916e740 1 --2- 172.21.15.31:0/374886718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfdc8ae8d0 0x55bfdc8b0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.492+0000 7f102916e740 1 --2- 172.21.15.31:0/374886718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfdc8a5b00 0x55bfdc8ae390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.492+0000 7f102916e740 1 --2- 172.21.15.31:0/374886718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8a51e0 0x55bfdc8a55c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.492+0000 7f102916e740 1 -- 172.21.15.31:0/374886718 >> 172.21.15.31:0/374886718 conn(0x55bfdc89fb00 msgr2=0x55bfdc89fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:42.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.493+0000 7f102916e740 1 -- 172.21.15.31:0/374886718 shutdown_connections 2024-03-21T10:39:42.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.493+0000 7f102916e740 1 -- 172.21.15.31:0/374886718 wait complete. 2024-03-21T10:39:42.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.494+0000 7f102916e740 1 -- start start 2024-03-21T10:39:42.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.494+0000 7f102916e740 1 Processor -- start 2024-03-21T10:39:42.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.495+0000 7f102916e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfdc8a51e0 0x55bfdc8ab940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:42.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.495+0000 7f102916e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8a5b00 0x55bfdc8abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:42.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.495+0000 7f102916e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfdc8ae8d0 0x55bfdc8ac3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:42.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.495+0000 7f102916e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfdc7c5a20 con 0x55bfdc8ae8d0 2024-03-21T10:39:42.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.495+0000 7f102916e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfdc8058d0 con 0x55bfdc8a5b00 2024-03-21T10:39:42.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.495+0000 7f102916e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfdc8b1500 con 0x55bfdc8a51e0 2024-03-21T10:39:42.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.495+0000 7f102916e740 10 client.0 _ll_register_callbacks cb 0x55bfdc8021c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:39:42.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.495+0000 7f101ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8a5b00 0x55bfdc8abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:42.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.495+0000 7f101ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8a5b00 0x55bfdc8abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:55936/0 (socket says 172.21.15.31:55936) 2024-03-21T10:39:42.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.495+0000 7f101ea00640 1 -- 172.21.15.31:0/355195614 learned_addr learned my addr 172.21.15.31:0/355195614 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:42.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.496+0000 7f101fe00640 1 --2- 172.21.15.31:0/355195614 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfdc8ae8d0 0x55bfdc8ac3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:42.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.496+0000 7f101fe00640 1 --2- 172.21.15.31:0/355195614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfdc8a51e0 0x55bfdc8ab940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:42.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.496+0000 7f101ea00640 1 -- 172.21.15.31:0/355195614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfdc8a51e0 msgr2=0x55bfdc8ab940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:42.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.496+0000 7f101ea00640 1 --2- 172.21.15.31:0/355195614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfdc8a51e0 0x55bfdc8ab940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.496+0000 7f101ea00640 1 -- 172.21.15.31:0/355195614 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfdc8ae8d0 msgr2=0x55bfdc8ac3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:42.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.496+0000 7f101ea00640 1 --2- 172.21.15.31:0/355195614 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfdc8ae8d0 0x55bfdc8ac3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.496+0000 7f101ea00640 1 -- 172.21.15.31:0/355195614 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bfdc51e210 con 0x55bfdc8a5b00 2024-03-21T10:39:42.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.496+0000 7f101fe00640 1 --2- 172.21.15.31:0/355195614 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfdc8ae8d0 0x55bfdc8ac3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:39:42.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.496+0000 7f101ea00640 1 --2- 172.21.15.31:0/355195614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8a5b00 0x55bfdc8abe80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f100c00bda0 tx=0x7f100c0117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d50c618c6b828323 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:42.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.497+0000 7f1009600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:39:42.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.497+0000 7f1009600640 1 -- 172.21.15.31:0/355195614 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100c004220 con 0x55bfdc8a5b00 2024-03-21T10:39:42.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.497+0000 7f1009600640 1 -- 172.21.15.31:0/355195614 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f100c0043c0 con 0x55bfdc8a5b00 2024-03-21T10:39:42.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.497+0000 7f102916e740 1 -- 172.21.15.31:0/355195614 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bfdc816d70 con 0x55bfdc8a5b00 2024-03-21T10:39:42.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.497+0000 7f1009600640 1 -- 172.21.15.31:0/355195614 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f100c025050 con 0x55bfdc8a5b00 2024-03-21T10:39:42.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.497+0000 7f102916e740 10 client.25970 fetch_fsmap learned FSMap version 24 2024-03-21T10:39:42.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.497+0000 7f102916e740 1 -- 172.21.15.31:0/355195614 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55bfdc5ea620 con 0x55bfdc8a5b00 2024-03-21T10:39:42.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.498+0000 7f1009600640 1 -- 172.21.15.31:0/355195614 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55bfdc816d70 con 0x55bfdc8a5b00 2024-03-21T10:39:42.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.498+0000 7f1009600640 1 -- 172.21.15.31:0/355195614 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f100c0039c0 con 0x55bfdc8a5b00 2024-03-21T10:39:42.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.498+0000 7f102916e740 10 client.25970 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:39:42.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.498+0000 7f102916e740 10 client.25970 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:39:42.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.498+0000 7f102916e740 20 client.25970 populate_metadata read hostname 'smithi031' 2024-03-21T10:39:42.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.498+0000 7f102916e740 1 --2- 172.21.15.31:0/355195614 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55bfdc8c7190 0x55bfdc8c9580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:42.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.498+0000 7f102916e740 4 client.25970 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:39:42.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.498+0000 7f102916e740 1 -- 172.21.15.31:0/355195614 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bfdc8110c0 con 0x55bfdc8c7190 2024-03-21T10:39:42.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.499+0000 7f101fe00640 1 --2- 172.21.15.31:0/355195614 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55bfdc8c7190 0x55bfdc8c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:42.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.499+0000 7f101fe00640 1 --2- 172.21.15.31:0/355195614 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55bfdc8c7190 0x55bfdc8c9580 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14b6d8d30dcefce4 server_cookie=558b1093e3afe520 in_seq=0 out_seq=0 2024-03-21T10:39:42.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.499+0000 7f1009600640 10 client.25970 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:42.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.500+0000 7f1009600640 1 -- 172.21.15.31:0/355195614 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55bfdc8110c0 con 0x55bfdc8c7190 2024-03-21T10:39:42.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.500+0000 7f1009600640 10 client.25970 handle_command_reply: tid=0 2024-03-21T10:39:42.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.500+0000 7f102916e740 1 client.25970 shutdown 2024-03-21T10:39:42.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.500+0000 7f102916e740 1 -- 172.21.15.31:0/355195614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8a5b00 msgr2=0x55bfdc8abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:42.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.500+0000 7f102916e740 1 --2- 172.21.15.31:0/355195614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8a5b00 0x55bfdc8abe80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f100c00bda0 tx=0x7f100c0117b0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.501+0000 7f102916e740 1 -- 172.21.15.31:0/355195614 shutdown_connections 2024-03-21T10:39:42.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.501+0000 7f102916e740 1 --2- 172.21.15.31:0/355195614 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55bfdc8c7190 0x55bfdc8c9580 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.501+0000 7f102916e740 1 --2- 172.21.15.31:0/355195614 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfdc8ae8d0 0x55bfdc8ac3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.501+0000 7f102916e740 1 --2- 172.21.15.31:0/355195614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8a5b00 0x55bfdc8abe80 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.501+0000 7f102916e740 1 --2- 172.21.15.31:0/355195614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfdc8a51e0 0x55bfdc8ab940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.501+0000 7f102916e740 1 -- 172.21.15.31:0/355195614 >> 172.21.15.31:0/355195614 conn(0x55bfdc89fb00 msgr2=0x55bfdc8a1850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:42.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.501+0000 7f1009600640 0 client.25970 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:42.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.501+0000 7f102916e740 1 -- 172.21.15.31:0/355195614 shutdown_connections 2024-03-21T10:39:42.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.501+0000 7f102916e740 1 -- 172.21.15.31:0/355195614 wait complete. 2024-03-21T10:39:42.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.501+0000 7f102916e740 20 client.25970 trim_cache size 0 max 16384 2024-03-21T10:39:42.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.531+0000 7f102916e740 1 Processor -- start 2024-03-21T10:39:42.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.531+0000 7f102916e740 1 -- start start 2024-03-21T10:39:42.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.531+0000 7f102916e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8a51e0 0x55bfdc8ae070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:42.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.531+0000 7f102916e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfdc8a5b00 0x55bfdc8c2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:42.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.531+0000 7f102916e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfdc8ae8d0 0x55bfdc8c29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:42.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.531+0000 7f102916e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10201aeef0 con 0x55bfdc8ae8d0 2024-03-21T10:39:42.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.531+0000 7f102916e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10201ac9f0 con 0x55bfdc8a51e0 2024-03-21T10:39:42.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.531+0000 7f102916e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10201aaa90 con 0x55bfdc8a5b00 2024-03-21T10:39:42.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.532+0000 7f101fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8a51e0 0x55bfdc8ae070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:42.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.532+0000 7f101fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8a51e0 0x55bfdc8ae070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:55938/0 (socket says 172.21.15.31:55938) 2024-03-21T10:39:42.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.532+0000 7f101fe00640 1 -- 172.21.15.31:0/1820039672 learned_addr learned my addr 172.21.15.31:0/1820039672 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:42.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.532+0000 7f101fe00640 1 --2- 172.21.15.31:0/1820039672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfdc8ae8d0 0x55bfdc8c29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:42.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.532+0000 7f101ea00640 1 --2- 172.21.15.31:0/1820039672 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfdc8a5b00 0x55bfdc8c2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:42.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.532+0000 7f101fe00640 1 -- 172.21.15.31:0/1820039672 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfdc8a5b00 msgr2=0x55bfdc8c2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:42.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.532+0000 7f101fe00640 1 --2- 172.21.15.31:0/1820039672 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfdc8a5b00 0x55bfdc8c2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.532+0000 7f101fe00640 1 -- 172.21.15.31:0/1820039672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfdc8ae8d0 msgr2=0x55bfdc8c29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:42.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.532+0000 7f101fe00640 1 --2- 172.21.15.31:0/1820039672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfdc8ae8d0 0x55bfdc8c29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.532+0000 7f101fe00640 1 -- 172.21.15.31:0/1820039672 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bfdc8c98f0 con 0x55bfdc8a51e0 2024-03-21T10:39:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.533+0000 7f101fe00640 1 --2- 172.21.15.31:0/1820039672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8a51e0 0x55bfdc8ae070 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1018009270 tx=0x7f1018000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b4bfbc2a5cca18e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.533+0000 7f100be00640 1 -- 172.21.15.31:0/1820039672 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1018010530 con 0x55bfdc8a51e0 2024-03-21T10:39:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.533+0000 7f100be00640 1 -- 172.21.15.31:0/1820039672 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1018020040 con 0x55bfdc8a51e0 2024-03-21T10:39:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.533+0000 7f100be00640 1 -- 172.21.15.31:0/1820039672 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1018011460 con 0x55bfdc8a51e0 2024-03-21T10:39:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.534+0000 7f102916e740 1 -- 172.21.15.31:0/1820039672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8a51e0 msgr2=0x55bfdc8ae070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.534+0000 7f102916e740 1 --2- 172.21.15.31:0/1820039672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8a51e0 0x55bfdc8ae070 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1018009270 tx=0x7f1018000f40 comp rx=0 tx=0).stop 2024-03-21T10:39:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.534+0000 7f102916e740 1 -- 172.21.15.31:0/1820039672 shutdown_connections 2024-03-21T10:39:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.534+0000 7f102916e740 1 --2- 172.21.15.31:0/1820039672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfdc8ae8d0 0x55bfdc8c29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.534+0000 7f102916e740 1 --2- 172.21.15.31:0/1820039672 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfdc8a5b00 0x55bfdc8c2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.534+0000 7f102916e740 1 --2- 172.21.15.31:0/1820039672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8a51e0 0x55bfdc8ae070 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.534+0000 7f102916e740 1 -- 172.21.15.31:0/1820039672 >> 172.21.15.31:0/1820039672 conn(0x55bfdc89fb00 msgr2=0x55bfdc8a6c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.534+0000 7f102916e740 1 -- 172.21.15.31:0/1820039672 shutdown_connections 2024-03-21T10:39:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.534+0000 7f102916e740 1 -- 172.21.15.31:0/1820039672 wait complete. 2024-03-21T10:39:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.535+0000 7f102916e740 1 -- start start 2024-03-21T10:39:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.535+0000 7f102916e740 1 Processor -- start 2024-03-21T10:39:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.536+0000 7f102916e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfdc8a51e0 0x55bfdc8ba560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.536+0000 7f102916e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfdc8a5b00 0x55bfdc8baaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.536+0000 7f102916e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8ae8d0 0x55bfdc8bafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.536+0000 7f102916e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10201ab090 con 0x55bfdc8a51e0 2024-03-21T10:39:42.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.536+0000 7f102916e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10201ac390 con 0x55bfdc8ae8d0 2024-03-21T10:39:42.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.536+0000 7f102916e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10201ab6d0 con 0x55bfdc8a5b00 2024-03-21T10:39:42.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.536+0000 7f102916e740 10 client.0 _ll_register_callbacks cb 0x7f100c00b9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:39:42.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.536+0000 7f101ea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfdc8a5b00 0x55bfdc8baaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:42.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.536+0000 7f101ea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfdc8a5b00 0x55bfdc8baaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:45908/0 (socket says 172.21.15.31:45908) 2024-03-21T10:39:42.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.536+0000 7f101ea00640 1 -- 172.21.15.31:0/409800191 learned_addr learned my addr 172.21.15.31:0/409800191 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:42.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.536+0000 7f101fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfdc8a51e0 0x55bfdc8ba560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:42.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.536+0000 7f101fe00640 1 --2- 172.21.15.31:0/409800191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8ae8d0 0x55bfdc8bafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:42.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.537+0000 7f101ea00640 1 -- 172.21.15.31:0/409800191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8ae8d0 msgr2=0x55bfdc8bafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:42.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.537+0000 7f101ea00640 1 --2- 172.21.15.31:0/409800191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8ae8d0 0x55bfdc8bafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.537+0000 7f101ea00640 1 -- 172.21.15.31:0/409800191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfdc8a51e0 msgr2=0x55bfdc8ba560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:42.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.537+0000 7f101ea00640 1 --2- 172.21.15.31:0/409800191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfdc8a51e0 0x55bfdc8ba560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.537+0000 7f101ea00640 1 -- 172.21.15.31:0/409800191 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bfdc921b20 con 0x55bfdc8a5b00 2024-03-21T10:39:42.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.537+0000 7f101fe00640 1 --2- 172.21.15.31:0/409800191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8ae8d0 0x55bfdc8bafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:39:42.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.537+0000 7f101ea00640 1 --2- 172.21.15.31:0/409800191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfdc8a5b00 0x55bfdc8baaa0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f100c006010 tx=0x7f100c00b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ed13ebbfb97be33 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:42.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.537+0000 7f1009600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:39:42.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.537+0000 7f1009600640 1 -- 172.21.15.31:0/409800191 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100c005850 con 0x55bfdc8a5b00 2024-03-21T10:39:42.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.538+0000 7f1009600640 1 -- 172.21.15.31:0/409800191 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f100c004110 con 0x55bfdc8a5b00 2024-03-21T10:39:42.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.538+0000 7f102916e740 1 -- 172.21.15.31:0/409800191 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1018020040 con 0x55bfdc8a5b00 2024-03-21T10:39:42.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.538+0000 7f1009600640 1 -- 172.21.15.31:0/409800191 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f100c03a050 con 0x55bfdc8a5b00 2024-03-21T10:39:42.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.538+0000 7f102916e740 10 client.25957 fetch_fsmap learned FSMap version 24 2024-03-21T10:39:42.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.538+0000 7f102916e740 1 -- 172.21.15.31:0/409800191 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55bfdc922e30 con 0x55bfdc8a5b00 2024-03-21T10:39:42.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.539+0000 7f1009600640 1 -- 172.21.15.31:0/409800191 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1018020040 con 0x55bfdc8a5b00 2024-03-21T10:39:42.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.539+0000 7f1009600640 1 -- 172.21.15.31:0/409800191 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f100c0054d0 con 0x55bfdc8a5b00 2024-03-21T10:39:42.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.539+0000 7f102916e740 10 client.25957 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:39:42.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.539+0000 7f102916e740 10 client.25957 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:39:42.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.539+0000 7f102916e740 20 client.25957 populate_metadata read hostname 'smithi031' 2024-03-21T10:39:42.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.539+0000 7f102916e740 1 --2- 172.21.15.31:0/409800191 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55bfdc923530 0x55bfdc925950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:42.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.539+0000 7f102916e740 4 client.25957 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:39:42.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.539+0000 7f102916e740 1 -- 172.21.15.31:0/409800191 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55bfdc78f5d0 con 0x55bfdc923530 2024-03-21T10:39:42.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.539+0000 7f101fe00640 1 --2- 172.21.15.31:0/409800191 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55bfdc923530 0x55bfdc925950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.540+0000 7f101fe00640 1 --2- 172.21.15.31:0/409800191 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55bfdc923530 0x55bfdc925950 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cd9add67ae2cbd0 server_cookie=4de2e3ea1e37f621 in_seq=0 out_seq=0 2024-03-21T10:39:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.540+0000 7f1009600640 10 client.25957 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:42.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.543+0000 7f1009600640 1 -- 172.21.15.31:0/409800191 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55bfdc78f5d0 con 0x55bfdc923530 2024-03-21T10:39:42.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.543+0000 7f1009600640 10 client.25957 handle_command_reply: tid=0 2024-03-21T10:39:42.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.543+0000 7f102916e740 1 client.25957 shutdown 2024-03-21T10:39:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.544+0000 7f102916e740 1 -- 172.21.15.31:0/409800191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfdc8a5b00 msgr2=0x55bfdc8baaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.544+0000 7f102916e740 1 --2- 172.21.15.31:0/409800191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfdc8a5b00 0x55bfdc8baaa0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f100c006010 tx=0x7f100c00b940 comp rx=0 tx=0).stop 2024-03-21T10:39:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.544+0000 7f102916e740 1 -- 172.21.15.31:0/409800191 shutdown_connections 2024-03-21T10:39:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.544+0000 7f102916e740 1 --2- 172.21.15.31:0/409800191 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55bfdc923530 0x55bfdc925950 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.544+0000 7f1009600640 0 client.25957 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.544+0000 7f102916e740 1 --2- 172.21.15.31:0/409800191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bfdc8ae8d0 0x55bfdc8bafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.544+0000 7f102916e740 1 --2- 172.21.15.31:0/409800191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfdc8a5b00 0x55bfdc8baaa0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.544+0000 7f102916e740 1 --2- 172.21.15.31:0/409800191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfdc8a51e0 0x55bfdc8ba560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.544+0000 7f102916e740 1 -- 172.21.15.31:0/409800191 >> 172.21.15.31:0/409800191 conn(0x55bfdc89fb00 msgr2=0x55bfdc8a0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.546+0000 7f102916e740 1 -- 172.21.15.31:0/409800191 shutdown_connections 2024-03-21T10:39:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.546+0000 7f102916e740 1 -- 172.21.15.31:0/409800191 wait complete. 2024-03-21T10:39:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.546+0000 7f102916e740 20 client.25957 trim_cache size 0 max 16384 2024-03-21T10:39:42.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:39:42.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:39:42.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:39:42.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:39:42.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.547+0000 7f1024e00640 1 -- 172.21.15.31:0/1055138071 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f10040a3290 msgr2=0x7f10040a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:42.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.547+0000 7f1024e00640 1 --2- 172.21.15.31:0/1055138071 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f10040a3290 0x7f10040a5750 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f10100eee40 tx=0x7f10100035f0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.547+0000 7f1024e00640 1 -- 172.21.15.31:0/1055138071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1020103690 msgr2=0x7f102019a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:42.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.547+0000 7f1024e00640 1 --2- 172.21.15.31:0/1055138071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1020103690 0x7f102019a1b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1010008000 tx=0x7f10100042e0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.547+0000 7f1024e00640 1 -- 172.21.15.31:0/1055138071 shutdown_connections 2024-03-21T10:39:42.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.547+0000 7f1024e00640 1 --2- 172.21.15.31:0/1055138071 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f10040a3290 0x7f10040a5750 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.547+0000 7f1024e00640 1 --2- 172.21.15.31:0/1055138071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f102019ea70 0x7f102019ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.547+0000 7f1024e00640 1 --2- 172.21.15.31:0/1055138071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1020104050 0x7f102019a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.547+0000 7f1024e00640 1 --2- 172.21.15.31:0/1055138071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1020103690 0x7f102019a1b0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:42.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.547+0000 7f1024e00640 1 -- 172.21.15.31:0/1055138071 >> 172.21.15.31:0/1055138071 conn(0x7f10200ff180 msgr2=0x7f10200ffa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.548+0000 7f1024e00640 1 -- 172.21.15.31:0/1055138071 shutdown_connections 2024-03-21T10:39:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:42.548+0000 7f1024e00640 1 -- 172.21.15.31:0/1055138071 wait complete. 2024-03-21T10:39:42.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:42 smithi040 ceph-mon[36154]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-21T10:39:42.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:42 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:39:43.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:42 smithi031 ceph-mon[31673]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-21T10:39:43.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:42 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:39:43.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:42 smithi070 ceph-mon[36330]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-21T10:39:43.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:42 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:39:44.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:44 smithi040 ceph-mon[36154]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-21T10:39:44.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:44 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:39:44.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:44 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:39:44.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:44 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:39:45.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:44 smithi031 ceph-mon[31673]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-21T10:39:45.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:44 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:39:45.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:44 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:39:45.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:44 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:39:45.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:44 smithi070 ceph-mon[36330]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-21T10:39:45.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:44 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:39:45.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:44 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:39:45.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:44 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:39:46.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:46 smithi040 ceph-mon[36154]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-21T10:39:47.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:46 smithi031 ceph-mon[31673]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-21T10:39:47.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:46 smithi070 ceph-mon[36330]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-21T10:39:48.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:48 smithi040 ceph-mon[36154]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-21T10:39:49.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:48 smithi031 ceph-mon[31673]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-21T10:39:49.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:48 smithi070 ceph-mon[36330]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-21T10:39:50.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:50 smithi040 ceph-mon[36154]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-21T10:39:51.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:50 smithi031 ceph-mon[31673]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-21T10:39:51.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:50 smithi070 ceph-mon[36330]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-21T10:39:52.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:52 smithi040 ceph-mon[36154]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-21T10:39:53.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:52 smithi031 ceph-mon[31673]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-21T10:39:53.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:52 smithi070 ceph-mon[36330]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-21T10:39:54.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:54 smithi040 ceph-mon[36154]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-21T10:39:55.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:54 smithi031 ceph-mon[31673]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-21T10:39:55.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:54 smithi070 ceph-mon[36330]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-21T10:39:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.689+0000 7f4219400640 1 -- 172.21.15.31:0/3184697840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4214073a50 msgr2=0x7f4214073e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.689+0000 7f4219400640 1 --2- 172.21.15.31:0/3184697840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4214073a50 0x7f4214073e30 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4204009900 tx=0x7f420402bcc0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.690+0000 7f4219400640 1 -- 172.21.15.31:0/3184697840 shutdown_connections 2024-03-21T10:39:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.690+0000 7f4219400640 1 --2- 172.21.15.31:0/3184697840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f421410df10 0x7f4214110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.690+0000 7f4219400640 1 --2- 172.21.15.31:0/3184697840 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4214074370 0x7f421410d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.690+0000 7f4219400640 1 --2- 172.21.15.31:0/3184697840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4214073a50 0x7f4214073e30 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.690+0000 7f4219400640 1 -- 172.21.15.31:0/3184697840 >> 172.21.15.31:0/3184697840 conn(0x7f42140fd120 msgr2=0x7f42140ff540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.690+0000 7f4219400640 1 -- 172.21.15.31:0/3184697840 shutdown_connections 2024-03-21T10:39:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.690+0000 7f4219400640 1 -- 172.21.15.31:0/3184697840 wait complete. 2024-03-21T10:39:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.691+0000 7f4219400640 1 Processor -- start 2024-03-21T10:39:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.691+0000 7f4219400640 1 -- start start 2024-03-21T10:39:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.691+0000 7f4219400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4214073a50 0x7f421419a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.691+0000 7f4219400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4214074370 0x7f421419a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.691+0000 7f4219400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f421410df10 0x7f421419ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.691+0000 7f4219400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4214113c40 con 0x7f4214074370 2024-03-21T10:39:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.691+0000 7f4219400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4214113ac0 con 0x7f4214073a50 2024-03-21T10:39:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.691+0000 7f4219400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4214113dc0 con 0x7f421410df10 2024-03-21T10:39:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.692+0000 7f4212800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4214073a50 0x7f421419a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.692+0000 7f4212800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4214073a50 0x7f421419a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44034/0 (socket says 172.21.15.31:44034) 2024-03-21T10:39:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.692+0000 7f4212800640 1 -- 172.21.15.31:0/472898258 learned_addr learned my addr 172.21.15.31:0/472898258 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:55.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.692+0000 7f4211e00640 1 --2- 172.21.15.31:0/472898258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4214074370 0x7f421419a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:55.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.692+0000 7f4213200640 1 --2- 172.21.15.31:0/472898258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f421410df10 0x7f421419ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:55.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.692+0000 7f4212800640 1 -- 172.21.15.31:0/472898258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f421410df10 msgr2=0x7f421419ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:55.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.692+0000 7f4212800640 1 --2- 172.21.15.31:0/472898258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f421410df10 0x7f421419ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.692+0000 7f4212800640 1 -- 172.21.15.31:0/472898258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4214074370 msgr2=0x7f421419a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:55.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.692+0000 7f4212800640 1 --2- 172.21.15.31:0/472898258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4214074370 0x7f421419a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.692+0000 7f4212800640 1 -- 172.21.15.31:0/472898258 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f421419f140 con 0x7f4214073a50 2024-03-21T10:39:55.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.692+0000 7f4213200640 1 --2- 172.21.15.31:0/472898258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f421410df10 0x7f421419ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:39:55.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.693+0000 7f4212800640 1 --2- 172.21.15.31:0/472898258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4214073a50 0x7f421419a1a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f4204000c00 tx=0x7f4204004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:55.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.693+0000 7f420b400640 1 -- 172.21.15.31:0/472898258 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4204042020 con 0x7f4214073a50 2024-03-21T10:39:55.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.693+0000 7f420b400640 1 -- 172.21.15.31:0/472898258 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4204005000 con 0x7f4214073a50 2024-03-21T10:39:55.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.693+0000 7f420b400640 1 -- 172.21.15.31:0/472898258 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4204003b20 con 0x7f4214073a50 2024-03-21T10:39:55.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.693+0000 7f4219400640 1 -- 172.21.15.31:0/472898258 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f421419f3d0 con 0x7f4214073a50 2024-03-21T10:39:55.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.693+0000 7f4219400640 1 -- 172.21.15.31:0/472898258 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42141a6cc0 con 0x7f4214073a50 2024-03-21T10:39:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.698+0000 7f421c019740 1 Processor -- start 2024-03-21T10:39:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.698+0000 7f420b400640 1 -- 172.21.15.31:0/472898258 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) v1 ==== 138087+0+0 (secure 0 0 0) 0x7f4204045020 con 0x7f4214073a50 2024-03-21T10:39:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.699+0000 7f421c019740 1 -- start start 2024-03-21T10:39:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.699+0000 7f421c019740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5558b7892fd0 0x5558b78933b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.699+0000 7f421c019740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5558b78938f0 0x5558b789c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.699+0000 7f421c019740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5558b789c6c0 0x5558b789eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.699+0000 7f421c019740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558b78553b0 con 0x5558b78938f0 2024-03-21T10:39:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.699+0000 7f421c019740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558b77f7830 con 0x5558b7892fd0 2024-03-21T10:39:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.699+0000 7f421c019740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558b77a05d0 con 0x5558b789c6c0 2024-03-21T10:39:55.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.700+0000 7f420b400640 1 --2- 172.21.15.31:0/472898258 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f41f80a3220 0x7f41f80a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:55.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.700+0000 7f4211e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5558b789c6c0 0x5558b789eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:55.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.700+0000 7f4211e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5558b789c6c0 0x5558b789eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:58772/0 (socket says 172.21.15.31:58772) 2024-03-21T10:39:55.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.700+0000 7f4211e00640 1 -- 172.21.15.31:0/1709164873 learned_addr learned my addr 172.21.15.31:0/1709164873 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:55.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.700+0000 7f420b400640 1 -- 172.21.15.31:0/472898258 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f42040f1960 con 0x7f4214073a50 2024-03-21T10:39:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.700+0000 7f4213200640 1 --2- 172.21.15.31:0/472898258 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f41f80a3220 0x7f41f80a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.701+0000 7f4211e00640 1 -- 172.21.15.31:0/1709164873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5558b7892fd0 msgr2=0x5558b78933b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.701+0000 7f4211e00640 1 --2- 172.21.15.31:0/1709164873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5558b7892fd0 0x5558b78933b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.701+0000 7f4211e00640 1 -- 172.21.15.31:0/1709164873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5558b78938f0 msgr2=0x5558b789c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.700+0000 7f4212800640 1 --2- 172.21.15.31:0/1709164873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5558b78938f0 0x5558b789c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.701+0000 7f4211e00640 1 --2- 172.21.15.31:0/1709164873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5558b78938f0 0x5558b789c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.701+0000 7f4211e00640 1 -- 172.21.15.31:0/1709164873 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558b77b82c0 con 0x5558b789c6c0 2024-03-21T10:39:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.701+0000 7f4212800640 1 --2- 172.21.15.31:0/1709164873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5558b78938f0 0x5558b789c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:39:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.701+0000 7f4211e00640 1 --2- 172.21.15.31:0/1709164873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5558b789c6c0 0x5558b789eab0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4200009c20 tx=0x7f42000117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43909edc891a0d24 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.701+0000 7f4213200640 1 --2- 172.21.15.31:0/472898258 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f41f80a3220 0x7f41f80a56e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f420c009270 tx=0x7f420c0091e0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.702+0000 7f41ff400640 1 -- 172.21.15.31:0/1709164873 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4200012450 con 0x5558b789c6c0 2024-03-21T10:39:55.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.702+0000 7f41ff400640 1 -- 172.21.15.31:0/1709164873 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4200012ad0 con 0x5558b789c6c0 2024-03-21T10:39:55.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.702+0000 7f41ff400640 1 -- 172.21.15.31:0/1709164873 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420000fbf0 con 0x5558b789c6c0 2024-03-21T10:39:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.703+0000 7f421c019740 1 -- 172.21.15.31:0/1709164873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5558b789c6c0 msgr2=0x5558b789eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.703+0000 7f421c019740 1 --2- 172.21.15.31:0/1709164873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5558b789c6c0 0x5558b789eab0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4200009c20 tx=0x7f42000117b0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.703+0000 7f421c019740 1 -- 172.21.15.31:0/1709164873 shutdown_connections 2024-03-21T10:39:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.703+0000 7f421c019740 1 --2- 172.21.15.31:0/1709164873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5558b789c6c0 0x5558b789eab0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.703+0000 7f421c019740 1 --2- 172.21.15.31:0/1709164873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5558b78938f0 0x5558b789c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.703+0000 7f421c019740 1 --2- 172.21.15.31:0/1709164873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5558b7892fd0 0x5558b78933b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.703+0000 7f421c019740 1 -- 172.21.15.31:0/1709164873 >> 172.21.15.31:0/1709164873 conn(0x5558b788d8f0 msgr2=0x5558b788dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.703+0000 7f421c019740 1 -- 172.21.15.31:0/1709164873 shutdown_connections 2024-03-21T10:39:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.703+0000 7f421c019740 1 -- 172.21.15.31:0/1709164873 wait complete. 2024-03-21T10:39:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.704+0000 7f421c019740 1 -- start start 2024-03-21T10:39:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.704+0000 7f421c019740 1 Processor -- start 2024-03-21T10:39:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.705+0000 7f421c019740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5558b7892fd0 0x5558b7899710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.705+0000 7f421c019740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5558b78938f0 0x5558b7899c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.705+0000 7f421c019740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5558b789c6c0 0x5558b789a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.705+0000 7f421c019740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558b77b3a20 con 0x5558b7892fd0 2024-03-21T10:39:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.705+0000 7f421c019740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558b77f38d0 con 0x5558b789c6c0 2024-03-21T10:39:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.705+0000 7f421c019740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558b789f2f0 con 0x5558b78938f0 2024-03-21T10:39:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.705+0000 7f421c019740 10 client.0 _ll_register_callbacks cb 0x5558b77f01c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:39:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.705+0000 7f4212800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5558b7892fd0 0x5558b7899710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.705+0000 7f4213200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5558b789c6c0 0x5558b789a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.705+0000 7f4212800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5558b7892fd0 0x5558b7899710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42774/0 (socket says 172.21.15.31:42774) 2024-03-21T10:39:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.705+0000 7f4212800640 1 -- 172.21.15.31:0/1016892855 learned_addr learned my addr 172.21.15.31:0/1016892855 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.705+0000 7f4211e00640 1 --2- 172.21.15.31:0/1016892855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5558b78938f0 0x5558b7899c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.706+0000 7f4211e00640 1 -- 172.21.15.31:0/1016892855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5558b789c6c0 msgr2=0x5558b789a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.706+0000 7f4211e00640 1 --2- 172.21.15.31:0/1016892855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5558b789c6c0 0x5558b789a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.706+0000 7f4211e00640 1 -- 172.21.15.31:0/1016892855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5558b7892fd0 msgr2=0x5558b7899710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.706+0000 7f4211e00640 1 --2- 172.21.15.31:0/1016892855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5558b7892fd0 0x5558b7899710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.706+0000 7f4211e00640 1 -- 172.21.15.31:0/1016892855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5558b750c210 con 0x5558b78938f0 2024-03-21T10:39:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.706+0000 7f4211e00640 1 --2- 172.21.15.31:0/1016892855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5558b78938f0 0x5558b7899c50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f420001c040 tx=0x7f4200012eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e47d7d56437fb1c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.706+0000 7f41f3e00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:39:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.706+0000 7f41f3e00640 1 -- 172.21.15.31:0/1016892855 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4200028020 con 0x5558b78938f0 2024-03-21T10:39:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.706+0000 7f41f3e00640 1 -- 172.21.15.31:0/1016892855 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f420000f930 con 0x5558b78938f0 2024-03-21T10:39:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.707+0000 7f421c019740 1 -- 172.21.15.31:0/1016892855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4200012ad0 con 0x5558b78938f0 2024-03-21T10:39:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.707+0000 7f41f3e00640 1 -- 172.21.15.31:0/1016892855 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4200033050 con 0x5558b78938f0 2024-03-21T10:39:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.707+0000 7f421c019740 10 client.25969 fetch_fsmap learned FSMap version 24 2024-03-21T10:39:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.707+0000 7f421c019740 1 -- 172.21.15.31:0/1016892855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5558b75d8620 con 0x5558b78938f0 2024-03-21T10:39:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.707+0000 7f41f3e00640 1 -- 172.21.15.31:0/1016892855 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4200012ad0 con 0x5558b78938f0 2024-03-21T10:39:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.707+0000 7f41f3e00640 1 -- 172.21.15.31:0/1016892855 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f42000046c0 con 0x5558b78938f0 2024-03-21T10:39:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.708+0000 7f421c019740 10 client.25969 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:39:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.708+0000 7f421c019740 10 client.25969 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:39:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.708+0000 7f421c019740 20 client.25969 populate_metadata read hostname 'smithi031' 2024-03-21T10:39:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.708+0000 7f421c019740 1 --2- 172.21.15.31:0/1016892855 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5558b78ac5f0 0x5558b78ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.708+0000 7f421c019740 4 client.25969 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:39:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.708+0000 7f421c019740 1 -- 172.21.15.31:0/1016892855 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5558b7804d70 con 0x5558b78ac5f0 2024-03-21T10:39:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.708+0000 7f4212800640 1 --2- 172.21.15.31:0/1016892855 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5558b78ac5f0 0x5558b78ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.708+0000 7f41f3e00640 10 client.25969 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.709+0000 7f4212800640 1 --2- 172.21.15.31:0/1016892855 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5558b78ac5f0 0x5558b78ae9e0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a4f01480880a0a3 server_cookie=1f03b18b5aec164a in_seq=0 out_seq=0 2024-03-21T10:39:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.710+0000 7f41f3e00640 1 -- 172.21.15.31:0/1016892855 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5558b7804d70 con 0x5558b78ac5f0 2024-03-21T10:39:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.710+0000 7f41f3e00640 10 client.25969 handle_command_reply: tid=0 2024-03-21T10:39:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.710+0000 7f421c019740 1 client.25969 shutdown 2024-03-21T10:39:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.710+0000 7f421c019740 1 -- 172.21.15.31:0/1016892855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5558b78938f0 msgr2=0x5558b7899c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.710+0000 7f421c019740 1 --2- 172.21.15.31:0/1016892855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5558b78938f0 0x5558b7899c50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f420001c040 tx=0x7f4200012eb0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.710+0000 7f421c019740 1 -- 172.21.15.31:0/1016892855 shutdown_connections 2024-03-21T10:39:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.710+0000 7f421c019740 1 --2- 172.21.15.31:0/1016892855 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5558b78ac5f0 0x5558b78ae9e0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.710+0000 7f421c019740 1 --2- 172.21.15.31:0/1016892855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5558b789c6c0 0x5558b789a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.710+0000 7f421c019740 1 --2- 172.21.15.31:0/1016892855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5558b78938f0 0x5558b7899c50 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.710+0000 7f421c019740 1 --2- 172.21.15.31:0/1016892855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5558b7892fd0 0x5558b7899710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.710+0000 7f421c019740 1 -- 172.21.15.31:0/1016892855 >> 172.21.15.31:0/1016892855 conn(0x5558b788d8f0 msgr2=0x5558b788f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.710+0000 7f41f3e00640 0 client.25969 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.711+0000 7f421c019740 1 -- 172.21.15.31:0/1016892855 shutdown_connections 2024-03-21T10:39:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.711+0000 7f421c019740 1 -- 172.21.15.31:0/1016892855 wait complete. 2024-03-21T10:39:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.711+0000 7f421c019740 20 client.25969 trim_cache size 0 max 16384 2024-03-21T10:39:55.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.726+0000 7f421c019740 1 Processor -- start 2024-03-21T10:39:55.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.726+0000 7f421c019740 1 -- start start 2024-03-21T10:39:55.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.726+0000 7f421c019740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5558b7892fd0 0x5558b7899fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:55.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.726+0000 7f421c019740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5558b78938f0 0x5558b788f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:55.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.726+0000 7f421c019740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5558b789c6c0 0x5558b788f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:55.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.726+0000 7f421c019740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42141b3090 con 0x5558b78938f0 2024-03-21T10:39:55.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.726+0000 7f421c019740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42141b0b90 con 0x5558b789c6c0 2024-03-21T10:39:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.726+0000 7f421c019740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42141aec30 con 0x5558b7892fd0 2024-03-21T10:39:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.727+0000 7f4212800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5558b7892fd0 0x5558b7899fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.727+0000 7f4212800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5558b7892fd0 0x5558b7899fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:58780/0 (socket says 172.21.15.31:58780) 2024-03-21T10:39:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.727+0000 7f4212800640 1 -- 172.21.15.31:0/1110749455 learned_addr learned my addr 172.21.15.31:0/1110749455 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.727+0000 7f4211e00640 1 --2- 172.21.15.31:0/1110749455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5558b78938f0 0x5558b788f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.727+0000 7f4213200640 1 --2- 172.21.15.31:0/1110749455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5558b789c6c0 0x5558b788f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.727+0000 7f4212800640 1 -- 172.21.15.31:0/1110749455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5558b789c6c0 msgr2=0x5558b788f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.727+0000 7f4212800640 1 --2- 172.21.15.31:0/1110749455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5558b789c6c0 0x5558b788f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.727+0000 7f4212800640 1 -- 172.21.15.31:0/1110749455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5558b78938f0 msgr2=0x5558b788f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.727+0000 7f4212800640 1 --2- 172.21.15.31:0/1110749455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5558b78938f0 0x5558b788f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.727+0000 7f4212800640 1 -- 172.21.15.31:0/1110749455 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558b7897c80 con 0x5558b7892fd0 2024-03-21T10:39:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.728+0000 7f4212800640 1 --2- 172.21.15.31:0/1110749455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5558b7892fd0 0x5558b7899fb0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f42040b9d30 tx=0x7f4204005400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fb9a2aa09e75b84 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.728+0000 7f41fea00640 1 -- 172.21.15.31:0/1110749455 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42040bdfd0 con 0x5558b7892fd0 2024-03-21T10:39:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.728+0000 7f41fea00640 1 -- 172.21.15.31:0/1110749455 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42040bd8d0 con 0x5558b7892fd0 2024-03-21T10:39:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.728+0000 7f41fea00640 1 -- 172.21.15.31:0/1110749455 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42040b9f40 con 0x5558b7892fd0 2024-03-21T10:39:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.729+0000 7f421c019740 1 -- 172.21.15.31:0/1110749455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5558b7892fd0 msgr2=0x5558b7899fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.729+0000 7f421c019740 1 --2- 172.21.15.31:0/1110749455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5558b7892fd0 0x5558b7899fb0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f42040b9d30 tx=0x7f4204005400 comp rx=0 tx=0).stop 2024-03-21T10:39:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.729+0000 7f421c019740 1 -- 172.21.15.31:0/1110749455 shutdown_connections 2024-03-21T10:39:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.729+0000 7f421c019740 1 --2- 172.21.15.31:0/1110749455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5558b789c6c0 0x5558b788f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.729+0000 7f421c019740 1 --2- 172.21.15.31:0/1110749455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5558b78938f0 0x5558b788f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.729+0000 7f421c019740 1 --2- 172.21.15.31:0/1110749455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5558b7892fd0 0x5558b7899fb0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.729+0000 7f421c019740 1 -- 172.21.15.31:0/1110749455 >> 172.21.15.31:0/1110749455 conn(0x5558b788d8f0 msgr2=0x5558b788eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:55.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.729+0000 7f421c019740 1 -- 172.21.15.31:0/1110749455 shutdown_connections 2024-03-21T10:39:55.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.729+0000 7f421c019740 1 -- 172.21.15.31:0/1110749455 wait complete. 2024-03-21T10:39:55.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.730+0000 7f421c019740 1 -- start start 2024-03-21T10:39:55.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.730+0000 7f421c019740 1 Processor -- start 2024-03-21T10:39:55.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.730+0000 7f421c019740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5558b7892fd0 0x5558b78969c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:55.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.730+0000 7f421c019740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5558b78938f0 0x5558b7896f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:55.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.730+0000 7f421c019740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5558b789c6c0 0x5558b78ffc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:55.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.730+0000 7f421c019740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42141af230 con 0x5558b789c6c0 2024-03-21T10:39:55.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.730+0000 7f421c019740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42141b0530 con 0x5558b7892fd0 2024-03-21T10:39:55.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.730+0000 7f421c019740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42141af870 con 0x5558b78938f0 2024-03-21T10:39:55.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.730+0000 7f421c019740 10 client.0 _ll_register_callbacks cb 0x7f4200009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:39:55.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.731+0000 7f4212800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5558b7892fd0 0x5558b78969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:55.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.731+0000 7f4212800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5558b7892fd0 0x5558b78969c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44068/0 (socket says 172.21.15.31:44068) 2024-03-21T10:39:55.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.731+0000 7f4212800640 1 -- 172.21.15.31:0/573763270 learned_addr learned my addr 172.21.15.31:0/573763270 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:39:55.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.731+0000 7f4211e00640 1 --2- 172.21.15.31:0/573763270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5558b78938f0 0x5558b7896f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:55.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.731+0000 7f4213200640 1 --2- 172.21.15.31:0/573763270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5558b789c6c0 0x5558b78ffc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:55.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.731+0000 7f4212800640 1 -- 172.21.15.31:0/573763270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5558b78938f0 msgr2=0x5558b7896f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:55.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.731+0000 7f4212800640 1 --2- 172.21.15.31:0/573763270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5558b78938f0 0x5558b7896f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.731+0000 7f4212800640 1 -- 172.21.15.31:0/573763270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5558b789c6c0 msgr2=0x5558b78ffc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:39:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.731+0000 7f4212800640 1 --2- 172.21.15.31:0/573763270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5558b789c6c0 0x5558b78ffc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.731+0000 7f4212800640 1 -- 172.21.15.31:0/573763270 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5558b7900450 con 0x5558b7892fd0 2024-03-21T10:39:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.731+0000 7f4211e00640 1 --2- 172.21.15.31:0/573763270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5558b78938f0 0x5558b7896f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:39:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.731+0000 7f4213200640 1 --2- 172.21.15.31:0/573763270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5558b789c6c0 0x5558b78ffc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:39:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.731+0000 7f4212800640 1 --2- 172.21.15.31:0/573763270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5558b7892fd0 0x5558b78969c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f42040a8440 tx=0x7f420402bcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf0a6ee7afc7ea47 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:39:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.732+0000 7f41f3e00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:39:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.732+0000 7f41f3e00640 1 -- 172.21.15.31:0/573763270 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42040bdfd0 con 0x5558b7892fd0 2024-03-21T10:39:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.732+0000 7f41f3e00640 1 -- 172.21.15.31:0/573763270 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f4204003cc0 con 0x5558b7892fd0 2024-03-21T10:39:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.732+0000 7f421c019740 1 -- 172.21.15.31:0/573763270 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f42040bd8d0 con 0x5558b7892fd0 2024-03-21T10:39:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.732+0000 7f41f3e00640 1 -- 172.21.15.31:0/573763270 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f42040b9f40 con 0x5558b7892fd0 2024-03-21T10:39:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.732+0000 7f421c019740 10 client.25991 fetch_fsmap learned FSMap version 24 2024-03-21T10:39:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.732+0000 7f421c019740 1 -- 172.21.15.31:0/573763270 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5558b79087e0 con 0x5558b7892fd0 2024-03-21T10:39:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.732+0000 7f41f3e00640 1 -- 172.21.15.31:0/573763270 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42040bd8d0 con 0x5558b7892fd0 2024-03-21T10:39:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.733+0000 7f41f3e00640 1 -- 172.21.15.31:0/573763270 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f42040c5e50 con 0x5558b7892fd0 2024-03-21T10:39:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.733+0000 7f421c019740 10 client.25991 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:39:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.733+0000 7f421c019740 10 client.25991 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:39:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.733+0000 7f421c019740 20 client.25991 populate_metadata read hostname 'smithi031' 2024-03-21T10:39:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.733+0000 7f421c019740 1 --2- 172.21.15.31:0/573763270 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5558b7908ec0 0x5558b790b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:39:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.733+0000 7f421c019740 4 client.25991 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:39:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.733+0000 7f421c019740 1 -- 172.21.15.31:0/573763270 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5558b77ff0c0 con 0x5558b7908ec0 2024-03-21T10:39:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.733+0000 7f4211e00640 1 --2- 172.21.15.31:0/573763270 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5558b7908ec0 0x5558b790b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:39:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.734+0000 7f41f3e00640 10 client.25991 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.734+0000 7f4211e00640 1 --2- 172.21.15.31:0/573763270 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5558b7908ec0 0x5558b790b2e0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9298c31a8c29584 server_cookie=601d3f483fd7cda5 in_seq=0 out_seq=0 2024-03-21T10:39:55.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.737+0000 7f41f3e00640 1 -- 172.21.15.31:0/573763270 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5558b77ff0c0 con 0x5558b7908ec0 2024-03-21T10:39:55.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.737+0000 7f41f3e00640 10 client.25991 handle_command_reply: tid=0 2024-03-21T10:39:55.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.737+0000 7f421c019740 1 client.25991 shutdown 2024-03-21T10:39:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.737+0000 7f421c019740 1 -- 172.21.15.31:0/573763270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5558b7892fd0 msgr2=0x5558b78969c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.737+0000 7f421c019740 1 --2- 172.21.15.31:0/573763270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5558b7892fd0 0x5558b78969c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f42040a8440 tx=0x7f420402bcc0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.737+0000 7f421c019740 1 -- 172.21.15.31:0/573763270 shutdown_connections 2024-03-21T10:39:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.737+0000 7f421c019740 1 --2- 172.21.15.31:0/573763270 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5558b7908ec0 0x5558b790b2e0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.737+0000 7f421c019740 1 --2- 172.21.15.31:0/573763270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5558b789c6c0 0x5558b78ffc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.737+0000 7f421c019740 1 --2- 172.21.15.31:0/573763270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5558b78938f0 0x5558b7896f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.737+0000 7f421c019740 1 --2- 172.21.15.31:0/573763270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5558b7892fd0 0x5558b78969c0 secure :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f42040a8440 tx=0x7f420402bcc0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.737+0000 7f421c019740 1 -- 172.21.15.31:0/573763270 >> 172.21.15.31:0/573763270 conn(0x5558b788d8f0 msgr2=0x5558b78b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.737+0000 7f41f3e00640 0 client.25991 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:39:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.738+0000 7f421c019740 1 -- 172.21.15.31:0/573763270 shutdown_connections 2024-03-21T10:39:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.738+0000 7f421c019740 1 -- 172.21.15.31:0/573763270 wait complete. 2024-03-21T10:39:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.738+0000 7f421c019740 20 client.25991 trim_cache size 0 max 16384 2024-03-21T10:39:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:39:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:39:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:39:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:39:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.739+0000 7f4219400640 1 -- 172.21.15.31:0/472898258 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f41f80a3220 msgr2=0x7f41f80a56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.739+0000 7f4219400640 1 --2- 172.21.15.31:0/472898258 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f41f80a3220 0x7f41f80a56e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f420c009270 tx=0x7f420c0091e0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.739+0000 7f4219400640 1 -- 172.21.15.31:0/472898258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4214073a50 msgr2=0x7f421419a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:39:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.739+0000 7f4219400640 1 --2- 172.21.15.31:0/472898258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4214073a50 0x7f421419a1a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f4204000c00 tx=0x7f4204004360 comp rx=0 tx=0).stop 2024-03-21T10:39:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.739+0000 7f4219400640 1 -- 172.21.15.31:0/472898258 shutdown_connections 2024-03-21T10:39:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.739+0000 7f4219400640 1 --2- 172.21.15.31:0/472898258 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f41f80a3220 0x7f41f80a56e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.739+0000 7f4219400640 1 --2- 172.21.15.31:0/472898258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f421410df10 0x7f421419ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.739+0000 7f4219400640 1 --2- 172.21.15.31:0/472898258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4214074370 0x7f421419a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.739+0000 7f4219400640 1 --2- 172.21.15.31:0/472898258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4214073a50 0x7f421419a1a0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:39:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.739+0000 7f4219400640 1 -- 172.21.15.31:0/472898258 >> 172.21.15.31:0/472898258 conn(0x7f42140fd120 msgr2=0x7f42140fe950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:39:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.740+0000 7f4219400640 1 -- 172.21.15.31:0/472898258 shutdown_connections 2024-03-21T10:39:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:39:55.740+0000 7f4219400640 1 -- 172.21.15.31:0/472898258 wait complete. 2024-03-21T10:39:56.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:56 smithi040 ceph-mon[36154]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-21T10:39:57.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:56 smithi031 ceph-mon[31673]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-21T10:39:57.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:56 smithi070 ceph-mon[36330]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-21T10:39:59.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:39:58 smithi031 ceph-mon[31673]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:39:59.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:39:58 smithi070 ceph-mon[36330]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:39:59.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:39:58 smithi040 ceph-mon[36154]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:40:00.095 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:40:00.134 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:40:00.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.353+0000 7fbe75c00640 1 -- 172.21.15.31:0/1227062462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe70104040 msgr2=0x7fbe70110550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:00.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.353+0000 7fbe75c00640 1 --2- 172.21.15.31:0/1227062462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe70104040 0x7fbe70110550 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fbe68009be0 tx=0x7fbe6802bef0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.353+0000 7fbe75c00640 1 -- 172.21.15.31:0/1227062462 shutdown_connections 2024-03-21T10:40:00.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.353+0000 7fbe75c00640 1 --2- 172.21.15.31:0/1227062462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe70104040 0x7fbe70110550 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.353+0000 7fbe75c00640 1 --2- 172.21.15.31:0/1227062462 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe70103680 0x7fbe70103b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.353+0000 7fbe75c00640 1 --2- 172.21.15.31:0/1227062462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe70109690 0x7fbe70109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.353+0000 7fbe75c00640 1 -- 172.21.15.31:0/1227062462 >> 172.21.15.31:0/1227062462 conn(0x7fbe700ff110 msgr2=0x7fbe70101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:00.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.353+0000 7fbe75c00640 1 -- 172.21.15.31:0/1227062462 shutdown_connections 2024-03-21T10:40:00.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.353+0000 7fbe75c00640 1 -- 172.21.15.31:0/1227062462 wait complete. 2024-03-21T10:40:00.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.355+0000 7fbe75c00640 1 Processor -- start 2024-03-21T10:40:00.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.355+0000 7fbe75c00640 1 -- start start 2024-03-21T10:40:00.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.355+0000 7fbe75c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe70103680 0x7fbe70199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:00.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.355+0000 7fbe75c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe70104040 0x7fbe7019a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:00.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.355+0000 7fbe75c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe70109690 0x7fbe7019e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:00.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.356+0000 7fbe75c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe70113af0 con 0x7fbe70103680 2024-03-21T10:40:00.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.356+0000 7fbe75c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe70113970 con 0x7fbe70104040 2024-03-21T10:40:00.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.356+0000 7fbe75c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe70113c70 con 0x7fbe70109690 2024-03-21T10:40:00.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.356+0000 7fbe75200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe70109690 0x7fbe7019e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:00.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.356+0000 7fbe75200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe70109690 0x7fbe7019e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42366/0 (socket says 172.21.15.31:42366) 2024-03-21T10:40:00.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.356+0000 7fbe75200640 1 -- 172.21.15.31:0/2032336435 learned_addr learned my addr 172.21.15.31:0/2032336435 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:00.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.356+0000 7fbe6f400640 1 --2- 172.21.15.31:0/2032336435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe70104040 0x7fbe7019a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:00.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.356+0000 7fbe6fe00640 1 --2- 172.21.15.31:0/2032336435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe70103680 0x7fbe70199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:00.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.356+0000 7fbe75200640 1 -- 172.21.15.31:0/2032336435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe70104040 msgr2=0x7fbe7019a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:00.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.356+0000 7fbe75200640 1 --2- 172.21.15.31:0/2032336435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe70104040 0x7fbe7019a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.356+0000 7fbe75200640 1 -- 172.21.15.31:0/2032336435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe70103680 msgr2=0x7fbe70199f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:00.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.356+0000 7fbe75200640 1 --2- 172.21.15.31:0/2032336435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe70103680 0x7fbe70199f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.356+0000 7fbe75200640 1 -- 172.21.15.31:0/2032336435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe7019ef60 con 0x7fbe70109690 2024-03-21T10:40:00.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.356+0000 7fbe6fe00640 1 --2- 172.21.15.31:0/2032336435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe70103680 0x7fbe70199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:00.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.357+0000 7fbe75200640 1 --2- 172.21.15.31:0/2032336435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe70109690 0x7fbe7019e830 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbe68033420 tx=0x7fbe680336b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:00.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.357+0000 7fbe6cc00640 1 -- 172.21.15.31:0/2032336435 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe68043020 con 0x7fbe70109690 2024-03-21T10:40:00.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.357+0000 7fbe75c00640 1 -- 172.21.15.31:0/2032336435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe7019f1f0 con 0x7fbe70109690 2024-03-21T10:40:00.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.357+0000 7fbe6cc00640 1 -- 172.21.15.31:0/2032336435 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe68003e50 con 0x7fbe70109690 2024-03-21T10:40:00.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.357+0000 7fbe6cc00640 1 -- 172.21.15.31:0/2032336435 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6803e6b0 con 0x7fbe70109690 2024-03-21T10:40:00.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.357+0000 7fbe75c00640 1 -- 172.21.15.31:0/2032336435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe701a6ae0 con 0x7fbe70109690 2024-03-21T10:40:00.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.361+0000 7fbe6cc00640 1 -- 172.21.15.31:0/2032336435 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) v1 ==== 138087+0+0 (secure 0 0 0) 0x7fbe68046020 con 0x7fbe70109690 2024-03-21T10:40:00.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.362+0000 7fbe79fab740 1 Processor -- start 2024-03-21T10:40:00.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.362+0000 7fbe6cc00640 1 --2- 172.21.15.31:0/2032336435 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbe540a3230 0x7fbe540a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:00.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.362+0000 7fbe79fab740 1 -- start start 2024-03-21T10:40:00.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.363+0000 7fbe6cc00640 1 -- 172.21.15.31:0/2032336435 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fbe680f22b0 con 0x7fbe70109690 2024-03-21T10:40:00.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.363+0000 7fbe79fab740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bf9080dd0 0x555bf907fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:00.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.363+0000 7fbe79fab740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bf90800f0 0x555bf9080570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:00.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.363+0000 7fbe79fab740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf9089460 0x555bf908b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:00.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.363+0000 7fbe79fab740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bf8fe4250 con 0x555bf90800f0 2024-03-21T10:40:00.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.363+0000 7fbe79fab740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bf8f8ca30 con 0x555bf9080dd0 2024-03-21T10:40:00.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.363+0000 7fbe79fab740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bf8fe02f0 con 0x555bf9089460 2024-03-21T10:40:00.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.363+0000 7fbe6fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bf9080dd0 0x555bf907fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:00.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.363+0000 7fbe6f400640 1 --2- 172.21.15.31:0/2032336435 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbe540a3230 0x7fbe540a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:00.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.363+0000 7fbe6fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bf9080dd0 0x555bf907fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:41810/0 (socket says 172.21.15.31:41810) 2024-03-21T10:40:00.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.363+0000 7fbe6fe00640 1 -- 172.21.15.31:0/1915009091 learned_addr learned my addr 172.21.15.31:0/1915009091 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:00.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.364+0000 7fbe75200640 1 --2- 172.21.15.31:0/1915009091 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf9089460 0x555bf908b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:00.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.364+0000 7fbe75200640 1 -- 172.21.15.31:0/1915009091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bf9080dd0 msgr2=0x555bf907fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:00.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.364+0000 7fbe75200640 1 --2- 172.21.15.31:0/1915009091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bf9080dd0 0x555bf907fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.364+0000 7fbe75200640 1 -- 172.21.15.31:0/1915009091 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bf90800f0 msgr2=0x555bf9080570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:00.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.364+0000 7fbe75200640 1 --2- 172.21.15.31:0/1915009091 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bf90800f0 0x555bf9080570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.364+0000 7fbe75200640 1 -- 172.21.15.31:0/1915009091 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555bf8f77690 con 0x555bf9089460 2024-03-21T10:40:00.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.364+0000 7fbe75200640 1 --2- 172.21.15.31:0/1915009091 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf9089460 0x555bf908b850 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fbe680ef5f0 tx=0x7fbe68009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0509f1438b9006b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:00.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.365+0000 7fbe6f400640 1 --2- 172.21.15.31:0/2032336435 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbe540a3230 0x7fbe540a56f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c009710 tx=0x7fbe5c009920 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:00.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.365+0000 7fbe65600640 1 -- 172.21.15.31:0/1915009091 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe68004040 con 0x555bf9089460 2024-03-21T10:40:00.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.365+0000 7fbe65600640 1 -- 172.21.15.31:0/1915009091 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe680be8c0 con 0x555bf9089460 2024-03-21T10:40:00.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.365+0000 7fbe65600640 1 -- 172.21.15.31:0/1915009091 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe680ba830 con 0x555bf9089460 2024-03-21T10:40:00.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.366+0000 7fbe79fab740 1 -- 172.21.15.31:0/1915009091 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf9089460 msgr2=0x555bf908b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:00.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.366+0000 7fbe79fab740 1 --2- 172.21.15.31:0/1915009091 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf9089460 0x555bf908b850 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fbe680ef5f0 tx=0x7fbe68009500 comp rx=0 tx=0).stop 2024-03-21T10:40:00.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.366+0000 7fbe79fab740 1 -- 172.21.15.31:0/1915009091 shutdown_connections 2024-03-21T10:40:00.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.366+0000 7fbe79fab740 1 --2- 172.21.15.31:0/1915009091 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf9089460 0x555bf908b850 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.366+0000 7fbe79fab740 1 --2- 172.21.15.31:0/1915009091 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bf90800f0 0x555bf9080570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.366+0000 7fbe79fab740 1 --2- 172.21.15.31:0/1915009091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bf9080dd0 0x555bf907fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.366+0000 7fbe79fab740 1 -- 172.21.15.31:0/1915009091 >> 172.21.15.31:0/1915009091 conn(0x555bf9047200 msgr2=0x555bf907ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:00.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.366+0000 7fbe79fab740 1 -- 172.21.15.31:0/1915009091 shutdown_connections 2024-03-21T10:40:00.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.366+0000 7fbe79fab740 1 -- 172.21.15.31:0/1915009091 wait complete. 2024-03-21T10:40:00.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.367+0000 7fbe79fab740 1 -- start start 2024-03-21T10:40:00.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.367+0000 7fbe79fab740 1 Processor -- start 2024-03-21T10:40:00.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.368+0000 7fbe79fab740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bf90800f0 0x555bf9088bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:00.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.368+0000 7fbe79fab740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf9080dd0 0x555bf9085f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:00.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.368+0000 7fbe79fab740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bf9089460 0x555bf90864c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:00.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.368+0000 7fbe79fab740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bf8fe02f0 con 0x555bf9089460 2024-03-21T10:40:00.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.368+0000 7fbe79fab740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bf8f9fe80 con 0x555bf90800f0 2024-03-21T10:40:00.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.368+0000 7fbe79fab740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bf908c090 con 0x555bf9080dd0 2024-03-21T10:40:00.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.368+0000 7fbe79fab740 10 client.0 _ll_register_callbacks cb 0x555bf8fb2130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:00.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.368+0000 7fbe6f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf9080dd0 0x555bf9085f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:00.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.368+0000 7fbe6f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf9080dd0 0x555bf9085f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42382/0 (socket says 172.21.15.31:42382) 2024-03-21T10:40:00.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.368+0000 7fbe6f400640 1 -- 172.21.15.31:0/2028634904 learned_addr learned my addr 172.21.15.31:0/2028634904 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:00.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.368+0000 7fbe6fe00640 1 --2- 172.21.15.31:0/2028634904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bf90800f0 0x555bf9088bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:00.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.368+0000 7fbe75200640 1 --2- 172.21.15.31:0/2028634904 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bf9089460 0x555bf90864c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:00.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.368+0000 7fbe6f400640 1 -- 172.21.15.31:0/2028634904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bf90800f0 msgr2=0x555bf9088bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:00.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.368+0000 7fbe6f400640 1 --2- 172.21.15.31:0/2028634904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bf90800f0 0x555bf9088bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.368+0000 7fbe6f400640 1 -- 172.21.15.31:0/2028634904 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bf9089460 msgr2=0x555bf90864c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:00.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.369+0000 7fbe6f400640 1 --2- 172.21.15.31:0/2028634904 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bf9089460 0x555bf90864c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.369+0000 7fbe6f400640 1 -- 172.21.15.31:0/2028634904 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555bf8dc27d0 con 0x555bf9080dd0 2024-03-21T10:40:00.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.369+0000 7fbe75200640 1 --2- 172.21.15.31:0/2028634904 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bf9089460 0x555bf90864c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:00.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.369+0000 7fbe6f400640 1 --2- 172.21.15.31:0/2028634904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf9080dd0 0x555bf9085f80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c0055d0 tx=0x7fbe5c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7fefa4b89cbb5d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:00.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.369+0000 7fbe5a000640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:40:00.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.369+0000 7fbe5a000640 1 -- 172.21.15.31:0/2028634904 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c004c30 con 0x555bf9080dd0 2024-03-21T10:40:00.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.369+0000 7fbe79fab740 1 -- 172.21.15.31:0/2028634904 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbe680be8c0 con 0x555bf9080dd0 2024-03-21T10:40:00.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.369+0000 7fbe5a000640 1 -- 172.21.15.31:0/2028634904 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fbe5c004dd0 con 0x555bf9080dd0 2024-03-21T10:40:00.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.370+0000 7fbe5a000640 1 -- 172.21.15.31:0/2028634904 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe5c026440 con 0x555bf9080dd0 2024-03-21T10:40:00.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.370+0000 7fbe79fab740 10 client.25987 fetch_fsmap learned FSMap version 24 2024-03-21T10:40:00.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.370+0000 7fbe79fab740 1 -- 172.21.15.31:0/2028634904 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x555bf8cf83f0 con 0x555bf9080dd0 2024-03-21T10:40:00.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.370+0000 7fbe5a000640 1 -- 172.21.15.31:0/2028634904 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe680be8c0 con 0x555bf9080dd0 2024-03-21T10:40:00.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.370+0000 7fbe5a000640 1 -- 172.21.15.31:0/2028634904 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fbe5c024070 con 0x555bf9080dd0 2024-03-21T10:40:00.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.371+0000 7fbe79fab740 10 client.25987 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:40:00.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.371+0000 7fbe79fab740 10 client.25987 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:40:00.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.371+0000 7fbe79fab740 20 client.25987 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:00.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.371+0000 7fbe79fab740 1 --2- 172.21.15.31:0/2028634904 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x555bf908d650 0x555bf908fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:00.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.371+0000 7fbe79fab740 4 client.25987 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:40:00.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.371+0000 7fbe79fab740 1 -- 172.21.15.31:0/2028634904 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555bf8ff1790 con 0x555bf908d650 2024-03-21T10:40:00.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.372+0000 7fbe6fe00640 1 --2- 172.21.15.31:0/2028634904 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x555bf908d650 0x555bf908fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:00.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.372+0000 7fbe6fe00640 1 --2- 172.21.15.31:0/2028634904 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x555bf908d650 0x555bf908fa60 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4351f81767a6c74c server_cookie=d18fc3ec24d3cdaa in_seq=0 out_seq=0 2024-03-21T10:40:00.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.372+0000 7fbe5a000640 10 client.25987 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:00.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.373+0000 7fbe5a000640 1 -- 172.21.15.31:0/2028634904 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555bf8ff1790 con 0x555bf908d650 2024-03-21T10:40:00.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.373+0000 7fbe5a000640 10 client.25987 handle_command_reply: tid=0 2024-03-21T10:40:00.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.373+0000 7fbe79fab740 1 client.25987 shutdown 2024-03-21T10:40:00.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.373+0000 7fbe79fab740 1 -- 172.21.15.31:0/2028634904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf9080dd0 msgr2=0x555bf9085f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:00.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.373+0000 7fbe79fab740 1 --2- 172.21.15.31:0/2028634904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf9080dd0 0x555bf9085f80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c0055d0 tx=0x7fbe5c0046c0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.374+0000 7fbe79fab740 1 -- 172.21.15.31:0/2028634904 shutdown_connections 2024-03-21T10:40:00.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.374+0000 7fbe79fab740 1 --2- 172.21.15.31:0/2028634904 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x555bf908d650 0x555bf908fa60 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.374+0000 7fbe79fab740 1 --2- 172.21.15.31:0/2028634904 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bf9089460 0x555bf90864c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.374+0000 7fbe79fab740 1 --2- 172.21.15.31:0/2028634904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf9080dd0 0x555bf9085f80 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.374+0000 7fbe79fab740 1 --2- 172.21.15.31:0/2028634904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bf90800f0 0x555bf9088bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.374+0000 7fbe5a000640 0 client.25987 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:00.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.374+0000 7fbe79fab740 1 -- 172.21.15.31:0/2028634904 >> 172.21.15.31:0/2028634904 conn(0x555bf9047200 msgr2=0x555bf907c890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:00.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.374+0000 7fbe79fab740 1 -- 172.21.15.31:0/2028634904 shutdown_connections 2024-03-21T10:40:00.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.374+0000 7fbe79fab740 1 -- 172.21.15.31:0/2028634904 wait complete. 2024-03-21T10:40:00.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.374+0000 7fbe79fab740 20 client.25987 trim_cache size 0 max 16384 2024-03-21T10:40:00.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.396+0000 7fbe79fab740 1 Processor -- start 2024-03-21T10:40:00.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.396+0000 7fbe79fab740 1 -- start start 2024-03-21T10:40:00.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.396+0000 7fbe79fab740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bf90800f0 0x555bf909d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:00.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.396+0000 7fbe79fab740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bf9080dd0 0x555bf90886b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:00.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.396+0000 7fbe79fab740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf9089460 0x555bf9088bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:00.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.396+0000 7fbe79fab740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe701ae9d0 con 0x555bf90800f0 2024-03-21T10:40:00.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.396+0000 7fbe79fab740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe701ac4d0 con 0x555bf9080dd0 2024-03-21T10:40:00.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.396+0000 7fbe79fab740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe701aa590 con 0x555bf9089460 2024-03-21T10:40:00.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.397+0000 7fbe6f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bf9080dd0 0x555bf90886b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:00.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.397+0000 7fbe6f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bf9080dd0 0x555bf90886b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:41834/0 (socket says 172.21.15.31:41834) 2024-03-21T10:40:00.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.397+0000 7fbe6f400640 1 -- 172.21.15.31:0/3692336942 learned_addr learned my addr 172.21.15.31:0/3692336942 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:00.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.397+0000 7fbe75200640 1 --2- 172.21.15.31:0/3692336942 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf9089460 0x555bf9088bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:00.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.397+0000 7fbe6fe00640 1 --2- 172.21.15.31:0/3692336942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bf90800f0 0x555bf909d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:00.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.397+0000 7fbe6f400640 1 -- 172.21.15.31:0/3692336942 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf9089460 msgr2=0x555bf9088bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:00.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.397+0000 7fbe6f400640 1 --2- 172.21.15.31:0/3692336942 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf9089460 0x555bf9088bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.397+0000 7fbe6f400640 1 -- 172.21.15.31:0/3692336942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bf90800f0 msgr2=0x555bf909d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:00.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.397+0000 7fbe6f400640 1 --2- 172.21.15.31:0/3692336942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bf90800f0 0x555bf909d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.397+0000 7fbe6f400640 1 -- 172.21.15.31:0/3692336942 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555bf8cb80f0 con 0x555bf9080dd0 2024-03-21T10:40:00.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.397+0000 7fbe6fe00640 1 --2- 172.21.15.31:0/3692336942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bf90800f0 0x555bf909d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:00.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.398+0000 7fbe6f400640 1 --2- 172.21.15.31:0/3692336942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bf9080dd0 0x555bf90886b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c018040 tx=0x7fbe5c011d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=300ed8f4fd50ca6a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:00.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.398+0000 7fbe64c00640 1 -- 172.21.15.31:0/3692336942 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c030670 con 0x555bf9080dd0 2024-03-21T10:40:00.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.398+0000 7fbe64c00640 1 -- 172.21.15.31:0/3692336942 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe5c035070 con 0x555bf9080dd0 2024-03-21T10:40:00.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.399+0000 7fbe64c00640 1 -- 172.21.15.31:0/3692336942 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c026c90 con 0x555bf9080dd0 2024-03-21T10:40:00.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.399+0000 7fbe79fab740 1 -- 172.21.15.31:0/3692336942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bf9080dd0 msgr2=0x555bf90886b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:00.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.399+0000 7fbe79fab740 1 --2- 172.21.15.31:0/3692336942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bf9080dd0 0x555bf90886b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c018040 tx=0x7fbe5c011d80 comp rx=0 tx=0).stop 2024-03-21T10:40:00.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.400+0000 7fbe79fab740 1 -- 172.21.15.31:0/3692336942 shutdown_connections 2024-03-21T10:40:00.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.400+0000 7fbe79fab740 1 --2- 172.21.15.31:0/3692336942 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf9089460 0x555bf9088bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.400+0000 7fbe79fab740 1 --2- 172.21.15.31:0/3692336942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bf9080dd0 0x555bf90886b0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.400+0000 7fbe79fab740 1 --2- 172.21.15.31:0/3692336942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bf90800f0 0x555bf909d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.400+0000 7fbe79fab740 1 -- 172.21.15.31:0/3692336942 >> 172.21.15.31:0/3692336942 conn(0x555bf9047200 msgr2=0x555bf907a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:00.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.400+0000 7fbe79fab740 1 -- 172.21.15.31:0/3692336942 shutdown_connections 2024-03-21T10:40:00.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.400+0000 7fbe79fab740 1 -- 172.21.15.31:0/3692336942 wait complete. 2024-03-21T10:40:00.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.401+0000 7fbe79fab740 1 -- start start 2024-03-21T10:40:00.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.401+0000 7fbe79fab740 1 Processor -- start 2024-03-21T10:40:00.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.402+0000 7fbe79fab740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bf90800f0 0x555bf90a0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:00.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.402+0000 7fbe79fab740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf9080dd0 0x555bf90a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:00.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.402+0000 7fbe79fab740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bf9089460 0x555bf90ed190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:00.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.402+0000 7fbe79fab740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe701aab90 con 0x555bf90800f0 2024-03-21T10:40:00.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.402+0000 7fbe79fab740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe701abe70 con 0x555bf9089460 2024-03-21T10:40:00.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.402+0000 7fbe79fab740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe701ab1b0 con 0x555bf9080dd0 2024-03-21T10:40:00.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.402+0000 7fbe79fab740 10 client.0 _ll_register_callbacks cb 0x7fbe5c009ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:00.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.402+0000 7fbe6f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf9080dd0 0x555bf90a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:00.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.402+0000 7fbe6f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf9080dd0 0x555bf90a0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42388/0 (socket says 172.21.15.31:42388) 2024-03-21T10:40:00.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.402+0000 7fbe6f400640 1 -- 172.21.15.31:0/1417255674 learned_addr learned my addr 172.21.15.31:0/1417255674 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:00.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.402+0000 7fbe75200640 1 --2- 172.21.15.31:0/1417255674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bf9089460 0x555bf90ed190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:00.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.403+0000 7fbe6fe00640 1 --2- 172.21.15.31:0/1417255674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bf90800f0 0x555bf90a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:00.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.403+0000 7fbe6f400640 1 -- 172.21.15.31:0/1417255674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bf9089460 msgr2=0x555bf90ed190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:00.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.403+0000 7fbe6f400640 1 --2- 172.21.15.31:0/1417255674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bf9089460 0x555bf90ed190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.403+0000 7fbe6f400640 1 -- 172.21.15.31:0/1417255674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bf90800f0 msgr2=0x555bf90a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:00.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.403+0000 7fbe6f400640 1 --2- 172.21.15.31:0/1417255674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bf90800f0 0x555bf90a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.403+0000 7fbe6f400640 1 -- 172.21.15.31:0/1417255674 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555bf90ed7d0 con 0x555bf9080dd0 2024-03-21T10:40:00.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.403+0000 7fbe6fe00640 1 --2- 172.21.15.31:0/1417255674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bf90800f0 0x555bf90a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:40:00.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.404+0000 7fbe6f400640 1 --2- 172.21.15.31:0/1417255674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf9080dd0 0x555bf90a0b60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c0309c0 tx=0x7fbe5c030720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d09e2295396eb86 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:00.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.404+0000 7fbe5a000640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:40:00.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.404+0000 7fbe5a000640 1 -- 172.21.15.31:0/1417255674 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c041030 con 0x555bf9080dd0 2024-03-21T10:40:00.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.404+0000 7fbe5a000640 1 -- 172.21.15.31:0/1417255674 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fbe5c004d10 con 0x555bf9080dd0 2024-03-21T10:40:00.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.404+0000 7fbe79fab740 1 -- 172.21.15.31:0/1417255674 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbe5c035070 con 0x555bf9080dd0 2024-03-21T10:40:00.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.405+0000 7fbe5a000640 1 -- 172.21.15.31:0/1417255674 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe5c030b70 con 0x555bf9080dd0 2024-03-21T10:40:00.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.405+0000 7fbe79fab740 10 client.25993 fetch_fsmap learned FSMap version 24 2024-03-21T10:40:00.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.405+0000 7fbe79fab740 1 -- 172.21.15.31:0/1417255674 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x555bf90f4b50 con 0x555bf9080dd0 2024-03-21T10:40:00.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.405+0000 7fbe5a000640 1 -- 172.21.15.31:0/1417255674 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe5c035070 con 0x555bf9080dd0 2024-03-21T10:40:00.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.405+0000 7fbe5a000640 1 -- 172.21.15.31:0/1417255674 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fbe5c024040 con 0x555bf9080dd0 2024-03-21T10:40:00.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.406+0000 7fbe79fab740 10 client.25993 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:40:00.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.406+0000 7fbe79fab740 10 client.25993 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:40:00.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.406+0000 7fbe79fab740 20 client.25993 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:00.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.406+0000 7fbe79fab740 1 --2- 172.21.15.31:0/1417255674 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x555bf90f5160 0x555bf90f7580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:00.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.406+0000 7fbe79fab740 4 client.25993 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:40:00.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.406+0000 7fbe79fab740 1 -- 172.21.15.31:0/1417255674 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555bf8febae0 con 0x555bf90f5160 2024-03-21T10:40:00.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.406+0000 7fbe6fe00640 1 --2- 172.21.15.31:0/1417255674 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x555bf90f5160 0x555bf90f7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:00.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.407+0000 7fbe6fe00640 1 --2- 172.21.15.31:0/1417255674 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x555bf90f5160 0x555bf90f7580 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfe92d566b9dc442 server_cookie=4a228cc31c05cc02 in_seq=0 out_seq=0 2024-03-21T10:40:00.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.407+0000 7fbe5a000640 10 client.25993 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:00.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.407+0000 7fbe5a000640 1 -- 172.21.15.31:0/1417255674 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x555bf8febae0 con 0x555bf90f5160 2024-03-21T10:40:00.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.407+0000 7fbe5a000640 10 client.25993 handle_command_reply: tid=0 2024-03-21T10:40:00.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.407+0000 7fbe79fab740 1 client.25993 shutdown 2024-03-21T10:40:00.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.408+0000 7fbe79fab740 1 -- 172.21.15.31:0/1417255674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf9080dd0 msgr2=0x555bf90a0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:00.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.408+0000 7fbe79fab740 1 --2- 172.21.15.31:0/1417255674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf9080dd0 0x555bf90a0b60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c0309c0 tx=0x7fbe5c030720 comp rx=0 tx=0).stop 2024-03-21T10:40:00.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.408+0000 7fbe79fab740 1 -- 172.21.15.31:0/1417255674 shutdown_connections 2024-03-21T10:40:00.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.408+0000 7fbe79fab740 1 --2- 172.21.15.31:0/1417255674 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x555bf90f5160 0x555bf90f7580 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.408+0000 7fbe79fab740 1 --2- 172.21.15.31:0/1417255674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bf9089460 0x555bf90ed190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.408+0000 7fbe79fab740 1 --2- 172.21.15.31:0/1417255674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf9080dd0 0x555bf90a0b60 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.408+0000 7fbe79fab740 1 --2- 172.21.15.31:0/1417255674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bf90800f0 0x555bf90a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.408+0000 7fbe79fab740 1 -- 172.21.15.31:0/1417255674 >> 172.21.15.31:0/1417255674 conn(0x555bf9047200 msgr2=0x555bf9084850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:00.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.408+0000 7fbe5a000640 0 client.25993 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:00.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.408+0000 7fbe79fab740 1 -- 172.21.15.31:0/1417255674 shutdown_connections 2024-03-21T10:40:00.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.408+0000 7fbe79fab740 1 -- 172.21.15.31:0/1417255674 wait complete. 2024-03-21T10:40:00.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.408+0000 7fbe79fab740 20 client.25993 trim_cache size 0 max 16384 2024-03-21T10:40:00.407 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:40:00.407 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-03-21T10:40:00.407 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-03-21T10:40:00.408 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:40:00.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.411+0000 7fbe75c00640 1 -- 172.21.15.31:0/2032336435 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbe540a3230 msgr2=0x7fbe540a56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:00.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.411+0000 7fbe75c00640 1 --2- 172.21.15.31:0/2032336435 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbe540a3230 0x7fbe540a56f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c009710 tx=0x7fbe5c009920 comp rx=0 tx=0).stop 2024-03-21T10:40:00.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.411+0000 7fbe75c00640 1 -- 172.21.15.31:0/2032336435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe70109690 msgr2=0x7fbe7019e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:00.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.411+0000 7fbe75c00640 1 --2- 172.21.15.31:0/2032336435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe70109690 0x7fbe7019e830 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbe68033420 tx=0x7fbe680336b0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.411+0000 7fbe75c00640 1 -- 172.21.15.31:0/2032336435 shutdown_connections 2024-03-21T10:40:00.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.411+0000 7fbe75c00640 1 --2- 172.21.15.31:0/2032336435 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbe540a3230 0x7fbe540a56f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.411+0000 7fbe75c00640 1 --2- 172.21.15.31:0/2032336435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe70109690 0x7fbe7019e830 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.411+0000 7fbe75c00640 1 --2- 172.21.15.31:0/2032336435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe70104040 0x7fbe7019a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.411+0000 7fbe75c00640 1 --2- 172.21.15.31:0/2032336435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe70103680 0x7fbe70199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.411+0000 7fbe75c00640 1 -- 172.21.15.31:0/2032336435 >> 172.21.15.31:0/2032336435 conn(0x7fbe700ff110 msgr2=0x7fbe700ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:00.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.413+0000 7fbe75c00640 1 -- 172.21.15.31:0/2032336435 shutdown_connections 2024-03-21T10:40:00.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.413+0000 7fbe75c00640 1 -- 172.21.15.31:0/2032336435 wait complete. 2024-03-21T10:40:00.441 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-21T10:40:00.441 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:40:00.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.660+0000 7f85ab200640 1 -- 172.21.15.31:0/3400208209 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85a4107e50 msgr2=0x7f85a4108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:00.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.660+0000 7f85ab200640 1 --2- 172.21.15.31:0/3400208209 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85a4107e50 0x7f85a4108230 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8598009900 tx=0x7f859802bc30 comp rx=0 tx=0).stop 2024-03-21T10:40:00.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.660+0000 7f85ab200640 1 -- 172.21.15.31:0/3400208209 shutdown_connections 2024-03-21T10:40:00.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.660+0000 7f85ab200640 1 --2- 172.21.15.31:0/3400208209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a410cea0 0x7f85a4114a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.660+0000 7f85ab200640 1 --2- 172.21.15.31:0/3400208209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85a4108800 0x7f85a410c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.660+0000 7f85ab200640 1 --2- 172.21.15.31:0/3400208209 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85a4107e50 0x7f85a4108230 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.660+0000 7f85ab200640 1 -- 172.21.15.31:0/3400208209 >> 172.21.15.31:0/3400208209 conn(0x7f85a4077470 msgr2=0x7f85a41064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:00.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.661+0000 7f85ab200640 1 -- 172.21.15.31:0/3400208209 shutdown_connections 2024-03-21T10:40:00.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.661+0000 7f85ab200640 1 -- 172.21.15.31:0/3400208209 wait complete. 2024-03-21T10:40:00.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.662+0000 7f85ab200640 1 Processor -- start 2024-03-21T10:40:00.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.662+0000 7f85ab200640 1 -- start start 2024-03-21T10:40:00.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.663+0000 7f85ab200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85a4107e50 0x7f85a419e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:00.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.663+0000 7f85ab200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a410cea0 0x7f85a419e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:00.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.663+0000 7f85ab200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85a41a2ce0 0x7f85a41a3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:00.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.663+0000 7f85ab200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a4117fc0 con 0x7f85a41a2ce0 2024-03-21T10:40:00.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.663+0000 7f85ab200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a4117e40 con 0x7f85a410cea0 2024-03-21T10:40:00.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.663+0000 7f85ab200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a4118140 con 0x7f85a4107e50 2024-03-21T10:40:00.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.663+0000 7f85a9e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85a4107e50 0x7f85a419e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:00.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.663+0000 7f85a9e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85a4107e50 0x7f85a419e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42404/0 (socket says 172.21.15.31:42404) 2024-03-21T10:40:00.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.663+0000 7f85a9e00640 1 -- 172.21.15.31:0/4251981843 learned_addr learned my addr 172.21.15.31:0/4251981843 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:00.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.663+0000 7f85a9400640 1 --2- 172.21.15.31:0/4251981843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a410cea0 0x7f85a419e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:00.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.663+0000 7f85aa800640 1 --2- 172.21.15.31:0/4251981843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85a41a2ce0 0x7f85a41a3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:00.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.663+0000 7f85a9e00640 1 -- 172.21.15.31:0/4251981843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a410cea0 msgr2=0x7f85a419e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:00.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.663+0000 7f85a9e00640 1 --2- 172.21.15.31:0/4251981843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a410cea0 0x7f85a419e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.663+0000 7f85a9e00640 1 -- 172.21.15.31:0/4251981843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85a41a2ce0 msgr2=0x7f85a41a3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:00.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.663+0000 7f85a9e00640 1 --2- 172.21.15.31:0/4251981843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85a41a2ce0 0x7f85a41a3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:00.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.663+0000 7f85a9e00640 1 -- 172.21.15.31:0/4251981843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a41a37d0 con 0x7f85a4107e50 2024-03-21T10:40:00.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.664+0000 7f85a9e00640 1 --2- 172.21.15.31:0/4251981843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85a4107e50 0x7f85a419e420 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f859802c240 tx=0x7f85980042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:00.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.664+0000 7f859ea00640 1 -- 172.21.15.31:0/4251981843 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8598043020 con 0x7f85a4107e50 2024-03-21T10:40:00.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.664+0000 7f859ea00640 1 -- 172.21.15.31:0/4251981843 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f85980054e0 con 0x7f85a4107e50 2024-03-21T10:40:00.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.664+0000 7f859ea00640 1 -- 172.21.15.31:0/4251981843 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8598003b30 con 0x7f85a4107e50 2024-03-21T10:40:00.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.664+0000 7f85ab200640 1 -- 172.21.15.31:0/4251981843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85a41a3a60 con 0x7f85a4107e50 2024-03-21T10:40:00.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.665+0000 7f85ab200640 1 -- 172.21.15.31:0/4251981843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85a41ab1d0 con 0x7f85a4107e50 2024-03-21T10:40:00.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.666+0000 7f85ab200640 1 -- 172.21.15.31:0/4251981843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85a404f470 con 0x7f85a4107e50 2024-03-21T10:40:00.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.669+0000 7f859ea00640 1 -- 172.21.15.31:0/4251981843 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) v1 ==== 138087+0+0 (secure 0 0 0) 0x7f8598046020 con 0x7f85a4107e50 2024-03-21T10:40:00.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.671+0000 7f859ea00640 1 --2- 172.21.15.31:0/4251981843 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f858c0a3220 0x7f858c0a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:00.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.672+0000 7f859ea00640 1 -- 172.21.15.31:0/4251981843 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f85980f1c30 con 0x7f85a4107e50 2024-03-21T10:40:00.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.672+0000 7f85a9400640 1 --2- 172.21.15.31:0/4251981843 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f858c0a3220 0x7f858c0a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:00.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.673+0000 7f85a9400640 1 --2- 172.21.15.31:0/4251981843 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f858c0a3220 0x7f858c0a56e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f85a410e240 tx=0x7f8594005f90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:00.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:00.675+0000 7f859ea00640 1 -- 172.21.15.31:0/4251981843 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f859804d020 con 0x7f85a4107e50 2024-03-21T10:40:00.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:00 smithi031 ceph-mon[31673]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-21T10:40:00.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:00 smithi031 ceph-mon[31673]: overall HEALTH_OK 2024-03-21T10:40:01.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.062+0000 7f85ab200640 1 -- 172.21.15.31:0/4251981843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85a4108800 con 0x7f85a4107e50 2024-03-21T10:40:01.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.063+0000 7f859ea00640 1 -- 172.21.15.31:0/4251981843 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11384 (secure 0 0 0) 0x7f85980be240 con 0x7f85a4107e50 2024-03-21T10:40:01.061 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:40:01.061 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:38:12.319122+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24547,24596]},"id":1}]} 2024-03-21T10:40:01.061 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 24 2024-03-21T10:40:01.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.066+0000 7f85ab200640 1 -- 172.21.15.31:0/4251981843 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f858c0a3220 msgr2=0x7f858c0a56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.066+0000 7f85ab200640 1 --2- 172.21.15.31:0/4251981843 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f858c0a3220 0x7f858c0a56e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f85a410e240 tx=0x7f8594005f90 comp rx=0 tx=0).stop 2024-03-21T10:40:01.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.066+0000 7f85ab200640 1 -- 172.21.15.31:0/4251981843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85a4107e50 msgr2=0x7f85a419e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.066+0000 7f85ab200640 1 --2- 172.21.15.31:0/4251981843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85a4107e50 0x7f85a419e420 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f859802c240 tx=0x7f85980042f0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.066+0000 7f85ab200640 1 -- 172.21.15.31:0/4251981843 shutdown_connections 2024-03-21T10:40:01.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.066+0000 7f85ab200640 1 --2- 172.21.15.31:0/4251981843 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f858c0a3220 0x7f858c0a56e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.066+0000 7f85ab200640 1 --2- 172.21.15.31:0/4251981843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85a41a2ce0 0x7f85a41a3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.066+0000 7f85ab200640 1 --2- 172.21.15.31:0/4251981843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a410cea0 0x7f85a419e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.066+0000 7f85ab200640 1 --2- 172.21.15.31:0/4251981843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85a4107e50 0x7f85a419e420 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.066+0000 7f85ab200640 1 -- 172.21.15.31:0/4251981843 >> 172.21.15.31:0/4251981843 conn(0x7f85a4077470 msgr2=0x7f85a4077a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:01.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.066+0000 7f85ab200640 1 -- 172.21.15.31:0/4251981843 shutdown_connections 2024-03-21T10:40:01.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.066+0000 7f85ab200640 1 -- 172.21.15.31:0/4251981843 wait complete. 2024-03-21T10:40:01.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:00 smithi070 ceph-mon[36330]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-21T10:40:01.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:00 smithi070 ceph-mon[36330]: overall HEALTH_OK 2024-03-21T10:40:01.081 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:40:01.081 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-21T10:40:01.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:00 smithi040 ceph-mon[36154]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-21T10:40:01.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:00 smithi040 ceph-mon[36154]: overall HEALTH_OK 2024-03-21T10:40:01.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.275+0000 7fbfd6e00640 1 -- 172.21.15.31:0/3775271774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfd0103660 msgr2=0x7fbfd0103ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.275+0000 7fbfd6e00640 1 --2- 172.21.15.31:0/3775271774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfd0103660 0x7fbfd0103ae0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc009860 tx=0x7fbfbc02bb70 comp rx=0 tx=0).stop 2024-03-21T10:40:01.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.276+0000 7fbfd6e00640 1 -- 172.21.15.31:0/3775271774 shutdown_connections 2024-03-21T10:40:01.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.276+0000 7fbfd6e00640 1 --2- 172.21.15.31:0/3775271774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfd0104020 0x7fbfd0110530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.276+0000 7fbfd6e00640 1 --2- 172.21.15.31:0/3775271774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfd0103660 0x7fbfd0103ae0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.276+0000 7fbfd6e00640 1 --2- 172.21.15.31:0/3775271774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfd0109670 0x7fbfd0109a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.276+0000 7fbfd6e00640 1 -- 172.21.15.31:0/3775271774 >> 172.21.15.31:0/3775271774 conn(0x7fbfd00ff110 msgr2=0x7fbfd0101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:01.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.276+0000 7fbfd6e00640 1 -- 172.21.15.31:0/3775271774 shutdown_connections 2024-03-21T10:40:01.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.276+0000 7fbfd6e00640 1 -- 172.21.15.31:0/3775271774 wait complete. 2024-03-21T10:40:01.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.277+0000 7fbfd6e00640 1 Processor -- start 2024-03-21T10:40:01.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.277+0000 7fbfd6e00640 1 -- start start 2024-03-21T10:40:01.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.278+0000 7fbfd6e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfd0103660 0x7fbfd0199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.278+0000 7fbfd6e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfd0104020 0x7fbfd019a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.278+0000 7fbfd6e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfd0109670 0x7fbfd019e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.278+0000 7fbfd6e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd0113a80 con 0x7fbfd0109670 2024-03-21T10:40:01.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.278+0000 7fbfd6e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd0113900 con 0x7fbfd0104020 2024-03-21T10:40:01.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.278+0000 7fbfd6e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd0113c00 con 0x7fbfd0103660 2024-03-21T10:40:01.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.278+0000 7fbfd5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfd0104020 0x7fbfd019a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.278+0000 7fbfd5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfd0104020 0x7fbfd019a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:41874/0 (socket says 172.21.15.31:41874) 2024-03-21T10:40:01.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.278+0000 7fbfd5000640 1 -- 172.21.15.31:0/1933312316 learned_addr learned my addr 172.21.15.31:0/1933312316 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:01.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.278+0000 7fbfd5a00640 1 --2- 172.21.15.31:0/1933312316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfd0103660 0x7fbfd0199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.279+0000 7fbfd6400640 1 --2- 172.21.15.31:0/1933312316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfd0109670 0x7fbfd019e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.279+0000 7fbfd5000640 1 -- 172.21.15.31:0/1933312316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfd0103660 msgr2=0x7fbfd0199f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.279+0000 7fbfd5000640 1 --2- 172.21.15.31:0/1933312316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfd0103660 0x7fbfd0199f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.279+0000 7fbfd5000640 1 -- 172.21.15.31:0/1933312316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfd0109670 msgr2=0x7fbfd019e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.279+0000 7fbfd5000640 1 --2- 172.21.15.31:0/1933312316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfd0109670 0x7fbfd019e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.279+0000 7fbfd5000640 1 -- 172.21.15.31:0/1933312316 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd019ef70 con 0x7fbfd0104020 2024-03-21T10:40:01.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.279+0000 7fbfd6400640 1 --2- 172.21.15.31:0/1933312316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfd0109670 0x7fbfd019e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:40:01.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.279+0000 7fbfd5000640 1 --2- 172.21.15.31:0/1933312316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfd0104020 0x7fbfd019a4c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc037040 tx=0x7fbfbc004590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:01.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.280+0000 7fbfce000640 1 -- 172.21.15.31:0/1933312316 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc041020 con 0x7fbfd0104020 2024-03-21T10:40:01.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.280+0000 7fbfce000640 1 -- 172.21.15.31:0/1933312316 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbfbc02c9f0 con 0x7fbfd0104020 2024-03-21T10:40:01.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.280+0000 7fbfce000640 1 -- 172.21.15.31:0/1933312316 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc039040 con 0x7fbfd0104020 2024-03-21T10:40:01.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.280+0000 7fbfd6e00640 1 -- 172.21.15.31:0/1933312316 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd019f200 con 0x7fbfd0104020 2024-03-21T10:40:01.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.280+0000 7fbfd6e00640 1 -- 172.21.15.31:0/1933312316 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfd01a6af0 con 0x7fbfd0104020 2024-03-21T10:40:01.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.285+0000 7fbfce000640 1 -- 172.21.15.31:0/1933312316 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) v1 ==== 138087+0+0 (secure 0 0 0) 0x7fbfbc044020 con 0x7fbfd0104020 2024-03-21T10:40:01.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.285+0000 7fbfdb03c740 1 Processor -- start 2024-03-21T10:40:01.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.285+0000 7fbfdb03c740 1 -- start start 2024-03-21T10:40:01.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.286+0000 7fbfdb03c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3505378b0 0x55e350537c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.286+0000 7fbfdb03c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e350538260 0x55e350540810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.286+0000 7fbfdb03c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e350540de0 0x55e350543230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.286+0000 7fbfdb03c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e35049b2c0 con 0x55e350540de0 2024-03-21T10:40:01.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.286+0000 7fbfdb03c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e350443c90 con 0x55e3505378b0 2024-03-21T10:40:01.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.286+0000 7fbfdb03c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e350497360 con 0x55e350538260 2024-03-21T10:40:01.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.287+0000 7fbfd6400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3505378b0 0x55e350537c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.287+0000 7fbfd6400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3505378b0 0x55e350537c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:41882/0 (socket says 172.21.15.31:41882) 2024-03-21T10:40:01.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.287+0000 7fbfd6400640 1 -- 172.21.15.31:0/4155372599 learned_addr learned my addr 172.21.15.31:0/4155372599 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:01.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.287+0000 7fbfd5000640 1 --2- 172.21.15.31:0/4155372599 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e350540de0 0x55e350543230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.287+0000 7fbfd5a00640 1 --2- 172.21.15.31:0/4155372599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e350538260 0x55e350540810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.287+0000 7fbfce000640 1 --2- 172.21.15.31:0/1933312316 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbfb80a31b0 0x7fbfb80a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.287+0000 7fbfd5000640 1 -- 172.21.15.31:0/4155372599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e350538260 msgr2=0x55e350540810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:01.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.287+0000 7fbfd5000640 1 --2- 172.21.15.31:0/4155372599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e350538260 0x55e350540810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.287+0000 7fbfd5000640 1 -- 172.21.15.31:0/4155372599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3505378b0 msgr2=0x55e350537c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:01.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.287+0000 7fbfd5000640 1 --2- 172.21.15.31:0/4155372599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3505378b0 0x55e350537c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.287+0000 7fbfd5000640 1 -- 172.21.15.31:0/4155372599 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e350468080 con 0x55e350540de0 2024-03-21T10:40:01.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.288+0000 7fbfce000640 1 -- 172.21.15.31:0/1933312316 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fbfbc0f15e0 con 0x7fbfd0104020 2024-03-21T10:40:01.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.288+0000 7fbfd5000640 1 --2- 172.21.15.31:0/4155372599 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e350540de0 0x55e350543230 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc0ee980 tx=0x7fbfbc0bd150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=791e40207d90ef60 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:01.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.288+0000 7fbfc2a00640 1 -- 172.21.15.31:0/4155372599 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc0bdc50 con 0x55e350540de0 2024-03-21T10:40:01.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.288+0000 7fbfd6400640 1 --2- 172.21.15.31:0/1933312316 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbfb80a31b0 0x7fbfb80a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.289+0000 7fbfc2a00640 1 -- 172.21.15.31:0/4155372599 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbfbc032e50 con 0x55e350540de0 2024-03-21T10:40:01.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.289+0000 7fbfc2a00640 1 -- 172.21.15.31:0/4155372599 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc0c16a0 con 0x55e350540de0 2024-03-21T10:40:01.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.289+0000 7fbfdb03c740 1 -- 172.21.15.31:0/4155372599 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e350540de0 msgr2=0x55e350543230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.289+0000 7fbfdb03c740 1 --2- 172.21.15.31:0/4155372599 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e350540de0 0x55e350543230 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc0ee980 tx=0x7fbfbc0bd150 comp rx=0 tx=0).stop 2024-03-21T10:40:01.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.289+0000 7fbfd6400640 1 --2- 172.21.15.31:0/1933312316 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbfb80a31b0 0x7fbfb80a5670 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbfc80098d0 tx=0x7fbfc8009f70 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:01.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.290+0000 7fbfdb03c740 1 -- 172.21.15.31:0/4155372599 shutdown_connections 2024-03-21T10:40:01.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.290+0000 7fbfdb03c740 1 --2- 172.21.15.31:0/4155372599 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e350540de0 0x55e350543230 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.290+0000 7fbfdb03c740 1 --2- 172.21.15.31:0/4155372599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e350538260 0x55e350540810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.290+0000 7fbfdb03c740 1 --2- 172.21.15.31:0/4155372599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3505378b0 0x55e350537c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.290+0000 7fbfdb03c740 1 -- 172.21.15.31:0/4155372599 >> 172.21.15.31:0/4155372599 conn(0x55e350531f80 msgr2=0x55e350532360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:01.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.290+0000 7fbfdb03c740 1 -- 172.21.15.31:0/4155372599 shutdown_connections 2024-03-21T10:40:01.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.290+0000 7fbfdb03c740 1 -- 172.21.15.31:0/4155372599 wait complete. 2024-03-21T10:40:01.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.290+0000 7fbfdb03c740 1 -- start start 2024-03-21T10:40:01.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.291+0000 7fbfdb03c740 1 Processor -- start 2024-03-21T10:40:01.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.291+0000 7fbfdb03c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e3505378b0 0x55e350540570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.291+0000 7fbfdb03c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e350538260 0x55e35053d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.291+0000 7fbfdb03c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e350540de0 0x55e35053de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.291+0000 7fbfdb03c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e35049b2c0 con 0x55e350540de0 2024-03-21T10:40:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.291+0000 7fbfdb03c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3504570d0 con 0x55e350538260 2024-03-21T10:40:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.291+0000 7fbfdb03c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e350543b10 con 0x55e3505378b0 2024-03-21T10:40:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.291+0000 7fbfdb03c740 10 client.0 _ll_register_callbacks cb 0x55e35045be80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.291+0000 7fbfd5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e350538260 0x55e35053d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.291+0000 7fbfd5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e350538260 0x55e35053d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:41896/0 (socket says 172.21.15.31:41896) 2024-03-21T10:40:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.291+0000 7fbfd5000640 1 -- 172.21.15.31:0/946060479 learned_addr learned my addr 172.21.15.31:0/946060479 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.291+0000 7fbfd6400640 1 --2- 172.21.15.31:0/946060479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e350540de0 0x55e35053de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.292+0000 7fbfd5000640 1 -- 172.21.15.31:0/946060479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e3505378b0 msgr2=0x55e350540570 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:40:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.292+0000 7fbfd5000640 1 --2- 172.21.15.31:0/946060479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e3505378b0 0x55e350540570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.292+0000 7fbfd5000640 1 -- 172.21.15.31:0/946060479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e350540de0 msgr2=0x55e35053de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.292+0000 7fbfd5000640 1 --2- 172.21.15.31:0/946060479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e350540de0 0x55e35053de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.292+0000 7fbfd5000640 1 -- 172.21.15.31:0/946060479 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e3501af3f0 con 0x55e350538260 2024-03-21T10:40:01.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.292+0000 7fbfd5000640 1 --2- 172.21.15.31:0/946060479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e350538260 0x55e35053d900 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc000c00 tx=0x7fbfbc0bd1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fccea622d2da140 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:01.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.292+0000 7fbfb7400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:40:01.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.292+0000 7fbfb7400640 1 -- 172.21.15.31:0/946060479 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc0c16a0 con 0x55e350538260 2024-03-21T10:40:01.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.292+0000 7fbfdb03c740 1 -- 172.21.15.31:0/946060479 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbfbc032e50 con 0x55e350538260 2024-03-21T10:40:01.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.292+0000 7fbfb7400640 1 -- 172.21.15.31:0/946060479 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fbfbc0c5ce0 con 0x55e350538260 2024-03-21T10:40:01.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.293+0000 7fbfb7400640 1 -- 172.21.15.31:0/946060479 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbfbc034640 con 0x55e350538260 2024-03-21T10:40:01.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.293+0000 7fbfdb03c740 10 client.26027 fetch_fsmap learned FSMap version 24 2024-03-21T10:40:01.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.293+0000 7fbfdb03c740 1 -- 172.21.15.31:0/946060479 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55e3502797d0 con 0x55e350538260 2024-03-21T10:40:01.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.293+0000 7fbfb7400640 1 -- 172.21.15.31:0/946060479 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbfbc032e50 con 0x55e350538260 2024-03-21T10:40:01.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.293+0000 7fbfb7400640 1 -- 172.21.15.31:0/946060479 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fbfbc03c030 con 0x55e350538260 2024-03-21T10:40:01.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.293+0000 7fbfdb03c740 10 client.26027 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:40:01.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.293+0000 7fbfdb03c740 10 client.26027 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:40:01.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.293+0000 7fbfdb03c740 20 client.26027 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:01.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.293+0000 7fbfdb03c740 1 --2- 172.21.15.31:0/946060479 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55e350550630 0x55e350552a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.293+0000 7fbfdb03c740 4 client.26027 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:40:01.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.293+0000 7fbfdb03c740 1 -- 172.21.15.31:0/946060479 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e3504a8800 con 0x55e350550630 2024-03-21T10:40:01.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.294+0000 7fbfd5a00640 1 --2- 172.21.15.31:0/946060479 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55e350550630 0x55e350552a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.295+0000 7fbfb7400640 10 client.26027 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:40:01.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.295+0000 7fbfd5a00640 1 --2- 172.21.15.31:0/946060479 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55e350550630 0x55e350552a20 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=356eb3d898088030 server_cookie=8c10cc74957d1264 in_seq=0 out_seq=0 2024-03-21T10:40:01.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.296+0000 7fbfb7400640 1 -- 172.21.15.31:0/946060479 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e3504a8800 con 0x55e350550630 2024-03-21T10:40:01.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.296+0000 7fbfb7400640 10 client.26027 handle_command_reply: tid=0 2024-03-21T10:40:01.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.296+0000 7fbfdb03c740 1 client.26027 shutdown 2024-03-21T10:40:01.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.296+0000 7fbfdb03c740 1 -- 172.21.15.31:0/946060479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e350538260 msgr2=0x55e35053d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.296+0000 7fbfdb03c740 1 --2- 172.21.15.31:0/946060479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e350538260 0x55e35053d900 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc000c00 tx=0x7fbfbc0bd1c0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.296+0000 7fbfdb03c740 1 -- 172.21.15.31:0/946060479 shutdown_connections 2024-03-21T10:40:01.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.296+0000 7fbfdb03c740 1 --2- 172.21.15.31:0/946060479 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55e350550630 0x55e350552a20 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.296+0000 7fbfdb03c740 1 --2- 172.21.15.31:0/946060479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e350540de0 0x55e35053de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.296+0000 7fbfdb03c740 1 --2- 172.21.15.31:0/946060479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e350538260 0x55e35053d900 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.296+0000 7fbfdb03c740 1 --2- 172.21.15.31:0/946060479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e3505378b0 0x55e350540570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.296+0000 7fbfdb03c740 1 -- 172.21.15.31:0/946060479 >> 172.21.15.31:0/946060479 conn(0x55e350531f80 msgr2=0x55e350533cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:01.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.296+0000 7fbfb7400640 0 client.26027 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:40:01.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.297+0000 7fbfdb03c740 1 -- 172.21.15.31:0/946060479 shutdown_connections 2024-03-21T10:40:01.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.297+0000 7fbfdb03c740 1 -- 172.21.15.31:0/946060479 wait complete. 2024-03-21T10:40:01.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.297+0000 7fbfdb03c740 20 client.26027 trim_cache size 0 max 16384 2024-03-21T10:40:01.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.311+0000 7fbfdb03c740 1 Processor -- start 2024-03-21T10:40:01.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.311+0000 7fbfdb03c740 1 -- start start 2024-03-21T10:40:01.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.311+0000 7fbfdb03c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e350531f80 0x55e350540030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.311+0000 7fbfdb03c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3505378b0 0x55e350540570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.311+0000 7fbfdb03c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e350538260 0x55e35054c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.311+0000 7fbfdb03c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd01b2e80 con 0x55e3505378b0 2024-03-21T10:40:01.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.311+0000 7fbfdb03c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd01b0980 con 0x55e350531f80 2024-03-21T10:40:01.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.311+0000 7fbfdb03c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd01aea20 con 0x55e350538260 2024-03-21T10:40:01.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.312+0000 7fbfd5a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e350531f80 0x55e350540030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.312+0000 7fbfd5a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e350531f80 0x55e350540030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:41910/0 (socket says 172.21.15.31:41910) 2024-03-21T10:40:01.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.312+0000 7fbfd5a00640 1 -- 172.21.15.31:0/3306205317 learned_addr learned my addr 172.21.15.31:0/3306205317 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:01.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.312+0000 7fbfd6400640 1 --2- 172.21.15.31:0/3306205317 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e350538260 0x55e35054c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.312+0000 7fbfd5000640 1 --2- 172.21.15.31:0/3306205317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3505378b0 0x55e350540570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.312+0000 7fbfd5a00640 1 -- 172.21.15.31:0/3306205317 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e350538260 msgr2=0x55e35054c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:01.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.312+0000 7fbfd5a00640 1 --2- 172.21.15.31:0/3306205317 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e350538260 0x55e35054c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.312+0000 7fbfd5a00640 1 -- 172.21.15.31:0/3306205317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3505378b0 msgr2=0x55e350540570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:01.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.312+0000 7fbfd5a00640 1 --2- 172.21.15.31:0/3306205317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3505378b0 0x55e350540570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.312+0000 7fbfd5a00640 1 -- 172.21.15.31:0/3306205317 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e350531db0 con 0x55e350531f80 2024-03-21T10:40:01.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.312+0000 7fbfd5a00640 1 --2- 172.21.15.31:0/3306205317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e350531f80 0x55e350540030 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbfc40099d0 tx=0x7fbfc4010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=506fccc5761b9ab2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:01.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.313+0000 7fbfc2000640 1 -- 172.21.15.31:0/3306205317 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4004050 con 0x55e350531f80 2024-03-21T10:40:01.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.313+0000 7fbfc2000640 1 -- 172.21.15.31:0/3306205317 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbfc4022070 con 0x55e350531f80 2024-03-21T10:40:01.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.313+0000 7fbfc2000640 1 -- 172.21.15.31:0/3306205317 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4003850 con 0x55e350531f80 2024-03-21T10:40:01.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.313+0000 7fbfdb03c740 1 -- 172.21.15.31:0/3306205317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e350531f80 msgr2=0x55e350540030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.313+0000 7fbfdb03c740 1 --2- 172.21.15.31:0/3306205317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e350531f80 0x55e350540030 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbfc40099d0 tx=0x7fbfc4010800 comp rx=0 tx=0).stop 2024-03-21T10:40:01.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.314+0000 7fbfdb03c740 1 -- 172.21.15.31:0/3306205317 shutdown_connections 2024-03-21T10:40:01.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.314+0000 7fbfdb03c740 1 --2- 172.21.15.31:0/3306205317 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e350538260 0x55e35054c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.314+0000 7fbfdb03c740 1 --2- 172.21.15.31:0/3306205317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3505378b0 0x55e350540570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.314+0000 7fbfdb03c740 1 --2- 172.21.15.31:0/3306205317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e350531f80 0x55e350540030 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.314+0000 7fbfdb03c740 1 -- 172.21.15.31:0/3306205317 >> 172.21.15.31:0/3306205317 conn(0x55e35052d440 msgr2=0x55e350538f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:01.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.314+0000 7fbfdb03c740 1 -- 172.21.15.31:0/3306205317 shutdown_connections 2024-03-21T10:40:01.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.314+0000 7fbfdb03c740 1 -- 172.21.15.31:0/3306205317 wait complete. 2024-03-21T10:40:01.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.314+0000 7fbfdb03c740 1 -- start start 2024-03-21T10:40:01.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.314+0000 7fbfdb03c740 1 Processor -- start 2024-03-21T10:40:01.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.315+0000 7fbfdb03c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e350531f80 0x55e350548b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.315+0000 7fbfdb03c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3505378b0 0x55e350544e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.315+0000 7fbfdb03c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e350538260 0x55e3505453c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.315+0000 7fbfdb03c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd01af020 con 0x55e3505378b0 2024-03-21T10:40:01.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.315+0000 7fbfdb03c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd01b0320 con 0x55e350538260 2024-03-21T10:40:01.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.315+0000 7fbfdb03c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd01af660 con 0x55e350531f80 2024-03-21T10:40:01.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.315+0000 7fbfdb03c740 10 client.0 _ll_register_callbacks cb 0x7fbfc8009a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:01.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.315+0000 7fbfd6400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e350538260 0x55e3505453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.315+0000 7fbfd6400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e350538260 0x55e3505453c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:41922/0 (socket says 172.21.15.31:41922) 2024-03-21T10:40:01.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.315+0000 7fbfd6400640 1 -- 172.21.15.31:0/560675845 learned_addr learned my addr 172.21.15.31:0/560675845 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:01.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.315+0000 7fbfd5000640 1 --2- 172.21.15.31:0/560675845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3505378b0 0x55e350544e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.316+0000 7fbfd5a00640 1 --2- 172.21.15.31:0/560675845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e350531f80 0x55e350548b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.316+0000 7fbfd6400640 1 -- 172.21.15.31:0/560675845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e350531f80 msgr2=0x55e350548b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:01.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.316+0000 7fbfd6400640 1 --2- 172.21.15.31:0/560675845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e350531f80 0x55e350548b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.316+0000 7fbfd6400640 1 -- 172.21.15.31:0/560675845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3505378b0 msgr2=0x55e350544e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:01.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.316+0000 7fbfd6400640 1 --2- 172.21.15.31:0/560675845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3505378b0 0x55e350544e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.316+0000 7fbfd6400640 1 -- 172.21.15.31:0/560675845 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e350552e80 con 0x55e350538260 2024-03-21T10:40:01.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.316+0000 7fbfd5000640 1 --2- 172.21.15.31:0/560675845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3505378b0 0x55e350544e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:01.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.316+0000 7fbfd6400640 1 --2- 172.21.15.31:0/560675845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e350538260 0x55e3505453c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fbfc801e970 tx=0x7fbfc8009d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5a1834266de6458 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:01.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.316+0000 7fbfb7400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:40:01.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.316+0000 7fbfb7400640 1 -- 172.21.15.31:0/560675845 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc802e060 con 0x55e350538260 2024-03-21T10:40:01.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.317+0000 7fbfdb03c740 1 -- 172.21.15.31:0/560675845 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbfc4022070 con 0x55e350538260 2024-03-21T10:40:01.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.317+0000 7fbfb7400640 1 -- 172.21.15.31:0/560675845 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fbfc8005250 con 0x55e350538260 2024-03-21T10:40:01.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.317+0000 7fbfb7400640 1 -- 172.21.15.31:0/560675845 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbfc8035050 con 0x55e350538260 2024-03-21T10:40:01.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.317+0000 7fbfdb03c740 10 client.26033 fetch_fsmap learned FSMap version 24 2024-03-21T10:40:01.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.317+0000 7fbfdb03c740 1 -- 172.21.15.31:0/560675845 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55e3505b52f0 con 0x55e350538260 2024-03-21T10:40:01.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.317+0000 7fbfb7400640 1 -- 172.21.15.31:0/560675845 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbfc4022070 con 0x55e350538260 2024-03-21T10:40:01.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.318+0000 7fbfb7400640 1 -- 172.21.15.31:0/560675845 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fbfc800b040 con 0x55e350538260 2024-03-21T10:40:01.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.318+0000 7fbfdb03c740 10 client.26033 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:40:01.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.318+0000 7fbfdb03c740 10 client.26033 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:40:01.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.318+0000 7fbfdb03c740 20 client.26033 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:01.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.318+0000 7fbfdb03c740 1 --2- 172.21.15.31:0/560675845 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55e350540de0 0x55e3505b7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.318+0000 7fbfdb03c740 4 client.26033 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:40:01.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.318+0000 7fbfdb03c740 1 -- 172.21.15.31:0/560675845 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e3504a2b50 con 0x55e350540de0 2024-03-21T10:40:01.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.318+0000 7fbfd5a00640 1 --2- 172.21.15.31:0/560675845 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55e350540de0 0x55e3505b7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.319+0000 7fbfd5a00640 1 --2- 172.21.15.31:0/560675845 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55e350540de0 0x55e3505b7a40 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5f32bcb47b3c51ec server_cookie=d73b88342deba9f1 in_seq=0 out_seq=0 2024-03-21T10:40:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.319+0000 7fbfb7400640 10 client.26033 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:40:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.319+0000 7fbfb7400640 1 -- 172.21.15.31:0/560675845 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e3504a2b50 con 0x55e350540de0 2024-03-21T10:40:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.319+0000 7fbfb7400640 10 client.26033 handle_command_reply: tid=0 2024-03-21T10:40:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.319+0000 7fbfdb03c740 1 client.26033 shutdown 2024-03-21T10:40:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.319+0000 7fbfdb03c740 1 -- 172.21.15.31:0/560675845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e350538260 msgr2=0x55e3505453c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.319+0000 7fbfdb03c740 1 --2- 172.21.15.31:0/560675845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e350538260 0x55e3505453c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fbfc801e970 tx=0x7fbfc8009d70 comp rx=0 tx=0).stop 2024-03-21T10:40:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.319+0000 7fbfdb03c740 1 -- 172.21.15.31:0/560675845 shutdown_connections 2024-03-21T10:40:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.319+0000 7fbfdb03c740 1 --2- 172.21.15.31:0/560675845 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55e350540de0 0x55e3505b7a40 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.319+0000 7fbfdb03c740 1 --2- 172.21.15.31:0/560675845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e350538260 0x55e3505453c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.319+0000 7fbfdb03c740 1 --2- 172.21.15.31:0/560675845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3505378b0 0x55e350544e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.319+0000 7fbfdb03c740 1 --2- 172.21.15.31:0/560675845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e350531f80 0x55e350548b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.319+0000 7fbfdb03c740 1 -- 172.21.15.31:0/560675845 >> 172.21.15.31:0/560675845 conn(0x55e35052d440 msgr2=0x55e350533300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.319+0000 7fbfb7400640 0 client.26033 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:40:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.319+0000 7fbfdb03c740 1 -- 172.21.15.31:0/560675845 shutdown_connections 2024-03-21T10:40:01.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.319+0000 7fbfdb03c740 1 -- 172.21.15.31:0/560675845 wait complete. 2024-03-21T10:40:01.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.320+0000 7fbfdb03c740 20 client.26033 trim_cache size 0 max 16384 2024-03-21T10:40:01.320 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:40:01.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.320+0000 7fbfd6e00640 1 -- 172.21.15.31:0/1933312316 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbfb80a31b0 msgr2=0x7fbfb80a5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.320+0000 7fbfd6e00640 1 --2- 172.21.15.31:0/1933312316 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbfb80a31b0 0x7fbfb80a5670 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbfc80098d0 tx=0x7fbfc8009f70 comp rx=0 tx=0).stop 2024-03-21T10:40:01.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.320+0000 7fbfd6e00640 1 -- 172.21.15.31:0/1933312316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfd0104020 msgr2=0x7fbfd019a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.320+0000 7fbfd6e00640 1 --2- 172.21.15.31:0/1933312316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfd0104020 0x7fbfd019a4c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc037040 tx=0x7fbfbc004590 comp rx=0 tx=0).stop 2024-03-21T10:40:01.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.320+0000 7fbfd6e00640 1 -- 172.21.15.31:0/1933312316 shutdown_connections 2024-03-21T10:40:01.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.320+0000 7fbfd6e00640 1 --2- 172.21.15.31:0/1933312316 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbfb80a31b0 0x7fbfb80a5670 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.320+0000 7fbfd6e00640 1 --2- 172.21.15.31:0/1933312316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfd0109670 0x7fbfd019e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.320+0000 7fbfd6e00640 1 --2- 172.21.15.31:0/1933312316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfd0104020 0x7fbfd019a4c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.320+0000 7fbfd6e00640 1 --2- 172.21.15.31:0/1933312316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfd0103660 0x7fbfd0199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.321+0000 7fbfd6e00640 1 -- 172.21.15.31:0/1933312316 >> 172.21.15.31:0/1933312316 conn(0x7fbfd00ff110 msgr2=0x7fbfd00ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:01.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.321+0000 7fbfd6e00640 1 -- 172.21.15.31:0/1933312316 shutdown_connections 2024-03-21T10:40:01.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.321+0000 7fbfd6e00640 1 -- 172.21.15.31:0/1933312316 wait complete. 2024-03-21T10:40:01.340 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:40:01.340 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-21T10:40:01.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.559+0000 7fe104a00640 1 -- 172.21.15.31:0/1693830784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe100109850 msgr2=0x7fe100109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.559+0000 7fe104a00640 1 --2- 172.21.15.31:0/1693830784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe100109850 0x7fe100109c30 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0009930 tx=0x7fe0f002bc30 comp rx=0 tx=0).stop 2024-03-21T10:40:01.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.560+0000 7fe104a00640 1 -- 172.21.15.31:0/1693830784 shutdown_connections 2024-03-21T10:40:01.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.560+0000 7fe104a00640 1 --2- 172.21.15.31:0/1693830784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe100104200 0x7fe100110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.560+0000 7fe104a00640 1 --2- 172.21.15.31:0/1693830784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe100103840 0x7fe100103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.560+0000 7fe104a00640 1 --2- 172.21.15.31:0/1693830784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe100109850 0x7fe100109c30 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.560+0000 7fe104a00640 1 -- 172.21.15.31:0/1693830784 >> 172.21.15.31:0/1693830784 conn(0x7fe1000ff2f0 msgr2=0x7fe100101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:01.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.560+0000 7fe104a00640 1 -- 172.21.15.31:0/1693830784 shutdown_connections 2024-03-21T10:40:01.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.560+0000 7fe104a00640 1 -- 172.21.15.31:0/1693830784 wait complete. 2024-03-21T10:40:01.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.561+0000 7fe104a00640 1 Processor -- start 2024-03-21T10:40:01.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.561+0000 7fe104a00640 1 -- start start 2024-03-21T10:40:01.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.561+0000 7fe104a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe100103840 0x7fe10010d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.561+0000 7fe104a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe100104200 0x7fe10010de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.561+0000 7fe104a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe10010e360 0x7fe1001a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.561+0000 7fe104a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe100113c70 con 0x7fe100103840 2024-03-21T10:40:01.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.561+0000 7fe104a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe100113af0 con 0x7fe10010e360 2024-03-21T10:40:01.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.561+0000 7fe104a00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe100113df0 con 0x7fe100104200 2024-03-21T10:40:01.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.562+0000 7fe0fd400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe100104200 0x7fe10010de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.562+0000 7fe0fd400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe100104200 0x7fe10010de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42498/0 (socket says 172.21.15.31:42498) 2024-03-21T10:40:01.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.562+0000 7fe0fd400640 1 -- 172.21.15.31:0/4292709169 learned_addr learned my addr 172.21.15.31:0/4292709169 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:01.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.562+0000 7fe0fde00640 1 --2- 172.21.15.31:0/4292709169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe100103840 0x7fe10010d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.562+0000 7fe0fe800640 1 --2- 172.21.15.31:0/4292709169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe10010e360 0x7fe1001a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.562+0000 7fe0fd400640 1 -- 172.21.15.31:0/4292709169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe10010e360 msgr2=0x7fe1001a0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.562+0000 7fe0fd400640 1 --2- 172.21.15.31:0/4292709169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe10010e360 0x7fe1001a0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.562+0000 7fe0fd400640 1 -- 172.21.15.31:0/4292709169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe100103840 msgr2=0x7fe10010d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.562+0000 7fe0fd400640 1 --2- 172.21.15.31:0/4292709169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe100103840 0x7fe10010d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.562+0000 7fe0fd400640 1 -- 172.21.15.31:0/4292709169 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1001a0690 con 0x7fe100104200 2024-03-21T10:40:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.562+0000 7fe0fe800640 1 --2- 172.21.15.31:0/4292709169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe10010e360 0x7fe1001a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.562+0000 7fe0fde00640 1 --2- 172.21.15.31:0/4292709169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe100103840 0x7fe10010d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.562+0000 7fe0fd400640 1 --2- 172.21.15.31:0/4292709169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe100104200 0x7fe10010de20 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec00e9f0 tx=0x7fe0ec00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.563+0000 7fe0f6a00640 1 -- 172.21.15.31:0/4292709169 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec00cd80 con 0x7fe100104200 2024-03-21T10:40:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.563+0000 7fe0f6a00640 1 -- 172.21.15.31:0/4292709169 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe0ec010910 con 0x7fe100104200 2024-03-21T10:40:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.563+0000 7fe0f6a00640 1 -- 172.21.15.31:0/4292709169 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec018980 con 0x7fe100104200 2024-03-21T10:40:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.563+0000 7fe104a00640 1 -- 172.21.15.31:0/4292709169 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1001a0980 con 0x7fe100104200 2024-03-21T10:40:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.563+0000 7fe104a00640 1 -- 172.21.15.31:0/4292709169 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1001a0e80 con 0x7fe100104200 2024-03-21T10:40:01.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.568+0000 7fe1077a2740 1 Processor -- start 2024-03-21T10:40:01.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.568+0000 7fe1077a2740 1 -- start start 2024-03-21T10:40:01.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.568+0000 7fe0f6a00640 1 -- 172.21.15.31:0/4292709169 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) v1 ==== 138087+0+0 (secure 0 0 0) 0x7fe0ec010ac0 con 0x7fe100104200 2024-03-21T10:40:01.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.569+0000 7fe1077a2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7edb81530 0x55f7edb81910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.569+0000 7fe1077a2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7edb81ee0 0x55f7edb97620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.569+0000 7fe1077a2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7edb97c40 0x55f7edb980c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.569+0000 7fe1077a2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7edae52c0 con 0x55f7edb81530 2024-03-21T10:40:01.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.569+0000 7fe1077a2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7eda8dc90 con 0x55f7edb81ee0 2024-03-21T10:40:01.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.569+0000 7fe1077a2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7edae1360 con 0x55f7edb97c40 2024-03-21T10:40:01.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.569+0000 7fe0fde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7edb81ee0 0x55f7edb97620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.569+0000 7fe0fde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7edb81ee0 0x55f7edb97620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:41942/0 (socket says 172.21.15.31:41942) 2024-03-21T10:40:01.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.569+0000 7fe0fde00640 1 -- 172.21.15.31:0/533752203 learned_addr learned my addr 172.21.15.31:0/533752203 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:01.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.569+0000 7fe0fd400640 1 --2- 172.21.15.31:0/533752203 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7edb97c40 0x55f7edb980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.570+0000 7fe0fde00640 1 -- 172.21.15.31:0/533752203 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7edb97c40 msgr2=0x55f7edb980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:01.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.570+0000 7fe0fde00640 1 --2- 172.21.15.31:0/533752203 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7edb97c40 0x55f7edb980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.570+0000 7fe0fde00640 1 -- 172.21.15.31:0/533752203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7edb81530 msgr2=0x55f7edb81910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:40:01.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.570+0000 7fe0fde00640 1 --2- 172.21.15.31:0/533752203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7edb81530 0x55f7edb81910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.570+0000 7fe0fde00640 1 -- 172.21.15.31:0/533752203 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f7edab2080 con 0x55f7edb81ee0 2024-03-21T10:40:01.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.570+0000 7fe0f6a00640 1 --2- 172.21.15.31:0/4292709169 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe0e40a3230 0x7fe0e40a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.570+0000 7fe0fde00640 1 --2- 172.21.15.31:0/533752203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7edb81ee0 0x55f7edb97620 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0036040 tx=0x7fe0f0005e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=768511f95aaba070 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:01.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.570+0000 7fe0f6a00640 1 -- 172.21.15.31:0/4292709169 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fe0ec0d01c0 con 0x7fe100104200 2024-03-21T10:40:01.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.571+0000 7fe0eb000640 1 -- 172.21.15.31:0/533752203 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f0066020 con 0x55f7edb81ee0 2024-03-21T10:40:01.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.571+0000 7fe0eb000640 1 -- 172.21.15.31:0/533752203 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe0f002c880 con 0x55f7edb81ee0 2024-03-21T10:40:01.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.571+0000 7fe0eb000640 1 -- 172.21.15.31:0/533752203 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f0005420 con 0x55f7edb81ee0 2024-03-21T10:40:01.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.571+0000 7fe0fe800640 1 --2- 172.21.15.31:0/4292709169 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe0e40a3230 0x7fe0e40a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.572+0000 7fe1077a2740 1 -- 172.21.15.31:0/533752203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7edb81ee0 msgr2=0x55f7edb97620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.572+0000 7fe1077a2740 1 --2- 172.21.15.31:0/533752203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7edb81ee0 0x55f7edb97620 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0036040 tx=0x7fe0f0005e60 comp rx=0 tx=0).stop 2024-03-21T10:40:01.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.572+0000 7fe1077a2740 1 -- 172.21.15.31:0/533752203 shutdown_connections 2024-03-21T10:40:01.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.572+0000 7fe1077a2740 1 --2- 172.21.15.31:0/533752203 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7edb97c40 0x55f7edb980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.572+0000 7fe1077a2740 1 --2- 172.21.15.31:0/533752203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7edb81ee0 0x55f7edb97620 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.572+0000 7fe1077a2740 1 --2- 172.21.15.31:0/533752203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7edb81530 0x55f7edb81910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.572+0000 7fe1077a2740 1 -- 172.21.15.31:0/533752203 >> 172.21.15.31:0/533752203 conn(0x55f7edb7bda0 msgr2=0x55f7edb7c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:01.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.572+0000 7fe1077a2740 1 -- 172.21.15.31:0/533752203 shutdown_connections 2024-03-21T10:40:01.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.572+0000 7fe1077a2740 1 -- 172.21.15.31:0/533752203 wait complete. 2024-03-21T10:40:01.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.572+0000 7fe0fe800640 1 --2- 172.21.15.31:0/4292709169 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe0e40a3230 0x7fe0e40a56f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8005fd0 tx=0x7fe0f8009500 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:01.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.572+0000 7fe1077a2740 1 -- start start 2024-03-21T10:40:01.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.573+0000 7fe1077a2740 1 Processor -- start 2024-03-21T10:40:01.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.573+0000 7fe1077a2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7edb81530 0x55f7edb88ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.573+0000 7fe1077a2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7edb81ee0 0x55f7edb88ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.573+0000 7fe1077a2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7edb97c40 0x55f7edb83eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.573+0000 7fe0fde00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7edb81530 0x55f7edb88ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.574+0000 7fe0fde00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7edb81530 0x55f7edb88ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42516/0 (socket says 172.21.15.31:42516) 2024-03-21T10:40:01.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.574+0000 7fe0fde00640 1 -- 172.21.15.31:0/1688978383 learned_addr learned my addr 172.21.15.31:0/1688978383 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:01.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.574+0000 7fe1077a2740 1 -- 172.21.15.31:0/1688978383 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7edb827b0 con 0x55f7edb97c40 2024-03-21T10:40:01.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.574+0000 7fe0fe800640 1 --2- 172.21.15.31:0/1688978383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7edb97c40 0x55f7edb83eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.574+0000 7fe1077a2740 1 -- 172.21.15.31:0/1688978383 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7edaa10d0 con 0x55f7edb81ee0 2024-03-21T10:40:01.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.574+0000 7fe1077a2740 1 -- 172.21.15.31:0/1688978383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7edb82930 con 0x55f7edb81530 2024-03-21T10:40:01.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.574+0000 7fe0fd400640 1 --2- 172.21.15.31:0/1688978383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7edb81ee0 0x55f7edb88ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.574+0000 7fe1077a2740 10 client.0 _ll_register_callbacks cb 0x55f7edaa5e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:01.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.574+0000 7fe0fde00640 1 -- 172.21.15.31:0/1688978383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7edb81ee0 msgr2=0x55f7edb88ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:01.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.574+0000 7fe0fde00640 1 --2- 172.21.15.31:0/1688978383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7edb81ee0 0x55f7edb88ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.574+0000 7fe0fde00640 1 -- 172.21.15.31:0/1688978383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7edb97c40 msgr2=0x55f7edb83eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:01.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.574+0000 7fe0fde00640 1 --2- 172.21.15.31:0/1688978383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7edb97c40 0x55f7edb83eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.574+0000 7fe0fde00640 1 -- 172.21.15.31:0/1688978383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f7ed7f93f0 con 0x55f7edb81530 2024-03-21T10:40:01.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.574+0000 7fe0fe800640 1 --2- 172.21.15.31:0/1688978383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7edb97c40 0x55f7edb83eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:40:01.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.575+0000 7fe0fde00640 1 --2- 172.21.15.31:0/1688978383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7edb81530 0x55f7edb88ab0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0006010 tx=0x7fe0f0005350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff22199832f8476c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:01.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.575+0000 7fe0df400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:40:01.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.575+0000 7fe0df400640 1 -- 172.21.15.31:0/1688978383 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f0066070 con 0x55f7edb81530 2024-03-21T10:40:01.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.575+0000 7fe0df400640 1 -- 172.21.15.31:0/1688978383 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fe0f005b040 con 0x55f7edb81530 2024-03-21T10:40:01.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.575+0000 7fe1077a2740 1 -- 172.21.15.31:0/1688978383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe0f002c880 con 0x55f7edb81530 2024-03-21T10:40:01.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.575+0000 7fe0df400640 1 -- 172.21.15.31:0/1688978383 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe0f0078050 con 0x55f7edb81530 2024-03-21T10:40:01.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.575+0000 7fe1077a2740 10 client.26026 fetch_fsmap learned FSMap version 24 2024-03-21T10:40:01.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.575+0000 7fe1077a2740 1 -- 172.21.15.31:0/1688978383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55f7ed8c37d0 con 0x55f7edb81530 2024-03-21T10:40:01.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.576+0000 7fe0df400640 1 -- 172.21.15.31:0/1688978383 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe0f002c880 con 0x55f7edb81530 2024-03-21T10:40:01.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.576+0000 7fe0df400640 1 -- 172.21.15.31:0/1688978383 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fe0f0062660 con 0x55f7edb81530 2024-03-21T10:40:01.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.576+0000 7fe1077a2740 10 client.26026 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:40:01.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.576+0000 7fe1077a2740 10 client.26026 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:40:01.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.576+0000 7fe1077a2740 20 client.26026 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:01.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.576+0000 7fe1077a2740 1 --2- 172.21.15.31:0/1688978383 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55f7edb93b00 0x55f7edb95f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.576+0000 7fe1077a2740 4 client.26026 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:40:01.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.576+0000 7fe1077a2740 1 -- 172.21.15.31:0/1688978383 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f7edaf2800 con 0x55f7edb93b00 2024-03-21T10:40:01.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.578+0000 7fe0fd400640 1 --2- 172.21.15.31:0/1688978383 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55f7edb93b00 0x55f7edb95f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.579+0000 7fe0fd400640 1 --2- 172.21.15.31:0/1688978383 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55f7edb93b00 0x55f7edb95f30 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=463b6a20b47c9d88 server_cookie=2c371ffffd4ff0a7 in_seq=0 out_seq=0 2024-03-21T10:40:01.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.579+0000 7fe0df400640 10 client.26026 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:40:01.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.580+0000 7fe0df400640 1 -- 172.21.15.31:0/1688978383 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f7edaf2800 con 0x55f7edb93b00 2024-03-21T10:40:01.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.580+0000 7fe0df400640 10 client.26026 handle_command_reply: tid=0 2024-03-21T10:40:01.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.580+0000 7fe1077a2740 1 client.26026 shutdown 2024-03-21T10:40:01.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.581+0000 7fe1077a2740 1 -- 172.21.15.31:0/1688978383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7edb81530 msgr2=0x55f7edb88ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.581+0000 7fe1077a2740 1 --2- 172.21.15.31:0/1688978383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7edb81530 0x55f7edb88ab0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0006010 tx=0x7fe0f0005350 comp rx=0 tx=0).stop 2024-03-21T10:40:01.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.581+0000 7fe1077a2740 1 -- 172.21.15.31:0/1688978383 shutdown_connections 2024-03-21T10:40:01.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.581+0000 7fe1077a2740 1 --2- 172.21.15.31:0/1688978383 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55f7edb93b00 0x55f7edb95f30 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.581+0000 7fe1077a2740 1 --2- 172.21.15.31:0/1688978383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7edb97c40 0x55f7edb83eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.581+0000 7fe1077a2740 1 --2- 172.21.15.31:0/1688978383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7edb81ee0 0x55f7edb88ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.581+0000 7fe1077a2740 1 --2- 172.21.15.31:0/1688978383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7edb81530 0x55f7edb88ab0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.581+0000 7fe1077a2740 1 -- 172.21.15.31:0/1688978383 >> 172.21.15.31:0/1688978383 conn(0x55f7edb7bda0 msgr2=0x55f7edb7daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:01.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.581+0000 7fe0df400640 0 client.26026 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:40:01.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.581+0000 7fe1077a2740 1 -- 172.21.15.31:0/1688978383 shutdown_connections 2024-03-21T10:40:01.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.581+0000 7fe1077a2740 1 -- 172.21.15.31:0/1688978383 wait complete. 2024-03-21T10:40:01.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.582+0000 7fe1077a2740 20 client.26026 trim_cache size 0 max 16384 2024-03-21T10:40:01.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.608+0000 7fe1077a2740 1 Processor -- start 2024-03-21T10:40:01.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.608+0000 7fe1077a2740 1 -- start start 2024-03-21T10:40:01.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.608+0000 7fe1077a2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7edb81530 0x55f7edba3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.608+0000 7fe1077a2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7edb81ee0 0x55f7edb83b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.608+0000 7fe1077a2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7edb97c40 0x55f7edb96f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.608+0000 7fe1077a2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1001b73c0 con 0x55f7edb81530 2024-03-21T10:40:01.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.608+0000 7fe1077a2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1001b7a20 con 0x55f7edb97c40 2024-03-21T10:40:01.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.608+0000 7fe1077a2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1001b5520 con 0x55f7edb81ee0 2024-03-21T10:40:01.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.609+0000 7fe0fd400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7edb81ee0 0x55f7edb83b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.609+0000 7fe0fde00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7edb81530 0x55f7edba3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.609+0000 7fe0fd400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7edb81ee0 0x55f7edb83b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42528/0 (socket says 172.21.15.31:42528) 2024-03-21T10:40:01.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.609+0000 7fe0fd400640 1 -- 172.21.15.31:0/1623391525 learned_addr learned my addr 172.21.15.31:0/1623391525 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:01.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.609+0000 7fe0fe800640 1 --2- 172.21.15.31:0/1623391525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7edb97c40 0x55f7edb96f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.609+0000 7fe0fd400640 1 -- 172.21.15.31:0/1623391525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7edb97c40 msgr2=0x55f7edb96f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:01.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.609+0000 7fe0fd400640 1 --2- 172.21.15.31:0/1623391525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7edb97c40 0x55f7edb96f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.609+0000 7fe0fd400640 1 -- 172.21.15.31:0/1623391525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7edb81530 msgr2=0x55f7edba3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:01.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.609+0000 7fe0fd400640 1 --2- 172.21.15.31:0/1623391525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7edb81530 0x55f7edba3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.609+0000 7fe0fd400640 1 -- 172.21.15.31:0/1623391525 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f7edb762c0 con 0x55f7edb81ee0 2024-03-21T10:40:01.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.609+0000 7fe0fe800640 1 --2- 172.21.15.31:0/1623391525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7edb97c40 0x55f7edb96f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:40:01.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.610+0000 7fe0fd400640 1 --2- 172.21.15.31:0/1623391525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7edb81ee0 0x55f7edb83b90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec0cdbd0 tx=0x7fe0ec0d3cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd389bcc2a197de6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:01.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.610+0000 7fe0ea600640 1 -- 172.21.15.31:0/1623391525 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec09bb60 con 0x55f7edb81ee0 2024-03-21T10:40:01.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.610+0000 7fe0ea600640 1 -- 172.21.15.31:0/1623391525 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe0ec09c720 con 0x55f7edb81ee0 2024-03-21T10:40:01.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.610+0000 7fe0ea600640 1 -- 172.21.15.31:0/1623391525 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec09b980 con 0x55f7edb81ee0 2024-03-21T10:40:01.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.611+0000 7fe1077a2740 1 -- 172.21.15.31:0/1623391525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7edb81ee0 msgr2=0x55f7edb83b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.611+0000 7fe1077a2740 1 --2- 172.21.15.31:0/1623391525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7edb81ee0 0x55f7edb83b90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec0cdbd0 tx=0x7fe0ec0d3cd0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.611+0000 7fe1077a2740 1 -- 172.21.15.31:0/1623391525 shutdown_connections 2024-03-21T10:40:01.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.611+0000 7fe1077a2740 1 --2- 172.21.15.31:0/1623391525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7edb97c40 0x55f7edb96f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.611+0000 7fe1077a2740 1 --2- 172.21.15.31:0/1623391525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7edb81ee0 0x55f7edb83b90 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.611+0000 7fe1077a2740 1 --2- 172.21.15.31:0/1623391525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7edb81530 0x55f7edba3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.611+0000 7fe1077a2740 1 -- 172.21.15.31:0/1623391525 >> 172.21.15.31:0/1623391525 conn(0x55f7edb7bda0 msgr2=0x55f7edb8ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:01.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.611+0000 7fe1077a2740 1 -- 172.21.15.31:0/1623391525 shutdown_connections 2024-03-21T10:40:01.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.611+0000 7fe1077a2740 1 -- 172.21.15.31:0/1623391525 wait complete. 2024-03-21T10:40:01.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.612+0000 7fe1077a2740 1 -- start start 2024-03-21T10:40:01.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.612+0000 7fe1077a2740 1 Processor -- start 2024-03-21T10:40:01.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.612+0000 7fe1077a2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7edb81530 0x55f7edba1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.612+0000 7fe1077a2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7edb81ee0 0x55f7edba19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.613+0000 7fe1077a2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7edb97c40 0x55f7edbee540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.613+0000 7fe1077a2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1001a16f0 con 0x55f7edb97c40 2024-03-21T10:40:01.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.613+0000 7fe1077a2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1001b4ec0 con 0x55f7edb81530 2024-03-21T10:40:01.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.613+0000 7fe1077a2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe100073190 con 0x55f7edb81ee0 2024-03-21T10:40:01.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.613+0000 7fe1077a2740 10 client.0 _ll_register_callbacks cb 0x7fe0ec0d3ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:01.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.613+0000 7fe0fde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7edb81530 0x55f7edba1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.613+0000 7fe0fde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7edb81530 0x55f7edba1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:41972/0 (socket says 172.21.15.31:41972) 2024-03-21T10:40:01.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.613+0000 7fe0fde00640 1 -- 172.21.15.31:0/2210824477 learned_addr learned my addr 172.21.15.31:0/2210824477 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:01.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.613+0000 7fe0fe800640 1 --2- 172.21.15.31:0/2210824477 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7edb97c40 0x55f7edbee540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.613+0000 7fe0fd400640 1 --2- 172.21.15.31:0/2210824477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7edb81ee0 0x55f7edba19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.613+0000 7fe0fde00640 1 -- 172.21.15.31:0/2210824477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7edb81ee0 msgr2=0x55f7edba19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:01.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.613+0000 7fe0fde00640 1 --2- 172.21.15.31:0/2210824477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7edb81ee0 0x55f7edba19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.614+0000 7fe0fde00640 1 -- 172.21.15.31:0/2210824477 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7edb97c40 msgr2=0x55f7edbee540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:01.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.614+0000 7fe0fde00640 1 --2- 172.21.15.31:0/2210824477 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7edb97c40 0x55f7edbee540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.614+0000 7fe0fde00640 1 -- 172.21.15.31:0/2210824477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f7edbeeb30 con 0x55f7edb81530 2024-03-21T10:40:01.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.614+0000 7fe0fe800640 1 --2- 172.21.15.31:0/2210824477 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7edb97c40 0x55f7edbee540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:40:01.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.614+0000 7fe0fde00640 1 --2- 172.21.15.31:0/2210824477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7edb81530 0x55f7edba1470 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0009800 tx=0x7fe0f00626b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1aea588133083732 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:01.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.614+0000 7fe0df400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:40:01.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.614+0000 7fe0df400640 1 -- 172.21.15.31:0/2210824477 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f0066030 con 0x55f7edb81530 2024-03-21T10:40:01.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.614+0000 7fe1077a2740 1 -- 172.21.15.31:0/2210824477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe0ec09c720 con 0x55f7edb81530 2024-03-21T10:40:01.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.614+0000 7fe0df400640 1 -- 172.21.15.31:0/2210824477 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fe0f0070040 con 0x55f7edb81530 2024-03-21T10:40:01.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.615+0000 7fe0df400640 1 -- 172.21.15.31:0/2210824477 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe0f002c4b0 con 0x55f7edb81530 2024-03-21T10:40:01.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.615+0000 7fe1077a2740 10 client.26051 fetch_fsmap learned FSMap version 24 2024-03-21T10:40:01.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.615+0000 7fe1077a2740 1 -- 172.21.15.31:0/2210824477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55f7edbf5eb0 con 0x55f7edb81530 2024-03-21T10:40:01.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.615+0000 7fe0df400640 1 -- 172.21.15.31:0/2210824477 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe0ec09c720 con 0x55f7edb81530 2024-03-21T10:40:01.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.615+0000 7fe0df400640 1 -- 172.21.15.31:0/2210824477 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fe0f0003ad0 con 0x55f7edb81530 2024-03-21T10:40:01.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.615+0000 7fe1077a2740 10 client.26051 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:40:01.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.615+0000 7fe1077a2740 10 client.26051 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:40:01.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.615+0000 7fe1077a2740 20 client.26051 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:01.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.615+0000 7fe1077a2740 1 --2- 172.21.15.31:0/2210824477 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55f7edbf62e0 0x55f7edbf8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.615+0000 7fe1077a2740 4 client.26051 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:40:01.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.615+0000 7fe1077a2740 1 -- 172.21.15.31:0/2210824477 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f7edaecb50 con 0x55f7edbf62e0 2024-03-21T10:40:01.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.616+0000 7fe0fd400640 1 --2- 172.21.15.31:0/2210824477 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55f7edbf62e0 0x55f7edbf8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.617+0000 7fe0fd400640 1 --2- 172.21.15.31:0/2210824477 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55f7edbf62e0 0x55f7edbf8700 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4eb1a50b26cfad7f server_cookie=179e6cffe592dbe1 in_seq=0 out_seq=0 2024-03-21T10:40:01.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.617+0000 7fe0df400640 10 client.26051 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:40:01.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.617+0000 7fe0df400640 1 -- 172.21.15.31:0/2210824477 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f7edaecb50 con 0x55f7edbf62e0 2024-03-21T10:40:01.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.617+0000 7fe0df400640 10 client.26051 handle_command_reply: tid=0 2024-03-21T10:40:01.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.617+0000 7fe1077a2740 1 client.26051 shutdown 2024-03-21T10:40:01.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.618+0000 7fe1077a2740 1 -- 172.21.15.31:0/2210824477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7edb81530 msgr2=0x55f7edba1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.618+0000 7fe1077a2740 1 --2- 172.21.15.31:0/2210824477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7edb81530 0x55f7edba1470 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0009800 tx=0x7fe0f00626b0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.618+0000 7fe1077a2740 1 -- 172.21.15.31:0/2210824477 shutdown_connections 2024-03-21T10:40:01.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.618+0000 7fe1077a2740 1 --2- 172.21.15.31:0/2210824477 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55f7edbf62e0 0x55f7edbf8700 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.618+0000 7fe1077a2740 1 --2- 172.21.15.31:0/2210824477 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7edb97c40 0x55f7edbee540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.618+0000 7fe1077a2740 1 --2- 172.21.15.31:0/2210824477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7edb81ee0 0x55f7edba19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.618+0000 7fe1077a2740 1 --2- 172.21.15.31:0/2210824477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7edb81530 0x55f7edba1470 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.618+0000 7fe1077a2740 1 -- 172.21.15.31:0/2210824477 >> 172.21.15.31:0/2210824477 conn(0x55f7edb7bda0 msgr2=0x55f7edb7a620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:01.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.618+0000 7fe0df400640 0 client.26051 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:40:01.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.618+0000 7fe1077a2740 1 -- 172.21.15.31:0/2210824477 shutdown_connections 2024-03-21T10:40:01.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.618+0000 7fe1077a2740 1 -- 172.21.15.31:0/2210824477 wait complete. 2024-03-21T10:40:01.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.619+0000 7fe1077a2740 20 client.26051 trim_cache size 0 max 16384 2024-03-21T10:40:01.617 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:40:01.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.619+0000 7fe104a00640 1 -- 172.21.15.31:0/4292709169 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe0e40a3230 msgr2=0x7fe0e40a56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.619+0000 7fe104a00640 1 --2- 172.21.15.31:0/4292709169 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe0e40a3230 0x7fe0e40a56f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8005fd0 tx=0x7fe0f8009500 comp rx=0 tx=0).stop 2024-03-21T10:40:01.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.619+0000 7fe104a00640 1 -- 172.21.15.31:0/4292709169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe100104200 msgr2=0x7fe10010de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.619+0000 7fe104a00640 1 --2- 172.21.15.31:0/4292709169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe100104200 0x7fe10010de20 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec00e9f0 tx=0x7fe0ec00eeb0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.620+0000 7fe104a00640 1 -- 172.21.15.31:0/4292709169 shutdown_connections 2024-03-21T10:40:01.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.620+0000 7fe104a00640 1 --2- 172.21.15.31:0/4292709169 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe0e40a3230 0x7fe0e40a56f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.620+0000 7fe104a00640 1 --2- 172.21.15.31:0/4292709169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe10010e360 0x7fe1001a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.620+0000 7fe104a00640 1 --2- 172.21.15.31:0/4292709169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe100104200 0x7fe10010de20 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.620+0000 7fe104a00640 1 --2- 172.21.15.31:0/4292709169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe100103840 0x7fe10010d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.620+0000 7fe104a00640 1 -- 172.21.15.31:0/4292709169 >> 172.21.15.31:0/4292709169 conn(0x7fe1000ff2f0 msgr2=0x7fe1000ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:01.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.620+0000 7fe104a00640 1 -- 172.21.15.31:0/4292709169 shutdown_connections 2024-03-21T10:40:01.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.620+0000 7fe104a00640 1 -- 172.21.15.31:0/4292709169 wait complete. 2024-03-21T10:40:01.636 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:40:01.636 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-21T10:40:01.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.833+0000 7fbf17200640 1 -- 172.21.15.31:0/802409603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf100764f0 msgr2=0x7fbf10111dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.833+0000 7fbf17200640 1 --2- 172.21.15.31:0/802409603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf100764f0 0x7fbf10111dc0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbf00009930 tx=0x7fbf0002bc20 comp rx=0 tx=0).stop 2024-03-21T10:40:01.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.833+0000 7fbf17200640 1 -- 172.21.15.31:0/802409603 shutdown_connections 2024-03-21T10:40:01.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.833+0000 7fbf17200640 1 --2- 172.21.15.31:0/802409603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf10112300 0x7fbf101146f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.833+0000 7fbf17200640 1 --2- 172.21.15.31:0/802409603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf100764f0 0x7fbf10111dc0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.833+0000 7fbf17200640 1 --2- 172.21.15.31:0/802409603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf10075bd0 0x7fbf10075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.833+0000 7fbf17200640 1 -- 172.21.15.31:0/802409603 >> 172.21.15.31:0/802409603 conn(0x7fbf100ff110 msgr2=0x7fbf10101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:01.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.833+0000 7fbf17200640 1 -- 172.21.15.31:0/802409603 shutdown_connections 2024-03-21T10:40:01.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.833+0000 7fbf17200640 1 -- 172.21.15.31:0/802409603 wait complete. 2024-03-21T10:40:01.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.834+0000 7fbf17200640 1 Processor -- start 2024-03-21T10:40:01.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.835+0000 7fbf17200640 1 -- start start 2024-03-21T10:40:01.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.835+0000 7fbf17200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf10075bd0 0x7fbf1019e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.835+0000 7fbf17200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf100764f0 0x7fbf1019e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.835+0000 7fbf17200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf10112300 0x7fbf101a2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.835+0000 7fbf17200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf10117eb0 con 0x7fbf10112300 2024-03-21T10:40:01.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.835+0000 7fbf17200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf10117d30 con 0x7fbf10075bd0 2024-03-21T10:40:01.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.835+0000 7fbf17200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf10118030 con 0x7fbf100764f0 2024-03-21T10:40:01.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.835+0000 7fbf15e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf10075bd0 0x7fbf1019e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.835+0000 7fbf15e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf10075bd0 0x7fbf1019e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:41992/0 (socket says 172.21.15.31:41992) 2024-03-21T10:40:01.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.835+0000 7fbf15e00640 1 -- 172.21.15.31:0/730977098 learned_addr learned my addr 172.21.15.31:0/730977098 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:01.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.836+0000 7fbf15400640 1 --2- 172.21.15.31:0/730977098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf100764f0 0x7fbf1019e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.836+0000 7fbf16800640 1 --2- 172.21.15.31:0/730977098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf10112300 0x7fbf101a2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.836+0000 7fbf15e00640 1 -- 172.21.15.31:0/730977098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf100764f0 msgr2=0x7fbf1019e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.836+0000 7fbf15e00640 1 --2- 172.21.15.31:0/730977098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf100764f0 0x7fbf1019e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.836+0000 7fbf15e00640 1 -- 172.21.15.31:0/730977098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf10112300 msgr2=0x7fbf101a2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.836+0000 7fbf15e00640 1 --2- 172.21.15.31:0/730977098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf10112300 0x7fbf101a2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.836+0000 7fbf15e00640 1 -- 172.21.15.31:0/730977098 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf101a3360 con 0x7fbf10075bd0 2024-03-21T10:40:01.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.836+0000 7fbf16800640 1 --2- 172.21.15.31:0/730977098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf10112300 0x7fbf101a2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:01.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.836+0000 7fbf15e00640 1 --2- 172.21.15.31:0/730977098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf10075bd0 0x7fbf1019e370 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbf0400c920 tx=0x7fbf0400cde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:01.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.837+0000 7fbf0aa00640 1 -- 172.21.15.31:0/730977098 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf04007c70 con 0x7fbf10075bd0 2024-03-21T10:40:01.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.837+0000 7fbf0aa00640 1 -- 172.21.15.31:0/730977098 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf04007e10 con 0x7fbf10075bd0 2024-03-21T10:40:01.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.837+0000 7fbf0aa00640 1 -- 172.21.15.31:0/730977098 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf04018890 con 0x7fbf10075bd0 2024-03-21T10:40:01.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.837+0000 7fbf17200640 1 -- 172.21.15.31:0/730977098 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf101a3650 con 0x7fbf10075bd0 2024-03-21T10:40:01.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.837+0000 7fbf17200640 1 -- 172.21.15.31:0/730977098 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf101aaee0 con 0x7fbf10075bd0 2024-03-21T10:40:01.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.840+0000 7fbf0aa00640 1 -- 172.21.15.31:0/730977098 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) v1 ==== 138087+0+0 (secure 0 0 0) 0x7fbf04018a50 con 0x7fbf10075bd0 2024-03-21T10:40:01.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.842+0000 7fbf1b476740 1 Processor -- start 2024-03-21T10:40:01.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.842+0000 7fbf1b476740 1 -- start start 2024-03-21T10:40:01.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.842+0000 7fbf0aa00640 1 --2- 172.21.15.31:0/730977098 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbef80a3230 0x7fbef80a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.842+0000 7fbf0aa00640 1 -- 172.21.15.31:0/730977098 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fbf040cff10 con 0x7fbf10075bd0 2024-03-21T10:40:01.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.843+0000 7fbf1b476740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e05bac08b0 0x55e05bac0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.843+0000 7fbf1b476740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e05bac1260 0x55e05bac9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.843+0000 7fbf16800640 1 --2- 172.21.15.31:0/730977098 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbef80a3230 0x7fbef80a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.843+0000 7fbf1b476740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05bac9de0 0x55e05bacc230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.843+0000 7fbf1b476740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e05ba242c0 con 0x55e05bac08b0 2024-03-21T10:40:01.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.843+0000 7fbf1b476740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e05b9ccc90 con 0x55e05bac1260 2024-03-21T10:40:01.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.843+0000 7fbf1b476740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e05ba20360 con 0x55e05bac9de0 2024-03-21T10:40:01.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.843+0000 7fbf15e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e05bac08b0 0x55e05bac0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.843+0000 7fbf15e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e05bac08b0 0x55e05bac0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36356/0 (socket says 172.21.15.31:36356) 2024-03-21T10:40:01.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.843+0000 7fbf15e00640 1 -- 172.21.15.31:0/14287491 learned_addr learned my addr 172.21.15.31:0/14287491 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:01.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.843+0000 7fbf16800640 1 --2- 172.21.15.31:0/14287491 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05bac9de0 0x55e05bacc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.843+0000 7fbf15400640 1 --2- 172.21.15.31:0/14287491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e05bac1260 0x55e05bac9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.843+0000 7fbf15e00640 1 -- 172.21.15.31:0/14287491 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05bac9de0 msgr2=0x55e05bacc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:01.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.843+0000 7fbf15e00640 1 --2- 172.21.15.31:0/14287491 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05bac9de0 0x55e05bacc230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.843+0000 7fbf15e00640 1 -- 172.21.15.31:0/14287491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e05bac1260 msgr2=0x55e05bac9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:01.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.843+0000 7fbf15e00640 1 --2- 172.21.15.31:0/14287491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e05bac1260 0x55e05bac9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.843+0000 7fbf15e00640 1 -- 172.21.15.31:0/14287491 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e05b9f1080 con 0x55e05bac08b0 2024-03-21T10:40:01.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.846+0000 7fbf15e00640 1 --2- 172.21.15.31:0/14287491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e05bac08b0 0x55e05bac0c90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbf0409bca0 tx=0x7fbf0409bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa839e39c4077f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:01.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.846+0000 7fbeff000640 1 -- 172.21.15.31:0/14287491 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf04018d40 con 0x55e05bac08b0 2024-03-21T10:40:01.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.846+0000 7fbeff000640 1 -- 172.21.15.31:0/14287491 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf0400fb30 con 0x55e05bac08b0 2024-03-21T10:40:01.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.846+0000 7fbeff000640 1 -- 172.21.15.31:0/14287491 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf040cd680 con 0x55e05bac08b0 2024-03-21T10:40:01.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.847+0000 7fbf1b476740 1 -- 172.21.15.31:0/14287491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e05bac08b0 msgr2=0x55e05bac0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.847+0000 7fbf1b476740 1 --2- 172.21.15.31:0/14287491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e05bac08b0 0x55e05bac0c90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbf0409bca0 tx=0x7fbf0409bdd0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.847+0000 7fbf16800640 1 --2- 172.21.15.31:0/730977098 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbef80a3230 0x7fbef80a56f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbf1006a9d0 tx=0x7fbf0c009d60 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:01.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.847+0000 7fbf1b476740 1 -- 172.21.15.31:0/14287491 shutdown_connections 2024-03-21T10:40:01.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.847+0000 7fbf1b476740 1 --2- 172.21.15.31:0/14287491 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05bac9de0 0x55e05bacc230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.847+0000 7fbf1b476740 1 --2- 172.21.15.31:0/14287491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e05bac1260 0x55e05bac9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.847+0000 7fbf1b476740 1 --2- 172.21.15.31:0/14287491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e05bac08b0 0x55e05bac0c90 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.847+0000 7fbf1b476740 1 -- 172.21.15.31:0/14287491 >> 172.21.15.31:0/14287491 conn(0x55e05babaf80 msgr2=0x55e05babb360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:01.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.847+0000 7fbf1b476740 1 -- 172.21.15.31:0/14287491 shutdown_connections 2024-03-21T10:40:01.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.847+0000 7fbf1b476740 1 -- 172.21.15.31:0/14287491 wait complete. 2024-03-21T10:40:01.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.848+0000 7fbf1b476740 1 -- start start 2024-03-21T10:40:01.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.848+0000 7fbf1b476740 1 Processor -- start 2024-03-21T10:40:01.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.849+0000 7fbf1b476740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05bac08b0 0x55e05bac9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.849+0000 7fbf1b476740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e05bac1260 0x55e05bac6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.849+0000 7fbf1b476740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e05bac9de0 0x55e05bac70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.849+0000 7fbf1b476740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e05baccd50 con 0x55e05bac9de0 2024-03-21T10:40:01.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.849+0000 7fbf1b476740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e05b9e00d0 con 0x55e05bac1260 2024-03-21T10:40:01.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.849+0000 7fbf1b476740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e05bacced0 con 0x55e05bac08b0 2024-03-21T10:40:01.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.849+0000 7fbf1b476740 10 client.0 _ll_register_callbacks cb 0x55e05b9e4e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:01.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.849+0000 7fbf15e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05bac08b0 0x55e05bac9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.849+0000 7fbf15e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05bac08b0 0x55e05bac9720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42570/0 (socket says 172.21.15.31:42570) 2024-03-21T10:40:01.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.849+0000 7fbf15e00640 1 -- 172.21.15.31:0/367338455 learned_addr learned my addr 172.21.15.31:0/367338455 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:01.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.849+0000 7fbf15400640 1 --2- 172.21.15.31:0/367338455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e05bac1260 0x55e05bac6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.849+0000 7fbf16800640 1 --2- 172.21.15.31:0/367338455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e05bac9de0 0x55e05bac70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.850+0000 7fbf15e00640 1 -- 172.21.15.31:0/367338455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e05bac1260 msgr2=0x55e05bac6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:01.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.850+0000 7fbf15e00640 1 --2- 172.21.15.31:0/367338455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e05bac1260 0x55e05bac6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.850+0000 7fbf15e00640 1 -- 172.21.15.31:0/367338455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e05bac9de0 msgr2=0x55e05bac70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:01.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.850+0000 7fbf15e00640 1 --2- 172.21.15.31:0/367338455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e05bac9de0 0x55e05bac70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.850+0000 7fbf15e00640 1 -- 172.21.15.31:0/367338455 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e05b7383f0 con 0x55e05bac08b0 2024-03-21T10:40:01.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.850+0000 7fbf16800640 1 --2- 172.21.15.31:0/367338455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e05bac9de0 0x55e05bac70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:01.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.850+0000 7fbf15e00640 1 --2- 172.21.15.31:0/367338455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05bac08b0 0x55e05bac9720 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbf04000c00 tx=0x7fbf04018f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b7f056ad27fac90 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:01.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.850+0000 7fbef3400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:40:01.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.850+0000 7fbef3400640 1 -- 172.21.15.31:0/367338455 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf040a44e0 con 0x55e05bac08b0 2024-03-21T10:40:01.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.850+0000 7fbef3400640 1 -- 172.21.15.31:0/367338455 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fbf0409a010 con 0x55e05bac08b0 2024-03-21T10:40:01.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.850+0000 7fbf1b476740 1 -- 172.21.15.31:0/367338455 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbf0400fb30 con 0x55e05bac08b0 2024-03-21T10:40:01.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.851+0000 7fbef3400640 1 -- 172.21.15.31:0/367338455 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf040cd080 con 0x55e05bac08b0 2024-03-21T10:40:01.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.851+0000 7fbf1b476740 10 client.26044 fetch_fsmap learned FSMap version 24 2024-03-21T10:40:01.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.851+0000 7fbf1b476740 1 -- 172.21.15.31:0/367338455 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55e05b8027d0 con 0x55e05bac08b0 2024-03-21T10:40:01.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.851+0000 7fbef3400640 1 -- 172.21.15.31:0/367338455 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf0400fb30 con 0x55e05bac08b0 2024-03-21T10:40:01.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.851+0000 7fbef3400640 1 -- 172.21.15.31:0/367338455 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fbf0409acf0 con 0x55e05bac08b0 2024-03-21T10:40:01.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.852+0000 7fbf1b476740 10 client.26044 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:40:01.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.852+0000 7fbf1b476740 10 client.26044 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:40:01.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.852+0000 7fbf1b476740 20 client.26044 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:01.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.852+0000 7fbf1b476740 1 --2- 172.21.15.31:0/367338455 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55e05bad9b60 0x55e05badbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.852+0000 7fbf1b476740 4 client.26044 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:40:01.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.852+0000 7fbf1b476740 1 -- 172.21.15.31:0/367338455 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e05ba31800 con 0x55e05bad9b60 2024-03-21T10:40:01.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.852+0000 7fbf15400640 1 --2- 172.21.15.31:0/367338455 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55e05bad9b60 0x55e05badbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.852+0000 7fbf15400640 1 --2- 172.21.15.31:0/367338455 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55e05bad9b60 0x55e05badbf50 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12625ca6a5d85023 server_cookie=86b4fa0e0c47c7c4 in_seq=0 out_seq=0 2024-03-21T10:40:01.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.852+0000 7fbef3400640 10 client.26044 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:01.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.853+0000 7fbef3400640 1 -- 172.21.15.31:0/367338455 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e05ba31800 con 0x55e05bad9b60 2024-03-21T10:40:01.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.853+0000 7fbef3400640 10 client.26044 handle_command_reply: tid=0 2024-03-21T10:40:01.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.853+0000 7fbf1b476740 1 client.26044 shutdown 2024-03-21T10:40:01.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.853+0000 7fbf1b476740 1 -- 172.21.15.31:0/367338455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05bac08b0 msgr2=0x55e05bac9720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.853+0000 7fbf1b476740 1 --2- 172.21.15.31:0/367338455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05bac08b0 0x55e05bac9720 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbf04000c00 tx=0x7fbf04018f50 comp rx=0 tx=0).stop 2024-03-21T10:40:01.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.854+0000 7fbf1b476740 1 -- 172.21.15.31:0/367338455 shutdown_connections 2024-03-21T10:40:01.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.854+0000 7fbf1b476740 1 --2- 172.21.15.31:0/367338455 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55e05bad9b60 0x55e05badbf50 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.854+0000 7fbf1b476740 1 --2- 172.21.15.31:0/367338455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e05bac9de0 0x55e05bac70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.854+0000 7fbf1b476740 1 --2- 172.21.15.31:0/367338455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e05bac1260 0x55e05bac6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.854+0000 7fbf1b476740 1 --2- 172.21.15.31:0/367338455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05bac08b0 0x55e05bac9720 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.854+0000 7fbf1b476740 1 -- 172.21.15.31:0/367338455 >> 172.21.15.31:0/367338455 conn(0x55e05babaf80 msgr2=0x55e05babccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:01.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.854+0000 7fbef3400640 0 client.26044 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:01.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.854+0000 7fbf1b476740 1 -- 172.21.15.31:0/367338455 shutdown_connections 2024-03-21T10:40:01.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.854+0000 7fbf1b476740 1 -- 172.21.15.31:0/367338455 wait complete. 2024-03-21T10:40:01.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.854+0000 7fbf1b476740 20 client.26044 trim_cache size 0 max 16384 2024-03-21T10:40:01.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.868+0000 7fbf1b476740 1 Processor -- start 2024-03-21T10:40:01.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.868+0000 7fbf1b476740 1 -- start start 2024-03-21T10:40:01.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.868+0000 7fbf1b476740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e05bac08b0 0x55e05bac6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.868+0000 7fbf1b476740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05bac1260 0x55e05bac91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.868+0000 7fbf1b476740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e05bac9de0 0x55e05bac9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.868+0000 7fbf1b476740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf101b7280 con 0x55e05bac9de0 2024-03-21T10:40:01.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.868+0000 7fbf1b476740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf101b4d80 con 0x55e05bac08b0 2024-03-21T10:40:01.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.868+0000 7fbf1b476740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf101b2e20 con 0x55e05bac1260 2024-03-21T10:40:01.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.868+0000 7fbf15400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05bac1260 0x55e05bac91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.868+0000 7fbf16800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e05bac9de0 0x55e05bac9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.869+0000 7fbf16800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e05bac9de0 0x55e05bac9720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36372/0 (socket says 172.21.15.31:36372) 2024-03-21T10:40:01.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.869+0000 7fbf16800640 1 -- 172.21.15.31:0/2290132840 learned_addr learned my addr 172.21.15.31:0/2290132840 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:01.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.869+0000 7fbf15e00640 1 --2- 172.21.15.31:0/2290132840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e05bac08b0 0x55e05bac6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.869+0000 7fbf15400640 1 -- 172.21.15.31:0/2290132840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e05bac08b0 msgr2=0x55e05bac6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:01.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.869+0000 7fbf15400640 1 --2- 172.21.15.31:0/2290132840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e05bac08b0 0x55e05bac6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.869+0000 7fbf15400640 1 -- 172.21.15.31:0/2290132840 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e05bac9de0 msgr2=0x55e05bac9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:01.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.869+0000 7fbf15400640 1 --2- 172.21.15.31:0/2290132840 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e05bac9de0 0x55e05bac9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.869+0000 7fbf15400640 1 -- 172.21.15.31:0/2290132840 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e05bacced0 con 0x55e05bac1260 2024-03-21T10:40:01.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.869+0000 7fbf15400640 1 --2- 172.21.15.31:0/2290132840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05bac1260 0x55e05bac91e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fbf00036040 tx=0x7fbf00005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ab7c85a04a6b78b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:01.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.870+0000 7fbefe600640 1 -- 172.21.15.31:0/2290132840 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf000041a0 con 0x55e05bac1260 2024-03-21T10:40:01.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.870+0000 7fbefe600640 1 -- 172.21.15.31:0/2290132840 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf00004340 con 0x55e05bac1260 2024-03-21T10:40:01.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.870+0000 7fbefe600640 1 -- 172.21.15.31:0/2290132840 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf00004ee0 con 0x55e05bac1260 2024-03-21T10:40:01.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.871+0000 7fbf1b476740 1 -- 172.21.15.31:0/2290132840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05bac1260 msgr2=0x55e05bac91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.871+0000 7fbf1b476740 1 --2- 172.21.15.31:0/2290132840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05bac1260 0x55e05bac91e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fbf00036040 tx=0x7fbf00005c00 comp rx=0 tx=0).stop 2024-03-21T10:40:01.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.871+0000 7fbf1b476740 1 -- 172.21.15.31:0/2290132840 shutdown_connections 2024-03-21T10:40:01.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.871+0000 7fbf1b476740 1 --2- 172.21.15.31:0/2290132840 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e05bac9de0 0x55e05bac9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.871+0000 7fbf1b476740 1 --2- 172.21.15.31:0/2290132840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05bac1260 0x55e05bac91e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.871+0000 7fbf1b476740 1 --2- 172.21.15.31:0/2290132840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e05bac08b0 0x55e05bac6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.871+0000 7fbf1b476740 1 -- 172.21.15.31:0/2290132840 >> 172.21.15.31:0/2290132840 conn(0x55e05babaf80 msgr2=0x55e05baba160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:01.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.871+0000 7fbf1b476740 1 -- 172.21.15.31:0/2290132840 shutdown_connections 2024-03-21T10:40:01.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.871+0000 7fbf1b476740 1 -- 172.21.15.31:0/2290132840 wait complete. 2024-03-21T10:40:01.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.871+0000 7fbf1b476740 1 -- start start 2024-03-21T10:40:01.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.872+0000 7fbf1b476740 1 Processor -- start 2024-03-21T10:40:01.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.872+0000 7fbf1b476740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e05bac08b0 0x55e05bade900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.872+0000 7fbf1b476740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e05bac1260 0x55e05badee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.872+0000 7fbf1b476740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05bac9de0 0x55e05badcde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.872+0000 7fbf1b476740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf101b3420 con 0x55e05bac1260 2024-03-21T10:40:01.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.872+0000 7fbf1b476740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf101b4720 con 0x55e05bac08b0 2024-03-21T10:40:01.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.872+0000 7fbf1b476740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf101b3a60 con 0x55e05bac9de0 2024-03-21T10:40:01.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.872+0000 7fbf1b476740 10 client.0 _ll_register_callbacks cb 0x7fbf0c0146f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:01.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.872+0000 7fbf15e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e05bac08b0 0x55e05bade900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.872+0000 7fbf15e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e05bac08b0 0x55e05bade900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42034/0 (socket says 172.21.15.31:42034) 2024-03-21T10:40:01.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.872+0000 7fbf15e00640 1 -- 172.21.15.31:0/318663445 learned_addr learned my addr 172.21.15.31:0/318663445 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:01.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.872+0000 7fbf16800640 1 --2- 172.21.15.31:0/318663445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05bac9de0 0x55e05badcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.873+0000 7fbf15e00640 1 -- 172.21.15.31:0/318663445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05bac9de0 msgr2=0x55e05badcde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:01.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.873+0000 7fbf15e00640 1 --2- 172.21.15.31:0/318663445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05bac9de0 0x55e05badcde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.873+0000 7fbf15e00640 1 -- 172.21.15.31:0/318663445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e05bac1260 msgr2=0x55e05badee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:01.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.873+0000 7fbf15e00640 1 --2- 172.21.15.31:0/318663445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e05bac1260 0x55e05badee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.873+0000 7fbf15e00640 1 -- 172.21.15.31:0/318663445 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e05bacc690 con 0x55e05bac08b0 2024-03-21T10:40:01.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.873+0000 7fbf15e00640 1 --2- 172.21.15.31:0/318663445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e05bac08b0 0x55e05bade900 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fbf0409bc50 tx=0x7fbf04010000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=122315d1b24ed750 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:01.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.873+0000 7fbef3400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:40:01.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.873+0000 7fbef3400640 1 -- 172.21.15.31:0/318663445 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0409c410 con 0x55e05bac08b0 2024-03-21T10:40:01.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.874+0000 7fbef3400640 1 -- 172.21.15.31:0/318663445 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fbf0409a010 con 0x55e05bac08b0 2024-03-21T10:40:01.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.874+0000 7fbf1b476740 1 -- 172.21.15.31:0/318663445 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbf00004340 con 0x55e05bac08b0 2024-03-21T10:40:01.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.874+0000 7fbef3400640 1 -- 172.21.15.31:0/318663445 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf040cd080 con 0x55e05bac08b0 2024-03-21T10:40:01.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.874+0000 7fbf1b476740 10 client.26066 fetch_fsmap learned FSMap version 24 2024-03-21T10:40:01.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.874+0000 7fbf1b476740 1 -- 172.21.15.31:0/318663445 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55e05bacc690 con 0x55e05bac08b0 2024-03-21T10:40:01.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.874+0000 7fbef3400640 1 -- 172.21.15.31:0/318663445 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf00004340 con 0x55e05bac08b0 2024-03-21T10:40:01.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.874+0000 7fbef3400640 1 -- 172.21.15.31:0/318663445 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fbf0409b160 con 0x55e05bac08b0 2024-03-21T10:40:01.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.874+0000 7fbf1b476740 10 client.26066 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:40:01.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.874+0000 7fbf1b476740 10 client.26066 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:40:01.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.874+0000 7fbf1b476740 20 client.26066 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:01.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.874+0000 7fbf1b476740 1 --2- 172.21.15.31:0/318663445 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55e05bb35d90 0x55e05bb38180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:01.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.874+0000 7fbf1b476740 4 client.26066 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:40:01.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.874+0000 7fbf1b476740 1 -- 172.21.15.31:0/318663445 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e05ba2bb50 con 0x55e05bb35d90 2024-03-21T10:40:01.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.875+0000 7fbf15400640 1 --2- 172.21.15.31:0/318663445 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55e05bb35d90 0x55e05bb38180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:01.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.875+0000 7fbf15400640 1 --2- 172.21.15.31:0/318663445 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55e05bb35d90 0x55e05bb38180 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85d1116e2a5ca89a server_cookie=3fb74306b246b9ae in_seq=0 out_seq=0 2024-03-21T10:40:01.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.875+0000 7fbef3400640 10 client.26066 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:01.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.875+0000 7fbef3400640 1 -- 172.21.15.31:0/318663445 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e05ba2bb50 con 0x55e05bb35d90 2024-03-21T10:40:01.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.875+0000 7fbef3400640 10 client.26066 handle_command_reply: tid=0 2024-03-21T10:40:01.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.875+0000 7fbf1b476740 1 client.26066 shutdown 2024-03-21T10:40:01.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.875+0000 7fbf1b476740 1 -- 172.21.15.31:0/318663445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e05bac08b0 msgr2=0x55e05bade900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.875+0000 7fbf1b476740 1 --2- 172.21.15.31:0/318663445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e05bac08b0 0x55e05bade900 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fbf0409bc50 tx=0x7fbf04010000 comp rx=0 tx=0).stop 2024-03-21T10:40:01.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.876+0000 7fbf1b476740 1 -- 172.21.15.31:0/318663445 shutdown_connections 2024-03-21T10:40:01.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.876+0000 7fbf1b476740 1 --2- 172.21.15.31:0/318663445 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55e05bb35d90 0x55e05bb38180 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.876+0000 7fbef3400640 0 client.26066 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:01.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.876+0000 7fbf1b476740 1 --2- 172.21.15.31:0/318663445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05bac9de0 0x55e05badcde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.876+0000 7fbf1b476740 1 --2- 172.21.15.31:0/318663445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e05bac1260 0x55e05badee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.876+0000 7fbf1b476740 1 --2- 172.21.15.31:0/318663445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e05bac08b0 0x55e05bade900 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.876+0000 7fbf1b476740 1 -- 172.21.15.31:0/318663445 >> 172.21.15.31:0/318663445 conn(0x55e05babaf80 msgr2=0x55e05bac1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:01.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.876+0000 7fbf1b476740 1 -- 172.21.15.31:0/318663445 shutdown_connections 2024-03-21T10:40:01.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.876+0000 7fbf1b476740 1 -- 172.21.15.31:0/318663445 wait complete. 2024-03-21T10:40:01.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.876+0000 7fbf1b476740 20 client.26066 trim_cache size 0 max 16384 2024-03-21T10:40:01.877 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:40:01.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.877+0000 7fbf17200640 1 -- 172.21.15.31:0/730977098 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbef80a3230 msgr2=0x7fbef80a56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.877+0000 7fbf17200640 1 --2- 172.21.15.31:0/730977098 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbef80a3230 0x7fbef80a56f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbf1006a9d0 tx=0x7fbf0c009d60 comp rx=0 tx=0).stop 2024-03-21T10:40:01.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.877+0000 7fbf17200640 1 -- 172.21.15.31:0/730977098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf10075bd0 msgr2=0x7fbf1019e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:01.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.877+0000 7fbf17200640 1 --2- 172.21.15.31:0/730977098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf10075bd0 0x7fbf1019e370 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbf0400c920 tx=0x7fbf0400cde0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.877+0000 7fbf17200640 1 -- 172.21.15.31:0/730977098 shutdown_connections 2024-03-21T10:40:01.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.877+0000 7fbf17200640 1 --2- 172.21.15.31:0/730977098 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbef80a3230 0x7fbef80a56f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.877+0000 7fbf17200640 1 --2- 172.21.15.31:0/730977098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf10112300 0x7fbf101a2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.877+0000 7fbf17200640 1 --2- 172.21.15.31:0/730977098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf100764f0 0x7fbf1019e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.877+0000 7fbf17200640 1 --2- 172.21.15.31:0/730977098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf10075bd0 0x7fbf1019e370 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:01.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.877+0000 7fbf17200640 1 -- 172.21.15.31:0/730977098 >> 172.21.15.31:0/730977098 conn(0x7fbf100ff110 msgr2=0x7fbf10113940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:01.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.877+0000 7fbf17200640 1 -- 172.21.15.31:0/730977098 shutdown_connections 2024-03-21T10:40:01.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:01.877+0000 7fbf17200640 1 -- 172.21.15.31:0/730977098 wait complete. 2024-03-21T10:40:01.897 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:01 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/4251981843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:40:02.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:01 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/4251981843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:40:02.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:01 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/4251981843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:40:02.900 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-21T10:40:02.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:40:02.900 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-21T10:40:02.975 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:02 smithi031 ceph-mon[31673]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-21T10:40:03.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:02 smithi070 ceph-mon[36330]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-21T10:40:03.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.124+0000 7ff50b200640 1 -- 172.21.15.31:0/3252957842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff50c073a50 msgr2=0x7ff50c073e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:03.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.124+0000 7ff50b200640 1 --2- 172.21.15.31:0/3252957842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff50c073a50 0x7ff50c073e30 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc009900 tx=0x7ff4fc02bcd0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.125+0000 7ff50b200640 1 -- 172.21.15.31:0/3252957842 shutdown_connections 2024-03-21T10:40:03.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.125+0000 7ff50b200640 1 --2- 172.21.15.31:0/3252957842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff50c10dd40 0x7ff50c110130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.125+0000 7ff50b200640 1 --2- 172.21.15.31:0/3252957842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff50c074370 0x7ff50c10d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.125+0000 7ff50b200640 1 --2- 172.21.15.31:0/3252957842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff50c073a50 0x7ff50c073e30 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.125+0000 7ff50b200640 1 -- 172.21.15.31:0/3252957842 >> 172.21.15.31:0/3252957842 conn(0x7ff50c0fcfb0 msgr2=0x7ff50c0ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:03.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.125+0000 7ff50b200640 1 -- 172.21.15.31:0/3252957842 shutdown_connections 2024-03-21T10:40:03.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.125+0000 7ff50b200640 1 -- 172.21.15.31:0/3252957842 wait complete. 2024-03-21T10:40:03.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.126+0000 7ff50b200640 1 Processor -- start 2024-03-21T10:40:03.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.126+0000 7ff50b200640 1 -- start start 2024-03-21T10:40:03.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.126+0000 7ff50b200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff50c073a50 0x7ff50c199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.126+0000 7ff50b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff50c074370 0x7ff50c19a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.126+0000 7ff50b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff50c10dd40 0x7ff50c19e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.126+0000 7ff50b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c113a70 con 0x7ff50c074370 2024-03-21T10:40:03.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.126+0000 7ff50b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c1138f0 con 0x7ff50c10dd40 2024-03-21T10:40:03.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.126+0000 7ff50b200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c113bf0 con 0x7ff50c073a50 2024-03-21T10:40:03.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.127+0000 7ff509e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff50c073a50 0x7ff50c199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.127+0000 7ff509e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff50c073a50 0x7ff50c199f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42608/0 (socket says 172.21.15.31:42608) 2024-03-21T10:40:03.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.127+0000 7ff509e00640 1 -- 172.21.15.31:0/2199998264 learned_addr learned my addr 172.21.15.31:0/2199998264 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:03.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.127+0000 7ff50a800640 1 --2- 172.21.15.31:0/2199998264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff50c10dd40 0x7ff50c19e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.127+0000 7ff509400640 1 --2- 172.21.15.31:0/2199998264 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff50c074370 0x7ff50c19a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.127+0000 7ff50a800640 1 -- 172.21.15.31:0/2199998264 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff50c073a50 msgr2=0x7ff50c199f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:03.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.127+0000 7ff50a800640 1 --2- 172.21.15.31:0/2199998264 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff50c073a50 0x7ff50c199f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.127+0000 7ff50a800640 1 -- 172.21.15.31:0/2199998264 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff50c074370 msgr2=0x7ff50c19a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:03.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.127+0000 7ff50a800640 1 --2- 172.21.15.31:0/2199998264 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff50c074370 0x7ff50c19a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.127+0000 7ff50a800640 1 -- 172.21.15.31:0/2199998264 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff50c19ef60 con 0x7ff50c10dd40 2024-03-21T10:40:03.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.127+0000 7ff50a800640 1 --2- 172.21.15.31:0/2199998264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff50c10dd40 0x7ff50c19e830 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff504009fd0 tx=0x7ff50400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:03.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.128+0000 7ff502a00640 1 -- 172.21.15.31:0/2199998264 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50400ce20 con 0x7ff50c10dd40 2024-03-21T10:40:03.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.128+0000 7ff502a00640 1 -- 172.21.15.31:0/2199998264 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff504005020 con 0x7ff50c10dd40 2024-03-21T10:40:03.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.128+0000 7ff502a00640 1 -- 172.21.15.31:0/2199998264 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504010720 con 0x7ff50c10dd40 2024-03-21T10:40:03.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.128+0000 7ff50b200640 1 -- 172.21.15.31:0/2199998264 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff50c19f250 con 0x7ff50c10dd40 2024-03-21T10:40:03.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.128+0000 7ff50b200640 1 -- 172.21.15.31:0/2199998264 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff50c101390 con 0x7ff50c10dd40 2024-03-21T10:40:03.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.132+0000 7ff5137d8740 1 Processor -- start 2024-03-21T10:40:03.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.132+0000 7ff502a00640 1 -- 172.21.15.31:0/2199998264 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) v1 ==== 138087+0+0 (secure 0 0 0) 0x7ff504005540 con 0x7ff50c10dd40 2024-03-21T10:40:03.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.133+0000 7ff5137d8740 1 -- start start 2024-03-21T10:40:03.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.133+0000 7ff5137d8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560005547290 0x560005547670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.133+0000 7ff5137d8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560005547bb0 0x560005550250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.133+0000 7ff5137d8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560005550790 0x560005552b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.133+0000 7ff5137d8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600054abeb0 con 0x560005547bb0 2024-03-21T10:40:03.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.133+0000 7ff5137d8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560005454a60 con 0x560005547290 2024-03-21T10:40:03.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.133+0000 7ff5137d8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600054a7f50 con 0x560005550790 2024-03-21T10:40:03.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.134+0000 7ff509400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560005547290 0x560005547670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.134+0000 7ff509400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560005547290 0x560005547670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42064/0 (socket says 172.21.15.31:42064) 2024-03-21T10:40:03.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.134+0000 7ff509400640 1 -- 172.21.15.31:0/1485608338 learned_addr learned my addr 172.21.15.31:0/1485608338 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:03.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.134+0000 7ff509e00640 1 --2- 172.21.15.31:0/1485608338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560005547bb0 0x560005550250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.134+0000 7ff509400640 1 --2- 172.21.15.31:0/1485608338 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560005550790 0x560005552b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.134+0000 7ff509400640 1 -- 172.21.15.31:0/1485608338 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560005550790 msgr2=0x560005552b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:03.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.135+0000 7ff509400640 1 --2- 172.21.15.31:0/1485608338 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560005550790 0x560005552b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.135+0000 7ff502a00640 1 --2- 172.21.15.31:0/2199998264 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff4f00a31b0 0x7ff4f00a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.135+0000 7ff509400640 1 -- 172.21.15.31:0/1485608338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560005547bb0 msgr2=0x560005550250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:03.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.135+0000 7ff509400640 1 --2- 172.21.15.31:0/1485608338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560005547bb0 0x560005550250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.135+0000 7ff509400640 1 -- 172.21.15.31:0/1485608338 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600052887e0 con 0x560005547290 2024-03-21T10:40:03.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.135+0000 7ff509e00640 1 --2- 172.21.15.31:0/1485608338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560005547bb0 0x560005550250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:40:03.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.135+0000 7ff509400640 1 --2- 172.21.15.31:0/1485608338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560005547290 0x560005547670 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8009fd0 tx=0x7ff4f8005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3f99d204191d2b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:03.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.135+0000 7ff502a00640 1 -- 172.21.15.31:0/2199998264 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7ff5040d04c0 con 0x7ff50c10dd40 2024-03-21T10:40:03.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.135+0000 7ff50a800640 1 --2- 172.21.15.31:0/2199998264 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff4f00a31b0 0x7ff4f00a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.135+0000 7ff4f7000640 1 -- 172.21.15.31:0/1485608338 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f8004030 con 0x560005547290 2024-03-21T10:40:03.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.135+0000 7ff4f7000640 1 -- 172.21.15.31:0/1485608338 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4f8005500 con 0x560005547290 2024-03-21T10:40:03.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.136+0000 7ff4f7000640 1 -- 172.21.15.31:0/1485608338 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f8004d10 con 0x560005547290 2024-03-21T10:40:03.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.136+0000 7ff5137d8740 1 -- 172.21.15.31:0/1485608338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560005547290 msgr2=0x560005547670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:03.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.136+0000 7ff50a800640 1 --2- 172.21.15.31:0/2199998264 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff4f00a31b0 0x7ff4f00a5670 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff5040cd750 tx=0x7ff5040054b0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:03.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.136+0000 7ff5137d8740 1 --2- 172.21.15.31:0/1485608338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560005547290 0x560005547670 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8009fd0 tx=0x7ff4f8005850 comp rx=0 tx=0).stop 2024-03-21T10:40:03.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.137+0000 7ff5137d8740 1 -- 172.21.15.31:0/1485608338 shutdown_connections 2024-03-21T10:40:03.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.137+0000 7ff5137d8740 1 --2- 172.21.15.31:0/1485608338 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560005550790 0x560005552b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.137+0000 7ff5137d8740 1 --2- 172.21.15.31:0/1485608338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560005547bb0 0x560005550250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.137+0000 7ff5137d8740 1 --2- 172.21.15.31:0/1485608338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560005547290 0x560005547670 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.137+0000 7ff5137d8740 1 -- 172.21.15.31:0/1485608338 >> 172.21.15.31:0/1485608338 conn(0x5600055419c0 msgr2=0x560005541da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:03.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.137+0000 7ff5137d8740 1 -- 172.21.15.31:0/1485608338 shutdown_connections 2024-03-21T10:40:03.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.138+0000 7ff5137d8740 1 -- 172.21.15.31:0/1485608338 wait complete. 2024-03-21T10:40:03.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.138+0000 7ff5137d8740 1 -- start start 2024-03-21T10:40:03.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.139+0000 7ff5137d8740 1 Processor -- start 2024-03-21T10:40:03.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.139+0000 7ff5137d8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560005547290 0x560005550130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.139+0000 7ff5137d8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560005547bb0 0x56000554d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.139+0000 7ff5137d8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560005550790 0x56000554da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.139+0000 7ff5137d8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560005553580 con 0x560005547290 2024-03-21T10:40:03.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.139+0000 7ff5137d8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560005467eb0 con 0x560005547bb0 2024-03-21T10:40:03.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.139+0000 7ff5137d8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x560005553700 con 0x560005550790 2024-03-21T10:40:03.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.139+0000 7ff5137d8740 10 client.0 _ll_register_callbacks cb 0x56000545c150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:03.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.140+0000 7ff509400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560005547290 0x560005550130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.140+0000 7ff509e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560005547bb0 0x56000554d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.140+0000 7ff509e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560005547bb0 0x56000554d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42080/0 (socket says 172.21.15.31:42080) 2024-03-21T10:40:03.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.140+0000 7ff509e00640 1 -- 172.21.15.31:0/3389715503 learned_addr learned my addr 172.21.15.31:0/3389715503 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:03.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.140+0000 7ff509e00640 1 -- 172.21.15.31:0/3389715503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560005550790 msgr2=0x56000554da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:03.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.140+0000 7ff509e00640 1 --2- 172.21.15.31:0/3389715503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560005550790 0x56000554da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.140+0000 7ff509e00640 1 -- 172.21.15.31:0/3389715503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560005547290 msgr2=0x560005550130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:03.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.140+0000 7ff509e00640 1 --2- 172.21.15.31:0/3389715503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560005547290 0x560005550130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.140+0000 7ff509e00640 1 -- 172.21.15.31:0/3389715503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5600052e8f70 con 0x560005547bb0 2024-03-21T10:40:03.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.140+0000 7ff509400640 1 --2- 172.21.15.31:0/3389715503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560005547290 0x560005550130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:03.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.141+0000 7ff509e00640 1 --2- 172.21.15.31:0/3389715503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560005547bb0 0x56000554d4c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc036040 tx=0x7ff4fc0045f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e799dfceb0072a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:03.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.141+0000 7ff4eb400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:40:03.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.141+0000 7ff4eb400640 1 -- 172.21.15.31:0/3389715503 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc043070 con 0x560005547bb0 2024-03-21T10:40:03.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.141+0000 7ff4eb400640 1 -- 172.21.15.31:0/3389715503 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7ff4fc0046c0 con 0x560005547bb0 2024-03-21T10:40:03.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.141+0000 7ff5137d8740 1 -- 172.21.15.31:0/3389715503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff4f8005500 con 0x560005547bb0 2024-03-21T10:40:03.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.141+0000 7ff4eb400640 1 -- 172.21.15.31:0/3389715503 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4fc050050 con 0x560005547bb0 2024-03-21T10:40:03.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.141+0000 7ff5137d8740 10 client.26078 fetch_fsmap learned FSMap version 24 2024-03-21T10:40:03.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.141+0000 7ff5137d8740 1 -- 172.21.15.31:0/3389715503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x560005180170 con 0x560005547bb0 2024-03-21T10:40:03.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.142+0000 7ff4eb400640 1 -- 172.21.15.31:0/3389715503 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4f8005500 con 0x560005547bb0 2024-03-21T10:40:03.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.142+0000 7ff4eb400640 1 -- 172.21.15.31:0/3389715503 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7ff4fc0035f0 con 0x560005547bb0 2024-03-21T10:40:03.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.142+0000 7ff5137d8740 10 client.26078 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:40:03.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.142+0000 7ff5137d8740 10 client.26078 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:40:03.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.142+0000 7ff5137d8740 20 client.26078 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:03.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.142+0000 7ff5137d8740 1 --2- 172.21.15.31:0/3389715503 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560005560820 0x560005562c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.142+0000 7ff5137d8740 4 client.26078 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:40:03.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.142+0000 7ff5137d8740 1 -- 172.21.15.31:0/3389715503 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5600054b93f0 con 0x560005560820 2024-03-21T10:40:03.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.143+0000 7ff509400640 1 --2- 172.21.15.31:0/3389715503 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560005560820 0x560005562c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.143+0000 7ff509400640 1 --2- 172.21.15.31:0/3389715503 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560005560820 0x560005562c10 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9f5e71196569217 server_cookie=7e3e1216c134c963 in_seq=0 out_seq=0 2024-03-21T10:40:03.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.143+0000 7ff4eb400640 10 client.26078 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:03.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.145+0000 7ff4eb400640 1 -- 172.21.15.31:0/3389715503 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5600054b93f0 con 0x560005560820 2024-03-21T10:40:03.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.145+0000 7ff4eb400640 10 client.26078 handle_command_reply: tid=0 2024-03-21T10:40:03.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.145+0000 7ff5137d8740 1 client.26078 shutdown 2024-03-21T10:40:03.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.145+0000 7ff5137d8740 1 -- 172.21.15.31:0/3389715503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560005547bb0 msgr2=0x56000554d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:03.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.145+0000 7ff5137d8740 1 --2- 172.21.15.31:0/3389715503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560005547bb0 0x56000554d4c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc036040 tx=0x7ff4fc0045f0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.145+0000 7ff5137d8740 1 -- 172.21.15.31:0/3389715503 shutdown_connections 2024-03-21T10:40:03.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.145+0000 7ff5137d8740 1 --2- 172.21.15.31:0/3389715503 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560005560820 0x560005562c10 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.145+0000 7ff5137d8740 1 --2- 172.21.15.31:0/3389715503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560005550790 0x56000554da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.145+0000 7ff5137d8740 1 --2- 172.21.15.31:0/3389715503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560005547bb0 0x56000554d4c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.145+0000 7ff5137d8740 1 --2- 172.21.15.31:0/3389715503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560005547290 0x560005550130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.145+0000 7ff5137d8740 1 -- 172.21.15.31:0/3389715503 >> 172.21.15.31:0/3389715503 conn(0x5600055419c0 msgr2=0x560005543710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:03.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.145+0000 7ff4eb400640 0 client.26078 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:03.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.145+0000 7ff5137d8740 1 -- 172.21.15.31:0/3389715503 shutdown_connections 2024-03-21T10:40:03.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.145+0000 7ff5137d8740 1 -- 172.21.15.31:0/3389715503 wait complete. 2024-03-21T10:40:03.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.146+0000 7ff5137d8740 20 client.26078 trim_cache size 0 max 16384 2024-03-21T10:40:03.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.170+0000 7ff5137d8740 1 Processor -- start 2024-03-21T10:40:03.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.171+0000 7ff5137d8740 1 -- start start 2024-03-21T10:40:03.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.171+0000 7ff5137d8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560005547290 0x56000554fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.171+0000 7ff5137d8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560005547bb0 0x560005550130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.171+0000 7ff5137d8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560005550790 0x56000555be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.171+0000 7ff5137d8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c1aeab0 con 0x560005547bb0 2024-03-21T10:40:03.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.171+0000 7ff5137d8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c1ac5b0 con 0x560005550790 2024-03-21T10:40:03.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.171+0000 7ff5137d8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c1aa650 con 0x560005547290 2024-03-21T10:40:03.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.171+0000 7ff509400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560005547290 0x56000554fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.171+0000 7ff509400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560005547290 0x56000554fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42650/0 (socket says 172.21.15.31:42650) 2024-03-21T10:40:03.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.171+0000 7ff509400640 1 -- 172.21.15.31:0/1035421220 learned_addr learned my addr 172.21.15.31:0/1035421220 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:03.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.171+0000 7ff509400640 1 --2- 172.21.15.31:0/1035421220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560005550790 0x56000555be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.172+0000 7ff509e00640 1 --2- 172.21.15.31:0/1035421220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560005547bb0 0x560005550130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.172+0000 7ff509400640 1 -- 172.21.15.31:0/1035421220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560005550790 msgr2=0x56000555be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:03.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.172+0000 7ff509400640 1 --2- 172.21.15.31:0/1035421220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560005550790 0x56000555be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.172+0000 7ff509400640 1 -- 172.21.15.31:0/1035421220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560005547bb0 msgr2=0x560005550130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:03.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.172+0000 7ff509400640 1 --2- 172.21.15.31:0/1035421220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560005547bb0 0x560005550130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.172+0000 7ff509400640 1 -- 172.21.15.31:0/1035421220 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56000554c470 con 0x560005547290 2024-03-21T10:40:03.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.172+0000 7ff509400640 1 --2- 172.21.15.31:0/1035421220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560005547290 0x56000554fbf0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8009740 tx=0x7ff4f8004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f69d0691a3c4ad0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:03.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.172+0000 7ff4f6600640 1 -- 172.21.15.31:0/1035421220 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f8029040 con 0x560005547290 2024-03-21T10:40:03.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.173+0000 7ff4f6600640 1 -- 172.21.15.31:0/1035421220 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4f8034d10 con 0x560005547290 2024-03-21T10:40:03.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.173+0000 7ff5137d8740 1 -- 172.21.15.31:0/1035421220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560005547290 msgr2=0x56000554fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:03.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.173+0000 7ff5137d8740 1 --2- 172.21.15.31:0/1035421220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560005547290 0x56000554fbf0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8009740 tx=0x7ff4f8004950 comp rx=0 tx=0).stop 2024-03-21T10:40:03.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.174+0000 7ff5137d8740 1 -- 172.21.15.31:0/1035421220 shutdown_connections 2024-03-21T10:40:03.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.174+0000 7ff5137d8740 1 --2- 172.21.15.31:0/1035421220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560005550790 0x56000555be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.174+0000 7ff5137d8740 1 --2- 172.21.15.31:0/1035421220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560005547bb0 0x560005550130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.174+0000 7ff5137d8740 1 --2- 172.21.15.31:0/1035421220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560005547290 0x56000554fbf0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.174+0000 7ff5137d8740 1 -- 172.21.15.31:0/1035421220 >> 172.21.15.31:0/1035421220 conn(0x5600055419c0 msgr2=0x560005543550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:03.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.174+0000 7ff5137d8740 1 -- 172.21.15.31:0/1035421220 shutdown_connections 2024-03-21T10:40:03.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.174+0000 7ff5137d8740 1 -- 172.21.15.31:0/1035421220 wait complete. 2024-03-21T10:40:03.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.174+0000 7ff5137d8740 1 -- start start 2024-03-21T10:40:03.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.175+0000 7ff5137d8740 1 Processor -- start 2024-03-21T10:40:03.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.175+0000 7ff5137d8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560005547290 0x560005558890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.175+0000 7ff5137d8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560005547bb0 0x560005554c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.175+0000 7ff5137d8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560005550790 0x560005555300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.175+0000 7ff5137d8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c1aac50 con 0x560005550790 2024-03-21T10:40:03.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.175+0000 7ff5137d8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c1abf50 con 0x560005547290 2024-03-21T10:40:03.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.175+0000 7ff5137d8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c1ab290 con 0x560005547bb0 2024-03-21T10:40:03.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.175+0000 7ff5137d8740 10 client.0 _ll_register_callbacks cb 0x7ff4fc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:03.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.176+0000 7ff509400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560005547290 0x560005558890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.176+0000 7ff509400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560005547290 0x560005558890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42092/0 (socket says 172.21.15.31:42092) 2024-03-21T10:40:03.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.176+0000 7ff509400640 1 -- 172.21.15.31:0/25860964 learned_addr learned my addr 172.21.15.31:0/25860964 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:03.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.176+0000 7ff509400640 1 --2- 172.21.15.31:0/25860964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560005550790 0x560005555300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.176+0000 7ff509e00640 1 --2- 172.21.15.31:0/25860964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560005547bb0 0x560005554c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.176+0000 7ff509400640 1 -- 172.21.15.31:0/25860964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560005547bb0 msgr2=0x560005554c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:03.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.176+0000 7ff509400640 1 --2- 172.21.15.31:0/25860964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560005547bb0 0x560005554c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.176+0000 7ff509400640 1 -- 172.21.15.31:0/25860964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560005550790 msgr2=0x560005555300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:03.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.176+0000 7ff509400640 1 --2- 172.21.15.31:0/25860964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560005550790 0x560005555300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.176+0000 7ff509400640 1 -- 172.21.15.31:0/25860964 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560005552ef0 con 0x560005547290 2024-03-21T10:40:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.177+0000 7ff509400640 1 --2- 172.21.15.31:0/25860964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560005547290 0x560005558890 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8022370 tx=0x7ff4f8016040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5fda24aa5607506 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.177+0000 7ff4eb400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:40:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.177+0000 7ff4eb400640 1 -- 172.21.15.31:0/25860964 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f8029040 con 0x560005547290 2024-03-21T10:40:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.177+0000 7ff4eb400640 1 -- 172.21.15.31:0/25860964 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7ff4f8014d50 con 0x560005547290 2024-03-21T10:40:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.177+0000 7ff5137d8740 1 -- 172.21.15.31:0/25860964 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff4f8034d10 con 0x560005547290 2024-03-21T10:40:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.178+0000 7ff4eb400640 1 -- 172.21.15.31:0/25860964 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4f8022db0 con 0x560005547290 2024-03-21T10:40:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.178+0000 7ff5137d8740 10 client.26084 fetch_fsmap learned FSMap version 24 2024-03-21T10:40:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.178+0000 7ff5137d8740 1 -- 172.21.15.31:0/25860964 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5600055bd100 con 0x560005547290 2024-03-21T10:40:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.178+0000 7ff4eb400640 1 -- 172.21.15.31:0/25860964 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4f8034d10 con 0x560005547290 2024-03-21T10:40:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.178+0000 7ff4eb400640 1 -- 172.21.15.31:0/25860964 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7ff4f8032450 con 0x560005547290 2024-03-21T10:40:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.178+0000 7ff5137d8740 10 client.26084 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:40:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.178+0000 7ff5137d8740 10 client.26084 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:40:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.178+0000 7ff5137d8740 20 client.26084 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.178+0000 7ff5137d8740 1 --2- 172.21.15.31:0/25860964 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5600055bd520 0x5600055bf910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.178+0000 7ff5137d8740 4 client.26084 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-21T10:40:03.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.178+0000 7ff5137d8740 1 -- 172.21.15.31:0/25860964 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5600054b3740 con 0x5600055bd520 2024-03-21T10:40:03.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.179+0000 7ff509e00640 1 --2- 172.21.15.31:0/25860964 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5600055bd520 0x5600055bf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.179+0000 7ff509e00640 1 --2- 172.21.15.31:0/25860964 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5600055bd520 0x5600055bf910 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8663b9c760e6ddea server_cookie=d664ae61fe59fabb in_seq=0 out_seq=0 2024-03-21T10:40:03.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.179+0000 7ff4eb400640 10 client.26084 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:03.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.180+0000 7ff4eb400640 1 -- 172.21.15.31:0/25860964 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5600054b3740 con 0x5600055bd520 2024-03-21T10:40:03.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.181+0000 7ff4eb400640 10 client.26084 handle_command_reply: tid=0 2024-03-21T10:40:03.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.181+0000 7ff5137d8740 1 client.26084 shutdown 2024-03-21T10:40:03.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.181+0000 7ff5137d8740 1 -- 172.21.15.31:0/25860964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560005547290 msgr2=0x560005558890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:03.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.181+0000 7ff5137d8740 1 --2- 172.21.15.31:0/25860964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560005547290 0x560005558890 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8022370 tx=0x7ff4f8016040 comp rx=0 tx=0).stop 2024-03-21T10:40:03.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.182+0000 7ff5137d8740 1 -- 172.21.15.31:0/25860964 shutdown_connections 2024-03-21T10:40:03.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.182+0000 7ff5137d8740 1 --2- 172.21.15.31:0/25860964 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5600055bd520 0x5600055bf910 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.182+0000 7ff5137d8740 1 --2- 172.21.15.31:0/25860964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560005550790 0x560005555300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.182+0000 7ff5137d8740 1 --2- 172.21.15.31:0/25860964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560005547bb0 0x560005554c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.182+0000 7ff5137d8740 1 --2- 172.21.15.31:0/25860964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560005547290 0x560005558890 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.182+0000 7ff4eb400640 0 client.26084 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:03.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.182+0000 7ff5137d8740 1 -- 172.21.15.31:0/25860964 >> 172.21.15.31:0/25860964 conn(0x5600055419c0 msgr2=0x560005542bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:03.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.182+0000 7ff5137d8740 1 -- 172.21.15.31:0/25860964 shutdown_connections 2024-03-21T10:40:03.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.182+0000 7ff5137d8740 1 -- 172.21.15.31:0/25860964 wait complete. 2024-03-21T10:40:03.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.182+0000 7ff5137d8740 20 client.26084 trim_cache size 0 max 16384 2024-03-21T10:40:03.181 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:40:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-03-21T10:40:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "66ee601a-875d-4d41-8aea-65c7a3b8fd19", 2024-03-21T10:40:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-03-21T10:40:03.181 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:40:03.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.183+0000 7ff50b200640 1 -- 172.21.15.31:0/2199998264 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff4f00a31b0 msgr2=0x7ff4f00a5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:03.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.183+0000 7ff50b200640 1 --2- 172.21.15.31:0/2199998264 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff4f00a31b0 0x7ff4f00a5670 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff5040cd750 tx=0x7ff5040054b0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.183+0000 7ff50b200640 1 -- 172.21.15.31:0/2199998264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff50c10dd40 msgr2=0x7ff50c19e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:03.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.183+0000 7ff50b200640 1 --2- 172.21.15.31:0/2199998264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff50c10dd40 0x7ff50c19e830 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff504009fd0 tx=0x7ff50400ef10 comp rx=0 tx=0).stop 2024-03-21T10:40:03.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.184+0000 7ff50b200640 1 -- 172.21.15.31:0/2199998264 shutdown_connections 2024-03-21T10:40:03.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.184+0000 7ff50b200640 1 --2- 172.21.15.31:0/2199998264 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff4f00a31b0 0x7ff4f00a5670 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.184+0000 7ff50b200640 1 --2- 172.21.15.31:0/2199998264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff50c10dd40 0x7ff50c19e830 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.184+0000 7ff50b200640 1 --2- 172.21.15.31:0/2199998264 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff50c074370 0x7ff50c19a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.184+0000 7ff50b200640 1 --2- 172.21.15.31:0/2199998264 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff50c073a50 0x7ff50c199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.184+0000 7ff50b200640 1 -- 172.21.15.31:0/2199998264 >> 172.21.15.31:0/2199998264 conn(0x7ff50c0fcfb0 msgr2=0x7ff50c0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:03.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.184+0000 7ff50b200640 1 -- 172.21.15.31:0/2199998264 shutdown_connections 2024-03-21T10:40:03.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.184+0000 7ff50b200640 1 -- 172.21.15.31:0/2199998264 wait complete. 2024-03-21T10:40:03.183 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:02 smithi040 ceph-mon[36154]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-21T10:40:03.207 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:40:03.207 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:40:03.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.501+0000 7fe54ce00640 1 -- 172.21.15.31:0/4056505114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe548102f50 msgr2=0x7fe54810b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:03.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.501+0000 7fe54ce00640 1 --2- 172.21.15.31:0/4056505114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe548102f50 0x7fe54810b430 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe538009900 tx=0x7fe53802bcd0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.502+0000 7fe54ce00640 1 -- 172.21.15.31:0/4056505114 shutdown_connections 2024-03-21T10:40:03.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.502+0000 7fe54ce00640 1 --2- 172.21.15.31:0/4056505114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe54810ba50 0x7fe54810ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.502+0000 7fe54ce00640 1 --2- 172.21.15.31:0/4056505114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe548102f50 0x7fe54810b430 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.502+0000 7fe54ce00640 1 --2- 172.21.15.31:0/4056505114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe548102630 0x7fe548102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.502+0000 7fe54ce00640 1 -- 172.21.15.31:0/4056505114 >> 172.21.15.31:0/4056505114 conn(0x7fe548077130 msgr2=0x7fe5480fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:03.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.502+0000 7fe54ce00640 1 -- 172.21.15.31:0/4056505114 shutdown_connections 2024-03-21T10:40:03.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.502+0000 7fe54ce00640 1 -- 172.21.15.31:0/4056505114 wait complete. 2024-03-21T10:40:03.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.503+0000 7fe54ce00640 1 Processor -- start 2024-03-21T10:40:03.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.504+0000 7fe54ce00640 1 -- start start 2024-03-21T10:40:03.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.504+0000 7fe54ce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe548102630 0x7fe54819c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.504+0000 7fe54ce00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe548102f50 0x7fe548195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.504+0000 7fe54ce00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe54810ba50 0x7fe548195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.504+0000 7fe54ce00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54810f690 con 0x7fe548102630 2024-03-21T10:40:03.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.504+0000 7fe54ce00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54810f510 con 0x7fe54810ba50 2024-03-21T10:40:03.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.504+0000 7fe54ce00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54810f810 con 0x7fe548102f50 2024-03-21T10:40:03.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.504+0000 7fe547400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe548102630 0x7fe54819c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.505+0000 7fe547400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe548102630 0x7fe54819c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36448/0 (socket says 172.21.15.31:36448) 2024-03-21T10:40:03.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.505+0000 7fe547400640 1 -- 172.21.15.31:0/677939463 learned_addr learned my addr 172.21.15.31:0/677939463 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:03.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.505+0000 7fe547e00640 1 --2- 172.21.15.31:0/677939463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe54810ba50 0x7fe548195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.505+0000 7fe547e00640 1 -- 172.21.15.31:0/677939463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe548102f50 msgr2=0x7fe548195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:03.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.505+0000 7fe53fe00640 1 --2- 172.21.15.31:0/677939463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe548102f50 0x7fe548195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.505+0000 7fe547e00640 1 --2- 172.21.15.31:0/677939463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe548102f50 0x7fe548195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.505+0000 7fe547e00640 1 -- 172.21.15.31:0/677939463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe548102630 msgr2=0x7fe54819c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:03.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.506+0000 7fe547e00640 1 --2- 172.21.15.31:0/677939463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe548102630 0x7fe54819c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.506+0000 7fe547e00640 1 -- 172.21.15.31:0/677939463 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe548196460 con 0x7fe54810ba50 2024-03-21T10:40:03.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.506+0000 7fe547400640 1 --2- 172.21.15.31:0/677939463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe548102630 0x7fe54819c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:40:03.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.506+0000 7fe53fe00640 1 --2- 172.21.15.31:0/677939463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe548102f50 0x7fe548195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:40:03.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.506+0000 7fe547e00640 1 --2- 172.21.15.31:0/677939463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe54810ba50 0x7fe548195ed0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe54000da10 tx=0x7fe54000ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:03.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.506+0000 7fe544c00640 1 -- 172.21.15.31:0/677939463 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54000bd70 con 0x7fe54810ba50 2024-03-21T10:40:03.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.506+0000 7fe544c00640 1 -- 172.21.15.31:0/677939463 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe540003e50 con 0x7fe54810ba50 2024-03-21T10:40:03.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.506+0000 7fe544c00640 1 -- 172.21.15.31:0/677939463 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5400106a0 con 0x7fe54810ba50 2024-03-21T10:40:03.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.506+0000 7fe54ce00640 1 -- 172.21.15.31:0/677939463 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe548196750 con 0x7fe54810ba50 2024-03-21T10:40:03.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.507+0000 7fe54ce00640 1 -- 172.21.15.31:0/677939463 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54819a6b0 con 0x7fe54810ba50 2024-03-21T10:40:03.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.511+0000 7fe55117a740 1 Processor -- start 2024-03-21T10:40:03.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.512+0000 7fe544c00640 1 -- 172.21.15.31:0/677939463 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) v1 ==== 138087+0+0 (secure 0 0 0) 0x7fe540003970 con 0x7fe54810ba50 2024-03-21T10:40:03.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.512+0000 7fe55117a740 1 -- start start 2024-03-21T10:40:03.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.512+0000 7fe55117a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648670d7dd0 0x5648670d6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.513+0000 7fe55117a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648670d70f0 0x5648670d7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.513+0000 7fe55117a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648670e0460 0x5648670e2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.513+0000 7fe55117a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56486703b250 con 0x5648670e0460 2024-03-21T10:40:03.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.513+0000 7fe55117a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564866fe3a30 con 0x5648670d7dd0 2024-03-21T10:40:03.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.513+0000 7fe55117a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648670372f0 con 0x5648670d70f0 2024-03-21T10:40:03.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.513+0000 7fe53fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648670d7dd0 0x5648670d6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.513+0000 7fe53fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648670d7dd0 0x5648670d6b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42110/0 (socket says 172.21.15.31:42110) 2024-03-21T10:40:03.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.513+0000 7fe53fe00640 1 -- 172.21.15.31:0/3439609467 learned_addr learned my addr 172.21.15.31:0/3439609467 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:03.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.513+0000 7fe53fe00640 1 -- 172.21.15.31:0/3439609467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648670d70f0 msgr2=0x5648670d7570 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:40:03.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.513+0000 7fe53fe00640 1 --2- 172.21.15.31:0/3439609467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648670d70f0 0x5648670d7570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.513+0000 7fe53fe00640 1 -- 172.21.15.31:0/3439609467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648670e0460 msgr2=0x5648670e2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:03.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.513+0000 7fe53fe00640 1 --2- 172.21.15.31:0/3439609467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648670e0460 0x5648670e2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.513+0000 7fe53fe00640 1 -- 172.21.15.31:0/3439609467 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564866fce690 con 0x5648670d7dd0 2024-03-21T10:40:03.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.514+0000 7fe53fe00640 1 --2- 172.21.15.31:0/3439609467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648670d7dd0 0x5648670d6b20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe538035000 tx=0x7fe538005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17c8d5a6a5629153 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:03.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.514+0000 7fe53cc00640 1 -- 172.21.15.31:0/3439609467 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe538044070 con 0x5648670d7dd0 2024-03-21T10:40:03.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.514+0000 7fe53cc00640 1 -- 172.21.15.31:0/3439609467 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe53802c6f0 con 0x5648670d7dd0 2024-03-21T10:40:03.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.514+0000 7fe53cc00640 1 -- 172.21.15.31:0/3439609467 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe538005450 con 0x5648670d7dd0 2024-03-21T10:40:03.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.514+0000 7fe544c00640 1 --2- 172.21.15.31:0/677939463 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe52c0a3230 0x7fe52c0a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.515+0000 7fe544c00640 1 -- 172.21.15.31:0/677939463 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fe5400d01c0 con 0x7fe54810ba50 2024-03-21T10:40:03.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.515+0000 7fe547400640 1 --2- 172.21.15.31:0/677939463 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe52c0a3230 0x7fe52c0a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.516+0000 7fe55117a740 1 -- 172.21.15.31:0/3439609467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648670d7dd0 msgr2=0x5648670d6b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:03.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.516+0000 7fe55117a740 1 --2- 172.21.15.31:0/3439609467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648670d7dd0 0x5648670d6b20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe538035000 tx=0x7fe538005d50 comp rx=0 tx=0).stop 2024-03-21T10:40:03.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.516+0000 7fe55117a740 1 -- 172.21.15.31:0/3439609467 shutdown_connections 2024-03-21T10:40:03.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.516+0000 7fe55117a740 1 --2- 172.21.15.31:0/3439609467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648670e0460 0x5648670e2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.516+0000 7fe55117a740 1 --2- 172.21.15.31:0/3439609467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648670d70f0 0x5648670d7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.516+0000 7fe55117a740 1 --2- 172.21.15.31:0/3439609467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648670d7dd0 0x5648670d6b20 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.516+0000 7fe55117a740 1 -- 172.21.15.31:0/3439609467 >> 172.21.15.31:0/3439609467 conn(0x56486709e200 msgr2=0x5648670d1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:03.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.517+0000 7fe547400640 1 --2- 172.21.15.31:0/677939463 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe52c0a3230 0x7fe52c0a56f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe53000f640 tx=0x7fe530011040 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:03.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.517+0000 7fe55117a740 1 -- 172.21.15.31:0/3439609467 shutdown_connections 2024-03-21T10:40:03.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.517+0000 7fe55117a740 1 -- 172.21.15.31:0/3439609467 wait complete. 2024-03-21T10:40:03.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.517+0000 7fe55117a740 1 -- start start 2024-03-21T10:40:03.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.518+0000 7fe55117a740 1 Processor -- start 2024-03-21T10:40:03.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.519+0000 7fe55117a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648670d70f0 0x5648670dfc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.519+0000 7fe55117a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648670d7dd0 0x5648670dd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.519+0000 7fe55117a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648670e0460 0x5648670dd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.519+0000 7fe55117a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648670e3270 con 0x5648670d70f0 2024-03-21T10:40:03.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.519+0000 7fe55117a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564866ff6e80 con 0x5648670d7dd0 2024-03-21T10:40:03.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.519+0000 7fe55117a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648670e33f0 con 0x5648670e0460 2024-03-21T10:40:03.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.519+0000 7fe53fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648670d7dd0 0x5648670dd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.519+0000 7fe53fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648670d7dd0 0x5648670dd010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42120/0 (socket says 172.21.15.31:42120) 2024-03-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.519+0000 7fe53fe00640 1 -- 172.21.15.31:0/347021201 learned_addr learned my addr 172.21.15.31:0/347021201 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.519+0000 7fe55117a740 10 client.0 _ll_register_callbacks cb 0x564867009130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.519+0000 7fe547400640 1 --2- 172.21.15.31:0/347021201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648670d70f0 0x5648670dfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.519+0000 7fe547e00640 1 --2- 172.21.15.31:0/347021201 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648670e0460 0x5648670dd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.519+0000 7fe53fe00640 1 -- 172.21.15.31:0/347021201 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648670e0460 msgr2=0x5648670dd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.519+0000 7fe53fe00640 1 --2- 172.21.15.31:0/347021201 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648670e0460 0x5648670dd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.519+0000 7fe53fe00640 1 -- 172.21.15.31:0/347021201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648670d70f0 msgr2=0x5648670dfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.519+0000 7fe53fe00640 1 --2- 172.21.15.31:0/347021201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648670d70f0 0x5648670dfc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.519+0000 7fe53fe00640 1 -- 172.21.15.31:0/347021201 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564866e197d0 con 0x5648670d7dd0 2024-03-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.520+0000 7fe547e00640 1 --2- 172.21.15.31:0/347021201 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648670e0460 0x5648670dd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.520+0000 7fe547400640 1 --2- 172.21.15.31:0/347021201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648670d70f0 0x5648670dfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.520+0000 7fe53fe00640 1 --2- 172.21.15.31:0/347021201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648670d7dd0 0x5648670dd010 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe538005ab0 tx=0x7fe53802cb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5cc1de6484d91c4b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.520+0000 7fe535600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:40:03.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.520+0000 7fe535600640 1 -- 172.21.15.31:0/347021201 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53804c020 con 0x5648670d7dd0 2024-03-21T10:40:03.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.520+0000 7fe55117a740 1 -- 172.21.15.31:0/347021201 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe53802c6f0 con 0x5648670d7dd0 2024-03-21T10:40:03.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.520+0000 7fe535600640 1 -- 172.21.15.31:0/347021201 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fe538004240 con 0x5648670d7dd0 2024-03-21T10:40:03.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.521+0000 7fe535600640 1 -- 172.21.15.31:0/347021201 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe538057050 con 0x5648670d7dd0 2024-03-21T10:40:03.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.521+0000 7fe55117a740 10 client.26096 fetch_fsmap learned FSMap version 24 2024-03-21T10:40:03.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.521+0000 7fe55117a740 1 -- 172.21.15.31:0/347021201 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x564866d4f3f0 con 0x5648670d7dd0 2024-03-21T10:40:03.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.521+0000 7fe535600640 1 -- 172.21.15.31:0/347021201 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe53802c6f0 con 0x5648670d7dd0 2024-03-21T10:40:03.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.521+0000 7fe535600640 1 -- 172.21.15.31:0/347021201 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fe538047030 con 0x5648670d7dd0 2024-03-21T10:40:03.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.521+0000 7fe55117a740 10 client.26096 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:40:03.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.521+0000 7fe55117a740 10 client.26096 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:40:03.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.521+0000 7fe55117a740 20 client.26096 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:03.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.521+0000 7fe55117a740 1 --2- 172.21.15.31:0/347021201 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5648670f02c0 0x5648670f26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.521+0000 7fe55117a740 4 client.26096 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:40:03.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.521+0000 7fe55117a740 1 -- 172.21.15.31:0/347021201 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564867048790 con 0x5648670f02c0 2024-03-21T10:40:03.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.522+0000 7fe547400640 1 --2- 172.21.15.31:0/347021201 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5648670f02c0 0x5648670f26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.524+0000 7fe547400640 1 --2- 172.21.15.31:0/347021201 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5648670f02c0 0x5648670f26b0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcee054a69857605 server_cookie=c7bbf3d76a1f2029 in_seq=0 out_seq=0 2024-03-21T10:40:03.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.525+0000 7fe535600640 10 client.26096 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:03.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.526+0000 7fe535600640 1 -- 172.21.15.31:0/347021201 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564867048790 con 0x5648670f02c0 2024-03-21T10:40:03.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.526+0000 7fe535600640 10 client.26096 handle_command_reply: tid=0 2024-03-21T10:40:03.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.526+0000 7fe55117a740 1 client.26096 shutdown 2024-03-21T10:40:03.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.527+0000 7fe55117a740 1 -- 172.21.15.31:0/347021201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648670d7dd0 msgr2=0x5648670dd010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:03.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.527+0000 7fe55117a740 1 --2- 172.21.15.31:0/347021201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648670d7dd0 0x5648670dd010 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe538005ab0 tx=0x7fe53802cb50 comp rx=0 tx=0).stop 2024-03-21T10:40:03.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.527+0000 7fe55117a740 1 -- 172.21.15.31:0/347021201 shutdown_connections 2024-03-21T10:40:03.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.527+0000 7fe55117a740 1 --2- 172.21.15.31:0/347021201 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5648670f02c0 0x5648670f26b0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.527+0000 7fe55117a740 1 --2- 172.21.15.31:0/347021201 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648670e0460 0x5648670dd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.527+0000 7fe55117a740 1 --2- 172.21.15.31:0/347021201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648670d7dd0 0x5648670dd010 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.527+0000 7fe55117a740 1 --2- 172.21.15.31:0/347021201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648670d70f0 0x5648670dfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.527+0000 7fe55117a740 1 -- 172.21.15.31:0/347021201 >> 172.21.15.31:0/347021201 conn(0x56486709e200 msgr2=0x5648670d3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:03.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.527+0000 7fe535600640 0 client.26096 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:03.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.528+0000 7fe55117a740 1 -- 172.21.15.31:0/347021201 shutdown_connections 2024-03-21T10:40:03.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.528+0000 7fe55117a740 1 -- 172.21.15.31:0/347021201 wait complete. 2024-03-21T10:40:03.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.529+0000 7fe55117a740 20 client.26096 trim_cache size 0 max 16384 2024-03-21T10:40:03.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.565+0000 7fe55117a740 1 Processor -- start 2024-03-21T10:40:03.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.565+0000 7fe55117a740 1 -- start start 2024-03-21T10:40:03.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.565+0000 7fe55117a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648670d70f0 0x5648670df740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.566+0000 7fe55117a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648670d7dd0 0x5648670dfc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.566+0000 7fe55117a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648670e0460 0x5648670e1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.566+0000 7fe55117a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5481bf9c0 con 0x5648670d70f0 2024-03-21T10:40:03.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.566+0000 7fe55117a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5481bd4c0 con 0x5648670d7dd0 2024-03-21T10:40:03.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.566+0000 7fe55117a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5481bb560 con 0x5648670e0460 2024-03-21T10:40:03.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.566+0000 7fe547400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648670d70f0 0x5648670df740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.566+0000 7fe53fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648670d7dd0 0x5648670dfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.566+0000 7fe53fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648670d7dd0 0x5648670dfc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42134/0 (socket says 172.21.15.31:42134) 2024-03-21T10:40:03.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.566+0000 7fe547400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648670d70f0 0x5648670df740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36478/0 (socket says 172.21.15.31:36478) 2024-03-21T10:40:03.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.566+0000 7fe53fe00640 1 -- 172.21.15.31:0/3890462717 learned_addr learned my addr 172.21.15.31:0/3890462717 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:03.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.567+0000 7fe547e00640 1 --2- 172.21.15.31:0/3890462717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648670e0460 0x5648670e1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.567+0000 7fe53fe00640 1 -- 172.21.15.31:0/3890462717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648670e0460 msgr2=0x5648670e1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:03.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.567+0000 7fe53fe00640 1 --2- 172.21.15.31:0/3890462717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648670e0460 0x5648670e1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.567+0000 7fe53fe00640 1 -- 172.21.15.31:0/3890462717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648670d70f0 msgr2=0x5648670df740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:03.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.567+0000 7fe53fe00640 1 --2- 172.21.15.31:0/3890462717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648670d70f0 0x5648670df740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.567+0000 7fe53fe00640 1 -- 172.21.15.31:0/3890462717 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564866d0f0f0 con 0x5648670d7dd0 2024-03-21T10:40:03.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.567+0000 7fe53fe00640 1 --2- 172.21.15.31:0/3890462717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648670d7dd0 0x5648670dfc80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe5380092a0 tx=0x7fe538000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf8f6955625e989c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:03.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.567+0000 7fe547400640 1 --2- 172.21.15.31:0/3890462717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648670d70f0 0x5648670df740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:40:03.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.568+0000 7fe537e00640 1 -- 172.21.15.31:0/3890462717 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53804c070 con 0x5648670d7dd0 2024-03-21T10:40:03.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.568+0000 7fe537e00640 1 -- 172.21.15.31:0/3890462717 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe53802c670 con 0x5648670d7dd0 2024-03-21T10:40:03.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.568+0000 7fe537e00640 1 -- 172.21.15.31:0/3890462717 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe538044040 con 0x5648670d7dd0 2024-03-21T10:40:03.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.569+0000 7fe55117a740 1 -- 172.21.15.31:0/3890462717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648670d7dd0 msgr2=0x5648670dfc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:03.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.569+0000 7fe55117a740 1 --2- 172.21.15.31:0/3890462717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648670d7dd0 0x5648670dfc80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe5380092a0 tx=0x7fe538000bc0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.569+0000 7fe55117a740 1 -- 172.21.15.31:0/3890462717 shutdown_connections 2024-03-21T10:40:03.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.569+0000 7fe55117a740 1 --2- 172.21.15.31:0/3890462717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648670e0460 0x5648670e1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.569+0000 7fe55117a740 1 --2- 172.21.15.31:0/3890462717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648670d7dd0 0x5648670dfc80 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.569+0000 7fe55117a740 1 --2- 172.21.15.31:0/3890462717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648670d70f0 0x5648670df740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.569+0000 7fe55117a740 1 -- 172.21.15.31:0/3890462717 >> 172.21.15.31:0/3890462717 conn(0x56486709e200 msgr2=0x5648670d3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:03.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.569+0000 7fe55117a740 1 -- 172.21.15.31:0/3890462717 shutdown_connections 2024-03-21T10:40:03.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.569+0000 7fe55117a740 1 -- 172.21.15.31:0/3890462717 wait complete. 2024-03-21T10:40:03.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.570+0000 7fe55117a740 1 -- start start 2024-03-21T10:40:03.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.570+0000 7fe55117a740 1 Processor -- start 2024-03-21T10:40:03.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.571+0000 7fe55117a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648670d70f0 0x564867144af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.571+0000 7fe55117a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648670d7dd0 0x564867145030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.571+0000 7fe55117a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648670e0460 0x5648670d9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.571+0000 7fe55117a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5481bbb60 con 0x5648670e0460 2024-03-21T10:40:03.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.571+0000 7fe55117a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5481bce60 con 0x5648670d70f0 2024-03-21T10:40:03.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.571+0000 7fe55117a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5481bc1a0 con 0x5648670d7dd0 2024-03-21T10:40:03.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.571+0000 7fe55117a740 10 client.0 _ll_register_callbacks cb 0x7fe538042070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:03.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.571+0000 7fe53fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648670d7dd0 0x564867145030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.571+0000 7fe547400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648670d70f0 0x564867144af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.571+0000 7fe53fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648670d7dd0 0x564867145030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42716/0 (socket says 172.21.15.31:42716) 2024-03-21T10:40:03.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.571+0000 7fe53fe00640 1 -- 172.21.15.31:0/1598002906 learned_addr learned my addr 172.21.15.31:0/1598002906 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:03.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.572+0000 7fe547e00640 1 --2- 172.21.15.31:0/1598002906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648670e0460 0x5648670d9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.572+0000 7fe53fe00640 1 -- 172.21.15.31:0/1598002906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648670d70f0 msgr2=0x564867144af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:03.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.572+0000 7fe53fe00640 1 --2- 172.21.15.31:0/1598002906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648670d70f0 0x564867144af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.572+0000 7fe53fe00640 1 -- 172.21.15.31:0/1598002906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648670e0460 msgr2=0x5648670d9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:03.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.572+0000 7fe53fe00640 1 --2- 172.21.15.31:0/1598002906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648670e0460 0x5648670d9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.572+0000 7fe53fe00640 1 -- 172.21.15.31:0/1598002906 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5648670da4c0 con 0x5648670d7dd0 2024-03-21T10:40:03.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.572+0000 7fe547e00640 1 --2- 172.21.15.31:0/1598002906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648670e0460 0x5648670d9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:03.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.572+0000 7fe53fe00640 1 --2- 172.21.15.31:0/1598002906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648670d7dd0 0x564867145030 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe538005ab0 tx=0x7fe538009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acdaf7d60420e45b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:03.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.572+0000 7fe535600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:40:03.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.573+0000 7fe535600640 1 -- 172.21.15.31:0/1598002906 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe538061060 con 0x5648670d7dd0 2024-03-21T10:40:03.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.573+0000 7fe535600640 1 -- 172.21.15.31:0/1598002906 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fe53802cd30 con 0x5648670d7dd0 2024-03-21T10:40:03.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.573+0000 7fe55117a740 1 -- 172.21.15.31:0/1598002906 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe53802c670 con 0x5648670d7dd0 2024-03-21T10:40:03.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.574+0000 7fe535600640 1 -- 172.21.15.31:0/1598002906 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe538068050 con 0x5648670d7dd0 2024-03-21T10:40:03.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.575+0000 7fe55117a740 10 client.26077 fetch_fsmap learned FSMap version 24 2024-03-21T10:40:03.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.575+0000 7fe55117a740 1 -- 172.21.15.31:0/1598002906 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x56486714c230 con 0x5648670d7dd0 2024-03-21T10:40:03.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.576+0000 7fe535600640 1 -- 172.21.15.31:0/1598002906 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe538004b40 con 0x5648670d7dd0 2024-03-21T10:40:03.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.576+0000 7fe535600640 1 -- 172.21.15.31:0/1598002906 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fe538047040 con 0x5648670d7dd0 2024-03-21T10:40:03.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.576+0000 7fe55117a740 10 client.26077 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:40:03.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.576+0000 7fe55117a740 10 client.26077 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:40:03.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.576+0000 7fe55117a740 20 client.26077 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:03.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.576+0000 7fe55117a740 1 --2- 172.21.15.31:0/1598002906 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56486714c880 0x56486714eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:03.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.576+0000 7fe55117a740 4 client.26077 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:40:03.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.576+0000 7fe55117a740 1 -- 172.21.15.31:0/1598002906 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7fe53802c670 con 0x56486714c880 2024-03-21T10:40:03.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.577+0000 7fe547400640 1 --2- 172.21.15.31:0/1598002906 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56486714c880 0x56486714eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:03.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.577+0000 7fe547400640 1 --2- 172.21.15.31:0/1598002906 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56486714c880 0x56486714eca0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=842d3ac45a279bc7 server_cookie=afb4ef3760b94df7 in_seq=0 out_seq=0 2024-03-21T10:40:03.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.577+0000 7fe535600640 10 client.26077 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:03.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.578+0000 7fe535600640 1 -- 172.21.15.31:0/1598002906 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x7fe53802c670 con 0x56486714c880 2024-03-21T10:40:03.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.578+0000 7fe535600640 10 client.26077 handle_command_reply: tid=0 2024-03-21T10:40:03.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.578+0000 7fe55117a740 1 client.26077 shutdown 2024-03-21T10:40:03.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.578+0000 7fe55117a740 1 -- 172.21.15.31:0/1598002906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648670d7dd0 msgr2=0x564867145030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:03.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.578+0000 7fe55117a740 1 --2- 172.21.15.31:0/1598002906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648670d7dd0 0x564867145030 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe538005ab0 tx=0x7fe538009800 comp rx=0 tx=0).stop 2024-03-21T10:40:03.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.578+0000 7fe55117a740 1 -- 172.21.15.31:0/1598002906 shutdown_connections 2024-03-21T10:40:03.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.578+0000 7fe55117a740 1 --2- 172.21.15.31:0/1598002906 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56486714c880 0x56486714eca0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.578+0000 7fe55117a740 1 --2- 172.21.15.31:0/1598002906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648670e0460 0x5648670d9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.578+0000 7fe55117a740 1 --2- 172.21.15.31:0/1598002906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648670d7dd0 0x564867145030 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.578+0000 7fe55117a740 1 --2- 172.21.15.31:0/1598002906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648670d70f0 0x564867144af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.578+0000 7fe55117a740 1 -- 172.21.15.31:0/1598002906 >> 172.21.15.31:0/1598002906 conn(0x56486709e200 msgr2=0x5648670d2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:03.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.578+0000 7fe535600640 0 client.26077 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:03.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.578+0000 7fe55117a740 1 -- 172.21.15.31:0/1598002906 shutdown_connections 2024-03-21T10:40:03.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.578+0000 7fe55117a740 1 -- 172.21.15.31:0/1598002906 wait complete. 2024-03-21T10:40:03.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.579+0000 7fe55117a740 20 client.26077 trim_cache size 0 max 16384 2024-03-21T10:40:03.578 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:40:03.578 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-21T10:40:03.578 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-03-21T10:40:03.578 INFO:teuthology.orchestra.run.smithi031.stdout: "66ee601a-875d-4d41-8aea-65c7a3b8fd19": { 2024-03-21T10:40:03.578 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-03-21T10:40:03.578 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "66ee601a-875d-4d41-8aea-65c7a3b8fd19", 2024-03-21T10:40:03.578 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-03-21T10:40:03.578 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:40:03.578 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:40:03.579 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:40:03.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.579+0000 7fe54ce00640 1 -- 172.21.15.31:0/677939463 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe52c0a3230 msgr2=0x7fe52c0a56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:03.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.579+0000 7fe54ce00640 1 --2- 172.21.15.31:0/677939463 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe52c0a3230 0x7fe52c0a56f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe53000f640 tx=0x7fe530011040 comp rx=0 tx=0).stop 2024-03-21T10:40:03.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.579+0000 7fe54ce00640 1 -- 172.21.15.31:0/677939463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe54810ba50 msgr2=0x7fe548195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:03.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.579+0000 7fe54ce00640 1 --2- 172.21.15.31:0/677939463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe54810ba50 0x7fe548195ed0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe54000da10 tx=0x7fe54000ded0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.580+0000 7fe54ce00640 1 -- 172.21.15.31:0/677939463 shutdown_connections 2024-03-21T10:40:03.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.580+0000 7fe54ce00640 1 --2- 172.21.15.31:0/677939463 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe52c0a3230 0x7fe52c0a56f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.580+0000 7fe54ce00640 1 --2- 172.21.15.31:0/677939463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe54810ba50 0x7fe548195ed0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.580+0000 7fe54ce00640 1 --2- 172.21.15.31:0/677939463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe548102f50 0x7fe548195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.580+0000 7fe54ce00640 1 --2- 172.21.15.31:0/677939463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe548102630 0x7fe54819c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:03.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.580+0000 7fe54ce00640 1 -- 172.21.15.31:0/677939463 >> 172.21.15.31:0/677939463 conn(0x7fe548077130 msgr2=0x7fe5480fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:03.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.581+0000 7fe54ce00640 1 -- 172.21.15.31:0/677939463 shutdown_connections 2024-03-21T10:40:03.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:03.581+0000 7fe54ce00640 1 -- 172.21.15.31:0/677939463 wait complete. 2024-03-21T10:40:03.603 INFO:tasks.cephfs.filesystem:scrub status for tag:66ee601a-875d-4d41-8aea-65c7a3b8fd19 - {'path': '/', 'tag': '66ee601a-875d-4d41-8aea-65c7a3b8fd19', 'options': 'recursive,force'} 2024-03-21T10:40:04.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:03 smithi031 ceph-mon[31673]: scrub queued for path: / 2024-03-21T10:40:04.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:03 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:40:04.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:03 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:40:04.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:03 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:40:04.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:03 smithi070 ceph-mon[36330]: scrub queued for path: / 2024-03-21T10:40:04.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:03 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:40:04.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:03 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:40:04.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:03 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:40:04.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:03 smithi040 ceph-mon[36154]: scrub queued for path: / 2024-03-21T10:40:04.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:03 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:40:04.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:03 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:40:04.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:03 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:40:05.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:04 smithi031 ceph-mon[31673]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-21T10:40:05.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:04 smithi070 ceph-mon[36330]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-21T10:40:05.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:04 smithi040 ceph-mon[36154]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-21T10:40:07.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:06 smithi031 ceph-mon[31673]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-21T10:40:07.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:06 smithi031 ceph-mon[31673]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-03-21T10:40:07.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:06 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:40:07.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:06 smithi070 ceph-mon[36330]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-21T10:40:07.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:06 smithi070 ceph-mon[36330]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-03-21T10:40:07.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:06 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:40:07.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:06 smithi040 ceph-mon[36154]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-21T10:40:07.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:06 smithi040 ceph-mon[36154]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-03-21T10:40:07.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:06 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:40:08.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.884+0000 7f6a6de00640 1 -- 172.21.15.31:0/1773002716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a681075b0 msgr2=0x7f6a68107990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:08.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.884+0000 7f6a6de00640 1 --2- 172.21.15.31:0/1773002716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a681075b0 0x7f6a68107990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c009930 tx=0x7f6a5c02bc20 comp rx=0 tx=0).stop 2024-03-21T10:40:08.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.885+0000 7f6a6de00640 1 -- 172.21.15.31:0/1773002716 shutdown_connections 2024-03-21T10:40:08.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.885+0000 7f6a6de00640 1 --2- 172.21.15.31:0/1773002716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a68069850 0x7f6a6810e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.885+0000 7f6a6de00640 1 --2- 172.21.15.31:0/1773002716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a68068e90 0x7f6a68069310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.885+0000 7f6a6de00640 1 --2- 172.21.15.31:0/1773002716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a681075b0 0x7f6a68107990 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.885+0000 7f6a6de00640 1 -- 172.21.15.31:0/1773002716 >> 172.21.15.31:0/1773002716 conn(0x7f6a68075f00 msgr2=0x7f6a68076310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:08.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.885+0000 7f6a6de00640 1 -- 172.21.15.31:0/1773002716 shutdown_connections 2024-03-21T10:40:08.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.885+0000 7f6a6de00640 1 -- 172.21.15.31:0/1773002716 wait complete. 2024-03-21T10:40:08.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.887+0000 7f6a6de00640 1 Processor -- start 2024-03-21T10:40:08.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.887+0000 7f6a6de00640 1 -- start start 2024-03-21T10:40:08.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.888+0000 7f6a6de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a68068e90 0x7f6a68197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:08.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.888+0000 7f6a6de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a68069850 0x7f6a68198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:08.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.888+0000 7f6a6de00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a681075b0 0x7f6a6819c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:08.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.888+0000 7f6a6de00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a68111830 con 0x7f6a68068e90 2024-03-21T10:40:08.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.888+0000 7f6a6de00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a681116b0 con 0x7f6a68069850 2024-03-21T10:40:08.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.888+0000 7f6a6de00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a681119b0 con 0x7f6a681075b0 2024-03-21T10:40:08.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.888+0000 7f6a67e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a68069850 0x7f6a68198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:08.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.888+0000 7f6a6d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a681075b0 0x7f6a6819c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:08.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.888+0000 7f6a6ca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a68068e90 0x7f6a68197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:08.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.888+0000 7f6a6d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a681075b0 0x7f6a6819c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:59780/0 (socket says 172.21.15.31:59780) 2024-03-21T10:40:08.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.888+0000 7f6a6d400640 1 -- 172.21.15.31:0/707147320 learned_addr learned my addr 172.21.15.31:0/707147320 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:08.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.888+0000 7f6a67e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a68069850 0x7f6a68198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:47460/0 (socket says 172.21.15.31:47460) 2024-03-21T10:40:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.889+0000 7f6a6d400640 1 -- 172.21.15.31:0/707147320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a68069850 msgr2=0x7f6a68198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.889+0000 7f6a6d400640 1 --2- 172.21.15.31:0/707147320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a68069850 0x7f6a68198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.889+0000 7f6a6d400640 1 -- 172.21.15.31:0/707147320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a68068e90 msgr2=0x7f6a68197d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.889+0000 7f6a6d400640 1 --2- 172.21.15.31:0/707147320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a68068e90 0x7f6a68197d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.889+0000 7f6a6d400640 1 -- 172.21.15.31:0/707147320 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6819cd20 con 0x7f6a681075b0 2024-03-21T10:40:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.889+0000 7f6a67e00640 1 --2- 172.21.15.31:0/707147320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a68069850 0x7f6a68198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:40:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.889+0000 7f6a6d400640 1 --2- 172.21.15.31:0/707147320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a681075b0 0x7f6a6819c5f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009fd0 tx=0x7f6a6000b260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.889+0000 7f6a65600640 1 -- 172.21.15.31:0/707147320 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a60017070 con 0x7f6a681075b0 2024-03-21T10:40:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.890+0000 7f6a65600640 1 -- 172.21.15.31:0/707147320 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a60003e50 con 0x7f6a681075b0 2024-03-21T10:40:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.890+0000 7f6a6de00640 1 -- 172.21.15.31:0/707147320 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a6819d010 con 0x7f6a681075b0 2024-03-21T10:40:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.890+0000 7f6a65600640 1 -- 172.21.15.31:0/707147320 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a600106c0 con 0x7f6a681075b0 2024-03-21T10:40:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.890+0000 7f6a6de00640 1 -- 172.21.15.31:0/707147320 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a681a48a0 con 0x7f6a681075b0 2024-03-21T10:40:08.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.893+0000 7f6a65600640 1 -- 172.21.15.31:0/707147320 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f6a60003970 con 0x7f6a681075b0 2024-03-21T10:40:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.894+0000 7f6a65600640 1 --2- 172.21.15.31:0/707147320 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6a500a30f0 0x7f6a500a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.894+0000 7f6a7203f740 1 Processor -- start 2024-03-21T10:40:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.894+0000 7f6a65600640 1 -- 172.21.15.31:0/707147320 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f6a600cf830 con 0x7f6a681075b0 2024-03-21T10:40:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.894+0000 7f6a7203f740 1 -- start start 2024-03-21T10:40:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.894+0000 7f6a6ca00640 1 --2- 172.21.15.31:0/707147320 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6a500a30f0 0x7f6a500a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.895+0000 7f6a7203f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bf88f80ba0 0x55bf88f80f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.895+0000 7f6a7203f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f814c0 0x55bf88f8e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.895+0000 7f6a7203f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bf88f8ece0 0x55bf88f910d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.895+0000 7f6a7203f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf88f433b0 con 0x55bf88f8ece0 2024-03-21T10:40:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.895+0000 7f6a7203f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf88ee5830 con 0x55bf88f814c0 2024-03-21T10:40:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.895+0000 7f6a7203f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf88e8e5d0 con 0x55bf88f80ba0 2024-03-21T10:40:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.895+0000 7f6a67e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f814c0 0x55bf88f8e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.895+0000 7f6a67e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f814c0 0x55bf88f8e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:47470/0 (socket says 172.21.15.31:47470) 2024-03-21T10:40:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.895+0000 7f6a67e00640 1 -- 172.21.15.31:0/4161186005 learned_addr learned my addr 172.21.15.31:0/4161186005 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:08.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.895+0000 7f6a6d400640 1 --2- 172.21.15.31:0/4161186005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bf88f8ece0 0x55bf88f910d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:08.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.895+0000 7f6a6ca00640 1 --2- 172.21.15.31:0/4161186005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bf88f80ba0 0x55bf88f80f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:08.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.895+0000 7f6a67e00640 1 -- 172.21.15.31:0/4161186005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bf88f80ba0 msgr2=0x55bf88f80f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:08.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.895+0000 7f6a67e00640 1 --2- 172.21.15.31:0/4161186005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bf88f80ba0 0x55bf88f80f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.895+0000 7f6a67e00640 1 -- 172.21.15.31:0/4161186005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bf88f8ece0 msgr2=0x55bf88f910d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:08.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.895+0000 7f6a67e00640 1 --2- 172.21.15.31:0/4161186005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bf88f8ece0 0x55bf88f910d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.895+0000 7f6a67e00640 1 -- 172.21.15.31:0/4161186005 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bf88ea62c0 con 0x55bf88f814c0 2024-03-21T10:40:08.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.896+0000 7f6a6ca00640 1 --2- 172.21.15.31:0/707147320 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6a500a30f0 0x7f6a500a55b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c009900 tx=0x7f6a5c000bc0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:08.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.896+0000 7f6a67e00640 1 --2- 172.21.15.31:0/4161186005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f814c0 0x55bf88f8e7a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f6a54010480 tx=0x7f6a54010940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd25384a7893d85d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:08.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.896+0000 7f6a59c00640 1 -- 172.21.15.31:0/4161186005 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a54011760 con 0x55bf88f814c0 2024-03-21T10:40:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.896+0000 7f6a59c00640 1 -- 172.21.15.31:0/4161186005 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a54011de0 con 0x55bf88f814c0 2024-03-21T10:40:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.896+0000 7f6a59c00640 1 -- 172.21.15.31:0/4161186005 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5400e430 con 0x55bf88f814c0 2024-03-21T10:40:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.897+0000 7f6a7203f740 1 -- 172.21.15.31:0/4161186005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f814c0 msgr2=0x55bf88f8e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.897+0000 7f6a7203f740 1 --2- 172.21.15.31:0/4161186005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f814c0 0x55bf88f8e7a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f6a54010480 tx=0x7f6a54010940 comp rx=0 tx=0).stop 2024-03-21T10:40:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.897+0000 7f6a7203f740 1 -- 172.21.15.31:0/4161186005 shutdown_connections 2024-03-21T10:40:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.897+0000 7f6a7203f740 1 --2- 172.21.15.31:0/4161186005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bf88f8ece0 0x55bf88f910d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.897+0000 7f6a7203f740 1 --2- 172.21.15.31:0/4161186005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f814c0 0x55bf88f8e7a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.897+0000 7f6a7203f740 1 --2- 172.21.15.31:0/4161186005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bf88f80ba0 0x55bf88f80f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.897+0000 7f6a7203f740 1 -- 172.21.15.31:0/4161186005 >> 172.21.15.31:0/4161186005 conn(0x55bf88f7bb00 msgr2=0x55bf88f7bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.897+0000 7f6a7203f740 1 -- 172.21.15.31:0/4161186005 shutdown_connections 2024-03-21T10:40:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.897+0000 7f6a7203f740 1 -- 172.21.15.31:0/4161186005 wait complete. 2024-03-21T10:40:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.898+0000 7f6a7203f740 1 -- start start 2024-03-21T10:40:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.898+0000 7f6a7203f740 1 Processor -- start 2024-03-21T10:40:08.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.899+0000 7f6a7203f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f80ba0 0x55bf88f8bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:08.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.899+0000 7f6a7203f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bf88f814c0 0x55bf88f8c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:08.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.899+0000 7f6a7203f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bf88f8ece0 0x55bf88f8c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:08.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.899+0000 7f6a7203f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf88ea1a20 con 0x55bf88f814c0 2024-03-21T10:40:08.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.899+0000 7f6a7203f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf88ee18d0 con 0x55bf88f80ba0 2024-03-21T10:40:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.899+0000 7f6a7203f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf88f91910 con 0x55bf88f8ece0 2024-03-21T10:40:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.899+0000 7f6a7203f740 10 client.0 _ll_register_callbacks cb 0x55bf88ede1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.899+0000 7f6a6ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f80ba0 0x55bf88f8bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.899+0000 7f6a6ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f80ba0 0x55bf88f8bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:47472/0 (socket says 172.21.15.31:47472) 2024-03-21T10:40:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.899+0000 7f6a67e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bf88f814c0 0x55bf88f8c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.899+0000 7f6a6ca00640 1 -- 172.21.15.31:0/4292963396 learned_addr learned my addr 172.21.15.31:0/4292963396 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.899+0000 7f6a6d400640 1 --2- 172.21.15.31:0/4292963396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bf88f8ece0 0x55bf88f8c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.899+0000 7f6a6ca00640 1 -- 172.21.15.31:0/4292963396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bf88f8ece0 msgr2=0x55bf88f8c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.899+0000 7f6a6ca00640 1 --2- 172.21.15.31:0/4292963396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bf88f8ece0 0x55bf88f8c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.899+0000 7f6a6ca00640 1 -- 172.21.15.31:0/4292963396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bf88f814c0 msgr2=0x55bf88f8c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.899+0000 7f6a6ca00640 1 --2- 172.21.15.31:0/4292963396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bf88f814c0 0x55bf88f8c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.899+0000 7f6a6ca00640 1 -- 172.21.15.31:0/4292963396 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bf88bfa210 con 0x55bf88f80ba0 2024-03-21T10:40:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.900+0000 7f6a6ca00640 1 --2- 172.21.15.31:0/4292963396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f80ba0 0x55bf88f8bdc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c009930 tx=0x7f6a5c02bc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5c361d2058c8aee server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:08.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.900+0000 7f6a4e000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:40:08.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.900+0000 7f6a4e000640 1 -- 172.21.15.31:0/4292963396 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c004d10 con 0x55bf88f80ba0 2024-03-21T10:40:08.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.900+0000 7f6a7203f740 1 -- 172.21.15.31:0/4292963396 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6a54011de0 con 0x55bf88f80ba0 2024-03-21T10:40:08.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.900+0000 7f6a4e000640 1 -- 172.21.15.31:0/4292963396 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f6a5c004ed0 con 0x55bf88f80ba0 2024-03-21T10:40:08.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.900+0000 7f6a4e000640 1 -- 172.21.15.31:0/4292963396 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a5c05c050 con 0x55bf88f80ba0 2024-03-21T10:40:08.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.900+0000 7f6a7203f740 10 client.26114 fetch_fsmap learned FSMap version 24 2024-03-21T10:40:08.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.900+0000 7f6a7203f740 1 -- 172.21.15.31:0/4292963396 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55bf88cc6620 con 0x55bf88f80ba0 2024-03-21T10:40:08.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.900+0000 7f6a4e000640 1 -- 172.21.15.31:0/4292963396 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a54011de0 con 0x55bf88f80ba0 2024-03-21T10:40:08.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.901+0000 7f6a4e000640 1 -- 172.21.15.31:0/4292963396 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f6a5c04b440 con 0x55bf88f80ba0 2024-03-21T10:40:08.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.901+0000 7f6a7203f740 10 client.26114 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:40:08.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.901+0000 7f6a7203f740 10 client.26114 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:40:08.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.901+0000 7f6a7203f740 20 client.26114 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:08.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.901+0000 7f6a7203f740 1 --2- 172.21.15.31:0/4292963396 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55bf88f9eda0 0x55bf88fa1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:08.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.901+0000 7f6a7203f740 4 client.26114 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:40:08.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.901+0000 7f6a7203f740 1 -- 172.21.15.31:0/4292963396 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bf88ef2d70 con 0x55bf88f9eda0 2024-03-21T10:40:08.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.901+0000 7f6a67e00640 1 --2- 172.21.15.31:0/4292963396 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55bf88f9eda0 0x55bf88fa1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:08.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.902+0000 7f6a67e00640 1 --2- 172.21.15.31:0/4292963396 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55bf88f9eda0 0x55bf88fa1190 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b50ad284dee354a server_cookie=d96718c9997fb63c in_seq=0 out_seq=0 2024-03-21T10:40:08.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.902+0000 7f6a4e000640 10 client.26114 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:08.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.902+0000 7f6a4e000640 1 -- 172.21.15.31:0/4292963396 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55bf88ef2d70 con 0x55bf88f9eda0 2024-03-21T10:40:08.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.902+0000 7f6a4e000640 10 client.26114 handle_command_reply: tid=0 2024-03-21T10:40:08.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.903+0000 7f6a7203f740 1 client.26114 shutdown 2024-03-21T10:40:08.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.903+0000 7f6a7203f740 1 -- 172.21.15.31:0/4292963396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f80ba0 msgr2=0x55bf88f8bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:08.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.903+0000 7f6a7203f740 1 --2- 172.21.15.31:0/4292963396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f80ba0 0x55bf88f8bdc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c009930 tx=0x7f6a5c02bc20 comp rx=0 tx=0).stop 2024-03-21T10:40:08.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.903+0000 7f6a7203f740 1 -- 172.21.15.31:0/4292963396 shutdown_connections 2024-03-21T10:40:08.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.903+0000 7f6a7203f740 1 --2- 172.21.15.31:0/4292963396 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55bf88f9eda0 0x55bf88fa1190 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.903+0000 7f6a7203f740 1 --2- 172.21.15.31:0/4292963396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bf88f8ece0 0x55bf88f8c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.903+0000 7f6a7203f740 1 --2- 172.21.15.31:0/4292963396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bf88f814c0 0x55bf88f8c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.903+0000 7f6a7203f740 1 --2- 172.21.15.31:0/4292963396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f80ba0 0x55bf88f8bdc0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.903+0000 7f6a7203f740 1 -- 172.21.15.31:0/4292963396 >> 172.21.15.31:0/4292963396 conn(0x55bf88f7bb00 msgr2=0x55bf88f7d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:08.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.903+0000 7f6a4e000640 0 client.26114 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:08.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.903+0000 7f6a7203f740 1 -- 172.21.15.31:0/4292963396 shutdown_connections 2024-03-21T10:40:08.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.903+0000 7f6a7203f740 1 -- 172.21.15.31:0/4292963396 wait complete. 2024-03-21T10:40:08.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.903+0000 7f6a7203f740 20 client.26114 trim_cache size 0 max 16384 2024-03-21T10:40:08.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.919+0000 7f6a7203f740 1 Processor -- start 2024-03-21T10:40:08.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.919+0000 7f6a7203f740 1 -- start start 2024-03-21T10:40:08.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.919+0000 7f6a7203f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f80ba0 0x55bf88f8e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:08.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.919+0000 7f6a7203f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bf88f814c0 0x55bf88f9a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:08.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.919+0000 7f6a7203f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bf88f8ece0 0x55bf88f9a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:08.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.919+0000 7f6a7203f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a681ae9e0 con 0x55bf88f814c0 2024-03-21T10:40:08.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.919+0000 7f6a7203f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a681ac4e0 con 0x55bf88f80ba0 2024-03-21T10:40:08.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.919+0000 7f6a7203f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a681aa580 con 0x55bf88f8ece0 2024-03-21T10:40:08.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.920+0000 7f6a6ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f80ba0 0x55bf88f8e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:08.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.920+0000 7f6a6ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f80ba0 0x55bf88f8e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:47482/0 (socket says 172.21.15.31:47482) 2024-03-21T10:40:08.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.920+0000 7f6a6ca00640 1 -- 172.21.15.31:0/3017810204 learned_addr learned my addr 172.21.15.31:0/3017810204 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:08.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.920+0000 7f6a6d400640 1 --2- 172.21.15.31:0/3017810204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bf88f8ece0 0x55bf88f9a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:08.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.920+0000 7f6a67e00640 1 --2- 172.21.15.31:0/3017810204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bf88f814c0 0x55bf88f9a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:08.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.920+0000 7f6a6ca00640 1 -- 172.21.15.31:0/3017810204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bf88f8ece0 msgr2=0x55bf88f9a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:08.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.920+0000 7f6a6ca00640 1 --2- 172.21.15.31:0/3017810204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bf88f8ece0 0x55bf88f9a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.920+0000 7f6a6ca00640 1 -- 172.21.15.31:0/3017810204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bf88f814c0 msgr2=0x55bf88f9a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:08.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.920+0000 7f6a6ca00640 1 --2- 172.21.15.31:0/3017810204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bf88f814c0 0x55bf88f9a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.920+0000 7f6a6ca00640 1 -- 172.21.15.31:0/3017810204 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bf88fa1500 con 0x55bf88f80ba0 2024-03-21T10:40:08.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.920+0000 7f6a6ca00640 1 --2- 172.21.15.31:0/3017810204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f80ba0 0x55bf88f8e4f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c02c780 tx=0x7f6a5c02c7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9905b5b827d067b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:08.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.921+0000 7f6a59200640 1 -- 172.21.15.31:0/3017810204 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c065070 con 0x55bf88f80ba0 2024-03-21T10:40:08.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.921+0000 7f6a59200640 1 -- 172.21.15.31:0/3017810204 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a5c02cc10 con 0x55bf88f80ba0 2024-03-21T10:40:08.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.921+0000 7f6a59200640 1 -- 172.21.15.31:0/3017810204 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c005680 con 0x55bf88f80ba0 2024-03-21T10:40:08.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.921+0000 7f6a7203f740 1 -- 172.21.15.31:0/3017810204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f80ba0 msgr2=0x55bf88f8e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:08.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.921+0000 7f6a7203f740 1 --2- 172.21.15.31:0/3017810204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f80ba0 0x55bf88f8e4f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c02c780 tx=0x7f6a5c02c7b0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.921+0000 7f6a7203f740 1 -- 172.21.15.31:0/3017810204 shutdown_connections 2024-03-21T10:40:08.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.921+0000 7f6a7203f740 1 --2- 172.21.15.31:0/3017810204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bf88f8ece0 0x55bf88f9a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.921+0000 7f6a7203f740 1 --2- 172.21.15.31:0/3017810204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bf88f814c0 0x55bf88f9a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.921+0000 7f6a7203f740 1 --2- 172.21.15.31:0/3017810204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f80ba0 0x55bf88f8e4f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.921+0000 7f6a7203f740 1 -- 172.21.15.31:0/3017810204 >> 172.21.15.31:0/3017810204 conn(0x55bf88f7bb00 msgr2=0x55bf88f7d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:08.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.922+0000 7f6a7203f740 1 -- 172.21.15.31:0/3017810204 shutdown_connections 2024-03-21T10:40:08.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.922+0000 7f6a7203f740 1 -- 172.21.15.31:0/3017810204 wait complete. 2024-03-21T10:40:08.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.922+0000 7f6a7203f740 1 -- start start 2024-03-21T10:40:08.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.922+0000 7f6a7203f740 1 Processor -- start 2024-03-21T10:40:08.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.923+0000 7f6a7203f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f80ba0 0x55bf88ff2940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:08.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.923+0000 7f6a7203f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bf88f814c0 0x55bf88ff2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:08.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.923+0000 7f6a7203f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bf88f8ece0 0x55bf88f88d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:08.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.923+0000 7f6a7203f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a681aab80 con 0x55bf88f8ece0 2024-03-21T10:40:08.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.923+0000 7f6a7203f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a681abe80 con 0x55bf88f80ba0 2024-03-21T10:40:08.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.923+0000 7f6a7203f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a681ab1c0 con 0x55bf88f814c0 2024-03-21T10:40:08.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.923+0000 7f6a7203f740 10 client.0 _ll_register_callbacks cb 0x7f6a5c042070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:08.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.923+0000 7f6a6ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f80ba0 0x55bf88ff2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:08.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.923+0000 7f6a67e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bf88f814c0 0x55bf88ff2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:08.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.923+0000 7f6a6ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f80ba0 0x55bf88ff2940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:47492/0 (socket says 172.21.15.31:47492) 2024-03-21T10:40:08.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.923+0000 7f6a67e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bf88f814c0 0x55bf88ff2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:59814/0 (socket says 172.21.15.31:59814) 2024-03-21T10:40:08.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.923+0000 7f6a6ca00640 1 -- 172.21.15.31:0/2588191122 learned_addr learned my addr 172.21.15.31:0/2588191122 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:08.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.923+0000 7f6a6d400640 1 --2- 172.21.15.31:0/2588191122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bf88f8ece0 0x55bf88f88d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:08.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.923+0000 7f6a6ca00640 1 -- 172.21.15.31:0/2588191122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bf88f814c0 msgr2=0x55bf88ff2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:08.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.923+0000 7f6a6ca00640 1 --2- 172.21.15.31:0/2588191122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bf88f814c0 0x55bf88ff2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.923+0000 7f6a6ca00640 1 -- 172.21.15.31:0/2588191122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bf88f8ece0 msgr2=0x55bf88f88d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:08.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.923+0000 7f6a6ca00640 1 --2- 172.21.15.31:0/2588191122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bf88f8ece0 0x55bf88f88d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.923+0000 7f6a6ca00640 1 -- 172.21.15.31:0/2588191122 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bf88f89420 con 0x55bf88f80ba0 2024-03-21T10:40:08.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.923+0000 7f6a67e00640 1 --2- 172.21.15.31:0/2588191122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bf88f814c0 0x55bf88ff2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:40:08.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.924+0000 7f6a6ca00640 1 --2- 172.21.15.31:0/2588191122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f80ba0 0x55bf88ff2940 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c006010 tx=0x7f6a5c04bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e57539825053e314 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:08.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.924+0000 7f6a4e000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:40:08.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.924+0000 7f6a4e000640 1 -- 172.21.15.31:0/2588191122 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c065060 con 0x55bf88f80ba0 2024-03-21T10:40:08.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.924+0000 7f6a7203f740 1 -- 172.21.15.31:0/2588191122 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bf88eed0c0 con 0x55bf88f80ba0 2024-03-21T10:40:08.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.924+0000 7f6a4e000640 1 -- 172.21.15.31:0/2588191122 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f6a5c0050d0 con 0x55bf88f80ba0 2024-03-21T10:40:08.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.924+0000 7f6a4e000640 1 -- 172.21.15.31:0/2588191122 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a5c04b5c0 con 0x55bf88f80ba0 2024-03-21T10:40:08.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.924+0000 7f6a7203f740 10 client.26120 fetch_fsmap learned FSMap version 24 2024-03-21T10:40:08.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.924+0000 7f6a7203f740 1 -- 172.21.15.31:0/2588191122 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55bf88ffb1b0 con 0x55bf88f80ba0 2024-03-21T10:40:08.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.924+0000 7f6a4e000640 1 -- 172.21.15.31:0/2588191122 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55bf88eed0c0 con 0x55bf88f80ba0 2024-03-21T10:40:08.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.924+0000 7f6a4e000640 1 -- 172.21.15.31:0/2588191122 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f6a5c068080 con 0x55bf88f80ba0 2024-03-21T10:40:08.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.925+0000 7f6a7203f740 10 client.26120 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:40:08.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.925+0000 7f6a7203f740 10 client.26120 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:40:08.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.925+0000 7f6a7203f740 20 client.26120 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.925+0000 7f6a7203f740 1 --2- 172.21.15.31:0/2588191122 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55bf88ffb760 0x55bf88ffdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.925+0000 7f6a7203f740 4 client.26120 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:40:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.925+0000 7f6a7203f740 1 -- 172.21.15.31:0/2588191122 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55bf88e6b5d0 con 0x55bf88ffb760 2024-03-21T10:40:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.925+0000 7f6a67e00640 1 --2- 172.21.15.31:0/2588191122 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55bf88ffb760 0x55bf88ffdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.925+0000 7f6a67e00640 1 --2- 172.21.15.31:0/2588191122 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55bf88ffb760 0x55bf88ffdb50 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=673f5836c2751567 server_cookie=1cf18e9e79ceb4a8 in_seq=0 out_seq=0 2024-03-21T10:40:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.925+0000 7f6a4e000640 10 client.26120 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:08.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.930+0000 7f6a4e000640 1 -- 172.21.15.31:0/2588191122 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55bf88e6b5d0 con 0x55bf88ffb760 2024-03-21T10:40:08.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.930+0000 7f6a4e000640 10 client.26120 handle_command_reply: tid=0 2024-03-21T10:40:08.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.930+0000 7f6a7203f740 1 client.26120 shutdown 2024-03-21T10:40:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.931+0000 7f6a7203f740 1 -- 172.21.15.31:0/2588191122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f80ba0 msgr2=0x55bf88ff2940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.931+0000 7f6a7203f740 1 --2- 172.21.15.31:0/2588191122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f80ba0 0x55bf88ff2940 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c006010 tx=0x7f6a5c04bd80 comp rx=0 tx=0).stop 2024-03-21T10:40:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.931+0000 7f6a7203f740 1 -- 172.21.15.31:0/2588191122 shutdown_connections 2024-03-21T10:40:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.931+0000 7f6a7203f740 1 --2- 172.21.15.31:0/2588191122 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55bf88ffb760 0x55bf88ffdb50 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.931+0000 7f6a7203f740 1 --2- 172.21.15.31:0/2588191122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bf88f8ece0 0x55bf88f88d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.931+0000 7f6a7203f740 1 --2- 172.21.15.31:0/2588191122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bf88f814c0 0x55bf88ff2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.931+0000 7f6a7203f740 1 --2- 172.21.15.31:0/2588191122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bf88f80ba0 0x55bf88ff2940 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.931+0000 7f6a4e000640 0 client.26120 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.931+0000 7f6a7203f740 1 -- 172.21.15.31:0/2588191122 >> 172.21.15.31:0/2588191122 conn(0x55bf88f7bb00 msgr2=0x55bf88f7a2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.931+0000 7f6a7203f740 1 -- 172.21.15.31:0/2588191122 shutdown_connections 2024-03-21T10:40:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.931+0000 7f6a7203f740 1 -- 172.21.15.31:0/2588191122 wait complete. 2024-03-21T10:40:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.931+0000 7f6a7203f740 20 client.26120 trim_cache size 0 max 16384 2024-03-21T10:40:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:40:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:40:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:40:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:40:08.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.932+0000 7f6a6de00640 1 -- 172.21.15.31:0/707147320 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6a500a30f0 msgr2=0x7f6a500a55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:08.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.932+0000 7f6a6de00640 1 --2- 172.21.15.31:0/707147320 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6a500a30f0 0x7f6a500a55b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c009900 tx=0x7f6a5c000bc0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.932+0000 7f6a6de00640 1 -- 172.21.15.31:0/707147320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a681075b0 msgr2=0x7f6a6819c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:08.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.932+0000 7f6a6de00640 1 --2- 172.21.15.31:0/707147320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a681075b0 0x7f6a6819c5f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009fd0 tx=0x7f6a6000b260 comp rx=0 tx=0).stop 2024-03-21T10:40:08.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.932+0000 7f6a6de00640 1 -- 172.21.15.31:0/707147320 shutdown_connections 2024-03-21T10:40:08.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.932+0000 7f6a6de00640 1 --2- 172.21.15.31:0/707147320 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6a500a30f0 0x7f6a500a55b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.932+0000 7f6a6de00640 1 --2- 172.21.15.31:0/707147320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a681075b0 0x7f6a6819c5f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.932+0000 7f6a6de00640 1 --2- 172.21.15.31:0/707147320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a68069850 0x7f6a68198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.932+0000 7f6a6de00640 1 --2- 172.21.15.31:0/707147320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a68068e90 0x7f6a68197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:08.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.932+0000 7f6a6de00640 1 -- 172.21.15.31:0/707147320 >> 172.21.15.31:0/707147320 conn(0x7f6a68075f00 msgr2=0x7f6a680ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:08.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.932+0000 7f6a6de00640 1 -- 172.21.15.31:0/707147320 shutdown_connections 2024-03-21T10:40:08.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:08.932+0000 7f6a6de00640 1 -- 172.21.15.31:0/707147320 wait complete. 2024-03-21T10:40:09.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:08 smithi031 ceph-mon[31673]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-21T10:40:09.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:08 smithi070 ceph-mon[36330]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-21T10:40:09.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:08 smithi040 ceph-mon[36154]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-21T10:40:09.459 INFO:tasks.workunit.client.0.smithi031.stderr:Updating files: 57% (7042/12271) Updating files: 58% (7118/12271) Updating files: 59% (7240/12271) Updating files: 60% (7363/12271) Updating files: 61% (7486/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 94% (11619/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-21T10:40:10.116 INFO:tasks.workunit.client.0.smithi031.stderr:Note: switching to '2edfee68ad7243bed422a049b6f5f58619300f11'. 2024-03-21T10:40:10.117 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-03-21T10:40:10.117 INFO:tasks.workunit.client.0.smithi031.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-21T10:40:10.117 INFO:tasks.workunit.client.0.smithi031.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-21T10:40:10.117 INFO:tasks.workunit.client.0.smithi031.stderr:state without impacting any branches by switching back to a branch. 2024-03-21T10:40:10.117 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-03-21T10:40:10.117 INFO:tasks.workunit.client.0.smithi031.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-21T10:40:10.117 INFO:tasks.workunit.client.0.smithi031.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-21T10:40:10.117 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-03-21T10:40:10.117 INFO:tasks.workunit.client.0.smithi031.stderr: git switch -c 2024-03-21T10:40:10.117 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-03-21T10:40:10.117 INFO:tasks.workunit.client.0.smithi031.stderr:Or undo this operation with: 2024-03-21T10:40:10.117 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-03-21T10:40:10.117 INFO:tasks.workunit.client.0.smithi031.stderr: git switch - 2024-03-21T10:40:10.117 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-03-21T10:40:10.118 INFO:tasks.workunit.client.0.smithi031.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-21T10:40:10.118 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-03-21T10:40:10.118 INFO:tasks.workunit.client.0.smithi031.stderr:HEAD is now at 2edfee68ad7 Merge PR #55768 into wip-vshankar-testing-20240320.053002 2024-03-21T10:40:10.127 DEBUG:teuthology.orchestra.run.smithi031:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-21T10:40:10.185 INFO:tasks.workunit.client.0.smithi031.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-21T10:40:10.194 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-21T10:40:10.194 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-21T10:40:11.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:10 smithi031 ceph-mon[31673]: pgmap v51: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-21T10:40:11.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:10 smithi070 ceph-mon[36330]: pgmap v51: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-21T10:40:11.173 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-21T10:40:11.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:10 smithi040 ceph-mon[36154]: pgmap v51: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-21T10:40:11.294 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-21T10:40:11.379 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-21T10:40:11.385 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-21T10:40:11.385 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-21T10:40:11.476 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-21T10:40:11.486 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-21T10:40:11.486 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-21T10:40:11.537 INFO:tasks.workunit:Running workunits matching suites/iogen.sh on client.0... 2024-03-21T10:40:11.539 INFO:tasks.workunit:Running workunit suites/iogen.sh... 2024-03-21T10:40:11.539 DEBUG:teuthology.orchestra.run.smithi031:workunit test suites/iogen.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2edfee68ad7243bed422a049b6f5f58619300f11 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iogen.sh 2024-03-21T10:40:11.613 INFO:tasks.workunit.client.0.smithi031.stdout:getting iogen 2024-03-21T10:40:11.613 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo 'getting iogen' 2024-03-21T10:40:11.613 INFO:tasks.workunit.client.0.smithi031.stderr:+ wget http://download.ceph.com/qa/iogen_3.1p0.tar 2024-03-21T10:40:11.664 INFO:tasks.workunit.client.0.smithi031.stderr:--2024-03-21 10:40:11-- http://download.ceph.com/qa/iogen_3.1p0.tar 2024-03-21T10:40:11.665 INFO:tasks.workunit.client.0.smithi031.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-03-21T10:40:11.686 INFO:tasks.workunit.client.0.smithi031.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-03-21T10:40:11.707 INFO:tasks.workunit.client.0.smithi031.stderr:HTTP request sent, awaiting response... 200 OK 2024-03-21T10:40:11.707 INFO:tasks.workunit.client.0.smithi031.stderr:Length: 8071 (7.9K) [application/octet-stream] 2024-03-21T10:40:11.710 INFO:tasks.workunit.client.0.smithi031.stderr:Saving to: ‘iogen_3.1p0.tar’ 2024-03-21T10:40:11.716 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-03-21T10:40:11.716 INFO:tasks.workunit.client.0.smithi031.stderr: 0K ....... 100% 226M=0s 2024-03-21T10:40:11.716 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-03-21T10:40:11.716 INFO:tasks.workunit.client.0.smithi031.stderr:2024-03-21 10:40:11 (226 MB/s) - ‘iogen_3.1p0.tar’ saved [8071/8071] 2024-03-21T10:40:11.716 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-03-21T10:40:11.717 INFO:tasks.workunit.client.0.smithi031.stderr:+ tar -xvzf iogen_3.1p0.tar 2024-03-21T10:40:11.739 INFO:tasks.workunit.client.0.smithi031.stdout:iogen_3.1p0 2024-03-21T10:40:11.740 INFO:tasks.workunit.client.0.smithi031.stdout:iogen_3.1p0/src 2024-03-21T10:40:11.741 INFO:tasks.workunit.client.0.smithi031.stdout:iogen_3.1p0/src/iogen.8 2024-03-21T10:40:11.742 INFO:tasks.workunit.client.0.smithi031.stdout:iogen_3.1p0/src/iogen.c 2024-03-21T10:40:11.743 INFO:tasks.workunit.client.0.smithi031.stdout:iogen_3.1p0/README 2024-03-21T10:40:11.744 INFO:tasks.workunit.client.0.smithi031.stdout:iogen_3.1p0/Makefile.bsd 2024-03-21T10:40:11.745 INFO:tasks.workunit.client.0.smithi031.stdout:iogen_3.1p0/Makefile 2024-03-21T10:40:11.746 INFO:tasks.workunit.client.0.smithi031.stderr:+ cd iogen_3.1p0 2024-03-21T10:40:11.746 INFO:tasks.workunit.client.0.smithi031.stdout:making iogen 2024-03-21T10:40:11.746 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo 'making iogen' 2024-03-21T10:40:11.746 INFO:tasks.workunit.client.0.smithi031.stderr:+ make 2024-03-21T10:40:11.758 INFO:tasks.workunit.client.0.smithi031.stdout:gcc -D_FILE_OFFSET_BITS=64 -DVERSION=\"3.1p0\" -Dstrlcpy=strncpy -Darc4random=rand -O2 src/iogen.c -o iogen 2024-03-21T10:40:12.446 INFO:tasks.workunit.client.0.smithi031.stdout:nroff -mandoc -Tascii src/iogen.8 > iogen.cat8 2024-03-21T10:40:12.591 INFO:tasks.workunit.client.0.smithi031.stdout:running iogen 2024-03-21T10:40:12.591 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo 'running iogen' 2024-03-21T10:40:12.592 INFO:tasks.workunit.client.0.smithi031.stderr:+ ./iogen -n 5 -s 2g 2024-03-21T10:40:12.594 INFO:tasks.workunit.client.0.smithi031.stderr:child spawned 90195 2024-03-21T10:40:12.594 INFO:tasks.workunit.client.0.smithi031.stderr:child spawned 90196 2024-03-21T10:40:12.595 INFO:tasks.workunit.client.0.smithi031.stderr:child spawned 90197 2024-03-21T10:40:12.595 INFO:tasks.workunit.client.0.smithi031.stderr:child spawned 90198 2024-03-21T10:40:12.596 INFO:tasks.workunit.client.0.smithi031.stderr:child spawned 90199 2024-03-21T10:40:12.597 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo 'sleep for 10 min' 2024-03-21T10:40:12.597 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 600 2024-03-21T10:40:12.597 INFO:tasks.workunit.client.0.smithi031.stdout:sleep for 10 min 2024-03-21T10:40:13.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:12 smithi070 ceph-mon[36330]: pgmap v52: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-21T10:40:13.087 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:12 smithi031 ceph-mon[31673]: pgmap v52: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-21T10:40:13.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:12 smithi040 ceph-mon[36154]: pgmap v52: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-21T10:40:15.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:14 smithi031 ceph-mon[31673]: pgmap v53: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-21T10:40:15.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:14 smithi070 ceph-mon[36330]: pgmap v53: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-21T10:40:15.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:14 smithi040 ceph-mon[36154]: pgmap v53: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-21T10:40:17.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:16 smithi031 ceph-mon[31673]: pgmap v54: 129 pgs: 129 active+clean; 78 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 6.4 MiB/s wr, 33 op/s 2024-03-21T10:40:17.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:16 smithi070 ceph-mon[36330]: pgmap v54: 129 pgs: 129 active+clean; 78 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 6.4 MiB/s wr, 33 op/s 2024-03-21T10:40:17.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:16 smithi040 ceph-mon[36154]: pgmap v54: 129 pgs: 129 active+clean; 78 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 6.4 MiB/s wr, 33 op/s 2024-03-21T10:40:19.063 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:18 smithi031 ceph-mon[31673]: pgmap v55: 129 pgs: 129 active+clean; 78 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.4 MiB/s wr, 13 op/s 2024-03-21T10:40:19.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:18 smithi070 ceph-mon[36330]: pgmap v55: 129 pgs: 129 active+clean; 78 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.4 MiB/s wr, 13 op/s 2024-03-21T10:40:19.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:18 smithi040 ceph-mon[36154]: pgmap v55: 129 pgs: 129 active+clean; 78 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.4 MiB/s wr, 13 op/s 2024-03-21T10:40:21.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:20 smithi031 ceph-mon[31673]: pgmap v56: 129 pgs: 129 active+clean; 1.4 GiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 116 MiB/s wr, 45 op/s 2024-03-21T10:40:21.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:20 smithi070 ceph-mon[36330]: pgmap v56: 129 pgs: 129 active+clean; 1.4 GiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 116 MiB/s wr, 45 op/s 2024-03-21T10:40:21.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:20 smithi040 ceph-mon[36154]: pgmap v56: 129 pgs: 129 active+clean; 1.4 GiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 116 MiB/s wr, 45 op/s 2024-03-21T10:40:22.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.108+0000 7f7c13a00640 1 -- 172.21.15.31:0/2561495730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c0c112520 msgr2=0x7f7c0c114910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:22.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.108+0000 7f7c13a00640 1 --2- 172.21.15.31:0/2561495730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c0c112520 0x7f7c0c114910 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc009900 tx=0x7f7bfc02bc40 comp rx=0 tx=0).stop 2024-03-21T10:40:22.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.109+0000 7f7c13a00640 1 -- 172.21.15.31:0/2561495730 shutdown_connections 2024-03-21T10:40:22.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.110+0000 7f7c13a00640 1 --2- 172.21.15.31:0/2561495730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c0c112520 0x7f7c0c114910 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.110+0000 7f7c13a00640 1 --2- 172.21.15.31:0/2561495730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c0c0764f0 0x7f7c0c111fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.110+0000 7f7c13a00640 1 --2- 172.21.15.31:0/2561495730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c0c075bd0 0x7f7c0c075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.111+0000 7f7c13a00640 1 -- 172.21.15.31:0/2561495730 >> 172.21.15.31:0/2561495730 conn(0x7f7c0c0ff390 msgr2=0x7f7c0c1017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.112+0000 7f7c13a00640 1 -- 172.21.15.31:0/2561495730 shutdown_connections 2024-03-21T10:40:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.112+0000 7f7c13a00640 1 -- 172.21.15.31:0/2561495730 wait complete. 2024-03-21T10:40:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.113+0000 7f7c13a00640 1 Processor -- start 2024-03-21T10:40:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.114+0000 7f7c13a00640 1 -- start start 2024-03-21T10:40:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.114+0000 7f7c13a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c0c075bd0 0x7f7c0c111b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.114+0000 7f7c13a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c0c0764f0 0x7f7c0c10ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.114+0000 7f7c13a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c0c112520 0x7f7c0c10b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.114+0000 7f7c13a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c118140 con 0x7f7c0c0764f0 2024-03-21T10:40:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.114+0000 7f7c13a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c117fc0 con 0x7f7c0c075bd0 2024-03-21T10:40:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.114+0000 7f7c13a00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c1182c0 con 0x7f7c0c112520 2024-03-21T10:40:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.115+0000 7f7c11000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c0c075bd0 0x7f7c0c111b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.115+0000 7f7c11000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c0c075bd0 0x7f7c0c111b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:57434/0 (socket says 172.21.15.31:57434) 2024-03-21T10:40:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.115+0000 7f7c11000640 1 -- 172.21.15.31:0/2792991409 learned_addr learned my addr 172.21.15.31:0/2792991409 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.116+0000 7f7c0be00640 1 --2- 172.21.15.31:0/2792991409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c0c0764f0 0x7f7c0c10ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:22.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.119+0000 7f7c11000640 1 -- 172.21.15.31:0/2792991409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c0c112520 msgr2=0x7f7c0c10b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:22.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.119+0000 7f7c11000640 1 --2- 172.21.15.31:0/2792991409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c0c112520 0x7f7c0c10b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.119+0000 7f7c11000640 1 -- 172.21.15.31:0/2792991409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c0c0764f0 msgr2=0x7f7c0c10ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:22.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.119+0000 7f7c11000640 1 --2- 172.21.15.31:0/2792991409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c0c0764f0 0x7f7c0c10ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.119+0000 7f7c11000640 1 -- 172.21.15.31:0/2792991409 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c0c10b820 con 0x7f7c0c075bd0 2024-03-21T10:40:22.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.119+0000 7f7c11000640 1 --2- 172.21.15.31:0/2792991409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c0c075bd0 0x7f7c0c111b50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7c0400ba50 tx=0x7f7c0400bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:22.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.120+0000 7f7c09600640 1 -- 172.21.15.31:0/2792991409 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c04019070 con 0x7f7c0c075bd0 2024-03-21T10:40:22.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.122+0000 7f7c09600640 1 -- 172.21.15.31:0/2792991409 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c0400cbe0 con 0x7f7c0c075bd0 2024-03-21T10:40:22.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.122+0000 7f7c09600640 1 -- 172.21.15.31:0/2792991409 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c040076e0 con 0x7f7c0c075bd0 2024-03-21T10:40:22.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.122+0000 7f7c13a00640 1 -- 172.21.15.31:0/2792991409 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c0c10f6e0 con 0x7f7c0c075bd0 2024-03-21T10:40:22.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.123+0000 7f7c13a00640 1 -- 172.21.15.31:0/2792991409 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c0c10fb80 con 0x7f7c0c075bd0 2024-03-21T10:40:22.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.131+0000 7f7c09600640 1 -- 172.21.15.31:0/2792991409 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f7c0401c070 con 0x7f7c0c075bd0 2024-03-21T10:40:22.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.132+0000 7f7c16604740 1 Processor -- start 2024-03-21T10:40:22.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.132+0000 7f7c16604740 1 -- start start 2024-03-21T10:40:22.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.132+0000 7f7c16604740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4ce29dfd0 0x55c4ce29e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.132+0000 7f7c16604740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4ce29e8f0 0x55c4ce2a7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.132+0000 7f7c16604740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4ce2a76c0 0x55c4ce2a9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.132+0000 7f7c16604740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4ce2603b0 con 0x55c4ce2a76c0 2024-03-21T10:40:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.132+0000 7f7c16604740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4ce202830 con 0x55c4ce29dfd0 2024-03-21T10:40:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.132+0000 7f7c16604740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4ce1ab5d0 con 0x55c4ce29e8f0 2024-03-21T10:40:22.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.133+0000 7f7c0be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4ce2a76c0 0x55c4ce2a9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:22.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.133+0000 7f7c0be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4ce2a76c0 0x55c4ce2a9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49570/0 (socket says 172.21.15.31:49570) 2024-03-21T10:40:22.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.133+0000 7f7c0be00640 1 -- 172.21.15.31:0/2241564055 learned_addr learned my addr 172.21.15.31:0/2241564055 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:22.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.133+0000 7f7c09600640 1 --2- 172.21.15.31:0/2792991409 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7bf00a3210 0x7f7bf00a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:22.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.134+0000 7f7c11a00640 1 --2- 172.21.15.31:0/2241564055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4ce29dfd0 0x55c4ce29e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:22.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.134+0000 7f7c11a00640 1 -- 172.21.15.31:0/2241564055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4ce29e8f0 msgr2=0x55c4ce2a7180 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:40:22.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.134+0000 7f7c11a00640 1 --2- 172.21.15.31:0/2241564055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4ce29e8f0 0x55c4ce2a7180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.134+0000 7f7c11a00640 1 -- 172.21.15.31:0/2241564055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4ce2a76c0 msgr2=0x55c4ce2a9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:22.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.134+0000 7f7c09600640 1 -- 172.21.15.31:0/2792991409 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f7c040cffe0 con 0x7f7c0c075bd0 2024-03-21T10:40:22.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.137+0000 7f7c11a00640 1 --2- 172.21.15.31:0/2241564055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4ce2a76c0 0x55c4ce2a9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.137+0000 7f7c11a00640 1 -- 172.21.15.31:0/2241564055 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4ce1c32c0 con 0x55c4ce29dfd0 2024-03-21T10:40:22.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.137+0000 7f7c0be00640 1 --2- 172.21.15.31:0/2241564055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4ce2a76c0 0x55c4ce2a9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:40:22.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.138+0000 7f7c11a00640 1 --2- 172.21.15.31:0/2241564055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4ce29dfd0 0x55c4ce29e3b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc009850 tx=0x7f7bfc02c750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b854c0df2734e960 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:22.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.138+0000 7f7c11a00640 1 --2- 172.21.15.31:0/2792991409 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7bf00a3210 0x7f7bf00a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:22.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.139+0000 7f7c02000640 1 -- 172.21.15.31:0/2241564055 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc004050 con 0x55c4ce29dfd0 2024-03-21T10:40:22.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.140+0000 7f7c02000640 1 -- 172.21.15.31:0/2241564055 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7bfc0054e0 con 0x55c4ce29dfd0 2024-03-21T10:40:22.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.140+0000 7f7c02000640 1 -- 172.21.15.31:0/2241564055 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc004a90 con 0x55c4ce29dfd0 2024-03-21T10:40:22.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.140+0000 7f7c11a00640 1 --2- 172.21.15.31:0/2792991409 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7bf00a3210 0x7f7bf00a56d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc02c860 tx=0x7f7bfc04b000 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:22.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.141+0000 7f7c16604740 1 -- 172.21.15.31:0/2241564055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4ce29dfd0 msgr2=0x55c4ce29e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:22.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.141+0000 7f7c16604740 1 --2- 172.21.15.31:0/2241564055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4ce29dfd0 0x55c4ce29e3b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc009850 tx=0x7f7bfc02c750 comp rx=0 tx=0).stop 2024-03-21T10:40:22.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.141+0000 7f7c16604740 1 -- 172.21.15.31:0/2241564055 shutdown_connections 2024-03-21T10:40:22.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.141+0000 7f7c16604740 1 --2- 172.21.15.31:0/2241564055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4ce2a76c0 0x55c4ce2a9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.141+0000 7f7c16604740 1 --2- 172.21.15.31:0/2241564055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4ce29e8f0 0x55c4ce2a7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.141+0000 7f7c16604740 1 --2- 172.21.15.31:0/2241564055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4ce29dfd0 0x55c4ce29e3b0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.141+0000 7f7c16604740 1 -- 172.21.15.31:0/2241564055 >> 172.21.15.31:0/2241564055 conn(0x55c4ce2988f0 msgr2=0x55c4ce298cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:22.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.141+0000 7f7c16604740 1 -- 172.21.15.31:0/2241564055 shutdown_connections 2024-03-21T10:40:22.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.141+0000 7f7c16604740 1 -- 172.21.15.31:0/2241564055 wait complete. 2024-03-21T10:40:22.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.142+0000 7f7c16604740 1 -- start start 2024-03-21T10:40:22.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.142+0000 7f7c16604740 1 Processor -- start 2024-03-21T10:40:22.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.144+0000 7f7c16604740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4ce29dfd0 0x55c4ce2a4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:22.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.144+0000 7f7c16604740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4ce29e8f0 0x55c4ce2a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:22.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.144+0000 7f7c16604740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4ce2a76c0 0x55c4ce2a5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:22.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.144+0000 7f7c16604740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4ce1bea20 con 0x55c4ce29dfd0 2024-03-21T10:40:22.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.144+0000 7f7c16604740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4ce1fe8d0 con 0x55c4ce29e8f0 2024-03-21T10:40:22.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.144+0000 7f7c16604740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4ce2aa2f0 con 0x55c4ce2a76c0 2024-03-21T10:40:22.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.144+0000 7f7c16604740 10 client.0 _ll_register_callbacks cb 0x55c4ce1fb1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:22.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.145+0000 7f7c11a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4ce2a76c0 0x55c4ce2a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:22.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.145+0000 7f7c11a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4ce2a76c0 0x55c4ce2a5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:58190/0 (socket says 172.21.15.31:58190) 2024-03-21T10:40:22.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.145+0000 7f7c11a00640 1 -- 172.21.15.31:0/3671859730 learned_addr learned my addr 172.21.15.31:0/3671859730 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:22.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.145+0000 7f7c11a00640 1 -- 172.21.15.31:0/3671859730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4ce29e8f0 msgr2=0x55c4ce2a4d50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:40:22.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.145+0000 7f7c11a00640 1 --2- 172.21.15.31:0/3671859730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4ce29e8f0 0x55c4ce2a4d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.145+0000 7f7c11a00640 1 -- 172.21.15.31:0/3671859730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4ce29dfd0 msgr2=0x55c4ce2a4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:22.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.145+0000 7f7c11a00640 1 --2- 172.21.15.31:0/3671859730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4ce29dfd0 0x55c4ce2a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.145+0000 7f7c11a00640 1 -- 172.21.15.31:0/3671859730 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c4cdf17210 con 0x55c4ce2a76c0 2024-03-21T10:40:22.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.146+0000 7f7c11a00640 1 --2- 172.21.15.31:0/3671859730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4ce2a76c0 0x55c4ce2a5290 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc0048f0 tx=0x7f7bfc004920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe29d5efe0957e38 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:22.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.146+0000 7f7bf6a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:40:22.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.146+0000 7f7bf6a00640 1 -- 172.21.15.31:0/3671859730 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc059060 con 0x55c4ce2a76c0 2024-03-21T10:40:22.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.146+0000 7f7bf6a00640 1 -- 172.21.15.31:0/3671859730 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f7bfc045070 con 0x55c4ce2a76c0 2024-03-21T10:40:22.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.146+0000 7f7c16604740 1 -- 172.21.15.31:0/3671859730 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7bfc0054e0 con 0x55c4ce2a76c0 2024-03-21T10:40:22.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.147+0000 7f7bf6a00640 1 -- 172.21.15.31:0/3671859730 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7bfc004b40 con 0x55c4ce2a76c0 2024-03-21T10:40:22.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.147+0000 7f7c16604740 10 client.26098 fetch_fsmap learned FSMap version 24 2024-03-21T10:40:22.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.147+0000 7f7c16604740 1 -- 172.21.15.31:0/3671859730 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55c4cdfe3620 con 0x55c4ce2a76c0 2024-03-21T10:40:22.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.148+0000 7f7bf6a00640 1 -- 172.21.15.31:0/3671859730 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7bfc0054e0 con 0x55c4ce2a76c0 2024-03-21T10:40:22.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.148+0000 7f7bf6a00640 1 -- 172.21.15.31:0/3671859730 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f7bfc049020 con 0x55c4ce2a76c0 2024-03-21T10:40:22.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.148+0000 7f7c16604740 10 client.26098 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:40:22.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.149+0000 7f7c16604740 10 client.26098 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:40:22.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.149+0000 7f7c16604740 20 client.26098 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:22.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.149+0000 7f7c16604740 1 --2- 172.21.15.31:0/3671859730 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c4ce2b7470 0x55c4ce2b9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:22.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.149+0000 7f7c16604740 4 client.26098 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:40:22.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.149+0000 7f7c16604740 1 -- 172.21.15.31:0/3671859730 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c4ce20fd70 con 0x55c4ce2b7470 2024-03-21T10:40:22.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.151+0000 7f7c11000640 1 --2- 172.21.15.31:0/3671859730 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c4ce2b7470 0x55c4ce2b9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:22.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.152+0000 7f7c11000640 1 --2- 172.21.15.31:0/3671859730 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c4ce2b7470 0x55c4ce2b9860 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2639f8c4370edf2a server_cookie=fe1c6c267f92cd6b in_seq=0 out_seq=0 2024-03-21T10:40:22.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.152+0000 7f7bf6a00640 10 client.26098 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:22.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.152+0000 7f7bf6a00640 1 -- 172.21.15.31:0/3671859730 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c4ce20fd70 con 0x55c4ce2b7470 2024-03-21T10:40:22.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.152+0000 7f7bf6a00640 10 client.26098 handle_command_reply: tid=0 2024-03-21T10:40:22.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.153+0000 7f7c16604740 1 client.26098 shutdown 2024-03-21T10:40:22.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.153+0000 7f7c16604740 1 -- 172.21.15.31:0/3671859730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4ce2a76c0 msgr2=0x55c4ce2a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:22.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.154+0000 7f7c16604740 1 --2- 172.21.15.31:0/3671859730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4ce2a76c0 0x55c4ce2a5290 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc0048f0 tx=0x7f7bfc004920 comp rx=0 tx=0).stop 2024-03-21T10:40:22.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.155+0000 7f7c16604740 1 -- 172.21.15.31:0/3671859730 shutdown_connections 2024-03-21T10:40:22.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.155+0000 7f7c16604740 1 --2- 172.21.15.31:0/3671859730 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c4ce2b7470 0x55c4ce2b9860 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.155+0000 7f7c16604740 1 --2- 172.21.15.31:0/3671859730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4ce2a76c0 0x55c4ce2a5290 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.155+0000 7f7c16604740 1 --2- 172.21.15.31:0/3671859730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4ce29e8f0 0x55c4ce2a4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.155+0000 7f7c16604740 1 --2- 172.21.15.31:0/3671859730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4ce29dfd0 0x55c4ce2a4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.155+0000 7f7c16604740 1 -- 172.21.15.31:0/3671859730 >> 172.21.15.31:0/3671859730 conn(0x55c4ce2988f0 msgr2=0x55c4ce29a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:22.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.155+0000 7f7bf6a00640 0 client.26098 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:22.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.158+0000 7f7c16604740 1 -- 172.21.15.31:0/3671859730 shutdown_connections 2024-03-21T10:40:22.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.158+0000 7f7c16604740 1 -- 172.21.15.31:0/3671859730 wait complete. 2024-03-21T10:40:22.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.158+0000 7f7c16604740 20 client.26098 trim_cache size 0 max 16384 2024-03-21T10:40:22.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.197+0000 7f7c16604740 1 Processor -- start 2024-03-21T10:40:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.197+0000 7f7c16604740 1 -- start start 2024-03-21T10:40:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.197+0000 7f7c16604740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4ce29dfd0 0x55c4ce2a6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.197+0000 7f7c16604740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4ce29e8f0 0x55c4ce2977d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.197+0000 7f7c16604740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4ce2a76c0 0x55c4ce2a1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.198+0000 7f7c16604740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c1d0af0 con 0x55c4ce29dfd0 2024-03-21T10:40:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.198+0000 7f7c16604740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c1ac5e0 con 0x55c4ce29e8f0 2024-03-21T10:40:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.198+0000 7f7c16604740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c04fc90 con 0x55c4ce2a76c0 2024-03-21T10:40:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.202+0000 7f7c11a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4ce2a76c0 0x55c4ce2a1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.202+0000 7f7c11a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4ce2a76c0 0x55c4ce2a1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:58198/0 (socket says 172.21.15.31:58198) 2024-03-21T10:40:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.202+0000 7f7c11a00640 1 -- 172.21.15.31:0/3839105725 learned_addr learned my addr 172.21.15.31:0/3839105725 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.203+0000 7f7c11000640 1 --2- 172.21.15.31:0/3839105725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4ce29dfd0 0x55c4ce2a6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:22.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.203+0000 7f7c11a00640 1 -- 172.21.15.31:0/3839105725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4ce29e8f0 msgr2=0x55c4ce2977d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:40:22.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.207+0000 7f7c11a00640 1 --2- 172.21.15.31:0/3839105725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4ce29e8f0 0x55c4ce2977d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.207+0000 7f7c11a00640 1 -- 172.21.15.31:0/3839105725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4ce29dfd0 msgr2=0x55c4ce2a6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:22.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.207+0000 7f7c11a00640 1 --2- 172.21.15.31:0/3839105725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4ce29dfd0 0x55c4ce2a6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.207+0000 7f7c11a00640 1 -- 172.21.15.31:0/3839105725 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4ce2a2da0 con 0x55c4ce2a76c0 2024-03-21T10:40:22.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.208+0000 7f7c11000640 1 --2- 172.21.15.31:0/3839105725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4ce29dfd0 0x55c4ce2a6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:40:22.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.212+0000 7f7c11a00640 1 --2- 172.21.15.31:0/3839105725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4ce2a76c0 0x55c4ce2a1ad0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc0095d0 tx=0x7f7bfc02c340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c649f9f62c4e7efe server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.213+0000 7f7c01600640 1 -- 172.21.15.31:0/3839105725 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc02c5e0 con 0x55c4ce2a76c0 2024-03-21T10:40:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.217+0000 7f7c01600640 1 -- 172.21.15.31:0/3839105725 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7bfc004990 con 0x55c4ce2a76c0 2024-03-21T10:40:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.217+0000 7f7c01600640 1 -- 172.21.15.31:0/3839105725 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc004fc0 con 0x55c4ce2a76c0 2024-03-21T10:40:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.218+0000 7f7c16604740 1 -- 172.21.15.31:0/3839105725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4ce2a76c0 msgr2=0x55c4ce2a1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.218+0000 7f7c16604740 1 --2- 172.21.15.31:0/3839105725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4ce2a76c0 0x55c4ce2a1ad0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc0095d0 tx=0x7f7bfc02c340 comp rx=0 tx=0).stop 2024-03-21T10:40:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.222+0000 7f7c16604740 1 -- 172.21.15.31:0/3839105725 shutdown_connections 2024-03-21T10:40:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.222+0000 7f7c16604740 1 --2- 172.21.15.31:0/3839105725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4ce2a76c0 0x55c4ce2a1ad0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.222+0000 7f7c16604740 1 --2- 172.21.15.31:0/3839105725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4ce29e8f0 0x55c4ce2977d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.222+0000 7f7c16604740 1 --2- 172.21.15.31:0/3839105725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4ce29dfd0 0x55c4ce2a6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.222+0000 7f7c16604740 1 -- 172.21.15.31:0/3839105725 >> 172.21.15.31:0/3839105725 conn(0x55c4ce2988f0 msgr2=0x55c4ce2970c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:22.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.236+0000 7f7c16604740 1 -- 172.21.15.31:0/3839105725 shutdown_connections 2024-03-21T10:40:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.237+0000 7f7c16604740 1 -- 172.21.15.31:0/3839105725 wait complete. 2024-03-21T10:40:22.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.238+0000 7f7c16604740 1 -- start start 2024-03-21T10:40:22.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.238+0000 7f7c16604740 1 Processor -- start 2024-03-21T10:40:22.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.248+0000 7f7c16604740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4ce29dfd0 0x55c4ce2bc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:22.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.248+0000 7f7c16604740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4ce29e8f0 0x55c4ce2ba840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:22.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.249+0000 7f7c0be00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4ce29e8f0 0x55c4ce2ba840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:22.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.249+0000 7f7c0be00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4ce29e8f0 0x55c4ce2ba840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:58210/0 (socket says 172.21.15.31:58210) 2024-03-21T10:40:22.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.250+0000 7f7c0be00640 1 -- 172.21.15.31:0/2651238330 learned_addr learned my addr 172.21.15.31:0/2651238330 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:22.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.251+0000 7f7c16604740 1 --2- 172.21.15.31:0/2651238330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4ce2a76c0 0x55c4ce2bad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:22.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.251+0000 7f7c16604740 1 -- 172.21.15.31:0/2651238330 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c1aad20 con 0x55c4ce29dfd0 2024-03-21T10:40:22.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.251+0000 7f7c16604740 1 -- 172.21.15.31:0/2651238330 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c1abf80 con 0x55c4ce2a76c0 2024-03-21T10:40:22.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.251+0000 7f7c16604740 1 -- 172.21.15.31:0/2651238330 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c1ab320 con 0x55c4ce29e8f0 2024-03-21T10:40:22.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.251+0000 7f7c0be00640 1 -- 172.21.15.31:0/2651238330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4ce2a76c0 msgr2=0x55c4ce2bad80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:40:22.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.251+0000 7f7c0be00640 1 --2- 172.21.15.31:0/2651238330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4ce2a76c0 0x55c4ce2bad80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.251+0000 7f7c0be00640 1 -- 172.21.15.31:0/2651238330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4ce29dfd0 msgr2=0x55c4ce2bc450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:40:22.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.252+0000 7f7c16604740 10 client.0 _ll_register_callbacks cb 0x7f7bf800b220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:22.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.254+0000 7f7c11000640 1 --2- 172.21.15.31:0/2651238330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4ce29dfd0 0x55c4ce2bc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:22.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.254+0000 7f7c0be00640 1 --2- 172.21.15.31:0/2651238330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4ce29dfd0 0x55c4ce2bc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.254+0000 7f7c0be00640 1 -- 172.21.15.31:0/2651238330 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c4ce2bb3f0 con 0x55c4ce29e8f0 2024-03-21T10:40:22.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.254+0000 7f7c16604740 1 -- 172.21.15.31:0/2651238330 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7bfc004990 con 0x55c4ce29e8f0 2024-03-21T10:40:22.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.254+0000 7f7c0be00640 1 --2- 172.21.15.31:0/2651238330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4ce29e8f0 0x55c4ce2ba840 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x55c4ce299a80 tx=0x7f7bf800d870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43bb0597c7ed705d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:22.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.256+0000 7f7bf6a00640 10 client.26104 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:40:22.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.256+0000 7f7bf6a00640 1 -- 172.21.15.31:0/2651238330 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf8015750 con 0x55c4ce29e8f0 2024-03-21T10:40:22.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.256+0000 7f7bf6a00640 1 -- 172.21.15.31:0/2651238330 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f7bf80158f0 con 0x55c4ce29e8f0 2024-03-21T10:40:22.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.256+0000 7f7bf6a00640 1 -- 172.21.15.31:0/2651238330 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7bf8015b30 con 0x55c4ce29e8f0 2024-03-21T10:40:22.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.257+0000 7f7c16604740 10 client.26104 fetch_fsmap learned FSMap version 24 2024-03-21T10:40:22.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.257+0000 7f7c16604740 1 -- 172.21.15.31:0/2651238330 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({fsmap=24}) v3 -- 0x55c4ce3136f0 con 0x55c4ce29e8f0 2024-03-21T10:40:22.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.258+0000 7f7bf6a00640 1 -- 172.21.15.31:0/2651238330 <== mon.1 v2:172.21.15.70:3300/0 4 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f7bf801f030 con 0x55c4ce29e8f0 2024-03-21T10:40:22.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.258+0000 7f7c16604740 10 client.26104 fetch_fsmap finished waiting for FSMap version 24 2024-03-21T10:40:22.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.258+0000 7f7c16604740 10 client.26104 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:40:22.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.259+0000 7f7c16604740 20 client.26104 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:22.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.259+0000 7f7c16604740 1 --2- 172.21.15.31:0/2651238330 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c4ce313cd0 0x55c4ce316150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:22.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.259+0000 7f7c16604740 4 client.26104 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:40:22.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.259+0000 7f7c16604740 1 -- 172.21.15.31:0/2651238330 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c4ce20a0c0 con 0x55c4ce313cd0 2024-03-21T10:40:22.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.260+0000 7f7c11000640 1 --2- 172.21.15.31:0/2651238330 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c4ce313cd0 0x55c4ce316150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.261+0000 7f7bf6a00640 10 client.26104 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.263+0000 7f7c11000640 1 --2- 172.21.15.31:0/2651238330 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c4ce313cd0 0x55c4ce316150 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebfd9009f8048518 server_cookie=c7902ef9f7badf75 in_seq=0 out_seq=0 2024-03-21T10:40:22.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.386+0000 7f7bf6a00640 1 -- 172.21.15.31:0/2651238330 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c4ce20a0c0 con 0x55c4ce313cd0 2024-03-21T10:40:22.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.386+0000 7f7bf6a00640 10 client.26104 handle_command_reply: tid=0 2024-03-21T10:40:22.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.386+0000 7f7c16604740 1 client.26104 shutdown 2024-03-21T10:40:22.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.386+0000 7f7c16604740 1 -- 172.21.15.31:0/2651238330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4ce29e8f0 msgr2=0x55c4ce2ba840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:22.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.386+0000 7f7c16604740 1 --2- 172.21.15.31:0/2651238330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4ce29e8f0 0x55c4ce2ba840 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x55c4ce299a80 tx=0x7f7bf800d870 comp rx=0 tx=0).stop 2024-03-21T10:40:22.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.386+0000 7f7c16604740 1 -- 172.21.15.31:0/2651238330 shutdown_connections 2024-03-21T10:40:22.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.386+0000 7f7c16604740 1 --2- 172.21.15.31:0/2651238330 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c4ce313cd0 0x55c4ce316150 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.386+0000 7f7c16604740 1 --2- 172.21.15.31:0/2651238330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4ce2a76c0 0x55c4ce2bad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.386+0000 7f7c16604740 1 --2- 172.21.15.31:0/2651238330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c4ce29e8f0 0x55c4ce2ba840 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.386+0000 7f7c16604740 1 --2- 172.21.15.31:0/2651238330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4ce29dfd0 0x55c4ce2bc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.386+0000 7f7c16604740 1 -- 172.21.15.31:0/2651238330 >> 172.21.15.31:0/2651238330 conn(0x55c4ce2988f0 msgr2=0x55c4ce297950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:22.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.386+0000 7f7bf6a00640 0 client.26104 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:22.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.387+0000 7f7c16604740 1 -- 172.21.15.31:0/2651238330 shutdown_connections 2024-03-21T10:40:22.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.387+0000 7f7c16604740 1 -- 172.21.15.31:0/2651238330 wait complete. 2024-03-21T10:40:22.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.387+0000 7f7c16604740 20 client.26104 trim_cache size 0 max 16384 2024-03-21T10:40:22.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:40:22.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:40:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:40:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:40:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.390+0000 7f7c13a00640 1 -- 172.21.15.31:0/2792991409 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7bf00a3210 msgr2=0x7f7bf00a56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.390+0000 7f7c13a00640 1 --2- 172.21.15.31:0/2792991409 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7bf00a3210 0x7f7bf00a56d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc02c860 tx=0x7f7bfc04b000 comp rx=0 tx=0).stop 2024-03-21T10:40:22.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.390+0000 7f7c13a00640 1 -- 172.21.15.31:0/2792991409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c0c075bd0 msgr2=0x7f7c0c111b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:22.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.390+0000 7f7c13a00640 1 --2- 172.21.15.31:0/2792991409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c0c075bd0 0x7f7c0c111b50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7c0400ba50 tx=0x7f7c0400bf10 comp rx=0 tx=0).stop 2024-03-21T10:40:22.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.391+0000 7f7c13a00640 1 -- 172.21.15.31:0/2792991409 shutdown_connections 2024-03-21T10:40:22.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.391+0000 7f7c13a00640 1 --2- 172.21.15.31:0/2792991409 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7bf00a3210 0x7f7bf00a56d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.391+0000 7f7c13a00640 1 --2- 172.21.15.31:0/2792991409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c0c112520 0x7f7c0c10b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.391+0000 7f7c13a00640 1 --2- 172.21.15.31:0/2792991409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c0c0764f0 0x7f7c0c10ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.391+0000 7f7c13a00640 1 --2- 172.21.15.31:0/2792991409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c0c075bd0 0x7f7c0c111b50 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:22.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.391+0000 7f7c13a00640 1 -- 172.21.15.31:0/2792991409 >> 172.21.15.31:0/2792991409 conn(0x7f7c0c0ff390 msgr2=0x7f7c0c113bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:22.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.391+0000 7f7c13a00640 1 -- 172.21.15.31:0/2792991409 shutdown_connections 2024-03-21T10:40:22.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:22.391+0000 7f7c13a00640 1 -- 172.21.15.31:0/2792991409 wait complete. 2024-03-21T10:40:23.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:22 smithi031 ceph-mon[31673]: pgmap v57: 129 pgs: 129 active+clean; 1.4 GiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 116 MiB/s wr, 42 op/s 2024-03-21T10:40:23.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:22 smithi070 ceph-mon[36330]: pgmap v57: 129 pgs: 129 active+clean; 1.4 GiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 116 MiB/s wr, 42 op/s 2024-03-21T10:40:23.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:22 smithi040 ceph-mon[36154]: pgmap v57: 129 pgs: 129 active+clean; 1.4 GiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 116 MiB/s wr, 42 op/s 2024-03-21T10:40:25.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:24 smithi031 ceph-mon[31673]: pgmap v58: 129 pgs: 129 active+clean; 1.4 GiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 116 MiB/s wr, 42 op/s 2024-03-21T10:40:25.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:24 smithi070 ceph-mon[36330]: pgmap v58: 129 pgs: 129 active+clean; 1.4 GiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 116 MiB/s wr, 42 op/s 2024-03-21T10:40:25.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:24 smithi040 ceph-mon[36154]: pgmap v58: 129 pgs: 129 active+clean; 1.4 GiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 116 MiB/s wr, 42 op/s 2024-03-21T10:40:27.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:26 smithi031 ceph-mon[31673]: pgmap v59: 129 pgs: 129 active+clean; 2.7 GiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 227 MiB/s wr, 82 op/s 2024-03-21T10:40:27.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:26 smithi070 ceph-mon[36330]: pgmap v59: 129 pgs: 129 active+clean; 2.7 GiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 227 MiB/s wr, 82 op/s 2024-03-21T10:40:27.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:26 smithi040 ceph-mon[36154]: pgmap v59: 129 pgs: 129 active+clean; 2.7 GiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 227 MiB/s wr, 82 op/s 2024-03-21T10:40:28.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:27 smithi031 ceph-mon[31673]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:40:28.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:27 smithi070 ceph-mon[36330]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:40:28.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:27 smithi040 ceph-mon[36154]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:40:29.074 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:28 smithi070 ceph-mon[36330]: pgmap v60: 129 pgs: 129 active+clean; 2.7 GiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 220 MiB/s wr, 72 op/s 2024-03-21T10:40:29.134 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:28 smithi031 ceph-mon[31673]: pgmap v60: 129 pgs: 129 active+clean; 2.7 GiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 220 MiB/s wr, 72 op/s 2024-03-21T10:40:29.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:28 smithi040 ceph-mon[36154]: pgmap v60: 129 pgs: 129 active+clean; 2.7 GiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 220 MiB/s wr, 72 op/s 2024-03-21T10:40:31.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:30 smithi040 ceph-mon[36154]: pgmap v61: 129 pgs: 129 active+clean; 3.9 GiB data, 895 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 330 MiB/s wr, 104 op/s 2024-03-21T10:40:31.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:30 smithi031 ceph-mon[31673]: pgmap v61: 129 pgs: 129 active+clean; 3.9 GiB data, 895 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 330 MiB/s wr, 104 op/s 2024-03-21T10:40:31.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:30 smithi070 ceph-mon[36330]: pgmap v61: 129 pgs: 129 active+clean; 3.9 GiB data, 895 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 330 MiB/s wr, 104 op/s 2024-03-21T10:40:33.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:32 smithi040 ceph-mon[36154]: pgmap v62: 129 pgs: 129 active+clean; 3.9 GiB data, 895 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 220 MiB/s wr, 72 op/s 2024-03-21T10:40:33.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:32 smithi031 ceph-mon[31673]: pgmap v62: 129 pgs: 129 active+clean; 3.9 GiB data, 895 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 220 MiB/s wr, 72 op/s 2024-03-21T10:40:33.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:32 smithi070 ceph-mon[36330]: pgmap v62: 129 pgs: 129 active+clean; 3.9 GiB data, 895 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 220 MiB/s wr, 72 op/s 2024-03-21T10:40:33.604 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:40:33.605 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:40:33.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.940+0000 7fcf99800640 1 -- 172.21.15.31:0/2236639852 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf940ff710 msgr2=0x7fcf940ffb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:33.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.940+0000 7fcf99800640 1 --2- 172.21.15.31:0/2236639852 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf940ff710 0x7fcf940ffb90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fcf84009890 tx=0x7fcf8402bb70 comp rx=0 tx=0).stop 2024-03-21T10:40:33.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.940+0000 7fcf99800640 1 -- 172.21.15.31:0/2236639852 shutdown_connections 2024-03-21T10:40:33.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.940+0000 7fcf99800640 1 --2- 172.21.15.31:0/2236639852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf941000d0 0x7fcf9410c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:33.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.940+0000 7fcf99800640 1 --2- 172.21.15.31:0/2236639852 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf940ff710 0x7fcf940ffb90 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:33.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.940+0000 7fcf99800640 1 --2- 172.21.15.31:0/2236639852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf941056a0 0x7fcf94105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:33.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.940+0000 7fcf99800640 1 -- 172.21.15.31:0/2236639852 >> 172.21.15.31:0/2236639852 conn(0x7fcf940fb840 msgr2=0x7fcf940fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:33.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.940+0000 7fcf99800640 1 -- 172.21.15.31:0/2236639852 shutdown_connections 2024-03-21T10:40:33.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.941+0000 7fcf99800640 1 -- 172.21.15.31:0/2236639852 wait complete. 2024-03-21T10:40:33.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.942+0000 7fcf99800640 1 Processor -- start 2024-03-21T10:40:33.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.942+0000 7fcf99800640 1 -- start start 2024-03-21T10:40:33.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.942+0000 7fcf99800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf940ff710 0x7fcf941a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:33.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.942+0000 7fcf99800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf941000d0 0x7fcf941a35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:33.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.942+0000 7fcf99800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf941056a0 0x7fcf9410b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:33.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.942+0000 7fcf99800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf941118c0 con 0x7fcf940ff710 2024-03-21T10:40:33.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.942+0000 7fcf99800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf94111740 con 0x7fcf941056a0 2024-03-21T10:40:33.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.942+0000 7fcf99800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf94111a40 con 0x7fcf941000d0 2024-03-21T10:40:33.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.942+0000 7fcf93e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf940ff710 0x7fcf941a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:33.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.943+0000 7fcf93e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf940ff710 0x7fcf941a3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39568/0 (socket says 172.21.15.31:39568) 2024-03-21T10:40:33.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.943+0000 7fcf93e00640 1 -- 172.21.15.31:0/3207268530 learned_addr learned my addr 172.21.15.31:0/3207268530 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:33.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.943+0000 7fcf93400640 1 --2- 172.21.15.31:0/3207268530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf941000d0 0x7fcf941a35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:33.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.943+0000 7fcf93400640 1 -- 172.21.15.31:0/3207268530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf941056a0 msgr2=0x7fcf9410b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:33.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.943+0000 7fcf98e00640 1 --2- 172.21.15.31:0/3207268530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf941056a0 0x7fcf9410b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:33.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.943+0000 7fcf93400640 1 --2- 172.21.15.31:0/3207268530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf941056a0 0x7fcf9410b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:33.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.943+0000 7fcf93400640 1 -- 172.21.15.31:0/3207268530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf940ff710 msgr2=0x7fcf941a3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:33.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.943+0000 7fcf93400640 1 --2- 172.21.15.31:0/3207268530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf940ff710 0x7fcf941a3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:33.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.943+0000 7fcf93400640 1 -- 172.21.15.31:0/3207268530 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9410bee0 con 0x7fcf941000d0 2024-03-21T10:40:33.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.943+0000 7fcf93e00640 1 --2- 172.21.15.31:0/3207268530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf940ff710 0x7fcf941a3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:33.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.943+0000 7fcf98e00640 1 --2- 172.21.15.31:0/3207268530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf941056a0 0x7fcf9410b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:40:33.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.944+0000 7fcf93400640 1 --2- 172.21.15.31:0/3207268530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf941000d0 0x7fcf941a35a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcf84000c00 tx=0x7fcf840045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:33.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.944+0000 7fcf90c00640 1 -- 172.21.15.31:0/3207268530 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf84041020 con 0x7fcf941000d0 2024-03-21T10:40:33.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.944+0000 7fcf90c00640 1 -- 172.21.15.31:0/3207268530 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf8402c9f0 con 0x7fcf941000d0 2024-03-21T10:40:33.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.944+0000 7fcf90c00640 1 -- 172.21.15.31:0/3207268530 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf84039040 con 0x7fcf941000d0 2024-03-21T10:40:33.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.944+0000 7fcf99800640 1 -- 172.21.15.31:0/3207268530 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf9410c170 con 0x7fcf941000d0 2024-03-21T10:40:33.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.944+0000 7fcf99800640 1 -- 172.21.15.31:0/3207268530 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf94196d90 con 0x7fcf941000d0 2024-03-21T10:40:33.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.948+0000 7fcf9db82740 1 Processor -- start 2024-03-21T10:40:33.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.948+0000 7fcf90c00640 1 -- 172.21.15.31:0/3207268530 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fcf84044020 con 0x7fcf941000d0 2024-03-21T10:40:33.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.948+0000 7fcf9db82740 1 -- start start 2024-03-21T10:40:33.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.949+0000 7fcf9db82740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55999942cdd0 0x55999942bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:33.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.949+0000 7fcf9db82740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55999942c0f0 0x55999942c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:33.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.949+0000 7fcf9db82740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559999435460 0x559999437850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:33.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.949+0000 7fcf9db82740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559999390250 con 0x55999942cdd0 2024-03-21T10:40:33.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.949+0000 7fcf9db82740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x559999338a30 con 0x559999435460 2024-03-21T10:40:33.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.949+0000 7fcf9db82740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55999938c2f0 con 0x55999942c0f0 2024-03-21T10:40:33.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.949+0000 7fcf98e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55999942cdd0 0x55999942bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:33.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.949+0000 7fcf98e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55999942cdd0 0x55999942bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39582/0 (socket says 172.21.15.31:39582) 2024-03-21T10:40:33.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.949+0000 7fcf98e00640 1 -- 172.21.15.31:0/3345627662 learned_addr learned my addr 172.21.15.31:0/3345627662 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:33.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.950+0000 7fcf93400640 1 --2- 172.21.15.31:0/3345627662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559999435460 0x559999437850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:33.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.950+0000 7fcf93e00640 1 --2- 172.21.15.31:0/3345627662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55999942c0f0 0x55999942c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:33.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.950+0000 7fcf98e00640 1 -- 172.21.15.31:0/3345627662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55999942c0f0 msgr2=0x55999942c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:33.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.950+0000 7fcf98e00640 1 --2- 172.21.15.31:0/3345627662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55999942c0f0 0x55999942c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:33.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.950+0000 7fcf98e00640 1 -- 172.21.15.31:0/3345627662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559999435460 msgr2=0x559999437850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:33.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.950+0000 7fcf98e00640 1 --2- 172.21.15.31:0/3345627662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559999435460 0x559999437850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:33.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.950+0000 7fcf98e00640 1 -- 172.21.15.31:0/3345627662 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559999323690 con 0x55999942cdd0 2024-03-21T10:40:33.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.950+0000 7fcf93e00640 1 --2- 172.21.15.31:0/3345627662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55999942c0f0 0x55999942c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:33.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.950+0000 7fcf93400640 1 --2- 172.21.15.31:0/3345627662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559999435460 0x559999437850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:40:33.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.951+0000 7fcf98e00640 1 --2- 172.21.15.31:0/3345627662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55999942cdd0 0x55999942bb20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c007e80 tx=0x7fcf8c00c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=307169ba09e372c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:33.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.951+0000 7fcf90c00640 1 --2- 172.21.15.31:0/3207268530 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fcf780a3150 0x7fcf780a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:33.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.951+0000 7fcf89600640 1 -- 172.21.15.31:0/3345627662 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c014070 con 0x55999942cdd0 2024-03-21T10:40:33.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.951+0000 7fcf89600640 1 -- 172.21.15.31:0/3345627662 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf8c00fd30 con 0x55999942cdd0 2024-03-21T10:40:33.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.951+0000 7fcf90c00640 1 -- 172.21.15.31:0/3207268530 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fcf840f1370 con 0x7fcf941000d0 2024-03-21T10:40:33.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.952+0000 7fcf89600640 1 -- 172.21.15.31:0/3345627662 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c00eca0 con 0x55999942cdd0 2024-03-21T10:40:33.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.952+0000 7fcf9db82740 1 -- 172.21.15.31:0/3345627662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55999942cdd0 msgr2=0x55999942bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:33.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.952+0000 7fcf9db82740 1 --2- 172.21.15.31:0/3345627662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55999942cdd0 0x55999942bb20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c007e80 tx=0x7fcf8c00c690 comp rx=0 tx=0).stop 2024-03-21T10:40:33.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.952+0000 7fcf93e00640 1 --2- 172.21.15.31:0/3207268530 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fcf780a3150 0x7fcf780a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:33.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.953+0000 7fcf9db82740 1 -- 172.21.15.31:0/3345627662 shutdown_connections 2024-03-21T10:40:33.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.953+0000 7fcf9db82740 1 --2- 172.21.15.31:0/3345627662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559999435460 0x559999437850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:33.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.953+0000 7fcf9db82740 1 --2- 172.21.15.31:0/3345627662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55999942c0f0 0x55999942c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:33.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.953+0000 7fcf9db82740 1 --2- 172.21.15.31:0/3345627662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55999942cdd0 0x55999942bb20 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:33.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.953+0000 7fcf9db82740 1 -- 172.21.15.31:0/3345627662 >> 172.21.15.31:0/3345627662 conn(0x5599993f3200 msgr2=0x559999426b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:33.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.953+0000 7fcf9db82740 1 -- 172.21.15.31:0/3345627662 shutdown_connections 2024-03-21T10:40:33.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.953+0000 7fcf9db82740 1 -- 172.21.15.31:0/3345627662 wait complete. 2024-03-21T10:40:33.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.954+0000 7fcf9db82740 1 -- start start 2024-03-21T10:40:33.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.954+0000 7fcf9db82740 1 Processor -- start 2024-03-21T10:40:33.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.954+0000 7fcf93e00640 1 --2- 172.21.15.31:0/3207268530 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fcf780a3150 0x7fcf780a5610 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcf80007320 tx=0x7fcf8000c040 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:33.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.955+0000 7fcf9db82740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55999942c0f0 0x5599994349b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:33.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.955+0000 7fcf9db82740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55999942cdd0 0x559999431db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:33.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.955+0000 7fcf9db82740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559999435460 0x5599994322f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:33.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.955+0000 7fcf93400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55999942cdd0 0x559999431db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:33.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.955+0000 7fcf93e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55999942c0f0 0x5599994349b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:33.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.955+0000 7fcf93e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55999942c0f0 0x5599994349b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34396/0 (socket says 172.21.15.31:34396) 2024-03-21T10:40:33.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.955+0000 7fcf93e00640 1 -- 172.21.15.31:0/335106012 learned_addr learned my addr 172.21.15.31:0/335106012 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:33.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.955+0000 7fcf9db82740 1 -- 172.21.15.31:0/335106012 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559999438370 con 0x559999435460 2024-03-21T10:40:33.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.955+0000 7fcf9db82740 1 -- 172.21.15.31:0/335106012 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55999934be80 con 0x55999942cdd0 2024-03-21T10:40:33.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.955+0000 7fcf9db82740 1 -- 172.21.15.31:0/335106012 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599994384f0 con 0x55999942c0f0 2024-03-21T10:40:33.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.955+0000 7fcf9db82740 10 client.0 _ll_register_callbacks cb 0x55999935e130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:33.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.955+0000 7fcf93400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55999942cdd0 0x559999431db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:48804/0 (socket says 172.21.15.31:48804) 2024-03-21T10:40:33.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.956+0000 7fcf98e00640 1 --2- 172.21.15.31:0/335106012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559999435460 0x5599994322f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:33.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.956+0000 7fcf93e00640 1 -- 172.21.15.31:0/335106012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55999942cdd0 msgr2=0x559999431db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:33.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.956+0000 7fcf93e00640 1 --2- 172.21.15.31:0/335106012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55999942cdd0 0x559999431db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:33.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.956+0000 7fcf93e00640 1 -- 172.21.15.31:0/335106012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559999435460 msgr2=0x5599994322f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:33.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.956+0000 7fcf93e00640 1 --2- 172.21.15.31:0/335106012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559999435460 0x5599994322f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:33.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.956+0000 7fcf93e00640 1 -- 172.21.15.31:0/335106012 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55999916e7d0 con 0x55999942c0f0 2024-03-21T10:40:33.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.956+0000 7fcf93e00640 1 --2- 172.21.15.31:0/335106012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55999942c0f0 0x5599994349b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcf800169f0 tx=0x7fcf80004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d227e5bfaa1ccfe9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:33.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.956+0000 7fcf7e000640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:40:33.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.956+0000 7fcf7e000640 1 -- 172.21.15.31:0/335106012 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8001d050 con 0x55999942c0f0 2024-03-21T10:40:33.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.956+0000 7fcf7e000640 1 -- 172.21.15.31:0/335106012 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fcf80004020 con 0x55999942c0f0 2024-03-21T10:40:33.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.956+0000 7fcf9db82740 1 -- 172.21.15.31:0/335106012 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf8c00fd30 con 0x55999942c0f0 2024-03-21T10:40:33.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.957+0000 7fcf7e000640 1 -- 172.21.15.31:0/335106012 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf80004220 con 0x55999942c0f0 2024-03-21T10:40:33.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.957+0000 7fcf9db82740 10 client.26116 fetch_fsmap learned FSMap version 25 2024-03-21T10:40:33.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.957+0000 7fcf9db82740 1 -- 172.21.15.31:0/335106012 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5599990a43f0 con 0x55999942c0f0 2024-03-21T10:40:33.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.957+0000 7fcf7e000640 1 -- 172.21.15.31:0/335106012 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf8c00fd30 con 0x55999942c0f0 2024-03-21T10:40:33.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.958+0000 7fcf7e000640 1 -- 172.21.15.31:0/335106012 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fcf80003820 con 0x55999942c0f0 2024-03-21T10:40:33.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.958+0000 7fcf9db82740 10 client.26116 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:40:33.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.958+0000 7fcf9db82740 10 client.26116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:40:33.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.958+0000 7fcf9db82740 20 client.26116 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:33.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.958+0000 7fcf9db82740 1 --2- 172.21.15.31:0/335106012 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5599994453e0 0x5599994477d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:33.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.958+0000 7fcf9db82740 4 client.26116 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:40:33.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.958+0000 7fcf9db82740 1 -- 172.21.15.31:0/335106012 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55999939d790 con 0x5599994453e0 2024-03-21T10:40:33.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.958+0000 7fcf93400640 1 --2- 172.21.15.31:0/335106012 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5599994453e0 0x5599994477d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:33.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.959+0000 7fcf93400640 1 --2- 172.21.15.31:0/335106012 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5599994453e0 0x5599994477d0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b09a8cd022d78b5 server_cookie=97deab01d7e1dbe7 in_seq=0 out_seq=0 2024-03-21T10:40:33.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.959+0000 7fcf7e000640 10 client.26116 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:33.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.960+0000 7fcf7e000640 1 -- 172.21.15.31:0/335106012 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55999939d790 con 0x5599994453e0 2024-03-21T10:40:33.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.960+0000 7fcf7e000640 10 client.26116 handle_command_reply: tid=0 2024-03-21T10:40:33.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.960+0000 7fcf9db82740 1 client.26116 shutdown 2024-03-21T10:40:33.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.961+0000 7fcf9db82740 1 -- 172.21.15.31:0/335106012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55999942c0f0 msgr2=0x5599994349b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:33.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.961+0000 7fcf9db82740 1 --2- 172.21.15.31:0/335106012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55999942c0f0 0x5599994349b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcf800169f0 tx=0x7fcf80004730 comp rx=0 tx=0).stop 2024-03-21T10:40:33.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.961+0000 7fcf9db82740 1 -- 172.21.15.31:0/335106012 shutdown_connections 2024-03-21T10:40:33.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.961+0000 7fcf9db82740 1 --2- 172.21.15.31:0/335106012 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5599994453e0 0x5599994477d0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:33.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.961+0000 7fcf9db82740 1 --2- 172.21.15.31:0/335106012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559999435460 0x5599994322f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:33.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.961+0000 7fcf9db82740 1 --2- 172.21.15.31:0/335106012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55999942cdd0 0x559999431db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:33.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.961+0000 7fcf9db82740 1 --2- 172.21.15.31:0/335106012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55999942c0f0 0x5599994349b0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:33.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.961+0000 7fcf9db82740 1 -- 172.21.15.31:0/335106012 >> 172.21.15.31:0/335106012 conn(0x5599993f3200 msgr2=0x559999428890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:33.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.961+0000 7fcf7e000640 0 client.26116 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:33.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.965+0000 7fcf9db82740 1 -- 172.21.15.31:0/335106012 shutdown_connections 2024-03-21T10:40:33.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.965+0000 7fcf9db82740 1 -- 172.21.15.31:0/335106012 wait complete. 2024-03-21T10:40:33.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.966+0000 7fcf9db82740 20 client.26116 trim_cache size 0 max 16384 2024-03-21T10:40:33.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.998+0000 7fcf9db82740 1 Processor -- start 2024-03-21T10:40:34.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.998+0000 7fcf9db82740 1 -- start start 2024-03-21T10:40:34.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.999+0000 7fcf9db82740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55999942cdd0 0x559999436850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:34.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.999+0000 7fcf9db82740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559999435460 0x5599994306a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:34.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.999+0000 7fcf9db82740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55999944d0c0 0x559999430be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:34.009 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:40:34.009 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-03-21T10:40:34.009 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-03-21T10:40:34.009 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:40:34.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.999+0000 7fcf9db82740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf941bfb00 con 0x55999944d0c0 2024-03-21T10:40:34.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.999+0000 7fcf9db82740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf941bd600 con 0x55999942cdd0 2024-03-21T10:40:34.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.999+0000 7fcf9db82740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9410c650 con 0x559999435460 2024-03-21T10:40:34.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.999+0000 7fcf93e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55999942cdd0 0x559999436850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:34.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.999+0000 7fcf93e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55999942cdd0 0x559999436850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:48816/0 (socket says 172.21.15.31:48816) 2024-03-21T10:40:34.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:33.999+0000 7fcf93e00640 1 -- 172.21.15.31:0/703247526 learned_addr learned my addr 172.21.15.31:0/703247526 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:34.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.000+0000 7fcf93400640 1 --2- 172.21.15.31:0/703247526 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559999435460 0x5599994306a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:34.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.000+0000 7fcf98e00640 1 --2- 172.21.15.31:0/703247526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55999944d0c0 0x559999430be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:34.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.000+0000 7fcf93e00640 1 -- 172.21.15.31:0/703247526 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559999435460 msgr2=0x5599994306a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:34.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.000+0000 7fcf93e00640 1 --2- 172.21.15.31:0/703247526 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559999435460 0x5599994306a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:34.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.000+0000 7fcf93e00640 1 -- 172.21.15.31:0/703247526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55999944d0c0 msgr2=0x559999430be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:34.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.000+0000 7fcf93e00640 1 --2- 172.21.15.31:0/703247526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55999944d0c0 0x559999430be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:34.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.000+0000 7fcf93e00640 1 -- 172.21.15.31:0/703247526 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5599990640f0 con 0x55999942cdd0 2024-03-21T10:40:34.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.000+0000 7fcf98e00640 1 --2- 172.21.15.31:0/703247526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55999944d0c0 0x559999430be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:40:34.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.000+0000 7fcf93400640 1 --2- 172.21.15.31:0/703247526 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559999435460 0x5599994306a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:34.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.000+0000 7fcf93e00640 1 --2- 172.21.15.31:0/703247526 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55999942cdd0 0x559999436850 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fcf800169c0 tx=0x7fcf8000a4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b773d1d4cbdd6ff0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:34.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.001+0000 7fcf88c00640 1 -- 172.21.15.31:0/703247526 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf80039070 con 0x55999942cdd0 2024-03-21T10:40:34.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.001+0000 7fcf88c00640 1 -- 172.21.15.31:0/703247526 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf8001e040 con 0x55999942cdd0 2024-03-21T10:40:34.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.001+0000 7fcf88c00640 1 -- 172.21.15.31:0/703247526 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8001d040 con 0x55999942cdd0 2024-03-21T10:40:34.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.001+0000 7fcf9db82740 1 -- 172.21.15.31:0/703247526 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55999942cdd0 msgr2=0x559999436850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:34.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.001+0000 7fcf9db82740 1 --2- 172.21.15.31:0/703247526 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55999942cdd0 0x559999436850 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fcf800169c0 tx=0x7fcf8000a4c0 comp rx=0 tx=0).stop 2024-03-21T10:40:34.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.002+0000 7fcf9db82740 1 -- 172.21.15.31:0/703247526 shutdown_connections 2024-03-21T10:40:34.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.002+0000 7fcf9db82740 1 --2- 172.21.15.31:0/703247526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55999944d0c0 0x559999430be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:34.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.002+0000 7fcf9db82740 1 --2- 172.21.15.31:0/703247526 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559999435460 0x5599994306a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:34.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.002+0000 7fcf9db82740 1 --2- 172.21.15.31:0/703247526 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55999942cdd0 0x559999436850 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:34.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.002+0000 7fcf9db82740 1 -- 172.21.15.31:0/703247526 >> 172.21.15.31:0/703247526 conn(0x5599993f3200 msgr2=0x559999426670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:34.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.002+0000 7fcf9db82740 1 -- 172.21.15.31:0/703247526 shutdown_connections 2024-03-21T10:40:34.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.002+0000 7fcf9db82740 1 -- 172.21.15.31:0/703247526 wait complete. 2024-03-21T10:40:34.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.002+0000 7fcf9db82740 1 -- start start 2024-03-21T10:40:34.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.003+0000 7fcf9db82740 1 Processor -- start 2024-03-21T10:40:34.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.003+0000 7fcf9db82740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55999942cdd0 0x5599994aa400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:34.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.003+0000 7fcf9db82740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559999435460 0x5599994aa940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:34.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.003+0000 7fcf9db82740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55999944d0c0 0x55999943a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:34.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.003+0000 7fcf9db82740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9419f2b0 con 0x559999435460 2024-03-21T10:40:34.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.003+0000 7fcf9db82740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf941bcfa0 con 0x55999942cdd0 2024-03-21T10:40:34.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.003+0000 7fcf9db82740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf941bc2e0 con 0x55999944d0c0 2024-03-21T10:40:34.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.003+0000 7fcf9db82740 10 client.0 _ll_register_callbacks cb 0x7fcf8000a2c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:34.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.003+0000 7fcf93e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55999942cdd0 0x5599994aa400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:34.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.004+0000 7fcf93e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55999942cdd0 0x5599994aa400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:48826/0 (socket says 172.21.15.31:48826) 2024-03-21T10:40:34.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.004+0000 7fcf93e00640 1 -- 172.21.15.31:0/2430973343 learned_addr learned my addr 172.21.15.31:0/2430973343 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:34.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.004+0000 7fcf98e00640 1 --2- 172.21.15.31:0/2430973343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55999944d0c0 0x55999943a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:34.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.004+0000 7fcf93400640 1 --2- 172.21.15.31:0/2430973343 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559999435460 0x5599994aa940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:34.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.004+0000 7fcf93e00640 1 -- 172.21.15.31:0/2430973343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55999944d0c0 msgr2=0x55999943a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:34.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.004+0000 7fcf93e00640 1 --2- 172.21.15.31:0/2430973343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55999944d0c0 0x55999943a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:34.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.004+0000 7fcf93e00640 1 -- 172.21.15.31:0/2430973343 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559999435460 msgr2=0x5599994aa940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:34.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.004+0000 7fcf93e00640 1 --2- 172.21.15.31:0/2430973343 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559999435460 0x5599994aa940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:34.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.004+0000 7fcf93e00640 1 -- 172.21.15.31:0/2430973343 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55999943d8f0 con 0x55999942cdd0 2024-03-21T10:40:34.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.005+0000 7fcf93e00640 1 --2- 172.21.15.31:0/2430973343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55999942cdd0 0x5599994aa400 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcf8000c010 tx=0x7fcf80022440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=826fe469763c12b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:34.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.005+0000 7fcf7e000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:40:34.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.005+0000 7fcf7e000640 1 -- 172.21.15.31:0/2430973343 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf80039070 con 0x55999942cdd0 2024-03-21T10:40:34.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.005+0000 7fcf7e000640 1 -- 172.21.15.31:0/2430973343 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fcf80004020 con 0x55999942cdd0 2024-03-21T10:40:34.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.005+0000 7fcf9db82740 1 -- 172.21.15.31:0/2430973343 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf8001e040 con 0x55999942cdd0 2024-03-21T10:40:34.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.005+0000 7fcf7e000640 1 -- 172.21.15.31:0/2430973343 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf8001d040 con 0x55999942cdd0 2024-03-21T10:40:34.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.006+0000 7fcf9db82740 10 client.26141 fetch_fsmap learned FSMap version 25 2024-03-21T10:40:34.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.006+0000 7fcf9db82740 1 -- 172.21.15.31:0/2430973343 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x559999498c40 con 0x55999942cdd0 2024-03-21T10:40:34.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.006+0000 7fcf7e000640 1 -- 172.21.15.31:0/2430973343 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf80007810 con 0x55999942cdd0 2024-03-21T10:40:34.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.006+0000 7fcf7e000640 1 -- 172.21.15.31:0/2430973343 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fcf800176c0 con 0x55999942cdd0 2024-03-21T10:40:34.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.007+0000 7fcf9db82740 10 client.26141 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:40:34.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.007+0000 7fcf9db82740 10 client.26141 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:40:34.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.007+0000 7fcf9db82740 20 client.26141 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:34.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.007+0000 7fcf9db82740 1 --2- 172.21.15.31:0/2430973343 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x559999499280 0x55999949b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:34.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.007+0000 7fcf9db82740 4 client.26141 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:40:34.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.007+0000 7fcf9db82740 1 -- 172.21.15.31:0/2430973343 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7fcf8001e040 con 0x559999499280 2024-03-21T10:40:34.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.007+0000 7fcf93400640 1 --2- 172.21.15.31:0/2430973343 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x559999499280 0x55999949b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:34.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.008+0000 7fcf93400640 1 --2- 172.21.15.31:0/2430973343 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x559999499280 0x55999949b6c0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b1a48c748d88437 server_cookie=5a561758aad6adfd in_seq=0 out_seq=0 2024-03-21T10:40:34.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.008+0000 7fcf7e000640 10 client.26141 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:34.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.008+0000 7fcf7e000640 1 -- 172.21.15.31:0/2430973343 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x7fcf8001e040 con 0x559999499280 2024-03-21T10:40:34.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.008+0000 7fcf7e000640 10 client.26141 handle_command_reply: tid=0 2024-03-21T10:40:34.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.008+0000 7fcf9db82740 1 client.26141 shutdown 2024-03-21T10:40:34.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.008+0000 7fcf9db82740 1 -- 172.21.15.31:0/2430973343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55999942cdd0 msgr2=0x5599994aa400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:34.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.008+0000 7fcf9db82740 1 --2- 172.21.15.31:0/2430973343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55999942cdd0 0x5599994aa400 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcf8000c010 tx=0x7fcf80022440 comp rx=0 tx=0).stop 2024-03-21T10:40:34.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.008+0000 7fcf9db82740 1 -- 172.21.15.31:0/2430973343 shutdown_connections 2024-03-21T10:40:34.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.008+0000 7fcf9db82740 1 --2- 172.21.15.31:0/2430973343 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x559999499280 0x55999949b6c0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:34.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.008+0000 7fcf9db82740 1 --2- 172.21.15.31:0/2430973343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55999944d0c0 0x55999943a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:34.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.008+0000 7fcf7e000640 0 client.26141 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:34.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.008+0000 7fcf9db82740 1 --2- 172.21.15.31:0/2430973343 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559999435460 0x5599994aa940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:34.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.008+0000 7fcf9db82740 1 --2- 172.21.15.31:0/2430973343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55999942cdd0 0x5599994aa400 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:34.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.008+0000 7fcf9db82740 1 -- 172.21.15.31:0/2430973343 >> 172.21.15.31:0/2430973343 conn(0x5599993f3200 msgr2=0x55999944c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:34.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.008+0000 7fcf9db82740 1 -- 172.21.15.31:0/2430973343 shutdown_connections 2024-03-21T10:40:34.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.009+0000 7fcf9db82740 1 -- 172.21.15.31:0/2430973343 wait complete. 2024-03-21T10:40:34.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.009+0000 7fcf9db82740 20 client.26141 trim_cache size 0 max 16384 2024-03-21T10:40:34.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.009+0000 7fcf99800640 1 -- 172.21.15.31:0/3207268530 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fcf780a3150 msgr2=0x7fcf780a5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:34.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.009+0000 7fcf99800640 1 --2- 172.21.15.31:0/3207268530 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fcf780a3150 0x7fcf780a5610 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcf80007320 tx=0x7fcf8000c040 comp rx=0 tx=0).stop 2024-03-21T10:40:34.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.009+0000 7fcf99800640 1 -- 172.21.15.31:0/3207268530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf941000d0 msgr2=0x7fcf941a35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:34.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.009+0000 7fcf99800640 1 --2- 172.21.15.31:0/3207268530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf941000d0 0x7fcf941a35a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcf84000c00 tx=0x7fcf840045e0 comp rx=0 tx=0).stop 2024-03-21T10:40:34.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.010+0000 7fcf99800640 1 -- 172.21.15.31:0/3207268530 shutdown_connections 2024-03-21T10:40:34.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.010+0000 7fcf99800640 1 --2- 172.21.15.31:0/3207268530 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fcf780a3150 0x7fcf780a5610 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:34.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.010+0000 7fcf99800640 1 --2- 172.21.15.31:0/3207268530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf941056a0 0x7fcf9410b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:34.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.010+0000 7fcf99800640 1 --2- 172.21.15.31:0/3207268530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf941000d0 0x7fcf941a35a0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:34.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.010+0000 7fcf99800640 1 --2- 172.21.15.31:0/3207268530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf940ff710 0x7fcf941a3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:34.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.010+0000 7fcf99800640 1 -- 172.21.15.31:0/3207268530 >> 172.21.15.31:0/3207268530 conn(0x7fcf940fb840 msgr2=0x7fcf94102350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:34.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.010+0000 7fcf99800640 1 -- 172.21.15.31:0/3207268530 shutdown_connections 2024-03-21T10:40:34.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.010+0000 7fcf99800640 1 -- 172.21.15.31:0/3207268530 wait complete. 2024-03-21T10:40:34.059 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-21T10:40:34.059 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:40:34.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.398+0000 7f6b40e00640 1 -- 172.21.15.31:0/2582185157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b3c103850 msgr2=0x7f6b3c103cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:34.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.398+0000 7f6b40e00640 1 --2- 172.21.15.31:0/2582185157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b3c103850 0x7f6b3c103cd0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6b28009900 tx=0x7f6b2802bc50 comp rx=0 tx=0).stop 2024-03-21T10:40:34.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.399+0000 7f6b40e00640 1 -- 172.21.15.31:0/2582185157 shutdown_connections 2024-03-21T10:40:34.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.399+0000 7f6b40e00640 1 --2- 172.21.15.31:0/2582185157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b3c104210 0x7f6b3c110720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:34.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.399+0000 7f6b40e00640 1 --2- 172.21.15.31:0/2582185157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b3c103850 0x7f6b3c103cd0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:34.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.399+0000 7f6b40e00640 1 --2- 172.21.15.31:0/2582185157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b3c109860 0x7f6b3c109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:34.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.399+0000 7f6b40e00640 1 -- 172.21.15.31:0/2582185157 >> 172.21.15.31:0/2582185157 conn(0x7f6b3c0ff360 msgr2=0x7f6b3c101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:34.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.401+0000 7f6b40e00640 1 -- 172.21.15.31:0/2582185157 shutdown_connections 2024-03-21T10:40:34.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.402+0000 7f6b40e00640 1 -- 172.21.15.31:0/2582185157 wait complete. 2024-03-21T10:40:34.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.403+0000 7f6b40e00640 1 Processor -- start 2024-03-21T10:40:34.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.403+0000 7f6b40e00640 1 -- start start 2024-03-21T10:40:34.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.404+0000 7f6b40e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b3c103850 0x7f6b3c19a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:34.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.404+0000 7f6b40e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b3c104210 0x7f6b3c19a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:34.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.404+0000 7f6b40e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b3c109860 0x7f6b3c19eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:34.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.404+0000 7f6b40e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c113ce0 con 0x7f6b3c109860 2024-03-21T10:40:34.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.404+0000 7f6b40e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c113b60 con 0x7f6b3c103850 2024-03-21T10:40:34.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.404+0000 7f6b40e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c113e60 con 0x7f6b3c104210 2024-03-21T10:40:34.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.404+0000 7f6b39400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b3c104210 0x7f6b3c19a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:34.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.405+0000 7f6b39400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b3c104210 0x7f6b3c19a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34430/0 (socket says 172.21.15.31:34430) 2024-03-21T10:40:34.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.405+0000 7f6b39400640 1 -- 172.21.15.31:0/2647804826 learned_addr learned my addr 172.21.15.31:0/2647804826 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:34.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.405+0000 7f6b39e00640 1 --2- 172.21.15.31:0/2647804826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b3c103850 0x7f6b3c19a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:34.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.405+0000 7f6b3a800640 1 --2- 172.21.15.31:0/2647804826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b3c109860 0x7f6b3c19eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:34.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.405+0000 7f6b39400640 1 -- 172.21.15.31:0/2647804826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b3c103850 msgr2=0x7f6b3c19a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:34.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.405+0000 7f6b39400640 1 --2- 172.21.15.31:0/2647804826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b3c103850 0x7f6b3c19a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:34.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.405+0000 7f6b39400640 1 -- 172.21.15.31:0/2647804826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b3c109860 msgr2=0x7f6b3c19eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:34.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.405+0000 7f6b39400640 1 --2- 172.21.15.31:0/2647804826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b3c109860 0x7f6b3c19eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:34.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.405+0000 7f6b39400640 1 -- 172.21.15.31:0/2647804826 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3c19f210 con 0x7f6b3c104210 2024-03-21T10:40:34.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.405+0000 7f6b3a800640 1 --2- 172.21.15.31:0/2647804826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b3c109860 0x7f6b3c19eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:40:34.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.405+0000 7f6b39400640 1 --2- 172.21.15.31:0/2647804826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b3c104210 0x7f6b3c19a760 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6b28000c00 tx=0x7f6b28004430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:34.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.406+0000 7f6b32a00640 1 -- 172.21.15.31:0/2647804826 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b28041070 con 0x7f6b3c104210 2024-03-21T10:40:34.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.406+0000 7f6b32a00640 1 -- 172.21.15.31:0/2647804826 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b280054e0 con 0x7f6b3c104210 2024-03-21T10:40:34.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.406+0000 7f6b32a00640 1 -- 172.21.15.31:0/2647804826 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b28003e00 con 0x7f6b3c104210 2024-03-21T10:40:34.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.406+0000 7f6b40e00640 1 -- 172.21.15.31:0/2647804826 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b3c19f4a0 con 0x7f6b3c104210 2024-03-21T10:40:34.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.406+0000 7f6b40e00640 1 -- 172.21.15.31:0/2647804826 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b3c1a6d90 con 0x7f6b3c104210 2024-03-21T10:40:34.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.408+0000 7f6b40e00640 1 -- 172.21.15.31:0/2647804826 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b3c068970 con 0x7f6b3c104210 2024-03-21T10:40:34.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.411+0000 7f6b32a00640 1 -- 172.21.15.31:0/2647804826 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f6b28044070 con 0x7f6b3c104210 2024-03-21T10:40:34.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.414+0000 7f6b32a00640 1 --2- 172.21.15.31:0/2647804826 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6b200a3110 0x7f6b200a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:34.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.415+0000 7f6b39e00640 1 --2- 172.21.15.31:0/2647804826 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6b200a3110 0x7f6b200a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:34.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.415+0000 7f6b32a00640 1 -- 172.21.15.31:0/2647804826 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f6b280f1620 con 0x7f6b3c104210 2024-03-21T10:40:34.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.416+0000 7f6b39e00640 1 --2- 172.21.15.31:0/2647804826 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6b200a3110 0x7f6b200a55d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c006fd0 tx=0x7f6b2c0090e0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:34.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:34.421+0000 7f6b32a00640 1 -- 172.21.15.31:0/2647804826 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6b280bdee0 con 0x7f6b3c104210 2024-03-21T10:40:35.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.015+0000 7f6b40e00640 1 -- 172.21.15.31:0/2647804826 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b3c10bdf0 con 0x7f6b3c104210 2024-03-21T10:40:35.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.016+0000 7f6b32a00640 1 -- 172.21.15.31:0/2647804826 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f6b280bd7e0 con 0x7f6b3c104210 2024-03-21T10:40:35.013 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:40:35.013 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:40:26.180798+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24596,24547]},"id":1}]} 2024-03-21T10:40:35.013 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 25 2024-03-21T10:40:35.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.018+0000 7f6b40e00640 1 -- 172.21.15.31:0/2647804826 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6b200a3110 msgr2=0x7f6b200a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.018+0000 7f6b40e00640 1 --2- 172.21.15.31:0/2647804826 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6b200a3110 0x7f6b200a55d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c006fd0 tx=0x7f6b2c0090e0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.018+0000 7f6b40e00640 1 -- 172.21.15.31:0/2647804826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b3c104210 msgr2=0x7f6b3c19a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.018+0000 7f6b40e00640 1 --2- 172.21.15.31:0/2647804826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b3c104210 0x7f6b3c19a760 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6b28000c00 tx=0x7f6b28004430 comp rx=0 tx=0).stop 2024-03-21T10:40:35.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.018+0000 7f6b40e00640 1 -- 172.21.15.31:0/2647804826 shutdown_connections 2024-03-21T10:40:35.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.019+0000 7f6b40e00640 1 --2- 172.21.15.31:0/2647804826 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6b200a3110 0x7f6b200a55d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.019+0000 7f6b40e00640 1 --2- 172.21.15.31:0/2647804826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b3c109860 0x7f6b3c19eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.019+0000 7f6b40e00640 1 --2- 172.21.15.31:0/2647804826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b3c104210 0x7f6b3c19a760 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.019+0000 7f6b40e00640 1 --2- 172.21.15.31:0/2647804826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b3c103850 0x7f6b3c19a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.019+0000 7f6b40e00640 1 -- 172.21.15.31:0/2647804826 >> 172.21.15.31:0/2647804826 conn(0x7f6b3c0ff360 msgr2=0x7f6b3c0ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:35.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.019+0000 7f6b40e00640 1 -- 172.21.15.31:0/2647804826 shutdown_connections 2024-03-21T10:40:35.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.019+0000 7f6b40e00640 1 -- 172.21.15.31:0/2647804826 wait complete. 2024-03-21T10:40:35.033 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:40:35.033 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-21T10:40:35.127 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:34 smithi031 ceph-mon[31673]: pgmap v63: 129 pgs: 129 active+clean; 3.9 GiB data, 895 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 220 MiB/s wr, 72 op/s 2024-03-21T10:40:35.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:34 smithi040 ceph-mon[36154]: pgmap v63: 129 pgs: 129 active+clean; 3.9 GiB data, 895 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 220 MiB/s wr, 72 op/s 2024-03-21T10:40:35.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:34 smithi070 ceph-mon[36330]: pgmap v63: 129 pgs: 129 active+clean; 3.9 GiB data, 895 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 220 MiB/s wr, 72 op/s 2024-03-21T10:40:35.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.340+0000 7f88db000640 1 -- 172.21.15.31:0/2120983586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88d41124e0 msgr2=0x7f88d41148d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.340+0000 7f88db000640 1 --2- 172.21.15.31:0/2120983586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88d41124e0 0x7f88d41148d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f88cc009be0 tx=0x7f88cc02bf10 comp rx=0 tx=0).stop 2024-03-21T10:40:35.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.340+0000 7f88db000640 1 -- 172.21.15.31:0/2120983586 shutdown_connections 2024-03-21T10:40:35.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.340+0000 7f88db000640 1 --2- 172.21.15.31:0/2120983586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88d41124e0 0x7f88d41148d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.340+0000 7f88db000640 1 --2- 172.21.15.31:0/2120983586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88d40764f0 0x7f88d4111fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.340+0000 7f88db000640 1 --2- 172.21.15.31:0/2120983586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d4075bd0 0x7f88d4075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.340+0000 7f88db000640 1 -- 172.21.15.31:0/2120983586 >> 172.21.15.31:0/2120983586 conn(0x7f88d40ff2f0 msgr2=0x7f88d4101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:35.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.341+0000 7f88db000640 1 -- 172.21.15.31:0/2120983586 shutdown_connections 2024-03-21T10:40:35.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.341+0000 7f88db000640 1 -- 172.21.15.31:0/2120983586 wait complete. 2024-03-21T10:40:35.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.343+0000 7f88db000640 1 Processor -- start 2024-03-21T10:40:35.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.343+0000 7f88db000640 1 -- start start 2024-03-21T10:40:35.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.343+0000 7f88db000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d4075bd0 0x7f88d4072c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.343+0000 7f88db000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88d40764f0 0x7f88d4073190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.343+0000 7f88db000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88d41124e0 0x7f88d406bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.344+0000 7f88d3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88d40764f0 0x7f88d4073190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.344+0000 7f88d3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88d40764f0 0x7f88d4073190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34452/0 (socket says 172.21.15.31:34452) 2024-03-21T10:40:35.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.344+0000 7f88d9000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88d41124e0 0x7f88d406bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.344+0000 7f88d3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d4075bd0 0x7f88d4072c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.344+0000 7f88d3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d4075bd0 0x7f88d4072c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:48880/0 (socket says 172.21.15.31:48880) 2024-03-21T10:40:35.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.344+0000 7f88db000640 1 -- 172.21.15.31:0/1246978710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d4118090 con 0x7f88d41124e0 2024-03-21T10:40:35.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.344+0000 7f88d3400640 1 -- 172.21.15.31:0/1246978710 learned_addr learned my addr 172.21.15.31:0/1246978710 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:35.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.344+0000 7f88db000640 1 -- 172.21.15.31:0/1246978710 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d4117f10 con 0x7f88d4075bd0 2024-03-21T10:40:35.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.344+0000 7f88db000640 1 -- 172.21.15.31:0/1246978710 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d4118210 con 0x7f88d40764f0 2024-03-21T10:40:35.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.345+0000 7f88d3e00640 1 -- 172.21.15.31:0/1246978710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88d40764f0 msgr2=0x7f88d4073190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.345+0000 7f88d3e00640 1 --2- 172.21.15.31:0/1246978710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88d40764f0 0x7f88d4073190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.345+0000 7f88d3e00640 1 -- 172.21.15.31:0/1246978710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88d41124e0 msgr2=0x7f88d406bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.345+0000 7f88d3e00640 1 --2- 172.21.15.31:0/1246978710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88d41124e0 0x7f88d406bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.345+0000 7f88d3e00640 1 -- 172.21.15.31:0/1246978710 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d406c5c0 con 0x7f88d4075bd0 2024-03-21T10:40:35.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.345+0000 7f88d9000640 1 --2- 172.21.15.31:0/1246978710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88d41124e0 0x7f88d406bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:40:35.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.346+0000 7f88d3e00640 1 --2- 172.21.15.31:0/1246978710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d4075bd0 0x7f88d4072c50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f88c400d950 tx=0x7f88c400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:35.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.346+0000 7f88d0c00640 1 -- 172.21.15.31:0/1246978710 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c4009c30 con 0x7f88d4075bd0 2024-03-21T10:40:35.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.346+0000 7f88d0c00640 1 -- 172.21.15.31:0/1246978710 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88c4009dd0 con 0x7f88d4075bd0 2024-03-21T10:40:35.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.347+0000 7f88d0c00640 1 -- 172.21.15.31:0/1246978710 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c4010460 con 0x7f88d4075bd0 2024-03-21T10:40:35.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.347+0000 7f88db000640 1 -- 172.21.15.31:0/1246978710 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d406c8b0 con 0x7f88d4075bd0 2024-03-21T10:40:35.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.347+0000 7f88db000640 1 -- 172.21.15.31:0/1246978710 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d4070950 con 0x7f88d4075bd0 2024-03-21T10:40:35.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.352+0000 7f88ddde2740 1 Processor -- start 2024-03-21T10:40:35.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.353+0000 7f88d0c00640 1 -- 172.21.15.31:0/1246978710 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f88c4003f60 con 0x7f88d4075bd0 2024-03-21T10:40:35.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.354+0000 7f88ddde2740 1 -- start start 2024-03-21T10:40:35.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.354+0000 7f88ddde2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606eec2d6d0 0x5606eec2dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.354+0000 7f88ddde2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5606eec2e080 0x5606eec36630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.354+0000 7f88ddde2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606eec36c00 0x5606eec39050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.354+0000 7f88ddde2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606eeb912c0 con 0x5606eec2d6d0 2024-03-21T10:40:35.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.354+0000 7f88ddde2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606eeb39c90 con 0x5606eec2e080 2024-03-21T10:40:35.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.354+0000 7f88ddde2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606eeb8d360 con 0x5606eec36c00 2024-03-21T10:40:35.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.355+0000 7f88d3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5606eec2e080 0x5606eec36630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.355+0000 7f88d3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5606eec2e080 0x5606eec36630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:48892/0 (socket says 172.21.15.31:48892) 2024-03-21T10:40:35.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.355+0000 7f88d3e00640 1 -- 172.21.15.31:0/3743148330 learned_addr learned my addr 172.21.15.31:0/3743148330 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:35.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.355+0000 7f88d3400640 1 --2- 172.21.15.31:0/3743148330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606eec36c00 0x5606eec39050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.355+0000 7f88d3e00640 1 -- 172.21.15.31:0/3743148330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606eec36c00 msgr2=0x5606eec39050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:35.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.355+0000 7f88d3e00640 1 --2- 172.21.15.31:0/3743148330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606eec36c00 0x5606eec39050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.355+0000 7f88d9000640 1 --2- 172.21.15.31:0/3743148330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606eec2d6d0 0x5606eec2dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.355+0000 7f88d3e00640 1 -- 172.21.15.31:0/3743148330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606eec2d6d0 msgr2=0x5606eec2dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:35.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.356+0000 7f88d3e00640 1 --2- 172.21.15.31:0/3743148330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606eec2d6d0 0x5606eec2dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.356+0000 7f88d3e00640 1 -- 172.21.15.31:0/3743148330 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606eeb5e080 con 0x5606eec2e080 2024-03-21T10:40:35.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.356+0000 7f88d0c00640 1 --2- 172.21.15.31:0/1246978710 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f88b80a3110 0x7f88b80a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.356+0000 7f88d9000640 1 --2- 172.21.15.31:0/3743148330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606eec2d6d0 0x5606eec2dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:40:35.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.356+0000 7f88d0c00640 1 -- 172.21.15.31:0/1246978710 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f88c40cf300 con 0x7f88d4075bd0 2024-03-21T10:40:35.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.356+0000 7f88d3e00640 1 --2- 172.21.15.31:0/3743148330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5606eec2e080 0x5606eec36630 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f88c40cc680 tx=0x7f88c40d79f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b602a6e6f82f3ba9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:35.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.357+0000 7f88c9600640 1 -- 172.21.15.31:0/3743148330 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c4020030 con 0x5606eec2e080 2024-03-21T10:40:35.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.357+0000 7f88c9600640 1 -- 172.21.15.31:0/3743148330 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88c409bab0 con 0x5606eec2e080 2024-03-21T10:40:35.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.357+0000 7f88c9600640 1 -- 172.21.15.31:0/3743148330 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c409aef0 con 0x5606eec2e080 2024-03-21T10:40:35.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.358+0000 7f88d9000640 1 --2- 172.21.15.31:0/1246978710 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f88b80a3110 0x7f88b80a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.358+0000 7f88ddde2740 1 -- 172.21.15.31:0/3743148330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5606eec2e080 msgr2=0x5606eec36630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.358+0000 7f88ddde2740 1 --2- 172.21.15.31:0/3743148330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5606eec2e080 0x5606eec36630 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f88c40cc680 tx=0x7f88c40d79f0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.358+0000 7f88ddde2740 1 -- 172.21.15.31:0/3743148330 shutdown_connections 2024-03-21T10:40:35.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.358+0000 7f88ddde2740 1 --2- 172.21.15.31:0/3743148330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606eec36c00 0x5606eec39050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.358+0000 7f88ddde2740 1 --2- 172.21.15.31:0/3743148330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5606eec2e080 0x5606eec36630 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.358+0000 7f88ddde2740 1 --2- 172.21.15.31:0/3743148330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606eec2d6d0 0x5606eec2dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.358+0000 7f88ddde2740 1 -- 172.21.15.31:0/3743148330 >> 172.21.15.31:0/3743148330 conn(0x5606eec27da0 msgr2=0x5606eec28180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:35.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.359+0000 7f88ddde2740 1 -- 172.21.15.31:0/3743148330 shutdown_connections 2024-03-21T10:40:35.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.359+0000 7f88d9000640 1 --2- 172.21.15.31:0/1246978710 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f88b80a3110 0x7f88b80a55d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f88cc02c450 tx=0x7f88cc02e040 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:35.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.359+0000 7f88ddde2740 1 -- 172.21.15.31:0/3743148330 wait complete. 2024-03-21T10:40:35.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.359+0000 7f88ddde2740 1 -- start start 2024-03-21T10:40:35.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.360+0000 7f88ddde2740 1 Processor -- start 2024-03-21T10:40:35.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.360+0000 7f88ddde2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606eec2d6d0 0x5606eec36600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.360+0000 7f88ddde2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5606eec2e080 0x5606eec33990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.361+0000 7f88d3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606eec2d6d0 0x5606eec36600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.361+0000 7f88ddde2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606eec36c00 0x5606eec34080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.361+0000 7f88d3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606eec2d6d0 0x5606eec36600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39658/0 (socket says 172.21.15.31:39658) 2024-03-21T10:40:35.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.362+0000 7f88ddde2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606eec39b10 con 0x5606eec2d6d0 2024-03-21T10:40:35.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.362+0000 7f88ddde2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606eeb4d0d0 con 0x5606eec2e080 2024-03-21T10:40:35.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.362+0000 7f88ddde2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606eec39c90 con 0x5606eec36c00 2024-03-21T10:40:35.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.362+0000 7f88ddde2740 10 client.0 _ll_register_callbacks cb 0x5606eeb51e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:35.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.362+0000 7f88d3e00640 1 -- 172.21.15.31:0/3487971516 learned_addr learned my addr 172.21.15.31:0/3487971516 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:35.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.362+0000 7f88d9000640 1 --2- 172.21.15.31:0/3487971516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606eec36c00 0x5606eec34080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.362+0000 7f88d3400640 1 --2- 172.21.15.31:0/3487971516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5606eec2e080 0x5606eec33990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.363+0000 7f88d3e00640 1 -- 172.21.15.31:0/3487971516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606eec36c00 msgr2=0x5606eec34080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:35.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.363+0000 7f88d3e00640 1 --2- 172.21.15.31:0/3487971516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606eec36c00 0x5606eec34080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.363+0000 7f88d3e00640 1 -- 172.21.15.31:0/3487971516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5606eec2e080 msgr2=0x5606eec33990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:35.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.363+0000 7f88d3e00640 1 --2- 172.21.15.31:0/3487971516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5606eec2e080 0x5606eec33990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.363+0000 7f88d3e00640 1 -- 172.21.15.31:0/3487971516 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5606ee8a53f0 con 0x5606eec2d6d0 2024-03-21T10:40:35.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.363+0000 7f88d9000640 1 --2- 172.21.15.31:0/3487971516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606eec36c00 0x5606eec34080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:40:35.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.364+0000 7f88d3e00640 1 --2- 172.21.15.31:0/3487971516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606eec2d6d0 0x5606eec36600 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f88c4099660 tx=0x7f88c40039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7992d950c9a9095d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:35.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.364+0000 7f88be000640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:40:35.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.364+0000 7f88be000640 1 -- 172.21.15.31:0/3487971516 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c40cce60 con 0x5606eec2d6d0 2024-03-21T10:40:35.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.364+0000 7f88be000640 1 -- 172.21.15.31:0/3487971516 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f88c40d7150 con 0x5606eec2d6d0 2024-03-21T10:40:35.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.364+0000 7f88ddde2740 1 -- 172.21.15.31:0/3487971516 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f88c409bab0 con 0x5606eec2d6d0 2024-03-21T10:40:35.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.365+0000 7f88be000640 1 -- 172.21.15.31:0/3487971516 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88c409ad10 con 0x5606eec2d6d0 2024-03-21T10:40:35.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.365+0000 7f88ddde2740 10 client.15948 fetch_fsmap learned FSMap version 25 2024-03-21T10:40:35.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.365+0000 7f88ddde2740 1 -- 172.21.15.31:0/3487971516 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5606ee96f7d0 con 0x5606eec2d6d0 2024-03-21T10:40:35.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.366+0000 7f88be000640 1 -- 172.21.15.31:0/3487971516 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88c409bab0 con 0x5606eec2d6d0 2024-03-21T10:40:35.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.366+0000 7f88be000640 1 -- 172.21.15.31:0/3487971516 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f88c4023060 con 0x5606eec2d6d0 2024-03-21T10:40:35.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.366+0000 7f88ddde2740 10 client.15948 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:40:35.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.366+0000 7f88ddde2740 10 client.15948 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:40:35.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.366+0000 7f88ddde2740 20 client.15948 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:35.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.366+0000 7f88ddde2740 1 --2- 172.21.15.31:0/3487971516 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5606eec468e0 0x5606eec48cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.366+0000 7f88ddde2740 4 client.15948 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:40:35.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.367+0000 7f88ddde2740 1 -- 172.21.15.31:0/3487971516 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5606eeb9e800 con 0x5606eec468e0 2024-03-21T10:40:35.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.367+0000 7f88d3400640 1 --2- 172.21.15.31:0/3487971516 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5606eec468e0 0x5606eec48cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.368+0000 7f88d3400640 1 --2- 172.21.15.31:0/3487971516 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5606eec468e0 0x5606eec48cd0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a0da2c2123a856b9 server_cookie=c4b91aa516277cf9 in_seq=0 out_seq=0 2024-03-21T10:40:35.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.368+0000 7f88be000640 10 client.15948 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:40:35.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.374+0000 7f88be000640 1 -- 172.21.15.31:0/3487971516 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5606eeb9e800 con 0x5606eec468e0 2024-03-21T10:40:35.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.374+0000 7f88be000640 10 client.15948 handle_command_reply: tid=0 2024-03-21T10:40:35.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.374+0000 7f88ddde2740 1 client.15948 shutdown 2024-03-21T10:40:35.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.375+0000 7f88ddde2740 1 -- 172.21.15.31:0/3487971516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606eec2d6d0 msgr2=0x5606eec36600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.375+0000 7f88ddde2740 1 --2- 172.21.15.31:0/3487971516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606eec2d6d0 0x5606eec36600 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f88c4099660 tx=0x7f88c40039c0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.376+0000 7f88ddde2740 1 -- 172.21.15.31:0/3487971516 shutdown_connections 2024-03-21T10:40:35.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.376+0000 7f88ddde2740 1 --2- 172.21.15.31:0/3487971516 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5606eec468e0 0x5606eec48cd0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.376+0000 7f88ddde2740 1 --2- 172.21.15.31:0/3487971516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606eec36c00 0x5606eec34080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.376+0000 7f88ddde2740 1 --2- 172.21.15.31:0/3487971516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5606eec2e080 0x5606eec33990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.376+0000 7f88ddde2740 1 --2- 172.21.15.31:0/3487971516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606eec2d6d0 0x5606eec36600 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.376+0000 7f88ddde2740 1 -- 172.21.15.31:0/3487971516 >> 172.21.15.31:0/3487971516 conn(0x5606eec27da0 msgr2=0x5606eec29af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:35.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.376+0000 7f88be000640 0 client.15948 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:40:35.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.377+0000 7f88ddde2740 1 -- 172.21.15.31:0/3487971516 shutdown_connections 2024-03-21T10:40:35.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.377+0000 7f88ddde2740 1 -- 172.21.15.31:0/3487971516 wait complete. 2024-03-21T10:40:35.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.377+0000 7f88ddde2740 20 client.15948 trim_cache size 0 max 16384 2024-03-21T10:40:35.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.410+0000 7f88ddde2740 1 Processor -- start 2024-03-21T10:40:35.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.410+0000 7f88ddde2740 1 -- start start 2024-03-21T10:40:35.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.411+0000 7f88ddde2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5606eec2d6d0 0x5606eec42ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.411+0000 7f88ddde2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606eec2e080 0x5606eec22d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.411+0000 7f88ddde2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606eec36c00 0x5606eec360c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.411+0000 7f88ddde2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d41bfa50 con 0x5606eec36c00 2024-03-21T10:40:35.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.411+0000 7f88ddde2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d41bd550 con 0x5606eec2d6d0 2024-03-21T10:40:35.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.411+0000 7f88ddde2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d41bb5f0 con 0x5606eec2e080 2024-03-21T10:40:35.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.412+0000 7f88d3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5606eec2d6d0 0x5606eec42ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.412+0000 7f88d3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5606eec2d6d0 0x5606eec42ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:48904/0 (socket says 172.21.15.31:48904) 2024-03-21T10:40:35.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.412+0000 7f88d3e00640 1 -- 172.21.15.31:0/1051231674 learned_addr learned my addr 172.21.15.31:0/1051231674 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:35.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.412+0000 7f88d9000640 1 --2- 172.21.15.31:0/1051231674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606eec36c00 0x5606eec360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.413+0000 7f88d3400640 1 --2- 172.21.15.31:0/1051231674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606eec2e080 0x5606eec22d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.414+0000 7f88d3400640 1 -- 172.21.15.31:0/1051231674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5606eec2d6d0 msgr2=0x5606eec42ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:35.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.414+0000 7f88d3400640 1 --2- 172.21.15.31:0/1051231674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5606eec2d6d0 0x5606eec42ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.414+0000 7f88d3400640 1 -- 172.21.15.31:0/1051231674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606eec36c00 msgr2=0x5606eec360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:35.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.414+0000 7f88d3400640 1 --2- 172.21.15.31:0/1051231674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606eec36c00 0x5606eec360c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.414+0000 7f88d3400640 1 -- 172.21.15.31:0/1051231674 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606eec23700 con 0x5606eec2e080 2024-03-21T10:40:35.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.414+0000 7f88d3e00640 1 --2- 172.21.15.31:0/1051231674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5606eec2d6d0 0x5606eec42ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:40:35.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.414+0000 7f88d9000640 1 --2- 172.21.15.31:0/1051231674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606eec36c00 0x5606eec360c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:40:35.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.415+0000 7f88d3400640 1 --2- 172.21.15.31:0/1051231674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606eec2e080 0x5606eec22d40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x5606eec36b40 tx=0x7f88c000e3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c2055b84d369fbf server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:35.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.416+0000 7f88c8c00640 1 -- 172.21.15.31:0/1051231674 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c0011640 con 0x5606eec2e080 2024-03-21T10:40:35.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.416+0000 7f88c8c00640 1 -- 172.21.15.31:0/1051231674 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88c0011cc0 con 0x5606eec2e080 2024-03-21T10:40:35.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.416+0000 7f88c8c00640 1 -- 172.21.15.31:0/1051231674 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c0010cc0 con 0x5606eec2e080 2024-03-21T10:40:35.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.417+0000 7f88ddde2740 1 -- 172.21.15.31:0/1051231674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606eec2e080 msgr2=0x5606eec22d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.417+0000 7f88ddde2740 1 --2- 172.21.15.31:0/1051231674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606eec2e080 0x5606eec22d40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x5606eec36b40 tx=0x7f88c000e3b0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.418+0000 7f88ddde2740 1 -- 172.21.15.31:0/1051231674 shutdown_connections 2024-03-21T10:40:35.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.418+0000 7f88ddde2740 1 --2- 172.21.15.31:0/1051231674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606eec36c00 0x5606eec360c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.418+0000 7f88ddde2740 1 --2- 172.21.15.31:0/1051231674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606eec2e080 0x5606eec22d40 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.418+0000 7f88ddde2740 1 --2- 172.21.15.31:0/1051231674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5606eec2d6d0 0x5606eec42ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.418+0000 7f88ddde2740 1 -- 172.21.15.31:0/1051231674 >> 172.21.15.31:0/1051231674 conn(0x5606eec27da0 msgr2=0x5606eec26eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:35.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.418+0000 7f88ddde2740 1 -- 172.21.15.31:0/1051231674 shutdown_connections 2024-03-21T10:40:35.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.418+0000 7f88ddde2740 1 -- 172.21.15.31:0/1051231674 wait complete. 2024-03-21T10:40:35.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.419+0000 7f88ddde2740 1 -- start start 2024-03-21T10:40:35.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.419+0000 7f88ddde2740 1 Processor -- start 2024-03-21T10:40:35.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.420+0000 7f88ddde2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5606eec2d6d0 0x5606eec49040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.420+0000 7f88ddde2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606eec2e080 0x5606eec49580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.420+0000 7f88ddde2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606eec36c00 0x5606eeca0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.420+0000 7f88ddde2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d41bbbf0 con 0x5606eec2e080 2024-03-21T10:40:35.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.420+0000 7f88ddde2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d41bcef0 con 0x5606eec2d6d0 2024-03-21T10:40:35.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.420+0000 7f88ddde2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d41bc230 con 0x5606eec36c00 2024-03-21T10:40:35.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.420+0000 7f88ddde2740 10 client.0 _ll_register_callbacks cb 0x7f88c40d73e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:35.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.421+0000 7f88d3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606eec2e080 0x5606eec49580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.421+0000 7f88d3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606eec2e080 0x5606eec49580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39684/0 (socket says 172.21.15.31:39684) 2024-03-21T10:40:35.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.421+0000 7f88d3400640 1 -- 172.21.15.31:0/1657031903 learned_addr learned my addr 172.21.15.31:0/1657031903 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:35.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.422+0000 7f88d3400640 1 -- 172.21.15.31:0/1657031903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606eec36c00 msgr2=0x5606eeca0e10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:40:35.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.422+0000 7f88d3400640 1 --2- 172.21.15.31:0/1657031903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606eec36c00 0x5606eeca0e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.422+0000 7f88d3400640 1 -- 172.21.15.31:0/1657031903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5606eec2d6d0 msgr2=0x5606eec49040 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:40:35.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.423+0000 7f88d3e00640 1 --2- 172.21.15.31:0/1657031903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5606eec2d6d0 0x5606eec49040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.423+0000 7f88d3400640 1 --2- 172.21.15.31:0/1657031903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5606eec2d6d0 0x5606eec49040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.423+0000 7f88d3400640 1 -- 172.21.15.31:0/1657031903 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5606eec49e10 con 0x5606eec2e080 2024-03-21T10:40:35.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.423+0000 7f88d3e00640 1 --2- 172.21.15.31:0/1657031903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5606eec2d6d0 0x5606eec49040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:40:35.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.423+0000 7f88d3400640 1 --2- 172.21.15.31:0/1657031903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606eec2e080 0x5606eec49580 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x5606eec36b40 tx=0x7f88c000e950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca41f2323602f868 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:35.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.424+0000 7f88be000640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:40:35.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.424+0000 7f88be000640 1 -- 172.21.15.31:0/1657031903 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c0004220 con 0x5606eec2e080 2024-03-21T10:40:35.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.424+0000 7f88ddde2740 1 -- 172.21.15.31:0/1657031903 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f88c0011cc0 con 0x5606eec2e080 2024-03-21T10:40:35.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.424+0000 7f88be000640 1 -- 172.21.15.31:0/1657031903 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f88c00043c0 con 0x5606eec2e080 2024-03-21T10:40:35.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.425+0000 7f88be000640 1 -- 172.21.15.31:0/1657031903 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88c0010940 con 0x5606eec2e080 2024-03-21T10:40:35.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.425+0000 7f88ddde2740 10 client.15954 fetch_fsmap learned FSMap version 25 2024-03-21T10:40:35.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.425+0000 7f88ddde2740 1 -- 172.21.15.31:0/1657031903 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5606eeca2420 con 0x5606eec2e080 2024-03-21T10:40:35.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.426+0000 7f88be000640 1 -- 172.21.15.31:0/1657031903 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88c0011cc0 con 0x5606eec2e080 2024-03-21T10:40:35.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.426+0000 7f88be000640 1 -- 172.21.15.31:0/1657031903 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f88c002a030 con 0x5606eec2e080 2024-03-21T10:40:35.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.426+0000 7f88ddde2740 10 client.15954 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:40:35.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.426+0000 7f88ddde2740 10 client.15954 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:40:35.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.426+0000 7f88ddde2740 20 client.15954 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:35.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.426+0000 7f88ddde2740 1 --2- 172.21.15.31:0/1657031903 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5606eeca2c50 0x5606eeca5070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.426+0000 7f88ddde2740 4 client.15954 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:40:35.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.426+0000 7f88ddde2740 1 -- 172.21.15.31:0/1657031903 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5606eeb98b50 con 0x5606eeca2c50 2024-03-21T10:40:35.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.427+0000 7f88d3e00640 1 --2- 172.21.15.31:0/1657031903 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5606eeca2c50 0x5606eeca5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.427+0000 7f88be000640 10 client.15954 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:40:35.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.428+0000 7f88d3e00640 1 --2- 172.21.15.31:0/1657031903 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5606eeca2c50 0x5606eeca5070 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=10914493a0159fc8 server_cookie=82fcf060cc212f4c in_seq=0 out_seq=0 2024-03-21T10:40:35.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.428+0000 7f88be000640 1 -- 172.21.15.31:0/1657031903 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5606eeb98b50 con 0x5606eeca2c50 2024-03-21T10:40:35.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.428+0000 7f88be000640 10 client.15954 handle_command_reply: tid=0 2024-03-21T10:40:35.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.429+0000 7f88ddde2740 1 client.15954 shutdown 2024-03-21T10:40:35.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.436+0000 7f88ddde2740 1 -- 172.21.15.31:0/1657031903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606eec2e080 msgr2=0x5606eec49580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.436+0000 7f88ddde2740 1 --2- 172.21.15.31:0/1657031903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606eec2e080 0x5606eec49580 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x5606eec36b40 tx=0x7f88c000e950 comp rx=0 tx=0).stop 2024-03-21T10:40:35.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.436+0000 7f88ddde2740 1 -- 172.21.15.31:0/1657031903 shutdown_connections 2024-03-21T10:40:35.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.436+0000 7f88ddde2740 1 --2- 172.21.15.31:0/1657031903 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5606eeca2c50 0x5606eeca5070 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.436+0000 7f88ddde2740 1 --2- 172.21.15.31:0/1657031903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606eec36c00 0x5606eeca0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.436+0000 7f88be000640 0 client.15954 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:40:35.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.436+0000 7f88ddde2740 1 --2- 172.21.15.31:0/1657031903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606eec2e080 0x5606eec49580 secure :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0x5606eec36b40 tx=0x7f88c000e950 comp rx=0 tx=0).stop 2024-03-21T10:40:35.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.437+0000 7f88ddde2740 1 --2- 172.21.15.31:0/1657031903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5606eec2d6d0 0x5606eec49040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.437+0000 7f88ddde2740 1 -- 172.21.15.31:0/1657031903 >> 172.21.15.31:0/1657031903 conn(0x5606eec27da0 msgr2=0x5606eec2e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:35.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.438+0000 7f88ddde2740 1 -- 172.21.15.31:0/1657031903 shutdown_connections 2024-03-21T10:40:35.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.441+0000 7f88ddde2740 1 -- 172.21.15.31:0/1657031903 wait complete. 2024-03-21T10:40:35.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.441+0000 7f88ddde2740 20 client.15954 trim_cache size 0 max 16384 2024-03-21T10:40:35.439 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:40:35.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.443+0000 7f88db000640 1 -- 172.21.15.31:0/1246978710 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f88b80a3110 msgr2=0x7f88b80a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.443+0000 7f88db000640 1 --2- 172.21.15.31:0/1246978710 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f88b80a3110 0x7f88b80a55d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f88cc02c450 tx=0x7f88cc02e040 comp rx=0 tx=0).stop 2024-03-21T10:40:35.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.443+0000 7f88db000640 1 -- 172.21.15.31:0/1246978710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d4075bd0 msgr2=0x7f88d4072c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.443+0000 7f88db000640 1 --2- 172.21.15.31:0/1246978710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d4075bd0 0x7f88d4072c50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f88c400d950 tx=0x7f88c400de10 comp rx=0 tx=0).stop 2024-03-21T10:40:35.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.443+0000 7f88db000640 1 -- 172.21.15.31:0/1246978710 shutdown_connections 2024-03-21T10:40:35.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.443+0000 7f88db000640 1 --2- 172.21.15.31:0/1246978710 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f88b80a3110 0x7f88b80a55d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.443+0000 7f88db000640 1 --2- 172.21.15.31:0/1246978710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88d41124e0 0x7f88d406bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.443+0000 7f88db000640 1 --2- 172.21.15.31:0/1246978710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88d40764f0 0x7f88d4073190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.443+0000 7f88db000640 1 --2- 172.21.15.31:0/1246978710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d4075bd0 0x7f88d4072c50 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.443+0000 7f88db000640 1 -- 172.21.15.31:0/1246978710 >> 172.21.15.31:0/1246978710 conn(0x7f88d40ff2f0 msgr2=0x7f88d41128c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:35.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.443+0000 7f88db000640 1 -- 172.21.15.31:0/1246978710 shutdown_connections 2024-03-21T10:40:35.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.444+0000 7f88db000640 1 -- 172.21.15.31:0/1246978710 wait complete. 2024-03-21T10:40:35.481 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:40:35.481 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-21T10:40:35.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.617+0000 7fdded400640 1 -- 172.21.15.31:0/332813041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdde810d3d0 msgr2=0x7fdde8114fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.618+0000 7fdded400640 1 --2- 172.21.15.31:0/332813041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdde810d3d0 0x7fdde8114fa0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009900 tx=0x7fdde002bc80 comp rx=0 tx=0).stop 2024-03-21T10:40:35.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.618+0000 7fdded400640 1 -- 172.21.15.31:0/332813041 shutdown_connections 2024-03-21T10:40:35.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.618+0000 7fdded400640 1 --2- 172.21.15.31:0/332813041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdde810d3d0 0x7fdde8114fa0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.618+0000 7fdded400640 1 --2- 172.21.15.31:0/332813041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdde81092c0 0x7fdde810ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.618+0000 7fdded400640 1 --2- 172.21.15.31:0/332813041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdde8108910 0x7fdde8108cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.618+0000 7fdded400640 1 -- 172.21.15.31:0/332813041 >> 172.21.15.31:0/332813041 conn(0x7fdde806c750 msgr2=0x7fdde806cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:35.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.620+0000 7fdded400640 1 -- 172.21.15.31:0/332813041 shutdown_connections 2024-03-21T10:40:35.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.620+0000 7fdded400640 1 -- 172.21.15.31:0/332813041 wait complete. 2024-03-21T10:40:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.622+0000 7fdded400640 1 Processor -- start 2024-03-21T10:40:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.622+0000 7fdded400640 1 -- start start 2024-03-21T10:40:35.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.623+0000 7fdded400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdde8108910 0x7fdde8109d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.623+0000 7fdded400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdde810d3d0 0x7fdde810a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.623+0000 7fdded400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdde810ba30 0x7fdde810a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.623+0000 7fdded400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde8118030 con 0x7fdde810d3d0 2024-03-21T10:40:35.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.623+0000 7fdded400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde8117eb0 con 0x7fdde8108910 2024-03-21T10:40:35.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.623+0000 7fdded400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde81181b0 con 0x7fdde810ba30 2024-03-21T10:40:35.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.624+0000 7fdde6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdde8108910 0x7fdde8109d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.624+0000 7fdde5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdde810d3d0 0x7fdde810a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.624+0000 7fdde7200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdde810ba30 0x7fdde810a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.624+0000 7fdde7200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdde810ba30 0x7fdde810a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34512/0 (socket says 172.21.15.31:34512) 2024-03-21T10:40:35.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.624+0000 7fdde7200640 1 -- 172.21.15.31:0/590699532 learned_addr learned my addr 172.21.15.31:0/590699532 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.625+0000 7fdde7200640 1 -- 172.21.15.31:0/590699532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdde8108910 msgr2=0x7fdde8109d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.625+0000 7fdde7200640 1 --2- 172.21.15.31:0/590699532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdde8108910 0x7fdde8109d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.625+0000 7fdde7200640 1 -- 172.21.15.31:0/590699532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdde810d3d0 msgr2=0x7fdde810a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.625+0000 7fdde7200640 1 --2- 172.21.15.31:0/590699532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdde810d3d0 0x7fdde810a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.625+0000 7fdde7200640 1 -- 172.21.15.31:0/590699532 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde81a4a60 con 0x7fdde810ba30 2024-03-21T10:40:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.626+0000 7fdde7200640 1 --2- 172.21.15.31:0/590699532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdde810ba30 0x7fdde810a810 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fdde002c210 tx=0x7fdde002eaa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.628+0000 7fdddf400640 1 -- 172.21.15.31:0/590699532 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde0041020 con 0x7fdde810ba30 2024-03-21T10:40:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.628+0000 7fdddf400640 1 -- 172.21.15.31:0/590699532 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdde002ee20 con 0x7fdde810ba30 2024-03-21T10:40:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.628+0000 7fdddf400640 1 -- 172.21.15.31:0/590699532 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde0004ec0 con 0x7fdde810ba30 2024-03-21T10:40:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.629+0000 7fdded400640 1 -- 172.21.15.31:0/590699532 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde81a4cf0 con 0x7fdde810ba30 2024-03-21T10:40:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.630+0000 7fdded400640 1 -- 172.21.15.31:0/590699532 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde81a5190 con 0x7fdde810ba30 2024-03-21T10:40:35.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.635+0000 7fddf0220740 1 Processor -- start 2024-03-21T10:40:35.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.637+0000 7fdddf400640 1 -- 172.21.15.31:0/590699532 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fdde0044020 con 0x7fdde810ba30 2024-03-21T10:40:35.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.637+0000 7fddf0220740 1 -- start start 2024-03-21T10:40:35.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.639+0000 7fddf0220740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55579b104fd0 0x55579b1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.639+0000 7fddf0220740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579b1058f0 0x55579b10e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.639+0000 7fddf0220740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55579b10e6c0 0x55579b110ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.639+0000 7fddf0220740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55579b0c73b0 con 0x55579b1058f0 2024-03-21T10:40:35.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.639+0000 7fddf0220740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55579b069830 con 0x55579b104fd0 2024-03-21T10:40:35.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.639+0000 7fddf0220740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55579b0125d0 con 0x55579b10e6c0 2024-03-21T10:40:35.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.639+0000 7fdde5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55579b104fd0 0x55579b1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.639+0000 7fdde5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55579b104fd0 0x55579b1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:48956/0 (socket says 172.21.15.31:48956) 2024-03-21T10:40:35.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.639+0000 7fdde5e00640 1 -- 172.21.15.31:0/2919952320 learned_addr learned my addr 172.21.15.31:0/2919952320 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:35.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.639+0000 7fdde5e00640 1 --2- 172.21.15.31:0/2919952320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55579b10e6c0 0x55579b110ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.640+0000 7fdde5e00640 1 -- 172.21.15.31:0/2919952320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55579b10e6c0 msgr2=0x55579b110ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:35.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.640+0000 7fdde5e00640 1 --2- 172.21.15.31:0/2919952320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55579b10e6c0 0x55579b110ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.640+0000 7fdde5e00640 1 -- 172.21.15.31:0/2919952320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579b1058f0 msgr2=0x55579b10e180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:40:35.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.640+0000 7fdddf400640 1 --2- 172.21.15.31:0/590699532 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fddcc0a3210 0x7fddcc0a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.640+0000 7fdddf400640 1 -- 172.21.15.31:0/590699532 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fdde00f10c0 con 0x7fdde810ba30 2024-03-21T10:40:35.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.641+0000 7fdde6800640 1 --2- 172.21.15.31:0/2919952320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579b1058f0 0x55579b10e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.641+0000 7fdde5e00640 1 --2- 172.21.15.31:0/2919952320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579b1058f0 0x55579b10e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.641+0000 7fdde5e00640 1 -- 172.21.15.31:0/2919952320 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55579b02a2c0 con 0x55579b104fd0 2024-03-21T10:40:35.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.641+0000 7fdde6800640 1 --2- 172.21.15.31:0/2919952320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579b1058f0 0x55579b10e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:40:35.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.643+0000 7fdde5e00640 1 --2- 172.21.15.31:0/2919952320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55579b104fd0 0x55579b1053b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fddd400a640 tx=0x7fddd400a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d715104df38507b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:35.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.643+0000 7fdde5e00640 1 --2- 172.21.15.31:0/590699532 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fddcc0a3210 0x7fddcc0a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.644+0000 7fddd3400640 1 -- 172.21.15.31:0/2919952320 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd401a770 con 0x55579b104fd0 2024-03-21T10:40:35.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.644+0000 7fddd3400640 1 -- 172.21.15.31:0/2919952320 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fddd401adf0 con 0x55579b104fd0 2024-03-21T10:40:35.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.644+0000 7fddd3400640 1 -- 172.21.15.31:0/2919952320 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd4026d30 con 0x55579b104fd0 2024-03-21T10:40:35.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.645+0000 7fddf0220740 1 -- 172.21.15.31:0/2919952320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55579b104fd0 msgr2=0x55579b1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.645+0000 7fddf0220740 1 --2- 172.21.15.31:0/2919952320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55579b104fd0 0x55579b1053b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fddd400a640 tx=0x7fddd400a5d0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.645+0000 7fdde5e00640 1 --2- 172.21.15.31:0/590699532 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fddcc0a3210 0x7fddcc0a56d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fddd4023000 tx=0x7fddd4024000 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:35.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.645+0000 7fddf0220740 1 -- 172.21.15.31:0/2919952320 shutdown_connections 2024-03-21T10:40:35.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.645+0000 7fddf0220740 1 --2- 172.21.15.31:0/2919952320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55579b10e6c0 0x55579b110ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.645+0000 7fddf0220740 1 --2- 172.21.15.31:0/2919952320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579b1058f0 0x55579b10e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.645+0000 7fddf0220740 1 --2- 172.21.15.31:0/2919952320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55579b104fd0 0x55579b1053b0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.645+0000 7fddf0220740 1 -- 172.21.15.31:0/2919952320 >> 172.21.15.31:0/2919952320 conn(0x55579b0ff8f0 msgr2=0x55579b0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:35.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.645+0000 7fddf0220740 1 -- 172.21.15.31:0/2919952320 shutdown_connections 2024-03-21T10:40:35.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.645+0000 7fddf0220740 1 -- 172.21.15.31:0/2919952320 wait complete. 2024-03-21T10:40:35.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.646+0000 7fddf0220740 1 -- start start 2024-03-21T10:40:35.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.646+0000 7fddf0220740 1 Processor -- start 2024-03-21T10:40:35.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.647+0000 7fddf0220740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55579b104fd0 0x55579b10b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.647+0000 7fddf0220740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55579b1058f0 0x55579b10bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.647+0000 7fddf0220740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579b10e6c0 0x55579b10c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.647+0000 7fddf0220740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55579b025a20 con 0x55579b10e6c0 2024-03-21T10:40:35.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.647+0000 7fddf0220740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55579b0658d0 con 0x55579b104fd0 2024-03-21T10:40:35.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.647+0000 7fddf0220740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55579b1112f0 con 0x55579b1058f0 2024-03-21T10:40:35.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.647+0000 7fddf0220740 10 client.0 _ll_register_callbacks cb 0x55579b0621c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:35.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.648+0000 7fdde7200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579b10e6c0 0x55579b10c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.648+0000 7fdde5e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55579b1058f0 0x55579b10bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.648+0000 7fdde5e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55579b1058f0 0x55579b10bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34538/0 (socket says 172.21.15.31:34538) 2024-03-21T10:40:35.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.648+0000 7fdde5e00640 1 -- 172.21.15.31:0/506969836 learned_addr learned my addr 172.21.15.31:0/506969836 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:35.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.648+0000 7fdde6800640 1 --2- 172.21.15.31:0/506969836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55579b104fd0 0x55579b10b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.648+0000 7fdde5e00640 1 -- 172.21.15.31:0/506969836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55579b104fd0 msgr2=0x55579b10b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:35.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.648+0000 7fdde5e00640 1 --2- 172.21.15.31:0/506969836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55579b104fd0 0x55579b10b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.648+0000 7fdde5e00640 1 -- 172.21.15.31:0/506969836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579b10e6c0 msgr2=0x55579b10c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:35.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.648+0000 7fdde5e00640 1 --2- 172.21.15.31:0/506969836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579b10e6c0 0x55579b10c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.648+0000 7fdde5e00640 1 -- 172.21.15.31:0/506969836 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55579ad7e210 con 0x55579b1058f0 2024-03-21T10:40:35.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.648+0000 7fdde7200640 1 --2- 172.21.15.31:0/506969836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579b10e6c0 0x55579b10c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:35.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.649+0000 7fdde5e00640 1 --2- 172.21.15.31:0/506969836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55579b1058f0 0x55579b10bb70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fddd4026650 tx=0x7fddd4026680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=baa632da00130da9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:35.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.649+0000 7fddc7e00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:40:35.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.649+0000 7fddc7e00640 1 -- 172.21.15.31:0/506969836 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd4005240 con 0x55579b1058f0 2024-03-21T10:40:35.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.649+0000 7fddc7e00640 1 -- 172.21.15.31:0/506969836 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fddd40053e0 con 0x55579b1058f0 2024-03-21T10:40:35.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.649+0000 7fddf0220740 1 -- 172.21.15.31:0/506969836 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fddd401adf0 con 0x55579b1058f0 2024-03-21T10:40:35.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.649+0000 7fddc7e00640 1 -- 172.21.15.31:0/506969836 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fddd403c050 con 0x55579b1058f0 2024-03-21T10:40:35.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.650+0000 7fddf0220740 10 client.26149 fetch_fsmap learned FSMap version 25 2024-03-21T10:40:35.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.650+0000 7fddf0220740 1 -- 172.21.15.31:0/506969836 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55579ae4a620 con 0x55579b1058f0 2024-03-21T10:40:35.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.650+0000 7fddc7e00640 1 -- 172.21.15.31:0/506969836 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fddd401adf0 con 0x55579b1058f0 2024-03-21T10:40:35.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.650+0000 7fddc7e00640 1 -- 172.21.15.31:0/506969836 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fddd402d070 con 0x55579b1058f0 2024-03-21T10:40:35.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.650+0000 7fddf0220740 10 client.26149 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:40:35.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.650+0000 7fddf0220740 10 client.26149 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:40:35.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.650+0000 7fddf0220740 20 client.26149 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:35.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.650+0000 7fddf0220740 1 --2- 172.21.15.31:0/506969836 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55579b11e390 0x55579b120780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.651+0000 7fddf0220740 4 client.26149 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:40:35.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.651+0000 7fddf0220740 1 -- 172.21.15.31:0/506969836 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55579b076d70 con 0x55579b11e390 2024-03-21T10:40:35.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.651+0000 7fdde6800640 1 --2- 172.21.15.31:0/506969836 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55579b11e390 0x55579b120780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.651+0000 7fdde6800640 1 --2- 172.21.15.31:0/506969836 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55579b11e390 0x55579b120780 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fd366126059ebb server_cookie=40fa22b7c81338ca in_seq=0 out_seq=0 2024-03-21T10:40:35.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.651+0000 7fddc7e00640 10 client.26149 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:35.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.652+0000 7fddc7e00640 1 -- 172.21.15.31:0/506969836 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55579b076d70 con 0x55579b11e390 2024-03-21T10:40:35.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.652+0000 7fddc7e00640 10 client.26149 handle_command_reply: tid=0 2024-03-21T10:40:35.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.653+0000 7fddf0220740 1 client.26149 shutdown 2024-03-21T10:40:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.653+0000 7fddf0220740 1 -- 172.21.15.31:0/506969836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55579b1058f0 msgr2=0x55579b10bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.653+0000 7fddf0220740 1 --2- 172.21.15.31:0/506969836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55579b1058f0 0x55579b10bb70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fddd4026650 tx=0x7fddd4026680 comp rx=0 tx=0).stop 2024-03-21T10:40:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.653+0000 7fddf0220740 1 -- 172.21.15.31:0/506969836 shutdown_connections 2024-03-21T10:40:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.653+0000 7fddf0220740 1 --2- 172.21.15.31:0/506969836 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55579b11e390 0x55579b120780 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.653+0000 7fddc7e00640 0 client.26149 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.653+0000 7fddf0220740 1 --2- 172.21.15.31:0/506969836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579b10e6c0 0x55579b10c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.653+0000 7fddf0220740 1 --2- 172.21.15.31:0/506969836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55579b1058f0 0x55579b10bb70 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.653+0000 7fddf0220740 1 --2- 172.21.15.31:0/506969836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55579b104fd0 0x55579b10b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.653+0000 7fddf0220740 1 -- 172.21.15.31:0/506969836 >> 172.21.15.31:0/506969836 conn(0x55579b0ff8f0 msgr2=0x55579b101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:35.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.654+0000 7fddf0220740 1 -- 172.21.15.31:0/506969836 shutdown_connections 2024-03-21T10:40:35.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.654+0000 7fddf0220740 1 -- 172.21.15.31:0/506969836 wait complete. 2024-03-21T10:40:35.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.654+0000 7fddf0220740 20 client.26149 trim_cache size 0 max 16384 2024-03-21T10:40:35.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.691+0000 7fddf0220740 1 Processor -- start 2024-03-21T10:40:35.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.691+0000 7fddf0220740 1 -- start start 2024-03-21T10:40:35.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.692+0000 7fddf0220740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55579b104fd0 0x55579b10dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.693+0000 7fddf0220740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579b1058f0 0x55579b119a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.693+0000 7fddf0220740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55579b10e6c0 0x55579b119fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.693+0000 7fdde7200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55579b10e6c0 0x55579b119fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.694+0000 7fdde6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55579b104fd0 0x55579b10dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.694+0000 7fddf0220740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc8012110 con 0x55579b1058f0 2024-03-21T10:40:35.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.694+0000 7fddf0220740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc800fc10 con 0x55579b104fd0 2024-03-21T10:40:35.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.695+0000 7fdde7200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55579b10e6c0 0x55579b119fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34552/0 (socket says 172.21.15.31:34552) 2024-03-21T10:40:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.695+0000 7fdde7200640 1 -- 172.21.15.31:0/1493507555 learned_addr learned my addr 172.21.15.31:0/1493507555 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.695+0000 7fddf0220740 1 -- 172.21.15.31:0/1493507555 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc800dcb0 con 0x55579b10e6c0 2024-03-21T10:40:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.695+0000 7fdde5e00640 1 --2- 172.21.15.31:0/1493507555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579b1058f0 0x55579b119a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.696+0000 7fdde7200640 1 -- 172.21.15.31:0/1493507555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55579b104fd0 msgr2=0x55579b10dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:35.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.696+0000 7fdde7200640 1 --2- 172.21.15.31:0/1493507555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55579b104fd0 0x55579b10dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.696+0000 7fdde7200640 1 -- 172.21.15.31:0/1493507555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579b1058f0 msgr2=0x55579b119a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:35.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.696+0000 7fdde7200640 1 --2- 172.21.15.31:0/1493507555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579b1058f0 0x55579b119a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.697+0000 7fdde7200640 1 -- 172.21.15.31:0/1493507555 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55579b10a620 con 0x55579b10e6c0 2024-03-21T10:40:35.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.697+0000 7fdde6800640 1 --2- 172.21.15.31:0/1493507555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55579b104fd0 0x55579b10dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:35.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.697+0000 7fdde5e00640 1 --2- 172.21.15.31:0/1493507555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579b1058f0 0x55579b119a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.700+0000 7fdde7200640 1 --2- 172.21.15.31:0/1493507555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55579b10e6c0 0x55579b119fa0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdde00ee690 tx=0x7fdde00bd160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9adc7f951908cc1c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.702+0000 7fddd2a00640 1 -- 172.21.15.31:0/1493507555 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde00d5070 con 0x55579b10e6c0 2024-03-21T10:40:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.702+0000 7fddd2a00640 1 -- 172.21.15.31:0/1493507555 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdde00bd280 con 0x55579b10e6c0 2024-03-21T10:40:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.702+0000 7fddd2a00640 1 -- 172.21.15.31:0/1493507555 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde00ee750 con 0x55579b10e6c0 2024-03-21T10:40:35.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.703+0000 7fddf0220740 1 -- 172.21.15.31:0/1493507555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55579b10e6c0 msgr2=0x55579b119fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.704+0000 7fddf0220740 1 --2- 172.21.15.31:0/1493507555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55579b10e6c0 0x55579b119fa0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdde00ee690 tx=0x7fdde00bd160 comp rx=0 tx=0).stop 2024-03-21T10:40:35.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.705+0000 7fddf0220740 1 -- 172.21.15.31:0/1493507555 shutdown_connections 2024-03-21T10:40:35.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.705+0000 7fddf0220740 1 --2- 172.21.15.31:0/1493507555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55579b10e6c0 0x55579b119fa0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.706+0000 7fddf0220740 1 --2- 172.21.15.31:0/1493507555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579b1058f0 0x55579b119a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.706+0000 7fddf0220740 1 --2- 172.21.15.31:0/1493507555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55579b104fd0 0x55579b10dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.706+0000 7fddf0220740 1 -- 172.21.15.31:0/1493507555 >> 172.21.15.31:0/1493507555 conn(0x55579b0ff8f0 msgr2=0x55579b0fe950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.709+0000 7fddf0220740 1 -- 172.21.15.31:0/1493507555 shutdown_connections 2024-03-21T10:40:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.709+0000 7fddf0220740 1 -- 172.21.15.31:0/1493507555 wait complete. 2024-03-21T10:40:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.710+0000 7fddf0220740 1 -- start start 2024-03-21T10:40:35.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.710+0000 7fddf0220740 1 Processor -- start 2024-03-21T10:40:35.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.711+0000 7fddf0220740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55579b104fd0 0x55579b113570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.712+0000 7fddf0220740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55579b1058f0 0x55579b113ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.712+0000 7fddf0220740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579b10e6c0 0x55579b113ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.713+0000 7fdde7200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579b10e6c0 0x55579b113ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.713+0000 7fdde7200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579b10e6c0 0x55579b113ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39714/0 (socket says 172.21.15.31:39714) 2024-03-21T10:40:35.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.713+0000 7fdde5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55579b1058f0 0x55579b113ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.713+0000 7fdde7200640 1 -- 172.21.15.31:0/1444852899 learned_addr learned my addr 172.21.15.31:0/1444852899 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.713+0000 7fddf0220740 1 -- 172.21.15.31:0/1444852899 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc800e2b0 con 0x55579b10e6c0 2024-03-21T10:40:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.713+0000 7fddf0220740 1 -- 172.21.15.31:0/1444852899 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc800f5b0 con 0x55579b1058f0 2024-03-21T10:40:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.713+0000 7fddf0220740 1 -- 172.21.15.31:0/1444852899 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc800e8f0 con 0x55579b104fd0 2024-03-21T10:40:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.713+0000 7fdde7200640 1 -- 172.21.15.31:0/1444852899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55579b104fd0 msgr2=0x55579b113570 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:40:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.713+0000 7fdde7200640 1 --2- 172.21.15.31:0/1444852899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55579b104fd0 0x55579b113570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.714+0000 7fdde7200640 1 -- 172.21.15.31:0/1444852899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55579b1058f0 msgr2=0x55579b113ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.714+0000 7fdde7200640 1 --2- 172.21.15.31:0/1444852899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55579b1058f0 0x55579b113ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.714+0000 7fdde7200640 1 -- 172.21.15.31:0/1444852899 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55579b110e20 con 0x55579b10e6c0 2024-03-21T10:40:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.714+0000 7fddf0220740 10 client.0 _ll_register_callbacks cb 0x7fddd400adf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.714+0000 7fddf0220740 1 -- 172.21.15.31:0/1444852899 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdde00bd280 con 0x55579b10e6c0 2024-03-21T10:40:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.715+0000 7fddc7e00640 10 client.15969 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:40:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.715+0000 7fdde7200640 1 --2- 172.21.15.31:0/1444852899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579b10e6c0 0x55579b113ff0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdde002cc90 tx=0x7fdde00bb760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c6a807436872b2a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.716+0000 7fddc7e00640 1 -- 172.21.15.31:0/1444852899 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde009e020 con 0x55579b10e6c0 2024-03-21T10:40:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.716+0000 7fddc7e00640 1 -- 172.21.15.31:0/1444852899 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fdde00c1170 con 0x55579b10e6c0 2024-03-21T10:40:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.716+0000 7fddc7e00640 1 -- 172.21.15.31:0/1444852899 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdde009e020 con 0x55579b10e6c0 2024-03-21T10:40:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.717+0000 7fddf0220740 10 client.15969 fetch_fsmap learned FSMap version 25 2024-03-21T10:40:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.717+0000 7fddf0220740 1 -- 172.21.15.31:0/1444852899 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({fsmap=25}) v3 -- 0x55579b17a820 con 0x55579b10e6c0 2024-03-21T10:40:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.718+0000 7fddc7e00640 1 -- 172.21.15.31:0/1444852899 <== mon.0 v2:172.21.15.31:3300/0 4 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fdde00b98f0 con 0x55579b10e6c0 2024-03-21T10:40:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.718+0000 7fddf0220740 10 client.15969 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:40:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.719+0000 7fddf0220740 10 client.15969 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:40:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.719+0000 7fddf0220740 20 client.15969 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.719+0000 7fddf0220740 1 --2- 172.21.15.31:0/1444852899 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55579b17ade0 0x55579b17d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.719+0000 7fddf0220740 4 client.15969 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:40:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.719+0000 7fddf0220740 1 -- 172.21.15.31:0/1444852899 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55579b0710c0 con 0x55579b17ade0 2024-03-21T10:40:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.719+0000 7fdde6800640 1 --2- 172.21.15.31:0/1444852899 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55579b17ade0 0x55579b17d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.721+0000 7fdde6800640 1 --2- 172.21.15.31:0/1444852899 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55579b17ade0 0x55579b17d240 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15fde21f4619d3e server_cookie=5bf7dc0beb412467 in_seq=0 out_seq=0 2024-03-21T10:40:35.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.721+0000 7fddc7e00640 10 client.15969 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:35.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.757+0000 7fddc7e00640 1 -- 172.21.15.31:0/1444852899 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55579b0710c0 con 0x55579b17ade0 2024-03-21T10:40:35.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.757+0000 7fddc7e00640 10 client.15969 handle_command_reply: tid=0 2024-03-21T10:40:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.757+0000 7fddf0220740 1 client.15969 shutdown 2024-03-21T10:40:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.759+0000 7fddf0220740 1 -- 172.21.15.31:0/1444852899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579b10e6c0 msgr2=0x55579b113ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.759+0000 7fddf0220740 1 --2- 172.21.15.31:0/1444852899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579b10e6c0 0x55579b113ff0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdde002cc90 tx=0x7fdde00bb760 comp rx=0 tx=0).stop 2024-03-21T10:40:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.759+0000 7fddf0220740 1 -- 172.21.15.31:0/1444852899 shutdown_connections 2024-03-21T10:40:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.759+0000 7fddf0220740 1 --2- 172.21.15.31:0/1444852899 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55579b17ade0 0x55579b17d240 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.759+0000 7fddf0220740 1 --2- 172.21.15.31:0/1444852899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579b10e6c0 0x55579b113ff0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.759+0000 7fddf0220740 1 --2- 172.21.15.31:0/1444852899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55579b1058f0 0x55579b113ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.759+0000 7fddf0220740 1 --2- 172.21.15.31:0/1444852899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55579b104fd0 0x55579b113570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.759+0000 7fddf0220740 1 -- 172.21.15.31:0/1444852899 >> 172.21.15.31:0/1444852899 conn(0x55579b0ff8f0 msgr2=0x55579b10b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.760+0000 7fddc7e00640 0 client.15969 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.761+0000 7fddf0220740 1 -- 172.21.15.31:0/1444852899 shutdown_connections 2024-03-21T10:40:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.761+0000 7fddf0220740 1 -- 172.21.15.31:0/1444852899 wait complete. 2024-03-21T10:40:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.761+0000 7fddf0220740 20 client.15969 trim_cache size 0 max 16384 2024-03-21T10:40:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:40:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:40:35.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:40:35.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:40:35.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.765+0000 7fdded400640 1 -- 172.21.15.31:0/590699532 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fddcc0a3210 msgr2=0x7fddcc0a56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.765+0000 7fdded400640 1 --2- 172.21.15.31:0/590699532 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fddcc0a3210 0x7fddcc0a56d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fddd4023000 tx=0x7fddd4024000 comp rx=0 tx=0).stop 2024-03-21T10:40:35.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.765+0000 7fdded400640 1 -- 172.21.15.31:0/590699532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdde810ba30 msgr2=0x7fdde810a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.765+0000 7fdded400640 1 --2- 172.21.15.31:0/590699532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdde810ba30 0x7fdde810a810 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fdde002c210 tx=0x7fdde002eaa0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.765+0000 7fdded400640 1 -- 172.21.15.31:0/590699532 shutdown_connections 2024-03-21T10:40:35.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.765+0000 7fdded400640 1 --2- 172.21.15.31:0/590699532 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fddcc0a3210 0x7fddcc0a56d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.765+0000 7fdded400640 1 --2- 172.21.15.31:0/590699532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdde810ba30 0x7fdde810a810 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.765+0000 7fdded400640 1 --2- 172.21.15.31:0/590699532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdde810d3d0 0x7fdde810a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.765+0000 7fdded400640 1 --2- 172.21.15.31:0/590699532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdde8108910 0x7fdde8109d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.765+0000 7fdded400640 1 -- 172.21.15.31:0/590699532 >> 172.21.15.31:0/590699532 conn(0x7fdde806c750 msgr2=0x7fdde810c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:35.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.766+0000 7fdded400640 1 -- 172.21.15.31:0/590699532 shutdown_connections 2024-03-21T10:40:35.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:35.766+0000 7fdded400640 1 -- 172.21.15.31:0/590699532 wait complete. 2024-03-21T10:40:35.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.873+0000 7f40ca800640 1 -- 172.21.15.31:0/1691706976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40c41001b0 msgr2=0x7f40c4100590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.873+0000 7f40ca800640 1 --2- 172.21.15.31:0/1691706976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40c41001b0 0x7f40c4100590 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f40bc009ca0 tx=0x7f40bc02c140 comp rx=0 tx=0).stop 2024-03-21T10:40:35.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.874+0000 7f40ca800640 1 -- 172.21.15.31:0/1691706976 shutdown_connections 2024-03-21T10:40:35.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.874+0000 7f40ca800640 1 --2- 172.21.15.31:0/1691706976 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40c410dd60 0x7f40c4110150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.874+0000 7f40ca800640 1 --2- 172.21.15.31:0/1691706976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40c4100b60 0x7f40c410d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.874+0000 7f40ca800640 1 --2- 172.21.15.31:0/1691706976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40c41001b0 0x7f40c4100590 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.874+0000 7f40ca800640 1 -- 172.21.15.31:0/1691706976 >> 172.21.15.31:0/1691706976 conn(0x7f40c4077130 msgr2=0x7f40c40fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:35.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.874+0000 7f40ca800640 1 -- 172.21.15.31:0/1691706976 shutdown_connections 2024-03-21T10:40:35.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.875+0000 7f40ca800640 1 -- 172.21.15.31:0/1691706976 wait complete. 2024-03-21T10:40:35.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.876+0000 7f40ca800640 1 Processor -- start 2024-03-21T10:40:35.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.877+0000 7f40ca800640 1 -- start start 2024-03-21T10:40:35.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.877+0000 7f40ca800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40c41001b0 0x7f40c410d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.877+0000 7f40ca800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40c4100b60 0x7f40c41048a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.877+0000 7f40ca800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40c410dd60 0x7f40c4104de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.877+0000 7f40ca800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c41040e0 con 0x7f40c410dd60 2024-03-21T10:40:35.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.877+0000 7f40ca800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c4103f60 con 0x7f40c4100b60 2024-03-21T10:40:35.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.877+0000 7f40ca800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c4104260 con 0x7f40c41001b0 2024-03-21T10:40:35.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.878+0000 7f40c9400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40c41001b0 0x7f40c410d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.878+0000 7f40c8a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40c4100b60 0x7f40c41048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.878+0000 7f40c8a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40c4100b60 0x7f40c41048a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49004/0 (socket says 172.21.15.31:49004) 2024-03-21T10:40:35.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.878+0000 7f40c8a00640 1 -- 172.21.15.31:0/2895681553 learned_addr learned my addr 172.21.15.31:0/2895681553 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:35.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.878+0000 7f40c9e00640 1 --2- 172.21.15.31:0/2895681553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40c410dd60 0x7f40c4104de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.879+0000 7f40c9400640 1 -- 172.21.15.31:0/2895681553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40c4100b60 msgr2=0x7f40c41048a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.879+0000 7f40c9400640 1 --2- 172.21.15.31:0/2895681553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40c4100b60 0x7f40c41048a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.879+0000 7f40c9400640 1 -- 172.21.15.31:0/2895681553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40c410dd60 msgr2=0x7f40c4104de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.879+0000 7f40c9400640 1 --2- 172.21.15.31:0/2895681553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40c410dd60 0x7f40c4104de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.879+0000 7f40c9400640 1 -- 172.21.15.31:0/2895681553 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c4105560 con 0x7f40c41001b0 2024-03-21T10:40:35.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.880+0000 7f40c9400640 1 --2- 172.21.15.31:0/2895681553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40c41001b0 0x7f40c410d4f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f40bc02c750 tx=0x7f40bc034d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:35.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.881+0000 7f40c2000640 1 -- 172.21.15.31:0/2895681553 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40bc041020 con 0x7f40c41001b0 2024-03-21T10:40:35.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.881+0000 7f40c2000640 1 -- 172.21.15.31:0/2895681553 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40bc0327b0 con 0x7f40c41001b0 2024-03-21T10:40:35.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.881+0000 7f40c2000640 1 -- 172.21.15.31:0/2895681553 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40bc039040 con 0x7f40c41001b0 2024-03-21T10:40:35.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.882+0000 7f40ca800640 1 -- 172.21.15.31:0/2895681553 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c4108f30 con 0x7f40c41001b0 2024-03-21T10:40:35.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.883+0000 7f40ca800640 1 -- 172.21.15.31:0/2895681553 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c41093a0 con 0x7f40c41001b0 2024-03-21T10:40:35.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.890+0000 7f40cec35740 1 Processor -- start 2024-03-21T10:40:35.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.890+0000 7f40cec35740 1 -- start start 2024-03-21T10:40:35.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.891+0000 7f40cec35740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569ae1128b0 0x5569ae112c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.891+0000 7f40cec35740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569ae113260 0x5569ae11b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.891+0000 7f40cec35740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569ae11bde0 0x5569ae11e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.891+0000 7f40cec35740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569ae0762c0 con 0x5569ae1128b0 2024-03-21T10:40:35.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.891+0000 7f40cec35740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569ae01ec90 con 0x5569ae113260 2024-03-21T10:40:35.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.891+0000 7f40cec35740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569ae072360 con 0x5569ae11bde0 2024-03-21T10:40:35.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.891+0000 7f40c2000640 1 -- 172.21.15.31:0/2895681553 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f40bc044020 con 0x7f40c41001b0 2024-03-21T10:40:35.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.892+0000 7f40c9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569ae1128b0 0x5569ae112c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.892+0000 7f40c9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569ae1128b0 0x5569ae112c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39750/0 (socket says 172.21.15.31:39750) 2024-03-21T10:40:35.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.892+0000 7f40c9e00640 1 -- 172.21.15.31:0/3636387532 learned_addr learned my addr 172.21.15.31:0/3636387532 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:35.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.892+0000 7f40c9400640 1 --2- 172.21.15.31:0/3636387532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569ae113260 0x5569ae11b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.892+0000 7f40c8a00640 1 --2- 172.21.15.31:0/3636387532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569ae11bde0 0x5569ae11e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.893+0000 7f40c9e00640 1 -- 172.21.15.31:0/3636387532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569ae11bde0 msgr2=0x5569ae11e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:35.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.893+0000 7f40c9e00640 1 --2- 172.21.15.31:0/3636387532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569ae11bde0 0x5569ae11e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.893+0000 7f40c9e00640 1 -- 172.21.15.31:0/3636387532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569ae113260 msgr2=0x5569ae11b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:35.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.893+0000 7f40c9e00640 1 --2- 172.21.15.31:0/3636387532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569ae113260 0x5569ae11b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.893+0000 7f40c9e00640 1 -- 172.21.15.31:0/3636387532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569ae043080 con 0x5569ae1128b0 2024-03-21T10:40:35.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.893+0000 7f40c2000640 1 --2- 172.21.15.31:0/2895681553 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f40ac0a3290 0x7f40ac0a5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.894+0000 7f40c9e00640 1 --2- 172.21.15.31:0/3636387532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569ae1128b0 0x5569ae112c90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f40b000cb10 tx=0x7f40b000e350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e401331e85872b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:35.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.894+0000 7f40c2000640 1 -- 172.21.15.31:0/2895681553 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f40bc0f1440 con 0x7f40c41001b0 2024-03-21T10:40:35.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.894+0000 7f40c9400640 1 --2- 172.21.15.31:0/2895681553 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f40ac0a3290 0x7f40ac0a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.894+0000 7f40b6600640 1 -- 172.21.15.31:0/3636387532 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b0005020 con 0x5569ae1128b0 2024-03-21T10:40:35.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.895+0000 7f40b6600640 1 -- 172.21.15.31:0/3636387532 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40b00047a0 con 0x5569ae1128b0 2024-03-21T10:40:35.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.895+0000 7f40b6600640 1 -- 172.21.15.31:0/3636387532 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b0005020 con 0x5569ae1128b0 2024-03-21T10:40:35.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.896+0000 7f40cec35740 1 -- 172.21.15.31:0/3636387532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569ae1128b0 msgr2=0x5569ae112c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.896+0000 7f40cec35740 1 --2- 172.21.15.31:0/3636387532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569ae1128b0 0x5569ae112c90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f40b000cb10 tx=0x7f40b000e350 comp rx=0 tx=0).stop 2024-03-21T10:40:35.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.897+0000 7f40c9400640 1 --2- 172.21.15.31:0/2895681553 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f40ac0a3290 0x7f40ac0a5750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x5569ae112380 tx=0x7f40bc0f4e80 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:35.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.900+0000 7f40cec35740 1 -- 172.21.15.31:0/3636387532 shutdown_connections 2024-03-21T10:40:35.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.900+0000 7f40cec35740 1 --2- 172.21.15.31:0/3636387532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569ae11bde0 0x5569ae11e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.900+0000 7f40cec35740 1 --2- 172.21.15.31:0/3636387532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569ae113260 0x5569ae11b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.900+0000 7f40cec35740 1 --2- 172.21.15.31:0/3636387532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569ae1128b0 0x5569ae112c90 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.900+0000 7f40cec35740 1 -- 172.21.15.31:0/3636387532 >> 172.21.15.31:0/3636387532 conn(0x5569ae10cf80 msgr2=0x5569ae10d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:35.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.900+0000 7f40cec35740 1 -- 172.21.15.31:0/3636387532 shutdown_connections 2024-03-21T10:40:35.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.900+0000 7f40cec35740 1 -- 172.21.15.31:0/3636387532 wait complete. 2024-03-21T10:40:35.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.901+0000 7f40cec35740 1 -- start start 2024-03-21T10:40:35.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.901+0000 7f40cec35740 1 Processor -- start 2024-03-21T10:40:35.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.903+0000 7f40cec35740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569ae1128b0 0x5569ae11b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.903+0000 7f40cec35740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569ae113260 0x5569ae118a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.903+0000 7f40cec35740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569ae11bde0 0x5569ae119030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.903+0000 7f40cec35740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569ae11ed50 con 0x5569ae11bde0 2024-03-21T10:40:35.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.903+0000 7f40cec35740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569ae0320d0 con 0x5569ae113260 2024-03-21T10:40:35.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.903+0000 7f40cec35740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569ae11eed0 con 0x5569ae1128b0 2024-03-21T10:40:35.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.903+0000 7f40cec35740 10 client.0 _ll_register_callbacks cb 0x5569ae036e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:35.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.906+0000 7f40c9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569ae11bde0 0x5569ae119030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.906+0000 7f40c9e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569ae1128b0 0x5569ae11b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.906+0000 7f40c9e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569ae1128b0 0x5569ae11b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34592/0 (socket says 172.21.15.31:34592) 2024-03-21T10:40:35.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.906+0000 7f40c9e00640 1 -- 172.21.15.31:0/690540985 learned_addr learned my addr 172.21.15.31:0/690540985 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:35.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.907+0000 7f40c8a00640 1 --2- 172.21.15.31:0/690540985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569ae113260 0x5569ae118a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.908+0000 7f40c9e00640 1 -- 172.21.15.31:0/690540985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569ae1128b0 msgr2=0x5569ae11b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:35.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.908+0000 7f40c9e00640 1 --2- 172.21.15.31:0/690540985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569ae1128b0 0x5569ae11b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.908+0000 7f40c9e00640 1 -- 172.21.15.31:0/690540985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569ae113260 msgr2=0x5569ae118a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:35.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.908+0000 7f40c9e00640 1 --2- 172.21.15.31:0/690540985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569ae113260 0x5569ae118a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.908+0000 7f40c9e00640 1 -- 172.21.15.31:0/690540985 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5569add8a3f0 con 0x5569ae11bde0 2024-03-21T10:40:35.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.909+0000 7f40c9e00640 1 --2- 172.21.15.31:0/690540985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569ae11bde0 0x5569ae119030 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f40b0005940 tx=0x7f40b000efa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77e8eb8159d91b2d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:35.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.909+0000 7f40aaa00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:40:35.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.910+0000 7f40aaa00640 1 -- 172.21.15.31:0/690540985 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b0025040 con 0x5569ae11bde0 2024-03-21T10:40:35.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.910+0000 7f40aaa00640 1 -- 172.21.15.31:0/690540985 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f40b00053d0 con 0x5569ae11bde0 2024-03-21T10:40:35.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.910+0000 7f40cec35740 1 -- 172.21.15.31:0/690540985 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f40b00047a0 con 0x5569ae11bde0 2024-03-21T10:40:35.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.911+0000 7f40aaa00640 1 -- 172.21.15.31:0/690540985 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40b0031050 con 0x5569ae11bde0 2024-03-21T10:40:35.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.911+0000 7f40cec35740 10 client.15981 fetch_fsmap learned FSMap version 25 2024-03-21T10:40:35.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.911+0000 7f40cec35740 1 -- 172.21.15.31:0/690540985 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5569ade547d0 con 0x5569ae11bde0 2024-03-21T10:40:35.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.912+0000 7f40aaa00640 1 -- 172.21.15.31:0/690540985 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40b00047a0 con 0x5569ae11bde0 2024-03-21T10:40:35.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.912+0000 7f40aaa00640 1 -- 172.21.15.31:0/690540985 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f40b001a410 con 0x5569ae11bde0 2024-03-21T10:40:35.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.912+0000 7f40cec35740 10 client.15981 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:40:35.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.912+0000 7f40cec35740 10 client.15981 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:40:35.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.912+0000 7f40cec35740 20 client.15981 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:35.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.912+0000 7f40cec35740 1 --2- 172.21.15.31:0/690540985 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x5569ae12bba0 0x5569ae12df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.912+0000 7f40cec35740 4 client.15981 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:40:35.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.912+0000 7f40cec35740 1 -- 172.21.15.31:0/690540985 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5569ae083800 con 0x5569ae12bba0 2024-03-21T10:40:35.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.913+0000 7f40c8a00640 1 --2- 172.21.15.31:0/690540985 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x5569ae12bba0 0x5569ae12df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.916+0000 7f40c8a00640 1 --2- 172.21.15.31:0/690540985 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x5569ae12bba0 0x5569ae12df90 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1ad19cb603c8cbc5 server_cookie=c6d57854364b20b9 in_seq=0 out_seq=0 2024-03-21T10:40:35.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.916+0000 7f40aaa00640 10 client.15981 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:40:35.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.918+0000 7f40aaa00640 1 -- 172.21.15.31:0/690540985 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5569ae083800 con 0x5569ae12bba0 2024-03-21T10:40:35.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.918+0000 7f40aaa00640 10 client.15981 handle_command_reply: tid=0 2024-03-21T10:40:35.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.918+0000 7f40cec35740 1 client.15981 shutdown 2024-03-21T10:40:35.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.919+0000 7f40cec35740 1 -- 172.21.15.31:0/690540985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569ae11bde0 msgr2=0x5569ae119030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.919+0000 7f40cec35740 1 --2- 172.21.15.31:0/690540985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569ae11bde0 0x5569ae119030 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f40b0005940 tx=0x7f40b000efa0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.919+0000 7f40cec35740 1 -- 172.21.15.31:0/690540985 shutdown_connections 2024-03-21T10:40:35.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.919+0000 7f40cec35740 1 --2- 172.21.15.31:0/690540985 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x5569ae12bba0 0x5569ae12df90 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.919+0000 7f40aaa00640 0 client.15981 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:40:35.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.919+0000 7f40cec35740 1 --2- 172.21.15.31:0/690540985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569ae11bde0 0x5569ae119030 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.920+0000 7f40cec35740 1 --2- 172.21.15.31:0/690540985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569ae113260 0x5569ae118a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.920+0000 7f40cec35740 1 --2- 172.21.15.31:0/690540985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569ae1128b0 0x5569ae11b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.920+0000 7f40cec35740 1 -- 172.21.15.31:0/690540985 >> 172.21.15.31:0/690540985 conn(0x5569ae10cf80 msgr2=0x5569ae10ecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:35.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.921+0000 7f40cec35740 1 -- 172.21.15.31:0/690540985 shutdown_connections 2024-03-21T10:40:35.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.921+0000 7f40cec35740 1 -- 172.21.15.31:0/690540985 wait complete. 2024-03-21T10:40:35.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.922+0000 7f40cec35740 20 client.15981 trim_cache size 0 max 16384 2024-03-21T10:40:35.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.953+0000 7f40cec35740 1 Processor -- start 2024-03-21T10:40:35.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.953+0000 7f40cec35740 1 -- start start 2024-03-21T10:40:35.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.954+0000 7f40cec35740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569ae1128b0 0x5569ae11b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.955+0000 7f40c9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569ae1128b0 0x5569ae11b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.955+0000 7f40cec35740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569ae113260 0x5569ae11b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.955+0000 7f40c9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569ae1128b0 0x5569ae11b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39770/0 (socket says 172.21.15.31:39770) 2024-03-21T10:40:35.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.955+0000 7f40c9e00640 1 -- 172.21.15.31:0/1940782743 learned_addr learned my addr 172.21.15.31:0/1940782743 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:35.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.956+0000 7f40cec35740 1 --2- 172.21.15.31:0/1940782743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569ae11bde0 0x5569ae115d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.956+0000 7f40cec35740 1 -- 172.21.15.31:0/1940782743 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c41bf8e0 con 0x5569ae1128b0 2024-03-21T10:40:35.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.956+0000 7f40cec35740 1 -- 172.21.15.31:0/1940782743 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c41bd3e0 con 0x5569ae11bde0 2024-03-21T10:40:35.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.956+0000 7f40cec35740 1 -- 172.21.15.31:0/1940782743 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c41bb480 con 0x5569ae113260 2024-03-21T10:40:35.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.956+0000 7f40c8a00640 1 --2- 172.21.15.31:0/1940782743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569ae113260 0x5569ae11b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.957+0000 7f40c9e00640 1 -- 172.21.15.31:0/1940782743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569ae113260 msgr2=0x5569ae11b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:35.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.957+0000 7f40c9e00640 1 --2- 172.21.15.31:0/1940782743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569ae113260 0x5569ae11b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.957+0000 7f40c9e00640 1 -- 172.21.15.31:0/1940782743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569ae11bde0 msgr2=0x5569ae115d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:40:35.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.957+0000 7f40c9e00640 1 --2- 172.21.15.31:0/1940782743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569ae11bde0 0x5569ae115d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.957+0000 7f40c9e00640 1 -- 172.21.15.31:0/1940782743 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569ae12e3f0 con 0x5569ae1128b0 2024-03-21T10:40:35.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.957+0000 7f40c9e00640 1 --2- 172.21.15.31:0/1940782743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569ae1128b0 0x5569ae11b1c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f40b0000c00 tx=0x7f40b000e8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15dfc6835545049a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:35.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.958+0000 7f40b5c00640 1 -- 172.21.15.31:0/1940782743 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b000c660 con 0x5569ae1128b0 2024-03-21T10:40:35.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.958+0000 7f40b5c00640 1 -- 172.21.15.31:0/1940782743 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40b0004cd0 con 0x5569ae1128b0 2024-03-21T10:40:35.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.958+0000 7f40b5c00640 1 -- 172.21.15.31:0/1940782743 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b0004290 con 0x5569ae1128b0 2024-03-21T10:40:35.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.959+0000 7f40cec35740 1 -- 172.21.15.31:0/1940782743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569ae1128b0 msgr2=0x5569ae11b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.960+0000 7f40cec35740 1 --2- 172.21.15.31:0/1940782743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569ae1128b0 0x5569ae11b1c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f40b0000c00 tx=0x7f40b000e8d0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.960+0000 7f40cec35740 1 -- 172.21.15.31:0/1940782743 shutdown_connections 2024-03-21T10:40:35.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.960+0000 7f40cec35740 1 --2- 172.21.15.31:0/1940782743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569ae11bde0 0x5569ae115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.960+0000 7f40cec35740 1 --2- 172.21.15.31:0/1940782743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569ae113260 0x5569ae11b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.960+0000 7f40cec35740 1 --2- 172.21.15.31:0/1940782743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569ae1128b0 0x5569ae11b1c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.960+0000 7f40cec35740 1 -- 172.21.15.31:0/1940782743 >> 172.21.15.31:0/1940782743 conn(0x5569ae10cf80 msgr2=0x5569ae10c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:35.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.961+0000 7f40cec35740 1 -- 172.21.15.31:0/1940782743 shutdown_connections 2024-03-21T10:40:35.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.961+0000 7f40cec35740 1 -- 172.21.15.31:0/1940782743 wait complete. 2024-03-21T10:40:35.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.962+0000 7f40cec35740 1 -- start start 2024-03-21T10:40:35.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.962+0000 7f40cec35740 1 Processor -- start 2024-03-21T10:40:35.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.963+0000 7f40cec35740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569ae1128b0 0x5569ae121b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.964+0000 7f40cec35740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569ae113260 0x5569ae1220a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.964+0000 7f40c9e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569ae1128b0 0x5569ae121b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.964+0000 7f40c9e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569ae1128b0 0x5569ae121b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34616/0 (socket says 172.21.15.31:34616) 2024-03-21T10:40:35.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.964+0000 7f40c9e00640 1 -- 172.21.15.31:0/2173696223 learned_addr learned my addr 172.21.15.31:0/2173696223 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:35.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.964+0000 7f40cec35740 1 --2- 172.21.15.31:0/2173696223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569ae11bde0 0x5569ae186250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.964+0000 7f40cec35740 1 -- 172.21.15.31:0/2173696223 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c41bba80 con 0x5569ae11bde0 2024-03-21T10:40:35.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.964+0000 7f40cec35740 1 -- 172.21.15.31:0/2173696223 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c41bcd80 con 0x5569ae113260 2024-03-21T10:40:35.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.964+0000 7f40cec35740 1 -- 172.21.15.31:0/2173696223 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c41bc0c0 con 0x5569ae1128b0 2024-03-21T10:40:35.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.965+0000 7f40cec35740 10 client.0 _ll_register_callbacks cb 0x7f40bc0f4d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:35.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.965+0000 7f40c8a00640 1 --2- 172.21.15.31:0/2173696223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569ae113260 0x5569ae1220a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.966+0000 7f40c9e00640 1 -- 172.21.15.31:0/2173696223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569ae113260 msgr2=0x5569ae1220a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:35.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.966+0000 7f40c9e00640 1 --2- 172.21.15.31:0/2173696223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569ae113260 0x5569ae1220a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.966+0000 7f40c9e00640 1 -- 172.21.15.31:0/2173696223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569ae11bde0 msgr2=0x5569ae186250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:35.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.966+0000 7f40c9e00640 1 --2- 172.21.15.31:0/2173696223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569ae11bde0 0x5569ae186250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.967+0000 7f40c9e00640 1 -- 172.21.15.31:0/2173696223 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5569ae186790 con 0x5569ae1128b0 2024-03-21T10:40:35.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.967+0000 7f40c8a00640 1 --2- 172.21.15.31:0/2173696223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569ae113260 0x5569ae1220a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:35.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.968+0000 7f40c9e00640 1 --2- 172.21.15.31:0/2173696223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569ae1128b0 0x5569ae121b60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f40b0005940 tx=0x7f40b001ae70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cef18649699d010 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:35.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.968+0000 7f40aaa00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:40:35.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.968+0000 7f40aaa00640 1 -- 172.21.15.31:0/2173696223 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b000c6e0 con 0x5569ae1128b0 2024-03-21T10:40:35.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.969+0000 7f40aaa00640 1 -- 172.21.15.31:0/2173696223 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f40b001aae0 con 0x5569ae1128b0 2024-03-21T10:40:35.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.969+0000 7f40cec35740 1 -- 172.21.15.31:0/2173696223 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f40b0004cd0 con 0x5569ae1128b0 2024-03-21T10:40:35.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.970+0000 7f40aaa00640 1 -- 172.21.15.31:0/2173696223 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40b00053c0 con 0x5569ae1128b0 2024-03-21T10:40:35.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.970+0000 7f40cec35740 10 client.26167 fetch_fsmap learned FSMap version 25 2024-03-21T10:40:35.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.970+0000 7f40cec35740 1 -- 172.21.15.31:0/2173696223 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5569ae187970 con 0x5569ae1128b0 2024-03-21T10:40:35.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.972+0000 7f40aaa00640 1 -- 172.21.15.31:0/2173696223 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40b0004cd0 con 0x5569ae1128b0 2024-03-21T10:40:35.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.975+0000 7f40aaa00640 1 -- 172.21.15.31:0/2173696223 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f40b0042460 con 0x5569ae1128b0 2024-03-21T10:40:35.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.975+0000 7f40cec35740 10 client.26167 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:40:35.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.975+0000 7f40cec35740 10 client.26167 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:40:35.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.975+0000 7f40cec35740 20 client.26167 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:35.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.975+0000 7f40cec35740 1 --2- 172.21.15.31:0/2173696223 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x5569ae12bba0 0x5569ae189e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:35.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.976+0000 7f40cec35740 4 client.26167 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:40:35.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.976+0000 7f40cec35740 1 -- 172.21.15.31:0/2173696223 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5569ae07db50 con 0x5569ae12bba0 2024-03-21T10:40:35.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.990+0000 7f40c8a00640 1 --2- 172.21.15.31:0/2173696223 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x5569ae12bba0 0x5569ae189e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:35.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.991+0000 7f40c8a00640 1 --2- 172.21.15.31:0/2173696223 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x5569ae12bba0 0x5569ae189e40 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dbb1a619e95cc35b server_cookie=75ab3b63a2e4d634 in_seq=0 out_seq=0 2024-03-21T10:40:35.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.991+0000 7f40aaa00640 10 client.26167 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:40:35.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.991+0000 7f40aaa00640 1 -- 172.21.15.31:0/2173696223 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5569ae07db50 con 0x5569ae12bba0 2024-03-21T10:40:35.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.991+0000 7f40aaa00640 10 client.26167 handle_command_reply: tid=0 2024-03-21T10:40:35.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.991+0000 7f40cec35740 1 client.26167 shutdown 2024-03-21T10:40:35.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.992+0000 7f40cec35740 1 -- 172.21.15.31:0/2173696223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569ae1128b0 msgr2=0x5569ae121b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.992+0000 7f40cec35740 1 --2- 172.21.15.31:0/2173696223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569ae1128b0 0x5569ae121b60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f40b0005940 tx=0x7f40b001ae70 comp rx=0 tx=0).stop 2024-03-21T10:40:35.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.992+0000 7f40cec35740 1 -- 172.21.15.31:0/2173696223 shutdown_connections 2024-03-21T10:40:35.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.992+0000 7f40cec35740 1 --2- 172.21.15.31:0/2173696223 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x5569ae12bba0 0x5569ae189e40 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.992+0000 7f40cec35740 1 --2- 172.21.15.31:0/2173696223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569ae11bde0 0x5569ae186250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.992+0000 7f40cec35740 1 --2- 172.21.15.31:0/2173696223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569ae113260 0x5569ae1220a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.992+0000 7f40cec35740 1 --2- 172.21.15.31:0/2173696223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569ae1128b0 0x5569ae121b60 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.992+0000 7f40cec35740 1 -- 172.21.15.31:0/2173696223 >> 172.21.15.31:0/2173696223 conn(0x5569ae10cf80 msgr2=0x5569ae10b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:35.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.992+0000 7f40aaa00640 0 client.26167 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:40:35.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.992+0000 7f40cec35740 1 -- 172.21.15.31:0/2173696223 shutdown_connections 2024-03-21T10:40:35.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.992+0000 7f40cec35740 1 -- 172.21.15.31:0/2173696223 wait complete. 2024-03-21T10:40:35.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.993+0000 7f40cec35740 20 client.26167 trim_cache size 0 max 16384 2024-03-21T10:40:35.991 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:40:35.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.994+0000 7f40ca800640 1 -- 172.21.15.31:0/2895681553 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f40ac0a3290 msgr2=0x7f40ac0a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.994+0000 7f40ca800640 1 --2- 172.21.15.31:0/2895681553 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f40ac0a3290 0x7f40ac0a5750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x5569ae112380 tx=0x7f40bc0f4e80 comp rx=0 tx=0).stop 2024-03-21T10:40:35.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.994+0000 7f40ca800640 1 -- 172.21.15.31:0/2895681553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40c41001b0 msgr2=0x7f40c410d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:35.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.994+0000 7f40ca800640 1 --2- 172.21.15.31:0/2895681553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40c41001b0 0x7f40c410d4f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f40bc02c750 tx=0x7f40bc034d00 comp rx=0 tx=0).stop 2024-03-21T10:40:35.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.994+0000 7f40ca800640 1 -- 172.21.15.31:0/2895681553 shutdown_connections 2024-03-21T10:40:35.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.994+0000 7f40ca800640 1 --2- 172.21.15.31:0/2895681553 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f40ac0a3290 0x7f40ac0a5750 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.994+0000 7f40ca800640 1 --2- 172.21.15.31:0/2895681553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40c410dd60 0x7f40c4104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.994+0000 7f40ca800640 1 --2- 172.21.15.31:0/2895681553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40c4100b60 0x7f40c41048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.994+0000 7f40ca800640 1 --2- 172.21.15.31:0/2895681553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40c41001b0 0x7f40c410d4f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:35.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.994+0000 7f40ca800640 1 -- 172.21.15.31:0/2895681553 >> 172.21.15.31:0/2895681553 conn(0x7f40c4077130 msgr2=0x7f40c40fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:35.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.997+0000 7f40ca800640 1 -- 172.21.15.31:0/2895681553 shutdown_connections 2024-03-21T10:40:35.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:35.997+0000 7f40ca800640 1 -- 172.21.15.31:0/2895681553 wait complete. 2024-03-21T10:40:36.067 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:40:36.067 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-21T10:40:36.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:35 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2647804826' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:40:36.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:35 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2647804826' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:40:36.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:35 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2647804826' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:40:36.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.372+0000 7f3826200640 1 -- 172.21.15.31:0/2370755849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3820107640 msgr2=0x7f3820107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:36.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.372+0000 7f3826200640 1 --2- 172.21.15.31:0/2370755849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3820107640 0x7f3820107a20 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3810009930 tx=0x7f381002bc20 comp rx=0 tx=0).stop 2024-03-21T10:40:36.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.372+0000 7f3826200640 1 -- 172.21.15.31:0/2370755849 shutdown_connections 2024-03-21T10:40:36.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.372+0000 7f3826200640 1 --2- 172.21.15.31:0/2370755849 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3820101ff0 0x7f382010e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.372+0000 7f3826200640 1 --2- 172.21.15.31:0/2370755849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3820101630 0x7f3820101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.372+0000 7f3826200640 1 --2- 172.21.15.31:0/2370755849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3820107640 0x7f3820107a20 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.372+0000 7f3826200640 1 -- 172.21.15.31:0/2370755849 >> 172.21.15.31:0/2370755849 conn(0x7f38200fd140 msgr2=0x7f38200ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:36.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.372+0000 7f3826200640 1 -- 172.21.15.31:0/2370755849 shutdown_connections 2024-03-21T10:40:36.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.372+0000 7f3826200640 1 -- 172.21.15.31:0/2370755849 wait complete. 2024-03-21T10:40:36.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.373+0000 7f3826200640 1 Processor -- start 2024-03-21T10:40:36.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.373+0000 7f3826200640 1 -- start start 2024-03-21T10:40:36.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.374+0000 7f3826200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3820101630 0x7f382019a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:36.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.374+0000 7f3826200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3820101ff0 0x7f382019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:36.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.374+0000 7f3826200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3820107640 0x7f382019ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:36.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.374+0000 7f3826200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3820111aa0 con 0x7f3820101ff0 2024-03-21T10:40:36.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.374+0000 7f3826200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3820111920 con 0x7f3820107640 2024-03-21T10:40:36.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.374+0000 7f3826200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3820111c20 con 0x7f3820101630 2024-03-21T10:40:36.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.374+0000 7f381fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3820107640 0x7f382019ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:36.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.374+0000 7f381f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3820101630 0x7f382019a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:36.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.374+0000 7f381fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3820107640 0x7f382019ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49058/0 (socket says 172.21.15.31:49058) 2024-03-21T10:40:36.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.374+0000 7f381f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3820101630 0x7f382019a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34626/0 (socket says 172.21.15.31:34626) 2024-03-21T10:40:36.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.374+0000 7f381fe00640 1 -- 172.21.15.31:0/1961338429 learned_addr learned my addr 172.21.15.31:0/1961338429 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:36.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.374+0000 7f381ea00640 1 --2- 172.21.15.31:0/1961338429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3820101ff0 0x7f382019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:36.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.375+0000 7f381fe00640 1 -- 172.21.15.31:0/1961338429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3820101630 msgr2=0x7f382019a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:36.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.375+0000 7f381fe00640 1 --2- 172.21.15.31:0/1961338429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3820101630 0x7f382019a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.375+0000 7f381fe00640 1 -- 172.21.15.31:0/1961338429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3820101ff0 msgr2=0x7f382019a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:36.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.375+0000 7f381fe00640 1 --2- 172.21.15.31:0/1961338429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3820101ff0 0x7f382019a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.375+0000 7f381fe00640 1 -- 172.21.15.31:0/1961338429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f382019f170 con 0x7f3820107640 2024-03-21T10:40:36.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.375+0000 7f381ea00640 1 --2- 172.21.15.31:0/1961338429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3820101ff0 0x7f382019a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:36.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.375+0000 7f381fe00640 1 --2- 172.21.15.31:0/1961338429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3820107640 0x7f382019ea40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f381800eb10 tx=0x7f381800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:36.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.375+0000 7f3817e00640 1 -- 172.21.15.31:0/1961338429 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3818004b20 con 0x7f3820107640 2024-03-21T10:40:36.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.375+0000 7f3817e00640 1 -- 172.21.15.31:0/1961338429 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3818004cc0 con 0x7f3820107640 2024-03-21T10:40:36.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.375+0000 7f3826200640 1 -- 172.21.15.31:0/1961338429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f382019f460 con 0x7f3820107640 2024-03-21T10:40:36.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.375+0000 7f3817e00640 1 -- 172.21.15.31:0/1961338429 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3818010790 con 0x7f3820107640 2024-03-21T10:40:36.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.375+0000 7f3826200640 1 -- 172.21.15.31:0/1961338429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38201a6cf0 con 0x7f3820107640 2024-03-21T10:40:36.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.380+0000 7f3828f6c740 1 Processor -- start 2024-03-21T10:40:36.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.380+0000 7f3817e00640 1 -- 172.21.15.31:0/1961338429 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f3818010930 con 0x7f3820107640 2024-03-21T10:40:36.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.381+0000 7f3828f6c740 1 -- start start 2024-03-21T10:40:36.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.381+0000 7f3828f6c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe441ae6d0 0x55fe441aeab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:36.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.381+0000 7f3828f6c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe441af080 0x55fe441b7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:36.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.381+0000 7f3828f6c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe441b7c00 0x55fe441ba050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:36.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.381+0000 7f3828f6c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe441122c0 con 0x55fe441ae6d0 2024-03-21T10:40:36.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.381+0000 7f3828f6c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe440bac90 con 0x55fe441b7c00 2024-03-21T10:40:36.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.381+0000 7f3828f6c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe4410e360 con 0x55fe441af080 2024-03-21T10:40:36.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.382+0000 7f381ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe441ae6d0 0x55fe441aeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:36.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.382+0000 7f381ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe441ae6d0 0x55fe441aeab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39790/0 (socket says 172.21.15.31:39790) 2024-03-21T10:40:36.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.382+0000 7f381ea00640 1 -- 172.21.15.31:0/2920255715 learned_addr learned my addr 172.21.15.31:0/2920255715 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:36.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.382+0000 7f381ea00640 1 --2- 172.21.15.31:0/2920255715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe441b7c00 0x55fe441ba050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:36.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.382+0000 7f3817e00640 1 --2- 172.21.15.31:0/1961338429 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f38040a3190 0x7f38040a5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:36.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.382+0000 7f381ea00640 1 -- 172.21.15.31:0/2920255715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe441af080 msgr2=0x55fe441b7630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:40:36.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.382+0000 7f381ea00640 1 --2- 172.21.15.31:0/2920255715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe441af080 0x55fe441b7630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.382+0000 7f381ea00640 1 -- 172.21.15.31:0/2920255715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe441b7c00 msgr2=0x55fe441ba050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:36.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.382+0000 7f381ea00640 1 --2- 172.21.15.31:0/2920255715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe441b7c00 0x55fe441ba050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.382+0000 7f381ea00640 1 -- 172.21.15.31:0/2920255715 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe440df080 con 0x55fe441ae6d0 2024-03-21T10:40:36.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.383+0000 7f3817e00640 1 -- 172.21.15.31:0/1961338429 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f38180cfb10 con 0x7f3820107640 2024-03-21T10:40:36.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.383+0000 7f381fe00640 1 --2- 172.21.15.31:0/1961338429 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f38040a3190 0x7f38040a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:36.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.383+0000 7f381ea00640 1 --2- 172.21.15.31:0/2920255715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe441ae6d0 0x55fe441aeab0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f380c00bf70 tx=0x7f380c00b500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0da5b2cdbc9249b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:36.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.384+0000 7f380be00640 1 -- 172.21.15.31:0/2920255715 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380c0140f0 con 0x55fe441ae6d0 2024-03-21T10:40:36.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.384+0000 7f381fe00640 1 --2- 172.21.15.31:0/1961338429 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f38040a3190 0x7f38040a5650 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f38180049d0 tx=0x7f381800c950 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:36.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.384+0000 7f380be00640 1 -- 172.21.15.31:0/2920255715 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f380c005020 con 0x55fe441ae6d0 2024-03-21T10:40:36.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.384+0000 7f380be00640 1 -- 172.21.15.31:0/2920255715 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380c004a00 con 0x55fe441ae6d0 2024-03-21T10:40:36.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.385+0000 7f3828f6c740 1 -- 172.21.15.31:0/2920255715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe441ae6d0 msgr2=0x55fe441aeab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:36.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.385+0000 7f3828f6c740 1 --2- 172.21.15.31:0/2920255715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe441ae6d0 0x55fe441aeab0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f380c00bf70 tx=0x7f380c00b500 comp rx=0 tx=0).stop 2024-03-21T10:40:36.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.385+0000 7f3828f6c740 1 -- 172.21.15.31:0/2920255715 shutdown_connections 2024-03-21T10:40:36.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.385+0000 7f3828f6c740 1 --2- 172.21.15.31:0/2920255715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe441b7c00 0x55fe441ba050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.385+0000 7f3828f6c740 1 --2- 172.21.15.31:0/2920255715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe441af080 0x55fe441b7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.385+0000 7f3828f6c740 1 --2- 172.21.15.31:0/2920255715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe441ae6d0 0x55fe441aeab0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.385+0000 7f3828f6c740 1 -- 172.21.15.31:0/2920255715 >> 172.21.15.31:0/2920255715 conn(0x55fe441a8da0 msgr2=0x55fe441a9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:36.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.385+0000 7f3828f6c740 1 -- 172.21.15.31:0/2920255715 shutdown_connections 2024-03-21T10:40:36.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.385+0000 7f3828f6c740 1 -- 172.21.15.31:0/2920255715 wait complete. 2024-03-21T10:40:36.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.385+0000 7f3828f6c740 1 -- start start 2024-03-21T10:40:36.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.386+0000 7f3828f6c740 1 Processor -- start 2024-03-21T10:40:36.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.386+0000 7f3828f6c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe441ae6d0 0x55fe441b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:36.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.386+0000 7f3828f6c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe441af080 0x55fe441b4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:36.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.386+0000 7f3828f6c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe441b7c00 0x55fe441b4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:36.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.386+0000 7f3828f6c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe441bab70 con 0x55fe441ae6d0 2024-03-21T10:40:36.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.386+0000 7f3828f6c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe440ce0d0 con 0x55fe441af080 2024-03-21T10:40:36.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.386+0000 7f3828f6c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe441bacf0 con 0x55fe441b7c00 2024-03-21T10:40:36.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.386+0000 7f3828f6c740 10 client.0 _ll_register_callbacks cb 0x55fe440d2e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:36.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.387+0000 7f381ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe441ae6d0 0x55fe441b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:36.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.387+0000 7f381ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe441ae6d0 0x55fe441b74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39794/0 (socket says 172.21.15.31:39794) 2024-03-21T10:40:36.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.387+0000 7f381ea00640 1 -- 172.21.15.31:0/3762637249 learned_addr learned my addr 172.21.15.31:0/3762637249 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:36.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.387+0000 7f381f400640 1 --2- 172.21.15.31:0/3762637249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe441af080 0x55fe441b4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:36.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.387+0000 7f381ea00640 1 --2- 172.21.15.31:0/3762637249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe441b7c00 0x55fe441b4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:36.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.387+0000 7f381f400640 1 -- 172.21.15.31:0/3762637249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe441b7c00 msgr2=0x55fe441b4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:36.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.387+0000 7f381f400640 1 --2- 172.21.15.31:0/3762637249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe441b7c00 0x55fe441b4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.387+0000 7f381f400640 1 -- 172.21.15.31:0/3762637249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe441ae6d0 msgr2=0x55fe441b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:36.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.387+0000 7f381f400640 1 --2- 172.21.15.31:0/3762637249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe441ae6d0 0x55fe441b74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.387+0000 7f381f400640 1 -- 172.21.15.31:0/3762637249 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe43e263f0 con 0x55fe441af080 2024-03-21T10:40:36.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.388+0000 7f381f400640 1 --2- 172.21.15.31:0/3762637249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe441af080 0x55fe441b4880 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3810036040 tx=0x7f38100045f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38f688321ef5f3d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:36.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.388+0000 7f3808c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:40:36.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.388+0000 7f3808c00640 1 -- 172.21.15.31:0/3762637249 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3810044020 con 0x55fe441af080 2024-03-21T10:40:36.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.388+0000 7f3808c00640 1 -- 172.21.15.31:0/3762637249 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f381002cc80 con 0x55fe441af080 2024-03-21T10:40:36.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.388+0000 7f3828f6c740 1 -- 172.21.15.31:0/3762637249 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f380c005020 con 0x55fe441af080 2024-03-21T10:40:36.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.389+0000 7f3808c00640 1 -- 172.21.15.31:0/3762637249 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f381004f050 con 0x55fe441af080 2024-03-21T10:40:36.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.389+0000 7f3828f6c740 10 client.26207 fetch_fsmap learned FSMap version 25 2024-03-21T10:40:36.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.389+0000 7f3828f6c740 1 -- 172.21.15.31:0/3762637249 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55fe43ef07d0 con 0x55fe441af080 2024-03-21T10:40:36.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.389+0000 7f3808c00640 1 -- 172.21.15.31:0/3762637249 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f380c005020 con 0x55fe441af080 2024-03-21T10:40:36.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.390+0000 7f3808c00640 1 -- 172.21.15.31:0/3762637249 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f3810004c30 con 0x55fe441af080 2024-03-21T10:40:36.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.390+0000 7f3828f6c740 10 client.26207 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:40:36.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.390+0000 7f3828f6c740 10 client.26207 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:40:36.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.390+0000 7f3828f6c740 20 client.26207 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:36.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.390+0000 7f3828f6c740 1 --2- 172.21.15.31:0/3762637249 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55fe441c7ab0 0x55fe441c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:36.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.390+0000 7f3828f6c740 4 client.26207 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:40:36.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.390+0000 7f3828f6c740 1 -- 172.21.15.31:0/3762637249 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fe4411f800 con 0x55fe441c7ab0 2024-03-21T10:40:36.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.390+0000 7f381ea00640 1 --2- 172.21.15.31:0/3762637249 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55fe441c7ab0 0x55fe441c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.391+0000 7f381ea00640 1 --2- 172.21.15.31:0/3762637249 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55fe441c7ab0 0x55fe441c9ed0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2d40c57139fd0f4 server_cookie=9072cd72005318aa in_seq=0 out_seq=0 2024-03-21T10:40:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.391+0000 7f3808c00640 10 client.26207 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.391+0000 7f3808c00640 1 -- 172.21.15.31:0/3762637249 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55fe4411f800 con 0x55fe441c7ab0 2024-03-21T10:40:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.391+0000 7f3808c00640 10 client.26207 handle_command_reply: tid=0 2024-03-21T10:40:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.392+0000 7f3828f6c740 1 client.26207 shutdown 2024-03-21T10:40:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.392+0000 7f3828f6c740 1 -- 172.21.15.31:0/3762637249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe441af080 msgr2=0x55fe441b4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.392+0000 7f3828f6c740 1 --2- 172.21.15.31:0/3762637249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe441af080 0x55fe441b4880 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3810036040 tx=0x7f38100045f0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.392+0000 7f3828f6c740 1 -- 172.21.15.31:0/3762637249 shutdown_connections 2024-03-21T10:40:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.392+0000 7f3828f6c740 1 --2- 172.21.15.31:0/3762637249 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55fe441c7ab0 0x55fe441c9ed0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.392+0000 7f3828f6c740 1 --2- 172.21.15.31:0/3762637249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe441b7c00 0x55fe441b4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.392+0000 7f3828f6c740 1 --2- 172.21.15.31:0/3762637249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe441af080 0x55fe441b4880 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.392+0000 7f3828f6c740 1 --2- 172.21.15.31:0/3762637249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe441ae6d0 0x55fe441b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.392+0000 7f3828f6c740 1 -- 172.21.15.31:0/3762637249 >> 172.21.15.31:0/3762637249 conn(0x55fe441a8da0 msgr2=0x55fe441aaaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:36.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.392+0000 7f3808c00640 0 client.26207 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:36.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.392+0000 7f3828f6c740 1 -- 172.21.15.31:0/3762637249 shutdown_connections 2024-03-21T10:40:36.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.392+0000 7f3828f6c740 1 -- 172.21.15.31:0/3762637249 wait complete. 2024-03-21T10:40:36.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.392+0000 7f3828f6c740 20 client.26207 trim_cache size 0 max 16384 2024-03-21T10:40:36.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.410+0000 7f3828f6c740 1 Processor -- start 2024-03-21T10:40:36.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.411+0000 7f3828f6c740 1 -- start start 2024-03-21T10:40:36.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.411+0000 7f3828f6c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe441ae6d0 0x55fe441b6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:36.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.411+0000 7f3828f6c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe441af080 0x55fe441b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:36.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.411+0000 7f3828f6c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe441b7c00 0x55fe441b1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:36.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.411+0000 7f3828f6c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38201b3050 con 0x55fe441b7c00 2024-03-21T10:40:36.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.411+0000 7f3828f6c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38201b0b50 con 0x55fe441af080 2024-03-21T10:40:36.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.411+0000 7f3828f6c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38201aebf0 con 0x55fe441ae6d0 2024-03-21T10:40:36.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.411+0000 7f381f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe441af080 0x55fe441b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:36.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.411+0000 7f381ea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe441ae6d0 0x55fe441b6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:36.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.412+0000 7f381f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe441af080 0x55fe441b74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49086/0 (socket says 172.21.15.31:49086) 2024-03-21T10:40:36.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.412+0000 7f381f400640 1 -- 172.21.15.31:0/1694012175 learned_addr learned my addr 172.21.15.31:0/1694012175 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:36.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.412+0000 7f381ea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe441ae6d0 0x55fe441b6fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34644/0 (socket says 172.21.15.31:34644) 2024-03-21T10:40:36.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.412+0000 7f381ea00640 1 --2- 172.21.15.31:0/1694012175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe441b7c00 0x55fe441b1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:36.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.412+0000 7f381f400640 1 -- 172.21.15.31:0/1694012175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe441ae6d0 msgr2=0x55fe441b6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:36.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.412+0000 7f381f400640 1 --2- 172.21.15.31:0/1694012175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe441ae6d0 0x55fe441b6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.412+0000 7f381f400640 1 -- 172.21.15.31:0/1694012175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe441b7c00 msgr2=0x55fe441b1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:36.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.412+0000 7f381f400640 1 --2- 172.21.15.31:0/1694012175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe441b7c00 0x55fe441b1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.412+0000 7f381f400640 1 -- 172.21.15.31:0/1694012175 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe441ae180 con 0x55fe441af080 2024-03-21T10:40:36.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.412+0000 7f381ea00640 1 --2- 172.21.15.31:0/1694012175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe441b7c00 0x55fe441b1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:36.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.412+0000 7f381f400640 1 --2- 172.21.15.31:0/1694012175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe441af080 0x55fe441b74f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3810004490 tx=0x7f3810000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10443df21f6d104a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:36.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.412+0000 7f380b400640 1 -- 172.21.15.31:0/1694012175 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3810000e60 con 0x55fe441af080 2024-03-21T10:40:36.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.412+0000 7f380b400640 1 -- 172.21.15.31:0/1694012175 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3810009520 con 0x55fe441af080 2024-03-21T10:40:36.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.412+0000 7f380b400640 1 -- 172.21.15.31:0/1694012175 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3810044040 con 0x55fe441af080 2024-03-21T10:40:36.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.413+0000 7f3828f6c740 1 -- 172.21.15.31:0/1694012175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe441af080 msgr2=0x55fe441b74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:36.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.413+0000 7f3828f6c740 1 --2- 172.21.15.31:0/1694012175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe441af080 0x55fe441b74f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3810004490 tx=0x7f3810000bc0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.413+0000 7f3828f6c740 1 -- 172.21.15.31:0/1694012175 shutdown_connections 2024-03-21T10:40:36.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.413+0000 7f3828f6c740 1 --2- 172.21.15.31:0/1694012175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe441b7c00 0x55fe441b1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.413+0000 7f3828f6c740 1 --2- 172.21.15.31:0/1694012175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe441af080 0x55fe441b74f0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.413+0000 7f3828f6c740 1 --2- 172.21.15.31:0/1694012175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe441ae6d0 0x55fe441b6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.413+0000 7f3828f6c740 1 -- 172.21.15.31:0/1694012175 >> 172.21.15.31:0/1694012175 conn(0x55fe441a8da0 msgr2=0x55fe441a7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:36.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.413+0000 7f3828f6c740 1 -- 172.21.15.31:0/1694012175 shutdown_connections 2024-03-21T10:40:36.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.413+0000 7f3828f6c740 1 -- 172.21.15.31:0/1694012175 wait complete. 2024-03-21T10:40:36.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.414+0000 7f3828f6c740 1 -- start start 2024-03-21T10:40:36.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.414+0000 7f3828f6c740 1 Processor -- start 2024-03-21T10:40:36.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.414+0000 7f3828f6c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe441ae6d0 0x55fe441bd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.414+0000 7f3828f6c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe441af080 0x55fe441bdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.414+0000 7f3828f6c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe441b7c00 0x55fe441be030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.414+0000 7f3828f6c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38201af1f0 con 0x55fe441ae6d0 2024-03-21T10:40:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.414+0000 7f3828f6c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38201b04f0 con 0x55fe441af080 2024-03-21T10:40:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.414+0000 7f3828f6c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38201af830 con 0x55fe441b7c00 2024-03-21T10:40:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.415+0000 7f3828f6c740 10 client.0 _ll_register_callbacks cb 0x7f38180093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.415+0000 7f381ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe441ae6d0 0x55fe441bd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.415+0000 7f381ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe441ae6d0 0x55fe441bd5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39816/0 (socket says 172.21.15.31:39816) 2024-03-21T10:40:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.415+0000 7f381ea00640 1 -- 172.21.15.31:0/2769518997 learned_addr learned my addr 172.21.15.31:0/2769518997 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.415+0000 7f381f400640 1 --2- 172.21.15.31:0/2769518997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe441af080 0x55fe441bdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.415+0000 7f381ea00640 1 --2- 172.21.15.31:0/2769518997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe441b7c00 0x55fe441be030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.415+0000 7f381f400640 1 -- 172.21.15.31:0/2769518997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe441b7c00 msgr2=0x55fe441be030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.415+0000 7f381f400640 1 --2- 172.21.15.31:0/2769518997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe441b7c00 0x55fe441be030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.415+0000 7f381f400640 1 -- 172.21.15.31:0/2769518997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe441ae6d0 msgr2=0x55fe441bd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.415+0000 7f381f400640 1 --2- 172.21.15.31:0/2769518997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe441ae6d0 0x55fe441bd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.415+0000 7f381f400640 1 -- 172.21.15.31:0/2769518997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe442225c0 con 0x55fe441af080 2024-03-21T10:40:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.415+0000 7f381ea00640 1 --2- 172.21.15.31:0/2769518997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe441ae6d0 0x55fe441bd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.416+0000 7f381f400640 1 --2- 172.21.15.31:0/2769518997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe441af080 0x55fe441bdaf0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x55fe441bae40 tx=0x7f381002c390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d8cb5802b8eaf9d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.416+0000 7f3808c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:40:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.416+0000 7f3808c00640 1 -- 172.21.15.31:0/2769518997 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3810044040 con 0x55fe441af080 2024-03-21T10:40:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.416+0000 7f3828f6c740 1 -- 172.21.15.31:0/2769518997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3810009520 con 0x55fe441af080 2024-03-21T10:40:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.416+0000 7f3808c00640 1 -- 172.21.15.31:0/2769518997 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f3810005240 con 0x55fe441af080 2024-03-21T10:40:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.416+0000 7f3808c00640 1 -- 172.21.15.31:0/2769518997 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3810000e40 con 0x55fe441af080 2024-03-21T10:40:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.417+0000 7f3828f6c740 10 client.26213 fetch_fsmap learned FSMap version 25 2024-03-21T10:40:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.417+0000 7f3828f6c740 1 -- 172.21.15.31:0/2769518997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55fe442238c0 con 0x55fe441af080 2024-03-21T10:40:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.417+0000 7f3808c00640 1 -- 172.21.15.31:0/2769518997 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3810009520 con 0x55fe441af080 2024-03-21T10:40:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.417+0000 7f3808c00640 1 -- 172.21.15.31:0/2769518997 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f38100035f0 con 0x55fe441af080 2024-03-21T10:40:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.417+0000 7f3828f6c740 10 client.26213 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:40:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.417+0000 7f3828f6c740 10 client.26213 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:40:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.417+0000 7f3828f6c740 20 client.26213 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.417+0000 7f3828f6c740 1 --2- 172.21.15.31:0/2769518997 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55fe44223ee0 0x55fe44226300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.417+0000 7f3828f6c740 4 client.26213 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:40:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.417+0000 7f3828f6c740 1 -- 172.21.15.31:0/2769518997 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55fe44119b50 con 0x55fe44223ee0 2024-03-21T10:40:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.418+0000 7f381ea00640 1 --2- 172.21.15.31:0/2769518997 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55fe44223ee0 0x55fe44226300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:36.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.421+0000 7f3808c00640 10 client.26213 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:36.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.421+0000 7f381ea00640 1 --2- 172.21.15.31:0/2769518997 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55fe44223ee0 0x55fe44226300 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e205f595e26837b1 server_cookie=ef7efbdc359a2964 in_seq=0 out_seq=0 2024-03-21T10:40:36.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.422+0000 7f3808c00640 1 -- 172.21.15.31:0/2769518997 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55fe44119b50 con 0x55fe44223ee0 2024-03-21T10:40:36.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.422+0000 7f3808c00640 10 client.26213 handle_command_reply: tid=0 2024-03-21T10:40:36.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.422+0000 7f3828f6c740 1 client.26213 shutdown 2024-03-21T10:40:36.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.422+0000 7f3828f6c740 1 -- 172.21.15.31:0/2769518997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe441af080 msgr2=0x55fe441bdaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:36.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.422+0000 7f3828f6c740 1 --2- 172.21.15.31:0/2769518997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe441af080 0x55fe441bdaf0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x55fe441bae40 tx=0x7f381002c390 comp rx=0 tx=0).stop 2024-03-21T10:40:36.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.422+0000 7f3828f6c740 1 -- 172.21.15.31:0/2769518997 shutdown_connections 2024-03-21T10:40:36.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.422+0000 7f3828f6c740 1 --2- 172.21.15.31:0/2769518997 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55fe44223ee0 0x55fe44226300 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.422+0000 7f3828f6c740 1 --2- 172.21.15.31:0/2769518997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe441b7c00 0x55fe441be030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.422+0000 7f3828f6c740 1 --2- 172.21.15.31:0/2769518997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe441af080 0x55fe441bdaf0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.422+0000 7f3828f6c740 1 --2- 172.21.15.31:0/2769518997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe441ae6d0 0x55fe441bd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.422+0000 7f3828f6c740 1 -- 172.21.15.31:0/2769518997 >> 172.21.15.31:0/2769518997 conn(0x55fe441a8da0 msgr2=0x55fe441aa0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:36.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.422+0000 7f3808c00640 0 client.26213 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:36.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.423+0000 7f3828f6c740 1 -- 172.21.15.31:0/2769518997 shutdown_connections 2024-03-21T10:40:36.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.423+0000 7f3828f6c740 1 -- 172.21.15.31:0/2769518997 wait complete. 2024-03-21T10:40:36.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.423+0000 7f3828f6c740 20 client.26213 trim_cache size 0 max 16384 2024-03-21T10:40:36.422 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:40:36.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.424+0000 7f3826200640 1 -- 172.21.15.31:0/1961338429 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f38040a3190 msgr2=0x7f38040a5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:36.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.424+0000 7f3826200640 1 --2- 172.21.15.31:0/1961338429 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f38040a3190 0x7f38040a5650 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f38180049d0 tx=0x7f381800c950 comp rx=0 tx=0).stop 2024-03-21T10:40:36.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.424+0000 7f3826200640 1 -- 172.21.15.31:0/1961338429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3820107640 msgr2=0x7f382019ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:36.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.424+0000 7f3826200640 1 --2- 172.21.15.31:0/1961338429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3820107640 0x7f382019ea40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f381800eb10 tx=0x7f381800c2d0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.424+0000 7f3826200640 1 -- 172.21.15.31:0/1961338429 shutdown_connections 2024-03-21T10:40:36.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.424+0000 7f3826200640 1 --2- 172.21.15.31:0/1961338429 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f38040a3190 0x7f38040a5650 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.424+0000 7f3826200640 1 --2- 172.21.15.31:0/1961338429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3820107640 0x7f382019ea40 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.424+0000 7f3826200640 1 --2- 172.21.15.31:0/1961338429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3820101ff0 0x7f382019a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.424+0000 7f3826200640 1 --2- 172.21.15.31:0/1961338429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3820101630 0x7f382019a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.424+0000 7f3826200640 1 -- 172.21.15.31:0/1961338429 >> 172.21.15.31:0/1961338429 conn(0x7f38200fd140 msgr2=0x7f38200fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.425+0000 7f3826200640 1 -- 172.21.15.31:0/1961338429 shutdown_connections 2024-03-21T10:40:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:36.425+0000 7f3826200640 1 -- 172.21.15.31:0/1961338429 wait complete. 2024-03-21T10:40:37.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:36 smithi040 ceph-mon[36154]: pgmap v64: 129 pgs: 129 active+clean; 5.2 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 328 MiB/s wr, 102 op/s 2024-03-21T10:40:37.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:36 smithi031 ceph-mon[31673]: pgmap v64: 129 pgs: 129 active+clean; 5.2 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 328 MiB/s wr, 102 op/s 2024-03-21T10:40:37.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:36 smithi070 ceph-mon[36330]: pgmap v64: 129 pgs: 129 active+clean; 5.2 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 328 MiB/s wr, 102 op/s 2024-03-21T10:40:37.450 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-21T10:40:37.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:40:37.450 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-21T10:40:37.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.797+0000 7f7ae3e00640 1 -- 172.21.15.31:0/1471387700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7adc105380 msgr2=0x7f7adc1057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:37.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.797+0000 7f7ae3e00640 1 --2- 172.21.15.31:0/1471387700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7adc105380 0x7f7adc1057e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f7ac8009890 tx=0x7f7ac802bc10 comp rx=0 tx=0).stop 2024-03-21T10:40:37.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.798+0000 7f7ae3e00640 1 -- 172.21.15.31:0/1471387700 shutdown_connections 2024-03-21T10:40:37.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.798+0000 7f7ae3e00640 1 --2- 172.21.15.31:0/1471387700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7adc105d20 0x7f7adc110850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.798+0000 7f7ae3e00640 1 --2- 172.21.15.31:0/1471387700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7adc105380 0x7f7adc1057e0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.798+0000 7f7ae3e00640 1 --2- 172.21.15.31:0/1471387700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7adc103c40 0x7f7adc104020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.798+0000 7f7ae3e00640 1 -- 172.21.15.31:0/1471387700 >> 172.21.15.31:0/1471387700 conn(0x7f7adc0ff350 msgr2=0x7f7adc101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:37.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.799+0000 7f7ae3e00640 1 -- 172.21.15.31:0/1471387700 shutdown_connections 2024-03-21T10:40:37.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.799+0000 7f7ae3e00640 1 -- 172.21.15.31:0/1471387700 wait complete. 2024-03-21T10:40:37.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.800+0000 7f7ae3e00640 1 Processor -- start 2024-03-21T10:40:37.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.800+0000 7f7ae3e00640 1 -- start start 2024-03-21T10:40:37.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.801+0000 7f7ae3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7adc103c40 0x7f7adc19a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:37.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.801+0000 7f7ae3e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7adc105380 0x7f7adc19a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:37.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.801+0000 7f7ae3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7adc105d20 0x7f7adc19ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:37.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.801+0000 7f7ae3e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc113da0 con 0x7f7adc105d20 2024-03-21T10:40:37.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.801+0000 7f7ae3e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc113c20 con 0x7f7adc103c40 2024-03-21T10:40:37.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.801+0000 7f7ae3e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc113f20 con 0x7f7adc105380 2024-03-21T10:40:37.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.801+0000 7f7ae1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7adc103c40 0x7f7adc19a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:37.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.801+0000 7f7ae1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7adc103c40 0x7f7adc19a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:59182/0 (socket says 172.21.15.31:59182) 2024-03-21T10:40:37.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.801+0000 7f7ae1400640 1 -- 172.21.15.31:0/1285648989 learned_addr learned my addr 172.21.15.31:0/1285648989 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:37.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.802+0000 7f7ae1400640 1 -- 172.21.15.31:0/1285648989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7adc105380 msgr2=0x7f7adc19a880 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-21T10:40:37.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.802+0000 7f7ae1400640 1 --2- 172.21.15.31:0/1285648989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7adc105380 0x7f7adc19a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.802+0000 7f7ae1400640 1 -- 172.21.15.31:0/1285648989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7adc105d20 msgr2=0x7f7adc19ec00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T10:40:37.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.802+0000 7f7ae1e00640 1 --2- 172.21.15.31:0/1285648989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7adc105d20 0x7f7adc19ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:37.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.802+0000 7f7ae1400640 1 --2- 172.21.15.31:0/1285648989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7adc105d20 0x7f7adc19ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.802+0000 7f7ae1400640 1 -- 172.21.15.31:0/1285648989 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7adc19f300 con 0x7f7adc103c40 2024-03-21T10:40:37.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.802+0000 7f7ae1e00640 1 --2- 172.21.15.31:0/1285648989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7adc105d20 0x7f7adc19ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:40:37.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.802+0000 7f7ae1400640 1 --2- 172.21.15.31:0/1285648989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7adc103c40 0x7f7adc19a340 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7ad000d980 tx=0x7f7ad000de40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:37.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.803+0000 7f7ada000640 1 -- 172.21.15.31:0/1285648989 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad0017070 con 0x7f7adc103c40 2024-03-21T10:40:37.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.803+0000 7f7ae3e00640 1 -- 172.21.15.31:0/1285648989 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7adc19f5f0 con 0x7f7adc103c40 2024-03-21T10:40:37.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.803+0000 7f7ae3e00640 1 -- 172.21.15.31:0/1285648989 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7adc06b0c0 con 0x7f7adc103c40 2024-03-21T10:40:37.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.804+0000 7f7ada000640 1 -- 172.21.15.31:0/1285648989 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ad0009c30 con 0x7f7adc103c40 2024-03-21T10:40:37.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.804+0000 7f7ada000640 1 -- 172.21.15.31:0/1285648989 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad000f620 con 0x7f7adc103c40 2024-03-21T10:40:37.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.810+0000 7f7ada000640 1 -- 172.21.15.31:0/1285648989 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f7ad000f7c0 con 0x7f7adc103c40 2024-03-21T10:40:37.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.810+0000 7f7ae6c1d740 1 Processor -- start 2024-03-21T10:40:37.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.811+0000 7f7ae6c1d740 1 -- start start 2024-03-21T10:40:37.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.811+0000 7f7ae6c1d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a35a330080 0x55a35a330460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:37.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.811+0000 7f7ae6c1d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35a3309a0 0x55a35a339040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:37.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.811+0000 7f7ae6c1d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35a339580 0x55a35a33b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:37.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.811+0000 7f7ae6c1d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a35a294eb0 con 0x55a35a3309a0 2024-03-21T10:40:37.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.811+0000 7f7ae6c1d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a35a23da60 con 0x55a35a339580 2024-03-21T10:40:37.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.811+0000 7f7ae6c1d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a35a290f50 con 0x55a35a330080 2024-03-21T10:40:37.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.812+0000 7f7ae1e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a35a330080 0x55a35a330460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:37.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.812+0000 7f7ae1e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a35a330080 0x55a35a330460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:51620/0 (socket says 172.21.15.31:51620) 2024-03-21T10:40:37.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.812+0000 7f7ae1e00640 1 -- 172.21.15.31:0/2522428272 learned_addr learned my addr 172.21.15.31:0/2522428272 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:37.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.812+0000 7f7ae1400640 1 --2- 172.21.15.31:0/2522428272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35a3309a0 0x55a35a339040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:37.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.812+0000 7f7ae1e00640 1 -- 172.21.15.31:0/2522428272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35a339580 msgr2=0x55a35a33b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:37.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.812+0000 7f7ae1e00640 1 --2- 172.21.15.31:0/2522428272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35a339580 0x55a35a33b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.812+0000 7f7ae1e00640 1 -- 172.21.15.31:0/2522428272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35a3309a0 msgr2=0x55a35a339040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:37.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.813+0000 7f7ae1e00640 1 --2- 172.21.15.31:0/2522428272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35a3309a0 0x55a35a339040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.813+0000 7f7ae1e00640 1 -- 172.21.15.31:0/2522428272 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a35a0717e0 con 0x55a35a330080 2024-03-21T10:40:37.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.813+0000 7f7ae1400640 1 --2- 172.21.15.31:0/2522428272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35a3309a0 0x55a35a339040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:37.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.813+0000 7f7ae1e00640 1 --2- 172.21.15.31:0/2522428272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a35a330080 0x55a35a330460 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4007f30 tx=0x7f7ad400c750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4a951217d78f266 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:37.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.814+0000 7f7ace600640 1 -- 172.21.15.31:0/2522428272 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad4014070 con 0x55a35a330080 2024-03-21T10:40:37.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.814+0000 7f7ace600640 1 -- 172.21.15.31:0/2522428272 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ad400fd30 con 0x55a35a330080 2024-03-21T10:40:37.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.814+0000 7f7ace600640 1 -- 172.21.15.31:0/2522428272 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad400ed60 con 0x55a35a330080 2024-03-21T10:40:37.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.814+0000 7f7ada000640 1 --2- 172.21.15.31:0/1285648989 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7ac40a3110 0x7f7ac40a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:37.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.815+0000 7f7ada000640 1 -- 172.21.15.31:0/1285648989 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f7ad00cf7e0 con 0x7f7adc103c40 2024-03-21T10:40:37.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.815+0000 7f7ae6c1d740 1 -- 172.21.15.31:0/2522428272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a35a330080 msgr2=0x55a35a330460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:37.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.815+0000 7f7ae6c1d740 1 --2- 172.21.15.31:0/2522428272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a35a330080 0x55a35a330460 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4007f30 tx=0x7f7ad400c750 comp rx=0 tx=0).stop 2024-03-21T10:40:37.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.815+0000 7f7ae6c1d740 1 -- 172.21.15.31:0/2522428272 shutdown_connections 2024-03-21T10:40:37.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.815+0000 7f7ae6c1d740 1 --2- 172.21.15.31:0/2522428272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35a339580 0x55a35a33b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.815+0000 7f7ae6c1d740 1 --2- 172.21.15.31:0/2522428272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35a3309a0 0x55a35a339040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.815+0000 7f7ae6c1d740 1 --2- 172.21.15.31:0/2522428272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a35a330080 0x55a35a330460 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.815+0000 7f7ae6c1d740 1 -- 172.21.15.31:0/2522428272 >> 172.21.15.31:0/2522428272 conn(0x55a35a32a7b0 msgr2=0x55a35a32ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:37.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.815+0000 7f7ae6c1d740 1 -- 172.21.15.31:0/2522428272 shutdown_connections 2024-03-21T10:40:37.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.815+0000 7f7ae1400640 1 --2- 172.21.15.31:0/1285648989 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7ac40a3110 0x7f7ac40a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:37.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.817+0000 7f7ae1400640 1 --2- 172.21.15.31:0/1285648989 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7ac40a3110 0x7f7ac40a55d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x55a35a32fb50 tx=0x7f7ad0010db0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:37.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.817+0000 7f7ae6c1d740 1 -- 172.21.15.31:0/2522428272 wait complete. 2024-03-21T10:40:37.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.817+0000 7f7ae6c1d740 1 -- start start 2024-03-21T10:40:37.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.817+0000 7f7ae6c1d740 1 Processor -- start 2024-03-21T10:40:37.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.818+0000 7f7ae6c1d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35a330080 0x55a35a338ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:37.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.818+0000 7f7ae6c1d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35a3309a0 0x55a35a336230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:37.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.818+0000 7f7ae6c1d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a35a339580 0x55a35a336770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:37.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.818+0000 7f7ae6c1d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a35a33c370 con 0x55a35a3309a0 2024-03-21T10:40:37.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.818+0000 7f7ae6c1d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a35a250eb0 con 0x55a35a330080 2024-03-21T10:40:37.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.818+0000 7f7ae6c1d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a35a33c4f0 con 0x55a35a339580 2024-03-21T10:40:37.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.818+0000 7f7ae6c1d740 10 client.0 _ll_register_callbacks cb 0x55a35a245150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:37.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.818+0000 7f7ae1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35a330080 0x55a35a338ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:37.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.818+0000 7f7ae1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35a330080 0x55a35a338ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:59198/0 (socket says 172.21.15.31:59198) 2024-03-21T10:40:37.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.818+0000 7f7ae1e00640 1 -- 172.21.15.31:0/2259760970 learned_addr learned my addr 172.21.15.31:0/2259760970 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:37.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.818+0000 7f7ae0a00640 1 --2- 172.21.15.31:0/2259760970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35a3309a0 0x55a35a336230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:37.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.818+0000 7f7ae1e00640 1 --2- 172.21.15.31:0/2259760970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a35a339580 0x55a35a336770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:37.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.819+0000 7f7ae1e00640 1 -- 172.21.15.31:0/2259760970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a35a339580 msgr2=0x55a35a336770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:37.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.819+0000 7f7ae1e00640 1 --2- 172.21.15.31:0/2259760970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a35a339580 0x55a35a336770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.819+0000 7f7ae1e00640 1 -- 172.21.15.31:0/2259760970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35a3309a0 msgr2=0x55a35a336230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:37.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.819+0000 7f7ae1e00640 1 --2- 172.21.15.31:0/2259760970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35a3309a0 0x55a35a336230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.819+0000 7f7ae1e00640 1 -- 172.21.15.31:0/2259760970 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a35a0d1f70 con 0x55a35a330080 2024-03-21T10:40:37.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.819+0000 7f7ae0a00640 1 --2- 172.21.15.31:0/2259760970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35a3309a0 0x55a35a336230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:37.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.819+0000 7f7ae1e00640 1 --2- 172.21.15.31:0/2259760970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35a330080 0x55a35a338ea0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7ad400ea40 tx=0x7f7ad400e940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a59332322f60a92 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:37.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.819+0000 7f7ac2a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:40:37.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.820+0000 7f7ac2a00640 1 -- 172.21.15.31:0/2259760970 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad4014050 con 0x55a35a330080 2024-03-21T10:40:37.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.820+0000 7f7ac2a00640 1 -- 172.21.15.31:0/2259760970 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f7ad400f650 con 0x55a35a330080 2024-03-21T10:40:37.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.820+0000 7f7ae6c1d740 1 -- 172.21.15.31:0/2259760970 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7ad400fd30 con 0x55a35a330080 2024-03-21T10:40:37.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.820+0000 7f7ac2a00640 1 -- 172.21.15.31:0/2259760970 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ad402e050 con 0x55a35a330080 2024-03-21T10:40:37.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.820+0000 7f7ae6c1d740 10 client.26222 fetch_fsmap learned FSMap version 25 2024-03-21T10:40:37.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.820+0000 7f7ae6c1d740 1 -- 172.21.15.31:0/2259760970 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a359f69170 con 0x55a35a330080 2024-03-21T10:40:37.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.821+0000 7f7ac2a00640 1 -- 172.21.15.31:0/2259760970 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ad400fd30 con 0x55a35a330080 2024-03-21T10:40:37.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.822+0000 7f7ac2a00640 1 -- 172.21.15.31:0/2259760970 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f7ad4019b50 con 0x55a35a330080 2024-03-21T10:40:37.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.822+0000 7f7ae6c1d740 10 client.26222 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:40:37.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.822+0000 7f7ae6c1d740 10 client.26222 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:40:37.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.822+0000 7f7ae6c1d740 20 client.26222 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.822+0000 7f7ae6c1d740 1 --2- 172.21.15.31:0/2259760970 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a35a349500 0x55a35a34b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.822+0000 7f7ae6c1d740 4 client.26222 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:40:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.822+0000 7f7ae6c1d740 1 -- 172.21.15.31:0/2259760970 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a35a2a23f0 con 0x55a35a349500 2024-03-21T10:40:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.822+0000 7f7ae0a00640 1 --2- 172.21.15.31:0/2259760970 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a35a349500 0x55a35a34b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.823+0000 7f7ae0a00640 1 --2- 172.21.15.31:0/2259760970 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a35a349500 0x55a35a34b8f0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d59f6e6bd59a9ad5 server_cookie=27086c0397b5aed9 in_seq=0 out_seq=0 2024-03-21T10:40:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.823+0000 7f7ac2a00640 10 client.26222 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:37.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.823+0000 7f7ac2a00640 1 -- 172.21.15.31:0/2259760970 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a35a2a23f0 con 0x55a35a349500 2024-03-21T10:40:37.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.823+0000 7f7ac2a00640 10 client.26222 handle_command_reply: tid=0 2024-03-21T10:40:37.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.824+0000 7f7ae6c1d740 1 client.26222 shutdown 2024-03-21T10:40:37.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.824+0000 7f7ae6c1d740 1 -- 172.21.15.31:0/2259760970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35a330080 msgr2=0x55a35a338ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:37.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.824+0000 7f7ae6c1d740 1 --2- 172.21.15.31:0/2259760970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35a330080 0x55a35a338ea0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7ad400ea40 tx=0x7f7ad400e940 comp rx=0 tx=0).stop 2024-03-21T10:40:37.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.824+0000 7f7ae6c1d740 1 -- 172.21.15.31:0/2259760970 shutdown_connections 2024-03-21T10:40:37.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.824+0000 7f7ae6c1d740 1 --2- 172.21.15.31:0/2259760970 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a35a349500 0x55a35a34b8f0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.824+0000 7f7ae6c1d740 1 --2- 172.21.15.31:0/2259760970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a35a339580 0x55a35a336770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.824+0000 7f7ae6c1d740 1 --2- 172.21.15.31:0/2259760970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35a3309a0 0x55a35a336230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.824+0000 7f7ae6c1d740 1 --2- 172.21.15.31:0/2259760970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35a330080 0x55a35a338ea0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.824+0000 7f7ae6c1d740 1 -- 172.21.15.31:0/2259760970 >> 172.21.15.31:0/2259760970 conn(0x55a35a32a7b0 msgr2=0x55a35a32c500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:37.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.824+0000 7f7ac2a00640 0 client.26222 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:37.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.825+0000 7f7ae6c1d740 1 -- 172.21.15.31:0/2259760970 shutdown_connections 2024-03-21T10:40:37.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.825+0000 7f7ae6c1d740 1 -- 172.21.15.31:0/2259760970 wait complete. 2024-03-21T10:40:37.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.825+0000 7f7ae6c1d740 20 client.26222 trim_cache size 0 max 16384 2024-03-21T10:40:37.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.857+0000 7f7ae6c1d740 1 Processor -- start 2024-03-21T10:40:37.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.857+0000 7f7ae6c1d740 1 -- start start 2024-03-21T10:40:37.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.858+0000 7f7ae6c1d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35a330080 0x55a35a338960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:37.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.858+0000 7f7ae6c1d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a35a3309a0 0x55a35a338ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:37.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.858+0000 7f7ae6c1d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35a339580 0x55a35a344c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:37.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.858+0000 7f7ae6c1d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc012160 con 0x55a35a339580 2024-03-21T10:40:37.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.858+0000 7f7ae6c1d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc00fc60 con 0x55a35a330080 2024-03-21T10:40:37.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.858+0000 7f7ae6c1d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc00dd00 con 0x55a35a3309a0 2024-03-21T10:40:37.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.858+0000 7f7ae0a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a35a3309a0 0x55a35a338ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:37.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.858+0000 7f7ae0a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a35a3309a0 0x55a35a338ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:51648/0 (socket says 172.21.15.31:51648) 2024-03-21T10:40:37.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.858+0000 7f7ae0a00640 1 -- 172.21.15.31:0/1092966688 learned_addr learned my addr 172.21.15.31:0/1092966688 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:37.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.858+0000 7f7ae1e00640 1 --2- 172.21.15.31:0/1092966688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35a330080 0x55a35a338960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:37.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.858+0000 7f7ae1e00640 1 --2- 172.21.15.31:0/1092966688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35a339580 0x55a35a344c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:37.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.859+0000 7f7ae0a00640 1 -- 172.21.15.31:0/1092966688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35a330080 msgr2=0x55a35a338960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:37.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.859+0000 7f7ae0a00640 1 --2- 172.21.15.31:0/1092966688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35a330080 0x55a35a338960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.859+0000 7f7ae0a00640 1 -- 172.21.15.31:0/1092966688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35a339580 msgr2=0x55a35a344c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:37.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.859+0000 7f7ae0a00640 1 --2- 172.21.15.31:0/1092966688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35a339580 0x55a35a344c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.859+0000 7f7ae0a00640 1 -- 172.21.15.31:0/1092966688 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a35a325c20 con 0x55a35a3309a0 2024-03-21T10:40:37.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.859+0000 7f7ae1e00640 1 --2- 172.21.15.31:0/1092966688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35a339580 0x55a35a344c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:37.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.859+0000 7f7ae0a00640 1 --2- 172.21.15.31:0/1092966688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a35a3309a0 0x55a35a338ea0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f7ac802bc50 tx=0x7f7ac8005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=212beb3a901193bc server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:37.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.859+0000 7f7acdc00640 1 -- 172.21.15.31:0/1092966688 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac8039040 con 0x55a35a3309a0 2024-03-21T10:40:37.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.860+0000 7f7acdc00640 1 -- 172.21.15.31:0/1092966688 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ac8034440 con 0x55a35a3309a0 2024-03-21T10:40:37.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.860+0000 7f7acdc00640 1 -- 172.21.15.31:0/1092966688 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac80054f0 con 0x55a35a3309a0 2024-03-21T10:40:37.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.861+0000 7f7ae6c1d740 1 -- 172.21.15.31:0/1092966688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a35a3309a0 msgr2=0x55a35a338ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:37.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.861+0000 7f7ae6c1d740 1 --2- 172.21.15.31:0/1092966688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a35a3309a0 0x55a35a338ea0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f7ac802bc50 tx=0x7f7ac8005850 comp rx=0 tx=0).stop 2024-03-21T10:40:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.861+0000 7f7ae6c1d740 1 -- 172.21.15.31:0/1092966688 shutdown_connections 2024-03-21T10:40:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.861+0000 7f7ae6c1d740 1 --2- 172.21.15.31:0/1092966688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35a339580 0x55a35a344c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.861+0000 7f7ae6c1d740 1 --2- 172.21.15.31:0/1092966688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a35a3309a0 0x55a35a338ea0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.861+0000 7f7ae6c1d740 1 --2- 172.21.15.31:0/1092966688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35a330080 0x55a35a338960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.861+0000 7f7ae6c1d740 1 -- 172.21.15.31:0/1092966688 >> 172.21.15.31:0/1092966688 conn(0x55a35a32a7b0 msgr2=0x55a35a32c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.861+0000 7f7ae6c1d740 1 -- 172.21.15.31:0/1092966688 shutdown_connections 2024-03-21T10:40:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.861+0000 7f7ae6c1d740 1 -- 172.21.15.31:0/1092966688 wait complete. 2024-03-21T10:40:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.862+0000 7f7ae6c1d740 1 -- start start 2024-03-21T10:40:37.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.862+0000 7f7ae6c1d740 1 Processor -- start 2024-03-21T10:40:37.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.863+0000 7f7ae6c1d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35a330080 0x55a35a33e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:37.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.863+0000 7f7ae6c1d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35a3309a0 0x55a35a33eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:37.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.863+0000 7f7ae6c1d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a35a339580 0x55a35a3ad3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:37.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.863+0000 7f7ae6c1d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc00e300 con 0x55a35a330080 2024-03-21T10:40:37.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.863+0000 7f7ae6c1d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc00f600 con 0x55a35a3309a0 2024-03-21T10:40:37.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.863+0000 7f7ae6c1d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc00e940 con 0x55a35a339580 2024-03-21T10:40:37.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.863+0000 7f7ae6c1d740 10 client.0 _ll_register_callbacks cb 0x7f7ad4007d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:37.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.863+0000 7f7ae1e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35a330080 0x55a35a33e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:37.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.863+0000 7f7ae1e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35a330080 0x55a35a33e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57868/0 (socket says 172.21.15.31:57868) 2024-03-21T10:40:37.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.864+0000 7f7ae1e00640 1 -- 172.21.15.31:0/4034316758 learned_addr learned my addr 172.21.15.31:0/4034316758 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:37.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.864+0000 7f7ae1e00640 1 --2- 172.21.15.31:0/4034316758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a35a339580 0x55a35a3ad3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:37.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.864+0000 7f7ae0a00640 1 --2- 172.21.15.31:0/4034316758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35a3309a0 0x55a35a33eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:37.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.864+0000 7f7ae1e00640 1 -- 172.21.15.31:0/4034316758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a35a339580 msgr2=0x55a35a3ad3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:37.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.864+0000 7f7ae1e00640 1 --2- 172.21.15.31:0/4034316758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a35a339580 0x55a35a3ad3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.864+0000 7f7ae1e00640 1 -- 172.21.15.31:0/4034316758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35a3309a0 msgr2=0x55a35a33eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:37.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.864+0000 7f7ae1e00640 1 --2- 172.21.15.31:0/4034316758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35a3309a0 0x55a35a33eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.864+0000 7f7ae1e00640 1 -- 172.21.15.31:0/4034316758 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a35a3ad900 con 0x55a35a330080 2024-03-21T10:40:37.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.865+0000 7f7ae1e00640 1 --2- 172.21.15.31:0/4034316758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35a330080 0x55a35a33e990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4002230 tx=0x7f7ad400e8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d432ccad556f5cce server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:37.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.865+0000 7f7ac2a00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:40:37.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.865+0000 7f7ac2a00640 1 -- 172.21.15.31:0/4034316758 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad4014070 con 0x55a35a330080 2024-03-21T10:40:37.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.865+0000 7f7ae6c1d740 1 -- 172.21.15.31:0/4034316758 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7ac8034440 con 0x55a35a330080 2024-03-21T10:40:37.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.865+0000 7f7ac2a00640 1 -- 172.21.15.31:0/4034316758 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f7ad4024040 con 0x55a35a330080 2024-03-21T10:40:37.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.866+0000 7f7ac2a00640 1 -- 172.21.15.31:0/4034316758 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ad4045050 con 0x55a35a330080 2024-03-21T10:40:37.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.866+0000 7f7ae6c1d740 10 client.16017 fetch_fsmap learned FSMap version 25 2024-03-21T10:40:37.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.866+0000 7f7ae6c1d740 1 -- 172.21.15.31:0/4034316758 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a35a3aebb0 con 0x55a35a330080 2024-03-21T10:40:37.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.866+0000 7f7ac2a00640 1 -- 172.21.15.31:0/4034316758 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ac8034440 con 0x55a35a330080 2024-03-21T10:40:37.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.866+0000 7f7ac2a00640 1 -- 172.21.15.31:0/4034316758 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f7ad4017460 con 0x55a35a330080 2024-03-21T10:40:37.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.866+0000 7f7ae6c1d740 10 client.16017 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:40:37.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.866+0000 7f7ae6c1d740 10 client.16017 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:40:37.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.866+0000 7f7ae6c1d740 20 client.16017 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:37.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.866+0000 7f7ae6c1d740 1 --2- 172.21.15.31:0/4034316758 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a35a3af190 0x55a35a3b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:37.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.866+0000 7f7ae6c1d740 4 client.16017 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-21T10:40:37.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.866+0000 7f7ae6c1d740 1 -- 172.21.15.31:0/4034316758 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55a35a29c740 con 0x55a35a3af190 2024-03-21T10:40:37.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.867+0000 7f7ae0a00640 1 --2- 172.21.15.31:0/4034316758 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a35a3af190 0x55a35a3b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:37.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.867+0000 7f7ac2a00640 10 client.16017 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:37.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.868+0000 7f7ae0a00640 1 --2- 172.21.15.31:0/4034316758 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a35a3af190 0x55a35a3b15b0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70df696e7389c177 server_cookie=36c6993775d444d in_seq=0 out_seq=0 2024-03-21T10:40:37.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.869+0000 7f7ac2a00640 1 -- 172.21.15.31:0/4034316758 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55a35a29c740 con 0x55a35a3af190 2024-03-21T10:40:37.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.869+0000 7f7ac2a00640 10 client.16017 handle_command_reply: tid=0 2024-03-21T10:40:37.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.869+0000 7f7ae6c1d740 1 client.16017 shutdown 2024-03-21T10:40:37.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.869+0000 7f7ae6c1d740 1 -- 172.21.15.31:0/4034316758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35a330080 msgr2=0x55a35a33e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:37.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.869+0000 7f7ae6c1d740 1 --2- 172.21.15.31:0/4034316758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35a330080 0x55a35a33e990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4002230 tx=0x7f7ad400e8a0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.870+0000 7f7ae6c1d740 1 -- 172.21.15.31:0/4034316758 shutdown_connections 2024-03-21T10:40:37.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.870+0000 7f7ae6c1d740 1 --2- 172.21.15.31:0/4034316758 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a35a3af190 0x55a35a3b15b0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.870+0000 7f7ac2a00640 0 client.16017 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:37.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.870+0000 7f7ae6c1d740 1 --2- 172.21.15.31:0/4034316758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a35a339580 0x55a35a3ad3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.870+0000 7f7ae6c1d740 1 --2- 172.21.15.31:0/4034316758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35a3309a0 0x55a35a33eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.870+0000 7f7ae6c1d740 1 --2- 172.21.15.31:0/4034316758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35a330080 0x55a35a33e990 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.870+0000 7f7ae6c1d740 1 -- 172.21.15.31:0/4034316758 >> 172.21.15.31:0/4034316758 conn(0x55a35a32a7b0 msgr2=0x55a35a328fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:37.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.870+0000 7f7ae6c1d740 1 -- 172.21.15.31:0/4034316758 shutdown_connections 2024-03-21T10:40:37.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.870+0000 7f7ae6c1d740 1 -- 172.21.15.31:0/4034316758 wait complete. 2024-03-21T10:40:37.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.870+0000 7f7ae6c1d740 20 client.16017 trim_cache size 0 max 16384 2024-03-21T10:40:37.869 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:40:37.870 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-03-21T10:40:37.870 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "14cc37d1-4a0c-4727-a49e-74350e41cd3e", 2024-03-21T10:40:37.870 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-03-21T10:40:37.870 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:40:37.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.871+0000 7f7ae3e00640 1 -- 172.21.15.31:0/1285648989 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7ac40a3110 msgr2=0x7f7ac40a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:37.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.871+0000 7f7ae3e00640 1 --2- 172.21.15.31:0/1285648989 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7ac40a3110 0x7f7ac40a55d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x55a35a32fb50 tx=0x7f7ad0010db0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.871+0000 7f7ae3e00640 1 -- 172.21.15.31:0/1285648989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7adc103c40 msgr2=0x7f7adc19a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:37.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.871+0000 7f7ae3e00640 1 --2- 172.21.15.31:0/1285648989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7adc103c40 0x7f7adc19a340 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7ad000d980 tx=0x7f7ad000de40 comp rx=0 tx=0).stop 2024-03-21T10:40:37.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.871+0000 7f7ae3e00640 1 -- 172.21.15.31:0/1285648989 shutdown_connections 2024-03-21T10:40:37.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.871+0000 7f7ae3e00640 1 --2- 172.21.15.31:0/1285648989 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7ac40a3110 0x7f7ac40a55d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.871+0000 7f7ae3e00640 1 --2- 172.21.15.31:0/1285648989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7adc105d20 0x7f7adc19ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.871+0000 7f7ae3e00640 1 --2- 172.21.15.31:0/1285648989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7adc105380 0x7f7adc19a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.871+0000 7f7ae3e00640 1 --2- 172.21.15.31:0/1285648989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7adc103c40 0x7f7adc19a340 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:37.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.871+0000 7f7ae3e00640 1 -- 172.21.15.31:0/1285648989 >> 172.21.15.31:0/1285648989 conn(0x7f7adc0ff350 msgr2=0x7f7adc107760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:37.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.871+0000 7f7ae3e00640 1 -- 172.21.15.31:0/1285648989 shutdown_connections 2024-03-21T10:40:37.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:37.872+0000 7f7ae3e00640 1 -- 172.21.15.31:0/1285648989 wait complete. 2024-03-21T10:40:37.892 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:40:37.893 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:40:38.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.156+0000 7fcf6d800640 1 -- 172.21.15.31:0/2275496112 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf68101630 msgr2=0x7fcf68101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:38.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.156+0000 7fcf6d800640 1 --2- 172.21.15.31:0/2275496112 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf68101630 0x7fcf68101ab0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fcf58009930 tx=0x7fcf5802bc30 comp rx=0 tx=0).stop 2024-03-21T10:40:38.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.157+0000 7fcf6d800640 1 -- 172.21.15.31:0/2275496112 shutdown_connections 2024-03-21T10:40:38.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.157+0000 7fcf6d800640 1 --2- 172.21.15.31:0/2275496112 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf68101ff0 0x7fcf6810e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.157+0000 7fcf6d800640 1 --2- 172.21.15.31:0/2275496112 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf68101630 0x7fcf68101ab0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.157+0000 7fcf6d800640 1 --2- 172.21.15.31:0/2275496112 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf68107640 0x7fcf68107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.157+0000 7fcf6d800640 1 -- 172.21.15.31:0/2275496112 >> 172.21.15.31:0/2275496112 conn(0x7fcf680fd140 msgr2=0x7fcf680ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:38.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.160+0000 7fcf6d800640 1 -- 172.21.15.31:0/2275496112 shutdown_connections 2024-03-21T10:40:38.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.160+0000 7fcf6d800640 1 -- 172.21.15.31:0/2275496112 wait complete. 2024-03-21T10:40:38.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.162+0000 7fcf6d800640 1 Processor -- start 2024-03-21T10:40:38.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.162+0000 7fcf6d800640 1 -- start start 2024-03-21T10:40:38.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.163+0000 7fcf6d800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf68101630 0x7fcf6819a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:38.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.163+0000 7fcf6d800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf68101ff0 0x7fcf6819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:38.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.163+0000 7fcf6d800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf68107640 0x7fcf6819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:38.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.163+0000 7fcf6d800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf68111a60 con 0x7fcf68101630 2024-03-21T10:40:38.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.163+0000 7fcf6d800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf681118e0 con 0x7fcf68107640 2024-03-21T10:40:38.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.163+0000 7fcf6d800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf68111be0 con 0x7fcf68101ff0 2024-03-21T10:40:38.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.163+0000 7fcf66a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf68101630 0x7fcf6819a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:38.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.163+0000 7fcf66a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf68101630 0x7fcf6819a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57892/0 (socket says 172.21.15.31:57892) 2024-03-21T10:40:38.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.163+0000 7fcf66a00640 1 -- 172.21.15.31:0/3279691968 learned_addr learned my addr 172.21.15.31:0/3279691968 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:38.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.163+0000 7fcf5e000640 1 --2- 172.21.15.31:0/3279691968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf68101ff0 0x7fcf6819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:38.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.163+0000 7fcf67400640 1 --2- 172.21.15.31:0/3279691968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf68107640 0x7fcf6819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:38.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.164+0000 7fcf5e000640 1 -- 172.21.15.31:0/3279691968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf68107640 msgr2=0x7fcf6819eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:38.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.164+0000 7fcf5e000640 1 --2- 172.21.15.31:0/3279691968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf68107640 0x7fcf6819eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.164+0000 7fcf5e000640 1 -- 172.21.15.31:0/3279691968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf68101630 msgr2=0x7fcf6819a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:38.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.165+0000 7fcf5e000640 1 --2- 172.21.15.31:0/3279691968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf68101630 0x7fcf6819a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.165+0000 7fcf5e000640 1 -- 172.21.15.31:0/3279691968 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf6819f1f0 con 0x7fcf68101ff0 2024-03-21T10:40:38.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.165+0000 7fcf66a00640 1 --2- 172.21.15.31:0/3279691968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf68101630 0x7fcf6819a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:40:38.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.165+0000 7fcf67400640 1 --2- 172.21.15.31:0/3279691968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf68107640 0x7fcf6819eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:40:38.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.165+0000 7fcf5e000640 1 --2- 172.21.15.31:0/3279691968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf68101ff0 0x7fcf6819a740 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fcf58036040 tx=0x7fcf58004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:38.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.166+0000 7fcf5fe00640 1 -- 172.21.15.31:0/3279691968 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf58043020 con 0x7fcf68101ff0 2024-03-21T10:40:38.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.166+0000 7fcf5fe00640 1 -- 172.21.15.31:0/3279691968 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf580054e0 con 0x7fcf68101ff0 2024-03-21T10:40:38.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.166+0000 7fcf5fe00640 1 -- 172.21.15.31:0/3279691968 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf58003b80 con 0x7fcf68101ff0 2024-03-21T10:40:38.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.166+0000 7fcf6d800640 1 -- 172.21.15.31:0/3279691968 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf6819f480 con 0x7fcf68101ff0 2024-03-21T10:40:38.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.167+0000 7fcf6d800640 1 -- 172.21.15.31:0/3279691968 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf681a6d70 con 0x7fcf68101ff0 2024-03-21T10:40:38.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.172+0000 7fcf5fe00640 1 -- 172.21.15.31:0/3279691968 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fcf58046020 con 0x7fcf68101ff0 2024-03-21T10:40:38.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.173+0000 7fcf70637740 1 Processor -- start 2024-03-21T10:40:38.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.175+0000 7fcf5fe00640 1 --2- 172.21.15.31:0/3279691968 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fcf4c0a31e0 0x7fcf4c0a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:38.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.175+0000 7fcf70637740 1 -- start start 2024-03-21T10:40:38.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.175+0000 7fcf5fe00640 1 -- 172.21.15.31:0/3279691968 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fcf580f19b0 con 0x7fcf68101ff0 2024-03-21T10:40:38.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.175+0000 7fcf70637740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cdba512410 0x55cdba511160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:38.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.175+0000 7fcf70637740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdba511730 0x55cdba511bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:38.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.176+0000 7fcf70637740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdba51aaa0 0x55cdba51ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:38.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.176+0000 7fcf70637740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdba46d250 con 0x55cdba512410 2024-03-21T10:40:38.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.176+0000 7fcf70637740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdba415a30 con 0x55cdba51aaa0 2024-03-21T10:40:38.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.176+0000 7fcf70637740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdba4692f0 con 0x55cdba511730 2024-03-21T10:40:38.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.176+0000 7fcf5e000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdba511730 0x55cdba511bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:38.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.176+0000 7fcf5e000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdba511730 0x55cdba511bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:51684/0 (socket says 172.21.15.31:51684) 2024-03-21T10:40:38.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.176+0000 7fcf5e000640 1 -- 172.21.15.31:0/1016804613 learned_addr learned my addr 172.21.15.31:0/1016804613 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:38.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.176+0000 7fcf67400640 1 --2- 172.21.15.31:0/1016804613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdba51aaa0 0x55cdba51ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:38.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.176+0000 7fcf5e000640 1 -- 172.21.15.31:0/1016804613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdba51aaa0 msgr2=0x55cdba51ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:38.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.176+0000 7fcf5e000640 1 --2- 172.21.15.31:0/1016804613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdba51aaa0 0x55cdba51ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.177+0000 7fcf5e000640 1 -- 172.21.15.31:0/1016804613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cdba512410 msgr2=0x55cdba511160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:40:38.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.177+0000 7fcf66a00640 1 --2- 172.21.15.31:0/1016804613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cdba512410 0x55cdba511160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:38.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.177+0000 7fcf5e000640 1 --2- 172.21.15.31:0/1016804613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cdba512410 0x55cdba511160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.177+0000 7fcf5e000640 1 -- 172.21.15.31:0/1016804613 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cdba400690 con 0x55cdba511730 2024-03-21T10:40:38.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.177+0000 7fcf67400640 1 --2- 172.21.15.31:0/1016804613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdba51aaa0 0x55cdba51ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:40:38.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.177+0000 7fcf67400640 1 --2- 172.21.15.31:0/3279691968 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fcf4c0a31e0 0x7fcf4c0a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:38.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.179+0000 7fcf66a00640 1 --2- 172.21.15.31:0/1016804613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cdba512410 0x55cdba511160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:40:38.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.179+0000 7fcf5e000640 1 --2- 172.21.15.31:0/1016804613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdba511730 0x55cdba511bb0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcf580ef3c0 tx=0x7fcf580bd4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb0783f568b929a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:38.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.180+0000 7fcf57e00640 1 -- 172.21.15.31:0/1016804613 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf580be270 con 0x55cdba511730 2024-03-21T10:40:38.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.180+0000 7fcf57e00640 1 -- 172.21.15.31:0/1016804613 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf580bdb70 con 0x55cdba511730 2024-03-21T10:40:38.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.180+0000 7fcf57e00640 1 -- 172.21.15.31:0/1016804613 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf580ba1e0 con 0x55cdba511730 2024-03-21T10:40:38.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.181+0000 7fcf70637740 1 -- 172.21.15.31:0/1016804613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdba511730 msgr2=0x55cdba511bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:38.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.181+0000 7fcf70637740 1 --2- 172.21.15.31:0/1016804613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdba511730 0x55cdba511bb0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcf580ef3c0 tx=0x7fcf580bd4d0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.182+0000 7fcf67400640 1 --2- 172.21.15.31:0/3279691968 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fcf4c0a31e0 0x7fcf4c0a56a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcf60009fd0 tx=0x7fcf60009500 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:38.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.182+0000 7fcf70637740 1 -- 172.21.15.31:0/1016804613 shutdown_connections 2024-03-21T10:40:38.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.182+0000 7fcf70637740 1 --2- 172.21.15.31:0/1016804613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdba51aaa0 0x55cdba51ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.182+0000 7fcf70637740 1 --2- 172.21.15.31:0/1016804613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdba511730 0x55cdba511bb0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.182+0000 7fcf70637740 1 --2- 172.21.15.31:0/1016804613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cdba512410 0x55cdba511160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.182+0000 7fcf70637740 1 -- 172.21.15.31:0/1016804613 >> 172.21.15.31:0/1016804613 conn(0x55cdba4d0200 msgr2=0x55cdba50c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:38.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.183+0000 7fcf70637740 1 -- 172.21.15.31:0/1016804613 shutdown_connections 2024-03-21T10:40:38.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.183+0000 7fcf70637740 1 -- 172.21.15.31:0/1016804613 wait complete. 2024-03-21T10:40:38.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.184+0000 7fcf70637740 1 -- start start 2024-03-21T10:40:38.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.184+0000 7fcf70637740 1 Processor -- start 2024-03-21T10:40:38.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.185+0000 7fcf70637740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdba511730 0x55cdba51a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:38.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.185+0000 7fcf70637740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cdba512410 0x55cdba517510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:38.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.185+0000 7fcf70637740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdba51aaa0 0x55cdba517a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:38.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.185+0000 7fcf70637740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdba51d6d0 con 0x55cdba512410 2024-03-21T10:40:38.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.185+0000 7fcf70637740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdba428e80 con 0x55cdba511730 2024-03-21T10:40:38.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.185+0000 7fcf70637740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdba51d850 con 0x55cdba51aaa0 2024-03-21T10:40:38.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.185+0000 7fcf70637740 10 client.0 _ll_register_callbacks cb 0x55cdba43b130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:38.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.186+0000 7fcf66a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdba511730 0x55cdba51a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:38.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.186+0000 7fcf66a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdba511730 0x55cdba51a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:59238/0 (socket says 172.21.15.31:59238) 2024-03-21T10:40:38.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.186+0000 7fcf66a00640 1 -- 172.21.15.31:0/1463787433 learned_addr learned my addr 172.21.15.31:0/1463787433 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:38.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.186+0000 7fcf67400640 1 --2- 172.21.15.31:0/1463787433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdba51aaa0 0x55cdba517a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:38.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.187+0000 7fcf66a00640 1 -- 172.21.15.31:0/1463787433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdba51aaa0 msgr2=0x55cdba517a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:38.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.187+0000 7fcf66a00640 1 --2- 172.21.15.31:0/1463787433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdba51aaa0 0x55cdba517a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.187+0000 7fcf66a00640 1 -- 172.21.15.31:0/1463787433 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cdba512410 msgr2=0x55cdba517510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:40:38.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.188+0000 7fcf5e000640 1 --2- 172.21.15.31:0/1463787433 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cdba512410 0x55cdba517510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:38.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.188+0000 7fcf66a00640 1 --2- 172.21.15.31:0/1463787433 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cdba512410 0x55cdba517510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.188+0000 7fcf66a00640 1 -- 172.21.15.31:0/1463787433 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cdba24b7d0 con 0x55cdba511730 2024-03-21T10:40:38.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.188+0000 7fcf66a00640 1 --2- 172.21.15.31:0/1463787433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdba511730 0x55cdba51a110 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fcf500117d0 tx=0x7fcf50011c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5d5e35b232ea4b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:38.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.188+0000 7fcf54c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:40:38.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.193+0000 7fcf54c00640 1 -- 172.21.15.31:0/1463787433 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf50018070 con 0x55cdba511730 2024-03-21T10:40:38.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.193+0000 7fcf54c00640 1 -- 172.21.15.31:0/1463787433 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fcf5000fd50 con 0x55cdba511730 2024-03-21T10:40:38.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.193+0000 7fcf70637740 1 -- 172.21.15.31:0/1463787433 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf580bdb70 con 0x55cdba511730 2024-03-21T10:40:38.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.194+0000 7fcf54c00640 1 -- 172.21.15.31:0/1463787433 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf50012450 con 0x55cdba511730 2024-03-21T10:40:38.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.194+0000 7fcf70637740 10 client.26234 fetch_fsmap learned FSMap version 25 2024-03-21T10:40:38.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.194+0000 7fcf70637740 1 -- 172.21.15.31:0/1463787433 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55cdba1813f0 con 0x55cdba511730 2024-03-21T10:40:38.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.195+0000 7fcf54c00640 1 -- 172.21.15.31:0/1463787433 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf50012ad0 con 0x55cdba511730 2024-03-21T10:40:38.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.195+0000 7fcf54c00640 1 -- 172.21.15.31:0/1463787433 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fcf50026840 con 0x55cdba511730 2024-03-21T10:40:38.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.195+0000 7fcf70637740 10 client.26234 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:40:38.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.195+0000 7fcf70637740 10 client.26234 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:40:38.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.195+0000 7fcf70637740 20 client.26234 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:38.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.195+0000 7fcf70637740 1 --2- 172.21.15.31:0/1463787433 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55cdba52a9e0 0x55cdba52ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:38.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.195+0000 7fcf70637740 4 client.26234 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:40:38.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.195+0000 7fcf70637740 1 -- 172.21.15.31:0/1463787433 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf580bdb70 con 0x55cdba52a9e0 2024-03-21T10:40:38.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.196+0000 7fcf5e000640 1 --2- 172.21.15.31:0/1463787433 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55cdba52a9e0 0x55cdba52ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:38.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.197+0000 7fcf54c00640 10 client.26234 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:38.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.197+0000 7fcf5e000640 1 --2- 172.21.15.31:0/1463787433 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55cdba52a9e0 0x55cdba52ce20 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1efbcbeace22945d server_cookie=3937273fc307d67d in_seq=0 out_seq=0 2024-03-21T10:40:38.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.198+0000 7fcf54c00640 1 -- 172.21.15.31:0/1463787433 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7fcf580bdb70 con 0x55cdba52a9e0 2024-03-21T10:40:38.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.198+0000 7fcf54c00640 10 client.26234 handle_command_reply: tid=0 2024-03-21T10:40:38.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.198+0000 7fcf70637740 1 client.26234 shutdown 2024-03-21T10:40:38.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.198+0000 7fcf70637740 1 -- 172.21.15.31:0/1463787433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdba511730 msgr2=0x55cdba51a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:38.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.198+0000 7fcf70637740 1 --2- 172.21.15.31:0/1463787433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdba511730 0x55cdba51a110 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fcf500117d0 tx=0x7fcf50011c90 comp rx=0 tx=0).stop 2024-03-21T10:40:38.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.198+0000 7fcf70637740 1 -- 172.21.15.31:0/1463787433 shutdown_connections 2024-03-21T10:40:38.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.198+0000 7fcf70637740 1 --2- 172.21.15.31:0/1463787433 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55cdba52a9e0 0x55cdba52ce20 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.198+0000 7fcf54c00640 0 client.26234 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:38.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.198+0000 7fcf70637740 1 --2- 172.21.15.31:0/1463787433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdba51aaa0 0x55cdba517a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.198+0000 7fcf70637740 1 --2- 172.21.15.31:0/1463787433 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cdba512410 0x55cdba517510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.198+0000 7fcf70637740 1 --2- 172.21.15.31:0/1463787433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdba511730 0x55cdba51a110 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.198+0000 7fcf70637740 1 -- 172.21.15.31:0/1463787433 >> 172.21.15.31:0/1463787433 conn(0x55cdba4d0200 msgr2=0x55cdba50ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:38.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.201+0000 7fcf70637740 1 -- 172.21.15.31:0/1463787433 shutdown_connections 2024-03-21T10:40:38.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.201+0000 7fcf70637740 1 -- 172.21.15.31:0/1463787433 wait complete. 2024-03-21T10:40:38.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.201+0000 7fcf70637740 20 client.26234 trim_cache size 0 max 16384 2024-03-21T10:40:38.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.245+0000 7fcf70637740 1 Processor -- start 2024-03-21T10:40:38.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.246+0000 7fcf70637740 1 -- start start 2024-03-21T10:40:38.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.246+0000 7fcf70637740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cdba512410 0x55cdba517aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:38.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.246+0000 7fcf70637740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdba51aaa0 0x55cdba519c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:38.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.246+0000 7fcf70637740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdba51a180 0x55cdba51be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:38.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.246+0000 7fcf70637740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf681aed50 con 0x55cdba512410 2024-03-21T10:40:38.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.246+0000 7fcf70637740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf681ac850 con 0x55cdba51aaa0 2024-03-21T10:40:38.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.246+0000 7fcf70637740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf681aa8f0 con 0x55cdba51a180 2024-03-21T10:40:38.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.246+0000 7fcf66a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cdba512410 0x55cdba517aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:38.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.246+0000 7fcf66a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cdba512410 0x55cdba517aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57912/0 (socket says 172.21.15.31:57912) 2024-03-21T10:40:38.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.246+0000 7fcf66a00640 1 -- 172.21.15.31:0/3013722034 learned_addr learned my addr 172.21.15.31:0/3013722034 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:38.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.247+0000 7fcf5e000640 1 --2- 172.21.15.31:0/3013722034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdba51aaa0 0x55cdba519c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:38.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.247+0000 7fcf67400640 1 --2- 172.21.15.31:0/3013722034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdba51a180 0x55cdba51be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:38.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.247+0000 7fcf66a00640 1 -- 172.21.15.31:0/3013722034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdba51a180 msgr2=0x55cdba51be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:38.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.247+0000 7fcf66a00640 1 --2- 172.21.15.31:0/3013722034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdba51a180 0x55cdba51be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.247+0000 7fcf66a00640 1 -- 172.21.15.31:0/3013722034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdba51aaa0 msgr2=0x55cdba519c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:38.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.247+0000 7fcf66a00640 1 --2- 172.21.15.31:0/3013722034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdba51aaa0 0x55cdba519c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.247+0000 7fcf66a00640 1 -- 172.21.15.31:0/3013722034 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cdba1410f0 con 0x55cdba512410 2024-03-21T10:40:38.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.247+0000 7fcf67400640 1 --2- 172.21.15.31:0/3013722034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdba51a180 0x55cdba51be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:40:38.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.247+0000 7fcf66a00640 1 --2- 172.21.15.31:0/3013722034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cdba512410 0x55cdba517aa0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcf500195f0 tx=0x7fcf50001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=324ac4e7cc6059f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:38.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.248+0000 7fcf57400640 1 -- 172.21.15.31:0/3013722034 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf50012450 con 0x55cdba512410 2024-03-21T10:40:38.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.248+0000 7fcf57400640 1 -- 172.21.15.31:0/3013722034 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf5002b020 con 0x55cdba512410 2024-03-21T10:40:38.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.248+0000 7fcf57400640 1 -- 172.21.15.31:0/3013722034 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf50026470 con 0x55cdba512410 2024-03-21T10:40:38.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.249+0000 7fcf70637740 1 -- 172.21.15.31:0/3013722034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cdba512410 msgr2=0x55cdba517aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:38.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.249+0000 7fcf70637740 1 --2- 172.21.15.31:0/3013722034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cdba512410 0x55cdba517aa0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcf500195f0 tx=0x7fcf50001e50 comp rx=0 tx=0).stop 2024-03-21T10:40:38.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.249+0000 7fcf70637740 1 -- 172.21.15.31:0/3013722034 shutdown_connections 2024-03-21T10:40:38.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.249+0000 7fcf70637740 1 --2- 172.21.15.31:0/3013722034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdba51a180 0x55cdba51be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.249+0000 7fcf70637740 1 --2- 172.21.15.31:0/3013722034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdba51aaa0 0x55cdba519c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.249+0000 7fcf70637740 1 --2- 172.21.15.31:0/3013722034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cdba512410 0x55cdba517aa0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.249+0000 7fcf70637740 1 -- 172.21.15.31:0/3013722034 >> 172.21.15.31:0/3013722034 conn(0x55cdba4d0200 msgr2=0x55cdba502a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:38.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.249+0000 7fcf70637740 1 -- 172.21.15.31:0/3013722034 shutdown_connections 2024-03-21T10:40:38.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.249+0000 7fcf70637740 1 -- 172.21.15.31:0/3013722034 wait complete. 2024-03-21T10:40:38.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.250+0000 7fcf70637740 1 -- start start 2024-03-21T10:40:38.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.250+0000 7fcf70637740 1 Processor -- start 2024-03-21T10:40:38.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.251+0000 7fcf70637740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdba512410 0x55cdba581a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:38.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.251+0000 7fcf70637740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdba51a180 0x55cdba521aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:38.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.251+0000 7fcf70637740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cdba51aaa0 0x55cdba521fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:38.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.251+0000 7fcf70637740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf681aaef0 con 0x55cdba51aaa0 2024-03-21T10:40:38.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.251+0000 7fcf70637740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf681ac1f0 con 0x55cdba512410 2024-03-21T10:40:38.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.251+0000 7fcf70637740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf681ab530 con 0x55cdba51a180 2024-03-21T10:40:38.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.251+0000 7fcf70637740 10 client.0 _ll_register_callbacks cb 0x7fcf5000bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:38.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.251+0000 7fcf66a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdba512410 0x55cdba581a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:38.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.251+0000 7fcf66a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdba512410 0x55cdba581a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:59264/0 (socket says 172.21.15.31:59264) 2024-03-21T10:40:38.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.251+0000 7fcf66a00640 1 -- 172.21.15.31:0/398916777 learned_addr learned my addr 172.21.15.31:0/398916777 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:38.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.252+0000 7fcf67400640 1 --2- 172.21.15.31:0/398916777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cdba51aaa0 0x55cdba521fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:38.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.252+0000 7fcf66a00640 1 -- 172.21.15.31:0/398916777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdba51a180 msgr2=0x55cdba521aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:38.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.252+0000 7fcf66a00640 1 --2- 172.21.15.31:0/398916777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdba51a180 0x55cdba521aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.252+0000 7fcf66a00640 1 -- 172.21.15.31:0/398916777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cdba51aaa0 msgr2=0x55cdba521fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:38.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.252+0000 7fcf66a00640 1 --2- 172.21.15.31:0/398916777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cdba51aaa0 0x55cdba521fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.252+0000 7fcf66a00640 1 -- 172.21.15.31:0/398916777 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cdba522520 con 0x55cdba512410 2024-03-21T10:40:38.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.252+0000 7fcf67400640 1 --2- 172.21.15.31:0/398916777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cdba51aaa0 0x55cdba521fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:38.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.256+0000 7fcf66a00640 1 --2- 172.21.15.31:0/398916777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdba512410 0x55cdba581a40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fcf50011cd0 tx=0x7fcf50005b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85f55b044681b01b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:38.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.256+0000 7fcf54c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:40:38.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.256+0000 7fcf54c00640 1 -- 172.21.15.31:0/398916777 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf50026680 con 0x55cdba512410 2024-03-21T10:40:38.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.257+0000 7fcf54c00640 1 -- 172.21.15.31:0/398916777 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fcf50026b60 con 0x55cdba512410 2024-03-21T10:40:38.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.257+0000 7fcf70637740 1 -- 172.21.15.31:0/398916777 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf5002b020 con 0x55cdba512410 2024-03-21T10:40:38.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.258+0000 7fcf54c00640 1 -- 172.21.15.31:0/398916777 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf50026d60 con 0x55cdba512410 2024-03-21T10:40:38.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.258+0000 7fcf70637740 10 client.26240 fetch_fsmap learned FSMap version 25 2024-03-21T10:40:38.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.258+0000 7fcf70637740 1 -- 172.21.15.31:0/398916777 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55cdba580ec0 con 0x55cdba512410 2024-03-21T10:40:38.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.258+0000 7fcf54c00640 1 -- 172.21.15.31:0/398916777 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf5002b020 con 0x55cdba512410 2024-03-21T10:40:38.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.259+0000 7fcf54c00640 1 -- 172.21.15.31:0/398916777 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fcf50026840 con 0x55cdba512410 2024-03-21T10:40:38.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.259+0000 7fcf70637740 10 client.26240 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:40:38.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.259+0000 7fcf70637740 10 client.26240 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:40:38.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.259+0000 7fcf70637740 20 client.26240 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:38.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.259+0000 7fcf70637740 1 --2- 172.21.15.31:0/398916777 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55cdba508390 0x55cdba50a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:38.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.259+0000 7fcf70637740 4 client.26240 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:40:38.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.259+0000 7fcf70637740 1 -- 172.21.15.31:0/398916777 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55cdba47a790 con 0x55cdba508390 2024-03-21T10:40:38.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.259+0000 7fcf5e000640 1 --2- 172.21.15.31:0/398916777 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55cdba508390 0x55cdba50a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:38.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.260+0000 7fcf5e000640 1 --2- 172.21.15.31:0/398916777 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55cdba508390 0x55cdba50a7b0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd33e35dfe5c468f server_cookie=28d269a5f8e32928 in_seq=0 out_seq=0 2024-03-21T10:40:38.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.260+0000 7fcf54c00640 10 client.26240 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:38.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.260+0000 7fcf54c00640 1 -- 172.21.15.31:0/398916777 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55cdba47a790 con 0x55cdba508390 2024-03-21T10:40:38.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.260+0000 7fcf54c00640 10 client.26240 handle_command_reply: tid=0 2024-03-21T10:40:38.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.260+0000 7fcf70637740 1 client.26240 shutdown 2024-03-21T10:40:38.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.261+0000 7fcf70637740 1 -- 172.21.15.31:0/398916777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdba512410 msgr2=0x55cdba581a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:38.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.261+0000 7fcf70637740 1 --2- 172.21.15.31:0/398916777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdba512410 0x55cdba581a40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fcf50011cd0 tx=0x7fcf50005b90 comp rx=0 tx=0).stop 2024-03-21T10:40:38.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.261+0000 7fcf70637740 1 -- 172.21.15.31:0/398916777 shutdown_connections 2024-03-21T10:40:38.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.261+0000 7fcf70637740 1 --2- 172.21.15.31:0/398916777 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55cdba508390 0x55cdba50a7b0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.261+0000 7fcf70637740 1 --2- 172.21.15.31:0/398916777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cdba51aaa0 0x55cdba521fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.261+0000 7fcf70637740 1 --2- 172.21.15.31:0/398916777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdba51a180 0x55cdba521aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.261+0000 7fcf54c00640 0 client.26240 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:38.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.261+0000 7fcf70637740 1 --2- 172.21.15.31:0/398916777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cdba512410 0x55cdba581a40 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.261+0000 7fcf70637740 1 -- 172.21.15.31:0/398916777 >> 172.21.15.31:0/398916777 conn(0x55cdba4d0200 msgr2=0x55cdba531720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:38.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.261+0000 7fcf70637740 1 -- 172.21.15.31:0/398916777 shutdown_connections 2024-03-21T10:40:38.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.261+0000 7fcf70637740 1 -- 172.21.15.31:0/398916777 wait complete. 2024-03-21T10:40:38.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.261+0000 7fcf70637740 20 client.26240 trim_cache size 0 max 16384 2024-03-21T10:40:38.261 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:40:38.262 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-21T10:40:38.262 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-03-21T10:40:38.262 INFO:teuthology.orchestra.run.smithi031.stdout: "14cc37d1-4a0c-4727-a49e-74350e41cd3e": { 2024-03-21T10:40:38.262 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-03-21T10:40:38.262 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "14cc37d1-4a0c-4727-a49e-74350e41cd3e", 2024-03-21T10:40:38.262 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-03-21T10:40:38.262 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:40:38.262 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:40:38.262 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:40:38.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.262+0000 7fcf6d800640 1 -- 172.21.15.31:0/3279691968 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fcf4c0a31e0 msgr2=0x7fcf4c0a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:38.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.262+0000 7fcf6d800640 1 --2- 172.21.15.31:0/3279691968 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fcf4c0a31e0 0x7fcf4c0a56a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcf60009fd0 tx=0x7fcf60009500 comp rx=0 tx=0).stop 2024-03-21T10:40:38.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.262+0000 7fcf6d800640 1 -- 172.21.15.31:0/3279691968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf68101ff0 msgr2=0x7fcf6819a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:38.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.262+0000 7fcf6d800640 1 --2- 172.21.15.31:0/3279691968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf68101ff0 0x7fcf6819a740 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fcf58036040 tx=0x7fcf58004360 comp rx=0 tx=0).stop 2024-03-21T10:40:38.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.262+0000 7fcf6d800640 1 -- 172.21.15.31:0/3279691968 shutdown_connections 2024-03-21T10:40:38.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.262+0000 7fcf6d800640 1 --2- 172.21.15.31:0/3279691968 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fcf4c0a31e0 0x7fcf4c0a56a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.262+0000 7fcf6d800640 1 --2- 172.21.15.31:0/3279691968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf68107640 0x7fcf6819eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.262+0000 7fcf6d800640 1 --2- 172.21.15.31:0/3279691968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf68101ff0 0x7fcf6819a740 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.262+0000 7fcf6d800640 1 --2- 172.21.15.31:0/3279691968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf68101630 0x7fcf6819a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:38.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.262+0000 7fcf6d800640 1 -- 172.21.15.31:0/3279691968 >> 172.21.15.31:0/3279691968 conn(0x7fcf680fd140 msgr2=0x7fcf6810ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:38.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.262+0000 7fcf6d800640 1 -- 172.21.15.31:0/3279691968 shutdown_connections 2024-03-21T10:40:38.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:40:38.263+0000 7fcf6d800640 1 -- 172.21.15.31:0/3279691968 wait complete. 2024-03-21T10:40:38.279 INFO:tasks.cephfs.filesystem:scrub status for tag:14cc37d1-4a0c-4727-a49e-74350e41cd3e - {'path': '/', 'tag': '14cc37d1-4a0c-4727-a49e-74350e41cd3e', 'options': 'recursive,force'} 2024-03-21T10:40:39.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:38 smithi040 ceph-mon[36154]: pgmap v65: 129 pgs: 129 active+clean; 5.2 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 217 MiB/s wr, 62 op/s 2024-03-21T10:40:39.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:38 smithi040 ceph-mon[36154]: scrub queued for path: / 2024-03-21T10:40:39.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:38 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:40:39.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:38 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:40:39.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:38 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:40:39.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:38 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:40:39.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:38 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:40:39.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:38 smithi031 ceph-mon[31673]: pgmap v65: 129 pgs: 129 active+clean; 5.2 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 217 MiB/s wr, 62 op/s 2024-03-21T10:40:39.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:38 smithi031 ceph-mon[31673]: scrub queued for path: / 2024-03-21T10:40:39.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:38 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:40:39.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:38 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:40:39.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:38 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:40:39.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:38 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:40:39.311 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:38 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:40:39.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:38 smithi070 ceph-mon[36330]: pgmap v65: 129 pgs: 129 active+clean; 5.2 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 217 MiB/s wr, 62 op/s 2024-03-21T10:40:39.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:38 smithi070 ceph-mon[36330]: scrub queued for path: / 2024-03-21T10:40:39.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:38 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:40:39.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:38 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:40:39.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:38 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:40:39.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:38 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:40:39.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:38 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:40:41.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:40 smithi040 ceph-mon[36154]: pgmap v66: 129 pgs: 129 active+clean; 6.4 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 317 MiB/s wr, 96 op/s 2024-03-21T10:40:41.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:40 smithi031 ceph-mon[31673]: pgmap v66: 129 pgs: 129 active+clean; 6.4 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 317 MiB/s wr, 96 op/s 2024-03-21T10:40:41.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:40 smithi070 ceph-mon[36330]: pgmap v66: 129 pgs: 129 active+clean; 6.4 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 317 MiB/s wr, 96 op/s 2024-03-21T10:40:43.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:42 smithi031 ceph-mon[31673]: pgmap v67: 129 pgs: 129 active+clean; 6.5 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 214 MiB/s wr, 66 op/s 2024-03-21T10:40:43.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:42 smithi070 ceph-mon[36330]: pgmap v67: 129 pgs: 129 active+clean; 6.5 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 214 MiB/s wr, 66 op/s 2024-03-21T10:40:43.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:42 smithi040 ceph-mon[36154]: pgmap v67: 129 pgs: 129 active+clean; 6.5 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 214 MiB/s wr, 66 op/s 2024-03-21T10:40:45.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:44 smithi031 ceph-mon[31673]: pgmap v68: 129 pgs: 129 active+clean; 6.5 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 214 MiB/s wr, 66 op/s 2024-03-21T10:40:45.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:44 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:40:45.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:44 smithi070 ceph-mon[36330]: pgmap v68: 129 pgs: 129 active+clean; 6.5 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 214 MiB/s wr, 66 op/s 2024-03-21T10:40:45.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:44 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:40:45.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:44 smithi040 ceph-mon[36154]: pgmap v68: 129 pgs: 129 active+clean; 6.5 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 214 MiB/s wr, 66 op/s 2024-03-21T10:40:45.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:44 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:40:47.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:46 smithi031 ceph-mon[31673]: pgmap v69: 129 pgs: 129 active+clean; 7.8 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 326 MiB/s wr, 97 op/s 2024-03-21T10:40:47.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:46 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:40:47.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:46 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:40:47.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:46 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:40:47.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:46 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:40:47.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:46 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:40:47.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:46 smithi070 ceph-mon[36330]: pgmap v69: 129 pgs: 129 active+clean; 7.8 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 326 MiB/s wr, 97 op/s 2024-03-21T10:40:47.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:46 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:40:47.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:46 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:40:47.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:46 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:40:47.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:46 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:40:47.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:46 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:40:47.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:46 smithi040 ceph-mon[36154]: pgmap v69: 129 pgs: 129 active+clean; 7.8 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 326 MiB/s wr, 97 op/s 2024-03-21T10:40:47.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:46 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:40:47.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:46 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:40:47.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:46 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:40:47.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:46 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:40:47.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:46 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:40:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:48.997+0000 7fccd1200640 1 -- 172.21.15.31:0/445326571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcccc069480 msgr2=0x7fcccc0698e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:48.997+0000 7fccd1200640 1 --2- 172.21.15.31:0/445326571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcccc069480 0x7fcccc0698e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fccb8009900 tx=0x7fccb802bc60 comp rx=0 tx=0).stop 2024-03-21T10:40:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:48.997+0000 7fccd1200640 1 -- 172.21.15.31:0/445326571 shutdown_connections 2024-03-21T10:40:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:48.997+0000 7fccd1200640 1 --2- 172.21.15.31:0/445326571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcccc104090 0x7fcccc104540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:48.997+0000 7fccd1200640 1 --2- 172.21.15.31:0/445326571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcccc069480 0x7fcccc0698e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:48.997+0000 7fccd1200640 1 --2- 172.21.15.31:0/445326571 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcccc106590 0x7fcccc068f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:48.997+0000 7fccd1200640 1 -- 172.21.15.31:0/445326571 >> 172.21.15.31:0/445326571 conn(0x7fcccc077690 msgr2=0x7fcccc077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:48.997+0000 7fccd1200640 1 -- 172.21.15.31:0/445326571 shutdown_connections 2024-03-21T10:40:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:48.997+0000 7fccd1200640 1 -- 172.21.15.31:0/445326571 wait complete. 2024-03-21T10:40:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:48.998+0000 7fccd1200640 1 Processor -- start 2024-03-21T10:40:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:48.998+0000 7fccd1200640 1 -- start start 2024-03-21T10:40:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:48.999+0000 7fccd1200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcccc069480 0x7fcccc19e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:48.999+0000 7fccd1200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcccc104090 0x7fcccc19ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:48.999+0000 7fccd1200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcccc106590 0x7fcccc1a2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:48.999+0000 7fccd1200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc118050 con 0x7fcccc069480 2024-03-21T10:40:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:48.999+0000 7fccd1200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc117ed0 con 0x7fcccc104090 2024-03-21T10:40:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:48.999+0000 7fccd1200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc1181d0 con 0x7fcccc106590 2024-03-21T10:40:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:48.999+0000 7fccc9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcccc069480 0x7fcccc19e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.000+0000 7fccca800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcccc106590 0x7fcccc1a2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.000+0000 7fccc9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcccc069480 0x7fcccc19e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50308/0 (socket says 172.21.15.31:50308) 2024-03-21T10:40:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.000+0000 7fccc9e00640 1 -- 172.21.15.31:0/3497890969 learned_addr learned my addr 172.21.15.31:0/3497890969 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.000+0000 7fccc9400640 1 --2- 172.21.15.31:0/3497890969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcccc104090 0x7fcccc19ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.001+0000 7fccc9e00640 1 -- 172.21.15.31:0/3497890969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcccc106590 msgr2=0x7fcccc1a2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.001+0000 7fccc9e00640 1 --2- 172.21.15.31:0/3497890969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcccc106590 0x7fcccc1a2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.001+0000 7fccc9e00640 1 -- 172.21.15.31:0/3497890969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcccc104090 msgr2=0x7fcccc19ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.001+0000 7fccc9e00640 1 --2- 172.21.15.31:0/3497890969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcccc104090 0x7fcccc19ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.001+0000 7fccc9e00640 1 -- 172.21.15.31:0/3497890969 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc1a3580 con 0x7fcccc069480 2024-03-21T10:40:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.001+0000 7fccc9400640 1 --2- 172.21.15.31:0/3497890969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcccc104090 0x7fcccc19ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.001+0000 7fccc9e00640 1 --2- 172.21.15.31:0/3497890969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcccc069480 0x7fcccc19e590 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fccbc00e9f0 tx=0x7fccbc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.002+0000 7fccc2a00640 1 -- 172.21.15.31:0/3497890969 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccbc00ccc0 con 0x7fcccc069480 2024-03-21T10:40:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.002+0000 7fccc2a00640 1 -- 172.21.15.31:0/3497890969 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fccbc010910 con 0x7fcccc069480 2024-03-21T10:40:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.002+0000 7fccd1200640 1 -- 172.21.15.31:0/3497890969 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcccc1a3870 con 0x7fcccc069480 2024-03-21T10:40:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.002+0000 7fccc2a00640 1 -- 172.21.15.31:0/3497890969 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccbc0188a0 con 0x7fcccc069480 2024-03-21T10:40:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.002+0000 7fccd1200640 1 -- 172.21.15.31:0/3497890969 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcccc1ab100 con 0x7fcccc069480 2024-03-21T10:40:49.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.006+0000 7fccc2a00640 1 -- 172.21.15.31:0/3497890969 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fccbc018a40 con 0x7fcccc069480 2024-03-21T10:40:49.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.006+0000 7fccd3f8a740 1 Processor -- start 2024-03-21T10:40:49.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.006+0000 7fccd3f8a740 1 -- start start 2024-03-21T10:40:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.007+0000 7fccd3f8a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 0x563797bfa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.007+0000 7fccd3f8a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563797bfa8f0 0x563797c03180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.007+0000 7fccd3f8a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563797c036c0 0x563797c05ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.007+0000 7fccd3f8a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563797bbc3b0 con 0x563797c036c0 2024-03-21T10:40:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.007+0000 7fccd3f8a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x563797b5e830 con 0x563797bfa8f0 2024-03-21T10:40:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.007+0000 7fccd3f8a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x563797b075d0 con 0x563797bf9fd0 2024-03-21T10:40:49.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.007+0000 7fccca800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 0x563797bfa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:49.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.007+0000 7fccca800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 0x563797bfa3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:57290/0 (socket says 172.21.15.31:57290) 2024-03-21T10:40:49.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.007+0000 7fccca800640 1 -- 172.21.15.31:0/1895318744 learned_addr learned my addr 172.21.15.31:0/1895318744 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:49.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.007+0000 7fccc9e00640 1 --2- 172.21.15.31:0/1895318744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563797bfa8f0 0x563797c03180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:49.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.008+0000 7fccca800640 1 -- 172.21.15.31:0/1895318744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563797bfa8f0 msgr2=0x563797c03180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:49.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.008+0000 7fccca800640 1 --2- 172.21.15.31:0/1895318744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563797bfa8f0 0x563797c03180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.008+0000 7fccca800640 1 -- 172.21.15.31:0/1895318744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563797c036c0 msgr2=0x563797c05ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:49.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.008+0000 7fccca800640 1 --2- 172.21.15.31:0/1895318744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563797c036c0 0x563797c05ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.008+0000 7fccca800640 1 -- 172.21.15.31:0/1895318744 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563797b1f2c0 con 0x563797bf9fd0 2024-03-21T10:40:49.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.008+0000 7fccc9e00640 1 --2- 172.21.15.31:0/1895318744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563797bfa8f0 0x563797c03180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:40:49.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.008+0000 7fccc2a00640 1 --2- 172.21.15.31:0/3497890969 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fccb00a3080 0x7fccb00a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:49.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.008+0000 7fccc2a00640 1 -- 172.21.15.31:0/3497890969 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fccbc0cff30 con 0x7fcccc069480 2024-03-21T10:40:49.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.008+0000 7fccc9e00640 1 --2- 172.21.15.31:0/3497890969 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fccb00a3080 0x7fccb00a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:49.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.008+0000 7fccca800640 1 --2- 172.21.15.31:0/1895318744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 0x563797bfa3b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fccc4011440 tx=0x7fccc4011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f890a2e25d2cce13 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:49.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.009+0000 7fccb7400640 1 -- 172.21.15.31:0/1895318744 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc4003c30 con 0x563797bf9fd0 2024-03-21T10:40:49.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.009+0000 7fccb7400640 1 -- 172.21.15.31:0/1895318744 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fccc4003dd0 con 0x563797bf9fd0 2024-03-21T10:40:49.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.009+0000 7fccb7400640 1 -- 172.21.15.31:0/1895318744 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc400fb50 con 0x563797bf9fd0 2024-03-21T10:40:49.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.010+0000 7fccd3f8a740 1 -- 172.21.15.31:0/1895318744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 msgr2=0x563797bfa3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:49.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.010+0000 7fccc9e00640 1 --2- 172.21.15.31:0/3497890969 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fccb00a3080 0x7fccb00a5540 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x563797bf9a80 tx=0x7fccbc0bc040 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:49.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.010+0000 7fccd3f8a740 1 --2- 172.21.15.31:0/1895318744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 0x563797bfa3b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fccc4011440 tx=0x7fccc4011900 comp rx=0 tx=0).stop 2024-03-21T10:40:49.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.010+0000 7fccd3f8a740 1 -- 172.21.15.31:0/1895318744 shutdown_connections 2024-03-21T10:40:49.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.010+0000 7fccd3f8a740 1 --2- 172.21.15.31:0/1895318744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563797c036c0 0x563797c05ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.010+0000 7fccd3f8a740 1 --2- 172.21.15.31:0/1895318744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563797bfa8f0 0x563797c03180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.010+0000 7fccd3f8a740 1 --2- 172.21.15.31:0/1895318744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 0x563797bfa3b0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.010+0000 7fccd3f8a740 1 -- 172.21.15.31:0/1895318744 >> 172.21.15.31:0/1895318744 conn(0x563797bf48f0 msgr2=0x563797bf4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:49.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.011+0000 7fccd3f8a740 1 -- 172.21.15.31:0/1895318744 shutdown_connections 2024-03-21T10:40:49.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.011+0000 7fccd3f8a740 1 -- 172.21.15.31:0/1895318744 wait complete. 2024-03-21T10:40:49.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.011+0000 7fccd3f8a740 1 -- start start 2024-03-21T10:40:49.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.011+0000 7fccd3f8a740 1 Processor -- start 2024-03-21T10:40:49.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.012+0000 7fccd3f8a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 0x563797c00740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:49.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.012+0000 7fccd3f8a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563797bfa8f0 0x563797c00c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:49.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.012+0000 7fccd3f8a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563797c036c0 0x563797c011c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:49.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.012+0000 7fccd3f8a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563797b1aa20 con 0x563797bfa8f0 2024-03-21T10:40:49.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.012+0000 7fccd3f8a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x563797b5a8d0 con 0x563797c036c0 2024-03-21T10:40:49.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.012+0000 7fccd3f8a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x563797c062f0 con 0x563797bf9fd0 2024-03-21T10:40:49.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.012+0000 7fccd3f8a740 10 client.0 _ll_register_callbacks cb 0x563797b571c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:49.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.012+0000 7fccca800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 0x563797c00740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:49.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.012+0000 7fccca800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 0x563797c00740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:57296/0 (socket says 172.21.15.31:57296) 2024-03-21T10:40:49.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.012+0000 7fccca800640 1 -- 172.21.15.31:0/2003963976 learned_addr learned my addr 172.21.15.31:0/2003963976 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:49.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.013+0000 7fccca800640 1 --2- 172.21.15.31:0/2003963976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563797c036c0 0x563797c011c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:49.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.013+0000 7fccc9400640 1 --2- 172.21.15.31:0/2003963976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563797bfa8f0 0x563797c00c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:49.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.013+0000 7fccca800640 1 -- 172.21.15.31:0/2003963976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563797c036c0 msgr2=0x563797c011c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:49.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.013+0000 7fccca800640 1 --2- 172.21.15.31:0/2003963976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563797c036c0 0x563797c011c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.013+0000 7fccca800640 1 -- 172.21.15.31:0/2003963976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563797bfa8f0 msgr2=0x563797c00c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:49.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.013+0000 7fccca800640 1 --2- 172.21.15.31:0/2003963976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563797bfa8f0 0x563797c00c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.013+0000 7fccca800640 1 -- 172.21.15.31:0/2003963976 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563797873210 con 0x563797bf9fd0 2024-03-21T10:40:49.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.013+0000 7fccca800640 1 --2- 172.21.15.31:0/2003963976 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 0x563797c00740 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fccc4000f80 tx=0x7fccc400f6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16d4a688eab097ac server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:49.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.013+0000 7fccabe00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:40:49.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.014+0000 7fccabe00640 1 -- 172.21.15.31:0/2003963976 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc400fd30 con 0x563797bf9fd0 2024-03-21T10:40:49.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.014+0000 7fccabe00640 1 -- 172.21.15.31:0/2003963976 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fccc4004000 con 0x563797bf9fd0 2024-03-21T10:40:49.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.014+0000 7fccd3f8a740 1 -- 172.21.15.31:0/2003963976 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fccc4003dd0 con 0x563797bf9fd0 2024-03-21T10:40:49.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.014+0000 7fccabe00640 1 -- 172.21.15.31:0/2003963976 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fccc4004200 con 0x563797bf9fd0 2024-03-21T10:40:49.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.014+0000 7fccd3f8a740 10 client.26215 fetch_fsmap learned FSMap version 25 2024-03-21T10:40:49.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.014+0000 7fccd3f8a740 1 -- 172.21.15.31:0/2003963976 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56379793f620 con 0x563797bf9fd0 2024-03-21T10:40:49.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.015+0000 7fccabe00640 1 -- 172.21.15.31:0/2003963976 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fccc4003dd0 con 0x563797bf9fd0 2024-03-21T10:40:49.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.015+0000 7fccabe00640 1 -- 172.21.15.31:0/2003963976 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fccc40198c0 con 0x563797bf9fd0 2024-03-21T10:40:49.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.016+0000 7fccd3f8a740 10 client.26215 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:40:49.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.016+0000 7fccd3f8a740 10 client.26215 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:40:49.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.016+0000 7fccd3f8a740 20 client.26215 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:49.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.016+0000 7fccd3f8a740 1 --2- 172.21.15.31:0/2003963976 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563797c13690 0x563797c15a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:49.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.016+0000 7fccd3f8a740 4 client.26215 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:40:49.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.016+0000 7fccd3f8a740 1 -- 172.21.15.31:0/2003963976 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563797b6bd70 con 0x563797c13690 2024-03-21T10:40:49.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.017+0000 7fccc9400640 1 --2- 172.21.15.31:0/2003963976 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563797c13690 0x563797c15a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:49.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.017+0000 7fccc9400640 1 --2- 172.21.15.31:0/2003963976 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563797c13690 0x563797c15a80 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6e73f0054189283 server_cookie=be9125b9dc45e66e in_seq=0 out_seq=0 2024-03-21T10:40:49.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.018+0000 7fccabe00640 10 client.26215 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:49.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.018+0000 7fccabe00640 1 -- 172.21.15.31:0/2003963976 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563797b6bd70 con 0x563797c13690 2024-03-21T10:40:49.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.018+0000 7fccabe00640 10 client.26215 handle_command_reply: tid=0 2024-03-21T10:40:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.019+0000 7fccd3f8a740 1 client.26215 shutdown 2024-03-21T10:40:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.022+0000 7fccd3f8a740 1 -- 172.21.15.31:0/2003963976 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 msgr2=0x563797c00740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.022+0000 7fccd3f8a740 1 --2- 172.21.15.31:0/2003963976 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 0x563797c00740 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fccc4000f80 tx=0x7fccc400f6f0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.023+0000 7fccd3f8a740 1 -- 172.21.15.31:0/2003963976 shutdown_connections 2024-03-21T10:40:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.023+0000 7fccd3f8a740 1 --2- 172.21.15.31:0/2003963976 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563797c13690 0x563797c15a80 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.023+0000 7fccd3f8a740 1 --2- 172.21.15.31:0/2003963976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563797c036c0 0x563797c011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.023+0000 7fccd3f8a740 1 --2- 172.21.15.31:0/2003963976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563797bfa8f0 0x563797c00c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.023+0000 7fccd3f8a740 1 --2- 172.21.15.31:0/2003963976 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 0x563797c00740 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.023+0000 7fccd3f8a740 1 -- 172.21.15.31:0/2003963976 >> 172.21.15.31:0/2003963976 conn(0x563797bf48f0 msgr2=0x563797bf6640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:49.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.023+0000 7fccabe00640 0 client.26215 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:49.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.024+0000 7fccd3f8a740 1 -- 172.21.15.31:0/2003963976 shutdown_connections 2024-03-21T10:40:49.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.026+0000 7fccd3f8a740 1 -- 172.21.15.31:0/2003963976 wait complete. 2024-03-21T10:40:49.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.026+0000 7fccd3f8a740 20 client.26215 trim_cache size 0 max 16384 2024-03-21T10:40:49.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.053+0000 7fccd3f8a740 1 Processor -- start 2024-03-21T10:40:49.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.053+0000 7fccd3f8a740 1 -- start start 2024-03-21T10:40:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.054+0000 7fccd3f8a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 0x563797c02e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.054+0000 7fccd3f8a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563797bfa8f0 0x563797c01000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.054+0000 7fccd3f8a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563797c036c0 0x563797c0ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.054+0000 7fccd3f8a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac012160 con 0x563797bfa8f0 2024-03-21T10:40:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.054+0000 7fccd3f8a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac00fc60 con 0x563797c036c0 2024-03-21T10:40:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.054+0000 7fccd3f8a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac00dd00 con 0x563797bf9fd0 2024-03-21T10:40:49.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.054+0000 7fccca800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 0x563797c02e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:49.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.054+0000 7fccca800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 0x563797c02e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:57312/0 (socket says 172.21.15.31:57312) 2024-03-21T10:40:49.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.055+0000 7fccca800640 1 -- 172.21.15.31:0/3190954271 learned_addr learned my addr 172.21.15.31:0/3190954271 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:49.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.055+0000 7fccca800640 1 --2- 172.21.15.31:0/3190954271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563797c036c0 0x563797c0ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:49.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.055+0000 7fccca800640 1 -- 172.21.15.31:0/3190954271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563797c036c0 msgr2=0x563797c0ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:49.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.055+0000 7fccca800640 1 --2- 172.21.15.31:0/3190954271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563797c036c0 0x563797c0ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.055+0000 7fccca800640 1 -- 172.21.15.31:0/3190954271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563797bfa8f0 msgr2=0x563797c01000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:49.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.055+0000 7fccca800640 1 --2- 172.21.15.31:0/3190954271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563797bfa8f0 0x563797c01000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.055+0000 7fccca800640 1 -- 172.21.15.31:0/3190954271 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563797bfecf0 con 0x563797bf9fd0 2024-03-21T10:40:49.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.055+0000 7fccca800640 1 --2- 172.21.15.31:0/3190954271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 0x563797c02e70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fccc401bff0 tx=0x7fccc4005cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da574c04c925f8e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:49.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.056+0000 7fccb6a00640 1 -- 172.21.15.31:0/3190954271 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc4004e70 con 0x563797bf9fd0 2024-03-21T10:40:49.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.056+0000 7fccb6a00640 1 -- 172.21.15.31:0/3190954271 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fccc402d070 con 0x563797bf9fd0 2024-03-21T10:40:49.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.056+0000 7fccb6a00640 1 -- 172.21.15.31:0/3190954271 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc4019430 con 0x563797bf9fd0 2024-03-21T10:40:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.059+0000 7fccd3f8a740 1 -- 172.21.15.31:0/3190954271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 msgr2=0x563797c02e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.059+0000 7fccd3f8a740 1 --2- 172.21.15.31:0/3190954271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 0x563797c02e70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fccc401bff0 tx=0x7fccc4005cf0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.060+0000 7fccd3f8a740 1 -- 172.21.15.31:0/3190954271 shutdown_connections 2024-03-21T10:40:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.060+0000 7fccd3f8a740 1 --2- 172.21.15.31:0/3190954271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563797c036c0 0x563797c0ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.060+0000 7fccd3f8a740 1 --2- 172.21.15.31:0/3190954271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563797bfa8f0 0x563797c01000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.060+0000 7fccd3f8a740 1 --2- 172.21.15.31:0/3190954271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 0x563797c02e70 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.060+0000 7fccd3f8a740 1 -- 172.21.15.31:0/3190954271 >> 172.21.15.31:0/3190954271 conn(0x563797bf48f0 msgr2=0x563797bf3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:49.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.068+0000 7fccd3f8a740 1 -- 172.21.15.31:0/3190954271 shutdown_connections 2024-03-21T10:40:49.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.068+0000 7fccd3f8a740 1 -- 172.21.15.31:0/3190954271 wait complete. 2024-03-21T10:40:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.069+0000 7fccd3f8a740 1 -- start start 2024-03-21T10:40:49.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.070+0000 7fccd3f8a740 1 Processor -- start 2024-03-21T10:40:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.072+0000 7fccd3f8a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 0x563797bfd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.072+0000 7fccd3f8a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563797bfa8f0 0x563797bfd990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.072+0000 7fccd3f8a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563797c036c0 0x563797c6dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.072+0000 7fccd3f8a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac00e300 con 0x563797bfa8f0 2024-03-21T10:40:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.072+0000 7fccd3f8a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac00f600 con 0x563797c036c0 2024-03-21T10:40:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.072+0000 7fccd3f8a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac00e940 con 0x563797bf9fd0 2024-03-21T10:40:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.072+0000 7fccd3f8a740 10 client.0 _ll_register_callbacks cb 0x7fccc4009eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:40:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.073+0000 7fccca800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 0x563797bfd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.073+0000 7fccca800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 0x563797bfd450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:57322/0 (socket says 172.21.15.31:57322) 2024-03-21T10:40:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.073+0000 7fccca800640 1 -- 172.21.15.31:0/2059194618 learned_addr learned my addr 172.21.15.31:0/2059194618 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:40:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.074+0000 7fccca800640 1 --2- 172.21.15.31:0/2059194618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563797c036c0 0x563797c6dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.074+0000 7fccc9400640 1 --2- 172.21.15.31:0/2059194618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563797bfa8f0 0x563797bfd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.074+0000 7fccca800640 1 -- 172.21.15.31:0/2059194618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563797c036c0 msgr2=0x563797c6dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.074+0000 7fccca800640 1 --2- 172.21.15.31:0/2059194618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563797c036c0 0x563797c6dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.074+0000 7fccca800640 1 -- 172.21.15.31:0/2059194618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563797bfa8f0 msgr2=0x563797bfd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:40:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.074+0000 7fccca800640 1 --2- 172.21.15.31:0/2059194618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563797bfa8f0 0x563797bfd990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.074+0000 7fccca800640 1 -- 172.21.15.31:0/2059194618 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563797c6e2c0 con 0x563797bf9fd0 2024-03-21T10:40:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.074+0000 7fccca800640 1 --2- 172.21.15.31:0/2059194618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 0x563797bfd450 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fccc4003eb0 tx=0x7fccc400f7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c99e36083d65159c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:40:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.075+0000 7fccabe00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:40:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.075+0000 7fccabe00640 1 -- 172.21.15.31:0/2059194618 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc40043a0 con 0x563797bf9fd0 2024-03-21T10:40:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.075+0000 7fccabe00640 1 -- 172.21.15.31:0/2059194618 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fccc4004540 con 0x563797bf9fd0 2024-03-21T10:40:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.075+0000 7fccd3f8a740 1 -- 172.21.15.31:0/2059194618 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fccc402d070 con 0x563797bf9fd0 2024-03-21T10:40:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.075+0000 7fccc9400640 1 --2- 172.21.15.31:0/2059194618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563797bfa8f0 0x563797bfd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:40:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.075+0000 7fccabe00640 1 -- 172.21.15.31:0/2059194618 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fccc4004740 con 0x563797bf9fd0 2024-03-21T10:40:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.078+0000 7fccd3f8a740 10 client.26221 fetch_fsmap learned FSMap version 25 2024-03-21T10:40:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.078+0000 7fccd3f8a740 1 -- 172.21.15.31:0/2059194618 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563797c80480 con 0x563797bf9fd0 2024-03-21T10:40:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.078+0000 7fccabe00640 1 -- 172.21.15.31:0/2059194618 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fccc402d070 con 0x563797bf9fd0 2024-03-21T10:40:49.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.079+0000 7fccabe00640 1 -- 172.21.15.31:0/2059194618 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fccc4019430 con 0x563797bf9fd0 2024-03-21T10:40:49.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.080+0000 7fccd3f8a740 10 client.26221 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:40:49.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.080+0000 7fccd3f8a740 10 client.26221 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:40:49.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.080+0000 7fccd3f8a740 20 client.26221 populate_metadata read hostname 'smithi031' 2024-03-21T10:40:49.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.080+0000 7fccd3f8a740 1 --2- 172.21.15.31:0/2059194618 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563797c80860 0x563797c82c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:40:49.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.080+0000 7fccd3f8a740 4 client.26221 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:40:49.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.080+0000 7fccd3f8a740 1 -- 172.21.15.31:0/2059194618 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563797b660c0 con 0x563797c80860 2024-03-21T10:40:49.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.080+0000 7fccc9400640 1 --2- 172.21.15.31:0/2059194618 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563797c80860 0x563797c82c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:40:49.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.081+0000 7fccc9400640 1 --2- 172.21.15.31:0/2059194618 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563797c80860 0x563797c82c50 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d67aeb30e5eba34 server_cookie=5934f734e3d0f66c in_seq=0 out_seq=0 2024-03-21T10:40:49.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.081+0000 7fccabe00640 10 client.26221 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:49.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.098+0000 7fccabe00640 1 -- 172.21.15.31:0/2059194618 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563797b660c0 con 0x563797c80860 2024-03-21T10:40:49.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.098+0000 7fccabe00640 10 client.26221 handle_command_reply: tid=0 2024-03-21T10:40:49.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.098+0000 7fccd3f8a740 1 client.26221 shutdown 2024-03-21T10:40:49.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.098+0000 7fccd3f8a740 1 -- 172.21.15.31:0/2059194618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 msgr2=0x563797bfd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:49.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.098+0000 7fccd3f8a740 1 --2- 172.21.15.31:0/2059194618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 0x563797bfd450 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fccc4003eb0 tx=0x7fccc400f7e0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.099+0000 7fccd3f8a740 1 -- 172.21.15.31:0/2059194618 shutdown_connections 2024-03-21T10:40:49.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.099+0000 7fccd3f8a740 1 --2- 172.21.15.31:0/2059194618 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563797c80860 0x563797c82c50 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.099+0000 7fccd3f8a740 1 --2- 172.21.15.31:0/2059194618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563797c036c0 0x563797c6dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.099+0000 7fccabe00640 0 client.26221 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:40:49.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.099+0000 7fccd3f8a740 1 --2- 172.21.15.31:0/2059194618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563797bfa8f0 0x563797bfd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.099+0000 7fccd3f8a740 1 --2- 172.21.15.31:0/2059194618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563797bf9fd0 0x563797bfd450 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.099+0000 7fccd3f8a740 1 -- 172.21.15.31:0/2059194618 >> 172.21.15.31:0/2059194618 conn(0x563797bf48f0 msgr2=0x563797c1a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:49.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.099+0000 7fccd3f8a740 1 -- 172.21.15.31:0/2059194618 shutdown_connections 2024-03-21T10:40:49.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.099+0000 7fccd3f8a740 1 -- 172.21.15.31:0/2059194618 wait complete. 2024-03-21T10:40:49.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.100+0000 7fccd3f8a740 20 client.26221 trim_cache size 0 max 16384 2024-03-21T10:40:49.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:40:49.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:40:49.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:40:49.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:40:49.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.101+0000 7fccd1200640 1 -- 172.21.15.31:0/3497890969 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fccb00a3080 msgr2=0x7fccb00a5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:49.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.101+0000 7fccd1200640 1 --2- 172.21.15.31:0/3497890969 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fccb00a3080 0x7fccb00a5540 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x563797bf9a80 tx=0x7fccbc0bc040 comp rx=0 tx=0).stop 2024-03-21T10:40:49.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.101+0000 7fccd1200640 1 -- 172.21.15.31:0/3497890969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcccc069480 msgr2=0x7fcccc19e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:40:49.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.101+0000 7fccd1200640 1 --2- 172.21.15.31:0/3497890969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcccc069480 0x7fcccc19e590 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fccbc00e9f0 tx=0x7fccbc00edf0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.102+0000 7fccd1200640 1 -- 172.21.15.31:0/3497890969 shutdown_connections 2024-03-21T10:40:49.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.102+0000 7fccd1200640 1 --2- 172.21.15.31:0/3497890969 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fccb00a3080 0x7fccb00a5540 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.102+0000 7fccd1200640 1 --2- 172.21.15.31:0/3497890969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcccc106590 0x7fcccc1a2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.102+0000 7fccd1200640 1 --2- 172.21.15.31:0/3497890969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcccc104090 0x7fcccc19ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.102+0000 7fccd1200640 1 --2- 172.21.15.31:0/3497890969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcccc069480 0x7fcccc19e590 secure :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fccbc00e9f0 tx=0x7fccbc00edf0 comp rx=0 tx=0).stop 2024-03-21T10:40:49.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.102+0000 7fccd1200640 1 -- 172.21.15.31:0/3497890969 >> 172.21.15.31:0/3497890969 conn(0x7fcccc077690 msgr2=0x7fcccc103070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:40:49.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.103+0000 7fccd1200640 1 -- 172.21.15.31:0/3497890969 shutdown_connections 2024-03-21T10:40:49.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:40:49.103+0000 7fccd1200640 1 -- 172.21.15.31:0/3497890969 wait complete. 2024-03-21T10:40:49.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:48 smithi031 ceph-mon[31673]: pgmap v70: 129 pgs: 129 active+clean; 7.8 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 219 MiB/s wr, 66 op/s 2024-03-21T10:40:49.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:48 smithi070 ceph-mon[36330]: pgmap v70: 129 pgs: 129 active+clean; 7.8 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 219 MiB/s wr, 66 op/s 2024-03-21T10:40:49.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:48 smithi040 ceph-mon[36154]: pgmap v70: 129 pgs: 129 active+clean; 7.8 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 219 MiB/s wr, 66 op/s 2024-03-21T10:40:51.232 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:50 smithi040 ceph-mon[36154]: pgmap v71: 129 pgs: 129 active+clean; 9.0 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 323 MiB/s wr, 98 op/s 2024-03-21T10:40:51.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:50 smithi031 ceph-mon[31673]: pgmap v71: 129 pgs: 129 active+clean; 9.0 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 323 MiB/s wr, 98 op/s 2024-03-21T10:40:51.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:50 smithi070 ceph-mon[36330]: pgmap v71: 129 pgs: 129 active+clean; 9.0 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 323 MiB/s wr, 98 op/s 2024-03-21T10:40:53.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:52 smithi040 ceph-mon[36154]: pgmap v72: 129 pgs: 129 active+clean; 9.0 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 222 MiB/s wr, 64 op/s 2024-03-21T10:40:53.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:52 smithi031 ceph-mon[31673]: pgmap v72: 129 pgs: 129 active+clean; 9.0 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 222 MiB/s wr, 64 op/s 2024-03-21T10:40:53.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:52 smithi070 ceph-mon[36330]: pgmap v72: 129 pgs: 129 active+clean; 9.0 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 222 MiB/s wr, 64 op/s 2024-03-21T10:40:55.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:54 smithi031 ceph-mon[31673]: pgmap v73: 129 pgs: 129 active+clean; 9.0 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 216 MiB/s wr, 62 op/s 2024-03-21T10:40:55.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:54 smithi070 ceph-mon[36330]: pgmap v73: 129 pgs: 129 active+clean; 9.0 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 216 MiB/s wr, 62 op/s 2024-03-21T10:40:55.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:54 smithi040 ceph-mon[36154]: pgmap v73: 129 pgs: 129 active+clean; 9.0 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 216 MiB/s wr, 62 op/s 2024-03-21T10:40:57.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:57 smithi031 ceph-mon[31673]: pgmap v74: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 318 MiB/s wr, 160 op/s 2024-03-21T10:40:57.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:57 smithi070 ceph-mon[36330]: pgmap v74: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 318 MiB/s wr, 160 op/s 2024-03-21T10:40:57.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:57 smithi040 ceph-mon[36154]: pgmap v74: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 318 MiB/s wr, 160 op/s 2024-03-21T10:40:59.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:40:59 smithi031 ceph-mon[31673]: pgmap v75: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 207 MiB/s wr, 129 op/s 2024-03-21T10:40:59.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:40:59 smithi070 ceph-mon[36330]: pgmap v75: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 207 MiB/s wr, 129 op/s 2024-03-21T10:40:59.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:40:59 smithi040 ceph-mon[36154]: pgmap v75: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 207 MiB/s wr, 129 op/s 2024-03-21T10:41:01.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:01 smithi031 ceph-mon[31673]: pgmap v76: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 291 MiB/s wr, 536 op/s 2024-03-21T10:41:01.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:01 smithi070 ceph-mon[36330]: pgmap v76: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 291 MiB/s wr, 536 op/s 2024-03-21T10:41:01.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:01 smithi040 ceph-mon[36154]: pgmap v76: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 291 MiB/s wr, 536 op/s 2024-03-21T10:41:02.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.324+0000 7f139b200640 1 -- 172.21.15.31:0/2924087797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1394108460 msgr2=0x7f1394114970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:02.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.324+0000 7f139b200640 1 --2- 172.21.15.31:0/2924087797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1394108460 0x7f1394114970 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1390009be0 tx=0x7f139002bef0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.325+0000 7f139b200640 1 -- 172.21.15.31:0/2924087797 shutdown_connections 2024-03-21T10:41:02.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.325+0000 7f139b200640 1 --2- 172.21.15.31:0/2924087797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1394108460 0x7f1394114970 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.325+0000 7f139b200640 1 --2- 172.21.15.31:0/2924087797 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1394107aa0 0x7f1394107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.325+0000 7f139b200640 1 --2- 172.21.15.31:0/2924087797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f139410dab0 0x7f139410de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.325+0000 7f139b200640 1 -- 172.21.15.31:0/2924087797 >> 172.21.15.31:0/2924087797 conn(0x7f1394077220 msgr2=0x7f1394079640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:02.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.325+0000 7f139b200640 1 -- 172.21.15.31:0/2924087797 shutdown_connections 2024-03-21T10:41:02.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.326+0000 7f139b200640 1 -- 172.21.15.31:0/2924087797 wait complete. 2024-03-21T10:41:02.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.327+0000 7f139b200640 1 Processor -- start 2024-03-21T10:41:02.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.328+0000 7f139b200640 1 -- start start 2024-03-21T10:41:02.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.329+0000 7f139b200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1394107aa0 0x7f1394081c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.331+0000 7f139b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1394108460 0x7f1394082180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.331+0000 7f1399e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1394107aa0 0x7f1394081c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.331+0000 7f1399e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1394107aa0 0x7f1394081c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:52148/0 (socket says 172.21.15.31:52148) 2024-03-21T10:41:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.331+0000 7f139b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f139410dab0 0x7f139407aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.331+0000 7f1399400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1394108460 0x7f1394082180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.331+0000 7f1399400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1394108460 0x7f1394082180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51734/0 (socket says 172.21.15.31:51734) 2024-03-21T10:41:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.331+0000 7f1399400640 1 -- 172.21.15.31:0/1842572978 learned_addr learned my addr 172.21.15.31:0/1842572978 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.331+0000 7f139b200640 1 -- 172.21.15.31:0/1842572978 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1394117f10 con 0x7f139410dab0 2024-03-21T10:41:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.331+0000 7f139b200640 1 -- 172.21.15.31:0/1842572978 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1394117d90 con 0x7f1394108460 2024-03-21T10:41:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.331+0000 7f139b200640 1 -- 172.21.15.31:0/1842572978 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1394118090 con 0x7f1394107aa0 2024-03-21T10:41:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.332+0000 7f1399e00640 1 -- 172.21.15.31:0/1842572978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1394108460 msgr2=0x7f1394082180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.332+0000 7f1399e00640 1 --2- 172.21.15.31:0/1842572978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1394108460 0x7f1394082180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.332+0000 7f1399e00640 1 -- 172.21.15.31:0/1842572978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f139410dab0 msgr2=0x7f139407aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.332+0000 7f1399e00640 1 --2- 172.21.15.31:0/1842572978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f139410dab0 0x7f139407aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.332+0000 7f1399e00640 1 -- 172.21.15.31:0/1842572978 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139407b760 con 0x7f1394107aa0 2024-03-21T10:41:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.332+0000 7f1399e00640 1 --2- 172.21.15.31:0/1842572978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1394107aa0 0x7f1394081c40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f138400e790 tx=0x7f138400ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.333+0000 7f138ea00640 1 -- 172.21.15.31:0/1842572978 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138400eea0 con 0x7f1394107aa0 2024-03-21T10:41:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.333+0000 7f138ea00640 1 -- 172.21.15.31:0/1842572978 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f138400ccc0 con 0x7f1394107aa0 2024-03-21T10:41:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.333+0000 7f138ea00640 1 -- 172.21.15.31:0/1842572978 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1384018780 con 0x7f1394107aa0 2024-03-21T10:41:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.333+0000 7f139b200640 1 -- 172.21.15.31:0/1842572978 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f139407ba50 con 0x7f1394107aa0 2024-03-21T10:41:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.333+0000 7f139b200640 1 -- 172.21.15.31:0/1842572978 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f139407fa00 con 0x7f1394107aa0 2024-03-21T10:41:02.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.341+0000 7f138ea00640 1 -- 172.21.15.31:0/1842572978 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f1384010430 con 0x7f1394107aa0 2024-03-21T10:41:02.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.349+0000 7f139f466740 1 Processor -- start 2024-03-21T10:41:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.350+0000 7f139f466740 1 -- start start 2024-03-21T10:41:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.350+0000 7f139f466740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c44d47c1e0 0x55c44d47c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.350+0000 7f139f466740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c44d47cb00 0x55c44d485390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.350+0000 7f139f466740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c44d4858d0 0x55c44d487cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.350+0000 7f139f466740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c44d43e3b0 con 0x55c44d47c1e0 2024-03-21T10:41:02.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.350+0000 7f139f466740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c44d3e0830 con 0x55c44d4858d0 2024-03-21T10:41:02.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.350+0000 7f139f466740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c44d3895d0 con 0x55c44d47cb00 2024-03-21T10:41:02.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.353+0000 7f138ea00640 1 --2- 172.21.15.31:0/1842572978 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f137c0a3170 0x7f137c0a5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:02.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.353+0000 7f138ea00640 1 -- 172.21.15.31:0/1842572978 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f13840cfcd0 con 0x7f1394107aa0 2024-03-21T10:41:02.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.354+0000 7f139a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c44d47c1e0 0x55c44d47c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:02.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.354+0000 7f139a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c44d47c1e0 0x55c44d47c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47488/0 (socket says 172.21.15.31:47488) 2024-03-21T10:41:02.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.354+0000 7f139a800640 1 -- 172.21.15.31:0/3017862372 learned_addr learned my addr 172.21.15.31:0/3017862372 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:02.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.354+0000 7f139a800640 1 --2- 172.21.15.31:0/1842572978 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f137c0a3170 0x7f137c0a5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:02.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.354+0000 7f1399400640 1 --2- 172.21.15.31:0/3017862372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c44d4858d0 0x55c44d487cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:02.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.354+0000 7f1399e00640 1 --2- 172.21.15.31:0/3017862372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c44d47cb00 0x55c44d485390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:02.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.355+0000 7f1399e00640 1 -- 172.21.15.31:0/3017862372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c44d4858d0 msgr2=0x55c44d487cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:02.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.355+0000 7f1399e00640 1 --2- 172.21.15.31:0/3017862372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c44d4858d0 0x55c44d487cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.355+0000 7f1399e00640 1 -- 172.21.15.31:0/3017862372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c44d47c1e0 msgr2=0x55c44d47c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:02.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.355+0000 7f1399e00640 1 --2- 172.21.15.31:0/3017862372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c44d47c1e0 0x55c44d47c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.355+0000 7f1399e00640 1 -- 172.21.15.31:0/3017862372 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c44d3a12c0 con 0x55c44d47cb00 2024-03-21T10:41:02.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.355+0000 7f1399400640 1 --2- 172.21.15.31:0/3017862372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c44d4858d0 0x55c44d487cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:41:02.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.355+0000 7f1399e00640 1 --2- 172.21.15.31:0/3017862372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c44d47cb00 0x55c44d485390 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f13840cd080 tx=0x7f13840980f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d56bb25d4d6ad2ef server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:02.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.356+0000 7f1383000640 1 -- 172.21.15.31:0/3017862372 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13840a4730 con 0x55c44d47cb00 2024-03-21T10:41:02.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.356+0000 7f1383000640 1 -- 172.21.15.31:0/3017862372 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1384010a30 con 0x55c44d47cb00 2024-03-21T10:41:02.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.356+0000 7f1383000640 1 -- 172.21.15.31:0/3017862372 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138409b170 con 0x55c44d47cb00 2024-03-21T10:41:02.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.358+0000 7f139f466740 1 -- 172.21.15.31:0/3017862372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c44d47cb00 msgr2=0x55c44d485390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:02.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.358+0000 7f139f466740 1 --2- 172.21.15.31:0/3017862372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c44d47cb00 0x55c44d485390 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f13840cd080 tx=0x7f13840980f0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.358+0000 7f139a800640 1 --2- 172.21.15.31:0/1842572978 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f137c0a3170 0x7f137c0a5630 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1390005ee0 tx=0x7f13900092a0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:02.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.359+0000 7f139f466740 1 -- 172.21.15.31:0/3017862372 shutdown_connections 2024-03-21T10:41:02.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.359+0000 7f139f466740 1 --2- 172.21.15.31:0/3017862372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c44d4858d0 0x55c44d487cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.359+0000 7f139f466740 1 --2- 172.21.15.31:0/3017862372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c44d47cb00 0x55c44d485390 secure :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f13840cd080 tx=0x7f13840980f0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.359+0000 7f139f466740 1 --2- 172.21.15.31:0/3017862372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c44d47c1e0 0x55c44d47c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.359+0000 7f139f466740 1 -- 172.21.15.31:0/3017862372 >> 172.21.15.31:0/3017862372 conn(0x55c44d476b00 msgr2=0x55c44d476ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:02.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.359+0000 7f139f466740 1 -- 172.21.15.31:0/3017862372 shutdown_connections 2024-03-21T10:41:02.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.362+0000 7f139f466740 1 -- 172.21.15.31:0/3017862372 wait complete. 2024-03-21T10:41:02.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.363+0000 7f139f466740 1 -- start start 2024-03-21T10:41:02.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.363+0000 7f139f466740 1 Processor -- start 2024-03-21T10:41:02.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.365+0000 7f139f466740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c44d47c1e0 0x55c44d482a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:02.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.365+0000 7f139f466740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c44d4858d0 0x55c44d482fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:02.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.365+0000 7f139f466740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c44d483510 0x55c44d4a2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:02.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.366+0000 7f1399400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c44d4858d0 0x55c44d482fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:02.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.366+0000 7f1399400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c44d4858d0 0x55c44d482fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51748/0 (socket says 172.21.15.31:51748) 2024-03-21T10:41:02.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.366+0000 7f1399e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c44d47c1e0 0x55c44d482a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.366+0000 7f1399e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c44d47c1e0 0x55c44d482a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:52160/0 (socket says 172.21.15.31:52160) 2024-03-21T10:41:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.367+0000 7f139a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c44d483510 0x55c44d4a2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.367+0000 7f139f466740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c44d39ca20 con 0x55c44d483510 2024-03-21T10:41:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.367+0000 7f139f466740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c44d3dc8d0 con 0x55c44d4858d0 2024-03-21T10:41:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.367+0000 7f139f466740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c44d4994b0 con 0x55c44d47c1e0 2024-03-21T10:41:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.367+0000 7f139f466740 10 client.0 _ll_register_callbacks cb 0x55c44d3d91c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.370+0000 7f139a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c44d483510 0x55c44d4a2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47494/0 (socket says 172.21.15.31:47494) 2024-03-21T10:41:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.370+0000 7f139a800640 1 -- 172.21.15.31:0/766335371 learned_addr learned my addr 172.21.15.31:0/766335371 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:02.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.371+0000 7f1399400640 1 -- 172.21.15.31:0/766335371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c44d47c1e0 msgr2=0x55c44d482a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:02.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.371+0000 7f1399400640 1 --2- 172.21.15.31:0/766335371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c44d47c1e0 0x55c44d482a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.371+0000 7f1399400640 1 -- 172.21.15.31:0/766335371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c44d483510 msgr2=0x55c44d4a2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:02.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.371+0000 7f1399400640 1 --2- 172.21.15.31:0/766335371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c44d483510 0x55c44d4a2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.371+0000 7f1399400640 1 -- 172.21.15.31:0/766335371 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c44d0f5210 con 0x55c44d4858d0 2024-03-21T10:41:02.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.372+0000 7f1399400640 1 --2- 172.21.15.31:0/766335371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c44d4858d0 0x55c44d482fd0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x55c44d477c90 tx=0x7f1388010c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0fa6aac6cfc3027 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:02.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.372+0000 7f1377400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:41:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.373+0000 7f1377400640 1 -- 172.21.15.31:0/766335371 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138800fc20 con 0x55c44d4858d0 2024-03-21T10:41:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.373+0000 7f1377400640 1 -- 172.21.15.31:0/766335371 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f138800fdc0 con 0x55c44d4858d0 2024-03-21T10:41:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.374+0000 7f139f466740 1 -- 172.21.15.31:0/766335371 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1384010a30 con 0x55c44d4858d0 2024-03-21T10:41:02.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.374+0000 7f1377400640 1 -- 172.21.15.31:0/766335371 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1388005020 con 0x55c44d4858d0 2024-03-21T10:41:02.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.375+0000 7f139f466740 10 client.26270 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:02.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.375+0000 7f139f466740 1 -- 172.21.15.31:0/766335371 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55c44d1c1620 con 0x55c44d4858d0 2024-03-21T10:41:02.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.375+0000 7f1377400640 1 -- 172.21.15.31:0/766335371 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1384010a30 con 0x55c44d4858d0 2024-03-21T10:41:02.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.376+0000 7f1377400640 1 -- 172.21.15.31:0/766335371 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f1388023480 con 0x55c44d4858d0 2024-03-21T10:41:02.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.376+0000 7f139f466740 10 client.26270 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:02.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.377+0000 7f139f466740 10 client.26270 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:41:02.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.377+0000 7f139f466740 20 client.26270 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:02.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.377+0000 7f139f466740 1 --2- 172.21.15.31:0/766335371 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c44d4a6ef0 0x55c44d4a9340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:02.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.377+0000 7f139f466740 4 client.26270 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:41:02.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.377+0000 7f139f466740 1 -- 172.21.15.31:0/766335371 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c44d3edd70 con 0x55c44d4a6ef0 2024-03-21T10:41:02.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.378+0000 7f1399e00640 1 --2- 172.21.15.31:0/766335371 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c44d4a6ef0 0x55c44d4a9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:02.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.379+0000 7f1399e00640 1 --2- 172.21.15.31:0/766335371 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c44d4a6ef0 0x55c44d4a9340 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=841f336828556721 server_cookie=8fb73d33cf4babfe in_seq=0 out_seq=0 2024-03-21T10:41:02.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.379+0000 7f1377400640 10 client.26270 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:02.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.379+0000 7f1377400640 1 -- 172.21.15.31:0/766335371 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c44d3edd70 con 0x55c44d4a6ef0 2024-03-21T10:41:02.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.380+0000 7f1377400640 10 client.26270 handle_command_reply: tid=0 2024-03-21T10:41:02.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.380+0000 7f139f466740 1 client.26270 shutdown 2024-03-21T10:41:02.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.381+0000 7f139f466740 1 -- 172.21.15.31:0/766335371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c44d4858d0 msgr2=0x55c44d482fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:02.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.381+0000 7f139f466740 1 --2- 172.21.15.31:0/766335371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c44d4858d0 0x55c44d482fd0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x55c44d477c90 tx=0x7f1388010c80 comp rx=0 tx=0).stop 2024-03-21T10:41:02.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.382+0000 7f139f466740 1 -- 172.21.15.31:0/766335371 shutdown_connections 2024-03-21T10:41:02.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.382+0000 7f139f466740 1 --2- 172.21.15.31:0/766335371 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c44d4a6ef0 0x55c44d4a9340 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.382+0000 7f139f466740 1 --2- 172.21.15.31:0/766335371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c44d483510 0x55c44d4a2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.382+0000 7f139f466740 1 --2- 172.21.15.31:0/766335371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c44d4858d0 0x55c44d482fd0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.382+0000 7f139f466740 1 --2- 172.21.15.31:0/766335371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c44d47c1e0 0x55c44d482a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.382+0000 7f139f466740 1 -- 172.21.15.31:0/766335371 >> 172.21.15.31:0/766335371 conn(0x55c44d476b00 msgr2=0x55c44d478850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:02.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.382+0000 7f1377400640 0 client.26270 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:02.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.383+0000 7f139f466740 1 -- 172.21.15.31:0/766335371 shutdown_connections 2024-03-21T10:41:02.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.383+0000 7f139f466740 1 -- 172.21.15.31:0/766335371 wait complete. 2024-03-21T10:41:02.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.384+0000 7f139f466740 20 client.26270 trim_cache size 0 max 16384 2024-03-21T10:41:02.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.408+0000 7f139f466740 1 Processor -- start 2024-03-21T10:41:02.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.408+0000 7f139f466740 1 -- start start 2024-03-21T10:41:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.409+0000 7f139f466740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c44d47c1e0 0x55c44d47fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.409+0000 7f139f466740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c44d483510 0x55c44d480290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.409+0000 7f139f466740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c44d4858d0 0x55c44d4752e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.409+0000 7f139f466740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13941b7350 con 0x55c44d47c1e0 2024-03-21T10:41:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.409+0000 7f139f466740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13941b4e50 con 0x55c44d4858d0 2024-03-21T10:41:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.409+0000 7f139f466740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1394067f10 con 0x55c44d483510 2024-03-21T10:41:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.409+0000 7f139a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c44d4858d0 0x55c44d4752e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.409+0000 7f139a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c44d4858d0 0x55c44d4752e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51750/0 (socket says 172.21.15.31:51750) 2024-03-21T10:41:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.409+0000 7f139a800640 1 -- 172.21.15.31:0/768481544 learned_addr learned my addr 172.21.15.31:0/768481544 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.410+0000 7f139a800640 1 -- 172.21.15.31:0/768481544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c44d483510 msgr2=0x55c44d480290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:41:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.410+0000 7f139a800640 1 --2- 172.21.15.31:0/768481544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c44d483510 0x55c44d480290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.410+0000 7f139a800640 1 -- 172.21.15.31:0/768481544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c44d47c1e0 msgr2=0x55c44d47fd50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:41:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.411+0000 7f1399e00640 1 --2- 172.21.15.31:0/768481544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c44d47c1e0 0x55c44d47fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:02.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.411+0000 7f139a800640 1 --2- 172.21.15.31:0/768481544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c44d47c1e0 0x55c44d47fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.411+0000 7f139a800640 1 -- 172.21.15.31:0/768481544 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c44d474f80 con 0x55c44d4858d0 2024-03-21T10:41:02.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.411+0000 7f139a800640 1 --2- 172.21.15.31:0/768481544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c44d4858d0 0x55c44d4752e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f139002c500 tx=0x7f1390045000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d02a3e5bca7d49 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:02.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.411+0000 7f1382600640 1 -- 172.21.15.31:0/768481544 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1390009800 con 0x55c44d4858d0 2024-03-21T10:41:02.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.411+0000 7f1382600640 1 -- 172.21.15.31:0/768481544 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1390003e50 con 0x55c44d4858d0 2024-03-21T10:41:02.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.411+0000 7f1382600640 1 -- 172.21.15.31:0/768481544 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1390004610 con 0x55c44d4858d0 2024-03-21T10:41:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.412+0000 7f139f466740 1 -- 172.21.15.31:0/768481544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c44d4858d0 msgr2=0x55c44d4752e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.412+0000 7f139f466740 1 --2- 172.21.15.31:0/768481544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c44d4858d0 0x55c44d4752e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f139002c500 tx=0x7f1390045000 comp rx=0 tx=0).stop 2024-03-21T10:41:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.413+0000 7f139f466740 1 -- 172.21.15.31:0/768481544 shutdown_connections 2024-03-21T10:41:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.413+0000 7f139f466740 1 --2- 172.21.15.31:0/768481544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c44d4858d0 0x55c44d4752e0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.413+0000 7f139f466740 1 --2- 172.21.15.31:0/768481544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c44d483510 0x55c44d480290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.413+0000 7f139f466740 1 --2- 172.21.15.31:0/768481544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c44d47c1e0 0x55c44d47fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.413+0000 7f139f466740 1 -- 172.21.15.31:0/768481544 >> 172.21.15.31:0/768481544 conn(0x55c44d476b00 msgr2=0x55c44d47ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.413+0000 7f139f466740 1 -- 172.21.15.31:0/768481544 shutdown_connections 2024-03-21T10:41:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.413+0000 7f139f466740 1 -- 172.21.15.31:0/768481544 wait complete. 2024-03-21T10:41:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.413+0000 7f139f466740 1 -- start start 2024-03-21T10:41:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.414+0000 7f139f466740 1 Processor -- start 2024-03-21T10:41:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.414+0000 7f139f466740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c44d47c1e0 0x55c44d49c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:02.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.414+0000 7f139f466740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c44d483510 0x55c44d49c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:02.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.414+0000 7f139f466740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c44d4858d0 0x55c44d5017e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:02.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.414+0000 7f139f466740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1394072b00 con 0x55c44d483510 2024-03-21T10:41:02.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.414+0000 7f139f466740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13941b47f0 con 0x55c44d4858d0 2024-03-21T10:41:02.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.414+0000 7f139f466740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1394073160 con 0x55c44d47c1e0 2024-03-21T10:41:02.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.414+0000 7f139f466740 10 client.0 _ll_register_callbacks cb 0x7f138800a870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:02.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.415+0000 7f139a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c44d4858d0 0x55c44d5017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:02.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.415+0000 7f1399400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c44d483510 0x55c44d49c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:02.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.415+0000 7f139a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c44d4858d0 0x55c44d5017e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51758/0 (socket says 172.21.15.31:51758) 2024-03-21T10:41:02.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.415+0000 7f139a800640 1 -- 172.21.15.31:0/2360091800 learned_addr learned my addr 172.21.15.31:0/2360091800 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:02.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.416+0000 7f1399400640 1 -- 172.21.15.31:0/2360091800 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c44d47c1e0 msgr2=0x55c44d49c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:02.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.416+0000 7f1399400640 1 --2- 172.21.15.31:0/2360091800 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c44d47c1e0 0x55c44d49c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.416+0000 7f1399400640 1 -- 172.21.15.31:0/2360091800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c44d4858d0 msgr2=0x55c44d5017e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:02.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.416+0000 7f1399400640 1 --2- 172.21.15.31:0/2360091800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c44d4858d0 0x55c44d5017e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.416+0000 7f1399400640 1 -- 172.21.15.31:0/2360091800 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c44d501d20 con 0x55c44d483510 2024-03-21T10:41:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.417+0000 7f1399400640 1 --2- 172.21.15.31:0/2360091800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c44d483510 0x55c44d49c760 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x55c44d483ec0 tx=0x7f13880104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e53131da6dde2d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.417+0000 7f1377400640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:41:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.418+0000 7f1377400640 1 -- 172.21.15.31:0/2360091800 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1388005020 con 0x55c44d483510 2024-03-21T10:41:02.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.419+0000 7f1377400640 1 -- 172.21.15.31:0/2360091800 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f13880051c0 con 0x55c44d483510 2024-03-21T10:41:02.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.419+0000 7f139f466740 1 -- 172.21.15.31:0/2360091800 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1390003e50 con 0x55c44d483510 2024-03-21T10:41:02.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.420+0000 7f1377400640 1 -- 172.21.15.31:0/2360091800 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1388033050 con 0x55c44d483510 2024-03-21T10:41:02.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.420+0000 7f139f466740 10 client.16047 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:02.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.420+0000 7f139f466740 1 -- 172.21.15.31:0/2360091800 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55c44d503000 con 0x55c44d483510 2024-03-21T10:41:02.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.420+0000 7f1377400640 1 -- 172.21.15.31:0/2360091800 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1390003e50 con 0x55c44d483510 2024-03-21T10:41:02.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.421+0000 7f1377400640 1 -- 172.21.15.31:0/2360091800 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f1388036020 con 0x55c44d483510 2024-03-21T10:41:02.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.422+0000 7f139f466740 10 client.16047 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:02.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.422+0000 7f139f466740 10 client.16047 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:41:02.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.422+0000 7f139f466740 20 client.16047 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:02.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.422+0000 7f139f466740 1 --2- 172.21.15.31:0/2360091800 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c44d5036c0 0x55c44d505ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:02.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.422+0000 7f139f466740 4 client.16047 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:41:02.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.422+0000 7f139f466740 1 -- 172.21.15.31:0/2360091800 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c44d3e80c0 con 0x55c44d5036c0 2024-03-21T10:41:02.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.422+0000 7f1399e00640 1 --2- 172.21.15.31:0/2360091800 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c44d5036c0 0x55c44d505ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:02.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.423+0000 7f1377400640 10 client.16047 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:02.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.423+0000 7f1399e00640 1 --2- 172.21.15.31:0/2360091800 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c44d5036c0 0x55c44d505ab0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=519159c274f3d3fd server_cookie=2eb855ffd54b8f39 in_seq=0 out_seq=0 2024-03-21T10:41:02.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.441+0000 7f1377400640 1 -- 172.21.15.31:0/2360091800 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c44d3e80c0 con 0x55c44d5036c0 2024-03-21T10:41:02.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.441+0000 7f1377400640 10 client.16047 handle_command_reply: tid=0 2024-03-21T10:41:02.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.442+0000 7f139f466740 1 client.16047 shutdown 2024-03-21T10:41:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.442+0000 7f139f466740 1 -- 172.21.15.31:0/2360091800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c44d483510 msgr2=0x55c44d49c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.442+0000 7f139f466740 1 --2- 172.21.15.31:0/2360091800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c44d483510 0x55c44d49c760 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x55c44d483ec0 tx=0x7f13880104a0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.443+0000 7f139f466740 1 -- 172.21.15.31:0/2360091800 shutdown_connections 2024-03-21T10:41:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.443+0000 7f139f466740 1 --2- 172.21.15.31:0/2360091800 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c44d5036c0 0x55c44d505ab0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.443+0000 7f139f466740 1 --2- 172.21.15.31:0/2360091800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c44d4858d0 0x55c44d5017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.443+0000 7f1377400640 0 client.16047 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.443+0000 7f139f466740 1 --2- 172.21.15.31:0/2360091800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c44d483510 0x55c44d49c760 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.443+0000 7f139f466740 1 --2- 172.21.15.31:0/2360091800 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c44d47c1e0 0x55c44d49c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.443+0000 7f139f466740 1 -- 172.21.15.31:0/2360091800 >> 172.21.15.31:0/2360091800 conn(0x55c44d476b00 msgr2=0x55c44d4752e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.444+0000 7f139f466740 1 -- 172.21.15.31:0/2360091800 shutdown_connections 2024-03-21T10:41:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.444+0000 7f139f466740 1 -- 172.21.15.31:0/2360091800 wait complete. 2024-03-21T10:41:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.444+0000 7f139f466740 20 client.16047 trim_cache size 0 max 16384 2024-03-21T10:41:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:41:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:41:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:41:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:41:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.446+0000 7f139b200640 1 -- 172.21.15.31:0/1842572978 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f137c0a3170 msgr2=0x7f137c0a5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.446+0000 7f139b200640 1 --2- 172.21.15.31:0/1842572978 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f137c0a3170 0x7f137c0a5630 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1390005ee0 tx=0x7f13900092a0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.446+0000 7f139b200640 1 -- 172.21.15.31:0/1842572978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1394107aa0 msgr2=0x7f1394081c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.446+0000 7f139b200640 1 --2- 172.21.15.31:0/1842572978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1394107aa0 0x7f1394081c40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f138400e790 tx=0x7f138400ec50 comp rx=0 tx=0).stop 2024-03-21T10:41:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.446+0000 7f139b200640 1 -- 172.21.15.31:0/1842572978 shutdown_connections 2024-03-21T10:41:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.446+0000 7f139b200640 1 --2- 172.21.15.31:0/1842572978 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f137c0a3170 0x7f137c0a5630 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.446+0000 7f139b200640 1 --2- 172.21.15.31:0/1842572978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f139410dab0 0x7f139407aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.446+0000 7f139b200640 1 --2- 172.21.15.31:0/1842572978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1394108460 0x7f1394082180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.446+0000 7f139b200640 1 --2- 172.21.15.31:0/1842572978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1394107aa0 0x7f1394081c40 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.446+0000 7f139b200640 1 -- 172.21.15.31:0/1842572978 >> 172.21.15.31:0/1842572978 conn(0x7f1394077220 msgr2=0x7f1394077980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.447+0000 7f139b200640 1 -- 172.21.15.31:0/1842572978 shutdown_connections 2024-03-21T10:41:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:02.447+0000 7f139b200640 1 -- 172.21.15.31:0/1842572978 wait complete. 2024-03-21T10:41:03.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:03 smithi040 ceph-mon[36154]: pgmap v77: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 196 MiB/s wr, 552 op/s 2024-03-21T10:41:03.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:03 smithi031 ceph-mon[31673]: pgmap v77: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 196 MiB/s wr, 552 op/s 2024-03-21T10:41:03.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:03 smithi070 ceph-mon[36330]: pgmap v77: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 196 MiB/s wr, 552 op/s 2024-03-21T10:41:05.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:05 smithi031 ceph-mon[31673]: pgmap v78: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 196 MiB/s wr, 552 op/s 2024-03-21T10:41:05.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:05 smithi070 ceph-mon[36330]: pgmap v78: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 196 MiB/s wr, 552 op/s 2024-03-21T10:41:05.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:05 smithi040 ceph-mon[36154]: pgmap v78: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 196 MiB/s wr, 552 op/s 2024-03-21T10:41:07.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:07 smithi031 ceph-mon[31673]: pgmap v79: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 285 MiB/s wr, 1.06k op/s 2024-03-21T10:41:07.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:07 smithi070 ceph-mon[36330]: pgmap v79: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 285 MiB/s wr, 1.06k op/s 2024-03-21T10:41:07.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:07 smithi040 ceph-mon[36154]: pgmap v79: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 285 MiB/s wr, 1.06k op/s 2024-03-21T10:41:08.279 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:41:08.280 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:41:08.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:08 smithi040 ceph-mon[36154]: pgmap v80: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 182 MiB/s wr, 967 op/s 2024-03-21T10:41:08.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.546+0000 7fb21ac00640 1 -- 172.21.15.31:0/3308482169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb214102f50 msgr2=0x7fb21410b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:08.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.546+0000 7fb21ac00640 1 --2- 172.21.15.31:0/3308482169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb214102f50 0x7fb21410b430 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb200009860 tx=0x7fb20002bc30 comp rx=0 tx=0).stop 2024-03-21T10:41:08.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.547+0000 7fb21ac00640 1 -- 172.21.15.31:0/3308482169 shutdown_connections 2024-03-21T10:41:08.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.547+0000 7fb21ac00640 1 --2- 172.21.15.31:0/3308482169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb21410ba50 0x7fb21410ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.547+0000 7fb21ac00640 1 --2- 172.21.15.31:0/3308482169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb214102f50 0x7fb21410b430 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.547+0000 7fb21ac00640 1 --2- 172.21.15.31:0/3308482169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb214102630 0x7fb214102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.547+0000 7fb21ac00640 1 -- 172.21.15.31:0/3308482169 >> 172.21.15.31:0/3308482169 conn(0x7fb214077130 msgr2=0x7fb2140fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:08.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.547+0000 7fb21ac00640 1 -- 172.21.15.31:0/3308482169 shutdown_connections 2024-03-21T10:41:08.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.547+0000 7fb21ac00640 1 -- 172.21.15.31:0/3308482169 wait complete. 2024-03-21T10:41:08.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.548+0000 7fb21ac00640 1 Processor -- start 2024-03-21T10:41:08.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.548+0000 7fb21ac00640 1 -- start start 2024-03-21T10:41:08.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.549+0000 7fb21ac00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb214102630 0x7fb214199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:08.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.549+0000 7fb21ac00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb214102f50 0x7fb21419a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:08.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.549+0000 7fb21ac00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb21410ba50 0x7fb21419e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:08.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.549+0000 7fb21ac00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb214113a90 con 0x7fb214102f50 2024-03-21T10:41:08.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.550+0000 7fb21a200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb21410ba50 0x7fb21419e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:08.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.550+0000 7fb21a200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb21410ba50 0x7fb21419e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60436/0 (socket says 172.21.15.31:60436) 2024-03-21T10:41:08.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.550+0000 7fb21a200640 1 -- 172.21.15.31:0/3758387240 learned_addr learned my addr 172.21.15.31:0/3758387240 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:08.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.550+0000 7fb21ac00640 1 -- 172.21.15.31:0/3758387240 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb214113910 con 0x7fb21410ba50 2024-03-21T10:41:08.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.550+0000 7fb21ac00640 1 -- 172.21.15.31:0/3758387240 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb214113c10 con 0x7fb214102630 2024-03-21T10:41:08.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.550+0000 7fb218e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb214102f50 0x7fb21419a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:08.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.549+0000 7fb219800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb214102630 0x7fb214199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:08.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.550+0000 7fb21a200640 1 -- 172.21.15.31:0/3758387240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb214102630 msgr2=0x7fb214199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:08.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.550+0000 7fb21a200640 1 --2- 172.21.15.31:0/3758387240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb214102630 0x7fb214199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.550+0000 7fb21a200640 1 -- 172.21.15.31:0/3758387240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb214102f50 msgr2=0x7fb21419a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:08.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.550+0000 7fb21a200640 1 --2- 172.21.15.31:0/3758387240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb214102f50 0x7fb21419a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.551+0000 7fb21a200640 1 -- 172.21.15.31:0/3758387240 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb21419ef10 con 0x7fb21410ba50 2024-03-21T10:41:08.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.551+0000 7fb218e00640 1 --2- 172.21.15.31:0/3758387240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb214102f50 0x7fb21419a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:41:08.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.551+0000 7fb21a200640 1 --2- 172.21.15.31:0/3758387240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb21410ba50 0x7fb21419e810 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb20c00ead0 tx=0x7fb20c00ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:08.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.551+0000 7fb212000640 1 -- 172.21.15.31:0/3758387240 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20c00cde0 con 0x7fb21410ba50 2024-03-21T10:41:08.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.551+0000 7fb212000640 1 -- 172.21.15.31:0/3758387240 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb20c005020 con 0x7fb21410ba50 2024-03-21T10:41:08.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.552+0000 7fb212000640 1 -- 172.21.15.31:0/3758387240 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20c0116d0 con 0x7fb21410ba50 2024-03-21T10:41:08.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.552+0000 7fb21ac00640 1 -- 172.21.15.31:0/3758387240 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb21419f200 con 0x7fb21410ba50 2024-03-21T10:41:08.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.552+0000 7fb21ac00640 1 -- 172.21.15.31:0/3758387240 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb21406b0c0 con 0x7fb21410ba50 2024-03-21T10:41:08.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.556+0000 7fb21efc1740 1 Processor -- start 2024-03-21T10:41:08.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:08 smithi031 ceph-mon[31673]: pgmap v80: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 182 MiB/s wr, 967 op/s 2024-03-21T10:41:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.561+0000 7fb212000640 1 -- 172.21.15.31:0/3758387240 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fb20c0051d0 con 0x7fb21410ba50 2024-03-21T10:41:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.561+0000 7fb21efc1740 1 -- start start 2024-03-21T10:41:08.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.561+0000 7fb21efc1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611eb4ac5e0 0x5611eb4ab330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:08.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.561+0000 7fb21efc1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5611eb4ab900 0x5611eb4abd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:08.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.561+0000 7fb21efc1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5611eb4b4c70 0x5611eb4b7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:08.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.561+0000 7fb21efc1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611eb407250 con 0x5611eb4b4c70 2024-03-21T10:41:08.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.561+0000 7fb21efc1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611eb3afa30 con 0x5611eb4ac5e0 2024-03-21T10:41:08.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:08 smithi070 ceph-mon[36330]: pgmap v80: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 182 MiB/s wr, 967 op/s 2024-03-21T10:41:08.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.561+0000 7fb21efc1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611eb4032f0 con 0x5611eb4ab900 2024-03-21T10:41:08.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.563+0000 7fb218e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611eb4ac5e0 0x5611eb4ab330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:08.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.563+0000 7fb218e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611eb4ac5e0 0x5611eb4ab330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60446/0 (socket says 172.21.15.31:60446) 2024-03-21T10:41:08.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.563+0000 7fb218e00640 1 -- 172.21.15.31:0/3750952962 learned_addr learned my addr 172.21.15.31:0/3750952962 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:08.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.563+0000 7fb212000640 1 --2- 172.21.15.31:0/3758387240 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb1fc0a3110 0x7fb1fc0a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:08.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.564+0000 7fb218e00640 1 --2- 172.21.15.31:0/3750952962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5611eb4b4c70 0x5611eb4b7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:08.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.564+0000 7fb212000640 1 -- 172.21.15.31:0/3758387240 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fb20c0cfa40 con 0x7fb21410ba50 2024-03-21T10:41:08.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.564+0000 7fb219800640 1 --2- 172.21.15.31:0/3750952962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5611eb4ab900 0x5611eb4abd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:08.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.565+0000 7fb218e00640 1 -- 172.21.15.31:0/3750952962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5611eb4ab900 msgr2=0x5611eb4abd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:08.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.565+0000 7fb218e00640 1 --2- 172.21.15.31:0/3750952962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5611eb4ab900 0x5611eb4abd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.565+0000 7fb218e00640 1 -- 172.21.15.31:0/3750952962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5611eb4b4c70 msgr2=0x5611eb4b7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:08.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.565+0000 7fb218e00640 1 --2- 172.21.15.31:0/3750952962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5611eb4b4c70 0x5611eb4b7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.565+0000 7fb218e00640 1 -- 172.21.15.31:0/3750952962 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611eb39a690 con 0x5611eb4ac5e0 2024-03-21T10:41:08.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.565+0000 7fb21a200640 1 --2- 172.21.15.31:0/3758387240 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb1fc0a3110 0x7fb1fc0a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:08.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.567+0000 7fb21a200640 1 --2- 172.21.15.31:0/3758387240 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb1fc0a3110 0x7fb1fc0a55d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb20c098460 tx=0x7fb20c0cfc90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:08.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.572+0000 7fb218e00640 1 --2- 172.21.15.31:0/3750952962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611eb4ac5e0 0x5611eb4ab330 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb2000057c0 tx=0x7fb200005710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f99167680bf5ee31 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:08.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.575+0000 7fb206a00640 1 -- 172.21.15.31:0/3750952962 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb200039070 con 0x5611eb4ac5e0 2024-03-21T10:41:08.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.575+0000 7fb206a00640 1 -- 172.21.15.31:0/3750952962 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb20003a040 con 0x5611eb4ac5e0 2024-03-21T10:41:08.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.575+0000 7fb206a00640 1 -- 172.21.15.31:0/3750952962 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb200005020 con 0x5611eb4ac5e0 2024-03-21T10:41:08.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.583+0000 7fb21efc1740 1 -- 172.21.15.31:0/3750952962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611eb4ac5e0 msgr2=0x5611eb4ab330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:08.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.583+0000 7fb21efc1740 1 --2- 172.21.15.31:0/3750952962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611eb4ac5e0 0x5611eb4ab330 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb2000057c0 tx=0x7fb200005710 comp rx=0 tx=0).stop 2024-03-21T10:41:08.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.584+0000 7fb21efc1740 1 -- 172.21.15.31:0/3750952962 shutdown_connections 2024-03-21T10:41:08.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.585+0000 7fb21efc1740 1 --2- 172.21.15.31:0/3750952962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5611eb4b4c70 0x5611eb4b7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.585+0000 7fb21efc1740 1 --2- 172.21.15.31:0/3750952962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5611eb4ab900 0x5611eb4abd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.585+0000 7fb21efc1740 1 --2- 172.21.15.31:0/3750952962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611eb4ac5e0 0x5611eb4ab330 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.585+0000 7fb21efc1740 1 -- 172.21.15.31:0/3750952962 >> 172.21.15.31:0/3750952962 conn(0x5611eb46a200 msgr2=0x5611eb4a6350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:08.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.589+0000 7fb21efc1740 1 -- 172.21.15.31:0/3750952962 shutdown_connections 2024-03-21T10:41:08.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.590+0000 7fb21efc1740 1 -- 172.21.15.31:0/3750952962 wait complete. 2024-03-21T10:41:08.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.591+0000 7fb21efc1740 1 -- start start 2024-03-21T10:41:08.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.591+0000 7fb21efc1740 1 Processor -- start 2024-03-21T10:41:08.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.596+0000 7fb21efc1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5611eb4ab900 0x5611eb4b4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:08.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.596+0000 7fb21efc1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611eb4ac5e0 0x5611eb4b18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:08.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.597+0000 7fb218e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5611eb4ab900 0x5611eb4b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:08.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.597+0000 7fb218e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5611eb4ab900 0x5611eb4b4540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:52710/0 (socket says 172.21.15.31:52710) 2024-03-21T10:41:08.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.597+0000 7fb21efc1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5611eb4b4c70 0x5611eb4b1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:08.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.597+0000 7fb21efc1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611eb49d5b0 con 0x5611eb4b4c70 2024-03-21T10:41:08.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.597+0000 7fb21efc1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611eb3c2e80 con 0x5611eb4ac5e0 2024-03-21T10:41:08.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.597+0000 7fb21efc1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611eb49d730 con 0x5611eb4ab900 2024-03-21T10:41:08.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.597+0000 7fb21efc1740 10 client.0 _ll_register_callbacks cb 0x5611eb3d5130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:08.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.597+0000 7fb218e00640 1 -- 172.21.15.31:0/790280298 learned_addr learned my addr 172.21.15.31:0/790280298 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:08.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.598+0000 7fb219800640 1 --2- 172.21.15.31:0/790280298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611eb4ac5e0 0x5611eb4b18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:08.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.598+0000 7fb218e00640 1 --2- 172.21.15.31:0/790280298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5611eb4b4c70 0x5611eb4b1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:08.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.598+0000 7fb218e00640 1 -- 172.21.15.31:0/790280298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611eb4ac5e0 msgr2=0x5611eb4b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:08.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.598+0000 7fb218e00640 1 --2- 172.21.15.31:0/790280298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611eb4ac5e0 0x5611eb4b18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.598+0000 7fb218e00640 1 -- 172.21.15.31:0/790280298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5611eb4b4c70 msgr2=0x5611eb4b1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:08.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.598+0000 7fb218e00640 1 --2- 172.21.15.31:0/790280298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5611eb4b4c70 0x5611eb4b1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.598+0000 7fb218e00640 1 -- 172.21.15.31:0/790280298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5611eb1e57d0 con 0x5611eb4ab900 2024-03-21T10:41:08.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.599+0000 7fb218e00640 1 --2- 172.21.15.31:0/790280298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5611eb4ab900 0x5611eb4b4540 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb200009450 tx=0x7fb20002bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68e2388b94556ea7 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:08.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.599+0000 7fb1fb400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:41:08.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.600+0000 7fb1fb400640 1 -- 172.21.15.31:0/790280298 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb200039070 con 0x5611eb4ab900 2024-03-21T10:41:08.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.600+0000 7fb1fb400640 1 -- 172.21.15.31:0/790280298 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fb200004120 con 0x5611eb4ab900 2024-03-21T10:41:08.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.600+0000 7fb21efc1740 1 -- 172.21.15.31:0/790280298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb20003a040 con 0x5611eb4ab900 2024-03-21T10:41:08.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.600+0000 7fb1fb400640 1 -- 172.21.15.31:0/790280298 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb200004320 con 0x5611eb4ab900 2024-03-21T10:41:08.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.601+0000 7fb21efc1740 10 client.26245 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:08.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.601+0000 7fb21efc1740 1 -- 172.21.15.31:0/790280298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5611eb11b3f0 con 0x5611eb4ab900 2024-03-21T10:41:08.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.601+0000 7fb1fb400640 1 -- 172.21.15.31:0/790280298 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb20003a040 con 0x5611eb4ab900 2024-03-21T10:41:08.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.601+0000 7fb1fb400640 1 -- 172.21.15.31:0/790280298 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fb20003f030 con 0x5611eb4ab900 2024-03-21T10:41:08.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.602+0000 7fb21efc1740 10 client.26245 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:08.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.602+0000 7fb21efc1740 10 client.26245 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:41:08.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.602+0000 7fb21efc1740 20 client.26245 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:08.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.602+0000 7fb21efc1740 1 --2- 172.21.15.31:0/790280298 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5611eb4bc290 0x5611eb4be680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:08.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.602+0000 7fb21efc1740 4 client.26245 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:41:08.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.602+0000 7fb21efc1740 1 -- 172.21.15.31:0/790280298 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5611eb414790 con 0x5611eb4bc290 2024-03-21T10:41:08.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.603+0000 7fb219800640 1 --2- 172.21.15.31:0/790280298 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5611eb4bc290 0x5611eb4be680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:08.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.604+0000 7fb219800640 1 --2- 172.21.15.31:0/790280298 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5611eb4bc290 0x5611eb4be680 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40e9ceb1685b9b58 server_cookie=d22b76b7fe53e142 in_seq=0 out_seq=0 2024-03-21T10:41:08.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.604+0000 7fb1fb400640 10 client.26245 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:08.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.605+0000 7fb1fb400640 1 -- 172.21.15.31:0/790280298 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5611eb414790 con 0x5611eb4bc290 2024-03-21T10:41:08.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.605+0000 7fb1fb400640 10 client.26245 handle_command_reply: tid=0 2024-03-21T10:41:08.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.605+0000 7fb21efc1740 1 client.26245 shutdown 2024-03-21T10:41:08.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.606+0000 7fb21efc1740 1 -- 172.21.15.31:0/790280298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5611eb4ab900 msgr2=0x5611eb4b4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:08.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.606+0000 7fb21efc1740 1 --2- 172.21.15.31:0/790280298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5611eb4ab900 0x5611eb4b4540 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb200009450 tx=0x7fb20002bc30 comp rx=0 tx=0).stop 2024-03-21T10:41:08.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.606+0000 7fb21efc1740 1 -- 172.21.15.31:0/790280298 shutdown_connections 2024-03-21T10:41:08.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.606+0000 7fb21efc1740 1 --2- 172.21.15.31:0/790280298 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5611eb4bc290 0x5611eb4be680 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.607+0000 7fb21efc1740 1 --2- 172.21.15.31:0/790280298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5611eb4b4c70 0x5611eb4b1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.607+0000 7fb21efc1740 1 --2- 172.21.15.31:0/790280298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611eb4ac5e0 0x5611eb4b18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.607+0000 7fb21efc1740 1 --2- 172.21.15.31:0/790280298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5611eb4ab900 0x5611eb4b4540 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.607+0000 7fb21efc1740 1 -- 172.21.15.31:0/790280298 >> 172.21.15.31:0/790280298 conn(0x5611eb46a200 msgr2=0x5611eb4a80a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:08.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.607+0000 7fb1fb400640 0 client.26245 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:08.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.607+0000 7fb21efc1740 1 -- 172.21.15.31:0/790280298 shutdown_connections 2024-03-21T10:41:08.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.607+0000 7fb21efc1740 1 -- 172.21.15.31:0/790280298 wait complete. 2024-03-21T10:41:08.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.608+0000 7fb21efc1740 20 client.26245 trim_cache size 0 max 16384 2024-03-21T10:41:08.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.643+0000 7fb21efc1740 1 Processor -- start 2024-03-21T10:41:08.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.643+0000 7fb21efc1740 1 -- start start 2024-03-21T10:41:08.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.644+0000 7fb21efc1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5611eb4ab900 0x5611eb49ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:08.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.644+0000 7fb21efc1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5611eb4ac5e0 0x5611eb4b4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:08.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.644+0000 7fb21efc1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611eb4b4c70 0x5611eb4b4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:08.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.644+0000 7fb21efc1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb214073130 con 0x5611eb4ac5e0 2024-03-21T10:41:08.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.644+0000 7fb21efc1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb214070c30 con 0x5611eb4b4c70 2024-03-21T10:41:08.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.644+0000 7fb21efc1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21406ecd0 con 0x5611eb4ab900 2024-03-21T10:41:08.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.644+0000 7fb218e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5611eb4ab900 0x5611eb49ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:08.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.644+0000 7fb218e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5611eb4ab900 0x5611eb49ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:52714/0 (socket says 172.21.15.31:52714) 2024-03-21T10:41:08.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.644+0000 7fb218e00640 1 -- 172.21.15.31:0/3906387583 learned_addr learned my addr 172.21.15.31:0/3906387583 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:08.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.645+0000 7fb219800640 1 --2- 172.21.15.31:0/3906387583 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5611eb4ac5e0 0x5611eb4b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:08.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.645+0000 7fb218e00640 1 --2- 172.21.15.31:0/3906387583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611eb4b4c70 0x5611eb4b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:08.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.646+0000 7fb218e00640 1 -- 172.21.15.31:0/3906387583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611eb4b4c70 msgr2=0x5611eb4b4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:08.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.646+0000 7fb218e00640 1 --2- 172.21.15.31:0/3906387583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611eb4b4c70 0x5611eb4b4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.646+0000 7fb218e00640 1 -- 172.21.15.31:0/3906387583 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5611eb4ac5e0 msgr2=0x5611eb4b4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:08.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.646+0000 7fb218e00640 1 --2- 172.21.15.31:0/3906387583 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5611eb4ac5e0 0x5611eb4b4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.646+0000 7fb218e00640 1 -- 172.21.15.31:0/3906387583 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611eb4b21a0 con 0x5611eb4ab900 2024-03-21T10:41:08.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.647+0000 7fb218e00640 1 --2- 172.21.15.31:0/3906387583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5611eb4ab900 0x5611eb49ca40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb2000347b0 tx=0x7fb200000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=726e840aa1854828 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:08.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.647+0000 7fb206000640 1 -- 172.21.15.31:0/3906387583 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb200039040 con 0x5611eb4ab900 2024-03-21T10:41:08.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.647+0000 7fb206000640 1 -- 172.21.15.31:0/3906387583 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb200081040 con 0x5611eb4ab900 2024-03-21T10:41:08.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.647+0000 7fb206000640 1 -- 172.21.15.31:0/3906387583 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb200039040 con 0x5611eb4ab900 2024-03-21T10:41:08.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.648+0000 7fb21efc1740 1 -- 172.21.15.31:0/3906387583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5611eb4ab900 msgr2=0x5611eb49ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:08.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.648+0000 7fb21efc1740 1 --2- 172.21.15.31:0/3906387583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5611eb4ab900 0x5611eb49ca40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb2000347b0 tx=0x7fb200000bc0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.648+0000 7fb21efc1740 1 -- 172.21.15.31:0/3906387583 shutdown_connections 2024-03-21T10:41:08.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.648+0000 7fb21efc1740 1 --2- 172.21.15.31:0/3906387583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611eb4b4c70 0x5611eb4b4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.648+0000 7fb21efc1740 1 --2- 172.21.15.31:0/3906387583 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5611eb4ac5e0 0x5611eb4b4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.648+0000 7fb21efc1740 1 --2- 172.21.15.31:0/3906387583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5611eb4ab900 0x5611eb49ca40 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.648+0000 7fb21efc1740 1 -- 172.21.15.31:0/3906387583 >> 172.21.15.31:0/3906387583 conn(0x5611eb46a200 msgr2=0x5611eb49c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:08.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.649+0000 7fb21efc1740 1 -- 172.21.15.31:0/3906387583 shutdown_connections 2024-03-21T10:41:08.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.649+0000 7fb21efc1740 1 -- 172.21.15.31:0/3906387583 wait complete. 2024-03-21T10:41:08.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.649+0000 7fb21efc1740 1 -- start start 2024-03-21T10:41:08.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.650+0000 7fb21efc1740 1 Processor -- start 2024-03-21T10:41:08.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.650+0000 7fb21efc1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5611eb4ab900 0x5611eb4a0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:08.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.650+0000 7fb21efc1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611eb4ac5e0 0x5611eb4a07a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:08.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.650+0000 7fb21efc1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5611eb4b4c70 0x5611eb5107f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:08.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.650+0000 7fb21efc1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21406f2d0 con 0x5611eb4b4c70 2024-03-21T10:41:08.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.650+0000 7fb21efc1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2140705d0 con 0x5611eb4ac5e0 2024-03-21T10:41:08.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.650+0000 7fb21efc1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21406f910 con 0x5611eb4ab900 2024-03-21T10:41:08.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.650+0000 7fb21efc1740 10 client.0 _ll_register_callbacks cb 0x7fb208005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:08.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.651+0000 7fb219800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611eb4ac5e0 0x5611eb4a07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:08.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.651+0000 7fb219800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611eb4ac5e0 0x5611eb4a07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60476/0 (socket says 172.21.15.31:60476) 2024-03-21T10:41:08.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.651+0000 7fb219800640 1 -- 172.21.15.31:0/2061012137 learned_addr learned my addr 172.21.15.31:0/2061012137 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:08.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.652+0000 7fb218e00640 1 --2- 172.21.15.31:0/2061012137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5611eb4b4c70 0x5611eb5107f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:08.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.652+0000 7fb219800640 1 -- 172.21.15.31:0/2061012137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5611eb4ab900 msgr2=0x5611eb4a0260 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:41:08.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.652+0000 7fb219800640 1 --2- 172.21.15.31:0/2061012137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5611eb4ab900 0x5611eb4a0260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.652+0000 7fb219800640 1 -- 172.21.15.31:0/2061012137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5611eb4b4c70 msgr2=0x5611eb5107f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:08.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.652+0000 7fb219800640 1 --2- 172.21.15.31:0/2061012137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5611eb4b4c70 0x5611eb5107f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.653+0000 7fb219800640 1 -- 172.21.15.31:0/2061012137 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5611eb0db0f0 con 0x5611eb4ac5e0 2024-03-21T10:41:08.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.653+0000 7fb218e00640 1 --2- 172.21.15.31:0/2061012137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5611eb4b4c70 0x5611eb5107f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:41:08.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.653+0000 7fb219800640 1 --2- 172.21.15.31:0/2061012137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611eb4ac5e0 0x5611eb4a07a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x5611eb4b1130 tx=0x7fb208012820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c25608c6337854a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:08.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.653+0000 7fb1fb400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:41:08.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.654+0000 7fb1fb400640 1 -- 172.21.15.31:0/2061012137 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb208004220 con 0x5611eb4ac5e0 2024-03-21T10:41:08.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.654+0000 7fb1fb400640 1 -- 172.21.15.31:0/2061012137 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fb2080043c0 con 0x5611eb4ac5e0 2024-03-21T10:41:08.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.655+0000 7fb21efc1740 1 -- 172.21.15.31:0/2061012137 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb200081040 con 0x5611eb4ac5e0 2024-03-21T10:41:08.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.655+0000 7fb1fb400640 1 -- 172.21.15.31:0/2061012137 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb20801e890 con 0x5611eb4ac5e0 2024-03-21T10:41:08.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.655+0000 7fb21efc1740 10 client.26294 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:08.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.655+0000 7fb21efc1740 1 -- 172.21.15.31:0/2061012137 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5611eb5180a0 con 0x5611eb4ac5e0 2024-03-21T10:41:08.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.656+0000 7fb1fb400640 1 -- 172.21.15.31:0/2061012137 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb200081040 con 0x5611eb4ac5e0 2024-03-21T10:41:08.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.657+0000 7fb1fb400640 1 -- 172.21.15.31:0/2061012137 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fb208003cb0 con 0x5611eb4ac5e0 2024-03-21T10:41:08.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.657+0000 7fb21efc1740 10 client.26294 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:08.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.657+0000 7fb21efc1740 10 client.26294 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:41:08.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.657+0000 7fb21efc1740 20 client.26294 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:08.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.657+0000 7fb21efc1740 1 --2- 172.21.15.31:0/2061012137 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5611eb518730 0x5611eb51ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:08.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.658+0000 7fb21efc1740 4 client.26294 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:41:08.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.658+0000 7fb21efc1740 1 -- 172.21.15.31:0/2061012137 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5611eb40eae0 con 0x5611eb518730 2024-03-21T10:41:08.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.658+0000 7fb218e00640 1 --2- 172.21.15.31:0/2061012137 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5611eb518730 0x5611eb51ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:08.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.659+0000 7fb218e00640 1 --2- 172.21.15.31:0/2061012137 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5611eb518730 0x5611eb51ab50 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c28c295ff5b5fe5 server_cookie=ba87d130eb5157b0 in_seq=0 out_seq=0 2024-03-21T10:41:08.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.659+0000 7fb1fb400640 10 client.26294 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:08.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.659+0000 7fb1fb400640 1 -- 172.21.15.31:0/2061012137 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5611eb40eae0 con 0x5611eb518730 2024-03-21T10:41:08.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.659+0000 7fb1fb400640 10 client.26294 handle_command_reply: tid=0 2024-03-21T10:41:08.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.659+0000 7fb21efc1740 1 client.26294 shutdown 2024-03-21T10:41:08.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.660+0000 7fb21efc1740 1 -- 172.21.15.31:0/2061012137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611eb4ac5e0 msgr2=0x5611eb4a07a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:08.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.660+0000 7fb21efc1740 1 --2- 172.21.15.31:0/2061012137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611eb4ac5e0 0x5611eb4a07a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x5611eb4b1130 tx=0x7fb208012820 comp rx=0 tx=0).stop 2024-03-21T10:41:08.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.660+0000 7fb21efc1740 1 -- 172.21.15.31:0/2061012137 shutdown_connections 2024-03-21T10:41:08.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.660+0000 7fb21efc1740 1 --2- 172.21.15.31:0/2061012137 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5611eb518730 0x5611eb51ab50 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.660+0000 7fb21efc1740 1 --2- 172.21.15.31:0/2061012137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5611eb4b4c70 0x5611eb5107f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.660+0000 7fb21efc1740 1 --2- 172.21.15.31:0/2061012137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611eb4ac5e0 0x5611eb4a07a0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.660+0000 7fb21efc1740 1 --2- 172.21.15.31:0/2061012137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5611eb4ab900 0x5611eb4a0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.660+0000 7fb21efc1740 1 -- 172.21.15.31:0/2061012137 >> 172.21.15.31:0/2061012137 conn(0x5611eb46a200 msgr2=0x5611eb49cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:08.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.660+0000 7fb1fb400640 0 client.26294 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:08.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.661+0000 7fb21efc1740 1 -- 172.21.15.31:0/2061012137 shutdown_connections 2024-03-21T10:41:08.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.661+0000 7fb21efc1740 1 -- 172.21.15.31:0/2061012137 wait complete. 2024-03-21T10:41:08.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.661+0000 7fb21efc1740 20 client.26294 trim_cache size 0 max 16384 2024-03-21T10:41:08.660 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:41:08.660 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-03-21T10:41:08.660 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-03-21T10:41:08.660 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:41:08.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.666+0000 7fb21ac00640 1 -- 172.21.15.31:0/3758387240 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb1fc0a3110 msgr2=0x7fb1fc0a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:08.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.666+0000 7fb21ac00640 1 --2- 172.21.15.31:0/3758387240 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb1fc0a3110 0x7fb1fc0a55d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb20c098460 tx=0x7fb20c0cfc90 comp rx=0 tx=0).stop 2024-03-21T10:41:08.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.666+0000 7fb21ac00640 1 -- 172.21.15.31:0/3758387240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb21410ba50 msgr2=0x7fb21419e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:08.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.666+0000 7fb21ac00640 1 --2- 172.21.15.31:0/3758387240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb21410ba50 0x7fb21419e810 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb20c00ead0 tx=0x7fb20c00ef90 comp rx=0 tx=0).stop 2024-03-21T10:41:08.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.666+0000 7fb21ac00640 1 -- 172.21.15.31:0/3758387240 shutdown_connections 2024-03-21T10:41:08.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.666+0000 7fb21ac00640 1 --2- 172.21.15.31:0/3758387240 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb1fc0a3110 0x7fb1fc0a55d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.666+0000 7fb21ac00640 1 --2- 172.21.15.31:0/3758387240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb21410ba50 0x7fb21419e810 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.666+0000 7fb21ac00640 1 --2- 172.21.15.31:0/3758387240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb214102f50 0x7fb21419a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.666+0000 7fb21ac00640 1 --2- 172.21.15.31:0/3758387240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb214102630 0x7fb214199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:08.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.666+0000 7fb21ac00640 1 -- 172.21.15.31:0/3758387240 >> 172.21.15.31:0/3758387240 conn(0x7fb214077130 msgr2=0x7fb2140fc650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:08.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.667+0000 7fb21ac00640 1 -- 172.21.15.31:0/3758387240 shutdown_connections 2024-03-21T10:41:08.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:08.667+0000 7fb21ac00640 1 -- 172.21.15.31:0/3758387240 wait complete. 2024-03-21T10:41:08.698 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-21T10:41:08.698 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:41:09.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.128+0000 7f4730400640 1 -- 172.21.15.31:0/2693131275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4728106590 msgr2=0x7f47280698d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:09.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.128+0000 7f4730400640 1 --2- 172.21.15.31:0/2693131275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4728106590 0x7f47280698d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4714009900 tx=0x7f471402bc30 comp rx=0 tx=0).stop 2024-03-21T10:41:09.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.128+0000 7f4730400640 1 -- 172.21.15.31:0/2693131275 shutdown_connections 2024-03-21T10:41:09.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.128+0000 7f4730400640 1 --2- 172.21.15.31:0/2693131275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4728104090 0x7f4728104540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:09.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.128+0000 7f4730400640 1 --2- 172.21.15.31:0/2693131275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4728069e10 0x7f472806a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:09.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.128+0000 7f4730400640 1 --2- 172.21.15.31:0/2693131275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4728106590 0x7f47280698d0 secure :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4714009900 tx=0x7f471402bc30 comp rx=0 tx=0).stop 2024-03-21T10:41:09.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.128+0000 7f4730400640 1 -- 172.21.15.31:0/2693131275 >> 172.21.15.31:0/2693131275 conn(0x7f4728077690 msgr2=0x7f4728077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:09.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.129+0000 7f4730400640 1 -- 172.21.15.31:0/2693131275 shutdown_connections 2024-03-21T10:41:09.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.129+0000 7f4730400640 1 -- 172.21.15.31:0/2693131275 wait complete. 2024-03-21T10:41:09.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.130+0000 7f4730400640 1 Processor -- start 2024-03-21T10:41:09.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.130+0000 7f4730400640 1 -- start start 2024-03-21T10:41:09.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.131+0000 7f4730400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4728069e10 0x7f472819e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:09.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.131+0000 7f4730400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4728104090 0x7f472819ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:09.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.131+0000 7f4730400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47281a3010 0x7f47281a34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:09.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.131+0000 7f4730400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4728118080 con 0x7f47281a3010 2024-03-21T10:41:09.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.132+0000 7f4730400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4728117f00 con 0x7f4728104090 2024-03-21T10:41:09.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.132+0000 7f4730400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4728118200 con 0x7f4728069e10 2024-03-21T10:41:09.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.132+0000 7f472d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4728104090 0x7f472819ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:09.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.132+0000 7f472d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4728104090 0x7f472819ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60496/0 (socket says 172.21.15.31:60496) 2024-03-21T10:41:09.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.132+0000 7f472d000640 1 -- 172.21.15.31:0/1058880533 learned_addr learned my addr 172.21.15.31:0/1058880533 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:09.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.132+0000 7f472e400640 1 --2- 172.21.15.31:0/1058880533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47281a3010 0x7f47281a34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:09.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.133+0000 7f472da00640 1 --2- 172.21.15.31:0/1058880533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4728069e10 0x7f472819e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:09.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.133+0000 7f472d000640 1 -- 172.21.15.31:0/1058880533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4728069e10 msgr2=0x7f472819e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:09.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.133+0000 7f472d000640 1 --2- 172.21.15.31:0/1058880533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4728069e10 0x7f472819e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:09.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.133+0000 7f472d000640 1 -- 172.21.15.31:0/1058880533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47281a3010 msgr2=0x7f47281a34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:09.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.133+0000 7f472d000640 1 --2- 172.21.15.31:0/1058880533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47281a3010 0x7f47281a34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:09.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.133+0000 7f472d000640 1 -- 172.21.15.31:0/1058880533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47281a3b00 con 0x7f4728104090 2024-03-21T10:41:09.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.133+0000 7f472da00640 1 --2- 172.21.15.31:0/1058880533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4728069e10 0x7f472819e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:41:09.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.134+0000 7f472d000640 1 --2- 172.21.15.31:0/1058880533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4728104090 0x7f472819ec90 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f472000b9e0 tx=0x7f472000bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:09.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.134+0000 7f4726000640 1 -- 172.21.15.31:0/1058880533 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4720019070 con 0x7f4728104090 2024-03-21T10:41:09.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.134+0000 7f4726000640 1 -- 172.21.15.31:0/1058880533 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4720014440 con 0x7f4728104090 2024-03-21T10:41:09.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.134+0000 7f4730400640 1 -- 172.21.15.31:0/1058880533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47281a3df0 con 0x7f4728104090 2024-03-21T10:41:09.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.134+0000 7f4726000640 1 -- 172.21.15.31:0/1058880533 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4720007610 con 0x7f4728104090 2024-03-21T10:41:09.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.134+0000 7f4730400640 1 -- 172.21.15.31:0/1058880533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47281ab630 con 0x7f4728104090 2024-03-21T10:41:09.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.136+0000 7f4730400640 1 -- 172.21.15.31:0/1058880533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4728067c90 con 0x7f4728104090 2024-03-21T10:41:09.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.140+0000 7f4726000640 1 -- 172.21.15.31:0/1058880533 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f472001c070 con 0x7f4728104090 2024-03-21T10:41:09.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.143+0000 7f4726000640 1 --2- 172.21.15.31:0/1058880533 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f47100a3110 0x7f47100a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:09.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.143+0000 7f4726000640 1 -- 172.21.15.31:0/1058880533 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f47200cf830 con 0x7f4728104090 2024-03-21T10:41:09.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.143+0000 7f4726000640 1 -- 172.21.15.31:0/1058880533 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f47200fd7e0 con 0x7f4728104090 2024-03-21T10:41:09.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.144+0000 7f472da00640 1 --2- 172.21.15.31:0/1058880533 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f47100a3110 0x7f47100a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:09.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.146+0000 7f472da00640 1 --2- 172.21.15.31:0/1058880533 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f47100a3110 0x7f47100a55d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4714006010 tx=0x7f4714000bc0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:09.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.725+0000 7f4730400640 1 -- 172.21.15.31:0/1058880533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47281ab9e0 con 0x7f4728104090 2024-03-21T10:41:09.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.727+0000 7f4726000640 1 -- 172.21.15.31:0/1058880533 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f472009bff0 con 0x7f4728104090 2024-03-21T10:41:09.727 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:41:09.727 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:40:26.180798+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24596,24547]},"id":1}]} 2024-03-21T10:41:09.728 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 25 2024-03-21T10:41:09.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.730+0000 7f4730400640 1 -- 172.21.15.31:0/1058880533 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f47100a3110 msgr2=0x7f47100a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:09.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.730+0000 7f4730400640 1 --2- 172.21.15.31:0/1058880533 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f47100a3110 0x7f47100a55d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4714006010 tx=0x7f4714000bc0 comp rx=0 tx=0).stop 2024-03-21T10:41:09.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.730+0000 7f4730400640 1 -- 172.21.15.31:0/1058880533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4728104090 msgr2=0x7f472819ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:09.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.730+0000 7f4730400640 1 --2- 172.21.15.31:0/1058880533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4728104090 0x7f472819ec90 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f472000b9e0 tx=0x7f472000bea0 comp rx=0 tx=0).stop 2024-03-21T10:41:09.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.730+0000 7f4730400640 1 -- 172.21.15.31:0/1058880533 shutdown_connections 2024-03-21T10:41:09.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.730+0000 7f4730400640 1 --2- 172.21.15.31:0/1058880533 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f47100a3110 0x7f47100a55d0 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4714006010 tx=0x7f4714000bc0 comp rx=0 tx=0).stop 2024-03-21T10:41:09.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.730+0000 7f4730400640 1 --2- 172.21.15.31:0/1058880533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47281a3010 0x7f47281a34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:09.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.730+0000 7f4730400640 1 --2- 172.21.15.31:0/1058880533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4728104090 0x7f472819ec90 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:09.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.730+0000 7f4730400640 1 --2- 172.21.15.31:0/1058880533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4728069e10 0x7f472819e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:09.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.730+0000 7f4730400640 1 -- 172.21.15.31:0/1058880533 >> 172.21.15.31:0/1058880533 conn(0x7f4728077690 msgr2=0x7f47280fd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:09.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.730+0000 7f4730400640 1 -- 172.21.15.31:0/1058880533 shutdown_connections 2024-03-21T10:41:09.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:09.730+0000 7f4730400640 1 -- 172.21.15.31:0/1058880533 wait complete. 2024-03-21T10:41:09.752 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:41:09.753 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-21T10:41:10.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.132+0000 7fdc83800640 1 -- 172.21.15.31:0/3189475563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc7c100f30 msgr2=0x7fdc7c10d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:10.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.132+0000 7fdc83800640 1 --2- 172.21.15.31:0/3189475563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc7c100f30 0x7fdc7c10d3f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fdc74009c10 tx=0x7fdc7402bef0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.133+0000 7fdc83800640 1 -- 172.21.15.31:0/3189475563 shutdown_connections 2024-03-21T10:41:10.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.133+0000 7fdc83800640 1 --2- 172.21.15.31:0/3189475563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc7c100f30 0x7fdc7c10d3f0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.133+0000 7fdc83800640 1 --2- 172.21.15.31:0/3189475563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc7c100570 0x7fdc7c1009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.133+0000 7fdc83800640 1 --2- 172.21.15.31:0/3189475563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc7c106630 0x7fdc7c106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.133+0000 7fdc83800640 1 -- 172.21.15.31:0/3189475563 >> 172.21.15.31:0/3189475563 conn(0x7fdc7c0776f0 msgr2=0x7fdc7c077b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:10.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.138+0000 7fdc83800640 1 -- 172.21.15.31:0/3189475563 shutdown_connections 2024-03-21T10:41:10.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.138+0000 7fdc83800640 1 -- 172.21.15.31:0/3189475563 wait complete. 2024-03-21T10:41:10.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.139+0000 7fdc83800640 1 Processor -- start 2024-03-21T10:41:10.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.139+0000 7fdc83800640 1 -- start start 2024-03-21T10:41:10.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.140+0000 7fdc83800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc7c100570 0x7fdc7c19dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.140+0000 7fdc83800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc7c100f30 0x7fdc7c19e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.141+0000 7fdc83800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc7c106630 0x7fdc7c195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.141+0000 7fdc80e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc7c100570 0x7fdc7c19dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.141+0000 7fdc83800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c110960 con 0x7fdc7c106630 2024-03-21T10:41:10.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.141+0000 7fdc83800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c1107e0 con 0x7fdc7c100f30 2024-03-21T10:41:10.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.141+0000 7fdc83800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c110ae0 con 0x7fdc7c100570 2024-03-21T10:41:10.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.141+0000 7fdc80e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc7c100570 0x7fdc7c19dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:52752/0 (socket says 172.21.15.31:52752) 2024-03-21T10:41:10.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.141+0000 7fdc80e00640 1 -- 172.21.15.31:0/2318681402 learned_addr learned my addr 172.21.15.31:0/2318681402 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:10.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.141+0000 7fdc80e00640 1 -- 172.21.15.31:0/2318681402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc7c100f30 msgr2=0x7fdc7c19e010 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-21T10:41:10.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.141+0000 7fdc80e00640 1 --2- 172.21.15.31:0/2318681402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc7c100f30 0x7fdc7c19e010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.141+0000 7fdc80e00640 1 -- 172.21.15.31:0/2318681402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc7c106630 msgr2=0x7fdc7c195e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T10:41:10.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.144+0000 7fdc81800640 1 --2- 172.21.15.31:0/2318681402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc7c106630 0x7fdc7c195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.146+0000 7fdc80e00640 1 --2- 172.21.15.31:0/2318681402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc7c106630 0x7fdc7c195e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.147+0000 7fdc80e00640 1 -- 172.21.15.31:0/2318681402 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c196730 con 0x7fdc7c100570 2024-03-21T10:41:10.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.147+0000 7fdc81800640 1 --2- 172.21.15.31:0/2318681402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc7c106630 0x7fdc7c195e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:41:10.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.150+0000 7fdc80e00640 1 --2- 172.21.15.31:0/2318681402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc7c100570 0x7fdc7c19dad0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fdc7000e9c0 tx=0x7fdc7000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:10.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.152+0000 7fdc79600640 1 -- 172.21.15.31:0/2318681402 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7000ccc0 con 0x7fdc7c100570 2024-03-21T10:41:10.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.152+0000 7fdc79600640 1 -- 172.21.15.31:0/2318681402 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc70005500 con 0x7fdc7c100570 2024-03-21T10:41:10.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.152+0000 7fdc79600640 1 -- 172.21.15.31:0/2318681402 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc70010430 con 0x7fdc7c100570 2024-03-21T10:41:10.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.152+0000 7fdc83800640 1 -- 172.21.15.31:0/2318681402 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc7c196a20 con 0x7fdc7c100570 2024-03-21T10:41:10.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.153+0000 7fdc83800640 1 -- 172.21.15.31:0/2318681402 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc7c1b3850 con 0x7fdc7c100570 2024-03-21T10:41:10.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.158+0000 7fdc86633740 1 Processor -- start 2024-03-21T10:41:10.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.159+0000 7fdc86633740 1 -- start start 2024-03-21T10:41:10.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.159+0000 7fdc86633740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56083a5d7530 0x56083a5d7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.159+0000 7fdc86633740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56083a5d7ee0 0x56083a5ed620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.159+0000 7fdc86633740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083a5edc40 0x56083a5ee0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.159+0000 7fdc86633740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56083a53b2c0 con 0x56083a5d7530 2024-03-21T10:41:10.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.159+0000 7fdc86633740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56083a4e3c90 con 0x56083a5d7ee0 2024-03-21T10:41:10.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.159+0000 7fdc86633740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56083a537360 con 0x56083a5edc40 2024-03-21T10:41:10.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.160+0000 7fdc7be00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083a5edc40 0x56083a5ee0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.160+0000 7fdc7be00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083a5edc40 0x56083a5ee0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:52758/0 (socket says 172.21.15.31:52758) 2024-03-21T10:41:10.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.160+0000 7fdc7be00640 1 -- 172.21.15.31:0/2886238701 learned_addr learned my addr 172.21.15.31:0/2886238701 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:10.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.161+0000 7fdc7be00640 1 -- 172.21.15.31:0/2886238701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56083a5d7ee0 msgr2=0x56083a5ed620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:41:10.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.161+0000 7fdc7be00640 1 --2- 172.21.15.31:0/2886238701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56083a5d7ee0 0x56083a5ed620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.161+0000 7fdc7be00640 1 -- 172.21.15.31:0/2886238701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56083a5d7530 msgr2=0x56083a5d7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:10.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.161+0000 7fdc7be00640 1 --2- 172.21.15.31:0/2886238701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56083a5d7530 0x56083a5d7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.161+0000 7fdc7be00640 1 -- 172.21.15.31:0/2886238701 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56083a508080 con 0x56083a5edc40 2024-03-21T10:41:10.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.161+0000 7fdc7be00640 1 --2- 172.21.15.31:0/2886238701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083a5edc40 0x56083a5ee0c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fdc6802c930 tx=0x7fdc6802cdf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9d21dda9c062c0b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:10.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.162+0000 7fdc6e000640 1 -- 172.21.15.31:0/2886238701 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc6802ed90 con 0x56083a5edc40 2024-03-21T10:41:10.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.162+0000 7fdc6e000640 1 -- 172.21.15.31:0/2886238701 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc68005500 con 0x56083a5edc40 2024-03-21T10:41:10.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.162+0000 7fdc6e000640 1 -- 172.21.15.31:0/2886238701 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc680316c0 con 0x56083a5edc40 2024-03-21T10:41:10.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.162+0000 7fdc79600640 1 -- 172.21.15.31:0/2318681402 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fdc700046c0 con 0x7fdc7c100570 2024-03-21T10:41:10.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.163+0000 7fdc86633740 1 -- 172.21.15.31:0/2886238701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083a5edc40 msgr2=0x56083a5ee0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:10.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.163+0000 7fdc86633740 1 --2- 172.21.15.31:0/2886238701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083a5edc40 0x56083a5ee0c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fdc6802c930 tx=0x7fdc6802cdf0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.164+0000 7fdc86633740 1 -- 172.21.15.31:0/2886238701 shutdown_connections 2024-03-21T10:41:10.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.164+0000 7fdc86633740 1 --2- 172.21.15.31:0/2886238701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083a5edc40 0x56083a5ee0c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.164+0000 7fdc86633740 1 --2- 172.21.15.31:0/2886238701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56083a5d7ee0 0x56083a5ed620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.164+0000 7fdc86633740 1 --2- 172.21.15.31:0/2886238701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56083a5d7530 0x56083a5d7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.164+0000 7fdc86633740 1 -- 172.21.15.31:0/2886238701 >> 172.21.15.31:0/2886238701 conn(0x56083a5d1da0 msgr2=0x56083a5d2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:10.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.165+0000 7fdc86633740 1 -- 172.21.15.31:0/2886238701 shutdown_connections 2024-03-21T10:41:10.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.165+0000 7fdc79600640 1 --2- 172.21.15.31:0/2318681402 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fdc640a3020 0x7fdc640a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.165+0000 7fdc86633740 1 -- 172.21.15.31:0/2886238701 wait complete. 2024-03-21T10:41:10.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.165+0000 7fdc79600640 1 -- 172.21.15.31:0/2318681402 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fdc700d0000 con 0x7fdc7c100570 2024-03-21T10:41:10.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.166+0000 7fdc7be00640 1 --2- 172.21.15.31:0/2318681402 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fdc640a3020 0x7fdc640a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.166+0000 7fdc86633740 1 -- start start 2024-03-21T10:41:10.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.167+0000 7fdc7be00640 1 --2- 172.21.15.31:0/2318681402 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fdc640a3020 0x7fdc640a54e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdc6802c900 tx=0x7fdc68004ce0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:10.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.167+0000 7fdc86633740 1 Processor -- start 2024-03-21T10:41:10.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.168+0000 7fdc86633740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56083a5d7530 0x56083a5deb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.169+0000 7fdc86633740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56083a5d7ee0 0x56083a5df0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.169+0000 7fdc86633740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083a5edc40 0x56083a5d9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.169+0000 7fdc86633740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56083a5d8810 con 0x56083a5d7530 2024-03-21T10:41:10.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.169+0000 7fdc86633740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56083a4f70d0 con 0x56083a5d7ee0 2024-03-21T10:41:10.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.169+0000 7fdc86633740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56083a5d8990 con 0x56083a5edc40 2024-03-21T10:41:10.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.169+0000 7fdc86633740 10 client.0 _ll_register_callbacks cb 0x56083a4fbe80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:10.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.169+0000 7fdc80e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56083a5d7530 0x56083a5deb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.169+0000 7fdc80e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56083a5d7530 0x56083a5deb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58954/0 (socket says 172.21.15.31:58954) 2024-03-21T10:41:10.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.169+0000 7fdc80e00640 1 -- 172.21.15.31:0/2530430286 learned_addr learned my addr 172.21.15.31:0/2530430286 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:10.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.170+0000 7fdc80e00640 1 -- 172.21.15.31:0/2530430286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083a5edc40 msgr2=0x56083a5d9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:10.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.170+0000 7fdc81800640 1 --2- 172.21.15.31:0/2530430286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083a5edc40 0x56083a5d9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.170+0000 7fdc80e00640 1 --2- 172.21.15.31:0/2530430286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083a5edc40 0x56083a5d9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.170+0000 7fdc80e00640 1 -- 172.21.15.31:0/2530430286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56083a5d7ee0 msgr2=0x56083a5df0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:10.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.170+0000 7fdc80e00640 1 --2- 172.21.15.31:0/2530430286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56083a5d7ee0 0x56083a5df0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.170+0000 7fdc80e00640 1 -- 172.21.15.31:0/2530430286 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56083a24f3f0 con 0x56083a5d7530 2024-03-21T10:41:10.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.170+0000 7fdc81800640 1 --2- 172.21.15.31:0/2530430286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083a5edc40 0x56083a5d9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:41:10.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.170+0000 7fdc80e00640 1 --2- 172.21.15.31:0/2530430286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56083a5d7530 0x56083a5deb70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdc7000c6a0 tx=0x7fdc7009c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=896970b72d2a9264 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:10.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.171+0000 7fdc62a00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:41:10.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.171+0000 7fdc62a00640 1 -- 172.21.15.31:0/2530430286 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc700a4a60 con 0x56083a5d7530 2024-03-21T10:41:10.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.171+0000 7fdc62a00640 1 -- 172.21.15.31:0/2530430286 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fdc700986c0 con 0x56083a5d7530 2024-03-21T10:41:10.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.171+0000 7fdc86633740 1 -- 172.21.15.31:0/2530430286 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdc68005500 con 0x56083a5d7530 2024-03-21T10:41:10.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.172+0000 7fdc62a00640 1 -- 172.21.15.31:0/2530430286 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc700cd630 con 0x56083a5d7530 2024-03-21T10:41:10.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.172+0000 7fdc86633740 10 client.16065 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:10.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.172+0000 7fdc86633740 1 -- 172.21.15.31:0/2530430286 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56083a3197d0 con 0x56083a5d7530 2024-03-21T10:41:10.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.174+0000 7fdc62a00640 1 -- 172.21.15.31:0/2530430286 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc68005500 con 0x56083a5d7530 2024-03-21T10:41:10.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.174+0000 7fdc62a00640 1 -- 172.21.15.31:0/2530430286 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fdc70096bc0 con 0x56083a5d7530 2024-03-21T10:41:10.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.174+0000 7fdc86633740 10 client.16065 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:10.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.174+0000 7fdc86633740 10 client.16065 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:41:10.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.174+0000 7fdc86633740 20 client.16065 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:10.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.174+0000 7fdc86633740 1 --2- 172.21.15.31:0/2530430286 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x56083a5e9db0 0x56083a5ec200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.174+0000 7fdc86633740 4 client.16065 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:41:10.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.174+0000 7fdc86633740 1 -- 172.21.15.31:0/2530430286 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56083a548800 con 0x56083a5e9db0 2024-03-21T10:41:10.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.177+0000 7fdc7be00640 1 --2- 172.21.15.31:0/2530430286 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x56083a5e9db0 0x56083a5ec200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.182+0000 7fdc7be00640 1 --2- 172.21.15.31:0/2530430286 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x56083a5e9db0 0x56083a5ec200 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e98b95b3eff823ba server_cookie=b4609f8c159588ae in_seq=0 out_seq=0 2024-03-21T10:41:10.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.182+0000 7fdc62a00640 10 client.16065 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:41:10.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.184+0000 7fdc62a00640 1 -- 172.21.15.31:0/2530430286 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56083a548800 con 0x56083a5e9db0 2024-03-21T10:41:10.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.184+0000 7fdc62a00640 10 client.16065 handle_command_reply: tid=0 2024-03-21T10:41:10.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.184+0000 7fdc86633740 1 client.16065 shutdown 2024-03-21T10:41:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.184+0000 7fdc86633740 1 -- 172.21.15.31:0/2530430286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56083a5d7530 msgr2=0x56083a5deb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.184+0000 7fdc86633740 1 --2- 172.21.15.31:0/2530430286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56083a5d7530 0x56083a5deb70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdc7000c6a0 tx=0x7fdc7009c750 comp rx=0 tx=0).stop 2024-03-21T10:41:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.184+0000 7fdc86633740 1 -- 172.21.15.31:0/2530430286 shutdown_connections 2024-03-21T10:41:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.184+0000 7fdc86633740 1 --2- 172.21.15.31:0/2530430286 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x56083a5e9db0 0x56083a5ec200 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.185+0000 7fdc86633740 1 --2- 172.21.15.31:0/2530430286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083a5edc40 0x56083a5d9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.185+0000 7fdc86633740 1 --2- 172.21.15.31:0/2530430286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56083a5d7ee0 0x56083a5df0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.185+0000 7fdc86633740 1 --2- 172.21.15.31:0/2530430286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56083a5d7530 0x56083a5deb70 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.185+0000 7fdc86633740 1 -- 172.21.15.31:0/2530430286 >> 172.21.15.31:0/2530430286 conn(0x56083a5d1da0 msgr2=0x56083a5d3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.185+0000 7fdc62a00640 0 client.16065 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:41:10.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.185+0000 7fdc86633740 1 -- 172.21.15.31:0/2530430286 shutdown_connections 2024-03-21T10:41:10.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.185+0000 7fdc86633740 1 -- 172.21.15.31:0/2530430286 wait complete. 2024-03-21T10:41:10.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.185+0000 7fdc86633740 20 client.16065 trim_cache size 0 max 16384 2024-03-21T10:41:10.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.215+0000 7fdc86633740 1 Processor -- start 2024-03-21T10:41:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.216+0000 7fdc86633740 1 -- start start 2024-03-21T10:41:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.216+0000 7fdc86633740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56083a5d7530 0x56083a5eca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.216+0000 7fdc86633740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56083a5d7ee0 0x56083a5d9c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.216+0000 7fdc86633740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083a5edc40 0x56083a5f2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.216+0000 7fdc86633740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c1b7480 con 0x56083a5d7530 2024-03-21T10:41:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.216+0000 7fdc86633740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c1b4f80 con 0x56083a5d7ee0 2024-03-21T10:41:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.216+0000 7fdc86633740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c067c90 con 0x56083a5edc40 2024-03-21T10:41:10.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.217+0000 7fdc7be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56083a5d7ee0 0x56083a5d9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.217+0000 7fdc7be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56083a5d7ee0 0x56083a5d9c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60548/0 (socket says 172.21.15.31:60548) 2024-03-21T10:41:10.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.217+0000 7fdc7be00640 1 -- 172.21.15.31:0/453162532 learned_addr learned my addr 172.21.15.31:0/453162532 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:10.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.217+0000 7fdc7be00640 1 -- 172.21.15.31:0/453162532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083a5edc40 msgr2=0x56083a5f2c50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:41:10.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.217+0000 7fdc7be00640 1 --2- 172.21.15.31:0/453162532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083a5edc40 0x56083a5f2c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.217+0000 7fdc7be00640 1 -- 172.21.15.31:0/453162532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56083a5d7530 msgr2=0x56083a5eca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:10.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.217+0000 7fdc80e00640 1 --2- 172.21.15.31:0/453162532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56083a5d7530 0x56083a5eca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.217+0000 7fdc7be00640 1 --2- 172.21.15.31:0/453162532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56083a5d7530 0x56083a5eca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.217+0000 7fdc7be00640 1 -- 172.21.15.31:0/453162532 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56083a5df420 con 0x56083a5d7ee0 2024-03-21T10:41:10.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.218+0000 7fdc80e00640 1 --2- 172.21.15.31:0/453162532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56083a5d7530 0x56083a5eca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:41:10.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.218+0000 7fdc7be00640 1 --2- 172.21.15.31:0/453162532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56083a5d7ee0 0x56083a5d9c20 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fdc68004d50 tx=0x7fdc68040000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1208fa65cebae7a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:10.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.218+0000 7fdc6d600640 1 -- 172.21.15.31:0/453162532 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc6802c440 con 0x56083a5d7ee0 2024-03-21T10:41:10.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.219+0000 7fdc6d600640 1 -- 172.21.15.31:0/453162532 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc68004eb0 con 0x56083a5d7ee0 2024-03-21T10:41:10.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.219+0000 7fdc6d600640 1 -- 172.21.15.31:0/453162532 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc68052600 con 0x56083a5d7ee0 2024-03-21T10:41:10.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.221+0000 7fdc86633740 1 -- 172.21.15.31:0/453162532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56083a5d7ee0 msgr2=0x56083a5d9c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:10.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.221+0000 7fdc86633740 1 --2- 172.21.15.31:0/453162532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56083a5d7ee0 0x56083a5d9c20 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fdc68004d50 tx=0x7fdc68040000 comp rx=0 tx=0).stop 2024-03-21T10:41:10.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.221+0000 7fdc86633740 1 -- 172.21.15.31:0/453162532 shutdown_connections 2024-03-21T10:41:10.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.222+0000 7fdc86633740 1 --2- 172.21.15.31:0/453162532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083a5edc40 0x56083a5f2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.222+0000 7fdc86633740 1 --2- 172.21.15.31:0/453162532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56083a5d7ee0 0x56083a5d9c20 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.222+0000 7fdc86633740 1 --2- 172.21.15.31:0/453162532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56083a5d7530 0x56083a5eca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.222+0000 7fdc86633740 1 -- 172.21.15.31:0/453162532 >> 172.21.15.31:0/453162532 conn(0x56083a5d1da0 msgr2=0x56083a5f0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:10.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.223+0000 7fdc86633740 1 -- 172.21.15.31:0/453162532 shutdown_connections 2024-03-21T10:41:10.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.223+0000 7fdc86633740 1 -- 172.21.15.31:0/453162532 wait complete. 2024-03-21T10:41:10.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.224+0000 7fdc86633740 1 -- start start 2024-03-21T10:41:10.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.224+0000 7fdc86633740 1 Processor -- start 2024-03-21T10:41:10.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.226+0000 7fdc86633740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083a5d7530 0x56083a5e2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.227+0000 7fdc86633740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56083a5d7ee0 0x56083a5e3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.227+0000 7fdc86633740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56083a5edc40 0x56083a64b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.229+0000 7fdc86633740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c0734a0 con 0x56083a5d7ee0 2024-03-21T10:41:10.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.229+0000 7fdc86633740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c1b4920 con 0x56083a5edc40 2024-03-21T10:41:10.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.229+0000 7fdc86633740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c1b3c60 con 0x56083a5d7530 2024-03-21T10:41:10.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.229+0000 7fdc86633740 10 client.0 _ll_register_callbacks cb 0x7fdc7000ce60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:10.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.227+0000 7fdc80e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083a5d7530 0x56083a5e2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.230+0000 7fdc80e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083a5d7530 0x56083a5e2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:52772/0 (socket says 172.21.15.31:52772) 2024-03-21T10:41:10.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.230+0000 7fdc80e00640 1 -- 172.21.15.31:0/425479122 learned_addr learned my addr 172.21.15.31:0/425479122 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:10.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.230+0000 7fdc81800640 1 --2- 172.21.15.31:0/425479122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56083a5edc40 0x56083a64b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.230+0000 7fdc7be00640 1 --2- 172.21.15.31:0/425479122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56083a5d7ee0 0x56083a5e3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.230+0000 7fdc81800640 1 -- 172.21.15.31:0/425479122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083a5d7530 msgr2=0x56083a5e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:10.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.230+0000 7fdc81800640 1 --2- 172.21.15.31:0/425479122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083a5d7530 0x56083a5e2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.231+0000 7fdc81800640 1 -- 172.21.15.31:0/425479122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56083a5d7ee0 msgr2=0x56083a5e3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:10.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.233+0000 7fdc81800640 1 --2- 172.21.15.31:0/425479122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56083a5d7ee0 0x56083a5e3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.233+0000 7fdc81800640 1 -- 172.21.15.31:0/425479122 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56083a64bab0 con 0x56083a5edc40 2024-03-21T10:41:10.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.233+0000 7fdc80e00640 1 --2- 172.21.15.31:0/425479122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083a5d7530 0x56083a5e2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:41:10.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.233+0000 7fdc7be00640 1 --2- 172.21.15.31:0/425479122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56083a5d7ee0 0x56083a5e3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:41:10.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.233+0000 7fdc81800640 1 --2- 172.21.15.31:0/425479122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56083a5edc40 0x56083a64b570 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x56083a5cc400 tx=0x7fdc740336b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c2ca7d29ecaf18d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:10.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.233+0000 7fdc62a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:41:10.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.234+0000 7fdc62a00640 1 -- 172.21.15.31:0/425479122 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc74043070 con 0x56083a5edc40 2024-03-21T10:41:10.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.234+0000 7fdc86633740 1 -- 172.21.15.31:0/425479122 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdc68004eb0 con 0x56083a5edc40 2024-03-21T10:41:10.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.234+0000 7fdc62a00640 1 -- 172.21.15.31:0/425479122 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fdc74003970 con 0x56083a5edc40 2024-03-21T10:41:10.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.234+0000 7fdc62a00640 1 -- 172.21.15.31:0/425479122 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc7404f050 con 0x56083a5edc40 2024-03-21T10:41:10.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.234+0000 7fdc86633740 10 client.26309 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:10.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.235+0000 7fdc86633740 1 -- 172.21.15.31:0/425479122 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56083a64cc10 con 0x56083a5edc40 2024-03-21T10:41:10.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.235+0000 7fdc62a00640 1 -- 172.21.15.31:0/425479122 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc68004eb0 con 0x56083a5edc40 2024-03-21T10:41:10.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.236+0000 7fdc62a00640 1 -- 172.21.15.31:0/425479122 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fdc74004140 con 0x56083a5edc40 2024-03-21T10:41:10.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.236+0000 7fdc86633740 10 client.26309 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:10.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.236+0000 7fdc86633740 10 client.26309 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:41:10.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.236+0000 7fdc86633740 20 client.26309 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:10.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.236+0000 7fdc86633740 1 --2- 172.21.15.31:0/425479122 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x56083a64d320 0x56083a64f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.236+0000 7fdc86633740 4 client.26309 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:41:10.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.236+0000 7fdc86633740 1 -- 172.21.15.31:0/425479122 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56083a542b50 con 0x56083a64d320 2024-03-21T10:41:10.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.237+0000 7fdc80e00640 1 --2- 172.21.15.31:0/425479122 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x56083a64d320 0x56083a64f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.239+0000 7fdc80e00640 1 --2- 172.21.15.31:0/425479122 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x56083a64d320 0x56083a64f740 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=26b1a0a5948dcb54 server_cookie=e9574e6da06275a3 in_seq=0 out_seq=0 2024-03-21T10:41:10.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.239+0000 7fdc62a00640 10 client.26309 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:41:10.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.239+0000 7fdc62a00640 1 -- 172.21.15.31:0/425479122 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56083a542b50 con 0x56083a64d320 2024-03-21T10:41:10.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.239+0000 7fdc62a00640 10 client.26309 handle_command_reply: tid=0 2024-03-21T10:41:10.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.239+0000 7fdc86633740 1 client.26309 shutdown 2024-03-21T10:41:10.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.240+0000 7fdc86633740 1 -- 172.21.15.31:0/425479122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56083a5edc40 msgr2=0x56083a64b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:10.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.240+0000 7fdc86633740 1 --2- 172.21.15.31:0/425479122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56083a5edc40 0x56083a64b570 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x56083a5cc400 tx=0x7fdc740336b0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.240+0000 7fdc86633740 1 -- 172.21.15.31:0/425479122 shutdown_connections 2024-03-21T10:41:10.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.240+0000 7fdc86633740 1 --2- 172.21.15.31:0/425479122 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x56083a64d320 0x56083a64f740 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.241+0000 7fdc86633740 1 --2- 172.21.15.31:0/425479122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56083a5edc40 0x56083a64b570 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.241+0000 7fdc86633740 1 --2- 172.21.15.31:0/425479122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56083a5d7ee0 0x56083a5e3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.241+0000 7fdc86633740 1 --2- 172.21.15.31:0/425479122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083a5d7530 0x56083a5e2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.241+0000 7fdc86633740 1 -- 172.21.15.31:0/425479122 >> 172.21.15.31:0/425479122 conn(0x56083a5d1da0 msgr2=0x56083a5f0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:10.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.241+0000 7fdc62a00640 0 client.26309 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:41:10.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.242+0000 7fdc86633740 1 -- 172.21.15.31:0/425479122 shutdown_connections 2024-03-21T10:41:10.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.242+0000 7fdc86633740 1 -- 172.21.15.31:0/425479122 wait complete. 2024-03-21T10:41:10.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.242+0000 7fdc86633740 20 client.26309 trim_cache size 0 max 16384 2024-03-21T10:41:10.243 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:41:10.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.246+0000 7fdc83800640 1 -- 172.21.15.31:0/2318681402 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fdc640a3020 msgr2=0x7fdc640a54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:10.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.246+0000 7fdc83800640 1 --2- 172.21.15.31:0/2318681402 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fdc640a3020 0x7fdc640a54e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdc6802c900 tx=0x7fdc68004ce0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.247+0000 7fdc83800640 1 -- 172.21.15.31:0/2318681402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc7c100570 msgr2=0x7fdc7c19dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:10.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.247+0000 7fdc83800640 1 --2- 172.21.15.31:0/2318681402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc7c100570 0x7fdc7c19dad0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fdc7000e9c0 tx=0x7fdc7000ee80 comp rx=0 tx=0).stop 2024-03-21T10:41:10.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.247+0000 7fdc83800640 1 -- 172.21.15.31:0/2318681402 shutdown_connections 2024-03-21T10:41:10.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.247+0000 7fdc83800640 1 --2- 172.21.15.31:0/2318681402 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fdc640a3020 0x7fdc640a54e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.247+0000 7fdc83800640 1 --2- 172.21.15.31:0/2318681402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc7c106630 0x7fdc7c195e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.247+0000 7fdc83800640 1 --2- 172.21.15.31:0/2318681402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc7c100f30 0x7fdc7c19e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.247+0000 7fdc83800640 1 --2- 172.21.15.31:0/2318681402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc7c100570 0x7fdc7c19dad0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.247+0000 7fdc83800640 1 -- 172.21.15.31:0/2318681402 >> 172.21.15.31:0/2318681402 conn(0x7fdc7c0776f0 msgr2=0x7fdc7c069880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:10.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.247+0000 7fdc83800640 1 -- 172.21.15.31:0/2318681402 shutdown_connections 2024-03-21T10:41:10.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.247+0000 7fdc83800640 1 -- 172.21.15.31:0/2318681402 wait complete. 2024-03-21T10:41:10.275 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:41:10.275 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-21T10:41:10.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.586+0000 7f50d7200640 1 -- 172.21.15.31:0/2186044952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50d8112470 msgr2=0x7f50d8114860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:10.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.586+0000 7f50d7200640 1 --2- 172.21.15.31:0/2186044952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50d8112470 0x7f50d8114860 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f50d0009ca0 tx=0x7f50d002c190 comp rx=0 tx=0).stop 2024-03-21T10:41:10.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.586+0000 7f50d7200640 1 -- 172.21.15.31:0/2186044952 shutdown_connections 2024-03-21T10:41:10.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.587+0000 7f50d7200640 1 --2- 172.21.15.31:0/2186044952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50d8112470 0x7f50d8114860 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.587+0000 7f50d7200640 1 --2- 172.21.15.31:0/2186044952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50d8074360 0x7f50d8111f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.587+0000 7f50d7200640 1 --2- 172.21.15.31:0/2186044952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50d8073a40 0x7f50d8073e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.587+0000 7f50d7200640 1 -- 172.21.15.31:0/2186044952 >> 172.21.15.31:0/2186044952 conn(0x7f50d80fcff0 msgr2=0x7f50d80ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:10.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.587+0000 7f50d7200640 1 -- 172.21.15.31:0/2186044952 shutdown_connections 2024-03-21T10:41:10.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.588+0000 7f50d7200640 1 -- 172.21.15.31:0/2186044952 wait complete. 2024-03-21T10:41:10.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.590+0000 7f50d7200640 1 Processor -- start 2024-03-21T10:41:10.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.591+0000 7f50d7200640 1 -- start start 2024-03-21T10:41:10.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.591+0000 7f50d7200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50d8073a40 0x7f50d8111ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.591+0000 7f50d7200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50d8074360 0x7f50d810abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.591+0000 7f50d7200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50d8112470 0x7f50d810b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.591+0000 7f50d7200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d8117e70 con 0x7f50d8074360 2024-03-21T10:41:10.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.591+0000 7f50d7200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d8117cf0 con 0x7f50d8073a40 2024-03-21T10:41:10.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.591+0000 7f50d7200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d8117ff0 con 0x7f50d8112470 2024-03-21T10:41:10.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.592+0000 7f50d6800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50d8112470 0x7f50d810b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.592+0000 7f50d5400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50d8074360 0x7f50d810abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.592+0000 7f50d5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50d8073a40 0x7f50d8111ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.592+0000 7f50d5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50d8073a40 0x7f50d8111ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60574/0 (socket says 172.21.15.31:60574) 2024-03-21T10:41:10.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.592+0000 7f50d5e00640 1 -- 172.21.15.31:0/4293998126 learned_addr learned my addr 172.21.15.31:0/4293998126 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:10.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.592+0000 7f50d5e00640 1 -- 172.21.15.31:0/4293998126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50d8112470 msgr2=0x7f50d810b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:10.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.594+0000 7f50d5e00640 1 --2- 172.21.15.31:0/4293998126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50d8112470 0x7f50d810b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.594+0000 7f50d5e00640 1 -- 172.21.15.31:0/4293998126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50d8074360 msgr2=0x7f50d810abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:10.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.594+0000 7f50d5e00640 1 --2- 172.21.15.31:0/4293998126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50d8074360 0x7f50d810abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.594+0000 7f50d5e00640 1 -- 172.21.15.31:0/4293998126 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d810b760 con 0x7f50d8073a40 2024-03-21T10:41:10.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.594+0000 7f50d5400640 1 --2- 172.21.15.31:0/4293998126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50d8074360 0x7f50d810abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:41:10.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.594+0000 7f50d5e00640 1 --2- 172.21.15.31:0/4293998126 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50d8073a40 0x7f50d8111ae0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f50c400d950 tx=0x7f50c400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:10.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.594+0000 7f50d6800640 1 --2- 172.21.15.31:0/4293998126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50d8112470 0x7f50d810b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:41:10.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.595+0000 7f50cea00640 1 -- 172.21.15.31:0/4293998126 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c4017070 con 0x7f50d8073a40 2024-03-21T10:41:10.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.595+0000 7f50cea00640 1 -- 172.21.15.31:0/4293998126 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50c4009c30 con 0x7f50d8073a40 2024-03-21T10:41:10.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.595+0000 7f50cea00640 1 -- 172.21.15.31:0/4293998126 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c400f6b0 con 0x7f50d8073a40 2024-03-21T10:41:10.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.595+0000 7f50d7200640 1 -- 172.21.15.31:0/4293998126 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50d810f600 con 0x7f50d8073a40 2024-03-21T10:41:10.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.595+0000 7f50d7200640 1 -- 172.21.15.31:0/4293998126 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50d810fb00 con 0x7f50d8073a40 2024-03-21T10:41:10.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.601+0000 7f50df995740 1 Processor -- start 2024-03-21T10:41:10.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.601+0000 7f50df995740 1 -- start start 2024-03-21T10:41:10.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.602+0000 7f50cea00640 1 -- 172.21.15.31:0/4293998126 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f50c400f850 con 0x7f50d8073a40 2024-03-21T10:41:10.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.602+0000 7f50df995740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562935a138b0 0x562935a13c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.602+0000 7f50df995740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562935a14260 0x562935a1c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.602+0000 7f50df995740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562935a1cde0 0x562935a1f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.602+0000 7f50df995740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629359772c0 con 0x562935a14260 2024-03-21T10:41:10.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.602+0000 7f50df995740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56293591fc90 con 0x562935a1cde0 2024-03-21T10:41:10.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.602+0000 7f50df995740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x562935973360 con 0x562935a138b0 2024-03-21T10:41:10.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.603+0000 7f50d5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562935a14260 0x562935a1c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.603+0000 7f50d6800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562935a138b0 0x562935a13c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.603+0000 7f50d6800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562935a138b0 0x562935a13c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:52810/0 (socket says 172.21.15.31:52810) 2024-03-21T10:41:10.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.603+0000 7f50d5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562935a14260 0x562935a1c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58990/0 (socket says 172.21.15.31:58990) 2024-03-21T10:41:10.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.603+0000 7f50d6800640 1 -- 172.21.15.31:0/2795133284 learned_addr learned my addr 172.21.15.31:0/2795133284 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:10.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.603+0000 7f50d5400640 1 --2- 172.21.15.31:0/2795133284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562935a1cde0 0x562935a1f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.605+0000 7f50cea00640 1 --2- 172.21.15.31:0/4293998126 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f50bc0a3150 0x7f50bc0a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.606+0000 7f50d6800640 1 -- 172.21.15.31:0/2795133284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562935a1cde0 msgr2=0x562935a1f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:10.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.608+0000 7f50d6800640 1 --2- 172.21.15.31:0/2795133284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562935a1cde0 0x562935a1f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.608+0000 7f50d6800640 1 -- 172.21.15.31:0/2795133284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562935a14260 msgr2=0x562935a1c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:10.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.608+0000 7f50cea00640 1 -- 172.21.15.31:0/4293998126 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f50c40cf5b0 con 0x7f50d8073a40 2024-03-21T10:41:10.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.610+0000 7f50d6800640 1 --2- 172.21.15.31:0/2795133284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562935a14260 0x562935a1c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.610+0000 7f50d6800640 1 -- 172.21.15.31:0/2795133284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562935944080 con 0x562935a138b0 2024-03-21T10:41:10.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.610+0000 7f50d5400640 1 --2- 172.21.15.31:0/2795133284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562935a1cde0 0x562935a1f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:41:10.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.610+0000 7f50d5e00640 1 --2- 172.21.15.31:0/2795133284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562935a14260 0x562935a1c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:41:10.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.616+0000 7f50d6800640 1 --2- 172.21.15.31:0/4293998126 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f50bc0a3150 0x7f50bc0a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.618+0000 7f50d6800640 1 --2- 172.21.15.31:0/2795133284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562935a138b0 0x562935a13c90 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f50d0009960 tx=0x7f50d0009990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81803c8e8e10bd12 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:10.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.620+0000 7f50c3000640 1 -- 172.21.15.31:0/2795133284 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d0039050 con 0x562935a138b0 2024-03-21T10:41:10.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.620+0000 7f50c3000640 1 -- 172.21.15.31:0/2795133284 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50d003a040 con 0x562935a138b0 2024-03-21T10:41:10.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.620+0000 7f50c3000640 1 -- 172.21.15.31:0/2795133284 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d00328d0 con 0x562935a138b0 2024-03-21T10:41:10.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.621+0000 7f50df995740 1 -- 172.21.15.31:0/2795133284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562935a138b0 msgr2=0x562935a13c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:10.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.621+0000 7f50df995740 1 --2- 172.21.15.31:0/2795133284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562935a138b0 0x562935a13c90 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f50d0009960 tx=0x7f50d0009990 comp rx=0 tx=0).stop 2024-03-21T10:41:10.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.624+0000 7f50d6800640 1 --2- 172.21.15.31:0/4293998126 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f50bc0a3150 0x7f50bc0a5610 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f50d0048000 tx=0x7f50d00048f0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:10.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.624+0000 7f50df995740 1 -- 172.21.15.31:0/2795133284 shutdown_connections 2024-03-21T10:41:10.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.624+0000 7f50df995740 1 --2- 172.21.15.31:0/2795133284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562935a1cde0 0x562935a1f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.624+0000 7f50df995740 1 --2- 172.21.15.31:0/2795133284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562935a14260 0x562935a1c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.624+0000 7f50df995740 1 --2- 172.21.15.31:0/2795133284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562935a138b0 0x562935a13c90 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.624+0000 7f50df995740 1 -- 172.21.15.31:0/2795133284 >> 172.21.15.31:0/2795133284 conn(0x562935a0df80 msgr2=0x562935a0e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:10.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.625+0000 7f50df995740 1 -- 172.21.15.31:0/2795133284 shutdown_connections 2024-03-21T10:41:10.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.625+0000 7f50df995740 1 -- 172.21.15.31:0/2795133284 wait complete. 2024-03-21T10:41:10.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.629+0000 7f50df995740 1 -- start start 2024-03-21T10:41:10.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.629+0000 7f50df995740 1 Processor -- start 2024-03-21T10:41:10.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.632+0000 7f50df995740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562935a138b0 0x562935a1c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.632+0000 7f50df995740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562935a14260 0x562935a19a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.633+0000 7f50d5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562935a138b0 0x562935a1c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.633+0000 7f50d5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562935a138b0 0x562935a1c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60586/0 (socket says 172.21.15.31:60586) 2024-03-21T10:41:10.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.633+0000 7f50d5e00640 1 -- 172.21.15.31:0/121132698 learned_addr learned my addr 172.21.15.31:0/121132698 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:10.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.633+0000 7f50d5400640 1 --2- 172.21.15.31:0/121132698 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562935a14260 0x562935a19a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.633+0000 7f50df995740 1 --2- 172.21.15.31:0/121132698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562935a1cde0 0x562935a19f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.634+0000 7f50df995740 1 -- 172.21.15.31:0/121132698 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562935a1fd50 con 0x562935a14260 2024-03-21T10:41:10.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.634+0000 7f50df995740 1 -- 172.21.15.31:0/121132698 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629359330d0 con 0x562935a138b0 2024-03-21T10:41:10.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.634+0000 7f50df995740 1 -- 172.21.15.31:0/121132698 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x562935a1fed0 con 0x562935a1cde0 2024-03-21T10:41:10.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.634+0000 7f50df995740 10 client.0 _ll_register_callbacks cb 0x562935937e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:10.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.634+0000 7f50d6800640 1 --2- 172.21.15.31:0/121132698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562935a1cde0 0x562935a19f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.634+0000 7f50d5e00640 1 -- 172.21.15.31:0/121132698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562935a1cde0 msgr2=0x562935a19f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:10.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.634+0000 7f50d5e00640 1 --2- 172.21.15.31:0/121132698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562935a1cde0 0x562935a19f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.634+0000 7f50d5e00640 1 -- 172.21.15.31:0/121132698 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562935a14260 msgr2=0x562935a19a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:10.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.634+0000 7f50d5e00640 1 --2- 172.21.15.31:0/121132698 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562935a14260 0x562935a19a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.634+0000 7f50d5e00640 1 -- 172.21.15.31:0/121132698 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56293568b3f0 con 0x562935a138b0 2024-03-21T10:41:10.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.634+0000 7f50d5400640 1 --2- 172.21.15.31:0/121132698 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562935a14260 0x562935a19a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:41:10.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.635+0000 7f50d5e00640 1 --2- 172.21.15.31:0/121132698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562935a138b0 0x562935a1c6a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f50c409c270 tx=0x7f50c409c340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=778432a33222109a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:10.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.635+0000 7f50b7400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:41:10.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.635+0000 7f50b7400640 1 -- 172.21.15.31:0/121132698 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c40cf070 con 0x562935a138b0 2024-03-21T10:41:10.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.635+0000 7f50df995740 1 -- 172.21.15.31:0/121132698 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f50d003a040 con 0x562935a138b0 2024-03-21T10:41:10.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.635+0000 7f50b7400640 1 -- 172.21.15.31:0/121132698 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f50c40109d0 con 0x562935a138b0 2024-03-21T10:41:10.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.636+0000 7f50b7400640 1 -- 172.21.15.31:0/121132698 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50c4020030 con 0x562935a138b0 2024-03-21T10:41:10.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.636+0000 7f50df995740 10 client.26321 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:10.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.636+0000 7f50df995740 1 -- 172.21.15.31:0/121132698 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5629357557d0 con 0x562935a138b0 2024-03-21T10:41:10.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.636+0000 7f50b7400640 1 -- 172.21.15.31:0/121132698 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50d003a040 con 0x562935a138b0 2024-03-21T10:41:10.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.636+0000 7f50b7400640 1 -- 172.21.15.31:0/121132698 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f50c4023060 con 0x562935a138b0 2024-03-21T10:41:10.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.637+0000 7f50df995740 10 client.26321 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:10.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.637+0000 7f50df995740 10 client.26321 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:41:10.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.637+0000 7f50df995740 20 client.26321 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:10.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.637+0000 7f50df995740 1 --2- 172.21.15.31:0/121132698 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x562935a2cb30 0x562935a2ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.637+0000 7f50df995740 4 client.26321 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:41:10.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.637+0000 7f50df995740 1 -- 172.21.15.31:0/121132698 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562935984800 con 0x562935a2cb30 2024-03-21T10:41:10.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.638+0000 7f50d5400640 1 --2- 172.21.15.31:0/121132698 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x562935a2cb30 0x562935a2ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.639+0000 7f50d5400640 1 --2- 172.21.15.31:0/121132698 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x562935a2cb30 0x562935a2ef40 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d9ecab6b1a24ff9a server_cookie=23baf1907da76d38 in_seq=0 out_seq=0 2024-03-21T10:41:10.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.639+0000 7f50b7400640 10 client.26321 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:41:10.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.640+0000 7f50b7400640 1 -- 172.21.15.31:0/121132698 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562935984800 con 0x562935a2cb30 2024-03-21T10:41:10.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.640+0000 7f50b7400640 10 client.26321 handle_command_reply: tid=0 2024-03-21T10:41:10.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.640+0000 7f50df995740 1 client.26321 shutdown 2024-03-21T10:41:10.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.641+0000 7f50df995740 1 -- 172.21.15.31:0/121132698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562935a138b0 msgr2=0x562935a1c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:10.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.641+0000 7f50df995740 1 --2- 172.21.15.31:0/121132698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562935a138b0 0x562935a1c6a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f50c409c270 tx=0x7f50c409c340 comp rx=0 tx=0).stop 2024-03-21T10:41:10.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.641+0000 7f50df995740 1 -- 172.21.15.31:0/121132698 shutdown_connections 2024-03-21T10:41:10.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.641+0000 7f50df995740 1 --2- 172.21.15.31:0/121132698 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x562935a2cb30 0x562935a2ef40 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.641+0000 7f50df995740 1 --2- 172.21.15.31:0/121132698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562935a1cde0 0x562935a19f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.641+0000 7f50df995740 1 --2- 172.21.15.31:0/121132698 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562935a14260 0x562935a19a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.641+0000 7f50df995740 1 --2- 172.21.15.31:0/121132698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562935a138b0 0x562935a1c6a0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.641+0000 7f50df995740 1 -- 172.21.15.31:0/121132698 >> 172.21.15.31:0/121132698 conn(0x562935a0df80 msgr2=0x562935a0fcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:10.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.641+0000 7f50b7400640 0 client.26321 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:41:10.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.641+0000 7f50df995740 1 -- 172.21.15.31:0/121132698 shutdown_connections 2024-03-21T10:41:10.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.641+0000 7f50df995740 1 -- 172.21.15.31:0/121132698 wait complete. 2024-03-21T10:41:10.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.641+0000 7f50df995740 20 client.26321 trim_cache size 0 max 16384 2024-03-21T10:41:10.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.667+0000 7f50df995740 1 Processor -- start 2024-03-21T10:41:10.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.667+0000 7f50df995740 1 -- start start 2024-03-21T10:41:10.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.668+0000 7f50df995740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562935a138b0 0x562935a1c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.668+0000 7f50df995740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562935a14260 0x562935a1c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.668+0000 7f50df995740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562935a1cde0 0x562935a16cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.668+0000 7f50df995740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d81bfac0 con 0x562935a1cde0 2024-03-21T10:41:10.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.668+0000 7f50df995740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d81bd5c0 con 0x562935a138b0 2024-03-21T10:41:10.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.668+0000 7f50df995740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d81bb660 con 0x562935a14260 2024-03-21T10:41:10.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.668+0000 7f50d5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562935a138b0 0x562935a1c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.668+0000 7f50d5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562935a138b0 0x562935a1c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60594/0 (socket says 172.21.15.31:60594) 2024-03-21T10:41:10.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.668+0000 7f50d5e00640 1 -- 172.21.15.31:0/3173496516 learned_addr learned my addr 172.21.15.31:0/3173496516 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:10.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.668+0000 7f50d5400640 1 --2- 172.21.15.31:0/3173496516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562935a14260 0x562935a1c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.668+0000 7f50d6800640 1 --2- 172.21.15.31:0/3173496516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562935a1cde0 0x562935a16cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.669+0000 7f50d5e00640 1 -- 172.21.15.31:0/3173496516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562935a14260 msgr2=0x562935a1c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:10.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.669+0000 7f50d5e00640 1 --2- 172.21.15.31:0/3173496516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562935a14260 0x562935a1c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.669+0000 7f50d5e00640 1 -- 172.21.15.31:0/3173496516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562935a1cde0 msgr2=0x562935a16cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:10.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.669+0000 7f50d5e00640 1 --2- 172.21.15.31:0/3173496516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562935a1cde0 0x562935a16cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.669+0000 7f50d5e00640 1 -- 172.21.15.31:0/3173496516 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562935a098b0 con 0x562935a138b0 2024-03-21T10:41:10.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.669+0000 7f50d6800640 1 --2- 172.21.15.31:0/3173496516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562935a1cde0 0x562935a16cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:41:10.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.670+0000 7f50d5e00640 1 --2- 172.21.15.31:0/3173496516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562935a138b0 0x562935a1c160 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f50c409c500 tx=0x7f50c4005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e92c49071042ae0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:10.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.670+0000 7f50c2600640 1 -- 172.21.15.31:0/3173496516 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c40cf040 con 0x562935a138b0 2024-03-21T10:41:10.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.670+0000 7f50c2600640 1 -- 172.21.15.31:0/3173496516 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50c409cc70 con 0x562935a138b0 2024-03-21T10:41:10.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.670+0000 7f50c2600640 1 -- 172.21.15.31:0/3173496516 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c40a40f0 con 0x562935a138b0 2024-03-21T10:41:10.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.671+0000 7f50df995740 1 -- 172.21.15.31:0/3173496516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562935a138b0 msgr2=0x562935a1c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:10.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.671+0000 7f50df995740 1 --2- 172.21.15.31:0/3173496516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562935a138b0 0x562935a1c160 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f50c409c500 tx=0x7f50c4005880 comp rx=0 tx=0).stop 2024-03-21T10:41:10.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.671+0000 7f50df995740 1 -- 172.21.15.31:0/3173496516 shutdown_connections 2024-03-21T10:41:10.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.671+0000 7f50df995740 1 --2- 172.21.15.31:0/3173496516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562935a1cde0 0x562935a16cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.671+0000 7f50df995740 1 --2- 172.21.15.31:0/3173496516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562935a14260 0x562935a1c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.671+0000 7f50df995740 1 --2- 172.21.15.31:0/3173496516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562935a138b0 0x562935a1c160 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.671+0000 7f50df995740 1 -- 172.21.15.31:0/3173496516 >> 172.21.15.31:0/3173496516 conn(0x562935a0df80 msgr2=0x562935a0d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:10.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.671+0000 7f50df995740 1 -- 172.21.15.31:0/3173496516 shutdown_connections 2024-03-21T10:41:10.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.672+0000 7f50df995740 1 -- 172.21.15.31:0/3173496516 wait complete. 2024-03-21T10:41:10.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.672+0000 7f50df995740 1 -- start start 2024-03-21T10:41:10.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.673+0000 7f50df995740 1 Processor -- start 2024-03-21T10:41:10.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.674+0000 7f50df995740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562935a138b0 0x562935a24be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.674+0000 7f50df995740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562935a14260 0x562935a20f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.674+0000 7f50df995740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562935a1cde0 0x562935a214a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.674+0000 7f50df995740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d81bbc60 con 0x562935a14260 2024-03-21T10:41:10.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.675+0000 7f50df995740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d81bcf60 con 0x562935a1cde0 2024-03-21T10:41:10.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.675+0000 7f50df995740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d81bc2a0 con 0x562935a138b0 2024-03-21T10:41:10.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.675+0000 7f50df995740 10 client.0 _ll_register_callbacks cb 0x7f50c40d2f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:10.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.675+0000 7f50d5e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562935a138b0 0x562935a24be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.675+0000 7f50d5400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562935a14260 0x562935a20f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.675+0000 7f50d5e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562935a138b0 0x562935a24be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:52832/0 (socket says 172.21.15.31:52832) 2024-03-21T10:41:10.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.675+0000 7f50d5e00640 1 -- 172.21.15.31:0/312108571 learned_addr learned my addr 172.21.15.31:0/312108571 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:10.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.675+0000 7f50d5e00640 1 -- 172.21.15.31:0/312108571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562935a1cde0 msgr2=0x562935a214a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:10.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.675+0000 7f50d5e00640 1 --2- 172.21.15.31:0/312108571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562935a1cde0 0x562935a214a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.675+0000 7f50d5e00640 1 -- 172.21.15.31:0/312108571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562935a14260 msgr2=0x562935a20f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:10.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.676+0000 7f50d5e00640 1 --2- 172.21.15.31:0/312108571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562935a14260 0x562935a20f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.676+0000 7f50d5e00640 1 -- 172.21.15.31:0/312108571 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562935a219e0 con 0x562935a138b0 2024-03-21T10:41:10.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.676+0000 7f50d5e00640 1 --2- 172.21.15.31:0/312108571 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562935a138b0 0x562935a24be0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f50c4098f40 tx=0x7f50c40cf4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4327ef4ad933a53 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:10.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.676+0000 7f50b7400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:41:10.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.677+0000 7f50b7400640 1 -- 172.21.15.31:0/312108571 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c4015030 con 0x562935a138b0 2024-03-21T10:41:10.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.677+0000 7f50b7400640 1 -- 172.21.15.31:0/312108571 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f50c409b9c0 con 0x562935a138b0 2024-03-21T10:41:10.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.677+0000 7f50df995740 1 -- 172.21.15.31:0/312108571 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f50c409cc70 con 0x562935a138b0 2024-03-21T10:41:10.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.679+0000 7f50b7400640 1 -- 172.21.15.31:0/312108571 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50c40cf040 con 0x562935a138b0 2024-03-21T10:41:10.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.679+0000 7f50df995740 10 client.26278 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:10.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.679+0000 7f50df995740 1 -- 172.21.15.31:0/312108571 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x562935a89110 con 0x562935a138b0 2024-03-21T10:41:10.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.680+0000 7f50b7400640 1 -- 172.21.15.31:0/312108571 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50c409cc70 con 0x562935a138b0 2024-03-21T10:41:10.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.680+0000 7f50b7400640 1 -- 172.21.15.31:0/312108571 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f50c40ba0a0 con 0x562935a138b0 2024-03-21T10:41:10.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.681+0000 7f50df995740 10 client.26278 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:10.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.681+0000 7f50df995740 10 client.26278 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:41:10.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.681+0000 7f50df995740 20 client.26278 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:10.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.681+0000 7f50df995740 1 --2- 172.21.15.31:0/312108571 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x562935a89430 0x562935a8b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:10.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.681+0000 7f50df995740 4 client.26278 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:41:10.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.681+0000 7f50df995740 1 -- 172.21.15.31:0/312108571 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56293597eb50 con 0x562935a89430 2024-03-21T10:41:10.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.683+0000 7f50d5400640 1 --2- 172.21.15.31:0/312108571 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x562935a89430 0x562935a8b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:10.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.683+0000 7f50d5400640 1 --2- 172.21.15.31:0/312108571 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x562935a89430 0x562935a8b850 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c20ea8e098ab09f0 server_cookie=3dc681066f68620b in_seq=0 out_seq=0 2024-03-21T10:41:10.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.684+0000 7f50b7400640 10 client.26278 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:41:10.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.684+0000 7f50b7400640 1 -- 172.21.15.31:0/312108571 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56293597eb50 con 0x562935a89430 2024-03-21T10:41:10.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.684+0000 7f50b7400640 10 client.26278 handle_command_reply: tid=0 2024-03-21T10:41:10.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.685+0000 7f50df995740 1 client.26278 shutdown 2024-03-21T10:41:10.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.685+0000 7f50df995740 1 -- 172.21.15.31:0/312108571 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562935a138b0 msgr2=0x562935a24be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:10.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.685+0000 7f50df995740 1 --2- 172.21.15.31:0/312108571 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562935a138b0 0x562935a24be0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f50c4098f40 tx=0x7f50c40cf4e0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.685+0000 7f50df995740 1 -- 172.21.15.31:0/312108571 shutdown_connections 2024-03-21T10:41:10.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.686+0000 7f50df995740 1 --2- 172.21.15.31:0/312108571 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x562935a89430 0x562935a8b850 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.686+0000 7f50df995740 1 --2- 172.21.15.31:0/312108571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562935a1cde0 0x562935a214a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.686+0000 7f50df995740 1 --2- 172.21.15.31:0/312108571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562935a14260 0x562935a20f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.686+0000 7f50df995740 1 --2- 172.21.15.31:0/312108571 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562935a138b0 0x562935a24be0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.686+0000 7f50df995740 1 -- 172.21.15.31:0/312108571 >> 172.21.15.31:0/312108571 conn(0x562935a0df80 msgr2=0x562935a0f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:10.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.687+0000 7f50b7400640 0 client.26278 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:41:10.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.687+0000 7f50df995740 1 -- 172.21.15.31:0/312108571 shutdown_connections 2024-03-21T10:41:10.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.688+0000 7f50df995740 1 -- 172.21.15.31:0/312108571 wait complete. 2024-03-21T10:41:10.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.688+0000 7f50df995740 20 client.26278 trim_cache size 0 max 16384 2024-03-21T10:41:10.685 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:41:10.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.689+0000 7f50d7200640 1 -- 172.21.15.31:0/4293998126 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f50bc0a3150 msgr2=0x7f50bc0a5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:10.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.689+0000 7f50d7200640 1 --2- 172.21.15.31:0/4293998126 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f50bc0a3150 0x7f50bc0a5610 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f50d0048000 tx=0x7f50d00048f0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.689+0000 7f50d7200640 1 -- 172.21.15.31:0/4293998126 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50d8073a40 msgr2=0x7f50d8111ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:10.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.689+0000 7f50d7200640 1 --2- 172.21.15.31:0/4293998126 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50d8073a40 0x7f50d8111ae0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f50c400d950 tx=0x7f50c400de10 comp rx=0 tx=0).stop 2024-03-21T10:41:10.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.690+0000 7f50d7200640 1 -- 172.21.15.31:0/4293998126 shutdown_connections 2024-03-21T10:41:10.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.690+0000 7f50d7200640 1 --2- 172.21.15.31:0/4293998126 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f50bc0a3150 0x7f50bc0a5610 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.690+0000 7f50d7200640 1 --2- 172.21.15.31:0/4293998126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50d8112470 0x7f50d810b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.690+0000 7f50d7200640 1 --2- 172.21.15.31:0/4293998126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50d8074360 0x7f50d810abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.690+0000 7f50d7200640 1 --2- 172.21.15.31:0/4293998126 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50d8073a40 0x7f50d8111ae0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:10.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.690+0000 7f50d7200640 1 -- 172.21.15.31:0/4293998126 >> 172.21.15.31:0/4293998126 conn(0x7f50d80fcff0 msgr2=0x7f50d80fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:10.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.690+0000 7f50d7200640 1 -- 172.21.15.31:0/4293998126 shutdown_connections 2024-03-21T10:41:10.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:10.690+0000 7f50d7200640 1 -- 172.21.15.31:0/4293998126 wait complete. 2024-03-21T10:41:10.715 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:41:10.716 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-21T10:41:10.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:10 smithi040 ceph-mon[36154]: pgmap v81: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 274 MiB/s wr, 1.45k op/s 2024-03-21T10:41:10.931 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:10 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1058880533' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:41:11.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.056+0000 7f1412600640 1 -- 172.21.15.31:0/629689708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f140c101ff0 msgr2=0x7f140c10e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:11.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.056+0000 7f1412600640 1 --2- 172.21.15.31:0/629689708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f140c101ff0 0x7f140c10e500 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1404009cd0 tx=0x7f140402c0e0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.056+0000 7f1412600640 1 -- 172.21.15.31:0/629689708 shutdown_connections 2024-03-21T10:41:11.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.056+0000 7f1412600640 1 --2- 172.21.15.31:0/629689708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f140c101ff0 0x7f140c10e500 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.056+0000 7f1412600640 1 --2- 172.21.15.31:0/629689708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f140c101630 0x7f140c101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.056+0000 7f1412600640 1 --2- 172.21.15.31:0/629689708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f140c107640 0x7f140c107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.056+0000 7f1412600640 1 -- 172.21.15.31:0/629689708 >> 172.21.15.31:0/629689708 conn(0x7f140c0fd140 msgr2=0x7f140c0ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:11.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.056+0000 7f1412600640 1 -- 172.21.15.31:0/629689708 shutdown_connections 2024-03-21T10:41:11.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.056+0000 7f1412600640 1 -- 172.21.15.31:0/629689708 wait complete. 2024-03-21T10:41:11.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.058+0000 7f1412600640 1 Processor -- start 2024-03-21T10:41:11.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:10 smithi031 ceph-mon[31673]: pgmap v81: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 274 MiB/s wr, 1.45k op/s 2024-03-21T10:41:11.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:10 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1058880533' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:41:11.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.058+0000 7f1412600640 1 -- start start 2024-03-21T10:41:11.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.058+0000 7f1412600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f140c101630 0x7f140c19a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:11.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.058+0000 7f1412600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f140c101ff0 0x7f140c19a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:11.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.058+0000 7f1412600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f140c107640 0x7f140c19ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:11.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.058+0000 7f1412600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140c111a50 con 0x7f140c101630 2024-03-21T10:41:11.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.058+0000 7f1412600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140c1118d0 con 0x7f140c101ff0 2024-03-21T10:41:11.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.058+0000 7f1412600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140c111bd0 con 0x7f140c107640 2024-03-21T10:41:11.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.059+0000 7f140aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f140c101ff0 0x7f140c19a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:11.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.059+0000 7f140be00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f140c107640 0x7f140c19ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:11.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.059+0000 7f140be00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f140c107640 0x7f140c19ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:52844/0 (socket says 172.21.15.31:52844) 2024-03-21T10:41:11.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.059+0000 7f140be00640 1 -- 172.21.15.31:0/252891727 learned_addr learned my addr 172.21.15.31:0/252891727 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:11.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.059+0000 7f140b400640 1 --2- 172.21.15.31:0/252891727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f140c101630 0x7f140c19a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:11.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.059+0000 7f140aa00640 1 -- 172.21.15.31:0/252891727 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f140c107640 msgr2=0x7f140c19ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:11.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.059+0000 7f140aa00640 1 --2- 172.21.15.31:0/252891727 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f140c107640 0x7f140c19ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.059+0000 7f140aa00640 1 -- 172.21.15.31:0/252891727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f140c101630 msgr2=0x7f140c19a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:11.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.060+0000 7f140aa00640 1 --2- 172.21.15.31:0/252891727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f140c101630 0x7f140c19a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.060+0000 7f140aa00640 1 -- 172.21.15.31:0/252891727 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f140c19f160 con 0x7f140c101ff0 2024-03-21T10:41:11.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.060+0000 7f140b400640 1 --2- 172.21.15.31:0/252891727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f140c101630 0x7f140c19a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:41:11.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.060+0000 7f140aa00640 1 --2- 172.21.15.31:0/252891727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f140c101ff0 0x7f140c19a6b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f13fc00e9f0 tx=0x7f13fc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:11.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.061+0000 7f1403e00640 1 -- 172.21.15.31:0/252891727 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13fc00ccc0 con 0x7f140c101ff0 2024-03-21T10:41:11.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.061+0000 7f1403e00640 1 -- 172.21.15.31:0/252891727 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f13fc005020 con 0x7f140c101ff0 2024-03-21T10:41:11.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.061+0000 7f1403e00640 1 -- 172.21.15.31:0/252891727 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13fc0049e0 con 0x7f140c101ff0 2024-03-21T10:41:11.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.062+0000 7f1412600640 1 -- 172.21.15.31:0/252891727 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f140c19f450 con 0x7f140c101ff0 2024-03-21T10:41:11.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.062+0000 7f1412600640 1 -- 172.21.15.31:0/252891727 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f140c1a6ce0 con 0x7f140c101ff0 2024-03-21T10:41:11.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:10 smithi070 ceph-mon[36330]: pgmap v81: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 274 MiB/s wr, 1.45k op/s 2024-03-21T10:41:11.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:10 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1058880533' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:41:11.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.076+0000 7f1403e00640 1 -- 172.21.15.31:0/252891727 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f13fc005560 con 0x7f140c101ff0 2024-03-21T10:41:11.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.078+0000 7f14153c0740 1 Processor -- start 2024-03-21T10:41:11.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.079+0000 7f14153c0740 1 -- start start 2024-03-21T10:41:11.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.079+0000 7f14153c0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a6a5f55b0 0x560a6a5f5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:11.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.079+0000 7f14153c0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a6a5f5f60 0x560a6a602e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:11.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.079+0000 7f14153c0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6a603460 0x560a6a6038c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:11.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.079+0000 7f14153c0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a6a5592c0 con 0x560a6a5f55b0 2024-03-21T10:41:11.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.079+0000 7f14153c0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a6a501c90 con 0x560a6a5f5f60 2024-03-21T10:41:11.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.079+0000 7f14153c0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a6a555360 con 0x560a6a603460 2024-03-21T10:41:11.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.081+0000 7f140aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6a603460 0x560a6a6038c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:11.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.081+0000 7f140aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6a603460 0x560a6a6038c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:52860/0 (socket says 172.21.15.31:52860) 2024-03-21T10:41:11.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.081+0000 7f140aa00640 1 -- 172.21.15.31:0/2152786244 learned_addr learned my addr 172.21.15.31:0/2152786244 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:11.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.081+0000 7f140b400640 1 --2- 172.21.15.31:0/2152786244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a6a5f5f60 0x560a6a602e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:11.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.082+0000 7f140be00640 1 --2- 172.21.15.31:0/2152786244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a6a5f55b0 0x560a6a5f5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:11.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.082+0000 7f140aa00640 1 -- 172.21.15.31:0/2152786244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a6a5f5f60 msgr2=0x560a6a602e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:11.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.082+0000 7f140aa00640 1 --2- 172.21.15.31:0/2152786244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a6a5f5f60 0x560a6a602e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.082+0000 7f140aa00640 1 -- 172.21.15.31:0/2152786244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a6a5f55b0 msgr2=0x560a6a5f5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:11.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.083+0000 7f140aa00640 1 --2- 172.21.15.31:0/2152786244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a6a5f55b0 0x560a6a5f5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.083+0000 7f140aa00640 1 -- 172.21.15.31:0/2152786244 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a6a526080 con 0x560a6a603460 2024-03-21T10:41:11.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.083+0000 7f140be00640 1 --2- 172.21.15.31:0/2152786244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a6a5f55b0 0x560a6a5f5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:41:11.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.085+0000 7f1403e00640 1 --2- 172.21.15.31:0/252891727 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f13f00a31e0 0x7f13f00a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:11.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.086+0000 7f1403e00640 1 -- 172.21.15.31:0/252891727 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f13fc0cfa60 con 0x7f140c101ff0 2024-03-21T10:41:11.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.085+0000 7f140aa00640 1 --2- 172.21.15.31:0/2152786244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6a603460 0x560a6a6038c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f13fc0ccde0 tx=0x7f13fc0d3d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=253b4d131b95a9fb server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:11.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.088+0000 7f1400c00640 1 -- 172.21.15.31:0/2152786244 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13fc01c070 con 0x560a6a603460 2024-03-21T10:41:11.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.088+0000 7f1400c00640 1 -- 172.21.15.31:0/2152786244 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f13fc0051d0 con 0x560a6a603460 2024-03-21T10:41:11.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.088+0000 7f1400c00640 1 -- 172.21.15.31:0/2152786244 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13fc09b650 con 0x560a6a603460 2024-03-21T10:41:11.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.089+0000 7f140be00640 1 --2- 172.21.15.31:0/252891727 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f13f00a31e0 0x7f13f00a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:11.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.089+0000 7f14153c0740 1 -- 172.21.15.31:0/2152786244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6a603460 msgr2=0x560a6a6038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:11.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.089+0000 7f14153c0740 1 --2- 172.21.15.31:0/2152786244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6a603460 0x560a6a6038c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f13fc0ccde0 tx=0x7f13fc0d3d30 comp rx=0 tx=0).stop 2024-03-21T10:41:11.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.090+0000 7f140be00640 1 --2- 172.21.15.31:0/252891727 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f13f00a31e0 0x7f13f00a56a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1404006010 tx=0x7f140402e040 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:11.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.091+0000 7f14153c0740 1 -- 172.21.15.31:0/2152786244 shutdown_connections 2024-03-21T10:41:11.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.091+0000 7f14153c0740 1 --2- 172.21.15.31:0/2152786244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6a603460 0x560a6a6038c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.092+0000 7f14153c0740 1 --2- 172.21.15.31:0/2152786244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a6a5f5f60 0x560a6a602e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.092+0000 7f14153c0740 1 --2- 172.21.15.31:0/2152786244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a6a5f55b0 0x560a6a5f5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.092+0000 7f14153c0740 1 -- 172.21.15.31:0/2152786244 >> 172.21.15.31:0/2152786244 conn(0x560a6a5efda0 msgr2=0x560a6a5f0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:11.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.093+0000 7f14153c0740 1 -- 172.21.15.31:0/2152786244 shutdown_connections 2024-03-21T10:41:11.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.093+0000 7f14153c0740 1 -- 172.21.15.31:0/2152786244 wait complete. 2024-03-21T10:41:11.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.094+0000 7f14153c0740 1 -- start start 2024-03-21T10:41:11.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.094+0000 7f14153c0740 1 Processor -- start 2024-03-21T10:41:11.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.096+0000 7f14153c0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a6a5f55b0 0x560a6a609890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:11.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.096+0000 7f14153c0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a6a5f5f60 0x560a6a606c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:11.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.097+0000 7f14153c0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6a603460 0x560a6a607180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:11.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.097+0000 7f14153c0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a6a60a3a0 con 0x560a6a5f5f60 2024-03-21T10:41:11.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.097+0000 7f14153c0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a6a5150d0 con 0x560a6a5f55b0 2024-03-21T10:41:11.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.097+0000 7f14153c0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a6a60a520 con 0x560a6a603460 2024-03-21T10:41:11.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.098+0000 7f14153c0740 10 client.0 _ll_register_callbacks cb 0x560a6a519e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:11.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.098+0000 7f140b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a6a5f55b0 0x560a6a609890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:11.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.098+0000 7f140b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a6a5f55b0 0x560a6a609890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60624/0 (socket says 172.21.15.31:60624) 2024-03-21T10:41:11.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.098+0000 7f140b400640 1 -- 172.21.15.31:0/202637525 learned_addr learned my addr 172.21.15.31:0/202637525 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:11.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.098+0000 7f140be00640 1 --2- 172.21.15.31:0/202637525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6a603460 0x560a6a607180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:11.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.098+0000 7f140aa00640 1 --2- 172.21.15.31:0/202637525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a6a5f5f60 0x560a6a606c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:11.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.099+0000 7f140b400640 1 -- 172.21.15.31:0/202637525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6a603460 msgr2=0x560a6a607180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:11.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.099+0000 7f140b400640 1 --2- 172.21.15.31:0/202637525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6a603460 0x560a6a607180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.099+0000 7f140b400640 1 -- 172.21.15.31:0/202637525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a6a5f5f60 msgr2=0x560a6a606c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:11.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.099+0000 7f140b400640 1 --2- 172.21.15.31:0/202637525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a6a5f5f60 0x560a6a606c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.099+0000 7f140b400640 1 -- 172.21.15.31:0/202637525 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560a6a26d3f0 con 0x560a6a5f55b0 2024-03-21T10:41:11.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.099+0000 7f140aa00640 1 --2- 172.21.15.31:0/202637525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a6a5f5f60 0x560a6a606c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:41:11.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.099+0000 7f140b400640 1 --2- 172.21.15.31:0/202637525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a6a5f55b0 0x560a6a609890 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f13f8010750 tx=0x7f13f8010c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e76b6a81dc90cd72 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:11.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.100+0000 7f13f5600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:41:11.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.101+0000 7f13f5600640 1 -- 172.21.15.31:0/202637525 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f801d060 con 0x560a6a5f55b0 2024-03-21T10:41:11.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.101+0000 7f13f5600640 1 -- 172.21.15.31:0/202637525 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f13f800eda0 con 0x560a6a5f55b0 2024-03-21T10:41:11.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.101+0000 7f14153c0740 1 -- 172.21.15.31:0/202637525 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f13fc0051d0 con 0x560a6a5f55b0 2024-03-21T10:41:11.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.102+0000 7f13f5600640 1 -- 172.21.15.31:0/202637525 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f13f8024050 con 0x560a6a5f55b0 2024-03-21T10:41:11.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.102+0000 7f14153c0740 10 client.26336 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:11.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.102+0000 7f14153c0740 1 -- 172.21.15.31:0/202637525 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x560a6a3377d0 con 0x560a6a5f55b0 2024-03-21T10:41:11.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.103+0000 7f13f5600640 1 -- 172.21.15.31:0/202637525 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f13fc0051d0 con 0x560a6a5f55b0 2024-03-21T10:41:11.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.103+0000 7f13f5600640 1 -- 172.21.15.31:0/202637525 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f13f8016570 con 0x560a6a5f55b0 2024-03-21T10:41:11.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.104+0000 7f14153c0740 10 client.26336 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:11.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.104+0000 7f14153c0740 10 client.26336 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:41:11.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.104+0000 7f14153c0740 20 client.26336 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:11.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.104+0000 7f14153c0740 1 --2- 172.21.15.31:0/202637525 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560a6a617280 0x560a6a6196b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:11.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.104+0000 7f14153c0740 4 client.26336 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:41:11.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.104+0000 7f14153c0740 1 -- 172.21.15.31:0/202637525 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560a6a566800 con 0x560a6a617280 2024-03-21T10:41:11.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.105+0000 7f140aa00640 1 --2- 172.21.15.31:0/202637525 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560a6a617280 0x560a6a6196b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:11.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.106+0000 7f13f5600640 10 client.26336 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:11.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.108+0000 7f140aa00640 1 --2- 172.21.15.31:0/202637525 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560a6a617280 0x560a6a6196b0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc7baccb852f87f9 server_cookie=d78478df16287b04 in_seq=0 out_seq=0 2024-03-21T10:41:11.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.109+0000 7f13f5600640 1 -- 172.21.15.31:0/202637525 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560a6a566800 con 0x560a6a617280 2024-03-21T10:41:11.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.109+0000 7f13f5600640 10 client.26336 handle_command_reply: tid=0 2024-03-21T10:41:11.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.109+0000 7f14153c0740 1 client.26336 shutdown 2024-03-21T10:41:11.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.111+0000 7f14153c0740 1 -- 172.21.15.31:0/202637525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a6a5f55b0 msgr2=0x560a6a609890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:11.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.111+0000 7f14153c0740 1 --2- 172.21.15.31:0/202637525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a6a5f55b0 0x560a6a609890 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f13f8010750 tx=0x7f13f8010c10 comp rx=0 tx=0).stop 2024-03-21T10:41:11.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.111+0000 7f14153c0740 1 -- 172.21.15.31:0/202637525 shutdown_connections 2024-03-21T10:41:11.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.111+0000 7f14153c0740 1 --2- 172.21.15.31:0/202637525 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560a6a617280 0x560a6a6196b0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.111+0000 7f14153c0740 1 --2- 172.21.15.31:0/202637525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6a603460 0x560a6a607180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.111+0000 7f14153c0740 1 --2- 172.21.15.31:0/202637525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a6a5f5f60 0x560a6a606c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.111+0000 7f14153c0740 1 --2- 172.21.15.31:0/202637525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a6a5f55b0 0x560a6a609890 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.111+0000 7f13f5600640 0 client.26336 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:11.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.111+0000 7f14153c0740 1 -- 172.21.15.31:0/202637525 >> 172.21.15.31:0/202637525 conn(0x560a6a5efda0 msgr2=0x560a6a5f1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:11.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.113+0000 7f14153c0740 1 -- 172.21.15.31:0/202637525 shutdown_connections 2024-03-21T10:41:11.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.115+0000 7f14153c0740 1 -- 172.21.15.31:0/202637525 wait complete. 2024-03-21T10:41:11.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.115+0000 7f14153c0740 20 client.26336 trim_cache size 0 max 16384 2024-03-21T10:41:11.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.149+0000 7f14153c0740 1 Processor -- start 2024-03-21T10:41:11.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.149+0000 7f14153c0740 1 -- start start 2024-03-21T10:41:11.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.150+0000 7f14153c0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a6a5f55b0 0x560a6a602c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:11.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.150+0000 7f14153c0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6a5f5f60 0x560a6a5eed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:11.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.150+0000 7f14153c0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a6a603460 0x560a6a609350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:11.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.150+0000 7f14153c0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140c1aec90 con 0x560a6a5f55b0 2024-03-21T10:41:11.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.150+0000 7f14153c0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140c1ac790 con 0x560a6a603460 2024-03-21T10:41:11.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.150+0000 7f14153c0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140c1aa890 con 0x560a6a5f5f60 2024-03-21T10:41:11.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.151+0000 7f140aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6a5f5f60 0x560a6a5eed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:11.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.151+0000 7f140be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a6a603460 0x560a6a609350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:11.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.151+0000 7f140aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6a5f5f60 0x560a6a5eed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:52886/0 (socket says 172.21.15.31:52886) 2024-03-21T10:41:11.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.151+0000 7f140aa00640 1 -- 172.21.15.31:0/1167286028 learned_addr learned my addr 172.21.15.31:0/1167286028 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:11.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.151+0000 7f140be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a6a603460 0x560a6a609350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60628/0 (socket says 172.21.15.31:60628) 2024-03-21T10:41:11.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.152+0000 7f140b400640 1 --2- 172.21.15.31:0/1167286028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a6a5f55b0 0x560a6a602c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:11.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.152+0000 7f140aa00640 1 -- 172.21.15.31:0/1167286028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a6a603460 msgr2=0x560a6a609350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:11.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.152+0000 7f140aa00640 1 --2- 172.21.15.31:0/1167286028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a6a603460 0x560a6a609350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.152+0000 7f140aa00640 1 -- 172.21.15.31:0/1167286028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a6a5f55b0 msgr2=0x560a6a602c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:11.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.152+0000 7f140aa00640 1 --2- 172.21.15.31:0/1167286028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a6a5f55b0 0x560a6a602c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.153+0000 7f140aa00640 1 -- 172.21.15.31:0/1167286028 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a6a605bd0 con 0x560a6a5f5f60 2024-03-21T10:41:11.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.153+0000 7f140aa00640 1 --2- 172.21.15.31:0/1167286028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6a5f5f60 0x560a6a5eed30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f13fc0984e0 tx=0x7f13fc0096e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1cf26c586b84fa6d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:11.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.154+0000 7f13f7e00640 1 -- 172.21.15.31:0/1167286028 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13fc01c070 con 0x560a6a5f5f60 2024-03-21T10:41:11.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.154+0000 7f13f7e00640 1 -- 172.21.15.31:0/1167286028 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f13fc0051d0 con 0x560a6a5f5f60 2024-03-21T10:41:11.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.154+0000 7f13f7e00640 1 -- 172.21.15.31:0/1167286028 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13fc09b470 con 0x560a6a5f5f60 2024-03-21T10:41:11.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.156+0000 7f14153c0740 1 -- 172.21.15.31:0/1167286028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6a5f5f60 msgr2=0x560a6a5eed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:11.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.156+0000 7f14153c0740 1 --2- 172.21.15.31:0/1167286028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6a5f5f60 0x560a6a5eed30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f13fc0984e0 tx=0x7f13fc0096e0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.156+0000 7f14153c0740 1 -- 172.21.15.31:0/1167286028 shutdown_connections 2024-03-21T10:41:11.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.156+0000 7f14153c0740 1 --2- 172.21.15.31:0/1167286028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a6a603460 0x560a6a609350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.156+0000 7f14153c0740 1 --2- 172.21.15.31:0/1167286028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6a5f5f60 0x560a6a5eed30 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.156+0000 7f14153c0740 1 --2- 172.21.15.31:0/1167286028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a6a5f55b0 0x560a6a602c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.156+0000 7f14153c0740 1 -- 172.21.15.31:0/1167286028 >> 172.21.15.31:0/1167286028 conn(0x560a6a5efda0 msgr2=0x560a6a5ee620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:11.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.157+0000 7f14153c0740 1 -- 172.21.15.31:0/1167286028 shutdown_connections 2024-03-21T10:41:11.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.157+0000 7f14153c0740 1 -- 172.21.15.31:0/1167286028 wait complete. 2024-03-21T10:41:11.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.158+0000 7f14153c0740 1 -- start start 2024-03-21T10:41:11.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.159+0000 7f14153c0740 1 Processor -- start 2024-03-21T10:41:11.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.160+0000 7f14153c0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6a5f55b0 0x560a6a60b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:11.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.161+0000 7f14153c0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a6a5f5f60 0x560a6a60b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:11.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.161+0000 7f14153c0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a6a603460 0x560a6a671b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:11.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.161+0000 7f14153c0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140c1aae90 con 0x560a6a5f5f60 2024-03-21T10:41:11.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.161+0000 7f14153c0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140c1ac130 con 0x560a6a603460 2024-03-21T10:41:11.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.161+0000 7f14153c0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140c1ab490 con 0x560a6a5f55b0 2024-03-21T10:41:11.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.161+0000 7f14153c0740 10 client.0 _ll_register_callbacks cb 0x7f13f800b5f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:11.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.161+0000 7f140b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6a5f55b0 0x560a6a60b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:11.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.162+0000 7f140b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6a5f55b0 0x560a6a60b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:52892/0 (socket says 172.21.15.31:52892) 2024-03-21T10:41:11.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.162+0000 7f140b400640 1 -- 172.21.15.31:0/3751801349 learned_addr learned my addr 172.21.15.31:0/3751801349 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:11.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.162+0000 7f140aa00640 1 --2- 172.21.15.31:0/3751801349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a6a5f5f60 0x560a6a60b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:11.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.162+0000 7f140be00640 1 --2- 172.21.15.31:0/3751801349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a6a603460 0x560a6a671b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:11.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.165+0000 7f140b400640 1 -- 172.21.15.31:0/3751801349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a6a603460 msgr2=0x560a6a671b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:11.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.165+0000 7f140b400640 1 --2- 172.21.15.31:0/3751801349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a6a603460 0x560a6a671b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.165+0000 7f140b400640 1 -- 172.21.15.31:0/3751801349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a6a5f5f60 msgr2=0x560a6a60b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:11.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.165+0000 7f140b400640 1 --2- 172.21.15.31:0/3751801349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a6a5f5f60 0x560a6a60b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.165+0000 7f140b400640 1 -- 172.21.15.31:0/3751801349 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560a6a6720d0 con 0x560a6a5f55b0 2024-03-21T10:41:11.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.165+0000 7f140b400640 1 --2- 172.21.15.31:0/3751801349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6a5f55b0 0x560a6a60b330 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f13f80169c0 tx=0x7f13f8016de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a781359d89d654a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:11.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.165+0000 7f13f5600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:41:11.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.166+0000 7f13f5600640 1 -- 172.21.15.31:0/3751801349 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f8010e60 con 0x560a6a5f55b0 2024-03-21T10:41:11.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.166+0000 7f13f5600640 1 -- 172.21.15.31:0/3751801349 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f13f800edb0 con 0x560a6a5f55b0 2024-03-21T10:41:11.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.166+0000 7f14153c0740 1 -- 172.21.15.31:0/3751801349 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f13fc0051d0 con 0x560a6a5f55b0 2024-03-21T10:41:11.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.166+0000 7f13f5600640 1 -- 172.21.15.31:0/3751801349 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f13f8036050 con 0x560a6a5f55b0 2024-03-21T10:41:11.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.166+0000 7f14153c0740 10 client.26296 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:11.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.166+0000 7f14153c0740 1 -- 172.21.15.31:0/3751801349 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x560a6a6733d0 con 0x560a6a5f55b0 2024-03-21T10:41:11.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.167+0000 7f13f5600640 1 -- 172.21.15.31:0/3751801349 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f13fc0051d0 con 0x560a6a5f55b0 2024-03-21T10:41:11.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.167+0000 7f13f5600640 1 -- 172.21.15.31:0/3751801349 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f13f8005480 con 0x560a6a5f55b0 2024-03-21T10:41:11.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.167+0000 7f14153c0740 10 client.26296 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:11.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.167+0000 7f14153c0740 10 client.26296 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:41:11.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.167+0000 7f14153c0740 20 client.26296 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:11.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.167+0000 7f14153c0740 1 --2- 172.21.15.31:0/3751801349 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560a6a673b20 0x560a6a675f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:11.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.167+0000 7f14153c0740 4 client.26296 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:41:11.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.167+0000 7f14153c0740 1 -- 172.21.15.31:0/3751801349 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560a6a560b50 con 0x560a6a673b20 2024-03-21T10:41:11.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.168+0000 7f140aa00640 1 --2- 172.21.15.31:0/3751801349 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560a6a673b20 0x560a6a675f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:11.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.168+0000 7f140aa00640 1 --2- 172.21.15.31:0/3751801349 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560a6a673b20 0x560a6a675f40 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53c7fe5f0efb80ff server_cookie=7e584658b185aa92 in_seq=0 out_seq=0 2024-03-21T10:41:11.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.168+0000 7f13f5600640 10 client.26296 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:11.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.169+0000 7f13f5600640 1 -- 172.21.15.31:0/3751801349 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560a6a560b50 con 0x560a6a673b20 2024-03-21T10:41:11.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.169+0000 7f13f5600640 10 client.26296 handle_command_reply: tid=0 2024-03-21T10:41:11.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.169+0000 7f14153c0740 1 client.26296 shutdown 2024-03-21T10:41:11.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.169+0000 7f14153c0740 1 -- 172.21.15.31:0/3751801349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6a5f55b0 msgr2=0x560a6a60b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:11.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.169+0000 7f14153c0740 1 --2- 172.21.15.31:0/3751801349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6a5f55b0 0x560a6a60b330 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f13f80169c0 tx=0x7f13f8016de0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.171+0000 7f14153c0740 1 -- 172.21.15.31:0/3751801349 shutdown_connections 2024-03-21T10:41:11.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.171+0000 7f14153c0740 1 --2- 172.21.15.31:0/3751801349 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560a6a673b20 0x560a6a675f40 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.171+0000 7f14153c0740 1 --2- 172.21.15.31:0/3751801349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a6a603460 0x560a6a671b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.171+0000 7f13f5600640 0 client.26296 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:11.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.171+0000 7f14153c0740 1 --2- 172.21.15.31:0/3751801349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a6a5f5f60 0x560a6a60b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.171+0000 7f14153c0740 1 --2- 172.21.15.31:0/3751801349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a6a5f55b0 0x560a6a60b330 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.171+0000 7f14153c0740 1 -- 172.21.15.31:0/3751801349 >> 172.21.15.31:0/3751801349 conn(0x560a6a5efda0 msgr2=0x560a6a5f1970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:11.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.173+0000 7f14153c0740 1 -- 172.21.15.31:0/3751801349 shutdown_connections 2024-03-21T10:41:11.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.173+0000 7f14153c0740 1 -- 172.21.15.31:0/3751801349 wait complete. 2024-03-21T10:41:11.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.173+0000 7f14153c0740 20 client.26296 trim_cache size 0 max 16384 2024-03-21T10:41:11.174 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:41:11.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.179+0000 7f1412600640 1 -- 172.21.15.31:0/252891727 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f13f00a31e0 msgr2=0x7f13f00a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:11.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.179+0000 7f1412600640 1 --2- 172.21.15.31:0/252891727 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f13f00a31e0 0x7f13f00a56a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1404006010 tx=0x7f140402e040 comp rx=0 tx=0).stop 2024-03-21T10:41:11.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.179+0000 7f1412600640 1 -- 172.21.15.31:0/252891727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f140c101ff0 msgr2=0x7f140c19a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:11.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.179+0000 7f1412600640 1 --2- 172.21.15.31:0/252891727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f140c101ff0 0x7f140c19a6b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f13fc00e9f0 tx=0x7f13fc00eeb0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.180+0000 7f1412600640 1 -- 172.21.15.31:0/252891727 shutdown_connections 2024-03-21T10:41:11.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.180+0000 7f1412600640 1 --2- 172.21.15.31:0/252891727 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f13f00a31e0 0x7f13f00a56a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.180+0000 7f1412600640 1 --2- 172.21.15.31:0/252891727 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f140c107640 0x7f140c19ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.180+0000 7f1412600640 1 --2- 172.21.15.31:0/252891727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f140c101ff0 0x7f140c19a6b0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.180+0000 7f1412600640 1 --2- 172.21.15.31:0/252891727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f140c101630 0x7f140c19a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:11.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.180+0000 7f1412600640 1 -- 172.21.15.31:0/252891727 >> 172.21.15.31:0/252891727 conn(0x7f140c0fd140 msgr2=0x7f140c10c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:11.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.180+0000 7f1412600640 1 -- 172.21.15.31:0/252891727 shutdown_connections 2024-03-21T10:41:11.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:11.180+0000 7f1412600640 1 -- 172.21.15.31:0/252891727 wait complete. 2024-03-21T10:41:12.214 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-21T10:41:12.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:41:12.214 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-21T10:41:12.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.426+0000 7efdb4c00640 1 -- 172.21.15.31:0/1323038835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdb0107dc0 msgr2=0x7efdb0108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:12.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.426+0000 7efdb4c00640 1 --2- 172.21.15.31:0/1323038835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdb0107dc0 0x7efdb0108220 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7efd9c009930 tx=0x7efd9c02bc20 comp rx=0 tx=0).stop 2024-03-21T10:41:12.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.427+0000 7efdb4c00640 1 -- 172.21.15.31:0/1323038835 shutdown_connections 2024-03-21T10:41:12.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.427+0000 7efdb4c00640 1 --2- 172.21.15.31:0/1323038835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdb0108760 0x7efdb0114c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.427+0000 7efdb4c00640 1 --2- 172.21.15.31:0/1323038835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdb0107dc0 0x7efdb0108220 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.427+0000 7efdb4c00640 1 --2- 172.21.15.31:0/1323038835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdb0109b70 0x7efdb0109f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.427+0000 7efdb4c00640 1 -- 172.21.15.31:0/1323038835 >> 172.21.15.31:0/1323038835 conn(0x7efdb0077540 msgr2=0x7efdb01064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:12.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.430+0000 7efdb4c00640 1 -- 172.21.15.31:0/1323038835 shutdown_connections 2024-03-21T10:41:12.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.430+0000 7efdb4c00640 1 -- 172.21.15.31:0/1323038835 wait complete. 2024-03-21T10:41:12.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.432+0000 7efdb4c00640 1 Processor -- start 2024-03-21T10:41:12.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.432+0000 7efdb4c00640 1 -- start start 2024-03-21T10:41:12.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.432+0000 7efdb4c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdb0107dc0 0x7efdb019e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:12.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.432+0000 7efdb4c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdb0108760 0x7efdb019eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:12.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.432+0000 7efdb4c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdb0109b70 0x7efdb01a2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:12.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.432+0000 7efdb4c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb0118140 con 0x7efdb0108760 2024-03-21T10:41:12.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.432+0000 7efdb4c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb0117fc0 con 0x7efdb0109b70 2024-03-21T10:41:12.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.433+0000 7efdb4c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb01182c0 con 0x7efdb0107dc0 2024-03-21T10:41:12.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.433+0000 7efdad400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdb0108760 0x7efdb019eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:12.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.435+0000 7efdade00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdb0107dc0 0x7efdb019e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:12.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.435+0000 7efdade00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdb0107dc0 0x7efdb019e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:52896/0 (socket says 172.21.15.31:52896) 2024-03-21T10:41:12.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.435+0000 7efdade00640 1 -- 172.21.15.31:0/2337110732 learned_addr learned my addr 172.21.15.31:0/2337110732 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:12.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.435+0000 7efdae800640 1 --2- 172.21.15.31:0/2337110732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdb0109b70 0x7efdb01a2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:12.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.436+0000 7efdade00640 1 -- 172.21.15.31:0/2337110732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdb0109b70 msgr2=0x7efdb01a2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:12.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.436+0000 7efdade00640 1 --2- 172.21.15.31:0/2337110732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdb0109b70 0x7efdb01a2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.436+0000 7efdade00640 1 -- 172.21.15.31:0/2337110732 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdb0108760 msgr2=0x7efdb019eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:12.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.436+0000 7efdade00640 1 --2- 172.21.15.31:0/2337110732 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdb0108760 0x7efdb019eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.436+0000 7efdade00640 1 -- 172.21.15.31:0/2337110732 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb01a3570 con 0x7efdb0107dc0 2024-03-21T10:41:12.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.436+0000 7efdad400640 1 --2- 172.21.15.31:0/2337110732 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdb0108760 0x7efdb019eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:41:12.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.436+0000 7efdade00640 1 --2- 172.21.15.31:0/2337110732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdb0107dc0 0x7efdb019e600 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7efda000ea90 tx=0x7efda000ef50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:12.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.436+0000 7efda6a00640 1 -- 172.21.15.31:0/2337110732 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda000ce40 con 0x7efdb0107dc0 2024-03-21T10:41:12.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.437+0000 7efdb4c00640 1 -- 172.21.15.31:0/2337110732 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb01a3860 con 0x7efdb0107dc0 2024-03-21T10:41:12.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.437+0000 7efda6a00640 1 -- 172.21.15.31:0/2337110732 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efda00035f0 con 0x7efdb0107dc0 2024-03-21T10:41:12.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.437+0000 7efda6a00640 1 -- 172.21.15.31:0/2337110732 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda0010690 con 0x7efdb0107dc0 2024-03-21T10:41:12.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.437+0000 7efdb4c00640 1 -- 172.21.15.31:0/2337110732 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb0079ef0 con 0x7efdb0107dc0 2024-03-21T10:41:12.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.442+0000 7efdb787f740 1 Processor -- start 2024-03-21T10:41:12.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.442+0000 7efdb787f740 1 -- start start 2024-03-21T10:41:12.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.443+0000 7efdb787f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f37db2b080 0x55f37db2b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:12.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.443+0000 7efdb787f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f37db2b9a0 0x55f37db34040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:12.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.443+0000 7efdb787f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f37db34580 0x55f37db36970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:12.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.443+0000 7efdb787f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f37da8feb0 con 0x55f37db2b9a0 2024-03-21T10:41:12.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.443+0000 7efdb787f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f37da38a60 con 0x55f37db2b080 2024-03-21T10:41:12.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.443+0000 7efdb787f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f37da8bf50 con 0x55f37db34580 2024-03-21T10:41:12.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.443+0000 7efda6a00640 1 -- 172.21.15.31:0/2337110732 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7efda0010830 con 0x7efdb0107dc0 2024-03-21T10:41:12.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.443+0000 7efdae800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f37db2b080 0x55f37db2b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:12.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.443+0000 7efdae800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f37db2b080 0x55f37db2b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60658/0 (socket says 172.21.15.31:60658) 2024-03-21T10:41:12.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.443+0000 7efdae800640 1 -- 172.21.15.31:0/1288456852 learned_addr learned my addr 172.21.15.31:0/1288456852 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:12.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.443+0000 7efdade00640 1 --2- 172.21.15.31:0/1288456852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f37db2b9a0 0x55f37db34040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:12.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.443+0000 7efdae800640 1 -- 172.21.15.31:0/1288456852 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f37db34580 msgr2=0x55f37db36970 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:41:12.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.443+0000 7efdae800640 1 --2- 172.21.15.31:0/1288456852 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f37db34580 0x55f37db36970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.443+0000 7efdae800640 1 -- 172.21.15.31:0/1288456852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f37db2b9a0 msgr2=0x55f37db34040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:12.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.443+0000 7efdae800640 1 --2- 172.21.15.31:0/1288456852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f37db2b9a0 0x55f37db34040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.443+0000 7efdae800640 1 -- 172.21.15.31:0/1288456852 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f37d86c7e0 con 0x55f37db2b080 2024-03-21T10:41:12.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.444+0000 7efdade00640 1 --2- 172.21.15.31:0/1288456852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f37db2b9a0 0x55f37db34040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:41:12.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.444+0000 7efdae800640 1 --2- 172.21.15.31:0/1288456852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f37db2b080 0x55f37db2b460 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7efda8009fc0 tx=0x7efda80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da8e35637270c9ef server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:12.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.444+0000 7efd9b000640 1 -- 172.21.15.31:0/1288456852 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda8004d20 con 0x55f37db2b080 2024-03-21T10:41:12.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.444+0000 7efd9b000640 1 -- 172.21.15.31:0/1288456852 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efda8004050 con 0x55f37db2b080 2024-03-21T10:41:12.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.444+0000 7efd9b000640 1 -- 172.21.15.31:0/1288456852 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda8005260 con 0x55f37db2b080 2024-03-21T10:41:12.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.445+0000 7efda6a00640 1 --2- 172.21.15.31:0/2337110732 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7efd940a3110 0x7efd940a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:12.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.445+0000 7efda6a00640 1 -- 172.21.15.31:0/2337110732 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7efda00cff90 con 0x7efdb0107dc0 2024-03-21T10:41:12.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.445+0000 7efdade00640 1 --2- 172.21.15.31:0/2337110732 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7efd940a3110 0x7efd940a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:12.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.445+0000 7efdb787f740 1 -- 172.21.15.31:0/1288456852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f37db2b080 msgr2=0x55f37db2b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:12.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.445+0000 7efdb787f740 1 --2- 172.21.15.31:0/1288456852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f37db2b080 0x55f37db2b460 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7efda8009fc0 tx=0x7efda80118d0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.445+0000 7efdb787f740 1 -- 172.21.15.31:0/1288456852 shutdown_connections 2024-03-21T10:41:12.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.445+0000 7efdb787f740 1 --2- 172.21.15.31:0/1288456852 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f37db34580 0x55f37db36970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.445+0000 7efdb787f740 1 --2- 172.21.15.31:0/1288456852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f37db2b9a0 0x55f37db34040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.445+0000 7efdb787f740 1 --2- 172.21.15.31:0/1288456852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f37db2b080 0x55f37db2b460 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.445+0000 7efdb787f740 1 -- 172.21.15.31:0/1288456852 >> 172.21.15.31:0/1288456852 conn(0x55f37db257b0 msgr2=0x55f37db25b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:12.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.446+0000 7efdb787f740 1 -- 172.21.15.31:0/1288456852 shutdown_connections 2024-03-21T10:41:12.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.446+0000 7efdb787f740 1 -- 172.21.15.31:0/1288456852 wait complete. 2024-03-21T10:41:12.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.446+0000 7efdade00640 1 --2- 172.21.15.31:0/2337110732 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7efd940a3110 0x7efd940a55d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efda00cd450 tx=0x7efda0003e10 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:12.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.446+0000 7efdb787f740 1 -- start start 2024-03-21T10:41:12.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.446+0000 7efdb787f740 1 Processor -- start 2024-03-21T10:41:12.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.447+0000 7efdb787f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f37db2b080 0x55f37db33fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:12.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.447+0000 7efdb787f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f37db2b9a0 0x55f37db31330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:12.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.447+0000 7efdb787f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f37db34580 0x55f37db31940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:12.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.447+0000 7efdb787f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f37db37370 con 0x55f37db34580 2024-03-21T10:41:12.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.447+0000 7efdb787f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f37da4beb0 con 0x55f37db2b9a0 2024-03-21T10:41:12.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.447+0000 7efdb787f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f37db374f0 con 0x55f37db2b080 2024-03-21T10:41:12.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.447+0000 7efdb787f740 10 client.0 _ll_register_callbacks cb 0x55f37da40150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:12.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.447+0000 7efdae800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f37db2b080 0x55f37db33fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:12.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.447+0000 7efdae800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f37db2b080 0x55f37db33fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:52918/0 (socket says 172.21.15.31:52918) 2024-03-21T10:41:12.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.447+0000 7efdae800640 1 -- 172.21.15.31:0/3174409690 learned_addr learned my addr 172.21.15.31:0/3174409690 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:12.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.447+0000 7efdae800640 1 --2- 172.21.15.31:0/3174409690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f37db34580 0x55f37db31940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:12.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.447+0000 7efdad400640 1 --2- 172.21.15.31:0/3174409690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f37db2b9a0 0x55f37db31330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:12.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.447+0000 7efdae800640 1 -- 172.21.15.31:0/3174409690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f37db2b9a0 msgr2=0x55f37db31330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:12.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.447+0000 7efdae800640 1 --2- 172.21.15.31:0/3174409690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f37db2b9a0 0x55f37db31330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.447+0000 7efdae800640 1 -- 172.21.15.31:0/3174409690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f37db34580 msgr2=0x55f37db31940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:12.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.447+0000 7efdae800640 1 --2- 172.21.15.31:0/3174409690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f37db34580 0x55f37db31940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.447+0000 7efdae800640 1 -- 172.21.15.31:0/3174409690 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f37d8ccf70 con 0x55f37db2b080 2024-03-21T10:41:12.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.448+0000 7efdae800640 1 --2- 172.21.15.31:0/3174409690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f37db2b080 0x55f37db33fa0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7efda80049d0 tx=0x7efda8005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70ae706555f229ab server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:12.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.448+0000 7efd8f400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:41:12.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.448+0000 7efd8f400640 1 -- 172.21.15.31:0/3174409690 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda800f4e0 con 0x55f37db2b080 2024-03-21T10:41:12.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.448+0000 7efdb787f740 1 -- 172.21.15.31:0/3174409690 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efda8004050 con 0x55f37db2b080 2024-03-21T10:41:12.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.448+0000 7efd8f400640 1 -- 172.21.15.31:0/3174409690 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7efda800f680 con 0x55f37db2b080 2024-03-21T10:41:12.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.448+0000 7efd8f400640 1 -- 172.21.15.31:0/3174409690 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7efda8036050 con 0x55f37db2b080 2024-03-21T10:41:12.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.448+0000 7efdb787f740 10 client.26305 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:12.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.448+0000 7efdb787f740 1 -- 172.21.15.31:0/3174409690 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55f37d764170 con 0x55f37db2b080 2024-03-21T10:41:12.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.448+0000 7efd8f400640 1 -- 172.21.15.31:0/3174409690 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efda8004050 con 0x55f37db2b080 2024-03-21T10:41:12.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.449+0000 7efd8f400640 1 -- 172.21.15.31:0/3174409690 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7efda8025040 con 0x55f37db2b080 2024-03-21T10:41:12.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.449+0000 7efdb787f740 10 client.26305 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:12.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.449+0000 7efdb787f740 10 client.26305 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:41:12.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.449+0000 7efdb787f740 20 client.26305 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:12.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.449+0000 7efdb787f740 1 --2- 172.21.15.31:0/3174409690 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f37db44540 0x55f37db46930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:12.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.449+0000 7efdb787f740 4 client.26305 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:41:12.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.449+0000 7efdb787f740 1 -- 172.21.15.31:0/3174409690 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f37da9d3f0 con 0x55f37db44540 2024-03-21T10:41:12.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.450+0000 7efdad400640 1 --2- 172.21.15.31:0/3174409690 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f37db44540 0x55f37db46930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:12.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.450+0000 7efdad400640 1 --2- 172.21.15.31:0/3174409690 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f37db44540 0x55f37db46930 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5da4c088605bfce server_cookie=22f91f2516f396de in_seq=0 out_seq=0 2024-03-21T10:41:12.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.450+0000 7efd8f400640 10 client.26305 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:12.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.451+0000 7efd8f400640 1 -- 172.21.15.31:0/3174409690 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f37da9d3f0 con 0x55f37db44540 2024-03-21T10:41:12.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.451+0000 7efd8f400640 10 client.26305 handle_command_reply: tid=0 2024-03-21T10:41:12.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.451+0000 7efdb787f740 1 client.26305 shutdown 2024-03-21T10:41:12.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.451+0000 7efdb787f740 1 -- 172.21.15.31:0/3174409690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f37db2b080 msgr2=0x55f37db33fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:12.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.451+0000 7efdb787f740 1 --2- 172.21.15.31:0/3174409690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f37db2b080 0x55f37db33fa0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7efda80049d0 tx=0x7efda8005bd0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.451+0000 7efdb787f740 1 -- 172.21.15.31:0/3174409690 shutdown_connections 2024-03-21T10:41:12.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.451+0000 7efdb787f740 1 --2- 172.21.15.31:0/3174409690 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f37db44540 0x55f37db46930 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.451+0000 7efdb787f740 1 --2- 172.21.15.31:0/3174409690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f37db34580 0x55f37db31940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.451+0000 7efdb787f740 1 --2- 172.21.15.31:0/3174409690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f37db2b9a0 0x55f37db31330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.451+0000 7efd8f400640 0 client.26305 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:12.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.451+0000 7efdb787f740 1 --2- 172.21.15.31:0/3174409690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f37db2b080 0x55f37db33fa0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.451+0000 7efdb787f740 1 -- 172.21.15.31:0/3174409690 >> 172.21.15.31:0/3174409690 conn(0x55f37db257b0 msgr2=0x55f37db27500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:12.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.452+0000 7efdb787f740 1 -- 172.21.15.31:0/3174409690 shutdown_connections 2024-03-21T10:41:12.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.453+0000 7efdb787f740 1 -- 172.21.15.31:0/3174409690 wait complete. 2024-03-21T10:41:12.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.453+0000 7efdb787f740 20 client.26305 trim_cache size 0 max 16384 2024-03-21T10:41:12.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.495+0000 7efdb787f740 1 Processor -- start 2024-03-21T10:41:12.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.495+0000 7efdb787f740 1 -- start start 2024-03-21T10:41:12.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.496+0000 7efdb787f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f37db2b080 0x55f37db33a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:12.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.496+0000 7efdb787f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f37db2b9a0 0x55f37db33fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:12.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.496+0000 7efdb787f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f37db34580 0x55f37db246b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:12.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.496+0000 7efdb787f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb01bfc00 con 0x55f37db2b080 2024-03-21T10:41:12.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.496+0000 7efdb787f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb01bd700 con 0x55f37db34580 2024-03-21T10:41:12.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.496+0000 7efdb787f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb0081d70 con 0x55f37db2b9a0 2024-03-21T10:41:12.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.496+0000 7efdae800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f37db2b080 0x55f37db33a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:12.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.496+0000 7efdae800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f37db2b080 0x55f37db33a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59100/0 (socket says 172.21.15.31:59100) 2024-03-21T10:41:12.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.497+0000 7efdae800640 1 -- 172.21.15.31:0/1125757491 learned_addr learned my addr 172.21.15.31:0/1125757491 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:12.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.497+0000 7efdae800640 1 --2- 172.21.15.31:0/1125757491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f37db34580 0x55f37db246b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:12.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.498+0000 7efdad400640 1 --2- 172.21.15.31:0/1125757491 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f37db2b9a0 0x55f37db33fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:12.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.499+0000 7efdae800640 1 -- 172.21.15.31:0/1125757491 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f37db2b9a0 msgr2=0x55f37db33fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:12.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.499+0000 7efdae800640 1 --2- 172.21.15.31:0/1125757491 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f37db2b9a0 0x55f37db33fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.500+0000 7efdae800640 1 -- 172.21.15.31:0/1125757491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f37db2b080 msgr2=0x55f37db33a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:12.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.500+0000 7efdae800640 1 --2- 172.21.15.31:0/1125757491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f37db2b080 0x55f37db33a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.500+0000 7efdae800640 1 -- 172.21.15.31:0/1125757491 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f37db20c20 con 0x55f37db34580 2024-03-21T10:41:12.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.500+0000 7efdad400640 1 --2- 172.21.15.31:0/1125757491 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f37db2b9a0 0x55f37db33fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:41:12.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.501+0000 7efdae800640 1 --2- 172.21.15.31:0/1125757491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f37db34580 0x55f37db246b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7efda8002230 tx=0x7efda8005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6772912b87960d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:12.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.503+0000 7efd9a600640 1 -- 172.21.15.31:0/1125757491 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda8004790 con 0x55f37db34580 2024-03-21T10:41:12.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.503+0000 7efd9a600640 1 -- 172.21.15.31:0/1125757491 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efda802d070 con 0x55f37db34580 2024-03-21T10:41:12.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.503+0000 7efd9a600640 1 -- 172.21.15.31:0/1125757491 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda8028520 con 0x55f37db34580 2024-03-21T10:41:12.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.504+0000 7efdb787f740 1 -- 172.21.15.31:0/1125757491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f37db34580 msgr2=0x55f37db246b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:12.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.504+0000 7efdb787f740 1 --2- 172.21.15.31:0/1125757491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f37db34580 0x55f37db246b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7efda8002230 tx=0x7efda8005bd0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.508+0000 7efdb787f740 1 -- 172.21.15.31:0/1125757491 shutdown_connections 2024-03-21T10:41:12.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.508+0000 7efdb787f740 1 --2- 172.21.15.31:0/1125757491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f37db34580 0x55f37db246b0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.509+0000 7efdb787f740 1 --2- 172.21.15.31:0/1125757491 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f37db2b9a0 0x55f37db33fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.509+0000 7efdb787f740 1 --2- 172.21.15.31:0/1125757491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f37db2b080 0x55f37db33a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.509+0000 7efdb787f740 1 -- 172.21.15.31:0/1125757491 >> 172.21.15.31:0/1125757491 conn(0x55f37db257b0 msgr2=0x55f37db23fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:12.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.513+0000 7efdb787f740 1 -- 172.21.15.31:0/1125757491 shutdown_connections 2024-03-21T10:41:12.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.513+0000 7efdb787f740 1 -- 172.21.15.31:0/1125757491 wait complete. 2024-03-21T10:41:12.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.514+0000 7efdb787f740 1 -- start start 2024-03-21T10:41:12.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.514+0000 7efdb787f740 1 Processor -- start 2024-03-21T10:41:12.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.516+0000 7efdb787f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f37db2b080 0x55f37db2efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:12.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.516+0000 7efdae800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f37db2b080 0x55f37db2efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:12.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.516+0000 7efdae800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f37db2b080 0x55f37db2efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:52936/0 (socket says 172.21.15.31:52936) 2024-03-21T10:41:12.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.516+0000 7efdae800640 1 -- 172.21.15.31:0/2505888554 learned_addr learned my addr 172.21.15.31:0/2505888554 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:12.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.519+0000 7efdb787f740 1 --2- 172.21.15.31:0/2505888554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f37db2b9a0 0x55f37db3b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:12.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.519+0000 7efdb787f740 1 --2- 172.21.15.31:0/2505888554 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f37db34580 0x55f37db3bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:12.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.519+0000 7efdb787f740 1 -- 172.21.15.31:0/2505888554 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb0082370 con 0x55f37db34580 2024-03-21T10:41:12.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.519+0000 7efdb787f740 1 -- 172.21.15.31:0/2505888554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb01bd0a0 con 0x55f37db2b9a0 2024-03-21T10:41:12.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.519+0000 7efdb787f740 1 -- 172.21.15.31:0/2505888554 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb01bc3e0 con 0x55f37db2b080 2024-03-21T10:41:12.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.520+0000 7efdb787f740 10 client.0 _ll_register_callbacks cb 0x7efda8009e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:12.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.520+0000 7efdad400640 1 --2- 172.21.15.31:0/2505888554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f37db2b9a0 0x55f37db3b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:12.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.520+0000 7efdae800640 1 -- 172.21.15.31:0/2505888554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f37db2b9a0 msgr2=0x55f37db3b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:12.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.520+0000 7efdae800640 1 --2- 172.21.15.31:0/2505888554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f37db2b9a0 0x55f37db3b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.520+0000 7efdae800640 1 -- 172.21.15.31:0/2505888554 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f37db34580 msgr2=0x55f37db3bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:12.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.520+0000 7efdae800640 1 --2- 172.21.15.31:0/2505888554 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f37db34580 0x55f37db3bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.520+0000 7efdae800640 1 -- 172.21.15.31:0/2505888554 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f37db3c450 con 0x55f37db2b080 2024-03-21T10:41:12.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.520+0000 7efdae800640 1 --2- 172.21.15.31:0/2505888554 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f37db2b080 0x55f37db2efd0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7efda8000f80 tx=0x7efda8009fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8938664386dc59b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:12.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.521+0000 7efdad400640 1 --2- 172.21.15.31:0/2505888554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f37db2b9a0 0x55f37db3b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:41:12.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.521+0000 7efd8f400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:41:12.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.521+0000 7efd8f400640 1 -- 172.21.15.31:0/2505888554 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda8004020 con 0x55f37db2b080 2024-03-21T10:41:12.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.522+0000 7efd8f400640 1 -- 172.21.15.31:0/2505888554 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7efda800fad0 con 0x55f37db2b080 2024-03-21T10:41:12.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.522+0000 7efdb787f740 1 -- 172.21.15.31:0/2505888554 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efda802d070 con 0x55f37db2b080 2024-03-21T10:41:12.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.523+0000 7efd8f400640 1 -- 172.21.15.31:0/2505888554 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7efda800fcf0 con 0x55f37db2b080 2024-03-21T10:41:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.527+0000 7efdb787f740 10 client.26308 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.527+0000 7efdb787f740 1 -- 172.21.15.31:0/2505888554 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55f37dbb1a30 con 0x55f37db2b080 2024-03-21T10:41:12.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.527+0000 7efd8f400640 1 -- 172.21.15.31:0/2505888554 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efda802d070 con 0x55f37db2b080 2024-03-21T10:41:12.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.528+0000 7efd8f400640 1 -- 172.21.15.31:0/2505888554 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7efda8028bf0 con 0x55f37db2b080 2024-03-21T10:41:12.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.529+0000 7efdb787f740 10 client.26308 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:12.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.529+0000 7efdb787f740 10 client.26308 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:41:12.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.529+0000 7efdb787f740 20 client.26308 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:12.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.529+0000 7efdb787f740 1 --2- 172.21.15.31:0/2505888554 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f37dbb1f20 0x55f37dbb4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:12.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.529+0000 7efdb787f740 4 client.26308 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-21T10:41:12.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.529+0000 7efdb787f740 1 -- 172.21.15.31:0/2505888554 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f37da97740 con 0x55f37dbb1f20 2024-03-21T10:41:12.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.530+0000 7efdad400640 1 --2- 172.21.15.31:0/2505888554 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f37dbb1f20 0x55f37dbb4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:12.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.531+0000 7efdad400640 1 --2- 172.21.15.31:0/2505888554 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f37dbb1f20 0x55f37dbb4340 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4060a15b0e817df0 server_cookie=24aff5f625baa40a in_seq=0 out_seq=0 2024-03-21T10:41:12.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.531+0000 7efd8f400640 10 client.26308 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:12.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.532+0000 7efd8f400640 1 -- 172.21.15.31:0/2505888554 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f37da97740 con 0x55f37dbb1f20 2024-03-21T10:41:12.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.533+0000 7efd8f400640 10 client.26308 handle_command_reply: tid=0 2024-03-21T10:41:12.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.533+0000 7efdb787f740 1 client.26308 shutdown 2024-03-21T10:41:12.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.537+0000 7efdb787f740 1 -- 172.21.15.31:0/2505888554 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f37db2b080 msgr2=0x55f37db2efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:12.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.537+0000 7efdb787f740 1 --2- 172.21.15.31:0/2505888554 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f37db2b080 0x55f37db2efd0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7efda8000f80 tx=0x7efda8009fb0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.543+0000 7efdb787f740 1 -- 172.21.15.31:0/2505888554 shutdown_connections 2024-03-21T10:41:12.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.543+0000 7efdb787f740 1 --2- 172.21.15.31:0/2505888554 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f37dbb1f20 0x55f37dbb4340 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.543+0000 7efdb787f740 1 --2- 172.21.15.31:0/2505888554 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f37db34580 0x55f37db3bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.543+0000 7efdb787f740 1 --2- 172.21.15.31:0/2505888554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f37db2b9a0 0x55f37db3b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.543+0000 7efdb787f740 1 --2- 172.21.15.31:0/2505888554 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f37db2b080 0x55f37db2efd0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.543+0000 7efdb787f740 1 -- 172.21.15.31:0/2505888554 >> 172.21.15.31:0/2505888554 conn(0x55f37db257b0 msgr2=0x55f37db24830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.543+0000 7efd8f400640 0 client.26308 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.548+0000 7efdb787f740 1 -- 172.21.15.31:0/2505888554 shutdown_connections 2024-03-21T10:41:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.548+0000 7efdb787f740 1 -- 172.21.15.31:0/2505888554 wait complete. 2024-03-21T10:41:12.546 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:41:12.546 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-03-21T10:41:12.546 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "5d3ca942-f766-4b0f-94f2-7ccad116396b", 2024-03-21T10:41:12.546 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-03-21T10:41:12.547 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:41:12.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.548+0000 7efdb787f740 20 client.26308 trim_cache size 0 max 16384 2024-03-21T10:41:12.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.557+0000 7efdb4c00640 1 -- 172.21.15.31:0/2337110732 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7efd940a3110 msgr2=0x7efd940a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:12.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.557+0000 7efdb4c00640 1 --2- 172.21.15.31:0/2337110732 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7efd940a3110 0x7efd940a55d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efda00cd450 tx=0x7efda0003e10 comp rx=0 tx=0).stop 2024-03-21T10:41:12.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.557+0000 7efdb4c00640 1 -- 172.21.15.31:0/2337110732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdb0107dc0 msgr2=0x7efdb019e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:12.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.557+0000 7efdb4c00640 1 --2- 172.21.15.31:0/2337110732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdb0107dc0 0x7efdb019e600 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7efda000ea90 tx=0x7efda000ef50 comp rx=0 tx=0).stop 2024-03-21T10:41:12.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.558+0000 7efdb4c00640 1 -- 172.21.15.31:0/2337110732 shutdown_connections 2024-03-21T10:41:12.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.558+0000 7efdb4c00640 1 --2- 172.21.15.31:0/2337110732 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7efd940a3110 0x7efd940a55d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.558+0000 7efdb4c00640 1 --2- 172.21.15.31:0/2337110732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdb0109b70 0x7efdb01a2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.558+0000 7efdb4c00640 1 --2- 172.21.15.31:0/2337110732 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdb0108760 0x7efdb019eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.558+0000 7efdb4c00640 1 --2- 172.21.15.31:0/2337110732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdb0107dc0 0x7efdb019e600 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.558+0000 7efdb4c00640 1 -- 172.21.15.31:0/2337110732 >> 172.21.15.31:0/2337110732 conn(0x7efdb0077540 msgr2=0x7efdb0077b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:12.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.562+0000 7efdb4c00640 1 -- 172.21.15.31:0/2337110732 shutdown_connections 2024-03-21T10:41:12.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.563+0000 7efdb4c00640 1 -- 172.21.15.31:0/2337110732 wait complete. 2024-03-21T10:41:12.605 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:41:12.605 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:41:12.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.999+0000 7f9b96a00640 1 -- 172.21.15.31:0/2927109816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b90100fd0 msgr2=0x7f9b90105490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:12.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.999+0000 7f9b96a00640 1 --2- 172.21.15.31:0/2927109816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b90100fd0 0x7f9b90105490 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c009890 tx=0x7f9b7c02bc60 comp rx=0 tx=0).stop 2024-03-21T10:41:12.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:12.999+0000 7f9b96a00640 1 -- 172.21.15.31:0/2927109816 shutdown_connections 2024-03-21T10:41:12.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.000+0000 7f9b96a00640 1 --2- 172.21.15.31:0/2927109816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b90105ab0 0x7f9b90105f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.000+0000 7f9b96a00640 1 --2- 172.21.15.31:0/2927109816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b90100fd0 0x7f9b90105490 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.000+0000 7f9b96a00640 1 --2- 172.21.15.31:0/2927109816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b901006b0 0x7f9b90100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:12.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.000+0000 7f9b96a00640 1 -- 172.21.15.31:0/2927109816 >> 172.21.15.31:0/2927109816 conn(0x7f9b900776f0 msgr2=0x7f9b90077b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:12.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.000+0000 7f9b96a00640 1 -- 172.21.15.31:0/2927109816 shutdown_connections 2024-03-21T10:41:13.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.001+0000 7f9b96a00640 1 -- 172.21.15.31:0/2927109816 wait complete. 2024-03-21T10:41:13.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.002+0000 7f9b96a00640 1 Processor -- start 2024-03-21T10:41:13.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.003+0000 7f9b96a00640 1 -- start start 2024-03-21T10:41:13.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.004+0000 7f9b96a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b901006b0 0x7f9b90072f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:13.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.004+0000 7f9b96a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b90100fd0 0x7f9b9006bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:13.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.005+0000 7f9b96a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b90105ab0 0x7f9b9006c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:13.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.005+0000 7f9b96a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b90118060 con 0x7f9b901006b0 2024-03-21T10:41:13.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.005+0000 7f9b96a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b90117ee0 con 0x7f9b90105ab0 2024-03-21T10:41:13.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.005+0000 7f9b96a00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b901181e0 con 0x7f9b90100fd0 2024-03-21T10:41:13.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.005+0000 7f9b8fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b901006b0 0x7f9b90072f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:13.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.005+0000 7f9b8fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b901006b0 0x7f9b90072f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59120/0 (socket says 172.21.15.31:59120) 2024-03-21T10:41:13.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.005+0000 7f9b8fe00640 1 -- 172.21.15.31:0/1447535520 learned_addr learned my addr 172.21.15.31:0/1447535520 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:13.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.006+0000 7f9b8f400640 1 --2- 172.21.15.31:0/1447535520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b90100fd0 0x7f9b9006bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:13.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.006+0000 7f9b8fe00640 1 -- 172.21.15.31:0/1447535520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b90100fd0 msgr2=0x7f9b9006bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:13.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.006+0000 7f9b8fe00640 1 --2- 172.21.15.31:0/1447535520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b90100fd0 0x7f9b9006bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.006+0000 7f9b8fe00640 1 -- 172.21.15.31:0/1447535520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b90105ab0 msgr2=0x7f9b9006c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:13.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.006+0000 7f9b8fe00640 1 --2- 172.21.15.31:0/1447535520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b90105ab0 0x7f9b9006c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.006+0000 7f9b8fe00640 1 -- 172.21.15.31:0/1447535520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9006cad0 con 0x7f9b901006b0 2024-03-21T10:41:13.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.007+0000 7f9b8f400640 1 --2- 172.21.15.31:0/1447535520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b90100fd0 0x7f9b9006bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:41:13.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.007+0000 7f9b8fe00640 1 --2- 172.21.15.31:0/1447535520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b901006b0 0x7f9b90072f00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9b8000e9f0 tx=0x7f9b8000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:13.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.007+0000 7f9b8cc00640 1 -- 172.21.15.31:0/1447535520 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8000cd80 con 0x7f9b901006b0 2024-03-21T10:41:13.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.008+0000 7f9b8cc00640 1 -- 172.21.15.31:0/1447535520 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b80010910 con 0x7f9b901006b0 2024-03-21T10:41:13.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.008+0000 7f9b8cc00640 1 -- 172.21.15.31:0/1447535520 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b800188f0 con 0x7f9b901006b0 2024-03-21T10:41:13.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.008+0000 7f9b96a00640 1 -- 172.21.15.31:0/1447535520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b9006cdc0 con 0x7f9b901006b0 2024-03-21T10:41:13.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.008+0000 7f9b96a00640 1 -- 172.21.15.31:0/1447535520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b90070d50 con 0x7f9b901006b0 2024-03-21T10:41:13.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.012+0000 7f9b99786740 1 Processor -- start 2024-03-21T10:41:13.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.013+0000 7f9b99786740 1 -- start start 2024-03-21T10:41:13.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.013+0000 7f9b8cc00640 1 -- 172.21.15.31:0/1447535520 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f9b80010ac0 con 0x7f9b901006b0 2024-03-21T10:41:13.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.014+0000 7f9b99786740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f913aec00 0x557f913ad950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:13.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.014+0000 7f9b99786740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557f913adf20 0x557f913ae3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:13.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.014+0000 7f9b99786740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f913b7290 0x557f913b9680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:13.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.014+0000 7f9b99786740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f91312250 con 0x557f913adf20 2024-03-21T10:41:13.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.014+0000 7f9b99786740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f912baa30 con 0x557f913b7290 2024-03-21T10:41:13.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.014+0000 7f9b99786740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9130e2f0 con 0x557f913aec00 2024-03-21T10:41:13.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.014+0000 7f9b94a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f913aec00 0x557f913ad950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:13.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.014+0000 7f9b94a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f913aec00 0x557f913ad950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:52948/0 (socket says 172.21.15.31:52948) 2024-03-21T10:41:13.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.014+0000 7f9b94a00640 1 -- 172.21.15.31:0/2066544509 learned_addr learned my addr 172.21.15.31:0/2066544509 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:13.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.015+0000 7f9b8f400640 1 --2- 172.21.15.31:0/2066544509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f913b7290 0x557f913b9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:13.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.015+0000 7f9b94a00640 1 -- 172.21.15.31:0/2066544509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f913b7290 msgr2=0x557f913b9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:13.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.015+0000 7f9b94a00640 1 --2- 172.21.15.31:0/2066544509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f913b7290 0x557f913b9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.015+0000 7f9b94a00640 1 -- 172.21.15.31:0/2066544509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557f913adf20 msgr2=0x557f913ae3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:13.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.015+0000 7f9b94a00640 1 --2- 172.21.15.31:0/2066544509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557f913adf20 0x557f913ae3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.015+0000 7f9b94a00640 1 -- 172.21.15.31:0/2066544509 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557f912a5690 con 0x557f913aec00 2024-03-21T10:41:13.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.016+0000 7f9b8cc00640 1 --2- 172.21.15.31:0/1447535520 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9b740a30d0 0x7f9b740a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:13.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.016+0000 7f9b8cc00640 1 -- 172.21.15.31:0/1447535520 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f9b800d0040 con 0x7f9b901006b0 2024-03-21T10:41:13.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.016+0000 7f9b94a00640 1 --2- 172.21.15.31:0/2066544509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f913aec00 0x557f913ad950 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9b88009d00 tx=0x7f9b8800e760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23b8644eaea1d221 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:13.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.016+0000 7f9b85200640 1 -- 172.21.15.31:0/2066544509 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8800f600 con 0x557f913aec00 2024-03-21T10:41:13.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.017+0000 7f9b85200640 1 -- 172.21.15.31:0/2066544509 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b8800fc80 con 0x557f913aec00 2024-03-21T10:41:13.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.017+0000 7f9b8fe00640 1 --2- 172.21.15.31:0/1447535520 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9b740a30d0 0x7f9b740a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:13.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.017+0000 7f9b85200640 1 -- 172.21.15.31:0/2066544509 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8800c3f0 con 0x557f913aec00 2024-03-21T10:41:13.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.018+0000 7f9b99786740 1 -- 172.21.15.31:0/2066544509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f913aec00 msgr2=0x557f913ad950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:13.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.018+0000 7f9b99786740 1 --2- 172.21.15.31:0/2066544509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f913aec00 0x557f913ad950 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9b88009d00 tx=0x7f9b8800e760 comp rx=0 tx=0).stop 2024-03-21T10:41:13.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.018+0000 7f9b99786740 1 -- 172.21.15.31:0/2066544509 shutdown_connections 2024-03-21T10:41:13.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.018+0000 7f9b99786740 1 --2- 172.21.15.31:0/2066544509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f913b7290 0x557f913b9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.018+0000 7f9b99786740 1 --2- 172.21.15.31:0/2066544509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557f913adf20 0x557f913ae3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.018+0000 7f9b99786740 1 --2- 172.21.15.31:0/2066544509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f913aec00 0x557f913ad950 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.018+0000 7f9b99786740 1 -- 172.21.15.31:0/2066544509 >> 172.21.15.31:0/2066544509 conn(0x557f91375200 msgr2=0x557f913a8970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:13.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.018+0000 7f9b8fe00640 1 --2- 172.21.15.31:0/1447535520 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9b740a30d0 0x7f9b740a5590 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x557f913af150 tx=0x7f9b8000cf20 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:13.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.019+0000 7f9b99786740 1 -- 172.21.15.31:0/2066544509 shutdown_connections 2024-03-21T10:41:13.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.019+0000 7f9b99786740 1 -- 172.21.15.31:0/2066544509 wait complete. 2024-03-21T10:41:13.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.020+0000 7f9b99786740 1 -- start start 2024-03-21T10:41:13.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.020+0000 7f9b99786740 1 Processor -- start 2024-03-21T10:41:13.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.021+0000 7f9b99786740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f913adf20 0x557f913b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:13.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.022+0000 7f9b99786740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f913aec00 0x557f913b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:13.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.022+0000 7f9b99786740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557f913b7290 0x557f913b4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:13.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.022+0000 7f9b99786740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f913ba1a0 con 0x557f913b7290 2024-03-21T10:41:13.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.022+0000 7f9b99786740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f912cde80 con 0x557f913aec00 2024-03-21T10:41:13.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.022+0000 7f9b99786740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f913ba320 con 0x557f913adf20 2024-03-21T10:41:13.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.022+0000 7f9b99786740 10 client.0 _ll_register_callbacks cb 0x557f912e0130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:13.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.022+0000 7f9b94a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f913adf20 0x557f913b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:13.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.022+0000 7f9b94a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f913adf20 0x557f913b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:52962/0 (socket says 172.21.15.31:52962) 2024-03-21T10:41:13.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.022+0000 7f9b94a00640 1 -- 172.21.15.31:0/2714600779 learned_addr learned my addr 172.21.15.31:0/2714600779 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:13.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.022+0000 7f9b8f400640 1 --2- 172.21.15.31:0/2714600779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f913aec00 0x557f913b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:13.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.024+0000 7f9b94a00640 1 --2- 172.21.15.31:0/2714600779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557f913b7290 0x557f913b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:13.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.024+0000 7f9b94a00640 1 -- 172.21.15.31:0/2714600779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f913aec00 msgr2=0x557f913b3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:13.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.024+0000 7f9b94a00640 1 --2- 172.21.15.31:0/2714600779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f913aec00 0x557f913b3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.024+0000 7f9b94a00640 1 -- 172.21.15.31:0/2714600779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557f913b7290 msgr2=0x557f913b4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:13.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.024+0000 7f9b94a00640 1 --2- 172.21.15.31:0/2714600779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557f913b7290 0x557f913b4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.024+0000 7f9b94a00640 1 -- 172.21.15.31:0/2714600779 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557f910f07d0 con 0x557f913adf20 2024-03-21T10:41:13.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.025+0000 7f9b94a00640 1 --2- 172.21.15.31:0/2714600779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f913adf20 0x557f913b68a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9b88017af0 tx=0x7f9b88005da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bcbaf9283ba7f13 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:13.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.025+0000 7f9b79600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:41:13.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.026+0000 7f9b79600640 1 -- 172.21.15.31:0/2714600779 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b88029020 con 0x557f913adf20 2024-03-21T10:41:13.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.026+0000 7f9b79600640 1 -- 172.21.15.31:0/2714600779 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f9b88004b20 con 0x557f913adf20 2024-03-21T10:41:13.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.026+0000 7f9b99786740 1 -- 172.21.15.31:0/2714600779 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9b8800fc80 con 0x557f913adf20 2024-03-21T10:41:13.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.026+0000 7f9b79600640 1 -- 172.21.15.31:0/2714600779 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b88004d20 con 0x557f913adf20 2024-03-21T10:41:13.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.026+0000 7f9b99786740 10 client.26317 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:13.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.026+0000 7f9b99786740 1 -- 172.21.15.31:0/2714600779 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557f910263f0 con 0x557f913adf20 2024-03-21T10:41:13.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.027+0000 7f9b79600640 1 -- 172.21.15.31:0/2714600779 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b8800fc80 con 0x557f913adf20 2024-03-21T10:41:13.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.027+0000 7f9b79600640 1 -- 172.21.15.31:0/2714600779 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f9b8800b030 con 0x557f913adf20 2024-03-21T10:41:13.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.027+0000 7f9b99786740 10 client.26317 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:13.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.027+0000 7f9b99786740 10 client.26317 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:41:13.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.027+0000 7f9b99786740 20 client.26317 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:13.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.027+0000 7f9b99786740 1 --2- 172.21.15.31:0/2714600779 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x557f913c70b0 0x557f913c94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:13.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.027+0000 7f9b99786740 4 client.26317 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:41:13.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.027+0000 7f9b99786740 1 -- 172.21.15.31:0/2714600779 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557f9131f790 con 0x557f913c70b0 2024-03-21T10:41:13.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.027+0000 7f9b8f400640 1 --2- 172.21.15.31:0/2714600779 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x557f913c70b0 0x557f913c94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:13.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.028+0000 7f9b8f400640 1 --2- 172.21.15.31:0/2714600779 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x557f913c70b0 0x557f913c94a0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56f58d0d4a2a0e33 server_cookie=ee013306e9791b4b in_seq=0 out_seq=0 2024-03-21T10:41:13.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.028+0000 7f9b79600640 10 client.26317 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.029+0000 7f9b79600640 1 -- 172.21.15.31:0/2714600779 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557f9131f790 con 0x557f913c70b0 2024-03-21T10:41:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.029+0000 7f9b79600640 10 client.26317 handle_command_reply: tid=0 2024-03-21T10:41:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.030+0000 7f9b99786740 1 client.26317 shutdown 2024-03-21T10:41:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.030+0000 7f9b99786740 1 -- 172.21.15.31:0/2714600779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f913adf20 msgr2=0x557f913b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.030+0000 7f9b99786740 1 --2- 172.21.15.31:0/2714600779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f913adf20 0x557f913b68a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9b88017af0 tx=0x7f9b88005da0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.030+0000 7f9b99786740 1 -- 172.21.15.31:0/2714600779 shutdown_connections 2024-03-21T10:41:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.030+0000 7f9b99786740 1 --2- 172.21.15.31:0/2714600779 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x557f913c70b0 0x557f913c94a0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.030+0000 7f9b99786740 1 --2- 172.21.15.31:0/2714600779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557f913b7290 0x557f913b4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.030+0000 7f9b99786740 1 --2- 172.21.15.31:0/2714600779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f913aec00 0x557f913b3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.030+0000 7f9b99786740 1 --2- 172.21.15.31:0/2714600779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f913adf20 0x557f913b68a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.030+0000 7f9b99786740 1 -- 172.21.15.31:0/2714600779 >> 172.21.15.31:0/2714600779 conn(0x557f91375200 msgr2=0x557f913aa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.030+0000 7f9b79600640 0 client.26317 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.030+0000 7f9b99786740 1 -- 172.21.15.31:0/2714600779 shutdown_connections 2024-03-21T10:41:13.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.031+0000 7f9b99786740 1 -- 172.21.15.31:0/2714600779 wait complete. 2024-03-21T10:41:13.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.031+0000 7f9b99786740 20 client.26317 trim_cache size 0 max 16384 2024-03-21T10:41:13.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.058+0000 7f9b99786740 1 Processor -- start 2024-03-21T10:41:13.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.058+0000 7f9b99786740 1 -- start start 2024-03-21T10:41:13.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:12 smithi031 ceph-mon[31673]: pgmap v82: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 198 MiB/s wr, 1.09k op/s 2024-03-21T10:41:13.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.058+0000 7f9b99786740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f913adf20 0x557f913aa460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:13.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.058+0000 7f9b99786740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f913aec00 0x557f913b6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:13.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.059+0000 7f9b99786740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557f913b7290 0x557f913b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:13.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.059+0000 7f9b99786740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b901bfec0 con 0x557f913b7290 2024-03-21T10:41:13.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.059+0000 7f9b99786740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b901bd9c0 con 0x557f913aec00 2024-03-21T10:41:13.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.059+0000 7f9b99786740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b901bba60 con 0x557f913adf20 2024-03-21T10:41:13.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.059+0000 7f9b94a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f913adf20 0x557f913aa460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:13.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.059+0000 7f9b94a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f913adf20 0x557f913aa460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:52972/0 (socket says 172.21.15.31:52972) 2024-03-21T10:41:13.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.059+0000 7f9b94a00640 1 -- 172.21.15.31:0/1436040957 learned_addr learned my addr 172.21.15.31:0/1436040957 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:13.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.059+0000 7f9b94a00640 1 --2- 172.21.15.31:0/1436040957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557f913b7290 0x557f913b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:13.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.060+0000 7f9b8f400640 1 --2- 172.21.15.31:0/1436040957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f913aec00 0x557f913b6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:13.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.060+0000 7f9b8f400640 1 -- 172.21.15.31:0/1436040957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f913adf20 msgr2=0x557f913aa460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:13.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.060+0000 7f9b8f400640 1 --2- 172.21.15.31:0/1436040957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f913adf20 0x557f913aa460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.060+0000 7f9b8f400640 1 -- 172.21.15.31:0/1436040957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557f913b7290 msgr2=0x557f913b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:13.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.060+0000 7f9b8f400640 1 --2- 172.21.15.31:0/1436040957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557f913b7290 0x557f913b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.060+0000 7f9b8f400640 1 -- 172.21.15.31:0/1436040957 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557f90fe60f0 con 0x557f913aec00 2024-03-21T10:41:13.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.061+0000 7f9b8f400640 1 --2- 172.21.15.31:0/1436040957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f913aec00 0x557f913b6360 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f9b9006d7d0 tx=0x7f9b7c02c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1c56250446114d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:13.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.061+0000 7f9b7be00640 1 -- 172.21.15.31:0/1436040957 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c039040 con 0x557f913aec00 2024-03-21T10:41:13.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.061+0000 7f9b7be00640 1 -- 172.21.15.31:0/1436040957 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b7c03a040 con 0x557f913aec00 2024-03-21T10:41:13.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.063+0000 7f9b7be00640 1 -- 172.21.15.31:0/1436040957 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c0052c0 con 0x557f913aec00 2024-03-21T10:41:13.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.064+0000 7f9b99786740 1 -- 172.21.15.31:0/1436040957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f913aec00 msgr2=0x557f913b6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:13.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.064+0000 7f9b99786740 1 --2- 172.21.15.31:0/1436040957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f913aec00 0x557f913b6360 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f9b9006d7d0 tx=0x7f9b7c02c400 comp rx=0 tx=0).stop 2024-03-21T10:41:13.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.071+0000 7f9b99786740 1 -- 172.21.15.31:0/1436040957 shutdown_connections 2024-03-21T10:41:13.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.071+0000 7f9b99786740 1 --2- 172.21.15.31:0/1436040957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557f913b7290 0x557f913b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.071+0000 7f9b99786740 1 --2- 172.21.15.31:0/1436040957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f913aec00 0x557f913b6360 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.071+0000 7f9b99786740 1 --2- 172.21.15.31:0/1436040957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f913adf20 0x557f913aa460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.071+0000 7f9b99786740 1 -- 172.21.15.31:0/1436040957 >> 172.21.15.31:0/1436040957 conn(0x557f91375200 msgr2=0x557f913a9d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:13.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:12 smithi070 ceph-mon[36330]: pgmap v82: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 198 MiB/s wr, 1.09k op/s 2024-03-21T10:41:13.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.077+0000 7f9b99786740 1 -- 172.21.15.31:0/1436040957 shutdown_connections 2024-03-21T10:41:13.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.077+0000 7f9b99786740 1 -- 172.21.15.31:0/1436040957 wait complete. 2024-03-21T10:41:13.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.078+0000 7f9b99786740 1 -- start start 2024-03-21T10:41:13.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.079+0000 7f9b99786740 1 Processor -- start 2024-03-21T10:41:13.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.085+0000 7f9b99786740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f913adf20 0x557f9141bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:13.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.085+0000 7f9b99786740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f913aec00 0x557f9141c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:13.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.085+0000 7f9b94a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f913adf20 0x557f9141bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:13.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.086+0000 7f9b94a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f913adf20 0x557f9141bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60736/0 (socket says 172.21.15.31:60736) 2024-03-21T10:41:13.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.086+0000 7f9b99786740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557f913b7290 0x557f913b0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:13.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.086+0000 7f9b99786740 1 -- 172.21.15.31:0/1524738307 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b901bc060 con 0x557f913b7290 2024-03-21T10:41:13.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.086+0000 7f9b94a00640 1 -- 172.21.15.31:0/1524738307 learned_addr learned my addr 172.21.15.31:0/1524738307 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:13.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.086+0000 7f9b99786740 1 -- 172.21.15.31:0/1524738307 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b901bd360 con 0x557f913adf20 2024-03-21T10:41:13.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.086+0000 7f9b99786740 1 -- 172.21.15.31:0/1524738307 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b901bc6a0 con 0x557f913aec00 2024-03-21T10:41:13.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.086+0000 7f9b99786740 10 client.0 _ll_register_callbacks cb 0x7f9b88009b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:13.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.087+0000 7f9b8f400640 1 --2- 172.21.15.31:0/1524738307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f913aec00 0x557f9141c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:13.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.087+0000 7f9b94a00640 1 -- 172.21.15.31:0/1524738307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f913aec00 msgr2=0x557f9141c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:13.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.087+0000 7f9b94a00640 1 --2- 172.21.15.31:0/1524738307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f913aec00 0x557f9141c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.087+0000 7f9b94a00640 1 -- 172.21.15.31:0/1524738307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557f913b7290 msgr2=0x557f913b0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:13.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.087+0000 7f9b94a00640 1 --2- 172.21.15.31:0/1524738307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557f913b7290 0x557f913b0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.087+0000 7f9b94a00640 1 -- 172.21.15.31:0/1524738307 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557f913b11b0 con 0x557f913adf20 2024-03-21T10:41:13.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.089+0000 7f9b94a00640 1 --2- 172.21.15.31:0/1524738307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f913adf20 0x557f9141bae0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9b88005e30 tx=0x7f9b88009540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4291e849e846c62 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:13.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.089+0000 7f9b79600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:41:13.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.090+0000 7f9b79600640 1 -- 172.21.15.31:0/1524738307 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8803d020 con 0x557f913adf20 2024-03-21T10:41:13.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.091+0000 7f9b79600640 1 -- 172.21.15.31:0/1524738307 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f9b88004860 con 0x557f913adf20 2024-03-21T10:41:13.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.091+0000 7f9b99786740 1 -- 172.21.15.31:0/1524738307 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9b7c03a040 con 0x557f913adf20 2024-03-21T10:41:13.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.091+0000 7f9b79600640 1 -- 172.21.15.31:0/1524738307 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b88045050 con 0x557f913adf20 2024-03-21T10:41:13.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.091+0000 7f9b99786740 10 client.26372 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:13.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.093+0000 7f9b99786740 1 -- 172.21.15.31:0/1524738307 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557f91423200 con 0x557f913adf20 2024-03-21T10:41:13.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.094+0000 7f9b79600640 1 -- 172.21.15.31:0/1524738307 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b7c03a040 con 0x557f913adf20 2024-03-21T10:41:13.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.094+0000 7f9b79600640 1 -- 172.21.15.31:0/1524738307 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f9b88036040 con 0x557f913adf20 2024-03-21T10:41:13.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.095+0000 7f9b99786740 10 client.26372 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:13.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.095+0000 7f9b99786740 10 client.26372 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:41:13.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.095+0000 7f9b99786740 20 client.26372 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:13.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.095+0000 7f9b99786740 1 --2- 172.21.15.31:0/1524738307 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x557f91423870 0x557f91425c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:13.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.095+0000 7f9b99786740 4 client.26372 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:41:13.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.095+0000 7f9b99786740 1 -- 172.21.15.31:0/1524738307 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557f91319ae0 con 0x557f91423870 2024-03-21T10:41:13.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.099+0000 7f9b8f400640 1 --2- 172.21.15.31:0/1524738307 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x557f91423870 0x557f91425c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:13.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.100+0000 7f9b8f400640 1 --2- 172.21.15.31:0/1524738307 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x557f91423870 0x557f91425c90 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4184f471f9b9928 server_cookie=ef1698f82cd2e711 in_seq=0 out_seq=0 2024-03-21T10:41:13.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.100+0000 7f9b79600640 10 client.26372 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:13.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.100+0000 7f9b79600640 1 -- 172.21.15.31:0/1524738307 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x557f91319ae0 con 0x557f91423870 2024-03-21T10:41:13.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.100+0000 7f9b79600640 10 client.26372 handle_command_reply: tid=0 2024-03-21T10:41:13.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.100+0000 7f9b99786740 1 client.26372 shutdown 2024-03-21T10:41:13.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.100+0000 7f9b99786740 1 -- 172.21.15.31:0/1524738307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f913adf20 msgr2=0x557f9141bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:13.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.101+0000 7f9b99786740 1 --2- 172.21.15.31:0/1524738307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f913adf20 0x557f9141bae0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9b88005e30 tx=0x7f9b88009540 comp rx=0 tx=0).stop 2024-03-21T10:41:13.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.101+0000 7f9b99786740 1 -- 172.21.15.31:0/1524738307 shutdown_connections 2024-03-21T10:41:13.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.101+0000 7f9b99786740 1 --2- 172.21.15.31:0/1524738307 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x557f91423870 0x557f91425c90 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.101+0000 7f9b79600640 0 client.26372 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:13.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.101+0000 7f9b99786740 1 --2- 172.21.15.31:0/1524738307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557f913b7290 0x557f913b0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.101+0000 7f9b99786740 1 --2- 172.21.15.31:0/1524738307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f913aec00 0x557f9141c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.101+0000 7f9b99786740 1 --2- 172.21.15.31:0/1524738307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557f913adf20 0x557f9141bae0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.101+0000 7f9b99786740 1 -- 172.21.15.31:0/1524738307 >> 172.21.15.31:0/1524738307 conn(0x557f91375200 msgr2=0x557f913aa5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:13.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.101+0000 7f9b99786740 1 -- 172.21.15.31:0/1524738307 shutdown_connections 2024-03-21T10:41:13.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.101+0000 7f9b99786740 1 -- 172.21.15.31:0/1524738307 wait complete. 2024-03-21T10:41:13.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.101+0000 7f9b99786740 20 client.26372 trim_cache size 0 max 16384 2024-03-21T10:41:13.100 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:41:13.100 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-21T10:41:13.100 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-03-21T10:41:13.100 INFO:teuthology.orchestra.run.smithi031.stdout: "5d3ca942-f766-4b0f-94f2-7ccad116396b": { 2024-03-21T10:41:13.100 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-03-21T10:41:13.100 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "5d3ca942-f766-4b0f-94f2-7ccad116396b", 2024-03-21T10:41:13.100 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-03-21T10:41:13.100 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:41:13.100 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:41:13.100 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:41:13.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.103+0000 7f9b96a00640 1 -- 172.21.15.31:0/1447535520 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9b740a30d0 msgr2=0x7f9b740a5590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:13.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.103+0000 7f9b96a00640 1 --2- 172.21.15.31:0/1447535520 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9b740a30d0 0x7f9b740a5590 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x557f913af150 tx=0x7f9b8000cf20 comp rx=0 tx=0).stop 2024-03-21T10:41:13.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.103+0000 7f9b96a00640 1 -- 172.21.15.31:0/1447535520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b901006b0 msgr2=0x7f9b90072f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:13.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.103+0000 7f9b96a00640 1 --2- 172.21.15.31:0/1447535520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b901006b0 0x7f9b90072f00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9b8000e9f0 tx=0x7f9b8000eeb0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.103+0000 7f9b96a00640 1 -- 172.21.15.31:0/1447535520 shutdown_connections 2024-03-21T10:41:13.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.103+0000 7f9b96a00640 1 --2- 172.21.15.31:0/1447535520 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9b740a30d0 0x7f9b740a5590 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.103+0000 7f9b96a00640 1 --2- 172.21.15.31:0/1447535520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b90105ab0 0x7f9b9006c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.103+0000 7f9b96a00640 1 --2- 172.21.15.31:0/1447535520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b90100fd0 0x7f9b9006bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.103+0000 7f9b96a00640 1 --2- 172.21.15.31:0/1447535520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b901006b0 0x7f9b90072f00 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:13.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.103+0000 7f9b96a00640 1 -- 172.21.15.31:0/1447535520 >> 172.21.15.31:0/1447535520 conn(0x7f9b900776f0 msgr2=0x7f9b90103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:13.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.104+0000 7f9b96a00640 1 -- 172.21.15.31:0/1447535520 shutdown_connections 2024-03-21T10:41:13.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:13.104+0000 7f9b96a00640 1 -- 172.21.15.31:0/1447535520 wait complete. 2024-03-21T10:41:13.137 INFO:tasks.cephfs.filesystem:scrub status for tag:5d3ca942-f766-4b0f-94f2-7ccad116396b - {'path': '/', 'tag': '5d3ca942-f766-4b0f-94f2-7ccad116396b', 'options': 'recursive,force'} 2024-03-21T10:41:13.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:12 smithi040 ceph-mon[36154]: pgmap v82: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 198 MiB/s wr, 1.09k op/s 2024-03-21T10:41:14.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:13 smithi031 ceph-mon[31673]: scrub queued for path: / 2024-03-21T10:41:14.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:13 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:41:14.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:13 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:41:14.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:13 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:41:14.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:13 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:41:14.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:13 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:41:14.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:13 smithi070 ceph-mon[36330]: scrub queued for path: / 2024-03-21T10:41:14.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:13 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:41:14.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:13 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:41:14.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:13 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:41:14.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:13 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:41:14.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:13 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:41:14.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:13 smithi040 ceph-mon[36154]: scrub queued for path: / 2024-03-21T10:41:14.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:13 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:41:14.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:13 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:41:14.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:13 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:41:14.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:13 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:41:14.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:13 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:41:15.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:14 smithi031 ceph-mon[31673]: pgmap v83: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 189 MiB/s wr, 1.04k op/s 2024-03-21T10:41:15.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:14 smithi070 ceph-mon[36330]: pgmap v83: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 189 MiB/s wr, 1.04k op/s 2024-03-21T10:41:15.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:14 smithi040 ceph-mon[36154]: pgmap v83: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 189 MiB/s wr, 1.04k op/s 2024-03-21T10:41:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.627+0000 7fc0b9e00640 1 -- 172.21.15.31:0/3562607091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0b4103880 msgr2=0x7fc0b4103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.627+0000 7fc0b9e00640 1 --2- 172.21.15.31:0/3562607091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0b4103880 0x7fc0b4103d00 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0009900 tx=0x7fc0a002bc30 comp rx=0 tx=0).stop 2024-03-21T10:41:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.627+0000 7fc0b9e00640 1 -- 172.21.15.31:0/3562607091 shutdown_connections 2024-03-21T10:41:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.627+0000 7fc0b9e00640 1 --2- 172.21.15.31:0/3562607091 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0b4104240 0x7fc0b4110750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.627+0000 7fc0b9e00640 1 --2- 172.21.15.31:0/3562607091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0b4103880 0x7fc0b4103d00 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.627+0000 7fc0b9e00640 1 --2- 172.21.15.31:0/3562607091 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0b4109890 0x7fc0b4109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.627+0000 7fc0b9e00640 1 -- 172.21.15.31:0/3562607091 >> 172.21.15.31:0/3562607091 conn(0x7fc0b40ff390 msgr2=0x7fc0b41017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.627+0000 7fc0b9e00640 1 -- 172.21.15.31:0/3562607091 shutdown_connections 2024-03-21T10:41:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.627+0000 7fc0b9e00640 1 -- 172.21.15.31:0/3562607091 wait complete. 2024-03-21T10:41:15.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.628+0000 7fc0b9e00640 1 Processor -- start 2024-03-21T10:41:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.629+0000 7fc0b9e00640 1 -- start start 2024-03-21T10:41:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.629+0000 7fc0b9e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0b4103880 0x7fc0b419a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.629+0000 7fc0b9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0b4104240 0x7fc0b419a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.629+0000 7fc0b9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0b4109890 0x7fc0b419ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.629+0000 7fc0b9e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b4113cf0 con 0x7fc0b4109890 2024-03-21T10:41:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.629+0000 7fc0b9e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b4113b70 con 0x7fc0b4104240 2024-03-21T10:41:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.629+0000 7fc0b9e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b4113e70 con 0x7fc0b4103880 2024-03-21T10:41:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.629+0000 7fc0b2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0b4103880 0x7fc0b419a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.629+0000 7fc0b2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0b4103880 0x7fc0b419a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:52992/0 (socket says 172.21.15.31:52992) 2024-03-21T10:41:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.629+0000 7fc0b2a00640 1 -- 172.21.15.31:0/581733028 learned_addr learned my addr 172.21.15.31:0/581733028 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.629+0000 7fc0b2000640 1 --2- 172.21.15.31:0/581733028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0b4104240 0x7fc0b419a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.630+0000 7fc0b3400640 1 --2- 172.21.15.31:0/581733028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0b4109890 0x7fc0b419ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.630+0000 7fc0b2a00640 1 -- 172.21.15.31:0/581733028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0b4104240 msgr2=0x7fc0b419a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.630+0000 7fc0b2a00640 1 --2- 172.21.15.31:0/581733028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0b4104240 0x7fc0b419a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.630+0000 7fc0b2a00640 1 -- 172.21.15.31:0/581733028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0b4109890 msgr2=0x7fc0b419ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.630+0000 7fc0b2a00640 1 --2- 172.21.15.31:0/581733028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0b4109890 0x7fc0b419ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.630+0000 7fc0b2a00640 1 -- 172.21.15.31:0/581733028 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b419f160 con 0x7fc0b4103880 2024-03-21T10:41:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.630+0000 7fc0b2000640 1 --2- 172.21.15.31:0/581733028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0b4104240 0x7fc0b419a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:41:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.630+0000 7fc0b3400640 1 --2- 172.21.15.31:0/581733028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0b4109890 0x7fc0b419ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:41:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.631+0000 7fc0b2a00640 1 --2- 172.21.15.31:0/581733028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0b4103880 0x7fc0b419a170 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc0a400e9f0 tx=0x7fc0a400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.631+0000 7fc0b8a00640 1 -- 172.21.15.31:0/581733028 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a400cd80 con 0x7fc0b4103880 2024-03-21T10:41:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.631+0000 7fc0b8a00640 1 -- 172.21.15.31:0/581733028 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc0a4005500 con 0x7fc0b4103880 2024-03-21T10:41:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.631+0000 7fc0b9e00640 1 -- 172.21.15.31:0/581733028 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b419f450 con 0x7fc0b4103880 2024-03-21T10:41:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.631+0000 7fc0b8a00640 1 -- 172.21.15.31:0/581733028 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a4010690 con 0x7fc0b4103880 2024-03-21T10:41:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.631+0000 7fc0b9e00640 1 -- 172.21.15.31:0/581733028 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b41a6ce0 con 0x7fc0b4103880 2024-03-21T10:41:15.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.634+0000 7fc0b8a00640 1 -- 172.21.15.31:0/581733028 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fc0a40046c0 con 0x7fc0b4103880 2024-03-21T10:41:15.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.634+0000 7fc0bcbdb740 1 Processor -- start 2024-03-21T10:41:15.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.635+0000 7fc0bcbdb740 1 -- start start 2024-03-21T10:41:15.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.635+0000 7fc0bcbdb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56492bd15fd0 0x56492bd163b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:15.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.635+0000 7fc0bcbdb740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492bd168f0 0x56492bd1f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:15.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.635+0000 7fc0bcbdb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56492bd1f6c0 0x56492bd21ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:15.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.635+0000 7fc0bcbdb740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56492bcd83b0 con 0x56492bd168f0 2024-03-21T10:41:15.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.635+0000 7fc0bcbdb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56492bc7a830 con 0x56492bd1f6c0 2024-03-21T10:41:15.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.635+0000 7fc0bcbdb740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56492bc235d0 con 0x56492bd15fd0 2024-03-21T10:41:15.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.635+0000 7fc0b2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56492bd1f6c0 0x56492bd21ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:15.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.635+0000 7fc0b2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56492bd1f6c0 0x56492bd21ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60770/0 (socket says 172.21.15.31:60770) 2024-03-21T10:41:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.635+0000 7fc0b2000640 1 -- 172.21.15.31:0/1210193919 learned_addr learned my addr 172.21.15.31:0/1210193919 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.635+0000 7fc0b3400640 1 --2- 172.21.15.31:0/1210193919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56492bd15fd0 0x56492bd163b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.636+0000 7fc0b2a00640 1 --2- 172.21.15.31:0/1210193919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492bd168f0 0x56492bd1f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.636+0000 7fc0b8a00640 1 --2- 172.21.15.31:0/581733028 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc0980a3080 0x7fc0980a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.636+0000 7fc0b2000640 1 -- 172.21.15.31:0/1210193919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56492bd15fd0 msgr2=0x56492bd163b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.636+0000 7fc0b2000640 1 --2- 172.21.15.31:0/1210193919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56492bd15fd0 0x56492bd163b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.636+0000 7fc0b2000640 1 -- 172.21.15.31:0/1210193919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492bd168f0 msgr2=0x56492bd1f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.636+0000 7fc0b2000640 1 --2- 172.21.15.31:0/1210193919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492bd168f0 0x56492bd1f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.636+0000 7fc0b2000640 1 -- 172.21.15.31:0/1210193919 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56492bc3b2c0 con 0x56492bd1f6c0 2024-03-21T10:41:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.636+0000 7fc0b2a00640 1 --2- 172.21.15.31:0/1210193919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492bd168f0 0x56492bd1f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:41:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.636+0000 7fc0b8a00640 1 -- 172.21.15.31:0/581733028 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fc0a40cff50 con 0x7fc0b4103880 2024-03-21T10:41:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.637+0000 7fc0b2000640 1 --2- 172.21.15.31:0/1210193919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56492bd1f6c0 0x56492bd21ab0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0009230 tx=0x7fc0a0009670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cb58e72546075c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.637+0000 7fc0b3400640 1 --2- 172.21.15.31:0/581733028 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc0980a3080 0x7fc0980a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.637+0000 7fc09fe00640 1 -- 172.21.15.31:0/1210193919 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a0042070 con 0x56492bd1f6c0 2024-03-21T10:41:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.637+0000 7fc09fe00640 1 -- 172.21.15.31:0/1210193919 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc0a0004a70 con 0x56492bd1f6c0 2024-03-21T10:41:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.637+0000 7fc09fe00640 1 -- 172.21.15.31:0/1210193919 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a00051c0 con 0x56492bd1f6c0 2024-03-21T10:41:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.638+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/1210193919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56492bd1f6c0 msgr2=0x56492bd21ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.638+0000 7fc0bcbdb740 1 --2- 172.21.15.31:0/1210193919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56492bd1f6c0 0x56492bd21ab0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0009230 tx=0x7fc0a0009670 comp rx=0 tx=0).stop 2024-03-21T10:41:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.639+0000 7fc0b3400640 1 --2- 172.21.15.31:0/581733028 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc0980a3080 0x7fc0980a5540 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac018fd0 tx=0x7fc0ac004a40 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.639+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/1210193919 shutdown_connections 2024-03-21T10:41:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.639+0000 7fc0bcbdb740 1 --2- 172.21.15.31:0/1210193919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56492bd1f6c0 0x56492bd21ab0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.639+0000 7fc0bcbdb740 1 --2- 172.21.15.31:0/1210193919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492bd168f0 0x56492bd1f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.639+0000 7fc0bcbdb740 1 --2- 172.21.15.31:0/1210193919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56492bd15fd0 0x56492bd163b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.639+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/1210193919 >> 172.21.15.31:0/1210193919 conn(0x56492bd108f0 msgr2=0x56492bd10cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.639+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/1210193919 shutdown_connections 2024-03-21T10:41:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.640+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/1210193919 wait complete. 2024-03-21T10:41:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.641+0000 7fc0bcbdb740 1 -- start start 2024-03-21T10:41:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.642+0000 7fc0bcbdb740 1 Processor -- start 2024-03-21T10:41:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.643+0000 7fc0bcbdb740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492bd15fd0 0x56492bd1c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.643+0000 7fc0bcbdb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56492bd168f0 0x56492bd1cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.643+0000 7fc0bcbdb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56492bd1f6c0 0x56492bd1d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.643+0000 7fc0bcbdb740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56492bc36a20 con 0x56492bd15fd0 2024-03-21T10:41:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.643+0000 7fc0b2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56492bd168f0 0x56492bd1cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.643+0000 7fc0b2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56492bd168f0 0x56492bd1cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60778/0 (socket says 172.21.15.31:60778) 2024-03-21T10:41:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.643+0000 7fc0b2000640 1 -- 172.21.15.31:0/351610404 learned_addr learned my addr 172.21.15.31:0/351610404 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.643+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/351610404 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56492bc768d0 con 0x56492bd168f0 2024-03-21T10:41:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.643+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/351610404 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56492bd22290 con 0x56492bd1f6c0 2024-03-21T10:41:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.643+0000 7fc0b2a00640 1 --2- 172.21.15.31:0/351610404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492bd15fd0 0x56492bd1c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.643+0000 7fc0bcbdb740 10 client.0 _ll_register_callbacks cb 0x56492bc731c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.643+0000 7fc0b3400640 1 --2- 172.21.15.31:0/351610404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56492bd1f6c0 0x56492bd1d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.644+0000 7fc0b2a00640 1 -- 172.21.15.31:0/351610404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56492bd1f6c0 msgr2=0x56492bd1d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.644+0000 7fc0b2a00640 1 --2- 172.21.15.31:0/351610404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56492bd1f6c0 0x56492bd1d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.644+0000 7fc0b2a00640 1 -- 172.21.15.31:0/351610404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56492bd168f0 msgr2=0x56492bd1cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.644+0000 7fc0b2a00640 1 --2- 172.21.15.31:0/351610404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56492bd168f0 0x56492bd1cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.644+0000 7fc0b2a00640 1 -- 172.21.15.31:0/351610404 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56492b98f210 con 0x56492bd15fd0 2024-03-21T10:41:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.644+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/351610404 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc0a0004a70 con 0x56492bd15fd0 2024-03-21T10:41:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.644+0000 7fc0b2a00640 1 --2- 172.21.15.31:0/351610404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492bd15fd0 0x56492bd1c800 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc0a40cd360 tx=0x7fc0a409bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aebc8630fb44b89 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.644+0000 7fc09cc00640 10 client.16116 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:41:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.645+0000 7fc09cc00640 1 -- 172.21.15.31:0/351610404 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a40b5060 con 0x56492bd15fd0 2024-03-21T10:41:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.645+0000 7fc09cc00640 1 -- 172.21.15.31:0/351610404 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fc0a4098670 con 0x56492bd15fd0 2024-03-21T10:41:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.645+0000 7fc09cc00640 1 -- 172.21.15.31:0/351610404 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0a40a0150 con 0x56492bd15fd0 2024-03-21T10:41:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.645+0000 7fc0bcbdb740 10 client.16116 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.645+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/351610404 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({fsmap=25}) v3 -- 0x56492ba5b620 con 0x56492bd15fd0 2024-03-21T10:41:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.645+0000 7fc09cc00640 1 -- 172.21.15.31:0/351610404 <== mon.0 v2:172.21.15.31:3300/0 4 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fc0a40a4580 con 0x56492bd15fd0 2024-03-21T10:41:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.646+0000 7fc0bcbdb740 10 client.16116 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.646+0000 7fc0bcbdb740 10 client.16116 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:41:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.646+0000 7fc0bcbdb740 20 client.16116 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.646+0000 7fc0bcbdb740 1 --2- 172.21.15.31:0/351610404 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56492bd2f510 0x56492bd31920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.646+0000 7fc0bcbdb740 4 client.16116 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:41:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.646+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/351610404 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56492bc87d70 con 0x56492bd2f510 2024-03-21T10:41:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.646+0000 7fc0b2000640 1 --2- 172.21.15.31:0/351610404 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56492bd2f510 0x56492bd31920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.646+0000 7fc0b2000640 1 --2- 172.21.15.31:0/351610404 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56492bd2f510 0x56492bd31920 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba4c152ac8d66f1 server_cookie=7ad088c7b647a809 in_seq=0 out_seq=0 2024-03-21T10:41:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.646+0000 7fc09cc00640 10 client.16116 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.647+0000 7fc09cc00640 1 -- 172.21.15.31:0/351610404 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56492bc87d70 con 0x56492bd2f510 2024-03-21T10:41:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.647+0000 7fc09cc00640 10 client.16116 handle_command_reply: tid=0 2024-03-21T10:41:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.649+0000 7fc0bcbdb740 1 client.16116 shutdown 2024-03-21T10:41:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.650+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/351610404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492bd15fd0 msgr2=0x56492bd1c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.650+0000 7fc0bcbdb740 1 --2- 172.21.15.31:0/351610404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492bd15fd0 0x56492bd1c800 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc0a40cd360 tx=0x7fc0a409bc00 comp rx=0 tx=0).stop 2024-03-21T10:41:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.655+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/351610404 shutdown_connections 2024-03-21T10:41:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.658+0000 7fc0bcbdb740 1 --2- 172.21.15.31:0/351610404 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56492bd2f510 0x56492bd31920 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.658+0000 7fc09cc00640 0 client.16116 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:15.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.658+0000 7fc0bcbdb740 1 --2- 172.21.15.31:0/351610404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56492bd1f6c0 0x56492bd1d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.658+0000 7fc0bcbdb740 1 --2- 172.21.15.31:0/351610404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56492bd168f0 0x56492bd1cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.658+0000 7fc0bcbdb740 1 --2- 172.21.15.31:0/351610404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492bd15fd0 0x56492bd1c800 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.658+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/351610404 >> 172.21.15.31:0/351610404 conn(0x56492bd108f0 msgr2=0x56492bd12640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:15.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.659+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/351610404 shutdown_connections 2024-03-21T10:41:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.659+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/351610404 wait complete. 2024-03-21T10:41:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.663+0000 7fc0bcbdb740 20 client.16116 trim_cache size 0 max 16384 2024-03-21T10:41:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.740+0000 7fc0bcbdb740 1 Processor -- start 2024-03-21T10:41:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.741+0000 7fc0bcbdb740 1 -- start start 2024-03-21T10:41:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.741+0000 7fc0bcbdb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56492bd15fd0 0x56492bd1ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.741+0000 7fc0bcbdb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56492bd168f0 0x56492bd2a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.741+0000 7fc0bcbdb740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492bd1f6c0 0x56492bd2aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.741+0000 7fc0bcbdb740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b4072dc0 con 0x56492bd1f6c0 2024-03-21T10:41:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.741+0000 7fc0bcbdb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b40708c0 con 0x56492bd168f0 2024-03-21T10:41:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.741+0000 7fc0bcbdb740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b406e980 con 0x56492bd15fd0 2024-03-21T10:41:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.741+0000 7fc0b2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56492bd15fd0 0x56492bd1ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.741+0000 7fc0b2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56492bd15fd0 0x56492bd1ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53016/0 (socket says 172.21.15.31:53016) 2024-03-21T10:41:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.741+0000 7fc0b2a00640 1 -- 172.21.15.31:0/145843931 learned_addr learned my addr 172.21.15.31:0/145843931 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.741+0000 7fc0b2000640 1 --2- 172.21.15.31:0/145843931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56492bd168f0 0x56492bd2a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.741+0000 7fc0b3400640 1 --2- 172.21.15.31:0/145843931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492bd1f6c0 0x56492bd2aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.742+0000 7fc0b2a00640 1 -- 172.21.15.31:0/145843931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56492bd168f0 msgr2=0x56492bd2a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.742+0000 7fc0b2a00640 1 --2- 172.21.15.31:0/145843931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56492bd168f0 0x56492bd2a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.742+0000 7fc0b2a00640 1 -- 172.21.15.31:0/145843931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492bd1f6c0 msgr2=0x56492bd2aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.742+0000 7fc0b2a00640 1 --2- 172.21.15.31:0/145843931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492bd1f6c0 0x56492bd2aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.742+0000 7fc0b2a00640 1 -- 172.21.15.31:0/145843931 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56492bd21de0 con 0x56492bd15fd0 2024-03-21T10:41:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.742+0000 7fc0b2a00640 1 --2- 172.21.15.31:0/145843931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56492bd15fd0 0x56492bd1ef30 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc0a409bd00 tx=0x7fc0a4019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36f4add697afbf2c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.742+0000 7fc09f400640 1 -- 172.21.15.31:0/145843931 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a40982a0 con 0x56492bd15fd0 2024-03-21T10:41:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.743+0000 7fc09f400640 1 -- 172.21.15.31:0/145843931 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc0a0004a70 con 0x56492bd15fd0 2024-03-21T10:41:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.743+0000 7fc09f400640 1 -- 172.21.15.31:0/145843931 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a40d2110 con 0x56492bd15fd0 2024-03-21T10:41:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.743+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/145843931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56492bd15fd0 msgr2=0x56492bd1ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.743+0000 7fc0bcbdb740 1 --2- 172.21.15.31:0/145843931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56492bd15fd0 0x56492bd1ef30 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc0a409bd00 tx=0x7fc0a4019000 comp rx=0 tx=0).stop 2024-03-21T10:41:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.744+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/145843931 shutdown_connections 2024-03-21T10:41:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.744+0000 7fc0bcbdb740 1 --2- 172.21.15.31:0/145843931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492bd1f6c0 0x56492bd2aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.744+0000 7fc0bcbdb740 1 --2- 172.21.15.31:0/145843931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56492bd168f0 0x56492bd2a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.744+0000 7fc0bcbdb740 1 --2- 172.21.15.31:0/145843931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56492bd15fd0 0x56492bd1ef30 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.744+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/145843931 >> 172.21.15.31:0/145843931 conn(0x56492bd108f0 msgr2=0x56492bd1b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.744+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/145843931 shutdown_connections 2024-03-21T10:41:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.744+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/145843931 wait complete. 2024-03-21T10:41:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.745+0000 7fc0bcbdb740 1 -- start start 2024-03-21T10:41:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.745+0000 7fc0bcbdb740 1 Processor -- start 2024-03-21T10:41:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.746+0000 7fc0bcbdb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56492bd168f0 0x56492bd22c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.746+0000 7fc0bcbdb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56492bd1f6c0 0x56492bd231d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.746+0000 7fc0bcbdb740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492bd2f510 0x56492bd89df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.746+0000 7fc0bcbdb740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b406ef80 con 0x56492bd2f510 2024-03-21T10:41:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.746+0000 7fc0bcbdb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b4070260 con 0x56492bd1f6c0 2024-03-21T10:41:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.746+0000 7fc0bcbdb740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b406f5a0 con 0x56492bd168f0 2024-03-21T10:41:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.746+0000 7fc0bcbdb740 10 client.0 _ll_register_callbacks cb 0x7fc0a400cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.747+0000 7fc0b2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56492bd168f0 0x56492bd22c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.747+0000 7fc0b2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56492bd168f0 0x56492bd22c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53030/0 (socket says 172.21.15.31:53030) 2024-03-21T10:41:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.747+0000 7fc0b2a00640 1 -- 172.21.15.31:0/1397101339 learned_addr learned my addr 172.21.15.31:0/1397101339 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.747+0000 7fc0b2000640 1 --2- 172.21.15.31:0/1397101339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56492bd1f6c0 0x56492bd231d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.747+0000 7fc0b2a00640 1 -- 172.21.15.31:0/1397101339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56492bd1f6c0 msgr2=0x56492bd231d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.747+0000 7fc0b2a00640 1 --2- 172.21.15.31:0/1397101339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56492bd1f6c0 0x56492bd231d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.747+0000 7fc0b2a00640 1 -- 172.21.15.31:0/1397101339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492bd2f510 msgr2=0x56492bd89df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:41:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.749+0000 7fc0b3400640 1 --2- 172.21.15.31:0/1397101339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492bd2f510 0x56492bd89df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.750+0000 7fc0b2a00640 1 --2- 172.21.15.31:0/1397101339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492bd2f510 0x56492bd89df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.754+0000 7fc0b2a00640 1 -- 172.21.15.31:0/1397101339 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56492bd8a330 con 0x56492bd168f0 2024-03-21T10:41:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.754+0000 7fc0b3400640 1 --2- 172.21.15.31:0/1397101339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492bd2f510 0x56492bd89df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:41:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.754+0000 7fc0b2a00640 1 --2- 172.21.15.31:0/1397101339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56492bd168f0 0x56492bd22c90 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4098460 tx=0x7fc0a4005e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cf54a2603d99266 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:15.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.755+0000 7fc09cc00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:41:15.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.755+0000 7fc09cc00640 1 -- 172.21.15.31:0/1397101339 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a40982a0 con 0x56492bd168f0 2024-03-21T10:41:15.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.755+0000 7fc09cc00640 1 -- 172.21.15.31:0/1397101339 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fc0a4096d70 con 0x56492bd168f0 2024-03-21T10:41:15.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.755+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/1397101339 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc0a0004a70 con 0x56492bd168f0 2024-03-21T10:41:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.756+0000 7fc09cc00640 1 -- 172.21.15.31:0/1397101339 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0a409bdd0 con 0x56492bd168f0 2024-03-21T10:41:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.756+0000 7fc0bcbdb740 10 client.26341 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.756+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/1397101339 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56492bd8b500 con 0x56492bd168f0 2024-03-21T10:41:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.757+0000 7fc09cc00640 1 -- 172.21.15.31:0/1397101339 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc0a0004a70 con 0x56492bd168f0 2024-03-21T10:41:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.757+0000 7fc09cc00640 1 -- 172.21.15.31:0/1397101339 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fc0a409aec0 con 0x56492bd168f0 2024-03-21T10:41:15.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.758+0000 7fc0bcbdb740 10 client.26341 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:15.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.758+0000 7fc0bcbdb740 10 client.26341 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:41:15.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.758+0000 7fc0bcbdb740 20 client.26341 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:15.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.758+0000 7fc0bcbdb740 1 --2- 172.21.15.31:0/1397101339 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56492bd8b890 0x56492bd8dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:15.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.758+0000 7fc0bcbdb740 4 client.26341 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:41:15.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.758+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/1397101339 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56492bc820c0 con 0x56492bd8b890 2024-03-21T10:41:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.759+0000 7fc0b2000640 1 --2- 172.21.15.31:0/1397101339 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56492bd8b890 0x56492bd8dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.761+0000 7fc0b2000640 1 --2- 172.21.15.31:0/1397101339 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56492bd8b890 0x56492bd8dc80 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f74ea6a8bc0877aa server_cookie=cc3a2434c958b6a6 in_seq=0 out_seq=0 2024-03-21T10:41:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.762+0000 7fc09cc00640 10 client.26341 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:15.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.779+0000 7fc09cc00640 1 -- 172.21.15.31:0/1397101339 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56492bc820c0 con 0x56492bd8b890 2024-03-21T10:41:15.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.779+0000 7fc09cc00640 10 client.26341 handle_command_reply: tid=0 2024-03-21T10:41:15.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.779+0000 7fc0bcbdb740 1 client.26341 shutdown 2024-03-21T10:41:15.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.779+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/1397101339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56492bd168f0 msgr2=0x56492bd22c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:15.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.779+0000 7fc0bcbdb740 1 --2- 172.21.15.31:0/1397101339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56492bd168f0 0x56492bd22c90 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4098460 tx=0x7fc0a4005e30 comp rx=0 tx=0).stop 2024-03-21T10:41:15.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.780+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/1397101339 shutdown_connections 2024-03-21T10:41:15.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.780+0000 7fc0bcbdb740 1 --2- 172.21.15.31:0/1397101339 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56492bd8b890 0x56492bd8dc80 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.780+0000 7fc0bcbdb740 1 --2- 172.21.15.31:0/1397101339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492bd2f510 0x56492bd89df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.780+0000 7fc0bcbdb740 1 --2- 172.21.15.31:0/1397101339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56492bd1f6c0 0x56492bd231d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.780+0000 7fc0bcbdb740 1 --2- 172.21.15.31:0/1397101339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56492bd168f0 0x56492bd22c90 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.780+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/1397101339 >> 172.21.15.31:0/1397101339 conn(0x56492bd108f0 msgr2=0x56492bd11c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:15.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.780+0000 7fc09cc00640 0 client.26341 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:15.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.780+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/1397101339 shutdown_connections 2024-03-21T10:41:15.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.780+0000 7fc0bcbdb740 1 -- 172.21.15.31:0/1397101339 wait complete. 2024-03-21T10:41:15.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.780+0000 7fc0bcbdb740 20 client.26341 trim_cache size 0 max 16384 2024-03-21T10:41:15.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:41:15.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:41:15.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:41:15.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:41:15.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.783+0000 7fc0b9e00640 1 -- 172.21.15.31:0/581733028 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc0980a3080 msgr2=0x7fc0980a5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:15.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.783+0000 7fc0b9e00640 1 --2- 172.21.15.31:0/581733028 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc0980a3080 0x7fc0980a5540 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac018fd0 tx=0x7fc0ac004a40 comp rx=0 tx=0).stop 2024-03-21T10:41:15.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.783+0000 7fc0b9e00640 1 -- 172.21.15.31:0/581733028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0b4103880 msgr2=0x7fc0b419a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:15.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.783+0000 7fc0b9e00640 1 --2- 172.21.15.31:0/581733028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0b4103880 0x7fc0b419a170 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc0a400e9f0 tx=0x7fc0a400edf0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.784+0000 7fc0b9e00640 1 -- 172.21.15.31:0/581733028 shutdown_connections 2024-03-21T10:41:15.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.784+0000 7fc0b9e00640 1 --2- 172.21.15.31:0/581733028 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc0980a3080 0x7fc0980a5540 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.784+0000 7fc0b9e00640 1 --2- 172.21.15.31:0/581733028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0b4109890 0x7fc0b419ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.784+0000 7fc0b9e00640 1 --2- 172.21.15.31:0/581733028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0b4104240 0x7fc0b419a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.784+0000 7fc0b9e00640 1 --2- 172.21.15.31:0/581733028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0b4103880 0x7fc0b419a170 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:15.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.784+0000 7fc0b9e00640 1 -- 172.21.15.31:0/581733028 >> 172.21.15.31:0/581733028 conn(0x7fc0b40ff390 msgr2=0x7fc0b40ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:15.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.785+0000 7fc0b9e00640 1 -- 172.21.15.31:0/581733028 shutdown_connections 2024-03-21T10:41:15.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:15.785+0000 7fc0b9e00640 1 -- 172.21.15.31:0/581733028 wait complete. 2024-03-21T10:41:17.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:16 smithi031 ceph-mon[31673]: pgmap v84: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 273 MiB/s wr, 1.70k op/s 2024-03-21T10:41:17.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:16 smithi070 ceph-mon[36330]: pgmap v84: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 273 MiB/s wr, 1.70k op/s 2024-03-21T10:41:17.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:16 smithi040 ceph-mon[36154]: pgmap v84: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 273 MiB/s wr, 1.70k op/s 2024-03-21T10:41:19.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:18 smithi031 ceph-mon[31673]: pgmap v85: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 185 MiB/s wr, 1.19k op/s 2024-03-21T10:41:19.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:18 smithi070 ceph-mon[36330]: pgmap v85: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 185 MiB/s wr, 1.19k op/s 2024-03-21T10:41:19.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:18 smithi040 ceph-mon[36154]: pgmap v85: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 185 MiB/s wr, 1.19k op/s 2024-03-21T10:41:21.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:20 smithi031 ceph-mon[31673]: pgmap v86: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 247 MiB/s wr, 1.73k op/s 2024-03-21T10:41:21.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:20 smithi070 ceph-mon[36330]: pgmap v86: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 247 MiB/s wr, 1.73k op/s 2024-03-21T10:41:21.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:20 smithi040 ceph-mon[36154]: pgmap v86: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 247 MiB/s wr, 1.73k op/s 2024-03-21T10:41:22.058 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:21 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:41:22.059 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:21 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:41:22.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:21 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:41:22.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:21 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:41:22.081 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:21 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:41:22.081 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:21 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:41:23.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:22 smithi031 ceph-mon[31673]: pgmap v87: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 185 MiB/s wr, 1.49k op/s 2024-03-21T10:41:23.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:22 smithi070 ceph-mon[36330]: pgmap v87: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 185 MiB/s wr, 1.49k op/s 2024-03-21T10:41:23.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:22 smithi040 ceph-mon[36154]: pgmap v87: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 185 MiB/s wr, 1.49k op/s 2024-03-21T10:41:25.062 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:24 smithi031 ceph-mon[31673]: pgmap v88: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 177 MiB/s wr, 1.44k op/s 2024-03-21T10:41:25.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:24 smithi070 ceph-mon[36330]: pgmap v88: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 177 MiB/s wr, 1.44k op/s 2024-03-21T10:41:25.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:24 smithi040 ceph-mon[36154]: pgmap v88: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 177 MiB/s wr, 1.44k op/s 2024-03-21T10:41:27.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:26 smithi070 ceph-mon[36330]: pgmap v89: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 276 MiB/s wr, 2.23k op/s 2024-03-21T10:41:27.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:26 smithi040 ceph-mon[36154]: pgmap v89: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 276 MiB/s wr, 2.23k op/s 2024-03-21T10:41:27.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:26 smithi031 ceph-mon[31673]: pgmap v89: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 276 MiB/s wr, 2.23k op/s 2024-03-21T10:41:29.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:28 smithi070 ceph-mon[36330]: pgmap v90: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 191 MiB/s wr, 1.57k op/s 2024-03-21T10:41:29.111 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:28 smithi031 ceph-mon[31673]: pgmap v90: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 191 MiB/s wr, 1.57k op/s 2024-03-21T10:41:29.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:28 smithi040 ceph-mon[36154]: pgmap v90: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 191 MiB/s wr, 1.57k op/s 2024-03-21T10:41:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.226+0000 7eff97200640 1 -- 172.21.15.31:0/3556341461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff9810e3a0 msgr2=0x7eff9811ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.226+0000 7eff97200640 1 --2- 172.21.15.31:0/3556341461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff9810e3a0 0x7eff9811ce90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7eff90009ca0 tx=0x7eff9002c180 comp rx=0 tx=0).stop 2024-03-21T10:41:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.227+0000 7eff97200640 1 -- 172.21.15.31:0/3556341461 shutdown_connections 2024-03-21T10:41:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.227+0000 7eff97200640 1 --2- 172.21.15.31:0/3556341461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff9810e3a0 0x7eff9811ce90 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.227+0000 7eff97200640 1 --2- 172.21.15.31:0/3556341461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff9810d810 0x7eff9810dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.227+0000 7eff97200640 1 --2- 172.21.15.31:0/3556341461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff9810fd40 0x7eff98110120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.227+0000 7eff97200640 1 -- 172.21.15.31:0/3556341461 >> 172.21.15.31:0/3556341461 conn(0x7eff9806c5f0 msgr2=0x7eff9806ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.227+0000 7eff97200640 1 -- 172.21.15.31:0/3556341461 shutdown_connections 2024-03-21T10:41:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.227+0000 7eff97200640 1 -- 172.21.15.31:0/3556341461 wait complete. 2024-03-21T10:41:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.228+0000 7eff97200640 1 Processor -- start 2024-03-21T10:41:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.228+0000 7eff97200640 1 -- start start 2024-03-21T10:41:29.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.229+0000 7eff97200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff9810d810 0x7eff981ac7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:29.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.229+0000 7eff97200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff9810e3a0 0x7eff981acd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:29.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.229+0000 7eff97200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff9810fd40 0x7eff981a6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:29.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.229+0000 7eff97200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff981206d0 con 0x7eff9810d810 2024-03-21T10:41:29.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.229+0000 7eff97200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff98120550 con 0x7eff9810fd40 2024-03-21T10:41:29.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.229+0000 7eff97200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff98120850 con 0x7eff9810e3a0 2024-03-21T10:41:29.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.229+0000 7eff95e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff9810d810 0x7eff981ac7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:29.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.229+0000 7eff96800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff9810fd40 0x7eff981a6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:29.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.229+0000 7eff96800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff9810fd40 0x7eff981a6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38988/0 (socket says 172.21.15.31:38988) 2024-03-21T10:41:29.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.229+0000 7eff96800640 1 -- 172.21.15.31:0/2593396237 learned_addr learned my addr 172.21.15.31:0/2593396237 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:29.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.229+0000 7eff95400640 1 --2- 172.21.15.31:0/2593396237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff9810e3a0 0x7eff981acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.230+0000 7eff95e00640 1 -- 172.21.15.31:0/2593396237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff9810e3a0 msgr2=0x7eff981acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.230+0000 7eff95e00640 1 --2- 172.21.15.31:0/2593396237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff9810e3a0 0x7eff981acd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.230+0000 7eff95e00640 1 -- 172.21.15.31:0/2593396237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff9810fd40 msgr2=0x7eff981a6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:29.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.230+0000 7eff95e00640 1 --2- 172.21.15.31:0/2593396237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff9810fd40 0x7eff981a6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.230+0000 7eff95e00640 1 -- 172.21.15.31:0/2593396237 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff981a70f0 con 0x7eff9810d810 2024-03-21T10:41:29.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.231+0000 7eff95e00640 1 --2- 172.21.15.31:0/2593396237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff9810d810 0x7eff981ac7c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7eff8800c970 tx=0x7eff8800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:29.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.231+0000 7eff8ea00640 1 -- 172.21.15.31:0/2593396237 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff88007bb0 con 0x7eff9810d810 2024-03-21T10:41:29.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.231+0000 7eff8ea00640 1 -- 172.21.15.31:0/2593396237 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff88007d50 con 0x7eff9810d810 2024-03-21T10:41:29.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.231+0000 7eff8ea00640 1 -- 172.21.15.31:0/2593396237 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff88018820 con 0x7eff9810d810 2024-03-21T10:41:29.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.231+0000 7eff97200640 1 -- 172.21.15.31:0/2593396237 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff981a73e0 con 0x7eff9810d810 2024-03-21T10:41:29.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.231+0000 7eff97200640 1 -- 172.21.15.31:0/2593396237 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff981b3680 con 0x7eff9810d810 2024-03-21T10:41:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.241+0000 7eff9fa14740 1 Processor -- start 2024-03-21T10:41:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.243+0000 7eff9fa14740 1 -- start start 2024-03-21T10:41:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.243+0000 7eff9fa14740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bb08071e0 0x562bb08075c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.243+0000 7eff9fa14740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bb0807b00 0x562bb0810390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.244+0000 7eff9fa14740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bb08108d0 0x562bb0812cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.244+0000 7eff9fa14740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bb07c93b0 con 0x562bb08108d0 2024-03-21T10:41:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.244+0000 7eff9fa14740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bb076b830 con 0x562bb08071e0 2024-03-21T10:41:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.244+0000 7eff9fa14740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bb07145d0 con 0x562bb0807b00 2024-03-21T10:41:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.244+0000 7eff96800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bb08071e0 0x562bb08075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.244+0000 7eff96800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bb08071e0 0x562bb08075c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39004/0 (socket says 172.21.15.31:39004) 2024-03-21T10:41:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.244+0000 7eff96800640 1 -- 172.21.15.31:0/3231558234 learned_addr learned my addr 172.21.15.31:0/3231558234 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.244+0000 7eff95400640 1 --2- 172.21.15.31:0/3231558234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bb08108d0 0x562bb0812cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.245+0000 7eff8ea00640 1 -- 172.21.15.31:0/2593396237 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7eff880189c0 con 0x7eff9810d810 2024-03-21T10:41:29.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.245+0000 7eff95400640 1 -- 172.21.15.31:0/3231558234 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bb0807b00 msgr2=0x562bb0810390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:41:29.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.245+0000 7eff95400640 1 --2- 172.21.15.31:0/3231558234 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bb0807b00 0x562bb0810390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.245+0000 7eff95400640 1 -- 172.21.15.31:0/3231558234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bb08071e0 msgr2=0x562bb08075c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:29.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.245+0000 7eff95400640 1 --2- 172.21.15.31:0/3231558234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bb08071e0 0x562bb08075c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.245+0000 7eff95400640 1 -- 172.21.15.31:0/3231558234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562bb072c2c0 con 0x562bb08108d0 2024-03-21T10:41:29.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.246+0000 7eff95400640 1 --2- 172.21.15.31:0/3231558234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bb08108d0 0x562bb0812cc0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7eff981a8160 tx=0x7eff840113e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1afebe3c2928f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:29.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.249+0000 7eff8ea00640 1 --2- 172.21.15.31:0/2593396237 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7eff7c0a3150 0x7eff7c0a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:29.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.249+0000 7eff83000640 1 -- 172.21.15.31:0/3231558234 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff84011de0 con 0x562bb08108d0 2024-03-21T10:41:29.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.249+0000 7eff83000640 1 -- 172.21.15.31:0/3231558234 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff84012930 con 0x562bb08108d0 2024-03-21T10:41:29.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.249+0000 7eff83000640 1 -- 172.21.15.31:0/3231558234 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8400faa0 con 0x562bb08108d0 2024-03-21T10:41:29.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.249+0000 7eff96800640 1 --2- 172.21.15.31:0/2593396237 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7eff7c0a3150 0x7eff7c0a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:29.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.250+0000 7eff8ea00640 1 -- 172.21.15.31:0/2593396237 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7eff880cfba0 con 0x7eff9810d810 2024-03-21T10:41:29.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.250+0000 7eff9fa14740 1 -- 172.21.15.31:0/3231558234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bb08108d0 msgr2=0x562bb0812cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:29.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.250+0000 7eff9fa14740 1 --2- 172.21.15.31:0/3231558234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bb08108d0 0x562bb0812cc0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7eff981a8160 tx=0x7eff840113e0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.251+0000 7eff96800640 1 --2- 172.21.15.31:0/2593396237 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7eff7c0a3150 0x7eff7c0a5610 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7eff90009a60 tx=0x7eff90009800 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:29.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.252+0000 7eff9fa14740 1 -- 172.21.15.31:0/3231558234 shutdown_connections 2024-03-21T10:41:29.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.252+0000 7eff9fa14740 1 --2- 172.21.15.31:0/3231558234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bb08108d0 0x562bb0812cc0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.252+0000 7eff9fa14740 1 --2- 172.21.15.31:0/3231558234 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bb0807b00 0x562bb0810390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.252+0000 7eff9fa14740 1 --2- 172.21.15.31:0/3231558234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bb08071e0 0x562bb08075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.252+0000 7eff9fa14740 1 -- 172.21.15.31:0/3231558234 >> 172.21.15.31:0/3231558234 conn(0x562bb0801b00 msgr2=0x562bb0801ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:29.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.252+0000 7eff9fa14740 1 -- 172.21.15.31:0/3231558234 shutdown_connections 2024-03-21T10:41:29.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.252+0000 7eff9fa14740 1 -- 172.21.15.31:0/3231558234 wait complete. 2024-03-21T10:41:29.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.252+0000 7eff9fa14740 1 -- start start 2024-03-21T10:41:29.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.252+0000 7eff9fa14740 1 Processor -- start 2024-03-21T10:41:29.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.253+0000 7eff9fa14740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bb08071e0 0x562bb080d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:29.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.253+0000 7eff9fa14740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bb0807b00 0x562bb080dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:29.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.253+0000 7eff9fa14740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bb08108d0 0x562bb080e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:29.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.253+0000 7eff9fa14740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bb0727a20 con 0x562bb08108d0 2024-03-21T10:41:29.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.253+0000 7eff9fa14740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bb07678d0 con 0x562bb08071e0 2024-03-21T10:41:29.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.253+0000 7eff9fa14740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bb0813500 con 0x562bb0807b00 2024-03-21T10:41:29.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.253+0000 7eff9fa14740 10 client.0 _ll_register_callbacks cb 0x562bb07641c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.254+0000 7eff96800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bb08108d0 0x562bb080e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.255+0000 7eff95e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bb08071e0 0x562bb080d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.255+0000 7eff96800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bb08108d0 0x562bb080e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39154/0 (socket says 172.21.15.31:39154) 2024-03-21T10:41:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.255+0000 7eff96800640 1 -- 172.21.15.31:0/3325893843 learned_addr learned my addr 172.21.15.31:0/3325893843 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.255+0000 7eff95400640 1 --2- 172.21.15.31:0/3325893843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bb0807b00 0x562bb080dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.256+0000 7eff95e00640 1 -- 172.21.15.31:0/3325893843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bb0807b00 msgr2=0x562bb080dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.256+0000 7eff95e00640 1 --2- 172.21.15.31:0/3325893843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bb0807b00 0x562bb080dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.256+0000 7eff95e00640 1 -- 172.21.15.31:0/3325893843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bb08108d0 msgr2=0x562bb080e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.256+0000 7eff95e00640 1 --2- 172.21.15.31:0/3325893843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bb08108d0 0x562bb080e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.256+0000 7eff95e00640 1 -- 172.21.15.31:0/3325893843 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562bb0480210 con 0x562bb08071e0 2024-03-21T10:41:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.256+0000 7eff95e00640 1 --2- 172.21.15.31:0/3325893843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bb08071e0 0x562bb080d9a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7eff880ccfb0 tx=0x7eff8809b6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8099045f79a3fca server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.256+0000 7eff77400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:41:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.256+0000 7eff96800640 1 --2- 172.21.15.31:0/3325893843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bb08108d0 0x562bb080e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:41:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.257+0000 7eff77400640 1 -- 172.21.15.31:0/3325893843 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8809bc50 con 0x562bb08071e0 2024-03-21T10:41:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.257+0000 7eff77400640 1 -- 172.21.15.31:0/3325893843 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7eff880a41d0 con 0x562bb08071e0 2024-03-21T10:41:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.257+0000 7eff9fa14740 1 -- 172.21.15.31:0/3325893843 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7eff84012930 con 0x562bb08071e0 2024-03-21T10:41:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.258+0000 7eff77400640 1 -- 172.21.15.31:0/3325893843 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff880a4450 con 0x562bb08071e0 2024-03-21T10:41:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.259+0000 7eff9fa14740 10 client.26399 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.259+0000 7eff9fa14740 1 -- 172.21.15.31:0/3325893843 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x562bb054c620 con 0x562bb08071e0 2024-03-21T10:41:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.259+0000 7eff77400640 1 -- 172.21.15.31:0/3325893843 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff84012930 con 0x562bb08071e0 2024-03-21T10:41:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.259+0000 7eff77400640 1 -- 172.21.15.31:0/3325893843 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7eff8809ab10 con 0x562bb08071e0 2024-03-21T10:41:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.260+0000 7eff9fa14740 10 client.26399 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.260+0000 7eff9fa14740 10 client.26399 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:41:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.260+0000 7eff9fa14740 20 client.26399 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.260+0000 7eff9fa14740 1 --2- 172.21.15.31:0/3325893843 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562bb0820950 0x562bb0822d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.260+0000 7eff9fa14740 4 client.26399 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:41:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.260+0000 7eff9fa14740 1 -- 172.21.15.31:0/3325893843 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562bb0778d70 con 0x562bb0820950 2024-03-21T10:41:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.261+0000 7eff95400640 1 --2- 172.21.15.31:0/3325893843 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562bb0820950 0x562bb0822d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.262+0000 7eff95400640 1 --2- 172.21.15.31:0/3325893843 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562bb0820950 0x562bb0822d40 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff15a5074fb86275 server_cookie=fd0a80ca84f4afba in_seq=0 out_seq=0 2024-03-21T10:41:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.262+0000 7eff77400640 10 client.26399 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:29.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.263+0000 7eff77400640 1 -- 172.21.15.31:0/3325893843 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562bb0778d70 con 0x562bb0820950 2024-03-21T10:41:29.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.263+0000 7eff77400640 10 client.26399 handle_command_reply: tid=0 2024-03-21T10:41:29.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.263+0000 7eff9fa14740 1 client.26399 shutdown 2024-03-21T10:41:29.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.263+0000 7eff9fa14740 1 -- 172.21.15.31:0/3325893843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bb08071e0 msgr2=0x562bb080d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:29.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.263+0000 7eff9fa14740 1 --2- 172.21.15.31:0/3325893843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bb08071e0 0x562bb080d9a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7eff880ccfb0 tx=0x7eff8809b6e0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.264+0000 7eff9fa14740 1 -- 172.21.15.31:0/3325893843 shutdown_connections 2024-03-21T10:41:29.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.264+0000 7eff9fa14740 1 --2- 172.21.15.31:0/3325893843 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562bb0820950 0x562bb0822d40 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.264+0000 7eff9fa14740 1 --2- 172.21.15.31:0/3325893843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bb08108d0 0x562bb080e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.264+0000 7eff9fa14740 1 --2- 172.21.15.31:0/3325893843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bb0807b00 0x562bb080dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.264+0000 7eff9fa14740 1 --2- 172.21.15.31:0/3325893843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bb08071e0 0x562bb080d9a0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.264+0000 7eff9fa14740 1 -- 172.21.15.31:0/3325893843 >> 172.21.15.31:0/3325893843 conn(0x562bb0801b00 msgr2=0x562bb0803850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.264+0000 7eff77400640 0 client.26399 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.264+0000 7eff9fa14740 1 -- 172.21.15.31:0/3325893843 shutdown_connections 2024-03-21T10:41:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.264+0000 7eff9fa14740 1 -- 172.21.15.31:0/3325893843 wait complete. 2024-03-21T10:41:29.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.264+0000 7eff9fa14740 20 client.26399 trim_cache size 0 max 16384 2024-03-21T10:41:29.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.334+0000 7eff9fa14740 1 Processor -- start 2024-03-21T10:41:29.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.337+0000 7eff9fa14740 1 -- start start 2024-03-21T10:41:29.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.338+0000 7eff9fa14740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bb08071e0 0x562bb080e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:29.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.340+0000 7eff9fa14740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bb0807b00 0x562bb08009f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:29.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.340+0000 7eff95e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bb08071e0 0x562bb080e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:29.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.340+0000 7eff95e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bb08071e0 0x562bb080e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39032/0 (socket says 172.21.15.31:39032) 2024-03-21T10:41:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.341+0000 7eff9fa14740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bb08108d0 0x562bb081bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.341+0000 7eff95e00640 1 -- 172.21.15.31:0/3729764850 learned_addr learned my addr 172.21.15.31:0/3729764850 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:29.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.342+0000 7eff9fa14740 1 -- 172.21.15.31:0/3729764850 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff981bfa30 con 0x562bb08108d0 2024-03-21T10:41:29.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.342+0000 7eff9fa14740 1 -- 172.21.15.31:0/3729764850 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff981bd530 con 0x562bb08071e0 2024-03-21T10:41:29.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.342+0000 7eff9fa14740 1 -- 172.21.15.31:0/3729764850 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff981bb5d0 con 0x562bb0807b00 2024-03-21T10:41:29.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.342+0000 7eff96800640 1 --2- 172.21.15.31:0/3729764850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bb08108d0 0x562bb081bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:29.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.342+0000 7eff95400640 1 --2- 172.21.15.31:0/3729764850 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bb0807b00 0x562bb08009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:29.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.349+0000 7eff95400640 1 -- 172.21.15.31:0/3729764850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bb08071e0 msgr2=0x562bb080e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:29.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.349+0000 7eff95400640 1 --2- 172.21.15.31:0/3729764850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bb08071e0 0x562bb080e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.349+0000 7eff95400640 1 -- 172.21.15.31:0/3729764850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bb08108d0 msgr2=0x562bb081bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:29.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.349+0000 7eff95400640 1 --2- 172.21.15.31:0/3729764850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bb08108d0 0x562bb081bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.350+0000 7eff95400640 1 -- 172.21.15.31:0/3729764850 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562bb0823300 con 0x562bb0807b00 2024-03-21T10:41:29.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.350+0000 7eff95e00640 1 --2- 172.21.15.31:0/3729764850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bb08071e0 0x562bb080e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:41:29.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.350+0000 7eff96800640 1 --2- 172.21.15.31:0/3729764850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bb08108d0 0x562bb081bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:41:29.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.351+0000 7eff95400640 1 --2- 172.21.15.31:0/3729764850 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bb0807b00 0x562bb08009f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7eff8401c040 tx=0x7eff84005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95a9979b8b9cca82 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:29.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.356+0000 7eff82600640 1 -- 172.21.15.31:0/3729764850 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff84011d00 con 0x562bb0807b00 2024-03-21T10:41:29.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.356+0000 7eff82600640 1 -- 172.21.15.31:0/3729764850 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff84005710 con 0x562bb0807b00 2024-03-21T10:41:29.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.356+0000 7eff82600640 1 -- 172.21.15.31:0/3729764850 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8400f440 con 0x562bb0807b00 2024-03-21T10:41:29.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.361+0000 7eff9fa14740 1 -- 172.21.15.31:0/3729764850 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bb0807b00 msgr2=0x562bb08009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:29.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.361+0000 7eff9fa14740 1 --2- 172.21.15.31:0/3729764850 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bb0807b00 0x562bb08009f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7eff8401c040 tx=0x7eff84005f90 comp rx=0 tx=0).stop 2024-03-21T10:41:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.364+0000 7eff9fa14740 1 -- 172.21.15.31:0/3729764850 shutdown_connections 2024-03-21T10:41:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.364+0000 7eff9fa14740 1 --2- 172.21.15.31:0/3729764850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bb08108d0 0x562bb081bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.364+0000 7eff9fa14740 1 --2- 172.21.15.31:0/3729764850 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bb0807b00 0x562bb08009f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.364+0000 7eff9fa14740 1 --2- 172.21.15.31:0/3729764850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bb08071e0 0x562bb080e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.364+0000 7eff9fa14740 1 -- 172.21.15.31:0/3729764850 >> 172.21.15.31:0/3729764850 conn(0x562bb0801b00 msgr2=0x562bb08002e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:29.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.371+0000 7eff9fa14740 1 -- 172.21.15.31:0/3729764850 shutdown_connections 2024-03-21T10:41:29.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.373+0000 7eff9fa14740 1 -- 172.21.15.31:0/3729764850 wait complete. 2024-03-21T10:41:29.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.373+0000 7eff9fa14740 1 -- start start 2024-03-21T10:41:29.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.373+0000 7eff9fa14740 1 Processor -- start 2024-03-21T10:41:29.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.377+0000 7eff9fa14740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bb08071e0 0x562bb0815720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:29.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.378+0000 7eff9fa14740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bb0807b00 0x562bb0815c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:29.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.378+0000 7eff95e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bb08071e0 0x562bb0815720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:29.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.379+0000 7eff95e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bb08071e0 0x562bb0815720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39178/0 (socket says 172.21.15.31:39178) 2024-03-21T10:41:29.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.379+0000 7eff95e00640 1 -- 172.21.15.31:0/372671071 learned_addr learned my addr 172.21.15.31:0/372671071 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:29.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.379+0000 7eff95400640 1 --2- 172.21.15.31:0/372671071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bb0807b00 0x562bb0815c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:29.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.379+0000 7eff9fa14740 1 --2- 172.21.15.31:0/372671071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bb08108d0 0x562bb08161a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:29.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.379+0000 7eff9fa14740 1 -- 172.21.15.31:0/372671071 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff981bbbd0 con 0x562bb08071e0 2024-03-21T10:41:29.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.379+0000 7eff9fa14740 1 -- 172.21.15.31:0/372671071 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff981bced0 con 0x562bb0807b00 2024-03-21T10:41:29.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.379+0000 7eff9fa14740 1 -- 172.21.15.31:0/372671071 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff981bc210 con 0x562bb08108d0 2024-03-21T10:41:29.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.379+0000 7eff9fa14740 10 client.0 _ll_register_callbacks cb 0x7eff8800fdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:29.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.379+0000 7eff96800640 1 --2- 172.21.15.31:0/372671071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bb08108d0 0x562bb08161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:29.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.380+0000 7eff95e00640 1 -- 172.21.15.31:0/372671071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bb08108d0 msgr2=0x562bb08161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:29.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.380+0000 7eff95e00640 1 --2- 172.21.15.31:0/372671071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bb08108d0 0x562bb08161a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.380+0000 7eff95e00640 1 -- 172.21.15.31:0/372671071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bb0807b00 msgr2=0x562bb0815c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:29.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.380+0000 7eff95e00640 1 --2- 172.21.15.31:0/372671071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bb0807b00 0x562bb0815c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.380+0000 7eff95e00640 1 -- 172.21.15.31:0/372671071 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562bb087c140 con 0x562bb08071e0 2024-03-21T10:41:29.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.380+0000 7eff95400640 1 --2- 172.21.15.31:0/372671071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bb0807b00 0x562bb0815c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:41:29.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.380+0000 7eff95e00640 1 --2- 172.21.15.31:0/372671071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bb08071e0 0x562bb0815720 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7eff880cd5b0 tx=0x7eff8809c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aded509ab6dbe0c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:29.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.381+0000 7eff77400640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:41:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.386+0000 7eff77400640 1 -- 172.21.15.31:0/372671071 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8807f080 con 0x562bb08071e0 2024-03-21T10:41:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.386+0000 7eff77400640 1 -- 172.21.15.31:0/372671071 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7eff880a41d0 con 0x562bb08071e0 2024-03-21T10:41:29.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.388+0000 7eff9fa14740 1 -- 172.21.15.31:0/372671071 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7eff84005710 con 0x562bb08071e0 2024-03-21T10:41:29.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.400+0000 7eff77400640 1 -- 172.21.15.31:0/372671071 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff8809b2a0 con 0x562bb08071e0 2024-03-21T10:41:29.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.403+0000 7eff9fa14740 10 client.16137 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:29.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.403+0000 7eff9fa14740 1 -- 172.21.15.31:0/372671071 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x562bb087d3b0 con 0x562bb08071e0 2024-03-21T10:41:29.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.409+0000 7eff77400640 1 -- 172.21.15.31:0/372671071 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff8809fda0 con 0x562bb08071e0 2024-03-21T10:41:29.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.409+0000 7eff77400640 1 -- 172.21.15.31:0/372671071 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7eff880a0f30 con 0x562bb08071e0 2024-03-21T10:41:29.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.409+0000 7eff9fa14740 10 client.16137 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:29.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.409+0000 7eff9fa14740 10 client.16137 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:41:29.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.409+0000 7eff9fa14740 20 client.16137 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:29.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.409+0000 7eff9fa14740 1 --2- 172.21.15.31:0/372671071 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562bb087d8a0 0x562bb087fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:29.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.409+0000 7eff9fa14740 4 client.16137 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:41:29.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.409+0000 7eff9fa14740 1 -- 172.21.15.31:0/372671071 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x7eff84005710 con 0x562bb087d8a0 2024-03-21T10:41:29.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.410+0000 7eff95400640 1 --2- 172.21.15.31:0/372671071 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562bb087d8a0 0x562bb087fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:29.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.411+0000 7eff77400640 10 client.16137 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:29.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.411+0000 7eff95400640 1 --2- 172.21.15.31:0/372671071 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562bb087d8a0 0x562bb087fcc0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a1af72378434940 server_cookie=e055eb9dd06b6c46 in_seq=0 out_seq=0 2024-03-21T10:41:29.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.448+0000 7eff77400640 1 -- 172.21.15.31:0/372671071 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x7eff84005710 con 0x562bb087d8a0 2024-03-21T10:41:29.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.448+0000 7eff77400640 10 client.16137 handle_command_reply: tid=0 2024-03-21T10:41:29.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.449+0000 7eff9fa14740 1 client.16137 shutdown 2024-03-21T10:41:29.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.452+0000 7eff9fa14740 1 -- 172.21.15.31:0/372671071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bb08071e0 msgr2=0x562bb0815720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:29.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.452+0000 7eff9fa14740 1 --2- 172.21.15.31:0/372671071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bb08071e0 0x562bb0815720 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7eff880cd5b0 tx=0x7eff8809c410 comp rx=0 tx=0).stop 2024-03-21T10:41:29.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.452+0000 7eff9fa14740 1 -- 172.21.15.31:0/372671071 shutdown_connections 2024-03-21T10:41:29.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.452+0000 7eff9fa14740 1 --2- 172.21.15.31:0/372671071 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562bb087d8a0 0x562bb087fcc0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.452+0000 7eff9fa14740 1 --2- 172.21.15.31:0/372671071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bb08108d0 0x562bb08161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.453+0000 7eff9fa14740 1 --2- 172.21.15.31:0/372671071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bb0807b00 0x562bb0815c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.453+0000 7eff9fa14740 1 --2- 172.21.15.31:0/372671071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bb08071e0 0x562bb0815720 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.453+0000 7eff9fa14740 1 -- 172.21.15.31:0/372671071 >> 172.21.15.31:0/372671071 conn(0x562bb0801b00 msgr2=0x562bb0800b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:29.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.453+0000 7eff77400640 0 client.16137 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:29.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.453+0000 7eff9fa14740 1 -- 172.21.15.31:0/372671071 shutdown_connections 2024-03-21T10:41:29.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.453+0000 7eff9fa14740 1 -- 172.21.15.31:0/372671071 wait complete. 2024-03-21T10:41:29.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.453+0000 7eff9fa14740 20 client.16137 trim_cache size 0 max 16384 2024-03-21T10:41:29.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:41:29.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:41:29.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:41:29.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:41:29.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.454+0000 7eff97200640 1 -- 172.21.15.31:0/2593396237 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7eff7c0a3150 msgr2=0x7eff7c0a5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:29.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.454+0000 7eff97200640 1 --2- 172.21.15.31:0/2593396237 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7eff7c0a3150 0x7eff7c0a5610 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7eff90009a60 tx=0x7eff90009800 comp rx=0 tx=0).stop 2024-03-21T10:41:29.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.454+0000 7eff97200640 1 -- 172.21.15.31:0/2593396237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff9810d810 msgr2=0x7eff981ac7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:29.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.454+0000 7eff97200640 1 --2- 172.21.15.31:0/2593396237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff9810d810 0x7eff981ac7c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7eff8800c970 tx=0x7eff8800ce30 comp rx=0 tx=0).stop 2024-03-21T10:41:29.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.455+0000 7eff97200640 1 -- 172.21.15.31:0/2593396237 shutdown_connections 2024-03-21T10:41:29.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.455+0000 7eff97200640 1 --2- 172.21.15.31:0/2593396237 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7eff7c0a3150 0x7eff7c0a5610 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.455+0000 7eff97200640 1 --2- 172.21.15.31:0/2593396237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff9810fd40 0x7eff981a6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.455+0000 7eff97200640 1 --2- 172.21.15.31:0/2593396237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff9810e3a0 0x7eff981acd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.455+0000 7eff97200640 1 --2- 172.21.15.31:0/2593396237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff9810d810 0x7eff981ac7c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:29.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.455+0000 7eff97200640 1 -- 172.21.15.31:0/2593396237 >> 172.21.15.31:0/2593396237 conn(0x7eff9806c5f0 msgr2=0x7eff98070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:29.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.455+0000 7eff97200640 1 -- 172.21.15.31:0/2593396237 shutdown_connections 2024-03-21T10:41:29.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:29.455+0000 7eff97200640 1 -- 172.21.15.31:0/2593396237 wait complete. 2024-03-21T10:41:31.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:30 smithi040 ceph-mon[36154]: pgmap v91: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 218 MiB/s wr, 1.82k op/s 2024-03-21T10:41:31.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:30 smithi031 ceph-mon[31673]: pgmap v91: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 218 MiB/s wr, 1.82k op/s 2024-03-21T10:41:31.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:30 smithi070 ceph-mon[36330]: pgmap v91: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 218 MiB/s wr, 1.82k op/s 2024-03-21T10:41:33.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:32 smithi040 ceph-mon[36154]: pgmap v92: 129 pgs: 129 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 208 MiB/s wr, 1.71k op/s 2024-03-21T10:41:33.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:32 smithi031 ceph-mon[31673]: pgmap v92: 129 pgs: 129 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 208 MiB/s wr, 1.71k op/s 2024-03-21T10:41:33.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:32 smithi070 ceph-mon[36330]: pgmap v92: 129 pgs: 129 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 208 MiB/s wr, 1.71k op/s 2024-03-21T10:41:35.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:34 smithi040 ceph-mon[36154]: pgmap v93: 129 pgs: 129 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 178 MiB/s wr, 1.46k op/s 2024-03-21T10:41:35.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:34 smithi031 ceph-mon[31673]: pgmap v93: 129 pgs: 129 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 178 MiB/s wr, 1.46k op/s 2024-03-21T10:41:35.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:34 smithi070 ceph-mon[36330]: pgmap v93: 129 pgs: 129 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 178 MiB/s wr, 1.46k op/s 2024-03-21T10:41:37.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:36 smithi040 ceph-mon[36154]: pgmap v94: 129 pgs: 129 active+clean; 12 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 268 MiB/s wr, 2.08k op/s 2024-03-21T10:41:37.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:36 smithi031 ceph-mon[31673]: pgmap v94: 129 pgs: 129 active+clean; 12 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 268 MiB/s wr, 2.08k op/s 2024-03-21T10:41:37.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:36 smithi070 ceph-mon[36330]: pgmap v94: 129 pgs: 129 active+clean; 12 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 268 MiB/s wr, 2.08k op/s 2024-03-21T10:41:39.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:38 smithi040 ceph-mon[36154]: pgmap v95: 129 pgs: 129 active+clean; 12 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 169 MiB/s wr, 1.29k op/s 2024-03-21T10:41:39.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:38 smithi031 ceph-mon[31673]: pgmap v95: 129 pgs: 129 active+clean; 12 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 169 MiB/s wr, 1.29k op/s 2024-03-21T10:41:39.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:38 smithi070 ceph-mon[36330]: pgmap v95: 129 pgs: 129 active+clean; 12 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 169 MiB/s wr, 1.29k op/s 2024-03-21T10:41:41.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:40 smithi040 ceph-mon[36154]: pgmap v96: 129 pgs: 129 active+clean; 13 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 203 MiB/s wr, 1.50k op/s 2024-03-21T10:41:41.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:40 smithi031 ceph-mon[31673]: pgmap v96: 129 pgs: 129 active+clean; 13 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 203 MiB/s wr, 1.50k op/s 2024-03-21T10:41:41.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:40 smithi070 ceph-mon[36330]: pgmap v96: 129 pgs: 129 active+clean; 13 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 203 MiB/s wr, 1.50k op/s 2024-03-21T10:41:42.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.702+0000 7fc2e1800640 1 -- 172.21.15.31:0/1410178803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2dc108910 msgr2=0x7fc2dc108cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:42.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.702+0000 7fc2e1800640 1 --2- 172.21.15.31:0/1410178803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2dc108910 0x7fc2dc108cf0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc2c8009900 tx=0x7fc2c802bcf0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.702+0000 7fc2e1800640 1 -- 172.21.15.31:0/1410178803 shutdown_connections 2024-03-21T10:41:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.702+0000 7fc2e1800640 1 --2- 172.21.15.31:0/1410178803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2dc10cfe0 0x7fc2dc0799b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.702+0000 7fc2e1800640 1 --2- 172.21.15.31:0/1410178803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2dc109230 0x7fc2dc10c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.702+0000 7fc2e1800640 1 --2- 172.21.15.31:0/1410178803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2dc108910 0x7fc2dc108cf0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.702+0000 7fc2e1800640 1 -- 172.21.15.31:0/1410178803 >> 172.21.15.31:0/1410178803 conn(0x7fc2dc06c750 msgr2=0x7fc2dc06cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.702+0000 7fc2e1800640 1 -- 172.21.15.31:0/1410178803 shutdown_connections 2024-03-21T10:41:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.703+0000 7fc2e1800640 1 -- 172.21.15.31:0/1410178803 wait complete. 2024-03-21T10:41:42.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.704+0000 7fc2e1800640 1 Processor -- start 2024-03-21T10:41:42.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.706+0000 7fc2e1800640 1 -- start start 2024-03-21T10:41:42.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.707+0000 7fc2e1800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2dc108910 0x7fc2dc0841d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:42.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.707+0000 7fc2e1800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2dc109230 0x7fc2dc07e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:42.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.707+0000 7fc2daa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2dc108910 0x7fc2dc0841d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:42.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.707+0000 7fc2daa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2dc108910 0x7fc2dc0841d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:51394/0 (socket says 172.21.15.31:51394) 2024-03-21T10:41:42.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.708+0000 7fc2daa00640 1 -- 172.21.15.31:0/3133911926 learned_addr learned my addr 172.21.15.31:0/3133911926 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:42.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.708+0000 7fc2e1800640 1 --2- 172.21.15.31:0/3133911926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2dc10cfe0 0x7fc2dc07e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:42.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.708+0000 7fc2e1800640 1 -- 172.21.15.31:0/3133911926 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc07cf20 con 0x7fc2dc109230 2024-03-21T10:41:42.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.708+0000 7fc2e1800640 1 -- 172.21.15.31:0/3133911926 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc07c4f0 con 0x7fc2dc10cfe0 2024-03-21T10:41:42.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.708+0000 7fc2da000640 1 --2- 172.21.15.31:0/3133911926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2dc109230 0x7fc2dc07e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:42.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.709+0000 7fc2e1800640 1 -- 172.21.15.31:0/3133911926 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc07d0a0 con 0x7fc2dc108910 2024-03-21T10:41:42.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.709+0000 7fc2db400640 1 --2- 172.21.15.31:0/3133911926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2dc10cfe0 0x7fc2dc07e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:42.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.710+0000 7fc2da000640 1 -- 172.21.15.31:0/3133911926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2dc108910 msgr2=0x7fc2dc0841d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:42.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.710+0000 7fc2da000640 1 --2- 172.21.15.31:0/3133911926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2dc108910 0x7fc2dc0841d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.710+0000 7fc2da000640 1 -- 172.21.15.31:0/3133911926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2dc10cfe0 msgr2=0x7fc2dc07e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:42.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.710+0000 7fc2da000640 1 --2- 172.21.15.31:0/3133911926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2dc10cfe0 0x7fc2dc07e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.711+0000 7fc2da000640 1 -- 172.21.15.31:0/3133911926 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2dc07f090 con 0x7fc2dc109230 2024-03-21T10:41:42.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.711+0000 7fc2db400640 1 --2- 172.21.15.31:0/3133911926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2dc10cfe0 0x7fc2dc07e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:41:42.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.712+0000 7fc2da000640 1 --2- 172.21.15.31:0/3133911926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2dc109230 0x7fc2dc07e290 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc005fd0 tx=0x7fc2cc00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:42.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.712+0000 7fc2d3400640 1 -- 172.21.15.31:0/3133911926 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc009c00 con 0x7fc2dc109230 2024-03-21T10:41:42.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.713+0000 7fc2d3400640 1 -- 172.21.15.31:0/3133911926 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2cc009da0 con 0x7fc2dc109230 2024-03-21T10:41:42.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.714+0000 7fc2e1800640 1 -- 172.21.15.31:0/3133911926 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2dc136c60 con 0x7fc2dc109230 2024-03-21T10:41:42.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.714+0000 7fc2e1800640 1 -- 172.21.15.31:0/3133911926 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2dc1371b0 con 0x7fc2dc109230 2024-03-21T10:41:42.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.714+0000 7fc2d3400640 1 -- 172.21.15.31:0/3133911926 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc009c00 con 0x7fc2dc109230 2024-03-21T10:41:42.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.722+0000 7fc2d3400640 1 -- 172.21.15.31:0/3133911926 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fc2cc005080 con 0x7fc2dc109230 2024-03-21T10:41:42.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.724+0000 7fc2d3400640 1 --2- 172.21.15.31:0/3133911926 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc2c00a34c0 0x7fc2c00a5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:42.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.725+0000 7fc2e45d2740 1 Processor -- start 2024-03-21T10:41:42.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.725+0000 7fc2e45d2740 1 -- start start 2024-03-21T10:41:42.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.725+0000 7fc2d3400640 1 -- 172.21.15.31:0/3133911926 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fc2cc009a00 con 0x7fc2dc109230 2024-03-21T10:41:42.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.726+0000 7fc2e45d2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570568a5fd0 0x5570568a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:42.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.726+0000 7fc2e45d2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570568a68f0 0x5570568af180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:42.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.726+0000 7fc2e45d2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5570568af6c0 0x5570568b1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:42.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.726+0000 7fc2e45d2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570568683b0 con 0x5570568af6c0 2024-03-21T10:41:42.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.726+0000 7fc2e45d2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55705680a830 con 0x5570568a5fd0 2024-03-21T10:41:42.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.726+0000 7fc2e45d2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570567b35d0 con 0x5570568a68f0 2024-03-21T10:41:42.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.726+0000 7fc2da000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570568a68f0 0x5570568af180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:42.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.726+0000 7fc2da000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570568a68f0 0x5570568af180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:51398/0 (socket says 172.21.15.31:51398) 2024-03-21T10:41:42.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.726+0000 7fc2da000640 1 -- 172.21.15.31:0/1065092576 learned_addr learned my addr 172.21.15.31:0/1065092576 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:42.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.726+0000 7fc2daa00640 1 --2- 172.21.15.31:0/1065092576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570568a5fd0 0x5570568a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:42.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.727+0000 7fc2da000640 1 -- 172.21.15.31:0/1065092576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570568a5fd0 msgr2=0x5570568a63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:42.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.727+0000 7fc2db400640 1 --2- 172.21.15.31:0/1065092576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5570568af6c0 0x5570568b1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:42.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.727+0000 7fc2da000640 1 --2- 172.21.15.31:0/1065092576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570568a5fd0 0x5570568a63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.727+0000 7fc2da000640 1 -- 172.21.15.31:0/1065092576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5570568af6c0 msgr2=0x5570568b1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:42.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.727+0000 7fc2da000640 1 --2- 172.21.15.31:0/1065092576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5570568af6c0 0x5570568b1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.727+0000 7fc2daa00640 1 --2- 172.21.15.31:0/3133911926 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc2c00a34c0 0x7fc2c00a5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:42.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.727+0000 7fc2da000640 1 -- 172.21.15.31:0/1065092576 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570567cb2c0 con 0x5570568a68f0 2024-03-21T10:41:42.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.727+0000 7fc2db400640 1 --2- 172.21.15.31:0/1065092576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5570568af6c0 0x5570568b1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:41:42.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.727+0000 7fc2da000640 1 --2- 172.21.15.31:0/1065092576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570568a68f0 0x5570568af180 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc005050 tx=0x7fc2cc09b390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ccff4df9e6d8316 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:42.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.728+0000 7fc2c7e00640 1 -- 172.21.15.31:0/1065092576 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc0a3610 con 0x5570568a68f0 2024-03-21T10:41:42.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.728+0000 7fc2c7e00640 1 -- 172.21.15.31:0/1065092576 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2cc009da0 con 0x5570568a68f0 2024-03-21T10:41:42.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.728+0000 7fc2c7e00640 1 -- 172.21.15.31:0/1065092576 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc0a3610 con 0x5570568a68f0 2024-03-21T10:41:42.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.729+0000 7fc2e45d2740 1 -- 172.21.15.31:0/1065092576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570568a68f0 msgr2=0x5570568af180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:42.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.729+0000 7fc2e45d2740 1 --2- 172.21.15.31:0/1065092576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570568a68f0 0x5570568af180 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc005050 tx=0x7fc2cc09b390 comp rx=0 tx=0).stop 2024-03-21T10:41:42.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.730+0000 7fc2e45d2740 1 -- 172.21.15.31:0/1065092576 shutdown_connections 2024-03-21T10:41:42.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.730+0000 7fc2e45d2740 1 --2- 172.21.15.31:0/1065092576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5570568af6c0 0x5570568b1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.730+0000 7fc2e45d2740 1 --2- 172.21.15.31:0/1065092576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570568a68f0 0x5570568af180 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.730+0000 7fc2e45d2740 1 --2- 172.21.15.31:0/1065092576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570568a5fd0 0x5570568a63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.730+0000 7fc2e45d2740 1 -- 172.21.15.31:0/1065092576 >> 172.21.15.31:0/1065092576 conn(0x5570568a08f0 msgr2=0x5570568a0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:42.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.734+0000 7fc2daa00640 1 --2- 172.21.15.31:0/3133911926 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc2c00a34c0 0x7fc2c00a5980 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc2c8005c40 tx=0x7fc2c8000bc0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:42.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.735+0000 7fc2e45d2740 1 -- 172.21.15.31:0/1065092576 shutdown_connections 2024-03-21T10:41:42.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.735+0000 7fc2e45d2740 1 -- 172.21.15.31:0/1065092576 wait complete. 2024-03-21T10:41:42.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.735+0000 7fc2e45d2740 1 -- start start 2024-03-21T10:41:42.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.736+0000 7fc2e45d2740 1 Processor -- start 2024-03-21T10:41:42.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.737+0000 7fc2e45d2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5570568a5fd0 0x5570568ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:42.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.737+0000 7fc2e45d2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570568a68f0 0x5570568accb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:42.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.737+0000 7fc2e45d2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570568af6c0 0x5570568ad1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:42.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.737+0000 7fc2e45d2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570567c6a20 con 0x5570568a5fd0 2024-03-21T10:41:42.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.737+0000 7fc2e45d2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570568068d0 con 0x5570568af6c0 2024-03-21T10:41:42.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.737+0000 7fc2e45d2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570568b22f0 con 0x5570568a68f0 2024-03-21T10:41:42.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.737+0000 7fc2e45d2740 10 client.0 _ll_register_callbacks cb 0x5570568031c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:42.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.737+0000 7fc2db400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570568af6c0 0x5570568ad1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:42.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.737+0000 7fc2db400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570568af6c0 0x5570568ad1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56364/0 (socket says 172.21.15.31:56364) 2024-03-21T10:41:42.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.737+0000 7fc2db400640 1 -- 172.21.15.31:0/3173000309 learned_addr learned my addr 172.21.15.31:0/3173000309 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:42.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.738+0000 7fc2da000640 1 --2- 172.21.15.31:0/3173000309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570568a68f0 0x5570568accb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:42.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.738+0000 7fc2daa00640 1 --2- 172.21.15.31:0/3173000309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5570568a5fd0 0x5570568ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:42.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.738+0000 7fc2db400640 1 -- 172.21.15.31:0/3173000309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570568a68f0 msgr2=0x5570568accb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:42.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.738+0000 7fc2db400640 1 --2- 172.21.15.31:0/3173000309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570568a68f0 0x5570568accb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.738+0000 7fc2db400640 1 -- 172.21.15.31:0/3173000309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5570568a5fd0 msgr2=0x5570568ac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:42.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.738+0000 7fc2db400640 1 --2- 172.21.15.31:0/3173000309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5570568a5fd0 0x5570568ac770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.738+0000 7fc2db400640 1 -- 172.21.15.31:0/3173000309 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55705651f210 con 0x5570568af6c0 2024-03-21T10:41:42.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.739+0000 7fc2da000640 1 --2- 172.21.15.31:0/3173000309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570568a68f0 0x5570568accb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:41:42.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.739+0000 7fc2daa00640 1 --2- 172.21.15.31:0/3173000309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5570568a5fd0 0x5570568ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:41:42.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.740+0000 7fc2db400640 1 --2- 172.21.15.31:0/3173000309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570568af6c0 0x5570568ad1f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc2d40117f0 tx=0x7fc2d4011cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75a141f1855d6536 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:42.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.740+0000 7fc2c4c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:41:42.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.740+0000 7fc2c4c00640 1 -- 172.21.15.31:0/3173000309 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4011ea0 con 0x5570568af6c0 2024-03-21T10:41:42.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.740+0000 7fc2c4c00640 1 -- 172.21.15.31:0/3173000309 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fc2d4004d20 con 0x5570568af6c0 2024-03-21T10:41:42.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.741+0000 7fc2e45d2740 1 -- 172.21.15.31:0/3173000309 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc2cc009da0 con 0x5570568af6c0 2024-03-21T10:41:42.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.742+0000 7fc2c4c00640 1 -- 172.21.15.31:0/3173000309 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc2d400cb20 con 0x5570568af6c0 2024-03-21T10:41:42.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.746+0000 7fc2e45d2740 10 client.26417 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:42.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.746+0000 7fc2e45d2740 1 -- 172.21.15.31:0/3173000309 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5570565eb620 con 0x5570568af6c0 2024-03-21T10:41:42.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.748+0000 7fc2c4c00640 1 -- 172.21.15.31:0/3173000309 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2cc009da0 con 0x5570568af6c0 2024-03-21T10:41:42.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.749+0000 7fc2c4c00640 1 -- 172.21.15.31:0/3173000309 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fc2d4024470 con 0x5570568af6c0 2024-03-21T10:41:42.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.749+0000 7fc2e45d2740 10 client.26417 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:42.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.749+0000 7fc2e45d2740 10 client.26417 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:41:42.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.749+0000 7fc2e45d2740 20 client.26417 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:42.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.750+0000 7fc2e45d2740 1 --2- 172.21.15.31:0/3173000309 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5570568bf8d0 0x5570568c1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:42.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.750+0000 7fc2e45d2740 4 client.26417 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:41:42.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.750+0000 7fc2e45d2740 1 -- 172.21.15.31:0/3173000309 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557056817d70 con 0x5570568bf8d0 2024-03-21T10:41:42.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.751+0000 7fc2daa00640 1 --2- 172.21.15.31:0/3173000309 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5570568bf8d0 0x5570568c1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:42.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.753+0000 7fc2c4c00640 10 client.26417 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:42.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.753+0000 7fc2daa00640 1 --2- 172.21.15.31:0/3173000309 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5570568bf8d0 0x5570568c1cc0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=daa4ee8399bdeeec server_cookie=de2d9ba60fd8a605 in_seq=0 out_seq=0 2024-03-21T10:41:42.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.754+0000 7fc2c4c00640 1 -- 172.21.15.31:0/3173000309 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557056817d70 con 0x5570568bf8d0 2024-03-21T10:41:42.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.754+0000 7fc2c4c00640 10 client.26417 handle_command_reply: tid=0 2024-03-21T10:41:42.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.755+0000 7fc2e45d2740 1 client.26417 shutdown 2024-03-21T10:41:42.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.756+0000 7fc2e45d2740 1 -- 172.21.15.31:0/3173000309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570568af6c0 msgr2=0x5570568ad1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:42.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.756+0000 7fc2e45d2740 1 --2- 172.21.15.31:0/3173000309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570568af6c0 0x5570568ad1f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc2d40117f0 tx=0x7fc2d4011cb0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.756+0000 7fc2e45d2740 1 -- 172.21.15.31:0/3173000309 shutdown_connections 2024-03-21T10:41:42.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.756+0000 7fc2e45d2740 1 --2- 172.21.15.31:0/3173000309 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5570568bf8d0 0x5570568c1cc0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.756+0000 7fc2e45d2740 1 --2- 172.21.15.31:0/3173000309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570568af6c0 0x5570568ad1f0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.756+0000 7fc2e45d2740 1 --2- 172.21.15.31:0/3173000309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570568a68f0 0x5570568accb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.756+0000 7fc2e45d2740 1 --2- 172.21.15.31:0/3173000309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5570568a5fd0 0x5570568ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.756+0000 7fc2e45d2740 1 -- 172.21.15.31:0/3173000309 >> 172.21.15.31:0/3173000309 conn(0x5570568a08f0 msgr2=0x5570568a2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:42.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.758+0000 7fc2c4c00640 0 client.26417 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:42.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.759+0000 7fc2e45d2740 1 -- 172.21.15.31:0/3173000309 shutdown_connections 2024-03-21T10:41:42.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.759+0000 7fc2e45d2740 1 -- 172.21.15.31:0/3173000309 wait complete. 2024-03-21T10:41:42.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.760+0000 7fc2e45d2740 20 client.26417 trim_cache size 0 max 16384 2024-03-21T10:41:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.799+0000 7fc2e45d2740 1 Processor -- start 2024-03-21T10:41:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.800+0000 7fc2e45d2740 1 -- start start 2024-03-21T10:41:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.800+0000 7fc2e45d2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5570568a5fd0 0x5570568aeea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.800+0000 7fc2e45d2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570568a68f0 0x5570568ab080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.800+0000 7fc2e45d2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570568af6c0 0x5570568acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.801+0000 7fc2e45d2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc1d9330 con 0x5570568a5fd0 2024-03-21T10:41:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.801+0000 7fc2e45d2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc1d6e30 con 0x5570568af6c0 2024-03-21T10:41:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.801+0000 7fc2db400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570568af6c0 0x5570568acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.801+0000 7fc2e45d2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc1d4ed0 con 0x5570568a68f0 2024-03-21T10:41:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.801+0000 7fc2db400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570568af6c0 0x5570568acd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56372/0 (socket says 172.21.15.31:56372) 2024-03-21T10:41:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.801+0000 7fc2db400640 1 -- 172.21.15.31:0/1588324270 learned_addr learned my addr 172.21.15.31:0/1588324270 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.802+0000 7fc2db400640 1 -- 172.21.15.31:0/1588324270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570568a68f0 msgr2=0x5570568ab080 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:41:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.802+0000 7fc2db400640 1 --2- 172.21.15.31:0/1588324270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570568a68f0 0x5570568ab080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.803+0000 7fc2db400640 1 -- 172.21.15.31:0/1588324270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5570568a5fd0 msgr2=0x5570568aeea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.804+0000 7fc2db400640 1 --2- 172.21.15.31:0/1588324270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5570568a5fd0 0x5570568aeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.804+0000 7fc2db400640 1 -- 172.21.15.31:0/1588324270 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570568ab760 con 0x5570568af6c0 2024-03-21T10:41:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.806+0000 7fc2db400640 1 --2- 172.21.15.31:0/1588324270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570568af6c0 0x5570568acd00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4002230 tx=0x7fc2d4005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb4c45c2cbdf0e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.806+0000 7fc2c7400640 1 -- 172.21.15.31:0/1588324270 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4011ea0 con 0x5570568af6c0 2024-03-21T10:41:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.806+0000 7fc2c7400640 1 -- 172.21.15.31:0/1588324270 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2d400c700 con 0x5570568af6c0 2024-03-21T10:41:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.807+0000 7fc2c7400640 1 -- 172.21.15.31:0/1588324270 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d40052a0 con 0x5570568af6c0 2024-03-21T10:41:42.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.808+0000 7fc2e45d2740 1 -- 172.21.15.31:0/1588324270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570568af6c0 msgr2=0x5570568acd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:42.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.809+0000 7fc2e45d2740 1 --2- 172.21.15.31:0/1588324270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570568af6c0 0x5570568acd00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4002230 tx=0x7fc2d4005fd0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.810+0000 7fc2e45d2740 1 -- 172.21.15.31:0/1588324270 shutdown_connections 2024-03-21T10:41:42.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.810+0000 7fc2e45d2740 1 --2- 172.21.15.31:0/1588324270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570568af6c0 0x5570568acd00 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.810+0000 7fc2e45d2740 1 --2- 172.21.15.31:0/1588324270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570568a68f0 0x5570568ab080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.810+0000 7fc2e45d2740 1 --2- 172.21.15.31:0/1588324270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5570568a5fd0 0x5570568aeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.810+0000 7fc2e45d2740 1 -- 172.21.15.31:0/1588324270 >> 172.21.15.31:0/1588324270 conn(0x5570568a08f0 msgr2=0x5570568a2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:42.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.811+0000 7fc2e45d2740 1 -- 172.21.15.31:0/1588324270 shutdown_connections 2024-03-21T10:41:42.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.812+0000 7fc2e45d2740 1 -- 172.21.15.31:0/1588324270 wait complete. 2024-03-21T10:41:42.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.813+0000 7fc2e45d2740 1 -- start start 2024-03-21T10:41:42.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.813+0000 7fc2e45d2740 1 Processor -- start 2024-03-21T10:41:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.817+0000 7fc2e45d2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570568a5fd0 0x5570568a9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.817+0000 7fc2e45d2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5570568a68f0 0x5570568a9580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.817+0000 7fc2e45d2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570568af6c0 0x5570568a9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.817+0000 7fc2e45d2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc1d54d0 con 0x5570568a68f0 2024-03-21T10:41:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.817+0000 7fc2e45d2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc1d67d0 con 0x5570568a5fd0 2024-03-21T10:41:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.817+0000 7fc2e45d2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc1d5b10 con 0x5570568af6c0 2024-03-21T10:41:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.817+0000 7fc2e45d2740 10 client.0 _ll_register_callbacks cb 0x7fc2d40095e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.817+0000 7fc2daa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570568a5fd0 0x5570568a9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.817+0000 7fc2daa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570568a5fd0 0x5570568a9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56386/0 (socket says 172.21.15.31:56386) 2024-03-21T10:41:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.817+0000 7fc2daa00640 1 -- 172.21.15.31:0/2684159181 learned_addr learned my addr 172.21.15.31:0/2684159181 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.818+0000 7fc2daa00640 1 -- 172.21.15.31:0/2684159181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570568af6c0 msgr2=0x5570568a9ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:41:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.818+0000 7fc2daa00640 1 --2- 172.21.15.31:0/2684159181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570568af6c0 0x5570568a9ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.819+0000 7fc2da000640 1 --2- 172.21.15.31:0/2684159181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5570568a68f0 0x5570568a9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.819+0000 7fc2daa00640 1 -- 172.21.15.31:0/2684159181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5570568a68f0 msgr2=0x5570568a9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.819+0000 7fc2daa00640 1 --2- 172.21.15.31:0/2684159181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5570568a68f0 0x5570568a9580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.819+0000 7fc2daa00640 1 -- 172.21.15.31:0/2684159181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55705691a210 con 0x5570568a5fd0 2024-03-21T10:41:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.819+0000 7fc2daa00640 1 --2- 172.21.15.31:0/2684159181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570568a5fd0 0x5570568a9040 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc2c802c790 tx=0x7fc2c802bcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c710cea1a49bbbdd server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.820+0000 7fc2c4c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:41:42.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.820+0000 7fc2c4c00640 1 -- 172.21.15.31:0/2684159181 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c8054160 con 0x5570568a5fd0 2024-03-21T10:41:42.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.821+0000 7fc2c4c00640 1 -- 172.21.15.31:0/2684159181 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fc2c8004c90 con 0x5570568a5fd0 2024-03-21T10:41:42.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.821+0000 7fc2e45d2740 1 -- 172.21.15.31:0/2684159181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc2d400c700 con 0x5570568a5fd0 2024-03-21T10:41:42.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.821+0000 7fc2c4c00640 1 -- 172.21.15.31:0/2684159181 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc2c8064050 con 0x5570568a5fd0 2024-03-21T10:41:42.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.821+0000 7fc2e45d2740 10 client.26423 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:42.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.821+0000 7fc2e45d2740 1 -- 172.21.15.31:0/2684159181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55705691b580 con 0x5570568a5fd0 2024-03-21T10:41:42.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.822+0000 7fc2c4c00640 1 -- 172.21.15.31:0/2684159181 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2d400c700 con 0x5570568a5fd0 2024-03-21T10:41:42.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.822+0000 7fc2c4c00640 1 -- 172.21.15.31:0/2684159181 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fc2c804e040 con 0x5570568a5fd0 2024-03-21T10:41:42.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.822+0000 7fc2e45d2740 10 client.26423 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:42.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.822+0000 7fc2e45d2740 10 client.26423 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:41:42.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.822+0000 7fc2e45d2740 20 client.26423 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:42.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.822+0000 7fc2e45d2740 1 --2- 172.21.15.31:0/2684159181 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55705691bd40 0x55705691e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:42.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.822+0000 7fc2e45d2740 4 client.26423 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:41:42.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.822+0000 7fc2e45d2740 1 -- 172.21.15.31:0/2684159181 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5570568120c0 con 0x55705691bd40 2024-03-21T10:41:42.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.823+0000 7fc2da000640 1 --2- 172.21.15.31:0/2684159181 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55705691bd40 0x55705691e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:42.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.825+0000 7fc2da000640 1 --2- 172.21.15.31:0/2684159181 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55705691bd40 0x55705691e160 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad7a1f872371f128 server_cookie=fda2ac3f8f2e60ce in_seq=0 out_seq=0 2024-03-21T10:41:42.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.825+0000 7fc2c4c00640 10 client.26423 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:42.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.863+0000 7fc2c4c00640 1 -- 172.21.15.31:0/2684159181 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5570568120c0 con 0x55705691bd40 2024-03-21T10:41:42.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.863+0000 7fc2c4c00640 10 client.26423 handle_command_reply: tid=0 2024-03-21T10:41:42.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.863+0000 7fc2e45d2740 1 client.26423 shutdown 2024-03-21T10:41:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.864+0000 7fc2e45d2740 1 -- 172.21.15.31:0/2684159181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570568a5fd0 msgr2=0x5570568a9040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.864+0000 7fc2e45d2740 1 --2- 172.21.15.31:0/2684159181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570568a5fd0 0x5570568a9040 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc2c802c790 tx=0x7fc2c802bcf0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.864+0000 7fc2e45d2740 1 -- 172.21.15.31:0/2684159181 shutdown_connections 2024-03-21T10:41:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.864+0000 7fc2e45d2740 1 --2- 172.21.15.31:0/2684159181 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55705691bd40 0x55705691e160 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.864+0000 7fc2e45d2740 1 --2- 172.21.15.31:0/2684159181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570568af6c0 0x5570568a9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.864+0000 7fc2e45d2740 1 --2- 172.21.15.31:0/2684159181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5570568a68f0 0x5570568a9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.864+0000 7fc2e45d2740 1 --2- 172.21.15.31:0/2684159181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570568a5fd0 0x5570568a9040 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.864+0000 7fc2e45d2740 1 -- 172.21.15.31:0/2684159181 >> 172.21.15.31:0/2684159181 conn(0x5570568a08f0 msgr2=0x55705689f950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.864+0000 7fc2c4c00640 0 client.26423 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:42.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.865+0000 7fc2e45d2740 1 -- 172.21.15.31:0/2684159181 shutdown_connections 2024-03-21T10:41:42.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.865+0000 7fc2e45d2740 1 -- 172.21.15.31:0/2684159181 wait complete. 2024-03-21T10:41:42.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.865+0000 7fc2e45d2740 20 client.26423 trim_cache size 0 max 16384 2024-03-21T10:41:42.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:41:42.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:41:42.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:41:42.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:41:42.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.867+0000 7fc2e1800640 1 -- 172.21.15.31:0/3133911926 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc2c00a34c0 msgr2=0x7fc2c00a5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:42.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.867+0000 7fc2e1800640 1 --2- 172.21.15.31:0/3133911926 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc2c00a34c0 0x7fc2c00a5980 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc2c8005c40 tx=0x7fc2c8000bc0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.867+0000 7fc2e1800640 1 -- 172.21.15.31:0/3133911926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2dc109230 msgr2=0x7fc2dc07e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:42.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.867+0000 7fc2e1800640 1 --2- 172.21.15.31:0/3133911926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2dc109230 0x7fc2dc07e290 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc005fd0 tx=0x7fc2cc00dde0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.867+0000 7fc2e1800640 1 -- 172.21.15.31:0/3133911926 shutdown_connections 2024-03-21T10:41:42.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.867+0000 7fc2e1800640 1 --2- 172.21.15.31:0/3133911926 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc2c00a34c0 0x7fc2c00a5980 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc2c8005c40 tx=0x7fc2c8000bc0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.867+0000 7fc2e1800640 1 --2- 172.21.15.31:0/3133911926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2dc10cfe0 0x7fc2dc07e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.867+0000 7fc2e1800640 1 --2- 172.21.15.31:0/3133911926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2dc109230 0x7fc2dc07e290 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.867+0000 7fc2e1800640 1 --2- 172.21.15.31:0/3133911926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2dc108910 0x7fc2dc0841d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:42.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.867+0000 7fc2e1800640 1 -- 172.21.15.31:0/3133911926 >> 172.21.15.31:0/3133911926 conn(0x7fc2dc06c750 msgr2=0x7fc2dc071010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.871+0000 7fc2e1800640 1 -- 172.21.15.31:0/3133911926 shutdown_connections 2024-03-21T10:41:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:42.871+0000 7fc2e1800640 1 -- 172.21.15.31:0/3133911926 wait complete. 2024-03-21T10:41:43.138 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:41:43.139 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:41:43.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:42 smithi040 ceph-mon[36154]: pgmap v97: 129 pgs: 129 active+clean; 13 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 233 MiB/s wr, 1.61k op/s 2024-03-21T10:41:43.214 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:42 smithi031 ceph-mon[31673]: pgmap v97: 129 pgs: 129 active+clean; 13 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 233 MiB/s wr, 1.61k op/s 2024-03-21T10:41:43.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:42 smithi070 ceph-mon[36330]: pgmap v97: 129 pgs: 129 active+clean; 13 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 233 MiB/s wr, 1.61k op/s 2024-03-21T10:41:43.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.475+0000 7fb861200640 1 -- 172.21.15.31:0/3979670795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb85c100420 msgr2=0x7fb85c100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:43.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.475+0000 7fb861200640 1 --2- 172.21.15.31:0/3979670795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb85c100420 0x7fb85c100800 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb84c009900 tx=0x7fb84c02bcd0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.475+0000 7fb861200640 1 -- 172.21.15.31:0/3979670795 shutdown_connections 2024-03-21T10:41:43.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.475+0000 7fb861200640 1 --2- 172.21.15.31:0/3979670795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb85c105820 0x7fb85c105c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.475+0000 7fb861200640 1 --2- 172.21.15.31:0/3979670795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb85c100d40 0x7fb85c105200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.475+0000 7fb861200640 1 --2- 172.21.15.31:0/3979670795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb85c100420 0x7fb85c100800 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.475+0000 7fb861200640 1 -- 172.21.15.31:0/3979670795 >> 172.21.15.31:0/3979670795 conn(0x7fb85c077130 msgr2=0x7fb85c0fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:43.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.475+0000 7fb861200640 1 -- 172.21.15.31:0/3979670795 shutdown_connections 2024-03-21T10:41:43.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.475+0000 7fb861200640 1 -- 172.21.15.31:0/3979670795 wait complete. 2024-03-21T10:41:43.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.476+0000 7fb861200640 1 Processor -- start 2024-03-21T10:41:43.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.476+0000 7fb861200640 1 -- start start 2024-03-21T10:41:43.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.477+0000 7fb861200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb85c100420 0x7fb85c19e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:43.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.477+0000 7fb861200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb85c100d40 0x7fb85c19e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:43.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.477+0000 7fb861200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb85c105820 0x7fb85c1a2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:43.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.477+0000 7fb861200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c117e80 con 0x7fb85c100d40 2024-03-21T10:41:43.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.477+0000 7fb861200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c117d00 con 0x7fb85c100420 2024-03-21T10:41:43.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.477+0000 7fb861200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c118000 con 0x7fb85c105820 2024-03-21T10:41:43.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.477+0000 7fb85b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb85c100420 0x7fb85c19e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:43.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.477+0000 7fb85b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb85c100420 0x7fb85c19e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56400/0 (socket says 172.21.15.31:56400) 2024-03-21T10:41:43.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.477+0000 7fb85b400640 1 -- 172.21.15.31:0/1143901329 learned_addr learned my addr 172.21.15.31:0/1143901329 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:43.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.477+0000 7fb85aa00640 1 --2- 172.21.15.31:0/1143901329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb85c100d40 0x7fb85c19e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:43.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.477+0000 7fb85be00640 1 --2- 172.21.15.31:0/1143901329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb85c105820 0x7fb85c1a2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:43.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.478+0000 7fb85b400640 1 -- 172.21.15.31:0/1143901329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb85c105820 msgr2=0x7fb85c1a2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:43.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.478+0000 7fb85b400640 1 --2- 172.21.15.31:0/1143901329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb85c105820 0x7fb85c1a2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.478+0000 7fb85b400640 1 -- 172.21.15.31:0/1143901329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb85c100d40 msgr2=0x7fb85c19e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:43.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.478+0000 7fb85b400640 1 --2- 172.21.15.31:0/1143901329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb85c100d40 0x7fb85c19e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.478+0000 7fb85b400640 1 -- 172.21.15.31:0/1143901329 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85c1a3310 con 0x7fb85c100420 2024-03-21T10:41:43.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.478+0000 7fb85b400640 1 --2- 172.21.15.31:0/1143901329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb85c100420 0x7fb85c19e320 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb84c000c00 tx=0x7fb84c004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:43.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.478+0000 7fb853e00640 1 -- 172.21.15.31:0/1143901329 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c042020 con 0x7fb85c100420 2024-03-21T10:41:43.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.478+0000 7fb861200640 1 -- 172.21.15.31:0/1143901329 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb85c1a35a0 con 0x7fb85c100420 2024-03-21T10:41:43.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.478+0000 7fb853e00640 1 -- 172.21.15.31:0/1143901329 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb84c0054e0 con 0x7fb85c100420 2024-03-21T10:41:43.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.479+0000 7fb861200640 1 -- 172.21.15.31:0/1143901329 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb85c1aae90 con 0x7fb85c100420 2024-03-21T10:41:43.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.479+0000 7fb853e00640 1 -- 172.21.15.31:0/1143901329 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c005000 con 0x7fb85c100420 2024-03-21T10:41:43.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.483+0000 7fb865413740 1 Processor -- start 2024-03-21T10:41:43.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.483+0000 7fb865413740 1 -- start start 2024-03-21T10:41:43.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.483+0000 7fb865413740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565222d29dd0 0x565222d28b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:43.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.484+0000 7fb865413740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565222d290f0 0x565222d29570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:43.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.484+0000 7fb865413740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565222d32460 0x565222d34850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:43.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.484+0000 7fb865413740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x565222c8d250 con 0x565222d32460 2024-03-21T10:41:43.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.484+0000 7fb865413740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x565222c35a30 con 0x565222d29dd0 2024-03-21T10:41:43.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.484+0000 7fb865413740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x565222c892f0 con 0x565222d290f0 2024-03-21T10:41:43.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.484+0000 7fb85be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565222d29dd0 0x565222d28b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:43.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.484+0000 7fb85be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565222d29dd0 0x565222d28b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56404/0 (socket says 172.21.15.31:56404) 2024-03-21T10:41:43.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.484+0000 7fb85be00640 1 -- 172.21.15.31:0/2291477461 learned_addr learned my addr 172.21.15.31:0/2291477461 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:43.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.484+0000 7fb85aa00640 1 --2- 172.21.15.31:0/2291477461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565222d32460 0x565222d34850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:43.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.484+0000 7fb85be00640 1 -- 172.21.15.31:0/2291477461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565222d290f0 msgr2=0x565222d29570 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:41:43.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.484+0000 7fb853e00640 1 -- 172.21.15.31:0/1143901329 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fb84c045020 con 0x7fb85c100420 2024-03-21T10:41:43.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.484+0000 7fb85be00640 1 --2- 172.21.15.31:0/2291477461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565222d290f0 0x565222d29570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.484+0000 7fb85be00640 1 -- 172.21.15.31:0/2291477461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565222d32460 msgr2=0x565222d34850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:43.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.484+0000 7fb85be00640 1 --2- 172.21.15.31:0/2291477461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565222d32460 0x565222d34850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.484+0000 7fb85be00640 1 -- 172.21.15.31:0/2291477461 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565222c20690 con 0x565222d29dd0 2024-03-21T10:41:43.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.485+0000 7fb85be00640 1 --2- 172.21.15.31:0/2291477461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565222d29dd0 0x565222d28b20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb854009f40 tx=0x7fb854010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad96677da471e206 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:43.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.485+0000 7fb850c00640 1 -- 172.21.15.31:0/2291477461 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb854003b70 con 0x565222d29dd0 2024-03-21T10:41:43.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.485+0000 7fb850c00640 1 -- 172.21.15.31:0/2291477461 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb854003d10 con 0x565222d29dd0 2024-03-21T10:41:43.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.485+0000 7fb850c00640 1 -- 172.21.15.31:0/2291477461 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb854011af0 con 0x565222d29dd0 2024-03-21T10:41:43.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.486+0000 7fb853e00640 1 --2- 172.21.15.31:0/1143901329 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb8400a3110 0x7fb8400a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:43.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.486+0000 7fb865413740 1 -- 172.21.15.31:0/2291477461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565222d29dd0 msgr2=0x565222d28b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:43.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.486+0000 7fb865413740 1 --2- 172.21.15.31:0/2291477461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565222d29dd0 0x565222d28b20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb854009f40 tx=0x7fb854010860 comp rx=0 tx=0).stop 2024-03-21T10:41:43.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.486+0000 7fb853e00640 1 -- 172.21.15.31:0/1143901329 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fb84c0f0da0 con 0x7fb85c100420 2024-03-21T10:41:43.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.486+0000 7fb85b400640 1 --2- 172.21.15.31:0/1143901329 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb8400a3110 0x7fb8400a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:43.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.487+0000 7fb865413740 1 -- 172.21.15.31:0/2291477461 shutdown_connections 2024-03-21T10:41:43.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.487+0000 7fb865413740 1 --2- 172.21.15.31:0/2291477461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565222d32460 0x565222d34850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.487+0000 7fb865413740 1 --2- 172.21.15.31:0/2291477461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565222d290f0 0x565222d29570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.487+0000 7fb865413740 1 --2- 172.21.15.31:0/2291477461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565222d29dd0 0x565222d28b20 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.487+0000 7fb865413740 1 -- 172.21.15.31:0/2291477461 >> 172.21.15.31:0/2291477461 conn(0x565222cf0200 msgr2=0x565222d23b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:43.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.487+0000 7fb865413740 1 -- 172.21.15.31:0/2291477461 shutdown_connections 2024-03-21T10:41:43.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.487+0000 7fb865413740 1 -- 172.21.15.31:0/2291477461 wait complete. 2024-03-21T10:41:43.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.488+0000 7fb85b400640 1 --2- 172.21.15.31:0/1143901329 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb8400a3110 0x7fb8400a55d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x565222d2a320 tx=0x7fb84c004610 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:43.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.488+0000 7fb865413740 1 -- start start 2024-03-21T10:41:43.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.489+0000 7fb865413740 1 Processor -- start 2024-03-21T10:41:43.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.489+0000 7fb865413740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565222d290f0 0x565222d31b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:43.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.489+0000 7fb865413740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565222d29dd0 0x565222d2eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:43.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.489+0000 7fb865413740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565222d32460 0x565222d2f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:43.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.489+0000 7fb865413740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x565222d352d0 con 0x565222d32460 2024-03-21T10:41:43.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.489+0000 7fb865413740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x565222c48e80 con 0x565222d29dd0 2024-03-21T10:41:43.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.489+0000 7fb865413740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x565222d35450 con 0x565222d290f0 2024-03-21T10:41:43.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.489+0000 7fb865413740 10 client.0 _ll_register_callbacks cb 0x565222c5b130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:43.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.490+0000 7fb85aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565222d29dd0 0x565222d2eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:43.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.490+0000 7fb85aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565222d29dd0 0x565222d2eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56410/0 (socket says 172.21.15.31:56410) 2024-03-21T10:41:43.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.490+0000 7fb85aa00640 1 -- 172.21.15.31:0/2002151471 learned_addr learned my addr 172.21.15.31:0/2002151471 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:43.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.490+0000 7fb85be00640 1 --2- 172.21.15.31:0/2002151471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565222d290f0 0x565222d31b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:43.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.490+0000 7fb85aa00640 1 -- 172.21.15.31:0/2002151471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565222d290f0 msgr2=0x565222d31b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:43.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.491+0000 7fb85aa00640 1 --2- 172.21.15.31:0/2002151471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565222d290f0 0x565222d31b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.491+0000 7fb85aa00640 1 -- 172.21.15.31:0/2002151471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565222d32460 msgr2=0x565222d2f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:43.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.491+0000 7fb85aa00640 1 --2- 172.21.15.31:0/2002151471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565222d32460 0x565222d2f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.491+0000 7fb85aa00640 1 -- 172.21.15.31:0/2002151471 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565222a6b7d0 con 0x565222d29dd0 2024-03-21T10:41:43.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.491+0000 7fb85be00640 1 --2- 172.21.15.31:0/2002151471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565222d290f0 0x565222d31b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:41:43.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.491+0000 7fb85aa00640 1 --2- 172.21.15.31:0/2002151471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565222d29dd0 0x565222d2eef0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb8480124d0 tx=0x7fb848012990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=893639e869510eff server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:43.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.491+0000 7fb845600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:41:43.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.491+0000 7fb845600640 1 -- 172.21.15.31:0/2002151471 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84801f060 con 0x565222d29dd0 2024-03-21T10:41:43.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.491+0000 7fb845600640 1 -- 172.21.15.31:0/2002151471 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fb8480107e0 con 0x565222d29dd0 2024-03-21T10:41:43.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.492+0000 7fb865413740 1 -- 172.21.15.31:0/2002151471 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb854003d10 con 0x565222d29dd0 2024-03-21T10:41:43.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.492+0000 7fb845600640 1 -- 172.21.15.31:0/2002151471 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb848026050 con 0x565222d29dd0 2024-03-21T10:41:43.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.492+0000 7fb865413740 10 client.26435 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:43.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.492+0000 7fb865413740 1 -- 172.21.15.31:0/2002151471 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5652229a13f0 con 0x565222d29dd0 2024-03-21T10:41:43.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.492+0000 7fb845600640 1 -- 172.21.15.31:0/2002151471 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb854003d10 con 0x565222d29dd0 2024-03-21T10:41:43.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.495+0000 7fb845600640 1 -- 172.21.15.31:0/2002151471 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fb84801ac40 con 0x565222d29dd0 2024-03-21T10:41:43.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.495+0000 7fb865413740 10 client.26435 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:43.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.495+0000 7fb865413740 10 client.26435 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:41:43.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.495+0000 7fb865413740 20 client.26435 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:43.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.495+0000 7fb865413740 1 --2- 172.21.15.31:0/2002151471 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x565222d42410 0x565222d44800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:43.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.495+0000 7fb865413740 4 client.26435 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:41:43.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.495+0000 7fb865413740 1 -- 172.21.15.31:0/2002151471 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565222c9a790 con 0x565222d42410 2024-03-21T10:41:43.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.495+0000 7fb85be00640 1 --2- 172.21.15.31:0/2002151471 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x565222d42410 0x565222d44800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:43.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.496+0000 7fb85be00640 1 --2- 172.21.15.31:0/2002151471 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x565222d42410 0x565222d44800 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2c523d1ae53e755 server_cookie=bff72f2c7a9a3a76 in_seq=0 out_seq=0 2024-03-21T10:41:43.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.496+0000 7fb845600640 10 client.26435 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:43.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.497+0000 7fb845600640 1 -- 172.21.15.31:0/2002151471 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x565222c9a790 con 0x565222d42410 2024-03-21T10:41:43.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.497+0000 7fb845600640 10 client.26435 handle_command_reply: tid=0 2024-03-21T10:41:43.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.497+0000 7fb865413740 1 client.26435 shutdown 2024-03-21T10:41:43.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.498+0000 7fb865413740 1 -- 172.21.15.31:0/2002151471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565222d29dd0 msgr2=0x565222d2eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:43.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.498+0000 7fb865413740 1 --2- 172.21.15.31:0/2002151471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565222d29dd0 0x565222d2eef0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb8480124d0 tx=0x7fb848012990 comp rx=0 tx=0).stop 2024-03-21T10:41:43.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.498+0000 7fb865413740 1 -- 172.21.15.31:0/2002151471 shutdown_connections 2024-03-21T10:41:43.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.498+0000 7fb865413740 1 --2- 172.21.15.31:0/2002151471 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x565222d42410 0x565222d44800 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.498+0000 7fb865413740 1 --2- 172.21.15.31:0/2002151471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565222d32460 0x565222d2f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.498+0000 7fb865413740 1 --2- 172.21.15.31:0/2002151471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565222d29dd0 0x565222d2eef0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.498+0000 7fb865413740 1 --2- 172.21.15.31:0/2002151471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565222d290f0 0x565222d31b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.498+0000 7fb865413740 1 -- 172.21.15.31:0/2002151471 >> 172.21.15.31:0/2002151471 conn(0x565222cf0200 msgr2=0x565222d25890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:43.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.499+0000 7fb845600640 0 client.26435 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:43.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.499+0000 7fb865413740 1 -- 172.21.15.31:0/2002151471 shutdown_connections 2024-03-21T10:41:43.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.499+0000 7fb865413740 1 -- 172.21.15.31:0/2002151471 wait complete. 2024-03-21T10:41:43.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.500+0000 7fb865413740 20 client.26435 trim_cache size 0 max 16384 2024-03-21T10:41:43.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.552+0000 7fb865413740 1 Processor -- start 2024-03-21T10:41:43.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.553+0000 7fb865413740 1 -- start start 2024-03-21T10:41:43.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.553+0000 7fb865413740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565222d290f0 0x565222d31620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:43.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.553+0000 7fb865413740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565222d29dd0 0x565222d31b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:43.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.553+0000 7fb865413740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565222d32460 0x565222d3dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:43.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.553+0000 7fb865413740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c1b7210 con 0x565222d29dd0 2024-03-21T10:41:43.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.553+0000 7fb865413740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c1b4d10 con 0x565222d290f0 2024-03-21T10:41:43.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.553+0000 7fb865413740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c1b2db0 con 0x565222d32460 2024-03-21T10:41:43.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.554+0000 7fb85be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565222d290f0 0x565222d31620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:43.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.554+0000 7fb85aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565222d29dd0 0x565222d31b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:43.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.554+0000 7fb85be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565222d290f0 0x565222d31620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56420/0 (socket says 172.21.15.31:56420) 2024-03-21T10:41:43.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.554+0000 7fb85be00640 1 -- 172.21.15.31:0/1754502913 learned_addr learned my addr 172.21.15.31:0/1754502913 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:43.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.554+0000 7fb85be00640 1 --2- 172.21.15.31:0/1754502913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565222d32460 0x565222d3dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:43.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.555+0000 7fb85be00640 1 -- 172.21.15.31:0/1754502913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565222d32460 msgr2=0x565222d3dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:43.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.555+0000 7fb85be00640 1 --2- 172.21.15.31:0/1754502913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565222d32460 0x565222d3dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.555+0000 7fb85be00640 1 -- 172.21.15.31:0/1754502913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565222d29dd0 msgr2=0x565222d31b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:43.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.555+0000 7fb85be00640 1 --2- 172.21.15.31:0/1754502913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565222d29dd0 0x565222d31b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.555+0000 7fb85be00640 1 -- 172.21.15.31:0/1754502913 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652229610f0 con 0x565222d290f0 2024-03-21T10:41:43.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.556+0000 7fb85be00640 1 --2- 172.21.15.31:0/1754502913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565222d290f0 0x565222d31620 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb854011760 tx=0x7fb8540116d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d81a0f6acd6fa416 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:43.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.556+0000 7fb847e00640 1 -- 172.21.15.31:0/1754502913 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb854011ac0 con 0x565222d290f0 2024-03-21T10:41:43.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.556+0000 7fb847e00640 1 -- 172.21.15.31:0/1754502913 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb854011c60 con 0x565222d290f0 2024-03-21T10:41:43.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.556+0000 7fb847e00640 1 -- 172.21.15.31:0/1754502913 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb854011ac0 con 0x565222d290f0 2024-03-21T10:41:43.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.557+0000 7fb865413740 1 -- 172.21.15.31:0/1754502913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565222d290f0 msgr2=0x565222d31620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:43.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.557+0000 7fb865413740 1 --2- 172.21.15.31:0/1754502913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565222d290f0 0x565222d31620 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb854011760 tx=0x7fb8540116d0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.557+0000 7fb865413740 1 -- 172.21.15.31:0/1754502913 shutdown_connections 2024-03-21T10:41:43.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.557+0000 7fb865413740 1 --2- 172.21.15.31:0/1754502913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565222d32460 0x565222d3dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.558+0000 7fb865413740 1 --2- 172.21.15.31:0/1754502913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565222d29dd0 0x565222d31b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.558+0000 7fb865413740 1 --2- 172.21.15.31:0/1754502913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565222d290f0 0x565222d31620 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.558+0000 7fb865413740 1 -- 172.21.15.31:0/1754502913 >> 172.21.15.31:0/1754502913 conn(0x565222cf0200 msgr2=0x565222d23670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:43.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.558+0000 7fb865413740 1 -- 172.21.15.31:0/1754502913 shutdown_connections 2024-03-21T10:41:43.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.558+0000 7fb865413740 1 -- 172.21.15.31:0/1754502913 wait complete. 2024-03-21T10:41:43.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.559+0000 7fb865413740 1 -- start start 2024-03-21T10:41:43.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.559+0000 7fb865413740 1 Processor -- start 2024-03-21T10:41:43.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.560+0000 7fb865413740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565222d290f0 0x565222d2c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:43.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.561+0000 7fb85be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565222d290f0 0x565222d2c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:43.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.561+0000 7fb865413740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565222d29dd0 0x565222d2c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:43.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.561+0000 7fb865413740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565222d32460 0x565222d32e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:43.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.561+0000 7fb85be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565222d290f0 0x565222d2c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50786/0 (socket says 172.21.15.31:50786) 2024-03-21T10:41:43.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.561+0000 7fb865413740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c1b33b0 con 0x565222d290f0 2024-03-21T10:41:43.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.561+0000 7fb85be00640 1 -- 172.21.15.31:0/3345416190 learned_addr learned my addr 172.21.15.31:0/3345416190 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:43.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.561+0000 7fb865413740 1 -- 172.21.15.31:0/3345416190 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c1b46b0 con 0x565222d32460 2024-03-21T10:41:43.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.561+0000 7fb865413740 1 -- 172.21.15.31:0/3345416190 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c1b39f0 con 0x565222d29dd0 2024-03-21T10:41:43.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.561+0000 7fb865413740 10 client.0 _ll_register_callbacks cb 0x7fb848009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:43.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.561+0000 7fb85be00640 1 --2- 172.21.15.31:0/3345416190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565222d32460 0x565222d32e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:43.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.561+0000 7fb85aa00640 1 --2- 172.21.15.31:0/3345416190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565222d29dd0 0x565222d2c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:43.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.562+0000 7fb85be00640 1 -- 172.21.15.31:0/3345416190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565222d29dd0 msgr2=0x565222d2c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:43.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.562+0000 7fb85be00640 1 --2- 172.21.15.31:0/3345416190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565222d29dd0 0x565222d2c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.562+0000 7fb85be00640 1 -- 172.21.15.31:0/3345416190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565222d32460 msgr2=0x565222d32e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:43.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.562+0000 7fb85be00640 1 --2- 172.21.15.31:0/3345416190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565222d32460 0x565222d32e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.562+0000 7fb85be00640 1 -- 172.21.15.31:0/3345416190 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565222d33380 con 0x565222d290f0 2024-03-21T10:41:43.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.562+0000 7fb85be00640 1 --2- 172.21.15.31:0/3345416190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565222d290f0 0x565222d2c3f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb854011600 tx=0x7fb854000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71c3adf10494579a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:43.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.562+0000 7fb845600640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:41:43.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.563+0000 7fb845600640 1 -- 172.21.15.31:0/3345416190 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb854027b10 con 0x565222d290f0 2024-03-21T10:41:43.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.563+0000 7fb845600640 1 -- 172.21.15.31:0/3345416190 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fb854027cb0 con 0x565222d290f0 2024-03-21T10:41:43.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.563+0000 7fb865413740 1 -- 172.21.15.31:0/3345416190 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb854011c60 con 0x565222d290f0 2024-03-21T10:41:43.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.563+0000 7fb845600640 1 -- 172.21.15.31:0/3345416190 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb85401b4c0 con 0x565222d290f0 2024-03-21T10:41:43.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.566+0000 7fb865413740 10 client.16158 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:43.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.566+0000 7fb865413740 1 -- 172.21.15.31:0/3345416190 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x565222d9e3f0 con 0x565222d290f0 2024-03-21T10:41:43.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.572+0000 7fb845600640 1 -- 172.21.15.31:0/3345416190 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb854011c60 con 0x565222d290f0 2024-03-21T10:41:43.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.572+0000 7fb845600640 1 -- 172.21.15.31:0/3345416190 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fb854041060 con 0x565222d290f0 2024-03-21T10:41:43.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.572+0000 7fb865413740 10 client.16158 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:43.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.572+0000 7fb865413740 10 client.16158 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:41:43.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.572+0000 7fb865413740 20 client.16158 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:43.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.572+0000 7fb865413740 1 --2- 172.21.15.31:0/3345416190 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x565222d9e720 0x565222da0b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:43.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.572+0000 7fb865413740 4 client.16158 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:41:43.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.572+0000 7fb865413740 1 -- 172.21.15.31:0/3345416190 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x565222c94ae0 con 0x565222d9e720 2024-03-21T10:41:43.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.575+0000 7fb85aa00640 1 --2- 172.21.15.31:0/3345416190 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x565222d9e720 0x565222da0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:43.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.576+0000 7fb85aa00640 1 --2- 172.21.15.31:0/3345416190 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x565222d9e720 0x565222da0b10 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29436e1beed6d5b8 server_cookie=8f3f0206a6455bb in_seq=0 out_seq=0 2024-03-21T10:41:43.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.577+0000 7fb845600640 10 client.16158 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:43.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.577+0000 7fb845600640 1 -- 172.21.15.31:0/3345416190 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x565222c94ae0 con 0x565222d9e720 2024-03-21T10:41:43.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.577+0000 7fb845600640 10 client.16158 handle_command_reply: tid=0 2024-03-21T10:41:43.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.577+0000 7fb865413740 1 client.16158 shutdown 2024-03-21T10:41:43.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.583+0000 7fb865413740 1 -- 172.21.15.31:0/3345416190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565222d290f0 msgr2=0x565222d2c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:43.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.583+0000 7fb865413740 1 --2- 172.21.15.31:0/3345416190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565222d290f0 0x565222d2c3f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb854011600 tx=0x7fb854000f40 comp rx=0 tx=0).stop 2024-03-21T10:41:43.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.584+0000 7fb865413740 1 -- 172.21.15.31:0/3345416190 shutdown_connections 2024-03-21T10:41:43.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.584+0000 7fb865413740 1 --2- 172.21.15.31:0/3345416190 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x565222d9e720 0x565222da0b10 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.584+0000 7fb865413740 1 --2- 172.21.15.31:0/3345416190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565222d32460 0x565222d32e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.584+0000 7fb865413740 1 --2- 172.21.15.31:0/3345416190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565222d29dd0 0x565222d2c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.584+0000 7fb865413740 1 --2- 172.21.15.31:0/3345416190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565222d290f0 0x565222d2c3f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.584+0000 7fb865413740 1 -- 172.21.15.31:0/3345416190 >> 172.21.15.31:0/3345416190 conn(0x565222cf0200 msgr2=0x565222d24f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:43.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.584+0000 7fb845600640 0 client.16158 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:43.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.585+0000 7fb865413740 1 -- 172.21.15.31:0/3345416190 shutdown_connections 2024-03-21T10:41:43.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.586+0000 7fb865413740 1 -- 172.21.15.31:0/3345416190 wait complete. 2024-03-21T10:41:43.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.586+0000 7fb865413740 20 client.16158 trim_cache size 0 max 16384 2024-03-21T10:41:43.594 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:41:43.594 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-03-21T10:41:43.594 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-03-21T10:41:43.594 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:41:43.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.600+0000 7fb861200640 1 -- 172.21.15.31:0/1143901329 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb8400a3110 msgr2=0x7fb8400a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:43.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.600+0000 7fb861200640 1 --2- 172.21.15.31:0/1143901329 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb8400a3110 0x7fb8400a55d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x565222d2a320 tx=0x7fb84c004610 comp rx=0 tx=0).stop 2024-03-21T10:41:43.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.600+0000 7fb861200640 1 -- 172.21.15.31:0/1143901329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb85c100420 msgr2=0x7fb85c19e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:43.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.600+0000 7fb861200640 1 --2- 172.21.15.31:0/1143901329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb85c100420 0x7fb85c19e320 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb84c000c00 tx=0x7fb84c004360 comp rx=0 tx=0).stop 2024-03-21T10:41:43.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.601+0000 7fb861200640 1 -- 172.21.15.31:0/1143901329 shutdown_connections 2024-03-21T10:41:43.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.601+0000 7fb861200640 1 --2- 172.21.15.31:0/1143901329 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb8400a3110 0x7fb8400a55d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.601+0000 7fb861200640 1 --2- 172.21.15.31:0/1143901329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb85c105820 0x7fb85c1a2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.601+0000 7fb861200640 1 --2- 172.21.15.31:0/1143901329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb85c100d40 0x7fb85c19e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.601+0000 7fb861200640 1 --2- 172.21.15.31:0/1143901329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb85c100420 0x7fb85c19e320 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.601+0000 7fb861200640 1 -- 172.21.15.31:0/1143901329 >> 172.21.15.31:0/1143901329 conn(0x7fb85c077130 msgr2=0x7fb85c1031d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:43.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.612+0000 7fb861200640 1 -- 172.21.15.31:0/1143901329 shutdown_connections 2024-03-21T10:41:43.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.612+0000 7fb861200640 1 -- 172.21.15.31:0/1143901329 wait complete. 2024-03-21T10:41:43.644 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-21T10:41:43.645 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:41:43.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.961+0000 7effd9e00640 1 -- 172.21.15.31:0/3360285932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effd4074370 msgr2=0x7effd410d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:43.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.961+0000 7effd9e00640 1 --2- 172.21.15.31:0/3360285932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effd4074370 0x7effd410d9c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7effc0009900 tx=0x7effc002bcd0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.961+0000 7effd9e00640 1 -- 172.21.15.31:0/3360285932 shutdown_connections 2024-03-21T10:41:43.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.961+0000 7effd9e00640 1 --2- 172.21.15.31:0/3360285932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd410df00 0x7effd41102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.962+0000 7effd9e00640 1 --2- 172.21.15.31:0/3360285932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effd4074370 0x7effd410d9c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.962+0000 7effd9e00640 1 --2- 172.21.15.31:0/3360285932 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effd4073a50 0x7effd4073e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.962+0000 7effd9e00640 1 -- 172.21.15.31:0/3360285932 >> 172.21.15.31:0/3360285932 conn(0x7effd40fd150 msgr2=0x7effd40ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:43.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.966+0000 7effd9e00640 1 -- 172.21.15.31:0/3360285932 shutdown_connections 2024-03-21T10:41:43.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.966+0000 7effd9e00640 1 -- 172.21.15.31:0/3360285932 wait complete. 2024-03-21T10:41:43.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.967+0000 7effd9e00640 1 Processor -- start 2024-03-21T10:41:43.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.969+0000 7effd9e00640 1 -- start start 2024-03-21T10:41:43.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.970+0000 7effd9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd4073a50 0x7effd419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:43.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.970+0000 7effd9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effd4074370 0x7effd419a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:43.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.970+0000 7effd9e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effd410df00 0x7effd419ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:43.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.970+0000 7effd9e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd4113c70 con 0x7effd4074370 2024-03-21T10:41:43.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.970+0000 7effd9e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd4113af0 con 0x7effd4073a50 2024-03-21T10:41:43.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.970+0000 7effd9e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd4113df0 con 0x7effd410df00 2024-03-21T10:41:43.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.970+0000 7effd2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd4073a50 0x7effd419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:43.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.970+0000 7effd3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effd410df00 0x7effd419ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:43.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.970+0000 7effd2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd4073a50 0x7effd419a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56442/0 (socket says 172.21.15.31:56442) 2024-03-21T10:41:43.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.970+0000 7effd2a00640 1 -- 172.21.15.31:0/1680761550 learned_addr learned my addr 172.21.15.31:0/1680761550 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:43.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.971+0000 7effd3400640 1 -- 172.21.15.31:0/1680761550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd4073a50 msgr2=0x7effd419a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:43.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.971+0000 7effd3400640 1 --2- 172.21.15.31:0/1680761550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd4073a50 0x7effd419a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.971+0000 7effd3400640 1 -- 172.21.15.31:0/1680761550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effd4074370 msgr2=0x7effd419a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:43.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.971+0000 7effd3400640 1 --2- 172.21.15.31:0/1680761550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effd4074370 0x7effd419a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:43.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.971+0000 7effd3400640 1 -- 172.21.15.31:0/1680761550 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd419f1c0 con 0x7effd410df00 2024-03-21T10:41:43.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.973+0000 7effd3400640 1 --2- 172.21.15.31:0/1680761550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effd410df00 0x7effd419ea90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7effcc009fd0 tx=0x7effcc00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:43.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.974+0000 7effd8a00640 1 -- 172.21.15.31:0/1680761550 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effcc00ce20 con 0x7effd410df00 2024-03-21T10:41:43.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.974+0000 7effd8a00640 1 -- 172.21.15.31:0/1680761550 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effcc005020 con 0x7effd410df00 2024-03-21T10:41:43.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.974+0000 7effd8a00640 1 -- 172.21.15.31:0/1680761550 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effcc010720 con 0x7effd410df00 2024-03-21T10:41:43.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.975+0000 7effd9e00640 1 -- 172.21.15.31:0/1680761550 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effd419f4b0 con 0x7effd410df00 2024-03-21T10:41:43.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.975+0000 7effd9e00640 1 -- 172.21.15.31:0/1680761550 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd41a6d40 con 0x7effd410df00 2024-03-21T10:41:43.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.976+0000 7effd9e00640 1 -- 172.21.15.31:0/1680761550 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effd4075100 con 0x7effd410df00 2024-03-21T10:41:43.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.980+0000 7effd8a00640 1 -- 172.21.15.31:0/1680761550 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7effcc005540 con 0x7effd410df00 2024-03-21T10:41:43.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.982+0000 7effd8a00640 1 --2- 172.21.15.31:0/1680761550 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7effb80a3210 0x7effb80a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:43.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.982+0000 7effd8a00640 1 -- 172.21.15.31:0/1680761550 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7effcc0d0650 con 0x7effd410df00 2024-03-21T10:41:43.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.983+0000 7effd2a00640 1 --2- 172.21.15.31:0/1680761550 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7effb80a3210 0x7effb80a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:43.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.984+0000 7effd2a00640 1 --2- 172.21.15.31:0/1680761550 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7effb80a3210 0x7effb80a56d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7effc4005fd0 tx=0x7effc400b2c0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:43.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:43.986+0000 7effd8a00640 1 -- 172.21.15.31:0/1680761550 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7effcc01d060 con 0x7effd410df00 2024-03-21T10:41:44.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.489+0000 7effd9e00640 1 -- 172.21.15.31:0/1680761550 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effd419b520 con 0x7effd410df00 2024-03-21T10:41:44.489 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:41:44.489 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:40:26.180798+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24596,24547]},"id":1}]} 2024-03-21T10:41:44.489 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 25 2024-03-21T10:41:44.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.491+0000 7effd8a00640 1 -- 172.21.15.31:0/1680761550 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11385 (secure 0 0 0) 0x7effcc0173d0 con 0x7effd410df00 2024-03-21T10:41:44.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.496+0000 7effd9e00640 1 -- 172.21.15.31:0/1680761550 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7effb80a3210 msgr2=0x7effb80a56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:44.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.496+0000 7effd9e00640 1 --2- 172.21.15.31:0/1680761550 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7effb80a3210 0x7effb80a56d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7effc4005fd0 tx=0x7effc400b2c0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.496+0000 7effd9e00640 1 -- 172.21.15.31:0/1680761550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effd410df00 msgr2=0x7effd419ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:44.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.496+0000 7effd9e00640 1 --2- 172.21.15.31:0/1680761550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effd410df00 0x7effd419ea90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7effcc009fd0 tx=0x7effcc00ef10 comp rx=0 tx=0).stop 2024-03-21T10:41:44.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.505+0000 7effd9e00640 1 -- 172.21.15.31:0/1680761550 shutdown_connections 2024-03-21T10:41:44.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.505+0000 7effd9e00640 1 --2- 172.21.15.31:0/1680761550 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7effb80a3210 0x7effb80a56d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.505+0000 7effd9e00640 1 --2- 172.21.15.31:0/1680761550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effd410df00 0x7effd419ea90 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.505+0000 7effd9e00640 1 --2- 172.21.15.31:0/1680761550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7effd4074370 0x7effd419a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.505+0000 7effd9e00640 1 --2- 172.21.15.31:0/1680761550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd4073a50 0x7effd419a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.505+0000 7effd9e00640 1 -- 172.21.15.31:0/1680761550 >> 172.21.15.31:0/1680761550 conn(0x7effd40fd150 msgr2=0x7effd40fe9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:44.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.507+0000 7effd9e00640 1 -- 172.21.15.31:0/1680761550 shutdown_connections 2024-03-21T10:41:44.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.507+0000 7effd9e00640 1 -- 172.21.15.31:0/1680761550 wait complete. 2024-03-21T10:41:44.542 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:41:44.542 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-21T10:41:44.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.898+0000 7f0603800640 1 -- 172.21.15.31:0/3755123149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05fc107dc0 msgr2=0x7f05fc108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:44.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.898+0000 7f0603800640 1 --2- 172.21.15.31:0/3755123149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05fc107dc0 0x7f05fc108220 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f05e8009890 tx=0x7f05e802bb70 comp rx=0 tx=0).stop 2024-03-21T10:41:44.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.904+0000 7f0603800640 1 -- 172.21.15.31:0/3755123149 shutdown_connections 2024-03-21T10:41:44.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.904+0000 7f0603800640 1 --2- 172.21.15.31:0/3755123149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05fc108760 0x7f05fc114c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.904+0000 7f0603800640 1 --2- 172.21.15.31:0/3755123149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05fc107dc0 0x7f05fc108220 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.904+0000 7f0603800640 1 --2- 172.21.15.31:0/3755123149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05fc109b70 0x7f05fc109f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.904+0000 7f0603800640 1 -- 172.21.15.31:0/3755123149 >> 172.21.15.31:0/3755123149 conn(0x7f05fc077540 msgr2=0x7f05fc1064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:44.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.905+0000 7f0603800640 1 -- 172.21.15.31:0/3755123149 shutdown_connections 2024-03-21T10:41:44.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.905+0000 7f0603800640 1 -- 172.21.15.31:0/3755123149 wait complete. 2024-03-21T10:41:44.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.907+0000 7f0603800640 1 Processor -- start 2024-03-21T10:41:44.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.907+0000 7f0603800640 1 -- start start 2024-03-21T10:41:44.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.908+0000 7f0603800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05fc107dc0 0x7f05fc072e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:44.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.909+0000 7f0603800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05fc108760 0x7f05fc073390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:44.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.909+0000 7f0600e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05fc107dc0 0x7f05fc072e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:44.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.909+0000 7f0603800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05fc109b70 0x7f05fc06c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:44.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.909+0000 7f0600e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05fc107dc0 0x7f05fc072e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:51530/0 (socket says 172.21.15.31:51530) 2024-03-21T10:41:44.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.909+0000 7f05fbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05fc108760 0x7f05fc073390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:44.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.909+0000 7f05fbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05fc108760 0x7f05fc073390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56466/0 (socket says 172.21.15.31:56466) 2024-03-21T10:41:44.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.910+0000 7f0601800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05fc109b70 0x7f05fc06c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:44.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.910+0000 7f0601800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05fc109b70 0x7f05fc06c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50822/0 (socket says 172.21.15.31:50822) 2024-03-21T10:41:44.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.910+0000 7f0600e00640 1 -- 172.21.15.31:0/700336398 learned_addr learned my addr 172.21.15.31:0/700336398 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:44.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.910+0000 7f0603800640 1 -- 172.21.15.31:0/700336398 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc1180f0 con 0x7f05fc109b70 2024-03-21T10:41:44.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.910+0000 7f0603800640 1 -- 172.21.15.31:0/700336398 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc117f70 con 0x7f05fc108760 2024-03-21T10:41:44.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.910+0000 7f0603800640 1 -- 172.21.15.31:0/700336398 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc118270 con 0x7f05fc107dc0 2024-03-21T10:41:44.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.910+0000 7f0600e00640 1 -- 172.21.15.31:0/700336398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05fc108760 msgr2=0x7f05fc073390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:44.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.910+0000 7f0600e00640 1 --2- 172.21.15.31:0/700336398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05fc108760 0x7f05fc073390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.910+0000 7f0600e00640 1 -- 172.21.15.31:0/700336398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05fc109b70 msgr2=0x7f05fc06c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:44.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.910+0000 7f0600e00640 1 --2- 172.21.15.31:0/700336398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05fc109b70 0x7f05fc06c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.910+0000 7f0600e00640 1 -- 172.21.15.31:0/700336398 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05fc06c970 con 0x7f05fc107dc0 2024-03-21T10:41:44.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.910+0000 7f0601800640 1 --2- 172.21.15.31:0/700336398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05fc109b70 0x7f05fc06c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:41:44.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.911+0000 7f0600e00640 1 --2- 172.21.15.31:0/700336398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05fc107dc0 0x7f05fc072e50 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f05ec00c920 tx=0x7f05ec00cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:44.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.911+0000 7f05f9600640 1 -- 172.21.15.31:0/700336398 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ec007c70 con 0x7f05fc107dc0 2024-03-21T10:41:44.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.911+0000 7f0603800640 1 -- 172.21.15.31:0/700336398 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05fc06cc60 con 0x7f05fc107dc0 2024-03-21T10:41:44.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.911+0000 7f05f9600640 1 -- 172.21.15.31:0/700336398 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05ec007e10 con 0x7f05fc107dc0 2024-03-21T10:41:44.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.911+0000 7f05f9600640 1 -- 172.21.15.31:0/700336398 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ec018890 con 0x7f05fc107dc0 2024-03-21T10:41:44.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.911+0000 7f0603800640 1 -- 172.21.15.31:0/700336398 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05fc070c60 con 0x7f05fc107dc0 2024-03-21T10:41:44.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.916+0000 7f05f9600640 1 -- 172.21.15.31:0/700336398 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f05ec018a70 con 0x7f05fc107dc0 2024-03-21T10:41:44.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.917+0000 7f0606597740 1 Processor -- start 2024-03-21T10:41:44.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.917+0000 7f0606597740 1 -- start start 2024-03-21T10:41:44.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.917+0000 7f0606597740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c8e3746d0 0x561c8e374ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:44.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.918+0000 7f0606597740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c8e375080 0x561c8e37d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:44.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.918+0000 7f0606597740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c8e37dc00 0x561c8e380050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:44.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.918+0000 7f0601800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c8e3746d0 0x561c8e374ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:44.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.918+0000 7f0601800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c8e3746d0 0x561c8e374ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56474/0 (socket says 172.21.15.31:56474) 2024-03-21T10:41:44.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.918+0000 7f0606597740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c8e2d82c0 con 0x561c8e37dc00 2024-03-21T10:41:44.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.918+0000 7f0606597740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c8e280c90 con 0x561c8e3746d0 2024-03-21T10:41:44.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.918+0000 7f0606597740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c8e2d4360 con 0x561c8e375080 2024-03-21T10:41:44.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.918+0000 7f0601800640 1 -- 172.21.15.31:0/1210772679 learned_addr learned my addr 172.21.15.31:0/1210772679 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:44.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.918+0000 7f0600e00640 1 --2- 172.21.15.31:0/1210772679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c8e375080 0x561c8e37d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:44.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.919+0000 7f05f9600640 1 --2- 172.21.15.31:0/700336398 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f05e00a3110 0x7f05e00a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:44.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.919+0000 7f0601800640 1 -- 172.21.15.31:0/1210772679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c8e375080 msgr2=0x561c8e37d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:44.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.919+0000 7f0601800640 1 --2- 172.21.15.31:0/1210772679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c8e375080 0x561c8e37d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.919+0000 7f0601800640 1 -- 172.21.15.31:0/1210772679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c8e37dc00 msgr2=0x561c8e380050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:44.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.919+0000 7f05fbe00640 1 --2- 172.21.15.31:0/1210772679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c8e37dc00 0x561c8e380050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:44.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.919+0000 7f0601800640 1 --2- 172.21.15.31:0/1210772679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c8e37dc00 0x561c8e380050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.919+0000 7f05f9600640 1 -- 172.21.15.31:0/700336398 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f05ec0cfcc0 con 0x7f05fc107dc0 2024-03-21T10:41:44.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.919+0000 7f0601800640 1 -- 172.21.15.31:0/1210772679 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c8e2a5080 con 0x561c8e3746d0 2024-03-21T10:41:44.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.919+0000 7f05fbe00640 1 --2- 172.21.15.31:0/1210772679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c8e37dc00 0x561c8e380050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:41:44.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.920+0000 7f0601800640 1 --2- 172.21.15.31:0/700336398 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f05e00a3110 0x7f05e00a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:44.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.920+0000 7f0601800640 1 --2- 172.21.15.31:0/1210772679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c8e3746d0 0x561c8e374ab0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f05f400f480 tx=0x7f05f400f880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d19cdcfb5c6ffab3 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:44.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.921+0000 7f0601800640 1 --2- 172.21.15.31:0/700336398 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f05e00a3110 0x7f05e00a55d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f05f4022000 tx=0x7f05f4023000 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:44.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.921+0000 7f05f2000640 1 -- 172.21.15.31:0/1210772679 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f400cd60 con 0x561c8e3746d0 2024-03-21T10:41:44.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.921+0000 7f05f2000640 1 -- 172.21.15.31:0/1210772679 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05f401bd50 con 0x561c8e3746d0 2024-03-21T10:41:44.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.921+0000 7f05f2000640 1 -- 172.21.15.31:0/1210772679 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f4019cd0 con 0x561c8e3746d0 2024-03-21T10:41:44.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.924+0000 7f0606597740 1 -- 172.21.15.31:0/1210772679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c8e3746d0 msgr2=0x561c8e374ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:44.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.924+0000 7f0606597740 1 --2- 172.21.15.31:0/1210772679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c8e3746d0 0x561c8e374ab0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f05f400f480 tx=0x7f05f400f880 comp rx=0 tx=0).stop 2024-03-21T10:41:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.924+0000 7f0606597740 1 -- 172.21.15.31:0/1210772679 shutdown_connections 2024-03-21T10:41:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.924+0000 7f0606597740 1 --2- 172.21.15.31:0/1210772679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c8e37dc00 0x561c8e380050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.924+0000 7f0606597740 1 --2- 172.21.15.31:0/1210772679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c8e375080 0x561c8e37d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.924+0000 7f0606597740 1 --2- 172.21.15.31:0/1210772679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c8e3746d0 0x561c8e374ab0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.924+0000 7f0606597740 1 -- 172.21.15.31:0/1210772679 >> 172.21.15.31:0/1210772679 conn(0x561c8e36eda0 msgr2=0x561c8e36f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.925+0000 7f0606597740 1 -- 172.21.15.31:0/1210772679 shutdown_connections 2024-03-21T10:41:44.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.925+0000 7f0606597740 1 -- 172.21.15.31:0/1210772679 wait complete. 2024-03-21T10:41:44.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.926+0000 7f0606597740 1 -- start start 2024-03-21T10:41:44.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.926+0000 7f0606597740 1 Processor -- start 2024-03-21T10:41:44.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.933+0000 7f0606597740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c8e3746d0 0x561c8e37d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:44.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.933+0000 7f0606597740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c8e375080 0x561c8e37a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:44.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.933+0000 7f0606597740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c8e37dc00 0x561c8e37b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:44.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.933+0000 7f0606597740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c8e380b70 con 0x561c8e37dc00 2024-03-21T10:41:44.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.934+0000 7f0606597740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c8e2940d0 con 0x561c8e375080 2024-03-21T10:41:44.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.934+0000 7f0606597740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c8e380cf0 con 0x561c8e3746d0 2024-03-21T10:41:44.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.934+0000 7f05fbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c8e375080 0x561c8e37a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:44.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.934+0000 7f0600e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c8e3746d0 0x561c8e37d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:44.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.934+0000 7f0606597740 10 client.0 _ll_register_callbacks cb 0x561c8e298e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:44.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.935+0000 7f05fbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c8e375080 0x561c8e37a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56478/0 (socket says 172.21.15.31:56478) 2024-03-21T10:41:44.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.935+0000 7f05fbe00640 1 -- 172.21.15.31:0/3146349640 learned_addr learned my addr 172.21.15.31:0/3146349640 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:44.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.935+0000 7f0601800640 1 --2- 172.21.15.31:0/3146349640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c8e37dc00 0x561c8e37b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:44.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.936+0000 7f0601800640 1 -- 172.21.15.31:0/3146349640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c8e3746d0 msgr2=0x561c8e37d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:44.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.936+0000 7f0601800640 1 --2- 172.21.15.31:0/3146349640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c8e3746d0 0x561c8e37d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.936+0000 7f0601800640 1 -- 172.21.15.31:0/3146349640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c8e375080 msgr2=0x561c8e37a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:44.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.936+0000 7f0601800640 1 --2- 172.21.15.31:0/3146349640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c8e375080 0x561c8e37a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.936+0000 7f0601800640 1 -- 172.21.15.31:0/3146349640 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561c8dfec3f0 con 0x561c8e37dc00 2024-03-21T10:41:44.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.937+0000 7f0601800640 1 --2- 172.21.15.31:0/3146349640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c8e37dc00 0x561c8e37b0b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f05f400fd80 tx=0x7f05f400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0a79f577700ea40 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:44.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.937+0000 7f05e6a00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:41:44.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.938+0000 7f05e6a00640 1 -- 172.21.15.31:0/3146349640 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f401ba00 con 0x561c8e37dc00 2024-03-21T10:41:44.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.938+0000 7f05e6a00640 1 -- 172.21.15.31:0/3146349640 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f05f4015040 con 0x561c8e37dc00 2024-03-21T10:41:44.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.938+0000 7f0606597740 1 -- 172.21.15.31:0/3146349640 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f05f401bd50 con 0x561c8e37dc00 2024-03-21T10:41:44.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.939+0000 7f05e6a00640 1 -- 172.21.15.31:0/3146349640 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05f401bba0 con 0x561c8e37dc00 2024-03-21T10:41:44.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.941+0000 7f0606597740 10 client.16170 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:44.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.941+0000 7f0606597740 1 -- 172.21.15.31:0/3146349640 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x561c8e0b67d0 con 0x561c8e37dc00 2024-03-21T10:41:44.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.941+0000 7f05e6a00640 1 -- 172.21.15.31:0/3146349640 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05f401bd50 con 0x561c8e37dc00 2024-03-21T10:41:44.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.942+0000 7f05e6a00640 1 -- 172.21.15.31:0/3146349640 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f05f400b040 con 0x561c8e37dc00 2024-03-21T10:41:44.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.942+0000 7f0606597740 10 client.16170 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:44.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.942+0000 7f0606597740 10 client.16170 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:41:44.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.942+0000 7f0606597740 20 client.16170 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:44.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.942+0000 7f0606597740 1 --2- 172.21.15.31:0/3146349640 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x561c8e38d960 0x561c8e38fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:44.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.942+0000 7f0606597740 4 client.16170 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:41:44.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.942+0000 7f0606597740 1 -- 172.21.15.31:0/3146349640 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561c8e2e5800 con 0x561c8e38d960 2024-03-21T10:41:44.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.943+0000 7f0600e00640 1 --2- 172.21.15.31:0/3146349640 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x561c8e38d960 0x561c8e38fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:44.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.944+0000 7f0600e00640 1 --2- 172.21.15.31:0/3146349640 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x561c8e38d960 0x561c8e38fd50 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=644557c2f7f5d677 server_cookie=583c143abfb96b29 in_seq=0 out_seq=0 2024-03-21T10:41:44.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.944+0000 7f05e6a00640 10 client.16170 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:41:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.945+0000 7f05e6a00640 1 -- 172.21.15.31:0/3146349640 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561c8e2e5800 con 0x561c8e38d960 2024-03-21T10:41:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.945+0000 7f05e6a00640 10 client.16170 handle_command_reply: tid=0 2024-03-21T10:41:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.945+0000 7f0606597740 1 client.16170 shutdown 2024-03-21T10:41:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.946+0000 7f0606597740 1 -- 172.21.15.31:0/3146349640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c8e37dc00 msgr2=0x561c8e37b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.946+0000 7f0606597740 1 --2- 172.21.15.31:0/3146349640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c8e37dc00 0x561c8e37b0b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f05f400fd80 tx=0x7f05f400cf90 comp rx=0 tx=0).stop 2024-03-21T10:41:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.946+0000 7f0606597740 1 -- 172.21.15.31:0/3146349640 shutdown_connections 2024-03-21T10:41:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.946+0000 7f0606597740 1 --2- 172.21.15.31:0/3146349640 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x561c8e38d960 0x561c8e38fd50 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.946+0000 7f05e6a00640 0 client.16170 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:41:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.946+0000 7f0606597740 1 --2- 172.21.15.31:0/3146349640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c8e37dc00 0x561c8e37b0b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.946+0000 7f0606597740 1 --2- 172.21.15.31:0/3146349640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c8e375080 0x561c8e37a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.946+0000 7f0606597740 1 --2- 172.21.15.31:0/3146349640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c8e3746d0 0x561c8e37d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.946+0000 7f0606597740 1 -- 172.21.15.31:0/3146349640 >> 172.21.15.31:0/3146349640 conn(0x561c8e36eda0 msgr2=0x561c8e370af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.947+0000 7f0606597740 1 -- 172.21.15.31:0/3146349640 shutdown_connections 2024-03-21T10:41:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.947+0000 7f0606597740 1 -- 172.21.15.31:0/3146349640 wait complete. 2024-03-21T10:41:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.947+0000 7f0606597740 20 client.16170 trim_cache size 0 max 16384 2024-03-21T10:41:44.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.972+0000 7f0606597740 1 Processor -- start 2024-03-21T10:41:44.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.972+0000 7f0606597740 1 -- start start 2024-03-21T10:41:44.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.972+0000 7f0606597740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c8e3746d0 0x561c8e36dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:44.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.972+0000 7f0606597740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c8e375080 0x561c8e37d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:44.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.972+0000 7f0606597740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c8e37dc00 0x561c8e37d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:44.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.972+0000 7f0606597740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc1bfb80 con 0x561c8e3746d0 2024-03-21T10:41:44.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.972+0000 7f0606597740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc1bd680 con 0x561c8e37dc00 2024-03-21T10:41:44.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.972+0000 7f0606597740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc1bb720 con 0x561c8e375080 2024-03-21T10:41:44.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.973+0000 7f05fbe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c8e375080 0x561c8e37d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:44.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.973+0000 7f05fbe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c8e375080 0x561c8e37d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:51554/0 (socket says 172.21.15.31:51554) 2024-03-21T10:41:44.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.973+0000 7f05fbe00640 1 -- 172.21.15.31:0/3013269470 learned_addr learned my addr 172.21.15.31:0/3013269470 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:44.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.973+0000 7f0600e00640 1 --2- 172.21.15.31:0/3013269470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c8e3746d0 0x561c8e36dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:44.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.973+0000 7f0601800640 1 --2- 172.21.15.31:0/3013269470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c8e37dc00 0x561c8e37d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:44.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.973+0000 7f05fbe00640 1 -- 172.21.15.31:0/3013269470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c8e37dc00 msgr2=0x561c8e37d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:44.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.973+0000 7f05fbe00640 1 --2- 172.21.15.31:0/3013269470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c8e37dc00 0x561c8e37d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.973+0000 7f05fbe00640 1 -- 172.21.15.31:0/3013269470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c8e3746d0 msgr2=0x561c8e36dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:44.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.973+0000 7f05fbe00640 1 --2- 172.21.15.31:0/3013269470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c8e3746d0 0x561c8e36dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.973+0000 7f05fbe00640 1 -- 172.21.15.31:0/3013269470 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c8e37bd60 con 0x561c8e375080 2024-03-21T10:41:44.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.973+0000 7f0601800640 1 --2- 172.21.15.31:0/3013269470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c8e37dc00 0x561c8e37d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:41:44.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.974+0000 7f05fbe00640 1 --2- 172.21.15.31:0/3013269470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c8e375080 0x561c8e37d0a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f05e8037040 tx=0x7f05e80323f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a856b67d0874edc server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:44.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.974+0000 7f05f1600640 1 -- 172.21.15.31:0/3013269470 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e8039040 con 0x561c8e375080 2024-03-21T10:41:44.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.974+0000 7f05f1600640 1 -- 172.21.15.31:0/3013269470 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05e8035d40 con 0x561c8e375080 2024-03-21T10:41:44.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.974+0000 7f05f1600640 1 -- 172.21.15.31:0/3013269470 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e80053a0 con 0x561c8e375080 2024-03-21T10:41:44.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.975+0000 7f0606597740 1 -- 172.21.15.31:0/3013269470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c8e375080 msgr2=0x561c8e37d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:44.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.975+0000 7f0606597740 1 --2- 172.21.15.31:0/3013269470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c8e375080 0x561c8e37d0a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f05e8037040 tx=0x7f05e80323f0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.975+0000 7f0606597740 1 -- 172.21.15.31:0/3013269470 shutdown_connections 2024-03-21T10:41:44.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.975+0000 7f0606597740 1 --2- 172.21.15.31:0/3013269470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c8e37dc00 0x561c8e37d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.975+0000 7f0606597740 1 --2- 172.21.15.31:0/3013269470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c8e375080 0x561c8e37d0a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.975+0000 7f0606597740 1 --2- 172.21.15.31:0/3013269470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c8e3746d0 0x561c8e36dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.975+0000 7f0606597740 1 -- 172.21.15.31:0/3013269470 >> 172.21.15.31:0/3013269470 conn(0x561c8e36eda0 msgr2=0x561c8e36d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:44.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.975+0000 7f0606597740 1 -- 172.21.15.31:0/3013269470 shutdown_connections 2024-03-21T10:41:44.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.975+0000 7f0606597740 1 -- 172.21.15.31:0/3013269470 wait complete. 2024-03-21T10:41:44.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.976+0000 7f0606597740 1 -- start start 2024-03-21T10:41:44.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.976+0000 7f0606597740 1 Processor -- start 2024-03-21T10:41:44.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.976+0000 7f0606597740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c8e3746d0 0x561c8e383510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:44.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.976+0000 7f0606597740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c8e375080 0x561c8e383a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:44.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.976+0000 7f0606597740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c8e37dc00 0x561c8e383f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:44.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.976+0000 7f0606597740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc1bbd20 con 0x561c8e3746d0 2024-03-21T10:41:44.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.976+0000 7f0606597740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc1bd020 con 0x561c8e37dc00 2024-03-21T10:41:44.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.976+0000 7f0606597740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc1bc360 con 0x561c8e375080 2024-03-21T10:41:44.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.976+0000 7f0606597740 10 client.0 _ll_register_callbacks cb 0x7f05ec0d3640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:44.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.977+0000 7f05fbe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c8e375080 0x561c8e383a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:44.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.977+0000 7f05fbe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c8e375080 0x561c8e383a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:51562/0 (socket says 172.21.15.31:51562) 2024-03-21T10:41:44.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.977+0000 7f05fbe00640 1 -- 172.21.15.31:0/967591188 learned_addr learned my addr 172.21.15.31:0/967591188 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:44.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.977+0000 7f0600e00640 1 --2- 172.21.15.31:0/967591188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c8e3746d0 0x561c8e383510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:44.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.977+0000 7f0601800640 1 --2- 172.21.15.31:0/967591188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c8e37dc00 0x561c8e383f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:44.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.977+0000 7f05fbe00640 1 -- 172.21.15.31:0/967591188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c8e37dc00 msgr2=0x561c8e383f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:44.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.977+0000 7f05fbe00640 1 --2- 172.21.15.31:0/967591188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c8e37dc00 0x561c8e383f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.977+0000 7f05fbe00640 1 -- 172.21.15.31:0/967591188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c8e3746d0 msgr2=0x561c8e383510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:44.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.977+0000 7f05fbe00640 1 --2- 172.21.15.31:0/967591188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c8e3746d0 0x561c8e383510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.977+0000 7f05fbe00640 1 -- 172.21.15.31:0/967591188 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561c8e3e8060 con 0x561c8e375080 2024-03-21T10:41:44.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.977+0000 7f0600e00640 1 --2- 172.21.15.31:0/967591188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c8e3746d0 0x561c8e383510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:41:44.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.977+0000 7f0601800640 1 --2- 172.21.15.31:0/967591188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c8e37dc00 0x561c8e383f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:41:44.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.977+0000 7f05fbe00640 1 --2- 172.21.15.31:0/967591188 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c8e375080 0x561c8e383a50 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f05e802c0b0 tx=0x7f05e8004530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc7038f3e599279b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:44.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.977+0000 7f05e6a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:41:44.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.978+0000 7f05e6a00640 1 -- 172.21.15.31:0/967591188 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e8039030 con 0x561c8e375080 2024-03-21T10:41:44.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.978+0000 7f05e6a00640 1 -- 172.21.15.31:0/967591188 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f05e8034850 con 0x561c8e375080 2024-03-21T10:41:44.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.978+0000 7f0606597740 1 -- 172.21.15.31:0/967591188 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f05e8035d40 con 0x561c8e375080 2024-03-21T10:41:44.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.978+0000 7f05e6a00640 1 -- 172.21.15.31:0/967591188 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05e8059050 con 0x561c8e375080 2024-03-21T10:41:44.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.978+0000 7f0606597740 10 client.26404 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:44.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.978+0000 7f0606597740 1 -- 172.21.15.31:0/967591188 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x561c8e3e9340 con 0x561c8e375080 2024-03-21T10:41:44.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.979+0000 7f05e6a00640 1 -- 172.21.15.31:0/967591188 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05e8035d40 con 0x561c8e375080 2024-03-21T10:41:44.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.979+0000 7f05e6a00640 1 -- 172.21.15.31:0/967591188 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f05e80046c0 con 0x561c8e375080 2024-03-21T10:41:44.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.979+0000 7f0606597740 10 client.26404 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:44.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.979+0000 7f0606597740 10 client.26404 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:41:44.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.979+0000 7f0606597740 20 client.26404 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:44.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.979+0000 7f0606597740 1 --2- 172.21.15.31:0/967591188 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x561c8e3e99f0 0x561c8e3ebe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:44.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.979+0000 7f0606597740 4 client.26404 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:41:44.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.979+0000 7f0606597740 1 -- 172.21.15.31:0/967591188 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561c8e2dfb50 con 0x561c8e3e99f0 2024-03-21T10:41:44.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.979+0000 7f0600e00640 1 --2- 172.21.15.31:0/967591188 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x561c8e3e99f0 0x561c8e3ebe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:44.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.980+0000 7f0600e00640 1 --2- 172.21.15.31:0/967591188 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x561c8e3e99f0 0x561c8e3ebe10 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=afb27cc39f72dc6c server_cookie=43a912379332f68b in_seq=0 out_seq=0 2024-03-21T10:41:44.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.980+0000 7f05e6a00640 10 client.26404 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:41:44.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.980+0000 7f05e6a00640 1 -- 172.21.15.31:0/967591188 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561c8e2dfb50 con 0x561c8e3e99f0 2024-03-21T10:41:44.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.980+0000 7f05e6a00640 10 client.26404 handle_command_reply: tid=0 2024-03-21T10:41:44.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.980+0000 7f0606597740 1 client.26404 shutdown 2024-03-21T10:41:44.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.980+0000 7f0606597740 1 -- 172.21.15.31:0/967591188 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c8e375080 msgr2=0x561c8e383a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:44.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.980+0000 7f0606597740 1 --2- 172.21.15.31:0/967591188 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c8e375080 0x561c8e383a50 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f05e802c0b0 tx=0x7f05e8004530 comp rx=0 tx=0).stop 2024-03-21T10:41:44.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.981+0000 7f0606597740 1 -- 172.21.15.31:0/967591188 shutdown_connections 2024-03-21T10:41:44.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.981+0000 7f0606597740 1 --2- 172.21.15.31:0/967591188 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x561c8e3e99f0 0x561c8e3ebe10 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.981+0000 7f0606597740 1 --2- 172.21.15.31:0/967591188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c8e37dc00 0x561c8e383f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.981+0000 7f05e6a00640 0 client.26404 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:41:44.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.981+0000 7f0606597740 1 --2- 172.21.15.31:0/967591188 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c8e375080 0x561c8e383a50 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.981+0000 7f0606597740 1 --2- 172.21.15.31:0/967591188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c8e3746d0 0x561c8e383510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.981+0000 7f0606597740 1 -- 172.21.15.31:0/967591188 >> 172.21.15.31:0/967591188 conn(0x561c8e36eda0 msgr2=0x561c8e36deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:44.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.981+0000 7f0606597740 1 -- 172.21.15.31:0/967591188 shutdown_connections 2024-03-21T10:41:44.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.981+0000 7f0606597740 1 -- 172.21.15.31:0/967591188 wait complete. 2024-03-21T10:41:44.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.981+0000 7f0606597740 20 client.26404 trim_cache size 0 max 16384 2024-03-21T10:41:44.981 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:41:44.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.982+0000 7f0603800640 1 -- 172.21.15.31:0/700336398 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f05e00a3110 msgr2=0x7f05e00a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:44.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.982+0000 7f0603800640 1 --2- 172.21.15.31:0/700336398 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f05e00a3110 0x7f05e00a55d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f05f4022000 tx=0x7f05f4023000 comp rx=0 tx=0).stop 2024-03-21T10:41:44.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.982+0000 7f0603800640 1 -- 172.21.15.31:0/700336398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05fc107dc0 msgr2=0x7f05fc072e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:44.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.982+0000 7f0603800640 1 --2- 172.21.15.31:0/700336398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05fc107dc0 0x7f05fc072e50 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f05ec00c920 tx=0x7f05ec00cde0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.983+0000 7f0603800640 1 -- 172.21.15.31:0/700336398 shutdown_connections 2024-03-21T10:41:44.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.983+0000 7f0603800640 1 --2- 172.21.15.31:0/700336398 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f05e00a3110 0x7f05e00a55d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.983+0000 7f0603800640 1 --2- 172.21.15.31:0/700336398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05fc109b70 0x7f05fc06c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.983+0000 7f0603800640 1 --2- 172.21.15.31:0/700336398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05fc108760 0x7f05fc073390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.983+0000 7f0603800640 1 --2- 172.21.15.31:0/700336398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05fc107dc0 0x7f05fc072e50 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:44.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.983+0000 7f0603800640 1 -- 172.21.15.31:0/700336398 >> 172.21.15.31:0/700336398 conn(0x7f05fc077540 msgr2=0x7f05fc077a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:44.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.987+0000 7f0603800640 1 -- 172.21.15.31:0/700336398 shutdown_connections 2024-03-21T10:41:44.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:44.987+0000 7f0603800640 1 -- 172.21.15.31:0/700336398 wait complete. 2024-03-21T10:41:45.014 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:41:45.015 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-21T10:41:45.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:44 smithi040 ceph-mon[36154]: pgmap v98: 129 pgs: 129 active+clean; 13 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 181 MiB/s wr, 1.19k op/s 2024-03-21T10:41:45.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:44 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1680761550' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:41:45.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:44 smithi031 ceph-mon[31673]: pgmap v98: 129 pgs: 129 active+clean; 13 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 181 MiB/s wr, 1.19k op/s 2024-03-21T10:41:45.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:44 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1680761550' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:41:45.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:44 smithi070 ceph-mon[36330]: pgmap v98: 129 pgs: 129 active+clean; 13 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 181 MiB/s wr, 1.19k op/s 2024-03-21T10:41:45.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:44 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1680761550' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:41:45.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.446+0000 7f4d6fc00640 1 -- 172.21.15.31:0/1653451494 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d68108910 msgr2=0x7f4d68108cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:45.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.446+0000 7f4d6fc00640 1 --2- 172.21.15.31:0/1653451494 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d68108910 0x7f4d68108cf0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4d60009be0 tx=0x7f4d6002bef0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.448+0000 7f4d6fc00640 1 -- 172.21.15.31:0/1653451494 shutdown_connections 2024-03-21T10:41:45.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.448+0000 7f4d6fc00640 1 --2- 172.21.15.31:0/1653451494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d6810d000 0x7f4d680799b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.448+0000 7f4d6fc00640 1 --2- 172.21.15.31:0/1653451494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d681092c0 0x7f4d6810c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.448+0000 7f4d6fc00640 1 --2- 172.21.15.31:0/1653451494 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d68108910 0x7f4d68108cf0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.448+0000 7f4d6fc00640 1 -- 172.21.15.31:0/1653451494 >> 172.21.15.31:0/1653451494 conn(0x7f4d6806c710 msgr2=0x7f4d6806cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:45.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.455+0000 7f4d6fc00640 1 -- 172.21.15.31:0/1653451494 shutdown_connections 2024-03-21T10:41:45.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.461+0000 7f4d6fc00640 1 -- 172.21.15.31:0/1653451494 wait complete. 2024-03-21T10:41:45.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.464+0000 7f4d6fc00640 1 Processor -- start 2024-03-21T10:41:45.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.465+0000 7f4d6fc00640 1 -- start start 2024-03-21T10:41:45.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.466+0000 7f4d6fc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d68108910 0x7f4d680799b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:45.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.466+0000 7f4d6fc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d6810d000 0x7f4d68073d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:45.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.466+0000 7f4d6fc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d68074350 0x7f4d68136b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:45.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.466+0000 7f4d6fc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6807c190 con 0x7f4d68074350 2024-03-21T10:41:45.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.466+0000 7f4d6fc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6807c010 con 0x7f4d6810d000 2024-03-21T10:41:45.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.466+0000 7f4d6fc00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6807c310 con 0x7f4d68108910 2024-03-21T10:41:45.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.466+0000 7f4d67e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d6810d000 0x7f4d68073d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:45.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.466+0000 7f4d67e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d6810d000 0x7f4d68073d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56512/0 (socket says 172.21.15.31:56512) 2024-03-21T10:41:45.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.466+0000 7f4d67e00640 1 -- 172.21.15.31:0/2318643340 learned_addr learned my addr 172.21.15.31:0/2318643340 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:45.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.467+0000 7f4d6d200640 1 --2- 172.21.15.31:0/2318643340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d68108910 0x7f4d680799b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:45.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.467+0000 7f4d67e00640 1 -- 172.21.15.31:0/2318643340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d68108910 msgr2=0x7f4d680799b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:45.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.467+0000 7f4d67e00640 1 --2- 172.21.15.31:0/2318643340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d68108910 0x7f4d680799b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.467+0000 7f4d67e00640 1 -- 172.21.15.31:0/2318643340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d68074350 msgr2=0x7f4d68136b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:45.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.467+0000 7f4d67e00640 1 --2- 172.21.15.31:0/2318643340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d68074350 0x7f4d68136b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.467+0000 7f4d67e00640 1 -- 172.21.15.31:0/2318643340 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d68137080 con 0x7f4d6810d000 2024-03-21T10:41:45.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.468+0000 7f4d67e00640 1 --2- 172.21.15.31:0/2318643340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d6810d000 0x7f4d68073d00 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f4d5400e9c0 tx=0x7f4d5400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:45.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.468+0000 7f4d65600640 1 -- 172.21.15.31:0/2318643340 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5400ccc0 con 0x7f4d6810d000 2024-03-21T10:41:45.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.468+0000 7f4d65600640 1 -- 172.21.15.31:0/2318643340 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d54010910 con 0x7f4d6810d000 2024-03-21T10:41:45.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.468+0000 7f4d65600640 1 -- 172.21.15.31:0/2318643340 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d54018840 con 0x7f4d6810d000 2024-03-21T10:41:45.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.469+0000 7f4d6fc00640 1 -- 172.21.15.31:0/2318643340 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d68137370 con 0x7f4d6810d000 2024-03-21T10:41:45.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.470+0000 7f4d6fc00640 1 -- 172.21.15.31:0/2318643340 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d68137870 con 0x7f4d6810d000 2024-03-21T10:41:45.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.480+0000 7f4d7280d740 1 Processor -- start 2024-03-21T10:41:45.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.481+0000 7f4d65600640 1 -- 172.21.15.31:0/2318643340 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f4d54010ac0 con 0x7f4d6810d000 2024-03-21T10:41:45.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.481+0000 7f4d7280d740 1 -- start start 2024-03-21T10:41:45.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.484+0000 7f4d7280d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5585852516d0 0x558585251ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:45.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.484+0000 7f4d7280d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558585252080 0x55858525a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:45.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.485+0000 7f4d6dc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5585852516d0 0x558585251ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:45.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.485+0000 7f4d6dc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5585852516d0 0x558585251ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56518/0 (socket says 172.21.15.31:56518) 2024-03-21T10:41:45.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.486+0000 7f4d7280d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55858525ac00 0x55858525d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:45.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.486+0000 7f4d7280d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585851b52c0 con 0x558585252080 2024-03-21T10:41:45.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.486+0000 7f4d7280d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55858515dc90 con 0x5585852516d0 2024-03-21T10:41:45.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.486+0000 7f4d7280d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585851b1360 con 0x55858525ac00 2024-03-21T10:41:45.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.487+0000 7f4d6dc00640 1 -- 172.21.15.31:0/3401940998 learned_addr learned my addr 172.21.15.31:0/3401940998 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:45.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.488+0000 7f4d65600640 1 --2- 172.21.15.31:0/2318643340 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4d4c0a3170 0x7f4d4c0a5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:45.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.488+0000 7f4d6d200640 1 --2- 172.21.15.31:0/3401940998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558585252080 0x55858525a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:45.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.488+0000 7f4d6dc00640 1 -- 172.21.15.31:0/3401940998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55858525ac00 msgr2=0x55858525d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:45.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.488+0000 7f4d6dc00640 1 --2- 172.21.15.31:0/3401940998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55858525ac00 0x55858525d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.488+0000 7f4d6dc00640 1 -- 172.21.15.31:0/3401940998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558585252080 msgr2=0x55858525a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:45.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.488+0000 7f4d6dc00640 1 --2- 172.21.15.31:0/3401940998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558585252080 0x55858525a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.488+0000 7f4d65600640 1 -- 172.21.15.31:0/2318643340 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f4d540cfdd0 con 0x7f4d6810d000 2024-03-21T10:41:45.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.488+0000 7f4d6dc00640 1 -- 172.21.15.31:0/3401940998 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558585182080 con 0x5585852516d0 2024-03-21T10:41:45.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.490+0000 7f4d6dc00640 1 --2- 172.21.15.31:0/3401940998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5585852516d0 0x558585251ab0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4d5800ba00 tx=0x7f4d5800bec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71843eae7c4f26a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:45.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.490+0000 7f4d6dc00640 1 --2- 172.21.15.31:0/2318643340 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4d4c0a3170 0x7f4d4c0a5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:45.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.491+0000 7f4d5e000640 1 -- 172.21.15.31:0/3401940998 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5801d660 con 0x5585852516d0 2024-03-21T10:41:45.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.491+0000 7f4d5e000640 1 -- 172.21.15.31:0/3401940998 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d58016040 con 0x5585852516d0 2024-03-21T10:41:45.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.492+0000 7f4d5e000640 1 -- 172.21.15.31:0/3401940998 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5801d660 con 0x5585852516d0 2024-03-21T10:41:45.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.494+0000 7f4d7280d740 1 -- 172.21.15.31:0/3401940998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5585852516d0 msgr2=0x558585251ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:45.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.494+0000 7f4d7280d740 1 --2- 172.21.15.31:0/3401940998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5585852516d0 0x558585251ab0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4d5800ba00 tx=0x7f4d5800bec0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.499+0000 7f4d6dc00640 1 --2- 172.21.15.31:0/2318643340 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4d4c0a3170 0x7f4d4c0a5630 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4d5801c000 tx=0x7f4d5801c790 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:45.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.499+0000 7f4d7280d740 1 -- 172.21.15.31:0/3401940998 shutdown_connections 2024-03-21T10:41:45.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.499+0000 7f4d7280d740 1 --2- 172.21.15.31:0/3401940998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55858525ac00 0x55858525d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.499+0000 7f4d7280d740 1 --2- 172.21.15.31:0/3401940998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558585252080 0x55858525a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.499+0000 7f4d7280d740 1 --2- 172.21.15.31:0/3401940998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5585852516d0 0x558585251ab0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.499+0000 7f4d7280d740 1 -- 172.21.15.31:0/3401940998 >> 172.21.15.31:0/3401940998 conn(0x55858524bda0 msgr2=0x55858524c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:45.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.507+0000 7f4d7280d740 1 -- 172.21.15.31:0/3401940998 shutdown_connections 2024-03-21T10:41:45.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.514+0000 7f4d7280d740 1 -- 172.21.15.31:0/3401940998 wait complete. 2024-03-21T10:41:45.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.514+0000 7f4d7280d740 1 -- start start 2024-03-21T10:41:45.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.515+0000 7f4d7280d740 1 Processor -- start 2024-03-21T10:41:45.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.517+0000 7f4d7280d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585852516d0 0x55858525a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:45.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.517+0000 7f4d7280d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558585252080 0x5585852577b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:45.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.517+0000 7f4d7280d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55858525ac00 0x558585257cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:45.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.523+0000 7f4d7280d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55858525dbc0 con 0x5585852516d0 2024-03-21T10:41:45.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.523+0000 7f4d7280d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585851710d0 con 0x558585252080 2024-03-21T10:41:45.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.523+0000 7f4d7280d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55858525dd40 con 0x55858525ac00 2024-03-21T10:41:45.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.523+0000 7f4d7280d740 10 client.0 _ll_register_callbacks cb 0x558585175e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:45.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.525+0000 7f4d6dc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55858525ac00 0x558585257cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:45.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.525+0000 7f4d6dc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55858525ac00 0x558585257cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:51600/0 (socket says 172.21.15.31:51600) 2024-03-21T10:41:45.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.525+0000 7f4d6dc00640 1 -- 172.21.15.31:0/1895201689 learned_addr learned my addr 172.21.15.31:0/1895201689 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:45.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.525+0000 7f4d6dc00640 1 -- 172.21.15.31:0/1895201689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558585252080 msgr2=0x5585852577b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:41:45.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.525+0000 7f4d6dc00640 1 --2- 172.21.15.31:0/1895201689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558585252080 0x5585852577b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.525+0000 7f4d6dc00640 1 -- 172.21.15.31:0/1895201689 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585852516d0 msgr2=0x55858525a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:45.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.525+0000 7f4d6dc00640 1 --2- 172.21.15.31:0/1895201689 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585852516d0 0x55858525a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.526+0000 7f4d6dc00640 1 -- 172.21.15.31:0/1895201689 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558584ec93f0 con 0x55858525ac00 2024-03-21T10:41:45.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.527+0000 7f4d6dc00640 1 --2- 172.21.15.31:0/1895201689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55858525ac00 0x558585257cf0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f4d5800bf00 tx=0x7f4d58014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9b4b630a77b8f70 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:45.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.527+0000 7f4d52a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:41:45.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.528+0000 7f4d52a00640 1 -- 172.21.15.31:0/1895201689 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d580106a0 con 0x55858525ac00 2024-03-21T10:41:45.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.528+0000 7f4d52a00640 1 -- 172.21.15.31:0/1895201689 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f4d5800dd00 con 0x55858525ac00 2024-03-21T10:41:45.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.531+0000 7f4d7280d740 1 -- 172.21.15.31:0/1895201689 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4d58016040 con 0x55858525ac00 2024-03-21T10:41:45.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.531+0000 7f4d52a00640 1 -- 172.21.15.31:0/1895201689 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d58010af0 con 0x55858525ac00 2024-03-21T10:41:45.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.532+0000 7f4d7280d740 10 client.26413 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:45.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.532+0000 7f4d7280d740 1 -- 172.21.15.31:0/1895201689 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558584f937d0 con 0x55858525ac00 2024-03-21T10:41:45.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.532+0000 7f4d52a00640 1 -- 172.21.15.31:0/1895201689 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d58016040 con 0x55858525ac00 2024-03-21T10:41:45.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.533+0000 7f4d52a00640 1 -- 172.21.15.31:0/1895201689 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f4d580039c0 con 0x55858525ac00 2024-03-21T10:41:45.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.534+0000 7f4d7280d740 10 client.26413 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:45.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.534+0000 7f4d7280d740 10 client.26413 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:41:45.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.536+0000 7f4d7280d740 20 client.26413 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:45.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.536+0000 7f4d7280d740 1 --2- 172.21.15.31:0/1895201689 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55858526a920 0x55858526cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:45.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.536+0000 7f4d7280d740 4 client.26413 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:41:45.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.536+0000 7f4d7280d740 1 -- 172.21.15.31:0/1895201689 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5585851c2800 con 0x55858526a920 2024-03-21T10:41:45.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.538+0000 7f4d6d200640 1 --2- 172.21.15.31:0/1895201689 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55858526a920 0x55858526cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:45.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.541+0000 7f4d6d200640 1 --2- 172.21.15.31:0/1895201689 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55858526a920 0x55858526cd10 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c14ffd844fdd2aef server_cookie=f01c840a7fd352f7 in_seq=0 out_seq=0 2024-03-21T10:41:45.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.541+0000 7f4d52a00640 10 client.26413 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:41:45.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.547+0000 7f4d52a00640 1 -- 172.21.15.31:0/1895201689 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5585851c2800 con 0x55858526a920 2024-03-21T10:41:45.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.547+0000 7f4d52a00640 10 client.26413 handle_command_reply: tid=0 2024-03-21T10:41:45.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.548+0000 7f4d7280d740 1 client.26413 shutdown 2024-03-21T10:41:45.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.549+0000 7f4d7280d740 1 -- 172.21.15.31:0/1895201689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55858525ac00 msgr2=0x558585257cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:45.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.549+0000 7f4d7280d740 1 --2- 172.21.15.31:0/1895201689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55858525ac00 0x558585257cf0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f4d5800bf00 tx=0x7f4d58014000 comp rx=0 tx=0).stop 2024-03-21T10:41:45.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.550+0000 7f4d7280d740 1 -- 172.21.15.31:0/1895201689 shutdown_connections 2024-03-21T10:41:45.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.551+0000 7f4d7280d740 1 --2- 172.21.15.31:0/1895201689 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55858526a920 0x55858526cd10 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.553+0000 7f4d52a00640 0 client.26413 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:41:45.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.553+0000 7f4d7280d740 1 --2- 172.21.15.31:0/1895201689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55858525ac00 0x558585257cf0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.553+0000 7f4d7280d740 1 --2- 172.21.15.31:0/1895201689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558585252080 0x5585852577b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.553+0000 7f4d7280d740 1 --2- 172.21.15.31:0/1895201689 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585852516d0 0x55858525a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.553+0000 7f4d7280d740 1 -- 172.21.15.31:0/1895201689 >> 172.21.15.31:0/1895201689 conn(0x55858524bda0 msgr2=0x55858524daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:45.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.553+0000 7f4d7280d740 1 -- 172.21.15.31:0/1895201689 shutdown_connections 2024-03-21T10:41:45.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.553+0000 7f4d7280d740 1 -- 172.21.15.31:0/1895201689 wait complete. 2024-03-21T10:41:45.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.555+0000 7f4d7280d740 20 client.26413 trim_cache size 0 max 16384 2024-03-21T10:41:45.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.590+0000 7f4d7280d740 1 Processor -- start 2024-03-21T10:41:45.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.594+0000 7f4d7280d740 1 -- start start 2024-03-21T10:41:45.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.594+0000 7f4d7280d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585852516d0 0x558585252cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:45.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.594+0000 7f4d7280d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558585252080 0x558585259ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:45.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.595+0000 7f4d6d200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585852516d0 0x558585252cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:45.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.595+0000 7f4d6d200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585852516d0 0x558585252cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:51604/0 (socket says 172.21.15.31:51604) 2024-03-21T10:41:45.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.595+0000 7f4d7280d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55858525ac00 0x55858525a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:45.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.595+0000 7f4d7280d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d681d93f0 con 0x558585252080 2024-03-21T10:41:45.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.595+0000 7f4d7280d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d681d6ef0 con 0x55858525ac00 2024-03-21T10:41:45.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.595+0000 7f4d7280d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d681d5040 con 0x5585852516d0 2024-03-21T10:41:45.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.595+0000 7f4d67e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558585252080 0x558585259ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:45.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.596+0000 7f4d67e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558585252080 0x558585259ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50928/0 (socket says 172.21.15.31:50928) 2024-03-21T10:41:45.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.596+0000 7f4d67e00640 1 -- 172.21.15.31:0/1369245741 learned_addr learned my addr 172.21.15.31:0/1369245741 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:45.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.597+0000 7f4d67e00640 1 -- 172.21.15.31:0/1369245741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585852516d0 msgr2=0x558585252cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:45.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.597+0000 7f4d67e00640 1 --2- 172.21.15.31:0/1369245741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585852516d0 0x558585252cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.597+0000 7f4d67e00640 1 -- 172.21.15.31:0/1369245741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55858525ac00 msgr2=0x55858525a420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:41:45.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.597+0000 7f4d67e00640 1 --2- 172.21.15.31:0/1369245741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55858525ac00 0x55858525a420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.597+0000 7f4d67e00640 1 -- 172.21.15.31:0/1369245741 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55858525dd40 con 0x558585252080 2024-03-21T10:41:45.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.597+0000 7f4d6d200640 1 --2- 172.21.15.31:0/1369245741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585852516d0 0x558585252cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-21T10:41:45.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.598+0000 7f4d67e00640 1 --2- 172.21.15.31:0/1369245741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558585252080 0x558585259ee0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4d5409bdb0 tx=0x7f4d5409b780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8da8cb625318f98 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:45.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.599+0000 7f4d5d600640 1 -- 172.21.15.31:0/1369245741 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5409ff70 con 0x558585252080 2024-03-21T10:41:45.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.600+0000 7f4d5d600640 1 -- 172.21.15.31:0/1369245741 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d540a4830 con 0x558585252080 2024-03-21T10:41:45.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.600+0000 7f4d5d600640 1 -- 172.21.15.31:0/1369245741 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d540badf0 con 0x558585252080 2024-03-21T10:41:45.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.600+0000 7f4d7280d740 1 -- 172.21.15.31:0/1369245741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558585252080 msgr2=0x558585259ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:45.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.600+0000 7f4d7280d740 1 --2- 172.21.15.31:0/1369245741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558585252080 0x558585259ee0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4d5409bdb0 tx=0x7f4d5409b780 comp rx=0 tx=0).stop 2024-03-21T10:41:45.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.601+0000 7f4d7280d740 1 -- 172.21.15.31:0/1369245741 shutdown_connections 2024-03-21T10:41:45.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.601+0000 7f4d7280d740 1 --2- 172.21.15.31:0/1369245741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55858525ac00 0x55858525a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.601+0000 7f4d7280d740 1 --2- 172.21.15.31:0/1369245741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558585252080 0x558585259ee0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.601+0000 7f4d7280d740 1 --2- 172.21.15.31:0/1369245741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585852516d0 0x558585252cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.601+0000 7f4d7280d740 1 -- 172.21.15.31:0/1369245741 >> 172.21.15.31:0/1369245741 conn(0x55858524bda0 msgr2=0x55858524da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:45.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.601+0000 7f4d7280d740 1 -- 172.21.15.31:0/1369245741 shutdown_connections 2024-03-21T10:41:45.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.601+0000 7f4d7280d740 1 -- 172.21.15.31:0/1369245741 wait complete. 2024-03-21T10:41:45.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.602+0000 7f4d7280d740 1 -- start start 2024-03-21T10:41:45.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.602+0000 7f4d7280d740 1 Processor -- start 2024-03-21T10:41:45.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.603+0000 7f4d7280d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585852516d0 0x55858525e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:45.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.603+0000 7f4d7280d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55858525ac00 0x55858525eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:45.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.603+0000 7f4d7280d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55858525f0c0 0x5585852c4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:45.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.604+0000 7f4d67e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55858525ac00 0x55858525eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:45.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.604+0000 7f4d67e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55858525ac00 0x55858525eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56544/0 (socket says 172.21.15.31:56544) 2024-03-21T10:41:45.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.604+0000 7f4d6dc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55858525f0c0 0x5585852c4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:45.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.604+0000 7f4d6dc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55858525f0c0 0x5585852c4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:51606/0 (socket says 172.21.15.31:51606) 2024-03-21T10:41:45.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.604+0000 7f4d7280d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d681d55b0 con 0x5585852516d0 2024-03-21T10:41:45.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.604+0000 7f4d7280d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d681d6890 con 0x55858525ac00 2024-03-21T10:41:45.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.604+0000 7f4d7280d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d681d5bd0 con 0x55858525f0c0 2024-03-21T10:41:45.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.605+0000 7f4d67e00640 1 -- 172.21.15.31:0/951928382 learned_addr learned my addr 172.21.15.31:0/951928382 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:45.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.605+0000 7f4d7280d740 10 client.0 _ll_register_callbacks cb 0x7f4d5800b850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:45.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.605+0000 7f4d67e00640 1 -- 172.21.15.31:0/951928382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55858525f0c0 msgr2=0x5585852c4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:45.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.605+0000 7f4d67e00640 1 --2- 172.21.15.31:0/951928382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55858525f0c0 0x5585852c4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.605+0000 7f4d67e00640 1 -- 172.21.15.31:0/951928382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585852516d0 msgr2=0x55858525e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:45.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.605+0000 7f4d67e00640 1 --2- 172.21.15.31:0/951928382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585852516d0 0x55858525e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.605+0000 7f4d67e00640 1 -- 172.21.15.31:0/951928382 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5585852c55c0 con 0x55858525ac00 2024-03-21T10:41:45.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.606+0000 7f4d67e00640 1 --2- 172.21.15.31:0/951928382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55858525ac00 0x55858525eb80 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4d5400c6a0 tx=0x7f4d5409b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b697393faa0514fc server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:45.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.606+0000 7f4d52a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:41:45.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.606+0000 7f4d52a00640 1 -- 172.21.15.31:0/951928382 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5409ff70 con 0x55858525ac00 2024-03-21T10:41:45.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.606+0000 7f4d52a00640 1 -- 172.21.15.31:0/951928382 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f4d5409b270 con 0x55858525ac00 2024-03-21T10:41:45.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.607+0000 7f4d7280d740 1 -- 172.21.15.31:0/951928382 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4d540a4830 con 0x55858525ac00 2024-03-21T10:41:45.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.608+0000 7f4d52a00640 1 -- 172.21.15.31:0/951928382 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d5409b4f0 con 0x55858525ac00 2024-03-21T10:41:45.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.608+0000 7f4d7280d740 10 client.26471 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:45.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.608+0000 7f4d7280d740 1 -- 172.21.15.31:0/951928382 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5585852c67f0 con 0x55858525ac00 2024-03-21T10:41:45.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.610+0000 7f4d52a00640 1 -- 172.21.15.31:0/951928382 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d540a4830 con 0x55858525ac00 2024-03-21T10:41:45.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.610+0000 7f4d52a00640 1 -- 172.21.15.31:0/951928382 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f4d5409aed0 con 0x55858525ac00 2024-03-21T10:41:45.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.610+0000 7f4d7280d740 10 client.26471 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:45.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.610+0000 7f4d7280d740 10 client.26471 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:41:45.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.610+0000 7f4d7280d740 20 client.26471 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:45.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.610+0000 7f4d7280d740 1 --2- 172.21.15.31:0/951928382 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x5585852c6e40 0x5585852c9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:45.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.611+0000 7f4d7280d740 4 client.26471 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:41:45.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.611+0000 7f4d7280d740 1 -- 172.21.15.31:0/951928382 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5585851bcb50 con 0x5585852c6e40 2024-03-21T10:41:45.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.613+0000 7f4d6d200640 1 --2- 172.21.15.31:0/951928382 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x5585852c6e40 0x5585852c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:45.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.615+0000 7f4d6d200640 1 --2- 172.21.15.31:0/951928382 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x5585852c6e40 0x5585852c9260 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ab1243024b94213e server_cookie=dc4a19f198b64bc in_seq=0 out_seq=0 2024-03-21T10:41:45.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.615+0000 7f4d52a00640 10 client.26471 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:41:45.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.618+0000 7f4d52a00640 1 -- 172.21.15.31:0/951928382 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5585851bcb50 con 0x5585852c6e40 2024-03-21T10:41:45.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.618+0000 7f4d52a00640 10 client.26471 handle_command_reply: tid=0 2024-03-21T10:41:45.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.622+0000 7f4d7280d740 1 client.26471 shutdown 2024-03-21T10:41:45.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.624+0000 7f4d7280d740 1 -- 172.21.15.31:0/951928382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55858525ac00 msgr2=0x55858525eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:45.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.624+0000 7f4d7280d740 1 --2- 172.21.15.31:0/951928382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55858525ac00 0x55858525eb80 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4d5400c6a0 tx=0x7f4d5409b8b0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.624+0000 7f4d7280d740 1 -- 172.21.15.31:0/951928382 shutdown_connections 2024-03-21T10:41:45.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.624+0000 7f4d7280d740 1 --2- 172.21.15.31:0/951928382 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x5585852c6e40 0x5585852c9260 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.624+0000 7f4d7280d740 1 --2- 172.21.15.31:0/951928382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55858525f0c0 0x5585852c4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.624+0000 7f4d7280d740 1 --2- 172.21.15.31:0/951928382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55858525ac00 0x55858525eb80 secure :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4d5400c6a0 tx=0x7f4d5409b8b0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.624+0000 7f4d7280d740 1 --2- 172.21.15.31:0/951928382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585852516d0 0x55858525e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.624+0000 7f4d7280d740 1 -- 172.21.15.31:0/951928382 >> 172.21.15.31:0/951928382 conn(0x55858524bda0 msgr2=0x558585252cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:45.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.624+0000 7f4d52a00640 0 client.26471 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:41:45.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.624+0000 7f4d7280d740 1 -- 172.21.15.31:0/951928382 shutdown_connections 2024-03-21T10:41:45.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.624+0000 7f4d7280d740 1 -- 172.21.15.31:0/951928382 wait complete. 2024-03-21T10:41:45.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.625+0000 7f4d7280d740 20 client.26471 trim_cache size 0 max 16384 2024-03-21T10:41:45.625 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:41:45.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.631+0000 7f4d6fc00640 1 -- 172.21.15.31:0/2318643340 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4d4c0a3170 msgr2=0x7f4d4c0a5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:45.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.631+0000 7f4d6fc00640 1 --2- 172.21.15.31:0/2318643340 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4d4c0a3170 0x7f4d4c0a5630 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4d5801c000 tx=0x7f4d5801c790 comp rx=0 tx=0).stop 2024-03-21T10:41:45.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.632+0000 7f4d6fc00640 1 -- 172.21.15.31:0/2318643340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d6810d000 msgr2=0x7f4d68073d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:45.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.632+0000 7f4d6fc00640 1 --2- 172.21.15.31:0/2318643340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d6810d000 0x7f4d68073d00 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f4d5400e9c0 tx=0x7f4d5400ee80 comp rx=0 tx=0).stop 2024-03-21T10:41:45.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.632+0000 7f4d6fc00640 1 -- 172.21.15.31:0/2318643340 shutdown_connections 2024-03-21T10:41:45.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.632+0000 7f4d6fc00640 1 --2- 172.21.15.31:0/2318643340 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4d4c0a3170 0x7f4d4c0a5630 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.632+0000 7f4d6fc00640 1 --2- 172.21.15.31:0/2318643340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d68074350 0x7f4d68136b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.632+0000 7f4d6fc00640 1 --2- 172.21.15.31:0/2318643340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d6810d000 0x7f4d68073d00 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.632+0000 7f4d6fc00640 1 --2- 172.21.15.31:0/2318643340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d68108910 0x7f4d680799b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:45.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.632+0000 7f4d6fc00640 1 -- 172.21.15.31:0/2318643340 >> 172.21.15.31:0/2318643340 conn(0x7f4d6806c710 msgr2=0x7f4d68072520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:45.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.634+0000 7f4d6fc00640 1 -- 172.21.15.31:0/2318643340 shutdown_connections 2024-03-21T10:41:45.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:45.634+0000 7f4d6fc00640 1 -- 172.21.15.31:0/2318643340 wait complete. 2024-03-21T10:41:45.675 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:41:45.676 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-21T10:41:46.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.092+0000 7fec62600640 1 -- 172.21.15.31:0/1677539057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec5c103680 msgr2=0x7fec5c103b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:46.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.092+0000 7fec62600640 1 --2- 172.21.15.31:0/1677539057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec5c103680 0x7fec5c103b00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fec48009890 tx=0x7fec4802bb90 comp rx=0 tx=0).stop 2024-03-21T10:41:46.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.092+0000 7fec62600640 1 -- 172.21.15.31:0/1677539057 shutdown_connections 2024-03-21T10:41:46.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.092+0000 7fec62600640 1 --2- 172.21.15.31:0/1677539057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec5c104040 0x7fec5c110550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.092+0000 7fec62600640 1 --2- 172.21.15.31:0/1677539057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec5c103680 0x7fec5c103b00 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.092+0000 7fec62600640 1 --2- 172.21.15.31:0/1677539057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec5c109690 0x7fec5c109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.092+0000 7fec62600640 1 -- 172.21.15.31:0/1677539057 >> 172.21.15.31:0/1677539057 conn(0x7fec5c0ff110 msgr2=0x7fec5c101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:46.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.093+0000 7fec62600640 1 -- 172.21.15.31:0/1677539057 shutdown_connections 2024-03-21T10:41:46.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.093+0000 7fec62600640 1 -- 172.21.15.31:0/1677539057 wait complete. 2024-03-21T10:41:46.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.094+0000 7fec62600640 1 Processor -- start 2024-03-21T10:41:46.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.095+0000 7fec62600640 1 -- start start 2024-03-21T10:41:46.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.095+0000 7fec62600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec5c103680 0x7fec5c1101d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:46.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.095+0000 7fec62600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec5c104040 0x7fec5c10d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:46.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.095+0000 7fec62600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec5c109690 0x7fec5c10d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:46.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.095+0000 7fec62600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c113ac0 con 0x7fec5c104040 2024-03-21T10:41:46.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.095+0000 7fec62600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c113940 con 0x7fec5c103680 2024-03-21T10:41:46.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.095+0000 7fec62600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c113c40 con 0x7fec5c109690 2024-03-21T10:41:46.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.095+0000 7fec5be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec5c104040 0x7fec5c10d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:46.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.095+0000 7fec61200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec5c103680 0x7fec5c1101d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:46.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.096+0000 7fec61200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec5c103680 0x7fec5c1101d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56576/0 (socket says 172.21.15.31:56576) 2024-03-21T10:41:46.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.096+0000 7fec61200640 1 -- 172.21.15.31:0/2417218935 learned_addr learned my addr 172.21.15.31:0/2417218935 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:46.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.096+0000 7fec61c00640 1 --2- 172.21.15.31:0/2417218935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec5c109690 0x7fec5c10d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:46.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.096+0000 7fec61c00640 1 -- 172.21.15.31:0/2417218935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec5c103680 msgr2=0x7fec5c1101d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:46.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.096+0000 7fec61c00640 1 --2- 172.21.15.31:0/2417218935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec5c103680 0x7fec5c1101d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.096+0000 7fec61c00640 1 -- 172.21.15.31:0/2417218935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec5c104040 msgr2=0x7fec5c10d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:46.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.096+0000 7fec61c00640 1 --2- 172.21.15.31:0/2417218935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec5c104040 0x7fec5c10d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.096+0000 7fec61c00640 1 -- 172.21.15.31:0/2417218935 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec5c10deb0 con 0x7fec5c109690 2024-03-21T10:41:46.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.097+0000 7fec61c00640 1 --2- 172.21.15.31:0/2417218935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec5c109690 0x7fec5c10d7b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fec5400d680 tx=0x7fec5400db40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:46.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.097+0000 7fec59600640 1 -- 172.21.15.31:0/2417218935 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec54017070 con 0x7fec5c109690 2024-03-21T10:41:46.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.097+0000 7fec59600640 1 -- 172.21.15.31:0/2417218935 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec54009820 con 0x7fec5c109690 2024-03-21T10:41:46.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.097+0000 7fec59600640 1 -- 172.21.15.31:0/2417218935 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec54004d00 con 0x7fec5c109690 2024-03-21T10:41:46.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.097+0000 7fec62600640 1 -- 172.21.15.31:0/2417218935 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec5c19ffe0 con 0x7fec5c109690 2024-03-21T10:41:46.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.097+0000 7fec62600640 1 -- 172.21.15.31:0/2417218935 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec5c1a04b0 con 0x7fec5c109690 2024-03-21T10:41:46.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.101+0000 7fec66813740 1 Processor -- start 2024-03-21T10:41:46.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.102+0000 7fec59600640 1 -- 172.21.15.31:0/2417218935 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fec540099d0 con 0x7fec5c109690 2024-03-21T10:41:46.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.103+0000 7fec66813740 1 -- start start 2024-03-21T10:41:46.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.104+0000 7fec66813740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4c5a168b0 0x55e4c5a16c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:46.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.104+0000 7fec66813740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4c5a17260 0x55e4c5a1f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:46.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.104+0000 7fec66813740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4c5a1fde0 0x55e4c5a22230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:46.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.104+0000 7fec66813740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4c597a2c0 con 0x55e4c5a168b0 2024-03-21T10:41:46.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.104+0000 7fec66813740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4c5922c90 con 0x55e4c5a1fde0 2024-03-21T10:41:46.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.104+0000 7fec66813740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4c5976360 con 0x55e4c5a17260 2024-03-21T10:41:46.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.104+0000 7fec5be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4c5a1fde0 0x55e4c5a22230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:46.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.105+0000 7fec5be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4c5a1fde0 0x55e4c5a22230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56586/0 (socket says 172.21.15.31:56586) 2024-03-21T10:41:46.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.105+0000 7fec5be00640 1 -- 172.21.15.31:0/672811255 learned_addr learned my addr 172.21.15.31:0/672811255 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:46.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.105+0000 7fec5be00640 1 --2- 172.21.15.31:0/672811255 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4c5a168b0 0x55e4c5a16c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:46.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.105+0000 7fec59600640 1 --2- 172.21.15.31:0/2417218935 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fec440a31e0 0x7fec440a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:46.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.105+0000 7fec59600640 1 -- 172.21.15.31:0/2417218935 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fec540cf280 con 0x7fec5c109690 2024-03-21T10:41:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.105+0000 7fec61200640 1 --2- 172.21.15.31:0/672811255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4c5a17260 0x55e4c5a1f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.106+0000 7fec5be00640 1 -- 172.21.15.31:0/672811255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4c5a17260 msgr2=0x55e4c5a1f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.106+0000 7fec5be00640 1 --2- 172.21.15.31:0/672811255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4c5a17260 0x55e4c5a1f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.106+0000 7fec5be00640 1 -- 172.21.15.31:0/672811255 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4c5a168b0 msgr2=0x55e4c5a16c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.106+0000 7fec5be00640 1 --2- 172.21.15.31:0/672811255 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4c5a168b0 0x55e4c5a16c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.106+0000 7fec61c00640 1 --2- 172.21.15.31:0/2417218935 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fec440a31e0 0x7fec440a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.106+0000 7fec5be00640 1 -- 172.21.15.31:0/672811255 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e4c5947080 con 0x55e4c5a1fde0 2024-03-21T10:41:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.107+0000 7fec5be00640 1 --2- 172.21.15.31:0/672811255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4c5a1fde0 0x55e4c5a22230 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fec4802c0d0 tx=0x7fec4802e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14dbd9f58815f883 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.107+0000 7fec61c00640 1 --2- 172.21.15.31:0/2417218935 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fec440a31e0 0x7fec440a56a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fec54097840 tx=0x7fec54097040 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.107+0000 7fec4e000640 1 -- 172.21.15.31:0/672811255 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4804a070 con 0x55e4c5a1fde0 2024-03-21T10:41:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.107+0000 7fec4e000640 1 -- 172.21.15.31:0/672811255 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec48004250 con 0x55e4c5a1fde0 2024-03-21T10:41:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.107+0000 7fec4e000640 1 -- 172.21.15.31:0/672811255 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec48039040 con 0x55e4c5a1fde0 2024-03-21T10:41:46.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.108+0000 7fec66813740 1 -- 172.21.15.31:0/672811255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4c5a1fde0 msgr2=0x55e4c5a22230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:46.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.108+0000 7fec66813740 1 --2- 172.21.15.31:0/672811255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4c5a1fde0 0x55e4c5a22230 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fec4802c0d0 tx=0x7fec4802e040 comp rx=0 tx=0).stop 2024-03-21T10:41:46.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.110+0000 7fec66813740 1 -- 172.21.15.31:0/672811255 shutdown_connections 2024-03-21T10:41:46.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.110+0000 7fec66813740 1 --2- 172.21.15.31:0/672811255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4c5a1fde0 0x55e4c5a22230 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.110+0000 7fec66813740 1 --2- 172.21.15.31:0/672811255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4c5a17260 0x55e4c5a1f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.110+0000 7fec66813740 1 --2- 172.21.15.31:0/672811255 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4c5a168b0 0x55e4c5a16c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.110+0000 7fec66813740 1 -- 172.21.15.31:0/672811255 >> 172.21.15.31:0/672811255 conn(0x55e4c5a10f80 msgr2=0x55e4c5a11360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:46.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.111+0000 7fec66813740 1 -- 172.21.15.31:0/672811255 shutdown_connections 2024-03-21T10:41:46.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.113+0000 7fec66813740 1 -- 172.21.15.31:0/672811255 wait complete. 2024-03-21T10:41:46.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.113+0000 7fec66813740 1 -- start start 2024-03-21T10:41:46.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.114+0000 7fec66813740 1 Processor -- start 2024-03-21T10:41:46.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.114+0000 7fec66813740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4c5a168b0 0x55e4c5a1cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:46.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.114+0000 7fec66813740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4c5a17260 0x55e4c5a1d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:46.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.114+0000 7fec66813740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4c5a1fde0 0x55e4c5a1d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:46.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.114+0000 7fec66813740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4c5a22cf0 con 0x55e4c5a1fde0 2024-03-21T10:41:46.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.114+0000 7fec66813740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4c59360d0 con 0x55e4c5a168b0 2024-03-21T10:41:46.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.114+0000 7fec66813740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4c5a22e70 con 0x55e4c5a17260 2024-03-21T10:41:46.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.115+0000 7fec66813740 10 client.0 _ll_register_callbacks cb 0x55e4c593ae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:46.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.115+0000 7fec5be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4c5a1fde0 0x55e4c5a1d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:46.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.115+0000 7fec5be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4c5a1fde0 0x55e4c5a1d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50966/0 (socket says 172.21.15.31:50966) 2024-03-21T10:41:46.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.115+0000 7fec5be00640 1 -- 172.21.15.31:0/2861936536 learned_addr learned my addr 172.21.15.31:0/2861936536 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:46.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.115+0000 7fec5be00640 1 --2- 172.21.15.31:0/2861936536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4c5a168b0 0x55e4c5a1cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:46.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.118+0000 7fec61200640 1 --2- 172.21.15.31:0/2861936536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4c5a17260 0x55e4c5a1d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:46.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.122+0000 7fec61200640 1 -- 172.21.15.31:0/2861936536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4c5a168b0 msgr2=0x55e4c5a1cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:46.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.122+0000 7fec61200640 1 --2- 172.21.15.31:0/2861936536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4c5a168b0 0x55e4c5a1cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.122+0000 7fec61200640 1 -- 172.21.15.31:0/2861936536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4c5a1fde0 msgr2=0x55e4c5a1d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:46.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.122+0000 7fec61200640 1 --2- 172.21.15.31:0/2861936536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4c5a1fde0 0x55e4c5a1d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.122+0000 7fec61200640 1 -- 172.21.15.31:0/2861936536 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e4c568e3f0 con 0x55e4c5a17260 2024-03-21T10:41:46.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.126+0000 7fec61200640 1 --2- 172.21.15.31:0/2861936536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4c5a17260 0x55e4c5a1d0f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x55e4c5a17660 tx=0x7fec5000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=914be25854c517aa server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:46.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.132+0000 7fec42a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:41:46.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.132+0000 7fec42a00640 1 -- 172.21.15.31:0/2861936536 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec50012110 con 0x55e4c5a17260 2024-03-21T10:41:46.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.132+0000 7fec42a00640 1 -- 172.21.15.31:0/2861936536 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fec50003970 con 0x55e4c5a17260 2024-03-21T10:41:46.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.134+0000 7fec66813740 1 -- 172.21.15.31:0/2861936536 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fec48004250 con 0x55e4c5a17260 2024-03-21T10:41:46.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.139+0000 7fec42a00640 1 -- 172.21.15.31:0/2861936536 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec500122b0 con 0x55e4c5a17260 2024-03-21T10:41:46.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.140+0000 7fec66813740 10 client.26428 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:46.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.140+0000 7fec66813740 1 -- 172.21.15.31:0/2861936536 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55e4c57587d0 con 0x55e4c5a17260 2024-03-21T10:41:46.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.143+0000 7fec42a00640 1 -- 172.21.15.31:0/2861936536 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec50003b90 con 0x55e4c5a17260 2024-03-21T10:41:46.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.143+0000 7fec42a00640 1 -- 172.21.15.31:0/2861936536 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fec50010630 con 0x55e4c5a17260 2024-03-21T10:41:46.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.143+0000 7fec66813740 10 client.26428 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:46.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.143+0000 7fec66813740 10 client.26428 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:41:46.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.143+0000 7fec66813740 20 client.26428 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:46.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.143+0000 7fec66813740 1 --2- 172.21.15.31:0/2861936536 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55e4c5a2fb10 0x55e4c5a31f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:46.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.143+0000 7fec66813740 4 client.26428 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:41:46.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.143+0000 7fec66813740 1 -- 172.21.15.31:0/2861936536 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec48004250 con 0x55e4c5a2fb10 2024-03-21T10:41:46.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.144+0000 7fec5be00640 1 --2- 172.21.15.31:0/2861936536 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55e4c5a2fb10 0x55e4c5a31f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:46.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.145+0000 7fec5be00640 1 --2- 172.21.15.31:0/2861936536 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55e4c5a2fb10 0x55e4c5a31f00 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8bb2f9ce77933c08 server_cookie=eb9d99844120aff9 in_seq=0 out_seq=0 2024-03-21T10:41:46.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.145+0000 7fec42a00640 10 client.26428 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:46.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.146+0000 7fec42a00640 1 -- 172.21.15.31:0/2861936536 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7fec48004250 con 0x55e4c5a2fb10 2024-03-21T10:41:46.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.146+0000 7fec42a00640 10 client.26428 handle_command_reply: tid=0 2024-03-21T10:41:46.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.146+0000 7fec66813740 1 client.26428 shutdown 2024-03-21T10:41:46.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.147+0000 7fec66813740 1 -- 172.21.15.31:0/2861936536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4c5a17260 msgr2=0x55e4c5a1d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:46.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.147+0000 7fec66813740 1 --2- 172.21.15.31:0/2861936536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4c5a17260 0x55e4c5a1d0f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x55e4c5a17660 tx=0x7fec5000f790 comp rx=0 tx=0).stop 2024-03-21T10:41:46.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.147+0000 7fec66813740 1 -- 172.21.15.31:0/2861936536 shutdown_connections 2024-03-21T10:41:46.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.147+0000 7fec66813740 1 --2- 172.21.15.31:0/2861936536 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55e4c5a2fb10 0x55e4c5a31f00 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.147+0000 7fec66813740 1 --2- 172.21.15.31:0/2861936536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4c5a1fde0 0x55e4c5a1d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.147+0000 7fec42a00640 0 client.26428 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:46.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.148+0000 7fec66813740 1 --2- 172.21.15.31:0/2861936536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4c5a17260 0x55e4c5a1d0f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.148+0000 7fec66813740 1 --2- 172.21.15.31:0/2861936536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4c5a168b0 0x55e4c5a1cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.148+0000 7fec66813740 1 -- 172.21.15.31:0/2861936536 >> 172.21.15.31:0/2861936536 conn(0x55e4c5a10f80 msgr2=0x55e4c5a12cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:46.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.152+0000 7fec66813740 1 -- 172.21.15.31:0/2861936536 shutdown_connections 2024-03-21T10:41:46.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.153+0000 7fec66813740 1 -- 172.21.15.31:0/2861936536 wait complete. 2024-03-21T10:41:46.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.157+0000 7fec66813740 20 client.26428 trim_cache size 0 max 16384 2024-03-21T10:41:46.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.190+0000 7fec66813740 1 Processor -- start 2024-03-21T10:41:46.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.190+0000 7fec66813740 1 -- start start 2024-03-21T10:41:46.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.190+0000 7fec66813740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4c5a168b0 0x55e4c5a1f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:46.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.190+0000 7fec66813740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4c5a17260 0x55e4c5a1d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:46.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.190+0000 7fec66813740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4c5a1fde0 0x55e4c5a19e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:46.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.190+0000 7fec66813740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c1c8110 con 0x55e4c5a168b0 2024-03-21T10:41:46.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.190+0000 7fec66813740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c1c5c10 con 0x55e4c5a17260 2024-03-21T10:41:46.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.190+0000 7fec66813740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c071db0 con 0x55e4c5a1fde0 2024-03-21T10:41:46.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.191+0000 7fec5be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4c5a168b0 0x55e4c5a1f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:46.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.191+0000 7fec61200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4c5a17260 0x55e4c5a1d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:46.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.191+0000 7fec61200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4c5a17260 0x55e4c5a1d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56614/0 (socket says 172.21.15.31:56614) 2024-03-21T10:41:46.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.191+0000 7fec61200640 1 -- 172.21.15.31:0/3977967309 learned_addr learned my addr 172.21.15.31:0/3977967309 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:46.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.191+0000 7fec5be00640 1 --2- 172.21.15.31:0/3977967309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4c5a1fde0 0x55e4c5a19e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:46.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.191+0000 7fec61200640 1 -- 172.21.15.31:0/3977967309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4c5a1fde0 msgr2=0x55e4c5a19e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:46.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.191+0000 7fec61200640 1 --2- 172.21.15.31:0/3977967309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4c5a1fde0 0x55e4c5a19e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.191+0000 7fec61200640 1 -- 172.21.15.31:0/3977967309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4c5a168b0 msgr2=0x55e4c5a1f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:46.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.191+0000 7fec61200640 1 --2- 172.21.15.31:0/3977967309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4c5a168b0 0x55e4c5a1f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.191+0000 7fec61200640 1 -- 172.21.15.31:0/3977967309 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e4c5a1b120 con 0x55e4c5a17260 2024-03-21T10:41:46.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.191+0000 7fec5be00640 1 --2- 172.21.15.31:0/3977967309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4c5a168b0 0x55e4c5a1f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:41:46.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.192+0000 7fec61200640 1 --2- 172.21.15.31:0/3977967309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4c5a17260 0x55e4c5a1d450 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fec50000f80 tx=0x7fec50009bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=600391481461818b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:46.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.192+0000 7fec4d600640 1 -- 172.21.15.31:0/3977967309 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec50012110 con 0x55e4c5a17260 2024-03-21T10:41:46.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.192+0000 7fec4d600640 1 -- 172.21.15.31:0/3977967309 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec50010d20 con 0x55e4c5a17260 2024-03-21T10:41:46.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.192+0000 7fec4d600640 1 -- 172.21.15.31:0/3977967309 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec500050d0 con 0x55e4c5a17260 2024-03-21T10:41:46.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.193+0000 7fec66813740 1 -- 172.21.15.31:0/3977967309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4c5a17260 msgr2=0x55e4c5a1d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:46.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.193+0000 7fec66813740 1 --2- 172.21.15.31:0/3977967309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4c5a17260 0x55e4c5a1d450 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fec50000f80 tx=0x7fec50009bf0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.193+0000 7fec66813740 1 -- 172.21.15.31:0/3977967309 shutdown_connections 2024-03-21T10:41:46.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.193+0000 7fec66813740 1 --2- 172.21.15.31:0/3977967309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4c5a1fde0 0x55e4c5a19e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.193+0000 7fec66813740 1 --2- 172.21.15.31:0/3977967309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4c5a17260 0x55e4c5a1d450 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.193+0000 7fec66813740 1 --2- 172.21.15.31:0/3977967309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4c5a168b0 0x55e4c5a1f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.193+0000 7fec66813740 1 -- 172.21.15.31:0/3977967309 >> 172.21.15.31:0/3977967309 conn(0x55e4c5a10f80 msgr2=0x55e4c5a17f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:46.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.193+0000 7fec66813740 1 -- 172.21.15.31:0/3977967309 shutdown_connections 2024-03-21T10:41:46.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.193+0000 7fec66813740 1 -- 172.21.15.31:0/3977967309 wait complete. 2024-03-21T10:41:46.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.194+0000 7fec66813740 1 -- start start 2024-03-21T10:41:46.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.194+0000 7fec66813740 1 Processor -- start 2024-03-21T10:41:46.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.194+0000 7fec66813740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4c5a168b0 0x55e4c5a25b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:46.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.194+0000 7fec66813740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4c5a17260 0x55e4c5a26070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:46.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.194+0000 7fec66813740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4c5a1fde0 0x55e4c5a8a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:46.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.194+0000 7fec66813740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c0723b0 con 0x55e4c5a168b0 2024-03-21T10:41:46.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.194+0000 7fec66813740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c0736b0 con 0x55e4c5a1fde0 2024-03-21T10:41:46.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.194+0000 7fec66813740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c0729f0 con 0x55e4c5a17260 2024-03-21T10:41:46.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.194+0000 7fec66813740 10 client.0 _ll_register_callbacks cb 0x7fec50009cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:46.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.195+0000 7fec61200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4c5a17260 0x55e4c5a26070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:46.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.195+0000 7fec61200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4c5a17260 0x55e4c5a26070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:51668/0 (socket says 172.21.15.31:51668) 2024-03-21T10:41:46.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.195+0000 7fec61200640 1 -- 172.21.15.31:0/4147732239 learned_addr learned my addr 172.21.15.31:0/4147732239 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:46.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.195+0000 7fec5be00640 1 --2- 172.21.15.31:0/4147732239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4c5a1fde0 0x55e4c5a8a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:46.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.195+0000 7fec5be00640 1 --2- 172.21.15.31:0/4147732239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4c5a168b0 0x55e4c5a25b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:46.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.195+0000 7fec61200640 1 -- 172.21.15.31:0/4147732239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4c5a1fde0 msgr2=0x55e4c5a8a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:46.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.195+0000 7fec61200640 1 --2- 172.21.15.31:0/4147732239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4c5a1fde0 0x55e4c5a8a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.195+0000 7fec61200640 1 -- 172.21.15.31:0/4147732239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4c5a168b0 msgr2=0x55e4c5a25b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:46.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.195+0000 7fec61200640 1 --2- 172.21.15.31:0/4147732239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4c5a168b0 0x55e4c5a25b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.195+0000 7fec61200640 1 -- 172.21.15.31:0/4147732239 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e4c5a8a870 con 0x55e4c5a17260 2024-03-21T10:41:46.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.195+0000 7fec5be00640 1 --2- 172.21.15.31:0/4147732239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4c5a168b0 0x55e4c5a25b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:41:46.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.195+0000 7fec61200640 1 --2- 172.21.15.31:0/4147732239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4c5a17260 0x55e4c5a26070 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fec50010010 tx=0x7fec50009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af47df3df20602e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:46.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.195+0000 7fec42a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:41:46.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.196+0000 7fec42a00640 1 -- 172.21.15.31:0/4147732239 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec50035060 con 0x55e4c5a17260 2024-03-21T10:41:46.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.196+0000 7fec42a00640 1 -- 172.21.15.31:0/4147732239 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fec50010430 con 0x55e4c5a17260 2024-03-21T10:41:46.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.196+0000 7fec66813740 1 -- 172.21.15.31:0/4147732239 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fec50010d20 con 0x55e4c5a17260 2024-03-21T10:41:46.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.196+0000 7fec42a00640 1 -- 172.21.15.31:0/4147732239 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec50010630 con 0x55e4c5a17260 2024-03-21T10:41:46.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.196+0000 7fec66813740 10 client.26434 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:46.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.196+0000 7fec66813740 1 -- 172.21.15.31:0/4147732239 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55e4c5a8baa0 con 0x55e4c5a17260 2024-03-21T10:41:46.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.197+0000 7fec42a00640 1 -- 172.21.15.31:0/4147732239 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec500107d0 con 0x55e4c5a17260 2024-03-21T10:41:46.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.197+0000 7fec42a00640 1 -- 172.21.15.31:0/4147732239 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fec50030550 con 0x55e4c5a17260 2024-03-21T10:41:46.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.198+0000 7fec66813740 10 client.26434 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:46.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.198+0000 7fec66813740 10 client.26434 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:41:46.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.198+0000 7fec66813740 20 client.26434 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:46.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.198+0000 7fec66813740 1 --2- 172.21.15.31:0/4147732239 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55e4c5a2fb10 0x55e4c5a8e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:46.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.198+0000 7fec66813740 4 client.26434 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:41:46.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.198+0000 7fec66813740 1 -- 172.21.15.31:0/4147732239 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7fec50010d20 con 0x55e4c5a2fb10 2024-03-21T10:41:46.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.198+0000 7fec5be00640 1 --2- 172.21.15.31:0/4147732239 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55e4c5a2fb10 0x55e4c5a8e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:46.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.199+0000 7fec5be00640 1 --2- 172.21.15.31:0/4147732239 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55e4c5a2fb10 0x55e4c5a8e060 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=693b8ce36450d873 server_cookie=5f12542fdc624dfe in_seq=0 out_seq=0 2024-03-21T10:41:46.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.199+0000 7fec42a00640 10 client.26434 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:46.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.199+0000 7fec42a00640 1 -- 172.21.15.31:0/4147732239 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7fec50010d20 con 0x55e4c5a2fb10 2024-03-21T10:41:46.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.199+0000 7fec42a00640 10 client.26434 handle_command_reply: tid=0 2024-03-21T10:41:46.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.199+0000 7fec66813740 1 client.26434 shutdown 2024-03-21T10:41:46.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.199+0000 7fec66813740 1 -- 172.21.15.31:0/4147732239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4c5a17260 msgr2=0x55e4c5a26070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:46.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.199+0000 7fec66813740 1 --2- 172.21.15.31:0/4147732239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4c5a17260 0x55e4c5a26070 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fec50010010 tx=0x7fec50009500 comp rx=0 tx=0).stop 2024-03-21T10:41:46.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.199+0000 7fec66813740 1 -- 172.21.15.31:0/4147732239 shutdown_connections 2024-03-21T10:41:46.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.199+0000 7fec66813740 1 --2- 172.21.15.31:0/4147732239 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55e4c5a2fb10 0x55e4c5a8e060 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.199+0000 7fec66813740 1 --2- 172.21.15.31:0/4147732239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4c5a1fde0 0x55e4c5a8a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.199+0000 7fec66813740 1 --2- 172.21.15.31:0/4147732239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4c5a17260 0x55e4c5a26070 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.199+0000 7fec42a00640 0 client.26434 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:46.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.199+0000 7fec66813740 1 --2- 172.21.15.31:0/4147732239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4c5a168b0 0x55e4c5a25b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.199+0000 7fec66813740 1 -- 172.21.15.31:0/4147732239 >> 172.21.15.31:0/4147732239 conn(0x55e4c5a10f80 msgr2=0x55e4c5a17640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:46.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.199+0000 7fec66813740 1 -- 172.21.15.31:0/4147732239 shutdown_connections 2024-03-21T10:41:46.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.199+0000 7fec66813740 1 -- 172.21.15.31:0/4147732239 wait complete. 2024-03-21T10:41:46.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.200+0000 7fec66813740 20 client.26434 trim_cache size 0 max 16384 2024-03-21T10:41:46.200 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:41:46.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.200+0000 7fec62600640 1 -- 172.21.15.31:0/2417218935 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fec440a31e0 msgr2=0x7fec440a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:46.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.200+0000 7fec62600640 1 --2- 172.21.15.31:0/2417218935 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fec440a31e0 0x7fec440a56a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fec54097840 tx=0x7fec54097040 comp rx=0 tx=0).stop 2024-03-21T10:41:46.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.200+0000 7fec62600640 1 -- 172.21.15.31:0/2417218935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec5c109690 msgr2=0x7fec5c10d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:46.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.200+0000 7fec62600640 1 --2- 172.21.15.31:0/2417218935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec5c109690 0x7fec5c10d7b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fec5400d680 tx=0x7fec5400db40 comp rx=0 tx=0).stop 2024-03-21T10:41:46.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.200+0000 7fec62600640 1 -- 172.21.15.31:0/2417218935 shutdown_connections 2024-03-21T10:41:46.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.200+0000 7fec62600640 1 --2- 172.21.15.31:0/2417218935 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fec440a31e0 0x7fec440a56a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.200+0000 7fec62600640 1 --2- 172.21.15.31:0/2417218935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec5c109690 0x7fec5c10d7b0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.200+0000 7fec62600640 1 --2- 172.21.15.31:0/2417218935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec5c104040 0x7fec5c10d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.200+0000 7fec62600640 1 --2- 172.21.15.31:0/2417218935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec5c103680 0x7fec5c1101d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:46.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.200+0000 7fec62600640 1 -- 172.21.15.31:0/2417218935 >> 172.21.15.31:0/2417218935 conn(0x7fec5c0ff110 msgr2=0x7fec5c0ff4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:46.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.201+0000 7fec62600640 1 -- 172.21.15.31:0/2417218935 shutdown_connections 2024-03-21T10:41:46.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:46.201+0000 7fec62600640 1 -- 172.21.15.31:0/2417218935 wait complete. 2024-03-21T10:41:47.223 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-21T10:41:47.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:41:47.223 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-21T10:41:47.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:46 smithi031 ceph-mon[31673]: pgmap v99: 129 pgs: 129 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 261 MiB/s wr, 1.61k op/s 2024-03-21T10:41:47.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:46 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:41:47.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:46 smithi070 ceph-mon[36330]: pgmap v99: 129 pgs: 129 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 261 MiB/s wr, 1.61k op/s 2024-03-21T10:41:47.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:46 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:41:47.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:46 smithi040 ceph-mon[36154]: pgmap v99: 129 pgs: 129 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 261 MiB/s wr, 1.61k op/s 2024-03-21T10:41:47.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:46 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:41:47.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.674+0000 7ff460c00640 1 -- 172.21.15.31:0/1462580279 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff45c114340 msgr2=0x7ff45c116730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:47.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.674+0000 7ff460c00640 1 --2- 172.21.15.31:0/1462580279 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff45c114340 0x7ff45c116730 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff454009be0 tx=0x7ff45402bef0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.674+0000 7ff460c00640 1 -- 172.21.15.31:0/1462580279 shutdown_connections 2024-03-21T10:41:47.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.674+0000 7ff460c00640 1 --2- 172.21.15.31:0/1462580279 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff45c114340 0x7ff45c116730 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.674+0000 7ff460c00640 1 --2- 172.21.15.31:0/1462580279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff45c108be0 0x7ff45c109060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.674+0000 7ff460c00640 1 --2- 172.21.15.31:0/1462580279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff45c06d8d0 0x7ff45c1086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.674+0000 7ff460c00640 1 -- 172.21.15.31:0/1462580279 >> 172.21.15.31:0/1462580279 conn(0x7ff45c06c580 msgr2=0x7ff45c06c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:47.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.675+0000 7ff460c00640 1 -- 172.21.15.31:0/1462580279 shutdown_connections 2024-03-21T10:41:47.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.675+0000 7ff460c00640 1 -- 172.21.15.31:0/1462580279 wait complete. 2024-03-21T10:41:47.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.676+0000 7ff460c00640 1 Processor -- start 2024-03-21T10:41:47.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.676+0000 7ff460c00640 1 -- start start 2024-03-21T10:41:47.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.677+0000 7ff460c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff45c06d8d0 0x7ff45c1a0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:47.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.677+0000 7ff460c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff45c108be0 0x7ff45c1a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:47.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.677+0000 7ff460c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff45c114340 0x7ff45c1a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:47.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.677+0000 7ff460c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c11a090 con 0x7ff45c114340 2024-03-21T10:41:47.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.677+0000 7ff460c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c119f10 con 0x7ff45c06d8d0 2024-03-21T10:41:47.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.677+0000 7ff460c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c11a210 con 0x7ff45c108be0 2024-03-21T10:41:47.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.677+0000 7ff45aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff45c108be0 0x7ff45c1a0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:47.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.677+0000 7ff45aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff45c108be0 0x7ff45c1a0a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:40282/0 (socket says 172.21.15.31:40282) 2024-03-21T10:41:47.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.677+0000 7ff45aa00640 1 -- 172.21.15.31:0/369087974 learned_addr learned my addr 172.21.15.31:0/369087974 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:47.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.677+0000 7ff45b400640 1 --2- 172.21.15.31:0/369087974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff45c06d8d0 0x7ff45c1a0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:47.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.678+0000 7ff45be00640 1 --2- 172.21.15.31:0/369087974 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff45c114340 0x7ff45c1a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:47.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.678+0000 7ff45b400640 1 -- 172.21.15.31:0/369087974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff45c108be0 msgr2=0x7ff45c1a0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:47.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.678+0000 7ff45b400640 1 --2- 172.21.15.31:0/369087974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff45c108be0 0x7ff45c1a0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.678+0000 7ff45b400640 1 -- 172.21.15.31:0/369087974 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff45c114340 msgr2=0x7ff45c1a4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:47.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.678+0000 7ff45b400640 1 --2- 172.21.15.31:0/369087974 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff45c114340 0x7ff45c1a4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.678+0000 7ff45b400640 1 -- 172.21.15.31:0/369087974 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45c1a5540 con 0x7ff45c06d8d0 2024-03-21T10:41:47.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.678+0000 7ff45aa00640 1 --2- 172.21.15.31:0/369087974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff45c108be0 0x7ff45c1a0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:41:47.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.678+0000 7ff45be00640 1 --2- 172.21.15.31:0/369087974 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff45c114340 0x7ff45c1a4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:41:47.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.678+0000 7ff45b400640 1 --2- 172.21.15.31:0/369087974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff45c06d8d0 0x7ff45c1a0550 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff44c00e790 tx=0x7ff44c00ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:47.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.678+0000 7ff453e00640 1 -- 172.21.15.31:0/369087974 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44c020070 con 0x7ff45c06d8d0 2024-03-21T10:41:47.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.679+0000 7ff460c00640 1 -- 172.21.15.31:0/369087974 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff45c1a5830 con 0x7ff45c06d8d0 2024-03-21T10:41:47.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.679+0000 7ff453e00640 1 -- 172.21.15.31:0/369087974 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff44c00ccc0 con 0x7ff45c06d8d0 2024-03-21T10:41:47.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.679+0000 7ff453e00640 1 -- 172.21.15.31:0/369087974 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44c01ba30 con 0x7ff45c06d8d0 2024-03-21T10:41:47.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.679+0000 7ff460c00640 1 -- 172.21.15.31:0/369087974 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff45c10a040 con 0x7ff45c06d8d0 2024-03-21T10:41:47.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.682+0000 7ff453e00640 1 -- 172.21.15.31:0/369087974 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7ff44c023070 con 0x7ff45c06d8d0 2024-03-21T10:41:47.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.684+0000 7ff464f65740 1 Processor -- start 2024-03-21T10:41:47.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.684+0000 7ff464f65740 1 -- start start 2024-03-21T10:41:47.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.684+0000 7ff464f65740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56395d2f6290 0x56395d2f6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:47.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.685+0000 7ff464f65740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d2f6bb0 0x56395d2ff250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:47.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.685+0000 7ff464f65740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56395d2ff790 0x56395d301b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:47.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.685+0000 7ff464f65740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56395d25aeb0 con 0x56395d2f6290 2024-03-21T10:41:47.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.685+0000 7ff464f65740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56395d203a60 con 0x56395d2ff790 2024-03-21T10:41:47.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.685+0000 7ff464f65740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56395d256f50 con 0x56395d2f6bb0 2024-03-21T10:41:47.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.685+0000 7ff453e00640 1 --2- 172.21.15.31:0/369087974 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff4400a3190 0x7ff4400a5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:47.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.685+0000 7ff453e00640 1 -- 172.21.15.31:0/369087974 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7ff44c0d3a70 con 0x7ff45c06d8d0 2024-03-21T10:41:47.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.686+0000 7ff45aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56395d2ff790 0x56395d301b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:47.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.686+0000 7ff45aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56395d2ff790 0x56395d301b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42784/0 (socket says 172.21.15.31:42784) 2024-03-21T10:41:47.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.686+0000 7ff45aa00640 1 -- 172.21.15.31:0/1043103104 learned_addr learned my addr 172.21.15.31:0/1043103104 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:47.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.686+0000 7ff45be00640 1 --2- 172.21.15.31:0/1043103104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56395d2f6290 0x56395d2f6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:47.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.687+0000 7ff45be00640 1 -- 172.21.15.31:0/1043103104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d2f6bb0 msgr2=0x56395d2ff250 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:41:47.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.687+0000 7ff45be00640 1 --2- 172.21.15.31:0/1043103104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d2f6bb0 0x56395d2ff250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.687+0000 7ff45be00640 1 -- 172.21.15.31:0/1043103104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56395d2ff790 msgr2=0x56395d301b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:47.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.687+0000 7ff45be00640 1 --2- 172.21.15.31:0/1043103104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56395d2ff790 0x56395d301b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.687+0000 7ff45be00640 1 -- 172.21.15.31:0/1043103104 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56395d0377e0 con 0x56395d2f6290 2024-03-21T10:41:47.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.688+0000 7ff45be00640 1 --2- 172.21.15.31:0/369087974 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff4400a3190 0x7ff4400a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:47.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.689+0000 7ff45be00640 1 --2- 172.21.15.31:0/1043103104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56395d2f6290 0x56395d2f6670 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff45402c430 tx=0x7ff4540098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ff921dd7a58e772 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:47.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.690+0000 7ff447e00640 1 -- 172.21.15.31:0/1043103104 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff454050070 con 0x56395d2f6290 2024-03-21T10:41:47.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.690+0000 7ff447e00640 1 -- 172.21.15.31:0/1043103104 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff454003e00 con 0x56395d2f6290 2024-03-21T10:41:47.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.690+0000 7ff447e00640 1 -- 172.21.15.31:0/1043103104 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff454004510 con 0x56395d2f6290 2024-03-21T10:41:47.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.690+0000 7ff45be00640 1 --2- 172.21.15.31:0/369087974 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff4400a3190 0x7ff4400a5650 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff454004af0 tx=0x7ff45402bef0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:47.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.692+0000 7ff464f65740 1 -- 172.21.15.31:0/1043103104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56395d2f6290 msgr2=0x56395d2f6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:47.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.692+0000 7ff464f65740 1 --2- 172.21.15.31:0/1043103104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56395d2f6290 0x56395d2f6670 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff45402c430 tx=0x7ff4540098e0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.692+0000 7ff464f65740 1 -- 172.21.15.31:0/1043103104 shutdown_connections 2024-03-21T10:41:47.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.692+0000 7ff464f65740 1 --2- 172.21.15.31:0/1043103104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56395d2ff790 0x56395d301b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.692+0000 7ff464f65740 1 --2- 172.21.15.31:0/1043103104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d2f6bb0 0x56395d2ff250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.692+0000 7ff464f65740 1 --2- 172.21.15.31:0/1043103104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56395d2f6290 0x56395d2f6670 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.692+0000 7ff464f65740 1 -- 172.21.15.31:0/1043103104 >> 172.21.15.31:0/1043103104 conn(0x56395d2f09c0 msgr2=0x56395d2f0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:47.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.693+0000 7ff464f65740 1 -- 172.21.15.31:0/1043103104 shutdown_connections 2024-03-21T10:41:47.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.693+0000 7ff464f65740 1 -- 172.21.15.31:0/1043103104 wait complete. 2024-03-21T10:41:47.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.694+0000 7ff464f65740 1 -- start start 2024-03-21T10:41:47.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.695+0000 7ff464f65740 1 Processor -- start 2024-03-21T10:41:47.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.699+0000 7ff464f65740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d2f6290 0x56395d2ff200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:47.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.699+0000 7ff464f65740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56395d2f6bb0 0x56395d2fc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:47.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.699+0000 7ff464f65740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56395d2ff790 0x56395d2fccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:47.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.699+0000 7ff464f65740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56395d302520 con 0x56395d2f6bb0 2024-03-21T10:41:47.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.699+0000 7ff464f65740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56395d216eb0 con 0x56395d2ff790 2024-03-21T10:41:47.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.699+0000 7ff464f65740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56395d3026a0 con 0x56395d2f6290 2024-03-21T10:41:47.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.699+0000 7ff464f65740 10 client.0 _ll_register_callbacks cb 0x56395d20b150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:47.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.700+0000 7ff45b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d2f6290 0x56395d2ff200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:47.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.700+0000 7ff45b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d2f6290 0x56395d2ff200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:40304/0 (socket says 172.21.15.31:40304) 2024-03-21T10:41:47.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.700+0000 7ff45b400640 1 -- 172.21.15.31:0/1278606009 learned_addr learned my addr 172.21.15.31:0/1278606009 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:47.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.700+0000 7ff45aa00640 1 --2- 172.21.15.31:0/1278606009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56395d2f6bb0 0x56395d2fc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:47.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.700+0000 7ff45be00640 1 --2- 172.21.15.31:0/1278606009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56395d2ff790 0x56395d2fccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:47.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.700+0000 7ff45b400640 1 -- 172.21.15.31:0/1278606009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56395d2ff790 msgr2=0x56395d2fccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:47.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.700+0000 7ff45b400640 1 --2- 172.21.15.31:0/1278606009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56395d2ff790 0x56395d2fccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.700+0000 7ff45b400640 1 -- 172.21.15.31:0/1278606009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56395d2f6bb0 msgr2=0x56395d2fc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:47.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.700+0000 7ff45b400640 1 --2- 172.21.15.31:0/1278606009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56395d2f6bb0 0x56395d2fc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.700+0000 7ff45b400640 1 -- 172.21.15.31:0/1278606009 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56395d097f70 con 0x56395d2f6290 2024-03-21T10:41:47.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.701+0000 7ff45b400640 1 --2- 172.21.15.31:0/1278606009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d2f6290 0x56395d2ff200 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff44c005fd0 tx=0x7ff44c09f880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c7ef3957358ab11 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:47.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.701+0000 7ff444c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:41:47.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.701+0000 7ff444c00640 1 -- 172.21.15.31:0/1278606009 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44c0d12b0 con 0x56395d2f6290 2024-03-21T10:41:47.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.701+0000 7ff444c00640 1 -- 172.21.15.31:0/1278606009 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7ff44c01bdd0 con 0x56395d2f6290 2024-03-21T10:41:47.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.701+0000 7ff464f65740 1 -- 172.21.15.31:0/1278606009 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff454003e00 con 0x56395d2f6290 2024-03-21T10:41:47.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.701+0000 7ff444c00640 1 -- 172.21.15.31:0/1278606009 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff44c0a85e0 con 0x56395d2f6290 2024-03-21T10:41:47.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.702+0000 7ff464f65740 10 client.26443 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:47.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.702+0000 7ff464f65740 1 -- 172.21.15.31:0/1278606009 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56395cf2f170 con 0x56395d2f6290 2024-03-21T10:41:47.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.702+0000 7ff444c00640 1 -- 172.21.15.31:0/1278606009 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff454003e00 con 0x56395d2f6290 2024-03-21T10:41:47.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.703+0000 7ff444c00640 1 -- 172.21.15.31:0/1278606009 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7ff44c09eb40 con 0x56395d2f6290 2024-03-21T10:41:47.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.703+0000 7ff464f65740 10 client.26443 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:47.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.703+0000 7ff464f65740 10 client.26443 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:41:47.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.703+0000 7ff464f65740 20 client.26443 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:47.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.703+0000 7ff464f65740 1 --2- 172.21.15.31:0/1278606009 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56395d30f8a0 0x56395d311c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:47.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.703+0000 7ff464f65740 4 client.26443 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:41:47.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.703+0000 7ff464f65740 1 -- 172.21.15.31:0/1278606009 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56395d2683f0 con 0x56395d30f8a0 2024-03-21T10:41:47.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.704+0000 7ff45aa00640 1 --2- 172.21.15.31:0/1278606009 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56395d30f8a0 0x56395d311c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:47.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.705+0000 7ff45aa00640 1 --2- 172.21.15.31:0/1278606009 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56395d30f8a0 0x56395d311c90 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac33b7de3e0c7dff server_cookie=f79dc2eace5c690f in_seq=0 out_seq=0 2024-03-21T10:41:47.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.705+0000 7ff444c00640 10 client.26443 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:47.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.706+0000 7ff444c00640 1 -- 172.21.15.31:0/1278606009 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56395d2683f0 con 0x56395d30f8a0 2024-03-21T10:41:47.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.706+0000 7ff444c00640 10 client.26443 handle_command_reply: tid=0 2024-03-21T10:41:47.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.706+0000 7ff464f65740 1 client.26443 shutdown 2024-03-21T10:41:47.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.707+0000 7ff464f65740 1 -- 172.21.15.31:0/1278606009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d2f6290 msgr2=0x56395d2ff200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:47.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.707+0000 7ff464f65740 1 --2- 172.21.15.31:0/1278606009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d2f6290 0x56395d2ff200 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff44c005fd0 tx=0x7ff44c09f880 comp rx=0 tx=0).stop 2024-03-21T10:41:47.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.711+0000 7ff464f65740 1 -- 172.21.15.31:0/1278606009 shutdown_connections 2024-03-21T10:41:47.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.711+0000 7ff464f65740 1 --2- 172.21.15.31:0/1278606009 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56395d30f8a0 0x56395d311c90 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.711+0000 7ff464f65740 1 --2- 172.21.15.31:0/1278606009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56395d2ff790 0x56395d2fccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.711+0000 7ff464f65740 1 --2- 172.21.15.31:0/1278606009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56395d2f6bb0 0x56395d2fc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.711+0000 7ff464f65740 1 --2- 172.21.15.31:0/1278606009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d2f6290 0x56395d2ff200 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.711+0000 7ff464f65740 1 -- 172.21.15.31:0/1278606009 >> 172.21.15.31:0/1278606009 conn(0x56395d2f09c0 msgr2=0x56395d2f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:47.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.711+0000 7ff444c00640 0 client.26443 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:47.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.711+0000 7ff464f65740 1 -- 172.21.15.31:0/1278606009 shutdown_connections 2024-03-21T10:41:47.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.712+0000 7ff464f65740 1 -- 172.21.15.31:0/1278606009 wait complete. 2024-03-21T10:41:47.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.713+0000 7ff464f65740 20 client.26443 trim_cache size 0 max 16384 2024-03-21T10:41:47.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.756+0000 7ff464f65740 1 Processor -- start 2024-03-21T10:41:47.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.756+0000 7ff464f65740 1 -- start start 2024-03-21T10:41:47.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.756+0000 7ff464f65740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d2f6290 0x56395d2fecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:47.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.756+0000 7ff464f65740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56395d2f6bb0 0x56395d2ff200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:47.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.757+0000 7ff464f65740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56395d2ff790 0x56395d30adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:47.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.757+0000 7ff464f65740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c1b7260 con 0x56395d2ff790 2024-03-21T10:41:47.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.757+0000 7ff464f65740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c1b4d60 con 0x56395d2f6bb0 2024-03-21T10:41:47.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.757+0000 7ff464f65740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c1b2e00 con 0x56395d2f6290 2024-03-21T10:41:47.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.757+0000 7ff45b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d2f6290 0x56395d2fecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:47.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.757+0000 7ff45b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d2f6290 0x56395d2fecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:40320/0 (socket says 172.21.15.31:40320) 2024-03-21T10:41:47.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.757+0000 7ff45b400640 1 -- 172.21.15.31:0/3988561377 learned_addr learned my addr 172.21.15.31:0/3988561377 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:47.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.757+0000 7ff45be00640 1 --2- 172.21.15.31:0/3988561377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56395d2ff790 0x56395d30adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:47.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.757+0000 7ff45b400640 1 -- 172.21.15.31:0/3988561377 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56395d2f6bb0 msgr2=0x56395d2ff200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:41:47.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.757+0000 7ff45b400640 1 --2- 172.21.15.31:0/3988561377 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56395d2f6bb0 0x56395d2ff200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.757+0000 7ff45b400640 1 -- 172.21.15.31:0/3988561377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56395d2ff790 msgr2=0x56395d30adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:47.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.757+0000 7ff45b400640 1 --2- 172.21.15.31:0/3988561377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56395d2ff790 0x56395d30adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.757+0000 7ff45b400640 1 -- 172.21.15.31:0/3988561377 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56395d2f5a30 con 0x56395d2f6290 2024-03-21T10:41:47.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.757+0000 7ff45b400640 1 --2- 172.21.15.31:0/3988561377 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d2f6290 0x56395d2fecc0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff44c09fe40 tx=0x7ff44c09f8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7627930267e46f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:47.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.758+0000 7ff447400640 1 -- 172.21.15.31:0/3988561377 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44c0d12b0 con 0x56395d2f6290 2024-03-21T10:41:47.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.758+0000 7ff447400640 1 -- 172.21.15.31:0/3988561377 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff44c082070 con 0x56395d2f6290 2024-03-21T10:41:47.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.758+0000 7ff447400640 1 -- 172.21.15.31:0/3988561377 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44c0a8150 con 0x56395d2f6290 2024-03-21T10:41:47.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.759+0000 7ff464f65740 1 -- 172.21.15.31:0/3988561377 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d2f6290 msgr2=0x56395d2fecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:47.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.759+0000 7ff464f65740 1 --2- 172.21.15.31:0/3988561377 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d2f6290 0x56395d2fecc0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff44c09fe40 tx=0x7ff44c09f8f0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.760+0000 7ff464f65740 1 -- 172.21.15.31:0/3988561377 shutdown_connections 2024-03-21T10:41:47.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.760+0000 7ff464f65740 1 --2- 172.21.15.31:0/3988561377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56395d2ff790 0x56395d30adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.760+0000 7ff464f65740 1 --2- 172.21.15.31:0/3988561377 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56395d2f6bb0 0x56395d2ff200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.760+0000 7ff464f65740 1 --2- 172.21.15.31:0/3988561377 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d2f6290 0x56395d2fecc0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.760+0000 7ff464f65740 1 -- 172.21.15.31:0/3988561377 >> 172.21.15.31:0/3988561377 conn(0x56395d2f09c0 msgr2=0x56395d2efa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:47.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.760+0000 7ff464f65740 1 -- 172.21.15.31:0/3988561377 shutdown_connections 2024-03-21T10:41:47.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.760+0000 7ff464f65740 1 -- 172.21.15.31:0/3988561377 wait complete. 2024-03-21T10:41:47.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.761+0000 7ff464f65740 1 -- start start 2024-03-21T10:41:47.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.761+0000 7ff464f65740 1 Processor -- start 2024-03-21T10:41:47.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.765+0000 7ff464f65740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56395d2f6290 0x56395d2f9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:47.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.765+0000 7ff464f65740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d2f6bb0 0x56395d2f9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:47.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.765+0000 7ff464f65740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56395d2ff790 0x56395d36c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:47.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.765+0000 7ff464f65740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c1b3400 con 0x56395d2ff790 2024-03-21T10:41:47.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.765+0000 7ff464f65740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c1b4700 con 0x56395d2f6290 2024-03-21T10:41:47.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.765+0000 7ff464f65740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c1b3a40 con 0x56395d2f6bb0 2024-03-21T10:41:47.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.765+0000 7ff464f65740 10 client.0 _ll_register_callbacks cb 0x7ff44c009600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:47.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.766+0000 7ff45b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56395d2f6290 0x56395d2f9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:47.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.766+0000 7ff45b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56395d2f6290 0x56395d2f9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42810/0 (socket says 172.21.15.31:42810) 2024-03-21T10:41:47.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.766+0000 7ff45b400640 1 -- 172.21.15.31:0/2257202555 learned_addr learned my addr 172.21.15.31:0/2257202555 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:47.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.766+0000 7ff45be00640 1 --2- 172.21.15.31:0/2257202555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56395d2ff790 0x56395d36c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:47.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.767+0000 7ff45b400640 1 -- 172.21.15.31:0/2257202555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d2f6bb0 msgr2=0x56395d2f9a50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:41:47.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.767+0000 7ff45b400640 1 --2- 172.21.15.31:0/2257202555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d2f6bb0 0x56395d2f9a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.767+0000 7ff45b400640 1 -- 172.21.15.31:0/2257202555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56395d2ff790 msgr2=0x56395d36c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:47.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.767+0000 7ff45b400640 1 --2- 172.21.15.31:0/2257202555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56395d2ff790 0x56395d36c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.767+0000 7ff45b400640 1 -- 172.21.15.31:0/2257202555 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56395d36ce80 con 0x56395d2f6290 2024-03-21T10:41:47.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.770+0000 7ff45b400640 1 --2- 172.21.15.31:0/2257202555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56395d2f6290 0x56395d2f9510 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff44c09c0e0 tx=0x7ff44c0a8380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8090540d84bd83d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:47.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.770+0000 7ff444c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:41:47.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.770+0000 7ff444c00640 1 -- 172.21.15.31:0/2257202555 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44c0d12b0 con 0x56395d2f6290 2024-03-21T10:41:47.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.770+0000 7ff444c00640 1 -- 172.21.15.31:0/2257202555 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7ff44c09f0d0 con 0x56395d2f6290 2024-03-21T10:41:47.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.771+0000 7ff45be00640 1 --2- 172.21.15.31:0/2257202555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56395d2ff790 0x56395d36c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:41:47.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.771+0000 7ff464f65740 1 -- 172.21.15.31:0/2257202555 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff44c082070 con 0x56395d2f6290 2024-03-21T10:41:47.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.772+0000 7ff444c00640 1 -- 172.21.15.31:0/2257202555 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff44c09f2d0 con 0x56395d2f6290 2024-03-21T10:41:47.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.774+0000 7ff464f65740 10 client.26501 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:47.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.774+0000 7ff464f65740 1 -- 172.21.15.31:0/2257202555 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56395d374240 con 0x56395d2f6290 2024-03-21T10:41:47.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.775+0000 7ff444c00640 1 -- 172.21.15.31:0/2257202555 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff44c082070 con 0x56395d2f6290 2024-03-21T10:41:47.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.775+0000 7ff444c00640 1 -- 172.21.15.31:0/2257202555 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7ff44c08a230 con 0x56395d2f6290 2024-03-21T10:41:47.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.776+0000 7ff464f65740 10 client.26501 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:47.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.776+0000 7ff464f65740 10 client.26501 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:41:47.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.776+0000 7ff464f65740 20 client.26501 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:47.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.776+0000 7ff464f65740 1 --2- 172.21.15.31:0/2257202555 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56395d374530 0x56395d376920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:47.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.776+0000 7ff464f65740 4 client.26501 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-21T10:41:47.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.776+0000 7ff464f65740 1 -- 172.21.15.31:0/2257202555 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56395d262740 con 0x56395d374530 2024-03-21T10:41:47.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.777+0000 7ff45aa00640 1 --2- 172.21.15.31:0/2257202555 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56395d374530 0x56395d376920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:47.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.780+0000 7ff45aa00640 1 --2- 172.21.15.31:0/2257202555 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56395d374530 0x56395d376920 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c0e9af5b5851bf4 server_cookie=3a6b6693d88a2201 in_seq=0 out_seq=0 2024-03-21T10:41:47.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.780+0000 7ff444c00640 10 client.26501 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:47.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.781+0000 7ff444c00640 1 -- 172.21.15.31:0/2257202555 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56395d262740 con 0x56395d374530 2024-03-21T10:41:47.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.781+0000 7ff444c00640 10 client.26501 handle_command_reply: tid=0 2024-03-21T10:41:47.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.781+0000 7ff464f65740 1 client.26501 shutdown 2024-03-21T10:41:47.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.782+0000 7ff464f65740 1 -- 172.21.15.31:0/2257202555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56395d2f6290 msgr2=0x56395d2f9510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:47.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.782+0000 7ff464f65740 1 --2- 172.21.15.31:0/2257202555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56395d2f6290 0x56395d2f9510 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff44c09c0e0 tx=0x7ff44c0a8380 comp rx=0 tx=0).stop 2024-03-21T10:41:47.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.783+0000 7ff464f65740 1 -- 172.21.15.31:0/2257202555 shutdown_connections 2024-03-21T10:41:47.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.783+0000 7ff464f65740 1 --2- 172.21.15.31:0/2257202555 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56395d374530 0x56395d376920 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.783+0000 7ff464f65740 1 --2- 172.21.15.31:0/2257202555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56395d2ff790 0x56395d36c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.783+0000 7ff464f65740 1 --2- 172.21.15.31:0/2257202555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56395d2f6bb0 0x56395d2f9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.783+0000 7ff464f65740 1 --2- 172.21.15.31:0/2257202555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56395d2f6290 0x56395d2f9510 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.783+0000 7ff464f65740 1 -- 172.21.15.31:0/2257202555 >> 172.21.15.31:0/2257202555 conn(0x56395d2f09c0 msgr2=0x56395d2f1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:47.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.783+0000 7ff444c00640 0 client.26501 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:47.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.783+0000 7ff464f65740 1 -- 172.21.15.31:0/2257202555 shutdown_connections 2024-03-21T10:41:47.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.783+0000 7ff464f65740 1 -- 172.21.15.31:0/2257202555 wait complete. 2024-03-21T10:41:47.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.783+0000 7ff464f65740 20 client.26501 trim_cache size 0 max 16384 2024-03-21T10:41:47.781 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:41:47.782 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-03-21T10:41:47.782 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "91665fb1-b0a9-4215-b526-d4b604163f30", 2024-03-21T10:41:47.782 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-03-21T10:41:47.782 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:41:47.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.789+0000 7ff460c00640 1 -- 172.21.15.31:0/369087974 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff4400a3190 msgr2=0x7ff4400a5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:47.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.789+0000 7ff460c00640 1 --2- 172.21.15.31:0/369087974 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff4400a3190 0x7ff4400a5650 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff454004af0 tx=0x7ff45402bef0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.789+0000 7ff460c00640 1 -- 172.21.15.31:0/369087974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff45c06d8d0 msgr2=0x7ff45c1a0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:47.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.789+0000 7ff460c00640 1 --2- 172.21.15.31:0/369087974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff45c06d8d0 0x7ff45c1a0550 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff44c00e790 tx=0x7ff44c00ec50 comp rx=0 tx=0).stop 2024-03-21T10:41:47.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.790+0000 7ff460c00640 1 -- 172.21.15.31:0/369087974 shutdown_connections 2024-03-21T10:41:47.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.790+0000 7ff460c00640 1 --2- 172.21.15.31:0/369087974 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff4400a3190 0x7ff4400a5650 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.790+0000 7ff460c00640 1 --2- 172.21.15.31:0/369087974 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff45c114340 0x7ff45c1a4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.790+0000 7ff460c00640 1 --2- 172.21.15.31:0/369087974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff45c108be0 0x7ff45c1a0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.790+0000 7ff460c00640 1 --2- 172.21.15.31:0/369087974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff45c06d8d0 0x7ff45c1a0550 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:47.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.790+0000 7ff460c00640 1 -- 172.21.15.31:0/369087974 >> 172.21.15.31:0/369087974 conn(0x7ff45c06c580 msgr2=0x7ff45c0714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:47.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.792+0000 7ff460c00640 1 -- 172.21.15.31:0/369087974 shutdown_connections 2024-03-21T10:41:47.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:47.792+0000 7ff460c00640 1 -- 172.21.15.31:0/369087974 wait complete. 2024-03-21T10:41:47.841 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:41:47.842 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:41:48.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.400+0000 7f6b35e00640 1 -- 172.21.15.31:0/2515249198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b30107dc0 msgr2=0x7f6b30108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:48.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.400+0000 7f6b35e00640 1 --2- 172.21.15.31:0/2515249198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b30107dc0 0x7f6b30108220 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f6b18009930 tx=0x7f6b1802bc60 comp rx=0 tx=0).stop 2024-03-21T10:41:48.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.402+0000 7f6b35e00640 1 -- 172.21.15.31:0/2515249198 shutdown_connections 2024-03-21T10:41:48.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.402+0000 7f6b35e00640 1 --2- 172.21.15.31:0/2515249198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b30108760 0x7f6b30114c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.402+0000 7f6b35e00640 1 --2- 172.21.15.31:0/2515249198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b30107dc0 0x7f6b30108220 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.402+0000 7f6b35e00640 1 --2- 172.21.15.31:0/2515249198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b30109b70 0x7f6b30109f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.402+0000 7f6b35e00640 1 -- 172.21.15.31:0/2515249198 >> 172.21.15.31:0/2515249198 conn(0x7f6b30077540 msgr2=0x7f6b301064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:48.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.403+0000 7f6b35e00640 1 -- 172.21.15.31:0/2515249198 shutdown_connections 2024-03-21T10:41:48.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.403+0000 7f6b35e00640 1 -- 172.21.15.31:0/2515249198 wait complete. 2024-03-21T10:41:48.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.406+0000 7f6b35e00640 1 Processor -- start 2024-03-21T10:41:48.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.407+0000 7f6b35e00640 1 -- start start 2024-03-21T10:41:48.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.407+0000 7f6b35e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b30107dc0 0x7f6b30072e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:48.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.407+0000 7f6b35e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b30108760 0x7f6b30073390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:48.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.408+0000 7f6b26000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b30108760 0x7f6b30073390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:48.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.408+0000 7f6b26000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b30108760 0x7f6b30073390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42838/0 (socket says 172.21.15.31:42838) 2024-03-21T10:41:48.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.408+0000 7f6b26000640 1 -- 172.21.15.31:0/690435527 learned_addr learned my addr 172.21.15.31:0/690435527 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:48.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.408+0000 7f6b35e00640 1 --2- 172.21.15.31:0/690435527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b30109b70 0x7f6b3006c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:48.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.409+0000 7f6b2f400640 1 --2- 172.21.15.31:0/690435527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b30109b70 0x7f6b3006c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:48.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.409+0000 7f6b35e00640 1 -- 172.21.15.31:0/690435527 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b301181c0 con 0x7f6b30107dc0 2024-03-21T10:41:48.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.409+0000 7f6b35e00640 1 -- 172.21.15.31:0/690435527 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b30118040 con 0x7f6b30108760 2024-03-21T10:41:48.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.409+0000 7f6b35e00640 1 -- 172.21.15.31:0/690435527 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b30118340 con 0x7f6b30109b70 2024-03-21T10:41:48.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.410+0000 7f6b2f400640 1 -- 172.21.15.31:0/690435527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b30108760 msgr2=0x7f6b30073390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:48.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.410+0000 7f6b2f400640 1 --2- 172.21.15.31:0/690435527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b30108760 0x7f6b30073390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.410+0000 7f6b2f400640 1 -- 172.21.15.31:0/690435527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b30107dc0 msgr2=0x7f6b30072e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:48.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.410+0000 7f6b2f400640 1 --2- 172.21.15.31:0/690435527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b30107dc0 0x7f6b30072e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.410+0000 7f6b2f400640 1 -- 172.21.15.31:0/690435527 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3006c970 con 0x7f6b30109b70 2024-03-21T10:41:48.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.411+0000 7f6b2f400640 1 --2- 172.21.15.31:0/690435527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b30109b70 0x7f6b3006c0b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f6b2800be20 tx=0x7f6b2800bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:48.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.411+0000 7f6b34a00640 1 -- 172.21.15.31:0/690435527 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b28019070 con 0x7f6b30109b70 2024-03-21T10:41:48.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.411+0000 7f6b34a00640 1 -- 172.21.15.31:0/690435527 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b2800cdf0 con 0x7f6b30109b70 2024-03-21T10:41:48.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.411+0000 7f6b34a00640 1 -- 172.21.15.31:0/690435527 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b280129b0 con 0x7f6b30109b70 2024-03-21T10:41:48.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.411+0000 7f6b35e00640 1 -- 172.21.15.31:0/690435527 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b3006cc60 con 0x7f6b30109b70 2024-03-21T10:41:48.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.411+0000 7f6b35e00640 1 -- 172.21.15.31:0/690435527 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b30070c10 con 0x7f6b30109b70 2024-03-21T10:41:48.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.419+0000 7f6b34a00640 1 -- 172.21.15.31:0/690435527 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f6b2801c070 con 0x7f6b30109b70 2024-03-21T10:41:48.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.419+0000 7f6b38ba5740 1 Processor -- start 2024-03-21T10:41:48.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.422+0000 7f6b34a00640 1 --2- 172.21.15.31:0/690435527 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6b100a3290 0x7f6b100a5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:48.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.422+0000 7f6b34a00640 1 -- 172.21.15.31:0/690435527 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f6b280cf380 con 0x7f6b30109b70 2024-03-21T10:41:48.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.423+0000 7f6b26000640 1 --2- 172.21.15.31:0/690435527 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6b100a3290 0x7f6b100a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:48.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.423+0000 7f6b38ba5740 1 -- start start 2024-03-21T10:41:48.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.424+0000 7f6b38ba5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2e88e3c00 0x55c2e88e2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:48.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.424+0000 7f6b38ba5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88e2f20 0x55c2e88e33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:48.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.424+0000 7f6b38ba5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c2e88ec290 0x55c2e88ee680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:48.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.424+0000 7f6b38ba5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e8847250 con 0x55c2e88ec290 2024-03-21T10:41:48.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.424+0000 7f6b38ba5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e87efa30 con 0x55c2e88e3c00 2024-03-21T10:41:48.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.424+0000 7f6b38ba5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e88432f0 con 0x55c2e88e2f20 2024-03-21T10:41:48.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.427+0000 7f6b2f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c2e88ec290 0x55c2e88ee680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:48.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.428+0000 7f6b26000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88e2f20 0x55c2e88e33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:48.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.428+0000 7f6b26000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88e2f20 0x55c2e88e33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:40348/0 (socket says 172.21.15.31:40348) 2024-03-21T10:41:48.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.428+0000 7f6b26000640 1 -- 172.21.15.31:0/868772237 learned_addr learned my addr 172.21.15.31:0/868772237 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:48.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.428+0000 7f6b2ea00640 1 --2- 172.21.15.31:0/868772237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2e88e3c00 0x55c2e88e2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:48.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.429+0000 7f6b26000640 1 --2- 172.21.15.31:0/690435527 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6b100a3290 0x7f6b100a5750 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6b3006d690 tx=0x7f6b18005850 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:48.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.429+0000 7f6b26000640 1 -- 172.21.15.31:0/868772237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2e88e3c00 msgr2=0x55c2e88e2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:48.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.429+0000 7f6b26000640 1 --2- 172.21.15.31:0/868772237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2e88e3c00 0x55c2e88e2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.429+0000 7f6b26000640 1 -- 172.21.15.31:0/868772237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c2e88ec290 msgr2=0x55c2e88ee680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:48.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.429+0000 7f6b26000640 1 --2- 172.21.15.31:0/868772237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c2e88ec290 0x55c2e88ee680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.429+0000 7f6b26000640 1 -- 172.21.15.31:0/868772237 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2e87da690 con 0x55c2e88e2f20 2024-03-21T10:41:48.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.429+0000 7f6b2f400640 1 --2- 172.21.15.31:0/868772237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c2e88ec290 0x55c2e88ee680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:41:48.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.430+0000 7f6b26000640 1 --2- 172.21.15.31:0/868772237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88e2f20 0x55c2e88e33a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6b1802c270 tx=0x7f6b1803d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c044bc0ac01b51f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:48.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.433+0000 7f6b1fe00640 1 -- 172.21.15.31:0/868772237 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1804e070 con 0x55c2e88e2f20 2024-03-21T10:41:48.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.433+0000 7f6b1fe00640 1 -- 172.21.15.31:0/868772237 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b18044070 con 0x55c2e88e2f20 2024-03-21T10:41:48.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.433+0000 7f6b1fe00640 1 -- 172.21.15.31:0/868772237 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b180035f0 con 0x55c2e88e2f20 2024-03-21T10:41:48.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.434+0000 7f6b38ba5740 1 -- 172.21.15.31:0/868772237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88e2f20 msgr2=0x55c2e88e33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:48.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.435+0000 7f6b38ba5740 1 --2- 172.21.15.31:0/868772237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88e2f20 0x55c2e88e33a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6b1802c270 tx=0x7f6b1803d000 comp rx=0 tx=0).stop 2024-03-21T10:41:48.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.435+0000 7f6b38ba5740 1 -- 172.21.15.31:0/868772237 shutdown_connections 2024-03-21T10:41:48.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.435+0000 7f6b38ba5740 1 --2- 172.21.15.31:0/868772237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c2e88ec290 0x55c2e88ee680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.435+0000 7f6b38ba5740 1 --2- 172.21.15.31:0/868772237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88e2f20 0x55c2e88e33a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.435+0000 7f6b38ba5740 1 --2- 172.21.15.31:0/868772237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2e88e3c00 0x55c2e88e2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.435+0000 7f6b38ba5740 1 -- 172.21.15.31:0/868772237 >> 172.21.15.31:0/868772237 conn(0x55c2e88aa200 msgr2=0x55c2e88dd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:48.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.435+0000 7f6b38ba5740 1 -- 172.21.15.31:0/868772237 shutdown_connections 2024-03-21T10:41:48.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.436+0000 7f6b38ba5740 1 -- 172.21.15.31:0/868772237 wait complete. 2024-03-21T10:41:48.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.436+0000 7f6b38ba5740 1 -- start start 2024-03-21T10:41:48.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.437+0000 7f6b38ba5740 1 Processor -- start 2024-03-21T10:41:48.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.441+0000 7f6b38ba5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2e88e2f20 0x55c2e88eb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:48.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.441+0000 7f6b38ba5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c2e88e3c00 0x55c2e88e8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:48.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.441+0000 7f6b38ba5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88ec290 0x55c2e88e92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:48.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.441+0000 7f6b38ba5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e88ef0a0 con 0x55c2e88e3c00 2024-03-21T10:41:48.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.441+0000 7f6b38ba5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e8802e80 con 0x55c2e88e2f20 2024-03-21T10:41:48.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.441+0000 7f6b38ba5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e88ef220 con 0x55c2e88ec290 2024-03-21T10:41:48.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.441+0000 7f6b38ba5740 10 client.0 _ll_register_callbacks cb 0x55c2e8815130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:48.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.441+0000 7f6b2f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88ec290 0x55c2e88e92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:48.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.441+0000 7f6b2f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88ec290 0x55c2e88e92a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:40352/0 (socket says 172.21.15.31:40352) 2024-03-21T10:41:48.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.441+0000 7f6b2f400640 1 -- 172.21.15.31:0/2688728708 learned_addr learned my addr 172.21.15.31:0/2688728708 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:48.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.442+0000 7f6b2ea00640 1 --2- 172.21.15.31:0/2688728708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2e88e2f20 0x55c2e88eb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:48.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.442+0000 7f6b26000640 1 --2- 172.21.15.31:0/2688728708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c2e88e3c00 0x55c2e88e8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:48.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.443+0000 7f6b2f400640 1 -- 172.21.15.31:0/2688728708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2e88e2f20 msgr2=0x55c2e88eb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:48.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.443+0000 7f6b2f400640 1 --2- 172.21.15.31:0/2688728708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2e88e2f20 0x55c2e88eb9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.443+0000 7f6b2f400640 1 -- 172.21.15.31:0/2688728708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c2e88e3c00 msgr2=0x55c2e88e8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:48.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.443+0000 7f6b2f400640 1 --2- 172.21.15.31:0/2688728708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c2e88e3c00 0x55c2e88e8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.443+0000 7f6b2f400640 1 -- 172.21.15.31:0/2688728708 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c2e86257d0 con 0x55c2e88ec290 2024-03-21T10:41:48.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.443+0000 7f6b2f400640 1 --2- 172.21.15.31:0/2688728708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88ec290 0x55c2e88e92a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6b2800bdf0 tx=0x7f6b2809b500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=184b8857ac9acc03 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:48.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.446+0000 7f6b1cc00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:41:48.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.446+0000 7f6b1cc00640 1 -- 172.21.15.31:0/2688728708 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b280b4060 con 0x55c2e88ec290 2024-03-21T10:41:48.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.446+0000 7f6b1cc00640 1 -- 172.21.15.31:0/2688728708 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f6b2809f170 con 0x55c2e88ec290 2024-03-21T10:41:48.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.446+0000 7f6b38ba5740 1 -- 172.21.15.31:0/2688728708 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6b18044070 con 0x55c2e88ec290 2024-03-21T10:41:48.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.449+0000 7f6b1cc00640 1 -- 172.21.15.31:0/2688728708 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b280b9050 con 0x55c2e88ec290 2024-03-21T10:41:48.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.449+0000 7f6b38ba5740 10 client.26458 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:48.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.452+0000 7f6b38ba5740 1 -- 172.21.15.31:0/2688728708 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55c2e855b3f0 con 0x55c2e88ec290 2024-03-21T10:41:48.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.453+0000 7f6b1cc00640 1 -- 172.21.15.31:0/2688728708 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b18044070 con 0x55c2e88ec290 2024-03-21T10:41:48.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.453+0000 7f6b1cc00640 1 -- 172.21.15.31:0/2688728708 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f6b2809a890 con 0x55c2e88ec290 2024-03-21T10:41:48.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.456+0000 7f6b38ba5740 10 client.26458 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:48.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.456+0000 7f6b38ba5740 10 client.26458 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:41:48.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.456+0000 7f6b38ba5740 20 client.26458 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:48.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.456+0000 7f6b38ba5740 1 --2- 172.21.15.31:0/2688728708 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c2e88fc250 0x55c2e88fe640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:48.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.456+0000 7f6b38ba5740 4 client.26458 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:41:48.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.456+0000 7f6b2ea00640 1 --2- 172.21.15.31:0/2688728708 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c2e88fc250 0x55c2e88fe640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:48.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.456+0000 7f6b38ba5740 1 -- 172.21.15.31:0/2688728708 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c2e8854790 con 0x55c2e88fc250 2024-03-21T10:41:48.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.458+0000 7f6b2ea00640 1 --2- 172.21.15.31:0/2688728708 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c2e88fc250 0x55c2e88fe640 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47fb1693cb235191 server_cookie=ce9d71a64aae720e in_seq=0 out_seq=0 2024-03-21T10:41:48.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.458+0000 7f6b1cc00640 10 client.26458 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:48.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.459+0000 7f6b1cc00640 1 -- 172.21.15.31:0/2688728708 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c2e8854790 con 0x55c2e88fc250 2024-03-21T10:41:48.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.459+0000 7f6b1cc00640 10 client.26458 handle_command_reply: tid=0 2024-03-21T10:41:48.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.459+0000 7f6b38ba5740 1 client.26458 shutdown 2024-03-21T10:41:48.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.461+0000 7f6b38ba5740 1 -- 172.21.15.31:0/2688728708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88ec290 msgr2=0x55c2e88e92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:48.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.461+0000 7f6b38ba5740 1 --2- 172.21.15.31:0/2688728708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88ec290 0x55c2e88e92a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6b2800bdf0 tx=0x7f6b2809b500 comp rx=0 tx=0).stop 2024-03-21T10:41:48.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.461+0000 7f6b38ba5740 1 -- 172.21.15.31:0/2688728708 shutdown_connections 2024-03-21T10:41:48.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.461+0000 7f6b38ba5740 1 --2- 172.21.15.31:0/2688728708 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c2e88fc250 0x55c2e88fe640 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.461+0000 7f6b38ba5740 1 --2- 172.21.15.31:0/2688728708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88ec290 0x55c2e88e92a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.461+0000 7f6b38ba5740 1 --2- 172.21.15.31:0/2688728708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c2e88e3c00 0x55c2e88e8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.461+0000 7f6b38ba5740 1 --2- 172.21.15.31:0/2688728708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2e88e2f20 0x55c2e88eb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.462+0000 7f6b1cc00640 0 client.26458 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:48.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.462+0000 7f6b38ba5740 1 -- 172.21.15.31:0/2688728708 >> 172.21.15.31:0/2688728708 conn(0x55c2e88aa200 msgr2=0x55c2e88df6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:48.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.462+0000 7f6b38ba5740 1 -- 172.21.15.31:0/2688728708 shutdown_connections 2024-03-21T10:41:48.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.462+0000 7f6b38ba5740 1 -- 172.21.15.31:0/2688728708 wait complete. 2024-03-21T10:41:48.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.463+0000 7f6b38ba5740 20 client.26458 trim_cache size 0 max 16384 2024-03-21T10:41:48.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.504+0000 7f6b38ba5740 1 Processor -- start 2024-03-21T10:41:48.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.505+0000 7f6b38ba5740 1 -- start start 2024-03-21T10:41:48.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.505+0000 7f6b38ba5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c2e88e2f20 0x55c2e88e7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:48.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.505+0000 7f6b38ba5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2e88e3c00 0x55c2e88eb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:48.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.505+0000 7f6b38ba5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88ec290 0x55c2e88eb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:48.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.506+0000 7f6b2ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c2e88e2f20 0x55c2e88e7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:48.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.506+0000 7f6b2ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c2e88e2f20 0x55c2e88e7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60612/0 (socket says 172.21.15.31:60612) 2024-03-21T10:41:48.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.507+0000 7f6b38ba5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b301c84f0 con 0x55c2e88e2f20 2024-03-21T10:41:48.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.507+0000 7f6b38ba5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b301c5ff0 con 0x55c2e88e3c00 2024-03-21T10:41:48.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.507+0000 7f6b38ba5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b301c4090 con 0x55c2e88ec290 2024-03-21T10:41:48.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.507+0000 7f6b2f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88ec290 0x55c2e88eb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:48.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.507+0000 7f6b2f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88ec290 0x55c2e88eb9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:40356/0 (socket says 172.21.15.31:40356) 2024-03-21T10:41:48.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.507+0000 7f6b2f400640 1 -- 172.21.15.31:0/316371999 learned_addr learned my addr 172.21.15.31:0/316371999 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:48.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.507+0000 7f6b26000640 1 --2- 172.21.15.31:0/316371999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2e88e3c00 0x55c2e88eb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:48.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.507+0000 7f6b2f400640 1 -- 172.21.15.31:0/316371999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2e88e3c00 msgr2=0x55c2e88eb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:48.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.507+0000 7f6b2f400640 1 --2- 172.21.15.31:0/316371999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2e88e3c00 0x55c2e88eb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.507+0000 7f6b2f400640 1 -- 172.21.15.31:0/316371999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c2e88e2f20 msgr2=0x55c2e88e7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:48.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.508+0000 7f6b2f400640 1 --2- 172.21.15.31:0/316371999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c2e88e2f20 0x55c2e88e7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.508+0000 7f6b2f400640 1 -- 172.21.15.31:0/316371999 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2e851b0f0 con 0x55c2e88ec290 2024-03-21T10:41:48.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.508+0000 7f6b2ea00640 1 --2- 172.21.15.31:0/316371999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c2e88e2f20 0x55c2e88e7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:41:48.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.508+0000 7f6b2f400640 1 --2- 172.21.15.31:0/316371999 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88ec290 0x55c2e88eb9d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6b2809b3d0 tx=0x7f6b28014e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85651cedcd0a3986 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:48.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.509+0000 7f6b1f400640 1 -- 172.21.15.31:0/316371999 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b280aa020 con 0x55c2e88ec290 2024-03-21T10:41:48.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.509+0000 7f6b1f400640 1 -- 172.21.15.31:0/316371999 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b2809f370 con 0x55c2e88ec290 2024-03-21T10:41:48.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.510+0000 7f6b38ba5740 1 -- 172.21.15.31:0/316371999 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88ec290 msgr2=0x55c2e88eb9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:48.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.510+0000 7f6b1f400640 1 -- 172.21.15.31:0/316371999 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b280aa020 con 0x55c2e88ec290 2024-03-21T10:41:48.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.510+0000 7f6b38ba5740 1 --2- 172.21.15.31:0/316371999 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88ec290 0x55c2e88eb9d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6b2809b3d0 tx=0x7f6b28014e90 comp rx=0 tx=0).stop 2024-03-21T10:41:48.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.511+0000 7f6b38ba5740 1 -- 172.21.15.31:0/316371999 shutdown_connections 2024-03-21T10:41:48.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.511+0000 7f6b38ba5740 1 --2- 172.21.15.31:0/316371999 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88ec290 0x55c2e88eb9d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.511+0000 7f6b38ba5740 1 --2- 172.21.15.31:0/316371999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2e88e3c00 0x55c2e88eb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.511+0000 7f6b38ba5740 1 --2- 172.21.15.31:0/316371999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c2e88e2f20 0x55c2e88e7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.511+0000 7f6b38ba5740 1 -- 172.21.15.31:0/316371999 >> 172.21.15.31:0/316371999 conn(0x55c2e88aa200 msgr2=0x55c2e88e7500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:48.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.513+0000 7f6b38ba5740 1 -- 172.21.15.31:0/316371999 shutdown_connections 2024-03-21T10:41:48.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.514+0000 7f6b38ba5740 1 -- 172.21.15.31:0/316371999 wait complete. 2024-03-21T10:41:48.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.515+0000 7f6b38ba5740 1 -- start start 2024-03-21T10:41:48.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.515+0000 7f6b38ba5740 1 Processor -- start 2024-03-21T10:41:48.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.517+0000 7f6b38ba5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c2e88e2f20 0x55c2e88eaad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:48.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.517+0000 7f6b38ba5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88e3c00 0x55c2e88eb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:48.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.517+0000 7f6b2ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c2e88e2f20 0x55c2e88eaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:48.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.518+0000 7f6b38ba5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2e88ec290 0x55c2e88f2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:48.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.518+0000 7f6b38ba5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b301c4690 con 0x55c2e88e2f20 2024-03-21T10:41:48.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.518+0000 7f6b2ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c2e88e2f20 0x55c2e88eaad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60628/0 (socket says 172.21.15.31:60628) 2024-03-21T10:41:48.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.518+0000 7f6b2ea00640 1 -- 172.21.15.31:0/4232831501 learned_addr learned my addr 172.21.15.31:0/4232831501 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:48.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.518+0000 7f6b26000640 1 --2- 172.21.15.31:0/4232831501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88e3c00 0x55c2e88eb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:48.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.518+0000 7f6b38ba5740 1 -- 172.21.15.31:0/4232831501 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b301c5990 con 0x55c2e88ec290 2024-03-21T10:41:48.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.518+0000 7f6b38ba5740 1 -- 172.21.15.31:0/4232831501 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b301c4cd0 con 0x55c2e88e3c00 2024-03-21T10:41:48.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.518+0000 7f6b38ba5740 10 client.0 _ll_register_callbacks cb 0x7f6b28097080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:48.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.520+0000 7f6b2f400640 1 --2- 172.21.15.31:0/4232831501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2e88ec290 0x55c2e88f2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:48.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.521+0000 7f6b26000640 1 -- 172.21.15.31:0/4232831501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2e88ec290 msgr2=0x55c2e88f2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:48.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.521+0000 7f6b26000640 1 --2- 172.21.15.31:0/4232831501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2e88ec290 0x55c2e88f2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.521+0000 7f6b26000640 1 -- 172.21.15.31:0/4232831501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c2e88e2f20 msgr2=0x55c2e88eaad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:48.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.521+0000 7f6b26000640 1 --2- 172.21.15.31:0/4232831501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c2e88e2f20 0x55c2e88eaad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.521+0000 7f6b26000640 1 -- 172.21.15.31:0/4232831501 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c2e88f2c20 con 0x55c2e88e3c00 2024-03-21T10:41:48.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.522+0000 7f6b1cc00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:41:48.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.522+0000 7f6b26000640 1 --2- 172.21.15.31:0/4232831501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88e3c00 0x55c2e88eb010 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f6b18009900 tx=0x7f6b180092d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34ba560217e09bf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:48.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.523+0000 7f6b1cc00640 1 -- 172.21.15.31:0/4232831501 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1804e040 con 0x55c2e88e3c00 2024-03-21T10:41:48.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.523+0000 7f6b1cc00640 1 -- 172.21.15.31:0/4232831501 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f6b180048c0 con 0x55c2e88e3c00 2024-03-21T10:41:48.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.524+0000 7f6b38ba5740 1 -- 172.21.15.31:0/4232831501 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c2e884eae0 con 0x55c2e88e3c00 2024-03-21T10:41:48.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.525+0000 7f6b1cc00640 1 -- 172.21.15.31:0/4232831501 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b18061050 con 0x55c2e88e3c00 2024-03-21T10:41:48.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.525+0000 7f6b38ba5740 10 client.26464 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:48.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.525+0000 7f6b38ba5740 1 -- 172.21.15.31:0/4232831501 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55c2e89582e0 con 0x55c2e88e3c00 2024-03-21T10:41:48.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.526+0000 7f6b1cc00640 1 -- 172.21.15.31:0/4232831501 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c2e884eae0 con 0x55c2e88e3c00 2024-03-21T10:41:48.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.527+0000 7f6b1cc00640 1 -- 172.21.15.31:0/4232831501 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f6b18049070 con 0x55c2e88e3c00 2024-03-21T10:41:48.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.527+0000 7f6b38ba5740 10 client.26464 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:48.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.527+0000 7f6b38ba5740 10 client.26464 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:41:48.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.527+0000 7f6b38ba5740 20 client.26464 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:48.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.527+0000 7f6b38ba5740 1 --2- 172.21.15.31:0/4232831501 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c2e8958610 0x55c2e895aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:48.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.527+0000 7f6b38ba5740 4 client.26464 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:41:48.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.527+0000 7f6b38ba5740 1 -- 172.21.15.31:0/4232831501 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c2e87cc3a0 con 0x55c2e8958610 2024-03-21T10:41:48.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.529+0000 7f6b2ea00640 1 --2- 172.21.15.31:0/4232831501 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c2e8958610 0x55c2e895aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:48.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.529+0000 7f6b2ea00640 1 --2- 172.21.15.31:0/4232831501 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c2e8958610 0x55c2e895aa00 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6dcc16c72cd067 server_cookie=198ba5bad82cf4e9 in_seq=0 out_seq=0 2024-03-21T10:41:48.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.529+0000 7f6b1cc00640 10 client.26464 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:48.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.530+0000 7f6b1cc00640 1 -- 172.21.15.31:0/4232831501 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55c2e87cc3a0 con 0x55c2e8958610 2024-03-21T10:41:48.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.530+0000 7f6b1cc00640 10 client.26464 handle_command_reply: tid=0 2024-03-21T10:41:48.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.530+0000 7f6b38ba5740 1 client.26464 shutdown 2024-03-21T10:41:48.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.530+0000 7f6b38ba5740 1 -- 172.21.15.31:0/4232831501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88e3c00 msgr2=0x55c2e88eb010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:48.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.530+0000 7f6b38ba5740 1 --2- 172.21.15.31:0/4232831501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88e3c00 0x55c2e88eb010 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f6b18009900 tx=0x7f6b180092d0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.531+0000 7f6b38ba5740 1 -- 172.21.15.31:0/4232831501 shutdown_connections 2024-03-21T10:41:48.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.531+0000 7f6b38ba5740 1 --2- 172.21.15.31:0/4232831501 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c2e8958610 0x55c2e895aa00 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.531+0000 7f6b38ba5740 1 --2- 172.21.15.31:0/4232831501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2e88ec290 0x55c2e88f2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.531+0000 7f6b38ba5740 1 --2- 172.21.15.31:0/4232831501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2e88e3c00 0x55c2e88eb010 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.531+0000 7f6b38ba5740 1 --2- 172.21.15.31:0/4232831501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c2e88e2f20 0x55c2e88eaad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.531+0000 7f6b38ba5740 1 -- 172.21.15.31:0/4232831501 >> 172.21.15.31:0/4232831501 conn(0x55c2e88aa200 msgr2=0x55c2e88e7d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:48.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.531+0000 7f6b1cc00640 0 client.26464 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:48.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.531+0000 7f6b38ba5740 1 -- 172.21.15.31:0/4232831501 shutdown_connections 2024-03-21T10:41:48.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.532+0000 7f6b38ba5740 1 -- 172.21.15.31:0/4232831501 wait complete. 2024-03-21T10:41:48.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.532+0000 7f6b38ba5740 20 client.26464 trim_cache size 0 max 16384 2024-03-21T10:41:48.530 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:41:48.530 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-21T10:41:48.531 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-03-21T10:41:48.531 INFO:teuthology.orchestra.run.smithi031.stdout: "91665fb1-b0a9-4215-b526-d4b604163f30": { 2024-03-21T10:41:48.531 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-03-21T10:41:48.531 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "91665fb1-b0a9-4215-b526-d4b604163f30", 2024-03-21T10:41:48.531 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-03-21T10:41:48.531 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:41:48.531 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:41:48.531 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:41:48.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.534+0000 7f6b35e00640 1 -- 172.21.15.31:0/690435527 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6b100a3290 msgr2=0x7f6b100a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:48.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.534+0000 7f6b35e00640 1 --2- 172.21.15.31:0/690435527 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6b100a3290 0x7f6b100a5750 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6b3006d690 tx=0x7f6b18005850 comp rx=0 tx=0).stop 2024-03-21T10:41:48.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.534+0000 7f6b35e00640 1 -- 172.21.15.31:0/690435527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b30109b70 msgr2=0x7f6b3006c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:48.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.534+0000 7f6b35e00640 1 --2- 172.21.15.31:0/690435527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b30109b70 0x7f6b3006c0b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f6b2800be20 tx=0x7f6b2800bf10 comp rx=0 tx=0).stop 2024-03-21T10:41:48.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.535+0000 7f6b35e00640 1 -- 172.21.15.31:0/690435527 shutdown_connections 2024-03-21T10:41:48.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.535+0000 7f6b35e00640 1 --2- 172.21.15.31:0/690435527 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6b100a3290 0x7f6b100a5750 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.535+0000 7f6b35e00640 1 --2- 172.21.15.31:0/690435527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b30109b70 0x7f6b3006c0b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.535+0000 7f6b35e00640 1 --2- 172.21.15.31:0/690435527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b30108760 0x7f6b30073390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.535+0000 7f6b35e00640 1 --2- 172.21.15.31:0/690435527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b30107dc0 0x7f6b30072e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:48.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.535+0000 7f6b35e00640 1 -- 172.21.15.31:0/690435527 >> 172.21.15.31:0/690435527 conn(0x7f6b30077540 msgr2=0x7f6b30077a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:48.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.535+0000 7f6b35e00640 1 -- 172.21.15.31:0/690435527 shutdown_connections 2024-03-21T10:41:48.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:41:48.540+0000 7f6b35e00640 1 -- 172.21.15.31:0/690435527 wait complete. 2024-03-21T10:41:48.576 INFO:tasks.cephfs.filesystem:scrub status for tag:91665fb1-b0a9-4215-b526-d4b604163f30 - {'path': '/', 'tag': '91665fb1-b0a9-4215-b526-d4b604163f30', 'options': 'recursive,force'} 2024-03-21T10:41:48.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:48 smithi031 ceph-mon[31673]: pgmap v100: 129 pgs: 129 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 182 MiB/s wr, 1.05k op/s 2024-03-21T10:41:48.939 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:48 smithi031 ceph-mon[31673]: scrub queued for path: / 2024-03-21T10:41:48.939 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:48 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:41:48.939 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:48 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:41:48.939 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:48 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:41:48.939 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:48 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:41:48.939 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:48 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:41:49.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:48 smithi070 ceph-mon[36330]: pgmap v100: 129 pgs: 129 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 182 MiB/s wr, 1.05k op/s 2024-03-21T10:41:49.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:48 smithi070 ceph-mon[36330]: scrub queued for path: / 2024-03-21T10:41:49.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:48 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:41:49.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:48 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:41:49.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:48 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:41:49.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:48 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:41:49.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:48 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:41:49.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:48 smithi040 ceph-mon[36154]: pgmap v100: 129 pgs: 129 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 182 MiB/s wr, 1.05k op/s 2024-03-21T10:41:49.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:48 smithi040 ceph-mon[36154]: scrub queued for path: / 2024-03-21T10:41:49.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:48 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:41:49.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:48 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:41:49.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:48 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:41:49.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:48 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:41:49.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:48 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:41:50.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:49 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:41:50.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:49 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:41:50.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:49 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:41:50.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:49 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:41:50.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:49 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:41:50.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:49 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:41:50.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:49 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:41:50.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:49 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:41:50.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:49 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:41:51.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:50 smithi031 ceph-mon[31673]: pgmap v101: 129 pgs: 129 active+clean; 15 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 215 MiB/s wr, 1.21k op/s 2024-03-21T10:41:51.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:50 smithi070 ceph-mon[36330]: pgmap v101: 129 pgs: 129 active+clean; 15 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 215 MiB/s wr, 1.21k op/s 2024-03-21T10:41:51.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:50 smithi040 ceph-mon[36154]: pgmap v101: 129 pgs: 129 active+clean; 15 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 215 MiB/s wr, 1.21k op/s 2024-03-21T10:41:53.232 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:52 smithi040 ceph-mon[36154]: pgmap v102: 129 pgs: 129 active+clean; 15 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 238 MiB/s wr, 1.28k op/s 2024-03-21T10:41:53.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:52 smithi031 ceph-mon[31673]: pgmap v102: 129 pgs: 129 active+clean; 15 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 238 MiB/s wr, 1.28k op/s 2024-03-21T10:41:53.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:52 smithi070 ceph-mon[36330]: pgmap v102: 129 pgs: 129 active+clean; 15 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 238 MiB/s wr, 1.28k op/s 2024-03-21T10:41:55.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:54 smithi031 ceph-mon[31673]: pgmap v103: 129 pgs: 129 active+clean; 15 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 180 MiB/s wr, 920 op/s 2024-03-21T10:41:55.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:54 smithi070 ceph-mon[36330]: pgmap v103: 129 pgs: 129 active+clean; 15 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 180 MiB/s wr, 920 op/s 2024-03-21T10:41:55.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:54 smithi040 ceph-mon[36154]: pgmap v103: 129 pgs: 129 active+clean; 15 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 180 MiB/s wr, 920 op/s 2024-03-21T10:41:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.161+0000 7f20ff200640 1 -- 172.21.15.31:0/1165297127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20f8009d40 msgr2=0x7f20f800a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.161+0000 7f20ff200640 1 --2- 172.21.15.31:0/1165297127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20f8009d40 0x7f20f800a120 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f2100050950 tx=0x7f2100097890 comp rx=0 tx=0).stop 2024-03-21T10:41:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.162+0000 7f20ff200640 1 -- 172.21.15.31:0/1165297127 shutdown_connections 2024-03-21T10:41:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.162+0000 7f20ff200640 1 --2- 172.21.15.31:0/1165297127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20f80a6bc0 0x7f20f80a8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.162+0000 7f20ff200640 1 --2- 172.21.15.31:0/1165297127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20f800a660 0x7f20f80a6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.162+0000 7f20ff200640 1 --2- 172.21.15.31:0/1165297127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20f8009d40 0x7f20f800a120 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.162+0000 7f20ff200640 1 -- 172.21.15.31:0/1165297127 >> 172.21.15.31:0/1165297127 conn(0x7f20f80939f0 msgr2=0x7f20f8095e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:56.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.162+0000 7f20ff200640 1 -- 172.21.15.31:0/1165297127 shutdown_connections 2024-03-21T10:41:56.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.162+0000 7f20ff200640 1 -- 172.21.15.31:0/1165297127 wait complete. 2024-03-21T10:41:56.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.164+0000 7f20ff200640 1 Processor -- start 2024-03-21T10:41:56.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.171+0000 7f20ff200640 1 -- start start 2024-03-21T10:41:56.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.179+0000 7f20ff200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20f8009d40 0x7f20f80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:56.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.179+0000 7f20ff200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20f800a660 0x7f20f809f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:56.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.179+0000 7f20ff200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20f80a6bc0 0x7f20f809f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:56.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.179+0000 7f20ff200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f80ac690 con 0x7f20f80a6bc0 2024-03-21T10:41:56.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.179+0000 7f20ff200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f80ac510 con 0x7f20f800a660 2024-03-21T10:41:56.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.179+0000 7f20ff200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f80ac810 con 0x7f20f8009d40 2024-03-21T10:41:56.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.179+0000 7f20fd400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20f800a660 0x7f20f809f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:56.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.179+0000 7f20fd400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20f800a660 0x7f20f809f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42880/0 (socket says 172.21.15.31:42880) 2024-03-21T10:41:56.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.179+0000 7f20fd400640 1 -- 172.21.15.31:0/3791815574 learned_addr learned my addr 172.21.15.31:0/3791815574 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:56.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.179+0000 7f20fde00640 1 --2- 172.21.15.31:0/3791815574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20f8009d40 0x7f20f80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:56.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.180+0000 7f20fd400640 1 -- 172.21.15.31:0/3791815574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20f8009d40 msgr2=0x7f20f80a6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:56.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.180+0000 7f20fd400640 1 --2- 172.21.15.31:0/3791815574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20f8009d40 0x7f20f80a6280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.180+0000 7f20fd400640 1 -- 172.21.15.31:0/3791815574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20f80a6bc0 msgr2=0x7f20f809f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:56.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.180+0000 7f20fd400640 1 --2- 172.21.15.31:0/3791815574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20f80a6bc0 0x7f20f809f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.180+0000 7f20fd400640 1 -- 172.21.15.31:0/3791815574 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f809fee0 con 0x7f20f800a660 2024-03-21T10:41:56.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.181+0000 7f20fd400640 1 --2- 172.21.15.31:0/3791815574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20f800a660 0x7f20f809f300 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f20ec00e9f0 tx=0x7f20ec00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:56.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.182+0000 7f20f6a00640 1 -- 172.21.15.31:0/3791815574 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ec00cd80 con 0x7f20f800a660 2024-03-21T10:41:56.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.183+0000 7f20f6a00640 1 -- 172.21.15.31:0/3791815574 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f20ec010910 con 0x7f20f800a660 2024-03-21T10:41:56.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.183+0000 7f20ff200640 1 -- 172.21.15.31:0/3791815574 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20f80a3da0 con 0x7f20f800a660 2024-03-21T10:41:56.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.183+0000 7f20ff200640 1 -- 172.21.15.31:0/3791815574 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20f80a42f0 con 0x7f20f800a660 2024-03-21T10:41:56.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.184+0000 7f20f6a00640 1 -- 172.21.15.31:0/3791815574 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ec0188b0 con 0x7f20f800a660 2024-03-21T10:41:56.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.208+0000 7f2107981740 1 Processor -- start 2024-03-21T10:41:56.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.208+0000 7f2107981740 1 -- start start 2024-03-21T10:41:56.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.208+0000 7f2107981740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728dda1a0 0x55a728dda580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:56.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.210+0000 7f2107981740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a728ddaac0 0x55a728de3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:56.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.210+0000 7f2107981740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a728de3890 0x55a728de5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:56.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.210+0000 7f2107981740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a728d9c3b0 con 0x55a728ddaac0 2024-03-21T10:41:56.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.210+0000 7f2107981740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a728d3e830 con 0x55a728dda1a0 2024-03-21T10:41:56.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.210+0000 7f2107981740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a728ce75d0 con 0x55a728de3890 2024-03-21T10:41:56.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.210+0000 7f20fe800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728dda1a0 0x55a728dda580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:56.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.210+0000 7f20fe800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728dda1a0 0x55a728dda580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42882/0 (socket says 172.21.15.31:42882) 2024-03-21T10:41:56.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.210+0000 7f20fe800640 1 -- 172.21.15.31:0/4207821569 learned_addr learned my addr 172.21.15.31:0/4207821569 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:56.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.211+0000 7f20f6a00640 1 -- 172.21.15.31:0/3791815574 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f20ec010ac0 con 0x7f20f800a660 2024-03-21T10:41:56.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.212+0000 7f20fe800640 1 -- 172.21.15.31:0/4207821569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a728de3890 msgr2=0x55a728de5c80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:41:56.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.212+0000 7f20fe800640 1 --2- 172.21.15.31:0/4207821569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a728de3890 0x55a728de5c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.212+0000 7f20fe800640 1 -- 172.21.15.31:0/4207821569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a728ddaac0 msgr2=0x55a728de3350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:41:56.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.212+0000 7f20fe800640 1 --2- 172.21.15.31:0/4207821569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a728ddaac0 0x55a728de3350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.212+0000 7f20fe800640 1 -- 172.21.15.31:0/4207821569 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a728cff2c0 con 0x55a728dda1a0 2024-03-21T10:41:56.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.213+0000 7f20fe800640 1 --2- 172.21.15.31:0/4207821569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728dda1a0 0x55a728dda580 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f20f000b9e0 tx=0x7f20f000bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3aa865cfa0c79d98 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:56.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.215+0000 7f20eb000640 1 -- 172.21.15.31:0/4207821569 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f0016070 con 0x55a728dda1a0 2024-03-21T10:41:56.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.215+0000 7f20eb000640 1 -- 172.21.15.31:0/4207821569 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f20f0005500 con 0x55a728dda1a0 2024-03-21T10:41:56.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.215+0000 7f20eb000640 1 -- 172.21.15.31:0/4207821569 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f0010460 con 0x55a728dda1a0 2024-03-21T10:41:56.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.215+0000 7f20f6a00640 1 --2- 172.21.15.31:0/3791815574 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f20e40a3260 0x7f20e40a5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:56.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.215+0000 7f20f6a00640 1 -- 172.21.15.31:0/3791815574 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f20ec0d00e0 con 0x7f20f800a660 2024-03-21T10:41:56.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.216+0000 7f2107981740 1 -- 172.21.15.31:0/4207821569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728dda1a0 msgr2=0x55a728dda580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:56.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.216+0000 7f2107981740 1 --2- 172.21.15.31:0/4207821569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728dda1a0 0x55a728dda580 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f20f000b9e0 tx=0x7f20f000bea0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.218+0000 7f2107981740 1 -- 172.21.15.31:0/4207821569 shutdown_connections 2024-03-21T10:41:56.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.218+0000 7f2107981740 1 --2- 172.21.15.31:0/4207821569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a728de3890 0x55a728de5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.218+0000 7f2107981740 1 --2- 172.21.15.31:0/4207821569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a728ddaac0 0x55a728de3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.219+0000 7f2107981740 1 --2- 172.21.15.31:0/4207821569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728dda1a0 0x55a728dda580 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.219+0000 7f2107981740 1 -- 172.21.15.31:0/4207821569 >> 172.21.15.31:0/4207821569 conn(0x55a728dd4ac0 msgr2=0x55a728dd4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:56.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.219+0000 7f20fde00640 1 --2- 172.21.15.31:0/3791815574 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f20e40a3260 0x7f20e40a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:56.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.220+0000 7f2107981740 1 -- 172.21.15.31:0/4207821569 shutdown_connections 2024-03-21T10:41:56.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.220+0000 7f2107981740 1 -- 172.21.15.31:0/4207821569 wait complete. 2024-03-21T10:41:56.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.221+0000 7f2107981740 1 -- start start 2024-03-21T10:41:56.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.222+0000 7f2107981740 1 Processor -- start 2024-03-21T10:41:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.224+0000 7f20fde00640 1 --2- 172.21.15.31:0/3791815574 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f20e40a3260 0x7f20e40a5720 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2100066e50 tx=0x7f2100067750 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.224+0000 7f2107981740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a728dda1a0 0x55a728de08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.224+0000 7f2107981740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728ddaac0 0x55a728de0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:56.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.225+0000 7f20fde00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a728dda1a0 0x55a728de08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:56.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.225+0000 7f20fd400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728ddaac0 0x55a728de0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.225+0000 7f20fd400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728ddaac0 0x55a728de0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42884/0 (socket says 172.21.15.31:42884) 2024-03-21T10:41:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.225+0000 7f20fd400640 1 -- 172.21.15.31:0/2253516537 learned_addr learned my addr 172.21.15.31:0/2253516537 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.226+0000 7f2107981740 1 --2- 172.21.15.31:0/2253516537 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a728de3890 0x55a728de1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.226+0000 7f2107981740 1 -- 172.21.15.31:0/2253516537 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a728cfaa20 con 0x55a728de3890 2024-03-21T10:41:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.226+0000 7f2107981740 1 -- 172.21.15.31:0/2253516537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a728d3a8d0 con 0x55a728ddaac0 2024-03-21T10:41:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.226+0000 7f2107981740 1 -- 172.21.15.31:0/2253516537 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a728de64c0 con 0x55a728dda1a0 2024-03-21T10:41:56.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.228+0000 7f20fe800640 1 --2- 172.21.15.31:0/2253516537 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a728de3890 0x55a728de1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:56.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.228+0000 7f20fd400640 1 -- 172.21.15.31:0/2253516537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a728dda1a0 msgr2=0x55a728de08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:56.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.228+0000 7f20fd400640 1 --2- 172.21.15.31:0/2253516537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a728dda1a0 0x55a728de08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.228+0000 7f20fd400640 1 -- 172.21.15.31:0/2253516537 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a728de3890 msgr2=0x55a728de1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:56.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.228+0000 7f20fd400640 1 --2- 172.21.15.31:0/2253516537 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a728de3890 0x55a728de1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.228+0000 7f20fd400640 1 -- 172.21.15.31:0/2253516537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a728a53210 con 0x55a728ddaac0 2024-03-21T10:41:56.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.228+0000 7f2107981740 10 client.0 _ll_register_callbacks cb 0x55a728d371c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:56.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.229+0000 7f2107981740 1 -- 172.21.15.31:0/2253516537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f20f0005500 con 0x55a728ddaac0 2024-03-21T10:41:56.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.230+0000 7f20fd400640 1 --2- 172.21.15.31:0/2253516537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728ddaac0 0x55a728de0df0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f20ec09c0c0 tx=0x7f20ec09ba90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=934625f4f7b5ba6e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:56.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.230+0000 7f20df400640 10 client.26528 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:41:56.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.231+0000 7f20df400640 1 -- 172.21.15.31:0/2253516537 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ec0a0280 con 0x55a728ddaac0 2024-03-21T10:41:56.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.231+0000 7f20df400640 1 -- 172.21.15.31:0/2253516537 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f20ec0cdbe0 con 0x55a728ddaac0 2024-03-21T10:41:56.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.231+0000 7f20df400640 1 -- 172.21.15.31:0/2253516537 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f20ec0cde40 con 0x55a728ddaac0 2024-03-21T10:41:56.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.231+0000 7f2107981740 10 client.26528 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:56.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.231+0000 7f2107981740 1 -- 172.21.15.31:0/2253516537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({fsmap=25}) v3 -- 0x55a728b1f620 con 0x55a728ddaac0 2024-03-21T10:41:56.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.233+0000 7f20df400640 1 -- 172.21.15.31:0/2253516537 <== mon.2 v2:172.21.15.40:3300/0 4 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f20ec0bc0b0 con 0x55a728ddaac0 2024-03-21T10:41:56.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.233+0000 7f2107981740 10 client.26528 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:56.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.233+0000 7f2107981740 10 client.26528 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:41:56.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.233+0000 7f2107981740 20 client.26528 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:56.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.233+0000 7f2107981740 1 --2- 172.21.15.31:0/2253516537 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a728df36c0 0x55a728df5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:56.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.233+0000 7f2107981740 4 client.26528 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:41:56.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.233+0000 7f2107981740 1 -- 172.21.15.31:0/2253516537 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a728d4bd70 con 0x55a728df36c0 2024-03-21T10:41:56.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.234+0000 7f20fde00640 1 --2- 172.21.15.31:0/2253516537 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a728df36c0 0x55a728df5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:56.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.238+0000 7f20fde00640 1 --2- 172.21.15.31:0/2253516537 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a728df36c0 0x55a728df5ab0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6dd0d3459aa2561 server_cookie=e0e2cdd090e4e895 in_seq=0 out_seq=0 2024-03-21T10:41:56.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.238+0000 7f20df400640 10 client.26528 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:56.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.239+0000 7f20df400640 1 -- 172.21.15.31:0/2253516537 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a728d4bd70 con 0x55a728df36c0 2024-03-21T10:41:56.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.239+0000 7f20df400640 10 client.26528 handle_command_reply: tid=0 2024-03-21T10:41:56.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.240+0000 7f2107981740 1 client.26528 shutdown 2024-03-21T10:41:56.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.241+0000 7f2107981740 1 -- 172.21.15.31:0/2253516537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728ddaac0 msgr2=0x55a728de0df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:56.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.241+0000 7f2107981740 1 --2- 172.21.15.31:0/2253516537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728ddaac0 0x55a728de0df0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f20ec09c0c0 tx=0x7f20ec09ba90 comp rx=0 tx=0).stop 2024-03-21T10:41:56.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.242+0000 7f2107981740 1 -- 172.21.15.31:0/2253516537 shutdown_connections 2024-03-21T10:41:56.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.242+0000 7f2107981740 1 --2- 172.21.15.31:0/2253516537 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a728df36c0 0x55a728df5ab0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.242+0000 7f2107981740 1 --2- 172.21.15.31:0/2253516537 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a728de3890 0x55a728de1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.242+0000 7f2107981740 1 --2- 172.21.15.31:0/2253516537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728ddaac0 0x55a728de0df0 secure :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f20ec09c0c0 tx=0x7f20ec09ba90 comp rx=0 tx=0).stop 2024-03-21T10:41:56.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.242+0000 7f2107981740 1 --2- 172.21.15.31:0/2253516537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a728dda1a0 0x55a728de08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.242+0000 7f2107981740 1 -- 172.21.15.31:0/2253516537 >> 172.21.15.31:0/2253516537 conn(0x55a728dd4ac0 msgr2=0x55a728dd6810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:56.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.242+0000 7f20df400640 0 client.26528 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:56.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.243+0000 7f2107981740 1 -- 172.21.15.31:0/2253516537 shutdown_connections 2024-03-21T10:41:56.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.244+0000 7f2107981740 1 -- 172.21.15.31:0/2253516537 wait complete. 2024-03-21T10:41:56.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.244+0000 7f2107981740 20 client.26528 trim_cache size 0 max 16384 2024-03-21T10:41:56.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.268+0000 7f2107981740 1 Processor -- start 2024-03-21T10:41:56.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.268+0000 7f2107981740 1 -- start start 2024-03-21T10:41:56.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.269+0000 7f2107981740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a728dda1a0 0x55a728de1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:56.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.269+0000 7f2107981740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728de3890 0x55a728ddbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:56.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.269+0000 7f2107981740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a728dfb3a0 0x55a728ddc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:56.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.269+0000 7f2107981740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e00121d0 con 0x55a728dda1a0 2024-03-21T10:41:56.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.269+0000 7f2107981740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e000fcd0 con 0x55a728de3890 2024-03-21T10:41:56.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.269+0000 7f2107981740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e000dd70 con 0x55a728dfb3a0 2024-03-21T10:41:56.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.269+0000 7f20fe800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a728dfb3a0 0x55a728ddc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:56.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.269+0000 7f20fd400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728de3890 0x55a728ddbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:56.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.269+0000 7f20fe800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a728dfb3a0 0x55a728ddc0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:40406/0 (socket says 172.21.15.31:40406) 2024-03-21T10:41:56.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.269+0000 7f20fe800640 1 -- 172.21.15.31:0/3602705941 learned_addr learned my addr 172.21.15.31:0/3602705941 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:56.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.269+0000 7f20fde00640 1 --2- 172.21.15.31:0/3602705941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a728dda1a0 0x55a728de1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:56.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.270+0000 7f20fd400640 1 -- 172.21.15.31:0/3602705941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a728dfb3a0 msgr2=0x55a728ddc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:56.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.270+0000 7f20fd400640 1 --2- 172.21.15.31:0/3602705941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a728dfb3a0 0x55a728ddc0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.270+0000 7f20fd400640 1 -- 172.21.15.31:0/3602705941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a728dda1a0 msgr2=0x55a728de1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:56.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.270+0000 7f20fd400640 1 --2- 172.21.15.31:0/3602705941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a728dda1a0 0x55a728de1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.270+0000 7f20fd400640 1 -- 172.21.15.31:0/3602705941 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a728de1690 con 0x55a728de3890 2024-03-21T10:41:56.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.271+0000 7f20fd400640 1 --2- 172.21.15.31:0/3602705941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728de3890 0x55a728ddbbb0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x55a728de1940 tx=0x7f20ec019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d358274133a1543 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:56.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.271+0000 7f20ea600640 1 -- 172.21.15.31:0/3602705941 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ec0cdbe0 con 0x55a728de3890 2024-03-21T10:41:56.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.271+0000 7f20ea600640 1 -- 172.21.15.31:0/3602705941 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f20f0005500 con 0x55a728de3890 2024-03-21T10:41:56.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.271+0000 7f20ea600640 1 -- 172.21.15.31:0/3602705941 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ec018b30 con 0x55a728de3890 2024-03-21T10:41:56.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.272+0000 7f2107981740 1 -- 172.21.15.31:0/3602705941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728de3890 msgr2=0x55a728ddbbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:56.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.272+0000 7f2107981740 1 --2- 172.21.15.31:0/3602705941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728de3890 0x55a728ddbbb0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x55a728de1940 tx=0x7f20ec019000 comp rx=0 tx=0).stop 2024-03-21T10:41:56.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.272+0000 7f2107981740 1 -- 172.21.15.31:0/3602705941 shutdown_connections 2024-03-21T10:41:56.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.272+0000 7f2107981740 1 --2- 172.21.15.31:0/3602705941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a728dfb3a0 0x55a728ddc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.272+0000 7f2107981740 1 --2- 172.21.15.31:0/3602705941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728de3890 0x55a728ddbbb0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.272+0000 7f2107981740 1 --2- 172.21.15.31:0/3602705941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a728dda1a0 0x55a728de1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.273+0000 7f2107981740 1 -- 172.21.15.31:0/3602705941 >> 172.21.15.31:0/3602705941 conn(0x55a728dd4ac0 msgr2=0x55a728dcf6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:56.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.273+0000 7f2107981740 1 -- 172.21.15.31:0/3602705941 shutdown_connections 2024-03-21T10:41:56.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.273+0000 7f2107981740 1 -- 172.21.15.31:0/3602705941 wait complete. 2024-03-21T10:41:56.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.273+0000 7f2107981740 1 -- start start 2024-03-21T10:41:56.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.274+0000 7f2107981740 1 Processor -- start 2024-03-21T10:41:56.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.274+0000 7f2107981740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a728dda1a0 0x55a728ddd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:56.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.274+0000 7f2107981740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728de3890 0x55a728dddb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:56.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.274+0000 7f2107981740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a728dfb3a0 0x55a728e4df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:56.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.274+0000 7f2107981740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e000e370 con 0x55a728dda1a0 2024-03-21T10:41:56.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.274+0000 7f2107981740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e000f670 con 0x55a728de3890 2024-03-21T10:41:56.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.274+0000 7f2107981740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e000e9b0 con 0x55a728dfb3a0 2024-03-21T10:41:56.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.274+0000 7f2107981740 10 client.0 _ll_register_callbacks cb 0x7f20ec09bf00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:41:56.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.275+0000 7f20fd400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728de3890 0x55a728dddb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:56.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.275+0000 7f20fd400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728de3890 0x55a728dddb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42902/0 (socket says 172.21.15.31:42902) 2024-03-21T10:41:56.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.275+0000 7f20fd400640 1 -- 172.21.15.31:0/2220558475 learned_addr learned my addr 172.21.15.31:0/2220558475 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:41:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.275+0000 7f20fd400640 1 -- 172.21.15.31:0/2220558475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a728dfb3a0 msgr2=0x55a728e4df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.275+0000 7f20fd400640 1 --2- 172.21.15.31:0/2220558475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a728dfb3a0 0x55a728e4df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.275+0000 7f20fd400640 1 -- 172.21.15.31:0/2220558475 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a728dda1a0 msgr2=0x55a728ddd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:41:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.275+0000 7f20fd400640 1 --2- 172.21.15.31:0/2220558475 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a728dda1a0 0x55a728ddd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.275+0000 7f20fd400640 1 -- 172.21.15.31:0/2220558475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a728e4e4a0 con 0x55a728de3890 2024-03-21T10:41:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.275+0000 7f20fd400640 1 --2- 172.21.15.31:0/2220558475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728de3890 0x55a728dddb40 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f20ec00e9c0 tx=0x7f20ec09bbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=990df671e327fb7e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:41:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.275+0000 7f20df400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:41:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.276+0000 7f20df400640 1 -- 172.21.15.31:0/2220558475 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ec09b580 con 0x55a728de3890 2024-03-21T10:41:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.276+0000 7f20df400640 1 -- 172.21.15.31:0/2220558475 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f20ec09b7d0 con 0x55a728de3890 2024-03-21T10:41:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.276+0000 7f2107981740 1 -- 172.21.15.31:0/2220558475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f20f0005500 con 0x55a728de3890 2024-03-21T10:41:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.276+0000 7f20df400640 1 -- 172.21.15.31:0/2220558475 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f20ec0a0280 con 0x55a728de3890 2024-03-21T10:41:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.276+0000 7f2107981740 10 client.26534 fetch_fsmap learned FSMap version 25 2024-03-21T10:41:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.276+0000 7f2107981740 1 -- 172.21.15.31:0/2220558475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a728e4f750 con 0x55a728de3890 2024-03-21T10:41:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.276+0000 7f20df400640 1 -- 172.21.15.31:0/2220558475 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f20f0005500 con 0x55a728de3890 2024-03-21T10:41:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.277+0000 7f20df400640 1 -- 172.21.15.31:0/2220558475 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f20ec0a1460 con 0x55a728de3890 2024-03-21T10:41:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.277+0000 7f2107981740 10 client.26534 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:41:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.277+0000 7f2107981740 10 client.26534 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:41:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.277+0000 7f2107981740 20 client.26534 populate_metadata read hostname 'smithi031' 2024-03-21T10:41:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.277+0000 7f2107981740 1 --2- 172.21.15.31:0/2220558475 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a728e4fd40 0x55a728e52160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:41:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.277+0000 7f2107981740 4 client.26534 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:41:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.277+0000 7f2107981740 1 -- 172.21.15.31:0/2220558475 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a728d460c0 con 0x55a728e4fd40 2024-03-21T10:41:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.277+0000 7f20fde00640 1 --2- 172.21.15.31:0/2220558475 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a728e4fd40 0x55a728e52160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:41:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.278+0000 7f20df400640 10 client.26534 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.278+0000 7f20fde00640 1 --2- 172.21.15.31:0/2220558475 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a728e4fd40 0x55a728e52160 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=675f0fbc8689b444 server_cookie=d07310b1a2d10e1d in_seq=0 out_seq=0 2024-03-21T10:41:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.298+0000 7f20df400640 1 -- 172.21.15.31:0/2220558475 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a728d460c0 con 0x55a728e4fd40 2024-03-21T10:41:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.299+0000 7f20df400640 10 client.26534 handle_command_reply: tid=0 2024-03-21T10:41:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.299+0000 7f2107981740 1 client.26534 shutdown 2024-03-21T10:41:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.299+0000 7f2107981740 1 -- 172.21.15.31:0/2220558475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728de3890 msgr2=0x55a728dddb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.299+0000 7f2107981740 1 --2- 172.21.15.31:0/2220558475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728de3890 0x55a728dddb40 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f20ec00e9c0 tx=0x7f20ec09bbc0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.299+0000 7f2107981740 1 -- 172.21.15.31:0/2220558475 shutdown_connections 2024-03-21T10:41:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.299+0000 7f2107981740 1 --2- 172.21.15.31:0/2220558475 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a728e4fd40 0x55a728e52160 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.300+0000 7f20df400640 0 client.26534 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:41:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.300+0000 7f2107981740 1 --2- 172.21.15.31:0/2220558475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a728dfb3a0 0x55a728e4df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.300+0000 7f2107981740 1 --2- 172.21.15.31:0/2220558475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a728de3890 0x55a728dddb40 secure :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f20ec00e9c0 tx=0x7f20ec09bbc0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.300+0000 7f2107981740 1 --2- 172.21.15.31:0/2220558475 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a728dda1a0 0x55a728ddd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.300+0000 7f2107981740 1 -- 172.21.15.31:0/2220558475 >> 172.21.15.31:0/2220558475 conn(0x55a728dd4ac0 msgr2=0x55a728ddbbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.306+0000 7f2107981740 1 -- 172.21.15.31:0/2220558475 shutdown_connections 2024-03-21T10:41:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.306+0000 7f2107981740 1 -- 172.21.15.31:0/2220558475 wait complete. 2024-03-21T10:41:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.308+0000 7f2107981740 20 client.26534 trim_cache size 0 max 16384 2024-03-21T10:41:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:41:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:41:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:41:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:41:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.311+0000 7f20ff200640 1 -- 172.21.15.31:0/3791815574 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f20e40a3260 msgr2=0x7f20e40a5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.311+0000 7f20ff200640 1 --2- 172.21.15.31:0/3791815574 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f20e40a3260 0x7f20e40a5720 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2100066e50 tx=0x7f2100067750 comp rx=0 tx=0).stop 2024-03-21T10:41:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.311+0000 7f20ff200640 1 -- 172.21.15.31:0/3791815574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20f800a660 msgr2=0x7f20f809f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:41:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.311+0000 7f20ff200640 1 --2- 172.21.15.31:0/3791815574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20f800a660 0x7f20f809f300 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f20ec00e9f0 tx=0x7f20ec00eeb0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.318+0000 7f20ff200640 1 -- 172.21.15.31:0/3791815574 shutdown_connections 2024-03-21T10:41:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.318+0000 7f20ff200640 1 --2- 172.21.15.31:0/3791815574 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f20e40a3260 0x7f20e40a5720 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.318+0000 7f20ff200640 1 --2- 172.21.15.31:0/3791815574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20f80a6bc0 0x7f20f809f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.318+0000 7f20ff200640 1 --2- 172.21.15.31:0/3791815574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20f800a660 0x7f20f809f300 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.318+0000 7f20ff200640 1 --2- 172.21.15.31:0/3791815574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20f8009d40 0x7f20f80a6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:41:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.318+0000 7f20ff200640 1 -- 172.21.15.31:0/3791815574 >> 172.21.15.31:0/3791815574 conn(0x7f20f80939f0 msgr2=0x7f20f8095de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:41:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.319+0000 7f20ff200640 1 -- 172.21.15.31:0/3791815574 shutdown_connections 2024-03-21T10:41:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:41:56.319+0000 7f20ff200640 1 -- 172.21.15.31:0/3791815574 wait complete. 2024-03-21T10:41:57.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:56 smithi031 ceph-mon[31673]: pgmap v104: 129 pgs: 129 active+clean; 16 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 260 MiB/s wr, 1.29k op/s 2024-03-21T10:41:57.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:56 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:41:57.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:56 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:41:57.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:56 smithi070 ceph-mon[36330]: pgmap v104: 129 pgs: 129 active+clean; 16 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 260 MiB/s wr, 1.29k op/s 2024-03-21T10:41:57.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:56 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:41:57.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:56 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:41:57.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:56 smithi040 ceph-mon[36154]: pgmap v104: 129 pgs: 129 active+clean; 16 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 260 MiB/s wr, 1.29k op/s 2024-03-21T10:41:57.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:56 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:41:57.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:56 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:41:59.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:41:59 smithi031 ceph-mon[31673]: pgmap v105: 129 pgs: 129 active+clean; 16 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 194 MiB/s wr, 946 op/s 2024-03-21T10:41:59.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:41:59 smithi070 ceph-mon[36330]: pgmap v105: 129 pgs: 129 active+clean; 16 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 194 MiB/s wr, 946 op/s 2024-03-21T10:41:59.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:41:59 smithi040 ceph-mon[36154]: pgmap v105: 129 pgs: 129 active+clean; 16 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 194 MiB/s wr, 946 op/s 2024-03-21T10:42:01.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:01 smithi031 ceph-mon[31673]: pgmap v106: 129 pgs: 129 active+clean; 16 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 221 MiB/s wr, 1.07k op/s 2024-03-21T10:42:01.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:01 smithi070 ceph-mon[36330]: pgmap v106: 129 pgs: 129 active+clean; 16 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 221 MiB/s wr, 1.07k op/s 2024-03-21T10:42:01.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:01 smithi040 ceph-mon[36154]: pgmap v106: 129 pgs: 129 active+clean; 16 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 221 MiB/s wr, 1.07k op/s 2024-03-21T10:42:03.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:03 smithi031 ceph-mon[31673]: pgmap v107: 129 pgs: 129 active+clean; 17 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 245 MiB/s wr, 1.20k op/s 2024-03-21T10:42:03.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:03 smithi070 ceph-mon[36330]: pgmap v107: 129 pgs: 129 active+clean; 17 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 245 MiB/s wr, 1.20k op/s 2024-03-21T10:42:03.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:03 smithi040 ceph-mon[36154]: pgmap v107: 129 pgs: 129 active+clean; 17 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 245 MiB/s wr, 1.20k op/s 2024-03-21T10:42:05.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:05 smithi031 ceph-mon[31673]: pgmap v108: 129 pgs: 129 active+clean; 17 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 188 MiB/s wr, 923 op/s 2024-03-21T10:42:05.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:05 smithi070 ceph-mon[36330]: pgmap v108: 129 pgs: 129 active+clean; 17 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 188 MiB/s wr, 923 op/s 2024-03-21T10:42:05.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:05 smithi040 ceph-mon[36154]: pgmap v108: 129 pgs: 129 active+clean; 17 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 188 MiB/s wr, 923 op/s 2024-03-21T10:42:07.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:07 smithi040 ceph-mon[36154]: pgmap v109: 129 pgs: 129 active+clean; 17 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 266 MiB/s wr, 1.40k op/s 2024-03-21T10:42:07.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:07 smithi031 ceph-mon[31673]: pgmap v109: 129 pgs: 129 active+clean; 17 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 266 MiB/s wr, 1.40k op/s 2024-03-21T10:42:07.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:07 smithi070 ceph-mon[36330]: pgmap v109: 129 pgs: 129 active+clean; 17 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 266 MiB/s wr, 1.40k op/s 2024-03-21T10:42:09.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:09 smithi040 ceph-mon[36154]: pgmap v110: 129 pgs: 129 active+clean; 17 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 204 MiB/s wr, 1.16k op/s 2024-03-21T10:42:09.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:09 smithi031 ceph-mon[31673]: pgmap v110: 129 pgs: 129 active+clean; 17 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 204 MiB/s wr, 1.16k op/s 2024-03-21T10:42:09.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.563+0000 7fe35ba00640 1 -- 172.21.15.31:0/2609971749 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe35410ded0 msgr2=0x7fe354110320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:09.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.563+0000 7fe35ba00640 1 --2- 172.21.15.31:0/2609971749 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe35410ded0 0x7fe354110320 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe34c009ca0 tx=0x7fe34c02c190 comp rx=0 tx=0).stop 2024-03-21T10:42:09.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.573+0000 7fe35ba00640 1 -- 172.21.15.31:0/2609971749 shutdown_connections 2024-03-21T10:42:09.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.573+0000 7fe35ba00640 1 --2- 172.21.15.31:0/2609971749 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe35410ded0 0x7fe354110320 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.573+0000 7fe35ba00640 1 --2- 172.21.15.31:0/2609971749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3540743f0 0x7fe35410d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.573+0000 7fe35ba00640 1 --2- 172.21.15.31:0/2609971749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe354073ad0 0x7fe354073eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.573+0000 7fe35ba00640 1 -- 172.21.15.31:0/2609971749 >> 172.21.15.31:0/2609971749 conn(0x7fe3540fd140 msgr2=0x7fe3540ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:09.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.573+0000 7fe35ba00640 1 -- 172.21.15.31:0/2609971749 shutdown_connections 2024-03-21T10:42:09.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.575+0000 7fe35ba00640 1 -- 172.21.15.31:0/2609971749 wait complete. 2024-03-21T10:42:09.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.576+0000 7fe35ba00640 1 Processor -- start 2024-03-21T10:42:09.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:09 smithi070 ceph-mon[36330]: pgmap v110: 129 pgs: 129 active+clean; 17 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 204 MiB/s wr, 1.16k op/s 2024-03-21T10:42:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.576+0000 7fe35ba00640 1 -- start start 2024-03-21T10:42:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.577+0000 7fe35ba00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe354073ad0 0x7fe3541015f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.577+0000 7fe35ba00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3540743f0 0x7fe354101b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.577+0000 7fe35ba00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe35410ded0 0x7fe354102070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.577+0000 7fe35ba00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe354113c70 con 0x7fe354073ad0 2024-03-21T10:42:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.577+0000 7fe35ba00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe354113af0 con 0x7fe3540743f0 2024-03-21T10:42:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.577+0000 7fe35ba00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe354113df0 con 0x7fe35410ded0 2024-03-21T10:42:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.578+0000 7fe359000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe354073ad0 0x7fe3541015f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.578+0000 7fe359a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe35410ded0 0x7fe354102070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.579+0000 7fe359a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe35410ded0 0x7fe354102070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:37488/0 (socket says 172.21.15.31:37488) 2024-03-21T10:42:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.579+0000 7fe359a00640 1 -- 172.21.15.31:0/120824799 learned_addr learned my addr 172.21.15.31:0/120824799 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.579+0000 7fe353e00640 1 --2- 172.21.15.31:0/120824799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3540743f0 0x7fe354101b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.579+0000 7fe359a00640 1 -- 172.21.15.31:0/120824799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3540743f0 msgr2=0x7fe354101b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.579+0000 7fe359a00640 1 --2- 172.21.15.31:0/120824799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3540743f0 0x7fe354101b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.579+0000 7fe359a00640 1 -- 172.21.15.31:0/120824799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe354073ad0 msgr2=0x7fe3541015f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.579+0000 7fe359a00640 1 --2- 172.21.15.31:0/120824799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe354073ad0 0x7fe3541015f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.579+0000 7fe359a00640 1 -- 172.21.15.31:0/120824799 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3541aa860 con 0x7fe35410ded0 2024-03-21T10:42:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.580+0000 7fe359a00640 1 --2- 172.21.15.31:0/120824799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe35410ded0 0x7fe354102070 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe34c009bf0 tx=0x7fe34c035e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.581+0000 7fe351600640 1 -- 172.21.15.31:0/120824799 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c039040 con 0x7fe35410ded0 2024-03-21T10:42:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.581+0000 7fe351600640 1 -- 172.21.15.31:0/120824799 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe34c044070 con 0x7fe35410ded0 2024-03-21T10:42:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.581+0000 7fe351600640 1 -- 172.21.15.31:0/120824799 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c032940 con 0x7fe35410ded0 2024-03-21T10:42:09.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.584+0000 7fe35ba00640 1 -- 172.21.15.31:0/120824799 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3541aaaf0 con 0x7fe35410ded0 2024-03-21T10:42:09.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.588+0000 7fe35ba00640 1 -- 172.21.15.31:0/120824799 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3541aaf60 con 0x7fe35410ded0 2024-03-21T10:42:09.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.597+0000 7fe35e793740 1 Processor -- start 2024-03-21T10:42:09.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.597+0000 7fe351600640 1 -- 172.21.15.31:0/120824799 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fe34c032ae0 con 0x7fe35410ded0 2024-03-21T10:42:09.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.599+0000 7fe351600640 1 --2- 172.21.15.31:0/120824799 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe3380a31d0 0x7fe3380a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:09.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.602+0000 7fe353e00640 1 --2- 172.21.15.31:0/120824799 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe3380a31d0 0x7fe3380a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:09.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.604+0000 7fe35e793740 1 -- start start 2024-03-21T10:42:09.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.605+0000 7fe35e793740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a17c8fd0 0x55a7a17c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:09.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.605+0000 7fe35e793740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7a17c98f0 0x55a7a17d2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:09.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.605+0000 7fe35e793740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7a17d26c0 0x55a7a17d4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:09.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.605+0000 7fe351600640 1 -- 172.21.15.31:0/120824799 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fe34c0bbaf0 con 0x7fe35410ded0 2024-03-21T10:42:09.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.606+0000 7fe35e793740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7a178b3b0 con 0x55a7a17c8fd0 2024-03-21T10:42:09.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.606+0000 7fe35e793740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7a172d830 con 0x55a7a17d26c0 2024-03-21T10:42:09.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.606+0000 7fe35e793740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7a16d65d0 con 0x55a7a17c98f0 2024-03-21T10:42:09.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.606+0000 7fe353e00640 1 --2- 172.21.15.31:0/120824799 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe3380a31d0 0x7fe3380a5690 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe340005e00 tx=0x7fe340005c50 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:09.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.607+0000 7fe359000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a17c8fd0 0x55a7a17c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:09.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.607+0000 7fe359a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7a17d26c0 0x55a7a17d4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:09.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.607+0000 7fe359a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7a17d26c0 0x55a7a17d4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60450/0 (socket says 172.21.15.31:60450) 2024-03-21T10:42:09.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.607+0000 7fe359a00640 1 -- 172.21.15.31:0/2769968681 learned_addr learned my addr 172.21.15.31:0/2769968681 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:09.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.607+0000 7fe353e00640 1 --2- 172.21.15.31:0/2769968681 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7a17c98f0 0x55a7a17d2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:09.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.607+0000 7fe359a00640 1 -- 172.21.15.31:0/2769968681 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7a17c98f0 msgr2=0x55a7a17d2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:09.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.607+0000 7fe359a00640 1 --2- 172.21.15.31:0/2769968681 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7a17c98f0 0x55a7a17d2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.607+0000 7fe359a00640 1 -- 172.21.15.31:0/2769968681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a17c8fd0 msgr2=0x55a7a17c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:09.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.607+0000 7fe359a00640 1 --2- 172.21.15.31:0/2769968681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a17c8fd0 0x55a7a17c93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.607+0000 7fe359a00640 1 -- 172.21.15.31:0/2769968681 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a7a16ee2c0 con 0x55a7a17d26c0 2024-03-21T10:42:09.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.607+0000 7fe359a00640 1 --2- 172.21.15.31:0/2769968681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7a17d26c0 0x55a7a17d4ab0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe34c0bd9e0 tx=0x7fe34c0c2cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c01fa5143aa24957 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:09.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.608+0000 7fe34a000640 1 -- 172.21.15.31:0/2769968681 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c0ba480 con 0x55a7a17d26c0 2024-03-21T10:42:09.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.608+0000 7fe34a000640 1 -- 172.21.15.31:0/2769968681 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe34c02fec0 con 0x55a7a17d26c0 2024-03-21T10:42:09.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.608+0000 7fe34a000640 1 -- 172.21.15.31:0/2769968681 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c0ba620 con 0x55a7a17d26c0 2024-03-21T10:42:09.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.609+0000 7fe35e793740 1 -- 172.21.15.31:0/2769968681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7a17d26c0 msgr2=0x55a7a17d4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:09.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.609+0000 7fe35e793740 1 --2- 172.21.15.31:0/2769968681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7a17d26c0 0x55a7a17d4ab0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe34c0bd9e0 tx=0x7fe34c0c2cc0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.609+0000 7fe35e793740 1 -- 172.21.15.31:0/2769968681 shutdown_connections 2024-03-21T10:42:09.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.609+0000 7fe35e793740 1 --2- 172.21.15.31:0/2769968681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7a17d26c0 0x55a7a17d4ab0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.609+0000 7fe35e793740 1 --2- 172.21.15.31:0/2769968681 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7a17c98f0 0x55a7a17d2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.609+0000 7fe35e793740 1 --2- 172.21.15.31:0/2769968681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a17c8fd0 0x55a7a17c93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.609+0000 7fe35e793740 1 -- 172.21.15.31:0/2769968681 >> 172.21.15.31:0/2769968681 conn(0x55a7a17c38f0 msgr2=0x55a7a17c3cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:09.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.610+0000 7fe35e793740 1 -- 172.21.15.31:0/2769968681 shutdown_connections 2024-03-21T10:42:09.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.610+0000 7fe35e793740 1 -- 172.21.15.31:0/2769968681 wait complete. 2024-03-21T10:42:09.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.610+0000 7fe35e793740 1 -- start start 2024-03-21T10:42:09.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.611+0000 7fe35e793740 1 Processor -- start 2024-03-21T10:42:09.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.612+0000 7fe35e793740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a17c8fd0 0x55a7a17cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:09.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.613+0000 7fe35e793740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7a17c98f0 0x55a7a17cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:09.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.613+0000 7fe35e793740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7a17d26c0 0x55a7a17d0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:09.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.613+0000 7fe35e793740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7a16e9a20 con 0x55a7a17c8fd0 2024-03-21T10:42:09.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.613+0000 7fe353e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7a17c98f0 0x55a7a17cfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:09.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.613+0000 7fe359a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7a17d26c0 0x55a7a17d0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:09.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.614+0000 7fe353e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7a17c98f0 0x55a7a17cfcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:37512/0 (socket says 172.21.15.31:37512) 2024-03-21T10:42:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.614+0000 7fe353e00640 1 -- 172.21.15.31:0/736750923 learned_addr learned my addr 172.21.15.31:0/736750923 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.614+0000 7fe35e793740 1 -- 172.21.15.31:0/736750923 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7a17298d0 con 0x55a7a17d26c0 2024-03-21T10:42:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.614+0000 7fe35e793740 1 -- 172.21.15.31:0/736750923 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7a17d50b0 con 0x55a7a17c98f0 2024-03-21T10:42:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.614+0000 7fe35e793740 10 client.0 _ll_register_callbacks cb 0x55a7a17261c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.614+0000 7fe359000640 1 --2- 172.21.15.31:0/736750923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a17c8fd0 0x55a7a17cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.615+0000 7fe353e00640 1 -- 172.21.15.31:0/736750923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7a17d26c0 msgr2=0x55a7a17d0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.615+0000 7fe353e00640 1 --2- 172.21.15.31:0/736750923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7a17d26c0 0x55a7a17d0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.615+0000 7fe353e00640 1 -- 172.21.15.31:0/736750923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a17c8fd0 msgr2=0x55a7a17cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.615+0000 7fe353e00640 1 --2- 172.21.15.31:0/736750923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a17c8fd0 0x55a7a17cf780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.615+0000 7fe353e00640 1 -- 172.21.15.31:0/736750923 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a7a1442210 con 0x55a7a17c98f0 2024-03-21T10:42:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.615+0000 7fe359a00640 1 --2- 172.21.15.31:0/736750923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7a17d26c0 0x55a7a17d0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:42:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.615+0000 7fe359000640 1 --2- 172.21.15.31:0/736750923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a17c8fd0 0x55a7a17cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:42:09.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.616+0000 7fe353e00640 1 --2- 172.21.15.31:0/736750923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7a17c98f0 0x55a7a17cfcc0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe34000ffe0 tx=0x7fe3400046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=567423deebeedabd server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:09.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.616+0000 7fe33ea00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:42:09.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.617+0000 7fe33ea00640 1 -- 172.21.15.31:0/736750923 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34001f840 con 0x55a7a17c98f0 2024-03-21T10:42:09.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.618+0000 7fe33ea00640 1 -- 172.21.15.31:0/736750923 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fe34001f9e0 con 0x55a7a17c98f0 2024-03-21T10:42:09.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.619+0000 7fe35e793740 1 -- 172.21.15.31:0/736750923 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe34c02fec0 con 0x55a7a17c98f0 2024-03-21T10:42:09.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.620+0000 7fe33ea00640 1 -- 172.21.15.31:0/736750923 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe34001fbe0 con 0x55a7a17c98f0 2024-03-21T10:42:09.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.620+0000 7fe35e793740 10 client.26488 fetch_fsmap learned FSMap version 25 2024-03-21T10:42:09.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.620+0000 7fe35e793740 1 -- 172.21.15.31:0/736750923 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a7a150e620 con 0x55a7a17c98f0 2024-03-21T10:42:09.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.621+0000 7fe33ea00640 1 -- 172.21.15.31:0/736750923 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe34c02fec0 con 0x55a7a17c98f0 2024-03-21T10:42:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.622+0000 7fe33ea00640 1 -- 172.21.15.31:0/736750923 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fe3400303d0 con 0x55a7a17c98f0 2024-03-21T10:42:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.622+0000 7fe35e793740 10 client.26488 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:42:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.622+0000 7fe35e793740 10 client.26488 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:42:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.622+0000 7fe35e793740 20 client.26488 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.622+0000 7fe35e793740 1 --2- 172.21.15.31:0/736750923 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a7a17e24a0 0x55a7a17e4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.622+0000 7fe35e793740 4 client.26488 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:42:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.622+0000 7fe35e793740 1 -- 172.21.15.31:0/736750923 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a7a173ad70 con 0x55a7a17e24a0 2024-03-21T10:42:09.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.624+0000 7fe359000640 1 --2- 172.21.15.31:0/736750923 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a7a17e24a0 0x55a7a17e4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:09.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.626+0000 7fe359000640 1 --2- 172.21.15.31:0/736750923 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a7a17e24a0 0x55a7a17e4890 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=457dcfc777b2740c server_cookie=bab33a3faebf2a in_seq=0 out_seq=0 2024-03-21T10:42:09.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.626+0000 7fe33ea00640 10 client.26488 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:09.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.627+0000 7fe33ea00640 1 -- 172.21.15.31:0/736750923 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a7a173ad70 con 0x55a7a17e24a0 2024-03-21T10:42:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.627+0000 7fe33ea00640 10 client.26488 handle_command_reply: tid=0 2024-03-21T10:42:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.628+0000 7fe35e793740 1 client.26488 shutdown 2024-03-21T10:42:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.628+0000 7fe35e793740 1 -- 172.21.15.31:0/736750923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7a17c98f0 msgr2=0x55a7a17cfcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.628+0000 7fe35e793740 1 --2- 172.21.15.31:0/736750923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7a17c98f0 0x55a7a17cfcc0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe34000ffe0 tx=0x7fe3400046c0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.629+0000 7fe35e793740 1 -- 172.21.15.31:0/736750923 shutdown_connections 2024-03-21T10:42:09.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.629+0000 7fe35e793740 1 --2- 172.21.15.31:0/736750923 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a7a17e24a0 0x55a7a17e4890 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.629+0000 7fe35e793740 1 --2- 172.21.15.31:0/736750923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7a17d26c0 0x55a7a17d0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.629+0000 7fe35e793740 1 --2- 172.21.15.31:0/736750923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7a17c98f0 0x55a7a17cfcc0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.629+0000 7fe35e793740 1 --2- 172.21.15.31:0/736750923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a17c8fd0 0x55a7a17cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.629+0000 7fe35e793740 1 -- 172.21.15.31:0/736750923 >> 172.21.15.31:0/736750923 conn(0x55a7a17c38f0 msgr2=0x55a7a17c5640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:09.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.629+0000 7fe33ea00640 0 client.26488 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:09.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.629+0000 7fe35e793740 1 -- 172.21.15.31:0/736750923 shutdown_connections 2024-03-21T10:42:09.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.629+0000 7fe35e793740 1 -- 172.21.15.31:0/736750923 wait complete. 2024-03-21T10:42:09.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.630+0000 7fe35e793740 20 client.26488 trim_cache size 0 max 16384 2024-03-21T10:42:09.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.666+0000 7fe35e793740 1 Processor -- start 2024-03-21T10:42:09.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.667+0000 7fe35e793740 1 -- start start 2024-03-21T10:42:09.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.667+0000 7fe35e793740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7a17c8fd0 0x55a7a17d1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:09.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.667+0000 7fe35e793740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a17c98f0 0x55a7a17dd730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:09.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.667+0000 7fe35e793740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7a17d26c0 0x55a7a17ddc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:09.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.667+0000 7fe35e793740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3541b74c0 con 0x55a7a17c98f0 2024-03-21T10:42:09.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.667+0000 7fe35e793740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3541b4fc0 con 0x55a7a17c8fd0 2024-03-21T10:42:09.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.667+0000 7fe35e793740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3541b3060 con 0x55a7a17d26c0 2024-03-21T10:42:09.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.668+0000 7fe359a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7a17d26c0 0x55a7a17ddc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:09.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.668+0000 7fe359a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7a17d26c0 0x55a7a17ddc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:37514/0 (socket says 172.21.15.31:37514) 2024-03-21T10:42:09.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.669+0000 7fe359a00640 1 -- 172.21.15.31:0/561923170 learned_addr learned my addr 172.21.15.31:0/561923170 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:09.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.669+0000 7fe359000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7a17c8fd0 0x55a7a17d1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:09.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.669+0000 7fe353e00640 1 --2- 172.21.15.31:0/561923170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a17c98f0 0x55a7a17dd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:09.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.670+0000 7fe359a00640 1 -- 172.21.15.31:0/561923170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7a17c8fd0 msgr2=0x55a7a17d1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:09.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.670+0000 7fe359a00640 1 --2- 172.21.15.31:0/561923170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7a17c8fd0 0x55a7a17d1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.670+0000 7fe359a00640 1 -- 172.21.15.31:0/561923170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a17c98f0 msgr2=0x55a7a17dd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.670+0000 7fe359a00640 1 --2- 172.21.15.31:0/561923170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a17c98f0 0x55a7a17dd730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.670+0000 7fe359a00640 1 -- 172.21.15.31:0/561923170 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a7a17e4ec0 con 0x55a7a17d26c0 2024-03-21T10:42:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.670+0000 7fe353e00640 1 --2- 172.21.15.31:0/561923170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a17c98f0 0x55a7a17dd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:42:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.670+0000 7fe359000640 1 --2- 172.21.15.31:0/561923170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7a17c8fd0 0x55a7a17d1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:42:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.671+0000 7fe359a00640 1 --2- 172.21.15.31:0/561923170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7a17d26c0 0x55a7a17ddc70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe34c006010 tx=0x7fe34c005cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=831a77a0f0e007ea server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.671+0000 7fe349600640 1 -- 172.21.15.31:0/561923170 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c0db070 con 0x55a7a17d26c0 2024-03-21T10:42:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.671+0000 7fe349600640 1 -- 172.21.15.31:0/561923170 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe34c003ce0 con 0x55a7a17d26c0 2024-03-21T10:42:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.672+0000 7fe349600640 1 -- 172.21.15.31:0/561923170 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c0ba7c0 con 0x55a7a17d26c0 2024-03-21T10:42:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.673+0000 7fe35e793740 1 -- 172.21.15.31:0/561923170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7a17d26c0 msgr2=0x55a7a17ddc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:09.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.675+0000 7fe35e793740 1 --2- 172.21.15.31:0/561923170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7a17d26c0 0x55a7a17ddc70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe34c006010 tx=0x7fe34c005cf0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.675+0000 7fe35e793740 1 -- 172.21.15.31:0/561923170 shutdown_connections 2024-03-21T10:42:09.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.675+0000 7fe35e793740 1 --2- 172.21.15.31:0/561923170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7a17d26c0 0x55a7a17ddc70 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.675+0000 7fe35e793740 1 --2- 172.21.15.31:0/561923170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a17c98f0 0x55a7a17dd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.675+0000 7fe35e793740 1 --2- 172.21.15.31:0/561923170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7a17c8fd0 0x55a7a17d1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.675+0000 7fe35e793740 1 -- 172.21.15.31:0/561923170 >> 172.21.15.31:0/561923170 conn(0x55a7a17c38f0 msgr2=0x55a7a17c55d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.676+0000 7fe35e793740 1 -- 172.21.15.31:0/561923170 shutdown_connections 2024-03-21T10:42:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.676+0000 7fe35e793740 1 -- 172.21.15.31:0/561923170 wait complete. 2024-03-21T10:42:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.677+0000 7fe35e793740 1 -- start start 2024-03-21T10:42:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.677+0000 7fe35e793740 1 Processor -- start 2024-03-21T10:42:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.678+0000 7fe35e793740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a17c8fd0 0x55a7a17cd390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:09.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.679+0000 7fe35e793740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7a17c98f0 0x55a7a17d7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:09.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.679+0000 7fe35e793740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7a17d26c0 0x55a7a17d7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.679+0000 7fe35e793740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3541b3660 con 0x55a7a17c8fd0 2024-03-21T10:42:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.679+0000 7fe35e793740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3541b4960 con 0x55a7a17c98f0 2024-03-21T10:42:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.679+0000 7fe35e793740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3541b3ca0 con 0x55a7a17d26c0 2024-03-21T10:42:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.679+0000 7fe359a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7a17d26c0 0x55a7a17d7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.680+0000 7fe359a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7a17d26c0 0x55a7a17d7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:37524/0 (socket says 172.21.15.31:37524) 2024-03-21T10:42:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.680+0000 7fe359a00640 1 -- 172.21.15.31:0/2348993294 learned_addr learned my addr 172.21.15.31:0/2348993294 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.680+0000 7fe35e793740 10 client.0 _ll_register_callbacks cb 0x7fe340015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.680+0000 7fe353e00640 1 --2- 172.21.15.31:0/2348993294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7a17c98f0 0x55a7a17d7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.681+0000 7fe359a00640 1 -- 172.21.15.31:0/2348993294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7a17c98f0 msgr2=0x55a7a17d7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.680+0000 7fe359000640 1 --2- 172.21.15.31:0/2348993294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a17c8fd0 0x55a7a17cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.681+0000 7fe359a00640 1 --2- 172.21.15.31:0/2348993294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7a17c98f0 0x55a7a17d7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.681+0000 7fe359a00640 1 -- 172.21.15.31:0/2348993294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a17c8fd0 msgr2=0x55a7a17cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.681+0000 7fe359a00640 1 --2- 172.21.15.31:0/2348993294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a17c8fd0 0x55a7a17cd390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.682+0000 7fe359a00640 1 -- 172.21.15.31:0/2348993294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a7a17d7e70 con 0x55a7a17d26c0 2024-03-21T10:42:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.682+0000 7fe353e00640 1 --2- 172.21.15.31:0/2348993294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7a17c98f0 0x55a7a17d7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:42:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.682+0000 7fe359000640 1 --2- 172.21.15.31:0/2348993294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a17c8fd0 0x55a7a17cd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:42:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.682+0000 7fe359a00640 1 --2- 172.21.15.31:0/2348993294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7a17d26c0 0x55a7a17d7780 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe34c0b9ff0 tx=0x7fe34c0bbf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f46f25d90882c897 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.682+0000 7fe33ea00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:42:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.683+0000 7fe33ea00640 1 -- 172.21.15.31:0/2348993294 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c0a1060 con 0x55a7a17d26c0 2024-03-21T10:42:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.684+0000 7fe33ea00640 1 -- 172.21.15.31:0/2348993294 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fe34c004b40 con 0x55a7a17d26c0 2024-03-21T10:42:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.684+0000 7fe35e793740 1 -- 172.21.15.31:0/2348993294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe34c003ce0 con 0x55a7a17d26c0 2024-03-21T10:42:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.684+0000 7fe33ea00640 1 -- 172.21.15.31:0/2348993294 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe34c0db070 con 0x55a7a17d26c0 2024-03-21T10:42:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.685+0000 7fe35e793740 10 client.26494 fetch_fsmap learned FSMap version 25 2024-03-21T10:42:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.685+0000 7fe35e793740 1 -- 172.21.15.31:0/2348993294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a7a183eb80 con 0x55a7a17d26c0 2024-03-21T10:42:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.685+0000 7fe33ea00640 1 -- 172.21.15.31:0/2348993294 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe34c003ce0 con 0x55a7a17d26c0 2024-03-21T10:42:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.685+0000 7fe33ea00640 1 -- 172.21.15.31:0/2348993294 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fe34c03c030 con 0x55a7a17d26c0 2024-03-21T10:42:09.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.686+0000 7fe35e793740 10 client.26494 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:42:09.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.686+0000 7fe35e793740 10 client.26494 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:42:09.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.686+0000 7fe35e793740 20 client.26494 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:09.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.686+0000 7fe35e793740 1 --2- 172.21.15.31:0/2348993294 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a7a183efb0 0x55a7a18413a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:09.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.686+0000 7fe35e793740 4 client.26494 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:42:09.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.686+0000 7fe35e793740 1 -- 172.21.15.31:0/2348993294 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a7a17350c0 con 0x55a7a183efb0 2024-03-21T10:42:09.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.686+0000 7fe359000640 1 --2- 172.21.15.31:0/2348993294 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a7a183efb0 0x55a7a18413a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:09.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.689+0000 7fe33ea00640 10 client.26494 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:09.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.689+0000 7fe359000640 1 --2- 172.21.15.31:0/2348993294 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a7a183efb0 0x55a7a18413a0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e52c2a85d88a584c server_cookie=6d83cced19f83c16 in_seq=0 out_seq=0 2024-03-21T10:42:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.714+0000 7fe33ea00640 1 -- 172.21.15.31:0/2348993294 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a7a17350c0 con 0x55a7a183efb0 2024-03-21T10:42:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.714+0000 7fe33ea00640 10 client.26494 handle_command_reply: tid=0 2024-03-21T10:42:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.714+0000 7fe35e793740 1 client.26494 shutdown 2024-03-21T10:42:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.715+0000 7fe35e793740 1 -- 172.21.15.31:0/2348993294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7a17d26c0 msgr2=0x55a7a17d7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.715+0000 7fe35e793740 1 --2- 172.21.15.31:0/2348993294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7a17d26c0 0x55a7a17d7780 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe34c0b9ff0 tx=0x7fe34c0bbf20 comp rx=0 tx=0).stop 2024-03-21T10:42:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.715+0000 7fe35e793740 1 -- 172.21.15.31:0/2348993294 shutdown_connections 2024-03-21T10:42:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.715+0000 7fe35e793740 1 --2- 172.21.15.31:0/2348993294 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a7a183efb0 0x55a7a18413a0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.715+0000 7fe35e793740 1 --2- 172.21.15.31:0/2348993294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7a17d26c0 0x55a7a17d7780 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.715+0000 7fe35e793740 1 --2- 172.21.15.31:0/2348993294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a7a17c98f0 0x55a7a17d7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.715+0000 7fe35e793740 1 --2- 172.21.15.31:0/2348993294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a7a17c8fd0 0x55a7a17cd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.715+0000 7fe35e793740 1 -- 172.21.15.31:0/2348993294 >> 172.21.15.31:0/2348993294 conn(0x55a7a17c38f0 msgr2=0x55a7a17c20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.715+0000 7fe33ea00640 0 client.26494 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.715+0000 7fe35e793740 1 -- 172.21.15.31:0/2348993294 shutdown_connections 2024-03-21T10:42:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.715+0000 7fe35e793740 1 -- 172.21.15.31:0/2348993294 wait complete. 2024-03-21T10:42:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.716+0000 7fe35e793740 20 client.26494 trim_cache size 0 max 16384 2024-03-21T10:42:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:42:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:42:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:42:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:42:09.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.717+0000 7fe35ba00640 1 -- 172.21.15.31:0/120824799 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe3380a31d0 msgr2=0x7fe3380a5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:09.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.717+0000 7fe35ba00640 1 --2- 172.21.15.31:0/120824799 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe3380a31d0 0x7fe3380a5690 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe340005e00 tx=0x7fe340005c50 comp rx=0 tx=0).stop 2024-03-21T10:42:09.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.717+0000 7fe35ba00640 1 -- 172.21.15.31:0/120824799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe35410ded0 msgr2=0x7fe354102070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:09.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.717+0000 7fe35ba00640 1 --2- 172.21.15.31:0/120824799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe35410ded0 0x7fe354102070 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe34c009bf0 tx=0x7fe34c035e50 comp rx=0 tx=0).stop 2024-03-21T10:42:09.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.717+0000 7fe35ba00640 1 -- 172.21.15.31:0/120824799 shutdown_connections 2024-03-21T10:42:09.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.717+0000 7fe35ba00640 1 --2- 172.21.15.31:0/120824799 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe3380a31d0 0x7fe3380a5690 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.717+0000 7fe35ba00640 1 --2- 172.21.15.31:0/120824799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe35410ded0 0x7fe354102070 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.717+0000 7fe35ba00640 1 --2- 172.21.15.31:0/120824799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3540743f0 0x7fe354101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.717+0000 7fe35ba00640 1 --2- 172.21.15.31:0/120824799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe354073ad0 0x7fe3541015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:09.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.717+0000 7fe35ba00640 1 -- 172.21.15.31:0/120824799 >> 172.21.15.31:0/120824799 conn(0x7fe3540fd140 msgr2=0x7fe3540fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:09.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.717+0000 7fe35ba00640 1 -- 172.21.15.31:0/120824799 shutdown_connections 2024-03-21T10:42:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:09.718+0000 7fe35ba00640 1 -- 172.21.15.31:0/120824799 wait complete. 2024-03-21T10:42:11.375 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:11 smithi070 ceph-mon[36330]: pgmap v111: 129 pgs: 129 active+clean; 18 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 224 MiB/s wr, 1.33k op/s 2024-03-21T10:42:11.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:11 smithi040 ceph-mon[36154]: pgmap v111: 129 pgs: 129 active+clean; 18 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 224 MiB/s wr, 1.33k op/s 2024-03-21T10:42:11.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:11 smithi031 ceph-mon[31673]: pgmap v111: 129 pgs: 129 active+clean; 18 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 224 MiB/s wr, 1.33k op/s 2024-03-21T10:42:13.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:13 smithi040 ceph-mon[36154]: pgmap v112: 129 pgs: 129 active+clean; 18 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 239 MiB/s wr, 1.54k op/s 2024-03-21T10:42:13.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:13 smithi031 ceph-mon[31673]: pgmap v112: 129 pgs: 129 active+clean; 18 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 239 MiB/s wr, 1.54k op/s 2024-03-21T10:42:13.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:13 smithi070 ceph-mon[36330]: pgmap v112: 129 pgs: 129 active+clean; 18 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 239 MiB/s wr, 1.54k op/s 2024-03-21T10:42:15.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:15 smithi040 ceph-mon[36154]: pgmap v113: 129 pgs: 129 active+clean; 18 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 183 MiB/s wr, 1.25k op/s 2024-03-21T10:42:15.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:15 smithi031 ceph-mon[31673]: pgmap v113: 129 pgs: 129 active+clean; 18 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 183 MiB/s wr, 1.25k op/s 2024-03-21T10:42:15.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:15 smithi070 ceph-mon[36330]: pgmap v113: 129 pgs: 129 active+clean; 18 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 183 MiB/s wr, 1.25k op/s 2024-03-21T10:42:17.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:17 smithi040 ceph-mon[36154]: pgmap v114: 129 pgs: 129 active+clean; 18 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 217 MiB/s wr, 1.70k op/s 2024-03-21T10:42:17.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:17 smithi031 ceph-mon[31673]: pgmap v114: 129 pgs: 129 active+clean; 18 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 217 MiB/s wr, 1.70k op/s 2024-03-21T10:42:17.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:17 smithi070 ceph-mon[36330]: pgmap v114: 129 pgs: 129 active+clean; 18 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 217 MiB/s wr, 1.70k op/s 2024-03-21T10:42:18.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:18 smithi040 ceph-mon[36154]: pgmap v115: 129 pgs: 129 active+clean; 18 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 147 MiB/s wr, 1.34k op/s 2024-03-21T10:42:18.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:18 smithi031 ceph-mon[31673]: pgmap v115: 129 pgs: 129 active+clean; 18 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 147 MiB/s wr, 1.34k op/s 2024-03-21T10:42:18.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:18 smithi070 ceph-mon[36330]: pgmap v115: 129 pgs: 129 active+clean; 18 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 147 MiB/s wr, 1.34k op/s 2024-03-21T10:42:18.577 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:42:18.577 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:42:18.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.936+0000 7fbb64e00640 1 -- 172.21.15.31:0/1597202921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb580ac0f0 msgr2=0x7fbb580b3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:18.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.936+0000 7fbb64e00640 1 --2- 172.21.15.31:0/1597202921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb580ac0f0 0x7fbb580b3c50 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fbb600674e0 tx=0x7fbb6008f3d0 comp rx=0 tx=0).stop 2024-03-21T10:42:18.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.937+0000 7fbb64e00640 1 -- 172.21.15.31:0/1597202921 shutdown_connections 2024-03-21T10:42:18.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.937+0000 7fbb64e00640 1 --2- 172.21.15.31:0/1597202921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb580ac0f0 0x7fbb580b3c50 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:18.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.939+0000 7fbb64e00640 1 --2- 172.21.15.31:0/1597202921 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb580a5960 0x7fbb580ab9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:18.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.939+0000 7fbb64e00640 1 --2- 172.21.15.31:0/1597202921 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb580a4fb0 0x7fbb580a5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:18.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.939+0000 7fbb64e00640 1 -- 172.21.15.31:0/1597202921 >> 172.21.15.31:0/1597202921 conn(0x7fbb5801a630 msgr2=0x7fbb5801aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:18.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.953+0000 7fbb64e00640 1 -- 172.21.15.31:0/1597202921 shutdown_connections 2024-03-21T10:42:18.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.953+0000 7fbb64e00640 1 -- 172.21.15.31:0/1597202921 wait complete. 2024-03-21T10:42:18.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.955+0000 7fbb64e00640 1 Processor -- start 2024-03-21T10:42:18.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.955+0000 7fbb64e00640 1 -- start start 2024-03-21T10:42:18.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.956+0000 7fbb64e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb580ac0f0 0x7fbb58145610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:18.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.956+0000 7fbb64e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb58145b50 0x7fbb5813f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:18.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.956+0000 7fbb64e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb5813fc10 0x7fbb581400c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:18.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.956+0000 7fbb64e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb580a7390 con 0x7fbb580ac0f0 2024-03-21T10:42:18.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.956+0000 7fbb64e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb580a7210 con 0x7fbb5813fc10 2024-03-21T10:42:18.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.956+0000 7fbb64e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb580a7510 con 0x7fbb58145b50 2024-03-21T10:42:18.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.957+0000 7fbb5d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb58145b50 0x7fbb5813f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:18.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.957+0000 7fbb5d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb58145b50 0x7fbb5813f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60158/0 (socket says 172.21.15.31:60158) 2024-03-21T10:42:18.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.957+0000 7fbb5d400640 1 -- 172.21.15.31:0/2597414652 learned_addr learned my addr 172.21.15.31:0/2597414652 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:18.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.957+0000 7fbb5de00640 1 --2- 172.21.15.31:0/2597414652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb580ac0f0 0x7fbb58145610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:18.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.957+0000 7fbb5e800640 1 --2- 172.21.15.31:0/2597414652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb5813fc10 0x7fbb581400c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:18.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.958+0000 7fbb5d400640 1 -- 172.21.15.31:0/2597414652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb5813fc10 msgr2=0x7fbb581400c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:18.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.958+0000 7fbb5d400640 1 --2- 172.21.15.31:0/2597414652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb5813fc10 0x7fbb581400c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:18.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.958+0000 7fbb5d400640 1 -- 172.21.15.31:0/2597414652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb580ac0f0 msgr2=0x7fbb58145610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:18.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.958+0000 7fbb5d400640 1 --2- 172.21.15.31:0/2597414652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb580ac0f0 0x7fbb58145610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:18.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.958+0000 7fbb5d400640 1 -- 172.21.15.31:0/2597414652 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb5800bcb0 con 0x7fbb58145b50 2024-03-21T10:42:18.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.959+0000 7fbb5d400640 1 --2- 172.21.15.31:0/2597414652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb58145b50 0x7fbb5813f6d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c006fd0 tx=0x7fbb4c00b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:18.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.959+0000 7fbb56a00640 1 -- 172.21.15.31:0/2597414652 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c011830 con 0x7fbb58145b50 2024-03-21T10:42:18.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.959+0000 7fbb56a00640 1 -- 172.21.15.31:0/2597414652 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb4c016070 con 0x7fbb58145b50 2024-03-21T10:42:18.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.959+0000 7fbb56a00640 1 -- 172.21.15.31:0/2597414652 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c00f2e0 con 0x7fbb58145b50 2024-03-21T10:42:18.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.959+0000 7fbb64e00640 1 -- 172.21.15.31:0/2597414652 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb5800bfa0 con 0x7fbb58145b50 2024-03-21T10:42:18.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.960+0000 7fbb64e00640 1 -- 172.21.15.31:0/2597414652 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb5800c570 con 0x7fbb58145b50 2024-03-21T10:42:18.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.973+0000 7fbb56a00640 1 -- 172.21.15.31:0/2597414652 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fbb4c01a410 con 0x7fbb58145b50 2024-03-21T10:42:18.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.974+0000 7fbb67bcd740 1 Processor -- start 2024-03-21T10:42:18.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.978+0000 7fbb56a00640 1 --2- 172.21.15.31:0/2597414652 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbb440a3210 0x7fbb440a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:18.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.978+0000 7fbb56a00640 1 -- 172.21.15.31:0/2597414652 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fbb4c0ce740 con 0x7fbb58145b50 2024-03-21T10:42:18.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.979+0000 7fbb67bcd740 1 -- start start 2024-03-21T10:42:18.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.979+0000 7fbb67bcd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a4525e0 0x56497a451330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:18.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.979+0000 7fbb67bcd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56497a451900 0x56497a451d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:18.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.979+0000 7fbb67bcd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56497a45ac70 0x56497a45d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:18.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.979+0000 7fbb67bcd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56497a3ad250 con 0x56497a451900 2024-03-21T10:42:18.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.979+0000 7fbb67bcd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56497a355a30 con 0x56497a45ac70 2024-03-21T10:42:18.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.979+0000 7fbb67bcd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56497a3a92f0 con 0x56497a4525e0 2024-03-21T10:42:18.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.980+0000 7fbb5e800640 1 --2- 172.21.15.31:0/2597414652 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbb440a3210 0x7fbb440a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:18.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.980+0000 7fbb5e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56497a45ac70 0x56497a45d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:18.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.980+0000 7fbb5e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56497a45ac70 0x56497a45d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60972/0 (socket says 172.21.15.31:60972) 2024-03-21T10:42:18.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.980+0000 7fbb5e800640 1 -- 172.21.15.31:0/374349699 learned_addr learned my addr 172.21.15.31:0/374349699 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:18.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.980+0000 7fbb5de00640 1 --2- 172.21.15.31:0/374349699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a4525e0 0x56497a451330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:18.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.981+0000 7fbb5de00640 1 -- 172.21.15.31:0/374349699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56497a45ac70 msgr2=0x56497a45d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:18.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.981+0000 7fbb5de00640 1 --2- 172.21.15.31:0/374349699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56497a45ac70 0x56497a45d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:18.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.981+0000 7fbb5de00640 1 -- 172.21.15.31:0/374349699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56497a451900 msgr2=0x56497a451d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:18.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.981+0000 7fbb5de00640 1 --2- 172.21.15.31:0/374349699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56497a451900 0x56497a451d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:18.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.981+0000 7fbb5de00640 1 -- 172.21.15.31:0/374349699 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56497a340690 con 0x56497a4525e0 2024-03-21T10:42:18.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.983+0000 7fbb5e800640 1 --2- 172.21.15.31:0/374349699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56497a45ac70 0x56497a45d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:42:18.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.984+0000 7fbb5de00640 1 --2- 172.21.15.31:0/374349699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a4525e0 0x56497a451330 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fbb5801b9b0 tx=0x7fbb5000bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c129c0653388a2d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:18.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.984+0000 7fbb4b400640 1 -- 172.21.15.31:0/374349699 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb50010e50 con 0x56497a4525e0 2024-03-21T10:42:18.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.985+0000 7fbb4b400640 1 -- 172.21.15.31:0/374349699 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb50011920 con 0x56497a4525e0 2024-03-21T10:42:18.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.985+0000 7fbb4b400640 1 -- 172.21.15.31:0/374349699 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb50010e50 con 0x56497a4525e0 2024-03-21T10:42:18.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.986+0000 7fbb67bcd740 1 -- 172.21.15.31:0/374349699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a4525e0 msgr2=0x56497a451330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:18.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.986+0000 7fbb67bcd740 1 --2- 172.21.15.31:0/374349699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a4525e0 0x56497a451330 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fbb5801b9b0 tx=0x7fbb5000bf80 comp rx=0 tx=0).stop 2024-03-21T10:42:18.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.987+0000 7fbb67bcd740 1 -- 172.21.15.31:0/374349699 shutdown_connections 2024-03-21T10:42:18.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.987+0000 7fbb67bcd740 1 --2- 172.21.15.31:0/374349699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56497a45ac70 0x56497a45d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:18.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.987+0000 7fbb67bcd740 1 --2- 172.21.15.31:0/374349699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56497a451900 0x56497a451d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:18.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.987+0000 7fbb67bcd740 1 --2- 172.21.15.31:0/374349699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a4525e0 0x56497a451330 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:18.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.987+0000 7fbb67bcd740 1 -- 172.21.15.31:0/374349699 >> 172.21.15.31:0/374349699 conn(0x56497a410200 msgr2=0x56497a44c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:18.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.990+0000 7fbb67bcd740 1 -- 172.21.15.31:0/374349699 shutdown_connections 2024-03-21T10:42:18.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.992+0000 7fbb67bcd740 1 -- 172.21.15.31:0/374349699 wait complete. 2024-03-21T10:42:18.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.992+0000 7fbb67bcd740 1 -- start start 2024-03-21T10:42:18.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.993+0000 7fbb67bcd740 1 Processor -- start 2024-03-21T10:42:18.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:18.994+0000 7fbb5e800640 1 --2- 172.21.15.31:0/2597414652 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbb440a3210 0x7fbb440a56d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbb600670d0 tx=0x7fbb60050420 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:18.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.001+0000 7fbb67bcd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56497a451900 0x56497a45a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:18.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.001+0000 7fbb67bcd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56497a4525e0 0x56497a457920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:18.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.001+0000 7fbb67bcd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a45ac70 0x56497a457ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:18.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.001+0000 7fbb67bcd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56497a4437f0 con 0x56497a4525e0 2024-03-21T10:42:18.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.001+0000 7fbb67bcd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56497a368e80 con 0x56497a451900 2024-03-21T10:42:18.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.001+0000 7fbb67bcd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56497a443970 con 0x56497a45ac70 2024-03-21T10:42:18.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.001+0000 7fbb5e800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a45ac70 0x56497a457ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:18.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.001+0000 7fbb5e800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a45ac70 0x56497a457ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60164/0 (socket says 172.21.15.31:60164) 2024-03-21T10:42:18.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.001+0000 7fbb5e800640 1 -- 172.21.15.31:0/2418197179 learned_addr learned my addr 172.21.15.31:0/2418197179 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:18.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.002+0000 7fbb5de00640 1 --2- 172.21.15.31:0/2418197179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56497a451900 0x56497a45a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:19.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.002+0000 7fbb5e800640 1 -- 172.21.15.31:0/2418197179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56497a451900 msgr2=0x56497a45a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:19.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.002+0000 7fbb5e800640 1 --2- 172.21.15.31:0/2418197179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56497a451900 0x56497a45a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.002+0000 7fbb5e800640 1 -- 172.21.15.31:0/2418197179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56497a4525e0 msgr2=0x56497a457920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:19.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.002+0000 7fbb5d400640 1 --2- 172.21.15.31:0/2418197179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56497a4525e0 0x56497a457920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:19.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.002+0000 7fbb67bcd740 10 client.0 _ll_register_callbacks cb 0x56497a37b130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:19.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.003+0000 7fbb5e800640 1 --2- 172.21.15.31:0/2418197179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56497a4525e0 0x56497a457920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.003+0000 7fbb5e800640 1 -- 172.21.15.31:0/2418197179 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56497a18b7d0 con 0x56497a45ac70 2024-03-21T10:42:19.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.003+0000 7fbb67bcd740 1 -- 172.21.15.31:0/2418197179 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbb50011920 con 0x56497a45ac70 2024-03-21T10:42:19.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.004+0000 7fbb3fe00640 10 client.26506 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:42:19.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.005+0000 7fbb5e800640 1 --2- 172.21.15.31:0/2418197179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a45ac70 0x56497a457ec0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fbb600501b0 tx=0x7fbb600676b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33bd938ff7500fff server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:19.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.005+0000 7fbb3fe00640 1 -- 172.21.15.31:0/2418197179 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb600b9060 con 0x56497a45ac70 2024-03-21T10:42:19.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.008+0000 7fbb3fe00640 1 -- 172.21.15.31:0/2418197179 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fbb600ab820 con 0x56497a45ac70 2024-03-21T10:42:19.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.008+0000 7fbb3fe00640 1 -- 172.21.15.31:0/2418197179 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb6009c040 con 0x56497a45ac70 2024-03-21T10:42:19.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.008+0000 7fbb67bcd740 10 client.26506 fetch_fsmap learned FSMap version 25 2024-03-21T10:42:19.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.008+0000 7fbb67bcd740 1 -- 172.21.15.31:0/2418197179 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({fsmap=25}) v3 -- 0x56497a0c13f0 con 0x56497a45ac70 2024-03-21T10:42:19.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.009+0000 7fbb3fe00640 1 -- 172.21.15.31:0/2418197179 <== mon.1 v2:172.21.15.70:3300/0 4 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fbb600aba60 con 0x56497a45ac70 2024-03-21T10:42:19.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.009+0000 7fbb67bcd740 10 client.26506 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:42:19.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.009+0000 7fbb67bcd740 10 client.26506 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:42:19.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.009+0000 7fbb67bcd740 20 client.26506 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:19.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.009+0000 7fbb67bcd740 1 --2- 172.21.15.31:0/2418197179 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56497a473460 0x56497a475920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:19.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.009+0000 7fbb67bcd740 4 client.26506 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:42:19.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.009+0000 7fbb67bcd740 1 -- 172.21.15.31:0/2418197179 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56497a3ba790 con 0x56497a473460 2024-03-21T10:42:19.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.010+0000 7fbb5de00640 1 --2- 172.21.15.31:0/2418197179 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56497a473460 0x56497a475920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:19.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.011+0000 7fbb3fe00640 10 client.26506 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:19.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.011+0000 7fbb5de00640 1 --2- 172.21.15.31:0/2418197179 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56497a473460 0x56497a475920 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b401101e93ab73f server_cookie=45e21096e1d6dbb in_seq=0 out_seq=0 2024-03-21T10:42:19.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.012+0000 7fbb3fe00640 1 -- 172.21.15.31:0/2418197179 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56497a3ba790 con 0x56497a473460 2024-03-21T10:42:19.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.012+0000 7fbb3fe00640 10 client.26506 handle_command_reply: tid=0 2024-03-21T10:42:19.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.012+0000 7fbb67bcd740 1 client.26506 shutdown 2024-03-21T10:42:19.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.013+0000 7fbb67bcd740 1 -- 172.21.15.31:0/2418197179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a45ac70 msgr2=0x56497a457ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:19.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.014+0000 7fbb67bcd740 1 --2- 172.21.15.31:0/2418197179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a45ac70 0x56497a457ec0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fbb600501b0 tx=0x7fbb600676b0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.016+0000 7fbb67bcd740 1 -- 172.21.15.31:0/2418197179 shutdown_connections 2024-03-21T10:42:19.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.016+0000 7fbb67bcd740 1 --2- 172.21.15.31:0/2418197179 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56497a473460 0x56497a475920 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.016+0000 7fbb3fe00640 0 client.26506 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:19.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.016+0000 7fbb67bcd740 1 --2- 172.21.15.31:0/2418197179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a45ac70 0x56497a457ec0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.016+0000 7fbb67bcd740 1 --2- 172.21.15.31:0/2418197179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56497a4525e0 0x56497a457920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.016+0000 7fbb67bcd740 1 --2- 172.21.15.31:0/2418197179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56497a451900 0x56497a45a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.016+0000 7fbb67bcd740 1 -- 172.21.15.31:0/2418197179 >> 172.21.15.31:0/2418197179 conn(0x56497a410200 msgr2=0x56497a44e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:19.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.016+0000 7fbb67bcd740 1 -- 172.21.15.31:0/2418197179 shutdown_connections 2024-03-21T10:42:19.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.017+0000 7fbb67bcd740 1 -- 172.21.15.31:0/2418197179 wait complete. 2024-03-21T10:42:19.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.017+0000 7fbb67bcd740 20 client.26506 trim_cache size 0 max 16384 2024-03-21T10:42:19.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.059+0000 7fbb67bcd740 1 Processor -- start 2024-03-21T10:42:19.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.060+0000 7fbb67bcd740 1 -- start start 2024-03-21T10:42:19.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.061+0000 7fbb67bcd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56497a451900 0x56497a45a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:19.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.061+0000 7fbb67bcd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56497a4525e0 0x56497a45a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:19.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.061+0000 7fbb67bcd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a45ac70 0x56497a45c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:19.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.061+0000 7fbb67bcd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb580140b0 con 0x56497a451900 2024-03-21T10:42:19.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.061+0000 7fbb67bcd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb58011bb0 con 0x56497a4525e0 2024-03-21T10:42:19.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.061+0000 7fbb67bcd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5800fc70 con 0x56497a45ac70 2024-03-21T10:42:19.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.063+0000 7fbb5e800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a45ac70 0x56497a45c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:19.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.063+0000 7fbb5e800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a45ac70 0x56497a45c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60170/0 (socket says 172.21.15.31:60170) 2024-03-21T10:42:19.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.063+0000 7fbb5e800640 1 -- 172.21.15.31:0/3273995301 learned_addr learned my addr 172.21.15.31:0/3273995301 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:19.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.063+0000 7fbb5d400640 1 --2- 172.21.15.31:0/3273995301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56497a4525e0 0x56497a45a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:19.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.064+0000 7fbb5e800640 1 -- 172.21.15.31:0/3273995301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56497a4525e0 msgr2=0x56497a45a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:19.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.064+0000 7fbb5e800640 1 --2- 172.21.15.31:0/3273995301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56497a4525e0 0x56497a45a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.064+0000 7fbb5e800640 1 -- 172.21.15.31:0/3273995301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56497a451900 msgr2=0x56497a45a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:19.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.064+0000 7fbb5e800640 1 --2- 172.21.15.31:0/3273995301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56497a451900 0x56497a45a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.064+0000 7fbb5e800640 1 -- 172.21.15.31:0/3273995301 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56497a0810f0 con 0x56497a45ac70 2024-03-21T10:42:19.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.067+0000 7fbb5e800640 1 --2- 172.21.15.31:0/3273995301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a45ac70 0x56497a45c060 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fbb60050be0 tx=0x7fbb600ad050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f87f00dad22a2156 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:19.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.067+0000 7fbb4aa00640 1 -- 172.21.15.31:0/3273995301 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb600699f0 con 0x56497a45ac70 2024-03-21T10:42:19.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.068+0000 7fbb4aa00640 1 -- 172.21.15.31:0/3273995301 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb50011920 con 0x56497a45ac70 2024-03-21T10:42:19.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.068+0000 7fbb4aa00640 1 -- 172.21.15.31:0/3273995301 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb600a9c00 con 0x56497a45ac70 2024-03-21T10:42:19.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.070+0000 7fbb67bcd740 1 -- 172.21.15.31:0/3273995301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a45ac70 msgr2=0x56497a45c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:19.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.070+0000 7fbb67bcd740 1 --2- 172.21.15.31:0/3273995301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a45ac70 0x56497a45c060 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fbb60050be0 tx=0x7fbb600ad050 comp rx=0 tx=0).stop 2024-03-21T10:42:19.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.071+0000 7fbb67bcd740 1 -- 172.21.15.31:0/3273995301 shutdown_connections 2024-03-21T10:42:19.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.071+0000 7fbb67bcd740 1 --2- 172.21.15.31:0/3273995301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a45ac70 0x56497a45c060 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.071+0000 7fbb67bcd740 1 --2- 172.21.15.31:0/3273995301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56497a4525e0 0x56497a45a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.071+0000 7fbb67bcd740 1 --2- 172.21.15.31:0/3273995301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56497a451900 0x56497a45a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.071+0000 7fbb67bcd740 1 -- 172.21.15.31:0/3273995301 >> 172.21.15.31:0/3273995301 conn(0x56497a410200 msgr2=0x56497a456230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:19.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.072+0000 7fbb67bcd740 1 -- 172.21.15.31:0/3273995301 shutdown_connections 2024-03-21T10:42:19.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.072+0000 7fbb67bcd740 1 -- 172.21.15.31:0/3273995301 wait complete. 2024-03-21T10:42:19.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.074+0000 7fbb67bcd740 1 -- start start 2024-03-21T10:42:19.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.075+0000 7fbb67bcd740 1 Processor -- start 2024-03-21T10:42:19.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.076+0000 7fbb67bcd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a451900 0x56497a4543c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:19.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.076+0000 7fbb67bcd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56497a4525e0 0x56497a454900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:19.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.077+0000 7fbb5de00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a451900 0x56497a4543c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:19.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.077+0000 7fbb67bcd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56497a45ac70 0x56497a449c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:19.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.077+0000 7fbb67bcd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb58010270 con 0x56497a4525e0 2024-03-21T10:42:19.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.077+0000 7fbb67bcd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb58011550 con 0x56497a45ac70 2024-03-21T10:42:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.077+0000 7fbb67bcd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb58010890 con 0x56497a451900 2024-03-21T10:42:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.077+0000 7fbb5de00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a451900 0x56497a4543c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60178/0 (socket says 172.21.15.31:60178) 2024-03-21T10:42:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.077+0000 7fbb5de00640 1 -- 172.21.15.31:0/4172668022 learned_addr learned my addr 172.21.15.31:0/4172668022 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.077+0000 7fbb67bcd740 10 client.0 _ll_register_callbacks cb 0x7fbb60069940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.078+0000 7fbb5e800640 1 --2- 172.21.15.31:0/4172668022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56497a45ac70 0x56497a449c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.078+0000 7fbb5d400640 1 --2- 172.21.15.31:0/4172668022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56497a4525e0 0x56497a454900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:19.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.080+0000 7fbb5de00640 1 -- 172.21.15.31:0/4172668022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56497a45ac70 msgr2=0x56497a449c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:19.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.080+0000 7fbb5de00640 1 --2- 172.21.15.31:0/4172668022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56497a45ac70 0x56497a449c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.080+0000 7fbb5de00640 1 -- 172.21.15.31:0/4172668022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56497a4525e0 msgr2=0x56497a454900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:19.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.080+0000 7fbb5de00640 1 --2- 172.21.15.31:0/4172668022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56497a4525e0 0x56497a454900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.080+0000 7fbb5de00640 1 -- 172.21.15.31:0/4172668022 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56497a44a2b0 con 0x56497a451900 2024-03-21T10:42:19.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.080+0000 7fbb5de00640 1 --2- 172.21.15.31:0/4172668022 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a451900 0x56497a4543c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fbb50000c00 tx=0x7fbb50010f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57a2bde51c3873af server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:19.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.080+0000 7fbb3fe00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:42:19.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.081+0000 7fbb3fe00640 1 -- 172.21.15.31:0/4172668022 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb50011440 con 0x56497a451900 2024-03-21T10:42:19.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.081+0000 7fbb67bcd740 1 -- 172.21.15.31:0/4172668022 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbb50011920 con 0x56497a451900 2024-03-21T10:42:19.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.084+0000 7fbb3fe00640 1 -- 172.21.15.31:0/4172668022 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fbb500115e0 con 0x56497a451900 2024-03-21T10:42:19.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.084+0000 7fbb3fe00640 1 -- 172.21.15.31:0/4172668022 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb50037050 con 0x56497a451900 2024-03-21T10:42:19.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.088+0000 7fbb67bcd740 10 client.26512 fetch_fsmap learned FSMap version 25 2024-03-21T10:42:19.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.088+0000 7fbb67bcd740 1 -- 172.21.15.31:0/4172668022 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56497a4cf700 con 0x56497a451900 2024-03-21T10:42:19.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.088+0000 7fbb3fe00640 1 -- 172.21.15.31:0/4172668022 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb50011920 con 0x56497a451900 2024-03-21T10:42:19.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.088+0000 7fbb3fe00640 1 -- 172.21.15.31:0/4172668022 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fbb5002a780 con 0x56497a451900 2024-03-21T10:42:19.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.089+0000 7fbb67bcd740 10 client.26512 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:42:19.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.089+0000 7fbb67bcd740 10 client.26512 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:42:19.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.089+0000 7fbb67bcd740 20 client.26512 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:19.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.089+0000 7fbb67bcd740 1 --2- 172.21.15.31:0/4172668022 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56497a473460 0x56497a4d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:19.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.089+0000 7fbb67bcd740 4 client.26512 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:42:19.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.089+0000 7fbb67bcd740 1 -- 172.21.15.31:0/4172668022 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56497a3b4ae0 con 0x56497a473460 2024-03-21T10:42:19.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.090+0000 7fbb5d400640 1 --2- 172.21.15.31:0/4172668022 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56497a473460 0x56497a4d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:19.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.091+0000 7fbb5d400640 1 --2- 172.21.15.31:0/4172668022 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56497a473460 0x56497a4d1b00 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e213a488e6e322fd server_cookie=e313c046bf1d62f0 in_seq=0 out_seq=0 2024-03-21T10:42:19.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.092+0000 7fbb3fe00640 10 client.26512 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:19.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.092+0000 7fbb3fe00640 1 -- 172.21.15.31:0/4172668022 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56497a3b4ae0 con 0x56497a473460 2024-03-21T10:42:19.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.092+0000 7fbb3fe00640 10 client.26512 handle_command_reply: tid=0 2024-03-21T10:42:19.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.092+0000 7fbb67bcd740 1 client.26512 shutdown 2024-03-21T10:42:19.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.093+0000 7fbb67bcd740 1 -- 172.21.15.31:0/4172668022 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a451900 msgr2=0x56497a4543c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:19.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.093+0000 7fbb67bcd740 1 --2- 172.21.15.31:0/4172668022 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a451900 0x56497a4543c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fbb50000c00 tx=0x7fbb50010f40 comp rx=0 tx=0).stop 2024-03-21T10:42:19.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.093+0000 7fbb67bcd740 1 -- 172.21.15.31:0/4172668022 shutdown_connections 2024-03-21T10:42:19.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.093+0000 7fbb67bcd740 1 --2- 172.21.15.31:0/4172668022 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56497a473460 0x56497a4d1b00 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.093+0000 7fbb67bcd740 1 --2- 172.21.15.31:0/4172668022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56497a45ac70 0x56497a449c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.093+0000 7fbb67bcd740 1 --2- 172.21.15.31:0/4172668022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56497a4525e0 0x56497a454900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.093+0000 7fbb67bcd740 1 --2- 172.21.15.31:0/4172668022 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56497a451900 0x56497a4543c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.093+0000 7fbb67bcd740 1 -- 172.21.15.31:0/4172668022 >> 172.21.15.31:0/4172668022 conn(0x56497a410200 msgr2=0x56497a44d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:19.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.093+0000 7fbb3fe00640 0 client.26512 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:19.093 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:42:19.093 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-03-21T10:42:19.093 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-03-21T10:42:19.093 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:42:19.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.095+0000 7fbb67bcd740 1 -- 172.21.15.31:0/4172668022 shutdown_connections 2024-03-21T10:42:19.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.095+0000 7fbb67bcd740 1 -- 172.21.15.31:0/4172668022 wait complete. 2024-03-21T10:42:19.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.095+0000 7fbb67bcd740 20 client.26512 trim_cache size 0 max 16384 2024-03-21T10:42:19.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.096+0000 7fbb64e00640 1 -- 172.21.15.31:0/2597414652 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbb440a3210 msgr2=0x7fbb440a56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:19.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.096+0000 7fbb64e00640 1 --2- 172.21.15.31:0/2597414652 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbb440a3210 0x7fbb440a56d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbb600670d0 tx=0x7fbb60050420 comp rx=0 tx=0).stop 2024-03-21T10:42:19.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.096+0000 7fbb64e00640 1 -- 172.21.15.31:0/2597414652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb58145b50 msgr2=0x7fbb5813f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:19.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.096+0000 7fbb64e00640 1 --2- 172.21.15.31:0/2597414652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb58145b50 0x7fbb5813f6d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c006fd0 tx=0x7fbb4c00b8e0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.097+0000 7fbb64e00640 1 -- 172.21.15.31:0/2597414652 shutdown_connections 2024-03-21T10:42:19.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.097+0000 7fbb64e00640 1 --2- 172.21.15.31:0/2597414652 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbb440a3210 0x7fbb440a56d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.097+0000 7fbb64e00640 1 --2- 172.21.15.31:0/2597414652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb5813fc10 0x7fbb581400c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.097+0000 7fbb64e00640 1 --2- 172.21.15.31:0/2597414652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb58145b50 0x7fbb5813f6d0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.097+0000 7fbb64e00640 1 --2- 172.21.15.31:0/2597414652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb580ac0f0 0x7fbb58145610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.097+0000 7fbb64e00640 1 -- 172.21.15.31:0/2597414652 >> 172.21.15.31:0/2597414652 conn(0x7fbb5801a630 msgr2=0x7fbb580a2450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:19.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.097+0000 7fbb64e00640 1 -- 172.21.15.31:0/2597414652 shutdown_connections 2024-03-21T10:42:19.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.097+0000 7fbb64e00640 1 -- 172.21.15.31:0/2597414652 wait complete. 2024-03-21T10:42:19.139 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-21T10:42:19.140 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:42:19.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.449+0000 7f393f400640 1 -- 172.21.15.31:0/2570881862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f393810fe10 msgr2=0x7f39381086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:19.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.449+0000 7f393f400640 1 --2- 172.21.15.31:0/2570881862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f393810fe10 0x7f39381086a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f393001acb0 tx=0x7f393003d170 comp rx=0 tx=0).stop 2024-03-21T10:42:19.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.461+0000 7f393f400640 1 -- 172.21.15.31:0/2570881862 shutdown_connections 2024-03-21T10:42:19.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.461+0000 7f393f400640 1 --2- 172.21.15.31:0/2570881862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39381092c0 0x7f393811cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.461+0000 7f393f400640 1 --2- 172.21.15.31:0/2570881862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3938110280 0x7f3938108be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.461+0000 7f393f400640 1 --2- 172.21.15.31:0/2570881862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f393810fe10 0x7f39381086a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.461+0000 7f393f400640 1 -- 172.21.15.31:0/2570881862 >> 172.21.15.31:0/2570881862 conn(0x7f393806da00 msgr2=0x7f393806de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:19.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.461+0000 7f393f400640 1 -- 172.21.15.31:0/2570881862 shutdown_connections 2024-03-21T10:42:19.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.461+0000 7f393f400640 1 -- 172.21.15.31:0/2570881862 wait complete. 2024-03-21T10:42:19.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.463+0000 7f393f400640 1 Processor -- start 2024-03-21T10:42:19.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.467+0000 7f393f400640 1 -- start start 2024-03-21T10:42:19.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.468+0000 7f393f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f393810fe10 0x7f393811a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:19.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.469+0000 7f393f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3938110280 0x7f3938116690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:19.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.469+0000 7f393f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3938116bd0 0x7f3938117080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:19.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.469+0000 7f393f400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39381130a0 con 0x7f393810fe10 2024-03-21T10:42:19.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.469+0000 7f393f400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3938112f20 con 0x7f3938116bd0 2024-03-21T10:42:19.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.469+0000 7f393f400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3938113220 con 0x7f3938110280 2024-03-21T10:42:19.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.470+0000 7f3937e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3938110280 0x7f3938116690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:19.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.470+0000 7f3937e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3938110280 0x7f3938116690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60196/0 (socket says 172.21.15.31:60196) 2024-03-21T10:42:19.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.470+0000 7f3937e00640 1 -- 172.21.15.31:0/2899571957 learned_addr learned my addr 172.21.15.31:0/2899571957 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:19.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.471+0000 7f393d400640 1 --2- 172.21.15.31:0/2899571957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3938116bd0 0x7f3938117080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:19.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.471+0000 7f3937e00640 1 -- 172.21.15.31:0/2899571957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3938116bd0 msgr2=0x7f3938117080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:19.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.471+0000 7f3937e00640 1 --2- 172.21.15.31:0/2899571957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3938116bd0 0x7f3938117080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.471+0000 7f3937e00640 1 -- 172.21.15.31:0/2899571957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f393810fe10 msgr2=0x7f393811a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:19.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.471+0000 7f3937e00640 1 --2- 172.21.15.31:0/2899571957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f393810fe10 0x7f393811a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.472+0000 7f3937e00640 1 -- 172.21.15.31:0/2899571957 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39381aac50 con 0x7f3938110280 2024-03-21T10:42:19.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.472+0000 7f393d400640 1 --2- 172.21.15.31:0/2899571957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3938116bd0 0x7f3938117080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:42:19.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.476+0000 7f3937e00640 1 --2- 172.21.15.31:0/2899571957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3938110280 0x7f3938116690 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f392400b4a0 tx=0x7f392400b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:19.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.477+0000 7f3935600640 1 -- 172.21.15.31:0/2899571957 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3924014070 con 0x7f3938110280 2024-03-21T10:42:19.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.477+0000 7f3935600640 1 -- 172.21.15.31:0/2899571957 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3924010af0 con 0x7f3938110280 2024-03-21T10:42:19.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.477+0000 7f3935600640 1 -- 172.21.15.31:0/2899571957 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3924017d00 con 0x7f3938110280 2024-03-21T10:42:19.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.478+0000 7f393f400640 1 -- 172.21.15.31:0/2899571957 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39381aaf40 con 0x7f3938110280 2024-03-21T10:42:19.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.478+0000 7f393f400640 1 -- 172.21.15.31:0/2899571957 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39381ab3a0 con 0x7f3938110280 2024-03-21T10:42:19.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.480+0000 7f393f400640 1 -- 172.21.15.31:0/2899571957 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f393806ec00 con 0x7f3938110280 2024-03-21T10:42:19.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.483+0000 7f3935600640 1 -- 172.21.15.31:0/2899571957 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f3924017070 con 0x7f3938110280 2024-03-21T10:42:19.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.485+0000 7f3935600640 1 --2- 172.21.15.31:0/2899571957 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f391c0a3150 0x7f391c0a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:19.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.485+0000 7f3935600640 1 -- 172.21.15.31:0/2899571957 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f39240cee40 con 0x7f3938110280 2024-03-21T10:42:19.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.485+0000 7f393ca00640 1 --2- 172.21.15.31:0/2899571957 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f391c0a3150 0x7f391c0a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:19.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.489+0000 7f393ca00640 1 --2- 172.21.15.31:0/2899571957 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f391c0a3150 0x7f391c0a5610 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3930006010 tx=0x7f393001a990 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:19.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.490+0000 7f3935600640 1 -- 172.21.15.31:0/2899571957 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f392409b700 con 0x7f3938110280 2024-03-21T10:42:19.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.852+0000 7f393f400640 1 -- 172.21.15.31:0/2899571957 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39381094a0 con 0x7f3938110280 2024-03-21T10:42:19.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.853+0000 7f3935600640 1 -- 172.21.15.31:0/2899571957 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f392409b000 con 0x7f3938110280 2024-03-21T10:42:19.851 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:42:19.851 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:40:26.180798+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24596,24547]},"id":1}]} 2024-03-21T10:42:19.852 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 25 2024-03-21T10:42:19.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.856+0000 7f393f400640 1 -- 172.21.15.31:0/2899571957 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f391c0a3150 msgr2=0x7f391c0a5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:19.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.856+0000 7f393f400640 1 --2- 172.21.15.31:0/2899571957 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f391c0a3150 0x7f391c0a5610 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3930006010 tx=0x7f393001a990 comp rx=0 tx=0).stop 2024-03-21T10:42:19.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.857+0000 7f393f400640 1 -- 172.21.15.31:0/2899571957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3938110280 msgr2=0x7f3938116690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:19.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.857+0000 7f393f400640 1 --2- 172.21.15.31:0/2899571957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3938110280 0x7f3938116690 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f392400b4a0 tx=0x7f392400b960 comp rx=0 tx=0).stop 2024-03-21T10:42:19.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.857+0000 7f393f400640 1 -- 172.21.15.31:0/2899571957 shutdown_connections 2024-03-21T10:42:19.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.857+0000 7f393f400640 1 --2- 172.21.15.31:0/2899571957 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f391c0a3150 0x7f391c0a5610 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.857+0000 7f393f400640 1 --2- 172.21.15.31:0/2899571957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3938116bd0 0x7f3938117080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.857+0000 7f393f400640 1 --2- 172.21.15.31:0/2899571957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3938110280 0x7f3938116690 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.857+0000 7f393f400640 1 --2- 172.21.15.31:0/2899571957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f393810fe10 0x7f393811a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:19.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.857+0000 7f393f400640 1 -- 172.21.15.31:0/2899571957 >> 172.21.15.31:0/2899571957 conn(0x7f393806da00 msgr2=0x7f393810c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:19.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.857+0000 7f393f400640 1 -- 172.21.15.31:0/2899571957 shutdown_connections 2024-03-21T10:42:19.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:19.857+0000 7f393f400640 1 -- 172.21.15.31:0/2899571957 wait complete. 2024-03-21T10:42:19.880 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:42:19.880 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-21T10:42:20.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.130+0000 7fccfd000640 1 -- 172.21.15.31:0/3865469371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccf8109870 msgr2=0x7fccf8109c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:20.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.130+0000 7fccfd000640 1 --2- 172.21.15.31:0/3865469371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccf8109870 0x7fccf8109c50 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fcce8009900 tx=0x7fcce802bc30 comp rx=0 tx=0).stop 2024-03-21T10:42:20.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.131+0000 7fccfd000640 1 -- 172.21.15.31:0/3865469371 shutdown_connections 2024-03-21T10:42:20.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.131+0000 7fccfd000640 1 --2- 172.21.15.31:0/3865469371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccf8104220 0x7fccf8110730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.131+0000 7fccfd000640 1 --2- 172.21.15.31:0/3865469371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fccf8103860 0x7fccf8103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.131+0000 7fccfd000640 1 --2- 172.21.15.31:0/3865469371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccf8109870 0x7fccf8109c50 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.131+0000 7fccfd000640 1 -- 172.21.15.31:0/3865469371 >> 172.21.15.31:0/3865469371 conn(0x7fccf80ff310 msgr2=0x7fccf8101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:20.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.131+0000 7fccfd000640 1 -- 172.21.15.31:0/3865469371 shutdown_connections 2024-03-21T10:42:20.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.131+0000 7fccfd000640 1 -- 172.21.15.31:0/3865469371 wait complete. 2024-03-21T10:42:20.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.132+0000 7fccfd000640 1 Processor -- start 2024-03-21T10:42:20.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.132+0000 7fccfd000640 1 -- start start 2024-03-21T10:42:20.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.132+0000 7fccfd000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fccf8103860 0x7fccf819a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.132+0000 7fccfd000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccf8104220 0x7fccf819a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.132+0000 7fccfd000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccf8109870 0x7fccf819e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.132+0000 7fccfd000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf8113c80 con 0x7fccf8103860 2024-03-21T10:42:20.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.132+0000 7fccfd000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf8113b00 con 0x7fccf8109870 2024-03-21T10:42:20.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.132+0000 7fccfd000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf8113e00 con 0x7fccf8104220 2024-03-21T10:42:20.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.132+0000 7fccf5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fccf8103860 0x7fccf819a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.132+0000 7fccf5400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccf8104220 0x7fccf819a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.132+0000 7fccf5400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccf8104220 0x7fccf819a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60214/0 (socket says 172.21.15.31:60214) 2024-03-21T10:42:20.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.132+0000 7fccf5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fccf8103860 0x7fccf819a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51400/0 (socket says 172.21.15.31:51400) 2024-03-21T10:42:20.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.132+0000 7fccf5400640 1 -- 172.21.15.31:0/4194968020 learned_addr learned my addr 172.21.15.31:0/4194968020 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:20.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.132+0000 7fccf6800640 1 --2- 172.21.15.31:0/4194968020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccf8109870 0x7fccf819e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.133+0000 7fccf5400640 1 -- 172.21.15.31:0/4194968020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccf8109870 msgr2=0x7fccf819e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:20.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.133+0000 7fccf5400640 1 --2- 172.21.15.31:0/4194968020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccf8109870 0x7fccf819e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.133+0000 7fccf5400640 1 -- 172.21.15.31:0/4194968020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fccf8103860 msgr2=0x7fccf819a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:20.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.133+0000 7fccf5400640 1 --2- 172.21.15.31:0/4194968020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fccf8103860 0x7fccf819a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.133+0000 7fccf5400640 1 -- 172.21.15.31:0/4194968020 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf819f0f0 con 0x7fccf8104220 2024-03-21T10:42:20.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.133+0000 7fccf5400640 1 --2- 172.21.15.31:0/4194968020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccf8104220 0x7fccf819a640 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcce400e9f0 tx=0x7fcce400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:20.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.133+0000 7fcceea00640 1 -- 172.21.15.31:0/4194968020 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce400cd80 con 0x7fccf8104220 2024-03-21T10:42:20.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.133+0000 7fcceea00640 1 -- 172.21.15.31:0/4194968020 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcce4010910 con 0x7fccf8104220 2024-03-21T10:42:20.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.133+0000 7fcceea00640 1 -- 172.21.15.31:0/4194968020 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce4018990 con 0x7fccf8104220 2024-03-21T10:42:20.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.133+0000 7fccfd000640 1 -- 172.21.15.31:0/4194968020 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccf819f3e0 con 0x7fccf8104220 2024-03-21T10:42:20.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.133+0000 7fccfd000640 1 -- 172.21.15.31:0/4194968020 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccf81a6c70 con 0x7fccf8104220 2024-03-21T10:42:20.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.137+0000 7fcceea00640 1 -- 172.21.15.31:0/4194968020 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fcce4010ac0 con 0x7fccf8104220 2024-03-21T10:42:20.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.138+0000 7fccffc25740 1 Processor -- start 2024-03-21T10:42:20.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.138+0000 7fccffc25740 1 -- start start 2024-03-21T10:42:20.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.139+0000 7fccffc25740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0edc6d0 0x5620f0edcab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.139+0000 7fcceea00640 1 --2- 172.21.15.31:0/4194968020 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fccdc0a3110 0x7fccdc0a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.139+0000 7fccffc25740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620f0edd080 0x5620f0ee5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.139+0000 7fccffc25740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620f0ee5c00 0x5620f0ee8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.139+0000 7fccffc25740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620f0e402c0 con 0x5620f0edd080 2024-03-21T10:42:20.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.139+0000 7fccffc25740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620f0de8c90 con 0x5620f0ee5c00 2024-03-21T10:42:20.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.139+0000 7fccffc25740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620f0e3c360 con 0x5620f0edc6d0 2024-03-21T10:42:20.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.139+0000 7fcceea00640 1 -- 172.21.15.31:0/4194968020 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fcce40cfe10 con 0x7fccf8104220 2024-03-21T10:42:20.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.139+0000 7fccf6800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0edc6d0 0x5620f0edcab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.139+0000 7fccf6800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0edc6d0 0x5620f0edcab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60222/0 (socket says 172.21.15.31:60222) 2024-03-21T10:42:20.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.139+0000 7fccf6800640 1 -- 172.21.15.31:0/2175565634 learned_addr learned my addr 172.21.15.31:0/2175565634 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:20.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.139+0000 7fccf6800640 1 --2- 172.21.15.31:0/2175565634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620f0ee5c00 0x5620f0ee8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.139+0000 7fccf5400640 1 --2- 172.21.15.31:0/2175565634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620f0edd080 0x5620f0ee5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.139+0000 7fccf5e00640 1 --2- 172.21.15.31:0/4194968020 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fccdc0a3110 0x7fccdc0a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.139+0000 7fccf6800640 1 -- 172.21.15.31:0/2175565634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620f0ee5c00 msgr2=0x5620f0ee8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:20.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.139+0000 7fccf6800640 1 --2- 172.21.15.31:0/2175565634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620f0ee5c00 0x5620f0ee8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.139+0000 7fccf6800640 1 -- 172.21.15.31:0/2175565634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620f0edd080 msgr2=0x5620f0ee5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:20.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.139+0000 7fccf6800640 1 --2- 172.21.15.31:0/2175565634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620f0edd080 0x5620f0ee5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.139+0000 7fccf6800640 1 -- 172.21.15.31:0/2175565634 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5620f0e0d080 con 0x5620f0edc6d0 2024-03-21T10:42:20.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.139+0000 7fccf5400640 1 --2- 172.21.15.31:0/2175565634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620f0edd080 0x5620f0ee5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:42:20.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.140+0000 7fccf6800640 1 --2- 172.21.15.31:0/2175565634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0edc6d0 0x5620f0edcab0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fccf0005f40 tx=0x7fccf0005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=567d391c146d3568 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:20.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.140+0000 7fcce3400640 1 -- 172.21.15.31:0/2175565634 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf0011130 con 0x5620f0edc6d0 2024-03-21T10:42:20.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.140+0000 7fcce3400640 1 -- 172.21.15.31:0/2175565634 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fccf00112d0 con 0x5620f0edc6d0 2024-03-21T10:42:20.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.140+0000 7fcce3400640 1 -- 172.21.15.31:0/2175565634 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf000fbe0 con 0x5620f0edc6d0 2024-03-21T10:42:20.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.141+0000 7fccffc25740 1 -- 172.21.15.31:0/2175565634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0edc6d0 msgr2=0x5620f0edcab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:20.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.141+0000 7fccffc25740 1 --2- 172.21.15.31:0/2175565634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0edc6d0 0x5620f0edcab0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fccf0005f40 tx=0x7fccf0005bd0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.141+0000 7fccf5e00640 1 --2- 172.21.15.31:0/4194968020 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fccdc0a3110 0x7fccdc0a55d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcce8009230 tx=0x7fcce8005fd0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:20.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.141+0000 7fccffc25740 1 -- 172.21.15.31:0/2175565634 shutdown_connections 2024-03-21T10:42:20.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.141+0000 7fccffc25740 1 --2- 172.21.15.31:0/2175565634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620f0ee5c00 0x5620f0ee8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.141+0000 7fccffc25740 1 --2- 172.21.15.31:0/2175565634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620f0edd080 0x5620f0ee5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.141+0000 7fccffc25740 1 --2- 172.21.15.31:0/2175565634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0edc6d0 0x5620f0edcab0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.141+0000 7fccffc25740 1 -- 172.21.15.31:0/2175565634 >> 172.21.15.31:0/2175565634 conn(0x5620f0ed6da0 msgr2=0x5620f0ed7180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:20.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.141+0000 7fccffc25740 1 -- 172.21.15.31:0/2175565634 shutdown_connections 2024-03-21T10:42:20.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.141+0000 7fccffc25740 1 -- 172.21.15.31:0/2175565634 wait complete. 2024-03-21T10:42:20.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.141+0000 7fccffc25740 1 -- start start 2024-03-21T10:42:20.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.141+0000 7fccffc25740 1 Processor -- start 2024-03-21T10:42:20.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.142+0000 7fccffc25740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620f0edc6d0 0x5620f0ee5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.142+0000 7fccffc25740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0edd080 0x5620f0ee28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.142+0000 7fccffc25740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620f0ee5c00 0x5620f0ee2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.142+0000 7fccffc25740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620f0ee8b70 con 0x5620f0ee5c00 2024-03-21T10:42:20.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.142+0000 7fccffc25740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620f0dfc0d0 con 0x5620f0edc6d0 2024-03-21T10:42:20.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.142+0000 7fccffc25740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620f0ee8cf0 con 0x5620f0edd080 2024-03-21T10:42:20.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.142+0000 7fccffc25740 10 client.0 _ll_register_callbacks cb 0x5620f0e00e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:20.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.142+0000 7fccf5400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0edd080 0x5620f0ee28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.142+0000 7fccf5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620f0edc6d0 0x5620f0ee5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.142+0000 7fccf5400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0edd080 0x5620f0ee28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60226/0 (socket says 172.21.15.31:60226) 2024-03-21T10:42:20.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.142+0000 7fccf5400640 1 -- 172.21.15.31:0/1123892419 learned_addr learned my addr 172.21.15.31:0/1123892419 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:20.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.142+0000 7fccf6800640 1 --2- 172.21.15.31:0/1123892419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620f0ee5c00 0x5620f0ee2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.143+0000 7fccf5400640 1 -- 172.21.15.31:0/1123892419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620f0edc6d0 msgr2=0x5620f0ee5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:20.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.143+0000 7fccf5400640 1 --2- 172.21.15.31:0/1123892419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620f0edc6d0 0x5620f0ee5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.143+0000 7fccf5400640 1 -- 172.21.15.31:0/1123892419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620f0ee5c00 msgr2=0x5620f0ee2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:20.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.143+0000 7fccf5400640 1 --2- 172.21.15.31:0/1123892419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620f0ee5c00 0x5620f0ee2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.143+0000 7fccf5400640 1 -- 172.21.15.31:0/1123892419 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5620f0b543f0 con 0x5620f0edd080 2024-03-21T10:42:20.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.143+0000 7fccf5400640 1 --2- 172.21.15.31:0/1123892419 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0edd080 0x5620f0ee28b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fcce4009380 tx=0x7fcce40a1200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d037c0f937e0990b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:20.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.143+0000 7fccd7e00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:42:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.143+0000 7fccd7e00640 1 -- 172.21.15.31:0/1123892419 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce40cd650 con 0x5620f0edd080 2024-03-21T10:42:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.143+0000 7fccd7e00640 1 -- 172.21.15.31:0/1123892419 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fcce40babd0 con 0x5620f0edd080 2024-03-21T10:42:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.143+0000 7fccffc25740 1 -- 172.21.15.31:0/1123892419 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fccf00112d0 con 0x5620f0edd080 2024-03-21T10:42:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.143+0000 7fccd7e00640 1 -- 172.21.15.31:0/1123892419 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcce40a0ee0 con 0x5620f0edd080 2024-03-21T10:42:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.143+0000 7fccffc25740 10 client.26530 fetch_fsmap learned FSMap version 25 2024-03-21T10:42:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.143+0000 7fccffc25740 1 -- 172.21.15.31:0/1123892419 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5620f0c1e7d0 con 0x5620f0edd080 2024-03-21T10:42:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.144+0000 7fccd7e00640 1 -- 172.21.15.31:0/1123892419 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fccf00112d0 con 0x5620f0edd080 2024-03-21T10:42:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.144+0000 7fccd7e00640 1 -- 172.21.15.31:0/1123892419 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fcce40a20d0 con 0x5620f0edd080 2024-03-21T10:42:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.144+0000 7fccffc25740 10 client.26530 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:42:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.144+0000 7fccffc25740 10 client.26530 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:42:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.144+0000 7fccffc25740 20 client.26530 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.144+0000 7fccffc25740 1 --2- 172.21.15.31:0/1123892419 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5620f0ef5a20 0x5620f0ef7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.144+0000 7fccffc25740 4 client.26530 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:42:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.144+0000 7fccffc25740 1 -- 172.21.15.31:0/1123892419 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5620f0e4d800 con 0x5620f0ef5a20 2024-03-21T10:42:20.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.144+0000 7fccf5e00640 1 --2- 172.21.15.31:0/1123892419 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5620f0ef5a20 0x5620f0ef7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.145+0000 7fccf5e00640 1 --2- 172.21.15.31:0/1123892419 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5620f0ef5a20 0x5620f0ef7e10 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=96be3b5b6b5f611e server_cookie=bcc979792b691876 in_seq=0 out_seq=0 2024-03-21T10:42:20.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.145+0000 7fccd7e00640 10 client.26530 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:42:20.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.145+0000 7fccd7e00640 1 -- 172.21.15.31:0/1123892419 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5620f0e4d800 con 0x5620f0ef5a20 2024-03-21T10:42:20.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.145+0000 7fccd7e00640 10 client.26530 handle_command_reply: tid=0 2024-03-21T10:42:20.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.145+0000 7fccffc25740 1 client.26530 shutdown 2024-03-21T10:42:20.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.146+0000 7fccffc25740 1 -- 172.21.15.31:0/1123892419 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0edd080 msgr2=0x5620f0ee28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:20.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.146+0000 7fccffc25740 1 --2- 172.21.15.31:0/1123892419 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0edd080 0x5620f0ee28b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fcce4009380 tx=0x7fcce40a1200 comp rx=0 tx=0).stop 2024-03-21T10:42:20.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.146+0000 7fccffc25740 1 -- 172.21.15.31:0/1123892419 shutdown_connections 2024-03-21T10:42:20.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.146+0000 7fccffc25740 1 --2- 172.21.15.31:0/1123892419 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5620f0ef5a20 0x5620f0ef7e10 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.146+0000 7fccffc25740 1 --2- 172.21.15.31:0/1123892419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620f0ee5c00 0x5620f0ee2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.146+0000 7fccffc25740 1 --2- 172.21.15.31:0/1123892419 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0edd080 0x5620f0ee28b0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.146+0000 7fccffc25740 1 --2- 172.21.15.31:0/1123892419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620f0edc6d0 0x5620f0ee5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.146+0000 7fccffc25740 1 -- 172.21.15.31:0/1123892419 >> 172.21.15.31:0/1123892419 conn(0x5620f0ed6da0 msgr2=0x5620f0ed8af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:20.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.146+0000 7fccd7e00640 0 client.26530 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:42:20.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.146+0000 7fccffc25740 1 -- 172.21.15.31:0/1123892419 shutdown_connections 2024-03-21T10:42:20.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.146+0000 7fccffc25740 1 -- 172.21.15.31:0/1123892419 wait complete. 2024-03-21T10:42:20.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.146+0000 7fccffc25740 20 client.26530 trim_cache size 0 max 16384 2024-03-21T10:42:20.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.161+0000 7fccffc25740 1 Processor -- start 2024-03-21T10:42:20.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.161+0000 7fccffc25740 1 -- start start 2024-03-21T10:42:20.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.161+0000 7fccffc25740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620f0edc6d0 0x5620f0ee4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.161+0000 7fccffc25740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620f0edd080 0x5620f0ee5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.161+0000 7fccffc25740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0ee5c00 0x5620f0edfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.161+0000 7fccffc25740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf81b2f40 con 0x5620f0edd080 2024-03-21T10:42:20.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.161+0000 7fccffc25740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf81b0a40 con 0x5620f0edc6d0 2024-03-21T10:42:20.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.161+0000 7fccffc25740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf81aeae0 con 0x5620f0ee5c00 2024-03-21T10:42:20.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.162+0000 7fccf6800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0ee5c00 0x5620f0edfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.162+0000 7fccf6800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0ee5c00 0x5620f0edfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60236/0 (socket says 172.21.15.31:60236) 2024-03-21T10:42:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.162+0000 7fccf6800640 1 -- 172.21.15.31:0/1321657991 learned_addr learned my addr 172.21.15.31:0/1321657991 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.162+0000 7fccf5e00640 1 --2- 172.21.15.31:0/1321657991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620f0edc6d0 0x5620f0ee4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.162+0000 7fccf5400640 1 --2- 172.21.15.31:0/1321657991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620f0edd080 0x5620f0ee5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.162+0000 7fccf6800640 1 -- 172.21.15.31:0/1321657991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620f0edc6d0 msgr2=0x5620f0ee4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.162+0000 7fccf6800640 1 --2- 172.21.15.31:0/1321657991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620f0edc6d0 0x5620f0ee4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.162+0000 7fccf6800640 1 -- 172.21.15.31:0/1321657991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620f0edd080 msgr2=0x5620f0ee5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.162+0000 7fccf6800640 1 --2- 172.21.15.31:0/1321657991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620f0edd080 0x5620f0ee5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.162+0000 7fccf6800640 1 -- 172.21.15.31:0/1321657991 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5620f0ef8180 con 0x5620f0ee5c00 2024-03-21T10:42:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.162+0000 7fccf5e00640 1 --2- 172.21.15.31:0/1321657991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620f0edc6d0 0x5620f0ee4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:42:20.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.163+0000 7fccf6800640 1 --2- 172.21.15.31:0/1321657991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0ee5c00 0x5620f0edfb30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fccf00118b0 tx=0x7fccf0009e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1dcd9eec55c35616 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:20.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.163+0000 7fcce2a00640 1 -- 172.21.15.31:0/1321657991 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf0011230 con 0x5620f0ee5c00 2024-03-21T10:42:20.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.163+0000 7fcce2a00640 1 -- 172.21.15.31:0/1321657991 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fccf00113d0 con 0x5620f0ee5c00 2024-03-21T10:42:20.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.163+0000 7fcce2a00640 1 -- 172.21.15.31:0/1321657991 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf0005370 con 0x5620f0ee5c00 2024-03-21T10:42:20.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.164+0000 7fccffc25740 1 -- 172.21.15.31:0/1321657991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0ee5c00 msgr2=0x5620f0edfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:20.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.164+0000 7fccffc25740 1 --2- 172.21.15.31:0/1321657991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0ee5c00 0x5620f0edfb30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fccf00118b0 tx=0x7fccf0009e40 comp rx=0 tx=0).stop 2024-03-21T10:42:20.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.164+0000 7fccffc25740 1 -- 172.21.15.31:0/1321657991 shutdown_connections 2024-03-21T10:42:20.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.164+0000 7fccffc25740 1 --2- 172.21.15.31:0/1321657991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0ee5c00 0x5620f0edfb30 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.164+0000 7fccffc25740 1 --2- 172.21.15.31:0/1321657991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620f0edd080 0x5620f0ee5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.164+0000 7fccffc25740 1 --2- 172.21.15.31:0/1321657991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620f0edc6d0 0x5620f0ee4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.164+0000 7fccffc25740 1 -- 172.21.15.31:0/1321657991 >> 172.21.15.31:0/1321657991 conn(0x5620f0ed6da0 msgr2=0x5620f0ed5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:20.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.164+0000 7fccffc25740 1 -- 172.21.15.31:0/1321657991 shutdown_connections 2024-03-21T10:42:20.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.164+0000 7fccffc25740 1 -- 172.21.15.31:0/1321657991 wait complete. 2024-03-21T10:42:20.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.164+0000 7fccffc25740 1 -- start start 2024-03-21T10:42:20.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.165+0000 7fccffc25740 1 Processor -- start 2024-03-21T10:42:20.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.165+0000 7fccffc25740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620f0edc6d0 0x5620f0eedbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.165+0000 7fccffc25740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620f0edd080 0x5620f0eebf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.165+0000 7fccffc25740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0ee5c00 0x5620f0eec4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.165+0000 7fccffc25740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf81af0e0 con 0x5620f0edd080 2024-03-21T10:42:20.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.165+0000 7fccffc25740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf81b03e0 con 0x5620f0edc6d0 2024-03-21T10:42:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.165+0000 7fccffc25740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf81af720 con 0x5620f0ee5c00 2024-03-21T10:42:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.165+0000 7fccffc25740 10 client.0 _ll_register_callbacks cb 0x7fcce4009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.166+0000 7fccf5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620f0edc6d0 0x5620f0eedbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.166+0000 7fccf6800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0ee5c00 0x5620f0eec4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.166+0000 7fccf5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620f0edc6d0 0x5620f0eedbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:32860/0 (socket says 172.21.15.31:32860) 2024-03-21T10:42:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.166+0000 7fccf6800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0ee5c00 0x5620f0eec4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60248/0 (socket says 172.21.15.31:60248) 2024-03-21T10:42:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.166+0000 7fccf5e00640 1 -- 172.21.15.31:0/2932582211 learned_addr learned my addr 172.21.15.31:0/2932582211 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.166+0000 7fccf5400640 1 --2- 172.21.15.31:0/2932582211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620f0edd080 0x5620f0eebf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.166+0000 7fccf6800640 1 -- 172.21.15.31:0/2932582211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620f0edc6d0 msgr2=0x5620f0eedbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.166+0000 7fccf6800640 1 --2- 172.21.15.31:0/2932582211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620f0edc6d0 0x5620f0eedbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.166+0000 7fccf6800640 1 -- 172.21.15.31:0/2932582211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620f0edd080 msgr2=0x5620f0eebf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.166+0000 7fccf6800640 1 --2- 172.21.15.31:0/2932582211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620f0edd080 0x5620f0eebf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.166+0000 7fccf6800640 1 -- 172.21.15.31:0/2932582211 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5620f0eecc00 con 0x5620f0ee5c00 2024-03-21T10:42:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.166+0000 7fccf6800640 1 --2- 172.21.15.31:0/2932582211 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0ee5c00 0x5620f0eec4c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fccf00115a0 tx=0x7fccf0009720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e0e8e7d8320bf71 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.166+0000 7fccd7e00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:42:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.166+0000 7fccd7e00640 1 -- 172.21.15.31:0/2932582211 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf0003d80 con 0x5620f0ee5c00 2024-03-21T10:42:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.167+0000 7fccffc25740 1 -- 172.21.15.31:0/2932582211 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fccf00113d0 con 0x5620f0ee5c00 2024-03-21T10:42:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.167+0000 7fccd7e00640 1 -- 172.21.15.31:0/2932582211 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fccf0005110 con 0x5620f0ee5c00 2024-03-21T10:42:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.167+0000 7fccd7e00640 1 -- 172.21.15.31:0/2932582211 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fccf003c050 con 0x5620f0ee5c00 2024-03-21T10:42:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.167+0000 7fccffc25740 10 client.26536 fetch_fsmap learned FSMap version 25 2024-03-21T10:42:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.167+0000 7fccffc25740 1 -- 172.21.15.31:0/2932582211 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5620f0f521c0 con 0x5620f0ee5c00 2024-03-21T10:42:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.167+0000 7fccd7e00640 1 -- 172.21.15.31:0/2932582211 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fccf00113d0 con 0x5620f0ee5c00 2024-03-21T10:42:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.167+0000 7fccd7e00640 1 -- 172.21.15.31:0/2932582211 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fccf001d020 con 0x5620f0ee5c00 2024-03-21T10:42:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.167+0000 7fccffc25740 10 client.26536 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:42:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.167+0000 7fccffc25740 10 client.26536 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:42:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.167+0000 7fccffc25740 20 client.26536 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.167+0000 7fccffc25740 1 --2- 172.21.15.31:0/2932582211 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5620f0f52640 0x5620f0f54a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.167+0000 7fccffc25740 4 client.26536 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:42:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.167+0000 7fccffc25740 1 -- 172.21.15.31:0/2932582211 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5620f0e47b50 con 0x5620f0f52640 2024-03-21T10:42:20.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.168+0000 7fccf5e00640 1 --2- 172.21.15.31:0/2932582211 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5620f0f52640 0x5620f0f54a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.168+0000 7fccf5e00640 1 --2- 172.21.15.31:0/2932582211 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5620f0f52640 0x5620f0f54a30 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ec68f7c00d22a222 server_cookie=9bc4cf763d100714 in_seq=0 out_seq=0 2024-03-21T10:42:20.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.168+0000 7fccd7e00640 10 client.26536 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:42:20.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.168+0000 7fccd7e00640 1 -- 172.21.15.31:0/2932582211 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5620f0e47b50 con 0x5620f0f52640 2024-03-21T10:42:20.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.168+0000 7fccd7e00640 10 client.26536 handle_command_reply: tid=0 2024-03-21T10:42:20.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.168+0000 7fccffc25740 1 client.26536 shutdown 2024-03-21T10:42:20.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.169+0000 7fccffc25740 1 -- 172.21.15.31:0/2932582211 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0ee5c00 msgr2=0x5620f0eec4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:20.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.169+0000 7fccffc25740 1 --2- 172.21.15.31:0/2932582211 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0ee5c00 0x5620f0eec4c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fccf00115a0 tx=0x7fccf0009720 comp rx=0 tx=0).stop 2024-03-21T10:42:20.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.169+0000 7fccffc25740 1 -- 172.21.15.31:0/2932582211 shutdown_connections 2024-03-21T10:42:20.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.169+0000 7fccffc25740 1 --2- 172.21.15.31:0/2932582211 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5620f0f52640 0x5620f0f54a30 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.169+0000 7fccffc25740 1 --2- 172.21.15.31:0/2932582211 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f0ee5c00 0x5620f0eec4c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.169+0000 7fccffc25740 1 --2- 172.21.15.31:0/2932582211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620f0edd080 0x5620f0eebf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.169+0000 7fccd7e00640 0 client.26536 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:42:20.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.169+0000 7fccffc25740 1 --2- 172.21.15.31:0/2932582211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620f0edc6d0 0x5620f0eedbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.169+0000 7fccffc25740 1 -- 172.21.15.31:0/2932582211 >> 172.21.15.31:0/2932582211 conn(0x5620f0ed6da0 msgr2=0x5620f0ed8080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:20.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.169+0000 7fccffc25740 1 -- 172.21.15.31:0/2932582211 shutdown_connections 2024-03-21T10:42:20.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.169+0000 7fccffc25740 1 -- 172.21.15.31:0/2932582211 wait complete. 2024-03-21T10:42:20.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.169+0000 7fccffc25740 20 client.26536 trim_cache size 0 max 16384 2024-03-21T10:42:20.170 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:42:20.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.170+0000 7fccfd000640 1 -- 172.21.15.31:0/4194968020 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fccdc0a3110 msgr2=0x7fccdc0a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:20.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.170+0000 7fccfd000640 1 --2- 172.21.15.31:0/4194968020 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fccdc0a3110 0x7fccdc0a55d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcce8009230 tx=0x7fcce8005fd0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.170+0000 7fccfd000640 1 -- 172.21.15.31:0/4194968020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccf8104220 msgr2=0x7fccf819a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:20.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.170+0000 7fccfd000640 1 --2- 172.21.15.31:0/4194968020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccf8104220 0x7fccf819a640 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcce400e9f0 tx=0x7fcce400eeb0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.170+0000 7fccfd000640 1 -- 172.21.15.31:0/4194968020 shutdown_connections 2024-03-21T10:42:20.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.170+0000 7fccfd000640 1 --2- 172.21.15.31:0/4194968020 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fccdc0a3110 0x7fccdc0a55d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.170+0000 7fccfd000640 1 --2- 172.21.15.31:0/4194968020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccf8109870 0x7fccf819e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.170+0000 7fccfd000640 1 --2- 172.21.15.31:0/4194968020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fccf8104220 0x7fccf819a640 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.170+0000 7fccfd000640 1 --2- 172.21.15.31:0/4194968020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fccf8103860 0x7fccf819a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.170+0000 7fccfd000640 1 -- 172.21.15.31:0/4194968020 >> 172.21.15.31:0/4194968020 conn(0x7fccf80ff310 msgr2=0x7fccf80ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:20.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.170+0000 7fccfd000640 1 -- 172.21.15.31:0/4194968020 shutdown_connections 2024-03-21T10:42:20.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.170+0000 7fccfd000640 1 -- 172.21.15.31:0/4194968020 wait complete. 2024-03-21T10:42:20.190 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:42:20.190 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-21T10:42:20.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.655+0000 7f47f5a00640 1 -- 172.21.15.31:0/917622123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47f0103840 msgr2=0x7f47f0103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:20.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.655+0000 7f47f5a00640 1 --2- 172.21.15.31:0/917622123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47f0103840 0x7f47f0103cc0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f47dc009930 tx=0x7f47dc02bc20 comp rx=0 tx=0).stop 2024-03-21T10:42:20.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.656+0000 7f47f5a00640 1 -- 172.21.15.31:0/917622123 shutdown_connections 2024-03-21T10:42:20.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.656+0000 7f47f5a00640 1 --2- 172.21.15.31:0/917622123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47f0104200 0x7f47f0110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.656+0000 7f47f5a00640 1 --2- 172.21.15.31:0/917622123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47f0103840 0x7f47f0103cc0 secure :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f47dc009930 tx=0x7f47dc02bc20 comp rx=0 tx=0).stop 2024-03-21T10:42:20.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.656+0000 7f47f5a00640 1 --2- 172.21.15.31:0/917622123 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f0109850 0x7f47f0109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.656+0000 7f47f5a00640 1 -- 172.21.15.31:0/917622123 >> 172.21.15.31:0/917622123 conn(0x7f47f00ff2f0 msgr2=0x7f47f0101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:20.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.657+0000 7f47f5a00640 1 -- 172.21.15.31:0/917622123 shutdown_connections 2024-03-21T10:42:20.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.657+0000 7f47f5a00640 1 -- 172.21.15.31:0/917622123 wait complete. 2024-03-21T10:42:20.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.659+0000 7f47f5a00640 1 Processor -- start 2024-03-21T10:42:20.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.659+0000 7f47f5a00640 1 -- start start 2024-03-21T10:42:20.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.661+0000 7f47f5a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47f0104200 0x7f47f0075bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.661+0000 7f47f5a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f0109850 0x7f47f00760f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.661+0000 7f47f5a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47f0076630 0x7f47f01aa900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.661+0000 7f47f5a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f0113c70 con 0x7f47f0109850 2024-03-21T10:42:20.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.661+0000 7f47f5a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f0113af0 con 0x7f47f0076630 2024-03-21T10:42:20.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.661+0000 7f47f5a00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f0113df0 con 0x7f47f0104200 2024-03-21T10:42:20.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.662+0000 7f47eea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47f0104200 0x7f47f0075bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.662+0000 7f47eea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47f0104200 0x7f47f0075bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60268/0 (socket says 172.21.15.31:60268) 2024-03-21T10:42:20.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.662+0000 7f47eea00640 1 -- 172.21.15.31:0/999273835 learned_addr learned my addr 172.21.15.31:0/999273835 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:20.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.662+0000 7f47ee000640 1 --2- 172.21.15.31:0/999273835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f0109850 0x7f47f00760f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.662+0000 7f47ef400640 1 --2- 172.21.15.31:0/999273835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47f0076630 0x7f47f01aa900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.662+0000 7f47eea00640 1 -- 172.21.15.31:0/999273835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47f0076630 msgr2=0x7f47f01aa900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:20.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.662+0000 7f47eea00640 1 --2- 172.21.15.31:0/999273835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47f0076630 0x7f47f01aa900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.662+0000 7f47eea00640 1 -- 172.21.15.31:0/999273835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f0109850 msgr2=0x7f47f00760f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:20.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.662+0000 7f47eea00640 1 --2- 172.21.15.31:0/999273835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f0109850 0x7f47f00760f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.662+0000 7f47eea00640 1 -- 172.21.15.31:0/999273835 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f01aae40 con 0x7f47f0104200 2024-03-21T10:42:20.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.662+0000 7f47ee000640 1 --2- 172.21.15.31:0/999273835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f0109850 0x7f47f00760f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:42:20.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.663+0000 7f47eea00640 1 --2- 172.21.15.31:0/999273835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47f0104200 0x7f47f0075bb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f47e000d950 tx=0x7f47e000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:20.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.663+0000 7f47e7400640 1 -- 172.21.15.31:0/999273835 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e0017070 con 0x7f47f0104200 2024-03-21T10:42:20.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.663+0000 7f47e7400640 1 -- 172.21.15.31:0/999273835 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47e0009c30 con 0x7f47f0104200 2024-03-21T10:42:20.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.663+0000 7f47e7400640 1 -- 172.21.15.31:0/999273835 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e000f6b0 con 0x7f47f0104200 2024-03-21T10:42:20.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.666+0000 7f47f5a00640 1 -- 172.21.15.31:0/999273835 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f006b120 con 0x7f47f0104200 2024-03-21T10:42:20.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.666+0000 7f47f5a00640 1 -- 172.21.15.31:0/999273835 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f006b5f0 con 0x7f47f0104200 2024-03-21T10:42:20.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.670+0000 7f47e7400640 1 -- 172.21.15.31:0/999273835 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f47e00109d0 con 0x7f47f0104200 2024-03-21T10:42:20.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.673+0000 7f47e7400640 1 --2- 172.21.15.31:0/999273835 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f47d40a3110 0x7f47d40a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.673+0000 7f47e7400640 1 -- 172.21.15.31:0/999273835 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f47e00cf7a0 con 0x7f47f0104200 2024-03-21T10:42:20.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.676+0000 7f47ee000640 1 --2- 172.21.15.31:0/999273835 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f47d40a3110 0x7f47d40a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.676+0000 7f47f87e2740 1 Processor -- start 2024-03-21T10:42:20.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.677+0000 7f47f87e2740 1 -- start start 2024-03-21T10:42:20.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.677+0000 7f47f87e2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55713ce576d0 0x55713ce57ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.677+0000 7f47f87e2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55713ce58080 0x55713ce60630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.677+0000 7f47f87e2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713ce60c00 0x55713ce63050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.677+0000 7f47f87e2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55713cdbb2c0 con 0x55713ce576d0 2024-03-21T10:42:20.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.677+0000 7f47f87e2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55713cd63c90 con 0x55713ce58080 2024-03-21T10:42:20.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.677+0000 7f47f87e2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55713cdb7360 con 0x55713ce60c00 2024-03-21T10:42:20.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.677+0000 7f47eea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55713ce576d0 0x55713ce57ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.678+0000 7f47eea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55713ce576d0 0x55713ce57ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51470/0 (socket says 172.21.15.31:51470) 2024-03-21T10:42:20.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.678+0000 7f47eea00640 1 -- 172.21.15.31:0/2587634642 learned_addr learned my addr 172.21.15.31:0/2587634642 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:20.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.678+0000 7f47eea00640 1 -- 172.21.15.31:0/2587634642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713ce60c00 msgr2=0x55713ce63050 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:42:20.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.678+0000 7f47eea00640 1 --2- 172.21.15.31:0/2587634642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713ce60c00 0x55713ce63050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.678+0000 7f47eea00640 1 -- 172.21.15.31:0/2587634642 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55713ce58080 msgr2=0x55713ce60630 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:42:20.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.678+0000 7f47eea00640 1 --2- 172.21.15.31:0/2587634642 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55713ce58080 0x55713ce60630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.678+0000 7f47eea00640 1 -- 172.21.15.31:0/2587634642 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55713cd88080 con 0x55713ce576d0 2024-03-21T10:42:20.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.681+0000 7f47eea00640 1 --2- 172.21.15.31:0/2587634642 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55713ce576d0 0x55713ce57ab0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f47e0098220 tx=0x7f47e0009de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e45467cd31a107d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:20.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.681+0000 7f47dbe00640 1 -- 172.21.15.31:0/2587634642 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e00b3020 con 0x55713ce576d0 2024-03-21T10:42:20.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.683+0000 7f47dbe00640 1 -- 172.21.15.31:0/2587634642 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47e009f9a0 con 0x55713ce576d0 2024-03-21T10:42:20.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.684+0000 7f47f87e2740 1 -- 172.21.15.31:0/2587634642 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55713ce576d0 msgr2=0x55713ce57ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:20.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.685+0000 7f47dbe00640 1 -- 172.21.15.31:0/2587634642 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e00cf420 con 0x55713ce576d0 2024-03-21T10:42:20.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.685+0000 7f47f87e2740 1 --2- 172.21.15.31:0/2587634642 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55713ce576d0 0x55713ce57ab0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f47e0098220 tx=0x7f47e0009de0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.686+0000 7f47f87e2740 1 -- 172.21.15.31:0/2587634642 shutdown_connections 2024-03-21T10:42:20.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.686+0000 7f47f87e2740 1 --2- 172.21.15.31:0/2587634642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713ce60c00 0x55713ce63050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.686+0000 7f47ee000640 1 --2- 172.21.15.31:0/999273835 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f47d40a3110 0x7f47d40a55d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f47dc009230 tx=0x7f47dc005fd0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:20.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.686+0000 7f47f87e2740 1 --2- 172.21.15.31:0/2587634642 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55713ce58080 0x55713ce60630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.686+0000 7f47f87e2740 1 --2- 172.21.15.31:0/2587634642 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55713ce576d0 0x55713ce57ab0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.686+0000 7f47f87e2740 1 -- 172.21.15.31:0/2587634642 >> 172.21.15.31:0/2587634642 conn(0x55713ce51da0 msgr2=0x55713ce52180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:20.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.686+0000 7f47f87e2740 1 -- 172.21.15.31:0/2587634642 shutdown_connections 2024-03-21T10:42:20.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.686+0000 7f47f87e2740 1 -- 172.21.15.31:0/2587634642 wait complete. 2024-03-21T10:42:20.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.687+0000 7f47f87e2740 1 -- start start 2024-03-21T10:42:20.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.688+0000 7f47f87e2740 1 Processor -- start 2024-03-21T10:42:20.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.689+0000 7f47f87e2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55713ce576d0 0x55713ce60590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.689+0000 7f47f87e2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713ce58080 0x55713ce5d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.689+0000 7f47f87e2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55713ce60c00 0x55713ce5df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.689+0000 7f47ee000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713ce58080 0x55713ce5d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.690+0000 7f47ef400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55713ce60c00 0x55713ce5df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.690+0000 7f47eea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55713ce576d0 0x55713ce60590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.690+0000 7f47eea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55713ce576d0 0x55713ce60590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:32886/0 (socket says 172.21.15.31:32886) 2024-03-21T10:42:20.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.690+0000 7f47eea00640 1 -- 172.21.15.31:0/123178868 learned_addr learned my addr 172.21.15.31:0/123178868 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:20.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.690+0000 7f47f87e2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55713ce74bd0 con 0x55713ce60c00 2024-03-21T10:42:20.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.690+0000 7f47f87e2740 1 -- 172.21.15.31:0/123178868 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55713cd770d0 con 0x55713ce576d0 2024-03-21T10:42:20.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.690+0000 7f47f87e2740 1 -- 172.21.15.31:0/123178868 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55713ce74d50 con 0x55713ce58080 2024-03-21T10:42:20.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.690+0000 7f47f87e2740 10 client.0 _ll_register_callbacks cb 0x55713cd7be80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:20.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.691+0000 7f47eea00640 1 -- 172.21.15.31:0/123178868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713ce58080 msgr2=0x55713ce5d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:20.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.693+0000 7f47eea00640 1 --2- 172.21.15.31:0/123178868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713ce58080 0x55713ce5d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.693+0000 7f47eea00640 1 -- 172.21.15.31:0/123178868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55713ce60c00 msgr2=0x55713ce5df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:20.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.693+0000 7f47eea00640 1 --2- 172.21.15.31:0/123178868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55713ce60c00 0x55713ce5df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.693+0000 7f47eea00640 1 -- 172.21.15.31:0/123178868 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55713cacf3f0 con 0x55713ce576d0 2024-03-21T10:42:20.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.694+0000 7f47ef400640 1 --2- 172.21.15.31:0/123178868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55713ce60c00 0x55713ce5df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:42:20.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.694+0000 7f47ee000640 1 --2- 172.21.15.31:0/123178868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713ce58080 0x55713ce5d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:42:20.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.694+0000 7f47d8c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:42:20.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.694+0000 7f47eea00640 1 --2- 172.21.15.31:0/123178868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55713ce576d0 0x55713ce60590 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f47e009b550 tx=0x7f47e0005f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=881fc9e69d83a74d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:20.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.694+0000 7f47d8c00640 1 -- 172.21.15.31:0/123178868 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e00a9060 con 0x55713ce576d0 2024-03-21T10:42:20.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.695+0000 7f47d8c00640 1 -- 172.21.15.31:0/123178868 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f47e00ba2b0 con 0x55713ce576d0 2024-03-21T10:42:20.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.700+0000 7f47f87e2740 1 -- 172.21.15.31:0/123178868 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f47e009f9a0 con 0x55713ce576d0 2024-03-21T10:42:20.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.701+0000 7f47d8c00640 1 -- 172.21.15.31:0/123178868 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47e00b3070 con 0x55713ce576d0 2024-03-21T10:42:20.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.701+0000 7f47f87e2740 10 client.26600 fetch_fsmap learned FSMap version 25 2024-03-21T10:42:20.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.701+0000 7f47f87e2740 1 -- 172.21.15.31:0/123178868 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55713cb997d0 con 0x55713ce576d0 2024-03-21T10:42:20.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.703+0000 7f47d8c00640 1 -- 172.21.15.31:0/123178868 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47e009f9a0 con 0x55713ce576d0 2024-03-21T10:42:20.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.703+0000 7f47d8c00640 1 -- 172.21.15.31:0/123178868 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f47e00ac050 con 0x55713ce576d0 2024-03-21T10:42:20.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.703+0000 7f47f87e2740 10 client.26600 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:42:20.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.703+0000 7f47f87e2740 10 client.26600 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:42:20.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.703+0000 7f47f87e2740 20 client.26600 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:20.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.705+0000 7f47f87e2740 1 --2- 172.21.15.31:0/123178868 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55713ce81a20 0x55713ce83e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.705+0000 7f47f87e2740 4 client.26600 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:42:20.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.705+0000 7f47f87e2740 1 -- 172.21.15.31:0/123178868 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55713cdc8800 con 0x55713ce81a20 2024-03-21T10:42:20.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.708+0000 7f47ee000640 1 --2- 172.21.15.31:0/123178868 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55713ce81a20 0x55713ce83e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.717+0000 7f47ee000640 1 --2- 172.21.15.31:0/123178868 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55713ce81a20 0x55713ce83e10 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e95f8e5e5b7d4456 server_cookie=6bafe51ea3188d05 in_seq=0 out_seq=0 2024-03-21T10:42:20.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.717+0000 7f47d8c00640 10 client.26600 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:42:20.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.725+0000 7f47d8c00640 1 -- 172.21.15.31:0/123178868 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55713cdc8800 con 0x55713ce81a20 2024-03-21T10:42:20.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.725+0000 7f47d8c00640 10 client.26600 handle_command_reply: tid=0 2024-03-21T10:42:20.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.725+0000 7f47f87e2740 1 client.26600 shutdown 2024-03-21T10:42:20.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.727+0000 7f47f87e2740 1 -- 172.21.15.31:0/123178868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55713ce576d0 msgr2=0x55713ce60590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:20.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.727+0000 7f47f87e2740 1 --2- 172.21.15.31:0/123178868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55713ce576d0 0x55713ce60590 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f47e009b550 tx=0x7f47e0005f70 comp rx=0 tx=0).stop 2024-03-21T10:42:20.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.727+0000 7f47f87e2740 1 -- 172.21.15.31:0/123178868 shutdown_connections 2024-03-21T10:42:20.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.727+0000 7f47f87e2740 1 --2- 172.21.15.31:0/123178868 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55713ce81a20 0x55713ce83e10 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.727+0000 7f47f87e2740 1 --2- 172.21.15.31:0/123178868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55713ce60c00 0x55713ce5df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.727+0000 7f47f87e2740 1 --2- 172.21.15.31:0/123178868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713ce58080 0x55713ce5d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.727+0000 7f47f87e2740 1 --2- 172.21.15.31:0/123178868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55713ce576d0 0x55713ce60590 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.727+0000 7f47f87e2740 1 -- 172.21.15.31:0/123178868 >> 172.21.15.31:0/123178868 conn(0x55713ce51da0 msgr2=0x55713ce53af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:20.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.727+0000 7f47d8c00640 0 client.26600 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:42:20.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.728+0000 7f47f87e2740 1 -- 172.21.15.31:0/123178868 shutdown_connections 2024-03-21T10:42:20.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.729+0000 7f47f87e2740 1 -- 172.21.15.31:0/123178868 wait complete. 2024-03-21T10:42:20.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.730+0000 7f47f87e2740 20 client.26600 trim_cache size 0 max 16384 2024-03-21T10:42:20.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.788+0000 7f47f87e2740 1 Processor -- start 2024-03-21T10:42:20.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.788+0000 7f47f87e2740 1 -- start start 2024-03-21T10:42:20.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.789+0000 7f47f87e2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55713ce576d0 0x55713ce7e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.789+0000 7f47f87e2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55713ce58080 0x55713ce60050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.789+0000 7f47f87e2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713ce60c00 0x55713ce60590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.789+0000 7f47f87e2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f019e230 con 0x55713ce576d0 2024-03-21T10:42:20.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.789+0000 7f47f87e2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f019bd30 con 0x55713ce58080 2024-03-21T10:42:20.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.789+0000 7f47f87e2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f00735f0 con 0x55713ce60c00 2024-03-21T10:42:20.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.790+0000 7f47ef400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713ce60c00 0x55713ce60590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.790+0000 7f47ef400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713ce60c00 0x55713ce60590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60292/0 (socket says 172.21.15.31:60292) 2024-03-21T10:42:20.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.790+0000 7f47ef400640 1 -- 172.21.15.31:0/949856870 learned_addr learned my addr 172.21.15.31:0/949856870 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:20.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.791+0000 7f47ee000640 1 --2- 172.21.15.31:0/949856870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55713ce58080 0x55713ce60050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.791+0000 7f47eea00640 1 --2- 172.21.15.31:0/949856870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55713ce576d0 0x55713ce7e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.791+0000 7f47ee000640 1 -- 172.21.15.31:0/949856870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713ce60c00 msgr2=0x55713ce60590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:20.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.791+0000 7f47ee000640 1 --2- 172.21.15.31:0/949856870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713ce60c00 0x55713ce60590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.791+0000 7f47ee000640 1 -- 172.21.15.31:0/949856870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55713ce576d0 msgr2=0x55713ce7e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:20.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.792+0000 7f47ee000640 1 --2- 172.21.15.31:0/949856870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55713ce576d0 0x55713ce7e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.792+0000 7f47ee000640 1 -- 172.21.15.31:0/949856870 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55713ce63220 con 0x55713ce58080 2024-03-21T10:42:20.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.793+0000 7f47ee000640 1 --2- 172.21.15.31:0/949856870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55713ce58080 0x55713ce60050 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f47dc008000 tx=0x7f47dc03d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7eb2960d826a129b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:20.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.794+0000 7f47db400640 1 -- 172.21.15.31:0/949856870 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc0501e0 con 0x55713ce58080 2024-03-21T10:42:20.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.795+0000 7f47db400640 1 -- 172.21.15.31:0/949856870 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47dc056070 con 0x55713ce58080 2024-03-21T10:42:20.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.795+0000 7f47db400640 1 -- 172.21.15.31:0/949856870 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc0501e0 con 0x55713ce58080 2024-03-21T10:42:20.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.797+0000 7f47f87e2740 1 -- 172.21.15.31:0/949856870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55713ce58080 msgr2=0x55713ce60050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:20.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.797+0000 7f47f87e2740 1 --2- 172.21.15.31:0/949856870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55713ce58080 0x55713ce60050 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f47dc008000 tx=0x7f47dc03d000 comp rx=0 tx=0).stop 2024-03-21T10:42:20.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.797+0000 7f47f87e2740 1 -- 172.21.15.31:0/949856870 shutdown_connections 2024-03-21T10:42:20.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.798+0000 7f47f87e2740 1 --2- 172.21.15.31:0/949856870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713ce60c00 0x55713ce60590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.798+0000 7f47f87e2740 1 --2- 172.21.15.31:0/949856870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55713ce58080 0x55713ce60050 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.798+0000 7f47f87e2740 1 --2- 172.21.15.31:0/949856870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55713ce576d0 0x55713ce7e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.798+0000 7f47f87e2740 1 -- 172.21.15.31:0/949856870 >> 172.21.15.31:0/949856870 conn(0x55713ce51da0 msgr2=0x55713ce53a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:20.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.798+0000 7f47f87e2740 1 -- 172.21.15.31:0/949856870 shutdown_connections 2024-03-21T10:42:20.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.799+0000 7f47f87e2740 1 -- 172.21.15.31:0/949856870 wait complete. 2024-03-21T10:42:20.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.801+0000 7f47f87e2740 1 -- start start 2024-03-21T10:42:20.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.802+0000 7f47f87e2740 1 Processor -- start 2024-03-21T10:42:20.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.804+0000 7f47f87e2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55713ce576d0 0x55713ce86840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.804+0000 7f47f87e2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713ce58080 0x55713ce86d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.804+0000 7f47f87e2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55713ce60c00 0x55713ce84dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.804+0000 7f47f87e2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f019a3d0 con 0x55713ce576d0 2024-03-21T10:42:20.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.804+0000 7f47f87e2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f019b6d0 con 0x55713ce60c00 2024-03-21T10:42:20.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.804+0000 7f47f87e2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f019aa10 con 0x55713ce58080 2024-03-21T10:42:20.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.804+0000 7f47f87e2740 10 client.0 _ll_register_callbacks cb 0x7f47e009b710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:20.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.807+0000 7f47ef400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55713ce60c00 0x55713ce84dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.807+0000 7f47ef400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55713ce60c00 0x55713ce84dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:32896/0 (socket says 172.21.15.31:32896) 2024-03-21T10:42:20.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.807+0000 7f47ee000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713ce58080 0x55713ce86d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.808+0000 7f47ee000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713ce58080 0x55713ce86d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60298/0 (socket says 172.21.15.31:60298) 2024-03-21T10:42:20.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.808+0000 7f47ee000640 1 -- 172.21.15.31:0/1587983789 learned_addr learned my addr 172.21.15.31:0/1587983789 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:20.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.812+0000 7f47ef400640 1 -- 172.21.15.31:0/1587983789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713ce58080 msgr2=0x55713ce86d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:20.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.814+0000 7f47ef400640 1 --2- 172.21.15.31:0/1587983789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713ce58080 0x55713ce86d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.814+0000 7f47ef400640 1 -- 172.21.15.31:0/1587983789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55713ce576d0 msgr2=0x55713ce86840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:20.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.814+0000 7f47eea00640 1 --2- 172.21.15.31:0/1587983789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55713ce576d0 0x55713ce86840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.814+0000 7f47ef400640 1 --2- 172.21.15.31:0/1587983789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55713ce576d0 0x55713ce86840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.814+0000 7f47ef400640 1 -- 172.21.15.31:0/1587983789 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55713ce855c0 con 0x55713ce60c00 2024-03-21T10:42:20.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.814+0000 7f47ee000640 1 --2- 172.21.15.31:0/1587983789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713ce58080 0x55713ce86d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:42:20.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.817+0000 7f47eea00640 1 --2- 172.21.15.31:0/1587983789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55713ce576d0 0x55713ce86840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:42:20.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.819+0000 7f47ef400640 1 --2- 172.21.15.31:0/1587983789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55713ce60c00 0x55713ce84dc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x55713ce43580 tx=0x7f47e8013b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=555f068e4dd42edc server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:20.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.820+0000 7f47d8c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:42:20.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.822+0000 7f47d8c00640 1 -- 172.21.15.31:0/1587983789 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e8003e40 con 0x55713ce60c00 2024-03-21T10:42:20.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.822+0000 7f47d8c00640 1 -- 172.21.15.31:0/1587983789 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f47e8011910 con 0x55713ce60c00 2024-03-21T10:42:20.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.822+0000 7f47f87e2740 1 -- 172.21.15.31:0/1587983789 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f47dc056070 con 0x55713ce60c00 2024-03-21T10:42:20.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.823+0000 7f47d8c00640 1 -- 172.21.15.31:0/1587983789 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47e8046050 con 0x55713ce60c00 2024-03-21T10:42:20.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.823+0000 7f47f87e2740 10 client.26606 fetch_fsmap learned FSMap version 25 2024-03-21T10:42:20.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.823+0000 7f47f87e2740 1 -- 172.21.15.31:0/1587983789 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55713ceddc50 con 0x55713ce60c00 2024-03-21T10:42:20.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.826+0000 7f47d8c00640 1 -- 172.21.15.31:0/1587983789 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47dc056070 con 0x55713ce60c00 2024-03-21T10:42:20.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.829+0000 7f47d8c00640 1 -- 172.21.15.31:0/1587983789 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f47e801a020 con 0x55713ce60c00 2024-03-21T10:42:20.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.830+0000 7f47f87e2740 10 client.26606 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:42:20.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.830+0000 7f47f87e2740 10 client.26606 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:42:20.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.830+0000 7f47f87e2740 20 client.26606 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:20.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.830+0000 7f47f87e2740 1 --2- 172.21.15.31:0/1587983789 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55713cede300 0x55713cee0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:20.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.830+0000 7f47f87e2740 4 client.26606 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:42:20.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.830+0000 7f47f87e2740 1 -- 172.21.15.31:0/1587983789 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55713cdc2b50 con 0x55713cede300 2024-03-21T10:42:20.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.830+0000 7f47eea00640 1 --2- 172.21.15.31:0/1587983789 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55713cede300 0x55713cee0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:20.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.832+0000 7f47d8c00640 10 client.26606 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:42:20.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.832+0000 7f47eea00640 1 --2- 172.21.15.31:0/1587983789 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55713cede300 0x55713cee0720 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7bbc8c0ac02be359 server_cookie=b200d7b162a0f196 in_seq=0 out_seq=0 2024-03-21T10:42:20.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.835+0000 7f47d8c00640 1 -- 172.21.15.31:0/1587983789 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55713cdc2b50 con 0x55713cede300 2024-03-21T10:42:20.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.835+0000 7f47d8c00640 10 client.26606 handle_command_reply: tid=0 2024-03-21T10:42:20.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.835+0000 7f47f87e2740 1 client.26606 shutdown 2024-03-21T10:42:20.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.838+0000 7f47f87e2740 1 -- 172.21.15.31:0/1587983789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55713ce60c00 msgr2=0x55713ce84dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:20.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.838+0000 7f47f87e2740 1 --2- 172.21.15.31:0/1587983789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55713ce60c00 0x55713ce84dc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x55713ce43580 tx=0x7f47e8013b70 comp rx=0 tx=0).stop 2024-03-21T10:42:20.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.839+0000 7f47f87e2740 1 -- 172.21.15.31:0/1587983789 shutdown_connections 2024-03-21T10:42:20.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.839+0000 7f47f87e2740 1 --2- 172.21.15.31:0/1587983789 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55713cede300 0x55713cee0720 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.839+0000 7f47f87e2740 1 --2- 172.21.15.31:0/1587983789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55713ce60c00 0x55713ce84dc0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.839+0000 7f47f87e2740 1 --2- 172.21.15.31:0/1587983789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55713ce58080 0x55713ce86d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.839+0000 7f47f87e2740 1 --2- 172.21.15.31:0/1587983789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55713ce576d0 0x55713ce86840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.839+0000 7f47d8c00640 0 client.26606 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:42:20.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.839+0000 7f47f87e2740 1 -- 172.21.15.31:0/1587983789 >> 172.21.15.31:0/1587983789 conn(0x55713ce51da0 msgr2=0x55713ce53130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:20.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.842+0000 7f47f87e2740 1 -- 172.21.15.31:0/1587983789 shutdown_connections 2024-03-21T10:42:20.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.842+0000 7f47f87e2740 1 -- 172.21.15.31:0/1587983789 wait complete. 2024-03-21T10:42:20.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.842+0000 7f47f87e2740 20 client.26606 trim_cache size 0 max 16384 2024-03-21T10:42:20.854 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:42:20.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.858+0000 7f47f5a00640 1 -- 172.21.15.31:0/999273835 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f47d40a3110 msgr2=0x7f47d40a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:20.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.858+0000 7f47f5a00640 1 --2- 172.21.15.31:0/999273835 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f47d40a3110 0x7f47d40a55d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f47dc009230 tx=0x7f47dc005fd0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.858+0000 7f47f5a00640 1 -- 172.21.15.31:0/999273835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47f0104200 msgr2=0x7f47f0075bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:20.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.858+0000 7f47f5a00640 1 --2- 172.21.15.31:0/999273835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47f0104200 0x7f47f0075bb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f47e000d950 tx=0x7f47e000de10 comp rx=0 tx=0).stop 2024-03-21T10:42:20.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.860+0000 7f47f5a00640 1 -- 172.21.15.31:0/999273835 shutdown_connections 2024-03-21T10:42:20.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.860+0000 7f47f5a00640 1 --2- 172.21.15.31:0/999273835 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f47d40a3110 0x7f47d40a55d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.860+0000 7f47f5a00640 1 --2- 172.21.15.31:0/999273835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47f0076630 0x7f47f01aa900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.860+0000 7f47f5a00640 1 --2- 172.21.15.31:0/999273835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47f0109850 0x7f47f00760f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.860+0000 7f47f5a00640 1 --2- 172.21.15.31:0/999273835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47f0104200 0x7f47f0075bb0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:20.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.860+0000 7f47f5a00640 1 -- 172.21.15.31:0/999273835 >> 172.21.15.31:0/999273835 conn(0x7f47f00ff2f0 msgr2=0x7f47f00ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:20.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.860+0000 7f47f5a00640 1 -- 172.21.15.31:0/999273835 shutdown_connections 2024-03-21T10:42:20.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:20.860+0000 7f47f5a00640 1 -- 172.21.15.31:0/999273835 wait complete. 2024-03-21T10:42:20.897 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:42:20.897 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-21T10:42:21.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:20 smithi031 ceph-mon[31673]: pgmap v116: 129 pgs: 129 active+clean; 18 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 203 MiB/s wr, 1.68k op/s 2024-03-21T10:42:21.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:20 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2899571957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:42:21.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:20 smithi070 ceph-mon[36330]: pgmap v116: 129 pgs: 129 active+clean; 18 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 203 MiB/s wr, 1.68k op/s 2024-03-21T10:42:21.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:20 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2899571957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:42:21.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:20 smithi040 ceph-mon[36154]: pgmap v116: 129 pgs: 129 active+clean; 18 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 203 MiB/s wr, 1.68k op/s 2024-03-21T10:42:21.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:20 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2899571957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:42:21.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.470+0000 7f9987c00640 1 -- 172.21.15.31:0/3839557401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9980101630 msgr2=0x7f9980101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:21.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.470+0000 7f9987c00640 1 --2- 172.21.15.31:0/3839557401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9980101630 0x7f9980101ab0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f9974009860 tx=0x7f997402bbb0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.470+0000 7f9987c00640 1 -- 172.21.15.31:0/3839557401 shutdown_connections 2024-03-21T10:42:21.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.470+0000 7f9987c00640 1 --2- 172.21.15.31:0/3839557401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9980101ff0 0x7f998010e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.470+0000 7f9987c00640 1 --2- 172.21.15.31:0/3839557401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9980101630 0x7f9980101ab0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.470+0000 7f9987c00640 1 --2- 172.21.15.31:0/3839557401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9980107640 0x7f9980107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.470+0000 7f9987c00640 1 -- 172.21.15.31:0/3839557401 >> 172.21.15.31:0/3839557401 conn(0x7f99800fd140 msgr2=0x7f99800ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:21.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.471+0000 7f9987c00640 1 -- 172.21.15.31:0/3839557401 shutdown_connections 2024-03-21T10:42:21.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.471+0000 7f9987c00640 1 -- 172.21.15.31:0/3839557401 wait complete. 2024-03-21T10:42:21.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.473+0000 7f9987c00640 1 Processor -- start 2024-03-21T10:42:21.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.476+0000 7f9987c00640 1 -- start start 2024-03-21T10:42:21.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.476+0000 7f9987c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9980101630 0x7f9980072ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:21.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.476+0000 7f9987c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9980101ff0 0x7f998006bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:21.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.476+0000 7f9987c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9980107640 0x7f998006c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:21.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.476+0000 7f9987c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998010f8a0 con 0x7f9980101630 2024-03-21T10:42:21.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.476+0000 7f9987c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998010f720 con 0x7f9980107640 2024-03-21T10:42:21.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.476+0000 7f9987c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998010fa20 con 0x7f9980101ff0 2024-03-21T10:42:21.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.479+0000 7f9985c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9980107640 0x7f998006c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:21.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.479+0000 7f9985c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9980107640 0x7f998006c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:32908/0 (socket says 172.21.15.31:32908) 2024-03-21T10:42:21.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.479+0000 7f9985c00640 1 -- 172.21.15.31:0/3622477988 learned_addr learned my addr 172.21.15.31:0/3622477988 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:21.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.479+0000 7f997fe00640 1 --2- 172.21.15.31:0/3622477988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9980101ff0 0x7f998006bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:21.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.480+0000 7f9985c00640 1 -- 172.21.15.31:0/3622477988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9980101ff0 msgr2=0x7f998006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:21.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.480+0000 7f9985c00640 1 --2- 172.21.15.31:0/3622477988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9980101ff0 0x7f998006bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.480+0000 7f9985c00640 1 -- 172.21.15.31:0/3622477988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9980101630 msgr2=0x7f9980072ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:21.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.480+0000 7f9985c00640 1 --2- 172.21.15.31:0/3622477988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9980101630 0x7f9980072ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.480+0000 7f9985c00640 1 -- 172.21.15.31:0/3622477988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f998006ca80 con 0x7f9980107640 2024-03-21T10:42:21.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.481+0000 7f9985c00640 1 --2- 172.21.15.31:0/3622477988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9980107640 0x7f998006c490 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f997800b970 tx=0x7f997800be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:21.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.482+0000 7f997d600640 1 -- 172.21.15.31:0/3622477988 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9978019070 con 0x7f9980107640 2024-03-21T10:42:21.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.482+0000 7f997d600640 1 -- 172.21.15.31:0/3622477988 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f997800cb40 con 0x7f9980107640 2024-03-21T10:42:21.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.482+0000 7f997d600640 1 -- 172.21.15.31:0/3622477988 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9978019070 con 0x7f9980107640 2024-03-21T10:42:21.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.484+0000 7f9987c00640 1 -- 172.21.15.31:0/3622477988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f998006cd70 con 0x7f9980107640 2024-03-21T10:42:21.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.484+0000 7f9987c00640 1 -- 172.21.15.31:0/3622477988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9980070c30 con 0x7f9980107640 2024-03-21T10:42:21.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.495+0000 7f998a814740 1 Processor -- start 2024-03-21T10:42:21.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.496+0000 7f997d600640 1 -- 172.21.15.31:0/3622477988 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f9978014440 con 0x7f9980107640 2024-03-21T10:42:21.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.497+0000 7f998a814740 1 -- start start 2024-03-21T10:42:21.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.497+0000 7f998a814740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c959ac6d0 0x559c959acab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:21.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.497+0000 7f998a814740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c959ad080 0x559c959b5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:21.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.497+0000 7f998a814740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c959b5c00 0x559c959b8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:21.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.497+0000 7f998a814740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c959102c0 con 0x559c959ad080 2024-03-21T10:42:21.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.497+0000 7f998a814740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c958b8c90 con 0x559c959ac6d0 2024-03-21T10:42:21.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.497+0000 7f998a814740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c9590c360 con 0x559c959b5c00 2024-03-21T10:42:21.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.499+0000 7f997fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c959ac6d0 0x559c959acab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:21.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.500+0000 7f997fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c959ac6d0 0x559c959acab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:32916/0 (socket says 172.21.15.31:32916) 2024-03-21T10:42:21.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.500+0000 7f997fe00640 1 -- 172.21.15.31:0/2778268867 learned_addr learned my addr 172.21.15.31:0/2778268867 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:21.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.500+0000 7f997fe00640 1 --2- 172.21.15.31:0/2778268867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c959b5c00 0x559c959b8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:21.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.500+0000 7f9985200640 1 --2- 172.21.15.31:0/2778268867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c959ad080 0x559c959b5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:21.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.502+0000 7f997d600640 1 --2- 172.21.15.31:0/3622477988 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f99680a3340 0x7f99680a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:21.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.502+0000 7f997d600640 1 -- 172.21.15.31:0/3622477988 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f99780ceaf0 con 0x7f9980107640 2024-03-21T10:42:21.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.502+0000 7f997fe00640 1 -- 172.21.15.31:0/2778268867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c959b5c00 msgr2=0x559c959b8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:21.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.502+0000 7f997fe00640 1 --2- 172.21.15.31:0/2778268867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c959b5c00 0x559c959b8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.502+0000 7f997fe00640 1 -- 172.21.15.31:0/2778268867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c959ad080 msgr2=0x559c959b5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:21.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.502+0000 7f997fe00640 1 --2- 172.21.15.31:0/2778268867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c959ad080 0x559c959b5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.502+0000 7f997fe00640 1 -- 172.21.15.31:0/2778268867 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c958dd080 con 0x559c959ac6d0 2024-03-21T10:42:21.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.503+0000 7f997fe00640 1 --2- 172.21.15.31:0/2778268867 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c959ac6d0 0x559c959acab0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f9974009780 tx=0x7f997402e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c6d2d8ad5de0c76 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:21.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.503+0000 7f9985c00640 1 --2- 172.21.15.31:0/3622477988 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f99680a3340 0x7f99680a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:21.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.503+0000 7f9972000640 1 -- 172.21.15.31:0/2778268867 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9974039040 con 0x559c959ac6d0 2024-03-21T10:42:21.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.504+0000 7f9972000640 1 -- 172.21.15.31:0/2778268867 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9974004230 con 0x559c959ac6d0 2024-03-21T10:42:21.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.504+0000 7f9972000640 1 -- 172.21.15.31:0/2778268867 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99740054a0 con 0x559c959ac6d0 2024-03-21T10:42:21.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.505+0000 7f998a814740 1 -- 172.21.15.31:0/2778268867 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c959ac6d0 msgr2=0x559c959acab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:21.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.505+0000 7f998a814740 1 --2- 172.21.15.31:0/2778268867 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c959ac6d0 0x559c959acab0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f9974009780 tx=0x7f997402e040 comp rx=0 tx=0).stop 2024-03-21T10:42:21.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.507+0000 7f998a814740 1 -- 172.21.15.31:0/2778268867 shutdown_connections 2024-03-21T10:42:21.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.507+0000 7f998a814740 1 --2- 172.21.15.31:0/2778268867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c959b5c00 0x559c959b8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.507+0000 7f998a814740 1 --2- 172.21.15.31:0/2778268867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c959ad080 0x559c959b5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.507+0000 7f998a814740 1 --2- 172.21.15.31:0/2778268867 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c959ac6d0 0x559c959acab0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.507+0000 7f998a814740 1 -- 172.21.15.31:0/2778268867 >> 172.21.15.31:0/2778268867 conn(0x559c959a6da0 msgr2=0x559c959a7180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:21.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.508+0000 7f9985c00640 1 --2- 172.21.15.31:0/3622477988 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f99680a3340 0x7f99680a5800 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f99780cc010 tx=0x7f997800cf60 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:21.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.508+0000 7f998a814740 1 -- 172.21.15.31:0/2778268867 shutdown_connections 2024-03-21T10:42:21.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.508+0000 7f998a814740 1 -- 172.21.15.31:0/2778268867 wait complete. 2024-03-21T10:42:21.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.509+0000 7f998a814740 1 -- start start 2024-03-21T10:42:21.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.509+0000 7f998a814740 1 Processor -- start 2024-03-21T10:42:21.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.511+0000 7f998a814740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c959ac6d0 0x559c959b5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:21.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.511+0000 7f998a814740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c959ad080 0x559c959b2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:21.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.512+0000 7f998a814740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c959b5c00 0x559c959b3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:21.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.512+0000 7f998a814740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c959c9ac0 con 0x559c959ac6d0 2024-03-21T10:42:21.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.513+0000 7f998a814740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c958cc0d0 con 0x559c959b5c00 2024-03-21T10:42:21.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.513+0000 7f998a814740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c959c9c40 con 0x559c959ad080 2024-03-21T10:42:21.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.513+0000 7f998a814740 10 client.0 _ll_register_callbacks cb 0x559c958d0e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:21.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.514+0000 7f9985200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c959ad080 0x559c959b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:21.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.514+0000 7f9985200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c959ad080 0x559c959b2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60344/0 (socket says 172.21.15.31:60344) 2024-03-21T10:42:21.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.514+0000 7f9985200640 1 -- 172.21.15.31:0/2420533946 learned_addr learned my addr 172.21.15.31:0/2420533946 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:21.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.514+0000 7f9985200640 1 -- 172.21.15.31:0/2420533946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c959b5c00 msgr2=0x559c959b3000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:42:21.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.514+0000 7f9985200640 1 --2- 172.21.15.31:0/2420533946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c959b5c00 0x559c959b3000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.514+0000 7f9985200640 1 -- 172.21.15.31:0/2420533946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c959ac6d0 msgr2=0x559c959b5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:21.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.514+0000 7f9985200640 1 --2- 172.21.15.31:0/2420533946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c959ac6d0 0x559c959b5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.515+0000 7f9985200640 1 -- 172.21.15.31:0/2420533946 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559c956243f0 con 0x559c959ad080 2024-03-21T10:42:21.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.515+0000 7f9985200640 1 --2- 172.21.15.31:0/2420533946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c959ad080 0x559c959b2980 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f996c00cdb0 tx=0x7f996c00a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e9fad5c2fdfea4f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:21.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.515+0000 7f9966a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:42:21.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.516+0000 7f9966a00640 1 -- 172.21.15.31:0/2420533946 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c004220 con 0x559c959ad080 2024-03-21T10:42:21.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.516+0000 7f9966a00640 1 -- 172.21.15.31:0/2420533946 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f996c0043c0 con 0x559c959ad080 2024-03-21T10:42:21.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.516+0000 7f998a814740 1 -- 172.21.15.31:0/2420533946 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9974004230 con 0x559c959ad080 2024-03-21T10:42:21.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.517+0000 7f9966a00640 1 -- 172.21.15.31:0/2420533946 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f996c005020 con 0x559c959ad080 2024-03-21T10:42:21.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.517+0000 7f998a814740 10 client.26560 fetch_fsmap learned FSMap version 25 2024-03-21T10:42:21.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.517+0000 7f998a814740 1 -- 172.21.15.31:0/2420533946 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x559c956ee7d0 con 0x559c959ad080 2024-03-21T10:42:21.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.518+0000 7f9966a00640 1 -- 172.21.15.31:0/2420533946 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f996c0047a0 con 0x559c959ad080 2024-03-21T10:42:21.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.518+0000 7f9966a00640 1 -- 172.21.15.31:0/2420533946 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f996c011070 con 0x559c959ad080 2024-03-21T10:42:21.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.518+0000 7f998a814740 10 client.26560 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:42:21.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.518+0000 7f998a814740 10 client.26560 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:42:21.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.518+0000 7f998a814740 20 client.26560 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:21.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.518+0000 7f998a814740 1 --2- 172.21.15.31:0/2420533946 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x559c959d6bb0 0x559c959d8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:21.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.518+0000 7f998a814740 4 client.26560 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:42:21.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.518+0000 7f998a814740 1 -- 172.21.15.31:0/2420533946 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9974004230 con 0x559c959d6bb0 2024-03-21T10:42:21.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.520+0000 7f997fe00640 1 --2- 172.21.15.31:0/2420533946 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x559c959d6bb0 0x559c959d8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:21.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.523+0000 7f997fe00640 1 --2- 172.21.15.31:0/2420533946 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x559c959d6bb0 0x559c959d8fa0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42fdadb52d3920a2 server_cookie=a2d9b9817cdf982b in_seq=0 out_seq=0 2024-03-21T10:42:21.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.523+0000 7f9966a00640 10 client.26560 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:21.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.524+0000 7f9966a00640 1 -- 172.21.15.31:0/2420533946 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7f9974004230 con 0x559c959d6bb0 2024-03-21T10:42:21.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.524+0000 7f9966a00640 10 client.26560 handle_command_reply: tid=0 2024-03-21T10:42:21.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.524+0000 7f998a814740 1 client.26560 shutdown 2024-03-21T10:42:21.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.525+0000 7f998a814740 1 -- 172.21.15.31:0/2420533946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c959ad080 msgr2=0x559c959b2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:21.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.525+0000 7f998a814740 1 --2- 172.21.15.31:0/2420533946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c959ad080 0x559c959b2980 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f996c00cdb0 tx=0x7f996c00a620 comp rx=0 tx=0).stop 2024-03-21T10:42:21.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.526+0000 7f998a814740 1 -- 172.21.15.31:0/2420533946 shutdown_connections 2024-03-21T10:42:21.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.526+0000 7f998a814740 1 --2- 172.21.15.31:0/2420533946 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x559c959d6bb0 0x559c959d8fa0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.526+0000 7f998a814740 1 --2- 172.21.15.31:0/2420533946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c959b5c00 0x559c959b3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.526+0000 7f998a814740 1 --2- 172.21.15.31:0/2420533946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c959ad080 0x559c959b2980 secure :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f996c00cdb0 tx=0x7f996c00a620 comp rx=0 tx=0).stop 2024-03-21T10:42:21.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.526+0000 7f998a814740 1 --2- 172.21.15.31:0/2420533946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c959ac6d0 0x559c959b5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.526+0000 7f998a814740 1 -- 172.21.15.31:0/2420533946 >> 172.21.15.31:0/2420533946 conn(0x559c959a6da0 msgr2=0x559c959a8af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:21.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.526+0000 7f9966a00640 0 client.26560 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:21.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.528+0000 7f998a814740 1 -- 172.21.15.31:0/2420533946 shutdown_connections 2024-03-21T10:42:21.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.531+0000 7f998a814740 1 -- 172.21.15.31:0/2420533946 wait complete. 2024-03-21T10:42:21.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.531+0000 7f998a814740 20 client.26560 trim_cache size 0 max 16384 2024-03-21T10:42:21.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.588+0000 7f998a814740 1 Processor -- start 2024-03-21T10:42:21.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.588+0000 7f998a814740 1 -- start start 2024-03-21T10:42:21.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.589+0000 7f998a814740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c959ac6d0 0x559c959b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:21.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.589+0000 7f998a814740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c959b5c00 0x559c959afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:21.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.589+0000 7f998a814740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c959b4cf0 0x559c959b0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:21.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.589+0000 7f998a814740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99801b73d0 con 0x559c959ac6d0 2024-03-21T10:42:21.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.589+0000 7f998a814740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99801b4ed0 con 0x559c959b4cf0 2024-03-21T10:42:21.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.589+0000 7f998a814740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99801b2f70 con 0x559c959b5c00 2024-03-21T10:42:21.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.589+0000 7f9985200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c959b5c00 0x559c959afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:21.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.589+0000 7f9985200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c959b5c00 0x559c959afc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60348/0 (socket says 172.21.15.31:60348) 2024-03-21T10:42:21.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.589+0000 7f9985200640 1 -- 172.21.15.31:0/1596495468 learned_addr learned my addr 172.21.15.31:0/1596495468 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:21.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.590+0000 7f9985200640 1 -- 172.21.15.31:0/1596495468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c959b4cf0 msgr2=0x559c959b0140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:42:21.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.590+0000 7f9985200640 1 --2- 172.21.15.31:0/1596495468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c959b4cf0 0x559c959b0140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.590+0000 7f9985200640 1 -- 172.21.15.31:0/1596495468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c959ac6d0 msgr2=0x559c959b47b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:42:21.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.591+0000 7f997fe00640 1 --2- 172.21.15.31:0/1596495468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c959ac6d0 0x559c959b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:21.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.591+0000 7f9985200640 1 --2- 172.21.15.31:0/1596495468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c959ac6d0 0x559c959b47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.591+0000 7f9985200640 1 -- 172.21.15.31:0/1596495468 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c959d9620 con 0x559c959b5c00 2024-03-21T10:42:21.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.592+0000 7f9985200640 1 --2- 172.21.15.31:0/1596495468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c959b5c00 0x559c959afc00 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f996c000c00 tx=0x7f996c00af40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f28110c424f1732 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:21.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.592+0000 7f9971600640 1 -- 172.21.15.31:0/1596495468 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c032070 con 0x559c959b5c00 2024-03-21T10:42:21.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.592+0000 7f9971600640 1 -- 172.21.15.31:0/1596495468 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f996c0110c0 con 0x559c959b5c00 2024-03-21T10:42:21.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.592+0000 7f9971600640 1 -- 172.21.15.31:0/1596495468 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c004020 con 0x559c959b5c00 2024-03-21T10:42:21.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.593+0000 7f998a814740 1 -- 172.21.15.31:0/1596495468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c959b5c00 msgr2=0x559c959afc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:21.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.593+0000 7f998a814740 1 --2- 172.21.15.31:0/1596495468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c959b5c00 0x559c959afc00 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f996c000c00 tx=0x7f996c00af40 comp rx=0 tx=0).stop 2024-03-21T10:42:21.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.594+0000 7f998a814740 1 -- 172.21.15.31:0/1596495468 shutdown_connections 2024-03-21T10:42:21.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.594+0000 7f998a814740 1 --2- 172.21.15.31:0/1596495468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c959b4cf0 0x559c959b0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.594+0000 7f998a814740 1 --2- 172.21.15.31:0/1596495468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c959b5c00 0x559c959afc00 secure :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f996c000c00 tx=0x7f996c00af40 comp rx=0 tx=0).stop 2024-03-21T10:42:21.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.594+0000 7f998a814740 1 --2- 172.21.15.31:0/1596495468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c959ac6d0 0x559c959b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.594+0000 7f998a814740 1 -- 172.21.15.31:0/1596495468 >> 172.21.15.31:0/1596495468 conn(0x559c959a6da0 msgr2=0x559c959b40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:21.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.595+0000 7f998a814740 1 -- 172.21.15.31:0/1596495468 shutdown_connections 2024-03-21T10:42:21.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.595+0000 7f998a814740 1 -- 172.21.15.31:0/1596495468 wait complete. 2024-03-21T10:42:21.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.595+0000 7f998a814740 1 -- start start 2024-03-21T10:42:21.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.596+0000 7f998a814740 1 Processor -- start 2024-03-21T10:42:21.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.598+0000 7f998a814740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c959ac6d0 0x559c959cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:21.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.599+0000 7f997fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c959ac6d0 0x559c959cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:21.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.600+0000 7f998a814740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c959b4cf0 0x559c959cd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:21.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.600+0000 7f998a814740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c959cde60 0x559c95a32490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:21.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.600+0000 7f998a814740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99801b3570 con 0x559c959cde60 2024-03-21T10:42:21.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.600+0000 7f998a814740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99801b4870 con 0x559c959b4cf0 2024-03-21T10:42:21.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.600+0000 7f998a814740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99801b3bb0 con 0x559c959ac6d0 2024-03-21T10:42:21.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.600+0000 7f997fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c959ac6d0 0x559c959cd3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60362/0 (socket says 172.21.15.31:60362) 2024-03-21T10:42:21.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.600+0000 7f997fe00640 1 -- 172.21.15.31:0/134723762 learned_addr learned my addr 172.21.15.31:0/134723762 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:21.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.601+0000 7f998a814740 10 client.0 _ll_register_callbacks cb 0x7f996c00c9c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:21.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.601+0000 7f9985200640 1 --2- 172.21.15.31:0/134723762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c959b4cf0 0x559c959cd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:21.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.601+0000 7f997fe00640 1 -- 172.21.15.31:0/134723762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c959b4cf0 msgr2=0x559c959cd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:21.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.602+0000 7f997fe00640 1 --2- 172.21.15.31:0/134723762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c959b4cf0 0x559c959cd920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.604+0000 7f997fe00640 1 -- 172.21.15.31:0/134723762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c959cde60 msgr2=0x559c95a32490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:21.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.604+0000 7f997fe00640 1 --2- 172.21.15.31:0/134723762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c959cde60 0x559c95a32490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.604+0000 7f997fe00640 1 -- 172.21.15.31:0/134723762 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559c95a329d0 con 0x559c959ac6d0 2024-03-21T10:42:21.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.604+0000 7f9985200640 1 --2- 172.21.15.31:0/134723762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c959b4cf0 0x559c959cd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:42:21.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.604+0000 7f997fe00640 1 --2- 172.21.15.31:0/134723762 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c959ac6d0 0x559c959cd3e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9974009250 tx=0x7f997402bbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eefb1a559313247c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:21.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.604+0000 7f9966a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:42:21.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.605+0000 7f9966a00640 1 -- 172.21.15.31:0/134723762 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9974059060 con 0x559c959ac6d0 2024-03-21T10:42:21.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.605+0000 7f9966a00640 1 -- 172.21.15.31:0/134723762 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f99740541b0 con 0x559c959ac6d0 2024-03-21T10:42:21.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.605+0000 7f998a814740 1 -- 172.21.15.31:0/134723762 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f996c0110c0 con 0x559c959ac6d0 2024-03-21T10:42:21.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.606+0000 7f9966a00640 1 -- 172.21.15.31:0/134723762 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9974039040 con 0x559c959ac6d0 2024-03-21T10:42:21.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.607+0000 7f998a814740 10 client.26566 fetch_fsmap learned FSMap version 25 2024-03-21T10:42:21.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.607+0000 7f998a814740 1 -- 172.21.15.31:0/134723762 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x559c95a33cb0 con 0x559c959ac6d0 2024-03-21T10:42:21.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.608+0000 7f9966a00640 1 -- 172.21.15.31:0/134723762 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f996c0110c0 con 0x559c959ac6d0 2024-03-21T10:42:21.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.610+0000 7f9966a00640 1 -- 172.21.15.31:0/134723762 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f997403f070 con 0x559c959ac6d0 2024-03-21T10:42:21.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.610+0000 7f998a814740 10 client.26566 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:42:21.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.610+0000 7f998a814740 10 client.26566 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:42:21.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.610+0000 7f998a814740 20 client.26566 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:21.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.611+0000 7f998a814740 1 --2- 172.21.15.31:0/134723762 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x559c95a33f90 0x559c95a36380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:21.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.611+0000 7f998a814740 4 client.26566 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:42:21.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.611+0000 7f998a814740 1 -- 172.21.15.31:0/134723762 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559c9591d800 con 0x559c95a33f90 2024-03-21T10:42:21.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.611+0000 7f9985200640 1 --2- 172.21.15.31:0/134723762 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x559c95a33f90 0x559c95a36380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:21.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.614+0000 7f9985200640 1 --2- 172.21.15.31:0/134723762 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x559c95a33f90 0x559c95a36380 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4cab3c65c9a060d server_cookie=bcaf6e78d040a8b6 in_seq=0 out_seq=0 2024-03-21T10:42:21.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.621+0000 7f9966a00640 10 client.26566 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:21.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.621+0000 7f9966a00640 1 -- 172.21.15.31:0/134723762 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559c9591d800 con 0x559c95a33f90 2024-03-21T10:42:21.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.621+0000 7f9966a00640 10 client.26566 handle_command_reply: tid=0 2024-03-21T10:42:21.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.621+0000 7f998a814740 1 client.26566 shutdown 2024-03-21T10:42:21.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.622+0000 7f998a814740 1 -- 172.21.15.31:0/134723762 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c959ac6d0 msgr2=0x559c959cd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:21.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.622+0000 7f998a814740 1 --2- 172.21.15.31:0/134723762 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c959ac6d0 0x559c959cd3e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9974009250 tx=0x7f997402bbb0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.622+0000 7f998a814740 1 -- 172.21.15.31:0/134723762 shutdown_connections 2024-03-21T10:42:21.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.622+0000 7f998a814740 1 --2- 172.21.15.31:0/134723762 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x559c95a33f90 0x559c95a36380 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.622+0000 7f998a814740 1 --2- 172.21.15.31:0/134723762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c959cde60 0x559c95a32490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.622+0000 7f998a814740 1 --2- 172.21.15.31:0/134723762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c959b4cf0 0x559c959cd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.622+0000 7f998a814740 1 --2- 172.21.15.31:0/134723762 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559c959ac6d0 0x559c959cd3e0 secure :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9974009250 tx=0x7f997402bbb0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.622+0000 7f998a814740 1 -- 172.21.15.31:0/134723762 >> 172.21.15.31:0/134723762 conn(0x559c959a6da0 msgr2=0x559c959b16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:21.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.623+0000 7f9966a00640 0 client.26566 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:21.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.623+0000 7f998a814740 1 -- 172.21.15.31:0/134723762 shutdown_connections 2024-03-21T10:42:21.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.624+0000 7f998a814740 1 -- 172.21.15.31:0/134723762 wait complete. 2024-03-21T10:42:21.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.624+0000 7f998a814740 20 client.26566 trim_cache size 0 max 16384 2024-03-21T10:42:21.622 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:42:21.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.630+0000 7f9987c00640 1 -- 172.21.15.31:0/3622477988 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f99680a3340 msgr2=0x7f99680a5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:21.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.630+0000 7f9987c00640 1 --2- 172.21.15.31:0/3622477988 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f99680a3340 0x7f99680a5800 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f99780cc010 tx=0x7f997800cf60 comp rx=0 tx=0).stop 2024-03-21T10:42:21.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.630+0000 7f9987c00640 1 -- 172.21.15.31:0/3622477988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9980107640 msgr2=0x7f998006c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:21.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.630+0000 7f9987c00640 1 --2- 172.21.15.31:0/3622477988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9980107640 0x7f998006c490 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f997800b970 tx=0x7f997800be30 comp rx=0 tx=0).stop 2024-03-21T10:42:21.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.654+0000 7f9987c00640 1 -- 172.21.15.31:0/3622477988 shutdown_connections 2024-03-21T10:42:21.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.654+0000 7f9987c00640 1 --2- 172.21.15.31:0/3622477988 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f99680a3340 0x7f99680a5800 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.654+0000 7f9987c00640 1 --2- 172.21.15.31:0/3622477988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9980107640 0x7f998006c490 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.654+0000 7f9987c00640 1 --2- 172.21.15.31:0/3622477988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9980101ff0 0x7f998006bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.654+0000 7f9987c00640 1 --2- 172.21.15.31:0/3622477988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9980101630 0x7f9980072ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:21.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.654+0000 7f9987c00640 1 -- 172.21.15.31:0/3622477988 >> 172.21.15.31:0/3622477988 conn(0x7f99800fd140 msgr2=0x7f99800fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:21.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.655+0000 7f9987c00640 1 -- 172.21.15.31:0/3622477988 shutdown_connections 2024-03-21T10:42:21.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:21.655+0000 7f9987c00640 1 -- 172.21.15.31:0/3622477988 wait complete. 2024-03-21T10:42:22.694 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-21T10:42:22.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:42:22.694 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-21T10:42:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.946+0000 7f1e00e00640 1 -- 172.21.15.31:0/2530967030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dfc108880 msgr2=0x7f1dfc108c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.946+0000 7f1e00e00640 1 --2- 172.21.15.31:0/2530967030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dfc108880 0x7f1dfc108c60 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1dec009900 tx=0x7f1dec02bc30 comp rx=0 tx=0).stop 2024-03-21T10:42:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.946+0000 7f1e00e00640 1 -- 172.21.15.31:0/2530967030 shutdown_connections 2024-03-21T10:42:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.946+0000 7f1e00e00640 1 --2- 172.21.15.31:0/2530967030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dfc1147f0 0x7f1dfc116be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.946+0000 7f1e00e00640 1 --2- 172.21.15.31:0/2530967030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dfc1091a0 0x7f1dfc1142b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.946+0000 7f1e00e00640 1 --2- 172.21.15.31:0/2530967030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dfc108880 0x7f1dfc108c60 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.946+0000 7f1e00e00640 1 -- 172.21.15.31:0/2530967030 >> 172.21.15.31:0/2530967030 conn(0x7f1dfc06c750 msgr2=0x7f1dfc06cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.946+0000 7f1e00e00640 1 -- 172.21.15.31:0/2530967030 shutdown_connections 2024-03-21T10:42:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.946+0000 7f1e00e00640 1 -- 172.21.15.31:0/2530967030 wait complete. 2024-03-21T10:42:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.948+0000 7f1e00e00640 1 Processor -- start 2024-03-21T10:42:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.948+0000 7f1e00e00640 1 -- start start 2024-03-21T10:42:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.948+0000 7f1e00e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dfc108880 0x7f1dfc113cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.948+0000 7f1e00e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dfc1091a0 0x7f1dfc114200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.948+0000 7f1e00e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dfc1147f0 0x7f1dfc10dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.948+0000 7f1e00e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dfc118080 con 0x7f1dfc1147f0 2024-03-21T10:42:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.948+0000 7f1e00e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dfc117f00 con 0x7f1dfc108880 2024-03-21T10:42:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.948+0000 7f1e00e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dfc118200 con 0x7f1dfc1091a0 2024-03-21T10:42:22.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.949+0000 7f1df9400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dfc1091a0 0x7f1dfc114200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:22.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.949+0000 7f1df9400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dfc1091a0 0x7f1dfc114200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60384/0 (socket says 172.21.15.31:60384) 2024-03-21T10:42:22.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.949+0000 7f1df9400640 1 -- 172.21.15.31:0/3413815713 learned_addr learned my addr 172.21.15.31:0/3413815713 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:22.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.949+0000 7f1dfa800640 1 --2- 172.21.15.31:0/3413815713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dfc1147f0 0x7f1dfc10dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:22.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.950+0000 7f1df9e00640 1 --2- 172.21.15.31:0/3413815713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dfc108880 0x7f1dfc113cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:22.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.950+0000 7f1dfa800640 1 -- 172.21.15.31:0/3413815713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dfc1091a0 msgr2=0x7f1dfc114200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:22.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.950+0000 7f1dfa800640 1 --2- 172.21.15.31:0/3413815713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dfc1091a0 0x7f1dfc114200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:22.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.950+0000 7f1dfa800640 1 -- 172.21.15.31:0/3413815713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dfc108880 msgr2=0x7f1dfc113cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:22.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.950+0000 7f1dfa800640 1 --2- 172.21.15.31:0/3413815713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dfc108880 0x7f1dfc113cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:22.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.950+0000 7f1dfa800640 1 -- 172.21.15.31:0/3413815713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dfc10e760 con 0x7f1dfc1147f0 2024-03-21T10:42:22.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.952+0000 7f1dfa800640 1 --2- 172.21.15.31:0/3413815713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dfc1147f0 0x7f1dfc10dea0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1df4005fd0 tx=0x7f1df400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.955+0000 7f1df2a00640 1 -- 172.21.15.31:0/3413815713 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df400cdf0 con 0x7f1dfc1147f0 2024-03-21T10:42:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.955+0000 7f1df2a00640 1 -- 172.21.15.31:0/3413815713 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1df4005020 con 0x7f1dfc1147f0 2024-03-21T10:42:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.955+0000 7f1df2a00640 1 -- 172.21.15.31:0/3413815713 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df40106d0 con 0x7f1dfc1147f0 2024-03-21T10:42:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.955+0000 7f1e00e00640 1 -- 172.21.15.31:0/3413815713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dfc10ea50 con 0x7f1dfc1147f0 2024-03-21T10:42:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.955+0000 7f1e00e00640 1 -- 172.21.15.31:0/3413815713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dfc1bbff0 con 0x7f1dfc1147f0 2024-03-21T10:42:22.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.960+0000 7f1e03a80740 1 Processor -- start 2024-03-21T10:42:22.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.962+0000 7f1df2a00640 1 -- 172.21.15.31:0/3413815713 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f1df4004b20 con 0x7f1dfc1147f0 2024-03-21T10:42:22.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.963+0000 7f1e03a80740 1 -- start start 2024-03-21T10:42:22.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.963+0000 7f1e03a80740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c9e445ffd0 0x55c9e44603b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:22.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.963+0000 7f1e03a80740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9e44608f0 0x55c9e4469180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:22.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.963+0000 7f1e03a80740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9e44696c0 0x55c9e446bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:22.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.963+0000 7f1e03a80740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9e44223b0 con 0x55c9e44608f0 2024-03-21T10:42:22.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.963+0000 7f1e03a80740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9e43c4830 con 0x55c9e44696c0 2024-03-21T10:42:22.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.963+0000 7f1e03a80740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9e436d5d0 con 0x55c9e445ffd0 2024-03-21T10:42:22.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.964+0000 7f1df9400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c9e445ffd0 0x55c9e44603b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:22.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.964+0000 7f1df9400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c9e445ffd0 0x55c9e44603b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60386/0 (socket says 172.21.15.31:60386) 2024-03-21T10:42:22.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.964+0000 7f1df9400640 1 -- 172.21.15.31:0/1309719032 learned_addr learned my addr 172.21.15.31:0/1309719032 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:22.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.964+0000 7f1df2a00640 1 --2- 172.21.15.31:0/3413815713 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f1de00a3230 0x7f1de00a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:22.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.964+0000 7f1df9400640 1 --2- 172.21.15.31:0/1309719032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9e44696c0 0x55c9e446bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:22.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.964+0000 7f1df2a00640 1 -- 172.21.15.31:0/3413815713 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f1df40d0090 con 0x7f1dfc1147f0 2024-03-21T10:42:22.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.965+0000 7f1dfa800640 1 --2- 172.21.15.31:0/3413815713 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f1de00a3230 0x7f1de00a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:22.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.965+0000 7f1df9400640 1 -- 172.21.15.31:0/1309719032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9e44696c0 msgr2=0x55c9e446bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:22.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.965+0000 7f1df9400640 1 --2- 172.21.15.31:0/1309719032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9e44696c0 0x55c9e446bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:22.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.965+0000 7f1df9400640 1 -- 172.21.15.31:0/1309719032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9e44608f0 msgr2=0x55c9e4469180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:42:22.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.965+0000 7f1df9400640 1 --2- 172.21.15.31:0/1309719032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9e44608f0 0x55c9e4469180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:22.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.965+0000 7f1df9400640 1 -- 172.21.15.31:0/1309719032 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c9e43852c0 con 0x55c9e445ffd0 2024-03-21T10:42:22.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.965+0000 7f1df9400640 1 --2- 172.21.15.31:0/1309719032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c9e445ffd0 0x55c9e44603b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1de8009fd0 tx=0x7f1de8009720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7130aee8a195450e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:22.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.965+0000 7f1de7400640 1 -- 172.21.15.31:0/1309719032 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de8013d30 con 0x55c9e445ffd0 2024-03-21T10:42:22.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.965+0000 7f1de7400640 1 -- 172.21.15.31:0/1309719032 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1de80130d0 con 0x55c9e445ffd0 2024-03-21T10:42:22.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.965+0000 7f1de7400640 1 -- 172.21.15.31:0/1309719032 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de8011890 con 0x55c9e445ffd0 2024-03-21T10:42:22.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.967+0000 7f1dfa800640 1 --2- 172.21.15.31:0/3413815713 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f1de00a3230 0x7f1de00a56f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1df4009fd0 tx=0x7f1df400c860 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:22.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.967+0000 7f1e03a80740 1 -- 172.21.15.31:0/1309719032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c9e445ffd0 msgr2=0x55c9e44603b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:22.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.967+0000 7f1e03a80740 1 --2- 172.21.15.31:0/1309719032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c9e445ffd0 0x55c9e44603b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1de8009fd0 tx=0x7f1de8009720 comp rx=0 tx=0).stop 2024-03-21T10:42:22.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.967+0000 7f1e03a80740 1 -- 172.21.15.31:0/1309719032 shutdown_connections 2024-03-21T10:42:22.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.967+0000 7f1e03a80740 1 --2- 172.21.15.31:0/1309719032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9e44696c0 0x55c9e446bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:22.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.968+0000 7f1e03a80740 1 --2- 172.21.15.31:0/1309719032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9e44608f0 0x55c9e4469180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:22.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.968+0000 7f1e03a80740 1 --2- 172.21.15.31:0/1309719032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c9e445ffd0 0x55c9e44603b0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:22.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.968+0000 7f1e03a80740 1 -- 172.21.15.31:0/1309719032 >> 172.21.15.31:0/1309719032 conn(0x55c9e445a8f0 msgr2=0x55c9e445acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:22.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.968+0000 7f1e03a80740 1 -- 172.21.15.31:0/1309719032 shutdown_connections 2024-03-21T10:42:22.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.968+0000 7f1e03a80740 1 -- 172.21.15.31:0/1309719032 wait complete. 2024-03-21T10:42:22.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.969+0000 7f1e03a80740 1 -- start start 2024-03-21T10:42:22.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.969+0000 7f1e03a80740 1 Processor -- start 2024-03-21T10:42:22.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.969+0000 7f1e03a80740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c9e445ffd0 0x55c9e4466740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:22.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.970+0000 7f1e03a80740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9e44608f0 0x55c9e4466c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:22.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.970+0000 7f1e03a80740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9e44696c0 0x55c9e44671c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:22.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.970+0000 7f1e03a80740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9e4380a20 con 0x55c9e44696c0 2024-03-21T10:42:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.970+0000 7f1e03a80740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9e43c08d0 con 0x55c9e44608f0 2024-03-21T10:42:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.970+0000 7f1e03a80740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9e446c290 con 0x55c9e445ffd0 2024-03-21T10:42:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.970+0000 7f1e03a80740 10 client.0 _ll_register_callbacks cb 0x55c9e43bd1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.970+0000 7f1df9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9e44608f0 0x55c9e4466c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.970+0000 7f1df9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9e44608f0 0x55c9e4466c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:32974/0 (socket says 172.21.15.31:32974) 2024-03-21T10:42:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.970+0000 7f1df9400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c9e445ffd0 0x55c9e4466740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.970+0000 7f1df9e00640 1 -- 172.21.15.31:0/1391819349 learned_addr learned my addr 172.21.15.31:0/1391819349 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.970+0000 7f1df9400640 1 --2- 172.21.15.31:0/1391819349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9e44696c0 0x55c9e44671c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.970+0000 7f1df9e00640 1 -- 172.21.15.31:0/1391819349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c9e445ffd0 msgr2=0x55c9e4466740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.970+0000 7f1df9e00640 1 --2- 172.21.15.31:0/1391819349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c9e445ffd0 0x55c9e4466740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.970+0000 7f1df9e00640 1 -- 172.21.15.31:0/1391819349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9e44696c0 msgr2=0x55c9e44671c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.970+0000 7f1df9e00640 1 --2- 172.21.15.31:0/1391819349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9e44696c0 0x55c9e44671c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.970+0000 7f1df9e00640 1 -- 172.21.15.31:0/1391819349 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c9e40d9210 con 0x55c9e44608f0 2024-03-21T10:42:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.971+0000 7f1df9e00640 1 --2- 172.21.15.31:0/1391819349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9e44608f0 0x55c9e4466c80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1dec035000 tx=0x7f1dec005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c144fb3baa83686f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:22.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.971+0000 7f1ddbe00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:42:22.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.972+0000 7f1ddbe00640 1 -- 172.21.15.31:0/1391819349 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dec042070 con 0x55c9e44608f0 2024-03-21T10:42:22.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.972+0000 7f1ddbe00640 1 -- 172.21.15.31:0/1391819349 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f1dec02cc90 con 0x55c9e44608f0 2024-03-21T10:42:22.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.972+0000 7f1e03a80740 1 -- 172.21.15.31:0/1391819349 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1de80130d0 con 0x55c9e44608f0 2024-03-21T10:42:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.972+0000 7f1ddbe00640 1 -- 172.21.15.31:0/1391819349 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1dec04c050 con 0x55c9e44608f0 2024-03-21T10:42:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.973+0000 7f1e03a80740 10 client.26630 fetch_fsmap learned FSMap version 25 2024-03-21T10:42:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.973+0000 7f1e03a80740 1 -- 172.21.15.31:0/1391819349 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55c9e41a5620 con 0x55c9e44608f0 2024-03-21T10:42:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.973+0000 7f1ddbe00640 1 -- 172.21.15.31:0/1391819349 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1de80130d0 con 0x55c9e44608f0 2024-03-21T10:42:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.973+0000 7f1ddbe00640 1 -- 172.21.15.31:0/1391819349 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f1dec004b30 con 0x55c9e44608f0 2024-03-21T10:42:22.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.973+0000 7f1e03a80740 10 client.26630 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:42:22.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.973+0000 7f1e03a80740 10 client.26630 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:42:22.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.973+0000 7f1e03a80740 20 client.26630 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:22.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.973+0000 7f1e03a80740 1 --2- 172.21.15.31:0/1391819349 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c9e44797a0 0x55c9e447bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:22.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.973+0000 7f1e03a80740 4 client.26630 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:42:22.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.973+0000 7f1e03a80740 1 -- 172.21.15.31:0/1391819349 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c9e43d1d70 con 0x55c9e44797a0 2024-03-21T10:42:22.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.974+0000 7f1df9400640 1 --2- 172.21.15.31:0/1391819349 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c9e44797a0 0x55c9e447bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:22.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.974+0000 7f1df9400640 1 --2- 172.21.15.31:0/1391819349 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c9e44797a0 0x55c9e447bbc0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9477b9542ea5705 server_cookie=b8ddf86a064ad775 in_seq=0 out_seq=0 2024-03-21T10:42:22.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.974+0000 7f1ddbe00640 10 client.26630 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:22.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.975+0000 7f1ddbe00640 1 -- 172.21.15.31:0/1391819349 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c9e43d1d70 con 0x55c9e44797a0 2024-03-21T10:42:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.975+0000 7f1ddbe00640 10 client.26630 handle_command_reply: tid=0 2024-03-21T10:42:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.976+0000 7f1e03a80740 1 client.26630 shutdown 2024-03-21T10:42:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.976+0000 7f1e03a80740 1 -- 172.21.15.31:0/1391819349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9e44608f0 msgr2=0x55c9e4466c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.976+0000 7f1e03a80740 1 --2- 172.21.15.31:0/1391819349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9e44608f0 0x55c9e4466c80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1dec035000 tx=0x7f1dec005b50 comp rx=0 tx=0).stop 2024-03-21T10:42:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.976+0000 7f1e03a80740 1 -- 172.21.15.31:0/1391819349 shutdown_connections 2024-03-21T10:42:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.976+0000 7f1e03a80740 1 --2- 172.21.15.31:0/1391819349 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c9e44797a0 0x55c9e447bbc0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.976+0000 7f1e03a80740 1 --2- 172.21.15.31:0/1391819349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9e44696c0 0x55c9e44671c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.976+0000 7f1e03a80740 1 --2- 172.21.15.31:0/1391819349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9e44608f0 0x55c9e4466c80 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.976+0000 7f1e03a80740 1 --2- 172.21.15.31:0/1391819349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c9e445ffd0 0x55c9e4466740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.976+0000 7f1e03a80740 1 -- 172.21.15.31:0/1391819349 >> 172.21.15.31:0/1391819349 conn(0x55c9e445a8f0 msgr2=0x55c9e445c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.976+0000 7f1ddbe00640 0 client.26630 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.976+0000 7f1e03a80740 1 -- 172.21.15.31:0/1391819349 shutdown_connections 2024-03-21T10:42:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.976+0000 7f1e03a80740 1 -- 172.21.15.31:0/1391819349 wait complete. 2024-03-21T10:42:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.976+0000 7f1e03a80740 20 client.26630 trim_cache size 0 max 16384 2024-03-21T10:42:22.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.994+0000 7f1e03a80740 1 Processor -- start 2024-03-21T10:42:22.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.994+0000 7f1e03a80740 1 -- start start 2024-03-21T10:42:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.995+0000 7f1e03a80740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9e445ffd0 0x55c9e4468e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.995+0000 7f1e03a80740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c9e44608f0 0x55c9e44597d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.995+0000 7f1e03a80740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9e44696c0 0x55c9e4459d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.995+0000 7f1e03a80740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc0121e0 con 0x55c9e445ffd0 2024-03-21T10:42:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.995+0000 7f1e03a80740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc00fce0 con 0x55c9e44696c0 2024-03-21T10:42:22.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.995+0000 7f1e03a80740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc00dd80 con 0x55c9e44608f0 2024-03-21T10:42:22.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.995+0000 7f1df9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9e445ffd0 0x55c9e4468e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:22.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.995+0000 7f1df9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9e445ffd0 0x55c9e4468e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51610/0 (socket says 172.21.15.31:51610) 2024-03-21T10:42:22.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.995+0000 7f1df9400640 1 -- 172.21.15.31:0/2420677403 learned_addr learned my addr 172.21.15.31:0/2420677403 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:22.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.995+0000 7f1df9e00640 1 --2- 172.21.15.31:0/2420677403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c9e44608f0 0x55c9e44597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:22.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.995+0000 7f1df9400640 1 --2- 172.21.15.31:0/2420677403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9e44696c0 0x55c9e4459d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.995+0000 7f1df9400640 1 -- 172.21.15.31:0/2420677403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c9e44608f0 msgr2=0x55c9e44597d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.995+0000 7f1df9400640 1 --2- 172.21.15.31:0/2420677403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c9e44608f0 0x55c9e44597d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.996+0000 7f1df9400640 1 -- 172.21.15.31:0/2420677403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9e44696c0 msgr2=0x55c9e4459d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.996+0000 7f1df9400640 1 --2- 172.21.15.31:0/2420677403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9e44696c0 0x55c9e4459d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.996+0000 7f1df9400640 1 -- 172.21.15.31:0/2420677403 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c9e4467ae0 con 0x55c9e445ffd0 2024-03-21T10:42:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.996+0000 7f1df9400640 1 --2- 172.21.15.31:0/2420677403 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9e445ffd0 0x55c9e4468e70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1de8010010 tx=0x7f1de8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dab3d7193f126fb server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.996+0000 7f1de6a00640 1 -- 172.21.15.31:0/2420677403 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de80130d0 con 0x55c9e445ffd0 2024-03-21T10:42:22.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:22.996+0000 7fd16c600640 1 -- 172.21.15.31:0/3078760681 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1641028c0 msgr2=0x7fd164102ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:22.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:22.996+0000 7fd16c600640 1 --2- 172.21.15.31:0/3078760681 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1641028c0 0x7fd164102ca0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd15800dc90 tx=0x7fd15802fc30 comp rx=0 tx=0).stop 2024-03-21T10:42:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.996+0000 7f1de6a00640 1 -- 172.21.15.31:0/2420677403 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1de800fc10 con 0x55c9e445ffd0 2024-03-21T10:42:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.996+0000 7f1de6a00640 1 -- 172.21.15.31:0/2420677403 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de8004650 con 0x55c9e445ffd0 2024-03-21T10:42:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.997+0000 7f1e03a80740 1 -- 172.21.15.31:0/2420677403 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9e445ffd0 msgr2=0x55c9e4468e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.997+0000 7f1e03a80740 1 --2- 172.21.15.31:0/2420677403 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9e445ffd0 0x55c9e4468e70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1de8010010 tx=0x7f1de8005f90 comp rx=0 tx=0).stop 2024-03-21T10:42:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.998+0000 7f1e03a80740 1 -- 172.21.15.31:0/2420677403 shutdown_connections 2024-03-21T10:42:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.998+0000 7f1e03a80740 1 --2- 172.21.15.31:0/2420677403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9e44696c0 0x55c9e4459d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.998+0000 7f1e03a80740 1 --2- 172.21.15.31:0/2420677403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c9e44608f0 0x55c9e44597d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.998+0000 7f1e03a80740 1 --2- 172.21.15.31:0/2420677403 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9e445ffd0 0x55c9e4468e70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.998+0000 7f1e03a80740 1 -- 172.21.15.31:0/2420677403 >> 172.21.15.31:0/2420677403 conn(0x55c9e445a8f0 msgr2=0x55c9e44590c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.998+0000 7f1e03a80740 1 -- 172.21.15.31:0/2420677403 shutdown_connections 2024-03-21T10:42:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.998+0000 7f1e03a80740 1 -- 172.21.15.31:0/2420677403 wait complete. 2024-03-21T10:42:22.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:22.997+0000 7fd16c600640 1 -- 172.21.15.31:0/3078760681 shutdown_connections 2024-03-21T10:42:22.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:22.997+0000 7fd16c600640 1 --2- 172.21.15.31:0/3078760681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd16410bc90 0x7fd16410e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:22.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:22.997+0000 7fd16c600640 1 --2- 172.21.15.31:0/3078760681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1641031e0 0x7fd16410b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:22.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:22.997+0000 7fd16c600640 1 --2- 172.21.15.31:0/3078760681 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1641028c0 0x7fd164102ca0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:22.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:22.997+0000 7fd16c600640 1 -- 172.21.15.31:0/3078760681 >> 172.21.15.31:0/3078760681 conn(0x7fd1640776c0 msgr2=0x7fd164077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:22.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:22.997+0000 7fd16c600640 1 -- 172.21.15.31:0/3078760681 shutdown_connections 2024-03-21T10:42:22.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:22.997+0000 7fd16c600640 1 -- 172.21.15.31:0/3078760681 wait complete. 2024-03-21T10:42:22.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:22.998+0000 7fd16c600640 1 Processor -- start 2024-03-21T10:42:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.998+0000 7f1e03a80740 1 -- start start 2024-03-21T10:42:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:22.999+0000 7f1e03a80740 1 Processor -- start 2024-03-21T10:42:23.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:22.999+0000 7fd16c600640 1 -- start start 2024-03-21T10:42:23.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:22.999+0000 7fd16c600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1641028c0 0x7fd16410b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:22.999+0000 7fd16c600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1641031e0 0x7fd16410b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:22.999+0000 7fd16c600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd16410bc90 0x7fd1641053c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.000+0000 7fd16a600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd16410bc90 0x7fd1641053c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.000+0000 7fd16a600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd16410bc90 0x7fd1641053c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60420/0 (socket says 172.21.15.31:60420) 2024-03-21T10:42:23.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.000+0000 7fd16a600640 1 -- 172.21.15.31:0/132261269 learned_addr learned my addr 172.21.15.31:0/132261269 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:23.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.000+0000 7fd16c600640 1 -- 172.21.15.31:0/132261269 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16406b890 con 0x7fd1641028c0 2024-03-21T10:42:23.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.000+0000 7fd16c600640 1 -- 172.21.15.31:0/132261269 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16406b710 con 0x7fd1641031e0 2024-03-21T10:42:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.000+0000 7fd16c600640 1 -- 172.21.15.31:0/132261269 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16406ba10 con 0x7fd16410bc90 2024-03-21T10:42:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.000+0000 7fd169200640 1 --2- 172.21.15.31:0/132261269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1641031e0 0x7fd16410b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.001+0000 7fd169c00640 1 --2- 172.21.15.31:0/132261269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1641028c0 0x7fd16410b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.001+0000 7fd16a600640 1 -- 172.21.15.31:0/132261269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1641031e0 msgr2=0x7fd16410b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.001+0000 7fd16a600640 1 --2- 172.21.15.31:0/132261269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1641031e0 0x7fd16410b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.001+0000 7fd16a600640 1 -- 172.21.15.31:0/132261269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1641028c0 msgr2=0x7fd16410b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.001+0000 7fd16a600640 1 --2- 172.21.15.31:0/132261269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1641028c0 0x7fd16410b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.001+0000 7fd16a600640 1 -- 172.21.15.31:0/132261269 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd164105c80 con 0x7fd16410bc90 2024-03-21T10:42:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.002+0000 7fd16a600640 1 --2- 172.21.15.31:0/132261269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd16410bc90 0x7fd1641053c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd15c00e9d0 tx=0x7fd15c00ee90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.003+0000 7fd162000640 1 -- 172.21.15.31:0/132261269 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c00cd80 con 0x7fd16410bc90 2024-03-21T10:42:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.003+0000 7fd162000640 1 -- 172.21.15.31:0/132261269 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd15c005020 con 0x7fd16410bc90 2024-03-21T10:42:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.003+0000 7fd162000640 1 -- 172.21.15.31:0/132261269 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c0116d0 con 0x7fd16410bc90 2024-03-21T10:42:23.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.004+0000 7fd16c600640 1 -- 172.21.15.31:0/132261269 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd164105f10 con 0x7fd16410bc90 2024-03-21T10:42:23.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.004+0000 7fd16c600640 1 -- 172.21.15.31:0/132261269 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1641ab020 con 0x7fd16410bc90 2024-03-21T10:42:23.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.001+0000 7f1e03a80740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9e445ffd0 0x55c9e44ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.001+0000 7f1e03a80740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9e44608f0 0x55c9e44ce8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.001+0000 7f1e03a80740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c9e44696c0 0x55c9e4463590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.001+0000 7f1e03a80740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc00e380 con 0x55c9e44608f0 2024-03-21T10:42:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.001+0000 7f1e03a80740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc00f680 con 0x55c9e445ffd0 2024-03-21T10:42:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.001+0000 7f1e03a80740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc00e9c0 con 0x55c9e44696c0 2024-03-21T10:42:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.001+0000 7f1e03a80740 10 client.0 _ll_register_callbacks cb 0x7f1de8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.001+0000 7f1df9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9e445ffd0 0x55c9e44ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.002+0000 7f1df9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9e445ffd0 0x55c9e44ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33000/0 (socket says 172.21.15.31:33000) 2024-03-21T10:42:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.002+0000 7f1df9400640 1 -- 172.21.15.31:0/2003609911 learned_addr learned my addr 172.21.15.31:0/2003609911 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.002+0000 7f1df9400640 1 --2- 172.21.15.31:0/2003609911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c9e44696c0 0x55c9e4463590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.002+0000 7f1df9400640 1 -- 172.21.15.31:0/2003609911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c9e44696c0 msgr2=0x55c9e4463590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.002+0000 7f1df9400640 1 --2- 172.21.15.31:0/2003609911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c9e44696c0 0x55c9e4463590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.002+0000 7f1df9400640 1 -- 172.21.15.31:0/2003609911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9e44608f0 msgr2=0x55c9e44ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.002+0000 7f1df9400640 1 --2- 172.21.15.31:0/2003609911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9e44608f0 0x55c9e44ce8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.002+0000 7f1df9400640 1 -- 172.21.15.31:0/2003609911 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c9e4463cd0 con 0x55c9e445ffd0 2024-03-21T10:42:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.003+0000 7f1df9400640 1 --2- 172.21.15.31:0/2003609911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9e445ffd0 0x55c9e44ce360 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1de8005230 tx=0x7f1de80055b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac98083c302bd392 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.003+0000 7f1ddbe00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:42:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.003+0000 7f1ddbe00640 1 -- 172.21.15.31:0/2003609911 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de800f730 con 0x55c9e445ffd0 2024-03-21T10:42:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.003+0000 7f1ddbe00640 1 -- 172.21.15.31:0/2003609911 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f1de801c040 con 0x55c9e445ffd0 2024-03-21T10:42:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.003+0000 7f1e03a80740 1 -- 172.21.15.31:0/2003609911 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1de800fc10 con 0x55c9e445ffd0 2024-03-21T10:42:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.004+0000 7f1ddbe00640 1 -- 172.21.15.31:0/2003609911 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1de800f8d0 con 0x55c9e445ffd0 2024-03-21T10:42:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.004+0000 7f1e03a80740 10 client.26642 fetch_fsmap learned FSMap version 25 2024-03-21T10:42:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.004+0000 7f1e03a80740 1 -- 172.21.15.31:0/2003609911 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55c9e44d5b20 con 0x55c9e445ffd0 2024-03-21T10:42:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.004+0000 7f1ddbe00640 1 -- 172.21.15.31:0/2003609911 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1de800fc10 con 0x55c9e445ffd0 2024-03-21T10:42:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.004+0000 7f1ddbe00640 1 -- 172.21.15.31:0/2003609911 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f1de80115e0 con 0x55c9e445ffd0 2024-03-21T10:42:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.005+0000 7f1e03a80740 10 client.26642 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:42:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.005+0000 7f1e03a80740 10 client.26642 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:42:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.005+0000 7f1e03a80740 20 client.26642 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.005+0000 7f1e03a80740 1 --2- 172.21.15.31:0/2003609911 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c9e44d5f20 0x55c9e44d8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.005+0000 7f1e03a80740 4 client.26642 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:42:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.005+0000 7f1e03a80740 1 -- 172.21.15.31:0/2003609911 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c9e43cc0c0 con 0x55c9e44d5f20 2024-03-21T10:42:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.005+0000 7f1df9e00640 1 --2- 172.21.15.31:0/2003609911 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c9e44d5f20 0x55c9e44d8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.006+0000 7f1ddbe00640 10 client.26642 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.006+0000 7f1df9e00640 1 --2- 172.21.15.31:0/2003609911 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c9e44d5f20 0x55c9e44d8340 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e2d3bce5eae55cc server_cookie=b4da83d418a25dfb in_seq=0 out_seq=0 2024-03-21T10:42:23.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.008+0000 7fd16f3b3740 1 Processor -- start 2024-03-21T10:42:23.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.010+0000 7fd162000640 1 -- 172.21.15.31:0/132261269 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fd15c0051d0 con 0x7fd16410bc90 2024-03-21T10:42:23.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.010+0000 7fd16f3b3740 1 -- start start 2024-03-21T10:42:23.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.011+0000 7fd16f3b3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d608247080 0x55d608247460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.011+0000 7fd16f3b3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6082479a0 0x55d608250040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.011+0000 7fd16f3b3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d608250580 0x55d608252970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.011+0000 7fd16f3b3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6081abeb0 con 0x55d608247080 2024-03-21T10:42:23.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.011+0000 7fd16f3b3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d608154a60 con 0x55d608250580 2024-03-21T10:42:23.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.011+0000 7fd16f3b3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6081a7f50 con 0x55d6082479a0 2024-03-21T10:42:23.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.011+0000 7fd169c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6082479a0 0x55d608250040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.011+0000 7fd169c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6082479a0 0x55d608250040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60444/0 (socket says 172.21.15.31:60444) 2024-03-21T10:42:23.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.011+0000 7fd169c00640 1 -- 172.21.15.31:0/1896801113 learned_addr learned my addr 172.21.15.31:0/1896801113 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:23.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.012+0000 7fd169c00640 1 -- 172.21.15.31:0/1896801113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d608250580 msgr2=0x55d608252970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:42:23.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.012+0000 7fd169c00640 1 --2- 172.21.15.31:0/1896801113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d608250580 0x55d608252970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.012+0000 7fd169c00640 1 -- 172.21.15.31:0/1896801113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d608247080 msgr2=0x55d608247460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:23.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.012+0000 7fd169c00640 1 --2- 172.21.15.31:0/1896801113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d608247080 0x55d608247460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.012+0000 7fd169c00640 1 -- 172.21.15.31:0/1896801113 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d607f887e0 con 0x55d6082479a0 2024-03-21T10:42:23.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.012+0000 7fd162000640 1 --2- 172.21.15.31:0/132261269 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd14c0a3210 0x7fd14c0a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.012+0000 7fd162000640 1 -- 172.21.15.31:0/132261269 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fd15c0cfec0 con 0x7fd16410bc90 2024-03-21T10:42:23.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.013+0000 7fd169c00640 1 --2- 172.21.15.31:0/1896801113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6082479a0 0x55d608250040 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd15800dc90 tx=0x7fd15800da20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a75b877a36cb501f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:23.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.014+0000 7fd156a00640 1 -- 172.21.15.31:0/1896801113 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd158044070 con 0x55d6082479a0 2024-03-21T10:42:23.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.014+0000 7fd156a00640 1 -- 172.21.15.31:0/1896801113 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd15800d3e0 con 0x55d6082479a0 2024-03-21T10:42:23.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.014+0000 7fd156a00640 1 -- 172.21.15.31:0/1896801113 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd158030830 con 0x55d6082479a0 2024-03-21T10:42:23.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.014+0000 7fd169200640 1 --2- 172.21.15.31:0/132261269 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd14c0a3210 0x7fd14c0a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.015+0000 7fd16f3b3740 1 -- 172.21.15.31:0/1896801113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6082479a0 msgr2=0x55d608250040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:23.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.015+0000 7fd16f3b3740 1 --2- 172.21.15.31:0/1896801113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6082479a0 0x55d608250040 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd15800dc90 tx=0x7fd15800da20 comp rx=0 tx=0).stop 2024-03-21T10:42:23.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.015+0000 7fd16f3b3740 1 -- 172.21.15.31:0/1896801113 shutdown_connections 2024-03-21T10:42:23.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.015+0000 7fd16f3b3740 1 --2- 172.21.15.31:0/1896801113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d608250580 0x55d608252970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.015+0000 7fd16f3b3740 1 --2- 172.21.15.31:0/1896801113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6082479a0 0x55d608250040 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.015+0000 7fd16f3b3740 1 --2- 172.21.15.31:0/1896801113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d608247080 0x55d608247460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.015+0000 7fd16f3b3740 1 -- 172.21.15.31:0/1896801113 >> 172.21.15.31:0/1896801113 conn(0x55d6082417b0 msgr2=0x55d608241b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:23.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.016+0000 7fd16f3b3740 1 -- 172.21.15.31:0/1896801113 shutdown_connections 2024-03-21T10:42:23.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.016+0000 7fd169200640 1 --2- 172.21.15.31:0/132261269 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd14c0a3210 0x7fd14c0a56d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x55d608242940 tx=0x7fd150009820 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:23.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.016+0000 7fd16f3b3740 1 -- 172.21.15.31:0/1896801113 wait complete. 2024-03-21T10:42:23.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.016+0000 7fd16f3b3740 1 -- start start 2024-03-21T10:42:23.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.017+0000 7fd16f3b3740 1 Processor -- start 2024-03-21T10:42:23.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.017+0000 7fd16f3b3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d608247080 0x55d60824ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.017+0000 7fd16f3b3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d6082479a0 0x55d60824d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.017+0000 7fd16f3b3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d608250580 0x55d60824d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.017+0000 7fd16f3b3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d608253360 con 0x55d6082479a0 2024-03-21T10:42:23.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.017+0000 7fd16f3b3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d608167eb0 con 0x55d608247080 2024-03-21T10:42:23.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.017+0000 7fd16f3b3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6082534e0 con 0x55d608250580 2024-03-21T10:42:23.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.017+0000 7fd16f3b3740 10 client.0 _ll_register_callbacks cb 0x55d60815c150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:23.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.017+0000 7fd16a600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d608250580 0x55d60824d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.017+0000 7fd169c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d608247080 0x55d60824ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.018+0000 7fd16a600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d608250580 0x55d60824d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60454/0 (socket says 172.21.15.31:60454) 2024-03-21T10:42:23.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.018+0000 7fd16a600640 1 -- 172.21.15.31:0/3997426273 learned_addr learned my addr 172.21.15.31:0/3997426273 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:23.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.018+0000 7fd169200640 1 --2- 172.21.15.31:0/3997426273 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d6082479a0 0x55d60824d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.018+0000 7fd16a600640 1 -- 172.21.15.31:0/3997426273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d608247080 msgr2=0x55d60824ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:23.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.018+0000 7fd16a600640 1 --2- 172.21.15.31:0/3997426273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d608247080 0x55d60824ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.018+0000 7fd16a600640 1 -- 172.21.15.31:0/3997426273 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d6082479a0 msgr2=0x55d60824d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:23.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.018+0000 7fd16a600640 1 --2- 172.21.15.31:0/3997426273 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d6082479a0 0x55d60824d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.018+0000 7fd16a600640 1 -- 172.21.15.31:0/3997426273 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d607fe8f70 con 0x55d608250580 2024-03-21T10:42:23.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.018+0000 7fd169c00640 1 --2- 172.21.15.31:0/3997426273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d608247080 0x55d60824ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:42:23.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.018+0000 7fd169200640 1 --2- 172.21.15.31:0/3997426273 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d6082479a0 0x55d60824d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:42:23.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.018+0000 7fd16a600640 1 --2- 172.21.15.31:0/3997426273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d608250580 0x55d60824d8f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd15c00cfd0 tx=0x7fd15c09bf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c913e0bf4bc6203 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:23.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.018+0000 7fd14b400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:42:23.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.018+0000 7fd14b400640 1 -- 172.21.15.31:0/3997426273 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c0b4070 con 0x55d608250580 2024-03-21T10:42:23.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.019+0000 7fd14b400640 1 -- 172.21.15.31:0/3997426273 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fd15c09b3e0 con 0x55d608250580 2024-03-21T10:42:23.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.019+0000 7fd16f3b3740 1 -- 172.21.15.31:0/3997426273 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd15800d3e0 con 0x55d608250580 2024-03-21T10:42:23.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.019+0000 7fd14b400640 1 -- 172.21.15.31:0/3997426273 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd15c09b5e0 con 0x55d608250580 2024-03-21T10:42:23.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.019+0000 7fd16f3b3740 10 client.26596 fetch_fsmap learned FSMap version 25 2024-03-21T10:42:23.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.019+0000 7fd16f3b3740 1 -- 172.21.15.31:0/3997426273 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d607e80170 con 0x55d608250580 2024-03-21T10:42:23.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.019+0000 7fd14b400640 1 -- 172.21.15.31:0/3997426273 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd15800d3e0 con 0x55d608250580 2024-03-21T10:42:23.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.019+0000 7fd14b400640 1 -- 172.21.15.31:0/3997426273 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fd15c00f040 con 0x55d608250580 2024-03-21T10:42:23.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.019+0000 7fd16f3b3740 10 client.26596 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:42:23.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.019+0000 7fd16f3b3740 10 client.26596 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:42:23.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.019+0000 7fd16f3b3740 20 client.26596 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:23.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.019+0000 7fd16f3b3740 1 --2- 172.21.15.31:0/3997426273 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d608260730 0x55d608262b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.019+0000 7fd16f3b3740 4 client.26596 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:42:23.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.019+0000 7fd16f3b3740 1 -- 172.21.15.31:0/3997426273 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d6081b93f0 con 0x55d608260730 2024-03-21T10:42:23.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.020+0000 7fd169c00640 1 --2- 172.21.15.31:0/3997426273 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d608260730 0x55d608262b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.020+0000 7fd169c00640 1 --2- 172.21.15.31:0/3997426273 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d608260730 0x55d608262b90 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca7c22650d7f0062 server_cookie=7e2fbef0e8844e58 in_seq=0 out_seq=0 2024-03-21T10:42:23.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.020+0000 7fd14b400640 10 client.26596 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.020+0000 7f1ddbe00640 1 -- 172.21.15.31:0/2003609911 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c9e43cc0c0 con 0x55c9e44d5f20 2024-03-21T10:42:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.020+0000 7f1ddbe00640 10 client.26642 handle_command_reply: tid=0 2024-03-21T10:42:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.020+0000 7f1e03a80740 1 client.26642 shutdown 2024-03-21T10:42:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.021+0000 7f1e03a80740 1 -- 172.21.15.31:0/2003609911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9e445ffd0 msgr2=0x55c9e44ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.021+0000 7f1e03a80740 1 --2- 172.21.15.31:0/2003609911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9e445ffd0 0x55c9e44ce360 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1de8005230 tx=0x7f1de80055b0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.021+0000 7f1e03a80740 1 -- 172.21.15.31:0/2003609911 shutdown_connections 2024-03-21T10:42:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.021+0000 7f1e03a80740 1 --2- 172.21.15.31:0/2003609911 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c9e44d5f20 0x55c9e44d8340 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.021+0000 7f1e03a80740 1 --2- 172.21.15.31:0/2003609911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c9e44696c0 0x55c9e4463590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.021+0000 7f1e03a80740 1 --2- 172.21.15.31:0/2003609911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9e44608f0 0x55c9e44ce8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.021+0000 7f1e03a80740 1 --2- 172.21.15.31:0/2003609911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9e445ffd0 0x55c9e44ce360 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.021+0000 7f1e03a80740 1 -- 172.21.15.31:0/2003609911 >> 172.21.15.31:0/2003609911 conn(0x55c9e445a8f0 msgr2=0x55c9e4466f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.021+0000 7f1ddbe00640 0 client.26642 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:23.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.021+0000 7fd14b400640 1 -- 172.21.15.31:0/3997426273 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d6081b93f0 con 0x55d608260730 2024-03-21T10:42:23.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.021+0000 7fd14b400640 10 client.26596 handle_command_reply: tid=0 2024-03-21T10:42:23.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.021+0000 7fd16f3b3740 1 client.26596 shutdown 2024-03-21T10:42:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.021+0000 7f1e03a80740 1 -- 172.21.15.31:0/2003609911 shutdown_connections 2024-03-21T10:42:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.021+0000 7f1e03a80740 1 -- 172.21.15.31:0/2003609911 wait complete. 2024-03-21T10:42:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.021+0000 7f1e03a80740 20 client.26642 trim_cache size 0 max 16384 2024-03-21T10:42:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:42:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:42:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:42:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:42:23.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.022+0000 7fd16f3b3740 1 -- 172.21.15.31:0/3997426273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d608250580 msgr2=0x55d60824d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:23.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.022+0000 7fd16f3b3740 1 --2- 172.21.15.31:0/3997426273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d608250580 0x55d60824d8f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd15c00cfd0 tx=0x7fd15c09bf90 comp rx=0 tx=0).stop 2024-03-21T10:42:23.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.022+0000 7fd16f3b3740 1 -- 172.21.15.31:0/3997426273 shutdown_connections 2024-03-21T10:42:23.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.022+0000 7fd16f3b3740 1 --2- 172.21.15.31:0/3997426273 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d608260730 0x55d608262b90 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.022+0000 7fd16f3b3740 1 --2- 172.21.15.31:0/3997426273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d608250580 0x55d60824d8f0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.022+0000 7fd16f3b3740 1 --2- 172.21.15.31:0/3997426273 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d6082479a0 0x55d60824d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.022+0000 7fd16f3b3740 1 --2- 172.21.15.31:0/3997426273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d608247080 0x55d60824ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.022+0000 7fd16f3b3740 1 -- 172.21.15.31:0/3997426273 >> 172.21.15.31:0/3997426273 conn(0x55d6082417b0 msgr2=0x55d608243500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:23.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.022+0000 7fd14b400640 0 client.26596 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:23.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.022+0000 7fd16f3b3740 1 -- 172.21.15.31:0/3997426273 shutdown_connections 2024-03-21T10:42:23.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.022+0000 7fd16f3b3740 1 -- 172.21.15.31:0/3997426273 wait complete. 2024-03-21T10:42:23.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.022+0000 7fd16f3b3740 20 client.26596 trim_cache size 0 max 16384 2024-03-21T10:42:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.022+0000 7f1e00e00640 1 -- 172.21.15.31:0/3413815713 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f1de00a3230 msgr2=0x7f1de00a56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.022+0000 7f1e00e00640 1 --2- 172.21.15.31:0/3413815713 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f1de00a3230 0x7f1de00a56f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1df4009fd0 tx=0x7f1df400c860 comp rx=0 tx=0).stop 2024-03-21T10:42:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.022+0000 7f1e00e00640 1 -- 172.21.15.31:0/3413815713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dfc1147f0 msgr2=0x7f1dfc10dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.022+0000 7f1e00e00640 1 --2- 172.21.15.31:0/3413815713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dfc1147f0 0x7f1dfc10dea0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1df4005fd0 tx=0x7f1df400ef10 comp rx=0 tx=0).stop 2024-03-21T10:42:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.023+0000 7f1e00e00640 1 -- 172.21.15.31:0/3413815713 shutdown_connections 2024-03-21T10:42:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.023+0000 7f1e00e00640 1 --2- 172.21.15.31:0/3413815713 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f1de00a3230 0x7f1de00a56f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.023+0000 7f1e00e00640 1 --2- 172.21.15.31:0/3413815713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dfc1147f0 0x7f1dfc10dea0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.023+0000 7f1e00e00640 1 --2- 172.21.15.31:0/3413815713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1dfc1091a0 0x7f1dfc114200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.023+0000 7f1e00e00640 1 --2- 172.21.15.31:0/3413815713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dfc108880 0x7f1dfc113cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.023+0000 7f1e00e00640 1 -- 172.21.15.31:0/3413815713 >> 172.21.15.31:0/3413815713 conn(0x7f1dfc06c750 msgr2=0x7f1dfc0702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.023+0000 7f1e00e00640 1 -- 172.21.15.31:0/3413815713 shutdown_connections 2024-03-21T10:42:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:23.023+0000 7f1e00e00640 1 -- 172.21.15.31:0/3413815713 wait complete. 2024-03-21T10:42:23.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.040+0000 7fd16f3b3740 1 Processor -- start 2024-03-21T10:42:23.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.049+0000 7fd16f3b3740 1 -- start start 2024-03-21T10:42:23.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.049+0000 7fd16f3b3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d608247080 0x55d60824fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.049+0000 7fd16f3b3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d6082479a0 0x55d60824ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.049+0000 7fd16f3b3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d608250580 0x55d60825bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.049+0000 7fd16f3b3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1641c8250 con 0x55d6082479a0 2024-03-21T10:42:23.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.049+0000 7fd16f3b3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1641c5d50 con 0x55d608247080 2024-03-21T10:42:23.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.049+0000 7fd16f3b3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1641c3df0 con 0x55d608250580 2024-03-21T10:42:23.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.050+0000 7fd16a600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d608250580 0x55d60825bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.050+0000 7fd16a600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d608250580 0x55d60825bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60468/0 (socket says 172.21.15.31:60468) 2024-03-21T10:42:23.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.051+0000 7fd169c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d608247080 0x55d60824fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.051+0000 7fd169c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d608247080 0x55d60824fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33012/0 (socket says 172.21.15.31:33012) 2024-03-21T10:42:23.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.051+0000 7fd169c00640 1 -- 172.21.15.31:0/1442719461 learned_addr learned my addr 172.21.15.31:0/1442719461 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:23.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.052+0000 7fd169c00640 1 -- 172.21.15.31:0/1442719461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d608250580 msgr2=0x55d60825bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:23.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.052+0000 7fd169c00640 1 --2- 172.21.15.31:0/1442719461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d608250580 0x55d60825bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.052+0000 7fd169c00640 1 -- 172.21.15.31:0/1442719461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d6082479a0 msgr2=0x55d60824ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:23.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.052+0000 7fd169c00640 1 --2- 172.21.15.31:0/1442719461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d6082479a0 0x55d60824ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.052+0000 7fd169c00640 1 -- 172.21.15.31:0/1442719461 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d60823cc20 con 0x55d608247080 2024-03-21T10:42:23.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.054+0000 7fd169c00640 1 --2- 172.21.15.31:0/1442719461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d608247080 0x55d60824fa10 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fd158003d20 tx=0x7fd15802fc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26bf2539034d9885 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:23.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.055+0000 7fd156000640 1 -- 172.21.15.31:0/1442719461 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd158044070 con 0x55d608247080 2024-03-21T10:42:23.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.056+0000 7fd156000640 1 -- 172.21.15.31:0/1442719461 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd158045040 con 0x55d608247080 2024-03-21T10:42:23.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.056+0000 7fd156000640 1 -- 172.21.15.31:0/1442719461 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15804db10 con 0x55d608247080 2024-03-21T10:42:23.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:22 smithi031 ceph-mon[31673]: pgmap v117: 129 pgs: 129 active+clean; 18 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 209 MiB/s wr, 1.66k op/s 2024-03-21T10:42:23.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.060+0000 7fd16f3b3740 1 -- 172.21.15.31:0/1442719461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d608247080 msgr2=0x55d60824fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:23.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.061+0000 7fd16f3b3740 1 --2- 172.21.15.31:0/1442719461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d608247080 0x55d60824fa10 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fd158003d20 tx=0x7fd15802fc30 comp rx=0 tx=0).stop 2024-03-21T10:42:23.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.062+0000 7fd16f3b3740 1 -- 172.21.15.31:0/1442719461 shutdown_connections 2024-03-21T10:42:23.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.062+0000 7fd16f3b3740 1 --2- 172.21.15.31:0/1442719461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d608250580 0x55d60825bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.064+0000 7fd16f3b3740 1 --2- 172.21.15.31:0/1442719461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d6082479a0 0x55d60824ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.064+0000 7fd16f3b3740 1 --2- 172.21.15.31:0/1442719461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d608247080 0x55d60824fa10 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.064+0000 7fd16f3b3740 1 -- 172.21.15.31:0/1442719461 >> 172.21.15.31:0/1442719461 conn(0x55d6082417b0 msgr2=0x55d608240830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:23.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.064+0000 7fd16f3b3740 1 -- 172.21.15.31:0/1442719461 shutdown_connections 2024-03-21T10:42:23.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.064+0000 7fd16f3b3740 1 -- 172.21.15.31:0/1442719461 wait complete. 2024-03-21T10:42:23.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.065+0000 7fd16f3b3740 1 -- start start 2024-03-21T10:42:23.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.065+0000 7fd16f3b3740 1 Processor -- start 2024-03-21T10:42:23.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.067+0000 7fd16f3b3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d608247080 0x55d608257e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.067+0000 7fd16f3b3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6082479a0 0x55d608258370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.067+0000 7fd16f3b3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d608250580 0x55d608254300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.070+0000 7fd169200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6082479a0 0x55d608258370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.070+0000 7fd169200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6082479a0 0x55d608258370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60476/0 (socket says 172.21.15.31:60476) 2024-03-21T10:42:23.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.070+0000 7fd169200640 1 -- 172.21.15.31:0/3457226309 learned_addr learned my addr 172.21.15.31:0/3457226309 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:23.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.071+0000 7fd16a600640 1 --2- 172.21.15.31:0/3457226309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d608250580 0x55d608254300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.071+0000 7fd16f3b3740 1 -- 172.21.15.31:0/3457226309 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1641c43f0 con 0x55d608247080 2024-03-21T10:42:23.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.071+0000 7fd16f3b3740 1 -- 172.21.15.31:0/3457226309 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1641c56f0 con 0x55d608250580 2024-03-21T10:42:23.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.072+0000 7fd16f3b3740 1 -- 172.21.15.31:0/3457226309 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1641c4a30 con 0x55d6082479a0 2024-03-21T10:42:23.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.073+0000 7fd16f3b3740 10 client.0 _ll_register_callbacks cb 0x7fd15c098230 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:23.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.073+0000 7fd169200640 1 -- 172.21.15.31:0/3457226309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d608250580 msgr2=0x55d608254300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:23.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.074+0000 7fd169200640 1 --2- 172.21.15.31:0/3457226309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d608250580 0x55d608254300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.074+0000 7fd169c00640 1 --2- 172.21.15.31:0/3457226309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d608247080 0x55d608257e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.074+0000 7fd169200640 1 -- 172.21.15.31:0/3457226309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d608247080 msgr2=0x55d608257e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:23.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.074+0000 7fd169200640 1 --2- 172.21.15.31:0/3457226309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d608247080 0x55d608257e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.074+0000 7fd169200640 1 -- 172.21.15.31:0/3457226309 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d608254a90 con 0x55d6082479a0 2024-03-21T10:42:23.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.074+0000 7fd16a600640 1 --2- 172.21.15.31:0/3457226309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d608250580 0x55d608254300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:42:23.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.075+0000 7fd169c00640 1 --2- 172.21.15.31:0/3457226309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d608247080 0x55d608257e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:42:23.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:22 smithi070 ceph-mon[36330]: pgmap v117: 129 pgs: 129 active+clean; 18 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 209 MiB/s wr, 1.66k op/s 2024-03-21T10:42:23.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.077+0000 7fd169200640 1 --2- 172.21.15.31:0/3457226309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6082479a0 0x55d608258370 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd150004700 tx=0x7fd150004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb17c2b29d836261 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:23.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.078+0000 7fd14b400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:42:23.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.078+0000 7fd14b400640 1 -- 172.21.15.31:0/3457226309 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd150005220 con 0x55d6082479a0 2024-03-21T10:42:23.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.078+0000 7fd14b400640 1 -- 172.21.15.31:0/3457226309 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fd1500053c0 con 0x55d6082479a0 2024-03-21T10:42:23.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.085+0000 7fd16f3b3740 1 -- 172.21.15.31:0/3457226309 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd158045040 con 0x55d6082479a0 2024-03-21T10:42:23.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.087+0000 7fd14b400640 1 -- 172.21.15.31:0/3457226309 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1500055c0 con 0x55d6082479a0 2024-03-21T10:42:23.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.087+0000 7fd16f3b3740 10 client.26602 fetch_fsmap learned FSMap version 25 2024-03-21T10:42:23.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.088+0000 7fd16f3b3740 1 -- 172.21.15.31:0/3457226309 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d6082bcb40 con 0x55d6082479a0 2024-03-21T10:42:23.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.090+0000 7fd14b400640 1 -- 172.21.15.31:0/3457226309 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd158045040 con 0x55d6082479a0 2024-03-21T10:42:23.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.090+0000 7fd14b400640 1 -- 172.21.15.31:0/3457226309 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fd15001ccc0 con 0x55d6082479a0 2024-03-21T10:42:23.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.096+0000 7fd16f3b3740 10 client.26602 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:42:23.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.096+0000 7fd16f3b3740 10 client.26602 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:42:23.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.096+0000 7fd16f3b3740 20 client.26602 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:23.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.096+0000 7fd16f3b3740 1 --2- 172.21.15.31:0/3457226309 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d6082bd200 0x55d6082bf620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.096+0000 7fd16f3b3740 4 client.26602 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-21T10:42:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.096+0000 7fd16f3b3740 1 -- 172.21.15.31:0/3457226309 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55d6081b3740 con 0x55d6082bd200 2024-03-21T10:42:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.096+0000 7fd169c00640 1 --2- 172.21.15.31:0/3457226309 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d6082bd200 0x55d6082bf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.101+0000 7fd169c00640 1 --2- 172.21.15.31:0/3457226309 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d6082bd200 0x55d6082bf620 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ec67badb6aa8dfd server_cookie=65de4998b1ec0051 in_seq=0 out_seq=0 2024-03-21T10:42:23.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.101+0000 7fd14b400640 10 client.26602 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:23.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.103+0000 7fd14b400640 1 -- 172.21.15.31:0/3457226309 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55d6081b3740 con 0x55d6082bd200 2024-03-21T10:42:23.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.103+0000 7fd14b400640 10 client.26602 handle_command_reply: tid=0 2024-03-21T10:42:23.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.103+0000 7fd16f3b3740 1 client.26602 shutdown 2024-03-21T10:42:23.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.108+0000 7fd16f3b3740 1 -- 172.21.15.31:0/3457226309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6082479a0 msgr2=0x55d608258370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:23.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.108+0000 7fd16f3b3740 1 --2- 172.21.15.31:0/3457226309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6082479a0 0x55d608258370 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd150004700 tx=0x7fd150004730 comp rx=0 tx=0).stop 2024-03-21T10:42:23.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.113+0000 7fd16f3b3740 1 -- 172.21.15.31:0/3457226309 shutdown_connections 2024-03-21T10:42:23.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.113+0000 7fd16f3b3740 1 --2- 172.21.15.31:0/3457226309 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d6082bd200 0x55d6082bf620 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.113+0000 7fd16f3b3740 1 --2- 172.21.15.31:0/3457226309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d608250580 0x55d608254300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.113+0000 7fd16f3b3740 1 --2- 172.21.15.31:0/3457226309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d6082479a0 0x55d608258370 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.113+0000 7fd16f3b3740 1 --2- 172.21.15.31:0/3457226309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d608247080 0x55d608257e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.113+0000 7fd16f3b3740 1 -- 172.21.15.31:0/3457226309 >> 172.21.15.31:0/3457226309 conn(0x55d6082417b0 msgr2=0x55d60823ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:23.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.119+0000 7fd14b400640 0 client.26602 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:23.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.119+0000 7fd16f3b3740 1 -- 172.21.15.31:0/3457226309 shutdown_connections 2024-03-21T10:42:23.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.120+0000 7fd16f3b3740 1 -- 172.21.15.31:0/3457226309 wait complete. 2024-03-21T10:42:23.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.120+0000 7fd16f3b3740 20 client.26602 trim_cache size 0 max 16384 2024-03-21T10:42:23.118 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:42:23.118 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-03-21T10:42:23.118 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "7518468c-2861-4d4e-be5a-9dec219c3b76", 2024-03-21T10:42:23.118 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-03-21T10:42:23.118 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:42:23.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.123+0000 7fd16c600640 1 -- 172.21.15.31:0/132261269 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd14c0a3210 msgr2=0x7fd14c0a56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:23.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.123+0000 7fd16c600640 1 --2- 172.21.15.31:0/132261269 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd14c0a3210 0x7fd14c0a56d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x55d608242940 tx=0x7fd150009820 comp rx=0 tx=0).stop 2024-03-21T10:42:23.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.123+0000 7fd16c600640 1 -- 172.21.15.31:0/132261269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd16410bc90 msgr2=0x7fd1641053c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:23.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.123+0000 7fd16c600640 1 --2- 172.21.15.31:0/132261269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd16410bc90 0x7fd1641053c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd15c00e9d0 tx=0x7fd15c00ee90 comp rx=0 tx=0).stop 2024-03-21T10:42:23.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.139+0000 7fd16c600640 1 -- 172.21.15.31:0/132261269 shutdown_connections 2024-03-21T10:42:23.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.139+0000 7fd16c600640 1 --2- 172.21.15.31:0/132261269 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd14c0a3210 0x7fd14c0a56d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.139+0000 7fd16c600640 1 --2- 172.21.15.31:0/132261269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd16410bc90 0x7fd1641053c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.139+0000 7fd16c600640 1 --2- 172.21.15.31:0/132261269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1641031e0 0x7fd16410b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.139+0000 7fd16c600640 1 --2- 172.21.15.31:0/132261269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1641028c0 0x7fd16410b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.139+0000 7fd16c600640 1 -- 172.21.15.31:0/132261269 >> 172.21.15.31:0/132261269 conn(0x7fd1640776c0 msgr2=0x7fd1640fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:23.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.146+0000 7fd16c600640 1 -- 172.21.15.31:0/132261269 shutdown_connections 2024-03-21T10:42:23.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.148+0000 7fd16c600640 1 -- 172.21.15.31:0/132261269 wait complete. 2024-03-21T10:42:23.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:22 smithi040 ceph-mon[36154]: pgmap v117: 129 pgs: 129 active+clean; 18 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 209 MiB/s wr, 1.66k op/s 2024-03-21T10:42:23.196 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:42:23.196 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:42:23.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.595+0000 7f88bc600640 1 -- 172.21.15.31:0/2277154917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88b4104220 msgr2=0x7f88b4110730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:23.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.595+0000 7f88bc600640 1 --2- 172.21.15.31:0/2277154917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88b4104220 0x7f88b4110730 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f88ac009cd0 tx=0x7f88ac02c0f0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.599+0000 7f88bc600640 1 -- 172.21.15.31:0/2277154917 shutdown_connections 2024-03-21T10:42:23.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.599+0000 7f88bc600640 1 --2- 172.21.15.31:0/2277154917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88b4104220 0x7f88b4110730 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.599+0000 7f88bc600640 1 --2- 172.21.15.31:0/2277154917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88b4103860 0x7f88b4103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.599+0000 7f88bc600640 1 --2- 172.21.15.31:0/2277154917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88b4109870 0x7f88b4109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.599+0000 7f88bc600640 1 -- 172.21.15.31:0/2277154917 >> 172.21.15.31:0/2277154917 conn(0x7f88b40ff310 msgr2=0x7f88b4101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:23.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.600+0000 7f88bc600640 1 -- 172.21.15.31:0/2277154917 shutdown_connections 2024-03-21T10:42:23.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.600+0000 7f88bc600640 1 -- 172.21.15.31:0/2277154917 wait complete. 2024-03-21T10:42:23.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.601+0000 7f88bc600640 1 Processor -- start 2024-03-21T10:42:23.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.602+0000 7f88bc600640 1 -- start start 2024-03-21T10:42:23.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.603+0000 7f88bc600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88b4103860 0x7f88b410f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.603+0000 7f88bc600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88b4104220 0x7f88b410f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.603+0000 7f88bc600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88b4109870 0x7f88b410fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.603+0000 7f88b9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88b4104220 0x7f88b410f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.603+0000 7f88b9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88b4104220 0x7f88b410f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33044/0 (socket says 172.21.15.31:33044) 2024-03-21T10:42:23.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.603+0000 7f88b9c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88b4103860 0x7f88b410f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.603+0000 7f88b9c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88b4103860 0x7f88b410f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60488/0 (socket says 172.21.15.31:60488) 2024-03-21T10:42:23.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.604+0000 7f88b9200640 1 -- 172.21.15.31:0/363600413 learned_addr learned my addr 172.21.15.31:0/363600413 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:23.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.604+0000 7f88bc600640 1 -- 172.21.15.31:0/363600413 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b4076da0 con 0x7f88b4109870 2024-03-21T10:42:23.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.604+0000 7f88bc600640 1 -- 172.21.15.31:0/363600413 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b4076c20 con 0x7f88b4104220 2024-03-21T10:42:23.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.604+0000 7f88bc600640 1 -- 172.21.15.31:0/363600413 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b4076f20 con 0x7f88b4103860 2024-03-21T10:42:23.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.604+0000 7f88ba600640 1 --2- 172.21.15.31:0/363600413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88b4109870 0x7f88b410fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.604+0000 7f88b9200640 1 -- 172.21.15.31:0/363600413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88b4103860 msgr2=0x7f88b410f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:23.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.604+0000 7f88b9200640 1 --2- 172.21.15.31:0/363600413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88b4103860 0x7f88b410f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.605+0000 7f88b9200640 1 -- 172.21.15.31:0/363600413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88b4109870 msgr2=0x7f88b410fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:23.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.605+0000 7f88b9200640 1 --2- 172.21.15.31:0/363600413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88b4109870 0x7f88b410fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.605+0000 7f88b9200640 1 -- 172.21.15.31:0/363600413 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b41b2f90 con 0x7f88b4104220 2024-03-21T10:42:23.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.606+0000 7f88b9200640 1 --2- 172.21.15.31:0/363600413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88b4104220 0x7f88b410f640 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f88a000e990 tx=0x7f88a000ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:23.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.606+0000 7f88b2000640 1 -- 172.21.15.31:0/363600413 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a000cbe0 con 0x7f88b4104220 2024-03-21T10:42:23.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.607+0000 7f88b2000640 1 -- 172.21.15.31:0/363600413 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88a000cd80 con 0x7f88b4104220 2024-03-21T10:42:23.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.607+0000 7f88bc600640 1 -- 172.21.15.31:0/363600413 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b41b3280 con 0x7f88b4104220 2024-03-21T10:42:23.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.607+0000 7f88b2000640 1 -- 172.21.15.31:0/363600413 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a0011470 con 0x7f88b4104220 2024-03-21T10:42:23.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.607+0000 7f88bc600640 1 -- 172.21.15.31:0/363600413 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b41b37d0 con 0x7f88b4104220 2024-03-21T10:42:23.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.613+0000 7f88bf379740 1 Processor -- start 2024-03-21T10:42:23.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.616+0000 7f88bf379740 1 -- start start 2024-03-21T10:42:23.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.616+0000 7f88bf379740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ba7f78c00 0x555ba7f77950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.616+0000 7f88bf379740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ba7f77f20 0x555ba7f783a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.616+0000 7f88bf379740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ba7f922a0 0x555ba7f94690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.616+0000 7f88bf379740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ba7edc250 con 0x555ba7f922a0 2024-03-21T10:42:23.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.616+0000 7f88bf379740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ba7e84a30 con 0x555ba7f78c00 2024-03-21T10:42:23.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.616+0000 7f88bf379740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ba7ed82f0 con 0x555ba7f77f20 2024-03-21T10:42:23.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.617+0000 7f88b9200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ba7f922a0 0x555ba7f94690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.618+0000 7f88b2000640 1 -- 172.21.15.31:0/363600413 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f88a00046c0 con 0x7f88b4104220 2024-03-21T10:42:23.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.618+0000 7f88b9200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ba7f922a0 0x555ba7f94690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51680/0 (socket says 172.21.15.31:51680) 2024-03-21T10:42:23.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.618+0000 7f88b9200640 1 -- 172.21.15.31:0/2115203522 learned_addr learned my addr 172.21.15.31:0/2115203522 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:23.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.619+0000 7f88b9200640 1 -- 172.21.15.31:0/2115203522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ba7f77f20 msgr2=0x555ba7f783a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:42:23.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.619+0000 7f88b9200640 1 --2- 172.21.15.31:0/2115203522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ba7f77f20 0x555ba7f783a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.619+0000 7f88b9200640 1 -- 172.21.15.31:0/2115203522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ba7f78c00 msgr2=0x555ba7f77950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:42:23.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.619+0000 7f88b9200640 1 --2- 172.21.15.31:0/2115203522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ba7f78c00 0x555ba7f77950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.619+0000 7f88b9200640 1 -- 172.21.15.31:0/2115203522 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ba7e6f690 con 0x555ba7f922a0 2024-03-21T10:42:23.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.620+0000 7f88b9200640 1 --2- 172.21.15.31:0/2115203522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ba7f922a0 0x555ba7f94690 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f88a00cceb0 tx=0x7f88a00d86c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5915304097d2561e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:23.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.621+0000 7f88a6a00640 1 -- 172.21.15.31:0/2115203522 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a00df070 con 0x555ba7f922a0 2024-03-21T10:42:23.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.621+0000 7f88a6a00640 1 -- 172.21.15.31:0/2115203522 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88a00db950 con 0x555ba7f922a0 2024-03-21T10:42:23.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.621+0000 7f88a6a00640 1 -- 172.21.15.31:0/2115203522 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a00da8f0 con 0x555ba7f922a0 2024-03-21T10:42:23.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.622+0000 7f88bf379740 1 -- 172.21.15.31:0/2115203522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ba7f922a0 msgr2=0x555ba7f94690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:23.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.622+0000 7f88bf379740 1 --2- 172.21.15.31:0/2115203522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ba7f922a0 0x555ba7f94690 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f88a00cceb0 tx=0x7f88a00d86c0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.625+0000 7f88b2000640 1 --2- 172.21.15.31:0/363600413 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f889c0a3290 0x7f889c0a5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.625+0000 7f88b2000640 1 -- 172.21.15.31:0/363600413 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f88a00cfbc0 con 0x7f88b4104220 2024-03-21T10:42:23.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.624+0000 7f88bf379740 1 -- 172.21.15.31:0/2115203522 shutdown_connections 2024-03-21T10:42:23.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.626+0000 7f88bf379740 1 --2- 172.21.15.31:0/2115203522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ba7f922a0 0x555ba7f94690 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.627+0000 7f88bf379740 1 --2- 172.21.15.31:0/2115203522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ba7f77f20 0x555ba7f783a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.627+0000 7f88bf379740 1 --2- 172.21.15.31:0/2115203522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ba7f78c00 0x555ba7f77950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.627+0000 7f88bf379740 1 -- 172.21.15.31:0/2115203522 >> 172.21.15.31:0/2115203522 conn(0x555ba7f3f200 msgr2=0x555ba7f72970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:23.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.627+0000 7f88ba600640 1 --2- 172.21.15.31:0/363600413 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f889c0a3290 0x7f889c0a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.628+0000 7f88bf379740 1 -- 172.21.15.31:0/2115203522 shutdown_connections 2024-03-21T10:42:23.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.628+0000 7f88bf379740 1 -- 172.21.15.31:0/2115203522 wait complete. 2024-03-21T10:42:23.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.629+0000 7f88bf379740 1 -- start start 2024-03-21T10:42:23.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.630+0000 7f88bf379740 1 Processor -- start 2024-03-21T10:42:23.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.638+0000 7f88bf379740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ba7f77f20 0x555ba7f919f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.638+0000 7f88bf379740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ba7f78c00 0x555ba7f8ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.638+0000 7f88bf379740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ba7f922a0 0x555ba7f8f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.638+0000 7f88b9c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ba7f77f20 0x555ba7f919f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.638+0000 7f88b9c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ba7f77f20 0x555ba7f919f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60502/0 (socket says 172.21.15.31:60502) 2024-03-21T10:42:23.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.638+0000 7f88b9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ba7f78c00 0x555ba7f8ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.638+0000 7f88b9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ba7f78c00 0x555ba7f8ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33060/0 (socket says 172.21.15.31:33060) 2024-03-21T10:42:23.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.640+0000 7f88bf379740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ba7f95110 con 0x555ba7f922a0 2024-03-21T10:42:23.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.640+0000 7f88bf379740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ba7e97e80 con 0x555ba7f78c00 2024-03-21T10:42:23.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.640+0000 7f88bf379740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ba7f95290 con 0x555ba7f77f20 2024-03-21T10:42:23.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.640+0000 7f88b9c00640 1 -- 172.21.15.31:0/3267802639 learned_addr learned my addr 172.21.15.31:0/3267802639 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:23.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.640+0000 7f88bf379740 10 client.0 _ll_register_callbacks cb 0x555ba7eaa130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:23.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.640+0000 7f88b9200640 1 -- 172.21.15.31:0/3267802639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ba7f77f20 msgr2=0x555ba7f919f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:23.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.640+0000 7f88b9200640 1 --2- 172.21.15.31:0/3267802639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ba7f77f20 0x555ba7f919f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.640+0000 7f88b9200640 1 -- 172.21.15.31:0/3267802639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ba7f922a0 msgr2=0x555ba7f8f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:23.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.640+0000 7f88b9200640 1 --2- 172.21.15.31:0/3267802639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ba7f922a0 0x555ba7f8f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.640+0000 7f88b9200640 1 -- 172.21.15.31:0/3267802639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555ba7cba7d0 con 0x555ba7f78c00 2024-03-21T10:42:23.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.641+0000 7f88ba600640 1 --2- 172.21.15.31:0/363600413 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f889c0a3290 0x7f889c0a5750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f88ac037040 tx=0x7f88ac009860 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:23.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.642+0000 7f88b9200640 1 --2- 172.21.15.31:0/3267802639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ba7f78c00 0x555ba7f8ed80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f88a000c640 tx=0x7f88a0005700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=477e6700d9405793 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:23.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.642+0000 7f889b400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:42:23.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.643+0000 7f889b400640 1 -- 172.21.15.31:0/3267802639 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a001f070 con 0x555ba7f78c00 2024-03-21T10:42:23.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.643+0000 7f889b400640 1 -- 172.21.15.31:0/3267802639 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f88a00cd1f0 con 0x555ba7f78c00 2024-03-21T10:42:23.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.644+0000 7f88bf379740 1 -- 172.21.15.31:0/3267802639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f88a00db950 con 0x555ba7f78c00 2024-03-21T10:42:23.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.644+0000 7f889b400640 1 -- 172.21.15.31:0/3267802639 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88a00df040 con 0x555ba7f78c00 2024-03-21T10:42:23.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.644+0000 7f88bf379740 10 client.26660 fetch_fsmap learned FSMap version 25 2024-03-21T10:42:23.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.644+0000 7f88bf379740 1 -- 172.21.15.31:0/3267802639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x555ba7bf03f0 con 0x555ba7f78c00 2024-03-21T10:42:23.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.647+0000 7f889b400640 1 -- 172.21.15.31:0/3267802639 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88a00db950 con 0x555ba7f78c00 2024-03-21T10:42:23.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.648+0000 7f889b400640 1 -- 172.21.15.31:0/3267802639 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f88a00d9040 con 0x555ba7f78c00 2024-03-21T10:42:23.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.649+0000 7f88bf379740 10 client.26660 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:42:23.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.649+0000 7f88bf379740 10 client.26660 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:42:23.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.649+0000 7f88bf379740 20 client.26660 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:23.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.649+0000 7f88bf379740 1 --2- 172.21.15.31:0/3267802639 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x555ba7fa2210 0x555ba7fa4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.649+0000 7f88bf379740 4 client.26660 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:42:23.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.649+0000 7f88bf379740 1 -- 172.21.15.31:0/3267802639 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555ba7ee9790 con 0x555ba7fa2210 2024-03-21T10:42:23.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.650+0000 7f88b9c00640 1 --2- 172.21.15.31:0/3267802639 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x555ba7fa2210 0x555ba7fa4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.651+0000 7f88b9c00640 1 --2- 172.21.15.31:0/3267802639 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x555ba7fa2210 0x555ba7fa4600 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f98e0ccb539fa771 server_cookie=3da66cf2fbdc99c7 in_seq=0 out_seq=0 2024-03-21T10:42:23.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.651+0000 7f889b400640 10 client.26660 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:23.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.652+0000 7f889b400640 1 -- 172.21.15.31:0/3267802639 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555ba7ee9790 con 0x555ba7fa2210 2024-03-21T10:42:23.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.653+0000 7f889b400640 10 client.26660 handle_command_reply: tid=0 2024-03-21T10:42:23.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.653+0000 7f88bf379740 1 client.26660 shutdown 2024-03-21T10:42:23.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.653+0000 7f88bf379740 1 -- 172.21.15.31:0/3267802639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ba7f78c00 msgr2=0x555ba7f8ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:23.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.653+0000 7f88bf379740 1 --2- 172.21.15.31:0/3267802639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ba7f78c00 0x555ba7f8ed80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f88a000c640 tx=0x7f88a0005700 comp rx=0 tx=0).stop 2024-03-21T10:42:23.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.653+0000 7f88bf379740 1 -- 172.21.15.31:0/3267802639 shutdown_connections 2024-03-21T10:42:23.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.653+0000 7f88bf379740 1 --2- 172.21.15.31:0/3267802639 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x555ba7fa2210 0x555ba7fa4600 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.653+0000 7f88bf379740 1 --2- 172.21.15.31:0/3267802639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ba7f922a0 0x555ba7f8f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.654+0000 7f88bf379740 1 --2- 172.21.15.31:0/3267802639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ba7f78c00 0x555ba7f8ed80 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.654+0000 7f88bf379740 1 --2- 172.21.15.31:0/3267802639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ba7f77f20 0x555ba7f919f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.654+0000 7f889b400640 0 client.26660 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:23.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.654+0000 7f88bf379740 1 -- 172.21.15.31:0/3267802639 >> 172.21.15.31:0/3267802639 conn(0x555ba7f3f200 msgr2=0x555ba7f746c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:23.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.654+0000 7f88bf379740 1 -- 172.21.15.31:0/3267802639 shutdown_connections 2024-03-21T10:42:23.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.654+0000 7f88bf379740 1 -- 172.21.15.31:0/3267802639 wait complete. 2024-03-21T10:42:23.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.655+0000 7f88bf379740 20 client.26660 trim_cache size 0 max 16384 2024-03-21T10:42:23.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.704+0000 7f88bf379740 1 Processor -- start 2024-03-21T10:42:23.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.704+0000 7f88bf379740 1 -- start start 2024-03-21T10:42:23.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.706+0000 7f88bf379740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ba7f77f20 0x555ba7f914b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.706+0000 7f88bf379740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ba7f78c00 0x555ba7f919f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.707+0000 7f88bf379740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ba7f922a0 0x555ba7f93690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.707+0000 7f88bf379740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b41b7360 con 0x555ba7f77f20 2024-03-21T10:42:23.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.707+0000 7f88bf379740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b41b4e60 con 0x555ba7f78c00 2024-03-21T10:42:23.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.707+0000 7f88bf379740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b411a650 con 0x555ba7f922a0 2024-03-21T10:42:23.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.707+0000 7f88ba600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ba7f922a0 0x555ba7f93690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.708+0000 7f88ba600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ba7f922a0 0x555ba7f93690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60508/0 (socket says 172.21.15.31:60508) 2024-03-21T10:42:23.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.708+0000 7f88ba600640 1 -- 172.21.15.31:0/3387215264 learned_addr learned my addr 172.21.15.31:0/3387215264 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:23.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.708+0000 7f88b9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ba7f78c00 0x555ba7f919f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.709+0000 7f88b9200640 1 -- 172.21.15.31:0/3387215264 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ba7f922a0 msgr2=0x555ba7f93690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:23.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.709+0000 7f88b9200640 1 --2- 172.21.15.31:0/3387215264 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ba7f922a0 0x555ba7f93690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.709+0000 7f88b9200640 1 -- 172.21.15.31:0/3387215264 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ba7f77f20 msgr2=0x555ba7f914b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:42:23.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.709+0000 7f88b9200640 1 --2- 172.21.15.31:0/3387215264 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ba7f77f20 0x555ba7f914b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.710+0000 7f88b9200640 1 -- 172.21.15.31:0/3387215264 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ba7bb00f0 con 0x555ba7f78c00 2024-03-21T10:42:23.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.710+0000 7f88ba600640 1 --2- 172.21.15.31:0/3387215264 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ba7f922a0 0x555ba7f93690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:42:23.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.710+0000 7f88b9200640 1 --2- 172.21.15.31:0/3387215264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ba7f78c00 0x555ba7f919f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f88a0098070 tx=0x7f88a009a120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcea5e9bca2c1cf server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:23.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.712+0000 7f88a6000640 1 -- 172.21.15.31:0/3387215264 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a001f070 con 0x555ba7f78c00 2024-03-21T10:42:23.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.712+0000 7f88a6000640 1 -- 172.21.15.31:0/3387215264 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88a009bc10 con 0x555ba7f78c00 2024-03-21T10:42:23.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.712+0000 7f88a6000640 1 -- 172.21.15.31:0/3387215264 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a0097960 con 0x555ba7f78c00 2024-03-21T10:42:23.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.713+0000 7f88bf379740 1 -- 172.21.15.31:0/3387215264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ba7f78c00 msgr2=0x555ba7f919f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:23.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.713+0000 7f88bf379740 1 --2- 172.21.15.31:0/3387215264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ba7f78c00 0x555ba7f919f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f88a0098070 tx=0x7f88a009a120 comp rx=0 tx=0).stop 2024-03-21T10:42:23.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.714+0000 7f88bf379740 1 -- 172.21.15.31:0/3387215264 shutdown_connections 2024-03-21T10:42:23.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.715+0000 7f88bf379740 1 --2- 172.21.15.31:0/3387215264 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ba7f922a0 0x555ba7f93690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.715+0000 7f88bf379740 1 --2- 172.21.15.31:0/3387215264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ba7f78c00 0x555ba7f919f0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.715+0000 7f88bf379740 1 --2- 172.21.15.31:0/3387215264 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ba7f77f20 0x555ba7f914b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.715+0000 7f88bf379740 1 -- 172.21.15.31:0/3387215264 >> 172.21.15.31:0/3387215264 conn(0x555ba7f3f200 msgr2=0x555ba7f724a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:23.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.715+0000 7f88bf379740 1 -- 172.21.15.31:0/3387215264 shutdown_connections 2024-03-21T10:42:23.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.716+0000 7f88bf379740 1 -- 172.21.15.31:0/3387215264 wait complete. 2024-03-21T10:42:23.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.716+0000 7f88bf379740 1 -- start start 2024-03-21T10:42:23.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.716+0000 7f88bf379740 1 Processor -- start 2024-03-21T10:42:23.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.719+0000 7f88bf379740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ba7f77f20 0x555ba7f8c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.719+0000 7f88bf379740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ba7f78c00 0x555ba7f8aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.720+0000 7f88b9c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ba7f77f20 0x555ba7f8c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.720+0000 7f88b9c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ba7f77f20 0x555ba7f8c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60520/0 (socket says 172.21.15.31:60520) 2024-03-21T10:42:23.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.720+0000 7f88b9200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ba7f78c00 0x555ba7f8aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.721+0000 7f88bf379740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ba7f922a0 0x555ba7f8b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.721+0000 7f88b9200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ba7f78c00 0x555ba7f8aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51710/0 (socket says 172.21.15.31:51710) 2024-03-21T10:42:23.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.722+0000 7f88b9c00640 1 -- 172.21.15.31:0/1356862992 learned_addr learned my addr 172.21.15.31:0/1356862992 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:23.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.723+0000 7f88bf379740 1 -- 172.21.15.31:0/1356862992 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b411abc0 con 0x555ba7f78c00 2024-03-21T10:42:23.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.723+0000 7f88bf379740 1 -- 172.21.15.31:0/1356862992 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b41b4800 con 0x555ba7f922a0 2024-03-21T10:42:23.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.723+0000 7f88bf379740 1 -- 172.21.15.31:0/1356862992 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b41b3b40 con 0x555ba7f77f20 2024-03-21T10:42:23.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.723+0000 7f88ba600640 1 --2- 172.21.15.31:0/1356862992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ba7f922a0 0x555ba7f8b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.723+0000 7f88bf379740 10 client.0 _ll_register_callbacks cb 0x7f88a00d3b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:23.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.724+0000 7f88ba600640 1 -- 172.21.15.31:0/1356862992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ba7f77f20 msgr2=0x555ba7f8c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:23.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.724+0000 7f88ba600640 1 --2- 172.21.15.31:0/1356862992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ba7f77f20 0x555ba7f8c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.724+0000 7f88ba600640 1 -- 172.21.15.31:0/1356862992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ba7f78c00 msgr2=0x555ba7f8aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:23.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.724+0000 7f88ba600640 1 --2- 172.21.15.31:0/1356862992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ba7f78c00 0x555ba7f8aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.724+0000 7f88ba600640 1 -- 172.21.15.31:0/1356862992 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555ba7f8bbd0 con 0x555ba7f922a0 2024-03-21T10:42:23.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.724+0000 7f88b9c00640 1 --2- 172.21.15.31:0/1356862992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ba7f77f20 0x555ba7f8c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:42:23.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.726+0000 7f88ba600640 1 --2- 172.21.15.31:0/1356862992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ba7f922a0 0x555ba7f8b530 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f88ac02c700 tx=0x7f88ac032620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=839abeae3a53d05 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:23.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.726+0000 7f889b400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:42:23.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.735+0000 7f889b400640 1 -- 172.21.15.31:0/1356862992 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ac04f020 con 0x555ba7f922a0 2024-03-21T10:42:23.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.735+0000 7f88bf379740 1 -- 172.21.15.31:0/1356862992 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f88a009bc10 con 0x555ba7f922a0 2024-03-21T10:42:23.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.735+0000 7f889b400640 1 -- 172.21.15.31:0/1356862992 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f88ac004040 con 0x555ba7f922a0 2024-03-21T10:42:23.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.738+0000 7f889b400640 1 -- 172.21.15.31:0/1356862992 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88ac05a050 con 0x555ba7f922a0 2024-03-21T10:42:23.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.738+0000 7f88bf379740 10 client.26666 fetch_fsmap learned FSMap version 25 2024-03-21T10:42:23.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.738+0000 7f88bf379740 1 -- 172.21.15.31:0/1356862992 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x555ba7ffd610 con 0x555ba7f922a0 2024-03-21T10:42:23.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.739+0000 7f889b400640 1 -- 172.21.15.31:0/1356862992 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88a009bc10 con 0x555ba7f922a0 2024-03-21T10:42:23.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.739+0000 7f889b400640 1 -- 172.21.15.31:0/1356862992 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f88ac04b9d0 con 0x555ba7f922a0 2024-03-21T10:42:23.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.739+0000 7f88bf379740 10 client.26666 fetch_fsmap finished waiting for FSMap version 25 2024-03-21T10:42:23.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.739+0000 7f88bf379740 10 client.26666 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:42:23.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.739+0000 7f88bf379740 20 client.26666 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:23.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.740+0000 7f88bf379740 1 --2- 172.21.15.31:0/1356862992 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x555ba7ffdd40 0x555ba8000160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:23.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.740+0000 7f88bf379740 4 client.26666 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:42:23.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.740+0000 7f88bf379740 1 -- 172.21.15.31:0/1356862992 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555ba7ee3ae0 con 0x555ba7ffdd40 2024-03-21T10:42:23.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.742+0000 7f88b9c00640 1 --2- 172.21.15.31:0/1356862992 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x555ba7ffdd40 0x555ba8000160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:23.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.744+0000 7f88b9c00640 1 --2- 172.21.15.31:0/1356862992 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x555ba7ffdd40 0x555ba8000160 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ffcecd9ec50c9f3 server_cookie=a44793b61d498a4f in_seq=0 out_seq=0 2024-03-21T10:42:23.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.744+0000 7f889b400640 10 client.26666 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:23.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.744+0000 7f889b400640 1 -- 172.21.15.31:0/1356862992 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x555ba7ee3ae0 con 0x555ba7ffdd40 2024-03-21T10:42:23.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.744+0000 7f889b400640 10 client.26666 handle_command_reply: tid=0 2024-03-21T10:42:23.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.744+0000 7f88bf379740 1 client.26666 shutdown 2024-03-21T10:42:23.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.746+0000 7f88bf379740 1 -- 172.21.15.31:0/1356862992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ba7f922a0 msgr2=0x555ba7f8b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:23.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.746+0000 7f88bf379740 1 --2- 172.21.15.31:0/1356862992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ba7f922a0 0x555ba7f8b530 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f88ac02c700 tx=0x7f88ac032620 comp rx=0 tx=0).stop 2024-03-21T10:42:23.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.747+0000 7f88bf379740 1 -- 172.21.15.31:0/1356862992 shutdown_connections 2024-03-21T10:42:23.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.747+0000 7f88bf379740 1 --2- 172.21.15.31:0/1356862992 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x555ba7ffdd40 0x555ba8000160 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.747+0000 7f88bf379740 1 --2- 172.21.15.31:0/1356862992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ba7f922a0 0x555ba7f8b530 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.747+0000 7f88bf379740 1 --2- 172.21.15.31:0/1356862992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ba7f78c00 0x555ba7f8aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.747+0000 7f88bf379740 1 --2- 172.21.15.31:0/1356862992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555ba7f77f20 0x555ba7f8c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.747+0000 7f889b400640 0 client.26666 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:23.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.747+0000 7f88bf379740 1 -- 172.21.15.31:0/1356862992 >> 172.21.15.31:0/1356862992 conn(0x555ba7f3f200 msgr2=0x555ba7f71200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:23.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.748+0000 7f88bf379740 1 -- 172.21.15.31:0/1356862992 shutdown_connections 2024-03-21T10:42:23.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.749+0000 7f88bf379740 1 -- 172.21.15.31:0/1356862992 wait complete. 2024-03-21T10:42:23.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.749+0000 7f88bf379740 20 client.26666 trim_cache size 0 max 16384 2024-03-21T10:42:23.746 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:42:23.746 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "scrub active (0 inodes in the stack)", 2024-03-21T10:42:23.746 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-03-21T10:42:23.747 INFO:teuthology.orchestra.run.smithi031.stdout: "7518468c-2861-4d4e-be5a-9dec219c3b76": { 2024-03-21T10:42:23.747 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-03-21T10:42:23.747 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "7518468c-2861-4d4e-be5a-9dec219c3b76", 2024-03-21T10:42:23.747 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-03-21T10:42:23.747 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:42:23.747 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:42:23.747 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:42:23.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.751+0000 7f88bc600640 1 -- 172.21.15.31:0/363600413 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f889c0a3290 msgr2=0x7f889c0a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:23.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.751+0000 7f88bc600640 1 --2- 172.21.15.31:0/363600413 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f889c0a3290 0x7f889c0a5750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f88ac037040 tx=0x7f88ac009860 comp rx=0 tx=0).stop 2024-03-21T10:42:23.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.751+0000 7f88bc600640 1 -- 172.21.15.31:0/363600413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88b4104220 msgr2=0x7f88b410f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:23.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.751+0000 7f88bc600640 1 --2- 172.21.15.31:0/363600413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88b4104220 0x7f88b410f640 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f88a000e990 tx=0x7f88a000ee50 comp rx=0 tx=0).stop 2024-03-21T10:42:23.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.751+0000 7f88bc600640 1 -- 172.21.15.31:0/363600413 shutdown_connections 2024-03-21T10:42:23.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.751+0000 7f88bc600640 1 --2- 172.21.15.31:0/363600413 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f889c0a3290 0x7f889c0a5750 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.751+0000 7f88bc600640 1 --2- 172.21.15.31:0/363600413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88b4109870 0x7f88b410fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.751+0000 7f88bc600640 1 --2- 172.21.15.31:0/363600413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88b4104220 0x7f88b410f640 secure :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f88a000e990 tx=0x7f88a000ee50 comp rx=0 tx=0).stop 2024-03-21T10:42:23.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.751+0000 7f88bc600640 1 --2- 172.21.15.31:0/363600413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88b4103860 0x7f88b410f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:23.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.751+0000 7f88bc600640 1 -- 172.21.15.31:0/363600413 >> 172.21.15.31:0/363600413 conn(0x7f88b40ff310 msgr2=0x7f88b4101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:23.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.753+0000 7f88bc600640 1 -- 172.21.15.31:0/363600413 shutdown_connections 2024-03-21T10:42:23.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:23.753+0000 7f88bc600640 1 -- 172.21.15.31:0/363600413 wait complete. 2024-03-21T10:42:23.793 INFO:tasks.cephfs.filesystem:scrub status for tag:7518468c-2861-4d4e-be5a-9dec219c3b76 - {'path': '/', 'tag': '7518468c-2861-4d4e-be5a-9dec219c3b76', 'options': 'recursive,force'} 2024-03-21T10:42:24.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:23 smithi031 ceph-mon[31673]: scrub queued for path: / 2024-03-21T10:42:24.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:23 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:42:24.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:23 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:42:24.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:23 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:42:24.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:23 smithi070 ceph-mon[36330]: scrub queued for path: / 2024-03-21T10:42:24.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:23 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:42:24.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:23 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:42:24.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:23 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:42:24.081 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:23 smithi040 ceph-mon[36154]: scrub queued for path: / 2024-03-21T10:42:24.081 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:23 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:42:24.081 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:23 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:42:24.082 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:23 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:42:25.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:24 smithi031 ceph-mon[31673]: pgmap v118: 129 pgs: 129 active+clean; 18 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 155 MiB/s wr, 1.26k op/s 2024-03-21T10:42:25.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:24 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:42:25.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:24 smithi070 ceph-mon[36330]: pgmap v118: 129 pgs: 129 active+clean; 18 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 155 MiB/s wr, 1.26k op/s 2024-03-21T10:42:25.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:24 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:42:25.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:24 smithi040 ceph-mon[36154]: pgmap v118: 129 pgs: 129 active+clean; 18 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 155 MiB/s wr, 1.26k op/s 2024-03-21T10:42:25.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:24 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:42:27.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:26 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:42:27.060 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:26 smithi031 ceph-mon[31673]: pgmap v119: 129 pgs: 129 active+clean; 18 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 234 MiB/s wr, 1.84k op/s 2024-03-21T10:42:27.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:26 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:42:27.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:26 smithi070 ceph-mon[36330]: pgmap v119: 129 pgs: 129 active+clean; 18 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 234 MiB/s wr, 1.84k op/s 2024-03-21T10:42:27.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:26 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:42:27.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:26 smithi040 ceph-mon[36154]: pgmap v119: 129 pgs: 129 active+clean; 18 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 234 MiB/s wr, 1.84k op/s 2024-03-21T10:42:29.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:28 smithi070 ceph-mon[36330]: pgmap v120: 129 pgs: 129 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 213 MiB/s wr, 1.51k op/s 2024-03-21T10:42:29.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:28 smithi040 ceph-mon[36154]: pgmap v120: 129 pgs: 129 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 213 MiB/s wr, 1.51k op/s 2024-03-21T10:42:29.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:28 smithi031 ceph-mon[31673]: pgmap v120: 129 pgs: 129 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 213 MiB/s wr, 1.51k op/s 2024-03-21T10:42:31.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:30 smithi040 ceph-mon[36154]: pgmap v121: 129 pgs: 129 active+clean; 19 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 236 MiB/s wr, 1.72k op/s 2024-03-21T10:42:31.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:30 smithi031 ceph-mon[31673]: pgmap v121: 129 pgs: 129 active+clean; 19 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 236 MiB/s wr, 1.72k op/s 2024-03-21T10:42:31.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:30 smithi070 ceph-mon[36330]: pgmap v121: 129 pgs: 129 active+clean; 19 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 236 MiB/s wr, 1.72k op/s 2024-03-21T10:42:32.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:31 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:42:32.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:31 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:42:32.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:31 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:42:32.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:31 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:42:32.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:31 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:42:32.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:31 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:42:33.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:32 smithi040 ceph-mon[36154]: pgmap v122: 129 pgs: 129 active+clean; 19 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 187 MiB/s wr, 1.48k op/s 2024-03-21T10:42:33.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:32 smithi040 ceph-mon[36154]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:42:33.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:32 smithi031 ceph-mon[31673]: pgmap v122: 129 pgs: 129 active+clean; 19 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 187 MiB/s wr, 1.48k op/s 2024-03-21T10:42:33.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:32 smithi031 ceph-mon[31673]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:42:33.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:32 smithi070 ceph-mon[36330]: pgmap v122: 129 pgs: 129 active+clean; 19 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 187 MiB/s wr, 1.48k op/s 2024-03-21T10:42:33.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:32 smithi070 ceph-mon[36330]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:42:35.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:34 smithi040 ceph-mon[36154]: pgmap v123: 129 pgs: 129 active+clean; 19 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 148 MiB/s wr, 1.25k op/s 2024-03-21T10:42:35.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:34 smithi031 ceph-mon[31673]: pgmap v123: 129 pgs: 129 active+clean; 19 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 148 MiB/s wr, 1.25k op/s 2024-03-21T10:42:35.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:34 smithi070 ceph-mon[36330]: pgmap v123: 129 pgs: 129 active+clean; 19 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 148 MiB/s wr, 1.25k op/s 2024-03-21T10:42:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.287+0000 7f00e4000640 1 -- 172.21.15.31:0/1246859073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00dc11ae80 msgr2=0x7f00dc11d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.287+0000 7f00e4000640 1 --2- 172.21.15.31:0/1246859073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00dc11ae80 0x7f00dc11d270 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f00d0009900 tx=0x7f00d002bc60 comp rx=0 tx=0).stop 2024-03-21T10:42:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.287+0000 7f00e4000640 1 -- 172.21.15.31:0/1246859073 shutdown_connections 2024-03-21T10:42:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.287+0000 7f00e4000640 1 --2- 172.21.15.31:0/1246859073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00dc11ae80 0x7f00dc11d270 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.287+0000 7f00e4000640 1 --2- 172.21.15.31:0/1246859073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00dc110280 0x7f00dc10def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.287+0000 7f00e4000640 1 --2- 172.21.15.31:0/1246859073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00dc10fe10 0x7f00dc10d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.287+0000 7f00e4000640 1 -- 172.21.15.31:0/1246859073 >> 172.21.15.31:0/1246859073 conn(0x7f00dc06c750 msgr2=0x7f00dc06cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.287+0000 7f00e4000640 1 -- 172.21.15.31:0/1246859073 shutdown_connections 2024-03-21T10:42:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.287+0000 7f00e4000640 1 -- 172.21.15.31:0/1246859073 wait complete. 2024-03-21T10:42:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.289+0000 7f00e4000640 1 Processor -- start 2024-03-21T10:42:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.289+0000 7f00e4000640 1 -- start start 2024-03-21T10:42:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.289+0000 7f00e4000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00dc10fe10 0x7f00dc1ac940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.290+0000 7f00e1600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00dc10fe10 0x7f00dc1ac940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.290+0000 7f00e4000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00dc110280 0x7f00dc1ace80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.290+0000 7f00e4000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00dc11ae80 0x7f00dc1a6af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.290+0000 7f00e4000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00dc120870 con 0x7f00dc10fe10 2024-03-21T10:42:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.290+0000 7f00e4000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00dc1206f0 con 0x7f00dc11ae80 2024-03-21T10:42:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.290+0000 7f00e4000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00dc1209f0 con 0x7f00dc110280 2024-03-21T10:42:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.290+0000 7f00e1600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00dc10fe10 0x7f00dc1ac940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60482/0 (socket says 172.21.15.31:60482) 2024-03-21T10:42:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.290+0000 7f00e1600640 1 -- 172.21.15.31:0/2504409447 learned_addr learned my addr 172.21.15.31:0/2504409447 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.291+0000 7f00e1600640 1 -- 172.21.15.31:0/2504409447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00dc110280 msgr2=0x7f00dc1ace80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T10:42:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.291+0000 7f00e1600640 1 --2- 172.21.15.31:0/2504409447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00dc110280 0x7f00dc1ace80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.291+0000 7f00e1600640 1 -- 172.21.15.31:0/2504409447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00dc11ae80 msgr2=0x7f00dc1a6af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T10:42:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.291+0000 7f00e1600640 1 --2- 172.21.15.31:0/2504409447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00dc11ae80 0x7f00dc1a6af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.291+0000 7f00e1600640 1 -- 172.21.15.31:0/2504409447 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00dc1a7270 con 0x7f00dc10fe10 2024-03-21T10:42:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.292+0000 7f00e1600640 1 --2- 172.21.15.31:0/2504409447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00dc10fe10 0x7f00dc1ac940 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f00d400cce0 tx=0x7f00d400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.292+0000 7f00da000640 1 -- 172.21.15.31:0/2504409447 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d4004b20 con 0x7f00dc10fe10 2024-03-21T10:42:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.292+0000 7f00da000640 1 -- 172.21.15.31:0/2504409447 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00d4004cc0 con 0x7f00dc10fe10 2024-03-21T10:42:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.292+0000 7f00da000640 1 -- 172.21.15.31:0/2504409447 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d400f7d0 con 0x7f00dc10fe10 2024-03-21T10:42:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.293+0000 7f00e4000640 1 -- 172.21.15.31:0/2504409447 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00dc1a7560 con 0x7f00dc10fe10 2024-03-21T10:42:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.293+0000 7f00e4000640 1 -- 172.21.15.31:0/2504409447 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00dc1b3800 con 0x7f00dc10fe10 2024-03-21T10:42:36.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.297+0000 7f00da000640 1 -- 172.21.15.31:0/2504409447 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f00d400f970 con 0x7f00dc10fe10 2024-03-21T10:42:36.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.298+0000 7f00e6e15740 1 Processor -- start 2024-03-21T10:42:36.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.298+0000 7f00e6e15740 1 -- start start 2024-03-21T10:42:36.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.298+0000 7f00e6e15740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643455f9fd0 0x5643455fa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:36.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.299+0000 7f00e6e15740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5643455fa8f0 0x564345603180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:36.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.299+0000 7f00e6e15740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643456036c0 0x564345605ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:36.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.299+0000 7f00e6e15740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643455bc3b0 con 0x5643455f9fd0 2024-03-21T10:42:36.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.299+0000 7f00e6e15740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56434555e830 con 0x5643455fa8f0 2024-03-21T10:42:36.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.299+0000 7f00e6e15740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643455075d0 con 0x5643456036c0 2024-03-21T10:42:36.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.299+0000 7f00e2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643455f9fd0 0x5643455fa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:36.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.299+0000 7f00e2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643455f9fd0 0x5643455fa3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60498/0 (socket says 172.21.15.31:60498) 2024-03-21T10:42:36.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.299+0000 7f00e2000640 1 -- 172.21.15.31:0/215846280 learned_addr learned my addr 172.21.15.31:0/215846280 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:36.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.299+0000 7f00e0c00640 1 --2- 172.21.15.31:0/215846280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643456036c0 0x564345605ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:36.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.299+0000 7f00e1600640 1 --2- 172.21.15.31:0/215846280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5643455fa8f0 0x564345603180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:36.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.299+0000 7f00e2000640 1 -- 172.21.15.31:0/215846280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643456036c0 msgr2=0x564345605ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:36.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.299+0000 7f00e2000640 1 --2- 172.21.15.31:0/215846280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643456036c0 0x564345605ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.299+0000 7f00e2000640 1 -- 172.21.15.31:0/215846280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5643455fa8f0 msgr2=0x564345603180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:36.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.299+0000 7f00e2000640 1 --2- 172.21.15.31:0/215846280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5643455fa8f0 0x564345603180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.299+0000 7f00e2000640 1 -- 172.21.15.31:0/215846280 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56434551f2c0 con 0x5643455f9fd0 2024-03-21T10:42:36.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.299+0000 7f00e1600640 1 --2- 172.21.15.31:0/215846280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5643455fa8f0 0x564345603180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:42:36.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.300+0000 7f00e2000640 1 --2- 172.21.15.31:0/215846280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643455f9fd0 0x5643455fa3b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f00d0005c20 tx=0x7f00d002c600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd96a4b2e3d60467 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:36.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.301+0000 7f00ce600640 1 -- 172.21.15.31:0/215846280 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d002c870 con 0x5643455f9fd0 2024-03-21T10:42:36.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.301+0000 7f00ce600640 1 -- 172.21.15.31:0/215846280 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00d0043070 con 0x5643455f9fd0 2024-03-21T10:42:36.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.301+0000 7f00ce600640 1 -- 172.21.15.31:0/215846280 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d00046c0 con 0x5643455f9fd0 2024-03-21T10:42:36.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.302+0000 7f00da000640 1 --2- 172.21.15.31:0/2504409447 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f00c40a3080 0x7f00c40a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:36.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.302+0000 7f00e6e15740 1 -- 172.21.15.31:0/215846280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643455f9fd0 msgr2=0x5643455fa3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:36.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.302+0000 7f00e6e15740 1 --2- 172.21.15.31:0/215846280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643455f9fd0 0x5643455fa3b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f00d0005c20 tx=0x7f00d002c600 comp rx=0 tx=0).stop 2024-03-21T10:42:36.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.302+0000 7f00e1600640 1 --2- 172.21.15.31:0/2504409447 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f00c40a3080 0x7f00c40a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:36.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.303+0000 7f00e6e15740 1 -- 172.21.15.31:0/215846280 shutdown_connections 2024-03-21T10:42:36.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.303+0000 7f00e6e15740 1 --2- 172.21.15.31:0/215846280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643456036c0 0x564345605ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.303+0000 7f00e6e15740 1 --2- 172.21.15.31:0/215846280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5643455fa8f0 0x564345603180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.303+0000 7f00e6e15740 1 --2- 172.21.15.31:0/215846280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643455f9fd0 0x5643455fa3b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.303+0000 7f00e6e15740 1 -- 172.21.15.31:0/215846280 >> 172.21.15.31:0/215846280 conn(0x5643455f48f0 msgr2=0x5643455f4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:36.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.303+0000 7f00e6e15740 1 -- 172.21.15.31:0/215846280 shutdown_connections 2024-03-21T10:42:36.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.304+0000 7f00e6e15740 1 -- 172.21.15.31:0/215846280 wait complete. 2024-03-21T10:42:36.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.304+0000 7f00da000640 1 -- 172.21.15.31:0/2504409447 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f00d40cf970 con 0x7f00dc10fe10 2024-03-21T10:42:36.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.304+0000 7f00e6e15740 1 -- start start 2024-03-21T10:42:36.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.304+0000 7f00e6e15740 1 Processor -- start 2024-03-21T10:42:36.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.305+0000 7f00e1600640 1 --2- 172.21.15.31:0/2504409447 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f00c40a3080 0x7f00c40a5540 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x5643455f9a80 tx=0x7f00d400af30 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:36.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.305+0000 7f00e6e15740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643455f9fd0 0x5643456006b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:36.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.305+0000 7f00e6e15740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5643455fa8f0 0x564345600bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.306+0000 7f00e2000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643455f9fd0 0x5643456006b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.306+0000 7f00e2000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643455f9fd0 0x5643456006b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:45828/0 (socket says 172.21.15.31:45828) 2024-03-21T10:42:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.306+0000 7f00e0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5643455fa8f0 0x564345600bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.306+0000 7f00e0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5643455fa8f0 0x564345600bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60142/0 (socket says 172.21.15.31:60142) 2024-03-21T10:42:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.306+0000 7f00e6e15740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643456036c0 0x564345601130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.306+0000 7f00e2000640 1 -- 172.21.15.31:0/833107991 learned_addr learned my addr 172.21.15.31:0/833107991 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.306+0000 7f00e6e15740 1 -- 172.21.15.31:0/833107991 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56434551aa20 con 0x5643456036c0 2024-03-21T10:42:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.306+0000 7f00e6e15740 1 -- 172.21.15.31:0/833107991 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56434555a8d0 con 0x5643455fa8f0 2024-03-21T10:42:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.306+0000 7f00e6e15740 1 -- 172.21.15.31:0/833107991 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x564345606290 con 0x5643455f9fd0 2024-03-21T10:42:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.307+0000 7f00e6e15740 10 client.0 _ll_register_callbacks cb 0x5643455571c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.307+0000 7f00e2000640 1 -- 172.21.15.31:0/833107991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5643455fa8f0 msgr2=0x564345600bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.307+0000 7f00e2000640 1 --2- 172.21.15.31:0/833107991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5643455fa8f0 0x564345600bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.307+0000 7f00e2000640 1 -- 172.21.15.31:0/833107991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643456036c0 msgr2=0x564345601130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.307+0000 7f00e2000640 1 --2- 172.21.15.31:0/833107991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643456036c0 0x564345601130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.308+0000 7f00e2000640 1 -- 172.21.15.31:0/833107991 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564345273210 con 0x5643455f9fd0 2024-03-21T10:42:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.308+0000 7f00e0c00640 1 --2- 172.21.15.31:0/833107991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5643455fa8f0 0x564345600bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:42:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.308+0000 7f00e2000640 1 --2- 172.21.15.31:0/833107991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643455f9fd0 0x5643456006b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f00d0009850 tx=0x7f00d00056a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79a95189cfacf4ee server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.308+0000 7f00c2a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:42:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.309+0000 7f00c2a00640 1 -- 172.21.15.31:0/833107991 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d004e020 con 0x5643455f9fd0 2024-03-21T10:42:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.309+0000 7f00c2a00640 1 -- 172.21.15.31:0/833107991 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f00d0004240 con 0x5643455f9fd0 2024-03-21T10:42:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.309+0000 7f00e6e15740 1 -- 172.21.15.31:0/833107991 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f00d0043070 con 0x5643455f9fd0 2024-03-21T10:42:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.309+0000 7f00c2a00640 1 -- 172.21.15.31:0/833107991 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00d0059050 con 0x5643455f9fd0 2024-03-21T10:42:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.309+0000 7f00e6e15740 10 client.26626 fetch_fsmap learned FSMap version 26 2024-03-21T10:42:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.309+0000 7f00e6e15740 1 -- 172.21.15.31:0/833107991 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x56434533f620 con 0x5643455f9fd0 2024-03-21T10:42:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.310+0000 7f00c2a00640 1 -- 172.21.15.31:0/833107991 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00d0043070 con 0x5643455f9fd0 2024-03-21T10:42:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.311+0000 7f00c2a00640 1 -- 172.21.15.31:0/833107991 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f00d0004830 con 0x5643455f9fd0 2024-03-21T10:42:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.311+0000 7f00e6e15740 10 client.26626 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:42:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.311+0000 7f00e6e15740 10 client.26626 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:42:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.311+0000 7f00e6e15740 20 client.26626 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.311+0000 7f00e6e15740 1 --2- 172.21.15.31:0/833107991 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564345613460 0x564345615850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.311+0000 7f00e6e15740 4 client.26626 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:42:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.311+0000 7f00e6e15740 1 -- 172.21.15.31:0/833107991 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56434556bd70 con 0x564345613460 2024-03-21T10:42:36.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.311+0000 7f00e0c00640 1 --2- 172.21.15.31:0/833107991 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564345613460 0x564345615850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:36.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.312+0000 7f00c2a00640 10 client.26626 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:36.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.318+0000 7f00e0c00640 1 --2- 172.21.15.31:0/833107991 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564345613460 0x564345615850 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d714dbdd588d53a3 server_cookie=6e049b9f28f9120e in_seq=0 out_seq=0 2024-03-21T10:42:36.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.318+0000 7f00c2a00640 1 -- 172.21.15.31:0/833107991 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56434556bd70 con 0x564345613460 2024-03-21T10:42:36.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.318+0000 7f00c2a00640 10 client.26626 handle_command_reply: tid=0 2024-03-21T10:42:36.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.319+0000 7f00e6e15740 1 client.26626 shutdown 2024-03-21T10:42:36.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.319+0000 7f00e6e15740 1 -- 172.21.15.31:0/833107991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643455f9fd0 msgr2=0x5643456006b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:36.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.319+0000 7f00e6e15740 1 --2- 172.21.15.31:0/833107991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643455f9fd0 0x5643456006b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f00d0009850 tx=0x7f00d00056a0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.319+0000 7f00e6e15740 1 -- 172.21.15.31:0/833107991 shutdown_connections 2024-03-21T10:42:36.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.319+0000 7f00e6e15740 1 --2- 172.21.15.31:0/833107991 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564345613460 0x564345615850 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.319+0000 7f00e6e15740 1 --2- 172.21.15.31:0/833107991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643456036c0 0x564345601130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.319+0000 7f00e6e15740 1 --2- 172.21.15.31:0/833107991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5643455fa8f0 0x564345600bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.319+0000 7f00e6e15740 1 --2- 172.21.15.31:0/833107991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643455f9fd0 0x5643456006b0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.319+0000 7f00e6e15740 1 -- 172.21.15.31:0/833107991 >> 172.21.15.31:0/833107991 conn(0x5643455f48f0 msgr2=0x5643455f6640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:36.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.319+0000 7f00c2a00640 0 client.26626 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:36.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.319+0000 7f00e6e15740 1 -- 172.21.15.31:0/833107991 shutdown_connections 2024-03-21T10:42:36.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.320+0000 7f00e6e15740 1 -- 172.21.15.31:0/833107991 wait complete. 2024-03-21T10:42:36.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.320+0000 7f00e6e15740 20 client.26626 trim_cache size 0 max 16384 2024-03-21T10:42:36.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.358+0000 7f00e6e15740 1 Processor -- start 2024-03-21T10:42:36.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.358+0000 7f00e6e15740 1 -- start start 2024-03-21T10:42:36.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.358+0000 7f00e6e15740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643455f9fd0 0x564345602de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:36.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.358+0000 7f00e6e15740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5643455fa8f0 0x5643455fef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:36.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.358+0000 7f00e6e15740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643456036c0 0x5643455ff4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:36.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.358+0000 7f00e6e15740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00dc1bfb70 con 0x5643456036c0 2024-03-21T10:42:36.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.358+0000 7f00e6e15740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00dc1bd670 con 0x5643455fa8f0 2024-03-21T10:42:36.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.358+0000 7f00e6e15740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00dc1bb710 con 0x5643455f9fd0 2024-03-21T10:42:36.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.359+0000 7f00e0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5643455fa8f0 0x5643455fef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:36.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.359+0000 7f00e0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5643455fa8f0 0x5643455fef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60158/0 (socket says 172.21.15.31:60158) 2024-03-21T10:42:36.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.359+0000 7f00e0c00640 1 -- 172.21.15.31:0/3545824716 learned_addr learned my addr 172.21.15.31:0/3545824716 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:36.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.359+0000 7f00e2000640 1 --2- 172.21.15.31:0/3545824716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643455f9fd0 0x564345602de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:36.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.359+0000 7f00e2000640 1 --2- 172.21.15.31:0/3545824716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643456036c0 0x5643455ff4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:36.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.359+0000 7f00e2000640 1 -- 172.21.15.31:0/3545824716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5643455fa8f0 msgr2=0x5643455fef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:36.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.359+0000 7f00e2000640 1 --2- 172.21.15.31:0/3545824716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5643455fa8f0 0x5643455fef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.359+0000 7f00e2000640 1 -- 172.21.15.31:0/3545824716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643456036c0 msgr2=0x5643455ff4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:36.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.359+0000 7f00e2000640 1 --2- 172.21.15.31:0/3545824716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643456036c0 0x5643455ff4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.359+0000 7f00e2000640 1 -- 172.21.15.31:0/3545824716 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5643455fa6e0 con 0x5643455f9fd0 2024-03-21T10:42:36.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.360+0000 7f00e2000640 1 --2- 172.21.15.31:0/3545824716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643455f9fd0 0x564345602de0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f00d002c1a0 tx=0x7f00d002e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc6d081542aa252f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:36.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.361+0000 7f00cdc00640 1 -- 172.21.15.31:0/3545824716 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d0062070 con 0x5643455f9fd0 2024-03-21T10:42:36.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.361+0000 7f00cdc00640 1 -- 172.21.15.31:0/3545824716 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00d0043070 con 0x5643455f9fd0 2024-03-21T10:42:36.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.361+0000 7f00cdc00640 1 -- 172.21.15.31:0/3545824716 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d004e030 con 0x5643455f9fd0 2024-03-21T10:42:36.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.362+0000 7f00e6e15740 1 -- 172.21.15.31:0/3545824716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643455f9fd0 msgr2=0x564345602de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:36.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.362+0000 7f00e6e15740 1 --2- 172.21.15.31:0/3545824716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643455f9fd0 0x564345602de0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f00d002c1a0 tx=0x7f00d002e040 comp rx=0 tx=0).stop 2024-03-21T10:42:36.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.362+0000 7f00e6e15740 1 -- 172.21.15.31:0/3545824716 shutdown_connections 2024-03-21T10:42:36.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.362+0000 7f00e6e15740 1 --2- 172.21.15.31:0/3545824716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643456036c0 0x5643455ff4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.362+0000 7f00e6e15740 1 --2- 172.21.15.31:0/3545824716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5643455fa8f0 0x5643455fef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.362+0000 7f00e6e15740 1 --2- 172.21.15.31:0/3545824716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643455f9fd0 0x564345602de0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.362+0000 7f00e6e15740 1 -- 172.21.15.31:0/3545824716 >> 172.21.15.31:0/3545824716 conn(0x5643455f48f0 msgr2=0x5643455fba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:36.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.362+0000 7f00e6e15740 1 -- 172.21.15.31:0/3545824716 shutdown_connections 2024-03-21T10:42:36.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.362+0000 7f00e6e15740 1 -- 172.21.15.31:0/3545824716 wait complete. 2024-03-21T10:42:36.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.362+0000 7f00e6e15740 1 -- start start 2024-03-21T10:42:36.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.363+0000 7f00e6e15740 1 Processor -- start 2024-03-21T10:42:36.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.363+0000 7f00e6e15740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5643455f9fd0 0x564345615660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:36.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.363+0000 7f00e6e15740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643455fa8f0 0x564345615ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:36.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.363+0000 7f00e6e15740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643456036c0 0x56434566c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:36.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.363+0000 7f00e6e15740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00dc1bbd10 con 0x5643455fa8f0 2024-03-21T10:42:36.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.363+0000 7f00e6e15740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00dc1bd010 con 0x5643455f9fd0 2024-03-21T10:42:36.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.363+0000 7f00e6e15740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00dc1bc350 con 0x5643456036c0 2024-03-21T10:42:36.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.363+0000 7f00e6e15740 10 client.0 _ll_register_callbacks cb 0x7f00c8009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:36.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.363+0000 7f00e2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5643455f9fd0 0x564345615660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:36.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.363+0000 7f00e2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5643455f9fd0 0x564345615660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60172/0 (socket says 172.21.15.31:60172) 2024-03-21T10:42:36.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.363+0000 7f00e2000640 1 -- 172.21.15.31:0/3130200308 learned_addr learned my addr 172.21.15.31:0/3130200308 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:36.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.363+0000 7f00e2000640 1 --2- 172.21.15.31:0/3130200308 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643456036c0 0x56434566c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:36.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.363+0000 7f00e0c00640 1 --2- 172.21.15.31:0/3130200308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643455fa8f0 0x564345615ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.364+0000 7f00e0c00640 1 -- 172.21.15.31:0/3130200308 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643456036c0 msgr2=0x56434566c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.364+0000 7f00e0c00640 1 --2- 172.21.15.31:0/3130200308 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643456036c0 0x56434566c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.364+0000 7f00e0c00640 1 -- 172.21.15.31:0/3130200308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5643455f9fd0 msgr2=0x564345615660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.364+0000 7f00e0c00640 1 --2- 172.21.15.31:0/3130200308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5643455f9fd0 0x564345615660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.364+0000 7f00e0c00640 1 -- 172.21.15.31:0/3130200308 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56434566cef0 con 0x5643455fa8f0 2024-03-21T10:42:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.364+0000 7f00e0c00640 1 --2- 172.21.15.31:0/3130200308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643455fa8f0 0x564345615ba0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x5643455f4880 tx=0x7f00c801fa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3e057434b8389b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.364+0000 7f00c2a00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:42:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.364+0000 7f00c2a00640 1 -- 172.21.15.31:0/3130200308 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00c8004030 con 0x5643455fa8f0 2024-03-21T10:42:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.365+0000 7f00c2a00640 1 -- 172.21.15.31:0/3130200308 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f00c80041d0 con 0x5643455fa8f0 2024-03-21T10:42:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.365+0000 7f00e6e15740 1 -- 172.21.15.31:0/3130200308 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f00d0043070 con 0x5643455fa8f0 2024-03-21T10:42:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.365+0000 7f00c2a00640 1 -- 172.21.15.31:0/3130200308 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00c802e050 con 0x5643455fa8f0 2024-03-21T10:42:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.365+0000 7f00e6e15740 10 client.16317 fetch_fsmap learned FSMap version 26 2024-03-21T10:42:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.365+0000 7f00e6e15740 1 -- 172.21.15.31:0/3130200308 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x56434566e250 con 0x5643455fa8f0 2024-03-21T10:42:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.365+0000 7f00c2a00640 1 -- 172.21.15.31:0/3130200308 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00d0043070 con 0x5643455fa8f0 2024-03-21T10:42:36.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.365+0000 7f00c2a00640 1 -- 172.21.15.31:0/3130200308 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f00c8003710 con 0x5643455fa8f0 2024-03-21T10:42:36.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.365+0000 7f00e6e15740 10 client.16317 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:42:36.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.365+0000 7f00e6e15740 10 client.16317 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:42:36.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.365+0000 7f00e6e15740 20 client.16317 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:36.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.366+0000 7f00e6e15740 1 --2- 172.21.15.31:0/3130200308 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56434566ea00 0x564345670ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:36.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.366+0000 7f00e6e15740 4 client.16317 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:42:36.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.366+0000 7f00e6e15740 1 -- 172.21.15.31:0/3130200308 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5643455660c0 con 0x56434566ea00 2024-03-21T10:42:36.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.366+0000 7f00e2000640 1 --2- 172.21.15.31:0/3130200308 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56434566ea00 0x564345670ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:36.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.367+0000 7f00c2a00640 10 client.16317 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:36.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.367+0000 7f00e2000640 1 --2- 172.21.15.31:0/3130200308 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56434566ea00 0x564345670ec0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56398b2bdedb1dc3 server_cookie=be0c41c0b53394ec in_seq=0 out_seq=0 2024-03-21T10:42:36.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.383+0000 7f00c2a00640 1 -- 172.21.15.31:0/3130200308 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5643455660c0 con 0x56434566ea00 2024-03-21T10:42:36.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.383+0000 7f00c2a00640 10 client.16317 handle_command_reply: tid=0 2024-03-21T10:42:36.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.383+0000 7f00e6e15740 1 client.16317 shutdown 2024-03-21T10:42:36.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.383+0000 7f00e6e15740 1 -- 172.21.15.31:0/3130200308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643455fa8f0 msgr2=0x564345615ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:36.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.383+0000 7f00e6e15740 1 --2- 172.21.15.31:0/3130200308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643455fa8f0 0x564345615ba0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x5643455f4880 tx=0x7f00c801fa60 comp rx=0 tx=0).stop 2024-03-21T10:42:36.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.383+0000 7f00e6e15740 1 -- 172.21.15.31:0/3130200308 shutdown_connections 2024-03-21T10:42:36.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.383+0000 7f00e6e15740 1 --2- 172.21.15.31:0/3130200308 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56434566ea00 0x564345670ec0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.383+0000 7f00e6e15740 1 --2- 172.21.15.31:0/3130200308 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643456036c0 0x56434566c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.383+0000 7f00e6e15740 1 --2- 172.21.15.31:0/3130200308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643455fa8f0 0x564345615ba0 secure :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0x5643455f4880 tx=0x7f00c801fa60 comp rx=0 tx=0).stop 2024-03-21T10:42:36.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.383+0000 7f00e6e15740 1 --2- 172.21.15.31:0/3130200308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5643455f9fd0 0x564345615660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.383+0000 7f00c2a00640 0 client.16317 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:36.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.383+0000 7f00e6e15740 1 -- 172.21.15.31:0/3130200308 >> 172.21.15.31:0/3130200308 conn(0x5643455f48f0 msgr2=0x5643455fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:36.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.386+0000 7f00e6e15740 1 -- 172.21.15.31:0/3130200308 shutdown_connections 2024-03-21T10:42:36.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.386+0000 7f00e6e15740 1 -- 172.21.15.31:0/3130200308 wait complete. 2024-03-21T10:42:36.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.386+0000 7f00e6e15740 20 client.16317 trim_cache size 0 max 16384 2024-03-21T10:42:36.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:42:36.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:42:36.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:42:36.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:42:36.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.388+0000 7f00e4000640 1 -- 172.21.15.31:0/2504409447 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f00c40a3080 msgr2=0x7f00c40a5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:36.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.388+0000 7f00e4000640 1 --2- 172.21.15.31:0/2504409447 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f00c40a3080 0x7f00c40a5540 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x5643455f9a80 tx=0x7f00d400af30 comp rx=0 tx=0).stop 2024-03-21T10:42:36.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.388+0000 7f00e4000640 1 -- 172.21.15.31:0/2504409447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00dc10fe10 msgr2=0x7f00dc1ac940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:36.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.388+0000 7f00e4000640 1 --2- 172.21.15.31:0/2504409447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00dc10fe10 0x7f00dc1ac940 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f00d400cce0 tx=0x7f00d400a620 comp rx=0 tx=0).stop 2024-03-21T10:42:36.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.389+0000 7f00e4000640 1 -- 172.21.15.31:0/2504409447 shutdown_connections 2024-03-21T10:42:36.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.389+0000 7f00e4000640 1 --2- 172.21.15.31:0/2504409447 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f00c40a3080 0x7f00c40a5540 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x5643455f9a80 tx=0x7f00d400af30 comp rx=0 tx=0).stop 2024-03-21T10:42:36.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.389+0000 7f00e4000640 1 --2- 172.21.15.31:0/2504409447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00dc11ae80 0x7f00dc1a6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.389+0000 7f00e4000640 1 --2- 172.21.15.31:0/2504409447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00dc110280 0x7f00dc1ace80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:36.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.389+0000 7f00e4000640 1 --2- 172.21.15.31:0/2504409447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00dc10fe10 0x7f00dc1ac940 secure :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f00d400cce0 tx=0x7f00d400a620 comp rx=0 tx=0).stop 2024-03-21T10:42:36.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.389+0000 7f00e4000640 1 -- 172.21.15.31:0/2504409447 >> 172.21.15.31:0/2504409447 conn(0x7f00dc06c750 msgr2=0x7f00dc119870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:36.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.392+0000 7f00e4000640 1 -- 172.21.15.31:0/2504409447 shutdown_connections 2024-03-21T10:42:36.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:36.393+0000 7f00e4000640 1 -- 172.21.15.31:0/2504409447 wait complete. 2024-03-21T10:42:37.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:36 smithi031 ceph-mon[31673]: pgmap v124: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 216 MiB/s wr, 1.79k op/s 2024-03-21T10:42:37.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:36 smithi070 ceph-mon[36330]: pgmap v124: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 216 MiB/s wr, 1.79k op/s 2024-03-21T10:42:37.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:36 smithi040 ceph-mon[36154]: pgmap v124: 129 pgs: 129 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 216 MiB/s wr, 1.79k op/s 2024-03-21T10:42:39.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:38 smithi031 ceph-mon[31673]: pgmap v125: 129 pgs: 129 active+clean; 20 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 157 MiB/s wr, 1.36k op/s 2024-03-21T10:42:39.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:38 smithi070 ceph-mon[36330]: pgmap v125: 129 pgs: 129 active+clean; 20 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 157 MiB/s wr, 1.36k op/s 2024-03-21T10:42:39.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:38 smithi040 ceph-mon[36154]: pgmap v125: 129 pgs: 129 active+clean; 20 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 157 MiB/s wr, 1.36k op/s 2024-03-21T10:42:41.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:40 smithi031 ceph-mon[31673]: pgmap v126: 129 pgs: 129 active+clean; 21 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 210 MiB/s wr, 1.67k op/s 2024-03-21T10:42:41.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:40 smithi070 ceph-mon[36330]: pgmap v126: 129 pgs: 129 active+clean; 21 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 210 MiB/s wr, 1.67k op/s 2024-03-21T10:42:41.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:40 smithi040 ceph-mon[36154]: pgmap v126: 129 pgs: 129 active+clean; 21 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 210 MiB/s wr, 1.67k op/s 2024-03-21T10:42:43.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:42 smithi031 ceph-mon[31673]: pgmap v127: 129 pgs: 129 active+clean; 21 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 209 MiB/s wr, 1.53k op/s 2024-03-21T10:42:43.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:42 smithi070 ceph-mon[36330]: pgmap v127: 129 pgs: 129 active+clean; 21 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 209 MiB/s wr, 1.53k op/s 2024-03-21T10:42:43.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:42 smithi040 ceph-mon[36154]: pgmap v127: 129 pgs: 129 active+clean; 21 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 209 MiB/s wr, 1.53k op/s 2024-03-21T10:42:45.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:45 smithi031 ceph-mon[31673]: pgmap v128: 129 pgs: 129 active+clean; 21 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 185 MiB/s wr, 1.31k op/s 2024-03-21T10:42:45.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:45 smithi070 ceph-mon[36330]: pgmap v128: 129 pgs: 129 active+clean; 21 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 185 MiB/s wr, 1.31k op/s 2024-03-21T10:42:45.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:45 smithi040 ceph-mon[36154]: pgmap v128: 129 pgs: 129 active+clean; 21 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 185 MiB/s wr, 1.31k op/s 2024-03-21T10:42:47.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:47 smithi031 ceph-mon[31673]: pgmap v129: 129 pgs: 129 active+clean; 22 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 259 MiB/s wr, 1.69k op/s 2024-03-21T10:42:47.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:47 smithi070 ceph-mon[36330]: pgmap v129: 129 pgs: 129 active+clean; 22 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 259 MiB/s wr, 1.69k op/s 2024-03-21T10:42:47.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:47 smithi040 ceph-mon[36154]: pgmap v129: 129 pgs: 129 active+clean; 22 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 259 MiB/s wr, 1.69k op/s 2024-03-21T10:42:49.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:49 smithi031 ceph-mon[31673]: pgmap v130: 129 pgs: 129 active+clean; 22 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 212 MiB/s wr, 1.25k op/s 2024-03-21T10:42:49.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:49 smithi070 ceph-mon[36330]: pgmap v130: 129 pgs: 129 active+clean; 22 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 212 MiB/s wr, 1.25k op/s 2024-03-21T10:42:49.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:49 smithi040 ceph-mon[36154]: pgmap v130: 129 pgs: 129 active+clean; 22 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 212 MiB/s wr, 1.25k op/s 2024-03-21T10:42:49.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.612+0000 7fced9a00640 1 -- 172.21.15.31:0/1132542055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fced410d230 msgr2=0x7fced410d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:49.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.612+0000 7fced9a00640 1 --2- 172.21.15.31:0/1132542055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fced410d230 0x7fced410d6e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fcec4009be0 tx=0x7fcec402bf10 comp rx=0 tx=0).stop 2024-03-21T10:42:49.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.612+0000 7fced9a00640 1 -- 172.21.15.31:0/1132542055 shutdown_connections 2024-03-21T10:42:49.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.612+0000 7fced9a00640 1 --2- 172.21.15.31:0/1132542055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fced410d230 0x7fced410d6e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.612+0000 7fced9a00640 1 --2- 172.21.15.31:0/1132542055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fced4108fc0 0x7fced410ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.613+0000 7fced9a00640 1 --2- 172.21.15.31:0/1132542055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fced41086a0 0x7fced4108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.613+0000 7fced9a00640 1 -- 172.21.15.31:0/1132542055 >> 172.21.15.31:0/1132542055 conn(0x7fced406cbe0 msgr2=0x7fced406cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:49.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.613+0000 7fced9a00640 1 -- 172.21.15.31:0/1132542055 shutdown_connections 2024-03-21T10:42:49.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.613+0000 7fced9a00640 1 -- 172.21.15.31:0/1132542055 wait complete. 2024-03-21T10:42:49.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.614+0000 7fced9a00640 1 Processor -- start 2024-03-21T10:42:49.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.615+0000 7fced9a00640 1 -- start start 2024-03-21T10:42:49.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.615+0000 7fced9a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fced41086a0 0x7fced411c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:49.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.615+0000 7fced9a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fced4108fc0 0x7fced4119140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:49.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.615+0000 7fced9a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fced410d230 0x7fced41196e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:49.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.615+0000 7fced2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fced41086a0 0x7fced411c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:49.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.615+0000 7fced2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fced41086a0 0x7fced411c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58590/0 (socket says 172.21.15.31:58590) 2024-03-21T10:42:49.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.616+0000 7fced2000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fced4108fc0 0x7fced4119140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:49.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.616+0000 7fced3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fced410d230 0x7fced41196e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:49.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.616+0000 7fced3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fced410d230 0x7fced41196e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43796/0 (socket says 172.21.15.31:43796) 2024-03-21T10:42:49.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.616+0000 7fced2000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fced4108fc0 0x7fced4119140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:41384/0 (socket says 172.21.15.31:41384) 2024-03-21T10:42:49.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.616+0000 7fced9a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced411c6d0 con 0x7fced410d230 2024-03-21T10:42:49.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.616+0000 7fced2a00640 1 -- 172.21.15.31:0/3890481927 learned_addr learned my addr 172.21.15.31:0/3890481927 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:49.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.616+0000 7fced9a00640 1 -- 172.21.15.31:0/3890481927 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced4111fc0 con 0x7fced41086a0 2024-03-21T10:42:49.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.616+0000 7fced9a00640 1 -- 172.21.15.31:0/3890481927 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced411c850 con 0x7fced4108fc0 2024-03-21T10:42:49.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.616+0000 7fced2000640 1 -- 172.21.15.31:0/3890481927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fced41086a0 msgr2=0x7fced411c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:49.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.616+0000 7fced2000640 1 --2- 172.21.15.31:0/3890481927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fced41086a0 0x7fced411c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.617+0000 7fced2000640 1 -- 172.21.15.31:0/3890481927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fced410d230 msgr2=0x7fced41196e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:49.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.617+0000 7fced2000640 1 --2- 172.21.15.31:0/3890481927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fced410d230 0x7fced41196e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.617+0000 7fced2000640 1 -- 172.21.15.31:0/3890481927 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced41a8990 con 0x7fced4108fc0 2024-03-21T10:42:49.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.618+0000 7fced2000640 1 --2- 172.21.15.31:0/3890481927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fced4108fc0 0x7fced4119140 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fcebc00c970 tx=0x7fcebc00ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:49.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.618+0000 7fcecb400640 1 -- 172.21.15.31:0/3890481927 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcebc007bb0 con 0x7fced4108fc0 2024-03-21T10:42:49.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.618+0000 7fcecb400640 1 -- 172.21.15.31:0/3890481927 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcebc007d50 con 0x7fced4108fc0 2024-03-21T10:42:49.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.618+0000 7fcecb400640 1 -- 172.21.15.31:0/3890481927 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcebc00f450 con 0x7fced4108fc0 2024-03-21T10:42:49.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.619+0000 7fced9a00640 1 -- 172.21.15.31:0/3890481927 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced41a8c80 con 0x7fced4108fc0 2024-03-21T10:42:49.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.619+0000 7fced9a00640 1 -- 172.21.15.31:0/3890481927 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced41a9150 con 0x7fced4108fc0 2024-03-21T10:42:49.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.625+0000 7fcedc61b740 1 Processor -- start 2024-03-21T10:42:49.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.627+0000 7fcecb400640 1 -- 172.21.15.31:0/3890481927 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fcebc004eb0 con 0x7fced4108fc0 2024-03-21T10:42:49.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.629+0000 7fcedc61b740 1 -- start start 2024-03-21T10:42:49.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.629+0000 7fcedc61b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bd141fd0 0x5635bd1423b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:49.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.629+0000 7fcedc61b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635bd1428f0 0x5635bd14b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:49.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.629+0000 7fcedc61b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd14b6c0 0x5635bd14dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:49.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.629+0000 7fcedc61b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635bd1043b0 con 0x5635bd141fd0 2024-03-21T10:42:49.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.629+0000 7fcedc61b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635bd0a6830 con 0x5635bd14b6c0 2024-03-21T10:42:49.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.629+0000 7fcedc61b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635bd04f5d0 con 0x5635bd1428f0 2024-03-21T10:42:49.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.630+0000 7fcecb400640 1 --2- 172.21.15.31:0/3890481927 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fceb80a31d0 0x7fceb80a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:49.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.630+0000 7fcecb400640 1 -- 172.21.15.31:0/3890481927 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fcebc0cf9f0 con 0x7fced4108fc0 2024-03-21T10:42:49.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.631+0000 7fced3400640 1 --2- 172.21.15.31:0/3890481927 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fceb80a31d0 0x7fceb80a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:49.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.632+0000 7fced2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd14b6c0 0x5635bd14dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:49.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.632+0000 7fced2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd14b6c0 0x5635bd14dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58600/0 (socket says 172.21.15.31:58600) 2024-03-21T10:42:49.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.632+0000 7fced2000640 1 -- 172.21.15.31:0/762695661 learned_addr learned my addr 172.21.15.31:0/762695661 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:49.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.632+0000 7fced2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635bd1428f0 0x5635bd14b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:49.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.632+0000 7fced3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bd141fd0 0x5635bd1423b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:49.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.634+0000 7fced2000640 1 -- 172.21.15.31:0/762695661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635bd1428f0 msgr2=0x5635bd14b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:49.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.634+0000 7fced2000640 1 --2- 172.21.15.31:0/762695661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635bd1428f0 0x5635bd14b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.634+0000 7fced2000640 1 -- 172.21.15.31:0/762695661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bd141fd0 msgr2=0x5635bd1423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:49.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.634+0000 7fced2000640 1 --2- 172.21.15.31:0/762695661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bd141fd0 0x5635bd1423b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.634+0000 7fced2000640 1 -- 172.21.15.31:0/762695661 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635bd0672c0 con 0x5635bd14b6c0 2024-03-21T10:42:49.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.635+0000 7fced2000640 1 --2- 172.21.15.31:0/762695661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd14b6c0 0x5635bd14dab0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fcebc0a3f20 tx=0x7fcebc09b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5db9a1ab95652675 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:49.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.635+0000 7fced3400640 1 --2- 172.21.15.31:0/3890481927 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fceb80a31d0 0x7fceb80a5690 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcec4004af0 tx=0x7fcec4033bf0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:49.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.636+0000 7fcec3e00640 1 -- 172.21.15.31:0/762695661 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcebc0a4450 con 0x5635bd14b6c0 2024-03-21T10:42:49.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.636+0000 7fcec3e00640 1 -- 172.21.15.31:0/762695661 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcebc099ae0 con 0x5635bd14b6c0 2024-03-21T10:42:49.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.637+0000 7fcec3e00640 1 -- 172.21.15.31:0/762695661 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcebc0a4220 con 0x5635bd14b6c0 2024-03-21T10:42:49.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.638+0000 7fcedc61b740 1 -- 172.21.15.31:0/762695661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd14b6c0 msgr2=0x5635bd14dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:49.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.638+0000 7fcedc61b740 1 --2- 172.21.15.31:0/762695661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd14b6c0 0x5635bd14dab0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fcebc0a3f20 tx=0x7fcebc09b940 comp rx=0 tx=0).stop 2024-03-21T10:42:49.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.638+0000 7fcedc61b740 1 -- 172.21.15.31:0/762695661 shutdown_connections 2024-03-21T10:42:49.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.638+0000 7fcedc61b740 1 --2- 172.21.15.31:0/762695661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd14b6c0 0x5635bd14dab0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.638+0000 7fcedc61b740 1 --2- 172.21.15.31:0/762695661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635bd1428f0 0x5635bd14b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.638+0000 7fcedc61b740 1 --2- 172.21.15.31:0/762695661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bd141fd0 0x5635bd1423b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.638+0000 7fcedc61b740 1 -- 172.21.15.31:0/762695661 >> 172.21.15.31:0/762695661 conn(0x5635bd13c8f0 msgr2=0x5635bd13ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:49.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.638+0000 7fcedc61b740 1 -- 172.21.15.31:0/762695661 shutdown_connections 2024-03-21T10:42:49.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.639+0000 7fcedc61b740 1 -- 172.21.15.31:0/762695661 wait complete. 2024-03-21T10:42:49.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.639+0000 7fcedc61b740 1 -- start start 2024-03-21T10:42:49.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.639+0000 7fcedc61b740 1 Processor -- start 2024-03-21T10:42:49.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.640+0000 7fcedc61b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bd141fd0 0x5635bd1487e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:49.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.640+0000 7fcedc61b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635bd1428f0 0x5635bd148d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.640+0000 7fcedc61b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd14b6c0 0x5635bd149260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.641+0000 7fced2000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635bd1428f0 0x5635bd148d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.641+0000 7fced2000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635bd1428f0 0x5635bd148d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:41400/0 (socket says 172.21.15.31:41400) 2024-03-21T10:42:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.641+0000 7fced2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bd141fd0 0x5635bd1487e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.641+0000 7fced2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bd141fd0 0x5635bd1487e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43826/0 (socket says 172.21.15.31:43826) 2024-03-21T10:42:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.641+0000 7fcedc61b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635bd062a20 con 0x5635bd141fd0 2024-03-21T10:42:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.641+0000 7fced3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd14b6c0 0x5635bd149260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.641+0000 7fced3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd14b6c0 0x5635bd149260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58614/0 (socket says 172.21.15.31:58614) 2024-03-21T10:42:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.641+0000 7fced3400640 1 -- 172.21.15.31:0/1956472529 learned_addr learned my addr 172.21.15.31:0/1956472529 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.641+0000 7fcedc61b740 1 -- 172.21.15.31:0/1956472529 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635bd0a28d0 con 0x5635bd14b6c0 2024-03-21T10:42:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.641+0000 7fcedc61b740 1 -- 172.21.15.31:0/1956472529 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635bd14e2f0 con 0x5635bd1428f0 2024-03-21T10:42:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.642+0000 7fcedc61b740 10 client.0 _ll_register_callbacks cb 0x5635bd09f1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.642+0000 7fced3400640 1 -- 172.21.15.31:0/1956472529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635bd1428f0 msgr2=0x5635bd148d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.642+0000 7fced3400640 1 --2- 172.21.15.31:0/1956472529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635bd1428f0 0x5635bd148d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.642+0000 7fced3400640 1 -- 172.21.15.31:0/1956472529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bd141fd0 msgr2=0x5635bd1487e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.642+0000 7fced3400640 1 --2- 172.21.15.31:0/1956472529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bd141fd0 0x5635bd1487e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.642+0000 7fced3400640 1 -- 172.21.15.31:0/1956472529 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5635bcdbb210 con 0x5635bd14b6c0 2024-03-21T10:42:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.643+0000 7fced3400640 1 --2- 172.21.15.31:0/1956472529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd14b6c0 0x5635bd149260 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fcec4009be0 tx=0x7fcec4005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e23e40ed413bc5b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.643+0000 7fcec0c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:42:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.644+0000 7fcec0c00640 1 -- 172.21.15.31:0/1956472529 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec4004250 con 0x5635bd14b6c0 2024-03-21T10:42:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.644+0000 7fcec0c00640 1 -- 172.21.15.31:0/1956472529 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fcec40043f0 con 0x5635bd14b6c0 2024-03-21T10:42:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.645+0000 7fcedc61b740 1 -- 172.21.15.31:0/1956472529 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcebc099ae0 con 0x5635bd14b6c0 2024-03-21T10:42:49.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.646+0000 7fcec0c00640 1 -- 172.21.15.31:0/1956472529 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcec40045f0 con 0x5635bd14b6c0 2024-03-21T10:42:49.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.646+0000 7fcedc61b740 10 client.26690 fetch_fsmap learned FSMap version 26 2024-03-21T10:42:49.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.646+0000 7fcedc61b740 1 -- 172.21.15.31:0/1956472529 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5635bce87620 con 0x5635bd14b6c0 2024-03-21T10:42:49.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.647+0000 7fcec0c00640 1 -- 172.21.15.31:0/1956472529 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcebc099ae0 con 0x5635bd14b6c0 2024-03-21T10:42:49.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.647+0000 7fcec0c00640 1 -- 172.21.15.31:0/1956472529 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fcec4042440 con 0x5635bd14b6c0 2024-03-21T10:42:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.648+0000 7fcedc61b740 10 client.26690 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:42:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.648+0000 7fcedc61b740 10 client.26690 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:42:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.648+0000 7fcedc61b740 20 client.26690 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.648+0000 7fcedc61b740 1 --2- 172.21.15.31:0/1956472529 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5635bd15b840 0x5635bd15dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.648+0000 7fcedc61b740 4 client.26690 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:42:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.648+0000 7fcedc61b740 1 -- 172.21.15.31:0/1956472529 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5635bd0b3d70 con 0x5635bd15b840 2024-03-21T10:42:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.649+0000 7fced2a00640 1 --2- 172.21.15.31:0/1956472529 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5635bd15b840 0x5635bd15dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.649+0000 7fcec0c00640 10 client.26690 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:49.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.650+0000 7fced2a00640 1 --2- 172.21.15.31:0/1956472529 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5635bd15b840 0x5635bd15dc30 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52f1ee2ec6d24244 server_cookie=480103785512b924 in_seq=0 out_seq=0 2024-03-21T10:42:49.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.652+0000 7fcec0c00640 1 -- 172.21.15.31:0/1956472529 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5635bd0b3d70 con 0x5635bd15b840 2024-03-21T10:42:49.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.652+0000 7fcec0c00640 10 client.26690 handle_command_reply: tid=0 2024-03-21T10:42:49.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.653+0000 7fcedc61b740 1 client.26690 shutdown 2024-03-21T10:42:49.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.653+0000 7fcedc61b740 1 -- 172.21.15.31:0/1956472529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd14b6c0 msgr2=0x5635bd149260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:49.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.653+0000 7fcedc61b740 1 --2- 172.21.15.31:0/1956472529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd14b6c0 0x5635bd149260 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fcec4009be0 tx=0x7fcec4005fd0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.654+0000 7fcedc61b740 1 -- 172.21.15.31:0/1956472529 shutdown_connections 2024-03-21T10:42:49.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.654+0000 7fcedc61b740 1 --2- 172.21.15.31:0/1956472529 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5635bd15b840 0x5635bd15dc30 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.654+0000 7fcedc61b740 1 --2- 172.21.15.31:0/1956472529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd14b6c0 0x5635bd149260 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.654+0000 7fcedc61b740 1 --2- 172.21.15.31:0/1956472529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635bd1428f0 0x5635bd148d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.654+0000 7fcedc61b740 1 --2- 172.21.15.31:0/1956472529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bd141fd0 0x5635bd1487e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.654+0000 7fcedc61b740 1 -- 172.21.15.31:0/1956472529 >> 172.21.15.31:0/1956472529 conn(0x5635bd13c8f0 msgr2=0x5635bd13e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:49.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.654+0000 7fcec0c00640 0 client.26690 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:49.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.654+0000 7fcedc61b740 1 -- 172.21.15.31:0/1956472529 shutdown_connections 2024-03-21T10:42:49.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.654+0000 7fcedc61b740 1 -- 172.21.15.31:0/1956472529 wait complete. 2024-03-21T10:42:49.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.654+0000 7fcedc61b740 20 client.26690 trim_cache size 0 max 16384 2024-03-21T10:42:49.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.681+0000 7fcedc61b740 1 Processor -- start 2024-03-21T10:42:49.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.691+0000 7fcedc61b740 1 -- start start 2024-03-21T10:42:49.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.692+0000 7fcedc61b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd141fd0 0x5635bd14af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:49.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.692+0000 7fcedc61b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635bd1428f0 0x5635bd156a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:49.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.692+0000 7fcedc61b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bd14b6c0 0x5635bd156f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:49.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.692+0000 7fcedc61b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced41c8460 con 0x5635bd14b6c0 2024-03-21T10:42:49.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.692+0000 7fcedc61b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced41c5f60 con 0x5635bd141fd0 2024-03-21T10:42:49.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.692+0000 7fcedc61b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced41c4000 con 0x5635bd1428f0 2024-03-21T10:42:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.693+0000 7fced2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd141fd0 0x5635bd14af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.693+0000 7fced2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd141fd0 0x5635bd14af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58618/0 (socket says 172.21.15.31:58618) 2024-03-21T10:42:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.693+0000 7fced2a00640 1 -- 172.21.15.31:0/2425724182 learned_addr learned my addr 172.21.15.31:0/2425724182 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.693+0000 7fced3400640 1 --2- 172.21.15.31:0/2425724182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bd14b6c0 0x5635bd156f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.693+0000 7fced2a00640 1 -- 172.21.15.31:0/2425724182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635bd1428f0 msgr2=0x5635bd156a00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:42:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.693+0000 7fced2a00640 1 --2- 172.21.15.31:0/2425724182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635bd1428f0 0x5635bd156a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.693+0000 7fced2a00640 1 -- 172.21.15.31:0/2425724182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bd14b6c0 msgr2=0x5635bd156f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.693+0000 7fced2a00640 1 --2- 172.21.15.31:0/2425724182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bd14b6c0 0x5635bd156f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.693+0000 7fced2a00640 1 -- 172.21.15.31:0/2425724182 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635bd157fb0 con 0x5635bd141fd0 2024-03-21T10:42:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.694+0000 7fced2a00640 1 --2- 172.21.15.31:0/2425724182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd141fd0 0x5635bd14af10 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fcecc00be90 tx=0x7fcecc01db10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55272d7d299b7b13 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.695+0000 7fcec3400640 1 -- 172.21.15.31:0/2425724182 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcecc0130d0 con 0x5635bd141fd0 2024-03-21T10:42:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.695+0000 7fcec3400640 1 -- 172.21.15.31:0/2425724182 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcecc013270 con 0x5635bd141fd0 2024-03-21T10:42:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.695+0000 7fcec3400640 1 -- 172.21.15.31:0/2425724182 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcecc005420 con 0x5635bd141fd0 2024-03-21T10:42:49.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.696+0000 7fcedc61b740 1 -- 172.21.15.31:0/2425724182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd141fd0 msgr2=0x5635bd14af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:49.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.697+0000 7fcedc61b740 1 --2- 172.21.15.31:0/2425724182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd141fd0 0x5635bd14af10 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fcecc00be90 tx=0x7fcecc01db10 comp rx=0 tx=0).stop 2024-03-21T10:42:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.697+0000 7fcedc61b740 1 -- 172.21.15.31:0/2425724182 shutdown_connections 2024-03-21T10:42:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.697+0000 7fcedc61b740 1 --2- 172.21.15.31:0/2425724182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bd14b6c0 0x5635bd156f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.697+0000 7fcedc61b740 1 --2- 172.21.15.31:0/2425724182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635bd1428f0 0x5635bd156a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.697+0000 7fcedc61b740 1 --2- 172.21.15.31:0/2425724182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd141fd0 0x5635bd14af10 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.697+0000 7fcedc61b740 1 -- 172.21.15.31:0/2425724182 >> 172.21.15.31:0/2425724182 conn(0x5635bd13c8f0 msgr2=0x5635bd1470d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.698+0000 7fcedc61b740 1 -- 172.21.15.31:0/2425724182 shutdown_connections 2024-03-21T10:42:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.698+0000 7fcedc61b740 1 -- 172.21.15.31:0/2425724182 wait complete. 2024-03-21T10:42:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.698+0000 7fcedc61b740 1 -- start start 2024-03-21T10:42:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.699+0000 7fcedc61b740 1 Processor -- start 2024-03-21T10:42:49.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.702+0000 7fcedc61b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bd141fd0 0x5635bd145030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:49.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.703+0000 7fcedc61b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635bd1428f0 0x5635bd145570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:49.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.703+0000 7fcedc61b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd14b6c0 0x5635bd145ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:49.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.703+0000 7fced2000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635bd1428f0 0x5635bd145570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:49.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.703+0000 7fcedc61b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced41c4600 con 0x5635bd141fd0 2024-03-21T10:42:49.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.703+0000 7fcedc61b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced41c5900 con 0x5635bd14b6c0 2024-03-21T10:42:49.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.703+0000 7fcedc61b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced41c4c40 con 0x5635bd1428f0 2024-03-21T10:42:49.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.703+0000 7fced2000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635bd1428f0 0x5635bd145570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:41408/0 (socket says 172.21.15.31:41408) 2024-03-21T10:42:49.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.703+0000 7fced2000640 1 -- 172.21.15.31:0/220116011 learned_addr learned my addr 172.21.15.31:0/220116011 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:49.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.703+0000 7fced3400640 1 --2- 172.21.15.31:0/220116011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd14b6c0 0x5635bd145ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:49.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.704+0000 7fcedc61b740 10 client.0 _ll_register_callbacks cb 0x7fcec4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:49.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.704+0000 7fced2000640 1 -- 172.21.15.31:0/220116011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd14b6c0 msgr2=0x5635bd145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:49.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.704+0000 7fced2000640 1 --2- 172.21.15.31:0/220116011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd14b6c0 0x5635bd145ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.704+0000 7fced2000640 1 -- 172.21.15.31:0/220116011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bd141fd0 msgr2=0x5635bd145030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:49.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.704+0000 7fced2a00640 1 --2- 172.21.15.31:0/220116011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bd141fd0 0x5635bd145030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:49.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.704+0000 7fced2000640 1 --2- 172.21.15.31:0/220116011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bd141fd0 0x5635bd145030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.704+0000 7fced2000640 1 -- 172.21.15.31:0/220116011 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5635bd1b6100 con 0x5635bd1428f0 2024-03-21T10:42:49.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.704+0000 7fced3400640 1 --2- 172.21.15.31:0/220116011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd14b6c0 0x5635bd145ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:42:49.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.704+0000 7fcedc61b740 1 -- 172.21.15.31:0/220116011 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcecc013270 con 0x5635bd1428f0 2024-03-21T10:42:49.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.704+0000 7fced2a00640 1 --2- 172.21.15.31:0/220116011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bd141fd0 0x5635bd145030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:42:49.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.706+0000 7fced2000640 1 --2- 172.21.15.31:0/220116011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635bd1428f0 0x5635bd145570 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fcebc09bc00 tx=0x7fcebc00af20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48190c2d2879dc8f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:49.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.707+0000 7fcec0c00640 10 client.26647 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:42:49.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.707+0000 7fcec0c00640 1 -- 172.21.15.31:0/220116011 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcebc0a9060 con 0x5635bd1428f0 2024-03-21T10:42:49.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.707+0000 7fcec0c00640 1 -- 172.21.15.31:0/220116011 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fcebc0cd020 con 0x5635bd1428f0 2024-03-21T10:42:49.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.708+0000 7fcec0c00640 1 -- 172.21.15.31:0/220116011 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcebc0a4450 con 0x5635bd1428f0 2024-03-21T10:42:49.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.708+0000 7fcedc61b740 10 client.26647 fetch_fsmap learned FSMap version 26 2024-03-21T10:42:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.708+0000 7fcedc61b740 1 -- 172.21.15.31:0/220116011 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({fsmap=26}) v3 -- 0x5635bd1b7330 con 0x5635bd1428f0 2024-03-21T10:42:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.709+0000 7fcec0c00640 1 -- 172.21.15.31:0/220116011 <== mon.1 v2:172.21.15.70:3300/0 4 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fcebc0ba320 con 0x5635bd1428f0 2024-03-21T10:42:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.709+0000 7fcedc61b740 10 client.26647 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:42:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.709+0000 7fcedc61b740 10 client.26647 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:42:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.709+0000 7fcedc61b740 20 client.26647 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.709+0000 7fcedc61b740 1 --2- 172.21.15.31:0/220116011 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5635bd1b7560 0x5635bd1b9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.709+0000 7fcedc61b740 4 client.26647 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:42:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.709+0000 7fcedc61b740 1 -- 172.21.15.31:0/220116011 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5635bd0ae0c0 con 0x5635bd1b7560 2024-03-21T10:42:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.710+0000 7fced2a00640 1 --2- 172.21.15.31:0/220116011 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5635bd1b7560 0x5635bd1b9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.711+0000 7fced2a00640 1 --2- 172.21.15.31:0/220116011 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5635bd1b7560 0x5635bd1b9950 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66e777c3b1335acf server_cookie=69cfffb105f930f2 in_seq=0 out_seq=0 2024-03-21T10:42:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.711+0000 7fcec0c00640 10 client.26647 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.753+0000 7fcec0c00640 1 -- 172.21.15.31:0/220116011 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5635bd0ae0c0 con 0x5635bd1b7560 2024-03-21T10:42:49.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.753+0000 7fcec0c00640 10 client.26647 handle_command_reply: tid=0 2024-03-21T10:42:49.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.753+0000 7fcedc61b740 1 client.26647 shutdown 2024-03-21T10:42:49.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.761+0000 7fcedc61b740 1 -- 172.21.15.31:0/220116011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635bd1428f0 msgr2=0x5635bd145570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:49.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.761+0000 7fcedc61b740 1 --2- 172.21.15.31:0/220116011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635bd1428f0 0x5635bd145570 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fcebc09bc00 tx=0x7fcebc00af20 comp rx=0 tx=0).stop 2024-03-21T10:42:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.762+0000 7fcedc61b740 1 -- 172.21.15.31:0/220116011 shutdown_connections 2024-03-21T10:42:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.762+0000 7fcedc61b740 1 --2- 172.21.15.31:0/220116011 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5635bd1b7560 0x5635bd1b9950 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.762+0000 7fcedc61b740 1 --2- 172.21.15.31:0/220116011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635bd14b6c0 0x5635bd145ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.762+0000 7fcedc61b740 1 --2- 172.21.15.31:0/220116011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635bd1428f0 0x5635bd145570 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.762+0000 7fcedc61b740 1 --2- 172.21.15.31:0/220116011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bd141fd0 0x5635bd145030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.762+0000 7fcedc61b740 1 -- 172.21.15.31:0/220116011 >> 172.21.15.31:0/220116011 conn(0x5635bd13c8f0 msgr2=0x5635bd1487a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.762+0000 7fcec0c00640 0 client.26647 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:49.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.763+0000 7fcedc61b740 1 -- 172.21.15.31:0/220116011 shutdown_connections 2024-03-21T10:42:49.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.763+0000 7fcedc61b740 1 -- 172.21.15.31:0/220116011 wait complete. 2024-03-21T10:42:49.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.764+0000 7fcedc61b740 20 client.26647 trim_cache size 0 max 16384 2024-03-21T10:42:49.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:42:49.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:42:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:42:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:42:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.767+0000 7fced9a00640 1 -- 172.21.15.31:0/3890481927 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fceb80a31d0 msgr2=0x7fceb80a5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.767+0000 7fced9a00640 1 --2- 172.21.15.31:0/3890481927 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fceb80a31d0 0x7fceb80a5690 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcec4004af0 tx=0x7fcec4033bf0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.767+0000 7fced9a00640 1 -- 172.21.15.31:0/3890481927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fced4108fc0 msgr2=0x7fced4119140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.767+0000 7fced9a00640 1 --2- 172.21.15.31:0/3890481927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fced4108fc0 0x7fced4119140 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fcebc00c970 tx=0x7fcebc00ce30 comp rx=0 tx=0).stop 2024-03-21T10:42:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.769+0000 7fced9a00640 1 -- 172.21.15.31:0/3890481927 shutdown_connections 2024-03-21T10:42:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.769+0000 7fced9a00640 1 --2- 172.21.15.31:0/3890481927 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fceb80a31d0 0x7fceb80a5690 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.769+0000 7fced9a00640 1 --2- 172.21.15.31:0/3890481927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fced410d230 0x7fced41196e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.769+0000 7fced9a00640 1 --2- 172.21.15.31:0/3890481927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fced4108fc0 0x7fced4119140 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.769+0000 7fced9a00640 1 --2- 172.21.15.31:0/3890481927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fced41086a0 0x7fced411c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.770+0000 7fced9a00640 1 -- 172.21.15.31:0/3890481927 >> 172.21.15.31:0/3890481927 conn(0x7fced406cbe0 msgr2=0x7fced410ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.777+0000 7fced9a00640 1 -- 172.21.15.31:0/3890481927 shutdown_connections 2024-03-21T10:42:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:42:49.782+0000 7fced9a00640 1 -- 172.21.15.31:0/3890481927 wait complete. 2024-03-21T10:42:50.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:50 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:42:50.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:50 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:42:50.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:50 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:42:51.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:51 smithi070 ceph-mon[36330]: pgmap v131: 129 pgs: 129 active+clean; 23 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 265 MiB/s wr, 1.41k op/s 2024-03-21T10:42:51.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:51 smithi040 ceph-mon[36154]: pgmap v131: 129 pgs: 129 active+clean; 23 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 265 MiB/s wr, 1.41k op/s 2024-03-21T10:42:51.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:51 smithi031 ceph-mon[31673]: pgmap v131: 129 pgs: 129 active+clean; 23 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 265 MiB/s wr, 1.41k op/s 2024-03-21T10:42:52.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:52 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:42:52.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:52 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:42:52.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:52 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:42:52.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:52 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:42:52.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:52 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:42:52.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:52 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:42:52.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:52 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:42:52.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:52 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:42:52.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:52 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:42:53.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:53 smithi040 ceph-mon[36154]: pgmap v132: 129 pgs: 129 active+clean; 24 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 227 MiB/s wr, 1.09k op/s 2024-03-21T10:42:53.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:53 smithi031 ceph-mon[31673]: pgmap v132: 129 pgs: 129 active+clean; 24 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 227 MiB/s wr, 1.09k op/s 2024-03-21T10:42:53.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:53 smithi070 ceph-mon[36330]: pgmap v132: 129 pgs: 129 active+clean; 24 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 227 MiB/s wr, 1.09k op/s 2024-03-21T10:42:53.794 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:42:53.794 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:42:54.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.218+0000 7f63fb600640 1 -- 172.21.15.31:0/3534166050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63f4100560 msgr2=0x7f63f41009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:54.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.218+0000 7f63fb600640 1 --2- 172.21.15.31:0/3534166050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63f4100560 0x7f63f41009e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f63e0009930 tx=0x7f63e002bc30 comp rx=0 tx=0).stop 2024-03-21T10:42:54.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.218+0000 7f63fb600640 1 -- 172.21.15.31:0/3534166050 shutdown_connections 2024-03-21T10:42:54.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.218+0000 7f63fb600640 1 --2- 172.21.15.31:0/3534166050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63f4100f20 0x7f63f410d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.218+0000 7f63fb600640 1 --2- 172.21.15.31:0/3534166050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63f4100560 0x7f63f41009e0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.218+0000 7f63fb600640 1 --2- 172.21.15.31:0/3534166050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63f4106620 0x7f63f4106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.218+0000 7f63fb600640 1 -- 172.21.15.31:0/3534166050 >> 172.21.15.31:0/3534166050 conn(0x7f63f4077750 msgr2=0x7f63f4077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:54.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.218+0000 7f63fb600640 1 -- 172.21.15.31:0/3534166050 shutdown_connections 2024-03-21T10:42:54.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.218+0000 7f63fb600640 1 -- 172.21.15.31:0/3534166050 wait complete. 2024-03-21T10:42:54.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.220+0000 7f63fb600640 1 Processor -- start 2024-03-21T10:42:54.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.220+0000 7f63fb600640 1 -- start start 2024-03-21T10:42:54.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.220+0000 7f63fb600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63f4100560 0x7f63f419a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:54.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.221+0000 7f63fb600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63f4100f20 0x7f63f419a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:54.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.221+0000 7f63fb600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63f4106620 0x7f63f419e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:54.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.221+0000 7f63fb600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f4110930 con 0x7f63f4100560 2024-03-21T10:42:54.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.221+0000 7f63fb600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f41107b0 con 0x7f63f4100f20 2024-03-21T10:42:54.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.221+0000 7f63fb600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f4110ab0 con 0x7f63f4106620 2024-03-21T10:42:54.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.221+0000 7f63f3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63f4100f20 0x7f63f419a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:54.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.221+0000 7f63f3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63f4100f20 0x7f63f419a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58648/0 (socket says 172.21.15.31:58648) 2024-03-21T10:42:54.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.221+0000 7f63f3e00640 1 -- 172.21.15.31:0/1217447315 learned_addr learned my addr 172.21.15.31:0/1217447315 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:54.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.221+0000 7f63f9600640 1 --2- 172.21.15.31:0/1217447315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63f4106620 0x7f63f419e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:54.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.221+0000 7f63f8c00640 1 --2- 172.21.15.31:0/1217447315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63f4100560 0x7f63f419a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:54.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.221+0000 7f63f3e00640 1 -- 172.21.15.31:0/1217447315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63f4106620 msgr2=0x7f63f419e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:54.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.221+0000 7f63f3e00640 1 --2- 172.21.15.31:0/1217447315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63f4106620 0x7f63f419e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.221+0000 7f63f3e00640 1 -- 172.21.15.31:0/1217447315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63f4100560 msgr2=0x7f63f419a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:54.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.221+0000 7f63f3e00640 1 --2- 172.21.15.31:0/1217447315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63f4100560 0x7f63f419a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.221+0000 7f63f3e00640 1 -- 172.21.15.31:0/1217447315 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f419f100 con 0x7f63f4100f20 2024-03-21T10:42:54.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.221+0000 7f63f8c00640 1 --2- 172.21.15.31:0/1217447315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63f4100560 0x7f63f419a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:42:54.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.222+0000 7f63f3e00640 1 --2- 172.21.15.31:0/1217447315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63f4100f20 0x7f63f419a650 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f63e0009900 tx=0x7f63e00042e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:54.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.222+0000 7f63f1600640 1 -- 172.21.15.31:0/1217447315 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e0043020 con 0x7f63f4100f20 2024-03-21T10:42:54.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.222+0000 7f63f1600640 1 -- 172.21.15.31:0/1217447315 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f63e00044d0 con 0x7f63f4100f20 2024-03-21T10:42:54.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.222+0000 7f63f1600640 1 -- 172.21.15.31:0/1217447315 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e0003be0 con 0x7f63f4100f20 2024-03-21T10:42:54.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.222+0000 7f63fb600640 1 -- 172.21.15.31:0/1217447315 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f419f390 con 0x7f63f4100f20 2024-03-21T10:42:54.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.222+0000 7f63fb600640 1 -- 172.21.15.31:0/1217447315 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f41a6c80 con 0x7f63f4100f20 2024-03-21T10:42:54.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.226+0000 7f63fe40e740 1 Processor -- start 2024-03-21T10:42:54.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.226+0000 7f63fe40e740 1 -- start start 2024-03-21T10:42:54.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.227+0000 7f63fe40e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650854a4c00 0x5650854a3950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:54.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.227+0000 7f63fe40e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650854a3f20 0x5650854a43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:54.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.227+0000 7f63fe40e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650854ad290 0x5650854af680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:54.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.227+0000 7f63fe40e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x565085408250 con 0x5650854a3f20 2024-03-21T10:42:54.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.227+0000 7f63fe40e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650853b0a30 con 0x5650854a4c00 2024-03-21T10:42:54.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.227+0000 7f63fe40e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650854042f0 con 0x5650854ad290 2024-03-21T10:42:54.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.227+0000 7f63f9600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650854a4c00 0x5650854a3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:54.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.227+0000 7f63f8c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650854a3f20 0x5650854a43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:54.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.227+0000 7f63f9600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650854a4c00 0x5650854a3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58658/0 (socket says 172.21.15.31:58658) 2024-03-21T10:42:54.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.227+0000 7f63f9600640 1 -- 172.21.15.31:0/552814136 learned_addr learned my addr 172.21.15.31:0/552814136 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:54.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.227+0000 7f63f1600640 1 -- 172.21.15.31:0/1217447315 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f63e0046020 con 0x7f63f4100f20 2024-03-21T10:42:54.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.228+0000 7f63f3e00640 1 --2- 172.21.15.31:0/552814136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650854ad290 0x5650854af680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:54.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.228+0000 7f63f8c00640 1 -- 172.21.15.31:0/552814136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650854ad290 msgr2=0x5650854af680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:54.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.228+0000 7f63f8c00640 1 --2- 172.21.15.31:0/552814136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650854ad290 0x5650854af680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.228+0000 7f63f8c00640 1 -- 172.21.15.31:0/552814136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650854a4c00 msgr2=0x5650854a3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:54.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.228+0000 7f63f8c00640 1 --2- 172.21.15.31:0/552814136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650854a4c00 0x5650854a3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.228+0000 7f63f8c00640 1 -- 172.21.15.31:0/552814136 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56508539b690 con 0x5650854a3f20 2024-03-21T10:42:54.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.229+0000 7f63f3e00640 1 --2- 172.21.15.31:0/552814136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650854ad290 0x5650854af680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:42:54.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.229+0000 7f63f8c00640 1 --2- 172.21.15.31:0/552814136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650854a3f20 0x5650854a43a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f63e8009c40 tx=0x7f63e8010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d376052304c1ce83 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:54.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.229+0000 7f63f1600640 1 --2- 172.21.15.31:0/1217447315 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f63dc0a30d0 0x7f63dc0a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:54.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.229+0000 7f63e5c00640 1 -- 172.21.15.31:0/552814136 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e8004030 con 0x5650854a3f20 2024-03-21T10:42:54.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.230+0000 7f63e5c00640 1 -- 172.21.15.31:0/552814136 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f63e80041d0 con 0x5650854a3f20 2024-03-21T10:42:54.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.230+0000 7f63e5c00640 1 -- 172.21.15.31:0/552814136 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e8012740 con 0x5650854a3f20 2024-03-21T10:42:54.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.230+0000 7f63f1600640 1 -- 172.21.15.31:0/1217447315 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f63e00f1ac0 con 0x7f63f4100f20 2024-03-21T10:42:54.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.231+0000 7f63f9600640 1 --2- 172.21.15.31:0/1217447315 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f63dc0a30d0 0x7f63dc0a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:54.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.231+0000 7f63fe40e740 1 -- 172.21.15.31:0/552814136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650854a3f20 msgr2=0x5650854a43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:54.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.231+0000 7f63fe40e740 1 --2- 172.21.15.31:0/552814136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650854a3f20 0x5650854a43a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f63e8009c40 tx=0x7f63e8010740 comp rx=0 tx=0).stop 2024-03-21T10:42:54.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.231+0000 7f63fe40e740 1 -- 172.21.15.31:0/552814136 shutdown_connections 2024-03-21T10:42:54.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.231+0000 7f63fe40e740 1 --2- 172.21.15.31:0/552814136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650854ad290 0x5650854af680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.231+0000 7f63fe40e740 1 --2- 172.21.15.31:0/552814136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650854a3f20 0x5650854a43a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.231+0000 7f63fe40e740 1 --2- 172.21.15.31:0/552814136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650854a4c00 0x5650854a3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.231+0000 7f63fe40e740 1 -- 172.21.15.31:0/552814136 >> 172.21.15.31:0/552814136 conn(0x56508546b200 msgr2=0x56508549e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:54.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.231+0000 7f63fe40e740 1 -- 172.21.15.31:0/552814136 shutdown_connections 2024-03-21T10:42:54.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.232+0000 7f63f9600640 1 --2- 172.21.15.31:0/1217447315 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f63dc0a30d0 0x7f63dc0a5590 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f63ec012640 tx=0x7f63ec009500 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:54.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.232+0000 7f63fe40e740 1 -- 172.21.15.31:0/552814136 wait complete. 2024-03-21T10:42:54.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.232+0000 7f63fe40e740 1 -- start start 2024-03-21T10:42:54.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.232+0000 7f63fe40e740 1 Processor -- start 2024-03-21T10:42:54.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.233+0000 7f63fe40e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650854a3f20 0x5650854ac990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:54.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.233+0000 7f63fe40e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650854a4c00 0x5650854a9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:54.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.233+0000 7f63fe40e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650854ad290 0x5650854aa260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:54.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.233+0000 7f63fe40e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650854b0100 con 0x5650854ad290 2024-03-21T10:42:54.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.233+0000 7f63fe40e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650853c3e80 con 0x5650854a4c00 2024-03-21T10:42:54.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.233+0000 7f63fe40e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650854b0280 con 0x5650854a3f20 2024-03-21T10:42:54.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.233+0000 7f63fe40e740 10 client.0 _ll_register_callbacks cb 0x5650853d6130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:54.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.233+0000 7f63f3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650854a4c00 0x5650854a9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:54.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.234+0000 7f63f9600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650854ad290 0x5650854aa260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:54.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.234+0000 7f63f3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650854a4c00 0x5650854a9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58668/0 (socket says 172.21.15.31:58668) 2024-03-21T10:42:54.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.234+0000 7f63f3e00640 1 -- 172.21.15.31:0/1246343422 learned_addr learned my addr 172.21.15.31:0/1246343422 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:54.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.235+0000 7f63f8c00640 1 --2- 172.21.15.31:0/1246343422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650854a3f20 0x5650854ac990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:54.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.235+0000 7f63f9600640 1 -- 172.21.15.31:0/1246343422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650854a3f20 msgr2=0x5650854ac990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:54.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.235+0000 7f63f9600640 1 --2- 172.21.15.31:0/1246343422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650854a3f20 0x5650854ac990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.235+0000 7f63f9600640 1 -- 172.21.15.31:0/1246343422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650854a4c00 msgr2=0x5650854a9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:54.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.235+0000 7f63f9600640 1 --2- 172.21.15.31:0/1246343422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650854a4c00 0x5650854a9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.235+0000 7f63f9600640 1 -- 172.21.15.31:0/1246343422 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5650851e67d0 con 0x5650854ad290 2024-03-21T10:42:54.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.236+0000 7f63f9600640 1 --2- 172.21.15.31:0/1246343422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650854ad290 0x5650854aa260 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f63ec00fa70 tx=0x7f63ec004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=853f13a04b48e078 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:54.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.236+0000 7f63da000640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:42:54.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.237+0000 7f63da000640 1 -- 172.21.15.31:0/1246343422 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ec02d060 con 0x5650854ad290 2024-03-21T10:42:54.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.237+0000 7f63da000640 1 -- 172.21.15.31:0/1246343422 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f63ec018030 con 0x5650854ad290 2024-03-21T10:42:54.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.237+0000 7f63fe40e740 1 -- 172.21.15.31:0/1246343422 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f63e80041d0 con 0x5650854ad290 2024-03-21T10:42:54.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.237+0000 7f63da000640 1 -- 172.21.15.31:0/1246343422 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63ec034050 con 0x5650854ad290 2024-03-21T10:42:54.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.237+0000 7f63fe40e740 10 client.16335 fetch_fsmap learned FSMap version 26 2024-03-21T10:42:54.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.237+0000 7f63fe40e740 1 -- 172.21.15.31:0/1246343422 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x56508511c3f0 con 0x5650854ad290 2024-03-21T10:42:54.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.237+0000 7f63da000640 1 -- 172.21.15.31:0/1246343422 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f63e80041d0 con 0x5650854ad290 2024-03-21T10:42:54.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.238+0000 7f63da000640 1 -- 172.21.15.31:0/1246343422 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f63ec0289c0 con 0x5650854ad290 2024-03-21T10:42:54.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.238+0000 7f63fe40e740 10 client.16335 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:42:54.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.238+0000 7f63fe40e740 10 client.16335 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:42:54.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.239+0000 7f63fe40e740 20 client.16335 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:54.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.239+0000 7f63fe40e740 1 --2- 172.21.15.31:0/1246343422 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5650854c5a20 0x5650854c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:54.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.239+0000 7f63fe40e740 4 client.16335 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:42:54.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.239+0000 7f63fe40e740 1 -- 172.21.15.31:0/1246343422 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565085415790 con 0x5650854c5a20 2024-03-21T10:42:54.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.239+0000 7f63f8c00640 1 --2- 172.21.15.31:0/1246343422 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5650854c5a20 0x5650854c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:54.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.240+0000 7f63f8c00640 1 --2- 172.21.15.31:0/1246343422 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5650854c5a20 0x5650854c7e10 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dddadc1f1feb33aa server_cookie=1a60af301998d23b in_seq=0 out_seq=0 2024-03-21T10:42:54.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.240+0000 7f63da000640 10 client.16335 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:54.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.240+0000 7f63da000640 1 -- 172.21.15.31:0/1246343422 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x565085415790 con 0x5650854c5a20 2024-03-21T10:42:54.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.240+0000 7f63da000640 10 client.16335 handle_command_reply: tid=0 2024-03-21T10:42:54.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.241+0000 7f63fe40e740 1 client.16335 shutdown 2024-03-21T10:42:54.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.241+0000 7f63fe40e740 1 -- 172.21.15.31:0/1246343422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650854ad290 msgr2=0x5650854aa260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:54.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.241+0000 7f63fe40e740 1 --2- 172.21.15.31:0/1246343422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650854ad290 0x5650854aa260 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f63ec00fa70 tx=0x7f63ec004ab0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.242+0000 7f63fe40e740 1 -- 172.21.15.31:0/1246343422 shutdown_connections 2024-03-21T10:42:54.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.242+0000 7f63fe40e740 1 --2- 172.21.15.31:0/1246343422 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5650854c5a20 0x5650854c7e10 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.242+0000 7f63da000640 0 client.16335 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:54.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.242+0000 7f63fe40e740 1 --2- 172.21.15.31:0/1246343422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650854ad290 0x5650854aa260 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.242+0000 7f63fe40e740 1 --2- 172.21.15.31:0/1246343422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650854a4c00 0x5650854a9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.242+0000 7f63fe40e740 1 --2- 172.21.15.31:0/1246343422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650854a3f20 0x5650854ac990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.242+0000 7f63fe40e740 1 -- 172.21.15.31:0/1246343422 >> 172.21.15.31:0/1246343422 conn(0x56508546b200 msgr2=0x5650854a06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:54.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.242+0000 7f63fe40e740 1 -- 172.21.15.31:0/1246343422 shutdown_connections 2024-03-21T10:42:54.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.242+0000 7f63fe40e740 1 -- 172.21.15.31:0/1246343422 wait complete. 2024-03-21T10:42:54.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.242+0000 7f63fe40e740 20 client.16335 trim_cache size 0 max 16384 2024-03-21T10:42:54.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.264+0000 7f63fe40e740 1 Processor -- start 2024-03-21T10:42:54.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.266+0000 7f63fe40e740 1 -- start start 2024-03-21T10:42:54.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.267+0000 7f63fe40e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650854a3f20 0x5650854ac450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:54.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.267+0000 7f63fe40e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650854a4c00 0x5650854ac990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:54.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.267+0000 7f63fe40e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650854ad290 0x5650854c1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:54.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.267+0000 7f63fe40e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f41aebf0 con 0x5650854a4c00 2024-03-21T10:42:54.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.267+0000 7f63fe40e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f41ac6f0 con 0x5650854ad290 2024-03-21T10:42:54.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.267+0000 7f63fe40e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f41aa7b0 con 0x5650854a3f20 2024-03-21T10:42:54.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.267+0000 7f63f9600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650854ad290 0x5650854c1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:54.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.267+0000 7f63f9600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650854ad290 0x5650854c1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58676/0 (socket says 172.21.15.31:58676) 2024-03-21T10:42:54.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.267+0000 7f63f9600640 1 -- 172.21.15.31:0/3032973484 learned_addr learned my addr 172.21.15.31:0/3032973484 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:54.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.268+0000 7f63f9600640 1 -- 172.21.15.31:0/3032973484 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650854a3f20 msgr2=0x5650854ac450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:42:54.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.268+0000 7f63f9600640 1 --2- 172.21.15.31:0/3032973484 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650854a3f20 0x5650854ac450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.268+0000 7f63f9600640 1 -- 172.21.15.31:0/3032973484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650854a4c00 msgr2=0x5650854ac990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:42:54.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.268+0000 7f63f3e00640 1 --2- 172.21.15.31:0/3032973484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650854a4c00 0x5650854ac990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:54.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.269+0000 7f63f9600640 1 --2- 172.21.15.31:0/3032973484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650854a4c00 0x5650854ac990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.269+0000 7f63f9600640 1 -- 172.21.15.31:0/3032973484 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5650850dc0f0 con 0x5650854ad290 2024-03-21T10:42:54.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.269+0000 7f63f3e00640 1 --2- 172.21.15.31:0/3032973484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650854a4c00 0x5650854ac990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:42:54.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.269+0000 7f63f9600640 1 --2- 172.21.15.31:0/3032973484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650854ad290 0x5650854c1260 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f63ec012ed0 tx=0x7f63ec009fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4fc8a252bb4383e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:54.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.270+0000 7f63e5200640 1 -- 172.21.15.31:0/3032973484 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ec012040 con 0x5650854ad290 2024-03-21T10:42:54.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.270+0000 7f63e5200640 1 -- 172.21.15.31:0/3032973484 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f63ec0121e0 con 0x5650854ad290 2024-03-21T10:42:54.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.270+0000 7f63e5200640 1 -- 172.21.15.31:0/3032973484 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ec01dca0 con 0x5650854ad290 2024-03-21T10:42:54.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.271+0000 7f63fe40e740 1 -- 172.21.15.31:0/3032973484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650854ad290 msgr2=0x5650854c1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:54.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.271+0000 7f63fe40e740 1 --2- 172.21.15.31:0/3032973484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650854ad290 0x5650854c1260 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f63ec012ed0 tx=0x7f63ec009fb0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.272+0000 7f63fe40e740 1 -- 172.21.15.31:0/3032973484 shutdown_connections 2024-03-21T10:42:54.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.272+0000 7f63fe40e740 1 --2- 172.21.15.31:0/3032973484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650854ad290 0x5650854c1260 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.272+0000 7f63fe40e740 1 --2- 172.21.15.31:0/3032973484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650854a4c00 0x5650854ac990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.272+0000 7f63fe40e740 1 --2- 172.21.15.31:0/3032973484 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650854a3f20 0x5650854ac450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.272+0000 7f63fe40e740 1 -- 172.21.15.31:0/3032973484 >> 172.21.15.31:0/3032973484 conn(0x56508546b200 msgr2=0x56508549e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:54.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.272+0000 7f63fe40e740 1 -- 172.21.15.31:0/3032973484 shutdown_connections 2024-03-21T10:42:54.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.272+0000 7f63fe40e740 1 -- 172.21.15.31:0/3032973484 wait complete. 2024-03-21T10:42:54.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.272+0000 7f63fe40e740 1 -- start start 2024-03-21T10:42:54.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.273+0000 7f63fe40e740 1 Processor -- start 2024-03-21T10:42:54.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.273+0000 7f63fe40e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650854a3f20 0x5650854a6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:54.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.273+0000 7f63fe40e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650854a4c00 0x5650854a7500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:54.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.273+0000 7f63fe40e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650854ad290 0x5650854adc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:54.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.273+0000 7f63fe40e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f41aadb0 con 0x5650854a4c00 2024-03-21T10:42:54.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.273+0000 7f63fe40e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f41ac090 con 0x5650854a3f20 2024-03-21T10:42:54.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.273+0000 7f63fe40e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f41ab3d0 con 0x5650854ad290 2024-03-21T10:42:54.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.273+0000 7f63fe40e740 10 client.0 _ll_register_callbacks cb 0x7f63ec009da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:54.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.273+0000 7f63f9600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650854ad290 0x5650854adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:54.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.273+0000 7f63f9600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650854ad290 0x5650854adc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:41464/0 (socket says 172.21.15.31:41464) 2024-03-21T10:42:54.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.273+0000 7f63f9600640 1 -- 172.21.15.31:0/480034471 learned_addr learned my addr 172.21.15.31:0/480034471 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:54.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.273+0000 7f63f8c00640 1 --2- 172.21.15.31:0/480034471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650854a3f20 0x5650854a6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:54.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.274+0000 7f63f3e00640 1 --2- 172.21.15.31:0/480034471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650854a4c00 0x5650854a7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:54.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.274+0000 7f63f9600640 1 -- 172.21.15.31:0/480034471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650854a3f20 msgr2=0x5650854a6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:54.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.274+0000 7f63f9600640 1 --2- 172.21.15.31:0/480034471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650854a3f20 0x5650854a6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.274+0000 7f63f9600640 1 -- 172.21.15.31:0/480034471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650854a4c00 msgr2=0x5650854a7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:54.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.274+0000 7f63f9600640 1 --2- 172.21.15.31:0/480034471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650854a4c00 0x5650854a7500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.274+0000 7f63f9600640 1 -- 172.21.15.31:0/480034471 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5650854ae1b0 con 0x5650854ad290 2024-03-21T10:42:54.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.274+0000 7f63f3e00640 1 --2- 172.21.15.31:0/480034471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650854a4c00 0x5650854a7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:42:54.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.274+0000 7f63f9600640 1 --2- 172.21.15.31:0/480034471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650854ad290 0x5650854adc70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f63ec01ec60 tx=0x7f63ec01ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73117f8e375688f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:54.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.274+0000 7f63da000640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:42:54.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.274+0000 7f63da000640 1 -- 172.21.15.31:0/480034471 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ec012390 con 0x5650854ad290 2024-03-21T10:42:54.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.274+0000 7f63da000640 1 -- 172.21.15.31:0/480034471 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f63ec004c30 con 0x5650854ad290 2024-03-21T10:42:54.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.275+0000 7f63fe40e740 1 -- 172.21.15.31:0/480034471 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f63ec0121e0 con 0x5650854ad290 2024-03-21T10:42:54.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.275+0000 7f63da000640 1 -- 172.21.15.31:0/480034471 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63ec01c3a0 con 0x5650854ad290 2024-03-21T10:42:54.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.275+0000 7f63fe40e740 10 client.26662 fetch_fsmap learned FSMap version 26 2024-03-21T10:42:54.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.275+0000 7f63fe40e740 1 -- 172.21.15.31:0/480034471 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x565085521940 con 0x5650854ad290 2024-03-21T10:42:54.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.275+0000 7f63da000640 1 -- 172.21.15.31:0/480034471 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f63ec0121e0 con 0x5650854ad290 2024-03-21T10:42:54.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.275+0000 7f63da000640 1 -- 172.21.15.31:0/480034471 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f63ec01c740 con 0x5650854ad290 2024-03-21T10:42:54.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.275+0000 7f63fe40e740 10 client.26662 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:42:54.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.275+0000 7f63fe40e740 10 client.26662 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:42:54.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.275+0000 7f63fe40e740 20 client.26662 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:54.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.275+0000 7f63fe40e740 1 --2- 172.21.15.31:0/480034471 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5650854c5a20 0x565085523f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:54.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.275+0000 7f63fe40e740 4 client.26662 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:42:54.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.275+0000 7f63fe40e740 1 -- 172.21.15.31:0/480034471 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56508540fae0 con 0x5650854c5a20 2024-03-21T10:42:54.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.276+0000 7f63f8c00640 1 --2- 172.21.15.31:0/480034471 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5650854c5a20 0x565085523f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:54.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.276+0000 7f63f8c00640 1 --2- 172.21.15.31:0/480034471 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5650854c5a20 0x565085523f60 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9e1d31b6a517805 server_cookie=19ec9c6b9272a31 in_seq=0 out_seq=0 2024-03-21T10:42:54.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.276+0000 7f63da000640 10 client.26662 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:54.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.276+0000 7f63da000640 1 -- 172.21.15.31:0/480034471 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56508540fae0 con 0x5650854c5a20 2024-03-21T10:42:54.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.276+0000 7f63da000640 10 client.26662 handle_command_reply: tid=0 2024-03-21T10:42:54.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.276+0000 7f63fe40e740 1 client.26662 shutdown 2024-03-21T10:42:54.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.277+0000 7f63fe40e740 1 -- 172.21.15.31:0/480034471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650854ad290 msgr2=0x5650854adc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:54.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.277+0000 7f63fe40e740 1 --2- 172.21.15.31:0/480034471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650854ad290 0x5650854adc70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f63ec01ec60 tx=0x7f63ec01ec90 comp rx=0 tx=0).stop 2024-03-21T10:42:54.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.277+0000 7f63fe40e740 1 -- 172.21.15.31:0/480034471 shutdown_connections 2024-03-21T10:42:54.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.277+0000 7f63fe40e740 1 --2- 172.21.15.31:0/480034471 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5650854c5a20 0x565085523f60 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.278+0000 7f63fe40e740 1 --2- 172.21.15.31:0/480034471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650854ad290 0x5650854adc70 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.278+0000 7f63fe40e740 1 --2- 172.21.15.31:0/480034471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650854a4c00 0x5650854a7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.278+0000 7f63fe40e740 1 --2- 172.21.15.31:0/480034471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650854a3f20 0x5650854a6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.278+0000 7f63fe40e740 1 -- 172.21.15.31:0/480034471 >> 172.21.15.31:0/480034471 conn(0x56508546b200 msgr2=0x5650854a8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:54.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.278+0000 7f63da000640 0 client.26662 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:54.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.278+0000 7f63fe40e740 1 -- 172.21.15.31:0/480034471 shutdown_connections 2024-03-21T10:42:54.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.278+0000 7f63fe40e740 1 -- 172.21.15.31:0/480034471 wait complete. 2024-03-21T10:42:54.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.278+0000 7f63fe40e740 20 client.26662 trim_cache size 0 max 16384 2024-03-21T10:42:54.278 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:42:54.278 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-03-21T10:42:54.278 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-03-21T10:42:54.278 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:42:54.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.279+0000 7f63fb600640 1 -- 172.21.15.31:0/1217447315 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f63dc0a30d0 msgr2=0x7f63dc0a5590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:54.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.279+0000 7f63fb600640 1 --2- 172.21.15.31:0/1217447315 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f63dc0a30d0 0x7f63dc0a5590 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f63ec012640 tx=0x7f63ec009500 comp rx=0 tx=0).stop 2024-03-21T10:42:54.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.279+0000 7f63fb600640 1 -- 172.21.15.31:0/1217447315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63f4100f20 msgr2=0x7f63f419a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:54.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.279+0000 7f63fb600640 1 --2- 172.21.15.31:0/1217447315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63f4100f20 0x7f63f419a650 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f63e0009900 tx=0x7f63e00042e0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.279+0000 7f63fb600640 1 -- 172.21.15.31:0/1217447315 shutdown_connections 2024-03-21T10:42:54.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.279+0000 7f63fb600640 1 --2- 172.21.15.31:0/1217447315 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f63dc0a30d0 0x7f63dc0a5590 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.279+0000 7f63fb600640 1 --2- 172.21.15.31:0/1217447315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63f4106620 0x7f63f419e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.279+0000 7f63fb600640 1 --2- 172.21.15.31:0/1217447315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63f4100f20 0x7f63f419a650 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.279+0000 7f63fb600640 1 --2- 172.21.15.31:0/1217447315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63f4100560 0x7f63f419a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.279+0000 7f63fb600640 1 -- 172.21.15.31:0/1217447315 >> 172.21.15.31:0/1217447315 conn(0x7f63f4077750 msgr2=0x7f63f4068990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:54.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.279+0000 7f63fb600640 1 -- 172.21.15.31:0/1217447315 shutdown_connections 2024-03-21T10:42:54.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.279+0000 7f63fb600640 1 -- 172.21.15.31:0/1217447315 wait complete. 2024-03-21T10:42:54.294 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-21T10:42:54.295 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:42:54.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.576+0000 7f8487600640 1 -- 172.21.15.31:0/1664760587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84801147f0 msgr2=0x7f8480116be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:54.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.576+0000 7f8487600640 1 --2- 172.21.15.31:0/1664760587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84801147f0 0x7f8480116be0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f846c009860 tx=0x7f846c02bb70 comp rx=0 tx=0).stop 2024-03-21T10:42:54.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.576+0000 7f8487600640 1 -- 172.21.15.31:0/1664760587 shutdown_connections 2024-03-21T10:42:54.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.576+0000 7f8487600640 1 --2- 172.21.15.31:0/1664760587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84801147f0 0x7f8480116be0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.576+0000 7f8487600640 1 --2- 172.21.15.31:0/1664760587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8480108fc0 0x7f84801142b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.576+0000 7f8487600640 1 --2- 172.21.15.31:0/1664760587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84801086a0 0x7f8480108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.576+0000 7f8487600640 1 -- 172.21.15.31:0/1664760587 >> 172.21.15.31:0/1664760587 conn(0x7f848006da00 msgr2=0x7f848006de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:54.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.581+0000 7f8487600640 1 -- 172.21.15.31:0/1664760587 shutdown_connections 2024-03-21T10:42:54.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.581+0000 7f8487600640 1 -- 172.21.15.31:0/1664760587 wait complete. 2024-03-21T10:42:54.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.583+0000 7f8487600640 1 Processor -- start 2024-03-21T10:42:54.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.583+0000 7f8487600640 1 -- start start 2024-03-21T10:42:54.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.583+0000 7f8487600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84801086a0 0x7f84801a8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:54.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.583+0000 7f8487600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8480108fc0 0x7f84801a8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:54.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.584+0000 7f8487600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84801147f0 0x7f84801a2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:54.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.584+0000 7f8487600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848011a1f0 con 0x7f8480108fc0 2024-03-21T10:42:54.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.584+0000 7f8487600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848011a070 con 0x7f84801147f0 2024-03-21T10:42:54.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.584+0000 7f8487600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848011a370 con 0x7f84801086a0 2024-03-21T10:42:54.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.584+0000 7f8485600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84801147f0 0x7f84801a2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:54.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.584+0000 7f8485600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84801147f0 0x7f84801a2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58698/0 (socket says 172.21.15.31:58698) 2024-03-21T10:42:54.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.584+0000 7f8485600640 1 -- 172.21.15.31:0/2688156212 learned_addr learned my addr 172.21.15.31:0/2688156212 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:54.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.585+0000 7f8485600640 1 -- 172.21.15.31:0/2688156212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84801086a0 msgr2=0x7f84801a8640 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-21T10:42:54.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.585+0000 7f8485600640 1 --2- 172.21.15.31:0/2688156212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84801086a0 0x7f84801a8640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.585+0000 7f8485600640 1 -- 172.21.15.31:0/2688156212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8480108fc0 msgr2=0x7f84801a8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:54.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.585+0000 7f8485600640 1 --2- 172.21.15.31:0/2688156212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8480108fc0 0x7f84801a8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:54.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.585+0000 7f8485600640 1 -- 172.21.15.31:0/2688156212 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84801a2fc0 con 0x7f84801147f0 2024-03-21T10:42:54.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.586+0000 7f8485600640 1 --2- 172.21.15.31:0/2688156212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84801147f0 0x7f84801a2700 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f846c02c180 tx=0x7f846c0045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:54.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.588+0000 7f847d600640 1 -- 172.21.15.31:0/2688156212 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c041030 con 0x7f84801147f0 2024-03-21T10:42:54.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.588+0000 7f847d600640 1 -- 172.21.15.31:0/2688156212 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f846c02c510 con 0x7f84801147f0 2024-03-21T10:42:54.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.588+0000 7f847d600640 1 -- 172.21.15.31:0/2688156212 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c039040 con 0x7f84801147f0 2024-03-21T10:42:54.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.589+0000 7f8487600640 1 -- 172.21.15.31:0/2688156212 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84801a3250 con 0x7f84801147f0 2024-03-21T10:42:54.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.590+0000 7f8487600640 1 -- 172.21.15.31:0/2688156212 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84801af3e0 con 0x7f84801147f0 2024-03-21T10:42:54.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.594+0000 7f8487600640 1 -- 172.21.15.31:0/2688156212 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8480109d90 con 0x7f84801147f0 2024-03-21T10:42:54.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.595+0000 7f847d600640 1 -- 172.21.15.31:0/2688156212 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f846c044070 con 0x7f84801147f0 2024-03-21T10:42:54.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.597+0000 7f847d600640 1 --2- 172.21.15.31:0/2688156212 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f84640a31e0 0x7f84640a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:54.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.597+0000 7f847d600640 1 -- 172.21.15.31:0/2688156212 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f846c0f0400 con 0x7f84801147f0 2024-03-21T10:42:54.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.598+0000 7f8484c00640 1 --2- 172.21.15.31:0/2688156212 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f84640a31e0 0x7f84640a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:54.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.599+0000 7f8484c00640 1 --2- 172.21.15.31:0/2688156212 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f84640a31e0 0x7f84640a56a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8478005c40 tx=0x7f8478005bd0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:54.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:54.601+0000 7f847d600640 1 -- 172.21.15.31:0/2688156212 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f846c04a020 con 0x7f84801147f0 2024-03-21T10:42:55.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.120+0000 7f8487600640 1 -- 172.21.15.31:0/2688156212 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84801af5b0 con 0x7f84801147f0 2024-03-21T10:42:55.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.121+0000 7f847d600640 1 -- 172.21.15.31:0/2688156212 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+11384 (secure 0 0 0) 0x7f846c0b8020 con 0x7f84801147f0 2024-03-21T10:42:55.122 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:42:55.122 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:42:31.182549+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24547,24596]},"id":1}]} 2024-03-21T10:42:55.123 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2024-03-21T10:42:55.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.125+0000 7f8487600640 1 -- 172.21.15.31:0/2688156212 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f84640a31e0 msgr2=0x7f84640a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:55.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.125+0000 7f8487600640 1 --2- 172.21.15.31:0/2688156212 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f84640a31e0 0x7f84640a56a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8478005c40 tx=0x7f8478005bd0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.125+0000 7f8487600640 1 -- 172.21.15.31:0/2688156212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84801147f0 msgr2=0x7f84801a2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:55.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.125+0000 7f8487600640 1 --2- 172.21.15.31:0/2688156212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84801147f0 0x7f84801a2700 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f846c02c180 tx=0x7f846c0045e0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.126+0000 7f8487600640 1 -- 172.21.15.31:0/2688156212 shutdown_connections 2024-03-21T10:42:55.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.126+0000 7f8487600640 1 --2- 172.21.15.31:0/2688156212 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f84640a31e0 0x7f84640a56a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.126+0000 7f8487600640 1 --2- 172.21.15.31:0/2688156212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84801147f0 0x7f84801a2700 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.126+0000 7f8487600640 1 --2- 172.21.15.31:0/2688156212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8480108fc0 0x7f84801a8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.126+0000 7f8487600640 1 --2- 172.21.15.31:0/2688156212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84801086a0 0x7f84801a8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.126+0000 7f8487600640 1 -- 172.21.15.31:0/2688156212 >> 172.21.15.31:0/2688156212 conn(0x7f848006da00 msgr2=0x7f8480070100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:55.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.126+0000 7f8487600640 1 -- 172.21.15.31:0/2688156212 shutdown_connections 2024-03-21T10:42:55.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.126+0000 7f8487600640 1 -- 172.21.15.31:0/2688156212 wait complete. 2024-03-21T10:42:55.160 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:42:55.160 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-21T10:42:55.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:55 smithi040 ceph-mon[36154]: pgmap v133: 129 pgs: 129 active+clean; 24 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 196 MiB/s wr, 902 op/s 2024-03-21T10:42:55.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.537+0000 7f4495000640 1 -- 172.21.15.31:0/3630686717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44900764f0 msgr2=0x7f4490111de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:55.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.538+0000 7f4495000640 1 --2- 172.21.15.31:0/3630686717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44900764f0 0x7f4490111de0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f447c01ab70 tx=0x7f447c03cf00 comp rx=0 tx=0).stop 2024-03-21T10:42:55.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.543+0000 7f4495000640 1 -- 172.21.15.31:0/3630686717 shutdown_connections 2024-03-21T10:42:55.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.543+0000 7f4495000640 1 --2- 172.21.15.31:0/3630686717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4490112320 0x7f4490114710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.543+0000 7f4495000640 1 --2- 172.21.15.31:0/3630686717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44900764f0 0x7f4490111de0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.543+0000 7f4495000640 1 --2- 172.21.15.31:0/3630686717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4490075bd0 0x7f4490075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.543+0000 7f4495000640 1 -- 172.21.15.31:0/3630686717 >> 172.21.15.31:0/3630686717 conn(0x7f44900ff110 msgr2=0x7f4490101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:55.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.547+0000 7f4495000640 1 -- 172.21.15.31:0/3630686717 shutdown_connections 2024-03-21T10:42:55.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.547+0000 7f4495000640 1 -- 172.21.15.31:0/3630686717 wait complete. 2024-03-21T10:42:55.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.549+0000 7f4495000640 1 Processor -- start 2024-03-21T10:42:55.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.549+0000 7f4495000640 1 -- start start 2024-03-21T10:42:55.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.550+0000 7f4495000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4490075bd0 0x7f4490111be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:55.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.550+0000 7f4495000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44900764f0 0x7f449010dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:55.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.550+0000 7f4495000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4490112320 0x7f449010e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:55.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.551+0000 7f448ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44900764f0 0x7f449010dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:55.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.551+0000 7f448f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4490075bd0 0x7f4490111be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:55.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.551+0000 7f448f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4490075bd0 0x7f4490111be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:41494/0 (socket says 172.21.15.31:41494) 2024-03-21T10:42:55.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.552+0000 7f448fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4490112320 0x7f449010e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:55.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.552+0000 7f448fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4490112320 0x7f449010e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58714/0 (socket says 172.21.15.31:58714) 2024-03-21T10:42:55.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.552+0000 7f448ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44900764f0 0x7f449010dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43922/0 (socket says 172.21.15.31:43922) 2024-03-21T10:42:55.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.552+0000 7f448f400640 1 -- 172.21.15.31:0/3362081362 learned_addr learned my addr 172.21.15.31:0/3362081362 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:55.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.552+0000 7f4495000640 1 -- 172.21.15.31:0/3362081362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4490106020 con 0x7f44900764f0 2024-03-21T10:42:55.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.552+0000 7f4495000640 1 -- 172.21.15.31:0/3362081362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4490105ea0 con 0x7f4490112320 2024-03-21T10:42:55.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.552+0000 7f4495000640 1 -- 172.21.15.31:0/3362081362 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44901061a0 con 0x7f4490075bd0 2024-03-21T10:42:55.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.554+0000 7f448fe00640 1 -- 172.21.15.31:0/3362081362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4490075bd0 msgr2=0x7f4490111be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:55.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.554+0000 7f448fe00640 1 --2- 172.21.15.31:0/3362081362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4490075bd0 0x7f4490111be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.554+0000 7f448fe00640 1 -- 172.21.15.31:0/3362081362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44900764f0 msgr2=0x7f449010dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:55.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.554+0000 7f448fe00640 1 --2- 172.21.15.31:0/3362081362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44900764f0 0x7f449010dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.554+0000 7f448fe00640 1 -- 172.21.15.31:0/3362081362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449010ea40 con 0x7f4490112320 2024-03-21T10:42:55.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.555+0000 7f448fe00640 1 --2- 172.21.15.31:0/3362081362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4490112320 0x7f449010e1b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f44900691e0 tx=0x7f448000b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:55.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.555+0000 7f4487e00640 1 -- 172.21.15.31:0/3362081362 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4480015020 con 0x7f4490112320 2024-03-21T10:42:55.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.555+0000 7f4487e00640 1 -- 172.21.15.31:0/3362081362 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4480011d10 con 0x7f4490112320 2024-03-21T10:42:55.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.555+0000 7f4487e00640 1 -- 172.21.15.31:0/3362081362 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448000f2e0 con 0x7f4490112320 2024-03-21T10:42:55.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.555+0000 7f4495000640 1 -- 172.21.15.31:0/3362081362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44901a22b0 con 0x7f4490112320 2024-03-21T10:42:55.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.556+0000 7f4495000640 1 -- 172.21.15.31:0/3362081362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44901a2780 con 0x7f4490112320 2024-03-21T10:42:55.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:55 smithi031 ceph-mon[31673]: pgmap v133: 129 pgs: 129 active+clean; 24 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 196 MiB/s wr, 902 op/s 2024-03-21T10:42:55.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.560+0000 7f4499252740 1 Processor -- start 2024-03-21T10:42:55.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.561+0000 7f4487e00640 1 -- 172.21.15.31:0/3362081362 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f4480018020 con 0x7f4490112320 2024-03-21T10:42:55.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.563+0000 7f4487e00640 1 --2- 172.21.15.31:0/3362081362 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f44740a3090 0x7f44740a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:55.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.563+0000 7f4487e00640 1 -- 172.21.15.31:0/3362081362 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f44800cf210 con 0x7f4490112320 2024-03-21T10:42:55.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.564+0000 7f448ea00640 1 --2- 172.21.15.31:0/3362081362 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f44740a3090 0x7f44740a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:55.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.564+0000 7f4499252740 1 -- start start 2024-03-21T10:42:55.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.564+0000 7f4499252740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ed0e1b68b0 0x55ed0e1b6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:55.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.564+0000 7f4499252740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0e1b7260 0x55ed0e1bf810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:55.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.564+0000 7f4499252740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed0e1bfde0 0x55ed0e1c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:55.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.565+0000 7f4499252740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed0e11a2c0 con 0x55ed0e1b68b0 2024-03-21T10:42:55.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.565+0000 7f4499252740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed0e0c2c90 con 0x55ed0e1bfde0 2024-03-21T10:42:55.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.565+0000 7f4499252740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed0e116360 con 0x55ed0e1b7260 2024-03-21T10:42:55.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.565+0000 7f448fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed0e1bfde0 0x55ed0e1c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:55.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.565+0000 7f448fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed0e1bfde0 0x55ed0e1c2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58718/0 (socket says 172.21.15.31:58718) 2024-03-21T10:42:55.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.565+0000 7f448fe00640 1 -- 172.21.15.31:0/1665153535 learned_addr learned my addr 172.21.15.31:0/1665153535 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:55.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.565+0000 7f448f400640 1 --2- 172.21.15.31:0/1665153535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ed0e1b68b0 0x55ed0e1b6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:55.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.565+0000 7f448fe00640 1 -- 172.21.15.31:0/1665153535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0e1b7260 msgr2=0x55ed0e1bf810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:42:55.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.565+0000 7f448fe00640 1 --2- 172.21.15.31:0/1665153535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0e1b7260 0x55ed0e1bf810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.565+0000 7f448fe00640 1 -- 172.21.15.31:0/1665153535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ed0e1b68b0 msgr2=0x55ed0e1b6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:55.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.565+0000 7f448fe00640 1 --2- 172.21.15.31:0/1665153535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ed0e1b68b0 0x55ed0e1b6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.565+0000 7f448fe00640 1 -- 172.21.15.31:0/1665153535 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed0e0e7080 con 0x55ed0e1bfde0 2024-03-21T10:42:55.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.566+0000 7f448ea00640 1 --2- 172.21.15.31:0/3362081362 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f44740a3090 0x7f44740a5550 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f447c006010 tx=0x7f447c03f040 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:55.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.566+0000 7f448fe00640 1 --2- 172.21.15.31:0/1665153535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed0e1bfde0 0x55ed0e1c2230 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4480097da0 tx=0x7f448009ad30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e84fdeb406304a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:55.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.567+0000 7f4484c00640 1 -- 172.21.15.31:0/1665153535 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44800b4070 con 0x55ed0e1bfde0 2024-03-21T10:42:55.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.567+0000 7f4484c00640 1 -- 172.21.15.31:0/1665153535 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f448009f520 con 0x55ed0e1bfde0 2024-03-21T10:42:55.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.567+0000 7f4484c00640 1 -- 172.21.15.31:0/1665153535 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448009f170 con 0x55ed0e1bfde0 2024-03-21T10:42:55.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.568+0000 7f4499252740 1 -- 172.21.15.31:0/1665153535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed0e1bfde0 msgr2=0x55ed0e1c2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:55.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.568+0000 7f4499252740 1 --2- 172.21.15.31:0/1665153535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed0e1bfde0 0x55ed0e1c2230 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4480097da0 tx=0x7f448009ad30 comp rx=0 tx=0).stop 2024-03-21T10:42:55.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.568+0000 7f4499252740 1 -- 172.21.15.31:0/1665153535 shutdown_connections 2024-03-21T10:42:55.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.568+0000 7f4499252740 1 --2- 172.21.15.31:0/1665153535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed0e1bfde0 0x55ed0e1c2230 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.568+0000 7f4499252740 1 --2- 172.21.15.31:0/1665153535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0e1b7260 0x55ed0e1bf810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.568+0000 7f4499252740 1 --2- 172.21.15.31:0/1665153535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ed0e1b68b0 0x55ed0e1b6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.568+0000 7f4499252740 1 -- 172.21.15.31:0/1665153535 >> 172.21.15.31:0/1665153535 conn(0x55ed0e1b0f80 msgr2=0x55ed0e1b1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:55.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.569+0000 7f4499252740 1 -- 172.21.15.31:0/1665153535 shutdown_connections 2024-03-21T10:42:55.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.569+0000 7f4499252740 1 -- 172.21.15.31:0/1665153535 wait complete. 2024-03-21T10:42:55.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.569+0000 7f4499252740 1 -- start start 2024-03-21T10:42:55.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.570+0000 7f4499252740 1 Processor -- start 2024-03-21T10:42:55.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.570+0000 7f4499252740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0e1b68b0 0x55ed0e1bf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:55.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.570+0000 7f4499252740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed0e1b7260 0x55ed0e1bcad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:55.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.570+0000 7f4499252740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ed0e1bfde0 0x55ed0e1bd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:55.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.570+0000 7f4499252740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed0e1c2d50 con 0x55ed0e1bfde0 2024-03-21T10:42:55.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.570+0000 7f4499252740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed0e0d60d0 con 0x55ed0e1b7260 2024-03-21T10:42:55.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.570+0000 7f4499252740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed0e1c2ed0 con 0x55ed0e1b68b0 2024-03-21T10:42:55.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.570+0000 7f4499252740 10 client.0 _ll_register_callbacks cb 0x55ed0e0dae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:55.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.571+0000 7f448f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0e1b68b0 0x55ed0e1bf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:55.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.571+0000 7f448f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0e1b68b0 0x55ed0e1bf740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:41506/0 (socket says 172.21.15.31:41506) 2024-03-21T10:42:55.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.571+0000 7f448f400640 1 -- 172.21.15.31:0/1684146035 learned_addr learned my addr 172.21.15.31:0/1684146035 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:55.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.571+0000 7f448ea00640 1 --2- 172.21.15.31:0/1684146035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed0e1b7260 0x55ed0e1bcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:55.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.571+0000 7f448fe00640 1 --2- 172.21.15.31:0/1684146035 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ed0e1bfde0 0x55ed0e1bd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:55.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.571+0000 7f448ea00640 1 -- 172.21.15.31:0/1684146035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0e1b68b0 msgr2=0x55ed0e1bf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:55.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.571+0000 7f448ea00640 1 --2- 172.21.15.31:0/1684146035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0e1b68b0 0x55ed0e1bf740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.571+0000 7f448ea00640 1 -- 172.21.15.31:0/1684146035 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ed0e1bfde0 msgr2=0x55ed0e1bd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:55.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.571+0000 7f448ea00640 1 --2- 172.21.15.31:0/1684146035 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ed0e1bfde0 0x55ed0e1bd0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.571+0000 7f448ea00640 1 -- 172.21.15.31:0/1684146035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed0de2e3f0 con 0x55ed0e1b7260 2024-03-21T10:42:55.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.572+0000 7f448ea00640 1 --2- 172.21.15.31:0/1684146035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed0e1b7260 0x55ed0e1bcad0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f447c000c00 tx=0x7f447c01a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74adc058a3a8a302 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:55.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.572+0000 7f4479600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:42:55.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.572+0000 7f4479600640 1 -- 172.21.15.31:0/1684146035 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447c05f020 con 0x55ed0e1b7260 2024-03-21T10:42:55.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.572+0000 7f4479600640 1 -- 172.21.15.31:0/1684146035 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f447c003d40 con 0x55ed0e1b7260 2024-03-21T10:42:55.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.573+0000 7f4499252740 1 -- 172.21.15.31:0/1684146035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f448009f520 con 0x55ed0e1b7260 2024-03-21T10:42:55.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.573+0000 7f4479600640 1 -- 172.21.15.31:0/1684146035 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f447c0454e0 con 0x55ed0e1b7260 2024-03-21T10:42:55.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.573+0000 7f4499252740 10 client.26729 fetch_fsmap learned FSMap version 26 2024-03-21T10:42:55.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.573+0000 7f4499252740 1 -- 172.21.15.31:0/1684146035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55ed0def87d0 con 0x55ed0e1b7260 2024-03-21T10:42:55.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.573+0000 7f4479600640 1 -- 172.21.15.31:0/1684146035 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f448009f520 con 0x55ed0e1b7260 2024-03-21T10:42:55.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.573+0000 7f4479600640 1 -- 172.21.15.31:0/1684146035 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f447c04f840 con 0x55ed0e1b7260 2024-03-21T10:42:55.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.574+0000 7f4499252740 10 client.26729 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:42:55.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.574+0000 7f4499252740 10 client.26729 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:42:55.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.574+0000 7f4499252740 20 client.26729 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:55.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.574+0000 7f4499252740 1 --2- 172.21.15.31:0/1684146035 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55ed0e1cfc40 0x55ed0e1d2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:55.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.574+0000 7f4499252740 4 client.26729 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:42:55.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.574+0000 7f4499252740 1 -- 172.21.15.31:0/1684146035 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ed0e127800 con 0x55ed0e1cfc40 2024-03-21T10:42:55.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.574+0000 7f448f400640 1 --2- 172.21.15.31:0/1684146035 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55ed0e1cfc40 0x55ed0e1d2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:55.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.575+0000 7f4479600640 10 client.26729 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:42:55.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:55 smithi070 ceph-mon[36330]: pgmap v133: 129 pgs: 129 active+clean; 24 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 196 MiB/s wr, 902 op/s 2024-03-21T10:42:55.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.579+0000 7f448f400640 1 --2- 172.21.15.31:0/1684146035 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55ed0e1cfc40 0x55ed0e1d2030 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1ad57aa4ff783fdf server_cookie=8b22f7bb1ce2a751 in_seq=0 out_seq=0 2024-03-21T10:42:55.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.581+0000 7f4479600640 1 -- 172.21.15.31:0/1684146035 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ed0e127800 con 0x55ed0e1cfc40 2024-03-21T10:42:55.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.581+0000 7f4479600640 10 client.26729 handle_command_reply: tid=0 2024-03-21T10:42:55.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.581+0000 7f4499252740 1 client.26729 shutdown 2024-03-21T10:42:55.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.581+0000 7f4499252740 1 -- 172.21.15.31:0/1684146035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed0e1b7260 msgr2=0x55ed0e1bcad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:55.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.581+0000 7f4499252740 1 --2- 172.21.15.31:0/1684146035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed0e1b7260 0x55ed0e1bcad0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f447c000c00 tx=0x7f447c01a270 comp rx=0 tx=0).stop 2024-03-21T10:42:55.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.581+0000 7f4499252740 1 -- 172.21.15.31:0/1684146035 shutdown_connections 2024-03-21T10:42:55.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.581+0000 7f4499252740 1 --2- 172.21.15.31:0/1684146035 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55ed0e1cfc40 0x55ed0e1d2030 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.581+0000 7f4499252740 1 --2- 172.21.15.31:0/1684146035 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ed0e1bfde0 0x55ed0e1bd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.581+0000 7f4499252740 1 --2- 172.21.15.31:0/1684146035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed0e1b7260 0x55ed0e1bcad0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.581+0000 7f4499252740 1 --2- 172.21.15.31:0/1684146035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0e1b68b0 0x55ed0e1bf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.581+0000 7f4499252740 1 -- 172.21.15.31:0/1684146035 >> 172.21.15.31:0/1684146035 conn(0x55ed0e1b0f80 msgr2=0x55ed0e1b2cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:55.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.582+0000 7f4479600640 0 client.26729 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:42:55.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.582+0000 7f4499252740 1 -- 172.21.15.31:0/1684146035 shutdown_connections 2024-03-21T10:42:55.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.582+0000 7f4499252740 1 -- 172.21.15.31:0/1684146035 wait complete. 2024-03-21T10:42:55.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.582+0000 7f4499252740 20 client.26729 trim_cache size 0 max 16384 2024-03-21T10:42:55.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.611+0000 7f4499252740 1 Processor -- start 2024-03-21T10:42:55.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.611+0000 7f4499252740 1 -- start start 2024-03-21T10:42:55.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.612+0000 7f4499252740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed0e1b68b0 0x55ed0e1bb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:55.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.612+0000 7f4499252740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0e1b7260 0x55ed0e1bf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:55.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.612+0000 7f4499252740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ed0e1bfde0 0x55ed0e1bf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:55.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.612+0000 7f4499252740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44901aeab0 con 0x55ed0e1bfde0 2024-03-21T10:42:55.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.613+0000 7f4499252740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44901ac5b0 con 0x55ed0e1b68b0 2024-03-21T10:42:55.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.613+0000 7f4499252740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44901aa610 con 0x55ed0e1b7260 2024-03-21T10:42:55.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.613+0000 7f448fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ed0e1bfde0 0x55ed0e1bf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:55.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.615+0000 7f448fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ed0e1bfde0 0x55ed0e1bf740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43944/0 (socket says 172.21.15.31:43944) 2024-03-21T10:42:55.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.615+0000 7f448fe00640 1 -- 172.21.15.31:0/1763488227 learned_addr learned my addr 172.21.15.31:0/1763488227 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:55.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.619+0000 7f448ea00640 1 --2- 172.21.15.31:0/1763488227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0e1b7260 0x55ed0e1bf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:55.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.621+0000 7f448fe00640 1 -- 172.21.15.31:0/1763488227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0e1b7260 msgr2=0x55ed0e1bf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:55.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.621+0000 7f448fe00640 1 --2- 172.21.15.31:0/1763488227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0e1b7260 0x55ed0e1bf200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.621+0000 7f448fe00640 1 -- 172.21.15.31:0/1763488227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed0e1b68b0 msgr2=0x55ed0e1bb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:55.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.621+0000 7f448fe00640 1 --2- 172.21.15.31:0/1763488227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed0e1b68b0 0x55ed0e1bb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.621+0000 7f448fe00640 1 -- 172.21.15.31:0/1763488227 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed0e1bba80 con 0x55ed0e1bfde0 2024-03-21T10:42:55.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.621+0000 7f448ea00640 1 --2- 172.21.15.31:0/1763488227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0e1b7260 0x55ed0e1bf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:42:55.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.628+0000 7f448fe00640 1 --2- 172.21.15.31:0/1763488227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ed0e1bfde0 0x55ed0e1bf740 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4480000c00 tx=0x7f4480010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5637393f2e46bf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:55.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.631+0000 7f447be00640 1 -- 172.21.15.31:0/1763488227 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44800b4040 con 0x55ed0e1bfde0 2024-03-21T10:42:55.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.632+0000 7f447be00640 1 -- 172.21.15.31:0/1763488227 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44800996d0 con 0x55ed0e1bfde0 2024-03-21T10:42:55.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.632+0000 7f447be00640 1 -- 172.21.15.31:0/1763488227 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44800b4040 con 0x55ed0e1bfde0 2024-03-21T10:42:55.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.633+0000 7f4499252740 1 -- 172.21.15.31:0/1763488227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ed0e1bfde0 msgr2=0x55ed0e1bf740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:55.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.633+0000 7f4499252740 1 --2- 172.21.15.31:0/1763488227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ed0e1bfde0 0x55ed0e1bf740 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4480000c00 tx=0x7f4480010740 comp rx=0 tx=0).stop 2024-03-21T10:42:55.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.633+0000 7f4499252740 1 -- 172.21.15.31:0/1763488227 shutdown_connections 2024-03-21T10:42:55.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.633+0000 7f4499252740 1 --2- 172.21.15.31:0/1763488227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ed0e1bfde0 0x55ed0e1bf740 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.633+0000 7f4499252740 1 --2- 172.21.15.31:0/1763488227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0e1b7260 0x55ed0e1bf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.633+0000 7f4499252740 1 --2- 172.21.15.31:0/1763488227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed0e1b68b0 0x55ed0e1bb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.633+0000 7f4499252740 1 -- 172.21.15.31:0/1763488227 >> 172.21.15.31:0/1763488227 conn(0x55ed0e1b0f80 msgr2=0x55ed0e1b0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:55.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.634+0000 7f4499252740 1 -- 172.21.15.31:0/1763488227 shutdown_connections 2024-03-21T10:42:55.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.634+0000 7f4499252740 1 -- 172.21.15.31:0/1763488227 wait complete. 2024-03-21T10:42:55.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.635+0000 7f4499252740 1 -- start start 2024-03-21T10:42:55.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.635+0000 7f4499252740 1 Processor -- start 2024-03-21T10:42:55.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.636+0000 7f4499252740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ed0e1b68b0 0x55ed0e1c7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:55.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.636+0000 7f4499252740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0e1b7260 0x55ed0e1c5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:55.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.637+0000 7f4499252740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed0e1bfde0 0x55ed0e1c6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:55.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.637+0000 7f448f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ed0e1b68b0 0x55ed0e1c7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:55.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.637+0000 7f4499252740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44901aac30 con 0x55ed0e1b68b0 2024-03-21T10:42:55.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.637+0000 7f4499252740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44901abf50 con 0x55ed0e1bfde0 2024-03-21T10:42:55.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.637+0000 7f4499252740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44901ab290 con 0x55ed0e1b7260 2024-03-21T10:42:55.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.637+0000 7f448fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed0e1bfde0 0x55ed0e1c6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:55.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.637+0000 7f448fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed0e1bfde0 0x55ed0e1c6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58754/0 (socket says 172.21.15.31:58754) 2024-03-21T10:42:55.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.637+0000 7f448fe00640 1 -- 172.21.15.31:0/3040516795 learned_addr learned my addr 172.21.15.31:0/3040516795 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:55.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.637+0000 7f4499252740 10 client.0 _ll_register_callbacks cb 0x7f447c005a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:55.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.638+0000 7f448fe00640 1 -- 172.21.15.31:0/3040516795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0e1b7260 msgr2=0x55ed0e1c5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:55.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.638+0000 7f448fe00640 1 --2- 172.21.15.31:0/3040516795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0e1b7260 0x55ed0e1c5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.638+0000 7f448fe00640 1 -- 172.21.15.31:0/3040516795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ed0e1b68b0 msgr2=0x55ed0e1c7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:55.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.638+0000 7f448fe00640 1 --2- 172.21.15.31:0/3040516795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ed0e1b68b0 0x55ed0e1c7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.638+0000 7f448fe00640 1 -- 172.21.15.31:0/3040516795 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed0e1c6990 con 0x55ed0e1bfde0 2024-03-21T10:42:55.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.638+0000 7f448f400640 1 --2- 172.21.15.31:0/3040516795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ed0e1b68b0 0x55ed0e1c7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:42:55.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.639+0000 7f448fe00640 1 --2- 172.21.15.31:0/3040516795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed0e1bfde0 0x55ed0e1c6450 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f4480010450 tx=0x7f44800108e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dac7cc24fb3b6299 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:55.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.639+0000 7f4479600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:42:55.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.639+0000 7f4479600640 1 -- 172.21.15.31:0/3040516795 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44800b4040 con 0x55ed0e1bfde0 2024-03-21T10:42:55.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.639+0000 7f4479600640 1 -- 172.21.15.31:0/3040516795 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f448009f170 con 0x55ed0e1bfde0 2024-03-21T10:42:55.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.640+0000 7f4499252740 1 -- 172.21.15.31:0/3040516795 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f44800996d0 con 0x55ed0e1bfde0 2024-03-21T10:42:55.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.641+0000 7f4479600640 1 -- 172.21.15.31:0/3040516795 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f44800b41e0 con 0x55ed0e1bfde0 2024-03-21T10:42:55.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.641+0000 7f4499252740 10 client.26732 fetch_fsmap learned FSMap version 26 2024-03-21T10:42:55.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.641+0000 7f4499252740 1 -- 172.21.15.31:0/3040516795 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55ed0e22c110 con 0x55ed0e1bfde0 2024-03-21T10:42:55.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.643+0000 7f4479600640 1 -- 172.21.15.31:0/3040516795 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44800996d0 con 0x55ed0e1bfde0 2024-03-21T10:42:55.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.644+0000 7f4479600640 1 -- 172.21.15.31:0/3040516795 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f448007d070 con 0x55ed0e1bfde0 2024-03-21T10:42:55.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.644+0000 7f4499252740 10 client.26732 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:42:55.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.644+0000 7f4499252740 10 client.26732 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:42:55.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.644+0000 7f4499252740 20 client.26732 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:55.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.644+0000 7f4499252740 1 --2- 172.21.15.31:0/3040516795 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55ed0e22c2e0 0x55ed0e22e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:55.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.644+0000 7f4499252740 4 client.26732 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:42:55.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.644+0000 7f4499252740 1 -- 172.21.15.31:0/3040516795 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ed0e121b50 con 0x55ed0e22c2e0 2024-03-21T10:42:55.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.645+0000 7f448f400640 1 --2- 172.21.15.31:0/3040516795 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55ed0e22c2e0 0x55ed0e22e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:55.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.646+0000 7f448f400640 1 --2- 172.21.15.31:0/3040516795 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55ed0e22c2e0 0x55ed0e22e6d0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3df145b5de6b0b72 server_cookie=36721ab5a0a7541c in_seq=0 out_seq=0 2024-03-21T10:42:55.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.646+0000 7f4479600640 10 client.26732 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:42:55.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.646+0000 7f4479600640 1 -- 172.21.15.31:0/3040516795 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ed0e121b50 con 0x55ed0e22c2e0 2024-03-21T10:42:55.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.646+0000 7f4479600640 10 client.26732 handle_command_reply: tid=0 2024-03-21T10:42:55.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.646+0000 7f4499252740 1 client.26732 shutdown 2024-03-21T10:42:55.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.648+0000 7f4499252740 1 -- 172.21.15.31:0/3040516795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed0e1bfde0 msgr2=0x55ed0e1c6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:55.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.648+0000 7f4499252740 1 --2- 172.21.15.31:0/3040516795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed0e1bfde0 0x55ed0e1c6450 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f4480010450 tx=0x7f44800108e0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.649+0000 7f4499252740 1 -- 172.21.15.31:0/3040516795 shutdown_connections 2024-03-21T10:42:55.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.649+0000 7f4499252740 1 --2- 172.21.15.31:0/3040516795 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55ed0e22c2e0 0x55ed0e22e6d0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.649+0000 7f4499252740 1 --2- 172.21.15.31:0/3040516795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed0e1bfde0 0x55ed0e1c6450 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.649+0000 7f4479600640 0 client.26732 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:42:55.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.649+0000 7f4499252740 1 --2- 172.21.15.31:0/3040516795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0e1b7260 0x55ed0e1c5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.649+0000 7f4499252740 1 --2- 172.21.15.31:0/3040516795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ed0e1b68b0 0x55ed0e1c7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.649+0000 7f4499252740 1 -- 172.21.15.31:0/3040516795 >> 172.21.15.31:0/3040516795 conn(0x55ed0e1b0f80 msgr2=0x55ed0e1b7640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:55.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.651+0000 7f4499252740 1 -- 172.21.15.31:0/3040516795 shutdown_connections 2024-03-21T10:42:55.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.653+0000 7f4499252740 1 -- 172.21.15.31:0/3040516795 wait complete. 2024-03-21T10:42:55.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.653+0000 7f4499252740 20 client.26732 trim_cache size 0 max 16384 2024-03-21T10:42:55.653 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:42:55.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.656+0000 7f4495000640 1 -- 172.21.15.31:0/3362081362 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f44740a3090 msgr2=0x7f44740a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:55.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.656+0000 7f4495000640 1 --2- 172.21.15.31:0/3362081362 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f44740a3090 0x7f44740a5550 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f447c006010 tx=0x7f447c03f040 comp rx=0 tx=0).stop 2024-03-21T10:42:55.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.656+0000 7f4495000640 1 -- 172.21.15.31:0/3362081362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4490112320 msgr2=0x7f449010e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:55.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.657+0000 7f4495000640 1 --2- 172.21.15.31:0/3362081362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4490112320 0x7f449010e1b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f44900691e0 tx=0x7f448000b870 comp rx=0 tx=0).stop 2024-03-21T10:42:55.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.657+0000 7f4495000640 1 -- 172.21.15.31:0/3362081362 shutdown_connections 2024-03-21T10:42:55.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.657+0000 7f4495000640 1 --2- 172.21.15.31:0/3362081362 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f44740a3090 0x7f44740a5550 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.657+0000 7f4495000640 1 --2- 172.21.15.31:0/3362081362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4490112320 0x7f449010e1b0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.657+0000 7f4495000640 1 --2- 172.21.15.31:0/3362081362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44900764f0 0x7f449010dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.657+0000 7f4495000640 1 --2- 172.21.15.31:0/3362081362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4490075bd0 0x7f4490111be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:55.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.658+0000 7f4495000640 1 -- 172.21.15.31:0/3362081362 >> 172.21.15.31:0/3362081362 conn(0x7f44900ff110 msgr2=0x7f4490101500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:55.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.658+0000 7f4495000640 1 -- 172.21.15.31:0/3362081362 shutdown_connections 2024-03-21T10:42:55.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:55.659+0000 7f4495000640 1 -- 172.21.15.31:0/3362081362 wait complete. 2024-03-21T10:42:55.692 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:42:55.692 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-21T10:42:56.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.081+0000 7fc633e00640 1 -- 172.21.15.31:0/734157745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc634102630 msgr2=0x7fc634102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:56.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.081+0000 7fc633e00640 1 --2- 172.21.15.31:0/734157745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc634102630 0x7fc634102a10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc620009860 tx=0x7fc62002bc50 comp rx=0 tx=0).stop 2024-03-21T10:42:56.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.081+0000 7fc633e00640 1 -- 172.21.15.31:0/734157745 shutdown_connections 2024-03-21T10:42:56.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.081+0000 7fc633e00640 1 --2- 172.21.15.31:0/734157745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc63410ba50 0x7fc63410ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.081+0000 7fc633e00640 1 --2- 172.21.15.31:0/734157745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc634102f50 0x7fc63410b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.082+0000 7fc633e00640 1 --2- 172.21.15.31:0/734157745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc634102630 0x7fc634102a10 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.082+0000 7fc633e00640 1 -- 172.21.15.31:0/734157745 >> 172.21.15.31:0/734157745 conn(0x7fc634077130 msgr2=0x7fc6340fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:56.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.083+0000 7fc633e00640 1 -- 172.21.15.31:0/734157745 shutdown_connections 2024-03-21T10:42:56.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.083+0000 7fc633e00640 1 -- 172.21.15.31:0/734157745 wait complete. 2024-03-21T10:42:56.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.085+0000 7fc633e00640 1 Processor -- start 2024-03-21T10:42:56.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.085+0000 7fc633e00640 1 -- start start 2024-03-21T10:42:56.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.086+0000 7fc633e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc634102630 0x7fc6341002a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.086+0000 7fc633e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc634102f50 0x7fc6341007e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.086+0000 7fc633e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc63410ba50 0x7fc634100d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.086+0000 7fc633e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc634111850 con 0x7fc634102630 2024-03-21T10:42:56.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.086+0000 7fc633e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6341116d0 con 0x7fc634102f50 2024-03-21T10:42:56.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.086+0000 7fc633e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6341119d0 con 0x7fc63410ba50 2024-03-21T10:42:56.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.086+0000 7fc633400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc63410ba50 0x7fc634100d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.086+0000 7fc633400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc63410ba50 0x7fc634100d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:41544/0 (socket says 172.21.15.31:41544) 2024-03-21T10:42:56.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.086+0000 7fc633400640 1 -- 172.21.15.31:0/1877393641 learned_addr learned my addr 172.21.15.31:0/1877393641 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:56.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.087+0000 7fc632000640 1 --2- 172.21.15.31:0/1877393641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc634102f50 0x7fc6341007e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.087+0000 7fc632000640 1 -- 172.21.15.31:0/1877393641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc63410ba50 msgr2=0x7fc634100d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:56.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.087+0000 7fc632000640 1 --2- 172.21.15.31:0/1877393641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc63410ba50 0x7fc634100d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.087+0000 7fc632000640 1 -- 172.21.15.31:0/1877393641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc634102630 msgr2=0x7fc6341002a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T10:42:56.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.088+0000 7fc632a00640 1 --2- 172.21.15.31:0/1877393641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc634102630 0x7fc6341002a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.088+0000 7fc632000640 1 --2- 172.21.15.31:0/1877393641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc634102630 0x7fc6341002a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.088+0000 7fc632000640 1 -- 172.21.15.31:0/1877393641 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc63419bef0 con 0x7fc634102f50 2024-03-21T10:42:56.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.088+0000 7fc632000640 1 --2- 172.21.15.31:0/1877393641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc634102f50 0x7fc6341007e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fc62400e790 tx=0x7fc62400ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:56.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.088+0000 7fc62b400640 1 -- 172.21.15.31:0/1877393641 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62400eea0 con 0x7fc634102f50 2024-03-21T10:42:56.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.088+0000 7fc633e00640 1 -- 172.21.15.31:0/1877393641 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc63419c1e0 con 0x7fc634102f50 2024-03-21T10:42:56.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.089+0000 7fc62b400640 1 -- 172.21.15.31:0/1877393641 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc62400ccc0 con 0x7fc634102f50 2024-03-21T10:42:56.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.089+0000 7fc62b400640 1 -- 172.21.15.31:0/1877393641 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc624018760 con 0x7fc634102f50 2024-03-21T10:42:56.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.089+0000 7fc633e00640 1 -- 172.21.15.31:0/1877393641 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc63419c6c0 con 0x7fc634102f50 2024-03-21T10:42:56.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.099+0000 7fc63c97b740 1 Processor -- start 2024-03-21T10:42:56.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.099+0000 7fc62b400640 1 -- 172.21.15.31:0/1877393641 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fc624010430 con 0x7fc634102f50 2024-03-21T10:42:56.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.100+0000 7fc63c97b740 1 -- start start 2024-03-21T10:42:56.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.100+0000 7fc63c97b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557111d198b0 0x557111d19c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.100+0000 7fc63c97b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557111d1a260 0x557111d22810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.100+0000 7fc63c97b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557111d22de0 0x557111d25230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.100+0000 7fc63c97b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557111c7d2c0 con 0x557111d1a260 2024-03-21T10:42:56.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.100+0000 7fc63c97b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557111c25c90 con 0x557111d22de0 2024-03-21T10:42:56.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.101+0000 7fc63c97b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x557111c79360 con 0x557111d198b0 2024-03-21T10:42:56.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.102+0000 7fc632000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557111d22de0 0x557111d25230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.102+0000 7fc632000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557111d22de0 0x557111d25230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58778/0 (socket says 172.21.15.31:58778) 2024-03-21T10:42:56.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.102+0000 7fc632000640 1 -- 172.21.15.31:0/2136075695 learned_addr learned my addr 172.21.15.31:0/2136075695 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:56.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.102+0000 7fc633400640 1 --2- 172.21.15.31:0/2136075695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557111d198b0 0x557111d19c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.102+0000 7fc632000640 1 -- 172.21.15.31:0/2136075695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557111d198b0 msgr2=0x557111d19c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:56.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.102+0000 7fc632000640 1 --2- 172.21.15.31:0/2136075695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557111d198b0 0x557111d19c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.102+0000 7fc632000640 1 -- 172.21.15.31:0/2136075695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557111d1a260 msgr2=0x557111d22810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:42:56.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.102+0000 7fc632000640 1 --2- 172.21.15.31:0/2136075695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557111d1a260 0x557111d22810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.102+0000 7fc632000640 1 -- 172.21.15.31:0/2136075695 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557111c4a080 con 0x557111d22de0 2024-03-21T10:42:56.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.103+0000 7fc632000640 1 --2- 172.21.15.31:0/2136075695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557111d22de0 0x557111d25230 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fc6240cd080 tx=0x7fc6240d67c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40ce557d5164f634 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:56.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.104+0000 7fc61fe00640 1 -- 172.21.15.31:0/2136075695 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6240e5070 con 0x557111d22de0 2024-03-21T10:42:56.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.104+0000 7fc62b400640 1 --2- 172.21.15.31:0/1877393641 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc6180a3150 0x7fc6180a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.104+0000 7fc62b400640 1 -- 172.21.15.31:0/1877393641 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fc6240cfd00 con 0x7fc634102f50 2024-03-21T10:42:56.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.105+0000 7fc61fe00640 1 -- 172.21.15.31:0/2136075695 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc6240d4ac0 con 0x557111d22de0 2024-03-21T10:42:56.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.105+0000 7fc61fe00640 1 -- 172.21.15.31:0/2136075695 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62409b8f0 con 0x557111d22de0 2024-03-21T10:42:56.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.107+0000 7fc633400640 1 --2- 172.21.15.31:0/1877393641 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc6180a3150 0x7fc6180a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.109+0000 7fc63c97b740 1 -- 172.21.15.31:0/2136075695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557111d22de0 msgr2=0x557111d25230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:56.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.109+0000 7fc63c97b740 1 --2- 172.21.15.31:0/2136075695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557111d22de0 0x557111d25230 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fc6240cd080 tx=0x7fc6240d67c0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.109+0000 7fc63c97b740 1 -- 172.21.15.31:0/2136075695 shutdown_connections 2024-03-21T10:42:56.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.109+0000 7fc63c97b740 1 --2- 172.21.15.31:0/2136075695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557111d22de0 0x557111d25230 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.111+0000 7fc63c97b740 1 --2- 172.21.15.31:0/2136075695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557111d1a260 0x557111d22810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.111+0000 7fc63c97b740 1 --2- 172.21.15.31:0/2136075695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557111d198b0 0x557111d19c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.111+0000 7fc63c97b740 1 -- 172.21.15.31:0/2136075695 >> 172.21.15.31:0/2136075695 conn(0x557111d13f80 msgr2=0x557111d14360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:56.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.111+0000 7fc63c97b740 1 -- 172.21.15.31:0/2136075695 shutdown_connections 2024-03-21T10:42:56.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.112+0000 7fc63c97b740 1 -- 172.21.15.31:0/2136075695 wait complete. 2024-03-21T10:42:56.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.113+0000 7fc63c97b740 1 -- start start 2024-03-21T10:42:56.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.114+0000 7fc63c97b740 1 Processor -- start 2024-03-21T10:42:56.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.115+0000 7fc633400640 1 --2- 172.21.15.31:0/1877393641 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc6180a3150 0x7fc6180a5610 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc62c0049d0 tx=0x7fc62c009500 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:56.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.116+0000 7fc63c97b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557111d198b0 0x557111d22680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.116+0000 7fc63c97b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557111d1a260 0x557111d1fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.116+0000 7fc63c97b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557111d22de0 0x557111d1ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.116+0000 7fc63c97b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557111d36d60 con 0x557111d1a260 2024-03-21T10:42:56.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.117+0000 7fc633400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557111d22de0 0x557111d1ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.117+0000 7fc633400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557111d22de0 0x557111d1ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:41564/0 (socket says 172.21.15.31:41564) 2024-03-21T10:42:56.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.117+0000 7fc633400640 1 -- 172.21.15.31:0/996899008 learned_addr learned my addr 172.21.15.31:0/996899008 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:56.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.117+0000 7fc63c97b740 1 -- 172.21.15.31:0/996899008 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557111c390d0 con 0x557111d198b0 2024-03-21T10:42:56.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.117+0000 7fc63c97b740 1 -- 172.21.15.31:0/996899008 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x557111d36ee0 con 0x557111d22de0 2024-03-21T10:42:56.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.117+0000 7fc632a00640 1 --2- 172.21.15.31:0/996899008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557111d198b0 0x557111d22680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.117+0000 7fc63c97b740 10 client.0 _ll_register_callbacks cb 0x557111c3de80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:56.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.117+0000 7fc633400640 1 -- 172.21.15.31:0/996899008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557111d198b0 msgr2=0x557111d22680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:56.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.117+0000 7fc633400640 1 --2- 172.21.15.31:0/996899008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557111d198b0 0x557111d22680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.117+0000 7fc633400640 1 -- 172.21.15.31:0/996899008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557111d1a260 msgr2=0x557111d1fa10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:42:56.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.117+0000 7fc633400640 1 --2- 172.21.15.31:0/996899008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557111d1a260 0x557111d1fa10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.117+0000 7fc633400640 1 -- 172.21.15.31:0/996899008 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5571119913f0 con 0x557111d22de0 2024-03-21T10:42:56.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.117+0000 7fc633400640 1 --2- 172.21.15.31:0/996899008 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557111d22de0 0x557111d1ff50 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc62c01d9d0 tx=0x7fc62c004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30a2fce9f6898ed4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:56.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.117+0000 7fc61cc00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:42:56.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.118+0000 7fc61cc00640 1 -- 172.21.15.31:0/996899008 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c02d060 con 0x557111d22de0 2024-03-21T10:42:56.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.119+0000 7fc61cc00640 1 -- 172.21.15.31:0/996899008 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fc62c017030 con 0x557111d22de0 2024-03-21T10:42:56.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.119+0000 7fc63c97b740 1 -- 172.21.15.31:0/996899008 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc6240d4ac0 con 0x557111d22de0 2024-03-21T10:42:56.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.119+0000 7fc61cc00640 1 -- 172.21.15.31:0/996899008 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc62c00f540 con 0x557111d22de0 2024-03-21T10:42:56.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.122+0000 7fc63c97b740 10 client.26683 fetch_fsmap learned FSMap version 26 2024-03-21T10:42:56.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.123+0000 7fc63c97b740 1 -- 172.21.15.31:0/996899008 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({fsmap=26}) v3 -- 0x557111a5b7d0 con 0x557111d22de0 2024-03-21T10:42:56.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.124+0000 7fc61cc00640 1 -- 172.21.15.31:0/996899008 <== mon.1 v2:172.21.15.70:3300/0 4 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fc62c00f6e0 con 0x557111d22de0 2024-03-21T10:42:56.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.124+0000 7fc63c97b740 10 client.26683 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:42:56.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.124+0000 7fc63c97b740 10 client.26683 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:42:56.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.124+0000 7fc63c97b740 20 client.26683 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:56.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.124+0000 7fc63c97b740 1 --2- 172.21.15.31:0/996899008 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x557111d43b00 0x557111d45ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.124+0000 7fc63c97b740 4 client.26683 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:42:56.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.124+0000 7fc63c97b740 1 -- 172.21.15.31:0/996899008 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557111c8a800 con 0x557111d43b00 2024-03-21T10:42:56.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.125+0000 7fc632a00640 1 --2- 172.21.15.31:0/996899008 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x557111d43b00 0x557111d45ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.125+0000 7fc632a00640 1 --2- 172.21.15.31:0/996899008 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x557111d43b00 0x557111d45ef0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b0e6b4d1313b840 server_cookie=3ce35a3470e1b26e in_seq=0 out_seq=0 2024-03-21T10:42:56.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.125+0000 7fc61cc00640 10 client.26683 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:42:56.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.128+0000 7fc61cc00640 1 -- 172.21.15.31:0/996899008 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557111c8a800 con 0x557111d43b00 2024-03-21T10:42:56.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.128+0000 7fc61cc00640 10 client.26683 handle_command_reply: tid=0 2024-03-21T10:42:56.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.130+0000 7fc63c97b740 1 client.26683 shutdown 2024-03-21T10:42:56.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.133+0000 7fc63c97b740 1 -- 172.21.15.31:0/996899008 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557111d22de0 msgr2=0x557111d1ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:56.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.133+0000 7fc63c97b740 1 --2- 172.21.15.31:0/996899008 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557111d22de0 0x557111d1ff50 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc62c01d9d0 tx=0x7fc62c004ab0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.133+0000 7fc63c97b740 1 -- 172.21.15.31:0/996899008 shutdown_connections 2024-03-21T10:42:56.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.133+0000 7fc63c97b740 1 --2- 172.21.15.31:0/996899008 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x557111d43b00 0x557111d45ef0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.134+0000 7fc63c97b740 1 --2- 172.21.15.31:0/996899008 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557111d22de0 0x557111d1ff50 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.134+0000 7fc63c97b740 1 --2- 172.21.15.31:0/996899008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557111d1a260 0x557111d1fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.134+0000 7fc63c97b740 1 --2- 172.21.15.31:0/996899008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557111d198b0 0x557111d22680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.134+0000 7fc63c97b740 1 -- 172.21.15.31:0/996899008 >> 172.21.15.31:0/996899008 conn(0x557111d13f80 msgr2=0x557111d15cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:56.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.134+0000 7fc61cc00640 0 client.26683 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:42:56.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.140+0000 7fc63c97b740 1 -- 172.21.15.31:0/996899008 shutdown_connections 2024-03-21T10:42:56.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.140+0000 7fc63c97b740 1 -- 172.21.15.31:0/996899008 wait complete. 2024-03-21T10:42:56.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.140+0000 7fc63c97b740 20 client.26683 trim_cache size 0 max 16384 2024-03-21T10:42:56.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.181+0000 7fc63c97b740 1 Processor -- start 2024-03-21T10:42:56.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.182+0000 7fc63c97b740 1 -- start start 2024-03-21T10:42:56.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.182+0000 7fc63c97b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557111d198b0 0x557111d12fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.182+0000 7fc63c97b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557111d1a260 0x557111d22140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.182+0000 7fc63c97b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557111d22de0 0x557111d22680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.182+0000 7fc63c97b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63406f520 con 0x557111d198b0 2024-03-21T10:42:56.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.182+0000 7fc63c97b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63406d020 con 0x557111d1a260 2024-03-21T10:42:56.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.182+0000 7fc63c97b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63406b0c0 con 0x557111d22de0 2024-03-21T10:42:56.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.184+0000 7fc633400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557111d22de0 0x557111d22680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.184+0000 7fc633400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557111d22de0 0x557111d22680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:41570/0 (socket says 172.21.15.31:41570) 2024-03-21T10:42:56.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.184+0000 7fc633400640 1 -- 172.21.15.31:0/1244887225 learned_addr learned my addr 172.21.15.31:0/1244887225 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:56.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.184+0000 7fc632000640 1 --2- 172.21.15.31:0/1244887225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557111d1a260 0x557111d22140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.185+0000 7fc632000640 1 -- 172.21.15.31:0/1244887225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557111d22de0 msgr2=0x557111d22680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:56.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.185+0000 7fc632000640 1 --2- 172.21.15.31:0/1244887225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557111d22de0 0x557111d22680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.185+0000 7fc632a00640 1 --2- 172.21.15.31:0/1244887225 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557111d198b0 0x557111d12fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.185+0000 7fc632000640 1 -- 172.21.15.31:0/1244887225 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557111d198b0 msgr2=0x557111d12fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:56.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.185+0000 7fc632000640 1 --2- 172.21.15.31:0/1244887225 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557111d198b0 0x557111d12fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.185+0000 7fc632000640 1 -- 172.21.15.31:0/1244887225 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557111d46260 con 0x557111d1a260 2024-03-21T10:42:56.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.185+0000 7fc633400640 1 --2- 172.21.15.31:0/1244887225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557111d22de0 0x557111d22680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:42:56.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.186+0000 7fc632a00640 1 --2- 172.21.15.31:0/1244887225 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557111d198b0 0x557111d12fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:42:56.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.187+0000 7fc632000640 1 --2- 172.21.15.31:0/1244887225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557111d1a260 0x557111d22140 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc6240d35f0 tx=0x7fc6240d6f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=892b8c57ef69f1af server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:56.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.192+0000 7fc61f400640 1 -- 172.21.15.31:0/1244887225 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62409b710 con 0x557111d1a260 2024-03-21T10:42:56.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.193+0000 7fc61f400640 1 -- 172.21.15.31:0/1244887225 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc62409f020 con 0x557111d1a260 2024-03-21T10:42:56.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.193+0000 7fc61f400640 1 -- 172.21.15.31:0/1244887225 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62409c4b0 con 0x557111d1a260 2024-03-21T10:42:56.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.194+0000 7fc63c97b740 1 -- 172.21.15.31:0/1244887225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557111d1a260 msgr2=0x557111d22140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:56.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.194+0000 7fc63c97b740 1 --2- 172.21.15.31:0/1244887225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557111d1a260 0x557111d22140 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc6240d35f0 tx=0x7fc6240d6f60 comp rx=0 tx=0).stop 2024-03-21T10:42:56.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.194+0000 7fc63c97b740 1 -- 172.21.15.31:0/1244887225 shutdown_connections 2024-03-21T10:42:56.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.195+0000 7fc63c97b740 1 --2- 172.21.15.31:0/1244887225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557111d22de0 0x557111d22680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.195+0000 7fc63c97b740 1 --2- 172.21.15.31:0/1244887225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557111d1a260 0x557111d22140 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.195+0000 7fc63c97b740 1 --2- 172.21.15.31:0/1244887225 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557111d198b0 0x557111d12fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.195+0000 7fc63c97b740 1 -- 172.21.15.31:0/1244887225 >> 172.21.15.31:0/1244887225 conn(0x557111d13f80 msgr2=0x557111d128d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:56.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.195+0000 7fc63c97b740 1 -- 172.21.15.31:0/1244887225 shutdown_connections 2024-03-21T10:42:56.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.195+0000 7fc63c97b740 1 -- 172.21.15.31:0/1244887225 wait complete. 2024-03-21T10:42:56.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.196+0000 7fc63c97b740 1 -- start start 2024-03-21T10:42:56.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.196+0000 7fc63c97b740 1 Processor -- start 2024-03-21T10:42:56.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.197+0000 7fc63c97b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557111d198b0 0x557111d97610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.197+0000 7fc63c97b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557111d1a260 0x557111d97b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.197+0000 7fc63c97b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557111d22de0 0x557111d37e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.197+0000 7fc63c97b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63406b6c0 con 0x557111d22de0 2024-03-21T10:42:56.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.197+0000 7fc63c97b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63406c9c0 con 0x557111d1a260 2024-03-21T10:42:56.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.197+0000 7fc63c97b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63406bd00 con 0x557111d198b0 2024-03-21T10:42:56.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.197+0000 7fc63c97b740 10 client.0 _ll_register_callbacks cb 0x7fc62c01d890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:56.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.198+0000 7fc632000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557111d1a260 0x557111d97b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.198+0000 7fc632000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557111d1a260 0x557111d97b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58798/0 (socket says 172.21.15.31:58798) 2024-03-21T10:42:56.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.198+0000 7fc632000640 1 -- 172.21.15.31:0/973601599 learned_addr learned my addr 172.21.15.31:0/973601599 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:56.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.198+0000 7fc632a00640 1 --2- 172.21.15.31:0/973601599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557111d198b0 0x557111d97610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.199+0000 7fc632000640 1 -- 172.21.15.31:0/973601599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557111d198b0 msgr2=0x557111d97610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:56.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.199+0000 7fc632000640 1 --2- 172.21.15.31:0/973601599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557111d198b0 0x557111d97610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.199+0000 7fc632000640 1 -- 172.21.15.31:0/973601599 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557111d22de0 msgr2=0x557111d37e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:56.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.199+0000 7fc632000640 1 --2- 172.21.15.31:0/973601599 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557111d22de0 0x557111d37e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.199+0000 7fc632000640 1 -- 172.21.15.31:0/973601599 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557111d38510 con 0x557111d1a260 2024-03-21T10:42:56.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.200+0000 7fc61cc00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:42:56.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.200+0000 7fc632000640 1 --2- 172.21.15.31:0/973601599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557111d1a260 0x557111d97b50 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc62409bab0 tx=0x7fc62400c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=491caf334ff30f55 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:56.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.200+0000 7fc61cc00640 1 -- 172.21.15.31:0/973601599 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62409b710 con 0x557111d1a260 2024-03-21T10:42:56.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.200+0000 7fc61cc00640 1 -- 172.21.15.31:0/973601599 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fc6240dc070 con 0x557111d1a260 2024-03-21T10:42:56.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.201+0000 7fc63c97b740 1 -- 172.21.15.31:0/973601599 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc62409f020 con 0x557111d1a260 2024-03-21T10:42:56.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.201+0000 7fc61cc00640 1 -- 172.21.15.31:0/973601599 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6240b7020 con 0x557111d1a260 2024-03-21T10:42:56.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.201+0000 7fc63c97b740 10 client.26747 fetch_fsmap learned FSMap version 26 2024-03-21T10:42:56.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.201+0000 7fc63c97b740 1 -- 172.21.15.31:0/973601599 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x557111d9fe50 con 0x557111d1a260 2024-03-21T10:42:56.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.201+0000 7fc61cc00640 1 -- 172.21.15.31:0/973601599 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc62409c4b0 con 0x557111d1a260 2024-03-21T10:42:56.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.202+0000 7fc61cc00640 1 -- 172.21.15.31:0/973601599 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fc624086040 con 0x557111d1a260 2024-03-21T10:42:56.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.202+0000 7fc63c97b740 10 client.26747 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:42:56.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.202+0000 7fc63c97b740 10 client.26747 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:42:56.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.202+0000 7fc63c97b740 20 client.26747 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:56.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.202+0000 7fc63c97b740 1 --2- 172.21.15.31:0/973601599 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x557111d9fe50 0x557111da2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.203+0000 7fc63c97b740 4 client.26747 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:42:56.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.203+0000 7fc63c97b740 1 -- 172.21.15.31:0/973601599 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7fc62409f020 con 0x557111d9fe50 2024-03-21T10:42:56.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.205+0000 7fc632a00640 1 --2- 172.21.15.31:0/973601599 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x557111d9fe50 0x557111da2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.206+0000 7fc632a00640 1 --2- 172.21.15.31:0/973601599 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x557111d9fe50 0x557111da2240 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c737f53738ab63bc server_cookie=9793cc40f240b4d0 in_seq=0 out_seq=0 2024-03-21T10:42:56.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.206+0000 7fc61cc00640 10 client.26747 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:42:56.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.207+0000 7fc61cc00640 1 -- 172.21.15.31:0/973601599 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7fc62409f020 con 0x557111d9fe50 2024-03-21T10:42:56.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.207+0000 7fc61cc00640 10 client.26747 handle_command_reply: tid=0 2024-03-21T10:42:56.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.207+0000 7fc63c97b740 1 client.26747 shutdown 2024-03-21T10:42:56.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.208+0000 7fc63c97b740 1 -- 172.21.15.31:0/973601599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557111d1a260 msgr2=0x557111d97b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:56.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.208+0000 7fc63c97b740 1 --2- 172.21.15.31:0/973601599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557111d1a260 0x557111d97b50 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc62409bab0 tx=0x7fc62400c7d0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.209+0000 7fc63c97b740 1 -- 172.21.15.31:0/973601599 shutdown_connections 2024-03-21T10:42:56.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.209+0000 7fc63c97b740 1 --2- 172.21.15.31:0/973601599 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x557111d9fe50 0x557111da2240 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.209+0000 7fc63c97b740 1 --2- 172.21.15.31:0/973601599 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557111d22de0 0x557111d37e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.209+0000 7fc63c97b740 1 --2- 172.21.15.31:0/973601599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557111d1a260 0x557111d97b50 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.209+0000 7fc63c97b740 1 --2- 172.21.15.31:0/973601599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557111d198b0 0x557111d97610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.209+0000 7fc61cc00640 0 client.26747 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:42:56.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.209+0000 7fc63c97b740 1 -- 172.21.15.31:0/973601599 >> 172.21.15.31:0/973601599 conn(0x557111d13f80 msgr2=0x557111d13160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:56.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.210+0000 7fc63c97b740 1 -- 172.21.15.31:0/973601599 shutdown_connections 2024-03-21T10:42:56.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.210+0000 7fc63c97b740 1 -- 172.21.15.31:0/973601599 wait complete. 2024-03-21T10:42:56.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.210+0000 7fc63c97b740 20 client.26747 trim_cache size 0 max 16384 2024-03-21T10:42:56.209 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:42:56.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.212+0000 7fc633e00640 1 -- 172.21.15.31:0/1877393641 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc6180a3150 msgr2=0x7fc6180a5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:56.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.212+0000 7fc633e00640 1 --2- 172.21.15.31:0/1877393641 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc6180a3150 0x7fc6180a5610 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc62c0049d0 tx=0x7fc62c009500 comp rx=0 tx=0).stop 2024-03-21T10:42:56.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.212+0000 7fc633e00640 1 -- 172.21.15.31:0/1877393641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc634102f50 msgr2=0x7fc6341007e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:56.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.212+0000 7fc633e00640 1 --2- 172.21.15.31:0/1877393641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc634102f50 0x7fc6341007e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fc62400e790 tx=0x7fc62400ec50 comp rx=0 tx=0).stop 2024-03-21T10:42:56.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.212+0000 7fc633e00640 1 -- 172.21.15.31:0/1877393641 shutdown_connections 2024-03-21T10:42:56.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.212+0000 7fc633e00640 1 --2- 172.21.15.31:0/1877393641 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc6180a3150 0x7fc6180a5610 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.212+0000 7fc633e00640 1 --2- 172.21.15.31:0/1877393641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc63410ba50 0x7fc634100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.212+0000 7fc633e00640 1 --2- 172.21.15.31:0/1877393641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc634102f50 0x7fc6341007e0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.212+0000 7fc633e00640 1 --2- 172.21.15.31:0/1877393641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc634102630 0x7fc6341002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.212+0000 7fc633e00640 1 -- 172.21.15.31:0/1877393641 >> 172.21.15.31:0/1877393641 conn(0x7fc634077130 msgr2=0x7fc6340fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:56.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.212+0000 7fc633e00640 1 -- 172.21.15.31:0/1877393641 shutdown_connections 2024-03-21T10:42:56.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.212+0000 7fc633e00640 1 -- 172.21.15.31:0/1877393641 wait complete. 2024-03-21T10:42:56.232 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:42:56.232 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-21T10:42:56.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:56 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2688156212' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:42:56.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.499+0000 7f8ba1800640 1 -- 172.21.15.31:0/823533570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b9c103840 msgr2=0x7f8b9c103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:56.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.499+0000 7f8ba1800640 1 --2- 172.21.15.31:0/823533570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b9c103840 0x7f8b9c103cc0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c009900 tx=0x7f8b8c02bc30 comp rx=0 tx=0).stop 2024-03-21T10:42:56.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.500+0000 7f8ba1800640 1 -- 172.21.15.31:0/823533570 shutdown_connections 2024-03-21T10:42:56.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.500+0000 7f8ba1800640 1 --2- 172.21.15.31:0/823533570 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b9c104200 0x7f8b9c110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.500+0000 7f8ba1800640 1 --2- 172.21.15.31:0/823533570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b9c103840 0x7f8b9c103cc0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.500+0000 7f8ba1800640 1 --2- 172.21.15.31:0/823533570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b9c109850 0x7f8b9c109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.500+0000 7f8ba1800640 1 -- 172.21.15.31:0/823533570 >> 172.21.15.31:0/823533570 conn(0x7f8b9c0ff2f0 msgr2=0x7f8b9c101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:56.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.500+0000 7f8ba1800640 1 -- 172.21.15.31:0/823533570 shutdown_connections 2024-03-21T10:42:56.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.500+0000 7f8ba1800640 1 -- 172.21.15.31:0/823533570 wait complete. 2024-03-21T10:42:56.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.502+0000 7f8ba1800640 1 Processor -- start 2024-03-21T10:42:56.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.502+0000 7f8ba1800640 1 -- start start 2024-03-21T10:42:56.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.502+0000 7f8ba1800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b9c103840 0x7f8b9c19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.502+0000 7f8ba1800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b9c104200 0x7f8b9c19a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.502+0000 7f8ba1800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b9c109850 0x7f8b9c19ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.502+0000 7f8ba1800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c113c20 con 0x7f8b9c109850 2024-03-21T10:42:56.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.502+0000 7f8ba1800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c113aa0 con 0x7f8b9c103840 2024-03-21T10:42:56.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.502+0000 7f8ba1800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c113da0 con 0x7f8b9c104200 2024-03-21T10:42:56.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.503+0000 7f8b9aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b9c103840 0x7f8b9c19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.503+0000 7f8b9aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b9c103840 0x7f8b9c19a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58812/0 (socket says 172.21.15.31:58812) 2024-03-21T10:42:56.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.503+0000 7f8b9aa00640 1 -- 172.21.15.31:0/3320739779 learned_addr learned my addr 172.21.15.31:0/3320739779 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:56.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.503+0000 7f8b9a000640 1 --2- 172.21.15.31:0/3320739779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b9c104200 0x7f8b9c19a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.503+0000 7f8b9aa00640 1 -- 172.21.15.31:0/3320739779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b9c104200 msgr2=0x7f8b9c19a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:56.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.503+0000 7f8b9aa00640 1 --2- 172.21.15.31:0/3320739779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b9c104200 0x7f8b9c19a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.503+0000 7f8b9aa00640 1 -- 172.21.15.31:0/3320739779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b9c109850 msgr2=0x7f8b9c19ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:56.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.503+0000 7f8b9aa00640 1 --2- 172.21.15.31:0/3320739779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b9c109850 0x7f8b9c19ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.503+0000 7f8b9aa00640 1 -- 172.21.15.31:0/3320739779 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b9c19f140 con 0x7f8b9c103840 2024-03-21T10:42:56.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.504+0000 7f8b9aa00640 1 --2- 172.21.15.31:0/3320739779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b9c103840 0x7f8b9c19a1a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f8b8800ea60 tx=0x7f8b8800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:56.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.504+0000 7f8b93400640 1 -- 172.21.15.31:0/3320739779 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8800cde0 con 0x7f8b9c103840 2024-03-21T10:42:56.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.504+0000 7f8b93400640 1 -- 172.21.15.31:0/3320739779 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8b880035f0 con 0x7f8b9c103840 2024-03-21T10:42:56.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.504+0000 7f8ba1800640 1 -- 172.21.15.31:0/3320739779 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b9c19f430 con 0x7f8b9c103840 2024-03-21T10:42:56.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.504+0000 7f8b93400640 1 -- 172.21.15.31:0/3320739779 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b88010690 con 0x7f8b9c103840 2024-03-21T10:42:56.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.505+0000 7f8ba1800640 1 -- 172.21.15.31:0/3320739779 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b9c1a6cc0 con 0x7f8b9c103840 2024-03-21T10:42:56.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.509+0000 7f8b93400640 1 -- 172.21.15.31:0/3320739779 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f8b880037a0 con 0x7f8b9c103840 2024-03-21T10:42:56.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.510+0000 7f8ba458b740 1 Processor -- start 2024-03-21T10:42:56.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.510+0000 7f8b93400640 1 --2- 172.21.15.31:0/3320739779 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f8b800a3190 0x7f8b800a5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.510+0000 7f8b93400640 1 -- 172.21.15.31:0/3320739779 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f8b880cfea0 con 0x7f8b9c103840 2024-03-21T10:42:56.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.510+0000 7f8ba458b740 1 -- start start 2024-03-21T10:42:56.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.511+0000 7f8b9b400640 1 --2- 172.21.15.31:0/3320739779 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f8b800a3190 0x7f8b800a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.511+0000 7f8ba458b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564055883670 0x564055883a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.511+0000 7f8ba458b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564055884020 0x564055894e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.511+0000 7f8ba458b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564055895410 0x564055897820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.511+0000 7f8ba458b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640557e72c0 con 0x564055884020 2024-03-21T10:42:56.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.511+0000 7f8ba458b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56405578fc90 con 0x564055895410 2024-03-21T10:42:56.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.511+0000 7f8ba458b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640557e3360 con 0x564055883670 2024-03-21T10:42:56.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.511+0000 7f8b9aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564055883670 0x564055883a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.511+0000 7f8b9aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564055883670 0x564055883a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:41600/0 (socket says 172.21.15.31:41600) 2024-03-21T10:42:56.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.511+0000 7f8b9aa00640 1 -- 172.21.15.31:0/2339396849 learned_addr learned my addr 172.21.15.31:0/2339396849 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:56.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.511+0000 7f8b9a000640 1 --2- 172.21.15.31:0/2339396849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564055884020 0x564055894e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.511+0000 7f8b9aa00640 1 -- 172.21.15.31:0/2339396849 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564055895410 msgr2=0x564055897820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:42:56.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.512+0000 7f8b9aa00640 1 --2- 172.21.15.31:0/2339396849 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564055895410 0x564055897820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.512+0000 7f8b9aa00640 1 -- 172.21.15.31:0/2339396849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564055884020 msgr2=0x564055894e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:56.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.512+0000 7f8b9aa00640 1 --2- 172.21.15.31:0/2339396849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564055884020 0x564055894e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.512+0000 7f8b9aa00640 1 -- 172.21.15.31:0/2339396849 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5640557b4080 con 0x564055883670 2024-03-21T10:42:56.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.512+0000 7f8b9a000640 1 --2- 172.21.15.31:0/2339396849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564055884020 0x564055894e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:42:56.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.512+0000 7f8b9aa00640 1 --2- 172.21.15.31:0/2339396849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564055883670 0x564055883a50 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8b880a43d0 tx=0x7f8b8809bdf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72c2ee7bec6421ce server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:56.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.512+0000 7f8b87e00640 1 -- 172.21.15.31:0/2339396849 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b880bc0f0 con 0x564055883670 2024-03-21T10:42:56.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.512+0000 7f8b87e00640 1 -- 172.21.15.31:0/2339396849 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8b880a0040 con 0x564055883670 2024-03-21T10:42:56.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.512+0000 7f8b87e00640 1 -- 172.21.15.31:0/2339396849 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b880cd4d0 con 0x564055883670 2024-03-21T10:42:56.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.513+0000 7f8ba458b740 1 -- 172.21.15.31:0/2339396849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564055883670 msgr2=0x564055883a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:56.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.513+0000 7f8b9b400640 1 --2- 172.21.15.31:0/3320739779 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f8b800a3190 0x7f8b800a5650 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8b94009270 tx=0x7f8b940091e0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:56.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.513+0000 7f8ba458b740 1 --2- 172.21.15.31:0/2339396849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564055883670 0x564055883a50 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8b880a43d0 tx=0x7f8b8809bdf0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.514+0000 7f8ba458b740 1 -- 172.21.15.31:0/2339396849 shutdown_connections 2024-03-21T10:42:56.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.514+0000 7f8ba458b740 1 --2- 172.21.15.31:0/2339396849 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564055895410 0x564055897820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.514+0000 7f8ba458b740 1 --2- 172.21.15.31:0/2339396849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564055884020 0x564055894e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.514+0000 7f8ba458b740 1 --2- 172.21.15.31:0/2339396849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564055883670 0x564055883a50 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.514+0000 7f8ba458b740 1 -- 172.21.15.31:0/2339396849 >> 172.21.15.31:0/2339396849 conn(0x56405587dda0 msgr2=0x56405587e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:56.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.515+0000 7f8ba458b740 1 -- 172.21.15.31:0/2339396849 shutdown_connections 2024-03-21T10:42:56.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.516+0000 7f8ba458b740 1 -- 172.21.15.31:0/2339396849 wait complete. 2024-03-21T10:42:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.517+0000 7f8ba458b740 1 -- start start 2024-03-21T10:42:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.517+0000 7f8ba458b740 1 Processor -- start 2024-03-21T10:42:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.518+0000 7f8ba458b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564055883670 0x564055894db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.518+0000 7f8ba458b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564055884020 0x564055892140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.518+0000 7f8ba458b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564055895410 0x564055892880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.518+0000 7f8ba458b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564055898340 con 0x564055884020 2024-03-21T10:42:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.518+0000 7f8ba458b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640557a30d0 con 0x564055883670 2024-03-21T10:42:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.518+0000 7f8ba458b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640558984c0 con 0x564055895410 2024-03-21T10:42:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.518+0000 7f8ba458b740 10 client.0 _ll_register_callbacks cb 0x5640557a7e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.518+0000 7f8b9aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564055883670 0x564055894db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.519+0000 7f8b9aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564055883670 0x564055894db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58832/0 (socket says 172.21.15.31:58832) 2024-03-21T10:42:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.519+0000 7f8b9aa00640 1 -- 172.21.15.31:0/3358394983 learned_addr learned my addr 172.21.15.31:0/3358394983 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.519+0000 7f8b9b400640 1 --2- 172.21.15.31:0/3358394983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564055895410 0x564055892880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.519+0000 7f8b9aa00640 1 -- 172.21.15.31:0/3358394983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564055895410 msgr2=0x564055892880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:56.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.519+0000 7f8b9aa00640 1 --2- 172.21.15.31:0/3358394983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564055895410 0x564055892880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.519+0000 7f8b9aa00640 1 -- 172.21.15.31:0/3358394983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564055884020 msgr2=0x564055892140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:42:56.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.519+0000 7f8b9aa00640 1 --2- 172.21.15.31:0/3358394983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564055884020 0x564055892140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.519+0000 7f8b9aa00640 1 -- 172.21.15.31:0/3358394983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5640554fb3f0 con 0x564055883670 2024-03-21T10:42:56.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.519+0000 7f8b9aa00640 1 --2- 172.21.15.31:0/3358394983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564055883670 0x564055894db0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f8b8800efd0 tx=0x7f8b880cd240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad79d544fd5cb414 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:56.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.519+0000 7f8b84c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:42:56.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.520+0000 7f8b84c00640 1 -- 172.21.15.31:0/3358394983 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b88003ad0 con 0x564055883670 2024-03-21T10:42:56.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.520+0000 7f8b84c00640 1 -- 172.21.15.31:0/3358394983 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f8b880cd4d0 con 0x564055883670 2024-03-21T10:42:56.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.520+0000 7f8ba458b740 1 -- 172.21.15.31:0/3358394983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8b880a0040 con 0x564055883670 2024-03-21T10:42:56.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.520+0000 7f8b84c00640 1 -- 172.21.15.31:0/3358394983 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b8809b340 con 0x564055883670 2024-03-21T10:42:56.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.520+0000 7f8ba458b740 10 client.26756 fetch_fsmap learned FSMap version 26 2024-03-21T10:42:56.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.520+0000 7f8ba458b740 1 -- 172.21.15.31:0/3358394983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5640555c57d0 con 0x564055883670 2024-03-21T10:42:56.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.521+0000 7f8b84c00640 1 -- 172.21.15.31:0/3358394983 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8b880a0040 con 0x564055883670 2024-03-21T10:42:56.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.521+0000 7f8b84c00640 1 -- 172.21.15.31:0/3358394983 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f8b88097e40 con 0x564055883670 2024-03-21T10:42:56.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.522+0000 7f8ba458b740 10 client.26756 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:42:56.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.522+0000 7f8ba458b740 10 client.26756 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:42:56.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.522+0000 7f8ba458b740 20 client.26756 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:56.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.522+0000 7f8ba458b740 1 --2- 172.21.15.31:0/3358394983 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564055899910 0x56405589bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.522+0000 7f8ba458b740 4 client.26756 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:42:56.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.522+0000 7f8ba458b740 1 -- 172.21.15.31:0/3358394983 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5640557f4800 con 0x564055899910 2024-03-21T10:42:56.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.522+0000 7f8b9a000640 1 --2- 172.21.15.31:0/3358394983 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564055899910 0x56405589bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.523+0000 7f8b9a000640 1 --2- 172.21.15.31:0/3358394983 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564055899910 0x56405589bd60 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70f854c3bb2a4722 server_cookie=ada52833b552b8b1 in_seq=0 out_seq=0 2024-03-21T10:42:56.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.523+0000 7f8b84c00640 10 client.26756 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:56.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.524+0000 7f8b84c00640 1 -- 172.21.15.31:0/3358394983 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5640557f4800 con 0x564055899910 2024-03-21T10:42:56.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.524+0000 7f8b84c00640 10 client.26756 handle_command_reply: tid=0 2024-03-21T10:42:56.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.524+0000 7f8ba458b740 1 client.26756 shutdown 2024-03-21T10:42:56.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.524+0000 7f8ba458b740 1 -- 172.21.15.31:0/3358394983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564055883670 msgr2=0x564055894db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:56.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.524+0000 7f8ba458b740 1 --2- 172.21.15.31:0/3358394983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564055883670 0x564055894db0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f8b8800efd0 tx=0x7f8b880cd240 comp rx=0 tx=0).stop 2024-03-21T10:42:56.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.524+0000 7f8ba458b740 1 -- 172.21.15.31:0/3358394983 shutdown_connections 2024-03-21T10:42:56.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.524+0000 7f8ba458b740 1 --2- 172.21.15.31:0/3358394983 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564055899910 0x56405589bd60 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.524+0000 7f8ba458b740 1 --2- 172.21.15.31:0/3358394983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564055895410 0x564055892880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.524+0000 7f8ba458b740 1 --2- 172.21.15.31:0/3358394983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564055884020 0x564055892140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.524+0000 7f8b84c00640 0 client.26756 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:56.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.524+0000 7f8ba458b740 1 --2- 172.21.15.31:0/3358394983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564055883670 0x564055894db0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.524+0000 7f8ba458b740 1 -- 172.21.15.31:0/3358394983 >> 172.21.15.31:0/3358394983 conn(0x56405587dda0 msgr2=0x56405587faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:56.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.524+0000 7f8ba458b740 1 -- 172.21.15.31:0/3358394983 shutdown_connections 2024-03-21T10:42:56.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.525+0000 7f8ba458b740 1 -- 172.21.15.31:0/3358394983 wait complete. 2024-03-21T10:42:56.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.525+0000 7f8ba458b740 20 client.26756 trim_cache size 0 max 16384 2024-03-21T10:42:56.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.546+0000 7f8ba458b740 1 Processor -- start 2024-03-21T10:42:56.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.546+0000 7f8ba458b740 1 -- start start 2024-03-21T10:42:56.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.546+0000 7f8ba458b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564055883670 0x5640558a0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.546+0000 7f8ba458b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564055884020 0x564055894870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.546+0000 7f8ba458b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564055895410 0x564055894db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.546+0000 7f8ba458b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c1aebf0 con 0x564055895410 2024-03-21T10:42:56.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.546+0000 7f8ba458b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c1ac6f0 con 0x564055884020 2024-03-21T10:42:56.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.546+0000 7f8ba458b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c1aa790 con 0x564055883670 2024-03-21T10:42:56.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.546+0000 7f8b9aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564055883670 0x5640558a0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.547+0000 7f8b9aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564055883670 0x5640558a0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:41624/0 (socket says 172.21.15.31:41624) 2024-03-21T10:42:56.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.547+0000 7f8b9aa00640 1 -- 172.21.15.31:0/1929859560 learned_addr learned my addr 172.21.15.31:0/1929859560 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:56.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.547+0000 7f8b9a000640 1 --2- 172.21.15.31:0/1929859560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564055884020 0x564055894870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.547+0000 7f8b9aa00640 1 -- 172.21.15.31:0/1929859560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564055884020 msgr2=0x564055894870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:56.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.547+0000 7f8b9aa00640 1 --2- 172.21.15.31:0/1929859560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564055884020 0x564055894870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.547+0000 7f8b9aa00640 1 -- 172.21.15.31:0/1929859560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564055895410 msgr2=0x564055894db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:56.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.547+0000 7f8b9aa00640 1 --2- 172.21.15.31:0/1929859560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564055895410 0x564055894db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.547+0000 7f8b9aa00640 1 -- 172.21.15.31:0/1929859560 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56405587efd0 con 0x564055883670 2024-03-21T10:42:56.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.547+0000 7f8b9a000640 1 --2- 172.21.15.31:0/1929859560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564055884020 0x564055894870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:42:56.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.548+0000 7f8b9aa00640 1 --2- 172.21.15.31:0/1929859560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564055883670 0x5640558a0d30 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8b8809be80 tx=0x7f8b880a49c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2880bbe7955e7443 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:56.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.548+0000 7f8b87400640 1 -- 172.21.15.31:0/1929859560 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8809adb0 con 0x564055883670 2024-03-21T10:42:56.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.548+0000 7f8b87400640 1 -- 172.21.15.31:0/1929859560 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8b880a0040 con 0x564055883670 2024-03-21T10:42:56.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.549+0000 7f8b87400640 1 -- 172.21.15.31:0/1929859560 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b880cd4d0 con 0x564055883670 2024-03-21T10:42:56.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.550+0000 7f8ba458b740 1 -- 172.21.15.31:0/1929859560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564055883670 msgr2=0x5640558a0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:56.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.550+0000 7f8ba458b740 1 --2- 172.21.15.31:0/1929859560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564055883670 0x5640558a0d30 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8b8809be80 tx=0x7f8b880a49c0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.550+0000 7f8ba458b740 1 -- 172.21.15.31:0/1929859560 shutdown_connections 2024-03-21T10:42:56.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.550+0000 7f8ba458b740 1 --2- 172.21.15.31:0/1929859560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564055895410 0x564055894db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.550+0000 7f8ba458b740 1 --2- 172.21.15.31:0/1929859560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564055884020 0x564055894870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.550+0000 7f8ba458b740 1 --2- 172.21.15.31:0/1929859560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564055883670 0x5640558a0d30 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.550+0000 7f8ba458b740 1 -- 172.21.15.31:0/1929859560 >> 172.21.15.31:0/1929859560 conn(0x56405587dda0 msgr2=0x56405587fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:56.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.551+0000 7f8ba458b740 1 -- 172.21.15.31:0/1929859560 shutdown_connections 2024-03-21T10:42:56.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.551+0000 7f8ba458b740 1 -- 172.21.15.31:0/1929859560 wait complete. 2024-03-21T10:42:56.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.551+0000 7f8ba458b740 1 -- start start 2024-03-21T10:42:56.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.551+0000 7f8ba458b740 1 Processor -- start 2024-03-21T10:42:56.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.552+0000 7f8ba458b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564055883670 0x564055886a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.552+0000 7f8ba458b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564055884020 0x564055886f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.552+0000 7f8ba458b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564055895410 0x564055887490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.552+0000 7f8ba458b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c1aad90 con 0x564055884020 2024-03-21T10:42:56.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.552+0000 7f8ba458b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c1ac090 con 0x564055895410 2024-03-21T10:42:56.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.552+0000 7f8ba458b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c1ab3d0 con 0x564055883670 2024-03-21T10:42:56.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.552+0000 7f8ba458b740 10 client.0 _ll_register_callbacks cb 0x7f8b8809ba30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:56.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.552+0000 7f8b9aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564055883670 0x564055886a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.552+0000 7f8b9aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564055883670 0x564055886a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:41634/0 (socket says 172.21.15.31:41634) 2024-03-21T10:42:56.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.552+0000 7f8b9aa00640 1 -- 172.21.15.31:0/672820625 learned_addr learned my addr 172.21.15.31:0/672820625 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:56.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.552+0000 7f8b9b400640 1 --2- 172.21.15.31:0/672820625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564055895410 0x564055887490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.553+0000 7f8b9aa00640 1 -- 172.21.15.31:0/672820625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564055895410 msgr2=0x564055887490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:56.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.553+0000 7f8b9aa00640 1 --2- 172.21.15.31:0/672820625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564055895410 0x564055887490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.553+0000 7f8b9a000640 1 --2- 172.21.15.31:0/672820625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564055884020 0x564055886f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.553+0000 7f8b9aa00640 1 -- 172.21.15.31:0/672820625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564055884020 msgr2=0x564055886f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:56.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.553+0000 7f8b9aa00640 1 --2- 172.21.15.31:0/672820625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564055884020 0x564055886f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.553+0000 7f8b9aa00640 1 -- 172.21.15.31:0/672820625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564055919cb0 con 0x564055883670 2024-03-21T10:42:56.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.553+0000 7f8b9a000640 1 --2- 172.21.15.31:0/672820625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564055884020 0x564055886f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:42:56.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.553+0000 7f8b9aa00640 1 --2- 172.21.15.31:0/672820625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564055883670 0x564055886a10 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8b8809bbf0 tx=0x7f8b88003ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2e81efb5ef72070 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:56.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.553+0000 7f8b84c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:42:56.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.553+0000 7f8b84c00640 1 -- 172.21.15.31:0/672820625 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8809adb0 con 0x564055883670 2024-03-21T10:42:56.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.553+0000 7f8b84c00640 1 -- 172.21.15.31:0/672820625 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f8b88004000 con 0x564055883670 2024-03-21T10:42:56.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.554+0000 7f8ba458b740 1 -- 172.21.15.31:0/672820625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8b880a0040 con 0x564055883670 2024-03-21T10:42:56.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.554+0000 7f8b84c00640 1 -- 172.21.15.31:0/672820625 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b880cd6d0 con 0x564055883670 2024-03-21T10:42:56.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.554+0000 7f8ba458b740 10 client.26707 fetch_fsmap learned FSMap version 26 2024-03-21T10:42:56.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.554+0000 7f8ba458b740 1 -- 172.21.15.31:0/672820625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x564055919cb0 con 0x564055883670 2024-03-21T10:42:56.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.554+0000 7f8b84c00640 1 -- 172.21.15.31:0/672820625 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8b880a0040 con 0x564055883670 2024-03-21T10:42:56.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.555+0000 7f8b84c00640 1 -- 172.21.15.31:0/672820625 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f8b880a11d0 con 0x564055883670 2024-03-21T10:42:56.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.555+0000 7f8ba458b740 10 client.26707 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:42:56.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.555+0000 7f8ba458b740 10 client.26707 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:42:56.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.555+0000 7f8ba458b740 20 client.26707 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:56.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.555+0000 7f8ba458b740 1 --2- 172.21.15.31:0/672820625 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56405591b1b0 0x56405591d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:56.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.555+0000 7f8ba458b740 4 client.26707 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:42:56.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.555+0000 7f8ba458b740 1 -- 172.21.15.31:0/672820625 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5640557eeb50 con 0x56405591b1b0 2024-03-21T10:42:56.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.555+0000 7f8b9a000640 1 --2- 172.21.15.31:0/672820625 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56405591b1b0 0x56405591d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:56.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.556+0000 7f8b9a000640 1 --2- 172.21.15.31:0/672820625 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56405591b1b0 0x56405591d5a0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca5b3efb2f9e059c server_cookie=7ae5ebce2e229b80 in_seq=0 out_seq=0 2024-03-21T10:42:56.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.556+0000 7f8b84c00640 10 client.26707 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:56.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.556+0000 7f8b84c00640 1 -- 172.21.15.31:0/672820625 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5640557eeb50 con 0x56405591b1b0 2024-03-21T10:42:56.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.556+0000 7f8b84c00640 10 client.26707 handle_command_reply: tid=0 2024-03-21T10:42:56.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.556+0000 7f8ba458b740 1 client.26707 shutdown 2024-03-21T10:42:56.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.557+0000 7f8ba458b740 1 -- 172.21.15.31:0/672820625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564055883670 msgr2=0x564055886a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:56.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.557+0000 7f8ba458b740 1 --2- 172.21.15.31:0/672820625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564055883670 0x564055886a10 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8b8809bbf0 tx=0x7f8b88003ed0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.557+0000 7f8ba458b740 1 -- 172.21.15.31:0/672820625 shutdown_connections 2024-03-21T10:42:56.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.557+0000 7f8ba458b740 1 --2- 172.21.15.31:0/672820625 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56405591b1b0 0x56405591d5a0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.557+0000 7f8ba458b740 1 --2- 172.21.15.31:0/672820625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564055895410 0x564055887490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.557+0000 7f8ba458b740 1 --2- 172.21.15.31:0/672820625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564055884020 0x564055886f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.557+0000 7f8ba458b740 1 --2- 172.21.15.31:0/672820625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564055883670 0x564055886a10 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.557+0000 7f8ba458b740 1 -- 172.21.15.31:0/672820625 >> 172.21.15.31:0/672820625 conn(0x56405587dda0 msgr2=0x56405587c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:56.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.557+0000 7f8b84c00640 0 client.26707 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:56.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:56 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2688156212' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:42:56.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.559+0000 7f8ba458b740 1 -- 172.21.15.31:0/672820625 shutdown_connections 2024-03-21T10:42:56.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.559+0000 7f8ba458b740 1 -- 172.21.15.31:0/672820625 wait complete. 2024-03-21T10:42:56.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.559+0000 7f8ba458b740 20 client.26707 trim_cache size 0 max 16384 2024-03-21T10:42:56.557 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:42:56.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.562+0000 7f8ba1800640 1 -- 172.21.15.31:0/3320739779 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f8b800a3190 msgr2=0x7f8b800a5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:56.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.562+0000 7f8ba1800640 1 --2- 172.21.15.31:0/3320739779 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f8b800a3190 0x7f8b800a5650 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8b94009270 tx=0x7f8b940091e0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.562+0000 7f8ba1800640 1 -- 172.21.15.31:0/3320739779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b9c103840 msgr2=0x7f8b9c19a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:56.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.562+0000 7f8ba1800640 1 --2- 172.21.15.31:0/3320739779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b9c103840 0x7f8b9c19a1a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f8b8800ea60 tx=0x7f8b8800ef20 comp rx=0 tx=0).stop 2024-03-21T10:42:56.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.562+0000 7f8ba1800640 1 -- 172.21.15.31:0/3320739779 shutdown_connections 2024-03-21T10:42:56.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.562+0000 7f8ba1800640 1 --2- 172.21.15.31:0/3320739779 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f8b800a3190 0x7f8b800a5650 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.562+0000 7f8ba1800640 1 --2- 172.21.15.31:0/3320739779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b9c109850 0x7f8b9c19ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.562+0000 7f8ba1800640 1 --2- 172.21.15.31:0/3320739779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b9c104200 0x7f8b9c19a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.562+0000 7f8ba1800640 1 --2- 172.21.15.31:0/3320739779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b9c103840 0x7f8b9c19a1a0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:56.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.562+0000 7f8ba1800640 1 -- 172.21.15.31:0/3320739779 >> 172.21.15.31:0/3320739779 conn(0x7f8b9c0ff2f0 msgr2=0x7f8b9c0ffad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:56.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.562+0000 7f8ba1800640 1 -- 172.21.15.31:0/3320739779 shutdown_connections 2024-03-21T10:42:56.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:56.562+0000 7f8ba1800640 1 -- 172.21.15.31:0/3320739779 wait complete. 2024-03-21T10:42:56.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:56 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2688156212' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:42:57.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:57 smithi040 ceph-mon[36154]: pgmap v134: 129 pgs: 129 active+clean; 24 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 266 MiB/s wr, 1.15k op/s 2024-03-21T10:42:57.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:57 smithi031 ceph-mon[31673]: pgmap v134: 129 pgs: 129 active+clean; 24 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 266 MiB/s wr, 1.15k op/s 2024-03-21T10:42:57.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:57 smithi070 ceph-mon[36330]: pgmap v134: 129 pgs: 129 active+clean; 24 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 266 MiB/s wr, 1.15k op/s 2024-03-21T10:42:57.588 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-21T10:42:57.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:42:57.588 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-21T10:42:57.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.930+0000 7fd0b1c00640 1 -- 172.21.15.31:0/1114553611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0ac10bc90 msgr2=0x7fd0ac10e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:57.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.930+0000 7fd0b1c00640 1 --2- 172.21.15.31:0/1114553611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0ac10bc90 0x7fd0ac10e0e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4009ca0 tx=0x7fd0a402c1a0 comp rx=0 tx=0).stop 2024-03-21T10:42:57.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.931+0000 7fd0b1c00640 1 -- 172.21.15.31:0/1114553611 shutdown_connections 2024-03-21T10:42:57.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.931+0000 7fd0b1c00640 1 --2- 172.21.15.31:0/1114553611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0ac10bc90 0x7fd0ac10e0e0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:57.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.931+0000 7fd0b1c00640 1 --2- 172.21.15.31:0/1114553611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0ac1031e0 0x7fd0ac10b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:57.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.931+0000 7fd0b1c00640 1 --2- 172.21.15.31:0/1114553611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0ac1028c0 0x7fd0ac102ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:57.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.931+0000 7fd0b1c00640 1 -- 172.21.15.31:0/1114553611 >> 172.21.15.31:0/1114553611 conn(0x7fd0ac0776f0 msgr2=0x7fd0ac077b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:57.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.931+0000 7fd0b1c00640 1 -- 172.21.15.31:0/1114553611 shutdown_connections 2024-03-21T10:42:57.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.931+0000 7fd0b1c00640 1 -- 172.21.15.31:0/1114553611 wait complete. 2024-03-21T10:42:57.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.934+0000 7fd0b1c00640 1 Processor -- start 2024-03-21T10:42:57.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.935+0000 7fd0b1c00640 1 -- start start 2024-03-21T10:42:57.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.936+0000 7fd0b1c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0ac1028c0 0x7fd0ac197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:57.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.936+0000 7fd0b1c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0ac1031e0 0x7fd0ac1984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:57.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.936+0000 7fd0b1c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0ac10bc90 0x7fd0ac19c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:57.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.936+0000 7fd0b1c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ac111a70 con 0x7fd0ac1028c0 2024-03-21T10:42:57.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.936+0000 7fd0b1c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ac1118f0 con 0x7fd0ac10bc90 2024-03-21T10:42:57.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.936+0000 7fd0b1c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ac111bf0 con 0x7fd0ac1031e0 2024-03-21T10:42:57.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.936+0000 7fd0ab400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0ac10bc90 0x7fd0ac19c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:57.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.936+0000 7fd0ab400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0ac10bc90 0x7fd0ac19c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:52218/0 (socket says 172.21.15.31:52218) 2024-03-21T10:42:57.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.936+0000 7fd0ab400640 1 -- 172.21.15.31:0/1642434627 learned_addr learned my addr 172.21.15.31:0/1642434627 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:57.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.937+0000 7fd0aa000640 1 --2- 172.21.15.31:0/1642434627 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0ac1031e0 0x7fd0ac1984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:57.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.938+0000 7fd0aaa00640 1 --2- 172.21.15.31:0/1642434627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0ac1028c0 0x7fd0ac197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:57.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.940+0000 7fd0aa000640 1 -- 172.21.15.31:0/1642434627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0ac10bc90 msgr2=0x7fd0ac19c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:57.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.940+0000 7fd0aa000640 1 --2- 172.21.15.31:0/1642434627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0ac10bc90 0x7fd0ac19c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:57.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.940+0000 7fd0aa000640 1 -- 172.21.15.31:0/1642434627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0ac1028c0 msgr2=0x7fd0ac197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:57.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.942+0000 7fd0aa000640 1 --2- 172.21.15.31:0/1642434627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0ac1028c0 0x7fd0ac197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:57.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.942+0000 7fd0aa000640 1 -- 172.21.15.31:0/1642434627 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ac19cfa0 con 0x7fd0ac1031e0 2024-03-21T10:42:57.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.943+0000 7fd0aaa00640 1 --2- 172.21.15.31:0/1642434627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0ac1028c0 0x7fd0ac197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:42:57.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.943+0000 7fd0aa000640 1 --2- 172.21.15.31:0/1642434627 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0ac1031e0 0x7fd0ac1984f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd09800c8f0 tx=0x7fd09800cdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:57.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.946+0000 7fd0a3400640 1 -- 172.21.15.31:0/1642434627 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09800ae60 con 0x7fd0ac1031e0 2024-03-21T10:42:57.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.956+0000 7fd0a3400640 1 -- 172.21.15.31:0/1642434627 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd098005500 con 0x7fd0ac1031e0 2024-03-21T10:42:57.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.964+0000 7fd0a3400640 1 -- 172.21.15.31:0/1642434627 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd098004e30 con 0x7fd0ac1031e0 2024-03-21T10:42:57.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.964+0000 7fd0b1c00640 1 -- 172.21.15.31:0/1642434627 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0ac19d290 con 0x7fd0ac1031e0 2024-03-21T10:42:57.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.964+0000 7fd0b1c00640 1 -- 172.21.15.31:0/1642434627 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0ac1a4b20 con 0x7fd0ac1031e0 2024-03-21T10:42:57.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.979+0000 7fd0b4812740 1 Processor -- start 2024-03-21T10:42:57.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.980+0000 7fd0a3400640 1 -- 172.21.15.31:0/1642434627 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fd098004fd0 con 0x7fd0ac1031e0 2024-03-21T10:42:57.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.983+0000 7fd0a3400640 1 --2- 172.21.15.31:0/1642434627 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd0900a3210 0x7fd0900a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:57.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.983+0000 7fd0a3400640 1 -- 172.21.15.31:0/1642434627 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fd0980cf910 con 0x7fd0ac1031e0 2024-03-21T10:42:57.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.989+0000 7fd0ab400640 1 --2- 172.21.15.31:0/1642434627 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd0900a3210 0x7fd0900a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:57.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.990+0000 7fd0b4812740 1 -- start start 2024-03-21T10:42:57.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.991+0000 7fd0b4812740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474c4080 0x5581474c4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:57.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.991+0000 7fd0b4812740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5581474c49a0 0x5581474cd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:57.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.992+0000 7fd0b4812740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5581474cd580 0x5581474cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:57.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.992+0000 7fd0b4812740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558147428eb0 con 0x5581474cd580 2024-03-21T10:42:57.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.992+0000 7fd0b4812740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581473d1a60 con 0x5581474c49a0 2024-03-21T10:42:57.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.992+0000 7fd0b4812740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x558147424f50 con 0x5581474c4080 2024-03-21T10:42:57.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.993+0000 7fd0aaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474c4080 0x5581474c4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:57.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.993+0000 7fd0aaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474c4080 0x5581474c4460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34602/0 (socket says 172.21.15.31:34602) 2024-03-21T10:42:57.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.993+0000 7fd0aaa00640 1 -- 172.21.15.31:0/3498832697 learned_addr learned my addr 172.21.15.31:0/3498832697 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:57.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.993+0000 7fd0aa000640 1 --2- 172.21.15.31:0/3498832697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5581474c49a0 0x5581474cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:57.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.994+0000 7fd0aa000640 1 -- 172.21.15.31:0/3498832697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474c4080 msgr2=0x5581474c4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:57.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.994+0000 7fd0aa000640 1 --2- 172.21.15.31:0/3498832697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474c4080 0x5581474c4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:57.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.994+0000 7fd0aa000640 1 -- 172.21.15.31:0/3498832697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5581474cd580 msgr2=0x5581474cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:57.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.994+0000 7fd0ab400640 1 --2- 172.21.15.31:0/3498832697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5581474cd580 0x5581474cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:57.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.994+0000 7fd0aa000640 1 --2- 172.21.15.31:0/3498832697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5581474cd580 0x5581474cf970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:57.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.995+0000 7fd0aa000640 1 -- 172.21.15.31:0/3498832697 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581472057e0 con 0x5581474c49a0 2024-03-21T10:42:57.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.995+0000 7fd0aaa00640 1 --2- 172.21.15.31:0/3498832697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474c4080 0x5581474c4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:42:57.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.995+0000 7fd0ab400640 1 --2- 172.21.15.31:0/3498832697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5581474cd580 0x5581474cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:42:57.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.997+0000 7fd0aa000640 1 --2- 172.21.15.31:0/3498832697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5581474c49a0 0x5581474cd040 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd0980ccd20 tx=0x7fd09809c280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db1cd7c518c3e0d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:57.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.998+0000 7fd097e00640 1 -- 172.21.15.31:0/3498832697 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0980a4450 con 0x5581474c49a0 2024-03-21T10:42:57.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.000+0000 7fd097e00640 1 -- 172.21.15.31:0/3498832697 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd098099ae0 con 0x5581474c49a0 2024-03-21T10:42:57.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.000+0000 7fd097e00640 1 -- 172.21.15.31:0/3498832697 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0980980b0 con 0x5581474c49a0 2024-03-21T10:42:57.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:57.998+0000 7fd0ab400640 1 --2- 172.21.15.31:0/1642434627 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd0900a3210 0x7fd0900a56d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4009a60 tx=0x7fd0a40324f0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:57.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.001+0000 7fd0b4812740 1 -- 172.21.15.31:0/3498832697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5581474c49a0 msgr2=0x5581474cd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:57.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.001+0000 7fd0b4812740 1 --2- 172.21.15.31:0/3498832697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5581474c49a0 0x5581474cd040 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd0980ccd20 tx=0x7fd09809c280 comp rx=0 tx=0).stop 2024-03-21T10:42:57.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.001+0000 7fd0b4812740 1 -- 172.21.15.31:0/3498832697 shutdown_connections 2024-03-21T10:42:57.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.002+0000 7fd0b4812740 1 --2- 172.21.15.31:0/3498832697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5581474cd580 0x5581474cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:57.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.002+0000 7fd0b4812740 1 --2- 172.21.15.31:0/3498832697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5581474c49a0 0x5581474cd040 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:57.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.002+0000 7fd0b4812740 1 --2- 172.21.15.31:0/3498832697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474c4080 0x5581474c4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:57.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.002+0000 7fd0b4812740 1 -- 172.21.15.31:0/3498832697 >> 172.21.15.31:0/3498832697 conn(0x5581474be7b0 msgr2=0x5581474beb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:57.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.002+0000 7fd0b4812740 1 -- 172.21.15.31:0/3498832697 shutdown_connections 2024-03-21T10:42:57.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.002+0000 7fd0b4812740 1 -- 172.21.15.31:0/3498832697 wait complete. 2024-03-21T10:42:58.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.003+0000 7fd0b4812740 1 -- start start 2024-03-21T10:42:58.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.003+0000 7fd0b4812740 1 Processor -- start 2024-03-21T10:42:58.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.009+0000 7fd0b4812740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5581474c4080 0x5581474ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.010+0000 7fd0b4812740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5581474c49a0 0x5581474ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.010+0000 7fd0b4812740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474cd580 0x5581474ca900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.010+0000 7fd0b4812740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581474d0370 con 0x5581474c49a0 2024-03-21T10:42:58.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.010+0000 7fd0b4812740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581473e4eb0 con 0x5581474c4080 2024-03-21T10:42:58.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.010+0000 7fd0b4812740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581474d04f0 con 0x5581474cd580 2024-03-21T10:42:58.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.010+0000 7fd0b4812740 10 client.0 _ll_register_callbacks cb 0x5581473d9150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:58.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.011+0000 7fd0aaa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5581474c4080 0x5581474ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.011+0000 7fd0ab400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474cd580 0x5581474ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.011+0000 7fd0ab400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474cd580 0x5581474ca900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34612/0 (socket says 172.21.15.31:34612) 2024-03-21T10:42:58.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.011+0000 7fd0ab400640 1 -- 172.21.15.31:0/2268571876 learned_addr learned my addr 172.21.15.31:0/2268571876 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:58.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.011+0000 7fd0aaa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5581474c4080 0x5581474ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:52228/0 (socket says 172.21.15.31:52228) 2024-03-21T10:42:58.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.012+0000 7fd0aaa00640 1 -- 172.21.15.31:0/2268571876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474cd580 msgr2=0x5581474ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:58.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.012+0000 7fd0aa000640 1 --2- 172.21.15.31:0/2268571876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5581474c49a0 0x5581474ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.012+0000 7fd0aaa00640 1 --2- 172.21.15.31:0/2268571876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474cd580 0x5581474ca900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.012+0000 7fd0aaa00640 1 -- 172.21.15.31:0/2268571876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5581474c49a0 msgr2=0x5581474ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:58.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.012+0000 7fd0aaa00640 1 --2- 172.21.15.31:0/2268571876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5581474c49a0 0x5581474ca2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.012+0000 7fd0aaa00640 1 -- 172.21.15.31:0/2268571876 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558147265f70 con 0x5581474c4080 2024-03-21T10:42:58.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.013+0000 7fd0aaa00640 1 --2- 172.21.15.31:0/2268571876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5581474c4080 0x5581474ccf60 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd09c00f4c0 tx=0x7fd09c00f980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=134c706fb1a1e42d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:58.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.013+0000 7fd094c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:42:58.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.013+0000 7fd094c00640 1 -- 172.21.15.31:0/2268571876 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09c013850 con 0x5581474c4080 2024-03-21T10:42:58.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.013+0000 7fd094c00640 1 -- 172.21.15.31:0/2268571876 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fd09c0139f0 con 0x5581474c4080 2024-03-21T10:42:58.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.014+0000 7fd0b4812740 1 -- 172.21.15.31:0/2268571876 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd098099ae0 con 0x5581474c4080 2024-03-21T10:42:58.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.014+0000 7fd094c00640 1 -- 172.21.15.31:0/2268571876 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd09c013c10 con 0x5581474c4080 2024-03-21T10:42:58.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.014+0000 7fd0b4812740 10 client.26774 fetch_fsmap learned FSMap version 26 2024-03-21T10:42:58.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.014+0000 7fd0b4812740 1 -- 172.21.15.31:0/2268571876 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5581470fd170 con 0x5581474c4080 2024-03-21T10:42:58.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.015+0000 7fd094c00640 1 -- 172.21.15.31:0/2268571876 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd098099ae0 con 0x5581474c4080 2024-03-21T10:42:58.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.015+0000 7fd094c00640 1 -- 172.21.15.31:0/2268571876 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd09c01b420 con 0x5581474c4080 2024-03-21T10:42:58.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.015+0000 7fd0b4812740 10 client.26774 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:42:58.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.015+0000 7fd0b4812740 10 client.26774 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:42:58.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.015+0000 7fd0b4812740 20 client.26774 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:58.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.015+0000 7fd0b4812740 1 --2- 172.21.15.31:0/2268571876 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5581474e5dd0 0x5581474e81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.015+0000 7fd0b4812740 4 client.26774 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:42:58.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.015+0000 7fd0b4812740 1 -- 172.21.15.31:0/2268571876 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5581474363f0 con 0x5581474e5dd0 2024-03-21T10:42:58.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.016+0000 7fd0aa000640 1 --2- 172.21.15.31:0/2268571876 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5581474e5dd0 0x5581474e81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.017+0000 7fd094c00640 10 client.26774 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:58.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.018+0000 7fd0aa000640 1 --2- 172.21.15.31:0/2268571876 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5581474e5dd0 0x5581474e81c0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af664420886f1400 server_cookie=36991bb1a16e911c in_seq=0 out_seq=0 2024-03-21T10:42:58.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.018+0000 7fd094c00640 1 -- 172.21.15.31:0/2268571876 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5581474363f0 con 0x5581474e5dd0 2024-03-21T10:42:58.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.018+0000 7fd094c00640 10 client.26774 handle_command_reply: tid=0 2024-03-21T10:42:58.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.019+0000 7fd0b4812740 1 client.26774 shutdown 2024-03-21T10:42:58.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.019+0000 7fd0b4812740 1 -- 172.21.15.31:0/2268571876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5581474c4080 msgr2=0x5581474ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:58.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.019+0000 7fd0b4812740 1 --2- 172.21.15.31:0/2268571876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5581474c4080 0x5581474ccf60 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd09c00f4c0 tx=0x7fd09c00f980 comp rx=0 tx=0).stop 2024-03-21T10:42:58.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.021+0000 7fd0b4812740 1 -- 172.21.15.31:0/2268571876 shutdown_connections 2024-03-21T10:42:58.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.021+0000 7fd0b4812740 1 --2- 172.21.15.31:0/2268571876 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5581474e5dd0 0x5581474e81c0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.021+0000 7fd0b4812740 1 --2- 172.21.15.31:0/2268571876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474cd580 0x5581474ca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.021+0000 7fd094c00640 0 client.26774 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:58.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.021+0000 7fd0b4812740 1 --2- 172.21.15.31:0/2268571876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5581474c49a0 0x5581474ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.021+0000 7fd0b4812740 1 --2- 172.21.15.31:0/2268571876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5581474c4080 0x5581474ccf60 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.021+0000 7fd0b4812740 1 -- 172.21.15.31:0/2268571876 >> 172.21.15.31:0/2268571876 conn(0x5581474be7b0 msgr2=0x5581474c0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:58.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.022+0000 7fd0b4812740 1 -- 172.21.15.31:0/2268571876 shutdown_connections 2024-03-21T10:42:58.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.022+0000 7fd0b4812740 1 -- 172.21.15.31:0/2268571876 wait complete. 2024-03-21T10:42:58.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.022+0000 7fd0b4812740 20 client.26774 trim_cache size 0 max 16384 2024-03-21T10:42:58.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.061+0000 7fd0b4812740 1 Processor -- start 2024-03-21T10:42:58.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.061+0000 7fd0b4812740 1 -- start start 2024-03-21T10:42:58.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.061+0000 7fd0b4812740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474c4080 0x5581474cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.061+0000 7fd0b4812740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5581474c49a0 0x5581474ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.061+0000 7fd0b4812740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5581474cd580 0x5581474d8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.061+0000 7fd0b4812740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ac1aeda0 con 0x5581474cd580 2024-03-21T10:42:58.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.061+0000 7fd0b4812740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ac1ac8a0 con 0x5581474c49a0 2024-03-21T10:42:58.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.061+0000 7fd0b4812740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ac1aa940 con 0x5581474c4080 2024-03-21T10:42:58.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.062+0000 7fd0aa000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5581474c49a0 0x5581474ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.062+0000 7fd0aaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474c4080 0x5581474cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.062+0000 7fd0aaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474c4080 0x5581474cca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34628/0 (socket says 172.21.15.31:34628) 2024-03-21T10:42:58.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.062+0000 7fd0aaa00640 1 -- 172.21.15.31:0/2538170287 learned_addr learned my addr 172.21.15.31:0/2538170287 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:58.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.062+0000 7fd0aa000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5581474c49a0 0x5581474ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:52242/0 (socket says 172.21.15.31:52242) 2024-03-21T10:42:58.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.062+0000 7fd0ab400640 1 --2- 172.21.15.31:0/2538170287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5581474cd580 0x5581474d8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.062+0000 7fd0aaa00640 1 -- 172.21.15.31:0/2538170287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5581474c49a0 msgr2=0x5581474ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:58.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.062+0000 7fd0aaa00640 1 --2- 172.21.15.31:0/2538170287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5581474c49a0 0x5581474ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.062+0000 7fd0aaa00640 1 -- 172.21.15.31:0/2538170287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5581474cd580 msgr2=0x5581474d8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:58.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.062+0000 7fd0aaa00640 1 --2- 172.21.15.31:0/2538170287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5581474cd580 0x5581474d8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.062+0000 7fd0aaa00640 1 -- 172.21.15.31:0/2538170287 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581474b9c20 con 0x5581474c4080 2024-03-21T10:42:58.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.062+0000 7fd0aa000640 1 --2- 172.21.15.31:0/2538170287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5581474c49a0 0x5581474ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:42:58.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.062+0000 7fd0aaa00640 1 --2- 172.21.15.31:0/2538170287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474c4080 0x5581474cca20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd09c00a120 tx=0x7fd09c00a150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=171562ed9dce7a7b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:58.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.063+0000 7fd097400640 1 -- 172.21.15.31:0/2538170287 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09c006ba0 con 0x5581474c4080 2024-03-21T10:42:58.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.063+0000 7fd097400640 1 -- 172.21.15.31:0/2538170287 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd09c02a070 con 0x5581474c4080 2024-03-21T10:42:58.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.063+0000 7fd097400640 1 -- 172.21.15.31:0/2538170287 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09c025820 con 0x5581474c4080 2024-03-21T10:42:58.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.064+0000 7fd0b4812740 1 -- 172.21.15.31:0/2538170287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474c4080 msgr2=0x5581474cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:58.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.064+0000 7fd0b4812740 1 --2- 172.21.15.31:0/2538170287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474c4080 0x5581474cca20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd09c00a120 tx=0x7fd09c00a150 comp rx=0 tx=0).stop 2024-03-21T10:42:58.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.064+0000 7fd0b4812740 1 -- 172.21.15.31:0/2538170287 shutdown_connections 2024-03-21T10:42:58.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.064+0000 7fd0b4812740 1 --2- 172.21.15.31:0/2538170287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5581474cd580 0x5581474d8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.064+0000 7fd0b4812740 1 --2- 172.21.15.31:0/2538170287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5581474c49a0 0x5581474ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.064+0000 7fd0b4812740 1 --2- 172.21.15.31:0/2538170287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474c4080 0x5581474cca20 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.064+0000 7fd0b4812740 1 -- 172.21.15.31:0/2538170287 >> 172.21.15.31:0/2538170287 conn(0x5581474be7b0 msgr2=0x5581474c0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:58.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.064+0000 7fd0b4812740 1 -- 172.21.15.31:0/2538170287 shutdown_connections 2024-03-21T10:42:58.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.064+0000 7fd0b4812740 1 -- 172.21.15.31:0/2538170287 wait complete. 2024-03-21T10:42:58.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.065+0000 7fd0b4812740 1 -- start start 2024-03-21T10:42:58.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.065+0000 7fd0b4812740 1 Processor -- start 2024-03-21T10:42:58.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.065+0000 7fd0b4812740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474c4080 0x5581474d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.065+0000 7fd0b4812740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5581474c49a0 0x5581474d2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.065+0000 7fd0b4812740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5581474cd580 0x5581474d3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.065+0000 7fd0b4812740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ac1aaf40 con 0x5581474c49a0 2024-03-21T10:42:58.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.065+0000 7fd0b4812740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ac1ac240 con 0x5581474cd580 2024-03-21T10:42:58.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.065+0000 7fd0b4812740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ac1ab580 con 0x5581474c4080 2024-03-21T10:42:58.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.065+0000 7fd0b4812740 10 client.0 _ll_register_callbacks cb 0x7fd09c00adb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:58.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.065+0000 7fd0aaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474c4080 0x5581474d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.066+0000 7fd0aaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474c4080 0x5581474d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34632/0 (socket says 172.21.15.31:34632) 2024-03-21T10:42:58.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.066+0000 7fd0aaa00640 1 -- 172.21.15.31:0/4162771928 learned_addr learned my addr 172.21.15.31:0/4162771928 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:58.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.066+0000 7fd0aa000640 1 --2- 172.21.15.31:0/4162771928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5581474c49a0 0x5581474d2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.066+0000 7fd0ab400640 1 --2- 172.21.15.31:0/4162771928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5581474cd580 0x5581474d3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.066+0000 7fd0aaa00640 1 -- 172.21.15.31:0/4162771928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5581474cd580 msgr2=0x5581474d3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:58.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.066+0000 7fd0aaa00640 1 --2- 172.21.15.31:0/4162771928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5581474cd580 0x5581474d3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.066+0000 7fd0aaa00640 1 -- 172.21.15.31:0/4162771928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5581474c49a0 msgr2=0x5581474d2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:58.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.066+0000 7fd0aaa00640 1 --2- 172.21.15.31:0/4162771928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5581474c49a0 0x5581474d2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.066+0000 7fd0aaa00640 1 -- 172.21.15.31:0/4162771928 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5581474c8380 con 0x5581474c4080 2024-03-21T10:42:58.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.066+0000 7fd0aa000640 1 --2- 172.21.15.31:0/4162771928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5581474c49a0 0x5581474d2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:42:58.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.066+0000 7fd0ab400640 1 --2- 172.21.15.31:0/4162771928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5581474cd580 0x5581474d3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:42:58.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.066+0000 7fd0aaa00640 1 --2- 172.21.15.31:0/4162771928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474c4080 0x5581474d2710 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd09c000c00 tx=0x7fd09c006bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa4432a71da4d51d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:58.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.066+0000 7fd094c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:42:58.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.067+0000 7fd094c00640 1 -- 172.21.15.31:0/4162771928 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09c03c060 con 0x5581474c4080 2024-03-21T10:42:58.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.067+0000 7fd094c00640 1 -- 172.21.15.31:0/4162771928 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fd09c025620 con 0x5581474c4080 2024-03-21T10:42:58.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.067+0000 7fd0b4812740 1 -- 172.21.15.31:0/4162771928 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd09c02a070 con 0x5581474c4080 2024-03-21T10:42:58.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.068+0000 7fd094c00640 1 -- 172.21.15.31:0/4162771928 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd09c025820 con 0x5581474c4080 2024-03-21T10:42:58.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.068+0000 7fd0b4812740 10 client.26722 fetch_fsmap learned FSMap version 26 2024-03-21T10:42:58.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.068+0000 7fd0b4812740 1 -- 172.21.15.31:0/4162771928 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x558147542710 con 0x5581474c4080 2024-03-21T10:42:58.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.068+0000 7fd094c00640 1 -- 172.21.15.31:0/4162771928 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd09c02a070 con 0x5581474c4080 2024-03-21T10:42:58.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.069+0000 7fd094c00640 1 -- 172.21.15.31:0/4162771928 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd09c02e070 con 0x5581474c4080 2024-03-21T10:42:58.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.069+0000 7fd0b4812740 10 client.26722 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:42:58.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.069+0000 7fd0b4812740 10 client.26722 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:42:58.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.069+0000 7fd0b4812740 20 client.26722 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:58.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.069+0000 7fd0b4812740 1 --2- 172.21.15.31:0/4162771928 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5581474e5dd0 0x558147544be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.069+0000 7fd0b4812740 4 client.26722 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-21T10:42:58.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.069+0000 7fd0b4812740 1 -- 172.21.15.31:0/4162771928 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x558147430740 con 0x5581474e5dd0 2024-03-21T10:42:58.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.070+0000 7fd0aa000640 1 --2- 172.21.15.31:0/4162771928 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5581474e5dd0 0x558147544be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.070+0000 7fd0aa000640 1 --2- 172.21.15.31:0/4162771928 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5581474e5dd0 0x558147544be0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5711fbbcdf05db8b server_cookie=cc3f15add257b04b in_seq=0 out_seq=0 2024-03-21T10:42:58.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.070+0000 7fd094c00640 10 client.26722 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:58.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.072+0000 7fd094c00640 1 -- 172.21.15.31:0/4162771928 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x558147430740 con 0x5581474e5dd0 2024-03-21T10:42:58.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.072+0000 7fd094c00640 10 client.26722 handle_command_reply: tid=0 2024-03-21T10:42:58.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.072+0000 7fd0b4812740 1 client.26722 shutdown 2024-03-21T10:42:58.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.073+0000 7fd0b4812740 1 -- 172.21.15.31:0/4162771928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474c4080 msgr2=0x5581474d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:58.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.073+0000 7fd0b4812740 1 --2- 172.21.15.31:0/4162771928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474c4080 0x5581474d2710 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd09c000c00 tx=0x7fd09c006bc0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.074+0000 7fd0b4812740 1 -- 172.21.15.31:0/4162771928 shutdown_connections 2024-03-21T10:42:58.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.074+0000 7fd0b4812740 1 --2- 172.21.15.31:0/4162771928 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5581474e5dd0 0x558147544be0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.074+0000 7fd0b4812740 1 --2- 172.21.15.31:0/4162771928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5581474cd580 0x5581474d3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.074+0000 7fd0b4812740 1 --2- 172.21.15.31:0/4162771928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5581474c49a0 0x5581474d2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.074+0000 7fd0b4812740 1 --2- 172.21.15.31:0/4162771928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581474c4080 0x5581474d2710 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.074+0000 7fd0b4812740 1 -- 172.21.15.31:0/4162771928 >> 172.21.15.31:0/4162771928 conn(0x5581474be7b0 msgr2=0x5581474c8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:58.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.074+0000 7fd094c00640 0 client.26722 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:58.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.074+0000 7fd0b4812740 1 -- 172.21.15.31:0/4162771928 shutdown_connections 2024-03-21T10:42:58.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.075+0000 7fd0b4812740 1 -- 172.21.15.31:0/4162771928 wait complete. 2024-03-21T10:42:58.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.076+0000 7fd0b4812740 20 client.26722 trim_cache size 0 max 16384 2024-03-21T10:42:58.074 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:42:58.074 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-03-21T10:42:58.074 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "bb644869-d5b4-4038-b428-1348ff8b6a68", 2024-03-21T10:42:58.074 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-03-21T10:42:58.074 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:42:58.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.078+0000 7fd0b1c00640 1 -- 172.21.15.31:0/1642434627 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd0900a3210 msgr2=0x7fd0900a56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:58.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.078+0000 7fd0b1c00640 1 --2- 172.21.15.31:0/1642434627 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd0900a3210 0x7fd0900a56d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4009a60 tx=0x7fd0a40324f0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.078+0000 7fd0b1c00640 1 -- 172.21.15.31:0/1642434627 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0ac1031e0 msgr2=0x7fd0ac1984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:58.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.078+0000 7fd0b1c00640 1 --2- 172.21.15.31:0/1642434627 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0ac1031e0 0x7fd0ac1984f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd09800c8f0 tx=0x7fd09800cdb0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.081+0000 7fd0b1c00640 1 -- 172.21.15.31:0/1642434627 shutdown_connections 2024-03-21T10:42:58.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.081+0000 7fd0b1c00640 1 --2- 172.21.15.31:0/1642434627 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd0900a3210 0x7fd0900a56d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.081+0000 7fd0b1c00640 1 --2- 172.21.15.31:0/1642434627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0ac10bc90 0x7fd0ac19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.081+0000 7fd0b1c00640 1 --2- 172.21.15.31:0/1642434627 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0ac1031e0 0x7fd0ac1984f0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.081+0000 7fd0b1c00640 1 --2- 172.21.15.31:0/1642434627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0ac1028c0 0x7fd0ac197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.081+0000 7fd0b1c00640 1 -- 172.21.15.31:0/1642434627 >> 172.21.15.31:0/1642434627 conn(0x7fd0ac0776f0 msgr2=0x7fd0ac0fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:58.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.082+0000 7fd0b1c00640 1 -- 172.21.15.31:0/1642434627 shutdown_connections 2024-03-21T10:42:58.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.082+0000 7fd0b1c00640 1 -- 172.21.15.31:0/1642434627 wait complete. 2024-03-21T10:42:58.104 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:42:58.105 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:42:58.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.484+0000 7f2857e00640 1 -- 172.21.15.31:0/291520438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2850108be0 msgr2=0x7f2850109040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:58.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.484+0000 7f2857e00640 1 --2- 172.21.15.31:0/291520438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2850108be0 0x7f2850109040 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f283c009860 tx=0x7f283c02bbb0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.484+0000 7f2857e00640 1 -- 172.21.15.31:0/291520438 shutdown_connections 2024-03-21T10:42:58.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.484+0000 7f2857e00640 1 --2- 172.21.15.31:0/291520438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f285011ae80 0x7f285011d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.484+0000 7f2857e00640 1 --2- 172.21.15.31:0/291520438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2850108be0 0x7f2850109040 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.484+0000 7f2857e00640 1 --2- 172.21.15.31:0/291520438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f285010fe30 0x7f28501086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.484+0000 7f2857e00640 1 -- 172.21.15.31:0/291520438 >> 172.21.15.31:0/291520438 conn(0x7f285006c710 msgr2=0x7f285006cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:58.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.484+0000 7f2857e00640 1 -- 172.21.15.31:0/291520438 shutdown_connections 2024-03-21T10:42:58.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.484+0000 7f2857e00640 1 -- 172.21.15.31:0/291520438 wait complete. 2024-03-21T10:42:58.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.486+0000 7f2857e00640 1 Processor -- start 2024-03-21T10:42:58.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.487+0000 7f2857e00640 1 -- start start 2024-03-21T10:42:58.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.487+0000 7f2857e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2850108be0 0x7f285011a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.488+0000 7f2855400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2850108be0 0x7f285011a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.488+0000 7f2855400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2850108be0 0x7f285011a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47248/0 (socket says 172.21.15.31:47248) 2024-03-21T10:42:58.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.488+0000 7f2855400640 1 -- 172.21.15.31:0/3561726227 learned_addr learned my addr 172.21.15.31:0/3561726227 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:58.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.489+0000 7f2857e00640 1 --2- 172.21.15.31:0/3561726227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f285010fe30 0x7f2850116690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.489+0000 7f2857e00640 1 --2- 172.21.15.31:0/3561726227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f285011ae80 0x7f2850116bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.490+0000 7f2855e00640 1 --2- 172.21.15.31:0/3561726227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f285011ae80 0x7f2850116bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.490+0000 7f2857e00640 1 -- 172.21.15.31:0/3561726227 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2850112ff0 con 0x7f2850108be0 2024-03-21T10:42:58.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.490+0000 7f2857e00640 1 -- 172.21.15.31:0/3561726227 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2850112e70 con 0x7f285011ae80 2024-03-21T10:42:58.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.490+0000 7f2857e00640 1 -- 172.21.15.31:0/3561726227 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2850113170 con 0x7f285010fe30 2024-03-21T10:42:58.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.490+0000 7f2854a00640 1 --2- 172.21.15.31:0/3561726227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f285010fe30 0x7f2850116690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.491+0000 7f2855e00640 1 -- 172.21.15.31:0/3561726227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f285010fe30 msgr2=0x7f2850116690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:58.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.491+0000 7f2855e00640 1 --2- 172.21.15.31:0/3561726227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f285010fe30 0x7f2850116690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.491+0000 7f2855e00640 1 -- 172.21.15.31:0/3561726227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2850108be0 msgr2=0x7f285011a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:58.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.491+0000 7f2855e00640 1 --2- 172.21.15.31:0/3561726227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2850108be0 0x7f285011a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.491+0000 7f2855e00640 1 -- 172.21.15.31:0/3561726227 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2850117300 con 0x7f285011ae80 2024-03-21T10:42:58.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.492+0000 7f2855e00640 1 --2- 172.21.15.31:0/3561726227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f285011ae80 0x7f2850116bd0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2848005fd0 tx=0x7f284801d380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:58.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.493+0000 7f284e000640 1 -- 172.21.15.31:0/3561726227 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284801ddc0 con 0x7f285011ae80 2024-03-21T10:42:58.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.493+0000 7f284e000640 1 -- 172.21.15.31:0/3561726227 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2848003970 con 0x7f285011ae80 2024-03-21T10:42:58.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.493+0000 7f284e000640 1 -- 172.21.15.31:0/3561726227 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2848004530 con 0x7f285011ae80 2024-03-21T10:42:58.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.493+0000 7f2857e00640 1 -- 172.21.15.31:0/3561726227 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28501aac70 con 0x7f285011ae80 2024-03-21T10:42:58.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.494+0000 7f2857e00640 1 -- 172.21.15.31:0/3561726227 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28501ab080 con 0x7f285011ae80 2024-03-21T10:42:58.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.500+0000 7f284e000640 1 -- 172.21.15.31:0/3561726227 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f284802e020 con 0x7f285011ae80 2024-03-21T10:42:58.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.502+0000 7f284e000640 1 --2- 172.21.15.31:0/3561726227 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f28380a31e0 0x7f28380a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.502+0000 7f284e000640 1 -- 172.21.15.31:0/3561726227 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f28480e03d0 con 0x7f285011ae80 2024-03-21T10:42:58.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.502+0000 7f2855400640 1 --2- 172.21.15.31:0/3561726227 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f28380a31e0 0x7f28380a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.504+0000 7f285ab9f740 1 Processor -- start 2024-03-21T10:42:58.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.504+0000 7f285ab9f740 1 -- start start 2024-03-21T10:42:58.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.505+0000 7f285ab9f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea14cc00 0x55aeea14b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.505+0000 7f285ab9f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea14bf20 0x55aeea14c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.505+0000 7f285ab9f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aeea155290 0x55aeea157680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.505+0000 7f285ab9f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeea0b0250 con 0x55aeea155290 2024-03-21T10:42:58.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.505+0000 7f285ab9f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeea058a30 con 0x55aeea14cc00 2024-03-21T10:42:58.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.505+0000 7f285ab9f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeea0ac2f0 con 0x55aeea14bf20 2024-03-21T10:42:58.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.506+0000 7f2855400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea14cc00 0x55aeea14b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.506+0000 7f2855400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea14cc00 0x55aeea14b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:52272/0 (socket says 172.21.15.31:52272) 2024-03-21T10:42:58.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.506+0000 7f2855400640 1 -- 172.21.15.31:0/1049595249 learned_addr learned my addr 172.21.15.31:0/1049595249 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:58.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.506+0000 7f2855e00640 1 --2- 172.21.15.31:0/1049595249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aeea155290 0x55aeea157680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.506+0000 7f2854a00640 1 --2- 172.21.15.31:0/1049595249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea14bf20 0x55aeea14c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.506+0000 7f2855400640 1 --2- 172.21.15.31:0/3561726227 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f28380a31e0 0x7f28380a56a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2844005fd0 tx=0x7f284400f730 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:58.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.506+0000 7f2855400640 1 -- 172.21.15.31:0/1049595249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea14bf20 msgr2=0x55aeea14c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:58.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.506+0000 7f2855400640 1 --2- 172.21.15.31:0/1049595249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea14bf20 0x55aeea14c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.506+0000 7f2855400640 1 -- 172.21.15.31:0/1049595249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aeea155290 msgr2=0x55aeea157680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:58.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.506+0000 7f2855400640 1 --2- 172.21.15.31:0/1049595249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aeea155290 0x55aeea157680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.507+0000 7f2855400640 1 -- 172.21.15.31:0/1049595249 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aeea043690 con 0x55aeea14cc00 2024-03-21T10:42:58.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.507+0000 7f2855e00640 1 --2- 172.21.15.31:0/1049595249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aeea155290 0x55aeea157680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:42:58.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.507+0000 7f2855400640 1 --2- 172.21.15.31:0/1049595249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea14cc00 0x55aeea14b950 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f284400bfd0 tx=0x7f284401a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1164cc191b941db8 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:58.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.508+0000 7f2842600640 1 -- 172.21.15.31:0/1049595249 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2844005220 con 0x55aeea14cc00 2024-03-21T10:42:58.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.508+0000 7f2842600640 1 -- 172.21.15.31:0/1049595249 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2844004c80 con 0x55aeea14cc00 2024-03-21T10:42:58.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.508+0000 7f2842600640 1 -- 172.21.15.31:0/1049595249 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284401daf0 con 0x55aeea14cc00 2024-03-21T10:42:58.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.508+0000 7f285ab9f740 1 -- 172.21.15.31:0/1049595249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea14cc00 msgr2=0x55aeea14b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:58.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.508+0000 7f285ab9f740 1 --2- 172.21.15.31:0/1049595249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea14cc00 0x55aeea14b950 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f284400bfd0 tx=0x7f284401a000 comp rx=0 tx=0).stop 2024-03-21T10:42:58.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.509+0000 7f285ab9f740 1 -- 172.21.15.31:0/1049595249 shutdown_connections 2024-03-21T10:42:58.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.509+0000 7f285ab9f740 1 --2- 172.21.15.31:0/1049595249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aeea155290 0x55aeea157680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.509+0000 7f285ab9f740 1 --2- 172.21.15.31:0/1049595249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea14bf20 0x55aeea14c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.509+0000 7f285ab9f740 1 --2- 172.21.15.31:0/1049595249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea14cc00 0x55aeea14b950 secure :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f284400bfd0 tx=0x7f284401a000 comp rx=0 tx=0).stop 2024-03-21T10:42:58.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.509+0000 7f285ab9f740 1 -- 172.21.15.31:0/1049595249 >> 172.21.15.31:0/1049595249 conn(0x55aeea113200 msgr2=0x55aeea146970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:58.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.510+0000 7f285ab9f740 1 -- 172.21.15.31:0/1049595249 shutdown_connections 2024-03-21T10:42:58.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.510+0000 7f285ab9f740 1 -- 172.21.15.31:0/1049595249 wait complete. 2024-03-21T10:42:58.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.510+0000 7f285ab9f740 1 -- start start 2024-03-21T10:42:58.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.511+0000 7f285ab9f740 1 Processor -- start 2024-03-21T10:42:58.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.511+0000 7f285ab9f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aeea14bf20 0x55aeea154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.511+0000 7f285ab9f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea155290 0x55aeea151e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.512+0000 7f285ab9f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea152350 0x55aeea1527d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.512+0000 7f285ab9f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeea158100 con 0x55aeea14bf20 2024-03-21T10:42:58.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.512+0000 7f285ab9f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeea06be80 con 0x55aeea152350 2024-03-21T10:42:58.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.512+0000 7f285ab9f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeea158280 con 0x55aeea155290 2024-03-21T10:42:58.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.513+0000 7f285ab9f740 10 client.0 _ll_register_callbacks cb 0x55aeea07e130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:58.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.513+0000 7f2855400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aeea14bf20 0x55aeea154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.513+0000 7f2855e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea152350 0x55aeea1527d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.513+0000 7f2854a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea155290 0x55aeea151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.513+0000 7f2855e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea152350 0x55aeea1527d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:52280/0 (socket says 172.21.15.31:52280) 2024-03-21T10:42:58.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.513+0000 7f2855e00640 1 -- 172.21.15.31:0/3734405141 learned_addr learned my addr 172.21.15.31:0/3734405141 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:58.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.515+0000 7f2855e00640 1 -- 172.21.15.31:0/3734405141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea155290 msgr2=0x55aeea151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:58.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.515+0000 7f2855e00640 1 --2- 172.21.15.31:0/3734405141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea155290 0x55aeea151e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.515+0000 7f2855e00640 1 -- 172.21.15.31:0/3734405141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aeea14bf20 msgr2=0x55aeea154ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:58.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.515+0000 7f2855e00640 1 --2- 172.21.15.31:0/3734405141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aeea14bf20 0x55aeea154ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.515+0000 7f2855e00640 1 -- 172.21.15.31:0/3734405141 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aee9e8e7d0 con 0x55aeea152350 2024-03-21T10:42:58.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.516+0000 7f2855e00640 1 --2- 172.21.15.31:0/3734405141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea152350 0x55aeea1527d0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f28480a8e50 tx=0x7f2848004000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d6a4ad705065908 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:58.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.516+0000 7f2836a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:42:58.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.520+0000 7f2836a00640 1 -- 172.21.15.31:0/3734405141 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284802b040 con 0x55aeea152350 2024-03-21T10:42:58.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.520+0000 7f2836a00640 1 -- 172.21.15.31:0/3734405141 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f28480a8af0 con 0x55aeea152350 2024-03-21T10:42:58.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.521+0000 7f285ab9f740 1 -- 172.21.15.31:0/3734405141 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2844004c80 con 0x55aeea152350 2024-03-21T10:42:58.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.521+0000 7f2836a00640 1 -- 172.21.15.31:0/3734405141 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28480ac480 con 0x55aeea152350 2024-03-21T10:42:58.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.522+0000 7f285ab9f740 10 client.26792 fetch_fsmap learned FSMap version 26 2024-03-21T10:42:58.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.522+0000 7f285ab9f740 1 -- 172.21.15.31:0/3734405141 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55aee9dc43f0 con 0x55aeea152350 2024-03-21T10:42:58.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.523+0000 7f2836a00640 1 -- 172.21.15.31:0/3734405141 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2844004c80 con 0x55aeea152350 2024-03-21T10:42:58.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.523+0000 7f2836a00640 1 -- 172.21.15.31:0/3734405141 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f28480b0160 con 0x55aeea152350 2024-03-21T10:42:58.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.524+0000 7f285ab9f740 10 client.26792 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:42:58.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.524+0000 7f285ab9f740 10 client.26792 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:42:58.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.524+0000 7f285ab9f740 20 client.26792 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:58.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.524+0000 7f285ab9f740 1 --2- 172.21.15.31:0/3734405141 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55aeea1657b0 0x55aeea167c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.524+0000 7f285ab9f740 4 client.26792 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:42:58.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.524+0000 7f285ab9f740 1 -- 172.21.15.31:0/3734405141 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aeea0bd790 con 0x55aeea1657b0 2024-03-21T10:42:58.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.524+0000 7f2855400640 1 --2- 172.21.15.31:0/3734405141 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55aeea1657b0 0x55aeea167c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.525+0000 7f2855400640 1 --2- 172.21.15.31:0/3734405141 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55aeea1657b0 0x55aeea167c10 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f52605cd5212c6 server_cookie=1570f026fd89f9a in_seq=0 out_seq=0 2024-03-21T10:42:58.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.525+0000 7f2836a00640 10 client.26792 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:58.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.526+0000 7f2836a00640 1 -- 172.21.15.31:0/3734405141 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55aeea0bd790 con 0x55aeea1657b0 2024-03-21T10:42:58.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.526+0000 7f2836a00640 10 client.26792 handle_command_reply: tid=0 2024-03-21T10:42:58.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.526+0000 7f285ab9f740 1 client.26792 shutdown 2024-03-21T10:42:58.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.526+0000 7f285ab9f740 1 -- 172.21.15.31:0/3734405141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea152350 msgr2=0x55aeea1527d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:58.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.526+0000 7f285ab9f740 1 --2- 172.21.15.31:0/3734405141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea152350 0x55aeea1527d0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f28480a8e50 tx=0x7f2848004000 comp rx=0 tx=0).stop 2024-03-21T10:42:58.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.527+0000 7f285ab9f740 1 -- 172.21.15.31:0/3734405141 shutdown_connections 2024-03-21T10:42:58.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.527+0000 7f285ab9f740 1 --2- 172.21.15.31:0/3734405141 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55aeea1657b0 0x55aeea167c10 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.527+0000 7f285ab9f740 1 --2- 172.21.15.31:0/3734405141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea152350 0x55aeea1527d0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.527+0000 7f285ab9f740 1 --2- 172.21.15.31:0/3734405141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea155290 0x55aeea151e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.527+0000 7f285ab9f740 1 --2- 172.21.15.31:0/3734405141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aeea14bf20 0x55aeea154ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.527+0000 7f285ab9f740 1 -- 172.21.15.31:0/3734405141 >> 172.21.15.31:0/3734405141 conn(0x55aeea113200 msgr2=0x55aeea1486c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:58.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.527+0000 7f2836a00640 0 client.26792 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:58.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.527+0000 7f285ab9f740 1 -- 172.21.15.31:0/3734405141 shutdown_connections 2024-03-21T10:42:58.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.527+0000 7f285ab9f740 1 -- 172.21.15.31:0/3734405141 wait complete. 2024-03-21T10:42:58.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.528+0000 7f285ab9f740 20 client.26792 trim_cache size 0 max 16384 2024-03-21T10:42:58.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.570+0000 7f285ab9f740 1 Processor -- start 2024-03-21T10:42:58.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.580+0000 7f285ab9f740 1 -- start start 2024-03-21T10:42:58.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.581+0000 7f285ab9f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea113200 0x55aeea14b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.581+0000 7f285ab9f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea14bf20 0x55aeea156680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.581+0000 7f285ab9f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aeea152350 0x55aeea156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.581+0000 7f285ab9f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28501b7370 con 0x55aeea152350 2024-03-21T10:42:58.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.581+0000 7f285ab9f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28501b4e70 con 0x55aeea14bf20 2024-03-21T10:42:58.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.581+0000 7f285ab9f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28501b2f10 con 0x55aeea113200 2024-03-21T10:42:58.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.581+0000 7f2854a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea14bf20 0x55aeea156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.581+0000 7f2854a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea14bf20 0x55aeea156680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:52296/0 (socket says 172.21.15.31:52296) 2024-03-21T10:42:58.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.581+0000 7f2854a00640 1 -- 172.21.15.31:0/3758247927 learned_addr learned my addr 172.21.15.31:0/3758247927 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:58.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.581+0000 7f2855400640 1 --2- 172.21.15.31:0/3758247927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea113200 0x55aeea14b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.582+0000 7f2855e00640 1 --2- 172.21.15.31:0/3758247927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aeea152350 0x55aeea156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.582+0000 7f2854a00640 1 -- 172.21.15.31:0/3758247927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea113200 msgr2=0x55aeea14b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:58.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.582+0000 7f2854a00640 1 --2- 172.21.15.31:0/3758247927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea113200 0x55aeea14b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.582+0000 7f2854a00640 1 -- 172.21.15.31:0/3758247927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aeea152350 msgr2=0x55aeea156bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:58.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.582+0000 7f2854a00640 1 --2- 172.21.15.31:0/3758247927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aeea152350 0x55aeea156bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.582+0000 7f2854a00640 1 -- 172.21.15.31:0/3758247927 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aee9d840f0 con 0x55aeea14bf20 2024-03-21T10:42:58.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.582+0000 7f2855400640 1 --2- 172.21.15.31:0/3758247927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea113200 0x55aeea14b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:42:58.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.584+0000 7f2854a00640 1 --2- 172.21.15.31:0/3758247927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea14bf20 0x55aeea156680 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f283c005b60 tx=0x7f283c035a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aaf3091c47ecb2a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:58.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.584+0000 7f2841c00640 1 -- 172.21.15.31:0/3758247927 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283c039040 con 0x55aeea14bf20 2024-03-21T10:42:58.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.584+0000 7f2841c00640 1 -- 172.21.15.31:0/3758247927 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f283c004030 con 0x55aeea14bf20 2024-03-21T10:42:58.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.584+0000 7f2841c00640 1 -- 172.21.15.31:0/3758247927 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283c005410 con 0x55aeea14bf20 2024-03-21T10:42:58.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.589+0000 7f285ab9f740 1 -- 172.21.15.31:0/3758247927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea14bf20 msgr2=0x55aeea156680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:58.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.589+0000 7f285ab9f740 1 --2- 172.21.15.31:0/3758247927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea14bf20 0x55aeea156680 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f283c005b60 tx=0x7f283c035a60 comp rx=0 tx=0).stop 2024-03-21T10:42:58.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.592+0000 7f285ab9f740 1 -- 172.21.15.31:0/3758247927 shutdown_connections 2024-03-21T10:42:58.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.592+0000 7f285ab9f740 1 --2- 172.21.15.31:0/3758247927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aeea152350 0x55aeea156bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.592+0000 7f285ab9f740 1 --2- 172.21.15.31:0/3758247927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea14bf20 0x55aeea156680 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.592+0000 7f285ab9f740 1 --2- 172.21.15.31:0/3758247927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea113200 0x55aeea14b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.592+0000 7f285ab9f740 1 -- 172.21.15.31:0/3758247927 >> 172.21.15.31:0/3758247927 conn(0x55aeea14d830 msgr2=0x55aeea14b1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:58.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.593+0000 7f285ab9f740 1 -- 172.21.15.31:0/3758247927 shutdown_connections 2024-03-21T10:42:58.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.593+0000 7f285ab9f740 1 -- 172.21.15.31:0/3758247927 wait complete. 2024-03-21T10:42:58.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.593+0000 7f285ab9f740 1 -- start start 2024-03-21T10:42:58.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.594+0000 7f285ab9f740 1 Processor -- start 2024-03-21T10:42:58.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.594+0000 7f285ab9f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea113200 0x55aeea150610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.594+0000 7f285ab9f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aeea14bf20 0x55aeea150b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.594+0000 7f285ab9f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea152350 0x55aeea15c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.594+0000 7f285ab9f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28501b3510 con 0x55aeea14bf20 2024-03-21T10:42:58.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.594+0000 7f285ab9f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28501b4810 con 0x55aeea113200 2024-03-21T10:42:58.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.594+0000 7f285ab9f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28501b3b50 con 0x55aeea152350 2024-03-21T10:42:58.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.595+0000 7f285ab9f740 10 client.0 _ll_register_callbacks cb 0x7f28480a8720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:42:58.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.596+0000 7f2855400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea113200 0x55aeea150610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.596+0000 7f2854a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aeea14bf20 0x55aeea150b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.596+0000 7f2854a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aeea14bf20 0x55aeea150b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47280/0 (socket says 172.21.15.31:47280) 2024-03-21T10:42:58.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.596+0000 7f2854a00640 1 -- 172.21.15.31:0/1460214937 learned_addr learned my addr 172.21.15.31:0/1460214937 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:42:58.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.596+0000 7f2855e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea152350 0x55aeea15c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.597+0000 7f2855e00640 1 -- 172.21.15.31:0/1460214937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea113200 msgr2=0x55aeea150610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:58.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.597+0000 7f2855e00640 1 --2- 172.21.15.31:0/1460214937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea113200 0x55aeea150610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.598+0000 7f2855e00640 1 -- 172.21.15.31:0/1460214937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aeea14bf20 msgr2=0x55aeea150b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:42:58.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.598+0000 7f2855e00640 1 --2- 172.21.15.31:0/1460214937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aeea14bf20 0x55aeea150b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.598+0000 7f2855e00640 1 -- 172.21.15.31:0/1460214937 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aeea15ca50 con 0x55aeea152350 2024-03-21T10:42:58.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.599+0000 7f2855400640 1 --2- 172.21.15.31:0/1460214937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea113200 0x55aeea150610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:42:58.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.599+0000 7f2854a00640 1 --2- 172.21.15.31:0/1460214937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aeea14bf20 0x55aeea150b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:42:58.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.600+0000 7f2855e00640 1 --2- 172.21.15.31:0/1460214937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea152350 0x55aeea15c510 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f28480aa730 tx=0x7f28480a87d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69c817b777fb764b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:42:58.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.600+0000 7f2836a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:42:58.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.600+0000 7f2836a00640 1 -- 172.21.15.31:0/1460214937 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284802b070 con 0x55aeea152350 2024-03-21T10:42:58.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.600+0000 7f285ab9f740 1 -- 172.21.15.31:0/1460214937 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f283c004030 con 0x55aeea152350 2024-03-21T10:42:58.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.601+0000 7f2836a00640 1 -- 172.21.15.31:0/1460214937 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f28480b0160 con 0x55aeea152350 2024-03-21T10:42:58.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.601+0000 7f2836a00640 1 -- 172.21.15.31:0/1460214937 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f284802b070 con 0x55aeea152350 2024-03-21T10:42:58.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.602+0000 7f285ab9f740 10 client.26740 fetch_fsmap learned FSMap version 26 2024-03-21T10:42:58.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.602+0000 7f285ab9f740 1 -- 172.21.15.31:0/1460214937 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55aeea1c1280 con 0x55aeea152350 2024-03-21T10:42:58.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.603+0000 7f2836a00640 1 -- 172.21.15.31:0/1460214937 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f283c004030 con 0x55aeea152350 2024-03-21T10:42:58.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.605+0000 7f2836a00640 1 -- 172.21.15.31:0/1460214937 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f28480ba020 con 0x55aeea152350 2024-03-21T10:42:58.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.605+0000 7f285ab9f740 10 client.26740 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:42:58.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.605+0000 7f285ab9f740 10 client.26740 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:42:58.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.605+0000 7f285ab9f740 20 client.26740 populate_metadata read hostname 'smithi031' 2024-03-21T10:42:58.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.605+0000 7f285ab9f740 1 --2- 172.21.15.31:0/1460214937 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55aeea155290 0x55aeea1c39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:42:58.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.605+0000 7f285ab9f740 4 client.26740 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:42:58.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.605+0000 7f285ab9f740 1 -- 172.21.15.31:0/1460214937 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55aeea0b7ae0 con 0x55aeea155290 2024-03-21T10:42:58.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.606+0000 7f2855400640 1 --2- 172.21.15.31:0/1460214937 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55aeea155290 0x55aeea1c39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:42:58.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.607+0000 7f2855400640 1 --2- 172.21.15.31:0/1460214937 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55aeea155290 0x55aeea1c39e0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c3107b3bd1fc1ca server_cookie=ccae1ea0c4045e55 in_seq=0 out_seq=0 2024-03-21T10:42:58.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.607+0000 7f2836a00640 10 client.26740 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:58.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.607+0000 7f2836a00640 1 -- 172.21.15.31:0/1460214937 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55aeea0b7ae0 con 0x55aeea155290 2024-03-21T10:42:58.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.607+0000 7f2836a00640 10 client.26740 handle_command_reply: tid=0 2024-03-21T10:42:58.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.607+0000 7f285ab9f740 1 client.26740 shutdown 2024-03-21T10:42:58.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.609+0000 7f285ab9f740 1 -- 172.21.15.31:0/1460214937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea152350 msgr2=0x55aeea15c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:58.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.609+0000 7f285ab9f740 1 --2- 172.21.15.31:0/1460214937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea152350 0x55aeea15c510 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f28480aa730 tx=0x7f28480a87d0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.609+0000 7f285ab9f740 1 -- 172.21.15.31:0/1460214937 shutdown_connections 2024-03-21T10:42:58.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.609+0000 7f285ab9f740 1 --2- 172.21.15.31:0/1460214937 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55aeea155290 0x55aeea1c39e0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.609+0000 7f285ab9f740 1 --2- 172.21.15.31:0/1460214937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea152350 0x55aeea15c510 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.609+0000 7f285ab9f740 1 --2- 172.21.15.31:0/1460214937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aeea14bf20 0x55aeea150b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.609+0000 7f285ab9f740 1 --2- 172.21.15.31:0/1460214937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aeea113200 0x55aeea150610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.609+0000 7f285ab9f740 1 -- 172.21.15.31:0/1460214937 >> 172.21.15.31:0/1460214937 conn(0x55aeea14d830 msgr2=0x55aeea153640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:58.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.609+0000 7f2836a00640 0 client.26740 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:42:58.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.609+0000 7f285ab9f740 1 -- 172.21.15.31:0/1460214937 shutdown_connections 2024-03-21T10:42:58.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.609+0000 7f285ab9f740 1 -- 172.21.15.31:0/1460214937 wait complete. 2024-03-21T10:42:58.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.609+0000 7f285ab9f740 20 client.26740 trim_cache size 0 max 16384 2024-03-21T10:42:58.608 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:42:58.608 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-21T10:42:58.608 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-03-21T10:42:58.608 INFO:teuthology.orchestra.run.smithi031.stdout: "bb644869-d5b4-4038-b428-1348ff8b6a68": { 2024-03-21T10:42:58.608 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-03-21T10:42:58.608 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "bb644869-d5b4-4038-b428-1348ff8b6a68", 2024-03-21T10:42:58.609 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-03-21T10:42:58.609 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:42:58.609 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:42:58.609 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:42:58.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.610+0000 7f2857e00640 1 -- 172.21.15.31:0/3561726227 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f28380a31e0 msgr2=0x7f28380a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:58.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.611+0000 7f2857e00640 1 --2- 172.21.15.31:0/3561726227 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f28380a31e0 0x7f28380a56a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2844005fd0 tx=0x7f284400f730 comp rx=0 tx=0).stop 2024-03-21T10:42:58.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.611+0000 7f2857e00640 1 -- 172.21.15.31:0/3561726227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f285011ae80 msgr2=0x7f2850116bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:42:58.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.611+0000 7f2857e00640 1 --2- 172.21.15.31:0/3561726227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f285011ae80 0x7f2850116bd0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2848005fd0 tx=0x7f284801d380 comp rx=0 tx=0).stop 2024-03-21T10:42:58.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.611+0000 7f2857e00640 1 -- 172.21.15.31:0/3561726227 shutdown_connections 2024-03-21T10:42:58.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.611+0000 7f2857e00640 1 --2- 172.21.15.31:0/3561726227 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f28380a31e0 0x7f28380a56a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.611+0000 7f2857e00640 1 --2- 172.21.15.31:0/3561726227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f285011ae80 0x7f2850116bd0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.611+0000 7f2857e00640 1 --2- 172.21.15.31:0/3561726227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f285010fe30 0x7f2850116690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.611+0000 7f2857e00640 1 --2- 172.21.15.31:0/3561726227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2850108be0 0x7f285011a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:42:58.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.611+0000 7f2857e00640 1 -- 172.21.15.31:0/3561726227 >> 172.21.15.31:0/3561726227 conn(0x7f285006c710 msgr2=0x7f285010c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:42:58.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.611+0000 7f2857e00640 1 -- 172.21.15.31:0/3561726227 shutdown_connections 2024-03-21T10:42:58.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:42:58.611+0000 7f2857e00640 1 -- 172.21.15.31:0/3561726227 wait complete. 2024-03-21T10:42:58.631 INFO:tasks.cephfs.filesystem:scrub status for tag:bb644869-d5b4-4038-b428-1348ff8b6a68 - {'path': '/', 'tag': 'bb644869-d5b4-4038-b428-1348ff8b6a68', 'options': 'recursive,force'} 2024-03-21T10:42:59.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:59 smithi040 ceph-mon[36154]: pgmap v135: 129 pgs: 129 active+clean; 25 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 218 MiB/s wr, 866 op/s 2024-03-21T10:42:59.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:59 smithi040 ceph-mon[36154]: scrub queued for path: / 2024-03-21T10:42:59.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:59 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:42:59.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:59 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:42:59.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:59 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:42:59.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:59 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:42:59.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:42:59 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:42:59.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:59 smithi031 ceph-mon[31673]: pgmap v135: 129 pgs: 129 active+clean; 25 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 218 MiB/s wr, 866 op/s 2024-03-21T10:42:59.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:59 smithi031 ceph-mon[31673]: scrub queued for path: / 2024-03-21T10:42:59.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:59 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:42:59.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:59 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:42:59.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:59 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:42:59.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:59 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:42:59.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:42:59 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:42:59.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:59 smithi070 ceph-mon[36330]: pgmap v135: 129 pgs: 129 active+clean; 25 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 218 MiB/s wr, 866 op/s 2024-03-21T10:42:59.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:59 smithi070 ceph-mon[36330]: scrub queued for path: / 2024-03-21T10:42:59.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:59 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:42:59.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:59 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:42:59.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:59 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:42:59.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:59 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:42:59.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:42:59 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:43:00.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:00 smithi040 ceph-mon[36154]: pgmap v136: 129 pgs: 129 active+clean; 25 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 260 MiB/s wr, 1.04k op/s 2024-03-21T10:43:00.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:00 smithi031 ceph-mon[31673]: pgmap v136: 129 pgs: 129 active+clean; 25 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 260 MiB/s wr, 1.04k op/s 2024-03-21T10:43:00.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:00 smithi070 ceph-mon[36330]: pgmap v136: 129 pgs: 129 active+clean; 25 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 260 MiB/s wr, 1.04k op/s 2024-03-21T10:43:03.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.052+0000 7fa7c4e00640 1 -- 172.21.15.31:0/3210905465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7c01098a0 msgr2=0x7fa7c0109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:03.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.052+0000 7fa7c4e00640 1 --2- 172.21.15.31:0/3210905465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7c01098a0 0x7fa7c0109c80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8009cd0 tx=0x7fa7b802c0e0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.052+0000 7fa7c4e00640 1 -- 172.21.15.31:0/3210905465 shutdown_connections 2024-03-21T10:43:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.053+0000 7fa7c4e00640 1 --2- 172.21.15.31:0/3210905465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7c0104250 0x7fa7c0110760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.053+0000 7fa7c4e00640 1 --2- 172.21.15.31:0/3210905465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7c0103890 0x7fa7c0103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.053+0000 7fa7c4e00640 1 --2- 172.21.15.31:0/3210905465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7c01098a0 0x7fa7c0109c80 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.053+0000 7fa7c4e00640 1 -- 172.21.15.31:0/3210905465 >> 172.21.15.31:0/3210905465 conn(0x7fa7c00ff320 msgr2=0x7fa7c0101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:03.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.054+0000 7fa7c4e00640 1 -- 172.21.15.31:0/3210905465 shutdown_connections 2024-03-21T10:43:03.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.055+0000 7fa7c4e00640 1 -- 172.21.15.31:0/3210905465 wait complete. 2024-03-21T10:43:03.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:02 smithi031 ceph-mon[31673]: pgmap v137: 129 pgs: 129 active+clean; 25 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 219 MiB/s wr, 877 op/s 2024-03-21T10:43:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.058+0000 7fa7c4e00640 1 Processor -- start 2024-03-21T10:43:03.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.059+0000 7fa7c4e00640 1 -- start start 2024-03-21T10:43:03.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.062+0000 7fa7c4e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7c0103890 0x7fa7c0110610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:03.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.062+0000 7fa7c4e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7c0104250 0x7fa7c010d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:03.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.062+0000 7fa7c4e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7c01098a0 0x7fa7c010dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:03.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.062+0000 7fa7c4e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c0113c70 con 0x7fa7c0103890 2024-03-21T10:43:03.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.062+0000 7fa7c4e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c0113af0 con 0x7fa7c01098a0 2024-03-21T10:43:03.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.062+0000 7fa7c4e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c0113df0 con 0x7fa7c0104250 2024-03-21T10:43:03.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.063+0000 7fa7bde00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7c0103890 0x7fa7c0110610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:03.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.063+0000 7fa7bde00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7c0103890 0x7fa7c0110610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47298/0 (socket says 172.21.15.31:47298) 2024-03-21T10:43:03.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.063+0000 7fa7bde00640 1 -- 172.21.15.31:0/4049050466 learned_addr learned my addr 172.21.15.31:0/4049050466 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:03.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.063+0000 7fa7bde00640 1 -- 172.21.15.31:0/4049050466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7c0104250 msgr2=0x7fa7c010d6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-21T10:43:03.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.063+0000 7fa7bde00640 1 --2- 172.21.15.31:0/4049050466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7c0104250 0x7fa7c010d6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.063+0000 7fa7bde00640 1 -- 172.21.15.31:0/4049050466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7c01098a0 msgr2=0x7fa7c010dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:03.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.065+0000 7fa7be800640 1 --2- 172.21.15.31:0/4049050466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7c01098a0 0x7fa7c010dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:03.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.066+0000 7fa7bde00640 1 --2- 172.21.15.31:0/4049050466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7c01098a0 0x7fa7c010dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.066+0000 7fa7bde00640 1 -- 172.21.15.31:0/4049050466 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c010e4d0 con 0x7fa7c0103890 2024-03-21T10:43:03.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.066+0000 7fa7be800640 1 --2- 172.21.15.31:0/4049050466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7c01098a0 0x7fa7c010dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:43:03.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.066+0000 7fa7bde00640 1 --2- 172.21.15.31:0/4049050466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7c0103890 0x7fa7c0110610 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8000f80 tx=0x7fa7b8035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:03.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.067+0000 7fa7b6a00640 1 -- 172.21.15.31:0/4049050466 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b8039040 con 0x7fa7c0103890 2024-03-21T10:43:03.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.067+0000 7fa7b6a00640 1 -- 172.21.15.31:0/4049050466 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7b8044070 con 0x7fa7c0103890 2024-03-21T10:43:03.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.067+0000 7fa7c4e00640 1 -- 172.21.15.31:0/4049050466 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7c01aaa30 con 0x7fa7c0103890 2024-03-21T10:43:03.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.069+0000 7fa7b6a00640 1 -- 172.21.15.31:0/4049050466 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b8039040 con 0x7fa7c0103890 2024-03-21T10:43:03.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.069+0000 7fa7c4e00640 1 -- 172.21.15.31:0/4049050466 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c01aaf20 con 0x7fa7c0103890 2024-03-21T10:43:03.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.075+0000 7fa7b6a00640 1 -- 172.21.15.31:0/4049050466 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fa7b8032790 con 0x7fa7c0103890 2024-03-21T10:43:03.072 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:02 smithi070 ceph-mon[36330]: pgmap v137: 129 pgs: 129 active+clean; 25 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 219 MiB/s wr, 877 op/s 2024-03-21T10:43:03.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.080+0000 7fa7b6a00640 1 --2- 172.21.15.31:0/4049050466 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa7a40a3300 0x7fa7a40a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:03.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.080+0000 7fa7b6a00640 1 -- 172.21.15.31:0/4049050466 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fa7b80f1d40 con 0x7fa7c0103890 2024-03-21T10:43:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.095+0000 7fa7bd400640 1 --2- 172.21.15.31:0/4049050466 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa7a40a3300 0x7fa7a40a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:03.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.096+0000 7fa7c79df740 1 Processor -- start 2024-03-21T10:43:03.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.096+0000 7fa7c79df740 1 -- start start 2024-03-21T10:43:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.099+0000 7fa7c79df740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558bd2fb2fd0 0x558bd2fb33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.099+0000 7fa7c79df740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bd2fb38f0 0x558bd2fbc180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.099+0000 7fa7c79df740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bd2fbc6c0 0x558bd2fbeab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.099+0000 7fa7c79df740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bd2f753b0 con 0x558bd2fb2fd0 2024-03-21T10:43:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.099+0000 7fa7c79df740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bd2f17830 con 0x558bd2fbc6c0 2024-03-21T10:43:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.099+0000 7fa7c79df740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bd2ec05d0 con 0x558bd2fb38f0 2024-03-21T10:43:03.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.099+0000 7fa7bde00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558bd2fb2fd0 0x558bd2fb33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:03.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.099+0000 7fa7be800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bd2fbc6c0 0x558bd2fbeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:03.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.100+0000 7fa7bde00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558bd2fb2fd0 0x558bd2fb33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47304/0 (socket says 172.21.15.31:47304) 2024-03-21T10:43:03.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.100+0000 7fa7be800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bd2fbc6c0 0x558bd2fbeab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:52320/0 (socket says 172.21.15.31:52320) 2024-03-21T10:43:03.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.100+0000 7fa7bde00640 1 -- 172.21.15.31:0/472221210 learned_addr learned my addr 172.21.15.31:0/472221210 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:03.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.100+0000 7fa7bd400640 1 --2- 172.21.15.31:0/472221210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bd2fb38f0 0x558bd2fbc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:03.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.100+0000 7fa7bde00640 1 -- 172.21.15.31:0/472221210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bd2fb38f0 msgr2=0x558bd2fbc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:03.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.100+0000 7fa7bde00640 1 --2- 172.21.15.31:0/472221210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bd2fb38f0 0x558bd2fbc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.100+0000 7fa7bde00640 1 -- 172.21.15.31:0/472221210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bd2fbc6c0 msgr2=0x558bd2fbeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:03.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.100+0000 7fa7bde00640 1 --2- 172.21.15.31:0/472221210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bd2fbc6c0 0x558bd2fbeab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.100+0000 7fa7bde00640 1 -- 172.21.15.31:0/472221210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558bd2ed82c0 con 0x558bd2fb2fd0 2024-03-21T10:43:03.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.101+0000 7fa7bde00640 1 --2- 172.21.15.31:0/472221210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558bd2fb2fd0 0x558bd2fb33b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8032530 tx=0x7fa7b80bdd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16a70de508668c15 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:03.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.102+0000 7fa7bd400640 1 --2- 172.21.15.31:0/4049050466 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa7a40a3300 0x7fa7a40a57c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac005860 tx=0x7fa7ac0147e0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.103+0000 7fa7ab400640 1 -- 172.21.15.31:0/472221210 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b80be5a0 con 0x558bd2fb2fd0 2024-03-21T10:43:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.105+0000 7fa7ab400640 1 -- 172.21.15.31:0/472221210 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7b80bdea0 con 0x558bd2fb2fd0 2024-03-21T10:43:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.107+0000 7fa7ab400640 1 -- 172.21.15.31:0/472221210 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b80ef370 con 0x558bd2fb2fd0 2024-03-21T10:43:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.108+0000 7fa7c79df740 1 -- 172.21.15.31:0/472221210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558bd2fb2fd0 msgr2=0x558bd2fb33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.108+0000 7fa7c79df740 1 --2- 172.21.15.31:0/472221210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558bd2fb2fd0 0x558bd2fb33b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8032530 tx=0x7fa7b80bdd30 comp rx=0 tx=0).stop 2024-03-21T10:43:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.108+0000 7fa7c79df740 1 -- 172.21.15.31:0/472221210 shutdown_connections 2024-03-21T10:43:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.112+0000 7fa7c79df740 1 --2- 172.21.15.31:0/472221210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bd2fbc6c0 0x558bd2fbeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.112+0000 7fa7c79df740 1 --2- 172.21.15.31:0/472221210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bd2fb38f0 0x558bd2fbc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.112+0000 7fa7c79df740 1 --2- 172.21.15.31:0/472221210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558bd2fb2fd0 0x558bd2fb33b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.112+0000 7fa7c79df740 1 -- 172.21.15.31:0/472221210 >> 172.21.15.31:0/472221210 conn(0x558bd2fad8f0 msgr2=0x558bd2fadcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.112+0000 7fa7c79df740 1 -- 172.21.15.31:0/472221210 shutdown_connections 2024-03-21T10:43:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.112+0000 7fa7c79df740 1 -- 172.21.15.31:0/472221210 wait complete. 2024-03-21T10:43:03.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.113+0000 7fa7c79df740 1 -- start start 2024-03-21T10:43:03.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.114+0000 7fa7c79df740 1 Processor -- start 2024-03-21T10:43:03.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.115+0000 7fa7c79df740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bd2fb2fd0 0x558bd2fb97a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:03.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.115+0000 7fa7c79df740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bd2fb38f0 0x558bd2fb9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:03.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.115+0000 7fa7c79df740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558bd2fbc6c0 0x558bd2fba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:03.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.115+0000 7fa7c79df740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bd2ed3a20 con 0x558bd2fbc6c0 2024-03-21T10:43:03.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.115+0000 7fa7c79df740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bd2f138d0 con 0x558bd2fb38f0 2024-03-21T10:43:03.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.115+0000 7fa7c79df740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bd2fbf0b0 con 0x558bd2fb2fd0 2024-03-21T10:43:03.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.116+0000 7fa7c79df740 10 client.0 _ll_register_callbacks cb 0x558bd2f101c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:43:03.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.118+0000 7fa7be800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558bd2fbc6c0 0x558bd2fba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:03.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.118+0000 7fa7bde00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bd2fb2fd0 0x558bd2fb97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:03.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.118+0000 7fa7bde00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bd2fb2fd0 0x558bd2fb97a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34718/0 (socket says 172.21.15.31:34718) 2024-03-21T10:43:03.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.118+0000 7fa7bd400640 1 --2- 172.21.15.31:0/3712850079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bd2fb38f0 0x558bd2fb9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:03.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.119+0000 7fa7bd400640 1 -- 172.21.15.31:0/3712850079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bd2fb2fd0 msgr2=0x558bd2fb97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:03.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.119+0000 7fa7bd400640 1 --2- 172.21.15.31:0/3712850079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bd2fb2fd0 0x558bd2fb97a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.119+0000 7fa7bd400640 1 -- 172.21.15.31:0/3712850079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558bd2fbc6c0 msgr2=0x558bd2fba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:03.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.119+0000 7fa7bd400640 1 --2- 172.21.15.31:0/3712850079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558bd2fbc6c0 0x558bd2fba220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.119+0000 7fa7bd400640 1 -- 172.21.15.31:0/3712850079 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558bd2c2c210 con 0x558bd2fb38f0 2024-03-21T10:43:03.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.118+0000 7fa7bde00640 1 -- 172.21.15.31:0/3712850079 learned_addr learned my addr 172.21.15.31:0/3712850079 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:03.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.119+0000 7fa7bde00640 1 --2- 172.21.15.31:0/3712850079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bd2fb2fd0 0x558bd2fb97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-21T10:43:03.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.120+0000 7fa7bd400640 1 --2- 172.21.15.31:0/3712850079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bd2fb38f0 0x558bd2fb9ce0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac015fd0 tx=0x7fa7ac0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63127a0597c9c3f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:03.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.120+0000 7fa79fe00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:43:03.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.120+0000 7fa79fe00640 1 -- 172.21.15.31:0/3712850079 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac022020 con 0x558bd2fb38f0 2024-03-21T10:43:03.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.120+0000 7fa79fe00640 1 -- 172.21.15.31:0/3712850079 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fa7ac010850 con 0x558bd2fb38f0 2024-03-21T10:43:03.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.121+0000 7fa7c79df740 1 -- 172.21.15.31:0/3712850079 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa7b80bdea0 con 0x558bd2fb38f0 2024-03-21T10:43:03.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.122+0000 7fa79fe00640 1 -- 172.21.15.31:0/3712850079 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7ac010a50 con 0x558bd2fb38f0 2024-03-21T10:43:03.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.122+0000 7fa7c79df740 10 client.26801 fetch_fsmap learned FSMap version 26 2024-03-21T10:43:03.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.122+0000 7fa7c79df740 1 -- 172.21.15.31:0/3712850079 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x558bd2cf8620 con 0x558bd2fb38f0 2024-03-21T10:43:03.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.122+0000 7fa79fe00640 1 -- 172.21.15.31:0/3712850079 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7b80bdea0 con 0x558bd2fb38f0 2024-03-21T10:43:03.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.123+0000 7fa79fe00640 1 -- 172.21.15.31:0/3712850079 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa7ac01ecd0 con 0x558bd2fb38f0 2024-03-21T10:43:03.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.123+0000 7fa7c79df740 10 client.26801 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:43:03.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.123+0000 7fa7c79df740 10 client.26801 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:43:03.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.123+0000 7fa7c79df740 20 client.26801 populate_metadata read hostname 'smithi031' 2024-03-21T10:43:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.123+0000 7fa7c79df740 1 --2- 172.21.15.31:0/3712850079 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558bd2fdd500 0x558bd2fdf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.123+0000 7fa7c79df740 4 client.26801 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:43:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.123+0000 7fa7c79df740 1 -- 172.21.15.31:0/3712850079 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558bd2f24d70 con 0x558bd2fdd500 2024-03-21T10:43:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.126+0000 7fa7bde00640 1 --2- 172.21.15.31:0/3712850079 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558bd2fdd500 0x558bd2fdf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:03.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.127+0000 7fa7bde00640 1 --2- 172.21.15.31:0/3712850079 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558bd2fdd500 0x558bd2fdf8f0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=811d455e4cf77727 server_cookie=290d5eeb35705e56 in_seq=0 out_seq=0 2024-03-21T10:43:03.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.129+0000 7fa79fe00640 10 client.26801 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:03.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.129+0000 7fa79fe00640 1 -- 172.21.15.31:0/3712850079 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558bd2f24d70 con 0x558bd2fdd500 2024-03-21T10:43:03.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.129+0000 7fa79fe00640 10 client.26801 handle_command_reply: tid=0 2024-03-21T10:43:03.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.129+0000 7fa7c79df740 1 client.26801 shutdown 2024-03-21T10:43:03.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.130+0000 7fa7c79df740 1 -- 172.21.15.31:0/3712850079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bd2fb38f0 msgr2=0x558bd2fb9ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:03.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.131+0000 7fa7c79df740 1 --2- 172.21.15.31:0/3712850079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bd2fb38f0 0x558bd2fb9ce0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac015fd0 tx=0x7fa7ac0046c0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.131+0000 7fa7c79df740 1 -- 172.21.15.31:0/3712850079 shutdown_connections 2024-03-21T10:43:03.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.131+0000 7fa7c79df740 1 --2- 172.21.15.31:0/3712850079 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558bd2fdd500 0x558bd2fdf8f0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.131+0000 7fa7c79df740 1 --2- 172.21.15.31:0/3712850079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558bd2fbc6c0 0x558bd2fba220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.131+0000 7fa7c79df740 1 --2- 172.21.15.31:0/3712850079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bd2fb38f0 0x558bd2fb9ce0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.131+0000 7fa7c79df740 1 --2- 172.21.15.31:0/3712850079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bd2fb2fd0 0x558bd2fb97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.131+0000 7fa7c79df740 1 -- 172.21.15.31:0/3712850079 >> 172.21.15.31:0/3712850079 conn(0x558bd2fad8f0 msgr2=0x558bd2faf640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:03.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.131+0000 7fa79fe00640 0 client.26801 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:03.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.132+0000 7fa7c79df740 1 -- 172.21.15.31:0/3712850079 shutdown_connections 2024-03-21T10:43:03.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.132+0000 7fa7c79df740 1 -- 172.21.15.31:0/3712850079 wait complete. 2024-03-21T10:43:03.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.132+0000 7fa7c79df740 20 client.26801 trim_cache size 0 max 16384 2024-03-21T10:43:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.168+0000 7fa7c79df740 1 Processor -- start 2024-03-21T10:43:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.168+0000 7fa7c79df740 1 -- start start 2024-03-21T10:43:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.169+0000 7fa7c79df740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bd2fb2fd0 0x558bd2fbbed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.169+0000 7fa7c79df740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bd2fb38f0 0x558bd2fb4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.169+0000 7fa7c79df740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558bd2fbc6c0 0x558bd2fba040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.169+0000 7fa7c79df740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c01aed30 con 0x558bd2fbc6c0 2024-03-21T10:43:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.169+0000 7fa7c79df740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c01ac830 con 0x558bd2fb2fd0 2024-03-21T10:43:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.169+0000 7fa7c79df740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c0072f50 con 0x558bd2fb38f0 2024-03-21T10:43:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.169+0000 7fa7bd400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bd2fb38f0 0x558bd2fb4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.169+0000 7fa7bd400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bd2fb38f0 0x558bd2fb4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34722/0 (socket says 172.21.15.31:34722) 2024-03-21T10:43:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.169+0000 7fa7bd400640 1 -- 172.21.15.31:0/4161465980 learned_addr learned my addr 172.21.15.31:0/4161465980 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.170+0000 7fa7bde00640 1 --2- 172.21.15.31:0/4161465980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bd2fb2fd0 0x558bd2fbbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:03.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.170+0000 7fa7bd400640 1 -- 172.21.15.31:0/4161465980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bd2fb2fd0 msgr2=0x558bd2fbbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:03.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.170+0000 7fa7bd400640 1 --2- 172.21.15.31:0/4161465980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bd2fb2fd0 0x558bd2fbbed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.170+0000 7fa7bd400640 1 -- 172.21.15.31:0/4161465980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558bd2fbc6c0 msgr2=0x558bd2fba040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:03.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.170+0000 7fa7bd400640 1 --2- 172.21.15.31:0/4161465980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558bd2fbc6c0 0x558bd2fba040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.170+0000 7fa7bd400640 1 -- 172.21.15.31:0/4161465980 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558bd2fa7920 con 0x558bd2fb38f0 2024-03-21T10:43:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.176+0000 7fa7bd400640 1 --2- 172.21.15.31:0/4161465980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bd2fb38f0 0x558bd2fb4ae0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac0143e0 tx=0x7fa7ac012680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf090cb17a1fe4aa server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.176+0000 7fa7aaa00640 1 -- 172.21.15.31:0/4161465980 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac033070 con 0x558bd2fb38f0 2024-03-21T10:43:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.177+0000 7fa7aaa00640 1 -- 172.21.15.31:0/4161465980 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7ac012e10 con 0x558bd2fb38f0 2024-03-21T10:43:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.177+0000 7fa7aaa00640 1 -- 172.21.15.31:0/4161465980 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac022040 con 0x558bd2fb38f0 2024-03-21T10:43:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.179+0000 7fa7c79df740 1 -- 172.21.15.31:0/4161465980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bd2fb38f0 msgr2=0x558bd2fb4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.179+0000 7fa7c79df740 1 --2- 172.21.15.31:0/4161465980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bd2fb38f0 0x558bd2fb4ae0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac0143e0 tx=0x7fa7ac012680 comp rx=0 tx=0).stop 2024-03-21T10:43:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.179+0000 7fa7c79df740 1 -- 172.21.15.31:0/4161465980 shutdown_connections 2024-03-21T10:43:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.179+0000 7fa7c79df740 1 --2- 172.21.15.31:0/4161465980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558bd2fbc6c0 0x558bd2fba040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.179+0000 7fa7c79df740 1 --2- 172.21.15.31:0/4161465980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bd2fb38f0 0x558bd2fb4ae0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.179+0000 7fa7c79df740 1 --2- 172.21.15.31:0/4161465980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bd2fb2fd0 0x558bd2fbbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.179+0000 7fa7c79df740 1 -- 172.21.15.31:0/4161465980 >> 172.21.15.31:0/4161465980 conn(0x558bd2fad8f0 msgr2=0x558bd2faf5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:03.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.180+0000 7fa7c79df740 1 -- 172.21.15.31:0/4161465980 shutdown_connections 2024-03-21T10:43:03.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.180+0000 7fa7c79df740 1 -- 172.21.15.31:0/4161465980 wait complete. 2024-03-21T10:43:03.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.181+0000 7fa7c79df740 1 -- start start 2024-03-21T10:43:03.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.181+0000 7fa7c79df740 1 Processor -- start 2024-03-21T10:43:03.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.182+0000 7fa7c79df740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558bd2fb2fd0 0x558bd2fb6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:03.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.182+0000 7fa7c79df740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bd2fb38f0 0x558bd2fb6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:03.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.182+0000 7fa7c79df740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bd2fbc6c0 0x558bd2fb6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:03.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.182+0000 7fa7c79df740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c0073550 con 0x558bd2fb2fd0 2024-03-21T10:43:03.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.182+0000 7fa7c79df740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c01ac1d0 con 0x558bd2fbc6c0 2024-03-21T10:43:03.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.182+0000 7fa7c79df740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c01ab510 con 0x558bd2fb38f0 2024-03-21T10:43:03.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.182+0000 7fa7c79df740 10 client.0 _ll_register_callbacks cb 0x7fa7ac0140a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:43:03.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:02 smithi040 ceph-mon[36154]: pgmap v137: 129 pgs: 129 active+clean; 25 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 219 MiB/s wr, 877 op/s 2024-03-21T10:43:03.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.183+0000 7fa7bde00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558bd2fb2fd0 0x558bd2fb6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:03.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.183+0000 7fa7bd400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bd2fb38f0 0x558bd2fb6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:03.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.183+0000 7fa7bd400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bd2fb38f0 0x558bd2fb6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34738/0 (socket says 172.21.15.31:34738) 2024-03-21T10:43:03.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.183+0000 7fa7bd400640 1 -- 172.21.15.31:0/1286661421 learned_addr learned my addr 172.21.15.31:0/1286661421 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:03.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.183+0000 7fa7be800640 1 --2- 172.21.15.31:0/1286661421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bd2fbc6c0 0x558bd2fb6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:03.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.184+0000 7fa7bd400640 1 -- 172.21.15.31:0/1286661421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bd2fbc6c0 msgr2=0x558bd2fb6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:03.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.184+0000 7fa7bd400640 1 --2- 172.21.15.31:0/1286661421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bd2fbc6c0 0x558bd2fb6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.184+0000 7fa7bd400640 1 -- 172.21.15.31:0/1286661421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558bd2fb2fd0 msgr2=0x558bd2fb6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:03.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.184+0000 7fa7bd400640 1 --2- 172.21.15.31:0/1286661421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558bd2fb2fd0 0x558bd2fb6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.184+0000 7fa7bd400640 1 -- 172.21.15.31:0/1286661421 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558bd3038070 con 0x558bd2fb38f0 2024-03-21T10:43:03.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.184+0000 7fa7be800640 1 --2- 172.21.15.31:0/1286661421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bd2fbc6c0 0x558bd2fb6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:43:03.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.185+0000 7fa7bd400640 1 --2- 172.21.15.31:0/1286661421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bd2fb38f0 0x558bd2fb6690 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac015fd0 tx=0x7fa7ac014e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e38168a07bf61b65 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:03.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.185+0000 7fa79fe00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:43:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.186+0000 7fa79fe00640 1 -- 172.21.15.31:0/1286661421 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac033040 con 0x558bd2fb38f0 2024-03-21T10:43:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.187+0000 7fa79fe00640 1 -- 172.21.15.31:0/1286661421 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fa7ac012b40 con 0x558bd2fb38f0 2024-03-21T10:43:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.187+0000 7fa7c79df740 1 -- 172.21.15.31:0/1286661421 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa7ac012e10 con 0x558bd2fb38f0 2024-03-21T10:43:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.187+0000 7fa79fe00640 1 -- 172.21.15.31:0/1286661421 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7ac033040 con 0x558bd2fb38f0 2024-03-21T10:43:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.188+0000 7fa7c79df740 10 client.26752 fetch_fsmap learned FSMap version 26 2024-03-21T10:43:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.188+0000 7fa7c79df740 1 -- 172.21.15.31:0/1286661421 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x558bd3039380 con 0x558bd2fb38f0 2024-03-21T10:43:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.188+0000 7fa79fe00640 1 -- 172.21.15.31:0/1286661421 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7ac012e10 con 0x558bd2fb38f0 2024-03-21T10:43:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.188+0000 7fa79fe00640 1 -- 172.21.15.31:0/1286661421 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa7ac038450 con 0x558bd2fb38f0 2024-03-21T10:43:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.189+0000 7fa7c79df740 10 client.26752 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:43:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.189+0000 7fa7c79df740 10 client.26752 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:43:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.189+0000 7fa7c79df740 20 client.26752 populate_metadata read hostname 'smithi031' 2024-03-21T10:43:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.189+0000 7fa7c79df740 1 --2- 172.21.15.31:0/1286661421 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558bd30399f0 0x558bd303be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.189+0000 7fa7c79df740 4 client.26752 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:43:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.189+0000 7fa7c79df740 1 -- 172.21.15.31:0/1286661421 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558bd2f1f0c0 con 0x558bd30399f0 2024-03-21T10:43:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.190+0000 7fa7bde00640 1 --2- 172.21.15.31:0/1286661421 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558bd30399f0 0x558bd303be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.191+0000 7fa7bde00640 1 --2- 172.21.15.31:0/1286661421 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558bd30399f0 0x558bd303be10 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2818c60e44c01ed server_cookie=1d4f5ca0155adec in_seq=0 out_seq=0 2024-03-21T10:43:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.191+0000 7fa79fe00640 10 client.26752 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:03.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.242+0000 7fa79fe00640 1 -- 172.21.15.31:0/1286661421 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558bd2f1f0c0 con 0x558bd30399f0 2024-03-21T10:43:03.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.242+0000 7fa79fe00640 10 client.26752 handle_command_reply: tid=0 2024-03-21T10:43:03.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.242+0000 7fa7c79df740 1 client.26752 shutdown 2024-03-21T10:43:03.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.243+0000 7fa7c79df740 1 -- 172.21.15.31:0/1286661421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bd2fb38f0 msgr2=0x558bd2fb6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:03.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.243+0000 7fa7c79df740 1 --2- 172.21.15.31:0/1286661421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bd2fb38f0 0x558bd2fb6690 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac015fd0 tx=0x7fa7ac014e70 comp rx=0 tx=0).stop 2024-03-21T10:43:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.244+0000 7fa7c79df740 1 -- 172.21.15.31:0/1286661421 shutdown_connections 2024-03-21T10:43:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.244+0000 7fa7c79df740 1 --2- 172.21.15.31:0/1286661421 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558bd30399f0 0x558bd303be10 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.244+0000 7fa7c79df740 1 --2- 172.21.15.31:0/1286661421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bd2fbc6c0 0x558bd2fb6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.244+0000 7fa7c79df740 1 --2- 172.21.15.31:0/1286661421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558bd2fb38f0 0x558bd2fb6690 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.244+0000 7fa7c79df740 1 --2- 172.21.15.31:0/1286661421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558bd2fb2fd0 0x558bd2fb6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.244+0000 7fa79fe00640 0 client.26752 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.244+0000 7fa7c79df740 1 -- 172.21.15.31:0/1286661421 >> 172.21.15.31:0/1286661421 conn(0x558bd2fad8f0 msgr2=0x558bd2fac0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:03.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.245+0000 7fa7c79df740 1 -- 172.21.15.31:0/1286661421 shutdown_connections 2024-03-21T10:43:03.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.246+0000 7fa7c79df740 1 -- 172.21.15.31:0/1286661421 wait complete. 2024-03-21T10:43:03.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.246+0000 7fa7c79df740 20 client.26752 trim_cache size 0 max 16384 2024-03-21T10:43:03.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:43:03.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:43:03.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:43:03.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:43:03.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.247+0000 7fa7c4e00640 1 -- 172.21.15.31:0/4049050466 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa7a40a3300 msgr2=0x7fa7a40a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:03.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.247+0000 7fa7c4e00640 1 --2- 172.21.15.31:0/4049050466 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa7a40a3300 0x7fa7a40a57c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac005860 tx=0x7fa7ac0147e0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.247+0000 7fa7c4e00640 1 -- 172.21.15.31:0/4049050466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7c0103890 msgr2=0x7fa7c0110610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.247+0000 7fa7c4e00640 1 --2- 172.21.15.31:0/4049050466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7c0103890 0x7fa7c0110610 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8000f80 tx=0x7fa7b8035d10 comp rx=0 tx=0).stop 2024-03-21T10:43:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.248+0000 7fa7c4e00640 1 -- 172.21.15.31:0/4049050466 shutdown_connections 2024-03-21T10:43:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.248+0000 7fa7c4e00640 1 --2- 172.21.15.31:0/4049050466 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa7a40a3300 0x7fa7a40a57c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.248+0000 7fa7c4e00640 1 --2- 172.21.15.31:0/4049050466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7c01098a0 0x7fa7c010dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.248+0000 7fa7c4e00640 1 --2- 172.21.15.31:0/4049050466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7c0104250 0x7fa7c010d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.248+0000 7fa7c4e00640 1 --2- 172.21.15.31:0/4049050466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7c0103890 0x7fa7c0110610 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.248+0000 7fa7c4e00640 1 -- 172.21.15.31:0/4049050466 >> 172.21.15.31:0/4049050466 conn(0x7fa7c00ff320 msgr2=0x7fa7c00ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.249+0000 7fa7c4e00640 1 -- 172.21.15.31:0/4049050466 shutdown_connections 2024-03-21T10:43:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:03.249+0000 7fa7c4e00640 1 -- 172.21.15.31:0/4049050466 wait complete. 2024-03-21T10:43:05.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:04 smithi031 ceph-mon[31673]: pgmap v138: 129 pgs: 129 active+clean; 25 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 190 MiB/s wr, 771 op/s 2024-03-21T10:43:05.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:04 smithi070 ceph-mon[36330]: pgmap v138: 129 pgs: 129 active+clean; 25 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 190 MiB/s wr, 771 op/s 2024-03-21T10:43:05.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:04 smithi040 ceph-mon[36154]: pgmap v138: 129 pgs: 129 active+clean; 25 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 190 MiB/s wr, 771 op/s 2024-03-21T10:43:07.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:06 smithi031 ceph-mon[31673]: pgmap v139: 129 pgs: 129 active+clean; 26 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 258 MiB/s wr, 1.15k op/s 2024-03-21T10:43:07.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:06 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:43:07.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:06 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:43:07.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:06 smithi070 ceph-mon[36330]: pgmap v139: 129 pgs: 129 active+clean; 26 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 258 MiB/s wr, 1.15k op/s 2024-03-21T10:43:07.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:06 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:43:07.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:06 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:43:07.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:06 smithi040 ceph-mon[36154]: pgmap v139: 129 pgs: 129 active+clean; 26 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 258 MiB/s wr, 1.15k op/s 2024-03-21T10:43:07.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:06 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:43:07.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:06 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:43:09.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:08 smithi070 ceph-mon[36330]: pgmap v140: 129 pgs: 129 active+clean; 26 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 211 MiB/s wr, 1.05k op/s 2024-03-21T10:43:09.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:08 smithi040 ceph-mon[36154]: pgmap v140: 129 pgs: 129 active+clean; 26 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 211 MiB/s wr, 1.05k op/s 2024-03-21T10:43:09.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:08 smithi031 ceph-mon[31673]: pgmap v140: 129 pgs: 129 active+clean; 26 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 211 MiB/s wr, 1.05k op/s 2024-03-21T10:43:11.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:10 smithi070 ceph-mon[36330]: pgmap v141: 129 pgs: 129 active+clean; 26 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 250 MiB/s wr, 1.39k op/s 2024-03-21T10:43:11.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:10 smithi040 ceph-mon[36154]: pgmap v141: 129 pgs: 129 active+clean; 26 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 250 MiB/s wr, 1.39k op/s 2024-03-21T10:43:11.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:10 smithi031 ceph-mon[31673]: pgmap v141: 129 pgs: 129 active+clean; 26 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 250 MiB/s wr, 1.39k op/s 2024-03-21T10:43:13.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:12 smithi070 ceph-mon[36330]: pgmap v142: 129 pgs: 129 active+clean; 26 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 213 MiB/s wr, 1.31k op/s 2024-03-21T10:43:13.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:12 smithi040 ceph-mon[36154]: pgmap v142: 129 pgs: 129 active+clean; 26 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 213 MiB/s wr, 1.31k op/s 2024-03-21T10:43:13.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:12 smithi031 ceph-mon[31673]: pgmap v142: 129 pgs: 129 active+clean; 26 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 213 MiB/s wr, 1.31k op/s 2024-03-21T10:43:15.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:14 smithi070 ceph-mon[36330]: pgmap v143: 129 pgs: 129 active+clean; 26 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 182 MiB/s wr, 1.16k op/s 2024-03-21T10:43:15.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:14 smithi040 ceph-mon[36154]: pgmap v143: 129 pgs: 129 active+clean; 26 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 182 MiB/s wr, 1.16k op/s 2024-03-21T10:43:15.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:14 smithi031 ceph-mon[31673]: pgmap v143: 129 pgs: 129 active+clean; 26 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 182 MiB/s wr, 1.16k op/s 2024-03-21T10:43:16.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.559+0000 7f9d66000640 1 -- 172.21.15.31:0/1646962967 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d6010a960 msgr2=0x7f9d601130b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:16.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.559+0000 7f9d66000640 1 --2- 172.21.15.31:0/1646962967 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d6010a960 0x7f9d601130b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f9d50009860 tx=0x7f9d5002bb90 comp rx=0 tx=0).stop 2024-03-21T10:43:16.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.559+0000 7f9d66000640 1 -- 172.21.15.31:0/1646962967 shutdown_connections 2024-03-21T10:43:16.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.559+0000 7f9d66000640 1 --2- 172.21.15.31:0/1646962967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d601135f0 0x7f9d601159e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:16.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.559+0000 7f9d66000640 1 --2- 172.21.15.31:0/1646962967 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d6010a960 0x7f9d601130b0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:16.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.559+0000 7f9d66000640 1 --2- 172.21.15.31:0/1646962967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d6010a040 0x7f9d6010a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:16.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.559+0000 7f9d66000640 1 -- 172.21.15.31:0/1646962967 >> 172.21.15.31:0/1646962967 conn(0x7f9d6006cbe0 msgr2=0x7f9d6006cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:16.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.559+0000 7f9d66000640 1 -- 172.21.15.31:0/1646962967 shutdown_connections 2024-03-21T10:43:16.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.559+0000 7f9d66000640 1 -- 172.21.15.31:0/1646962967 wait complete. 2024-03-21T10:43:16.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.561+0000 7f9d66000640 1 Processor -- start 2024-03-21T10:43:16.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.562+0000 7f9d66000640 1 -- start start 2024-03-21T10:43:16.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.564+0000 7f9d66000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d6010a040 0x7f9d6019e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:16.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.564+0000 7f9d66000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d6010a960 0x7f9d6019ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:16.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.564+0000 7f9d66000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d601135f0 0x7f9d601a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:16.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.564+0000 7f9d5f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d6010a040 0x7f9d6019e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:16.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.564+0000 7f9d5f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d6010a040 0x7f9d6019e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:34018/0 (socket says 172.21.15.31:34018) 2024-03-21T10:43:16.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.564+0000 7f9d5ea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d6010a960 0x7f9d6019ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:16.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.564+0000 7f9d5ea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d6010a960 0x7f9d6019ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:33864/0 (socket says 172.21.15.31:33864) 2024-03-21T10:43:16.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.564+0000 7f9d5f400640 1 -- 172.21.15.31:0/3318498470 learned_addr learned my addr 172.21.15.31:0/3318498470 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:16.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.564+0000 7f9d66000640 1 -- 172.21.15.31:0/3318498470 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d60118040 con 0x7f9d601135f0 2024-03-21T10:43:16.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.564+0000 7f9d66000640 1 -- 172.21.15.31:0/3318498470 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d60117ec0 con 0x7f9d6010a040 2024-03-21T10:43:16.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.564+0000 7f9d66000640 1 -- 172.21.15.31:0/3318498470 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d601181c0 con 0x7f9d6010a960 2024-03-21T10:43:16.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.565+0000 7f9d5fe00640 1 --2- 172.21.15.31:0/3318498470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d601135f0 0x7f9d601a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:16.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.565+0000 7f9d5ea00640 1 -- 172.21.15.31:0/3318498470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d6010a040 msgr2=0x7f9d6019e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:16.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.565+0000 7f9d5ea00640 1 --2- 172.21.15.31:0/3318498470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d6010a040 0x7f9d6019e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:16.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.565+0000 7f9d5ea00640 1 -- 172.21.15.31:0/3318498470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d601135f0 msgr2=0x7f9d601a2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:16.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.565+0000 7f9d5ea00640 1 --2- 172.21.15.31:0/3318498470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d601135f0 0x7f9d601a2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:16.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.565+0000 7f9d5ea00640 1 -- 172.21.15.31:0/3318498470 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d601a34d0 con 0x7f9d6010a960 2024-03-21T10:43:16.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.565+0000 7f9d5fe00640 1 --2- 172.21.15.31:0/3318498470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d601135f0 0x7f9d601a2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:43:16.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.565+0000 7f9d5ea00640 1 --2- 172.21.15.31:0/3318498470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d6010a960 0x7f9d6019ea20 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f9d50037040 tx=0x7f9d500349b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:16.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.566+0000 7f9d57e00640 1 -- 172.21.15.31:0/3318498470 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d50041020 con 0x7f9d6010a960 2024-03-21T10:43:16.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.566+0000 7f9d57e00640 1 -- 172.21.15.31:0/3318498470 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d5002ca10 con 0x7f9d6010a960 2024-03-21T10:43:16.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.566+0000 7f9d57e00640 1 -- 172.21.15.31:0/3318498470 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d50039040 con 0x7f9d6010a960 2024-03-21T10:43:16.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.566+0000 7f9d66000640 1 -- 172.21.15.31:0/3318498470 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d601a3760 con 0x7f9d6010a960 2024-03-21T10:43:16.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.567+0000 7f9d66000640 1 -- 172.21.15.31:0/3318498470 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d601a3bd0 con 0x7f9d6010a960 2024-03-21T10:43:16.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.578+0000 7f9d68df9740 1 Processor -- start 2024-03-21T10:43:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.579+0000 7f9d68df9740 1 -- start start 2024-03-21T10:43:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.579+0000 7f9d68df9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e06cc3fd0 0x563e06cc43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.579+0000 7f9d68df9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 0x563e06ccd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.579+0000 7f9d68df9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e06ccd6c0 0x563e06ccfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.579+0000 7f9d68df9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e06c863b0 con 0x563e06cc3fd0 2024-03-21T10:43:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.579+0000 7f9d68df9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e06c28830 con 0x563e06cc48f0 2024-03-21T10:43:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.579+0000 7f9d68df9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e06bd15d0 con 0x563e06ccd6c0 2024-03-21T10:43:16.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.581+0000 7f9d5f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 0x563e06ccd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:16.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.582+0000 7f9d5f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 0x563e06ccd180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:34026/0 (socket says 172.21.15.31:34026) 2024-03-21T10:43:16.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.582+0000 7f9d5f400640 1 -- 172.21.15.31:0/1241246671 learned_addr learned my addr 172.21.15.31:0/1241246671 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:16.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.582+0000 7f9d5fe00640 1 --2- 172.21.15.31:0/1241246671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e06cc3fd0 0x563e06cc43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:16.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.582+0000 7f9d5f400640 1 -- 172.21.15.31:0/1241246671 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e06ccd6c0 msgr2=0x563e06ccfab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:43:16.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.582+0000 7f9d5f400640 1 --2- 172.21.15.31:0/1241246671 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e06ccd6c0 0x563e06ccfab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:16.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.582+0000 7f9d5f400640 1 -- 172.21.15.31:0/1241246671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e06cc3fd0 msgr2=0x563e06cc43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:16.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.582+0000 7f9d5f400640 1 --2- 172.21.15.31:0/1241246671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e06cc3fd0 0x563e06cc43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:16.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.582+0000 7f9d5f400640 1 -- 172.21.15.31:0/1241246671 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563e06be92c0 con 0x563e06cc48f0 2024-03-21T10:43:16.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.583+0000 7f9d5f400640 1 --2- 172.21.15.31:0/1241246671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 0x563e06ccd180 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9d5800f480 tx=0x7f9d5800f940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3622a5f7e0f29c84 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:16.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.583+0000 7f9d4be00640 1 -- 172.21.15.31:0/1241246671 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5800e800 con 0x563e06cc48f0 2024-03-21T10:43:16.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.584+0000 7f9d4be00640 1 -- 172.21.15.31:0/1241246671 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d5800ee80 con 0x563e06cc48f0 2024-03-21T10:43:16.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.584+0000 7f9d4be00640 1 -- 172.21.15.31:0/1241246671 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5800e800 con 0x563e06cc48f0 2024-03-21T10:43:16.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.585+0000 7f9d68df9740 1 -- 172.21.15.31:0/1241246671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 msgr2=0x563e06ccd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:16.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.585+0000 7f9d68df9740 1 --2- 172.21.15.31:0/1241246671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 0x563e06ccd180 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9d5800f480 tx=0x7f9d5800f940 comp rx=0 tx=0).stop 2024-03-21T10:43:16.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.586+0000 7f9d68df9740 1 -- 172.21.15.31:0/1241246671 shutdown_connections 2024-03-21T10:43:16.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.586+0000 7f9d68df9740 1 --2- 172.21.15.31:0/1241246671 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e06ccd6c0 0x563e06ccfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:16.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.586+0000 7f9d68df9740 1 --2- 172.21.15.31:0/1241246671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 0x563e06ccd180 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:16.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.586+0000 7f9d68df9740 1 --2- 172.21.15.31:0/1241246671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e06cc3fd0 0x563e06cc43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:16.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.586+0000 7f9d68df9740 1 -- 172.21.15.31:0/1241246671 >> 172.21.15.31:0/1241246671 conn(0x563e06cbe8f0 msgr2=0x563e06cbecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:16.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.587+0000 7f9d57e00640 1 -- 172.21.15.31:0/3318498470 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f9d50044020 con 0x7f9d6010a960 2024-03-21T10:43:16.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.592+0000 7f9d68df9740 1 -- 172.21.15.31:0/1241246671 shutdown_connections 2024-03-21T10:43:16.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.592+0000 7f9d68df9740 1 -- 172.21.15.31:0/1241246671 wait complete. 2024-03-21T10:43:16.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.593+0000 7f9d68df9740 1 -- start start 2024-03-21T10:43:16.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.593+0000 7f9d68df9740 1 Processor -- start 2024-03-21T10:43:16.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.596+0000 7f9d57e00640 1 --2- 172.21.15.31:0/3318498470 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9d440a7580 0x7f9d440a9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:16.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.597+0000 7f9d57e00640 1 -- 172.21.15.31:0/3318498470 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f9d500f1180 con 0x7f9d6010a960 2024-03-21T10:43:16.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.598+0000 7f9d68df9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e06cc3fd0 0x563e06cca770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.598+0000 7f9d68df9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 0x563e06ccacb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.598+0000 7f9d68df9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e06ccb1f0 0x563e06ce1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.598+0000 7f9d5ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 0x563e06ccacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.598+0000 7f9d68df9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e06be4a20 con 0x563e06ccb1f0 2024-03-21T10:43:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.598+0000 7f9d5ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 0x563e06ccacb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:34030/0 (socket says 172.21.15.31:34030) 2024-03-21T10:43:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.598+0000 7f9d5ea00640 1 -- 172.21.15.31:0/2546900359 learned_addr learned my addr 172.21.15.31:0/2546900359 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:16.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.598+0000 7f9d68df9740 1 -- 172.21.15.31:0/2546900359 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e06c248d0 con 0x563e06cc48f0 2024-03-21T10:43:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.598+0000 7f9d68df9740 1 -- 172.21.15.31:0/2546900359 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e06cd02f0 con 0x563e06cc3fd0 2024-03-21T10:43:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.598+0000 7f9d68df9740 10 client.0 _ll_register_callbacks cb 0x563e06c211c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:43:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.599+0000 7f9d5f400640 1 --2- 172.21.15.31:0/2546900359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e06cc3fd0 0x563e06cca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.599+0000 7f9d5ea00640 1 -- 172.21.15.31:0/2546900359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e06cc3fd0 msgr2=0x563e06cca770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.599+0000 7f9d5ea00640 1 --2- 172.21.15.31:0/2546900359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e06cc3fd0 0x563e06cca770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.599+0000 7f9d5ea00640 1 -- 172.21.15.31:0/2546900359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e06ccb1f0 msgr2=0x563e06ce1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.599+0000 7f9d5ea00640 1 --2- 172.21.15.31:0/2546900359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e06ccb1f0 0x563e06ce1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.600+0000 7f9d5ea00640 1 -- 172.21.15.31:0/2546900359 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563e0693d210 con 0x563e06cc48f0 2024-03-21T10:43:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.600+0000 7f9d5fe00640 1 --2- 172.21.15.31:0/3318498470 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9d440a7580 0x7f9d440a9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.600+0000 7f9d5ea00640 1 --2- 172.21.15.31:0/2546900359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 0x563e06ccacb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f9d5002c1a0 tx=0x7f9d500bcc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a9f4a49350f72b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.601+0000 7f9d48c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:43:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.601+0000 7f9d48c00640 1 -- 172.21.15.31:0/2546900359 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d500c1430 con 0x563e06cc48f0 2024-03-21T10:43:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.601+0000 7f9d48c00640 1 -- 172.21.15.31:0/2546900359 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f9d5002cbc0 con 0x563e06cc48f0 2024-03-21T10:43:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.601+0000 7f9d68df9740 1 -- 172.21.15.31:0/2546900359 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563e06c35d70 con 0x563e06cc48f0 2024-03-21T10:43:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.606+0000 7f9d5fe00640 1 --2- 172.21.15.31:0/3318498470 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9d440a7580 0x7f9d440a9a40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c005d00 tx=0x7f9d4c00f040 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:16.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.615+0000 7f9d48c00640 1 -- 172.21.15.31:0/2546900359 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d500db050 con 0x563e06cc48f0 2024-03-21T10:43:16.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.615+0000 7f9d68df9740 10 client.26819 fetch_fsmap learned FSMap version 26 2024-03-21T10:43:16.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.615+0000 7f9d68df9740 1 -- 172.21.15.31:0/2546900359 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x563e06a09620 con 0x563e06cc48f0 2024-03-21T10:43:16.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.616+0000 7f9d48c00640 1 -- 172.21.15.31:0/2546900359 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563e06c35d70 con 0x563e06cc48f0 2024-03-21T10:43:16.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.616+0000 7f9d48c00640 1 -- 172.21.15.31:0/2546900359 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f9d5003c030 con 0x563e06cc48f0 2024-03-21T10:43:16.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.616+0000 7f9d68df9740 10 client.26819 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:43:16.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.616+0000 7f9d68df9740 10 client.26819 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:43:16.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.616+0000 7f9d68df9740 20 client.26819 populate_metadata read hostname 'smithi031' 2024-03-21T10:43:16.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.616+0000 7f9d68df9740 1 --2- 172.21.15.31:0/2546900359 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563e06ce6300 0x563e06ce8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:16.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.616+0000 7f9d68df9740 4 client.26819 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:43:16.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.616+0000 7f9d68df9740 1 -- 172.21.15.31:0/2546900359 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563e06c300c0 con 0x563e06ce6300 2024-03-21T10:43:16.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.619+0000 7f9d5f400640 1 --2- 172.21.15.31:0/2546900359 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563e06ce6300 0x563e06ce8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.622+0000 7f9d48c00640 10 client.26819 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.622+0000 7f9d5f400640 1 --2- 172.21.15.31:0/2546900359 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563e06ce6300 0x563e06ce8720 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98c8a72ad78dbbb1 server_cookie=47cc3fecc5c1d01b in_seq=0 out_seq=0 2024-03-21T10:43:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.624+0000 7f9d48c00640 1 -- 172.21.15.31:0/2546900359 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563e06c300c0 con 0x563e06ce6300 2024-03-21T10:43:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.624+0000 7f9d48c00640 10 client.26819 handle_command_reply: tid=0 2024-03-21T10:43:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.624+0000 7f9d68df9740 1 client.26819 shutdown 2024-03-21T10:43:16.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.625+0000 7f9d68df9740 1 -- 172.21.15.31:0/2546900359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 msgr2=0x563e06ccacb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:16.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.625+0000 7f9d68df9740 1 --2- 172.21.15.31:0/2546900359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 0x563e06ccacb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f9d5002c1a0 tx=0x7f9d500bcc40 comp rx=0 tx=0).stop 2024-03-21T10:43:16.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.625+0000 7f9d68df9740 1 -- 172.21.15.31:0/2546900359 shutdown_connections 2024-03-21T10:43:16.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.625+0000 7f9d68df9740 1 --2- 172.21.15.31:0/2546900359 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563e06ce6300 0x563e06ce8720 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:16.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.625+0000 7f9d68df9740 1 --2- 172.21.15.31:0/2546900359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e06ccb1f0 0x563e06ce1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:16.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.625+0000 7f9d68df9740 1 --2- 172.21.15.31:0/2546900359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 0x563e06ccacb0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:16.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.625+0000 7f9d68df9740 1 --2- 172.21.15.31:0/2546900359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e06cc3fd0 0x563e06cca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:16.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.625+0000 7f9d68df9740 1 -- 172.21.15.31:0/2546900359 >> 172.21.15.31:0/2546900359 conn(0x563e06cbe8f0 msgr2=0x563e06cc0640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:16.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.625+0000 7f9d48c00640 0 client.26819 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:16.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.625+0000 7f9d68df9740 1 -- 172.21.15.31:0/2546900359 shutdown_connections 2024-03-21T10:43:16.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.626+0000 7f9d68df9740 1 -- 172.21.15.31:0/2546900359 wait complete. 2024-03-21T10:43:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.627+0000 7f9d68df9740 20 client.26819 trim_cache size 0 max 16384 2024-03-21T10:43:16.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.676+0000 7f9d68df9740 1 Processor -- start 2024-03-21T10:43:16.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.677+0000 7f9d68df9740 1 -- start start 2024-03-21T10:43:16.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.677+0000 7f9d68df9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e06cc3fd0 0x563e06cb95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:16.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.677+0000 7f9d68df9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 0x563e06cc79f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:16.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.677+0000 7f9d68df9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e06ccb1f0 0x563e06cc7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:16.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.677+0000 7f9d68df9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d601b7450 con 0x563e06ccb1f0 2024-03-21T10:43:16.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.677+0000 7f9d68df9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d601b4f50 con 0x563e06cc48f0 2024-03-21T10:43:16.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.677+0000 7f9d68df9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d601b2ff0 con 0x563e06cc3fd0 2024-03-21T10:43:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.678+0000 7f9d5ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 0x563e06cc79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.678+0000 7f9d5fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e06ccb1f0 0x563e06cc7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.678+0000 7f9d5ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 0x563e06cc79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:34046/0 (socket says 172.21.15.31:34046) 2024-03-21T10:43:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.678+0000 7f9d5ea00640 1 -- 172.21.15.31:0/105618087 learned_addr learned my addr 172.21.15.31:0/105618087 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.678+0000 7f9d5f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e06cc3fd0 0x563e06cb95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.681+0000 7f9d5ea00640 1 -- 172.21.15.31:0/105618087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e06cc3fd0 msgr2=0x563e06cb95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.681+0000 7f9d5ea00640 1 --2- 172.21.15.31:0/105618087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e06cc3fd0 0x563e06cb95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.681+0000 7f9d5ea00640 1 -- 172.21.15.31:0/105618087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e06ccb1f0 msgr2=0x563e06cc7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.681+0000 7f9d5ea00640 1 --2- 172.21.15.31:0/105618087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e06ccb1f0 0x563e06cc7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.681+0000 7f9d5ea00640 1 -- 172.21.15.31:0/105618087 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563e06ce60b0 con 0x563e06cc48f0 2024-03-21T10:43:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.681+0000 7f9d5f400640 1 --2- 172.21.15.31:0/105618087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e06cc3fd0 0x563e06cb95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:43:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.681+0000 7f9d5ea00640 1 --2- 172.21.15.31:0/105618087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 0x563e06cc79f0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9d500b9cb0 tx=0x7f9d500bcd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d455305cf876798 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.682+0000 7f9d4b400640 1 -- 172.21.15.31:0/105618087 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d500c1430 con 0x563e06cc48f0 2024-03-21T10:43:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.682+0000 7f9d4b400640 1 -- 172.21.15.31:0/105618087 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d500b7e50 con 0x563e06cc48f0 2024-03-21T10:43:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.682+0000 7f9d4b400640 1 -- 172.21.15.31:0/105618087 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d500c10d0 con 0x563e06cc48f0 2024-03-21T10:43:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.684+0000 7f9d5fe00640 1 --2- 172.21.15.31:0/105618087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e06ccb1f0 0x563e06cc7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:43:16.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.684+0000 7f9d68df9740 1 -- 172.21.15.31:0/105618087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 msgr2=0x563e06cc79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:16.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.684+0000 7f9d68df9740 1 --2- 172.21.15.31:0/105618087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 0x563e06cc79f0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9d500b9cb0 tx=0x7f9d500bcd70 comp rx=0 tx=0).stop 2024-03-21T10:43:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.686+0000 7f9d68df9740 1 -- 172.21.15.31:0/105618087 shutdown_connections 2024-03-21T10:43:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.686+0000 7f9d68df9740 1 --2- 172.21.15.31:0/105618087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e06ccb1f0 0x563e06cc7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.686+0000 7f9d68df9740 1 --2- 172.21.15.31:0/105618087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 0x563e06cc79f0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.686+0000 7f9d68df9740 1 --2- 172.21.15.31:0/105618087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e06cc3fd0 0x563e06cb95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.686+0000 7f9d68df9740 1 -- 172.21.15.31:0/105618087 >> 172.21.15.31:0/105618087 conn(0x563e06cbe8f0 msgr2=0x563e06cc9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.686+0000 7f9d68df9740 1 -- 172.21.15.31:0/105618087 shutdown_connections 2024-03-21T10:43:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.686+0000 7f9d68df9740 1 -- 172.21.15.31:0/105618087 wait complete. 2024-03-21T10:43:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.687+0000 7f9d68df9740 1 -- start start 2024-03-21T10:43:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.687+0000 7f9d68df9740 1 Processor -- start 2024-03-21T10:43:16.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.688+0000 7f9d68df9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e06cc3fd0 0x563e06cdb480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:16.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.688+0000 7f9d68df9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 0x563e06cdb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:16.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.689+0000 7f9d68df9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e06ccb1f0 0x563e06cdbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:16.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.689+0000 7f9d68df9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d601b35f0 con 0x563e06cc3fd0 2024-03-21T10:43:16.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.689+0000 7f9d5ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 0x563e06cdb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:16.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.689+0000 7f9d68df9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d601b48f0 con 0x563e06cc48f0 2024-03-21T10:43:16.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.689+0000 7f9d68df9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d601b3c30 con 0x563e06ccb1f0 2024-03-21T10:43:16.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.689+0000 7f9d68df9740 10 client.0 _ll_register_callbacks cb 0x7f9d500bd0b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:43:16.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.689+0000 7f9d5f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e06cc3fd0 0x563e06cdb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.689+0000 7f9d5ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 0x563e06cdb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:34060/0 (socket says 172.21.15.31:34060) 2024-03-21T10:43:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.689+0000 7f9d5ea00640 1 -- 172.21.15.31:0/3987469442 learned_addr learned my addr 172.21.15.31:0/3987469442 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.689+0000 7f9d5fe00640 1 --2- 172.21.15.31:0/3987469442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e06ccb1f0 0x563e06cdbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.691+0000 7f9d5fe00640 1 -- 172.21.15.31:0/3987469442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 msgr2=0x563e06cdb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.691+0000 7f9d5fe00640 1 --2- 172.21.15.31:0/3987469442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 0x563e06cdb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.691+0000 7f9d5fe00640 1 -- 172.21.15.31:0/3987469442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e06cc3fd0 msgr2=0x563e06cdb480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.692+0000 7f9d5fe00640 1 --2- 172.21.15.31:0/3987469442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e06cc3fd0 0x563e06cdb480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:16.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.692+0000 7f9d5fe00640 1 -- 172.21.15.31:0/3987469442 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563e06d40dc0 con 0x563e06ccb1f0 2024-03-21T10:43:16.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.692+0000 7f9d5ea00640 1 --2- 172.21.15.31:0/3987469442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 0x563e06cdb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:43:16.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.693+0000 7f9d48c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:43:16.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.693+0000 7f9d5fe00640 1 --2- 172.21.15.31:0/3987469442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e06ccb1f0 0x563e06cdbf00 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c004400 tx=0x7f9d4c005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9014a26eb517dec6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:16.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.694+0000 7f9d48c00640 1 -- 172.21.15.31:0/3987469442 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c017780 con 0x563e06ccb1f0 2024-03-21T10:43:16.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.694+0000 7f9d48c00640 1 -- 172.21.15.31:0/3987469442 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f9d4c017580 con 0x563e06ccb1f0 2024-03-21T10:43:16.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.694+0000 7f9d68df9740 1 -- 172.21.15.31:0/3987469442 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9d500b7e50 con 0x563e06ccb1f0 2024-03-21T10:43:16.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.694+0000 7f9d48c00640 1 -- 172.21.15.31:0/3987469442 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d4c017780 con 0x563e06ccb1f0 2024-03-21T10:43:16.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.694+0000 7f9d68df9740 10 client.26767 fetch_fsmap learned FSMap version 26 2024-03-21T10:43:16.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.694+0000 7f9d68df9740 1 -- 172.21.15.31:0/3987469442 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x563e06d420f0 con 0x563e06ccb1f0 2024-03-21T10:43:16.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.695+0000 7f9d48c00640 1 -- 172.21.15.31:0/3987469442 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d4c029070 con 0x563e06ccb1f0 2024-03-21T10:43:16.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.697+0000 7f9d48c00640 1 -- 172.21.15.31:0/3987469442 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f9d4c02eb60 con 0x563e06ccb1f0 2024-03-21T10:43:16.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.697+0000 7f9d68df9740 10 client.26767 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:43:16.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.697+0000 7f9d68df9740 10 client.26767 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:43:16.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.697+0000 7f9d68df9740 20 client.26767 populate_metadata read hostname 'smithi031' 2024-03-21T10:43:16.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.697+0000 7f9d68df9740 1 --2- 172.21.15.31:0/3987469442 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563e06d42810 0x563e06d44c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.697+0000 7f9d68df9740 4 client.26767 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:43:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.697+0000 7f9d68df9740 1 -- 172.21.15.31:0/3987469442 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x7f9d500b7e50 con 0x563e06d42810 2024-03-21T10:43:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.699+0000 7f9d5f400640 1 --2- 172.21.15.31:0/3987469442 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563e06d42810 0x563e06d44c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.701+0000 7f9d5f400640 1 --2- 172.21.15.31:0/3987469442 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563e06d42810 0x563e06d44c00 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38182efc97098753 server_cookie=30d426142ccff32a in_seq=0 out_seq=0 2024-03-21T10:43:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:16.701+0000 7f9d48c00640 10 client.26767 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:17.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.163+0000 7f9d48c00640 1 -- 172.21.15.31:0/3987469442 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x7f9d500b7e50 con 0x563e06d42810 2024-03-21T10:43:17.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.163+0000 7f9d48c00640 10 client.26767 handle_command_reply: tid=0 2024-03-21T10:43:17.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.163+0000 7f9d68df9740 1 client.26767 shutdown 2024-03-21T10:43:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.170+0000 7f9d68df9740 1 -- 172.21.15.31:0/3987469442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e06ccb1f0 msgr2=0x563e06cdbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.171+0000 7f9d68df9740 1 --2- 172.21.15.31:0/3987469442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e06ccb1f0 0x563e06cdbf00 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c004400 tx=0x7f9d4c005f90 comp rx=0 tx=0).stop 2024-03-21T10:43:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.171+0000 7f9d68df9740 1 -- 172.21.15.31:0/3987469442 shutdown_connections 2024-03-21T10:43:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.171+0000 7f9d68df9740 1 --2- 172.21.15.31:0/3987469442 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563e06d42810 0x563e06d44c00 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.171+0000 7f9d48c00640 0 client.26767 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.171+0000 7f9d68df9740 1 --2- 172.21.15.31:0/3987469442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e06ccb1f0 0x563e06cdbf00 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.172+0000 7f9d68df9740 1 --2- 172.21.15.31:0/3987469442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563e06cc48f0 0x563e06cdb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.172+0000 7f9d68df9740 1 --2- 172.21.15.31:0/3987469442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e06cc3fd0 0x563e06cdb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.172+0000 7f9d68df9740 1 -- 172.21.15.31:0/3987469442 >> 172.21.15.31:0/3987469442 conn(0x563e06cbe8f0 msgr2=0x563e06cbd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.173+0000 7f9d68df9740 1 -- 172.21.15.31:0/3987469442 shutdown_connections 2024-03-21T10:43:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.173+0000 7f9d68df9740 1 -- 172.21.15.31:0/3987469442 wait complete. 2024-03-21T10:43:17.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.173+0000 7f9d68df9740 20 client.26767 trim_cache size 0 max 16384 2024-03-21T10:43:17.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:43:17.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:43:17.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:43:17.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:43:17.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.176+0000 7f9d66000640 1 -- 172.21.15.31:0/3318498470 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9d440a7580 msgr2=0x7f9d440a9a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:17.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.176+0000 7f9d66000640 1 --2- 172.21.15.31:0/3318498470 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9d440a7580 0x7f9d440a9a40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c005d00 tx=0x7f9d4c00f040 comp rx=0 tx=0).stop 2024-03-21T10:43:17.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.176+0000 7f9d66000640 1 -- 172.21.15.31:0/3318498470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d6010a960 msgr2=0x7f9d6019ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:17.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.176+0000 7f9d66000640 1 --2- 172.21.15.31:0/3318498470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d6010a960 0x7f9d6019ea20 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f9d50037040 tx=0x7f9d500349b0 comp rx=0 tx=0).stop 2024-03-21T10:43:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.178+0000 7f9d66000640 1 -- 172.21.15.31:0/3318498470 shutdown_connections 2024-03-21T10:43:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.178+0000 7f9d66000640 1 --2- 172.21.15.31:0/3318498470 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9d440a7580 0x7f9d440a9a40 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.178+0000 7f9d66000640 1 --2- 172.21.15.31:0/3318498470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d601135f0 0x7f9d601a2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.178+0000 7f9d66000640 1 --2- 172.21.15.31:0/3318498470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d6010a960 0x7f9d6019ea20 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.178+0000 7f9d66000640 1 --2- 172.21.15.31:0/3318498470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d6010a040 0x7f9d6019e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.178+0000 7f9d66000640 1 -- 172.21.15.31:0/3318498470 >> 172.21.15.31:0/3318498470 conn(0x7f9d6006cbe0 msgr2=0x7f9d60070360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.178+0000 7f9d66000640 1 -- 172.21.15.31:0/3318498470 shutdown_connections 2024-03-21T10:43:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:17.178+0000 7f9d66000640 1 -- 172.21.15.31:0/3318498470 wait complete. 2024-03-21T10:43:17.182 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:16 smithi040 ceph-mon[36154]: pgmap v144: 129 pgs: 129 active+clean; 27 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 260 MiB/s wr, 1.81k op/s 2024-03-21T10:43:17.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:16 smithi031 ceph-mon[31673]: pgmap v144: 129 pgs: 129 active+clean; 27 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 260 MiB/s wr, 1.81k op/s 2024-03-21T10:43:17.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:16 smithi070 ceph-mon[36330]: pgmap v144: 129 pgs: 129 active+clean; 27 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 260 MiB/s wr, 1.81k op/s 2024-03-21T10:43:19.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:18 smithi040 ceph-mon[36154]: pgmap v145: 129 pgs: 129 active+clean; 27 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 213 MiB/s wr, 1.64k op/s 2024-03-21T10:43:19.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:18 smithi031 ceph-mon[31673]: pgmap v145: 129 pgs: 129 active+clean; 27 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 213 MiB/s wr, 1.64k op/s 2024-03-21T10:43:19.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:18 smithi070 ceph-mon[36330]: pgmap v145: 129 pgs: 129 active+clean; 27 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 213 MiB/s wr, 1.64k op/s 2024-03-21T10:43:21.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:20 smithi040 ceph-mon[36154]: pgmap v146: 129 pgs: 129 active+clean; 27 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 243 MiB/s wr, 2.04k op/s 2024-03-21T10:43:21.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:20 smithi031 ceph-mon[31673]: pgmap v146: 129 pgs: 129 active+clean; 27 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 243 MiB/s wr, 2.04k op/s 2024-03-21T10:43:21.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:20 smithi070 ceph-mon[36330]: pgmap v146: 129 pgs: 129 active+clean; 27 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 243 MiB/s wr, 2.04k op/s 2024-03-21T10:43:23.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:22 smithi040 ceph-mon[36154]: pgmap v147: 129 pgs: 129 active+clean; 27 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 187 MiB/s wr, 1.69k op/s 2024-03-21T10:43:23.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:22 smithi031 ceph-mon[31673]: pgmap v147: 129 pgs: 129 active+clean; 27 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 187 MiB/s wr, 1.69k op/s 2024-03-21T10:43:23.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:22 smithi070 ceph-mon[36330]: pgmap v147: 129 pgs: 129 active+clean; 27 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 187 MiB/s wr, 1.69k op/s 2024-03-21T10:43:25.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:24 smithi040 ceph-mon[36154]: pgmap v148: 129 pgs: 129 active+clean; 27 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 170 MiB/s wr, 1.59k op/s 2024-03-21T10:43:25.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:24 smithi031 ceph-mon[31673]: pgmap v148: 129 pgs: 129 active+clean; 27 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 170 MiB/s wr, 1.59k op/s 2024-03-21T10:43:25.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:24 smithi070 ceph-mon[36330]: pgmap v148: 129 pgs: 129 active+clean; 27 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 170 MiB/s wr, 1.59k op/s 2024-03-21T10:43:27.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:26 smithi040 ceph-mon[36154]: pgmap v149: 129 pgs: 129 active+clean; 27 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 252 MiB/s wr, 2.20k op/s 2024-03-21T10:43:27.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:26 smithi031 ceph-mon[31673]: pgmap v149: 129 pgs: 129 active+clean; 27 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 252 MiB/s wr, 2.20k op/s 2024-03-21T10:43:27.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:26 smithi070 ceph-mon[36330]: pgmap v149: 129 pgs: 129 active+clean; 27 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 252 MiB/s wr, 2.20k op/s 2024-03-21T10:43:28.632 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:43:28.633 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:43:29.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.018+0000 7f6246600640 1 -- 172.21.15.31:0/1189604031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f624010ba50 msgr2=0x7f624010ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:29.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.018+0000 7f6246600640 1 --2- 172.21.15.31:0/1189604031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f624010ba50 0x7f624010ded0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f622c009860 tx=0x7f622c02bc30 comp rx=0 tx=0).stop 2024-03-21T10:43:29.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.019+0000 7f6246600640 1 -- 172.21.15.31:0/1189604031 shutdown_connections 2024-03-21T10:43:29.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.019+0000 7f6246600640 1 --2- 172.21.15.31:0/1189604031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f624010ba50 0x7f624010ded0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.019+0000 7f6246600640 1 --2- 172.21.15.31:0/1189604031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6240102f50 0x7f624010b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.019+0000 7f6246600640 1 --2- 172.21.15.31:0/1189604031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6240102630 0x7f6240102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.019+0000 7f6246600640 1 -- 172.21.15.31:0/1189604031 >> 172.21.15.31:0/1189604031 conn(0x7f6240077130 msgr2=0x7f62400fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:29.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.022+0000 7f6246600640 1 -- 172.21.15.31:0/1189604031 shutdown_connections 2024-03-21T10:43:29.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.023+0000 7f6246600640 1 -- 172.21.15.31:0/1189604031 wait complete. 2024-03-21T10:43:29.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.024+0000 7f6246600640 1 Processor -- start 2024-03-21T10:43:29.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.024+0000 7f6246600640 1 -- start start 2024-03-21T10:43:29.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.025+0000 7f6246600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6240102630 0x7f624010a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:29.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.025+0000 7f6246600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6240102f50 0x7f624010ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:29.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.025+0000 7f6246600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f624010ba50 0x7f62401a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:29.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.025+0000 7f6246600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6240111810 con 0x7f6240102630 2024-03-21T10:43:29.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.025+0000 7f6246600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6240111690 con 0x7f6240102f50 2024-03-21T10:43:29.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.025+0000 7f6246600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6240111990 con 0x7f624010ba50 2024-03-21T10:43:29.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.027+0000 7f623fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6240102f50 0x7f624010ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:29.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.027+0000 7f623fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6240102f50 0x7f624010ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58044/0 (socket says 172.21.15.31:58044) 2024-03-21T10:43:29.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.028+0000 7f623fe00640 1 -- 172.21.15.31:0/4042805778 learned_addr learned my addr 172.21.15.31:0/4042805778 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:29.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.028+0000 7f6245c00640 1 --2- 172.21.15.31:0/4042805778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f624010ba50 0x7f62401a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.028+0000 7f6245c00640 1 -- 172.21.15.31:0/4042805778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6240102f50 msgr2=0x7f624010ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.028+0000 7f6245c00640 1 --2- 172.21.15.31:0/4042805778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6240102f50 0x7f624010ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.028+0000 7f6245c00640 1 -- 172.21.15.31:0/4042805778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6240102630 msgr2=0x7f624010a7a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T10:43:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.029+0000 7f6245200640 1 --2- 172.21.15.31:0/4042805778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6240102630 0x7f624010a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:29.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.029+0000 7f6245c00640 1 --2- 172.21.15.31:0/4042805778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6240102630 0x7f624010a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.029+0000 7f6245c00640 1 -- 172.21.15.31:0/4042805778 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62401a32c0 con 0x7f624010ba50 2024-03-21T10:43:29.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.031+0000 7f6245c00640 1 --2- 172.21.15.31:0/4042805778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f624010ba50 0x7f62401a2b90 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f622c02c170 tx=0x7f622c034440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:29.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.036+0000 7f623d600640 1 -- 172.21.15.31:0/4042805778 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622c039040 con 0x7f624010ba50 2024-03-21T10:43:29.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.036+0000 7f623d600640 1 -- 172.21.15.31:0/4042805778 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f622c042070 con 0x7f624010ba50 2024-03-21T10:43:29.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.036+0000 7f623d600640 1 -- 172.21.15.31:0/4042805778 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622c02ce60 con 0x7f624010ba50 2024-03-21T10:43:29.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.041+0000 7f6246600640 1 -- 172.21.15.31:0/4042805778 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62401a3550 con 0x7f624010ba50 2024-03-21T10:43:29.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.043+0000 7f6246600640 1 -- 172.21.15.31:0/4042805778 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62401a39c0 con 0x7f624010ba50 2024-03-21T10:43:29.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.048+0000 7f624a849740 1 Processor -- start 2024-03-21T10:43:29.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.050+0000 7f624a849740 1 -- start start 2024-03-21T10:43:29.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.051+0000 7f624a849740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad19f42dd0 0x55ad19f41b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:29.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.051+0000 7f624a849740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad19f420f0 0x55ad19f42570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:29.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.051+0000 7f624a849740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad19f4b460 0x55ad19f4d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:29.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.051+0000 7f624a849740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad19ea6250 con 0x55ad19f42dd0 2024-03-21T10:43:29.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.051+0000 7f624a849740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad19e4ea30 con 0x55ad19f420f0 2024-03-21T10:43:29.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.051+0000 7f623d600640 1 -- 172.21.15.31:0/4042805778 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f622c02c5d0 con 0x7f624010ba50 2024-03-21T10:43:29.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.051+0000 7f623fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad19f42dd0 0x55ad19f41b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:29.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.052+0000 7f623fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad19f42dd0 0x55ad19f41b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48906/0 (socket says 172.21.15.31:48906) 2024-03-21T10:43:29.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.052+0000 7f623fe00640 1 -- 172.21.15.31:0/2168986751 learned_addr learned my addr 172.21.15.31:0/2168986751 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:29.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.052+0000 7f6245200640 1 --2- 172.21.15.31:0/2168986751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad19f420f0 0x55ad19f42570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:29.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.053+0000 7f624a849740 1 -- 172.21.15.31:0/2168986751 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad19ea22f0 con 0x55ad19f4b460 2024-03-21T10:43:29.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.053+0000 7f623fe00640 1 --2- 172.21.15.31:0/2168986751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad19f4b460 0x55ad19f4d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:29.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.053+0000 7f623fe00640 1 -- 172.21.15.31:0/2168986751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad19f420f0 msgr2=0x55ad19f42570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:29.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.053+0000 7f623fe00640 1 --2- 172.21.15.31:0/2168986751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad19f420f0 0x55ad19f42570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.053+0000 7f623fe00640 1 -- 172.21.15.31:0/2168986751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad19f42dd0 msgr2=0x55ad19f41b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:29.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.053+0000 7f623fe00640 1 --2- 172.21.15.31:0/2168986751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad19f42dd0 0x55ad19f41b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.053+0000 7f623fe00640 1 -- 172.21.15.31:0/2168986751 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad19e39690 con 0x55ad19f4b460 2024-03-21T10:43:29.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.053+0000 7f6245200640 1 --2- 172.21.15.31:0/2168986751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad19f420f0 0x55ad19f42570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:43:29.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.055+0000 7f623d600640 1 --2- 172.21.15.31:0/4042805778 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f62240a3260 0x7f62240a5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:29.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.055+0000 7f623d600640 1 -- 172.21.15.31:0/4042805778 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f622c0f0e70 con 0x7f624010ba50 2024-03-21T10:43:29.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.055+0000 7f623fe00640 1 --2- 172.21.15.31:0/2168986751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad19f4b460 0x55ad19f4d850 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f623000afc0 tx=0x7f623000a380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=610bc32b924758dd server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:29.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.055+0000 7f6236000640 1 -- 172.21.15.31:0/2168986751 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6230005220 con 0x55ad19f4b460 2024-03-21T10:43:29.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.055+0000 7f6236000640 1 -- 172.21.15.31:0/2168986751 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6230004c80 con 0x55ad19f4b460 2024-03-21T10:43:29.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.056+0000 7f6236000640 1 -- 172.21.15.31:0/2168986751 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623000fc90 con 0x55ad19f4b460 2024-03-21T10:43:29.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:28 smithi031 ceph-mon[31673]: pgmap v150: 129 pgs: 129 active+clean; 28 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 190 MiB/s wr, 1.64k op/s 2024-03-21T10:43:29.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.056+0000 7f624a849740 1 -- 172.21.15.31:0/2168986751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad19f4b460 msgr2=0x55ad19f4d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:29.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.057+0000 7f624a849740 1 --2- 172.21.15.31:0/2168986751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad19f4b460 0x55ad19f4d850 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f623000afc0 tx=0x7f623000a380 comp rx=0 tx=0).stop 2024-03-21T10:43:29.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.057+0000 7f624a849740 1 -- 172.21.15.31:0/2168986751 shutdown_connections 2024-03-21T10:43:29.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.057+0000 7f624a849740 1 --2- 172.21.15.31:0/2168986751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad19f4b460 0x55ad19f4d850 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.057+0000 7f624a849740 1 --2- 172.21.15.31:0/2168986751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad19f420f0 0x55ad19f42570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.057+0000 7f624a849740 1 --2- 172.21.15.31:0/2168986751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad19f42dd0 0x55ad19f41b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.057+0000 7f624a849740 1 -- 172.21.15.31:0/2168986751 >> 172.21.15.31:0/2168986751 conn(0x55ad19f09200 msgr2=0x55ad19f3cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:29.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.060+0000 7f6245200640 1 --2- 172.21.15.31:0/4042805778 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f62240a3260 0x7f62240a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.061+0000 7f6245200640 1 --2- 172.21.15.31:0/4042805778 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f62240a3260 0x7f62240a5720 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x55ad19f43320 tx=0x7f62380095c0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.061+0000 7f624a849740 1 -- 172.21.15.31:0/2168986751 shutdown_connections 2024-03-21T10:43:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.061+0000 7f624a849740 1 -- 172.21.15.31:0/2168986751 wait complete. 2024-03-21T10:43:29.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.062+0000 7f624a849740 1 -- start start 2024-03-21T10:43:29.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.063+0000 7f624a849740 1 Processor -- start 2024-03-21T10:43:29.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.065+0000 7f624a849740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad19f420f0 0x55ad19f4ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:29.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.066+0000 7f624a849740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad19f42dd0 0x55ad19f47ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:29.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.066+0000 7f6245200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad19f420f0 0x55ad19f4ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:29.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.066+0000 7f6245200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad19f420f0 0x55ad19f4ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48922/0 (socket says 172.21.15.31:48922) 2024-03-21T10:43:29.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.066+0000 7f6245200640 1 -- 172.21.15.31:0/2823731982 learned_addr learned my addr 172.21.15.31:0/2823731982 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:29.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.066+0000 7f623fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad19f42dd0 0x55ad19f47ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:29.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.066+0000 7f624a849740 1 --2- 172.21.15.31:0/2823731982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad19f4b460 0x55ad19f483e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:29.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.067+0000 7f6245c00640 1 --2- 172.21.15.31:0/2823731982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad19f4b460 0x55ad19f483e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:29.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.067+0000 7f624a849740 1 -- 172.21.15.31:0/2823731982 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad19f4e370 con 0x55ad19f420f0 2024-03-21T10:43:29.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.067+0000 7f624a849740 1 -- 172.21.15.31:0/2823731982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad19e61e80 con 0x55ad19f4b460 2024-03-21T10:43:29.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.067+0000 7f624a849740 1 -- 172.21.15.31:0/2823731982 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad19f4e4f0 con 0x55ad19f42dd0 2024-03-21T10:43:29.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.071+0000 7f624a849740 10 client.0 _ll_register_callbacks cb 0x55ad19e74130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:43:29.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.072+0000 7f6245c00640 1 -- 172.21.15.31:0/2823731982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad19f42dd0 msgr2=0x55ad19f47ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:29.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.072+0000 7f6245c00640 1 --2- 172.21.15.31:0/2823731982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad19f42dd0 0x55ad19f47ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.072+0000 7f6245c00640 1 -- 172.21.15.31:0/2823731982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad19f420f0 msgr2=0x55ad19f4ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:29.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.072+0000 7f6245c00640 1 --2- 172.21.15.31:0/2823731982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad19f420f0 0x55ad19f4ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.072+0000 7f6245c00640 1 -- 172.21.15.31:0/2823731982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad19c847d0 con 0x55ad19f4b460 2024-03-21T10:43:29.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.072+0000 7f623fe00640 1 --2- 172.21.15.31:0/2823731982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad19f42dd0 0x55ad19f47ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:43:29.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.073+0000 7f622aa00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:43:29.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.073+0000 7f6245c00640 1 --2- 172.21.15.31:0/2823731982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad19f4b460 0x55ad19f483e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f622c0ee300 tx=0x7f622c0bcb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae5b51f3a1580aec server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:29.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.073+0000 7f622aa00640 1 -- 172.21.15.31:0/2823731982 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622c02cac0 con 0x55ad19f4b460 2024-03-21T10:43:29.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.073+0000 7f622aa00640 1 -- 172.21.15.31:0/2823731982 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f622c02c8c0 con 0x55ad19f4b460 2024-03-21T10:43:29.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.074+0000 7f624a849740 1 -- 172.21.15.31:0/2823731982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6230004c80 con 0x55ad19f4b460 2024-03-21T10:43:29.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.074+0000 7f622aa00640 1 -- 172.21.15.31:0/2823731982 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f622c0bcfd0 con 0x55ad19f4b460 2024-03-21T10:43:29.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.074+0000 7f624a849740 10 client.26834 fetch_fsmap learned FSMap version 26 2024-03-21T10:43:29.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.074+0000 7f624a849740 1 -- 172.21.15.31:0/2823731982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55ad19bba3f0 con 0x55ad19f4b460 2024-03-21T10:43:29.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.075+0000 7f622aa00640 1 -- 172.21.15.31:0/2823731982 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6230004c80 con 0x55ad19f4b460 2024-03-21T10:43:29.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.075+0000 7f622aa00640 1 -- 172.21.15.31:0/2823731982 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f622c03c070 con 0x55ad19f4b460 2024-03-21T10:43:29.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.075+0000 7f624a849740 10 client.26834 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:43:29.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.076+0000 7f624a849740 10 client.26834 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:43:29.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.076+0000 7f624a849740 20 client.26834 populate_metadata read hostname 'smithi031' 2024-03-21T10:43:29.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.076+0000 7f624a849740 1 --2- 172.21.15.31:0/2823731982 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ad19f5b080 0x55ad19f5d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:29.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.083+0000 7f624a849740 4 client.26834 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:43:29.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.083+0000 7f624a849740 1 -- 172.21.15.31:0/2823731982 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ad19eb3790 con 0x55ad19f5b080 2024-03-21T10:43:29.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.083+0000 7f6245200640 1 --2- 172.21.15.31:0/2823731982 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ad19f5b080 0x55ad19f5d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:29.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.084+0000 7f6245200640 1 --2- 172.21.15.31:0/2823731982 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ad19f5b080 0x55ad19f5d490 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffb102acfd8d256b server_cookie=1ed4d84754337c72 in_seq=0 out_seq=0 2024-03-21T10:43:29.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.084+0000 7f622aa00640 10 client.26834 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:29.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.085+0000 7f622aa00640 1 -- 172.21.15.31:0/2823731982 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ad19eb3790 con 0x55ad19f5b080 2024-03-21T10:43:29.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.085+0000 7f622aa00640 10 client.26834 handle_command_reply: tid=0 2024-03-21T10:43:29.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.085+0000 7f624a849740 1 client.26834 shutdown 2024-03-21T10:43:29.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.090+0000 7f624a849740 1 -- 172.21.15.31:0/2823731982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad19f4b460 msgr2=0x55ad19f483e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:29.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.090+0000 7f624a849740 1 --2- 172.21.15.31:0/2823731982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad19f4b460 0x55ad19f483e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f622c0ee300 tx=0x7f622c0bcb90 comp rx=0 tx=0).stop 2024-03-21T10:43:29.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.091+0000 7f624a849740 1 -- 172.21.15.31:0/2823731982 shutdown_connections 2024-03-21T10:43:29.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.091+0000 7f624a849740 1 --2- 172.21.15.31:0/2823731982 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ad19f5b080 0x55ad19f5d490 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.091+0000 7f624a849740 1 --2- 172.21.15.31:0/2823731982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad19f4b460 0x55ad19f483e0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.091+0000 7f622aa00640 0 client.26834 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:29.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.091+0000 7f624a849740 1 --2- 172.21.15.31:0/2823731982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad19f42dd0 0x55ad19f47ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.091+0000 7f624a849740 1 --2- 172.21.15.31:0/2823731982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad19f420f0 0x55ad19f4ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.091+0000 7f624a849740 1 -- 172.21.15.31:0/2823731982 >> 172.21.15.31:0/2823731982 conn(0x55ad19f09200 msgr2=0x55ad19f3e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:29.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.091+0000 7f624a849740 1 -- 172.21.15.31:0/2823731982 shutdown_connections 2024-03-21T10:43:29.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.092+0000 7f624a849740 1 -- 172.21.15.31:0/2823731982 wait complete. 2024-03-21T10:43:29.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.093+0000 7f624a849740 20 client.26834 trim_cache size 0 max 16384 2024-03-21T10:43:29.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.144+0000 7f624a849740 1 Processor -- start 2024-03-21T10:43:29.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.144+0000 7f624a849740 1 -- start start 2024-03-21T10:43:29.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.146+0000 7f624a849740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad19f420f0 0x55ad19f3e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:29.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.146+0000 7f624a849740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad19f42dd0 0x55ad19f4c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:29.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.146+0000 7f624a849740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad19f4b460 0x55ad19f4cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:29.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.146+0000 7f624a849740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62401b7190 con 0x55ad19f420f0 2024-03-21T10:43:29.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.146+0000 7f624a849740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62401b4c90 con 0x55ad19f4b460 2024-03-21T10:43:29.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.147+0000 7f624a849740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62401b2d30 con 0x55ad19f42dd0 2024-03-21T10:43:29.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.147+0000 7f623fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad19f42dd0 0x55ad19f4c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:29.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.147+0000 7f623fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad19f42dd0 0x55ad19f4c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42080/0 (socket says 172.21.15.31:42080) 2024-03-21T10:43:29.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.147+0000 7f623fe00640 1 -- 172.21.15.31:0/3688195652 learned_addr learned my addr 172.21.15.31:0/3688195652 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:29.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.147+0000 7f6245c00640 1 --2- 172.21.15.31:0/3688195652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad19f4b460 0x55ad19f4cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:29.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.147+0000 7f6245c00640 1 -- 172.21.15.31:0/3688195652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad19f42dd0 msgr2=0x55ad19f4c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:29.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.147+0000 7f6245c00640 1 --2- 172.21.15.31:0/3688195652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad19f42dd0 0x55ad19f4c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.147+0000 7f6245c00640 1 -- 172.21.15.31:0/3688195652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad19f420f0 msgr2=0x55ad19f3e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:29.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.147+0000 7f6245c00640 1 --2- 172.21.15.31:0/3688195652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad19f420f0 0x55ad19f3e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.148+0000 7f6245c00640 1 -- 172.21.15.31:0/3688195652 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad19b7a0f0 con 0x55ad19f4b460 2024-03-21T10:43:29.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.148+0000 7f6245c00640 1 --2- 172.21.15.31:0/3688195652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad19f4b460 0x55ad19f4cd90 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f622c0b9950 tx=0x7f622c02bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f1f1c916a9f2552 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:29.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.148+0000 7f6235600640 1 -- 172.21.15.31:0/3688195652 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622c0b9270 con 0x55ad19f4b460 2024-03-21T10:43:29.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.149+0000 7f6235600640 1 -- 172.21.15.31:0/3688195652 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f622c02cac0 con 0x55ad19f4b460 2024-03-21T10:43:29.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.149+0000 7f6235600640 1 -- 172.21.15.31:0/3688195652 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622c0b9270 con 0x55ad19f4b460 2024-03-21T10:43:29.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.149+0000 7f624a849740 1 -- 172.21.15.31:0/3688195652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad19f4b460 msgr2=0x55ad19f4cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:29.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.149+0000 7f624a849740 1 --2- 172.21.15.31:0/3688195652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad19f4b460 0x55ad19f4cd90 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f622c0b9950 tx=0x7f622c02bc30 comp rx=0 tx=0).stop 2024-03-21T10:43:29.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.151+0000 7f624a849740 1 -- 172.21.15.31:0/3688195652 shutdown_connections 2024-03-21T10:43:29.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.151+0000 7f624a849740 1 --2- 172.21.15.31:0/3688195652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad19f4b460 0x55ad19f4cd90 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.151+0000 7f624a849740 1 --2- 172.21.15.31:0/3688195652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad19f42dd0 0x55ad19f4c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.151+0000 7f624a849740 1 --2- 172.21.15.31:0/3688195652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad19f420f0 0x55ad19f3e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.151+0000 7f624a849740 1 -- 172.21.15.31:0/3688195652 >> 172.21.15.31:0/3688195652 conn(0x55ad19f09200 msgr2=0x55ad19f3df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:29.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.151+0000 7f624a849740 1 -- 172.21.15.31:0/3688195652 shutdown_connections 2024-03-21T10:43:29.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.151+0000 7f624a849740 1 -- 172.21.15.31:0/3688195652 wait complete. 2024-03-21T10:43:29.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.151+0000 7f624a849740 1 -- start start 2024-03-21T10:43:29.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.152+0000 7f624a849740 1 Processor -- start 2024-03-21T10:43:29.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.153+0000 7f624a849740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad19f420f0 0x55ad19f44b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:29.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.153+0000 7f624a849740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad19f42dd0 0x55ad19f450a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:29.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.154+0000 7f624a849740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad19f4b460 0x55ad19fb5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:29.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.154+0000 7f624a849740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62401b3330 con 0x55ad19f42dd0 2024-03-21T10:43:29.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.154+0000 7f624a849740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62401b4630 con 0x55ad19f4b460 2024-03-21T10:43:29.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.154+0000 7f624a849740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62401b3970 con 0x55ad19f420f0 2024-03-21T10:43:29.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.154+0000 7f624a849740 10 client.0 _ll_register_callbacks cb 0x7f622c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:43:29.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.153+0000 7f6245200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad19f420f0 0x55ad19f44b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:29.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.154+0000 7f6245200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad19f420f0 0x55ad19f44b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42094/0 (socket says 172.21.15.31:42094) 2024-03-21T10:43:29.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.154+0000 7f6245200640 1 -- 172.21.15.31:0/4086528383 learned_addr learned my addr 172.21.15.31:0/4086528383 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:29.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.155+0000 7f6245200640 1 -- 172.21.15.31:0/4086528383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad19f4b460 msgr2=0x55ad19fb5920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:43:29.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.156+0000 7f623fe00640 1 --2- 172.21.15.31:0/4086528383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad19f42dd0 0x55ad19f450a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:29.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.156+0000 7f6245c00640 1 --2- 172.21.15.31:0/4086528383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad19f4b460 0x55ad19fb5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:29.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.156+0000 7f6245200640 1 --2- 172.21.15.31:0/4086528383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad19f4b460 0x55ad19fb5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.157+0000 7f6245200640 1 -- 172.21.15.31:0/4086528383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad19f42dd0 msgr2=0x55ad19f450a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:29.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.157+0000 7f6245200640 1 --2- 172.21.15.31:0/4086528383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad19f42dd0 0x55ad19f450a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.157+0000 7f6245200640 1 -- 172.21.15.31:0/4086528383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad19fb5e60 con 0x55ad19f420f0 2024-03-21T10:43:29.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.157+0000 7f6245c00640 1 --2- 172.21.15.31:0/4086528383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad19f4b460 0x55ad19fb5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:43:29.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.157+0000 7f623fe00640 1 --2- 172.21.15.31:0/4086528383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad19f42dd0 0x55ad19f450a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:43:29.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.158+0000 7f6245200640 1 --2- 172.21.15.31:0/4086528383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad19f420f0 0x55ad19f44b60 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6238005fd0 tx=0x7f6238004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cc1865e37b44680 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:29.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.158+0000 7f622aa00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:43:29.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.158+0000 7f622aa00640 1 -- 172.21.15.31:0/4086528383 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6238004660 con 0x55ad19f420f0 2024-03-21T10:43:29.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.159+0000 7f622aa00640 1 -- 172.21.15.31:0/4086528383 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f6238004800 con 0x55ad19f420f0 2024-03-21T10:43:29.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.159+0000 7f624a849740 1 -- 172.21.15.31:0/4086528383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f622c02cac0 con 0x55ad19f420f0 2024-03-21T10:43:29.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.163+0000 7f622aa00640 1 -- 172.21.15.31:0/4086528383 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6238036050 con 0x55ad19f420f0 2024-03-21T10:43:29.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.163+0000 7f624a849740 10 client.26785 fetch_fsmap learned FSMap version 26 2024-03-21T10:43:29.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.163+0000 7f624a849740 1 -- 172.21.15.31:0/4086528383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55ad19fb7160 con 0x55ad19f420f0 2024-03-21T10:43:29.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.165+0000 7f622aa00640 1 -- 172.21.15.31:0/4086528383 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f622c02cac0 con 0x55ad19f420f0 2024-03-21T10:43:29.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.165+0000 7f622aa00640 1 -- 172.21.15.31:0/4086528383 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f623800b040 con 0x55ad19f420f0 2024-03-21T10:43:29.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.168+0000 7f624a849740 10 client.26785 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:43:29.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.168+0000 7f624a849740 10 client.26785 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:43:29.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.168+0000 7f624a849740 20 client.26785 populate_metadata read hostname 'smithi031' 2024-03-21T10:43:29.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.168+0000 7f624a849740 1 --2- 172.21.15.31:0/4086528383 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ad19f5b080 0x55ad19fb9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:29.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.168+0000 7f624a849740 4 client.26785 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:43:29.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.168+0000 7f624a849740 1 -- 172.21.15.31:0/4086528383 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ad19eadae0 con 0x55ad19f5b080 2024-03-21T10:43:29.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.168+0000 7f623fe00640 1 --2- 172.21.15.31:0/4086528383 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ad19f5b080 0x55ad19fb9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:29.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.169+0000 7f623fe00640 1 --2- 172.21.15.31:0/4086528383 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ad19f5b080 0x55ad19fb9920 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=940ad5b9d682592a server_cookie=567f952aca377ec1 in_seq=0 out_seq=0 2024-03-21T10:43:29.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.169+0000 7f622aa00640 10 client.26785 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:29.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.169+0000 7f622aa00640 1 -- 172.21.15.31:0/4086528383 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55ad19eadae0 con 0x55ad19f5b080 2024-03-21T10:43:29.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.169+0000 7f622aa00640 10 client.26785 handle_command_reply: tid=0 2024-03-21T10:43:29.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.169+0000 7f624a849740 1 client.26785 shutdown 2024-03-21T10:43:29.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.170+0000 7f624a849740 1 -- 172.21.15.31:0/4086528383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad19f420f0 msgr2=0x55ad19f44b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:29.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.170+0000 7f624a849740 1 --2- 172.21.15.31:0/4086528383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad19f420f0 0x55ad19f44b60 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6238005fd0 tx=0x7f6238004ab0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.171+0000 7f624a849740 1 -- 172.21.15.31:0/4086528383 shutdown_connections 2024-03-21T10:43:29.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.171+0000 7f624a849740 1 --2- 172.21.15.31:0/4086528383 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ad19f5b080 0x55ad19fb9920 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.171+0000 7f624a849740 1 --2- 172.21.15.31:0/4086528383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad19f4b460 0x55ad19fb5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.171+0000 7f624a849740 1 --2- 172.21.15.31:0/4086528383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad19f42dd0 0x55ad19f450a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.171+0000 7f624a849740 1 --2- 172.21.15.31:0/4086528383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ad19f420f0 0x55ad19f44b60 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.171+0000 7f622aa00640 0 client.26785 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:29.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.171+0000 7f624a849740 1 -- 172.21.15.31:0/4086528383 >> 172.21.15.31:0/4086528383 conn(0x55ad19f09200 msgr2=0x55ad19f3bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:29.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.171+0000 7f624a849740 1 -- 172.21.15.31:0/4086528383 shutdown_connections 2024-03-21T10:43:29.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.171+0000 7f624a849740 1 -- 172.21.15.31:0/4086528383 wait complete. 2024-03-21T10:43:29.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.172+0000 7f624a849740 20 client.26785 trim_cache size 0 max 16384 2024-03-21T10:43:29.170 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:43:29.171 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-03-21T10:43:29.171 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-03-21T10:43:29.171 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:43:29.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.174+0000 7f6246600640 1 -- 172.21.15.31:0/4042805778 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f62240a3260 msgr2=0x7f62240a5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:29.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.174+0000 7f6246600640 1 --2- 172.21.15.31:0/4042805778 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f62240a3260 0x7f62240a5720 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x55ad19f43320 tx=0x7f62380095c0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.174+0000 7f6246600640 1 -- 172.21.15.31:0/4042805778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f624010ba50 msgr2=0x7f62401a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:29.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.174+0000 7f6246600640 1 --2- 172.21.15.31:0/4042805778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f624010ba50 0x7f62401a2b90 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f622c02c170 tx=0x7f622c034440 comp rx=0 tx=0).stop 2024-03-21T10:43:29.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.175+0000 7f6246600640 1 -- 172.21.15.31:0/4042805778 shutdown_connections 2024-03-21T10:43:29.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.175+0000 7f6246600640 1 --2- 172.21.15.31:0/4042805778 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f62240a3260 0x7f62240a5720 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.175+0000 7f6246600640 1 --2- 172.21.15.31:0/4042805778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f624010ba50 0x7f62401a2b90 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.175+0000 7f6246600640 1 --2- 172.21.15.31:0/4042805778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6240102f50 0x7f624010ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.175+0000 7f6246600640 1 --2- 172.21.15.31:0/4042805778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6240102630 0x7f624010a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.175+0000 7f6246600640 1 -- 172.21.15.31:0/4042805778 >> 172.21.15.31:0/4042805778 conn(0x7f6240077130 msgr2=0x7f62401001b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:29.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.175+0000 7f6246600640 1 -- 172.21.15.31:0/4042805778 shutdown_connections 2024-03-21T10:43:29.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.175+0000 7f6246600640 1 -- 172.21.15.31:0/4042805778 wait complete. 2024-03-21T10:43:29.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:28 smithi040 ceph-mon[36154]: pgmap v150: 129 pgs: 129 active+clean; 28 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 190 MiB/s wr, 1.64k op/s 2024-03-21T10:43:29.221 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-21T10:43:29.221 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:43:29.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:28 smithi070 ceph-mon[36330]: pgmap v150: 129 pgs: 129 active+clean; 28 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 190 MiB/s wr, 1.64k op/s 2024-03-21T10:43:29.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.599+0000 7fe7a1800640 1 -- 172.21.15.31:0/1103464032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe79c107eb0 msgr2=0x7fe79c108290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:29.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.599+0000 7fe7a1800640 1 --2- 172.21.15.31:0/1103464032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe79c107eb0 0x7fe79c108290 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe78c009900 tx=0x7fe78c02bc40 comp rx=0 tx=0).stop 2024-03-21T10:43:29.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.599+0000 7fe7a1800640 1 -- 172.21.15.31:0/1103464032 shutdown_connections 2024-03-21T10:43:29.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.599+0000 7fe7a1800640 1 --2- 172.21.15.31:0/1103464032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe79c10cf80 0x7fe79c114b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.599+0000 7fe7a1800640 1 --2- 172.21.15.31:0/1103464032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe79c108860 0x7fe79c10c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.599+0000 7fe7a1800640 1 --2- 172.21.15.31:0/1103464032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe79c107eb0 0x7fe79c108290 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.599+0000 7fe7a1800640 1 -- 172.21.15.31:0/1103464032 >> 172.21.15.31:0/1103464032 conn(0x7fe79c0773c0 msgr2=0x7fe79c0797e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:29.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.600+0000 7fe7a1800640 1 -- 172.21.15.31:0/1103464032 shutdown_connections 2024-03-21T10:43:29.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.600+0000 7fe7a1800640 1 -- 172.21.15.31:0/1103464032 wait complete. 2024-03-21T10:43:29.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.602+0000 7fe7a1800640 1 Processor -- start 2024-03-21T10:43:29.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.605+0000 7fe7a1800640 1 -- start start 2024-03-21T10:43:29.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.606+0000 7fe7a1800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe79c107eb0 0x7fe79c072d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:29.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.606+0000 7fe7a1800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe79c10cf80 0x7fe79c0732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:29.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.607+0000 7fe7a1800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe79c06ce50 0x7fe79c06d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:29.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.607+0000 7fe7a1800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79c06bad0 con 0x7fe79c107eb0 2024-03-21T10:43:29.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.607+0000 7fe7a1800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79c06b950 con 0x7fe79c10cf80 2024-03-21T10:43:29.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.607+0000 7fe7a1800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79c06bc50 con 0x7fe79c06ce50 2024-03-21T10:43:29.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.607+0000 7fe79a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe79c10cf80 0x7fe79c0732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:29.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.607+0000 7fe79a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe79c10cf80 0x7fe79c0732d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58098/0 (socket says 172.21.15.31:58098) 2024-03-21T10:43:29.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.607+0000 7fe79a000640 1 -- 172.21.15.31:0/1876765928 learned_addr learned my addr 172.21.15.31:0/1876765928 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:29.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.607+0000 7fe79aa00640 1 --2- 172.21.15.31:0/1876765928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe79c107eb0 0x7fe79c072d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:29.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.607+0000 7fe79a000640 1 -- 172.21.15.31:0/1876765928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe79c06ce50 msgr2=0x7fe79c06d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:29.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.607+0000 7fe79a000640 1 --2- 172.21.15.31:0/1876765928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe79c06ce50 0x7fe79c06d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.607+0000 7fe79a000640 1 -- 172.21.15.31:0/1876765928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe79c107eb0 msgr2=0x7fe79c072d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:29.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.607+0000 7fe79a000640 1 --2- 172.21.15.31:0/1876765928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe79c107eb0 0x7fe79c072d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:29.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.608+0000 7fe79a000640 1 -- 172.21.15.31:0/1876765928 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe79c06dba0 con 0x7fe79c10cf80 2024-03-21T10:43:29.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.608+0000 7fe79a000640 1 --2- 172.21.15.31:0/1876765928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe79c10cf80 0x7fe79c0732d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe78400c970 tx=0x7fe78400ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:29.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.608+0000 7fe793400640 1 -- 172.21.15.31:0/1876765928 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe784014070 con 0x7fe79c10cf80 2024-03-21T10:43:29.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.608+0000 7fe793400640 1 -- 172.21.15.31:0/1876765928 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe784004440 con 0x7fe79c10cf80 2024-03-21T10:43:29.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.608+0000 7fe793400640 1 -- 172.21.15.31:0/1876765928 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78400f450 con 0x7fe79c10cf80 2024-03-21T10:43:29.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.610+0000 7fe7a1800640 1 -- 172.21.15.31:0/1876765928 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe79c11d760 con 0x7fe79c10cf80 2024-03-21T10:43:29.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.610+0000 7fe7a1800640 1 -- 172.21.15.31:0/1876765928 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe79c11dc30 con 0x7fe79c10cf80 2024-03-21T10:43:29.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.613+0000 7fe7a1800640 1 -- 172.21.15.31:0/1876765928 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe79c109380 con 0x7fe79c10cf80 2024-03-21T10:43:29.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.621+0000 7fe793400640 1 -- 172.21.15.31:0/1876765928 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fe784017070 con 0x7fe79c10cf80 2024-03-21T10:43:29.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.624+0000 7fe793400640 1 --2- 172.21.15.31:0/1876765928 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe7800a31e0 0x7fe7800a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:29.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.625+0000 7fe793400640 1 -- 172.21.15.31:0/1876765928 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fe7840cf9c0 con 0x7fe79c10cf80 2024-03-21T10:43:29.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.625+0000 7fe793400640 1 -- 172.21.15.31:0/1876765928 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe7840fd7e0 con 0x7fe79c10cf80 2024-03-21T10:43:29.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.628+0000 7fe79aa00640 1 --2- 172.21.15.31:0/1876765928 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe7800a31e0 0x7fe7800a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:29.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:29.635+0000 7fe79aa00640 1 --2- 172.21.15.31:0/1876765928 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe7800a31e0 0x7fe7800a56a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe78c02efd0 tx=0x7fe78c000bc0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:30.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.163+0000 7fe7a1800640 1 -- 172.21.15.31:0/1876765928 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe79c068970 con 0x7fe79c10cf80 2024-03-21T10:43:30.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.164+0000 7fe793400640 1 -- 172.21.15.31:0/1876765928 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+11384 (secure 0 0 0) 0x7fe78409c210 con 0x7fe79c10cf80 2024-03-21T10:43:30.162 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:43:30.162 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:42:31.182549+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24547,24596]},"id":1}]} 2024-03-21T10:43:30.163 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2024-03-21T10:43:30.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.168+0000 7fe7a1800640 1 -- 172.21.15.31:0/1876765928 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe7800a31e0 msgr2=0x7fe7800a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:30.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.168+0000 7fe7a1800640 1 --2- 172.21.15.31:0/1876765928 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe7800a31e0 0x7fe7800a56a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe78c02efd0 tx=0x7fe78c000bc0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.168+0000 7fe7a1800640 1 -- 172.21.15.31:0/1876765928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe79c10cf80 msgr2=0x7fe79c0732d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:30.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.168+0000 7fe7a1800640 1 --2- 172.21.15.31:0/1876765928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe79c10cf80 0x7fe79c0732d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe78400c970 tx=0x7fe78400ce30 comp rx=0 tx=0).stop 2024-03-21T10:43:30.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.168+0000 7fe7a1800640 1 -- 172.21.15.31:0/1876765928 shutdown_connections 2024-03-21T10:43:30.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.168+0000 7fe7a1800640 1 --2- 172.21.15.31:0/1876765928 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe7800a31e0 0x7fe7800a56a0 secure :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe78c02efd0 tx=0x7fe78c000bc0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.168+0000 7fe7a1800640 1 --2- 172.21.15.31:0/1876765928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe79c06ce50 0x7fe79c06d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.168+0000 7fe7a1800640 1 --2- 172.21.15.31:0/1876765928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe79c10cf80 0x7fe79c0732d0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.168+0000 7fe7a1800640 1 --2- 172.21.15.31:0/1876765928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe79c107eb0 0x7fe79c072d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.168+0000 7fe7a1800640 1 -- 172.21.15.31:0/1876765928 >> 172.21.15.31:0/1876765928 conn(0x7fe79c0773c0 msgr2=0x7fe79c077b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:30.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.169+0000 7fe7a1800640 1 -- 172.21.15.31:0/1876765928 shutdown_connections 2024-03-21T10:43:30.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.169+0000 7fe7a1800640 1 -- 172.21.15.31:0/1876765928 wait complete. 2024-03-21T10:43:30.188 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:43:30.188 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-21T10:43:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.417+0000 7f02b7200640 1 -- 172.21.15.31:0/410212732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02b0106590 msgr2=0x7f02b0068f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.417+0000 7f02b7200640 1 --2- 172.21.15.31:0/410212732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02b0106590 0x7f02b0068f40 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f029c009900 tx=0x7f029c02bc60 comp rx=0 tx=0).stop 2024-03-21T10:43:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.417+0000 7f02b7200640 1 -- 172.21.15.31:0/410212732 shutdown_connections 2024-03-21T10:43:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.417+0000 7f02b7200640 1 --2- 172.21.15.31:0/410212732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02b0104090 0x7f02b0104540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.417+0000 7f02b7200640 1 --2- 172.21.15.31:0/410212732 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02b0069480 0x7f02b00698e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.417+0000 7f02b7200640 1 --2- 172.21.15.31:0/410212732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02b0106590 0x7f02b0068f40 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.417+0000 7f02b7200640 1 -- 172.21.15.31:0/410212732 >> 172.21.15.31:0/410212732 conn(0x7f02b0077690 msgr2=0x7f02b0077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.417+0000 7f02b7200640 1 -- 172.21.15.31:0/410212732 shutdown_connections 2024-03-21T10:43:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.417+0000 7f02b7200640 1 -- 172.21.15.31:0/410212732 wait complete. 2024-03-21T10:43:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.418+0000 7f02b7200640 1 Processor -- start 2024-03-21T10:43:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.418+0000 7f02b7200640 1 -- start start 2024-03-21T10:43:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.419+0000 7f02b7200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02b0069480 0x7f02b019e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.419+0000 7f02b7200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02b0104090 0x7f02b019e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.419+0000 7f02b7200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02b0106590 0x7f02b01a2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.419+0000 7f02b7200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b0118040 con 0x7f02b0069480 2024-03-21T10:43:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.419+0000 7f02b7200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b0117ec0 con 0x7f02b0104090 2024-03-21T10:43:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.419+0000 7f02b7200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b01181c0 con 0x7f02b0106590 2024-03-21T10:43:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.419+0000 7f02af400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02b0104090 0x7f02b019e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.419+0000 7f02afe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02b0069480 0x7f02b019e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.419+0000 7f02af400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02b0104090 0x7f02b019e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58106/0 (socket says 172.21.15.31:58106) 2024-03-21T10:43:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.419+0000 7f02af400640 1 -- 172.21.15.31:0/4193800009 learned_addr learned my addr 172.21.15.31:0/4193800009 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.419+0000 7f02b5200640 1 --2- 172.21.15.31:0/4193800009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02b0106590 0x7f02b01a2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.419+0000 7f02af400640 1 -- 172.21.15.31:0/4193800009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02b0106590 msgr2=0x7f02b01a2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.419+0000 7f02af400640 1 --2- 172.21.15.31:0/4193800009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02b0106590 0x7f02b01a2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.419+0000 7f02af400640 1 -- 172.21.15.31:0/4193800009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02b0069480 msgr2=0x7f02b019e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.419+0000 7f02af400640 1 --2- 172.21.15.31:0/4193800009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02b0069480 0x7f02b019e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.419+0000 7f02af400640 1 -- 172.21.15.31:0/4193800009 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b01a3490 con 0x7f02b0104090 2024-03-21T10:43:30.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.420+0000 7f02af400640 1 --2- 172.21.15.31:0/4193800009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02b0104090 0x7f02b019e9e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f02a000e9f0 tx=0x7f02a000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:30.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.420+0000 7f02acc00640 1 -- 172.21.15.31:0/4193800009 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a000ccc0 con 0x7f02b0104090 2024-03-21T10:43:30.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.420+0000 7f02acc00640 1 -- 172.21.15.31:0/4193800009 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f02a0010910 con 0x7f02b0104090 2024-03-21T10:43:30.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.420+0000 7f02b7200640 1 -- 172.21.15.31:0/4193800009 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b01a3780 con 0x7f02b0104090 2024-03-21T10:43:30.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.420+0000 7f02acc00640 1 -- 172.21.15.31:0/4193800009 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a00188e0 con 0x7f02b0104090 2024-03-21T10:43:30.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.420+0000 7f02b7200640 1 -- 172.21.15.31:0/4193800009 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b01ab010 con 0x7f02b0104090 2024-03-21T10:43:30.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.423+0000 7f02acc00640 1 -- 172.21.15.31:0/4193800009 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f02a0010ac0 con 0x7f02b0104090 2024-03-21T10:43:30.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.425+0000 7f02acc00640 1 --2- 172.21.15.31:0/4193800009 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f02940a3080 0x7f02940a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.425+0000 7f02acc00640 1 -- 172.21.15.31:0/4193800009 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f02a00cfdd0 con 0x7f02b0104090 2024-03-21T10:43:30.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.425+0000 7f02b9e31740 1 Processor -- start 2024-03-21T10:43:30.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.425+0000 7f02b9e31740 1 -- start start 2024-03-21T10:43:30.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.425+0000 7f02afe00640 1 --2- 172.21.15.31:0/4193800009 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f02940a3080 0x7f02940a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.426+0000 7f02b9e31740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a202aafd0 0x560a202ab3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.426+0000 7f02b9e31740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a202ab8f0 0x560a202b4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.426+0000 7f02b9e31740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a202b46c0 0x560a202b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.426+0000 7f02b9e31740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a2026d3b0 con 0x560a202b46c0 2024-03-21T10:43:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.426+0000 7f02b9e31740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a2020f830 con 0x560a202aafd0 2024-03-21T10:43:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.426+0000 7f02b9e31740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a201b85d0 con 0x560a202ab8f0 2024-03-21T10:43:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.426+0000 7f02af400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a202ab8f0 0x560a202b4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.426+0000 7f02af400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a202ab8f0 0x560a202b4180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42138/0 (socket says 172.21.15.31:42138) 2024-03-21T10:43:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.426+0000 7f02af400640 1 -- 172.21.15.31:0/3931167719 learned_addr learned my addr 172.21.15.31:0/3931167719 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.427+0000 7f02afe00640 1 --2- 172.21.15.31:0/4193800009 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f02940a3080 0x7f02940a5540 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f029c000c00 tx=0x7f029c005a20 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.427+0000 7f02af400640 1 -- 172.21.15.31:0/3931167719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a202aafd0 msgr2=0x560a202ab3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.427+0000 7f02afe00640 1 --2- 172.21.15.31:0/3931167719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a202aafd0 0x560a202ab3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.427+0000 7f02af400640 1 --2- 172.21.15.31:0/3931167719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a202aafd0 0x560a202ab3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.427+0000 7f02af400640 1 -- 172.21.15.31:0/3931167719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a202b46c0 msgr2=0x560a202b6ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:43:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.427+0000 7f02af400640 1 --2- 172.21.15.31:0/3931167719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a202b46c0 0x560a202b6ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.427+0000 7f02af400640 1 -- 172.21.15.31:0/3931167719 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a201d02c0 con 0x560a202ab8f0 2024-03-21T10:43:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.427+0000 7f02afe00640 1 --2- 172.21.15.31:0/3931167719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a202aafd0 0x560a202ab3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:43:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.427+0000 7f02af400640 1 --2- 172.21.15.31:0/3931167719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a202ab8f0 0x560a202b4180 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f02a00a4300 tx=0x7f02a00bc110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3682962314940a54 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.428+0000 7f02a5600640 1 -- 172.21.15.31:0/3931167719 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a0010de0 con 0x560a202ab8f0 2024-03-21T10:43:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.428+0000 7f02a5600640 1 -- 172.21.15.31:0/3931167719 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f02a00cd400 con 0x560a202ab8f0 2024-03-21T10:43:30.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.428+0000 7f02a5600640 1 -- 172.21.15.31:0/3931167719 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a009b580 con 0x560a202ab8f0 2024-03-21T10:43:30.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.429+0000 7f02b9e31740 1 -- 172.21.15.31:0/3931167719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a202ab8f0 msgr2=0x560a202b4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:30.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.429+0000 7f02b9e31740 1 --2- 172.21.15.31:0/3931167719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a202ab8f0 0x560a202b4180 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f02a00a4300 tx=0x7f02a00bc110 comp rx=0 tx=0).stop 2024-03-21T10:43:30.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.429+0000 7f02b9e31740 1 -- 172.21.15.31:0/3931167719 shutdown_connections 2024-03-21T10:43:30.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.429+0000 7f02b9e31740 1 --2- 172.21.15.31:0/3931167719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a202b46c0 0x560a202b6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.429+0000 7f02b9e31740 1 --2- 172.21.15.31:0/3931167719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a202ab8f0 0x560a202b4180 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.429+0000 7f02b9e31740 1 --2- 172.21.15.31:0/3931167719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a202aafd0 0x560a202ab3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.429+0000 7f02b9e31740 1 -- 172.21.15.31:0/3931167719 >> 172.21.15.31:0/3931167719 conn(0x560a202a58f0 msgr2=0x560a202a5cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:30.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.429+0000 7f02b9e31740 1 -- 172.21.15.31:0/3931167719 shutdown_connections 2024-03-21T10:43:30.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.429+0000 7f02b9e31740 1 -- 172.21.15.31:0/3931167719 wait complete. 2024-03-21T10:43:30.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.430+0000 7f02b9e31740 1 -- start start 2024-03-21T10:43:30.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.430+0000 7f02b9e31740 1 Processor -- start 2024-03-21T10:43:30.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.431+0000 7f02b9e31740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a202aafd0 0x560a202b1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.431+0000 7f02b9e31740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a202ab8f0 0x560a202b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.431+0000 7f02b9e31740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a202b46c0 0x560a202b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.431+0000 7f02b9e31740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a201cba20 con 0x560a202aafd0 2024-03-21T10:43:30.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.431+0000 7f02b9e31740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a2020b8d0 con 0x560a202ab8f0 2024-03-21T10:43:30.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.431+0000 7f02b9e31740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a202b7290 con 0x560a202b46c0 2024-03-21T10:43:30.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.431+0000 7f02b9e31740 10 client.0 _ll_register_callbacks cb 0x560a202081c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:43:30.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.431+0000 7f02af400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a202ab8f0 0x560a202b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.431+0000 7f02af400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a202ab8f0 0x560a202b1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58126/0 (socket says 172.21.15.31:58126) 2024-03-21T10:43:30.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.432+0000 7f02af400640 1 -- 172.21.15.31:0/1778167336 learned_addr learned my addr 172.21.15.31:0/1778167336 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:30.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.432+0000 7f02b5200640 1 --2- 172.21.15.31:0/1778167336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a202b46c0 0x560a202b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.432+0000 7f02af400640 1 -- 172.21.15.31:0/1778167336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a202b46c0 msgr2=0x560a202b21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.432+0000 7f02af400640 1 --2- 172.21.15.31:0/1778167336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a202b46c0 0x560a202b21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.432+0000 7f02af400640 1 -- 172.21.15.31:0/1778167336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a202aafd0 msgr2=0x560a202b1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.432+0000 7f02af400640 1 --2- 172.21.15.31:0/1778167336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a202aafd0 0x560a202b1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.432+0000 7f02af400640 1 -- 172.21.15.31:0/1778167336 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560a1ff24210 con 0x560a202ab8f0 2024-03-21T10:43:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.433+0000 7f02af400640 1 --2- 172.21.15.31:0/1778167336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a202ab8f0 0x560a202b1ca0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f02a009c090 tx=0x7f02a0009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=caa7703c42d885b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.434+0000 7f029a000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:43:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.434+0000 7f029a000640 1 -- 172.21.15.31:0/1778167336 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a00b9070 con 0x560a202ab8f0 2024-03-21T10:43:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.434+0000 7f029a000640 1 -- 172.21.15.31:0/1778167336 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f02a0010640 con 0x560a202ab8f0 2024-03-21T10:43:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.434+0000 7f02b9e31740 1 -- 172.21.15.31:0/1778167336 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f02a00cd400 con 0x560a202ab8f0 2024-03-21T10:43:30.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.434+0000 7f029a000640 1 -- 172.21.15.31:0/1778167336 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02a0010de0 con 0x560a202ab8f0 2024-03-21T10:43:30.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.434+0000 7f02b9e31740 10 client.26852 fetch_fsmap learned FSMap version 26 2024-03-21T10:43:30.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.434+0000 7f02b9e31740 1 -- 172.21.15.31:0/1778167336 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x560a1fff0620 con 0x560a202ab8f0 2024-03-21T10:43:30.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.435+0000 7f029a000640 1 -- 172.21.15.31:0/1778167336 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f02a00cd400 con 0x560a202ab8f0 2024-03-21T10:43:30.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.435+0000 7f029a000640 1 -- 172.21.15.31:0/1778167336 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f02a00a1100 con 0x560a202ab8f0 2024-03-21T10:43:30.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.435+0000 7f02b9e31740 10 client.26852 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:43:30.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.435+0000 7f02b9e31740 10 client.26852 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:43:30.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.435+0000 7f02b9e31740 20 client.26852 populate_metadata read hostname 'smithi031' 2024-03-21T10:43:30.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.435+0000 7f02b9e31740 1 --2- 172.21.15.31:0/1778167336 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560a202c44c0 0x560a202c68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.435+0000 7f02b9e31740 4 client.26852 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:43:30.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.435+0000 7f02b9e31740 1 -- 172.21.15.31:0/1778167336 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560a2021cd70 con 0x560a202c44c0 2024-03-21T10:43:30.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.436+0000 7f02afe00640 1 --2- 172.21.15.31:0/1778167336 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560a202c44c0 0x560a202c68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.437+0000 7f02afe00640 1 --2- 172.21.15.31:0/1778167336 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560a202c44c0 0x560a202c68b0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f244e10ec71785fb server_cookie=8ee67e384139824f in_seq=0 out_seq=0 2024-03-21T10:43:30.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.437+0000 7f029a000640 10 client.26852 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:30.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.437+0000 7f029a000640 1 -- 172.21.15.31:0/1778167336 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560a2021cd70 con 0x560a202c44c0 2024-03-21T10:43:30.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.437+0000 7f029a000640 10 client.26852 handle_command_reply: tid=0 2024-03-21T10:43:30.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.438+0000 7f02b9e31740 1 client.26852 shutdown 2024-03-21T10:43:30.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.438+0000 7f02b9e31740 1 -- 172.21.15.31:0/1778167336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a202ab8f0 msgr2=0x560a202b1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:30.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.438+0000 7f02b9e31740 1 --2- 172.21.15.31:0/1778167336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a202ab8f0 0x560a202b1ca0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f02a009c090 tx=0x7f02a0009f90 comp rx=0 tx=0).stop 2024-03-21T10:43:30.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.439+0000 7f02b9e31740 1 -- 172.21.15.31:0/1778167336 shutdown_connections 2024-03-21T10:43:30.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.439+0000 7f02b9e31740 1 --2- 172.21.15.31:0/1778167336 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560a202c44c0 0x560a202c68b0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.439+0000 7f029a000640 0 client.26852 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:30.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.439+0000 7f02b9e31740 1 --2- 172.21.15.31:0/1778167336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a202b46c0 0x560a202b21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.439+0000 7f02b9e31740 1 --2- 172.21.15.31:0/1778167336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a202ab8f0 0x560a202b1ca0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.439+0000 7f02b9e31740 1 --2- 172.21.15.31:0/1778167336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a202aafd0 0x560a202b1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.439+0000 7f02b9e31740 1 -- 172.21.15.31:0/1778167336 >> 172.21.15.31:0/1778167336 conn(0x560a202a58f0 msgr2=0x560a202a7640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:30.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.439+0000 7f02b9e31740 1 -- 172.21.15.31:0/1778167336 shutdown_connections 2024-03-21T10:43:30.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.439+0000 7f02b9e31740 1 -- 172.21.15.31:0/1778167336 wait complete. 2024-03-21T10:43:30.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.439+0000 7f02b9e31740 20 client.26852 trim_cache size 0 max 16384 2024-03-21T10:43:30.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.468+0000 7f02b9e31740 1 Processor -- start 2024-03-21T10:43:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.468+0000 7f02b9e31740 1 -- start start 2024-03-21T10:43:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.468+0000 7f02b9e31740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a202aafd0 0x560a202b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.468+0000 7f02b9e31740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a202ab8f0 0x560a202b2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.468+0000 7f02b9e31740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a202b46c0 0x560a202ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.468+0000 7f02b9e31740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b01b74e0 con 0x560a202b46c0 2024-03-21T10:43:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.468+0000 7f02b9e31740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b01b4fe0 con 0x560a202ab8f0 2024-03-21T10:43:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.469+0000 7f02b9e31740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b01b3080 con 0x560a202aafd0 2024-03-21T10:43:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.469+0000 7f02af400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a202ab8f0 0x560a202b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.469+0000 7f02af400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a202ab8f0 0x560a202b2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58128/0 (socket says 172.21.15.31:58128) 2024-03-21T10:43:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.469+0000 7f02af400640 1 -- 172.21.15.31:0/1415137533 learned_addr learned my addr 172.21.15.31:0/1415137533 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.469+0000 7f02af400640 1 -- 172.21.15.31:0/1415137533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a202aafd0 msgr2=0x560a202b3e90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:43:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.469+0000 7f02af400640 1 --2- 172.21.15.31:0/1415137533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a202aafd0 0x560a202b3e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.469+0000 7f02af400640 1 -- 172.21.15.31:0/1415137533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a202b46c0 msgr2=0x560a202ae9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:43:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.469+0000 7f02b5200640 1 --2- 172.21.15.31:0/1415137533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a202b46c0 0x560a202ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.469+0000 7f02af400640 1 --2- 172.21.15.31:0/1415137533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a202b46c0 0x560a202ae9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.469+0000 7f02af400640 1 -- 172.21.15.31:0/1415137533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a202cc0c0 con 0x560a202ab8f0 2024-03-21T10:43:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.470+0000 7f02af400640 1 --2- 172.21.15.31:0/1415137533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a202ab8f0 0x560a202b2000 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f02a009be60 tx=0x7f02a00a49b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=becbc845ee668722 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.470+0000 7f02a4c00640 1 -- 172.21.15.31:0/1415137533 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a00b9070 con 0x560a202ab8f0 2024-03-21T10:43:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.470+0000 7f02a4c00640 1 -- 172.21.15.31:0/1415137533 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f02a009b630 con 0x560a202ab8f0 2024-03-21T10:43:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.470+0000 7f02a4c00640 1 -- 172.21.15.31:0/1415137533 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a00a1100 con 0x560a202ab8f0 2024-03-21T10:43:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.471+0000 7f02b9e31740 1 -- 172.21.15.31:0/1415137533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a202ab8f0 msgr2=0x560a202b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.471+0000 7f02b9e31740 1 --2- 172.21.15.31:0/1415137533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a202ab8f0 0x560a202b2000 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f02a009be60 tx=0x7f02a00a49b0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.471+0000 7f02b9e31740 1 -- 172.21.15.31:0/1415137533 shutdown_connections 2024-03-21T10:43:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.471+0000 7f02b9e31740 1 --2- 172.21.15.31:0/1415137533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a202b46c0 0x560a202ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.471+0000 7f02b9e31740 1 --2- 172.21.15.31:0/1415137533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a202ab8f0 0x560a202b2000 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.471+0000 7f02b9e31740 1 --2- 172.21.15.31:0/1415137533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a202aafd0 0x560a202b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.471+0000 7f02b9e31740 1 -- 172.21.15.31:0/1415137533 >> 172.21.15.31:0/1415137533 conn(0x560a202a58f0 msgr2=0x560a202b0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.471+0000 7f02b9e31740 1 -- 172.21.15.31:0/1415137533 shutdown_connections 2024-03-21T10:43:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.471+0000 7f02b9e31740 1 -- 172.21.15.31:0/1415137533 wait complete. 2024-03-21T10:43:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.472+0000 7f02b9e31740 1 -- start start 2024-03-21T10:43:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.472+0000 7f02b9e31740 1 Processor -- start 2024-03-21T10:43:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.472+0000 7f02b9e31740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a202aafd0 0x560a20318500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.472+0000 7f02b9e31740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a202ab8f0 0x560a20318a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.472+0000 7f02b9e31740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a202b46c0 0x560a202c79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.472+0000 7f02b9e31740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b01b3680 con 0x560a202aafd0 2024-03-21T10:43:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.472+0000 7f02b9e31740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b01b4980 con 0x560a202b46c0 2024-03-21T10:43:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.472+0000 7f02b9e31740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b01b3cc0 con 0x560a202ab8f0 2024-03-21T10:43:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.472+0000 7f02b9e31740 10 client.0 _ll_register_callbacks cb 0x7f02a009b830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:43:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.472+0000 7f02afe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a202aafd0 0x560a20318500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.472+0000 7f02afe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a202aafd0 0x560a20318500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49010/0 (socket says 172.21.15.31:49010) 2024-03-21T10:43:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.473+0000 7f02afe00640 1 -- 172.21.15.31:0/132153626 learned_addr learned my addr 172.21.15.31:0/132153626 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.473+0000 7f02b5200640 1 --2- 172.21.15.31:0/132153626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a202b46c0 0x560a202c79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.473+0000 7f02afe00640 1 -- 172.21.15.31:0/132153626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a202ab8f0 msgr2=0x560a20318a40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:43:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.473+0000 7f02afe00640 1 --2- 172.21.15.31:0/132153626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a202ab8f0 0x560a20318a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.473+0000 7f02afe00640 1 -- 172.21.15.31:0/132153626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a202b46c0 msgr2=0x560a202c79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.473+0000 7f02afe00640 1 --2- 172.21.15.31:0/132153626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a202b46c0 0x560a202c79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.473+0000 7f02afe00640 1 -- 172.21.15.31:0/132153626 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560a202c7ef0 con 0x560a202aafd0 2024-03-21T10:43:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.473+0000 7f02afe00640 1 --2- 172.21.15.31:0/132153626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a202aafd0 0x560a20318500 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f029c02c270 tx=0x7f029c02bc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b0f54354a8625c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.473+0000 7f029a000640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:43:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.474+0000 7f029a000640 1 -- 172.21.15.31:0/132153626 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029c0522d0 con 0x560a202aafd0 2024-03-21T10:43:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.474+0000 7f02b9e31740 1 -- 172.21.15.31:0/132153626 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560a202170c0 con 0x560a202aafd0 2024-03-21T10:43:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.474+0000 7f029a000640 1 -- 172.21.15.31:0/132153626 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f029c0046c0 con 0x560a202aafd0 2024-03-21T10:43:30.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.474+0000 7f029a000640 1 -- 172.21.15.31:0/132153626 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f029c062050 con 0x560a202aafd0 2024-03-21T10:43:30.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.474+0000 7f02b9e31740 10 client.16419 fetch_fsmap learned FSMap version 26 2024-03-21T10:43:30.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.474+0000 7f02b9e31740 1 -- 172.21.15.31:0/132153626 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x560a20331260 con 0x560a202aafd0 2024-03-21T10:43:30.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.474+0000 7f029a000640 1 -- 172.21.15.31:0/132153626 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x560a202170c0 con 0x560a202aafd0 2024-03-21T10:43:30.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.475+0000 7f029a000640 1 -- 172.21.15.31:0/132153626 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f029c0054f0 con 0x560a202aafd0 2024-03-21T10:43:30.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.475+0000 7f02b9e31740 10 client.16419 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:43:30.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.475+0000 7f02b9e31740 10 client.16419 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:43:30.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.475+0000 7f02b9e31740 20 client.16419 populate_metadata read hostname 'smithi031' 2024-03-21T10:43:30.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.475+0000 7f02b9e31740 1 --2- 172.21.15.31:0/132153626 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560a202c44c0 0x560a20333980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.475+0000 7f02b9e31740 4 client.16419 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:43:30.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.475+0000 7f02b9e31740 1 -- 172.21.15.31:0/132153626 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560a201955d0 con 0x560a202c44c0 2024-03-21T10:43:30.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.475+0000 7f02af400640 1 --2- 172.21.15.31:0/132153626 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560a202c44c0 0x560a20333980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.476+0000 7f02af400640 1 --2- 172.21.15.31:0/132153626 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560a202c44c0 0x560a20333980 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e25f1de91bd09ab0 server_cookie=76b3679a1de18284 in_seq=0 out_seq=0 2024-03-21T10:43:30.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.476+0000 7f029a000640 10 client.16419 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:30.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.491+0000 7f029a000640 1 -- 172.21.15.31:0/132153626 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560a201955d0 con 0x560a202c44c0 2024-03-21T10:43:30.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.492+0000 7f029a000640 10 client.16419 handle_command_reply: tid=0 2024-03-21T10:43:30.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.492+0000 7f02b9e31740 1 client.16419 shutdown 2024-03-21T10:43:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.492+0000 7f02b9e31740 1 -- 172.21.15.31:0/132153626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a202aafd0 msgr2=0x560a20318500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.492+0000 7f02b9e31740 1 --2- 172.21.15.31:0/132153626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a202aafd0 0x560a20318500 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f029c02c270 tx=0x7f029c02bc60 comp rx=0 tx=0).stop 2024-03-21T10:43:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.492+0000 7f02b9e31740 1 -- 172.21.15.31:0/132153626 shutdown_connections 2024-03-21T10:43:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.492+0000 7f02b9e31740 1 --2- 172.21.15.31:0/132153626 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560a202c44c0 0x560a20333980 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.492+0000 7f02b9e31740 1 --2- 172.21.15.31:0/132153626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a202b46c0 0x560a202c79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.492+0000 7f02b9e31740 1 --2- 172.21.15.31:0/132153626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560a202ab8f0 0x560a20318a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.492+0000 7f02b9e31740 1 --2- 172.21.15.31:0/132153626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a202aafd0 0x560a20318500 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.492+0000 7f02b9e31740 1 -- 172.21.15.31:0/132153626 >> 172.21.15.31:0/132153626 conn(0x560a202a58f0 msgr2=0x560a202bf4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.492+0000 7f029a000640 0 client.16419 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.492+0000 7f02b9e31740 1 -- 172.21.15.31:0/132153626 shutdown_connections 2024-03-21T10:43:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.492+0000 7f02b9e31740 1 -- 172.21.15.31:0/132153626 wait complete. 2024-03-21T10:43:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.492+0000 7f02b9e31740 20 client.16419 trim_cache size 0 max 16384 2024-03-21T10:43:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:43:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:43:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:43:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:43:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.493+0000 7f02b7200640 1 -- 172.21.15.31:0/4193800009 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f02940a3080 msgr2=0x7f02940a5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.493+0000 7f02b7200640 1 --2- 172.21.15.31:0/4193800009 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f02940a3080 0x7f02940a5540 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f029c000c00 tx=0x7f029c005a20 comp rx=0 tx=0).stop 2024-03-21T10:43:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.493+0000 7f02b7200640 1 -- 172.21.15.31:0/4193800009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02b0104090 msgr2=0x7f02b019e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.493+0000 7f02b7200640 1 --2- 172.21.15.31:0/4193800009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02b0104090 0x7f02b019e9e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f02a000e9f0 tx=0x7f02a000eeb0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.494+0000 7f02b7200640 1 -- 172.21.15.31:0/4193800009 shutdown_connections 2024-03-21T10:43:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.494+0000 7f02b7200640 1 --2- 172.21.15.31:0/4193800009 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f02940a3080 0x7f02940a5540 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.494+0000 7f02b7200640 1 --2- 172.21.15.31:0/4193800009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02b0106590 0x7f02b01a2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.494+0000 7f02b7200640 1 --2- 172.21.15.31:0/4193800009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02b0104090 0x7f02b019e9e0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.494+0000 7f02b7200640 1 --2- 172.21.15.31:0/4193800009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02b0069480 0x7f02b019e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.494+0000 7f02b7200640 1 -- 172.21.15.31:0/4193800009 >> 172.21.15.31:0/4193800009 conn(0x7f02b0077690 msgr2=0x7f02b0103070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.494+0000 7f02b7200640 1 -- 172.21.15.31:0/4193800009 shutdown_connections 2024-03-21T10:43:30.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:30.494+0000 7f02b7200640 1 -- 172.21.15.31:0/4193800009 wait complete. 2024-03-21T10:43:30.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.522+0000 7f739a600640 1 -- 172.21.15.31:0/4097315729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f739410fe30 msgr2=0x7f73941086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:30.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.522+0000 7f739a600640 1 --2- 172.21.15.31:0/4097315729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f739410fe30 0x7f73941086a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f7384009900 tx=0x7f738402bc30 comp rx=0 tx=0).stop 2024-03-21T10:43:30.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.523+0000 7f739a600640 1 -- 172.21.15.31:0/4097315729 shutdown_connections 2024-03-21T10:43:30.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.523+0000 7f739a600640 1 --2- 172.21.15.31:0/4097315729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f739411ae80 0x7f739411d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.523+0000 7f739a600640 1 --2- 172.21.15.31:0/4097315729 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7394108be0 0x7f7394109040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.523+0000 7f739a600640 1 --2- 172.21.15.31:0/4097315729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f739410fe30 0x7f73941086a0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.523+0000 7f739a600640 1 -- 172.21.15.31:0/4097315729 >> 172.21.15.31:0/4097315729 conn(0x7f739406c710 msgr2=0x7f739406cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:30.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.525+0000 7f739a600640 1 -- 172.21.15.31:0/4097315729 shutdown_connections 2024-03-21T10:43:30.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.529+0000 7f739a600640 1 -- 172.21.15.31:0/4097315729 wait complete. 2024-03-21T10:43:30.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.530+0000 7f739a600640 1 Processor -- start 2024-03-21T10:43:30.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.532+0000 7f739a600640 1 -- start start 2024-03-21T10:43:30.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.535+0000 7f739a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7394108be0 0x7f73941ac950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.535+0000 7f739a600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f739410fe30 0x7f73941ace90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.535+0000 7f739a600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f739411ae80 0x7f73941a6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.535+0000 7f739a600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73941208e0 con 0x7f739411ae80 2024-03-21T10:43:30.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.535+0000 7f739a600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7394120760 con 0x7f7394108be0 2024-03-21T10:43:30.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.535+0000 7f739a600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7394120a60 con 0x7f739410fe30 2024-03-21T10:43:30.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.535+0000 7f7392a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f739410fe30 0x7f73941ace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.535+0000 7f7392a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f739410fe30 0x7f73941ace90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42184/0 (socket says 172.21.15.31:42184) 2024-03-21T10:43:30.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.536+0000 7f7392a00640 1 -- 172.21.15.31:0/454713804 learned_addr learned my addr 172.21.15.31:0/454713804 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:30.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.536+0000 7f7393400640 1 --2- 172.21.15.31:0/454713804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7394108be0 0x7f73941ac950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.536+0000 7f7393e00640 1 --2- 172.21.15.31:0/454713804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f739411ae80 0x7f73941a6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.536+0000 7f7392a00640 1 -- 172.21.15.31:0/454713804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7394108be0 msgr2=0x7f73941ac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:30.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.536+0000 7f7392a00640 1 --2- 172.21.15.31:0/454713804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7394108be0 0x7f73941ac950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.536+0000 7f7392a00640 1 -- 172.21.15.31:0/454713804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f739411ae80 msgr2=0x7f73941a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:30.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.536+0000 7f7392a00640 1 --2- 172.21.15.31:0/454713804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f739411ae80 0x7f73941a6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.536+0000 7f7392a00640 1 -- 172.21.15.31:0/454713804 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73941a7280 con 0x7f739410fe30 2024-03-21T10:43:30.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.537+0000 7f7393e00640 1 --2- 172.21.15.31:0/454713804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f739411ae80 0x7f73941a6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:43:30.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.537+0000 7f7392a00640 1 --2- 172.21.15.31:0/454713804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f739410fe30 0x7f73941ace90 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f738000c920 tx=0x7f738000cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:30.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.539+0000 7f738be00640 1 -- 172.21.15.31:0/454713804 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7380007c70 con 0x7f739410fe30 2024-03-21T10:43:30.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.539+0000 7f739a600640 1 -- 172.21.15.31:0/454713804 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73941a7570 con 0x7f739410fe30 2024-03-21T10:43:30.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.539+0000 7f738be00640 1 -- 172.21.15.31:0/454713804 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7380007e10 con 0x7f739410fe30 2024-03-21T10:43:30.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.539+0000 7f739a600640 1 -- 172.21.15.31:0/454713804 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7394110660 con 0x7f739410fe30 2024-03-21T10:43:30.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.539+0000 7f738be00640 1 -- 172.21.15.31:0/454713804 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7380018890 con 0x7f739410fe30 2024-03-21T10:43:30.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.548+0000 7f738be00640 1 -- 172.21.15.31:0/454713804 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f7380018b30 con 0x7f739410fe30 2024-03-21T10:43:30.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.559+0000 7f738be00640 1 --2- 172.21.15.31:0/454713804 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f73780a31d0 0x7f73780a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.559+0000 7f738be00640 1 -- 172.21.15.31:0/454713804 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f73800cfc80 con 0x7f739410fe30 2024-03-21T10:43:30.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.560+0000 7f739d230740 1 Processor -- start 2024-03-21T10:43:30.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.560+0000 7f739d230740 1 -- start start 2024-03-21T10:43:30.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.561+0000 7f7393400640 1 --2- 172.21.15.31:0/454713804 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f73780a31d0 0x7f73780a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.565+0000 7f739d230740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eee39306d0 0x55eee3930ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.565+0000 7f739d230740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eee3931080 0x55eee3939630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.565+0000 7f739d230740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eee3939c00 0x55eee393c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.565+0000 7f739d230740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eee38942c0 con 0x55eee39306d0 2024-03-21T10:43:30.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.565+0000 7f739d230740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eee383cc90 con 0x55eee3931080 2024-03-21T10:43:30.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.565+0000 7f739d230740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eee3890360 con 0x55eee3939c00 2024-03-21T10:43:30.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.568+0000 7f7393400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eee39306d0 0x55eee3930ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.569+0000 7f7393400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eee39306d0 0x55eee3930ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49034/0 (socket says 172.21.15.31:49034) 2024-03-21T10:43:30.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.569+0000 7f7393400640 1 -- 172.21.15.31:0/3459937025 learned_addr learned my addr 172.21.15.31:0/3459937025 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:30.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.570+0000 7f7393400640 1 --2- 172.21.15.31:0/454713804 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f73780a31d0 0x7f73780a5690 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7384000c00 tx=0x7f738402df90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:30.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.571+0000 7f7393400640 1 -- 172.21.15.31:0/3459937025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eee3939c00 msgr2=0x55eee393c050 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:43:30.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.572+0000 7f7393400640 1 --2- 172.21.15.31:0/3459937025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eee3939c00 0x55eee393c050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.572+0000 7f7393400640 1 -- 172.21.15.31:0/3459937025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eee3931080 msgr2=0x55eee3939630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:43:30.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.572+0000 7f7393400640 1 --2- 172.21.15.31:0/3459937025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eee3931080 0x55eee3939630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.572+0000 7f7393400640 1 -- 172.21.15.31:0/3459937025 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eee3861080 con 0x55eee39306d0 2024-03-21T10:43:30.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.577+0000 7f7393400640 1 --2- 172.21.15.31:0/3459937025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eee39306d0 0x55eee3930ab0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7384009230 tx=0x7f7384047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76ce1f7dd1b45688 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:30.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.579+0000 7f7388c00640 1 -- 172.21.15.31:0/3459937025 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738404f070 con 0x55eee39306d0 2024-03-21T10:43:30.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.579+0000 7f7388c00640 1 -- 172.21.15.31:0/3459937025 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f73840046c0 con 0x55eee39306d0 2024-03-21T10:43:30.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.579+0000 7f7388c00640 1 -- 172.21.15.31:0/3459937025 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7384005580 con 0x55eee39306d0 2024-03-21T10:43:30.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.584+0000 7f739d230740 1 -- 172.21.15.31:0/3459937025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eee39306d0 msgr2=0x55eee3930ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:30.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.584+0000 7f739d230740 1 --2- 172.21.15.31:0/3459937025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eee39306d0 0x55eee3930ab0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7384009230 tx=0x7f7384047000 comp rx=0 tx=0).stop 2024-03-21T10:43:30.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.585+0000 7f739d230740 1 -- 172.21.15.31:0/3459937025 shutdown_connections 2024-03-21T10:43:30.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.585+0000 7f739d230740 1 --2- 172.21.15.31:0/3459937025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eee3939c00 0x55eee393c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.585+0000 7f739d230740 1 --2- 172.21.15.31:0/3459937025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eee3931080 0x55eee3939630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.585+0000 7f739d230740 1 --2- 172.21.15.31:0/3459937025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eee39306d0 0x55eee3930ab0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.585+0000 7f739d230740 1 -- 172.21.15.31:0/3459937025 >> 172.21.15.31:0/3459937025 conn(0x55eee392ada0 msgr2=0x55eee392b180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:30.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.587+0000 7f739d230740 1 -- 172.21.15.31:0/3459937025 shutdown_connections 2024-03-21T10:43:30.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.587+0000 7f739d230740 1 -- 172.21.15.31:0/3459937025 wait complete. 2024-03-21T10:43:30.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.588+0000 7f739d230740 1 -- start start 2024-03-21T10:43:30.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.588+0000 7f739d230740 1 Processor -- start 2024-03-21T10:43:30.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.590+0000 7f739d230740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eee39306d0 0x55eee3939530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.590+0000 7f739d230740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eee3931080 0x55eee39368c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.590+0000 7f739d230740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eee3939c00 0x55eee3936ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.592+0000 7f739d230740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eee393cb10 con 0x55eee3931080 2024-03-21T10:43:30.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.592+0000 7f739d230740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eee38500d0 con 0x55eee39306d0 2024-03-21T10:43:30.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.592+0000 7f739d230740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eee393cc90 con 0x55eee3939c00 2024-03-21T10:43:30.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.592+0000 7f7393e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eee3939c00 0x55eee3936ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.593+0000 7f739d230740 10 client.0 _ll_register_callbacks cb 0x55eee3854e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:43:30.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.593+0000 7f7393e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eee3939c00 0x55eee3936ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42208/0 (socket says 172.21.15.31:42208) 2024-03-21T10:43:30.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.593+0000 7f7393e00640 1 -- 172.21.15.31:0/1503150101 learned_addr learned my addr 172.21.15.31:0/1503150101 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:30.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.594+0000 7f7393400640 1 --2- 172.21.15.31:0/1503150101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eee39306d0 0x55eee3939530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.597+0000 7f7392a00640 1 --2- 172.21.15.31:0/1503150101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eee3931080 0x55eee39368c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.603+0000 7f7392a00640 1 -- 172.21.15.31:0/1503150101 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eee3939c00 msgr2=0x55eee3936ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:30.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.603+0000 7f7392a00640 1 --2- 172.21.15.31:0/1503150101 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eee3939c00 0x55eee3936ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.603+0000 7f7392a00640 1 -- 172.21.15.31:0/1503150101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eee39306d0 msgr2=0x55eee3939530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:30.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.603+0000 7f7392a00640 1 --2- 172.21.15.31:0/1503150101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eee39306d0 0x55eee3939530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.603+0000 7f7392a00640 1 -- 172.21.15.31:0/1503150101 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eee35a83f0 con 0x55eee3931080 2024-03-21T10:43:30.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.603+0000 7f7393e00640 1 --2- 172.21.15.31:0/1503150101 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eee3939c00 0x55eee3936ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:43:30.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.603+0000 7f7393400640 1 --2- 172.21.15.31:0/1503150101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eee39306d0 0x55eee3939530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:43:30.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.605+0000 7f7392a00640 1 --2- 172.21.15.31:0/1503150101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eee3931080 0x55eee39368c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f73800cd030 tx=0x7f7380018e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c9a9a122189c8e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:30.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.606+0000 7f737d600640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:43:30.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.606+0000 7f737d600640 1 -- 172.21.15.31:0/1503150101 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73800cd4b0 con 0x55eee3931080 2024-03-21T10:43:30.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.606+0000 7f737d600640 1 -- 172.21.15.31:0/1503150101 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f73800a4250 con 0x55eee3931080 2024-03-21T10:43:30.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.606+0000 7f739d230740 1 -- 172.21.15.31:0/1503150101 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f73840046c0 con 0x55eee3931080 2024-03-21T10:43:30.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.607+0000 7f737d600640 1 -- 172.21.15.31:0/1503150101 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f738009bcd0 con 0x55eee3931080 2024-03-21T10:43:30.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.607+0000 7f739d230740 10 client.16425 fetch_fsmap learned FSMap version 26 2024-03-21T10:43:30.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.607+0000 7f739d230740 1 -- 172.21.15.31:0/1503150101 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55eee36727d0 con 0x55eee3931080 2024-03-21T10:43:30.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.610+0000 7f737d600640 1 -- 172.21.15.31:0/1503150101 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f73840046c0 con 0x55eee3931080 2024-03-21T10:43:30.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.610+0000 7f737d600640 1 -- 172.21.15.31:0/1503150101 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f738009ab90 con 0x55eee3931080 2024-03-21T10:43:30.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.611+0000 7f739d230740 10 client.16425 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:43:30.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.611+0000 7f739d230740 10 client.16425 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:43:30.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.611+0000 7f739d230740 20 client.16425 populate_metadata read hostname 'smithi031' 2024-03-21T10:43:30.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.611+0000 7f739d230740 1 --2- 172.21.15.31:0/1503150101 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55eee3949a20 0x55eee394be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.611+0000 7f739d230740 4 client.16425 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:43:30.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.611+0000 7f739d230740 1 -- 172.21.15.31:0/1503150101 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eee38a1800 con 0x55eee3949a20 2024-03-21T10:43:30.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.613+0000 7f7393400640 1 --2- 172.21.15.31:0/1503150101 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55eee3949a20 0x55eee394be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.614+0000 7f7393400640 1 --2- 172.21.15.31:0/1503150101 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55eee3949a20 0x55eee394be10 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=161500b8a170f722 server_cookie=8244775f30e26fa0 in_seq=0 out_seq=0 2024-03-21T10:43:30.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.615+0000 7f737d600640 10 client.16425 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:43:30.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.615+0000 7f737d600640 1 -- 172.21.15.31:0/1503150101 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55eee38a1800 con 0x55eee3949a20 2024-03-21T10:43:30.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.615+0000 7f737d600640 10 client.16425 handle_command_reply: tid=0 2024-03-21T10:43:30.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.615+0000 7f739d230740 1 client.16425 shutdown 2024-03-21T10:43:30.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.616+0000 7f739d230740 1 -- 172.21.15.31:0/1503150101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eee3931080 msgr2=0x55eee39368c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:30.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.616+0000 7f739d230740 1 --2- 172.21.15.31:0/1503150101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eee3931080 0x55eee39368c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f73800cd030 tx=0x7f7380018e20 comp rx=0 tx=0).stop 2024-03-21T10:43:30.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.616+0000 7f739d230740 1 -- 172.21.15.31:0/1503150101 shutdown_connections 2024-03-21T10:43:30.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.616+0000 7f739d230740 1 --2- 172.21.15.31:0/1503150101 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55eee3949a20 0x55eee394be10 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.616+0000 7f739d230740 1 --2- 172.21.15.31:0/1503150101 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eee3939c00 0x55eee3936ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.616+0000 7f739d230740 1 --2- 172.21.15.31:0/1503150101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eee3931080 0x55eee39368c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.616+0000 7f739d230740 1 --2- 172.21.15.31:0/1503150101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eee39306d0 0x55eee3939530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.616+0000 7f739d230740 1 -- 172.21.15.31:0/1503150101 >> 172.21.15.31:0/1503150101 conn(0x55eee392ada0 msgr2=0x55eee392caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:30.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.616+0000 7f737d600640 0 client.16425 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:43:30.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.617+0000 7f739d230740 1 -- 172.21.15.31:0/1503150101 shutdown_connections 2024-03-21T10:43:30.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.617+0000 7f739d230740 1 -- 172.21.15.31:0/1503150101 wait complete. 2024-03-21T10:43:30.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.618+0000 7f739d230740 20 client.16425 trim_cache size 0 max 16384 2024-03-21T10:43:30.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.643+0000 7f739d230740 1 Processor -- start 2024-03-21T10:43:30.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.644+0000 7f739d230740 1 -- start start 2024-03-21T10:43:30.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.645+0000 7f739d230740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eee39306d0 0x55eee3938ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.645+0000 7f739d230740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eee3939c00 0x55eee3939530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.645+0000 7f739d230740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eee3951550 0x55eee3933b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.645+0000 7f739d230740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73941b7420 con 0x55eee3939c00 2024-03-21T10:43:30.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.645+0000 7f739d230740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73941b4f20 con 0x55eee39306d0 2024-03-21T10:43:30.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.645+0000 7f7393400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eee39306d0 0x55eee3938ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.645+0000 7f7393400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eee39306d0 0x55eee3938ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58162/0 (socket says 172.21.15.31:58162) 2024-03-21T10:43:30.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.645+0000 7f7393400640 1 -- 172.21.15.31:0/229320710 learned_addr learned my addr 172.21.15.31:0/229320710 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:30.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.645+0000 7f739d230740 1 -- 172.21.15.31:0/229320710 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7394068520 con 0x55eee3951550 2024-03-21T10:43:30.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.645+0000 7f7393e00640 1 --2- 172.21.15.31:0/229320710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eee3951550 0x55eee3933b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.645+0000 7f7392a00640 1 --2- 172.21.15.31:0/229320710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eee3939c00 0x55eee3939530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.646+0000 7f7393400640 1 -- 172.21.15.31:0/229320710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eee3951550 msgr2=0x55eee3933b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:30.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.646+0000 7f7393400640 1 --2- 172.21.15.31:0/229320710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eee3951550 0x55eee3933b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.646+0000 7f7393400640 1 -- 172.21.15.31:0/229320710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eee3939c00 msgr2=0x55eee3939530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:30.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.646+0000 7f7393400640 1 --2- 172.21.15.31:0/229320710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eee3939c00 0x55eee3939530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.646+0000 7f7393400640 1 -- 172.21.15.31:0/229320710 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eee393ca50 con 0x55eee39306d0 2024-03-21T10:43:30.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.646+0000 7f7393400640 1 --2- 172.21.15.31:0/229320710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eee39306d0 0x55eee3938ff0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f7384009900 tx=0x7f738402bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8057d8528d190e8a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:30.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.646+0000 7f737fe00640 1 -- 172.21.15.31:0/229320710 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738404f040 con 0x55eee39306d0 2024-03-21T10:43:30.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.646+0000 7f737fe00640 1 -- 172.21.15.31:0/229320710 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f738402cd20 con 0x55eee39306d0 2024-03-21T10:43:30.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.647+0000 7f739d230740 1 -- 172.21.15.31:0/229320710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eee39306d0 msgr2=0x55eee3938ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:30.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.647+0000 7f739d230740 1 --2- 172.21.15.31:0/229320710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eee39306d0 0x55eee3938ff0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f7384009900 tx=0x7f738402bc30 comp rx=0 tx=0).stop 2024-03-21T10:43:30.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.647+0000 7f739d230740 1 -- 172.21.15.31:0/229320710 shutdown_connections 2024-03-21T10:43:30.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.647+0000 7f739d230740 1 --2- 172.21.15.31:0/229320710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eee3951550 0x55eee3933b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.647+0000 7f739d230740 1 --2- 172.21.15.31:0/229320710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eee3939c00 0x55eee3939530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.647+0000 7f739d230740 1 --2- 172.21.15.31:0/229320710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eee39306d0 0x55eee3938ff0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.647+0000 7f739d230740 1 -- 172.21.15.31:0/229320710 >> 172.21.15.31:0/229320710 conn(0x55eee392ada0 msgr2=0x55eee3929eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:30.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.648+0000 7f739d230740 1 -- 172.21.15.31:0/229320710 shutdown_connections 2024-03-21T10:43:30.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.648+0000 7f739d230740 1 -- 172.21.15.31:0/229320710 wait complete. 2024-03-21T10:43:30.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.648+0000 7f739d230740 1 -- start start 2024-03-21T10:43:30.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.649+0000 7f739d230740 1 Processor -- start 2024-03-21T10:43:30.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.654+0000 7f739d230740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eee39306d0 0x55eee39404d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.654+0000 7f739d230740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eee3939c00 0x55eee3940a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.654+0000 7f739d230740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eee3951550 0x55eee3940f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.654+0000 7f739d230740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7394118c90 con 0x55eee3939c00 2024-03-21T10:43:30.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.654+0000 7f739d230740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73941b48c0 con 0x55eee39306d0 2024-03-21T10:43:30.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.654+0000 7f739d230740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73941b3c00 con 0x55eee3951550 2024-03-21T10:43:30.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.654+0000 7f739d230740 10 client.0 _ll_register_callbacks cb 0x7f738000fe30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:43:30.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.655+0000 7f7393400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eee39306d0 0x55eee39404d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.655+0000 7f7393400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eee39306d0 0x55eee39404d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58164/0 (socket says 172.21.15.31:58164) 2024-03-21T10:43:30.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.655+0000 7f7393400640 1 -- 172.21.15.31:0/2978708579 learned_addr learned my addr 172.21.15.31:0/2978708579 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:30.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.655+0000 7f7392a00640 1 --2- 172.21.15.31:0/2978708579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eee3939c00 0x55eee3940a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.655+0000 7f7393400640 1 -- 172.21.15.31:0/2978708579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eee3951550 msgr2=0x55eee3940f50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:43:30.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.655+0000 7f7393400640 1 --2- 172.21.15.31:0/2978708579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eee3951550 0x55eee3940f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.655+0000 7f7393400640 1 -- 172.21.15.31:0/2978708579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eee3939c00 msgr2=0x55eee3940a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:30.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.655+0000 7f7393400640 1 --2- 172.21.15.31:0/2978708579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eee3939c00 0x55eee3940a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.656+0000 7f7393400640 1 -- 172.21.15.31:0/2978708579 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eee39a52b0 con 0x55eee39306d0 2024-03-21T10:43:30.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.656+0000 7f7393400640 1 --2- 172.21.15.31:0/2978708579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eee39306d0 0x55eee39404d0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f738402c820 tx=0x7f7384009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2dd257dae1c3289b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:30.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.656+0000 7f737d600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:43:30.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.656+0000 7f737d600640 1 -- 172.21.15.31:0/2978708579 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738404f070 con 0x55eee39306d0 2024-03-21T10:43:30.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.657+0000 7f737d600640 1 -- 172.21.15.31:0/2978708579 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f7384005220 con 0x55eee39306d0 2024-03-21T10:43:30.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.657+0000 7f739d230740 1 -- 172.21.15.31:0/2978708579 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55eee389bb50 con 0x55eee39306d0 2024-03-21T10:43:30.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.657+0000 7f737d600640 1 -- 172.21.15.31:0/2978708579 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7384005420 con 0x55eee39306d0 2024-03-21T10:43:30.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.657+0000 7f739d230740 10 client.26873 fetch_fsmap learned FSMap version 26 2024-03-21T10:43:30.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.657+0000 7f739d230740 1 -- 172.21.15.31:0/2978708579 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55eee39a6590 con 0x55eee39306d0 2024-03-21T10:43:30.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.658+0000 7f737d600640 1 -- 172.21.15.31:0/2978708579 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55eee389bb50 con 0x55eee39306d0 2024-03-21T10:43:30.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.659+0000 7f737d600640 1 -- 172.21.15.31:0/2978708579 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f7384054450 con 0x55eee39306d0 2024-03-21T10:43:30.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.659+0000 7f739d230740 10 client.26873 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:43:30.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.659+0000 7f739d230740 10 client.26873 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:43:30.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.659+0000 7f739d230740 20 client.26873 populate_metadata read hostname 'smithi031' 2024-03-21T10:43:30.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.659+0000 7f739d230740 1 --2- 172.21.15.31:0/2978708579 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55eee39a6810 0x55eee39a8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:30.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.659+0000 7f739d230740 4 client.26873 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:43:30.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.659+0000 7f739d230740 1 -- 172.21.15.31:0/2978708579 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55eee3819500 con 0x55eee39a6810 2024-03-21T10:43:30.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.659+0000 7f7392a00640 1 --2- 172.21.15.31:0/2978708579 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55eee39a6810 0x55eee39a8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:30.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.660+0000 7f7392a00640 1 --2- 172.21.15.31:0/2978708579 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55eee39a6810 0x55eee39a8c00 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=83c4c051c54b9ec3 server_cookie=d61727cdbc1357a4 in_seq=0 out_seq=0 2024-03-21T10:43:30.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.660+0000 7f737d600640 10 client.26873 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:43:30.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.660+0000 7f737d600640 1 -- 172.21.15.31:0/2978708579 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55eee3819500 con 0x55eee39a6810 2024-03-21T10:43:30.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.660+0000 7f737d600640 10 client.26873 handle_command_reply: tid=0 2024-03-21T10:43:30.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.660+0000 7f739d230740 1 client.26873 shutdown 2024-03-21T10:43:30.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.661+0000 7f739d230740 1 -- 172.21.15.31:0/2978708579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eee39306d0 msgr2=0x55eee39404d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:30.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.661+0000 7f739d230740 1 --2- 172.21.15.31:0/2978708579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eee39306d0 0x55eee39404d0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f738402c820 tx=0x7f7384009260 comp rx=0 tx=0).stop 2024-03-21T10:43:30.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.661+0000 7f739d230740 1 -- 172.21.15.31:0/2978708579 shutdown_connections 2024-03-21T10:43:30.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.661+0000 7f739d230740 1 --2- 172.21.15.31:0/2978708579 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55eee39a6810 0x55eee39a8c00 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.661+0000 7f739d230740 1 --2- 172.21.15.31:0/2978708579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eee3951550 0x55eee3940f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.661+0000 7f739d230740 1 --2- 172.21.15.31:0/2978708579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eee3939c00 0x55eee3940a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.661+0000 7f739d230740 1 --2- 172.21.15.31:0/2978708579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eee39306d0 0x55eee39404d0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.661+0000 7f739d230740 1 -- 172.21.15.31:0/2978708579 >> 172.21.15.31:0/2978708579 conn(0x55eee392ada0 msgr2=0x55eee392c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:30.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.661+0000 7f737d600640 0 client.26873 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:43:30.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.662+0000 7f739d230740 1 -- 172.21.15.31:0/2978708579 shutdown_connections 2024-03-21T10:43:30.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.662+0000 7f739d230740 1 -- 172.21.15.31:0/2978708579 wait complete. 2024-03-21T10:43:30.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.662+0000 7f739d230740 20 client.26873 trim_cache size 0 max 16384 2024-03-21T10:43:30.660 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:43:30.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.663+0000 7f739a600640 1 -- 172.21.15.31:0/454713804 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f73780a31d0 msgr2=0x7f73780a5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:30.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.663+0000 7f739a600640 1 --2- 172.21.15.31:0/454713804 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f73780a31d0 0x7f73780a5690 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7384000c00 tx=0x7f738402df90 comp rx=0 tx=0).stop 2024-03-21T10:43:30.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.663+0000 7f739a600640 1 -- 172.21.15.31:0/454713804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f739410fe30 msgr2=0x7f73941ace90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:30.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.663+0000 7f739a600640 1 --2- 172.21.15.31:0/454713804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f739410fe30 0x7f73941ace90 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f738000c920 tx=0x7f738000cde0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.664+0000 7f739a600640 1 -- 172.21.15.31:0/454713804 shutdown_connections 2024-03-21T10:43:30.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.664+0000 7f739a600640 1 --2- 172.21.15.31:0/454713804 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f73780a31d0 0x7f73780a5690 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.664+0000 7f739a600640 1 --2- 172.21.15.31:0/454713804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f739411ae80 0x7f73941a6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.664+0000 7f739a600640 1 --2- 172.21.15.31:0/454713804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f739410fe30 0x7f73941ace90 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.664+0000 7f739a600640 1 --2- 172.21.15.31:0/454713804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7394108be0 0x7f73941ac950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:30.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.664+0000 7f739a600640 1 -- 172.21.15.31:0/454713804 >> 172.21.15.31:0/454713804 conn(0x7f739406c710 msgr2=0x7f7394118e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:30.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.664+0000 7f739a600640 1 -- 172.21.15.31:0/454713804 shutdown_connections 2024-03-21T10:43:30.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:30.664+0000 7f739a600640 1 -- 172.21.15.31:0/454713804 wait complete. 2024-03-21T10:43:30.684 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:43:30.684 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-21T10:43:31.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.010+0000 7ff52be00640 1 -- 172.21.15.31:0/1739663701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff52409acf0 msgr2=0x7ff524004cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:31.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.010+0000 7ff52be00640 1 --2- 172.21.15.31:0/1739663701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff52409acf0 0x7ff524004cb0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff51c009900 tx=0x7ff51c02bc30 comp rx=0 tx=0).stop 2024-03-21T10:43:31.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.011+0000 7ff52be00640 1 -- 172.21.15.31:0/1739663701 shutdown_connections 2024-03-21T10:43:31.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.011+0000 7ff52be00640 1 --2- 172.21.15.31:0/1739663701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5240986e0 0x7ff524098b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.011+0000 7ff52be00640 1 --2- 172.21.15.31:0/1739663701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5240051f0 0x7ff524005670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.011+0000 7ff52be00640 1 --2- 172.21.15.31:0/1739663701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff52409acf0 0x7ff524004cb0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.011+0000 7ff52be00640 1 -- 172.21.15.31:0/1739663701 >> 172.21.15.31:0/1739663701 conn(0x7ff52400a740 msgr2=0x7ff52400ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:31.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.011+0000 7ff52be00640 1 -- 172.21.15.31:0/1739663701 shutdown_connections 2024-03-21T10:43:31.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.011+0000 7ff52be00640 1 -- 172.21.15.31:0/1739663701 wait complete. 2024-03-21T10:43:31.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.012+0000 7ff52be00640 1 Processor -- start 2024-03-21T10:43:31.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.013+0000 7ff52be00640 1 -- start start 2024-03-21T10:43:31.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.013+0000 7ff52be00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5240051f0 0x7ff5241387a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.013+0000 7ff52be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5240986e0 0x7ff524138ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.013+0000 7ff52be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff52409acf0 0x7ff524132950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.013+0000 7ff52be00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5240ac5b0 con 0x7ff5240986e0 2024-03-21T10:43:31.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.013+0000 7ff52be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5240ac430 con 0x7ff52409acf0 2024-03-21T10:43:31.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.013+0000 7ff52be00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5240ac730 con 0x7ff5240051f0 2024-03-21T10:43:31.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.014+0000 7ff52b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff52409acf0 0x7ff524132950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.014+0000 7ff52b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff52409acf0 0x7ff524132950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58182/0 (socket says 172.21.15.31:58182) 2024-03-21T10:43:31.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.014+0000 7ff52b400640 1 -- 172.21.15.31:0/1560435586 learned_addr learned my addr 172.21.15.31:0/1560435586 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:31.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.014+0000 7ff52b400640 1 -- 172.21.15.31:0/1560435586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5240051f0 msgr2=0x7ff5241387a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-21T10:43:31.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.014+0000 7ff52b400640 1 --2- 172.21.15.31:0/1560435586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5240051f0 0x7ff5241387a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.014+0000 7ff52b400640 1 -- 172.21.15.31:0/1560435586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5240986e0 msgr2=0x7ff524138ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T10:43:31.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.014+0000 7ff52a000640 1 --2- 172.21.15.31:0/1560435586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5240986e0 0x7ff524138ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.015+0000 7ff52b400640 1 --2- 172.21.15.31:0/1560435586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5240986e0 0x7ff524138ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.015+0000 7ff52b400640 1 -- 172.21.15.31:0/1560435586 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5241330d0 con 0x7ff52409acf0 2024-03-21T10:43:31.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.015+0000 7ff52b400640 1 --2- 172.21.15.31:0/1560435586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff52409acf0 0x7ff524132950 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff52c050d00 tx=0x7ff52c07aee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:31.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.015+0000 7ff52a000640 1 --2- 172.21.15.31:0/1560435586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5240986e0 0x7ff524138ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:43:31.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.015+0000 7ff530a00640 1 -- 172.21.15.31:0/1560435586 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c078b40 con 0x7ff52409acf0 2024-03-21T10:43:31.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.015+0000 7ff52be00640 1 -- 172.21.15.31:0/1560435586 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff524133360 con 0x7ff52409acf0 2024-03-21T10:43:31.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.016+0000 7ff52be00640 1 -- 172.21.15.31:0/1560435586 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff52413f660 con 0x7ff52409acf0 2024-03-21T10:43:31.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.017+0000 7ff530a00640 1 -- 172.21.15.31:0/1560435586 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff52c085070 con 0x7ff52409acf0 2024-03-21T10:43:31.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.017+0000 7ff530a00640 1 -- 172.21.15.31:0/1560435586 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c069330 con 0x7ff52409acf0 2024-03-21T10:43:31.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.021+0000 7ff534a2a740 1 Processor -- start 2024-03-21T10:43:31.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.022+0000 7ff534a2a740 1 -- start start 2024-03-21T10:43:31.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.022+0000 7ff530a00640 1 -- 172.21.15.31:0/1560435586 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7ff52c07e050 con 0x7ff52409acf0 2024-03-21T10:43:31.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.023+0000 7ff534a2a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daa77a8b0 0x563daa77ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.023+0000 7ff534a2a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563daa77b260 0x563daa783810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.023+0000 7ff534a2a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563daa783de0 0x563daa786230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.023+0000 7ff534a2a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563daa6de2c0 con 0x563daa783de0 2024-03-21T10:43:31.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.023+0000 7ff534a2a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x563daa686c90 con 0x563daa77a8b0 2024-03-21T10:43:31.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.023+0000 7ff534a2a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x563daa6da360 con 0x563daa77b260 2024-03-21T10:43:31.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.023+0000 7ff52a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daa77a8b0 0x563daa77ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.024+0000 7ff52a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daa77a8b0 0x563daa77ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58196/0 (socket says 172.21.15.31:58196) 2024-03-21T10:43:31.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.024+0000 7ff52a000640 1 -- 172.21.15.31:0/485377730 learned_addr learned my addr 172.21.15.31:0/485377730 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:31.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.024+0000 7ff52aa00640 1 --2- 172.21.15.31:0/485377730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563daa77b260 0x563daa783810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.024+0000 7ff52aa00640 1 -- 172.21.15.31:0/485377730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daa77a8b0 msgr2=0x563daa77ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:31.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.024+0000 7ff52a000640 1 --2- 172.21.15.31:0/485377730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563daa783de0 0x563daa786230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.024+0000 7ff52aa00640 1 --2- 172.21.15.31:0/485377730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daa77a8b0 0x563daa77ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.024+0000 7ff52aa00640 1 -- 172.21.15.31:0/485377730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563daa783de0 msgr2=0x563daa786230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:31.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.024+0000 7ff52aa00640 1 --2- 172.21.15.31:0/485377730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563daa783de0 0x563daa786230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.024+0000 7ff52aa00640 1 -- 172.21.15.31:0/485377730 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563daa6ab080 con 0x563daa77b260 2024-03-21T10:43:31.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.025+0000 7ff52aa00640 1 --2- 172.21.15.31:0/485377730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563daa77b260 0x563daa783810 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff51c036040 tx=0x7ff51c005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=797f7f9bdd297bf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:31.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.025+0000 7ff530a00640 1 --2- 172.21.15.31:0/1560435586 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff5100a3190 0x7ff5100a5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.026+0000 7ff517e00640 1 -- 172.21.15.31:0/485377730 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c041020 con 0x563daa77b260 2024-03-21T10:43:31.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.026+0000 7ff517e00640 1 -- 172.21.15.31:0/485377730 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff51c02c690 con 0x563daa77b260 2024-03-21T10:43:31.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.026+0000 7ff517e00640 1 -- 172.21.15.31:0/485377730 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c0046c0 con 0x563daa77b260 2024-03-21T10:43:31.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.026+0000 7ff530a00640 1 -- 172.21.15.31:0/1560435586 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7ff52c13b410 con 0x7ff52409acf0 2024-03-21T10:43:31.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.027+0000 7ff52a000640 1 --2- 172.21.15.31:0/1560435586 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff5100a3190 0x7ff5100a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.027+0000 7ff534a2a740 1 -- 172.21.15.31:0/485377730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563daa77b260 msgr2=0x563daa783810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:31.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.027+0000 7ff534a2a740 1 --2- 172.21.15.31:0/485377730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563daa77b260 0x563daa783810 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff51c036040 tx=0x7ff51c005e80 comp rx=0 tx=0).stop 2024-03-21T10:43:31.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.027+0000 7ff534a2a740 1 -- 172.21.15.31:0/485377730 shutdown_connections 2024-03-21T10:43:31.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.027+0000 7ff534a2a740 1 --2- 172.21.15.31:0/485377730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563daa783de0 0x563daa786230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.027+0000 7ff534a2a740 1 --2- 172.21.15.31:0/485377730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563daa77b260 0x563daa783810 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.027+0000 7ff534a2a740 1 --2- 172.21.15.31:0/485377730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daa77a8b0 0x563daa77ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.027+0000 7ff534a2a740 1 -- 172.21.15.31:0/485377730 >> 172.21.15.31:0/485377730 conn(0x563daa774f80 msgr2=0x563daa775360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:31.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.027+0000 7ff534a2a740 1 -- 172.21.15.31:0/485377730 shutdown_connections 2024-03-21T10:43:31.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.029+0000 7ff534a2a740 1 -- 172.21.15.31:0/485377730 wait complete. 2024-03-21T10:43:31.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.029+0000 7ff52a000640 1 --2- 172.21.15.31:0/1560435586 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff5100a3190 0x7ff5100a5650 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff518005fd0 tx=0x7ff518004730 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:31.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.030+0000 7ff534a2a740 1 -- start start 2024-03-21T10:43:31.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.031+0000 7ff534a2a740 1 Processor -- start 2024-03-21T10:43:31.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.033+0000 7ff534a2a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563daa77a8b0 0x563daa783670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.033+0000 7ff534a2a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563daa77b260 0x563daa780a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.033+0000 7ff534a2a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daa783de0 0x563daa780f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.033+0000 7ff534a2a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563daa786d50 con 0x563daa77a8b0 2024-03-21T10:43:31.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.033+0000 7ff534a2a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x563daa69a0d0 con 0x563daa783de0 2024-03-21T10:43:31.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.033+0000 7ff534a2a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x563daa786ed0 con 0x563daa77b260 2024-03-21T10:43:31.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.033+0000 7ff534a2a740 10 client.0 _ll_register_callbacks cb 0x563daa69ee80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:43:31.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.034+0000 7ff52b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daa783de0 0x563daa780f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.034+0000 7ff52b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daa783de0 0x563daa780f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58200/0 (socket says 172.21.15.31:58200) 2024-03-21T10:43:31.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.034+0000 7ff52b400640 1 -- 172.21.15.31:0/2711189408 learned_addr learned my addr 172.21.15.31:0/2711189408 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:31.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.034+0000 7ff52a000640 1 --2- 172.21.15.31:0/2711189408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563daa77b260 0x563daa780a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.034+0000 7ff52aa00640 1 --2- 172.21.15.31:0/2711189408 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563daa77a8b0 0x563daa783670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.034+0000 7ff52b400640 1 -- 172.21.15.31:0/2711189408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563daa77b260 msgr2=0x563daa780a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:31.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.034+0000 7ff52b400640 1 --2- 172.21.15.31:0/2711189408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563daa77b260 0x563daa780a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.034+0000 7ff52b400640 1 -- 172.21.15.31:0/2711189408 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563daa77a8b0 msgr2=0x563daa783670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:31.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.034+0000 7ff52b400640 1 --2- 172.21.15.31:0/2711189408 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563daa77a8b0 0x563daa783670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.034+0000 7ff52b400640 1 -- 172.21.15.31:0/2711189408 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563daa3f23f0 con 0x563daa783de0 2024-03-21T10:43:31.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.038+0000 7ff514c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:43:31.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.038+0000 7ff52b400640 1 --2- 172.21.15.31:0/2711189408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daa783de0 0x563daa780f40 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff52c0670d0 tx=0x7ff52c0675b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e3f40201c1b98c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:31.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.038+0000 7ff514c00640 1 -- 172.21.15.31:0/2711189408 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c10b5b0 con 0x563daa783de0 2024-03-21T10:43:31.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.038+0000 7ff514c00640 1 -- 172.21.15.31:0/2711189408 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7ff52c13b170 con 0x563daa783de0 2024-03-21T10:43:31.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.038+0000 7ff534a2a740 1 -- 172.21.15.31:0/2711189408 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563daa6eb800 con 0x563daa783de0 2024-03-21T10:43:31.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.039+0000 7ff514c00640 1 -- 172.21.15.31:0/2711189408 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff52c10fe70 con 0x563daa783de0 2024-03-21T10:43:31.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.039+0000 7ff534a2a740 10 client.26885 fetch_fsmap learned FSMap version 26 2024-03-21T10:43:31.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.039+0000 7ff534a2a740 1 -- 172.21.15.31:0/2711189408 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x563daa4bc7d0 con 0x563daa783de0 2024-03-21T10:43:31.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.039+0000 7ff514c00640 1 -- 172.21.15.31:0/2711189408 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563daa6eb800 con 0x563daa783de0 2024-03-21T10:43:31.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.039+0000 7ff514c00640 1 -- 172.21.15.31:0/2711189408 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7ff52c10b1b0 con 0x563daa783de0 2024-03-21T10:43:31.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.040+0000 7ff534a2a740 10 client.26885 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:43:31.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.040+0000 7ff534a2a740 10 client.26885 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:43:31.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.040+0000 7ff534a2a740 20 client.26885 populate_metadata read hostname 'smithi031' 2024-03-21T10:43:31.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.040+0000 7ff534a2a740 1 --2- 172.21.15.31:0/2711189408 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x563daa793c80 0x563daa7960b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.040+0000 7ff534a2a740 4 client.26885 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:43:31.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.040+0000 7ff534a2a740 1 -- 172.21.15.31:0/2711189408 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563daa6e5b50 con 0x563daa793c80 2024-03-21T10:43:31.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.040+0000 7ff52aa00640 1 --2- 172.21.15.31:0/2711189408 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x563daa793c80 0x563daa7960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.041+0000 7ff52aa00640 1 --2- 172.21.15.31:0/2711189408 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x563daa793c80 0x563daa7960b0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=eced236455ac6db7 server_cookie=31060740c1751160 in_seq=0 out_seq=0 2024-03-21T10:43:31.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.041+0000 7ff514c00640 10 client.26885 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:43:31.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.042+0000 7ff514c00640 1 -- 172.21.15.31:0/2711189408 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563daa6e5b50 con 0x563daa793c80 2024-03-21T10:43:31.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.042+0000 7ff514c00640 10 client.26885 handle_command_reply: tid=0 2024-03-21T10:43:31.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.043+0000 7ff534a2a740 1 client.26885 shutdown 2024-03-21T10:43:31.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.043+0000 7ff534a2a740 1 -- 172.21.15.31:0/2711189408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daa783de0 msgr2=0x563daa780f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:31.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.043+0000 7ff534a2a740 1 --2- 172.21.15.31:0/2711189408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daa783de0 0x563daa780f40 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff52c0670d0 tx=0x7ff52c0675b0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.043+0000 7ff534a2a740 1 -- 172.21.15.31:0/2711189408 shutdown_connections 2024-03-21T10:43:31.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.043+0000 7ff534a2a740 1 --2- 172.21.15.31:0/2711189408 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x563daa793c80 0x563daa7960b0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.043+0000 7ff534a2a740 1 --2- 172.21.15.31:0/2711189408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daa783de0 0x563daa780f40 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.043+0000 7ff534a2a740 1 --2- 172.21.15.31:0/2711189408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563daa77b260 0x563daa780a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.043+0000 7ff534a2a740 1 --2- 172.21.15.31:0/2711189408 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563daa77a8b0 0x563daa783670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.043+0000 7ff534a2a740 1 -- 172.21.15.31:0/2711189408 >> 172.21.15.31:0/2711189408 conn(0x563daa774f80 msgr2=0x563daa776cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:31.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.043+0000 7ff514c00640 0 client.26885 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:43:31.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.044+0000 7ff534a2a740 1 -- 172.21.15.31:0/2711189408 shutdown_connections 2024-03-21T10:43:31.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.044+0000 7ff534a2a740 1 -- 172.21.15.31:0/2711189408 wait complete. 2024-03-21T10:43:31.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.044+0000 7ff534a2a740 20 client.26885 trim_cache size 0 max 16384 2024-03-21T10:43:31.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:30 smithi031 ceph-mon[31673]: pgmap v151: 129 pgs: 129 active+clean; 28 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 212 MiB/s wr, 1.71k op/s 2024-03-21T10:43:31.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:30 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1876765928' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:43:31.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.077+0000 7ff534a2a740 1 Processor -- start 2024-03-21T10:43:31.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.077+0000 7ff534a2a740 1 -- start start 2024-03-21T10:43:31.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.078+0000 7ff534a2a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563daa77a8b0 0x563daa774160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.078+0000 7ff534a2a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563daa77b260 0x563daa77bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.078+0000 7ff534a2a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daa783de0 0x563daa783130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.078+0000 7ff534a2a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c012110 con 0x563daa77a8b0 2024-03-21T10:43:31.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.078+0000 7ff534a2a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c00fc10 con 0x563daa783de0 2024-03-21T10:43:31.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.078+0000 7ff534a2a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c00dcb0 con 0x563daa77b260 2024-03-21T10:43:31.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.078+0000 7ff52aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563daa77a8b0 0x563daa774160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.079+0000 7ff52aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563daa77a8b0 0x563daa774160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49104/0 (socket says 172.21.15.31:49104) 2024-03-21T10:43:31.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.079+0000 7ff52aa00640 1 -- 172.21.15.31:0/3579101622 learned_addr learned my addr 172.21.15.31:0/3579101622 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:31.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.079+0000 7ff52a000640 1 --2- 172.21.15.31:0/3579101622 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563daa77b260 0x563daa77bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.079+0000 7ff52aa00640 1 -- 172.21.15.31:0/3579101622 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563daa77b260 msgr2=0x563daa77bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:31.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.079+0000 7ff52aa00640 1 --2- 172.21.15.31:0/3579101622 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563daa77b260 0x563daa77bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.079+0000 7ff52aa00640 1 -- 172.21.15.31:0/3579101622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daa783de0 msgr2=0x563daa783130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:31.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.080+0000 7ff52b400640 1 --2- 172.21.15.31:0/3579101622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daa783de0 0x563daa783130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.080+0000 7ff52aa00640 1 --2- 172.21.15.31:0/3579101622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daa783de0 0x563daa783130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.080+0000 7ff52aa00640 1 -- 172.21.15.31:0/3579101622 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563daa781270 con 0x563daa77a8b0 2024-03-21T10:43:31.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.080+0000 7ff52b400640 1 --2- 172.21.15.31:0/3579101622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daa783de0 0x563daa783130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:43:31.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.081+0000 7ff52aa00640 1 --2- 172.21.15.31:0/3579101622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563daa77a8b0 0x563daa774160 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff51c02c170 tx=0x7ff51c048040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29c87ea22cc22efc server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:31.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.081+0000 7ff517400640 1 -- 172.21.15.31:0/3579101622 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c041040 con 0x563daa77a8b0 2024-03-21T10:43:31.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.081+0000 7ff517400640 1 -- 172.21.15.31:0/3579101622 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563daa6e5b50 con 0x563daa77a8b0 2024-03-21T10:43:31.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.081+0000 7ff517400640 1 -- 172.21.15.31:0/3579101622 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c004d30 con 0x563daa77a8b0 2024-03-21T10:43:31.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.082+0000 7ff534a2a740 1 -- 172.21.15.31:0/3579101622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563daa77a8b0 msgr2=0x563daa774160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:31.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.082+0000 7ff534a2a740 1 --2- 172.21.15.31:0/3579101622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563daa77a8b0 0x563daa774160 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff51c02c170 tx=0x7ff51c048040 comp rx=0 tx=0).stop 2024-03-21T10:43:31.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.083+0000 7ff534a2a740 1 -- 172.21.15.31:0/3579101622 shutdown_connections 2024-03-21T10:43:31.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.083+0000 7ff534a2a740 1 --2- 172.21.15.31:0/3579101622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daa783de0 0x563daa783130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.083+0000 7ff534a2a740 1 --2- 172.21.15.31:0/3579101622 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563daa77b260 0x563daa77bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.083+0000 7ff534a2a740 1 --2- 172.21.15.31:0/3579101622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563daa77a8b0 0x563daa774160 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.083+0000 7ff534a2a740 1 -- 172.21.15.31:0/3579101622 >> 172.21.15.31:0/3579101622 conn(0x563daa774f80 msgr2=0x563daa77b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:31.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.083+0000 7ff534a2a740 1 -- 172.21.15.31:0/3579101622 shutdown_connections 2024-03-21T10:43:31.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.083+0000 7ff534a2a740 1 -- 172.21.15.31:0/3579101622 wait complete. 2024-03-21T10:43:31.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.083+0000 7ff534a2a740 1 -- start start 2024-03-21T10:43:31.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.084+0000 7ff534a2a740 1 Processor -- start 2024-03-21T10:43:31.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.084+0000 7ff534a2a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daa77a8b0 0x563daa7e8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.084+0000 7ff534a2a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563daa77b260 0x563daa7e9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.084+0000 7ff534a2a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563daa783de0 0x563daa78a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.084+0000 7ff534a2a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c00e2b0 con 0x563daa783de0 2024-03-21T10:43:31.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.084+0000 7ff534a2a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c00f5b0 con 0x563daa77a8b0 2024-03-21T10:43:31.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.084+0000 7ff534a2a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c00e8f0 con 0x563daa77b260 2024-03-21T10:43:31.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.085+0000 7ff534a2a740 10 client.0 _ll_register_callbacks cb 0x7ff52c106fa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:43:31.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.085+0000 7ff52b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563daa783de0 0x563daa78a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.085+0000 7ff52b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563daa783de0 0x563daa78a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49106/0 (socket says 172.21.15.31:49106) 2024-03-21T10:43:31.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.085+0000 7ff52b400640 1 -- 172.21.15.31:0/1988561679 learned_addr learned my addr 172.21.15.31:0/1988561679 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:31.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.085+0000 7ff52a000640 1 --2- 172.21.15.31:0/1988561679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563daa77b260 0x563daa7e9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.085+0000 7ff52aa00640 1 --2- 172.21.15.31:0/1988561679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daa77a8b0 0x563daa7e8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.086+0000 7ff52b400640 1 -- 172.21.15.31:0/1988561679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563daa77b260 msgr2=0x563daa7e9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:31.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.086+0000 7ff52b400640 1 --2- 172.21.15.31:0/1988561679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563daa77b260 0x563daa7e9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.086+0000 7ff52b400640 1 -- 172.21.15.31:0/1988561679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daa77a8b0 msgr2=0x563daa7e8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:31.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.086+0000 7ff52b400640 1 --2- 172.21.15.31:0/1988561679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daa77a8b0 0x563daa7e8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.086+0000 7ff52b400640 1 -- 172.21.15.31:0/1988561679 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563daa78a760 con 0x563daa783de0 2024-03-21T10:43:31.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.086+0000 7ff52a000640 1 --2- 172.21.15.31:0/1988561679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563daa77b260 0x563daa7e9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:43:31.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.086+0000 7ff52b400640 1 --2- 172.21.15.31:0/1988561679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563daa783de0 0x563daa78a170 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff52c050670 tx=0x7ff52c103830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd006eca5ad57bb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:31.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.086+0000 7ff514c00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:43:31.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.087+0000 7ff514c00640 1 -- 172.21.15.31:0/1988561679 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c10b5b0 con 0x563daa783de0 2024-03-21T10:43:31.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.087+0000 7ff534a2a740 1 -- 172.21.15.31:0/1988561679 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563daa6e5b50 con 0x563daa783de0 2024-03-21T10:43:31.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.087+0000 7ff514c00640 1 -- 172.21.15.31:0/1988561679 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7ff52c138a40 con 0x563daa783de0 2024-03-21T10:43:31.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.087+0000 7ff514c00640 1 -- 172.21.15.31:0/1988561679 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff52c087520 con 0x563daa783de0 2024-03-21T10:43:31.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.087+0000 7ff534a2a740 10 client.16443 fetch_fsmap learned FSMap version 26 2024-03-21T10:43:31.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.087+0000 7ff534a2a740 1 -- 172.21.15.31:0/1988561679 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x563daa78ba20 con 0x563daa783de0 2024-03-21T10:43:31.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.087+0000 7ff514c00640 1 -- 172.21.15.31:0/1988561679 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563daa6e5b50 con 0x563daa783de0 2024-03-21T10:43:31.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.088+0000 7ff514c00640 1 -- 172.21.15.31:0/1988561679 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7ff52c11bab0 con 0x563daa783de0 2024-03-21T10:43:31.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.088+0000 7ff534a2a740 10 client.16443 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:43:31.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.088+0000 7ff534a2a740 10 client.16443 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:43:31.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.088+0000 7ff534a2a740 20 client.16443 populate_metadata read hostname 'smithi031' 2024-03-21T10:43:31.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.088+0000 7ff534a2a740 1 --2- 172.21.15.31:0/1988561679 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x563daa78c0f0 0x563daa8045b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.088+0000 7ff534a2a740 4 client.16443 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:43:31.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.088+0000 7ff534a2a740 1 -- 172.21.15.31:0/1988561679 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563daa663500 con 0x563daa78c0f0 2024-03-21T10:43:31.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.088+0000 7ff52aa00640 1 --2- 172.21.15.31:0/1988561679 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x563daa78c0f0 0x563daa8045b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.089+0000 7ff52aa00640 1 --2- 172.21.15.31:0/1988561679 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x563daa78c0f0 0x563daa8045b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f571d5d4b3579c79 server_cookie=a7b524f3ac48021a in_seq=0 out_seq=0 2024-03-21T10:43:31.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.089+0000 7ff514c00640 10 client.16443 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:43:31.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.089+0000 7ff514c00640 1 -- 172.21.15.31:0/1988561679 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563daa663500 con 0x563daa78c0f0 2024-03-21T10:43:31.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.089+0000 7ff514c00640 10 client.16443 handle_command_reply: tid=0 2024-03-21T10:43:31.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.089+0000 7ff534a2a740 1 client.16443 shutdown 2024-03-21T10:43:31.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.090+0000 7ff534a2a740 1 -- 172.21.15.31:0/1988561679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563daa783de0 msgr2=0x563daa78a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:31.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.090+0000 7ff534a2a740 1 --2- 172.21.15.31:0/1988561679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563daa783de0 0x563daa78a170 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff52c050670 tx=0x7ff52c103830 comp rx=0 tx=0).stop 2024-03-21T10:43:31.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.090+0000 7ff534a2a740 1 -- 172.21.15.31:0/1988561679 shutdown_connections 2024-03-21T10:43:31.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.090+0000 7ff534a2a740 1 --2- 172.21.15.31:0/1988561679 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x563daa78c0f0 0x563daa8045b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.090+0000 7ff534a2a740 1 --2- 172.21.15.31:0/1988561679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563daa783de0 0x563daa78a170 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.090+0000 7ff534a2a740 1 --2- 172.21.15.31:0/1988561679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563daa77b260 0x563daa7e9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.090+0000 7ff534a2a740 1 --2- 172.21.15.31:0/1988561679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daa77a8b0 0x563daa7e8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.090+0000 7ff514c00640 0 client.16443 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:43:31.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.090+0000 7ff534a2a740 1 -- 172.21.15.31:0/1988561679 >> 172.21.15.31:0/1988561679 conn(0x563daa774f80 msgr2=0x563daa77bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:31.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.090+0000 7ff534a2a740 1 -- 172.21.15.31:0/1988561679 shutdown_connections 2024-03-21T10:43:31.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.091+0000 7ff534a2a740 1 -- 172.21.15.31:0/1988561679 wait complete. 2024-03-21T10:43:31.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.093+0000 7ff534a2a740 20 client.16443 trim_cache size 0 max 16384 2024-03-21T10:43:31.090 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:43:31.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.094+0000 7ff52be00640 1 -- 172.21.15.31:0/1560435586 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff5100a3190 msgr2=0x7ff5100a5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:31.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.094+0000 7ff52be00640 1 --2- 172.21.15.31:0/1560435586 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff5100a3190 0x7ff5100a5650 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff518005fd0 tx=0x7ff518004730 comp rx=0 tx=0).stop 2024-03-21T10:43:31.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.094+0000 7ff52be00640 1 -- 172.21.15.31:0/1560435586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff52409acf0 msgr2=0x7ff524132950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:31.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.094+0000 7ff52be00640 1 --2- 172.21.15.31:0/1560435586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff52409acf0 0x7ff524132950 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff52c050d00 tx=0x7ff52c07aee0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.094+0000 7ff52be00640 1 -- 172.21.15.31:0/1560435586 shutdown_connections 2024-03-21T10:43:31.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.094+0000 7ff52be00640 1 --2- 172.21.15.31:0/1560435586 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff5100a3190 0x7ff5100a5650 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.094+0000 7ff52be00640 1 --2- 172.21.15.31:0/1560435586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff52409acf0 0x7ff524132950 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.094+0000 7ff52be00640 1 --2- 172.21.15.31:0/1560435586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5240986e0 0x7ff524138ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.094+0000 7ff52be00640 1 --2- 172.21.15.31:0/1560435586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5240051f0 0x7ff5241387a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.094+0000 7ff52be00640 1 -- 172.21.15.31:0/1560435586 >> 172.21.15.31:0/1560435586 conn(0x7ff52400a740 msgr2=0x7ff5240976c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:31.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.094+0000 7ff52be00640 1 -- 172.21.15.31:0/1560435586 shutdown_connections 2024-03-21T10:43:31.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.094+0000 7ff52be00640 1 -- 172.21.15.31:0/1560435586 wait complete. 2024-03-21T10:43:31.110 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:43:31.110 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-21T10:43:31.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:30 smithi040 ceph-mon[36154]: pgmap v151: 129 pgs: 129 active+clean; 28 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 212 MiB/s wr, 1.71k op/s 2024-03-21T10:43:31.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:30 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1876765928' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:43:31.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:30 smithi070 ceph-mon[36330]: pgmap v151: 129 pgs: 129 active+clean; 28 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 212 MiB/s wr, 1.71k op/s 2024-03-21T10:43:31.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:30 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1876765928' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:43:31.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.386+0000 7f57b0c00640 1 -- 172.21.15.31:0/3771276523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57ac101ff0 msgr2=0x7f57ac10e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:31.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.386+0000 7f57b0c00640 1 --2- 172.21.15.31:0/3771276523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57ac101ff0 0x7f57ac10e500 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f57a4009ca0 tx=0x7f57a402c100 comp rx=0 tx=0).stop 2024-03-21T10:43:31.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.387+0000 7f57b0c00640 1 -- 172.21.15.31:0/3771276523 shutdown_connections 2024-03-21T10:43:31.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.387+0000 7f57b0c00640 1 --2- 172.21.15.31:0/3771276523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57ac101ff0 0x7f57ac10e500 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.387+0000 7f57b0c00640 1 --2- 172.21.15.31:0/3771276523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57ac101630 0x7f57ac101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.387+0000 7f57b0c00640 1 --2- 172.21.15.31:0/3771276523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57ac107640 0x7f57ac107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.387+0000 7f57b0c00640 1 -- 172.21.15.31:0/3771276523 >> 172.21.15.31:0/3771276523 conn(0x7f57ac0fd140 msgr2=0x7f57ac0ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:31.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.388+0000 7f57b0c00640 1 -- 172.21.15.31:0/3771276523 shutdown_connections 2024-03-21T10:43:31.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.388+0000 7f57b0c00640 1 -- 172.21.15.31:0/3771276523 wait complete. 2024-03-21T10:43:31.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.389+0000 7f57b0c00640 1 Processor -- start 2024-03-21T10:43:31.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.390+0000 7f57b0c00640 1 -- start start 2024-03-21T10:43:31.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.390+0000 7f57b0c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57ac101630 0x7f57ac10e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.390+0000 7f57b0c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57ac101ff0 0x7f57ac10b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.390+0000 7f57b0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57ac107640 0x7f57ac10bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.390+0000 7f57b0c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ac075470 con 0x7f57ac101630 2024-03-21T10:43:31.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.390+0000 7f57b0c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ac0752f0 con 0x7f57ac107640 2024-03-21T10:43:31.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.390+0000 7f57b0c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ac0755f0 con 0x7f57ac101ff0 2024-03-21T10:43:31.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.391+0000 7f57a9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57ac101630 0x7f57ac10e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.391+0000 7f57a9400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57ac101ff0 0x7f57ac10b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.391+0000 7f57a9400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57ac101ff0 0x7f57ac10b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42316/0 (socket says 172.21.15.31:42316) 2024-03-21T10:43:31.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.391+0000 7f57a9400640 1 -- 172.21.15.31:0/2631866525 learned_addr learned my addr 172.21.15.31:0/2631866525 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:31.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.391+0000 7f57a9e00640 1 -- 172.21.15.31:0/2631866525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57ac101ff0 msgr2=0x7f57ac10b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:31.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.391+0000 7f57a9e00640 1 --2- 172.21.15.31:0/2631866525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57ac101ff0 0x7f57ac10b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.391+0000 7f57a9e00640 1 -- 172.21.15.31:0/2631866525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57ac107640 msgr2=0x7f57ac10bd00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-21T10:43:31.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.391+0000 7f57a9e00640 1 --2- 172.21.15.31:0/2631866525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57ac107640 0x7f57ac10bd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.391+0000 7f57a9e00640 1 -- 172.21.15.31:0/2631866525 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ac1aa840 con 0x7f57ac101630 2024-03-21T10:43:31.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.392+0000 7f57a9e00640 1 --2- 172.21.15.31:0/2631866525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57ac101630 0x7f57ac10e4f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f579c00e9f0 tx=0x7f579c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:31.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.393+0000 7f57a2a00640 1 -- 172.21.15.31:0/2631866525 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579c00cd20 con 0x7f57ac101630 2024-03-21T10:43:31.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.393+0000 7f57a2a00640 1 -- 172.21.15.31:0/2631866525 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f579c0035f0 con 0x7f57ac101630 2024-03-21T10:43:31.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.393+0000 7f57a2a00640 1 -- 172.21.15.31:0/2631866525 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579c010690 con 0x7f57ac101630 2024-03-21T10:43:31.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.393+0000 7f57b0c00640 1 -- 172.21.15.31:0/2631866525 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57ac1aaad0 con 0x7f57ac101630 2024-03-21T10:43:31.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.393+0000 7f57b0c00640 1 -- 172.21.15.31:0/2631866525 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57ac1aafa0 con 0x7f57ac101630 2024-03-21T10:43:31.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.398+0000 7f57b382a740 1 Processor -- start 2024-03-21T10:43:31.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.398+0000 7f57b382a740 1 -- start start 2024-03-21T10:43:31.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.399+0000 7f57b382a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37530 0x561b92f37910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.399+0000 7f57b382a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b92f37ee0 0x561b92f4d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.399+0000 7f57b382a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b92f4dc40 0x561b92f4e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.399+0000 7f57aa800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37530 0x561b92f37910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.399+0000 7f57aa800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37530 0x561b92f37910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58230/0 (socket says 172.21.15.31:58230) 2024-03-21T10:43:31.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.399+0000 7f57aa800640 1 -- 172.21.15.31:0/3714492659 learned_addr learned my addr 172.21.15.31:0/3714492659 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:31.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.399+0000 7f57b382a740 1 -- 172.21.15.31:0/3714492659 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b92e9b2c0 con 0x561b92f37ee0 2024-03-21T10:43:31.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.399+0000 7f57b382a740 1 -- 172.21.15.31:0/3714492659 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b92e43c90 con 0x561b92f37530 2024-03-21T10:43:31.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.399+0000 7f57b382a740 1 -- 172.21.15.31:0/3714492659 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b92e97360 con 0x561b92f4dc40 2024-03-21T10:43:31.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.402+0000 7f57a9400640 1 --2- 172.21.15.31:0/3714492659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b92f4dc40 0x561b92f4e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.404+0000 7f57aa800640 1 -- 172.21.15.31:0/3714492659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b92f4dc40 msgr2=0x561b92f4e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:31.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.404+0000 7f57aa800640 1 --2- 172.21.15.31:0/3714492659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b92f4dc40 0x561b92f4e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.404+0000 7f57aa800640 1 -- 172.21.15.31:0/3714492659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b92f37ee0 msgr2=0x561b92f4d620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:43:31.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.404+0000 7f57aa800640 1 --2- 172.21.15.31:0/3714492659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b92f37ee0 0x561b92f4d620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.404+0000 7f57aa800640 1 -- 172.21.15.31:0/3714492659 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b92e68080 con 0x561b92f37530 2024-03-21T10:43:31.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.405+0000 7f57a9400640 1 --2- 172.21.15.31:0/3714492659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b92f4dc40 0x561b92f4e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:43:31.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.421+0000 7f57a2a00640 1 -- 172.21.15.31:0/2631866525 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f579c0037a0 con 0x7f57ac101630 2024-03-21T10:43:31.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.424+0000 7f57a2a00640 1 --2- 172.21.15.31:0/2631866525 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f57900a3150 0x7f57900a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.425+0000 7f57a9e00640 1 --2- 172.21.15.31:0/2631866525 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f57900a3150 0x7f57900a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.426+0000 7f57a2a00640 1 -- 172.21.15.31:0/2631866525 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f579c099e80 con 0x7f57ac101630 2024-03-21T10:43:31.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.427+0000 7f57a9e00640 1 --2- 172.21.15.31:0/2631866525 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f57900a3150 0x7f57900a5610 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f579c0968b0 tx=0x7f579c0a0120 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:31.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.445+0000 7f57aa800640 1 --2- 172.21.15.31:0/3714492659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37530 0x561b92f37910 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f57a402c690 tx=0x7f57a4005fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80995405225327e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:31.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.450+0000 7f5797000640 1 -- 172.21.15.31:0/3714492659 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a4063020 con 0x561b92f37530 2024-03-21T10:43:31.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.450+0000 7f5797000640 1 -- 172.21.15.31:0/3714492659 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57a405b040 con 0x561b92f37530 2024-03-21T10:43:31.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.450+0000 7f5797000640 1 -- 172.21.15.31:0/3714492659 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a4004040 con 0x561b92f37530 2024-03-21T10:43:31.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.451+0000 7f57b382a740 1 -- 172.21.15.31:0/3714492659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37530 msgr2=0x561b92f37910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:31.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.451+0000 7f57b382a740 1 --2- 172.21.15.31:0/3714492659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37530 0x561b92f37910 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f57a402c690 tx=0x7f57a4005fc0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.451+0000 7f57b382a740 1 -- 172.21.15.31:0/3714492659 shutdown_connections 2024-03-21T10:43:31.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.451+0000 7f57b382a740 1 --2- 172.21.15.31:0/3714492659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b92f4dc40 0x561b92f4e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.451+0000 7f57b382a740 1 --2- 172.21.15.31:0/3714492659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b92f37ee0 0x561b92f4d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.451+0000 7f57b382a740 1 --2- 172.21.15.31:0/3714492659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37530 0x561b92f37910 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.451+0000 7f57b382a740 1 -- 172.21.15.31:0/3714492659 >> 172.21.15.31:0/3714492659 conn(0x561b92f31da0 msgr2=0x561b92f32180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:31.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.452+0000 7f57b382a740 1 -- 172.21.15.31:0/3714492659 shutdown_connections 2024-03-21T10:43:31.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.452+0000 7f57b382a740 1 -- 172.21.15.31:0/3714492659 wait complete. 2024-03-21T10:43:31.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.453+0000 7f57b382a740 1 -- start start 2024-03-21T10:43:31.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.453+0000 7f57b382a740 1 Processor -- start 2024-03-21T10:43:31.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.456+0000 7f57b382a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b92f37530 0x561b92f3eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.456+0000 7f57b382a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37ee0 0x561b92f3f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.456+0000 7f57b382a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b92f4dc40 0x561b92f39f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.456+0000 7f57b382a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b92f387b0 con 0x561b92f37530 2024-03-21T10:43:31.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.456+0000 7f57b382a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b92e570d0 con 0x561b92f37ee0 2024-03-21T10:43:31.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.456+0000 7f57b382a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b92f38930 con 0x561b92f4dc40 2024-03-21T10:43:31.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.456+0000 7f57b382a740 10 client.0 _ll_register_callbacks cb 0x561b92e5be80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:43:31.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.462+0000 7f57aa800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b92f4dc40 0x561b92f39f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.462+0000 7f57a9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37ee0 0x561b92f3f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.462+0000 7f57a9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37ee0 0x561b92f3f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58234/0 (socket says 172.21.15.31:58234) 2024-03-21T10:43:31.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.462+0000 7f57a9400640 1 -- 172.21.15.31:0/1275266138 learned_addr learned my addr 172.21.15.31:0/1275266138 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:31.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.462+0000 7f57a9400640 1 -- 172.21.15.31:0/1275266138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b92f4dc40 msgr2=0x561b92f39f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:31.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.464+0000 7f57a9400640 1 --2- 172.21.15.31:0/1275266138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b92f4dc40 0x561b92f39f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.464+0000 7f57a9400640 1 -- 172.21.15.31:0/1275266138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b92f37530 msgr2=0x561b92f3eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:31.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.464+0000 7f57a9400640 1 --2- 172.21.15.31:0/1275266138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b92f37530 0x561b92f3eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.464+0000 7f57a9400640 1 -- 172.21.15.31:0/1275266138 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561b92baf3f0 con 0x561b92f37ee0 2024-03-21T10:43:31.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.468+0000 7f57aa800640 1 --2- 172.21.15.31:0/1275266138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b92f4dc40 0x561b92f39f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:43:31.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.468+0000 7f57a9400640 1 --2- 172.21.15.31:0/1275266138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37ee0 0x561b92f3f040 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5798011750 tx=0x7f5798011c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6431b51666416bd server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:31.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.468+0000 7f578b400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:43:31.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.469+0000 7f578b400640 1 -- 172.21.15.31:0/1275266138 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579800fd50 con 0x561b92f37ee0 2024-03-21T10:43:31.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.469+0000 7f57b382a740 1 -- 172.21.15.31:0/1275266138 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f57a405b040 con 0x561b92f37ee0 2024-03-21T10:43:31.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.470+0000 7f578b400640 1 -- 172.21.15.31:0/1275266138 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f579800fb50 con 0x561b92f37ee0 2024-03-21T10:43:31.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.472+0000 7f578b400640 1 -- 172.21.15.31:0/1275266138 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f579800fd50 con 0x561b92f37ee0 2024-03-21T10:43:31.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.472+0000 7f57b382a740 10 client.26897 fetch_fsmap learned FSMap version 26 2024-03-21T10:43:31.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.472+0000 7f57b382a740 1 -- 172.21.15.31:0/1275266138 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x561b92c797d0 con 0x561b92f37ee0 2024-03-21T10:43:31.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.473+0000 7f578b400640 1 -- 172.21.15.31:0/1275266138 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57a405b040 con 0x561b92f37ee0 2024-03-21T10:43:31.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.475+0000 7f578b400640 1 -- 172.21.15.31:0/1275266138 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f5798004a40 con 0x561b92f37ee0 2024-03-21T10:43:31.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.475+0000 7f57b382a740 10 client.26897 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:43:31.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.475+0000 7f57b382a740 10 client.26897 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:43:31.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.475+0000 7f57b382a740 20 client.26897 populate_metadata read hostname 'smithi031' 2024-03-21T10:43:31.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.475+0000 7f57b382a740 1 --2- 172.21.15.31:0/1275266138 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561b92f49cb0 0x561b92f4c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.475+0000 7f57b382a740 4 client.26897 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:43:31.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.475+0000 7f57b382a740 1 -- 172.21.15.31:0/1275266138 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561b92ea8800 con 0x561b92f49cb0 2024-03-21T10:43:31.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.476+0000 7f57aa800640 1 --2- 172.21.15.31:0/1275266138 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561b92f49cb0 0x561b92f4c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.477+0000 7f578b400640 10 client.26897 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:31.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.477+0000 7f57aa800640 1 --2- 172.21.15.31:0/1275266138 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561b92f49cb0 0x561b92f4c170 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9acf9148f7838a9 server_cookie=42405c2495c29a9b in_seq=0 out_seq=0 2024-03-21T10:43:31.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.479+0000 7f578b400640 1 -- 172.21.15.31:0/1275266138 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561b92ea8800 con 0x561b92f49cb0 2024-03-21T10:43:31.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.479+0000 7f578b400640 10 client.26897 handle_command_reply: tid=0 2024-03-21T10:43:31.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.479+0000 7f57b382a740 1 client.26897 shutdown 2024-03-21T10:43:31.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.481+0000 7f57b382a740 1 -- 172.21.15.31:0/1275266138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37ee0 msgr2=0x561b92f3f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:31.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.481+0000 7f57b382a740 1 --2- 172.21.15.31:0/1275266138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37ee0 0x561b92f3f040 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5798011750 tx=0x7f5798011c10 comp rx=0 tx=0).stop 2024-03-21T10:43:31.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.481+0000 7f57b382a740 1 -- 172.21.15.31:0/1275266138 shutdown_connections 2024-03-21T10:43:31.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.481+0000 7f57b382a740 1 --2- 172.21.15.31:0/1275266138 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561b92f49cb0 0x561b92f4c170 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.481+0000 7f57b382a740 1 --2- 172.21.15.31:0/1275266138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b92f4dc40 0x561b92f39f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.481+0000 7f57b382a740 1 --2- 172.21.15.31:0/1275266138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37ee0 0x561b92f3f040 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.481+0000 7f578b400640 0 client.26897 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:31.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.481+0000 7f57b382a740 1 --2- 172.21.15.31:0/1275266138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b92f37530 0x561b92f3eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.481+0000 7f57b382a740 1 -- 172.21.15.31:0/1275266138 >> 172.21.15.31:0/1275266138 conn(0x561b92f31da0 msgr2=0x561b92f33af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:31.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.482+0000 7f57b382a740 1 -- 172.21.15.31:0/1275266138 shutdown_connections 2024-03-21T10:43:31.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.482+0000 7f57b382a740 1 -- 172.21.15.31:0/1275266138 wait complete. 2024-03-21T10:43:31.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.482+0000 7f57b382a740 20 client.26897 trim_cache size 0 max 16384 2024-03-21T10:43:31.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.519+0000 7f57b382a740 1 Processor -- start 2024-03-21T10:43:31.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.519+0000 7f57b382a740 1 -- start start 2024-03-21T10:43:31.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.520+0000 7f57b382a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37530 0x561b92f51040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.520+0000 7f57b382a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b92f37ee0 0x561b92f310f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.520+0000 7f57b382a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b92f4dc40 0x561b92f52bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.520+0000 7f57b382a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ac117f00 con 0x561b92f37ee0 2024-03-21T10:43:31.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.520+0000 7f57b382a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ac115a00 con 0x561b92f37530 2024-03-21T10:43:31.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.520+0000 7f57b382a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ac073030 con 0x561b92f4dc40 2024-03-21T10:43:31.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.521+0000 7f57aa800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37530 0x561b92f51040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.521+0000 7f57aa800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37530 0x561b92f51040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58240/0 (socket says 172.21.15.31:58240) 2024-03-21T10:43:31.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.521+0000 7f57aa800640 1 -- 172.21.15.31:0/3241478825 learned_addr learned my addr 172.21.15.31:0/3241478825 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:31.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.521+0000 7f57aa800640 1 --2- 172.21.15.31:0/3241478825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b92f4dc40 0x561b92f52bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.521+0000 7f57a9400640 1 --2- 172.21.15.31:0/3241478825 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b92f37ee0 0x561b92f310f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.522+0000 7f57a9400640 1 -- 172.21.15.31:0/3241478825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b92f4dc40 msgr2=0x561b92f52bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:31.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.522+0000 7f57a9400640 1 --2- 172.21.15.31:0/3241478825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b92f4dc40 0x561b92f52bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.522+0000 7f57a9400640 1 -- 172.21.15.31:0/3241478825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37530 msgr2=0x561b92f51040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:31.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.522+0000 7f57a9400640 1 --2- 172.21.15.31:0/3241478825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37530 0x561b92f51040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.522+0000 7f57a9400640 1 -- 172.21.15.31:0/3241478825 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b92f53ec0 con 0x561b92f37ee0 2024-03-21T10:43:31.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.522+0000 7f57aa800640 1 --2- 172.21.15.31:0/3241478825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b92f4dc40 0x561b92f52bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:43:31.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.523+0000 7f57a9400640 1 --2- 172.21.15.31:0/3241478825 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b92f37ee0 0x561b92f310f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5798004e30 tx=0x7f5798011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56270743e00144a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:31.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.524+0000 7f5796600640 1 -- 172.21.15.31:0/3241478825 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5798028040 con 0x561b92f37ee0 2024-03-21T10:43:31.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.524+0000 7f5796600640 1 -- 172.21.15.31:0/3241478825 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5798032070 con 0x561b92f37ee0 2024-03-21T10:43:31.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.524+0000 7f5796600640 1 -- 172.21.15.31:0/3241478825 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579800fd50 con 0x561b92f37ee0 2024-03-21T10:43:31.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.528+0000 7f57b382a740 1 -- 172.21.15.31:0/3241478825 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b92f37ee0 msgr2=0x561b92f310f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:31.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.528+0000 7f57b382a740 1 --2- 172.21.15.31:0/3241478825 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b92f37ee0 0x561b92f310f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5798004e30 tx=0x7f5798011e80 comp rx=0 tx=0).stop 2024-03-21T10:43:31.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.528+0000 7f57b382a740 1 -- 172.21.15.31:0/3241478825 shutdown_connections 2024-03-21T10:43:31.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.528+0000 7f57b382a740 1 --2- 172.21.15.31:0/3241478825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b92f4dc40 0x561b92f52bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.528+0000 7f57b382a740 1 --2- 172.21.15.31:0/3241478825 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b92f37ee0 0x561b92f310f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.532+0000 7f57b382a740 1 --2- 172.21.15.31:0/3241478825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37530 0x561b92f51040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.532+0000 7f57b382a740 1 -- 172.21.15.31:0/3241478825 >> 172.21.15.31:0/3241478825 conn(0x561b92f31da0 msgr2=0x561b92f4d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:31.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.532+0000 7f57b382a740 1 -- 172.21.15.31:0/3241478825 shutdown_connections 2024-03-21T10:43:31.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.532+0000 7f57b382a740 1 -- 172.21.15.31:0/3241478825 wait complete. 2024-03-21T10:43:31.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.533+0000 7f57b382a740 1 -- start start 2024-03-21T10:43:31.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.533+0000 7f57b382a740 1 Processor -- start 2024-03-21T10:43:31.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.534+0000 7f57b382a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b92f37530 0x561b92fb6970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.537+0000 7f57aa800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b92f37530 0x561b92fb6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.537+0000 7f57b382a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37ee0 0x561b92fb6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.537+0000 7f57b382a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b92f4dc40 0x561b92f43160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.537+0000 7f57b382a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ac073690 con 0x561b92f37530 2024-03-21T10:43:31.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.537+0000 7f57b382a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ac1153a0 con 0x561b92f37ee0 2024-03-21T10:43:31.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.537+0000 7f57b382a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ac1146e0 con 0x561b92f4dc40 2024-03-21T10:43:31.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.537+0000 7f57b382a740 10 client.0 _ll_register_callbacks cb 0x7f579800a660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:43:31.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.538+0000 7f57a9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37ee0 0x561b92fb6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.538+0000 7f57a9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37ee0 0x561b92fb6eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58242/0 (socket says 172.21.15.31:58242) 2024-03-21T10:43:31.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.538+0000 7f57a9400640 1 -- 172.21.15.31:0/1852886627 learned_addr learned my addr 172.21.15.31:0/1852886627 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:31.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.539+0000 7f57a9400640 1 -- 172.21.15.31:0/1852886627 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b92f4dc40 msgr2=0x561b92f43160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:31.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.539+0000 7f57a9400640 1 --2- 172.21.15.31:0/1852886627 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b92f4dc40 0x561b92f43160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.539+0000 7f57a9400640 1 -- 172.21.15.31:0/1852886627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b92f37530 msgr2=0x561b92fb6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:31.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.539+0000 7f57a9400640 1 --2- 172.21.15.31:0/1852886627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b92f37530 0x561b92fb6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.539+0000 7f57a9400640 1 -- 172.21.15.31:0/1852886627 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561b92f43850 con 0x561b92f37ee0 2024-03-21T10:43:31.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.540+0000 7f57a9400640 1 --2- 172.21.15.31:0/1852886627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37ee0 0x561b92fb6eb0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f579802f040 tx=0x7f5798005e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7130715982034cb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:31.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.540+0000 7f578b400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:43:31.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.540+0000 7f578b400640 1 -- 172.21.15.31:0/1852886627 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5798028040 con 0x561b92f37ee0 2024-03-21T10:43:31.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.541+0000 7f578b400640 1 -- 172.21.15.31:0/1852886627 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f57980129b0 con 0x561b92f37ee0 2024-03-21T10:43:31.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.541+0000 7f57b382a740 1 -- 172.21.15.31:0/1852886627 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5798032070 con 0x561b92f37ee0 2024-03-21T10:43:31.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.542+0000 7f578b400640 1 -- 172.21.15.31:0/1852886627 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5798044050 con 0x561b92f37ee0 2024-03-21T10:43:31.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.543+0000 7f57b382a740 10 client.26903 fetch_fsmap learned FSMap version 26 2024-03-21T10:43:31.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.543+0000 7f57b382a740 1 -- 172.21.15.31:0/1852886627 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x561b92fbe170 con 0x561b92f37ee0 2024-03-21T10:43:31.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.544+0000 7f578b400640 1 -- 172.21.15.31:0/1852886627 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5798032070 con 0x561b92f37ee0 2024-03-21T10:43:31.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.550+0000 7f578b400640 1 -- 172.21.15.31:0/1852886627 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f5798005530 con 0x561b92f37ee0 2024-03-21T10:43:31.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.550+0000 7f57b382a740 10 client.26903 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:43:31.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.550+0000 7f57b382a740 10 client.26903 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:43:31.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.550+0000 7f57b382a740 20 client.26903 populate_metadata read hostname 'smithi031' 2024-03-21T10:43:31.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.550+0000 7f57b382a740 1 --2- 172.21.15.31:0/1852886627 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561b92fbe780 0x561b92fc0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:31.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.551+0000 7f57b382a740 4 client.26903 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:43:31.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.551+0000 7f57b382a740 1 -- 172.21.15.31:0/1852886627 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561b92ea2b50 con 0x561b92fbe780 2024-03-21T10:43:31.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.551+0000 7f57aa800640 1 --2- 172.21.15.31:0/1852886627 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561b92fbe780 0x561b92fc0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:31.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.553+0000 7f578b400640 10 client.26903 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:31.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.553+0000 7f57aa800640 1 --2- 172.21.15.31:0/1852886627 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561b92fbe780 0x561b92fc0ba0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca84c5477b57fd64 server_cookie=e80cd13322c551e7 in_seq=0 out_seq=0 2024-03-21T10:43:31.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.553+0000 7f578b400640 1 -- 172.21.15.31:0/1852886627 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561b92ea2b50 con 0x561b92fbe780 2024-03-21T10:43:31.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.553+0000 7f578b400640 10 client.26903 handle_command_reply: tid=0 2024-03-21T10:43:31.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.555+0000 7f57b382a740 1 client.26903 shutdown 2024-03-21T10:43:31.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.555+0000 7f57b382a740 1 -- 172.21.15.31:0/1852886627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37ee0 msgr2=0x561b92fb6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:31.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.555+0000 7f57b382a740 1 --2- 172.21.15.31:0/1852886627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37ee0 0x561b92fb6eb0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f579802f040 tx=0x7f5798005e40 comp rx=0 tx=0).stop 2024-03-21T10:43:31.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.555+0000 7f57b382a740 1 -- 172.21.15.31:0/1852886627 shutdown_connections 2024-03-21T10:43:31.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.555+0000 7f57b382a740 1 --2- 172.21.15.31:0/1852886627 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561b92fbe780 0x561b92fc0ba0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.555+0000 7f57b382a740 1 --2- 172.21.15.31:0/1852886627 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b92f4dc40 0x561b92f43160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.556+0000 7f57b382a740 1 --2- 172.21.15.31:0/1852886627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b92f37ee0 0x561b92fb6eb0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.556+0000 7f57b382a740 1 --2- 172.21.15.31:0/1852886627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b92f37530 0x561b92fb6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.556+0000 7f578b400640 0 client.26903 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:31.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.556+0000 7f57b382a740 1 -- 172.21.15.31:0/1852886627 >> 172.21.15.31:0/1852886627 conn(0x561b92f31da0 msgr2=0x561b92f32f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:31.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.557+0000 7f57b382a740 1 -- 172.21.15.31:0/1852886627 shutdown_connections 2024-03-21T10:43:31.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.557+0000 7f57b382a740 1 -- 172.21.15.31:0/1852886627 wait complete. 2024-03-21T10:43:31.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.557+0000 7f57b382a740 20 client.26903 trim_cache size 0 max 16384 2024-03-21T10:43:31.555 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:43:31.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.558+0000 7f57b0c00640 1 -- 172.21.15.31:0/2631866525 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f57900a3150 msgr2=0x7f57900a5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:31.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.558+0000 7f57b0c00640 1 --2- 172.21.15.31:0/2631866525 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f57900a3150 0x7f57900a5610 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f579c0968b0 tx=0x7f579c0a0120 comp rx=0 tx=0).stop 2024-03-21T10:43:31.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.558+0000 7f57b0c00640 1 -- 172.21.15.31:0/2631866525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57ac101630 msgr2=0x7f57ac10e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:31.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.558+0000 7f57b0c00640 1 --2- 172.21.15.31:0/2631866525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57ac101630 0x7f57ac10e4f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f579c00e9f0 tx=0x7f579c00eeb0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.559+0000 7f57b0c00640 1 -- 172.21.15.31:0/2631866525 shutdown_connections 2024-03-21T10:43:31.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.559+0000 7f57b0c00640 1 --2- 172.21.15.31:0/2631866525 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f57900a3150 0x7f57900a5610 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.559+0000 7f57b0c00640 1 --2- 172.21.15.31:0/2631866525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57ac107640 0x7f57ac10bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.559+0000 7f57b0c00640 1 --2- 172.21.15.31:0/2631866525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57ac101ff0 0x7f57ac10b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.559+0000 7f57b0c00640 1 --2- 172.21.15.31:0/2631866525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57ac101630 0x7f57ac10e4f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:31.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.559+0000 7f57b0c00640 1 -- 172.21.15.31:0/2631866525 >> 172.21.15.31:0/2631866525 conn(0x7f57ac0fd140 msgr2=0x7f57ac0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:31.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.559+0000 7f57b0c00640 1 -- 172.21.15.31:0/2631866525 shutdown_connections 2024-03-21T10:43:31.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:31.560+0000 7f57b0c00640 1 -- 172.21.15.31:0/2631866525 wait complete. 2024-03-21T10:43:32.664 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-21T10:43:32.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:43:32.664 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-21T10:43:32.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.961+0000 7f487a600640 1 -- 172.21.15.31:0/3483341334 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4874101440 msgr2=0x7f48741018c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:32.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.961+0000 7f487a600640 1 --2- 172.21.15.31:0/3483341334 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4874101440 0x7f48741018c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f486000fb10 tx=0x7f486002fdd0 comp rx=0 tx=0).stop 2024-03-21T10:43:32.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.962+0000 7f487a600640 1 -- 172.21.15.31:0/3483341334 shutdown_connections 2024-03-21T10:43:32.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.962+0000 7f487a600640 1 --2- 172.21.15.31:0/3483341334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4874101e00 0x7f487410e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:32.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.962+0000 7f487a600640 1 --2- 172.21.15.31:0/3483341334 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4874101440 0x7f48741018c0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:32.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.962+0000 7f487a600640 1 --2- 172.21.15.31:0/3483341334 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48741074e0 0x7f48741078c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:32.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.962+0000 7f487a600640 1 -- 172.21.15.31:0/3483341334 >> 172.21.15.31:0/3483341334 conn(0x7f48740fcf50 msgr2=0x7f48740ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:32.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.962+0000 7f487a600640 1 -- 172.21.15.31:0/3483341334 shutdown_connections 2024-03-21T10:43:32.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.963+0000 7f487a600640 1 -- 172.21.15.31:0/3483341334 wait complete. 2024-03-21T10:43:32.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.965+0000 7f487a600640 1 Processor -- start 2024-03-21T10:43:32.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.965+0000 7f487a600640 1 -- start start 2024-03-21T10:43:32.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.966+0000 7f487a600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4874101440 0x7f487419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:32.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.966+0000 7f487a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4874101e00 0x7f487419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:32.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.966+0000 7f487a600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48741074e0 0x7f4874195a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:32.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.966+0000 7f487a600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487410f6b0 con 0x7f48741074e0 2024-03-21T10:43:32.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.966+0000 7f487a600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487410f530 con 0x7f4874101e00 2024-03-21T10:43:32.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.966+0000 7f487a600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487410f830 con 0x7f4874101440 2024-03-21T10:43:32.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.966+0000 7f4873e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4874101e00 0x7f487419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:32.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.967+0000 7f4873e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4874101e00 0x7f487419cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58248/0 (socket says 172.21.15.31:58248) 2024-03-21T10:43:32.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.967+0000 7f4879c00640 1 --2- 172.21.15.31:0/1413617661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48741074e0 0x7f4874195a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:32.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.967+0000 7f4873e00640 1 -- 172.21.15.31:0/1413617661 learned_addr learned my addr 172.21.15.31:0/1413617661 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:32.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.967+0000 7f4879c00640 1 -- 172.21.15.31:0/1413617661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4874101440 msgr2=0x7f487419c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:32.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.967+0000 7f4879c00640 1 --2- 172.21.15.31:0/1413617661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4874101440 0x7f487419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:32.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.967+0000 7f4879c00640 1 -- 172.21.15.31:0/1413617661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4874101e00 msgr2=0x7f487419cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:32.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.967+0000 7f4879c00640 1 --2- 172.21.15.31:0/1413617661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4874101e00 0x7f487419cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:32.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.967+0000 7f4879c00640 1 -- 172.21.15.31:0/1413617661 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4874196340 con 0x7f48741074e0 2024-03-21T10:43:32.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.968+0000 7f4879c00640 1 --2- 172.21.15.31:0/1413617661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48741074e0 0x7f4874195a80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f486c009fd0 tx=0x7f486c00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:32.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.969+0000 7f4871600640 1 -- 172.21.15.31:0/1413617661 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486c00ce20 con 0x7f48741074e0 2024-03-21T10:43:32.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.969+0000 7f487a600640 1 -- 172.21.15.31:0/1413617661 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4874196630 con 0x7f48741074e0 2024-03-21T10:43:32.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.969+0000 7f487a600640 1 -- 172.21.15.31:0/1413617661 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f487419a5d0 con 0x7f48741074e0 2024-03-21T10:43:32.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.971+0000 7f4871600640 1 -- 172.21.15.31:0/1413617661 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f486c005020 con 0x7f48741074e0 2024-03-21T10:43:32.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.971+0000 7f4871600640 1 -- 172.21.15.31:0/1413617661 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486c010720 con 0x7f48741074e0 2024-03-21T10:43:32.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.983+0000 7f487e997740 1 Processor -- start 2024-03-21T10:43:32.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.983+0000 7f487e997740 1 -- start start 2024-03-21T10:43:32.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.983+0000 7f4871600640 1 -- 172.21.15.31:0/1413617661 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f486c010960 con 0x7f48741074e0 2024-03-21T10:43:32.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.984+0000 7f487e997740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5d2b77290 0x55f5d2b77670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:32.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.985+0000 7f487e997740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5d2b77bb0 0x55f5d2b80250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:32.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.986+0000 7f4879200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5d2b77bb0 0x55f5d2b80250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:32.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.986+0000 7f487e997740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5d2b80790 0x55f5d2b82b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:32.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.986+0000 7f487e997740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5d2adbeb0 con 0x55f5d2b77bb0 2024-03-21T10:43:32.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.986+0000 7f487e997740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5d2a84a60 con 0x55f5d2b80790 2024-03-21T10:43:32.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.986+0000 7f487e997740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5d2ad7f50 con 0x55f5d2b77290 2024-03-21T10:43:32.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.987+0000 7f4879200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5d2b77bb0 0x55f5d2b80250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49176/0 (socket says 172.21.15.31:49176) 2024-03-21T10:43:32.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.987+0000 7f4879200640 1 -- 172.21.15.31:0/4281764717 learned_addr learned my addr 172.21.15.31:0/4281764717 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:32.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.987+0000 7f4871600640 1 --2- 172.21.15.31:0/1413617661 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f485c0a3190 0x7f485c0a5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:32.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.988+0000 7f4879200640 1 -- 172.21.15.31:0/4281764717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5d2b77290 msgr2=0x55f5d2b77670 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:43:32.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.989+0000 7f4879200640 1 --2- 172.21.15.31:0/4281764717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5d2b77290 0x55f5d2b77670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:32.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.989+0000 7f4879200640 1 -- 172.21.15.31:0/4281764717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5d2b80790 msgr2=0x55f5d2b82b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:43:32.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.989+0000 7f4879200640 1 --2- 172.21.15.31:0/4281764717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5d2b80790 0x55f5d2b82b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:32.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.989+0000 7f4871600640 1 -- 172.21.15.31:0/1413617661 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f486c0d0470 con 0x7f48741074e0 2024-03-21T10:43:32.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.989+0000 7f4879200640 1 -- 172.21.15.31:0/4281764717 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f5d28b87e0 con 0x55f5d2b77bb0 2024-03-21T10:43:32.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.989+0000 7f4879c00640 1 --2- 172.21.15.31:0/1413617661 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f485c0a3190 0x7f485c0a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:32.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.990+0000 7f4879200640 1 --2- 172.21.15.31:0/4281764717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5d2b77bb0 0x55f5d2b80250 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f486800aa10 tx=0x7f486800aed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1da668c239b62fca server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:32.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.990+0000 7f4879c00640 1 --2- 172.21.15.31:0/1413617661 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f485c0a3190 0x7f485c0a5650 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f486c09d100 tx=0x7f486c00c890 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:32.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.991+0000 7f4866000640 1 -- 172.21.15.31:0/4281764717 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486800cde0 con 0x55f5d2b77bb0 2024-03-21T10:43:32.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.991+0000 7f4866000640 1 -- 172.21.15.31:0/4281764717 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48680035f0 con 0x55f5d2b77bb0 2024-03-21T10:43:32.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.991+0000 7f4866000640 1 -- 172.21.15.31:0/4281764717 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486800f6f0 con 0x55f5d2b77bb0 2024-03-21T10:43:32.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.992+0000 7f487e997740 1 -- 172.21.15.31:0/4281764717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5d2b77bb0 msgr2=0x55f5d2b80250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:32.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.992+0000 7f487e997740 1 --2- 172.21.15.31:0/4281764717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5d2b77bb0 0x55f5d2b80250 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f486800aa10 tx=0x7f486800aed0 comp rx=0 tx=0).stop 2024-03-21T10:43:32.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.992+0000 7f487e997740 1 -- 172.21.15.31:0/4281764717 shutdown_connections 2024-03-21T10:43:32.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.992+0000 7f487e997740 1 --2- 172.21.15.31:0/4281764717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5d2b80790 0x55f5d2b82b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:32.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.992+0000 7f487e997740 1 --2- 172.21.15.31:0/4281764717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5d2b77bb0 0x55f5d2b80250 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:32.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.992+0000 7f487e997740 1 --2- 172.21.15.31:0/4281764717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5d2b77290 0x55f5d2b77670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:32.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:32.992+0000 7f487e997740 1 -- 172.21.15.31:0/4281764717 >> 172.21.15.31:0/4281764717 conn(0x55f5d2b719c0 msgr2=0x55f5d2b71da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:33.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.014+0000 7f487e997740 1 -- 172.21.15.31:0/4281764717 shutdown_connections 2024-03-21T10:43:33.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.014+0000 7f487e997740 1 -- 172.21.15.31:0/4281764717 wait complete. 2024-03-21T10:43:33.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.015+0000 7f487e997740 1 -- start start 2024-03-21T10:43:33.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.015+0000 7f487e997740 1 Processor -- start 2024-03-21T10:43:33.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.017+0000 7f487e997740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5d2b77bb0 0x55f5d2b80220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.017+0000 7f4873e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5d2b77bb0 0x55f5d2b80220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:33.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.017+0000 7f4873e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5d2b77bb0 0x55f5d2b80220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42376/0 (socket says 172.21.15.31:42376) 2024-03-21T10:43:33.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.017+0000 7f4873e00640 1 -- 172.21.15.31:0/4286499873 learned_addr learned my addr 172.21.15.31:0/4286499873 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:33.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.017+0000 7f487e997740 1 --2- 172.21.15.31:0/4286499873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5d2b7d510 0x55f5d2b7d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.017+0000 7f487e997740 1 --2- 172.21.15.31:0/4286499873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5d2b7dfe0 0x55f5d2b8cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.017+0000 7f487e997740 1 -- 172.21.15.31:0/4286499873 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5d2b83580 con 0x55f5d2b7d510 2024-03-21T10:43:33.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.018+0000 7f487e997740 1 -- 172.21.15.31:0/4286499873 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5d2a97eb0 con 0x55f5d2b7dfe0 2024-03-21T10:43:33.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.018+0000 7f487e997740 1 -- 172.21.15.31:0/4286499873 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5d2b83700 con 0x55f5d2b77bb0 2024-03-21T10:43:33.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.018+0000 7f487e997740 10 client.0 _ll_register_callbacks cb 0x55f5d2a8c150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:43:33.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.019+0000 7f4873e00640 1 --2- 172.21.15.31:0/4286499873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5d2b7dfe0 0x55f5d2b8cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:33.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.019+0000 7f4873e00640 1 -- 172.21.15.31:0/4286499873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5d2b7dfe0 msgr2=0x55f5d2b8cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:33.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.019+0000 7f4873e00640 1 --2- 172.21.15.31:0/4286499873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5d2b7dfe0 0x55f5d2b8cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.019+0000 7f4873e00640 1 -- 172.21.15.31:0/4286499873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5d2b7d510 msgr2=0x55f5d2b7d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:33.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.019+0000 7f4873e00640 1 --2- 172.21.15.31:0/4286499873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5d2b7d510 0x55f5d2b7d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.019+0000 7f4873e00640 1 -- 172.21.15.31:0/4286499873 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f5d2918f70 con 0x55f5d2b77bb0 2024-03-21T10:43:33.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.022+0000 7f485aa00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:43:33.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.022+0000 7f4873e00640 1 --2- 172.21.15.31:0/4286499873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5d2b77bb0 0x55f5d2b80220 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f4860006010 tx=0x7f4860030870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36d0065c4b63e7c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:33.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.022+0000 7f485aa00640 1 -- 172.21.15.31:0/4286499873 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48600046c0 con 0x55f5d2b77bb0 2024-03-21T10:43:33.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.023+0000 7f485aa00640 1 -- 172.21.15.31:0/4286499873 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f486000f5d0 con 0x55f5d2b77bb0 2024-03-21T10:43:33.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.023+0000 7f487e997740 1 -- 172.21.15.31:0/4286499873 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f48680035f0 con 0x55f5d2b77bb0 2024-03-21T10:43:33.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.023+0000 7f485aa00640 1 -- 172.21.15.31:0/4286499873 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48600046c0 con 0x55f5d2b77bb0 2024-03-21T10:43:33.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.023+0000 7f487e997740 10 client.26836 fetch_fsmap learned FSMap version 26 2024-03-21T10:43:33.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.023+0000 7f487e997740 1 -- 172.21.15.31:0/4286499873 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55f5d27b0170 con 0x55f5d2b77bb0 2024-03-21T10:43:33.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.024+0000 7f485aa00640 1 -- 172.21.15.31:0/4286499873 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f486003e040 con 0x55f5d2b77bb0 2024-03-21T10:43:33.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.026+0000 7f485aa00640 1 -- 172.21.15.31:0/4286499873 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f4860044870 con 0x55f5d2b77bb0 2024-03-21T10:43:33.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.026+0000 7f487e997740 10 client.26836 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:43:33.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.026+0000 7f487e997740 10 client.26836 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:43:33.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.026+0000 7f487e997740 20 client.26836 populate_metadata read hostname 'smithi031' 2024-03-21T10:43:33.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.026+0000 7f487e997740 1 --2- 172.21.15.31:0/4286499873 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f5d2b911d0 0x55f5d2b93610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.026+0000 7f487e997740 4 client.26836 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:43:33.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.026+0000 7f487e997740 1 -- 172.21.15.31:0/4286499873 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f48680035f0 con 0x55f5d2b911d0 2024-03-21T10:43:33.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.026+0000 7f4879200640 1 --2- 172.21.15.31:0/4286499873 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f5d2b911d0 0x55f5d2b93610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:33.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.028+0000 7f4879200640 1 --2- 172.21.15.31:0/4286499873 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f5d2b911d0 0x55f5d2b93610 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a3e2bccc664a7f5 server_cookie=fe29677627ed6299 in_seq=0 out_seq=0 2024-03-21T10:43:33.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.028+0000 7f485aa00640 10 client.26836 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:33.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.028+0000 7f485aa00640 1 -- 172.21.15.31:0/4286499873 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7f48680035f0 con 0x55f5d2b911d0 2024-03-21T10:43:33.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.028+0000 7f485aa00640 10 client.26836 handle_command_reply: tid=0 2024-03-21T10:43:33.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.029+0000 7f487e997740 1 client.26836 shutdown 2024-03-21T10:43:33.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.029+0000 7f487e997740 1 -- 172.21.15.31:0/4286499873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5d2b77bb0 msgr2=0x55f5d2b80220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:33.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.030+0000 7f487e997740 1 --2- 172.21.15.31:0/4286499873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5d2b77bb0 0x55f5d2b80220 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f4860006010 tx=0x7f4860030870 comp rx=0 tx=0).stop 2024-03-21T10:43:33.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.030+0000 7f487e997740 1 -- 172.21.15.31:0/4286499873 shutdown_connections 2024-03-21T10:43:33.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.030+0000 7f487e997740 1 --2- 172.21.15.31:0/4286499873 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f5d2b911d0 0x55f5d2b93610 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.030+0000 7f487e997740 1 --2- 172.21.15.31:0/4286499873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5d2b7dfe0 0x55f5d2b8cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.030+0000 7f487e997740 1 --2- 172.21.15.31:0/4286499873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5d2b7d510 0x55f5d2b7d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.030+0000 7f487e997740 1 --2- 172.21.15.31:0/4286499873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5d2b77bb0 0x55f5d2b80220 secure :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f4860006010 tx=0x7f4860030870 comp rx=0 tx=0).stop 2024-03-21T10:43:33.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.030+0000 7f487e997740 1 -- 172.21.15.31:0/4286499873 >> 172.21.15.31:0/4286499873 conn(0x55f5d2b719c0 msgr2=0x55f5d2b73710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:33.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.030+0000 7f485aa00640 0 client.26836 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:33.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.030+0000 7f487e997740 1 -- 172.21.15.31:0/4286499873 shutdown_connections 2024-03-21T10:43:33.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.032+0000 7f487e997740 1 -- 172.21.15.31:0/4286499873 wait complete. 2024-03-21T10:43:33.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.033+0000 7f487e997740 20 client.26836 trim_cache size 0 max 16384 2024-03-21T10:43:33.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:32 smithi031 ceph-mon[31673]: pgmap v152: 129 pgs: 129 active+clean; 28 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 189 MiB/s wr, 1.36k op/s 2024-03-21T10:43:33.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.075+0000 7f487e997740 1 Processor -- start 2024-03-21T10:43:33.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.076+0000 7f487e997740 1 -- start start 2024-03-21T10:43:33.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.076+0000 7f487e997740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5d2b7d510 0x55f5d2b7fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.077+0000 7f487e997740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5d2b7dfe0 0x55f5d2b70900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.077+0000 7f487e997740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5d2b98f20 0x55f5d2b70e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.077+0000 7f487e997740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4854012090 con 0x55f5d2b7dfe0 2024-03-21T10:43:33.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.077+0000 7f487e997740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485400fb90 con 0x55f5d2b98f20 2024-03-21T10:43:33.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.077+0000 7f487e997740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485400dce0 con 0x55f5d2b7d510 2024-03-21T10:43:33.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.077+0000 7f4873e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5d2b7d510 0x55f5d2b7fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:33.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.077+0000 7f4873e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5d2b7d510 0x55f5d2b7fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42390/0 (socket says 172.21.15.31:42390) 2024-03-21T10:43:33.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.078+0000 7f4873e00640 1 -- 172.21.15.31:0/2776812714 learned_addr learned my addr 172.21.15.31:0/2776812714 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:33.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.078+0000 7f4873e00640 1 --2- 172.21.15.31:0/2776812714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5d2b98f20 0x55f5d2b70e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:33.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.079+0000 7f4873e00640 1 -- 172.21.15.31:0/2776812714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5d2b98f20 msgr2=0x55f5d2b70e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:33.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.079+0000 7f4873e00640 1 --2- 172.21.15.31:0/2776812714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5d2b98f20 0x55f5d2b70e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.079+0000 7f4873e00640 1 -- 172.21.15.31:0/2776812714 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5d2b7dfe0 msgr2=0x55f5d2b70900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:33.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.079+0000 7f4873e00640 1 --2- 172.21.15.31:0/2776812714 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5d2b7dfe0 0x55f5d2b70900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.079+0000 7f4873e00640 1 -- 172.21.15.31:0/2776812714 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f5d2b787b0 con 0x55f5d2b7d510 2024-03-21T10:43:33.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.080+0000 7f4873e00640 1 --2- 172.21.15.31:0/2776812714 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5d2b7d510 0x55f5d2b7fd50 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f4860044ce0 tx=0x7f486002fdd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba4d0264c27f2c7c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:33.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.080+0000 7f4865600640 1 -- 172.21.15.31:0/2776812714 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4860052040 con 0x55f5d2b7d510 2024-03-21T10:43:33.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.081+0000 7f4865600640 1 -- 172.21.15.31:0/2776812714 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4860053040 con 0x55f5d2b7d510 2024-03-21T10:43:33.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.081+0000 7f4865600640 1 -- 172.21.15.31:0/2776812714 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4860052040 con 0x55f5d2b7d510 2024-03-21T10:43:33.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.083+0000 7f487e997740 1 -- 172.21.15.31:0/2776812714 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5d2b7d510 msgr2=0x55f5d2b7fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:33.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.083+0000 7f487e997740 1 --2- 172.21.15.31:0/2776812714 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5d2b7d510 0x55f5d2b7fd50 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f4860044ce0 tx=0x7f486002fdd0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.083+0000 7f487e997740 1 -- 172.21.15.31:0/2776812714 shutdown_connections 2024-03-21T10:43:33.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.083+0000 7f487e997740 1 --2- 172.21.15.31:0/2776812714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5d2b98f20 0x55f5d2b70e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.083+0000 7f487e997740 1 --2- 172.21.15.31:0/2776812714 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5d2b7dfe0 0x55f5d2b70900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.083+0000 7f487e997740 1 --2- 172.21.15.31:0/2776812714 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5d2b7d510 0x55f5d2b7fd50 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.083+0000 7f487e997740 1 -- 172.21.15.31:0/2776812714 >> 172.21.15.31:0/2776812714 conn(0x55f5d2b719c0 msgr2=0x55f5d2b701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:33.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.084+0000 7f487e997740 1 -- 172.21.15.31:0/2776812714 shutdown_connections 2024-03-21T10:43:33.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.084+0000 7f487e997740 1 -- 172.21.15.31:0/2776812714 wait complete. 2024-03-21T10:43:33.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.085+0000 7f487e997740 1 -- start start 2024-03-21T10:43:33.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.085+0000 7f487e997740 1 Processor -- start 2024-03-21T10:43:33.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.086+0000 7f487e997740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5d2b7d510 0x55f5d2b87550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.087+0000 7f487e997740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5d2b7dfe0 0x55f5d2b87a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.087+0000 7f487e997740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5d2b98f20 0x55f5d2becb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.087+0000 7f487e997740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485400e250 con 0x55f5d2b7d510 2024-03-21T10:43:33.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.087+0000 7f487e997740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485400f530 con 0x55f5d2b7dfe0 2024-03-21T10:43:33.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.087+0000 7f487e997740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485400e870 con 0x55f5d2b98f20 2024-03-21T10:43:33.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.087+0000 7f487e997740 10 client.0 _ll_register_callbacks cb 0x7f486000f990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:43:33.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.087+0000 7f4873e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5d2b7d510 0x55f5d2b87550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:33.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.087+0000 7f4879200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5d2b7dfe0 0x55f5d2b87a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:33.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.087+0000 7f4879200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5d2b7dfe0 0x55f5d2b87a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58286/0 (socket says 172.21.15.31:58286) 2024-03-21T10:43:33.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.087+0000 7f4879200640 1 -- 172.21.15.31:0/2369006942 learned_addr learned my addr 172.21.15.31:0/2369006942 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:33.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.088+0000 7f4879200640 1 -- 172.21.15.31:0/2369006942 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5d2b98f20 msgr2=0x55f5d2becb00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:43:33.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.088+0000 7f4879200640 1 --2- 172.21.15.31:0/2369006942 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5d2b98f20 0x55f5d2becb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.088+0000 7f4879200640 1 -- 172.21.15.31:0/2369006942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5d2b7d510 msgr2=0x55f5d2b87550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:33.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.088+0000 7f4879200640 1 --2- 172.21.15.31:0/2369006942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5d2b7d510 0x55f5d2b87550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.088+0000 7f4879200640 1 -- 172.21.15.31:0/2369006942 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f5d2bed040 con 0x55f5d2b7dfe0 2024-03-21T10:43:33.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.088+0000 7f4879200640 1 --2- 172.21.15.31:0/2369006942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5d2b7dfe0 0x55f5d2b87a90 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4868005fd0 tx=0x7f4868003d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57065b46e88d1763 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:33.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.088+0000 7f485aa00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:43:33.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.088+0000 7f485aa00640 1 -- 172.21.15.31:0/2369006942 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4868024080 con 0x55f5d2b7dfe0 2024-03-21T10:43:33.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.088+0000 7f485aa00640 1 -- 172.21.15.31:0/2369006942 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f4868024220 con 0x55f5d2b7dfe0 2024-03-21T10:43:33.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.088+0000 7f487e997740 1 -- 172.21.15.31:0/2369006942 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4860053040 con 0x55f5d2b7dfe0 2024-03-21T10:43:33.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.089+0000 7f485aa00640 1 -- 172.21.15.31:0/2369006942 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4868003fc0 con 0x55f5d2b7dfe0 2024-03-21T10:43:33.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.089+0000 7f487e997740 10 client.26918 fetch_fsmap learned FSMap version 26 2024-03-21T10:43:33.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.089+0000 7f487e997740 1 -- 172.21.15.31:0/2369006942 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55f5d2bee3f0 con 0x55f5d2b7dfe0 2024-03-21T10:43:33.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.089+0000 7f485aa00640 1 -- 172.21.15.31:0/2369006942 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4860053040 con 0x55f5d2b7dfe0 2024-03-21T10:43:33.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.090+0000 7f485aa00640 1 -- 172.21.15.31:0/2369006942 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f48680246e0 con 0x55f5d2b7dfe0 2024-03-21T10:43:33.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.090+0000 7f487e997740 10 client.26918 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:43:33.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.090+0000 7f487e997740 10 client.26918 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:43:33.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.090+0000 7f487e997740 20 client.26918 populate_metadata read hostname 'smithi031' 2024-03-21T10:43:33.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.090+0000 7f487e997740 1 --2- 172.21.15.31:0/2369006942 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f5d2bee990 0x55f5d2bf0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.090+0000 7f487e997740 4 client.26918 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-21T10:43:33.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.090+0000 7f487e997740 1 -- 172.21.15.31:0/2369006942 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f5d2ae93f0 con 0x55f5d2bee990 2024-03-21T10:43:33.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.090+0000 7f4873e00640 1 --2- 172.21.15.31:0/2369006942 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f5d2bee990 0x55f5d2bf0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:33.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.091+0000 7f4873e00640 1 --2- 172.21.15.31:0/2369006942 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f5d2bee990 0x55f5d2bf0db0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb9ae7cc5f6b7e4 server_cookie=3c4fcc214486e759 in_seq=0 out_seq=0 2024-03-21T10:43:33.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.091+0000 7f485aa00640 10 client.26918 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:33.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.093+0000 7f485aa00640 1 -- 172.21.15.31:0/2369006942 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f5d2ae93f0 con 0x55f5d2bee990 2024-03-21T10:43:33.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.093+0000 7f485aa00640 10 client.26918 handle_command_reply: tid=0 2024-03-21T10:43:33.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.093+0000 7f487e997740 1 client.26918 shutdown 2024-03-21T10:43:33.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.094+0000 7f487e997740 1 -- 172.21.15.31:0/2369006942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5d2b7dfe0 msgr2=0x55f5d2b87a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:33.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.094+0000 7f487e997740 1 --2- 172.21.15.31:0/2369006942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5d2b7dfe0 0x55f5d2b87a90 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4868005fd0 tx=0x7f4868003d00 comp rx=0 tx=0).stop 2024-03-21T10:43:33.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.094+0000 7f487e997740 1 -- 172.21.15.31:0/2369006942 shutdown_connections 2024-03-21T10:43:33.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.094+0000 7f487e997740 1 --2- 172.21.15.31:0/2369006942 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f5d2bee990 0x55f5d2bf0db0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.094+0000 7f487e997740 1 --2- 172.21.15.31:0/2369006942 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5d2b98f20 0x55f5d2becb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.094+0000 7f487e997740 1 --2- 172.21.15.31:0/2369006942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5d2b7dfe0 0x55f5d2b87a90 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.094+0000 7f487e997740 1 --2- 172.21.15.31:0/2369006942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5d2b7d510 0x55f5d2b87550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.094+0000 7f485aa00640 0 client.26918 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:33.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.094+0000 7f487e997740 1 -- 172.21.15.31:0/2369006942 >> 172.21.15.31:0/2369006942 conn(0x55f5d2b719c0 msgr2=0x55f5d2b987a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:33.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.094+0000 7f487e997740 1 -- 172.21.15.31:0/2369006942 shutdown_connections 2024-03-21T10:43:33.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.095+0000 7f487e997740 1 -- 172.21.15.31:0/2369006942 wait complete. 2024-03-21T10:43:33.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.095+0000 7f487e997740 20 client.26918 trim_cache size 0 max 16384 2024-03-21T10:43:33.093 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:43:33.093 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-03-21T10:43:33.093 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "bda1d8f6-d34f-404e-9843-17aa7c34bc1d", 2024-03-21T10:43:33.093 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-03-21T10:43:33.093 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:43:33.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.096+0000 7f487a600640 1 -- 172.21.15.31:0/1413617661 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f485c0a3190 msgr2=0x7f485c0a5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:33.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.096+0000 7f487a600640 1 --2- 172.21.15.31:0/1413617661 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f485c0a3190 0x7f485c0a5650 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f486c09d100 tx=0x7f486c00c890 comp rx=0 tx=0).stop 2024-03-21T10:43:33.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.096+0000 7f487a600640 1 -- 172.21.15.31:0/1413617661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48741074e0 msgr2=0x7f4874195a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:33.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.096+0000 7f487a600640 1 --2- 172.21.15.31:0/1413617661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48741074e0 0x7f4874195a80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f486c009fd0 tx=0x7f486c00ef10 comp rx=0 tx=0).stop 2024-03-21T10:43:33.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.100+0000 7f487a600640 1 -- 172.21.15.31:0/1413617661 shutdown_connections 2024-03-21T10:43:33.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.101+0000 7f487a600640 1 --2- 172.21.15.31:0/1413617661 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f485c0a3190 0x7f485c0a5650 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.101+0000 7f487a600640 1 --2- 172.21.15.31:0/1413617661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48741074e0 0x7f4874195a80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.101+0000 7f487a600640 1 --2- 172.21.15.31:0/1413617661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4874101e00 0x7f487419cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.101+0000 7f487a600640 1 --2- 172.21.15.31:0/1413617661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4874101440 0x7f487419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.101+0000 7f487a600640 1 -- 172.21.15.31:0/1413617661 >> 172.21.15.31:0/1413617661 conn(0x7f48740fcf50 msgr2=0x7f48740fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:33.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.102+0000 7f487a600640 1 -- 172.21.15.31:0/1413617661 shutdown_connections 2024-03-21T10:43:33.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.102+0000 7f487a600640 1 -- 172.21.15.31:0/1413617661 wait complete. 2024-03-21T10:43:33.132 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:43:33.133 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:43:33.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:32 smithi040 ceph-mon[36154]: pgmap v152: 129 pgs: 129 active+clean; 28 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 189 MiB/s wr, 1.36k op/s 2024-03-21T10:43:33.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:32 smithi070 ceph-mon[36330]: pgmap v152: 129 pgs: 129 active+clean; 28 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 189 MiB/s wr, 1.36k op/s 2024-03-21T10:43:33.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.424+0000 7f7871200640 1 -- 172.21.15.31:0/915267441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f786c109b70 msgr2=0x7f786c109f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:33.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.424+0000 7f7871200640 1 --2- 172.21.15.31:0/915267441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f786c109b70 0x7f786c109f50 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f785c009930 tx=0x7f785c02bc30 comp rx=0 tx=0).stop 2024-03-21T10:43:33.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.424+0000 7f7871200640 1 -- 172.21.15.31:0/915267441 shutdown_connections 2024-03-21T10:43:33.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.424+0000 7f7871200640 1 --2- 172.21.15.31:0/915267441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f786c108760 0x7f786c114c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.424+0000 7f7871200640 1 --2- 172.21.15.31:0/915267441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f786c107dc0 0x7f786c108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.424+0000 7f7871200640 1 --2- 172.21.15.31:0/915267441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f786c109b70 0x7f786c109f50 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.424+0000 7f7871200640 1 -- 172.21.15.31:0/915267441 >> 172.21.15.31:0/915267441 conn(0x7f786c077540 msgr2=0x7f786c1064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:33.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.425+0000 7f7871200640 1 -- 172.21.15.31:0/915267441 shutdown_connections 2024-03-21T10:43:33.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.425+0000 7f7871200640 1 -- 172.21.15.31:0/915267441 wait complete. 2024-03-21T10:43:33.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.426+0000 7f7871200640 1 Processor -- start 2024-03-21T10:43:33.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.426+0000 7f7871200640 1 -- start start 2024-03-21T10:43:33.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.427+0000 7f7871200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f786c107dc0 0x7f786c19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.427+0000 7f7871200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f786c108760 0x7f786c19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.427+0000 7f7871200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f786c109b70 0x7f786c1a2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.427+0000 7f7871200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c118150 con 0x7f786c108760 2024-03-21T10:43:33.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.427+0000 7f7871200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c117fd0 con 0x7f786c107dc0 2024-03-21T10:43:33.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.427+0000 7f7871200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c1182d0 con 0x7f786c109b70 2024-03-21T10:43:33.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.427+0000 7f786a800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f786c109b70 0x7f786c1a2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:33.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.427+0000 7f7869400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f786c108760 0x7f786c19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:33.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.427+0000 7f786a800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f786c109b70 0x7f786c1a2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42430/0 (socket says 172.21.15.31:42430) 2024-03-21T10:43:33.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.427+0000 7f786a800640 1 -- 172.21.15.31:0/2400449994 learned_addr learned my addr 172.21.15.31:0/2400449994 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:33.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.428+0000 7f7869e00640 1 --2- 172.21.15.31:0/2400449994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f786c107dc0 0x7f786c19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:33.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.428+0000 7f786a800640 1 -- 172.21.15.31:0/2400449994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f786c107dc0 msgr2=0x7f786c19e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:33.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.428+0000 7f786a800640 1 --2- 172.21.15.31:0/2400449994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f786c107dc0 0x7f786c19e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.428+0000 7f786a800640 1 -- 172.21.15.31:0/2400449994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f786c108760 msgr2=0x7f786c19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:33.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.428+0000 7f786a800640 1 --2- 172.21.15.31:0/2400449994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f786c108760 0x7f786c19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.428+0000 7f786a800640 1 -- 172.21.15.31:0/2400449994 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786c1a35f0 con 0x7f786c109b70 2024-03-21T10:43:33.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.428+0000 7f786a800640 1 --2- 172.21.15.31:0/2400449994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f786c109b70 0x7f786c1a2ec0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f786400da40 tx=0x7f786400df00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:33.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.428+0000 7f7862a00640 1 -- 172.21.15.31:0/2400449994 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7864017070 con 0x7f786c109b70 2024-03-21T10:43:33.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.428+0000 7f7862a00640 1 -- 172.21.15.31:0/2400449994 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7864003e50 con 0x7f786c109b70 2024-03-21T10:43:33.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.428+0000 7f7871200640 1 -- 172.21.15.31:0/2400449994 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f786c1a38e0 con 0x7f786c109b70 2024-03-21T10:43:33.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.429+0000 7f7862a00640 1 -- 172.21.15.31:0/2400449994 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7864010460 con 0x7f786c109b70 2024-03-21T10:43:33.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.429+0000 7f7871200640 1 -- 172.21.15.31:0/2400449994 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f786c1ab170 con 0x7f786c109b70 2024-03-21T10:43:33.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.432+0000 7f7862a00640 1 -- 172.21.15.31:0/2400449994 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f7864003970 con 0x7f786c109b70 2024-03-21T10:43:33.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.434+0000 7f7862a00640 1 --2- 172.21.15.31:0/2400449994 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f78500a3110 0x7f78500a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.435+0000 7f7873f82740 1 Processor -- start 2024-03-21T10:43:33.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.435+0000 7f7862a00640 1 -- 172.21.15.31:0/2400449994 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f78640cf770 con 0x7f786c109b70 2024-03-21T10:43:33.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.435+0000 7f7873f82740 1 -- start start 2024-03-21T10:43:33.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.435+0000 7f7873f82740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557461eb9c00 0x557461eb8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.435+0000 7f7873f82740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557461eb8f20 0x557461eb93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.435+0000 7f7873f82740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557461ec2290 0x557461ec4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.435+0000 7f7873f82740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557461e1d250 con 0x557461eb9c00 2024-03-21T10:43:33.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.435+0000 7f7873f82740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557461dc5a30 con 0x557461ec2290 2024-03-21T10:43:33.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.435+0000 7f7873f82740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x557461e192f0 con 0x557461eb8f20 2024-03-21T10:43:33.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.435+0000 7f7869e00640 1 --2- 172.21.15.31:0/2400449994 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f78500a3110 0x7f78500a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:33.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.436+0000 7f786a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557461ec2290 0x557461ec4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:33.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.436+0000 7f786a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557461ec2290 0x557461ec4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58316/0 (socket says 172.21.15.31:58316) 2024-03-21T10:43:33.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.436+0000 7f786a800640 1 -- 172.21.15.31:0/874895111 learned_addr learned my addr 172.21.15.31:0/874895111 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:33.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.436+0000 7f7869400640 1 --2- 172.21.15.31:0/874895111 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557461eb8f20 0x557461eb93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:33.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.436+0000 7f786a800640 1 -- 172.21.15.31:0/874895111 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557461eb8f20 msgr2=0x557461eb93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:33.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.436+0000 7f786a800640 1 --2- 172.21.15.31:0/874895111 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557461eb8f20 0x557461eb93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.436+0000 7f786a800640 1 -- 172.21.15.31:0/874895111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557461eb9c00 msgr2=0x557461eb8950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:43:33.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.436+0000 7f786a800640 1 --2- 172.21.15.31:0/874895111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557461eb9c00 0x557461eb8950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.436+0000 7f786a800640 1 -- 172.21.15.31:0/874895111 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557461db0690 con 0x557461ec2290 2024-03-21T10:43:33.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.436+0000 7f786a800640 1 --2- 172.21.15.31:0/874895111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557461ec2290 0x557461ec4680 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7864098190 tx=0x7f786409b120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81106851b4222bfd server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:33.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.437+0000 7f785b400640 1 -- 172.21.15.31:0/874895111 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7864020070 con 0x557461ec2290 2024-03-21T10:43:33.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.437+0000 7f785b400640 1 -- 172.21.15.31:0/874895111 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f786409b7c0 con 0x557461ec2290 2024-03-21T10:43:33.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.437+0000 7f785b400640 1 -- 172.21.15.31:0/874895111 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78640ccda0 con 0x557461ec2290 2024-03-21T10:43:33.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.438+0000 7f7873f82740 1 -- 172.21.15.31:0/874895111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557461ec2290 msgr2=0x557461ec4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:33.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.438+0000 7f7873f82740 1 --2- 172.21.15.31:0/874895111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557461ec2290 0x557461ec4680 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7864098190 tx=0x7f786409b120 comp rx=0 tx=0).stop 2024-03-21T10:43:33.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.438+0000 7f7869e00640 1 --2- 172.21.15.31:0/2400449994 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f78500a3110 0x7f78500a55d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f785c036040 tx=0x7f785c02df90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:33.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.438+0000 7f7873f82740 1 -- 172.21.15.31:0/874895111 shutdown_connections 2024-03-21T10:43:33.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.438+0000 7f7873f82740 1 --2- 172.21.15.31:0/874895111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557461ec2290 0x557461ec4680 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.438+0000 7f7873f82740 1 --2- 172.21.15.31:0/874895111 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557461eb8f20 0x557461eb93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.438+0000 7f7873f82740 1 --2- 172.21.15.31:0/874895111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557461eb9c00 0x557461eb8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.438+0000 7f7873f82740 1 -- 172.21.15.31:0/874895111 >> 172.21.15.31:0/874895111 conn(0x557461e80200 msgr2=0x557461eb3970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:33.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.438+0000 7f7873f82740 1 -- 172.21.15.31:0/874895111 shutdown_connections 2024-03-21T10:43:33.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.438+0000 7f7873f82740 1 -- 172.21.15.31:0/874895111 wait complete. 2024-03-21T10:43:33.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.439+0000 7f7873f82740 1 -- start start 2024-03-21T10:43:33.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.439+0000 7f7873f82740 1 Processor -- start 2024-03-21T10:43:33.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.439+0000 7f7873f82740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557461eb8f20 0x557461ec1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.440+0000 7f7873f82740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557461eb9c00 0x557461ebeca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.440+0000 7f7873f82740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557461ec2290 0x557461ebf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.440+0000 7f7873f82740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557461ec50a0 con 0x557461ec2290 2024-03-21T10:43:33.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.440+0000 7f7873f82740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557461dd8e80 con 0x557461eb8f20 2024-03-21T10:43:33.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.440+0000 7f7873f82740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x557461ec5220 con 0x557461eb9c00 2024-03-21T10:43:33.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.440+0000 7f7873f82740 10 client.0 _ll_register_callbacks cb 0x557461deb130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:43:33.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.440+0000 7f7869e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557461eb8f20 0x557461ec1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:33.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.440+0000 7f7869e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557461eb8f20 0x557461ec1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58320/0 (socket says 172.21.15.31:58320) 2024-03-21T10:43:33.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.440+0000 7f7869e00640 1 -- 172.21.15.31:0/3651543468 learned_addr learned my addr 172.21.15.31:0/3651543468 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:33.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.441+0000 7f7869e00640 1 -- 172.21.15.31:0/3651543468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557461eb9c00 msgr2=0x557461ebeca0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:43:33.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.441+0000 7f7869e00640 1 --2- 172.21.15.31:0/3651543468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557461eb9c00 0x557461ebeca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.441+0000 7f7869e00640 1 -- 172.21.15.31:0/3651543468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557461ec2290 msgr2=0x557461ebf1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:33.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.441+0000 7f7869e00640 1 --2- 172.21.15.31:0/3651543468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557461ec2290 0x557461ebf1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.441+0000 7f7869e00640 1 -- 172.21.15.31:0/3651543468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557461bfb7d0 con 0x557461eb8f20 2024-03-21T10:43:33.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.442+0000 7f7869e00640 1 --2- 172.21.15.31:0/3651543468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557461eb8f20 0x557461ec1910 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f785c035000 tx=0x7f785c005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca9885cc1c917145 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:33.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.442+0000 7f784fe00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:43:33.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.443+0000 7f784fe00640 1 -- 172.21.15.31:0/3651543468 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785c050060 con 0x557461eb8f20 2024-03-21T10:43:33.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.443+0000 7f784fe00640 1 -- 172.21.15.31:0/3651543468 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f785c03a040 con 0x557461eb8f20 2024-03-21T10:43:33.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.444+0000 7f7873f82740 1 -- 172.21.15.31:0/3651543468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f786409b7c0 con 0x557461eb8f20 2024-03-21T10:43:33.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.444+0000 7f784fe00640 1 -- 172.21.15.31:0/3651543468 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f785c02cbc0 con 0x557461eb8f20 2024-03-21T10:43:33.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.444+0000 7f7873f82740 10 client.26927 fetch_fsmap learned FSMap version 26 2024-03-21T10:43:33.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.444+0000 7f7873f82740 1 -- 172.21.15.31:0/3651543468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x557461b313f0 con 0x557461eb8f20 2024-03-21T10:43:33.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.447+0000 7f784fe00640 1 -- 172.21.15.31:0/3651543468 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f786409b7c0 con 0x557461eb8f20 2024-03-21T10:43:33.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.447+0000 7f784fe00640 1 -- 172.21.15.31:0/3651543468 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f785c003960 con 0x557461eb8f20 2024-03-21T10:43:33.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.448+0000 7f7873f82740 10 client.26927 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:43:33.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.448+0000 7f7873f82740 10 client.26927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:43:33.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.448+0000 7f7873f82740 20 client.26927 populate_metadata read hostname 'smithi031' 2024-03-21T10:43:33.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.448+0000 7f7873f82740 1 --2- 172.21.15.31:0/3651543468 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x557461ed20d0 0x557461ed44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.448+0000 7f7873f82740 4 client.26927 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:43:33.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.448+0000 7f7873f82740 1 -- 172.21.15.31:0/3651543468 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557461e2a790 con 0x557461ed20d0 2024-03-21T10:43:33.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.449+0000 7f7869400640 1 --2- 172.21.15.31:0/3651543468 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x557461ed20d0 0x557461ed44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:33.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.450+0000 7f7869400640 1 --2- 172.21.15.31:0/3651543468 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x557461ed20d0 0x557461ed44c0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b42f6e065b0fe7b server_cookie=c7071bd9ab51b603 in_seq=0 out_seq=0 2024-03-21T10:43:33.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.450+0000 7f784fe00640 10 client.26927 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:33.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.450+0000 7f784fe00640 1 -- 172.21.15.31:0/3651543468 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557461e2a790 con 0x557461ed20d0 2024-03-21T10:43:33.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.450+0000 7f784fe00640 10 client.26927 handle_command_reply: tid=0 2024-03-21T10:43:33.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.450+0000 7f7873f82740 1 client.26927 shutdown 2024-03-21T10:43:33.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.451+0000 7f7873f82740 1 -- 172.21.15.31:0/3651543468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557461eb8f20 msgr2=0x557461ec1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:33.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.451+0000 7f7873f82740 1 --2- 172.21.15.31:0/3651543468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557461eb8f20 0x557461ec1910 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f785c035000 tx=0x7f785c005fd0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.451+0000 7f7873f82740 1 -- 172.21.15.31:0/3651543468 shutdown_connections 2024-03-21T10:43:33.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.451+0000 7f7873f82740 1 --2- 172.21.15.31:0/3651543468 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x557461ed20d0 0x557461ed44c0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.451+0000 7f7873f82740 1 --2- 172.21.15.31:0/3651543468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557461ec2290 0x557461ebf1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.451+0000 7f7873f82740 1 --2- 172.21.15.31:0/3651543468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557461eb9c00 0x557461ebeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.451+0000 7f7873f82740 1 --2- 172.21.15.31:0/3651543468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557461eb8f20 0x557461ec1910 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.451+0000 7f7873f82740 1 -- 172.21.15.31:0/3651543468 >> 172.21.15.31:0/3651543468 conn(0x557461e80200 msgr2=0x557461eb56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:33.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.451+0000 7f784fe00640 0 client.26927 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:33.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.452+0000 7f7873f82740 1 -- 172.21.15.31:0/3651543468 shutdown_connections 2024-03-21T10:43:33.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.452+0000 7f7873f82740 1 -- 172.21.15.31:0/3651543468 wait complete. 2024-03-21T10:43:33.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.452+0000 7f7873f82740 20 client.26927 trim_cache size 0 max 16384 2024-03-21T10:43:33.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.498+0000 7f7873f82740 1 Processor -- start 2024-03-21T10:43:33.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.498+0000 7f7873f82740 1 -- start start 2024-03-21T10:43:33.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.498+0000 7f7873f82740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557461eb8f20 0x557461ec13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.499+0000 7f7873f82740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557461eb9c00 0x557461ec1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.499+0000 7f7873f82740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557461ec2290 0x557461ec3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.499+0000 7f7873f82740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c1b74f0 con 0x557461ec2290 2024-03-21T10:43:33.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.499+0000 7f7873f82740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c1b4ff0 con 0x557461eb9c00 2024-03-21T10:43:33.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.499+0000 7f7873f82740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c1b3090 con 0x557461eb8f20 2024-03-21T10:43:33.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.499+0000 7f786a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557461ec2290 0x557461ec3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:33.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.500+0000 7f7869e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557461eb8f20 0x557461ec13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:33.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.500+0000 7f7869e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557461eb8f20 0x557461ec13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42458/0 (socket says 172.21.15.31:42458) 2024-03-21T10:43:33.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.500+0000 7f7869e00640 1 -- 172.21.15.31:0/2820138182 learned_addr learned my addr 172.21.15.31:0/2820138182 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:33.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.500+0000 7f7869400640 1 --2- 172.21.15.31:0/2820138182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557461eb9c00 0x557461ec1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:33.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.501+0000 7f7869e00640 1 -- 172.21.15.31:0/2820138182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557461eb9c00 msgr2=0x557461ec1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:33.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.501+0000 7f7869e00640 1 --2- 172.21.15.31:0/2820138182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557461eb9c00 0x557461ec1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.501+0000 7f7869e00640 1 -- 172.21.15.31:0/2820138182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557461ec2290 msgr2=0x557461ec3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:33.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.501+0000 7f7869e00640 1 --2- 172.21.15.31:0/2820138182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557461ec2290 0x557461ec3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.501+0000 7f7869e00640 1 -- 172.21.15.31:0/2820138182 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557461af10f0 con 0x557461eb8f20 2024-03-21T10:43:33.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.501+0000 7f7869e00640 1 --2- 172.21.15.31:0/2820138182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557461eb8f20 0x557461ec13d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f785c02c170 tx=0x7f785c003f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53871cf4c8e11a19 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:33.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.503+0000 7f785aa00640 1 -- 172.21.15.31:0/2820138182 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785c050070 con 0x557461eb8f20 2024-03-21T10:43:33.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.503+0000 7f785aa00640 1 -- 172.21.15.31:0/2820138182 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f785c003ad0 con 0x557461eb8f20 2024-03-21T10:43:33.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.503+0000 7f785aa00640 1 -- 172.21.15.31:0/2820138182 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785c050070 con 0x557461eb8f20 2024-03-21T10:43:33.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.504+0000 7f7873f82740 1 -- 172.21.15.31:0/2820138182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557461eb8f20 msgr2=0x557461ec13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:33.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.504+0000 7f7873f82740 1 --2- 172.21.15.31:0/2820138182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557461eb8f20 0x557461ec13d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f785c02c170 tx=0x7f785c003f20 comp rx=0 tx=0).stop 2024-03-21T10:43:33.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.508+0000 7f7873f82740 1 -- 172.21.15.31:0/2820138182 shutdown_connections 2024-03-21T10:43:33.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.508+0000 7f7873f82740 1 --2- 172.21.15.31:0/2820138182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557461ec2290 0x557461ec3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.508+0000 7f7873f82740 1 --2- 172.21.15.31:0/2820138182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557461eb9c00 0x557461ec1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.508+0000 7f7873f82740 1 --2- 172.21.15.31:0/2820138182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557461eb8f20 0x557461ec13d0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.508+0000 7f7873f82740 1 -- 172.21.15.31:0/2820138182 >> 172.21.15.31:0/2820138182 conn(0x557461e80200 msgr2=0x557461eb34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:33.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.508+0000 7f7873f82740 1 -- 172.21.15.31:0/2820138182 shutdown_connections 2024-03-21T10:43:33.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.509+0000 7f7873f82740 1 -- 172.21.15.31:0/2820138182 wait complete. 2024-03-21T10:43:33.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.510+0000 7f7873f82740 1 -- start start 2024-03-21T10:43:33.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.511+0000 7f7873f82740 1 Processor -- start 2024-03-21T10:43:33.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.512+0000 7f7873f82740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557461eb8f20 0x557461ec5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.512+0000 7f7873f82740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557461eb9c00 0x557461ec6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.512+0000 7f7873f82740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557461ec2290 0x557461f3d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.512+0000 7f7873f82740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c1b3690 con 0x557461eb8f20 2024-03-21T10:43:33.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.512+0000 7f7873f82740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c1b4990 con 0x557461eb9c00 2024-03-21T10:43:33.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.512+0000 7f7873f82740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c1b3cd0 con 0x557461ec2290 2024-03-21T10:43:33.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.512+0000 7f7873f82740 10 client.0 _ll_register_callbacks cb 0x7f7854005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:43:33.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.513+0000 7f7869400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557461eb9c00 0x557461ec6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:33.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.513+0000 7f7869400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557461eb9c00 0x557461ec6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58344/0 (socket says 172.21.15.31:58344) 2024-03-21T10:43:33.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.513+0000 7f786a800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557461ec2290 0x557461f3d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:33.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.513+0000 7f786a800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557461ec2290 0x557461f3d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42462/0 (socket says 172.21.15.31:42462) 2024-03-21T10:43:33.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.513+0000 7f786a800640 1 -- 172.21.15.31:0/666891225 learned_addr learned my addr 172.21.15.31:0/666891225 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:33.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.514+0000 7f7869e00640 1 --2- 172.21.15.31:0/666891225 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557461eb8f20 0x557461ec5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:33.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.517+0000 7f7869400640 1 -- 172.21.15.31:0/666891225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557461ec2290 msgr2=0x557461f3d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:33.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.517+0000 7f7869400640 1 --2- 172.21.15.31:0/666891225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557461ec2290 0x557461f3d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.517+0000 7f7869400640 1 -- 172.21.15.31:0/666891225 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557461eb8f20 msgr2=0x557461ec5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:33.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.517+0000 7f7869400640 1 --2- 172.21.15.31:0/666891225 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557461eb8f20 0x557461ec5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.517+0000 7f7869400640 1 -- 172.21.15.31:0/666891225 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557461ec69d0 con 0x557461eb9c00 2024-03-21T10:43:33.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.517+0000 7f786a800640 1 --2- 172.21.15.31:0/666891225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557461ec2290 0x557461f3d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:43:33.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.517+0000 7f7869e00640 1 --2- 172.21.15.31:0/666891225 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557461eb8f20 0x557461ec5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:43:33.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.517+0000 7f7869400640 1 --2- 172.21.15.31:0/666891225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557461eb9c00 0x557461ec6100 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x557461ed4cc0 tx=0x7f7854004410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69f3928bc46773bf server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:33.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.517+0000 7f784fe00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:43:33.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.518+0000 7f784fe00640 1 -- 172.21.15.31:0/666891225 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7854021070 con 0x557461eb9c00 2024-03-21T10:43:33.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.518+0000 7f784fe00640 1 -- 172.21.15.31:0/666891225 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f785401d680 con 0x557461eb9c00 2024-03-21T10:43:33.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.518+0000 7f7873f82740 1 -- 172.21.15.31:0/666891225 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f785c003ad0 con 0x557461eb9c00 2024-03-21T10:43:33.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.518+0000 7f784fe00640 1 -- 172.21.15.31:0/666891225 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f785402b050 con 0x557461eb9c00 2024-03-21T10:43:33.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.518+0000 7f7873f82740 10 client.26933 fetch_fsmap learned FSMap version 26 2024-03-21T10:43:33.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.519+0000 7f7873f82740 1 -- 172.21.15.31:0/666891225 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x557461f4ffb0 con 0x557461eb9c00 2024-03-21T10:43:33.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.519+0000 7f784fe00640 1 -- 172.21.15.31:0/666891225 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f785c003ad0 con 0x557461eb9c00 2024-03-21T10:43:33.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.521+0000 7f784fe00640 1 -- 172.21.15.31:0/666891225 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f7854026a50 con 0x557461eb9c00 2024-03-21T10:43:33.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.521+0000 7f7873f82740 10 client.26933 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:43:33.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.521+0000 7f7873f82740 10 client.26933 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:43:33.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.521+0000 7f7873f82740 20 client.26933 populate_metadata read hostname 'smithi031' 2024-03-21T10:43:33.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.521+0000 7f7873f82740 1 --2- 172.21.15.31:0/666891225 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x557461f50630 0x557461f52a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:33.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.521+0000 7f7873f82740 4 client.26933 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:43:33.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.521+0000 7f7873f82740 1 -- 172.21.15.31:0/666891225 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557461e24ae0 con 0x557461f50630 2024-03-21T10:43:33.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.522+0000 7f7869e00640 1 --2- 172.21.15.31:0/666891225 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x557461f50630 0x557461f52a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:33.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.525+0000 7f784fe00640 10 client.26933 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:33.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.525+0000 7f7869e00640 1 --2- 172.21.15.31:0/666891225 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x557461f50630 0x557461f52a20 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8abb3081e42991c server_cookie=2e308b6e31624a79 in_seq=0 out_seq=0 2024-03-21T10:43:33.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.526+0000 7f784fe00640 1 -- 172.21.15.31:0/666891225 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x557461e24ae0 con 0x557461f50630 2024-03-21T10:43:33.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.526+0000 7f784fe00640 10 client.26933 handle_command_reply: tid=0 2024-03-21T10:43:33.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.526+0000 7f7873f82740 1 client.26933 shutdown 2024-03-21T10:43:33.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.529+0000 7f7873f82740 1 -- 172.21.15.31:0/666891225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557461eb9c00 msgr2=0x557461ec6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:33.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.529+0000 7f7873f82740 1 --2- 172.21.15.31:0/666891225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557461eb9c00 0x557461ec6100 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x557461ed4cc0 tx=0x7f7854004410 comp rx=0 tx=0).stop 2024-03-21T10:43:33.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.531+0000 7f7873f82740 1 -- 172.21.15.31:0/666891225 shutdown_connections 2024-03-21T10:43:33.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.531+0000 7f7873f82740 1 --2- 172.21.15.31:0/666891225 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x557461f50630 0x557461f52a20 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.531+0000 7f7873f82740 1 --2- 172.21.15.31:0/666891225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557461ec2290 0x557461f3d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.531+0000 7f7873f82740 1 --2- 172.21.15.31:0/666891225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557461eb9c00 0x557461ec6100 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.531+0000 7f7873f82740 1 --2- 172.21.15.31:0/666891225 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557461eb8f20 0x557461ec5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.531+0000 7f7873f82740 1 -- 172.21.15.31:0/666891225 >> 172.21.15.31:0/666891225 conn(0x557461e80200 msgr2=0x557461ebd530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:33.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.531+0000 7f784fe00640 0 client.26933 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:33.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.533+0000 7f7873f82740 1 -- 172.21.15.31:0/666891225 shutdown_connections 2024-03-21T10:43:33.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.533+0000 7f7873f82740 1 -- 172.21.15.31:0/666891225 wait complete. 2024-03-21T10:43:33.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.533+0000 7f7873f82740 20 client.26933 trim_cache size 0 max 16384 2024-03-21T10:43:33.532 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:43:33.532 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-21T10:43:33.532 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-03-21T10:43:33.532 INFO:teuthology.orchestra.run.smithi031.stdout: "bda1d8f6-d34f-404e-9843-17aa7c34bc1d": { 2024-03-21T10:43:33.532 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-03-21T10:43:33.532 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "bda1d8f6-d34f-404e-9843-17aa7c34bc1d", 2024-03-21T10:43:33.532 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-03-21T10:43:33.532 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:43:33.533 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:43:33.533 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:43:33.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.536+0000 7f7871200640 1 -- 172.21.15.31:0/2400449994 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f78500a3110 msgr2=0x7f78500a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:33.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.536+0000 7f7871200640 1 --2- 172.21.15.31:0/2400449994 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f78500a3110 0x7f78500a55d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f785c036040 tx=0x7f785c02df90 comp rx=0 tx=0).stop 2024-03-21T10:43:33.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.536+0000 7f7871200640 1 -- 172.21.15.31:0/2400449994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f786c109b70 msgr2=0x7f786c1a2ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:33.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.536+0000 7f7871200640 1 --2- 172.21.15.31:0/2400449994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f786c109b70 0x7f786c1a2ec0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f786400da40 tx=0x7f786400df00 comp rx=0 tx=0).stop 2024-03-21T10:43:33.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.536+0000 7f7871200640 1 -- 172.21.15.31:0/2400449994 shutdown_connections 2024-03-21T10:43:33.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.536+0000 7f7871200640 1 --2- 172.21.15.31:0/2400449994 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f78500a3110 0x7f78500a55d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.536+0000 7f7871200640 1 --2- 172.21.15.31:0/2400449994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f786c109b70 0x7f786c1a2ec0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.536+0000 7f7871200640 1 --2- 172.21.15.31:0/2400449994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f786c108760 0x7f786c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.536+0000 7f7871200640 1 --2- 172.21.15.31:0/2400449994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f786c107dc0 0x7f786c19e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:33.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.536+0000 7f7871200640 1 -- 172.21.15.31:0/2400449994 >> 172.21.15.31:0/2400449994 conn(0x7f786c077540 msgr2=0x7f786c077b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:33.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.538+0000 7f7871200640 1 -- 172.21.15.31:0/2400449994 shutdown_connections 2024-03-21T10:43:33.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:43:33.540+0000 7f7871200640 1 -- 172.21.15.31:0/2400449994 wait complete. 2024-03-21T10:43:33.582 INFO:tasks.cephfs.filesystem:scrub status for tag:bda1d8f6-d34f-404e-9843-17aa7c34bc1d - {'path': '/', 'tag': 'bda1d8f6-d34f-404e-9843-17aa7c34bc1d', 'options': 'recursive,force'} 2024-03-21T10:43:33.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:33 smithi031 ceph-mon[31673]: scrub queued for path: / 2024-03-21T10:43:33.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:33 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:43:33.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:33 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:43:33.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:33 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:43:33.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:33 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:43:33.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:33 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:43:34.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:33 smithi070 ceph-mon[36330]: scrub queued for path: / 2024-03-21T10:43:34.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:33 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:43:34.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:33 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:43:34.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:33 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:43:34.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:33 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:43:34.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:33 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:43:34.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:33 smithi040 ceph-mon[36154]: scrub queued for path: / 2024-03-21T10:43:34.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:33 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:43:34.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:33 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:43:34.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:33 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:43:34.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:33 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:43:34.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:33 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:43:35.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:34 smithi031 ceph-mon[31673]: pgmap v153: 129 pgs: 129 active+clean; 29 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 188 MiB/s wr, 1.32k op/s 2024-03-21T10:43:35.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:34 smithi070 ceph-mon[36330]: pgmap v153: 129 pgs: 129 active+clean; 29 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 188 MiB/s wr, 1.32k op/s 2024-03-21T10:43:35.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:34 smithi040 ceph-mon[36154]: pgmap v153: 129 pgs: 129 active+clean; 29 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 188 MiB/s wr, 1.32k op/s 2024-03-21T10:43:37.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:36 smithi031 ceph-mon[31673]: pgmap v154: 129 pgs: 129 active+clean; 29 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 252 MiB/s wr, 1.74k op/s 2024-03-21T10:43:37.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:36 smithi070 ceph-mon[36330]: pgmap v154: 129 pgs: 129 active+clean; 29 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 252 MiB/s wr, 1.74k op/s 2024-03-21T10:43:37.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:36 smithi040 ceph-mon[36154]: pgmap v154: 129 pgs: 129 active+clean; 29 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 252 MiB/s wr, 1.74k op/s 2024-03-21T10:43:39.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:38 smithi031 ceph-mon[31673]: pgmap v155: 129 pgs: 129 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 186 MiB/s wr, 1.24k op/s 2024-03-21T10:43:39.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:38 smithi070 ceph-mon[36330]: pgmap v155: 129 pgs: 129 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 186 MiB/s wr, 1.24k op/s 2024-03-21T10:43:39.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:38 smithi040 ceph-mon[36154]: pgmap v155: 129 pgs: 129 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 186 MiB/s wr, 1.24k op/s 2024-03-21T10:43:41.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:41 smithi031 ceph-mon[31673]: pgmap v156: 129 pgs: 129 active+clean; 30 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 226 MiB/s wr, 1.49k op/s 2024-03-21T10:43:41.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:41 smithi070 ceph-mon[36330]: pgmap v156: 129 pgs: 129 active+clean; 30 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 226 MiB/s wr, 1.49k op/s 2024-03-21T10:43:41.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:41 smithi040 ceph-mon[36154]: pgmap v156: 129 pgs: 129 active+clean; 30 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 226 MiB/s wr, 1.49k op/s 2024-03-21T10:43:42.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:42 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:43:42.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:42 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:43:42.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:42 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:43:42.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:42 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:43:42.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:42 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:43:42.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:42 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:43:43.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:43 smithi031 ceph-mon[31673]: pgmap v157: 129 pgs: 129 active+clean; 31 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 213 MiB/s wr, 1.37k op/s 2024-03-21T10:43:43.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:43 smithi070 ceph-mon[36330]: pgmap v157: 129 pgs: 129 active+clean; 31 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 213 MiB/s wr, 1.37k op/s 2024-03-21T10:43:43.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:43 smithi040 ceph-mon[36154]: pgmap v157: 129 pgs: 129 active+clean; 31 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 213 MiB/s wr, 1.37k op/s 2024-03-21T10:43:43.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.789+0000 7f1628e00640 1 -- 172.21.15.31:0/3032502258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1624110280 msgr2=0x7f162410def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:43.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.789+0000 7f1628e00640 1 --2- 172.21.15.31:0/3032502258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1624110280 0x7f162410def0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f161001a910 tx=0x7f161003cc90 comp rx=0 tx=0).stop 2024-03-21T10:43:43.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.789+0000 7f1628e00640 1 -- 172.21.15.31:0/3032502258 shutdown_connections 2024-03-21T10:43:43.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.789+0000 7f1628e00640 1 --2- 172.21.15.31:0/3032502258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f162411ae80 0x7f162411d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.789+0000 7f1628e00640 1 --2- 172.21.15.31:0/3032502258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1624110280 0x7f162410def0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.789+0000 7f1628e00640 1 --2- 172.21.15.31:0/3032502258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f162410fe10 0x7f162410d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.789+0000 7f1628e00640 1 -- 172.21.15.31:0/3032502258 >> 172.21.15.31:0/3032502258 conn(0x7f162406c750 msgr2=0x7f162406cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:43.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.792+0000 7f1628e00640 1 -- 172.21.15.31:0/3032502258 shutdown_connections 2024-03-21T10:43:43.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.792+0000 7f1628e00640 1 -- 172.21.15.31:0/3032502258 wait complete. 2024-03-21T10:43:43.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.798+0000 7f1628e00640 1 Processor -- start 2024-03-21T10:43:43.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.799+0000 7f1628e00640 1 -- start start 2024-03-21T10:43:43.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.800+0000 7f1628e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f162410fe10 0x7f162411a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:43.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.800+0000 7f1628e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1624110280 0x7f1624116630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:43.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.800+0000 7f1628e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f162411ae80 0x7f1624116b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:43.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.801+0000 7f1621400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1624110280 0x7f1624116630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:43.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.801+0000 7f1628e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1624113080 con 0x7f162411ae80 2024-03-21T10:43:43.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.801+0000 7f1628e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1624112f00 con 0x7f162410fe10 2024-03-21T10:43:43.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.801+0000 7f1628e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1624113200 con 0x7f1624110280 2024-03-21T10:43:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.803+0000 7f1622800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f162411ae80 0x7f1624116b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.803+0000 7f1621e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f162410fe10 0x7f162411a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.803+0000 7f1621e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f162410fe10 0x7f162411a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:40546/0 (socket says 172.21.15.31:40546) 2024-03-21T10:43:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.803+0000 7f1621e00640 1 -- 172.21.15.31:0/3631039425 learned_addr learned my addr 172.21.15.31:0/3631039425 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:43.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.804+0000 7f1621e00640 1 -- 172.21.15.31:0/3631039425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1624110280 msgr2=0x7f1624116630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:43.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.804+0000 7f1621e00640 1 --2- 172.21.15.31:0/3631039425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1624110280 0x7f1624116630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.804+0000 7f1621e00640 1 -- 172.21.15.31:0/3631039425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f162411ae80 msgr2=0x7f1624116b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:43.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.804+0000 7f1621e00640 1 --2- 172.21.15.31:0/3631039425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f162411ae80 0x7f1624116b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.804+0000 7f1621e00640 1 -- 172.21.15.31:0/3631039425 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1624117270 con 0x7f162410fe10 2024-03-21T10:43:43.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.804+0000 7f1621e00640 1 --2- 172.21.15.31:0/3631039425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f162410fe10 0x7f162411a5a0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f161400c890 tx=0x7f161400cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:43.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.805+0000 7f161aa00640 1 -- 172.21.15.31:0/3631039425 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1614004220 con 0x7f162410fe10 2024-03-21T10:43:43.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.805+0000 7f161aa00640 1 -- 172.21.15.31:0/3631039425 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f16140043c0 con 0x7f162410fe10 2024-03-21T10:43:43.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.805+0000 7f161aa00640 1 -- 172.21.15.31:0/3631039425 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16140049e0 con 0x7f162410fe10 2024-03-21T10:43:43.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.805+0000 7f1628e00640 1 -- 172.21.15.31:0/3631039425 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16241ccc30 con 0x7f162410fe10 2024-03-21T10:43:43.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.805+0000 7f1628e00640 1 -- 172.21.15.31:0/3631039425 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16241cd060 con 0x7f162410fe10 2024-03-21T10:43:43.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.817+0000 7f161aa00640 1 -- 172.21.15.31:0/3631039425 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f1614004c40 con 0x7f162410fe10 2024-03-21T10:43:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.819+0000 7f162bb92740 1 Processor -- start 2024-03-21T10:43:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.820+0000 7f162bb92740 1 -- start start 2024-03-21T10:43:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.820+0000 7f162bb92740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646c0fc6fd0 0x5646c0fc73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.820+0000 7f162bb92740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5646c0fc78f0 0x5646c0fd0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.820+0000 7f162bb92740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5646c0fd06c0 0x5646c0fd2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.820+0000 7f162bb92740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646c0f893b0 con 0x5646c0fd06c0 2024-03-21T10:43:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.820+0000 7f162bb92740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646c0f2b830 con 0x5646c0fc78f0 2024-03-21T10:43:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.820+0000 7f162bb92740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646c0ed45d0 con 0x5646c0fc6fd0 2024-03-21T10:43:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.821+0000 7f161aa00640 1 --2- 172.21.15.31:0/3631039425 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f16080a3180 0x7f16080a5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.822+0000 7f161aa00640 1 -- 172.21.15.31:0/3631039425 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f16140d0ac0 con 0x7f162410fe10 2024-03-21T10:43:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.822+0000 7f1622800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646c0fc6fd0 0x5646c0fc73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.822+0000 7f1622800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646c0fc6fd0 0x5646c0fc73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:58070/0 (socket says 172.21.15.31:58070) 2024-03-21T10:43:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.822+0000 7f1622800640 1 -- 172.21.15.31:0/1709744287 learned_addr learned my addr 172.21.15.31:0/1709744287 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.823+0000 7f1621e00640 1 --2- 172.21.15.31:0/1709744287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5646c0fc78f0 0x5646c0fd0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.823+0000 7f1621e00640 1 -- 172.21.15.31:0/1709744287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646c0fc6fd0 msgr2=0x5646c0fc73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.823+0000 7f1621e00640 1 --2- 172.21.15.31:0/1709744287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646c0fc6fd0 0x5646c0fc73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.823+0000 7f1621e00640 1 -- 172.21.15.31:0/1709744287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5646c0fd06c0 msgr2=0x5646c0fd2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.823+0000 7f1621e00640 1 --2- 172.21.15.31:0/1709744287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5646c0fd06c0 0x5646c0fd2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.823+0000 7f1621e00640 1 -- 172.21.15.31:0/1709744287 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5646c0eec2c0 con 0x5646c0fc78f0 2024-03-21T10:43:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.824+0000 7f1622800640 1 --2- 172.21.15.31:0/3631039425 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f16080a3180 0x7f16080a5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.825+0000 7f1621e00640 1 --2- 172.21.15.31:0/1709744287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5646c0fc78f0 0x5646c0fd0180 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f16140cde70 tx=0x7f161409cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b258c2d7795f5cec server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:43.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.826+0000 7f160f400640 1 -- 172.21.15.31:0/1709744287 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161401d040 con 0x5646c0fc78f0 2024-03-21T10:43:43.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.826+0000 7f160f400640 1 -- 172.21.15.31:0/1709744287 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1614005300 con 0x5646c0fc78f0 2024-03-21T10:43:43.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.826+0000 7f160f400640 1 -- 172.21.15.31:0/1709744287 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16140a5090 con 0x5646c0fc78f0 2024-03-21T10:43:43.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.827+0000 7f1622800640 1 --2- 172.21.15.31:0/3631039425 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f16080a3180 0x7f16080a5640 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f161c009fd0 tx=0x7f161c004a60 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:43.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.831+0000 7f162bb92740 1 -- 172.21.15.31:0/1709744287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5646c0fc78f0 msgr2=0x5646c0fd0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:43.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.831+0000 7f162bb92740 1 --2- 172.21.15.31:0/1709744287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5646c0fc78f0 0x5646c0fd0180 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f16140cde70 tx=0x7f161409cc80 comp rx=0 tx=0).stop 2024-03-21T10:43:43.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.831+0000 7f162bb92740 1 -- 172.21.15.31:0/1709744287 shutdown_connections 2024-03-21T10:43:43.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.831+0000 7f162bb92740 1 --2- 172.21.15.31:0/1709744287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5646c0fd06c0 0x5646c0fd2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.831+0000 7f162bb92740 1 --2- 172.21.15.31:0/1709744287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5646c0fc78f0 0x5646c0fd0180 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.831+0000 7f162bb92740 1 --2- 172.21.15.31:0/1709744287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646c0fc6fd0 0x5646c0fc73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.831+0000 7f162bb92740 1 -- 172.21.15.31:0/1709744287 >> 172.21.15.31:0/1709744287 conn(0x5646c0fc18f0 msgr2=0x5646c0fc1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:43.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.832+0000 7f162bb92740 1 -- 172.21.15.31:0/1709744287 shutdown_connections 2024-03-21T10:43:43.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.832+0000 7f162bb92740 1 -- 172.21.15.31:0/1709744287 wait complete. 2024-03-21T10:43:43.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.832+0000 7f162bb92740 1 -- start start 2024-03-21T10:43:43.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.833+0000 7f162bb92740 1 Processor -- start 2024-03-21T10:43:43.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.834+0000 7f162bb92740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646c0fc6fd0 0x5646c0fcd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:43.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.834+0000 7f162bb92740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5646c0fc78f0 0x5646c0fcdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:43.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.834+0000 7f162bb92740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5646c0fd06c0 0x5646c0fce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:43.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.835+0000 7f162bb92740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646c0ee7a20 con 0x5646c0fc78f0 2024-03-21T10:43:43.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.835+0000 7f162bb92740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646c0f278d0 con 0x5646c0fd06c0 2024-03-21T10:43:43.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.835+0000 7f162bb92740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646c0fd32f0 con 0x5646c0fc6fd0 2024-03-21T10:43:43.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.835+0000 7f162bb92740 10 client.0 _ll_register_callbacks cb 0x5646c0f241c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:43:43.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.836+0000 7f1622800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5646c0fd06c0 0x5646c0fce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:43.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.836+0000 7f1622800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5646c0fd06c0 0x5646c0fce260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:40566/0 (socket says 172.21.15.31:40566) 2024-03-21T10:43:43.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.836+0000 7f1622800640 1 -- 172.21.15.31:0/3254881981 learned_addr learned my addr 172.21.15.31:0/3254881981 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:43.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.837+0000 7f1621e00640 1 --2- 172.21.15.31:0/3254881981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646c0fc6fd0 0x5646c0fcd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.837+0000 7f1622800640 1 -- 172.21.15.31:0/3254881981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646c0fc6fd0 msgr2=0x5646c0fcd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.837+0000 7f1622800640 1 --2- 172.21.15.31:0/3254881981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646c0fc6fd0 0x5646c0fcd7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.837+0000 7f1622800640 1 -- 172.21.15.31:0/3254881981 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5646c0fc78f0 msgr2=0x5646c0fcdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.837+0000 7f1622800640 1 --2- 172.21.15.31:0/3254881981 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5646c0fc78f0 0x5646c0fcdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.837+0000 7f1622800640 1 -- 172.21.15.31:0/3254881981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5646c0c40210 con 0x5646c0fd06c0 2024-03-21T10:43:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.840+0000 7f1622800640 1 --2- 172.21.15.31:0/3254881981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5646c0fd06c0 0x5646c0fce260 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f161c00a650 tx=0x7f161c004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8099311190a0679 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.840+0000 7f1603e00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:43:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.840+0000 7f1603e00640 1 -- 172.21.15.31:0/3254881981 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161c019e50 con 0x5646c0fd06c0 2024-03-21T10:43:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.840+0000 7f1603e00640 1 -- 172.21.15.31:0/3254881981 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f161c0233f0 con 0x5646c0fd06c0 2024-03-21T10:43:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.841+0000 7f162bb92740 1 -- 172.21.15.31:0/3254881981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1614005300 con 0x5646c0fd06c0 2024-03-21T10:43:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.842+0000 7f1603e00640 1 -- 172.21.15.31:0/3254881981 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f161c0235f0 con 0x5646c0fd06c0 2024-03-21T10:43:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.842+0000 7f162bb92740 10 client.26942 fetch_fsmap learned FSMap version 26 2024-03-21T10:43:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.842+0000 7f162bb92740 1 -- 172.21.15.31:0/3254881981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5646c0d0c620 con 0x5646c0fd06c0 2024-03-21T10:43:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.842+0000 7f1603e00640 1 -- 172.21.15.31:0/3254881981 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1614005300 con 0x5646c0fd06c0 2024-03-21T10:43:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.843+0000 7f1603e00640 1 -- 172.21.15.31:0/3254881981 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f161c02b9b0 con 0x5646c0fd06c0 2024-03-21T10:43:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.843+0000 7f162bb92740 10 client.26942 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:43:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.843+0000 7f162bb92740 10 client.26942 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:43:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.843+0000 7f162bb92740 20 client.26942 populate_metadata read hostname 'smithi031' 2024-03-21T10:43:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.843+0000 7f162bb92740 1 --2- 172.21.15.31:0/3254881981 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5646c0fe0650 0x5646c0fe2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.844+0000 7f162bb92740 4 client.26942 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:43:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.844+0000 7f162bb92740 1 -- 172.21.15.31:0/3254881981 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5646c0f38d70 con 0x5646c0fe0650 2024-03-21T10:43:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.845+0000 7f1621e00640 1 --2- 172.21.15.31:0/3254881981 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5646c0fe0650 0x5646c0fe2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.846+0000 7f1621e00640 1 --2- 172.21.15.31:0/3254881981 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5646c0fe0650 0x5646c0fe2ad0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2a791cfe4105a92 server_cookie=341475700602fb18 in_seq=0 out_seq=0 2024-03-21T10:43:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.847+0000 7f1603e00640 10 client.26942 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.847+0000 7f1603e00640 1 -- 172.21.15.31:0/3254881981 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5646c0f38d70 con 0x5646c0fe0650 2024-03-21T10:43:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.847+0000 7f1603e00640 10 client.26942 handle_command_reply: tid=0 2024-03-21T10:43:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.847+0000 7f162bb92740 1 client.26942 shutdown 2024-03-21T10:43:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.848+0000 7f162bb92740 1 -- 172.21.15.31:0/3254881981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5646c0fd06c0 msgr2=0x5646c0fce260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.848+0000 7f162bb92740 1 --2- 172.21.15.31:0/3254881981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5646c0fd06c0 0x5646c0fce260 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f161c00a650 tx=0x7f161c004ad0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.848+0000 7f162bb92740 1 -- 172.21.15.31:0/3254881981 shutdown_connections 2024-03-21T10:43:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.848+0000 7f162bb92740 1 --2- 172.21.15.31:0/3254881981 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5646c0fe0650 0x5646c0fe2ad0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.848+0000 7f162bb92740 1 --2- 172.21.15.31:0/3254881981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5646c0fd06c0 0x5646c0fce260 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.848+0000 7f162bb92740 1 --2- 172.21.15.31:0/3254881981 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5646c0fc78f0 0x5646c0fcdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.848+0000 7f162bb92740 1 --2- 172.21.15.31:0/3254881981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646c0fc6fd0 0x5646c0fcd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.848+0000 7f162bb92740 1 -- 172.21.15.31:0/3254881981 >> 172.21.15.31:0/3254881981 conn(0x5646c0fc18f0 msgr2=0x5646c0fc3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.848+0000 7f1603e00640 0 client.26942 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.850+0000 7f162bb92740 1 -- 172.21.15.31:0/3254881981 shutdown_connections 2024-03-21T10:43:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.850+0000 7f162bb92740 1 -- 172.21.15.31:0/3254881981 wait complete. 2024-03-21T10:43:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.850+0000 7f162bb92740 20 client.26942 trim_cache size 0 max 16384 2024-03-21T10:43:43.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.915+0000 7f162bb92740 1 Processor -- start 2024-03-21T10:43:43.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.915+0000 7f162bb92740 1 -- start start 2024-03-21T10:43:43.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.915+0000 7f162bb92740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5646c0fc6fd0 0x5646c0fcff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:43.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.915+0000 7f162bb92740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646c0fc78f0 0x5646c0fce080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:43.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.915+0000 7f162bb92740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5646c0fd06c0 0x5646c0fdba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:43.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.915+0000 7f162bb92740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1604012220 con 0x5646c0fc6fd0 2024-03-21T10:43:43.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.915+0000 7f162bb92740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160400fd20 con 0x5646c0fd06c0 2024-03-21T10:43:43.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.915+0000 7f162bb92740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160400ddc0 con 0x5646c0fc78f0 2024-03-21T10:43:43.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.916+0000 7f1622800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5646c0fd06c0 0x5646c0fdba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:43.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.916+0000 7f1622800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5646c0fd06c0 0x5646c0fdba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:40572/0 (socket says 172.21.15.31:40572) 2024-03-21T10:43:43.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.916+0000 7f1622800640 1 -- 172.21.15.31:0/3492309177 learned_addr learned my addr 172.21.15.31:0/3492309177 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.917+0000 7f1621e00640 1 --2- 172.21.15.31:0/3492309177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5646c0fc6fd0 0x5646c0fcff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.917+0000 7f1621400640 1 --2- 172.21.15.31:0/3492309177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646c0fc78f0 0x5646c0fce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.917+0000 7f1621400640 1 -- 172.21.15.31:0/3492309177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5646c0fd06c0 msgr2=0x5646c0fdba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.917+0000 7f1621400640 1 --2- 172.21.15.31:0/3492309177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5646c0fd06c0 0x5646c0fdba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.917+0000 7f1621400640 1 -- 172.21.15.31:0/3492309177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5646c0fc6fd0 msgr2=0x5646c0fcff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.917+0000 7f1621400640 1 --2- 172.21.15.31:0/3492309177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5646c0fc6fd0 0x5646c0fcff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.917+0000 7f1621400640 1 -- 172.21.15.31:0/3492309177 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5646c0fc7720 con 0x5646c0fc78f0 2024-03-21T10:43:43.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.920+0000 7f1621400640 1 --2- 172.21.15.31:0/3492309177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646c0fc78f0 0x5646c0fce080 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f161003d1d0 tx=0x7f1610004880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c539d4ad9aa2d92 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:43.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.923+0000 7f160ea00640 1 -- 172.21.15.31:0/3492309177 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1610004c50 con 0x5646c0fc78f0 2024-03-21T10:43:43.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.923+0000 7f160ea00640 1 -- 172.21.15.31:0/3492309177 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f161003d840 con 0x5646c0fc78f0 2024-03-21T10:43:43.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.923+0000 7f160ea00640 1 -- 172.21.15.31:0/3492309177 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1610005440 con 0x5646c0fc78f0 2024-03-21T10:43:43.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.924+0000 7f162bb92740 1 -- 172.21.15.31:0/3492309177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646c0fc78f0 msgr2=0x5646c0fce080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:43.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.924+0000 7f162bb92740 1 --2- 172.21.15.31:0/3492309177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646c0fc78f0 0x5646c0fce080 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f161003d1d0 tx=0x7f1610004880 comp rx=0 tx=0).stop 2024-03-21T10:43:43.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.925+0000 7f162bb92740 1 -- 172.21.15.31:0/3492309177 shutdown_connections 2024-03-21T10:43:43.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.925+0000 7f162bb92740 1 --2- 172.21.15.31:0/3492309177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5646c0fd06c0 0x5646c0fdba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.925+0000 7f162bb92740 1 --2- 172.21.15.31:0/3492309177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646c0fc78f0 0x5646c0fce080 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.925+0000 7f162bb92740 1 --2- 172.21.15.31:0/3492309177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5646c0fc6fd0 0x5646c0fcff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.925+0000 7f162bb92740 1 -- 172.21.15.31:0/3492309177 >> 172.21.15.31:0/3492309177 conn(0x5646c0fc18f0 msgr2=0x5646c0fc8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:43.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.925+0000 7f162bb92740 1 -- 172.21.15.31:0/3492309177 shutdown_connections 2024-03-21T10:43:43.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.925+0000 7f162bb92740 1 -- 172.21.15.31:0/3492309177 wait complete. 2024-03-21T10:43:43.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.926+0000 7f162bb92740 1 -- start start 2024-03-21T10:43:43.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.926+0000 7f162bb92740 1 Processor -- start 2024-03-21T10:43:43.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.927+0000 7f162bb92740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5646c0fc6fd0 0x5646c0fca220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:43.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.928+0000 7f162bb92740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5646c0fd06c0 0x5646c0fca760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:43.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.928+0000 7f162bb92740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646c0fcae00 0x5646c103afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:43.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.928+0000 7f1621e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5646c0fc6fd0 0x5646c0fca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:43.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.928+0000 7f162bb92740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160400e3c0 con 0x5646c0fd06c0 2024-03-21T10:43:43.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.928+0000 7f162bb92740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160400f6c0 con 0x5646c0fc6fd0 2024-03-21T10:43:43.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.928+0000 7f162bb92740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160400ea00 con 0x5646c0fcae00 2024-03-21T10:43:43.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.928+0000 7f162bb92740 10 client.0 _ll_register_callbacks cb 0x7f161c015d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:43:43.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.929+0000 7f1621e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5646c0fc6fd0 0x5646c0fca220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:40574/0 (socket says 172.21.15.31:40574) 2024-03-21T10:43:43.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.929+0000 7f1621e00640 1 -- 172.21.15.31:0/1866882781 learned_addr learned my addr 172.21.15.31:0/1866882781 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.929+0000 7f1621400640 1 --2- 172.21.15.31:0/1866882781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5646c0fd06c0 0x5646c0fca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.929+0000 7f1622800640 1 --2- 172.21.15.31:0/1866882781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646c0fcae00 0x5646c103afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.930+0000 7f1622800640 1 -- 172.21.15.31:0/1866882781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5646c0fc6fd0 msgr2=0x5646c0fca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.930+0000 7f1622800640 1 --2- 172.21.15.31:0/1866882781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5646c0fc6fd0 0x5646c0fca220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.930+0000 7f1622800640 1 -- 172.21.15.31:0/1866882781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5646c0fd06c0 msgr2=0x5646c0fca760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.930+0000 7f1622800640 1 --2- 172.21.15.31:0/1866882781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5646c0fd06c0 0x5646c0fca760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.930+0000 7f1622800640 1 -- 172.21.15.31:0/1866882781 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5646c103b7d0 con 0x5646c0fcae00 2024-03-21T10:43:43.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.930+0000 7f1621400640 1 --2- 172.21.15.31:0/1866882781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5646c0fd06c0 0x5646c0fca760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:43:43.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.930+0000 7f1621e00640 1 --2- 172.21.15.31:0/1866882781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5646c0fc6fd0 0x5646c0fca220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:43:43.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.930+0000 7f1622800640 1 --2- 172.21.15.31:0/1866882781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646c0fcae00 0x5646c103afa0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f161c02bd80 tx=0x7f161c023f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46932500327900fd server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:43.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.930+0000 7f1603e00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:43:43.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.931+0000 7f1603e00640 1 -- 172.21.15.31:0/1866882781 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161c0233f0 con 0x5646c0fcae00 2024-03-21T10:43:43.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.931+0000 7f162bb92740 1 -- 172.21.15.31:0/1866882781 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f161003d840 con 0x5646c0fcae00 2024-03-21T10:43:43.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.931+0000 7f1603e00640 1 -- 172.21.15.31:0/1866882781 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f161c03c020 con 0x5646c0fcae00 2024-03-21T10:43:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.932+0000 7f1603e00640 1 -- 172.21.15.31:0/1866882781 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f161c044050 con 0x5646c0fcae00 2024-03-21T10:43:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.932+0000 7f162bb92740 10 client.26872 fetch_fsmap learned FSMap version 26 2024-03-21T10:43:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.932+0000 7f162bb92740 1 -- 172.21.15.31:0/1866882781 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5646c103cae0 con 0x5646c0fcae00 2024-03-21T10:43:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.933+0000 7f1603e00640 1 -- 172.21.15.31:0/1866882781 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f161003d840 con 0x5646c0fcae00 2024-03-21T10:43:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.933+0000 7f1603e00640 1 -- 172.21.15.31:0/1866882781 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f161c02cc90 con 0x5646c0fcae00 2024-03-21T10:43:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.933+0000 7f162bb92740 10 client.26872 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:43:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.933+0000 7f162bb92740 10 client.26872 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:43:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.933+0000 7f162bb92740 20 client.26872 populate_metadata read hostname 'smithi031' 2024-03-21T10:43:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.933+0000 7f162bb92740 1 --2- 172.21.15.31:0/1866882781 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5646c103d200 0x5646c103f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.933+0000 7f162bb92740 4 client.26872 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:43:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.933+0000 7f162bb92740 1 -- 172.21.15.31:0/1866882781 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5646c0f330c0 con 0x5646c103d200 2024-03-21T10:43:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.934+0000 7f1621e00640 1 --2- 172.21.15.31:0/1866882781 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5646c103d200 0x5646c103f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.934+0000 7f1621e00640 1 --2- 172.21.15.31:0/1866882781 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5646c103d200 0x5646c103f620 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b8f16e210149e06 server_cookie=df608c6af9c6cd8c in_seq=0 out_seq=0 2024-03-21T10:43:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.934+0000 7f1603e00640 10 client.26872 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:43.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.981+0000 7f1603e00640 1 -- 172.21.15.31:0/1866882781 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5646c0f330c0 con 0x5646c103d200 2024-03-21T10:43:43.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.985+0000 7f1603e00640 10 client.26872 handle_command_reply: tid=0 2024-03-21T10:43:43.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.985+0000 7f162bb92740 1 client.26872 shutdown 2024-03-21T10:43:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.988+0000 7f162bb92740 1 -- 172.21.15.31:0/1866882781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646c0fcae00 msgr2=0x5646c103afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.988+0000 7f162bb92740 1 --2- 172.21.15.31:0/1866882781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646c0fcae00 0x5646c103afa0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f161c02bd80 tx=0x7f161c023f60 comp rx=0 tx=0).stop 2024-03-21T10:43:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.988+0000 7f162bb92740 1 -- 172.21.15.31:0/1866882781 shutdown_connections 2024-03-21T10:43:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.989+0000 7f162bb92740 1 --2- 172.21.15.31:0/1866882781 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5646c103d200 0x5646c103f620 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.989+0000 7f1603e00640 0 client.26872 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.989+0000 7f162bb92740 1 --2- 172.21.15.31:0/1866882781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646c0fcae00 0x5646c103afa0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.989+0000 7f162bb92740 1 --2- 172.21.15.31:0/1866882781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5646c0fd06c0 0x5646c0fca760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.989+0000 7f162bb92740 1 --2- 172.21.15.31:0/1866882781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5646c0fc6fd0 0x5646c0fca220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.989+0000 7f162bb92740 1 -- 172.21.15.31:0/1866882781 >> 172.21.15.31:0/1866882781 conn(0x5646c0fc18f0 msgr2=0x5646c0fc00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:43.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.990+0000 7f162bb92740 1 -- 172.21.15.31:0/1866882781 shutdown_connections 2024-03-21T10:43:43.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.991+0000 7f162bb92740 1 -- 172.21.15.31:0/1866882781 wait complete. 2024-03-21T10:43:43.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.991+0000 7f162bb92740 20 client.26872 trim_cache size 0 max 16384 2024-03-21T10:43:43.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:43:43.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:43:43.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:43:43.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:43:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.992+0000 7f1628e00640 1 -- 172.21.15.31:0/3631039425 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f16080a3180 msgr2=0x7f16080a5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.992+0000 7f1628e00640 1 --2- 172.21.15.31:0/3631039425 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f16080a3180 0x7f16080a5640 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f161c009fd0 tx=0x7f161c004a60 comp rx=0 tx=0).stop 2024-03-21T10:43:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.993+0000 7f1628e00640 1 -- 172.21.15.31:0/3631039425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f162410fe10 msgr2=0x7f162411a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.993+0000 7f1628e00640 1 --2- 172.21.15.31:0/3631039425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f162410fe10 0x7f162411a5a0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f161400c890 tx=0x7f161400cd50 comp rx=0 tx=0).stop 2024-03-21T10:43:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.993+0000 7f1628e00640 1 -- 172.21.15.31:0/3631039425 shutdown_connections 2024-03-21T10:43:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.993+0000 7f1628e00640 1 --2- 172.21.15.31:0/3631039425 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f16080a3180 0x7f16080a5640 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.993+0000 7f1628e00640 1 --2- 172.21.15.31:0/3631039425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f162411ae80 0x7f1624116b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.993+0000 7f1628e00640 1 --2- 172.21.15.31:0/3631039425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1624110280 0x7f1624116630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.993+0000 7f1628e00640 1 --2- 172.21.15.31:0/3631039425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f162410fe10 0x7f162411a5a0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.993+0000 7f1628e00640 1 -- 172.21.15.31:0/3631039425 >> 172.21.15.31:0/3631039425 conn(0x7f162406c750 msgr2=0x7f162411bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.994+0000 7f1628e00640 1 -- 172.21.15.31:0/3631039425 shutdown_connections 2024-03-21T10:43:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:43.994+0000 7f1628e00640 1 -- 172.21.15.31:0/3631039425 wait complete. 2024-03-21T10:43:45.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:45 smithi031 ceph-mon[31673]: pgmap v158: 129 pgs: 129 active+clean; 31 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 192 MiB/s wr, 1.21k op/s 2024-03-21T10:43:45.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:45 smithi070 ceph-mon[36330]: pgmap v158: 129 pgs: 129 active+clean; 31 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 192 MiB/s wr, 1.21k op/s 2024-03-21T10:43:45.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:45 smithi040 ceph-mon[36154]: pgmap v158: 129 pgs: 129 active+clean; 31 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 192 MiB/s wr, 1.21k op/s 2024-03-21T10:43:47.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:47 smithi070 ceph-mon[36330]: pgmap v159: 129 pgs: 129 active+clean; 32 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 260 MiB/s wr, 1.54k op/s 2024-03-21T10:43:47.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:47 smithi040 ceph-mon[36154]: pgmap v159: 129 pgs: 129 active+clean; 32 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 260 MiB/s wr, 1.54k op/s 2024-03-21T10:43:47.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:47 smithi031 ceph-mon[31673]: pgmap v159: 129 pgs: 129 active+clean; 32 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 260 MiB/s wr, 1.54k op/s 2024-03-21T10:43:49.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:49 smithi040 ceph-mon[36154]: pgmap v160: 129 pgs: 129 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 201 MiB/s wr, 1.10k op/s 2024-03-21T10:43:49.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:49 smithi031 ceph-mon[31673]: pgmap v160: 129 pgs: 129 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 201 MiB/s wr, 1.10k op/s 2024-03-21T10:43:49.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:49 smithi070 ceph-mon[36330]: pgmap v160: 129 pgs: 129 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 201 MiB/s wr, 1.10k op/s 2024-03-21T10:43:51.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:51 smithi040 ceph-mon[36154]: pgmap v161: 129 pgs: 129 active+clean; 32 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 235 MiB/s wr, 1.23k op/s 2024-03-21T10:43:51.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:51 smithi031 ceph-mon[31673]: pgmap v161: 129 pgs: 129 active+clean; 32 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 235 MiB/s wr, 1.23k op/s 2024-03-21T10:43:51.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:51 smithi070 ceph-mon[36330]: pgmap v161: 129 pgs: 129 active+clean; 32 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 235 MiB/s wr, 1.23k op/s 2024-03-21T10:43:53.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:53 smithi040 ceph-mon[36154]: pgmap v162: 129 pgs: 129 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 211 MiB/s wr, 1.04k op/s 2024-03-21T10:43:53.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:53 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:43:53.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:53 smithi031 ceph-mon[31673]: pgmap v162: 129 pgs: 129 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 211 MiB/s wr, 1.04k op/s 2024-03-21T10:43:53.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:53 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:43:53.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:53 smithi070 ceph-mon[36330]: pgmap v162: 129 pgs: 129 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 211 MiB/s wr, 1.04k op/s 2024-03-21T10:43:53.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:53 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:43:55.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:55 smithi040 ceph-mon[36154]: pgmap v163: 129 pgs: 129 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 189 MiB/s wr, 926 op/s 2024-03-21T10:43:55.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:55 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:43:55.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:55 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:43:55.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:55 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:43:55.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:55 smithi031 ceph-mon[31673]: pgmap v163: 129 pgs: 129 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 189 MiB/s wr, 926 op/s 2024-03-21T10:43:55.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:55 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:43:55.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:55 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:43:55.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:55 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:43:55.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:55 smithi070 ceph-mon[36330]: pgmap v163: 129 pgs: 129 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 189 MiB/s wr, 926 op/s 2024-03-21T10:43:55.604 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:55 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:43:55.605 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:55 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:43:55.605 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:55 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:43:57.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.248+0000 7f23cac00640 1 -- 172.21.15.31:0/2555713549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23c4109890 msgr2=0x7f23c4109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:57.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.248+0000 7f23cac00640 1 --2- 172.21.15.31:0/2555713549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23c4109890 0x7f23c4109c70 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f23ac009890 tx=0x7f23ac02bb90 comp rx=0 tx=0).stop 2024-03-21T10:43:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.249+0000 7f23cac00640 1 -- 172.21.15.31:0/2555713549 shutdown_connections 2024-03-21T10:43:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.249+0000 7f23cac00640 1 --2- 172.21.15.31:0/2555713549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23c4104240 0x7f23c4110750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.249+0000 7f23cac00640 1 --2- 172.21.15.31:0/2555713549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23c4103880 0x7f23c4103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.249+0000 7f23cac00640 1 --2- 172.21.15.31:0/2555713549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23c4109890 0x7f23c4109c70 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.249+0000 7f23cac00640 1 -- 172.21.15.31:0/2555713549 >> 172.21.15.31:0/2555713549 conn(0x7f23c40ff390 msgr2=0x7f23c41017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.251+0000 7f23cac00640 1 -- 172.21.15.31:0/2555713549 shutdown_connections 2024-03-21T10:43:57.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.251+0000 7f23cac00640 1 -- 172.21.15.31:0/2555713549 wait complete. 2024-03-21T10:43:57.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.253+0000 7f23cac00640 1 Processor -- start 2024-03-21T10:43:57.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.253+0000 7f23cac00640 1 -- start start 2024-03-21T10:43:57.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.254+0000 7f23cac00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23c4103880 0x7f23c410f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:57.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.254+0000 7f23cac00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23c4104240 0x7f23c410f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:57.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.254+0000 7f23cac00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23c4109890 0x7f23c410fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:57.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.254+0000 7f23cac00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c4076da0 con 0x7f23c4109890 2024-03-21T10:43:57.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.254+0000 7f23cac00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c4076c20 con 0x7f23c4103880 2024-03-21T10:43:57.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.254+0000 7f23cac00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c4076f20 con 0x7f23c4104240 2024-03-21T10:43:57.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.254+0000 7f23c3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23c4103880 0x7f23c410f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:57.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.254+0000 7f23c3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23c4103880 0x7f23c410f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51480/0 (socket says 172.21.15.31:51480) 2024-03-21T10:43:57.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.254+0000 7f23c3e00640 1 -- 172.21.15.31:0/636250040 learned_addr learned my addr 172.21.15.31:0/636250040 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:57.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.255+0000 7f23c3e00640 1 -- 172.21.15.31:0/636250040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23c4104240 msgr2=0x7f23c410f6f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T10:43:57.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.255+0000 7f23c3e00640 1 --2- 172.21.15.31:0/636250040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23c4104240 0x7f23c410f6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.255+0000 7f23c3e00640 1 -- 172.21.15.31:0/636250040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23c4109890 msgr2=0x7f23c410fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:57.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.255+0000 7f23c3e00640 1 --2- 172.21.15.31:0/636250040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23c4109890 0x7f23c410fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.255+0000 7f23c3e00640 1 -- 172.21.15.31:0/636250040 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c406b140 con 0x7f23c4103880 2024-03-21T10:43:57.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.255+0000 7f23c3e00640 1 --2- 172.21.15.31:0/636250040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23c4103880 0x7f23c410f1b0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f23ac009860 tx=0x7f23ac0349e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:57.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.257+0000 7f23c0c00640 1 -- 172.21.15.31:0/636250040 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac041030 con 0x7f23c4103880 2024-03-21T10:43:57.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.257+0000 7f23c0c00640 1 -- 172.21.15.31:0/636250040 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23ac004020 con 0x7f23c4103880 2024-03-21T10:43:57.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.257+0000 7f23c0c00640 1 -- 172.21.15.31:0/636250040 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac039040 con 0x7f23c4103880 2024-03-21T10:43:57.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.258+0000 7f23cac00640 1 -- 172.21.15.31:0/636250040 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c406b3d0 con 0x7f23c4103880 2024-03-21T10:43:57.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.258+0000 7f23cac00640 1 -- 172.21.15.31:0/636250040 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c406b870 con 0x7f23c4103880 2024-03-21T10:43:57.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.274+0000 7f23c0c00640 1 -- 172.21.15.31:0/636250040 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f23ac044070 con 0x7f23c4103880 2024-03-21T10:43:57.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.275+0000 7f23cd974740 1 Processor -- start 2024-03-21T10:43:57.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.282+0000 7f23cd974740 1 -- start start 2024-03-21T10:43:57.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.282+0000 7f23cd974740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597d3f0dfd0 0x5597d3f0e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:57.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.282+0000 7f23cd974740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d3f0e8f0 0x5597d3f17180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:57.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.282+0000 7f23cd974740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 0x5597d3f19ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:57.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.282+0000 7f23cd974740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597d3ed03b0 con 0x5597d3f0dfd0 2024-03-21T10:43:57.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.282+0000 7f23cd974740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597d3e72830 con 0x5597d3f176c0 2024-03-21T10:43:57.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.282+0000 7f23cd974740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597d3e1b5d0 con 0x5597d3f0e8f0 2024-03-21T10:43:57.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.283+0000 7f23c8c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597d3f0dfd0 0x5597d3f0e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:57.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.283+0000 7f23c3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 0x5597d3f19ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:57.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.283+0000 7f23c3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 0x5597d3f19ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51484/0 (socket says 172.21.15.31:51484) 2024-03-21T10:43:57.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.283+0000 7f23c3400640 1 -- 172.21.15.31:0/1880557533 learned_addr learned my addr 172.21.15.31:0/1880557533 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:57.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.284+0000 7f23c3e00640 1 --2- 172.21.15.31:0/1880557533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d3f0e8f0 0x5597d3f17180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:57.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.284+0000 7f23c3e00640 1 -- 172.21.15.31:0/1880557533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 msgr2=0x5597d3f19ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:57.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.284+0000 7f23c3e00640 1 --2- 172.21.15.31:0/1880557533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 0x5597d3f19ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.284+0000 7f23c3e00640 1 -- 172.21.15.31:0/1880557533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597d3f0dfd0 msgr2=0x5597d3f0e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:57.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.284+0000 7f23c3e00640 1 --2- 172.21.15.31:0/1880557533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597d3f0dfd0 0x5597d3f0e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.285+0000 7f23c3e00640 1 -- 172.21.15.31:0/1880557533 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5597d3e332c0 con 0x5597d3f0e8f0 2024-03-21T10:43:57.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.285+0000 7f23c3e00640 1 --2- 172.21.15.31:0/1880557533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d3f0e8f0 0x5597d3f17180 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f23ac0ee460 tx=0x7f23ac0f4f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d8e956101a83102 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:57.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.287+0000 7f23b9200640 1 -- 172.21.15.31:0/1880557533 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac0f9c20 con 0x5597d3f0e8f0 2024-03-21T10:43:57.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.287+0000 7f23b9200640 1 -- 172.21.15.31:0/1880557533 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23ac0fe070 con 0x5597d3f0e8f0 2024-03-21T10:43:57.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.287+0000 7f23b9200640 1 -- 172.21.15.31:0/1880557533 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac102740 con 0x5597d3f0e8f0 2024-03-21T10:43:57.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.289+0000 7f23c0c00640 1 --2- 172.21.15.31:0/636250040 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f23a80a3290 0x7f23a80a5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:57.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.289+0000 7f23c0c00640 1 -- 172.21.15.31:0/636250040 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f23ac0f0fa0 con 0x7f23c4103880 2024-03-21T10:43:57.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.291+0000 7f23c8c00640 1 --2- 172.21.15.31:0/636250040 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f23a80a3290 0x7f23a80a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:57.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.293+0000 7f23c8c00640 1 --2- 172.21.15.31:0/636250040 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f23a80a3290 0x7f23a80a5750 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f23bc0092f0 tx=0x7f23bc009280 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:57.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.296+0000 7f23cd974740 1 -- 172.21.15.31:0/1880557533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d3f0e8f0 msgr2=0x5597d3f17180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:57.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.296+0000 7f23cd974740 1 --2- 172.21.15.31:0/1880557533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d3f0e8f0 0x5597d3f17180 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f23ac0ee460 tx=0x7f23ac0f4f30 comp rx=0 tx=0).stop 2024-03-21T10:43:57.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.298+0000 7f23cd974740 1 -- 172.21.15.31:0/1880557533 shutdown_connections 2024-03-21T10:43:57.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.298+0000 7f23cd974740 1 --2- 172.21.15.31:0/1880557533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 0x5597d3f19ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.298+0000 7f23cd974740 1 --2- 172.21.15.31:0/1880557533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d3f0e8f0 0x5597d3f17180 secure :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f23ac0ee460 tx=0x7f23ac0f4f30 comp rx=0 tx=0).stop 2024-03-21T10:43:57.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.298+0000 7f23cd974740 1 --2- 172.21.15.31:0/1880557533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597d3f0dfd0 0x5597d3f0e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.298+0000 7f23cd974740 1 -- 172.21.15.31:0/1880557533 >> 172.21.15.31:0/1880557533 conn(0x5597d3f088f0 msgr2=0x5597d3f08cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:57.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.298+0000 7f23cd974740 1 -- 172.21.15.31:0/1880557533 shutdown_connections 2024-03-21T10:43:57.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.299+0000 7f23cd974740 1 -- 172.21.15.31:0/1880557533 wait complete. 2024-03-21T10:43:57.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.299+0000 7f23cd974740 1 -- start start 2024-03-21T10:43:57.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.300+0000 7f23cd974740 1 Processor -- start 2024-03-21T10:43:57.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.300+0000 7f23cd974740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d3f0dfd0 0x5597d3f14950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:57.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.300+0000 7f23cd974740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 0x5597d3f14e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:57.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.300+0000 7f23cd974740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597d3f15510 0x5597d3f23150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:57.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.300+0000 7f23cd974740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597d3e2ea20 con 0x5597d3f15510 2024-03-21T10:43:57.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.301+0000 7f23cd974740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597d3e6e8d0 con 0x5597d3f176c0 2024-03-21T10:43:57.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.301+0000 7f23cd974740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597d3e1b5d0 con 0x5597d3f0dfd0 2024-03-21T10:43:57.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.301+0000 7f23cd974740 10 client.0 _ll_register_callbacks cb 0x5597d3e6b1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:43:57.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.301+0000 7f23c3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 0x5597d3f14e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:57.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.301+0000 7f23c3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 0x5597d3f14e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51494/0 (socket says 172.21.15.31:51494) 2024-03-21T10:43:57.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.301+0000 7f23c3400640 1 -- 172.21.15.31:0/2303624197 learned_addr learned my addr 172.21.15.31:0/2303624197 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:57.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.301+0000 7f23c8c00640 1 --2- 172.21.15.31:0/2303624197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597d3f15510 0x5597d3f23150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:57.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.302+0000 7f23c3400640 1 -- 172.21.15.31:0/2303624197 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d3f0dfd0 msgr2=0x5597d3f14950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:43:57.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.302+0000 7f23c3400640 1 --2- 172.21.15.31:0/2303624197 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d3f0dfd0 0x5597d3f14950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.302+0000 7f23c3400640 1 -- 172.21.15.31:0/2303624197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597d3f15510 msgr2=0x5597d3f23150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:57.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.302+0000 7f23c3400640 1 --2- 172.21.15.31:0/2303624197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597d3f15510 0x5597d3f23150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.302+0000 7f23c3400640 1 -- 172.21.15.31:0/2303624197 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5597d3b87210 con 0x5597d3f176c0 2024-03-21T10:43:57.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.303+0000 7f23c3400640 1 --2- 172.21.15.31:0/2303624197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 0x5597d3f14e90 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f23b4011660 tx=0x7f23b4011b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58770f0cff1f6219 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:57.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.303+0000 7f23b1600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:43:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.303+0000 7f23b1600640 1 -- 172.21.15.31:0/2303624197 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b401f060 con 0x5597d3f176c0 2024-03-21T10:43:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.303+0000 7f23b1600640 1 -- 172.21.15.31:0/2303624197 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f23b400fac0 con 0x5597d3f176c0 2024-03-21T10:43:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.304+0000 7f23cd974740 1 -- 172.21.15.31:0/2303624197 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f23ac0fe070 con 0x5597d3f176c0 2024-03-21T10:43:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.304+0000 7f23b1600640 1 -- 172.21.15.31:0/2303624197 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23b4011e60 con 0x5597d3f176c0 2024-03-21T10:43:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.304+0000 7f23cd974740 10 client.26960 fetch_fsmap learned FSMap version 26 2024-03-21T10:43:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.304+0000 7f23cd974740 1 -- 172.21.15.31:0/2303624197 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5597d3c53620 con 0x5597d3f176c0 2024-03-21T10:43:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.305+0000 7f23b1600640 1 -- 172.21.15.31:0/2303624197 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23ac0fe070 con 0x5597d3f176c0 2024-03-21T10:43:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.305+0000 7f23b1600640 1 -- 172.21.15.31:0/2303624197 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f23b40233a0 con 0x5597d3f176c0 2024-03-21T10:43:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.305+0000 7f23cd974740 10 client.26960 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:43:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.305+0000 7f23cd974740 10 client.26960 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:43:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.305+0000 7f23cd974740 20 client.26960 populate_metadata read hostname 'smithi031' 2024-03-21T10:43:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.306+0000 7f23cd974740 1 --2- 172.21.15.31:0/2303624197 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5597d3f27980 0x5597d3f29d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:57.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.306+0000 7f23cd974740 4 client.26960 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:43:57.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.306+0000 7f23cd974740 1 -- 172.21.15.31:0/2303624197 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5597d3e7fd70 con 0x5597d3f27980 2024-03-21T10:43:57.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.307+0000 7f23c3e00640 1 --2- 172.21.15.31:0/2303624197 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5597d3f27980 0x5597d3f29d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:57.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.309+0000 7f23c3e00640 1 --2- 172.21.15.31:0/2303624197 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5597d3f27980 0x5597d3f29d70 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8941fb6d70be0e7d server_cookie=c40b3dec6bf303cc in_seq=0 out_seq=0 2024-03-21T10:43:57.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.309+0000 7f23b1600640 10 client.26960 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:57.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.310+0000 7f23b1600640 1 -- 172.21.15.31:0/2303624197 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5597d3e7fd70 con 0x5597d3f27980 2024-03-21T10:43:57.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.310+0000 7f23b1600640 10 client.26960 handle_command_reply: tid=0 2024-03-21T10:43:57.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.310+0000 7f23cd974740 1 client.26960 shutdown 2024-03-21T10:43:57.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.312+0000 7f23cd974740 1 -- 172.21.15.31:0/2303624197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 msgr2=0x5597d3f14e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:57.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.312+0000 7f23cd974740 1 --2- 172.21.15.31:0/2303624197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 0x5597d3f14e90 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f23b4011660 tx=0x7f23b4011b20 comp rx=0 tx=0).stop 2024-03-21T10:43:57.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.312+0000 7f23cd974740 1 -- 172.21.15.31:0/2303624197 shutdown_connections 2024-03-21T10:43:57.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.312+0000 7f23cd974740 1 --2- 172.21.15.31:0/2303624197 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5597d3f27980 0x5597d3f29d70 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.312+0000 7f23b1600640 0 client.26960 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:57.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.312+0000 7f23cd974740 1 --2- 172.21.15.31:0/2303624197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597d3f15510 0x5597d3f23150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.312+0000 7f23cd974740 1 --2- 172.21.15.31:0/2303624197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 0x5597d3f14e90 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.312+0000 7f23cd974740 1 --2- 172.21.15.31:0/2303624197 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d3f0dfd0 0x5597d3f14950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.312+0000 7f23cd974740 1 -- 172.21.15.31:0/2303624197 >> 172.21.15.31:0/2303624197 conn(0x5597d3f088f0 msgr2=0x5597d3f0a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:57.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.314+0000 7f23cd974740 1 -- 172.21.15.31:0/2303624197 shutdown_connections 2024-03-21T10:43:57.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.314+0000 7f23cd974740 1 -- 172.21.15.31:0/2303624197 wait complete. 2024-03-21T10:43:57.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.315+0000 7f23cd974740 20 client.26960 trim_cache size 0 max 16384 2024-03-21T10:43:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.337+0000 7f23cd974740 1 Processor -- start 2024-03-21T10:43:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.337+0000 7f23cd974740 1 -- start start 2024-03-21T10:43:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.338+0000 7f23cd974740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597d3f0dfd0 0x5597d3f11bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.338+0000 7f23cd974740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d3f15510 0x5597d3f12110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.338+0000 7f23cd974740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 0x5597d3f2e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.338+0000 7f23cd974740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c41c8510 con 0x5597d3f0dfd0 2024-03-21T10:43:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.338+0000 7f23cd974740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c41c6010 con 0x5597d3f176c0 2024-03-21T10:43:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.338+0000 7f23cd974740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c41106e0 con 0x5597d3f15510 2024-03-21T10:43:57.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.338+0000 7f23c8c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 0x5597d3f2e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:57.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.338+0000 7f23c8c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 0x5597d3f2e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51496/0 (socket says 172.21.15.31:51496) 2024-03-21T10:43:57.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.338+0000 7f23c8c00640 1 -- 172.21.15.31:0/3019323798 learned_addr learned my addr 172.21.15.31:0/3019323798 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:57.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.339+0000 7f23c3e00640 1 --2- 172.21.15.31:0/3019323798 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597d3f0dfd0 0x5597d3f11bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:57.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.339+0000 7f23c8c00640 1 -- 172.21.15.31:0/3019323798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d3f15510 msgr2=0x5597d3f12110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:43:57.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.339+0000 7f23c8c00640 1 --2- 172.21.15.31:0/3019323798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d3f15510 0x5597d3f12110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.339+0000 7f23c8c00640 1 -- 172.21.15.31:0/3019323798 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597d3f0dfd0 msgr2=0x5597d3f11bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:57.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.339+0000 7f23c8c00640 1 --2- 172.21.15.31:0/3019323798 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597d3f0dfd0 0x5597d3f11bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.339+0000 7f23c8c00640 1 -- 172.21.15.31:0/3019323798 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5597d3f19e20 con 0x5597d3f176c0 2024-03-21T10:43:57.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.341+0000 7f23c8c00640 1 --2- 172.21.15.31:0/3019323798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 0x5597d3f2e600 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f23bc009990 tx=0x7f23bc012000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce54e1fd14312e44 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.343+0000 7f23b3e00640 1 -- 172.21.15.31:0/3019323798 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23bc004d20 con 0x5597d3f176c0 2024-03-21T10:43:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.343+0000 7f23b3e00640 1 -- 172.21.15.31:0/3019323798 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23bc011e50 con 0x5597d3f176c0 2024-03-21T10:43:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.343+0000 7f23b3e00640 1 -- 172.21.15.31:0/3019323798 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23bc004d20 con 0x5597d3f176c0 2024-03-21T10:43:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.346+0000 7f23cd974740 1 -- 172.21.15.31:0/3019323798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 msgr2=0x5597d3f2e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.346+0000 7f23cd974740 1 --2- 172.21.15.31:0/3019323798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 0x5597d3f2e600 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f23bc009990 tx=0x7f23bc012000 comp rx=0 tx=0).stop 2024-03-21T10:43:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.352+0000 7f23cd974740 1 -- 172.21.15.31:0/3019323798 shutdown_connections 2024-03-21T10:43:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.352+0000 7f23cd974740 1 --2- 172.21.15.31:0/3019323798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 0x5597d3f2e600 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.352+0000 7f23cd974740 1 --2- 172.21.15.31:0/3019323798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d3f15510 0x5597d3f12110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.352+0000 7f23cd974740 1 --2- 172.21.15.31:0/3019323798 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597d3f0dfd0 0x5597d3f11bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.352+0000 7f23cd974740 1 -- 172.21.15.31:0/3019323798 >> 172.21.15.31:0/3019323798 conn(0x5597d3f088f0 msgr2=0x5597d3f07950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.356+0000 7f23cd974740 1 -- 172.21.15.31:0/3019323798 shutdown_connections 2024-03-21T10:43:57.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.362+0000 7f23cd974740 1 -- 172.21.15.31:0/3019323798 wait complete. 2024-03-21T10:43:57.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.362+0000 7f23cd974740 1 -- start start 2024-03-21T10:43:57.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.364+0000 7f23cd974740 1 Processor -- start 2024-03-21T10:43:57.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.366+0000 7f23cd974740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597d3f0dfd0 0x5597d3f1e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.369+0000 7f23cd974740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d3f15510 0x5597d3f1ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.369+0000 7f23cd974740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 0x5597d3f1cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.369+0000 7f23cd974740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c40736c0 con 0x5597d3f0dfd0 2024-03-21T10:43:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.369+0000 7f23cd974740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c41c59b0 con 0x5597d3f176c0 2024-03-21T10:43:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.369+0000 7f23cd974740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c41c4cf0 con 0x5597d3f15510 2024-03-21T10:43:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.369+0000 7f23cd974740 10 client.0 _ll_register_callbacks cb 0x7f23b4009600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:43:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.371+0000 7f23c8c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 0x5597d3f1cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.371+0000 7f23c8c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 0x5597d3f1cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51504/0 (socket says 172.21.15.31:51504) 2024-03-21T10:43:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.371+0000 7f23c8c00640 1 -- 172.21.15.31:0/1953531870 learned_addr learned my addr 172.21.15.31:0/1953531870 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:43:57.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.372+0000 7f23c8c00640 1 -- 172.21.15.31:0/1953531870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d3f15510 msgr2=0x5597d3f1ed80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:43:57.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.372+0000 7f23c8c00640 1 --2- 172.21.15.31:0/1953531870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d3f15510 0x5597d3f1ed80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.372+0000 7f23c8c00640 1 -- 172.21.15.31:0/1953531870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597d3f0dfd0 msgr2=0x5597d3f1e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:43:57.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.372+0000 7f23c8c00640 1 --2- 172.21.15.31:0/1953531870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597d3f0dfd0 0x5597d3f1e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.372+0000 7f23c8c00640 1 -- 172.21.15.31:0/1953531870 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5597d3f1d420 con 0x5597d3f176c0 2024-03-21T10:43:57.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.373+0000 7f23c8c00640 1 --2- 172.21.15.31:0/1953531870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 0x5597d3f1cce0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f23bc004a80 tx=0x7f23bc005170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebffe39f2975ed2c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:43:57.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.374+0000 7f23b1600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:43:57.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.374+0000 7f23b1600640 1 -- 172.21.15.31:0/1953531870 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23bc036060 con 0x5597d3f176c0 2024-03-21T10:43:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.374+0000 7f23b1600640 1 -- 172.21.15.31:0/1953531870 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f23bc0054f0 con 0x5597d3f176c0 2024-03-21T10:43:57.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.378+0000 7f23cd974740 1 -- 172.21.15.31:0/1953531870 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f23bc011e50 con 0x5597d3f176c0 2024-03-21T10:43:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.380+0000 7f23b1600640 1 -- 172.21.15.31:0/1953531870 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23bc004d20 con 0x5597d3f176c0 2024-03-21T10:43:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.381+0000 7f23cd974740 10 client.26966 fetch_fsmap learned FSMap version 26 2024-03-21T10:43:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.381+0000 7f23cd974740 1 -- 172.21.15.31:0/1953531870 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5597d3f83d40 con 0x5597d3f176c0 2024-03-21T10:43:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.381+0000 7f23b1600640 1 -- 172.21.15.31:0/1953531870 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23bc011e50 con 0x5597d3f176c0 2024-03-21T10:43:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.381+0000 7f23b1600640 1 -- 172.21.15.31:0/1953531870 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f23bc026400 con 0x5597d3f176c0 2024-03-21T10:43:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.381+0000 7f23cd974740 10 client.26966 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:43:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.381+0000 7f23cd974740 10 client.26966 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:43:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.382+0000 7f23cd974740 20 client.26966 populate_metadata read hostname 'smithi031' 2024-03-21T10:43:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.382+0000 7f23cd974740 1 --2- 172.21.15.31:0/1953531870 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5597d3f842f0 0x5597d3f86770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:43:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.382+0000 7f23cd974740 4 client.26966 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:43:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.382+0000 7f23cd974740 1 -- 172.21.15.31:0/1953531870 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5597d3e7a0c0 con 0x5597d3f842f0 2024-03-21T10:43:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.385+0000 7f23c3e00640 1 --2- 172.21.15.31:0/1953531870 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5597d3f842f0 0x5597d3f86770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:43:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.385+0000 7f23c3e00640 1 --2- 172.21.15.31:0/1953531870 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5597d3f842f0 0x5597d3f86770 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ecee2e005e73a1f server_cookie=368e2c8b85e4fe15 in_seq=0 out_seq=0 2024-03-21T10:43:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.386+0000 7f23b1600640 10 client.26966 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.401+0000 7f23b1600640 1 -- 172.21.15.31:0/1953531870 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5597d3e7a0c0 con 0x5597d3f842f0 2024-03-21T10:43:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.401+0000 7f23b1600640 10 client.26966 handle_command_reply: tid=0 2024-03-21T10:43:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.401+0000 7f23cd974740 1 client.26966 shutdown 2024-03-21T10:43:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.401+0000 7f23cd974740 1 -- 172.21.15.31:0/1953531870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 msgr2=0x5597d3f1cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.401+0000 7f23cd974740 1 --2- 172.21.15.31:0/1953531870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 0x5597d3f1cce0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f23bc004a80 tx=0x7f23bc005170 comp rx=0 tx=0).stop 2024-03-21T10:43:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.401+0000 7f23cd974740 1 -- 172.21.15.31:0/1953531870 shutdown_connections 2024-03-21T10:43:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.401+0000 7f23cd974740 1 --2- 172.21.15.31:0/1953531870 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5597d3f842f0 0x5597d3f86770 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.401+0000 7f23cd974740 1 --2- 172.21.15.31:0/1953531870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597d3f176c0 0x5597d3f1cce0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.401+0000 7f23b1600640 0 client.26966 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:43:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.401+0000 7f23cd974740 1 --2- 172.21.15.31:0/1953531870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d3f15510 0x5597d3f1ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.401+0000 7f23cd974740 1 --2- 172.21.15.31:0/1953531870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597d3f0dfd0 0x5597d3f1e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.401+0000 7f23cd974740 1 -- 172.21.15.31:0/1953531870 >> 172.21.15.31:0/1953531870 conn(0x5597d3f088f0 msgr2=0x5597d3f09b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.402+0000 7f23cd974740 1 -- 172.21.15.31:0/1953531870 shutdown_connections 2024-03-21T10:43:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.402+0000 7f23cd974740 1 -- 172.21.15.31:0/1953531870 wait complete. 2024-03-21T10:43:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.402+0000 7f23cd974740 20 client.26966 trim_cache size 0 max 16384 2024-03-21T10:43:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:43:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:43:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:43:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:43:57.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.402+0000 7f23cac00640 1 -- 172.21.15.31:0/636250040 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f23a80a3290 msgr2=0x7f23a80a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:57.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.402+0000 7f23cac00640 1 --2- 172.21.15.31:0/636250040 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f23a80a3290 0x7f23a80a5750 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f23bc0092f0 tx=0x7f23bc009280 comp rx=0 tx=0).stop 2024-03-21T10:43:57.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.402+0000 7f23cac00640 1 -- 172.21.15.31:0/636250040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23c4103880 msgr2=0x7f23c410f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:43:57.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.402+0000 7f23cac00640 1 --2- 172.21.15.31:0/636250040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23c4103880 0x7f23c410f1b0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f23ac009860 tx=0x7f23ac0349e0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.403+0000 7f23cac00640 1 -- 172.21.15.31:0/636250040 shutdown_connections 2024-03-21T10:43:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.403+0000 7f23cac00640 1 --2- 172.21.15.31:0/636250040 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f23a80a3290 0x7f23a80a5750 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.403+0000 7f23cac00640 1 --2- 172.21.15.31:0/636250040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23c4109890 0x7f23c410fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.403+0000 7f23cac00640 1 --2- 172.21.15.31:0/636250040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23c4104240 0x7f23c410f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.403+0000 7f23cac00640 1 --2- 172.21.15.31:0/636250040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23c4103880 0x7f23c410f1b0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:43:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.403+0000 7f23cac00640 1 -- 172.21.15.31:0/636250040 >> 172.21.15.31:0/636250040 conn(0x7f23c40ff390 msgr2=0x7f23c40ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:43:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.403+0000 7f23cac00640 1 -- 172.21.15.31:0/636250040 shutdown_connections 2024-03-21T10:43:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:43:57.403+0000 7f23cac00640 1 -- 172.21.15.31:0/636250040 wait complete. 2024-03-21T10:43:57.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:57 smithi031 ceph-mon[31673]: pgmap v164: 129 pgs: 129 active+clean; 33 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 256 MiB/s wr, 1.25k op/s 2024-03-21T10:43:57.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:57 smithi070 ceph-mon[36330]: pgmap v164: 129 pgs: 129 active+clean; 33 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 256 MiB/s wr, 1.25k op/s 2024-03-21T10:43:57.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:57 smithi040 ceph-mon[36154]: pgmap v164: 129 pgs: 129 active+clean; 33 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 256 MiB/s wr, 1.25k op/s 2024-03-21T10:43:59.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:43:59 smithi031 ceph-mon[31673]: pgmap v165: 129 pgs: 129 active+clean; 34 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 196 MiB/s wr, 948 op/s 2024-03-21T10:43:59.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:43:59 smithi070 ceph-mon[36330]: pgmap v165: 129 pgs: 129 active+clean; 34 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 196 MiB/s wr, 948 op/s 2024-03-21T10:43:59.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:43:59 smithi040 ceph-mon[36154]: pgmap v165: 129 pgs: 129 active+clean; 34 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 196 MiB/s wr, 948 op/s 2024-03-21T10:44:00.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:00 smithi031 ceph-mon[31673]: pgmap v166: 129 pgs: 129 active+clean; 34 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 238 MiB/s wr, 1.19k op/s 2024-03-21T10:44:00.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:00 smithi070 ceph-mon[36330]: pgmap v166: 129 pgs: 129 active+clean; 34 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 238 MiB/s wr, 1.19k op/s 2024-03-21T10:44:00.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:00 smithi040 ceph-mon[36154]: pgmap v166: 129 pgs: 129 active+clean; 34 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 238 MiB/s wr, 1.19k op/s 2024-03-21T10:44:03.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:02 smithi031 ceph-mon[31673]: pgmap v167: 129 pgs: 129 active+clean; 34 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 219 MiB/s wr, 1.14k op/s 2024-03-21T10:44:03.072 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:02 smithi070 ceph-mon[36330]: pgmap v167: 129 pgs: 129 active+clean; 34 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 219 MiB/s wr, 1.14k op/s 2024-03-21T10:44:03.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:02 smithi040 ceph-mon[36154]: pgmap v167: 129 pgs: 129 active+clean; 34 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 219 MiB/s wr, 1.14k op/s 2024-03-21T10:44:03.583 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:44:03.583 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:44:03.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.961+0000 7f0bb8e00640 1 -- 172.21.15.31:0/3595116613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0bb410ba50 msgr2=0x7f0bb410ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:03.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.961+0000 7f0bb8e00640 1 --2- 172.21.15.31:0/3595116613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0bb410ba50 0x7f0bb410ded0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0bac009900 tx=0x7f0bac02bce0 comp rx=0 tx=0).stop 2024-03-21T10:44:03.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.962+0000 7f0bb8e00640 1 -- 172.21.15.31:0/3595116613 shutdown_connections 2024-03-21T10:44:03.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.962+0000 7f0bb8e00640 1 --2- 172.21.15.31:0/3595116613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0bb410ba50 0x7f0bb410ded0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:03.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.962+0000 7f0bb8e00640 1 --2- 172.21.15.31:0/3595116613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0bb4102f50 0x7f0bb410b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:03.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.962+0000 7f0bb8e00640 1 --2- 172.21.15.31:0/3595116613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0bb4102630 0x7f0bb4102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:03.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.962+0000 7f0bb8e00640 1 -- 172.21.15.31:0/3595116613 >> 172.21.15.31:0/3595116613 conn(0x7f0bb4077130 msgr2=0x7f0bb40fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:03.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.962+0000 7f0bb8e00640 1 -- 172.21.15.31:0/3595116613 shutdown_connections 2024-03-21T10:44:03.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.962+0000 7f0bb8e00640 1 -- 172.21.15.31:0/3595116613 wait complete. 2024-03-21T10:44:03.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.964+0000 7f0bb8e00640 1 Processor -- start 2024-03-21T10:44:03.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.964+0000 7f0bb8e00640 1 -- start start 2024-03-21T10:44:03.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.964+0000 7f0bb8e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0bb4102630 0x7f0bb419e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:03.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.965+0000 7f0bb8e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0bb4102f50 0x7f0bb419e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:03.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.965+0000 7f0bb8e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0bb410ba50 0x7f0bb41a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:03.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.965+0000 7f0bb8e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb410f690 con 0x7f0bb4102630 2024-03-21T10:44:03.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.965+0000 7f0bb8e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb410f510 con 0x7f0bb410ba50 2024-03-21T10:44:03.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.965+0000 7f0bb8e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb410f810 con 0x7f0bb4102f50 2024-03-21T10:44:03.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.966+0000 7f0baaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0bb4102f50 0x7f0bb419e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:03.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.966+0000 7f0bb3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0bb410ba50 0x7f0bb41a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:03.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.966+0000 7f0bb3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0bb410ba50 0x7f0bb41a2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51524/0 (socket says 172.21.15.31:51524) 2024-03-21T10:44:03.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.966+0000 7f0bb3e00640 1 -- 172.21.15.31:0/1347154548 learned_addr learned my addr 172.21.15.31:0/1347154548 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:03.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.966+0000 7f0baaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0bb4102f50 0x7f0bb419e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:48044/0 (socket says 172.21.15.31:48044) 2024-03-21T10:44:03.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.967+0000 7f0bb3e00640 1 -- 172.21.15.31:0/1347154548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0bb4102f50 msgr2=0x7f0bb419e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:03.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.967+0000 7f0bb3e00640 1 --2- 172.21.15.31:0/1347154548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0bb4102f50 0x7f0bb419e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:03.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.967+0000 7f0bb3e00640 1 -- 172.21.15.31:0/1347154548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0bb4102630 msgr2=0x7f0bb419e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:03.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.967+0000 7f0bb3e00640 1 --2- 172.21.15.31:0/1347154548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0bb4102630 0x7f0bb419e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:03.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.967+0000 7f0bb3e00640 1 -- 172.21.15.31:0/1347154548 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb41a3370 con 0x7f0bb410ba50 2024-03-21T10:44:03.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.967+0000 7f0bb3e00640 1 --2- 172.21.15.31:0/1347154548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0bb410ba50 0x7f0bb41a2c40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0bac02c270 tx=0x7f0bac0045c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:03.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.969+0000 7f0bb0c00640 1 -- 172.21.15.31:0/1347154548 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac042020 con 0x7f0bb410ba50 2024-03-21T10:44:03.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.969+0000 7f0bb0c00640 1 -- 172.21.15.31:0/1347154548 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0bac005000 con 0x7f0bb410ba50 2024-03-21T10:44:03.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.969+0000 7f0bb0c00640 1 -- 172.21.15.31:0/1347154548 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac003a00 con 0x7f0bb410ba50 2024-03-21T10:44:03.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.969+0000 7f0bb8e00640 1 -- 172.21.15.31:0/1347154548 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb41a3600 con 0x7f0bb410ba50 2024-03-21T10:44:03.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.969+0000 7f0bb8e00640 1 -- 172.21.15.31:0/1347154548 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb41aaef0 con 0x7f0bb410ba50 2024-03-21T10:44:03.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.974+0000 7f0bbd1bf740 1 Processor -- start 2024-03-21T10:44:03.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.976+0000 7f0bb0c00640 1 -- 172.21.15.31:0/1347154548 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f0bac045020 con 0x7f0bb410ba50 2024-03-21T10:44:03.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.976+0000 7f0bbd1bf740 1 -- start start 2024-03-21T10:44:03.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.977+0000 7f0bbd1bf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651040c8dd0 0x5651040c7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:03.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.977+0000 7f0bbd1bf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651040c80f0 0x5651040c8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:03.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.977+0000 7f0bbd1bf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651040d1460 0x5651040d3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:03.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.977+0000 7f0bbd1bf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56510402c250 con 0x5651040d1460 2024-03-21T10:44:03.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.977+0000 7f0bbd1bf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x565103fd4a30 con 0x5651040c8dd0 2024-03-21T10:44:03.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.977+0000 7f0bbd1bf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651040282f0 con 0x5651040c80f0 2024-03-21T10:44:03.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.978+0000 7f0bb3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651040c80f0 0x5651040c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:03.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.978+0000 7f0bb3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651040c80f0 0x5651040c8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:48046/0 (socket says 172.21.15.31:48046) 2024-03-21T10:44:03.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.978+0000 7f0bb3400640 1 -- 172.21.15.31:0/2517238858 learned_addr learned my addr 172.21.15.31:0/2517238858 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:03.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.978+0000 7f0baaa00640 1 --2- 172.21.15.31:0/2517238858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651040c8dd0 0x5651040c7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:03.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.979+0000 7f0bb3400640 1 -- 172.21.15.31:0/2517238858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651040c8dd0 msgr2=0x5651040c7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:03.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.979+0000 7f0bb3400640 1 --2- 172.21.15.31:0/2517238858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651040c8dd0 0x5651040c7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:03.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.979+0000 7f0bb3400640 1 -- 172.21.15.31:0/2517238858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651040d1460 msgr2=0x5651040d3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:03.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.979+0000 7f0bb3400640 1 --2- 172.21.15.31:0/2517238858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651040d1460 0x5651040d3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:03.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.979+0000 7f0bb3400640 1 -- 172.21.15.31:0/2517238858 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565103fbf690 con 0x5651040c80f0 2024-03-21T10:44:03.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.980+0000 7f0bb0c00640 1 --2- 172.21.15.31:0/1347154548 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f0b980a3220 0x7f0b980a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:03.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.981+0000 7f0bb0c00640 1 -- 172.21.15.31:0/1347154548 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f0bac0f16c0 con 0x7f0bb410ba50 2024-03-21T10:44:03.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.981+0000 7f0baaa00640 1 --2- 172.21.15.31:0/1347154548 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f0b980a3220 0x7f0b980a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:03.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.982+0000 7f0bb3400640 1 --2- 172.21.15.31:0/2517238858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651040c80f0 0x5651040c8570 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c009820 tx=0x7f0b9c009ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed5c9c9400e8c464 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:03.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.983+0000 7f0ba8c00640 1 -- 172.21.15.31:0/2517238858 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c015020 con 0x5651040c80f0 2024-03-21T10:44:03.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.983+0000 7f0baaa00640 1 --2- 172.21.15.31:0/1347154548 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f0b980a3220 0x7f0b980a56e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0bb40ff710 tx=0x7f0ba4009c10 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:03.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.986+0000 7f0ba8c00640 1 -- 172.21.15.31:0/2517238858 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b9c011420 con 0x5651040c80f0 2024-03-21T10:44:03.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.988+0000 7f0ba8c00640 1 -- 172.21.15.31:0/2517238858 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c0105e0 con 0x5651040c80f0 2024-03-21T10:44:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.992+0000 7f0bbd1bf740 1 -- 172.21.15.31:0/2517238858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651040c80f0 msgr2=0x5651040c8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.992+0000 7f0bbd1bf740 1 --2- 172.21.15.31:0/2517238858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651040c80f0 0x5651040c8570 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c009820 tx=0x7f0b9c009ce0 comp rx=0 tx=0).stop 2024-03-21T10:44:03.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.993+0000 7f0bbd1bf740 1 -- 172.21.15.31:0/2517238858 shutdown_connections 2024-03-21T10:44:03.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.993+0000 7f0bbd1bf740 1 --2- 172.21.15.31:0/2517238858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651040d1460 0x5651040d3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:03.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.993+0000 7f0bbd1bf740 1 --2- 172.21.15.31:0/2517238858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651040c80f0 0x5651040c8570 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:03.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.993+0000 7f0bbd1bf740 1 --2- 172.21.15.31:0/2517238858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651040c8dd0 0x5651040c7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:03.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.993+0000 7f0bbd1bf740 1 -- 172.21.15.31:0/2517238858 >> 172.21.15.31:0/2517238858 conn(0x56510408f200 msgr2=0x5651040c2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:03.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.994+0000 7f0bbd1bf740 1 -- 172.21.15.31:0/2517238858 shutdown_connections 2024-03-21T10:44:03.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.995+0000 7f0bbd1bf740 1 -- 172.21.15.31:0/2517238858 wait complete. 2024-03-21T10:44:03.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.996+0000 7f0bbd1bf740 1 -- start start 2024-03-21T10:44:03.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.996+0000 7f0bbd1bf740 1 Processor -- start 2024-03-21T10:44:03.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.997+0000 7f0bbd1bf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651040c80f0 0x5651040d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:03.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.998+0000 7f0bbd1bf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651040c8dd0 0x5651040cde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:03.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.998+0000 7f0bbd1bf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651040d1460 0x5651040ce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:03.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.998+0000 7f0bbd1bf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651040d4370 con 0x5651040d1460 2024-03-21T10:44:03.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.998+0000 7f0bbd1bf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x565103fe7e80 con 0x5651040c80f0 2024-03-21T10:44:03.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.998+0000 7f0bbd1bf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651040d44f0 con 0x5651040c8dd0 2024-03-21T10:44:03.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.998+0000 7f0bbd1bf740 10 client.0 _ll_register_callbacks cb 0x565103ffa130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:03.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.998+0000 7f0bb3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651040c80f0 0x5651040d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:03.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.999+0000 7f0bb3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651040d1460 0x5651040ce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:03.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.999+0000 7f0bb3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651040c80f0 0x5651040d0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51528/0 (socket says 172.21.15.31:51528) 2024-03-21T10:44:03.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:03.999+0000 7f0bb3400640 1 -- 172.21.15.31:0/2210381198 learned_addr learned my addr 172.21.15.31:0/2210381198 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:03.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.000+0000 7f0baaa00640 1 --2- 172.21.15.31:0/2210381198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651040c8dd0 0x5651040cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:03.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.000+0000 7f0bb3e00640 1 -- 172.21.15.31:0/2210381198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651040c8dd0 msgr2=0x5651040cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:03.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.000+0000 7f0bb3e00640 1 --2- 172.21.15.31:0/2210381198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651040c8dd0 0x5651040cde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:03.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.000+0000 7f0bb3e00640 1 -- 172.21.15.31:0/2210381198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651040c80f0 msgr2=0x5651040d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:03.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.000+0000 7f0bb3e00640 1 --2- 172.21.15.31:0/2210381198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651040c80f0 0x5651040d0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:03.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.000+0000 7f0bb3e00640 1 -- 172.21.15.31:0/2210381198 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565103e0a7d0 con 0x5651040d1460 2024-03-21T10:44:04.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.006+0000 7f0ba1600640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:44:04.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.006+0000 7f0bb3e00640 1 --2- 172.21.15.31:0/2210381198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651040d1460 0x5651040ce3b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0bac0eebe0 tx=0x7f0bac0bd940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1d7d5113a6dfd1d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:04.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.006+0000 7f0ba1600640 1 -- 172.21.15.31:0/2210381198 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac0b9ef0 con 0x5651040d1460 2024-03-21T10:44:04.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.007+0000 7f0ba1600640 1 -- 172.21.15.31:0/2210381198 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f0bac0f1360 con 0x5651040d1460 2024-03-21T10:44:04.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.007+0000 7f0bbd1bf740 1 -- 172.21.15.31:0/2210381198 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0b9c011420 con 0x5651040d1460 2024-03-21T10:44:04.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.008+0000 7f0ba1600640 1 -- 172.21.15.31:0/2210381198 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0bac0db050 con 0x5651040d1460 2024-03-21T10:44:04.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.008+0000 7f0bbd1bf740 10 client.16485 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:04.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.008+0000 7f0bbd1bf740 1 -- 172.21.15.31:0/2210381198 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x565103d403f0 con 0x5651040d1460 2024-03-21T10:44:04.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.008+0000 7f0ba1600640 1 -- 172.21.15.31:0/2210381198 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b9c011420 con 0x5651040d1460 2024-03-21T10:44:04.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.009+0000 7f0ba1600640 1 -- 172.21.15.31:0/2210381198 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f0bac0d6bb0 con 0x5651040d1460 2024-03-21T10:44:04.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.009+0000 7f0bbd1bf740 10 client.16485 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:04.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.009+0000 7f0bbd1bf740 10 client.16485 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:44:04.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.009+0000 7f0bbd1bf740 20 client.16485 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:04.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.009+0000 7f0bbd1bf740 1 --2- 172.21.15.31:0/2210381198 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5651040e1340 0x5651040e3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:04.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.009+0000 7f0bbd1bf740 4 client.16485 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:44:04.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.009+0000 7f0bbd1bf740 1 -- 172.21.15.31:0/2210381198 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565104039790 con 0x5651040e1340 2024-03-21T10:44:04.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.010+0000 7f0bb3400640 1 --2- 172.21.15.31:0/2210381198 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5651040e1340 0x5651040e3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:04.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.012+0000 7f0bb3400640 1 --2- 172.21.15.31:0/2210381198 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5651040e1340 0x5651040e3730 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ad7393ac480cdc6 server_cookie=b61a936bbd21a934 in_seq=0 out_seq=0 2024-03-21T10:44:04.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.012+0000 7f0ba1600640 10 client.16485 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:04.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.013+0000 7f0ba1600640 1 -- 172.21.15.31:0/2210381198 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x565104039790 con 0x5651040e1340 2024-03-21T10:44:04.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.013+0000 7f0ba1600640 10 client.16485 handle_command_reply: tid=0 2024-03-21T10:44:04.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.013+0000 7f0bbd1bf740 1 client.16485 shutdown 2024-03-21T10:44:04.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.016+0000 7f0bbd1bf740 1 -- 172.21.15.31:0/2210381198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651040d1460 msgr2=0x5651040ce3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:04.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.016+0000 7f0bbd1bf740 1 --2- 172.21.15.31:0/2210381198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651040d1460 0x5651040ce3b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0bac0eebe0 tx=0x7f0bac0bd940 comp rx=0 tx=0).stop 2024-03-21T10:44:04.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.016+0000 7f0bbd1bf740 1 -- 172.21.15.31:0/2210381198 shutdown_connections 2024-03-21T10:44:04.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.016+0000 7f0bbd1bf740 1 --2- 172.21.15.31:0/2210381198 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5651040e1340 0x5651040e3730 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.016+0000 7f0bbd1bf740 1 --2- 172.21.15.31:0/2210381198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651040d1460 0x5651040ce3b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.016+0000 7f0bbd1bf740 1 --2- 172.21.15.31:0/2210381198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651040c8dd0 0x5651040cde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.016+0000 7f0bbd1bf740 1 --2- 172.21.15.31:0/2210381198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651040c80f0 0x5651040d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.016+0000 7f0bbd1bf740 1 -- 172.21.15.31:0/2210381198 >> 172.21.15.31:0/2210381198 conn(0x56510408f200 msgr2=0x5651040c4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:04.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.016+0000 7f0ba1600640 0 client.16485 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:04.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.017+0000 7f0bbd1bf740 1 -- 172.21.15.31:0/2210381198 shutdown_connections 2024-03-21T10:44:04.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.018+0000 7f0bbd1bf740 1 -- 172.21.15.31:0/2210381198 wait complete. 2024-03-21T10:44:04.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.018+0000 7f0bbd1bf740 20 client.16485 trim_cache size 0 max 16384 2024-03-21T10:44:04.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.060+0000 7f0bbd1bf740 1 Processor -- start 2024-03-21T10:44:04.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.061+0000 7f0bbd1bf740 1 -- start start 2024-03-21T10:44:04.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.061+0000 7f0bbd1bf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651040c80f0 0x5651040d2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:04.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.061+0000 7f0bbd1bf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651040c8dd0 0x5651040d2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:04.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.061+0000 7f0bbd1bf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651040d1460 0x5651040d05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:04.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.062+0000 7f0bb3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651040c80f0 0x5651040d2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:04.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.062+0000 7f0bb3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651040c80f0 0x5651040d2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51532/0 (socket says 172.21.15.31:51532) 2024-03-21T10:44:04.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.062+0000 7f0bb3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651040d1460 0x5651040d05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:04.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.062+0000 7f0bbd1bf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb41b7200 con 0x5651040d1460 2024-03-21T10:44:04.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.062+0000 7f0bbd1bf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb41b4d00 con 0x5651040c80f0 2024-03-21T10:44:04.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.062+0000 7f0bbd1bf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb41b2da0 con 0x5651040c8dd0 2024-03-21T10:44:04.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.062+0000 7f0baaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651040c8dd0 0x5651040d2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:04.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.062+0000 7f0bb3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651040d1460 0x5651040d05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52796/0 (socket says 172.21.15.31:52796) 2024-03-21T10:44:04.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.062+0000 7f0bb3e00640 1 -- 172.21.15.31:0/1573204256 learned_addr learned my addr 172.21.15.31:0/1573204256 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:04.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.063+0000 7f0bb3e00640 1 -- 172.21.15.31:0/1573204256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651040c8dd0 msgr2=0x5651040d2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:04.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.063+0000 7f0bb3e00640 1 --2- 172.21.15.31:0/1573204256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651040c8dd0 0x5651040d2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.063+0000 7f0bb3e00640 1 -- 172.21.15.31:0/1573204256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651040c80f0 msgr2=0x5651040d2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:04.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.063+0000 7f0bb3e00640 1 --2- 172.21.15.31:0/1573204256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651040c80f0 0x5651040d2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.063+0000 7f0bb3e00640 1 -- 172.21.15.31:0/1573204256 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5651040e3aa0 con 0x5651040d1460 2024-03-21T10:44:04.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.063+0000 7f0bb3400640 1 --2- 172.21.15.31:0/1573204256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651040c80f0 0x5651040d2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-21T10:44:04.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.064+0000 7f0baaa00640 1 --2- 172.21.15.31:0/1573204256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651040c8dd0 0x5651040d2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:44:04.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.064+0000 7f0bb3e00640 1 --2- 172.21.15.31:0/1573204256 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651040d1460 0x5651040d05a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0bac0ef0d0 tx=0x7f0bac0c6410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea003bbc42a63a81 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:04.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.064+0000 7f0ba3e00640 1 -- 172.21.15.31:0/1573204256 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac0c19d0 con 0x5651040d1460 2024-03-21T10:44:04.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.067+0000 7f0ba3e00640 1 -- 172.21.15.31:0/1573204256 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0bac0c5e00 con 0x5651040d1460 2024-03-21T10:44:04.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.067+0000 7f0ba3e00640 1 -- 172.21.15.31:0/1573204256 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac0dc3b0 con 0x5651040d1460 2024-03-21T10:44:04.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.068+0000 7f0bbd1bf740 1 -- 172.21.15.31:0/1573204256 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651040d1460 msgr2=0x5651040d05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:04.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.068+0000 7f0bbd1bf740 1 --2- 172.21.15.31:0/1573204256 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651040d1460 0x5651040d05a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0bac0ef0d0 tx=0x7f0bac0c6410 comp rx=0 tx=0).stop 2024-03-21T10:44:04.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.069+0000 7f0bbd1bf740 1 -- 172.21.15.31:0/1573204256 shutdown_connections 2024-03-21T10:44:04.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.069+0000 7f0bbd1bf740 1 --2- 172.21.15.31:0/1573204256 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651040d1460 0x5651040d05a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.069+0000 7f0bbd1bf740 1 --2- 172.21.15.31:0/1573204256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651040c8dd0 0x5651040d2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.069+0000 7f0bbd1bf740 1 --2- 172.21.15.31:0/1573204256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651040c80f0 0x5651040d2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.069+0000 7f0bbd1bf740 1 -- 172.21.15.31:0/1573204256 >> 172.21.15.31:0/1573204256 conn(0x56510408f200 msgr2=0x5651040c1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:04.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.070+0000 7f0bbd1bf740 1 -- 172.21.15.31:0/1573204256 shutdown_connections 2024-03-21T10:44:04.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.070+0000 7f0bbd1bf740 1 -- 172.21.15.31:0/1573204256 wait complete. 2024-03-21T10:44:04.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.071+0000 7f0bbd1bf740 1 -- start start 2024-03-21T10:44:04.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.071+0000 7f0bbd1bf740 1 Processor -- start 2024-03-21T10:44:04.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.072+0000 7f0bbd1bf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651040c80f0 0x565104134e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:04.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.072+0000 7f0bbd1bf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651040c8dd0 0x565104135390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:04.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.072+0000 7f0bbd1bf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651040d1460 0x5651040cade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:04.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.072+0000 7f0bbd1bf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb41b33a0 con 0x5651040c8dd0 2024-03-21T10:44:04.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.072+0000 7f0bbd1bf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb41b46a0 con 0x5651040c80f0 2024-03-21T10:44:04.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.072+0000 7f0bbd1bf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb41b39e0 con 0x5651040d1460 2024-03-21T10:44:04.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.072+0000 7f0bbd1bf740 10 client.0 _ll_register_callbacks cb 0x7f0bac0b9b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:04.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.072+0000 7f0baaa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651040c8dd0 0x565104135390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:04.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.073+0000 7f0bb3e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651040d1460 0x5651040cade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:04.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.073+0000 7f0bb3e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651040d1460 0x5651040cade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:48080/0 (socket says 172.21.15.31:48080) 2024-03-21T10:44:04.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.073+0000 7f0bb3e00640 1 -- 172.21.15.31:0/2926329762 learned_addr learned my addr 172.21.15.31:0/2926329762 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:04.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.073+0000 7f0bb3400640 1 --2- 172.21.15.31:0/2926329762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651040c80f0 0x565104134e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:04.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.074+0000 7f0baaa00640 1 -- 172.21.15.31:0/2926329762 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651040d1460 msgr2=0x5651040cade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:04.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.074+0000 7f0baaa00640 1 --2- 172.21.15.31:0/2926329762 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651040d1460 0x5651040cade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.074+0000 7f0baaa00640 1 -- 172.21.15.31:0/2926329762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651040c80f0 msgr2=0x565104134e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:04.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.074+0000 7f0baaa00640 1 --2- 172.21.15.31:0/2926329762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651040c80f0 0x565104134e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.074+0000 7f0baaa00640 1 -- 172.21.15.31:0/2926329762 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565103d000f0 con 0x5651040c8dd0 2024-03-21T10:44:04.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.074+0000 7f0ba1600640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:44:04.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.074+0000 7f0baaa00640 1 --2- 172.21.15.31:0/2926329762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651040c8dd0 0x565104135390 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x5651040bd070 tx=0x7f0ba4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd86d9878a4eed09 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:04.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.075+0000 7f0ba1600640 1 -- 172.21.15.31:0/2926329762 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4003b70 con 0x5651040c8dd0 2024-03-21T10:44:04.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.075+0000 7f0ba1600640 1 -- 172.21.15.31:0/2926329762 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f0ba4003970 con 0x5651040c8dd0 2024-03-21T10:44:04.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.075+0000 7f0bbd1bf740 1 -- 172.21.15.31:0/2926329762 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0bac0c5e00 con 0x5651040c8dd0 2024-03-21T10:44:04.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.076+0000 7f0ba1600640 1 -- 172.21.15.31:0/2926329762 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ba4003b70 con 0x5651040c8dd0 2024-03-21T10:44:04.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.076+0000 7f0bbd1bf740 10 client.16491 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:04.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.076+0000 7f0bbd1bf740 1 -- 172.21.15.31:0/2926329762 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x56510413d580 con 0x5651040c8dd0 2024-03-21T10:44:04.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.077+0000 7f0ba1600640 1 -- 172.21.15.31:0/2926329762 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ba4033950 con 0x5651040c8dd0 2024-03-21T10:44:04.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.077+0000 7f0ba1600640 1 -- 172.21.15.31:0/2926329762 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f0ba400f040 con 0x5651040c8dd0 2024-03-21T10:44:04.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.077+0000 7f0bbd1bf740 10 client.16491 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:04.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.077+0000 7f0bbd1bf740 10 client.16491 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:44:04.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.077+0000 7f0bbd1bf740 20 client.16491 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:04.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.077+0000 7f0bbd1bf740 1 --2- 172.21.15.31:0/2926329762 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56510413dcd0 0x5651041400f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:04.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.077+0000 7f0bbd1bf740 4 client.16491 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:44:04.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.077+0000 7f0bbd1bf740 1 -- 172.21.15.31:0/2926329762 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f0bac0c5e00 con 0x56510413dcd0 2024-03-21T10:44:04.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.078+0000 7f0bb3400640 1 --2- 172.21.15.31:0/2926329762 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56510413dcd0 0x5651041400f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:04.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.081+0000 7f0bb3400640 1 --2- 172.21.15.31:0/2926329762 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56510413dcd0 0x5651041400f0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9db079f5636e4ec2 server_cookie=263f786b08f83a76 in_seq=0 out_seq=0 2024-03-21T10:44:04.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.082+0000 7f0ba1600640 10 client.16491 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:04.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.083+0000 7f0ba1600640 1 -- 172.21.15.31:0/2926329762 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x7f0bac0c5e00 con 0x56510413dcd0 2024-03-21T10:44:04.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.083+0000 7f0ba1600640 10 client.16491 handle_command_reply: tid=0 2024-03-21T10:44:04.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.084+0000 7f0bbd1bf740 1 client.16491 shutdown 2024-03-21T10:44:04.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.085+0000 7f0bbd1bf740 1 -- 172.21.15.31:0/2926329762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651040c8dd0 msgr2=0x565104135390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:04.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.085+0000 7f0bbd1bf740 1 --2- 172.21.15.31:0/2926329762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651040c8dd0 0x565104135390 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x5651040bd070 tx=0x7f0ba4004ab0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.088+0000 7f0bbd1bf740 1 -- 172.21.15.31:0/2926329762 shutdown_connections 2024-03-21T10:44:04.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.088+0000 7f0bbd1bf740 1 --2- 172.21.15.31:0/2926329762 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56510413dcd0 0x5651041400f0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.088+0000 7f0bbd1bf740 1 --2- 172.21.15.31:0/2926329762 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5651040d1460 0x5651040cade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.088+0000 7f0bbd1bf740 1 --2- 172.21.15.31:0/2926329762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651040c8dd0 0x565104135390 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.088+0000 7f0bbd1bf740 1 --2- 172.21.15.31:0/2926329762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651040c80f0 0x565104134e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.088+0000 7f0bbd1bf740 1 -- 172.21.15.31:0/2926329762 >> 172.21.15.31:0/2926329762 conn(0x56510408f200 msgr2=0x5651040cd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:04.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.090+0000 7f0ba1600640 0 client.16491 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:04.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.094+0000 7f0bbd1bf740 1 -- 172.21.15.31:0/2926329762 shutdown_connections 2024-03-21T10:44:04.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.097+0000 7f0bbd1bf740 1 -- 172.21.15.31:0/2926329762 wait complete. 2024-03-21T10:44:04.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.098+0000 7f0bbd1bf740 20 client.16491 trim_cache size 0 max 16384 2024-03-21T10:44:04.100 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:44:04.100 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-03-21T10:44:04.100 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-03-21T10:44:04.100 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:44:04.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.105+0000 7f0bb8e00640 1 -- 172.21.15.31:0/1347154548 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f0b980a3220 msgr2=0x7f0b980a56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:04.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.105+0000 7f0bb8e00640 1 --2- 172.21.15.31:0/1347154548 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f0b980a3220 0x7f0b980a56e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0bb40ff710 tx=0x7f0ba4009c10 comp rx=0 tx=0).stop 2024-03-21T10:44:04.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.105+0000 7f0bb8e00640 1 -- 172.21.15.31:0/1347154548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0bb410ba50 msgr2=0x7f0bb41a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:04.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.105+0000 7f0bb8e00640 1 --2- 172.21.15.31:0/1347154548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0bb410ba50 0x7f0bb41a2c40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0bac02c270 tx=0x7f0bac0045c0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.105+0000 7f0bb8e00640 1 -- 172.21.15.31:0/1347154548 shutdown_connections 2024-03-21T10:44:04.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.105+0000 7f0bb8e00640 1 --2- 172.21.15.31:0/1347154548 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f0b980a3220 0x7f0b980a56e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.105+0000 7f0bb8e00640 1 --2- 172.21.15.31:0/1347154548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0bb410ba50 0x7f0bb41a2c40 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.105+0000 7f0bb8e00640 1 --2- 172.21.15.31:0/1347154548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0bb4102f50 0x7f0bb419e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.105+0000 7f0bb8e00640 1 --2- 172.21.15.31:0/1347154548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0bb4102630 0x7f0bb419e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.105+0000 7f0bb8e00640 1 -- 172.21.15.31:0/1347154548 >> 172.21.15.31:0/1347154548 conn(0x7f0bb4077130 msgr2=0x7f0bb40fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:04.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.108+0000 7f0bb8e00640 1 -- 172.21.15.31:0/1347154548 shutdown_connections 2024-03-21T10:44:04.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.110+0000 7f0bb8e00640 1 -- 172.21.15.31:0/1347154548 wait complete. 2024-03-21T10:44:04.145 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-21T10:44:04.146 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:44:04.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.479+0000 7f85fac00640 1 -- 172.21.15.31:0/1970533991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f40ff860 msgr2=0x7f85f40ffc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:04.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.479+0000 7f85fac00640 1 --2- 172.21.15.31:0/1970533991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f40ff860 0x7f85f40ffc40 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f85e401acb0 tx=0x7f85e403d1c0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.480+0000 7f85fac00640 1 -- 172.21.15.31:0/1970533991 shutdown_connections 2024-03-21T10:44:04.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.480+0000 7f85fac00640 1 --2- 172.21.15.31:0/1970533991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85f4112580 0x7f85f4114970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.480+0000 7f85fac00640 1 --2- 172.21.15.31:0/1970533991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85f4100180 0x7f85f4112040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.480+0000 7f85fac00640 1 --2- 172.21.15.31:0/1970533991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f40ff860 0x7f85f40ffc40 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.480+0000 7f85fac00640 1 -- 172.21.15.31:0/1970533991 >> 172.21.15.31:0/1970533991 conn(0x7f85f40fb990 msgr2=0x7f85f40fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:04.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.480+0000 7f85fac00640 1 -- 172.21.15.31:0/1970533991 shutdown_connections 2024-03-21T10:44:04.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.480+0000 7f85fac00640 1 -- 172.21.15.31:0/1970533991 wait complete. 2024-03-21T10:44:04.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.482+0000 7f85fac00640 1 Processor -- start 2024-03-21T10:44:04.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.483+0000 7f85fac00640 1 -- start start 2024-03-21T10:44:04.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.484+0000 7f85fac00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85f40ff860 0x7f85f4111c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:04.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.484+0000 7f85fac00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85f4100180 0x7f85f410dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:04.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.484+0000 7f85fac00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f4112580 0x7f85f410e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:04.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.484+0000 7f85ebe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85f4100180 0x7f85f410dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:04.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.485+0000 7f85ebe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85f4100180 0x7f85f410dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:48102/0 (socket says 172.21.15.31:48102) 2024-03-21T10:44:04.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.485+0000 7f85f8c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f4112580 0x7f85f410e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:04.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.485+0000 7f85f3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85f40ff860 0x7f85f4111c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:04.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.485+0000 7f85f8c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f4112580 0x7f85f410e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51548/0 (socket says 172.21.15.31:51548) 2024-03-21T10:44:04.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.485+0000 7f85ebe00640 1 -- 172.21.15.31:0/3553245904 learned_addr learned my addr 172.21.15.31:0/3553245904 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:04.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.485+0000 7f85fac00640 1 -- 172.21.15.31:0/3553245904 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f410a1b0 con 0x7f85f40ff860 2024-03-21T10:44:04.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.485+0000 7f85fac00640 1 -- 172.21.15.31:0/3553245904 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f410a030 con 0x7f85f4112580 2024-03-21T10:44:04.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.485+0000 7f85fac00640 1 -- 172.21.15.31:0/3553245904 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f410a330 con 0x7f85f4100180 2024-03-21T10:44:04.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.489+0000 7f85ebe00640 1 -- 172.21.15.31:0/3553245904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f4112580 msgr2=0x7f85f410e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:04.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.489+0000 7f85ebe00640 1 --2- 172.21.15.31:0/3553245904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f4112580 0x7f85f410e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.489+0000 7f85ebe00640 1 -- 172.21.15.31:0/3553245904 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85f40ff860 msgr2=0x7f85f4111c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:04.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.489+0000 7f85ebe00640 1 --2- 172.21.15.31:0/3553245904 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85f40ff860 0x7f85f4111c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:04.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.489+0000 7f85ebe00640 1 -- 172.21.15.31:0/3553245904 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f410e970 con 0x7f85f4100180 2024-03-21T10:44:04.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.489+0000 7f85f8c00640 1 --2- 172.21.15.31:0/3553245904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f4112580 0x7f85f410e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:44:04.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.490+0000 7f85ebe00640 1 --2- 172.21.15.31:0/3553245904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85f4100180 0x7f85f410dd00 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f85dc00d950 tx=0x7f85dc00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:04.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.490+0000 7f85f1600640 1 -- 172.21.15.31:0/3553245904 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc017070 con 0x7f85f4100180 2024-03-21T10:44:04.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.491+0000 7f85f1600640 1 -- 172.21.15.31:0/3553245904 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f85dc009c30 con 0x7f85f4100180 2024-03-21T10:44:04.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.491+0000 7f85f1600640 1 -- 172.21.15.31:0/3553245904 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc00f6b0 con 0x7f85f4100180 2024-03-21T10:44:04.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.491+0000 7f85fac00640 1 -- 172.21.15.31:0/3553245904 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f41a23f0 con 0x7f85f4100180 2024-03-21T10:44:04.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.491+0000 7f85fac00640 1 -- 172.21.15.31:0/3553245904 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f41a27a0 con 0x7f85f4100180 2024-03-21T10:44:04.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.495+0000 7f85fac00640 1 -- 172.21.15.31:0/3553245904 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f410eb40 con 0x7f85f4100180 2024-03-21T10:44:04.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.502+0000 7f85f1600640 1 -- 172.21.15.31:0/3553245904 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f85dc0109d0 con 0x7f85f4100180 2024-03-21T10:44:04.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.505+0000 7f85f1600640 1 --2- 172.21.15.31:0/3553245904 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f85d80a31e0 0x7f85d80a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:04.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.505+0000 7f85f1600640 1 -- 172.21.15.31:0/3553245904 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f85dc0cf5c0 con 0x7f85f4100180 2024-03-21T10:44:04.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.506+0000 7f85f3e00640 1 --2- 172.21.15.31:0/3553245904 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f85d80a31e0 0x7f85d80a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:04.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.507+0000 7f85f1600640 1 -- 172.21.15.31:0/3553245904 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f85dc009de0 con 0x7f85f4100180 2024-03-21T10:44:04.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:04.509+0000 7f85f3e00640 1 --2- 172.21.15.31:0/3553245904 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f85d80a31e0 0x7f85d80a56a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f85e4000f80 tx=0x7f85e401a370 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:05.073 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:04 smithi070 ceph-mon[36330]: pgmap v168: 129 pgs: 129 active+clean; 34 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 196 MiB/s wr, 1.03k op/s 2024-03-21T10:44:05.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.076+0000 7f85fac00640 1 -- 172.21.15.31:0/3553245904 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85f4100b60 con 0x7f85f4100180 2024-03-21T10:44:05.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.077+0000 7f85f1600640 1 -- 172.21.15.31:0/3553245904 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+11384 (secure 0 0 0) 0x7f85dc09be70 con 0x7f85f4100180 2024-03-21T10:44:05.079 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:44:05.079 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:42:31.182549+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24547,24596]},"id":1}]} 2024-03-21T10:44:05.079 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2024-03-21T10:44:05.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.087+0000 7f85fac00640 1 -- 172.21.15.31:0/3553245904 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f85d80a31e0 msgr2=0x7f85d80a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:05.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.087+0000 7f85fac00640 1 --2- 172.21.15.31:0/3553245904 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f85d80a31e0 0x7f85d80a56a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f85e4000f80 tx=0x7f85e401a370 comp rx=0 tx=0).stop 2024-03-21T10:44:05.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.087+0000 7f85fac00640 1 -- 172.21.15.31:0/3553245904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85f4100180 msgr2=0x7f85f410dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:05.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.087+0000 7f85fac00640 1 --2- 172.21.15.31:0/3553245904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85f4100180 0x7f85f410dd00 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f85dc00d950 tx=0x7f85dc00de10 comp rx=0 tx=0).stop 2024-03-21T10:44:05.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.088+0000 7f85fac00640 1 -- 172.21.15.31:0/3553245904 shutdown_connections 2024-03-21T10:44:05.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.088+0000 7f85fac00640 1 --2- 172.21.15.31:0/3553245904 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f85d80a31e0 0x7f85d80a56a0 secure :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f85e4000f80 tx=0x7f85e401a370 comp rx=0 tx=0).stop 2024-03-21T10:44:05.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.088+0000 7f85fac00640 1 --2- 172.21.15.31:0/3553245904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f4112580 0x7f85f410e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.088+0000 7f85fac00640 1 --2- 172.21.15.31:0/3553245904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85f4100180 0x7f85f410dd00 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.088+0000 7f85fac00640 1 --2- 172.21.15.31:0/3553245904 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85f40ff860 0x7f85f4111c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.088+0000 7f85fac00640 1 -- 172.21.15.31:0/3553245904 >> 172.21.15.31:0/3553245904 conn(0x7f85f40fb990 msgr2=0x7f85f4113730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:05.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.088+0000 7f85fac00640 1 -- 172.21.15.31:0/3553245904 shutdown_connections 2024-03-21T10:44:05.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.088+0000 7f85fac00640 1 -- 172.21.15.31:0/3553245904 wait complete. 2024-03-21T10:44:05.129 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:04 smithi031 ceph-mon[31673]: pgmap v168: 129 pgs: 129 active+clean; 34 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 196 MiB/s wr, 1.03k op/s 2024-03-21T10:44:05.134 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:44:05.134 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-21T10:44:05.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:04 smithi040 ceph-mon[36154]: pgmap v168: 129 pgs: 129 active+clean; 34 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 196 MiB/s wr, 1.03k op/s 2024-03-21T10:44:05.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.590+0000 7ff733e00640 1 -- 172.21.15.31:0/667957902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff734101e60 msgr2=0x7ff73410e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:05.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.590+0000 7ff733e00640 1 --2- 172.21.15.31:0/667957902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff734101e60 0x7ff73410e320 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff72c009ca0 tx=0x7ff72c02c100 comp rx=0 tx=0).stop 2024-03-21T10:44:05.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.590+0000 7ff733e00640 1 -- 172.21.15.31:0/667957902 shutdown_connections 2024-03-21T10:44:05.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.590+0000 7ff733e00640 1 --2- 172.21.15.31:0/667957902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff734101e60 0x7ff73410e320 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.590+0000 7ff733e00640 1 --2- 172.21.15.31:0/667957902 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7341014a0 0x7ff734101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.590+0000 7ff733e00640 1 --2- 172.21.15.31:0/667957902 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff734107560 0x7ff734107940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.590+0000 7ff733e00640 1 -- 172.21.15.31:0/667957902 >> 172.21.15.31:0/667957902 conn(0x7ff7340fcfb0 msgr2=0x7ff7340ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:05.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.591+0000 7ff733e00640 1 -- 172.21.15.31:0/667957902 shutdown_connections 2024-03-21T10:44:05.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.591+0000 7ff733e00640 1 -- 172.21.15.31:0/667957902 wait complete. 2024-03-21T10:44:05.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.593+0000 7ff733e00640 1 Processor -- start 2024-03-21T10:44:05.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.593+0000 7ff733e00640 1 -- start start 2024-03-21T10:44:05.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.593+0000 7ff733e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7341014a0 0x7ff73410cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:05.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.593+0000 7ff733e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff734101e60 0x7ff73410d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:05.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.594+0000 7ff733e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff734107560 0x7ff7341a02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:05.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.594+0000 7ff732a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7341014a0 0x7ff73410cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:05.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.594+0000 7ff732a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7341014a0 0x7ff73410cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51556/0 (socket says 172.21.15.31:51556) 2024-03-21T10:44:05.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.594+0000 7ff733400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff734107560 0x7ff7341a02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:05.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.594+0000 7ff733400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff734107560 0x7ff7341a02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:48134/0 (socket says 172.21.15.31:48134) 2024-03-21T10:44:05.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.594+0000 7ff733400640 1 -- 172.21.15.31:0/3847765622 learned_addr learned my addr 172.21.15.31:0/3847765622 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:05.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.594+0000 7ff733e00640 1 -- 172.21.15.31:0/3847765622 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff734113a90 con 0x7ff734101e60 2024-03-21T10:44:05.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.594+0000 7ff733e00640 1 -- 172.21.15.31:0/3847765622 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff734113910 con 0x7ff7341014a0 2024-03-21T10:44:05.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.594+0000 7ff733e00640 1 -- 172.21.15.31:0/3847765622 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff734113c10 con 0x7ff734107560 2024-03-21T10:44:05.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.594+0000 7ff732000640 1 --2- 172.21.15.31:0/3847765622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff734101e60 0x7ff73410d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:05.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.595+0000 7ff732a00640 1 -- 172.21.15.31:0/3847765622 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff734107560 msgr2=0x7ff7341a02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:05.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.595+0000 7ff732a00640 1 --2- 172.21.15.31:0/3847765622 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff734107560 0x7ff7341a02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.595+0000 7ff732a00640 1 -- 172.21.15.31:0/3847765622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff734101e60 msgr2=0x7ff73410d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:05.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.595+0000 7ff732a00640 1 --2- 172.21.15.31:0/3847765622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff734101e60 0x7ff73410d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.595+0000 7ff732a00640 1 -- 172.21.15.31:0/3847765622 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7341a0a70 con 0x7ff7341014a0 2024-03-21T10:44:05.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.595+0000 7ff732000640 1 --2- 172.21.15.31:0/3847765622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff734101e60 0x7ff73410d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:44:05.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.595+0000 7ff732a00640 1 --2- 172.21.15.31:0/3847765622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7341014a0 0x7ff73410cca0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff72400b3f0 tx=0x7ff72400b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:05.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.595+0000 7ff72b400640 1 -- 172.21.15.31:0/3847765622 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff724015020 con 0x7ff7341014a0 2024-03-21T10:44:05.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.595+0000 7ff72b400640 1 -- 172.21.15.31:0/3847765622 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff724011d10 con 0x7ff7341014a0 2024-03-21T10:44:05.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.595+0000 7ff72b400640 1 -- 172.21.15.31:0/3847765622 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72400f2e0 con 0x7ff7341014a0 2024-03-21T10:44:05.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.595+0000 7ff733e00640 1 -- 172.21.15.31:0/3847765622 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7341a0d00 con 0x7ff7341014a0 2024-03-21T10:44:05.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.595+0000 7ff733e00640 1 -- 172.21.15.31:0/3847765622 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff734111b40 con 0x7ff7341014a0 2024-03-21T10:44:05.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.601+0000 7ff73c3fd740 1 Processor -- start 2024-03-21T10:44:05.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.601+0000 7ff73c3fd740 1 -- start start 2024-03-21T10:44:05.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.602+0000 7ff73c3fd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84ad8b0 0x55afe84adc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:05.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.602+0000 7ff73c3fd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afe84ae260 0x55afe84b6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:05.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.602+0000 7ff73c3fd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55afe84b6de0 0x55afe84b9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:05.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.602+0000 7ff73c3fd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afe84112c0 con 0x55afe84ae260 2024-03-21T10:44:05.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.602+0000 7ff73c3fd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afe83b9c90 con 0x55afe84ad8b0 2024-03-21T10:44:05.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.602+0000 7ff73c3fd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afe840d360 con 0x55afe84b6de0 2024-03-21T10:44:05.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.603+0000 7ff733400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84ad8b0 0x55afe84adc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:05.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.603+0000 7ff733400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84ad8b0 0x55afe84adc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51566/0 (socket says 172.21.15.31:51566) 2024-03-21T10:44:05.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.603+0000 7ff733400640 1 -- 172.21.15.31:0/2284116300 learned_addr learned my addr 172.21.15.31:0/2284116300 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:05.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.603+0000 7ff72b400640 1 -- 172.21.15.31:0/3847765622 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7ff724018020 con 0x7ff7341014a0 2024-03-21T10:44:05.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.603+0000 7ff733400640 1 -- 172.21.15.31:0/2284116300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55afe84b6de0 msgr2=0x55afe84b9230 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:44:05.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.603+0000 7ff733400640 1 --2- 172.21.15.31:0/2284116300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55afe84b6de0 0x55afe84b9230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.603+0000 7ff733400640 1 -- 172.21.15.31:0/2284116300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afe84ae260 msgr2=0x55afe84b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:05.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.603+0000 7ff733400640 1 --2- 172.21.15.31:0/2284116300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afe84ae260 0x55afe84b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.603+0000 7ff733400640 1 -- 172.21.15.31:0/2284116300 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55afe83de080 con 0x55afe84ad8b0 2024-03-21T10:44:05.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.604+0000 7ff733400640 1 --2- 172.21.15.31:0/2284116300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84ad8b0 0x55afe84adc90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7ff72c009a00 tx=0x7ff72c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3024efce5164b1e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:05.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.604+0000 7ff71fe00640 1 -- 172.21.15.31:0/2284116300 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72c043070 con 0x55afe84ad8b0 2024-03-21T10:44:05.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.604+0000 7ff71fe00640 1 -- 172.21.15.31:0/2284116300 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff72c003970 con 0x55afe84ad8b0 2024-03-21T10:44:05.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.604+0000 7ff71fe00640 1 -- 172.21.15.31:0/2284116300 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72c039040 con 0x55afe84ad8b0 2024-03-21T10:44:05.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.605+0000 7ff73c3fd740 1 -- 172.21.15.31:0/2284116300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84ad8b0 msgr2=0x55afe84adc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:05.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.605+0000 7ff73c3fd740 1 --2- 172.21.15.31:0/2284116300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84ad8b0 0x55afe84adc90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7ff72c009a00 tx=0x7ff72c005f90 comp rx=0 tx=0).stop 2024-03-21T10:44:05.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.605+0000 7ff73c3fd740 1 -- 172.21.15.31:0/2284116300 shutdown_connections 2024-03-21T10:44:05.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.605+0000 7ff73c3fd740 1 --2- 172.21.15.31:0/2284116300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55afe84b6de0 0x55afe84b9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.605+0000 7ff73c3fd740 1 --2- 172.21.15.31:0/2284116300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afe84ae260 0x55afe84b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.605+0000 7ff73c3fd740 1 --2- 172.21.15.31:0/2284116300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84ad8b0 0x55afe84adc90 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.605+0000 7ff72b400640 1 --2- 172.21.15.31:0/3847765622 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff7180a3190 0x7ff7180a5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:05.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.605+0000 7ff73c3fd740 1 -- 172.21.15.31:0/2284116300 >> 172.21.15.31:0/2284116300 conn(0x55afe84a7f80 msgr2=0x55afe84a8360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:05.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.606+0000 7ff72b400640 1 -- 172.21.15.31:0/3847765622 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7ff7240cf2c0 con 0x7ff7341014a0 2024-03-21T10:44:05.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.606+0000 7ff733400640 1 --2- 172.21.15.31:0/3847765622 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff7180a3190 0x7ff7180a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:05.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.606+0000 7ff73c3fd740 1 -- 172.21.15.31:0/2284116300 shutdown_connections 2024-03-21T10:44:05.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.606+0000 7ff73c3fd740 1 -- 172.21.15.31:0/2284116300 wait complete. 2024-03-21T10:44:05.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.607+0000 7ff73c3fd740 1 -- start start 2024-03-21T10:44:05.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.607+0000 7ff733400640 1 --2- 172.21.15.31:0/3847765622 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff7180a3190 0x7ff7180a5650 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff72c009620 tx=0x7ff72c004040 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:05.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.607+0000 7ff73c3fd740 1 Processor -- start 2024-03-21T10:44:05.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.608+0000 7ff73c3fd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55afe84ad8b0 0x55afe84b66c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:05.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.608+0000 7ff73c3fd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84ae260 0x55afe84b3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:05.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.608+0000 7ff73c3fd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afe84b6de0 0x55afe84b3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:05.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.608+0000 7ff73c3fd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afe84b9cf0 con 0x55afe84b6de0 2024-03-21T10:44:05.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.608+0000 7ff73c3fd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afe83cd0d0 con 0x55afe84ae260 2024-03-21T10:44:05.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.608+0000 7ff73c3fd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afe84b9e70 con 0x55afe84ad8b0 2024-03-21T10:44:05.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.608+0000 7ff73c3fd740 10 client.0 _ll_register_callbacks cb 0x55afe83d1e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:05.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.608+0000 7ff732000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84ae260 0x55afe84b3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:05.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.608+0000 7ff732000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84ae260 0x55afe84b3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51568/0 (socket says 172.21.15.31:51568) 2024-03-21T10:44:05.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.608+0000 7ff732000640 1 -- 172.21.15.31:0/1128741476 learned_addr learned my addr 172.21.15.31:0/1128741476 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:05.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.608+0000 7ff733400640 1 --2- 172.21.15.31:0/1128741476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afe84b6de0 0x55afe84b3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:05.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.608+0000 7ff732a00640 1 --2- 172.21.15.31:0/1128741476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55afe84ad8b0 0x55afe84b66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:05.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.608+0000 7ff732000640 1 -- 172.21.15.31:0/1128741476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55afe84ad8b0 msgr2=0x55afe84b66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:05.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.609+0000 7ff732000640 1 --2- 172.21.15.31:0/1128741476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55afe84ad8b0 0x55afe84b66c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.609+0000 7ff732000640 1 -- 172.21.15.31:0/1128741476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afe84b6de0 msgr2=0x55afe84b3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:05.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.609+0000 7ff732000640 1 --2- 172.21.15.31:0/1128741476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afe84b6de0 0x55afe84b3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.609+0000 7ff732000640 1 -- 172.21.15.31:0/1128741476 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55afe81253f0 con 0x55afe84ae260 2024-03-21T10:44:05.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.609+0000 7ff733400640 1 --2- 172.21.15.31:0/1128741476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afe84b6de0 0x55afe84b3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:44:05.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.609+0000 7ff732a00640 1 --2- 172.21.15.31:0/1128741476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55afe84ad8b0 0x55afe84b66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:44:05.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.609+0000 7ff732000640 1 --2- 172.21.15.31:0/1128741476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84ae260 0x55afe84b3a50 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff720009fd0 tx=0x7ff72000e780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99bdfc51a90b4ef3 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:05.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.609+0000 7ff71cc00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:44:05.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.609+0000 7ff71cc00640 1 -- 172.21.15.31:0/1128741476 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff720016070 con 0x55afe84ae260 2024-03-21T10:44:05.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.609+0000 7ff73c3fd740 1 -- 172.21.15.31:0/1128741476 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff72c003970 con 0x55afe84ae260 2024-03-21T10:44:05.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.609+0000 7ff71cc00640 1 -- 172.21.15.31:0/1128741476 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7ff720011880 con 0x55afe84ae260 2024-03-21T10:44:05.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.609+0000 7ff71cc00640 1 -- 172.21.15.31:0/1128741476 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff720022050 con 0x55afe84ae260 2024-03-21T10:44:05.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.609+0000 7ff73c3fd740 10 client.26993 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:05.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.609+0000 7ff73c3fd740 1 -- 172.21.15.31:0/1128741476 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55afe81ef7d0 con 0x55afe84ae260 2024-03-21T10:44:05.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.610+0000 7ff71cc00640 1 -- 172.21.15.31:0/1128741476 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff72c003970 con 0x55afe84ae260 2024-03-21T10:44:05.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.610+0000 7ff71cc00640 1 -- 172.21.15.31:0/1128741476 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7ff72000f040 con 0x55afe84ae260 2024-03-21T10:44:05.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.610+0000 7ff73c3fd740 10 client.26993 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:05.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.610+0000 7ff73c3fd740 10 client.26993 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:44:05.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.610+0000 7ff73c3fd740 20 client.26993 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:05.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.610+0000 7ff73c3fd740 1 --2- 172.21.15.31:0/1128741476 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55afe84c6bc0 0x55afe84c8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:05.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.610+0000 7ff73c3fd740 4 client.26993 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:44:05.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.610+0000 7ff73c3fd740 1 -- 172.21.15.31:0/1128741476 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55afe841e800 con 0x55afe84c6bc0 2024-03-21T10:44:05.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.611+0000 7ff732a00640 1 --2- 172.21.15.31:0/1128741476 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55afe84c6bc0 0x55afe84c8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:05.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.612+0000 7ff732a00640 1 --2- 172.21.15.31:0/1128741476 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55afe84c6bc0 0x55afe84c8fb0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7f76fdf03c594403 server_cookie=81fc19eb14250fce in_seq=0 out_seq=0 2024-03-21T10:44:05.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.612+0000 7ff71cc00640 10 client.26993 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:44:05.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.613+0000 7ff71cc00640 1 -- 172.21.15.31:0/1128741476 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55afe841e800 con 0x55afe84c6bc0 2024-03-21T10:44:05.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.613+0000 7ff71cc00640 10 client.26993 handle_command_reply: tid=0 2024-03-21T10:44:05.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.613+0000 7ff73c3fd740 1 client.26993 shutdown 2024-03-21T10:44:05.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.613+0000 7ff73c3fd740 1 -- 172.21.15.31:0/1128741476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84ae260 msgr2=0x55afe84b3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:05.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.613+0000 7ff73c3fd740 1 --2- 172.21.15.31:0/1128741476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84ae260 0x55afe84b3a50 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff720009fd0 tx=0x7ff72000e780 comp rx=0 tx=0).stop 2024-03-21T10:44:05.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.614+0000 7ff73c3fd740 1 -- 172.21.15.31:0/1128741476 shutdown_connections 2024-03-21T10:44:05.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.614+0000 7ff73c3fd740 1 --2- 172.21.15.31:0/1128741476 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55afe84c6bc0 0x55afe84c8fb0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.614+0000 7ff73c3fd740 1 --2- 172.21.15.31:0/1128741476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afe84b6de0 0x55afe84b3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.614+0000 7ff73c3fd740 1 --2- 172.21.15.31:0/1128741476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84ae260 0x55afe84b3a50 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.614+0000 7ff73c3fd740 1 --2- 172.21.15.31:0/1128741476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55afe84ad8b0 0x55afe84b66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.614+0000 7ff73c3fd740 1 -- 172.21.15.31:0/1128741476 >> 172.21.15.31:0/1128741476 conn(0x55afe84a7f80 msgr2=0x55afe84a9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:05.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.614+0000 7ff71cc00640 0 client.26993 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:44:05.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.614+0000 7ff73c3fd740 1 -- 172.21.15.31:0/1128741476 shutdown_connections 2024-03-21T10:44:05.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.614+0000 7ff73c3fd740 1 -- 172.21.15.31:0/1128741476 wait complete. 2024-03-21T10:44:05.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.614+0000 7ff73c3fd740 20 client.26993 trim_cache size 0 max 16384 2024-03-21T10:44:05.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.642+0000 7ff73c3fd740 1 Processor -- start 2024-03-21T10:44:05.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.642+0000 7ff73c3fd740 1 -- start start 2024-03-21T10:44:05.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.642+0000 7ff73c3fd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afe84ad8b0 0x55afe84a6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:05.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.642+0000 7ff73c3fd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55afe84ae260 0x55afe84b6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:05.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.642+0000 7ff73c3fd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84b6de0 0x55afe84b66c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:05.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.642+0000 7ff73c3fd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7341c81b0 con 0x55afe84ad8b0 2024-03-21T10:44:05.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.642+0000 7ff73c3fd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7341c5cb0 con 0x55afe84b6de0 2024-03-21T10:44:05.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.642+0000 7ff73c3fd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff734071e90 con 0x55afe84ae260 2024-03-21T10:44:05.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.643+0000 7ff732000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55afe84ae260 0x55afe84b6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:05.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.643+0000 7ff732000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55afe84ae260 0x55afe84b6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:48146/0 (socket says 172.21.15.31:48146) 2024-03-21T10:44:05.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.643+0000 7ff732000640 1 -- 172.21.15.31:0/1867990282 learned_addr learned my addr 172.21.15.31:0/1867990282 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:05.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.643+0000 7ff732a00640 1 --2- 172.21.15.31:0/1867990282 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afe84ad8b0 0x55afe84a6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:05.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.643+0000 7ff733400640 1 --2- 172.21.15.31:0/1867990282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84b6de0 0x55afe84b66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:05.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.643+0000 7ff732000640 1 -- 172.21.15.31:0/1867990282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84b6de0 msgr2=0x55afe84b66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:05.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.643+0000 7ff732000640 1 --2- 172.21.15.31:0/1867990282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84b6de0 0x55afe84b66c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.643+0000 7ff732000640 1 -- 172.21.15.31:0/1867990282 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afe84ad8b0 msgr2=0x55afe84a6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:05.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.644+0000 7ff732000640 1 --2- 172.21.15.31:0/1867990282 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afe84ad8b0 0x55afe84a6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.644+0000 7ff732000640 1 -- 172.21.15.31:0/1867990282 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55afe84b4320 con 0x55afe84ae260 2024-03-21T10:44:05.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.644+0000 7ff733400640 1 --2- 172.21.15.31:0/1867990282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84b6de0 0x55afe84b66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:44:05.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.644+0000 7ff732a00640 1 --2- 172.21.15.31:0/1867990282 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afe84ad8b0 0x55afe84a6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:44:05.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.644+0000 7ff732000640 1 --2- 172.21.15.31:0/1867990282 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55afe84ae260 0x55afe84b6180 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff720001eb0 tx=0x7ff720005690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52399dc240637caa server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:05.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.645+0000 7ff71f400640 1 -- 172.21.15.31:0/1867990282 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff720016070 con 0x55afe84ae260 2024-03-21T10:44:05.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.645+0000 7ff71f400640 1 -- 172.21.15.31:0/1867990282 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff72001ac10 con 0x55afe84ae260 2024-03-21T10:44:05.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.645+0000 7ff71f400640 1 -- 172.21.15.31:0/1867990282 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff720010a20 con 0x55afe84ae260 2024-03-21T10:44:05.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.646+0000 7ff73c3fd740 1 -- 172.21.15.31:0/1867990282 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55afe84ae260 msgr2=0x55afe84b6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:05.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.646+0000 7ff73c3fd740 1 --2- 172.21.15.31:0/1867990282 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55afe84ae260 0x55afe84b6180 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff720001eb0 tx=0x7ff720005690 comp rx=0 tx=0).stop 2024-03-21T10:44:05.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.646+0000 7ff73c3fd740 1 -- 172.21.15.31:0/1867990282 shutdown_connections 2024-03-21T10:44:05.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.646+0000 7ff73c3fd740 1 --2- 172.21.15.31:0/1867990282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84b6de0 0x55afe84b66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.646+0000 7ff73c3fd740 1 --2- 172.21.15.31:0/1867990282 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55afe84ae260 0x55afe84b6180 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.646+0000 7ff73c3fd740 1 --2- 172.21.15.31:0/1867990282 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afe84ad8b0 0x55afe84a6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.646+0000 7ff73c3fd740 1 -- 172.21.15.31:0/1867990282 >> 172.21.15.31:0/1867990282 conn(0x55afe84a7f80 msgr2=0x55afe84a68d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:05.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.647+0000 7ff73c3fd740 1 -- 172.21.15.31:0/1867990282 shutdown_connections 2024-03-21T10:44:05.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.647+0000 7ff73c3fd740 1 -- 172.21.15.31:0/1867990282 wait complete. 2024-03-21T10:44:05.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.647+0000 7ff73c3fd740 1 -- start start 2024-03-21T10:44:05.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.647+0000 7ff73c3fd740 1 Processor -- start 2024-03-21T10:44:05.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.648+0000 7ff73c3fd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84ad8b0 0x55afe84ba8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:05.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.648+0000 7ff73c3fd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afe84ae260 0x55afe84bae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:05.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.648+0000 7ff73c3fd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55afe84b6de0 0x55afe85213d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:05.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.648+0000 7ff73c3fd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff734072490 con 0x55afe84ae260 2024-03-21T10:44:05.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.648+0000 7ff73c3fd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7341c57a0 con 0x55afe84ad8b0 2024-03-21T10:44:05.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.648+0000 7ff73c3fd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff734072ab0 con 0x55afe84b6de0 2024-03-21T10:44:05.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.648+0000 7ff73c3fd740 10 client.0 _ll_register_callbacks cb 0x7ff7200092a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:05.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.648+0000 7ff732a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84ad8b0 0x55afe84ba8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:05.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.648+0000 7ff732a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84ad8b0 0x55afe84ba8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51594/0 (socket says 172.21.15.31:51594) 2024-03-21T10:44:05.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.649+0000 7ff732a00640 1 -- 172.21.15.31:0/2286923611 learned_addr learned my addr 172.21.15.31:0/2286923611 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:05.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.649+0000 7ff733400640 1 --2- 172.21.15.31:0/2286923611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55afe84b6de0 0x55afe85213d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:05.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.649+0000 7ff732000640 1 --2- 172.21.15.31:0/2286923611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afe84ae260 0x55afe84bae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:05.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.649+0000 7ff732a00640 1 -- 172.21.15.31:0/2286923611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55afe84b6de0 msgr2=0x55afe85213d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:05.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.649+0000 7ff732a00640 1 --2- 172.21.15.31:0/2286923611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55afe84b6de0 0x55afe85213d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.649+0000 7ff732a00640 1 -- 172.21.15.31:0/2286923611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afe84ae260 msgr2=0x55afe84bae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:05.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.649+0000 7ff732a00640 1 --2- 172.21.15.31:0/2286923611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afe84ae260 0x55afe84bae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.649+0000 7ff732a00640 1 -- 172.21.15.31:0/2286923611 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55afe8521910 con 0x55afe84ad8b0 2024-03-21T10:44:05.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.649+0000 7ff732000640 1 --2- 172.21.15.31:0/2286923611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afe84ae260 0x55afe84bae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:44:05.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.650+0000 7ff732a00640 1 --2- 172.21.15.31:0/2286923611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84ad8b0 0x55afe84ba8c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff72409b3b0 tx=0x7ff7240047d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ff057f6387f005b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:05.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.650+0000 7ff71cc00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:44:05.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.650+0000 7ff71cc00640 1 -- 172.21.15.31:0/2286923611 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72409f570 con 0x55afe84ad8b0 2024-03-21T10:44:05.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.650+0000 7ff71cc00640 1 -- 172.21.15.31:0/2286923611 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7ff7240105a0 con 0x55afe84ad8b0 2024-03-21T10:44:05.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.650+0000 7ff73c3fd740 1 -- 172.21.15.31:0/2286923611 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff72001ac10 con 0x55afe84ad8b0 2024-03-21T10:44:05.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.650+0000 7ff71cc00640 1 -- 172.21.15.31:0/2286923611 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff7240a3e30 con 0x55afe84ad8b0 2024-03-21T10:44:05.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.650+0000 7ff73c3fd740 10 client.26999 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:05.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.651+0000 7ff73c3fd740 1 -- 172.21.15.31:0/2286923611 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55afe8522a90 con 0x55afe84ad8b0 2024-03-21T10:44:05.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.651+0000 7ff71cc00640 1 -- 172.21.15.31:0/2286923611 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff72001ac10 con 0x55afe84ad8b0 2024-03-21T10:44:05.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.651+0000 7ff71cc00640 1 -- 172.21.15.31:0/2286923611 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7ff72409f120 con 0x55afe84ad8b0 2024-03-21T10:44:05.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.651+0000 7ff73c3fd740 10 client.26999 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:05.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.651+0000 7ff73c3fd740 10 client.26999 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:44:05.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.651+0000 7ff73c3fd740 20 client.26999 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:05.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.651+0000 7ff73c3fd740 1 --2- 172.21.15.31:0/2286923611 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55afe8523190 0x55afe85255b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:05.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.651+0000 7ff73c3fd740 4 client.26999 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:44:05.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.651+0000 7ff73c3fd740 1 -- 172.21.15.31:0/2286923611 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55afe8418b50 con 0x55afe8523190 2024-03-21T10:44:05.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.651+0000 7ff732000640 1 --2- 172.21.15.31:0/2286923611 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55afe8523190 0x55afe85255b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:05.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.653+0000 7ff732000640 1 --2- 172.21.15.31:0/2286923611 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55afe8523190 0x55afe85255b0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=de6f7545d53e2ce3 server_cookie=abd3528d4e56c132 in_seq=0 out_seq=0 2024-03-21T10:44:05.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.653+0000 7ff71cc00640 10 client.26999 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:44:05.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.653+0000 7ff71cc00640 1 -- 172.21.15.31:0/2286923611 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55afe8418b50 con 0x55afe8523190 2024-03-21T10:44:05.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.653+0000 7ff71cc00640 10 client.26999 handle_command_reply: tid=0 2024-03-21T10:44:05.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.653+0000 7ff73c3fd740 1 client.26999 shutdown 2024-03-21T10:44:05.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.654+0000 7ff73c3fd740 1 -- 172.21.15.31:0/2286923611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84ad8b0 msgr2=0x55afe84ba8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:05.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.654+0000 7ff73c3fd740 1 --2- 172.21.15.31:0/2286923611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84ad8b0 0x55afe84ba8c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff72409b3b0 tx=0x7ff7240047d0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.654+0000 7ff73c3fd740 1 -- 172.21.15.31:0/2286923611 shutdown_connections 2024-03-21T10:44:05.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.654+0000 7ff73c3fd740 1 --2- 172.21.15.31:0/2286923611 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55afe8523190 0x55afe85255b0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.654+0000 7ff73c3fd740 1 --2- 172.21.15.31:0/2286923611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55afe84b6de0 0x55afe85213d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.654+0000 7ff73c3fd740 1 --2- 172.21.15.31:0/2286923611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afe84ae260 0x55afe84bae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.654+0000 7ff73c3fd740 1 --2- 172.21.15.31:0/2286923611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55afe84ad8b0 0x55afe84ba8c0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.654+0000 7ff73c3fd740 1 -- 172.21.15.31:0/2286923611 >> 172.21.15.31:0/2286923611 conn(0x55afe84a7f80 msgr2=0x55afe84a9af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:05.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.654+0000 7ff71cc00640 0 client.26999 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:44:05.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.655+0000 7ff73c3fd740 1 -- 172.21.15.31:0/2286923611 shutdown_connections 2024-03-21T10:44:05.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.655+0000 7ff73c3fd740 1 -- 172.21.15.31:0/2286923611 wait complete. 2024-03-21T10:44:05.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.655+0000 7ff73c3fd740 20 client.26999 trim_cache size 0 max 16384 2024-03-21T10:44:05.653 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:44:05.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.656+0000 7ff733e00640 1 -- 172.21.15.31:0/3847765622 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff7180a3190 msgr2=0x7ff7180a5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:05.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.656+0000 7ff733e00640 1 --2- 172.21.15.31:0/3847765622 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff7180a3190 0x7ff7180a5650 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff72c009620 tx=0x7ff72c004040 comp rx=0 tx=0).stop 2024-03-21T10:44:05.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.656+0000 7ff733e00640 1 -- 172.21.15.31:0/3847765622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7341014a0 msgr2=0x7ff73410cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:05.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.656+0000 7ff733e00640 1 --2- 172.21.15.31:0/3847765622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7341014a0 0x7ff73410cca0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff72400b3f0 tx=0x7ff72400b8b0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.656+0000 7ff733e00640 1 -- 172.21.15.31:0/3847765622 shutdown_connections 2024-03-21T10:44:05.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.656+0000 7ff733e00640 1 --2- 172.21.15.31:0/3847765622 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff7180a3190 0x7ff7180a5650 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.656+0000 7ff733e00640 1 --2- 172.21.15.31:0/3847765622 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff734107560 0x7ff7341a02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.656+0000 7ff733e00640 1 --2- 172.21.15.31:0/3847765622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff734101e60 0x7ff73410d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.656+0000 7ff733e00640 1 --2- 172.21.15.31:0/3847765622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7341014a0 0x7ff73410cca0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:05.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.656+0000 7ff733e00640 1 -- 172.21.15.31:0/3847765622 >> 172.21.15.31:0/3847765622 conn(0x7ff7340fcfb0 msgr2=0x7ff7340fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:05.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.657+0000 7ff733e00640 1 -- 172.21.15.31:0/3847765622 shutdown_connections 2024-03-21T10:44:05.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:05.657+0000 7ff733e00640 1 -- 172.21.15.31:0/3847765622 wait complete. 2024-03-21T10:44:05.683 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:44:05.683 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-21T10:44:06.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.045+0000 7f04e1c00640 1 -- 172.21.15.31:0/1793271543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04dc075bd0 msgr2=0x7f04dc075fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:06.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.045+0000 7f04e1c00640 1 --2- 172.21.15.31:0/1793271543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04dc075bd0 0x7f04dc075fb0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f04cc009930 tx=0x7f04cc02bc20 comp rx=0 tx=0).stop 2024-03-21T10:44:06.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.046+0000 7f04e1c00640 1 -- 172.21.15.31:0/1793271543 shutdown_connections 2024-03-21T10:44:06.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.046+0000 7f04e1c00640 1 --2- 172.21.15.31:0/1793271543 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04dc1124d0 0x7f04dc1148c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.046+0000 7f04e1c00640 1 --2- 172.21.15.31:0/1793271543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04dc0764f0 0x7f04dc111f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.046+0000 7f04e1c00640 1 --2- 172.21.15.31:0/1793271543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04dc075bd0 0x7f04dc075fb0 secure :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f04cc009930 tx=0x7f04cc02bc20 comp rx=0 tx=0).stop 2024-03-21T10:44:06.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.046+0000 7f04e1c00640 1 -- 172.21.15.31:0/1793271543 >> 172.21.15.31:0/1793271543 conn(0x7f04dc0ff2c0 msgr2=0x7f04dc1016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:06.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.046+0000 7f04e1c00640 1 -- 172.21.15.31:0/1793271543 shutdown_connections 2024-03-21T10:44:06.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.046+0000 7f04e1c00640 1 -- 172.21.15.31:0/1793271543 wait complete. 2024-03-21T10:44:06.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.050+0000 7f04e1c00640 1 Processor -- start 2024-03-21T10:44:06.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.050+0000 7f04e1c00640 1 -- start start 2024-03-21T10:44:06.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.052+0000 7f04e1c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04dc0764f0 0x7f04dc1a4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.052+0000 7f04e1c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04dc1124d0 0x7f04dc1a49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.052+0000 7f04e1c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04dc19e670 0x7f04dc19eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.052+0000 7f04e1c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc1180e0 con 0x7f04dc1124d0 2024-03-21T10:44:06.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.052+0000 7f04e1c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc117f60 con 0x7f04dc0764f0 2024-03-21T10:44:06.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.052+0000 7f04e1c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc118260 con 0x7f04dc19e670 2024-03-21T10:44:06.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.053+0000 7f04daa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04dc0764f0 0x7f04dc1a4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.053+0000 7f04db400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04dc19e670 0x7f04dc19eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.053+0000 7f04db400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04dc19e670 0x7f04dc19eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:48168/0 (socket says 172.21.15.31:48168) 2024-03-21T10:44:06.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.053+0000 7f04db400640 1 -- 172.21.15.31:0/3363594300 learned_addr learned my addr 172.21.15.31:0/3363594300 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:06.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.053+0000 7f04daa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04dc0764f0 0x7f04dc1a4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51616/0 (socket says 172.21.15.31:51616) 2024-03-21T10:44:06.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.053+0000 7f04da000640 1 --2- 172.21.15.31:0/3363594300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04dc1124d0 0x7f04dc1a49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.054+0000 7f04db400640 1 -- 172.21.15.31:0/3363594300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04dc0764f0 msgr2=0x7f04dc1a4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:06.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.054+0000 7f04db400640 1 --2- 172.21.15.31:0/3363594300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04dc0764f0 0x7f04dc1a4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.054+0000 7f04db400640 1 -- 172.21.15.31:0/3363594300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04dc1124d0 msgr2=0x7f04dc1a49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:06.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.054+0000 7f04db400640 1 --2- 172.21.15.31:0/3363594300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04dc1124d0 0x7f04dc1a49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.054+0000 7f04db400640 1 -- 172.21.15.31:0/3363594300 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04dc19f190 con 0x7f04dc19e670 2024-03-21T10:44:06.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.055+0000 7f04db400640 1 --2- 172.21.15.31:0/3363594300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04dc19e670 0x7f04dc19eb20 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f04d400e9d0 tx=0x7f04d400edd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:06.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.055+0000 7f04d3400640 1 -- 172.21.15.31:0/3363594300 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d4004d20 con 0x7f04dc19e670 2024-03-21T10:44:06.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.055+0000 7f04e1c00640 1 -- 172.21.15.31:0/3363594300 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04dc19f480 con 0x7f04dc19e670 2024-03-21T10:44:06.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.055+0000 7f04d3400640 1 -- 172.21.15.31:0/3363594300 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04d4005020 con 0x7f04dc19e670 2024-03-21T10:44:06.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.055+0000 7f04d3400640 1 -- 172.21.15.31:0/3363594300 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d4011690 con 0x7f04dc19e670 2024-03-21T10:44:06.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.055+0000 7f04e1c00640 1 -- 172.21.15.31:0/3363594300 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04dc1ab680 con 0x7f04dc19e670 2024-03-21T10:44:06.057 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:05 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3553245904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:44:06.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.060+0000 7f04d3400640 1 -- 172.21.15.31:0/3363594300 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f04d4011830 con 0x7f04dc19e670 2024-03-21T10:44:06.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.063+0000 7f04d3400640 1 --2- 172.21.15.31:0/3363594300 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f04c00a3090 0x7f04c00a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.063+0000 7f04d3400640 1 -- 172.21.15.31:0/3363594300 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f04d400cb20 con 0x7f04dc19e670 2024-03-21T10:44:06.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.063+0000 7f04e49e3740 1 Processor -- start 2024-03-21T10:44:06.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.063+0000 7f04e49e3740 1 -- start start 2024-03-21T10:44:06.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.064+0000 7f04e49e3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a12f82d6d0 0x55a12f82dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.064+0000 7f04e49e3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12f82e080 0x55a12f836630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.064+0000 7f04e49e3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a12f836c00 0x55a12f839050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.065+0000 7f04e49e3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a12f7912c0 con 0x55a12f836c00 2024-03-21T10:44:06.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.065+0000 7f04e49e3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a12f739c90 con 0x55a12f82e080 2024-03-21T10:44:06.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.065+0000 7f04e49e3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a12f78d360 con 0x55a12f82d6d0 2024-03-21T10:44:06.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.065+0000 7f04daa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a12f82d6d0 0x55a12f82dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.065+0000 7f04daa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a12f82d6d0 0x55a12f82dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:48176/0 (socket says 172.21.15.31:48176) 2024-03-21T10:44:06.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.065+0000 7f04daa00640 1 -- 172.21.15.31:0/2210622965 learned_addr learned my addr 172.21.15.31:0/2210622965 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:06.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.065+0000 7f04daa00640 1 --2- 172.21.15.31:0/3363594300 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f04c00a3090 0x7f04c00a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.066+0000 7f04db400640 1 --2- 172.21.15.31:0/2210622965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a12f836c00 0x55a12f839050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.066+0000 7f04db400640 1 -- 172.21.15.31:0/2210622965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a12f82d6d0 msgr2=0x55a12f82dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:06.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.066+0000 7f04db400640 1 --2- 172.21.15.31:0/2210622965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a12f82d6d0 0x55a12f82dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.066+0000 7f04db400640 1 -- 172.21.15.31:0/2210622965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12f82e080 msgr2=0x55a12f836630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:06.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.066+0000 7f04db400640 1 --2- 172.21.15.31:0/2210622965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12f82e080 0x55a12f836630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.066+0000 7f04db400640 1 -- 172.21.15.31:0/2210622965 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a12f75e080 con 0x55a12f836c00 2024-03-21T10:44:06.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.067+0000 7f04db400640 1 --2- 172.21.15.31:0/2210622965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a12f836c00 0x55a12f839050 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f04d40ce100 tx=0x7f04d4005300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5b889aff5cfa477 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:06.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.067+0000 7f04daa00640 1 --2- 172.21.15.31:0/3363594300 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f04c00a3090 0x7f04c00a5550 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f04cc02c810 tx=0x7f04cc005fd0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:06.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.067+0000 7f04c7e00640 1 -- 172.21.15.31:0/2210622965 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d40a0070 con 0x55a12f836c00 2024-03-21T10:44:06.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.068+0000 7f04c7e00640 1 -- 172.21.15.31:0/2210622965 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04d40a0ef0 con 0x55a12f836c00 2024-03-21T10:44:06.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.069+0000 7f04e49e3740 1 -- 172.21.15.31:0/2210622965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a12f836c00 msgr2=0x55a12f839050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:06.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.069+0000 7f04c7e00640 1 -- 172.21.15.31:0/2210622965 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d401f030 con 0x55a12f836c00 2024-03-21T10:44:06.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.069+0000 7f04e49e3740 1 --2- 172.21.15.31:0/2210622965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a12f836c00 0x55a12f839050 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f04d40ce100 tx=0x7f04d4005300 comp rx=0 tx=0).stop 2024-03-21T10:44:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.071+0000 7f04e49e3740 1 -- 172.21.15.31:0/2210622965 shutdown_connections 2024-03-21T10:44:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.071+0000 7f04e49e3740 1 --2- 172.21.15.31:0/2210622965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a12f836c00 0x55a12f839050 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.071+0000 7f04e49e3740 1 --2- 172.21.15.31:0/2210622965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12f82e080 0x55a12f836630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.071+0000 7f04e49e3740 1 --2- 172.21.15.31:0/2210622965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a12f82d6d0 0x55a12f82dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.071+0000 7f04e49e3740 1 -- 172.21.15.31:0/2210622965 >> 172.21.15.31:0/2210622965 conn(0x55a12f827da0 msgr2=0x55a12f828180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.072+0000 7f04e49e3740 1 -- 172.21.15.31:0/2210622965 shutdown_connections 2024-03-21T10:44:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.072+0000 7f04e49e3740 1 -- 172.21.15.31:0/2210622965 wait complete. 2024-03-21T10:44:06.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.072+0000 7f04e49e3740 1 -- start start 2024-03-21T10:44:06.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.073+0000 7f04e49e3740 1 Processor -- start 2024-03-21T10:44:06.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.074+0000 7f04e49e3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a12f82d6d0 0x55a12f836500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.074+0000 7f04e49e3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12f82e080 0x55a12f833890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.074+0000 7f04e49e3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a12f836c00 0x55a12f833e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.074+0000 7f04e49e3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a12f84abd0 con 0x55a12f836c00 2024-03-21T10:44:06.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.074+0000 7f04e49e3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a12f74d0d0 con 0x55a12f82e080 2024-03-21T10:44:06.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.074+0000 7f04e49e3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a12f84ad50 con 0x55a12f82d6d0 2024-03-21T10:44:06.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.074+0000 7f04e49e3740 10 client.0 _ll_register_callbacks cb 0x55a12f751e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:06.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.074+0000 7f04da000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12f82e080 0x55a12f833890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.074+0000 7f04da000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12f82e080 0x55a12f833890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51632/0 (socket says 172.21.15.31:51632) 2024-03-21T10:44:06.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.074+0000 7f04da000640 1 -- 172.21.15.31:0/305136276 learned_addr learned my addr 172.21.15.31:0/305136276 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:06.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.074+0000 7f04db400640 1 --2- 172.21.15.31:0/305136276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a12f836c00 0x55a12f833e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.075+0000 7f04da000640 1 -- 172.21.15.31:0/305136276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a12f82d6d0 msgr2=0x55a12f836500 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:44:06.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.075+0000 7f04da000640 1 --2- 172.21.15.31:0/305136276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a12f82d6d0 0x55a12f836500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.075+0000 7f04da000640 1 -- 172.21.15.31:0/305136276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a12f836c00 msgr2=0x55a12f833e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:06.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.075+0000 7f04da000640 1 --2- 172.21.15.31:0/305136276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a12f836c00 0x55a12f833e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.075+0000 7f04da000640 1 -- 172.21.15.31:0/305136276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a12f4a53f0 con 0x55a12f82e080 2024-03-21T10:44:06.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.075+0000 7f04da000640 1 --2- 172.21.15.31:0/305136276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12f82e080 0x55a12f833890 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f04c8009f90 tx=0x7f04c800e660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fb6ff944fb04734 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:06.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.075+0000 7f04c4c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:44:06.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.076+0000 7f04c4c00640 1 -- 172.21.15.31:0/305136276 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c8017070 con 0x55a12f82e080 2024-03-21T10:44:06.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.076+0000 7f04e49e3740 1 -- 172.21.15.31:0/305136276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f04d40a0ef0 con 0x55a12f82e080 2024-03-21T10:44:06.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.077+0000 7f04c4c00640 1 -- 172.21.15.31:0/305136276 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f04c8013470 con 0x55a12f82e080 2024-03-21T10:44:06.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.077+0000 7f04c4c00640 1 -- 172.21.15.31:0/305136276 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f04c8021050 con 0x55a12f82e080 2024-03-21T10:44:06.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.077+0000 7f04e49e3740 10 client.27008 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:06.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.077+0000 7f04e49e3740 1 -- 172.21.15.31:0/305136276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55a12f56f7d0 con 0x55a12f82e080 2024-03-21T10:44:06.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.078+0000 7f04c4c00640 1 -- 172.21.15.31:0/305136276 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04d40a0ef0 con 0x55a12f82e080 2024-03-21T10:44:06.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.078+0000 7f04c4c00640 1 -- 172.21.15.31:0/305136276 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f04c801c8b0 con 0x55a12f82e080 2024-03-21T10:44:06.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.078+0000 7f04e49e3740 10 client.27008 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:06.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.078+0000 7f04e49e3740 10 client.27008 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:44:06.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.078+0000 7f04e49e3740 20 client.27008 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:06.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.078+0000 7f04e49e3740 1 --2- 172.21.15.31:0/305136276 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55a12f868d50 0x55a12f86b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.078+0000 7f04e49e3740 4 client.27008 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:44:06.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.078+0000 7f04e49e3740 1 -- 172.21.15.31:0/305136276 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a12f79e800 con 0x55a12f868d50 2024-03-21T10:44:06.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.081+0000 7f04daa00640 1 --2- 172.21.15.31:0/305136276 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55a12f868d50 0x55a12f86b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.082+0000 7f04daa00640 1 --2- 172.21.15.31:0/305136276 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55a12f868d50 0x55a12f86b170 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=be0dfc38181ea1f9 server_cookie=9907a2f3a49a5f0c in_seq=0 out_seq=0 2024-03-21T10:44:06.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.082+0000 7f04c4c00640 10 client.27008 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:44:06.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.085+0000 7f04c4c00640 1 -- 172.21.15.31:0/305136276 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a12f79e800 con 0x55a12f868d50 2024-03-21T10:44:06.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.085+0000 7f04c4c00640 10 client.27008 handle_command_reply: tid=0 2024-03-21T10:44:06.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.085+0000 7f04e49e3740 1 client.27008 shutdown 2024-03-21T10:44:06.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.085+0000 7f04e49e3740 1 -- 172.21.15.31:0/305136276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12f82e080 msgr2=0x55a12f833890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:06.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.085+0000 7f04e49e3740 1 --2- 172.21.15.31:0/305136276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12f82e080 0x55a12f833890 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f04c8009f90 tx=0x7f04c800e660 comp rx=0 tx=0).stop 2024-03-21T10:44:06.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.085+0000 7f04e49e3740 1 -- 172.21.15.31:0/305136276 shutdown_connections 2024-03-21T10:44:06.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.085+0000 7f04e49e3740 1 --2- 172.21.15.31:0/305136276 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55a12f868d50 0x55a12f86b170 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.085+0000 7f04e49e3740 1 --2- 172.21.15.31:0/305136276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a12f836c00 0x55a12f833e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.085+0000 7f04e49e3740 1 --2- 172.21.15.31:0/305136276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12f82e080 0x55a12f833890 secure :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f04c8009f90 tx=0x7f04c800e660 comp rx=0 tx=0).stop 2024-03-21T10:44:06.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.085+0000 7f04e49e3740 1 --2- 172.21.15.31:0/305136276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a12f82d6d0 0x55a12f836500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.085+0000 7f04e49e3740 1 -- 172.21.15.31:0/305136276 >> 172.21.15.31:0/305136276 conn(0x55a12f827da0 msgr2=0x55a12f829af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:06.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.086+0000 7f04c4c00640 0 client.27008 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:44:06.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.086+0000 7f04e49e3740 1 -- 172.21.15.31:0/305136276 shutdown_connections 2024-03-21T10:44:06.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.086+0000 7f04e49e3740 1 -- 172.21.15.31:0/305136276 wait complete. 2024-03-21T10:44:06.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.086+0000 7f04e49e3740 20 client.27008 trim_cache size 0 max 16384 2024-03-21T10:44:06.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.136+0000 7f04e49e3740 1 Processor -- start 2024-03-21T10:44:06.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.136+0000 7f04e49e3740 1 -- start start 2024-03-21T10:44:06.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.136+0000 7f04e49e3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a12f82d6d0 0x55a12f822d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.136+0000 7f04e49e3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12f836c00 0x55a12f835fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.136+0000 7f04e49e3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a12f870a60 0x55a12f836500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.136+0000 7f04e49e3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc108b10 con 0x55a12f82d6d0 2024-03-21T10:44:06.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.136+0000 7f04e49e3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc106610 con 0x55a12f836c00 2024-03-21T10:44:06.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.136+0000 7f04e49e3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc0735b0 con 0x55a12f870a60 2024-03-21T10:44:06.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.137+0000 7f04db400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a12f870a60 0x55a12f836500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.138+0000 7f04db400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a12f870a60 0x55a12f836500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:48198/0 (socket says 172.21.15.31:48198) 2024-03-21T10:44:06.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.139+0000 7f04da000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12f836c00 0x55a12f835fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.139+0000 7f04da000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12f836c00 0x55a12f835fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51638/0 (socket says 172.21.15.31:51638) 2024-03-21T10:44:06.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.139+0000 7f04daa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a12f82d6d0 0x55a12f822d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.139+0000 7f04da000640 1 -- 172.21.15.31:0/3498598629 learned_addr learned my addr 172.21.15.31:0/3498598629 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:06.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.140+0000 7f04da000640 1 -- 172.21.15.31:0/3498598629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a12f870a60 msgr2=0x55a12f836500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:06.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.142+0000 7f04da000640 1 --2- 172.21.15.31:0/3498598629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a12f870a60 0x55a12f836500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.142+0000 7f04da000640 1 -- 172.21.15.31:0/3498598629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a12f82d6d0 msgr2=0x55a12f822d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:06.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.142+0000 7f04da000640 1 --2- 172.21.15.31:0/3498598629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a12f82d6d0 0x55a12f822d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.142+0000 7f04da000640 1 -- 172.21.15.31:0/3498598629 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a12f831e00 con 0x55a12f836c00 2024-03-21T10:44:06.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.142+0000 7f04db400640 1 --2- 172.21.15.31:0/3498598629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a12f870a60 0x55a12f836500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:44:06.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.143+0000 7f04da000640 1 --2- 172.21.15.31:0/3498598629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12f836c00 0x55a12f835fc0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f04c8010a90 tx=0x7f04c8011620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2519fea906d4c135 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:06.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.145+0000 7f04c7400640 1 -- 172.21.15.31:0/3498598629 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c802b070 con 0x55a12f836c00 2024-03-21T10:44:06.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.145+0000 7f04c7400640 1 -- 172.21.15.31:0/3498598629 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04c8025390 con 0x55a12f836c00 2024-03-21T10:44:06.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.145+0000 7f04c7400640 1 -- 172.21.15.31:0/3498598629 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c8021050 con 0x55a12f836c00 2024-03-21T10:44:06.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.146+0000 7f04e49e3740 1 -- 172.21.15.31:0/3498598629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12f836c00 msgr2=0x55a12f835fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:06.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.146+0000 7f04e49e3740 1 --2- 172.21.15.31:0/3498598629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12f836c00 0x55a12f835fc0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f04c8010a90 tx=0x7f04c8011620 comp rx=0 tx=0).stop 2024-03-21T10:44:06.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.146+0000 7f04e49e3740 1 -- 172.21.15.31:0/3498598629 shutdown_connections 2024-03-21T10:44:06.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.146+0000 7f04e49e3740 1 --2- 172.21.15.31:0/3498598629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a12f870a60 0x55a12f836500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.146+0000 7f04e49e3740 1 --2- 172.21.15.31:0/3498598629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12f836c00 0x55a12f835fc0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.146+0000 7f04e49e3740 1 --2- 172.21.15.31:0/3498598629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a12f82d6d0 0x55a12f822d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.146+0000 7f04e49e3740 1 -- 172.21.15.31:0/3498598629 >> 172.21.15.31:0/3498598629 conn(0x55a12f827da0 msgr2=0x55a12f833680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:06.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.146+0000 7f04e49e3740 1 -- 172.21.15.31:0/3498598629 shutdown_connections 2024-03-21T10:44:06.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.148+0000 7f04e49e3740 1 -- 172.21.15.31:0/3498598629 wait complete. 2024-03-21T10:44:06.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.148+0000 7f04e49e3740 1 -- start start 2024-03-21T10:44:06.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.149+0000 7f04e49e3740 1 Processor -- start 2024-03-21T10:44:06.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.150+0000 7f04e49e3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a12f82d6d0 0x55a12f8301e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.150+0000 7f04e49e3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12f836c00 0x55a12f830720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.150+0000 7f04e49e3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a12f870a60 0x55a12f830c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.150+0000 7f04e49e3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc104c90 con 0x55a12f870a60 2024-03-21T10:44:06.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.150+0000 7f04e49e3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc105fb0 con 0x55a12f836c00 2024-03-21T10:44:06.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.150+0000 7f04e49e3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc1052f0 con 0x55a12f82d6d0 2024-03-21T10:44:06.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.150+0000 7f04e49e3740 10 client.0 _ll_register_callbacks cb 0x7f04c8006bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:06.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.151+0000 7f04db400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a12f870a60 0x55a12f830c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.151+0000 7f04db400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a12f870a60 0x55a12f830c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52904/0 (socket says 172.21.15.31:52904) 2024-03-21T10:44:06.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.151+0000 7f04db400640 1 -- 172.21.15.31:0/2736377851 learned_addr learned my addr 172.21.15.31:0/2736377851 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:06.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.152+0000 7f04da000640 1 --2- 172.21.15.31:0/2736377851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12f836c00 0x55a12f830720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.153+0000 7f04daa00640 1 --2- 172.21.15.31:0/2736377851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a12f82d6d0 0x55a12f8301e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.153+0000 7f04da000640 1 -- 172.21.15.31:0/2736377851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a12f82d6d0 msgr2=0x55a12f8301e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:06.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.154+0000 7f04da000640 1 --2- 172.21.15.31:0/2736377851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a12f82d6d0 0x55a12f8301e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.155+0000 7f04da000640 1 -- 172.21.15.31:0/2736377851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a12f870a60 msgr2=0x55a12f830c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:06.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.155+0000 7f04da000640 1 --2- 172.21.15.31:0/2736377851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a12f870a60 0x55a12f830c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.155+0000 7f04da000640 1 -- 172.21.15.31:0/2736377851 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a12f8c38c0 con 0x55a12f836c00 2024-03-21T10:44:06.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.155+0000 7f04daa00640 1 --2- 172.21.15.31:0/2736377851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a12f82d6d0 0x55a12f8301e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:44:06.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.155+0000 7f04db400640 1 --2- 172.21.15.31:0/2736377851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a12f870a60 0x55a12f830c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:44:06.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.157+0000 7f04da000640 1 --2- 172.21.15.31:0/2736377851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12f836c00 0x55a12f830720 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f04c8006cd0 tx=0x7f04c8009180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f747652003666b08 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:06.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.157+0000 7f04c4c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:44:06.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.159+0000 7f04c4c00640 1 -- 172.21.15.31:0/2736377851 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c8021070 con 0x55a12f836c00 2024-03-21T10:44:06.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.159+0000 7f04c4c00640 1 -- 172.21.15.31:0/2736377851 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f04c8011350 con 0x55a12f836c00 2024-03-21T10:44:06.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.160+0000 7f04e49e3740 1 -- 172.21.15.31:0/2736377851 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f04c8025390 con 0x55a12f836c00 2024-03-21T10:44:06.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.161+0000 7f04c4c00640 1 -- 172.21.15.31:0/2736377851 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f04c802b040 con 0x55a12f836c00 2024-03-21T10:44:06.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.162+0000 7f04e49e3740 10 client.27014 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:06.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.162+0000 7f04e49e3740 1 -- 172.21.15.31:0/2736377851 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55a12f8c4b10 con 0x55a12f836c00 2024-03-21T10:44:06.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.164+0000 7f04c4c00640 1 -- 172.21.15.31:0/2736377851 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04c8025390 con 0x55a12f836c00 2024-03-21T10:44:06.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.165+0000 7f04c4c00640 1 -- 172.21.15.31:0/2736377851 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f04c801c8b0 con 0x55a12f836c00 2024-03-21T10:44:06.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.165+0000 7f04e49e3740 10 client.27014 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:06.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.165+0000 7f04e49e3740 10 client.27014 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:44:06.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.165+0000 7f04e49e3740 20 client.27014 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:06.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.166+0000 7f04e49e3740 1 --2- 172.21.15.31:0/2736377851 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55a12f8c50b0 0x55a12f8c74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.166+0000 7f04e49e3740 4 client.27014 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:44:06.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.166+0000 7f04e49e3740 1 -- 172.21.15.31:0/2736377851 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a12f798b50 con 0x55a12f8c50b0 2024-03-21T10:44:06.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.167+0000 7f04daa00640 1 --2- 172.21.15.31:0/2736377851 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55a12f8c50b0 0x55a12f8c74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.171+0000 7f04c4c00640 10 client.27014 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:44:06.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.171+0000 7f04daa00640 1 --2- 172.21.15.31:0/2736377851 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55a12f8c50b0 0x55a12f8c74d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4e2e2b18396713ba server_cookie=5b853f93a40e8142 in_seq=0 out_seq=0 2024-03-21T10:44:06.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.172+0000 7f04c4c00640 1 -- 172.21.15.31:0/2736377851 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a12f798b50 con 0x55a12f8c50b0 2024-03-21T10:44:06.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.172+0000 7f04c4c00640 10 client.27014 handle_command_reply: tid=0 2024-03-21T10:44:06.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.172+0000 7f04e49e3740 1 client.27014 shutdown 2024-03-21T10:44:06.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.178+0000 7f04e49e3740 1 -- 172.21.15.31:0/2736377851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12f836c00 msgr2=0x55a12f830720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:06.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.178+0000 7f04e49e3740 1 --2- 172.21.15.31:0/2736377851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12f836c00 0x55a12f830720 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f04c8006cd0 tx=0x7f04c8009180 comp rx=0 tx=0).stop 2024-03-21T10:44:06.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.179+0000 7f04e49e3740 1 -- 172.21.15.31:0/2736377851 shutdown_connections 2024-03-21T10:44:06.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.179+0000 7f04e49e3740 1 --2- 172.21.15.31:0/2736377851 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55a12f8c50b0 0x55a12f8c74d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.179+0000 7f04e49e3740 1 --2- 172.21.15.31:0/2736377851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a12f870a60 0x55a12f830c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.179+0000 7f04e49e3740 1 --2- 172.21.15.31:0/2736377851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12f836c00 0x55a12f830720 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.179+0000 7f04e49e3740 1 --2- 172.21.15.31:0/2736377851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a12f82d6d0 0x55a12f8301e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.179+0000 7f04e49e3740 1 -- 172.21.15.31:0/2736377851 >> 172.21.15.31:0/2736377851 conn(0x55a12f827da0 msgr2=0x55a12f8291f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:06.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.179+0000 7f04c4c00640 0 client.27014 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:44:06.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.180+0000 7f04e49e3740 1 -- 172.21.15.31:0/2736377851 shutdown_connections 2024-03-21T10:44:06.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.180+0000 7f04e49e3740 1 -- 172.21.15.31:0/2736377851 wait complete. 2024-03-21T10:44:06.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.181+0000 7f04e49e3740 20 client.27014 trim_cache size 0 max 16384 2024-03-21T10:44:06.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:05 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3553245904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:44:06.182 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:44:06.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.186+0000 7f04e1c00640 1 -- 172.21.15.31:0/3363594300 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f04c00a3090 msgr2=0x7f04c00a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:06.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.186+0000 7f04e1c00640 1 --2- 172.21.15.31:0/3363594300 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f04c00a3090 0x7f04c00a5550 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f04cc02c810 tx=0x7f04cc005fd0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.186+0000 7f04e1c00640 1 -- 172.21.15.31:0/3363594300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04dc19e670 msgr2=0x7f04dc19eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:06.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.186+0000 7f04e1c00640 1 --2- 172.21.15.31:0/3363594300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04dc19e670 0x7f04dc19eb20 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f04d400e9d0 tx=0x7f04d400edd0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.186+0000 7f04e1c00640 1 -- 172.21.15.31:0/3363594300 shutdown_connections 2024-03-21T10:44:06.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.186+0000 7f04e1c00640 1 --2- 172.21.15.31:0/3363594300 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f04c00a3090 0x7f04c00a5550 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.186+0000 7f04e1c00640 1 --2- 172.21.15.31:0/3363594300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04dc19e670 0x7f04dc19eb20 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.186+0000 7f04e1c00640 1 --2- 172.21.15.31:0/3363594300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04dc1124d0 0x7f04dc1a49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.186+0000 7f04e1c00640 1 --2- 172.21.15.31:0/3363594300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04dc0764f0 0x7f04dc1a4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.186+0000 7f04e1c00640 1 -- 172.21.15.31:0/3363594300 >> 172.21.15.31:0/3363594300 conn(0x7f04dc0ff2c0 msgr2=0x7f04dc113bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:06.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.187+0000 7f04e1c00640 1 -- 172.21.15.31:0/3363594300 shutdown_connections 2024-03-21T10:44:06.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.187+0000 7f04e1c00640 1 -- 172.21.15.31:0/3363594300 wait complete. 2024-03-21T10:44:06.222 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:44:06.222 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-21T10:44:06.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:05 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3553245904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:44:06.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.581+0000 7fd084a00640 1 -- 172.21.15.31:0/1981562325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd08010e310 msgr2=0x7fd08011ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:06.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.581+0000 7fd084a00640 1 --2- 172.21.15.31:0/1981562325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd08010e310 0x7fd08011ce90 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd078009cd0 tx=0x7fd07802c0e0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.583+0000 7fd084a00640 1 -- 172.21.15.31:0/1981562325 shutdown_connections 2024-03-21T10:44:06.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.583+0000 7fd084a00640 1 --2- 172.21.15.31:0/1981562325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd08010e310 0x7fd08011ce90 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.583+0000 7fd084a00640 1 --2- 172.21.15.31:0/1981562325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd08010d780 0x7fd08010dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.583+0000 7fd084a00640 1 --2- 172.21.15.31:0/1981562325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd08010fc90 0x7fd080110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.583+0000 7fd084a00640 1 -- 172.21.15.31:0/1981562325 >> 172.21.15.31:0/1981562325 conn(0x7fd08006c580 msgr2=0x7fd08006c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:06.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.584+0000 7fd084a00640 1 -- 172.21.15.31:0/1981562325 shutdown_connections 2024-03-21T10:44:06.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.584+0000 7fd084a00640 1 -- 172.21.15.31:0/1981562325 wait complete. 2024-03-21T10:44:06.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.585+0000 7fd084a00640 1 Processor -- start 2024-03-21T10:44:06.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.585+0000 7fd084a00640 1 -- start start 2024-03-21T10:44:06.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.586+0000 7fd084a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd08010d780 0x7fd0801ac760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.586+0000 7fd084a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd08010e310 0x7fd0801acca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.586+0000 7fd084a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd08010fc90 0x7fd0801a6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.586+0000 7fd084a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd080120670 con 0x7fd08010fc90 2024-03-21T10:44:06.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.586+0000 7fd084a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0801204f0 con 0x7fd08010d780 2024-03-21T10:44:06.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.586+0000 7fd084a00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0801207f0 con 0x7fd08010e310 2024-03-21T10:44:06.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.586+0000 7fd07ea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd08010e310 0x7fd0801acca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.586+0000 7fd07ea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd08010e310 0x7fd0801acca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:48236/0 (socket says 172.21.15.31:48236) 2024-03-21T10:44:06.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.586+0000 7fd07ea00640 1 -- 172.21.15.31:0/3740759521 learned_addr learned my addr 172.21.15.31:0/3740759521 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:06.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.586+0000 7fd07f400640 1 --2- 172.21.15.31:0/3740759521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd08010d780 0x7fd0801ac760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.587+0000 7fd07ea00640 1 -- 172.21.15.31:0/3740759521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd08010d780 msgr2=0x7fd0801ac760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:06.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.587+0000 7fd07ea00640 1 --2- 172.21.15.31:0/3740759521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd08010d780 0x7fd0801ac760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.587+0000 7fd07ea00640 1 -- 172.21.15.31:0/3740759521 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd08010fc90 msgr2=0x7fd0801a6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:06.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.587+0000 7fd07ea00640 1 --2- 172.21.15.31:0/3740759521 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd08010fc90 0x7fd0801a6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.587+0000 7fd07ea00640 1 -- 172.21.15.31:0/3740759521 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0801a7090 con 0x7fd08010e310 2024-03-21T10:44:06.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.587+0000 7fd07ea00640 1 --2- 172.21.15.31:0/3740759521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd08010e310 0x7fd0801acca0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd07000b4a0 tx=0x7fd07000b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:06.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.587+0000 7fd077e00640 1 -- 172.21.15.31:0/3740759521 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070014070 con 0x7fd08010e310 2024-03-21T10:44:06.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.587+0000 7fd077e00640 1 -- 172.21.15.31:0/3740759521 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd070010af0 con 0x7fd08010e310 2024-03-21T10:44:06.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.587+0000 7fd077e00640 1 -- 172.21.15.31:0/3740759521 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070017d10 con 0x7fd08010e310 2024-03-21T10:44:06.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.588+0000 7fd084a00640 1 -- 172.21.15.31:0/3740759521 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0801a7380 con 0x7fd08010e310 2024-03-21T10:44:06.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.588+0000 7fd084a00640 1 -- 172.21.15.31:0/3740759521 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0801b3620 con 0x7fd08010e310 2024-03-21T10:44:06.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.594+0000 7fd077e00640 1 -- 172.21.15.31:0/3740759521 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fd070017070 con 0x7fd08010e310 2024-03-21T10:44:06.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.595+0000 7fd088de2740 1 Processor -- start 2024-03-21T10:44:06.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.599+0000 7fd077e00640 1 --2- 172.21.15.31:0/3740759521 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd0640a3110 0x7fd0640a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.600+0000 7fd088de2740 1 -- start start 2024-03-21T10:44:06.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.601+0000 7fd088de2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b341798b0 0x556b34179c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.601+0000 7fd088de2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b3417a260 0x556b34182810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.601+0000 7fd088de2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b34182de0 0x556b34185230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.601+0000 7fd077e00640 1 -- 172.21.15.31:0/3740759521 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fd0700cf130 con 0x7fd08010e310 2024-03-21T10:44:06.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.601+0000 7fd07ea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b3417a260 0x556b34182810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.601+0000 7fd07ea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b3417a260 0x556b34182810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:48250/0 (socket says 172.21.15.31:48250) 2024-03-21T10:44:06.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.601+0000 7fd07ea00640 1 -- 172.21.15.31:0/4212255996 learned_addr learned my addr 172.21.15.31:0/4212255996 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:06.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.601+0000 7fd088de2740 1 -- 172.21.15.31:0/4212255996 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b340dd2c0 con 0x556b34182de0 2024-03-21T10:44:06.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.601+0000 7fd088de2740 1 -- 172.21.15.31:0/4212255996 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b34085c90 con 0x556b341798b0 2024-03-21T10:44:06.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.601+0000 7fd07fe00640 1 --2- 172.21.15.31:0/4212255996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b34182de0 0x556b34185230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.601+0000 7fd088de2740 1 -- 172.21.15.31:0/4212255996 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b340d9360 con 0x556b3417a260 2024-03-21T10:44:06.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.602+0000 7fd07fe00640 1 --2- 172.21.15.31:0/3740759521 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd0640a3110 0x7fd0640a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.604+0000 7fd07ea00640 1 -- 172.21.15.31:0/4212255996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b341798b0 msgr2=0x556b34179c90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:44:06.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.604+0000 7fd07ea00640 1 --2- 172.21.15.31:0/4212255996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b341798b0 0x556b34179c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.604+0000 7fd07ea00640 1 -- 172.21.15.31:0/4212255996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b34182de0 msgr2=0x556b34185230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:06.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.604+0000 7fd07ea00640 1 --2- 172.21.15.31:0/4212255996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b34182de0 0x556b34185230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.604+0000 7fd07ea00640 1 -- 172.21.15.31:0/4212255996 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b340aa080 con 0x556b3417a260 2024-03-21T10:44:06.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.604+0000 7fd07fe00640 1 --2- 172.21.15.31:0/4212255996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b34182de0 0x556b34185230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:44:06.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.605+0000 7fd07ea00640 1 --2- 172.21.15.31:0/4212255996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b3417a260 0x556b34182810 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd0700a3770 tx=0x7fd0700046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53bbce7d4ef54544 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:06.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.605+0000 7fd07fe00640 1 --2- 172.21.15.31:0/3740759521 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd0640a3110 0x7fd0640a55d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd078037040 tx=0x7fd078005fb0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:06.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.605+0000 7fd06be00640 1 -- 172.21.15.31:0/4212255996 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07009b990 con 0x556b3417a260 2024-03-21T10:44:06.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.605+0000 7fd06be00640 1 -- 172.21.15.31:0/4212255996 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd07009b290 con 0x556b3417a260 2024-03-21T10:44:06.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.605+0000 7fd06be00640 1 -- 172.21.15.31:0/4212255996 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070097900 con 0x556b3417a260 2024-03-21T10:44:06.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.611+0000 7fd088de2740 1 -- 172.21.15.31:0/4212255996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b3417a260 msgr2=0x556b34182810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:06.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.611+0000 7fd088de2740 1 --2- 172.21.15.31:0/4212255996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b3417a260 0x556b34182810 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd0700a3770 tx=0x7fd0700046c0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.611+0000 7fd088de2740 1 -- 172.21.15.31:0/4212255996 shutdown_connections 2024-03-21T10:44:06.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.611+0000 7fd088de2740 1 --2- 172.21.15.31:0/4212255996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b34182de0 0x556b34185230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.611+0000 7fd088de2740 1 --2- 172.21.15.31:0/4212255996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b3417a260 0x556b34182810 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.611+0000 7fd088de2740 1 --2- 172.21.15.31:0/4212255996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b341798b0 0x556b34179c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.611+0000 7fd088de2740 1 -- 172.21.15.31:0/4212255996 >> 172.21.15.31:0/4212255996 conn(0x556b34173f80 msgr2=0x556b34174360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:06.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.612+0000 7fd088de2740 1 -- 172.21.15.31:0/4212255996 shutdown_connections 2024-03-21T10:44:06.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.613+0000 7fd088de2740 1 -- 172.21.15.31:0/4212255996 wait complete. 2024-03-21T10:44:06.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.613+0000 7fd088de2740 1 -- start start 2024-03-21T10:44:06.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.617+0000 7fd088de2740 1 Processor -- start 2024-03-21T10:44:06.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.623+0000 7fd088de2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b341798b0 0x556b3417fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.623+0000 7fd088de2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b34182de0 0x556b341800b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.623+0000 7fd088de2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b341805f0 0x556b3418f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.623+0000 7fd088de2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b34185cf0 con 0x556b341805f0 2024-03-21T10:44:06.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.623+0000 7fd088de2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b340990d0 con 0x556b34182de0 2024-03-21T10:44:06.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.623+0000 7fd088de2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b34185e70 con 0x556b341798b0 2024-03-21T10:44:06.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.624+0000 7fd088de2740 10 client.0 _ll_register_callbacks cb 0x556b3409de80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:06.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.624+0000 7fd07f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b341798b0 0x556b3417fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.624+0000 7fd07f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b341798b0 0x556b3417fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:48256/0 (socket says 172.21.15.31:48256) 2024-03-21T10:44:06.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.624+0000 7fd07f400640 1 -- 172.21.15.31:0/1484900677 learned_addr learned my addr 172.21.15.31:0/1484900677 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:06.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.625+0000 7fd07f400640 1 -- 172.21.15.31:0/1484900677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b34182de0 msgr2=0x556b341800b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:44:06.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.625+0000 7fd07fe00640 1 --2- 172.21.15.31:0/1484900677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b341805f0 0x556b3418f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.625+0000 7fd07f400640 1 --2- 172.21.15.31:0/1484900677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b34182de0 0x556b341800b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.625+0000 7fd07f400640 1 -- 172.21.15.31:0/1484900677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b341805f0 msgr2=0x556b3418f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:06.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.625+0000 7fd07f400640 1 --2- 172.21.15.31:0/1484900677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b341805f0 0x556b3418f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.625+0000 7fd07f400640 1 -- 172.21.15.31:0/1484900677 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556b33df13f0 con 0x556b341798b0 2024-03-21T10:44:06.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.628+0000 7fd07f400640 1 --2- 172.21.15.31:0/1484900677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b341798b0 0x556b3417fb70 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd06c01f430 tx=0x7fd06c01f8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee5add1f656ed03c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:06.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.628+0000 7fd068c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:44:06.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.629+0000 7fd068c00640 1 -- 172.21.15.31:0/1484900677 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06c021910 con 0x556b341798b0 2024-03-21T10:44:06.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.629+0000 7fd068c00640 1 -- 172.21.15.31:0/1484900677 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fd06c021710 con 0x556b341798b0 2024-03-21T10:44:06.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.633+0000 7fd088de2740 1 -- 172.21.15.31:0/1484900677 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd07009b290 con 0x556b341798b0 2024-03-21T10:44:06.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.634+0000 7fd068c00640 1 -- 172.21.15.31:0/1484900677 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd06c021910 con 0x556b341798b0 2024-03-21T10:44:06.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.634+0000 7fd088de2740 10 client.26935 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:06.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.634+0000 7fd088de2740 1 -- 172.21.15.31:0/1484900677 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x556b33ebb7d0 con 0x556b341798b0 2024-03-21T10:44:06.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.635+0000 7fd068c00640 1 -- 172.21.15.31:0/1484900677 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd06c020430 con 0x556b341798b0 2024-03-21T10:44:06.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.635+0000 7fd068c00640 1 -- 172.21.15.31:0/1484900677 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd06c035420 con 0x556b341798b0 2024-03-21T10:44:06.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.635+0000 7fd088de2740 10 client.26935 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:06.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.635+0000 7fd088de2740 10 client.26935 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:44:06.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.635+0000 7fd088de2740 20 client.26935 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:06.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.636+0000 7fd088de2740 1 --2- 172.21.15.31:0/1484900677 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556b341931b0 0x556b341955a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.636+0000 7fd088de2740 4 client.26935 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:44:06.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.636+0000 7fd088de2740 1 -- 172.21.15.31:0/1484900677 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd07009b290 con 0x556b341931b0 2024-03-21T10:44:06.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.636+0000 7fd07ea00640 1 --2- 172.21.15.31:0/1484900677 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556b341931b0 0x556b341955a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.638+0000 7fd07ea00640 1 --2- 172.21.15.31:0/1484900677 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556b341931b0 0x556b341955a0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1559368652b0edee server_cookie=42d0203f50aada57 in_seq=0 out_seq=0 2024-03-21T10:44:06.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.638+0000 7fd068c00640 10 client.26935 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:06.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.639+0000 7fd068c00640 1 -- 172.21.15.31:0/1484900677 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7fd07009b290 con 0x556b341931b0 2024-03-21T10:44:06.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.639+0000 7fd068c00640 10 client.26935 handle_command_reply: tid=0 2024-03-21T10:44:06.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.639+0000 7fd088de2740 1 client.26935 shutdown 2024-03-21T10:44:06.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.639+0000 7fd088de2740 1 -- 172.21.15.31:0/1484900677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b341798b0 msgr2=0x556b3417fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:06.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.640+0000 7fd088de2740 1 --2- 172.21.15.31:0/1484900677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b341798b0 0x556b3417fb70 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd06c01f430 tx=0x7fd06c01f8f0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.641+0000 7fd088de2740 1 -- 172.21.15.31:0/1484900677 shutdown_connections 2024-03-21T10:44:06.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.641+0000 7fd088de2740 1 --2- 172.21.15.31:0/1484900677 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556b341931b0 0x556b341955a0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.641+0000 7fd088de2740 1 --2- 172.21.15.31:0/1484900677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b341805f0 0x556b3418f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.641+0000 7fd088de2740 1 --2- 172.21.15.31:0/1484900677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b34182de0 0x556b341800b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.641+0000 7fd088de2740 1 --2- 172.21.15.31:0/1484900677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b341798b0 0x556b3417fb70 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.641+0000 7fd088de2740 1 -- 172.21.15.31:0/1484900677 >> 172.21.15.31:0/1484900677 conn(0x556b34173f80 msgr2=0x556b34175cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:06.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.641+0000 7fd068c00640 0 client.26935 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:06.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.641+0000 7fd088de2740 1 -- 172.21.15.31:0/1484900677 shutdown_connections 2024-03-21T10:44:06.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.642+0000 7fd088de2740 1 -- 172.21.15.31:0/1484900677 wait complete. 2024-03-21T10:44:06.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.642+0000 7fd088de2740 20 client.26935 trim_cache size 0 max 16384 2024-03-21T10:44:06.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.670+0000 7fd088de2740 1 Processor -- start 2024-03-21T10:44:06.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.670+0000 7fd088de2740 1 -- start start 2024-03-21T10:44:06.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.670+0000 7fd088de2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b341798b0 0x556b3417ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.671+0000 7fd088de2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b341805f0 0x556b3417d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.671+0000 7fd088de2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b34182de0 0x556b34199e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.671+0000 7fd088de2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0801b7320 con 0x556b341805f0 2024-03-21T10:44:06.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.671+0000 7fd088de2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0801b4e20 con 0x556b34182de0 2024-03-21T10:44:06.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.671+0000 7fd088de2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08004f400 con 0x556b341798b0 2024-03-21T10:44:06.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.672+0000 7fd07fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b34182de0 0x556b34199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.672+0000 7fd07fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b34182de0 0x556b34199e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51696/0 (socket says 172.21.15.31:51696) 2024-03-21T10:44:06.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.672+0000 7fd07fe00640 1 -- 172.21.15.31:0/441075967 learned_addr learned my addr 172.21.15.31:0/441075967 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:06.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.673+0000 7fd07fe00640 1 -- 172.21.15.31:0/441075967 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b341798b0 msgr2=0x556b3417ce10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:44:06.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.673+0000 7fd07fe00640 1 --2- 172.21.15.31:0/441075967 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b341798b0 0x556b3417ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.673+0000 7fd07fe00640 1 -- 172.21.15.31:0/441075967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b341805f0 msgr2=0x556b3417d350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:44:06.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.673+0000 7fd07fe00640 1 --2- 172.21.15.31:0/441075967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b341805f0 0x556b3417d350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.673+0000 7fd07fe00640 1 -- 172.21.15.31:0/441075967 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b3417e0e0 con 0x556b34182de0 2024-03-21T10:44:06.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.674+0000 7fd07fe00640 1 --2- 172.21.15.31:0/441075967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b34182de0 0x556b34199e60 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fd078009650 tx=0x7fd078044000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2aba14179252e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:06.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.674+0000 7fd06b400640 1 -- 172.21.15.31:0/441075967 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd078050070 con 0x556b34182de0 2024-03-21T10:44:06.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.675+0000 7fd06b400640 1 -- 172.21.15.31:0/441075967 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd078004040 con 0x556b34182de0 2024-03-21T10:44:06.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.675+0000 7fd06b400640 1 -- 172.21.15.31:0/441075967 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd078039040 con 0x556b34182de0 2024-03-21T10:44:06.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.676+0000 7fd088de2740 1 -- 172.21.15.31:0/441075967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b34182de0 msgr2=0x556b34199e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:06.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.676+0000 7fd088de2740 1 --2- 172.21.15.31:0/441075967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b34182de0 0x556b34199e60 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fd078009650 tx=0x7fd078044000 comp rx=0 tx=0).stop 2024-03-21T10:44:06.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.678+0000 7fd088de2740 1 -- 172.21.15.31:0/441075967 shutdown_connections 2024-03-21T10:44:06.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.678+0000 7fd088de2740 1 --2- 172.21.15.31:0/441075967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b34182de0 0x556b34199e60 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.678+0000 7fd088de2740 1 --2- 172.21.15.31:0/441075967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b341805f0 0x556b3417d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.678+0000 7fd088de2740 1 --2- 172.21.15.31:0/441075967 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b341798b0 0x556b3417ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.678+0000 7fd088de2740 1 -- 172.21.15.31:0/441075967 >> 172.21.15.31:0/441075967 conn(0x556b34173f80 msgr2=0x556b34173160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:06.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.682+0000 7fd088de2740 1 -- 172.21.15.31:0/441075967 shutdown_connections 2024-03-21T10:44:06.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.682+0000 7fd088de2740 1 -- 172.21.15.31:0/441075967 wait complete. 2024-03-21T10:44:06.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.683+0000 7fd088de2740 1 -- start start 2024-03-21T10:44:06.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.683+0000 7fd088de2740 1 Processor -- start 2024-03-21T10:44:06.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.687+0000 7fd088de2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b34182de0 0x556b341896a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.687+0000 7fd088de2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b34189be0 0x556b3418a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.687+0000 7fd088de2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b341eca40 0x556b341eee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.687+0000 7fd088de2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd080118a90 con 0x556b34182de0 2024-03-21T10:44:06.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.687+0000 7fd088de2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0801b47c0 con 0x556b341eca40 2024-03-21T10:44:06.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.687+0000 7fd088de2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0801b3b20 con 0x556b34189be0 2024-03-21T10:44:06.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.687+0000 7fd088de2740 10 client.0 _ll_register_callbacks cb 0x7fd06c005e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:06.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.687+0000 7fd07fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b341eca40 0x556b341eee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.687+0000 7fd07f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b34182de0 0x556b341896a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.687+0000 7fd07fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b341eca40 0x556b341eee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51700/0 (socket says 172.21.15.31:51700) 2024-03-21T10:44:06.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.687+0000 7fd07fe00640 1 -- 172.21.15.31:0/2914291956 learned_addr learned my addr 172.21.15.31:0/2914291956 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:06.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.688+0000 7fd07fe00640 1 -- 172.21.15.31:0/2914291956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b34189be0 msgr2=0x556b3418a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:06.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.688+0000 7fd07ea00640 1 --2- 172.21.15.31:0/2914291956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b34189be0 0x556b3418a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.688+0000 7fd07fe00640 1 --2- 172.21.15.31:0/2914291956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b34189be0 0x556b3418a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.688+0000 7fd07fe00640 1 -- 172.21.15.31:0/2914291956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b34182de0 msgr2=0x556b341896a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:06.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.688+0000 7fd07fe00640 1 --2- 172.21.15.31:0/2914291956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b34182de0 0x556b341896a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.688+0000 7fd07fe00640 1 -- 172.21.15.31:0/2914291956 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556b341ef4d0 con 0x556b341eca40 2024-03-21T10:44:06.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.688+0000 7fd07fe00640 1 --2- 172.21.15.31:0/2914291956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b341eca40 0x556b341eee30 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fd07804e040 tx=0x7fd078009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38c3c6ce17539f12 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:06.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.688+0000 7fd068c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:44:06.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.689+0000 7fd068c00640 1 -- 172.21.15.31:0/2914291956 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd078059060 con 0x556b341eca40 2024-03-21T10:44:06.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.689+0000 7fd068c00640 1 -- 172.21.15.31:0/2914291956 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fd0780043b0 con 0x556b341eca40 2024-03-21T10:44:06.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.689+0000 7fd088de2740 1 -- 172.21.15.31:0/2914291956 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd078004040 con 0x556b341eca40 2024-03-21T10:44:06.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.691+0000 7fd068c00640 1 -- 172.21.15.31:0/2914291956 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd078050040 con 0x556b341eca40 2024-03-21T10:44:06.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.691+0000 7fd088de2740 10 client.27023 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:06.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.691+0000 7fd088de2740 1 -- 172.21.15.31:0/2914291956 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x556b341f07d0 con 0x556b341eca40 2024-03-21T10:44:06.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.692+0000 7fd068c00640 1 -- 172.21.15.31:0/2914291956 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd078004040 con 0x556b341eca40 2024-03-21T10:44:06.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.692+0000 7fd068c00640 1 -- 172.21.15.31:0/2914291956 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd078034440 con 0x556b341eca40 2024-03-21T10:44:06.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.693+0000 7fd088de2740 10 client.27023 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:06.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.693+0000 7fd088de2740 10 client.27023 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:44:06.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.693+0000 7fd088de2740 20 client.27023 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:06.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.693+0000 7fd088de2740 1 --2- 172.21.15.31:0/2914291956 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556b341f0de0 0x556b341f3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:06.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.693+0000 7fd088de2740 4 client.27023 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:44:06.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.693+0000 7fd088de2740 1 -- 172.21.15.31:0/2914291956 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556b340ea800 con 0x556b341f0de0 2024-03-21T10:44:06.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.696+0000 7fd07f400640 1 --2- 172.21.15.31:0/2914291956 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556b341f0de0 0x556b341f3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:06.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.697+0000 7fd07f400640 1 --2- 172.21.15.31:0/2914291956 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556b341f0de0 0x556b341f3200 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1277e000ae4aed6 server_cookie=9909fb0c585c5fa3 in_seq=0 out_seq=0 2024-03-21T10:44:06.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.697+0000 7fd068c00640 10 client.27023 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:06.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.698+0000 7fd068c00640 1 -- 172.21.15.31:0/2914291956 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556b340ea800 con 0x556b341f0de0 2024-03-21T10:44:06.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.698+0000 7fd068c00640 10 client.27023 handle_command_reply: tid=0 2024-03-21T10:44:06.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.698+0000 7fd088de2740 1 client.27023 shutdown 2024-03-21T10:44:06.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.700+0000 7fd088de2740 1 -- 172.21.15.31:0/2914291956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b341eca40 msgr2=0x556b341eee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:06.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.700+0000 7fd088de2740 1 --2- 172.21.15.31:0/2914291956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b341eca40 0x556b341eee30 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fd07804e040 tx=0x7fd078009680 comp rx=0 tx=0).stop 2024-03-21T10:44:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.701+0000 7fd088de2740 1 -- 172.21.15.31:0/2914291956 shutdown_connections 2024-03-21T10:44:06.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.702+0000 7fd088de2740 1 --2- 172.21.15.31:0/2914291956 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556b341f0de0 0x556b341f3200 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.702+0000 7fd088de2740 1 --2- 172.21.15.31:0/2914291956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b341eca40 0x556b341eee30 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.702+0000 7fd088de2740 1 --2- 172.21.15.31:0/2914291956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b34189be0 0x556b3418a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.702+0000 7fd088de2740 1 --2- 172.21.15.31:0/2914291956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b34182de0 0x556b341896a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.702+0000 7fd068c00640 0 client.27023 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:06.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.702+0000 7fd088de2740 1 -- 172.21.15.31:0/2914291956 >> 172.21.15.31:0/2914291956 conn(0x556b34173f80 msgr2=0x556b3417a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:06.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.703+0000 7fd088de2740 1 -- 172.21.15.31:0/2914291956 shutdown_connections 2024-03-21T10:44:06.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.703+0000 7fd088de2740 1 -- 172.21.15.31:0/2914291956 wait complete. 2024-03-21T10:44:06.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.711+0000 7fd088de2740 20 client.27023 trim_cache size 0 max 16384 2024-03-21T10:44:06.708 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:44:06.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.712+0000 7fd084a00640 1 -- 172.21.15.31:0/3740759521 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd0640a3110 msgr2=0x7fd0640a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:06.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.712+0000 7fd084a00640 1 --2- 172.21.15.31:0/3740759521 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd0640a3110 0x7fd0640a55d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd078037040 tx=0x7fd078005fb0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.712+0000 7fd084a00640 1 -- 172.21.15.31:0/3740759521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd08010e310 msgr2=0x7fd0801acca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:06.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.712+0000 7fd084a00640 1 --2- 172.21.15.31:0/3740759521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd08010e310 0x7fd0801acca0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd07000b4a0 tx=0x7fd07000b960 comp rx=0 tx=0).stop 2024-03-21T10:44:06.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.715+0000 7fd084a00640 1 -- 172.21.15.31:0/3740759521 shutdown_connections 2024-03-21T10:44:06.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.715+0000 7fd084a00640 1 --2- 172.21.15.31:0/3740759521 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd0640a3110 0x7fd0640a55d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.715+0000 7fd084a00640 1 --2- 172.21.15.31:0/3740759521 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd08010fc90 0x7fd0801a6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.715+0000 7fd084a00640 1 --2- 172.21.15.31:0/3740759521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd08010e310 0x7fd0801acca0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.715+0000 7fd084a00640 1 --2- 172.21.15.31:0/3740759521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd08010d780 0x7fd0801ac760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:06.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.715+0000 7fd084a00640 1 -- 172.21.15.31:0/3740759521 >> 172.21.15.31:0/3740759521 conn(0x7fd08006c580 msgr2=0x7fd0800709a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:06.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.716+0000 7fd084a00640 1 -- 172.21.15.31:0/3740759521 shutdown_connections 2024-03-21T10:44:06.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:06.716+0000 7fd084a00640 1 -- 172.21.15.31:0/3740759521 wait complete. 2024-03-21T10:44:07.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:06 smithi031 ceph-mon[31673]: pgmap v169: 129 pgs: 129 active+clean; 35 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 269 MiB/s wr, 1.50k op/s 2024-03-21T10:44:07.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:06 smithi040 ceph-mon[36154]: pgmap v169: 129 pgs: 129 active+clean; 35 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 269 MiB/s wr, 1.50k op/s 2024-03-21T10:44:07.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:06 smithi070 ceph-mon[36330]: pgmap v169: 129 pgs: 129 active+clean; 35 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 269 MiB/s wr, 1.50k op/s 2024-03-21T10:44:07.752 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-21T10:44:07.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:44:07.752 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-21T10:44:08.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.125+0000 7f61cf400640 1 --2- 172.21.15.31:0/433905438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61c80957f0 0x7f61c80a1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:44:08.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.126+0000 7f61d5e00640 1 -- 172.21.15.31:0/433905438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61c8094e50 msgr2=0x7f61c80952b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:08.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.126+0000 7f61d5e00640 1 --2- 172.21.15.31:0/433905438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61c8094e50 0x7f61c80952b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f61bc009900 tx=0x7f61bc02bc50 comp rx=0 tx=0).stop 2024-03-21T10:44:08.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.126+0000 7f61d5e00640 1 -- 172.21.15.31:0/433905438 shutdown_connections 2024-03-21T10:44:08.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.126+0000 7f61d5e00640 1 --2- 172.21.15.31:0/433905438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61c80957f0 0x7f61c80a1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.126+0000 7f61d5e00640 1 --2- 172.21.15.31:0/433905438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61c8094e50 0x7f61c80952b0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.126+0000 7f61d5e00640 1 --2- 172.21.15.31:0/433905438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61c8099bc0 0x7f61c8099fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.126+0000 7f61d5e00640 1 -- 172.21.15.31:0/433905438 >> 172.21.15.31:0/433905438 conn(0x7f61c800a740 msgr2=0x7f61c800ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:08.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.126+0000 7f61d5e00640 1 -- 172.21.15.31:0/433905438 shutdown_connections 2024-03-21T10:44:08.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.126+0000 7f61d5e00640 1 -- 172.21.15.31:0/433905438 wait complete. 2024-03-21T10:44:08.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.128+0000 7f61d5e00640 1 Processor -- start 2024-03-21T10:44:08.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.128+0000 7f61d5e00640 1 -- start start 2024-03-21T10:44:08.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.128+0000 7f61d5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61c8094e50 0x7f61c812a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.129+0000 7f61d5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61c80957f0 0x7f61c812a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.129+0000 7f61d5e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61c8099bc0 0x7f61c812ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.129+0000 7f61d5e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c80a3f60 con 0x7f61c8094e50 2024-03-21T10:44:08.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.129+0000 7f61d5e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c80a3de0 con 0x7f61c80957f0 2024-03-21T10:44:08.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.129+0000 7f61d5e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c80a40e0 con 0x7f61c8099bc0 2024-03-21T10:44:08.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.129+0000 7f61ce000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61c80957f0 0x7f61c812a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.130+0000 7f61ce000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61c80957f0 0x7f61c812a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:57174/0 (socket says 172.21.15.31:57174) 2024-03-21T10:44:08.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.130+0000 7f61cf400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61c8099bc0 0x7f61c812ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.130+0000 7f61ce000640 1 -- 172.21.15.31:0/2180449772 learned_addr learned my addr 172.21.15.31:0/2180449772 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:08.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.130+0000 7f61cea00640 1 --2- 172.21.15.31:0/2180449772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61c8094e50 0x7f61c812a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.130+0000 7f61cf400640 1 -- 172.21.15.31:0/2180449772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61c80957f0 msgr2=0x7f61c812a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:08.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.130+0000 7f61cf400640 1 --2- 172.21.15.31:0/2180449772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61c80957f0 0x7f61c812a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.130+0000 7f61cf400640 1 -- 172.21.15.31:0/2180449772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61c8094e50 msgr2=0x7f61c812a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:08.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.130+0000 7f61cf400640 1 --2- 172.21.15.31:0/2180449772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61c8094e50 0x7f61c812a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.130+0000 7f61cf400640 1 -- 172.21.15.31:0/2180449772 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c812f2c0 con 0x7f61c8099bc0 2024-03-21T10:44:08.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.130+0000 7f61cea00640 1 --2- 172.21.15.31:0/2180449772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61c8094e50 0x7f61c812a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:44:08.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.131+0000 7f61cf400640 1 --2- 172.21.15.31:0/2180449772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61c8099bc0 0x7f61c812ebc0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f61d00674e0 tx=0x7f61d006d4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:08.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.131+0000 7f61d4a00640 1 -- 172.21.15.31:0/2180449772 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d0078020 con 0x7f61c8099bc0 2024-03-21T10:44:08.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.131+0000 7f61d4a00640 1 -- 172.21.15.31:0/2180449772 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61d0073550 con 0x7f61c8099bc0 2024-03-21T10:44:08.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.131+0000 7f61d4a00640 1 -- 172.21.15.31:0/2180449772 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d0069940 con 0x7f61c8099bc0 2024-03-21T10:44:08.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.131+0000 7f61d5e00640 1 -- 172.21.15.31:0/2180449772 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61c812f5b0 con 0x7f61c8099bc0 2024-03-21T10:44:08.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.132+0000 7f61d5e00640 1 -- 172.21.15.31:0/2180449772 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61c8147e80 con 0x7f61c8099bc0 2024-03-21T10:44:08.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.136+0000 7f61d8c27740 1 Processor -- start 2024-03-21T10:44:08.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.137+0000 7f61d4a00640 1 -- 172.21.15.31:0/2180449772 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f61d007b020 con 0x7f61c8099bc0 2024-03-21T10:44:08.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.137+0000 7f61d8c27740 1 -- start start 2024-03-21T10:44:08.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.138+0000 7f61d8c27740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0e990a0 0x5569f0e99480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.138+0000 7f61d8c27740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569f0e99a50 0x5569f0eaaa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.138+0000 7f61d8c27740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569f0eaafa0 0x5569f0ead390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.138+0000 7f61d8c27740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569f0dfdeb0 con 0x5569f0eaafa0 2024-03-21T10:44:08.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.138+0000 7f61d8c27740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569f0da6a60 con 0x5569f0e99a50 2024-03-21T10:44:08.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.138+0000 7f61d8c27740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569f0df9f50 con 0x5569f0e990a0 2024-03-21T10:44:08.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.139+0000 7f61ce000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0e990a0 0x5569f0e99480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.139+0000 7f61ce000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0e990a0 0x5569f0e99480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:45712/0 (socket says 172.21.15.31:45712) 2024-03-21T10:44:08.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.139+0000 7f61ce000640 1 -- 172.21.15.31:0/2259488525 learned_addr learned my addr 172.21.15.31:0/2259488525 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:08.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.139+0000 7f61cea00640 1 --2- 172.21.15.31:0/2259488525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569f0e99a50 0x5569f0eaaa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.139+0000 7f61ce000640 1 --2- 172.21.15.31:0/2259488525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569f0eaafa0 0x5569f0ead390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.139+0000 7f61cea00640 1 -- 172.21.15.31:0/2259488525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0e990a0 msgr2=0x5569f0e99480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:08.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.139+0000 7f61cea00640 1 --2- 172.21.15.31:0/2259488525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0e990a0 0x5569f0e99480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.139+0000 7f61cea00640 1 -- 172.21.15.31:0/2259488525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569f0eaafa0 msgr2=0x5569f0ead390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:08.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.139+0000 7f61cea00640 1 --2- 172.21.15.31:0/2259488525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569f0eaafa0 0x5569f0ead390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.139+0000 7f61cea00640 1 -- 172.21.15.31:0/2259488525 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569f0bda7e0 con 0x5569f0e99a50 2024-03-21T10:44:08.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.140+0000 7f61ce000640 1 --2- 172.21.15.31:0/2259488525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569f0eaafa0 0x5569f0ead390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:44:08.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.140+0000 7f61d4a00640 1 --2- 172.21.15.31:0/2180449772 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f61b40a3170 0x7f61b40a5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.140+0000 7f61cea00640 1 --2- 172.21.15.31:0/2259488525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569f0e99a50 0x5569f0eaaa60 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f61c0009bc0 tx=0x7f61c000e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc2671a20f458d03 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:08.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.140+0000 7f61d4a00640 1 -- 172.21.15.31:0/2180449772 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f61d0132de0 con 0x7f61c8099bc0 2024-03-21T10:44:08.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.140+0000 7f61bbe00640 1 -- 172.21.15.31:0/2259488525 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c0016070 con 0x5569f0e99a50 2024-03-21T10:44:08.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.140+0000 7f61bbe00640 1 -- 172.21.15.31:0/2259488525 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61c0005500 con 0x5569f0e99a50 2024-03-21T10:44:08.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.140+0000 7f61bbe00640 1 -- 172.21.15.31:0/2259488525 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c0011470 con 0x5569f0e99a50 2024-03-21T10:44:08.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.140+0000 7f61ce000640 1 --2- 172.21.15.31:0/2180449772 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f61b40a3170 0x7f61b40a5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.141+0000 7f61d8c27740 1 -- 172.21.15.31:0/2259488525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569f0e99a50 msgr2=0x5569f0eaaa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:08.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.141+0000 7f61d8c27740 1 --2- 172.21.15.31:0/2259488525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569f0e99a50 0x5569f0eaaa60 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f61c0009bc0 tx=0x7f61c000e6c0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.142+0000 7f61d8c27740 1 -- 172.21.15.31:0/2259488525 shutdown_connections 2024-03-21T10:44:08.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.142+0000 7f61d8c27740 1 --2- 172.21.15.31:0/2259488525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569f0eaafa0 0x5569f0ead390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.142+0000 7f61d8c27740 1 --2- 172.21.15.31:0/2259488525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569f0e99a50 0x5569f0eaaa60 secure :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f61c0009bc0 tx=0x7f61c000e6c0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.142+0000 7f61d8c27740 1 --2- 172.21.15.31:0/2259488525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0e990a0 0x5569f0e99480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.142+0000 7f61d8c27740 1 -- 172.21.15.31:0/2259488525 >> 172.21.15.31:0/2259488525 conn(0x5569f0e939c0 msgr2=0x5569f0e93da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:08.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.144+0000 7f61ce000640 1 --2- 172.21.15.31:0/2180449772 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f61b40a3170 0x7f61b40a5630 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f61bc02c870 tx=0x7f61bc000bc0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:08.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.144+0000 7f61d8c27740 1 -- 172.21.15.31:0/2259488525 shutdown_connections 2024-03-21T10:44:08.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.144+0000 7f61d8c27740 1 -- 172.21.15.31:0/2259488525 wait complete. 2024-03-21T10:44:08.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.145+0000 7f61d8c27740 1 -- start start 2024-03-21T10:44:08.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.145+0000 7f61d8c27740 1 Processor -- start 2024-03-21T10:44:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.147+0000 7f61d8c27740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0e990a0 0x5569f0ea7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.147+0000 7f61d8c27740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569f0eaafa0 0x5569f0ea82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.147+0000 7f61d8c27740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569f0ea87f0 0x5569f0eb72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.147+0000 7f61d8c27740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569f0eadd60 con 0x5569f0ea87f0 2024-03-21T10:44:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.147+0000 7f61d8c27740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569f0db9eb0 con 0x5569f0eaafa0 2024-03-21T10:44:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.147+0000 7f61d8c27740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569f0eadee0 con 0x5569f0e990a0 2024-03-21T10:44:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.147+0000 7f61d8c27740 10 client.0 _ll_register_callbacks cb 0x5569f0dae150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:08.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.148+0000 7f61ce000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569f0eaafa0 0x5569f0ea82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.149+0000 7f61ce000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569f0eaafa0 0x5569f0ea82b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:57200/0 (socket says 172.21.15.31:57200) 2024-03-21T10:44:08.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.148+0000 7f61cea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0e990a0 0x5569f0ea7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.149+0000 7f61cea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0e990a0 0x5569f0ea7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:45716/0 (socket says 172.21.15.31:45716) 2024-03-21T10:44:08.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.149+0000 7f61cea00640 1 -- 172.21.15.31:0/751900341 learned_addr learned my addr 172.21.15.31:0/751900341 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:08.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.149+0000 7f61cf400640 1 --2- 172.21.15.31:0/751900341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569f0ea87f0 0x5569f0eb72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.149+0000 7f61cea00640 1 -- 172.21.15.31:0/751900341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569f0eaafa0 msgr2=0x5569f0ea82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:08.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.149+0000 7f61cea00640 1 --2- 172.21.15.31:0/751900341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569f0eaafa0 0x5569f0ea82b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.149+0000 7f61cea00640 1 -- 172.21.15.31:0/751900341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569f0ea87f0 msgr2=0x5569f0eb72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:08.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.149+0000 7f61cea00640 1 --2- 172.21.15.31:0/751900341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569f0ea87f0 0x5569f0eb72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.149+0000 7f61cea00640 1 -- 172.21.15.31:0/751900341 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5569f0c3af70 con 0x5569f0e990a0 2024-03-21T10:44:08.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.150+0000 7f61cea00640 1 --2- 172.21.15.31:0/751900341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0e990a0 0x5569f0ea7d70 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f61c0000c00 tx=0x7f61c0004990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35e6a1f0198959f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:08.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.150+0000 7f61b8c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:44:08.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.150+0000 7f61b8c00640 1 -- 172.21.15.31:0/751900341 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c0016040 con 0x5569f0e990a0 2024-03-21T10:44:08.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.151+0000 7f61b8c00640 1 -- 172.21.15.31:0/751900341 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f61c0005000 con 0x5569f0e990a0 2024-03-21T10:44:08.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.151+0000 7f61d8c27740 1 -- 172.21.15.31:0/751900341 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f61c0005500 con 0x5569f0e990a0 2024-03-21T10:44:08.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.151+0000 7f61b8c00640 1 -- 172.21.15.31:0/751900341 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61c002d050 con 0x5569f0e990a0 2024-03-21T10:44:08.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.152+0000 7f61d8c27740 10 client.26944 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:08.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.152+0000 7f61d8c27740 1 -- 172.21.15.31:0/751900341 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5569f0ad2170 con 0x5569f0e990a0 2024-03-21T10:44:08.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.152+0000 7f61b8c00640 1 -- 172.21.15.31:0/751900341 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61c0009780 con 0x5569f0e990a0 2024-03-21T10:44:08.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.153+0000 7f61b8c00640 1 -- 172.21.15.31:0/751900341 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f61c000f040 con 0x5569f0e990a0 2024-03-21T10:44:08.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.153+0000 7f61d8c27740 10 client.26944 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:08.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.153+0000 7f61d8c27740 10 client.26944 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:44:08.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.153+0000 7f61d8c27740 20 client.26944 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:08.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.153+0000 7f61d8c27740 1 --2- 172.21.15.31:0/751900341 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5569f0e9de40 0x5569f0ea0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.153+0000 7f61d8c27740 4 client.26944 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:44:08.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.153+0000 7f61d8c27740 1 -- 172.21.15.31:0/751900341 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f61c0005500 con 0x5569f0e9de40 2024-03-21T10:44:08.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.154+0000 7f61ce000640 1 --2- 172.21.15.31:0/751900341 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5569f0e9de40 0x5569f0ea0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.155+0000 7f61b8c00640 10 client.26944 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:08.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.155+0000 7f61ce000640 1 --2- 172.21.15.31:0/751900341 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5569f0e9de40 0x5569f0ea0280 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27237083f2d2bd9a server_cookie=1a37a7ca0ef33aef in_seq=0 out_seq=0 2024-03-21T10:44:08.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.156+0000 7f61b8c00640 1 -- 172.21.15.31:0/751900341 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7f61c0005500 con 0x5569f0e9de40 2024-03-21T10:44:08.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.156+0000 7f61b8c00640 10 client.26944 handle_command_reply: tid=0 2024-03-21T10:44:08.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.157+0000 7f61d8c27740 1 client.26944 shutdown 2024-03-21T10:44:08.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.158+0000 7f61d8c27740 1 -- 172.21.15.31:0/751900341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0e990a0 msgr2=0x5569f0ea7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:08.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.158+0000 7f61d8c27740 1 --2- 172.21.15.31:0/751900341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0e990a0 0x5569f0ea7d70 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f61c0000c00 tx=0x7f61c0004990 comp rx=0 tx=0).stop 2024-03-21T10:44:08.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.158+0000 7f61d8c27740 1 -- 172.21.15.31:0/751900341 shutdown_connections 2024-03-21T10:44:08.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.159+0000 7f61d8c27740 1 --2- 172.21.15.31:0/751900341 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5569f0e9de40 0x5569f0ea0280 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.159+0000 7f61d8c27740 1 --2- 172.21.15.31:0/751900341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569f0ea87f0 0x5569f0eb72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.159+0000 7f61d8c27740 1 --2- 172.21.15.31:0/751900341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569f0eaafa0 0x5569f0ea82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.159+0000 7f61d8c27740 1 --2- 172.21.15.31:0/751900341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0e990a0 0x5569f0ea7d70 secure :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f61c0000c00 tx=0x7f61c0004990 comp rx=0 tx=0).stop 2024-03-21T10:44:08.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.159+0000 7f61d8c27740 1 -- 172.21.15.31:0/751900341 >> 172.21.15.31:0/751900341 conn(0x5569f0e939c0 msgr2=0x5569f0e95710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:08.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.159+0000 7f61b8c00640 0 client.26944 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:08.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.159+0000 7f61d8c27740 1 -- 172.21.15.31:0/751900341 shutdown_connections 2024-03-21T10:44:08.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.160+0000 7f61d8c27740 1 -- 172.21.15.31:0/751900341 wait complete. 2024-03-21T10:44:08.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.160+0000 7f61d8c27740 20 client.26944 trim_cache size 0 max 16384 2024-03-21T10:44:08.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.197+0000 7f61d8c27740 1 Processor -- start 2024-03-21T10:44:08.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.197+0000 7f61d8c27740 1 -- start start 2024-03-21T10:44:08.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.198+0000 7f61d8c27740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0ea87f0 0x5569f0eb7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.198+0000 7f61d8c27740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569f0eaafa0 0x5569f0ea2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.198+0000 7f61d8c27740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569f0eaa5b0 0x5569f0ea33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.198+0000 7f61d8c27740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c8154160 con 0x5569f0eaa5b0 2024-03-21T10:44:08.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.198+0000 7f61d8c27740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c8151c60 con 0x5569f0eaafa0 2024-03-21T10:44:08.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.198+0000 7f61d8c27740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c814fd00 con 0x5569f0ea87f0 2024-03-21T10:44:08.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.198+0000 7f61cea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0ea87f0 0x5569f0eb7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.198+0000 7f61cea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0ea87f0 0x5569f0eb7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:45732/0 (socket says 172.21.15.31:45732) 2024-03-21T10:44:08.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.198+0000 7f61cea00640 1 -- 172.21.15.31:0/616831548 learned_addr learned my addr 172.21.15.31:0/616831548 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:08.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.198+0000 7f61ce000640 1 --2- 172.21.15.31:0/616831548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569f0eaafa0 0x5569f0ea2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.199+0000 7f61cea00640 1 -- 172.21.15.31:0/616831548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569f0eaafa0 msgr2=0x5569f0ea2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:08.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.199+0000 7f61cea00640 1 --2- 172.21.15.31:0/616831548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569f0eaafa0 0x5569f0ea2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.199+0000 7f61cea00640 1 -- 172.21.15.31:0/616831548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569f0eaa5b0 msgr2=0x5569f0ea33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:08.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.199+0000 7f61cf400640 1 --2- 172.21.15.31:0/616831548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569f0eaa5b0 0x5569f0ea33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.199+0000 7f61cea00640 1 --2- 172.21.15.31:0/616831548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569f0eaa5b0 0x5569f0ea33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.199+0000 7f61cea00640 1 -- 172.21.15.31:0/616831548 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569f0e93200 con 0x5569f0ea87f0 2024-03-21T10:44:08.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.200+0000 7f61cea00640 1 --2- 172.21.15.31:0/616831548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0ea87f0 0x5569f0eb7210 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f61c00039e0 tx=0x7f61c00094a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8abf55a00424490 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:08.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.200+0000 7f61bb400640 1 -- 172.21.15.31:0/616831548 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c0016030 con 0x5569f0ea87f0 2024-03-21T10:44:08.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.200+0000 7f61bb400640 1 -- 172.21.15.31:0/616831548 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61c0004460 con 0x5569f0ea87f0 2024-03-21T10:44:08.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.200+0000 7f61bb400640 1 -- 172.21.15.31:0/616831548 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c0003770 con 0x5569f0ea87f0 2024-03-21T10:44:08.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.201+0000 7f61d8c27740 1 -- 172.21.15.31:0/616831548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0ea87f0 msgr2=0x5569f0eb7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:08.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.201+0000 7f61d8c27740 1 --2- 172.21.15.31:0/616831548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0ea87f0 0x5569f0eb7210 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f61c00039e0 tx=0x7f61c00094a0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.202+0000 7f61d8c27740 1 -- 172.21.15.31:0/616831548 shutdown_connections 2024-03-21T10:44:08.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.202+0000 7f61d8c27740 1 --2- 172.21.15.31:0/616831548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569f0eaa5b0 0x5569f0ea33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.202+0000 7f61d8c27740 1 --2- 172.21.15.31:0/616831548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569f0eaafa0 0x5569f0ea2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.202+0000 7f61d8c27740 1 --2- 172.21.15.31:0/616831548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0ea87f0 0x5569f0eb7210 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.202+0000 7f61d8c27740 1 -- 172.21.15.31:0/616831548 >> 172.21.15.31:0/616831548 conn(0x5569f0e939c0 msgr2=0x5569f0ea7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:08.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.203+0000 7f61d8c27740 1 -- 172.21.15.31:0/616831548 shutdown_connections 2024-03-21T10:44:08.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.203+0000 7f61d8c27740 1 -- 172.21.15.31:0/616831548 wait complete. 2024-03-21T10:44:08.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.203+0000 7f61d8c27740 1 -- start start 2024-03-21T10:44:08.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.204+0000 7f61d8c27740 1 Processor -- start 2024-03-21T10:44:08.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.204+0000 7f61d8c27740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0ea87f0 0x5569f0ea10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.204+0000 7f61d8c27740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569f0eaa5b0 0x5569f0ea15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.204+0000 7f61d8c27740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569f0eaafa0 0x5569f0f1e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.204+0000 7f61d8c27740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c8150300 con 0x5569f0eaa5b0 2024-03-21T10:44:08.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.204+0000 7f61d8c27740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c8151600 con 0x5569f0eaafa0 2024-03-21T10:44:08.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.204+0000 7f61d8c27740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c8150940 con 0x5569f0ea87f0 2024-03-21T10:44:08.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.205+0000 7f61d8c27740 10 client.0 _ll_register_callbacks cb 0x7f61bc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:08.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.205+0000 7f61cea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0ea87f0 0x5569f0ea10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.206+0000 7f61cea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0ea87f0 0x5569f0ea10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:45736/0 (socket says 172.21.15.31:45736) 2024-03-21T10:44:08.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.206+0000 7f61cea00640 1 -- 172.21.15.31:0/2218284415 learned_addr learned my addr 172.21.15.31:0/2218284415 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:08.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.206+0000 7f61cf400640 1 --2- 172.21.15.31:0/2218284415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569f0eaafa0 0x5569f0f1e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.206+0000 7f61ce000640 1 --2- 172.21.15.31:0/2218284415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569f0eaa5b0 0x5569f0ea15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.207+0000 7f61cea00640 1 -- 172.21.15.31:0/2218284415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569f0eaafa0 msgr2=0x5569f0f1e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:08.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.207+0000 7f61cea00640 1 --2- 172.21.15.31:0/2218284415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569f0eaafa0 0x5569f0f1e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.207+0000 7f61cea00640 1 -- 172.21.15.31:0/2218284415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569f0eaa5b0 msgr2=0x5569f0ea15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:08.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.207+0000 7f61cea00640 1 --2- 172.21.15.31:0/2218284415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569f0eaa5b0 0x5569f0ea15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.207+0000 7f61cea00640 1 -- 172.21.15.31:0/2218284415 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5569f0f1e920 con 0x5569f0ea87f0 2024-03-21T10:44:08.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.207+0000 7f61ce000640 1 --2- 172.21.15.31:0/2218284415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569f0eaa5b0 0x5569f0ea15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:44:08.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.207+0000 7f61cea00640 1 --2- 172.21.15.31:0/2218284415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0ea87f0 0x5569f0ea10b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f61c0009bc0 tx=0x7f61c0004a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d4bd43ec588fd82 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:08.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.207+0000 7f61b8c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:44:08.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.208+0000 7f61b8c00640 1 -- 172.21.15.31:0/2218284415 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c0016040 con 0x5569f0ea87f0 2024-03-21T10:44:08.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.208+0000 7f61d8c27740 1 -- 172.21.15.31:0/2218284415 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f61c0004460 con 0x5569f0ea87f0 2024-03-21T10:44:08.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.208+0000 7f61b8c00640 1 -- 172.21.15.31:0/2218284415 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f61c0020070 con 0x5569f0ea87f0 2024-03-21T10:44:08.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.208+0000 7f61b8c00640 1 -- 172.21.15.31:0/2218284415 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61c0041050 con 0x5569f0ea87f0 2024-03-21T10:44:08.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.208+0000 7f61d8c27740 10 client.26950 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:08.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.208+0000 7f61d8c27740 1 -- 172.21.15.31:0/2218284415 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5569f0f1fb60 con 0x5569f0ea87f0 2024-03-21T10:44:08.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.209+0000 7f61b8c00640 1 -- 172.21.15.31:0/2218284415 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61c0004460 con 0x5569f0ea87f0 2024-03-21T10:44:08.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.209+0000 7f61b8c00640 1 -- 172.21.15.31:0/2218284415 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f61c0024470 con 0x5569f0ea87f0 2024-03-21T10:44:08.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.209+0000 7f61d8c27740 10 client.26950 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:08.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.209+0000 7f61d8c27740 10 client.26950 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:44:08.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.209+0000 7f61d8c27740 20 client.26950 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:08.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.209+0000 7f61d8c27740 1 --2- 172.21.15.31:0/2218284415 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5569f0f20110 0x5569f0f22500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.209+0000 7f61d8c27740 4 client.26950 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-21T10:44:08.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.209+0000 7f61d8c27740 1 -- 172.21.15.31:0/2218284415 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5569f0e0b3f0 con 0x5569f0f20110 2024-03-21T10:44:08.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.209+0000 7f61ce000640 1 --2- 172.21.15.31:0/2218284415 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5569f0f20110 0x5569f0f22500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.210+0000 7f61ce000640 1 --2- 172.21.15.31:0/2218284415 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5569f0f20110 0x5569f0f22500 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29497aa489004eb3 server_cookie=f265cb8b2898c1f4 in_seq=0 out_seq=0 2024-03-21T10:44:08.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.210+0000 7f61b8c00640 10 client.26950 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:08.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.212+0000 7f61b8c00640 1 -- 172.21.15.31:0/2218284415 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5569f0e0b3f0 con 0x5569f0f20110 2024-03-21T10:44:08.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.212+0000 7f61b8c00640 10 client.26950 handle_command_reply: tid=0 2024-03-21T10:44:08.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.212+0000 7f61d8c27740 1 client.26950 shutdown 2024-03-21T10:44:08.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.212+0000 7f61d8c27740 1 -- 172.21.15.31:0/2218284415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0ea87f0 msgr2=0x5569f0ea10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:08.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.212+0000 7f61d8c27740 1 --2- 172.21.15.31:0/2218284415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0ea87f0 0x5569f0ea10b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f61c0009bc0 tx=0x7f61c0004a00 comp rx=0 tx=0).stop 2024-03-21T10:44:08.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.213+0000 7f61d8c27740 1 -- 172.21.15.31:0/2218284415 shutdown_connections 2024-03-21T10:44:08.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.213+0000 7f61d8c27740 1 --2- 172.21.15.31:0/2218284415 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5569f0f20110 0x5569f0f22500 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.213+0000 7f61d8c27740 1 --2- 172.21.15.31:0/2218284415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5569f0eaafa0 0x5569f0f1e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.213+0000 7f61d8c27740 1 --2- 172.21.15.31:0/2218284415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5569f0eaa5b0 0x5569f0ea15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.213+0000 7f61d8c27740 1 --2- 172.21.15.31:0/2218284415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f0ea87f0 0x5569f0ea10b0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.213+0000 7f61d8c27740 1 -- 172.21.15.31:0/2218284415 >> 172.21.15.31:0/2218284415 conn(0x5569f0e939c0 msgr2=0x5569f0e921f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:08.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.213+0000 7f61b8c00640 0 client.26950 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:08.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.213+0000 7f61d8c27740 1 -- 172.21.15.31:0/2218284415 shutdown_connections 2024-03-21T10:44:08.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.214+0000 7f61d8c27740 1 -- 172.21.15.31:0/2218284415 wait complete. 2024-03-21T10:44:08.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.214+0000 7f61d8c27740 20 client.26950 trim_cache size 0 max 16384 2024-03-21T10:44:08.211 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:44:08.212 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-03-21T10:44:08.212 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "0df2c906-e04c-4df4-97f4-5d11c0338140", 2024-03-21T10:44:08.212 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-03-21T10:44:08.212 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:44:08.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.217+0000 7f61d5e00640 1 -- 172.21.15.31:0/2180449772 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f61b40a3170 msgr2=0x7f61b40a5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:08.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.217+0000 7f61d5e00640 1 --2- 172.21.15.31:0/2180449772 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f61b40a3170 0x7f61b40a5630 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f61bc02c870 tx=0x7f61bc000bc0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.217+0000 7f61d5e00640 1 -- 172.21.15.31:0/2180449772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61c8099bc0 msgr2=0x7f61c812ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:08.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.217+0000 7f61d5e00640 1 --2- 172.21.15.31:0/2180449772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61c8099bc0 0x7f61c812ebc0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f61d00674e0 tx=0x7f61d006d4a0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.217+0000 7f61d5e00640 1 -- 172.21.15.31:0/2180449772 shutdown_connections 2024-03-21T10:44:08.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.217+0000 7f61d5e00640 1 --2- 172.21.15.31:0/2180449772 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f61b40a3170 0x7f61b40a5630 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.217+0000 7f61d5e00640 1 --2- 172.21.15.31:0/2180449772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61c8099bc0 0x7f61c812ebc0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.217+0000 7f61d5e00640 1 --2- 172.21.15.31:0/2180449772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61c80957f0 0x7f61c812a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.217+0000 7f61d5e00640 1 --2- 172.21.15.31:0/2180449772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61c8094e50 0x7f61c812a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.217+0000 7f61d5e00640 1 -- 172.21.15.31:0/2180449772 >> 172.21.15.31:0/2180449772 conn(0x7f61c800a740 msgr2=0x7f61c80929b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:08.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.218+0000 7f61d5e00640 1 -- 172.21.15.31:0/2180449772 shutdown_connections 2024-03-21T10:44:08.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.219+0000 7f61d5e00640 1 -- 172.21.15.31:0/2180449772 wait complete. 2024-03-21T10:44:08.255 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:44:08.256 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:44:08.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.621+0000 7f4724800640 1 -- 172.21.15.31:0/3567725993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f471c109870 msgr2=0x7f471c109c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:08.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.621+0000 7f4724800640 1 --2- 172.21.15.31:0/3567725993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f471c109870 0x7f471c109c50 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f4710009930 tx=0x7f471002bc30 comp rx=0 tx=0).stop 2024-03-21T10:44:08.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.631+0000 7f4724800640 1 -- 172.21.15.31:0/3567725993 shutdown_connections 2024-03-21T10:44:08.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.631+0000 7f4724800640 1 --2- 172.21.15.31:0/3567725993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f471c104220 0x7f471c110730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.631+0000 7f4724800640 1 --2- 172.21.15.31:0/3567725993 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f471c103860 0x7f471c103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.631+0000 7f4724800640 1 --2- 172.21.15.31:0/3567725993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f471c109870 0x7f471c109c50 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.631+0000 7f4724800640 1 -- 172.21.15.31:0/3567725993 >> 172.21.15.31:0/3567725993 conn(0x7f471c0ff310 msgr2=0x7f471c101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:08.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.631+0000 7f4724800640 1 -- 172.21.15.31:0/3567725993 shutdown_connections 2024-03-21T10:44:08.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.631+0000 7f4724800640 1 -- 172.21.15.31:0/3567725993 wait complete. 2024-03-21T10:44:08.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.633+0000 7f4724800640 1 Processor -- start 2024-03-21T10:44:08.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.633+0000 7f4724800640 1 -- start start 2024-03-21T10:44:08.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.635+0000 7f4724800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f471c103860 0x7f471c10f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.636+0000 7f4724800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f471c104220 0x7f471c10f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.636+0000 7f4724800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f471c109870 0x7f471c10fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.636+0000 7f4721400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f471c104220 0x7f471c10f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.636+0000 7f4721400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f471c104220 0x7f471c10f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:45744/0 (socket says 172.21.15.31:45744) 2024-03-21T10:44:08.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.636+0000 7f4721400640 1 -- 172.21.15.31:0/2432457568 learned_addr learned my addr 172.21.15.31:0/2432457568 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:08.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.636+0000 7f4724800640 1 -- 172.21.15.31:0/2432457568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c076de0 con 0x7f471c103860 2024-03-21T10:44:08.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.636+0000 7f4724800640 1 -- 172.21.15.31:0/2432457568 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c076c60 con 0x7f471c109870 2024-03-21T10:44:08.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.636+0000 7f4724800640 1 -- 172.21.15.31:0/2432457568 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c076f60 con 0x7f471c104220 2024-03-21T10:44:08.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.637+0000 7f4722800640 1 --2- 172.21.15.31:0/2432457568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f471c109870 0x7f471c10fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.637+0000 7f4721e00640 1 --2- 172.21.15.31:0/2432457568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f471c103860 0x7f471c10f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.638+0000 7f4722800640 1 -- 172.21.15.31:0/2432457568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f471c104220 msgr2=0x7f471c10f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:08.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.638+0000 7f4722800640 1 --2- 172.21.15.31:0/2432457568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f471c104220 0x7f471c10f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.638+0000 7f4722800640 1 -- 172.21.15.31:0/2432457568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f471c103860 msgr2=0x7f471c10f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:08.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.638+0000 7f4722800640 1 --2- 172.21.15.31:0/2432457568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f471c103860 0x7f471c10f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.638+0000 7f4722800640 1 -- 172.21.15.31:0/2432457568 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471c114c90 con 0x7f471c109870 2024-03-21T10:44:08.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.641+0000 7f4722800640 1 --2- 172.21.15.31:0/2432457568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f471c109870 0x7f471c10fcb0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f471800be20 tx=0x7f471800bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:08.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.642+0000 7f4716a00640 1 -- 172.21.15.31:0/2432457568 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4718019070 con 0x7f471c109870 2024-03-21T10:44:08.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.643+0000 7f4724800640 1 -- 172.21.15.31:0/2432457568 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f471c114f80 con 0x7f471c109870 2024-03-21T10:44:08.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.644+0000 7f4724800640 1 -- 172.21.15.31:0/2432457568 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f471c1154d0 con 0x7f471c109870 2024-03-21T10:44:08.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.644+0000 7f4716a00640 1 -- 172.21.15.31:0/2432457568 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4718014920 con 0x7f471c109870 2024-03-21T10:44:08.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.644+0000 7f4716a00640 1 -- 172.21.15.31:0/2432457568 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4718019070 con 0x7f471c109870 2024-03-21T10:44:08.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.659+0000 7f4716a00640 1 -- 172.21.15.31:0/2432457568 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f471801c070 con 0x7f471c109870 2024-03-21T10:44:08.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.660+0000 7f47271fd740 1 Processor -- start 2024-03-21T10:44:08.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.661+0000 7f47271fd740 1 -- start start 2024-03-21T10:44:08.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.661+0000 7f47271fd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224bc00 0x562f1224a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.661+0000 7f47271fd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1224af20 0x562f1224b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.661+0000 7f47271fd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562f12254290 0x562f12256680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.661+0000 7f47271fd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f121af250 con 0x562f12254290 2024-03-21T10:44:08.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.661+0000 7f47271fd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f12157a30 con 0x562f1224bc00 2024-03-21T10:44:08.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.661+0000 7f47271fd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f121ab2f0 con 0x562f1224af20 2024-03-21T10:44:08.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.661+0000 7f4716a00640 1 --2- 172.21.15.31:0/2432457568 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f47040a3310 0x7f47040a57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.662+0000 7f4721400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224bc00 0x562f1224a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.662+0000 7f4721e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1224af20 0x562f1224b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.662+0000 7f4721400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224bc00 0x562f1224a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:57248/0 (socket says 172.21.15.31:57248) 2024-03-21T10:44:08.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.662+0000 7f4721400640 1 -- 172.21.15.31:0/115230037 learned_addr learned my addr 172.21.15.31:0/115230037 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:08.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.662+0000 7f4722800640 1 --2- 172.21.15.31:0/2432457568 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f47040a3310 0x7f47040a57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.662+0000 7f4716a00640 1 -- 172.21.15.31:0/2432457568 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f47180cec80 con 0x7f471c109870 2024-03-21T10:44:08.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.662+0000 7f4721e00640 1 -- 172.21.15.31:0/115230037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224bc00 msgr2=0x562f1224a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:08.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.662+0000 7f4721e00640 1 --2- 172.21.15.31:0/115230037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224bc00 0x562f1224a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.662+0000 7f4721e00640 1 -- 172.21.15.31:0/115230037 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562f12254290 msgr2=0x562f12256680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:08.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.662+0000 7f4721e00640 1 --2- 172.21.15.31:0/115230037 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562f12254290 0x562f12256680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.662+0000 7f4721e00640 1 -- 172.21.15.31:0/115230037 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562f12142690 con 0x562f1224af20 2024-03-21T10:44:08.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.663+0000 7f4722800640 1 --2- 172.21.15.31:0/2432457568 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f47040a3310 0x7f47040a57d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4718098a60 tx=0x7f4718004b50 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:08.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.663+0000 7f4721e00640 1 --2- 172.21.15.31:0/115230037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1224af20 0x562f1224b3a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f4710036040 tx=0x7f4710005e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c27f94b701fdf9e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:08.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.664+0000 7f470b000640 1 -- 172.21.15.31:0/115230037 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4710044020 con 0x562f1224af20 2024-03-21T10:44:08.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.664+0000 7f470b000640 1 -- 172.21.15.31:0/115230037 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4710004240 con 0x562f1224af20 2024-03-21T10:44:08.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.664+0000 7f470b000640 1 -- 172.21.15.31:0/115230037 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47100046c0 con 0x562f1224af20 2024-03-21T10:44:08.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.666+0000 7f47271fd740 1 -- 172.21.15.31:0/115230037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1224af20 msgr2=0x562f1224b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:08.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.666+0000 7f47271fd740 1 --2- 172.21.15.31:0/115230037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1224af20 0x562f1224b3a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f4710036040 tx=0x7f4710005e60 comp rx=0 tx=0).stop 2024-03-21T10:44:08.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.668+0000 7f47271fd740 1 -- 172.21.15.31:0/115230037 shutdown_connections 2024-03-21T10:44:08.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.668+0000 7f47271fd740 1 --2- 172.21.15.31:0/115230037 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562f12254290 0x562f12256680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.668+0000 7f47271fd740 1 --2- 172.21.15.31:0/115230037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f1224af20 0x562f1224b3a0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.668+0000 7f47271fd740 1 --2- 172.21.15.31:0/115230037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224bc00 0x562f1224a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.668+0000 7f47271fd740 1 -- 172.21.15.31:0/115230037 >> 172.21.15.31:0/115230037 conn(0x562f12212200 msgr2=0x562f12245970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:08.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.668+0000 7f47271fd740 1 -- 172.21.15.31:0/115230037 shutdown_connections 2024-03-21T10:44:08.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.668+0000 7f47271fd740 1 -- 172.21.15.31:0/115230037 wait complete. 2024-03-21T10:44:08.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.671+0000 7f47271fd740 1 -- start start 2024-03-21T10:44:08.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.671+0000 7f47271fd740 1 Processor -- start 2024-03-21T10:44:08.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.672+0000 7f47271fd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224af20 0x562f122539b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.672+0000 7f47271fd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562f1224bc00 0x562f12250d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.672+0000 7f47271fd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f12254290 0x562f12251280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.672+0000 7f47271fd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f122570a0 con 0x562f1224bc00 2024-03-21T10:44:08.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.673+0000 7f47271fd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f1216ae80 con 0x562f1224af20 2024-03-21T10:44:08.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.673+0000 7f47271fd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f12257220 con 0x562f12254290 2024-03-21T10:44:08.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.673+0000 7f47271fd740 10 client.0 _ll_register_callbacks cb 0x562f1217d130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:08.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.676+0000 7f4721400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224af20 0x562f122539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.676+0000 7f4721400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224af20 0x562f122539b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:57252/0 (socket says 172.21.15.31:57252) 2024-03-21T10:44:08.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.676+0000 7f4721400640 1 -- 172.21.15.31:0/882747661 learned_addr learned my addr 172.21.15.31:0/882747661 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:08.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.676+0000 7f4721400640 1 --2- 172.21.15.31:0/882747661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f12254290 0x562f12251280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.676+0000 7f4721400640 1 -- 172.21.15.31:0/882747661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f12254290 msgr2=0x562f12251280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:08.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.676+0000 7f4721400640 1 --2- 172.21.15.31:0/882747661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f12254290 0x562f12251280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.676+0000 7f4721400640 1 -- 172.21.15.31:0/882747661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562f1224bc00 msgr2=0x562f12250d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:08.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.676+0000 7f4721400640 1 --2- 172.21.15.31:0/882747661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562f1224bc00 0x562f12250d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.676+0000 7f4721400640 1 -- 172.21.15.31:0/882747661 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562f11f8d7d0 con 0x562f1224af20 2024-03-21T10:44:08.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.677+0000 7f4721400640 1 --2- 172.21.15.31:0/882747661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224af20 0x562f122539b0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f470c011d10 tx=0x7f470c009970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72fd13723e1cf4cb server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:08.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.677+0000 7f46ff400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:44:08.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.678+0000 7f46ff400640 1 -- 172.21.15.31:0/882747661 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c01a040 con 0x562f1224af20 2024-03-21T10:44:08.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.678+0000 7f46ff400640 1 -- 172.21.15.31:0/882747661 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f470c004250 con 0x562f1224af20 2024-03-21T10:44:08.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.678+0000 7f47271fd740 1 -- 172.21.15.31:0/882747661 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4710004240 con 0x562f1224af20 2024-03-21T10:44:08.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.679+0000 7f46ff400640 1 -- 172.21.15.31:0/882747661 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f470c026050 con 0x562f1224af20 2024-03-21T10:44:08.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.679+0000 7f47271fd740 10 client.27050 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:08.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.679+0000 7f47271fd740 1 -- 172.21.15.31:0/882747661 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x562f11ec33f0 con 0x562f1224af20 2024-03-21T10:44:08.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.680+0000 7f46ff400640 1 -- 172.21.15.31:0/882747661 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4710004240 con 0x562f1224af20 2024-03-21T10:44:08.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.682+0000 7f46ff400640 1 -- 172.21.15.31:0/882747661 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f470c0204e0 con 0x562f1224af20 2024-03-21T10:44:08.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.683+0000 7f47271fd740 10 client.27050 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:08.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.683+0000 7f47271fd740 10 client.27050 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:44:08.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.683+0000 7f47271fd740 20 client.27050 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:08.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.683+0000 7f47271fd740 1 --2- 172.21.15.31:0/882747661 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562f12264260 0x562f12266670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.683+0000 7f47271fd740 4 client.27050 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:44:08.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.683+0000 7f47271fd740 1 -- 172.21.15.31:0/882747661 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562f121bc790 con 0x562f12264260 2024-03-21T10:44:08.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.683+0000 7f4721e00640 1 --2- 172.21.15.31:0/882747661 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562f12264260 0x562f12266670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.684+0000 7f4721e00640 1 --2- 172.21.15.31:0/882747661 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562f12264260 0x562f12266670 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb6e2bee8e57927b server_cookie=48825104a52209c6 in_seq=0 out_seq=0 2024-03-21T10:44:08.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.684+0000 7f46ff400640 10 client.27050 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:08.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.688+0000 7f46ff400640 1 -- 172.21.15.31:0/882747661 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562f121bc790 con 0x562f12264260 2024-03-21T10:44:08.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.688+0000 7f46ff400640 10 client.27050 handle_command_reply: tid=0 2024-03-21T10:44:08.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.688+0000 7f47271fd740 1 client.27050 shutdown 2024-03-21T10:44:08.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.688+0000 7f47271fd740 1 -- 172.21.15.31:0/882747661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224af20 msgr2=0x562f122539b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:08.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.688+0000 7f47271fd740 1 --2- 172.21.15.31:0/882747661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224af20 0x562f122539b0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f470c011d10 tx=0x7f470c009970 comp rx=0 tx=0).stop 2024-03-21T10:44:08.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.688+0000 7f47271fd740 1 -- 172.21.15.31:0/882747661 shutdown_connections 2024-03-21T10:44:08.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.688+0000 7f47271fd740 1 --2- 172.21.15.31:0/882747661 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562f12264260 0x562f12266670 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.688+0000 7f47271fd740 1 --2- 172.21.15.31:0/882747661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f12254290 0x562f12251280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.688+0000 7f47271fd740 1 --2- 172.21.15.31:0/882747661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562f1224bc00 0x562f12250d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.688+0000 7f47271fd740 1 --2- 172.21.15.31:0/882747661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224af20 0x562f122539b0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.688+0000 7f47271fd740 1 -- 172.21.15.31:0/882747661 >> 172.21.15.31:0/882747661 conn(0x562f12212200 msgr2=0x562f122476c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:08.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.689+0000 7f46ff400640 0 client.27050 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:08.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.689+0000 7f47271fd740 1 -- 172.21.15.31:0/882747661 shutdown_connections 2024-03-21T10:44:08.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.689+0000 7f47271fd740 1 -- 172.21.15.31:0/882747661 wait complete. 2024-03-21T10:44:08.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.690+0000 7f47271fd740 20 client.27050 trim_cache size 0 max 16384 2024-03-21T10:44:08.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.730+0000 7f47271fd740 1 Processor -- start 2024-03-21T10:44:08.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.731+0000 7f47271fd740 1 -- start start 2024-03-21T10:44:08.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.731+0000 7f47271fd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562f1224af20 0x562f12244910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.731+0000 7f47271fd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224bc00 0x562f12253470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.731+0000 7f47271fd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f12254290 0x562f122539b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.731+0000 7f47271fd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4700012190 con 0x562f1224af20 2024-03-21T10:44:08.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.731+0000 7f47271fd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470000fc90 con 0x562f1224bc00 2024-03-21T10:44:08.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.731+0000 7f47271fd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470000dcf0 con 0x562f12254290 2024-03-21T10:44:08.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.732+0000 7f4721e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224bc00 0x562f12253470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.732+0000 7f4721e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224bc00 0x562f12253470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:57264/0 (socket says 172.21.15.31:57264) 2024-03-21T10:44:08.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.732+0000 7f4721e00640 1 -- 172.21.15.31:0/634809754 learned_addr learned my addr 172.21.15.31:0/634809754 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:08.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.732+0000 7f4721400640 1 --2- 172.21.15.31:0/634809754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f12254290 0x562f122539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.732+0000 7f4721400640 1 --2- 172.21.15.31:0/634809754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562f1224af20 0x562f12244910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.733+0000 7f4721e00640 1 -- 172.21.15.31:0/634809754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f12254290 msgr2=0x562f122539b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:08.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.733+0000 7f4721e00640 1 --2- 172.21.15.31:0/634809754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f12254290 0x562f122539b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.733+0000 7f4721e00640 1 -- 172.21.15.31:0/634809754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562f1224af20 msgr2=0x562f12244910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:08.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.733+0000 7f4721e00640 1 --2- 172.21.15.31:0/634809754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562f1224af20 0x562f12244910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.733+0000 7f4721e00640 1 -- 172.21.15.31:0/634809754 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562f11e830f0 con 0x562f1224bc00 2024-03-21T10:44:08.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.733+0000 7f4721400640 1 --2- 172.21.15.31:0/634809754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562f1224af20 0x562f12244910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:44:08.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.734+0000 7f4721e00640 1 --2- 172.21.15.31:0/634809754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224bc00 0x562f12253470 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f471002c240 tx=0x7f47100095f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee45af2108c64915 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:08.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.734+0000 7f470a600640 1 -- 172.21.15.31:0/634809754 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471004d070 con 0x562f1224bc00 2024-03-21T10:44:08.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.734+0000 7f470a600640 1 -- 172.21.15.31:0/634809754 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f471002c880 con 0x562f1224bc00 2024-03-21T10:44:08.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.734+0000 7f470a600640 1 -- 172.21.15.31:0/634809754 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471004d070 con 0x562f1224bc00 2024-03-21T10:44:08.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.735+0000 7f47271fd740 1 -- 172.21.15.31:0/634809754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224bc00 msgr2=0x562f12253470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:08.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.735+0000 7f47271fd740 1 --2- 172.21.15.31:0/634809754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224bc00 0x562f12253470 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f471002c240 tx=0x7f47100095f0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.737+0000 7f47271fd740 1 -- 172.21.15.31:0/634809754 shutdown_connections 2024-03-21T10:44:08.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.737+0000 7f47271fd740 1 --2- 172.21.15.31:0/634809754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f12254290 0x562f122539b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.737+0000 7f47271fd740 1 --2- 172.21.15.31:0/634809754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224bc00 0x562f12253470 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.737+0000 7f47271fd740 1 --2- 172.21.15.31:0/634809754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562f1224af20 0x562f12244910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.737+0000 7f47271fd740 1 -- 172.21.15.31:0/634809754 >> 172.21.15.31:0/634809754 conn(0x562f12212200 msgr2=0x562f12244200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:08.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.737+0000 7f47271fd740 1 -- 172.21.15.31:0/634809754 shutdown_connections 2024-03-21T10:44:08.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.737+0000 7f47271fd740 1 -- 172.21.15.31:0/634809754 wait complete. 2024-03-21T10:44:08.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.738+0000 7f47271fd740 1 -- start start 2024-03-21T10:44:08.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.738+0000 7f47271fd740 1 Processor -- start 2024-03-21T10:44:08.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.739+0000 7f47271fd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562f1224af20 0x562f1224e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.739+0000 7f47271fd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224bc00 0x562f1225f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.739+0000 7f47271fd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f12254290 0x562f1225f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.739+0000 7f47271fd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470000e310 con 0x562f1224af20 2024-03-21T10:44:08.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.739+0000 7f47271fd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470000f630 con 0x562f1224bc00 2024-03-21T10:44:08.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.739+0000 7f47271fd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470000e970 con 0x562f12254290 2024-03-21T10:44:08.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.740+0000 7f47271fd740 10 client.0 _ll_register_callbacks cb 0x7f470c009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:08.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.740+0000 7f4721400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562f1224af20 0x562f1224e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.740+0000 7f4721e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224bc00 0x562f1225f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.740+0000 7f4721e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224bc00 0x562f1225f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:57272/0 (socket says 172.21.15.31:57272) 2024-03-21T10:44:08.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.740+0000 7f4721e00640 1 -- 172.21.15.31:0/317111538 learned_addr learned my addr 172.21.15.31:0/317111538 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:08.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.740+0000 7f4721400640 1 --2- 172.21.15.31:0/317111538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f12254290 0x562f1225f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.741+0000 7f4721e00640 1 -- 172.21.15.31:0/317111538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f12254290 msgr2=0x562f1225f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:08.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.741+0000 7f4721e00640 1 --2- 172.21.15.31:0/317111538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f12254290 0x562f1225f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.741+0000 7f4721e00640 1 -- 172.21.15.31:0/317111538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562f1224af20 msgr2=0x562f1224e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:08.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.741+0000 7f4721e00640 1 --2- 172.21.15.31:0/317111538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562f1224af20 0x562f1224e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.741+0000 7f4721e00640 1 -- 172.21.15.31:0/317111538 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562f1224ede0 con 0x562f1224bc00 2024-03-21T10:44:08.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.742+0000 7f4721e00640 1 --2- 172.21.15.31:0/317111538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224bc00 0x562f1225f250 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x562f12246790 tx=0x7f4710005700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42aa9fe9952800fa server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:08.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.742+0000 7f46ff400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:44:08.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.742+0000 7f46ff400640 1 -- 172.21.15.31:0/317111538 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471004d070 con 0x562f1224bc00 2024-03-21T10:44:08.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.742+0000 7f46ff400640 1 -- 172.21.15.31:0/317111538 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f4710004230 con 0x562f1224bc00 2024-03-21T10:44:08.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.742+0000 7f47271fd740 1 -- 172.21.15.31:0/317111538 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f471002c880 con 0x562f1224bc00 2024-03-21T10:44:08.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.742+0000 7f46ff400640 1 -- 172.21.15.31:0/317111538 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4710044040 con 0x562f1224bc00 2024-03-21T10:44:08.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.743+0000 7f47271fd740 10 client.27056 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:08.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.743+0000 7f47271fd740 1 -- 172.21.15.31:0/317111538 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x562f122c0660 con 0x562f1224bc00 2024-03-21T10:44:08.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.744+0000 7f46ff400640 1 -- 172.21.15.31:0/317111538 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f471002c880 con 0x562f1224bc00 2024-03-21T10:44:08.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.744+0000 7f46ff400640 1 -- 172.21.15.31:0/317111538 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f4710040470 con 0x562f1224bc00 2024-03-21T10:44:08.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.745+0000 7f47271fd740 10 client.27056 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:08.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.745+0000 7f47271fd740 10 client.27056 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:44:08.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.745+0000 7f47271fd740 20 client.27056 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:08.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.745+0000 7f47271fd740 1 --2- 172.21.15.31:0/317111538 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562f122c08e0 0x562f122c2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:08.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.745+0000 7f47271fd740 4 client.27056 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:44:08.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.745+0000 7f47271fd740 1 -- 172.21.15.31:0/317111538 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562f121b6ae0 con 0x562f122c08e0 2024-03-21T10:44:08.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.745+0000 7f4721400640 1 --2- 172.21.15.31:0/317111538 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562f122c08e0 0x562f122c2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:08.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.746+0000 7f4721400640 1 --2- 172.21.15.31:0/317111538 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562f122c08e0 0x562f122c2cd0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46f8337d06192887 server_cookie=9cf8afaaa1c9b4fd in_seq=0 out_seq=0 2024-03-21T10:44:08.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.746+0000 7f46ff400640 10 client.27056 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:08.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.746+0000 7f46ff400640 1 -- 172.21.15.31:0/317111538 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x562f121b6ae0 con 0x562f122c08e0 2024-03-21T10:44:08.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.746+0000 7f46ff400640 10 client.27056 handle_command_reply: tid=0 2024-03-21T10:44:08.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.747+0000 7f47271fd740 1 client.27056 shutdown 2024-03-21T10:44:08.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.747+0000 7f47271fd740 1 -- 172.21.15.31:0/317111538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224bc00 msgr2=0x562f1225f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:08.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.747+0000 7f47271fd740 1 --2- 172.21.15.31:0/317111538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224bc00 0x562f1225f250 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x562f12246790 tx=0x7f4710005700 comp rx=0 tx=0).stop 2024-03-21T10:44:08.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.747+0000 7f47271fd740 1 -- 172.21.15.31:0/317111538 shutdown_connections 2024-03-21T10:44:08.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.747+0000 7f47271fd740 1 --2- 172.21.15.31:0/317111538 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562f122c08e0 0x562f122c2cd0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.747+0000 7f47271fd740 1 --2- 172.21.15.31:0/317111538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562f12254290 0x562f1225f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.747+0000 7f47271fd740 1 --2- 172.21.15.31:0/317111538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562f1224bc00 0x562f1225f250 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.747+0000 7f47271fd740 1 --2- 172.21.15.31:0/317111538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562f1224af20 0x562f1224e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.747+0000 7f46ff400640 0 client.27056 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:08.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.747+0000 7f47271fd740 1 -- 172.21.15.31:0/317111538 >> 172.21.15.31:0/317111538 conn(0x562f12212200 msgr2=0x562f1224fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:08.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.748+0000 7f47271fd740 1 -- 172.21.15.31:0/317111538 shutdown_connections 2024-03-21T10:44:08.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.749+0000 7f47271fd740 1 -- 172.21.15.31:0/317111538 wait complete. 2024-03-21T10:44:08.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.749+0000 7f47271fd740 20 client.27056 trim_cache size 0 max 16384 2024-03-21T10:44:08.748 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:44:08.748 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-21T10:44:08.748 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-03-21T10:44:08.748 INFO:teuthology.orchestra.run.smithi031.stdout: "0df2c906-e04c-4df4-97f4-5d11c0338140": { 2024-03-21T10:44:08.748 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-03-21T10:44:08.748 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "0df2c906-e04c-4df4-97f4-5d11c0338140", 2024-03-21T10:44:08.748 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-03-21T10:44:08.748 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:44:08.749 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:44:08.749 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:44:08.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.751+0000 7f4724800640 1 -- 172.21.15.31:0/2432457568 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f47040a3310 msgr2=0x7f47040a57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:08.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.751+0000 7f4724800640 1 --2- 172.21.15.31:0/2432457568 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f47040a3310 0x7f47040a57d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4718098a60 tx=0x7f4718004b50 comp rx=0 tx=0).stop 2024-03-21T10:44:08.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.752+0000 7f4724800640 1 -- 172.21.15.31:0/2432457568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f471c109870 msgr2=0x7f471c10fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:08.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.752+0000 7f4724800640 1 --2- 172.21.15.31:0/2432457568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f471c109870 0x7f471c10fcb0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f471800be20 tx=0x7f471800bf10 comp rx=0 tx=0).stop 2024-03-21T10:44:08.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.753+0000 7f4724800640 1 -- 172.21.15.31:0/2432457568 shutdown_connections 2024-03-21T10:44:08.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.753+0000 7f4724800640 1 --2- 172.21.15.31:0/2432457568 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f47040a3310 0x7f47040a57d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.753+0000 7f4724800640 1 --2- 172.21.15.31:0/2432457568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f471c109870 0x7f471c10fcb0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.753+0000 7f4724800640 1 --2- 172.21.15.31:0/2432457568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f471c104220 0x7f471c10f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.753+0000 7f4724800640 1 --2- 172.21.15.31:0/2432457568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f471c103860 0x7f471c10f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:08.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.753+0000 7f4724800640 1 -- 172.21.15.31:0/2432457568 >> 172.21.15.31:0/2432457568 conn(0x7f471c0ff310 msgr2=0x7f471c0ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:08.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.753+0000 7f4724800640 1 -- 172.21.15.31:0/2432457568 shutdown_connections 2024-03-21T10:44:08.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:08.757+0000 7f4724800640 1 -- 172.21.15.31:0/2432457568 wait complete. 2024-03-21T10:44:08.785 INFO:tasks.cephfs.filesystem:scrub status for tag:0df2c906-e04c-4df4-97f4-5d11c0338140 - {'path': '/', 'tag': '0df2c906-e04c-4df4-97f4-5d11c0338140', 'options': 'recursive,force'} 2024-03-21T10:44:09.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:08 smithi031 ceph-mon[31673]: pgmap v170: 129 pgs: 129 active+clean; 35 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 210 MiB/s wr, 1.24k op/s 2024-03-21T10:44:09.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:08 smithi031 ceph-mon[31673]: scrub queued for path: / 2024-03-21T10:44:09.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:08 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:44:09.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:08 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:44:09.055 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:08 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:44:09.056 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:08 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:44:09.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:08 smithi040 ceph-mon[36154]: pgmap v170: 129 pgs: 129 active+clean; 35 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 210 MiB/s wr, 1.24k op/s 2024-03-21T10:44:09.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:08 smithi040 ceph-mon[36154]: scrub queued for path: / 2024-03-21T10:44:09.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:08 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:44:09.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:08 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:44:09.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:08 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:44:09.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:08 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:44:09.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:08 smithi070 ceph-mon[36330]: pgmap v170: 129 pgs: 129 active+clean; 35 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 210 MiB/s wr, 1.24k op/s 2024-03-21T10:44:09.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:08 smithi070 ceph-mon[36330]: scrub queued for path: / 2024-03-21T10:44:09.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:08 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:44:09.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:08 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:44:09.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:08 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:44:09.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:08 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:44:10.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:09 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:44:10.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:09 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:44:10.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:09 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:44:10.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.709+0000 7fa7bf200640 1 -- 172.21.15.31:0/1804861289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7b409be10 msgr2=0x7fa7b409c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:10.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.709+0000 7fa7bf200640 1 --2- 172.21.15.31:0/1804861289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7b409be10 0x7fa7b409c1f0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8009900 tx=0x7fa7a802bc30 comp rx=0 tx=0).stop 2024-03-21T10:44:10.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.711+0000 7fa7bf200640 1 -- 172.21.15.31:0/1804861289 shutdown_connections 2024-03-21T10:44:10.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.711+0000 7fa7bf200640 1 --2- 172.21.15.31:0/1804861289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b4005670 0x7fa7b40a2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.711+0000 7fa7bf200640 1 --2- 172.21.15.31:0/1804861289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7b4004cb0 0x7fa7b4005130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.711+0000 7fa7bf200640 1 --2- 172.21.15.31:0/1804861289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7b409be10 0x7fa7b409c1f0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.711+0000 7fa7bf200640 1 -- 172.21.15.31:0/1804861289 >> 172.21.15.31:0/1804861289 conn(0x7fa7b400a740 msgr2=0x7fa7b400ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:10.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.711+0000 7fa7bf200640 1 -- 172.21.15.31:0/1804861289 shutdown_connections 2024-03-21T10:44:10.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.711+0000 7fa7bf200640 1 -- 172.21.15.31:0/1804861289 wait complete. 2024-03-21T10:44:10.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.713+0000 7fa7bf200640 1 Processor -- start 2024-03-21T10:44:10.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.713+0000 7fa7bf200640 1 -- start start 2024-03-21T10:44:10.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.713+0000 7fa7bf200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7b4004cb0 0x7fa7b40ab440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:10.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.713+0000 7fa7bf200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7b4005670 0x7fa7b40ab980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:10.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.713+0000 7fa7bf200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b409be10 0x7fa7b40a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:10.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.714+0000 7fa7bde00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7b4004cb0 0x7fa7b40ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:10.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.714+0000 7fa7bde00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7b4004cb0 0x7fa7b40ab440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:45802/0 (socket says 172.21.15.31:45802) 2024-03-21T10:44:10.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.714+0000 7fa7bde00640 1 -- 172.21.15.31:0/507095545 learned_addr learned my addr 172.21.15.31:0/507095545 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:10.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.714+0000 7fa7bf200640 1 -- 172.21.15.31:0/507095545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b40a3ee0 con 0x7fa7b409be10 2024-03-21T10:44:10.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.714+0000 7fa7bf200640 1 -- 172.21.15.31:0/507095545 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b40a3d60 con 0x7fa7b4005670 2024-03-21T10:44:10.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.714+0000 7fa7bf200640 1 -- 172.21.15.31:0/507095545 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b40a4060 con 0x7fa7b4004cb0 2024-03-21T10:44:10.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.714+0000 7fa7be800640 1 --2- 172.21.15.31:0/507095545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b409be10 0x7fa7b40a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:10.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.714+0000 7fa7bd400640 1 --2- 172.21.15.31:0/507095545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7b4005670 0x7fa7b40ab980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:10.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.715+0000 7fa7bd400640 1 -- 172.21.15.31:0/507095545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7b4004cb0 msgr2=0x7fa7b40ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:10.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.715+0000 7fa7bd400640 1 --2- 172.21.15.31:0/507095545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7b4004cb0 0x7fa7b40ab440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.715+0000 7fa7bd400640 1 -- 172.21.15.31:0/507095545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b409be10 msgr2=0x7fa7b40a55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:10.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.715+0000 7fa7bd400640 1 --2- 172.21.15.31:0/507095545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b409be10 0x7fa7b40a55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.715+0000 7fa7bd400640 1 -- 172.21.15.31:0/507095545 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b40a5e70 con 0x7fa7b4005670 2024-03-21T10:44:10.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.715+0000 7fa7bd400640 1 --2- 172.21.15.31:0/507095545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7b4005670 0x7fa7b40ab980 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fa7b804f6d0 tx=0x7fa7b807ca70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:10.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.716+0000 7fa7b2a00640 1 -- 172.21.15.31:0/507095545 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b8082070 con 0x7fa7b4005670 2024-03-21T10:44:10.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.716+0000 7fa7bde00640 1 --2- 172.21.15.31:0/507095545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7b4004cb0 0x7fa7b40ab440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:44:10.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.716+0000 7fa7b2a00640 1 -- 172.21.15.31:0/507095545 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7b807ebe0 con 0x7fa7b4005670 2024-03-21T10:44:10.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.716+0000 7fa7be800640 1 --2- 172.21.15.31:0/507095545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b409be10 0x7fa7b40a55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:44:10.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.716+0000 7fa7b2a00640 1 -- 172.21.15.31:0/507095545 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b8082070 con 0x7fa7b4005670 2024-03-21T10:44:10.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.716+0000 7fa7bf200640 1 -- 172.21.15.31:0/507095545 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b40a6160 con 0x7fa7b4005670 2024-03-21T10:44:10.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.716+0000 7fa7bf200640 1 -- 172.21.15.31:0/507095545 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b4147ec0 con 0x7fa7b4005670 2024-03-21T10:44:10.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.725+0000 7fa7b2a00640 1 -- 172.21.15.31:0/507095545 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fa7b8085070 con 0x7fa7b4005670 2024-03-21T10:44:10.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.726+0000 7fa7c35de740 1 Processor -- start 2024-03-21T10:44:10.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.726+0000 7fa7c35de740 1 -- start start 2024-03-21T10:44:10.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.726+0000 7fa7c35de740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bcf63b1a0 0x562bcf63b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:10.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.726+0000 7fa7c35de740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bcf63bac0 0x562bcf644350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:10.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.726+0000 7fa7c35de740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bcf644890 0x562bcf646c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:10.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.726+0000 7fa7c35de740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bcf5fd3b0 con 0x562bcf63b1a0 2024-03-21T10:44:10.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.726+0000 7fa7c35de740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bcf59f830 con 0x562bcf644890 2024-03-21T10:44:10.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.726+0000 7fa7c35de740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bcf5485d0 con 0x562bcf63bac0 2024-03-21T10:44:10.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.727+0000 7fa7bde00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bcf63bac0 0x562bcf644350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:10.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.727+0000 7fa7bde00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bcf63bac0 0x562bcf644350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:45812/0 (socket says 172.21.15.31:45812) 2024-03-21T10:44:10.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.727+0000 7fa7bde00640 1 -- 172.21.15.31:0/1147538729 learned_addr learned my addr 172.21.15.31:0/1147538729 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:10.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.727+0000 7fa7bd400640 1 --2- 172.21.15.31:0/1147538729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bcf644890 0x562bcf646c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:10.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.727+0000 7fa7be800640 1 --2- 172.21.15.31:0/1147538729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bcf63b1a0 0x562bcf63b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.728+0000 7fa7bde00640 1 -- 172.21.15.31:0/1147538729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bcf644890 msgr2=0x562bcf646c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.728+0000 7fa7bde00640 1 --2- 172.21.15.31:0/1147538729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bcf644890 0x562bcf646c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.728+0000 7fa7bde00640 1 -- 172.21.15.31:0/1147538729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bcf63b1a0 msgr2=0x562bcf63b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.728+0000 7fa7bde00640 1 --2- 172.21.15.31:0/1147538729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bcf63b1a0 0x562bcf63b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.728+0000 7fa7bde00640 1 -- 172.21.15.31:0/1147538729 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562bcf5602c0 con 0x562bcf63bac0 2024-03-21T10:44:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.729+0000 7fa7be800640 1 --2- 172.21.15.31:0/1147538729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bcf63b1a0 0x562bcf63b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:44:10.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.729+0000 7fa7bde00640 1 --2- 172.21.15.31:0/1147538729 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bcf63bac0 0x562bcf644350 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa7a802efd0 tx=0x7fa7a802c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=561071e84b0c4596 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:10.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.730+0000 7fa7b2a00640 1 --2- 172.21.15.31:0/507095545 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa7a00a3540 0x7fa7a00a5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:10.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.730+0000 7fa7b2a00640 1 -- 172.21.15.31:0/507095545 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fa7b8077040 con 0x7fa7b4005670 2024-03-21T10:44:10.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.732+0000 7fa7a7000640 1 -- 172.21.15.31:0/1147538729 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a8041020 con 0x562bcf63bac0 2024-03-21T10:44:10.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.732+0000 7fa7be800640 1 --2- 172.21.15.31:0/507095545 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa7a00a3540 0x7fa7a00a5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:10.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.732+0000 7fa7a7000640 1 -- 172.21.15.31:0/1147538729 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7a8004240 con 0x562bcf63bac0 2024-03-21T10:44:10.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.732+0000 7fa7a7000640 1 -- 172.21.15.31:0/1147538729 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a8004700 con 0x562bcf63bac0 2024-03-21T10:44:10.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.733+0000 7fa7c35de740 1 -- 172.21.15.31:0/1147538729 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bcf63bac0 msgr2=0x562bcf644350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:10.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.733+0000 7fa7c35de740 1 --2- 172.21.15.31:0/1147538729 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bcf63bac0 0x562bcf644350 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa7a802efd0 tx=0x7fa7a802c660 comp rx=0 tx=0).stop 2024-03-21T10:44:10.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.733+0000 7fa7c35de740 1 -- 172.21.15.31:0/1147538729 shutdown_connections 2024-03-21T10:44:10.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.733+0000 7fa7c35de740 1 --2- 172.21.15.31:0/1147538729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bcf644890 0x562bcf646c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.733+0000 7fa7c35de740 1 --2- 172.21.15.31:0/1147538729 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bcf63bac0 0x562bcf644350 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.733+0000 7fa7c35de740 1 --2- 172.21.15.31:0/1147538729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bcf63b1a0 0x562bcf63b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.733+0000 7fa7c35de740 1 -- 172.21.15.31:0/1147538729 >> 172.21.15.31:0/1147538729 conn(0x562bcf635ac0 msgr2=0x562bcf635ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.733+0000 7fa7c35de740 1 -- 172.21.15.31:0/1147538729 shutdown_connections 2024-03-21T10:44:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.734+0000 7fa7c35de740 1 -- 172.21.15.31:0/1147538729 wait complete. 2024-03-21T10:44:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.734+0000 7fa7c35de740 1 -- start start 2024-03-21T10:44:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.734+0000 7fa7c35de740 1 Processor -- start 2024-03-21T10:44:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.735+0000 7fa7be800640 1 --2- 172.21.15.31:0/507095545 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa7a00a3540 0x7fa7a00a5a00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac005d80 tx=0x7fa7ac005ed0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:10.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.736+0000 7fa7c35de740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bcf63b1a0 0x562bcf641850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:10.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.736+0000 7fa7c35de740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bcf63bac0 0x562bcf641d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:10.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.736+0000 7fa7c35de740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bcf644890 0x562bcf6422d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:10.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.736+0000 7fa7c35de740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bcf55ba20 con 0x562bcf63b1a0 2024-03-21T10:44:10.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.736+0000 7fa7c35de740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bcf59b8d0 con 0x562bcf644890 2024-03-21T10:44:10.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.736+0000 7fa7c35de740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bcf647460 con 0x562bcf63bac0 2024-03-21T10:44:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.736+0000 7fa7c35de740 10 client.0 _ll_register_callbacks cb 0x562bcf5981c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.736+0000 7fa7bde00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bcf63b1a0 0x562bcf641850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.736+0000 7fa7be800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bcf644890 0x562bcf6422d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.736+0000 7fa7bd400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bcf63bac0 0x562bcf641d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.736+0000 7fa7bd400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bcf63bac0 0x562bcf641d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:45820/0 (socket says 172.21.15.31:45820) 2024-03-21T10:44:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.736+0000 7fa7bd400640 1 -- 172.21.15.31:0/3700638504 learned_addr learned my addr 172.21.15.31:0/3700638504 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.737+0000 7fa7bde00640 1 -- 172.21.15.31:0/3700638504 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bcf63bac0 msgr2=0x562bcf641d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.737+0000 7fa7bde00640 1 --2- 172.21.15.31:0/3700638504 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bcf63bac0 0x562bcf641d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.737+0000 7fa7bde00640 1 -- 172.21.15.31:0/3700638504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bcf644890 msgr2=0x562bcf6422d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.737+0000 7fa7bde00640 1 --2- 172.21.15.31:0/3700638504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bcf644890 0x562bcf6422d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.737+0000 7fa7bde00640 1 -- 172.21.15.31:0/3700638504 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562bcf2b4210 con 0x562bcf63b1a0 2024-03-21T10:44:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.737+0000 7fa7be800640 1 --2- 172.21.15.31:0/3700638504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bcf644890 0x562bcf6422d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:44:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.738+0000 7fa7bde00640 1 --2- 172.21.15.31:0/3700638504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bcf63b1a0 0x562bcf641850 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8036040 tx=0x7fa7a8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0e57f601288bafc server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:10.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.738+0000 7fa79b400640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:44:10.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.739+0000 7fa79b400640 1 -- 172.21.15.31:0/3700638504 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a8041050 con 0x562bcf63b1a0 2024-03-21T10:44:10.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.739+0000 7fa79b400640 1 -- 172.21.15.31:0/3700638504 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fa7a80048c0 con 0x562bcf63b1a0 2024-03-21T10:44:10.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.739+0000 7fa7c35de740 1 -- 172.21.15.31:0/3700638504 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa7a8004240 con 0x562bcf63b1a0 2024-03-21T10:44:10.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.740+0000 7fa79b400640 1 -- 172.21.15.31:0/3700638504 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7a8004ac0 con 0x562bcf63b1a0 2024-03-21T10:44:10.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.740+0000 7fa7c35de740 10 client.16548 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:10.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.740+0000 7fa7c35de740 1 -- 172.21.15.31:0/3700638504 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x562bcf380620 con 0x562bcf63b1a0 2024-03-21T10:44:10.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.741+0000 7fa79b400640 1 -- 172.21.15.31:0/3700638504 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7a8004240 con 0x562bcf63b1a0 2024-03-21T10:44:10.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.741+0000 7fa79b400640 1 -- 172.21.15.31:0/3700638504 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa7a804faf0 con 0x562bcf63b1a0 2024-03-21T10:44:10.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.741+0000 7fa7c35de740 10 client.16548 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:10.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.741+0000 7fa7c35de740 10 client.16548 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:44:10.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.741+0000 7fa7c35de740 20 client.16548 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:10.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.742+0000 7fa7c35de740 1 --2- 172.21.15.31:0/3700638504 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562bcf6549f0 0x562bcf656e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:10.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.742+0000 7fa7c35de740 4 client.16548 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:44:10.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.742+0000 7fa7c35de740 1 -- 172.21.15.31:0/3700638504 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562bcf5acd70 con 0x562bcf6549f0 2024-03-21T10:44:10.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.742+0000 7fa7bd400640 1 --2- 172.21.15.31:0/3700638504 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562bcf6549f0 0x562bcf656e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:10.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.742+0000 7fa79b400640 10 client.16548 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:10.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.743+0000 7fa7bd400640 1 --2- 172.21.15.31:0/3700638504 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562bcf6549f0 0x562bcf656e60 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2361a0c32fa6f75 server_cookie=c9e689aa551928ba in_seq=0 out_seq=0 2024-03-21T10:44:10.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.743+0000 7fa79b400640 1 -- 172.21.15.31:0/3700638504 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562bcf5acd70 con 0x562bcf6549f0 2024-03-21T10:44:10.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.743+0000 7fa79b400640 10 client.16548 handle_command_reply: tid=0 2024-03-21T10:44:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.743+0000 7fa7c35de740 1 client.16548 shutdown 2024-03-21T10:44:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.744+0000 7fa7c35de740 1 -- 172.21.15.31:0/3700638504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bcf63b1a0 msgr2=0x562bcf641850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.744+0000 7fa7c35de740 1 --2- 172.21.15.31:0/3700638504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bcf63b1a0 0x562bcf641850 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8036040 tx=0x7fa7a8009260 comp rx=0 tx=0).stop 2024-03-21T10:44:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.744+0000 7fa7c35de740 1 -- 172.21.15.31:0/3700638504 shutdown_connections 2024-03-21T10:44:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.744+0000 7fa7c35de740 1 --2- 172.21.15.31:0/3700638504 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562bcf6549f0 0x562bcf656e60 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.744+0000 7fa7c35de740 1 --2- 172.21.15.31:0/3700638504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bcf644890 0x562bcf6422d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.744+0000 7fa7c35de740 1 --2- 172.21.15.31:0/3700638504 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bcf63bac0 0x562bcf641d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.744+0000 7fa7c35de740 1 --2- 172.21.15.31:0/3700638504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bcf63b1a0 0x562bcf641850 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.744+0000 7fa7c35de740 1 -- 172.21.15.31:0/3700638504 >> 172.21.15.31:0/3700638504 conn(0x562bcf635ac0 msgr2=0x562bcf637810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.744+0000 7fa79b400640 0 client.16548 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.744+0000 7fa7c35de740 1 -- 172.21.15.31:0/3700638504 shutdown_connections 2024-03-21T10:44:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.745+0000 7fa7c35de740 1 -- 172.21.15.31:0/3700638504 wait complete. 2024-03-21T10:44:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.745+0000 7fa7c35de740 20 client.16548 trim_cache size 0 max 16384 2024-03-21T10:44:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.771+0000 7fa7c35de740 1 Processor -- start 2024-03-21T10:44:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.772+0000 7fa7c35de740 1 -- start start 2024-03-21T10:44:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.772+0000 7fa7c35de740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bcf63bac0 0x562bcf643f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.772+0000 7fa7c35de740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bcf644890 0x562bcf64fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.773+0000 7fa7c35de740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bcf63d220 0x562bcf650110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.773+0000 7fa7bde00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bcf63bac0 0x562bcf643f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.773+0000 7fa7bde00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bcf63bac0 0x562bcf643f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50434/0 (socket says 172.21.15.31:50434) 2024-03-21T10:44:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.773+0000 7fa7c35de740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b4154400 con 0x562bcf63bac0 2024-03-21T10:44:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.773+0000 7fa7c35de740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b4151f00 con 0x562bcf644890 2024-03-21T10:44:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.773+0000 7fa7c35de740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b414ffa0 con 0x562bcf63d220 2024-03-21T10:44:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.773+0000 7fa7bde00640 1 -- 172.21.15.31:0/209214696 learned_addr learned my addr 172.21.15.31:0/209214696 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.774+0000 7fa7be800640 1 --2- 172.21.15.31:0/209214696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bcf63d220 0x562bcf650110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.774+0000 7fa7bde00640 1 -- 172.21.15.31:0/209214696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bcf63d220 msgr2=0x562bcf650110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.774+0000 7fa7bde00640 1 --2- 172.21.15.31:0/209214696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bcf63d220 0x562bcf650110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.774+0000 7fa7bde00640 1 -- 172.21.15.31:0/209214696 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bcf644890 msgr2=0x562bcf64fbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:44:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.774+0000 7fa7bde00640 1 --2- 172.21.15.31:0/209214696 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bcf644890 0x562bcf64fbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.774+0000 7fa7bde00640 1 -- 172.21.15.31:0/209214696 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562bcf63a970 con 0x562bcf63bac0 2024-03-21T10:44:10.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.774+0000 7fa7bde00640 1 --2- 172.21.15.31:0/209214696 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bcf63bac0 0x562bcf643f80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa7a804fec0 tx=0x7fa7a8005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77b4f2193232cae8 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:10.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.775+0000 7fa7a6600640 1 -- 172.21.15.31:0/209214696 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a8041070 con 0x562bcf63bac0 2024-03-21T10:44:10.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.776+0000 7fa7a6600640 1 -- 172.21.15.31:0/209214696 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7a804a070 con 0x562bcf63bac0 2024-03-21T10:44:10.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.776+0000 7fa7a6600640 1 -- 172.21.15.31:0/209214696 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a803d850 con 0x562bcf63bac0 2024-03-21T10:44:10.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.777+0000 7fa7c35de740 1 -- 172.21.15.31:0/209214696 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bcf63bac0 msgr2=0x562bcf643f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:10.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.777+0000 7fa7c35de740 1 --2- 172.21.15.31:0/209214696 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bcf63bac0 0x562bcf643f80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa7a804fec0 tx=0x7fa7a8005c90 comp rx=0 tx=0).stop 2024-03-21T10:44:10.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.778+0000 7fa7c35de740 1 -- 172.21.15.31:0/209214696 shutdown_connections 2024-03-21T10:44:10.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.778+0000 7fa7c35de740 1 --2- 172.21.15.31:0/209214696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bcf63d220 0x562bcf650110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.778+0000 7fa7c35de740 1 --2- 172.21.15.31:0/209214696 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bcf644890 0x562bcf64fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.778+0000 7fa7c35de740 1 --2- 172.21.15.31:0/209214696 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bcf63bac0 0x562bcf643f80 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.778+0000 7fa7c35de740 1 -- 172.21.15.31:0/209214696 >> 172.21.15.31:0/209214696 conn(0x562bcf635ac0 msgr2=0x562bcf63cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:10.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.778+0000 7fa7c35de740 1 -- 172.21.15.31:0/209214696 shutdown_connections 2024-03-21T10:44:10.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.778+0000 7fa7c35de740 1 -- 172.21.15.31:0/209214696 wait complete. 2024-03-21T10:44:10.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.779+0000 7fa7c35de740 1 -- start start 2024-03-21T10:44:10.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.780+0000 7fa7c35de740 1 Processor -- start 2024-03-21T10:44:10.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.781+0000 7fa7c35de740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bcf63bac0 0x562bcf648270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:10.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.781+0000 7fa7c35de740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bcf63d220 0x562bcf6487b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:10.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.781+0000 7fa7c35de740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bcf644890 0x562bcf6af300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:10.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.781+0000 7fa7c35de740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b41505a0 con 0x562bcf63bac0 2024-03-21T10:44:10.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.781+0000 7fa7c35de740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b41518a0 con 0x562bcf63d220 2024-03-21T10:44:10.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.781+0000 7fa7c35de740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b4150be0 con 0x562bcf644890 2024-03-21T10:44:10.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.781+0000 7fa7c35de740 10 client.0 _ll_register_callbacks cb 0x7fa7a8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.782+0000 7fa7be800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bcf644890 0x562bcf6af300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.782+0000 7fa7be800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bcf644890 0x562bcf6af300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:45838/0 (socket says 172.21.15.31:45838) 2024-03-21T10:44:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.782+0000 7fa7be800640 1 -- 172.21.15.31:0/4248942496 learned_addr learned my addr 172.21.15.31:0/4248942496 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.782+0000 7fa7bd400640 1 --2- 172.21.15.31:0/4248942496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bcf63d220 0x562bcf6487b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.782+0000 7fa7bde00640 1 --2- 172.21.15.31:0/4248942496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bcf63bac0 0x562bcf648270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.783+0000 7fa7bd400640 1 -- 172.21.15.31:0/4248942496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bcf644890 msgr2=0x562bcf6af300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.783+0000 7fa7bd400640 1 --2- 172.21.15.31:0/4248942496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bcf644890 0x562bcf6af300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.783+0000 7fa7bd400640 1 -- 172.21.15.31:0/4248942496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bcf63bac0 msgr2=0x562bcf648270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:10.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.783+0000 7fa7bd400640 1 --2- 172.21.15.31:0/4248942496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bcf63bac0 0x562bcf648270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.783+0000 7fa7bd400640 1 -- 172.21.15.31:0/4248942496 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562bcf6af890 con 0x562bcf63d220 2024-03-21T10:44:10.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.784+0000 7fa7bd400640 1 --2- 172.21.15.31:0/4248942496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bcf63d220 0x562bcf6487b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8103cd0 tx=0x7fa7b807c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdb5d56d62cd211 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.787+0000 7fa79b400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:44:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.787+0000 7fa79b400640 1 -- 172.21.15.31:0/4248942496 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b808c040 con 0x562bcf63d220 2024-03-21T10:44:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.787+0000 7fa79b400640 1 -- 172.21.15.31:0/4248942496 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fa7b813e660 con 0x562bcf63d220 2024-03-21T10:44:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.788+0000 7fa7c35de740 1 -- 172.21.15.31:0/4248942496 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa7a804a070 con 0x562bcf63d220 2024-03-21T10:44:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.788+0000 7fa79b400640 1 -- 172.21.15.31:0/4248942496 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7b8082040 con 0x562bcf63d220 2024-03-21T10:44:10.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.789+0000 7fa7c35de740 10 client.27068 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:10.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.789+0000 7fa7c35de740 1 -- 172.21.15.31:0/4248942496 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x562bcf6b0bd0 con 0x562bcf63d220 2024-03-21T10:44:10.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.789+0000 7fa79b400640 1 -- 172.21.15.31:0/4248942496 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7a804a070 con 0x562bcf63d220 2024-03-21T10:44:10.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.790+0000 7fa79b400640 1 -- 172.21.15.31:0/4248942496 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa7b8138850 con 0x562bcf63d220 2024-03-21T10:44:10.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.790+0000 7fa7c35de740 10 client.27068 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:10.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.790+0000 7fa7c35de740 10 client.27068 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:44:10.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.790+0000 7fa7c35de740 20 client.27068 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:10.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.791+0000 7fa7c35de740 1 --2- 172.21.15.31:0/4248942496 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562bcf6b12e0 0x562bcf6b3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:10.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.791+0000 7fa7c35de740 4 client.27068 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:44:10.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.791+0000 7fa7c35de740 1 -- 172.21.15.31:0/4248942496 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562bcf5a70c0 con 0x562bcf6b12e0 2024-03-21T10:44:10.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.791+0000 7fa7bde00640 1 --2- 172.21.15.31:0/4248942496 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562bcf6b12e0 0x562bcf6b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:10.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.792+0000 7fa7bde00640 1 --2- 172.21.15.31:0/4248942496 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562bcf6b12e0 0x562bcf6b3700 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f266fbe544af0b9f server_cookie=ed93e8d0dce8053f in_seq=0 out_seq=0 2024-03-21T10:44:10.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.792+0000 7fa79b400640 10 client.27068 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:10.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.813+0000 7fa79b400640 1 -- 172.21.15.31:0/4248942496 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562bcf5a70c0 con 0x562bcf6b12e0 2024-03-21T10:44:10.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.813+0000 7fa79b400640 10 client.27068 handle_command_reply: tid=0 2024-03-21T10:44:10.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.814+0000 7fa7c35de740 1 client.27068 shutdown 2024-03-21T10:44:10.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.814+0000 7fa7c35de740 1 -- 172.21.15.31:0/4248942496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bcf63d220 msgr2=0x562bcf6487b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:10.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.814+0000 7fa7c35de740 1 --2- 172.21.15.31:0/4248942496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bcf63d220 0x562bcf6487b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8103cd0 tx=0x7fa7b807c560 comp rx=0 tx=0).stop 2024-03-21T10:44:10.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.814+0000 7fa7c35de740 1 -- 172.21.15.31:0/4248942496 shutdown_connections 2024-03-21T10:44:10.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.814+0000 7fa7c35de740 1 --2- 172.21.15.31:0/4248942496 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562bcf6b12e0 0x562bcf6b3700 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.814+0000 7fa7c35de740 1 --2- 172.21.15.31:0/4248942496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bcf644890 0x562bcf6af300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.814+0000 7fa7c35de740 1 --2- 172.21.15.31:0/4248942496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562bcf63d220 0x562bcf6487b0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.814+0000 7fa7c35de740 1 --2- 172.21.15.31:0/4248942496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562bcf63bac0 0x562bcf648270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.814+0000 7fa7c35de740 1 -- 172.21.15.31:0/4248942496 >> 172.21.15.31:0/4248942496 conn(0x562bcf635ac0 msgr2=0x562bcf634290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:10.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.814+0000 7fa79b400640 0 client.27068 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:10.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.815+0000 7fa7c35de740 1 -- 172.21.15.31:0/4248942496 shutdown_connections 2024-03-21T10:44:10.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.815+0000 7fa7c35de740 1 -- 172.21.15.31:0/4248942496 wait complete. 2024-03-21T10:44:10.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.815+0000 7fa7c35de740 20 client.27068 trim_cache size 0 max 16384 2024-03-21T10:44:10.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:44:10.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:44:10.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:44:10.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:44:10.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.817+0000 7fa7bf200640 1 -- 172.21.15.31:0/507095545 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa7a00a3540 msgr2=0x7fa7a00a5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:10.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.817+0000 7fa7bf200640 1 --2- 172.21.15.31:0/507095545 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa7a00a3540 0x7fa7a00a5a00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac005d80 tx=0x7fa7ac005ed0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.817+0000 7fa7bf200640 1 -- 172.21.15.31:0/507095545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7b4005670 msgr2=0x7fa7b40ab980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:10.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.817+0000 7fa7bf200640 1 --2- 172.21.15.31:0/507095545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7b4005670 0x7fa7b40ab980 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fa7b804f6d0 tx=0x7fa7b807ca70 comp rx=0 tx=0).stop 2024-03-21T10:44:10.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.817+0000 7fa7bf200640 1 -- 172.21.15.31:0/507095545 shutdown_connections 2024-03-21T10:44:10.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.817+0000 7fa7bf200640 1 --2- 172.21.15.31:0/507095545 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa7a00a3540 0x7fa7a00a5a00 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.817+0000 7fa7bf200640 1 --2- 172.21.15.31:0/507095545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b409be10 0x7fa7b40a55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.817+0000 7fa7bf200640 1 --2- 172.21.15.31:0/507095545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7b4005670 0x7fa7b40ab980 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.817+0000 7fa7bf200640 1 --2- 172.21.15.31:0/507095545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7b4004cb0 0x7fa7b40ab440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:10.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.817+0000 7fa7bf200640 1 -- 172.21.15.31:0/507095545 >> 172.21.15.31:0/507095545 conn(0x7fa7b400a740 msgr2=0x7fa7b4093b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:10.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.817+0000 7fa7bf200640 1 -- 172.21.15.31:0/507095545 shutdown_connections 2024-03-21T10:44:10.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:10.818+0000 7fa7bf200640 1 -- 172.21.15.31:0/507095545 wait complete. 2024-03-21T10:44:11.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:10 smithi040 ceph-mon[36154]: pgmap v171: 129 pgs: 129 active+clean; 35 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 242 MiB/s wr, 1.49k op/s 2024-03-21T10:44:11.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:10 smithi031 ceph-mon[31673]: pgmap v171: 129 pgs: 129 active+clean; 35 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 242 MiB/s wr, 1.49k op/s 2024-03-21T10:44:11.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:10 smithi070 ceph-mon[36330]: pgmap v171: 129 pgs: 129 active+clean; 35 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 242 MiB/s wr, 1.49k op/s 2024-03-21T10:44:13.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:12 smithi031 ceph-mon[31673]: pgmap v172: 129 pgs: 129 active+clean; 35 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 216 MiB/s wr, 1.40k op/s 2024-03-21T10:44:13.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:12 smithi070 ceph-mon[36330]: pgmap v172: 129 pgs: 129 active+clean; 35 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 216 MiB/s wr, 1.40k op/s 2024-03-21T10:44:13.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:12 smithi040 ceph-mon[36154]: pgmap v172: 129 pgs: 129 active+clean; 35 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 216 MiB/s wr, 1.40k op/s 2024-03-21T10:44:15.308 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:14 smithi031 ceph-mon[31673]: pgmap v173: 129 pgs: 129 active+clean; 35 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 194 MiB/s wr, 1.28k op/s 2024-03-21T10:44:15.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:14 smithi070 ceph-mon[36330]: pgmap v173: 129 pgs: 129 active+clean; 35 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 194 MiB/s wr, 1.28k op/s 2024-03-21T10:44:15.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:14 smithi040 ceph-mon[36154]: pgmap v173: 129 pgs: 129 active+clean; 35 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 194 MiB/s wr, 1.28k op/s 2024-03-21T10:44:17.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:17 smithi031 ceph-mon[31673]: pgmap v174: 129 pgs: 129 active+clean; 35 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 260 MiB/s wr, 1.81k op/s 2024-03-21T10:44:17.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:17 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:44:17.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:17 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:44:17.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:17 smithi070 ceph-mon[36330]: pgmap v174: 129 pgs: 129 active+clean; 35 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 260 MiB/s wr, 1.81k op/s 2024-03-21T10:44:17.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:17 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:44:17.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:17 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:44:17.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:17 smithi040 ceph-mon[36154]: pgmap v174: 129 pgs: 129 active+clean; 35 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 260 MiB/s wr, 1.81k op/s 2024-03-21T10:44:17.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:17 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:44:17.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:17 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:44:18.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:18 smithi031 ceph-mon[31673]: pgmap v175: 129 pgs: 129 active+clean; 35 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 191 MiB/s wr, 1.39k op/s 2024-03-21T10:44:18.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:18 smithi070 ceph-mon[36330]: pgmap v175: 129 pgs: 129 active+clean; 35 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 191 MiB/s wr, 1.39k op/s 2024-03-21T10:44:18.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:18 smithi040 ceph-mon[36154]: pgmap v175: 129 pgs: 129 active+clean; 35 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 191 MiB/s wr, 1.39k op/s 2024-03-21T10:44:21.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:20 smithi031 ceph-mon[31673]: pgmap v176: 129 pgs: 129 active+clean; 36 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 248 MiB/s wr, 1.85k op/s 2024-03-21T10:44:21.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:20 smithi070 ceph-mon[36330]: pgmap v176: 129 pgs: 129 active+clean; 36 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 248 MiB/s wr, 1.85k op/s 2024-03-21T10:44:21.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:20 smithi040 ceph-mon[36154]: pgmap v176: 129 pgs: 129 active+clean; 36 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 248 MiB/s wr, 1.85k op/s 2024-03-21T10:44:23.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:22 smithi031 ceph-mon[31673]: pgmap v177: 129 pgs: 129 active+clean; 36 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 216 MiB/s wr, 1.63k op/s 2024-03-21T10:44:23.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:22 smithi070 ceph-mon[36330]: pgmap v177: 129 pgs: 129 active+clean; 36 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 216 MiB/s wr, 1.63k op/s 2024-03-21T10:44:23.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:22 smithi040 ceph-mon[36154]: pgmap v177: 129 pgs: 129 active+clean; 36 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 216 MiB/s wr, 1.63k op/s 2024-03-21T10:44:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.057+0000 7f15dca00640 1 -- 172.21.15.31:0/377015250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15d81147f0 msgr2=0x7f15d8116be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.057+0000 7f15dca00640 1 --2- 172.21.15.31:0/377015250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15d81147f0 0x7f15d8116be0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f15c4009900 tx=0x7f15c402bc60 comp rx=0 tx=0).stop 2024-03-21T10:44:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.057+0000 7f15dca00640 1 -- 172.21.15.31:0/377015250 shutdown_connections 2024-03-21T10:44:24.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.057+0000 7f15dca00640 1 --2- 172.21.15.31:0/377015250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15d81147f0 0x7f15d8116be0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.057+0000 7f15dca00640 1 --2- 172.21.15.31:0/377015250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15d81091a0 0x7f15d81142b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.057+0000 7f15dca00640 1 --2- 172.21.15.31:0/377015250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15d8108880 0x7f15d8108c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.057+0000 7f15dca00640 1 -- 172.21.15.31:0/377015250 >> 172.21.15.31:0/377015250 conn(0x7f15d806c750 msgr2=0x7f15d806cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:24.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.057+0000 7f15dca00640 1 -- 172.21.15.31:0/377015250 shutdown_connections 2024-03-21T10:44:24.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.058+0000 7f15dca00640 1 -- 172.21.15.31:0/377015250 wait complete. 2024-03-21T10:44:24.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.059+0000 7f15dca00640 1 Processor -- start 2024-03-21T10:44:24.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.062+0000 7f15dca00640 1 -- start start 2024-03-21T10:44:24.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.062+0000 7f15dca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15d8108880 0x7f15d810a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:24.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.062+0000 7f15dca00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15d81091a0 0x7f15d810a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.063+0000 7f15dca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15d81147f0 0x7f15d810ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.063+0000 7f15dca00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d811a260 con 0x7f15d8108880 2024-03-21T10:44:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.063+0000 7f15dca00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d811a0e0 con 0x7f15d81147f0 2024-03-21T10:44:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.063+0000 7f15dca00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d811a3e0 con 0x7f15d81091a0 2024-03-21T10:44:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.063+0000 7f15d5400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15d81091a0 0x7f15d810a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.063+0000 7f15d5400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15d81091a0 0x7f15d810a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:33912/0 (socket says 172.21.15.31:33912) 2024-03-21T10:44:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.063+0000 7f15d5400640 1 -- 172.21.15.31:0/1965074394 learned_addr learned my addr 172.21.15.31:0/1965074394 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.063+0000 7f15d5e00640 1 --2- 172.21.15.31:0/1965074394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15d8108880 0x7f15d810a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.063+0000 7f15d6800640 1 --2- 172.21.15.31:0/1965074394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15d81147f0 0x7f15d810ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.064+0000 7f15d5400640 1 -- 172.21.15.31:0/1965074394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15d81147f0 msgr2=0x7f15d810ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.064+0000 7f15d5400640 1 --2- 172.21.15.31:0/1965074394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15d81147f0 0x7f15d810ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.064+0000 7f15d5400640 1 -- 172.21.15.31:0/1965074394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15d8108880 msgr2=0x7f15d810a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.064+0000 7f15d5400640 1 --2- 172.21.15.31:0/1965074394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15d8108880 0x7f15d810a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.064+0000 7f15d5400640 1 -- 172.21.15.31:0/1965074394 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d81aec60 con 0x7f15d81091a0 2024-03-21T10:44:24.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.064+0000 7f15d5e00640 1 --2- 172.21.15.31:0/1965074394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15d8108880 0x7f15d810a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:44:24.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.064+0000 7f15d5400640 1 --2- 172.21.15.31:0/1965074394 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15d81091a0 0x7f15d810a650 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f15d000cce0 tx=0x7f15d000a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:24.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.065+0000 7f15cea00640 1 -- 172.21.15.31:0/1965074394 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d0004b20 con 0x7f15d81091a0 2024-03-21T10:44:24.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.065+0000 7f15cea00640 1 -- 172.21.15.31:0/1965074394 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15d0004cc0 con 0x7f15d81091a0 2024-03-21T10:44:24.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.065+0000 7f15dca00640 1 -- 172.21.15.31:0/1965074394 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15d81aef50 con 0x7f15d81091a0 2024-03-21T10:44:24.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.065+0000 7f15cea00640 1 -- 172.21.15.31:0/1965074394 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d000f7c0 con 0x7f15d81091a0 2024-03-21T10:44:24.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.065+0000 7f15dca00640 1 -- 172.21.15.31:0/1965074394 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15d81af460 con 0x7f15d81091a0 2024-03-21T10:44:24.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.076+0000 7f15cea00640 1 -- 172.21.15.31:0/1965074394 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f15d000f960 con 0x7f15d81091a0 2024-03-21T10:44:24.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.077+0000 7f15df60a740 1 Processor -- start 2024-03-21T10:44:24.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.077+0000 7f15df60a740 1 -- start start 2024-03-21T10:44:24.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.078+0000 7f15df60a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56161a492fd0 0x56161a4933b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:24.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.079+0000 7f15d6800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56161a492fd0 0x56161a4933b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:24.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.079+0000 7f15d6800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56161a492fd0 0x56161a4933b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45614/0 (socket says 172.21.15.31:45614) 2024-03-21T10:44:24.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.079+0000 7f15d6800640 1 -- 172.21.15.31:0/1998022558 learned_addr learned my addr 172.21.15.31:0/1998022558 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:24.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.079+0000 7f15df60a740 1 --2- 172.21.15.31:0/1998022558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56161a4938f0 0x56161a4a4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:24.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.080+0000 7f15df60a740 1 --2- 172.21.15.31:0/1998022558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56161a4a4ed0 0x56161a4a72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:24.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.080+0000 7f15df60a740 1 -- 172.21.15.31:0/1998022558 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56161a4553b0 con 0x56161a492fd0 2024-03-21T10:44:24.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.080+0000 7f15df60a740 1 -- 172.21.15.31:0/1998022558 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56161a3f7830 con 0x56161a4938f0 2024-03-21T10:44:24.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.080+0000 7f15df60a740 1 -- 172.21.15.31:0/1998022558 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56161a3a05d0 con 0x56161a4a4ed0 2024-03-21T10:44:24.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.080+0000 7f15d5e00640 1 --2- 172.21.15.31:0/1998022558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56161a4938f0 0x56161a4a4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:24.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.081+0000 7f15d5400640 1 --2- 172.21.15.31:0/1998022558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56161a4a4ed0 0x56161a4a72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:24.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.081+0000 7f15d6800640 1 -- 172.21.15.31:0/1998022558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56161a4a4ed0 msgr2=0x56161a4a72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:24.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.081+0000 7f15d6800640 1 --2- 172.21.15.31:0/1998022558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56161a4a4ed0 0x56161a4a72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.081+0000 7f15d6800640 1 -- 172.21.15.31:0/1998022558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56161a4938f0 msgr2=0x56161a4a4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:24.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.081+0000 7f15d6800640 1 --2- 172.21.15.31:0/1998022558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56161a4938f0 0x56161a4a4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.081+0000 7f15d6800640 1 -- 172.21.15.31:0/1998022558 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56161a3b82c0 con 0x56161a492fd0 2024-03-21T10:44:24.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.081+0000 7f15d5e00640 1 --2- 172.21.15.31:0/1998022558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56161a4938f0 0x56161a4a4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:44:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.082+0000 7f15cea00640 1 --2- 172.21.15.31:0/1965074394 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f15bc0a31d0 0x7f15bc0a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.082+0000 7f15cea00640 1 -- 172.21.15.31:0/1965074394 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f15d00cfe90 con 0x7f15d81091a0 2024-03-21T10:44:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.082+0000 7f15d5400640 1 --2- 172.21.15.31:0/1998022558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56161a4a4ed0 0x56161a4a72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:44:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.082+0000 7f15d5e00640 1 --2- 172.21.15.31:0/1965074394 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f15bc0a31d0 0x7f15bc0a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.083+0000 7f15d6800640 1 --2- 172.21.15.31:0/1998022558 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56161a492fd0 0x56161a4933b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f15c4009900 tx=0x7f15c4004980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fb7abf648178d00 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:24.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.083+0000 7f15c3000640 1 -- 172.21.15.31:0/1998022558 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c4043070 con 0x56161a492fd0 2024-03-21T10:44:24.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.083+0000 7f15c3000640 1 -- 172.21.15.31:0/1998022558 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15c4004a70 con 0x56161a492fd0 2024-03-21T10:44:24.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.083+0000 7f15c3000640 1 -- 172.21.15.31:0/1998022558 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c4005360 con 0x56161a492fd0 2024-03-21T10:44:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.084+0000 7f15df60a740 1 -- 172.21.15.31:0/1998022558 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56161a492fd0 msgr2=0x56161a4933b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.084+0000 7f15df60a740 1 --2- 172.21.15.31:0/1998022558 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56161a492fd0 0x56161a4933b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f15c4009900 tx=0x7f15c4004980 comp rx=0 tx=0).stop 2024-03-21T10:44:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.085+0000 7f15d5e00640 1 --2- 172.21.15.31:0/1965074394 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f15bc0a31d0 0x7f15bc0a5690 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x56161a492a80 tx=0x7f15c8008040 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.085+0000 7f15df60a740 1 -- 172.21.15.31:0/1998022558 shutdown_connections 2024-03-21T10:44:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.085+0000 7f15df60a740 1 --2- 172.21.15.31:0/1998022558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56161a4a4ed0 0x56161a4a72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.085+0000 7f15df60a740 1 --2- 172.21.15.31:0/1998022558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56161a4938f0 0x56161a4a4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.085+0000 7f15df60a740 1 --2- 172.21.15.31:0/1998022558 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56161a492fd0 0x56161a4933b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.085+0000 7f15df60a740 1 -- 172.21.15.31:0/1998022558 >> 172.21.15.31:0/1998022558 conn(0x56161a48d8f0 msgr2=0x56161a48dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.086+0000 7f15df60a740 1 -- 172.21.15.31:0/1998022558 shutdown_connections 2024-03-21T10:44:24.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.086+0000 7f15df60a740 1 -- 172.21.15.31:0/1998022558 wait complete. 2024-03-21T10:44:24.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.087+0000 7f15df60a740 1 -- start start 2024-03-21T10:44:24.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.087+0000 7f15df60a740 1 Processor -- start 2024-03-21T10:44:24.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.091+0000 7f15df60a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56161a492fd0 0x56161a4a1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:24.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.091+0000 7f15df60a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56161a4938f0 0x56161a4a2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:24.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.092+0000 7f15df60a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56161a4a4ed0 0x56161a4a2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:24.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.092+0000 7f15d5400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56161a4938f0 0x56161a4a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:24.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.092+0000 7f15d6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56161a4a4ed0 0x56161a4a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:24.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.092+0000 7f15d6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56161a4a4ed0 0x56161a4a2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:34452/0 (socket says 172.21.15.31:34452) 2024-03-21T10:44:24.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.092+0000 7f15d6800640 1 -- 172.21.15.31:0/288796269 learned_addr learned my addr 172.21.15.31:0/288796269 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:24.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.092+0000 7f15d5e00640 1 --2- 172.21.15.31:0/288796269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56161a492fd0 0x56161a4a1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:24.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.093+0000 7f15df60a740 1 -- 172.21.15.31:0/288796269 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56161a3b3a20 con 0x56161a4938f0 2024-03-21T10:44:24.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.093+0000 7f15df60a740 1 -- 172.21.15.31:0/288796269 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56161a3f38d0 con 0x56161a4a4ed0 2024-03-21T10:44:24.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.093+0000 7f15df60a740 1 -- 172.21.15.31:0/288796269 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56161a4a7aa0 con 0x56161a492fd0 2024-03-21T10:44:24.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.093+0000 7f15df60a740 10 client.0 _ll_register_callbacks cb 0x56161a3f01c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:24.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.094+0000 7f15d5400640 1 -- 172.21.15.31:0/288796269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56161a492fd0 msgr2=0x56161a4a1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:24.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.094+0000 7f15d5400640 1 --2- 172.21.15.31:0/288796269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56161a492fd0 0x56161a4a1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.094+0000 7f15d5400640 1 -- 172.21.15.31:0/288796269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56161a4a4ed0 msgr2=0x56161a4a2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:24.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.094+0000 7f15d5400640 1 --2- 172.21.15.31:0/288796269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56161a4a4ed0 0x56161a4a2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.094+0000 7f15d5400640 1 -- 172.21.15.31:0/288796269 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56161a10c210 con 0x56161a4938f0 2024-03-21T10:44:24.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.094+0000 7f15d6800640 1 --2- 172.21.15.31:0/288796269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56161a4a4ed0 0x56161a4a2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:44:24.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.094+0000 7f15d5e00640 1 --2- 172.21.15.31:0/288796269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56161a492fd0 0x56161a4a1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:44:24.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.096+0000 7f15d5400640 1 --2- 172.21.15.31:0/288796269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56161a4938f0 0x56161a4a2440 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f15d00cd210 tx=0x7f15d009ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19a7731b1a4eb46b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:24.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.096+0000 7f15b7400640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:44:24.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.096+0000 7f15b7400640 1 -- 172.21.15.31:0/288796269 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d001d040 con 0x56161a4938f0 2024-03-21T10:44:24.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.096+0000 7f15b7400640 1 -- 172.21.15.31:0/288796269 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f15d00a4460 con 0x56161a4938f0 2024-03-21T10:44:24.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.096+0000 7f15df60a740 1 -- 172.21.15.31:0/288796269 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f15c4004a70 con 0x56161a4938f0 2024-03-21T10:44:24.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.097+0000 7f15b7400640 1 -- 172.21.15.31:0/288796269 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15d009bee0 con 0x56161a4938f0 2024-03-21T10:44:24.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.098+0000 7f15df60a740 10 client.16563 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:24.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.098+0000 7f15df60a740 1 -- 172.21.15.31:0/288796269 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x56161a1d8620 con 0x56161a4938f0 2024-03-21T10:44:24.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.098+0000 7f15b7400640 1 -- 172.21.15.31:0/288796269 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15c4004a70 con 0x56161a4938f0 2024-03-21T10:44:24.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.103+0000 7f15b7400640 1 -- 172.21.15.31:0/288796269 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f15d009ada0 con 0x56161a4938f0 2024-03-21T10:44:24.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.103+0000 7f15df60a740 10 client.16563 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:24.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.103+0000 7f15df60a740 10 client.16563 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:44:24.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.103+0000 7f15df60a740 20 client.16563 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:24.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.103+0000 7f15df60a740 1 --2- 172.21.15.31:0/288796269 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56161a4b4d90 0x56161a4b7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:24.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.103+0000 7f15df60a740 4 client.16563 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:44:24.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.104+0000 7f15df60a740 1 -- 172.21.15.31:0/288796269 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56161a404d70 con 0x56161a4b4d90 2024-03-21T10:44:24.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.104+0000 7f15d5e00640 1 --2- 172.21.15.31:0/288796269 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56161a4b4d90 0x56161a4b7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:24.103 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.106+0000 7f15d5e00640 1 --2- 172.21.15.31:0/288796269 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56161a4b4d90 0x56161a4b7180 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dba927690e139128 server_cookie=57a9d4ba3ef44dee in_seq=0 out_seq=0 2024-03-21T10:44:24.103 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.106+0000 7f15b7400640 10 client.16563 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:24.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.107+0000 7f15b7400640 1 -- 172.21.15.31:0/288796269 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56161a404d70 con 0x56161a4b4d90 2024-03-21T10:44:24.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.107+0000 7f15b7400640 10 client.16563 handle_command_reply: tid=0 2024-03-21T10:44:24.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.108+0000 7f15df60a740 1 client.16563 shutdown 2024-03-21T10:44:24.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.109+0000 7f15df60a740 1 -- 172.21.15.31:0/288796269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56161a4938f0 msgr2=0x56161a4a2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:24.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.109+0000 7f15df60a740 1 --2- 172.21.15.31:0/288796269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56161a4938f0 0x56161a4a2440 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f15d00cd210 tx=0x7f15d009ba20 comp rx=0 tx=0).stop 2024-03-21T10:44:24.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.113+0000 7f15df60a740 1 -- 172.21.15.31:0/288796269 shutdown_connections 2024-03-21T10:44:24.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.113+0000 7f15df60a740 1 --2- 172.21.15.31:0/288796269 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56161a4b4d90 0x56161a4b7180 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.113+0000 7f15df60a740 1 --2- 172.21.15.31:0/288796269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56161a4a4ed0 0x56161a4a2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.113+0000 7f15df60a740 1 --2- 172.21.15.31:0/288796269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56161a4938f0 0x56161a4a2440 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.113+0000 7f15df60a740 1 --2- 172.21.15.31:0/288796269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56161a492fd0 0x56161a4a1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.113+0000 7f15b7400640 0 client.16563 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:24.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.113+0000 7f15df60a740 1 -- 172.21.15.31:0/288796269 >> 172.21.15.31:0/288796269 conn(0x56161a48d8f0 msgr2=0x56161a48f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:24.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.113+0000 7f15df60a740 1 -- 172.21.15.31:0/288796269 shutdown_connections 2024-03-21T10:44:24.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.113+0000 7f15df60a740 1 -- 172.21.15.31:0/288796269 wait complete. 2024-03-21T10:44:24.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.113+0000 7f15df60a740 20 client.16563 trim_cache size 0 max 16384 2024-03-21T10:44:24.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.143+0000 7f15df60a740 1 Processor -- start 2024-03-21T10:44:24.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.144+0000 7f15df60a740 1 -- start start 2024-03-21T10:44:24.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.144+0000 7f15df60a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56161a492fd0 0x56161a4a4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:24.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.144+0000 7f15df60a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56161a4938f0 0x56161a4b7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:24.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.144+0000 7f15df60a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56161a4a4ed0 0x56161a48c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:24.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.144+0000 7f15df60a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d81b7470 con 0x56161a4938f0 2024-03-21T10:44:24.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.144+0000 7f15df60a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d81b4f70 con 0x56161a492fd0 2024-03-21T10:44:24.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.144+0000 7f15df60a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d81b3010 con 0x56161a4a4ed0 2024-03-21T10:44:24.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.144+0000 7f15d6800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56161a4a4ed0 0x56161a48c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:24.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.144+0000 7f15d6800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56161a4a4ed0 0x56161a48c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:33950/0 (socket says 172.21.15.31:33950) 2024-03-21T10:44:24.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.144+0000 7f15d6800640 1 -- 172.21.15.31:0/805860093 learned_addr learned my addr 172.21.15.31:0/805860093 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:24.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.145+0000 7f15d5e00640 1 --2- 172.21.15.31:0/805860093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56161a492fd0 0x56161a4a4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:24.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.145+0000 7f15d6800640 1 -- 172.21.15.31:0/805860093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56161a492fd0 msgr2=0x56161a4a4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:24.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.145+0000 7f15d6800640 1 --2- 172.21.15.31:0/805860093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56161a492fd0 0x56161a4a4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.145+0000 7f15d6800640 1 -- 172.21.15.31:0/805860093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56161a4938f0 msgr2=0x56161a4b7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:24.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.145+0000 7f15d6800640 1 --2- 172.21.15.31:0/805860093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56161a4938f0 0x56161a4b7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.145+0000 7f15d6800640 1 -- 172.21.15.31:0/805860093 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56161a4b7760 con 0x56161a4a4ed0 2024-03-21T10:44:24.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.145+0000 7f15d6800640 1 --2- 172.21.15.31:0/805860093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56161a4a4ed0 0x56161a48c7d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f15c4009850 tx=0x7f15c4004f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6fd02185d38f411 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:24.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.146+0000 7f15c2600640 1 -- 172.21.15.31:0/805860093 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c4043030 con 0x56161a4a4ed0 2024-03-21T10:44:24.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.146+0000 7f15c2600640 1 -- 172.21.15.31:0/805860093 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15c40053f0 con 0x56161a4a4ed0 2024-03-21T10:44:24.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.146+0000 7f15c2600640 1 -- 172.21.15.31:0/805860093 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c403f6b0 con 0x56161a4a4ed0 2024-03-21T10:44:24.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.147+0000 7f15df60a740 1 -- 172.21.15.31:0/805860093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56161a4a4ed0 msgr2=0x56161a48c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:24.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.147+0000 7f15df60a740 1 --2- 172.21.15.31:0/805860093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56161a4a4ed0 0x56161a48c7d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f15c4009850 tx=0x7f15c4004f40 comp rx=0 tx=0).stop 2024-03-21T10:44:24.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.147+0000 7f15df60a740 1 -- 172.21.15.31:0/805860093 shutdown_connections 2024-03-21T10:44:24.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.147+0000 7f15df60a740 1 --2- 172.21.15.31:0/805860093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56161a4a4ed0 0x56161a48c7d0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.147+0000 7f15df60a740 1 --2- 172.21.15.31:0/805860093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56161a4938f0 0x56161a4b7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.147+0000 7f15df60a740 1 --2- 172.21.15.31:0/805860093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56161a492fd0 0x56161a4a4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.147+0000 7f15df60a740 1 -- 172.21.15.31:0/805860093 >> 172.21.15.31:0/805860093 conn(0x56161a48d8f0 msgr2=0x56161a48c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:24.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.147+0000 7f15df60a740 1 -- 172.21.15.31:0/805860093 shutdown_connections 2024-03-21T10:44:24.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.148+0000 7f15df60a740 1 -- 172.21.15.31:0/805860093 wait complete. 2024-03-21T10:44:24.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.148+0000 7f15df60a740 1 -- start start 2024-03-21T10:44:24.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.149+0000 7f15df60a740 1 Processor -- start 2024-03-21T10:44:24.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.150+0000 7f15df60a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56161a492fd0 0x56161a4a83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:24.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.151+0000 7f15df60a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56161a4938f0 0x56161a4a88f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:24.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.151+0000 7f15df60a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56161a4a4ed0 0x56161a4a8e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:24.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.151+0000 7f15df60a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d81b3610 con 0x56161a492fd0 2024-03-21T10:44:24.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.151+0000 7f15df60a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d81b4910 con 0x56161a4938f0 2024-03-21T10:44:24.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.151+0000 7f15df60a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d81b3c50 con 0x56161a4a4ed0 2024-03-21T10:44:24.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.151+0000 7f15df60a740 10 client.0 _ll_register_callbacks cb 0x7f15d0098180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:24.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.151+0000 7f15d5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56161a492fd0 0x56161a4a83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:24.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.151+0000 7f15d5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56161a492fd0 0x56161a4a83b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45644/0 (socket says 172.21.15.31:45644) 2024-03-21T10:44:24.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.151+0000 7f15d5e00640 1 -- 172.21.15.31:0/1367753666 learned_addr learned my addr 172.21.15.31:0/1367753666 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:24.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.152+0000 7f15d6800640 1 --2- 172.21.15.31:0/1367753666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56161a4a4ed0 0x56161a4a8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.152+0000 7f15d5400640 1 --2- 172.21.15.31:0/1367753666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56161a4938f0 0x56161a4a88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.153+0000 7f15d5e00640 1 -- 172.21.15.31:0/1367753666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56161a4a4ed0 msgr2=0x56161a4a8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.153+0000 7f15d5e00640 1 --2- 172.21.15.31:0/1367753666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56161a4a4ed0 0x56161a4a8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.153+0000 7f15d5e00640 1 -- 172.21.15.31:0/1367753666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56161a4938f0 msgr2=0x56161a4a88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.153+0000 7f15d5e00640 1 --2- 172.21.15.31:0/1367753666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56161a4938f0 0x56161a4a88f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.153+0000 7f15d5e00640 1 -- 172.21.15.31:0/1367753666 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56161a50f850 con 0x56161a492fd0 2024-03-21T10:44:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.153+0000 7f15d5400640 1 --2- 172.21.15.31:0/1367753666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56161a4938f0 0x56161a4a88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:44:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.154+0000 7f15d5e00640 1 --2- 172.21.15.31:0/1367753666 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56161a492fd0 0x56161a4a83b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f15c8009fd0 tx=0x7f15c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc673deaea8a8916 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.154+0000 7f15b7400640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:44:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.154+0000 7f15b7400640 1 -- 172.21.15.31:0/1367753666 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c800f040 con 0x56161a492fd0 2024-03-21T10:44:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.154+0000 7f15b7400640 1 -- 172.21.15.31:0/1367753666 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f15c8027a80 con 0x56161a492fd0 2024-03-21T10:44:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.155+0000 7f15df60a740 1 -- 172.21.15.31:0/1367753666 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f15c40053f0 con 0x56161a492fd0 2024-03-21T10:44:24.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.156+0000 7f15b7400640 1 -- 172.21.15.31:0/1367753666 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15c8035050 con 0x56161a492fd0 2024-03-21T10:44:24.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.156+0000 7f15df60a740 10 client.16566 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:24.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.156+0000 7f15df60a740 1 -- 172.21.15.31:0/1367753666 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x56161a510bc0 con 0x56161a492fd0 2024-03-21T10:44:24.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.157+0000 7f15b7400640 1 -- 172.21.15.31:0/1367753666 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15c40053f0 con 0x56161a492fd0 2024-03-21T10:44:24.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.158+0000 7f15b7400640 1 -- 172.21.15.31:0/1367753666 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f15c8012030 con 0x56161a492fd0 2024-03-21T10:44:24.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.158+0000 7f15df60a740 10 client.16566 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:24.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.158+0000 7f15df60a740 10 client.16566 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:44:24.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.158+0000 7f15df60a740 20 client.16566 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:24.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.158+0000 7f15df60a740 1 --2- 172.21.15.31:0/1367753666 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56161a511290 0x56161a5136b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:24.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.158+0000 7f15df60a740 4 client.16566 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:44:24.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.158+0000 7f15df60a740 1 -- 172.21.15.31:0/1367753666 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56161a3ff0c0 con 0x56161a511290 2024-03-21T10:44:24.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.159+0000 7f15d5400640 1 --2- 172.21.15.31:0/1367753666 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56161a511290 0x56161a5136b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:24.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.160+0000 7f15d5400640 1 --2- 172.21.15.31:0/1367753666 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56161a511290 0x56161a5136b0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cadc91938259389 server_cookie=eee0a47e011fcd96 in_seq=0 out_seq=0 2024-03-21T10:44:24.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.161+0000 7f15b7400640 10 client.16566 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:24.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.349+0000 7f15b7400640 1 -- 172.21.15.31:0/1367753666 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56161a3ff0c0 con 0x56161a511290 2024-03-21T10:44:24.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.349+0000 7f15b7400640 10 client.16566 handle_command_reply: tid=0 2024-03-21T10:44:24.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.349+0000 7f15df60a740 1 client.16566 shutdown 2024-03-21T10:44:24.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.351+0000 7f15df60a740 1 -- 172.21.15.31:0/1367753666 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56161a492fd0 msgr2=0x56161a4a83b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:24.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.351+0000 7f15df60a740 1 --2- 172.21.15.31:0/1367753666 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56161a492fd0 0x56161a4a83b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f15c8009fd0 tx=0x7f15c80046c0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.352+0000 7f15df60a740 1 -- 172.21.15.31:0/1367753666 shutdown_connections 2024-03-21T10:44:24.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.352+0000 7f15df60a740 1 --2- 172.21.15.31:0/1367753666 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56161a511290 0x56161a5136b0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.352+0000 7f15b7400640 0 client.16566 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:24.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.352+0000 7f15df60a740 1 --2- 172.21.15.31:0/1367753666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56161a4a4ed0 0x56161a4a8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.352+0000 7f15df60a740 1 --2- 172.21.15.31:0/1367753666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56161a4938f0 0x56161a4a88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.352+0000 7f15df60a740 1 --2- 172.21.15.31:0/1367753666 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56161a492fd0 0x56161a4a83b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.352+0000 7f15df60a740 1 -- 172.21.15.31:0/1367753666 >> 172.21.15.31:0/1367753666 conn(0x56161a48d8f0 msgr2=0x56161a4a2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:24.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.353+0000 7f15df60a740 1 -- 172.21.15.31:0/1367753666 shutdown_connections 2024-03-21T10:44:24.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.353+0000 7f15df60a740 1 -- 172.21.15.31:0/1367753666 wait complete. 2024-03-21T10:44:24.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.354+0000 7f15df60a740 20 client.16566 trim_cache size 0 max 16384 2024-03-21T10:44:24.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:44:24.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:44:24.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:44:24.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:44:24.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.364+0000 7f15dca00640 1 -- 172.21.15.31:0/1965074394 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f15bc0a31d0 msgr2=0x7f15bc0a5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:24.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.364+0000 7f15dca00640 1 --2- 172.21.15.31:0/1965074394 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f15bc0a31d0 0x7f15bc0a5690 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x56161a492a80 tx=0x7f15c8008040 comp rx=0 tx=0).stop 2024-03-21T10:44:24.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.364+0000 7f15dca00640 1 -- 172.21.15.31:0/1965074394 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15d81091a0 msgr2=0x7f15d810a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:24.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.364+0000 7f15dca00640 1 --2- 172.21.15.31:0/1965074394 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15d81091a0 0x7f15d810a650 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f15d000cce0 tx=0x7f15d000a620 comp rx=0 tx=0).stop 2024-03-21T10:44:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.368+0000 7f15dca00640 1 -- 172.21.15.31:0/1965074394 shutdown_connections 2024-03-21T10:44:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.368+0000 7f15dca00640 1 --2- 172.21.15.31:0/1965074394 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f15bc0a31d0 0x7f15bc0a5690 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.368+0000 7f15dca00640 1 --2- 172.21.15.31:0/1965074394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15d81147f0 0x7f15d810ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.368+0000 7f15dca00640 1 --2- 172.21.15.31:0/1965074394 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15d81091a0 0x7f15d810a650 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.368+0000 7f15dca00640 1 --2- 172.21.15.31:0/1965074394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15d8108880 0x7f15d810a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.368+0000 7f15dca00640 1 -- 172.21.15.31:0/1965074394 >> 172.21.15.31:0/1965074394 conn(0x7f15d806c750 msgr2=0x7f15d8070300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.368+0000 7f15dca00640 1 -- 172.21.15.31:0/1965074394 shutdown_connections 2024-03-21T10:44:24.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:24.369+0000 7f15dca00640 1 -- 172.21.15.31:0/1965074394 wait complete. 2024-03-21T10:44:25.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:24 smithi031 ceph-mon[31673]: pgmap v178: 129 pgs: 129 active+clean; 36 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 205 MiB/s wr, 1.55k op/s 2024-03-21T10:44:25.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:24 smithi070 ceph-mon[36330]: pgmap v178: 129 pgs: 129 active+clean; 36 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 205 MiB/s wr, 1.55k op/s 2024-03-21T10:44:25.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:24 smithi040 ceph-mon[36154]: pgmap v178: 129 pgs: 129 active+clean; 36 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 205 MiB/s wr, 1.55k op/s 2024-03-21T10:44:27.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:26 smithi031 ceph-mon[31673]: pgmap v179: 129 pgs: 129 active+clean; 36 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 280 MiB/s wr, 2.12k op/s 2024-03-21T10:44:27.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:26 smithi070 ceph-mon[36330]: pgmap v179: 129 pgs: 129 active+clean; 36 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 280 MiB/s wr, 2.12k op/s 2024-03-21T10:44:27.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:26 smithi040 ceph-mon[36154]: pgmap v179: 129 pgs: 129 active+clean; 36 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 280 MiB/s wr, 2.12k op/s 2024-03-21T10:44:29.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:29 smithi031 ceph-mon[31673]: pgmap v180: 129 pgs: 129 active+clean; 36 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 219 MiB/s wr, 1.67k op/s 2024-03-21T10:44:29.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:29 smithi070 ceph-mon[36330]: pgmap v180: 129 pgs: 129 active+clean; 36 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 219 MiB/s wr, 1.67k op/s 2024-03-21T10:44:29.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:29 smithi040 ceph-mon[36154]: pgmap v180: 129 pgs: 129 active+clean; 36 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 219 MiB/s wr, 1.67k op/s 2024-03-21T10:44:31.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:31 smithi031 ceph-mon[31673]: pgmap v181: 129 pgs: 129 active+clean; 37 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 261 MiB/s wr, 2.02k op/s 2024-03-21T10:44:31.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:31 smithi070 ceph-mon[36330]: pgmap v181: 129 pgs: 129 active+clean; 37 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 261 MiB/s wr, 2.02k op/s 2024-03-21T10:44:31.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:31 smithi040 ceph-mon[36154]: pgmap v181: 129 pgs: 129 active+clean; 37 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 261 MiB/s wr, 2.02k op/s 2024-03-21T10:44:33.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:33 smithi031 ceph-mon[31673]: pgmap v182: 129 pgs: 129 active+clean; 37 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 198 MiB/s wr, 1.56k op/s 2024-03-21T10:44:33.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:33 smithi070 ceph-mon[36330]: pgmap v182: 129 pgs: 129 active+clean; 37 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 198 MiB/s wr, 1.56k op/s 2024-03-21T10:44:33.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:33 smithi040 ceph-mon[36154]: pgmap v182: 129 pgs: 129 active+clean; 37 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 198 MiB/s wr, 1.56k op/s 2024-03-21T10:44:35.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:35 smithi040 ceph-mon[36154]: pgmap v183: 129 pgs: 129 active+clean; 37 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 196 MiB/s wr, 1.55k op/s 2024-03-21T10:44:35.560 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:35 smithi031 ceph-mon[31673]: pgmap v183: 129 pgs: 129 active+clean; 37 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 196 MiB/s wr, 1.55k op/s 2024-03-21T10:44:35.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:35 smithi070 ceph-mon[36330]: pgmap v183: 129 pgs: 129 active+clean; 37 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 196 MiB/s wr, 1.55k op/s 2024-03-21T10:44:37.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:37 smithi040 ceph-mon[36154]: pgmap v184: 129 pgs: 129 active+clean; 38 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 253 MiB/s wr, 2.02k op/s 2024-03-21T10:44:37.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:37 smithi031 ceph-mon[31673]: pgmap v184: 129 pgs: 129 active+clean; 38 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 253 MiB/s wr, 2.02k op/s 2024-03-21T10:44:37.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:37 smithi070 ceph-mon[36330]: pgmap v184: 129 pgs: 129 active+clean; 38 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 253 MiB/s wr, 2.02k op/s 2024-03-21T10:44:37.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.644+0000 7f5452a00640 1 -- 172.21.15.31:0/486169528 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f544c10caf0 msgr2=0x7f544c10ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:37.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.644+0000 7f5452a00640 1 --2- 172.21.15.31:0/486169528 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f544c10caf0 0x7f544c10ced0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f5438009900 tx=0x7f543802cd40 comp rx=0 tx=0).stop 2024-03-21T10:44:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.647+0000 7f5452a00640 1 -- 172.21.15.31:0/486169528 shutdown_connections 2024-03-21T10:44:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.647+0000 7f5452a00640 1 --2- 172.21.15.31:0/486169528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f544c10e170 0x7f544c0798c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.647+0000 7f5452a00640 1 --2- 172.21.15.31:0/486169528 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f544c10d7d0 0x7f544c10dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.647+0000 7f5452a00640 1 --2- 172.21.15.31:0/486169528 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f544c10caf0 0x7f544c10ced0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.647+0000 7f5452a00640 1 -- 172.21.15.31:0/486169528 >> 172.21.15.31:0/486169528 conn(0x7f544c06c5f0 msgr2=0x7f544c06ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.647+0000 7f5452a00640 1 -- 172.21.15.31:0/486169528 shutdown_connections 2024-03-21T10:44:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.647+0000 7f5452a00640 1 -- 172.21.15.31:0/486169528 wait complete. 2024-03-21T10:44:37.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.649+0000 7f5452a00640 1 Processor -- start 2024-03-21T10:44:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.652+0000 7f5452a00640 1 -- start start 2024-03-21T10:44:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.652+0000 7f5452a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f544c10caf0 0x7f544c082940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.652+0000 7f5452a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f544c10d7d0 0x7f544c082e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.652+0000 7f5452a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f544c10e170 0x7f544c0833c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.652+0000 7f5452a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544c07cfb0 con 0x7f544c10caf0 2024-03-21T10:44:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.652+0000 7f5452a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544c07ce30 con 0x7f544c10d7d0 2024-03-21T10:44:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.652+0000 7f5452a00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544c07d130 con 0x7f544c10e170 2024-03-21T10:44:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.653+0000 7f5451600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f544c10caf0 0x7f544c082940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.653+0000 7f5450c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f544c10d7d0 0x7f544c082e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.653+0000 7f5452000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f544c10e170 0x7f544c0833c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.653+0000 7f5450c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f544c10d7d0 0x7f544c082e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37188/0 (socket says 172.21.15.31:37188) 2024-03-21T10:44:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.653+0000 7f5450c00640 1 -- 172.21.15.31:0/574178540 learned_addr learned my addr 172.21.15.31:0/574178540 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:37.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.654+0000 7f5452000640 1 -- 172.21.15.31:0/574178540 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f544c10d7d0 msgr2=0x7f544c082e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:37.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.654+0000 7f5452000640 1 --2- 172.21.15.31:0/574178540 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f544c10d7d0 0x7f544c082e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.654+0000 7f5452000640 1 -- 172.21.15.31:0/574178540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f544c10caf0 msgr2=0x7f544c082940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:37.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.654+0000 7f5452000640 1 --2- 172.21.15.31:0/574178540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f544c10caf0 0x7f544c082940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.654+0000 7f5452000640 1 -- 172.21.15.31:0/574178540 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f544c1b5760 con 0x7f544c10e170 2024-03-21T10:44:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.655+0000 7f5451600640 1 --2- 172.21.15.31:0/574178540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f544c10caf0 0x7f544c082940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:44:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.655+0000 7f5450c00640 1 --2- 172.21.15.31:0/574178540 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f544c10d7d0 0x7f544c082e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:44:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.655+0000 7f5452000640 1 --2- 172.21.15.31:0/574178540 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f544c10e170 0x7f544c0833c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5444005fd0 tx=0x7f544400edd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:37.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.656+0000 7f544a000640 1 -- 172.21.15.31:0/574178540 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5444004d20 con 0x7f544c10e170 2024-03-21T10:44:37.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.656+0000 7f544a000640 1 -- 172.21.15.31:0/574178540 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5444005020 con 0x7f544c10e170 2024-03-21T10:44:37.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.656+0000 7f544a000640 1 -- 172.21.15.31:0/574178540 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54440116d0 con 0x7f544c10e170 2024-03-21T10:44:37.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.657+0000 7f5452a00640 1 -- 172.21.15.31:0/574178540 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f544c1b5a50 con 0x7f544c10e170 2024-03-21T10:44:37.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.657+0000 7f5452a00640 1 -- 172.21.15.31:0/574178540 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f544c1b5f50 con 0x7f544c10e170 2024-03-21T10:44:37.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.662+0000 7f544a000640 1 -- 172.21.15.31:0/574178540 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f5444005540 con 0x7f544c10e170 2024-03-21T10:44:37.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.662+0000 7f5456e0c740 1 Processor -- start 2024-03-21T10:44:37.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.663+0000 7f5456e0c740 1 -- start start 2024-03-21T10:44:37.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.663+0000 7f5456e0c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f83cf6b1e0 0x55f83cf6b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.664+0000 7f5456e0c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f83cf6bb00 0x55f83cf74390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.664+0000 7f5456e0c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f83cf748d0 0x55f83cf76cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.664+0000 7f5456e0c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f83cf2d3b0 con 0x55f83cf748d0 2024-03-21T10:44:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.664+0000 7f5456e0c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f83cecf830 con 0x55f83cf6b1e0 2024-03-21T10:44:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.664+0000 7f5450c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f83cf6b1e0 0x55f83cf6b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.664+0000 7f5456e0c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f83ce785d0 con 0x55f83cf6bb00 2024-03-21T10:44:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.664+0000 7f5450c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f83cf6b1e0 0x55f83cf6b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37198/0 (socket says 172.21.15.31:37198) 2024-03-21T10:44:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.664+0000 7f5450c00640 1 -- 172.21.15.31:0/3176355262 learned_addr learned my addr 172.21.15.31:0/3176355262 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.665+0000 7f5451600640 1 --2- 172.21.15.31:0/3176355262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f83cf6bb00 0x55f83cf74390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.665+0000 7f544a000640 1 --2- 172.21.15.31:0/574178540 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f54340a31c0 0x7f54340a5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.665+0000 7f544a000640 1 -- 172.21.15.31:0/574178540 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f54440cfe00 con 0x7f544c10e170 2024-03-21T10:44:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.665+0000 7f5450c00640 1 -- 172.21.15.31:0/3176355262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f83cf6bb00 msgr2=0x55f83cf74390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.665+0000 7f5450c00640 1 --2- 172.21.15.31:0/3176355262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f83cf6bb00 0x55f83cf74390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.665+0000 7f5450c00640 1 -- 172.21.15.31:0/3176355262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f83cf748d0 msgr2=0x55f83cf76cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.665+0000 7f5450c00640 1 --2- 172.21.15.31:0/3176355262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f83cf748d0 0x55f83cf76cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.665+0000 7f5450c00640 1 -- 172.21.15.31:0/3176355262 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f83ce902c0 con 0x55f83cf6b1e0 2024-03-21T10:44:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.666+0000 7f5450c00640 1 --2- 172.21.15.31:0/3176355262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f83cf6b1e0 0x55f83cf6b5c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f5440006aa0 tx=0x7f54400105c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adcb1800a9fff11a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.666+0000 7f5452000640 1 --2- 172.21.15.31:0/574178540 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f54340a31c0 0x7f54340a5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.666+0000 7f543e600640 1 -- 172.21.15.31:0/3176355262 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5440018020 con 0x55f83cf6b1e0 2024-03-21T10:44:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.666+0000 7f543e600640 1 -- 172.21.15.31:0/3176355262 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5440014b20 con 0x55f83cf6b1e0 2024-03-21T10:44:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.666+0000 7f543e600640 1 -- 172.21.15.31:0/3176355262 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5440018020 con 0x55f83cf6b1e0 2024-03-21T10:44:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.667+0000 7f5456e0c740 1 -- 172.21.15.31:0/3176355262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f83cf6b1e0 msgr2=0x55f83cf6b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.668+0000 7f5456e0c740 1 --2- 172.21.15.31:0/3176355262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f83cf6b1e0 0x55f83cf6b5c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f5440006aa0 tx=0x7f54400105c0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.669+0000 7f5452000640 1 --2- 172.21.15.31:0/574178540 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f54340a31c0 0x7f54340a5680 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5444005ba0 tx=0x7f5444004a40 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.671+0000 7f5456e0c740 1 -- 172.21.15.31:0/3176355262 shutdown_connections 2024-03-21T10:44:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.671+0000 7f5456e0c740 1 --2- 172.21.15.31:0/3176355262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f83cf748d0 0x55f83cf76cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.671+0000 7f5456e0c740 1 --2- 172.21.15.31:0/3176355262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f83cf6bb00 0x55f83cf74390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.671+0000 7f5456e0c740 1 --2- 172.21.15.31:0/3176355262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f83cf6b1e0 0x55f83cf6b5c0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.671+0000 7f5456e0c740 1 -- 172.21.15.31:0/3176355262 >> 172.21.15.31:0/3176355262 conn(0x55f83cf65b00 msgr2=0x55f83cf65ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:37.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.672+0000 7f5456e0c740 1 -- 172.21.15.31:0/3176355262 shutdown_connections 2024-03-21T10:44:37.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.672+0000 7f5456e0c740 1 -- 172.21.15.31:0/3176355262 wait complete. 2024-03-21T10:44:37.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.672+0000 7f5456e0c740 1 -- start start 2024-03-21T10:44:37.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.673+0000 7f5456e0c740 1 Processor -- start 2024-03-21T10:44:37.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.673+0000 7f5456e0c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f83cf6b1e0 0x55f83cf71940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:37.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.673+0000 7f5456e0c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f83cf6bb00 0x55f83cf71e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:37.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.673+0000 7f5456e0c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f83cf748d0 0x55f83cf723c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:37.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.673+0000 7f5456e0c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f83ce8ba20 con 0x55f83cf6bb00 2024-03-21T10:44:37.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.673+0000 7f5456e0c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f83cecb8d0 con 0x55f83cf6b1e0 2024-03-21T10:44:37.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.673+0000 7f5456e0c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f83cf77500 con 0x55f83cf748d0 2024-03-21T10:44:37.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.673+0000 7f5456e0c740 10 client.0 _ll_register_callbacks cb 0x55f83cec81c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:37.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.674+0000 7f5450c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f83cf6b1e0 0x55f83cf71940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:37.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.674+0000 7f5450c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f83cf748d0 0x55f83cf723c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:37.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.674+0000 7f5450c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f83cf748d0 0x55f83cf723c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:59884/0 (socket says 172.21.15.31:59884) 2024-03-21T10:44:37.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.674+0000 7f5450c00640 1 -- 172.21.15.31:0/1901159971 learned_addr learned my addr 172.21.15.31:0/1901159971 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:37.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.674+0000 7f5451600640 1 --2- 172.21.15.31:0/1901159971 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f83cf6bb00 0x55f83cf71e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:37.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.675+0000 7f5451600640 1 -- 172.21.15.31:0/1901159971 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f83cf748d0 msgr2=0x55f83cf723c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.675+0000 7f5451600640 1 --2- 172.21.15.31:0/1901159971 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f83cf748d0 0x55f83cf723c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.675+0000 7f5451600640 1 -- 172.21.15.31:0/1901159971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f83cf6b1e0 msgr2=0x55f83cf71940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.675+0000 7f5451600640 1 --2- 172.21.15.31:0/1901159971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f83cf6b1e0 0x55f83cf71940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.675+0000 7f5451600640 1 -- 172.21.15.31:0/1901159971 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f83cbe4210 con 0x55f83cf6bb00 2024-03-21T10:44:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.676+0000 7f5451600640 1 --2- 172.21.15.31:0/1901159971 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f83cf6bb00 0x55f83cf71e80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f543802d280 tx=0x7f5438004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b05e36b8313ebb49 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.676+0000 7f5432a00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:44:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.677+0000 7f5432a00640 1 -- 172.21.15.31:0/1901159971 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543802db90 con 0x55f83cf6bb00 2024-03-21T10:44:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.677+0000 7f5432a00640 1 -- 172.21.15.31:0/1901159971 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f543802dd30 con 0x55f83cf6bb00 2024-03-21T10:44:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.677+0000 7f5456e0c740 1 -- 172.21.15.31:0/1901159971 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5440014b20 con 0x55f83cf6bb00 2024-03-21T10:44:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.677+0000 7f5432a00640 1 -- 172.21.15.31:0/1901159971 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5438004ef0 con 0x55f83cf6bb00 2024-03-21T10:44:37.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.678+0000 7f5456e0c740 10 client.16572 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:37.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.678+0000 7f5456e0c740 1 -- 172.21.15.31:0/1901159971 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55f83ccb0620 con 0x55f83cf6bb00 2024-03-21T10:44:37.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.678+0000 7f5432a00640 1 -- 172.21.15.31:0/1901159971 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5440014b20 con 0x55f83cf6bb00 2024-03-21T10:44:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.679+0000 7f5432a00640 1 -- 172.21.15.31:0/1901159971 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f5438003ab0 con 0x55f83cf6bb00 2024-03-21T10:44:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.679+0000 7f5456e0c740 10 client.16572 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.679+0000 7f5456e0c740 10 client.16572 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:44:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.679+0000 7f5456e0c740 20 client.16572 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.679+0000 7f5456e0c740 1 --2- 172.21.15.31:0/1901159971 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f83cf84920 0x55f83cf86d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.679+0000 7f5456e0c740 4 client.16572 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:44:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.679+0000 7f5456e0c740 1 -- 172.21.15.31:0/1901159971 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f83cedcd70 con 0x55f83cf84920 2024-03-21T10:44:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.680+0000 7f5450c00640 1 --2- 172.21.15.31:0/1901159971 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f83cf84920 0x55f83cf86d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:37.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.680+0000 7f5450c00640 1 --2- 172.21.15.31:0/1901159971 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f83cf84920 0x55f83cf86d10 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae4408bf302994b server_cookie=89739094de66629b in_seq=0 out_seq=0 2024-03-21T10:44:37.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.680+0000 7f5432a00640 10 client.16572 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:37.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.681+0000 7f5432a00640 1 -- 172.21.15.31:0/1901159971 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f83cedcd70 con 0x55f83cf84920 2024-03-21T10:44:37.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.681+0000 7f5432a00640 10 client.16572 handle_command_reply: tid=0 2024-03-21T10:44:37.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.681+0000 7f5456e0c740 1 client.16572 shutdown 2024-03-21T10:44:37.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.681+0000 7f5456e0c740 1 -- 172.21.15.31:0/1901159971 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f83cf6bb00 msgr2=0x55f83cf71e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:37.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.681+0000 7f5456e0c740 1 --2- 172.21.15.31:0/1901159971 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f83cf6bb00 0x55f83cf71e80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f543802d280 tx=0x7f5438004270 comp rx=0 tx=0).stop 2024-03-21T10:44:37.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.681+0000 7f5456e0c740 1 -- 172.21.15.31:0/1901159971 shutdown_connections 2024-03-21T10:44:37.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.681+0000 7f5456e0c740 1 --2- 172.21.15.31:0/1901159971 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f83cf84920 0x55f83cf86d10 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.681+0000 7f5456e0c740 1 --2- 172.21.15.31:0/1901159971 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f83cf748d0 0x55f83cf723c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.681+0000 7f5456e0c740 1 --2- 172.21.15.31:0/1901159971 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f83cf6bb00 0x55f83cf71e80 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.681+0000 7f5456e0c740 1 --2- 172.21.15.31:0/1901159971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f83cf6b1e0 0x55f83cf71940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.681+0000 7f5456e0c740 1 -- 172.21.15.31:0/1901159971 >> 172.21.15.31:0/1901159971 conn(0x55f83cf65b00 msgr2=0x55f83cf67850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:37.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.681+0000 7f5432a00640 0 client.16572 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:37.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.682+0000 7f5456e0c740 1 -- 172.21.15.31:0/1901159971 shutdown_connections 2024-03-21T10:44:37.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.682+0000 7f5456e0c740 1 -- 172.21.15.31:0/1901159971 wait complete. 2024-03-21T10:44:37.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.682+0000 7f5456e0c740 20 client.16572 trim_cache size 0 max 16384 2024-03-21T10:44:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.697+0000 7f5456e0c740 1 Processor -- start 2024-03-21T10:44:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.697+0000 7f5456e0c740 1 -- start start 2024-03-21T10:44:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.697+0000 7f5456e0c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f83cf65b00 0x55f83cf73770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.697+0000 7f5456e0c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f83cf6b1e0 0x55f83cf7fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.697+0000 7f5456e0c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f83cf748d0 0x55f83cf80150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.697+0000 7f5456e0c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544c1d92c0 con 0x55f83cf6b1e0 2024-03-21T10:44:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.697+0000 7f5456e0c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544c1d6dc0 con 0x55f83cf748d0 2024-03-21T10:44:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.697+0000 7f5456e0c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544c1d4eb0 con 0x55f83cf65b00 2024-03-21T10:44:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.698+0000 7f5450c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f83cf65b00 0x55f83cf73770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.698+0000 7f5450c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f83cf65b00 0x55f83cf73770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:59890/0 (socket says 172.21.15.31:59890) 2024-03-21T10:44:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.698+0000 7f5450c00640 1 -- 172.21.15.31:0/1928704662 learned_addr learned my addr 172.21.15.31:0/1928704662 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.698+0000 7f5450c00640 1 --2- 172.21.15.31:0/1928704662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f83cf748d0 0x55f83cf80150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.698+0000 7f5451600640 1 --2- 172.21.15.31:0/1928704662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f83cf6b1e0 0x55f83cf7fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.698+0000 7f5450c00640 1 -- 172.21.15.31:0/1928704662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f83cf65b00 msgr2=0x55f83cf73770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.698+0000 7f5450c00640 1 --2- 172.21.15.31:0/1928704662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f83cf65b00 0x55f83cf73770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.698+0000 7f5450c00640 1 -- 172.21.15.31:0/1928704662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f83cf6b1e0 msgr2=0x55f83cf7fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.698+0000 7f5450c00640 1 --2- 172.21.15.31:0/1928704662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f83cf6b1e0 0x55f83cf7fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.698+0000 7f5450c00640 1 -- 172.21.15.31:0/1928704662 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f83cf63f80 con 0x55f83cf748d0 2024-03-21T10:44:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.698+0000 7f5451600640 1 --2- 172.21.15.31:0/1928704662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f83cf6b1e0 0x55f83cf7fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:44:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.698+0000 7f5450c00640 1 --2- 172.21.15.31:0/1928704662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f83cf748d0 0x55f83cf80150 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f5440009870 tx=0x7f54400091c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa749eaff2ecaae9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.699+0000 7f543dc00640 1 -- 172.21.15.31:0/1928704662 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5440018040 con 0x55f83cf748d0 2024-03-21T10:44:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.699+0000 7f543dc00640 1 -- 172.21.15.31:0/1928704662 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5440014920 con 0x55f83cf748d0 2024-03-21T10:44:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.699+0000 7f543dc00640 1 -- 172.21.15.31:0/1928704662 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544001d5a0 con 0x55f83cf748d0 2024-03-21T10:44:37.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.700+0000 7f5456e0c740 1 -- 172.21.15.31:0/1928704662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f83cf748d0 msgr2=0x55f83cf80150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:37.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.700+0000 7f5456e0c740 1 --2- 172.21.15.31:0/1928704662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f83cf748d0 0x55f83cf80150 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f5440009870 tx=0x7f54400091c0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.700+0000 7f5456e0c740 1 -- 172.21.15.31:0/1928704662 shutdown_connections 2024-03-21T10:44:37.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.700+0000 7f5456e0c740 1 --2- 172.21.15.31:0/1928704662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f83cf748d0 0x55f83cf80150 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.700+0000 7f5456e0c740 1 --2- 172.21.15.31:0/1928704662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f83cf6b1e0 0x55f83cf7fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.700+0000 7f5456e0c740 1 --2- 172.21.15.31:0/1928704662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f83cf65b00 0x55f83cf73770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.700+0000 7f5456e0c740 1 -- 172.21.15.31:0/1928704662 >> 172.21.15.31:0/1928704662 conn(0x55f83cf87080 msgr2=0x55f83cf73060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.701+0000 7f5456e0c740 1 -- 172.21.15.31:0/1928704662 shutdown_connections 2024-03-21T10:44:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.701+0000 7f5456e0c740 1 -- 172.21.15.31:0/1928704662 wait complete. 2024-03-21T10:44:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.701+0000 7f5456e0c740 1 -- start start 2024-03-21T10:44:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.702+0000 7f5456e0c740 1 Processor -- start 2024-03-21T10:44:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.702+0000 7f5456e0c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f83cf65b00 0x55f83cf79720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.702+0000 7f5456e0c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f83cf6b1e0 0x55f83cf79c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.702+0000 7f5456e0c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f83cf748d0 0x55f83cf7a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.702+0000 7f5456e0c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544c1d5460 con 0x55f83cf65b00 2024-03-21T10:44:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.702+0000 7f5456e0c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544c1d6760 con 0x55f83cf6b1e0 2024-03-21T10:44:37.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.702+0000 7f5456e0c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544c1d5aa0 con 0x55f83cf748d0 2024-03-21T10:44:37.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.702+0000 7f5456e0c740 10 client.0 _ll_register_callbacks cb 0x7f5438009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:37.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.703+0000 7f5451600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f83cf6b1e0 0x55f83cf79c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:37.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.703+0000 7f5451600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f83cf6b1e0 0x55f83cf79c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37226/0 (socket says 172.21.15.31:37226) 2024-03-21T10:44:37.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.703+0000 7f5451600640 1 -- 172.21.15.31:0/775473897 learned_addr learned my addr 172.21.15.31:0/775473897 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:37.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.703+0000 7f5450c00640 1 --2- 172.21.15.31:0/775473897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f83cf748d0 0x55f83cf7a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:37.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.704+0000 7f5450c00640 1 -- 172.21.15.31:0/775473897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f83cf6b1e0 msgr2=0x55f83cf79c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:37.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.704+0000 7f5450c00640 1 --2- 172.21.15.31:0/775473897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f83cf6b1e0 0x55f83cf79c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.704+0000 7f5450c00640 1 -- 172.21.15.31:0/775473897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f83cf65b00 msgr2=0x55f83cf79720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:37.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.704+0000 7f5450c00640 1 --2- 172.21.15.31:0/775473897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f83cf65b00 0x55f83cf79720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.704+0000 7f5450c00640 1 -- 172.21.15.31:0/775473897 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f83cf6f9b0 con 0x55f83cf748d0 2024-03-21T10:44:37.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.705+0000 7f5432a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:44:37.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.705+0000 7f5450c00640 1 --2- 172.21.15.31:0/775473897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f83cf748d0 0x55f83cf7a1a0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f5440010ac0 tx=0x7f5440010bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c0527273c88f380 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:37.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.705+0000 7f5432a00640 1 -- 172.21.15.31:0/775473897 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5440032030 con 0x55f83cf748d0 2024-03-21T10:44:37.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.705+0000 7f5432a00640 1 -- 172.21.15.31:0/775473897 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f5440033070 con 0x55f83cf748d0 2024-03-21T10:44:37.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.705+0000 7f5456e0c740 1 -- 172.21.15.31:0/775473897 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5440014920 con 0x55f83cf748d0 2024-03-21T10:44:37.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.706+0000 7f5432a00640 1 -- 172.21.15.31:0/775473897 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f544003b050 con 0x55f83cf748d0 2024-03-21T10:44:37.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.706+0000 7f5456e0c740 10 client.27016 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:37.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.706+0000 7f5456e0c740 1 -- 172.21.15.31:0/775473897 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55f83cfe1180 con 0x55f83cf748d0 2024-03-21T10:44:37.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.706+0000 7f5432a00640 1 -- 172.21.15.31:0/775473897 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5440014920 con 0x55f83cf748d0 2024-03-21T10:44:37.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.707+0000 7f5432a00640 1 -- 172.21.15.31:0/775473897 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f54400061f0 con 0x55f83cf748d0 2024-03-21T10:44:37.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.707+0000 7f5456e0c740 10 client.27016 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:37.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.707+0000 7f5456e0c740 10 client.27016 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:44:37.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.707+0000 7f5456e0c740 20 client.27016 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:37.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.707+0000 7f5456e0c740 1 --2- 172.21.15.31:0/775473897 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f83cfe1350 0x55f83cfe3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:37.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.707+0000 7f5456e0c740 4 client.27016 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:44:37.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.707+0000 7f5456e0c740 1 -- 172.21.15.31:0/775473897 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f83ced70c0 con 0x55f83cfe1350 2024-03-21T10:44:37.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.708+0000 7f5451600640 1 --2- 172.21.15.31:0/775473897 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f83cfe1350 0x55f83cfe3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:37.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.708+0000 7f5432a00640 10 client.27016 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:37.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.708+0000 7f5451600640 1 --2- 172.21.15.31:0/775473897 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f83cfe1350 0x55f83cfe3740 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e7afc123e8ab5c4 server_cookie=5cfb708a0b468bdd in_seq=0 out_seq=0 2024-03-21T10:44:37.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.729+0000 7f5432a00640 1 -- 172.21.15.31:0/775473897 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f83ced70c0 con 0x55f83cfe1350 2024-03-21T10:44:37.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.729+0000 7f5432a00640 10 client.27016 handle_command_reply: tid=0 2024-03-21T10:44:37.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.729+0000 7f5456e0c740 1 client.27016 shutdown 2024-03-21T10:44:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.730+0000 7f5456e0c740 1 -- 172.21.15.31:0/775473897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f83cf748d0 msgr2=0x55f83cf7a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.730+0000 7f5456e0c740 1 --2- 172.21.15.31:0/775473897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f83cf748d0 0x55f83cf7a1a0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f5440010ac0 tx=0x7f5440010bb0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.730+0000 7f5456e0c740 1 -- 172.21.15.31:0/775473897 shutdown_connections 2024-03-21T10:44:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.730+0000 7f5456e0c740 1 --2- 172.21.15.31:0/775473897 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f83cfe1350 0x55f83cfe3740 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.731+0000 7f5456e0c740 1 --2- 172.21.15.31:0/775473897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f83cf748d0 0x55f83cf7a1a0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.731+0000 7f5432a00640 0 client.27016 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.731+0000 7f5456e0c740 1 --2- 172.21.15.31:0/775473897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f83cf6b1e0 0x55f83cf79c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.731+0000 7f5456e0c740 1 --2- 172.21.15.31:0/775473897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f83cf65b00 0x55f83cf79720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.731+0000 7f5456e0c740 1 -- 172.21.15.31:0/775473897 >> 172.21.15.31:0/775473897 conn(0x55f83cf87080 msgr2=0x55f83cf72170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:37.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.731+0000 7f5456e0c740 1 -- 172.21.15.31:0/775473897 shutdown_connections 2024-03-21T10:44:37.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.731+0000 7f5456e0c740 1 -- 172.21.15.31:0/775473897 wait complete. 2024-03-21T10:44:37.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.731+0000 7f5456e0c740 20 client.27016 trim_cache size 0 max 16384 2024-03-21T10:44:37.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:44:37.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:44:37.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:44:37.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:44:37.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.732+0000 7f5452a00640 1 -- 172.21.15.31:0/574178540 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f54340a31c0 msgr2=0x7f54340a5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.732+0000 7f5452a00640 1 --2- 172.21.15.31:0/574178540 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f54340a31c0 0x7f54340a5680 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5444005ba0 tx=0x7f5444004a40 comp rx=0 tx=0).stop 2024-03-21T10:44:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.732+0000 7f5452a00640 1 -- 172.21.15.31:0/574178540 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f544c10e170 msgr2=0x7f544c0833c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.732+0000 7f5452a00640 1 --2- 172.21.15.31:0/574178540 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f544c10e170 0x7f544c0833c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5444005fd0 tx=0x7f544400edd0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.732+0000 7f5452a00640 1 -- 172.21.15.31:0/574178540 shutdown_connections 2024-03-21T10:44:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.732+0000 7f5452a00640 1 --2- 172.21.15.31:0/574178540 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f54340a31c0 0x7f54340a5680 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.732+0000 7f5452a00640 1 --2- 172.21.15.31:0/574178540 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f544c10e170 0x7f544c0833c0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.732+0000 7f5452a00640 1 --2- 172.21.15.31:0/574178540 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f544c10d7d0 0x7f544c082e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.732+0000 7f5452a00640 1 --2- 172.21.15.31:0/574178540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f544c10caf0 0x7f544c082940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.732+0000 7f5452a00640 1 -- 172.21.15.31:0/574178540 >> 172.21.15.31:0/574178540 conn(0x7f544c06c5f0 msgr2=0x7f544c071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.733+0000 7f5452a00640 1 -- 172.21.15.31:0/574178540 shutdown_connections 2024-03-21T10:44:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:37.733+0000 7f5452a00640 1 -- 172.21.15.31:0/574178540 wait complete. 2024-03-21T10:44:38.785 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:44:38.786 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:44:39.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.079+0000 7fa7b8a00640 1 -- 172.21.15.31:0/1076877673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7b41014a0 msgr2=0x7fa7b4101920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:39.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.079+0000 7fa7b8a00640 1 --2- 172.21.15.31:0/1076877673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7b41014a0 0x7fa7b4101920 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0009890 tx=0x7fa7a002bb70 comp rx=0 tx=0).stop 2024-03-21T10:44:39.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.079+0000 7fa7b8a00640 1 -- 172.21.15.31:0/1076877673 shutdown_connections 2024-03-21T10:44:39.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.079+0000 7fa7b8a00640 1 --2- 172.21.15.31:0/1076877673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b4101e60 0x7fa7b410e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.079+0000 7fa7b8a00640 1 --2- 172.21.15.31:0/1076877673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7b41014a0 0x7fa7b4101920 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.079+0000 7fa7b8a00640 1 --2- 172.21.15.31:0/1076877673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7b4107560 0x7fa7b4107940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.079+0000 7fa7b8a00640 1 -- 172.21.15.31:0/1076877673 >> 172.21.15.31:0/1076877673 conn(0x7fa7b40fcfb0 msgr2=0x7fa7b40ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:39.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.083+0000 7fa7b8a00640 1 -- 172.21.15.31:0/1076877673 shutdown_connections 2024-03-21T10:44:39.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.083+0000 7fa7b8a00640 1 -- 172.21.15.31:0/1076877673 wait complete. 2024-03-21T10:44:39.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.084+0000 7fa7b8a00640 1 Processor -- start 2024-03-21T10:44:39.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.085+0000 7fa7b8a00640 1 -- start start 2024-03-21T10:44:39.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.085+0000 7fa7b8a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b41014a0 0x7fa7b4197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:39.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.085+0000 7fa7b8a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7b4101e60 0x7fa7b4198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:39.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.086+0000 7fa7b2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7b4101e60 0x7fa7b4198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:39.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.086+0000 7fa7b2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7b4101e60 0x7fa7b4198360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:59920/0 (socket says 172.21.15.31:59920) 2024-03-21T10:44:39.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.090+0000 7fa7b2a00640 1 -- 172.21.15.31:0/1155712786 learned_addr learned my addr 172.21.15.31:0/1155712786 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:39.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.090+0000 7fa7b8a00640 1 --2- 172.21.15.31:0/1155712786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7b4107560 0x7fa7b419c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:39.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.090+0000 7fa7b8a00640 1 -- 172.21.15.31:0/1155712786 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b4111820 con 0x7fa7b41014a0 2024-03-21T10:44:39.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.090+0000 7fa7b8a00640 1 -- 172.21.15.31:0/1155712786 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b41116a0 con 0x7fa7b4107560 2024-03-21T10:44:39.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.090+0000 7fa7b8a00640 1 -- 172.21.15.31:0/1155712786 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b41119a0 con 0x7fa7b4101e60 2024-03-21T10:44:39.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.090+0000 7fa7b3400640 1 --2- 172.21.15.31:0/1155712786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b41014a0 0x7fa7b4197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:39.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.090+0000 7fa7b3e00640 1 --2- 172.21.15.31:0/1155712786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7b4107560 0x7fa7b419c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:39.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.090+0000 7fa7b2a00640 1 -- 172.21.15.31:0/1155712786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7b4107560 msgr2=0x7fa7b419c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:39.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.090+0000 7fa7b2a00640 1 --2- 172.21.15.31:0/1155712786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7b4107560 0x7fa7b419c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.090+0000 7fa7b2a00640 1 -- 172.21.15.31:0/1155712786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b41014a0 msgr2=0x7fa7b4197e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:39.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.090+0000 7fa7b2a00640 1 --2- 172.21.15.31:0/1155712786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b41014a0 0x7fa7b4197e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.090+0000 7fa7b2a00640 1 -- 172.21.15.31:0/1155712786 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b419cdc0 con 0x7fa7b4101e60 2024-03-21T10:44:39.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.091+0000 7fa7b2a00640 1 --2- 172.21.15.31:0/1155712786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7b4101e60 0x7fa7b4198360 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0000c00 tx=0x7fa7a0004550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:39.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.091+0000 7fa7abe00640 1 -- 172.21.15.31:0/1155712786 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a0041020 con 0x7fa7b4101e60 2024-03-21T10:44:39.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.091+0000 7fa7abe00640 1 -- 172.21.15.31:0/1155712786 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7a002c9f0 con 0x7fa7b4101e60 2024-03-21T10:44:39.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.091+0000 7fa7abe00640 1 -- 172.21.15.31:0/1155712786 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a0039040 con 0x7fa7b4101e60 2024-03-21T10:44:39.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.092+0000 7fa7b8a00640 1 -- 172.21.15.31:0/1155712786 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b419d050 con 0x7fa7b4101e60 2024-03-21T10:44:39.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.092+0000 7fa7b8a00640 1 -- 172.21.15.31:0/1155712786 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b406d0d0 con 0x7fa7b4101e60 2024-03-21T10:44:39.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.095+0000 7fa7bcd6b740 1 Processor -- start 2024-03-21T10:44:39.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.096+0000 7fa7bcd6b740 1 -- start start 2024-03-21T10:44:39.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.096+0000 7fa7bcd6b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561df9b11820 0x561df9b11c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:39.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.096+0000 7fa7bcd6b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561df9b02090 0x561df9b02510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:39.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.096+0000 7fa7bcd6b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561df9b02a50 0x561df9b19490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:39.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.096+0000 7fa7bcd6b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561df9a69250 con 0x561df9b02090 2024-03-21T10:44:39.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.096+0000 7fa7bcd6b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561df9a11a30 con 0x561df9b02a50 2024-03-21T10:44:39.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.096+0000 7fa7bcd6b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x561df9a652f0 con 0x561df9b11820 2024-03-21T10:44:39.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.096+0000 7fa7b3e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561df9b11820 0x561df9b11c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:39.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.096+0000 7fa7b3e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561df9b11820 0x561df9b11c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:59926/0 (socket says 172.21.15.31:59926) 2024-03-21T10:44:39.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.096+0000 7fa7b3e00640 1 -- 172.21.15.31:0/1831905520 learned_addr learned my addr 172.21.15.31:0/1831905520 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:39.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.097+0000 7fa7b3e00640 1 -- 172.21.15.31:0/1831905520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561df9b02a50 msgr2=0x561df9b19490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:44:39.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.097+0000 7fa7b3e00640 1 --2- 172.21.15.31:0/1831905520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561df9b02a50 0x561df9b19490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.097+0000 7fa7b3e00640 1 -- 172.21.15.31:0/1831905520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561df9b02090 msgr2=0x561df9b02510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:44:39.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.098+0000 7fa7b3400640 1 --2- 172.21.15.31:0/1831905520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561df9b02090 0x561df9b02510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:39.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.098+0000 7fa7b3e00640 1 --2- 172.21.15.31:0/1831905520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561df9b02090 0x561df9b02510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.098+0000 7fa7b3e00640 1 -- 172.21.15.31:0/1831905520 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561df99fc690 con 0x561df9b11820 2024-03-21T10:44:39.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.098+0000 7fa7b3400640 1 --2- 172.21.15.31:0/1831905520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561df9b02090 0x561df9b02510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:44:39.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.098+0000 7fa7abe00640 1 -- 172.21.15.31:0/1155712786 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fa7a0044020 con 0x7fa7b4101e60 2024-03-21T10:44:39.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.098+0000 7fa7b3e00640 1 --2- 172.21.15.31:0/1831905520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561df9b11820 0x561df9b11c00 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac010400 tx=0x7fa7ac0108c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0824d9f336695cc server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:39.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.099+0000 7fa79fe00640 1 -- 172.21.15.31:0/1831905520 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac003b70 con 0x561df9b11820 2024-03-21T10:44:39.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.099+0000 7fa79fe00640 1 -- 172.21.15.31:0/1831905520 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7ac003d10 con 0x561df9b11820 2024-03-21T10:44:39.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.099+0000 7fa79fe00640 1 -- 172.21.15.31:0/1831905520 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac011b00 con 0x561df9b11820 2024-03-21T10:44:39.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.099+0000 7fa7bcd6b740 1 -- 172.21.15.31:0/1831905520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561df9b11820 msgr2=0x561df9b11c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:39.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.099+0000 7fa7bcd6b740 1 --2- 172.21.15.31:0/1831905520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561df9b11820 0x561df9b11c00 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac010400 tx=0x7fa7ac0108c0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.100+0000 7fa7bcd6b740 1 -- 172.21.15.31:0/1831905520 shutdown_connections 2024-03-21T10:44:39.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.100+0000 7fa7bcd6b740 1 --2- 172.21.15.31:0/1831905520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561df9b02a50 0x561df9b19490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.100+0000 7fa7bcd6b740 1 --2- 172.21.15.31:0/1831905520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561df9b02090 0x561df9b02510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.100+0000 7fa7bcd6b740 1 --2- 172.21.15.31:0/1831905520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561df9b11820 0x561df9b11c00 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.100+0000 7fa7bcd6b740 1 -- 172.21.15.31:0/1831905520 >> 172.21.15.31:0/1831905520 conn(0x561df9acc200 msgr2=0x561df9b00f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:39.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.100+0000 7fa7bcd6b740 1 -- 172.21.15.31:0/1831905520 shutdown_connections 2024-03-21T10:44:39.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.100+0000 7fa7bcd6b740 1 -- 172.21.15.31:0/1831905520 wait complete. 2024-03-21T10:44:39.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.100+0000 7fa7bcd6b740 1 -- start start 2024-03-21T10:44:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.101+0000 7fa7bcd6b740 1 Processor -- start 2024-03-21T10:44:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.101+0000 7fa7abe00640 1 --2- 172.21.15.31:0/1155712786 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa7980a3170 0x7fa7980a5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.101+0000 7fa7abe00640 1 -- 172.21.15.31:0/1155712786 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fa7a00f0980 con 0x7fa7b4101e60 2024-03-21T10:44:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.101+0000 7fa7bcd6b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561df9b02090 0x561df9b191a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.101+0000 7fa7bcd6b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561df9b02a50 0x561df9b16530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.101+0000 7fa7bcd6b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561df9b11820 0x561df9b16a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.101+0000 7fa7bcd6b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561df9b19b80 con 0x561df9b02090 2024-03-21T10:44:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.101+0000 7fa7bcd6b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561df9a24e80 con 0x561df9b02a50 2024-03-21T10:44:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.101+0000 7fa7bcd6b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x561df9b19d00 con 0x561df9b11820 2024-03-21T10:44:39.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.101+0000 7fa7b3e00640 1 --2- 172.21.15.31:0/1155712786 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa7980a3170 0x7fa7980a5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:39.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.101+0000 7fa7bcd6b740 10 client.0 _ll_register_callbacks cb 0x561df9a37130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:39.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.102+0000 7fa7b2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561df9b02a50 0x561df9b16530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:39.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.102+0000 7fa7b3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561df9b02090 0x561df9b191a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:39.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.102+0000 7fa7b2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561df9b02a50 0x561df9b16530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37258/0 (socket says 172.21.15.31:37258) 2024-03-21T10:44:39.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.102+0000 7fa7b2a00640 1 -- 172.21.15.31:0/197537094 learned_addr learned my addr 172.21.15.31:0/197537094 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:39.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.102+0000 7fa7b3400640 1 -- 172.21.15.31:0/197537094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561df9b11820 msgr2=0x561df9b16a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:39.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.102+0000 7fa7b3400640 1 --2- 172.21.15.31:0/197537094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561df9b11820 0x561df9b16a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.102+0000 7fa7b3400640 1 -- 172.21.15.31:0/197537094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561df9b02a50 msgr2=0x561df9b16530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:39.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.102+0000 7fa7b3400640 1 --2- 172.21.15.31:0/197537094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561df9b02a50 0x561df9b16530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.102+0000 7fa7b3400640 1 -- 172.21.15.31:0/197537094 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561df98477d0 con 0x561df9b02090 2024-03-21T10:44:39.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.102+0000 7fa7b2a00640 1 --2- 172.21.15.31:0/197537094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561df9b02a50 0x561df9b16530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:44:39.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.103+0000 7fa7b3400640 1 --2- 172.21.15.31:0/197537094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561df9b02090 0x561df9b191a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4005fd0 tx=0x7fa7a40107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a5437b2f4da7877 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:39.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.103+0000 7fa79cc00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:44:39.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.103+0000 7fa7b3e00640 1 --2- 172.21.15.31:0/1155712786 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa7980a3170 0x7fa7980a5630 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac009dd0 tx=0x7fa7ac011600 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:39.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.103+0000 7fa79cc00640 1 -- 172.21.15.31:0/197537094 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4005220 con 0x561df9b02090 2024-03-21T10:44:39.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.103+0000 7fa79cc00640 1 -- 172.21.15.31:0/197537094 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fa7a40053c0 con 0x561df9b02090 2024-03-21T10:44:39.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.103+0000 7fa7bcd6b740 1 -- 172.21.15.31:0/197537094 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa7ac003d10 con 0x561df9b02090 2024-03-21T10:44:39.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.103+0000 7fa79cc00640 1 -- 172.21.15.31:0/197537094 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7a40055c0 con 0x561df9b02090 2024-03-21T10:44:39.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.104+0000 7fa7bcd6b740 10 client.16581 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:39.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.104+0000 7fa7bcd6b740 1 -- 172.21.15.31:0/197537094 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x561df977d3f0 con 0x561df9b02090 2024-03-21T10:44:39.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.104+0000 7fa79cc00640 1 -- 172.21.15.31:0/197537094 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7ac003d10 con 0x561df9b02090 2024-03-21T10:44:39.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.105+0000 7fa79cc00640 1 -- 172.21.15.31:0/197537094 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa7a4011bb0 con 0x561df9b02090 2024-03-21T10:44:39.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.105+0000 7fa7bcd6b740 10 client.16581 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:39.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.105+0000 7fa7bcd6b740 10 client.16581 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:44:39.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.105+0000 7fa7bcd6b740 20 client.16581 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:39.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.105+0000 7fa7bcd6b740 1 --2- 172.21.15.31:0/197537094 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561df9b26c10 0x561df9b29000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:39.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.105+0000 7fa7bcd6b740 4 client.16581 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:44:39.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.105+0000 7fa7bcd6b740 1 -- 172.21.15.31:0/197537094 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561df9a76790 con 0x561df9b26c10 2024-03-21T10:44:39.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.105+0000 7fa7b2a00640 1 --2- 172.21.15.31:0/197537094 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561df9b26c10 0x561df9b29000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:39.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.106+0000 7fa79cc00640 10 client.16581 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:39.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.106+0000 7fa7b2a00640 1 --2- 172.21.15.31:0/197537094 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561df9b26c10 0x561df9b29000 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a07195e1008a27ce server_cookie=bb16781a13f71442 in_seq=0 out_seq=0 2024-03-21T10:44:39.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.107+0000 7fa79cc00640 1 -- 172.21.15.31:0/197537094 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561df9a76790 con 0x561df9b26c10 2024-03-21T10:44:39.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.107+0000 7fa79cc00640 10 client.16581 handle_command_reply: tid=0 2024-03-21T10:44:39.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.107+0000 7fa7bcd6b740 1 client.16581 shutdown 2024-03-21T10:44:39.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.107+0000 7fa7bcd6b740 1 -- 172.21.15.31:0/197537094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561df9b02090 msgr2=0x561df9b191a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:39.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.107+0000 7fa7bcd6b740 1 --2- 172.21.15.31:0/197537094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561df9b02090 0x561df9b191a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4005fd0 tx=0x7fa7a40107a0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.107+0000 7fa7bcd6b740 1 -- 172.21.15.31:0/197537094 shutdown_connections 2024-03-21T10:44:39.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.107+0000 7fa7bcd6b740 1 --2- 172.21.15.31:0/197537094 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561df9b26c10 0x561df9b29000 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.107+0000 7fa7bcd6b740 1 --2- 172.21.15.31:0/197537094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561df9b11820 0x561df9b16a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.107+0000 7fa79cc00640 0 client.16581 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:39.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.107+0000 7fa7bcd6b740 1 --2- 172.21.15.31:0/197537094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561df9b02a50 0x561df9b16530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.107+0000 7fa7bcd6b740 1 --2- 172.21.15.31:0/197537094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561df9b02090 0x561df9b191a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.107+0000 7fa7bcd6b740 1 -- 172.21.15.31:0/197537094 >> 172.21.15.31:0/197537094 conn(0x561df9acc200 msgr2=0x561df9affe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:39.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.107+0000 7fa7bcd6b740 1 -- 172.21.15.31:0/197537094 shutdown_connections 2024-03-21T10:44:39.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.107+0000 7fa7bcd6b740 1 -- 172.21.15.31:0/197537094 wait complete. 2024-03-21T10:44:39.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.108+0000 7fa7bcd6b740 20 client.16581 trim_cache size 0 max 16384 2024-03-21T10:44:39.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.136+0000 7fa7bcd6b740 1 Processor -- start 2024-03-21T10:44:39.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.141+0000 7fa7bcd6b740 1 -- start start 2024-03-21T10:44:39.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.141+0000 7fa7bcd6b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561df9b02090 0x561df9b15d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:39.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.141+0000 7fa7bcd6b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561df9b02a50 0x561df9b18c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:39.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.141+0000 7fa7bcd6b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561df9b11820 0x561df9b191a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:39.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.141+0000 7fa7bcd6b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b41b7390 con 0x561df9b11820 2024-03-21T10:44:39.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.141+0000 7fa7bcd6b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b41b4e90 con 0x561df9b02a50 2024-03-21T10:44:39.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.141+0000 7fa7bcd6b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b40750d0 con 0x561df9b02090 2024-03-21T10:44:39.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.142+0000 7fa7b3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561df9b02090 0x561df9b15d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:39.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.142+0000 7fa7b2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561df9b02a50 0x561df9b18c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:39.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.142+0000 7fa7b3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561df9b11820 0x561df9b191a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:39.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.142+0000 7fa7b3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561df9b02090 0x561df9b15d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:59946/0 (socket says 172.21.15.31:59946) 2024-03-21T10:44:39.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.142+0000 7fa7b3400640 1 -- 172.21.15.31:0/2198397552 learned_addr learned my addr 172.21.15.31:0/2198397552 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:39.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.143+0000 7fa7b3400640 1 -- 172.21.15.31:0/2198397552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561df9b02a50 msgr2=0x561df9b18c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:39.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.144+0000 7fa7b3400640 1 --2- 172.21.15.31:0/2198397552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561df9b02a50 0x561df9b18c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.144+0000 7fa7b3400640 1 -- 172.21.15.31:0/2198397552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561df9b11820 msgr2=0x561df9b191a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:39.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.144+0000 7fa7b3400640 1 --2- 172.21.15.31:0/2198397552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561df9b11820 0x561df9b191a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.144+0000 7fa7b3400640 1 -- 172.21.15.31:0/2198397552 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561df973d0f0 con 0x561df9b02090 2024-03-21T10:44:39.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.144+0000 7fa7b2a00640 1 --2- 172.21.15.31:0/2198397552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561df9b02a50 0x561df9b18c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:44:39.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.145+0000 7fa7b3400640 1 --2- 172.21.15.31:0/2198397552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561df9b02090 0x561df9b15d00 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4010ca0 tx=0x7fa7a4011700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd50a35fbceccd41 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:39.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.148+0000 7fa79f400640 1 -- 172.21.15.31:0/2198397552 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a402d070 con 0x561df9b02090 2024-03-21T10:44:39.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.149+0000 7fa79f400640 1 -- 172.21.15.31:0/2198397552 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7a4011a90 con 0x561df9b02090 2024-03-21T10:44:39.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.149+0000 7fa79f400640 1 -- 172.21.15.31:0/2198397552 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4029400 con 0x561df9b02090 2024-03-21T10:44:39.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.150+0000 7fa7bcd6b740 1 -- 172.21.15.31:0/2198397552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561df9b02090 msgr2=0x561df9b15d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:39.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.150+0000 7fa7bcd6b740 1 --2- 172.21.15.31:0/2198397552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561df9b02090 0x561df9b15d00 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4010ca0 tx=0x7fa7a4011700 comp rx=0 tx=0).stop 2024-03-21T10:44:39.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.151+0000 7fa7bcd6b740 1 -- 172.21.15.31:0/2198397552 shutdown_connections 2024-03-21T10:44:39.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.151+0000 7fa7bcd6b740 1 --2- 172.21.15.31:0/2198397552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561df9b11820 0x561df9b191a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.151+0000 7fa7bcd6b740 1 --2- 172.21.15.31:0/2198397552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561df9b02a50 0x561df9b18c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.151+0000 7fa7bcd6b740 1 --2- 172.21.15.31:0/2198397552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561df9b02090 0x561df9b15d00 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.151+0000 7fa7bcd6b740 1 -- 172.21.15.31:0/2198397552 >> 172.21.15.31:0/2198397552 conn(0x561df9acc200 msgr2=0x561df9afebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:39.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.151+0000 7fa7bcd6b740 1 -- 172.21.15.31:0/2198397552 shutdown_connections 2024-03-21T10:44:39.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.151+0000 7fa7bcd6b740 1 -- 172.21.15.31:0/2198397552 wait complete. 2024-03-21T10:44:39.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.152+0000 7fa7bcd6b740 1 -- start start 2024-03-21T10:44:39.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.152+0000 7fa7bcd6b740 1 Processor -- start 2024-03-21T10:44:39.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.153+0000 7fa7bcd6b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561df9b02090 0x561df9b1e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:39.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.153+0000 7fa7bcd6b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561df9b02a50 0x561df9b13650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:39.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.153+0000 7fa7bcd6b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561df9b11820 0x561df9b13b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:39.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.153+0000 7fa7bcd6b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b40756d0 con 0x561df9b02a50 2024-03-21T10:44:39.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.153+0000 7fa7bcd6b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b41b4830 con 0x561df9b02090 2024-03-21T10:44:39.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.153+0000 7fa7bcd6b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b41b3b70 con 0x561df9b11820 2024-03-21T10:44:39.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.153+0000 7fa7bcd6b740 10 client.0 _ll_register_callbacks cb 0x7fa7a4009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:39.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.153+0000 7fa7b3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561df9b02090 0x561df9b1e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:39.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.154+0000 7fa7b3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561df9b02090 0x561df9b1e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37280/0 (socket says 172.21.15.31:37280) 2024-03-21T10:44:39.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.154+0000 7fa7b3400640 1 -- 172.21.15.31:0/124769618 learned_addr learned my addr 172.21.15.31:0/124769618 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:39.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.154+0000 7fa7b2a00640 1 --2- 172.21.15.31:0/124769618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561df9b02a50 0x561df9b13650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:39.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.154+0000 7fa7b3e00640 1 --2- 172.21.15.31:0/124769618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561df9b11820 0x561df9b13b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:39.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.154+0000 7fa7b3400640 1 -- 172.21.15.31:0/124769618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561df9b11820 msgr2=0x561df9b13b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:39.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.154+0000 7fa7b3400640 1 --2- 172.21.15.31:0/124769618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561df9b11820 0x561df9b13b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.154+0000 7fa7b3400640 1 -- 172.21.15.31:0/124769618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561df9b02a50 msgr2=0x561df9b13650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:39.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.154+0000 7fa7b3400640 1 --2- 172.21.15.31:0/124769618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561df9b02a50 0x561df9b13650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.154+0000 7fa7b3400640 1 -- 172.21.15.31:0/124769618 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561df9b140d0 con 0x561df9b02090 2024-03-21T10:44:39.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.154+0000 7fa7b2a00640 1 --2- 172.21.15.31:0/124769618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561df9b02a50 0x561df9b13650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:44:39.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.155+0000 7fa7b3400640 1 --2- 172.21.15.31:0/124769618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561df9b02090 0x561df9b1e9e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4011e70 tx=0x7fa7a401bb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62f726b2143ff336 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:39.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.155+0000 7fa79cc00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:44:39.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.155+0000 7fa79cc00640 1 -- 172.21.15.31:0/124769618 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4036030 con 0x561df9b02090 2024-03-21T10:44:39.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.155+0000 7fa79cc00640 1 -- 172.21.15.31:0/124769618 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fa7a4011890 con 0x561df9b02090 2024-03-21T10:44:39.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.155+0000 7fa7bcd6b740 1 -- 172.21.15.31:0/124769618 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa7a4011a90 con 0x561df9b02090 2024-03-21T10:44:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.155+0000 7fa79cc00640 1 -- 172.21.15.31:0/124769618 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7a4041050 con 0x561df9b02090 2024-03-21T10:44:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.156+0000 7fa7bcd6b740 10 client.27110 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.156+0000 7fa7bcd6b740 1 -- 172.21.15.31:0/124769618 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x561df9b830a0 con 0x561df9b02090 2024-03-21T10:44:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.156+0000 7fa79cc00640 1 -- 172.21.15.31:0/124769618 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7a4011a90 con 0x561df9b02090 2024-03-21T10:44:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.156+0000 7fa79cc00640 1 -- 172.21.15.31:0/124769618 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa7a4029490 con 0x561df9b02090 2024-03-21T10:44:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.156+0000 7fa7bcd6b740 10 client.27110 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.156+0000 7fa7bcd6b740 10 client.27110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:44:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.156+0000 7fa7bcd6b740 20 client.27110 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.156+0000 7fa7bcd6b740 1 --2- 172.21.15.31:0/124769618 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561df9b83700 0x561df9b85b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:39.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.156+0000 7fa7bcd6b740 4 client.27110 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:44:39.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.156+0000 7fa7bcd6b740 1 -- 172.21.15.31:0/124769618 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561df9a70ae0 con 0x561df9b83700 2024-03-21T10:44:39.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.157+0000 7fa7b2a00640 1 --2- 172.21.15.31:0/124769618 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561df9b83700 0x561df9b85b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:39.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.160+0000 7fa7b2a00640 1 --2- 172.21.15.31:0/124769618 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561df9b83700 0x561df9b85b20 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6247a6c713130d1 server_cookie=89eb20b12fb45f2d in_seq=0 out_seq=0 2024-03-21T10:44:39.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.160+0000 7fa79cc00640 10 client.27110 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:39.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.160+0000 7fa79cc00640 1 -- 172.21.15.31:0/124769618 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x561df9a70ae0 con 0x561df9b83700 2024-03-21T10:44:39.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.160+0000 7fa79cc00640 10 client.27110 handle_command_reply: tid=0 2024-03-21T10:44:39.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.160+0000 7fa7bcd6b740 1 client.27110 shutdown 2024-03-21T10:44:39.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.164+0000 7fa7bcd6b740 1 -- 172.21.15.31:0/124769618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561df9b02090 msgr2=0x561df9b1e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:39.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.164+0000 7fa7bcd6b740 1 --2- 172.21.15.31:0/124769618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561df9b02090 0x561df9b1e9e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4011e70 tx=0x7fa7a401bb60 comp rx=0 tx=0).stop 2024-03-21T10:44:39.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.164+0000 7fa7bcd6b740 1 -- 172.21.15.31:0/124769618 shutdown_connections 2024-03-21T10:44:39.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.164+0000 7fa7bcd6b740 1 --2- 172.21.15.31:0/124769618 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561df9b83700 0x561df9b85b20 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.164+0000 7fa7bcd6b740 1 --2- 172.21.15.31:0/124769618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561df9b11820 0x561df9b13b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.164+0000 7fa7bcd6b740 1 --2- 172.21.15.31:0/124769618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561df9b02a50 0x561df9b13650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.164+0000 7fa7bcd6b740 1 --2- 172.21.15.31:0/124769618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561df9b02090 0x561df9b1e9e0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.164+0000 7fa7bcd6b740 1 -- 172.21.15.31:0/124769618 >> 172.21.15.31:0/124769618 conn(0x561df9acc200 msgr2=0x561df9b102b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:39.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.164+0000 7fa79cc00640 0 client.27110 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:39.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.165+0000 7fa7bcd6b740 1 -- 172.21.15.31:0/124769618 shutdown_connections 2024-03-21T10:44:39.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.165+0000 7fa7bcd6b740 1 -- 172.21.15.31:0/124769618 wait complete. 2024-03-21T10:44:39.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.165+0000 7fa7bcd6b740 20 client.27110 trim_cache size 0 max 16384 2024-03-21T10:44:39.166 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:44:39.167 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-03-21T10:44:39.167 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-03-21T10:44:39.167 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:44:39.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.174+0000 7fa7b8a00640 1 -- 172.21.15.31:0/1155712786 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa7980a3170 msgr2=0x7fa7980a5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:39.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.174+0000 7fa7b8a00640 1 --2- 172.21.15.31:0/1155712786 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa7980a3170 0x7fa7980a5630 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac009dd0 tx=0x7fa7ac011600 comp rx=0 tx=0).stop 2024-03-21T10:44:39.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.174+0000 7fa7b8a00640 1 -- 172.21.15.31:0/1155712786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7b4101e60 msgr2=0x7fa7b4198360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:39.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.174+0000 7fa7b8a00640 1 --2- 172.21.15.31:0/1155712786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7b4101e60 0x7fa7b4198360 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0000c00 tx=0x7fa7a0004550 comp rx=0 tx=0).stop 2024-03-21T10:44:39.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.175+0000 7fa7b8a00640 1 -- 172.21.15.31:0/1155712786 shutdown_connections 2024-03-21T10:44:39.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.175+0000 7fa7b8a00640 1 --2- 172.21.15.31:0/1155712786 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa7980a3170 0x7fa7980a5630 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.175+0000 7fa7b8a00640 1 --2- 172.21.15.31:0/1155712786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7b4107560 0x7fa7b419c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.175+0000 7fa7b8a00640 1 --2- 172.21.15.31:0/1155712786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7b4101e60 0x7fa7b4198360 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.175+0000 7fa7b8a00640 1 --2- 172.21.15.31:0/1155712786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7b41014a0 0x7fa7b4197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.175+0000 7fa7b8a00640 1 -- 172.21.15.31:0/1155712786 >> 172.21.15.31:0/1155712786 conn(0x7fa7b40fcfb0 msgr2=0x7fa7b40fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:39.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.177+0000 7fa7b8a00640 1 -- 172.21.15.31:0/1155712786 shutdown_connections 2024-03-21T10:44:39.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.177+0000 7fa7b8a00640 1 -- 172.21.15.31:0/1155712786 wait complete. 2024-03-21T10:44:39.202 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:39 smithi031 ceph-mon[31673]: pgmap v185: 129 pgs: 129 active+clean; 38 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 184 MiB/s wr, 1.49k op/s 2024-03-21T10:44:39.207 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-21T10:44:39.207 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:44:39.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:39 smithi040 ceph-mon[36154]: pgmap v185: 129 pgs: 129 active+clean; 38 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 184 MiB/s wr, 1.49k op/s 2024-03-21T10:44:39.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.517+0000 7f7c7e200640 1 -- 172.21.15.31:0/4145196590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c781006b0 msgr2=0x7f7c78100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:39.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.517+0000 7f7c7e200640 1 --2- 172.21.15.31:0/4145196590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c781006b0 0x7f7c78100a90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7c70009cd0 tx=0x7f7c7002c1c0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.522+0000 7f7c7e200640 1 -- 172.21.15.31:0/4145196590 shutdown_connections 2024-03-21T10:44:39.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.522+0000 7f7c7e200640 1 --2- 172.21.15.31:0/4145196590 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c78105ab0 0x7f7c78105f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.522+0000 7f7c7e200640 1 --2- 172.21.15.31:0/4145196590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c78100fd0 0x7f7c78105490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.522+0000 7f7c7e200640 1 --2- 172.21.15.31:0/4145196590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c781006b0 0x7f7c78100a90 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.522+0000 7f7c7e200640 1 -- 172.21.15.31:0/4145196590 >> 172.21.15.31:0/4145196590 conn(0x7f7c78077720 msgr2=0x7f7c78077b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:39.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.523+0000 7f7c7e200640 1 -- 172.21.15.31:0/4145196590 shutdown_connections 2024-03-21T10:44:39.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.524+0000 7f7c7e200640 1 -- 172.21.15.31:0/4145196590 wait complete. 2024-03-21T10:44:39.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.526+0000 7f7c7e200640 1 Processor -- start 2024-03-21T10:44:39.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.527+0000 7f7c7e200640 1 -- start start 2024-03-21T10:44:39.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.528+0000 7f7c7e200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c781006b0 0x7f7c78072ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:39.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.528+0000 7f7c7e200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c78100fd0 0x7f7c7806bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:39.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.528+0000 7f7c7e200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c78105ab0 0x7f7c7806c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:39.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.528+0000 7f7c7e200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c78118060 con 0x7f7c78105ab0 2024-03-21T10:44:39.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.528+0000 7f7c7e200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c78117ee0 con 0x7f7c78100fd0 2024-03-21T10:44:39.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.528+0000 7f7c7e200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c781181e0 con 0x7f7c781006b0 2024-03-21T10:44:39.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.528+0000 7f7c76a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c78100fd0 0x7f7c7806bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:39.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.528+0000 7f7c76a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c78100fd0 0x7f7c7806bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37300/0 (socket says 172.21.15.31:37300) 2024-03-21T10:44:39.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.528+0000 7f7c76a00640 1 -- 172.21.15.31:0/2547651262 learned_addr learned my addr 172.21.15.31:0/2547651262 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:39.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.528+0000 7f7c77400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c781006b0 0x7f7c78072ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:39.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.528+0000 7f7c77e00640 1 --2- 172.21.15.31:0/2547651262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c78105ab0 0x7f7c7806c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:39.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.529+0000 7f7c77400640 1 -- 172.21.15.31:0/2547651262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c78100fd0 msgr2=0x7f7c7806bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:39.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.529+0000 7f7c77400640 1 --2- 172.21.15.31:0/2547651262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c78100fd0 0x7f7c7806bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.529+0000 7f7c77400640 1 -- 172.21.15.31:0/2547651262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c78105ab0 msgr2=0x7f7c7806c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:39.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.529+0000 7f7c77400640 1 --2- 172.21.15.31:0/2547651262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c78105ab0 0x7f7c7806c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:39.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.529+0000 7f7c77400640 1 -- 172.21.15.31:0/2547651262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7806cac0 con 0x7f7c781006b0 2024-03-21T10:44:39.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.530+0000 7f7c77400640 1 --2- 172.21.15.31:0/2547651262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c781006b0 0x7f7c78072ec0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7c70009bf0 tx=0x7f7c70035d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:39.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.531+0000 7f7c6fe00640 1 -- 172.21.15.31:0/2547651262 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c70039040 con 0x7f7c781006b0 2024-03-21T10:44:39.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.531+0000 7f7c6fe00640 1 -- 172.21.15.31:0/2547651262 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c70003e50 con 0x7f7c781006b0 2024-03-21T10:44:39.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.531+0000 7f7c6fe00640 1 -- 172.21.15.31:0/2547651262 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c70032860 con 0x7f7c781006b0 2024-03-21T10:44:39.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.532+0000 7f7c7e200640 1 -- 172.21.15.31:0/2547651262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c7806cd50 con 0x7f7c781006b0 2024-03-21T10:44:39.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.532+0000 7f7c7e200640 1 -- 172.21.15.31:0/2547651262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c78070c20 con 0x7f7c781006b0 2024-03-21T10:44:39.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.535+0000 7f7c7e200640 1 -- 172.21.15.31:0/2547651262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c780691b0 con 0x7f7c781006b0 2024-03-21T10:44:39.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.536+0000 7f7c6fe00640 1 -- 172.21.15.31:0/2547651262 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f7c70046030 con 0x7f7c781006b0 2024-03-21T10:44:39.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.538+0000 7f7c6fe00640 1 --2- 172.21.15.31:0/2547651262 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7c5c0a31d0 0x7f7c5c0a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:39.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.538+0000 7f7c6fe00640 1 -- 172.21.15.31:0/2547651262 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f7c700f11e0 con 0x7f7c781006b0 2024-03-21T10:44:39.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.539+0000 7f7c76a00640 1 --2- 172.21.15.31:0/2547651262 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7c5c0a31d0 0x7f7c5c0a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:39.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.544+0000 7f7c6fe00640 1 -- 172.21.15.31:0/2547651262 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7c7004d060 con 0x7f7c781006b0 2024-03-21T10:44:39.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:39.545+0000 7f7c76a00640 1 --2- 172.21.15.31:0/2547651262 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7c5c0a31d0 0x7f7c5c0a5690 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7c7806d700 tx=0x7f7c6800b4a0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:39.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:39 smithi070 ceph-mon[36330]: pgmap v185: 129 pgs: 129 active+clean; 38 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 184 MiB/s wr, 1.49k op/s 2024-03-21T10:44:40.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.192+0000 7f7c7e200640 1 -- 172.21.15.31:0/2547651262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c78070f40 con 0x7f7c781006b0 2024-03-21T10:44:40.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.194+0000 7f7c6fe00640 1 -- 172.21.15.31:0/2547651262 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+11384 (secure 0 0 0) 0x7f7c700bdad0 con 0x7f7c781006b0 2024-03-21T10:44:40.196 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:44:40.196 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:42:31.182549+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24547,24596]},"id":1}]} 2024-03-21T10:44:40.197 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2024-03-21T10:44:40.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.199+0000 7f7c7e200640 1 -- 172.21.15.31:0/2547651262 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7c5c0a31d0 msgr2=0x7f7c5c0a5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:40.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.199+0000 7f7c7e200640 1 --2- 172.21.15.31:0/2547651262 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7c5c0a31d0 0x7f7c5c0a5690 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7c7806d700 tx=0x7f7c6800b4a0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.199+0000 7f7c7e200640 1 -- 172.21.15.31:0/2547651262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c781006b0 msgr2=0x7f7c78072ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:40.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.199+0000 7f7c7e200640 1 --2- 172.21.15.31:0/2547651262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c781006b0 0x7f7c78072ec0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7c70009bf0 tx=0x7f7c70035d80 comp rx=0 tx=0).stop 2024-03-21T10:44:40.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.199+0000 7f7c7e200640 1 -- 172.21.15.31:0/2547651262 shutdown_connections 2024-03-21T10:44:40.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.199+0000 7f7c7e200640 1 --2- 172.21.15.31:0/2547651262 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7c5c0a31d0 0x7f7c5c0a5690 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.199+0000 7f7c7e200640 1 --2- 172.21.15.31:0/2547651262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c78105ab0 0x7f7c7806c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.199+0000 7f7c7e200640 1 --2- 172.21.15.31:0/2547651262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c78100fd0 0x7f7c7806bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.199+0000 7f7c7e200640 1 --2- 172.21.15.31:0/2547651262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c781006b0 0x7f7c78072ec0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.199+0000 7f7c7e200640 1 -- 172.21.15.31:0/2547651262 >> 172.21.15.31:0/2547651262 conn(0x7f7c78077720 msgr2=0x7f7c78103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:40.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.199+0000 7f7c7e200640 1 -- 172.21.15.31:0/2547651262 shutdown_connections 2024-03-21T10:44:40.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.199+0000 7f7c7e200640 1 -- 172.21.15.31:0/2547651262 wait complete. 2024-03-21T10:44:40.240 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:44:40.240 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-21T10:44:40.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.502+0000 7f9acde00640 1 -- 172.21.15.31:0/234098032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ac8100550 msgr2=0x7f9ac81009d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:40.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.502+0000 7f9acde00640 1 --2- 172.21.15.31:0/234098032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ac8100550 0x7f9ac81009d0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4009930 tx=0x7f9ab402bc30 comp rx=0 tx=0).stop 2024-03-21T10:44:40.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.502+0000 7f9acde00640 1 -- 172.21.15.31:0/234098032 shutdown_connections 2024-03-21T10:44:40.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.502+0000 7f9acde00640 1 --2- 172.21.15.31:0/234098032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ac8100f10 0x7f9ac810d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.502+0000 7f9acde00640 1 --2- 172.21.15.31:0/234098032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ac8100550 0x7f9ac81009d0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.502+0000 7f9acde00640 1 --2- 172.21.15.31:0/234098032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ac8106600 0x7f9ac81069e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.502+0000 7f9acde00640 1 -- 172.21.15.31:0/234098032 >> 172.21.15.31:0/234098032 conn(0x7f9ac80776d0 msgr2=0x7f9ac8077ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:40.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.502+0000 7f9acde00640 1 -- 172.21.15.31:0/234098032 shutdown_connections 2024-03-21T10:44:40.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.502+0000 7f9acde00640 1 -- 172.21.15.31:0/234098032 wait complete. 2024-03-21T10:44:40.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.503+0000 7f9acde00640 1 Processor -- start 2024-03-21T10:44:40.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.504+0000 7f9acde00640 1 -- start start 2024-03-21T10:44:40.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.504+0000 7f9acde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ac8100550 0x7f9ac819a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:40.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.504+0000 7f9acde00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ac8100f10 0x7f9ac819a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:40.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.504+0000 7f9acde00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ac8106600 0x7f9ac819e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:40.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.504+0000 7f9acde00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac81108c0 con 0x7f9ac8100f10 2024-03-21T10:44:40.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.504+0000 7f9acde00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac8110740 con 0x7f9ac8100550 2024-03-21T10:44:40.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.504+0000 7f9acde00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac8110a40 con 0x7f9ac8106600 2024-03-21T10:44:40.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.505+0000 7f9ac6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ac8100550 0x7f9ac819a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:40.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.505+0000 7f9ac6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ac8100550 0x7f9ac819a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37314/0 (socket says 172.21.15.31:37314) 2024-03-21T10:44:40.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.505+0000 7f9ac6a00640 1 -- 172.21.15.31:0/726987598 learned_addr learned my addr 172.21.15.31:0/726987598 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:40.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.505+0000 7f9ac7400640 1 --2- 172.21.15.31:0/726987598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ac8106600 0x7f9ac819e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:40.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.505+0000 7f9ac6000640 1 --2- 172.21.15.31:0/726987598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ac8100f10 0x7f9ac819a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:40.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.505+0000 7f9ac6a00640 1 -- 172.21.15.31:0/726987598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ac8106600 msgr2=0x7f9ac819e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:40.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.505+0000 7f9ac6a00640 1 --2- 172.21.15.31:0/726987598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ac8106600 0x7f9ac819e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.505+0000 7f9ac6a00640 1 -- 172.21.15.31:0/726987598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ac8100f10 msgr2=0x7f9ac819a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:40.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.505+0000 7f9ac6a00640 1 --2- 172.21.15.31:0/726987598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ac8100f10 0x7f9ac819a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.505+0000 7f9ac6a00640 1 -- 172.21.15.31:0/726987598 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac819f0e0 con 0x7f9ac8100550 2024-03-21T10:44:40.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.505+0000 7f9ac6a00640 1 --2- 172.21.15.31:0/726987598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ac8100550 0x7f9ac819a0f0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f9ab800e9f0 tx=0x7f9ab800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:40.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.506+0000 7f9acca00640 1 -- 172.21.15.31:0/726987598 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab800cd80 con 0x7f9ac8100550 2024-03-21T10:44:40.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.506+0000 7f9acde00640 1 -- 172.21.15.31:0/726987598 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac819f3d0 con 0x7f9ac8100550 2024-03-21T10:44:40.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.506+0000 7f9acca00640 1 -- 172.21.15.31:0/726987598 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ab8010910 con 0x7f9ac8100550 2024-03-21T10:44:40.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.506+0000 7f9acca00640 1 -- 172.21.15.31:0/726987598 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab80188f0 con 0x7f9ac8100550 2024-03-21T10:44:40.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.506+0000 7f9acde00640 1 -- 172.21.15.31:0/726987598 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac81a6c60 con 0x7f9ac8100550 2024-03-21T10:44:40.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.510+0000 7f9ad0bce740 1 Processor -- start 2024-03-21T10:44:40.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.511+0000 7f9acca00640 1 -- 172.21.15.31:0/726987598 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f9ab8010ac0 con 0x7f9ac8100550 2024-03-21T10:44:40.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.512+0000 7f9ad0bce740 1 -- start start 2024-03-21T10:44:40.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.513+0000 7f9ad0bce740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560dd3a76ee0 0x560dd3a772c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:40.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.513+0000 7f9ad0bce740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560dd3a77890 0x560dd3a7fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:40.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.513+0000 7f9ad0bce740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dd3a80410 0x560dd3a82860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:40.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.513+0000 7f9ad0bce740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560dd39d22c0 con 0x560dd3a76ee0 2024-03-21T10:44:40.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.513+0000 7f9ad0bce740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560dd397ac90 con 0x560dd3a77890 2024-03-21T10:44:40.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.513+0000 7f9ad0bce740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x560dd39ce360 con 0x560dd3a80410 2024-03-21T10:44:40.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.513+0000 7f9acca00640 1 --2- 172.21.15.31:0/726987598 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9aac0a3110 0x7f9aac0a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:40.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.513+0000 7f9ac7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560dd3a76ee0 0x560dd3a772c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:40.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.513+0000 7f9acca00640 1 -- 172.21.15.31:0/726987598 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f9ab80cff40 con 0x7f9ac8100550 2024-03-21T10:44:40.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.514+0000 7f9ac6000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dd3a80410 0x560dd3a82860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:40.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.514+0000 7f9ac6000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dd3a80410 0x560dd3a82860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60006/0 (socket says 172.21.15.31:60006) 2024-03-21T10:44:40.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.514+0000 7f9ac6000640 1 -- 172.21.15.31:0/2194541736 learned_addr learned my addr 172.21.15.31:0/2194541736 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:40.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.514+0000 7f9ac6a00640 1 --2- 172.21.15.31:0/2194541736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560dd3a77890 0x560dd3a7fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:40.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.514+0000 7f9ac6000640 1 -- 172.21.15.31:0/2194541736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560dd3a77890 msgr2=0x560dd3a7fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:40.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.514+0000 7f9ac6000640 1 --2- 172.21.15.31:0/2194541736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560dd3a77890 0x560dd3a7fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.514+0000 7f9ac7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560dd3a76ee0 0x560dd3a772c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41634/0 (socket says 172.21.15.31:41634) 2024-03-21T10:44:40.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.514+0000 7f9ac6000640 1 -- 172.21.15.31:0/2194541736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560dd3a76ee0 msgr2=0x560dd3a772c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:40.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.514+0000 7f9ac6000640 1 --2- 172.21.15.31:0/2194541736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560dd3a76ee0 0x560dd3a772c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.514+0000 7f9ac6000640 1 -- 172.21.15.31:0/2194541736 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560dd399f080 con 0x560dd3a80410 2024-03-21T10:44:40.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.515+0000 7f9ac7400640 1 --2- 172.21.15.31:0/2194541736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560dd3a76ee0 0x560dd3a772c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:44:40.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.515+0000 7f9ac7400640 1 --2- 172.21.15.31:0/726987598 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9aac0a3110 0x7f9aac0a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:40.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.515+0000 7f9ac6000640 1 --2- 172.21.15.31:0/2194541736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dd3a80410 0x560dd3a82860 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9ab402cfd0 tx=0x7f9ab40095f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddf0913f41e0ef00 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:40.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.516+0000 7f9ab3e00640 1 -- 172.21.15.31:0/2194541736 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4042070 con 0x560dd3a80410 2024-03-21T10:44:40.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.516+0000 7f9ab3e00640 1 -- 172.21.15.31:0/2194541736 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ab4004030 con 0x560dd3a80410 2024-03-21T10:44:40.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.516+0000 7f9ab3e00640 1 -- 172.21.15.31:0/2194541736 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4004c90 con 0x560dd3a80410 2024-03-21T10:44:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.516+0000 7f9ad0bce740 1 -- 172.21.15.31:0/2194541736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dd3a80410 msgr2=0x560dd3a82860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.516+0000 7f9ad0bce740 1 --2- 172.21.15.31:0/2194541736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dd3a80410 0x560dd3a82860 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9ab402cfd0 tx=0x7f9ab40095f0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.517+0000 7f9ad0bce740 1 -- 172.21.15.31:0/2194541736 shutdown_connections 2024-03-21T10:44:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.517+0000 7f9ad0bce740 1 --2- 172.21.15.31:0/2194541736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dd3a80410 0x560dd3a82860 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.517+0000 7f9ad0bce740 1 --2- 172.21.15.31:0/2194541736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560dd3a77890 0x560dd3a7fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.517+0000 7f9ad0bce740 1 --2- 172.21.15.31:0/2194541736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560dd3a76ee0 0x560dd3a772c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.517+0000 7f9ad0bce740 1 -- 172.21.15.31:0/2194541736 >> 172.21.15.31:0/2194541736 conn(0x560dd3a715b0 msgr2=0x560dd3a71990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.519+0000 7f9ad0bce740 1 -- 172.21.15.31:0/2194541736 shutdown_connections 2024-03-21T10:44:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.521+0000 7f9ad0bce740 1 -- 172.21.15.31:0/2194541736 wait complete. 2024-03-21T10:44:40.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.526+0000 7f9ad0bce740 1 -- start start 2024-03-21T10:44:40.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.526+0000 7f9ad0bce740 1 Processor -- start 2024-03-21T10:44:40.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.522+0000 7f9ac7400640 1 --2- 172.21.15.31:0/726987598 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9aac0a3110 0x7f9aac0a55d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0005fd0 tx=0x7f9ac0004a40 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:40.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.528+0000 7f9ad0bce740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dd3a76ee0 0x560dd3a7fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:40.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.528+0000 7f9ad0bce740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560dd3a77890 0x560dd3a7d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:40.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.528+0000 7f9ad0bce740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560dd3a80410 0x560dd3a7d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:40.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.529+0000 7f9ac7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560dd3a80410 0x560dd3a7d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:40.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.529+0000 7f9ac7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560dd3a80410 0x560dd3a7d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41638/0 (socket says 172.21.15.31:41638) 2024-03-21T10:44:40.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.529+0000 7f9ac7400640 1 -- 172.21.15.31:0/4176156320 learned_addr learned my addr 172.21.15.31:0/4176156320 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:40.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.529+0000 7f9ad0bce740 1 -- 172.21.15.31:0/4176156320 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560dd3a94330 con 0x560dd3a80410 2024-03-21T10:44:40.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.530+0000 7f9ad0bce740 1 -- 172.21.15.31:0/4176156320 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560dd398e0d0 con 0x560dd3a77890 2024-03-21T10:44:40.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.530+0000 7f9ad0bce740 1 -- 172.21.15.31:0/4176156320 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x560dd3a944b0 con 0x560dd3a76ee0 2024-03-21T10:44:40.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.530+0000 7f9ad0bce740 10 client.0 _ll_register_callbacks cb 0x560dd3992e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:40.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.531+0000 7f9ac7400640 1 -- 172.21.15.31:0/4176156320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dd3a76ee0 msgr2=0x560dd3a7fd10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:44:40.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.531+0000 7f9ac7400640 1 --2- 172.21.15.31:0/4176156320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dd3a76ee0 0x560dd3a7fd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.531+0000 7f9ac7400640 1 -- 172.21.15.31:0/4176156320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560dd3a77890 msgr2=0x560dd3a7d0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:44:40.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.531+0000 7f9ac7400640 1 --2- 172.21.15.31:0/4176156320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560dd3a77890 0x560dd3a7d0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.531+0000 7f9ac7400640 1 -- 172.21.15.31:0/4176156320 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560dd36e63f0 con 0x560dd3a80410 2024-03-21T10:44:40.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.532+0000 7f9ab0c00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:44:40.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.532+0000 7f9ac7400640 1 --2- 172.21.15.31:0/4176156320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560dd3a80410 0x560dd3a7d620 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0012010 tx=0x7f9ac0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4408a802187ae2f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:40.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.533+0000 7f9ab0c00640 1 -- 172.21.15.31:0/4176156320 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac0029420 con 0x560dd3a80410 2024-03-21T10:44:40.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.533+0000 7f9ab0c00640 1 -- 172.21.15.31:0/4176156320 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f9ac00295c0 con 0x560dd3a80410 2024-03-21T10:44:40.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.534+0000 7f9ad0bce740 1 -- 172.21.15.31:0/4176156320 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9ab4004030 con 0x560dd3a80410 2024-03-21T10:44:40.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.534+0000 7f9ab0c00640 1 -- 172.21.15.31:0/4176156320 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ac00297c0 con 0x560dd3a80410 2024-03-21T10:44:40.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.535+0000 7f9ad0bce740 10 client.16599 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:40.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.535+0000 7f9ad0bce740 1 -- 172.21.15.31:0/4176156320 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x560dd37b07d0 con 0x560dd3a80410 2024-03-21T10:44:40.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.536+0000 7f9ab0c00640 1 -- 172.21.15.31:0/4176156320 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ab4004030 con 0x560dd3a80410 2024-03-21T10:44:40.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.536+0000 7f9ab0c00640 1 -- 172.21.15.31:0/4176156320 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f9ac0033b90 con 0x560dd3a80410 2024-03-21T10:44:40.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.536+0000 7f9ad0bce740 10 client.16599 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:40.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.536+0000 7f9ad0bce740 10 client.16599 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:44:40.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.536+0000 7f9ad0bce740 20 client.16599 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:40.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.536+0000 7f9ad0bce740 1 --2- 172.21.15.31:0/4176156320 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x560dd3aa1240 0x560dd3aa3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:40.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.536+0000 7f9ad0bce740 4 client.16599 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:44:40.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.536+0000 7f9ad0bce740 1 -- 172.21.15.31:0/4176156320 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560dd39df800 con 0x560dd3aa1240 2024-03-21T10:44:40.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.538+0000 7f9ac6a00640 1 --2- 172.21.15.31:0/4176156320 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x560dd3aa1240 0x560dd3aa3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:40.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.541+0000 7f9ab0c00640 10 client.16599 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:44:40.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.541+0000 7f9ac6a00640 1 --2- 172.21.15.31:0/4176156320 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x560dd3aa1240 0x560dd3aa3690 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e617d7e5b66296fe server_cookie=491676fb9f6268c4 in_seq=0 out_seq=0 2024-03-21T10:44:40.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.543+0000 7f9ab0c00640 1 -- 172.21.15.31:0/4176156320 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560dd39df800 con 0x560dd3aa1240 2024-03-21T10:44:40.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.543+0000 7f9ab0c00640 10 client.16599 handle_command_reply: tid=0 2024-03-21T10:44:40.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.547+0000 7f9ad0bce740 1 client.16599 shutdown 2024-03-21T10:44:40.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.549+0000 7f9ad0bce740 1 -- 172.21.15.31:0/4176156320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560dd3a80410 msgr2=0x560dd3a7d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:40.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.549+0000 7f9ad0bce740 1 --2- 172.21.15.31:0/4176156320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560dd3a80410 0x560dd3a7d620 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0012010 tx=0x7f9ac0004ab0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.552+0000 7f9ad0bce740 1 -- 172.21.15.31:0/4176156320 shutdown_connections 2024-03-21T10:44:40.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.552+0000 7f9ad0bce740 1 --2- 172.21.15.31:0/4176156320 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x560dd3aa1240 0x560dd3aa3690 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.552+0000 7f9ad0bce740 1 --2- 172.21.15.31:0/4176156320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560dd3a80410 0x560dd3a7d620 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.552+0000 7f9ad0bce740 1 --2- 172.21.15.31:0/4176156320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560dd3a77890 0x560dd3a7d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.552+0000 7f9ad0bce740 1 --2- 172.21.15.31:0/4176156320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dd3a76ee0 0x560dd3a7fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.552+0000 7f9ad0bce740 1 -- 172.21.15.31:0/4176156320 >> 172.21.15.31:0/4176156320 conn(0x560dd3a715b0 msgr2=0x560dd3a73300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.552+0000 7f9ab0c00640 0 client.16599 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:44:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.552+0000 7f9ad0bce740 1 -- 172.21.15.31:0/4176156320 shutdown_connections 2024-03-21T10:44:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.552+0000 7f9ad0bce740 1 -- 172.21.15.31:0/4176156320 wait complete. 2024-03-21T10:44:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.552+0000 7f9ad0bce740 20 client.16599 trim_cache size 0 max 16384 2024-03-21T10:44:40.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.611+0000 7f9ad0bce740 1 Processor -- start 2024-03-21T10:44:40.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.612+0000 7f9ad0bce740 1 -- start start 2024-03-21T10:44:40.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.612+0000 7f9ad0bce740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560dd3a76ee0 0x560dd3a67d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:40.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.612+0000 7f9ad0bce740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dd3a77890 0x560dd3a7f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:40.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.612+0000 7f9ad0bce740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560dd3a80410 0x560dd3a7fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:40.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.612+0000 7f9ad0bce740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac8072ec0 con 0x560dd3a76ee0 2024-03-21T10:44:40.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.612+0000 7f9ad0bce740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac80709c0 con 0x560dd3a80410 2024-03-21T10:44:40.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.612+0000 7f9ad0bce740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac806eac0 con 0x560dd3a77890 2024-03-21T10:44:40.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.613+0000 7f9ac7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560dd3a80410 0x560dd3a7fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:40.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.613+0000 7f9ac6000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dd3a77890 0x560dd3a7f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:40.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.613+0000 7f9ac7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560dd3a80410 0x560dd3a7fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37320/0 (socket says 172.21.15.31:37320) 2024-03-21T10:44:40.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.613+0000 7f9ac6000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dd3a77890 0x560dd3a7f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60018/0 (socket says 172.21.15.31:60018) 2024-03-21T10:44:40.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.613+0000 7f9ac7400640 1 -- 172.21.15.31:0/1477475196 learned_addr learned my addr 172.21.15.31:0/1477475196 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:40.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.613+0000 7f9ac6a00640 1 --2- 172.21.15.31:0/1477475196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560dd3a76ee0 0x560dd3a67d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:40.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.613+0000 7f9ac6000640 1 -- 172.21.15.31:0/1477475196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560dd3a80410 msgr2=0x560dd3a7fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:40.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.613+0000 7f9ac6000640 1 --2- 172.21.15.31:0/1477475196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560dd3a80410 0x560dd3a7fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.613+0000 7f9ac6000640 1 -- 172.21.15.31:0/1477475196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560dd3a76ee0 msgr2=0x560dd3a67d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:40.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.613+0000 7f9ac6000640 1 --2- 172.21.15.31:0/1477475196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560dd3a76ee0 0x560dd3a67d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.613+0000 7f9ac6000640 1 -- 172.21.15.31:0/1477475196 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560dd3a7c200 con 0x560dd3a77890 2024-03-21T10:44:40.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.613+0000 7f9ac7400640 1 --2- 172.21.15.31:0/1477475196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560dd3a80410 0x560dd3a7fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:44:40.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.613+0000 7f9ac6a00640 1 --2- 172.21.15.31:0/1477475196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560dd3a76ee0 0x560dd3a67d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:44:40.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.613+0000 7f9ac6000640 1 --2- 172.21.15.31:0/1477475196 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dd3a77890 0x560dd3a7f7d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f9ab402c170 tx=0x7f9ab4005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95d954e60746a1bc server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:40.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.614+0000 7f9ab3400640 1 -- 172.21.15.31:0/1477475196 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4042070 con 0x560dd3a77890 2024-03-21T10:44:40.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.614+0000 7f9ab3400640 1 -- 172.21.15.31:0/1477475196 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ab4004030 con 0x560dd3a77890 2024-03-21T10:44:40.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.614+0000 7f9ab3400640 1 -- 172.21.15.31:0/1477475196 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4004a60 con 0x560dd3a77890 2024-03-21T10:44:40.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.615+0000 7f9ad0bce740 1 -- 172.21.15.31:0/1477475196 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dd3a77890 msgr2=0x560dd3a7f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:40.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.615+0000 7f9ad0bce740 1 --2- 172.21.15.31:0/1477475196 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dd3a77890 0x560dd3a7f7d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f9ab402c170 tx=0x7f9ab4005b50 comp rx=0 tx=0).stop 2024-03-21T10:44:40.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.615+0000 7f9ad0bce740 1 -- 172.21.15.31:0/1477475196 shutdown_connections 2024-03-21T10:44:40.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.615+0000 7f9ad0bce740 1 --2- 172.21.15.31:0/1477475196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560dd3a80410 0x560dd3a7fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.615+0000 7f9ad0bce740 1 --2- 172.21.15.31:0/1477475196 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dd3a77890 0x560dd3a7f7d0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.615+0000 7f9ad0bce740 1 --2- 172.21.15.31:0/1477475196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560dd3a76ee0 0x560dd3a67d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.615+0000 7f9ad0bce740 1 -- 172.21.15.31:0/1477475196 >> 172.21.15.31:0/1477475196 conn(0x560dd3a715b0 msgr2=0x560dd3a67620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:40.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.615+0000 7f9ad0bce740 1 -- 172.21.15.31:0/1477475196 shutdown_connections 2024-03-21T10:44:40.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.615+0000 7f9ad0bce740 1 -- 172.21.15.31:0/1477475196 wait complete. 2024-03-21T10:44:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.616+0000 7f9ad0bce740 1 -- start start 2024-03-21T10:44:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.616+0000 7f9ad0bce740 1 Processor -- start 2024-03-21T10:44:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.617+0000 7f9ad0bce740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560dd3a76ee0 0x560dd3a98e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.617+0000 7f9ad0bce740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560dd3a77890 0x560dd3a99350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.617+0000 7f9ad0bce740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dd3a80410 0x560dd3a97340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.617+0000 7f9ad0bce740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac806f0c0 con 0x560dd3a77890 2024-03-21T10:44:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.617+0000 7f9ad0bce740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac8070360 con 0x560dd3a76ee0 2024-03-21T10:44:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.617+0000 7f9ad0bce740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac806f6c0 con 0x560dd3a80410 2024-03-21T10:44:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.617+0000 7f9ad0bce740 10 client.0 _ll_register_callbacks cb 0x7f9ac0009d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.617+0000 7f9ac6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560dd3a76ee0 0x560dd3a98e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:40.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.617+0000 7f9ac6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560dd3a76ee0 0x560dd3a98e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37322/0 (socket says 172.21.15.31:37322) 2024-03-21T10:44:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.617+0000 7f9ac6a00640 1 -- 172.21.15.31:0/4145234775 learned_addr learned my addr 172.21.15.31:0/4145234775 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.617+0000 7f9ac6000640 1 --2- 172.21.15.31:0/4145234775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560dd3a77890 0x560dd3a99350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.617+0000 7f9ac7400640 1 --2- 172.21.15.31:0/4145234775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dd3a80410 0x560dd3a97340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.617+0000 7f9ac6a00640 1 -- 172.21.15.31:0/4145234775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dd3a80410 msgr2=0x560dd3a97340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.617+0000 7f9ac6a00640 1 --2- 172.21.15.31:0/4145234775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dd3a80410 0x560dd3a97340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.617+0000 7f9ac6a00640 1 -- 172.21.15.31:0/4145234775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560dd3a77890 msgr2=0x560dd3a99350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.617+0000 7f9ac6a00640 1 --2- 172.21.15.31:0/4145234775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560dd3a77890 0x560dd3a99350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.617+0000 7f9ac6a00640 1 -- 172.21.15.31:0/4145234775 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560dd3a979e0 con 0x560dd3a76ee0 2024-03-21T10:44:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.618+0000 7f9ac6a00640 1 --2- 172.21.15.31:0/4145234775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560dd3a76ee0 0x560dd3a98e10 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9ab800e9c0 tx=0x7f9ab800cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86625735ab0feb5a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.618+0000 7f9ab0c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:44:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.618+0000 7f9ab0c00640 1 -- 172.21.15.31:0/4145234775 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab80986b0 con 0x560dd3a76ee0 2024-03-21T10:44:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.618+0000 7f9ad0bce740 1 -- 172.21.15.31:0/4145234775 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9ab4004030 con 0x560dd3a76ee0 2024-03-21T10:44:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.618+0000 7f9ab0c00640 1 -- 172.21.15.31:0/4145234775 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f9ab8096bb0 con 0x560dd3a76ee0 2024-03-21T10:44:40.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.618+0000 7f9ab0c00640 1 -- 172.21.15.31:0/4145234775 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ab8098850 con 0x560dd3a76ee0 2024-03-21T10:44:40.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.618+0000 7f9ad0bce740 10 client.27128 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:40.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.618+0000 7f9ad0bce740 1 -- 172.21.15.31:0/4145234775 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x560dd3afd5e0 con 0x560dd3a76ee0 2024-03-21T10:44:40.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.619+0000 7f9ab0c00640 1 -- 172.21.15.31:0/4145234775 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ab4004030 con 0x560dd3a76ee0 2024-03-21T10:44:40.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.619+0000 7f9ab0c00640 1 -- 172.21.15.31:0/4145234775 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f9ab809b490 con 0x560dd3a76ee0 2024-03-21T10:44:40.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.619+0000 7f9ad0bce740 10 client.27128 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:40.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.619+0000 7f9ad0bce740 10 client.27128 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:44:40.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.619+0000 7f9ad0bce740 20 client.27128 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:40.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.619+0000 7f9ad0bce740 1 --2- 172.21.15.31:0/4145234775 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x560dd3afddb0 0x560dd3b001d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:40.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.619+0000 7f9ad0bce740 4 client.27128 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:44:40.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.619+0000 7f9ad0bce740 1 -- 172.21.15.31:0/4145234775 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560dd39d9b50 con 0x560dd3afddb0 2024-03-21T10:44:40.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.620+0000 7f9ac6000640 1 --2- 172.21.15.31:0/4145234775 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x560dd3afddb0 0x560dd3b001d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:40.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.620+0000 7f9ac6000640 1 --2- 172.21.15.31:0/4145234775 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x560dd3afddb0 0x560dd3b001d0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=25155bb5261acaa8 server_cookie=6858bce59251be1e in_seq=0 out_seq=0 2024-03-21T10:44:40.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.620+0000 7f9ab0c00640 10 client.27128 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:44:40.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.621+0000 7f9ab0c00640 1 -- 172.21.15.31:0/4145234775 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560dd39d9b50 con 0x560dd3afddb0 2024-03-21T10:44:40.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.621+0000 7f9ab0c00640 10 client.27128 handle_command_reply: tid=0 2024-03-21T10:44:40.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.621+0000 7f9ad0bce740 1 client.27128 shutdown 2024-03-21T10:44:40.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.621+0000 7f9ad0bce740 1 -- 172.21.15.31:0/4145234775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560dd3a76ee0 msgr2=0x560dd3a98e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:40.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.621+0000 7f9ad0bce740 1 --2- 172.21.15.31:0/4145234775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560dd3a76ee0 0x560dd3a98e10 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9ab800e9c0 tx=0x7f9ab800cf90 comp rx=0 tx=0).stop 2024-03-21T10:44:40.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.621+0000 7f9ad0bce740 1 -- 172.21.15.31:0/4145234775 shutdown_connections 2024-03-21T10:44:40.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.621+0000 7f9ad0bce740 1 --2- 172.21.15.31:0/4145234775 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x560dd3afddb0 0x560dd3b001d0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.621+0000 7f9ad0bce740 1 --2- 172.21.15.31:0/4145234775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dd3a80410 0x560dd3a97340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.621+0000 7f9ad0bce740 1 --2- 172.21.15.31:0/4145234775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560dd3a77890 0x560dd3a99350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.621+0000 7f9ad0bce740 1 --2- 172.21.15.31:0/4145234775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560dd3a76ee0 0x560dd3a98e10 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.621+0000 7f9ab0c00640 0 client.27128 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:44:40.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.621+0000 7f9ad0bce740 1 -- 172.21.15.31:0/4145234775 >> 172.21.15.31:0/4145234775 conn(0x560dd3a715b0 msgr2=0x560dd3a67eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:40.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.622+0000 7f9ad0bce740 1 -- 172.21.15.31:0/4145234775 shutdown_connections 2024-03-21T10:44:40.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.622+0000 7f9ad0bce740 1 -- 172.21.15.31:0/4145234775 wait complete. 2024-03-21T10:44:40.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.622+0000 7f9ad0bce740 20 client.27128 trim_cache size 0 max 16384 2024-03-21T10:44:40.622 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:44:40.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.623+0000 7f9acde00640 1 -- 172.21.15.31:0/726987598 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9aac0a3110 msgr2=0x7f9aac0a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.623+0000 7f9acde00640 1 --2- 172.21.15.31:0/726987598 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9aac0a3110 0x7f9aac0a55d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0005fd0 tx=0x7f9ac0004a40 comp rx=0 tx=0).stop 2024-03-21T10:44:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.623+0000 7f9acde00640 1 -- 172.21.15.31:0/726987598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ac8100550 msgr2=0x7f9ac819a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.623+0000 7f9acde00640 1 --2- 172.21.15.31:0/726987598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ac8100550 0x7f9ac819a0f0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f9ab800e9f0 tx=0x7f9ab800eeb0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.623+0000 7f9acde00640 1 -- 172.21.15.31:0/726987598 shutdown_connections 2024-03-21T10:44:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.623+0000 7f9acde00640 1 --2- 172.21.15.31:0/726987598 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9aac0a3110 0x7f9aac0a55d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.623+0000 7f9acde00640 1 --2- 172.21.15.31:0/726987598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ac8106600 0x7f9ac819e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.623+0000 7f9acde00640 1 --2- 172.21.15.31:0/726987598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ac8100f10 0x7f9ac819a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.623+0000 7f9acde00640 1 --2- 172.21.15.31:0/726987598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ac8100550 0x7f9ac819a0f0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.623+0000 7f9acde00640 1 -- 172.21.15.31:0/726987598 >> 172.21.15.31:0/726987598 conn(0x7f9ac80776d0 msgr2=0x7f9ac8069d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.623+0000 7f9acde00640 1 -- 172.21.15.31:0/726987598 shutdown_connections 2024-03-21T10:44:40.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.624+0000 7f9acde00640 1 -- 172.21.15.31:0/726987598 wait complete. 2024-03-21T10:44:40.642 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:44:40.643 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-21T10:44:40.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.979+0000 7f6eaa600640 1 -- 172.21.15.31:0/3786697920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ea4107640 msgr2=0x7f6ea4107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:40.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.979+0000 7f6eaa600640 1 --2- 172.21.15.31:0/3786697920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ea4107640 0x7f6ea4107a20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6e9000dcc0 tx=0x7f6e9002b790 comp rx=0 tx=0).stop 2024-03-21T10:44:40.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.980+0000 7f6eaa600640 1 -- 172.21.15.31:0/3786697920 shutdown_connections 2024-03-21T10:44:40.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.980+0000 7f6eaa600640 1 --2- 172.21.15.31:0/3786697920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ea4101ff0 0x7f6ea410e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.980+0000 7f6eaa600640 1 --2- 172.21.15.31:0/3786697920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea4101630 0x7f6ea4101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.980+0000 7f6eaa600640 1 --2- 172.21.15.31:0/3786697920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ea4107640 0x7f6ea4107a20 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.980+0000 7f6eaa600640 1 -- 172.21.15.31:0/3786697920 >> 172.21.15.31:0/3786697920 conn(0x7f6ea40fd140 msgr2=0x7f6ea40ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:40.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.980+0000 7f6eaa600640 1 -- 172.21.15.31:0/3786697920 shutdown_connections 2024-03-21T10:44:40.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.983+0000 7f6eaa600640 1 -- 172.21.15.31:0/3786697920 wait complete. 2024-03-21T10:44:40.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.986+0000 7f6eaa600640 1 Processor -- start 2024-03-21T10:44:40.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.986+0000 7f6eaa600640 1 -- start start 2024-03-21T10:44:40.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.987+0000 7f6eaa600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea4101630 0x7f6ea4195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:40.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.987+0000 7f6eaa600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ea4101ff0 0x7f6ea41962c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:40.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.987+0000 7f6eaa600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ea4107640 0x7f6ea419a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:40.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.988+0000 7f6ea3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea4101630 0x7f6ea4195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:40.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.988+0000 7f6ea3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ea4107640 0x7f6ea419a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:40.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.988+0000 7f6ea3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ea4107640 0x7f6ea419a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41670/0 (socket says 172.21.15.31:41670) 2024-03-21T10:44:40.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.988+0000 7f6ea3e00640 1 -- 172.21.15.31:0/4063828831 learned_addr learned my addr 172.21.15.31:0/4063828831 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:40.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.988+0000 7f6ea2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ea4101ff0 0x7f6ea41962c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:40.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.988+0000 7f6eaa600640 1 -- 172.21.15.31:0/4063828831 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea410f8a0 con 0x7f6ea4107640 2024-03-21T10:44:40.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.988+0000 7f6eaa600640 1 -- 172.21.15.31:0/4063828831 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea410f720 con 0x7f6ea4101630 2024-03-21T10:44:40.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.988+0000 7f6eaa600640 1 -- 172.21.15.31:0/4063828831 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea410fa20 con 0x7f6ea4101ff0 2024-03-21T10:44:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.990+0000 7f6ea3e00640 1 -- 172.21.15.31:0/4063828831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ea4101ff0 msgr2=0x7f6ea41962c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.990+0000 7f6ea3e00640 1 --2- 172.21.15.31:0/4063828831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ea4101ff0 0x7f6ea41962c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.990+0000 7f6ea3e00640 1 -- 172.21.15.31:0/4063828831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea4101630 msgr2=0x7f6ea4195d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.990+0000 7f6ea3e00640 1 --2- 172.21.15.31:0/4063828831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea4101630 0x7f6ea4195d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.990+0000 7f6ea3e00640 1 -- 172.21.15.31:0/4063828831 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea419ad70 con 0x7f6ea4107640 2024-03-21T10:44:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.990+0000 7f6ea2a00640 1 --2- 172.21.15.31:0/4063828831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ea4101ff0 0x7f6ea41962c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:44:40.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.991+0000 7f6ea3e00640 1 --2- 172.21.15.31:0/4063828831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ea4107640 0x7f6ea419a640 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c00eaa0 tx=0x7f6e9c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:40.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.991+0000 7f6e9be00640 1 -- 172.21.15.31:0/4063828831 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c00cdb0 con 0x7f6ea4107640 2024-03-21T10:44:40.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.992+0000 7f6eaa600640 1 -- 172.21.15.31:0/4063828831 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea419b060 con 0x7f6ea4107640 2024-03-21T10:44:40.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.992+0000 7f6eaa600640 1 -- 172.21.15.31:0/4063828831 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea41a28f0 con 0x7f6ea4107640 2024-03-21T10:44:40.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.992+0000 7f6e9be00640 1 -- 172.21.15.31:0/4063828831 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e9c005020 con 0x7f6ea4107640 2024-03-21T10:44:40.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:40.998+0000 7f6e9be00640 1 -- 172.21.15.31:0/4063828831 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c01f060 con 0x7f6ea4107640 2024-03-21T10:44:40.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.002+0000 7f6ead1fe740 1 Processor -- start 2024-03-21T10:44:41.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.005+0000 7f6e9be00640 1 -- 172.21.15.31:0/4063828831 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f6e9c004b20 con 0x7f6ea4107640 2024-03-21T10:44:41.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.008+0000 7f6e9be00640 1 --2- 172.21.15.31:0/4063828831 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6e880a3090 0x7f6e880a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.008+0000 7f6e9be00640 1 -- 172.21.15.31:0/4063828831 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f6e9c004e10 con 0x7f6ea4107640 2024-03-21T10:44:41.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.009+0000 7f6ea2a00640 1 --2- 172.21.15.31:0/4063828831 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6e880a3090 0x7f6e880a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.009+0000 7f6ead1fe740 1 -- start start 2024-03-21T10:44:41.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.010+0000 7f6ead1fe740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6cf3066d0 0x55a6cf306ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.010+0000 7f6ead1fe740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6cf307080 0x55a6cf30f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.010+0000 7f6ead1fe740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6cf30fc00 0x55a6cf312050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.010+0000 7f6ead1fe740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6cf26a2c0 con 0x55a6cf3066d0 2024-03-21T10:44:41.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.010+0000 7f6ead1fe740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6cf212c90 con 0x55a6cf30fc00 2024-03-21T10:44:41.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.010+0000 7f6ead1fe740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6cf266360 con 0x55a6cf307080 2024-03-21T10:44:41.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.011+0000 7f6ea2a00640 1 --2- 172.21.15.31:0/4063828831 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6e880a3090 0x7f6e880a5550 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6ea40ff7a0 tx=0x7f6e94013040 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:41.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.011+0000 7f6ea3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6cf30fc00 0x55a6cf312050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.011+0000 7f6ea3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6cf30fc00 0x55a6cf312050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37344/0 (socket says 172.21.15.31:37344) 2024-03-21T10:44:41.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.011+0000 7f6ea3e00640 1 -- 172.21.15.31:0/2164059202 learned_addr learned my addr 172.21.15.31:0/2164059202 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:41.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.012+0000 7f6ea3e00640 1 -- 172.21.15.31:0/2164059202 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6cf307080 msgr2=0x55a6cf30f630 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:44:41.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.012+0000 7f6ea3e00640 1 --2- 172.21.15.31:0/2164059202 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6cf307080 0x55a6cf30f630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.012+0000 7f6ea3e00640 1 -- 172.21.15.31:0/2164059202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6cf3066d0 msgr2=0x55a6cf306ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:44:41.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.013+0000 7f6ea3400640 1 --2- 172.21.15.31:0/2164059202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6cf3066d0 0x55a6cf306ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.013+0000 7f6ea3e00640 1 --2- 172.21.15.31:0/2164059202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6cf3066d0 0x55a6cf306ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.013+0000 7f6ea3e00640 1 -- 172.21.15.31:0/2164059202 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6cf237080 con 0x55a6cf30fc00 2024-03-21T10:44:41.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.014+0000 7f6ea3e00640 1 --2- 172.21.15.31:0/2164059202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6cf30fc00 0x55a6cf312050 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c098fa0 tx=0x7f6e9c0a0f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cb42158d926563 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:41.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.014+0000 7f6e98c00640 1 -- 172.21.15.31:0/2164059202 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c098750 con 0x55a6cf30fc00 2024-03-21T10:44:41.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.014+0000 7f6e98c00640 1 -- 172.21.15.31:0/2164059202 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e9c014ec0 con 0x55a6cf30fc00 2024-03-21T10:44:41.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.014+0000 7f6e98c00640 1 -- 172.21.15.31:0/2164059202 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0988f0 con 0x55a6cf30fc00 2024-03-21T10:44:41.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.015+0000 7f6ead1fe740 1 -- 172.21.15.31:0/2164059202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6cf30fc00 msgr2=0x55a6cf312050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:41.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.015+0000 7f6ead1fe740 1 --2- 172.21.15.31:0/2164059202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6cf30fc00 0x55a6cf312050 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c098fa0 tx=0x7f6e9c0a0f90 comp rx=0 tx=0).stop 2024-03-21T10:44:41.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.017+0000 7f6ead1fe740 1 -- 172.21.15.31:0/2164059202 shutdown_connections 2024-03-21T10:44:41.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.017+0000 7f6ead1fe740 1 --2- 172.21.15.31:0/2164059202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6cf30fc00 0x55a6cf312050 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.017+0000 7f6ead1fe740 1 --2- 172.21.15.31:0/2164059202 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6cf307080 0x55a6cf30f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.017+0000 7f6ead1fe740 1 --2- 172.21.15.31:0/2164059202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6cf3066d0 0x55a6cf306ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.017+0000 7f6ead1fe740 1 -- 172.21.15.31:0/2164059202 >> 172.21.15.31:0/2164059202 conn(0x55a6cf300da0 msgr2=0x55a6cf301180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.017+0000 7f6ead1fe740 1 -- 172.21.15.31:0/2164059202 shutdown_connections 2024-03-21T10:44:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.018+0000 7f6ead1fe740 1 -- 172.21.15.31:0/2164059202 wait complete. 2024-03-21T10:44:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.018+0000 7f6ead1fe740 1 -- start start 2024-03-21T10:44:41.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.019+0000 7f6ead1fe740 1 Processor -- start 2024-03-21T10:44:41.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.021+0000 7f6ead1fe740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6cf3066d0 0x55a6cf30f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.021+0000 7f6ead1fe740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6cf307080 0x55a6cf30c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.021+0000 7f6ead1fe740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6cf30fc00 0x55a6cf30d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.021+0000 7f6ead1fe740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6cf312a80 con 0x55a6cf307080 2024-03-21T10:44:41.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.021+0000 7f6ead1fe740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6cf2260d0 con 0x55a6cf30fc00 2024-03-21T10:44:41.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.021+0000 7f6ead1fe740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6cf312c00 con 0x55a6cf3066d0 2024-03-21T10:44:41.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.021+0000 7f6ead1fe740 10 client.0 _ll_register_callbacks cb 0x55a6cf22ae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:41.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.022+0000 7f6ea3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6cf3066d0 0x55a6cf30f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.022+0000 7f6ea3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6cf30fc00 0x55a6cf30d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.022+0000 7f6ea3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6cf30fc00 0x55a6cf30d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37356/0 (socket says 172.21.15.31:37356) 2024-03-21T10:44:41.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.022+0000 7f6ea3e00640 1 -- 172.21.15.31:0/2239191008 learned_addr learned my addr 172.21.15.31:0/2239191008 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:41.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.022+0000 7f6ea2a00640 1 --2- 172.21.15.31:0/2239191008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6cf307080 0x55a6cf30c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.023+0000 7f6ea2a00640 1 -- 172.21.15.31:0/2239191008 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6cf3066d0 msgr2=0x55a6cf30f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:41.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.023+0000 7f6ea2a00640 1 --2- 172.21.15.31:0/2239191008 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6cf3066d0 0x55a6cf30f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.023+0000 7f6ea2a00640 1 -- 172.21.15.31:0/2239191008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6cf30fc00 msgr2=0x55a6cf30d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:41.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.023+0000 7f6ea2a00640 1 --2- 172.21.15.31:0/2239191008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6cf30fc00 0x55a6cf30d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.023+0000 7f6ea2a00640 1 -- 172.21.15.31:0/2239191008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a6cef7e3f0 con 0x55a6cf307080 2024-03-21T10:44:41.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.023+0000 7f6e8d600640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:44:41.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.023+0000 7f6ea2a00640 1 --2- 172.21.15.31:0/2239191008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6cf307080 0x55a6cf30c950 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x55a6cf307480 tx=0x7f6e940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b0212571dfdda37 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:41.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.023+0000 7f6e8d600640 1 -- 172.21.15.31:0/2239191008 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9401c620 con 0x55a6cf307080 2024-03-21T10:44:41.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.024+0000 7f6e8d600640 1 -- 172.21.15.31:0/2239191008 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f6e9401c7c0 con 0x55a6cf307080 2024-03-21T10:44:41.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.024+0000 7f6ead1fe740 1 -- 172.21.15.31:0/2239191008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6e9c014ec0 con 0x55a6cf307080 2024-03-21T10:44:41.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.024+0000 7f6e8d600640 1 -- 172.21.15.31:0/2239191008 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e9401c9c0 con 0x55a6cf307080 2024-03-21T10:44:41.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.024+0000 7f6ead1fe740 10 client.16611 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:41.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.024+0000 7f6ead1fe740 1 -- 172.21.15.31:0/2239191008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55a6cf0487d0 con 0x55a6cf307080 2024-03-21T10:44:41.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.025+0000 7f6e8d600640 1 -- 172.21.15.31:0/2239191008 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e9c014ec0 con 0x55a6cf307080 2024-03-21T10:44:41.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.026+0000 7f6e8d600640 1 -- 172.21.15.31:0/2239191008 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f6e9401cb60 con 0x55a6cf307080 2024-03-21T10:44:41.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.026+0000 7f6ead1fe740 10 client.16611 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:41.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.026+0000 7f6ead1fe740 10 client.16611 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:44:41.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.026+0000 7f6ead1fe740 20 client.16611 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:41.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.026+0000 7f6ead1fe740 1 --2- 172.21.15.31:0/2239191008 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55a6cf31f8b0 0x55a6cf321d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.026+0000 7f6ead1fe740 4 client.16611 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:44:41.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.026+0000 7f6ead1fe740 1 -- 172.21.15.31:0/2239191008 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a6cf277800 con 0x55a6cf31f8b0 2024-03-21T10:44:41.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.037+0000 7f6ea3400640 1 --2- 172.21.15.31:0/2239191008 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55a6cf31f8b0 0x55a6cf321d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.040+0000 7f6ea3400640 1 --2- 172.21.15.31:0/2239191008 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55a6cf31f8b0 0x55a6cf321d50 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c5477ebc9c8872d6 server_cookie=d40cf1e76b8acd6b in_seq=0 out_seq=0 2024-03-21T10:44:41.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.040+0000 7f6e8d600640 10 client.16611 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:44:41.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.044+0000 7f6e8d600640 1 -- 172.21.15.31:0/2239191008 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a6cf277800 con 0x55a6cf31f8b0 2024-03-21T10:44:41.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.044+0000 7f6e8d600640 10 client.16611 handle_command_reply: tid=0 2024-03-21T10:44:41.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.045+0000 7f6ead1fe740 1 client.16611 shutdown 2024-03-21T10:44:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.047+0000 7f6ead1fe740 1 -- 172.21.15.31:0/2239191008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6cf307080 msgr2=0x55a6cf30c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.047+0000 7f6ead1fe740 1 --2- 172.21.15.31:0/2239191008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6cf307080 0x55a6cf30c950 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x55a6cf307480 tx=0x7f6e940046c0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.048+0000 7f6ead1fe740 1 -- 172.21.15.31:0/2239191008 shutdown_connections 2024-03-21T10:44:41.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.048+0000 7f6ead1fe740 1 --2- 172.21.15.31:0/2239191008 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55a6cf31f8b0 0x55a6cf321d50 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.049+0000 7f6ead1fe740 1 --2- 172.21.15.31:0/2239191008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6cf30fc00 0x55a6cf30d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.049+0000 7f6ead1fe740 1 --2- 172.21.15.31:0/2239191008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6cf307080 0x55a6cf30c950 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.049+0000 7f6ead1fe740 1 --2- 172.21.15.31:0/2239191008 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6cf3066d0 0x55a6cf30f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.049+0000 7f6e8d600640 0 client.16611 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:44:41.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.049+0000 7f6ead1fe740 1 -- 172.21.15.31:0/2239191008 >> 172.21.15.31:0/2239191008 conn(0x55a6cf300da0 msgr2=0x55a6cf302af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:41.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.049+0000 7f6ead1fe740 1 -- 172.21.15.31:0/2239191008 shutdown_connections 2024-03-21T10:44:41.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.049+0000 7f6ead1fe740 1 -- 172.21.15.31:0/2239191008 wait complete. 2024-03-21T10:44:41.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.050+0000 7f6ead1fe740 20 client.16611 trim_cache size 0 max 16384 2024-03-21T10:44:41.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.072+0000 7f6ead1fe740 1 Processor -- start 2024-03-21T10:44:41.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.072+0000 7f6ead1fe740 1 -- start start 2024-03-21T10:44:41.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.072+0000 7f6ead1fe740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6cf3066d0 0x55a6cf30f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.072+0000 7f6ead1fe740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6cf307080 0x55a6cf30f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.072+0000 7f6ead1fe740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6cf30fc00 0x55a6cf309bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.072+0000 7f6ead1fe740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e84012160 con 0x55a6cf30fc00 2024-03-21T10:44:41.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.072+0000 7f6ead1fe740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8400fc60 con 0x55a6cf3066d0 2024-03-21T10:44:41.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.072+0000 7f6ead1fe740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8400dd00 con 0x55a6cf307080 2024-03-21T10:44:41.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.073+0000 7f6ea3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6cf30fc00 0x55a6cf309bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.073+0000 7f6ea3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6cf3066d0 0x55a6cf30f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.073+0000 7f6ea3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6cf30fc00 0x55a6cf309bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41702/0 (socket says 172.21.15.31:41702) 2024-03-21T10:44:41.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.073+0000 7f6ea3e00640 1 -- 172.21.15.31:0/813513946 learned_addr learned my addr 172.21.15.31:0/813513946 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:41.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.073+0000 7f6ea3e00640 1 -- 172.21.15.31:0/813513946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6cf307080 msgr2=0x55a6cf30f5c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:44:41.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.073+0000 7f6ea3e00640 1 --2- 172.21.15.31:0/813513946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6cf307080 0x55a6cf30f5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.074+0000 7f6ea3e00640 1 -- 172.21.15.31:0/813513946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6cf3066d0 msgr2=0x55a6cf30f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:41.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.074+0000 7f6ea3e00640 1 --2- 172.21.15.31:0/813513946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6cf3066d0 0x55a6cf30f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.074+0000 7f6ea3e00640 1 -- 172.21.15.31:0/813513946 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6cf30aea0 con 0x55a6cf30fc00 2024-03-21T10:44:41.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.074+0000 7f6ea3e00640 1 --2- 172.21.15.31:0/813513946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6cf30fc00 0x55a6cf309bd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c098fa0 tx=0x7f6e9c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a7b74c574718077 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:41.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.074+0000 7f6e8fe00640 1 -- 172.21.15.31:0/813513946 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c098a90 con 0x55a6cf30fc00 2024-03-21T10:44:41.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.074+0000 7f6e8fe00640 1 -- 172.21.15.31:0/813513946 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e9c09baf0 con 0x55a6cf30fc00 2024-03-21T10:44:41.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.074+0000 7f6e8fe00640 1 -- 172.21.15.31:0/813513946 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c098c60 con 0x55a6cf30fc00 2024-03-21T10:44:41.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.075+0000 7f6ead1fe740 1 -- 172.21.15.31:0/813513946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6cf30fc00 msgr2=0x55a6cf309bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:41.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.075+0000 7f6ead1fe740 1 --2- 172.21.15.31:0/813513946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6cf30fc00 0x55a6cf309bd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c098fa0 tx=0x7f6e9c00cf90 comp rx=0 tx=0).stop 2024-03-21T10:44:41.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.076+0000 7f6ead1fe740 1 -- 172.21.15.31:0/813513946 shutdown_connections 2024-03-21T10:44:41.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.076+0000 7f6ead1fe740 1 --2- 172.21.15.31:0/813513946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6cf30fc00 0x55a6cf309bd0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.076+0000 7f6ead1fe740 1 --2- 172.21.15.31:0/813513946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6cf307080 0x55a6cf30f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.076+0000 7f6ead1fe740 1 --2- 172.21.15.31:0/813513946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6cf3066d0 0x55a6cf30f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.076+0000 7f6ead1fe740 1 -- 172.21.15.31:0/813513946 >> 172.21.15.31:0/813513946 conn(0x55a6cf300da0 msgr2=0x55a6cf302a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:41.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.076+0000 7f6ead1fe740 1 -- 172.21.15.31:0/813513946 shutdown_connections 2024-03-21T10:44:41.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.076+0000 7f6ead1fe740 1 -- 172.21.15.31:0/813513946 wait complete. 2024-03-21T10:44:41.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.077+0000 7f6ead1fe740 1 -- start start 2024-03-21T10:44:41.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.077+0000 7f6ead1fe740 1 Processor -- start 2024-03-21T10:44:41.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.078+0000 7f6ead1fe740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6cf3066d0 0x55a6cf317520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.078+0000 7f6ead1fe740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6cf307080 0x55a6cf317a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.079+0000 7f6ea3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6cf3066d0 0x55a6cf317520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.079+0000 7f6ea3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6cf3066d0 0x55a6cf317520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41708/0 (socket says 172.21.15.31:41708) 2024-03-21T10:44:41.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.079+0000 7f6ea3400640 1 -- 172.21.15.31:0/3171908435 learned_addr learned my addr 172.21.15.31:0/3171908435 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:41.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.079+0000 7f6ea2a00640 1 --2- 172.21.15.31:0/3171908435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6cf307080 0x55a6cf317a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.079+0000 7f6ead1fe740 1 --2- 172.21.15.31:0/3171908435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6cf30fc00 0x55a6cf315b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.079+0000 7f6ead1fe740 1 -- 172.21.15.31:0/3171908435 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8400e300 con 0x55a6cf3066d0 2024-03-21T10:44:41.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.079+0000 7f6ead1fe740 1 -- 172.21.15.31:0/3171908435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8400f600 con 0x55a6cf307080 2024-03-21T10:44:41.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.079+0000 7f6ead1fe740 1 -- 172.21.15.31:0/3171908435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8400e940 con 0x55a6cf30fc00 2024-03-21T10:44:41.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.080+0000 7f6ead1fe740 10 client.0 _ll_register_callbacks cb 0x7f6e940043a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:41.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.080+0000 7f6ea3400640 1 -- 172.21.15.31:0/3171908435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6cf30fc00 msgr2=0x55a6cf315b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:41.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.080+0000 7f6ea3400640 1 --2- 172.21.15.31:0/3171908435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6cf30fc00 0x55a6cf315b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.080+0000 7f6ea3400640 1 -- 172.21.15.31:0/3171908435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6cf307080 msgr2=0x55a6cf317a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:41.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.080+0000 7f6ea3400640 1 --2- 172.21.15.31:0/3171908435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6cf307080 0x55a6cf317a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.080+0000 7f6ea3400640 1 -- 172.21.15.31:0/3171908435 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a6cf316220 con 0x55a6cf3066d0 2024-03-21T10:44:41.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.081+0000 7f6ea3400640 1 --2- 172.21.15.31:0/3171908435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6cf3066d0 0x55a6cf317520 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6e9000d300 tx=0x7f6e90004110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f87026e13475c90 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:41.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.081+0000 7f6e8d600640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:44:41.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.083+0000 7f6e8d600640 1 -- 172.21.15.31:0/3171908435 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e90004340 con 0x55a6cf3066d0 2024-03-21T10:44:41.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.083+0000 7f6ead1fe740 1 -- 172.21.15.31:0/3171908435 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6e9c09baf0 con 0x55a6cf3066d0 2024-03-21T10:44:41.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.084+0000 7f6e8d600640 1 -- 172.21.15.31:0/3171908435 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f6e90039040 con 0x55a6cf3066d0 2024-03-21T10:44:41.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.085+0000 7f6e8d600640 1 -- 172.21.15.31:0/3171908435 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e90004340 con 0x55a6cf3066d0 2024-03-21T10:44:41.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.085+0000 7f6ead1fe740 10 client.16617 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:41.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.085+0000 7f6ead1fe740 1 -- 172.21.15.31:0/3171908435 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55a6cf37bd70 con 0x55a6cf3066d0 2024-03-21T10:44:41.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.087+0000 7f6e8d600640 1 -- 172.21.15.31:0/3171908435 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e9c09baf0 con 0x55a6cf3066d0 2024-03-21T10:44:41.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.087+0000 7f6e8d600640 1 -- 172.21.15.31:0/3171908435 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f6e90049030 con 0x55a6cf3066d0 2024-03-21T10:44:41.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.089+0000 7f6ead1fe740 10 client.16617 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:41.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.089+0000 7f6ead1fe740 10 client.16617 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:44:41.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.089+0000 7f6ead1fe740 20 client.16617 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:41.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.089+0000 7f6ead1fe740 1 --2- 172.21.15.31:0/3171908435 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55a6cf37c440 0x55a6cf37e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.089+0000 7f6ead1fe740 4 client.16617 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:44:41.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.089+0000 7f6ead1fe740 1 -- 172.21.15.31:0/3171908435 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a6cf271b50 con 0x55a6cf37c440 2024-03-21T10:44:41.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.090+0000 7f6ea2a00640 1 --2- 172.21.15.31:0/3171908435 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55a6cf37c440 0x55a6cf37e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.091+0000 7f6ea2a00640 1 --2- 172.21.15.31:0/3171908435 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55a6cf37c440 0x55a6cf37e860 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=245e171ea907f2eb server_cookie=f1cd75be88a84ebb in_seq=0 out_seq=0 2024-03-21T10:44:41.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.091+0000 7f6e8d600640 10 client.16617 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:44:41.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.092+0000 7f6e8d600640 1 -- 172.21.15.31:0/3171908435 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a6cf271b50 con 0x55a6cf37c440 2024-03-21T10:44:41.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.092+0000 7f6e8d600640 10 client.16617 handle_command_reply: tid=0 2024-03-21T10:44:41.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.092+0000 7f6ead1fe740 1 client.16617 shutdown 2024-03-21T10:44:41.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.093+0000 7f6ead1fe740 1 -- 172.21.15.31:0/3171908435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6cf3066d0 msgr2=0x55a6cf317520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:41.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.093+0000 7f6ead1fe740 1 --2- 172.21.15.31:0/3171908435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6cf3066d0 0x55a6cf317520 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6e9000d300 tx=0x7f6e90004110 comp rx=0 tx=0).stop 2024-03-21T10:44:41.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.093+0000 7f6ead1fe740 1 -- 172.21.15.31:0/3171908435 shutdown_connections 2024-03-21T10:44:41.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.093+0000 7f6ead1fe740 1 --2- 172.21.15.31:0/3171908435 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55a6cf37c440 0x55a6cf37e860 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.094+0000 7f6ead1fe740 1 --2- 172.21.15.31:0/3171908435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a6cf30fc00 0x55a6cf315b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.094+0000 7f6ead1fe740 1 --2- 172.21.15.31:0/3171908435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6cf307080 0x55a6cf317a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.094+0000 7f6ead1fe740 1 --2- 172.21.15.31:0/3171908435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6cf3066d0 0x55a6cf317520 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.094+0000 7f6ead1fe740 1 -- 172.21.15.31:0/3171908435 >> 172.21.15.31:0/3171908435 conn(0x55a6cf300da0 msgr2=0x55a6cf3075d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:41.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.094+0000 7f6e8d600640 0 client.16617 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:44:41.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.094+0000 7f6ead1fe740 1 -- 172.21.15.31:0/3171908435 shutdown_connections 2024-03-21T10:44:41.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.097+0000 7f6ead1fe740 1 -- 172.21.15.31:0/3171908435 wait complete. 2024-03-21T10:44:41.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.097+0000 7f6ead1fe740 20 client.16617 trim_cache size 0 max 16384 2024-03-21T10:44:41.095 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:44:41.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.101+0000 7f6eaa600640 1 -- 172.21.15.31:0/4063828831 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6e880a3090 msgr2=0x7f6e880a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:41.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.101+0000 7f6eaa600640 1 --2- 172.21.15.31:0/4063828831 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6e880a3090 0x7f6e880a5550 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6ea40ff7a0 tx=0x7f6e94013040 comp rx=0 tx=0).stop 2024-03-21T10:44:41.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.102+0000 7f6eaa600640 1 -- 172.21.15.31:0/4063828831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ea4107640 msgr2=0x7f6ea419a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:41.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.102+0000 7f6eaa600640 1 --2- 172.21.15.31:0/4063828831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ea4107640 0x7f6ea419a640 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c00eaa0 tx=0x7f6e9c00ef60 comp rx=0 tx=0).stop 2024-03-21T10:44:41.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.104+0000 7f6eaa600640 1 -- 172.21.15.31:0/4063828831 shutdown_connections 2024-03-21T10:44:41.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.104+0000 7f6eaa600640 1 --2- 172.21.15.31:0/4063828831 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6e880a3090 0x7f6e880a5550 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.104+0000 7f6eaa600640 1 --2- 172.21.15.31:0/4063828831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ea4107640 0x7f6ea419a640 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.104+0000 7f6eaa600640 1 --2- 172.21.15.31:0/4063828831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ea4101ff0 0x7f6ea41962c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.104+0000 7f6eaa600640 1 --2- 172.21.15.31:0/4063828831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea4101630 0x7f6ea4195d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.104+0000 7f6eaa600640 1 -- 172.21.15.31:0/4063828831 >> 172.21.15.31:0/4063828831 conn(0x7f6ea40fd140 msgr2=0x7f6ea40fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:41.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.104+0000 7f6eaa600640 1 -- 172.21.15.31:0/4063828831 shutdown_connections 2024-03-21T10:44:41.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.105+0000 7f6eaa600640 1 -- 172.21.15.31:0/4063828831 wait complete. 2024-03-21T10:44:41.159 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:44:41.159 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-21T10:44:41.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:41 smithi040 ceph-mon[36154]: pgmap v186: 129 pgs: 129 active+clean; 39 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 229 MiB/s wr, 1.74k op/s 2024-03-21T10:44:41.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:41 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2547651262' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:44:41.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.534+0000 7f1e70400640 1 -- 172.21.15.31:0/971472585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e68109b70 msgr2=0x7f1e68109f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:41.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.534+0000 7f1e70400640 1 --2- 172.21.15.31:0/971472585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e68109b70 0x7f1e68109f50 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c009930 tx=0x7f1e5c02bc30 comp rx=0 tx=0).stop 2024-03-21T10:44:41.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.536+0000 7f1e70400640 1 -- 172.21.15.31:0/971472585 shutdown_connections 2024-03-21T10:44:41.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.536+0000 7f1e70400640 1 --2- 172.21.15.31:0/971472585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e68108760 0x7f1e68114c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.536+0000 7f1e70400640 1 --2- 172.21.15.31:0/971472585 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e68107dc0 0x7f1e68108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.536+0000 7f1e70400640 1 --2- 172.21.15.31:0/971472585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e68109b70 0x7f1e68109f50 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.536+0000 7f1e70400640 1 -- 172.21.15.31:0/971472585 >> 172.21.15.31:0/971472585 conn(0x7f1e68077540 msgr2=0x7f1e681064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:41.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.536+0000 7f1e70400640 1 -- 172.21.15.31:0/971472585 shutdown_connections 2024-03-21T10:44:41.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.536+0000 7f1e70400640 1 -- 172.21.15.31:0/971472585 wait complete. 2024-03-21T10:44:41.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.538+0000 7f1e70400640 1 Processor -- start 2024-03-21T10:44:41.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.538+0000 7f1e70400640 1 -- start start 2024-03-21T10:44:41.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.538+0000 7f1e70400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e68107dc0 0x7f1e6819e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.538+0000 7f1e70400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e68108760 0x7f1e6819eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.538+0000 7f1e70400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e68109b70 0x7f1e681a2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.538+0000 7f1e70400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e68118150 con 0x7f1e68108760 2024-03-21T10:44:41.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.538+0000 7f1e70400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e68117fd0 con 0x7f1e68109b70 2024-03-21T10:44:41.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.538+0000 7f1e70400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e681182d0 con 0x7f1e68107dc0 2024-03-21T10:44:41.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.539+0000 7f1e6da00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e68107dc0 0x7f1e6819e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.539+0000 7f1e6da00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e68107dc0 0x7f1e6819e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60102/0 (socket says 172.21.15.31:60102) 2024-03-21T10:44:41.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.539+0000 7f1e6da00640 1 -- 172.21.15.31:0/603245899 learned_addr learned my addr 172.21.15.31:0/603245899 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:41.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.540+0000 7f1e6da00640 1 -- 172.21.15.31:0/603245899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e68109b70 msgr2=0x7f1e681a2eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T10:44:41.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.540+0000 7f1e6da00640 1 --2- 172.21.15.31:0/603245899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e68109b70 0x7f1e681a2eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.540+0000 7f1e6da00640 1 -- 172.21.15.31:0/603245899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e68108760 msgr2=0x7f1e6819eb30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T10:44:41.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.541+0000 7f1e6d000640 1 --2- 172.21.15.31:0/603245899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e68108760 0x7f1e6819eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.541+0000 7f1e6da00640 1 --2- 172.21.15.31:0/603245899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e68108760 0x7f1e6819eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.541+0000 7f1e6da00640 1 -- 172.21.15.31:0/603245899 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e681a35e0 con 0x7f1e68107dc0 2024-03-21T10:44:41.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.541+0000 7f1e6d000640 1 --2- 172.21.15.31:0/603245899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e68108760 0x7f1e6819eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:44:41.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.541+0000 7f1e6da00640 1 --2- 172.21.15.31:0/603245899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e68107dc0 0x7f1e6819e5f0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c009900 tx=0x7f1e5c0043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:41.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.542+0000 7f1e66000640 1 -- 172.21.15.31:0/603245899 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c043020 con 0x7f1e68107dc0 2024-03-21T10:44:41.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.542+0000 7f1e66000640 1 -- 172.21.15.31:0/603245899 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1e5c0054e0 con 0x7f1e68107dc0 2024-03-21T10:44:41.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.542+0000 7f1e66000640 1 -- 172.21.15.31:0/603245899 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c003c50 con 0x7f1e68107dc0 2024-03-21T10:44:41.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.542+0000 7f1e70400640 1 -- 172.21.15.31:0/603245899 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e681a3870 con 0x7f1e68107dc0 2024-03-21T10:44:41.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.543+0000 7f1e70400640 1 -- 172.21.15.31:0/603245899 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e681ab160 con 0x7f1e68107dc0 2024-03-21T10:44:41.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.548+0000 7f1e66000640 1 -- 172.21.15.31:0/603245899 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f1e5c046020 con 0x7f1e68107dc0 2024-03-21T10:44:41.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.549+0000 7f1e72ff6740 1 Processor -- start 2024-03-21T10:44:41.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.550+0000 7f1e72ff6740 1 -- start start 2024-03-21T10:44:41.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.551+0000 7f1e72ff6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e05ee6d0 0x55d1e05eeab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.551+0000 7f1e72ff6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1e05ef080 0x55d1e05f7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.551+0000 7f1e72ff6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1e05f7c00 0x55d1e05fa050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.551+0000 7f1e72ff6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1e05522c0 con 0x55d1e05f7c00 2024-03-21T10:44:41.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.551+0000 7f1e72ff6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1e04fac90 con 0x55d1e05ee6d0 2024-03-21T10:44:41.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.551+0000 7f1e72ff6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1e054e360 con 0x55d1e05ef080 2024-03-21T10:44:41.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.552+0000 7f1e66000640 1 --2- 172.21.15.31:0/603245899 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f1e500a3150 0x7f1e500a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.552+0000 7f1e66000640 1 -- 172.21.15.31:0/603245899 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f1e5c0f1c70 con 0x7f1e68107dc0 2024-03-21T10:44:41.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.553+0000 7f1e6da00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1e05ef080 0x55d1e05f7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.553+0000 7f1e6e400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e05ee6d0 0x55d1e05eeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.553+0000 7f1e6da00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1e05ef080 0x55d1e05f7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60108/0 (socket says 172.21.15.31:60108) 2024-03-21T10:44:41.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.553+0000 7f1e6da00640 1 -- 172.21.15.31:0/343132505 learned_addr learned my addr 172.21.15.31:0/343132505 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:41.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.554+0000 7f1e6da00640 1 -- 172.21.15.31:0/343132505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e05ee6d0 msgr2=0x55d1e05eeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:41.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.554+0000 7f1e6da00640 1 --2- 172.21.15.31:0/343132505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e05ee6d0 0x55d1e05eeab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.554+0000 7f1e6da00640 1 -- 172.21.15.31:0/343132505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1e05f7c00 msgr2=0x55d1e05fa050 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:44:41.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.554+0000 7f1e6d000640 1 --2- 172.21.15.31:0/343132505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1e05f7c00 0x55d1e05fa050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.554+0000 7f1e6da00640 1 --2- 172.21.15.31:0/343132505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1e05f7c00 0x55d1e05fa050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.554+0000 7f1e6da00640 1 -- 172.21.15.31:0/343132505 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d1e051f080 con 0x55d1e05ef080 2024-03-21T10:44:41.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.554+0000 7f1e6e400640 1 --2- 172.21.15.31:0/343132505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e05ee6d0 0x55d1e05eeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:44:41.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.555+0000 7f1e6da00640 1 --2- 172.21.15.31:0/343132505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1e05ef080 0x55d1e05f7630 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c0ba1a0 tx=0x7f1e5c0ba110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a1f1b80f8135fec server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:41.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.555+0000 7f1e6e400640 1 --2- 172.21.15.31:0/603245899 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f1e500a3150 0x7f1e500a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.555+0000 7f1e5aa00640 1 -- 172.21.15.31:0/343132505 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c0d6020 con 0x55d1e05ef080 2024-03-21T10:44:41.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.556+0000 7f1e5aa00640 1 -- 172.21.15.31:0/343132505 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1e5c0c1070 con 0x55d1e05ef080 2024-03-21T10:44:41.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.556+0000 7f1e5aa00640 1 -- 172.21.15.31:0/343132505 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c0ef2a0 con 0x55d1e05ef080 2024-03-21T10:44:41.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:41 smithi031 ceph-mon[31673]: pgmap v186: 129 pgs: 129 active+clean; 39 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 229 MiB/s wr, 1.74k op/s 2024-03-21T10:44:41.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:41 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2547651262' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:44:41.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.559+0000 7f1e72ff6740 1 -- 172.21.15.31:0/343132505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1e05ef080 msgr2=0x55d1e05f7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:41.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.559+0000 7f1e6e400640 1 --2- 172.21.15.31:0/603245899 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f1e500a3150 0x7f1e500a5610 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1e60011440 tx=0x7f1e60004a40 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:41.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.559+0000 7f1e72ff6740 1 --2- 172.21.15.31:0/343132505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1e05ef080 0x55d1e05f7630 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c0ba1a0 tx=0x7f1e5c0ba110 comp rx=0 tx=0).stop 2024-03-21T10:44:41.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.559+0000 7f1e72ff6740 1 -- 172.21.15.31:0/343132505 shutdown_connections 2024-03-21T10:44:41.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.559+0000 7f1e72ff6740 1 --2- 172.21.15.31:0/343132505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1e05f7c00 0x55d1e05fa050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.559+0000 7f1e72ff6740 1 --2- 172.21.15.31:0/343132505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1e05ef080 0x55d1e05f7630 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.559+0000 7f1e72ff6740 1 --2- 172.21.15.31:0/343132505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e05ee6d0 0x55d1e05eeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.559+0000 7f1e72ff6740 1 -- 172.21.15.31:0/343132505 >> 172.21.15.31:0/343132505 conn(0x55d1e05e8da0 msgr2=0x55d1e05e9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:41.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.562+0000 7f1e72ff6740 1 -- 172.21.15.31:0/343132505 shutdown_connections 2024-03-21T10:44:41.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.562+0000 7f1e72ff6740 1 -- 172.21.15.31:0/343132505 wait complete. 2024-03-21T10:44:41.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.563+0000 7f1e72ff6740 1 -- start start 2024-03-21T10:44:41.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.563+0000 7f1e72ff6740 1 Processor -- start 2024-03-21T10:44:41.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.564+0000 7f1e72ff6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1e05ee6d0 0x55d1e05f7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.564+0000 7f1e72ff6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1e05f7c00 0x55d1e05f4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.564+0000 7f1e72ff6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e05f4fb0 0x55d1e0603d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.564+0000 7f1e72ff6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1e05fa930 con 0x55d1e05ee6d0 2024-03-21T10:44:41.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.564+0000 7f1e72ff6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1e050e0d0 con 0x55d1e05f4fb0 2024-03-21T10:44:41.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.564+0000 7f1e72ff6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1e05faab0 con 0x55d1e05f7c00 2024-03-21T10:44:41.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.564+0000 7f1e72ff6740 10 client.0 _ll_register_callbacks cb 0x55d1e0512e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:41.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.565+0000 7f1e6d000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1e05f7c00 0x55d1e05f4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.565+0000 7f1e6d000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1e05f7c00 0x55d1e05f4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60118/0 (socket says 172.21.15.31:60118) 2024-03-21T10:44:41.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.565+0000 7f1e6d000640 1 -- 172.21.15.31:0/2615822428 learned_addr learned my addr 172.21.15.31:0/2615822428 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:41.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.565+0000 7f1e6da00640 1 --2- 172.21.15.31:0/2615822428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1e05ee6d0 0x55d1e05f7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.565+0000 7f1e6e400640 1 --2- 172.21.15.31:0/2615822428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e05f4fb0 0x55d1e0603d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.566+0000 7f1e6d000640 1 -- 172.21.15.31:0/2615822428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e05f4fb0 msgr2=0x55d1e0603d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:41.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.566+0000 7f1e6d000640 1 --2- 172.21.15.31:0/2615822428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e05f4fb0 0x55d1e0603d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.566+0000 7f1e6d000640 1 -- 172.21.15.31:0/2615822428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1e05ee6d0 msgr2=0x55d1e05f7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:41.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.566+0000 7f1e6d000640 1 --2- 172.21.15.31:0/2615822428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1e05ee6d0 0x55d1e05f7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.566+0000 7f1e6d000640 1 -- 172.21.15.31:0/2615822428 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d1e02663f0 con 0x55d1e05f7c00 2024-03-21T10:44:41.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.566+0000 7f1e6d000640 1 --2- 172.21.15.31:0/2615822428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1e05f7c00 0x55d1e05f4910 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f1e5400dd80 tx=0x7f1e5400f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71229164646630d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:41.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.566+0000 7f1e4f400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:44:41.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.567+0000 7f1e4f400640 1 -- 172.21.15.31:0/2615822428 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e54005020 con 0x55d1e05f7c00 2024-03-21T10:44:41.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.567+0000 7f1e4f400640 1 -- 172.21.15.31:0/2615822428 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f1e540051c0 con 0x55d1e05f7c00 2024-03-21T10:44:41.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.567+0000 7f1e72ff6740 1 -- 172.21.15.31:0/2615822428 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1e5c0c1070 con 0x55d1e05f7c00 2024-03-21T10:44:41.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.567+0000 7f1e4f400640 1 -- 172.21.15.31:0/2615822428 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e540053c0 con 0x55d1e05f7c00 2024-03-21T10:44:41.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.567+0000 7f1e72ff6740 10 client.27070 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:41.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.567+0000 7f1e72ff6740 1 -- 172.21.15.31:0/2615822428 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55d1e03307d0 con 0x55d1e05f7c00 2024-03-21T10:44:41.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.568+0000 7f1e4f400640 1 -- 172.21.15.31:0/2615822428 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1e5c0c1070 con 0x55d1e05f7c00 2024-03-21T10:44:41.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.568+0000 7f1e4f400640 1 -- 172.21.15.31:0/2615822428 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f1e54011940 con 0x55d1e05f7c00 2024-03-21T10:44:41.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.568+0000 7f1e72ff6740 10 client.27070 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:41.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.568+0000 7f1e72ff6740 10 client.27070 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:44:41.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.568+0000 7f1e72ff6740 20 client.27070 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:41.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.568+0000 7f1e72ff6740 1 --2- 172.21.15.31:0/2615822428 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d1e0607cb0 0x55d1e060a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.569+0000 7f1e72ff6740 4 client.27070 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:44:41.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.569+0000 7f1e72ff6740 1 -- 172.21.15.31:0/2615822428 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d1e055f800 con 0x55d1e0607cb0 2024-03-21T10:44:41.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.569+0000 7f1e6da00640 1 --2- 172.21.15.31:0/2615822428 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d1e0607cb0 0x55d1e060a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.570+0000 7f1e6da00640 1 --2- 172.21.15.31:0/2615822428 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d1e0607cb0 0x55d1e060a0a0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a96636b44592ce9 server_cookie=1b85bfee65e0e453 in_seq=0 out_seq=0 2024-03-21T10:44:41.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.570+0000 7f1e4f400640 10 client.27070 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:41.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.571+0000 7f1e4f400640 1 -- 172.21.15.31:0/2615822428 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d1e055f800 con 0x55d1e0607cb0 2024-03-21T10:44:41.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.571+0000 7f1e4f400640 10 client.27070 handle_command_reply: tid=0 2024-03-21T10:44:41.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.571+0000 7f1e72ff6740 1 client.27070 shutdown 2024-03-21T10:44:41.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.572+0000 7f1e72ff6740 1 -- 172.21.15.31:0/2615822428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1e05f7c00 msgr2=0x55d1e05f4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:41.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.572+0000 7f1e72ff6740 1 --2- 172.21.15.31:0/2615822428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1e05f7c00 0x55d1e05f4910 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f1e5400dd80 tx=0x7f1e5400f730 comp rx=0 tx=0).stop 2024-03-21T10:44:41.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.572+0000 7f1e72ff6740 1 -- 172.21.15.31:0/2615822428 shutdown_connections 2024-03-21T10:44:41.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.572+0000 7f1e72ff6740 1 --2- 172.21.15.31:0/2615822428 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d1e0607cb0 0x55d1e060a0a0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.572+0000 7f1e72ff6740 1 --2- 172.21.15.31:0/2615822428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e05f4fb0 0x55d1e0603d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.572+0000 7f1e72ff6740 1 --2- 172.21.15.31:0/2615822428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1e05f7c00 0x55d1e05f4910 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.572+0000 7f1e72ff6740 1 --2- 172.21.15.31:0/2615822428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1e05ee6d0 0x55d1e05f7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.572+0000 7f1e72ff6740 1 -- 172.21.15.31:0/2615822428 >> 172.21.15.31:0/2615822428 conn(0x55d1e05e8da0 msgr2=0x55d1e05eaaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:41.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.572+0000 7f1e4f400640 0 client.27070 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:41.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.573+0000 7f1e72ff6740 1 -- 172.21.15.31:0/2615822428 shutdown_connections 2024-03-21T10:44:41.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.573+0000 7f1e72ff6740 1 -- 172.21.15.31:0/2615822428 wait complete. 2024-03-21T10:44:41.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.573+0000 7f1e72ff6740 20 client.27070 trim_cache size 0 max 16384 2024-03-21T10:44:41.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:41 smithi070 ceph-mon[36330]: pgmap v186: 129 pgs: 129 active+clean; 39 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 229 MiB/s wr, 1.74k op/s 2024-03-21T10:44:41.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:41 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2547651262' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:44:41.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.601+0000 7f1e72ff6740 1 Processor -- start 2024-03-21T10:44:41.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.601+0000 7f1e72ff6740 1 -- start start 2024-03-21T10:44:41.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.601+0000 7f1e72ff6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e05ee6d0 0x55d1e05f7150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.601+0000 7f1e72ff6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1e05f4fb0 0x55d1e05f1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.601+0000 7f1e72ff6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1e05f7c00 0x55d1e05f20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.601+0000 7f1e72ff6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e681b74d0 con 0x55d1e05f4fb0 2024-03-21T10:44:41.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.601+0000 7f1e72ff6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e681b4fd0 con 0x55d1e05ee6d0 2024-03-21T10:44:41.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.601+0000 7f1e72ff6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e681b3030 con 0x55d1e05f7c00 2024-03-21T10:44:41.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.602+0000 7f1e6da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e05ee6d0 0x55d1e05f7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.602+0000 7f1e6da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e05ee6d0 0x55d1e05f7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37400/0 (socket says 172.21.15.31:37400) 2024-03-21T10:44:41.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.602+0000 7f1e6da00640 1 -- 172.21.15.31:0/989600105 learned_addr learned my addr 172.21.15.31:0/989600105 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:41.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.602+0000 7f1e6d000640 1 --2- 172.21.15.31:0/989600105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1e05f4fb0 0x55d1e05f1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.602+0000 7f1e6e400640 1 --2- 172.21.15.31:0/989600105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1e05f7c00 0x55d1e05f20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.602+0000 7f1e6da00640 1 -- 172.21.15.31:0/989600105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1e05f7c00 msgr2=0x55d1e05f20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:41.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.602+0000 7f1e6da00640 1 --2- 172.21.15.31:0/989600105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1e05f7c00 0x55d1e05f20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.602+0000 7f1e6da00640 1 -- 172.21.15.31:0/989600105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1e05f4fb0 msgr2=0x55d1e05f1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:41.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.602+0000 7f1e6da00640 1 --2- 172.21.15.31:0/989600105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1e05f4fb0 0x55d1e05f1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.602+0000 7f1e6da00640 1 -- 172.21.15.31:0/989600105 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d1e05ee1d0 con 0x55d1e05ee6d0 2024-03-21T10:44:41.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.602+0000 7f1e6d000640 1 --2- 172.21.15.31:0/989600105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1e05f4fb0 0x55d1e05f1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:44:41.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.602+0000 7f1e6da00640 1 --2- 172.21.15.31:0/989600105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e05ee6d0 0x55d1e05f7150 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c006010 tx=0x7f1e5c02bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c402d1f09a421a91 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:41.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.603+0000 7f1e5a000640 1 -- 172.21.15.31:0/989600105 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c0d6070 con 0x55d1e05ee6d0 2024-03-21T10:44:41.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.603+0000 7f1e5a000640 1 -- 172.21.15.31:0/989600105 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1e5c0c1040 con 0x55d1e05ee6d0 2024-03-21T10:44:41.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.603+0000 7f1e5a000640 1 -- 172.21.15.31:0/989600105 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c0ef2a0 con 0x55d1e05ee6d0 2024-03-21T10:44:41.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.604+0000 7f1e72ff6740 1 -- 172.21.15.31:0/989600105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e05ee6d0 msgr2=0x55d1e05f7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:41.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.604+0000 7f1e72ff6740 1 --2- 172.21.15.31:0/989600105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e05ee6d0 0x55d1e05f7150 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c006010 tx=0x7f1e5c02bc30 comp rx=0 tx=0).stop 2024-03-21T10:44:41.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.604+0000 7f1e72ff6740 1 -- 172.21.15.31:0/989600105 shutdown_connections 2024-03-21T10:44:41.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.604+0000 7f1e72ff6740 1 --2- 172.21.15.31:0/989600105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1e05f7c00 0x55d1e05f20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.604+0000 7f1e72ff6740 1 --2- 172.21.15.31:0/989600105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1e05f4fb0 0x55d1e05f1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.604+0000 7f1e72ff6740 1 --2- 172.21.15.31:0/989600105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e05ee6d0 0x55d1e05f7150 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.604+0000 7f1e72ff6740 1 -- 172.21.15.31:0/989600105 >> 172.21.15.31:0/989600105 conn(0x55d1e05e8da0 msgr2=0x55d1e05e7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:41.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.604+0000 7f1e72ff6740 1 -- 172.21.15.31:0/989600105 shutdown_connections 2024-03-21T10:44:41.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.604+0000 7f1e72ff6740 1 -- 172.21.15.31:0/989600105 wait complete. 2024-03-21T10:44:41.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.604+0000 7f1e72ff6740 1 -- start start 2024-03-21T10:44:41.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.605+0000 7f1e72ff6740 1 Processor -- start 2024-03-21T10:44:41.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.605+0000 7f1e72ff6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1e05ee6d0 0x55d1e05fd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.605+0000 7f1e72ff6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e05f4fb0 0x55d1e05fdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.605+0000 7f1e72ff6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1e05f7c00 0x55d1e0662560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.606+0000 7f1e72ff6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e681b3650 con 0x55d1e05ee6d0 2024-03-21T10:44:41.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.606+0000 7f1e72ff6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e681b4970 con 0x55d1e05f4fb0 2024-03-21T10:44:41.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.606+0000 7f1e72ff6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e681b3cb0 con 0x55d1e05f7c00 2024-03-21T10:44:41.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.606+0000 7f1e72ff6740 10 client.0 _ll_register_callbacks cb 0x7f1e5400d2e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:41.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.606+0000 7f1e6d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e05f4fb0 0x55d1e05fdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.606+0000 7f1e6da00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1e05ee6d0 0x55d1e05fd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.606+0000 7f1e6d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e05f4fb0 0x55d1e05fdd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37404/0 (socket says 172.21.15.31:37404) 2024-03-21T10:44:41.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.606+0000 7f1e6d000640 1 -- 172.21.15.31:0/1341965263 learned_addr learned my addr 172.21.15.31:0/1341965263 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:41.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.606+0000 7f1e6e400640 1 --2- 172.21.15.31:0/1341965263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1e05f7c00 0x55d1e0662560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.606+0000 7f1e6d000640 1 -- 172.21.15.31:0/1341965263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1e05f7c00 msgr2=0x55d1e0662560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:41.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.606+0000 7f1e6d000640 1 --2- 172.21.15.31:0/1341965263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1e05f7c00 0x55d1e0662560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.606+0000 7f1e6d000640 1 -- 172.21.15.31:0/1341965263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1e05ee6d0 msgr2=0x55d1e05fd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:41.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.606+0000 7f1e6d000640 1 --2- 172.21.15.31:0/1341965263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1e05ee6d0 0x55d1e05fd7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.606+0000 7f1e6d000640 1 -- 172.21.15.31:0/1341965263 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d1e0662b70 con 0x55d1e05f4fb0 2024-03-21T10:44:41.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.607+0000 7f1e6d000640 1 --2- 172.21.15.31:0/1341965263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e05f4fb0 0x55d1e05fdd10 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f1e54017ac0 tx=0x7f1e5400d540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5802ed427042fd99 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:41.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.607+0000 7f1e4f400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:44:41.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.607+0000 7f1e4f400640 1 -- 172.21.15.31:0/1341965263 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e540053c0 con 0x55d1e05f4fb0 2024-03-21T10:44:41.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.607+0000 7f1e72ff6740 1 -- 172.21.15.31:0/1341965263 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1e5c0c1040 con 0x55d1e05f4fb0 2024-03-21T10:44:41.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.607+0000 7f1e4f400640 1 -- 172.21.15.31:0/1341965263 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f1e540051c0 con 0x55d1e05f4fb0 2024-03-21T10:44:41.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.607+0000 7f1e4f400640 1 -- 172.21.15.31:0/1341965263 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e54036050 con 0x55d1e05f4fb0 2024-03-21T10:44:41.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.607+0000 7f1e72ff6740 10 client.27161 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:41.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.607+0000 7f1e72ff6740 1 -- 172.21.15.31:0/1341965263 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55d1e0663ea0 con 0x55d1e05f4fb0 2024-03-21T10:44:41.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.608+0000 7f1e4f400640 1 -- 172.21.15.31:0/1341965263 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1e5c0c1040 con 0x55d1e05f4fb0 2024-03-21T10:44:41.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.608+0000 7f1e4f400640 1 -- 172.21.15.31:0/1341965263 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f1e54011940 con 0x55d1e05f4fb0 2024-03-21T10:44:41.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.608+0000 7f1e72ff6740 10 client.27161 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:41.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.608+0000 7f1e72ff6740 10 client.27161 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:44:41.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.608+0000 7f1e72ff6740 20 client.27161 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:41.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.608+0000 7f1e72ff6740 1 --2- 172.21.15.31:0/1341965263 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d1e06645f0 0x55d1e0666a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:41.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.608+0000 7f1e72ff6740 4 client.27161 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:44:41.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.608+0000 7f1e72ff6740 1 -- 172.21.15.31:0/1341965263 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d1e0559b50 con 0x55d1e06645f0 2024-03-21T10:44:41.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.608+0000 7f1e6da00640 1 --2- 172.21.15.31:0/1341965263 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d1e06645f0 0x55d1e0666a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:41.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.609+0000 7f1e6da00640 1 --2- 172.21.15.31:0/1341965263 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d1e06645f0 0x55d1e0666a10 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9ab33dd470a222b server_cookie=99f913e877322430 in_seq=0 out_seq=0 2024-03-21T10:44:41.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.609+0000 7f1e4f400640 10 client.27161 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:41.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.609+0000 7f1e4f400640 1 -- 172.21.15.31:0/1341965263 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d1e0559b50 con 0x55d1e06645f0 2024-03-21T10:44:41.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.609+0000 7f1e4f400640 10 client.27161 handle_command_reply: tid=0 2024-03-21T10:44:41.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.609+0000 7f1e72ff6740 1 client.27161 shutdown 2024-03-21T10:44:41.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.609+0000 7f1e72ff6740 1 -- 172.21.15.31:0/1341965263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e05f4fb0 msgr2=0x55d1e05fdd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:41.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.609+0000 7f1e72ff6740 1 --2- 172.21.15.31:0/1341965263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e05f4fb0 0x55d1e05fdd10 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f1e54017ac0 tx=0x7f1e5400d540 comp rx=0 tx=0).stop 2024-03-21T10:44:41.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.609+0000 7f1e72ff6740 1 -- 172.21.15.31:0/1341965263 shutdown_connections 2024-03-21T10:44:41.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.609+0000 7f1e72ff6740 1 --2- 172.21.15.31:0/1341965263 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d1e06645f0 0x55d1e0666a10 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.609+0000 7f1e72ff6740 1 --2- 172.21.15.31:0/1341965263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d1e05f7c00 0x55d1e0662560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.609+0000 7f1e72ff6740 1 --2- 172.21.15.31:0/1341965263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e05f4fb0 0x55d1e05fdd10 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.609+0000 7f1e4f400640 0 client.27161 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:41.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.609+0000 7f1e72ff6740 1 --2- 172.21.15.31:0/1341965263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1e05ee6d0 0x55d1e05fd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.609+0000 7f1e72ff6740 1 -- 172.21.15.31:0/1341965263 >> 172.21.15.31:0/1341965263 conn(0x55d1e05e8da0 msgr2=0x55d1e05ef460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:41.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.609+0000 7f1e72ff6740 1 -- 172.21.15.31:0/1341965263 shutdown_connections 2024-03-21T10:44:41.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.609+0000 7f1e72ff6740 1 -- 172.21.15.31:0/1341965263 wait complete. 2024-03-21T10:44:41.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.610+0000 7f1e72ff6740 20 client.27161 trim_cache size 0 max 16384 2024-03-21T10:44:41.610 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:44:41.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.610+0000 7f1e70400640 1 -- 172.21.15.31:0/603245899 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f1e500a3150 msgr2=0x7f1e500a5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:41.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.610+0000 7f1e70400640 1 --2- 172.21.15.31:0/603245899 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f1e500a3150 0x7f1e500a5610 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1e60011440 tx=0x7f1e60004a40 comp rx=0 tx=0).stop 2024-03-21T10:44:41.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.610+0000 7f1e70400640 1 -- 172.21.15.31:0/603245899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e68107dc0 msgr2=0x7f1e6819e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:41.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.610+0000 7f1e70400640 1 --2- 172.21.15.31:0/603245899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e68107dc0 0x7f1e6819e5f0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c009900 tx=0x7f1e5c0043b0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.610+0000 7f1e70400640 1 -- 172.21.15.31:0/603245899 shutdown_connections 2024-03-21T10:44:41.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.610+0000 7f1e70400640 1 --2- 172.21.15.31:0/603245899 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f1e500a3150 0x7f1e500a5610 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.610+0000 7f1e70400640 1 --2- 172.21.15.31:0/603245899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e68109b70 0x7f1e681a2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.610+0000 7f1e70400640 1 --2- 172.21.15.31:0/603245899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e68108760 0x7f1e6819eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.610+0000 7f1e70400640 1 --2- 172.21.15.31:0/603245899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e68107dc0 0x7f1e6819e5f0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:41.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.610+0000 7f1e70400640 1 -- 172.21.15.31:0/603245899 >> 172.21.15.31:0/603245899 conn(0x7f1e68077540 msgr2=0x7f1e68077a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:41.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.610+0000 7f1e70400640 1 -- 172.21.15.31:0/603245899 shutdown_connections 2024-03-21T10:44:41.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:41.611+0000 7f1e70400640 1 -- 172.21.15.31:0/603245899 wait complete. 2024-03-21T10:44:42.632 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-21T10:44:42.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:44:42.632 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-21T10:44:43.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.105+0000 7fb497e00640 1 -- 172.21.15.31:0/1364983737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb498100fc0 msgr2=0x7fb498104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:43.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.105+0000 7fb497e00640 1 --2- 172.21.15.31:0/1364983737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb498100fc0 0x7fb498104fb0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fb490009be0 tx=0x7fb49002bef0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.105+0000 7fb497e00640 1 -- 172.21.15.31:0/1364983737 shutdown_connections 2024-03-21T10:44:43.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.105+0000 7fb497e00640 1 --2- 172.21.15.31:0/1364983737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb498105690 0x7fb49810d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.105+0000 7fb497e00640 1 --2- 172.21.15.31:0/1364983737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb498100fc0 0x7fb498104fb0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.105+0000 7fb497e00640 1 --2- 172.21.15.31:0/1364983737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb498100610 0x7fb4981009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.105+0000 7fb497e00640 1 -- 172.21.15.31:0/1364983737 >> 172.21.15.31:0/1364983737 conn(0x7fb498077130 msgr2=0x7fb4980fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:43.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.105+0000 7fb497e00640 1 -- 172.21.15.31:0/1364983737 shutdown_connections 2024-03-21T10:44:43.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.106+0000 7fb497e00640 1 -- 172.21.15.31:0/1364983737 wait complete. 2024-03-21T10:44:43.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.107+0000 7fb497e00640 1 Processor -- start 2024-03-21T10:44:43.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.109+0000 7fb497e00640 1 -- start start 2024-03-21T10:44:43.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.109+0000 7fb497e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb498100610 0x7fb49810aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:43.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.109+0000 7fb497e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb498105690 0x7fb4980fe6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:43.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.109+0000 7fb497e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4980fec30 0x7fb4980ff0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:43.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.109+0000 7fb497e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49806daa0 con 0x7fb498100610 2024-03-21T10:44:43.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.109+0000 7fb497e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49806d920 con 0x7fb498105690 2024-03-21T10:44:43.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.109+0000 7fb497e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49806dc20 con 0x7fb4980fec30 2024-03-21T10:44:43.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.110+0000 7fb497400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4980fec30 0x7fb4980ff0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:43.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.110+0000 7fb497400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4980fec30 0x7fb4980ff0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60168/0 (socket says 172.21.15.31:60168) 2024-03-21T10:44:43.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.110+0000 7fb497400640 1 -- 172.21.15.31:0/257804405 learned_addr learned my addr 172.21.15.31:0/257804405 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:43.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.110+0000 7fb496000640 1 --2- 172.21.15.31:0/257804405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb498105690 0x7fb4980fe6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:43.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.110+0000 7fb497400640 1 -- 172.21.15.31:0/257804405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb498105690 msgr2=0x7fb4980fe6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:43.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.110+0000 7fb497400640 1 --2- 172.21.15.31:0/257804405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb498105690 0x7fb4980fe6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.110+0000 7fb497400640 1 -- 172.21.15.31:0/257804405 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb498100610 msgr2=0x7fb49810aca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T10:44:43.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.113+0000 7fb496a00640 1 --2- 172.21.15.31:0/257804405 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb498100610 0x7fb49810aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:43.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.113+0000 7fb497400640 1 --2- 172.21.15.31:0/257804405 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb498100610 0x7fb49810aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.113+0000 7fb497400640 1 -- 172.21.15.31:0/257804405 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4981a2310 con 0x7fb4980fec30 2024-03-21T10:44:43.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.114+0000 7fb497400640 1 --2- 172.21.15.31:0/257804405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4980fec30 0x7fb4980ff0e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fb48800d950 tx=0x7fb48800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:43.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.114+0000 7fb49ca00640 1 -- 172.21.15.31:0/257804405 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb488017030 con 0x7fb4980fec30 2024-03-21T10:44:43.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.114+0000 7fb49ca00640 1 -- 172.21.15.31:0/257804405 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb488004020 con 0x7fb4980fec30 2024-03-21T10:44:43.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.114+0000 7fb49ca00640 1 -- 172.21.15.31:0/257804405 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb488003c00 con 0x7fb4980fec30 2024-03-21T10:44:43.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.114+0000 7fb497e00640 1 -- 172.21.15.31:0/257804405 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4981a25a0 con 0x7fb4980fec30 2024-03-21T10:44:43.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.114+0000 7fb497e00640 1 -- 172.21.15.31:0/257804405 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4981a2a70 con 0x7fb4980fec30 2024-03-21T10:44:43.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.123+0000 7fb49ca00640 1 -- 172.21.15.31:0/257804405 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fb48801a070 con 0x7fb4980fec30 2024-03-21T10:44:43.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.125+0000 7fb49ca00640 1 --2- 172.21.15.31:0/257804405 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb47c0a3110 0x7fb47c0a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:43.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.126+0000 7fb49ca00640 1 -- 172.21.15.31:0/257804405 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fb488098a50 con 0x7fb4980fec30 2024-03-21T10:44:43.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.127+0000 7fb496a00640 1 --2- 172.21.15.31:0/257804405 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb47c0a3110 0x7fb47c0a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:43.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.127+0000 7fb4a0a3e740 1 Processor -- start 2024-03-21T10:44:43.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.128+0000 7fb4a0a3e740 1 -- start start 2024-03-21T10:44:43.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.128+0000 7fb496a00640 1 --2- 172.21.15.31:0/257804405 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb47c0a3110 0x7fb47c0a55d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb484005fd0 tx=0x7fb48401a270 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:43.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.128+0000 7fb4a0a3e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56209b630290 0x56209b630670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:43.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.128+0000 7fb4a0a3e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56209b630bb0 0x56209b639250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:43.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.128+0000 7fb4a0a3e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56209b639790 0x56209b63bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:43.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.128+0000 7fb4a0a3e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56209b594eb0 con 0x56209b630290 2024-03-21T10:44:43.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.128+0000 7fb4a0a3e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56209b53da60 con 0x56209b639790 2024-03-21T10:44:43.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.128+0000 7fb4a0a3e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56209b590f50 con 0x56209b630bb0 2024-03-21T10:44:43.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.129+0000 7fb497400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56209b639790 0x56209b63bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:43.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.129+0000 7fb496000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56209b630bb0 0x56209b639250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:43.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.130+0000 7fb496a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56209b630290 0x56209b630670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:43.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.130+0000 7fb497400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56209b639790 0x56209b63bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37432/0 (socket says 172.21.15.31:37432) 2024-03-21T10:44:43.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.130+0000 7fb497400640 1 -- 172.21.15.31:0/1360844496 learned_addr learned my addr 172.21.15.31:0/1360844496 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:43.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.132+0000 7fb497400640 1 -- 172.21.15.31:0/1360844496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56209b630bb0 msgr2=0x56209b639250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:43.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.132+0000 7fb497400640 1 --2- 172.21.15.31:0/1360844496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56209b630bb0 0x56209b639250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.132+0000 7fb497400640 1 -- 172.21.15.31:0/1360844496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56209b630290 msgr2=0x56209b630670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:43.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.132+0000 7fb497400640 1 --2- 172.21.15.31:0/1360844496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56209b630290 0x56209b630670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.132+0000 7fb497400640 1 -- 172.21.15.31:0/1360844496 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56209b3717e0 con 0x56209b639790 2024-03-21T10:44:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.134+0000 7fb497400640 1 --2- 172.21.15.31:0/1360844496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56209b639790 0x56209b63bb80 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb488097e00 tx=0x7fb48809fc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1963e31df4185c5e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:43.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.136+0000 7fb483e00640 1 -- 172.21.15.31:0/1360844496 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4880973e0 con 0x56209b639790 2024-03-21T10:44:43.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.136+0000 7fb483e00640 1 -- 172.21.15.31:0/1360844496 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb48801ce50 con 0x56209b639790 2024-03-21T10:44:43.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.138+0000 7fb483e00640 1 -- 172.21.15.31:0/1360844496 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4880973e0 con 0x56209b639790 2024-03-21T10:44:43.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.139+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/1360844496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56209b639790 msgr2=0x56209b63bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:43.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.139+0000 7fb4a0a3e740 1 --2- 172.21.15.31:0/1360844496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56209b639790 0x56209b63bb80 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb488097e00 tx=0x7fb48809fc20 comp rx=0 tx=0).stop 2024-03-21T10:44:43.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.139+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/1360844496 shutdown_connections 2024-03-21T10:44:43.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.139+0000 7fb4a0a3e740 1 --2- 172.21.15.31:0/1360844496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56209b639790 0x56209b63bb80 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.139+0000 7fb4a0a3e740 1 --2- 172.21.15.31:0/1360844496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56209b630bb0 0x56209b639250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.140+0000 7fb4a0a3e740 1 --2- 172.21.15.31:0/1360844496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56209b630290 0x56209b630670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.140+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/1360844496 >> 172.21.15.31:0/1360844496 conn(0x56209b62a9c0 msgr2=0x56209b62ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:43.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.140+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/1360844496 shutdown_connections 2024-03-21T10:44:43.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.140+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/1360844496 wait complete. 2024-03-21T10:44:43.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.141+0000 7fb4a0a3e740 1 -- start start 2024-03-21T10:44:43.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.141+0000 7fb4a0a3e740 1 Processor -- start 2024-03-21T10:44:43.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.143+0000 7fb4a0a3e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56209b630290 0x56209b639150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:43.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.143+0000 7fb4a0a3e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56209b630bb0 0x56209b6364e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:43.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.143+0000 7fb4a0a3e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56209b639790 0x56209b636af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:43.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.143+0000 7fb496a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56209b630290 0x56209b639150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:43.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.144+0000 7fb4a0a3e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56209b64d4d0 con 0x56209b630290 2024-03-21T10:44:43.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.144+0000 7fb496a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56209b630290 0x56209b639150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41814/0 (socket says 172.21.15.31:41814) 2024-03-21T10:44:43.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.144+0000 7fb496a00640 1 -- 172.21.15.31:0/3271342958 learned_addr learned my addr 172.21.15.31:0/3271342958 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:43.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.144+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/3271342958 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56209b550eb0 con 0x56209b639790 2024-03-21T10:44:43.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.144+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/3271342958 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56209b64d650 con 0x56209b630bb0 2024-03-21T10:44:43.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.144+0000 7fb4a0a3e740 10 client.0 _ll_register_callbacks cb 0x56209b545150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:43.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.144+0000 7fb496000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56209b630bb0 0x56209b6364e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:43.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.145+0000 7fb496a00640 1 -- 172.21.15.31:0/3271342958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56209b630bb0 msgr2=0x56209b6364e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:43.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.145+0000 7fb496a00640 1 --2- 172.21.15.31:0/3271342958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56209b630bb0 0x56209b6364e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.145+0000 7fb496a00640 1 -- 172.21.15.31:0/3271342958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56209b639790 msgr2=0x56209b636af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:43.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.145+0000 7fb496a00640 1 --2- 172.21.15.31:0/3271342958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56209b639790 0x56209b636af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.145+0000 7fb496a00640 1 -- 172.21.15.31:0/3271342958 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56209b3d1f70 con 0x56209b630290 2024-03-21T10:44:43.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.146+0000 7fb496a00640 1 --2- 172.21.15.31:0/3271342958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56209b630290 0x56209b639150 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb484022dc0 tx=0x7fb484004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=929f92c38a3cd8f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:43.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.146+0000 7fb480c00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:44:43.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.147+0000 7fb480c00640 1 -- 172.21.15.31:0/3271342958 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb484021c10 con 0x56209b630290 2024-03-21T10:44:43.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.147+0000 7fb480c00640 1 -- 172.21.15.31:0/3271342958 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fb484021db0 con 0x56209b630290 2024-03-21T10:44:43.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.148+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/3271342958 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb48801ce50 con 0x56209b630290 2024-03-21T10:44:43.147 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:43 smithi031 ceph-mon[31673]: pgmap v187: 129 pgs: 129 active+clean; 39 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 187 MiB/s wr, 1.37k op/s 2024-03-21T10:44:43.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.170+0000 7fb480c00640 1 -- 172.21.15.31:0/3271342958 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb48403b480 con 0x56209b630290 2024-03-21T10:44:43.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.170+0000 7fb4a0a3e740 10 client.16635 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:43.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.170+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/3271342958 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x56209b269170 con 0x56209b630290 2024-03-21T10:44:43.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.171+0000 7fb480c00640 1 -- 172.21.15.31:0/3271342958 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb48801ce50 con 0x56209b630290 2024-03-21T10:44:43.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.174+0000 7fb480c00640 1 -- 172.21.15.31:0/3271342958 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fb48401f050 con 0x56209b630290 2024-03-21T10:44:43.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.178+0000 7fb4a0a3e740 10 client.16635 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:43.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.178+0000 7fb4a0a3e740 10 client.16635 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:44:43.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.178+0000 7fb4a0a3e740 20 client.16635 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:43.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.178+0000 7fb4a0a3e740 1 --2- 172.21.15.31:0/3271342958 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56209b65a800 0x56209b65cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:43.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.178+0000 7fb4a0a3e740 4 client.16635 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:44:43.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.178+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/3271342958 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56209b5a23f0 con 0x56209b65a800 2024-03-21T10:44:43.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.179+0000 7fb496000640 1 --2- 172.21.15.31:0/3271342958 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56209b65a800 0x56209b65cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:43.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.192+0000 7fb480c00640 10 client.16635 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:43.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.192+0000 7fb496000640 1 --2- 172.21.15.31:0/3271342958 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56209b65a800 0x56209b65cc50 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7dfbc9d2433e7d4 server_cookie=8f9033ff4a4d2529 in_seq=0 out_seq=0 2024-03-21T10:44:43.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.194+0000 7fb480c00640 1 -- 172.21.15.31:0/3271342958 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56209b5a23f0 con 0x56209b65a800 2024-03-21T10:44:43.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.194+0000 7fb480c00640 10 client.16635 handle_command_reply: tid=0 2024-03-21T10:44:43.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.194+0000 7fb4a0a3e740 1 client.16635 shutdown 2024-03-21T10:44:43.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.196+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/3271342958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56209b630290 msgr2=0x56209b639150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:43.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.196+0000 7fb4a0a3e740 1 --2- 172.21.15.31:0/3271342958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56209b630290 0x56209b639150 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb484022dc0 tx=0x7fb484004730 comp rx=0 tx=0).stop 2024-03-21T10:44:43.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.197+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/3271342958 shutdown_connections 2024-03-21T10:44:43.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.197+0000 7fb4a0a3e740 1 --2- 172.21.15.31:0/3271342958 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56209b65a800 0x56209b65cc50 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.197+0000 7fb4a0a3e740 1 --2- 172.21.15.31:0/3271342958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56209b639790 0x56209b636af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.197+0000 7fb4a0a3e740 1 --2- 172.21.15.31:0/3271342958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56209b630bb0 0x56209b6364e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.197+0000 7fb4a0a3e740 1 --2- 172.21.15.31:0/3271342958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56209b630290 0x56209b639150 secure :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb484022dc0 tx=0x7fb484004730 comp rx=0 tx=0).stop 2024-03-21T10:44:43.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.197+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/3271342958 >> 172.21.15.31:0/3271342958 conn(0x56209b62a9c0 msgr2=0x56209b62c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:43.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.199+0000 7fb480c00640 0 client.16635 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:43.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.199+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/3271342958 shutdown_connections 2024-03-21T10:44:43.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.200+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/3271342958 wait complete. 2024-03-21T10:44:43.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.201+0000 7fb4a0a3e740 20 client.16635 trim_cache size 0 max 16384 2024-03-21T10:44:43.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.288+0000 7fb4a0a3e740 1 Processor -- start 2024-03-21T10:44:43.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.288+0000 7fb4a0a3e740 1 -- start start 2024-03-21T10:44:43.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.289+0000 7fb4a0a3e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56209b630bb0 0x56209b638c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:43.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.289+0000 7fb4a0a3e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56209b639790 0x56209b639150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:43.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.289+0000 7fb4a0a3e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56209b662560 0x56209b655d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:43.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.290+0000 7fb4a0a3e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb478012160 con 0x56209b630bb0 2024-03-21T10:44:43.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.290+0000 7fb4a0a3e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47800fc60 con 0x56209b662560 2024-03-21T10:44:43.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.290+0000 7fb4a0a3e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47800dd00 con 0x56209b639790 2024-03-21T10:44:43.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.290+0000 7fb497400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56209b662560 0x56209b655d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:43.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.290+0000 7fb497400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56209b662560 0x56209b655d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37446/0 (socket says 172.21.15.31:37446) 2024-03-21T10:44:43.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.290+0000 7fb497400640 1 -- 172.21.15.31:0/2746271683 learned_addr learned my addr 172.21.15.31:0/2746271683 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:43.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.291+0000 7fb497400640 1 -- 172.21.15.31:0/2746271683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56209b639790 msgr2=0x56209b639150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:44:43.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.291+0000 7fb497400640 1 --2- 172.21.15.31:0/2746271683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56209b639790 0x56209b639150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.291+0000 7fb497400640 1 -- 172.21.15.31:0/2746271683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56209b630bb0 msgr2=0x56209b638c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:44:43.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.291+0000 7fb497400640 1 --2- 172.21.15.31:0/2746271683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56209b630bb0 0x56209b638c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.291+0000 7fb497400640 1 -- 172.21.15.31:0/2746271683 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56209b636e60 con 0x56209b662560 2024-03-21T10:44:43.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.292+0000 7fb497400640 1 --2- 172.21.15.31:0/2746271683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56209b662560 0x56209b655d90 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb488096e10 tx=0x7fb488005bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3b79e90a93b30ed server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:43.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.293+0000 7fb483400640 1 -- 172.21.15.31:0/2746271683 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4880b2070 con 0x56209b662560 2024-03-21T10:44:43.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.293+0000 7fb483400640 1 -- 172.21.15.31:0/2746271683 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb4880a2050 con 0x56209b662560 2024-03-21T10:44:43.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.293+0000 7fb483400640 1 -- 172.21.15.31:0/2746271683 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb488097580 con 0x56209b662560 2024-03-21T10:44:43.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.296+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/2746271683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56209b662560 msgr2=0x56209b655d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:43.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.296+0000 7fb4a0a3e740 1 --2- 172.21.15.31:0/2746271683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56209b662560 0x56209b655d90 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb488096e10 tx=0x7fb488005bf0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.302+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/2746271683 shutdown_connections 2024-03-21T10:44:43.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.302+0000 7fb4a0a3e740 1 --2- 172.21.15.31:0/2746271683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56209b662560 0x56209b655d90 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.302+0000 7fb4a0a3e740 1 --2- 172.21.15.31:0/2746271683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56209b639790 0x56209b639150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.302+0000 7fb4a0a3e740 1 --2- 172.21.15.31:0/2746271683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56209b630bb0 0x56209b638c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.302+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/2746271683 >> 172.21.15.31:0/2746271683 conn(0x56209b62a9c0 msgr2=0x56209b629a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:43.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.302+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/2746271683 shutdown_connections 2024-03-21T10:44:43.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.304+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/2746271683 wait complete. 2024-03-21T10:44:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.305+0000 7fb4a0a3e740 1 -- start start 2024-03-21T10:44:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.306+0000 7fb4a0a3e740 1 Processor -- start 2024-03-21T10:44:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.307+0000 7fb4a0a3e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56209b630bb0 0x56209b634160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.307+0000 7fb4a0a3e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56209b639790 0x56209b64f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.307+0000 7fb4a0a3e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56209b662560 0x56209b64fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:43.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.307+0000 7fb4a0a3e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47800e300 con 0x56209b630bb0 2024-03-21T10:44:43.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.307+0000 7fb4a0a3e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47800f600 con 0x56209b639790 2024-03-21T10:44:43.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.307+0000 7fb4a0a3e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47800e940 con 0x56209b662560 2024-03-21T10:44:43.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.307+0000 7fb4a0a3e740 10 client.0 _ll_register_callbacks cb 0x7fb484021b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:43.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.307+0000 7fb497400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56209b662560 0x56209b64fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:43.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.307+0000 7fb497400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56209b662560 0x56209b64fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60200/0 (socket says 172.21.15.31:60200) 2024-03-21T10:44:43.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.307+0000 7fb497400640 1 -- 172.21.15.31:0/3964018308 learned_addr learned my addr 172.21.15.31:0/3964018308 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:43.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.308+0000 7fb497400640 1 -- 172.21.15.31:0/3964018308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56209b639790 msgr2=0x56209b64f8a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:44:43.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.308+0000 7fb497400640 1 --2- 172.21.15.31:0/3964018308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56209b639790 0x56209b64f8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.308+0000 7fb497400640 1 -- 172.21.15.31:0/3964018308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56209b630bb0 msgr2=0x56209b634160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:44:43.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.308+0000 7fb497400640 1 --2- 172.21.15.31:0/3964018308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56209b630bb0 0x56209b634160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.308+0000 7fb497400640 1 -- 172.21.15.31:0/3964018308 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56209b6504f0 con 0x56209b662560 2024-03-21T10:44:43.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.316+0000 7fb497400640 1 --2- 172.21.15.31:0/3964018308 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56209b662560 0x56209b64fde0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fb488005fd0 tx=0x7fb48809aba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45bb4f7b01d3c2a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:43.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.316+0000 7fb480c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:44:43.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.316+0000 7fb480c00640 1 -- 172.21.15.31:0/3964018308 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4880b7070 con 0x56209b662560 2024-03-21T10:44:43.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.316+0000 7fb480c00640 1 -- 172.21.15.31:0/3964018308 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fb488096a80 con 0x56209b662560 2024-03-21T10:44:43.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.317+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/3964018308 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb4880a2050 con 0x56209b662560 2024-03-21T10:44:43.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.317+0000 7fb480c00640 1 -- 172.21.15.31:0/3964018308 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb4880b2070 con 0x56209b662560 2024-03-21T10:44:43.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.317+0000 7fb4a0a3e740 10 client.27091 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:43.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.317+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/3964018308 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x56209b63c180 con 0x56209b662560 2024-03-21T10:44:43.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.317+0000 7fb480c00640 1 -- 172.21.15.31:0/3964018308 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb4880b9050 con 0x56209b662560 2024-03-21T10:44:43.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.318+0000 7fb480c00640 1 -- 172.21.15.31:0/3964018308 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fb48809bce0 con 0x56209b662560 2024-03-21T10:44:43.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.318+0000 7fb4a0a3e740 10 client.27091 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:43.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.318+0000 7fb4a0a3e740 10 client.27091 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:44:43.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.318+0000 7fb4a0a3e740 20 client.27091 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:43.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.318+0000 7fb4a0a3e740 1 --2- 172.21.15.31:0/3964018308 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56209b63c790 0x56209b63ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:43.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.318+0000 7fb4a0a3e740 4 client.27091 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-21T10:44:43.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.318+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/3964018308 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x7fb4880a2050 con 0x56209b63c790 2024-03-21T10:44:43.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.319+0000 7fb496a00640 1 --2- 172.21.15.31:0/3964018308 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56209b63c790 0x56209b63ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:43.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.321+0000 7fb480c00640 10 client.27091 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:43.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.321+0000 7fb496a00640 1 --2- 172.21.15.31:0/3964018308 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56209b63c790 0x56209b63ebb0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=baec32e2665369e6 server_cookie=4ecd4392d0eb18ec in_seq=0 out_seq=0 2024-03-21T10:44:43.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.322+0000 7fb480c00640 1 -- 172.21.15.31:0/3964018308 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x7fb4880a2050 con 0x56209b63c790 2024-03-21T10:44:43.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.322+0000 7fb480c00640 10 client.27091 handle_command_reply: tid=0 2024-03-21T10:44:43.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.322+0000 7fb4a0a3e740 1 client.27091 shutdown 2024-03-21T10:44:43.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.323+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/3964018308 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56209b662560 msgr2=0x56209b64fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:43.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.323+0000 7fb4a0a3e740 1 --2- 172.21.15.31:0/3964018308 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56209b662560 0x56209b64fde0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fb488005fd0 tx=0x7fb48809aba0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.323+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/3964018308 shutdown_connections 2024-03-21T10:44:43.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.323+0000 7fb4a0a3e740 1 --2- 172.21.15.31:0/3964018308 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56209b63c790 0x56209b63ebb0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.323+0000 7fb4a0a3e740 1 --2- 172.21.15.31:0/3964018308 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56209b662560 0x56209b64fde0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.323+0000 7fb4a0a3e740 1 --2- 172.21.15.31:0/3964018308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56209b639790 0x56209b64f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.323+0000 7fb4a0a3e740 1 --2- 172.21.15.31:0/3964018308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56209b630bb0 0x56209b634160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.323+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/3964018308 >> 172.21.15.31:0/3964018308 conn(0x56209b62a9c0 msgr2=0x56209b661550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:43.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.323+0000 7fb480c00640 0 client.27091 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:43.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.327+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/3964018308 shutdown_connections 2024-03-21T10:44:43.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.327+0000 7fb4a0a3e740 1 -- 172.21.15.31:0/3964018308 wait complete. 2024-03-21T10:44:43.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.328+0000 7fb4a0a3e740 20 client.27091 trim_cache size 0 max 16384 2024-03-21T10:44:43.326 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:44:43.326 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-03-21T10:44:43.326 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "f5b66cc1-dd46-405b-bf57-467912e3e749", 2024-03-21T10:44:43.326 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-03-21T10:44:43.326 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:44:43.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.331+0000 7fb497e00640 1 -- 172.21.15.31:0/257804405 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb47c0a3110 msgr2=0x7fb47c0a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:43.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.331+0000 7fb497e00640 1 --2- 172.21.15.31:0/257804405 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb47c0a3110 0x7fb47c0a55d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb484005fd0 tx=0x7fb48401a270 comp rx=0 tx=0).stop 2024-03-21T10:44:43.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.331+0000 7fb497e00640 1 -- 172.21.15.31:0/257804405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4980fec30 msgr2=0x7fb4980ff0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:43.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.331+0000 7fb497e00640 1 --2- 172.21.15.31:0/257804405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4980fec30 0x7fb4980ff0e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fb48800d950 tx=0x7fb48800de10 comp rx=0 tx=0).stop 2024-03-21T10:44:43.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.335+0000 7fb497e00640 1 -- 172.21.15.31:0/257804405 shutdown_connections 2024-03-21T10:44:43.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.335+0000 7fb497e00640 1 --2- 172.21.15.31:0/257804405 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb47c0a3110 0x7fb47c0a55d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.335+0000 7fb497e00640 1 --2- 172.21.15.31:0/257804405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4980fec30 0x7fb4980ff0e0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.335+0000 7fb497e00640 1 --2- 172.21.15.31:0/257804405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb498105690 0x7fb4980fe6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.335+0000 7fb497e00640 1 --2- 172.21.15.31:0/257804405 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb498100610 0x7fb49810aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:43.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.335+0000 7fb497e00640 1 -- 172.21.15.31:0/257804405 >> 172.21.15.31:0/257804405 conn(0x7fb498077130 msgr2=0x7fb498070360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:43.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.336+0000 7fb497e00640 1 -- 172.21.15.31:0/257804405 shutdown_connections 2024-03-21T10:44:43.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:43.336+0000 7fb497e00640 1 -- 172.21.15.31:0/257804405 wait complete. 2024-03-21T10:44:43.391 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:44:43.392 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:44:43.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:43 smithi040 ceph-mon[36154]: pgmap v187: 129 pgs: 129 active+clean; 39 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 187 MiB/s wr, 1.37k op/s 2024-03-21T10:44:43.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:43 smithi070 ceph-mon[36330]: pgmap v187: 129 pgs: 129 active+clean; 39 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 187 MiB/s wr, 1.37k op/s 2024-03-21T10:44:44.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.088+0000 7ff9db200640 1 -- 172.21.15.31:0/920093102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9dc10fc90 msgr2=0x7ff9dc110070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:44.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.088+0000 7ff9db200640 1 --2- 172.21.15.31:0/920093102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9dc10fc90 0x7ff9dc110070 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc009890 tx=0x7ff9cc02bb90 comp rx=0 tx=0).stop 2024-03-21T10:44:44.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.089+0000 7ff9db200640 1 -- 172.21.15.31:0/920093102 shutdown_connections 2024-03-21T10:44:44.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.089+0000 7ff9db200640 1 --2- 172.21.15.31:0/920093102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9dc10e310 0x7ff9dc11ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.089+0000 7ff9db200640 1 --2- 172.21.15.31:0/920093102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9dc10d780 0x7ff9dc10dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.089+0000 7ff9db200640 1 --2- 172.21.15.31:0/920093102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9dc10fc90 0x7ff9dc110070 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.089+0000 7ff9db200640 1 -- 172.21.15.31:0/920093102 >> 172.21.15.31:0/920093102 conn(0x7ff9dc06c580 msgr2=0x7ff9dc06c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:44.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.089+0000 7ff9db200640 1 -- 172.21.15.31:0/920093102 shutdown_connections 2024-03-21T10:44:44.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.089+0000 7ff9db200640 1 -- 172.21.15.31:0/920093102 wait complete. 2024-03-21T10:44:44.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.090+0000 7ff9db200640 1 Processor -- start 2024-03-21T10:44:44.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.091+0000 7ff9db200640 1 -- start start 2024-03-21T10:44:44.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.092+0000 7ff9db200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9dc10d780 0x7ff9dc1ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:44.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.092+0000 7ff9db200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9dc10e310 0x7ff9dc1acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:44.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.092+0000 7ff9db200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9dc10fc90 0x7ff9dc1a6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:44.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.093+0000 7ff9db200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9dc120660 con 0x7ff9dc10e310 2024-03-21T10:44:44.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.093+0000 7ff9da800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9dc10fc90 0x7ff9dc1a6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:44.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.093+0000 7ff9da800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9dc10fc90 0x7ff9dc1a6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:60218/0 (socket says 172.21.15.31:60218) 2024-03-21T10:44:44.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.093+0000 7ff9da800640 1 -- 172.21.15.31:0/2737884736 learned_addr learned my addr 172.21.15.31:0/2737884736 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:44.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.093+0000 7ff9d9400640 1 --2- 172.21.15.31:0/2737884736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9dc10e310 0x7ff9dc1acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:44.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.093+0000 7ff9db200640 1 -- 172.21.15.31:0/2737884736 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9dc1204e0 con 0x7ff9dc10d780 2024-03-21T10:44:44.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.094+0000 7ff9db200640 1 -- 172.21.15.31:0/2737884736 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9dc1207e0 con 0x7ff9dc10fc90 2024-03-21T10:44:44.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.094+0000 7ff9d9e00640 1 --2- 172.21.15.31:0/2737884736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9dc10d780 0x7ff9dc1ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:44.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.095+0000 7ff9da800640 1 -- 172.21.15.31:0/2737884736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9dc10d780 msgr2=0x7ff9dc1ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:44.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.095+0000 7ff9da800640 1 --2- 172.21.15.31:0/2737884736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9dc10d780 0x7ff9dc1ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.095+0000 7ff9da800640 1 -- 172.21.15.31:0/2737884736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9dc10e310 msgr2=0x7ff9dc1acd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:44.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.095+0000 7ff9da800640 1 --2- 172.21.15.31:0/2737884736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9dc10e310 0x7ff9dc1acd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.095+0000 7ff9da800640 1 -- 172.21.15.31:0/2737884736 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9dc1a7180 con 0x7ff9dc10fc90 2024-03-21T10:44:44.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.096+0000 7ff9da800640 1 --2- 172.21.15.31:0/2737884736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9dc10fc90 0x7ff9dc1a6a00 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff9d400cad0 tx=0x7ff9d400ced0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:44.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.096+0000 7ff9d2a00640 1 -- 172.21.15.31:0/2737884736 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d400ad60 con 0x7ff9dc10fc90 2024-03-21T10:44:44.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.096+0000 7ff9d2a00640 1 -- 172.21.15.31:0/2737884736 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9d4005020 con 0x7ff9dc10fc90 2024-03-21T10:44:44.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.096+0000 7ff9d2a00640 1 -- 172.21.15.31:0/2737884736 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d400f6b0 con 0x7ff9dc10fc90 2024-03-21T10:44:44.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.096+0000 7ff9db200640 1 -- 172.21.15.31:0/2737884736 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9dc1a7470 con 0x7ff9dc10fc90 2024-03-21T10:44:44.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.096+0000 7ff9db200640 1 -- 172.21.15.31:0/2737884736 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9dc1b3710 con 0x7ff9dc10fc90 2024-03-21T10:44:44.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.103+0000 7ff9d2a00640 1 -- 172.21.15.31:0/2737884736 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7ff9d4004b20 con 0x7ff9dc10fc90 2024-03-21T10:44:44.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.105+0000 7ff9d2a00640 1 --2- 172.21.15.31:0/2737884736 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff9c00a3290 0x7ff9c00a5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:44.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.106+0000 7ff9d2a00640 1 -- 172.21.15.31:0/2737884736 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7ff9d40cf940 con 0x7ff9dc10fc90 2024-03-21T10:44:44.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.107+0000 7ff9d9e00640 1 --2- 172.21.15.31:0/2737884736 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff9c00a3290 0x7ff9c00a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:44.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.108+0000 7ff9d9e00640 1 --2- 172.21.15.31:0/2737884736 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff9c00a3290 0x7ff9c00a5750 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc000c00 tx=0x7ff9cc005a90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:44.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.109+0000 7ff9e3821740 1 Processor -- start 2024-03-21T10:44:44.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.109+0000 7ff9e3821740 1 -- start start 2024-03-21T10:44:44.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.110+0000 7ff9e3821740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556bf0244dd0 0x556bf0243b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:44.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.110+0000 7ff9e3821740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bf02440f0 0x556bf0244570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:44.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.110+0000 7ff9e3821740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bf024d460 0x556bf024f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:44.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.110+0000 7ff9e3821740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bf01a8250 con 0x556bf0244dd0 2024-03-21T10:44:44.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.110+0000 7ff9e3821740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bf0150a30 con 0x556bf024d460 2024-03-21T10:44:44.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.110+0000 7ff9e3821740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bf01a42f0 con 0x556bf02440f0 2024-03-21T10:44:44.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.110+0000 7ff9da800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bf024d460 0x556bf024f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:44.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.110+0000 7ff9d9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556bf0244dd0 0x556bf0243b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:44.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.111+0000 7ff9d9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556bf0244dd0 0x556bf0243b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41846/0 (socket says 172.21.15.31:41846) 2024-03-21T10:44:44.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.111+0000 7ff9d9e00640 1 -- 172.21.15.31:0/3990248530 learned_addr learned my addr 172.21.15.31:0/3990248530 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:44.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.111+0000 7ff9d9400640 1 --2- 172.21.15.31:0/3990248530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bf02440f0 0x556bf0244570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:44.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.111+0000 7ff9da800640 1 -- 172.21.15.31:0/3990248530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bf02440f0 msgr2=0x556bf0244570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:44.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.111+0000 7ff9da800640 1 --2- 172.21.15.31:0/3990248530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bf02440f0 0x556bf0244570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.111+0000 7ff9da800640 1 -- 172.21.15.31:0/3990248530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556bf0244dd0 msgr2=0x556bf0243b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:44.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.111+0000 7ff9da800640 1 --2- 172.21.15.31:0/3990248530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556bf0244dd0 0x556bf0243b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.111+0000 7ff9da800640 1 -- 172.21.15.31:0/3990248530 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556bf013b690 con 0x556bf024d460 2024-03-21T10:44:44.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.113+0000 7ff9da800640 1 --2- 172.21.15.31:0/3990248530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bf024d460 0x556bf024f850 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40cccf0 tx=0x7ff9d409fae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d50703a2aae69f79 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:44.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.114+0000 7ff9c7000640 1 -- 172.21.15.31:0/3990248530 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d401d040 con 0x556bf024d460 2024-03-21T10:44:44.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.114+0000 7ff9c7000640 1 -- 172.21.15.31:0/3990248530 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9d4098000 con 0x556bf024d460 2024-03-21T10:44:44.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.114+0000 7ff9c7000640 1 -- 172.21.15.31:0/3990248530 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d409a850 con 0x556bf024d460 2024-03-21T10:44:44.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.115+0000 7ff9e3821740 1 -- 172.21.15.31:0/3990248530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bf024d460 msgr2=0x556bf024f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:44.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.115+0000 7ff9e3821740 1 --2- 172.21.15.31:0/3990248530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bf024d460 0x556bf024f850 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40cccf0 tx=0x7ff9d409fae0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.116+0000 7ff9e3821740 1 -- 172.21.15.31:0/3990248530 shutdown_connections 2024-03-21T10:44:44.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.116+0000 7ff9e3821740 1 --2- 172.21.15.31:0/3990248530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bf024d460 0x556bf024f850 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.116+0000 7ff9e3821740 1 --2- 172.21.15.31:0/3990248530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bf02440f0 0x556bf0244570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.116+0000 7ff9e3821740 1 --2- 172.21.15.31:0/3990248530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556bf0244dd0 0x556bf0243b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.116+0000 7ff9e3821740 1 -- 172.21.15.31:0/3990248530 >> 172.21.15.31:0/3990248530 conn(0x556bf020b200 msgr2=0x556bf023eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:44.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.116+0000 7ff9e3821740 1 -- 172.21.15.31:0/3990248530 shutdown_connections 2024-03-21T10:44:44.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.116+0000 7ff9e3821740 1 -- 172.21.15.31:0/3990248530 wait complete. 2024-03-21T10:44:44.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.117+0000 7ff9e3821740 1 -- start start 2024-03-21T10:44:44.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.117+0000 7ff9e3821740 1 Processor -- start 2024-03-21T10:44:44.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.118+0000 7ff9e3821740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bf02440f0 0x556bf024cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:44.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.119+0000 7ff9e3821740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556bf0244dd0 0x556bf0249f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:44.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.119+0000 7ff9d9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bf02440f0 0x556bf024cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:44.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.119+0000 7ff9d9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bf02440f0 0x556bf024cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37476/0 (socket says 172.21.15.31:37476) 2024-03-21T10:44:44.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.119+0000 7ff9e3821740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bf024d460 0x556bf024a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:44.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.119+0000 7ff9d9e00640 1 -- 172.21.15.31:0/3945462551 learned_addr learned my addr 172.21.15.31:0/3945462551 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:44.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.120+0000 7ff9da800640 1 --2- 172.21.15.31:0/3945462551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bf024d460 0x556bf024a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:44.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.120+0000 7ff9e3821740 1 -- 172.21.15.31:0/3945462551 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bf0250270 con 0x556bf0244dd0 2024-03-21T10:44:44.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.120+0000 7ff9e3821740 1 -- 172.21.15.31:0/3945462551 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bf0163e80 con 0x556bf02440f0 2024-03-21T10:44:44.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.120+0000 7ff9e3821740 1 -- 172.21.15.31:0/3945462551 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bf02503f0 con 0x556bf024d460 2024-03-21T10:44:44.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.120+0000 7ff9e3821740 10 client.0 _ll_register_callbacks cb 0x556bf0176130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:44.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.120+0000 7ff9da800640 1 -- 172.21.15.31:0/3945462551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bf02440f0 msgr2=0x556bf024cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:44.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.120+0000 7ff9da800640 1 --2- 172.21.15.31:0/3945462551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bf02440f0 0x556bf024cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.120+0000 7ff9da800640 1 -- 172.21.15.31:0/3945462551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556bf0244dd0 msgr2=0x556bf0249f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:44.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.120+0000 7ff9da800640 1 --2- 172.21.15.31:0/3945462551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556bf0244dd0 0x556bf0249f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.120+0000 7ff9da800640 1 -- 172.21.15.31:0/3945462551 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556beff867d0 con 0x556bf024d460 2024-03-21T10:44:44.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.121+0000 7ff9da800640 1 --2- 172.21.15.31:0/3945462551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bf024d460 0x556bf024a450 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40ccd20 tx=0x7ff9d4004f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d296f1661d468a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:44.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.121+0000 7ff9bb400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:44:44.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.121+0000 7ff9bb400640 1 -- 172.21.15.31:0/3945462551 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d40b9060 con 0x556bf024d460 2024-03-21T10:44:44.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.121+0000 7ff9bb400640 1 -- 172.21.15.31:0/3945462551 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7ff9d40976e0 con 0x556bf024d460 2024-03-21T10:44:44.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.122+0000 7ff9e3821740 1 -- 172.21.15.31:0/3945462551 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff9d4098000 con 0x556bf024d460 2024-03-21T10:44:44.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.122+0000 7ff9bb400640 1 -- 172.21.15.31:0/3945462551 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9d401d040 con 0x556bf024d460 2024-03-21T10:44:44.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.123+0000 7ff9e3821740 10 client.27103 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:44.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.123+0000 7ff9e3821740 1 -- 172.21.15.31:0/3945462551 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x556befebc3f0 con 0x556bf024d460 2024-03-21T10:44:44.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.123+0000 7ff9bb400640 1 -- 172.21.15.31:0/3945462551 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9d4098000 con 0x556bf024d460 2024-03-21T10:44:44.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.124+0000 7ff9bb400640 1 -- 172.21.15.31:0/3945462551 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7ff9d40a0c70 con 0x556bf024d460 2024-03-21T10:44:44.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.124+0000 7ff9e3821740 10 client.27103 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:44.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.124+0000 7ff9e3821740 10 client.27103 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:44:44.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.124+0000 7ff9e3821740 20 client.27103 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:44.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.124+0000 7ff9e3821740 1 --2- 172.21.15.31:0/3945462551 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556bf025d1c0 0x556bf025f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:44.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.124+0000 7ff9e3821740 4 client.27103 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:44:44.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.124+0000 7ff9e3821740 1 -- 172.21.15.31:0/3945462551 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556bf01b5790 con 0x556bf025d1c0 2024-03-21T10:44:44.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.126+0000 7ff9d9e00640 1 --2- 172.21.15.31:0/3945462551 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556bf025d1c0 0x556bf025f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:44.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.127+0000 7ff9d9e00640 1 --2- 172.21.15.31:0/3945462551 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556bf025d1c0 0x556bf025f5b0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eef1896f2b294e51 server_cookie=20746bdc74d9662e in_seq=0 out_seq=0 2024-03-21T10:44:44.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.127+0000 7ff9bb400640 10 client.27103 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:44.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.128+0000 7ff9bb400640 1 -- 172.21.15.31:0/3945462551 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556bf01b5790 con 0x556bf025d1c0 2024-03-21T10:44:44.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.128+0000 7ff9bb400640 10 client.27103 handle_command_reply: tid=0 2024-03-21T10:44:44.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.128+0000 7ff9e3821740 1 client.27103 shutdown 2024-03-21T10:44:44.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.128+0000 7ff9e3821740 1 -- 172.21.15.31:0/3945462551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bf024d460 msgr2=0x556bf024a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:44.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.128+0000 7ff9e3821740 1 --2- 172.21.15.31:0/3945462551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bf024d460 0x556bf024a450 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40ccd20 tx=0x7ff9d4004f60 comp rx=0 tx=0).stop 2024-03-21T10:44:44.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.129+0000 7ff9e3821740 1 -- 172.21.15.31:0/3945462551 shutdown_connections 2024-03-21T10:44:44.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.129+0000 7ff9e3821740 1 --2- 172.21.15.31:0/3945462551 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556bf025d1c0 0x556bf025f5b0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.129+0000 7ff9e3821740 1 --2- 172.21.15.31:0/3945462551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bf024d460 0x556bf024a450 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.129+0000 7ff9bb400640 0 client.27103 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:44.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.129+0000 7ff9e3821740 1 --2- 172.21.15.31:0/3945462551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556bf0244dd0 0x556bf0249f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.129+0000 7ff9e3821740 1 --2- 172.21.15.31:0/3945462551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bf02440f0 0x556bf024cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.129+0000 7ff9e3821740 1 -- 172.21.15.31:0/3945462551 >> 172.21.15.31:0/3945462551 conn(0x556bf020b200 msgr2=0x556bf0240890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:44.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.129+0000 7ff9e3821740 1 -- 172.21.15.31:0/3945462551 shutdown_connections 2024-03-21T10:44:44.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.129+0000 7ff9e3821740 1 -- 172.21.15.31:0/3945462551 wait complete. 2024-03-21T10:44:44.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.130+0000 7ff9e3821740 20 client.27103 trim_cache size 0 max 16384 2024-03-21T10:44:44.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.165+0000 7ff9e3821740 1 Processor -- start 2024-03-21T10:44:44.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.169+0000 7ff9e3821740 1 -- start start 2024-03-21T10:44:44.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.169+0000 7ff9e3821740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bf02440f0 0x556bf024c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:44.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.169+0000 7ff9e3821740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bf0244dd0 0x556bf024cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:44.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.169+0000 7ff9e3821740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556bf024d460 0x556bf024e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:44.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.170+0000 7ff9d9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bf02440f0 0x556bf024c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:44.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.170+0000 7ff9d9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bf02440f0 0x556bf024c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37486/0 (socket says 172.21.15.31:37486) 2024-03-21T10:44:44.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.172+0000 7ff9e3821740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9bc012180 con 0x556bf024d460 2024-03-21T10:44:44.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.172+0000 7ff9e3821740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9bc00fc80 con 0x556bf02440f0 2024-03-21T10:44:44.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.172+0000 7ff9e3821740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9bc00dd20 con 0x556bf0244dd0 2024-03-21T10:44:44.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.171+0000 7ff9da800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556bf024d460 0x556bf024e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:44.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.172+0000 7ff9da800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556bf024d460 0x556bf024e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41862/0 (socket says 172.21.15.31:41862) 2024-03-21T10:44:44.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.172+0000 7ff9da800640 1 -- 172.21.15.31:0/2076847681 learned_addr learned my addr 172.21.15.31:0/2076847681 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:44.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.173+0000 7ff9d9e00640 1 -- 172.21.15.31:0/2076847681 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bf0244dd0 msgr2=0x556bf024cb80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:44:44.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.176+0000 7ff9d9400640 1 --2- 172.21.15.31:0/2076847681 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bf0244dd0 0x556bf024cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:44.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.176+0000 7ff9d9e00640 1 --2- 172.21.15.31:0/2076847681 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bf0244dd0 0x556bf024cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.177+0000 7ff9d9e00640 1 -- 172.21.15.31:0/2076847681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556bf024d460 msgr2=0x556bf024e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:44.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.177+0000 7ff9d9e00640 1 --2- 172.21.15.31:0/2076847681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556bf024d460 0x556bf024e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.177+0000 7ff9d9e00640 1 -- 172.21.15.31:0/2076847681 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556befe7c0f0 con 0x556bf02440f0 2024-03-21T10:44:44.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.178+0000 7ff9da800640 1 --2- 172.21.15.31:0/2076847681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556bf024d460 0x556bf024e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:44:44.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.181+0000 7ff9d9e00640 1 --2- 172.21.15.31:0/2076847681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bf02440f0 0x556bf024c640 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40cccf0 tx=0x7ff9cc03e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be7d1391112f357 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:44.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.181+0000 7ff9d9400640 1 --2- 172.21.15.31:0/2076847681 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bf0244dd0 0x556bf024cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:44:44.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.183+0000 7ff9c6600640 1 -- 172.21.15.31:0/2076847681 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc059070 con 0x556bf02440f0 2024-03-21T10:44:44.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.184+0000 7ff9c6600640 1 -- 172.21.15.31:0/2076847681 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9cc005520 con 0x556bf02440f0 2024-03-21T10:44:44.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.185+0000 7ff9c6600640 1 -- 172.21.15.31:0/2076847681 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc059070 con 0x556bf02440f0 2024-03-21T10:44:44.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.188+0000 7ff9e3821740 1 -- 172.21.15.31:0/2076847681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bf02440f0 msgr2=0x556bf024c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:44.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.188+0000 7ff9e3821740 1 --2- 172.21.15.31:0/2076847681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bf02440f0 0x556bf024c640 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40cccf0 tx=0x7ff9cc03e000 comp rx=0 tx=0).stop 2024-03-21T10:44:44.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.188+0000 7ff9e3821740 1 -- 172.21.15.31:0/2076847681 shutdown_connections 2024-03-21T10:44:44.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.188+0000 7ff9e3821740 1 --2- 172.21.15.31:0/2076847681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556bf024d460 0x556bf024e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.188+0000 7ff9e3821740 1 --2- 172.21.15.31:0/2076847681 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bf0244dd0 0x556bf024cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.188+0000 7ff9e3821740 1 --2- 172.21.15.31:0/2076847681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bf02440f0 0x556bf024c640 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.189+0000 7ff9e3821740 1 -- 172.21.15.31:0/2076847681 >> 172.21.15.31:0/2076847681 conn(0x556bf020b200 msgr2=0x556bf023e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:44.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.191+0000 7ff9e3821740 1 -- 172.21.15.31:0/2076847681 shutdown_connections 2024-03-21T10:44:44.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.192+0000 7ff9e3821740 1 -- 172.21.15.31:0/2076847681 wait complete. 2024-03-21T10:44:44.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.194+0000 7ff9e3821740 1 -- start start 2024-03-21T10:44:44.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.195+0000 7ff9e3821740 1 Processor -- start 2024-03-21T10:44:44.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.198+0000 7ff9e3821740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bf02440f0 0x556bf0246d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:44.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.198+0000 7ff9e3821740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556bf0244dd0 0x556bf02472b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:44.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.200+0000 7ff9d9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bf02440f0 0x556bf0246d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:44.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.200+0000 7ff9d9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bf02440f0 0x556bf0246d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37494/0 (socket says 172.21.15.31:37494) 2024-03-21T10:44:44.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.202+0000 7ff9d9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556bf0244dd0 0x556bf02472b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:44.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.203+0000 7ff9e3821740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bf024d460 0x556bf02b76e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:44.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.203+0000 7ff9e3821740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9bc00e320 con 0x556bf0244dd0 2024-03-21T10:44:44.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.203+0000 7ff9e3821740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9bc00f620 con 0x556bf02440f0 2024-03-21T10:44:44.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.203+0000 7ff9e3821740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9bc00e960 con 0x556bf024d460 2024-03-21T10:44:44.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.203+0000 7ff9e3821740 10 client.0 _ll_register_callbacks cb 0x7ff9c8005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:44.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.203+0000 7ff9d9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556bf0244dd0 0x556bf02472b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41866/0 (socket says 172.21.15.31:41866) 2024-03-21T10:44:44.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.203+0000 7ff9d9400640 1 -- 172.21.15.31:0/2902463068 learned_addr learned my addr 172.21.15.31:0/2902463068 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:44.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.203+0000 7ff9da800640 1 --2- 172.21.15.31:0/2902463068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bf024d460 0x556bf02b76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:44.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.204+0000 7ff9d9e00640 1 -- 172.21.15.31:0/2902463068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bf024d460 msgr2=0x556bf02b76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:44.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.204+0000 7ff9d9e00640 1 --2- 172.21.15.31:0/2902463068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bf024d460 0x556bf02b76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.204+0000 7ff9d9e00640 1 -- 172.21.15.31:0/2902463068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556bf0244dd0 msgr2=0x556bf02472b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:44.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.204+0000 7ff9d9e00640 1 --2- 172.21.15.31:0/2902463068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556bf0244dd0 0x556bf02472b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.204+0000 7ff9d9e00640 1 -- 172.21.15.31:0/2902463068 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556bf02b7c70 con 0x556bf02440f0 2024-03-21T10:44:44.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.204+0000 7ff9d9e00640 1 --2- 172.21.15.31:0/2902463068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bf02440f0 0x556bf0246d70 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc035bd0 tx=0x7ff9cc005f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77e12ac65ffbb417 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:44.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.204+0000 7ff9bb400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:44:44.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.205+0000 7ff9bb400640 1 -- 172.21.15.31:0/2902463068 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc059040 con 0x556bf02440f0 2024-03-21T10:44:44.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.205+0000 7ff9bb400640 1 -- 172.21.15.31:0/2902463068 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7ff9cc034640 con 0x556bf02440f0 2024-03-21T10:44:44.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.205+0000 7ff9e3821740 1 -- 172.21.15.31:0/2902463068 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff9cc005520 con 0x556bf02440f0 2024-03-21T10:44:44.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.205+0000 7ff9bb400640 1 -- 172.21.15.31:0/2902463068 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9cc039040 con 0x556bf02440f0 2024-03-21T10:44:44.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.206+0000 7ff9e3821740 10 client.27188 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:44.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.206+0000 7ff9e3821740 1 -- 172.21.15.31:0/2902463068 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x556bf02b8f00 con 0x556bf02440f0 2024-03-21T10:44:44.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.206+0000 7ff9bb400640 1 -- 172.21.15.31:0/2902463068 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9cc005520 con 0x556bf02440f0 2024-03-21T10:44:44.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.207+0000 7ff9bb400640 1 -- 172.21.15.31:0/2902463068 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7ff9cc0046c0 con 0x556bf02440f0 2024-03-21T10:44:44.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.207+0000 7ff9e3821740 10 client.27188 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:44.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.207+0000 7ff9e3821740 10 client.27188 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:44:44.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.207+0000 7ff9e3821740 20 client.27188 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:44.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.207+0000 7ff9e3821740 1 --2- 172.21.15.31:0/2902463068 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556bf02b9580 0x556bf02bb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:44.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.207+0000 7ff9e3821740 4 client.27188 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:44:44.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.207+0000 7ff9e3821740 1 -- 172.21.15.31:0/2902463068 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556bf01afae0 con 0x556bf02b9580 2024-03-21T10:44:44.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.207+0000 7ff9d9400640 1 --2- 172.21.15.31:0/2902463068 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556bf02b9580 0x556bf02bb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:44.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.212+0000 7ff9d9400640 1 --2- 172.21.15.31:0/2902463068 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556bf02b9580 0x556bf02bb9a0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e01d62d9d22da53 server_cookie=cea8cd7a1df437bc in_seq=0 out_seq=0 2024-03-21T10:44:44.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.212+0000 7ff9bb400640 10 client.27188 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:44.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.212+0000 7ff9bb400640 1 -- 172.21.15.31:0/2902463068 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x556bf01afae0 con 0x556bf02b9580 2024-03-21T10:44:44.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.212+0000 7ff9bb400640 10 client.27188 handle_command_reply: tid=0 2024-03-21T10:44:44.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.212+0000 7ff9e3821740 1 client.27188 shutdown 2024-03-21T10:44:44.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.218+0000 7ff9e3821740 1 -- 172.21.15.31:0/2902463068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bf02440f0 msgr2=0x556bf0246d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:44.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.218+0000 7ff9e3821740 1 --2- 172.21.15.31:0/2902463068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bf02440f0 0x556bf0246d70 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc035bd0 tx=0x7ff9cc005f30 comp rx=0 tx=0).stop 2024-03-21T10:44:44.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.221+0000 7ff9e3821740 1 -- 172.21.15.31:0/2902463068 shutdown_connections 2024-03-21T10:44:44.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.221+0000 7ff9e3821740 1 --2- 172.21.15.31:0/2902463068 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556bf02b9580 0x556bf02bb9a0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.221+0000 7ff9e3821740 1 --2- 172.21.15.31:0/2902463068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bf024d460 0x556bf02b76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.221+0000 7ff9e3821740 1 --2- 172.21.15.31:0/2902463068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556bf0244dd0 0x556bf02472b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.221+0000 7ff9e3821740 1 --2- 172.21.15.31:0/2902463068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bf02440f0 0x556bf0246d70 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.221+0000 7ff9e3821740 1 -- 172.21.15.31:0/2902463068 >> 172.21.15.31:0/2902463068 conn(0x556bf020b200 msgr2=0x556bf0249d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:44.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.222+0000 7ff9bb400640 0 client.27188 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:44.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.224+0000 7ff9e3821740 1 -- 172.21.15.31:0/2902463068 shutdown_connections 2024-03-21T10:44:44.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.225+0000 7ff9e3821740 1 -- 172.21.15.31:0/2902463068 wait complete. 2024-03-21T10:44:44.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.225+0000 7ff9e3821740 20 client.27188 trim_cache size 0 max 16384 2024-03-21T10:44:44.224 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:44:44.224 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-21T10:44:44.224 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-03-21T10:44:44.224 INFO:teuthology.orchestra.run.smithi031.stdout: "f5b66cc1-dd46-405b-bf57-467912e3e749": { 2024-03-21T10:44:44.224 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-03-21T10:44:44.224 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "f5b66cc1-dd46-405b-bf57-467912e3e749", 2024-03-21T10:44:44.224 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-03-21T10:44:44.224 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:44:44.224 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:44:44.224 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:44:44.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.229+0000 7ff9db200640 1 -- 172.21.15.31:0/2737884736 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff9c00a3290 msgr2=0x7ff9c00a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:44.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.229+0000 7ff9db200640 1 --2- 172.21.15.31:0/2737884736 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff9c00a3290 0x7ff9c00a5750 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc000c00 tx=0x7ff9cc005a90 comp rx=0 tx=0).stop 2024-03-21T10:44:44.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.229+0000 7ff9db200640 1 -- 172.21.15.31:0/2737884736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9dc10fc90 msgr2=0x7ff9dc1a6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:44.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.229+0000 7ff9db200640 1 --2- 172.21.15.31:0/2737884736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9dc10fc90 0x7ff9dc1a6a00 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff9d400cad0 tx=0x7ff9d400ced0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.229+0000 7ff9db200640 1 -- 172.21.15.31:0/2737884736 shutdown_connections 2024-03-21T10:44:44.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.229+0000 7ff9db200640 1 --2- 172.21.15.31:0/2737884736 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff9c00a3290 0x7ff9c00a5750 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.229+0000 7ff9db200640 1 --2- 172.21.15.31:0/2737884736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9dc10fc90 0x7ff9dc1a6a00 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.229+0000 7ff9db200640 1 --2- 172.21.15.31:0/2737884736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9dc10e310 0x7ff9dc1acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.229+0000 7ff9db200640 1 --2- 172.21.15.31:0/2737884736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9dc10d780 0x7ff9dc1ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:44.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.229+0000 7ff9db200640 1 -- 172.21.15.31:0/2737884736 >> 172.21.15.31:0/2737884736 conn(0x7ff9dc06c580 msgr2=0x7ff9dc070a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:44.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.231+0000 7ff9db200640 1 -- 172.21.15.31:0/2737884736 shutdown_connections 2024-03-21T10:44:44.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:44:44.232+0000 7ff9db200640 1 -- 172.21.15.31:0/2737884736 wait complete. 2024-03-21T10:44:44.272 INFO:tasks.cephfs.filesystem:scrub status for tag:f5b66cc1-dd46-405b-bf57-467912e3e749 - {'path': '/', 'tag': 'f5b66cc1-dd46-405b-bf57-467912e3e749', 'options': 'recursive,force'} 2024-03-21T10:44:44.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:44 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:44:44.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:44 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:44:44.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:44 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:44:44.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:44 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:44:44.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:44 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:44:44.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:44 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:44:45.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:45 smithi070 ceph-mon[36330]: scrub queued for path: / 2024-03-21T10:44:45.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:45 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:44:45.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:45 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:44:45.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:45 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:44:45.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:45 smithi070 ceph-mon[36330]: pgmap v188: 129 pgs: 129 active+clean; 39 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 187 MiB/s wr, 1.34k op/s 2024-03-21T10:44:45.574 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:45 smithi031 ceph-mon[31673]: scrub queued for path: / 2024-03-21T10:44:45.574 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:45 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:44:45.575 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:45 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:44:45.575 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:45 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:44:45.575 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:45 smithi031 ceph-mon[31673]: pgmap v188: 129 pgs: 129 active+clean; 39 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 187 MiB/s wr, 1.34k op/s 2024-03-21T10:44:45.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:45 smithi040 ceph-mon[36154]: scrub queued for path: / 2024-03-21T10:44:45.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:45 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:44:45.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:45 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:44:45.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:45 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:44:45.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:45 smithi040 ceph-mon[36154]: pgmap v188: 129 pgs: 129 active+clean; 39 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 187 MiB/s wr, 1.34k op/s 2024-03-21T10:44:47.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:47 smithi031 ceph-mon[31673]: pgmap v189: 129 pgs: 129 active+clean; 40 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 242 MiB/s wr, 1.62k op/s 2024-03-21T10:44:47.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:47 smithi070 ceph-mon[36330]: pgmap v189: 129 pgs: 129 active+clean; 40 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 242 MiB/s wr, 1.62k op/s 2024-03-21T10:44:47.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:47 smithi040 ceph-mon[36154]: pgmap v189: 129 pgs: 129 active+clean; 40 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 242 MiB/s wr, 1.62k op/s 2024-03-21T10:44:49.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:49 smithi031 ceph-mon[31673]: pgmap v190: 129 pgs: 129 active+clean; 40 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 180 MiB/s wr, 1.11k op/s 2024-03-21T10:44:49.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:49 smithi070 ceph-mon[36330]: pgmap v190: 129 pgs: 129 active+clean; 40 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 180 MiB/s wr, 1.11k op/s 2024-03-21T10:44:49.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:49 smithi040 ceph-mon[36154]: pgmap v190: 129 pgs: 129 active+clean; 40 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 180 MiB/s wr, 1.11k op/s 2024-03-21T10:44:51.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.440+0000 7fe942000640 1 -- 172.21.15.31:0/2311935023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe93c10fe10 msgr2=0x7fe93c10d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:51.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.440+0000 7fe942000640 1 --2- 172.21.15.31:0/2311935023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe93c10fe10 0x7fe93c10d9b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe92c009900 tx=0x7fe92c02bc30 comp rx=0 tx=0).stop 2024-03-21T10:44:51.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.441+0000 7fe942000640 1 -- 172.21.15.31:0/2311935023 shutdown_connections 2024-03-21T10:44:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.441+0000 7fe942000640 1 --2- 172.21.15.31:0/2311935023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe93c11ae80 0x7fe93c11d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.441+0000 7fe942000640 1 --2- 172.21.15.31:0/2311935023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe93c110280 0x7fe93c10def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.441+0000 7fe942000640 1 --2- 172.21.15.31:0/2311935023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe93c10fe10 0x7fe93c10d9b0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.441+0000 7fe942000640 1 -- 172.21.15.31:0/2311935023 >> 172.21.15.31:0/2311935023 conn(0x7fe93c06c750 msgr2=0x7fe93c06cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.442+0000 7fe942000640 1 -- 172.21.15.31:0/2311935023 shutdown_connections 2024-03-21T10:44:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.442+0000 7fe942000640 1 -- 172.21.15.31:0/2311935023 wait complete. 2024-03-21T10:44:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.444+0000 7fe942000640 1 Processor -- start 2024-03-21T10:44:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.444+0000 7fe942000640 1 -- start start 2024-03-21T10:44:51.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.447+0000 7fe942000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe93c10fe10 0x7fe93c1ac910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:51.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.447+0000 7fe942000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe93c110280 0x7fe93c1ace50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:51.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.447+0000 7fe942000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe93c11ae80 0x7fe93c1a6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:51.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.447+0000 7fe942000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c1208a0 con 0x7fe93c110280 2024-03-21T10:44:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.447+0000 7fe942000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c120720 con 0x7fe93c11ae80 2024-03-21T10:44:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.447+0000 7fe942000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c120a20 con 0x7fe93c10fe10 2024-03-21T10:44:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.447+0000 7fe93b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe93c10fe10 0x7fe93c1ac910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.447+0000 7fe93b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe93c10fe10 0x7fe93c1ac910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:52746/0 (socket says 172.21.15.31:52746) 2024-03-21T10:44:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.447+0000 7fe93b400640 1 -- 172.21.15.31:0/3085259513 learned_addr learned my addr 172.21.15.31:0/3085259513 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:51.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.448+0000 7fe93b400640 1 -- 172.21.15.31:0/3085259513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe93c11ae80 msgr2=0x7fe93c1a6ac0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-21T10:44:51.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.448+0000 7fe93b400640 1 --2- 172.21.15.31:0/3085259513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe93c11ae80 0x7fe93c1a6ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.448+0000 7fe93b400640 1 -- 172.21.15.31:0/3085259513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe93c110280 msgr2=0x7fe93c1ace50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T10:44:51.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.449+0000 7fe93b400640 1 --2- 172.21.15.31:0/3085259513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe93c110280 0x7fe93c1ace50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.449+0000 7fe93b400640 1 -- 172.21.15.31:0/3085259513 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93c1a7240 con 0x7fe93c10fe10 2024-03-21T10:44:51.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.450+0000 7fe93b400640 1 --2- 172.21.15.31:0/3085259513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe93c10fe10 0x7fe93c1ac910 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe92c000c00 tx=0x7fe92c0045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.450+0000 7fe933e00640 1 -- 172.21.15.31:0/3085259513 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c043020 con 0x7fe93c10fe10 2024-03-21T10:44:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.451+0000 7fe933e00640 1 -- 172.21.15.31:0/3085259513 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe92c0054e0 con 0x7fe93c10fe10 2024-03-21T10:44:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.451+0000 7fe933e00640 1 -- 172.21.15.31:0/3085259513 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c003db0 con 0x7fe93c10fe10 2024-03-21T10:44:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.451+0000 7fe942000640 1 -- 172.21.15.31:0/3085259513 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe93c1a74d0 con 0x7fe93c10fe10 2024-03-21T10:44:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.452+0000 7fe942000640 1 -- 172.21.15.31:0/3085259513 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe93c1b37d0 con 0x7fe93c10fe10 2024-03-21T10:44:51.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.458+0000 7fe933e00640 1 -- 172.21.15.31:0/3085259513 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fe92c046020 con 0x7fe93c10fe10 2024-03-21T10:44:51.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.463+0000 7fe933e00640 1 --2- 172.21.15.31:0/3085259513 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe9200a3210 0x7fe9200a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:51.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.464+0000 7fe933e00640 1 -- 172.21.15.31:0/3085259513 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fe92c0f1830 con 0x7fe93c10fe10 2024-03-21T10:44:51.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.467+0000 7fe93aa00640 1 --2- 172.21.15.31:0/3085259513 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe9200a3210 0x7fe9200a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.518+0000 7fe93aa00640 1 --2- 172.21.15.31:0/3085259513 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe9200a3210 0x7fe9200a56d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe93c073190 tx=0x7fe928005bf0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.522+0000 7fe944d94740 1 Processor -- start 2024-03-21T10:44:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.522+0000 7fe944d94740 1 -- start start 2024-03-21T10:44:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.523+0000 7fe944d94740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612c8264fd0 0x5612c82653b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:51.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.525+0000 7fe944d94740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612c82658f0 0x5612c826e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:51.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.525+0000 7fe944d94740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612c826e6c0 0x5612c8270ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:51.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.526+0000 7fe93b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612c8264fd0 0x5612c82653b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:51.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.526+0000 7fe93be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612c826e6c0 0x5612c8270ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:51.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.526+0000 7fe93be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612c826e6c0 0x5612c8270ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:48486/0 (socket says 172.21.15.31:48486) 2024-03-21T10:44:51.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.526+0000 7fe93be00640 1 -- 172.21.15.31:0/1754741930 learned_addr learned my addr 172.21.15.31:0/1754741930 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:51.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.526+0000 7fe944d94740 1 -- 172.21.15.31:0/1754741930 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612c82273b0 con 0x5612c82658f0 2024-03-21T10:44:51.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.526+0000 7fe944d94740 1 -- 172.21.15.31:0/1754741930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612c81c9830 con 0x5612c826e6c0 2024-03-21T10:44:51.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.526+0000 7fe944d94740 1 -- 172.21.15.31:0/1754741930 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612c81725d0 con 0x5612c8264fd0 2024-03-21T10:44:51.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.527+0000 7fe93be00640 1 -- 172.21.15.31:0/1754741930 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612c8264fd0 msgr2=0x5612c82653b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:51.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.527+0000 7fe93be00640 1 --2- 172.21.15.31:0/1754741930 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612c8264fd0 0x5612c82653b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.527+0000 7fe93be00640 1 -- 172.21.15.31:0/1754741930 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612c82658f0 msgr2=0x5612c826e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:51.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.528+0000 7fe93be00640 1 --2- 172.21.15.31:0/1754741930 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612c82658f0 0x5612c826e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.528+0000 7fe93be00640 1 -- 172.21.15.31:0/1754741930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5612c818a2c0 con 0x5612c826e6c0 2024-03-21T10:44:51.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.528+0000 7fe93be00640 1 --2- 172.21.15.31:0/1754741930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612c826e6c0 0x5612c8270ab0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fe934007c40 tx=0x7fe93400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adf8c6131747ba22 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:51.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.529+0000 7fe927e00640 1 -- 172.21.15.31:0/1754741930 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe934003b70 con 0x5612c826e6c0 2024-03-21T10:44:51.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.535+0000 7fe927e00640 1 -- 172.21.15.31:0/1754741930 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe934003d10 con 0x5612c826e6c0 2024-03-21T10:44:51.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.535+0000 7fe927e00640 1 -- 172.21.15.31:0/1754741930 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe934003b70 con 0x5612c826e6c0 2024-03-21T10:44:51.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.537+0000 7fe944d94740 1 -- 172.21.15.31:0/1754741930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612c826e6c0 msgr2=0x5612c8270ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:51.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.537+0000 7fe944d94740 1 --2- 172.21.15.31:0/1754741930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612c826e6c0 0x5612c8270ab0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fe934007c40 tx=0x7fe93400c6a0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.546+0000 7fe944d94740 1 -- 172.21.15.31:0/1754741930 shutdown_connections 2024-03-21T10:44:51.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.546+0000 7fe944d94740 1 --2- 172.21.15.31:0/1754741930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612c826e6c0 0x5612c8270ab0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.546+0000 7fe944d94740 1 --2- 172.21.15.31:0/1754741930 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612c82658f0 0x5612c826e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.546+0000 7fe944d94740 1 --2- 172.21.15.31:0/1754741930 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612c8264fd0 0x5612c82653b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.546+0000 7fe944d94740 1 -- 172.21.15.31:0/1754741930 >> 172.21.15.31:0/1754741930 conn(0x5612c825f8f0 msgr2=0x5612c825fcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:51.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.554+0000 7fe944d94740 1 -- 172.21.15.31:0/1754741930 shutdown_connections 2024-03-21T10:44:51.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.555+0000 7fe944d94740 1 -- 172.21.15.31:0/1754741930 wait complete. 2024-03-21T10:44:51.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.555+0000 7fe944d94740 1 -- start start 2024-03-21T10:44:51.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.556+0000 7fe944d94740 1 Processor -- start 2024-03-21T10:44:51.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:51 smithi031 ceph-mon[31673]: pgmap v191: 129 pgs: 129 active+clean; 41 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 229 MiB/s wr, 1.37k op/s 2024-03-21T10:44:51.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:51 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:44:51.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.568+0000 7fe944d94740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612c8264fd0 0x5612c826b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:51.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.571+0000 7fe944d94740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612c82658f0 0x5612c826bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:51.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.571+0000 7fe93b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612c8264fd0 0x5612c826b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:51.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.571+0000 7fe93b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612c8264fd0 0x5612c826b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:52772/0 (socket says 172.21.15.31:52772) 2024-03-21T10:44:51.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:51 smithi070 ceph-mon[36330]: pgmap v191: 129 pgs: 129 active+clean; 41 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 229 MiB/s wr, 1.37k op/s 2024-03-21T10:44:51.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:51 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:44:51.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.571+0000 7fe93b400640 1 -- 172.21.15.31:0/1832131063 learned_addr learned my addr 172.21.15.31:0/1832131063 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:51.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.573+0000 7fe944d94740 1 --2- 172.21.15.31:0/1832131063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612c826e6c0 0x5612c826c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:51.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.577+0000 7fe944d94740 1 -- 172.21.15.31:0/1832131063 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612c8185a20 con 0x5612c82658f0 2024-03-21T10:44:51.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.577+0000 7fe944d94740 1 -- 172.21.15.31:0/1832131063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612c81c58d0 con 0x5612c826e6c0 2024-03-21T10:44:51.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.577+0000 7fe944d94740 1 -- 172.21.15.31:0/1832131063 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612c82712f0 con 0x5612c8264fd0 2024-03-21T10:44:51.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.577+0000 7fe944d94740 10 client.0 _ll_register_callbacks cb 0x5612c81c21c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:51.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.578+0000 7fe93aa00640 1 --2- 172.21.15.31:0/1832131063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612c82658f0 0x5612c826bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:51.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.579+0000 7fe93be00640 1 --2- 172.21.15.31:0/1832131063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612c826e6c0 0x5612c826c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:51.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.580+0000 7fe93be00640 1 -- 172.21.15.31:0/1832131063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612c8264fd0 msgr2=0x5612c826b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:51.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.580+0000 7fe93be00640 1 --2- 172.21.15.31:0/1832131063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612c8264fd0 0x5612c826b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.580+0000 7fe93be00640 1 -- 172.21.15.31:0/1832131063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612c82658f0 msgr2=0x5612c826bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:51.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.580+0000 7fe93be00640 1 --2- 172.21.15.31:0/1832131063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612c82658f0 0x5612c826bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.580+0000 7fe93be00640 1 -- 172.21.15.31:0/1832131063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5612c7ede210 con 0x5612c826e6c0 2024-03-21T10:44:51.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.587+0000 7fe93b400640 1 --2- 172.21.15.31:0/1832131063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612c8264fd0 0x5612c826b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:44:51.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.587+0000 7fe93be00640 1 --2- 172.21.15.31:0/1832131063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612c826e6c0 0x5612c826c2e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fe934004950 tx=0x7fe9340040f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51ac144faa23e7da server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:51.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.587+0000 7fe924c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:44:51.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.589+0000 7fe924c00640 1 -- 172.21.15.31:0/1832131063 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9340043f0 con 0x5612c826e6c0 2024-03-21T10:44:51.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.590+0000 7fe924c00640 1 -- 172.21.15.31:0/1832131063 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fe934004a40 con 0x5612c826e6c0 2024-03-21T10:44:51.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.593+0000 7fe944d94740 1 -- 172.21.15.31:0/1832131063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe934003d10 con 0x5612c826e6c0 2024-03-21T10:44:51.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.602+0000 7fe924c00640 1 -- 172.21.15.31:0/1832131063 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe934004590 con 0x5612c826e6c0 2024-03-21T10:44:51.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.604+0000 7fe944d94740 10 client.27194 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:51.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.604+0000 7fe944d94740 1 -- 172.21.15.31:0/1832131063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5612c7faa620 con 0x5612c826e6c0 2024-03-21T10:44:51.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.604+0000 7fe924c00640 1 -- 172.21.15.31:0/1832131063 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe934004c40 con 0x5612c826e6c0 2024-03-21T10:44:51.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.605+0000 7fe924c00640 1 -- 172.21.15.31:0/1832131063 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fe93402d410 con 0x5612c826e6c0 2024-03-21T10:44:51.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.605+0000 7fe944d94740 10 client.27194 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:51.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.605+0000 7fe944d94740 10 client.27194 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:44:51.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.605+0000 7fe944d94740 20 client.27194 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:51.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.605+0000 7fe944d94740 1 --2- 172.21.15.31:0/1832131063 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5612c827e780 0x5612c8280b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:51.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.605+0000 7fe944d94740 4 client.27194 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:44:51.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.605+0000 7fe944d94740 1 -- 172.21.15.31:0/1832131063 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe934003d10 con 0x5612c827e780 2024-03-21T10:44:51.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.605+0000 7fe93b400640 1 --2- 172.21.15.31:0/1832131063 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5612c827e780 0x5612c8280b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:51.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.606+0000 7fe93b400640 1 --2- 172.21.15.31:0/1832131063 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5612c827e780 0x5612c8280b70 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7456bd3c0421496 server_cookie=b917ab289d703fe7 in_seq=0 out_seq=0 2024-03-21T10:44:51.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.606+0000 7fe924c00640 10 client.27194 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:51.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.608+0000 7fe924c00640 1 -- 172.21.15.31:0/1832131063 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7fe934003d10 con 0x5612c827e780 2024-03-21T10:44:51.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.608+0000 7fe924c00640 10 client.27194 handle_command_reply: tid=0 2024-03-21T10:44:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.608+0000 7fe944d94740 1 client.27194 shutdown 2024-03-21T10:44:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.609+0000 7fe944d94740 1 -- 172.21.15.31:0/1832131063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612c826e6c0 msgr2=0x5612c826c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.609+0000 7fe944d94740 1 --2- 172.21.15.31:0/1832131063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612c826e6c0 0x5612c826c2e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fe934004950 tx=0x7fe9340040f0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.609+0000 7fe944d94740 1 -- 172.21.15.31:0/1832131063 shutdown_connections 2024-03-21T10:44:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.609+0000 7fe944d94740 1 --2- 172.21.15.31:0/1832131063 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5612c827e780 0x5612c8280b70 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.609+0000 7fe944d94740 1 --2- 172.21.15.31:0/1832131063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612c826e6c0 0x5612c826c2e0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.609+0000 7fe944d94740 1 --2- 172.21.15.31:0/1832131063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612c82658f0 0x5612c826bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.609+0000 7fe944d94740 1 --2- 172.21.15.31:0/1832131063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612c8264fd0 0x5612c826b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.609+0000 7fe944d94740 1 -- 172.21.15.31:0/1832131063 >> 172.21.15.31:0/1832131063 conn(0x5612c825f8f0 msgr2=0x5612c8261640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.609+0000 7fe924c00640 0 client.27194 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.610+0000 7fe944d94740 1 -- 172.21.15.31:0/1832131063 shutdown_connections 2024-03-21T10:44:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.610+0000 7fe944d94740 1 -- 172.21.15.31:0/1832131063 wait complete. 2024-03-21T10:44:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.612+0000 7fe944d94740 20 client.27194 trim_cache size 0 max 16384 2024-03-21T10:44:51.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.662+0000 7fe944d94740 1 Processor -- start 2024-03-21T10:44:51.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.662+0000 7fe944d94740 1 -- start start 2024-03-21T10:44:51.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.663+0000 7fe944d94740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612c8264fd0 0x5612c826df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:51.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.663+0000 7fe944d94740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612c82658f0 0x5612c8261450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.666+0000 7fe93aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612c82658f0 0x5612c8261450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.666+0000 7fe93aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612c82658f0 0x5612c8261450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:52782/0 (socket says 172.21.15.31:52782) 2024-03-21T10:44:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.666+0000 7fe93aa00640 1 -- 172.21.15.31:0/4069111630 learned_addr learned my addr 172.21.15.31:0/4069111630 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:51.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.666+0000 7fe944d94740 1 --2- 172.21.15.31:0/4069111630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612c8261990 0x5612c8268ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:51.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.666+0000 7fe93b400640 1 --2- 172.21.15.31:0/4069111630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612c8264fd0 0x5612c826df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:51.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.667+0000 7fe944d94740 1 -- 172.21.15.31:0/4069111630 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c1bfc80 con 0x5612c8264fd0 2024-03-21T10:44:51.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.667+0000 7fe944d94740 1 -- 172.21.15.31:0/4069111630 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c1bd780 con 0x5612c8261990 2024-03-21T10:44:51.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.667+0000 7fe944d94740 1 -- 172.21.15.31:0/4069111630 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c1bb820 con 0x5612c82658f0 2024-03-21T10:44:51.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.667+0000 7fe93be00640 1 --2- 172.21.15.31:0/4069111630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612c8261990 0x5612c8268ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:51.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.670+0000 7fe93aa00640 1 -- 172.21.15.31:0/4069111630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612c8261990 msgr2=0x5612c8268ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:51.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.670+0000 7fe93aa00640 1 --2- 172.21.15.31:0/4069111630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612c8261990 0x5612c8268ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.670+0000 7fe93aa00640 1 -- 172.21.15.31:0/4069111630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612c8264fd0 msgr2=0x5612c826df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:51.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.670+0000 7fe93aa00640 1 --2- 172.21.15.31:0/4069111630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612c8264fd0 0x5612c826df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.670+0000 7fe93aa00640 1 -- 172.21.15.31:0/4069111630 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5612c82655f0 con 0x5612c82658f0 2024-03-21T10:44:51.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.671+0000 7fe93aa00640 1 --2- 172.21.15.31:0/4069111630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612c82658f0 0x5612c8261450 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe92800cc40 tx=0x7fe9280046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b39bbc889c89118 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:51.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.672+0000 7fe927400640 1 -- 172.21.15.31:0/4069111630 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe928021030 con 0x5612c82658f0 2024-03-21T10:44:51.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.672+0000 7fe927400640 1 -- 172.21.15.31:0/4069111630 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe928016040 con 0x5612c82658f0 2024-03-21T10:44:51.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.673+0000 7fe927400640 1 -- 172.21.15.31:0/4069111630 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92800e610 con 0x5612c82658f0 2024-03-21T10:44:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.674+0000 7fe944d94740 1 -- 172.21.15.31:0/4069111630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612c82658f0 msgr2=0x5612c8261450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.674+0000 7fe944d94740 1 --2- 172.21.15.31:0/4069111630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612c82658f0 0x5612c8261450 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe92800cc40 tx=0x7fe9280046c0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.677+0000 7fe944d94740 1 -- 172.21.15.31:0/4069111630 shutdown_connections 2024-03-21T10:44:51.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.677+0000 7fe944d94740 1 --2- 172.21.15.31:0/4069111630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612c8261990 0x5612c8268ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.677+0000 7fe944d94740 1 --2- 172.21.15.31:0/4069111630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612c82658f0 0x5612c8261450 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.677+0000 7fe944d94740 1 --2- 172.21.15.31:0/4069111630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612c8264fd0 0x5612c826df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.677+0000 7fe944d94740 1 -- 172.21.15.31:0/4069111630 >> 172.21.15.31:0/4069111630 conn(0x5612c825f8f0 msgr2=0x5612c8260d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:51.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.678+0000 7fe944d94740 1 -- 172.21.15.31:0/4069111630 shutdown_connections 2024-03-21T10:44:51.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.678+0000 7fe944d94740 1 -- 172.21.15.31:0/4069111630 wait complete. 2024-03-21T10:44:51.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.679+0000 7fe944d94740 1 -- start start 2024-03-21T10:44:51.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.679+0000 7fe944d94740 1 Processor -- start 2024-03-21T10:44:51.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.680+0000 7fe944d94740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612c8261990 0x5612c8278050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:51.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.681+0000 7fe944d94740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612c8264fd0 0x5612c82743d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:51.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.681+0000 7fe944d94740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612c82658f0 0x5612c8274b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:51.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.681+0000 7fe93b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612c8261990 0x5612c8278050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:51.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.681+0000 7fe93b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612c8261990 0x5612c8278050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:52794/0 (socket says 172.21.15.31:52794) 2024-03-21T10:44:51.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.681+0000 7fe944d94740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c1bbe20 con 0x5612c82658f0 2024-03-21T10:44:51.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.681+0000 7fe944d94740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c1bd120 con 0x5612c8264fd0 2024-03-21T10:44:51.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.681+0000 7fe944d94740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c1bc460 con 0x5612c8261990 2024-03-21T10:44:51.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.681+0000 7fe944d94740 10 client.0 _ll_register_callbacks cb 0x7fe92c009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:44:51.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.682+0000 7fe93b400640 1 -- 172.21.15.31:0/309284208 learned_addr learned my addr 172.21.15.31:0/309284208 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:44:51.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.682+0000 7fe93aa00640 1 --2- 172.21.15.31:0/309284208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612c8264fd0 0x5612c82743d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:51.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.683+0000 7fe93b400640 1 -- 172.21.15.31:0/309284208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612c8264fd0 msgr2=0x5612c82743d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:51.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.683+0000 7fe93b400640 1 --2- 172.21.15.31:0/309284208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612c8264fd0 0x5612c82743d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.683+0000 7fe93b400640 1 -- 172.21.15.31:0/309284208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612c82658f0 msgr2=0x5612c8274b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:44:51.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.683+0000 7fe93b400640 1 --2- 172.21.15.31:0/309284208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612c82658f0 0x5612c8274b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:51 smithi040 ceph-mon[36154]: pgmap v191: 129 pgs: 129 active+clean; 41 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 229 MiB/s wr, 1.37k op/s 2024-03-21T10:44:51.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:51 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:44:51.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.684+0000 7fe93b400640 1 -- 172.21.15.31:0/309284208 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5612c82dbe10 con 0x5612c8261990 2024-03-21T10:44:51.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.684+0000 7fe93b400640 1 --2- 172.21.15.31:0/309284208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612c8261990 0x5612c8278050 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe92c0ba2d0 tx=0x7fe92c02cdf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8ec79c076ab7e05 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:44:51.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.684+0000 7fe924c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:44:51.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.685+0000 7fe924c00640 1 -- 172.21.15.31:0/309284208 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c0bd990 con 0x5612c8261990 2024-03-21T10:44:51.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.686+0000 7fe924c00640 1 -- 172.21.15.31:0/309284208 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7fe92c0b9f70 con 0x5612c8261990 2024-03-21T10:44:51.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.687+0000 7fe944d94740 1 -- 172.21.15.31:0/309284208 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe928016040 con 0x5612c8261990 2024-03-21T10:44:51.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.687+0000 7fe924c00640 1 -- 172.21.15.31:0/309284208 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe92c0c1ae0 con 0x5612c8261990 2024-03-21T10:44:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.696+0000 7fe944d94740 10 client.27118 fetch_fsmap learned FSMap version 26 2024-03-21T10:44:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.696+0000 7fe944d94740 1 -- 172.21.15.31:0/309284208 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5612c82dd040 con 0x5612c8261990 2024-03-21T10:44:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.696+0000 7fe924c00640 1 -- 172.21.15.31:0/309284208 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe928016040 con 0x5612c8261990 2024-03-21T10:44:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.699+0000 7fe924c00640 1 -- 172.21.15.31:0/309284208 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fe92c0bdbf0 con 0x5612c8261990 2024-03-21T10:44:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.699+0000 7fe944d94740 10 client.27118 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:44:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.699+0000 7fe944d94740 10 client.27118 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:44:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.699+0000 7fe944d94740 20 client.27118 populate_metadata read hostname 'smithi031' 2024-03-21T10:44:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.699+0000 7fe944d94740 1 --2- 172.21.15.31:0/309284208 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5612c82dd6c0 0x5612c82dfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:44:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.699+0000 7fe944d94740 4 client.27118 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:44:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.699+0000 7fe944d94740 1 -- 172.21.15.31:0/309284208 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5612c81d6d70 con 0x5612c82dd6c0 2024-03-21T10:44:51.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.699+0000 7fe93aa00640 1 --2- 172.21.15.31:0/309284208 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5612c82dd6c0 0x5612c82dfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:44:51.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.700+0000 7fe93aa00640 1 --2- 172.21.15.31:0/309284208 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5612c82dd6c0 0x5612c82dfab0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29343e15a4dfa047 server_cookie=9518af4b3ad8b2ce in_seq=0 out_seq=0 2024-03-21T10:44:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.701+0000 7fe924c00640 10 client.27118 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.962+0000 7fe924c00640 1 -- 172.21.15.31:0/309284208 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5612c81d6d70 con 0x5612c82dd6c0 2024-03-21T10:44:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.962+0000 7fe924c00640 10 client.27118 handle_command_reply: tid=0 2024-03-21T10:44:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.962+0000 7fe944d94740 1 client.27118 shutdown 2024-03-21T10:44:51.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.966+0000 7fe944d94740 1 -- 172.21.15.31:0/309284208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612c8261990 msgr2=0x5612c8278050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:51.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.966+0000 7fe944d94740 1 --2- 172.21.15.31:0/309284208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612c8261990 0x5612c8278050 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe92c0ba2d0 tx=0x7fe92c02cdf0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.967+0000 7fe944d94740 1 -- 172.21.15.31:0/309284208 shutdown_connections 2024-03-21T10:44:51.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.967+0000 7fe944d94740 1 --2- 172.21.15.31:0/309284208 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5612c82dd6c0 0x5612c82dfab0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.967+0000 7fe924c00640 0 client.27118 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:44:51.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.967+0000 7fe944d94740 1 --2- 172.21.15.31:0/309284208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612c82658f0 0x5612c8274b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.967+0000 7fe944d94740 1 --2- 172.21.15.31:0/309284208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612c8264fd0 0x5612c82743d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.967+0000 7fe944d94740 1 --2- 172.21.15.31:0/309284208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612c8261990 0x5612c8278050 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.967+0000 7fe944d94740 1 -- 172.21.15.31:0/309284208 >> 172.21.15.31:0/309284208 conn(0x5612c825f8f0 msgr2=0x5612c825e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:51.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.968+0000 7fe944d94740 1 -- 172.21.15.31:0/309284208 shutdown_connections 2024-03-21T10:44:51.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.968+0000 7fe944d94740 1 -- 172.21.15.31:0/309284208 wait complete. 2024-03-21T10:44:51.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.968+0000 7fe944d94740 20 client.27118 trim_cache size 0 max 16384 2024-03-21T10:44:51.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:44:51.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:44:51.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:44:51.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:44:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.973+0000 7fe942000640 1 -- 172.21.15.31:0/3085259513 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe9200a3210 msgr2=0x7fe9200a56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.973+0000 7fe942000640 1 --2- 172.21.15.31:0/3085259513 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe9200a3210 0x7fe9200a56d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe93c073190 tx=0x7fe928005bf0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.973+0000 7fe942000640 1 -- 172.21.15.31:0/3085259513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe93c10fe10 msgr2=0x7fe93c1ac910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:44:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.973+0000 7fe942000640 1 --2- 172.21.15.31:0/3085259513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe93c10fe10 0x7fe93c1ac910 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe92c000c00 tx=0x7fe92c0045e0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.973+0000 7fe942000640 1 -- 172.21.15.31:0/3085259513 shutdown_connections 2024-03-21T10:44:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.973+0000 7fe942000640 1 --2- 172.21.15.31:0/3085259513 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe9200a3210 0x7fe9200a56d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.973+0000 7fe942000640 1 --2- 172.21.15.31:0/3085259513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe93c11ae80 0x7fe93c1a6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.973+0000 7fe942000640 1 --2- 172.21.15.31:0/3085259513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe93c110280 0x7fe93c1ace50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.973+0000 7fe942000640 1 --2- 172.21.15.31:0/3085259513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe93c10fe10 0x7fe93c1ac910 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:44:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.973+0000 7fe942000640 1 -- 172.21.15.31:0/3085259513 >> 172.21.15.31:0/3085259513 conn(0x7fe93c06c750 msgr2=0x7fe93c119860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:44:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.977+0000 7fe942000640 1 -- 172.21.15.31:0/3085259513 shutdown_connections 2024-03-21T10:44:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:44:51.978+0000 7fe942000640 1 -- 172.21.15.31:0/3085259513 wait complete. 2024-03-21T10:44:52.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:52 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:44:52.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:52 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:44:52.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:52 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:44:53.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:53 smithi031 ceph-mon[31673]: pgmap v192: 129 pgs: 129 active+clean; 41 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 182 MiB/s wr, 1.06k op/s 2024-03-21T10:44:53.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:53 smithi070 ceph-mon[36330]: pgmap v192: 129 pgs: 129 active+clean; 41 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 182 MiB/s wr, 1.06k op/s 2024-03-21T10:44:53.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:53 smithi040 ceph-mon[36154]: pgmap v192: 129 pgs: 129 active+clean; 41 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 182 MiB/s wr, 1.06k op/s 2024-03-21T10:44:55.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:55 smithi031 ceph-mon[31673]: pgmap v193: 129 pgs: 129 active+clean; 41 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 185 MiB/s wr, 1.07k op/s 2024-03-21T10:44:55.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:55 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:44:55.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:55 smithi070 ceph-mon[36330]: pgmap v193: 129 pgs: 129 active+clean; 41 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 185 MiB/s wr, 1.07k op/s 2024-03-21T10:44:55.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:55 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:44:55.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:55 smithi040 ceph-mon[36154]: pgmap v193: 129 pgs: 129 active+clean; 41 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 185 MiB/s wr, 1.07k op/s 2024-03-21T10:44:55.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:55 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:44:56.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:56 smithi040 ceph-mon[36154]: pgmap v194: 129 pgs: 129 active+clean; 42 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 252 MiB/s wr, 1.45k op/s 2024-03-21T10:44:56.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:56 smithi031 ceph-mon[31673]: pgmap v194: 129 pgs: 129 active+clean; 42 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 252 MiB/s wr, 1.45k op/s 2024-03-21T10:44:56.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:56 smithi070 ceph-mon[36330]: pgmap v194: 129 pgs: 129 active+clean; 42 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 252 MiB/s wr, 1.45k op/s 2024-03-21T10:44:59.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:58 smithi040 ceph-mon[36154]: pgmap v195: 129 pgs: 129 active+clean; 42 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 198 MiB/s wr, 1.14k op/s 2024-03-21T10:44:59.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:58 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:44:59.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:58 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:44:59.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:44:58 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:44:59.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:58 smithi031 ceph-mon[31673]: pgmap v195: 129 pgs: 129 active+clean; 42 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 198 MiB/s wr, 1.14k op/s 2024-03-21T10:44:59.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:58 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:44:59.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:58 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:44:59.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:44:58 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:44:59.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:58 smithi070 ceph-mon[36330]: pgmap v195: 129 pgs: 129 active+clean; 42 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 198 MiB/s wr, 1.14k op/s 2024-03-21T10:44:59.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:58 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:44:59.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:58 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:44:59.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:44:58 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:45:01.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:00 smithi040 ceph-mon[36154]: pgmap v196: 129 pgs: 129 active+clean; 42 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 240 MiB/s wr, 1.36k op/s 2024-03-21T10:45:01.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:00 smithi031 ceph-mon[31673]: pgmap v196: 129 pgs: 129 active+clean; 42 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 240 MiB/s wr, 1.36k op/s 2024-03-21T10:45:01.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:00 smithi070 ceph-mon[36330]: pgmap v196: 129 pgs: 129 active+clean; 42 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 240 MiB/s wr, 1.36k op/s 2024-03-21T10:45:03.132 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:02 smithi031 ceph-mon[31673]: pgmap v197: 129 pgs: 129 active+clean; 42 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 190 MiB/s wr, 1.08k op/s 2024-03-21T10:45:03.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:02 smithi040 ceph-mon[36154]: pgmap v197: 129 pgs: 129 active+clean; 42 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 190 MiB/s wr, 1.08k op/s 2024-03-21T10:45:03.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:02 smithi070 ceph-mon[36330]: pgmap v197: 129 pgs: 129 active+clean; 42 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 190 MiB/s wr, 1.08k op/s 2024-03-21T10:45:05.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:04 smithi040 ceph-mon[36154]: pgmap v198: 129 pgs: 129 active+clean; 42 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 192 MiB/s wr, 1.09k op/s 2024-03-21T10:45:05.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:04 smithi031 ceph-mon[31673]: pgmap v198: 129 pgs: 129 active+clean; 42 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 192 MiB/s wr, 1.09k op/s 2024-03-21T10:45:05.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:04 smithi070 ceph-mon[36330]: pgmap v198: 129 pgs: 129 active+clean; 42 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 192 MiB/s wr, 1.09k op/s 2024-03-21T10:45:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.607+0000 7f30f6c00640 1 -- 172.21.15.31:0/994285589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30f010cfc0 msgr2=0x7f30f0114b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.608+0000 7f30f6c00640 1 --2- 172.21.15.31:0/994285589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30f010cfc0 0x7f30f0114b90 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f30e8009ca0 tx=0x7f30e802c100 comp rx=0 tx=0).stop 2024-03-21T10:45:05.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.608+0000 7f30f6c00640 1 -- 172.21.15.31:0/994285589 shutdown_connections 2024-03-21T10:45:05.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.609+0000 7f30f6c00640 1 --2- 172.21.15.31:0/994285589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30f010cfc0 0x7f30f0114b90 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.609+0000 7f30f6c00640 1 --2- 172.21.15.31:0/994285589 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30f01088f0 0x7f30f010c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.609+0000 7f30f6c00640 1 --2- 172.21.15.31:0/994285589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30f0107f40 0x7f30f0108320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.609+0000 7f30f6c00640 1 -- 172.21.15.31:0/994285589 >> 172.21.15.31:0/994285589 conn(0x7f30f00774e0 msgr2=0x7f30f01064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:05.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.609+0000 7f30f6c00640 1 -- 172.21.15.31:0/994285589 shutdown_connections 2024-03-21T10:45:05.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.610+0000 7f30f6c00640 1 -- 172.21.15.31:0/994285589 wait complete. 2024-03-21T10:45:05.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.611+0000 7f30f6c00640 1 Processor -- start 2024-03-21T10:45:05.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.611+0000 7f30f6c00640 1 -- start start 2024-03-21T10:45:05.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.612+0000 7f30f6c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30f0107f40 0x7f30f0081d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:05.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.612+0000 7f30f6c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30f010cfc0 0x7f30f007ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:05.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.613+0000 7f30f6c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30f007b380 0x7f30f007b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:05.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.613+0000 7f30f6c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f0118070 con 0x7f30f007b380 2024-03-21T10:45:05.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.613+0000 7f30f6c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f0117ef0 con 0x7f30f010cfc0 2024-03-21T10:45:05.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.613+0000 7f30f6c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f01181f0 con 0x7f30f0107f40 2024-03-21T10:45:05.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.613+0000 7f30f4c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30f007b380 0x7f30f007b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:05.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.614+0000 7f30f4c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30f007b380 0x7f30f007b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37788/0 (socket says 172.21.15.31:37788) 2024-03-21T10:45:05.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.614+0000 7f30f4c00640 1 -- 172.21.15.31:0/112091725 learned_addr learned my addr 172.21.15.31:0/112091725 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:05.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.614+0000 7f30ef400640 1 --2- 172.21.15.31:0/112091725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30f010cfc0 0x7f30f007ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:05.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.614+0000 7f30f4c00640 1 -- 172.21.15.31:0/112091725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30f0107f40 msgr2=0x7f30f0081d20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-21T10:45:05.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.614+0000 7f30f4c00640 1 --2- 172.21.15.31:0/112091725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30f0107f40 0x7f30f0081d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.614+0000 7f30f4c00640 1 -- 172.21.15.31:0/112091725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30f010cfc0 msgr2=0x7f30f007ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:05.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.614+0000 7f30f4c00640 1 --2- 172.21.15.31:0/112091725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30f010cfc0 0x7f30f007ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.614+0000 7f30f4c00640 1 -- 172.21.15.31:0/112091725 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f007f840 con 0x7f30f007b380 2024-03-21T10:45:05.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.615+0000 7f30f4c00640 1 --2- 172.21.15.31:0/112091725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30f007b380 0x7f30f007b830 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f30e8037040 tx=0x7f30e8032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:05.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.615+0000 7f30ecc00640 1 -- 172.21.15.31:0/112091725 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e8039040 con 0x7f30f007b380 2024-03-21T10:45:05.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.615+0000 7f30ecc00640 1 -- 172.21.15.31:0/112091725 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30e8044070 con 0x7f30f007b380 2024-03-21T10:45:05.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.615+0000 7f30ecc00640 1 -- 172.21.15.31:0/112091725 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e8039040 con 0x7f30f007b380 2024-03-21T10:45:05.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.615+0000 7f30f6c00640 1 -- 172.21.15.31:0/112091725 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f007fad0 con 0x7f30f007b380 2024-03-21T10:45:05.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.616+0000 7f30f6c00640 1 -- 172.21.15.31:0/112091725 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f007ff70 con 0x7f30f007b380 2024-03-21T10:45:05.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.621+0000 7f30ecc00640 1 -- 172.21.15.31:0/112091725 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f30e8003970 con 0x7f30f007b380 2024-03-21T10:45:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.622+0000 7f30ecc00640 1 --2- 172.21.15.31:0/112091725 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f30d80a3380 0x7f30d80a5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.623+0000 7f30ecc00640 1 -- 172.21.15.31:0/112091725 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f30e80f1aa0 con 0x7f30f007b380 2024-03-21T10:45:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.623+0000 7f30efe00640 1 --2- 172.21.15.31:0/112091725 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f30d80a3380 0x7f30d80a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:05.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.625+0000 7f30efe00640 1 --2- 172.21.15.31:0/112091725 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f30d80a3380 0x7f30d80a5840 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f30e400b990 tx=0x7f30e400b370 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:05.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.691+0000 7f30f9865740 1 Processor -- start 2024-03-21T10:45:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.692+0000 7f30f9865740 1 -- start start 2024-03-21T10:45:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.692+0000 7f30f9865740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55767f26efd0 0x55767f26f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.692+0000 7f30f9865740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f26f8f0 0x55767f278180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.692+0000 7f30f9865740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55767f2786c0 0x55767f27aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.692+0000 7f30f9865740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55767f2313b0 con 0x55767f26efd0 2024-03-21T10:45:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.692+0000 7f30f9865740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55767f1d3830 con 0x55767f2786c0 2024-03-21T10:45:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.692+0000 7f30f9865740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55767f17c5d0 con 0x55767f26f8f0 2024-03-21T10:45:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.693+0000 7f30f4c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55767f2786c0 0x55767f27aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.693+0000 7f30ef400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f26f8f0 0x55767f278180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.693+0000 7f30ef400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f26f8f0 0x55767f278180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:46074/0 (socket says 172.21.15.31:46074) 2024-03-21T10:45:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.693+0000 7f30ef400640 1 -- 172.21.15.31:0/2333044817 learned_addr learned my addr 172.21.15.31:0/2333044817 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:05.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.694+0000 7f30ef400640 1 -- 172.21.15.31:0/2333044817 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55767f2786c0 msgr2=0x55767f27aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:05.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.694+0000 7f30ef400640 1 --2- 172.21.15.31:0/2333044817 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55767f2786c0 0x55767f27aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.694+0000 7f30ef400640 1 -- 172.21.15.31:0/2333044817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55767f26efd0 msgr2=0x55767f26f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:05.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.694+0000 7f30ef400640 1 --2- 172.21.15.31:0/2333044817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55767f26efd0 0x55767f26f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.694+0000 7f30ef400640 1 -- 172.21.15.31:0/2333044817 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55767f1942c0 con 0x55767f26f8f0 2024-03-21T10:45:05.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.695+0000 7f30ef400640 1 --2- 172.21.15.31:0/2333044817 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f26f8f0 0x55767f278180 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f30dc00dba0 tx=0x7f30dc00f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df1542e6dd650117 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:05.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.697+0000 7f30e1200640 1 -- 172.21.15.31:0/2333044817 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30dc00de90 con 0x55767f26f8f0 2024-03-21T10:45:05.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.697+0000 7f30e1200640 1 -- 172.21.15.31:0/2333044817 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30dc005500 con 0x55767f26f8f0 2024-03-21T10:45:05.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.697+0000 7f30e1200640 1 -- 172.21.15.31:0/2333044817 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30dc0116a0 con 0x55767f26f8f0 2024-03-21T10:45:05.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.699+0000 7f30f9865740 1 -- 172.21.15.31:0/2333044817 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f26f8f0 msgr2=0x55767f278180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:05.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.699+0000 7f30f9865740 1 --2- 172.21.15.31:0/2333044817 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f26f8f0 0x55767f278180 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f30dc00dba0 tx=0x7f30dc00f730 comp rx=0 tx=0).stop 2024-03-21T10:45:05.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.699+0000 7f30f9865740 1 -- 172.21.15.31:0/2333044817 shutdown_connections 2024-03-21T10:45:05.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.699+0000 7f30f9865740 1 --2- 172.21.15.31:0/2333044817 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55767f2786c0 0x55767f27aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.699+0000 7f30f9865740 1 --2- 172.21.15.31:0/2333044817 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f26f8f0 0x55767f278180 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.699+0000 7f30f9865740 1 --2- 172.21.15.31:0/2333044817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55767f26efd0 0x55767f26f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.699+0000 7f30f9865740 1 -- 172.21.15.31:0/2333044817 >> 172.21.15.31:0/2333044817 conn(0x55767f2698f0 msgr2=0x55767f269cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:05.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.701+0000 7f30f9865740 1 -- 172.21.15.31:0/2333044817 shutdown_connections 2024-03-21T10:45:05.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.701+0000 7f30f9865740 1 -- 172.21.15.31:0/2333044817 wait complete. 2024-03-21T10:45:05.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.702+0000 7f30f9865740 1 -- start start 2024-03-21T10:45:05.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.702+0000 7f30f9865740 1 Processor -- start 2024-03-21T10:45:05.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.703+0000 7f30f9865740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55767f26efd0 0x55767f275740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:05.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.703+0000 7f30f9865740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f26f8f0 0x55767f275c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:05.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.703+0000 7f30f9865740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55767f2786c0 0x55767f2761c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:05.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.703+0000 7f30f9865740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55767f18fa20 con 0x55767f26efd0 2024-03-21T10:45:05.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.703+0000 7f30f9865740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55767f1cf8d0 con 0x55767f2786c0 2024-03-21T10:45:05.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.703+0000 7f30f9865740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55767f28c300 con 0x55767f26f8f0 2024-03-21T10:45:05.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.703+0000 7f30f9865740 10 client.0 _ll_register_callbacks cb 0x55767f1cc1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:05.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.704+0000 7f30f4c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55767f2786c0 0x55767f2761c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:05.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.704+0000 7f30efe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55767f26efd0 0x55767f275740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:05.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.704+0000 7f30ef400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f26f8f0 0x55767f275c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:05.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.704+0000 7f30ef400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f26f8f0 0x55767f275c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:46090/0 (socket says 172.21.15.31:46090) 2024-03-21T10:45:05.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.704+0000 7f30ef400640 1 -- 172.21.15.31:0/1191965483 learned_addr learned my addr 172.21.15.31:0/1191965483 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:05.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.705+0000 7f30f4c00640 1 -- 172.21.15.31:0/1191965483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f26f8f0 msgr2=0x55767f275c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:05.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.705+0000 7f30f4c00640 1 --2- 172.21.15.31:0/1191965483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f26f8f0 0x55767f275c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.705+0000 7f30f4c00640 1 -- 172.21.15.31:0/1191965483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55767f26efd0 msgr2=0x55767f275740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:05.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.705+0000 7f30f4c00640 1 --2- 172.21.15.31:0/1191965483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55767f26efd0 0x55767f275740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.705+0000 7f30f4c00640 1 -- 172.21.15.31:0/1191965483 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55767eee8210 con 0x55767f2786c0 2024-03-21T10:45:05.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.705+0000 7f30efe00640 1 --2- 172.21.15.31:0/1191965483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55767f26efd0 0x55767f275740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:45:05.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.706+0000 7f30f4c00640 1 --2- 172.21.15.31:0/1191965483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55767f2786c0 0x55767f2761c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f30e80eef30 tx=0x7f30e8035f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef0b91b804ed0e86 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:05.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.706+0000 7f30d5600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:45:05.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.706+0000 7f30d5600640 1 -- 172.21.15.31:0/1191965483 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e80ef2d0 con 0x55767f2786c0 2024-03-21T10:45:05.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.706+0000 7f30d5600640 1 -- 172.21.15.31:0/1191965483 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f30e80bddc0 con 0x55767f2786c0 2024-03-21T10:45:05.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.706+0000 7f30f9865740 1 -- 172.21.15.31:0/1191965483 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f30dc005500 con 0x55767f2786c0 2024-03-21T10:45:05.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.707+0000 7f30d5600640 1 -- 172.21.15.31:0/1191965483 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30e80c1cb0 con 0x55767f2786c0 2024-03-21T10:45:05.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.707+0000 7f30f9865740 10 client.27212 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:05.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.707+0000 7f30f9865740 1 -- 172.21.15.31:0/1191965483 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55767efb4620 con 0x55767f2786c0 2024-03-21T10:45:05.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.707+0000 7f30d5600640 1 -- 172.21.15.31:0/1191965483 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30dc005500 con 0x55767f2786c0 2024-03-21T10:45:05.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.708+0000 7f30d5600640 1 -- 172.21.15.31:0/1191965483 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f30e803c030 con 0x55767f2786c0 2024-03-21T10:45:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.708+0000 7f30f9865740 10 client.27212 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.708+0000 7f30f9865740 10 client.27212 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:45:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.708+0000 7f30f9865740 20 client.27212 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.708+0000 7f30f9865740 1 --2- 172.21.15.31:0/1191965483 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55767f2996c0 0x55767f29bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.708+0000 7f30f9865740 4 client.27212 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:45:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.708+0000 7f30f9865740 1 -- 172.21.15.31:0/1191965483 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55767f1e0d70 con 0x55767f2996c0 2024-03-21T10:45:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.708+0000 7f30efe00640 1 --2- 172.21.15.31:0/1191965483 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55767f2996c0 0x55767f29bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.709+0000 7f30d5600640 10 client.27212 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.712+0000 7f30efe00640 1 --2- 172.21.15.31:0/1191965483 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55767f2996c0 0x55767f29bab0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77f109db03b9ad94 server_cookie=19d724130fa682a6 in_seq=0 out_seq=0 2024-03-21T10:45:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.713+0000 7f30d5600640 1 -- 172.21.15.31:0/1191965483 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55767f1e0d70 con 0x55767f2996c0 2024-03-21T10:45:05.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.713+0000 7f30d5600640 10 client.27212 handle_command_reply: tid=0 2024-03-21T10:45:05.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.713+0000 7f30f9865740 1 client.27212 shutdown 2024-03-21T10:45:05.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.716+0000 7f30f9865740 1 -- 172.21.15.31:0/1191965483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55767f2786c0 msgr2=0x55767f2761c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:05.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.716+0000 7f30f9865740 1 --2- 172.21.15.31:0/1191965483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55767f2786c0 0x55767f2761c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f30e80eef30 tx=0x7f30e8035f20 comp rx=0 tx=0).stop 2024-03-21T10:45:05.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.717+0000 7f30f9865740 1 -- 172.21.15.31:0/1191965483 shutdown_connections 2024-03-21T10:45:05.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.717+0000 7f30f9865740 1 --2- 172.21.15.31:0/1191965483 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55767f2996c0 0x55767f29bab0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.717+0000 7f30f9865740 1 --2- 172.21.15.31:0/1191965483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55767f2786c0 0x55767f2761c0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.717+0000 7f30f9865740 1 --2- 172.21.15.31:0/1191965483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f26f8f0 0x55767f275c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.717+0000 7f30f9865740 1 --2- 172.21.15.31:0/1191965483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55767f26efd0 0x55767f275740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.717+0000 7f30f9865740 1 -- 172.21.15.31:0/1191965483 >> 172.21.15.31:0/1191965483 conn(0x55767f2698f0 msgr2=0x55767f26b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:05.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.717+0000 7f30d5600640 0 client.27212 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:05.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.718+0000 7f30f9865740 1 -- 172.21.15.31:0/1191965483 shutdown_connections 2024-03-21T10:45:05.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.718+0000 7f30f9865740 1 -- 172.21.15.31:0/1191965483 wait complete. 2024-03-21T10:45:05.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.719+0000 7f30f9865740 20 client.27212 trim_cache size 0 max 16384 2024-03-21T10:45:05.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.751+0000 7f30f9865740 1 Processor -- start 2024-03-21T10:45:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.751+0000 7f30f9865740 1 -- start start 2024-03-21T10:45:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.752+0000 7f30f9865740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55767f26efd0 0x55767f277e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.752+0000 7f30f9865740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55767f26f8f0 0x55767f275fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.752+0000 7f30f9865740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f2786c0 0x55767f270a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.752+0000 7f30f9865740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f01b7450 con 0x55767f26efd0 2024-03-21T10:45:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.752+0000 7f30f9865740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f01b4f50 con 0x55767f26f8f0 2024-03-21T10:45:05.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.752+0000 7f30f9865740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f0109400 con 0x55767f2786c0 2024-03-21T10:45:05.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.752+0000 7f30f4c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f2786c0 0x55767f270a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:05.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.752+0000 7f30f4c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f2786c0 0x55767f270a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:46100/0 (socket says 172.21.15.31:46100) 2024-03-21T10:45:05.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.752+0000 7f30f4c00640 1 -- 172.21.15.31:0/631234505 learned_addr learned my addr 172.21.15.31:0/631234505 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:05.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.752+0000 7f30efe00640 1 --2- 172.21.15.31:0/631234505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55767f26efd0 0x55767f277e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:05.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.753+0000 7f30f4c00640 1 -- 172.21.15.31:0/631234505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55767f26f8f0 msgr2=0x55767f275fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:45:05.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.753+0000 7f30f4c00640 1 --2- 172.21.15.31:0/631234505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55767f26f8f0 0x55767f275fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.753+0000 7f30f4c00640 1 -- 172.21.15.31:0/631234505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55767f26efd0 msgr2=0x55767f277e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:05.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.753+0000 7f30f4c00640 1 --2- 172.21.15.31:0/631234505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55767f26efd0 0x55767f277e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.753+0000 7f30f4c00640 1 -- 172.21.15.31:0/631234505 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55767f26f720 con 0x55767f2786c0 2024-03-21T10:45:05.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.753+0000 7f30f4c00640 1 --2- 172.21.15.31:0/631234505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f2786c0 0x55767f270a40 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f30e80b9fc0 tx=0x7f30e80bdce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff6546c5a6819f58 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:05.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.755+0000 7f30d7e00640 1 -- 172.21.15.31:0/631234505 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e80bd4c0 con 0x55767f2786c0 2024-03-21T10:45:05.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.755+0000 7f30d7e00640 1 -- 172.21.15.31:0/631234505 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30e80c6570 con 0x55767f2786c0 2024-03-21T10:45:05.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.756+0000 7f30d7e00640 1 -- 172.21.15.31:0/631234505 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e80bca20 con 0x55767f2786c0 2024-03-21T10:45:05.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.757+0000 7f30f9865740 1 -- 172.21.15.31:0/631234505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f2786c0 msgr2=0x55767f270a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:05.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.757+0000 7f30f9865740 1 --2- 172.21.15.31:0/631234505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f2786c0 0x55767f270a40 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f30e80b9fc0 tx=0x7f30e80bdce0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.757+0000 7f30f9865740 1 -- 172.21.15.31:0/631234505 shutdown_connections 2024-03-21T10:45:05.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.757+0000 7f30f9865740 1 --2- 172.21.15.31:0/631234505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f2786c0 0x55767f270a40 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.757+0000 7f30f9865740 1 --2- 172.21.15.31:0/631234505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55767f26f8f0 0x55767f275fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.757+0000 7f30f9865740 1 --2- 172.21.15.31:0/631234505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55767f26efd0 0x55767f277e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.757+0000 7f30f9865740 1 -- 172.21.15.31:0/631234505 >> 172.21.15.31:0/631234505 conn(0x55767f2698f0 msgr2=0x55767f274010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:05.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.757+0000 7f30f9865740 1 -- 172.21.15.31:0/631234505 shutdown_connections 2024-03-21T10:45:05.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.757+0000 7f30f9865740 1 -- 172.21.15.31:0/631234505 wait complete. 2024-03-21T10:45:05.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.758+0000 7f30f9865740 1 -- start start 2024-03-21T10:45:05.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.758+0000 7f30f9865740 1 Processor -- start 2024-03-21T10:45:05.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.759+0000 7f30f9865740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55767f26efd0 0x55767f272cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.760+0000 7f30f9865740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55767f26f8f0 0x55767f273200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.760+0000 7f30f9865740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f2786c0 0x55767f2719b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.760+0000 7f30f4c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f2786c0 0x55767f2719b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.760+0000 7f30f4c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f2786c0 0x55767f2719b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:46110/0 (socket says 172.21.15.31:46110) 2024-03-21T10:45:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.760+0000 7f30f4c00640 1 -- 172.21.15.31:0/2610829677 learned_addr learned my addr 172.21.15.31:0/2610829677 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.760+0000 7f30f9865740 1 -- 172.21.15.31:0/2610829677 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f00729a0 con 0x55767f26efd0 2024-03-21T10:45:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.760+0000 7f30f9865740 1 -- 172.21.15.31:0/2610829677 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f01b48f0 con 0x55767f26f8f0 2024-03-21T10:45:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.760+0000 7f30f9865740 1 -- 172.21.15.31:0/2610829677 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f0072fa0 con 0x55767f2786c0 2024-03-21T10:45:05.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.761+0000 7f30f9865740 10 client.0 _ll_register_callbacks cb 0x7f30e80bd930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:05.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.761+0000 7f30f4c00640 1 -- 172.21.15.31:0/2610829677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55767f26f8f0 msgr2=0x55767f273200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:45:05.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.761+0000 7f30f4c00640 1 --2- 172.21.15.31:0/2610829677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55767f26f8f0 0x55767f273200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.761+0000 7f30f4c00640 1 -- 172.21.15.31:0/2610829677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55767f26efd0 msgr2=0x55767f272cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:45:05.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.761+0000 7f30efe00640 1 --2- 172.21.15.31:0/2610829677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55767f26efd0 0x55767f272cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:05.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.762+0000 7f30f4c00640 1 --2- 172.21.15.31:0/2610829677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55767f26efd0 0x55767f272cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.762+0000 7f30f4c00640 1 -- 172.21.15.31:0/2610829677 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55767f2721b0 con 0x55767f2786c0 2024-03-21T10:45:05.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.762+0000 7f30f9865740 1 -- 172.21.15.31:0/2610829677 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f30e80c6570 con 0x55767f2786c0 2024-03-21T10:45:05.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.762+0000 7f30efe00640 1 --2- 172.21.15.31:0/2610829677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55767f26efd0 0x55767f272cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:45:05.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.763+0000 7f30f4c00640 1 --2- 172.21.15.31:0/2610829677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f2786c0 0x55767f2719b0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f30e80c1e70 tx=0x7f30e80c6310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9463f01708c18f67 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:05.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.763+0000 7f30d5600640 10 client.27133 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:45:05.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.763+0000 7f30d5600640 1 -- 172.21.15.31:0/2610829677 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e80bd4c0 con 0x55767f2786c0 2024-03-21T10:45:05.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.763+0000 7f30d5600640 1 -- 172.21.15.31:0/2610829677 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9358+0+0 (secure 0 0 0) 0x7f30e80b9ab0 con 0x55767f2786c0 2024-03-21T10:45:05.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.763+0000 7f30d5600640 1 -- 172.21.15.31:0/2610829677 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30e80bd2c0 con 0x55767f2786c0 2024-03-21T10:45:05.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.763+0000 7f30f9865740 10 client.27133 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:05.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.763+0000 7f30f9865740 1 -- 172.21.15.31:0/2610829677 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({fsmap=26}) v3 -- 0x55767f2721b0 con 0x55767f2786c0 2024-03-21T10:45:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.764+0000 7f30d5600640 1 -- 172.21.15.31:0/2610829677 <== mon.1 v2:172.21.15.70:3300/0 4 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f30e80a2020 con 0x55767f2786c0 2024-03-21T10:45:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.764+0000 7f30f9865740 10 client.27133 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.764+0000 7f30f9865740 10 client.27133 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:45:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.764+0000 7f30f9865740 20 client.27133 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.764+0000 7f30f9865740 1 --2- 172.21.15.31:0/2610829677 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55767f2f4980 0x55767f2f6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.764+0000 7f30f9865740 4 client.27133 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:45:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.764+0000 7f30f9865740 1 -- 172.21.15.31:0/2610829677 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55767f1db0c0 con 0x55767f2f4980 2024-03-21T10:45:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.765+0000 7f30efe00640 1 --2- 172.21.15.31:0/2610829677 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55767f2f4980 0x55767f2f6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.765+0000 7f30efe00640 1 --2- 172.21.15.31:0/2610829677 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55767f2f4980 0x55767f2f6d70 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5e992b4db9a733b server_cookie=d8a16f893ac1cdf in_seq=0 out_seq=0 2024-03-21T10:45:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.765+0000 7f30d5600640 10 client.27133 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.799+0000 7f30d5600640 1 -- 172.21.15.31:0/2610829677 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55767f1db0c0 con 0x55767f2f4980 2024-03-21T10:45:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.799+0000 7f30d5600640 10 client.27133 handle_command_reply: tid=0 2024-03-21T10:45:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.799+0000 7f30f9865740 1 client.27133 shutdown 2024-03-21T10:45:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.799+0000 7f30f9865740 1 -- 172.21.15.31:0/2610829677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f2786c0 msgr2=0x55767f2719b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.799+0000 7f30f9865740 1 --2- 172.21.15.31:0/2610829677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f2786c0 0x55767f2719b0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f30e80c1e70 tx=0x7f30e80c6310 comp rx=0 tx=0).stop 2024-03-21T10:45:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.799+0000 7f30f9865740 1 -- 172.21.15.31:0/2610829677 shutdown_connections 2024-03-21T10:45:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.799+0000 7f30f9865740 1 --2- 172.21.15.31:0/2610829677 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55767f2f4980 0x55767f2f6d70 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.799+0000 7f30f9865740 1 --2- 172.21.15.31:0/2610829677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55767f2786c0 0x55767f2719b0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.799+0000 7f30f9865740 1 --2- 172.21.15.31:0/2610829677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55767f26f8f0 0x55767f273200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.799+0000 7f30f9865740 1 --2- 172.21.15.31:0/2610829677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55767f26efd0 0x55767f272cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.799+0000 7f30f9865740 1 -- 172.21.15.31:0/2610829677 >> 172.21.15.31:0/2610829677 conn(0x55767f2698f0 msgr2=0x55767f270a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.800+0000 7f30d5600640 0 client.27133 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.800+0000 7f30f9865740 1 -- 172.21.15.31:0/2610829677 shutdown_connections 2024-03-21T10:45:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.800+0000 7f30f9865740 1 -- 172.21.15.31:0/2610829677 wait complete. 2024-03-21T10:45:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:45:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:45:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:45:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:45:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.800+0000 7f30f9865740 20 client.27133 trim_cache size 0 max 16384 2024-03-21T10:45:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.801+0000 7f30f6c00640 1 -- 172.21.15.31:0/112091725 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f30d80a3380 msgr2=0x7f30d80a5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.801+0000 7f30f6c00640 1 --2- 172.21.15.31:0/112091725 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f30d80a3380 0x7f30d80a5840 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f30e400b990 tx=0x7f30e400b370 comp rx=0 tx=0).stop 2024-03-21T10:45:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.801+0000 7f30f6c00640 1 -- 172.21.15.31:0/112091725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30f007b380 msgr2=0x7f30f007b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.801+0000 7f30f6c00640 1 --2- 172.21.15.31:0/112091725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30f007b380 0x7f30f007b830 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f30e8037040 tx=0x7f30e8032f90 comp rx=0 tx=0).stop 2024-03-21T10:45:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.801+0000 7f30f6c00640 1 -- 172.21.15.31:0/112091725 shutdown_connections 2024-03-21T10:45:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.802+0000 7f30f6c00640 1 --2- 172.21.15.31:0/112091725 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f30d80a3380 0x7f30d80a5840 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.802+0000 7f30f6c00640 1 --2- 172.21.15.31:0/112091725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30f007b380 0x7f30f007b830 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.802+0000 7f30f6c00640 1 --2- 172.21.15.31:0/112091725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30f010cfc0 0x7f30f007ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.802+0000 7f30f6c00640 1 --2- 172.21.15.31:0/112091725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30f0107f40 0x7f30f0081d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.802+0000 7f30f6c00640 1 -- 172.21.15.31:0/112091725 >> 172.21.15.31:0/112091725 conn(0x7f30f00774e0 msgr2=0x7f30f0077b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.802+0000 7f30f6c00640 1 -- 172.21.15.31:0/112091725 shutdown_connections 2024-03-21T10:45:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:05.802+0000 7f30f6c00640 1 -- 172.21.15.31:0/112091725 wait complete. 2024-03-21T10:45:07.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:06 smithi040 ceph-mon[36154]: pgmap v199: 129 pgs: 129 active+clean; 43 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 256 MiB/s wr, 1.56k op/s 2024-03-21T10:45:07.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:06 smithi031 ceph-mon[31673]: pgmap v199: 129 pgs: 129 active+clean; 43 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 256 MiB/s wr, 1.56k op/s 2024-03-21T10:45:07.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:06 smithi070 ceph-mon[36330]: pgmap v199: 129 pgs: 129 active+clean; 43 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 256 MiB/s wr, 1.56k op/s 2024-03-21T10:45:08.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:07 smithi040 ceph-mon[36154]: osdmap e95: 12 total, 12 up, 12 in 2024-03-21T10:45:08.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:07 smithi031 ceph-mon[31673]: osdmap e95: 12 total, 12 up, 12 in 2024-03-21T10:45:08.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:07 smithi070 ceph-mon[36330]: osdmap e95: 12 total, 12 up, 12 in 2024-03-21T10:45:09.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:08 smithi031 ceph-mon[31673]: pgmap v201: 129 pgs: 129 active+clean; 43 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 213 MiB/s wr, 1.37k op/s 2024-03-21T10:45:09.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:08 smithi070 ceph-mon[36330]: pgmap v201: 129 pgs: 129 active+clean; 43 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 213 MiB/s wr, 1.37k op/s 2024-03-21T10:45:09.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:08 smithi040 ceph-mon[36154]: pgmap v201: 129 pgs: 129 active+clean; 43 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 213 MiB/s wr, 1.37k op/s 2024-03-21T10:45:11.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:10 smithi031 ceph-mon[31673]: pgmap v202: 129 pgs: 129 active+clean; 43 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 228 MiB/s wr, 1.64k op/s 2024-03-21T10:45:11.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:10 smithi070 ceph-mon[36330]: pgmap v202: 129 pgs: 129 active+clean; 43 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 228 MiB/s wr, 1.64k op/s 2024-03-21T10:45:11.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:10 smithi040 ceph-mon[36154]: pgmap v202: 129 pgs: 129 active+clean; 43 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 228 MiB/s wr, 1.64k op/s 2024-03-21T10:45:13.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:12 smithi031 ceph-mon[31673]: pgmap v203: 129 pgs: 129 active+clean; 43 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 234 MiB/s wr, 1.68k op/s 2024-03-21T10:45:13.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:12 smithi070 ceph-mon[36330]: pgmap v203: 129 pgs: 129 active+clean; 43 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 234 MiB/s wr, 1.68k op/s 2024-03-21T10:45:13.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:12 smithi040 ceph-mon[36154]: pgmap v203: 129 pgs: 129 active+clean; 43 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 234 MiB/s wr, 1.68k op/s 2024-03-21T10:45:14.273 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:45:14.274 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:45:15.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.117+0000 7fbda3200640 1 -- 172.21.15.31:0/1664401680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbda410fc90 msgr2=0x7fbda4110070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:15.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.117+0000 7fbda3200640 1 --2- 172.21.15.31:0/1664401680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbda410fc90 0x7fbda4110070 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fbd90009900 tx=0x7fbd9002bc60 comp rx=0 tx=0).stop 2024-03-21T10:45:15.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.122+0000 7fbda3200640 1 -- 172.21.15.31:0/1664401680 shutdown_connections 2024-03-21T10:45:15.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.122+0000 7fbda3200640 1 --2- 172.21.15.31:0/1664401680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbda410e310 0x7fbda411ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.122+0000 7fbda3200640 1 --2- 172.21.15.31:0/1664401680 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbda410d780 0x7fbda410dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.122+0000 7fbda3200640 1 --2- 172.21.15.31:0/1664401680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbda410fc90 0x7fbda4110070 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.122+0000 7fbda3200640 1 -- 172.21.15.31:0/1664401680 >> 172.21.15.31:0/1664401680 conn(0x7fbda406c580 msgr2=0x7fbda406c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:15.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.126+0000 7fbda3200640 1 -- 172.21.15.31:0/1664401680 shutdown_connections 2024-03-21T10:45:15.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.126+0000 7fbda3200640 1 -- 172.21.15.31:0/1664401680 wait complete. 2024-03-21T10:45:15.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.128+0000 7fbda3200640 1 Processor -- start 2024-03-21T10:45:15.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.128+0000 7fbda3200640 1 -- start start 2024-03-21T10:45:15.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.128+0000 7fbda3200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbda410d780 0x7fbda41ac790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:15.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.128+0000 7fbda3200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbda410e310 0x7fbda41accd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:15.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.128+0000 7fbda3200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbda410fc90 0x7fbda41a6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:15.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.129+0000 7fbda1e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbda410d780 0x7fbda41ac790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:15.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.129+0000 7fbda1e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbda410d780 0x7fbda41ac790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47594/0 (socket says 172.21.15.31:47594) 2024-03-21T10:45:15.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.130+0000 7fbda1e00640 1 -- 172.21.15.31:0/262640199 learned_addr learned my addr 172.21.15.31:0/262640199 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:15.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.130+0000 7fbda3200640 1 -- 172.21.15.31:0/262640199 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda41206e0 con 0x7fbda410d780 2024-03-21T10:45:15.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.130+0000 7fbda3200640 1 -- 172.21.15.31:0/262640199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda4120560 con 0x7fbda410e310 2024-03-21T10:45:15.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.130+0000 7fbda3200640 1 -- 172.21.15.31:0/262640199 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda4120860 con 0x7fbda410fc90 2024-03-21T10:45:15.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.131+0000 7fbda1400640 1 --2- 172.21.15.31:0/262640199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbda410e310 0x7fbda41accd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:15.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.131+0000 7fbda2800640 1 --2- 172.21.15.31:0/262640199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbda410fc90 0x7fbda41a6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:15.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.131+0000 7fbda1400640 1 -- 172.21.15.31:0/262640199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbda410fc90 msgr2=0x7fbda41a6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:15.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.131+0000 7fbda1400640 1 --2- 172.21.15.31:0/262640199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbda410fc90 0x7fbda41a6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.132+0000 7fbda1400640 1 -- 172.21.15.31:0/262640199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbda410d780 msgr2=0x7fbda41ac790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:15.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.132+0000 7fbda1400640 1 --2- 172.21.15.31:0/262640199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbda410d780 0x7fbda41ac790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.132+0000 7fbda1400640 1 -- 172.21.15.31:0/262640199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda41a70c0 con 0x7fbda410e310 2024-03-21T10:45:15.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.132+0000 7fbda1400640 1 --2- 172.21.15.31:0/262640199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbda410e310 0x7fbda41accd0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fbd9400d6d0 tx=0x7fbd9400db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:15.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.136+0000 7fbd9aa00640 1 -- 172.21.15.31:0/262640199 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd94017070 con 0x7fbda410e310 2024-03-21T10:45:15.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.136+0000 7fbd9aa00640 1 -- 172.21.15.31:0/262640199 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd94009820 con 0x7fbda410e310 2024-03-21T10:45:15.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.136+0000 7fbd9aa00640 1 -- 172.21.15.31:0/262640199 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd94004d00 con 0x7fbda410e310 2024-03-21T10:45:15.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.137+0000 7fbda3200640 1 -- 172.21.15.31:0/262640199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda41a73b0 con 0x7fbda410e310 2024-03-21T10:45:15.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.137+0000 7fbda3200640 1 -- 172.21.15.31:0/262640199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbda41b3650 con 0x7fbda410e310 2024-03-21T10:45:15.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.144+0000 7fbd9aa00640 1 -- 172.21.15.31:0/262640199 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fbd940099d0 con 0x7fbda410e310 2024-03-21T10:45:15.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.147+0000 7fbd9aa00640 1 --2- 172.21.15.31:0/262640199 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbd880a3290 0x7fbd880a5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:15.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.147+0000 7fbd9aa00640 1 -- 172.21.15.31:0/262640199 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9534+0+0 (secure 0 0 0) 0x7fbd940cf810 con 0x7fbda410e310 2024-03-21T10:45:15.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.149+0000 7fbdabda2740 1 Processor -- start 2024-03-21T10:45:15.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.149+0000 7fbdabda2740 1 -- start start 2024-03-21T10:45:15.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.150+0000 7fbda1e00640 1 --2- 172.21.15.31:0/262640199 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbd880a3290 0x7fbd880a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:15.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.152+0000 7fbdabda2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604af304dd0 0x5604af303b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:15.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.152+0000 7fbdabda2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604af3040f0 0x5604af304570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:15.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.152+0000 7fbdabda2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604af30d460 0x5604af30f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:15.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.152+0000 7fbdabda2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604af268250 con 0x5604af3040f0 2024-03-21T10:45:15.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.152+0000 7fbdabda2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604af210a30 con 0x5604af304dd0 2024-03-21T10:45:15.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.152+0000 7fbdabda2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604af2642f0 con 0x5604af30d460 2024-03-21T10:45:15.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.152+0000 7fbda2800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604af30d460 0x5604af30f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:15.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.152+0000 7fbda1e00640 1 --2- 172.21.15.31:0/262640199 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbd880a3290 0x7fbd880a5750 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbd90000c00 tx=0x7fbd9002c180 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:15.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.152+0000 7fbda2800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604af30d460 0x5604af30f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:45724/0 (socket says 172.21.15.31:45724) 2024-03-21T10:45:15.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.152+0000 7fbda2800640 1 -- 172.21.15.31:0/78800584 learned_addr learned my addr 172.21.15.31:0/78800584 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:15.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.153+0000 7fbda1e00640 1 --2- 172.21.15.31:0/78800584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604af304dd0 0x5604af303b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:15.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.153+0000 7fbda1400640 1 --2- 172.21.15.31:0/78800584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604af3040f0 0x5604af304570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:15.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.154+0000 7fbda1e00640 1 -- 172.21.15.31:0/78800584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604af30d460 msgr2=0x5604af30f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:15.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.154+0000 7fbda1e00640 1 --2- 172.21.15.31:0/78800584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604af30d460 0x5604af30f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.154+0000 7fbda1e00640 1 -- 172.21.15.31:0/78800584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604af3040f0 msgr2=0x5604af304570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:15.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.154+0000 7fbda1e00640 1 --2- 172.21.15.31:0/78800584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604af3040f0 0x5604af304570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.154+0000 7fbda1e00640 1 -- 172.21.15.31:0/78800584 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604af1fb690 con 0x5604af304dd0 2024-03-21T10:45:15.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.156+0000 7fbda1e00640 1 --2- 172.21.15.31:0/78800584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604af304dd0 0x5604af303b20 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fbd9002c270 tx=0x7fbd9003e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=980cbc7878228bd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:15.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.157+0000 7fbd8f400640 1 -- 172.21.15.31:0/78800584 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd90005000 con 0x5604af304dd0 2024-03-21T10:45:15.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.157+0000 7fbd8f400640 1 -- 172.21.15.31:0/78800584 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd90050070 con 0x5604af304dd0 2024-03-21T10:45:15.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.157+0000 7fbd8f400640 1 -- 172.21.15.31:0/78800584 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd900035f0 con 0x5604af304dd0 2024-03-21T10:45:15.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.158+0000 7fbdabda2740 1 -- 172.21.15.31:0/78800584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604af304dd0 msgr2=0x5604af303b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:15.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.158+0000 7fbdabda2740 1 --2- 172.21.15.31:0/78800584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604af304dd0 0x5604af303b20 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fbd9002c270 tx=0x7fbd9003e000 comp rx=0 tx=0).stop 2024-03-21T10:45:15.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.158+0000 7fbdabda2740 1 -- 172.21.15.31:0/78800584 shutdown_connections 2024-03-21T10:45:15.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.158+0000 7fbdabda2740 1 --2- 172.21.15.31:0/78800584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604af30d460 0x5604af30f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.158+0000 7fbdabda2740 1 --2- 172.21.15.31:0/78800584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604af3040f0 0x5604af304570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.158+0000 7fbdabda2740 1 --2- 172.21.15.31:0/78800584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604af304dd0 0x5604af303b20 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.158+0000 7fbdabda2740 1 -- 172.21.15.31:0/78800584 >> 172.21.15.31:0/78800584 conn(0x5604af2cb200 msgr2=0x5604af2feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:15.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.158+0000 7fbdabda2740 1 -- 172.21.15.31:0/78800584 shutdown_connections 2024-03-21T10:45:15.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.158+0000 7fbdabda2740 1 -- 172.21.15.31:0/78800584 wait complete. 2024-03-21T10:45:15.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.159+0000 7fbdabda2740 1 -- start start 2024-03-21T10:45:15.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.159+0000 7fbdabda2740 1 Processor -- start 2024-03-21T10:45:15.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.160+0000 7fbdabda2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604af3040f0 0x5604af30cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:15.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.160+0000 7fbdabda2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604af304dd0 0x5604af309fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:15.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.160+0000 7fbdabda2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604af30d460 0x5604af30a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:15.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.160+0000 7fbdabda2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604af310270 con 0x5604af3040f0 2024-03-21T10:45:15.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.160+0000 7fbdabda2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604af223e80 con 0x5604af304dd0 2024-03-21T10:45:15.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.160+0000 7fbdabda2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604af3103f0 con 0x5604af30d460 2024-03-21T10:45:15.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.161+0000 7fbdabda2740 10 client.0 _ll_register_callbacks cb 0x5604af236130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:15.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.162+0000 7fbda1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604af304dd0 0x5604af309fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:15.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.162+0000 7fbda1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604af304dd0 0x5604af309fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51014/0 (socket says 172.21.15.31:51014) 2024-03-21T10:45:15.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.162+0000 7fbda1400640 1 -- 172.21.15.31:0/180393505 learned_addr learned my addr 172.21.15.31:0/180393505 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:15.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.163+0000 7fbda2800640 1 --2- 172.21.15.31:0/180393505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604af30d460 0x5604af30a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:15.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.163+0000 7fbda1400640 1 -- 172.21.15.31:0/180393505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604af30d460 msgr2=0x5604af30a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:15.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.163+0000 7fbda1400640 1 --2- 172.21.15.31:0/180393505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604af30d460 0x5604af30a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.163+0000 7fbda1400640 1 -- 172.21.15.31:0/180393505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604af3040f0 msgr2=0x5604af30cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:15.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.163+0000 7fbda1400640 1 --2- 172.21.15.31:0/180393505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604af3040f0 0x5604af30cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.163+0000 7fbda1400640 1 -- 172.21.15.31:0/180393505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5604af0467d0 con 0x5604af304dd0 2024-03-21T10:45:15.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.163+0000 7fbda1400640 1 --2- 172.21.15.31:0/180393505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604af304dd0 0x5604af309fa0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fbd94009610 tx=0x7fbd9409b2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=686fe60cf312da21 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:15.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.163+0000 7fbd83e00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:45:15.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.164+0000 7fbd83e00640 1 -- 172.21.15.31:0/180393505 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd940b3070 con 0x5604af304dd0 2024-03-21T10:45:15.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.164+0000 7fbd83e00640 1 -- 172.21.15.31:0/180393505 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9534+0+0 (secure 0 0 0) 0x7fbd940a3db0 con 0x5604af304dd0 2024-03-21T10:45:15.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.164+0000 7fbdabda2740 1 -- 172.21.15.31:0/180393505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbd90050070 con 0x5604af304dd0 2024-03-21T10:45:15.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.164+0000 7fbd83e00640 1 -- 172.21.15.31:0/180393505 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd940ccf90 con 0x5604af304dd0 2024-03-21T10:45:15.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.164+0000 7fbdabda2740 10 client.27224 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:15.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.165+0000 7fbdabda2740 1 -- 172.21.15.31:0/180393505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5604aef7c3f0 con 0x5604af304dd0 2024-03-21T10:45:15.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.165+0000 7fbd83e00640 1 -- 172.21.15.31:0/180393505 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd90050070 con 0x5604af304dd0 2024-03-21T10:45:15.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.166+0000 7fbd83e00640 1 -- 172.21.15.31:0/180393505 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fbd9409cb60 con 0x5604af304dd0 2024-03-21T10:45:15.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.166+0000 7fbdabda2740 10 client.27224 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:15.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.166+0000 7fbdabda2740 10 client.27224 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:45:15.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.166+0000 7fbdabda2740 20 client.27224 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:15.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.166+0000 7fbdabda2740 1 --2- 172.21.15.31:0/180393505 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5604af31d110 0x5604af31f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:15.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.167+0000 7fbdabda2740 4 client.27224 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:45:15.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.167+0000 7fbdabda2740 1 -- 172.21.15.31:0/180393505 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5604af275790 con 0x5604af31d110 2024-03-21T10:45:15.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.167+0000 7fbda1e00640 1 --2- 172.21.15.31:0/180393505 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5604af31d110 0x5604af31f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:15.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.167+0000 7fbda1e00640 1 --2- 172.21.15.31:0/180393505 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5604af31d110 0x5604af31f5d0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0f6d1fb54cfdc72 server_cookie=57b6c3fe612143a3 in_seq=0 out_seq=0 2024-03-21T10:45:15.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.167+0000 7fbd83e00640 10 client.27224 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:15.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.168+0000 7fbd83e00640 1 -- 172.21.15.31:0/180393505 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5604af275790 con 0x5604af31d110 2024-03-21T10:45:15.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.168+0000 7fbd83e00640 10 client.27224 handle_command_reply: tid=0 2024-03-21T10:45:15.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.169+0000 7fbdabda2740 1 client.27224 shutdown 2024-03-21T10:45:15.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.169+0000 7fbdabda2740 1 -- 172.21.15.31:0/180393505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604af304dd0 msgr2=0x5604af309fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:15.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.169+0000 7fbdabda2740 1 --2- 172.21.15.31:0/180393505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604af304dd0 0x5604af309fa0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fbd94009610 tx=0x7fbd9409b2c0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.170+0000 7fbdabda2740 1 -- 172.21.15.31:0/180393505 shutdown_connections 2024-03-21T10:45:15.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.170+0000 7fbdabda2740 1 --2- 172.21.15.31:0/180393505 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5604af31d110 0x5604af31f5d0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.170+0000 7fbdabda2740 1 --2- 172.21.15.31:0/180393505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604af30d460 0x5604af30a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.170+0000 7fbdabda2740 1 --2- 172.21.15.31:0/180393505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604af304dd0 0x5604af309fa0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.170+0000 7fbdabda2740 1 --2- 172.21.15.31:0/180393505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604af3040f0 0x5604af30cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.170+0000 7fbdabda2740 1 -- 172.21.15.31:0/180393505 >> 172.21.15.31:0/180393505 conn(0x5604af2cb200 msgr2=0x5604af300890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:15.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.170+0000 7fbd83e00640 0 client.27224 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:15.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.170+0000 7fbdabda2740 1 -- 172.21.15.31:0/180393505 shutdown_connections 2024-03-21T10:45:15.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.170+0000 7fbdabda2740 1 -- 172.21.15.31:0/180393505 wait complete. 2024-03-21T10:45:15.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.170+0000 7fbdabda2740 20 client.27224 trim_cache size 0 max 16384 2024-03-21T10:45:15.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.203+0000 7fbdabda2740 1 Processor -- start 2024-03-21T10:45:15.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.203+0000 7fbdabda2740 1 -- start start 2024-03-21T10:45:15.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.203+0000 7fbdabda2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604af3040f0 0x5604af30c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:15.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.203+0000 7fbdabda2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604af304dd0 0x5604af30cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:15.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.203+0000 7fbdabda2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604af30d460 0x5604af318b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:15.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.203+0000 7fbdabda2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda41bfc60 con 0x5604af304dd0 2024-03-21T10:45:15.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.203+0000 7fbdabda2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda41bd760 con 0x5604af30d460 2024-03-21T10:45:15.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.203+0000 7fbdabda2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda41bb820 con 0x5604af3040f0 2024-03-21T10:45:15.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.204+0000 7fbda2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604af30d460 0x5604af318b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:15.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.204+0000 7fbda1e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604af3040f0 0x5604af30c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:15.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.204+0000 7fbda1e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604af3040f0 0x5604af30c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:45734/0 (socket says 172.21.15.31:45734) 2024-03-21T10:45:15.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.204+0000 7fbda1e00640 1 -- 172.21.15.31:0/3627936823 learned_addr learned my addr 172.21.15.31:0/3627936823 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:15.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.205+0000 7fbda1400640 1 --2- 172.21.15.31:0/3627936823 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604af304dd0 0x5604af30cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:15.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.205+0000 7fbda1e00640 1 -- 172.21.15.31:0/3627936823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604af30d460 msgr2=0x5604af318b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:15.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.205+0000 7fbda1e00640 1 --2- 172.21.15.31:0/3627936823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604af30d460 0x5604af318b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.205+0000 7fbda1e00640 1 -- 172.21.15.31:0/3627936823 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604af304dd0 msgr2=0x5604af30cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:15.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.205+0000 7fbda1e00640 1 --2- 172.21.15.31:0/3627936823 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604af304dd0 0x5604af30cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.205+0000 7fbda1e00640 1 -- 172.21.15.31:0/3627936823 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604aef3c0f0 con 0x5604af3040f0 2024-03-21T10:45:15.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.205+0000 7fbda1400640 1 --2- 172.21.15.31:0/3627936823 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604af304dd0 0x5604af30cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:45:15.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.205+0000 7fbda1e00640 1 --2- 172.21.15.31:0/3627936823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604af3040f0 0x5604af30c6d0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fbd90004be0 tx=0x7fbd9002c720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80714137951e8fd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:15.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.205+0000 7fbd8ea00640 1 -- 172.21.15.31:0/3627936823 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9005b1d0 con 0x5604af3040f0 2024-03-21T10:45:15.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.206+0000 7fbd8ea00640 1 -- 172.21.15.31:0/3627936823 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd90050070 con 0x5604af3040f0 2024-03-21T10:45:15.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.206+0000 7fbd8ea00640 1 -- 172.21.15.31:0/3627936823 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd900049b0 con 0x5604af3040f0 2024-03-21T10:45:15.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.206+0000 7fbdabda2740 1 -- 172.21.15.31:0/3627936823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604af3040f0 msgr2=0x5604af30c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:15.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.206+0000 7fbdabda2740 1 --2- 172.21.15.31:0/3627936823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604af3040f0 0x5604af30c6d0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fbd90004be0 tx=0x7fbd9002c720 comp rx=0 tx=0).stop 2024-03-21T10:45:15.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.206+0000 7fbdabda2740 1 -- 172.21.15.31:0/3627936823 shutdown_connections 2024-03-21T10:45:15.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.206+0000 7fbdabda2740 1 --2- 172.21.15.31:0/3627936823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604af30d460 0x5604af318b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.206+0000 7fbdabda2740 1 --2- 172.21.15.31:0/3627936823 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604af304dd0 0x5604af30cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.206+0000 7fbdabda2740 1 --2- 172.21.15.31:0/3627936823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604af3040f0 0x5604af30c6d0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.206+0000 7fbdabda2740 1 -- 172.21.15.31:0/3627936823 >> 172.21.15.31:0/3627936823 conn(0x5604af2cb200 msgr2=0x5604af2fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:15.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.206+0000 7fbdabda2740 1 -- 172.21.15.31:0/3627936823 shutdown_connections 2024-03-21T10:45:15.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.207+0000 7fbdabda2740 1 -- 172.21.15.31:0/3627936823 wait complete. 2024-03-21T10:45:15.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.207+0000 7fbdabda2740 1 -- start start 2024-03-21T10:45:15.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.207+0000 7fbdabda2740 1 Processor -- start 2024-03-21T10:45:15.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.208+0000 7fbdabda2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604af3040f0 0x5604af307410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:15.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.208+0000 7fbdabda2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604af304dd0 0x5604af307950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:15.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.208+0000 7fbdabda2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604af30d460 0x5604af375a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:15.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.208+0000 7fbdabda2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda41bbe20 con 0x5604af30d460 2024-03-21T10:45:15.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.208+0000 7fbdabda2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda41bd100 con 0x5604af304dd0 2024-03-21T10:45:15.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.208+0000 7fbdabda2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda41bc440 con 0x5604af3040f0 2024-03-21T10:45:15.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.208+0000 7fbdabda2740 10 client.0 _ll_register_callbacks cb 0x7fbd94004ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:15.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.208+0000 7fbda2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604af30d460 0x5604af375a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:15.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.208+0000 7fbda2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604af30d460 0x5604af375a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47636/0 (socket says 172.21.15.31:47636) 2024-03-21T10:45:15.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.208+0000 7fbda2800640 1 -- 172.21.15.31:0/2102121004 learned_addr learned my addr 172.21.15.31:0/2102121004 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:15.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.208+0000 7fbda1e00640 1 --2- 172.21.15.31:0/2102121004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604af3040f0 0x5604af307410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:15.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.209+0000 7fbda2800640 1 -- 172.21.15.31:0/2102121004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604af3040f0 msgr2=0x5604af307410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:15.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.209+0000 7fbda2800640 1 --2- 172.21.15.31:0/2102121004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604af3040f0 0x5604af307410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.209+0000 7fbda2800640 1 -- 172.21.15.31:0/2102121004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604af304dd0 msgr2=0x5604af307950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:15.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.209+0000 7fbda2800640 1 --2- 172.21.15.31:0/2102121004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604af304dd0 0x5604af307950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.209+0000 7fbda2800640 1 -- 172.21.15.31:0/2102121004 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5604af375f90 con 0x5604af30d460 2024-03-21T10:45:15.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.209+0000 7fbda2800640 1 --2- 172.21.15.31:0/2102121004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604af30d460 0x5604af375a50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c00ae40 tx=0x7fbd9c017760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1755c6302336751c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:15.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.209+0000 7fbd83e00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:45:15.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.210+0000 7fbd83e00640 1 -- 172.21.15.31:0/2102121004 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c025060 con 0x5604af30d460 2024-03-21T10:45:15.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.210+0000 7fbd83e00640 1 -- 172.21.15.31:0/2102121004 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9534+0+0 (secure 0 0 0) 0x7fbd9c020640 con 0x5604af30d460 2024-03-21T10:45:15.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.210+0000 7fbdabda2740 1 -- 172.21.15.31:0/2102121004 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbd90050070 con 0x5604af30d460 2024-03-21T10:45:15.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.211+0000 7fbd83e00640 1 -- 172.21.15.31:0/2102121004 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd9c02c050 con 0x5604af30d460 2024-03-21T10:45:15.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.211+0000 7fbdabda2740 10 client.16677 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:15.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.212+0000 7fbdabda2740 1 -- 172.21.15.31:0/2102121004 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5604af30e100 con 0x5604af30d460 2024-03-21T10:45:15.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.213+0000 7fbd83e00640 1 -- 172.21.15.31:0/2102121004 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd90050070 con 0x5604af30d460 2024-03-21T10:45:15.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.213+0000 7fbd83e00640 1 -- 172.21.15.31:0/2102121004 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fbd9c01e640 con 0x5604af30d460 2024-03-21T10:45:15.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.213+0000 7fbdabda2740 10 client.16677 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:15.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.213+0000 7fbdabda2740 10 client.16677 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:45:15.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.213+0000 7fbdabda2740 20 client.16677 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:15.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.213+0000 7fbdabda2740 1 --2- 172.21.15.31:0/2102121004 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5604af314670 0x5604af314a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:15.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.213+0000 7fbdabda2740 4 client.16677 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:45:15.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.213+0000 7fbdabda2740 1 -- 172.21.15.31:0/2102121004 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5604af26fae0 con 0x5604af314670 2024-03-21T10:45:15.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.217+0000 7fbda1e00640 1 --2- 172.21.15.31:0/2102121004 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5604af314670 0x5604af314a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:15.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.218+0000 7fbda1e00640 1 --2- 172.21.15.31:0/2102121004 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5604af314670 0x5604af314a80 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a0e2f484a4e2e06 server_cookie=f43eeb244c8772b9 in_seq=0 out_seq=0 2024-03-21T10:45:15.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.218+0000 7fbd83e00640 10 client.16677 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:15.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.218+0000 7fbd83e00640 1 -- 172.21.15.31:0/2102121004 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5604af26fae0 con 0x5604af314670 2024-03-21T10:45:15.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.218+0000 7fbd83e00640 10 client.16677 handle_command_reply: tid=0 2024-03-21T10:45:15.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.219+0000 7fbdabda2740 1 client.16677 shutdown 2024-03-21T10:45:15.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.219+0000 7fbdabda2740 1 -- 172.21.15.31:0/2102121004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604af30d460 msgr2=0x5604af375a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:15.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.219+0000 7fbdabda2740 1 --2- 172.21.15.31:0/2102121004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604af30d460 0x5604af375a50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c00ae40 tx=0x7fbd9c017760 comp rx=0 tx=0).stop 2024-03-21T10:45:15.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.221+0000 7fbdabda2740 1 -- 172.21.15.31:0/2102121004 shutdown_connections 2024-03-21T10:45:15.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.221+0000 7fbdabda2740 1 --2- 172.21.15.31:0/2102121004 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5604af314670 0x5604af314a80 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.221+0000 7fbdabda2740 1 --2- 172.21.15.31:0/2102121004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604af30d460 0x5604af375a50 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.221+0000 7fbdabda2740 1 --2- 172.21.15.31:0/2102121004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604af304dd0 0x5604af307950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.221+0000 7fbdabda2740 1 --2- 172.21.15.31:0/2102121004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604af3040f0 0x5604af307410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.221+0000 7fbdabda2740 1 -- 172.21.15.31:0/2102121004 >> 172.21.15.31:0/2102121004 conn(0x5604af2cb200 msgr2=0x5604af308800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:15.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.221+0000 7fbd83e00640 0 client.16677 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:15.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.222+0000 7fbdabda2740 1 -- 172.21.15.31:0/2102121004 shutdown_connections 2024-03-21T10:45:15.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.222+0000 7fbdabda2740 1 -- 172.21.15.31:0/2102121004 wait complete. 2024-03-21T10:45:15.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.222+0000 7fbdabda2740 20 client.16677 trim_cache size 0 max 16384 2024-03-21T10:45:15.220 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:45:15.220 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-03-21T10:45:15.220 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-03-21T10:45:15.220 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:45:15.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.223+0000 7fbda3200640 1 -- 172.21.15.31:0/262640199 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbd880a3290 msgr2=0x7fbd880a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:15.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.223+0000 7fbda3200640 1 --2- 172.21.15.31:0/262640199 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbd880a3290 0x7fbd880a5750 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbd90000c00 tx=0x7fbd9002c180 comp rx=0 tx=0).stop 2024-03-21T10:45:15.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.223+0000 7fbda3200640 1 -- 172.21.15.31:0/262640199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbda410e310 msgr2=0x7fbda41accd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:15.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.223+0000 7fbda3200640 1 --2- 172.21.15.31:0/262640199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbda410e310 0x7fbda41accd0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fbd9400d6d0 tx=0x7fbd9400db90 comp rx=0 tx=0).stop 2024-03-21T10:45:15.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.223+0000 7fbda3200640 1 -- 172.21.15.31:0/262640199 shutdown_connections 2024-03-21T10:45:15.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.223+0000 7fbda3200640 1 --2- 172.21.15.31:0/262640199 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbd880a3290 0x7fbd880a5750 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.223+0000 7fbda3200640 1 --2- 172.21.15.31:0/262640199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbda410fc90 0x7fbda41a6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.223+0000 7fbda3200640 1 --2- 172.21.15.31:0/262640199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbda410e310 0x7fbda41accd0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.223+0000 7fbda3200640 1 --2- 172.21.15.31:0/262640199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbda410d780 0x7fbda41ac790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.223+0000 7fbda3200640 1 -- 172.21.15.31:0/262640199 >> 172.21.15.31:0/262640199 conn(0x7fbda406c580 msgr2=0x7fbda4070960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:15.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.224+0000 7fbda3200640 1 -- 172.21.15.31:0/262640199 shutdown_connections 2024-03-21T10:45:15.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.224+0000 7fbda3200640 1 -- 172.21.15.31:0/262640199 wait complete. 2024-03-21T10:45:15.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:14 smithi031 ceph-mon[31673]: pgmap v204: 129 pgs: 129 active+clean; 43 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 234 MiB/s wr, 1.75k op/s 2024-03-21T10:45:15.246 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-21T10:45:15.246 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:45:15.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:14 smithi070 ceph-mon[36330]: pgmap v204: 129 pgs: 129 active+clean; 43 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 234 MiB/s wr, 1.75k op/s 2024-03-21T10:45:15.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:14 smithi040 ceph-mon[36154]: pgmap v204: 129 pgs: 129 active+clean; 43 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 234 MiB/s wr, 1.75k op/s 2024-03-21T10:45:15.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.651+0000 7fc365a00640 1 -- 172.21.15.31:0/2679101206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc360107640 msgr2=0x7fc360107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:15.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.651+0000 7fc365a00640 1 --2- 172.21.15.31:0/2679101206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc360107640 0x7fc360107a20 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fc35000fb40 tx=0x7fc35002b9a0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.651+0000 7fc365a00640 1 -- 172.21.15.31:0/2679101206 shutdown_connections 2024-03-21T10:45:15.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.651+0000 7fc365a00640 1 --2- 172.21.15.31:0/2679101206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc360101f40 0x7fc36010e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.651+0000 7fc365a00640 1 --2- 172.21.15.31:0/2679101206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc360101580 0x7fc360101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.651+0000 7fc365a00640 1 --2- 172.21.15.31:0/2679101206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc360107640 0x7fc360107a20 secure :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fc35000fb40 tx=0x7fc35002b9a0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.651+0000 7fc365a00640 1 -- 172.21.15.31:0/2679101206 >> 172.21.15.31:0/2679101206 conn(0x7fc3600fd090 msgr2=0x7fc3600ff4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:15.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.653+0000 7fc365a00640 1 -- 172.21.15.31:0/2679101206 shutdown_connections 2024-03-21T10:45:15.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.654+0000 7fc365a00640 1 -- 172.21.15.31:0/2679101206 wait complete. 2024-03-21T10:45:15.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.655+0000 7fc365a00640 1 Processor -- start 2024-03-21T10:45:15.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.656+0000 7fc365a00640 1 -- start start 2024-03-21T10:45:15.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.656+0000 7fc365a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc360101580 0x7fc360195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:15.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.656+0000 7fc365a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc360101f40 0x7fc360196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:15.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.656+0000 7fc365a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc36019a790 0x7fc36019ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:15.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.657+0000 7fc35ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc360101580 0x7fc360195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:15.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.657+0000 7fc35ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc360101580 0x7fc360195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51066/0 (socket says 172.21.15.31:51066) 2024-03-21T10:45:15.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.657+0000 7fc35f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc36019a790 0x7fc36019ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:15.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.657+0000 7fc35e000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc360101f40 0x7fc360196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:15.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.657+0000 7fc35e000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc360101f40 0x7fc360196410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:45754/0 (socket says 172.21.15.31:45754) 2024-03-21T10:45:15.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.658+0000 7fc35f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc36019a790 0x7fc36019ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47654/0 (socket says 172.21.15.31:47654) 2024-03-21T10:45:15.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.658+0000 7fc365a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36010f7e0 con 0x7fc36019a790 2024-03-21T10:45:15.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.658+0000 7fc35ea00640 1 -- 172.21.15.31:0/900010738 learned_addr learned my addr 172.21.15.31:0/900010738 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:15.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.658+0000 7fc365a00640 1 -- 172.21.15.31:0/900010738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36010f660 con 0x7fc360101580 2024-03-21T10:45:15.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.658+0000 7fc365a00640 1 -- 172.21.15.31:0/900010738 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36010f960 con 0x7fc360101f40 2024-03-21T10:45:15.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.659+0000 7fc35ea00640 1 -- 172.21.15.31:0/900010738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc360101f40 msgr2=0x7fc360196410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:15.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.659+0000 7fc35ea00640 1 --2- 172.21.15.31:0/900010738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc360101f40 0x7fc360196410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.659+0000 7fc35ea00640 1 -- 172.21.15.31:0/900010738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc36019a790 msgr2=0x7fc36019ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:15.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.659+0000 7fc35ea00640 1 --2- 172.21.15.31:0/900010738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc36019a790 0x7fc36019ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:15.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.659+0000 7fc35ea00640 1 -- 172.21.15.31:0/900010738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36019b280 con 0x7fc360101580 2024-03-21T10:45:15.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.659+0000 7fc35e000640 1 --2- 172.21.15.31:0/900010738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc360101f40 0x7fc360196410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-21T10:45:15.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.659+0000 7fc35f400640 1 --2- 172.21.15.31:0/900010738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc36019a790 0x7fc36019ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-21T10:45:15.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.662+0000 7fc35ea00640 1 --2- 172.21.15.31:0/900010738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc360101580 0x7fc360195ed0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fc35002bfb0 tx=0x7fc350004720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:15.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.663+0000 7fc357400640 1 -- 172.21.15.31:0/900010738 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc350004dc0 con 0x7fc360101580 2024-03-21T10:45:15.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.663+0000 7fc357400640 1 -- 172.21.15.31:0/900010738 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc35000e660 con 0x7fc360101580 2024-03-21T10:45:15.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.663+0000 7fc365a00640 1 -- 172.21.15.31:0/900010738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc36019b510 con 0x7fc360101580 2024-03-21T10:45:15.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.664+0000 7fc365a00640 1 -- 172.21.15.31:0/900010738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3601a2cd0 con 0x7fc360101580 2024-03-21T10:45:15.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.664+0000 7fc357400640 1 -- 172.21.15.31:0/900010738 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc350003850 con 0x7fc360101580 2024-03-21T10:45:15.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.666+0000 7fc365a00640 1 -- 172.21.15.31:0/900010738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc36004f470 con 0x7fc360101580 2024-03-21T10:45:15.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.675+0000 7fc357400640 1 -- 172.21.15.31:0/900010738 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fc350037070 con 0x7fc360101580 2024-03-21T10:45:15.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.681+0000 7fc357400640 1 --2- 172.21.15.31:0/900010738 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc3440a3100 0x7fc3440a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:15.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.682+0000 7fc357400640 1 -- 172.21.15.31:0/900010738 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9534+0+0 (secure 0 0 0) 0x7fc3500f1190 con 0x7fc360101580 2024-03-21T10:45:15.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.684+0000 7fc35e000640 1 --2- 172.21.15.31:0/900010738 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc3440a3100 0x7fc3440a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:15.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.684+0000 7fc357400640 1 -- 172.21.15.31:0/900010738 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc3500bd890 con 0x7fc360101580 2024-03-21T10:45:15.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:15.688+0000 7fc35e000640 1 --2- 172.21.15.31:0/900010738 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc3440a3100 0x7fc3440a55c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc360106de0 tx=0x7fc34c005f90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:16.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.249+0000 7fc365a00640 1 -- 172.21.15.31:0/900010738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc360109a90 con 0x7fc360101580 2024-03-21T10:45:16.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.262+0000 7fc357400640 1 -- 172.21.15.31:0/900010738 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+11384 (secure 0 0 0) 0x7fc3500bd190 con 0x7fc360101580 2024-03-21T10:45:16.261 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:45:16.262 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:42:31.182549+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24547,24596]},"id":1}]} 2024-03-21T10:45:16.262 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2024-03-21T10:45:16.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.278+0000 7fc365a00640 1 -- 172.21.15.31:0/900010738 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc3440a3100 msgr2=0x7fc3440a55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:16.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.278+0000 7fc365a00640 1 --2- 172.21.15.31:0/900010738 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc3440a3100 0x7fc3440a55c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc360106de0 tx=0x7fc34c005f90 comp rx=0 tx=0).stop 2024-03-21T10:45:16.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.278+0000 7fc365a00640 1 -- 172.21.15.31:0/900010738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc360101580 msgr2=0x7fc360195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:16.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.278+0000 7fc365a00640 1 --2- 172.21.15.31:0/900010738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc360101580 0x7fc360195ed0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fc35002bfb0 tx=0x7fc350004720 comp rx=0 tx=0).stop 2024-03-21T10:45:16.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.278+0000 7fc365a00640 1 -- 172.21.15.31:0/900010738 shutdown_connections 2024-03-21T10:45:16.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.278+0000 7fc365a00640 1 --2- 172.21.15.31:0/900010738 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc3440a3100 0x7fc3440a55c0 secure :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc360106de0 tx=0x7fc34c005f90 comp rx=0 tx=0).stop 2024-03-21T10:45:16.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.278+0000 7fc365a00640 1 --2- 172.21.15.31:0/900010738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc36019a790 0x7fc36019ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.278+0000 7fc365a00640 1 --2- 172.21.15.31:0/900010738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc360101f40 0x7fc360196410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.278+0000 7fc365a00640 1 --2- 172.21.15.31:0/900010738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc360101580 0x7fc360195ed0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.278+0000 7fc365a00640 1 -- 172.21.15.31:0/900010738 >> 172.21.15.31:0/900010738 conn(0x7fc3600fd090 msgr2=0x7fc3600fea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:16.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.281+0000 7fc365a00640 1 -- 172.21.15.31:0/900010738 shutdown_connections 2024-03-21T10:45:16.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.281+0000 7fc365a00640 1 -- 172.21.15.31:0/900010738 wait complete. 2024-03-21T10:45:16.328 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:45:16.328 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-21T10:45:16.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.780+0000 7fa07f600640 1 -- 172.21.15.31:0/3365138183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa078105ab0 msgr2=0x7fa078105f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:16.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.780+0000 7fa07f600640 1 --2- 172.21.15.31:0/3365138183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa078105ab0 0x7fa078105f60 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fa070009ca0 tx=0x7fa07002c1b0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.783+0000 7fa07f600640 1 -- 172.21.15.31:0/3365138183 shutdown_connections 2024-03-21T10:45:16.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.783+0000 7fa07f600640 1 --2- 172.21.15.31:0/3365138183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa078105ab0 0x7fa078105f60 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.783+0000 7fa07f600640 1 --2- 172.21.15.31:0/3365138183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa078100fd0 0x7fa078105490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.783+0000 7fa07f600640 1 --2- 172.21.15.31:0/3365138183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0781006b0 0x7fa078100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.783+0000 7fa07f600640 1 -- 172.21.15.31:0/3365138183 >> 172.21.15.31:0/3365138183 conn(0x7fa078077750 msgr2=0x7fa078077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:16.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.784+0000 7fa07f600640 1 -- 172.21.15.31:0/3365138183 shutdown_connections 2024-03-21T10:45:16.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.784+0000 7fa07f600640 1 -- 172.21.15.31:0/3365138183 wait complete. 2024-03-21T10:45:16.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.785+0000 7fa07f600640 1 Processor -- start 2024-03-21T10:45:16.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.790+0000 7fa07f600640 1 -- start start 2024-03-21T10:45:16.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.790+0000 7fa07f600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0781006b0 0x7fa07810c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:16.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.790+0000 7fa07f600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa078100fd0 0x7fa07810c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:16.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.790+0000 7fa07f600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa078105ab0 0x7fa078106a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:16.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.790+0000 7fa07f600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa07806bad0 con 0x7fa0781006b0 2024-03-21T10:45:16.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.790+0000 7fa07f600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa07806b950 con 0x7fa078100fd0 2024-03-21T10:45:16.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.790+0000 7fa07f600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa07806bc50 con 0x7fa078105ab0 2024-03-21T10:45:16.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.791+0000 7fa07d600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa078105ab0 0x7fa078106a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:16.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.791+0000 7fa07d600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa078105ab0 0x7fa078106a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:45762/0 (socket says 172.21.15.31:45762) 2024-03-21T10:45:16.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.792+0000 7fa077e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa078100fd0 0x7fa07810c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:16.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.792+0000 7fa077e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa078100fd0 0x7fa07810c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51078/0 (socket says 172.21.15.31:51078) 2024-03-21T10:45:16.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.792+0000 7fa077e00640 1 -- 172.21.15.31:0/3570070991 learned_addr learned my addr 172.21.15.31:0/3570070991 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:16.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.793+0000 7fa07cc00640 1 --2- 172.21.15.31:0/3570070991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0781006b0 0x7fa07810c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:16.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.794+0000 7fa077e00640 1 -- 172.21.15.31:0/3570070991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa078105ab0 msgr2=0x7fa078106a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:16.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.794+0000 7fa077e00640 1 --2- 172.21.15.31:0/3570070991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa078105ab0 0x7fa078106a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.795+0000 7fa077e00640 1 -- 172.21.15.31:0/3570070991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0781006b0 msgr2=0x7fa07810c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:16.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.795+0000 7fa077e00640 1 --2- 172.21.15.31:0/3570070991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0781006b0 0x7fa07810c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.795+0000 7fa077e00640 1 -- 172.21.15.31:0/3570070991 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa078107230 con 0x7fa078100fd0 2024-03-21T10:45:16.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.796+0000 7fa07d600640 1 --2- 172.21.15.31:0/3570070991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa078105ab0 0x7fa078106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:45:16.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.797+0000 7fa077e00640 1 --2- 172.21.15.31:0/3570070991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa078100fd0 0x7fa07810c670 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa06400c920 tx=0x7fa06400cde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:16.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.799+0000 7fa075600640 1 -- 172.21.15.31:0/3570070991 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa064007c70 con 0x7fa078100fd0 2024-03-21T10:45:16.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.799+0000 7fa075600640 1 -- 172.21.15.31:0/3570070991 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa064007e10 con 0x7fa078100fd0 2024-03-21T10:45:16.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.799+0000 7fa075600640 1 -- 172.21.15.31:0/3570070991 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa064018890 con 0x7fa078100fd0 2024-03-21T10:45:16.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.799+0000 7fa07f600640 1 -- 172.21.15.31:0/3570070991 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa078107520 con 0x7fa078100fd0 2024-03-21T10:45:16.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.799+0000 7fa07f600640 1 -- 172.21.15.31:0/3570070991 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa078112eb0 con 0x7fa078100fd0 2024-03-21T10:45:16.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.807+0000 7fa082271740 1 Processor -- start 2024-03-21T10:45:16.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.807+0000 7fa082271740 1 -- start start 2024-03-21T10:45:16.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.811+0000 7fa082271740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650dd37c6d0 0x5650dd37cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:16.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.811+0000 7fa082271740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650dd37d080 0x5650dd385630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:16.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.811+0000 7fa082271740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dd385c00 0x5650dd388050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:16.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.811+0000 7fa082271740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650dd2e02c0 con 0x5650dd37d080 2024-03-21T10:45:16.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.811+0000 7fa082271740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650dd288c90 con 0x5650dd37c6d0 2024-03-21T10:45:16.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.811+0000 7fa082271740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650dd2dc360 con 0x5650dd385c00 2024-03-21T10:45:16.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.812+0000 7fa07d600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650dd37c6d0 0x5650dd37cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:16.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.812+0000 7fa07d600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650dd37c6d0 0x5650dd37cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51086/0 (socket says 172.21.15.31:51086) 2024-03-21T10:45:16.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.812+0000 7fa07d600640 1 -- 172.21.15.31:0/4191521506 learned_addr learned my addr 172.21.15.31:0/4191521506 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:16.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.812+0000 7fa07d600640 1 -- 172.21.15.31:0/4191521506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dd385c00 msgr2=0x5650dd388050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:45:16.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.812+0000 7fa07d600640 1 --2- 172.21.15.31:0/4191521506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dd385c00 0x5650dd388050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.812+0000 7fa07d600640 1 -- 172.21.15.31:0/4191521506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650dd37d080 msgr2=0x5650dd385630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:16.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.812+0000 7fa07d600640 1 --2- 172.21.15.31:0/4191521506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650dd37d080 0x5650dd385630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.812+0000 7fa07d600640 1 -- 172.21.15.31:0/4191521506 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5650dd2ad080 con 0x5650dd37c6d0 2024-03-21T10:45:16.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.813+0000 7fa075600640 1 -- 172.21.15.31:0/3570070991 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fa064018ab0 con 0x7fa078100fd0 2024-03-21T10:45:16.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.813+0000 7fa07d600640 1 --2- 172.21.15.31:0/4191521506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650dd37c6d0 0x5650dd37cab0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa070009ca0 tx=0x7fa0700324b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e7aa37f111fef76 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:16.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.814+0000 7fa069c00640 1 -- 172.21.15.31:0/4191521506 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa070039040 con 0x5650dd37c6d0 2024-03-21T10:45:16.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.815+0000 7fa069c00640 1 -- 172.21.15.31:0/4191521506 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa070032c70 con 0x5650dd37c6d0 2024-03-21T10:45:16.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.815+0000 7fa082271740 1 -- 172.21.15.31:0/4191521506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650dd37c6d0 msgr2=0x5650dd37cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:16.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.816+0000 7fa075600640 1 --2- 172.21.15.31:0/3570070991 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa0600a31e0 0x7fa0600a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:16.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.816+0000 7fa069c00640 1 -- 172.21.15.31:0/4191521506 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa070039040 con 0x5650dd37c6d0 2024-03-21T10:45:16.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.816+0000 7fa082271740 1 --2- 172.21.15.31:0/4191521506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650dd37c6d0 0x5650dd37cab0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa070009ca0 tx=0x7fa0700324b0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.816+0000 7fa075600640 1 -- 172.21.15.31:0/3570070991 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fa0640cf480 con 0x7fa078100fd0 2024-03-21T10:45:16.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.817+0000 7fa082271740 1 -- 172.21.15.31:0/4191521506 shutdown_connections 2024-03-21T10:45:16.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.817+0000 7fa082271740 1 --2- 172.21.15.31:0/4191521506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dd385c00 0x5650dd388050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.817+0000 7fa082271740 1 --2- 172.21.15.31:0/4191521506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650dd37d080 0x5650dd385630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.817+0000 7fa082271740 1 --2- 172.21.15.31:0/4191521506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650dd37c6d0 0x5650dd37cab0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.817+0000 7fa082271740 1 -- 172.21.15.31:0/4191521506 >> 172.21.15.31:0/4191521506 conn(0x5650dd376da0 msgr2=0x5650dd377180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:16.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.817+0000 7fa082271740 1 -- 172.21.15.31:0/4191521506 shutdown_connections 2024-03-21T10:45:16.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.817+0000 7fa082271740 1 -- 172.21.15.31:0/4191521506 wait complete. 2024-03-21T10:45:16.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.817+0000 7fa07cc00640 1 --2- 172.21.15.31:0/3570070991 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa0600a31e0 0x7fa0600a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:16.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.818+0000 7fa082271740 1 -- start start 2024-03-21T10:45:16.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.818+0000 7fa082271740 1 Processor -- start 2024-03-21T10:45:16.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.819+0000 7fa07cc00640 1 --2- 172.21.15.31:0/3570070991 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa0600a31e0 0x7fa0600a56a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x5650dd377e10 tx=0x7fa06c0046c0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:16.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.820+0000 7fa082271740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650dd37c6d0 0x5650dd3853e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:16.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.820+0000 7fa082271740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650dd37d080 0x5650dd382770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:16.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.820+0000 7fa082271740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dd385c00 0x5650dd382cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:16.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.822+0000 7fa07d600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dd385c00 0x5650dd382cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:16.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.822+0000 7fa07d600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dd385c00 0x5650dd382cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:45766/0 (socket says 172.21.15.31:45766) 2024-03-21T10:45:16.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.822+0000 7fa07cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650dd37c6d0 0x5650dd3853e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:16.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.822+0000 7fa07cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650dd37c6d0 0x5650dd3853e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51096/0 (socket says 172.21.15.31:51096) 2024-03-21T10:45:16.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.822+0000 7fa07d600640 1 -- 172.21.15.31:0/3665783295 learned_addr learned my addr 172.21.15.31:0/3665783295 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:16.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.822+0000 7fa082271740 1 -- 172.21.15.31:0/3665783295 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650dd388b60 con 0x5650dd37d080 2024-03-21T10:45:16.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.822+0000 7fa082271740 1 -- 172.21.15.31:0/3665783295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650dd29c0d0 con 0x5650dd37c6d0 2024-03-21T10:45:16.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.822+0000 7fa082271740 1 -- 172.21.15.31:0/3665783295 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650dd388ce0 con 0x5650dd385c00 2024-03-21T10:45:16.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.822+0000 7fa082271740 10 client.0 _ll_register_callbacks cb 0x5650dd2a0e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:16.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.823+0000 7fa07d600640 1 -- 172.21.15.31:0/3665783295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650dd37c6d0 msgr2=0x5650dd3853e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:16.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.823+0000 7fa07d600640 1 --2- 172.21.15.31:0/3665783295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650dd37c6d0 0x5650dd3853e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.823+0000 7fa07d600640 1 -- 172.21.15.31:0/3665783295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650dd37d080 msgr2=0x5650dd382770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:16.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.823+0000 7fa07d600640 1 --2- 172.21.15.31:0/3665783295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650dd37d080 0x5650dd382770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.823+0000 7fa07d600640 1 -- 172.21.15.31:0/3665783295 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5650dcff43f0 con 0x5650dd385c00 2024-03-21T10:45:16.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.825+0000 7fa05e000640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:45:16.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.825+0000 7fa07d600640 1 --2- 172.21.15.31:0/3665783295 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dd385c00 0x5650dd382cb0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa070009bf0 tx=0x7fa07003ff80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ad29cba671610c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:16.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.825+0000 7fa05e000640 1 -- 172.21.15.31:0/3665783295 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa07004c030 con 0x5650dd385c00 2024-03-21T10:45:16.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.826+0000 7fa05e000640 1 -- 172.21.15.31:0/3665783295 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fa070032590 con 0x5650dd385c00 2024-03-21T10:45:16.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.826+0000 7fa082271740 1 -- 172.21.15.31:0/3665783295 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa070032c70 con 0x5650dd385c00 2024-03-21T10:45:16.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.829+0000 7fa05e000640 1 -- 172.21.15.31:0/3665783295 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa070057050 con 0x5650dd385c00 2024-03-21T10:45:16.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.829+0000 7fa082271740 10 client.27163 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:16.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.829+0000 7fa082271740 1 -- 172.21.15.31:0/3665783295 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5650dd0be7d0 con 0x5650dd385c00 2024-03-21T10:45:16.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.831+0000 7fa05e000640 1 -- 172.21.15.31:0/3665783295 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa070032c70 con 0x5650dd385c00 2024-03-21T10:45:16.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.835+0000 7fa05e000640 1 -- 172.21.15.31:0/3665783295 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa070032790 con 0x5650dd385c00 2024-03-21T10:45:16.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.835+0000 7fa082271740 10 client.27163 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:16.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.835+0000 7fa082271740 10 client.27163 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:45:16.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.835+0000 7fa082271740 20 client.27163 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:16.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.835+0000 7fa082271740 1 --2- 172.21.15.31:0/3665783295 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5650dd395660 0x5650dd397a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:16.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.836+0000 7fa082271740 4 client.27163 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:45:16.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.836+0000 7fa082271740 1 -- 172.21.15.31:0/3665783295 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5650dd2ed800 con 0x5650dd395660 2024-03-21T10:45:16.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.837+0000 7fa07cc00640 1 --2- 172.21.15.31:0/3665783295 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5650dd395660 0x5650dd397a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:16.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.838+0000 7fa07cc00640 1 --2- 172.21.15.31:0/3665783295 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5650dd395660 0x5650dd397a50 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c86fa99314109b94 server_cookie=6ad44a19834a2db2 in_seq=0 out_seq=0 2024-03-21T10:45:16.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.838+0000 7fa05e000640 10 client.27163 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:45:16.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.839+0000 7fa05e000640 1 -- 172.21.15.31:0/3665783295 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5650dd2ed800 con 0x5650dd395660 2024-03-21T10:45:16.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.839+0000 7fa05e000640 10 client.27163 handle_command_reply: tid=0 2024-03-21T10:45:16.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.839+0000 7fa082271740 1 client.27163 shutdown 2024-03-21T10:45:16.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.840+0000 7fa082271740 1 -- 172.21.15.31:0/3665783295 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dd385c00 msgr2=0x5650dd382cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:16.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.840+0000 7fa082271740 1 --2- 172.21.15.31:0/3665783295 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dd385c00 0x5650dd382cb0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa070009bf0 tx=0x7fa07003ff80 comp rx=0 tx=0).stop 2024-03-21T10:45:16.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.840+0000 7fa082271740 1 -- 172.21.15.31:0/3665783295 shutdown_connections 2024-03-21T10:45:16.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.841+0000 7fa082271740 1 --2- 172.21.15.31:0/3665783295 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5650dd395660 0x5650dd397a50 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.842+0000 7fa082271740 1 --2- 172.21.15.31:0/3665783295 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dd385c00 0x5650dd382cb0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.842+0000 7fa082271740 1 --2- 172.21.15.31:0/3665783295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650dd37d080 0x5650dd382770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.842+0000 7fa082271740 1 --2- 172.21.15.31:0/3665783295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650dd37c6d0 0x5650dd3853e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.842+0000 7fa082271740 1 -- 172.21.15.31:0/3665783295 >> 172.21.15.31:0/3665783295 conn(0x5650dd376da0 msgr2=0x5650dd378260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:16.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.842+0000 7fa05e000640 0 client.27163 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:45:16.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.848+0000 7fa082271740 1 -- 172.21.15.31:0/3665783295 shutdown_connections 2024-03-21T10:45:16.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.849+0000 7fa082271740 1 -- 172.21.15.31:0/3665783295 wait complete. 2024-03-21T10:45:16.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.854+0000 7fa082271740 20 client.27163 trim_cache size 0 max 16384 2024-03-21T10:45:16.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.942+0000 7fa082271740 1 Processor -- start 2024-03-21T10:45:16.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.942+0000 7fa082271740 1 -- start start 2024-03-21T10:45:16.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.943+0000 7fa082271740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650dd37c6d0 0x5650dd37dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:16.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.944+0000 7fa082271740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dd37d080 0x5650dd384ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:16.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.944+0000 7fa07cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650dd37c6d0 0x5650dd37dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:16.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.944+0000 7fa07cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650dd37c6d0 0x5650dd37dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56418/0 (socket says 172.21.15.31:56418) 2024-03-21T10:45:16.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.944+0000 7fa07cc00640 1 -- 172.21.15.31:0/2966889763 learned_addr learned my addr 172.21.15.31:0/2966889763 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:16.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.945+0000 7fa077e00640 1 --2- 172.21.15.31:0/2966889763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dd37d080 0x5650dd384ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:16.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.945+0000 7fa082271740 1 --2- 172.21.15.31:0/2966889763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650dd385c00 0x5650dd3853e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:16.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.945+0000 7fa082271740 1 -- 172.21.15.31:0/2966889763 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0781c83e0 con 0x5650dd385c00 2024-03-21T10:45:16.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.945+0000 7fa082271740 1 -- 172.21.15.31:0/2966889763 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0781c5ee0 con 0x5650dd37c6d0 2024-03-21T10:45:16.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.945+0000 7fa082271740 1 -- 172.21.15.31:0/2966889763 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0781c3fa0 con 0x5650dd37d080 2024-03-21T10:45:16.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.946+0000 7fa07cc00640 1 -- 172.21.15.31:0/2966889763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dd37d080 msgr2=0x5650dd384ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:16.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.946+0000 7fa07cc00640 1 --2- 172.21.15.31:0/2966889763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dd37d080 0x5650dd384ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.946+0000 7fa07cc00640 1 -- 172.21.15.31:0/2966889763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650dd385c00 msgr2=0x5650dd3853e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:16.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.946+0000 7fa07cc00640 1 --2- 172.21.15.31:0/2966889763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650dd385c00 0x5650dd3853e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.946+0000 7fa07cc00640 1 -- 172.21.15.31:0/2966889763 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5650dd37b760 con 0x5650dd37c6d0 2024-03-21T10:45:16.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.948+0000 7fa077e00640 1 --2- 172.21.15.31:0/2966889763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dd37d080 0x5650dd384ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:45:16.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.948+0000 7fa07cc00640 1 --2- 172.21.15.31:0/2966889763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650dd37c6d0 0x5650dd37dbc0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fa06c009fd0 tx=0x7fa06c014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b31b17a7a2f251d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:16.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.948+0000 7fa069200640 1 -- 172.21.15.31:0/2966889763 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06c003b00 con 0x5650dd37c6d0 2024-03-21T10:45:16.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.949+0000 7fa069200640 1 -- 172.21.15.31:0/2966889763 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa06c005500 con 0x5650dd37c6d0 2024-03-21T10:45:16.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.949+0000 7fa069200640 1 -- 172.21.15.31:0/2966889763 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06c004d40 con 0x5650dd37c6d0 2024-03-21T10:45:16.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.950+0000 7fa082271740 1 -- 172.21.15.31:0/2966889763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650dd37c6d0 msgr2=0x5650dd37dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:16.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.950+0000 7fa082271740 1 --2- 172.21.15.31:0/2966889763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650dd37c6d0 0x5650dd37dbc0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fa06c009fd0 tx=0x7fa06c014000 comp rx=0 tx=0).stop 2024-03-21T10:45:16.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.951+0000 7fa082271740 1 -- 172.21.15.31:0/2966889763 shutdown_connections 2024-03-21T10:45:16.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.951+0000 7fa082271740 1 --2- 172.21.15.31:0/2966889763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650dd385c00 0x5650dd3853e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.951+0000 7fa082271740 1 --2- 172.21.15.31:0/2966889763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dd37d080 0x5650dd384ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.952+0000 7fa082271740 1 --2- 172.21.15.31:0/2966889763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650dd37c6d0 0x5650dd37dbc0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.952+0000 7fa082271740 1 -- 172.21.15.31:0/2966889763 >> 172.21.15.31:0/2966889763 conn(0x5650dd376da0 msgr2=0x5650dd37d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:16.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.952+0000 7fa082271740 1 -- 172.21.15.31:0/2966889763 shutdown_connections 2024-03-21T10:45:16.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.954+0000 7fa082271740 1 -- 172.21.15.31:0/2966889763 wait complete. 2024-03-21T10:45:16.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.955+0000 7fa082271740 1 -- start start 2024-03-21T10:45:16.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.955+0000 7fa082271740 1 Processor -- start 2024-03-21T10:45:16.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.956+0000 7fa082271740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dd37c6d0 0x5650dd398030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:16.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.957+0000 7fa082271740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650dd37d080 0x5650dd398570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:16.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.957+0000 7fa07cc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dd37c6d0 0x5650dd398030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:16.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.957+0000 7fa07cc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dd37c6d0 0x5650dd398030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34868/0 (socket says 172.21.15.31:34868) 2024-03-21T10:45:16.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.957+0000 7fa07cc00640 1 -- 172.21.15.31:0/523810346 learned_addr learned my addr 172.21.15.31:0/523810346 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:16.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.957+0000 7fa082271740 1 --2- 172.21.15.31:0/523810346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650dd385c00 0x5650dd398ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:16.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.957+0000 7fa082271740 1 -- 172.21.15.31:0/523810346 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0781c45a0 con 0x5650dd385c00 2024-03-21T10:45:16.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.957+0000 7fa082271740 1 -- 172.21.15.31:0/523810346 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0781c5880 con 0x5650dd37d080 2024-03-21T10:45:16.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.957+0000 7fa082271740 1 -- 172.21.15.31:0/523810346 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0781c4bc0 con 0x5650dd37c6d0 2024-03-21T10:45:16.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.958+0000 7fa082271740 10 client.0 _ll_register_callbacks cb 0x7fa070037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:16.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.958+0000 7fa07d600640 1 --2- 172.21.15.31:0/523810346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650dd385c00 0x5650dd398ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:16.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.959+0000 7fa07d600640 1 -- 172.21.15.31:0/523810346 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dd37c6d0 msgr2=0x5650dd398030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:16.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.959+0000 7fa07d600640 1 --2- 172.21.15.31:0/523810346 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dd37c6d0 0x5650dd398030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.959+0000 7fa07d600640 1 -- 172.21.15.31:0/523810346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650dd37d080 msgr2=0x5650dd398570 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:45:16.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.959+0000 7fa07d600640 1 --2- 172.21.15.31:0/523810346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650dd37d080 0x5650dd398570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:16.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.959+0000 7fa07d600640 1 -- 172.21.15.31:0/523810346 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5650dd3efd60 con 0x5650dd385c00 2024-03-21T10:45:16.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.960+0000 7fa07d600640 1 --2- 172.21.15.31:0/523810346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650dd385c00 0x5650dd398ab0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa07002c6f0 tx=0x7fa07004a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dee407729dbd755d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:16.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.960+0000 7fa05e000640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:45:16.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.961+0000 7fa05e000640 1 -- 172.21.15.31:0/523810346 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa07004c070 con 0x5650dd385c00 2024-03-21T10:45:16.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.961+0000 7fa05e000640 1 -- 172.21.15.31:0/523810346 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fa070034440 con 0x5650dd385c00 2024-03-21T10:45:16.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.962+0000 7fa082271740 1 -- 172.21.15.31:0/523810346 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa06c005500 con 0x5650dd385c00 2024-03-21T10:45:16.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.962+0000 7fa05e000640 1 -- 172.21.15.31:0/523810346 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa070039040 con 0x5650dd385c00 2024-03-21T10:45:16.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.966+0000 7fa082271740 10 client.16686 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:16.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.966+0000 7fa082271740 1 -- 172.21.15.31:0/523810346 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5650dd3f0f90 con 0x5650dd385c00 2024-03-21T10:45:16.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.967+0000 7fa05e000640 1 -- 172.21.15.31:0/523810346 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa06c005500 con 0x5650dd385c00 2024-03-21T10:45:16.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.967+0000 7fa05e000640 1 -- 172.21.15.31:0/523810346 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa070040680 con 0x5650dd385c00 2024-03-21T10:45:16.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.967+0000 7fa082271740 10 client.16686 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:16.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.967+0000 7fa082271740 10 client.16686 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:45:16.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.968+0000 7fa082271740 20 client.16686 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:16.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.968+0000 7fa082271740 1 --2- 172.21.15.31:0/523810346 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5650dd3f1600 0x5650dd3f3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:16.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.968+0000 7fa082271740 4 client.16686 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:45:16.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.968+0000 7fa082271740 1 -- 172.21.15.31:0/523810346 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5650dd2e7b50 con 0x5650dd3f1600 2024-03-21T10:45:16.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.971+0000 7fa07cc00640 1 --2- 172.21.15.31:0/523810346 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5650dd3f1600 0x5650dd3f3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:16.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.974+0000 7fa07cc00640 1 --2- 172.21.15.31:0/523810346 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5650dd3f1600 0x5650dd3f3a20 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=48cc1eb67eb99586 server_cookie=fa7a250e6848f4c1 in_seq=0 out_seq=0 2024-03-21T10:45:16.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.974+0000 7fa05e000640 10 client.16686 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:45:16.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.976+0000 7fa05e000640 1 -- 172.21.15.31:0/523810346 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5650dd2e7b50 con 0x5650dd3f1600 2024-03-21T10:45:16.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.976+0000 7fa05e000640 10 client.16686 handle_command_reply: tid=0 2024-03-21T10:45:16.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:16.976+0000 7fa082271740 1 client.16686 shutdown 2024-03-21T10:45:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.006+0000 7fa082271740 1 -- 172.21.15.31:0/523810346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650dd385c00 msgr2=0x5650dd398ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.006+0000 7fa082271740 1 --2- 172.21.15.31:0/523810346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650dd385c00 0x5650dd398ab0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa07002c6f0 tx=0x7fa07004a000 comp rx=0 tx=0).stop 2024-03-21T10:45:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.006+0000 7fa082271740 1 -- 172.21.15.31:0/523810346 shutdown_connections 2024-03-21T10:45:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.006+0000 7fa082271740 1 --2- 172.21.15.31:0/523810346 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5650dd3f1600 0x5650dd3f3a20 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.006+0000 7fa082271740 1 --2- 172.21.15.31:0/523810346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5650dd385c00 0x5650dd398ab0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.006+0000 7fa082271740 1 --2- 172.21.15.31:0/523810346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650dd37d080 0x5650dd398570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.006+0000 7fa082271740 1 --2- 172.21.15.31:0/523810346 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dd37c6d0 0x5650dd398030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.006+0000 7fa082271740 1 -- 172.21.15.31:0/523810346 >> 172.21.15.31:0/523810346 conn(0x5650dd376da0 msgr2=0x5650dd37dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:17.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.006+0000 7fa05e000640 0 client.16686 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:45:17.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.007+0000 7fa082271740 1 -- 172.21.15.31:0/523810346 shutdown_connections 2024-03-21T10:45:17.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.007+0000 7fa082271740 1 -- 172.21.15.31:0/523810346 wait complete. 2024-03-21T10:45:17.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.007+0000 7fa082271740 20 client.16686 trim_cache size 0 max 16384 2024-03-21T10:45:17.007 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:45:17.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.011+0000 7fa07f600640 1 -- 172.21.15.31:0/3570070991 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa0600a31e0 msgr2=0x7fa0600a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:17.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.011+0000 7fa07f600640 1 --2- 172.21.15.31:0/3570070991 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa0600a31e0 0x7fa0600a56a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x5650dd377e10 tx=0x7fa06c0046c0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.011+0000 7fa07f600640 1 -- 172.21.15.31:0/3570070991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa078100fd0 msgr2=0x7fa07810c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:17.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.011+0000 7fa07f600640 1 --2- 172.21.15.31:0/3570070991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa078100fd0 0x7fa07810c670 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa06400c920 tx=0x7fa06400cde0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.011+0000 7fa07f600640 1 -- 172.21.15.31:0/3570070991 shutdown_connections 2024-03-21T10:45:17.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.011+0000 7fa07f600640 1 --2- 172.21.15.31:0/3570070991 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa0600a31e0 0x7fa0600a56a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.011+0000 7fa07f600640 1 --2- 172.21.15.31:0/3570070991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa078105ab0 0x7fa078106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.011+0000 7fa07f600640 1 --2- 172.21.15.31:0/3570070991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa078100fd0 0x7fa07810c670 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.011+0000 7fa07f600640 1 --2- 172.21.15.31:0/3570070991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0781006b0 0x7fa07810c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.011+0000 7fa07f600640 1 -- 172.21.15.31:0/3570070991 >> 172.21.15.31:0/3570070991 conn(0x7fa078077750 msgr2=0x7fa078103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:17.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.012+0000 7fa07f600640 1 -- 172.21.15.31:0/3570070991 shutdown_connections 2024-03-21T10:45:17.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.012+0000 7fa07f600640 1 -- 172.21.15.31:0/3570070991 wait complete. 2024-03-21T10:45:17.056 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:45:17.057 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-21T10:45:17.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:17 smithi031 ceph-mon[31673]: pgmap v205: 129 pgs: 129 active+clean; 44 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 216 MiB/s wr, 1.85k op/s 2024-03-21T10:45:17.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:17 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/900010738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:45:17.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:17 smithi031 ceph-mon[31673]: osdmap e96: 12 total, 12 up, 12 in 2024-03-21T10:45:17.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.520+0000 7fd051a00640 1 -- 172.21.15.31:0/1666318841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd04c108be0 msgr2=0x7fd04c109040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:17.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.520+0000 7fd051a00640 1 --2- 172.21.15.31:0/1666318841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd04c108be0 0x7fd04c109040 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd03c009900 tx=0x7fd03c02bc80 comp rx=0 tx=0).stop 2024-03-21T10:45:17.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.521+0000 7fd051a00640 1 -- 172.21.15.31:0/1666318841 shutdown_connections 2024-03-21T10:45:17.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.521+0000 7fd051a00640 1 --2- 172.21.15.31:0/1666318841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd04c11ae80 0x7fd04c11d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.521+0000 7fd051a00640 1 --2- 172.21.15.31:0/1666318841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd04c108be0 0x7fd04c109040 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.521+0000 7fd051a00640 1 --2- 172.21.15.31:0/1666318841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd04c10fe30 0x7fd04c1086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.521+0000 7fd051a00640 1 -- 172.21.15.31:0/1666318841 >> 172.21.15.31:0/1666318841 conn(0x7fd04c06c710 msgr2=0x7fd04c06cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:17.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.522+0000 7fd051a00640 1 -- 172.21.15.31:0/1666318841 shutdown_connections 2024-03-21T10:45:17.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.523+0000 7fd051a00640 1 -- 172.21.15.31:0/1666318841 wait complete. 2024-03-21T10:45:17.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.525+0000 7fd051a00640 1 Processor -- start 2024-03-21T10:45:17.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.525+0000 7fd051a00640 1 -- start start 2024-03-21T10:45:17.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.528+0000 7fd051a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd04c108be0 0x7fd04c1a6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:17.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.528+0000 7fd051a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd04c10fe30 0x7fd04c1a74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:17.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.528+0000 7fd051a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd04c11ae80 0x7fd04c1ab830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:17.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.529+0000 7fd04aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd04c108be0 0x7fd04c1a6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:17.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.529+0000 7fd04aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd04c108be0 0x7fd04c1a6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56434/0 (socket says 172.21.15.31:56434) 2024-03-21T10:45:17.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.529+0000 7fd051a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c1209f0 con 0x7fd04c11ae80 2024-03-21T10:45:17.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.529+0000 7fd051a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c120870 con 0x7fd04c108be0 2024-03-21T10:45:17.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.529+0000 7fd051a00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c120b70 con 0x7fd04c10fe30 2024-03-21T10:45:17.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.529+0000 7fd04aa00640 1 -- 172.21.15.31:0/332827406 learned_addr learned my addr 172.21.15.31:0/332827406 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:17.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.529+0000 7fd04b400640 1 --2- 172.21.15.31:0/332827406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd04c11ae80 0x7fd04c1ab830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:17.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.530+0000 7fd04aa00640 1 -- 172.21.15.31:0/332827406 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd04c10fe30 msgr2=0x7fd04c1a74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:17.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.529+0000 7fd04a000640 1 --2- 172.21.15.31:0/332827406 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd04c10fe30 0x7fd04c1a74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:17.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.531+0000 7fd04aa00640 1 --2- 172.21.15.31:0/332827406 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd04c10fe30 0x7fd04c1a74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.531+0000 7fd04aa00640 1 -- 172.21.15.31:0/332827406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd04c11ae80 msgr2=0x7fd04c1ab830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:17.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.531+0000 7fd04aa00640 1 --2- 172.21.15.31:0/332827406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd04c11ae80 0x7fd04c1ab830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.531+0000 7fd04aa00640 1 -- 172.21.15.31:0/332827406 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04c1abf60 con 0x7fd04c108be0 2024-03-21T10:45:17.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.531+0000 7fd04a000640 1 --2- 172.21.15.31:0/332827406 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd04c10fe30 0x7fd04c1a74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:45:17.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.532+0000 7fd04aa00640 1 --2- 172.21.15.31:0/332827406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd04c108be0 0x7fd04c1a6f70 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd04400ba50 tx=0x7fd04400bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:17.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.532+0000 7fd043400640 1 -- 172.21.15.31:0/332827406 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd044019070 con 0x7fd04c108be0 2024-03-21T10:45:17.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.532+0000 7fd051a00640 1 -- 172.21.15.31:0/332827406 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd04c1ac250 con 0x7fd04c108be0 2024-03-21T10:45:17.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.533+0000 7fd051a00640 1 -- 172.21.15.31:0/332827406 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd04c1b3ae0 con 0x7fd04c108be0 2024-03-21T10:45:17.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.535+0000 7fd043400640 1 -- 172.21.15.31:0/332827406 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd04400cc10 con 0x7fd04c108be0 2024-03-21T10:45:17.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.535+0000 7fd043400640 1 -- 172.21.15.31:0/332827406 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd044019070 con 0x7fd04c108be0 2024-03-21T10:45:17.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.538+0000 7fd05466d740 1 Processor -- start 2024-03-21T10:45:17.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.538+0000 7fd05466d740 1 -- start start 2024-03-21T10:45:17.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.539+0000 7fd05466d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563555c686d0 0x563555c68ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:17.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.539+0000 7fd05466d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563555c69080 0x563555c71630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:17.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.539+0000 7fd05466d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563555c71c00 0x563555c74050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:17.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.539+0000 7fd05466d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563555bcc2c0 con 0x563555c69080 2024-03-21T10:45:17.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.539+0000 7fd05466d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x563555b74c90 con 0x563555c686d0 2024-03-21T10:45:17.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.539+0000 7fd05466d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x563555bc8360 con 0x563555c71c00 2024-03-21T10:45:17.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.539+0000 7fd04a000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563555c71c00 0x563555c74050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:17.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.540+0000 7fd04a000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563555c71c00 0x563555c74050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34900/0 (socket says 172.21.15.31:34900) 2024-03-21T10:45:17.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.540+0000 7fd04a000640 1 -- 172.21.15.31:0/3119500718 learned_addr learned my addr 172.21.15.31:0/3119500718 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:17.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.540+0000 7fd043400640 1 -- 172.21.15.31:0/332827406 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fd04401c070 con 0x7fd04c108be0 2024-03-21T10:45:17.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.540+0000 7fd04b400640 1 --2- 172.21.15.31:0/3119500718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563555c686d0 0x563555c68ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:17.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.540+0000 7fd04a000640 1 -- 172.21.15.31:0/3119500718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563555c686d0 msgr2=0x563555c68ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:17.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.540+0000 7fd04aa00640 1 --2- 172.21.15.31:0/3119500718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563555c69080 0x563555c71630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:17.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.540+0000 7fd04a000640 1 --2- 172.21.15.31:0/3119500718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563555c686d0 0x563555c68ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.540+0000 7fd04a000640 1 -- 172.21.15.31:0/3119500718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563555c69080 msgr2=0x563555c71630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:17.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.540+0000 7fd04a000640 1 --2- 172.21.15.31:0/3119500718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563555c69080 0x563555c71630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.540+0000 7fd04a000640 1 -- 172.21.15.31:0/3119500718 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563555b99080 con 0x563555c71c00 2024-03-21T10:45:17.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.540+0000 7fd04b400640 1 --2- 172.21.15.31:0/3119500718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563555c686d0 0x563555c68ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:45:17.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.541+0000 7fd04a000640 1 --2- 172.21.15.31:0/3119500718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563555c71c00 0x563555c74050 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd03c006010 tx=0x7fd03c02c760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=146d05823155b5d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:17.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.541+0000 7fd037e00640 1 -- 172.21.15.31:0/3119500718 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03c042070 con 0x563555c71c00 2024-03-21T10:45:17.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.541+0000 7fd037e00640 1 -- 172.21.15.31:0/3119500718 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd03c004340 con 0x563555c71c00 2024-03-21T10:45:17.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.541+0000 7fd037e00640 1 -- 172.21.15.31:0/3119500718 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03c004920 con 0x563555c71c00 2024-03-21T10:45:17.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.541+0000 7fd043400640 1 --2- 172.21.15.31:0/332827406 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd0300a3340 0x7fd0300a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:17.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.542+0000 7fd043400640 1 -- 172.21.15.31:0/332827406 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fd0440ceb90 con 0x7fd04c108be0 2024-03-21T10:45:17.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.542+0000 7fd04b400640 1 --2- 172.21.15.31:0/332827406 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd0300a3340 0x7fd0300a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:17.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.542+0000 7fd05466d740 1 -- 172.21.15.31:0/3119500718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563555c71c00 msgr2=0x563555c74050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:17.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.542+0000 7fd05466d740 1 --2- 172.21.15.31:0/3119500718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563555c71c00 0x563555c74050 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd03c006010 tx=0x7fd03c02c760 comp rx=0 tx=0).stop 2024-03-21T10:45:17.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.542+0000 7fd05466d740 1 -- 172.21.15.31:0/3119500718 shutdown_connections 2024-03-21T10:45:17.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.542+0000 7fd05466d740 1 --2- 172.21.15.31:0/3119500718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563555c71c00 0x563555c74050 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.542+0000 7fd05466d740 1 --2- 172.21.15.31:0/3119500718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563555c69080 0x563555c71630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.542+0000 7fd05466d740 1 --2- 172.21.15.31:0/3119500718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563555c686d0 0x563555c68ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.542+0000 7fd05466d740 1 -- 172.21.15.31:0/3119500718 >> 172.21.15.31:0/3119500718 conn(0x563555c62da0 msgr2=0x563555c63180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:17.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.543+0000 7fd05466d740 1 -- 172.21.15.31:0/3119500718 shutdown_connections 2024-03-21T10:45:17.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.543+0000 7fd05466d740 1 -- 172.21.15.31:0/3119500718 wait complete. 2024-03-21T10:45:17.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.543+0000 7fd05466d740 1 -- start start 2024-03-21T10:45:17.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.543+0000 7fd04b400640 1 --2- 172.21.15.31:0/332827406 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd0300a3340 0x7fd0300a5800 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd038006fd0 tx=0x7fd038008040 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:17.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.543+0000 7fd05466d740 1 Processor -- start 2024-03-21T10:45:17.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.544+0000 7fd05466d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563555c686d0 0x563555c714c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:17.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.544+0000 7fd05466d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563555c69080 0x563555c6e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:17.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.544+0000 7fd05466d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563555c71c00 0x563555c6edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:17.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.544+0000 7fd05466d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563555c74b10 con 0x563555c71c00 2024-03-21T10:45:17.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.544+0000 7fd05466d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x563555b880d0 con 0x563555c69080 2024-03-21T10:45:17.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.544+0000 7fd05466d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x563555c74c90 con 0x563555c686d0 2024-03-21T10:45:17.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.544+0000 7fd05466d740 10 client.0 _ll_register_callbacks cb 0x563555b8ce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:17.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.544+0000 7fd04a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563555c69080 0x563555c6e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:17.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.544+0000 7fd04a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563555c69080 0x563555c6e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56448/0 (socket says 172.21.15.31:56448) 2024-03-21T10:45:17.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.544+0000 7fd04a000640 1 -- 172.21.15.31:0/192427857 learned_addr learned my addr 172.21.15.31:0/192427857 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:17.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.544+0000 7fd04b400640 1 --2- 172.21.15.31:0/192427857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563555c71c00 0x563555c6edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:17.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.544+0000 7fd04aa00640 1 --2- 172.21.15.31:0/192427857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563555c686d0 0x563555c714c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:17.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.545+0000 7fd04a000640 1 -- 172.21.15.31:0/192427857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563555c686d0 msgr2=0x563555c714c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:17.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.545+0000 7fd04a000640 1 --2- 172.21.15.31:0/192427857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563555c686d0 0x563555c714c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.545+0000 7fd04a000640 1 -- 172.21.15.31:0/192427857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563555c71c00 msgr2=0x563555c6edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:17.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.545+0000 7fd04a000640 1 --2- 172.21.15.31:0/192427857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563555c71c00 0x563555c6edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.545+0000 7fd04a000640 1 -- 172.21.15.31:0/192427857 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5635558e03f0 con 0x563555c69080 2024-03-21T10:45:17.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.545+0000 7fd04a000640 1 --2- 172.21.15.31:0/192427857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563555c69080 0x563555c6e850 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fd03c000c00 tx=0x7fd03c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9769df4d71e19bc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:17.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.545+0000 7fd034c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:45:17.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.545+0000 7fd034c00640 1 -- 172.21.15.31:0/192427857 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03c042040 con 0x563555c69080 2024-03-21T10:45:17.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.545+0000 7fd034c00640 1 -- 172.21.15.31:0/192427857 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fd03c0046c0 con 0x563555c69080 2024-03-21T10:45:17.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.546+0000 7fd05466d740 1 -- 172.21.15.31:0/192427857 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd03c004340 con 0x563555c69080 2024-03-21T10:45:17.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.546+0000 7fd034c00640 1 -- 172.21.15.31:0/192427857 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd03c0048c0 con 0x563555c69080 2024-03-21T10:45:17.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.546+0000 7fd05466d740 10 client.27254 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:17.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.547+0000 7fd05466d740 1 -- 172.21.15.31:0/192427857 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5635559aa7d0 con 0x563555c69080 2024-03-21T10:45:17.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.547+0000 7fd034c00640 1 -- 172.21.15.31:0/192427857 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd03c004340 con 0x563555c69080 2024-03-21T10:45:17.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.548+0000 7fd034c00640 1 -- 172.21.15.31:0/192427857 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd03c003ba0 con 0x563555c69080 2024-03-21T10:45:17.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.548+0000 7fd05466d740 10 client.27254 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:17.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.548+0000 7fd05466d740 10 client.27254 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:45:17.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.548+0000 7fd05466d740 20 client.27254 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:17.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.548+0000 7fd05466d740 1 --2- 172.21.15.31:0/192427857 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x563555c817e0 0x563555c83bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:17.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.548+0000 7fd05466d740 4 client.27254 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:45:17.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.548+0000 7fd05466d740 1 -- 172.21.15.31:0/192427857 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563555bd9800 con 0x563555c817e0 2024-03-21T10:45:17.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.548+0000 7fd04aa00640 1 --2- 172.21.15.31:0/192427857 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x563555c817e0 0x563555c83bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:17.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.549+0000 7fd04aa00640 1 --2- 172.21.15.31:0/192427857 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x563555c817e0 0x563555c83bd0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ed96f9a26f661651 server_cookie=281995bbaf04e721 in_seq=0 out_seq=0 2024-03-21T10:45:17.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.549+0000 7fd034c00640 10 client.27254 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:45:17.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.550+0000 7fd034c00640 1 -- 172.21.15.31:0/192427857 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563555bd9800 con 0x563555c817e0 2024-03-21T10:45:17.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.550+0000 7fd034c00640 10 client.27254 handle_command_reply: tid=0 2024-03-21T10:45:17.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.550+0000 7fd05466d740 1 client.27254 shutdown 2024-03-21T10:45:17.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.551+0000 7fd05466d740 1 -- 172.21.15.31:0/192427857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563555c69080 msgr2=0x563555c6e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:17.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.551+0000 7fd05466d740 1 --2- 172.21.15.31:0/192427857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563555c69080 0x563555c6e850 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fd03c000c00 tx=0x7fd03c009260 comp rx=0 tx=0).stop 2024-03-21T10:45:17.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.551+0000 7fd05466d740 1 -- 172.21.15.31:0/192427857 shutdown_connections 2024-03-21T10:45:17.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.551+0000 7fd05466d740 1 --2- 172.21.15.31:0/192427857 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x563555c817e0 0x563555c83bd0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.551+0000 7fd05466d740 1 --2- 172.21.15.31:0/192427857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563555c71c00 0x563555c6edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.551+0000 7fd05466d740 1 --2- 172.21.15.31:0/192427857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563555c69080 0x563555c6e850 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.551+0000 7fd05466d740 1 --2- 172.21.15.31:0/192427857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563555c686d0 0x563555c714c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.551+0000 7fd05466d740 1 -- 172.21.15.31:0/192427857 >> 172.21.15.31:0/192427857 conn(0x563555c62da0 msgr2=0x563555c64af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:17.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.551+0000 7fd034c00640 0 client.27254 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:45:17.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.552+0000 7fd05466d740 1 -- 172.21.15.31:0/192427857 shutdown_connections 2024-03-21T10:45:17.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.552+0000 7fd05466d740 1 -- 172.21.15.31:0/192427857 wait complete. 2024-03-21T10:45:17.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.552+0000 7fd05466d740 20 client.27254 trim_cache size 0 max 16384 2024-03-21T10:45:17.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.566+0000 7fd05466d740 1 Processor -- start 2024-03-21T10:45:17.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.566+0000 7fd05466d740 1 -- start start 2024-03-21T10:45:17.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.566+0000 7fd05466d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563555c686d0 0x563555c61eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:17.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.566+0000 7fd05466d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563555c69080 0x563555c70f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:17.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.566+0000 7fd05466d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563555c71c00 0x563555c714c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:17.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.566+0000 7fd05466d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c012280 con 0x563555c71c00 2024-03-21T10:45:17.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.566+0000 7fd05466d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c00fd80 con 0x563555c686d0 2024-03-21T10:45:17.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.566+0000 7fd05466d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c00de40 con 0x563555c69080 2024-03-21T10:45:17.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.566+0000 7fd04aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563555c686d0 0x563555c61eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:17.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.566+0000 7fd04aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563555c686d0 0x563555c61eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56458/0 (socket says 172.21.15.31:56458) 2024-03-21T10:45:17.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.566+0000 7fd04aa00640 1 -- 172.21.15.31:0/187554532 learned_addr learned my addr 172.21.15.31:0/187554532 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:17.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.566+0000 7fd04b400640 1 --2- 172.21.15.31:0/187554532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563555c71c00 0x563555c714c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:17.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.566+0000 7fd04a000640 1 --2- 172.21.15.31:0/187554532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563555c69080 0x563555c70f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:17.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.567+0000 7fd04aa00640 1 -- 172.21.15.31:0/187554532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563555c69080 msgr2=0x563555c70f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:17.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.567+0000 7fd04aa00640 1 --2- 172.21.15.31:0/187554532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563555c69080 0x563555c70f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.567+0000 7fd04aa00640 1 -- 172.21.15.31:0/187554532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563555c71c00 msgr2=0x563555c714c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:17.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.567+0000 7fd04aa00640 1 --2- 172.21.15.31:0/187554532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563555c71c00 0x563555c714c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.567+0000 7fd04aa00640 1 -- 172.21.15.31:0/187554532 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563555c74c00 con 0x563555c686d0 2024-03-21T10:45:17.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.567+0000 7fd04aa00640 1 --2- 172.21.15.31:0/187554532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563555c686d0 0x563555c61eb0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fd04409a9f0 tx=0x7fd0440d1110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f252f11a7d749cf server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:17.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.567+0000 7fd037400640 1 -- 172.21.15.31:0/187554532 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0440a3700 con 0x563555c686d0 2024-03-21T10:45:17.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.567+0000 7fd037400640 1 -- 172.21.15.31:0/187554532 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd04400cdc0 con 0x563555c686d0 2024-03-21T10:45:17.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.567+0000 7fd037400640 1 -- 172.21.15.31:0/187554532 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0440a3270 con 0x563555c686d0 2024-03-21T10:45:17.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.568+0000 7fd05466d740 1 -- 172.21.15.31:0/187554532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563555c686d0 msgr2=0x563555c61eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:17.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.568+0000 7fd05466d740 1 --2- 172.21.15.31:0/187554532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563555c686d0 0x563555c61eb0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fd04409a9f0 tx=0x7fd0440d1110 comp rx=0 tx=0).stop 2024-03-21T10:45:17.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.568+0000 7fd05466d740 1 -- 172.21.15.31:0/187554532 shutdown_connections 2024-03-21T10:45:17.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.568+0000 7fd05466d740 1 --2- 172.21.15.31:0/187554532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563555c71c00 0x563555c714c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.568+0000 7fd05466d740 1 --2- 172.21.15.31:0/187554532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563555c69080 0x563555c70f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.568+0000 7fd05466d740 1 --2- 172.21.15.31:0/187554532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563555c686d0 0x563555c61eb0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.568+0000 7fd05466d740 1 -- 172.21.15.31:0/187554532 >> 172.21.15.31:0/187554532 conn(0x563555c62da0 msgr2=0x563555c69460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:17.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.568+0000 7fd05466d740 1 -- 172.21.15.31:0/187554532 shutdown_connections 2024-03-21T10:45:17.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.568+0000 7fd05466d740 1 -- 172.21.15.31:0/187554532 wait complete. 2024-03-21T10:45:17.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.569+0000 7fd05466d740 1 -- start start 2024-03-21T10:45:17.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.569+0000 7fd05466d740 1 Processor -- start 2024-03-21T10:45:17.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.569+0000 7fd05466d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563555c686d0 0x563555c79ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:17.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.569+0000 7fd05466d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563555c69080 0x563555c75e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:17.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.569+0000 7fd05466d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563555c71c00 0x563555c76380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:17.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.569+0000 7fd05466d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c00e440 con 0x563555c71c00 2024-03-21T10:45:17.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.569+0000 7fd05466d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c00f720 con 0x563555c686d0 2024-03-21T10:45:17.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.569+0000 7fd05466d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c00ea60 con 0x563555c69080 2024-03-21T10:45:17.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.569+0000 7fd05466d740 10 client.0 _ll_register_callbacks cb 0x7fd03c02c6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:17.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.569+0000 7fd04b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563555c71c00 0x563555c76380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:17.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.570+0000 7fd04b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563555c71c00 0x563555c76380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43426/0 (socket says 172.21.15.31:43426) 2024-03-21T10:45:17.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.570+0000 7fd04b400640 1 -- 172.21.15.31:0/2201053602 learned_addr learned my addr 172.21.15.31:0/2201053602 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:17.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.570+0000 7fd04a000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563555c69080 0x563555c75e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:17.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.570+0000 7fd04aa00640 1 --2- 172.21.15.31:0/2201053602 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563555c686d0 0x563555c79ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:17.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.570+0000 7fd04b400640 1 -- 172.21.15.31:0/2201053602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563555c69080 msgr2=0x563555c75e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:17.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.570+0000 7fd04b400640 1 --2- 172.21.15.31:0/2201053602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563555c69080 0x563555c75e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.570+0000 7fd04b400640 1 -- 172.21.15.31:0/2201053602 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563555c686d0 msgr2=0x563555c79ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:17.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.570+0000 7fd04b400640 1 --2- 172.21.15.31:0/2201053602 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563555c686d0 0x563555c79ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.570+0000 7fd04b400640 1 -- 172.21.15.31:0/2201053602 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563555c769c0 con 0x563555c71c00 2024-03-21T10:45:17.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.570+0000 7fd04aa00640 1 --2- 172.21.15.31:0/2201053602 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563555c686d0 0x563555c79ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:45:17.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.570+0000 7fd04b400640 1 --2- 172.21.15.31:0/2201053602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563555c71c00 0x563555c76380 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd03800f000 tx=0x7fd0380046e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b7744ec78c05648 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:17.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.570+0000 7fd034c00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:45:17.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.571+0000 7fd034c00640 1 -- 172.21.15.31:0/2201053602 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd038027070 con 0x563555c71c00 2024-03-21T10:45:17.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.571+0000 7fd034c00640 1 -- 172.21.15.31:0/2201053602 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fd038023440 con 0x563555c71c00 2024-03-21T10:45:17.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.571+0000 7fd05466d740 1 -- 172.21.15.31:0/2201053602 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd04400cdc0 con 0x563555c71c00 2024-03-21T10:45:17.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.571+0000 7fd034c00640 1 -- 172.21.15.31:0/2201053602 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd038031050 con 0x563555c71c00 2024-03-21T10:45:17.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.571+0000 7fd05466d740 10 client.16701 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:17.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.571+0000 7fd05466d740 1 -- 172.21.15.31:0/2201053602 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x563555cddc20 con 0x563555c71c00 2024-03-21T10:45:17.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.571+0000 7fd034c00640 1 -- 172.21.15.31:0/2201053602 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd04400cdc0 con 0x563555c71c00 2024-03-21T10:45:17.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.572+0000 7fd034c00640 1 -- 172.21.15.31:0/2201053602 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd038012030 con 0x563555c71c00 2024-03-21T10:45:17.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.572+0000 7fd05466d740 10 client.16701 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:17.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.572+0000 7fd05466d740 10 client.16701 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:45:17.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.572+0000 7fd05466d740 20 client.16701 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:17.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.572+0000 7fd05466d740 1 --2- 172.21.15.31:0/2201053602 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x563555cde3d0 0x563555ce07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:17.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.572+0000 7fd05466d740 4 client.16701 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:45:17.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.572+0000 7fd05466d740 1 -- 172.21.15.31:0/2201053602 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563555bd3b50 con 0x563555cde3d0 2024-03-21T10:45:17.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.572+0000 7fd04aa00640 1 --2- 172.21.15.31:0/2201053602 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x563555cde3d0 0x563555ce07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:17.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.573+0000 7fd034c00640 10 client.16701 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:45:17.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.572+0000 7fd04aa00640 1 --2- 172.21.15.31:0/2201053602 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x563555cde3d0 0x563555ce07f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9f9e7624ba4584f8 server_cookie=656cdd040693647 in_seq=0 out_seq=0 2024-03-21T10:45:17.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.573+0000 7fd034c00640 1 -- 172.21.15.31:0/2201053602 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563555bd3b50 con 0x563555cde3d0 2024-03-21T10:45:17.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.573+0000 7fd034c00640 10 client.16701 handle_command_reply: tid=0 2024-03-21T10:45:17.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.573+0000 7fd05466d740 1 client.16701 shutdown 2024-03-21T10:45:17.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.573+0000 7fd05466d740 1 -- 172.21.15.31:0/2201053602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563555c71c00 msgr2=0x563555c76380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:17.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.573+0000 7fd05466d740 1 --2- 172.21.15.31:0/2201053602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563555c71c00 0x563555c76380 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd03800f000 tx=0x7fd0380046e0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.573+0000 7fd05466d740 1 -- 172.21.15.31:0/2201053602 shutdown_connections 2024-03-21T10:45:17.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.573+0000 7fd05466d740 1 --2- 172.21.15.31:0/2201053602 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x563555cde3d0 0x563555ce07f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.573+0000 7fd05466d740 1 --2- 172.21.15.31:0/2201053602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563555c71c00 0x563555c76380 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.573+0000 7fd05466d740 1 --2- 172.21.15.31:0/2201053602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563555c69080 0x563555c75e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.573+0000 7fd05466d740 1 --2- 172.21.15.31:0/2201053602 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563555c686d0 0x563555c79ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.573+0000 7fd05466d740 1 -- 172.21.15.31:0/2201053602 >> 172.21.15.31:0/2201053602 conn(0x563555c62da0 msgr2=0x563555c61d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:17.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.573+0000 7fd034c00640 0 client.16701 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:45:17.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.573+0000 7fd05466d740 1 -- 172.21.15.31:0/2201053602 shutdown_connections 2024-03-21T10:45:17.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.573+0000 7fd05466d740 1 -- 172.21.15.31:0/2201053602 wait complete. 2024-03-21T10:45:17.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.574+0000 7fd05466d740 20 client.16701 trim_cache size 0 max 16384 2024-03-21T10:45:17.574 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:45:17.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.574+0000 7fd051a00640 1 -- 172.21.15.31:0/332827406 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd0300a3340 msgr2=0x7fd0300a5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:17.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.574+0000 7fd051a00640 1 --2- 172.21.15.31:0/332827406 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd0300a3340 0x7fd0300a5800 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd038006fd0 tx=0x7fd038008040 comp rx=0 tx=0).stop 2024-03-21T10:45:17.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.574+0000 7fd051a00640 1 -- 172.21.15.31:0/332827406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd04c108be0 msgr2=0x7fd04c1a6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:17.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.574+0000 7fd051a00640 1 --2- 172.21.15.31:0/332827406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd04c108be0 0x7fd04c1a6f70 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd04400ba50 tx=0x7fd04400bf10 comp rx=0 tx=0).stop 2024-03-21T10:45:17.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.574+0000 7fd051a00640 1 -- 172.21.15.31:0/332827406 shutdown_connections 2024-03-21T10:45:17.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.574+0000 7fd051a00640 1 --2- 172.21.15.31:0/332827406 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd0300a3340 0x7fd0300a5800 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.574+0000 7fd051a00640 1 --2- 172.21.15.31:0/332827406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd04c11ae80 0x7fd04c1ab830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.574+0000 7fd051a00640 1 --2- 172.21.15.31:0/332827406 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd04c10fe30 0x7fd04c1a74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.574+0000 7fd051a00640 1 --2- 172.21.15.31:0/332827406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd04c108be0 0x7fd04c1a6f70 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:17.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.574+0000 7fd051a00640 1 -- 172.21.15.31:0/332827406 >> 172.21.15.31:0/332827406 conn(0x7fd04c06c710 msgr2=0x7fd04c118e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:17.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.574+0000 7fd051a00640 1 -- 172.21.15.31:0/332827406 shutdown_connections 2024-03-21T10:45:17.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:17.574+0000 7fd051a00640 1 -- 172.21.15.31:0/332827406 wait complete. 2024-03-21T10:45:17.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:17 smithi070 ceph-mon[36330]: pgmap v205: 129 pgs: 129 active+clean; 44 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 216 MiB/s wr, 1.85k op/s 2024-03-21T10:45:17.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:17 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/900010738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:45:17.576 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:17 smithi070 ceph-mon[36330]: osdmap e96: 12 total, 12 up, 12 in 2024-03-21T10:45:17.628 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:45:17.628 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-21T10:45:17.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:17 smithi040 ceph-mon[36154]: pgmap v205: 129 pgs: 129 active+clean; 44 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 216 MiB/s wr, 1.85k op/s 2024-03-21T10:45:17.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:17 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/900010738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:45:17.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:17 smithi040 ceph-mon[36154]: osdmap e96: 12 total, 12 up, 12 in 2024-03-21T10:45:18.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.258+0000 7fd243e00640 1 -- 172.21.15.31:0/1845095176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd244102630 msgr2=0x7fd244102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:18.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.258+0000 7fd243e00640 1 --2- 172.21.15.31:0/1845095176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd244102630 0x7fd244102a10 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fd23c009be0 tx=0x7fd23c02bfa0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.270+0000 7fd243e00640 1 -- 172.21.15.31:0/1845095176 shutdown_connections 2024-03-21T10:45:18.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.270+0000 7fd243e00640 1 --2- 172.21.15.31:0/1845095176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd24410ba50 0x7fd24410ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.270+0000 7fd243e00640 1 --2- 172.21.15.31:0/1845095176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd244102f50 0x7fd24410b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.270+0000 7fd243e00640 1 --2- 172.21.15.31:0/1845095176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd244102630 0x7fd244102a10 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.270+0000 7fd243e00640 1 -- 172.21.15.31:0/1845095176 >> 172.21.15.31:0/1845095176 conn(0x7fd244077130 msgr2=0x7fd2440fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:18.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.271+0000 7fd243e00640 1 -- 172.21.15.31:0/1845095176 shutdown_connections 2024-03-21T10:45:18.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.271+0000 7fd243e00640 1 -- 172.21.15.31:0/1845095176 wait complete. 2024-03-21T10:45:18.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.273+0000 7fd243e00640 1 Processor -- start 2024-03-21T10:45:18.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.274+0000 7fd243e00640 1 -- start start 2024-03-21T10:45:18.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.276+0000 7fd243e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd244102630 0x7fd244115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:18.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.277+0000 7fd243e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd244102f50 0x7fd2441161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:18.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.277+0000 7fd243e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd24410ba50 0x7fd24411b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:18.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.277+0000 7fd243e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd244101f50 con 0x7fd244102630 2024-03-21T10:45:18.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.277+0000 7fd243e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd244101dd0 con 0x7fd24410ba50 2024-03-21T10:45:18.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.277+0000 7fd243e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2441020d0 con 0x7fd244102f50 2024-03-21T10:45:18.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.277+0000 7fd243400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd24410ba50 0x7fd24411b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:18.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.277+0000 7fd243400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd24410ba50 0x7fd24411b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56484/0 (socket says 172.21.15.31:56484) 2024-03-21T10:45:18.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.277+0000 7fd243400640 1 -- 172.21.15.31:0/3907562410 learned_addr learned my addr 172.21.15.31:0/3907562410 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:18.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.278+0000 7fd242000640 1 --2- 172.21.15.31:0/3907562410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd244102f50 0x7fd2441161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:18.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.278+0000 7fd242a00640 1 --2- 172.21.15.31:0/3907562410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd244102630 0x7fd244115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:18.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.279+0000 7fd243400640 1 -- 172.21.15.31:0/3907562410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd244102f50 msgr2=0x7fd2441161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:18.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.279+0000 7fd243400640 1 --2- 172.21.15.31:0/3907562410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd244102f50 0x7fd2441161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.279+0000 7fd243400640 1 -- 172.21.15.31:0/3907562410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd244102630 msgr2=0x7fd244115c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:18.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.279+0000 7fd243400640 1 --2- 172.21.15.31:0/3907562410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd244102630 0x7fd244115c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.279+0000 7fd243400640 1 -- 172.21.15.31:0/3907562410 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24411c240 con 0x7fd24410ba50 2024-03-21T10:45:18.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.279+0000 7fd242a00640 1 --2- 172.21.15.31:0/3907562410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd244102630 0x7fd244115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:45:18.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.279+0000 7fd243400640 1 --2- 172.21.15.31:0/3907562410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd24410ba50 0x7fd24411b980 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fd23400b3f0 tx=0x7fd23400b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:18.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.279+0000 7fd23b400640 1 -- 172.21.15.31:0/3907562410 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd234015020 con 0x7fd24410ba50 2024-03-21T10:45:18.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.279+0000 7fd243e00640 1 -- 172.21.15.31:0/3907562410 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd24411c530 con 0x7fd24410ba50 2024-03-21T10:45:18.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.280+0000 7fd243e00640 1 -- 172.21.15.31:0/3907562410 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2441110a0 con 0x7fd24410ba50 2024-03-21T10:45:18.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.279+0000 7fd242000640 1 --2- 172.21.15.31:0/3907562410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd244102f50 0x7fd2441161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:45:18.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.280+0000 7fd23b400640 1 -- 172.21.15.31:0/3907562410 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd234011d10 con 0x7fd24410ba50 2024-03-21T10:45:18.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.280+0000 7fd23b400640 1 -- 172.21.15.31:0/3907562410 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23400f2e0 con 0x7fd24410ba50 2024-03-21T10:45:18.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.291+0000 7fd23b400640 1 -- 172.21.15.31:0/3907562410 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fd234018020 con 0x7fd24410ba50 2024-03-21T10:45:18.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.294+0000 7fd24c76a740 1 Processor -- start 2024-03-21T10:45:18.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.295+0000 7fd24c76a740 1 -- start start 2024-03-21T10:45:18.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.296+0000 7fd23b400640 1 --2- 172.21.15.31:0/3907562410 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd2280a3190 0x7fd2280a5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:18.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.297+0000 7fd23b400640 1 -- 172.21.15.31:0/3907562410 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fd2340d03c0 con 0x7fd24410ba50 2024-03-21T10:45:18.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.298+0000 7fd24c76a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556496bad8b0 0x556496badc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:18.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.299+0000 7fd24c76a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556496bae260 0x556496bb6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:18.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.299+0000 7fd242a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556496bad8b0 0x556496badc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.299+0000 7fd242000640 1 --2- 172.21.15.31:0/3907562410 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd2280a3190 0x7fd2280a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.299+0000 7fd24c76a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556496bb6de0 0x556496bb9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.300+0000 7fd242000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556496bae260 0x556496bb6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.300+0000 7fd242000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556496bae260 0x556496bb6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43456/0 (socket says 172.21.15.31:43456) 2024-03-21T10:45:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.301+0000 7fd242a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556496bad8b0 0x556496badc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34956/0 (socket says 172.21.15.31:34956) 2024-03-21T10:45:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.301+0000 7fd242000640 1 -- 172.21.15.31:0/3143024158 learned_addr learned my addr 172.21.15.31:0/3143024158 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.301+0000 7fd24c76a740 1 -- 172.21.15.31:0/3143024158 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556496b112c0 con 0x556496bae260 2024-03-21T10:45:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.301+0000 7fd24c76a740 1 -- 172.21.15.31:0/3143024158 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556496ab9c90 con 0x556496bb6de0 2024-03-21T10:45:18.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.301+0000 7fd24c76a740 1 -- 172.21.15.31:0/3143024158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x556496b0d360 con 0x556496bad8b0 2024-03-21T10:45:18.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.301+0000 7fd242000640 1 --2- 172.21.15.31:0/3907562410 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd2280a3190 0x7fd2280a5650 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd244112540 tx=0x7fd230009480 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:18.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.302+0000 7fd242a00640 1 -- 172.21.15.31:0/3143024158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556496bb6de0 msgr2=0x556496bb9230 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:45:18.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.302+0000 7fd242a00640 1 --2- 172.21.15.31:0/3143024158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556496bb6de0 0x556496bb9230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.302+0000 7fd242a00640 1 -- 172.21.15.31:0/3143024158 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556496bae260 msgr2=0x556496bb6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:18.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.302+0000 7fd242a00640 1 --2- 172.21.15.31:0/3143024158 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556496bae260 0x556496bb6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.303+0000 7fd242a00640 1 -- 172.21.15.31:0/3143024158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556496ade080 con 0x556496bad8b0 2024-03-21T10:45:18.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.303+0000 7fd242a00640 1 --2- 172.21.15.31:0/3143024158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556496bad8b0 0x556496badc90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd23c006010 tx=0x7fd23c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42896e780f42a189 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:18.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.304+0000 7fd22fe00640 1 -- 172.21.15.31:0/3143024158 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c042070 con 0x556496bad8b0 2024-03-21T10:45:18.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.304+0000 7fd22fe00640 1 -- 172.21.15.31:0/3143024158 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd23c02ce20 con 0x556496bad8b0 2024-03-21T10:45:18.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.305+0000 7fd22fe00640 1 -- 172.21.15.31:0/3143024158 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c042070 con 0x556496bad8b0 2024-03-21T10:45:18.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.307+0000 7fd24c76a740 1 -- 172.21.15.31:0/3143024158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556496bad8b0 msgr2=0x556496badc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:18.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.307+0000 7fd24c76a740 1 --2- 172.21.15.31:0/3143024158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556496bad8b0 0x556496badc90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd23c006010 tx=0x7fd23c009840 comp rx=0 tx=0).stop 2024-03-21T10:45:18.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.313+0000 7fd24c76a740 1 -- 172.21.15.31:0/3143024158 shutdown_connections 2024-03-21T10:45:18.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.313+0000 7fd24c76a740 1 --2- 172.21.15.31:0/3143024158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556496bb6de0 0x556496bb9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.314+0000 7fd24c76a740 1 --2- 172.21.15.31:0/3143024158 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556496bae260 0x556496bb6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.314+0000 7fd24c76a740 1 --2- 172.21.15.31:0/3143024158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556496bad8b0 0x556496badc90 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.314+0000 7fd24c76a740 1 -- 172.21.15.31:0/3143024158 >> 172.21.15.31:0/3143024158 conn(0x556496ba7f80 msgr2=0x556496ba8360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:18.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.314+0000 7fd24c76a740 1 -- 172.21.15.31:0/3143024158 shutdown_connections 2024-03-21T10:45:18.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.314+0000 7fd24c76a740 1 -- 172.21.15.31:0/3143024158 wait complete. 2024-03-21T10:45:18.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.315+0000 7fd24c76a740 1 -- start start 2024-03-21T10:45:18.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.316+0000 7fd24c76a740 1 Processor -- start 2024-03-21T10:45:18.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.317+0000 7fd24c76a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556496bad8b0 0x556496bb6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:18.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.317+0000 7fd24c76a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556496bae260 0x556496bb3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:18.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.317+0000 7fd24c76a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556496bb6de0 0x556496bb3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:18.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.318+0000 7fd24c76a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556496bb9d40 con 0x556496bad8b0 2024-03-21T10:45:18.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.318+0000 7fd24c76a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556496acd0d0 con 0x556496bb6de0 2024-03-21T10:45:18.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.318+0000 7fd24c76a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x556496bb9ec0 con 0x556496bae260 2024-03-21T10:45:18.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.318+0000 7fd242a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556496bad8b0 0x556496bb6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:18.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.318+0000 7fd24c76a740 10 client.0 _ll_register_callbacks cb 0x556496ad1e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:18.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.319+0000 7fd242a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556496bad8b0 0x556496bb6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43468/0 (socket says 172.21.15.31:43468) 2024-03-21T10:45:18.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.319+0000 7fd242a00640 1 -- 172.21.15.31:0/1687623013 learned_addr learned my addr 172.21.15.31:0/1687623013 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:18.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.319+0000 7fd242000640 1 --2- 172.21.15.31:0/1687623013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556496bae260 0x556496bb3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:18.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.319+0000 7fd242a00640 1 -- 172.21.15.31:0/1687623013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556496bae260 msgr2=0x556496bb3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:18.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.320+0000 7fd242a00640 1 --2- 172.21.15.31:0/1687623013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556496bae260 0x556496bb3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.320+0000 7fd242a00640 1 -- 172.21.15.31:0/1687623013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556496bb6de0 msgr2=0x556496bb3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:18.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.320+0000 7fd242a00640 1 --2- 172.21.15.31:0/1687623013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556496bb6de0 0x556496bb3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.320+0000 7fd242a00640 1 -- 172.21.15.31:0/1687623013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5564968253f0 con 0x556496bad8b0 2024-03-21T10:45:18.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.321+0000 7fd242000640 1 --2- 172.21.15.31:0/1687623013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556496bae260 0x556496bb3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:45:18.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.323+0000 7fd242a00640 1 --2- 172.21.15.31:0/1687623013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556496bad8b0 0x556496bb6690 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd23c036040 tx=0x7fd23c02cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d58b4c510254edf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:18.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.323+0000 7fd22cc00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:45:18.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.328+0000 7fd22cc00640 1 -- 172.21.15.31:0/1687623013 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c042040 con 0x556496bad8b0 2024-03-21T10:45:18.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.328+0000 7fd22cc00640 1 -- 172.21.15.31:0/1687623013 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fd23c003e80 con 0x556496bad8b0 2024-03-21T10:45:18.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.328+0000 7fd24c76a740 1 -- 172.21.15.31:0/1687623013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd23c02ce20 con 0x556496bad8b0 2024-03-21T10:45:18.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.345+0000 7fd22cc00640 1 -- 172.21.15.31:0/1687623013 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd23c004080 con 0x556496bad8b0 2024-03-21T10:45:18.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.345+0000 7fd24c76a740 10 client.16710 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:18.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.345+0000 7fd24c76a740 1 -- 172.21.15.31:0/1687623013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5564968ef7d0 con 0x556496bad8b0 2024-03-21T10:45:18.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.346+0000 7fd22cc00640 1 -- 172.21.15.31:0/1687623013 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd23c02ce20 con 0x556496bad8b0 2024-03-21T10:45:18.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.347+0000 7fd22cc00640 1 -- 172.21.15.31:0/1687623013 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd23c047750 con 0x556496bad8b0 2024-03-21T10:45:18.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.350+0000 7fd24c76a740 10 client.16710 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:18.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.350+0000 7fd24c76a740 10 client.16710 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:45:18.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.351+0000 7fd24c76a740 20 client.16710 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:18.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.351+0000 7fd24c76a740 1 --2- 172.21.15.31:0/1687623013 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556496bc6a60 0x556496bc8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:18.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.351+0000 7fd24c76a740 4 client.16710 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:45:18.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.351+0000 7fd24c76a740 1 -- 172.21.15.31:0/1687623013 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556496b1e800 con 0x556496bc6a60 2024-03-21T10:45:18.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.351+0000 7fd242000640 1 --2- 172.21.15.31:0/1687623013 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556496bc6a60 0x556496bc8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:18.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.354+0000 7fd22cc00640 10 client.16710 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:18.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.354+0000 7fd242000640 1 --2- 172.21.15.31:0/1687623013 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556496bc6a60 0x556496bc8ee0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20c6bf9b2d8058b9 server_cookie=22e93ac0f07c2440 in_seq=0 out_seq=0 2024-03-21T10:45:18.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.357+0000 7fd22cc00640 1 -- 172.21.15.31:0/1687623013 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556496b1e800 con 0x556496bc6a60 2024-03-21T10:45:18.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.357+0000 7fd22cc00640 10 client.16710 handle_command_reply: tid=0 2024-03-21T10:45:18.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.358+0000 7fd24c76a740 1 client.16710 shutdown 2024-03-21T10:45:18.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.358+0000 7fd24c76a740 1 -- 172.21.15.31:0/1687623013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556496bad8b0 msgr2=0x556496bb6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:18.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.358+0000 7fd24c76a740 1 --2- 172.21.15.31:0/1687623013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556496bad8b0 0x556496bb6690 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd23c036040 tx=0x7fd23c02cc70 comp rx=0 tx=0).stop 2024-03-21T10:45:18.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.358+0000 7fd24c76a740 1 -- 172.21.15.31:0/1687623013 shutdown_connections 2024-03-21T10:45:18.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.358+0000 7fd24c76a740 1 --2- 172.21.15.31:0/1687623013 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556496bc6a60 0x556496bc8ee0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.359+0000 7fd24c76a740 1 --2- 172.21.15.31:0/1687623013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556496bb6de0 0x556496bb3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.359+0000 7fd24c76a740 1 --2- 172.21.15.31:0/1687623013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556496bae260 0x556496bb3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.359+0000 7fd24c76a740 1 --2- 172.21.15.31:0/1687623013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556496bad8b0 0x556496bb6690 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.359+0000 7fd24c76a740 1 -- 172.21.15.31:0/1687623013 >> 172.21.15.31:0/1687623013 conn(0x556496ba7f80 msgr2=0x556496ba9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:18.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.359+0000 7fd22cc00640 0 client.16710 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:18.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.359+0000 7fd24c76a740 1 -- 172.21.15.31:0/1687623013 shutdown_connections 2024-03-21T10:45:18.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.359+0000 7fd24c76a740 1 -- 172.21.15.31:0/1687623013 wait complete. 2024-03-21T10:45:18.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.360+0000 7fd24c76a740 20 client.16710 trim_cache size 0 max 16384 2024-03-21T10:45:18.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.408+0000 7fd24c76a740 1 Processor -- start 2024-03-21T10:45:18.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.409+0000 7fd24c76a740 1 -- start start 2024-03-21T10:45:18.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.410+0000 7fd24c76a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556496ba7f80 0x556496bce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:18.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.410+0000 7fd24c76a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556496bae260 0x556496bb22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:18.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.410+0000 7fd24c76a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556496bb6de0 0x556496bb6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:18.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.410+0000 7fd24c76a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd224012280 con 0x556496bae260 2024-03-21T10:45:18.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.410+0000 7fd24c76a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22400fd80 con 0x556496ba7f80 2024-03-21T10:45:18.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.410+0000 7fd24c76a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22400de40 con 0x556496bb6de0 2024-03-21T10:45:18.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.411+0000 7fd242a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556496ba7f80 0x556496bce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:18.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.411+0000 7fd242a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556496ba7f80 0x556496bce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56520/0 (socket says 172.21.15.31:56520) 2024-03-21T10:45:18.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.411+0000 7fd242a00640 1 -- 172.21.15.31:0/2035784047 learned_addr learned my addr 172.21.15.31:0/2035784047 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:18.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.411+0000 7fd243400640 1 --2- 172.21.15.31:0/2035784047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556496bb6de0 0x556496bb6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:18.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.411+0000 7fd242000640 1 --2- 172.21.15.31:0/2035784047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556496bae260 0x556496bb22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:18.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.412+0000 7fd243400640 1 -- 172.21.15.31:0/2035784047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556496ba7f80 msgr2=0x556496bce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:18.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.412+0000 7fd243400640 1 --2- 172.21.15.31:0/2035784047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556496ba7f80 0x556496bce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.412+0000 7fd243400640 1 -- 172.21.15.31:0/2035784047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556496bae260 msgr2=0x556496bb22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:18.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.412+0000 7fd243400640 1 --2- 172.21.15.31:0/2035784047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556496bae260 0x556496bb22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.412+0000 7fd243400640 1 -- 172.21.15.31:0/2035784047 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556496bb29d0 con 0x556496bb6de0 2024-03-21T10:45:18.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.413+0000 7fd243400640 1 --2- 172.21.15.31:0/2035784047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556496bb6de0 0x556496bb6150 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fd2340cd820 tx=0x7fd23409c3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c6fd2de913c874a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:18.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.414+0000 7fd22f400640 1 -- 172.21.15.31:0/2035784047 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2340cde60 con 0x556496bb6de0 2024-03-21T10:45:18.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.414+0000 7fd22f400640 1 -- 172.21.15.31:0/2035784047 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd23409c4c0 con 0x556496bb6de0 2024-03-21T10:45:18.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.414+0000 7fd22f400640 1 -- 172.21.15.31:0/2035784047 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2340a0120 con 0x556496bb6de0 2024-03-21T10:45:18.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.415+0000 7fd24c76a740 1 -- 172.21.15.31:0/2035784047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556496bb6de0 msgr2=0x556496bb6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:18.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.415+0000 7fd24c76a740 1 --2- 172.21.15.31:0/2035784047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556496bb6de0 0x556496bb6150 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fd2340cd820 tx=0x7fd23409c3b0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.415+0000 7fd24c76a740 1 -- 172.21.15.31:0/2035784047 shutdown_connections 2024-03-21T10:45:18.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.415+0000 7fd24c76a740 1 --2- 172.21.15.31:0/2035784047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556496bb6de0 0x556496bb6150 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.415+0000 7fd24c76a740 1 --2- 172.21.15.31:0/2035784047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556496bae260 0x556496bb22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.415+0000 7fd24c76a740 1 --2- 172.21.15.31:0/2035784047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556496ba7f80 0x556496bce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.415+0000 7fd24c76a740 1 -- 172.21.15.31:0/2035784047 >> 172.21.15.31:0/2035784047 conn(0x556496bb40c0 msgr2=0x556496ba7160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:18.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.415+0000 7fd24c76a740 1 -- 172.21.15.31:0/2035784047 shutdown_connections 2024-03-21T10:45:18.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.416+0000 7fd24c76a740 1 -- 172.21.15.31:0/2035784047 wait complete. 2024-03-21T10:45:18.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.416+0000 7fd24c76a740 1 -- start start 2024-03-21T10:45:18.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.417+0000 7fd24c76a740 1 Processor -- start 2024-03-21T10:45:18.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.418+0000 7fd24c76a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556496ba7f80 0x556496bcc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:18.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.418+0000 7fd24c76a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556496bae260 0x556496bcc820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:18.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.418+0000 7fd242a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556496ba7f80 0x556496bcc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:18.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.419+0000 7fd242a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556496ba7f80 0x556496bcc2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34988/0 (socket says 172.21.15.31:34988) 2024-03-21T10:45:18.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.419+0000 7fd242a00640 1 -- 172.21.15.31:0/3924560846 learned_addr learned my addr 172.21.15.31:0/3924560846 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:18.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.419+0000 7fd24c76a740 1 --2- 172.21.15.31:0/3924560846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556496bb6de0 0x556496c1b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:18.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.419+0000 7fd24c76a740 1 -- 172.21.15.31:0/3924560846 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22400e440 con 0x556496bb6de0 2024-03-21T10:45:18.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.419+0000 7fd242000640 1 --2- 172.21.15.31:0/3924560846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556496bae260 0x556496bcc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:18.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.420+0000 7fd24c76a740 1 -- 172.21.15.31:0/3924560846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22400f720 con 0x556496bae260 2024-03-21T10:45:18.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.420+0000 7fd24c76a740 1 -- 172.21.15.31:0/3924560846 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22400ea60 con 0x556496ba7f80 2024-03-21T10:45:18.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.420+0000 7fd243400640 1 --2- 172.21.15.31:0/3924560846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556496bb6de0 0x556496c1b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:18.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.421+0000 7fd24c76a740 10 client.0 _ll_register_callbacks cb 0x7fd230009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:18.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.421+0000 7fd242000640 1 -- 172.21.15.31:0/3924560846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556496ba7f80 msgr2=0x556496bcc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:18.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.421+0000 7fd242000640 1 --2- 172.21.15.31:0/3924560846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556496ba7f80 0x556496bcc2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.421+0000 7fd242000640 1 -- 172.21.15.31:0/3924560846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556496bb6de0 msgr2=0x556496c1b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:18.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.421+0000 7fd242000640 1 --2- 172.21.15.31:0/3924560846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556496bb6de0 0x556496c1b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.421+0000 7fd242000640 1 -- 172.21.15.31:0/3924560846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556496bae000 con 0x556496bae260 2024-03-21T10:45:18.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.421+0000 7fd242a00640 1 --2- 172.21.15.31:0/3924560846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556496ba7f80 0x556496bcc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:45:18.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.422+0000 7fd243400640 1 --2- 172.21.15.31:0/3924560846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556496bb6de0 0x556496c1b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:45:18.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.422+0000 7fd22cc00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:45:18.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.422+0000 7fd242000640 1 --2- 172.21.15.31:0/3924560846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556496bae260 0x556496bcc820 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x556496bb3280 tx=0x7fd230009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f08997f3708b4fc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:18.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.423+0000 7fd22cc00640 1 -- 172.21.15.31:0/3924560846 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd230004020 con 0x556496bae260 2024-03-21T10:45:18.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.423+0000 7fd22cc00640 1 -- 172.21.15.31:0/3924560846 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fd2300041c0 con 0x556496bae260 2024-03-21T10:45:18.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.423+0000 7fd24c76a740 1 -- 172.21.15.31:0/3924560846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd23409c4c0 con 0x556496bae260 2024-03-21T10:45:18.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.424+0000 7fd22cc00640 1 -- 172.21.15.31:0/3924560846 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd230038050 con 0x556496bae260 2024-03-21T10:45:18.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.424+0000 7fd24c76a740 10 client.27272 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:18.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.424+0000 7fd24c76a740 1 -- 172.21.15.31:0/3924560846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x556496c22a90 con 0x556496bae260 2024-03-21T10:45:18.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.424+0000 7fd22cc00640 1 -- 172.21.15.31:0/3924560846 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd23409c4c0 con 0x556496bae260 2024-03-21T10:45:18.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.425+0000 7fd22cc00640 1 -- 172.21.15.31:0/3924560846 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd2300285b0 con 0x556496bae260 2024-03-21T10:45:18.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.425+0000 7fd24c76a740 10 client.27272 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:18.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.425+0000 7fd24c76a740 10 client.27272 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:45:18.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.425+0000 7fd24c76a740 20 client.27272 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:18.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.425+0000 7fd24c76a740 1 --2- 172.21.15.31:0/3924560846 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556496c23170 0x556496c25590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:18.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.425+0000 7fd24c76a740 4 client.27272 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:45:18.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.425+0000 7fd24c76a740 1 -- 172.21.15.31:0/3924560846 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556496b18b50 con 0x556496c23170 2024-03-21T10:45:18.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.426+0000 7fd242a00640 1 --2- 172.21.15.31:0/3924560846 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556496c23170 0x556496c25590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:18.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.427+0000 7fd22cc00640 10 client.27272 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:18.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.428+0000 7fd242a00640 1 --2- 172.21.15.31:0/3924560846 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556496c23170 0x556496c25590 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=212a11179ce89c9c server_cookie=f5e054a5de2d3854 in_seq=0 out_seq=0 2024-03-21T10:45:18.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.428+0000 7fd22cc00640 1 -- 172.21.15.31:0/3924560846 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556496b18b50 con 0x556496c23170 2024-03-21T10:45:18.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.428+0000 7fd22cc00640 10 client.27272 handle_command_reply: tid=0 2024-03-21T10:45:18.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.428+0000 7fd24c76a740 1 client.27272 shutdown 2024-03-21T10:45:18.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.429+0000 7fd24c76a740 1 -- 172.21.15.31:0/3924560846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556496bae260 msgr2=0x556496bcc820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:18.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.429+0000 7fd24c76a740 1 --2- 172.21.15.31:0/3924560846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556496bae260 0x556496bcc820 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x556496bb3280 tx=0x7fd230009260 comp rx=0 tx=0).stop 2024-03-21T10:45:18.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.429+0000 7fd24c76a740 1 -- 172.21.15.31:0/3924560846 shutdown_connections 2024-03-21T10:45:18.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.429+0000 7fd24c76a740 1 --2- 172.21.15.31:0/3924560846 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556496c23170 0x556496c25590 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.429+0000 7fd24c76a740 1 --2- 172.21.15.31:0/3924560846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556496bb6de0 0x556496c1b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.429+0000 7fd24c76a740 1 --2- 172.21.15.31:0/3924560846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556496bae260 0x556496bcc820 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.429+0000 7fd24c76a740 1 --2- 172.21.15.31:0/3924560846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556496ba7f80 0x556496bcc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.429+0000 7fd24c76a740 1 -- 172.21.15.31:0/3924560846 >> 172.21.15.31:0/3924560846 conn(0x556496bb40c0 msgr2=0x556496ba7160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:18.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.429+0000 7fd22cc00640 0 client.27272 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:18.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.433+0000 7fd24c76a740 1 -- 172.21.15.31:0/3924560846 shutdown_connections 2024-03-21T10:45:18.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.433+0000 7fd24c76a740 1 -- 172.21.15.31:0/3924560846 wait complete. 2024-03-21T10:45:18.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.433+0000 7fd24c76a740 20 client.27272 trim_cache size 0 max 16384 2024-03-21T10:45:18.431 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:45:18.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.439+0000 7fd243e00640 1 -- 172.21.15.31:0/3907562410 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd2280a3190 msgr2=0x7fd2280a5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:18.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.439+0000 7fd243e00640 1 --2- 172.21.15.31:0/3907562410 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd2280a3190 0x7fd2280a5650 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd244112540 tx=0x7fd230009480 comp rx=0 tx=0).stop 2024-03-21T10:45:18.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.439+0000 7fd243e00640 1 -- 172.21.15.31:0/3907562410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd24410ba50 msgr2=0x7fd24411b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:18.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.439+0000 7fd243e00640 1 --2- 172.21.15.31:0/3907562410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd24410ba50 0x7fd24411b980 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fd23400b3f0 tx=0x7fd23400b8b0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.439+0000 7fd243e00640 1 -- 172.21.15.31:0/3907562410 shutdown_connections 2024-03-21T10:45:18.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.439+0000 7fd243e00640 1 --2- 172.21.15.31:0/3907562410 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd2280a3190 0x7fd2280a5650 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.439+0000 7fd243e00640 1 --2- 172.21.15.31:0/3907562410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd24410ba50 0x7fd24411b980 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.439+0000 7fd243e00640 1 --2- 172.21.15.31:0/3907562410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd244102f50 0x7fd2441161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.439+0000 7fd243e00640 1 --2- 172.21.15.31:0/3907562410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd244102630 0x7fd244115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:18.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.439+0000 7fd243e00640 1 -- 172.21.15.31:0/3907562410 >> 172.21.15.31:0/3907562410 conn(0x7fd244077130 msgr2=0x7fd2440fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:18.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.440+0000 7fd243e00640 1 -- 172.21.15.31:0/3907562410 shutdown_connections 2024-03-21T10:45:18.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:18.440+0000 7fd243e00640 1 -- 172.21.15.31:0/3907562410 wait complete. 2024-03-21T10:45:19.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.037+0000 7fd5e1800640 1 -- 172.21.15.31:0/749082684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5dc107f40 msgr2=0x7fd5dc108320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:19.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.037+0000 7fd5e1800640 1 --2- 172.21.15.31:0/749082684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5dc107f40 0x7fd5dc108320 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4009be0 tx=0x7fd5d402bef0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.039+0000 7fd5e1800640 1 -- 172.21.15.31:0/749082684 shutdown_connections 2024-03-21T10:45:19.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.039+0000 7fd5e1800640 1 --2- 172.21.15.31:0/749082684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5dc10cfc0 0x7fd5dc114b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.039+0000 7fd5e1800640 1 --2- 172.21.15.31:0/749082684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5dc1088f0 0x7fd5dc10c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.039+0000 7fd5e1800640 1 --2- 172.21.15.31:0/749082684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5dc107f40 0x7fd5dc108320 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.039+0000 7fd5e1800640 1 -- 172.21.15.31:0/749082684 >> 172.21.15.31:0/749082684 conn(0x7fd5dc0774e0 msgr2=0x7fd5dc1064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:19.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.040+0000 7fd5e1800640 1 -- 172.21.15.31:0/749082684 shutdown_connections 2024-03-21T10:45:19.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.040+0000 7fd5e1800640 1 -- 172.21.15.31:0/749082684 wait complete. 2024-03-21T10:45:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.041+0000 7fd5e1800640 1 Processor -- start 2024-03-21T10:45:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.041+0000 7fd5e1800640 1 -- start start 2024-03-21T10:45:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.042+0000 7fd5e1800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5dc107f40 0x7fd5dc081a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.042+0000 7fd5daa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5dc107f40 0x7fd5dc081a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.043+0000 7fd5daa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5dc107f40 0x7fd5dc081a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:35016/0 (socket says 172.21.15.31:35016) 2024-03-21T10:45:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.043+0000 7fd5e1800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5dc10cfc0 0x7fd5dc081f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.043+0000 7fd5e1800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5dc07ab20 0x7fd5dc07afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.043+0000 7fd5e1800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5dc1180b0 con 0x7fd5dc10cfc0 2024-03-21T10:45:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.043+0000 7fd5e1800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5dc117f30 con 0x7fd5dc07ab20 2024-03-21T10:45:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.043+0000 7fd5e1800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5dc118230 con 0x7fd5dc107f40 2024-03-21T10:45:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.045+0000 7fd5daa00640 1 -- 172.21.15.31:0/3232613592 learned_addr learned my addr 172.21.15.31:0/3232613592 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.045+0000 7fd5daa00640 1 -- 172.21.15.31:0/3232613592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5dc07ab20 msgr2=0x7fd5dc07afd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T10:45:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.045+0000 7fd5daa00640 1 --2- 172.21.15.31:0/3232613592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5dc07ab20 0x7fd5dc07afd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.045+0000 7fd5daa00640 1 -- 172.21.15.31:0/3232613592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5dc10cfc0 msgr2=0x7fd5dc081f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.045+0000 7fd5daa00640 1 --2- 172.21.15.31:0/3232613592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5dc10cfc0 0x7fd5dc081f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.045+0000 7fd5daa00640 1 -- 172.21.15.31:0/3232613592 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5dc07b770 con 0x7fd5dc107f40 2024-03-21T10:45:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.046+0000 7fd5daa00640 1 --2- 172.21.15.31:0/3232613592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5dc107f40 0x7fd5dc081a50 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4004af0 tx=0x7fd5d402c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.047+0000 7fd5d3400640 1 -- 172.21.15.31:0/3232613592 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d4054020 con 0x7fd5dc107f40 2024-03-21T10:45:19.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.047+0000 7fd5d3400640 1 -- 172.21.15.31:0/3232613592 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5d4004040 con 0x7fd5dc107f40 2024-03-21T10:45:19.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.047+0000 7fd5d3400640 1 -- 172.21.15.31:0/3232613592 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d4054020 con 0x7fd5dc107f40 2024-03-21T10:45:19.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.047+0000 7fd5e1800640 1 -- 172.21.15.31:0/3232613592 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5dc07f570 con 0x7fd5dc107f40 2024-03-21T10:45:19.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.048+0000 7fd5e1800640 1 -- 172.21.15.31:0/3232613592 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5dc07fa70 con 0x7fd5dc107f40 2024-03-21T10:45:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.054+0000 7fd5e4607740 1 Processor -- start 2024-03-21T10:45:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.055+0000 7fd5e4607740 1 -- start start 2024-03-21T10:45:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.055+0000 7fd5e4607740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605db05ffd0 0x5605db0603b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.057+0000 7fd5d3400640 1 -- 172.21.15.31:0/3232613592 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fd5d4057020 con 0x7fd5dc107f40 2024-03-21T10:45:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.057+0000 7fd5e4607740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5605db0608f0 0x5605db069180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.057+0000 7fd5e4607740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605db0696c0 0x5605db06bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.057+0000 7fd5e4607740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605db0223b0 con 0x5605db0696c0 2024-03-21T10:45:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.057+0000 7fd5e4607740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605dafc4830 con 0x5605db0608f0 2024-03-21T10:45:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.057+0000 7fd5e4607740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605daf6d5d0 con 0x5605db05ffd0 2024-03-21T10:45:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.058+0000 7fd5db400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605db05ffd0 0x5605db0603b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.058+0000 7fd5db400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605db05ffd0 0x5605db0603b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:35026/0 (socket says 172.21.15.31:35026) 2024-03-21T10:45:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.058+0000 7fd5db400640 1 -- 172.21.15.31:0/1321454559 learned_addr learned my addr 172.21.15.31:0/1321454559 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.058+0000 7fd5daa00640 1 --2- 172.21.15.31:0/1321454559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5605db0608f0 0x5605db069180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.058+0000 7fd5da000640 1 --2- 172.21.15.31:0/1321454559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605db0696c0 0x5605db06bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.059+0000 7fd5db400640 1 -- 172.21.15.31:0/1321454559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5605db0608f0 msgr2=0x5605db069180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:19.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.059+0000 7fd5db400640 1 --2- 172.21.15.31:0/1321454559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5605db0608f0 0x5605db069180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.059+0000 7fd5db400640 1 -- 172.21.15.31:0/1321454559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605db0696c0 msgr2=0x5605db06bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:19.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.059+0000 7fd5db400640 1 --2- 172.21.15.31:0/1321454559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605db0696c0 0x5605db06bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.059+0000 7fd5db400640 1 -- 172.21.15.31:0/1321454559 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605daf852c0 con 0x5605db05ffd0 2024-03-21T10:45:19.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.059+0000 7fd5db400640 1 --2- 172.21.15.31:0/1321454559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605db05ffd0 0x5605db0603b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fd5c800c900 tx=0x7fd5c800cdc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70df328b5763bad9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:19.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.064+0000 7fd5c7e00640 1 -- 172.21.15.31:0/1321454559 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c8004020 con 0x5605db05ffd0 2024-03-21T10:45:19.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.064+0000 7fd5c7e00640 1 -- 172.21.15.31:0/1321454559 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5c800ee70 con 0x5605db05ffd0 2024-03-21T10:45:19.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.064+0000 7fd5c7e00640 1 -- 172.21.15.31:0/1321454559 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c8003cd0 con 0x5605db05ffd0 2024-03-21T10:45:19.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.065+0000 7fd5e4607740 1 -- 172.21.15.31:0/1321454559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605db05ffd0 msgr2=0x5605db0603b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:19.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.065+0000 7fd5e4607740 1 --2- 172.21.15.31:0/1321454559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605db05ffd0 0x5605db0603b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fd5c800c900 tx=0x7fd5c800cdc0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.065+0000 7fd5e4607740 1 -- 172.21.15.31:0/1321454559 shutdown_connections 2024-03-21T10:45:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.065+0000 7fd5e4607740 1 --2- 172.21.15.31:0/1321454559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605db0696c0 0x5605db06bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.065+0000 7fd5e4607740 1 --2- 172.21.15.31:0/1321454559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5605db0608f0 0x5605db069180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.065+0000 7fd5e4607740 1 --2- 172.21.15.31:0/1321454559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605db05ffd0 0x5605db0603b0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.065+0000 7fd5e4607740 1 -- 172.21.15.31:0/1321454559 >> 172.21.15.31:0/1321454559 conn(0x5605db05a8f0 msgr2=0x5605db05acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.069+0000 7fd5e4607740 1 -- 172.21.15.31:0/1321454559 shutdown_connections 2024-03-21T10:45:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.069+0000 7fd5e4607740 1 -- 172.21.15.31:0/1321454559 wait complete. 2024-03-21T10:45:19.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.069+0000 7fd5e4607740 1 -- start start 2024-03-21T10:45:19.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.070+0000 7fd5e4607740 1 Processor -- start 2024-03-21T10:45:19.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.070+0000 7fd5d3400640 1 --2- 172.21.15.31:0/3232613592 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd5c00a3330 0x7fd5c00a57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.070+0000 7fd5d3400640 1 -- 172.21.15.31:0/3232613592 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fd5d4102e40 con 0x7fd5dc107f40 2024-03-21T10:45:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.075+0000 7fd5e4607740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5605db05ffd0 0x5605db066700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.075+0000 7fd5db400640 1 --2- 172.21.15.31:0/3232613592 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd5c00a3330 0x7fd5c00a57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.076+0000 7fd5e4607740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605db0608f0 0x5605db066c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.076+0000 7fd5daa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5605db05ffd0 0x5605db066700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.076+0000 7fd5daa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5605db05ffd0 0x5605db066700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56566/0 (socket says 172.21.15.31:56566) 2024-03-21T10:45:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.076+0000 7fd5e4607740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605db0696c0 0x5605db067180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.076+0000 7fd5e4607740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605daf80a20 con 0x5605db0608f0 2024-03-21T10:45:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.076+0000 7fd5e4607740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605dafc08d0 con 0x5605db05ffd0 2024-03-21T10:45:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.076+0000 7fd5e4607740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605db06c2f0 con 0x5605db0696c0 2024-03-21T10:45:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.076+0000 7fd5daa00640 1 -- 172.21.15.31:0/496882936 learned_addr learned my addr 172.21.15.31:0/496882936 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.076+0000 7fd5e4607740 10 client.0 _ll_register_callbacks cb 0x5605dafbd1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.076+0000 7fd5daa00640 1 -- 172.21.15.31:0/496882936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605db0696c0 msgr2=0x5605db067180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:45:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.076+0000 7fd5daa00640 1 --2- 172.21.15.31:0/496882936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605db0696c0 0x5605db067180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.076+0000 7fd5daa00640 1 -- 172.21.15.31:0/496882936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605db0608f0 msgr2=0x5605db066c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.076+0000 7fd5daa00640 1 --2- 172.21.15.31:0/496882936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605db0608f0 0x5605db066c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.076+0000 7fd5daa00640 1 -- 172.21.15.31:0/496882936 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5605dacd9210 con 0x5605db05ffd0 2024-03-21T10:45:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.077+0000 7fd5daa00640 1 --2- 172.21.15.31:0/496882936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5605db05ffd0 0x5605db066700 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4038040 tx=0x7fd5d40098b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d5b5599793613b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.077+0000 7fd5c4c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:45:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.077+0000 7fd5c4c00640 1 -- 172.21.15.31:0/496882936 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d40cea80 con 0x5605db05ffd0 2024-03-21T10:45:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.078+0000 7fd5c4c00640 1 -- 172.21.15.31:0/496882936 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fd5d404e3c0 con 0x5605db05ffd0 2024-03-21T10:45:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.078+0000 7fd5e4607740 1 -- 172.21.15.31:0/496882936 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd5c800ee70 con 0x5605db05ffd0 2024-03-21T10:45:19.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.081+0000 7fd5c4c00640 1 -- 172.21.15.31:0/496882936 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5d40ce8a0 con 0x5605db05ffd0 2024-03-21T10:45:19.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.081+0000 7fd5e4607740 10 client.27281 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:19.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.081+0000 7fd5e4607740 1 -- 172.21.15.31:0/496882936 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5605dada5620 con 0x5605db05ffd0 2024-03-21T10:45:19.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.081+0000 7fd5c4c00640 1 -- 172.21.15.31:0/496882936 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5c800ee70 con 0x5605db05ffd0 2024-03-21T10:45:19.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.082+0000 7fd5c4c00640 1 -- 172.21.15.31:0/496882936 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd5d4061060 con 0x5605db05ffd0 2024-03-21T10:45:19.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.082+0000 7fd5e4607740 10 client.27281 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:19.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.082+0000 7fd5e4607740 10 client.27281 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:45:19.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.082+0000 7fd5e4607740 20 client.27281 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:19.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.082+0000 7fd5e4607740 1 --2- 172.21.15.31:0/496882936 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5605db079790 0x5605db07bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.083+0000 7fd5e4607740 4 client.27281 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:45:19.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.083+0000 7fd5e4607740 1 -- 172.21.15.31:0/496882936 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5605dafd1d70 con 0x5605db079790 2024-03-21T10:45:19.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.083+0000 7fd5da000640 1 --2- 172.21.15.31:0/496882936 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5605db079790 0x5605db07bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.085+0000 7fd5da000640 1 --2- 172.21.15.31:0/496882936 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5605db079790 0x5605db07bc50 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d58c18d8eb39a543 server_cookie=f46c919935e76fba in_seq=0 out_seq=0 2024-03-21T10:45:19.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.085+0000 7fd5c4c00640 10 client.27281 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:19.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.086+0000 7fd5c4c00640 1 -- 172.21.15.31:0/496882936 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5605dafd1d70 con 0x5605db079790 2024-03-21T10:45:19.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.086+0000 7fd5c4c00640 10 client.27281 handle_command_reply: tid=0 2024-03-21T10:45:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.086+0000 7fd5db400640 1 --2- 172.21.15.31:0/3232613592 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd5c00a3330 0x7fd5c00a57f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd5c800d010 tx=0x7fd5c8003810 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.086+0000 7fd5e4607740 1 client.27281 shutdown 2024-03-21T10:45:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.086+0000 7fd5e4607740 1 -- 172.21.15.31:0/496882936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5605db05ffd0 msgr2=0x5605db066700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.086+0000 7fd5e4607740 1 --2- 172.21.15.31:0/496882936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5605db05ffd0 0x5605db066700 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4038040 tx=0x7fd5d40098b0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.086+0000 7fd5e4607740 1 -- 172.21.15.31:0/496882936 shutdown_connections 2024-03-21T10:45:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.086+0000 7fd5e4607740 1 --2- 172.21.15.31:0/496882936 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5605db079790 0x5605db07bc50 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.087+0000 7fd5e4607740 1 --2- 172.21.15.31:0/496882936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605db0696c0 0x5605db067180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.087+0000 7fd5e4607740 1 --2- 172.21.15.31:0/496882936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605db0608f0 0x5605db066c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.087+0000 7fd5e4607740 1 --2- 172.21.15.31:0/496882936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5605db05ffd0 0x5605db066700 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.087+0000 7fd5e4607740 1 -- 172.21.15.31:0/496882936 >> 172.21.15.31:0/496882936 conn(0x5605db05a8f0 msgr2=0x5605db05c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:19.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.087+0000 7fd5c4c00640 0 client.27281 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:19.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.087+0000 7fd5e4607740 1 -- 172.21.15.31:0/496882936 shutdown_connections 2024-03-21T10:45:19.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.087+0000 7fd5e4607740 1 -- 172.21.15.31:0/496882936 wait complete. 2024-03-21T10:45:19.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.087+0000 7fd5e4607740 20 client.27281 trim_cache size 0 max 16384 2024-03-21T10:45:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.151+0000 7fd5e4607740 1 Processor -- start 2024-03-21T10:45:19.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.155+0000 7fd5e4607740 1 -- start start 2024-03-21T10:45:19.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.155+0000 7fd5e4607740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605db05ffd0 0x5605db068e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.155+0000 7fd5e4607740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605db0608f0 0x5605db074a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.155+0000 7fd5e4607740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5605db0696c0 0x5605db074fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.155+0000 7fd5e4607740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc012110 con 0x5605db0608f0 2024-03-21T10:45:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.155+0000 7fd5e4607740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc00fc10 con 0x5605db0696c0 2024-03-21T10:45:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.155+0000 7fd5e4607740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc00dd60 con 0x5605db05ffd0 2024-03-21T10:45:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.155+0000 7fd5daa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605db05ffd0 0x5605db068e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.156+0000 7fd5daa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605db05ffd0 0x5605db068e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:35028/0 (socket says 172.21.15.31:35028) 2024-03-21T10:45:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.156+0000 7fd5daa00640 1 -- 172.21.15.31:0/796758461 learned_addr learned my addr 172.21.15.31:0/796758461 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.156+0000 7fd5da000640 1 --2- 172.21.15.31:0/796758461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605db0608f0 0x5605db074a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.156+0000 7fd5db400640 1 --2- 172.21.15.31:0/796758461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5605db0696c0 0x5605db074fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.157+0000 7fd5db400640 1 -- 172.21.15.31:0/796758461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605db05ffd0 msgr2=0x5605db068e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.157+0000 7fd5db400640 1 --2- 172.21.15.31:0/796758461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605db05ffd0 0x5605db068e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.157+0000 7fd5db400640 1 -- 172.21.15.31:0/796758461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605db0608f0 msgr2=0x5605db074a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.157+0000 7fd5db400640 1 --2- 172.21.15.31:0/796758461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605db0608f0 0x5605db074a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.158+0000 7fd5db400640 1 -- 172.21.15.31:0/796758461 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605db064c90 con 0x5605db0696c0 2024-03-21T10:45:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.162+0000 7fd5db400640 1 --2- 172.21.15.31:0/796758461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5605db0696c0 0x5605db074fa0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd5c80039b0 tx=0x7fd5c8021000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fe114f744cd152c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.162+0000 7fd5c7400640 1 -- 172.21.15.31:0/796758461 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c80118c0 con 0x5605db0696c0 2024-03-21T10:45:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.162+0000 7fd5c7400640 1 -- 172.21.15.31:0/796758461 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5c8052470 con 0x5605db0696c0 2024-03-21T10:45:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.162+0000 7fd5c7400640 1 -- 172.21.15.31:0/796758461 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c8024640 con 0x5605db0696c0 2024-03-21T10:45:19.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.163+0000 7fd5e4607740 1 -- 172.21.15.31:0/796758461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5605db0696c0 msgr2=0x5605db074fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:19.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.163+0000 7fd5e4607740 1 --2- 172.21.15.31:0/796758461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5605db0696c0 0x5605db074fa0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd5c80039b0 tx=0x7fd5c8021000 comp rx=0 tx=0).stop 2024-03-21T10:45:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.176+0000 7fd5e4607740 1 -- 172.21.15.31:0/796758461 shutdown_connections 2024-03-21T10:45:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.176+0000 7fd5e4607740 1 --2- 172.21.15.31:0/796758461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5605db0696c0 0x5605db074fa0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.176+0000 7fd5e4607740 1 --2- 172.21.15.31:0/796758461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605db0608f0 0x5605db074a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.176+0000 7fd5e4607740 1 --2- 172.21.15.31:0/796758461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605db05ffd0 0x5605db068e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.176+0000 7fd5e4607740 1 -- 172.21.15.31:0/796758461 >> 172.21.15.31:0/796758461 conn(0x5605db05a8f0 msgr2=0x5605db05c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:19.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.178+0000 7fd5e4607740 1 -- 172.21.15.31:0/796758461 shutdown_connections 2024-03-21T10:45:19.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.179+0000 7fd5e4607740 1 -- 172.21.15.31:0/796758461 wait complete. 2024-03-21T10:45:19.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.179+0000 7fd5e4607740 1 -- start start 2024-03-21T10:45:19.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.182+0000 7fd5e4607740 1 Processor -- start 2024-03-21T10:45:19.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.183+0000 7fd5e4607740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5605db05ffd0 0x5605db0ce3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.184+0000 7fd5daa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5605db05ffd0 0x5605db0ce3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.184+0000 7fd5daa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5605db05ffd0 0x5605db0ce3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56582/0 (socket says 172.21.15.31:56582) 2024-03-21T10:45:19.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.184+0000 7fd5e4607740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605db0608f0 0x5605db0ce910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.184+0000 7fd5daa00640 1 -- 172.21.15.31:0/2794829677 learned_addr learned my addr 172.21.15.31:0/2794829677 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:19.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.184+0000 7fd5da000640 1 --2- 172.21.15.31:0/2794829677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605db0608f0 0x5605db0ce910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.185+0000 7fd5e4607740 1 --2- 172.21.15.31:0/2794829677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605db0696c0 0x5605db063520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.186+0000 7fd5e4607740 1 -- 172.21.15.31:0/2794829677 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc00e2d0 con 0x5605db0696c0 2024-03-21T10:45:19.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.188+0000 7fd5e4607740 1 -- 172.21.15.31:0/2794829677 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc00f5b0 con 0x5605db05ffd0 2024-03-21T10:45:19.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.188+0000 7fd5e4607740 1 -- 172.21.15.31:0/2794829677 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc00e8f0 con 0x5605db0608f0 2024-03-21T10:45:19.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.188+0000 7fd5e4607740 10 client.0 _ll_register_callbacks cb 0x7fd5d4005f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:19.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.189+0000 7fd5db400640 1 --2- 172.21.15.31:0/2794829677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605db0696c0 0x5605db063520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.193+0000 7fd5daa00640 1 -- 172.21.15.31:0/2794829677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605db0608f0 msgr2=0x5605db0ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.193+0000 7fd5daa00640 1 --2- 172.21.15.31:0/2794829677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605db0608f0 0x5605db0ce910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.193+0000 7fd5daa00640 1 -- 172.21.15.31:0/2794829677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605db0696c0 msgr2=0x5605db063520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:19.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.193+0000 7fd5daa00640 1 --2- 172.21.15.31:0/2794829677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605db0696c0 0x5605db063520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.193+0000 7fd5daa00640 1 -- 172.21.15.31:0/2794829677 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5605db063c60 con 0x5605db05ffd0 2024-03-21T10:45:19.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.193+0000 7fd5daa00640 1 --2- 172.21.15.31:0/2794829677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5605db05ffd0 0x5605db0ce3d0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fd5d40cb3a0 tx=0x7fd5d41065d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3da9300966b2971 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:19.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.194+0000 7fd5c4c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:45:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.194+0000 7fd5c4c00640 1 -- 172.21.15.31:0/2794829677 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d40cea80 con 0x5605db05ffd0 2024-03-21T10:45:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.194+0000 7fd5c4c00640 1 -- 172.21.15.31:0/2794829677 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fd5d40ed900 con 0x5605db05ffd0 2024-03-21T10:45:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.194+0000 7fd5e4607740 1 -- 172.21.15.31:0/2794829677 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd5c8052470 con 0x5605db05ffd0 2024-03-21T10:45:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.195+0000 7fd5c4c00640 1 -- 172.21.15.31:0/2794829677 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5d40e8040 con 0x5605db05ffd0 2024-03-21T10:45:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.195+0000 7fd5e4607740 10 client.27287 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.195+0000 7fd5e4607740 1 -- 172.21.15.31:0/2794829677 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5605db063c60 con 0x5605db05ffd0 2024-03-21T10:45:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.195+0000 7fd5c4c00640 1 -- 172.21.15.31:0/2794829677 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5c8052470 con 0x5605db05ffd0 2024-03-21T10:45:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.195+0000 7fd5c4c00640 1 -- 172.21.15.31:0/2794829677 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd5d4004b80 con 0x5605db05ffd0 2024-03-21T10:45:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.199+0000 7fd5e4607740 10 client.27287 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.199+0000 7fd5e4607740 10 client.27287 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:45:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.199+0000 7fd5e4607740 20 client.27287 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.199+0000 7fd5e4607740 1 --2- 172.21.15.31:0/2794829677 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5605db0d5e40 0x5605db0d8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.199+0000 7fd5e4607740 4 client.27287 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:45:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.199+0000 7fd5e4607740 1 -- 172.21.15.31:0/2794829677 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5605dafcc0c0 con 0x5605db0d5e40 2024-03-21T10:45:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.199+0000 7fd5da000640 1 --2- 172.21.15.31:0/2794829677 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5605db0d5e40 0x5605db0d8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.200+0000 7fd5da000640 1 --2- 172.21.15.31:0/2794829677 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5605db0d5e40 0x5605db0d8230 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e968e710269b2c0 server_cookie=98e23911f49b1d91 in_seq=0 out_seq=0 2024-03-21T10:45:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.200+0000 7fd5c4c00640 10 client.27287 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:19.481 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-21T10:45:19.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:45:19.481 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-21T10:45:19.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.499+0000 7fd5c4c00640 1 -- 172.21.15.31:0/2794829677 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5605dafcc0c0 con 0x5605db0d5e40 2024-03-21T10:45:19.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.499+0000 7fd5c4c00640 10 client.27287 handle_command_reply: tid=0 2024-03-21T10:45:19.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.499+0000 7fd5e4607740 1 client.27287 shutdown 2024-03-21T10:45:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.500+0000 7fd5e4607740 1 -- 172.21.15.31:0/2794829677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5605db05ffd0 msgr2=0x5605db0ce3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.500+0000 7fd5e4607740 1 --2- 172.21.15.31:0/2794829677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5605db05ffd0 0x5605db0ce3d0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fd5d40cb3a0 tx=0x7fd5d41065d0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.500+0000 7fd5e4607740 1 -- 172.21.15.31:0/2794829677 shutdown_connections 2024-03-21T10:45:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.500+0000 7fd5e4607740 1 --2- 172.21.15.31:0/2794829677 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5605db0d5e40 0x5605db0d8230 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.501+0000 7fd5e4607740 1 --2- 172.21.15.31:0/2794829677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605db0696c0 0x5605db063520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.501+0000 7fd5e4607740 1 --2- 172.21.15.31:0/2794829677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605db0608f0 0x5605db0ce910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.501+0000 7fd5e4607740 1 --2- 172.21.15.31:0/2794829677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5605db05ffd0 0x5605db0ce3d0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.501+0000 7fd5e4607740 1 -- 172.21.15.31:0/2794829677 >> 172.21.15.31:0/2794829677 conn(0x5605db05a8f0 msgr2=0x5605db0590c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.501+0000 7fd5c4c00640 0 client.27287 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.501+0000 7fd5e4607740 1 -- 172.21.15.31:0/2794829677 shutdown_connections 2024-03-21T10:45:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.502+0000 7fd5e4607740 1 -- 172.21.15.31:0/2794829677 wait complete. 2024-03-21T10:45:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.502+0000 7fd5e4607740 20 client.27287 trim_cache size 0 max 16384 2024-03-21T10:45:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:45:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:45:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:45:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:45:19.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.504+0000 7fd5e1800640 1 -- 172.21.15.31:0/3232613592 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd5c00a3330 msgr2=0x7fd5c00a57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:19.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.504+0000 7fd5e1800640 1 --2- 172.21.15.31:0/3232613592 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd5c00a3330 0x7fd5c00a57f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd5c800d010 tx=0x7fd5c8003810 comp rx=0 tx=0).stop 2024-03-21T10:45:19.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.505+0000 7fd5e1800640 1 -- 172.21.15.31:0/3232613592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5dc107f40 msgr2=0x7fd5dc081a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:19.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.505+0000 7fd5e1800640 1 --2- 172.21.15.31:0/3232613592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5dc107f40 0x7fd5dc081a50 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4004af0 tx=0x7fd5d402c910 comp rx=0 tx=0).stop 2024-03-21T10:45:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.507+0000 7fd5e1800640 1 -- 172.21.15.31:0/3232613592 shutdown_connections 2024-03-21T10:45:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.507+0000 7fd5e1800640 1 --2- 172.21.15.31:0/3232613592 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd5c00a3330 0x7fd5c00a57f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.507+0000 7fd5e1800640 1 --2- 172.21.15.31:0/3232613592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5dc07ab20 0x7fd5dc07afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.507+0000 7fd5e1800640 1 --2- 172.21.15.31:0/3232613592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5dc10cfc0 0x7fd5dc081f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.507+0000 7fd5e1800640 1 --2- 172.21.15.31:0/3232613592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5dc107f40 0x7fd5dc081a50 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.507+0000 7fd5e1800640 1 -- 172.21.15.31:0/3232613592 >> 172.21.15.31:0/3232613592 conn(0x7fd5dc0774e0 msgr2=0x7fd5dc0778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:19.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.507+0000 7fd5e1800640 1 -- 172.21.15.31:0/3232613592 shutdown_connections 2024-03-21T10:45:19.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:19.508+0000 7fd5e1800640 1 -- 172.21.15.31:0/3232613592 wait complete. 2024-03-21T10:45:19.557 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:19 smithi031 ceph-mon[31673]: pgmap v207: 129 pgs: 129 active+clean; 44 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 208 MiB/s wr, 1.81k op/s 2024-03-21T10:45:19.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:19 smithi070 ceph-mon[36330]: pgmap v207: 129 pgs: 129 active+clean; 44 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 208 MiB/s wr, 1.81k op/s 2024-03-21T10:45:19.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:19 smithi040 ceph-mon[36154]: pgmap v207: 129 pgs: 129 active+clean; 44 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 208 MiB/s wr, 1.81k op/s 2024-03-21T10:45:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.893+0000 7f1f8dc00640 1 -- 172.21.15.31:0/3070554405 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f8810e310 msgr2=0x7f1f8811ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.893+0000 7f1f8dc00640 1 --2- 172.21.15.31:0/3070554405 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f8810e310 0x7f1f8811ce90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1f80009c10 tx=0x7f1f8002bef0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.894+0000 7f1f8dc00640 1 -- 172.21.15.31:0/3070554405 shutdown_connections 2024-03-21T10:45:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.894+0000 7f1f8dc00640 1 --2- 172.21.15.31:0/3070554405 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f8810e310 0x7f1f8811ce90 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.894+0000 7f1f8dc00640 1 --2- 172.21.15.31:0/3070554405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f8810d780 0x7f1f8810dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.894+0000 7f1f8dc00640 1 --2- 172.21.15.31:0/3070554405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f8810fc90 0x7f1f88110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.894+0000 7f1f8dc00640 1 -- 172.21.15.31:0/3070554405 >> 172.21.15.31:0/3070554405 conn(0x7f1f8806c580 msgr2=0x7f1f8806c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.894+0000 7f1f8dc00640 1 -- 172.21.15.31:0/3070554405 shutdown_connections 2024-03-21T10:45:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.894+0000 7f1f8dc00640 1 -- 172.21.15.31:0/3070554405 wait complete. 2024-03-21T10:45:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.896+0000 7f1f8dc00640 1 Processor -- start 2024-03-21T10:45:19.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.896+0000 7f1f8dc00640 1 -- start start 2024-03-21T10:45:19.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.896+0000 7f1f8dc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f8810d780 0x7f1f881ac730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.896+0000 7f1f8dc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f8810e310 0x7f1f881acc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.896+0000 7f1f8dc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f8810fc90 0x7f1f881a68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.896+0000 7f1f8dc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f88120640 con 0x7f1f8810e310 2024-03-21T10:45:19.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.897+0000 7f1f8dc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f881204c0 con 0x7f1f8810d780 2024-03-21T10:45:19.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.897+0000 7f1f8dc00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f881207c0 con 0x7f1f8810fc90 2024-03-21T10:45:19.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.897+0000 7f1f7fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f8810e310 0x7f1f881acc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.897+0000 7f1f7fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f8810e310 0x7f1f881acc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43562/0 (socket says 172.21.15.31:43562) 2024-03-21T10:45:19.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.897+0000 7f1f7fe00640 1 -- 172.21.15.31:0/1750580119 learned_addr learned my addr 172.21.15.31:0/1750580119 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:19.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.898+0000 7f1f87e00640 1 --2- 172.21.15.31:0/1750580119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f8810d780 0x7f1f881ac730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.898+0000 7f1f7fe00640 1 -- 172.21.15.31:0/1750580119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f8810fc90 msgr2=0x7f1f881a68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:19.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.898+0000 7f1f7fe00640 1 --2- 172.21.15.31:0/1750580119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f8810fc90 0x7f1f881a68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.898+0000 7f1f7fe00640 1 -- 172.21.15.31:0/1750580119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f8810d780 msgr2=0x7f1f881ac730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:19.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.898+0000 7f1f7fe00640 1 --2- 172.21.15.31:0/1750580119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f8810d780 0x7f1f881ac730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.898+0000 7f1f7fe00640 1 -- 172.21.15.31:0/1750580119 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f881a7060 con 0x7f1f8810e310 2024-03-21T10:45:19.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.898+0000 7f1f87e00640 1 --2- 172.21.15.31:0/1750580119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f8810d780 0x7f1f881ac730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:45:19.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.898+0000 7f1f7fe00640 1 --2- 172.21.15.31:0/1750580119 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f8810e310 0x7f1f881acc70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1f7000e9f0 tx=0x7f1f7000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:19.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.899+0000 7f1f85600640 1 -- 172.21.15.31:0/1750580119 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7000cd80 con 0x7f1f8810e310 2024-03-21T10:45:19.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.899+0000 7f1f85600640 1 -- 172.21.15.31:0/1750580119 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f70010910 con 0x7f1f8810e310 2024-03-21T10:45:19.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.899+0000 7f1f85600640 1 -- 172.21.15.31:0/1750580119 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f70018980 con 0x7f1f8810e310 2024-03-21T10:45:19.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.900+0000 7f1f8dc00640 1 -- 172.21.15.31:0/1750580119 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f881a7350 con 0x7f1f8810e310 2024-03-21T10:45:19.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.900+0000 7f1f8dc00640 1 -- 172.21.15.31:0/1750580119 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f881b35f0 con 0x7f1f8810e310 2024-03-21T10:45:19.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.908+0000 7f1f85600640 1 -- 172.21.15.31:0/1750580119 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f1f70010ac0 con 0x7f1f8810e310 2024-03-21T10:45:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.910+0000 7f1f85600640 1 --2- 172.21.15.31:0/1750580119 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f1f6c0a3260 0x7f1f6c0a5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.911+0000 7f1f85600640 1 -- 172.21.15.31:0/1750580119 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f1f700d0390 con 0x7f1f8810e310 2024-03-21T10:45:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.913+0000 7f1f87e00640 1 --2- 172.21.15.31:0/1750580119 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f1f6c0a3260 0x7f1f6c0a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.914+0000 7f1f91f88740 1 Processor -- start 2024-03-21T10:45:19.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.914+0000 7f1f91f88740 1 -- start start 2024-03-21T10:45:19.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.914+0000 7f1f91f88740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565292290290 0x565292290670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.914+0000 7f1f91f88740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565292290bb0 0x565292299250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.914+0000 7f1f91f88740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565292299790 0x56529229bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.914+0000 7f1f91f88740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652921f4eb0 con 0x565292299790 2024-03-21T10:45:19.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.914+0000 7f1f91f88740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529219da60 con 0x565292290bb0 2024-03-21T10:45:19.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.914+0000 7f1f91f88740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652921f0f50 con 0x565292290290 2024-03-21T10:45:19.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.915+0000 7f1f8d200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565292299790 0x56529229bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.915+0000 7f1f7fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565292290bb0 0x565292299250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.915+0000 7f1f7fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565292290bb0 0x565292299250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56602/0 (socket says 172.21.15.31:56602) 2024-03-21T10:45:19.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.915+0000 7f1f7fe00640 1 -- 172.21.15.31:0/913693173 learned_addr learned my addr 172.21.15.31:0/913693173 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:19.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.916+0000 7f1f8d200640 1 -- 172.21.15.31:0/913693173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565292290290 msgr2=0x565292290670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:45:19.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.916+0000 7f1f8d200640 1 --2- 172.21.15.31:0/913693173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565292290290 0x565292290670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.916+0000 7f1f8d200640 1 -- 172.21.15.31:0/913693173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565292290bb0 msgr2=0x565292299250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:19.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.916+0000 7f1f8d200640 1 --2- 172.21.15.31:0/913693173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565292290bb0 0x565292299250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.916+0000 7f1f8d200640 1 -- 172.21.15.31:0/913693173 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565291fd17e0 con 0x565292299790 2024-03-21T10:45:19.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.917+0000 7f1f8d200640 1 --2- 172.21.15.31:0/913693173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565292299790 0x56529229bb80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1f80036040 tx=0x7f1f80009330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4e5c11f1c401ad4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:19.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.918+0000 7f1f7d600640 1 -- 172.21.15.31:0/913693173 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f80042020 con 0x565292299790 2024-03-21T10:45:19.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.918+0000 7f1f7d600640 1 -- 172.21.15.31:0/913693173 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f80033480 con 0x565292299790 2024-03-21T10:45:19.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.918+0000 7f1f7d600640 1 -- 172.21.15.31:0/913693173 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8002c890 con 0x565292299790 2024-03-21T10:45:19.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.919+0000 7f1f91f88740 1 -- 172.21.15.31:0/913693173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565292299790 msgr2=0x56529229bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:19.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.919+0000 7f1f91f88740 1 --2- 172.21.15.31:0/913693173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565292299790 0x56529229bb80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1f80036040 tx=0x7f1f80009330 comp rx=0 tx=0).stop 2024-03-21T10:45:19.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.919+0000 7f1f87e00640 1 --2- 172.21.15.31:0/1750580119 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f1f6c0a3260 0x7f1f6c0a5720 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1f881a7a20 tx=0x7f1f78009260 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:19.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.919+0000 7f1f91f88740 1 -- 172.21.15.31:0/913693173 shutdown_connections 2024-03-21T10:45:19.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.919+0000 7f1f91f88740 1 --2- 172.21.15.31:0/913693173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565292299790 0x56529229bb80 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.919+0000 7f1f91f88740 1 --2- 172.21.15.31:0/913693173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565292290bb0 0x565292299250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.919+0000 7f1f91f88740 1 --2- 172.21.15.31:0/913693173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565292290290 0x565292290670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.920+0000 7f1f91f88740 1 -- 172.21.15.31:0/913693173 >> 172.21.15.31:0/913693173 conn(0x56529228a9c0 msgr2=0x56529228ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:19.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.920+0000 7f1f91f88740 1 -- 172.21.15.31:0/913693173 shutdown_connections 2024-03-21T10:45:19.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.921+0000 7f1f91f88740 1 -- 172.21.15.31:0/913693173 wait complete. 2024-03-21T10:45:19.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.921+0000 7f1f91f88740 1 -- start start 2024-03-21T10:45:19.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.922+0000 7f1f91f88740 1 Processor -- start 2024-03-21T10:45:19.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.924+0000 7f1f91f88740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565292290290 0x565292299110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.924+0000 7f1f91f88740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565292290bb0 0x5652922964a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.925+0000 7f1f91f88740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565292299790 0x565292296a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.925+0000 7f1f91f88740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529229c520 con 0x565292290bb0 2024-03-21T10:45:19.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.925+0000 7f1f91f88740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652921b0eb0 con 0x565292299790 2024-03-21T10:45:19.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.925+0000 7f1f91f88740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529229c6a0 con 0x565292290290 2024-03-21T10:45:19.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.925+0000 7f1f91f88740 10 client.0 _ll_register_callbacks cb 0x5652921a5150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:19.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.926+0000 7f1f87e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565292290290 0x565292299110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.926+0000 7f1f87e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565292290290 0x565292299110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:35062/0 (socket says 172.21.15.31:35062) 2024-03-21T10:45:19.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.926+0000 7f1f87e00640 1 -- 172.21.15.31:0/1800526105 learned_addr learned my addr 172.21.15.31:0/1800526105 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:19.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.926+0000 7f1f8d200640 1 --2- 172.21.15.31:0/1800526105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565292299790 0x565292296a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.926+0000 7f1f7fe00640 1 --2- 172.21.15.31:0/1800526105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565292290bb0 0x5652922964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.927+0000 7f1f87e00640 1 -- 172.21.15.31:0/1800526105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565292299790 msgr2=0x565292296a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:19.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.927+0000 7f1f87e00640 1 --2- 172.21.15.31:0/1800526105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565292299790 0x565292296a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.927+0000 7f1f87e00640 1 -- 172.21.15.31:0/1800526105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565292290bb0 msgr2=0x5652922964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:19.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.927+0000 7f1f87e00640 1 --2- 172.21.15.31:0/1800526105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565292290bb0 0x5652922964a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.927+0000 7f1f87e00640 1 -- 172.21.15.31:0/1800526105 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565292031f70 con 0x565292290290 2024-03-21T10:45:19.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.927+0000 7f1f7fe00640 1 --2- 172.21.15.31:0/1800526105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565292290bb0 0x5652922964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:45:19.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.928+0000 7f1f87e00640 1 --2- 172.21.15.31:0/1800526105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565292290290 0x565292299110 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f1f78012480 tx=0x7f1f780046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=241df95947434712 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:19.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.928+0000 7f1f76000640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:45:19.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.930+0000 7f1f76000640 1 -- 172.21.15.31:0/1800526105 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f78023070 con 0x565292290290 2024-03-21T10:45:19.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.930+0000 7f1f76000640 1 -- 172.21.15.31:0/1800526105 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f1f78004220 con 0x565292290290 2024-03-21T10:45:19.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.931+0000 7f1f91f88740 1 -- 172.21.15.31:0/1800526105 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1f80033480 con 0x565292290290 2024-03-21T10:45:19.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.932+0000 7f1f76000640 1 -- 172.21.15.31:0/1800526105 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f7802c050 con 0x565292290290 2024-03-21T10:45:19.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.932+0000 7f1f91f88740 10 client.27208 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:19.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.932+0000 7f1f91f88740 1 -- 172.21.15.31:0/1800526105 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x565291ec9170 con 0x565292290290 2024-03-21T10:45:19.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.932+0000 7f1f76000640 1 -- 172.21.15.31:0/1800526105 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f80033480 con 0x565292290290 2024-03-21T10:45:19.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.935+0000 7f1f76000640 1 -- 172.21.15.31:0/1800526105 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f1f7800f040 con 0x565292290290 2024-03-21T10:45:19.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.935+0000 7f1f91f88740 10 client.27208 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:19.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.935+0000 7f1f91f88740 10 client.27208 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:45:19.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.935+0000 7f1f91f88740 20 client.27208 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:19.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.936+0000 7f1f91f88740 1 --2- 172.21.15.31:0/1800526105 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5652922a9cb0 0x5652922ac0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.936+0000 7f1f91f88740 4 client.27208 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:45:19.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.936+0000 7f1f91f88740 1 -- 172.21.15.31:0/1800526105 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5652922023f0 con 0x5652922a9cb0 2024-03-21T10:45:19.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.937+0000 7f1f7fe00640 1 --2- 172.21.15.31:0/1800526105 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5652922a9cb0 0x5652922ac0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.939+0000 7f1f7fe00640 1 --2- 172.21.15.31:0/1800526105 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5652922a9cb0 0x5652922ac0e0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a38958108dc96d6 server_cookie=680061e84237b7fc in_seq=0 out_seq=0 2024-03-21T10:45:19.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.939+0000 7f1f76000640 10 client.27208 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:19.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.940+0000 7f1f76000640 1 -- 172.21.15.31:0/1800526105 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5652922023f0 con 0x5652922a9cb0 2024-03-21T10:45:19.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.940+0000 7f1f76000640 10 client.27208 handle_command_reply: tid=0 2024-03-21T10:45:19.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.940+0000 7f1f91f88740 1 client.27208 shutdown 2024-03-21T10:45:19.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.944+0000 7f1f91f88740 1 -- 172.21.15.31:0/1800526105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565292290290 msgr2=0x565292299110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:19.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.944+0000 7f1f91f88740 1 --2- 172.21.15.31:0/1800526105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565292290290 0x565292299110 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f1f78012480 tx=0x7f1f780046c0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.947+0000 7f1f91f88740 1 -- 172.21.15.31:0/1800526105 shutdown_connections 2024-03-21T10:45:19.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.947+0000 7f1f91f88740 1 --2- 172.21.15.31:0/1800526105 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5652922a9cb0 0x5652922ac0e0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.948+0000 7f1f91f88740 1 --2- 172.21.15.31:0/1800526105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565292299790 0x565292296a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.948+0000 7f1f91f88740 1 --2- 172.21.15.31:0/1800526105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565292290bb0 0x5652922964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.948+0000 7f1f91f88740 1 --2- 172.21.15.31:0/1800526105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565292290290 0x565292299110 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.948+0000 7f1f91f88740 1 -- 172.21.15.31:0/1800526105 >> 172.21.15.31:0/1800526105 conn(0x56529228a9c0 msgr2=0x56529228c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:19.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.948+0000 7f1f76000640 0 client.27208 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:19.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.959+0000 7f1f91f88740 1 -- 172.21.15.31:0/1800526105 shutdown_connections 2024-03-21T10:45:19.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.960+0000 7f1f91f88740 1 -- 172.21.15.31:0/1800526105 wait complete. 2024-03-21T10:45:19.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.960+0000 7f1f91f88740 20 client.27208 trim_cache size 0 max 16384 2024-03-21T10:45:19.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.991+0000 7f1f91f88740 1 Processor -- start 2024-03-21T10:45:19.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.992+0000 7f1f91f88740 1 -- start start 2024-03-21T10:45:19.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.992+0000 7f1f91f88740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565292290290 0x565292298bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.992+0000 7f1f91f88740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565292290bb0 0x565292299110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.992+0000 7f1f87e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565292290290 0x565292298bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.992+0000 7f1f87e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565292290290 0x565292298bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:35076/0 (socket says 172.21.15.31:35076) 2024-03-21T10:45:19.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.992+0000 7f1f7fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565292290bb0 0x565292299110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.992+0000 7f1f7fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565292290bb0 0x565292299110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43586/0 (socket says 172.21.15.31:43586) 2024-03-21T10:45:19.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.992+0000 7f1f91f88740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565292299790 0x5652922a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.992+0000 7f1f7fe00640 1 -- 172.21.15.31:0/2405122997 learned_addr learned my addr 172.21.15.31:0/2405122997 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:19.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.992+0000 7f1f91f88740 1 -- 172.21.15.31:0/2405122997 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f881bfa70 con 0x565292290bb0 2024-03-21T10:45:19.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.993+0000 7f1f91f88740 1 -- 172.21.15.31:0/2405122997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f881bd570 con 0x565292299790 2024-03-21T10:45:19.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.993+0000 7f1f91f88740 1 -- 172.21.15.31:0/2405122997 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f881bb630 con 0x565292290290 2024-03-21T10:45:19.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.993+0000 7f1f8d200640 1 --2- 172.21.15.31:0/2405122997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565292299790 0x5652922a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.993+0000 7f1f87e00640 1 -- 172.21.15.31:0/2405122997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565292299790 msgr2=0x5652922a4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:19.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.993+0000 7f1f87e00640 1 --2- 172.21.15.31:0/2405122997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565292299790 0x5652922a4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.993+0000 7f1f87e00640 1 -- 172.21.15.31:0/2405122997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565292290bb0 msgr2=0x565292299110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:19.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.993+0000 7f1f87e00640 1 --2- 172.21.15.31:0/2405122997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565292290bb0 0x565292299110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.993+0000 7f1f87e00640 1 -- 172.21.15.31:0/2405122997 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652922a9a60 con 0x565292290290 2024-03-21T10:45:19.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.993+0000 7f1f7fe00640 1 --2- 172.21.15.31:0/2405122997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565292290bb0 0x565292299110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:45:19.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.993+0000 7f1f87e00640 1 --2- 172.21.15.31:0/2405122997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565292290290 0x565292298bd0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f1f78012450 tx=0x7f1f7801f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=458855445d0bbf93 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:19.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.994+0000 7f1f7cc00640 1 -- 172.21.15.31:0/2405122997 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f78023040 con 0x565292290290 2024-03-21T10:45:19.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.994+0000 7f1f7cc00640 1 -- 172.21.15.31:0/2405122997 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f78005550 con 0x565292290290 2024-03-21T10:45:19.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.994+0000 7f1f7cc00640 1 -- 172.21.15.31:0/2405122997 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7801f8a0 con 0x565292290290 2024-03-21T10:45:19.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.994+0000 7f1f91f88740 1 -- 172.21.15.31:0/2405122997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565292290290 msgr2=0x565292298bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:19.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.994+0000 7f1f91f88740 1 --2- 172.21.15.31:0/2405122997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565292290290 0x565292298bd0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f1f78012450 tx=0x7f1f7801f000 comp rx=0 tx=0).stop 2024-03-21T10:45:19.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.995+0000 7f1f91f88740 1 -- 172.21.15.31:0/2405122997 shutdown_connections 2024-03-21T10:45:19.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.995+0000 7f1f91f88740 1 --2- 172.21.15.31:0/2405122997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565292299790 0x5652922a4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.995+0000 7f1f91f88740 1 --2- 172.21.15.31:0/2405122997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565292290bb0 0x565292299110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.995+0000 7f1f91f88740 1 --2- 172.21.15.31:0/2405122997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565292290290 0x565292298bd0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.995+0000 7f1f91f88740 1 -- 172.21.15.31:0/2405122997 >> 172.21.15.31:0/2405122997 conn(0x56529228a9c0 msgr2=0x56529228c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:19.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.996+0000 7f1f91f88740 1 -- 172.21.15.31:0/2405122997 shutdown_connections 2024-03-21T10:45:19.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.998+0000 7f1f91f88740 1 -- 172.21.15.31:0/2405122997 wait complete. 2024-03-21T10:45:19.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.998+0000 7f1f91f88740 1 -- start start 2024-03-21T10:45:19.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.999+0000 7f1f91f88740 1 Processor -- start 2024-03-21T10:45:19.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.999+0000 7f1f91f88740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565292290290 0x56529229e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.999+0000 7f1f91f88740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565292290bb0 0x56529229ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.999+0000 7f1f91f88740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565292299790 0x56529229f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:19.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.999+0000 7f1f91f88740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f881bbc30 con 0x565292290bb0 2024-03-21T10:45:19.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.999+0000 7f1f91f88740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f881bcf10 con 0x565292290290 2024-03-21T10:45:19.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.999+0000 7f1f91f88740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f881bc250 con 0x565292299790 2024-03-21T10:45:19.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:19.999+0000 7f1f91f88740 10 client.0 _ll_register_callbacks cb 0x7f1f78016070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:19.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.000+0000 7f1f87e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565292290290 0x56529229e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.000+0000 7f1f87e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565292290290 0x56529229e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56632/0 (socket says 172.21.15.31:56632) 2024-03-21T10:45:19.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.000+0000 7f1f87e00640 1 -- 172.21.15.31:0/3721136261 learned_addr learned my addr 172.21.15.31:0/3721136261 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:19.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.000+0000 7f1f87e00640 1 -- 172.21.15.31:0/3721136261 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565292299790 msgr2=0x56529229f3c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:45:19.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.000+0000 7f1f87e00640 1 --2- 172.21.15.31:0/3721136261 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565292299790 0x56529229f3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.000+0000 7f1f87e00640 1 -- 172.21.15.31:0/3721136261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565292290bb0 msgr2=0x56529229ee80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:45:19.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.001+0000 7f1f7fe00640 1 --2- 172.21.15.31:0/3721136261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565292290bb0 0x56529229ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:19.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.001+0000 7f1f87e00640 1 --2- 172.21.15.31:0/3721136261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565292290bb0 0x56529229ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:19.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.001+0000 7f1f87e00640 1 -- 172.21.15.31:0/3721136261 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565292294480 con 0x565292290290 2024-03-21T10:45:20.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.002+0000 7f1f87e00640 1 --2- 172.21.15.31:0/3721136261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565292290290 0x56529229e940 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f1f780038c0 tx=0x7f1f78005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61fdc6912ff945ac server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:20.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.002+0000 7f1f76000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:45:20.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.002+0000 7f1f76000640 1 -- 172.21.15.31:0/3721136261 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7803e020 con 0x565292290290 2024-03-21T10:45:20.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.002+0000 7f1f76000640 1 -- 172.21.15.31:0/3721136261 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f1f7801fc20 con 0x565292290290 2024-03-21T10:45:20.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.002+0000 7f1f91f88740 1 -- 172.21.15.31:0/3721136261 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1f78005550 con 0x565292290290 2024-03-21T10:45:20.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.003+0000 7f1f76000640 1 -- 172.21.15.31:0/3721136261 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f78023040 con 0x565292290290 2024-03-21T10:45:20.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.003+0000 7f1f91f88740 10 client.27302 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:20.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.003+0000 7f1f91f88740 1 -- 172.21.15.31:0/3721136261 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x565292306610 con 0x565292290290 2024-03-21T10:45:20.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.003+0000 7f1f76000640 1 -- 172.21.15.31:0/3721136261 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f78005550 con 0x565292290290 2024-03-21T10:45:20.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.003+0000 7f1f76000640 1 -- 172.21.15.31:0/3721136261 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f1f7803a930 con 0x565292290290 2024-03-21T10:45:20.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.004+0000 7f1f91f88740 10 client.27302 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:20.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.004+0000 7f1f91f88740 10 client.27302 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:45:20.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.004+0000 7f1f91f88740 20 client.27302 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:20.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.004+0000 7f1f91f88740 1 --2- 172.21.15.31:0/3721136261 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x565292306bc0 0x565292308fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:20.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.004+0000 7f1f91f88740 4 client.27302 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-21T10:45:20.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.004+0000 7f1f91f88740 1 -- 172.21.15.31:0/3721136261 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5652921fc740 con 0x565292306bc0 2024-03-21T10:45:20.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.004+0000 7f1f7fe00640 1 --2- 172.21.15.31:0/3721136261 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x565292306bc0 0x565292308fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:20.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.005+0000 7f1f7fe00640 1 --2- 172.21.15.31:0/3721136261 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x565292306bc0 0x565292308fb0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c4bdc55ffaf1821 server_cookie=67539430a2ff59c0 in_seq=0 out_seq=0 2024-03-21T10:45:20.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.005+0000 7f1f76000640 10 client.27302 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:20.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.007+0000 7f1f76000640 1 -- 172.21.15.31:0/3721136261 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5652921fc740 con 0x565292306bc0 2024-03-21T10:45:20.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.007+0000 7f1f76000640 10 client.27302 handle_command_reply: tid=0 2024-03-21T10:45:20.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.007+0000 7f1f91f88740 1 client.27302 shutdown 2024-03-21T10:45:20.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.007+0000 7f1f91f88740 1 -- 172.21.15.31:0/3721136261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565292290290 msgr2=0x56529229e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:20.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.007+0000 7f1f91f88740 1 --2- 172.21.15.31:0/3721136261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565292290290 0x56529229e940 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f1f780038c0 tx=0x7f1f78005fd0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.007+0000 7f1f91f88740 1 -- 172.21.15.31:0/3721136261 shutdown_connections 2024-03-21T10:45:20.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.007+0000 7f1f91f88740 1 --2- 172.21.15.31:0/3721136261 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x565292306bc0 0x565292308fb0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.007+0000 7f1f91f88740 1 --2- 172.21.15.31:0/3721136261 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565292299790 0x56529229f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.008+0000 7f1f76000640 0 client.27302 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:20.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.008+0000 7f1f91f88740 1 --2- 172.21.15.31:0/3721136261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565292290bb0 0x56529229ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.008+0000 7f1f91f88740 1 --2- 172.21.15.31:0/3721136261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565292290290 0x56529229e940 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.008+0000 7f1f91f88740 1 -- 172.21.15.31:0/3721136261 >> 172.21.15.31:0/3721136261 conn(0x56529228a9c0 msgr2=0x5652922891f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:20.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.008+0000 7f1f91f88740 1 -- 172.21.15.31:0/3721136261 shutdown_connections 2024-03-21T10:45:20.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.008+0000 7f1f91f88740 1 -- 172.21.15.31:0/3721136261 wait complete. 2024-03-21T10:45:20.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.008+0000 7f1f91f88740 20 client.27302 trim_cache size 0 max 16384 2024-03-21T10:45:20.006 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:45:20.006 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-03-21T10:45:20.007 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "6ebaf4cd-bac5-41c8-a332-36ac1b2ce3c9", 2024-03-21T10:45:20.007 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-03-21T10:45:20.007 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:45:20.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.009+0000 7f1f8dc00640 1 -- 172.21.15.31:0/1750580119 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f1f6c0a3260 msgr2=0x7f1f6c0a5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:20.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.009+0000 7f1f8dc00640 1 --2- 172.21.15.31:0/1750580119 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f1f6c0a3260 0x7f1f6c0a5720 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1f881a7a20 tx=0x7f1f78009260 comp rx=0 tx=0).stop 2024-03-21T10:45:20.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.009+0000 7f1f8dc00640 1 -- 172.21.15.31:0/1750580119 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f8810e310 msgr2=0x7f1f881acc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:20.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.009+0000 7f1f8dc00640 1 --2- 172.21.15.31:0/1750580119 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f8810e310 0x7f1f881acc70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1f7000e9f0 tx=0x7f1f7000eeb0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.009+0000 7f1f8dc00640 1 -- 172.21.15.31:0/1750580119 shutdown_connections 2024-03-21T10:45:20.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.009+0000 7f1f8dc00640 1 --2- 172.21.15.31:0/1750580119 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f1f6c0a3260 0x7f1f6c0a5720 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.009+0000 7f1f8dc00640 1 --2- 172.21.15.31:0/1750580119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f8810fc90 0x7f1f881a68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.009+0000 7f1f8dc00640 1 --2- 172.21.15.31:0/1750580119 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f8810e310 0x7f1f881acc70 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.009+0000 7f1f8dc00640 1 --2- 172.21.15.31:0/1750580119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f8810d780 0x7f1f881ac730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.009+0000 7f1f8dc00640 1 -- 172.21.15.31:0/1750580119 >> 172.21.15.31:0/1750580119 conn(0x7f1f8806c580 msgr2=0x7f1f88070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:20.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.010+0000 7f1f8dc00640 1 -- 172.21.15.31:0/1750580119 shutdown_connections 2024-03-21T10:45:20.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.010+0000 7f1f8dc00640 1 -- 172.21.15.31:0/1750580119 wait complete. 2024-03-21T10:45:20.033 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:45:20.033 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:45:20.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:20 smithi031 ceph-mon[31673]: scrub queued for path: / 2024-03-21T10:45:20.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:20 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:45:20.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:20 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:45:20.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:20 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:45:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.450+0000 7f45d9c00640 1 -- 172.21.15.31:0/250829316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45d4100440 msgr2=0x7f45d4100820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.450+0000 7f45d9c00640 1 --2- 172.21.15.31:0/250829316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45d4100440 0x7f45d4100820 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f45c0009900 tx=0x7f45c002bcc0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.451+0000 7f45d9c00640 1 -- 172.21.15.31:0/250829316 shutdown_connections 2024-03-21T10:45:20.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.451+0000 7f45d9c00640 1 --2- 172.21.15.31:0/250829316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45d410dff0 0x7f45d41103e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.451+0000 7f45d9c00640 1 --2- 172.21.15.31:0/250829316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45d4100d60 0x7f45d410dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.451+0000 7f45d9c00640 1 --2- 172.21.15.31:0/250829316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45d4100440 0x7f45d4100820 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.451+0000 7f45d9c00640 1 -- 172.21.15.31:0/250829316 >> 172.21.15.31:0/250829316 conn(0x7f45d4077750 msgr2=0x7f45d4077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:20.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.451+0000 7f45d9c00640 1 -- 172.21.15.31:0/250829316 shutdown_connections 2024-03-21T10:45:20.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.451+0000 7f45d9c00640 1 -- 172.21.15.31:0/250829316 wait complete. 2024-03-21T10:45:20.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.453+0000 7f45d9c00640 1 Processor -- start 2024-03-21T10:45:20.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.453+0000 7f45d9c00640 1 -- start start 2024-03-21T10:45:20.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.453+0000 7f45d9c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45d4100440 0x7f45d419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:20.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.453+0000 7f45d9c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45d4100d60 0x7f45d419a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:20.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.453+0000 7f45d9c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45d410dff0 0x7f45d419ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:20.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.453+0000 7f45d9c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d4113cc0 con 0x7f45d410dff0 2024-03-21T10:45:20.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.453+0000 7f45d9c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d4113b40 con 0x7f45d4100d60 2024-03-21T10:45:20.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.453+0000 7f45d9c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d4113e40 con 0x7f45d4100440 2024-03-21T10:45:20.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.454+0000 7f45d2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45d4100440 0x7f45d419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:20.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.454+0000 7f45d2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45d4100440 0x7f45d419a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:35084/0 (socket says 172.21.15.31:35084) 2024-03-21T10:45:20.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.454+0000 7f45d2a00640 1 -- 172.21.15.31:0/3909453736 learned_addr learned my addr 172.21.15.31:0/3909453736 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:20.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.454+0000 7f45d2000640 1 --2- 172.21.15.31:0/3909453736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45d4100d60 0x7f45d419a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:20.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.454+0000 7f45d2a00640 1 -- 172.21.15.31:0/3909453736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45d4100d60 msgr2=0x7f45d419a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:20.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.454+0000 7f45d2a00640 1 --2- 172.21.15.31:0/3909453736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45d4100d60 0x7f45d419a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.454+0000 7f45d2a00640 1 -- 172.21.15.31:0/3909453736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45d410dff0 msgr2=0x7f45d419ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:20.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.454+0000 7f45d2a00640 1 --2- 172.21.15.31:0/3909453736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45d410dff0 0x7f45d419ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.454+0000 7f45d2a00640 1 -- 172.21.15.31:0/3909453736 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d419f120 con 0x7f45d4100440 2024-03-21T10:45:20.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.456+0000 7f45d2a00640 1 --2- 172.21.15.31:0/3909453736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45d4100440 0x7f45d419a160 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f45c002c2d0 tx=0x7f45c0004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:20.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.456+0000 7f45cb400640 1 -- 172.21.15.31:0/3909453736 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c0041020 con 0x7f45d4100440 2024-03-21T10:45:20.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.456+0000 7f45cb400640 1 -- 172.21.15.31:0/3909453736 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45c0005000 con 0x7f45d4100440 2024-03-21T10:45:20.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.456+0000 7f45d9c00640 1 -- 172.21.15.31:0/3909453736 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45d419f3b0 con 0x7f45d4100440 2024-03-21T10:45:20.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.456+0000 7f45cb400640 1 -- 172.21.15.31:0/3909453736 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c0003b20 con 0x7f45d4100440 2024-03-21T10:45:20.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.456+0000 7f45d9c00640 1 -- 172.21.15.31:0/3909453736 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d406b0c0 con 0x7f45d4100440 2024-03-21T10:45:20.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.460+0000 7f45dca2c740 1 Processor -- start 2024-03-21T10:45:20.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.461+0000 7f45dca2c740 1 -- start start 2024-03-21T10:45:20.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.461+0000 7f45cb400640 1 -- 172.21.15.31:0/3909453736 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f45c0044020 con 0x7f45d4100440 2024-03-21T10:45:20.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.461+0000 7f45dca2c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea80497410 0x55ea80496160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:20.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.461+0000 7f45dca2c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea80496730 0x55ea80496bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:20.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.461+0000 7f45dca2c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea8049faa0 0x55ea804a1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:20.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.461+0000 7f45dca2c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea803f2250 con 0x55ea8049faa0 2024-03-21T10:45:20.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.461+0000 7f45dca2c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea8039aa30 con 0x55ea80497410 2024-03-21T10:45:20.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.461+0000 7f45dca2c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea803ee2f0 con 0x55ea80496730 2024-03-21T10:45:20.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.462+0000 7f45d3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea80497410 0x55ea80496160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:20.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.462+0000 7f45d3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea80497410 0x55ea80496160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56648/0 (socket says 172.21.15.31:56648) 2024-03-21T10:45:20.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.462+0000 7f45d3400640 1 -- 172.21.15.31:0/2683692292 learned_addr learned my addr 172.21.15.31:0/2683692292 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:20.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.462+0000 7f45d2a00640 1 --2- 172.21.15.31:0/2683692292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea80496730 0x55ea80496bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:20.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.462+0000 7f45d2000640 1 --2- 172.21.15.31:0/2683692292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea8049faa0 0x55ea804a1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:20.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.462+0000 7f45d3400640 1 -- 172.21.15.31:0/2683692292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea80496730 msgr2=0x55ea80496bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:20.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.462+0000 7f45d3400640 1 --2- 172.21.15.31:0/2683692292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea80496730 0x55ea80496bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.462+0000 7f45d3400640 1 -- 172.21.15.31:0/2683692292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea8049faa0 msgr2=0x55ea804a1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:20.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.463+0000 7f45d3400640 1 --2- 172.21.15.31:0/2683692292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea8049faa0 0x55ea804a1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.463+0000 7f45d3400640 1 -- 172.21.15.31:0/2683692292 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea80385690 con 0x55ea80497410 2024-03-21T10:45:20.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.463+0000 7f45d2000640 1 --2- 172.21.15.31:0/2683692292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea8049faa0 0x55ea804a1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:45:20.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.463+0000 7f45d3400640 1 --2- 172.21.15.31:0/2683692292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea80497410 0x55ea80496160 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009db0 tx=0x7f45cc00e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81ef0e38e36356a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:20.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.463+0000 7f45bfe00640 1 -- 172.21.15.31:0/2683692292 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc004b20 con 0x55ea80497410 2024-03-21T10:45:20.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.463+0000 7f45cb400640 1 --2- 172.21.15.31:0/3909453736 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f45b80a3110 0x7f45b80a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:20.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.463+0000 7f45bfe00640 1 -- 172.21.15.31:0/2683692292 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45cc003e50 con 0x55ea80497410 2024-03-21T10:45:20.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.463+0000 7f45bfe00640 1 -- 172.21.15.31:0/2683692292 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc005020 con 0x55ea80497410 2024-03-21T10:45:20.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.463+0000 7f45cb400640 1 -- 172.21.15.31:0/3909453736 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f45c00f1270 con 0x7f45d4100440 2024-03-21T10:45:20.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.464+0000 7f45d2a00640 1 --2- 172.21.15.31:0/3909453736 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f45b80a3110 0x7f45b80a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:20.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.464+0000 7f45dca2c740 1 -- 172.21.15.31:0/2683692292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea80497410 msgr2=0x55ea80496160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:20.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.464+0000 7f45dca2c740 1 --2- 172.21.15.31:0/2683692292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea80497410 0x55ea80496160 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009db0 tx=0x7f45cc00e720 comp rx=0 tx=0).stop 2024-03-21T10:45:20.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.464+0000 7f45dca2c740 1 -- 172.21.15.31:0/2683692292 shutdown_connections 2024-03-21T10:45:20.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.464+0000 7f45dca2c740 1 --2- 172.21.15.31:0/2683692292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea8049faa0 0x55ea804a1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.464+0000 7f45dca2c740 1 --2- 172.21.15.31:0/2683692292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea80496730 0x55ea80496bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.464+0000 7f45dca2c740 1 --2- 172.21.15.31:0/2683692292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea80497410 0x55ea80496160 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.464+0000 7f45dca2c740 1 -- 172.21.15.31:0/2683692292 >> 172.21.15.31:0/2683692292 conn(0x55ea80455200 msgr2=0x55ea80491180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:20.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.465+0000 7f45dca2c740 1 -- 172.21.15.31:0/2683692292 shutdown_connections 2024-03-21T10:45:20.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.465+0000 7f45dca2c740 1 -- 172.21.15.31:0/2683692292 wait complete. 2024-03-21T10:45:20.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.465+0000 7f45d2a00640 1 --2- 172.21.15.31:0/3909453736 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f45b80a3110 0x7f45b80a55d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x55ea80497960 tx=0x7f45c00f4ca0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:20.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.465+0000 7f45dca2c740 1 -- start start 2024-03-21T10:45:20.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.465+0000 7f45dca2c740 1 Processor -- start 2024-03-21T10:45:20.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.466+0000 7f45dca2c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea80496730 0x55ea8049f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:20.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.466+0000 7f45dca2c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea80497410 0x55ea8049c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:20.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.466+0000 7f45dca2c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea8049faa0 0x55ea8049c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:20.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.466+0000 7f45dca2c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea804a2910 con 0x55ea80497410 2024-03-21T10:45:20.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.466+0000 7f45dca2c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea803ade80 con 0x55ea80496730 2024-03-21T10:45:20.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.466+0000 7f45dca2c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea804a2a90 con 0x55ea8049faa0 2024-03-21T10:45:20.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.466+0000 7f45dca2c740 10 client.0 _ll_register_callbacks cb 0x55ea803c0130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:20.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.466+0000 7f45d3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea80496730 0x55ea8049f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:20.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.466+0000 7f45d3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea80496730 0x55ea8049f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56650/0 (socket says 172.21.15.31:56650) 2024-03-21T10:45:20.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.466+0000 7f45d3400640 1 -- 172.21.15.31:0/3559086172 learned_addr learned my addr 172.21.15.31:0/3559086172 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:20.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.466+0000 7f45d2000640 1 --2- 172.21.15.31:0/3559086172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea80497410 0x55ea8049c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:20.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.466+0000 7f45d3400640 1 --2- 172.21.15.31:0/3559086172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea8049faa0 0x55ea8049c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:20.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.467+0000 7f45d3400640 1 -- 172.21.15.31:0/3559086172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea8049faa0 msgr2=0x55ea8049c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:20.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.467+0000 7f45d3400640 1 --2- 172.21.15.31:0/3559086172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea8049faa0 0x55ea8049c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.467+0000 7f45d3400640 1 -- 172.21.15.31:0/3559086172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea80497410 msgr2=0x55ea8049c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:20.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.467+0000 7f45d3400640 1 --2- 172.21.15.31:0/3559086172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea80497410 0x55ea8049c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.467+0000 7f45d3400640 1 -- 172.21.15.31:0/3559086172 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ea801d07d0 con 0x55ea80496730 2024-03-21T10:45:20.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.467+0000 7f45d2000640 1 --2- 172.21.15.31:0/3559086172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea80497410 0x55ea8049c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:45:20.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.467+0000 7f45d3400640 1 --2- 172.21.15.31:0/3559086172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea80496730 0x55ea8049f080 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009bf0 tx=0x7f45cc004590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6865371078710f99 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:20.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.467+0000 7f45bcc00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:45:20.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.467+0000 7f45bcc00640 1 -- 172.21.15.31:0/3559086172 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc01e020 con 0x55ea80496730 2024-03-21T10:45:20.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.468+0000 7f45dca2c740 1 -- 172.21.15.31:0/3559086172 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f45cc003e50 con 0x55ea80496730 2024-03-21T10:45:20.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.468+0000 7f45bcc00640 1 -- 172.21.15.31:0/3559086172 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f45cc003b70 con 0x55ea80496730 2024-03-21T10:45:20.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.468+0000 7f45bcc00640 1 -- 172.21.15.31:0/3559086172 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45cc021070 con 0x55ea80496730 2024-03-21T10:45:20.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.468+0000 7f45dca2c740 10 client.27314 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:20.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.468+0000 7f45dca2c740 1 -- 172.21.15.31:0/3559086172 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55ea801063f0 con 0x55ea80496730 2024-03-21T10:45:20.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.468+0000 7f45bcc00640 1 -- 172.21.15.31:0/3559086172 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45cc003e50 con 0x55ea80496730 2024-03-21T10:45:20.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.468+0000 7f45bcc00640 1 -- 172.21.15.31:0/3559086172 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f45cc018800 con 0x55ea80496730 2024-03-21T10:45:20.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.468+0000 7f45dca2c740 10 client.27314 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:20.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.468+0000 7f45dca2c740 10 client.27314 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:45:20.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.468+0000 7f45dca2c740 20 client.27314 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:20.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.469+0000 7f45dca2c740 1 --2- 172.21.15.31:0/3559086172 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ea804af7c0 0x55ea804b1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:20.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.469+0000 7f45dca2c740 4 client.27314 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:45:20.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.469+0000 7f45dca2c740 1 -- 172.21.15.31:0/3559086172 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ea803ff790 con 0x55ea804af7c0 2024-03-21T10:45:20.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.469+0000 7f45d2000640 1 --2- 172.21.15.31:0/3559086172 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ea804af7c0 0x55ea804b1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:20.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.469+0000 7f45d2000640 1 --2- 172.21.15.31:0/3559086172 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ea804af7c0 0x55ea804b1bb0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eaff733f25d45065 server_cookie=5bf2d764586c1577 in_seq=0 out_seq=0 2024-03-21T10:45:20.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.469+0000 7f45bcc00640 10 client.27314 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:20.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.471+0000 7f45bcc00640 1 -- 172.21.15.31:0/3559086172 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ea803ff790 con 0x55ea804af7c0 2024-03-21T10:45:20.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.471+0000 7f45bcc00640 10 client.27314 handle_command_reply: tid=0 2024-03-21T10:45:20.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.471+0000 7f45dca2c740 1 client.27314 shutdown 2024-03-21T10:45:20.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.471+0000 7f45dca2c740 1 -- 172.21.15.31:0/3559086172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea80496730 msgr2=0x55ea8049f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:20.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.471+0000 7f45dca2c740 1 --2- 172.21.15.31:0/3559086172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea80496730 0x55ea8049f080 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009bf0 tx=0x7f45cc004590 comp rx=0 tx=0).stop 2024-03-21T10:45:20.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.471+0000 7f45dca2c740 1 -- 172.21.15.31:0/3559086172 shutdown_connections 2024-03-21T10:45:20.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.471+0000 7f45dca2c740 1 --2- 172.21.15.31:0/3559086172 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ea804af7c0 0x55ea804b1bb0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.471+0000 7f45dca2c740 1 --2- 172.21.15.31:0/3559086172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea8049faa0 0x55ea8049c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.471+0000 7f45dca2c740 1 --2- 172.21.15.31:0/3559086172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea80497410 0x55ea8049c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.471+0000 7f45bcc00640 0 client.27314 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:20.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.471+0000 7f45dca2c740 1 --2- 172.21.15.31:0/3559086172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea80496730 0x55ea8049f080 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.471+0000 7f45dca2c740 1 -- 172.21.15.31:0/3559086172 >> 172.21.15.31:0/3559086172 conn(0x55ea80455200 msgr2=0x55ea80492ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:20.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.471+0000 7f45dca2c740 1 -- 172.21.15.31:0/3559086172 shutdown_connections 2024-03-21T10:45:20.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.471+0000 7f45dca2c740 1 -- 172.21.15.31:0/3559086172 wait complete. 2024-03-21T10:45:20.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.471+0000 7f45dca2c740 20 client.27314 trim_cache size 0 max 16384 2024-03-21T10:45:20.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.500+0000 7f45dca2c740 1 Processor -- start 2024-03-21T10:45:20.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.505+0000 7f45dca2c740 1 -- start start 2024-03-21T10:45:20.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.505+0000 7f45dca2c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea80496730 0x55ea804a0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:20.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.505+0000 7f45dca2c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea80497410 0x55ea8049dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:20.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.505+0000 7f45dca2c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea8049faa0 0x55ea8049e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:20.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.505+0000 7f45dca2c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b4012270 con 0x55ea80496730 2024-03-21T10:45:20.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.505+0000 7f45dca2c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b400fd70 con 0x55ea80497410 2024-03-21T10:45:20.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.505+0000 7f45dca2c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b400de30 con 0x55ea8049faa0 2024-03-21T10:45:20.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.512+0000 7f45d3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea80496730 0x55ea804a0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:20.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.512+0000 7f45d3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea80496730 0x55ea804a0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43646/0 (socket says 172.21.15.31:43646) 2024-03-21T10:45:20.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.512+0000 7f45d3400640 1 -- 172.21.15.31:0/2582857617 learned_addr learned my addr 172.21.15.31:0/2582857617 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:20.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.513+0000 7f45d3400640 1 --2- 172.21.15.31:0/2582857617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea8049faa0 0x55ea8049e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:20.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.514+0000 7f45d3400640 1 -- 172.21.15.31:0/2582857617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea80497410 msgr2=0x55ea8049dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:20.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.514+0000 7f45d3400640 1 --2- 172.21.15.31:0/2582857617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea80497410 0x55ea8049dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.514+0000 7f45d3400640 1 -- 172.21.15.31:0/2582857617 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea80496730 msgr2=0x55ea804a0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:20.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.514+0000 7f45d3400640 1 --2- 172.21.15.31:0/2582857617 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea80496730 0x55ea804a0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.514+0000 7f45d3400640 1 -- 172.21.15.31:0/2582857617 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea800c60f0 con 0x55ea8049faa0 2024-03-21T10:45:20.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.515+0000 7f45d3400640 1 --2- 172.21.15.31:0/2582857617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea8049faa0 0x55ea8049e000 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f45cc004170 tx=0x7f45cc0042b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=103a1e156e320503 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:20.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.522+0000 7f45bf400640 1 -- 172.21.15.31:0/2582857617 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc03c070 con 0x55ea8049faa0 2024-03-21T10:45:20.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.522+0000 7f45bf400640 1 -- 172.21.15.31:0/2582857617 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45cc018950 con 0x55ea8049faa0 2024-03-21T10:45:20.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.522+0000 7f45bf400640 1 -- 172.21.15.31:0/2582857617 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc01e030 con 0x55ea8049faa0 2024-03-21T10:45:20.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.523+0000 7f45dca2c740 1 -- 172.21.15.31:0/2582857617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea8049faa0 msgr2=0x55ea8049e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:20.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.523+0000 7f45dca2c740 1 --2- 172.21.15.31:0/2582857617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea8049faa0 0x55ea8049e000 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f45cc004170 tx=0x7f45cc0042b0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.538+0000 7f45dca2c740 1 -- 172.21.15.31:0/2582857617 shutdown_connections 2024-03-21T10:45:20.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.538+0000 7f45dca2c740 1 --2- 172.21.15.31:0/2582857617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea8049faa0 0x55ea8049e000 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.539+0000 7f45dca2c740 1 --2- 172.21.15.31:0/2582857617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea80497410 0x55ea8049dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.539+0000 7f45dca2c740 1 --2- 172.21.15.31:0/2582857617 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea80496730 0x55ea804a0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.539+0000 7f45dca2c740 1 -- 172.21.15.31:0/2582857617 >> 172.21.15.31:0/2582857617 conn(0x55ea80455200 msgr2=0x55ea80490cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:20.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.539+0000 7f45dca2c740 1 -- 172.21.15.31:0/2582857617 shutdown_connections 2024-03-21T10:45:20.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.541+0000 7f45dca2c740 1 -- 172.21.15.31:0/2582857617 wait complete. 2024-03-21T10:45:20.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.542+0000 7f45dca2c740 1 -- start start 2024-03-21T10:45:20.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.544+0000 7f45dca2c740 1 Processor -- start 2024-03-21T10:45:20.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.549+0000 7f45dca2c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea80496730 0x55ea804a35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:20.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.550+0000 7f45dca2c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea80497410 0x55ea804a3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:20.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.550+0000 7f45dca2c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea8049faa0 0x55ea80509d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:20.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.550+0000 7f45dca2c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b400e430 con 0x55ea80497410 2024-03-21T10:45:20.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.550+0000 7f45dca2c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b400f710 con 0x55ea8049faa0 2024-03-21T10:45:20.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.550+0000 7f45dca2c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b400ea50 con 0x55ea80496730 2024-03-21T10:45:20.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.551+0000 7f45d3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea80496730 0x55ea804a35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:20.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.551+0000 7f45d3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea80496730 0x55ea804a35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:35118/0 (socket says 172.21.15.31:35118) 2024-03-21T10:45:20.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.551+0000 7f45d3400640 1 -- 172.21.15.31:0/1835581934 learned_addr learned my addr 172.21.15.31:0/1835581934 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:20.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.551+0000 7f45dca2c740 10 client.0 _ll_register_callbacks cb 0x7f45c00f4b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:20.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.552+0000 7f45d3400640 1 --2- 172.21.15.31:0/1835581934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea8049faa0 0x55ea80509d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:20.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.553+0000 7f45d3400640 1 -- 172.21.15.31:0/1835581934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea8049faa0 msgr2=0x55ea80509d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:20.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.553+0000 7f45d2000640 1 --2- 172.21.15.31:0/1835581934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea80497410 0x55ea804a3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:20.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.553+0000 7f45d3400640 1 --2- 172.21.15.31:0/1835581934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea8049faa0 0x55ea80509d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.553+0000 7f45d3400640 1 -- 172.21.15.31:0/1835581934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea80497410 msgr2=0x55ea804a3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:20.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.553+0000 7f45d3400640 1 --2- 172.21.15.31:0/1835581934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea80497410 0x55ea804a3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.553+0000 7f45d3400640 1 -- 172.21.15.31:0/1835581934 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ea8050a250 con 0x55ea80496730 2024-03-21T10:45:20.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.553+0000 7f45d2000640 1 --2- 172.21.15.31:0/1835581934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea80497410 0x55ea804a3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:45:20.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.554+0000 7f45bcc00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:45:20.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.554+0000 7f45d3400640 1 --2- 172.21.15.31:0/1835581934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea80496730 0x55ea804a35a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009af0 tx=0x7f45cc0021d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d6ab5475b9cbf77 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:20.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.554+0000 7f45bcc00640 1 -- 172.21.15.31:0/1835581934 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc047060 con 0x55ea80496730 2024-03-21T10:45:20.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.555+0000 7f45bcc00640 1 -- 172.21.15.31:0/1835581934 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f45cc018470 con 0x55ea80496730 2024-03-21T10:45:20.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.555+0000 7f45dca2c740 1 -- 172.21.15.31:0/1835581934 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f45cc018950 con 0x55ea80496730 2024-03-21T10:45:20.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.556+0000 7f45bcc00640 1 -- 172.21.15.31:0/1835581934 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45cc04e050 con 0x55ea80496730 2024-03-21T10:45:20.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.556+0000 7f45dca2c740 10 client.27229 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:20.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.556+0000 7f45dca2c740 1 -- 172.21.15.31:0/1835581934 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55ea8050b3a0 con 0x55ea80496730 2024-03-21T10:45:20.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.558+0000 7f45bcc00640 1 -- 172.21.15.31:0/1835581934 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45cc018950 con 0x55ea80496730 2024-03-21T10:45:20.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.558+0000 7f45bcc00640 1 -- 172.21.15.31:0/1835581934 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f45cc0104a0 con 0x55ea80496730 2024-03-21T10:45:20.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.559+0000 7f45dca2c740 10 client.27229 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:20.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.559+0000 7f45dca2c740 10 client.27229 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:45:20.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.559+0000 7f45dca2c740 20 client.27229 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:20.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.559+0000 7f45dca2c740 1 --2- 172.21.15.31:0/1835581934 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ea8050b910 0x55ea8050dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:20.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.559+0000 7f45dca2c740 4 client.27229 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:45:20.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.559+0000 7f45dca2c740 1 -- 172.21.15.31:0/1835581934 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ea803f9ae0 con 0x55ea8050b910 2024-03-21T10:45:20.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.559+0000 7f45d2000640 1 --2- 172.21.15.31:0/1835581934 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ea8050b910 0x55ea8050dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:20.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.560+0000 7f45bcc00640 10 client.27229 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:20.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.560+0000 7f45d2000640 1 --2- 172.21.15.31:0/1835581934 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ea8050b910 0x55ea8050dd00 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b83080d010165d1 server_cookie=495106c7a7e55df8 in_seq=0 out_seq=0 2024-03-21T10:45:20.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.561+0000 7f45bcc00640 1 -- 172.21.15.31:0/1835581934 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55ea803f9ae0 con 0x55ea8050b910 2024-03-21T10:45:20.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.561+0000 7f45bcc00640 10 client.27229 handle_command_reply: tid=0 2024-03-21T10:45:20.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.561+0000 7f45dca2c740 1 client.27229 shutdown 2024-03-21T10:45:20.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.562+0000 7f45dca2c740 1 -- 172.21.15.31:0/1835581934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea80496730 msgr2=0x55ea804a35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:20.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.562+0000 7f45dca2c740 1 --2- 172.21.15.31:0/1835581934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea80496730 0x55ea804a35a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009af0 tx=0x7f45cc0021d0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.562+0000 7f45dca2c740 1 -- 172.21.15.31:0/1835581934 shutdown_connections 2024-03-21T10:45:20.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.562+0000 7f45dca2c740 1 --2- 172.21.15.31:0/1835581934 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ea8050b910 0x55ea8050dd00 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.562+0000 7f45dca2c740 1 --2- 172.21.15.31:0/1835581934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea8049faa0 0x55ea80509d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.562+0000 7f45dca2c740 1 --2- 172.21.15.31:0/1835581934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea80497410 0x55ea804a3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.562+0000 7f45dca2c740 1 --2- 172.21.15.31:0/1835581934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea80496730 0x55ea804a35a0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.562+0000 7f45dca2c740 1 -- 172.21.15.31:0/1835581934 >> 172.21.15.31:0/1835581934 conn(0x55ea80455200 msgr2=0x55ea80487200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:20.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.562+0000 7f45bcc00640 0 client.27229 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:20.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.563+0000 7f45dca2c740 1 -- 172.21.15.31:0/1835581934 shutdown_connections 2024-03-21T10:45:20.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.563+0000 7f45dca2c740 1 -- 172.21.15.31:0/1835581934 wait complete. 2024-03-21T10:45:20.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.564+0000 7f45dca2c740 20 client.27229 trim_cache size 0 max 16384 2024-03-21T10:45:20.562 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:45:20.563 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-21T10:45:20.564 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-03-21T10:45:20.564 INFO:teuthology.orchestra.run.smithi031.stdout: "6ebaf4cd-bac5-41c8-a332-36ac1b2ce3c9": { 2024-03-21T10:45:20.564 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-03-21T10:45:20.564 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "6ebaf4cd-bac5-41c8-a332-36ac1b2ce3c9", 2024-03-21T10:45:20.564 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-03-21T10:45:20.565 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:45:20.565 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:45:20.565 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:45:20.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.566+0000 7f45d9c00640 1 -- 172.21.15.31:0/3909453736 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f45b80a3110 msgr2=0x7f45b80a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:20.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.566+0000 7f45d9c00640 1 --2- 172.21.15.31:0/3909453736 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f45b80a3110 0x7f45b80a55d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x55ea80497960 tx=0x7f45c00f4ca0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.566+0000 7f45d9c00640 1 -- 172.21.15.31:0/3909453736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45d4100440 msgr2=0x7f45d419a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:20.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.566+0000 7f45d9c00640 1 --2- 172.21.15.31:0/3909453736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45d4100440 0x7f45d419a160 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f45c002c2d0 tx=0x7f45c0004360 comp rx=0 tx=0).stop 2024-03-21T10:45:20.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.567+0000 7f45d9c00640 1 -- 172.21.15.31:0/3909453736 shutdown_connections 2024-03-21T10:45:20.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.567+0000 7f45d9c00640 1 --2- 172.21.15.31:0/3909453736 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f45b80a3110 0x7f45b80a55d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.567+0000 7f45d9c00640 1 --2- 172.21.15.31:0/3909453736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45d410dff0 0x7f45d419ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.567+0000 7f45d9c00640 1 --2- 172.21.15.31:0/3909453736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45d4100d60 0x7f45d419a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.567+0000 7f45d9c00640 1 --2- 172.21.15.31:0/3909453736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45d4100440 0x7f45d419a160 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:20.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.567+0000 7f45d9c00640 1 -- 172.21.15.31:0/3909453736 >> 172.21.15.31:0/3909453736 conn(0x7f45d4077750 msgr2=0x7f45d410f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:20.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.568+0000 7f45d9c00640 1 -- 172.21.15.31:0/3909453736 shutdown_connections 2024-03-21T10:45:20.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:20.568+0000 7f45d9c00640 1 -- 172.21.15.31:0/3909453736 wait complete. 2024-03-21T10:45:20.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:20 smithi070 ceph-mon[36330]: scrub queued for path: / 2024-03-21T10:45:20.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:20 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:45:20.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:20 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:45:20.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:20 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:45:20.612 INFO:tasks.cephfs.filesystem:scrub status for tag:6ebaf4cd-bac5-41c8-a332-36ac1b2ce3c9 - {'path': '/', 'tag': '6ebaf4cd-bac5-41c8-a332-36ac1b2ce3c9', 'options': 'recursive,force'} 2024-03-21T10:45:20.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:20 smithi040 ceph-mon[36154]: scrub queued for path: / 2024-03-21T10:45:20.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:20 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:45:20.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:20 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:45:20.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:20 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:45:21.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:21 smithi070 ceph-mon[36330]: pgmap v208: 129 pgs: 129 active+clean; 44 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 206 MiB/s wr, 1.63k op/s 2024-03-21T10:45:21.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:21 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:45:21.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:21 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:45:21.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:21 smithi040 ceph-mon[36154]: pgmap v208: 129 pgs: 129 active+clean; 44 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 206 MiB/s wr, 1.63k op/s 2024-03-21T10:45:21.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:21 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:45:21.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:21 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:45:21.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:21 smithi031 ceph-mon[31673]: pgmap v208: 129 pgs: 129 active+clean; 44 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 206 MiB/s wr, 1.63k op/s 2024-03-21T10:45:21.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:21 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:45:21.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:21 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:45:23.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:23 smithi040 ceph-mon[36154]: pgmap v209: 129 pgs: 129 active+clean; 44 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 201 MiB/s wr, 1.54k op/s 2024-03-21T10:45:23.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:23 smithi031 ceph-mon[31673]: pgmap v209: 129 pgs: 129 active+clean; 44 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 201 MiB/s wr, 1.54k op/s 2024-03-21T10:45:23.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:23 smithi070 ceph-mon[36330]: pgmap v209: 129 pgs: 129 active+clean; 44 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 201 MiB/s wr, 1.54k op/s 2024-03-21T10:45:24.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:24 smithi040 ceph-mon[36154]: pgmap v210: 129 pgs: 129 active+clean; 44 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 202 MiB/s wr, 1.43k op/s 2024-03-21T10:45:24.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:24 smithi031 ceph-mon[31673]: pgmap v210: 129 pgs: 129 active+clean; 44 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 202 MiB/s wr, 1.43k op/s 2024-03-21T10:45:24.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:24 smithi070 ceph-mon[36330]: pgmap v210: 129 pgs: 129 active+clean; 44 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 202 MiB/s wr, 1.43k op/s 2024-03-21T10:45:27.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:26 smithi040 ceph-mon[36154]: pgmap v211: 129 pgs: 129 active+clean; 44 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 223 MiB/s wr, 1.15k op/s 2024-03-21T10:45:27.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:26 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:45:27.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:26 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:45:27.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:26 smithi031 ceph-mon[31673]: pgmap v211: 129 pgs: 129 active+clean; 44 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 223 MiB/s wr, 1.15k op/s 2024-03-21T10:45:27.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:26 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:45:27.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:26 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:45:27.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:26 smithi070 ceph-mon[36330]: pgmap v211: 129 pgs: 129 active+clean; 44 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 223 MiB/s wr, 1.15k op/s 2024-03-21T10:45:27.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:26 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:45:27.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:26 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:45:29.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:28 smithi040 ceph-mon[36154]: pgmap v212: 129 pgs: 129 active+clean; 44 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 193 MiB/s wr, 956 op/s 2024-03-21T10:45:29.310 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:28 smithi031 ceph-mon[31673]: pgmap v212: 129 pgs: 129 active+clean; 44 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 193 MiB/s wr, 956 op/s 2024-03-21T10:45:29.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:28 smithi070 ceph-mon[36330]: pgmap v212: 129 pgs: 129 active+clean; 44 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 193 MiB/s wr, 956 op/s 2024-03-21T10:45:31.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:30 smithi040 ceph-mon[36154]: pgmap v213: 129 pgs: 129 active+clean; 45 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 252 MiB/s wr, 1.37k op/s 2024-03-21T10:45:31.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:30 smithi031 ceph-mon[31673]: pgmap v213: 129 pgs: 129 active+clean; 45 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 252 MiB/s wr, 1.37k op/s 2024-03-21T10:45:31.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:30 smithi070 ceph-mon[36330]: pgmap v213: 129 pgs: 129 active+clean; 45 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 252 MiB/s wr, 1.37k op/s 2024-03-21T10:45:32.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.912+0000 7f9c4b200640 1 -- 172.21.15.31:0/581263070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c4c105760 msgr2=0x7f9c4c10d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:32.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.912+0000 7f9c4b200640 1 --2- 172.21.15.31:0/581263070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c4c105760 0x7f9c4c10d2e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800dcc0 tx=0x7f9c3802fbe0 comp rx=0 tx=0).stop 2024-03-21T10:45:32.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.913+0000 7f9c4b200640 1 -- 172.21.15.31:0/581263070 shutdown_connections 2024-03-21T10:45:32.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.913+0000 7f9c4b200640 1 --2- 172.21.15.31:0/581263070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c4c105760 0x7f9c4c10d2e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:32.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.913+0000 7f9c4b200640 1 --2- 172.21.15.31:0/581263070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c4c101240 0x7f9c4c105030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:32.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.913+0000 7f9c4b200640 1 --2- 172.21.15.31:0/581263070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c4c100890 0x7f9c4c100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:32.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.913+0000 7f9c4b200640 1 -- 172.21.15.31:0/581263070 >> 172.21.15.31:0/581263070 conn(0x7f9c4c075f00 msgr2=0x7f9c4c076310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:32.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.914+0000 7f9c4b200640 1 -- 172.21.15.31:0/581263070 shutdown_connections 2024-03-21T10:45:32.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.914+0000 7f9c4b200640 1 -- 172.21.15.31:0/581263070 wait complete. 2024-03-21T10:45:32.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.916+0000 7f9c4b200640 1 Processor -- start 2024-03-21T10:45:32.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.918+0000 7f9c4b200640 1 -- start start 2024-03-21T10:45:32.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.919+0000 7f9c4b200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c4c100890 0x7f9c4c19c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:32.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.919+0000 7f9c4b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c4c105760 0x7f9c4c19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:32.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.919+0000 7f9c4b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c4c195950 0x7f9c4c195de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:32.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.921+0000 7f9c49e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c4c100890 0x7f9c4c19c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:32.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.921+0000 7f9c49e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c4c100890 0x7f9c4c19c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:39378/0 (socket says 172.21.15.31:39378) 2024-03-21T10:45:32.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.921+0000 7f9c4b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c10f6a0 con 0x7f9c4c195950 2024-03-21T10:45:32.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.921+0000 7f9c4b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c10f520 con 0x7f9c4c105760 2024-03-21T10:45:32.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.921+0000 7f9c4b200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c10f820 con 0x7f9c4c100890 2024-03-21T10:45:32.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.921+0000 7f9c49e00640 1 -- 172.21.15.31:0/4242924345 learned_addr learned my addr 172.21.15.31:0/4242924345 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:32.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.924+0000 7f9c4a800640 1 --2- 172.21.15.31:0/4242924345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c4c195950 0x7f9c4c195de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:32.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.924+0000 7f9c49e00640 1 -- 172.21.15.31:0/4242924345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c4c105760 msgr2=0x7f9c4c19cd20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T10:45:32.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.924+0000 7f9c49e00640 1 --2- 172.21.15.31:0/4242924345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c4c105760 0x7f9c4c19cd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:32.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.924+0000 7f9c49e00640 1 -- 172.21.15.31:0/4242924345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c4c195950 msgr2=0x7f9c4c195de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:32.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.924+0000 7f9c49e00640 1 --2- 172.21.15.31:0/4242924345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c4c195950 0x7f9c4c195de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:32.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.924+0000 7f9c49e00640 1 -- 172.21.15.31:0/4242924345 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c4c1966a0 con 0x7f9c4c100890 2024-03-21T10:45:32.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.925+0000 7f9c49e00640 1 --2- 172.21.15.31:0/4242924345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c4c100890 0x7f9c4c19c7e0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f9c4400eab0 tx=0x7f9c4400ef70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:32.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.926+0000 7f9c42a00640 1 -- 172.21.15.31:0/4242924345 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4400ce40 con 0x7f9c4c100890 2024-03-21T10:45:32.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.926+0000 7f9c4b200640 1 -- 172.21.15.31:0/4242924345 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c4c19a3c0 con 0x7f9c4c100890 2024-03-21T10:45:32.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.926+0000 7f9c4b200640 1 -- 172.21.15.31:0/4242924345 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c4c19a960 con 0x7f9c4c100890 2024-03-21T10:45:32.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.927+0000 7f9c42a00640 1 -- 172.21.15.31:0/4242924345 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c44011950 con 0x7f9c4c100890 2024-03-21T10:45:32.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.927+0000 7f9c42a00640 1 -- 172.21.15.31:0/4242924345 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c44010aa0 con 0x7f9c4c100890 2024-03-21T10:45:32.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.930+0000 7f9c53e60740 1 Processor -- start 2024-03-21T10:45:32.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.931+0000 7f9c53e60740 1 -- start start 2024-03-21T10:45:32.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.931+0000 7f9c53e60740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5567cf7fb1a0 0x5567cf7fb580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:32.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.931+0000 7f9c53e60740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567cf7fbac0 0x5567cf804350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:32.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.931+0000 7f9c53e60740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567cf804890 0x5567cf806c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:32.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.932+0000 7f9c49e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567cf7fbac0 0x5567cf804350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:32.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.932+0000 7f9c49e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567cf7fbac0 0x5567cf804350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:39392/0 (socket says 172.21.15.31:39392) 2024-03-21T10:45:32.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.933+0000 7f9c49400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567cf804890 0x5567cf806c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:32.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.933+0000 7f9c49400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567cf804890 0x5567cf806c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:48100/0 (socket says 172.21.15.31:48100) 2024-03-21T10:45:32.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.934+0000 7f9c4a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5567cf7fb1a0 0x5567cf7fb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:32.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.934+0000 7f9c4a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5567cf7fb1a0 0x5567cf7fb580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41558/0 (socket says 172.21.15.31:41558) 2024-03-21T10:45:32.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.934+0000 7f9c53e60740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567cf7bd3b0 con 0x5567cf7fb1a0 2024-03-21T10:45:32.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.935+0000 7f9c53e60740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567cf75f830 con 0x5567cf804890 2024-03-21T10:45:32.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.936+0000 7f9c53e60740 1 -- 172.21.15.31:0/2163456344 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567cf7085d0 con 0x5567cf7fbac0 2024-03-21T10:45:32.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.936+0000 7f9c49e00640 1 -- 172.21.15.31:0/2163456344 learned_addr learned my addr 172.21.15.31:0/2163456344 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:32.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.936+0000 7f9c49400640 1 -- 172.21.15.31:0/2163456344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567cf7fbac0 msgr2=0x5567cf804350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:32.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.936+0000 7f9c49400640 1 --2- 172.21.15.31:0/2163456344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567cf7fbac0 0x5567cf804350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.936+0000 7f9c49400640 1 -- 172.21.15.31:0/2163456344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5567cf7fb1a0 msgr2=0x5567cf7fb580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.936+0000 7f9c49400640 1 --2- 172.21.15.31:0/2163456344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5567cf7fb1a0 0x5567cf7fb580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.936+0000 7f9c49400640 1 -- 172.21.15.31:0/2163456344 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5567cf7202c0 con 0x5567cf804890 2024-03-21T10:45:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.939+0000 7f9c49400640 1 --2- 172.21.15.31:0/2163456344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567cf804890 0x5567cf806c80 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c005940 tx=0x7f9c3c00ad80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f0757c1da08579e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.939+0000 7f9c37400640 1 -- 172.21.15.31:0/2163456344 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c00cc90 con 0x5567cf804890 2024-03-21T10:45:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.939+0000 7f9c37400640 1 -- 172.21.15.31:0/2163456344 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c3c00ce30 con 0x5567cf804890 2024-03-21T10:45:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.939+0000 7f9c37400640 1 -- 172.21.15.31:0/2163456344 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c00f6b0 con 0x5567cf804890 2024-03-21T10:45:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.940+0000 7f9c53e60740 1 -- 172.21.15.31:0/2163456344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567cf804890 msgr2=0x5567cf806c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.940+0000 7f9c53e60740 1 --2- 172.21.15.31:0/2163456344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567cf804890 0x5567cf806c80 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c005940 tx=0x7f9c3c00ad80 comp rx=0 tx=0).stop 2024-03-21T10:45:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.940+0000 7f9c53e60740 1 -- 172.21.15.31:0/2163456344 shutdown_connections 2024-03-21T10:45:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.940+0000 7f9c53e60740 1 --2- 172.21.15.31:0/2163456344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567cf804890 0x5567cf806c80 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.940+0000 7f9c53e60740 1 --2- 172.21.15.31:0/2163456344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567cf7fbac0 0x5567cf804350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.941+0000 7f9c53e60740 1 --2- 172.21.15.31:0/2163456344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5567cf7fb1a0 0x5567cf7fb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.941+0000 7f9c53e60740 1 -- 172.21.15.31:0/2163456344 >> 172.21.15.31:0/2163456344 conn(0x5567cf7f5ac0 msgr2=0x5567cf7f5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:32.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.948+0000 7f9c42a00640 1 -- 172.21.15.31:0/4242924345 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f9c44010c40 con 0x7f9c4c100890 2024-03-21T10:45:32.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.953+0000 7f9c53e60740 1 -- 172.21.15.31:0/2163456344 shutdown_connections 2024-03-21T10:45:32.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.953+0000 7f9c53e60740 1 -- 172.21.15.31:0/2163456344 wait complete. 2024-03-21T10:45:32.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.955+0000 7f9c53e60740 1 -- start start 2024-03-21T10:45:32.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.955+0000 7f9c53e60740 1 Processor -- start 2024-03-21T10:45:32.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.958+0000 7f9c53e60740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567cf7fb1a0 0x5567cf801850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:32.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.958+0000 7f9c53e60740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567cf7fbac0 0x5567cf801d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:32.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.958+0000 7f9c53e60740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5567cf804890 0x5567cf8022d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:32.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.959+0000 7f9c4a800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567cf7fb1a0 0x5567cf801850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:32.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.959+0000 7f9c4a800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567cf7fb1a0 0x5567cf801850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:39408/0 (socket says 172.21.15.31:39408) 2024-03-21T10:45:32.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.959+0000 7f9c4a800640 1 -- 172.21.15.31:0/1135865972 learned_addr learned my addr 172.21.15.31:0/1135865972 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.959+0000 7f9c53e60740 1 -- 172.21.15.31:0/1135865972 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567cf71ba20 con 0x5567cf804890 2024-03-21T10:45:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.959+0000 7f9c53e60740 1 -- 172.21.15.31:0/1135865972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567cf75b8d0 con 0x5567cf7fbac0 2024-03-21T10:45:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.959+0000 7f9c53e60740 1 -- 172.21.15.31:0/1135865972 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567cf8074c0 con 0x5567cf7fb1a0 2024-03-21T10:45:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.959+0000 7f9c53e60740 10 client.0 _ll_register_callbacks cb 0x5567cf7581c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.959+0000 7f9c49e00640 1 --2- 172.21.15.31:0/1135865972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567cf7fbac0 0x5567cf801d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.960+0000 7f9c4a800640 1 -- 172.21.15.31:0/1135865972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567cf7fbac0 msgr2=0x5567cf801d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.960+0000 7f9c4a800640 1 --2- 172.21.15.31:0/1135865972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567cf7fbac0 0x5567cf801d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.960+0000 7f9c4a800640 1 -- 172.21.15.31:0/1135865972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5567cf804890 msgr2=0x5567cf8022d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.960+0000 7f9c4a800640 1 --2- 172.21.15.31:0/1135865972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5567cf804890 0x5567cf8022d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.960+0000 7f9c4a800640 1 -- 172.21.15.31:0/1135865972 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5567cf474210 con 0x5567cf7fb1a0 2024-03-21T10:45:32.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.961+0000 7f9c4a800640 1 --2- 172.21.15.31:0/1135865972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567cf7fb1a0 0x5567cf801850 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800dad0 tx=0x7f9c38004440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9104188083bb36f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:32.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.961+0000 7f9c42a00640 1 --2- 172.21.15.31:0/4242924345 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9c300a30e0 0x7f9c300a55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:32.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.961+0000 7f9c2be00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:45:32.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.961+0000 7f9c2be00640 1 -- 172.21.15.31:0/1135865972 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c38045060 con 0x5567cf7fb1a0 2024-03-21T10:45:32.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.961+0000 7f9c42a00640 1 -- 172.21.15.31:0/4242924345 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f9c440d0200 con 0x7f9c4c100890 2024-03-21T10:45:32.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.961+0000 7f9c2be00640 1 -- 172.21.15.31:0/1135865972 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f9c38003800 con 0x5567cf7fb1a0 2024-03-21T10:45:32.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.962+0000 7f9c53e60740 1 -- 172.21.15.31:0/1135865972 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9c3c00ce30 con 0x5567cf7fb1a0 2024-03-21T10:45:32.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.962+0000 7f9c2be00640 1 -- 172.21.15.31:0/1135865972 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c38003a00 con 0x5567cf7fb1a0 2024-03-21T10:45:32.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.962+0000 7f9c53e60740 10 client.27241 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:32.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.962+0000 7f9c53e60740 1 -- 172.21.15.31:0/1135865972 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5567cf540620 con 0x5567cf7fb1a0 2024-03-21T10:45:32.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.963+0000 7f9c2be00640 1 -- 172.21.15.31:0/1135865972 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c3c00ce30 con 0x5567cf7fb1a0 2024-03-21T10:45:32.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.963+0000 7f9c2be00640 1 -- 172.21.15.31:0/1135865972 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f9c38038400 con 0x5567cf7fb1a0 2024-03-21T10:45:32.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.963+0000 7f9c53e60740 10 client.27241 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:32.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.963+0000 7f9c53e60740 10 client.27241 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:45:32.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.963+0000 7f9c53e60740 20 client.27241 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:32.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.963+0000 7f9c53e60740 1 --2- 172.21.15.31:0/1135865972 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5567cf8146c0 0x5567cf816b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:32.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.964+0000 7f9c49e00640 1 --2- 172.21.15.31:0/4242924345 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9c300a30e0 0x7f9c300a55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:32.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.964+0000 7f9c53e60740 4 client.27241 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:45:32.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.964+0000 7f9c53e60740 1 -- 172.21.15.31:0/1135865972 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5567cf76cd70 con 0x5567cf8146c0 2024-03-21T10:45:32.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.964+0000 7f9c49400640 1 --2- 172.21.15.31:0/1135865972 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5567cf8146c0 0x5567cf816b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:32.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.965+0000 7f9c49e00640 1 --2- 172.21.15.31:0/4242924345 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9c300a30e0 0x7f9c300a55a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9c4401b000 tx=0x7f9c440d3cf0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:32.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.966+0000 7f9c49400640 1 --2- 172.21.15.31:0/1135865972 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5567cf8146c0 0x5567cf816b60 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1e661e0f0b2cc1 server_cookie=f7ae9cb672ae8073 in_seq=0 out_seq=0 2024-03-21T10:45:32.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.966+0000 7f9c2be00640 10 client.27241 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:32.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.968+0000 7f9c2be00640 1 -- 172.21.15.31:0/1135865972 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5567cf76cd70 con 0x5567cf8146c0 2024-03-21T10:45:32.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.968+0000 7f9c2be00640 10 client.27241 handle_command_reply: tid=0 2024-03-21T10:45:32.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.968+0000 7f9c53e60740 1 client.27241 shutdown 2024-03-21T10:45:32.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.969+0000 7f9c53e60740 1 -- 172.21.15.31:0/1135865972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567cf7fb1a0 msgr2=0x5567cf801850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:32.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.969+0000 7f9c53e60740 1 --2- 172.21.15.31:0/1135865972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567cf7fb1a0 0x5567cf801850 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800dad0 tx=0x7f9c38004440 comp rx=0 tx=0).stop 2024-03-21T10:45:32.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.970+0000 7f9c53e60740 1 -- 172.21.15.31:0/1135865972 shutdown_connections 2024-03-21T10:45:32.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.970+0000 7f9c53e60740 1 --2- 172.21.15.31:0/1135865972 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5567cf8146c0 0x5567cf816b60 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:32.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.971+0000 7f9c53e60740 1 --2- 172.21.15.31:0/1135865972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5567cf804890 0x5567cf8022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:32.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.971+0000 7f9c53e60740 1 --2- 172.21.15.31:0/1135865972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567cf7fbac0 0x5567cf801d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:32.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.971+0000 7f9c2be00640 0 client.27241 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:32.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.971+0000 7f9c53e60740 1 --2- 172.21.15.31:0/1135865972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567cf7fb1a0 0x5567cf801850 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:32.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.971+0000 7f9c53e60740 1 -- 172.21.15.31:0/1135865972 >> 172.21.15.31:0/1135865972 conn(0x5567cf7f5ac0 msgr2=0x5567cf7f7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:32.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.971+0000 7f9c53e60740 1 -- 172.21.15.31:0/1135865972 shutdown_connections 2024-03-21T10:45:32.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.971+0000 7f9c53e60740 1 -- 172.21.15.31:0/1135865972 wait complete. 2024-03-21T10:45:32.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:32.972+0000 7f9c53e60740 20 client.27241 trim_cache size 0 max 16384 2024-03-21T10:45:33.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.004+0000 7f9c53e60740 1 Processor -- start 2024-03-21T10:45:33.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.004+0000 7f9c53e60740 1 -- start start 2024-03-21T10:45:33.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.005+0000 7f9c53e60740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567cf7fb1a0 0x5567cf803f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:33.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.005+0000 7f9c4a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567cf7fb1a0 0x5567cf803f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:33.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.005+0000 7f9c4a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567cf7fb1a0 0x5567cf803f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:48104/0 (socket says 172.21.15.31:48104) 2024-03-21T10:45:33.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.005+0000 7f9c4a800640 1 -- 172.21.15.31:0/975381352 learned_addr learned my addr 172.21.15.31:0/975381352 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:33.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.005+0000 7f9c53e60740 1 --2- 172.21.15.31:0/975381352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5567cf7fbac0 0x5567cf7f77a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:33.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.006+0000 7f9c53e60740 1 --2- 172.21.15.31:0/975381352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567cf804890 0x5567cf80fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:33.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.006+0000 7f9c53e60740 1 -- 172.21.15.31:0/975381352 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c012350 con 0x5567cf7fbac0 2024-03-21T10:45:33.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.006+0000 7f9c53e60740 1 -- 172.21.15.31:0/975381352 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c00fe50 con 0x5567cf7fb1a0 2024-03-21T10:45:33.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.006+0000 7f9c53e60740 1 -- 172.21.15.31:0/975381352 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c00df10 con 0x5567cf804890 2024-03-21T10:45:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.007+0000 7f9c49400640 1 --2- 172.21.15.31:0/975381352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5567cf7fbac0 0x5567cf7f77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.007+0000 7f9c4a800640 1 --2- 172.21.15.31:0/975381352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567cf804890 0x5567cf80fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.007+0000 7f9c4a800640 1 -- 172.21.15.31:0/975381352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567cf804890 msgr2=0x5567cf80fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:33.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.007+0000 7f9c4a800640 1 --2- 172.21.15.31:0/975381352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567cf804890 0x5567cf80fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:33.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.007+0000 7f9c4a800640 1 -- 172.21.15.31:0/975381352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5567cf7fbac0 msgr2=0x5567cf7f77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:33.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.007+0000 7f9c4a800640 1 --2- 172.21.15.31:0/975381352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5567cf7fbac0 0x5567cf7f77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:33.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.007+0000 7f9c4a800640 1 -- 172.21.15.31:0/975381352 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5567cf800880 con 0x5567cf7fb1a0 2024-03-21T10:45:33.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.007+0000 7f9c4a800640 1 --2- 172.21.15.31:0/975381352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567cf7fb1a0 0x5567cf803f80 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800dcc0 tx=0x7f9c38030380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aaec3886c937e6e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:33.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.008+0000 7f9c36a00640 1 -- 172.21.15.31:0/975381352 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c38045070 con 0x5567cf7fb1a0 2024-03-21T10:45:33.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.008+0000 7f9c36a00640 1 -- 172.21.15.31:0/975381352 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c38005500 con 0x5567cf7fb1a0 2024-03-21T10:45:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.009+0000 7f9c36a00640 1 -- 172.21.15.31:0/975381352 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c38045070 con 0x5567cf7fb1a0 2024-03-21T10:45:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.009+0000 7f9c53e60740 1 -- 172.21.15.31:0/975381352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567cf7fb1a0 msgr2=0x5567cf803f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.009+0000 7f9c53e60740 1 --2- 172.21.15.31:0/975381352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567cf7fb1a0 0x5567cf803f80 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800dcc0 tx=0x7f9c38030380 comp rx=0 tx=0).stop 2024-03-21T10:45:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.009+0000 7f9c53e60740 1 -- 172.21.15.31:0/975381352 shutdown_connections 2024-03-21T10:45:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.009+0000 7f9c53e60740 1 --2- 172.21.15.31:0/975381352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567cf804890 0x5567cf80fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.009+0000 7f9c53e60740 1 --2- 172.21.15.31:0/975381352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5567cf7fbac0 0x5567cf7f77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.009+0000 7f9c53e60740 1 --2- 172.21.15.31:0/975381352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567cf7fb1a0 0x5567cf803f80 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.009+0000 7f9c53e60740 1 -- 172.21.15.31:0/975381352 >> 172.21.15.31:0/975381352 conn(0x5567cf7f5ac0 msgr2=0x5567cf7f5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.009+0000 7f9c53e60740 1 -- 172.21.15.31:0/975381352 shutdown_connections 2024-03-21T10:45:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.010+0000 7f9c53e60740 1 -- 172.21.15.31:0/975381352 wait complete. 2024-03-21T10:45:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.010+0000 7f9c53e60740 1 -- start start 2024-03-21T10:45:33.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.010+0000 7f9c53e60740 1 Processor -- start 2024-03-21T10:45:33.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.012+0000 7f9c53e60740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567cf7fb1a0 0x5567cf7ff5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:33.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.012+0000 7f9c4a800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567cf7fb1a0 0x5567cf7ff5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:33.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.012+0000 7f9c4a800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567cf7fb1a0 0x5567cf7ff5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:39424/0 (socket says 172.21.15.31:39424) 2024-03-21T10:45:33.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.012+0000 7f9c53e60740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5567cf7fbac0 0x5567cf809740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:33.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.012+0000 7f9c4a800640 1 -- 172.21.15.31:0/1018274797 learned_addr learned my addr 172.21.15.31:0/1018274797 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:33.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.013+0000 7f9c53e60740 1 --2- 172.21.15.31:0/1018274797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567cf804890 0x5567cf809c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:33.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.013+0000 7f9c53e60740 1 -- 172.21.15.31:0/1018274797 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c00e510 con 0x5567cf7fbac0 2024-03-21T10:45:33.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.013+0000 7f9c53e60740 1 -- 172.21.15.31:0/1018274797 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c00f7f0 con 0x5567cf804890 2024-03-21T10:45:33.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.013+0000 7f9c53e60740 1 -- 172.21.15.31:0/1018274797 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c00eb30 con 0x5567cf7fb1a0 2024-03-21T10:45:33.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.013+0000 7f9c53e60740 10 client.0 _ll_register_callbacks cb 0x7f9c3800db90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:33.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.013+0000 7f9c49400640 1 --2- 172.21.15.31:0/1018274797 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5567cf7fbac0 0x5567cf809740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:33.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.013+0000 7f9c4a800640 1 --2- 172.21.15.31:0/1018274797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567cf804890 0x5567cf809c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:33.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.013+0000 7f9c4a800640 1 -- 172.21.15.31:0/1018274797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567cf804890 msgr2=0x5567cf809c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:33.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.013+0000 7f9c4a800640 1 --2- 172.21.15.31:0/1018274797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567cf804890 0x5567cf809c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:33.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.013+0000 7f9c4a800640 1 -- 172.21.15.31:0/1018274797 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5567cf7fbac0 msgr2=0x5567cf809740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:33.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.014+0000 7f9c4a800640 1 --2- 172.21.15.31:0/1018274797 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5567cf7fbac0 0x5567cf809740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:33.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.014+0000 7f9c4a800640 1 -- 172.21.15.31:0/1018274797 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5567cf806ff0 con 0x5567cf7fb1a0 2024-03-21T10:45:33.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.014+0000 7f9c49400640 1 --2- 172.21.15.31:0/1018274797 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5567cf7fbac0 0x5567cf809740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:45:33.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.014+0000 7f9c4a800640 1 --2- 172.21.15.31:0/1018274797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567cf7fb1a0 0x5567cf7ff5f0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f9c380059d0 tx=0x7f9c38038bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f5df0c6035f49d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:33.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.014+0000 7f9c2be00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:45:33.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.014+0000 7f9c2be00640 1 -- 172.21.15.31:0/1018274797 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c38045070 con 0x5567cf7fb1a0 2024-03-21T10:45:33.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.014+0000 7f9c2be00640 1 -- 172.21.15.31:0/1018274797 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f9c380307e0 con 0x5567cf7fb1a0 2024-03-21T10:45:33.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.016+0000 7f9c53e60740 1 -- 172.21.15.31:0/1018274797 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9c38005500 con 0x5567cf7fb1a0 2024-03-21T10:45:33.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.018+0000 7f9c2be00640 1 -- 172.21.15.31:0/1018274797 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c3804f080 con 0x5567cf7fb1a0 2024-03-21T10:45:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.025+0000 7f9c53e60740 10 client.27247 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.025+0000 7f9c53e60740 1 -- 172.21.15.31:0/1018274797 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5567cf870e10 con 0x5567cf7fb1a0 2024-03-21T10:45:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.027+0000 7f9c2be00640 1 -- 172.21.15.31:0/1018274797 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c38005500 con 0x5567cf7fb1a0 2024-03-21T10:45:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.027+0000 7f9c2be00640 1 -- 172.21.15.31:0/1018274797 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f9c38040410 con 0x5567cf7fb1a0 2024-03-21T10:45:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.027+0000 7f9c53e60740 10 client.27247 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.027+0000 7f9c53e60740 10 client.27247 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:45:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.027+0000 7f9c53e60740 20 client.27247 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.028+0000 7f9c53e60740 1 --2- 172.21.15.31:0/1018274797 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5567cf8713c0 0x5567cf8737b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.028+0000 7f9c53e60740 4 client.27247 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:45:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.028+0000 7f9c53e60740 1 -- 172.21.15.31:0/1018274797 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5567cf7670c0 con 0x5567cf8713c0 2024-03-21T10:45:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.028+0000 7f9c49400640 1 --2- 172.21.15.31:0/1018274797 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5567cf8713c0 0x5567cf8737b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.033+0000 7f9c2be00640 10 client.27247 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.033+0000 7f9c49400640 1 --2- 172.21.15.31:0/1018274797 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5567cf8713c0 0x5567cf8737b0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b56be416a7039c8 server_cookie=89db8b00b199c898 in_seq=0 out_seq=0 2024-03-21T10:45:33.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.168+0000 7f9c2be00640 1 -- 172.21.15.31:0/1018274797 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5567cf7670c0 con 0x5567cf8713c0 2024-03-21T10:45:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.169+0000 7f9c2be00640 10 client.27247 handle_command_reply: tid=0 2024-03-21T10:45:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.170+0000 7f9c53e60740 1 client.27247 shutdown 2024-03-21T10:45:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.170+0000 7f9c53e60740 1 -- 172.21.15.31:0/1018274797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567cf7fb1a0 msgr2=0x5567cf7ff5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.170+0000 7f9c53e60740 1 --2- 172.21.15.31:0/1018274797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567cf7fb1a0 0x5567cf7ff5f0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f9c380059d0 tx=0x7f9c38038bd0 comp rx=0 tx=0).stop 2024-03-21T10:45:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.170+0000 7f9c53e60740 1 -- 172.21.15.31:0/1018274797 shutdown_connections 2024-03-21T10:45:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.170+0000 7f9c53e60740 1 --2- 172.21.15.31:0/1018274797 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5567cf8713c0 0x5567cf8737b0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.171+0000 7f9c2be00640 0 client.27247 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.171+0000 7f9c53e60740 1 --2- 172.21.15.31:0/1018274797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567cf804890 0x5567cf809c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.171+0000 7f9c53e60740 1 --2- 172.21.15.31:0/1018274797 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5567cf7fbac0 0x5567cf809740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.171+0000 7f9c53e60740 1 --2- 172.21.15.31:0/1018274797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567cf7fb1a0 0x5567cf7ff5f0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.171+0000 7f9c53e60740 1 -- 172.21.15.31:0/1018274797 >> 172.21.15.31:0/1018274797 conn(0x5567cf7f5ac0 msgr2=0x5567cf7f77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.171+0000 7f9c53e60740 1 -- 172.21.15.31:0/1018274797 shutdown_connections 2024-03-21T10:45:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.171+0000 7f9c53e60740 1 -- 172.21.15.31:0/1018274797 wait complete. 2024-03-21T10:45:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.172+0000 7f9c53e60740 20 client.27247 trim_cache size 0 max 16384 2024-03-21T10:45:33.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:45:33.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:45:33.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:45:33.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:45:33.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.177+0000 7f9c4b200640 1 -- 172.21.15.31:0/4242924345 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9c300a30e0 msgr2=0x7f9c300a55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:33.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.177+0000 7f9c4b200640 1 --2- 172.21.15.31:0/4242924345 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9c300a30e0 0x7f9c300a55a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9c4401b000 tx=0x7f9c440d3cf0 comp rx=0 tx=0).stop 2024-03-21T10:45:33.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.177+0000 7f9c4b200640 1 -- 172.21.15.31:0/4242924345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c4c100890 msgr2=0x7f9c4c19c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:33.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.177+0000 7f9c4b200640 1 --2- 172.21.15.31:0/4242924345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c4c100890 0x7f9c4c19c7e0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f9c4400eab0 tx=0x7f9c4400ef70 comp rx=0 tx=0).stop 2024-03-21T10:45:33.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.178+0000 7f9c4b200640 1 -- 172.21.15.31:0/4242924345 shutdown_connections 2024-03-21T10:45:33.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.178+0000 7f9c4b200640 1 --2- 172.21.15.31:0/4242924345 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9c300a30e0 0x7f9c300a55a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:33.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.178+0000 7f9c4b200640 1 --2- 172.21.15.31:0/4242924345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c4c195950 0x7f9c4c195de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:33.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.178+0000 7f9c4b200640 1 --2- 172.21.15.31:0/4242924345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c4c105760 0x7f9c4c19cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:33.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.178+0000 7f9c4b200640 1 --2- 172.21.15.31:0/4242924345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c4c100890 0x7f9c4c19c7e0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:33.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.178+0000 7f9c4b200640 1 -- 172.21.15.31:0/4242924345 >> 172.21.15.31:0/4242924345 conn(0x7f9c4c075f00 msgr2=0x7f9c4c0ff790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:33.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.179+0000 7f9c4b200640 1 -- 172.21.15.31:0/4242924345 shutdown_connections 2024-03-21T10:45:33.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:33.182+0000 7f9c4b200640 1 -- 172.21.15.31:0/4242924345 wait complete. 2024-03-21T10:45:33.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:32 smithi040 ceph-mon[36154]: pgmap v214: 129 pgs: 129 active+clean; 45 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 189 MiB/s wr, 1.03k op/s 2024-03-21T10:45:33.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:32 smithi031 ceph-mon[31673]: pgmap v214: 129 pgs: 129 active+clean; 45 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 189 MiB/s wr, 1.03k op/s 2024-03-21T10:45:33.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:32 smithi070 ceph-mon[36330]: pgmap v214: 129 pgs: 129 active+clean; 45 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 189 MiB/s wr, 1.03k op/s 2024-03-21T10:45:35.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:34 smithi031 ceph-mon[31673]: pgmap v215: 129 pgs: 129 active+clean; 45 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 198 MiB/s wr, 1.11k op/s 2024-03-21T10:45:35.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:34 smithi070 ceph-mon[36330]: pgmap v215: 129 pgs: 129 active+clean; 45 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 198 MiB/s wr, 1.11k op/s 2024-03-21T10:45:35.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:34 smithi040 ceph-mon[36154]: pgmap v215: 129 pgs: 129 active+clean; 45 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 198 MiB/s wr, 1.11k op/s 2024-03-21T10:45:37.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:36 smithi031 ceph-mon[31673]: pgmap v216: 129 pgs: 129 active+clean; 46 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 239 MiB/s wr, 1.45k op/s 2024-03-21T10:45:37.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:36 smithi070 ceph-mon[36330]: pgmap v216: 129 pgs: 129 active+clean; 46 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 239 MiB/s wr, 1.45k op/s 2024-03-21T10:45:37.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:36 smithi040 ceph-mon[36154]: pgmap v216: 129 pgs: 129 active+clean; 46 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 239 MiB/s wr, 1.45k op/s 2024-03-21T10:45:39.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:39 smithi031 ceph-mon[31673]: pgmap v217: 129 pgs: 129 active+clean; 46 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 156 MiB/s wr, 1.04k op/s 2024-03-21T10:45:39.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:39 smithi070 ceph-mon[36330]: pgmap v217: 129 pgs: 129 active+clean; 46 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 156 MiB/s wr, 1.04k op/s 2024-03-21T10:45:39.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:39 smithi040 ceph-mon[36154]: pgmap v217: 129 pgs: 129 active+clean; 46 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 156 MiB/s wr, 1.04k op/s 2024-03-21T10:45:41.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:41 smithi031 ceph-mon[31673]: pgmap v218: 129 pgs: 129 active+clean; 47 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 225 MiB/s wr, 1.51k op/s 2024-03-21T10:45:41.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:41 smithi070 ceph-mon[36330]: pgmap v218: 129 pgs: 129 active+clean; 47 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 225 MiB/s wr, 1.51k op/s 2024-03-21T10:45:41.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:41 smithi040 ceph-mon[36154]: pgmap v218: 129 pgs: 129 active+clean; 47 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 225 MiB/s wr, 1.51k op/s 2024-03-21T10:45:43.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:43 smithi031 ceph-mon[31673]: pgmap v219: 129 pgs: 129 active+clean; 47 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 164 MiB/s wr, 1.10k op/s 2024-03-21T10:45:43.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:43 smithi070 ceph-mon[36330]: pgmap v219: 129 pgs: 129 active+clean; 47 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 164 MiB/s wr, 1.10k op/s 2024-03-21T10:45:43.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:43 smithi040 ceph-mon[36154]: pgmap v219: 129 pgs: 129 active+clean; 47 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 164 MiB/s wr, 1.10k op/s 2024-03-21T10:45:45.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:45 smithi031 ceph-mon[31673]: pgmap v220: 129 pgs: 129 active+clean; 47 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 183 MiB/s wr, 1.21k op/s 2024-03-21T10:45:45.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:45 smithi070 ceph-mon[36330]: pgmap v220: 129 pgs: 129 active+clean; 47 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 183 MiB/s wr, 1.21k op/s 2024-03-21T10:45:45.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:45 smithi040 ceph-mon[36154]: pgmap v220: 129 pgs: 129 active+clean; 47 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 183 MiB/s wr, 1.21k op/s 2024-03-21T10:45:46.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.476+0000 7f2488200640 1 -- 172.21.15.31:0/2043971324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2480068f40 msgr2=0x7f2480069320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.476+0000 7f2488200640 1 --2- 172.21.15.31:0/2043971324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2480068f40 0x7f2480069320 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f2474009900 tx=0x7f247402bc30 comp rx=0 tx=0).stop 2024-03-21T10:45:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.476+0000 7f2488200640 1 -- 172.21.15.31:0/2043971324 shutdown_connections 2024-03-21T10:45:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.476+0000 7f2488200640 1 --2- 172.21.15.31:0/2043971324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f248010e040 0x7f2480110430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.476+0000 7f2488200640 1 --2- 172.21.15.31:0/2043971324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2480069860 0x7f248010db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.476+0000 7f2488200640 1 --2- 172.21.15.31:0/2043971324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2480068f40 0x7f2480069320 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.476+0000 7f2488200640 1 -- 172.21.15.31:0/2043971324 >> 172.21.15.31:0/2043971324 conn(0x7f2480077690 msgr2=0x7f2480077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.476+0000 7f2488200640 1 -- 172.21.15.31:0/2043971324 shutdown_connections 2024-03-21T10:45:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.476+0000 7f2488200640 1 -- 172.21.15.31:0/2043971324 wait complete. 2024-03-21T10:45:46.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.478+0000 7f2488200640 1 Processor -- start 2024-03-21T10:45:46.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.478+0000 7f2488200640 1 -- start start 2024-03-21T10:45:46.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.478+0000 7f2488200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2480068f40 0x7f248019a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:46.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.478+0000 7f2488200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2480069860 0x7f248019a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:46.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.478+0000 7f2488200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f248010e040 0x7f248019e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:46.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.478+0000 7f2488200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2480113c70 con 0x7f2480068f40 2024-03-21T10:45:46.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.478+0000 7f2488200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2480113af0 con 0x7f2480069860 2024-03-21T10:45:46.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.478+0000 7f2488200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2480113df0 con 0x7f248010e040 2024-03-21T10:45:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.479+0000 7f2485800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2480068f40 0x7f248019a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.479+0000 7f2484e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2480069860 0x7f248019a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.479+0000 7f2485800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2480068f40 0x7f248019a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:32924/0 (socket says 172.21.15.31:32924) 2024-03-21T10:45:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.479+0000 7f2484e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2480069860 0x7f248019a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37996/0 (socket says 172.21.15.31:37996) 2024-03-21T10:45:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.479+0000 7f2485800640 1 -- 172.21.15.31:0/2828704144 learned_addr learned my addr 172.21.15.31:0/2828704144 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.479+0000 7f2486200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f248010e040 0x7f248019e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.479+0000 7f2484e00640 1 -- 172.21.15.31:0/2828704144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f248010e040 msgr2=0x7f248019e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.479+0000 7f2484e00640 1 --2- 172.21.15.31:0/2828704144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f248010e040 0x7f248019e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.479+0000 7f2484e00640 1 -- 172.21.15.31:0/2828704144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2480068f40 msgr2=0x7f248019a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.479+0000 7f2484e00640 1 --2- 172.21.15.31:0/2828704144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2480068f40 0x7f248019a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.479+0000 7f2484e00640 1 -- 172.21.15.31:0/2828704144 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248019f0e0 con 0x7f2480069860 2024-03-21T10:45:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.480+0000 7f2484e00640 1 --2- 172.21.15.31:0/2828704144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2480069860 0x7f248019a630 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f246c00e9f0 tx=0x7f246c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.480+0000 7f247e000640 1 -- 172.21.15.31:0/2828704144 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c00ccc0 con 0x7f2480069860 2024-03-21T10:45:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.480+0000 7f247e000640 1 -- 172.21.15.31:0/2828704144 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f246c010910 con 0x7f2480069860 2024-03-21T10:45:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.480+0000 7f247e000640 1 -- 172.21.15.31:0/2828704144 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c018800 con 0x7f2480069860 2024-03-21T10:45:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.480+0000 7f2488200640 1 -- 172.21.15.31:0/2828704144 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f248019f3d0 con 0x7f2480069860 2024-03-21T10:45:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.480+0000 7f2488200640 1 -- 172.21.15.31:0/2828704144 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24801a6c60 con 0x7f2480069860 2024-03-21T10:45:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.484+0000 7f248ae44740 1 Processor -- start 2024-03-21T10:45:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.485+0000 7f248ae44740 1 -- start start 2024-03-21T10:45:46.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.485+0000 7f248ae44740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560faf60ffd0 0x560faf6103b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:46.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.485+0000 7f248ae44740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560faf6108f0 0x560faf619180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:46.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.485+0000 7f248ae44740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf6196c0 0x560faf61bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:46.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.485+0000 7f248ae44740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560faf5d23b0 con 0x560faf6108f0 2024-03-21T10:45:46.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.485+0000 7f248ae44740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560faf574830 con 0x560faf6196c0 2024-03-21T10:45:46.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.485+0000 7f248ae44740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x560faf51d5d0 con 0x560faf60ffd0 2024-03-21T10:45:46.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.485+0000 7f2486200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560faf60ffd0 0x560faf6103b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:46.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.485+0000 7f2486200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560faf60ffd0 0x560faf6103b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:41144/0 (socket says 172.21.15.31:41144) 2024-03-21T10:45:46.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.485+0000 7f2486200640 1 -- 172.21.15.31:0/315951545 learned_addr learned my addr 172.21.15.31:0/315951545 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:46.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.486+0000 7f2485800640 1 --2- 172.21.15.31:0/315951545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560faf6108f0 0x560faf619180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:46.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.486+0000 7f2486200640 1 -- 172.21.15.31:0/315951545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf6196c0 msgr2=0x560faf61bab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:45:46.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.486+0000 7f2486200640 1 --2- 172.21.15.31:0/315951545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf6196c0 0x560faf61bab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.486+0000 7f2486200640 1 -- 172.21.15.31:0/315951545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560faf6108f0 msgr2=0x560faf619180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:46.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.486+0000 7f2486200640 1 --2- 172.21.15.31:0/315951545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560faf6108f0 0x560faf619180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.486+0000 7f2486200640 1 -- 172.21.15.31:0/315951545 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560faf5352c0 con 0x560faf60ffd0 2024-03-21T10:45:46.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.486+0000 7f247e000640 1 -- 172.21.15.31:0/2828704144 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f246c010ac0 con 0x7f2480069860 2024-03-21T10:45:46.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.486+0000 7f2486200640 1 --2- 172.21.15.31:0/315951545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560faf60ffd0 0x560faf6103b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2478009f60 tx=0x7f2478009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e022d6a61decb607 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:46.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.487+0000 7f2472a00640 1 -- 172.21.15.31:0/315951545 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247801e070 con 0x560faf60ffd0 2024-03-21T10:45:46.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.487+0000 7f2472a00640 1 -- 172.21.15.31:0/315951545 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2478003e50 con 0x560faf60ffd0 2024-03-21T10:45:46.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.487+0000 7f2472a00640 1 -- 172.21.15.31:0/315951545 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247800f7a0 con 0x560faf60ffd0 2024-03-21T10:45:46.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.487+0000 7f248ae44740 1 -- 172.21.15.31:0/315951545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560faf60ffd0 msgr2=0x560faf6103b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:46.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.487+0000 7f248ae44740 1 --2- 172.21.15.31:0/315951545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560faf60ffd0 0x560faf6103b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2478009f60 tx=0x7f2478009f90 comp rx=0 tx=0).stop 2024-03-21T10:45:46.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.487+0000 7f248ae44740 1 -- 172.21.15.31:0/315951545 shutdown_connections 2024-03-21T10:45:46.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.488+0000 7f248ae44740 1 --2- 172.21.15.31:0/315951545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf6196c0 0x560faf61bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.488+0000 7f248ae44740 1 --2- 172.21.15.31:0/315951545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560faf6108f0 0x560faf619180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.488+0000 7f248ae44740 1 --2- 172.21.15.31:0/315951545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560faf60ffd0 0x560faf6103b0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.488+0000 7f248ae44740 1 -- 172.21.15.31:0/315951545 >> 172.21.15.31:0/315951545 conn(0x560faf60a8f0 msgr2=0x560faf60acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:46.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.488+0000 7f248ae44740 1 -- 172.21.15.31:0/315951545 shutdown_connections 2024-03-21T10:45:46.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.488+0000 7f248ae44740 1 -- 172.21.15.31:0/315951545 wait complete. 2024-03-21T10:45:46.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.488+0000 7f248ae44740 1 -- start start 2024-03-21T10:45:46.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.488+0000 7f247e000640 1 --2- 172.21.15.31:0/2828704144 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24680a2f90 0x7f24680a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:46.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.489+0000 7f248ae44740 1 Processor -- start 2024-03-21T10:45:46.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.489+0000 7f247e000640 1 -- 172.21.15.31:0/2828704144 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f246c0cff60 con 0x7f2480069860 2024-03-21T10:45:46.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.489+0000 7f2486200640 1 --2- 172.21.15.31:0/2828704144 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24680a2f90 0x7f24680a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.489+0000 7f248ae44740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf60ffd0 0x560faf6167d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.489+0000 7f248ae44740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560faf6108f0 0x560faf616d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.490+0000 7f248ae44740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560faf6196c0 0x560faf617250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.490+0000 7f248ae44740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560faf530a20 con 0x560faf6108f0 2024-03-21T10:45:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.490+0000 7f248ae44740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560faf5708d0 con 0x560faf60ffd0 2024-03-21T10:45:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.490+0000 7f248ae44740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x560faf61c2f0 con 0x560faf6196c0 2024-03-21T10:45:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.490+0000 7f248ae44740 10 client.0 _ll_register_callbacks cb 0x560faf56d1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.490+0000 7f2485800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf60ffd0 0x560faf6167d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.490+0000 7f2485800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf60ffd0 0x560faf6167d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38002/0 (socket says 172.21.15.31:38002) 2024-03-21T10:45:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.490+0000 7f2485800640 1 -- 172.21.15.31:0/337789820 learned_addr learned my addr 172.21.15.31:0/337789820 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.490+0000 7f2486200640 1 --2- 172.21.15.31:0/2828704144 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24680a2f90 0x7f24680a5450 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2478009d70 tx=0x7f2478005fd0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.490+0000 7f2484e00640 1 --2- 172.21.15.31:0/337789820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560faf6108f0 0x560faf616d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.490+0000 7f2485800640 1 -- 172.21.15.31:0/337789820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560faf6196c0 msgr2=0x560faf617250 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:45:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.490+0000 7f2485800640 1 --2- 172.21.15.31:0/337789820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560faf6196c0 0x560faf617250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.490+0000 7f2485800640 1 -- 172.21.15.31:0/337789820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560faf6108f0 msgr2=0x560faf616d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.490+0000 7f2485800640 1 --2- 172.21.15.31:0/337789820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560faf6108f0 0x560faf616d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.490+0000 7f2485800640 1 -- 172.21.15.31:0/337789820 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560faf289210 con 0x560faf60ffd0 2024-03-21T10:45:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.490+0000 7f2484e00640 1 --2- 172.21.15.31:0/337789820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560faf6108f0 0x560faf616d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:45:46.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.491+0000 7f2485800640 1 --2- 172.21.15.31:0/337789820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf60ffd0 0x560faf6167d0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f247402c800 tx=0x7f2474005a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3580d7a8e59a8c08 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:46.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.491+0000 7f2467400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:45:46.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.491+0000 7f2467400640 1 -- 172.21.15.31:0/337789820 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247402ca90 con 0x560faf60ffd0 2024-03-21T10:45:46.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.491+0000 7f2467400640 1 -- 172.21.15.31:0/337789820 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f247402cc30 con 0x560faf60ffd0 2024-03-21T10:45:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.491+0000 7f248ae44740 1 -- 172.21.15.31:0/337789820 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2478003e50 con 0x560faf60ffd0 2024-03-21T10:45:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.491+0000 7f2467400640 1 -- 172.21.15.31:0/337789820 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24740046c0 con 0x560faf60ffd0 2024-03-21T10:45:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.491+0000 7f248ae44740 10 client.27338 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.491+0000 7f248ae44740 1 -- 172.21.15.31:0/337789820 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x560faf355620 con 0x560faf60ffd0 2024-03-21T10:45:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.491+0000 7f2467400640 1 -- 172.21.15.31:0/337789820 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2478003e50 con 0x560faf60ffd0 2024-03-21T10:45:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.492+0000 7f2467400640 1 -- 172.21.15.31:0/337789820 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f24740035f0 con 0x560faf60ffd0 2024-03-21T10:45:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.492+0000 7f248ae44740 10 client.27338 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.492+0000 7f248ae44740 10 client.27338 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:45:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.492+0000 7f248ae44740 20 client.27338 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.492+0000 7f248ae44740 1 --2- 172.21.15.31:0/337789820 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560faf629780 0x560faf62bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.492+0000 7f248ae44740 4 client.27338 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:45:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.492+0000 7f248ae44740 1 -- 172.21.15.31:0/337789820 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560faf581d70 con 0x560faf629780 2024-03-21T10:45:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.492+0000 7f2484e00640 1 --2- 172.21.15.31:0/337789820 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560faf629780 0x560faf62bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.492+0000 7f2467400640 10 client.27338 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.493+0000 7f2484e00640 1 --2- 172.21.15.31:0/337789820 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560faf629780 0x560faf62bb70 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5f9e1adb8bab246 server_cookie=248600b02116eba3 in_seq=0 out_seq=0 2024-03-21T10:45:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.494+0000 7f2467400640 1 -- 172.21.15.31:0/337789820 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560faf581d70 con 0x560faf629780 2024-03-21T10:45:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.494+0000 7f2467400640 10 client.27338 handle_command_reply: tid=0 2024-03-21T10:45:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.494+0000 7f248ae44740 1 client.27338 shutdown 2024-03-21T10:45:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.494+0000 7f248ae44740 1 -- 172.21.15.31:0/337789820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf60ffd0 msgr2=0x560faf6167d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.494+0000 7f248ae44740 1 --2- 172.21.15.31:0/337789820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf60ffd0 0x560faf6167d0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f247402c800 tx=0x7f2474005a60 comp rx=0 tx=0).stop 2024-03-21T10:45:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.494+0000 7f248ae44740 1 -- 172.21.15.31:0/337789820 shutdown_connections 2024-03-21T10:45:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.494+0000 7f248ae44740 1 --2- 172.21.15.31:0/337789820 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560faf629780 0x560faf62bb70 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.494+0000 7f248ae44740 1 --2- 172.21.15.31:0/337789820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560faf6196c0 0x560faf617250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.494+0000 7f248ae44740 1 --2- 172.21.15.31:0/337789820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560faf6108f0 0x560faf616d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.494+0000 7f248ae44740 1 --2- 172.21.15.31:0/337789820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf60ffd0 0x560faf6167d0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.494+0000 7f2467400640 0 client.27338 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.494+0000 7f248ae44740 1 -- 172.21.15.31:0/337789820 >> 172.21.15.31:0/337789820 conn(0x560faf60a8f0 msgr2=0x560faf60c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.494+0000 7f248ae44740 1 -- 172.21.15.31:0/337789820 shutdown_connections 2024-03-21T10:45:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.494+0000 7f248ae44740 1 -- 172.21.15.31:0/337789820 wait complete. 2024-03-21T10:45:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.494+0000 7f248ae44740 20 client.27338 trim_cache size 0 max 16384 2024-03-21T10:45:46.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.515+0000 7f248ae44740 1 Processor -- start 2024-03-21T10:45:46.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.515+0000 7f248ae44740 1 -- start start 2024-03-21T10:45:46.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.515+0000 7f248ae44740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560faf60ffd0 0x560faf618f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:46.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.515+0000 7f248ae44740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560faf6108f0 0x560faf60c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:46.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.515+0000 7f248ae44740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf6196c0 0x560faf624a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:46.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.515+0000 7f248ae44740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24801b3170 con 0x560faf60ffd0 2024-03-21T10:45:46.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.515+0000 7f248ae44740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24801b0c70 con 0x560faf6196c0 2024-03-21T10:45:46.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.515+0000 7f248ae44740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24801aed30 con 0x560faf6108f0 2024-03-21T10:45:46.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.516+0000 7f2486200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf6196c0 0x560faf624a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:46.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.516+0000 7f2485800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560faf60ffd0 0x560faf618f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:46.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.516+0000 7f2484e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560faf6108f0 0x560faf60c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:46.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.516+0000 7f2486200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf6196c0 0x560faf624a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38010/0 (socket says 172.21.15.31:38010) 2024-03-21T10:45:46.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.516+0000 7f2486200640 1 -- 172.21.15.31:0/2666466769 learned_addr learned my addr 172.21.15.31:0/2666466769 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:46.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.516+0000 7f2486200640 1 -- 172.21.15.31:0/2666466769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560faf6108f0 msgr2=0x560faf60c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:46.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.516+0000 7f2486200640 1 --2- 172.21.15.31:0/2666466769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560faf6108f0 0x560faf60c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.516+0000 7f2486200640 1 -- 172.21.15.31:0/2666466769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560faf60ffd0 msgr2=0x560faf618f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.516+0000 7f2486200640 1 --2- 172.21.15.31:0/2666466769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560faf60ffd0 0x560faf618f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.516+0000 7f2486200640 1 -- 172.21.15.31:0/2666466769 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560faf608eb0 con 0x560faf6196c0 2024-03-21T10:45:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.516+0000 7f2485800640 1 --2- 172.21.15.31:0/2666466769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560faf60ffd0 0x560faf618f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:45:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.516+0000 7f2484e00640 1 --2- 172.21.15.31:0/2666466769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560faf6108f0 0x560faf60c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:45:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.517+0000 7f2486200640 1 --2- 172.21.15.31:0/2666466769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf6196c0 0x560faf624a00 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f2478009f00 tx=0x7f2478004820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=411e60ab5e40bbd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.517+0000 7f2472000640 1 -- 172.21.15.31:0/2666466769 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247801e040 con 0x560faf6196c0 2024-03-21T10:45:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.517+0000 7f2472000640 1 -- 172.21.15.31:0/2666466769 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f24780044d0 con 0x560faf6196c0 2024-03-21T10:45:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.517+0000 7f2472000640 1 -- 172.21.15.31:0/2666466769 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2478024440 con 0x560faf6196c0 2024-03-21T10:45:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.518+0000 7f248ae44740 1 -- 172.21.15.31:0/2666466769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf6196c0 msgr2=0x560faf624a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.518+0000 7f248ae44740 1 --2- 172.21.15.31:0/2666466769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf6196c0 0x560faf624a00 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f2478009f00 tx=0x7f2478004820 comp rx=0 tx=0).stop 2024-03-21T10:45:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.518+0000 7f248ae44740 1 -- 172.21.15.31:0/2666466769 shutdown_connections 2024-03-21T10:45:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.518+0000 7f248ae44740 1 --2- 172.21.15.31:0/2666466769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf6196c0 0x560faf624a00 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.518+0000 7f248ae44740 1 --2- 172.21.15.31:0/2666466769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560faf6108f0 0x560faf60c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.518+0000 7f248ae44740 1 --2- 172.21.15.31:0/2666466769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560faf60ffd0 0x560faf618f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.518+0000 7f248ae44740 1 -- 172.21.15.31:0/2666466769 >> 172.21.15.31:0/2666466769 conn(0x560faf60a8f0 msgr2=0x560faf60bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.518+0000 7f248ae44740 1 -- 172.21.15.31:0/2666466769 shutdown_connections 2024-03-21T10:45:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.518+0000 7f248ae44740 1 -- 172.21.15.31:0/2666466769 wait complete. 2024-03-21T10:45:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.519+0000 7f248ae44740 1 -- start start 2024-03-21T10:45:46.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.519+0000 7f248ae44740 1 Processor -- start 2024-03-21T10:45:46.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.519+0000 7f248ae44740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf60ffd0 0x560faf620b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.519+0000 7f248ae44740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560faf6108f0 0x560faf621080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.519+0000 7f248ae44740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560faf6196c0 0x560faf61d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.519+0000 7f248ae44740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24801af330 con 0x560faf6196c0 2024-03-21T10:45:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.519+0000 7f248ae44740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24801b0610 con 0x560faf60ffd0 2024-03-21T10:45:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.519+0000 7f248ae44740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24801af950 con 0x560faf6108f0 2024-03-21T10:45:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.519+0000 7f248ae44740 10 client.0 _ll_register_callbacks cb 0x7f24780093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.520+0000 7f2485800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf60ffd0 0x560faf620b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.520+0000 7f2485800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf60ffd0 0x560faf620b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38026/0 (socket says 172.21.15.31:38026) 2024-03-21T10:45:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.520+0000 7f2485800640 1 -- 172.21.15.31:0/1768145660 learned_addr learned my addr 172.21.15.31:0/1768145660 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.520+0000 7f2484e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560faf6108f0 0x560faf621080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.520+0000 7f2486200640 1 --2- 172.21.15.31:0/1768145660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560faf6196c0 0x560faf61d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.520+0000 7f2485800640 1 -- 172.21.15.31:0/1768145660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560faf6108f0 msgr2=0x560faf621080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.520+0000 7f2485800640 1 --2- 172.21.15.31:0/1768145660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560faf6108f0 0x560faf621080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.520+0000 7f2485800640 1 -- 172.21.15.31:0/1768145660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560faf6196c0 msgr2=0x560faf61d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.520+0000 7f2485800640 1 --2- 172.21.15.31:0/1768145660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560faf6196c0 0x560faf61d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.520+0000 7f2485800640 1 -- 172.21.15.31:0/1768145660 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560faf61d810 con 0x560faf60ffd0 2024-03-21T10:45:46.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.520+0000 7f2485800640 1 --2- 172.21.15.31:0/1768145660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf60ffd0 0x560faf620b40 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f2474036040 tx=0x7f2474003900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bad77ee8c83eb732 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:46.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.520+0000 7f2467400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:45:46.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.521+0000 7f2467400640 1 -- 172.21.15.31:0/1768145660 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247405b020 con 0x560faf60ffd0 2024-03-21T10:45:46.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.521+0000 7f2467400640 1 -- 172.21.15.31:0/1768145660 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f247402cc90 con 0x560faf60ffd0 2024-03-21T10:45:46.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.521+0000 7f248ae44740 1 -- 172.21.15.31:0/1768145660 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f24780044d0 con 0x560faf60ffd0 2024-03-21T10:45:46.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.521+0000 7f2467400640 1 -- 172.21.15.31:0/1768145660 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2474066050 con 0x560faf60ffd0 2024-03-21T10:45:46.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.521+0000 7f248ae44740 10 client.27344 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:46.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.521+0000 7f248ae44740 1 -- 172.21.15.31:0/1768145660 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x560faf68e230 con 0x560faf60ffd0 2024-03-21T10:45:46.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.521+0000 7f2467400640 1 -- 172.21.15.31:0/1768145660 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f24780044d0 con 0x560faf60ffd0 2024-03-21T10:45:46.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.521+0000 7f2467400640 1 -- 172.21.15.31:0/1768145660 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f2474057480 con 0x560faf60ffd0 2024-03-21T10:45:46.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.521+0000 7f248ae44740 10 client.27344 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:46.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.522+0000 7f248ae44740 10 client.27344 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:45:46.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.522+0000 7f248ae44740 20 client.27344 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:46.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.522+0000 7f248ae44740 1 --2- 172.21.15.31:0/1768145660 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560faf68e950 0x560faf690d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:46.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.522+0000 7f248ae44740 4 client.27344 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:45:46.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.522+0000 7f248ae44740 1 -- 172.21.15.31:0/1768145660 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560faf57c0c0 con 0x560faf68e950 2024-03-21T10:45:46.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.522+0000 7f2484e00640 1 --2- 172.21.15.31:0/1768145660 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560faf68e950 0x560faf690d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:46.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.523+0000 7f2484e00640 1 --2- 172.21.15.31:0/1768145660 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560faf68e950 0x560faf690d70 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f3effffe536b3c3 server_cookie=f82d6b6d7b1de6db in_seq=0 out_seq=0 2024-03-21T10:45:46.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.523+0000 7f2467400640 10 client.27344 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:46.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.537+0000 7f2467400640 1 -- 172.21.15.31:0/1768145660 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560faf57c0c0 con 0x560faf68e950 2024-03-21T10:45:46.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.537+0000 7f2467400640 10 client.27344 handle_command_reply: tid=0 2024-03-21T10:45:46.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.537+0000 7f248ae44740 1 client.27344 shutdown 2024-03-21T10:45:46.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.537+0000 7f248ae44740 1 -- 172.21.15.31:0/1768145660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf60ffd0 msgr2=0x560faf620b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:46.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.537+0000 7f248ae44740 1 --2- 172.21.15.31:0/1768145660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf60ffd0 0x560faf620b40 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f2474036040 tx=0x7f2474003900 comp rx=0 tx=0).stop 2024-03-21T10:45:46.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.538+0000 7f248ae44740 1 -- 172.21.15.31:0/1768145660 shutdown_connections 2024-03-21T10:45:46.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.538+0000 7f248ae44740 1 --2- 172.21.15.31:0/1768145660 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560faf68e950 0x560faf690d70 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.538+0000 7f248ae44740 1 --2- 172.21.15.31:0/1768145660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560faf6196c0 0x560faf61d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.538+0000 7f248ae44740 1 --2- 172.21.15.31:0/1768145660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560faf6108f0 0x560faf621080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.538+0000 7f248ae44740 1 --2- 172.21.15.31:0/1768145660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560faf60ffd0 0x560faf620b40 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.538+0000 7f248ae44740 1 -- 172.21.15.31:0/1768145660 >> 172.21.15.31:0/1768145660 conn(0x560faf60a8f0 msgr2=0x560faf60c480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:46.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.538+0000 7f2467400640 0 client.27344 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:46.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.538+0000 7f248ae44740 1 -- 172.21.15.31:0/1768145660 shutdown_connections 2024-03-21T10:45:46.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.538+0000 7f248ae44740 1 -- 172.21.15.31:0/1768145660 wait complete. 2024-03-21T10:45:46.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.538+0000 7f248ae44740 20 client.27344 trim_cache size 0 max 16384 2024-03-21T10:45:46.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:45:46.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:45:46.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:45:46.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:45:46.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.538+0000 7f2488200640 1 -- 172.21.15.31:0/2828704144 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24680a2f90 msgr2=0x7f24680a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:46.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.538+0000 7f2488200640 1 --2- 172.21.15.31:0/2828704144 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24680a2f90 0x7f24680a5450 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2478009d70 tx=0x7f2478005fd0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.539+0000 7f2488200640 1 -- 172.21.15.31:0/2828704144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2480069860 msgr2=0x7f248019a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:46.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.539+0000 7f2488200640 1 --2- 172.21.15.31:0/2828704144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2480069860 0x7f248019a630 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f246c00e9f0 tx=0x7f246c00edf0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.539+0000 7f2488200640 1 -- 172.21.15.31:0/2828704144 shutdown_connections 2024-03-21T10:45:46.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.539+0000 7f2488200640 1 --2- 172.21.15.31:0/2828704144 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24680a2f90 0x7f24680a5450 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.539+0000 7f2488200640 1 --2- 172.21.15.31:0/2828704144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f248010e040 0x7f248019e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.539+0000 7f2488200640 1 --2- 172.21.15.31:0/2828704144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2480069860 0x7f248019a630 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.539+0000 7f2488200640 1 --2- 172.21.15.31:0/2828704144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2480068f40 0x7f248019a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:46.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.539+0000 7f2488200640 1 -- 172.21.15.31:0/2828704144 >> 172.21.15.31:0/2828704144 conn(0x7f2480077690 msgr2=0x7f248010f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:46.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.539+0000 7f2488200640 1 -- 172.21.15.31:0/2828704144 shutdown_connections 2024-03-21T10:45:46.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:46.539+0000 7f2488200640 1 -- 172.21.15.31:0/2828704144 wait complete. 2024-03-21T10:45:47.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:47 smithi070 ceph-mon[36330]: pgmap v221: 129 pgs: 129 active+clean; 48 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 226 MiB/s wr, 1.55k op/s 2024-03-21T10:45:47.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:47 smithi040 ceph-mon[36154]: pgmap v221: 129 pgs: 129 active+clean; 48 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 226 MiB/s wr, 1.55k op/s 2024-03-21T10:45:47.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:47 smithi031 ceph-mon[31673]: pgmap v221: 129 pgs: 129 active+clean; 48 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 226 MiB/s wr, 1.55k op/s 2024-03-21T10:45:49.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:49 smithi040 ceph-mon[36154]: pgmap v222: 129 pgs: 129 active+clean; 48 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 169 MiB/s wr, 1.16k op/s 2024-03-21T10:45:49.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:49 smithi031 ceph-mon[31673]: pgmap v222: 129 pgs: 129 active+clean; 48 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 169 MiB/s wr, 1.16k op/s 2024-03-21T10:45:49.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:49 smithi070 ceph-mon[36330]: pgmap v222: 129 pgs: 129 active+clean; 48 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 169 MiB/s wr, 1.16k op/s 2024-03-21T10:45:50.613 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:45:50.613 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:45:51.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.009+0000 7fd602400640 1 -- 172.21.15.31:0/3809752124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5fc102840 msgr2=0x7fd5fc0ffef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:51.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.009+0000 7fd602400640 1 --2- 172.21.15.31:0/3809752124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5fc102840 0x7fd5fc0ffef0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec009930 tx=0x7fd5ec02bc40 comp rx=0 tx=0).stop 2024-03-21T10:45:51.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.009+0000 7fd602400640 1 -- 172.21.15.31:0/3809752124 shutdown_connections 2024-03-21T10:45:51.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.009+0000 7fd602400640 1 --2- 172.21.15.31:0/3809752124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5fc10e130 0x7fd5fc110520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.009+0000 7fd602400640 1 --2- 172.21.15.31:0/3809752124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5fc102840 0x7fd5fc0ffef0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.009+0000 7fd602400640 1 --2- 172.21.15.31:0/3809752124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5fc1023d0 0x7fd5fc0ff9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.009+0000 7fd602400640 1 -- 172.21.15.31:0/3809752124 >> 172.21.15.31:0/3809752124 conn(0x7fd5fc0768b0 msgr2=0x7fd5fc0fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:51.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.009+0000 7fd602400640 1 -- 172.21.15.31:0/3809752124 shutdown_connections 2024-03-21T10:45:51.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.009+0000 7fd602400640 1 -- 172.21.15.31:0/3809752124 wait complete. 2024-03-21T10:45:51.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.011+0000 7fd602400640 1 Processor -- start 2024-03-21T10:45:51.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.011+0000 7fd602400640 1 -- start start 2024-03-21T10:45:51.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.012+0000 7fd602400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5fc1023d0 0x7fd5fc199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:51.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.012+0000 7fd602400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5fc102840 0x7fd5fc19a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:51.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.012+0000 7fd602400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5fc10e130 0x7fd5fc19e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:51.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.012+0000 7fd602400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc113a60 con 0x7fd5fc102840 2024-03-21T10:45:51.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.012+0000 7fd602400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc1138e0 con 0x7fd5fc1023d0 2024-03-21T10:45:51.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.012+0000 7fd602400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc113be0 con 0x7fd5fc10e130 2024-03-21T10:45:51.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.012+0000 7fd601000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5fc1023d0 0x7fd5fc199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:51.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.012+0000 7fd601000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5fc1023d0 0x7fd5fc199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:35714/0 (socket says 172.21.15.31:35714) 2024-03-21T10:45:51.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.012+0000 7fd601000640 1 -- 172.21.15.31:0/3874808159 learned_addr learned my addr 172.21.15.31:0/3874808159 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:51.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.012+0000 7fd5fbe00640 1 --2- 172.21.15.31:0/3874808159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5fc102840 0x7fd5fc19a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:51.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.014+0000 7fd601a00640 1 --2- 172.21.15.31:0/3874808159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5fc10e130 0x7fd5fc19e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:51.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.014+0000 7fd601000640 1 -- 172.21.15.31:0/3874808159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5fc10e130 msgr2=0x7fd5fc19e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:51.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.014+0000 7fd601000640 1 --2- 172.21.15.31:0/3874808159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5fc10e130 0x7fd5fc19e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.014+0000 7fd601000640 1 -- 172.21.15.31:0/3874808159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5fc102840 msgr2=0x7fd5fc19a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:51.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.014+0000 7fd601000640 1 --2- 172.21.15.31:0/3874808159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5fc102840 0x7fd5fc19a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.014+0000 7fd601000640 1 -- 172.21.15.31:0/3874808159 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5fc19ef20 con 0x7fd5fc1023d0 2024-03-21T10:45:51.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.016+0000 7fd601000640 1 --2- 172.21.15.31:0/3874808159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5fc1023d0 0x7fd5fc199f60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400eab0 tx=0x7fd5f400ef70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:51.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.018+0000 7fd5f9600640 1 -- 172.21.15.31:0/3874808159 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f400ce40 con 0x7fd5fc1023d0 2024-03-21T10:45:51.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.018+0000 7fd5f9600640 1 -- 172.21.15.31:0/3874808159 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5f4011950 con 0x7fd5fc1023d0 2024-03-21T10:45:51.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.018+0000 7fd5f9600640 1 -- 172.21.15.31:0/3874808159 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f4010ae0 con 0x7fd5fc1023d0 2024-03-21T10:45:51.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.018+0000 7fd602400640 1 -- 172.21.15.31:0/3874808159 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5fc19f210 con 0x7fd5fc1023d0 2024-03-21T10:45:51.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.018+0000 7fd602400640 1 -- 172.21.15.31:0/3874808159 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5fc06b0c0 con 0x7fd5fc1023d0 2024-03-21T10:45:51.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.022+0000 7fd60679c740 1 Processor -- start 2024-03-21T10:45:51.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.023+0000 7fd60679c740 1 -- start start 2024-03-21T10:45:51.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.023+0000 7fd60679c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5617e13d9dd0 0x5617e13d8b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:51.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.023+0000 7fd60679c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617e13d90f0 0x5617e13d9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:51.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.023+0000 7fd60679c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617e13e2460 0x5617e13e4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:51.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.023+0000 7fd601a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5617e13d9dd0 0x5617e13d8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:51.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.024+0000 7fd601a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5617e13d9dd0 0x5617e13d8b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:35728/0 (socket says 172.21.15.31:35728) 2024-03-21T10:45:51.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.024+0000 7fd601a00640 1 -- 172.21.15.31:0/3827618311 learned_addr learned my addr 172.21.15.31:0/3827618311 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:51.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.024+0000 7fd60679c740 1 -- 172.21.15.31:0/3827618311 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617e133d250 con 0x5617e13d90f0 2024-03-21T10:45:51.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.024+0000 7fd60679c740 1 -- 172.21.15.31:0/3827618311 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617e12e5a30 con 0x5617e13d9dd0 2024-03-21T10:45:51.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.024+0000 7fd60679c740 1 -- 172.21.15.31:0/3827618311 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617e13392f0 con 0x5617e13e2460 2024-03-21T10:45:51.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.024+0000 7fd5fbe00640 1 --2- 172.21.15.31:0/3827618311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617e13e2460 0x5617e13e4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:51.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.024+0000 7fd5fbe00640 1 -- 172.21.15.31:0/3827618311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5617e13d9dd0 msgr2=0x5617e13d8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:51.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.024+0000 7fd5fbe00640 1 --2- 172.21.15.31:0/3827618311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5617e13d9dd0 0x5617e13d8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.024+0000 7fd5fbe00640 1 -- 172.21.15.31:0/3827618311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617e13d90f0 msgr2=0x5617e13d9570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:45:51.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.024+0000 7fd5fbe00640 1 --2- 172.21.15.31:0/3827618311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617e13d90f0 0x5617e13d9570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.024+0000 7fd5fbe00640 1 -- 172.21.15.31:0/3827618311 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617e12d0690 con 0x5617e13e2460 2024-03-21T10:45:51.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.026+0000 7fd5fbe00640 1 --2- 172.21.15.31:0/3827618311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617e13e2460 0x5617e13e4850 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec009230 tx=0x7fd5ec004740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebb2543862a905bc server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:51.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.026+0000 7fd5f2000640 1 -- 172.21.15.31:0/3827618311 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec042020 con 0x5617e13e2460 2024-03-21T10:45:51.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.026+0000 7fd5f2000640 1 -- 172.21.15.31:0/3827618311 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5ec004930 con 0x5617e13e2460 2024-03-21T10:45:51.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.026+0000 7fd5f2000640 1 -- 172.21.15.31:0/3827618311 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec02c5e0 con 0x5617e13e2460 2024-03-21T10:45:51.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.027+0000 7fd60679c740 1 -- 172.21.15.31:0/3827618311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617e13e2460 msgr2=0x5617e13e4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:51.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.027+0000 7fd60679c740 1 --2- 172.21.15.31:0/3827618311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617e13e2460 0x5617e13e4850 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec009230 tx=0x7fd5ec004740 comp rx=0 tx=0).stop 2024-03-21T10:45:51.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.029+0000 7fd60679c740 1 -- 172.21.15.31:0/3827618311 shutdown_connections 2024-03-21T10:45:51.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.029+0000 7fd60679c740 1 --2- 172.21.15.31:0/3827618311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617e13e2460 0x5617e13e4850 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.029+0000 7fd60679c740 1 --2- 172.21.15.31:0/3827618311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617e13d90f0 0x5617e13d9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.029+0000 7fd60679c740 1 --2- 172.21.15.31:0/3827618311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5617e13d9dd0 0x5617e13d8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.029+0000 7fd60679c740 1 -- 172.21.15.31:0/3827618311 >> 172.21.15.31:0/3827618311 conn(0x5617e13a0200 msgr2=0x5617e13d3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:51.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.029+0000 7fd5f9600640 1 -- 172.21.15.31:0/3874808159 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fd5f4010c80 con 0x7fd5fc1023d0 2024-03-21T10:45:51.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.031+0000 7fd60679c740 1 -- 172.21.15.31:0/3827618311 shutdown_connections 2024-03-21T10:45:51.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.032+0000 7fd60679c740 1 -- 172.21.15.31:0/3827618311 wait complete. 2024-03-21T10:45:51.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.032+0000 7fd5f9600640 1 --2- 172.21.15.31:0/3874808159 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd5e00a3050 0x7fd5e00a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:51.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.033+0000 7fd60679c740 1 -- start start 2024-03-21T10:45:51.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.033+0000 7fd5f9600640 1 -- 172.21.15.31:0/3874808159 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fd5f40d0240 con 0x7fd5fc1023d0 2024-03-21T10:45:51.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.033+0000 7fd60679c740 1 Processor -- start 2024-03-21T10:45:51.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.033+0000 7fd601a00640 1 --2- 172.21.15.31:0/3874808159 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd5e00a3050 0x7fd5e00a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:51.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.034+0000 7fd601a00640 1 --2- 172.21.15.31:0/3874808159 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd5e00a3050 0x7fd5e00a5510 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd5e800bbe0 tx=0x7fd5e800b4d0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:51.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.035+0000 7fd60679c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617e13d9dd0 0x5617e13e1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:51.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.036+0000 7fd60679c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5617e13e2460 0x5617e13defe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:51.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.036+0000 7fd60679c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617e13df520 0x5617e13df9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:51.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.036+0000 7fd60679c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617e13e5270 con 0x5617e13d9dd0 2024-03-21T10:45:51.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.036+0000 7fd60679c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617e12f8e80 con 0x5617e13e2460 2024-03-21T10:45:51.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.036+0000 7fd60679c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617e13e53f0 con 0x5617e13df520 2024-03-21T10:45:51.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.036+0000 7fd60679c740 10 client.0 _ll_register_callbacks cb 0x5617e130b130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:51.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.036+0000 7fd5fbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5617e13e2460 0x5617e13defe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:51.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.036+0000 7fd5fbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5617e13e2460 0x5617e13defe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:35744/0 (socket says 172.21.15.31:35744) 2024-03-21T10:45:51.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.036+0000 7fd5fbe00640 1 -- 172.21.15.31:0/3348492986 learned_addr learned my addr 172.21.15.31:0/3348492986 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:51.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.037+0000 7fd601000640 1 --2- 172.21.15.31:0/3348492986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617e13d9dd0 0x5617e13e1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:51.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.037+0000 7fd601a00640 1 --2- 172.21.15.31:0/3348492986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617e13df520 0x5617e13df9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:51.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.037+0000 7fd5fbe00640 1 -- 172.21.15.31:0/3348492986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617e13df520 msgr2=0x5617e13df9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:51.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.042+0000 7fd5fbe00640 1 --2- 172.21.15.31:0/3348492986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617e13df520 0x5617e13df9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.042+0000 7fd5fbe00640 1 -- 172.21.15.31:0/3348492986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617e13d9dd0 msgr2=0x5617e13e1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:51.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.042+0000 7fd5fbe00640 1 --2- 172.21.15.31:0/3348492986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617e13d9dd0 0x5617e13e1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.042+0000 7fd5fbe00640 1 -- 172.21.15.31:0/3348492986 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5617e111b7d0 con 0x5617e13e2460 2024-03-21T10:45:51.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.042+0000 7fd601a00640 1 --2- 172.21.15.31:0/3348492986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617e13df520 0x5617e13df9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:45:51.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.042+0000 7fd601000640 1 --2- 172.21.15.31:0/3348492986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617e13d9dd0 0x5617e13e1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:45:51.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.042+0000 7fd5fbe00640 1 --2- 172.21.15.31:0/3348492986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5617e13e2460 0x5617e13defe0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec000c00 tx=0x7fd5ec02c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef5d538872bcc69f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:51.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.045+0000 7fd5e6a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:45:51.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.045+0000 7fd5e6a00640 1 -- 172.21.15.31:0/3348492986 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec04b070 con 0x5617e13e2460 2024-03-21T10:45:51.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.045+0000 7fd5e6a00640 1 -- 172.21.15.31:0/3348492986 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fd5ec02cb90 con 0x5617e13e2460 2024-03-21T10:45:51.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.045+0000 7fd60679c740 1 -- 172.21.15.31:0/3348492986 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd5ec004930 con 0x5617e13e2460 2024-03-21T10:45:51.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.046+0000 7fd5e6a00640 1 -- 172.21.15.31:0/3348492986 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5ec055050 con 0x5617e13e2460 2024-03-21T10:45:51.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.049+0000 7fd60679c740 10 client.27356 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:51.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.049+0000 7fd60679c740 1 -- 172.21.15.31:0/3348492986 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5617e10513f0 con 0x5617e13e2460 2024-03-21T10:45:51.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.051+0000 7fd5e6a00640 1 -- 172.21.15.31:0/3348492986 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5ec004930 con 0x5617e13e2460 2024-03-21T10:45:51.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.054+0000 7fd5e6a00640 1 -- 172.21.15.31:0/3348492986 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd5ec004cf0 con 0x5617e13e2460 2024-03-21T10:45:51.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.054+0000 7fd60679c740 10 client.27356 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:51.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.054+0000 7fd60679c740 10 client.27356 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:45:51.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.054+0000 7fd60679c740 20 client.27356 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:51.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.054+0000 7fd60679c740 1 --2- 172.21.15.31:0/3348492986 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5617e13f2730 0x5617e13f4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:51.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.054+0000 7fd60679c740 4 client.27356 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:45:51.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.054+0000 7fd60679c740 1 -- 172.21.15.31:0/3348492986 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5617e134a790 con 0x5617e13f2730 2024-03-21T10:45:51.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.055+0000 7fd601000640 1 --2- 172.21.15.31:0/3348492986 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5617e13f2730 0x5617e13f4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:51.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.056+0000 7fd601000640 1 --2- 172.21.15.31:0/3348492986 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5617e13f2730 0x5617e13f4b20 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=809ba794d3c1bdf server_cookie=4d6c89216b8d1e33 in_seq=0 out_seq=0 2024-03-21T10:45:51.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.056+0000 7fd5e6a00640 10 client.27356 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:51.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.061+0000 7fd5e6a00640 1 -- 172.21.15.31:0/3348492986 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5617e134a790 con 0x5617e13f2730 2024-03-21T10:45:51.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.061+0000 7fd5e6a00640 10 client.27356 handle_command_reply: tid=0 2024-03-21T10:45:51.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.061+0000 7fd60679c740 1 client.27356 shutdown 2024-03-21T10:45:51.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.062+0000 7fd60679c740 1 -- 172.21.15.31:0/3348492986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5617e13e2460 msgr2=0x5617e13defe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:51.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.062+0000 7fd60679c740 1 --2- 172.21.15.31:0/3348492986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5617e13e2460 0x5617e13defe0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec000c00 tx=0x7fd5ec02c810 comp rx=0 tx=0).stop 2024-03-21T10:45:51.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.062+0000 7fd60679c740 1 -- 172.21.15.31:0/3348492986 shutdown_connections 2024-03-21T10:45:51.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.062+0000 7fd60679c740 1 --2- 172.21.15.31:0/3348492986 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5617e13f2730 0x5617e13f4b20 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.062+0000 7fd60679c740 1 --2- 172.21.15.31:0/3348492986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617e13df520 0x5617e13df9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.062+0000 7fd60679c740 1 --2- 172.21.15.31:0/3348492986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5617e13e2460 0x5617e13defe0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.062+0000 7fd5e6a00640 0 client.27356 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:51.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.062+0000 7fd60679c740 1 --2- 172.21.15.31:0/3348492986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617e13d9dd0 0x5617e13e1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.062+0000 7fd60679c740 1 -- 172.21.15.31:0/3348492986 >> 172.21.15.31:0/3348492986 conn(0x5617e13a0200 msgr2=0x5617e13d5890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:51.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.063+0000 7fd60679c740 1 -- 172.21.15.31:0/3348492986 shutdown_connections 2024-03-21T10:45:51.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.063+0000 7fd60679c740 1 -- 172.21.15.31:0/3348492986 wait complete. 2024-03-21T10:45:51.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.063+0000 7fd60679c740 20 client.27356 trim_cache size 0 max 16384 2024-03-21T10:45:51.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.102+0000 7fd60679c740 1 Processor -- start 2024-03-21T10:45:51.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.103+0000 7fd60679c740 1 -- start start 2024-03-21T10:45:51.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.104+0000 7fd60679c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617e13d9dd0 0x5617e13ddfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:51.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.104+0000 7fd601000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617e13d9dd0 0x5617e13ddfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:51.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.105+0000 7fd601000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617e13d9dd0 0x5617e13ddfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44600/0 (socket says 172.21.15.31:44600) 2024-03-21T10:45:51.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.105+0000 7fd60679c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5617e13df520 0x5617e13e1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:51.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.105+0000 7fd60679c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617e13e2460 0x5617e13e3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:51.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.105+0000 7fd60679c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc1aa5d0 con 0x5617e13d9dd0 2024-03-21T10:45:51.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.105+0000 7fd60679c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc1a80d0 con 0x5617e13df520 2024-03-21T10:45:51.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.105+0000 7fd601000640 1 -- 172.21.15.31:0/1094134995 learned_addr learned my addr 172.21.15.31:0/1094134995 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:51.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.105+0000 7fd60679c740 1 -- 172.21.15.31:0/1094134995 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc1009c0 con 0x5617e13e2460 2024-03-21T10:45:51.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.105+0000 7fd5fbe00640 1 --2- 172.21.15.31:0/1094134995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5617e13df520 0x5617e13e1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:51.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.105+0000 7fd601a00640 1 --2- 172.21.15.31:0/1094134995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617e13e2460 0x5617e13e3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:51.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.106+0000 7fd5fbe00640 1 -- 172.21.15.31:0/1094134995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617e13e2460 msgr2=0x5617e13e3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:51.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.106+0000 7fd5fbe00640 1 --2- 172.21.15.31:0/1094134995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617e13e2460 0x5617e13e3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.106+0000 7fd5fbe00640 1 -- 172.21.15.31:0/1094134995 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617e13d9dd0 msgr2=0x5617e13ddfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:51.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.106+0000 7fd5fbe00640 1 --2- 172.21.15.31:0/1094134995 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617e13d9dd0 0x5617e13ddfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.106+0000 7fd5fbe00640 1 -- 172.21.15.31:0/1094134995 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617e10110f0 con 0x5617e13df520 2024-03-21T10:45:51.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.107+0000 7fd5fbe00640 1 --2- 172.21.15.31:0/1094134995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5617e13df520 0x5617e13e1820 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec0050c0 tx=0x7fd5ec009230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd13d4fc8dbc7b65 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:51.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.107+0000 7fd5f1600640 1 -- 172.21.15.31:0/1094134995 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec042070 con 0x5617e13df520 2024-03-21T10:45:51.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.107+0000 7fd5f1600640 1 -- 172.21.15.31:0/1094134995 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5ec0048b0 con 0x5617e13df520 2024-03-21T10:45:51.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.107+0000 7fd5f1600640 1 -- 172.21.15.31:0/1094134995 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec04b040 con 0x5617e13df520 2024-03-21T10:45:51.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.108+0000 7fd60679c740 1 -- 172.21.15.31:0/1094134995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5617e13df520 msgr2=0x5617e13e1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:51.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.108+0000 7fd60679c740 1 --2- 172.21.15.31:0/1094134995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5617e13df520 0x5617e13e1820 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec0050c0 tx=0x7fd5ec009230 comp rx=0 tx=0).stop 2024-03-21T10:45:51.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.108+0000 7fd60679c740 1 -- 172.21.15.31:0/1094134995 shutdown_connections 2024-03-21T10:45:51.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.108+0000 7fd60679c740 1 --2- 172.21.15.31:0/1094134995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617e13e2460 0x5617e13e3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.108+0000 7fd60679c740 1 --2- 172.21.15.31:0/1094134995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5617e13df520 0x5617e13e1820 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.108+0000 7fd60679c740 1 --2- 172.21.15.31:0/1094134995 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617e13d9dd0 0x5617e13ddfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.108+0000 7fd60679c740 1 -- 172.21.15.31:0/1094134995 >> 172.21.15.31:0/1094134995 conn(0x5617e13a0200 msgr2=0x5617e13dd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:51.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.109+0000 7fd60679c740 1 -- 172.21.15.31:0/1094134995 shutdown_connections 2024-03-21T10:45:51.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.109+0000 7fd60679c740 1 -- 172.21.15.31:0/1094134995 wait complete. 2024-03-21T10:45:51.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.110+0000 7fd60679c740 1 -- start start 2024-03-21T10:45:51.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.110+0000 7fd60679c740 1 Processor -- start 2024-03-21T10:45:51.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.111+0000 7fd60679c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617e13d9dd0 0x5617e13db8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:51.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.112+0000 7fd60679c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5617e13df520 0x5617e13dbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:51.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.112+0000 7fd60679c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617e13e2460 0x5617e13dc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:51.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.112+0000 7fd601a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617e13e2460 0x5617e13dc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:51.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.112+0000 7fd60679c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc0736b0 con 0x5617e13e2460 2024-03-21T10:45:51.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.112+0000 7fd60679c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc1a7a70 con 0x5617e13df520 2024-03-21T10:45:51.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.112+0000 7fd60679c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc1a6dd0 con 0x5617e13d9dd0 2024-03-21T10:45:51.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.112+0000 7fd60679c740 10 client.0 _ll_register_callbacks cb 0x7fd5e800b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:51.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.113+0000 7fd601a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617e13e2460 0x5617e13dc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44604/0 (socket says 172.21.15.31:44604) 2024-03-21T10:45:51.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.113+0000 7fd601a00640 1 -- 172.21.15.31:0/4054887520 learned_addr learned my addr 172.21.15.31:0/4054887520 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:51.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.113+0000 7fd5fbe00640 1 --2- 172.21.15.31:0/4054887520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5617e13df520 0x5617e13dbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:51.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.113+0000 7fd601000640 1 --2- 172.21.15.31:0/4054887520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617e13d9dd0 0x5617e13db8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:51.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.114+0000 7fd601a00640 1 -- 172.21.15.31:0/4054887520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617e13d9dd0 msgr2=0x5617e13db8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:51.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.114+0000 7fd601a00640 1 --2- 172.21.15.31:0/4054887520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617e13d9dd0 0x5617e13db8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.114+0000 7fd601a00640 1 -- 172.21.15.31:0/4054887520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5617e13df520 msgr2=0x5617e13dbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:51.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.114+0000 7fd601a00640 1 --2- 172.21.15.31:0/4054887520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5617e13df520 0x5617e13dbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.115+0000 7fd601a00640 1 -- 172.21.15.31:0/4054887520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5617e144cd00 con 0x5617e13e2460 2024-03-21T10:45:51.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.115+0000 7fd5fbe00640 1 --2- 172.21.15.31:0/4054887520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5617e13df520 0x5617e13dbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:45:51.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.116+0000 7fd601a00640 1 --2- 172.21.15.31:0/4054887520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617e13e2460 0x5617e13dc330 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd5e803ebf0 tx=0x7fd5e8036040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8ab61033431f231 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:51.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.116+0000 7fd5e6a00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:45:51.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.117+0000 7fd5e6a00640 1 -- 172.21.15.31:0/4054887520 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8032c70 con 0x5617e13e2460 2024-03-21T10:45:51.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.117+0000 7fd5e6a00640 1 -- 172.21.15.31:0/4054887520 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fd5e804d020 con 0x5617e13e2460 2024-03-21T10:45:51.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.117+0000 7fd60679c740 1 -- 172.21.15.31:0/4054887520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd5ec0048b0 con 0x5617e13e2460 2024-03-21T10:45:51.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.118+0000 7fd5e6a00640 1 -- 172.21.15.31:0/4054887520 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5e8032c70 con 0x5617e13e2460 2024-03-21T10:45:51.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.118+0000 7fd60679c740 10 client.16785 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:51.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.118+0000 7fd60679c740 1 -- 172.21.15.31:0/4054887520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5617e144e050 con 0x5617e13e2460 2024-03-21T10:45:51.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.119+0000 7fd5e6a00640 1 -- 172.21.15.31:0/4054887520 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5ec0048b0 con 0x5617e13e2460 2024-03-21T10:45:51.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.119+0000 7fd5e6a00640 1 -- 172.21.15.31:0/4054887520 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd5e803b040 con 0x5617e13e2460 2024-03-21T10:45:51.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.120+0000 7fd60679c740 10 client.16785 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:51.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.120+0000 7fd60679c740 10 client.16785 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:45:51.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.120+0000 7fd60679c740 20 client.16785 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:51.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.120+0000 7fd60679c740 1 --2- 172.21.15.31:0/4054887520 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5617e13f2730 0x5617e1450810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:51.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.120+0000 7fd60679c740 4 client.16785 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:45:51.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.120+0000 7fd60679c740 1 -- 172.21.15.31:0/4054887520 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5617e1344ae0 con 0x5617e13f2730 2024-03-21T10:45:51.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.120+0000 7fd601000640 1 --2- 172.21.15.31:0/4054887520 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5617e13f2730 0x5617e1450810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:51.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.121+0000 7fd5e6a00640 10 client.16785 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:51.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.122+0000 7fd601000640 1 --2- 172.21.15.31:0/4054887520 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5617e13f2730 0x5617e1450810 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=178500c6dbd8684d server_cookie=a00519a6dd9c4ef5 in_seq=0 out_seq=0 2024-03-21T10:45:51.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.122+0000 7fd5e6a00640 1 -- 172.21.15.31:0/4054887520 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5617e1344ae0 con 0x5617e13f2730 2024-03-21T10:45:51.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.122+0000 7fd5e6a00640 10 client.16785 handle_command_reply: tid=0 2024-03-21T10:45:51.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.122+0000 7fd60679c740 1 client.16785 shutdown 2024-03-21T10:45:51.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.123+0000 7fd60679c740 1 -- 172.21.15.31:0/4054887520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617e13e2460 msgr2=0x5617e13dc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:51.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.123+0000 7fd60679c740 1 --2- 172.21.15.31:0/4054887520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617e13e2460 0x5617e13dc330 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd5e803ebf0 tx=0x7fd5e8036040 comp rx=0 tx=0).stop 2024-03-21T10:45:51.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.124+0000 7fd60679c740 1 -- 172.21.15.31:0/4054887520 shutdown_connections 2024-03-21T10:45:51.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.124+0000 7fd60679c740 1 --2- 172.21.15.31:0/4054887520 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5617e13f2730 0x5617e1450810 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.124+0000 7fd60679c740 1 --2- 172.21.15.31:0/4054887520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617e13e2460 0x5617e13dc330 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.124+0000 7fd60679c740 1 --2- 172.21.15.31:0/4054887520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5617e13df520 0x5617e13dbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.124+0000 7fd60679c740 1 --2- 172.21.15.31:0/4054887520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617e13d9dd0 0x5617e13db8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.124+0000 7fd60679c740 1 -- 172.21.15.31:0/4054887520 >> 172.21.15.31:0/4054887520 conn(0x5617e13a0200 msgr2=0x5617e13d4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:51.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.124+0000 7fd5e6a00640 0 client.16785 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:51.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.124+0000 7fd60679c740 1 -- 172.21.15.31:0/4054887520 shutdown_connections 2024-03-21T10:45:51.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.125+0000 7fd60679c740 1 -- 172.21.15.31:0/4054887520 wait complete. 2024-03-21T10:45:51.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.125+0000 7fd60679c740 20 client.16785 trim_cache size 0 max 16384 2024-03-21T10:45:51.123 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:45:51.123 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-03-21T10:45:51.123 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-03-21T10:45:51.123 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:45:51.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.127+0000 7fd602400640 1 -- 172.21.15.31:0/3874808159 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd5e00a3050 msgr2=0x7fd5e00a5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:51.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.127+0000 7fd602400640 1 --2- 172.21.15.31:0/3874808159 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd5e00a3050 0x7fd5e00a5510 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd5e800bbe0 tx=0x7fd5e800b4d0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.127+0000 7fd602400640 1 -- 172.21.15.31:0/3874808159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5fc1023d0 msgr2=0x7fd5fc199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:51.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.127+0000 7fd602400640 1 --2- 172.21.15.31:0/3874808159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5fc1023d0 0x7fd5fc199f60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400eab0 tx=0x7fd5f400ef70 comp rx=0 tx=0).stop 2024-03-21T10:45:51.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.128+0000 7fd602400640 1 -- 172.21.15.31:0/3874808159 shutdown_connections 2024-03-21T10:45:51.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.128+0000 7fd602400640 1 --2- 172.21.15.31:0/3874808159 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd5e00a3050 0x7fd5e00a5510 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.128+0000 7fd602400640 1 --2- 172.21.15.31:0/3874808159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5fc10e130 0x7fd5fc19e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.128+0000 7fd602400640 1 --2- 172.21.15.31:0/3874808159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5fc102840 0x7fd5fc19a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.128+0000 7fd602400640 1 --2- 172.21.15.31:0/3874808159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5fc1023d0 0x7fd5fc199f60 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.128+0000 7fd602400640 1 -- 172.21.15.31:0/3874808159 >> 172.21.15.31:0/3874808159 conn(0x7fd5fc0768b0 msgr2=0x7fd5fc10f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:51.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.132+0000 7fd602400640 1 -- 172.21.15.31:0/3874808159 shutdown_connections 2024-03-21T10:45:51.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.132+0000 7fd602400640 1 -- 172.21.15.31:0/3874808159 wait complete. 2024-03-21T10:45:51.159 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-21T10:45:51.159 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:45:51.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.408+0000 7f8015e00640 1 -- 172.21.15.31:0/998063868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8010101540 msgr2=0x7f80101019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:51.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.408+0000 7f8015e00640 1 --2- 172.21.15.31:0/998063868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8010101540 0x7f80101019c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc009930 tx=0x7f7ffc02bc30 comp rx=0 tx=0).stop 2024-03-21T10:45:51.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.408+0000 7f8015e00640 1 -- 172.21.15.31:0/998063868 shutdown_connections 2024-03-21T10:45:51.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.408+0000 7f8015e00640 1 --2- 172.21.15.31:0/998063868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8010101f00 0x7f801010e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.408+0000 7f8015e00640 1 --2- 172.21.15.31:0/998063868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8010101540 0x7f80101019c0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.408+0000 7f8015e00640 1 --2- 172.21.15.31:0/998063868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8010107600 0x7f80101079e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.408+0000 7f8015e00640 1 -- 172.21.15.31:0/998063868 >> 172.21.15.31:0/998063868 conn(0x7f80100fd050 msgr2=0x7f80100ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:51.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.408+0000 7f8015e00640 1 -- 172.21.15.31:0/998063868 shutdown_connections 2024-03-21T10:45:51.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.408+0000 7f8015e00640 1 -- 172.21.15.31:0/998063868 wait complete. 2024-03-21T10:45:51.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.409+0000 7f8015e00640 1 Processor -- start 2024-03-21T10:45:51.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.409+0000 7f8015e00640 1 -- start start 2024-03-21T10:45:51.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.410+0000 7f8015e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8010101540 0x7f80100739a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:51.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.410+0000 7f8015e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8010101f00 0x7f8010073ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:51.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.410+0000 7f8015e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8010107600 0x7f8010074420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:51.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.410+0000 7f8015e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8010111930 con 0x7f8010107600 2024-03-21T10:45:51.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.410+0000 7f8015e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80101117b0 con 0x7f8010101f00 2024-03-21T10:45:51.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.410+0000 7f8015e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8010111ab0 con 0x7f8010101540 2024-03-21T10:45:51.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.410+0000 7f800fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8010101f00 0x7f8010073ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:51.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.410+0000 7f8014a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8010101540 0x7f80100739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:51.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.410+0000 7f800fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8010101f00 0x7f8010073ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:35790/0 (socket says 172.21.15.31:35790) 2024-03-21T10:45:51.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.410+0000 7f8014a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8010101540 0x7f80100739a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:50742/0 (socket says 172.21.15.31:50742) 2024-03-21T10:45:51.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.410+0000 7f800fe00640 1 -- 172.21.15.31:0/34414577 learned_addr learned my addr 172.21.15.31:0/34414577 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:51.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.410+0000 7f8015400640 1 --2- 172.21.15.31:0/34414577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8010107600 0x7f8010074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:51.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.410+0000 7f8014a00640 1 -- 172.21.15.31:0/34414577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8010101f00 msgr2=0x7f8010073ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:51.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.410+0000 7f8014a00640 1 --2- 172.21.15.31:0/34414577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8010101f00 0x7f8010073ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.410+0000 7f8014a00640 1 -- 172.21.15.31:0/34414577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8010107600 msgr2=0x7f8010074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:51.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.410+0000 7f8014a00640 1 --2- 172.21.15.31:0/34414577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8010107600 0x7f8010074420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.410+0000 7f8014a00640 1 -- 172.21.15.31:0/34414577 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80101a62b0 con 0x7f8010101540 2024-03-21T10:45:51.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.410+0000 7f8015400640 1 --2- 172.21.15.31:0/34414577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8010107600 0x7f8010074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:45:51.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.411+0000 7f8014a00640 1 --2- 172.21.15.31:0/34414577 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8010101540 0x7f80100739a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f800400e9f0 tx=0x7f800400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:51.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.411+0000 7f800d600640 1 -- 172.21.15.31:0/34414577 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800400ccc0 con 0x7f8010101540 2024-03-21T10:45:51.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.411+0000 7f800d600640 1 -- 172.21.15.31:0/34414577 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8004005020 con 0x7f8010101540 2024-03-21T10:45:51.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.411+0000 7f8015e00640 1 -- 172.21.15.31:0/34414577 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80101a65a0 con 0x7f8010101540 2024-03-21T10:45:51.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.411+0000 7f800d600640 1 -- 172.21.15.31:0/34414577 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80040049e0 con 0x7f8010101540 2024-03-21T10:45:51.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.411+0000 7f8015e00640 1 -- 172.21.15.31:0/34414577 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80101a6af0 con 0x7f8010101540 2024-03-21T10:45:51.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.412+0000 7f8015e00640 1 -- 172.21.15.31:0/34414577 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f801004f470 con 0x7f8010101540 2024-03-21T10:45:51.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.418+0000 7f800d600640 1 -- 172.21.15.31:0/34414577 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f8004005560 con 0x7f8010101540 2024-03-21T10:45:51.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.421+0000 7f800d600640 1 --2- 172.21.15.31:0/34414577 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7ff80a3180 0x7f7ff80a5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:51.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.421+0000 7f800d600640 1 -- 172.21.15.31:0/34414577 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f80040cfaf0 con 0x7f8010101540 2024-03-21T10:45:51.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.426+0000 7f800d600640 1 -- 172.21.15.31:0/34414577 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f80040051d0 con 0x7f8010101540 2024-03-21T10:45:51.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.428+0000 7f800fe00640 1 --2- 172.21.15.31:0/34414577 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7ff80a3180 0x7f7ff80a5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:51.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.429+0000 7f800fe00640 1 --2- 172.21.15.31:0/34414577 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7ff80a3180 0x7f7ff80a5640 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc0096c0 tx=0x7f7ffc02df90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:51.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:51 smithi031 ceph-mon[31673]: pgmap v223: 129 pgs: 129 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 253 MiB/s wr, 1.71k op/s 2024-03-21T10:45:51.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:51 smithi040 ceph-mon[36154]: pgmap v223: 129 pgs: 129 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 253 MiB/s wr, 1.71k op/s 2024-03-21T10:45:51.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.778+0000 7f8015e00640 1 -- 172.21.15.31:0/34414577 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8010109a10 con 0x7f8010101540 2024-03-21T10:45:51.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.779+0000 7f800d600640 1 -- 172.21.15.31:0/34414577 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+11384 (secure 0 0 0) 0x7f800409c1e0 con 0x7f8010101540 2024-03-21T10:45:51.777 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:45:51.777 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:42:31.182549+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24547,24596]},"id":1}]} 2024-03-21T10:45:51.777 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2024-03-21T10:45:51.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.784+0000 7f8015e00640 1 -- 172.21.15.31:0/34414577 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7ff80a3180 msgr2=0x7f7ff80a5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:51.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.784+0000 7f8015e00640 1 --2- 172.21.15.31:0/34414577 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7ff80a3180 0x7f7ff80a5640 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc0096c0 tx=0x7f7ffc02df90 comp rx=0 tx=0).stop 2024-03-21T10:45:51.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.784+0000 7f8015e00640 1 -- 172.21.15.31:0/34414577 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8010101540 msgr2=0x7f80100739a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:51.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.784+0000 7f8015e00640 1 --2- 172.21.15.31:0/34414577 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8010101540 0x7f80100739a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f800400e9f0 tx=0x7f800400eeb0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.788+0000 7f8015e00640 1 -- 172.21.15.31:0/34414577 shutdown_connections 2024-03-21T10:45:51.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.788+0000 7f8015e00640 1 --2- 172.21.15.31:0/34414577 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7ff80a3180 0x7f7ff80a5640 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.788+0000 7f8015e00640 1 --2- 172.21.15.31:0/34414577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8010107600 0x7f8010074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.788+0000 7f8015e00640 1 --2- 172.21.15.31:0/34414577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8010101f00 0x7f8010073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.788+0000 7f8015e00640 1 --2- 172.21.15.31:0/34414577 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8010101540 0x7f80100739a0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:51.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.788+0000 7f8015e00640 1 -- 172.21.15.31:0/34414577 >> 172.21.15.31:0/34414577 conn(0x7f80100fd050 msgr2=0x7f80100fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:51.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.788+0000 7f8015e00640 1 -- 172.21.15.31:0/34414577 shutdown_connections 2024-03-21T10:45:51.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:51.788+0000 7f8015e00640 1 -- 172.21.15.31:0/34414577 wait complete. 2024-03-21T10:45:51.809 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:45:51.809 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-21T10:45:51.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:51 smithi070 ceph-mon[36330]: pgmap v223: 129 pgs: 129 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 253 MiB/s wr, 1.71k op/s 2024-03-21T10:45:52.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.091+0000 7fbade000640 1 -- 172.21.15.31:0/4275594794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbad811ae80 msgr2=0x7fbad811d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:52.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.091+0000 7fbade000640 1 --2- 172.21.15.31:0/4275594794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbad811ae80 0x7fbad811d270 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fbac4009900 tx=0x7fbac402bc80 comp rx=0 tx=0).stop 2024-03-21T10:45:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.108+0000 7fbade000640 1 -- 172.21.15.31:0/4275594794 shutdown_connections 2024-03-21T10:45:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.108+0000 7fbade000640 1 --2- 172.21.15.31:0/4275594794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbad811ae80 0x7fbad811d270 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.108+0000 7fbade000640 1 --2- 172.21.15.31:0/4275594794 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbad8108be0 0x7fbad8109040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.108+0000 7fbade000640 1 --2- 172.21.15.31:0/4275594794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad810fe30 0x7fbad81086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.108+0000 7fbade000640 1 -- 172.21.15.31:0/4275594794 >> 172.21.15.31:0/4275594794 conn(0x7fbad806c710 msgr2=0x7fbad806cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:52.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.110+0000 7fbade000640 1 -- 172.21.15.31:0/4275594794 shutdown_connections 2024-03-21T10:45:52.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.110+0000 7fbade000640 1 -- 172.21.15.31:0/4275594794 wait complete. 2024-03-21T10:45:52.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.116+0000 7fbade000640 1 Processor -- start 2024-03-21T10:45:52.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.119+0000 7fbade000640 1 -- start start 2024-03-21T10:45:52.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.120+0000 7fbade000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad8108be0 0x7fbad81a6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.120+0000 7fbade000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbad810fe30 0x7fbad81a7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.120+0000 7fbade000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbad811ae80 0x7fbad81ab6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.120+0000 7fbade000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad8120930 con 0x7fbad811ae80 2024-03-21T10:45:52.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.120+0000 7fbade000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad81207b0 con 0x7fbad8108be0 2024-03-21T10:45:52.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.120+0000 7fbade000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad8120ab0 con 0x7fbad810fe30 2024-03-21T10:45:52.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.121+0000 7fbad6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbad810fe30 0x7fbad81a7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.121+0000 7fbad6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbad810fe30 0x7fbad81a7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:50760/0 (socket says 172.21.15.31:50760) 2024-03-21T10:45:52.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.121+0000 7fbad6a00640 1 -- 172.21.15.31:0/1302533584 learned_addr learned my addr 172.21.15.31:0/1302533584 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:52.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.122+0000 7fbad7e00640 1 --2- 172.21.15.31:0/1302533584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbad811ae80 0x7fbad81ab6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.123+0000 7fbad7400640 1 --2- 172.21.15.31:0/1302533584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad8108be0 0x7fbad81a6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.123+0000 7fbad6a00640 1 -- 172.21.15.31:0/1302533584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad8108be0 msgr2=0x7fbad81a6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:52.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.123+0000 7fbad6a00640 1 --2- 172.21.15.31:0/1302533584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad8108be0 0x7fbad81a6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.123+0000 7fbad6a00640 1 -- 172.21.15.31:0/1302533584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbad811ae80 msgr2=0x7fbad81ab6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:52.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.123+0000 7fbad6a00640 1 --2- 172.21.15.31:0/1302533584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbad811ae80 0x7fbad81ab6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.123+0000 7fbad6a00640 1 -- 172.21.15.31:0/1302533584 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad81abe00 con 0x7fbad810fe30 2024-03-21T10:45:52.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.125+0000 7fbad6a00640 1 --2- 172.21.15.31:0/1302533584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbad810fe30 0x7fbad81a7350 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fbac800b4a0 tx=0x7fbac800b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:52.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.126+0000 7fbacfe00640 1 -- 172.21.15.31:0/1302533584 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac8014070 con 0x7fbad810fe30 2024-03-21T10:45:52.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.126+0000 7fbacfe00640 1 -- 172.21.15.31:0/1302533584 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbac8010af0 con 0x7fbad810fe30 2024-03-21T10:45:52.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.126+0000 7fbacfe00640 1 -- 172.21.15.31:0/1302533584 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac8017d10 con 0x7fbad810fe30 2024-03-21T10:45:52.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.127+0000 7fbade000640 1 -- 172.21.15.31:0/1302533584 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad81ac0f0 con 0x7fbad810fe30 2024-03-21T10:45:52.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.127+0000 7fbade000640 1 -- 172.21.15.31:0/1302533584 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad81b3980 con 0x7fbad810fe30 2024-03-21T10:45:52.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.138+0000 7fbae0dc9740 1 Processor -- start 2024-03-21T10:45:52.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.138+0000 7fbae0dc9740 1 -- start start 2024-03-21T10:45:52.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.139+0000 7fbacfe00640 1 -- 172.21.15.31:0/1302533584 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fbac8017070 con 0x7fbad810fe30 2024-03-21T10:45:52.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.142+0000 7fbacfe00640 1 --2- 172.21.15.31:0/1302533584 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbabc0a3110 0x7fbabc0a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.142+0000 7fbacfe00640 1 -- 172.21.15.31:0/1302533584 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fbac80cedb0 con 0x7fbad810fe30 2024-03-21T10:45:52.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.143+0000 7fbae0dc9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc0da736d0 0x55fc0da73ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.143+0000 7fbae0dc9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc0da74080 0x55fc0da7c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.143+0000 7fbae0dc9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc0da7cc00 0x55fc0da7f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.144+0000 7fbad7e00640 1 --2- 172.21.15.31:0/1302533584 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbabc0a3110 0x7fbabc0a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.145+0000 7fbad7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc0da7cc00 0x55fc0da7f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.145+0000 7fbad7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc0da7cc00 0x55fc0da7f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:50772/0 (socket says 172.21.15.31:50772) 2024-03-21T10:45:52.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.145+0000 7fbae0dc9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc0d9d72c0 con 0x55fc0da736d0 2024-03-21T10:45:52.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.145+0000 7fbae0dc9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc0d97fc90 con 0x55fc0da74080 2024-03-21T10:45:52.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.145+0000 7fbae0dc9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc0d9d3360 con 0x55fc0da7cc00 2024-03-21T10:45:52.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.145+0000 7fbad7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc0da736d0 0x55fc0da73ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.145+0000 7fbad7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc0da736d0 0x55fc0da73ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44656/0 (socket says 172.21.15.31:44656) 2024-03-21T10:45:52.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.145+0000 7fbad7400640 1 -- 172.21.15.31:0/2447839745 learned_addr learned my addr 172.21.15.31:0/2447839745 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:52.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.146+0000 7fbad6a00640 1 --2- 172.21.15.31:0/2447839745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc0da74080 0x55fc0da7c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.146+0000 7fbad6a00640 1 -- 172.21.15.31:0/2447839745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc0da7cc00 msgr2=0x55fc0da7f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:52.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.146+0000 7fbad6a00640 1 --2- 172.21.15.31:0/2447839745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc0da7cc00 0x55fc0da7f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.146+0000 7fbad6a00640 1 -- 172.21.15.31:0/2447839745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc0da736d0 msgr2=0x55fc0da73ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:52.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.146+0000 7fbad6a00640 1 --2- 172.21.15.31:0/2447839745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc0da736d0 0x55fc0da73ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.146+0000 7fbad6a00640 1 -- 172.21.15.31:0/2447839745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fc0d9a4080 con 0x55fc0da74080 2024-03-21T10:45:52.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.147+0000 7fbad7e00640 1 --2- 172.21.15.31:0/2447839745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc0da7cc00 0x55fc0da7f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-21T10:45:52.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.147+0000 7fbad6a00640 1 --2- 172.21.15.31:0/2447839745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc0da74080 0x55fc0da7c630 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fbac80cc270 tx=0x7fbac800bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9152bd1dd8b36548 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:52.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.148+0000 7fbac3e00640 1 -- 172.21.15.31:0/2447839745 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac801e050 con 0x55fc0da74080 2024-03-21T10:45:52.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.148+0000 7fbac3e00640 1 -- 172.21.15.31:0/2447839745 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbac809f060 con 0x55fc0da74080 2024-03-21T10:45:52.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.148+0000 7fbac3e00640 1 -- 172.21.15.31:0/2447839745 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac8010ca0 con 0x55fc0da74080 2024-03-21T10:45:52.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.149+0000 7fbae0dc9740 1 -- 172.21.15.31:0/2447839745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc0da74080 msgr2=0x55fc0da7c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:52.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.149+0000 7fbae0dc9740 1 --2- 172.21.15.31:0/2447839745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc0da74080 0x55fc0da7c630 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fbac80cc270 tx=0x7fbac800bf90 comp rx=0 tx=0).stop 2024-03-21T10:45:52.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.153+0000 7fbad7e00640 1 --2- 172.21.15.31:0/1302533584 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbabc0a3110 0x7fbabc0a55d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbac402c1c0 tx=0x7fbac4005fd0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:52.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.153+0000 7fbae0dc9740 1 -- 172.21.15.31:0/2447839745 shutdown_connections 2024-03-21T10:45:52.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.153+0000 7fbae0dc9740 1 --2- 172.21.15.31:0/2447839745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc0da7cc00 0x55fc0da7f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.153+0000 7fbae0dc9740 1 --2- 172.21.15.31:0/2447839745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc0da74080 0x55fc0da7c630 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.153+0000 7fbae0dc9740 1 --2- 172.21.15.31:0/2447839745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc0da736d0 0x55fc0da73ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.153+0000 7fbae0dc9740 1 -- 172.21.15.31:0/2447839745 >> 172.21.15.31:0/2447839745 conn(0x55fc0da6dda0 msgr2=0x55fc0da6e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:52.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.155+0000 7fbae0dc9740 1 -- 172.21.15.31:0/2447839745 shutdown_connections 2024-03-21T10:45:52.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.156+0000 7fbae0dc9740 1 -- 172.21.15.31:0/2447839745 wait complete. 2024-03-21T10:45:52.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.158+0000 7fbae0dc9740 1 -- start start 2024-03-21T10:45:52.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.159+0000 7fbae0dc9740 1 Processor -- start 2024-03-21T10:45:52.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.161+0000 7fbae0dc9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc0da736d0 0x55fc0da7c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.163+0000 7fbae0dc9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc0da74080 0x55fc0da79870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.163+0000 7fbad7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc0da736d0 0x55fc0da7c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.163+0000 7fbad7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc0da736d0 0x55fc0da7c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:35814/0 (socket says 172.21.15.31:35814) 2024-03-21T10:45:52.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.164+0000 7fbad7400640 1 -- 172.21.15.31:0/4060621930 learned_addr learned my addr 172.21.15.31:0/4060621930 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:52.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.164+0000 7fbad6a00640 1 --2- 172.21.15.31:0/4060621930 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc0da74080 0x55fc0da79870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.164+0000 7fbae0dc9740 1 --2- 172.21.15.31:0/4060621930 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc0da7cc00 0x55fc0da79dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.165+0000 7fbad7e00640 1 --2- 172.21.15.31:0/4060621930 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc0da7cc00 0x55fc0da79dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.165+0000 7fbae0dc9740 1 -- 172.21.15.31:0/4060621930 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc0da7fb70 con 0x55fc0da74080 2024-03-21T10:45:52.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.165+0000 7fbae0dc9740 1 -- 172.21.15.31:0/4060621930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc0d9930d0 con 0x55fc0da736d0 2024-03-21T10:45:52.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.165+0000 7fbae0dc9740 1 -- 172.21.15.31:0/4060621930 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc0da7fcf0 con 0x55fc0da7cc00 2024-03-21T10:45:52.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.165+0000 7fbae0dc9740 10 client.0 _ll_register_callbacks cb 0x55fc0d997e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:52.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.166+0000 7fbad7400640 1 -- 172.21.15.31:0/4060621930 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc0da7cc00 msgr2=0x55fc0da79dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:52.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.166+0000 7fbad7400640 1 --2- 172.21.15.31:0/4060621930 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc0da7cc00 0x55fc0da79dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.166+0000 7fbad7400640 1 -- 172.21.15.31:0/4060621930 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc0da74080 msgr2=0x55fc0da79870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:52.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.166+0000 7fbad7400640 1 --2- 172.21.15.31:0/4060621930 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc0da74080 0x55fc0da79870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.166+0000 7fbad7400640 1 -- 172.21.15.31:0/4060621930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fc0d6eb3f0 con 0x55fc0da736d0 2024-03-21T10:45:52.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.166+0000 7fbad7e00640 1 --2- 172.21.15.31:0/4060621930 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc0da7cc00 0x55fc0da79dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:45:52.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.167+0000 7fbad6a00640 1 --2- 172.21.15.31:0/4060621930 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc0da74080 0x55fc0da79870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:45:52.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.167+0000 7fbac0c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:45:52.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.167+0000 7fbad7400640 1 --2- 172.21.15.31:0/4060621930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc0da736d0 0x55fc0da7c4e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fbad0011690 tx=0x7fbad0011b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16c3a521b20e7ad5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:52.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.168+0000 7fbac0c00640 1 -- 172.21.15.31:0/4060621930 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad000fb40 con 0x55fc0da736d0 2024-03-21T10:45:52.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.168+0000 7fbac0c00640 1 -- 172.21.15.31:0/4060621930 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fbad000fce0 con 0x55fc0da736d0 2024-03-21T10:45:52.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.169+0000 7fbae0dc9740 1 -- 172.21.15.31:0/4060621930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbac809f060 con 0x55fc0da736d0 2024-03-21T10:45:52.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.169+0000 7fbac0c00640 1 -- 172.21.15.31:0/4060621930 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbad0011e90 con 0x55fc0da736d0 2024-03-21T10:45:52.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.173+0000 7fbae0dc9740 10 client.27377 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:52.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.173+0000 7fbae0dc9740 1 -- 172.21.15.31:0/4060621930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55fc0d7b57d0 con 0x55fc0da736d0 2024-03-21T10:45:52.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.174+0000 7fbac0c00640 1 -- 172.21.15.31:0/4060621930 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbac809f060 con 0x55fc0da736d0 2024-03-21T10:45:52.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.174+0000 7fbac0c00640 1 -- 172.21.15.31:0/4060621930 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fbad0012640 con 0x55fc0da736d0 2024-03-21T10:45:52.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.176+0000 7fbae0dc9740 10 client.27377 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:52.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.176+0000 7fbae0dc9740 10 client.27377 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:45:52.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.176+0000 7fbae0dc9740 20 client.27377 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:52.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.176+0000 7fbae0dc9740 1 --2- 172.21.15.31:0/4060621930 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55fc0da8c9b0 0x55fc0da8eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.176+0000 7fbae0dc9740 4 client.27377 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:45:52.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.176+0000 7fbae0dc9740 1 -- 172.21.15.31:0/4060621930 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fc0d9e4800 con 0x55fc0da8c9b0 2024-03-21T10:45:52.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.178+0000 7fbad6a00640 1 --2- 172.21.15.31:0/4060621930 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55fc0da8c9b0 0x55fc0da8eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.178+0000 7fbac0c00640 10 client.27377 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:45:52.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.179+0000 7fbad6a00640 1 --2- 172.21.15.31:0/4060621930 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55fc0da8c9b0 0x55fc0da8eda0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=76f5e9c3b9b6a0ed server_cookie=6101e69f0391b2b6 in_seq=0 out_seq=0 2024-03-21T10:45:52.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.180+0000 7fbac0c00640 1 -- 172.21.15.31:0/4060621930 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55fc0d9e4800 con 0x55fc0da8c9b0 2024-03-21T10:45:52.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.180+0000 7fbac0c00640 10 client.27377 handle_command_reply: tid=0 2024-03-21T10:45:52.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.180+0000 7fbae0dc9740 1 client.27377 shutdown 2024-03-21T10:45:52.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.182+0000 7fbae0dc9740 1 -- 172.21.15.31:0/4060621930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc0da736d0 msgr2=0x55fc0da7c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:52.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.182+0000 7fbae0dc9740 1 --2- 172.21.15.31:0/4060621930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc0da736d0 0x55fc0da7c4e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fbad0011690 tx=0x7fbad0011b50 comp rx=0 tx=0).stop 2024-03-21T10:45:52.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.182+0000 7fbae0dc9740 1 -- 172.21.15.31:0/4060621930 shutdown_connections 2024-03-21T10:45:52.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.182+0000 7fbae0dc9740 1 --2- 172.21.15.31:0/4060621930 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55fc0da8c9b0 0x55fc0da8eda0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.182+0000 7fbae0dc9740 1 --2- 172.21.15.31:0/4060621930 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc0da7cc00 0x55fc0da79dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.182+0000 7fbae0dc9740 1 --2- 172.21.15.31:0/4060621930 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc0da74080 0x55fc0da79870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.182+0000 7fbae0dc9740 1 --2- 172.21.15.31:0/4060621930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc0da736d0 0x55fc0da7c4e0 secure :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fbad0011690 tx=0x7fbad0011b50 comp rx=0 tx=0).stop 2024-03-21T10:45:52.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.182+0000 7fbae0dc9740 1 -- 172.21.15.31:0/4060621930 >> 172.21.15.31:0/4060621930 conn(0x55fc0da6dda0 msgr2=0x55fc0da6faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:52.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.182+0000 7fbac0c00640 0 client.27377 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:45:52.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.189+0000 7fbae0dc9740 1 -- 172.21.15.31:0/4060621930 shutdown_connections 2024-03-21T10:45:52.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.189+0000 7fbae0dc9740 1 -- 172.21.15.31:0/4060621930 wait complete. 2024-03-21T10:45:52.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.189+0000 7fbae0dc9740 20 client.27377 trim_cache size 0 max 16384 2024-03-21T10:45:52.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.239+0000 7fbae0dc9740 1 Processor -- start 2024-03-21T10:45:52.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.241+0000 7fbae0dc9740 1 -- start start 2024-03-21T10:45:52.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.241+0000 7fbae0dc9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc0da74080 0x55fc0da7bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.241+0000 7fbae0dc9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc0da7cc00 0x55fc0da7c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.242+0000 7fbae0dc9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc0da785e0 0x55fc0da74b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.242+0000 7fbae0dc9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad81bff10 con 0x55fc0da785e0 2024-03-21T10:45:52.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.242+0000 7fbae0dc9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad81bda10 con 0x55fc0da74080 2024-03-21T10:45:52.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.242+0000 7fbae0dc9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad81bbad0 con 0x55fc0da7cc00 2024-03-21T10:45:52.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.242+0000 7fbad7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc0da74080 0x55fc0da7bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.242+0000 7fbad7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc0da74080 0x55fc0da7bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:35820/0 (socket says 172.21.15.31:35820) 2024-03-21T10:45:52.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.242+0000 7fbad7400640 1 -- 172.21.15.31:0/3045580887 learned_addr learned my addr 172.21.15.31:0/3045580887 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:52.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.242+0000 7fbad7e00640 1 --2- 172.21.15.31:0/3045580887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc0da785e0 0x55fc0da74b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.242+0000 7fbad7400640 1 -- 172.21.15.31:0/3045580887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc0da7cc00 msgr2=0x55fc0da7c4e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:45:52.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.242+0000 7fbad7400640 1 --2- 172.21.15.31:0/3045580887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc0da7cc00 0x55fc0da7c4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.242+0000 7fbad7400640 1 -- 172.21.15.31:0/3045580887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc0da785e0 msgr2=0x55fc0da74b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:52.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.242+0000 7fbad7400640 1 --2- 172.21.15.31:0/3045580887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc0da785e0 0x55fc0da74b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.242+0000 7fbad7400640 1 -- 172.21.15.31:0/3045580887 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fc0da6d520 con 0x55fc0da74080 2024-03-21T10:45:52.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.243+0000 7fbad7400640 1 --2- 172.21.15.31:0/3045580887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc0da74080 0x55fc0da7bfa0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fbad0002230 tx=0x7fbad0005df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=153cbd79b8848ab4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:52.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.244+0000 7fbac3400640 1 -- 172.21.15.31:0/3045580887 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad002d040 con 0x55fc0da74080 2024-03-21T10:45:52.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.244+0000 7fbac3400640 1 -- 172.21.15.31:0/3045580887 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbad0011e90 con 0x55fc0da74080 2024-03-21T10:45:52.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.244+0000 7fbac3400640 1 -- 172.21.15.31:0/3045580887 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad0004cd0 con 0x55fc0da74080 2024-03-21T10:45:52.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.245+0000 7fbae0dc9740 1 -- 172.21.15.31:0/3045580887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc0da74080 msgr2=0x55fc0da7bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:52.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.245+0000 7fbae0dc9740 1 --2- 172.21.15.31:0/3045580887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc0da74080 0x55fc0da7bfa0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fbad0002230 tx=0x7fbad0005df0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.246+0000 7fbae0dc9740 1 -- 172.21.15.31:0/3045580887 shutdown_connections 2024-03-21T10:45:52.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.249+0000 7fbae0dc9740 1 --2- 172.21.15.31:0/3045580887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc0da785e0 0x55fc0da74b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.249+0000 7fbae0dc9740 1 --2- 172.21.15.31:0/3045580887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc0da7cc00 0x55fc0da7c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.249+0000 7fbae0dc9740 1 --2- 172.21.15.31:0/3045580887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc0da74080 0x55fc0da7bfa0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.249+0000 7fbae0dc9740 1 -- 172.21.15.31:0/3045580887 >> 172.21.15.31:0/3045580887 conn(0x55fc0da6dda0 msgr2=0x55fc0da74460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:52.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.253+0000 7fbae0dc9740 1 -- 172.21.15.31:0/3045580887 shutdown_connections 2024-03-21T10:45:52.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.254+0000 7fbae0dc9740 1 -- 172.21.15.31:0/3045580887 wait complete. 2024-03-21T10:45:52.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.256+0000 7fbae0dc9740 1 -- start start 2024-03-21T10:45:52.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.269+0000 7fbae0dc9740 1 Processor -- start 2024-03-21T10:45:52.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.279+0000 7fbae0dc9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc0da74080 0x55fc0dae3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.279+0000 7fbae0dc9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc0da785e0 0x55fc0dae3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.279+0000 7fbae0dc9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc0da7cc00 0x55fc0dae3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.280+0000 7fbae0dc9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad81bc0d0 con 0x55fc0da785e0 2024-03-21T10:45:52.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.280+0000 7fbae0dc9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad81bd3b0 con 0x55fc0da7cc00 2024-03-21T10:45:52.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.280+0000 7fbae0dc9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad81bc6f0 con 0x55fc0da74080 2024-03-21T10:45:52.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.281+0000 7fbae0dc9740 10 client.0 _ll_register_callbacks cb 0x7fbad00099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:52.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.281+0000 7fbad6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc0da785e0 0x55fc0dae3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.282+0000 7fbad6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc0da785e0 0x55fc0dae3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44684/0 (socket says 172.21.15.31:44684) 2024-03-21T10:45:52.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.282+0000 7fbad6a00640 1 -- 172.21.15.31:0/2901733532 learned_addr learned my addr 172.21.15.31:0/2901733532 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:52.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.282+0000 7fbad6a00640 1 -- 172.21.15.31:0/2901733532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc0da74080 msgr2=0x55fc0dae3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:52.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.283+0000 7fbad6a00640 1 --2- 172.21.15.31:0/2901733532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc0da74080 0x55fc0dae3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.283+0000 7fbad6a00640 1 -- 172.21.15.31:0/2901733532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc0da7cc00 msgr2=0x55fc0dae3ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:45:52.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.283+0000 7fbad6a00640 1 --2- 172.21.15.31:0/2901733532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc0da7cc00 0x55fc0dae3ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.283+0000 7fbad6a00640 1 -- 172.21.15.31:0/2901733532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fc0dae1ad0 con 0x55fc0da785e0 2024-03-21T10:45:52.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.284+0000 7fbad6a00640 1 --2- 172.21.15.31:0/2901733532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc0da785e0 0x55fc0dae3590 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fbac8009420 tx=0x7fbac809b090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3e4d60bf9b1bf5e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:52.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.284+0000 7fbac0c00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:45:52.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.285+0000 7fbac0c00640 1 -- 172.21.15.31:0/2901733532 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac807e060 con 0x55fc0da785e0 2024-03-21T10:45:52.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.286+0000 7fbac0c00640 1 -- 172.21.15.31:0/2901733532 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fbac8017360 con 0x55fc0da785e0 2024-03-21T10:45:52.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.286+0000 7fbae0dc9740 1 -- 172.21.15.31:0/2901733532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbad0011e90 con 0x55fc0da785e0 2024-03-21T10:45:52.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.287+0000 7fbac0c00640 1 -- 172.21.15.31:0/2901733532 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbac801e070 con 0x55fc0da785e0 2024-03-21T10:45:52.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.287+0000 7fbae0dc9740 10 client.16806 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:52.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.287+0000 7fbae0dc9740 1 -- 172.21.15.31:0/2901733532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55fc0da844d0 con 0x55fc0da785e0 2024-03-21T10:45:52.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.291+0000 7fbac0c00640 1 -- 172.21.15.31:0/2901733532 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbad0011e90 con 0x55fc0da785e0 2024-03-21T10:45:52.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.291+0000 7fbac0c00640 1 -- 172.21.15.31:0/2901733532 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fbac8097580 con 0x55fc0da785e0 2024-03-21T10:45:52.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.292+0000 7fbae0dc9740 10 client.16806 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:52.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.292+0000 7fbae0dc9740 10 client.16806 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:45:52.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.292+0000 7fbae0dc9740 20 client.16806 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:52.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.292+0000 7fbae0dc9740 1 --2- 172.21.15.31:0/2901733532 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55fc0dafb1b0 0x55fc0dafd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.292+0000 7fbae0dc9740 4 client.16806 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:45:52.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.292+0000 7fbae0dc9740 1 -- 172.21.15.31:0/2901733532 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55fc0d9deb50 con 0x55fc0dafb1b0 2024-03-21T10:45:52.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.297+0000 7fbad7400640 1 --2- 172.21.15.31:0/2901733532 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55fc0dafb1b0 0x55fc0dafd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.303+0000 7fbad7400640 1 --2- 172.21.15.31:0/2901733532 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55fc0dafb1b0 0x55fc0dafd5a0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=42687274a4c3fde4 server_cookie=f850218e48596f89 in_seq=0 out_seq=0 2024-03-21T10:45:52.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.303+0000 7fbac0c00640 10 client.16806 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:45:52.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.303+0000 7fbac0c00640 1 -- 172.21.15.31:0/2901733532 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55fc0d9deb50 con 0x55fc0dafb1b0 2024-03-21T10:45:52.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.304+0000 7fbac0c00640 10 client.16806 handle_command_reply: tid=0 2024-03-21T10:45:52.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.304+0000 7fbae0dc9740 1 client.16806 shutdown 2024-03-21T10:45:52.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.306+0000 7fbae0dc9740 1 -- 172.21.15.31:0/2901733532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc0da785e0 msgr2=0x55fc0dae3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:52.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.306+0000 7fbae0dc9740 1 --2- 172.21.15.31:0/2901733532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc0da785e0 0x55fc0dae3590 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fbac8009420 tx=0x7fbac809b090 comp rx=0 tx=0).stop 2024-03-21T10:45:52.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.307+0000 7fbae0dc9740 1 -- 172.21.15.31:0/2901733532 shutdown_connections 2024-03-21T10:45:52.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.307+0000 7fbae0dc9740 1 --2- 172.21.15.31:0/2901733532 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55fc0dafb1b0 0x55fc0dafd5a0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.307+0000 7fbac0c00640 0 client.16806 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:45:52.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.307+0000 7fbae0dc9740 1 --2- 172.21.15.31:0/2901733532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fc0da7cc00 0x55fc0dae3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.307+0000 7fbae0dc9740 1 --2- 172.21.15.31:0/2901733532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc0da785e0 0x55fc0dae3590 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.307+0000 7fbae0dc9740 1 --2- 172.21.15.31:0/2901733532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc0da74080 0x55fc0dae3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.307+0000 7fbae0dc9740 1 -- 172.21.15.31:0/2901733532 >> 172.21.15.31:0/2901733532 conn(0x55fc0da6dda0 msgr2=0x55fc0da6f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:52.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.308+0000 7fbae0dc9740 1 -- 172.21.15.31:0/2901733532 shutdown_connections 2024-03-21T10:45:52.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.308+0000 7fbae0dc9740 1 -- 172.21.15.31:0/2901733532 wait complete. 2024-03-21T10:45:52.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.309+0000 7fbae0dc9740 20 client.16806 trim_cache size 0 max 16384 2024-03-21T10:45:52.308 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:45:52.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.315+0000 7fbade000640 1 -- 172.21.15.31:0/1302533584 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbabc0a3110 msgr2=0x7fbabc0a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:52.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.315+0000 7fbade000640 1 --2- 172.21.15.31:0/1302533584 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbabc0a3110 0x7fbabc0a55d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbac402c1c0 tx=0x7fbac4005fd0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.315+0000 7fbade000640 1 -- 172.21.15.31:0/1302533584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbad810fe30 msgr2=0x7fbad81a7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:52.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.315+0000 7fbade000640 1 --2- 172.21.15.31:0/1302533584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbad810fe30 0x7fbad81a7350 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fbac800b4a0 tx=0x7fbac800b960 comp rx=0 tx=0).stop 2024-03-21T10:45:52.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.316+0000 7fbade000640 1 -- 172.21.15.31:0/1302533584 shutdown_connections 2024-03-21T10:45:52.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.316+0000 7fbade000640 1 --2- 172.21.15.31:0/1302533584 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbabc0a3110 0x7fbabc0a55d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.316+0000 7fbade000640 1 --2- 172.21.15.31:0/1302533584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbad811ae80 0x7fbad81ab6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.316+0000 7fbade000640 1 --2- 172.21.15.31:0/1302533584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbad810fe30 0x7fbad81a7350 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.316+0000 7fbade000640 1 --2- 172.21.15.31:0/1302533584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad8108be0 0x7fbad81a6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.316+0000 7fbade000640 1 -- 172.21.15.31:0/1302533584 >> 172.21.15.31:0/1302533584 conn(0x7fbad806c710 msgr2=0x7fbad8118e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:52.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.316+0000 7fbade000640 1 -- 172.21.15.31:0/1302533584 shutdown_connections 2024-03-21T10:45:52.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.317+0000 7fbade000640 1 -- 172.21.15.31:0/1302533584 wait complete. 2024-03-21T10:45:52.344 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:45:52.344 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-21T10:45:52.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:52 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/34414577' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:45:52.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.735+0000 7fe41ec00640 1 -- 172.21.15.31:0/2988109442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe418104200 msgr2=0x7fe418110710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:52.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.735+0000 7fe41ec00640 1 --2- 172.21.15.31:0/2988109442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe418104200 0x7fe418110710 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe408009890 tx=0x7fe40802bb70 comp rx=0 tx=0).stop 2024-03-21T10:45:52.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.738+0000 7fe41ec00640 1 -- 172.21.15.31:0/2988109442 shutdown_connections 2024-03-21T10:45:52.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.738+0000 7fe41ec00640 1 --2- 172.21.15.31:0/2988109442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe418104200 0x7fe418110710 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.738+0000 7fe41ec00640 1 --2- 172.21.15.31:0/2988109442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe418103840 0x7fe418103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.738+0000 7fe41ec00640 1 --2- 172.21.15.31:0/2988109442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe418109850 0x7fe418109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.738+0000 7fe41ec00640 1 -- 172.21.15.31:0/2988109442 >> 172.21.15.31:0/2988109442 conn(0x7fe4180772d0 msgr2=0x7fe418101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:52.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.741+0000 7fe41ec00640 1 -- 172.21.15.31:0/2988109442 shutdown_connections 2024-03-21T10:45:52.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.742+0000 7fe41ec00640 1 -- 172.21.15.31:0/2988109442 wait complete. 2024-03-21T10:45:52.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.744+0000 7fe41ec00640 1 Processor -- start 2024-03-21T10:45:52.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.747+0000 7fe41ec00640 1 -- start start 2024-03-21T10:45:52.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.747+0000 7fe41ec00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe418103840 0x7fe418072eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.747+0000 7fe41ec00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe418104200 0x7fe41806bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.747+0000 7fe41ec00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe418109850 0x7fe41806c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.747+0000 7fe41ec00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe418113c20 con 0x7fe418104200 2024-03-21T10:45:52.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.747+0000 7fe41ec00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe418113aa0 con 0x7fe418103840 2024-03-21T10:45:52.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.747+0000 7fe41ec00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe418113da0 con 0x7fe418109850 2024-03-21T10:45:52.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.748+0000 7fe41cc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe418109850 0x7fe41806c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.748+0000 7fe417e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe418103840 0x7fe418072eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.748+0000 7fe41cc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe418109850 0x7fe41806c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:50820/0 (socket says 172.21.15.31:50820) 2024-03-21T10:45:52.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.748+0000 7fe41cc00640 1 -- 172.21.15.31:0/1356325718 learned_addr learned my addr 172.21.15.31:0/1356325718 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:52.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.748+0000 7fe41cc00640 1 -- 172.21.15.31:0/1356325718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe418103840 msgr2=0x7fe418072eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:52.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.748+0000 7fe41cc00640 1 --2- 172.21.15.31:0/1356325718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe418103840 0x7fe418072eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.748+0000 7fe41cc00640 1 -- 172.21.15.31:0/1356325718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe418104200 msgr2=0x7fe41806bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:52.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.748+0000 7fe41cc00640 1 --2- 172.21.15.31:0/1356325718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe418104200 0x7fe41806bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.748+0000 7fe41cc00640 1 -- 172.21.15.31:0/1356325718 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4180735e0 con 0x7fe418109850 2024-03-21T10:45:52.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.749+0000 7fe41cc00640 1 --2- 172.21.15.31:0/1356325718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe418109850 0x7fe41806c470 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe408037040 tx=0x7fe408004530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:52.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.751+0000 7fe414c00640 1 -- 172.21.15.31:0/1356325718 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe408041020 con 0x7fe418109850 2024-03-21T10:45:52.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.752+0000 7fe414c00640 1 -- 172.21.15.31:0/1356325718 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe40802c9f0 con 0x7fe418109850 2024-03-21T10:45:52.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.752+0000 7fe414c00640 1 -- 172.21.15.31:0/1356325718 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe408039040 con 0x7fe418109850 2024-03-21T10:45:52.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.756+0000 7fe41ec00640 1 -- 172.21.15.31:0/1356325718 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe41806c9b0 con 0x7fe418109850 2024-03-21T10:45:52.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.756+0000 7fe41ec00640 1 -- 172.21.15.31:0/1356325718 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4180709d0 con 0x7fe418109850 2024-03-21T10:45:52.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.767+0000 7fe421996740 1 Processor -- start 2024-03-21T10:45:52.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.770+0000 7fe414c00640 1 -- 172.21.15.31:0/1356325718 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fe408044020 con 0x7fe418109850 2024-03-21T10:45:52.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.771+0000 7fe421996740 1 -- start start 2024-03-21T10:45:52.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.771+0000 7fe421996740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584ea69d6d0 0x5584ea69dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.771+0000 7fe421996740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5584ea69e080 0x5584ea6a6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.771+0000 7fe421996740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584ea6a6c00 0x5584ea6a9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.771+0000 7fe421996740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584ea6012c0 con 0x5584ea69e080 2024-03-21T10:45:52.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.771+0000 7fe421996740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584ea5a9c90 con 0x5584ea6a6c00 2024-03-21T10:45:52.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.771+0000 7fe421996740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584ea5fd360 con 0x5584ea69d6d0 2024-03-21T10:45:52.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.772+0000 7fe417e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5584ea69e080 0x5584ea6a6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.772+0000 7fe417e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5584ea69e080 0x5584ea6a6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44708/0 (socket says 172.21.15.31:44708) 2024-03-21T10:45:52.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.772+0000 7fe417e00640 1 -- 172.21.15.31:0/396930838 learned_addr learned my addr 172.21.15.31:0/396930838 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:52.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.773+0000 7fe417400640 1 --2- 172.21.15.31:0/396930838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584ea6a6c00 0x5584ea6a9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.773+0000 7fe417400640 1 --2- 172.21.15.31:0/396930838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584ea69d6d0 0x5584ea69dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.773+0000 7fe417e00640 1 -- 172.21.15.31:0/396930838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584ea69d6d0 msgr2=0x5584ea69dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:52.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.773+0000 7fe417e00640 1 --2- 172.21.15.31:0/396930838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584ea69d6d0 0x5584ea69dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.773+0000 7fe417e00640 1 -- 172.21.15.31:0/396930838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584ea6a6c00 msgr2=0x5584ea6a9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:52.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.773+0000 7fe417e00640 1 --2- 172.21.15.31:0/396930838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584ea6a6c00 0x5584ea6a9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.773+0000 7fe417e00640 1 -- 172.21.15.31:0/396930838 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5584ea5ce080 con 0x5584ea69e080 2024-03-21T10:45:52.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.774+0000 7fe417e00640 1 --2- 172.21.15.31:0/396930838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5584ea69e080 0x5584ea6a6630 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe410009fd0 tx=0x7fe4100095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42c0c5bbf0eaae1 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:52.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.775+0000 7fe40d200640 1 -- 172.21.15.31:0/396930838 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41001d070 con 0x5584ea69e080 2024-03-21T10:45:52.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.776+0000 7fe414c00640 1 --2- 172.21.15.31:0/1356325718 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe3fc0a3260 0x7fe3fc0a5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.776+0000 7fe414c00640 1 -- 172.21.15.31:0/1356325718 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fe4080f09b0 con 0x7fe418109850 2024-03-21T10:45:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.776+0000 7fe40d200640 1 -- 172.21.15.31:0/396930838 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe410003e40 con 0x5584ea69e080 2024-03-21T10:45:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.777+0000 7fe421996740 1 -- 172.21.15.31:0/396930838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5584ea69e080 msgr2=0x5584ea6a6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.777+0000 7fe421996740 1 --2- 172.21.15.31:0/396930838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5584ea69e080 0x5584ea6a6630 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe410009fd0 tx=0x7fe4100095c0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.778+0000 7fe417400640 1 --2- 172.21.15.31:0/1356325718 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe3fc0a3260 0x7fe3fc0a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.779+0000 7fe421996740 1 -- 172.21.15.31:0/396930838 shutdown_connections 2024-03-21T10:45:52.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.781+0000 7fe421996740 1 --2- 172.21.15.31:0/396930838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584ea6a6c00 0x5584ea6a9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.781+0000 7fe421996740 1 --2- 172.21.15.31:0/396930838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5584ea69e080 0x5584ea6a6630 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.781+0000 7fe417400640 1 --2- 172.21.15.31:0/1356325718 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe3fc0a3260 0x7fe3fc0a5720 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe4000058c0 tx=0x7fe4000046c0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:52.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.781+0000 7fe421996740 1 --2- 172.21.15.31:0/396930838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584ea69d6d0 0x5584ea69dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.781+0000 7fe421996740 1 -- 172.21.15.31:0/396930838 >> 172.21.15.31:0/396930838 conn(0x5584ea697da0 msgr2=0x5584ea698180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:52.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.782+0000 7fe421996740 1 -- 172.21.15.31:0/396930838 shutdown_connections 2024-03-21T10:45:52.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.782+0000 7fe421996740 1 -- 172.21.15.31:0/396930838 wait complete. 2024-03-21T10:45:52.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.782+0000 7fe421996740 1 -- start start 2024-03-21T10:45:52.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.783+0000 7fe421996740 1 Processor -- start 2024-03-21T10:45:52.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.784+0000 7fe421996740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584ea69d6d0 0x5584ea6a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.785+0000 7fe421996740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5584ea69e080 0x5584ea6a3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.785+0000 7fe417e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584ea69d6d0 0x5584ea6a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.785+0000 7fe417e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584ea69d6d0 0x5584ea6a63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:50840/0 (socket says 172.21.15.31:50840) 2024-03-21T10:45:52.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.785+0000 7fe421996740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584ea6a6c00 0x5584ea6a3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.787+0000 7fe417e00640 1 -- 172.21.15.31:0/4205004760 learned_addr learned my addr 172.21.15.31:0/4205004760 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:52.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.787+0000 7fe421996740 1 -- 172.21.15.31:0/4205004760 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584ea6a9c10 con 0x5584ea69e080 2024-03-21T10:45:52.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.787+0000 7fe421996740 1 -- 172.21.15.31:0/4205004760 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584ea5bd0d0 con 0x5584ea6a6c00 2024-03-21T10:45:52.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.787+0000 7fe421996740 1 -- 172.21.15.31:0/4205004760 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584ea6a9d90 con 0x5584ea69d6d0 2024-03-21T10:45:52.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.788+0000 7fe421996740 10 client.0 _ll_register_callbacks cb 0x5584ea5c1e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:52.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.788+0000 7fe41cc00640 1 --2- 172.21.15.31:0/4205004760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584ea6a6c00 0x5584ea6a3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.788+0000 7fe417400640 1 --2- 172.21.15.31:0/4205004760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5584ea69e080 0x5584ea6a3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.789+0000 7fe41cc00640 1 -- 172.21.15.31:0/4205004760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584ea69d6d0 msgr2=0x5584ea6a63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:52.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.789+0000 7fe41cc00640 1 --2- 172.21.15.31:0/4205004760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584ea69d6d0 0x5584ea6a63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.789+0000 7fe41cc00640 1 -- 172.21.15.31:0/4205004760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5584ea69e080 msgr2=0x5584ea6a3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:52.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.789+0000 7fe41cc00640 1 --2- 172.21.15.31:0/4205004760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5584ea69e080 0x5584ea6a3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.789+0000 7fe41cc00640 1 -- 172.21.15.31:0/4205004760 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5584ea3153f0 con 0x5584ea6a6c00 2024-03-21T10:45:52.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.793+0000 7fe41cc00640 1 --2- 172.21.15.31:0/4205004760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584ea6a6c00 0x5584ea6a3cc0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe4080ede10 tx=0x7fe4080bc9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7964f3663fbd93e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:52.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.793+0000 7fe405600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:45:52.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.793+0000 7fe405600640 1 -- 172.21.15.31:0/4205004760 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4080d5e60 con 0x5584ea6a6c00 2024-03-21T10:45:52.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.793+0000 7fe405600640 1 -- 172.21.15.31:0/4205004760 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fe408032cb0 con 0x5584ea6a6c00 2024-03-21T10:45:52.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.794+0000 7fe421996740 1 -- 172.21.15.31:0/4205004760 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5584ea60e800 con 0x5584ea6a6c00 2024-03-21T10:45:52.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.798+0000 7fe405600640 1 -- 172.21.15.31:0/4205004760 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4080bde60 con 0x5584ea6a6c00 2024-03-21T10:45:52.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.799+0000 7fe421996740 10 client.27392 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:52.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.799+0000 7fe421996740 1 -- 172.21.15.31:0/4205004760 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5584ea3df7d0 con 0x5584ea6a6c00 2024-03-21T10:45:52.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.800+0000 7fe405600640 1 -- 172.21.15.31:0/4205004760 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4080d2050 con 0x5584ea6a6c00 2024-03-21T10:45:52.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.801+0000 7fe405600640 1 -- 172.21.15.31:0/4205004760 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fe40803c030 con 0x5584ea6a6c00 2024-03-21T10:45:52.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.801+0000 7fe421996740 10 client.27392 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:52.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.801+0000 7fe421996740 10 client.27392 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:45:52.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.801+0000 7fe421996740 20 client.27392 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:52.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.801+0000 7fe421996740 1 --2- 172.21.15.31:0/4205004760 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x5584ea6b69f0 0x5584ea6b8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.801+0000 7fe421996740 4 client.27392 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:45:52.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.801+0000 7fe421996740 1 -- 172.21.15.31:0/4205004760 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5584ea60e800 con 0x5584ea6b69f0 2024-03-21T10:45:52.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.806+0000 7fe417e00640 1 --2- 172.21.15.31:0/4205004760 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x5584ea6b69f0 0x5584ea6b8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:52 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/34414577' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:45:52.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.809+0000 7fe417e00640 1 --2- 172.21.15.31:0/4205004760 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x5584ea6b69f0 0x5584ea6b8e10 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a9448eb202d91045 server_cookie=8f7da30444becb88 in_seq=0 out_seq=0 2024-03-21T10:45:52.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.809+0000 7fe405600640 10 client.27392 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:45:52.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.812+0000 7fe405600640 1 -- 172.21.15.31:0/4205004760 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5584ea60e800 con 0x5584ea6b69f0 2024-03-21T10:45:52.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.812+0000 7fe405600640 10 client.27392 handle_command_reply: tid=0 2024-03-21T10:45:52.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.812+0000 7fe421996740 1 client.27392 shutdown 2024-03-21T10:45:52.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.814+0000 7fe421996740 1 -- 172.21.15.31:0/4205004760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584ea6a6c00 msgr2=0x5584ea6a3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:52.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.814+0000 7fe421996740 1 --2- 172.21.15.31:0/4205004760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584ea6a6c00 0x5584ea6a3cc0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe4080ede10 tx=0x7fe4080bc9c0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.814+0000 7fe421996740 1 -- 172.21.15.31:0/4205004760 shutdown_connections 2024-03-21T10:45:52.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.814+0000 7fe421996740 1 --2- 172.21.15.31:0/4205004760 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x5584ea6b69f0 0x5584ea6b8e10 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.815+0000 7fe421996740 1 --2- 172.21.15.31:0/4205004760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584ea6a6c00 0x5584ea6a3cc0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.815+0000 7fe421996740 1 --2- 172.21.15.31:0/4205004760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5584ea69e080 0x5584ea6a3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.815+0000 7fe421996740 1 --2- 172.21.15.31:0/4205004760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584ea69d6d0 0x5584ea6a63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.815+0000 7fe421996740 1 -- 172.21.15.31:0/4205004760 >> 172.21.15.31:0/4205004760 conn(0x5584ea697da0 msgr2=0x5584ea699af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:52.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.815+0000 7fe405600640 0 client.27392 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:45:52.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.819+0000 7fe421996740 1 -- 172.21.15.31:0/4205004760 shutdown_connections 2024-03-21T10:45:52.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.819+0000 7fe421996740 1 -- 172.21.15.31:0/4205004760 wait complete. 2024-03-21T10:45:52.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.820+0000 7fe421996740 20 client.27392 trim_cache size 0 max 16384 2024-03-21T10:45:52.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:52 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/34414577' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:45:52.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.850+0000 7fe421996740 1 Processor -- start 2024-03-21T10:45:52.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.851+0000 7fe421996740 1 -- start start 2024-03-21T10:45:52.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.851+0000 7fe421996740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584ea69d6d0 0x5584ea69e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.851+0000 7fe421996740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5584ea69e080 0x5584ea6a5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.852+0000 7fe421996740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584ea6a6c00 0x5584ea6a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.852+0000 7fe421996740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4181bfdf0 con 0x5584ea69e080 2024-03-21T10:45:52.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.852+0000 7fe421996740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4181bd8f0 con 0x5584ea69d6d0 2024-03-21T10:45:52.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.852+0000 7fe421996740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4181bb9b0 con 0x5584ea6a6c00 2024-03-21T10:45:52.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.852+0000 7fe417400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5584ea69e080 0x5584ea6a5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.852+0000 7fe417400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5584ea69e080 0x5584ea6a5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44732/0 (socket says 172.21.15.31:44732) 2024-03-21T10:45:52.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.853+0000 7fe417400640 1 -- 172.21.15.31:0/3351901989 learned_addr learned my addr 172.21.15.31:0/3351901989 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:52.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.853+0000 7fe41cc00640 1 --2- 172.21.15.31:0/3351901989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584ea6a6c00 0x5584ea6a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.853+0000 7fe417e00640 1 --2- 172.21.15.31:0/3351901989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584ea69d6d0 0x5584ea69e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.853+0000 7fe41cc00640 1 -- 172.21.15.31:0/3351901989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584ea69d6d0 msgr2=0x5584ea69e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:52.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.853+0000 7fe41cc00640 1 --2- 172.21.15.31:0/3351901989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584ea69d6d0 0x5584ea69e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.853+0000 7fe41cc00640 1 -- 172.21.15.31:0/3351901989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5584ea69e080 msgr2=0x5584ea6a5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:52.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.854+0000 7fe41cc00640 1 --2- 172.21.15.31:0/3351901989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5584ea69e080 0x5584ea6a5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.854+0000 7fe41cc00640 1 -- 172.21.15.31:0/3351901989 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5584ea697bd0 con 0x5584ea6a6c00 2024-03-21T10:45:52.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.854+0000 7fe417400640 1 --2- 172.21.15.31:0/3351901989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5584ea69e080 0x5584ea6a5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:45:52.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.854+0000 7fe41cc00640 1 --2- 172.21.15.31:0/3351901989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584ea6a6c00 0x5584ea6a63f0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fe4080bc830 tx=0x7fe4080bca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1930297f6d2898d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:52.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.854+0000 7fe407e00640 1 -- 172.21.15.31:0/3351901989 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4080d5e60 con 0x5584ea6a6c00 2024-03-21T10:45:52.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.857+0000 7fe407e00640 1 -- 172.21.15.31:0/3351901989 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4080d2050 con 0x5584ea6a6c00 2024-03-21T10:45:52.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.857+0000 7fe407e00640 1 -- 172.21.15.31:0/3351901989 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4080ca080 con 0x5584ea6a6c00 2024-03-21T10:45:52.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.858+0000 7fe421996740 1 -- 172.21.15.31:0/3351901989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584ea6a6c00 msgr2=0x5584ea6a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:52.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.858+0000 7fe421996740 1 --2- 172.21.15.31:0/3351901989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584ea6a6c00 0x5584ea6a63f0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fe4080bc830 tx=0x7fe4080bca30 comp rx=0 tx=0).stop 2024-03-21T10:45:52.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.858+0000 7fe421996740 1 -- 172.21.15.31:0/3351901989 shutdown_connections 2024-03-21T10:45:52.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.858+0000 7fe421996740 1 --2- 172.21.15.31:0/3351901989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584ea6a6c00 0x5584ea6a63f0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.858+0000 7fe421996740 1 --2- 172.21.15.31:0/3351901989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5584ea69e080 0x5584ea6a5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.858+0000 7fe421996740 1 --2- 172.21.15.31:0/3351901989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584ea69d6d0 0x5584ea69e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.858+0000 7fe421996740 1 -- 172.21.15.31:0/3351901989 >> 172.21.15.31:0/3351901989 conn(0x5584ea697da0 msgr2=0x5584ea696eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:52.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.858+0000 7fe421996740 1 -- 172.21.15.31:0/3351901989 shutdown_connections 2024-03-21T10:45:52.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.858+0000 7fe421996740 1 -- 172.21.15.31:0/3351901989 wait complete. 2024-03-21T10:45:52.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.859+0000 7fe421996740 1 -- start start 2024-03-21T10:45:52.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.859+0000 7fe421996740 1 Processor -- start 2024-03-21T10:45:52.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.862+0000 7fe421996740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584ea69d6d0 0x5584ea6aab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.862+0000 7fe421996740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584ea69e080 0x5584ea6ab060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.862+0000 7fe421996740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5584ea6a6c00 0x5584ea7222f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.863+0000 7fe417400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584ea69e080 0x5584ea6ab060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.863+0000 7fe417e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584ea69d6d0 0x5584ea6aab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.863+0000 7fe417400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584ea69e080 0x5584ea6ab060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:35864/0 (socket says 172.21.15.31:35864) 2024-03-21T10:45:52.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.863+0000 7fe417400640 1 -- 172.21.15.31:0/161663568 learned_addr learned my addr 172.21.15.31:0/161663568 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:52.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.863+0000 7fe421996740 1 -- 172.21.15.31:0/161663568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4181bbfb0 con 0x5584ea6a6c00 2024-03-21T10:45:52.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.863+0000 7fe421996740 1 -- 172.21.15.31:0/161663568 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4181bd290 con 0x5584ea69e080 2024-03-21T10:45:52.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.864+0000 7fe421996740 1 -- 172.21.15.31:0/161663568 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4181bc5d0 con 0x5584ea69d6d0 2024-03-21T10:45:52.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.864+0000 7fe421996740 10 client.0 _ll_register_callbacks cb 0x7fe408032640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:52.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.864+0000 7fe41cc00640 1 --2- 172.21.15.31:0/161663568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5584ea6a6c00 0x5584ea7222f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.865+0000 7fe417e00640 1 -- 172.21.15.31:0/161663568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584ea69e080 msgr2=0x5584ea6ab060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:52.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.865+0000 7fe417e00640 1 --2- 172.21.15.31:0/161663568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584ea69e080 0x5584ea6ab060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.865+0000 7fe417e00640 1 -- 172.21.15.31:0/161663568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5584ea6a6c00 msgr2=0x5584ea7222f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:52.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.865+0000 7fe417e00640 1 --2- 172.21.15.31:0/161663568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5584ea6a6c00 0x5584ea7222f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.865+0000 7fe417e00640 1 -- 172.21.15.31:0/161663568 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5584ea693510 con 0x5584ea69d6d0 2024-03-21T10:45:52.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.865+0000 7fe41cc00640 1 --2- 172.21.15.31:0/161663568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5584ea6a6c00 0x5584ea7222f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:45:52.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.866+0000 7fe417e00640 1 --2- 172.21.15.31:0/161663568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584ea69d6d0 0x5584ea6aab20 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fe4100099a0 tx=0x7fe410003970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dac0f04e76c11311 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:52.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.866+0000 7fe405600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:45:52.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.867+0000 7fe405600640 1 -- 172.21.15.31:0/161663568 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41000de00 con 0x5584ea69d6d0 2024-03-21T10:45:52.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.868+0000 7fe421996740 1 -- 172.21.15.31:0/161663568 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe4080d2050 con 0x5584ea69d6d0 2024-03-21T10:45:52.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.868+0000 7fe405600640 1 -- 172.21.15.31:0/161663568 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fe4100050d0 con 0x5584ea69d6d0 2024-03-21T10:45:52.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.870+0000 7fe405600640 1 -- 172.21.15.31:0/161663568 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe410009c30 con 0x5584ea69d6d0 2024-03-21T10:45:52.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.870+0000 7fe421996740 10 client.27310 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:52.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.870+0000 7fe421996740 1 -- 172.21.15.31:0/161663568 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5584ea723900 con 0x5584ea69d6d0 2024-03-21T10:45:52.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.870+0000 7fe405600640 1 -- 172.21.15.31:0/161663568 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4080d2050 con 0x5584ea69d6d0 2024-03-21T10:45:52.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.870+0000 7fe405600640 1 -- 172.21.15.31:0/161663568 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fe410020040 con 0x5584ea69d6d0 2024-03-21T10:45:52.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.874+0000 7fe421996740 10 client.27310 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:52.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.874+0000 7fe421996740 10 client.27310 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:45:52.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.874+0000 7fe421996740 20 client.27310 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:52.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.874+0000 7fe421996740 1 --2- 172.21.15.31:0/161663568 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x5584ea723e30 0x5584ea726220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:52.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.874+0000 7fe421996740 4 client.27310 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:45:52.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.874+0000 7fe421996740 1 -- 172.21.15.31:0/161663568 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5584ea608b50 con 0x5584ea723e30 2024-03-21T10:45:52.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.876+0000 7fe417400640 1 --2- 172.21.15.31:0/161663568 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x5584ea723e30 0x5584ea726220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:52.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.879+0000 7fe417400640 1 --2- 172.21.15.31:0/161663568 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x5584ea723e30 0x5584ea726220 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e5c548925f2d6e06 server_cookie=c22cb36b80bf373d in_seq=0 out_seq=0 2024-03-21T10:45:52.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.879+0000 7fe405600640 10 client.27310 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:45:52.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.880+0000 7fe405600640 1 -- 172.21.15.31:0/161663568 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5584ea608b50 con 0x5584ea723e30 2024-03-21T10:45:52.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.880+0000 7fe405600640 10 client.27310 handle_command_reply: tid=0 2024-03-21T10:45:52.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.880+0000 7fe421996740 1 client.27310 shutdown 2024-03-21T10:45:52.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.881+0000 7fe421996740 1 -- 172.21.15.31:0/161663568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584ea69d6d0 msgr2=0x5584ea6aab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:52.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.881+0000 7fe421996740 1 --2- 172.21.15.31:0/161663568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584ea69d6d0 0x5584ea6aab20 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fe4100099a0 tx=0x7fe410003970 comp rx=0 tx=0).stop 2024-03-21T10:45:52.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.881+0000 7fe421996740 1 -- 172.21.15.31:0/161663568 shutdown_connections 2024-03-21T10:45:52.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.881+0000 7fe421996740 1 --2- 172.21.15.31:0/161663568 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x5584ea723e30 0x5584ea726220 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.881+0000 7fe421996740 1 --2- 172.21.15.31:0/161663568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5584ea6a6c00 0x5584ea7222f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.881+0000 7fe405600640 0 client.27310 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:45:52.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.881+0000 7fe421996740 1 --2- 172.21.15.31:0/161663568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584ea69e080 0x5584ea6ab060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.881+0000 7fe421996740 1 --2- 172.21.15.31:0/161663568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584ea69d6d0 0x5584ea6aab20 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.881+0000 7fe421996740 1 -- 172.21.15.31:0/161663568 >> 172.21.15.31:0/161663568 conn(0x5584ea697da0 msgr2=0x5584ea696620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:52.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.883+0000 7fe421996740 1 -- 172.21.15.31:0/161663568 shutdown_connections 2024-03-21T10:45:52.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.884+0000 7fe421996740 1 -- 172.21.15.31:0/161663568 wait complete. 2024-03-21T10:45:52.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.884+0000 7fe421996740 20 client.27310 trim_cache size 0 max 16384 2024-03-21T10:45:52.883 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:45:52.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.895+0000 7fe41ec00640 1 -- 172.21.15.31:0/1356325718 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe3fc0a3260 msgr2=0x7fe3fc0a5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:52.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.895+0000 7fe41ec00640 1 --2- 172.21.15.31:0/1356325718 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe3fc0a3260 0x7fe3fc0a5720 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe4000058c0 tx=0x7fe4000046c0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.895+0000 7fe41ec00640 1 -- 172.21.15.31:0/1356325718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe418109850 msgr2=0x7fe41806c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:52.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.895+0000 7fe41ec00640 1 --2- 172.21.15.31:0/1356325718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe418109850 0x7fe41806c470 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe408037040 tx=0x7fe408004530 comp rx=0 tx=0).stop 2024-03-21T10:45:52.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.896+0000 7fe41ec00640 1 -- 172.21.15.31:0/1356325718 shutdown_connections 2024-03-21T10:45:52.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.896+0000 7fe41ec00640 1 --2- 172.21.15.31:0/1356325718 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe3fc0a3260 0x7fe3fc0a5720 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.896+0000 7fe41ec00640 1 --2- 172.21.15.31:0/1356325718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe418109850 0x7fe41806c470 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.896+0000 7fe41ec00640 1 --2- 172.21.15.31:0/1356325718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe418104200 0x7fe41806bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.896+0000 7fe41ec00640 1 --2- 172.21.15.31:0/1356325718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe418103840 0x7fe418072eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:52.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.896+0000 7fe41ec00640 1 -- 172.21.15.31:0/1356325718 >> 172.21.15.31:0/1356325718 conn(0x7fe4180772d0 msgr2=0x7fe4180778f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:52.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.898+0000 7fe41ec00640 1 -- 172.21.15.31:0/1356325718 shutdown_connections 2024-03-21T10:45:52.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:52.898+0000 7fe41ec00640 1 -- 172.21.15.31:0/1356325718 wait complete. 2024-03-21T10:45:52.946 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:45:52.946 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-21T10:45:53.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.427+0000 7f5efd000640 1 -- 172.21.15.31:0/1641037100 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ef8108760 msgr2=0x7f5ef8114c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:53.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.427+0000 7f5efd000640 1 --2- 172.21.15.31:0/1641037100 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ef8108760 0x7f5ef8114c30 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0009930 tx=0x7f5ee002bc60 comp rx=0 tx=0).stop 2024-03-21T10:45:53.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.430+0000 7f5efd000640 1 -- 172.21.15.31:0/1641037100 shutdown_connections 2024-03-21T10:45:53.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.430+0000 7f5efd000640 1 --2- 172.21.15.31:0/1641037100 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ef8108760 0x7f5ef8114c30 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.430+0000 7f5efd000640 1 --2- 172.21.15.31:0/1641037100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ef8107dc0 0x7f5ef8108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.430+0000 7f5efd000640 1 --2- 172.21.15.31:0/1641037100 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ef8109b70 0x7f5ef8109f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.431+0000 7f5efd000640 1 -- 172.21.15.31:0/1641037100 >> 172.21.15.31:0/1641037100 conn(0x7f5ef8077540 msgr2=0x7f5ef81064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:53.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.432+0000 7f5efd000640 1 -- 172.21.15.31:0/1641037100 shutdown_connections 2024-03-21T10:45:53.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.432+0000 7f5efd000640 1 -- 172.21.15.31:0/1641037100 wait complete. 2024-03-21T10:45:53.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.435+0000 7f5efd000640 1 Processor -- start 2024-03-21T10:45:53.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.435+0000 7f5efd000640 1 -- start start 2024-03-21T10:45:53.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.435+0000 7f5efd000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ef8107dc0 0x7f5ef8081c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:53.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.436+0000 7f5efd000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ef8108760 0x7f5ef8082170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:53.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.436+0000 7f5efd000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ef8109b70 0x7f5ef807ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:53.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.436+0000 7f5efd000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef8118230 con 0x7f5ef8107dc0 2024-03-21T10:45:53.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.436+0000 7f5efd000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef81180b0 con 0x7f5ef8109b70 2024-03-21T10:45:53.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.436+0000 7f5efd000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef81183b0 con 0x7f5ef8108760 2024-03-21T10:45:53.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.439+0000 7f5ef6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ef8109b70 0x7f5ef807ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:53.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.439+0000 7f5ef5400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ef8108760 0x7f5ef8082170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:53.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.439+0000 7f5ef6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ef8109b70 0x7f5ef807ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:35894/0 (socket says 172.21.15.31:35894) 2024-03-21T10:45:53.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.439+0000 7f5ef6800640 1 -- 172.21.15.31:0/2460739311 learned_addr learned my addr 172.21.15.31:0/2460739311 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:53.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.440+0000 7f5ef5400640 1 -- 172.21.15.31:0/2460739311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ef8109b70 msgr2=0x7f5ef807ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:53.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.440+0000 7f5ef5400640 1 --2- 172.21.15.31:0/2460739311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ef8109b70 0x7f5ef807ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.440+0000 7f5ef5400640 1 -- 172.21.15.31:0/2460739311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ef8107dc0 msgr2=0x7f5ef8081c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:53.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.440+0000 7f5ef5400640 1 --2- 172.21.15.31:0/2460739311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ef8107dc0 0x7f5ef8081c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.440+0000 7f5ef5400640 1 -- 172.21.15.31:0/2460739311 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef807b700 con 0x7f5ef8108760 2024-03-21T10:45:53.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.441+0000 7f5ef5400640 1 --2- 172.21.15.31:0/2460739311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ef8108760 0x7f5ef8082170 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8009fd0 tx=0x7f5ee800b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:53.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.442+0000 7f5eeea00640 1 -- 172.21.15.31:0/2460739311 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee8015020 con 0x7f5ef8108760 2024-03-21T10:45:53.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.442+0000 7f5eeea00640 1 -- 172.21.15.31:0/2460739311 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ee8011d10 con 0x7f5ef8108760 2024-03-21T10:45:53.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.442+0000 7f5eeea00640 1 -- 172.21.15.31:0/2460739311 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee800f2e0 con 0x7f5ef8108760 2024-03-21T10:45:53.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.442+0000 7f5efd000640 1 -- 172.21.15.31:0/2460739311 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef807b9f0 con 0x7f5ef8108760 2024-03-21T10:45:53.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.442+0000 7f5efd000640 1 -- 172.21.15.31:0/2460739311 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef807f990 con 0x7f5ef8108760 2024-03-21T10:45:53.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.448+0000 7f5effc15740 1 Processor -- start 2024-03-21T10:45:53.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.449+0000 7f5effc15740 1 -- start start 2024-03-21T10:45:53.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.450+0000 7f5effc15740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560065de36d0 0x560065de3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:53.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.450+0000 7f5effc15740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560065de4080 0x560065dec630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:53.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.450+0000 7f5effc15740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560065decc00 0x560065def050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:53.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.450+0000 7f5eeea00640 1 -- 172.21.15.31:0/2460739311 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f5ee8018020 con 0x7f5ef8108760 2024-03-21T10:45:53.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.450+0000 7f5effc15740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560065d472c0 con 0x560065decc00 2024-03-21T10:45:53.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.450+0000 7f5effc15740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560065cefc90 con 0x560065de36d0 2024-03-21T10:45:53.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.450+0000 7f5effc15740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x560065d43360 con 0x560065de4080 2024-03-21T10:45:53.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.451+0000 7f5ef6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560065de36d0 0x560065de3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:53.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.451+0000 7f5ef6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560065de36d0 0x560065de3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:35908/0 (socket says 172.21.15.31:35908) 2024-03-21T10:45:53.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.451+0000 7f5ef6800640 1 -- 172.21.15.31:0/3169525722 learned_addr learned my addr 172.21.15.31:0/3169525722 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:53.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.451+0000 7f5ef5e00640 1 --2- 172.21.15.31:0/3169525722 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560065de4080 0x560065dec630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:53.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.453+0000 7f5eeea00640 1 --2- 172.21.15.31:0/2460739311 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f5edc0a3290 0x7f5edc0a5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:53.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.453+0000 7f5ef6800640 1 -- 172.21.15.31:0/3169525722 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560065de4080 msgr2=0x560065dec630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:53.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.453+0000 7f5ef6800640 1 --2- 172.21.15.31:0/3169525722 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560065de4080 0x560065dec630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.453+0000 7f5ef6800640 1 -- 172.21.15.31:0/3169525722 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560065decc00 msgr2=0x560065def050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:53.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.453+0000 7f5ef6800640 1 --2- 172.21.15.31:0/3169525722 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560065decc00 0x560065def050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.453+0000 7f5ef6800640 1 -- 172.21.15.31:0/3169525722 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560065d14080 con 0x560065de36d0 2024-03-21T10:45:53.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.453+0000 7f5eeea00640 1 -- 172.21.15.31:0/2460739311 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f5ee80cf2f0 con 0x7f5ef8108760 2024-03-21T10:45:53.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.454+0000 7f5ef6800640 1 --2- 172.21.15.31:0/2460739311 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f5edc0a3290 0x7f5edc0a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:53.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.454+0000 7f5ef6800640 1 --2- 172.21.15.31:0/3169525722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560065de36d0 0x560065de3ab0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5ee002c710 tx=0x7f5ee0009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bb225b16aca80ef server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:53.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.455+0000 7f5ee7400640 1 -- 172.21.15.31:0/3169525722 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee00043e0 con 0x560065de36d0 2024-03-21T10:45:53.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.455+0000 7f5ee7400640 1 -- 172.21.15.31:0/3169525722 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ee00046c0 con 0x560065de36d0 2024-03-21T10:45:53.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.455+0000 7f5ee7400640 1 -- 172.21.15.31:0/3169525722 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee0005510 con 0x560065de36d0 2024-03-21T10:45:53.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.456+0000 7f5effc15740 1 -- 172.21.15.31:0/3169525722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560065de36d0 msgr2=0x560065de3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:53.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.456+0000 7f5effc15740 1 --2- 172.21.15.31:0/3169525722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560065de36d0 0x560065de3ab0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5ee002c710 tx=0x7f5ee0009680 comp rx=0 tx=0).stop 2024-03-21T10:45:53.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.456+0000 7f5ef6800640 1 --2- 172.21.15.31:0/2460739311 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f5edc0a3290 0x7f5edc0a5750 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0048000 tx=0x7f5ee00035f0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:53.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.456+0000 7f5effc15740 1 -- 172.21.15.31:0/3169525722 shutdown_connections 2024-03-21T10:45:53.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.456+0000 7f5effc15740 1 --2- 172.21.15.31:0/3169525722 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560065decc00 0x560065def050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.456+0000 7f5effc15740 1 --2- 172.21.15.31:0/3169525722 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560065de4080 0x560065dec630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.456+0000 7f5effc15740 1 --2- 172.21.15.31:0/3169525722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560065de36d0 0x560065de3ab0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.456+0000 7f5effc15740 1 -- 172.21.15.31:0/3169525722 >> 172.21.15.31:0/3169525722 conn(0x560065dddda0 msgr2=0x560065dde180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:53.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.458+0000 7f5effc15740 1 -- 172.21.15.31:0/3169525722 shutdown_connections 2024-03-21T10:45:53.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.458+0000 7f5effc15740 1 -- 172.21.15.31:0/3169525722 wait complete. 2024-03-21T10:45:53.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.459+0000 7f5effc15740 1 -- start start 2024-03-21T10:45:53.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.459+0000 7f5effc15740 1 Processor -- start 2024-03-21T10:45:53.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.460+0000 7f5effc15740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560065de36d0 0x560065dec5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:53.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.460+0000 7f5effc15740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560065de4080 0x560065de9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:53.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.460+0000 7f5effc15740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560065decc00 0x560065dea080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:53.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.460+0000 7f5effc15740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560065defb70 con 0x560065de4080 2024-03-21T10:45:53.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.460+0000 7f5effc15740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560065d030d0 con 0x560065de36d0 2024-03-21T10:45:53.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.460+0000 7f5effc15740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x560065defcf0 con 0x560065decc00 2024-03-21T10:45:53.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.460+0000 7f5effc15740 10 client.0 _ll_register_callbacks cb 0x560065d07e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:53.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.461+0000 7f5ef5400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560065de4080 0x560065de9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:53.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.461+0000 7f5ef5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560065de36d0 0x560065dec5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:53.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.462+0000 7f5ef6800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560065decc00 0x560065dea080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:53.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.462+0000 7f5ef6800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560065decc00 0x560065dea080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:50902/0 (socket says 172.21.15.31:50902) 2024-03-21T10:45:53.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.462+0000 7f5ef5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560065de36d0 0x560065dec5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:35912/0 (socket says 172.21.15.31:35912) 2024-03-21T10:45:53.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.462+0000 7f5ef6800640 1 -- 172.21.15.31:0/713850922 learned_addr learned my addr 172.21.15.31:0/713850922 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:53.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.462+0000 7f5ef6800640 1 -- 172.21.15.31:0/713850922 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560065de36d0 msgr2=0x560065dec5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:53.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.462+0000 7f5ef6800640 1 --2- 172.21.15.31:0/713850922 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560065de36d0 0x560065dec5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.462+0000 7f5ef6800640 1 -- 172.21.15.31:0/713850922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560065de4080 msgr2=0x560065de9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:53.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.462+0000 7f5ef6800640 1 --2- 172.21.15.31:0/713850922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560065de4080 0x560065de9940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.462+0000 7f5ef6800640 1 -- 172.21.15.31:0/713850922 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560065a5b3f0 con 0x560065decc00 2024-03-21T10:45:53.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.462+0000 7f5ef5400640 1 --2- 172.21.15.31:0/713850922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560065de4080 0x560065de9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:45:53.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.463+0000 7f5ef6800640 1 --2- 172.21.15.31:0/713850922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560065decc00 0x560065dea080 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f5ee00096c0 tx=0x7f5ee003f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c496549fc0c56ebe server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:53.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.463+0000 7f5edbe00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:45:53.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.463+0000 7f5edbe00640 1 -- 172.21.15.31:0/713850922 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee0062060 con 0x560065decc00 2024-03-21T10:45:53.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.463+0000 7f5edbe00640 1 -- 172.21.15.31:0/713850922 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f5ee0004380 con 0x560065decc00 2024-03-21T10:45:53.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.463+0000 7f5effc15740 1 -- 172.21.15.31:0/713850922 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5ee00046c0 con 0x560065decc00 2024-03-21T10:45:53.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.464+0000 7f5edbe00640 1 -- 172.21.15.31:0/713850922 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ee0069050 con 0x560065decc00 2024-03-21T10:45:53.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.464+0000 7f5effc15740 10 client.27322 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:53.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.464+0000 7f5effc15740 1 -- 172.21.15.31:0/713850922 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x560065b257d0 con 0x560065decc00 2024-03-21T10:45:53.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.464+0000 7f5edbe00640 1 -- 172.21.15.31:0/713850922 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ee00046c0 con 0x560065decc00 2024-03-21T10:45:53.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.465+0000 7f5edbe00640 1 -- 172.21.15.31:0/713850922 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f5ee002c8b0 con 0x560065decc00 2024-03-21T10:45:53.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.465+0000 7f5effc15740 10 client.27322 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:53.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.465+0000 7f5effc15740 10 client.27322 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:45:53.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.465+0000 7f5effc15740 20 client.27322 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:53.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.465+0000 7f5effc15740 1 --2- 172.21.15.31:0/713850922 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560065dfc990 0x560065dfed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:53.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.466+0000 7f5effc15740 4 client.27322 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:45:53.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.466+0000 7f5effc15740 1 -- 172.21.15.31:0/713850922 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560065d54800 con 0x560065dfc990 2024-03-21T10:45:53.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.466+0000 7f5ef5e00640 1 --2- 172.21.15.31:0/713850922 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560065dfc990 0x560065dfed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:53.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.467+0000 7f5edbe00640 10 client.27322 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:53.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.467+0000 7f5ef5e00640 1 --2- 172.21.15.31:0/713850922 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560065dfc990 0x560065dfed80 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0e88f1da4794046 server_cookie=70d34712ba72bed2 in_seq=0 out_seq=0 2024-03-21T10:45:53.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.468+0000 7f5edbe00640 1 -- 172.21.15.31:0/713850922 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560065d54800 con 0x560065dfc990 2024-03-21T10:45:53.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.468+0000 7f5edbe00640 10 client.27322 handle_command_reply: tid=0 2024-03-21T10:45:53.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.468+0000 7f5effc15740 1 client.27322 shutdown 2024-03-21T10:45:53.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.469+0000 7f5effc15740 1 -- 172.21.15.31:0/713850922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560065decc00 msgr2=0x560065dea080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:53.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.469+0000 7f5effc15740 1 --2- 172.21.15.31:0/713850922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560065decc00 0x560065dea080 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f5ee00096c0 tx=0x7f5ee003f000 comp rx=0 tx=0).stop 2024-03-21T10:45:53.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.469+0000 7f5effc15740 1 -- 172.21.15.31:0/713850922 shutdown_connections 2024-03-21T10:45:53.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.469+0000 7f5effc15740 1 --2- 172.21.15.31:0/713850922 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560065dfc990 0x560065dfed80 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.469+0000 7f5effc15740 1 --2- 172.21.15.31:0/713850922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560065decc00 0x560065dea080 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.469+0000 7f5effc15740 1 --2- 172.21.15.31:0/713850922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560065de4080 0x560065de9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.469+0000 7f5effc15740 1 --2- 172.21.15.31:0/713850922 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560065de36d0 0x560065dec5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.469+0000 7f5effc15740 1 -- 172.21.15.31:0/713850922 >> 172.21.15.31:0/713850922 conn(0x560065dddda0 msgr2=0x560065ddfaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:53.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.469+0000 7f5edbe00640 0 client.27322 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:53.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.470+0000 7f5effc15740 1 -- 172.21.15.31:0/713850922 shutdown_connections 2024-03-21T10:45:53.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.470+0000 7f5effc15740 1 -- 172.21.15.31:0/713850922 wait complete. 2024-03-21T10:45:53.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.470+0000 7f5effc15740 20 client.27322 trim_cache size 0 max 16384 2024-03-21T10:45:53.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.498+0000 7f5effc15740 1 Processor -- start 2024-03-21T10:45:53.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.498+0000 7f5effc15740 1 -- start start 2024-03-21T10:45:53.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.499+0000 7f5effc15740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560065de36d0 0x560065ddfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:53.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.500+0000 7f5effc15740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560065de4080 0x560065dec070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:53.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.500+0000 7f5effc15740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560065decc00 0x560065dec5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:53.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.500+0000 7f5effc15740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef81bfd50 con 0x560065de4080 2024-03-21T10:45:53.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.500+0000 7f5ef5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560065de36d0 0x560065ddfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:53.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.501+0000 7f5ef5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560065de36d0 0x560065ddfcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:35920/0 (socket says 172.21.15.31:35920) 2024-03-21T10:45:53.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.501+0000 7f5ef6800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560065decc00 0x560065dec5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:53.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.501+0000 7f5ef5e00640 1 -- 172.21.15.31:0/823344373 learned_addr learned my addr 172.21.15.31:0/823344373 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:53.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.501+0000 7f5effc15740 1 -- 172.21.15.31:0/823344373 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef81bd850 con 0x560065de36d0 2024-03-21T10:45:53.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.501+0000 7f5effc15740 1 -- 172.21.15.31:0/823344373 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef81bb910 con 0x560065decc00 2024-03-21T10:45:53.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.501+0000 7f5ef5400640 1 --2- 172.21.15.31:0/823344373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560065de4080 0x560065dec070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:53.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.501+0000 7f5ef6800640 1 -- 172.21.15.31:0/823344373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560065de36d0 msgr2=0x560065ddfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:53.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.501+0000 7f5ef6800640 1 --2- 172.21.15.31:0/823344373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560065de36d0 0x560065ddfcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.501+0000 7f5ef6800640 1 -- 172.21.15.31:0/823344373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560065de4080 msgr2=0x560065dec070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:53.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.501+0000 7f5ef6800640 1 --2- 172.21.15.31:0/823344373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560065de4080 0x560065dec070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.501+0000 7f5ef6800640 1 -- 172.21.15.31:0/823344373 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560065dff0f0 con 0x560065decc00 2024-03-21T10:45:53.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.502+0000 7f5ef6800640 1 --2- 172.21.15.31:0/823344373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560065decc00 0x560065dec5b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0003db0 tx=0x7f5ee0003840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e6c9f13a0cf01b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:53.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.502+0000 7f5ee6a00640 1 -- 172.21.15.31:0/823344373 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee0073070 con 0x560065decc00 2024-03-21T10:45:53.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.502+0000 7f5ee6a00640 1 -- 172.21.15.31:0/823344373 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ee0004410 con 0x560065decc00 2024-03-21T10:45:53.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.502+0000 7f5ee6a00640 1 -- 172.21.15.31:0/823344373 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee0062050 con 0x560065decc00 2024-03-21T10:45:53.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.503+0000 7f5effc15740 1 -- 172.21.15.31:0/823344373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560065decc00 msgr2=0x560065dec5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:53.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.503+0000 7f5effc15740 1 --2- 172.21.15.31:0/823344373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560065decc00 0x560065dec5b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0003db0 tx=0x7f5ee0003840 comp rx=0 tx=0).stop 2024-03-21T10:45:53.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.503+0000 7f5effc15740 1 -- 172.21.15.31:0/823344373 shutdown_connections 2024-03-21T10:45:53.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.503+0000 7f5effc15740 1 --2- 172.21.15.31:0/823344373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560065decc00 0x560065dec5b0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.503+0000 7f5effc15740 1 --2- 172.21.15.31:0/823344373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560065de4080 0x560065dec070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.503+0000 7f5effc15740 1 --2- 172.21.15.31:0/823344373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560065de36d0 0x560065ddfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.503+0000 7f5effc15740 1 -- 172.21.15.31:0/823344373 >> 172.21.15.31:0/823344373 conn(0x560065dddda0 msgr2=0x560065ddceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:53.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.503+0000 7f5effc15740 1 -- 172.21.15.31:0/823344373 shutdown_connections 2024-03-21T10:45:53.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.503+0000 7f5effc15740 1 -- 172.21.15.31:0/823344373 wait complete. 2024-03-21T10:45:53.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.503+0000 7f5effc15740 1 -- start start 2024-03-21T10:45:53.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.504+0000 7f5effc15740 1 Processor -- start 2024-03-21T10:45:53.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.504+0000 7f5effc15740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560065de36d0 0x560065df4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:53.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.504+0000 7f5effc15740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560065de4080 0x560065df2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:53.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.504+0000 7f5effc15740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560065decc00 0x560065df34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:53.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.504+0000 7f5effc15740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef81bbf10 con 0x560065decc00 2024-03-21T10:45:53.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.504+0000 7f5effc15740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef81bd1f0 con 0x560065de4080 2024-03-21T10:45:53.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.504+0000 7f5effc15740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef81bc530 con 0x560065de36d0 2024-03-21T10:45:53.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.504+0000 7f5effc15740 10 client.0 _ll_register_callbacks cb 0x7f5ee0056040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:53.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.504+0000 7f5ef5e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560065de36d0 0x560065df4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:53.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.504+0000 7f5ef5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560065de4080 0x560065df2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:53.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.504+0000 7f5ef6800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560065decc00 0x560065df34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:53.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.504+0000 7f5ef5e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560065de36d0 0x560065df4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:50932/0 (socket says 172.21.15.31:50932) 2024-03-21T10:45:53.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.504+0000 7f5ef5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560065de4080 0x560065df2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:35928/0 (socket says 172.21.15.31:35928) 2024-03-21T10:45:53.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.504+0000 7f5ef5e00640 1 -- 172.21.15.31:0/3708536705 learned_addr learned my addr 172.21.15.31:0/3708536705 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:53.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.505+0000 7f5ef5e00640 1 -- 172.21.15.31:0/3708536705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560065de4080 msgr2=0x560065df2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:53.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.505+0000 7f5ef5e00640 1 --2- 172.21.15.31:0/3708536705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560065de4080 0x560065df2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.505+0000 7f5ef5e00640 1 -- 172.21.15.31:0/3708536705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560065decc00 msgr2=0x560065df34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:53.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.505+0000 7f5ef5e00640 1 --2- 172.21.15.31:0/3708536705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560065decc00 0x560065df34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.505+0000 7f5ef5e00640 1 -- 172.21.15.31:0/3708536705 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560065df3b10 con 0x560065de36d0 2024-03-21T10:45:53.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.505+0000 7f5ef5e00640 1 --2- 172.21.15.31:0/3708536705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560065de36d0 0x560065df4b70 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x560065de31a0 tx=0x7f5ef000a560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8303d5c4d581aa3b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:53.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.505+0000 7f5edbe00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:45:53.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.506+0000 7f5edbe00640 1 -- 172.21.15.31:0/3708536705 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef000ad90 con 0x560065de36d0 2024-03-21T10:45:53.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.506+0000 7f5effc15740 1 -- 172.21.15.31:0/3708536705 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5ee0004410 con 0x560065de36d0 2024-03-21T10:45:53.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.506+0000 7f5edbe00640 1 -- 172.21.15.31:0/3708536705 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f5ef0024420 con 0x560065de36d0 2024-03-21T10:45:53.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.506+0000 7f5edbe00640 1 -- 172.21.15.31:0/3708536705 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ef0024660 con 0x560065de36d0 2024-03-21T10:45:53.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.506+0000 7f5effc15740 10 client.27328 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:53.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.506+0000 7f5effc15740 1 -- 172.21.15.31:0/3708536705 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x560065e58e90 con 0x560065de36d0 2024-03-21T10:45:53.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.506+0000 7f5edbe00640 1 -- 172.21.15.31:0/3708536705 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ee0004410 con 0x560065de36d0 2024-03-21T10:45:53.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.507+0000 7f5edbe00640 1 -- 172.21.15.31:0/3708536705 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f5ef0018960 con 0x560065de36d0 2024-03-21T10:45:53.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.507+0000 7f5effc15740 10 client.27328 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:53.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.507+0000 7f5effc15740 10 client.27328 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:45:53.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.507+0000 7f5effc15740 20 client.27328 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:53.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.507+0000 7f5effc15740 1 --2- 172.21.15.31:0/3708536705 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560065e595f0 0x560065e5ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:53.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.507+0000 7f5effc15740 4 client.27328 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:45:53.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.507+0000 7f5effc15740 1 -- 172.21.15.31:0/3708536705 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560065d4eb50 con 0x560065e595f0 2024-03-21T10:45:53.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.507+0000 7f5ef5400640 1 --2- 172.21.15.31:0/3708536705 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560065e595f0 0x560065e5ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:53.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.507+0000 7f5ef5400640 1 --2- 172.21.15.31:0/3708536705 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560065e595f0 0x560065e5ba10 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5da4dd2e6ed7e00 server_cookie=9c1cb88352bb0c30 in_seq=0 out_seq=0 2024-03-21T10:45:53.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.507+0000 7f5edbe00640 10 client.27328 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:53.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.508+0000 7f5edbe00640 1 -- 172.21.15.31:0/3708536705 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560065d4eb50 con 0x560065e595f0 2024-03-21T10:45:53.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.508+0000 7f5edbe00640 10 client.27328 handle_command_reply: tid=0 2024-03-21T10:45:53.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.508+0000 7f5effc15740 1 client.27328 shutdown 2024-03-21T10:45:53.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.508+0000 7f5effc15740 1 -- 172.21.15.31:0/3708536705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560065de36d0 msgr2=0x560065df4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:53.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.508+0000 7f5effc15740 1 --2- 172.21.15.31:0/3708536705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560065de36d0 0x560065df4b70 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x560065de31a0 tx=0x7f5ef000a560 comp rx=0 tx=0).stop 2024-03-21T10:45:53.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.508+0000 7f5effc15740 1 -- 172.21.15.31:0/3708536705 shutdown_connections 2024-03-21T10:45:53.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.508+0000 7f5effc15740 1 --2- 172.21.15.31:0/3708536705 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560065e595f0 0x560065e5ba10 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.508+0000 7f5effc15740 1 --2- 172.21.15.31:0/3708536705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560065decc00 0x560065df34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.508+0000 7f5effc15740 1 --2- 172.21.15.31:0/3708536705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560065de4080 0x560065df2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.508+0000 7f5effc15740 1 --2- 172.21.15.31:0/3708536705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560065de36d0 0x560065df4b70 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.508+0000 7f5edbe00640 0 client.27328 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:53.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.508+0000 7f5effc15740 1 -- 172.21.15.31:0/3708536705 >> 172.21.15.31:0/3708536705 conn(0x560065dddda0 msgr2=0x560065de44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:53.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.508+0000 7f5effc15740 1 -- 172.21.15.31:0/3708536705 shutdown_connections 2024-03-21T10:45:53.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.508+0000 7f5effc15740 1 -- 172.21.15.31:0/3708536705 wait complete. 2024-03-21T10:45:53.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.509+0000 7f5effc15740 20 client.27328 trim_cache size 0 max 16384 2024-03-21T10:45:53.508 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:45:53.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.509+0000 7f5efd000640 1 -- 172.21.15.31:0/2460739311 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f5edc0a3290 msgr2=0x7f5edc0a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:53.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.509+0000 7f5efd000640 1 --2- 172.21.15.31:0/2460739311 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f5edc0a3290 0x7f5edc0a5750 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0048000 tx=0x7f5ee00035f0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.509+0000 7f5efd000640 1 -- 172.21.15.31:0/2460739311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ef8108760 msgr2=0x7f5ef8082170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:53.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.509+0000 7f5efd000640 1 --2- 172.21.15.31:0/2460739311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ef8108760 0x7f5ef8082170 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8009fd0 tx=0x7f5ee800b870 comp rx=0 tx=0).stop 2024-03-21T10:45:53.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.509+0000 7f5efd000640 1 -- 172.21.15.31:0/2460739311 shutdown_connections 2024-03-21T10:45:53.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.509+0000 7f5efd000640 1 --2- 172.21.15.31:0/2460739311 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f5edc0a3290 0x7f5edc0a5750 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.509+0000 7f5efd000640 1 --2- 172.21.15.31:0/2460739311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ef8109b70 0x7f5ef807ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.509+0000 7f5efd000640 1 --2- 172.21.15.31:0/2460739311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ef8108760 0x7f5ef8082170 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.509+0000 7f5efd000640 1 --2- 172.21.15.31:0/2460739311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ef8107dc0 0x7f5ef8081c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:53.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.510+0000 7f5efd000640 1 -- 172.21.15.31:0/2460739311 >> 172.21.15.31:0/2460739311 conn(0x7f5ef8077540 msgr2=0x7f5ef8077a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:53.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.510+0000 7f5efd000640 1 -- 172.21.15.31:0/2460739311 shutdown_connections 2024-03-21T10:45:53.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:53.510+0000 7f5efd000640 1 -- 172.21.15.31:0/2460739311 wait complete. 2024-03-21T10:45:53.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:53 smithi040 ceph-mon[36154]: pgmap v224: 129 pgs: 129 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 181 MiB/s wr, 1.22k op/s 2024-03-21T10:45:53.807 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:53 smithi031 ceph-mon[31673]: pgmap v224: 129 pgs: 129 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 181 MiB/s wr, 1.22k op/s 2024-03-21T10:45:53.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:53 smithi070 ceph-mon[36330]: pgmap v224: 129 pgs: 129 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 181 MiB/s wr, 1.22k op/s 2024-03-21T10:45:54.524 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-21T10:45:54.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:45:54.525 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-21T10:45:54.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.867+0000 7f6104c00640 1 -- 172.21.15.31:0/2180160867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6100101640 msgr2=0x7f6100101ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:54.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.867+0000 7f6104c00640 1 --2- 172.21.15.31:0/2180160867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6100101640 0x7f6100101ac0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f60ec009890 tx=0x7f60ec02bb70 comp rx=0 tx=0).stop 2024-03-21T10:45:54.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.867+0000 7f6104c00640 1 -- 172.21.15.31:0/2180160867 shutdown_connections 2024-03-21T10:45:54.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.867+0000 7f6104c00640 1 --2- 172.21.15.31:0/2180160867 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6100102000 0x7f610010e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.867+0000 7f6104c00640 1 --2- 172.21.15.31:0/2180160867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6100101640 0x7f6100101ac0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.867+0000 7f6104c00640 1 --2- 172.21.15.31:0/2180160867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6100107650 0x7f6100107a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.867+0000 7f6104c00640 1 -- 172.21.15.31:0/2180160867 >> 172.21.15.31:0/2180160867 conn(0x7f61000fd130 msgr2=0x7f61000ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:54.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.868+0000 7f6104c00640 1 -- 172.21.15.31:0/2180160867 shutdown_connections 2024-03-21T10:45:54.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.868+0000 7f6104c00640 1 -- 172.21.15.31:0/2180160867 wait complete. 2024-03-21T10:45:54.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.869+0000 7f6104c00640 1 Processor -- start 2024-03-21T10:45:54.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.870+0000 7f6104c00640 1 -- start start 2024-03-21T10:45:54.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.871+0000 7f6104c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6100101640 0x7f610019e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:54.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.871+0000 7f6104c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6100102000 0x7f610019eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:54.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.871+0000 7f6104c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6100107650 0x7f61001a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:54.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.871+0000 7f6104c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f610010f850 con 0x7f6100102000 2024-03-21T10:45:54.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.871+0000 7f6104c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f610010f6d0 con 0x7f6100107650 2024-03-21T10:45:54.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.871+0000 7f6104c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f610010f9d0 con 0x7f6100101640 2024-03-21T10:45:54.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.872+0000 7f60fd400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6100102000 0x7f610019eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:54.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.872+0000 7f60fd400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6100102000 0x7f610019eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44810/0 (socket says 172.21.15.31:44810) 2024-03-21T10:45:54.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.872+0000 7f60fd400640 1 -- 172.21.15.31:0/3845036754 learned_addr learned my addr 172.21.15.31:0/3845036754 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:54.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.873+0000 7f60fd400640 1 -- 172.21.15.31:0/3845036754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6100101640 msgr2=0x7f610019e560 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-21T10:45:54.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.873+0000 7f60fd400640 1 --2- 172.21.15.31:0/3845036754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6100101640 0x7f610019e560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.873+0000 7f60fe800640 1 --2- 172.21.15.31:0/3845036754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6100107650 0x7f61001a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:54.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.873+0000 7f60fd400640 1 -- 172.21.15.31:0/3845036754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6100107650 msgr2=0x7f61001a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:54.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.873+0000 7f60fd400640 1 --2- 172.21.15.31:0/3845036754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6100107650 0x7f61001a2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.873+0000 7f60fd400640 1 -- 172.21.15.31:0/3845036754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61001a3550 con 0x7f6100102000 2024-03-21T10:45:54.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.873+0000 7f60fe800640 1 --2- 172.21.15.31:0/3845036754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6100107650 0x7f61001a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:45:54.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.874+0000 7f60fd400640 1 --2- 172.21.15.31:0/3845036754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6100102000 0x7f610019eaa0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f60ec0043c0 tx=0x7f60ec0043f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:54.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.875+0000 7f60f6a00640 1 -- 172.21.15.31:0/3845036754 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ec041020 con 0x7f6100102000 2024-03-21T10:45:54.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.875+0000 7f60f6a00640 1 -- 172.21.15.31:0/3845036754 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60ec02c9f0 con 0x7f6100102000 2024-03-21T10:45:54.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.875+0000 7f60f6a00640 1 -- 172.21.15.31:0/3845036754 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ec039040 con 0x7f6100102000 2024-03-21T10:45:54.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.875+0000 7f6104c00640 1 -- 172.21.15.31:0/3845036754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61001a37e0 con 0x7f6100102000 2024-03-21T10:45:54.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.876+0000 7f6104c00640 1 -- 172.21.15.31:0/3845036754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6100195aa0 con 0x7f6100102000 2024-03-21T10:45:54.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.885+0000 7f610783a740 1 Processor -- start 2024-03-21T10:45:54.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.887+0000 7f610783a740 1 -- start start 2024-03-21T10:45:54.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.887+0000 7f60f6a00640 1 -- 172.21.15.31:0/3845036754 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f60ec044020 con 0x7f6100102000 2024-03-21T10:45:54.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.888+0000 7f610783a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619a07bd080 0x5619a07bd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:54.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.888+0000 7f610783a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619a07bd9a0 0x5619a07c6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:54.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.888+0000 7f610783a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5619a07c6580 0x5619a07c8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:54.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.888+0000 7f610783a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619a0721eb0 con 0x5619a07bd080 2024-03-21T10:45:54.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.888+0000 7f610783a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619a06caa60 con 0x5619a07bd9a0 2024-03-21T10:45:54.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.888+0000 7f610783a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619a071df50 con 0x5619a07c6580 2024-03-21T10:45:54.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.888+0000 7f60fe800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619a07bd080 0x5619a07bd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:54.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.889+0000 7f60fe800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619a07bd080 0x5619a07bd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44812/0 (socket says 172.21.15.31:44812) 2024-03-21T10:45:54.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.889+0000 7f60fe800640 1 -- 172.21.15.31:0/3667652473 learned_addr learned my addr 172.21.15.31:0/3667652473 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:54.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.889+0000 7f60fd400640 1 --2- 172.21.15.31:0/3667652473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5619a07c6580 0x5619a07c8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:54.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.889+0000 7f60fde00640 1 --2- 172.21.15.31:0/3667652473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619a07bd9a0 0x5619a07c6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:54.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.890+0000 7f60fd400640 1 -- 172.21.15.31:0/3667652473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619a07bd9a0 msgr2=0x5619a07c6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:54.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.890+0000 7f60fd400640 1 --2- 172.21.15.31:0/3667652473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619a07bd9a0 0x5619a07c6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.890+0000 7f60fd400640 1 -- 172.21.15.31:0/3667652473 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619a07bd080 msgr2=0x5619a07bd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:54.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.890+0000 7f60fd400640 1 --2- 172.21.15.31:0/3667652473 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619a07bd080 0x5619a07bd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.890+0000 7f60fd400640 1 -- 172.21.15.31:0/3667652473 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5619a04fe7e0 con 0x5619a07c6580 2024-03-21T10:45:54.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.890+0000 7f60fde00640 1 --2- 172.21.15.31:0/3667652473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619a07bd9a0 0x5619a07c6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:45:54.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.890+0000 7f60fd400640 1 --2- 172.21.15.31:0/3667652473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5619a07c6580 0x5619a07c8970 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f60ec0ee900 tx=0x7f60ec0f7550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6b5bcd03029bc51 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:54.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.891+0000 7f60f6a00640 1 --2- 172.21.15.31:0/3845036754 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f60e40a31d0 0x7f60e40a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:54.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.891+0000 7f60fe800640 1 --2- 172.21.15.31:0/3845036754 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f60e40a31d0 0x7f60e40a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:54.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.891+0000 7f60f6a00640 1 -- 172.21.15.31:0/3845036754 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f60ec0f1440 con 0x7f6100102000 2024-03-21T10:45:54.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.892+0000 7f60eb000640 1 -- 172.21.15.31:0/3667652473 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ec0b9840 con 0x5619a07c6580 2024-03-21T10:45:54.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.892+0000 7f60eb000640 1 -- 172.21.15.31:0/3667652473 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60ec0fe030 con 0x5619a07c6580 2024-03-21T10:45:54.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.892+0000 7f60eb000640 1 -- 172.21.15.31:0/3667652473 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ec0bcf00 con 0x5619a07c6580 2024-03-21T10:45:54.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.893+0000 7f610783a740 1 -- 172.21.15.31:0/3667652473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5619a07c6580 msgr2=0x5619a07c8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:54.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.893+0000 7f610783a740 1 --2- 172.21.15.31:0/3667652473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5619a07c6580 0x5619a07c8970 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f60ec0ee900 tx=0x7f60ec0f7550 comp rx=0 tx=0).stop 2024-03-21T10:45:54.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.893+0000 7f60fe800640 1 --2- 172.21.15.31:0/3845036754 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f60e40a31d0 0x7f60e40a5690 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f60f80098d0 tx=0x7f60f80095c0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:54.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.893+0000 7f610783a740 1 -- 172.21.15.31:0/3667652473 shutdown_connections 2024-03-21T10:45:54.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.893+0000 7f610783a740 1 --2- 172.21.15.31:0/3667652473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5619a07c6580 0x5619a07c8970 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.893+0000 7f610783a740 1 --2- 172.21.15.31:0/3667652473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619a07bd9a0 0x5619a07c6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.893+0000 7f610783a740 1 --2- 172.21.15.31:0/3667652473 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619a07bd080 0x5619a07bd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.893+0000 7f610783a740 1 -- 172.21.15.31:0/3667652473 >> 172.21.15.31:0/3667652473 conn(0x5619a07b77b0 msgr2=0x5619a07b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:54.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.893+0000 7f610783a740 1 -- 172.21.15.31:0/3667652473 shutdown_connections 2024-03-21T10:45:54.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.893+0000 7f610783a740 1 -- 172.21.15.31:0/3667652473 wait complete. 2024-03-21T10:45:54.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.894+0000 7f610783a740 1 -- start start 2024-03-21T10:45:54.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.894+0000 7f610783a740 1 Processor -- start 2024-03-21T10:45:54.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.895+0000 7f610783a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619a07bd080 0x5619a07c5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:54.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.895+0000 7f610783a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619a07bd9a0 0x5619a07c3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:54.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.895+0000 7f610783a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5619a07c6580 0x5619a07c3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:54.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.895+0000 7f610783a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619a07c9130 con 0x5619a07bd9a0 2024-03-21T10:45:54.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.895+0000 7f610783a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619a06ddeb0 con 0x5619a07bd080 2024-03-21T10:45:54.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.895+0000 7f610783a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619a07c92b0 con 0x5619a07c6580 2024-03-21T10:45:54.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.895+0000 7f610783a740 10 client.0 _ll_register_callbacks cb 0x5619a06d2150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:54.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.896+0000 7f60fde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619a07bd080 0x5619a07c5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:54.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.896+0000 7f60fde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619a07bd080 0x5619a07c5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:35966/0 (socket says 172.21.15.31:35966) 2024-03-21T10:45:54.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.896+0000 7f60fde00640 1 -- 172.21.15.31:0/1320928815 learned_addr learned my addr 172.21.15.31:0/1320928815 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:54.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.896+0000 7f60fe800640 1 --2- 172.21.15.31:0/1320928815 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5619a07c6580 0x5619a07c3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:54.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.896+0000 7f60fd400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619a07bd9a0 0x5619a07c3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:54.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.896+0000 7f60fde00640 1 -- 172.21.15.31:0/1320928815 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5619a07c6580 msgr2=0x5619a07c3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:54.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.896+0000 7f60fde00640 1 --2- 172.21.15.31:0/1320928815 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5619a07c6580 0x5619a07c3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.896+0000 7f60fde00640 1 -- 172.21.15.31:0/1320928815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619a07bd9a0 msgr2=0x5619a07c3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:54.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.896+0000 7f60fde00640 1 --2- 172.21.15.31:0/1320928815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619a07bd9a0 0x5619a07c3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.896+0000 7f60fde00640 1 -- 172.21.15.31:0/1320928815 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5619a055ef70 con 0x5619a07bd080 2024-03-21T10:45:54.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.897+0000 7f60fd400640 1 --2- 172.21.15.31:0/1320928815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619a07bd9a0 0x5619a07c3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:45:54.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.897+0000 7f60fde00640 1 --2- 172.21.15.31:0/1320928815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619a07bd080 0x5619a07c5ec0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f60f00038e0 tx=0x7f60f0003da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ad1a2e966f8edef server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:54.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.897+0000 7f60df400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:45:54.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.897+0000 7f60df400640 1 -- 172.21.15.31:0/1320928815 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f001a470 con 0x5619a07bd080 2024-03-21T10:45:54.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.897+0000 7f60df400640 1 -- 172.21.15.31:0/1320928815 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f60f0011440 con 0x5619a07bd080 2024-03-21T10:45:54.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.897+0000 7f610783a740 1 -- 172.21.15.31:0/1320928815 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f60ec0fe030 con 0x5619a07bd080 2024-03-21T10:45:54.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.898+0000 7f60df400640 1 -- 172.21.15.31:0/1320928815 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60f0011640 con 0x5619a07bd080 2024-03-21T10:45:54.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.898+0000 7f610783a740 10 client.27422 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:54.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.898+0000 7f610783a740 1 -- 172.21.15.31:0/1320928815 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5619a03f6170 con 0x5619a07bd080 2024-03-21T10:45:54.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.898+0000 7f60df400640 1 -- 172.21.15.31:0/1320928815 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60ec0fe030 con 0x5619a07bd080 2024-03-21T10:45:54.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.899+0000 7f60df400640 1 -- 172.21.15.31:0/1320928815 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f60f001ab90 con 0x5619a07bd080 2024-03-21T10:45:54.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.899+0000 7f610783a740 10 client.27422 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:54.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.899+0000 7f610783a740 10 client.27422 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:45:54.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.899+0000 7f610783a740 20 client.27422 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:54.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.899+0000 7f610783a740 1 --2- 172.21.15.31:0/1320928815 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5619a07d36e0 0x5619a07d5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:54.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.899+0000 7f610783a740 4 client.27422 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:45:54.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.899+0000 7f610783a740 1 -- 172.21.15.31:0/1320928815 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5619a072f3f0 con 0x5619a07d36e0 2024-03-21T10:45:54.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.899+0000 7f60fd400640 1 --2- 172.21.15.31:0/1320928815 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5619a07d36e0 0x5619a07d5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:54.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.900+0000 7f60fd400640 1 --2- 172.21.15.31:0/1320928815 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5619a07d36e0 0x5619a07d5b60 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=646f0643d84e0946 server_cookie=f811a640bd31e76 in_seq=0 out_seq=0 2024-03-21T10:45:54.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.900+0000 7f60df400640 10 client.27422 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:54.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.901+0000 7f60df400640 1 -- 172.21.15.31:0/1320928815 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5619a072f3f0 con 0x5619a07d36e0 2024-03-21T10:45:54.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.901+0000 7f60df400640 10 client.27422 handle_command_reply: tid=0 2024-03-21T10:45:54.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.901+0000 7f610783a740 1 client.27422 shutdown 2024-03-21T10:45:54.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.901+0000 7f610783a740 1 -- 172.21.15.31:0/1320928815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619a07bd080 msgr2=0x5619a07c5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:54.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.901+0000 7f610783a740 1 --2- 172.21.15.31:0/1320928815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619a07bd080 0x5619a07c5ec0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f60f00038e0 tx=0x7f60f0003da0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.901+0000 7f610783a740 1 -- 172.21.15.31:0/1320928815 shutdown_connections 2024-03-21T10:45:54.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.901+0000 7f610783a740 1 --2- 172.21.15.31:0/1320928815 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5619a07d36e0 0x5619a07d5b60 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.901+0000 7f610783a740 1 --2- 172.21.15.31:0/1320928815 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5619a07c6580 0x5619a07c3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.901+0000 7f610783a740 1 --2- 172.21.15.31:0/1320928815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619a07bd9a0 0x5619a07c3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.901+0000 7f610783a740 1 --2- 172.21.15.31:0/1320928815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619a07bd080 0x5619a07c5ec0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.901+0000 7f60df400640 0 client.27422 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:54.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.901+0000 7f610783a740 1 -- 172.21.15.31:0/1320928815 >> 172.21.15.31:0/1320928815 conn(0x5619a07b77b0 msgr2=0x5619a07b9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:54.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.901+0000 7f610783a740 1 -- 172.21.15.31:0/1320928815 shutdown_connections 2024-03-21T10:45:54.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.902+0000 7f610783a740 1 -- 172.21.15.31:0/1320928815 wait complete. 2024-03-21T10:45:54.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.902+0000 7f610783a740 20 client.27422 trim_cache size 0 max 16384 2024-03-21T10:45:54.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.947+0000 7f610783a740 1 Processor -- start 2024-03-21T10:45:54.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.948+0000 7f610783a740 1 -- start start 2024-03-21T10:45:54.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.948+0000 7f610783a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619a07bd080 0x5619a07f3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:54.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.948+0000 7f610783a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5619a07bd9a0 0x5619a07c5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:54.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.948+0000 7f610783a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619a07c6580 0x5619a07c5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:54.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.948+0000 7f610783a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61001aed20 con 0x5619a07bd080 2024-03-21T10:45:54.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.948+0000 7f610783a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61001ac820 con 0x5619a07c6580 2024-03-21T10:45:54.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.948+0000 7f610783a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f610019dac0 con 0x5619a07bd9a0 2024-03-21T10:45:54.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.949+0000 7f60fe800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619a07c6580 0x5619a07c5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:54.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.949+0000 7f60fd400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5619a07bd9a0 0x5619a07c5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:54.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.949+0000 7f60fd400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5619a07bd9a0 0x5619a07c5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:50976/0 (socket says 172.21.15.31:50976) 2024-03-21T10:45:54.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.949+0000 7f60fd400640 1 -- 172.21.15.31:0/2788991182 learned_addr learned my addr 172.21.15.31:0/2788991182 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:54.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.950+0000 7f60fd400640 1 -- 172.21.15.31:0/2788991182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619a07c6580 msgr2=0x5619a07c5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:54.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.950+0000 7f60fd400640 1 --2- 172.21.15.31:0/2788991182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619a07c6580 0x5619a07c5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.950+0000 7f60fd400640 1 -- 172.21.15.31:0/2788991182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619a07bd080 msgr2=0x5619a07f3e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:45:54.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.950+0000 7f60fd400640 1 --2- 172.21.15.31:0/2788991182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619a07bd080 0x5619a07f3e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.950+0000 7f60fd400640 1 -- 172.21.15.31:0/2788991182 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5619a07c2200 con 0x5619a07bd9a0 2024-03-21T10:45:54.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.951+0000 7f60fd400640 1 --2- 172.21.15.31:0/2788991182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5619a07bd9a0 0x5619a07c5980 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f60ec0eedf0 tx=0x7f60ec005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5b62ce89fa6abe0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:54.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.951+0000 7f60ea600640 1 -- 172.21.15.31:0/2788991182 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ec0bd0e0 con 0x5619a07bd9a0 2024-03-21T10:45:54.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.951+0000 7f60ea600640 1 -- 172.21.15.31:0/2788991182 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60ec0fe040 con 0x5619a07bd9a0 2024-03-21T10:45:54.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.952+0000 7f60ea600640 1 -- 172.21.15.31:0/2788991182 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ec0bd0e0 con 0x5619a07bd9a0 2024-03-21T10:45:54.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.954+0000 7f610783a740 1 -- 172.21.15.31:0/2788991182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5619a07bd9a0 msgr2=0x5619a07c5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:54.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.954+0000 7f610783a740 1 --2- 172.21.15.31:0/2788991182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5619a07bd9a0 0x5619a07c5980 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f60ec0eedf0 tx=0x7f60ec005f90 comp rx=0 tx=0).stop 2024-03-21T10:45:54.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.954+0000 7f610783a740 1 -- 172.21.15.31:0/2788991182 shutdown_connections 2024-03-21T10:45:54.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.954+0000 7f610783a740 1 --2- 172.21.15.31:0/2788991182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619a07c6580 0x5619a07c5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.954+0000 7f610783a740 1 --2- 172.21.15.31:0/2788991182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5619a07bd9a0 0x5619a07c5980 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.954+0000 7f610783a740 1 --2- 172.21.15.31:0/2788991182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619a07bd080 0x5619a07f3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.954+0000 7f610783a740 1 -- 172.21.15.31:0/2788991182 >> 172.21.15.31:0/2788991182 conn(0x5619a07b77b0 msgr2=0x5619a07b6830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:54.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.958+0000 7f610783a740 1 -- 172.21.15.31:0/2788991182 shutdown_connections 2024-03-21T10:45:54.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.958+0000 7f610783a740 1 -- 172.21.15.31:0/2788991182 wait complete. 2024-03-21T10:45:54.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.959+0000 7f610783a740 1 -- start start 2024-03-21T10:45:54.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.959+0000 7f610783a740 1 Processor -- start 2024-03-21T10:45:54.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.965+0000 7f610783a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5619a07bd9a0 0x5619a07c05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:54.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.966+0000 7f610783a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619a07c6580 0x5619a07c0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:54.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.966+0000 7f610783a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619a07bf4c0 0x5619a07bf940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:54.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.967+0000 7f610783a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f610019e0c0 con 0x5619a07c6580 2024-03-21T10:45:54.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.967+0000 7f610783a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61001ac1c0 con 0x5619a07bf4c0 2024-03-21T10:45:54.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.967+0000 7f610783a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61001ab500 con 0x5619a07bd9a0 2024-03-21T10:45:54.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.967+0000 7f610783a740 10 client.0 _ll_register_callbacks cb 0x7f60f00041a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:54.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.967+0000 7f60fde00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5619a07bd9a0 0x5619a07c05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:54.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.967+0000 7f60fde00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5619a07bd9a0 0x5619a07c05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:50988/0 (socket says 172.21.15.31:50988) 2024-03-21T10:45:54.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.967+0000 7f60fde00640 1 -- 172.21.15.31:0/2877921697 learned_addr learned my addr 172.21.15.31:0/2877921697 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:54.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.968+0000 7f60fe800640 1 --2- 172.21.15.31:0/2877921697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619a07bf4c0 0x5619a07bf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:54.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.968+0000 7f60fd400640 1 --2- 172.21.15.31:0/2877921697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619a07c6580 0x5619a07c0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:54.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.969+0000 7f60fde00640 1 -- 172.21.15.31:0/2877921697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619a07bf4c0 msgr2=0x5619a07bf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:54.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.969+0000 7f60fde00640 1 --2- 172.21.15.31:0/2877921697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619a07bf4c0 0x5619a07bf940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.969+0000 7f60fde00640 1 -- 172.21.15.31:0/2877921697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619a07c6580 msgr2=0x5619a07c0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:54.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.969+0000 7f60fde00640 1 --2- 172.21.15.31:0/2877921697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619a07c6580 0x5619a07c0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.970+0000 7f60fde00640 1 -- 172.21.15.31:0/2877921697 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5619a07bfe80 con 0x5619a07bd9a0 2024-03-21T10:45:54.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.973+0000 7f60fde00640 1 --2- 172.21.15.31:0/2877921697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5619a07bd9a0 0x5619a07c05e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f60f0010490 tx=0x7f60f0005700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=147865e506f1f9d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:54.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.973+0000 7f60df400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:45:54.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.974+0000 7f60df400640 1 -- 172.21.15.31:0/2877921697 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f00119f0 con 0x5619a07bd9a0 2024-03-21T10:45:54.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.974+0000 7f60df400640 1 -- 172.21.15.31:0/2877921697 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f60f0011b90 con 0x5619a07bd9a0 2024-03-21T10:45:54.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.974+0000 7f610783a740 1 -- 172.21.15.31:0/2877921697 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f60ec0fe040 con 0x5619a07bd9a0 2024-03-21T10:45:54.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.975+0000 7f60df400640 1 -- 172.21.15.31:0/2877921697 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60f0011db0 con 0x5619a07bd9a0 2024-03-21T10:45:54.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.975+0000 7f610783a740 10 client.27343 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:54.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.975+0000 7f610783a740 1 -- 172.21.15.31:0/2877921697 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5619a0877010 con 0x5619a07bd9a0 2024-03-21T10:45:54.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.976+0000 7f60df400640 1 -- 172.21.15.31:0/2877921697 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60ec0fe040 con 0x5619a07bd9a0 2024-03-21T10:45:54.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.981+0000 7f60df400640 1 -- 172.21.15.31:0/2877921697 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f60f00106d0 con 0x5619a07bd9a0 2024-03-21T10:45:54.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.981+0000 7f610783a740 10 client.27343 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:54.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.981+0000 7f610783a740 10 client.27343 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:45:54.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.981+0000 7f610783a740 20 client.27343 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:54.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.981+0000 7f610783a740 1 --2- 172.21.15.31:0/2877921697 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5619a08776e0 0x5619a0879b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:54.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.981+0000 7f610783a740 4 client.27343 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-21T10:45:54.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.981+0000 7f610783a740 1 -- 172.21.15.31:0/2877921697 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5619a0729740 con 0x5619a08776e0 2024-03-21T10:45:54.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.987+0000 7f60fd400640 1 --2- 172.21.15.31:0/2877921697 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5619a08776e0 0x5619a0879b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:54.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.987+0000 7f60df400640 10 client.27343 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:54.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.988+0000 7f60fd400640 1 --2- 172.21.15.31:0/2877921697 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5619a08776e0 0x5619a0879b00 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a868375a0275c7f3 server_cookie=fbb97877ac460f08 in_seq=0 out_seq=0 2024-03-21T10:45:54.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.989+0000 7f60df400640 1 -- 172.21.15.31:0/2877921697 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5619a0729740 con 0x5619a08776e0 2024-03-21T10:45:54.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.989+0000 7f60df400640 10 client.27343 handle_command_reply: tid=0 2024-03-21T10:45:54.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.989+0000 7f610783a740 1 client.27343 shutdown 2024-03-21T10:45:54.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.989+0000 7f610783a740 1 -- 172.21.15.31:0/2877921697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5619a07bd9a0 msgr2=0x5619a07c05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:54.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.989+0000 7f610783a740 1 --2- 172.21.15.31:0/2877921697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5619a07bd9a0 0x5619a07c05e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f60f0010490 tx=0x7f60f0005700 comp rx=0 tx=0).stop 2024-03-21T10:45:54.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.990+0000 7f610783a740 1 -- 172.21.15.31:0/2877921697 shutdown_connections 2024-03-21T10:45:54.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.990+0000 7f610783a740 1 --2- 172.21.15.31:0/2877921697 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5619a08776e0 0x5619a0879b00 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.990+0000 7f60df400640 0 client.27343 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:54.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.990+0000 7f610783a740 1 --2- 172.21.15.31:0/2877921697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5619a07bf4c0 0x5619a07bf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.990+0000 7f610783a740 1 --2- 172.21.15.31:0/2877921697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619a07c6580 0x5619a07c0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.990+0000 7f610783a740 1 --2- 172.21.15.31:0/2877921697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5619a07bd9a0 0x5619a07c05e0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.990+0000 7f610783a740 1 -- 172.21.15.31:0/2877921697 >> 172.21.15.31:0/2877921697 conn(0x5619a07b77b0 msgr2=0x5619a07b5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:54.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.990+0000 7f610783a740 1 -- 172.21.15.31:0/2877921697 shutdown_connections 2024-03-21T10:45:54.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.991+0000 7f610783a740 1 -- 172.21.15.31:0/2877921697 wait complete. 2024-03-21T10:45:54.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.991+0000 7f610783a740 20 client.27343 trim_cache size 0 max 16384 2024-03-21T10:45:54.988 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:45:54.988 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-03-21T10:45:54.989 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "e262e02a-3a00-4aa6-af0a-8e505443abfe", 2024-03-21T10:45:54.989 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-03-21T10:45:54.989 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:45:54.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.992+0000 7f6104c00640 1 -- 172.21.15.31:0/3845036754 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f60e40a31d0 msgr2=0x7f60e40a5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:54.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.992+0000 7f6104c00640 1 --2- 172.21.15.31:0/3845036754 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f60e40a31d0 0x7f60e40a5690 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f60f80098d0 tx=0x7f60f80095c0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.992+0000 7f6104c00640 1 -- 172.21.15.31:0/3845036754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6100102000 msgr2=0x7f610019eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:54.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.992+0000 7f6104c00640 1 --2- 172.21.15.31:0/3845036754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6100102000 0x7f610019eaa0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f60ec0043c0 tx=0x7f60ec0043f0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.992+0000 7f6104c00640 1 -- 172.21.15.31:0/3845036754 shutdown_connections 2024-03-21T10:45:54.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.992+0000 7f6104c00640 1 --2- 172.21.15.31:0/3845036754 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f60e40a31d0 0x7f60e40a5690 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.992+0000 7f6104c00640 1 --2- 172.21.15.31:0/3845036754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6100107650 0x7f61001a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.992+0000 7f6104c00640 1 --2- 172.21.15.31:0/3845036754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6100102000 0x7f610019eaa0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.992+0000 7f6104c00640 1 --2- 172.21.15.31:0/3845036754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6100101640 0x7f610019e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:54.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.992+0000 7f6104c00640 1 -- 172.21.15.31:0/3845036754 >> 172.21.15.31:0/3845036754 conn(0x7f61000fd130 msgr2=0x7f61000fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:54.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.993+0000 7f6104c00640 1 -- 172.21.15.31:0/3845036754 shutdown_connections 2024-03-21T10:45:54.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:54.993+0000 7f6104c00640 1 -- 172.21.15.31:0/3845036754 wait complete. 2024-03-21T10:45:55.018 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:45:55.018 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:45:55.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.414+0000 7f7cfa800640 1 -- 172.21.15.31:0/1480247349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cf41086a0 msgr2=0x7f7cf4108a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:55.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.414+0000 7f7cfa800640 1 --2- 172.21.15.31:0/1480247349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cf41086a0 0x7f7cf4108a80 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4009900 tx=0x7f7ce402bc50 comp rx=0 tx=0).stop 2024-03-21T10:45:55.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.416+0000 7f7cfa800640 1 -- 172.21.15.31:0/1480247349 shutdown_connections 2024-03-21T10:45:55.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.416+0000 7f7cfa800640 1 --2- 172.21.15.31:0/1480247349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7cf41147f0 0x7f7cf4116be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.416+0000 7f7cfa800640 1 --2- 172.21.15.31:0/1480247349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cf4108fc0 0x7f7cf41142b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.416+0000 7f7cfa800640 1 --2- 172.21.15.31:0/1480247349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cf41086a0 0x7f7cf4108a80 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.416+0000 7f7cfa800640 1 -- 172.21.15.31:0/1480247349 >> 172.21.15.31:0/1480247349 conn(0x7f7cf406c710 msgr2=0x7f7cf406cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:55.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.417+0000 7f7cfa800640 1 -- 172.21.15.31:0/1480247349 shutdown_connections 2024-03-21T10:45:55.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.417+0000 7f7cfa800640 1 -- 172.21.15.31:0/1480247349 wait complete. 2024-03-21T10:45:55.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.418+0000 7f7cfa800640 1 Processor -- start 2024-03-21T10:45:55.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.420+0000 7f7cfa800640 1 -- start start 2024-03-21T10:45:55.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.420+0000 7f7cfa800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cf41086a0 0x7f7cf411e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:55.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.420+0000 7f7cfa800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cf4108fc0 0x7f7cf411e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:55.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.420+0000 7f7cfa800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7cf41147f0 0x7f7cf41195d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:55.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.420+0000 7f7cfa800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf4118070 con 0x7f7cf41086a0 2024-03-21T10:45:55.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.420+0000 7f7cfa800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf4117ef0 con 0x7f7cf4108fc0 2024-03-21T10:45:55.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.420+0000 7f7cfa800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf41181f0 con 0x7f7cf41147f0 2024-03-21T10:45:55.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.421+0000 7f7cf2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cf4108fc0 0x7f7cf411e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:55.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.421+0000 7f7cf2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cf4108fc0 0x7f7cf411e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:35986/0 (socket says 172.21.15.31:35986) 2024-03-21T10:45:55.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.421+0000 7f7cf2a00640 1 -- 172.21.15.31:0/3201928804 learned_addr learned my addr 172.21.15.31:0/3201928804 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:55.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.421+0000 7f7cf3400640 1 --2- 172.21.15.31:0/3201928804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cf41086a0 0x7f7cf411e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:55.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.421+0000 7f7cf3e00640 1 --2- 172.21.15.31:0/3201928804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7cf41147f0 0x7f7cf41195d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:55.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.422+0000 7f7cf3400640 1 -- 172.21.15.31:0/3201928804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7cf41147f0 msgr2=0x7f7cf41195d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:55.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.422+0000 7f7cf3400640 1 --2- 172.21.15.31:0/3201928804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7cf41147f0 0x7f7cf41195d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.422+0000 7f7cf3400640 1 -- 172.21.15.31:0/3201928804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cf4108fc0 msgr2=0x7f7cf411e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:55.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.422+0000 7f7cf3400640 1 --2- 172.21.15.31:0/3201928804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cf4108fc0 0x7f7cf411e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.422+0000 7f7cf3400640 1 -- 172.21.15.31:0/3201928804 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf4119b90 con 0x7f7cf41086a0 2024-03-21T10:45:55.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.422+0000 7f7cf2a00640 1 --2- 172.21.15.31:0/3201928804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cf4108fc0 0x7f7cf411e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:45:55.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.422+0000 7f7cf3400640 1 --2- 172.21.15.31:0/3201928804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cf41086a0 0x7f7cf411e470 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4009850 tx=0x7f7ce40045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:55.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.426+0000 7f7cebe00640 1 -- 172.21.15.31:0/3201928804 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce4043020 con 0x7f7cf41086a0 2024-03-21T10:45:55.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.426+0000 7f7cebe00640 1 -- 172.21.15.31:0/3201928804 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ce40054e0 con 0x7f7cf41086a0 2024-03-21T10:45:55.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.426+0000 7f7cebe00640 1 -- 172.21.15.31:0/3201928804 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce4043020 con 0x7f7cf41086a0 2024-03-21T10:45:55.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.427+0000 7f7cfa800640 1 -- 172.21.15.31:0/3201928804 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf4119e20 con 0x7f7cf41086a0 2024-03-21T10:45:55.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.427+0000 7f7cfa800640 1 -- 172.21.15.31:0/3201928804 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf411a230 con 0x7f7cf41086a0 2024-03-21T10:45:55.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.437+0000 7f7cebe00640 1 -- 172.21.15.31:0/3201928804 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f7ce4046020 con 0x7f7cf41086a0 2024-03-21T10:45:55.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.440+0000 7f7cfd592740 1 Processor -- start 2024-03-21T10:45:55.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.441+0000 7f7cebe00640 1 --2- 172.21.15.31:0/3201928804 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7cd80a34d0 0x7f7cd80a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:55.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.441+0000 7f7cebe00640 1 -- 172.21.15.31:0/3201928804 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f7ce403a040 con 0x7f7cf41086a0 2024-03-21T10:45:55.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.442+0000 7f7cf3e00640 1 --2- 172.21.15.31:0/3201928804 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7cd80a34d0 0x7f7cd80a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:55.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.442+0000 7f7cfd592740 1 -- start start 2024-03-21T10:45:55.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.443+0000 7f7cfd592740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea48d7ac00 0x55ea48d79950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:55.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.443+0000 7f7cfd592740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea48d79f20 0x55ea48d7a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:55.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.443+0000 7f7cfd592740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea48d83290 0x55ea48d85680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:55.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.443+0000 7f7cfd592740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea48cde250 con 0x55ea48d79f20 2024-03-21T10:45:55.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.443+0000 7f7cfd592740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea48c86a30 con 0x55ea48d83290 2024-03-21T10:45:55.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.443+0000 7f7cfd592740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea48cda2f0 con 0x55ea48d7ac00 2024-03-21T10:45:55.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.445+0000 7f7cf3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea48d7ac00 0x55ea48d79950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:55.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.445+0000 7f7cf3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea48d7ac00 0x55ea48d79950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:51004/0 (socket says 172.21.15.31:51004) 2024-03-21T10:45:55.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.445+0000 7f7cf3400640 1 -- 172.21.15.31:0/3440999212 learned_addr learned my addr 172.21.15.31:0/3440999212 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:55.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.445+0000 7f7cf3e00640 1 --2- 172.21.15.31:0/3440999212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea48d83290 0x55ea48d85680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:55.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.446+0000 7f7cf3400640 1 -- 172.21.15.31:0/3440999212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea48d83290 msgr2=0x55ea48d85680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:55.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.446+0000 7f7cf3400640 1 --2- 172.21.15.31:0/3440999212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea48d83290 0x55ea48d85680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.446+0000 7f7cf3400640 1 -- 172.21.15.31:0/3440999212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea48d79f20 msgr2=0x55ea48d7a3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:45:55.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.446+0000 7f7cf3400640 1 --2- 172.21.15.31:0/3440999212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea48d79f20 0x55ea48d7a3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.446+0000 7f7cf3400640 1 -- 172.21.15.31:0/3440999212 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea48c71690 con 0x55ea48d7ac00 2024-03-21T10:45:55.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.447+0000 7f7cf3400640 1 --2- 172.21.15.31:0/3440999212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea48d7ac00 0x55ea48d79950 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4006010 tx=0x7f7ce40baac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e64b2bf2893b89f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:55.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.448+0000 7f7ce8c00640 1 -- 172.21.15.31:0/3440999212 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce4043070 con 0x55ea48d7ac00 2024-03-21T10:45:55.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.449+0000 7f7ce8c00640 1 -- 172.21.15.31:0/3440999212 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ce40f1070 con 0x55ea48d7ac00 2024-03-21T10:45:55.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.450+0000 7f7ce8c00640 1 -- 172.21.15.31:0/3440999212 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce4043070 con 0x55ea48d7ac00 2024-03-21T10:45:55.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.454+0000 7f7cfd592740 1 -- 172.21.15.31:0/3440999212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea48d7ac00 msgr2=0x55ea48d79950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:55.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.454+0000 7f7cfd592740 1 --2- 172.21.15.31:0/3440999212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea48d7ac00 0x55ea48d79950 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4006010 tx=0x7f7ce40baac0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.455+0000 7f7cfd592740 1 -- 172.21.15.31:0/3440999212 shutdown_connections 2024-03-21T10:45:55.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.455+0000 7f7cfd592740 1 --2- 172.21.15.31:0/3440999212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea48d83290 0x55ea48d85680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.455+0000 7f7cfd592740 1 --2- 172.21.15.31:0/3440999212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea48d79f20 0x55ea48d7a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.455+0000 7f7cfd592740 1 --2- 172.21.15.31:0/3440999212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea48d7ac00 0x55ea48d79950 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.455+0000 7f7cfd592740 1 -- 172.21.15.31:0/3440999212 >> 172.21.15.31:0/3440999212 conn(0x55ea48d41200 msgr2=0x55ea48d74970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:55.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.455+0000 7f7cf3e00640 1 --2- 172.21.15.31:0/3201928804 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7cd80a34d0 0x7f7cd80a5990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4072be0 tx=0x7f7cec0079b0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:55.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.456+0000 7f7cfd592740 1 -- 172.21.15.31:0/3440999212 shutdown_connections 2024-03-21T10:45:55.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.456+0000 7f7cfd592740 1 -- 172.21.15.31:0/3440999212 wait complete. 2024-03-21T10:45:55.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.457+0000 7f7cfd592740 1 -- start start 2024-03-21T10:45:55.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.458+0000 7f7cfd592740 1 Processor -- start 2024-03-21T10:45:55.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.459+0000 7f7cfd592740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea48d79f20 0x55ea48d82990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:55.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.459+0000 7f7cfd592740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea48d7ac00 0x55ea48d7fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:55.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.459+0000 7f7cfd592740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea48d83290 0x55ea48d80260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:55.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.459+0000 7f7cfd592740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea48d85ec0 con 0x55ea48d79f20 2024-03-21T10:45:55.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.459+0000 7f7cfd592740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea48c99e80 con 0x55ea48d83290 2024-03-21T10:45:55.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.459+0000 7f7cfd592740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea48d86040 con 0x55ea48d7ac00 2024-03-21T10:45:55.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.460+0000 7f7cfd592740 10 client.0 _ll_register_callbacks cb 0x55ea48cac130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:55.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.460+0000 7f7cf2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea48d7ac00 0x55ea48d7fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:55.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.460+0000 7f7cf2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea48d7ac00 0x55ea48d7fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:51018/0 (socket says 172.21.15.31:51018) 2024-03-21T10:45:55.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.460+0000 7f7cf2a00640 1 -- 172.21.15.31:0/2028205761 learned_addr learned my addr 172.21.15.31:0/2028205761 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:55.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.460+0000 7f7cf2a00640 1 -- 172.21.15.31:0/2028205761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea48d83290 msgr2=0x55ea48d80260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:45:55.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.460+0000 7f7cf2a00640 1 --2- 172.21.15.31:0/2028205761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea48d83290 0x55ea48d80260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.460+0000 7f7cf2a00640 1 -- 172.21.15.31:0/2028205761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea48d79f20 msgr2=0x55ea48d82990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:45:55.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.460+0000 7f7cf2a00640 1 --2- 172.21.15.31:0/2028205761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea48d79f20 0x55ea48d82990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.460+0000 7f7cf2a00640 1 -- 172.21.15.31:0/2028205761 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ea48abc7d0 con 0x55ea48d7ac00 2024-03-21T10:45:55.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.465+0000 7f7cf2a00640 1 --2- 172.21.15.31:0/2028205761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea48d7ac00 0x55ea48d7fd20 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f7ce000e710 tx=0x7f7ce000ebd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd22f5e10ff14d7a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:55.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.465+0000 7f7cdd600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:45:55.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.465+0000 7f7cdd600640 1 -- 172.21.15.31:0/2028205761 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce0017070 con 0x55ea48d7ac00 2024-03-21T10:45:55.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.465+0000 7f7cdd600640 1 -- 172.21.15.31:0/2028205761 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f7ce000c6b0 con 0x55ea48d7ac00 2024-03-21T10:45:55.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.466+0000 7f7cfd592740 1 -- 172.21.15.31:0/2028205761 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7ce40f1070 con 0x55ea48d7ac00 2024-03-21T10:45:55.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.466+0000 7f7cdd600640 1 -- 172.21.15.31:0/2028205761 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ce0021050 con 0x55ea48d7ac00 2024-03-21T10:45:55.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.466+0000 7f7cfd592740 10 client.27352 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:55.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.466+0000 7f7cfd592740 1 -- 172.21.15.31:0/2028205761 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55ea489f23f0 con 0x55ea48d7ac00 2024-03-21T10:45:55.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.467+0000 7f7cdd600640 1 -- 172.21.15.31:0/2028205761 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ce0013470 con 0x55ea48d7ac00 2024-03-21T10:45:55.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.467+0000 7f7cdd600640 1 -- 172.21.15.31:0/2028205761 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f7ce001c440 con 0x55ea48d7ac00 2024-03-21T10:45:55.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.467+0000 7f7cfd592740 10 client.27352 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:55.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.467+0000 7f7cfd592740 10 client.27352 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:45:55.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.468+0000 7f7cfd592740 20 client.27352 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:55.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.468+0000 7f7cfd592740 1 --2- 172.21.15.31:0/2028205761 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ea48d92f50 0x55ea48d95340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:55.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.468+0000 7f7cfd592740 4 client.27352 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:45:55.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.468+0000 7f7cfd592740 1 -- 172.21.15.31:0/2028205761 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ce40f1070 con 0x55ea48d92f50 2024-03-21T10:45:55.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.469+0000 7f7cf3400640 1 --2- 172.21.15.31:0/2028205761 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ea48d92f50 0x55ea48d95340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:55.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.469+0000 7f7cf3400640 1 --2- 172.21.15.31:0/2028205761 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ea48d92f50 0x55ea48d95340 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=387e2f3de6a71f2b server_cookie=ed1645cfee2d6374 in_seq=0 out_seq=0 2024-03-21T10:45:55.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.469+0000 7f7cdd600640 10 client.27352 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.474+0000 7f7cdd600640 1 -- 172.21.15.31:0/2028205761 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7f7ce40f1070 con 0x55ea48d92f50 2024-03-21T10:45:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.474+0000 7f7cdd600640 10 client.27352 handle_command_reply: tid=0 2024-03-21T10:45:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.474+0000 7f7cfd592740 1 client.27352 shutdown 2024-03-21T10:45:55.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.475+0000 7f7cfd592740 1 -- 172.21.15.31:0/2028205761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea48d7ac00 msgr2=0x55ea48d7fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:55.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.475+0000 7f7cfd592740 1 --2- 172.21.15.31:0/2028205761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea48d7ac00 0x55ea48d7fd20 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f7ce000e710 tx=0x7f7ce000ebd0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.475+0000 7f7cfd592740 1 -- 172.21.15.31:0/2028205761 shutdown_connections 2024-03-21T10:45:55.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.475+0000 7f7cfd592740 1 --2- 172.21.15.31:0/2028205761 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ea48d92f50 0x55ea48d95340 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.475+0000 7f7cfd592740 1 --2- 172.21.15.31:0/2028205761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea48d83290 0x55ea48d80260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.475+0000 7f7cfd592740 1 --2- 172.21.15.31:0/2028205761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea48d7ac00 0x55ea48d7fd20 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.475+0000 7f7cfd592740 1 --2- 172.21.15.31:0/2028205761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea48d79f20 0x55ea48d82990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.475+0000 7f7cfd592740 1 -- 172.21.15.31:0/2028205761 >> 172.21.15.31:0/2028205761 conn(0x55ea48d41200 msgr2=0x55ea48d766c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:55.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.475+0000 7f7cdd600640 0 client.27352 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:55.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.476+0000 7f7cfd592740 1 -- 172.21.15.31:0/2028205761 shutdown_connections 2024-03-21T10:45:55.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.477+0000 7f7cfd592740 1 -- 172.21.15.31:0/2028205761 wait complete. 2024-03-21T10:45:55.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.477+0000 7f7cfd592740 20 client.27352 trim_cache size 0 max 16384 2024-03-21T10:45:55.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.530+0000 7f7cfd592740 1 Processor -- start 2024-03-21T10:45:55.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.530+0000 7f7cfd592740 1 -- start start 2024-03-21T10:45:55.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.530+0000 7f7cfd592740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea48d79f20 0x55ea48d76460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:55.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.530+0000 7f7cfd592740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea48d7ac00 0x55ea48d82450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:55.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.530+0000 7f7cfd592740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea48d83290 0x55ea48d82990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:55.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.530+0000 7f7cfd592740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf41c84d0 con 0x55ea48d83290 2024-03-21T10:45:55.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.530+0000 7f7cfd592740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf41c5fd0 con 0x55ea48d7ac00 2024-03-21T10:45:55.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.530+0000 7f7cfd592740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf41c4090 con 0x55ea48d79f20 2024-03-21T10:45:55.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.531+0000 7f7cf2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea48d7ac00 0x55ea48d82450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:55.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.531+0000 7f7cf2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea48d7ac00 0x55ea48d82450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:35996/0 (socket says 172.21.15.31:35996) 2024-03-21T10:45:55.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.531+0000 7f7cf2a00640 1 -- 172.21.15.31:0/443603579 learned_addr learned my addr 172.21.15.31:0/443603579 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:55.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.532+0000 7f7cf3400640 1 --2- 172.21.15.31:0/443603579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea48d79f20 0x55ea48d76460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:55.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.532+0000 7f7cf3e00640 1 --2- 172.21.15.31:0/443603579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea48d83290 0x55ea48d82990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:55.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.533+0000 7f7cf3400640 1 -- 172.21.15.31:0/443603579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea48d7ac00 msgr2=0x55ea48d82450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:55.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.533+0000 7f7cf3400640 1 --2- 172.21.15.31:0/443603579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea48d7ac00 0x55ea48d82450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.533+0000 7f7cf3400640 1 -- 172.21.15.31:0/443603579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea48d83290 msgr2=0x55ea48d82990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:55.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.533+0000 7f7cf3400640 1 --2- 172.21.15.31:0/443603579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea48d83290 0x55ea48d82990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.533+0000 7f7cf3400640 1 -- 172.21.15.31:0/443603579 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea489b20f0 con 0x55ea48d79f20 2024-03-21T10:45:55.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.536+0000 7f7cf3400640 1 --2- 172.21.15.31:0/443603579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea48d79f20 0x55ea48d76460 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4049000 tx=0x7f7ce4041040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8dae56cf2afe350 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:55.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.537+0000 7f7cdfe00640 1 -- 172.21.15.31:0/443603579 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce4043040 con 0x55ea48d79f20 2024-03-21T10:45:55.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.537+0000 7f7cdfe00640 1 -- 172.21.15.31:0/443603579 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ce40f1040 con 0x55ea48d79f20 2024-03-21T10:45:55.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.537+0000 7f7cdfe00640 1 -- 172.21.15.31:0/443603579 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce40beee0 con 0x55ea48d79f20 2024-03-21T10:45:55.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.540+0000 7f7cfd592740 1 -- 172.21.15.31:0/443603579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea48d79f20 msgr2=0x55ea48d76460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:55.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.540+0000 7f7cfd592740 1 --2- 172.21.15.31:0/443603579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea48d79f20 0x55ea48d76460 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4049000 tx=0x7f7ce4041040 comp rx=0 tx=0).stop 2024-03-21T10:45:55.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.541+0000 7f7cfd592740 1 -- 172.21.15.31:0/443603579 shutdown_connections 2024-03-21T10:45:55.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.541+0000 7f7cfd592740 1 --2- 172.21.15.31:0/443603579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea48d83290 0x55ea48d82990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.541+0000 7f7cfd592740 1 --2- 172.21.15.31:0/443603579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea48d7ac00 0x55ea48d82450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.541+0000 7f7cfd592740 1 --2- 172.21.15.31:0/443603579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea48d79f20 0x55ea48d76460 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.541+0000 7f7cfd592740 1 -- 172.21.15.31:0/443603579 >> 172.21.15.31:0/443603579 conn(0x55ea48d41200 msgr2=0x55ea48d75d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:55.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.541+0000 7f7cfd592740 1 -- 172.21.15.31:0/443603579 shutdown_connections 2024-03-21T10:45:55.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.541+0000 7f7cfd592740 1 -- 172.21.15.31:0/443603579 wait complete. 2024-03-21T10:45:55.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.544+0000 7f7cfd592740 1 -- start start 2024-03-21T10:45:55.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.545+0000 7f7cfd592740 1 Processor -- start 2024-03-21T10:45:55.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.546+0000 7f7cfd592740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea48d79f20 0x55ea48d7d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:55.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.546+0000 7f7cfd592740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea48d7ac00 0x55ea48d7d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:55.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.546+0000 7f7cfd592740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea48d83290 0x55ea48d8dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:55.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.546+0000 7f7cf3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea48d79f20 0x55ea48d7d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:55.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.547+0000 7f7cfd592740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf41c4690 con 0x55ea48d7ac00 2024-03-21T10:45:55.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.547+0000 7f7cf3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea48d83290 0x55ea48d8dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:55.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.547+0000 7f7cf3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea48d83290 0x55ea48d8dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:36000/0 (socket says 172.21.15.31:36000) 2024-03-21T10:45:55.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.547+0000 7f7cf3e00640 1 -- 172.21.15.31:0/2629478820 learned_addr learned my addr 172.21.15.31:0/2629478820 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:55.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.547+0000 7f7cfd592740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf41c5970 con 0x55ea48d83290 2024-03-21T10:45:55.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.548+0000 7f7cfd592740 1 -- 172.21.15.31:0/2629478820 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf41c4cb0 con 0x55ea48d79f20 2024-03-21T10:45:55.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.548+0000 7f7cfd592740 10 client.0 _ll_register_callbacks cb 0x7f7ce0009840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:55.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.548+0000 7f7cf3e00640 1 -- 172.21.15.31:0/2629478820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea48d79f20 msgr2=0x55ea48d7d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:55.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.548+0000 7f7cf3e00640 1 --2- 172.21.15.31:0/2629478820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea48d79f20 0x55ea48d7d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.548+0000 7f7cf3e00640 1 -- 172.21.15.31:0/2629478820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea48d7ac00 msgr2=0x55ea48d7d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:55.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.548+0000 7f7cf3e00640 1 --2- 172.21.15.31:0/2629478820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea48d7ac00 0x55ea48d7d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.548+0000 7f7cf3e00640 1 -- 172.21.15.31:0/2629478820 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ea48d8e310 con 0x55ea48d83290 2024-03-21T10:45:55.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.550+0000 7f7cf3e00640 1 --2- 172.21.15.31:0/2629478820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea48d83290 0x55ea48d8dcd0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f7cec00f900 tx=0x7f7cec004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f85970a58e831138 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:55.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.550+0000 7f7cdd600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:45:55.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.550+0000 7f7cdd600640 1 -- 172.21.15.31:0/2629478820 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec024070 con 0x55ea48d83290 2024-03-21T10:45:55.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.550+0000 7f7cdd600640 1 -- 172.21.15.31:0/2629478820 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f7cec00ed20 con 0x55ea48d83290 2024-03-21T10:45:55.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.550+0000 7f7cfd592740 1 -- 172.21.15.31:0/2629478820 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7ce40f1040 con 0x55ea48d83290 2024-03-21T10:45:55.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.551+0000 7f7cdd600640 1 -- 172.21.15.31:0/2629478820 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7cec02e050 con 0x55ea48d83290 2024-03-21T10:45:55.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.551+0000 7f7cfd592740 10 client.27443 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:55.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.551+0000 7f7cfd592740 1 -- 172.21.15.31:0/2629478820 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55ea48def160 con 0x55ea48d83290 2024-03-21T10:45:55.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.553+0000 7f7cdd600640 1 -- 172.21.15.31:0/2629478820 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ce40f1040 con 0x55ea48d83290 2024-03-21T10:45:55.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.554+0000 7f7cdd600640 1 -- 172.21.15.31:0/2629478820 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f7cec020440 con 0x55ea48d83290 2024-03-21T10:45:55.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.554+0000 7f7cfd592740 10 client.27443 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:55.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.554+0000 7f7cfd592740 10 client.27443 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:45:55.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.554+0000 7f7cfd592740 20 client.27443 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:55.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.554+0000 7f7cfd592740 1 --2- 172.21.15.31:0/2629478820 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ea48def840 0x55ea48df1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:55.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.554+0000 7f7cfd592740 4 client.27443 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:45:55.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.554+0000 7f7cfd592740 1 -- 172.21.15.31:0/2629478820 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ea48ceb790 con 0x55ea48def840 2024-03-21T10:45:55.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.554+0000 7f7cf3400640 1 --2- 172.21.15.31:0/2629478820 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ea48def840 0x55ea48df1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:55.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.556+0000 7f7cf3400640 1 --2- 172.21.15.31:0/2629478820 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ea48def840 0x55ea48df1c80 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7328d76f300dd174 server_cookie=5d900e57c2fc9402 in_seq=0 out_seq=0 2024-03-21T10:45:55.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.562+0000 7f7cdd600640 10 client.27443 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:55.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.562+0000 7f7cdd600640 1 -- 172.21.15.31:0/2629478820 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55ea48ceb790 con 0x55ea48def840 2024-03-21T10:45:55.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.562+0000 7f7cdd600640 10 client.27443 handle_command_reply: tid=0 2024-03-21T10:45:55.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.562+0000 7f7cfd592740 1 client.27443 shutdown 2024-03-21T10:45:55.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.562+0000 7f7cfd592740 1 -- 172.21.15.31:0/2629478820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea48d83290 msgr2=0x55ea48d8dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:55.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.562+0000 7f7cfd592740 1 --2- 172.21.15.31:0/2629478820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea48d83290 0x55ea48d8dcd0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f7cec00f900 tx=0x7f7cec004a40 comp rx=0 tx=0).stop 2024-03-21T10:45:55.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.562+0000 7f7cfd592740 1 -- 172.21.15.31:0/2629478820 shutdown_connections 2024-03-21T10:45:55.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.562+0000 7f7cfd592740 1 --2- 172.21.15.31:0/2629478820 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ea48def840 0x55ea48df1c80 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.562+0000 7f7cfd592740 1 --2- 172.21.15.31:0/2629478820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ea48d83290 0x55ea48d8dcd0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.562+0000 7f7cfd592740 1 --2- 172.21.15.31:0/2629478820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea48d7ac00 0x55ea48d7d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.562+0000 7f7cfd592740 1 --2- 172.21.15.31:0/2629478820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea48d79f20 0x55ea48d7d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.562+0000 7f7cfd592740 1 -- 172.21.15.31:0/2629478820 >> 172.21.15.31:0/2629478820 conn(0x55ea48d41200 msgr2=0x55ea48d73a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:55.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.562+0000 7f7cdd600640 0 client.27443 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:55.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.563+0000 7f7cfd592740 1 -- 172.21.15.31:0/2629478820 shutdown_connections 2024-03-21T10:45:55.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.563+0000 7f7cfd592740 1 -- 172.21.15.31:0/2629478820 wait complete. 2024-03-21T10:45:55.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.563+0000 7f7cfd592740 20 client.27443 trim_cache size 0 max 16384 2024-03-21T10:45:55.561 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:45:55.561 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-21T10:45:55.561 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-03-21T10:45:55.562 INFO:teuthology.orchestra.run.smithi031.stdout: "e262e02a-3a00-4aa6-af0a-8e505443abfe": { 2024-03-21T10:45:55.562 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-03-21T10:45:55.562 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "e262e02a-3a00-4aa6-af0a-8e505443abfe", 2024-03-21T10:45:55.562 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-03-21T10:45:55.562 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:45:55.562 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:45:55.562 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:45:55.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.568+0000 7f7cfa800640 1 -- 172.21.15.31:0/3201928804 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7cd80a34d0 msgr2=0x7f7cd80a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:55.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.568+0000 7f7cfa800640 1 --2- 172.21.15.31:0/3201928804 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7cd80a34d0 0x7f7cd80a5990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4072be0 tx=0x7f7cec0079b0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.568+0000 7f7cfa800640 1 -- 172.21.15.31:0/3201928804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cf41086a0 msgr2=0x7f7cf411e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:55.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.568+0000 7f7cfa800640 1 --2- 172.21.15.31:0/3201928804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cf41086a0 0x7f7cf411e470 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4009850 tx=0x7f7ce40045e0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.569+0000 7f7cfa800640 1 -- 172.21.15.31:0/3201928804 shutdown_connections 2024-03-21T10:45:55.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.569+0000 7f7cfa800640 1 --2- 172.21.15.31:0/3201928804 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7cd80a34d0 0x7f7cd80a5990 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.569+0000 7f7cfa800640 1 --2- 172.21.15.31:0/3201928804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7cf41147f0 0x7f7cf41195d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.569+0000 7f7cfa800640 1 --2- 172.21.15.31:0/3201928804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cf4108fc0 0x7f7cf411e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.569+0000 7f7cfa800640 1 --2- 172.21.15.31:0/3201928804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cf41086a0 0x7f7cf411e470 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:55.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.569+0000 7f7cfa800640 1 -- 172.21.15.31:0/3201928804 >> 172.21.15.31:0/3201928804 conn(0x7f7cf406c710 msgr2=0x7f7cf4070210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:55.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.571+0000 7f7cfa800640 1 -- 172.21.15.31:0/3201928804 shutdown_connections 2024-03-21T10:45:55.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:45:55.574+0000 7f7cfa800640 1 -- 172.21.15.31:0/3201928804 wait complete. 2024-03-21T10:45:55.621 INFO:tasks.cephfs.filesystem:scrub status for tag:e262e02a-3a00-4aa6-af0a-8e505443abfe - {'path': '/', 'tag': 'e262e02a-3a00-4aa6-af0a-8e505443abfe', 'options': 'recursive,force'} 2024-03-21T10:45:55.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:55 smithi040 ceph-mon[36154]: pgmap v225: 129 pgs: 129 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 193 MiB/s wr, 1.31k op/s 2024-03-21T10:45:55.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:55 smithi040 ceph-mon[36154]: scrub queued for path: / 2024-03-21T10:45:55.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:55 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:45:55.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:55 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:45:55.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:55 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:45:55.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:55 smithi031 ceph-mon[31673]: pgmap v225: 129 pgs: 129 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 193 MiB/s wr, 1.31k op/s 2024-03-21T10:45:55.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:55 smithi031 ceph-mon[31673]: scrub queued for path: / 2024-03-21T10:45:55.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:55 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:45:55.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:55 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:45:55.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:55 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:45:55.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:55 smithi070 ceph-mon[36330]: pgmap v225: 129 pgs: 129 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 193 MiB/s wr, 1.31k op/s 2024-03-21T10:45:55.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:55 smithi070 ceph-mon[36330]: scrub queued for path: / 2024-03-21T10:45:55.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:55 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:45:55.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:55 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:45:55.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:55 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:45:56.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:56 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:45:56.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:56 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:45:56.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:56 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:45:56.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:56 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:45:56.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:56 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:45:56.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:56 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:45:57.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:57 smithi040 ceph-mon[36154]: pgmap v226: 129 pgs: 129 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 224 MiB/s wr, 1.53k op/s 2024-03-21T10:45:57.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:57 smithi031 ceph-mon[31673]: pgmap v226: 129 pgs: 129 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 224 MiB/s wr, 1.53k op/s 2024-03-21T10:45:57.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:57 smithi070 ceph-mon[36330]: pgmap v226: 129 pgs: 129 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 224 MiB/s wr, 1.53k op/s 2024-03-21T10:45:58.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:58 smithi040 ceph-mon[36154]: pgmap v227: 129 pgs: 129 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 160 MiB/s wr, 1.06k op/s 2024-03-21T10:45:58.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:45:58 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:45:58.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:58 smithi031 ceph-mon[31673]: pgmap v227: 129 pgs: 129 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 160 MiB/s wr, 1.06k op/s 2024-03-21T10:45:58.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:45:58 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:45:58.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:58 smithi070 ceph-mon[36330]: pgmap v227: 129 pgs: 129 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 160 MiB/s wr, 1.06k op/s 2024-03-21T10:45:58.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:45:58 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:45:59.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.871+0000 7f44d2400640 1 -- 172.21.15.31:0/1178929997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44cc10a040 msgr2=0x7f44cc10a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:59.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.871+0000 7f44d2400640 1 --2- 172.21.15.31:0/1178929997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44cc10a040 0x7f44cc10a420 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f44b4009900 tx=0x7f44b402bc30 comp rx=0 tx=0).stop 2024-03-21T10:45:59.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.872+0000 7f44d2400640 1 -- 172.21.15.31:0/1178929997 shutdown_connections 2024-03-21T10:45:59.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.872+0000 7f44d2400640 1 --2- 172.21.15.31:0/1178929997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44cc108ed0 0x7f44cc109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:59.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.872+0000 7f44d2400640 1 --2- 172.21.15.31:0/1178929997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44cc10a9f0 0x7f44cc1130b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:59.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.872+0000 7f44d2400640 1 --2- 172.21.15.31:0/1178929997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44cc10a040 0x7f44cc10a420 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:59.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.872+0000 7f44d2400640 1 -- 172.21.15.31:0/1178929997 >> 172.21.15.31:0/1178929997 conn(0x7f44cc06cbe0 msgr2=0x7f44cc06cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:59.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.872+0000 7f44d2400640 1 -- 172.21.15.31:0/1178929997 shutdown_connections 2024-03-21T10:45:59.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.872+0000 7f44d2400640 1 -- 172.21.15.31:0/1178929997 wait complete. 2024-03-21T10:45:59.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.874+0000 7f44d2400640 1 Processor -- start 2024-03-21T10:45:59.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.874+0000 7f44d2400640 1 -- start start 2024-03-21T10:45:59.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.875+0000 7f44d2400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44cc108ed0 0x7f44cc112f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:59.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.875+0000 7f44d2400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44cc10a040 0x7f44cc10fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:59.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.876+0000 7f44cb400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44cc108ed0 0x7f44cc112f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:59.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.876+0000 7f44c2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44cc10a040 0x7f44cc10fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:59.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.876+0000 7f44cb400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44cc108ed0 0x7f44cc112f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:35386/0 (socket says 172.21.15.31:35386) 2024-03-21T10:45:59.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.876+0000 7f44c2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44cc10a040 0x7f44cc10fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46622/0 (socket says 172.21.15.31:46622) 2024-03-21T10:45:59.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.876+0000 7f44d2400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44cc10a9f0 0x7f44cc110530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:59.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.876+0000 7f44cb400640 1 -- 172.21.15.31:0/239393819 learned_addr learned my addr 172.21.15.31:0/239393819 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:59.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.876+0000 7f44d2400640 1 -- 172.21.15.31:0/239393819 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44cc11c520 con 0x7f44cc10a9f0 2024-03-21T10:45:59.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.876+0000 7f44d2400640 1 -- 172.21.15.31:0/239393819 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44cc11c3a0 con 0x7f44cc10a040 2024-03-21T10:45:59.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.876+0000 7f44d2400640 1 -- 172.21.15.31:0/239393819 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44cc11c6a0 con 0x7f44cc108ed0 2024-03-21T10:45:59.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.877+0000 7f44cb400640 1 -- 172.21.15.31:0/239393819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44cc10a040 msgr2=0x7f44cc10fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:59.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.877+0000 7f44cb400640 1 --2- 172.21.15.31:0/239393819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44cc10a040 0x7f44cc10fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:59.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.877+0000 7f44cb400640 1 -- 172.21.15.31:0/239393819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44cc10a9f0 msgr2=0x7f44cc110530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:59.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.877+0000 7f44cb400640 1 --2- 172.21.15.31:0/239393819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44cc10a9f0 0x7f44cc110530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:59.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.877+0000 7f44cb400640 1 -- 172.21.15.31:0/239393819 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44cc110df0 con 0x7f44cc108ed0 2024-03-21T10:45:59.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.877+0000 7f44cb400640 1 --2- 172.21.15.31:0/239393819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44cc108ed0 0x7f44cc112f50 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f44b402c240 tx=0x7f44b4004340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:59.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.878+0000 7f44c8c00640 1 -- 172.21.15.31:0/239393819 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b4043020 con 0x7f44cc108ed0 2024-03-21T10:45:59.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.878+0000 7f44c8c00640 1 -- 172.21.15.31:0/239393819 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44b40054e0 con 0x7f44cc108ed0 2024-03-21T10:45:59.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.878+0000 7f44c8c00640 1 -- 172.21.15.31:0/239393819 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b4003c50 con 0x7f44cc108ed0 2024-03-21T10:45:59.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.878+0000 7f44d2400640 1 -- 172.21.15.31:0/239393819 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44cc1a8b00 con 0x7f44cc108ed0 2024-03-21T10:45:59.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.878+0000 7f44d2400640 1 -- 172.21.15.31:0/239393819 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44cc1a9000 con 0x7f44cc108ed0 2024-03-21T10:45:59.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.883+0000 7f44d5169740 1 Processor -- start 2024-03-21T10:45:59.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.883+0000 7f44c8c00640 1 -- 172.21.15.31:0/239393819 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f44b4046020 con 0x7f44cc108ed0 2024-03-21T10:45:59.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.883+0000 7f44d5169740 1 -- start start 2024-03-21T10:45:59.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.884+0000 7f44d5169740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1db5dfd0 0x560d1db5e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:59.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.884+0000 7f44d5169740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db5e8f0 0x560d1db67180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:59.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.884+0000 7f44d5169740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d1db676c0 0x560d1db69ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:59.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.884+0000 7f44d5169740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d1db203b0 con 0x560d1db676c0 2024-03-21T10:45:59.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.884+0000 7f44d5169740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d1dac2830 con 0x560d1db5dfd0 2024-03-21T10:45:59.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.884+0000 7f44d5169740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d1da6b5d0 con 0x560d1db5e8f0 2024-03-21T10:45:59.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.884+0000 7f44c2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d1db676c0 0x560d1db69ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:59.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.884+0000 7f44cb400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db5e8f0 0x560d1db67180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.884+0000 7f44cb400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db5e8f0 0x560d1db67180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:35394/0 (socket says 172.21.15.31:35394) 2024-03-21T10:45:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.884+0000 7f44cb400640 1 -- 172.21.15.31:0/1438613925 learned_addr learned my addr 172.21.15.31:0/1438613925 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.885+0000 7f44c2a00640 1 -- 172.21.15.31:0/1438613925 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db5e8f0 msgr2=0x560d1db67180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.885+0000 7f44c2a00640 1 --2- 172.21.15.31:0/1438613925 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db5e8f0 0x560d1db67180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.885+0000 7f44c2a00640 1 -- 172.21.15.31:0/1438613925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1db5dfd0 msgr2=0x560d1db5e3b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:45:59.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.885+0000 7f44c2a00640 1 --2- 172.21.15.31:0/1438613925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1db5dfd0 0x560d1db5e3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:59.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.885+0000 7f44c2a00640 1 -- 172.21.15.31:0/1438613925 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560d1da832c0 con 0x560d1db676c0 2024-03-21T10:45:59.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.885+0000 7f44cb400640 1 --2- 172.21.15.31:0/1438613925 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db5e8f0 0x560d1db67180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:45:59.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.886+0000 7f44c2a00640 1 --2- 172.21.15.31:0/1438613925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d1db676c0 0x560d1db69ab0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f44cc111840 tx=0x7f44bc0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c075a766d6af2d08 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.887+0000 7f44c8c00640 1 --2- 172.21.15.31:0/239393819 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f44b00a31d0 0x7f44b00a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:59.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.887+0000 7f44c8c00640 1 -- 172.21.15.31:0/239393819 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f44b40f1c80 con 0x7f44cc108ed0 2024-03-21T10:45:59.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.895+0000 7f44cbe00640 1 --2- 172.21.15.31:0/239393819 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f44b00a31d0 0x7f44b00a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:59.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.902+0000 7f44c0c00640 1 -- 172.21.15.31:0/1438613925 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44bc003c30 con 0x560d1db676c0 2024-03-21T10:45:59.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.902+0000 7f44c0c00640 1 -- 172.21.15.31:0/1438613925 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44bc003dd0 con 0x560d1db676c0 2024-03-21T10:45:59.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.902+0000 7f44c0c00640 1 -- 172.21.15.31:0/1438613925 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44bc00fb50 con 0x560d1db676c0 2024-03-21T10:45:59.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.903+0000 7f44cbe00640 1 --2- 172.21.15.31:0/239393819 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f44b00a31d0 0x7f44b00a5690 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f44cc073150 tx=0x7f44c4006c70 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:59.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.908+0000 7f44d5169740 1 -- 172.21.15.31:0/1438613925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d1db676c0 msgr2=0x560d1db69ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:59.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.908+0000 7f44d5169740 1 --2- 172.21.15.31:0/1438613925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d1db676c0 0x560d1db69ab0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f44cc111840 tx=0x7f44bc0118d0 comp rx=0 tx=0).stop 2024-03-21T10:45:59.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.909+0000 7f44d5169740 1 -- 172.21.15.31:0/1438613925 shutdown_connections 2024-03-21T10:45:59.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.909+0000 7f44d5169740 1 --2- 172.21.15.31:0/1438613925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d1db676c0 0x560d1db69ab0 secure :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f44cc111840 tx=0x7f44bc0118d0 comp rx=0 tx=0).stop 2024-03-21T10:45:59.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.909+0000 7f44d5169740 1 --2- 172.21.15.31:0/1438613925 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db5e8f0 0x560d1db67180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:59.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.909+0000 7f44d5169740 1 --2- 172.21.15.31:0/1438613925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1db5dfd0 0x560d1db5e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:59.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.909+0000 7f44d5169740 1 -- 172.21.15.31:0/1438613925 >> 172.21.15.31:0/1438613925 conn(0x560d1db588f0 msgr2=0x560d1db58cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:59.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.915+0000 7f44d5169740 1 -- 172.21.15.31:0/1438613925 shutdown_connections 2024-03-21T10:45:59.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.918+0000 7f44d5169740 1 -- 172.21.15.31:0/1438613925 wait complete. 2024-03-21T10:45:59.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.918+0000 7f44d5169740 1 -- start start 2024-03-21T10:45:59.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.919+0000 7f44d5169740 1 Processor -- start 2024-03-21T10:45:59.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.920+0000 7f44d5169740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1db5dfd0 0x560d1db648f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:59.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.920+0000 7f44d5169740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d1db5e8f0 0x560d1db64e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:59.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.920+0000 7f44d5169740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db653c0 0x560d1db735a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:59.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.921+0000 7f44c2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d1db5e8f0 0x560d1db64e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:59.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.921+0000 7f44cb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1db5dfd0 0x560d1db648f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:59.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.921+0000 7f44cb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1db5dfd0 0x560d1db648f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46636/0 (socket says 172.21.15.31:46636) 2024-03-21T10:45:59.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.921+0000 7f44d5169740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d1da7ea20 con 0x560d1db5e8f0 2024-03-21T10:45:59.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.921+0000 7f44d5169740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d1dabe8d0 con 0x560d1db5dfd0 2024-03-21T10:45:59.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.921+0000 7f44d5169740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d1db6a2f0 con 0x560d1db653c0 2024-03-21T10:45:59.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.921+0000 7f44d5169740 10 client.0 _ll_register_callbacks cb 0x560d1dabb1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:45:59.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.921+0000 7f44cb400640 1 -- 172.21.15.31:0/1579284945 learned_addr learned my addr 172.21.15.31:0/1579284945 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:45:59.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.922+0000 7f44cbe00640 1 --2- 172.21.15.31:0/1579284945 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db653c0 0x560d1db735a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:59.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.923+0000 7f44cbe00640 1 -- 172.21.15.31:0/1579284945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1db5dfd0 msgr2=0x560d1db648f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:59.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.923+0000 7f44cbe00640 1 --2- 172.21.15.31:0/1579284945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1db5dfd0 0x560d1db648f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:59.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.923+0000 7f44cbe00640 1 -- 172.21.15.31:0/1579284945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d1db5e8f0 msgr2=0x560d1db64e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:45:59.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.923+0000 7f44cbe00640 1 --2- 172.21.15.31:0/1579284945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d1db5e8f0 0x560d1db64e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:59.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.923+0000 7f44cbe00640 1 -- 172.21.15.31:0/1579284945 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560d1d7d7210 con 0x560d1db653c0 2024-03-21T10:45:59.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.924+0000 7f44c2a00640 1 --2- 172.21.15.31:0/1579284945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d1db5e8f0 0x560d1db64e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:45:59.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.925+0000 7f44cbe00640 1 --2- 172.21.15.31:0/1579284945 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db653c0 0x560d1db735a0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f44c4013ec0 tx=0x7f44c40047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b3698efae0d1e12 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:45:59.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.926+0000 7f44b9600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:45:59.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.930+0000 7f44b9600640 1 -- 172.21.15.31:0/1579284945 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c401d5f0 con 0x560d1db653c0 2024-03-21T10:45:59.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.930+0000 7f44d5169740 1 -- 172.21.15.31:0/1579284945 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f44bc003dd0 con 0x560d1db653c0 2024-03-21T10:45:59.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.930+0000 7f44b9600640 1 -- 172.21.15.31:0/1579284945 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f44c401d790 con 0x560d1db653c0 2024-03-21T10:45:59.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.936+0000 7f44b9600640 1 -- 172.21.15.31:0/1579284945 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f44c401d990 con 0x560d1db653c0 2024-03-21T10:45:59.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.936+0000 7f44d5169740 10 client.27367 fetch_fsmap learned FSMap version 26 2024-03-21T10:45:59.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.936+0000 7f44d5169740 1 -- 172.21.15.31:0/1579284945 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x560d1d8a3620 con 0x560d1db653c0 2024-03-21T10:45:59.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.937+0000 7f44b9600640 1 -- 172.21.15.31:0/1579284945 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44bc003dd0 con 0x560d1db653c0 2024-03-21T10:45:59.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.937+0000 7f44b9600640 1 -- 172.21.15.31:0/1579284945 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f44c4012070 con 0x560d1db653c0 2024-03-21T10:45:59.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.943+0000 7f44d5169740 10 client.27367 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:45:59.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.943+0000 7f44d5169740 10 client.27367 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:45:59.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.943+0000 7f44d5169740 20 client.27367 populate_metadata read hostname 'smithi031' 2024-03-21T10:45:59.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.943+0000 7f44d5169740 1 --2- 172.21.15.31:0/1579284945 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560d1db77e00 0x560d1db7a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:45:59.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.943+0000 7f44d5169740 4 client.27367 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:45:59.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.943+0000 7f44d5169740 1 -- 172.21.15.31:0/1579284945 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560d1dacfd70 con 0x560d1db77e00 2024-03-21T10:45:59.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.944+0000 7f44cb400640 1 --2- 172.21.15.31:0/1579284945 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560d1db77e00 0x560d1db7a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:45:59.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.946+0000 7f44cb400640 1 --2- 172.21.15.31:0/1579284945 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560d1db77e00 0x560d1db7a1f0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=102eaccf8fec74bb server_cookie=a1c8ef559690a25b in_seq=0 out_seq=0 2024-03-21T10:45:59.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.946+0000 7f44b9600640 10 client.27367 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:59.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.947+0000 7f44b9600640 1 -- 172.21.15.31:0/1579284945 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560d1dacfd70 con 0x560d1db77e00 2024-03-21T10:45:59.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.947+0000 7f44b9600640 10 client.27367 handle_command_reply: tid=0 2024-03-21T10:45:59.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.948+0000 7f44d5169740 1 client.27367 shutdown 2024-03-21T10:45:59.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.953+0000 7f44d5169740 1 -- 172.21.15.31:0/1579284945 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db653c0 msgr2=0x560d1db735a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:45:59.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.953+0000 7f44d5169740 1 --2- 172.21.15.31:0/1579284945 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db653c0 0x560d1db735a0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f44c4013ec0 tx=0x7f44c40047d0 comp rx=0 tx=0).stop 2024-03-21T10:45:59.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.953+0000 7f44d5169740 1 -- 172.21.15.31:0/1579284945 shutdown_connections 2024-03-21T10:45:59.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.953+0000 7f44d5169740 1 --2- 172.21.15.31:0/1579284945 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560d1db77e00 0x560d1db7a1f0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:59.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.953+0000 7f44d5169740 1 --2- 172.21.15.31:0/1579284945 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db653c0 0x560d1db735a0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:59.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.953+0000 7f44d5169740 1 --2- 172.21.15.31:0/1579284945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d1db5e8f0 0x560d1db64e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:59.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.953+0000 7f44d5169740 1 --2- 172.21.15.31:0/1579284945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1db5dfd0 0x560d1db648f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:45:59.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.953+0000 7f44d5169740 1 -- 172.21.15.31:0/1579284945 >> 172.21.15.31:0/1579284945 conn(0x560d1db588f0 msgr2=0x560d1db5a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:45:59.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.953+0000 7f44b9600640 0 client.27367 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:45:59.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.956+0000 7f44d5169740 1 -- 172.21.15.31:0/1579284945 shutdown_connections 2024-03-21T10:45:59.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.958+0000 7f44d5169740 1 -- 172.21.15.31:0/1579284945 wait complete. 2024-03-21T10:45:59.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:45:59.959+0000 7f44d5169740 20 client.27367 trim_cache size 0 max 16384 2024-03-21T10:46:00.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.016+0000 7f44d5169740 1 Processor -- start 2024-03-21T10:46:00.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.016+0000 7f44d5169740 1 -- start start 2024-03-21T10:46:00.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.016+0000 7f44d5169740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1db5dfd0 0x560d1db5a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:00.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.017+0000 7f44d5169740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db5e8f0 0x560d1db61b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:00.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.017+0000 7f44d5169740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d1db653c0 0x560d1db620b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:00.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.017+0000 7f44d5169740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44cc1c85a0 con 0x560d1db653c0 2024-03-21T10:46:00.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.017+0000 7f44d5169740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44cc1c60a0 con 0x560d1db5dfd0 2024-03-21T10:46:00.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.017+0000 7f44d5169740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44cc1c4160 con 0x560d1db5e8f0 2024-03-21T10:46:00.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.017+0000 7f44c2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db5e8f0 0x560d1db61b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:00.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.017+0000 7f44c2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db5e8f0 0x560d1db61b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:35416/0 (socket says 172.21.15.31:35416) 2024-03-21T10:46:00.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.017+0000 7f44c2a00640 1 -- 172.21.15.31:0/2950298933 learned_addr learned my addr 172.21.15.31:0/2950298933 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:00.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.017+0000 7f44cb400640 1 --2- 172.21.15.31:0/2950298933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1db5dfd0 0x560d1db5a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:00.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.017+0000 7f44c2a00640 1 -- 172.21.15.31:0/2950298933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1db5dfd0 msgr2=0x560d1db5a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:00.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.017+0000 7f44c2a00640 1 --2- 172.21.15.31:0/2950298933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1db5dfd0 0x560d1db5a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:00.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.018+0000 7f44c2a00640 1 -- 172.21.15.31:0/2950298933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d1db653c0 msgr2=0x560d1db620b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:00.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.018+0000 7f44c2a00640 1 --2- 172.21.15.31:0/2950298933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d1db653c0 0x560d1db620b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:00.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.018+0000 7f44c2a00640 1 -- 172.21.15.31:0/2950298933 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560d1db594c0 con 0x560d1db5e8f0 2024-03-21T10:46:00.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.018+0000 7f44c2a00640 1 --2- 172.21.15.31:0/2950298933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db5e8f0 0x560d1db61b70 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f44bc006010 tx=0x7f44bc009ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65256211ddfefd8d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:00.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.018+0000 7f44bbe00640 1 -- 172.21.15.31:0/2950298933 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44bc01f040 con 0x560d1db5e8f0 2024-03-21T10:46:00.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.018+0000 7f44bbe00640 1 -- 172.21.15.31:0/2950298933 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44bc02a070 con 0x560d1db5e8f0 2024-03-21T10:46:00.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.018+0000 7f44bbe00640 1 -- 172.21.15.31:0/2950298933 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44bc0256e0 con 0x560d1db5e8f0 2024-03-21T10:46:00.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.019+0000 7f44d5169740 1 -- 172.21.15.31:0/2950298933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db5e8f0 msgr2=0x560d1db61b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:00.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.019+0000 7f44d5169740 1 --2- 172.21.15.31:0/2950298933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db5e8f0 0x560d1db61b70 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f44bc006010 tx=0x7f44bc009ce0 comp rx=0 tx=0).stop 2024-03-21T10:46:00.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.019+0000 7f44d5169740 1 -- 172.21.15.31:0/2950298933 shutdown_connections 2024-03-21T10:46:00.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.020+0000 7f44d5169740 1 --2- 172.21.15.31:0/2950298933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d1db653c0 0x560d1db620b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:00.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.020+0000 7f44d5169740 1 --2- 172.21.15.31:0/2950298933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db5e8f0 0x560d1db61b70 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:00.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.020+0000 7f44d5169740 1 --2- 172.21.15.31:0/2950298933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1db5dfd0 0x560d1db5a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:00.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.020+0000 7f44d5169740 1 -- 172.21.15.31:0/2950298933 >> 172.21.15.31:0/2950298933 conn(0x560d1db588f0 msgr2=0x560d1db59d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:00.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.020+0000 7f44d5169740 1 -- 172.21.15.31:0/2950298933 shutdown_connections 2024-03-21T10:46:00.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.020+0000 7f44d5169740 1 -- 172.21.15.31:0/2950298933 wait complete. 2024-03-21T10:46:00.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.020+0000 7f44d5169740 1 -- start start 2024-03-21T10:46:00.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.021+0000 7f44d5169740 1 Processor -- start 2024-03-21T10:46:00.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.022+0000 7f44d5169740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d1db5dfd0 0x560d1db6caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:00.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.022+0000 7f44d5169740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1db5e8f0 0x560d1db6d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:00.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.022+0000 7f44d5169740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db653c0 0x560d1db6d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:00.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.022+0000 7f44d5169740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44cc1c4760 con 0x560d1db5dfd0 2024-03-21T10:46:00.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.022+0000 7f44d5169740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44cc1c5a40 con 0x560d1db5e8f0 2024-03-21T10:46:00.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.022+0000 7f44d5169740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44cc1c4d80 con 0x560d1db653c0 2024-03-21T10:46:00.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.022+0000 7f44d5169740 10 client.0 _ll_register_callbacks cb 0x7f44c4006ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:46:00.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.022+0000 7f44cbe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db653c0 0x560d1db6d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:00.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.023+0000 7f44cbe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db653c0 0x560d1db6d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:35426/0 (socket says 172.21.15.31:35426) 2024-03-21T10:46:00.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.023+0000 7f44cbe00640 1 -- 172.21.15.31:0/1233325172 learned_addr learned my addr 172.21.15.31:0/1233325172 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:00.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.023+0000 7f44cb400640 1 --2- 172.21.15.31:0/1233325172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d1db5dfd0 0x560d1db6caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:00.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.023+0000 7f44c2a00640 1 --2- 172.21.15.31:0/1233325172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1db5e8f0 0x560d1db6d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:00.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.023+0000 7f44cbe00640 1 -- 172.21.15.31:0/1233325172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1db5e8f0 msgr2=0x560d1db6d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:00.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.023+0000 7f44cbe00640 1 --2- 172.21.15.31:0/1233325172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1db5e8f0 0x560d1db6d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:00.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.023+0000 7f44cbe00640 1 -- 172.21.15.31:0/1233325172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d1db5dfd0 msgr2=0x560d1db6caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:00.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.023+0000 7f44cbe00640 1 --2- 172.21.15.31:0/1233325172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d1db5dfd0 0x560d1db6caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:00.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.023+0000 7f44cbe00640 1 -- 172.21.15.31:0/1233325172 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560d1dbd2740 con 0x560d1db653c0 2024-03-21T10:46:00.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.023+0000 7f44c2a00640 1 --2- 172.21.15.31:0/1233325172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1db5e8f0 0x560d1db6d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:46:00.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.024+0000 7f44cbe00640 1 --2- 172.21.15.31:0/1233325172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db653c0 0x560d1db6d570 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f44c4009e90 tx=0x7f44c401ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4913d1ccc4179c2b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:00.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.024+0000 7f44b9600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:46:00.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.024+0000 7f44b9600640 1 -- 172.21.15.31:0/1233325172 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c401d600 con 0x560d1db653c0 2024-03-21T10:46:00.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.024+0000 7f44b9600640 1 -- 172.21.15.31:0/1233325172 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f44c4005a30 con 0x560d1db653c0 2024-03-21T10:46:00.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.024+0000 7f44d5169740 1 -- 172.21.15.31:0/1233325172 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f44bc02a070 con 0x560d1db653c0 2024-03-21T10:46:00.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.026+0000 7f44b9600640 1 -- 172.21.15.31:0/1233325172 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f44c401d7a0 con 0x560d1db653c0 2024-03-21T10:46:00.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.026+0000 7f44d5169740 10 client.27373 fetch_fsmap learned FSMap version 26 2024-03-21T10:46:00.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.026+0000 7f44d5169740 1 -- 172.21.15.31:0/1233325172 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x560d1dbd3a50 con 0x560d1db653c0 2024-03-21T10:46:00.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.034+0000 7f44b9600640 1 -- 172.21.15.31:0/1233325172 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44bc02a070 con 0x560d1db653c0 2024-03-21T10:46:00.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.034+0000 7f44b9600640 1 -- 172.21.15.31:0/1233325172 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f44c4025350 con 0x560d1db653c0 2024-03-21T10:46:00.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.034+0000 7f44d5169740 10 client.27373 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:46:00.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.040+0000 7f44d5169740 10 client.27373 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:46:00.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.040+0000 7f44d5169740 20 client.27373 populate_metadata read hostname 'smithi031' 2024-03-21T10:46:00.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.040+0000 7f44d5169740 1 --2- 172.21.15.31:0/1233325172 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560d1dbd4160 0x560d1dbd6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:00.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.040+0000 7f44d5169740 4 client.27373 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:46:00.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.040+0000 7f44d5169740 1 -- 172.21.15.31:0/1233325172 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560d1daca0c0 con 0x560d1dbd4160 2024-03-21T10:46:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.042+0000 7f44cb400640 1 --2- 172.21.15.31:0/1233325172 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560d1dbd4160 0x560d1dbd6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:00.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.044+0000 7f44cb400640 1 --2- 172.21.15.31:0/1233325172 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560d1dbd4160 0x560d1dbd6580 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=289676803a226dee server_cookie=475c601d65e430d3 in_seq=0 out_seq=0 2024-03-21T10:46:00.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.044+0000 7f44b9600640 10 client.27373 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.070+0000 7f44b9600640 1 -- 172.21.15.31:0/1233325172 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560d1daca0c0 con 0x560d1dbd4160 2024-03-21T10:46:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.070+0000 7f44b9600640 10 client.27373 handle_command_reply: tid=0 2024-03-21T10:46:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.074+0000 7f44d5169740 1 client.27373 shutdown 2024-03-21T10:46:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.076+0000 7f44d5169740 1 -- 172.21.15.31:0/1233325172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db653c0 msgr2=0x560d1db6d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.076+0000 7f44d5169740 1 --2- 172.21.15.31:0/1233325172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db653c0 0x560d1db6d570 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f44c4009e90 tx=0x7f44c401ed70 comp rx=0 tx=0).stop 2024-03-21T10:46:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.076+0000 7f44d5169740 1 -- 172.21.15.31:0/1233325172 shutdown_connections 2024-03-21T10:46:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.076+0000 7f44d5169740 1 --2- 172.21.15.31:0/1233325172 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560d1dbd4160 0x560d1dbd6580 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.076+0000 7f44d5169740 1 --2- 172.21.15.31:0/1233325172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d1db653c0 0x560d1db6d570 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.076+0000 7f44d5169740 1 --2- 172.21.15.31:0/1233325172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1db5e8f0 0x560d1db6d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.076+0000 7f44d5169740 1 --2- 172.21.15.31:0/1233325172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d1db5dfd0 0x560d1db6caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.076+0000 7f44d5169740 1 -- 172.21.15.31:0/1233325172 >> 172.21.15.31:0/1233325172 conn(0x560d1db588f0 msgr2=0x560d1db649e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.077+0000 7f44b9600640 0 client.27373 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.077+0000 7f44d5169740 1 -- 172.21.15.31:0/1233325172 shutdown_connections 2024-03-21T10:46:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.077+0000 7f44d5169740 1 -- 172.21.15.31:0/1233325172 wait complete. 2024-03-21T10:46:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.078+0000 7f44d5169740 20 client.27373 trim_cache size 0 max 16384 2024-03-21T10:46:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:46:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:46:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:46:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:46:00.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.079+0000 7f44d2400640 1 -- 172.21.15.31:0/239393819 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f44b00a31d0 msgr2=0x7f44b00a5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:00.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.079+0000 7f44d2400640 1 --2- 172.21.15.31:0/239393819 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f44b00a31d0 0x7f44b00a5690 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f44cc073150 tx=0x7f44c4006c70 comp rx=0 tx=0).stop 2024-03-21T10:46:00.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.079+0000 7f44d2400640 1 -- 172.21.15.31:0/239393819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44cc108ed0 msgr2=0x7f44cc112f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:00.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.079+0000 7f44d2400640 1 --2- 172.21.15.31:0/239393819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44cc108ed0 0x7f44cc112f50 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f44b402c240 tx=0x7f44b4004340 comp rx=0 tx=0).stop 2024-03-21T10:46:00.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.080+0000 7f44d2400640 1 -- 172.21.15.31:0/239393819 shutdown_connections 2024-03-21T10:46:00.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.080+0000 7f44d2400640 1 --2- 172.21.15.31:0/239393819 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f44b00a31d0 0x7f44b00a5690 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:00.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.080+0000 7f44d2400640 1 --2- 172.21.15.31:0/239393819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44cc10a9f0 0x7f44cc110530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:00.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.080+0000 7f44d2400640 1 --2- 172.21.15.31:0/239393819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44cc10a040 0x7f44cc10fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:00.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.080+0000 7f44d2400640 1 --2- 172.21.15.31:0/239393819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44cc108ed0 0x7f44cc112f50 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:00.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.080+0000 7f44d2400640 1 -- 172.21.15.31:0/239393819 >> 172.21.15.31:0/239393819 conn(0x7f44cc06cbe0 msgr2=0x7f44cc070190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:00.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.080+0000 7f44d2400640 1 -- 172.21.15.31:0/239393819 shutdown_connections 2024-03-21T10:46:00.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:00.080+0000 7f44d2400640 1 -- 172.21.15.31:0/239393819 wait complete. 2024-03-21T10:46:01.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:00 smithi040 ceph-mon[36154]: pgmap v228: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 232 MiB/s wr, 1.55k op/s 2024-03-21T10:46:01.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:00 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:46:01.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:00 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:46:01.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:00 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:46:01.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:00 smithi031 ceph-mon[31673]: pgmap v228: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 232 MiB/s wr, 1.55k op/s 2024-03-21T10:46:01.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:00 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:46:01.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:00 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:46:01.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:00 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:46:01.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:00 smithi070 ceph-mon[36330]: pgmap v228: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 232 MiB/s wr, 1.55k op/s 2024-03-21T10:46:01.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:00 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:46:01.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:00 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:46:01.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:00 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:46:02.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:01 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:46:02.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:01 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:46:02.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:01 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:46:02.309 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:01 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:46:02.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:01 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:46:02.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:01 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:46:03.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:02 smithi040 ceph-mon[36154]: pgmap v229: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 151 MiB/s wr, 1.03k op/s 2024-03-21T10:46:03.212 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:02 smithi031 ceph-mon[31673]: pgmap v229: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 151 MiB/s wr, 1.03k op/s 2024-03-21T10:46:03.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:02 smithi070 ceph-mon[36330]: pgmap v229: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 151 MiB/s wr, 1.03k op/s 2024-03-21T10:46:05.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:04 smithi040 ceph-mon[36154]: pgmap v230: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 167 MiB/s wr, 1.15k op/s 2024-03-21T10:46:05.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:04 smithi031 ceph-mon[31673]: pgmap v230: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 167 MiB/s wr, 1.15k op/s 2024-03-21T10:46:05.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:04 smithi070 ceph-mon[36330]: pgmap v230: 129 pgs: 129 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 167 MiB/s wr, 1.15k op/s 2024-03-21T10:46:07.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:06 smithi031 ceph-mon[31673]: pgmap v231: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 221 MiB/s wr, 1.40k op/s 2024-03-21T10:46:07.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:06 smithi070 ceph-mon[36330]: pgmap v231: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 221 MiB/s wr, 1.40k op/s 2024-03-21T10:46:07.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:06 smithi040 ceph-mon[36154]: pgmap v231: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 221 MiB/s wr, 1.40k op/s 2024-03-21T10:46:09.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:08 smithi031 ceph-mon[31673]: pgmap v232: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 167 MiB/s wr, 1.03k op/s 2024-03-21T10:46:09.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:08 smithi070 ceph-mon[36330]: pgmap v232: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 167 MiB/s wr, 1.03k op/s 2024-03-21T10:46:09.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:08 smithi040 ceph-mon[36154]: pgmap v232: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 167 MiB/s wr, 1.03k op/s 2024-03-21T10:46:11.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:10 smithi031 ceph-mon[31673]: pgmap v233: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 192 MiB/s wr, 1.11k op/s 2024-03-21T10:46:11.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:10 smithi070 ceph-mon[36330]: pgmap v233: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 192 MiB/s wr, 1.11k op/s 2024-03-21T10:46:11.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:10 smithi040 ceph-mon[36154]: pgmap v233: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 192 MiB/s wr, 1.11k op/s 2024-03-21T10:46:13.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:12 smithi031 ceph-mon[31673]: pgmap v234: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 120 MiB/s wr, 627 op/s 2024-03-21T10:46:13.307 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:12 smithi031 ceph-mon[31673]: osdmap e97: 12 total, 12 up, 12 in 2024-03-21T10:46:13.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:12 smithi070 ceph-mon[36330]: pgmap v234: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 120 MiB/s wr, 627 op/s 2024-03-21T10:46:13.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:12 smithi070 ceph-mon[36330]: osdmap e97: 12 total, 12 up, 12 in 2024-03-21T10:46:13.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.358+0000 7fbc79a00640 1 -- 172.21.15.31:0/3900662588 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc74109060 msgr2=0x7fbc74114d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:13.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.358+0000 7fbc79a00640 1 --2- 172.21.15.31:0/3900662588 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc74109060 0x7fbc74114d80 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c0077b0 tx=0x7fbc6c02cc30 comp rx=0 tx=0).stop 2024-03-21T10:46:13.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.358+0000 7fbc79a00640 1 -- 172.21.15.31:0/3900662588 shutdown_connections 2024-03-21T10:46:13.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.358+0000 7fbc79a00640 1 --2- 172.21.15.31:0/3900662588 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc74109060 0x7fbc74114d80 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.358+0000 7fbc79a00640 1 --2- 172.21.15.31:0/3900662588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc741086a0 0x7fbc74108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.358+0000 7fbc79a00640 1 --2- 172.21.15.31:0/3900662588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc7410cef0 0x7fbc7410d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.358+0000 7fbc79a00640 1 -- 172.21.15.31:0/3900662588 >> 172.21.15.31:0/3900662588 conn(0x7fbc7406ca50 msgr2=0x7fbc7406ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:13.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.358+0000 7fbc79a00640 1 -- 172.21.15.31:0/3900662588 shutdown_connections 2024-03-21T10:46:13.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.358+0000 7fbc79a00640 1 -- 172.21.15.31:0/3900662588 wait complete. 2024-03-21T10:46:13.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.360+0000 7fbc79a00640 1 Processor -- start 2024-03-21T10:46:13.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.360+0000 7fbc79a00640 1 -- start start 2024-03-21T10:46:13.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.360+0000 7fbc79a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc741086a0 0x7fbc741147f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:13.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.360+0000 7fbc79a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc74109060 0x7fbc74114d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:13.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.360+0000 7fbc79a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc7410cef0 0x7fbc741aaae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:13.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.360+0000 7fbc79a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc74117030 con 0x7fbc7410cef0 2024-03-21T10:46:13.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.360+0000 7fbc79a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc74116eb0 con 0x7fbc74109060 2024-03-21T10:46:13.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.360+0000 7fbc79a00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc741171b0 con 0x7fbc741086a0 2024-03-21T10:46:13.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.361+0000 7fbc73e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc741086a0 0x7fbc741147f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.361+0000 7fbc73e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc741086a0 0x7fbc741147f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:49790/0 (socket says 172.21.15.31:49790) 2024-03-21T10:46:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.361+0000 7fbc73400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc74109060 0x7fbc74114d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.361+0000 7fbc73e00640 1 -- 172.21.15.31:0/1148924307 learned_addr learned my addr 172.21.15.31:0/1148924307 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.361+0000 7fbc79000640 1 --2- 172.21.15.31:0/1148924307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc7410cef0 0x7fbc741aaae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.361+0000 7fbc73e00640 1 -- 172.21.15.31:0/1148924307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc74109060 msgr2=0x7fbc74114d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.361+0000 7fbc73e00640 1 --2- 172.21.15.31:0/1148924307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc74109060 0x7fbc74114d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.361+0000 7fbc73e00640 1 -- 172.21.15.31:0/1148924307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc7410cef0 msgr2=0x7fbc741aaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.361+0000 7fbc73e00640 1 --2- 172.21.15.31:0/1148924307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc7410cef0 0x7fbc741aaae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.361+0000 7fbc73e00640 1 -- 172.21.15.31:0/1148924307 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc74110b30 con 0x7fbc741086a0 2024-03-21T10:46:13.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.362+0000 7fbc73e00640 1 --2- 172.21.15.31:0/1148924307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc741086a0 0x7fbc741147f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fbc6400e9f0 tx=0x7fbc6400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:13.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.362+0000 7fbc70c00640 1 -- 172.21.15.31:0/1148924307 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6400cd80 con 0x7fbc741086a0 2024-03-21T10:46:13.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.362+0000 7fbc70c00640 1 -- 172.21.15.31:0/1148924307 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc64010910 con 0x7fbc741086a0 2024-03-21T10:46:13.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.362+0000 7fbc70c00640 1 -- 172.21.15.31:0/1148924307 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc64018980 con 0x7fbc741086a0 2024-03-21T10:46:13.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.363+0000 7fbc79a00640 1 -- 172.21.15.31:0/1148924307 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc74110e20 con 0x7fbc741086a0 2024-03-21T10:46:13.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.363+0000 7fbc79a00640 1 -- 172.21.15.31:0/1148924307 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc74111320 con 0x7fbc741086a0 2024-03-21T10:46:13.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.366+0000 7fbc70c00640 1 -- 172.21.15.31:0/1148924307 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fbc64010ac0 con 0x7fbc741086a0 2024-03-21T10:46:13.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.366+0000 7fbc7dde3740 1 Processor -- start 2024-03-21T10:46:13.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.367+0000 7fbc7dde3740 1 -- start start 2024-03-21T10:46:13.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.367+0000 7fbc7dde3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c2403b1a0 0x558c2403b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:13.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.368+0000 7fbc7dde3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c2403bac0 0x558c24044350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:13.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.368+0000 7fbc7dde3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c24044890 0x558c24046c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:13.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.368+0000 7fbc7dde3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c23ffd3b0 con 0x558c2403b1a0 2024-03-21T10:46:13.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.368+0000 7fbc7dde3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c23f9f830 con 0x558c2403bac0 2024-03-21T10:46:13.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.368+0000 7fbc7dde3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c23f485d0 con 0x558c24044890 2024-03-21T10:46:13.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.368+0000 7fbc70c00640 1 --2- 172.21.15.31:0/1148924307 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbc580a3070 0x7fbc580a5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:13.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.368+0000 7fbc73400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c24044890 0x558c24046c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:13.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.368+0000 7fbc70c00640 1 -- 172.21.15.31:0/1148924307 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9534+0+0 (secure 0 0 0) 0x7fbc640cff40 con 0x7fbc741086a0 2024-03-21T10:46:13.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.368+0000 7fbc73e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c2403bac0 0x558c24044350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:13.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.368+0000 7fbc73e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c2403bac0 0x558c24044350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:54830/0 (socket says 172.21.15.31:54830) 2024-03-21T10:46:13.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.368+0000 7fbc73400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c24044890 0x558c24046c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:49798/0 (socket says 172.21.15.31:49798) 2024-03-21T10:46:13.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.368+0000 7fbc73e00640 1 -- 172.21.15.31:0/1746134544 learned_addr learned my addr 172.21.15.31:0/1746134544 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:13.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.368+0000 7fbc79000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c2403b1a0 0x558c2403b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:13.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.369+0000 7fbc73e00640 1 -- 172.21.15.31:0/1746134544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c24044890 msgr2=0x558c24046c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:13.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.369+0000 7fbc73e00640 1 --2- 172.21.15.31:0/1746134544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c24044890 0x558c24046c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.369+0000 7fbc73e00640 1 -- 172.21.15.31:0/1746134544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c2403b1a0 msgr2=0x558c2403b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:13.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.369+0000 7fbc73e00640 1 --2- 172.21.15.31:0/1746134544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c2403b1a0 0x558c2403b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.369+0000 7fbc73e00640 1 -- 172.21.15.31:0/1746134544 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558c23f602c0 con 0x558c2403bac0 2024-03-21T10:46:13.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.369+0000 7fbc73e00640 1 --2- 172.21.15.31:0/1746134544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c2403bac0 0x558c24044350 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fbc6400e9c0 tx=0x7fbc6409bad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac3955db9f84e1cc server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:13.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.369+0000 7fbc79000640 1 --2- 172.21.15.31:0/1148924307 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbc580a3070 0x7fbc580a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:13.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.369+0000 7fbc69600640 1 -- 172.21.15.31:0/1746134544 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc640a0030 con 0x558c2403bac0 2024-03-21T10:46:13.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.369+0000 7fbc69600640 1 -- 172.21.15.31:0/1746134544 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc640a48f0 con 0x558c2403bac0 2024-03-21T10:46:13.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.369+0000 7fbc69600640 1 -- 172.21.15.31:0/1746134544 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6409b640 con 0x558c2403bac0 2024-03-21T10:46:13.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.370+0000 7fbc7dde3740 1 -- 172.21.15.31:0/1746134544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c2403bac0 msgr2=0x558c24044350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:13.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.370+0000 7fbc79000640 1 --2- 172.21.15.31:0/1148924307 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbc580a3070 0x7fbc580a5530 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c00d090 tx=0x7fbc6c02cc30 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:13.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.370+0000 7fbc7dde3740 1 --2- 172.21.15.31:0/1746134544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c2403bac0 0x558c24044350 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fbc6400e9c0 tx=0x7fbc6409bad0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.370+0000 7fbc7dde3740 1 -- 172.21.15.31:0/1746134544 shutdown_connections 2024-03-21T10:46:13.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.370+0000 7fbc7dde3740 1 --2- 172.21.15.31:0/1746134544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c24044890 0x558c24046c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.370+0000 7fbc7dde3740 1 --2- 172.21.15.31:0/1746134544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c2403bac0 0x558c24044350 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.370+0000 7fbc7dde3740 1 --2- 172.21.15.31:0/1746134544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c2403b1a0 0x558c2403b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.370+0000 7fbc7dde3740 1 -- 172.21.15.31:0/1746134544 >> 172.21.15.31:0/1746134544 conn(0x558c24035ac0 msgr2=0x558c24035ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:13.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.370+0000 7fbc7dde3740 1 -- 172.21.15.31:0/1746134544 shutdown_connections 2024-03-21T10:46:13.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.370+0000 7fbc7dde3740 1 -- 172.21.15.31:0/1746134544 wait complete. 2024-03-21T10:46:13.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.371+0000 7fbc7dde3740 1 -- start start 2024-03-21T10:46:13.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.371+0000 7fbc7dde3740 1 Processor -- start 2024-03-21T10:46:13.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.371+0000 7fbc7dde3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c2403b1a0 0x558c24041940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:13.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.372+0000 7fbc7dde3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c2403bac0 0x558c24041e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:13.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.372+0000 7fbc7dde3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c24044890 0x558c240423c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:13.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.372+0000 7fbc7dde3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c23f5ba20 con 0x558c2403bac0 2024-03-21T10:46:13.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.372+0000 7fbc7dde3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c23f9b8d0 con 0x558c24044890 2024-03-21T10:46:13.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.372+0000 7fbc7dde3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c240474c0 con 0x558c2403b1a0 2024-03-21T10:46:13.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.372+0000 7fbc73e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c2403b1a0 0x558c24041940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:13.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.372+0000 7fbc73e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c2403b1a0 0x558c24041940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:49808/0 (socket says 172.21.15.31:49808) 2024-03-21T10:46:13.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.372+0000 7fbc73e00640 1 -- 172.21.15.31:0/355041936 learned_addr learned my addr 172.21.15.31:0/355041936 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:13.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.372+0000 7fbc73400640 1 --2- 172.21.15.31:0/355041936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c2403bac0 0x558c24041e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:13.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.372+0000 7fbc79000640 1 --2- 172.21.15.31:0/355041936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c24044890 0x558c240423c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:13.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.372+0000 7fbc7dde3740 10 client.0 _ll_register_callbacks cb 0x558c23f981c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:46:13.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.372+0000 7fbc73e00640 1 -- 172.21.15.31:0/355041936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c24044890 msgr2=0x558c240423c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:13.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.372+0000 7fbc73e00640 1 --2- 172.21.15.31:0/355041936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c24044890 0x558c240423c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.372+0000 7fbc73e00640 1 -- 172.21.15.31:0/355041936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c2403bac0 msgr2=0x558c24041e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:13.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.372+0000 7fbc73e00640 1 --2- 172.21.15.31:0/355041936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c2403bac0 0x558c24041e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.372+0000 7fbc73e00640 1 -- 172.21.15.31:0/355041936 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558c23cb4210 con 0x558c2403b1a0 2024-03-21T10:46:13.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.372+0000 7fbc7dde3740 1 -- 172.21.15.31:0/355041936 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbc640a48f0 con 0x558c2403b1a0 2024-03-21T10:46:13.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.372+0000 7fbc73e00640 1 --2- 172.21.15.31:0/355041936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c2403b1a0 0x558c24041940 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fbc6409bc90 tx=0x7fbc64098050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fc040f5ced27465 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:13.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.372+0000 7fbc5e000640 10 client.27385 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:46:13.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.373+0000 7fbc5e000640 1 -- 172.21.15.31:0/355041936 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6407f050 con 0x558c2403b1a0 2024-03-21T10:46:13.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.373+0000 7fbc5e000640 1 -- 172.21.15.31:0/355041936 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9534+0+0 (secure 0 0 0) 0x7fbc64018d20 con 0x558c2403b1a0 2024-03-21T10:46:13.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.373+0000 7fbc5e000640 1 -- 172.21.15.31:0/355041936 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbc640b4070 con 0x558c2403b1a0 2024-03-21T10:46:13.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.373+0000 7fbc7dde3740 10 client.27385 fetch_fsmap learned FSMap version 26 2024-03-21T10:46:13.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.373+0000 7fbc7dde3740 1 -- 172.21.15.31:0/355041936 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({fsmap=26}) v3 -- 0x558c23d80620 con 0x558c2403b1a0 2024-03-21T10:46:13.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.373+0000 7fbc5e000640 1 -- 172.21.15.31:0/355041936 <== mon.1 v2:172.21.15.70:3300/0 4 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fbc64010430 con 0x558c2403b1a0 2024-03-21T10:46:13.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.373+0000 7fbc7dde3740 10 client.27385 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:46:13.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.373+0000 7fbc7dde3740 10 client.27385 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:46:13.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.373+0000 7fbc7dde3740 20 client.27385 populate_metadata read hostname 'smithi031' 2024-03-21T10:46:13.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.373+0000 7fbc7dde3740 1 --2- 172.21.15.31:0/355041936 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558c240546a0 0x558c24056a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:13.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.373+0000 7fbc7dde3740 4 client.27385 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:46:13.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.373+0000 7fbc7dde3740 1 -- 172.21.15.31:0/355041936 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558c23facd70 con 0x558c240546a0 2024-03-21T10:46:13.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.374+0000 7fbc73400640 1 --2- 172.21.15.31:0/355041936 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558c240546a0 0x558c24056a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:13.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.374+0000 7fbc73400640 1 --2- 172.21.15.31:0/355041936 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558c240546a0 0x558c24056a90 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d10582a43d79b40 server_cookie=2fab9f3a1a1768f6 in_seq=0 out_seq=0 2024-03-21T10:46:13.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.374+0000 7fbc5e000640 10 client.27385 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:13.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.375+0000 7fbc5e000640 1 -- 172.21.15.31:0/355041936 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558c23facd70 con 0x558c240546a0 2024-03-21T10:46:13.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.375+0000 7fbc5e000640 10 client.27385 handle_command_reply: tid=0 2024-03-21T10:46:13.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.375+0000 7fbc7dde3740 1 client.27385 shutdown 2024-03-21T10:46:13.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.375+0000 7fbc7dde3740 1 -- 172.21.15.31:0/355041936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c2403b1a0 msgr2=0x558c24041940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:13.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.375+0000 7fbc7dde3740 1 --2- 172.21.15.31:0/355041936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c2403b1a0 0x558c24041940 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fbc6409bc90 tx=0x7fbc64098050 comp rx=0 tx=0).stop 2024-03-21T10:46:13.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.375+0000 7fbc7dde3740 1 -- 172.21.15.31:0/355041936 shutdown_connections 2024-03-21T10:46:13.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.375+0000 7fbc7dde3740 1 --2- 172.21.15.31:0/355041936 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558c240546a0 0x558c24056a90 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.375+0000 7fbc7dde3740 1 --2- 172.21.15.31:0/355041936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c24044890 0x558c240423c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.375+0000 7fbc7dde3740 1 --2- 172.21.15.31:0/355041936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c2403bac0 0x558c24041e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.375+0000 7fbc7dde3740 1 --2- 172.21.15.31:0/355041936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c2403b1a0 0x558c24041940 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.375+0000 7fbc5e000640 0 client.27385 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:13.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.375+0000 7fbc7dde3740 1 -- 172.21.15.31:0/355041936 >> 172.21.15.31:0/355041936 conn(0x558c24035ac0 msgr2=0x558c24037810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:13.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.375+0000 7fbc7dde3740 1 -- 172.21.15.31:0/355041936 shutdown_connections 2024-03-21T10:46:13.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.375+0000 7fbc7dde3740 1 -- 172.21.15.31:0/355041936 wait complete. 2024-03-21T10:46:13.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.375+0000 7fbc7dde3740 20 client.27385 trim_cache size 0 max 16384 2024-03-21T10:46:13.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.390+0000 7fbc7dde3740 1 Processor -- start 2024-03-21T10:46:13.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.390+0000 7fbc7dde3740 1 -- start start 2024-03-21T10:46:13.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.391+0000 7fbc7dde3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c2403b1a0 0x558c240410f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:13.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.391+0000 7fbc7dde3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c2403bac0 0x558c2403ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:13.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.391+0000 7fbc7dde3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c24044890 0x558c2403f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:13.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.391+0000 7fbc7dde3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc741c8420 con 0x558c24044890 2024-03-21T10:46:13.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.391+0000 7fbc7dde3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc741c5f20 con 0x558c2403bac0 2024-03-21T10:46:13.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.391+0000 7fbc7dde3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc741c3fe0 con 0x558c2403b1a0 2024-03-21T10:46:13.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.391+0000 7fbc73e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c2403b1a0 0x558c240410f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:13.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.391+0000 7fbc73400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c2403bac0 0x558c2403ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:13.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.391+0000 7fbc73400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c2403bac0 0x558c2403ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:54846/0 (socket says 172.21.15.31:54846) 2024-03-21T10:46:13.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.391+0000 7fbc73e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c2403b1a0 0x558c240410f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:49812/0 (socket says 172.21.15.31:49812) 2024-03-21T10:46:13.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.391+0000 7fbc73400640 1 -- 172.21.15.31:0/1382078539 learned_addr learned my addr 172.21.15.31:0/1382078539 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:13.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.391+0000 7fbc79000640 1 --2- 172.21.15.31:0/1382078539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c24044890 0x558c2403f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:13.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.391+0000 7fbc73400640 1 -- 172.21.15.31:0/1382078539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c2403b1a0 msgr2=0x558c240410f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:13.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.391+0000 7fbc73400640 1 --2- 172.21.15.31:0/1382078539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c2403b1a0 0x558c240410f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.391+0000 7fbc73400640 1 -- 172.21.15.31:0/1382078539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c24044890 msgr2=0x558c2403f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:13.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.391+0000 7fbc73400640 1 --2- 172.21.15.31:0/1382078539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c24044890 0x558c2403f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.391+0000 7fbc73400640 1 -- 172.21.15.31:0/1382078539 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558c2403c020 con 0x558c2403bac0 2024-03-21T10:46:13.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.391+0000 7fbc73e00640 1 --2- 172.21.15.31:0/1382078539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c2403b1a0 0x558c240410f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:46:13.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.392+0000 7fbc73400640 1 --2- 172.21.15.31:0/1382078539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c2403bac0 0x558c2403ebc0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fbc74112040 tx=0x7fbc6000afd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d82fa66ee006ce1 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:13.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.392+0000 7fbc68c00640 1 -- 172.21.15.31:0/1382078539 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc60004020 con 0x558c2403bac0 2024-03-21T10:46:13.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.392+0000 7fbc68c00640 1 -- 172.21.15.31:0/1382078539 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc60023070 con 0x558c2403bac0 2024-03-21T10:46:13.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.392+0000 7fbc68c00640 1 -- 172.21.15.31:0/1382078539 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc60010850 con 0x558c2403bac0 2024-03-21T10:46:13.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.393+0000 7fbc7dde3740 1 -- 172.21.15.31:0/1382078539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c2403bac0 msgr2=0x558c2403ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:13.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.393+0000 7fbc7dde3740 1 --2- 172.21.15.31:0/1382078539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c2403bac0 0x558c2403ebc0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fbc74112040 tx=0x7fbc6000afd0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.393+0000 7fbc7dde3740 1 -- 172.21.15.31:0/1382078539 shutdown_connections 2024-03-21T10:46:13.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.393+0000 7fbc7dde3740 1 --2- 172.21.15.31:0/1382078539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c24044890 0x558c2403f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.393+0000 7fbc7dde3740 1 --2- 172.21.15.31:0/1382078539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c2403bac0 0x558c2403ebc0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.393+0000 7fbc7dde3740 1 --2- 172.21.15.31:0/1382078539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c2403b1a0 0x558c240410f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.393+0000 7fbc7dde3740 1 -- 172.21.15.31:0/1382078539 >> 172.21.15.31:0/1382078539 conn(0x558c24035ac0 msgr2=0x558c240377a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:13.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.393+0000 7fbc7dde3740 1 -- 172.21.15.31:0/1382078539 shutdown_connections 2024-03-21T10:46:13.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.393+0000 7fbc7dde3740 1 -- 172.21.15.31:0/1382078539 wait complete. 2024-03-21T10:46:13.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.393+0000 7fbc7dde3740 1 -- start start 2024-03-21T10:46:13.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.393+0000 7fbc7dde3740 1 Processor -- start 2024-03-21T10:46:13.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.394+0000 7fbc7dde3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c2403b1a0 0x558c240599d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:13.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.394+0000 7fbc7dde3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c2403bac0 0x558c24056d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:13.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.394+0000 7fbc7dde3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c24044890 0x558c240574a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:13.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.394+0000 7fbc7dde3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc741c45e0 con 0x558c24044890 2024-03-21T10:46:13.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.394+0000 7fbc7dde3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc741c58c0 con 0x558c2403b1a0 2024-03-21T10:46:13.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.394+0000 7fbc7dde3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc741c4c00 con 0x558c2403bac0 2024-03-21T10:46:13.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.394+0000 7fbc7dde3740 10 client.0 _ll_register_callbacks cb 0x7fbc6000a590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:46:13.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.394+0000 7fbc73e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c2403b1a0 0x558c240599d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:13.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.394+0000 7fbc73400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c2403bac0 0x558c24056d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:13.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.394+0000 7fbc73e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c2403b1a0 0x558c240599d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:54860/0 (socket says 172.21.15.31:54860) 2024-03-21T10:46:13.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.394+0000 7fbc73e00640 1 -- 172.21.15.31:0/2682716821 learned_addr learned my addr 172.21.15.31:0/2682716821 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:13.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.394+0000 7fbc79000640 1 --2- 172.21.15.31:0/2682716821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c24044890 0x558c240574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:13.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.395+0000 7fbc73e00640 1 -- 172.21.15.31:0/2682716821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c2403bac0 msgr2=0x558c24056d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:13.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.395+0000 7fbc73e00640 1 --2- 172.21.15.31:0/2682716821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c2403bac0 0x558c24056d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.395+0000 7fbc73e00640 1 -- 172.21.15.31:0/2682716821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c24044890 msgr2=0x558c240574a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:13.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.395+0000 7fbc73e00640 1 --2- 172.21.15.31:0/2682716821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c24044890 0x558c240574a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.395+0000 7fbc73e00640 1 -- 172.21.15.31:0/2682716821 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558c240aebd0 con 0x558c2403b1a0 2024-03-21T10:46:13.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.395+0000 7fbc73e00640 1 --2- 172.21.15.31:0/2682716821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c2403b1a0 0x558c240599d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fbc64009230 tx=0x7fbc6409bad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=179345c1ad44c9ea server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:13.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.395+0000 7fbc5e000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:46:13.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.395+0000 7fbc5e000640 1 -- 172.21.15.31:0/2682716821 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6407f040 con 0x558c2403b1a0 2024-03-21T10:46:13.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.395+0000 7fbc5e000640 1 -- 172.21.15.31:0/2682716821 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9534+0+0 (secure 0 0 0) 0x7fbc64096b00 con 0x558c2403b1a0 2024-03-21T10:46:13.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.395+0000 7fbc7dde3740 1 -- 172.21.15.31:0/2682716821 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbc60023070 con 0x558c2403b1a0 2024-03-21T10:46:13.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.395+0000 7fbc5e000640 1 -- 172.21.15.31:0/2682716821 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbc640bc0c0 con 0x558c2403b1a0 2024-03-21T10:46:13.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.395+0000 7fbc7dde3740 10 client.27470 fetch_fsmap learned FSMap version 26 2024-03-21T10:46:13.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.395+0000 7fbc7dde3740 1 -- 172.21.15.31:0/2682716821 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x558c240afd80 con 0x558c2403b1a0 2024-03-21T10:46:13.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.396+0000 7fbc5e000640 1 -- 172.21.15.31:0/2682716821 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc60023070 con 0x558c2403b1a0 2024-03-21T10:46:13.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.396+0000 7fbc5e000640 1 -- 172.21.15.31:0/2682716821 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fbc640a11c0 con 0x558c2403b1a0 2024-03-21T10:46:13.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.396+0000 7fbc7dde3740 10 client.27470 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:46:13.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.396+0000 7fbc7dde3740 10 client.27470 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:46:13.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.396+0000 7fbc7dde3740 20 client.27470 populate_metadata read hostname 'smithi031' 2024-03-21T10:46:13.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.396+0000 7fbc7dde3740 1 --2- 172.21.15.31:0/2682716821 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558c240b0160 0x558c240b2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:13.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.396+0000 7fbc7dde3740 4 client.27470 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:46:13.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.396+0000 7fbc7dde3740 1 -- 172.21.15.31:0/2682716821 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558c23fa70c0 con 0x558c240b0160 2024-03-21T10:46:13.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.396+0000 7fbc73400640 1 --2- 172.21.15.31:0/2682716821 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558c240b0160 0x558c240b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:13.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.397+0000 7fbc73400640 1 --2- 172.21.15.31:0/2682716821 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558c240b0160 0x558c240b2550 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd4dd7c60deec8d8 server_cookie=4ea7bc2a11c7396b in_seq=0 out_seq=0 2024-03-21T10:46:13.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.397+0000 7fbc5e000640 10 client.27470 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:13.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.412+0000 7fbc5e000640 1 -- 172.21.15.31:0/2682716821 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558c23fa70c0 con 0x558c240b0160 2024-03-21T10:46:13.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.412+0000 7fbc5e000640 10 client.27470 handle_command_reply: tid=0 2024-03-21T10:46:13.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.412+0000 7fbc7dde3740 1 client.27470 shutdown 2024-03-21T10:46:13.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.413+0000 7fbc7dde3740 1 -- 172.21.15.31:0/2682716821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c2403b1a0 msgr2=0x558c240599d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:13.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.413+0000 7fbc7dde3740 1 --2- 172.21.15.31:0/2682716821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c2403b1a0 0x558c240599d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fbc64009230 tx=0x7fbc6409bad0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.413+0000 7fbc7dde3740 1 -- 172.21.15.31:0/2682716821 shutdown_connections 2024-03-21T10:46:13.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.413+0000 7fbc7dde3740 1 --2- 172.21.15.31:0/2682716821 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558c240b0160 0x558c240b2550 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.413+0000 7fbc7dde3740 1 --2- 172.21.15.31:0/2682716821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c24044890 0x558c240574a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.413+0000 7fbc7dde3740 1 --2- 172.21.15.31:0/2682716821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558c2403bac0 0x558c24056d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.413+0000 7fbc7dde3740 1 --2- 172.21.15.31:0/2682716821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558c2403b1a0 0x558c240599d0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.413+0000 7fbc7dde3740 1 -- 172.21.15.31:0/2682716821 >> 172.21.15.31:0/2682716821 conn(0x558c24035ac0 msgr2=0x558c240420f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:13.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.413+0000 7fbc5e000640 0 client.27470 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:13.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.414+0000 7fbc7dde3740 1 -- 172.21.15.31:0/2682716821 shutdown_connections 2024-03-21T10:46:13.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.414+0000 7fbc7dde3740 1 -- 172.21.15.31:0/2682716821 wait complete. 2024-03-21T10:46:13.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.414+0000 7fbc7dde3740 20 client.27470 trim_cache size 0 max 16384 2024-03-21T10:46:13.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:46:13.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:46:13.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:46:13.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:46:13.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.415+0000 7fbc79a00640 1 -- 172.21.15.31:0/1148924307 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbc580a3070 msgr2=0x7fbc580a5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:13.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.415+0000 7fbc79a00640 1 --2- 172.21.15.31:0/1148924307 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbc580a3070 0x7fbc580a5530 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c00d090 tx=0x7fbc6c02cc30 comp rx=0 tx=0).stop 2024-03-21T10:46:13.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.416+0000 7fbc79a00640 1 -- 172.21.15.31:0/1148924307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc741086a0 msgr2=0x7fbc741147f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:13.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.416+0000 7fbc79a00640 1 --2- 172.21.15.31:0/1148924307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc741086a0 0x7fbc741147f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fbc6400e9f0 tx=0x7fbc6400eeb0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.416+0000 7fbc79a00640 1 -- 172.21.15.31:0/1148924307 shutdown_connections 2024-03-21T10:46:13.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.416+0000 7fbc79a00640 1 --2- 172.21.15.31:0/1148924307 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fbc580a3070 0x7fbc580a5530 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.416+0000 7fbc79a00640 1 --2- 172.21.15.31:0/1148924307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc7410cef0 0x7fbc741aaae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.416+0000 7fbc79a00640 1 --2- 172.21.15.31:0/1148924307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc74109060 0x7fbc74114d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.416+0000 7fbc79a00640 1 --2- 172.21.15.31:0/1148924307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc741086a0 0x7fbc741147f0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:13.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.416+0000 7fbc79a00640 1 -- 172.21.15.31:0/1148924307 >> 172.21.15.31:0/1148924307 conn(0x7fbc7406ca50 msgr2=0x7fbc7410bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:13.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.419+0000 7fbc79a00640 1 -- 172.21.15.31:0/1148924307 shutdown_connections 2024-03-21T10:46:13.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:13.419+0000 7fbc79a00640 1 -- 172.21.15.31:0/1148924307 wait complete. 2024-03-21T10:46:13.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:12 smithi040 ceph-mon[36154]: pgmap v234: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 120 MiB/s wr, 627 op/s 2024-03-21T10:46:13.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:12 smithi040 ceph-mon[36154]: osdmap e97: 12 total, 12 up, 12 in 2024-03-21T10:46:15.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:14 smithi031 ceph-mon[31673]: pgmap v236: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 136 MiB/s wr, 576 op/s 2024-03-21T10:46:15.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:14 smithi070 ceph-mon[36330]: pgmap v236: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 136 MiB/s wr, 576 op/s 2024-03-21T10:46:15.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:14 smithi040 ceph-mon[36154]: pgmap v236: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 136 MiB/s wr, 576 op/s 2024-03-21T10:46:17.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:17 smithi040 ceph-mon[36154]: pgmap v237: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 121 MiB/s wr, 284 op/s 2024-03-21T10:46:17.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:17 smithi040 ceph-mon[36154]: osdmap e98: 12 total, 12 up, 12 in 2024-03-21T10:46:17.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:17 smithi031 ceph-mon[31673]: pgmap v237: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 121 MiB/s wr, 284 op/s 2024-03-21T10:46:17.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:17 smithi031 ceph-mon[31673]: osdmap e98: 12 total, 12 up, 12 in 2024-03-21T10:46:17.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:17 smithi070 ceph-mon[36330]: pgmap v237: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 121 MiB/s wr, 284 op/s 2024-03-21T10:46:17.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:17 smithi070 ceph-mon[36330]: osdmap e98: 12 total, 12 up, 12 in 2024-03-21T10:46:19.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:19 smithi040 ceph-mon[36154]: pgmap v239: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 113 MiB/s wr, 234 op/s 2024-03-21T10:46:19.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:19 smithi031 ceph-mon[31673]: pgmap v239: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 113 MiB/s wr, 234 op/s 2024-03-21T10:46:19.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:19 smithi070 ceph-mon[36330]: pgmap v239: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 113 MiB/s wr, 234 op/s 2024-03-21T10:46:21.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:21 smithi040 ceph-mon[36154]: pgmap v240: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 295 MiB/s wr, 942 op/s 2024-03-21T10:46:21.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:21 smithi031 ceph-mon[31673]: pgmap v240: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 295 MiB/s wr, 942 op/s 2024-03-21T10:46:21.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:21 smithi070 ceph-mon[36330]: pgmap v240: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 295 MiB/s wr, 942 op/s 2024-03-21T10:46:23.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:23 smithi040 ceph-mon[36154]: pgmap v241: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 245 MiB/s wr, 787 op/s 2024-03-21T10:46:23.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:23 smithi031 ceph-mon[31673]: pgmap v241: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 245 MiB/s wr, 787 op/s 2024-03-21T10:46:23.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:23 smithi070 ceph-mon[36330]: pgmap v241: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 245 MiB/s wr, 787 op/s 2024-03-21T10:46:25.621 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:46:25.622 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:46:25.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:25 smithi040 ceph-mon[36154]: pgmap v242: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 242 MiB/s wr, 823 op/s 2024-03-21T10:46:25.712 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:25 smithi031 ceph-mon[31673]: pgmap v242: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 242 MiB/s wr, 823 op/s 2024-03-21T10:46:25.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:25 smithi070 ceph-mon[36330]: pgmap v242: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 242 MiB/s wr, 823 op/s 2024-03-21T10:46:25.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.861+0000 7f6c11600640 1 -- 172.21.15.31:0/2726779862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c0c074360 msgr2=0x7f6c0c111f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:25.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.861+0000 7f6c11600640 1 --2- 172.21.15.31:0/2726779862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c0c074360 0x7f6c0c111f10 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4009900 tx=0x7f6bf402bcd0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.861+0000 7f6c11600640 1 -- 172.21.15.31:0/2726779862 shutdown_connections 2024-03-21T10:46:25.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.861+0000 7f6c11600640 1 --2- 172.21.15.31:0/2726779862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c0c112450 0x7f6c0c114840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.861+0000 7f6c11600640 1 --2- 172.21.15.31:0/2726779862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c0c074360 0x7f6c0c111f10 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.861+0000 7f6c11600640 1 --2- 172.21.15.31:0/2726779862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c0c073a40 0x7f6c0c073e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.861+0000 7f6c11600640 1 -- 172.21.15.31:0/2726779862 >> 172.21.15.31:0/2726779862 conn(0x7f6c0c0fcf50 msgr2=0x7f6c0c0ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:25.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.861+0000 7f6c11600640 1 -- 172.21.15.31:0/2726779862 shutdown_connections 2024-03-21T10:46:25.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.861+0000 7f6c11600640 1 -- 172.21.15.31:0/2726779862 wait complete. 2024-03-21T10:46:25.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.863+0000 7f6c11600640 1 Processor -- start 2024-03-21T10:46:25.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.863+0000 7f6c11600640 1 -- start start 2024-03-21T10:46:25.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.863+0000 7f6c11600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c0c073a40 0x7f6c0c19e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:25.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.863+0000 7f6c11600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c0c074360 0x7f6c0c19e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:25.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.863+0000 7f6c11600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c0c112450 0x7f6c0c1a2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:25.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.863+0000 7f6c11600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c117ea0 con 0x7f6c0c112450 2024-03-21T10:46:25.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.863+0000 7f6c11600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c117d20 con 0x7f6c0c073a40 2024-03-21T10:46:25.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.863+0000 7f6c11600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c118020 con 0x7f6c0c074360 2024-03-21T10:46:25.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.864+0000 7f6c03e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c0c074360 0x7f6c0c19e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:25.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.864+0000 7f6c0be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c0c073a40 0x7f6c0c19e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:25.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.864+0000 7f6c03e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c0c074360 0x7f6c0c19e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53776/0 (socket says 172.21.15.31:53776) 2024-03-21T10:46:25.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.864+0000 7f6c03e00640 1 -- 172.21.15.31:0/1964136391 learned_addr learned my addr 172.21.15.31:0/1964136391 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:25.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.864+0000 7f6c10c00640 1 --2- 172.21.15.31:0/1964136391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c0c112450 0x7f6c0c1a2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:25.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.864+0000 7f6c03e00640 1 -- 172.21.15.31:0/1964136391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c0c073a40 msgr2=0x7f6c0c19e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:25.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.864+0000 7f6c03e00640 1 --2- 172.21.15.31:0/1964136391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c0c073a40 0x7f6c0c19e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.864+0000 7f6c03e00640 1 -- 172.21.15.31:0/1964136391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c0c112450 msgr2=0x7f6c0c1a2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:25.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.864+0000 7f6c03e00640 1 --2- 172.21.15.31:0/1964136391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c0c112450 0x7f6c0c1a2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.864+0000 7f6c03e00640 1 -- 172.21.15.31:0/1964136391 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c0c1a33b0 con 0x7f6c0c074360 2024-03-21T10:46:25.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.864+0000 7f6c0be00640 1 --2- 172.21.15.31:0/1964136391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c0c073a40 0x7f6c0c19e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:46:25.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.864+0000 7f6c03e00640 1 --2- 172.21.15.31:0/1964136391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c0c074360 0x7f6c0c19e900 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4000c00 tx=0x7f6bf4004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:25.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.865+0000 7f6c09600640 1 -- 172.21.15.31:0/1964136391 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4042020 con 0x7f6c0c074360 2024-03-21T10:46:25.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.865+0000 7f6c09600640 1 -- 172.21.15.31:0/1964136391 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6bf40054e0 con 0x7f6c0c074360 2024-03-21T10:46:25.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.865+0000 7f6c09600640 1 -- 172.21.15.31:0/1964136391 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4003af0 con 0x7f6c0c074360 2024-03-21T10:46:25.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.865+0000 7f6c11600640 1 -- 172.21.15.31:0/1964136391 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c0c1a3640 con 0x7f6c0c074360 2024-03-21T10:46:25.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.865+0000 7f6c11600640 1 -- 172.21.15.31:0/1964136391 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c0c1aaf30 con 0x7f6c0c074360 2024-03-21T10:46:25.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.868+0000 7f6c09600640 1 -- 172.21.15.31:0/1964136391 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f6bf4045020 con 0x7f6c0c074360 2024-03-21T10:46:25.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.868+0000 7f6c15814740 1 Processor -- start 2024-03-21T10:46:25.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.868+0000 7f6c15814740 1 -- start start 2024-03-21T10:46:25.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.869+0000 7f6c15814740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e59dcb2dd0 0x55e59dcb1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:25.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.869+0000 7f6c15814740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e59dcb20f0 0x55e59dcb2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:25.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.869+0000 7f6c15814740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e59dcbb460 0x55e59dcbd850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:25.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.869+0000 7f6c15814740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e59dc16250 con 0x55e59dcbb460 2024-03-21T10:46:25.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.869+0000 7f6c15814740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e59dbbea30 con 0x55e59dcb2dd0 2024-03-21T10:46:25.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.869+0000 7f6c15814740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e59dc122f0 con 0x55e59dcb20f0 2024-03-21T10:46:25.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.869+0000 7f6c10c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e59dcb2dd0 0x55e59dcb1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:25.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.869+0000 7f6c10c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e59dcb2dd0 0x55e59dcb1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38506/0 (socket says 172.21.15.31:38506) 2024-03-21T10:46:25.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.869+0000 7f6c10c00640 1 -- 172.21.15.31:0/1152717481 learned_addr learned my addr 172.21.15.31:0/1152717481 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:25.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.869+0000 7f6c0be00640 1 --2- 172.21.15.31:0/1152717481 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e59dcb20f0 0x55e59dcb2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:25.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.869+0000 7f6c03e00640 1 --2- 172.21.15.31:0/1152717481 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e59dcbb460 0x55e59dcbd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:25.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.870+0000 7f6c10c00640 1 -- 172.21.15.31:0/1152717481 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e59dcb20f0 msgr2=0x55e59dcb2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:25.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.870+0000 7f6c10c00640 1 --2- 172.21.15.31:0/1152717481 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e59dcb20f0 0x55e59dcb2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.870+0000 7f6c10c00640 1 -- 172.21.15.31:0/1152717481 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e59dcbb460 msgr2=0x55e59dcbd850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:25.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.870+0000 7f6c10c00640 1 --2- 172.21.15.31:0/1152717481 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e59dcbb460 0x55e59dcbd850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.870+0000 7f6c10c00640 1 -- 172.21.15.31:0/1152717481 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e59dba9690 con 0x55e59dcb2dd0 2024-03-21T10:46:25.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.870+0000 7f6c09600640 1 --2- 172.21.15.31:0/1964136391 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6bf00a3110 0x7f6bf00a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:25.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.870+0000 7f6c09600640 1 -- 172.21.15.31:0/1964136391 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f6bf40f1640 con 0x7f6c0c074360 2024-03-21T10:46:25.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.870+0000 7f6c10c00640 1 --2- 172.21.15.31:0/1152717481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e59dcb2dd0 0x55e59dcb1b20 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009f30 tx=0x7f6c040118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=204af66931c399c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:25.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.870+0000 7f6c0be00640 1 --2- 172.21.15.31:0/1964136391 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6bf00a3110 0x7f6bf00a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:25.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.870+0000 7f6c01200640 1 -- 172.21.15.31:0/1152717481 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c04004d20 con 0x55e59dcb2dd0 2024-03-21T10:46:25.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.870+0000 7f6c01200640 1 -- 172.21.15.31:0/1152717481 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6c04004050 con 0x55e59dcb2dd0 2024-03-21T10:46:25.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.870+0000 7f6c01200640 1 -- 172.21.15.31:0/1152717481 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c04005260 con 0x55e59dcb2dd0 2024-03-21T10:46:25.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.871+0000 7f6c15814740 1 -- 172.21.15.31:0/1152717481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e59dcb2dd0 msgr2=0x55e59dcb1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:25.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.871+0000 7f6c0be00640 1 --2- 172.21.15.31:0/1964136391 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6bf00a3110 0x7f6bf00a55d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x55e59dcb3320 tx=0x7f6bfc005f90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:25.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.871+0000 7f6c15814740 1 --2- 172.21.15.31:0/1152717481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e59dcb2dd0 0x55e59dcb1b20 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009f30 tx=0x7f6c040118d0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.871+0000 7f6c15814740 1 -- 172.21.15.31:0/1152717481 shutdown_connections 2024-03-21T10:46:25.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.871+0000 7f6c15814740 1 --2- 172.21.15.31:0/1152717481 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e59dcbb460 0x55e59dcbd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.871+0000 7f6c15814740 1 --2- 172.21.15.31:0/1152717481 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e59dcb20f0 0x55e59dcb2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.871+0000 7f6c15814740 1 --2- 172.21.15.31:0/1152717481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e59dcb2dd0 0x55e59dcb1b20 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.871+0000 7f6c15814740 1 -- 172.21.15.31:0/1152717481 >> 172.21.15.31:0/1152717481 conn(0x55e59dc79200 msgr2=0x55e59dcacb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:25.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.871+0000 7f6c15814740 1 -- 172.21.15.31:0/1152717481 shutdown_connections 2024-03-21T10:46:25.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.871+0000 7f6c15814740 1 -- 172.21.15.31:0/1152717481 wait complete. 2024-03-21T10:46:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.872+0000 7f6c15814740 1 -- start start 2024-03-21T10:46:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.872+0000 7f6c15814740 1 Processor -- start 2024-03-21T10:46:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.872+0000 7f6c15814740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e59dcb20f0 0x55e59dcbab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.872+0000 7f6c15814740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e59dcb2dd0 0x55e59dcb7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.872+0000 7f6c15814740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e59dcbb460 0x55e59dcb8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.873+0000 7f6c15814740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e59dcbe2d0 con 0x55e59dcb20f0 2024-03-21T10:46:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.873+0000 7f6c15814740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e59dbd1e80 con 0x55e59dcbb460 2024-03-21T10:46:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.873+0000 7f6c15814740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e59dcbe450 con 0x55e59dcb2dd0 2024-03-21T10:46:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.873+0000 7f6c15814740 10 client.0 _ll_register_callbacks cb 0x55e59dbe4130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:46:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.873+0000 7f6c0be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e59dcb20f0 0x55e59dcbab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.873+0000 7f6c10c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e59dcbb460 0x55e59dcb8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.873+0000 7f6c0be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e59dcb20f0 0x55e59dcbab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44816/0 (socket says 172.21.15.31:44816) 2024-03-21T10:46:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.873+0000 7f6c0be00640 1 -- 172.21.15.31:0/2404411714 learned_addr learned my addr 172.21.15.31:0/2404411714 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:25.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.873+0000 7f6c03e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e59dcb2dd0 0x55e59dcb7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:25.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.873+0000 7f6c10c00640 1 -- 172.21.15.31:0/2404411714 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e59dcb2dd0 msgr2=0x55e59dcb7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:25.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.873+0000 7f6c10c00640 1 --2- 172.21.15.31:0/2404411714 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e59dcb2dd0 0x55e59dcb7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.873+0000 7f6c10c00640 1 -- 172.21.15.31:0/2404411714 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e59dcb20f0 msgr2=0x55e59dcbab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:25.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.873+0000 7f6c10c00640 1 --2- 172.21.15.31:0/2404411714 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e59dcb20f0 0x55e59dcbab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.873+0000 7f6c10c00640 1 -- 172.21.15.31:0/2404411714 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e59d9f47d0 con 0x55e59dcbb460 2024-03-21T10:46:25.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.873+0000 7f6c0be00640 1 --2- 172.21.15.31:0/2404411714 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e59dcb20f0 0x55e59dcbab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:46:25.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.873+0000 7f6c03e00640 1 --2- 172.21.15.31:0/2404411714 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e59dcb2dd0 0x55e59dcb7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:46:25.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.874+0000 7f6c10c00640 1 --2- 172.21.15.31:0/2404411714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e59dcbb460 0x55e59dcb8430 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f6c04011dd0 tx=0x7f6c04003ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5f4e7676b37a240 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:25.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.874+0000 7f6bf9600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:46:25.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.874+0000 7f6bf9600640 1 -- 172.21.15.31:0/2404411714 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c04028020 con 0x55e59dcbb460 2024-03-21T10:46:25.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.874+0000 7f6bf9600640 1 -- 172.21.15.31:0/2404411714 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f6c04004c50 con 0x55e59dcbb460 2024-03-21T10:46:25.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.874+0000 7f6c15814740 1 -- 172.21.15.31:0/2404411714 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6c04004050 con 0x55e59dcbb460 2024-03-21T10:46:25.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.874+0000 7f6bf9600640 1 -- 172.21.15.31:0/2404411714 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c04033050 con 0x55e59dcbb460 2024-03-21T10:46:25.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.874+0000 7f6c15814740 10 client.27482 fetch_fsmap learned FSMap version 26 2024-03-21T10:46:25.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.874+0000 7f6c15814740 1 -- 172.21.15.31:0/2404411714 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55e59d92a3f0 con 0x55e59dcbb460 2024-03-21T10:46:25.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.874+0000 7f6bf9600640 1 -- 172.21.15.31:0/2404411714 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6c04004050 con 0x55e59dcbb460 2024-03-21T10:46:25.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.875+0000 7f6bf9600640 1 -- 172.21.15.31:0/2404411714 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f6c04023450 con 0x55e59dcbb460 2024-03-21T10:46:25.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.875+0000 7f6c15814740 10 client.27482 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:46:25.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.875+0000 7f6c15814740 10 client.27482 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:46:25.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.875+0000 7f6c15814740 20 client.27482 populate_metadata read hostname 'smithi031' 2024-03-21T10:46:25.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.875+0000 7f6c15814740 1 --2- 172.21.15.31:0/2404411714 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55e59dccb320 0x55e59dccd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:25.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.875+0000 7f6c15814740 4 client.27482 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:46:25.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.875+0000 7f6c15814740 1 -- 172.21.15.31:0/2404411714 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e59dc23790 con 0x55e59dccb320 2024-03-21T10:46:25.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.875+0000 7f6c0be00640 1 --2- 172.21.15.31:0/2404411714 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55e59dccb320 0x55e59dccd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:25.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.876+0000 7f6c0be00640 1 --2- 172.21.15.31:0/2404411714 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55e59dccb320 0x55e59dccd710 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=356b1fdd4246489f server_cookie=9d7ebc6e61117089 in_seq=0 out_seq=0 2024-03-21T10:46:25.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.876+0000 7f6bf9600640 10 client.27482 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:25.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.876+0000 7f6bf9600640 1 -- 172.21.15.31:0/2404411714 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e59dc23790 con 0x55e59dccb320 2024-03-21T10:46:25.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.876+0000 7f6bf9600640 10 client.27482 handle_command_reply: tid=0 2024-03-21T10:46:25.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.876+0000 7f6c15814740 1 client.27482 shutdown 2024-03-21T10:46:25.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.877+0000 7f6c15814740 1 -- 172.21.15.31:0/2404411714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e59dcbb460 msgr2=0x55e59dcb8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:25.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.877+0000 7f6c15814740 1 --2- 172.21.15.31:0/2404411714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e59dcbb460 0x55e59dcb8430 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f6c04011dd0 tx=0x7f6c04003ce0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.877+0000 7f6c15814740 1 -- 172.21.15.31:0/2404411714 shutdown_connections 2024-03-21T10:46:25.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.877+0000 7f6c15814740 1 --2- 172.21.15.31:0/2404411714 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55e59dccb320 0x55e59dccd710 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.877+0000 7f6c15814740 1 --2- 172.21.15.31:0/2404411714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e59dcbb460 0x55e59dcb8430 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.877+0000 7f6c15814740 1 --2- 172.21.15.31:0/2404411714 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e59dcb2dd0 0x55e59dcb7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.877+0000 7f6c15814740 1 --2- 172.21.15.31:0/2404411714 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e59dcb20f0 0x55e59dcbab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.877+0000 7f6c15814740 1 -- 172.21.15.31:0/2404411714 >> 172.21.15.31:0/2404411714 conn(0x55e59dc79200 msgr2=0x55e59dcae890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:25.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.877+0000 7f6bf9600640 0 client.27482 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:25.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.877+0000 7f6c15814740 1 -- 172.21.15.31:0/2404411714 shutdown_connections 2024-03-21T10:46:25.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.877+0000 7f6c15814740 1 -- 172.21.15.31:0/2404411714 wait complete. 2024-03-21T10:46:25.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.877+0000 7f6c15814740 20 client.27482 trim_cache size 0 max 16384 2024-03-21T10:46:25.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.895+0000 7f6c15814740 1 Processor -- start 2024-03-21T10:46:25.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.895+0000 7f6c15814740 1 -- start start 2024-03-21T10:46:25.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.895+0000 7f6c15814740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e59dcb20f0 0x55e59dcba620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:25.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.895+0000 7f6c15814740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e59dcb2dd0 0x55e59dcbab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:25.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.895+0000 7f6c15814740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e59dcbb460 0x55e59dcbc850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:25.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.895+0000 7f6c15814740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c1b7330 con 0x55e59dcb2dd0 2024-03-21T10:46:25.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.895+0000 7f6c15814740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c1b4e30 con 0x55e59dcb20f0 2024-03-21T10:46:25.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.895+0000 7f6c15814740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c1b2ef0 con 0x55e59dcbb460 2024-03-21T10:46:25.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.896+0000 7f6c10c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e59dcbb460 0x55e59dcbc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:25.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.896+0000 7f6c0be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e59dcb20f0 0x55e59dcba620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:25.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.896+0000 7f6c10c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e59dcbb460 0x55e59dcbc850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53802/0 (socket says 172.21.15.31:53802) 2024-03-21T10:46:25.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.896+0000 7f6c10c00640 1 -- 172.21.15.31:0/575541882 learned_addr learned my addr 172.21.15.31:0/575541882 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:25.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.896+0000 7f6c03e00640 1 --2- 172.21.15.31:0/575541882 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e59dcb2dd0 0x55e59dcbab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:25.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.896+0000 7f6c10c00640 1 -- 172.21.15.31:0/575541882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e59dcb20f0 msgr2=0x55e59dcba620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:25.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.896+0000 7f6c10c00640 1 --2- 172.21.15.31:0/575541882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e59dcb20f0 0x55e59dcba620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.896+0000 7f6c10c00640 1 -- 172.21.15.31:0/575541882 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e59dcb2dd0 msgr2=0x55e59dcbab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:25.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.896+0000 7f6c10c00640 1 --2- 172.21.15.31:0/575541882 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e59dcb2dd0 0x55e59dcbab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.896+0000 7f6c10c00640 1 -- 172.21.15.31:0/575541882 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e59d8ea0f0 con 0x55e59dcbb460 2024-03-21T10:46:25.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.896+0000 7f6c10c00640 1 --2- 172.21.15.31:0/575541882 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e59dcbb460 0x55e59dcbc850 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f6c04011e80 tx=0x7f6c04011eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cac2a811b848ea07 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:25.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.896+0000 7f6bfbe00640 1 -- 172.21.15.31:0/575541882 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c04028040 con 0x55e59dcbb460 2024-03-21T10:46:25.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.896+0000 7f6bfbe00640 1 -- 172.21.15.31:0/575541882 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6c04009e80 con 0x55e59dcbb460 2024-03-21T10:46:25.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.896+0000 7f6bfbe00640 1 -- 172.21.15.31:0/575541882 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c04023450 con 0x55e59dcbb460 2024-03-21T10:46:25.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.897+0000 7f6c15814740 1 -- 172.21.15.31:0/575541882 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e59dcbb460 msgr2=0x55e59dcbc850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.897+0000 7f6c15814740 1 --2- 172.21.15.31:0/575541882 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e59dcbb460 0x55e59dcbc850 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f6c04011e80 tx=0x7f6c04011eb0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.897+0000 7f6c15814740 1 -- 172.21.15.31:0/575541882 shutdown_connections 2024-03-21T10:46:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.897+0000 7f6c15814740 1 --2- 172.21.15.31:0/575541882 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e59dcbb460 0x55e59dcbc850 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.897+0000 7f6c15814740 1 --2- 172.21.15.31:0/575541882 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e59dcb2dd0 0x55e59dcbab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.897+0000 7f6c15814740 1 --2- 172.21.15.31:0/575541882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e59dcb20f0 0x55e59dcba620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.897+0000 7f6c15814740 1 -- 172.21.15.31:0/575541882 >> 172.21.15.31:0/575541882 conn(0x55e59dc79200 msgr2=0x55e59dcac670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.897+0000 7f6c15814740 1 -- 172.21.15.31:0/575541882 shutdown_connections 2024-03-21T10:46:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.897+0000 7f6c15814740 1 -- 172.21.15.31:0/575541882 wait complete. 2024-03-21T10:46:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.898+0000 7f6c15814740 1 -- start start 2024-03-21T10:46:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.898+0000 7f6c15814740 1 Processor -- start 2024-03-21T10:46:25.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.898+0000 7f6c15814740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e59dcb20f0 0x55e59dcbcd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:25.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.898+0000 7f6c15814740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e59dcb2dd0 0x55e59dcb5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:25.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.898+0000 7f6c15814740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e59dcbb460 0x55e59dcb56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:25.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.898+0000 7f6c15814740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c1b34f0 con 0x55e59dcbb460 2024-03-21T10:46:25.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.898+0000 7f6c15814740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c1b47d0 con 0x55e59dcb20f0 2024-03-21T10:46:25.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.898+0000 7f6c15814740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c1b3b10 con 0x55e59dcb2dd0 2024-03-21T10:46:25.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.898+0000 7f6c15814740 10 client.0 _ll_register_callbacks cb 0x7f6bfc009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:46:25.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.899+0000 7f6c0be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e59dcb20f0 0x55e59dcbcd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:25.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.899+0000 7f6c0be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e59dcb20f0 0x55e59dcbcd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38540/0 (socket says 172.21.15.31:38540) 2024-03-21T10:46:25.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.899+0000 7f6c0be00640 1 -- 172.21.15.31:0/922860638 learned_addr learned my addr 172.21.15.31:0/922860638 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:25.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.899+0000 7f6c03e00640 1 --2- 172.21.15.31:0/922860638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e59dcb2dd0 0x55e59dcb5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:25.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.899+0000 7f6c10c00640 1 --2- 172.21.15.31:0/922860638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e59dcbb460 0x55e59dcb56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:25.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.899+0000 7f6c0be00640 1 -- 172.21.15.31:0/922860638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e59dcb2dd0 msgr2=0x55e59dcb5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:25.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.899+0000 7f6c0be00640 1 --2- 172.21.15.31:0/922860638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e59dcb2dd0 0x55e59dcb5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.899+0000 7f6c0be00640 1 -- 172.21.15.31:0/922860638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e59dcbb460 msgr2=0x55e59dcb56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:25.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.899+0000 7f6c0be00640 1 --2- 172.21.15.31:0/922860638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e59dcbb460 0x55e59dcb56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.899+0000 7f6c0be00640 1 -- 172.21.15.31:0/922860638 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e59dcb5eb0 con 0x55e59dcb20f0 2024-03-21T10:46:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.899+0000 7f6c0be00640 1 --2- 172.21.15.31:0/922860638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e59dcb20f0 0x55e59dcbcd80 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x55e59dccdce0 tx=0x7f6bfc017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c960273669df984e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.899+0000 7f6bf9600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:46:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.900+0000 7f6bf9600640 1 -- 172.21.15.31:0/922860638 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc004020 con 0x55e59dcb20f0 2024-03-21T10:46:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.900+0000 7f6bf9600640 1 -- 172.21.15.31:0/922860638 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f6bfc0041c0 con 0x55e59dcb20f0 2024-03-21T10:46:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.900+0000 7f6c15814740 1 -- 172.21.15.31:0/922860638 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e59dc1dae0 con 0x55e59dcb20f0 2024-03-21T10:46:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.900+0000 7f6bf9600640 1 -- 172.21.15.31:0/922860638 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6bfc038050 con 0x55e59dcb20f0 2024-03-21T10:46:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.900+0000 7f6c15814740 10 client.27488 fetch_fsmap learned FSMap version 26 2024-03-21T10:46:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.900+0000 7f6c15814740 1 -- 172.21.15.31:0/922860638 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55e59dd269f0 con 0x55e59dcb20f0 2024-03-21T10:46:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.900+0000 7f6bf9600640 1 -- 172.21.15.31:0/922860638 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e59dc1dae0 con 0x55e59dcb20f0 2024-03-21T10:46:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.900+0000 7f6bf9600640 1 -- 172.21.15.31:0/922860638 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f6bfc005370 con 0x55e59dcb20f0 2024-03-21T10:46:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.900+0000 7f6c15814740 10 client.27488 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:46:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.900+0000 7f6c15814740 10 client.27488 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:46:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.900+0000 7f6c15814740 20 client.27488 populate_metadata read hostname 'smithi031' 2024-03-21T10:46:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.900+0000 7f6c15814740 1 --2- 172.21.15.31:0/922860638 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55e59dd270c0 0x55e59dd294e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.900+0000 7f6c15814740 4 client.27488 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:46:25.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.900+0000 7f6c15814740 1 -- 172.21.15.31:0/922860638 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e59db9b3a0 con 0x55e59dd270c0 2024-03-21T10:46:25.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.901+0000 7f6c03e00640 1 --2- 172.21.15.31:0/922860638 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55e59dd270c0 0x55e59dd294e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:25.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.901+0000 7f6c03e00640 1 --2- 172.21.15.31:0/922860638 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55e59dd270c0 0x55e59dd294e0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9b93c11f94ca73 server_cookie=80cbb04c761a299f in_seq=0 out_seq=0 2024-03-21T10:46:25.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.901+0000 7f6bf9600640 10 client.27488 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:25.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.901+0000 7f6bf9600640 1 -- 172.21.15.31:0/922860638 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55e59db9b3a0 con 0x55e59dd270c0 2024-03-21T10:46:25.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.901+0000 7f6bf9600640 10 client.27488 handle_command_reply: tid=0 2024-03-21T10:46:25.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.901+0000 7f6c15814740 1 client.27488 shutdown 2024-03-21T10:46:25.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.901+0000 7f6c15814740 1 -- 172.21.15.31:0/922860638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e59dcb20f0 msgr2=0x55e59dcbcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:25.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.901+0000 7f6c15814740 1 --2- 172.21.15.31:0/922860638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e59dcb20f0 0x55e59dcbcd80 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x55e59dccdce0 tx=0x7f6bfc017000 comp rx=0 tx=0).stop 2024-03-21T10:46:25.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.902+0000 7f6c15814740 1 -- 172.21.15.31:0/922860638 shutdown_connections 2024-03-21T10:46:25.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.902+0000 7f6c15814740 1 --2- 172.21.15.31:0/922860638 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55e59dd270c0 0x55e59dd294e0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.902+0000 7f6c15814740 1 --2- 172.21.15.31:0/922860638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e59dcbb460 0x55e59dcb56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.902+0000 7f6c15814740 1 --2- 172.21.15.31:0/922860638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e59dcb2dd0 0x55e59dcb5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.902+0000 7f6bf9600640 0 client.27488 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:25.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.902+0000 7f6c15814740 1 --2- 172.21.15.31:0/922860638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e59dcb20f0 0x55e59dcbcd80 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.902+0000 7f6c15814740 1 -- 172.21.15.31:0/922860638 >> 172.21.15.31:0/922860638 conn(0x55e59dc79200 msgr2=0x55e59dcadf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:25.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.902+0000 7f6c15814740 1 -- 172.21.15.31:0/922860638 shutdown_connections 2024-03-21T10:46:25.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.902+0000 7f6c15814740 1 -- 172.21.15.31:0/922860638 wait complete. 2024-03-21T10:46:25.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.902+0000 7f6c15814740 20 client.27488 trim_cache size 0 max 16384 2024-03-21T10:46:25.903 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:46:25.903 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-03-21T10:46:25.903 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-03-21T10:46:25.903 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:46:25.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.902+0000 7f6c11600640 1 -- 172.21.15.31:0/1964136391 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6bf00a3110 msgr2=0x7f6bf00a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:25.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.903+0000 7f6c11600640 1 --2- 172.21.15.31:0/1964136391 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6bf00a3110 0x7f6bf00a55d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x55e59dcb3320 tx=0x7f6bfc005f90 comp rx=0 tx=0).stop 2024-03-21T10:46:25.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.903+0000 7f6c11600640 1 -- 172.21.15.31:0/1964136391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c0c074360 msgr2=0x7f6c0c19e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:25.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.903+0000 7f6c11600640 1 --2- 172.21.15.31:0/1964136391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c0c074360 0x7f6c0c19e900 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4000c00 tx=0x7f6bf4004360 comp rx=0 tx=0).stop 2024-03-21T10:46:25.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.903+0000 7f6c11600640 1 -- 172.21.15.31:0/1964136391 shutdown_connections 2024-03-21T10:46:25.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.903+0000 7f6c11600640 1 --2- 172.21.15.31:0/1964136391 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6bf00a3110 0x7f6bf00a55d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.903+0000 7f6c11600640 1 --2- 172.21.15.31:0/1964136391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c0c112450 0x7f6c0c1a2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.903+0000 7f6c11600640 1 --2- 172.21.15.31:0/1964136391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c0c074360 0x7f6c0c19e900 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.903+0000 7f6c11600640 1 --2- 172.21.15.31:0/1964136391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c0c073a40 0x7f6c0c19e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:25.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.903+0000 7f6c11600640 1 -- 172.21.15.31:0/1964136391 >> 172.21.15.31:0/1964136391 conn(0x7f6c0c0fcf50 msgr2=0x7f6c0c0fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:25.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.903+0000 7f6c11600640 1 -- 172.21.15.31:0/1964136391 shutdown_connections 2024-03-21T10:46:25.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:25.903+0000 7f6c11600640 1 -- 172.21.15.31:0/1964136391 wait complete. 2024-03-21T10:46:25.917 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-21T10:46:25.918 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:46:26.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.126+0000 7fdcdac00640 1 -- 172.21.15.31:0/4270054785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcd4073a40 msgr2=0x7fdcd4073e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.126+0000 7fdcdac00640 1 --2- 172.21.15.31:0/4270054785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcd4073a40 0x7fdcd4073e20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8009900 tx=0x7fdcc802bcd0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.126+0000 7fdcdac00640 1 -- 172.21.15.31:0/4270054785 shutdown_connections 2024-03-21T10:46:26.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.126+0000 7fdcdac00640 1 --2- 172.21.15.31:0/4270054785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdcd4112530 0x7fdcd4114920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.126+0000 7fdcdac00640 1 --2- 172.21.15.31:0/4270054785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdcd4074360 0x7fdcd4111ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.126+0000 7fdcdac00640 1 --2- 172.21.15.31:0/4270054785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcd4073a40 0x7fdcd4073e20 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.126+0000 7fdcdac00640 1 -- 172.21.15.31:0/4270054785 >> 172.21.15.31:0/4270054785 conn(0x7fdcd40fd070 msgr2=0x7fdcd40ff490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:26.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.126+0000 7fdcdac00640 1 -- 172.21.15.31:0/4270054785 shutdown_connections 2024-03-21T10:46:26.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.126+0000 7fdcdac00640 1 -- 172.21.15.31:0/4270054785 wait complete. 2024-03-21T10:46:26.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.127+0000 7fdcdac00640 1 Processor -- start 2024-03-21T10:46:26.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.127+0000 7fdcdac00640 1 -- start start 2024-03-21T10:46:26.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.127+0000 7fdcdac00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcd4073a40 0x7fdcd419e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.128+0000 7fdcdac00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdcd4074360 0x7fdcd419e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.128+0000 7fdcdac00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdcd4112530 0x7fdcd41a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.128+0000 7fdcdac00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd4117f80 con 0x7fdcd4073a40 2024-03-21T10:46:26.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.128+0000 7fdcdac00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd4117e00 con 0x7fdcd4074360 2024-03-21T10:46:26.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.128+0000 7fdcdac00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd4118100 con 0x7fdcd4112530 2024-03-21T10:46:26.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.128+0000 7fdcd8e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdcd4074360 0x7fdcd419e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.128+0000 7fdcda200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdcd4112530 0x7fdcd41a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.128+0000 7fdcd9800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcd4073a40 0x7fdcd419e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.128+0000 7fdcda200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdcd4112530 0x7fdcd41a2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53836/0 (socket says 172.21.15.31:53836) 2024-03-21T10:46:26.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.128+0000 7fdcda200640 1 -- 172.21.15.31:0/689623757 learned_addr learned my addr 172.21.15.31:0/689623757 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:26.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.128+0000 7fdcda200640 1 -- 172.21.15.31:0/689623757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdcd4074360 msgr2=0x7fdcd419e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.128+0000 7fdcda200640 1 --2- 172.21.15.31:0/689623757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdcd4074360 0x7fdcd419e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.128+0000 7fdcda200640 1 -- 172.21.15.31:0/689623757 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcd4073a40 msgr2=0x7fdcd419e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.128+0000 7fdcda200640 1 --2- 172.21.15.31:0/689623757 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcd4073a40 0x7fdcd419e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.128+0000 7fdcda200640 1 -- 172.21.15.31:0/689623757 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd41a3470 con 0x7fdcd4112530 2024-03-21T10:46:26.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.128+0000 7fdcd8e00640 1 --2- 172.21.15.31:0/689623757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdcd4074360 0x7fdcd419e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:46:26.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.129+0000 7fdcda200640 1 --2- 172.21.15.31:0/689623757 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdcd4112530 0x7fdcd41a2d40 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fdccc00eb40 tx=0x7fdccc00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:26.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.129+0000 7fdcd2000640 1 -- 172.21.15.31:0/689623757 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc004b20 con 0x7fdcd4112530 2024-03-21T10:46:26.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.129+0000 7fdcd2000640 1 -- 172.21.15.31:0/689623757 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdccc004cc0 con 0x7fdcd4112530 2024-03-21T10:46:26.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.129+0000 7fdcdac00640 1 -- 172.21.15.31:0/689623757 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd41a3760 con 0x7fdcd4112530 2024-03-21T10:46:26.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.129+0000 7fdcd2000640 1 -- 172.21.15.31:0/689623757 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc010850 con 0x7fdcd4112530 2024-03-21T10:46:26.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.129+0000 7fdcdac00640 1 -- 172.21.15.31:0/689623757 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd41aaff0 con 0x7fdcd4112530 2024-03-21T10:46:26.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.130+0000 7fdcdac00640 1 -- 172.21.15.31:0/689623757 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd404f470 con 0x7fdcd4112530 2024-03-21T10:46:26.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.134+0000 7fdcd2000640 1 -- 172.21.15.31:0/689623757 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fdccc005020 con 0x7fdcd4112530 2024-03-21T10:46:26.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.135+0000 7fdcd2000640 1 --2- 172.21.15.31:0/689623757 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fdcbc0a3080 0x7fdcbc0a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.135+0000 7fdcd2000640 1 -- 172.21.15.31:0/689623757 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fdccc0d02f0 con 0x7fdcd4112530 2024-03-21T10:46:26.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.135+0000 7fdcd9800640 1 --2- 172.21.15.31:0/689623757 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fdcbc0a3080 0x7fdcbc0a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.136+0000 7fdcd9800640 1 --2- 172.21.15.31:0/689623757 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fdcbc0a3080 0x7fdcbc0a5540 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8005ab0 tx=0x7fdcc8005a20 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:26.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.139+0000 7fdcd2000640 1 -- 172.21.15.31:0/689623757 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdccc09c9e0 con 0x7fdcd4112530 2024-03-21T10:46:26.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.397+0000 7fdcdac00640 1 -- 172.21.15.31:0/689623757 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdcd4104a10 con 0x7fdcd4112530 2024-03-21T10:46:26.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.398+0000 7fdcd2000640 1 -- 172.21.15.31:0/689623757 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+11384 (secure 0 0 0) 0x7fdccc09c2e0 con 0x7fdcd4112530 2024-03-21T10:46:26.396 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:46:26.396 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:42:31.182549+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24547,24596]},"id":1}]} 2024-03-21T10:46:26.396 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2024-03-21T10:46:26.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.401+0000 7fdcdac00640 1 -- 172.21.15.31:0/689623757 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fdcbc0a3080 msgr2=0x7fdcbc0a5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.401+0000 7fdcdac00640 1 --2- 172.21.15.31:0/689623757 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fdcbc0a3080 0x7fdcbc0a5540 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8005ab0 tx=0x7fdcc8005a20 comp rx=0 tx=0).stop 2024-03-21T10:46:26.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.401+0000 7fdcdac00640 1 -- 172.21.15.31:0/689623757 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdcd4112530 msgr2=0x7fdcd41a2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.401+0000 7fdcdac00640 1 --2- 172.21.15.31:0/689623757 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdcd4112530 0x7fdcd41a2d40 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fdccc00eb40 tx=0x7fdccc00c2d0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.401+0000 7fdcdac00640 1 -- 172.21.15.31:0/689623757 shutdown_connections 2024-03-21T10:46:26.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.401+0000 7fdcdac00640 1 --2- 172.21.15.31:0/689623757 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fdcbc0a3080 0x7fdcbc0a5540 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.401+0000 7fdcdac00640 1 --2- 172.21.15.31:0/689623757 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdcd4112530 0x7fdcd41a2d40 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.401+0000 7fdcdac00640 1 --2- 172.21.15.31:0/689623757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdcd4074360 0x7fdcd419e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.401+0000 7fdcdac00640 1 --2- 172.21.15.31:0/689623757 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcd4073a40 0x7fdcd419e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.401+0000 7fdcdac00640 1 -- 172.21.15.31:0/689623757 >> 172.21.15.31:0/689623757 conn(0x7fdcd40fd070 msgr2=0x7fdcd40fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:26.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.402+0000 7fdcdac00640 1 -- 172.21.15.31:0/689623757 shutdown_connections 2024-03-21T10:46:26.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.402+0000 7fdcdac00640 1 -- 172.21.15.31:0/689623757 wait complete. 2024-03-21T10:46:26.421 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:46:26.421 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-21T10:46:26.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.571+0000 7f4b5f600640 1 -- 172.21.15.31:0/1411717613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b581098a0 msgr2=0x7f4b58109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.571+0000 7f4b5f600640 1 --2- 172.21.15.31:0/1411717613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b581098a0 0x7f4b58109c80 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c009930 tx=0x7f4b4c02bc20 comp rx=0 tx=0).stop 2024-03-21T10:46:26.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.571+0000 7f4b5f600640 1 -- 172.21.15.31:0/1411717613 shutdown_connections 2024-03-21T10:46:26.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.571+0000 7f4b5f600640 1 --2- 172.21.15.31:0/1411717613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b58104250 0x7f4b58110760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.571+0000 7f4b5f600640 1 --2- 172.21.15.31:0/1411717613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b58103890 0x7f4b58103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.571+0000 7f4b5f600640 1 --2- 172.21.15.31:0/1411717613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b581098a0 0x7f4b58109c80 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.571+0000 7f4b5f600640 1 -- 172.21.15.31:0/1411717613 >> 172.21.15.31:0/1411717613 conn(0x7f4b580ff320 msgr2=0x7f4b58101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:26.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.571+0000 7f4b5f600640 1 -- 172.21.15.31:0/1411717613 shutdown_connections 2024-03-21T10:46:26.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.571+0000 7f4b5f600640 1 -- 172.21.15.31:0/1411717613 wait complete. 2024-03-21T10:46:26.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.572+0000 7f4b5f600640 1 Processor -- start 2024-03-21T10:46:26.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.573+0000 7f4b5f600640 1 -- start start 2024-03-21T10:46:26.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.573+0000 7f4b5f600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b58103890 0x7f4b5819a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.573+0000 7f4b5f600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b58104250 0x7f4b5819a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.573+0000 7f4b5f600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b581098a0 0x7f4b5819ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.573+0000 7f4b5f600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b58113cb0 con 0x7f4b581098a0 2024-03-21T10:46:26.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.573+0000 7f4b5f600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b58113b30 con 0x7f4b58103890 2024-03-21T10:46:26.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.573+0000 7f4b5f600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b58113e30 con 0x7f4b58104250 2024-03-21T10:46:26.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.573+0000 7f4b57e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b58104250 0x7f4b5819a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.573+0000 7f4b5cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b58103890 0x7f4b5819a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.573+0000 7f4b57e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b58104250 0x7f4b5819a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53844/0 (socket says 172.21.15.31:53844) 2024-03-21T10:46:26.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.573+0000 7f4b57e00640 1 -- 172.21.15.31:0/3333865859 learned_addr learned my addr 172.21.15.31:0/3333865859 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:26.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.573+0000 7f4b5cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b58103890 0x7f4b5819a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38572/0 (socket says 172.21.15.31:38572) 2024-03-21T10:46:26.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.573+0000 7f4b5d600640 1 --2- 172.21.15.31:0/3333865859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b581098a0 0x7f4b5819ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.574+0000 7f4b57e00640 1 -- 172.21.15.31:0/3333865859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b58103890 msgr2=0x7f4b5819a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.574+0000 7f4b57e00640 1 --2- 172.21.15.31:0/3333865859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b58103890 0x7f4b5819a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.574+0000 7f4b57e00640 1 -- 172.21.15.31:0/3333865859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b581098a0 msgr2=0x7f4b5819ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.574+0000 7f4b57e00640 1 --2- 172.21.15.31:0/3333865859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b581098a0 0x7f4b5819ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.574+0000 7f4b57e00640 1 -- 172.21.15.31:0/3333865859 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5819f180 con 0x7f4b58104250 2024-03-21T10:46:26.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.574+0000 7f4b5cc00640 1 --2- 172.21.15.31:0/3333865859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b58103890 0x7f4b5819a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:46:26.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.574+0000 7f4b57e00640 1 --2- 172.21.15.31:0/3333865859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b58104250 0x7f4b5819a6d0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4b4400d950 tx=0x7f4b4400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:26.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.574+0000 7f4b55600640 1 -- 172.21.15.31:0/3333865859 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b44017070 con 0x7f4b58104250 2024-03-21T10:46:26.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.574+0000 7f4b5f600640 1 -- 172.21.15.31:0/3333865859 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b5819f470 con 0x7f4b58104250 2024-03-21T10:46:26.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.574+0000 7f4b55600640 1 -- 172.21.15.31:0/3333865859 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b44009c30 con 0x7f4b58104250 2024-03-21T10:46:26.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.574+0000 7f4b55600640 1 -- 172.21.15.31:0/3333865859 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4400f620 con 0x7f4b58104250 2024-03-21T10:46:26.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.575+0000 7f4b5f600640 1 -- 172.21.15.31:0/3333865859 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b581a6d00 con 0x7f4b58104250 2024-03-21T10:46:26.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.578+0000 7f4b55600640 1 -- 172.21.15.31:0/3333865859 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f4b4401a070 con 0x7f4b58104250 2024-03-21T10:46:26.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.579+0000 7f4b55600640 1 --2- 172.21.15.31:0/3333865859 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4b400a3080 0x7f4b400a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.579+0000 7f4b62377740 1 Processor -- start 2024-03-21T10:46:26.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.579+0000 7f4b55600640 1 -- 172.21.15.31:0/3333865859 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f4b440cf5e0 con 0x7f4b58104250 2024-03-21T10:46:26.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.579+0000 7f4b62377740 1 -- start start 2024-03-21T10:46:26.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.579+0000 7f4b5cc00640 1 --2- 172.21.15.31:0/3333865859 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4b400a3080 0x7f4b400a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.580+0000 7f4b62377740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d41043efd0 0x55d41043f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.580+0000 7f4b62377740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d41043f8f0 0x55d410448180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.580+0000 7f4b62377740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4104486c0 0x55d41044aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.580+0000 7f4b62377740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4104013b0 con 0x55d41043f8f0 2024-03-21T10:46:26.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.580+0000 7f4b62377740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4103a3830 con 0x55d4104486c0 2024-03-21T10:46:26.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.580+0000 7f4b62377740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d41034c5d0 con 0x55d41043efd0 2024-03-21T10:46:26.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.580+0000 7f4b57e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d41043f8f0 0x55d410448180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.580+0000 7f4b57e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d41043f8f0 0x55d410448180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44872/0 (socket says 172.21.15.31:44872) 2024-03-21T10:46:26.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.580+0000 7f4b57e00640 1 -- 172.21.15.31:0/2847685244 learned_addr learned my addr 172.21.15.31:0/2847685244 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:26.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.580+0000 7f4b5cc00640 1 --2- 172.21.15.31:0/2847685244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d41043efd0 0x55d41043f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.580+0000 7f4b5d600640 1 --2- 172.21.15.31:0/2847685244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4104486c0 0x55d41044aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.581+0000 7f4b5d600640 1 -- 172.21.15.31:0/2847685244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d41043efd0 msgr2=0x55d41043f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:26.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.581+0000 7f4b5d600640 1 --2- 172.21.15.31:0/2847685244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d41043efd0 0x55d41043f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.581+0000 7f4b5d600640 1 -- 172.21.15.31:0/2847685244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d41043f8f0 msgr2=0x55d410448180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:26.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.581+0000 7f4b5d600640 1 --2- 172.21.15.31:0/2847685244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d41043f8f0 0x55d410448180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.581+0000 7f4b5d600640 1 -- 172.21.15.31:0/2847685244 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4103642c0 con 0x55d4104486c0 2024-03-21T10:46:26.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.581+0000 7f4b5cc00640 1 --2- 172.21.15.31:0/3333865859 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4b400a3080 0x7f4b400a5540 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c009900 tx=0x7f4b4c000bc0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:26.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.581+0000 7f4b5d600640 1 --2- 172.21.15.31:0/2847685244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4104486c0 0x55d41044aab0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f4b50009f00 tx=0x7f4b50011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0692cf7f06cfbe5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:26.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.581+0000 7f4b49c00640 1 -- 172.21.15.31:0/2847685244 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b50004d20 con 0x55d4104486c0 2024-03-21T10:46:26.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.581+0000 7f4b49c00640 1 -- 172.21.15.31:0/2847685244 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b50004050 con 0x55d4104486c0 2024-03-21T10:46:26.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.581+0000 7f4b49c00640 1 -- 172.21.15.31:0/2847685244 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b50005260 con 0x55d4104486c0 2024-03-21T10:46:26.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.582+0000 7f4b62377740 1 -- 172.21.15.31:0/2847685244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4104486c0 msgr2=0x55d41044aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.582+0000 7f4b62377740 1 --2- 172.21.15.31:0/2847685244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4104486c0 0x55d41044aab0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f4b50009f00 tx=0x7f4b50011810 comp rx=0 tx=0).stop 2024-03-21T10:46:26.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.582+0000 7f4b62377740 1 -- 172.21.15.31:0/2847685244 shutdown_connections 2024-03-21T10:46:26.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.582+0000 7f4b62377740 1 --2- 172.21.15.31:0/2847685244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4104486c0 0x55d41044aab0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.582+0000 7f4b62377740 1 --2- 172.21.15.31:0/2847685244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d41043f8f0 0x55d410448180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.582+0000 7f4b62377740 1 --2- 172.21.15.31:0/2847685244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d41043efd0 0x55d41043f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.582+0000 7f4b62377740 1 -- 172.21.15.31:0/2847685244 >> 172.21.15.31:0/2847685244 conn(0x55d4104398f0 msgr2=0x55d410439cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:26.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.583+0000 7f4b62377740 1 -- 172.21.15.31:0/2847685244 shutdown_connections 2024-03-21T10:46:26.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.583+0000 7f4b62377740 1 -- 172.21.15.31:0/2847685244 wait complete. 2024-03-21T10:46:26.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.583+0000 7f4b62377740 1 -- start start 2024-03-21T10:46:26.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.583+0000 7f4b62377740 1 Processor -- start 2024-03-21T10:46:26.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.584+0000 7f4b62377740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d41043efd0 0x55d410445760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.584+0000 7f4b62377740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41043f8f0 0x55d410445ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.584+0000 7f4b62377740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4104486c0 0x55d4104461e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.584+0000 7f4b62377740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d41035fa20 con 0x55d41043efd0 2024-03-21T10:46:26.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.584+0000 7f4b62377740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d41039f8d0 con 0x55d41043f8f0 2024-03-21T10:46:26.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.584+0000 7f4b62377740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d41044b2f0 con 0x55d4104486c0 2024-03-21T10:46:26.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.584+0000 7f4b62377740 10 client.0 _ll_register_callbacks cb 0x55d41039c1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:46:26.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.584+0000 7f4b5d600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4104486c0 0x55d4104461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.585+0000 7f4b57e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41043f8f0 0x55d410445ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.585+0000 7f4b5d600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4104486c0 0x55d4104461e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53856/0 (socket says 172.21.15.31:53856) 2024-03-21T10:46:26.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.585+0000 7f4b5d600640 1 -- 172.21.15.31:0/2128163316 learned_addr learned my addr 172.21.15.31:0/2128163316 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.585+0000 7f4b5cc00640 1 --2- 172.21.15.31:0/2128163316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d41043efd0 0x55d410445760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.585+0000 7f4b57e00640 1 -- 172.21.15.31:0/2128163316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4104486c0 msgr2=0x55d4104461e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.585+0000 7f4b57e00640 1 --2- 172.21.15.31:0/2128163316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4104486c0 0x55d4104461e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.585+0000 7f4b57e00640 1 -- 172.21.15.31:0/2128163316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d41043efd0 msgr2=0x55d410445760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.585+0000 7f4b57e00640 1 --2- 172.21.15.31:0/2128163316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d41043efd0 0x55d410445760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.585+0000 7f4b57e00640 1 -- 172.21.15.31:0/2128163316 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d4100b8210 con 0x55d41043f8f0 2024-03-21T10:46:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.585+0000 7f4b5d600640 1 --2- 172.21.15.31:0/2128163316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4104486c0 0x55d4104461e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:46:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.585+0000 7f4b57e00640 1 --2- 172.21.15.31:0/2128163316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41043f8f0 0x55d410445ca0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f4b44098050 tx=0x7f4b4409b450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36b574a12b23169a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.585+0000 7f4b3e000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:46:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.585+0000 7f4b3e000640 1 -- 172.21.15.31:0/2128163316 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b440b4060 con 0x55d41043f8f0 2024-03-21T10:46:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.586+0000 7f4b3e000640 1 -- 172.21.15.31:0/2128163316 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f4b44097cf0 con 0x55d41043f8f0 2024-03-21T10:46:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.586+0000 7f4b62377740 1 -- 172.21.15.31:0/2128163316 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4b50004050 con 0x55d41043f8f0 2024-03-21T10:46:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.586+0000 7f4b3e000640 1 -- 172.21.15.31:0/2128163316 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b44021040 con 0x55d41043f8f0 2024-03-21T10:46:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.586+0000 7f4b62377740 10 client.27506 fetch_fsmap learned FSMap version 26 2024-03-21T10:46:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.586+0000 7f4b62377740 1 -- 172.21.15.31:0/2128163316 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55d410184620 con 0x55d41043f8f0 2024-03-21T10:46:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.586+0000 7f4b3e000640 1 -- 172.21.15.31:0/2128163316 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b50004050 con 0x55d41043f8f0 2024-03-21T10:46:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.586+0000 7f4b3e000640 1 -- 172.21.15.31:0/2128163316 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f4b440cf210 con 0x55d41043f8f0 2024-03-21T10:46:26.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.586+0000 7f4b62377740 10 client.27506 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:46:26.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.586+0000 7f4b62377740 10 client.27506 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:46:26.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.586+0000 7f4b62377740 20 client.27506 populate_metadata read hostname 'smithi031' 2024-03-21T10:46:26.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.586+0000 7f4b62377740 1 --2- 172.21.15.31:0/2128163316 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d410458720 0x55d41045ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.586+0000 7f4b62377740 4 client.27506 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:46:26.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.586+0000 7f4b62377740 1 -- 172.21.15.31:0/2128163316 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d4103b0d70 con 0x55d410458720 2024-03-21T10:46:26.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.587+0000 7f4b5cc00640 1 --2- 172.21.15.31:0/2128163316 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d410458720 0x55d41045ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.587+0000 7f4b5cc00640 1 --2- 172.21.15.31:0/2128163316 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d410458720 0x55d41045ab10 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee165db6446efef6 server_cookie=4143f2bfa2dd7366 in_seq=0 out_seq=0 2024-03-21T10:46:26.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.587+0000 7f4b3e000640 10 client.27506 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:26.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.588+0000 7f4b3e000640 1 -- 172.21.15.31:0/2128163316 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d4103b0d70 con 0x55d410458720 2024-03-21T10:46:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.588+0000 7f4b3e000640 10 client.27506 handle_command_reply: tid=0 2024-03-21T10:46:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.588+0000 7f4b62377740 1 client.27506 shutdown 2024-03-21T10:46:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.588+0000 7f4b62377740 1 -- 172.21.15.31:0/2128163316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41043f8f0 msgr2=0x55d410445ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.588+0000 7f4b62377740 1 --2- 172.21.15.31:0/2128163316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41043f8f0 0x55d410445ca0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f4b44098050 tx=0x7f4b4409b450 comp rx=0 tx=0).stop 2024-03-21T10:46:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.588+0000 7f4b62377740 1 -- 172.21.15.31:0/2128163316 shutdown_connections 2024-03-21T10:46:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.588+0000 7f4b62377740 1 --2- 172.21.15.31:0/2128163316 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d410458720 0x55d41045ab10 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.588+0000 7f4b62377740 1 --2- 172.21.15.31:0/2128163316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4104486c0 0x55d4104461e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.588+0000 7f4b3e000640 0 client.27506 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.588+0000 7f4b62377740 1 --2- 172.21.15.31:0/2128163316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41043f8f0 0x55d410445ca0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.588+0000 7f4b62377740 1 --2- 172.21.15.31:0/2128163316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d41043efd0 0x55d410445760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.588+0000 7f4b62377740 1 -- 172.21.15.31:0/2128163316 >> 172.21.15.31:0/2128163316 conn(0x55d4104398f0 msgr2=0x55d41043b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.589+0000 7f4b62377740 1 -- 172.21.15.31:0/2128163316 shutdown_connections 2024-03-21T10:46:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.589+0000 7f4b62377740 1 -- 172.21.15.31:0/2128163316 wait complete. 2024-03-21T10:46:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.589+0000 7f4b62377740 20 client.27506 trim_cache size 0 max 16384 2024-03-21T10:46:26.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.606+0000 7f4b62377740 1 Processor -- start 2024-03-21T10:46:26.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.607+0000 7f4b62377740 1 -- start start 2024-03-21T10:46:26.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.607+0000 7f4b62377740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d41043efd0 0x55d410447e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.607+0000 7f4b62377740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d41043f8f0 0x55d41043b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.607+0000 7f4b62377740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4104486c0 0x55d410453a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.607+0000 7f4b62377740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b581b30c0 con 0x55d41043f8f0 2024-03-21T10:46:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.607+0000 7f4b62377740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b581b0bc0 con 0x55d4104486c0 2024-03-21T10:46:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.607+0000 7f4b62377740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b581aec80 con 0x55d41043efd0 2024-03-21T10:46:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.607+0000 7f4b5d600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4104486c0 0x55d410453a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.607+0000 7f4b5cc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d41043efd0 0x55d410447e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.607+0000 7f4b5d600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4104486c0 0x55d410453a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38584/0 (socket says 172.21.15.31:38584) 2024-03-21T10:46:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.607+0000 7f4b5d600640 1 -- 172.21.15.31:0/3298960910 learned_addr learned my addr 172.21.15.31:0/3298960910 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.607+0000 7f4b57e00640 1 --2- 172.21.15.31:0/3298960910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d41043f8f0 0x55d41043b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.608+0000 7f4b5cc00640 1 -- 172.21.15.31:0/3298960910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4104486c0 msgr2=0x55d410453a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.608+0000 7f4b5cc00640 1 --2- 172.21.15.31:0/3298960910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4104486c0 0x55d410453a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.608+0000 7f4b5cc00640 1 -- 172.21.15.31:0/3298960910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d41043f8f0 msgr2=0x55d41043b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.608+0000 7f4b5cc00640 1 --2- 172.21.15.31:0/3298960910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d41043f8f0 0x55d41043b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.608+0000 7f4b5cc00640 1 -- 172.21.15.31:0/3298960910 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d410444710 con 0x55d41043efd0 2024-03-21T10:46:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.608+0000 7f4b5cc00640 1 --2- 172.21.15.31:0/3298960910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d41043efd0 0x55d410447e90 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c009230 tx=0x7f4b4c049000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5d189f6482dcc72 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.608+0000 7f4b49200640 1 -- 172.21.15.31:0/3298960910 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4c004240 con 0x55d41043efd0 2024-03-21T10:46:26.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.609+0000 7f4b49200640 1 -- 172.21.15.31:0/3298960910 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b4c0043e0 con 0x55d41043efd0 2024-03-21T10:46:26.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.609+0000 7f4b49200640 1 -- 172.21.15.31:0/3298960910 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4c057c90 con 0x55d41043efd0 2024-03-21T10:46:26.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.610+0000 7f4b62377740 1 -- 172.21.15.31:0/3298960910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d41043efd0 msgr2=0x55d410447e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.610+0000 7f4b62377740 1 --2- 172.21.15.31:0/3298960910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d41043efd0 0x55d410447e90 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c009230 tx=0x7f4b4c049000 comp rx=0 tx=0).stop 2024-03-21T10:46:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.610+0000 7f4b62377740 1 -- 172.21.15.31:0/3298960910 shutdown_connections 2024-03-21T10:46:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.610+0000 7f4b62377740 1 --2- 172.21.15.31:0/3298960910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4104486c0 0x55d410453a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.610+0000 7f4b62377740 1 --2- 172.21.15.31:0/3298960910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d41043f8f0 0x55d41043b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.610+0000 7f4b62377740 1 --2- 172.21.15.31:0/3298960910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d41043efd0 0x55d410447e90 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.610+0000 7f4b62377740 1 -- 172.21.15.31:0/3298960910 >> 172.21.15.31:0/3298960910 conn(0x55d4104398f0 msgr2=0x55d410439cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.610+0000 7f4b62377740 1 -- 172.21.15.31:0/3298960910 shutdown_connections 2024-03-21T10:46:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.610+0000 7f4b62377740 1 -- 172.21.15.31:0/3298960910 wait complete. 2024-03-21T10:46:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.610+0000 7f4b62377740 1 -- start start 2024-03-21T10:46:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.611+0000 7f4b62377740 1 Processor -- start 2024-03-21T10:46:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.611+0000 7f4b62377740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d41043efd0 0x55d410443380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.611+0000 7f4b62377740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d41043f8f0 0x55d41044d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.611+0000 7f4b62377740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4104486c0 0x55d41044da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.611+0000 7f4b62377740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b581af280 con 0x55d41043f8f0 2024-03-21T10:46:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.611+0000 7f4b62377740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b581b0560 con 0x55d4104486c0 2024-03-21T10:46:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.611+0000 7f4b62377740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b581af8a0 con 0x55d41043efd0 2024-03-21T10:46:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.611+0000 7f4b62377740 10 client.0 _ll_register_callbacks cb 0x7f4b440cc8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:46:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.612+0000 7f4b5cc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d41043efd0 0x55d410443380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.612+0000 7f4b5cc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d41043efd0 0x55d410443380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53884/0 (socket says 172.21.15.31:53884) 2024-03-21T10:46:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.612+0000 7f4b5cc00640 1 -- 172.21.15.31:0/1569586148 learned_addr learned my addr 172.21.15.31:0/1569586148 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.612+0000 7f4b5d600640 1 --2- 172.21.15.31:0/1569586148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4104486c0 0x55d41044da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.612+0000 7f4b57e00640 1 --2- 172.21.15.31:0/1569586148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d41043f8f0 0x55d41044d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.612+0000 7f4b5cc00640 1 -- 172.21.15.31:0/1569586148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4104486c0 msgr2=0x55d41044da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.612+0000 7f4b5cc00640 1 --2- 172.21.15.31:0/1569586148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4104486c0 0x55d41044da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.612+0000 7f4b5cc00640 1 -- 172.21.15.31:0/1569586148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d41043f8f0 msgr2=0x55d41044d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.612+0000 7f4b5cc00640 1 --2- 172.21.15.31:0/1569586148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d41043f8f0 0x55d41044d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.612+0000 7f4b5cc00640 1 -- 172.21.15.31:0/1569586148 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d41044e140 con 0x55d41043efd0 2024-03-21T10:46:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.612+0000 7f4b5cc00640 1 --2- 172.21.15.31:0/1569586148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d41043efd0 0x55d410443380 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c0092a0 tx=0x7f4b4c004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f61a332e0bc723df server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.612+0000 7f4b3e000640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:46:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.612+0000 7f4b3e000640 1 -- 172.21.15.31:0/1569586148 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4c057de0 con 0x55d41043efd0 2024-03-21T10:46:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.613+0000 7f4b3e000640 1 -- 172.21.15.31:0/1569586148 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f4b4c004ef0 con 0x55d41043efd0 2024-03-21T10:46:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.613+0000 7f4b62377740 1 -- 172.21.15.31:0/1569586148 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4b4c0043e0 con 0x55d41043efd0 2024-03-21T10:46:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.613+0000 7f4b3e000640 1 -- 172.21.15.31:0/1569586148 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b4c0050f0 con 0x55d41043efd0 2024-03-21T10:46:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.613+0000 7f4b62377740 10 client.27433 fetch_fsmap learned FSMap version 26 2024-03-21T10:46:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.613+0000 7f4b62377740 1 -- 172.21.15.31:0/1569586148 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55d4104b4e60 con 0x55d41043efd0 2024-03-21T10:46:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.613+0000 7f4b3e000640 1 -- 172.21.15.31:0/1569586148 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b4c0043e0 con 0x55d41043efd0 2024-03-21T10:46:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.613+0000 7f4b3e000640 1 -- 172.21.15.31:0/1569586148 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f4b4c054070 con 0x55d41043efd0 2024-03-21T10:46:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.613+0000 7f4b62377740 10 client.27433 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:46:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.613+0000 7f4b62377740 10 client.27433 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:46:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.613+0000 7f4b62377740 20 client.27433 populate_metadata read hostname 'smithi031' 2024-03-21T10:46:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.613+0000 7f4b62377740 1 --2- 172.21.15.31:0/1569586148 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d4104b5500 0x55d4104b7920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.613+0000 7f4b62377740 4 client.27433 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:46:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.613+0000 7f4b62377740 1 -- 172.21.15.31:0/1569586148 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d4103ab0c0 con 0x55d4104b5500 2024-03-21T10:46:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.614+0000 7f4b57e00640 1 --2- 172.21.15.31:0/1569586148 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d4104b5500 0x55d4104b7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.614+0000 7f4b57e00640 1 --2- 172.21.15.31:0/1569586148 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d4104b5500 0x55d4104b7920 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50df1f790d9b5d87 server_cookie=8355e559eb336aa1 in_seq=0 out_seq=0 2024-03-21T10:46:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.614+0000 7f4b3e000640 10 client.27433 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.627+0000 7f4b3e000640 1 -- 172.21.15.31:0/1569586148 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d4103ab0c0 con 0x55d4104b5500 2024-03-21T10:46:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.627+0000 7f4b3e000640 10 client.27433 handle_command_reply: tid=0 2024-03-21T10:46:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.627+0000 7f4b62377740 1 client.27433 shutdown 2024-03-21T10:46:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.627+0000 7f4b62377740 1 -- 172.21.15.31:0/1569586148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d41043efd0 msgr2=0x55d410443380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.627+0000 7f4b62377740 1 --2- 172.21.15.31:0/1569586148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d41043efd0 0x55d410443380 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c0092a0 tx=0x7f4b4c004270 comp rx=0 tx=0).stop 2024-03-21T10:46:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.627+0000 7f4b62377740 1 -- 172.21.15.31:0/1569586148 shutdown_connections 2024-03-21T10:46:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.627+0000 7f4b62377740 1 --2- 172.21.15.31:0/1569586148 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d4104b5500 0x55d4104b7920 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.627+0000 7f4b62377740 1 --2- 172.21.15.31:0/1569586148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4104486c0 0x55d41044da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.627+0000 7f4b62377740 1 --2- 172.21.15.31:0/1569586148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d41043f8f0 0x55d41044d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.627+0000 7f4b3e000640 0 client.27433 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.627+0000 7f4b62377740 1 --2- 172.21.15.31:0/1569586148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d41043efd0 0x55d410443380 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.627+0000 7f4b62377740 1 -- 172.21.15.31:0/1569586148 >> 172.21.15.31:0/1569586148 conn(0x55d4104398f0 msgr2=0x55d410445f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.628+0000 7f4b62377740 1 -- 172.21.15.31:0/1569586148 shutdown_connections 2024-03-21T10:46:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.628+0000 7f4b62377740 1 -- 172.21.15.31:0/1569586148 wait complete. 2024-03-21T10:46:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.628+0000 7f4b62377740 20 client.27433 trim_cache size 0 max 16384 2024-03-21T10:46:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:46:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:46:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:46:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:46:26.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.628+0000 7f4b5f600640 1 -- 172.21.15.31:0/3333865859 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4b400a3080 msgr2=0x7f4b400a5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.629+0000 7f4b5f600640 1 --2- 172.21.15.31:0/3333865859 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4b400a3080 0x7f4b400a5540 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c009900 tx=0x7f4b4c000bc0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.629+0000 7f4b5f600640 1 -- 172.21.15.31:0/3333865859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b58104250 msgr2=0x7f4b5819a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.629+0000 7f4b5f600640 1 --2- 172.21.15.31:0/3333865859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b58104250 0x7f4b5819a6d0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4b4400d950 tx=0x7f4b4400de10 comp rx=0 tx=0).stop 2024-03-21T10:46:26.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.629+0000 7f4b5f600640 1 -- 172.21.15.31:0/3333865859 shutdown_connections 2024-03-21T10:46:26.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.629+0000 7f4b5f600640 1 --2- 172.21.15.31:0/3333865859 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4b400a3080 0x7f4b400a5540 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.629+0000 7f4b5f600640 1 --2- 172.21.15.31:0/3333865859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b581098a0 0x7f4b5819ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.629+0000 7f4b5f600640 1 --2- 172.21.15.31:0/3333865859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b58104250 0x7f4b5819a6d0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.629+0000 7f4b5f600640 1 --2- 172.21.15.31:0/3333865859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b58103890 0x7f4b5819a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.629+0000 7f4b5f600640 1 -- 172.21.15.31:0/3333865859 >> 172.21.15.31:0/3333865859 conn(0x7f4b580ff320 msgr2=0x7f4b580ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.629+0000 7f4b5f600640 1 -- 172.21.15.31:0/3333865859 shutdown_connections 2024-03-21T10:46:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:26.629+0000 7f4b5f600640 1 -- 172.21.15.31:0/3333865859 wait complete. 2024-03-21T10:46:26.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.639+0000 7f0407200640 1 -- 172.21.15.31:0/409713836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0408102f50 msgr2=0x7f040810b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.639+0000 7f0407200640 1 --2- 172.21.15.31:0/409713836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0408102f50 0x7f040810b430 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f03f4009900 tx=0x7f03f402bcd0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.639+0000 7f0407200640 1 -- 172.21.15.31:0/409713836 shutdown_connections 2024-03-21T10:46:26.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.639+0000 7f0407200640 1 --2- 172.21.15.31:0/409713836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f040810ba50 0x7f040810ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.639+0000 7f0407200640 1 --2- 172.21.15.31:0/409713836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0408102f50 0x7f040810b430 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.639+0000 7f0407200640 1 --2- 172.21.15.31:0/409713836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0408102630 0x7f0408102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.640+0000 7f0407200640 1 -- 172.21.15.31:0/409713836 >> 172.21.15.31:0/409713836 conn(0x7f0408077130 msgr2=0x7f04080fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:26.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.640+0000 7f0407200640 1 -- 172.21.15.31:0/409713836 shutdown_connections 2024-03-21T10:46:26.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.640+0000 7f0407200640 1 -- 172.21.15.31:0/409713836 wait complete. 2024-03-21T10:46:26.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.641+0000 7f0407200640 1 Processor -- start 2024-03-21T10:46:26.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.641+0000 7f0407200640 1 -- start start 2024-03-21T10:46:26.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.641+0000 7f0407200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0408102630 0x7f0408199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.641+0000 7f0407200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0408102f50 0x7f040819a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.641+0000 7f0407200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f040810ba50 0x7f040819e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.641+0000 7f0407200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0408111850 con 0x7f0408102630 2024-03-21T10:46:26.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.641+0000 7f0407200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04081116d0 con 0x7f040810ba50 2024-03-21T10:46:26.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.641+0000 7f0407200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04081119d0 con 0x7f0408102f50 2024-03-21T10:46:26.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.642+0000 7f0405e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0408102630 0x7f0408199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.642+0000 7f0405e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0408102630 0x7f0408199fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44902/0 (socket says 172.21.15.31:44902) 2024-03-21T10:46:26.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.642+0000 7f0406800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f040810ba50 0x7f040819e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.642+0000 7f0405e00640 1 -- 172.21.15.31:0/1127237954 learned_addr learned my addr 172.21.15.31:0/1127237954 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:26.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.642+0000 7f0405400640 1 --2- 172.21.15.31:0/1127237954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0408102f50 0x7f040819a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.642+0000 7f0406800640 1 -- 172.21.15.31:0/1127237954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0408102f50 msgr2=0x7f040819a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.642+0000 7f0406800640 1 --2- 172.21.15.31:0/1127237954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0408102f50 0x7f040819a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.642+0000 7f0406800640 1 -- 172.21.15.31:0/1127237954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0408102630 msgr2=0x7f0408199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.642+0000 7f0406800640 1 --2- 172.21.15.31:0/1127237954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0408102630 0x7f0408199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.642+0000 7f0406800640 1 -- 172.21.15.31:0/1127237954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f040819efa0 con 0x7f040810ba50 2024-03-21T10:46:26.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.642+0000 7f0405400640 1 --2- 172.21.15.31:0/1127237954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0408102f50 0x7f040819a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:46:26.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.642+0000 7f0406800640 1 --2- 172.21.15.31:0/1127237954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f040810ba50 0x7f040819e870 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f040000ec50 tx=0x7f040000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:26.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.643+0000 7f03fea00640 1 -- 172.21.15.31:0/1127237954 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0400003b70 con 0x7f040810ba50 2024-03-21T10:46:26.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.643+0000 7f0407200640 1 -- 172.21.15.31:0/1127237954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f040819f290 con 0x7f040810ba50 2024-03-21T10:46:26.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.643+0000 7f03fea00640 1 -- 172.21.15.31:0/1127237954 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0400003d10 con 0x7f040810ba50 2024-03-21T10:46:26.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.643+0000 7f03fea00640 1 -- 172.21.15.31:0/1127237954 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0400010890 con 0x7f040810ba50 2024-03-21T10:46:26.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.643+0000 7f0407200640 1 -- 172.21.15.31:0/1127237954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04081a6b20 con 0x7f040810ba50 2024-03-21T10:46:26.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.646+0000 7f040fb85740 1 Processor -- start 2024-03-21T10:46:26.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.648+0000 7f03fea00640 1 -- 172.21.15.31:0/1127237954 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f0400003fc0 con 0x7f040810ba50 2024-03-21T10:46:26.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.648+0000 7f040fb85740 1 -- start start 2024-03-21T10:46:26.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.648+0000 7f040fb85740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5adb78b0 0x556d5adb7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.648+0000 7f040fb85740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5adb8260 0x556d5adc0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.648+0000 7f040fb85740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5adc0de0 0x556d5adc3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.648+0000 7f040fb85740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d5ad1b2c0 con 0x556d5adb78b0 2024-03-21T10:46:26.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.648+0000 7f040fb85740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d5acc3c90 con 0x556d5adc0de0 2024-03-21T10:46:26.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.648+0000 7f040fb85740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d5ad17360 con 0x556d5adb8260 2024-03-21T10:46:26.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.649+0000 7f0405e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5adb8260 0x556d5adc0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.649+0000 7f0405e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5adb8260 0x556d5adc0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53910/0 (socket says 172.21.15.31:53910) 2024-03-21T10:46:26.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.649+0000 7f0405e00640 1 -- 172.21.15.31:0/2737955058 learned_addr learned my addr 172.21.15.31:0/2737955058 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:26.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.649+0000 7f0405400640 1 --2- 172.21.15.31:0/2737955058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5adb78b0 0x556d5adb7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.649+0000 7f0405400640 1 --2- 172.21.15.31:0/2737955058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5adc0de0 0x556d5adc3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.649+0000 7f0405e00640 1 -- 172.21.15.31:0/2737955058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5adc0de0 msgr2=0x556d5adc3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:26.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.649+0000 7f0405e00640 1 --2- 172.21.15.31:0/2737955058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5adc0de0 0x556d5adc3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.649+0000 7f0405e00640 1 -- 172.21.15.31:0/2737955058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5adb78b0 msgr2=0x556d5adb7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:26.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.649+0000 7f0405e00640 1 --2- 172.21.15.31:0/2737955058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5adb78b0 0x556d5adb7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.649+0000 7f0405e00640 1 -- 172.21.15.31:0/2737955058 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d5ace8080 con 0x556d5adb8260 2024-03-21T10:46:26.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.649+0000 7f0405400640 1 --2- 172.21.15.31:0/2737955058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5adb78b0 0x556d5adb7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:46:26.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.649+0000 7f0405e00640 1 --2- 172.21.15.31:0/2737955058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5adb8260 0x556d5adc0810 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f03f8009dc0 tx=0x7f03f8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ed055dba428d744 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:26.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.649+0000 7f03fea00640 1 --2- 172.21.15.31:0/1127237954 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f03ec0a3090 0x7f03ec0a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.649+0000 7f03f3400640 1 -- 172.21.15.31:0/2737955058 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f8005220 con 0x556d5adb8260 2024-03-21T10:46:26.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.650+0000 7f03f3400640 1 -- 172.21.15.31:0/2737955058 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f03f8004c80 con 0x556d5adb8260 2024-03-21T10:46:26.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.650+0000 7f03f3400640 1 -- 172.21.15.31:0/2737955058 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f8019970 con 0x556d5adb8260 2024-03-21T10:46:26.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.650+0000 7f03fea00640 1 -- 172.21.15.31:0/1127237954 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f04000d0150 con 0x7f040810ba50 2024-03-21T10:46:26.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.650+0000 7f0405400640 1 --2- 172.21.15.31:0/1127237954 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f03ec0a3090 0x7f03ec0a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.650+0000 7f040fb85740 1 -- 172.21.15.31:0/2737955058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5adb8260 msgr2=0x556d5adc0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.650+0000 7f040fb85740 1 --2- 172.21.15.31:0/2737955058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5adb8260 0x556d5adc0810 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f03f8009dc0 tx=0x7f03f8011810 comp rx=0 tx=0).stop 2024-03-21T10:46:26.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.651+0000 7f040fb85740 1 -- 172.21.15.31:0/2737955058 shutdown_connections 2024-03-21T10:46:26.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.651+0000 7f040fb85740 1 --2- 172.21.15.31:0/2737955058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5adc0de0 0x556d5adc3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.651+0000 7f040fb85740 1 --2- 172.21.15.31:0/2737955058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5adb8260 0x556d5adc0810 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.651+0000 7f040fb85740 1 --2- 172.21.15.31:0/2737955058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5adb78b0 0x556d5adb7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.651+0000 7f040fb85740 1 -- 172.21.15.31:0/2737955058 >> 172.21.15.31:0/2737955058 conn(0x556d5adb1f80 msgr2=0x556d5adb2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:26.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.651+0000 7f0405400640 1 --2- 172.21.15.31:0/1127237954 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f03ec0a3090 0x7f03ec0a5550 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f03f4006010 tx=0x7f03f4005a20 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:26.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.651+0000 7f040fb85740 1 -- 172.21.15.31:0/2737955058 shutdown_connections 2024-03-21T10:46:26.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.651+0000 7f040fb85740 1 -- 172.21.15.31:0/2737955058 wait complete. 2024-03-21T10:46:26.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.651+0000 7f040fb85740 1 -- start start 2024-03-21T10:46:26.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.651+0000 7f040fb85740 1 Processor -- start 2024-03-21T10:46:26.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.652+0000 7f040fb85740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5adb78b0 0x556d5adc06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.652+0000 7f040fb85740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5adb8260 0x556d5adbda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.652+0000 7f040fb85740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5adc0de0 0x556d5adbdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.652+0000 7f040fb85740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d5adc3d50 con 0x556d5adc0de0 2024-03-21T10:46:26.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.652+0000 7f040fb85740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d5acd70d0 con 0x556d5adb78b0 2024-03-21T10:46:26.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.652+0000 7f040fb85740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d5adc3ed0 con 0x556d5adb8260 2024-03-21T10:46:26.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.652+0000 7f040fb85740 10 client.0 _ll_register_callbacks cb 0x556d5acdbe80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:46:26.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.652+0000 7f0405e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5adb78b0 0x556d5adc06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.652+0000 7f0405400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5adb8260 0x556d5adbda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.652+0000 7f0405e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5adb78b0 0x556d5adc06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38644/0 (socket says 172.21.15.31:38644) 2024-03-21T10:46:26.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.652+0000 7f0405e00640 1 -- 172.21.15.31:0/3100864058 learned_addr learned my addr 172.21.15.31:0/3100864058 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:26.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.652+0000 7f0406800640 1 --2- 172.21.15.31:0/3100864058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5adc0de0 0x556d5adbdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.653+0000 7f0405e00640 1 -- 172.21.15.31:0/3100864058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5adb8260 msgr2=0x556d5adbda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:26.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.653+0000 7f0405e00640 1 --2- 172.21.15.31:0/3100864058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5adb8260 0x556d5adbda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.653+0000 7f0405e00640 1 -- 172.21.15.31:0/3100864058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5adc0de0 msgr2=0x556d5adbdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:26.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.653+0000 7f0405e00640 1 --2- 172.21.15.31:0/3100864058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5adc0de0 0x556d5adbdfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.653+0000 7f0405e00640 1 -- 172.21.15.31:0/3100864058 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556d5aa2f3f0 con 0x556d5adb78b0 2024-03-21T10:46:26.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.653+0000 7f0406800640 1 --2- 172.21.15.31:0/3100864058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5adc0de0 0x556d5adbdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:46:26.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.653+0000 7f0405e00640 1 --2- 172.21.15.31:0/3100864058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5adb78b0 0x556d5adc06d0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f03f8019510 tx=0x7f03f8004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=674bedd136a6639d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:26.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.653+0000 7f03e7e00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:46:26.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.653+0000 7f03e7e00640 1 -- 172.21.15.31:0/3100864058 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f8027070 con 0x556d5adb78b0 2024-03-21T10:46:26.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.653+0000 7f040fb85740 1 -- 172.21.15.31:0/3100864058 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f03f8004c80 con 0x556d5adb78b0 2024-03-21T10:46:26.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.653+0000 7f03e7e00640 1 -- 172.21.15.31:0/3100864058 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f03f8005030 con 0x556d5adb78b0 2024-03-21T10:46:26.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.654+0000 7f03e7e00640 1 -- 172.21.15.31:0/3100864058 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03f8033050 con 0x556d5adb78b0 2024-03-21T10:46:26.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.654+0000 7f040fb85740 10 client.27524 fetch_fsmap learned FSMap version 26 2024-03-21T10:46:26.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.654+0000 7f040fb85740 1 -- 172.21.15.31:0/3100864058 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x556d5aaf97d0 con 0x556d5adb78b0 2024-03-21T10:46:26.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.654+0000 7f03e7e00640 1 -- 172.21.15.31:0/3100864058 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f03f8004c80 con 0x556d5adb78b0 2024-03-21T10:46:26.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.654+0000 7f03e7e00640 1 -- 172.21.15.31:0/3100864058 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f03f80035f0 con 0x556d5adb78b0 2024-03-21T10:46:26.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.654+0000 7f040fb85740 10 client.27524 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:46:26.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.654+0000 7f040fb85740 10 client.27524 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:46:26.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.654+0000 7f040fb85740 20 client.27524 populate_metadata read hostname 'smithi031' 2024-03-21T10:46:26.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.654+0000 7f040fb85740 1 --2- 172.21.15.31:0/3100864058 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x556d5add0b30 0x556d5add2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.654+0000 7f040fb85740 4 client.27524 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:46:26.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.654+0000 7f040fb85740 1 -- 172.21.15.31:0/3100864058 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556d5ad28800 con 0x556d5add0b30 2024-03-21T10:46:26.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.655+0000 7f0405400640 1 --2- 172.21.15.31:0/3100864058 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x556d5add0b30 0x556d5add2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.655+0000 7f0405400640 1 --2- 172.21.15.31:0/3100864058 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x556d5add0b30 0x556d5add2f20 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5da7660fbb9f174a server_cookie=8ee85180b0deaec6 in_seq=0 out_seq=0 2024-03-21T10:46:26.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.655+0000 7f03e7e00640 10 client.27524 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:46:26.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.656+0000 7f03e7e00640 1 -- 172.21.15.31:0/3100864058 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556d5ad28800 con 0x556d5add0b30 2024-03-21T10:46:26.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.656+0000 7f03e7e00640 10 client.27524 handle_command_reply: tid=0 2024-03-21T10:46:26.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.656+0000 7f040fb85740 1 client.27524 shutdown 2024-03-21T10:46:26.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.657+0000 7f040fb85740 1 -- 172.21.15.31:0/3100864058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5adb78b0 msgr2=0x556d5adc06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.657+0000 7f040fb85740 1 --2- 172.21.15.31:0/3100864058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5adb78b0 0x556d5adc06d0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f03f8019510 tx=0x7f03f8004270 comp rx=0 tx=0).stop 2024-03-21T10:46:26.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.657+0000 7f040fb85740 1 -- 172.21.15.31:0/3100864058 shutdown_connections 2024-03-21T10:46:26.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.657+0000 7f040fb85740 1 --2- 172.21.15.31:0/3100864058 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x556d5add0b30 0x556d5add2f20 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.657+0000 7f040fb85740 1 --2- 172.21.15.31:0/3100864058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5adc0de0 0x556d5adbdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.657+0000 7f040fb85740 1 --2- 172.21.15.31:0/3100864058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5adb8260 0x556d5adbda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.657+0000 7f040fb85740 1 --2- 172.21.15.31:0/3100864058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5adb78b0 0x556d5adc06d0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.657+0000 7f03e7e00640 0 client.27524 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:46:26.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.657+0000 7f040fb85740 1 -- 172.21.15.31:0/3100864058 >> 172.21.15.31:0/3100864058 conn(0x556d5adb1f80 msgr2=0x556d5adb3cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:26.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.657+0000 7f040fb85740 1 -- 172.21.15.31:0/3100864058 shutdown_connections 2024-03-21T10:46:26.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.657+0000 7f040fb85740 1 -- 172.21.15.31:0/3100864058 wait complete. 2024-03-21T10:46:26.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.657+0000 7f040fb85740 20 client.27524 trim_cache size 0 max 16384 2024-03-21T10:46:26.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.671+0000 7f040fb85740 1 Processor -- start 2024-03-21T10:46:26.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.671+0000 7f040fb85740 1 -- start start 2024-03-21T10:46:26.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.671+0000 7f040fb85740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5adb78b0 0x556d5adc0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.671+0000 7f040fb85740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5adb8260 0x556d5adc06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.671+0000 7f040fb85740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5adc0de0 0x556d5adbace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.671+0000 7f040fb85740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04081b2fc0 con 0x556d5adb78b0 2024-03-21T10:46:26.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.671+0000 7f040fb85740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04081b0ac0 con 0x556d5adc0de0 2024-03-21T10:46:26.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.671+0000 7f040fb85740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04081aeb80 con 0x556d5adb8260 2024-03-21T10:46:26.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.672+0000 7f0405e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5adb78b0 0x556d5adc0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.672+0000 7f0406800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5adc0de0 0x556d5adbace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.672+0000 7f0406800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5adc0de0 0x556d5adbace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38652/0 (socket says 172.21.15.31:38652) 2024-03-21T10:46:26.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.672+0000 7f0405e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5adb78b0 0x556d5adc0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44930/0 (socket says 172.21.15.31:44930) 2024-03-21T10:46:26.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.672+0000 7f0405400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5adb8260 0x556d5adc06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.672+0000 7f0406800640 1 -- 172.21.15.31:0/631378176 learned_addr learned my addr 172.21.15.31:0/631378176 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:26.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.672+0000 7f0406800640 1 -- 172.21.15.31:0/631378176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5adb8260 msgr2=0x556d5adc06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:26.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.672+0000 7f0406800640 1 --2- 172.21.15.31:0/631378176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5adb8260 0x556d5adc06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.672+0000 7f0406800640 1 -- 172.21.15.31:0/631378176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5adb78b0 msgr2=0x556d5adc0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:26.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.672+0000 7f0406800640 1 --2- 172.21.15.31:0/631378176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5adb78b0 0x556d5adc0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.672+0000 7f0406800640 1 -- 172.21.15.31:0/631378176 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d5adbbfb0 con 0x556d5adc0de0 2024-03-21T10:46:26.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.672+0000 7f0406800640 1 --2- 172.21.15.31:0/631378176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5adc0de0 0x556d5adbace0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f040000cba0 tx=0x7f040009bda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11ae5e41d6a358de server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:26.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.673+0000 7f03f2a00640 1 -- 172.21.15.31:0/631378176 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040009c8a0 con 0x556d5adc0de0 2024-03-21T10:46:26.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.673+0000 7f03f2a00640 1 -- 172.21.15.31:0/631378176 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04000b5070 con 0x556d5adc0de0 2024-03-21T10:46:26.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.673+0000 7f03f2a00640 1 -- 172.21.15.31:0/631378176 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04000cd780 con 0x556d5adc0de0 2024-03-21T10:46:26.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.673+0000 7f040fb85740 1 -- 172.21.15.31:0/631378176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5adc0de0 msgr2=0x556d5adbace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.673+0000 7f040fb85740 1 --2- 172.21.15.31:0/631378176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5adc0de0 0x556d5adbace0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f040000cba0 tx=0x7f040009bda0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.673+0000 7f040fb85740 1 -- 172.21.15.31:0/631378176 shutdown_connections 2024-03-21T10:46:26.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.673+0000 7f040fb85740 1 --2- 172.21.15.31:0/631378176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5adc0de0 0x556d5adbace0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.673+0000 7f040fb85740 1 --2- 172.21.15.31:0/631378176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5adb8260 0x556d5adc06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.673+0000 7f040fb85740 1 --2- 172.21.15.31:0/631378176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5adb78b0 0x556d5adc0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.673+0000 7f040fb85740 1 -- 172.21.15.31:0/631378176 >> 172.21.15.31:0/631378176 conn(0x556d5adb1f80 msgr2=0x556d5adb1160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:26.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.673+0000 7f040fb85740 1 -- 172.21.15.31:0/631378176 shutdown_connections 2024-03-21T10:46:26.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.673+0000 7f040fb85740 1 -- 172.21.15.31:0/631378176 wait complete. 2024-03-21T10:46:26.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.674+0000 7f040fb85740 1 -- start start 2024-03-21T10:46:26.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.674+0000 7f040fb85740 1 Processor -- start 2024-03-21T10:46:26.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.674+0000 7f040fb85740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5adb78b0 0x556d5adc8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.674+0000 7f040fb85740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5adb8260 0x556d5adc64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.674+0000 7f040fb85740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5adc0de0 0x556d5adc6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.674+0000 7f040fb85740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04081af180 con 0x556d5adb8260 2024-03-21T10:46:26.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.674+0000 7f040fb85740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04081b0460 con 0x556d5adb78b0 2024-03-21T10:46:26.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.674+0000 7f040fb85740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04081af7a0 con 0x556d5adc0de0 2024-03-21T10:46:26.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.674+0000 7f040fb85740 10 client.0 _ll_register_callbacks cb 0x7f03f8009c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:46:26.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.675+0000 7f0406800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5adc0de0 0x556d5adc6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.675+0000 7f0405e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5adb78b0 0x556d5adc8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.675+0000 7f0405400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5adb8260 0x556d5adc64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.675+0000 7f0406800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5adc0de0 0x556d5adc6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53936/0 (socket says 172.21.15.31:53936) 2024-03-21T10:46:26.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.675+0000 7f0406800640 1 -- 172.21.15.31:0/1284937835 learned_addr learned my addr 172.21.15.31:0/1284937835 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:26.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.675+0000 7f0406800640 1 -- 172.21.15.31:0/1284937835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5adb78b0 msgr2=0x556d5adc8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:26.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.675+0000 7f0406800640 1 --2- 172.21.15.31:0/1284937835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5adb78b0 0x556d5adc8ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.675+0000 7f0406800640 1 -- 172.21.15.31:0/1284937835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5adb8260 msgr2=0x556d5adc64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:26.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.675+0000 7f0406800640 1 --2- 172.21.15.31:0/1284937835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5adb8260 0x556d5adc64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.675+0000 7f0406800640 1 -- 172.21.15.31:0/1284937835 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556d5ae2b1a0 con 0x556d5adc0de0 2024-03-21T10:46:26.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.675+0000 7f0405e00640 1 --2- 172.21.15.31:0/1284937835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5adb78b0 0x556d5adc8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:46:26.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.675+0000 7f0405400640 1 --2- 172.21.15.31:0/1284937835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5adb8260 0x556d5adc64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:46:26.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.675+0000 7f0406800640 1 --2- 172.21.15.31:0/1284937835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5adc0de0 0x556d5adc6ae0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f04000a4680 tx=0x7f04000046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bec5bf2e5624850 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:26.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.675+0000 7f03e7e00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:46:26.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.676+0000 7f03e7e00640 1 -- 172.21.15.31:0/1284937835 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04000b9070 con 0x556d5adc0de0 2024-03-21T10:46:26.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.676+0000 7f03e7e00640 1 -- 172.21.15.31:0/1284937835 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f040009c400 con 0x556d5adc0de0 2024-03-21T10:46:26.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.676+0000 7f040fb85740 1 -- 172.21.15.31:0/1284937835 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f04000b5070 con 0x556d5adc0de0 2024-03-21T10:46:26.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.676+0000 7f03e7e00640 1 -- 172.21.15.31:0/1284937835 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f040009c1a0 con 0x556d5adc0de0 2024-03-21T10:46:26.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.676+0000 7f040fb85740 10 client.27451 fetch_fsmap learned FSMap version 26 2024-03-21T10:46:26.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.676+0000 7f040fb85740 1 -- 172.21.15.31:0/1284937835 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x556d5ae2c2f0 con 0x556d5adc0de0 2024-03-21T10:46:26.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.676+0000 7f03e7e00640 1 -- 172.21.15.31:0/1284937835 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04000b5070 con 0x556d5adc0de0 2024-03-21T10:46:26.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.676+0000 7f03e7e00640 1 -- 172.21.15.31:0/1284937835 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f04000a4720 con 0x556d5adc0de0 2024-03-21T10:46:26.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.676+0000 7f040fb85740 10 client.27451 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:46:26.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.676+0000 7f040fb85740 10 client.27451 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:46:26.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.676+0000 7f040fb85740 20 client.27451 populate_metadata read hostname 'smithi031' 2024-03-21T10:46:26.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.676+0000 7f040fb85740 1 --2- 172.21.15.31:0/1284937835 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x556d5ae2c9b0 0x556d5ae2edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.676+0000 7f040fb85740 4 client.27451 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:46:26.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.676+0000 7f040fb85740 1 -- 172.21.15.31:0/1284937835 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556d5ad22b50 con 0x556d5ae2c9b0 2024-03-21T10:46:26.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.677+0000 7f0405e00640 1 --2- 172.21.15.31:0/1284937835 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x556d5ae2c9b0 0x556d5ae2edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.677+0000 7f0405e00640 1 --2- 172.21.15.31:0/1284937835 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x556d5ae2c9b0 0x556d5ae2edd0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7d9516cd17c588f7 server_cookie=ab091721977990d0 in_seq=0 out_seq=0 2024-03-21T10:46:26.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.677+0000 7f03e7e00640 10 client.27451 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:46:26.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.677+0000 7f03e7e00640 1 -- 172.21.15.31:0/1284937835 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556d5ad22b50 con 0x556d5ae2c9b0 2024-03-21T10:46:26.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.677+0000 7f03e7e00640 10 client.27451 handle_command_reply: tid=0 2024-03-21T10:46:26.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.677+0000 7f040fb85740 1 client.27451 shutdown 2024-03-21T10:46:26.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.677+0000 7f040fb85740 1 -- 172.21.15.31:0/1284937835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5adc0de0 msgr2=0x556d5adc6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.677+0000 7f040fb85740 1 --2- 172.21.15.31:0/1284937835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5adc0de0 0x556d5adc6ae0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f04000a4680 tx=0x7f04000046c0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.677+0000 7f040fb85740 1 -- 172.21.15.31:0/1284937835 shutdown_connections 2024-03-21T10:46:26.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.677+0000 7f040fb85740 1 --2- 172.21.15.31:0/1284937835 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x556d5ae2c9b0 0x556d5ae2edd0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.677+0000 7f040fb85740 1 --2- 172.21.15.31:0/1284937835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5adc0de0 0x556d5adc6ae0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.677+0000 7f03e7e00640 0 client.27451 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:46:26.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.677+0000 7f040fb85740 1 --2- 172.21.15.31:0/1284937835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5adb8260 0x556d5adc64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.677+0000 7f040fb85740 1 --2- 172.21.15.31:0/1284937835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5adb78b0 0x556d5adc8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.677+0000 7f040fb85740 1 -- 172.21.15.31:0/1284937835 >> 172.21.15.31:0/1284937835 conn(0x556d5adb1f80 msgr2=0x556d5adb8640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:26.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.678+0000 7f040fb85740 1 -- 172.21.15.31:0/1284937835 shutdown_connections 2024-03-21T10:46:26.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.678+0000 7f040fb85740 1 -- 172.21.15.31:0/1284937835 wait complete. 2024-03-21T10:46:26.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.678+0000 7f040fb85740 20 client.27451 trim_cache size 0 max 16384 2024-03-21T10:46:26.680 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:46:26.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.678+0000 7f0407200640 1 -- 172.21.15.31:0/1127237954 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f03ec0a3090 msgr2=0x7f03ec0a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.678+0000 7f0407200640 1 --2- 172.21.15.31:0/1127237954 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f03ec0a3090 0x7f03ec0a5550 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f03f4006010 tx=0x7f03f4005a20 comp rx=0 tx=0).stop 2024-03-21T10:46:26.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.678+0000 7f0407200640 1 -- 172.21.15.31:0/1127237954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f040810ba50 msgr2=0x7f040819e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.678+0000 7f0407200640 1 --2- 172.21.15.31:0/1127237954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f040810ba50 0x7f040819e870 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f040000ec50 tx=0x7f040000c6a0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.678+0000 7f0407200640 1 -- 172.21.15.31:0/1127237954 shutdown_connections 2024-03-21T10:46:26.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.678+0000 7f0407200640 1 --2- 172.21.15.31:0/1127237954 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f03ec0a3090 0x7f03ec0a5550 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.678+0000 7f0407200640 1 --2- 172.21.15.31:0/1127237954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f040810ba50 0x7f040819e870 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.678+0000 7f0407200640 1 --2- 172.21.15.31:0/1127237954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0408102f50 0x7f040819a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.678+0000 7f0407200640 1 --2- 172.21.15.31:0/1127237954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0408102630 0x7f0408199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.678+0000 7f0407200640 1 -- 172.21.15.31:0/1127237954 >> 172.21.15.31:0/1127237954 conn(0x7f0408077130 msgr2=0x7f0408100250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:26.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.679+0000 7f0407200640 1 -- 172.21.15.31:0/1127237954 shutdown_connections 2024-03-21T10:46:26.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.679+0000 7f0407200640 1 -- 172.21.15.31:0/1127237954 wait complete. 2024-03-21T10:46:26.693 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:46:26.693 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-21T10:46:26.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.899+0000 7fee16a00640 1 -- 172.21.15.31:0/2070417353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee10100420 msgr2=0x7fee10100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.899+0000 7fee16a00640 1 --2- 172.21.15.31:0/2070417353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee10100420 0x7fee10100800 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fee04009900 tx=0x7fee0402bcc0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.899+0000 7fee16a00640 1 -- 172.21.15.31:0/2070417353 shutdown_connections 2024-03-21T10:46:26.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.899+0000 7fee16a00640 1 --2- 172.21.15.31:0/2070417353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee10105820 0x7fee10105c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.899+0000 7fee16a00640 1 --2- 172.21.15.31:0/2070417353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee10100d40 0x7fee10105200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.899+0000 7fee16a00640 1 --2- 172.21.15.31:0/2070417353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee10100420 0x7fee10100800 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.899+0000 7fee16a00640 1 -- 172.21.15.31:0/2070417353 >> 172.21.15.31:0/2070417353 conn(0x7fee10077130 msgr2=0x7fee100fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:26.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.899+0000 7fee16a00640 1 -- 172.21.15.31:0/2070417353 shutdown_connections 2024-03-21T10:46:26.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.899+0000 7fee16a00640 1 -- 172.21.15.31:0/2070417353 wait complete. 2024-03-21T10:46:26.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.901+0000 7fee16a00640 1 Processor -- start 2024-03-21T10:46:26.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.901+0000 7fee16a00640 1 -- start start 2024-03-21T10:46:26.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.901+0000 7fee16a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee10100420 0x7fee1019e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.901+0000 7fee16a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee10100d40 0x7fee1019e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.901+0000 7fee16a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee10105820 0x7fee101a2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.901+0000 7fee16a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee10117e80 con 0x7fee10100d40 2024-03-21T10:46:26.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.901+0000 7fee16a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee10117d00 con 0x7fee10105820 2024-03-21T10:46:26.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.901+0000 7fee16a00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee10118000 con 0x7fee10100420 2024-03-21T10:46:26.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.902+0000 7fee15600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee10100420 0x7fee1019e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.902+0000 7fee15600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee10100420 0x7fee1019e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53942/0 (socket says 172.21.15.31:53942) 2024-03-21T10:46:26.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.902+0000 7fee15600640 1 -- 172.21.15.31:0/3394188539 learned_addr learned my addr 172.21.15.31:0/3394188539 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:26.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.902+0000 7fee16000640 1 --2- 172.21.15.31:0/3394188539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee10105820 0x7fee101a2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.902+0000 7fee14c00640 1 --2- 172.21.15.31:0/3394188539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee10100d40 0x7fee1019e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.902+0000 7fee15600640 1 -- 172.21.15.31:0/3394188539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee10105820 msgr2=0x7fee101a2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.902+0000 7fee15600640 1 --2- 172.21.15.31:0/3394188539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee10105820 0x7fee101a2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.902+0000 7fee15600640 1 -- 172.21.15.31:0/3394188539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee10100d40 msgr2=0x7fee1019e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.902+0000 7fee15600640 1 --2- 172.21.15.31:0/3394188539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee10100d40 0x7fee1019e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.902+0000 7fee15600640 1 -- 172.21.15.31:0/3394188539 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee101a3310 con 0x7fee10100420 2024-03-21T10:46:26.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.902+0000 7fee14c00640 1 --2- 172.21.15.31:0/3394188539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee10100d40 0x7fee1019e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:46:26.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.902+0000 7fee15600640 1 --2- 172.21.15.31:0/3394188539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee10100420 0x7fee1019e320 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fee04000c00 tx=0x7fee040043c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:26.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.903+0000 7fee0e000640 1 -- 172.21.15.31:0/3394188539 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee04042020 con 0x7fee10100420 2024-03-21T10:46:26.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.903+0000 7fee0e000640 1 -- 172.21.15.31:0/3394188539 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee04005000 con 0x7fee10100420 2024-03-21T10:46:26.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.903+0000 7fee16a00640 1 -- 172.21.15.31:0/3394188539 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee101a35a0 con 0x7fee10100420 2024-03-21T10:46:26.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.903+0000 7fee0e000640 1 -- 172.21.15.31:0/3394188539 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee04003b00 con 0x7fee10100420 2024-03-21T10:46:26.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.903+0000 7fee16a00640 1 -- 172.21.15.31:0/3394188539 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee101aae90 con 0x7fee10100420 2024-03-21T10:46:26.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.907+0000 7fee1ad69740 1 Processor -- start 2024-03-21T10:46:26.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.908+0000 7fee1ad69740 1 -- start start 2024-03-21T10:46:26.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.908+0000 7fee0e000640 1 -- 172.21.15.31:0/3394188539 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fee04045020 con 0x7fee10100420 2024-03-21T10:46:26.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.908+0000 7fee1ad69740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6a18ee8b0 0x55c6a18eec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.908+0000 7fee1ad69740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6a18ef260 0x55c6a18f7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.908+0000 7fee1ad69740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6a18f7de0 0x55c6a18fa230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.908+0000 7fee1ad69740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6a18522c0 con 0x55c6a18ee8b0 2024-03-21T10:46:26.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.908+0000 7fee1ad69740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6a17fac90 con 0x55c6a18ef260 2024-03-21T10:46:26.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.908+0000 7fee1ad69740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6a184e360 con 0x55c6a18f7de0 2024-03-21T10:46:26.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.908+0000 7fee15600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6a18ef260 0x55c6a18f7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.908+0000 7fee16000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6a18ee8b0 0x55c6a18eec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.908+0000 7fee14c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6a18f7de0 0x55c6a18fa230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.908+0000 7fee15600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6a18ef260 0x55c6a18f7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38678/0 (socket says 172.21.15.31:38678) 2024-03-21T10:46:26.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.908+0000 7fee15600640 1 -- 172.21.15.31:0/2338624829 learned_addr learned my addr 172.21.15.31:0/2338624829 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:26.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.909+0000 7fee15600640 1 -- 172.21.15.31:0/2338624829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6a18f7de0 msgr2=0x55c6a18fa230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:26.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.909+0000 7fee15600640 1 --2- 172.21.15.31:0/2338624829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6a18f7de0 0x55c6a18fa230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.909+0000 7fee15600640 1 -- 172.21.15.31:0/2338624829 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6a18ee8b0 msgr2=0x55c6a18eec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:26.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.909+0000 7fee15600640 1 --2- 172.21.15.31:0/2338624829 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6a18ee8b0 0x55c6a18eec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.909+0000 7fee15600640 1 -- 172.21.15.31:0/2338624829 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6a181f080 con 0x55c6a18ef260 2024-03-21T10:46:26.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.909+0000 7fee15600640 1 --2- 172.21.15.31:0/2338624829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6a18ef260 0x55c6a18f7810 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fee040eec90 tx=0x7fee040f7b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99ec18114e9252fd server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:26.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.910+0000 7fee02600640 1 -- 172.21.15.31:0/2338624829 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee04100070 con 0x55c6a18ef260 2024-03-21T10:46:26.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.910+0000 7fee0e000640 1 --2- 172.21.15.31:0/3394188539 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fedf80a3090 0x7fedf80a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.910+0000 7fee02600640 1 -- 172.21.15.31:0/2338624829 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee040fc420 con 0x55c6a18ef260 2024-03-21T10:46:26.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.910+0000 7fee02600640 1 -- 172.21.15.31:0/2338624829 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee040f3330 con 0x55c6a18ef260 2024-03-21T10:46:26.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.910+0000 7fee0e000640 1 -- 172.21.15.31:0/3394188539 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fee040f18f0 con 0x7fee10100420 2024-03-21T10:46:26.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.910+0000 7fee16000640 1 --2- 172.21.15.31:0/3394188539 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fedf80a3090 0x7fedf80a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.910+0000 7fee1ad69740 1 -- 172.21.15.31:0/2338624829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6a18ef260 msgr2=0x55c6a18f7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.910+0000 7fee1ad69740 1 --2- 172.21.15.31:0/2338624829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6a18ef260 0x55c6a18f7810 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fee040eec90 tx=0x7fee040f7b20 comp rx=0 tx=0).stop 2024-03-21T10:46:26.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.911+0000 7fee1ad69740 1 -- 172.21.15.31:0/2338624829 shutdown_connections 2024-03-21T10:46:26.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.911+0000 7fee1ad69740 1 --2- 172.21.15.31:0/2338624829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6a18f7de0 0x55c6a18fa230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.911+0000 7fee1ad69740 1 --2- 172.21.15.31:0/2338624829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6a18ef260 0x55c6a18f7810 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.911+0000 7fee1ad69740 1 --2- 172.21.15.31:0/2338624829 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6a18ee8b0 0x55c6a18eec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.911+0000 7fee1ad69740 1 -- 172.21.15.31:0/2338624829 >> 172.21.15.31:0/2338624829 conn(0x55c6a18e8f80 msgr2=0x55c6a18e9360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:26.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.911+0000 7fee1ad69740 1 -- 172.21.15.31:0/2338624829 shutdown_connections 2024-03-21T10:46:26.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.911+0000 7fee1ad69740 1 -- 172.21.15.31:0/2338624829 wait complete. 2024-03-21T10:46:26.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.911+0000 7fee16000640 1 --2- 172.21.15.31:0/3394188539 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fedf80a3090 0x7fedf80a5550 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fee08005fd0 tx=0x7fee08009500 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:26.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.911+0000 7fee1ad69740 1 -- start start 2024-03-21T10:46:26.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.911+0000 7fee1ad69740 1 Processor -- start 2024-03-21T10:46:26.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.912+0000 7fee1ad69740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6a18ee8b0 0x55c6a18f7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.912+0000 7fee1ad69740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6a18ef260 0x55c6a18f4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.912+0000 7fee1ad69740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6a18f7de0 0x55c6a18f50e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.912+0000 7fee1ad69740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6a18fab10 con 0x55c6a18f7de0 2024-03-21T10:46:26.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.912+0000 7fee1ad69740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6a180e0d0 con 0x55c6a18ef260 2024-03-21T10:46:26.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.912+0000 7fee1ad69740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6a18fac90 con 0x55c6a18ee8b0 2024-03-21T10:46:26.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.912+0000 7fee1ad69740 10 client.0 _ll_register_callbacks cb 0x55c6a1812e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:46:26.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.912+0000 7fee14c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6a18ef260 0x55c6a18f4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.912+0000 7fee14c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6a18ef260 0x55c6a18f4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38684/0 (socket says 172.21.15.31:38684) 2024-03-21T10:46:26.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.912+0000 7fee14c00640 1 -- 172.21.15.31:0/629713555 learned_addr learned my addr 172.21.15.31:0/629713555 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:26.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.912+0000 7fee16000640 1 --2- 172.21.15.31:0/629713555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6a18f7de0 0x55c6a18f50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.912+0000 7fee15600640 1 --2- 172.21.15.31:0/629713555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6a18ee8b0 0x55c6a18f7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.913+0000 7fee14c00640 1 -- 172.21.15.31:0/629713555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6a18ee8b0 msgr2=0x55c6a18f7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:26.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.913+0000 7fee14c00640 1 --2- 172.21.15.31:0/629713555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6a18ee8b0 0x55c6a18f7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.913+0000 7fee14c00640 1 -- 172.21.15.31:0/629713555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6a18f7de0 msgr2=0x55c6a18f50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:26.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.913+0000 7fee14c00640 1 --2- 172.21.15.31:0/629713555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6a18f7de0 0x55c6a18f50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.913+0000 7fee14c00640 1 -- 172.21.15.31:0/629713555 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c6a15663f0 con 0x55c6a18ef260 2024-03-21T10:46:26.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.913+0000 7fee14c00640 1 --2- 172.21.15.31:0/629713555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6a18ef260 0x55c6a18f4ad0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00bf50 tx=0x7fedfc0128e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd0e4bd776f401e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:26.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.913+0000 7fedf6a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:46:26.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.913+0000 7fedf6a00640 1 -- 172.21.15.31:0/629713555 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc004440 con 0x55c6a18ef260 2024-03-21T10:46:26.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.913+0000 7fedf6a00640 1 -- 172.21.15.31:0/629713555 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fedfc0035f0 con 0x55c6a18ef260 2024-03-21T10:46:26.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.913+0000 7fee1ad69740 1 -- 172.21.15.31:0/629713555 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fee040fc420 con 0x55c6a18ef260 2024-03-21T10:46:26.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.913+0000 7fedf6a00640 1 -- 172.21.15.31:0/629713555 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedfc0037f0 con 0x55c6a18ef260 2024-03-21T10:46:26.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.914+0000 7fee1ad69740 10 client.27542 fetch_fsmap learned FSMap version 26 2024-03-21T10:46:26.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.914+0000 7fee1ad69740 1 -- 172.21.15.31:0/629713555 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55c6a16307d0 con 0x55c6a18ef260 2024-03-21T10:46:26.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.914+0000 7fedf6a00640 1 -- 172.21.15.31:0/629713555 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee040fc420 con 0x55c6a18ef260 2024-03-21T10:46:26.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.914+0000 7fedf6a00640 1 -- 172.21.15.31:0/629713555 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fedfc019070 con 0x55c6a18ef260 2024-03-21T10:46:26.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.914+0000 7fee1ad69740 10 client.27542 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:46:26.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.914+0000 7fee1ad69740 10 client.27542 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:46:26.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.914+0000 7fee1ad69740 20 client.27542 populate_metadata read hostname 'smithi031' 2024-03-21T10:46:26.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.914+0000 7fee1ad69740 1 --2- 172.21.15.31:0/629713555 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55c6a1907a80 0x55c6a1909ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.914+0000 7fee1ad69740 4 client.27542 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:46:26.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.914+0000 7fee1ad69740 1 -- 172.21.15.31:0/629713555 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c6a185f800 con 0x55c6a1907a80 2024-03-21T10:46:26.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.915+0000 7fee15600640 1 --2- 172.21.15.31:0/629713555 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55c6a1907a80 0x55c6a1909ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.915+0000 7fee15600640 1 --2- 172.21.15.31:0/629713555 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55c6a1907a80 0x55c6a1909ec0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cb2695cdf7d3186b server_cookie=6897590cae52d6a4 in_seq=0 out_seq=0 2024-03-21T10:46:26.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.915+0000 7fedf6a00640 10 client.27542 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:46:26.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.916+0000 7fedf6a00640 1 -- 172.21.15.31:0/629713555 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c6a185f800 con 0x55c6a1907a80 2024-03-21T10:46:26.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.916+0000 7fedf6a00640 10 client.27542 handle_command_reply: tid=0 2024-03-21T10:46:26.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.916+0000 7fee1ad69740 1 client.27542 shutdown 2024-03-21T10:46:26.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.916+0000 7fee1ad69740 1 -- 172.21.15.31:0/629713555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6a18ef260 msgr2=0x55c6a18f4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.916+0000 7fee1ad69740 1 --2- 172.21.15.31:0/629713555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6a18ef260 0x55c6a18f4ad0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00bf50 tx=0x7fedfc0128e0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.916+0000 7fee1ad69740 1 -- 172.21.15.31:0/629713555 shutdown_connections 2024-03-21T10:46:26.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.916+0000 7fee1ad69740 1 --2- 172.21.15.31:0/629713555 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55c6a1907a80 0x55c6a1909ec0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.916+0000 7fee1ad69740 1 --2- 172.21.15.31:0/629713555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6a18f7de0 0x55c6a18f50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.916+0000 7fee1ad69740 1 --2- 172.21.15.31:0/629713555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6a18ef260 0x55c6a18f4ad0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.916+0000 7fee1ad69740 1 --2- 172.21.15.31:0/629713555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6a18ee8b0 0x55c6a18f7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.916+0000 7fedf6a00640 0 client.27542 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:46:26.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.916+0000 7fee1ad69740 1 -- 172.21.15.31:0/629713555 >> 172.21.15.31:0/629713555 conn(0x55c6a18e8f80 msgr2=0x55c6a18eacd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:26.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.916+0000 7fee1ad69740 1 -- 172.21.15.31:0/629713555 shutdown_connections 2024-03-21T10:46:26.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.916+0000 7fee1ad69740 1 -- 172.21.15.31:0/629713555 wait complete. 2024-03-21T10:46:26.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.917+0000 7fee1ad69740 20 client.27542 trim_cache size 0 max 16384 2024-03-21T10:46:26.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.930+0000 7fee1ad69740 1 Processor -- start 2024-03-21T10:46:26.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.930+0000 7fee1ad69740 1 -- start start 2024-03-21T10:46:26.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.931+0000 7fee1ad69740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6a18ee8b0 0x55c6a18ffc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.931+0000 7fee1ad69740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6a18ef260 0x55c6a19041d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.931+0000 7fee1ad69740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6a18f7de0 0x55c6a18f7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.931+0000 7fee1ad69740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee101b7300 con 0x55c6a18ef260 2024-03-21T10:46:26.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.931+0000 7fee1ad69740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee101b4e00 con 0x55c6a18f7de0 2024-03-21T10:46:26.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.931+0000 7fee1ad69740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee101b2ec0 con 0x55c6a18ee8b0 2024-03-21T10:46:26.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.931+0000 7fee14c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6a18ef260 0x55c6a19041d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.931+0000 7fee15600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6a18ee8b0 0x55c6a18ffc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.931+0000 7fee16000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6a18f7de0 0x55c6a18f7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.931+0000 7fee15600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6a18ee8b0 0x55c6a18ffc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:45202/0 (socket says 172.21.15.31:45202) 2024-03-21T10:46:26.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.931+0000 7fee15600640 1 -- 172.21.15.31:0/2853952990 learned_addr learned my addr 172.21.15.31:0/2853952990 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:26.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.931+0000 7fee15600640 1 -- 172.21.15.31:0/2853952990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6a18f7de0 msgr2=0x55c6a18f7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:26.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.931+0000 7fee15600640 1 --2- 172.21.15.31:0/2853952990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6a18f7de0 0x55c6a18f7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.931+0000 7fee15600640 1 -- 172.21.15.31:0/2853952990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6a18ef260 msgr2=0x55c6a19041d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:26.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.931+0000 7fee15600640 1 --2- 172.21.15.31:0/2853952990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6a18ef260 0x55c6a19041d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.931+0000 7fee15600640 1 -- 172.21.15.31:0/2853952990 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6a18f5450 con 0x55c6a18ee8b0 2024-03-21T10:46:26.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.932+0000 7fee15600640 1 --2- 172.21.15.31:0/2853952990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6a18ee8b0 0x55c6a18ffc40 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fee040bd980 tx=0x7fee04005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95280ca41c0df7f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:26.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.932+0000 7fee01c00640 1 -- 172.21.15.31:0/2853952990 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee04100040 con 0x55c6a18ee8b0 2024-03-21T10:46:26.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.932+0000 7fee01c00640 1 -- 172.21.15.31:0/2853952990 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee040be0f0 con 0x55c6a18ee8b0 2024-03-21T10:46:26.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.932+0000 7fee01c00640 1 -- 172.21.15.31:0/2853952990 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee040bd530 con 0x55c6a18ee8b0 2024-03-21T10:46:26.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.932+0000 7fee1ad69740 1 -- 172.21.15.31:0/2853952990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6a18ee8b0 msgr2=0x55c6a18ffc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.932+0000 7fee1ad69740 1 --2- 172.21.15.31:0/2853952990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6a18ee8b0 0x55c6a18ffc40 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fee040bd980 tx=0x7fee04005d50 comp rx=0 tx=0).stop 2024-03-21T10:46:26.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.932+0000 7fee1ad69740 1 -- 172.21.15.31:0/2853952990 shutdown_connections 2024-03-21T10:46:26.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.932+0000 7fee1ad69740 1 --2- 172.21.15.31:0/2853952990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6a18f7de0 0x55c6a18f7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.932+0000 7fee1ad69740 1 --2- 172.21.15.31:0/2853952990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6a18ef260 0x55c6a19041d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.932+0000 7fee1ad69740 1 --2- 172.21.15.31:0/2853952990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6a18ee8b0 0x55c6a18ffc40 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.932+0000 7fee1ad69740 1 -- 172.21.15.31:0/2853952990 >> 172.21.15.31:0/2853952990 conn(0x55c6a18e8f80 msgr2=0x55c6a18e8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:26.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.933+0000 7fee1ad69740 1 -- 172.21.15.31:0/2853952990 shutdown_connections 2024-03-21T10:46:26.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.933+0000 7fee1ad69740 1 -- 172.21.15.31:0/2853952990 wait complete. 2024-03-21T10:46:26.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.933+0000 7fee1ad69740 1 -- start start 2024-03-21T10:46:26.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.933+0000 7fee1ad69740 1 Processor -- start 2024-03-21T10:46:26.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.934+0000 7fee1ad69740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6a18ee8b0 0x55c6a18fdca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.934+0000 7fee1ad69740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6a18ef260 0x55c6a18fe1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.934+0000 7fee1ad69740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6a18f7de0 0x55c6a18ff160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.934+0000 7fee1ad69740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee101b34c0 con 0x55c6a18ee8b0 2024-03-21T10:46:26.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.934+0000 7fee1ad69740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee101b47a0 con 0x55c6a18f7de0 2024-03-21T10:46:26.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.934+0000 7fee1ad69740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee101b3ae0 con 0x55c6a18ef260 2024-03-21T10:46:26.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.934+0000 7fee1ad69740 10 client.0 _ll_register_callbacks cb 0x7fedfc00b800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:46:26.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.934+0000 7fee14c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6a18ef260 0x55c6a18fe1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.934+0000 7fee15600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6a18ee8b0 0x55c6a18fdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.934+0000 7fee14c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6a18ef260 0x55c6a18fe1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:45216/0 (socket says 172.21.15.31:45216) 2024-03-21T10:46:26.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.934+0000 7fee14c00640 1 -- 172.21.15.31:0/3159301879 learned_addr learned my addr 172.21.15.31:0/3159301879 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:26.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.934+0000 7fee15600640 1 --2- 172.21.15.31:0/3159301879 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6a18ee8b0 0x55c6a18fdca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44558/0 (socket says 172.21.15.31:44558) 2024-03-21T10:46:26.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.934+0000 7fee16000640 1 --2- 172.21.15.31:0/3159301879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6a18f7de0 0x55c6a18ff160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.934+0000 7fee14c00640 1 -- 172.21.15.31:0/3159301879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6a18f7de0 msgr2=0x55c6a18ff160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:26.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.934+0000 7fee14c00640 1 --2- 172.21.15.31:0/3159301879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6a18f7de0 0x55c6a18ff160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.934+0000 7fee14c00640 1 -- 172.21.15.31:0/3159301879 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6a18ee8b0 msgr2=0x55c6a18fdca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:26.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.934+0000 7fee14c00640 1 --2- 172.21.15.31:0/3159301879 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6a18ee8b0 0x55c6a18fdca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.935+0000 7fee14c00640 1 -- 172.21.15.31:0/3159301879 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c6a18ff850 con 0x55c6a18ef260 2024-03-21T10:46:26.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.935+0000 7fee15600640 1 --2- 172.21.15.31:0/3159301879 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6a18ee8b0 0x55c6a18fdca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:46:26.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.935+0000 7fee14c00640 1 --2- 172.21.15.31:0/3159301879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6a18ef260 0x55c6a18fe1e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fedfc0059f0 tx=0x7fedfc003bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d97270d66ad5d10a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:26.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.935+0000 7fedf6a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:46:26.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.935+0000 7fedf6a00640 1 -- 172.21.15.31:0/3159301879 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc02d020 con 0x55c6a18ef260 2024-03-21T10:46:26.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.935+0000 7fee1ad69740 1 -- 172.21.15.31:0/3159301879 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fee040be0f0 con 0x55c6a18ef260 2024-03-21T10:46:26.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.935+0000 7fedf6a00640 1 -- 172.21.15.31:0/3159301879 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fedfc00ba90 con 0x55c6a18ef260 2024-03-21T10:46:26.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.935+0000 7fedf6a00640 1 -- 172.21.15.31:0/3159301879 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedfc038050 con 0x55c6a18ef260 2024-03-21T10:46:26.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.935+0000 7fee1ad69740 10 client.27469 fetch_fsmap learned FSMap version 26 2024-03-21T10:46:26.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.935+0000 7fee1ad69740 1 -- 172.21.15.31:0/3159301879 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55c6a1963f90 con 0x55c6a18ef260 2024-03-21T10:46:26.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.936+0000 7fedf6a00640 1 -- 172.21.15.31:0/3159301879 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee040be0f0 con 0x55c6a18ef260 2024-03-21T10:46:26.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.936+0000 7fedf6a00640 1 -- 172.21.15.31:0/3159301879 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fedfc019040 con 0x55c6a18ef260 2024-03-21T10:46:26.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.936+0000 7fee1ad69740 10 client.27469 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:46:26.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.936+0000 7fee1ad69740 10 client.27469 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:46:26.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.936+0000 7fee1ad69740 20 client.27469 populate_metadata read hostname 'smithi031' 2024-03-21T10:46:26.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.936+0000 7fee1ad69740 1 --2- 172.21.15.31:0/3159301879 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55c6a19646e0 0x55c6a1966b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:26.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.936+0000 7fee1ad69740 4 client.27469 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:46:26.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.936+0000 7fee1ad69740 1 -- 172.21.15.31:0/3159301879 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c6a1859b50 con 0x55c6a19646e0 2024-03-21T10:46:26.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.936+0000 7fee15600640 1 --2- 172.21.15.31:0/3159301879 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55c6a19646e0 0x55c6a1966b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:26.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.937+0000 7fee15600640 1 --2- 172.21.15.31:0/3159301879 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55c6a19646e0 0x55c6a1966b00 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=60a6456f8599cc1d server_cookie=f0e12a1959a50117 in_seq=0 out_seq=0 2024-03-21T10:46:26.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.937+0000 7fedf6a00640 10 client.27469 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:46:26.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.937+0000 7fedf6a00640 1 -- 172.21.15.31:0/3159301879 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c6a1859b50 con 0x55c6a19646e0 2024-03-21T10:46:26.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.937+0000 7fedf6a00640 10 client.27469 handle_command_reply: tid=0 2024-03-21T10:46:26.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.937+0000 7fee1ad69740 1 client.27469 shutdown 2024-03-21T10:46:26.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.937+0000 7fee1ad69740 1 -- 172.21.15.31:0/3159301879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6a18ef260 msgr2=0x55c6a18fe1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.937+0000 7fee1ad69740 1 --2- 172.21.15.31:0/3159301879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6a18ef260 0x55c6a18fe1e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fedfc0059f0 tx=0x7fedfc003bc0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.937+0000 7fee1ad69740 1 -- 172.21.15.31:0/3159301879 shutdown_connections 2024-03-21T10:46:26.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.937+0000 7fee1ad69740 1 --2- 172.21.15.31:0/3159301879 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55c6a19646e0 0x55c6a1966b00 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.937+0000 7fee1ad69740 1 --2- 172.21.15.31:0/3159301879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6a18f7de0 0x55c6a18ff160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.937+0000 7fee1ad69740 1 --2- 172.21.15.31:0/3159301879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c6a18ef260 0x55c6a18fe1e0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.937+0000 7fee1ad69740 1 --2- 172.21.15.31:0/3159301879 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6a18ee8b0 0x55c6a18fdca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.937+0000 7fedf6a00640 0 client.27469 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:46:26.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.937+0000 7fee1ad69740 1 -- 172.21.15.31:0/3159301879 >> 172.21.15.31:0/3159301879 conn(0x55c6a18e8f80 msgr2=0x55c6a18efed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:26.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.938+0000 7fee1ad69740 1 -- 172.21.15.31:0/3159301879 shutdown_connections 2024-03-21T10:46:26.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.938+0000 7fee1ad69740 1 -- 172.21.15.31:0/3159301879 wait complete. 2024-03-21T10:46:26.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.938+0000 7fee1ad69740 20 client.27469 trim_cache size 0 max 16384 2024-03-21T10:46:26.938 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:46:26.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.938+0000 7fee16a00640 1 -- 172.21.15.31:0/3394188539 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fedf80a3090 msgr2=0x7fedf80a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.938+0000 7fee16a00640 1 --2- 172.21.15.31:0/3394188539 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fedf80a3090 0x7fedf80a5550 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fee08005fd0 tx=0x7fee08009500 comp rx=0 tx=0).stop 2024-03-21T10:46:26.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.938+0000 7fee16a00640 1 -- 172.21.15.31:0/3394188539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee10100420 msgr2=0x7fee1019e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:26.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.938+0000 7fee16a00640 1 --2- 172.21.15.31:0/3394188539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee10100420 0x7fee1019e320 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fee04000c00 tx=0x7fee040043c0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.939+0000 7fee16a00640 1 -- 172.21.15.31:0/3394188539 shutdown_connections 2024-03-21T10:46:26.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.939+0000 7fee16a00640 1 --2- 172.21.15.31:0/3394188539 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fedf80a3090 0x7fedf80a5550 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.939+0000 7fee16a00640 1 --2- 172.21.15.31:0/3394188539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee10105820 0x7fee101a2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.939+0000 7fee16a00640 1 --2- 172.21.15.31:0/3394188539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee10100d40 0x7fee1019e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.939+0000 7fee16a00640 1 --2- 172.21.15.31:0/3394188539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee10100420 0x7fee1019e320 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:26.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.939+0000 7fee16a00640 1 -- 172.21.15.31:0/3394188539 >> 172.21.15.31:0/3394188539 conn(0x7fee10077130 msgr2=0x7fee101031d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:26.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.939+0000 7fee16a00640 1 -- 172.21.15.31:0/3394188539 shutdown_connections 2024-03-21T10:46:26.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:26.939+0000 7fee16a00640 1 -- 172.21.15.31:0/3394188539 wait complete. 2024-03-21T10:46:26.958 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:46:26.958 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-21T10:46:27.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.183+0000 7f88a9200640 1 -- 172.21.15.31:0/1060360467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88a4107aa0 msgr2=0x7f88a4107f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:27.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.183+0000 7f88a9200640 1 --2- 172.21.15.31:0/1060360467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88a4107aa0 0x7f88a4107f00 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8890009900 tx=0x7f889002bcd0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.183+0000 7f88a9200640 1 -- 172.21.15.31:0/1060360467 shutdown_connections 2024-03-21T10:46:27.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.183+0000 7f88a9200640 1 --2- 172.21.15.31:0/1060360467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88a4108440 0x7f88a4114900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.183+0000 7f88a9200640 1 --2- 172.21.15.31:0/1060360467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88a4107aa0 0x7f88a4107f00 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.183+0000 7f88a9200640 1 --2- 172.21.15.31:0/1060360467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88a410c980 0x7f88a410cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.183+0000 7f88a9200640 1 -- 172.21.15.31:0/1060360467 >> 172.21.15.31:0/1060360467 conn(0x7f88a40772a0 msgr2=0x7f88a40796c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:27.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.183+0000 7f88a9200640 1 -- 172.21.15.31:0/1060360467 shutdown_connections 2024-03-21T10:46:27.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.183+0000 7f88a9200640 1 -- 172.21.15.31:0/1060360467 wait complete. 2024-03-21T10:46:27.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.184+0000 7f88a9200640 1 Processor -- start 2024-03-21T10:46:27.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.184+0000 7f88a9200640 1 -- start start 2024-03-21T10:46:27.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.185+0000 7f88a9200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88a4107aa0 0x7f88a419e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:27.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.185+0000 7f88a9200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88a4108440 0x7f88a419e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:27.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.185+0000 7f88a9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88a410c980 0x7f88a41a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:27.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.185+0000 7f88a9200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a4117e90 con 0x7f88a4108440 2024-03-21T10:46:27.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.185+0000 7f88a9200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a4117d10 con 0x7f88a410c980 2024-03-21T10:46:27.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.185+0000 7f88a9200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a4118010 con 0x7f88a4107aa0 2024-03-21T10:46:27.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.185+0000 7f88a3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88a4107aa0 0x7f88a419e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:27.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.185+0000 7f88a3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88a4107aa0 0x7f88a419e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:45224/0 (socket says 172.21.15.31:45224) 2024-03-21T10:46:27.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.185+0000 7f88a3400640 1 -- 172.21.15.31:0/2192953701 learned_addr learned my addr 172.21.15.31:0/2192953701 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:27.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.185+0000 7f88a2a00640 1 --2- 172.21.15.31:0/2192953701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88a4108440 0x7f88a419e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:27.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.185+0000 7f88a3e00640 1 --2- 172.21.15.31:0/2192953701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88a410c980 0x7f88a41a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:27.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.186+0000 7f88a3400640 1 -- 172.21.15.31:0/2192953701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88a410c980 msgr2=0x7f88a41a2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:27.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.186+0000 7f88a3400640 1 --2- 172.21.15.31:0/2192953701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88a410c980 0x7f88a41a2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.186+0000 7f88a3400640 1 -- 172.21.15.31:0/2192953701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88a4108440 msgr2=0x7f88a419e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:27.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.186+0000 7f88a3400640 1 --2- 172.21.15.31:0/2192953701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88a4108440 0x7f88a419e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.186+0000 7f88a3400640 1 -- 172.21.15.31:0/2192953701 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a41a33a0 con 0x7f88a4107aa0 2024-03-21T10:46:27.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.186+0000 7f88a3e00640 1 --2- 172.21.15.31:0/2192953701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88a410c980 0x7f88a41a2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:46:27.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.186+0000 7f88a3400640 1 --2- 172.21.15.31:0/2192953701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88a4107aa0 0x7f88a419e3b0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f889400d950 tx=0x7f889400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:27.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.186+0000 7f889be00640 1 -- 172.21.15.31:0/2192953701 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8894017070 con 0x7f88a4107aa0 2024-03-21T10:46:27.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.186+0000 7f889be00640 1 -- 172.21.15.31:0/2192953701 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8894009c30 con 0x7f88a4107aa0 2024-03-21T10:46:27.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.186+0000 7f88a9200640 1 -- 172.21.15.31:0/2192953701 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a41a3690 con 0x7f88a4107aa0 2024-03-21T10:46:27.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.186+0000 7f889be00640 1 -- 172.21.15.31:0/2192953701 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889400f6b0 con 0x7f88a4107aa0 2024-03-21T10:46:27.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.186+0000 7f88a9200640 1 -- 172.21.15.31:0/2192953701 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a41aaf20 con 0x7f88a4107aa0 2024-03-21T10:46:27.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.189+0000 7f889be00640 1 -- 172.21.15.31:0/2192953701 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f88940109d0 con 0x7f88a4107aa0 2024-03-21T10:46:27.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.189+0000 7f88ad628740 1 Processor -- start 2024-03-21T10:46:27.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.190+0000 7f88ad628740 1 -- start start 2024-03-21T10:46:27.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.190+0000 7f88ad628740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ea84f98b0 0x564ea84f9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:27.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.190+0000 7f88ad628740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ea84fa260 0x564ea8502810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:27.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.190+0000 7f88ad628740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea8502de0 0x564ea8505230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:27.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.190+0000 7f88ad628740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ea845d2c0 con 0x564ea8502de0 2024-03-21T10:46:27.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.190+0000 7f88ad628740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ea8405c90 con 0x564ea84fa260 2024-03-21T10:46:27.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.190+0000 7f88ad628740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ea8459360 con 0x564ea84f98b0 2024-03-21T10:46:27.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.190+0000 7f88a3e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ea84f98b0 0x564ea84f9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:27.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.190+0000 7f88a3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ea84fa260 0x564ea8502810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:27.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.190+0000 7f88a3e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ea84f98b0 0x564ea84f9c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:45234/0 (socket says 172.21.15.31:45234) 2024-03-21T10:46:27.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.190+0000 7f88a3e00640 1 -- 172.21.15.31:0/346420459 learned_addr learned my addr 172.21.15.31:0/346420459 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:27.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.190+0000 7f88a2a00640 1 --2- 172.21.15.31:0/346420459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea8502de0 0x564ea8505230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:27.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.191+0000 7f88a3e00640 1 -- 172.21.15.31:0/346420459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ea84fa260 msgr2=0x564ea8502810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:27.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.191+0000 7f88a3e00640 1 --2- 172.21.15.31:0/346420459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ea84fa260 0x564ea8502810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.191+0000 7f88a3e00640 1 -- 172.21.15.31:0/346420459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea8502de0 msgr2=0x564ea8505230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:27.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.191+0000 7f88a3e00640 1 --2- 172.21.15.31:0/346420459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea8502de0 0x564ea8505230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.191+0000 7f88a3e00640 1 -- 172.21.15.31:0/346420459 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564ea842a080 con 0x564ea84f98b0 2024-03-21T10:46:27.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.191+0000 7f88a3e00640 1 --2- 172.21.15.31:0/346420459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ea84f98b0 0x564ea84f9c90 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f889c010750 tx=0x7f889c010c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19e0f7271981041e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:27.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.191+0000 7f8898c00640 1 -- 172.21.15.31:0/346420459 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889c00ead0 con 0x564ea84f98b0 2024-03-21T10:46:27.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.191+0000 7f8898c00640 1 -- 172.21.15.31:0/346420459 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f889c010e60 con 0x564ea84f98b0 2024-03-21T10:46:27.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.191+0000 7f8898c00640 1 -- 172.21.15.31:0/346420459 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889c011640 con 0x564ea84f98b0 2024-03-21T10:46:27.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.191+0000 7f889be00640 1 --2- 172.21.15.31:0/2192953701 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f88880a3110 0x7f88880a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:27.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.192+0000 7f889be00640 1 -- 172.21.15.31:0/2192953701 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f88940cf690 con 0x7f88a4107aa0 2024-03-21T10:46:27.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.192+0000 7f88a3400640 1 --2- 172.21.15.31:0/2192953701 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f88880a3110 0x7f88880a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:27.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.192+0000 7f88ad628740 1 -- 172.21.15.31:0/346420459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ea84f98b0 msgr2=0x564ea84f9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:27.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.192+0000 7f88ad628740 1 --2- 172.21.15.31:0/346420459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ea84f98b0 0x564ea84f9c90 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f889c010750 tx=0x7f889c010c10 comp rx=0 tx=0).stop 2024-03-21T10:46:27.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.192+0000 7f88ad628740 1 -- 172.21.15.31:0/346420459 shutdown_connections 2024-03-21T10:46:27.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.192+0000 7f88ad628740 1 --2- 172.21.15.31:0/346420459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea8502de0 0x564ea8505230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.192+0000 7f88ad628740 1 --2- 172.21.15.31:0/346420459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ea84fa260 0x564ea8502810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.192+0000 7f88ad628740 1 --2- 172.21.15.31:0/346420459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ea84f98b0 0x564ea84f9c90 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.192+0000 7f88ad628740 1 -- 172.21.15.31:0/346420459 >> 172.21.15.31:0/346420459 conn(0x564ea84f3f80 msgr2=0x564ea84f4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:27.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.192+0000 7f88ad628740 1 -- 172.21.15.31:0/346420459 shutdown_connections 2024-03-21T10:46:27.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.192+0000 7f88ad628740 1 -- 172.21.15.31:0/346420459 wait complete. 2024-03-21T10:46:27.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.192+0000 7f88a3400640 1 --2- 172.21.15.31:0/2192953701 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f88880a3110 0x7f88880a55d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x564ea84f9380 tx=0x7f88940d5530 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:27.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.193+0000 7f88ad628740 1 -- start start 2024-03-21T10:46:27.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.193+0000 7f88ad628740 1 Processor -- start 2024-03-21T10:46:27.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.193+0000 7f88ad628740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea84f98b0 0x564ea8502710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:27.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.194+0000 7f88ad628740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ea84fa260 0x564ea84ffaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:27.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.194+0000 7f88ad628740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ea8502de0 0x564ea85000b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:27.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.194+0000 7f88ad628740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ea8505d50 con 0x564ea84f98b0 2024-03-21T10:46:27.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.194+0000 7f88ad628740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ea84190d0 con 0x564ea84fa260 2024-03-21T10:46:27.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.194+0000 7f88ad628740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ea8505ed0 con 0x564ea8502de0 2024-03-21T10:46:27.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.194+0000 7f88ad628740 10 client.0 _ll_register_callbacks cb 0x564ea841de80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:46:27.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.194+0000 7f88a3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea84f98b0 0x564ea8502710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:27.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.194+0000 7f88a2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ea84fa260 0x564ea84ffaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:27.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.194+0000 7f88a3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea84f98b0 0x564ea8502710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44596/0 (socket says 172.21.15.31:44596) 2024-03-21T10:46:27.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.194+0000 7f88a3e00640 1 -- 172.21.15.31:0/1844583582 learned_addr learned my addr 172.21.15.31:0/1844583582 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:27.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.194+0000 7f88a3e00640 1 --2- 172.21.15.31:0/1844583582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ea8502de0 0x564ea85000b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:27.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.194+0000 7f88a2a00640 1 -- 172.21.15.31:0/1844583582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ea8502de0 msgr2=0x564ea85000b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:27.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.194+0000 7f88a2a00640 1 --2- 172.21.15.31:0/1844583582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ea8502de0 0x564ea85000b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.194+0000 7f88a2a00640 1 -- 172.21.15.31:0/1844583582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea84f98b0 msgr2=0x564ea8502710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:27.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.194+0000 7f88a2a00640 1 --2- 172.21.15.31:0/1844583582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea84f98b0 0x564ea8502710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.194+0000 7f88a2a00640 1 -- 172.21.15.31:0/1844583582 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564ea81713f0 con 0x564ea84fa260 2024-03-21T10:46:27.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.194+0000 7f88a3e00640 1 --2- 172.21.15.31:0/1844583582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea84f98b0 0x564ea8502710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:46:27.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.195+0000 7f88a2a00640 1 --2- 172.21.15.31:0/1844583582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ea84fa260 0x564ea84ffaa0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f889002c210 tx=0x7f8890009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a21e615be89caa6e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:27.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.195+0000 7f888d600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:46:27.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.195+0000 7f888d600640 1 -- 172.21.15.31:0/1844583582 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8890043070 con 0x564ea84fa260 2024-03-21T10:46:27.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.195+0000 7f888d600640 1 -- 172.21.15.31:0/1844583582 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f88900046c0 con 0x564ea84fa260 2024-03-21T10:46:27.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.195+0000 7f88ad628740 1 -- 172.21.15.31:0/1844583582 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f889c010e60 con 0x564ea84fa260 2024-03-21T10:46:27.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.195+0000 7f888d600640 1 -- 172.21.15.31:0/1844583582 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88900048c0 con 0x564ea84fa260 2024-03-21T10:46:27.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.195+0000 7f88ad628740 10 client.27560 fetch_fsmap learned FSMap version 26 2024-03-21T10:46:27.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.195+0000 7f88ad628740 1 -- 172.21.15.31:0/1844583582 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x564ea823b7d0 con 0x564ea84fa260 2024-03-21T10:46:27.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.196+0000 7f888d600640 1 -- 172.21.15.31:0/1844583582 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f889c010e60 con 0x564ea84fa260 2024-03-21T10:46:27.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.196+0000 7f888d600640 1 -- 172.21.15.31:0/1844583582 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f8890003710 con 0x564ea84fa260 2024-03-21T10:46:27.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.196+0000 7f88ad628740 10 client.27560 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:46:27.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.196+0000 7f88ad628740 10 client.27560 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:46:27.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.196+0000 7f88ad628740 20 client.27560 populate_metadata read hostname 'smithi031' 2024-03-21T10:46:27.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.196+0000 7f88ad628740 1 --2- 172.21.15.31:0/1844583582 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564ea8512c60 0x564ea8515050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:27.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.196+0000 7f88ad628740 4 client.27560 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:46:27.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.196+0000 7f88ad628740 1 -- 172.21.15.31:0/1844583582 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564ea846a800 con 0x564ea8512c60 2024-03-21T10:46:27.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.196+0000 7f88a3e00640 1 --2- 172.21.15.31:0/1844583582 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564ea8512c60 0x564ea8515050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:27.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.197+0000 7f888d600640 10 client.27560 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:27.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.197+0000 7f88a3e00640 1 --2- 172.21.15.31:0/1844583582 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564ea8512c60 0x564ea8515050 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5faf2fdece245c35 server_cookie=3959993e6d5205b1 in_seq=0 out_seq=0 2024-03-21T10:46:27.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.197+0000 7f888d600640 1 -- 172.21.15.31:0/1844583582 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564ea846a800 con 0x564ea8512c60 2024-03-21T10:46:27.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.197+0000 7f888d600640 10 client.27560 handle_command_reply: tid=0 2024-03-21T10:46:27.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.197+0000 7f88ad628740 1 client.27560 shutdown 2024-03-21T10:46:27.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.198+0000 7f88ad628740 1 -- 172.21.15.31:0/1844583582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ea84fa260 msgr2=0x564ea84ffaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:27.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.198+0000 7f88ad628740 1 --2- 172.21.15.31:0/1844583582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ea84fa260 0x564ea84ffaa0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f889002c210 tx=0x7f8890009560 comp rx=0 tx=0).stop 2024-03-21T10:46:27.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.198+0000 7f88ad628740 1 -- 172.21.15.31:0/1844583582 shutdown_connections 2024-03-21T10:46:27.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.198+0000 7f88ad628740 1 --2- 172.21.15.31:0/1844583582 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564ea8512c60 0x564ea8515050 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.198+0000 7f88ad628740 1 --2- 172.21.15.31:0/1844583582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ea8502de0 0x564ea85000b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.198+0000 7f88ad628740 1 --2- 172.21.15.31:0/1844583582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ea84fa260 0x564ea84ffaa0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.198+0000 7f88ad628740 1 --2- 172.21.15.31:0/1844583582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea84f98b0 0x564ea8502710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.198+0000 7f888d600640 0 client.27560 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:27.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.198+0000 7f88ad628740 1 -- 172.21.15.31:0/1844583582 >> 172.21.15.31:0/1844583582 conn(0x564ea84f3f80 msgr2=0x564ea84f5cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:27.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.198+0000 7f88ad628740 1 -- 172.21.15.31:0/1844583582 shutdown_connections 2024-03-21T10:46:27.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.198+0000 7f88ad628740 1 -- 172.21.15.31:0/1844583582 wait complete. 2024-03-21T10:46:27.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.198+0000 7f88ad628740 20 client.27560 trim_cache size 0 max 16384 2024-03-21T10:46:27.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.211+0000 7f88ad628740 1 Processor -- start 2024-03-21T10:46:27.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.212+0000 7f88ad628740 1 -- start start 2024-03-21T10:46:27.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.212+0000 7f88ad628740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea84f98b0 0x564ea85021d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:27.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.212+0000 7f88ad628740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ea84fa260 0x564ea8502710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:27.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.212+0000 7f88ad628740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ea8502de0 0x564ea84fcd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:27.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.212+0000 7f88ad628740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a41b73a0 con 0x564ea84f98b0 2024-03-21T10:46:27.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.212+0000 7f88ad628740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a41b4ea0 con 0x564ea84fa260 2024-03-21T10:46:27.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.212+0000 7f88ad628740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a41b2f60 con 0x564ea8502de0 2024-03-21T10:46:27.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.212+0000 7f88a2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ea84fa260 0x564ea8502710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:27.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.212+0000 7f88a2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ea84fa260 0x564ea8502710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:54498/0 (socket says 172.21.15.31:54498) 2024-03-21T10:46:27.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.212+0000 7f88a2a00640 1 -- 172.21.15.31:0/985047885 learned_addr learned my addr 172.21.15.31:0/985047885 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:27.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.212+0000 7f88a3e00640 1 --2- 172.21.15.31:0/985047885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea84f98b0 0x564ea85021d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:27.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.213+0000 7f88a3e00640 1 --2- 172.21.15.31:0/985047885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ea8502de0 0x564ea84fcd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:27.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.213+0000 7f88a2a00640 1 -- 172.21.15.31:0/985047885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ea8502de0 msgr2=0x564ea84fcd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:27.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.213+0000 7f88a2a00640 1 --2- 172.21.15.31:0/985047885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ea8502de0 0x564ea84fcd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.213+0000 7f88a2a00640 1 -- 172.21.15.31:0/985047885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea84f98b0 msgr2=0x564ea85021d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:27.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.213+0000 7f88a2a00640 1 --2- 172.21.15.31:0/985047885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea84f98b0 0x564ea85021d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.213+0000 7f88a2a00640 1 -- 172.21.15.31:0/985047885 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564ea85153c0 con 0x564ea84fa260 2024-03-21T10:46:27.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.213+0000 7f88a3e00640 1 --2- 172.21.15.31:0/985047885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea84f98b0 0x564ea85021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:46:27.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.213+0000 7f88a2a00640 1 --2- 172.21.15.31:0/985047885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ea84fa260 0x564ea8502710 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f889002f020 tx=0x7f8890000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=490cf8adc101fc2e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:27.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.213+0000 7f888fe00640 1 -- 172.21.15.31:0/985047885 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8890043040 con 0x564ea84fa260 2024-03-21T10:46:27.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.213+0000 7f888fe00640 1 -- 172.21.15.31:0/985047885 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8890004460 con 0x564ea84fa260 2024-03-21T10:46:27.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.213+0000 7f888fe00640 1 -- 172.21.15.31:0/985047885 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88900035f0 con 0x564ea84fa260 2024-03-21T10:46:27.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.214+0000 7f88ad628740 1 -- 172.21.15.31:0/985047885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ea84fa260 msgr2=0x564ea8502710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:27.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.214+0000 7f88ad628740 1 --2- 172.21.15.31:0/985047885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ea84fa260 0x564ea8502710 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f889002f020 tx=0x7f8890000bc0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.214+0000 7f88ad628740 1 -- 172.21.15.31:0/985047885 shutdown_connections 2024-03-21T10:46:27.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.214+0000 7f88ad628740 1 --2- 172.21.15.31:0/985047885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ea8502de0 0x564ea84fcd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.214+0000 7f88ad628740 1 --2- 172.21.15.31:0/985047885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ea84fa260 0x564ea8502710 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.214+0000 7f88ad628740 1 --2- 172.21.15.31:0/985047885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea84f98b0 0x564ea85021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.214+0000 7f88ad628740 1 -- 172.21.15.31:0/985047885 >> 172.21.15.31:0/985047885 conn(0x564ea84f3f80 msgr2=0x564ea84f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:27.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.214+0000 7f88ad628740 1 -- 172.21.15.31:0/985047885 shutdown_connections 2024-03-21T10:46:27.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.214+0000 7f88ad628740 1 -- 172.21.15.31:0/985047885 wait complete. 2024-03-21T10:46:27.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.214+0000 7f88ad628740 1 -- start start 2024-03-21T10:46:27.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.215+0000 7f88ad628740 1 Processor -- start 2024-03-21T10:46:27.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.215+0000 7f88ad628740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ea84f98b0 0x564ea850abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:27.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.215+0000 7f88ad628740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ea84fa260 0x564ea8508f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:27.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.215+0000 7f88ad628740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea8502de0 0x564ea8509490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:27.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.215+0000 7f88ad628740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a41b3560 con 0x564ea8502de0 2024-03-21T10:46:27.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.215+0000 7f88ad628740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a41b4840 con 0x564ea84fa260 2024-03-21T10:46:27.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.215+0000 7f88ad628740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a41b3b80 con 0x564ea84f98b0 2024-03-21T10:46:27.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.215+0000 7f88ad628740 10 client.0 _ll_register_callbacks cb 0x7f8890005d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:46:27.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.215+0000 7f88a2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ea84fa260 0x564ea8508f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:27.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.215+0000 7f88a2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ea84fa260 0x564ea8508f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:54512/0 (socket says 172.21.15.31:54512) 2024-03-21T10:46:27.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.215+0000 7f88a2a00640 1 -- 172.21.15.31:0/2028269379 learned_addr learned my addr 172.21.15.31:0/2028269379 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:27.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.216+0000 7f88a3e00640 1 --2- 172.21.15.31:0/2028269379 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ea84f98b0 0x564ea850abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:27.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.216+0000 7f88a3e00640 1 --2- 172.21.15.31:0/2028269379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea8502de0 0x564ea8509490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:27.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.216+0000 7f88a2a00640 1 -- 172.21.15.31:0/2028269379 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ea84f98b0 msgr2=0x564ea850abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:27.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.216+0000 7f88a2a00640 1 --2- 172.21.15.31:0/2028269379 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ea84f98b0 0x564ea850abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.216+0000 7f88a2a00640 1 -- 172.21.15.31:0/2028269379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea8502de0 msgr2=0x564ea8509490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:27.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.216+0000 7f88a2a00640 1 --2- 172.21.15.31:0/2028269379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea8502de0 0x564ea8509490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.216+0000 7f88a2a00640 1 -- 172.21.15.31:0/2028269379 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564ea85099d0 con 0x564ea84fa260 2024-03-21T10:46:27.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.216+0000 7f88a2a00640 1 --2- 172.21.15.31:0/2028269379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ea84fa260 0x564ea8508f50 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8890003bd0 tx=0x7f8890003f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b31d79dbdbf3fc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:27.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.216+0000 7f888d600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:46:27.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.216+0000 7f888d600640 1 -- 172.21.15.31:0/2028269379 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8890043040 con 0x564ea84fa260 2024-03-21T10:46:27.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.216+0000 7f888d600640 1 -- 172.21.15.31:0/2028269379 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f88900046c0 con 0x564ea84fa260 2024-03-21T10:46:27.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.216+0000 7f88ad628740 1 -- 172.21.15.31:0/2028269379 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8890004460 con 0x564ea84fa260 2024-03-21T10:46:27.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.217+0000 7f888d600640 1 -- 172.21.15.31:0/2028269379 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8890009520 con 0x564ea84fa260 2024-03-21T10:46:27.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.217+0000 7f88ad628740 10 client.27566 fetch_fsmap learned FSMap version 26 2024-03-21T10:46:27.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.217+0000 7f88ad628740 1 -- 172.21.15.31:0/2028269379 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x564ea856f200 con 0x564ea84fa260 2024-03-21T10:46:27.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.217+0000 7f888d600640 1 -- 172.21.15.31:0/2028269379 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8890004460 con 0x564ea84fa260 2024-03-21T10:46:27.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.217+0000 7f888d600640 1 -- 172.21.15.31:0/2028269379 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f8890034920 con 0x564ea84fa260 2024-03-21T10:46:27.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.217+0000 7f88ad628740 10 client.27566 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:46:27.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.217+0000 7f88ad628740 10 client.27566 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:46:27.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.217+0000 7f88ad628740 20 client.27566 populate_metadata read hostname 'smithi031' 2024-03-21T10:46:27.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.217+0000 7f88ad628740 1 --2- 172.21.15.31:0/2028269379 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564ea856f700 0x564ea8571b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:27.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.217+0000 7f88ad628740 4 client.27566 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:46:27.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.217+0000 7f88ad628740 1 -- 172.21.15.31:0/2028269379 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564ea8464b50 con 0x564ea856f700 2024-03-21T10:46:27.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.218+0000 7f88a3e00640 1 --2- 172.21.15.31:0/2028269379 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564ea856f700 0x564ea8571b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:27.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.218+0000 7f88a3e00640 1 --2- 172.21.15.31:0/2028269379 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564ea856f700 0x564ea8571b20 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1ce73470aae71b1 server_cookie=97a38f49879c6ed5 in_seq=0 out_seq=0 2024-03-21T10:46:27.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.218+0000 7f888d600640 10 client.27566 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:27.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.218+0000 7f888d600640 1 -- 172.21.15.31:0/2028269379 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564ea8464b50 con 0x564ea856f700 2024-03-21T10:46:27.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.218+0000 7f888d600640 10 client.27566 handle_command_reply: tid=0 2024-03-21T10:46:27.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.218+0000 7f88ad628740 1 client.27566 shutdown 2024-03-21T10:46:27.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.218+0000 7f88ad628740 1 -- 172.21.15.31:0/2028269379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ea84fa260 msgr2=0x564ea8508f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:27.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.218+0000 7f88ad628740 1 --2- 172.21.15.31:0/2028269379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ea84fa260 0x564ea8508f50 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8890003bd0 tx=0x7f8890003f00 comp rx=0 tx=0).stop 2024-03-21T10:46:27.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.218+0000 7f88ad628740 1 -- 172.21.15.31:0/2028269379 shutdown_connections 2024-03-21T10:46:27.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.218+0000 7f88ad628740 1 --2- 172.21.15.31:0/2028269379 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564ea856f700 0x564ea8571b20 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.219+0000 7f88ad628740 1 --2- 172.21.15.31:0/2028269379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea8502de0 0x564ea8509490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.219+0000 7f88ad628740 1 --2- 172.21.15.31:0/2028269379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ea84fa260 0x564ea8508f50 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.219+0000 7f888d600640 0 client.27566 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:27.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.219+0000 7f88ad628740 1 --2- 172.21.15.31:0/2028269379 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ea84f98b0 0x564ea850abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.219+0000 7f88ad628740 1 -- 172.21.15.31:0/2028269379 >> 172.21.15.31:0/2028269379 conn(0x564ea84f3f80 msgr2=0x564ea84f52d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:27.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.219+0000 7f88ad628740 1 -- 172.21.15.31:0/2028269379 shutdown_connections 2024-03-21T10:46:27.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.219+0000 7f88ad628740 1 -- 172.21.15.31:0/2028269379 wait complete. 2024-03-21T10:46:27.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.219+0000 7f88ad628740 20 client.27566 trim_cache size 0 max 16384 2024-03-21T10:46:27.219 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:46:27.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.219+0000 7f88a9200640 1 -- 172.21.15.31:0/2192953701 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f88880a3110 msgr2=0x7f88880a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:27.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.219+0000 7f88a9200640 1 --2- 172.21.15.31:0/2192953701 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f88880a3110 0x7f88880a55d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x564ea84f9380 tx=0x7f88940d5530 comp rx=0 tx=0).stop 2024-03-21T10:46:27.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.219+0000 7f88a9200640 1 -- 172.21.15.31:0/2192953701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88a4107aa0 msgr2=0x7f88a419e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:27.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.219+0000 7f88a9200640 1 --2- 172.21.15.31:0/2192953701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88a4107aa0 0x7f88a419e3b0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f889400d950 tx=0x7f889400de10 comp rx=0 tx=0).stop 2024-03-21T10:46:27.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.220+0000 7f88a9200640 1 -- 172.21.15.31:0/2192953701 shutdown_connections 2024-03-21T10:46:27.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.220+0000 7f88a9200640 1 --2- 172.21.15.31:0/2192953701 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f88880a3110 0x7f88880a55d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.220+0000 7f88a9200640 1 --2- 172.21.15.31:0/2192953701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88a410c980 0x7f88a41a2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.220+0000 7f88a9200640 1 --2- 172.21.15.31:0/2192953701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88a4108440 0x7f88a419e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.220+0000 7f88a9200640 1 --2- 172.21.15.31:0/2192953701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88a4107aa0 0x7f88a419e3b0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:27.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.220+0000 7f88a9200640 1 -- 172.21.15.31:0/2192953701 >> 172.21.15.31:0/2192953701 conn(0x7f88a40772a0 msgr2=0x7f88a4077b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:27.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.220+0000 7f88a9200640 1 -- 172.21.15.31:0/2192953701 shutdown_connections 2024-03-21T10:46:27.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:27.220+0000 7f88a9200640 1 -- 172.21.15.31:0/2192953701 wait complete. 2024-03-21T10:46:27.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:27 smithi031 ceph-mon[31673]: pgmap v243: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 263 MiB/s wr, 1.08k op/s 2024-03-21T10:46:27.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:27 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/689623757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:46:27.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:27 smithi040 ceph-mon[36154]: pgmap v243: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 263 MiB/s wr, 1.08k op/s 2024-03-21T10:46:27.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:27 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/689623757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:46:27.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:27 smithi070 ceph-mon[36330]: pgmap v243: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 263 MiB/s wr, 1.08k op/s 2024-03-21T10:46:27.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:27 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/689623757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:46:28.239 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-21T10:46:28.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:46:28.240 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-21T10:46:28.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.501+0000 7f3acfe00640 1 -- 172.21.15.31:0/4062202395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ad0104040 msgr2=0x7f3ad0110550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:28.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.501+0000 7f3acfe00640 1 --2- 172.21.15.31:0/4062202395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ad0104040 0x7f3ad0110550 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f3ac8009cd0 tx=0x7f3ac802c0f0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.501+0000 7f3acfe00640 1 -- 172.21.15.31:0/4062202395 shutdown_connections 2024-03-21T10:46:28.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.501+0000 7f3acfe00640 1 --2- 172.21.15.31:0/4062202395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ad0104040 0x7f3ad0110550 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.501+0000 7f3acfe00640 1 --2- 172.21.15.31:0/4062202395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ad0103680 0x7f3ad0103b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.501+0000 7f3acfe00640 1 --2- 172.21.15.31:0/4062202395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ad0109690 0x7f3ad0109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.501+0000 7f3acfe00640 1 -- 172.21.15.31:0/4062202395 >> 172.21.15.31:0/4062202395 conn(0x7f3ad00ff110 msgr2=0x7f3ad0101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:28.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.502+0000 7f3acfe00640 1 -- 172.21.15.31:0/4062202395 shutdown_connections 2024-03-21T10:46:28.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.502+0000 7f3acfe00640 1 -- 172.21.15.31:0/4062202395 wait complete. 2024-03-21T10:46:28.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.504+0000 7f3acfe00640 1 Processor -- start 2024-03-21T10:46:28.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.504+0000 7f3acfe00640 1 -- start start 2024-03-21T10:46:28.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.505+0000 7f3acfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ad0103680 0x7f3ad0199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.505+0000 7f3acfe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ad0104040 0x7f3ad019a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.505+0000 7f3acfe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ad0109690 0x7f3ad019e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.505+0000 7f3acfe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad0113a60 con 0x7f3ad0104040 2024-03-21T10:46:28.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.505+0000 7f3acfe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad01138e0 con 0x7f3ad0103680 2024-03-21T10:46:28.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.505+0000 7f3acfe00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad0113be0 con 0x7f3ad0109690 2024-03-21T10:46:28.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.505+0000 7f3acea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ad0103680 0x7f3ad0199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.505+0000 7f3ace000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ad0104040 0x7f3ad019a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.505+0000 7f3acea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ad0103680 0x7f3ad0199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:54518/0 (socket says 172.21.15.31:54518) 2024-03-21T10:46:28.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.505+0000 7f3acea00640 1 -- 172.21.15.31:0/3066921829 learned_addr learned my addr 172.21.15.31:0/3066921829 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:28.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.505+0000 7f3acf400640 1 --2- 172.21.15.31:0/3066921829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ad0109690 0x7f3ad019e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.505+0000 7f3acea00640 1 -- 172.21.15.31:0/3066921829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ad0109690 msgr2=0x7f3ad019e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:28.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.505+0000 7f3acea00640 1 --2- 172.21.15.31:0/3066921829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ad0109690 0x7f3ad019e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.505+0000 7f3acea00640 1 -- 172.21.15.31:0/3066921829 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ad0104040 msgr2=0x7f3ad019a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:28.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.505+0000 7f3acea00640 1 --2- 172.21.15.31:0/3066921829 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ad0104040 0x7f3ad019a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.505+0000 7f3acea00640 1 -- 172.21.15.31:0/3066921829 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad019ef00 con 0x7f3ad0103680 2024-03-21T10:46:28.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.505+0000 7f3ace000640 1 --2- 172.21.15.31:0/3066921829 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ad0104040 0x7f3ad019a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:46:28.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.506+0000 7f3acea00640 1 --2- 172.21.15.31:0/3066921829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ad0103680 0x7f3ad0199f60 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f3ac000e9f0 tx=0x7f3ac000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:28.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.506+0000 7f3ac7400640 1 -- 172.21.15.31:0/3066921829 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac000cd80 con 0x7f3ad0103680 2024-03-21T10:46:28.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.506+0000 7f3ac7400640 1 -- 172.21.15.31:0/3066921829 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ac0010910 con 0x7f3ad0103680 2024-03-21T10:46:28.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.506+0000 7f3acfe00640 1 -- 172.21.15.31:0/3066921829 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ad019f1f0 con 0x7f3ad0103680 2024-03-21T10:46:28.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.506+0000 7f3ac7400640 1 -- 172.21.15.31:0/3066921829 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac0018990 con 0x7f3ad0103680 2024-03-21T10:46:28.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.506+0000 7f3acfe00640 1 -- 172.21.15.31:0/3066921829 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ad01a6a80 con 0x7f3ad0103680 2024-03-21T10:46:28.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.509+0000 7f3ac7400640 1 -- 172.21.15.31:0/3066921829 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f3ac0010ac0 con 0x7f3ad0103680 2024-03-21T10:46:28.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.510+0000 7f3ad8838740 1 Processor -- start 2024-03-21T10:46:28.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.510+0000 7f3ad8838740 1 -- start start 2024-03-21T10:46:28.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.510+0000 7f3ad8838740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ee0f37290 0x561ee0f37670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.510+0000 7f3ad8838740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561ee0f37bb0 0x561ee0f40250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.510+0000 7f3ad8838740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ee0f40790 0x561ee0f42b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.510+0000 7f3ad8838740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ee0e9beb0 con 0x561ee0f37bb0 2024-03-21T10:46:28.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.510+0000 7f3ad8838740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ee0e44a60 con 0x561ee0f40790 2024-03-21T10:46:28.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.510+0000 7f3ad8838740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ee0e97f50 con 0x561ee0f37290 2024-03-21T10:46:28.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.510+0000 7f3acf400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ee0f37290 0x561ee0f37670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.510+0000 7f3acf400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ee0f37290 0x561ee0f37670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:45314/0 (socket says 172.21.15.31:45314) 2024-03-21T10:46:28.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.510+0000 7f3acf400640 1 -- 172.21.15.31:0/767204557 learned_addr learned my addr 172.21.15.31:0/767204557 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:28.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.511+0000 7f3ace000640 1 --2- 172.21.15.31:0/767204557 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ee0f40790 0x561ee0f42b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.511+0000 7f3acea00640 1 --2- 172.21.15.31:0/767204557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561ee0f37bb0 0x561ee0f40250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.511+0000 7f3acf400640 1 -- 172.21.15.31:0/767204557 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ee0f40790 msgr2=0x561ee0f42b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:28.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.511+0000 7f3acf400640 1 --2- 172.21.15.31:0/767204557 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ee0f40790 0x561ee0f42b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.511+0000 7f3acf400640 1 -- 172.21.15.31:0/767204557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561ee0f37bb0 msgr2=0x561ee0f40250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:28.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.511+0000 7f3acf400640 1 --2- 172.21.15.31:0/767204557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561ee0f37bb0 0x561ee0f40250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.511+0000 7f3acf400640 1 -- 172.21.15.31:0/767204557 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ee0c787e0 con 0x561ee0f37290 2024-03-21T10:46:28.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.511+0000 7f3acea00640 1 --2- 172.21.15.31:0/767204557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561ee0f37bb0 0x561ee0f40250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:46:28.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.511+0000 7f3ac7400640 1 --2- 172.21.15.31:0/3066921829 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3ab40a3090 0x7f3ab40a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.511+0000 7f3acf400640 1 --2- 172.21.15.31:0/767204557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ee0f37290 0x561ee0f37670 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f3ac8004af0 tx=0x7f3ac8005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95f5cfd32269715b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:28.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.511+0000 7f3ac7400640 1 -- 172.21.15.31:0/3066921829 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f3ac00cfe20 con 0x7f3ad0103680 2024-03-21T10:46:28.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.512+0000 7f3acea00640 1 --2- 172.21.15.31:0/3066921829 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3ab40a3090 0x7f3ab40a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.512+0000 7f3abbe00640 1 -- 172.21.15.31:0/767204557 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac8043070 con 0x561ee0f37290 2024-03-21T10:46:28.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.512+0000 7f3abbe00640 1 -- 172.21.15.31:0/767204557 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ac802cca0 con 0x561ee0f37290 2024-03-21T10:46:28.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.512+0000 7f3abbe00640 1 -- 172.21.15.31:0/767204557 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac8039040 con 0x561ee0f37290 2024-03-21T10:46:28.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.512+0000 7f3ad8838740 1 -- 172.21.15.31:0/767204557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ee0f37290 msgr2=0x561ee0f37670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:28.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.512+0000 7f3ad8838740 1 --2- 172.21.15.31:0/767204557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ee0f37290 0x561ee0f37670 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f3ac8004af0 tx=0x7f3ac8005f80 comp rx=0 tx=0).stop 2024-03-21T10:46:28.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.512+0000 7f3acea00640 1 --2- 172.21.15.31:0/3066921829 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3ab40a3090 0x7f3ab40a5550 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3ac00cd280 tx=0x7f3ac0000bc0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:28.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.512+0000 7f3ad8838740 1 -- 172.21.15.31:0/767204557 shutdown_connections 2024-03-21T10:46:28.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.512+0000 7f3ad8838740 1 --2- 172.21.15.31:0/767204557 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ee0f40790 0x561ee0f42b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.512+0000 7f3ad8838740 1 --2- 172.21.15.31:0/767204557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561ee0f37bb0 0x561ee0f40250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.512+0000 7f3ad8838740 1 --2- 172.21.15.31:0/767204557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ee0f37290 0x561ee0f37670 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.512+0000 7f3ad8838740 1 -- 172.21.15.31:0/767204557 >> 172.21.15.31:0/767204557 conn(0x561ee0f319c0 msgr2=0x561ee0f31da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:28.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.512+0000 7f3ad8838740 1 -- 172.21.15.31:0/767204557 shutdown_connections 2024-03-21T10:46:28.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.513+0000 7f3ad8838740 1 -- 172.21.15.31:0/767204557 wait complete. 2024-03-21T10:46:28.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.513+0000 7f3ad8838740 1 -- start start 2024-03-21T10:46:28.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.513+0000 7f3ad8838740 1 Processor -- start 2024-03-21T10:46:28.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.514+0000 7f3ad8838740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ee0f37290 0x561ee0f400d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.514+0000 7f3ad8838740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561ee0f37bb0 0x561ee0f3d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.514+0000 7f3ad8838740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ee0f40790 0x561ee0f3d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.514+0000 7f3ad8838740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ee0f43520 con 0x561ee0f37bb0 2024-03-21T10:46:28.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.514+0000 7f3ad8838740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ee0e57eb0 con 0x561ee0f40790 2024-03-21T10:46:28.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.514+0000 7f3ad8838740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ee0f436a0 con 0x561ee0f37290 2024-03-21T10:46:28.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.514+0000 7f3ad8838740 10 client.0 _ll_register_callbacks cb 0x561ee0e4c150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:46:28.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.514+0000 7f3acf400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ee0f37290 0x561ee0f400d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.514+0000 7f3ace000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561ee0f37bb0 0x561ee0f3d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.514+0000 7f3acf400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ee0f37290 0x561ee0f400d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:45326/0 (socket says 172.21.15.31:45326) 2024-03-21T10:46:28.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.514+0000 7f3acf400640 1 -- 172.21.15.31:0/3609994834 learned_addr learned my addr 172.21.15.31:0/3609994834 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:28.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.514+0000 7f3acf400640 1 --2- 172.21.15.31:0/3609994834 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ee0f40790 0x561ee0f3d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.514+0000 7f3ace000640 1 -- 172.21.15.31:0/3609994834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ee0f37290 msgr2=0x561ee0f400d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:28.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.514+0000 7f3ace000640 1 --2- 172.21.15.31:0/3609994834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ee0f37290 0x561ee0f400d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.514+0000 7f3ace000640 1 -- 172.21.15.31:0/3609994834 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ee0f40790 msgr2=0x561ee0f3d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:28.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.514+0000 7f3ace000640 1 --2- 172.21.15.31:0/3609994834 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ee0f40790 0x561ee0f3d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.514+0000 7f3ace000640 1 -- 172.21.15.31:0/3609994834 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561ee0cd8f70 con 0x561ee0f37bb0 2024-03-21T10:46:28.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.515+0000 7f3ace000640 1 --2- 172.21.15.31:0/3609994834 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561ee0f37bb0 0x561ee0f3d460 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0101740 tx=0x7f3abc011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67fb3c789e92d73c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:28.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.515+0000 7f3ab8c00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:46:28.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.515+0000 7f3ab8c00640 1 -- 172.21.15.31:0/3609994834 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc010c30 con 0x561ee0f37bb0 2024-03-21T10:46:28.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.515+0000 7f3ab8c00640 1 -- 172.21.15.31:0/3609994834 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f3abc010dd0 con 0x561ee0f37bb0 2024-03-21T10:46:28.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.515+0000 7f3ad8838740 1 -- 172.21.15.31:0/3609994834 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3ac802cca0 con 0x561ee0f37bb0 2024-03-21T10:46:28.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.515+0000 7f3ab8c00640 1 -- 172.21.15.31:0/3609994834 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3abc024050 con 0x561ee0f37bb0 2024-03-21T10:46:28.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.516+0000 7f3ad8838740 10 client.16974 fetch_fsmap learned FSMap version 26 2024-03-21T10:46:28.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.516+0000 7f3ad8838740 1 -- 172.21.15.31:0/3609994834 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x561ee0b70170 con 0x561ee0f37bb0 2024-03-21T10:46:28.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.516+0000 7f3ab8c00640 1 -- 172.21.15.31:0/3609994834 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ac802cca0 con 0x561ee0f37bb0 2024-03-21T10:46:28.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.516+0000 7f3ab8c00640 1 -- 172.21.15.31:0/3609994834 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f3abc0047a0 con 0x561ee0f37bb0 2024-03-21T10:46:28.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.516+0000 7f3ad8838740 10 client.16974 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:46:28.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.516+0000 7f3ad8838740 10 client.16974 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:46:28.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.516+0000 7f3ad8838740 20 client.16974 populate_metadata read hostname 'smithi031' 2024-03-21T10:46:28.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.516+0000 7f3ad8838740 1 --2- 172.21.15.31:0/3609994834 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561ee0f50760 0x561ee0f52b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.516+0000 7f3ad8838740 4 client.16974 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:46:28.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.516+0000 7f3ad8838740 1 -- 172.21.15.31:0/3609994834 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561ee0ea93f0 con 0x561ee0f50760 2024-03-21T10:46:28.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.517+0000 7f3acf400640 1 --2- 172.21.15.31:0/3609994834 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561ee0f50760 0x561ee0f52b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.517+0000 7f3acf400640 1 --2- 172.21.15.31:0/3609994834 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561ee0f50760 0x561ee0f52b50 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5df7564cbaa9c03e server_cookie=5f059a3b558c6fc4 in_seq=0 out_seq=0 2024-03-21T10:46:28.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.517+0000 7f3ab8c00640 10 client.16974 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:28.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.518+0000 7f3ab8c00640 1 -- 172.21.15.31:0/3609994834 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561ee0ea93f0 con 0x561ee0f50760 2024-03-21T10:46:28.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.518+0000 7f3ab8c00640 10 client.16974 handle_command_reply: tid=0 2024-03-21T10:46:28.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.518+0000 7f3ad8838740 1 client.16974 shutdown 2024-03-21T10:46:28.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.518+0000 7f3ad8838740 1 -- 172.21.15.31:0/3609994834 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561ee0f37bb0 msgr2=0x561ee0f3d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:28.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.518+0000 7f3ad8838740 1 --2- 172.21.15.31:0/3609994834 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561ee0f37bb0 0x561ee0f3d460 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0101740 tx=0x7f3abc011b50 comp rx=0 tx=0).stop 2024-03-21T10:46:28.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.518+0000 7f3ad8838740 1 -- 172.21.15.31:0/3609994834 shutdown_connections 2024-03-21T10:46:28.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.518+0000 7f3ad8838740 1 --2- 172.21.15.31:0/3609994834 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561ee0f50760 0x561ee0f52b50 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.518+0000 7f3ad8838740 1 --2- 172.21.15.31:0/3609994834 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ee0f40790 0x561ee0f3d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.518+0000 7f3ad8838740 1 --2- 172.21.15.31:0/3609994834 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561ee0f37bb0 0x561ee0f3d460 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.518+0000 7f3ad8838740 1 --2- 172.21.15.31:0/3609994834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ee0f37290 0x561ee0f400d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.518+0000 7f3ab8c00640 0 client.16974 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:28.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.518+0000 7f3ad8838740 1 -- 172.21.15.31:0/3609994834 >> 172.21.15.31:0/3609994834 conn(0x561ee0f319c0 msgr2=0x561ee0f33710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:28.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.518+0000 7f3ad8838740 1 -- 172.21.15.31:0/3609994834 shutdown_connections 2024-03-21T10:46:28.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.518+0000 7f3ad8838740 1 -- 172.21.15.31:0/3609994834 wait complete. 2024-03-21T10:46:28.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.518+0000 7f3ad8838740 20 client.16974 trim_cache size 0 max 16384 2024-03-21T10:46:28.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.536+0000 7f3ad8838740 1 Processor -- start 2024-03-21T10:46:28.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.536+0000 7f3ad8838740 1 -- start start 2024-03-21T10:46:28.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.537+0000 7f3ad8838740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561ee0f37290 0x561ee0f3fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.537+0000 7f3ad8838740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ee0f37bb0 0x561ee0f400d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.537+0000 7f3ad8838740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ee0f40790 0x561ee0f333d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.537+0000 7f3ad8838740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad01b2f60 con 0x561ee0f37290 2024-03-21T10:46:28.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.537+0000 7f3ad8838740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad01b0a60 con 0x561ee0f37bb0 2024-03-21T10:46:28.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.537+0000 7f3ad8838740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad01aeb20 con 0x561ee0f40790 2024-03-21T10:46:28.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.537+0000 7f3ace000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ee0f37bb0 0x561ee0f400d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.537+0000 7f3acf400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561ee0f37290 0x561ee0f3fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.537+0000 7f3ace000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ee0f37bb0 0x561ee0f400d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:54538/0 (socket says 172.21.15.31:54538) 2024-03-21T10:46:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.537+0000 7f3acf400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561ee0f37290 0x561ee0f3fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44678/0 (socket says 172.21.15.31:44678) 2024-03-21T10:46:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.537+0000 7f3ace000640 1 -- 172.21.15.31:0/3387205218 learned_addr learned my addr 172.21.15.31:0/3387205218 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.537+0000 7f3acf400640 1 --2- 172.21.15.31:0/3387205218 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ee0f40790 0x561ee0f333d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.537+0000 7f3ace000640 1 -- 172.21.15.31:0/3387205218 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ee0f40790 msgr2=0x561ee0f333d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.537+0000 7f3ace000640 1 --2- 172.21.15.31:0/3387205218 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ee0f40790 0x561ee0f333d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.537+0000 7f3ace000640 1 -- 172.21.15.31:0/3387205218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561ee0f37290 msgr2=0x561ee0f3fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.538+0000 7f3ace000640 1 --2- 172.21.15.31:0/3387205218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561ee0f37290 0x561ee0f3fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.538+0000 7f3ace000640 1 -- 172.21.15.31:0/3387205218 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ee0f31200 con 0x561ee0f37bb0 2024-03-21T10:46:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.538+0000 7f3acf400640 1 --2- 172.21.15.31:0/3387205218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561ee0f37290 0x561ee0f3fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:46:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.538+0000 7f3ace000640 1 --2- 172.21.15.31:0/3387205218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ee0f37bb0 0x561ee0f400d0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f3abc001eb0 tx=0x7f3abc00b400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a36e84cf4beb05aa server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.538+0000 7f3abb400640 1 -- 172.21.15.31:0/3387205218 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc000e60 con 0x561ee0f37bb0 2024-03-21T10:46:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.538+0000 7f3abb400640 1 -- 172.21.15.31:0/3387205218 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3abc005bb0 con 0x561ee0f37bb0 2024-03-21T10:46:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.538+0000 7f3abb400640 1 -- 172.21.15.31:0/3387205218 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc003850 con 0x561ee0f37bb0 2024-03-21T10:46:28.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.539+0000 7f3ad8838740 1 -- 172.21.15.31:0/3387205218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ee0f37bb0 msgr2=0x561ee0f400d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:28.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.539+0000 7f3ad8838740 1 --2- 172.21.15.31:0/3387205218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ee0f37bb0 0x561ee0f400d0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f3abc001eb0 tx=0x7f3abc00b400 comp rx=0 tx=0).stop 2024-03-21T10:46:28.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.539+0000 7f3ad8838740 1 -- 172.21.15.31:0/3387205218 shutdown_connections 2024-03-21T10:46:28.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.539+0000 7f3ad8838740 1 --2- 172.21.15.31:0/3387205218 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ee0f40790 0x561ee0f333d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.539+0000 7f3ad8838740 1 --2- 172.21.15.31:0/3387205218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ee0f37bb0 0x561ee0f400d0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.539+0000 7f3ad8838740 1 --2- 172.21.15.31:0/3387205218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561ee0f37290 0x561ee0f3fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.539+0000 7f3ad8838740 1 -- 172.21.15.31:0/3387205218 >> 172.21.15.31:0/3387205218 conn(0x561ee0f319c0 msgr2=0x561ee0f32cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:28.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.539+0000 7f3ad8838740 1 -- 172.21.15.31:0/3387205218 shutdown_connections 2024-03-21T10:46:28.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.539+0000 7f3ad8838740 1 -- 172.21.15.31:0/3387205218 wait complete. 2024-03-21T10:46:28.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.539+0000 7f3ad8838740 1 -- start start 2024-03-21T10:46:28.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.540+0000 7f3ad8838740 1 Processor -- start 2024-03-21T10:46:28.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.540+0000 7f3ad8838740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ee0f37290 0x561ee0f48590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.540+0000 7f3ad8838740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ee0f37bb0 0x561ee0f44910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.540+0000 7f3ad8838740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561ee0f40790 0x561ee0f44e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.540+0000 7f3ad8838740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad01af120 con 0x561ee0f40790 2024-03-21T10:46:28.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.540+0000 7f3ad8838740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad01b0400 con 0x561ee0f37290 2024-03-21T10:46:28.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.540+0000 7f3ad8838740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad01af740 con 0x561ee0f37bb0 2024-03-21T10:46:28.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.540+0000 7f3ad8838740 10 client.0 _ll_register_callbacks cb 0x7f3abc00bbd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:46:28.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.540+0000 7f3ace000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ee0f37bb0 0x561ee0f44910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.540+0000 7f3acf400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ee0f37290 0x561ee0f48590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.540+0000 7f3ace000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ee0f37bb0 0x561ee0f44910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:45340/0 (socket says 172.21.15.31:45340) 2024-03-21T10:46:28.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.540+0000 7f3ace000640 1 -- 172.21.15.31:0/4290463182 learned_addr learned my addr 172.21.15.31:0/4290463182 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:28.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.540+0000 7f3acf400640 1 --2- 172.21.15.31:0/4290463182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561ee0f40790 0x561ee0f44e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.541+0000 7f3ace000640 1 -- 172.21.15.31:0/4290463182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ee0f37290 msgr2=0x561ee0f48590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:28.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.541+0000 7f3ace000640 1 --2- 172.21.15.31:0/4290463182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ee0f37290 0x561ee0f48590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.541+0000 7f3ace000640 1 -- 172.21.15.31:0/4290463182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561ee0f40790 msgr2=0x561ee0f44e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:28.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.541+0000 7f3ace000640 1 --2- 172.21.15.31:0/4290463182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561ee0f40790 0x561ee0f44e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.541+0000 7f3ace000640 1 -- 172.21.15.31:0/4290463182 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561ee0f45540 con 0x561ee0f37bb0 2024-03-21T10:46:28.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.541+0000 7f3ace000640 1 --2- 172.21.15.31:0/4290463182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ee0f37bb0 0x561ee0f44910 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f3abc011690 tx=0x7f3abc0046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=602e606908bf4441 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:28.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.541+0000 7f3ab8c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:46:28.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.541+0000 7f3ab8c00640 1 -- 172.21.15.31:0/4290463182 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc000e40 con 0x561ee0f37bb0 2024-03-21T10:46:28.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.541+0000 7f3ad8838740 1 -- 172.21.15.31:0/4290463182 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3abc005bb0 con 0x561ee0f37bb0 2024-03-21T10:46:28.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.541+0000 7f3ab8c00640 1 -- 172.21.15.31:0/4290463182 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f3abc004230 con 0x561ee0f37bb0 2024-03-21T10:46:28.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.542+0000 7f3ab8c00640 1 -- 172.21.15.31:0/4290463182 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3abc043050 con 0x561ee0f37bb0 2024-03-21T10:46:28.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.542+0000 7f3ad8838740 10 client.27505 fetch_fsmap learned FSMap version 26 2024-03-21T10:46:28.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.542+0000 7f3ad8838740 1 -- 172.21.15.31:0/4290463182 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x561ee0facf30 con 0x561ee0f37bb0 2024-03-21T10:46:28.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.542+0000 7f3ab8c00640 1 -- 172.21.15.31:0/4290463182 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3abc005bb0 con 0x561ee0f37bb0 2024-03-21T10:46:28.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.542+0000 7f3ab8c00640 1 -- 172.21.15.31:0/4290463182 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f3abc038020 con 0x561ee0f37bb0 2024-03-21T10:46:28.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.542+0000 7f3ad8838740 10 client.27505 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:46:28.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.542+0000 7f3ad8838740 10 client.27505 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:46:28.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.542+0000 7f3ad8838740 20 client.27505 populate_metadata read hostname 'smithi031' 2024-03-21T10:46:28.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.542+0000 7f3ad8838740 1 --2- 172.21.15.31:0/4290463182 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561ee0fad540 0x561ee0faf930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.542+0000 7f3ad8838740 4 client.27505 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-21T10:46:28.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.542+0000 7f3ad8838740 1 -- 172.21.15.31:0/4290463182 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x561ee0ea3740 con 0x561ee0fad540 2024-03-21T10:46:28.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.543+0000 7f3acf400640 1 --2- 172.21.15.31:0/4290463182 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561ee0fad540 0x561ee0faf930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.543+0000 7f3acf400640 1 --2- 172.21.15.31:0/4290463182 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561ee0fad540 0x561ee0faf930 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c7d9d3f03c71016 server_cookie=69241e1fdfdcc958 in_seq=0 out_seq=0 2024-03-21T10:46:28.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.543+0000 7f3ab8c00640 10 client.27505 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:28.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.544+0000 7f3ab8c00640 1 -- 172.21.15.31:0/4290463182 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x561ee0ea3740 con 0x561ee0fad540 2024-03-21T10:46:28.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.544+0000 7f3ab8c00640 10 client.27505 handle_command_reply: tid=0 2024-03-21T10:46:28.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.544+0000 7f3ad8838740 1 client.27505 shutdown 2024-03-21T10:46:28.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.544+0000 7f3ad8838740 1 -- 172.21.15.31:0/4290463182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ee0f37bb0 msgr2=0x561ee0f44910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:28.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.544+0000 7f3ad8838740 1 --2- 172.21.15.31:0/4290463182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ee0f37bb0 0x561ee0f44910 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f3abc011690 tx=0x7f3abc0046d0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.544+0000 7f3ad8838740 1 -- 172.21.15.31:0/4290463182 shutdown_connections 2024-03-21T10:46:28.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.544+0000 7f3ad8838740 1 --2- 172.21.15.31:0/4290463182 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561ee0fad540 0x561ee0faf930 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.544+0000 7f3ad8838740 1 --2- 172.21.15.31:0/4290463182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561ee0f40790 0x561ee0f44e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.544+0000 7f3ad8838740 1 --2- 172.21.15.31:0/4290463182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ee0f37bb0 0x561ee0f44910 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.544+0000 7f3ad8838740 1 --2- 172.21.15.31:0/4290463182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ee0f37290 0x561ee0f48590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.544+0000 7f3ab8c00640 0 client.27505 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:28.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.544+0000 7f3ad8838740 1 -- 172.21.15.31:0/4290463182 >> 172.21.15.31:0/4290463182 conn(0x561ee0f319c0 msgr2=0x561ee0f30a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:28.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.545+0000 7f3ad8838740 1 -- 172.21.15.31:0/4290463182 shutdown_connections 2024-03-21T10:46:28.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.545+0000 7f3ad8838740 1 -- 172.21.15.31:0/4290463182 wait complete. 2024-03-21T10:46:28.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.545+0000 7f3ad8838740 20 client.27505 trim_cache size 0 max 16384 2024-03-21T10:46:28.546 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:46:28.546 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-03-21T10:46:28.546 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "f2fe42ea-c353-45b3-92c8-b120e23ffee6", 2024-03-21T10:46:28.546 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-03-21T10:46:28.546 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:46:28.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.545+0000 7f3acfe00640 1 -- 172.21.15.31:0/3066921829 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3ab40a3090 msgr2=0x7f3ab40a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:28.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.545+0000 7f3acfe00640 1 --2- 172.21.15.31:0/3066921829 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3ab40a3090 0x7f3ab40a5550 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3ac00cd280 tx=0x7f3ac0000bc0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.545+0000 7f3acfe00640 1 -- 172.21.15.31:0/3066921829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ad0103680 msgr2=0x7f3ad0199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:28.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.545+0000 7f3acfe00640 1 --2- 172.21.15.31:0/3066921829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ad0103680 0x7f3ad0199f60 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f3ac000e9f0 tx=0x7f3ac000eeb0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.546+0000 7f3acfe00640 1 -- 172.21.15.31:0/3066921829 shutdown_connections 2024-03-21T10:46:28.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.546+0000 7f3acfe00640 1 --2- 172.21.15.31:0/3066921829 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3ab40a3090 0x7f3ab40a5550 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.546+0000 7f3acfe00640 1 --2- 172.21.15.31:0/3066921829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ad0109690 0x7f3ad019e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.546+0000 7f3acfe00640 1 --2- 172.21.15.31:0/3066921829 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ad0104040 0x7f3ad019a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.546+0000 7f3acfe00640 1 --2- 172.21.15.31:0/3066921829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ad0103680 0x7f3ad0199f60 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.546+0000 7f3acfe00640 1 -- 172.21.15.31:0/3066921829 >> 172.21.15.31:0/3066921829 conn(0x7f3ad00ff110 msgr2=0x7f3ad00ff7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:28.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.546+0000 7f3acfe00640 1 -- 172.21.15.31:0/3066921829 shutdown_connections 2024-03-21T10:46:28.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.546+0000 7f3acfe00640 1 -- 172.21.15.31:0/3066921829 wait complete. 2024-03-21T10:46:28.567 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:46:28.568 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:46:28.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.814+0000 7f2303e00640 1 -- 172.21.15.31:0/3190570503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2304100fc0 msgr2=0x7f2304104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:28.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.814+0000 7f2303e00640 1 --2- 172.21.15.31:0/3190570503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2304100fc0 0x7f2304104fb0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f22f0009900 tx=0x7f22f002bc40 comp rx=0 tx=0).stop 2024-03-21T10:46:28.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.814+0000 7f2303e00640 1 -- 172.21.15.31:0/3190570503 shutdown_connections 2024-03-21T10:46:28.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.814+0000 7f2303e00640 1 --2- 172.21.15.31:0/3190570503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2304105690 0x7f230410d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.814+0000 7f2303e00640 1 --2- 172.21.15.31:0/3190570503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2304100fc0 0x7f2304104fb0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.814+0000 7f2303e00640 1 --2- 172.21.15.31:0/3190570503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2304100610 0x7f23041009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.814+0000 7f2303e00640 1 -- 172.21.15.31:0/3190570503 >> 172.21.15.31:0/3190570503 conn(0x7f2304077130 msgr2=0x7f23040fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:28.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.814+0000 7f2303e00640 1 -- 172.21.15.31:0/3190570503 shutdown_connections 2024-03-21T10:46:28.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.814+0000 7f2303e00640 1 -- 172.21.15.31:0/3190570503 wait complete. 2024-03-21T10:46:28.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.816+0000 7f2303e00640 1 Processor -- start 2024-03-21T10:46:28.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.816+0000 7f2303e00640 1 -- start start 2024-03-21T10:46:28.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.816+0000 7f2303e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2304100610 0x7f2304199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.816+0000 7f2303e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2304105690 0x7f230419a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.816+0000 7f2303e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f230419e800 0x7f230419ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.816+0000 7f2303e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2304110750 con 0x7f2304105690 2024-03-21T10:46:28.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.816+0000 7f2303e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23041105d0 con 0x7f230419e800 2024-03-21T10:46:28.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.816+0000 7f2303e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23041108d0 con 0x7f2304100610 2024-03-21T10:46:28.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.816+0000 7f2302a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2304100610 0x7f2304199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.816+0000 7f2302000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2304105690 0x7f230419a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.816+0000 7f2302a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2304100610 0x7f2304199f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:45352/0 (socket says 172.21.15.31:45352) 2024-03-21T10:46:28.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.816+0000 7f2302a00640 1 -- 172.21.15.31:0/3374135751 learned_addr learned my addr 172.21.15.31:0/3374135751 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:28.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.816+0000 7f2303400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f230419e800 0x7f230419ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.817+0000 7f2302a00640 1 -- 172.21.15.31:0/3374135751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f230419e800 msgr2=0x7f230419ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:28.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.817+0000 7f2302a00640 1 --2- 172.21.15.31:0/3374135751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f230419e800 0x7f230419ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.817+0000 7f2302a00640 1 -- 172.21.15.31:0/3374135751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2304105690 msgr2=0x7f230419a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:28.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.817+0000 7f2302a00640 1 --2- 172.21.15.31:0/3374135751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2304105690 0x7f230419a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.817+0000 7f2302a00640 1 -- 172.21.15.31:0/3374135751 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f230419f2f0 con 0x7f2304100610 2024-03-21T10:46:28.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.817+0000 7f2302a00640 1 --2- 172.21.15.31:0/3374135751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2304100610 0x7f2304199f40 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f22f400db80 tx=0x7f22f400b630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:28.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.817+0000 7f2308a00640 1 -- 172.21.15.31:0/3374135751 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f4017070 con 0x7f2304100610 2024-03-21T10:46:28.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.817+0000 7f2308a00640 1 -- 172.21.15.31:0/3374135751 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22f4003b70 con 0x7f2304100610 2024-03-21T10:46:28.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.817+0000 7f2308a00640 1 -- 172.21.15.31:0/3374135751 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f40106c0 con 0x7f2304100610 2024-03-21T10:46:28.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.818+0000 7f2303e00640 1 -- 172.21.15.31:0/3374135751 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f230419f5e0 con 0x7f2304100610 2024-03-21T10:46:28.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.818+0000 7f2303e00640 1 -- 172.21.15.31:0/3374135751 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23041a6d50 con 0x7f2304100610 2024-03-21T10:46:28.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.821+0000 7f230cb70740 1 Processor -- start 2024-03-21T10:46:28.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.821+0000 7f2308a00640 1 -- 172.21.15.31:0/3374135751 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f22f4004020 con 0x7f2304100610 2024-03-21T10:46:28.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.821+0000 7f230cb70740 1 -- start start 2024-03-21T10:46:28.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.821+0000 7f230cb70740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558aaa9c3dd0 0x558aaa9c2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.821+0000 7f230cb70740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558aaa9c30f0 0x558aaa9c3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.821+0000 7f230cb70740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558aaa9cc460 0x558aaa9ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.821+0000 7f230cb70740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558aaa927250 con 0x558aaa9cc460 2024-03-21T10:46:28.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.821+0000 7f230cb70740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x558aaa8cfa30 con 0x558aaa9c3dd0 2024-03-21T10:46:28.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.821+0000 7f230cb70740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x558aaa9232f0 con 0x558aaa9c30f0 2024-03-21T10:46:28.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.822+0000 7f2302a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558aaa9c30f0 0x558aaa9c3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.822+0000 7f2303400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558aaa9c3dd0 0x558aaa9c2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.822+0000 7f2303400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558aaa9c3dd0 0x558aaa9c2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:54558/0 (socket says 172.21.15.31:54558) 2024-03-21T10:46:28.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.822+0000 7f2303400640 1 -- 172.21.15.31:0/2332490199 learned_addr learned my addr 172.21.15.31:0/2332490199 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:28.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.822+0000 7f2302000640 1 --2- 172.21.15.31:0/2332490199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558aaa9cc460 0x558aaa9ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.822+0000 7f2303400640 1 -- 172.21.15.31:0/2332490199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558aaa9c30f0 msgr2=0x558aaa9c3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:28.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.822+0000 7f2303400640 1 --2- 172.21.15.31:0/2332490199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558aaa9c30f0 0x558aaa9c3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.822+0000 7f2303400640 1 -- 172.21.15.31:0/2332490199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558aaa9cc460 msgr2=0x558aaa9ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:28.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.822+0000 7f2303400640 1 --2- 172.21.15.31:0/2332490199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558aaa9cc460 0x558aaa9ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.822+0000 7f2303400640 1 -- 172.21.15.31:0/2332490199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558aaa8ba690 con 0x558aaa9c3dd0 2024-03-21T10:46:28.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.822+0000 7f2303400640 1 --2- 172.21.15.31:0/2332490199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558aaa9c3dd0 0x558aaa9c2b20 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f22fc009c20 tx=0x7f22fc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7c09cd24df02816 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:28.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.823+0000 7f22efe00640 1 -- 172.21.15.31:0/2332490199 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22fc01e070 con 0x558aaa9c3dd0 2024-03-21T10:46:28.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.823+0000 7f22efe00640 1 -- 172.21.15.31:0/2332490199 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22fc019b10 con 0x558aaa9c3dd0 2024-03-21T10:46:28.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.823+0000 7f22efe00640 1 -- 172.21.15.31:0/2332490199 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22fc00eae0 con 0x558aaa9c3dd0 2024-03-21T10:46:28.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.823+0000 7f230cb70740 1 -- 172.21.15.31:0/2332490199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558aaa9c3dd0 msgr2=0x558aaa9c2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:28.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.823+0000 7f230cb70740 1 --2- 172.21.15.31:0/2332490199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558aaa9c3dd0 0x558aaa9c2b20 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f22fc009c20 tx=0x7f22fc011810 comp rx=0 tx=0).stop 2024-03-21T10:46:28.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.823+0000 7f230cb70740 1 -- 172.21.15.31:0/2332490199 shutdown_connections 2024-03-21T10:46:28.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.823+0000 7f230cb70740 1 --2- 172.21.15.31:0/2332490199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558aaa9cc460 0x558aaa9ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.823+0000 7f230cb70740 1 --2- 172.21.15.31:0/2332490199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558aaa9c30f0 0x558aaa9c3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.823+0000 7f230cb70740 1 --2- 172.21.15.31:0/2332490199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558aaa9c3dd0 0x558aaa9c2b20 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.823+0000 7f230cb70740 1 -- 172.21.15.31:0/2332490199 >> 172.21.15.31:0/2332490199 conn(0x558aaa98a200 msgr2=0x558aaa9bdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:28.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.823+0000 7f2308a00640 1 --2- 172.21.15.31:0/3374135751 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f22e80a3110 0x7f22e80a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.823+0000 7f230cb70740 1 -- 172.21.15.31:0/2332490199 shutdown_connections 2024-03-21T10:46:28.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.824+0000 7f230cb70740 1 -- 172.21.15.31:0/2332490199 wait complete. 2024-03-21T10:46:28.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.824+0000 7f2308a00640 1 -- 172.21.15.31:0/3374135751 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f22f40cf8b0 con 0x7f2304100610 2024-03-21T10:46:28.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.824+0000 7f2303400640 1 --2- 172.21.15.31:0/3374135751 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f22e80a3110 0x7f22e80a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.824+0000 7f230cb70740 1 -- start start 2024-03-21T10:46:28.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.824+0000 7f230cb70740 1 Processor -- start 2024-03-21T10:46:28.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.825+0000 7f2303400640 1 --2- 172.21.15.31:0/3374135751 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f22e80a3110 0x7f22e80a55d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f22fc009a30 tx=0x7f22fc000bc0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:28.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.825+0000 7f230cb70740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558aaa9c30f0 0x558aaa9cbb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.825+0000 7f230cb70740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558aaa9c3dd0 0x558aaa9c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.825+0000 7f230cb70740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558aaa9cc460 0x558aaa9c9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.825+0000 7f230cb70740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558aaa9cf2d0 con 0x558aaa9c30f0 2024-03-21T10:46:28.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.825+0000 7f230cb70740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x558aaa8e2e80 con 0x558aaa9c3dd0 2024-03-21T10:46:28.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.825+0000 7f230cb70740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x558aaa9cf450 con 0x558aaa9cc460 2024-03-21T10:46:28.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.825+0000 7f230cb70740 10 client.0 _ll_register_callbacks cb 0x558aaa8f5130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:46:28.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.825+0000 7f2303400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558aaa9cc460 0x558aaa9c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.825+0000 7f2302000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558aaa9c3dd0 0x558aaa9c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.825+0000 7f2302a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558aaa9c30f0 0x558aaa9cbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.825+0000 7f2303400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558aaa9cc460 0x558aaa9c9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:45382/0 (socket says 172.21.15.31:45382) 2024-03-21T10:46:28.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.825+0000 7f2302000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558aaa9c3dd0 0x558aaa9c8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:54570/0 (socket says 172.21.15.31:54570) 2024-03-21T10:46:28.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.825+0000 7f2303400640 1 -- 172.21.15.31:0/1033201381 learned_addr learned my addr 172.21.15.31:0/1033201381 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:28.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.826+0000 7f2302000640 1 -- 172.21.15.31:0/1033201381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558aaa9cc460 msgr2=0x558aaa9c9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:28.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.826+0000 7f2302000640 1 --2- 172.21.15.31:0/1033201381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558aaa9cc460 0x558aaa9c9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.826+0000 7f2302000640 1 -- 172.21.15.31:0/1033201381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558aaa9c30f0 msgr2=0x558aaa9cbb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:28.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.826+0000 7f2302000640 1 --2- 172.21.15.31:0/1033201381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558aaa9c30f0 0x558aaa9cbb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.826+0000 7f2302000640 1 -- 172.21.15.31:0/1033201381 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558aaa7057d0 con 0x558aaa9c3dd0 2024-03-21T10:46:28.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.826+0000 7f2302000640 1 --2- 172.21.15.31:0/1033201381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558aaa9c3dd0 0x558aaa9c8ee0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f22f0036040 tx=0x7f22f00045f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0d1f5972d6098ec server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:28.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.826+0000 7f22ecc00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:46:28.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.826+0000 7f22ecc00640 1 -- 172.21.15.31:0/1033201381 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f002cca0 con 0x558aaa9c3dd0 2024-03-21T10:46:28.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.826+0000 7f230cb70740 1 -- 172.21.15.31:0/1033201381 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f22fc019b10 con 0x558aaa9c3dd0 2024-03-21T10:46:28.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.826+0000 7f22ecc00640 1 -- 172.21.15.31:0/1033201381 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f22f00046c0 con 0x558aaa9c3dd0 2024-03-21T10:46:28.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.827+0000 7f22ecc00640 1 -- 172.21.15.31:0/1033201381 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22f00048c0 con 0x558aaa9c3dd0 2024-03-21T10:46:28.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.827+0000 7f230cb70740 10 client.27596 fetch_fsmap learned FSMap version 26 2024-03-21T10:46:28.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.827+0000 7f230cb70740 1 -- 172.21.15.31:0/1033201381 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x558aaa63b3f0 con 0x558aaa9c3dd0 2024-03-21T10:46:28.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.827+0000 7f22ecc00640 1 -- 172.21.15.31:0/1033201381 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22fc019b10 con 0x558aaa9c3dd0 2024-03-21T10:46:28.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.827+0000 7f22ecc00640 1 -- 172.21.15.31:0/1033201381 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f22f0046070 con 0x558aaa9c3dd0 2024-03-21T10:46:28.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.827+0000 7f230cb70740 10 client.27596 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:46:28.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.827+0000 7f230cb70740 10 client.27596 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:46:28.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.827+0000 7f230cb70740 20 client.27596 populate_metadata read hostname 'smithi031' 2024-03-21T10:46:28.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.827+0000 7f230cb70740 1 --2- 172.21.15.31:0/1033201381 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558aaa9dc310 0x558aaa9de700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.827+0000 7f230cb70740 4 client.27596 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:46:28.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.827+0000 7f230cb70740 1 -- 172.21.15.31:0/1033201381 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558aaa934790 con 0x558aaa9dc310 2024-03-21T10:46:28.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.828+0000 7f2302a00640 1 --2- 172.21.15.31:0/1033201381 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558aaa9dc310 0x558aaa9de700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.828+0000 7f2302a00640 1 --2- 172.21.15.31:0/1033201381 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558aaa9dc310 0x558aaa9de700 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71a0caeaa0526415 server_cookie=6df82f734a078aef in_seq=0 out_seq=0 2024-03-21T10:46:28.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.828+0000 7f22ecc00640 10 client.27596 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:28.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.829+0000 7f22ecc00640 1 -- 172.21.15.31:0/1033201381 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558aaa934790 con 0x558aaa9dc310 2024-03-21T10:46:28.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.829+0000 7f22ecc00640 10 client.27596 handle_command_reply: tid=0 2024-03-21T10:46:28.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.829+0000 7f230cb70740 1 client.27596 shutdown 2024-03-21T10:46:28.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.829+0000 7f230cb70740 1 -- 172.21.15.31:0/1033201381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558aaa9c3dd0 msgr2=0x558aaa9c8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:28.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.829+0000 7f230cb70740 1 --2- 172.21.15.31:0/1033201381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558aaa9c3dd0 0x558aaa9c8ee0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f22f0036040 tx=0x7f22f00045f0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.829+0000 7f230cb70740 1 -- 172.21.15.31:0/1033201381 shutdown_connections 2024-03-21T10:46:28.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.829+0000 7f230cb70740 1 --2- 172.21.15.31:0/1033201381 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558aaa9dc310 0x558aaa9de700 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.829+0000 7f230cb70740 1 --2- 172.21.15.31:0/1033201381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558aaa9cc460 0x558aaa9c9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.829+0000 7f22ecc00640 0 client.27596 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:28.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.829+0000 7f230cb70740 1 --2- 172.21.15.31:0/1033201381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558aaa9c3dd0 0x558aaa9c8ee0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.829+0000 7f230cb70740 1 --2- 172.21.15.31:0/1033201381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558aaa9c30f0 0x558aaa9cbb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.829+0000 7f230cb70740 1 -- 172.21.15.31:0/1033201381 >> 172.21.15.31:0/1033201381 conn(0x558aaa98a200 msgr2=0x558aaa9bf890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:28.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.829+0000 7f230cb70740 1 -- 172.21.15.31:0/1033201381 shutdown_connections 2024-03-21T10:46:28.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.829+0000 7f230cb70740 1 -- 172.21.15.31:0/1033201381 wait complete. 2024-03-21T10:46:28.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.829+0000 7f230cb70740 20 client.27596 trim_cache size 0 max 16384 2024-03-21T10:46:28.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.847+0000 7f230cb70740 1 Processor -- start 2024-03-21T10:46:28.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.847+0000 7f230cb70740 1 -- start start 2024-03-21T10:46:28.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.847+0000 7f230cb70740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558aaa9c30f0 0x558aaa9cb610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.847+0000 7f230cb70740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558aaa9c3dd0 0x558aaa9cbb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.847+0000 7f230cb70740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558aaa9cc460 0x558aaa9cd850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.847+0000 7f230cb70740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23041b2f40 con 0x558aaa9c3dd0 2024-03-21T10:46:28.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.847+0000 7f230cb70740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23041b0a40 con 0x558aaa9c30f0 2024-03-21T10:46:28.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.847+0000 7f230cb70740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23041aeb00 con 0x558aaa9cc460 2024-03-21T10:46:28.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.848+0000 7f2302a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558aaa9c30f0 0x558aaa9cb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.848+0000 7f2302a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558aaa9c30f0 0x558aaa9cb610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:54584/0 (socket says 172.21.15.31:54584) 2024-03-21T10:46:28.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.848+0000 7f2302a00640 1 -- 172.21.15.31:0/1775550360 learned_addr learned my addr 172.21.15.31:0/1775550360 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:28.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.848+0000 7f2302000640 1 --2- 172.21.15.31:0/1775550360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558aaa9c3dd0 0x558aaa9cbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.848+0000 7f2303400640 1 --2- 172.21.15.31:0/1775550360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558aaa9cc460 0x558aaa9cd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.848+0000 7f2302a00640 1 -- 172.21.15.31:0/1775550360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558aaa9cc460 msgr2=0x558aaa9cd850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:28.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.848+0000 7f2302a00640 1 --2- 172.21.15.31:0/1775550360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558aaa9cc460 0x558aaa9cd850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.848+0000 7f2302a00640 1 -- 172.21.15.31:0/1775550360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558aaa9c3dd0 msgr2=0x558aaa9cbb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:28.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.848+0000 7f2302a00640 1 --2- 172.21.15.31:0/1775550360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558aaa9c3dd0 0x558aaa9cbb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.848+0000 7f2302a00640 1 -- 172.21.15.31:0/1775550360 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558aaa5fb0f0 con 0x558aaa9c30f0 2024-03-21T10:46:28.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.848+0000 7f2302a00640 1 --2- 172.21.15.31:0/1775550360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558aaa9c30f0 0x558aaa9cb610 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f22f400bb30 tx=0x7f22f4086040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8db9d4550f9d7fe server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:28.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.849+0000 7f22ef400640 1 -- 172.21.15.31:0/1775550360 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f409bfa0 con 0x558aaa9c30f0 2024-03-21T10:46:28.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.849+0000 7f22ef400640 1 -- 172.21.15.31:0/1775550360 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22f409b8a0 con 0x558aaa9c30f0 2024-03-21T10:46:28.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.849+0000 7f22ef400640 1 -- 172.21.15.31:0/1775550360 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f4003d20 con 0x558aaa9c30f0 2024-03-21T10:46:28.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.849+0000 7f230cb70740 1 -- 172.21.15.31:0/1775550360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558aaa9c30f0 msgr2=0x558aaa9cb610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:28.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.849+0000 7f230cb70740 1 --2- 172.21.15.31:0/1775550360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558aaa9c30f0 0x558aaa9cb610 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f22f400bb30 tx=0x7f22f4086040 comp rx=0 tx=0).stop 2024-03-21T10:46:28.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.849+0000 7f230cb70740 1 -- 172.21.15.31:0/1775550360 shutdown_connections 2024-03-21T10:46:28.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.849+0000 7f230cb70740 1 --2- 172.21.15.31:0/1775550360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558aaa9cc460 0x558aaa9cd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.849+0000 7f230cb70740 1 --2- 172.21.15.31:0/1775550360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558aaa9c3dd0 0x558aaa9cbb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.849+0000 7f230cb70740 1 --2- 172.21.15.31:0/1775550360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558aaa9c30f0 0x558aaa9cb610 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.849+0000 7f230cb70740 1 -- 172.21.15.31:0/1775550360 >> 172.21.15.31:0/1775550360 conn(0x558aaa98a200 msgr2=0x558aaa9c7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:28.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.849+0000 7f230cb70740 1 -- 172.21.15.31:0/1775550360 shutdown_connections 2024-03-21T10:46:28.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.849+0000 7f230cb70740 1 -- 172.21.15.31:0/1775550360 wait complete. 2024-03-21T10:46:28.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.850+0000 7f230cb70740 1 -- start start 2024-03-21T10:46:28.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.850+0000 7f230cb70740 1 Processor -- start 2024-03-21T10:46:28.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.850+0000 7f230cb70740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558aaa9c30f0 0x558aaa9d41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.850+0000 7f230cb70740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558aaa9c3dd0 0x558aaa9c60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.850+0000 7f230cb70740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558aaa9cc460 0x558aaa9c65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.851+0000 7f230cb70740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23041af100 con 0x558aaa9c3dd0 2024-03-21T10:46:28.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.851+0000 7f230cb70740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23041b03e0 con 0x558aaa9cc460 2024-03-21T10:46:28.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.851+0000 7f230cb70740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23041af720 con 0x558aaa9c30f0 2024-03-21T10:46:28.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.851+0000 7f230cb70740 10 client.0 _ll_register_callbacks cb 0x7f22f0009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:46:28.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.851+0000 7f2302a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558aaa9c30f0 0x558aaa9d41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.851+0000 7f2302a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558aaa9c30f0 0x558aaa9d41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:45394/0 (socket says 172.21.15.31:45394) 2024-03-21T10:46:28.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.851+0000 7f2302a00640 1 -- 172.21.15.31:0/1089262561 learned_addr learned my addr 172.21.15.31:0/1089262561 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:28.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.851+0000 7f2303400640 1 --2- 172.21.15.31:0/1089262561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558aaa9cc460 0x558aaa9c65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.851+0000 7f2302000640 1 --2- 172.21.15.31:0/1089262561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558aaa9c3dd0 0x558aaa9c60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.851+0000 7f2302a00640 1 -- 172.21.15.31:0/1089262561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558aaa9cc460 msgr2=0x558aaa9c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:28.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.851+0000 7f2302a00640 1 --2- 172.21.15.31:0/1089262561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558aaa9cc460 0x558aaa9c65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.851+0000 7f2302a00640 1 -- 172.21.15.31:0/1089262561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558aaa9c3dd0 msgr2=0x558aaa9c60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:28.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.851+0000 7f2302a00640 1 --2- 172.21.15.31:0/1089262561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558aaa9c3dd0 0x558aaa9c60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.851+0000 7f2302a00640 1 -- 172.21.15.31:0/1089262561 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558aaa9c6b50 con 0x558aaa9c30f0 2024-03-21T10:46:28.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.851+0000 7f2302a00640 1 --2- 172.21.15.31:0/1089262561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558aaa9c30f0 0x558aaa9d41f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f22f4000f80 tx=0x7f22f400df10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9c41ea0ae605b58 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:28.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.851+0000 7f22ecc00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:46:28.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.852+0000 7f22ecc00640 1 -- 172.21.15.31:0/1089262561 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f40a42b0 con 0x558aaa9c30f0 2024-03-21T10:46:28.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.852+0000 7f22ecc00640 1 -- 172.21.15.31:0/1089262561 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f22f4003d20 con 0x558aaa9c30f0 2024-03-21T10:46:28.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.852+0000 7f230cb70740 1 -- 172.21.15.31:0/1089262561 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f22f409b8a0 con 0x558aaa9c30f0 2024-03-21T10:46:28.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.852+0000 7f22ecc00640 1 -- 172.21.15.31:0/1089262561 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22f409bfa0 con 0x558aaa9c30f0 2024-03-21T10:46:28.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.852+0000 7f230cb70740 10 client.27523 fetch_fsmap learned FSMap version 26 2024-03-21T10:46:28.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.852+0000 7f230cb70740 1 -- 172.21.15.31:0/1089262561 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x558aaaa41140 con 0x558aaa9c30f0 2024-03-21T10:46:28.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.852+0000 7f22ecc00640 1 -- 172.21.15.31:0/1089262561 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22f409b8a0 con 0x558aaa9c30f0 2024-03-21T10:46:28.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.852+0000 7f22ecc00640 1 -- 172.21.15.31:0/1089262561 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f22f40860b0 con 0x558aaa9c30f0 2024-03-21T10:46:28.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.852+0000 7f230cb70740 10 client.27523 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:46:28.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.852+0000 7f230cb70740 10 client.27523 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:46:28.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.852+0000 7f230cb70740 20 client.27523 populate_metadata read hostname 'smithi031' 2024-03-21T10:46:28.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.852+0000 7f230cb70740 1 --2- 172.21.15.31:0/1089262561 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558aaaa417e0 0x558aaaa43c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:28.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.852+0000 7f230cb70740 4 client.27523 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:46:28.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.852+0000 7f230cb70740 1 -- 172.21.15.31:0/1089262561 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558aaa92eae0 con 0x558aaaa417e0 2024-03-21T10:46:28.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.853+0000 7f2302000640 1 --2- 172.21.15.31:0/1089262561 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558aaaa417e0 0x558aaaa43c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:28.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.853+0000 7f2302000640 1 --2- 172.21.15.31:0/1089262561 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558aaaa417e0 0x558aaaa43c00 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab0efaff9cc80902 server_cookie=b7b573ed74c8d4c8 in_seq=0 out_seq=0 2024-03-21T10:46:28.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.853+0000 7f22ecc00640 10 client.27523 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:28.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.853+0000 7f22ecc00640 1 -- 172.21.15.31:0/1089262561 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x558aaa92eae0 con 0x558aaaa417e0 2024-03-21T10:46:28.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.853+0000 7f22ecc00640 10 client.27523 handle_command_reply: tid=0 2024-03-21T10:46:28.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.853+0000 7f230cb70740 1 client.27523 shutdown 2024-03-21T10:46:28.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.853+0000 7f230cb70740 1 -- 172.21.15.31:0/1089262561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558aaa9c30f0 msgr2=0x558aaa9d41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:28.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.853+0000 7f230cb70740 1 --2- 172.21.15.31:0/1089262561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558aaa9c30f0 0x558aaa9d41f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f22f4000f80 tx=0x7f22f400df10 comp rx=0 tx=0).stop 2024-03-21T10:46:28.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.854+0000 7f230cb70740 1 -- 172.21.15.31:0/1089262561 shutdown_connections 2024-03-21T10:46:28.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.854+0000 7f230cb70740 1 --2- 172.21.15.31:0/1089262561 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558aaaa417e0 0x558aaaa43c00 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.854+0000 7f230cb70740 1 --2- 172.21.15.31:0/1089262561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558aaa9cc460 0x558aaa9c65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.854+0000 7f230cb70740 1 --2- 172.21.15.31:0/1089262561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558aaa9c3dd0 0x558aaa9c60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.854+0000 7f230cb70740 1 --2- 172.21.15.31:0/1089262561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558aaa9c30f0 0x558aaa9d41f0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.854+0000 7f230cb70740 1 -- 172.21.15.31:0/1089262561 >> 172.21.15.31:0/1089262561 conn(0x558aaa98a200 msgr2=0x558aaa9bc330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:28.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.854+0000 7f22ecc00640 0 client.27523 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:28.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.854+0000 7f230cb70740 1 -- 172.21.15.31:0/1089262561 shutdown_connections 2024-03-21T10:46:28.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.854+0000 7f230cb70740 1 -- 172.21.15.31:0/1089262561 wait complete. 2024-03-21T10:46:28.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.854+0000 7f230cb70740 20 client.27523 trim_cache size 0 max 16384 2024-03-21T10:46:28.855 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:46:28.855 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-21T10:46:28.855 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-03-21T10:46:28.855 INFO:teuthology.orchestra.run.smithi031.stdout: "f2fe42ea-c353-45b3-92c8-b120e23ffee6": { 2024-03-21T10:46:28.855 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-03-21T10:46:28.855 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "f2fe42ea-c353-45b3-92c8-b120e23ffee6", 2024-03-21T10:46:28.855 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-03-21T10:46:28.856 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:46:28.856 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:46:28.856 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:46:28.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.854+0000 7f2303e00640 1 -- 172.21.15.31:0/3374135751 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f22e80a3110 msgr2=0x7f22e80a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:28.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.854+0000 7f2303e00640 1 --2- 172.21.15.31:0/3374135751 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f22e80a3110 0x7f22e80a55d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f22fc009a30 tx=0x7f22fc000bc0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.854+0000 7f2303e00640 1 -- 172.21.15.31:0/3374135751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2304100610 msgr2=0x7f2304199f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:28.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.854+0000 7f2303e00640 1 --2- 172.21.15.31:0/3374135751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2304100610 0x7f2304199f40 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f22f400db80 tx=0x7f22f400b630 comp rx=0 tx=0).stop 2024-03-21T10:46:28.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.855+0000 7f2303e00640 1 -- 172.21.15.31:0/3374135751 shutdown_connections 2024-03-21T10:46:28.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.855+0000 7f2303e00640 1 --2- 172.21.15.31:0/3374135751 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f22e80a3110 0x7f22e80a55d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.855+0000 7f2303e00640 1 --2- 172.21.15.31:0/3374135751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f230419e800 0x7f230419ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.855+0000 7f2303e00640 1 --2- 172.21.15.31:0/3374135751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2304105690 0x7f230419a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.855+0000 7f2303e00640 1 --2- 172.21.15.31:0/3374135751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2304100610 0x7f2304199f40 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:28.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.855+0000 7f2303e00640 1 -- 172.21.15.31:0/3374135751 >> 172.21.15.31:0/3374135751 conn(0x7f2304077130 msgr2=0x7f2304069230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:28.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.855+0000 7f2303e00640 1 -- 172.21.15.31:0/3374135751 shutdown_connections 2024-03-21T10:46:28.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:28.855+0000 7f2303e00640 1 -- 172.21.15.31:0/3374135751 wait complete. 2024-03-21T10:46:28.869 INFO:tasks.cephfs.filesystem:scrub status for tag:f2fe42ea-c353-45b3-92c8-b120e23ffee6 - {'path': '/', 'tag': 'f2fe42ea-c353-45b3-92c8-b120e23ffee6', 'options': 'recursive,force'} 2024-03-21T10:46:29.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:29 smithi040 ceph-mon[36154]: pgmap v244: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 229 MiB/s wr, 943 op/s 2024-03-21T10:46:29.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:29 smithi040 ceph-mon[36154]: scrub queued for path: / 2024-03-21T10:46:29.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:29 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:46:29.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:29 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:46:29.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:29 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:46:29.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:29 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:46:29.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:29 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:46:29.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:29 smithi031 ceph-mon[31673]: pgmap v244: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 229 MiB/s wr, 943 op/s 2024-03-21T10:46:29.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:29 smithi031 ceph-mon[31673]: scrub queued for path: / 2024-03-21T10:46:29.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:29 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:46:29.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:29 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:46:29.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:29 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:46:29.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:29 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:46:29.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:29 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:46:29.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:29 smithi070 ceph-mon[36330]: pgmap v244: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 229 MiB/s wr, 943 op/s 2024-03-21T10:46:29.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:29 smithi070 ceph-mon[36330]: scrub queued for path: / 2024-03-21T10:46:29.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:29 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:46:29.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:29 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:46:29.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:29 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:46:29.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:29 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:46:29.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:29 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:46:31.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:31 smithi040 ceph-mon[36154]: pgmap v245: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 219 MiB/s wr, 912 op/s 2024-03-21T10:46:31.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:31 smithi031 ceph-mon[31673]: pgmap v245: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 219 MiB/s wr, 912 op/s 2024-03-21T10:46:31.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:31 smithi070 ceph-mon[36330]: pgmap v245: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 219 MiB/s wr, 912 op/s 2024-03-21T10:46:33.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:33 smithi040 ceph-mon[36154]: pgmap v246: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 98 MiB/s wr, 440 op/s 2024-03-21T10:46:33.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:33 smithi031 ceph-mon[31673]: pgmap v246: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 98 MiB/s wr, 440 op/s 2024-03-21T10:46:33.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:33 smithi070 ceph-mon[36330]: pgmap v246: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 98 MiB/s wr, 440 op/s 2024-03-21T10:46:35.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:35 smithi040 ceph-mon[36154]: pgmap v247: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 92 MiB/s wr, 420 op/s 2024-03-21T10:46:35.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:35 smithi031 ceph-mon[31673]: pgmap v247: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 92 MiB/s wr, 420 op/s 2024-03-21T10:46:35.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:35 smithi070 ceph-mon[36330]: pgmap v247: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 92 MiB/s wr, 420 op/s 2024-03-21T10:46:36.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:36 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:46:36.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:36 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:46:36.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:36 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:46:37.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:37 smithi040 ceph-mon[36154]: pgmap v248: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 MiB/s wr, 356 op/s 2024-03-21T10:46:37.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:37 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:46:37.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:37 smithi031 ceph-mon[31673]: pgmap v248: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 MiB/s wr, 356 op/s 2024-03-21T10:46:37.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:37 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:46:37.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:37 smithi070 ceph-mon[36330]: pgmap v248: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 MiB/s wr, 356 op/s 2024-03-21T10:46:37.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:37 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:46:38.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:38 smithi040 ceph-mon[36154]: pgmap v249: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 11 op/s 2024-03-21T10:46:38.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:38 smithi031 ceph-mon[31673]: pgmap v249: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 11 op/s 2024-03-21T10:46:38.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:38 smithi070 ceph-mon[36330]: pgmap v249: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 11 op/s 2024-03-21T10:46:39.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.775+0000 7f3fc2400640 1 -- 172.21.15.31:0/3235408040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fbc073a40 msgr2=0x7f3fbc073e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:39.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.775+0000 7f3fc2400640 1 --2- 172.21.15.31:0/3235408040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fbc073a40 0x7f3fbc073e20 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0009900 tx=0x7f3fb002bcc0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.775+0000 7f3fc2400640 1 -- 172.21.15.31:0/3235408040 shutdown_connections 2024-03-21T10:46:39.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.775+0000 7f3fc2400640 1 --2- 172.21.15.31:0/3235408040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fbc112490 0x7f3fbc114880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.775+0000 7f3fc2400640 1 --2- 172.21.15.31:0/3235408040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fbc074360 0x7f3fbc111f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.775+0000 7f3fc2400640 1 --2- 172.21.15.31:0/3235408040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fbc073a40 0x7f3fbc073e20 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.775+0000 7f3fc2400640 1 -- 172.21.15.31:0/3235408040 >> 172.21.15.31:0/3235408040 conn(0x7f3fbc0fcf50 msgr2=0x7f3fbc0ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:39.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.775+0000 7f3fc2400640 1 -- 172.21.15.31:0/3235408040 shutdown_connections 2024-03-21T10:46:39.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.775+0000 7f3fc2400640 1 -- 172.21.15.31:0/3235408040 wait complete. 2024-03-21T10:46:39.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.776+0000 7f3fc2400640 1 Processor -- start 2024-03-21T10:46:39.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.777+0000 7f3fc2400640 1 -- start start 2024-03-21T10:46:39.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.777+0000 7f3fc2400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fbc073a40 0x7f3fbc19e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:39.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.777+0000 7f3fc2400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fbc074360 0x7f3fbc19e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:39.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.777+0000 7f3fc2400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fbc112490 0x7f3fbc1a2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:39.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.777+0000 7f3fc2400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc117ee0 con 0x7f3fbc074360 2024-03-21T10:46:39.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.777+0000 7f3fc2400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc117d60 con 0x7f3fbc073a40 2024-03-21T10:46:39.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.777+0000 7f3fc2400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc118060 con 0x7f3fbc112490 2024-03-21T10:46:39.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.777+0000 7f3fc1000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fbc073a40 0x7f3fbc19e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:39.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.777+0000 7f3fc1000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fbc073a40 0x7f3fbc19e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:36332/0 (socket says 172.21.15.31:36332) 2024-03-21T10:46:39.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.777+0000 7f3fc1a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fbc112490 0x7f3fbc1a2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:39.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.777+0000 7f3fc1000640 1 -- 172.21.15.31:0/3021328672 learned_addr learned my addr 172.21.15.31:0/3021328672 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:39.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.777+0000 7f3fbbe00640 1 --2- 172.21.15.31:0/3021328672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fbc074360 0x7f3fbc19e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:39.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.778+0000 7f3fc1000640 1 -- 172.21.15.31:0/3021328672 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fbc112490 msgr2=0x7f3fbc1a2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:39.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.778+0000 7f3fc1000640 1 --2- 172.21.15.31:0/3021328672 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fbc112490 0x7f3fbc1a2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.778+0000 7f3fc1000640 1 -- 172.21.15.31:0/3021328672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fbc074360 msgr2=0x7f3fbc19e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:39.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.778+0000 7f3fc1000640 1 --2- 172.21.15.31:0/3021328672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fbc074360 0x7f3fbc19e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.778+0000 7f3fc1000640 1 -- 172.21.15.31:0/3021328672 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc1a33e0 con 0x7f3fbc073a40 2024-03-21T10:46:39.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.778+0000 7f3fc1a00640 1 --2- 172.21.15.31:0/3021328672 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fbc112490 0x7f3fbc1a2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:46:39.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.778+0000 7f3fbbe00640 1 --2- 172.21.15.31:0/3021328672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fbc074360 0x7f3fbc19e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:46:39.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.778+0000 7f3fc1000640 1 --2- 172.21.15.31:0/3021328672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fbc073a40 0x7f3fbc19e440 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0000c00 tx=0x7f3fb00043c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:39.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.778+0000 7f3fb9600640 1 -- 172.21.15.31:0/3021328672 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb0042020 con 0x7f3fbc073a40 2024-03-21T10:46:39.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.778+0000 7f3fb9600640 1 -- 172.21.15.31:0/3021328672 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3fb0005000 con 0x7f3fbc073a40 2024-03-21T10:46:39.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.778+0000 7f3fc2400640 1 -- 172.21.15.31:0/3021328672 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fbc1a3670 con 0x7f3fbc073a40 2024-03-21T10:46:39.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.778+0000 7f3fb9600640 1 -- 172.21.15.31:0/3021328672 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb0003b00 con 0x7f3fbc073a40 2024-03-21T10:46:39.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.779+0000 7f3fc2400640 1 -- 172.21.15.31:0/3021328672 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fbc1aaf60 con 0x7f3fbc073a40 2024-03-21T10:46:39.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.782+0000 7f3fb9600640 1 -- 172.21.15.31:0/3021328672 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f3fb0045020 con 0x7f3fbc073a40 2024-03-21T10:46:39.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.782+0000 7f3fc67c6740 1 Processor -- start 2024-03-21T10:46:39.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.782+0000 7f3fc67c6740 1 -- start start 2024-03-21T10:46:39.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.782+0000 7f3fc67c6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566e80471e0 0x5566e80475c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:39.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.782+0000 7f3fc67c6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566e8047b00 0x5566e8050390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:39.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.782+0000 7f3fc67c6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5566e80508d0 0x5566e8052cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:39.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.782+0000 7f3fc67c6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566e80093b0 con 0x5566e8047b00 2024-03-21T10:46:39.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.782+0000 7f3fc67c6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566e7fab830 con 0x5566e80508d0 2024-03-21T10:46:39.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.782+0000 7f3fc67c6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566e7f545d0 con 0x5566e80471e0 2024-03-21T10:46:39.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.782+0000 7f3fc1a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566e80471e0 0x5566e80475c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:39.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.783+0000 7f3fc1a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566e80471e0 0x5566e80475c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:37528/0 (socket says 172.21.15.31:37528) 2024-03-21T10:46:39.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.783+0000 7f3fc1a00640 1 -- 172.21.15.31:0/2691004840 learned_addr learned my addr 172.21.15.31:0/2691004840 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:39.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.783+0000 7f3fc1000640 1 --2- 172.21.15.31:0/2691004840 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566e8047b00 0x5566e8050390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:39.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.783+0000 7f3fbbe00640 1 --2- 172.21.15.31:0/2691004840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5566e80508d0 0x5566e8052cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:39.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.783+0000 7f3fc1a00640 1 -- 172.21.15.31:0/2691004840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5566e80508d0 msgr2=0x5566e8052cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:39.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.783+0000 7f3fc1a00640 1 --2- 172.21.15.31:0/2691004840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5566e80508d0 0x5566e8052cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.783+0000 7f3fc1a00640 1 -- 172.21.15.31:0/2691004840 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566e8047b00 msgr2=0x5566e8050390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:39.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.783+0000 7f3fc1a00640 1 --2- 172.21.15.31:0/2691004840 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566e8047b00 0x5566e8050390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.783+0000 7f3fc1a00640 1 -- 172.21.15.31:0/2691004840 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566e7f6c2c0 con 0x5566e80471e0 2024-03-21T10:46:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.783+0000 7f3fbbe00640 1 --2- 172.21.15.31:0/2691004840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5566e80508d0 0x5566e8052cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:46:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.783+0000 7f3fc1a00640 1 --2- 172.21.15.31:0/2691004840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566e80471e0 0x5566e80475c0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4011760 tx=0x7f3fb4011c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=536f89cbb1c25381 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.783+0000 7f3fae000640 1 -- 172.21.15.31:0/2691004840 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4004d20 con 0x5566e80471e0 2024-03-21T10:46:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.784+0000 7f3fae000640 1 -- 172.21.15.31:0/2691004840 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3fb400f9a0 con 0x5566e80471e0 2024-03-21T10:46:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.784+0000 7f3fae000640 1 -- 172.21.15.31:0/2691004840 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4004590 con 0x5566e80471e0 2024-03-21T10:46:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.784+0000 7f3fb9600640 1 --2- 172.21.15.31:0/3021328672 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3fa40a30f0 0x7f3fa40a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.784+0000 7f3fb9600640 1 -- 172.21.15.31:0/3021328672 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f3fb00f1840 con 0x7f3fbc073a40 2024-03-21T10:46:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.784+0000 7f3fc1000640 1 --2- 172.21.15.31:0/3021328672 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3fa40a30f0 0x7f3fa40a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.784+0000 7f3fc67c6740 1 -- 172.21.15.31:0/2691004840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566e80471e0 msgr2=0x5566e80475c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.784+0000 7f3fc67c6740 1 --2- 172.21.15.31:0/2691004840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566e80471e0 0x5566e80475c0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4011760 tx=0x7f3fb4011c20 comp rx=0 tx=0).stop 2024-03-21T10:46:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.784+0000 7f3fc67c6740 1 -- 172.21.15.31:0/2691004840 shutdown_connections 2024-03-21T10:46:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.784+0000 7f3fc67c6740 1 --2- 172.21.15.31:0/2691004840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5566e80508d0 0x5566e8052cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.784+0000 7f3fc67c6740 1 --2- 172.21.15.31:0/2691004840 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566e8047b00 0x5566e8050390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.784+0000 7f3fc67c6740 1 --2- 172.21.15.31:0/2691004840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566e80471e0 0x5566e80475c0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.784+0000 7f3fc67c6740 1 -- 172.21.15.31:0/2691004840 >> 172.21.15.31:0/2691004840 conn(0x5566e8041b00 msgr2=0x5566e8041ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.785+0000 7f3fc67c6740 1 -- 172.21.15.31:0/2691004840 shutdown_connections 2024-03-21T10:46:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.785+0000 7f3fc67c6740 1 -- 172.21.15.31:0/2691004840 wait complete. 2024-03-21T10:46:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.785+0000 7f3fc1000640 1 --2- 172.21.15.31:0/3021328672 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3fa40a30f0 0x7f3fa40a55b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x5566e8046c90 tx=0x7f3fb00035f0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.785+0000 7f3fc67c6740 1 -- start start 2024-03-21T10:46:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.785+0000 7f3fc67c6740 1 Processor -- start 2024-03-21T10:46:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.786+0000 7f3fc67c6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566e80471e0 0x5566e804d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.786+0000 7f3fc67c6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566e8047b00 0x5566e804deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.786+0000 7f3fc67c6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5566e80508d0 0x5566e804e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.786+0000 7f3fc67c6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566e7f67a20 con 0x5566e80471e0 2024-03-21T10:46:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.786+0000 7f3fc67c6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566e7fa78d0 con 0x5566e80508d0 2024-03-21T10:46:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.786+0000 7f3fc67c6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566e8053500 con 0x5566e8047b00 2024-03-21T10:46:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.786+0000 7f3fc67c6740 10 client.0 _ll_register_callbacks cb 0x5566e7fa41c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:46:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.786+0000 7f3fbbe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566e8047b00 0x5566e804deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.786+0000 7f3fbbe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566e8047b00 0x5566e804deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:37544/0 (socket says 172.21.15.31:37544) 2024-03-21T10:46:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.786+0000 7f3fbbe00640 1 -- 172.21.15.31:0/2030017691 learned_addr learned my addr 172.21.15.31:0/2030017691 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.786+0000 7f3fc1a00640 1 --2- 172.21.15.31:0/2030017691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566e80471e0 0x5566e804d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.786+0000 7f3fc1a00640 1 --2- 172.21.15.31:0/2030017691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5566e80508d0 0x5566e804e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.787+0000 7f3fbbe00640 1 -- 172.21.15.31:0/2030017691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5566e80508d0 msgr2=0x5566e804e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.787+0000 7f3fbbe00640 1 --2- 172.21.15.31:0/2030017691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5566e80508d0 0x5566e804e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.787+0000 7f3fbbe00640 1 -- 172.21.15.31:0/2030017691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566e80471e0 msgr2=0x5566e804d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.787+0000 7f3fbbe00640 1 --2- 172.21.15.31:0/2030017691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566e80471e0 0x5566e804d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.787+0000 7f3fbbe00640 1 -- 172.21.15.31:0/2030017691 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5566e7cc0210 con 0x5566e8047b00 2024-03-21T10:46:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.787+0000 7f3fc1a00640 1 --2- 172.21.15.31:0/2030017691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566e80471e0 0x5566e804d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:46:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.787+0000 7f3fbbe00640 1 --2- 172.21.15.31:0/2030017691 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566e8047b00 0x5566e804deb0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f3fa800bf00 tx=0x7f3fa80117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9cd2cc7133c887b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.787+0000 7f3fa2a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:46:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.787+0000 7f3fa2a00640 1 -- 172.21.15.31:0/2030017691 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa8012650 con 0x5566e8047b00 2024-03-21T10:46:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.787+0000 7f3fa2a00640 1 -- 172.21.15.31:0/2030017691 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f3fa80127f0 con 0x5566e8047b00 2024-03-21T10:46:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.787+0000 7f3fc67c6740 1 -- 172.21.15.31:0/2030017691 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3fb400f9a0 con 0x5566e8047b00 2024-03-21T10:46:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.787+0000 7f3fa2a00640 1 -- 172.21.15.31:0/2030017691 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3fa80129f0 con 0x5566e8047b00 2024-03-21T10:46:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.788+0000 7f3fc67c6740 10 client.27535 fetch_fsmap learned FSMap version 26 2024-03-21T10:46:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.788+0000 7f3fc67c6740 1 -- 172.21.15.31:0/2030017691 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5566e7d8c620 con 0x5566e8047b00 2024-03-21T10:46:39.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.788+0000 7f3fa2a00640 1 -- 172.21.15.31:0/2030017691 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3fb400f9a0 con 0x5566e8047b00 2024-03-21T10:46:39.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.788+0000 7f3fa2a00640 1 -- 172.21.15.31:0/2030017691 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f3fa801a3d0 con 0x5566e8047b00 2024-03-21T10:46:39.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.788+0000 7f3fc67c6740 10 client.27535 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:46:39.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.788+0000 7f3fc67c6740 10 client.27535 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:46:39.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.788+0000 7f3fc67c6740 20 client.27535 populate_metadata read hostname 'smithi031' 2024-03-21T10:46:39.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.788+0000 7f3fc67c6740 1 --2- 172.21.15.31:0/2030017691 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5566e8060a00 0x5566e8062df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:39.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.788+0000 7f3fc67c6740 4 client.27535 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:46:39.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.788+0000 7f3fc67c6740 1 -- 172.21.15.31:0/2030017691 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5566e7fb8d70 con 0x5566e8060a00 2024-03-21T10:46:39.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.788+0000 7f3fc1a00640 1 --2- 172.21.15.31:0/2030017691 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5566e8060a00 0x5566e8062df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:39.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.789+0000 7f3fc1a00640 1 --2- 172.21.15.31:0/2030017691 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5566e8060a00 0x5566e8062df0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb13c2a8b0558cf0 server_cookie=df27be23d7ccd7cc in_seq=0 out_seq=0 2024-03-21T10:46:39.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.789+0000 7f3fa2a00640 10 client.27535 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:39.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.790+0000 7f3fa2a00640 1 -- 172.21.15.31:0/2030017691 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5566e7fb8d70 con 0x5566e8060a00 2024-03-21T10:46:39.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.790+0000 7f3fa2a00640 10 client.27535 handle_command_reply: tid=0 2024-03-21T10:46:39.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.790+0000 7f3fc67c6740 1 client.27535 shutdown 2024-03-21T10:46:39.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.790+0000 7f3fc67c6740 1 -- 172.21.15.31:0/2030017691 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566e8047b00 msgr2=0x5566e804deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:39.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.790+0000 7f3fc67c6740 1 --2- 172.21.15.31:0/2030017691 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566e8047b00 0x5566e804deb0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f3fa800bf00 tx=0x7f3fa80117b0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.790+0000 7f3fc67c6740 1 -- 172.21.15.31:0/2030017691 shutdown_connections 2024-03-21T10:46:39.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.790+0000 7f3fc67c6740 1 --2- 172.21.15.31:0/2030017691 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5566e8060a00 0x5566e8062df0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.790+0000 7f3fc67c6740 1 --2- 172.21.15.31:0/2030017691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5566e80508d0 0x5566e804e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.790+0000 7f3fc67c6740 1 --2- 172.21.15.31:0/2030017691 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566e8047b00 0x5566e804deb0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.790+0000 7f3fa2a00640 0 client.27535 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:39.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.790+0000 7f3fc67c6740 1 --2- 172.21.15.31:0/2030017691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566e80471e0 0x5566e804d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.790+0000 7f3fc67c6740 1 -- 172.21.15.31:0/2030017691 >> 172.21.15.31:0/2030017691 conn(0x5566e8041b00 msgr2=0x5566e8043850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:39.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.790+0000 7f3fc67c6740 1 -- 172.21.15.31:0/2030017691 shutdown_connections 2024-03-21T10:46:39.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.790+0000 7f3fc67c6740 1 -- 172.21.15.31:0/2030017691 wait complete. 2024-03-21T10:46:39.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.790+0000 7f3fc67c6740 20 client.27535 trim_cache size 0 max 16384 2024-03-21T10:46:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.806+0000 7f3fc67c6740 1 Processor -- start 2024-03-21T10:46:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.806+0000 7f3fc67c6740 1 -- start start 2024-03-21T10:46:39.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.807+0000 7f3fc67c6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566e80471e0 0x5566e80500a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:39.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.807+0000 7f3fc67c6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5566e8047b00 0x5566e804e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:39.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.807+0000 7f3fc67c6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566e80508d0 0x5566e80409f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:39.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.807+0000 7f3fc67c6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc1b72e0 con 0x5566e80508d0 2024-03-21T10:46:39.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.807+0000 7f3fc67c6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc1b4de0 con 0x5566e8047b00 2024-03-21T10:46:39.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.807+0000 7f3fc67c6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc1b2ea0 con 0x5566e80471e0 2024-03-21T10:46:39.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.807+0000 7f3fc1a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566e80471e0 0x5566e80500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:39.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.807+0000 7f3fc1a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566e80471e0 0x5566e80500a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:37550/0 (socket says 172.21.15.31:37550) 2024-03-21T10:46:39.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.807+0000 7f3fc1a00640 1 -- 172.21.15.31:0/2010131613 learned_addr learned my addr 172.21.15.31:0/2010131613 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:39.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.807+0000 7f3fbbe00640 1 --2- 172.21.15.31:0/2010131613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5566e8047b00 0x5566e804e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:39.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.807+0000 7f3fc1a00640 1 --2- 172.21.15.31:0/2010131613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566e80508d0 0x5566e80409f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:39.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.808+0000 7f3fc1a00640 1 -- 172.21.15.31:0/2010131613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5566e8047b00 msgr2=0x5566e804e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:39.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.808+0000 7f3fc1a00640 1 --2- 172.21.15.31:0/2010131613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5566e8047b00 0x5566e804e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.808+0000 7f3fc1a00640 1 -- 172.21.15.31:0/2010131613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566e80508d0 msgr2=0x5566e80409f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:39.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.808+0000 7f3fc1a00640 1 --2- 172.21.15.31:0/2010131613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566e80508d0 0x5566e80409f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.808+0000 7f3fc1a00640 1 -- 172.21.15.31:0/2010131613 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566e8053030 con 0x5566e80471e0 2024-03-21T10:46:39.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.808+0000 7f3fc1a00640 1 --2- 172.21.15.31:0/2010131613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566e80471e0 0x5566e80500a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4011730 tx=0x7f3fb4005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1344361b799c5c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:39.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.808+0000 7f3fad600640 1 -- 172.21.15.31:0/2010131613 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb400f9a0 con 0x5566e80471e0 2024-03-21T10:46:39.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.808+0000 7f3fad600640 1 -- 172.21.15.31:0/2010131613 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3fb402cce0 con 0x5566e80471e0 2024-03-21T10:46:39.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.808+0000 7f3fad600640 1 -- 172.21.15.31:0/2010131613 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4003e10 con 0x5566e80471e0 2024-03-21T10:46:39.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.809+0000 7f3fc67c6740 1 -- 172.21.15.31:0/2010131613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566e80471e0 msgr2=0x5566e80500a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:39.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.809+0000 7f3fc67c6740 1 --2- 172.21.15.31:0/2010131613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566e80471e0 0x5566e80500a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4011730 tx=0x7f3fb4005fd0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.809+0000 7f3fc67c6740 1 -- 172.21.15.31:0/2010131613 shutdown_connections 2024-03-21T10:46:39.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.809+0000 7f3fc67c6740 1 --2- 172.21.15.31:0/2010131613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566e80508d0 0x5566e80409f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.809+0000 7f3fc67c6740 1 --2- 172.21.15.31:0/2010131613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5566e8047b00 0x5566e804e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.809+0000 7f3fc67c6740 1 --2- 172.21.15.31:0/2010131613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566e80471e0 0x5566e80500a0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.809+0000 7f3fc67c6740 1 -- 172.21.15.31:0/2010131613 >> 172.21.15.31:0/2010131613 conn(0x5566e8041b00 msgr2=0x5566e80402e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:39.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.809+0000 7f3fc67c6740 1 -- 172.21.15.31:0/2010131613 shutdown_connections 2024-03-21T10:46:39.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.809+0000 7f3fc67c6740 1 -- 172.21.15.31:0/2010131613 wait complete. 2024-03-21T10:46:39.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.809+0000 7f3fc67c6740 1 -- start start 2024-03-21T10:46:39.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.810+0000 7f3fc67c6740 1 Processor -- start 2024-03-21T10:46:39.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.810+0000 7f3fc67c6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566e80471e0 0x5566e8053f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:39.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.810+0000 7f3fc67c6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5566e8047b00 0x5566e8054460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:39.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.810+0000 7f3fc67c6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566e80508d0 0x5566e80bb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:39.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.810+0000 7f3fc67c6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc1b34a0 con 0x5566e80508d0 2024-03-21T10:46:39.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.810+0000 7f3fc67c6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc1b4780 con 0x5566e8047b00 2024-03-21T10:46:39.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.810+0000 7f3fc67c6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc1b3ac0 con 0x5566e80471e0 2024-03-21T10:46:39.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.810+0000 7f3fc67c6740 10 client.0 _ll_register_callbacks cb 0x7f3fa800b520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:46:39.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.810+0000 7f3fbbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5566e8047b00 0x5566e8054460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:39.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.810+0000 7f3fc1a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566e80471e0 0x5566e8053f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:39.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.810+0000 7f3fbbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5566e8047b00 0x5566e8054460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:36376/0 (socket says 172.21.15.31:36376) 2024-03-21T10:46:39.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.810+0000 7f3fbbe00640 1 -- 172.21.15.31:0/3305479039 learned_addr learned my addr 172.21.15.31:0/3305479039 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:39.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.810+0000 7f3fc1a00640 1 --2- 172.21.15.31:0/3305479039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566e80508d0 0x5566e80bb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:39.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.811+0000 7f3fbbe00640 1 -- 172.21.15.31:0/3305479039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566e80471e0 msgr2=0x5566e8053f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:39.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.811+0000 7f3fbbe00640 1 --2- 172.21.15.31:0/3305479039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566e80471e0 0x5566e8053f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.811+0000 7f3fbbe00640 1 -- 172.21.15.31:0/3305479039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566e80508d0 msgr2=0x5566e80bb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:39.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.811+0000 7f3fbbe00640 1 --2- 172.21.15.31:0/3305479039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566e80508d0 0x5566e80bb2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.811+0000 7f3fbbe00640 1 -- 172.21.15.31:0/3305479039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5566e80469b0 con 0x5566e8047b00 2024-03-21T10:46:39.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.811+0000 7f3fbbe00640 1 --2- 172.21.15.31:0/3305479039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5566e8047b00 0x5566e8054460 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x5566e804ef40 tx=0x7f3fa800bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73dbd0f37978bff9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:39.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.811+0000 7f3fa2a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:46:39.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.811+0000 7f3fa2a00640 1 -- 172.21.15.31:0/3305479039 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa8012450 con 0x5566e8047b00 2024-03-21T10:46:39.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.811+0000 7f3fa2a00640 1 -- 172.21.15.31:0/3305479039 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f3fa80125f0 con 0x5566e8047b00 2024-03-21T10:46:39.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.811+0000 7f3fc67c6740 1 -- 172.21.15.31:0/3305479039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3fb402cce0 con 0x5566e8047b00 2024-03-21T10:46:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.812+0000 7f3fa2a00640 1 -- 172.21.15.31:0/3305479039 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3fa803a050 con 0x5566e8047b00 2024-03-21T10:46:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.812+0000 7f3fc67c6740 10 client.27620 fetch_fsmap learned FSMap version 26 2024-03-21T10:46:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.812+0000 7f3fc67c6740 1 -- 172.21.15.31:0/3305479039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5566e80bc780 con 0x5566e8047b00 2024-03-21T10:46:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.812+0000 7f3fa2a00640 1 -- 172.21.15.31:0/3305479039 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3fb402cce0 con 0x5566e8047b00 2024-03-21T10:46:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.812+0000 7f3fa2a00640 1 -- 172.21.15.31:0/3305479039 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f3fa8005040 con 0x5566e8047b00 2024-03-21T10:46:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.812+0000 7f3fc67c6740 10 client.27620 fetch_fsmap finished waiting for FSMap version 26 2024-03-21T10:46:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.812+0000 7f3fc67c6740 10 client.27620 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:46:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.812+0000 7f3fc67c6740 20 client.27620 populate_metadata read hostname 'smithi031' 2024-03-21T10:46:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.812+0000 7f3fc67c6740 1 --2- 172.21.15.31:0/3305479039 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5566e80bce20 0x5566e80bf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.812+0000 7f3fc67c6740 4 client.27620 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:46:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.812+0000 7f3fc67c6740 1 -- 172.21.15.31:0/3305479039 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5566e7fb30c0 con 0x5566e80bce20 2024-03-21T10:46:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.812+0000 7f3fc1a00640 1 --2- 172.21.15.31:0/3305479039 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5566e80bce20 0x5566e80bf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.813+0000 7f3fc1a00640 1 --2- 172.21.15.31:0/3305479039 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5566e80bce20 0x5566e80bf240 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea6f93d25ce934f5 server_cookie=bbdaa9f7323195ed in_seq=0 out_seq=0 2024-03-21T10:46:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.813+0000 7f3fa2a00640 10 client.27620 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.823+0000 7f3fa2a00640 1 -- 172.21.15.31:0/3305479039 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5566e7fb30c0 con 0x5566e80bce20 2024-03-21T10:46:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.823+0000 7f3fa2a00640 10 client.27620 handle_command_reply: tid=0 2024-03-21T10:46:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.823+0000 7f3fc67c6740 1 client.27620 shutdown 2024-03-21T10:46:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.823+0000 7f3fc67c6740 1 -- 172.21.15.31:0/3305479039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5566e8047b00 msgr2=0x5566e8054460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.823+0000 7f3fc67c6740 1 --2- 172.21.15.31:0/3305479039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5566e8047b00 0x5566e8054460 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x5566e804ef40 tx=0x7f3fa800bb70 comp rx=0 tx=0).stop 2024-03-21T10:46:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.823+0000 7f3fc67c6740 1 -- 172.21.15.31:0/3305479039 shutdown_connections 2024-03-21T10:46:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.823+0000 7f3fc67c6740 1 --2- 172.21.15.31:0/3305479039 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5566e80bce20 0x5566e80bf240 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.823+0000 7f3fc67c6740 1 --2- 172.21.15.31:0/3305479039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566e80508d0 0x5566e80bb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.823+0000 7f3fc67c6740 1 --2- 172.21.15.31:0/3305479039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5566e8047b00 0x5566e8054460 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.823+0000 7f3fc67c6740 1 --2- 172.21.15.31:0/3305479039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566e80471e0 0x5566e8053f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.823+0000 7f3fa2a00640 0 client.27620 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.823+0000 7f3fc67c6740 1 -- 172.21.15.31:0/3305479039 >> 172.21.15.31:0/3305479039 conn(0x5566e8041b00 msgr2=0x5566e804be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.823+0000 7f3fc67c6740 1 -- 172.21.15.31:0/3305479039 shutdown_connections 2024-03-21T10:46:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.824+0000 7f3fc67c6740 1 -- 172.21.15.31:0/3305479039 wait complete. 2024-03-21T10:46:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.824+0000 7f3fc67c6740 20 client.27620 trim_cache size 0 max 16384 2024-03-21T10:46:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:46:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:46:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:46:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:46:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.824+0000 7f3fc2400640 1 -- 172.21.15.31:0/3021328672 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3fa40a30f0 msgr2=0x7f3fa40a55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.824+0000 7f3fc2400640 1 --2- 172.21.15.31:0/3021328672 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3fa40a30f0 0x7f3fa40a55b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x5566e8046c90 tx=0x7f3fb00035f0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.824+0000 7f3fc2400640 1 -- 172.21.15.31:0/3021328672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fbc073a40 msgr2=0x7f3fbc19e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.824+0000 7f3fc2400640 1 --2- 172.21.15.31:0/3021328672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fbc073a40 0x7f3fbc19e440 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0000c00 tx=0x7f3fb00043c0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.824+0000 7f3fc2400640 1 -- 172.21.15.31:0/3021328672 shutdown_connections 2024-03-21T10:46:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.824+0000 7f3fc2400640 1 --2- 172.21.15.31:0/3021328672 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3fa40a30f0 0x7f3fa40a55b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.824+0000 7f3fc2400640 1 --2- 172.21.15.31:0/3021328672 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fbc112490 0x7f3fbc1a2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.824+0000 7f3fc2400640 1 --2- 172.21.15.31:0/3021328672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fbc074360 0x7f3fbc19e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.824+0000 7f3fc2400640 1 --2- 172.21.15.31:0/3021328672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fbc073a40 0x7f3fbc19e440 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.824+0000 7f3fc2400640 1 -- 172.21.15.31:0/3021328672 >> 172.21.15.31:0/3021328672 conn(0x7f3fbc0fcf50 msgr2=0x7f3fbc0fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:39.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.825+0000 7f3fc2400640 1 -- 172.21.15.31:0/3021328672 shutdown_connections 2024-03-21T10:46:39.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:39.825+0000 7f3fc2400640 1 -- 172.21.15.31:0/3021328672 wait complete. 2024-03-21T10:46:41.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:40 smithi040 ceph-mon[36154]: pgmap v250: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-03-21T10:46:41.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:40 smithi031 ceph-mon[31673]: pgmap v250: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-03-21T10:46:41.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:40 smithi070 ceph-mon[36330]: pgmap v250: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-03-21T10:46:43.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:42 smithi040 ceph-mon[36154]: pgmap v251: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 3.0 KiB/s wr, 5 op/s 2024-03-21T10:46:43.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:42 smithi031 ceph-mon[31673]: pgmap v251: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 3.0 KiB/s wr, 5 op/s 2024-03-21T10:46:43.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:42 smithi070 ceph-mon[36330]: pgmap v251: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 3.0 KiB/s wr, 5 op/s 2024-03-21T10:46:45.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:44 smithi040 ceph-mon[36154]: pgmap v252: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.0 KiB/s wr, 6 op/s 2024-03-21T10:46:45.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:44 smithi031 ceph-mon[31673]: pgmap v252: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.0 KiB/s wr, 6 op/s 2024-03-21T10:46:45.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:44 smithi070 ceph-mon[36330]: pgmap v252: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.0 KiB/s wr, 6 op/s 2024-03-21T10:46:47.180 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:46 smithi040 ceph-mon[36154]: pgmap v253: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 10 op/s 2024-03-21T10:46:47.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:46 smithi031 ceph-mon[31673]: pgmap v253: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 10 op/s 2024-03-21T10:46:47.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:46 smithi070 ceph-mon[36330]: pgmap v253: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 10 op/s 2024-03-21T10:46:48.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:47 smithi031 ceph-mon[31673]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:46:48.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:47 smithi070 ceph-mon[36330]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:46:48.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:47 smithi040 ceph-mon[36154]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:46:49.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:48 smithi031 ceph-mon[31673]: pgmap v254: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 KiB/s wr, 7 op/s 2024-03-21T10:46:49.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:48 smithi070 ceph-mon[36330]: pgmap v254: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 KiB/s wr, 7 op/s 2024-03-21T10:46:49.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:48 smithi040 ceph-mon[36154]: pgmap v254: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 KiB/s wr, 7 op/s 2024-03-21T10:46:51.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:50 smithi031 ceph-mon[31673]: pgmap v255: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 11 op/s 2024-03-21T10:46:51.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:50 smithi070 ceph-mon[36330]: pgmap v255: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 11 op/s 2024-03-21T10:46:51.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:50 smithi040 ceph-mon[36154]: pgmap v255: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 11 op/s 2024-03-21T10:46:52.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:52.997+0000 7fdcd4800640 1 -- 172.21.15.31:0/1160787066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdccc1098a0 msgr2=0x7fdccc109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:52.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:52.997+0000 7fdcd4800640 1 --2- 172.21.15.31:0/1160787066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdccc1098a0 0x7fdccc109c80 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0009930 tx=0x7fdcc002bc20 comp rx=0 tx=0).stop 2024-03-21T10:46:52.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:52.997+0000 7fdcd4800640 1 -- 172.21.15.31:0/1160787066 shutdown_connections 2024-03-21T10:46:52.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:52.997+0000 7fdcd4800640 1 --2- 172.21.15.31:0/1160787066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdccc104250 0x7fdccc110760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:52.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:52.997+0000 7fdcd4800640 1 --2- 172.21.15.31:0/1160787066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdccc103890 0x7fdccc103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:52.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:52.997+0000 7fdcd4800640 1 --2- 172.21.15.31:0/1160787066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdccc1098a0 0x7fdccc109c80 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:52.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:52.997+0000 7fdcd4800640 1 -- 172.21.15.31:0/1160787066 >> 172.21.15.31:0/1160787066 conn(0x7fdccc0ff320 msgr2=0x7fdccc101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:52.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:52.997+0000 7fdcd4800640 1 -- 172.21.15.31:0/1160787066 shutdown_connections 2024-03-21T10:46:52.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:52.998+0000 7fdcd4800640 1 -- 172.21.15.31:0/1160787066 wait complete. 2024-03-21T10:46:52.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:52.999+0000 7fdcd4800640 1 Processor -- start 2024-03-21T10:46:52.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:52.999+0000 7fdcd4800640 1 -- start start 2024-03-21T10:46:52.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:52.999+0000 7fdcd4800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdccc103890 0x7fdccc19a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:52.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:52.999+0000 7fdcd4800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdccc104250 0x7fdccc19a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:52.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:52.999+0000 7fdcd4800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdccc1098a0 0x7fdccc19ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:52.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:52.999+0000 7fdcd4800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc113cc0 con 0x7fdccc1098a0 2024-03-21T10:46:52.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:52.999+0000 7fdcd4800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc113b40 con 0x7fdccc103890 2024-03-21T10:46:52.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:52.999+0000 7fdcd4800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc113e40 con 0x7fdccc104250 2024-03-21T10:46:52.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:52.999+0000 7fdcd2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdccc1098a0 0x7fdccc19ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:52.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:52.999+0000 7fdcd1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdccc103890 0x7fdccc19a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:52.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:52.999+0000 7fdcd1400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdccc104250 0x7fdccc19a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:52.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:52.999+0000 7fdcd2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdccc1098a0 0x7fdccc19ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35308/0 (socket says 172.21.15.31:35308) 2024-03-21T10:46:52.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:52.999+0000 7fdcd2800640 1 -- 172.21.15.31:0/967706087 learned_addr learned my addr 172.21.15.31:0/967706087 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:52.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:52.999+0000 7fdcd1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdccc103890 0x7fdccc19a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38996/0 (socket says 172.21.15.31:38996) 2024-03-21T10:46:52.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.000+0000 7fdcd1e00640 1 -- 172.21.15.31:0/967706087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdccc104250 msgr2=0x7fdccc19a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:52.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.000+0000 7fdcd1e00640 1 --2- 172.21.15.31:0/967706087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdccc104250 0x7fdccc19a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:52.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.000+0000 7fdcd1e00640 1 -- 172.21.15.31:0/967706087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdccc1098a0 msgr2=0x7fdccc19ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:52.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.000+0000 7fdcd1e00640 1 --2- 172.21.15.31:0/967706087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdccc1098a0 0x7fdccc19ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:52.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.000+0000 7fdcd1e00640 1 -- 172.21.15.31:0/967706087 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdccc19f170 con 0x7fdccc103890 2024-03-21T10:46:52.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.000+0000 7fdcd2800640 1 --2- 172.21.15.31:0/967706087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdccc1098a0 0x7fdccc19ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:46:52.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.000+0000 7fdcd1e00640 1 --2- 172.21.15.31:0/967706087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdccc103890 0x7fdccc19a180 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0036040 tx=0x7fdcc0004340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:52.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.000+0000 7fdcc6a00640 1 -- 172.21.15.31:0/967706087 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0043020 con 0x7fdccc103890 2024-03-21T10:46:52.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.001+0000 7fdcc6a00640 1 -- 172.21.15.31:0/967706087 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdcc00054e0 con 0x7fdccc103890 2024-03-21T10:46:52.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.001+0000 7fdcd4800640 1 -- 172.21.15.31:0/967706087 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdccc19f400 con 0x7fdccc103890 2024-03-21T10:46:52.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.001+0000 7fdcc6a00640 1 -- 172.21.15.31:0/967706087 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0003ab0 con 0x7fdccc103890 2024-03-21T10:46:52.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.001+0000 7fdcd4800640 1 -- 172.21.15.31:0/967706087 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdccc1a6cf0 con 0x7fdccc103890 2024-03-21T10:46:53.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.004+0000 7fdcc6a00640 1 -- 172.21.15.31:0/967706087 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fdcc0046020 con 0x7fdccc103890 2024-03-21T10:46:53.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.004+0000 7fdcd75d1740 1 Processor -- start 2024-03-21T10:46:53.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.004+0000 7fdcd75d1740 1 -- start start 2024-03-21T10:46:53.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.005+0000 7fdcd75d1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a07d5a7fd0 0x55a07d5a83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:53.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.005+0000 7fdcd75d1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07d5a88f0 0x55a07d5b1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:53.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.005+0000 7fdcd75d1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5b16c0 0x55a07d5b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:53.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.005+0000 7fdcd75d1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a07d56a3b0 con 0x55a07d5a7fd0 2024-03-21T10:46:53.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.005+0000 7fdcd75d1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a07d50c830 con 0x55a07d5b16c0 2024-03-21T10:46:53.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.005+0000 7fdcd75d1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a07d4b55d0 con 0x55a07d5a88f0 2024-03-21T10:46:53.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.005+0000 7fdcc6a00640 1 --2- 172.21.15.31:0/967706087 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fdcb40a3100 0x7fdcb40a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:53.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.005+0000 7fdcd1e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07d5a88f0 0x55a07d5b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:53.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.005+0000 7fdcd1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5b16c0 0x55a07d5b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:53.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.005+0000 7fdcd1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5b16c0 0x55a07d5b3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38998/0 (socket says 172.21.15.31:38998) 2024-03-21T10:46:53.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.005+0000 7fdcd1400640 1 -- 172.21.15.31:0/3759511395 learned_addr learned my addr 172.21.15.31:0/3759511395 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:53.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.005+0000 7fdcd2800640 1 --2- 172.21.15.31:0/3759511395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a07d5a7fd0 0x55a07d5a83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:53.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.005+0000 7fdcc6a00640 1 -- 172.21.15.31:0/967706087 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fdcc00f18b0 con 0x7fdccc103890 2024-03-21T10:46:53.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.005+0000 7fdcd2800640 1 --2- 172.21.15.31:0/967706087 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fdcb40a3100 0x7fdcb40a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:53.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.006+0000 7fdcd1e00640 1 -- 172.21.15.31:0/3759511395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5b16c0 msgr2=0x55a07d5b3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:53.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.006+0000 7fdcd1e00640 1 --2- 172.21.15.31:0/3759511395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5b16c0 0x55a07d5b3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.006+0000 7fdcd1e00640 1 -- 172.21.15.31:0/3759511395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a07d5a7fd0 msgr2=0x55a07d5a83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:53.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.006+0000 7fdcd1e00640 1 --2- 172.21.15.31:0/3759511395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a07d5a7fd0 0x55a07d5a83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.006+0000 7fdcd1e00640 1 -- 172.21.15.31:0/3759511395 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a07d4cd2c0 con 0x55a07d5a88f0 2024-03-21T10:46:53.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.006+0000 7fdcd1400640 1 --2- 172.21.15.31:0/3759511395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5b16c0 0x55a07d5b3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:46:53.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.006+0000 7fdcd1e00640 1 --2- 172.21.15.31:0/3759511395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07d5a88f0 0x55a07d5b1180 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fdcc00eed70 tx=0x7fdcc0005e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13c46f91bbe04c22 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:53.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.006+0000 7fdcbb000640 1 -- 172.21.15.31:0/3759511395 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc00bd550 con 0x55a07d5a88f0 2024-03-21T10:46:53.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.006+0000 7fdcbb000640 1 -- 172.21.15.31:0/3759511395 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdcc0004c80 con 0x55a07d5a88f0 2024-03-21T10:46:53.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.006+0000 7fdcbb000640 1 -- 172.21.15.31:0/3759511395 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc00bd370 con 0x55a07d5a88f0 2024-03-21T10:46:53.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.007+0000 7fdcd75d1740 1 -- 172.21.15.31:0/3759511395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07d5a88f0 msgr2=0x55a07d5b1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:53.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.007+0000 7fdcd75d1740 1 --2- 172.21.15.31:0/3759511395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07d5a88f0 0x55a07d5b1180 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fdcc00eed70 tx=0x7fdcc0005e60 comp rx=0 tx=0).stop 2024-03-21T10:46:53.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.007+0000 7fdcd2800640 1 --2- 172.21.15.31:0/967706087 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fdcb40a3100 0x7fdcb40a55c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8005c40 tx=0x7fdcc8009680 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:53.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.007+0000 7fdcd75d1740 1 -- 172.21.15.31:0/3759511395 shutdown_connections 2024-03-21T10:46:53.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.007+0000 7fdcd75d1740 1 --2- 172.21.15.31:0/3759511395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5b16c0 0x55a07d5b3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.007+0000 7fdcd75d1740 1 --2- 172.21.15.31:0/3759511395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07d5a88f0 0x55a07d5b1180 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.007+0000 7fdcd75d1740 1 --2- 172.21.15.31:0/3759511395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a07d5a7fd0 0x55a07d5a83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.007+0000 7fdcd75d1740 1 -- 172.21.15.31:0/3759511395 >> 172.21.15.31:0/3759511395 conn(0x55a07d5a28f0 msgr2=0x55a07d5a2cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:53.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.007+0000 7fdcd75d1740 1 -- 172.21.15.31:0/3759511395 shutdown_connections 2024-03-21T10:46:53.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.007+0000 7fdcd75d1740 1 -- 172.21.15.31:0/3759511395 wait complete. 2024-03-21T10:46:53.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.008+0000 7fdcd75d1740 1 -- start start 2024-03-21T10:46:53.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.008+0000 7fdcd75d1740 1 Processor -- start 2024-03-21T10:46:53.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.008+0000 7fdcd75d1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5a7fd0 0x55a07d5ae7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:53.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.008+0000 7fdcd75d1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a07d5a88f0 0x55a07d5aed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:53.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.008+0000 7fdcd75d1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07d5b16c0 0x55a07d5af240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:53.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.008+0000 7fdcd75d1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a07d4c8a20 con 0x55a07d5a88f0 2024-03-21T10:46:53.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.008+0000 7fdcd75d1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a07d5088d0 con 0x55a07d5a7fd0 2024-03-21T10:46:53.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.008+0000 7fdcd75d1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a07d5b40b0 con 0x55a07d5b16c0 2024-03-21T10:46:53.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.009+0000 7fdcd75d1740 10 client.0 _ll_register_callbacks cb 0x55a07d5051c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:46:53.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.009+0000 7fdcd1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5a7fd0 0x55a07d5ae7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:53.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.009+0000 7fdcd1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a07d5a88f0 0x55a07d5aed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:53.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.009+0000 7fdcd2800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07d5b16c0 0x55a07d5af240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:53.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.009+0000 7fdcd1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5a7fd0 0x55a07d5ae7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39010/0 (socket says 172.21.15.31:39010) 2024-03-21T10:46:53.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.009+0000 7fdcd2800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07d5b16c0 0x55a07d5af240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38912/0 (socket says 172.21.15.31:38912) 2024-03-21T10:46:53.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.009+0000 7fdcd1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a07d5a88f0 0x55a07d5aed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35326/0 (socket says 172.21.15.31:35326) 2024-03-21T10:46:53.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.009+0000 7fdcd1e00640 1 -- 172.21.15.31:0/1820228550 learned_addr learned my addr 172.21.15.31:0/1820228550 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:53.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.009+0000 7fdcd1e00640 1 -- 172.21.15.31:0/1820228550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07d5b16c0 msgr2=0x55a07d5af240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:53.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.009+0000 7fdcd1e00640 1 --2- 172.21.15.31:0/1820228550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07d5b16c0 0x55a07d5af240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.009+0000 7fdcd1e00640 1 -- 172.21.15.31:0/1820228550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a07d5a88f0 msgr2=0x55a07d5aed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:53.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.009+0000 7fdcd1e00640 1 --2- 172.21.15.31:0/1820228550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a07d5a88f0 0x55a07d5aed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.009+0000 7fdcd1e00640 1 -- 172.21.15.31:0/1820228550 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a07d221210 con 0x55a07d5a7fd0 2024-03-21T10:46:53.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.009+0000 7fdcd2800640 1 --2- 172.21.15.31:0/1820228550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07d5b16c0 0x55a07d5af240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:46:53.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.009+0000 7fdcd1400640 1 --2- 172.21.15.31:0/1820228550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a07d5a88f0 0x55a07d5aed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:46:53.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.010+0000 7fdcd1e00640 1 --2- 172.21.15.31:0/1820228550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5a7fd0 0x55a07d5ae7c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fdcc00ba3e0 tx=0x7fdcc00ba290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=203cf2d981e423bd server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:53.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.010+0000 7fdcaf400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:46:53.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.010+0000 7fdcaf400640 1 -- 172.21.15.31:0/1820228550 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc00db060 con 0x55a07d5a7fd0 2024-03-21T10:46:53.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.010+0000 7fdcaf400640 1 -- 172.21.15.31:0/1820228550 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fdcc00381c0 con 0x55a07d5a7fd0 2024-03-21T10:46:53.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.010+0000 7fdcd75d1740 1 -- 172.21.15.31:0/1820228550 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdcc0004c80 con 0x55a07d5a7fd0 2024-03-21T10:46:53.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.010+0000 7fdcaf400640 1 -- 172.21.15.31:0/1820228550 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdcc00d6070 con 0x55a07d5a7fd0 2024-03-21T10:46:53.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.010+0000 7fdcd75d1740 10 client.27632 fetch_fsmap learned FSMap version 28 2024-03-21T10:46:53.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.010+0000 7fdcd75d1740 1 -- 172.21.15.31:0/1820228550 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a07d2ed620 con 0x55a07d5a7fd0 2024-03-21T10:46:53.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.010+0000 7fdcaf400640 1 -- 172.21.15.31:0/1820228550 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdcc0004c80 con 0x55a07d5a7fd0 2024-03-21T10:46:53.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.010+0000 7fdcaf400640 1 -- 172.21.15.31:0/1820228550 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fdcc0003c50 con 0x55a07d5a7fd0 2024-03-21T10:46:53.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.011+0000 7fdcd75d1740 10 client.27632 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:46:53.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.011+0000 7fdcd75d1740 10 client.27632 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:46:53.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.011+0000 7fdcd75d1740 20 client.27632 populate_metadata read hostname 'smithi031' 2024-03-21T10:46:53.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.011+0000 7fdcd75d1740 1 --2- 172.21.15.31:0/1820228550 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a07d5c1360 0x55a07d5c3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:53.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.011+0000 7fdcd75d1740 4 client.27632 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:46:53.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.011+0000 7fdcd75d1740 1 -- 172.21.15.31:0/1820228550 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a07d519d70 con 0x55a07d5c1360 2024-03-21T10:46:53.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.011+0000 7fdcd1400640 1 --2- 172.21.15.31:0/1820228550 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a07d5c1360 0x55a07d5c3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:53.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.011+0000 7fdcd1400640 1 --2- 172.21.15.31:0/1820228550 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a07d5c1360 0x55a07d5c3750 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe260c416f48e643 server_cookie=7491ff8067f6ed0 in_seq=0 out_seq=0 2024-03-21T10:46:53.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.011+0000 7fdcaf400640 10 client.27632 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:53.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.012+0000 7fdcaf400640 1 -- 172.21.15.31:0/1820228550 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a07d519d70 con 0x55a07d5c1360 2024-03-21T10:46:53.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.012+0000 7fdcaf400640 10 client.27632 handle_command_reply: tid=0 2024-03-21T10:46:53.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.012+0000 7fdcd75d1740 1 client.27632 shutdown 2024-03-21T10:46:53.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.012+0000 7fdcd75d1740 1 -- 172.21.15.31:0/1820228550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5a7fd0 msgr2=0x55a07d5ae7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:53.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.012+0000 7fdcd75d1740 1 --2- 172.21.15.31:0/1820228550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5a7fd0 0x55a07d5ae7c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fdcc00ba3e0 tx=0x7fdcc00ba290 comp rx=0 tx=0).stop 2024-03-21T10:46:53.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.013+0000 7fdcd75d1740 1 -- 172.21.15.31:0/1820228550 shutdown_connections 2024-03-21T10:46:53.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.013+0000 7fdcd75d1740 1 --2- 172.21.15.31:0/1820228550 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a07d5c1360 0x55a07d5c3750 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.013+0000 7fdcd75d1740 1 --2- 172.21.15.31:0/1820228550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07d5b16c0 0x55a07d5af240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.013+0000 7fdcd75d1740 1 --2- 172.21.15.31:0/1820228550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a07d5a88f0 0x55a07d5aed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.013+0000 7fdcaf400640 0 client.27632 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:53.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.013+0000 7fdcd75d1740 1 --2- 172.21.15.31:0/1820228550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5a7fd0 0x55a07d5ae7c0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.013+0000 7fdcd75d1740 1 -- 172.21.15.31:0/1820228550 >> 172.21.15.31:0/1820228550 conn(0x55a07d5a28f0 msgr2=0x55a07d5a4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:53.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.013+0000 7fdcd75d1740 1 -- 172.21.15.31:0/1820228550 shutdown_connections 2024-03-21T10:46:53.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.013+0000 7fdcd75d1740 1 -- 172.21.15.31:0/1820228550 wait complete. 2024-03-21T10:46:53.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.013+0000 7fdcd75d1740 20 client.27632 trim_cache size 0 max 16384 2024-03-21T10:46:53.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.031+0000 7fdcd75d1740 1 Processor -- start 2024-03-21T10:46:53.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.032+0000 7fdcd75d1740 1 -- start start 2024-03-21T10:46:53.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.032+0000 7fdcd75d1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a07d5a7fd0 0x55a07d5b0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:53.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.032+0000 7fdcd75d1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07d5a88f0 0x55a07d5bdf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:53.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.032+0000 7fdcd75d1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5b16c0 0x55a07d5ad920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:53.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.032+0000 7fdcd75d1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc1b3150 con 0x55a07d5a7fd0 2024-03-21T10:46:53.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.032+0000 7fdcd75d1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc1b0c50 con 0x55a07d5b16c0 2024-03-21T10:46:53.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.032+0000 7fdcd75d1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc1aed10 con 0x55a07d5a88f0 2024-03-21T10:46:53.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.032+0000 7fdcd2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5b16c0 0x55a07d5ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:53.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.032+0000 7fdcd2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5b16c0 0x55a07d5ad920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39018/0 (socket says 172.21.15.31:39018) 2024-03-21T10:46:53.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.032+0000 7fdcd2800640 1 -- 172.21.15.31:0/3034123048 learned_addr learned my addr 172.21.15.31:0/3034123048 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:53.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.032+0000 7fdcd1e00640 1 --2- 172.21.15.31:0/3034123048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a07d5a7fd0 0x55a07d5b0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:53.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.032+0000 7fdcd1400640 1 --2- 172.21.15.31:0/3034123048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07d5a88f0 0x55a07d5bdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:53.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.033+0000 7fdcd2800640 1 -- 172.21.15.31:0/3034123048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07d5a88f0 msgr2=0x55a07d5bdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:53.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.033+0000 7fdcd2800640 1 --2- 172.21.15.31:0/3034123048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07d5a88f0 0x55a07d5bdf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.033+0000 7fdcd2800640 1 -- 172.21.15.31:0/3034123048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a07d5a7fd0 msgr2=0x55a07d5b0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:53.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.033+0000 7fdcd2800640 1 --2- 172.21.15.31:0/3034123048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a07d5a7fd0 0x55a07d5b0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.033+0000 7fdcd2800640 1 -- 172.21.15.31:0/3034123048 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a07d5a8720 con 0x55a07d5b16c0 2024-03-21T10:46:53.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.033+0000 7fdcd1e00640 1 --2- 172.21.15.31:0/3034123048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a07d5a7fd0 0x55a07d5b0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:46:53.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.033+0000 7fdcd2800640 1 --2- 172.21.15.31:0/3034123048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5b16c0 0x55a07d5ad920 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fdcc800ef20 tx=0x7fdcc8005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa042a3d66d580de server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:53.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.033+0000 7fdcba600640 1 -- 172.21.15.31:0/3034123048 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8004d20 con 0x55a07d5b16c0 2024-03-21T10:46:53.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.033+0000 7fdcba600640 1 -- 172.21.15.31:0/3034123048 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdcc801b070 con 0x55a07d5b16c0 2024-03-21T10:46:53.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.033+0000 7fdcba600640 1 -- 172.21.15.31:0/3034123048 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8021780 con 0x55a07d5b16c0 2024-03-21T10:46:53.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.034+0000 7fdcd75d1740 1 -- 172.21.15.31:0/3034123048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5b16c0 msgr2=0x55a07d5ad920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:53.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.034+0000 7fdcd75d1740 1 --2- 172.21.15.31:0/3034123048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5b16c0 0x55a07d5ad920 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fdcc800ef20 tx=0x7fdcc8005bd0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.034+0000 7fdcd75d1740 1 -- 172.21.15.31:0/3034123048 shutdown_connections 2024-03-21T10:46:53.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.034+0000 7fdcd75d1740 1 --2- 172.21.15.31:0/3034123048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5b16c0 0x55a07d5ad920 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.034+0000 7fdcd75d1740 1 --2- 172.21.15.31:0/3034123048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07d5a88f0 0x55a07d5bdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.034+0000 7fdcd75d1740 1 --2- 172.21.15.31:0/3034123048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a07d5a7fd0 0x55a07d5b0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.034+0000 7fdcd75d1740 1 -- 172.21.15.31:0/3034123048 >> 172.21.15.31:0/3034123048 conn(0x55a07d5a28f0 msgr2=0x55a07d5a10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:53.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.034+0000 7fdcd75d1740 1 -- 172.21.15.31:0/3034123048 shutdown_connections 2024-03-21T10:46:53.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.034+0000 7fdcd75d1740 1 -- 172.21.15.31:0/3034123048 wait complete. 2024-03-21T10:46:53.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.035+0000 7fdcd75d1740 1 -- start start 2024-03-21T10:46:53.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.035+0000 7fdcd75d1740 1 Processor -- start 2024-03-21T10:46:53.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.036+0000 7fdcd75d1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5a7fd0 0x55a07d5b4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:53.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.036+0000 7fdcd75d1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07d5a88f0 0x55a07d5b4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:53.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.036+0000 7fdcd75d1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a07d5b16c0 0x55a07d5b52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:53.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.036+0000 7fdcd75d1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc1af310 con 0x55a07d5b16c0 2024-03-21T10:46:53.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.036+0000 7fdcd75d1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc1b05f0 con 0x55a07d5a7fd0 2024-03-21T10:46:53.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.036+0000 7fdcd75d1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc1af930 con 0x55a07d5a88f0 2024-03-21T10:46:53.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.036+0000 7fdcd75d1740 10 client.0 _ll_register_callbacks cb 0x7fdcbc009580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:46:53.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.036+0000 7fdcd1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5a7fd0 0x55a07d5b4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:53.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.036+0000 7fdcd1400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07d5a88f0 0x55a07d5b4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:53.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.036+0000 7fdcd1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5a7fd0 0x55a07d5b4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39032/0 (socket says 172.21.15.31:39032) 2024-03-21T10:46:53.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.036+0000 7fdcd1e00640 1 -- 172.21.15.31:0/2520568381 learned_addr learned my addr 172.21.15.31:0/2520568381 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:53.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.036+0000 7fdcd2800640 1 --2- 172.21.15.31:0/2520568381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a07d5b16c0 0x55a07d5b52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:53.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.037+0000 7fdcd1400640 1 -- 172.21.15.31:0/2520568381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5a7fd0 msgr2=0x55a07d5b4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:53.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.037+0000 7fdcd1400640 1 --2- 172.21.15.31:0/2520568381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5a7fd0 0x55a07d5b4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.037+0000 7fdcd1400640 1 -- 172.21.15.31:0/2520568381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a07d5b16c0 msgr2=0x55a07d5b52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:53.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.037+0000 7fdcd1400640 1 --2- 172.21.15.31:0/2520568381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a07d5b16c0 0x55a07d5b52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.037+0000 7fdcd1400640 1 -- 172.21.15.31:0/2520568381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a07d61bdc0 con 0x55a07d5a88f0 2024-03-21T10:46:53.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.037+0000 7fdcd1e00640 1 --2- 172.21.15.31:0/2520568381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5a7fd0 0x55a07d5b4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:46:53.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.037+0000 7fdcd1400640 1 --2- 172.21.15.31:0/2520568381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07d5a88f0 0x55a07d5b4da0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x55a07d5afd90 tx=0x7fdcbc020b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac1143d512b65842 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:53.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.037+0000 7fdcaf400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:46:53.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.037+0000 7fdcaf400640 1 -- 172.21.15.31:0/2520568381 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc020270 con 0x55a07d5a88f0 2024-03-21T10:46:53.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.037+0000 7fdcaf400640 1 -- 172.21.15.31:0/2520568381 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fdcbc012b80 con 0x55a07d5a88f0 2024-03-21T10:46:53.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.037+0000 7fdcd75d1740 1 -- 172.21.15.31:0/2520568381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdcc801b070 con 0x55a07d5a88f0 2024-03-21T10:46:53.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.038+0000 7fdcaf400640 1 -- 172.21.15.31:0/2520568381 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdcbc031050 con 0x55a07d5a88f0 2024-03-21T10:46:53.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.038+0000 7fdcd75d1740 10 client.27559 fetch_fsmap learned FSMap version 28 2024-03-21T10:46:53.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.038+0000 7fdcd75d1740 1 -- 172.21.15.31:0/2520568381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a07d61d110 con 0x55a07d5a88f0 2024-03-21T10:46:53.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.038+0000 7fdcaf400640 1 -- 172.21.15.31:0/2520568381 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdcc801b070 con 0x55a07d5a88f0 2024-03-21T10:46:53.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.038+0000 7fdcaf400640 1 -- 172.21.15.31:0/2520568381 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fdcbc029420 con 0x55a07d5a88f0 2024-03-21T10:46:53.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.038+0000 7fdcd75d1740 10 client.27559 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:46:53.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.038+0000 7fdcd75d1740 10 client.27559 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:46:53.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.038+0000 7fdcd75d1740 20 client.27559 populate_metadata read hostname 'smithi031' 2024-03-21T10:46:53.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.038+0000 7fdcd75d1740 1 --2- 172.21.15.31:0/2520568381 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a07d61d840 0x55a07d61fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:53.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.038+0000 7fdcd75d1740 4 client.27559 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:46:53.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.038+0000 7fdcd75d1740 1 -- 172.21.15.31:0/2520568381 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a07d5140c0 con 0x55a07d61d840 2024-03-21T10:46:53.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.038+0000 7fdcd1e00640 1 --2- 172.21.15.31:0/2520568381 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a07d61d840 0x55a07d61fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:53.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.039+0000 7fdcd1e00640 1 --2- 172.21.15.31:0/2520568381 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a07d61d840 0x55a07d61fc60 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3499dac7ef6e4f11 server_cookie=de9bdbfa1aa23abe in_seq=0 out_seq=0 2024-03-21T10:46:53.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.039+0000 7fdcaf400640 10 client.27559 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:53.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.047+0000 7fdcaf400640 1 -- 172.21.15.31:0/2520568381 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a07d5140c0 con 0x55a07d61d840 2024-03-21T10:46:53.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.047+0000 7fdcaf400640 10 client.27559 handle_command_reply: tid=0 2024-03-21T10:46:53.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.048+0000 7fdcd75d1740 1 client.27559 shutdown 2024-03-21T10:46:53.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.048+0000 7fdcd75d1740 1 -- 172.21.15.31:0/2520568381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07d5a88f0 msgr2=0x55a07d5b4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:53.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.048+0000 7fdcd75d1740 1 --2- 172.21.15.31:0/2520568381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07d5a88f0 0x55a07d5b4da0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x55a07d5afd90 tx=0x7fdcbc020b40 comp rx=0 tx=0).stop 2024-03-21T10:46:53.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.048+0000 7fdcd75d1740 1 -- 172.21.15.31:0/2520568381 shutdown_connections 2024-03-21T10:46:53.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.048+0000 7fdcd75d1740 1 --2- 172.21.15.31:0/2520568381 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a07d61d840 0x55a07d61fc60 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.048+0000 7fdcd75d1740 1 --2- 172.21.15.31:0/2520568381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a07d5b16c0 0x55a07d5b52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.048+0000 7fdcd75d1740 1 --2- 172.21.15.31:0/2520568381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07d5a88f0 0x55a07d5b4da0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.048+0000 7fdcd75d1740 1 --2- 172.21.15.31:0/2520568381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a07d5a7fd0 0x55a07d5b4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.048+0000 7fdcd75d1740 1 -- 172.21.15.31:0/2520568381 >> 172.21.15.31:0/2520568381 conn(0x55a07d5a28f0 msgr2=0x55a07d5a1950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:53.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.048+0000 7fdcaf400640 0 client.27559 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:53.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.048+0000 7fdcd75d1740 1 -- 172.21.15.31:0/2520568381 shutdown_connections 2024-03-21T10:46:53.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.048+0000 7fdcd75d1740 1 -- 172.21.15.31:0/2520568381 wait complete. 2024-03-21T10:46:53.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.048+0000 7fdcd75d1740 20 client.27559 trim_cache size 0 max 16384 2024-03-21T10:46:53.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:46:53.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:46:53.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:46:53.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:46:53.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.049+0000 7fdcd4800640 1 -- 172.21.15.31:0/967706087 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fdcb40a3100 msgr2=0x7fdcb40a55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:53.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.049+0000 7fdcd4800640 1 --2- 172.21.15.31:0/967706087 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fdcb40a3100 0x7fdcb40a55c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8005c40 tx=0x7fdcc8009680 comp rx=0 tx=0).stop 2024-03-21T10:46:53.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.049+0000 7fdcd4800640 1 -- 172.21.15.31:0/967706087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdccc103890 msgr2=0x7fdccc19a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:53.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.049+0000 7fdcd4800640 1 --2- 172.21.15.31:0/967706087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdccc103890 0x7fdccc19a180 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0036040 tx=0x7fdcc0004340 comp rx=0 tx=0).stop 2024-03-21T10:46:53.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.049+0000 7fdcd4800640 1 -- 172.21.15.31:0/967706087 shutdown_connections 2024-03-21T10:46:53.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.049+0000 7fdcd4800640 1 --2- 172.21.15.31:0/967706087 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fdcb40a3100 0x7fdcb40a55c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.049+0000 7fdcd4800640 1 --2- 172.21.15.31:0/967706087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdccc1098a0 0x7fdccc19ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.049+0000 7fdcd4800640 1 --2- 172.21.15.31:0/967706087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdccc104250 0x7fdccc19a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.049+0000 7fdcd4800640 1 --2- 172.21.15.31:0/967706087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdccc103890 0x7fdccc19a180 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:53.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.049+0000 7fdcd4800640 1 -- 172.21.15.31:0/967706087 >> 172.21.15.31:0/967706087 conn(0x7fdccc0ff320 msgr2=0x7fdccc0ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:53.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.049+0000 7fdcd4800640 1 -- 172.21.15.31:0/967706087 shutdown_connections 2024-03-21T10:46:53.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:46:53.049+0000 7fdcd4800640 1 -- 172.21.15.31:0/967706087 wait complete. 2024-03-21T10:46:53.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:52 smithi031 ceph-mon[31673]: pgmap v256: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 8 op/s 2024-03-21T10:46:53.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:52 smithi031 ceph-mon[31673]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:46:53.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:52 smithi070 ceph-mon[36330]: pgmap v256: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 8 op/s 2024-03-21T10:46:53.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:52 smithi070 ceph-mon[36330]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:46:53.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:52 smithi040 ceph-mon[36154]: pgmap v256: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 8 op/s 2024-03-21T10:46:53.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:52 smithi040 ceph-mon[36154]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:46:55.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:54 smithi031 ceph-mon[31673]: pgmap v257: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 17 KiB/s wr, 8 op/s 2024-03-21T10:46:55.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:54 smithi070 ceph-mon[36330]: pgmap v257: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 17 KiB/s wr, 8 op/s 2024-03-21T10:46:55.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:54 smithi040 ceph-mon[36154]: pgmap v257: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 17 KiB/s wr, 8 op/s 2024-03-21T10:46:57.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:56 smithi031 ceph-mon[31673]: pgmap v258: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-03-21T10:46:57.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:56 smithi070 ceph-mon[36330]: pgmap v258: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-03-21T10:46:57.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:56 smithi040 ceph-mon[36154]: pgmap v258: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-03-21T10:46:58.871 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:46:58.871 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:46:59.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.098+0000 7f2473e00640 1 -- 172.21.15.31:0/3232442205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f247410ba50 msgr2=0x7f247410ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:59.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.098+0000 7f2473e00640 1 --2- 172.21.15.31:0/3232442205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f247410ba50 0x7f247410ded0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f246c009be0 tx=0x7f246c02bf80 comp rx=0 tx=0).stop 2024-03-21T10:46:59.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.098+0000 7f2473e00640 1 -- 172.21.15.31:0/3232442205 shutdown_connections 2024-03-21T10:46:59.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.098+0000 7f2473e00640 1 --2- 172.21.15.31:0/3232442205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f247410ba50 0x7f247410ded0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.098+0000 7f2473e00640 1 --2- 172.21.15.31:0/3232442205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2474102f50 0x7f247410b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.098+0000 7f2473e00640 1 --2- 172.21.15.31:0/3232442205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2474102630 0x7f2474102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.098+0000 7f2473e00640 1 -- 172.21.15.31:0/3232442205 >> 172.21.15.31:0/3232442205 conn(0x7f2474077130 msgr2=0x7f24740fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:59.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.098+0000 7f2473e00640 1 -- 172.21.15.31:0/3232442205 shutdown_connections 2024-03-21T10:46:59.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.098+0000 7f2473e00640 1 -- 172.21.15.31:0/3232442205 wait complete. 2024-03-21T10:46:59.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.099+0000 7f2473e00640 1 Processor -- start 2024-03-21T10:46:59.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.099+0000 7f2473e00640 1 -- start start 2024-03-21T10:46:59.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.100+0000 7f2473e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2474102630 0x7f2474199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.100+0000 7f2473e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2474102f50 0x7f247419a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.100+0000 7f2473e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f247410ba50 0x7f247419e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.100+0000 7f2473e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2474111840 con 0x7f2474102630 2024-03-21T10:46:59.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.100+0000 7f2473e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24741116c0 con 0x7f247410ba50 2024-03-21T10:46:59.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.100+0000 7f2473e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24741119c0 con 0x7f2474102f50 2024-03-21T10:46:59.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.100+0000 7f2473400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f247410ba50 0x7f247419e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.100+0000 7f246be00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2474102f50 0x7f247419a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.100+0000 7f2473400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f247410ba50 0x7f247419e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42346/0 (socket says 172.21.15.31:42346) 2024-03-21T10:46:59.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.100+0000 7f2473400640 1 -- 172.21.15.31:0/2737264276 learned_addr learned my addr 172.21.15.31:0/2737264276 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:59.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.100+0000 7f2472a00640 1 --2- 172.21.15.31:0/2737264276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2474102630 0x7f2474199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.100+0000 7f2473400640 1 -- 172.21.15.31:0/2737264276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2474102f50 msgr2=0x7f247419a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:59.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.100+0000 7f2473400640 1 --2- 172.21.15.31:0/2737264276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2474102f50 0x7f247419a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.100+0000 7f2473400640 1 -- 172.21.15.31:0/2737264276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2474102630 msgr2=0x7f2474199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:59.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.101+0000 7f2473400640 1 --2- 172.21.15.31:0/2737264276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2474102630 0x7f2474199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.101+0000 7f2473400640 1 -- 172.21.15.31:0/2737264276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247419eed0 con 0x7f247410ba50 2024-03-21T10:46:59.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.101+0000 7f2472a00640 1 --2- 172.21.15.31:0/2737264276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2474102630 0x7f2474199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:46:59.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.101+0000 7f246be00640 1 --2- 172.21.15.31:0/2737264276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2474102f50 0x7f247419a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:46:59.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.101+0000 7f2473400640 1 --2- 172.21.15.31:0/2737264276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f247410ba50 0x7f247419e7a0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f246c036040 tx=0x7f246c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:59.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.101+0000 7f2478a00640 1 -- 172.21.15.31:0/2737264276 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c043020 con 0x7f247410ba50 2024-03-21T10:46:59.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.101+0000 7f2478a00640 1 -- 172.21.15.31:0/2737264276 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f246c0045f0 con 0x7f247410ba50 2024-03-21T10:46:59.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.101+0000 7f2473e00640 1 -- 172.21.15.31:0/2737264276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f247419f160 con 0x7f247410ba50 2024-03-21T10:46:59.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.101+0000 7f2478a00640 1 -- 172.21.15.31:0/2737264276 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c03e6b0 con 0x7f247410ba50 2024-03-21T10:46:59.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.101+0000 7f2473e00640 1 -- 172.21.15.31:0/2737264276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24741a6a50 con 0x7f247410ba50 2024-03-21T10:46:59.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.105+0000 7f2478a00640 1 -- 172.21.15.31:0/2737264276 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f246c046020 con 0x7f247410ba50 2024-03-21T10:46:59.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.106+0000 7f247cb7d740 1 Processor -- start 2024-03-21T10:46:59.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.106+0000 7f247cb7d740 1 -- start start 2024-03-21T10:46:59.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.106+0000 7f247cb7d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568c0712dd0 0x5568c0711b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.107+0000 7f247cb7d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568c07120f0 0x5568c0712570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.107+0000 7f247cb7d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568c071b460 0x5568c071d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.107+0000 7f247cb7d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568c0676250 con 0x5568c071b460 2024-03-21T10:46:59.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.107+0000 7f247cb7d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568c061ea30 con 0x5568c07120f0 2024-03-21T10:46:59.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.107+0000 7f247cb7d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568c06722f0 con 0x5568c0712dd0 2024-03-21T10:46:59.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.107+0000 7f2478a00640 1 --2- 172.21.15.31:0/2737264276 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24580a3090 0x7f24580a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.107+0000 7f2472a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568c07120f0 0x5568c0712570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.107+0000 7f2472a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568c07120f0 0x5568c0712570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42358/0 (socket says 172.21.15.31:42358) 2024-03-21T10:46:59.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.107+0000 7f2472a00640 1 -- 172.21.15.31:0/1175416271 learned_addr learned my addr 172.21.15.31:0/1175416271 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:59.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.107+0000 7f246be00640 1 --2- 172.21.15.31:0/1175416271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568c0712dd0 0x5568c0711b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.107+0000 7f2478a00640 1 -- 172.21.15.31:0/2737264276 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f246c0f2260 con 0x7f247410ba50 2024-03-21T10:46:59.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.107+0000 7f2473400640 1 --2- 172.21.15.31:0/2737264276 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24580a3090 0x7f24580a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.107+0000 7f246be00640 1 --2- 172.21.15.31:0/1175416271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568c071b460 0x5568c071d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.107+0000 7f2472a00640 1 -- 172.21.15.31:0/1175416271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568c0712dd0 msgr2=0x5568c0711b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:59.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.107+0000 7f2472a00640 1 --2- 172.21.15.31:0/1175416271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568c0712dd0 0x5568c0711b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.107+0000 7f2472a00640 1 -- 172.21.15.31:0/1175416271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568c071b460 msgr2=0x5568c071d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:59.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.107+0000 7f2472a00640 1 --2- 172.21.15.31:0/1175416271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568c071b460 0x5568c071d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.107+0000 7f2472a00640 1 -- 172.21.15.31:0/1175416271 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5568c0609690 con 0x5568c07120f0 2024-03-21T10:46:59.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.108+0000 7f2472a00640 1 --2- 172.21.15.31:0/1175416271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568c07120f0 0x5568c0712570 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f2464009cf0 tx=0x7f24640117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f39e88a78bc502f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:59.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.108+0000 7f2463e00640 1 -- 172.21.15.31:0/1175416271 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2464012450 con 0x5568c07120f0 2024-03-21T10:46:59.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.108+0000 7f2463e00640 1 -- 172.21.15.31:0/1175416271 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2464012ad0 con 0x5568c07120f0 2024-03-21T10:46:59.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.108+0000 7f2463e00640 1 -- 172.21.15.31:0/1175416271 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246400fc70 con 0x5568c07120f0 2024-03-21T10:46:59.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.109+0000 7f247cb7d740 1 -- 172.21.15.31:0/1175416271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568c07120f0 msgr2=0x5568c0712570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:59.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.109+0000 7f2473400640 1 --2- 172.21.15.31:0/2737264276 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24580a3090 0x7f24580a5550 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f246c004af0 tx=0x7f246c02bf80 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:59.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.109+0000 7f247cb7d740 1 --2- 172.21.15.31:0/1175416271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568c07120f0 0x5568c0712570 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f2464009cf0 tx=0x7f24640117b0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.109+0000 7f247cb7d740 1 -- 172.21.15.31:0/1175416271 shutdown_connections 2024-03-21T10:46:59.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.109+0000 7f247cb7d740 1 --2- 172.21.15.31:0/1175416271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568c071b460 0x5568c071d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.109+0000 7f247cb7d740 1 --2- 172.21.15.31:0/1175416271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568c07120f0 0x5568c0712570 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.109+0000 7f247cb7d740 1 --2- 172.21.15.31:0/1175416271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568c0712dd0 0x5568c0711b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.109+0000 7f247cb7d740 1 -- 172.21.15.31:0/1175416271 >> 172.21.15.31:0/1175416271 conn(0x5568c06d9200 msgr2=0x5568c070cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:59.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.109+0000 7f247cb7d740 1 -- 172.21.15.31:0/1175416271 shutdown_connections 2024-03-21T10:46:59.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.109+0000 7f247cb7d740 1 -- 172.21.15.31:0/1175416271 wait complete. 2024-03-21T10:46:59.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.110+0000 7f247cb7d740 1 -- start start 2024-03-21T10:46:59.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.110+0000 7f247cb7d740 1 Processor -- start 2024-03-21T10:46:59.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.110+0000 7f247cb7d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568c07120f0 0x5568c071ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.110+0000 7f247cb7d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568c0712dd0 0x5568c0717ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.110+0000 7f247cb7d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568c071b460 0x5568c0718400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.110+0000 7f247cb7d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568c071e2d0 con 0x5568c071b460 2024-03-21T10:46:59.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.110+0000 7f247cb7d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568c0631e80 con 0x5568c07120f0 2024-03-21T10:46:59.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.110+0000 7f247cb7d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568c071e450 con 0x5568c0712dd0 2024-03-21T10:46:59.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.111+0000 7f247cb7d740 10 client.0 _ll_register_callbacks cb 0x5568c0644130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:46:59.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.111+0000 7f246be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568c07120f0 0x5568c071ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.111+0000 7f2472a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568c0712dd0 0x5568c0717ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.111+0000 7f2472a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568c0712dd0 0x5568c0717ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53682/0 (socket says 172.21.15.31:53682) 2024-03-21T10:46:59.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.111+0000 7f2472a00640 1 -- 172.21.15.31:0/1922907377 learned_addr learned my addr 172.21.15.31:0/1922907377 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:59.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.111+0000 7f246be00640 1 --2- 172.21.15.31:0/1922907377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568c071b460 0x5568c0718400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.111+0000 7f2472a00640 1 -- 172.21.15.31:0/1922907377 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568c07120f0 msgr2=0x5568c071ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:59.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.111+0000 7f2472a00640 1 --2- 172.21.15.31:0/1922907377 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568c07120f0 0x5568c071ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.111+0000 7f2472a00640 1 -- 172.21.15.31:0/1922907377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568c071b460 msgr2=0x5568c0718400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:59.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.111+0000 7f2472a00640 1 --2- 172.21.15.31:0/1922907377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568c071b460 0x5568c0718400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.111+0000 7f2472a00640 1 -- 172.21.15.31:0/1922907377 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5568c04547d0 con 0x5568c0712dd0 2024-03-21T10:46:59.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.111+0000 7f2472a00640 1 --2- 172.21.15.31:0/1922907377 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568c0712dd0 0x5568c0717ec0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f2464006010 tx=0x7f24640056d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=103d211ff99f458f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:59.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.112+0000 7f2460c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:46:59.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.112+0000 7f2460c00640 1 -- 172.21.15.31:0/1922907377 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2464028070 con 0x5568c0712dd0 2024-03-21T10:46:59.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.112+0000 7f2460c00640 1 -- 172.21.15.31:0/1922907377 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f2464004450 con 0x5568c0712dd0 2024-03-21T10:46:59.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.112+0000 7f247cb7d740 1 -- 172.21.15.31:0/1922907377 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2464012ad0 con 0x5568c0712dd0 2024-03-21T10:46:59.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.112+0000 7f2460c00640 1 -- 172.21.15.31:0/1922907377 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2464034050 con 0x5568c0712dd0 2024-03-21T10:46:59.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.112+0000 7f247cb7d740 10 client.27571 fetch_fsmap learned FSMap version 28 2024-03-21T10:46:59.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.112+0000 7f247cb7d740 1 -- 172.21.15.31:0/1922907377 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5568c038a3f0 con 0x5568c0712dd0 2024-03-21T10:46:59.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.112+0000 7f2460c00640 1 -- 172.21.15.31:0/1922907377 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2464012ad0 con 0x5568c0712dd0 2024-03-21T10:46:59.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.113+0000 7f2460c00640 1 -- 172.21.15.31:0/1922907377 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f24640035f0 con 0x5568c0712dd0 2024-03-21T10:46:59.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.113+0000 7f247cb7d740 10 client.27571 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:46:59.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.113+0000 7f247cb7d740 10 client.27571 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:46:59.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.113+0000 7f247cb7d740 20 client.27571 populate_metadata read hostname 'smithi031' 2024-03-21T10:46:59.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.113+0000 7f247cb7d740 1 --2- 172.21.15.31:0/1922907377 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5568c072b280 0x5568c072d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.113+0000 7f247cb7d740 4 client.27571 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:46:59.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.113+0000 7f247cb7d740 1 -- 172.21.15.31:0/1922907377 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5568c0683790 con 0x5568c072b280 2024-03-21T10:46:59.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.113+0000 7f246be00640 1 --2- 172.21.15.31:0/1922907377 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5568c072b280 0x5568c072d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.114+0000 7f246be00640 1 --2- 172.21.15.31:0/1922907377 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5568c072b280 0x5568c072d670 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44391ffff3e48c94 server_cookie=a5c5f9feba2a364c in_seq=0 out_seq=0 2024-03-21T10:46:59.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.114+0000 7f2460c00640 10 client.27571 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:59.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.115+0000 7f2460c00640 1 -- 172.21.15.31:0/1922907377 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5568c0683790 con 0x5568c072b280 2024-03-21T10:46:59.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.115+0000 7f2460c00640 10 client.27571 handle_command_reply: tid=0 2024-03-21T10:46:59.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.115+0000 7f247cb7d740 1 client.27571 shutdown 2024-03-21T10:46:59.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.115+0000 7f247cb7d740 1 -- 172.21.15.31:0/1922907377 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568c0712dd0 msgr2=0x5568c0717ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:59.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.115+0000 7f247cb7d740 1 --2- 172.21.15.31:0/1922907377 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568c0712dd0 0x5568c0717ec0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f2464006010 tx=0x7f24640056d0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.115+0000 7f247cb7d740 1 -- 172.21.15.31:0/1922907377 shutdown_connections 2024-03-21T10:46:59.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.115+0000 7f247cb7d740 1 --2- 172.21.15.31:0/1922907377 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5568c072b280 0x5568c072d670 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.115+0000 7f247cb7d740 1 --2- 172.21.15.31:0/1922907377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568c071b460 0x5568c0718400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.115+0000 7f247cb7d740 1 --2- 172.21.15.31:0/1922907377 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568c0712dd0 0x5568c0717ec0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.115+0000 7f247cb7d740 1 --2- 172.21.15.31:0/1922907377 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568c07120f0 0x5568c071ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.115+0000 7f247cb7d740 1 -- 172.21.15.31:0/1922907377 >> 172.21.15.31:0/1922907377 conn(0x5568c06d9200 msgr2=0x5568c070e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:59.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.115+0000 7f2460c00640 0 client.27571 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:59.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.115+0000 7f247cb7d740 1 -- 172.21.15.31:0/1922907377 shutdown_connections 2024-03-21T10:46:59.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.115+0000 7f247cb7d740 1 -- 172.21.15.31:0/1922907377 wait complete. 2024-03-21T10:46:59.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.115+0000 7f247cb7d740 20 client.27571 trim_cache size 0 max 16384 2024-03-21T10:46:59.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.134+0000 7f247cb7d740 1 Processor -- start 2024-03-21T10:46:59.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.134+0000 7f247cb7d740 1 -- start start 2024-03-21T10:46:59.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.134+0000 7f247cb7d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568c07120f0 0x5568c071a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.134+0000 7f247cb7d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568c0712dd0 0x5568c071ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.134+0000 7f247cb7d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568c071b460 0x5568c071c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.134+0000 7f247cb7d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24741b2f10 con 0x5568c071b460 2024-03-21T10:46:59.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.134+0000 7f247cb7d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24741b0a10 con 0x5568c0712dd0 2024-03-21T10:46:59.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.134+0000 7f247cb7d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24741aeab0 con 0x5568c07120f0 2024-03-21T10:46:59.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.135+0000 7f2472a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568c0712dd0 0x5568c071ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.135+0000 7f2472a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568c0712dd0 0x5568c071ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42380/0 (socket says 172.21.15.31:42380) 2024-03-21T10:46:59.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.135+0000 7f2472a00640 1 -- 172.21.15.31:0/285770478 learned_addr learned my addr 172.21.15.31:0/285770478 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:59.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.135+0000 7f246be00640 1 --2- 172.21.15.31:0/285770478 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568c07120f0 0x5568c071a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.135+0000 7f246be00640 1 --2- 172.21.15.31:0/285770478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568c071b460 0x5568c071c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.135+0000 7f2472a00640 1 -- 172.21.15.31:0/285770478 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568c07120f0 msgr2=0x5568c071a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:59.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.135+0000 7f2472a00640 1 --2- 172.21.15.31:0/285770478 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568c07120f0 0x5568c071a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.135+0000 7f2472a00640 1 -- 172.21.15.31:0/285770478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568c071b460 msgr2=0x5568c071c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:59.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.135+0000 7f2472a00640 1 --2- 172.21.15.31:0/285770478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568c071b460 0x5568c071c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.135+0000 7f2472a00640 1 -- 172.21.15.31:0/285770478 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5568c034a0f0 con 0x5568c0712dd0 2024-03-21T10:46:59.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.135+0000 7f2472a00640 1 --2- 172.21.15.31:0/285770478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568c0712dd0 0x5568c071ab30 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f2464003d20 tx=0x7f2464003f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ae9a3a1d6459016 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:59.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.136+0000 7f2463400640 1 -- 172.21.15.31:0/285770478 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246403d070 con 0x5568c0712dd0 2024-03-21T10:46:59.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.136+0000 7f2463400640 1 -- 172.21.15.31:0/285770478 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f246400fe00 con 0x5568c0712dd0 2024-03-21T10:46:59.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.136+0000 7f2463400640 1 -- 172.21.15.31:0/285770478 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2464028040 con 0x5568c0712dd0 2024-03-21T10:46:59.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.136+0000 7f247cb7d740 1 -- 172.21.15.31:0/285770478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568c0712dd0 msgr2=0x5568c071ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:59.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.136+0000 7f247cb7d740 1 --2- 172.21.15.31:0/285770478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568c0712dd0 0x5568c071ab30 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f2464003d20 tx=0x7f2464003f60 comp rx=0 tx=0).stop 2024-03-21T10:46:59.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.136+0000 7f247cb7d740 1 -- 172.21.15.31:0/285770478 shutdown_connections 2024-03-21T10:46:59.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.136+0000 7f247cb7d740 1 --2- 172.21.15.31:0/285770478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568c071b460 0x5568c071c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.136+0000 7f247cb7d740 1 --2- 172.21.15.31:0/285770478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568c0712dd0 0x5568c071ab30 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.136+0000 7f247cb7d740 1 --2- 172.21.15.31:0/285770478 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568c07120f0 0x5568c071a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.136+0000 7f247cb7d740 1 -- 172.21.15.31:0/285770478 >> 172.21.15.31:0/285770478 conn(0x5568c06d9200 msgr2=0x5568c0716790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:59.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.137+0000 7f247cb7d740 1 -- 172.21.15.31:0/285770478 shutdown_connections 2024-03-21T10:46:59.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.137+0000 7f247cb7d740 1 -- 172.21.15.31:0/285770478 wait complete. 2024-03-21T10:46:59.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.137+0000 7f247cb7d740 1 -- start start 2024-03-21T10:46:59.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.137+0000 7f247cb7d740 1 Processor -- start 2024-03-21T10:46:59.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.137+0000 7f247cb7d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568c07120f0 0x5568c0714d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.138+0000 7f247cb7d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568c0712dd0 0x5568c0715260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.138+0000 7f247cb7d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568c071b460 0x5568c078e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.138+0000 7f247cb7d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24741af0b0 con 0x5568c071b460 2024-03-21T10:46:59.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.138+0000 7f247cb7d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24741b03b0 con 0x5568c0712dd0 2024-03-21T10:46:59.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.138+0000 7f247cb7d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24741af6f0 con 0x5568c07120f0 2024-03-21T10:46:59.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.138+0000 7f247cb7d740 10 client.0 _ll_register_callbacks cb 0x7f2464009b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:46:59.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.138+0000 7f2472a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568c0712dd0 0x5568c0715260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.138+0000 7f2472a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568c0712dd0 0x5568c0715260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42396/0 (socket says 172.21.15.31:42396) 2024-03-21T10:46:59.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.138+0000 7f2472a00640 1 -- 172.21.15.31:0/2232392379 learned_addr learned my addr 172.21.15.31:0/2232392379 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:59.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.138+0000 7f246be00640 1 --2- 172.21.15.31:0/2232392379 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568c07120f0 0x5568c0714d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.138+0000 7f246be00640 1 --2- 172.21.15.31:0/2232392379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568c071b460 0x5568c078e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.138+0000 7f2472a00640 1 -- 172.21.15.31:0/2232392379 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568c07120f0 msgr2=0x5568c0714d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:59.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.138+0000 7f2472a00640 1 --2- 172.21.15.31:0/2232392379 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568c07120f0 0x5568c0714d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.138+0000 7f2472a00640 1 -- 172.21.15.31:0/2232392379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568c071b460 msgr2=0x5568c078e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:59.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.138+0000 7f2472a00640 1 --2- 172.21.15.31:0/2232392379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568c071b460 0x5568c078e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.138+0000 7f2472a00640 1 -- 172.21.15.31:0/2232392379 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5568c078e630 con 0x5568c0712dd0 2024-03-21T10:46:59.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.138+0000 7f246be00640 1 --2- 172.21.15.31:0/2232392379 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568c07120f0 0x5568c0714d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:46:59.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.138+0000 7f2472a00640 1 --2- 172.21.15.31:0/2232392379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568c0712dd0 0x5568c0715260 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f2464003d20 tx=0x7f24640117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7432b3c188a5b9de server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:59.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.139+0000 7f2460c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:46:59.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.139+0000 7f2460c00640 1 -- 172.21.15.31:0/2232392379 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246403d070 con 0x5568c0712dd0 2024-03-21T10:46:59.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.139+0000 7f247cb7d740 1 -- 172.21.15.31:0/2232392379 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f246400fe00 con 0x5568c0712dd0 2024-03-21T10:46:59.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.139+0000 7f2460c00640 1 -- 172.21.15.31:0/2232392379 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f2464004460 con 0x5568c0712dd0 2024-03-21T10:46:59.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.139+0000 7f2460c00640 1 -- 172.21.15.31:0/2232392379 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2464048050 con 0x5568c0712dd0 2024-03-21T10:46:59.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.139+0000 7f247cb7d740 10 client.27656 fetch_fsmap learned FSMap version 28 2024-03-21T10:46:59.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.139+0000 7f247cb7d740 1 -- 172.21.15.31:0/2232392379 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5568c078f940 con 0x5568c0712dd0 2024-03-21T10:46:59.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.139+0000 7f2460c00640 1 -- 172.21.15.31:0/2232392379 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f246400fe00 con 0x5568c0712dd0 2024-03-21T10:46:59.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.140+0000 7f2460c00640 1 -- 172.21.15.31:0/2232392379 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f2464012830 con 0x5568c0712dd0 2024-03-21T10:46:59.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.140+0000 7f247cb7d740 10 client.27656 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:46:59.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.140+0000 7f247cb7d740 10 client.27656 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:46:59.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.140+0000 7f247cb7d740 20 client.27656 populate_metadata read hostname 'smithi031' 2024-03-21T10:46:59.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.140+0000 7f247cb7d740 1 --2- 172.21.15.31:0/2232392379 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5568c078ff60 0x5568c0792380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.140+0000 7f247cb7d740 4 client.27656 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:46:59.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.140+0000 7f247cb7d740 1 -- 172.21.15.31:0/2232392379 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5568c067dae0 con 0x5568c078ff60 2024-03-21T10:46:59.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.140+0000 7f246be00640 1 --2- 172.21.15.31:0/2232392379 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5568c078ff60 0x5568c0792380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.141+0000 7f246be00640 1 --2- 172.21.15.31:0/2232392379 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5568c078ff60 0x5568c0792380 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ae8377d7b869636 server_cookie=add676827ea5c8ba in_seq=0 out_seq=0 2024-03-21T10:46:59.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.141+0000 7f2460c00640 10 client.27656 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:59.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.141+0000 7f2460c00640 1 -- 172.21.15.31:0/2232392379 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5568c067dae0 con 0x5568c078ff60 2024-03-21T10:46:59.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.141+0000 7f2460c00640 10 client.27656 handle_command_reply: tid=0 2024-03-21T10:46:59.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.141+0000 7f247cb7d740 1 client.27656 shutdown 2024-03-21T10:46:59.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.141+0000 7f247cb7d740 1 -- 172.21.15.31:0/2232392379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568c0712dd0 msgr2=0x5568c0715260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:59.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.141+0000 7f247cb7d740 1 --2- 172.21.15.31:0/2232392379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568c0712dd0 0x5568c0715260 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f2464003d20 tx=0x7f24640117b0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.141+0000 7f247cb7d740 1 -- 172.21.15.31:0/2232392379 shutdown_connections 2024-03-21T10:46:59.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.141+0000 7f247cb7d740 1 --2- 172.21.15.31:0/2232392379 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5568c078ff60 0x5568c0792380 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.141+0000 7f247cb7d740 1 --2- 172.21.15.31:0/2232392379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568c071b460 0x5568c078e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.141+0000 7f247cb7d740 1 --2- 172.21.15.31:0/2232392379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568c0712dd0 0x5568c0715260 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.141+0000 7f247cb7d740 1 --2- 172.21.15.31:0/2232392379 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5568c07120f0 0x5568c0714d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.141+0000 7f247cb7d740 1 -- 172.21.15.31:0/2232392379 >> 172.21.15.31:0/2232392379 conn(0x5568c06d9200 msgr2=0x5568c070b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:59.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.141+0000 7f2460c00640 0 client.27656 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:46:59.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.142+0000 7f247cb7d740 1 -- 172.21.15.31:0/2232392379 shutdown_connections 2024-03-21T10:46:59.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.142+0000 7f247cb7d740 1 -- 172.21.15.31:0/2232392379 wait complete. 2024-03-21T10:46:59.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.142+0000 7f247cb7d740 20 client.27656 trim_cache size 0 max 16384 2024-03-21T10:46:59.144 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:46:59.144 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-03-21T10:46:59.144 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-03-21T10:46:59.144 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:46:59.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.142+0000 7f2473e00640 1 -- 172.21.15.31:0/2737264276 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24580a3090 msgr2=0x7f24580a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:59.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.142+0000 7f2473e00640 1 --2- 172.21.15.31:0/2737264276 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24580a3090 0x7f24580a5550 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f246c004af0 tx=0x7f246c02bf80 comp rx=0 tx=0).stop 2024-03-21T10:46:59.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.142+0000 7f2473e00640 1 -- 172.21.15.31:0/2737264276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f247410ba50 msgr2=0x7f247419e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:59.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.142+0000 7f2473e00640 1 --2- 172.21.15.31:0/2737264276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f247410ba50 0x7f247419e7a0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f246c036040 tx=0x7f246c004270 comp rx=0 tx=0).stop 2024-03-21T10:46:59.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.143+0000 7f2473e00640 1 -- 172.21.15.31:0/2737264276 shutdown_connections 2024-03-21T10:46:59.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.143+0000 7f2473e00640 1 --2- 172.21.15.31:0/2737264276 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24580a3090 0x7f24580a5550 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.143+0000 7f2473e00640 1 --2- 172.21.15.31:0/2737264276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f247410ba50 0x7f247419e7a0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.143+0000 7f2473e00640 1 --2- 172.21.15.31:0/2737264276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2474102f50 0x7f247419a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.143+0000 7f2473e00640 1 --2- 172.21.15.31:0/2737264276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2474102630 0x7f2474199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.143+0000 7f2473e00640 1 -- 172.21.15.31:0/2737264276 >> 172.21.15.31:0/2737264276 conn(0x7f2474077130 msgr2=0x7f24740fc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:59.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.143+0000 7f2473e00640 1 -- 172.21.15.31:0/2737264276 shutdown_connections 2024-03-21T10:46:59.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.143+0000 7f2473e00640 1 -- 172.21.15.31:0/2737264276 wait complete. 2024-03-21T10:46:59.159 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-21T10:46:59.159 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:46:59.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:58 smithi031 ceph-mon[31673]: pgmap v259: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 7 op/s 2024-03-21T10:46:59.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:58 smithi070 ceph-mon[36330]: pgmap v259: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 7 op/s 2024-03-21T10:46:59.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.369+0000 7faaffe00640 1 -- 172.21.15.31:0/929101837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab000764f0 msgr2=0x7fab00111ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:59.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.369+0000 7faaffe00640 1 --2- 172.21.15.31:0/929101837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab000764f0 0x7fab00111ea0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7faae8009930 tx=0x7faae802bc20 comp rx=0 tx=0).stop 2024-03-21T10:46:59.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.369+0000 7faaffe00640 1 -- 172.21.15.31:0/929101837 shutdown_connections 2024-03-21T10:46:59.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.369+0000 7faaffe00640 1 --2- 172.21.15.31:0/929101837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab001123e0 0x7fab001147d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.369+0000 7faaffe00640 1 --2- 172.21.15.31:0/929101837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab000764f0 0x7fab00111ea0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.369+0000 7faaffe00640 1 --2- 172.21.15.31:0/929101837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab00075bd0 0x7fab00075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.369+0000 7faaffe00640 1 -- 172.21.15.31:0/929101837 >> 172.21.15.31:0/929101837 conn(0x7fab000ff210 msgr2=0x7fab00101630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:59.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.369+0000 7faaffe00640 1 -- 172.21.15.31:0/929101837 shutdown_connections 2024-03-21T10:46:59.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.369+0000 7faaffe00640 1 -- 172.21.15.31:0/929101837 wait complete. 2024-03-21T10:46:59.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.370+0000 7faaffe00640 1 Processor -- start 2024-03-21T10:46:59.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.370+0000 7faaffe00640 1 -- start start 2024-03-21T10:46:59.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.371+0000 7faaffe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab00075bd0 0x7fab0019e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.371+0000 7faaffe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab000764f0 0x7fab0019e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.371+0000 7faaffe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab001123e0 0x7fab001a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.371+0000 7faaffe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab00117f90 con 0x7fab001123e0 2024-03-21T10:46:59.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.371+0000 7faaffe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab00117e10 con 0x7fab00075bd0 2024-03-21T10:46:59.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.371+0000 7faaffe00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab00118110 con 0x7fab000764f0 2024-03-21T10:46:59.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.371+0000 7faafea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab00075bd0 0x7fab0019e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.371+0000 7faafea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab00075bd0 0x7fab0019e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42412/0 (socket says 172.21.15.31:42412) 2024-03-21T10:46:59.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.371+0000 7faafea00640 1 -- 172.21.15.31:0/3455612563 learned_addr learned my addr 172.21.15.31:0/3455612563 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:59.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.371+0000 7faaf7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab000764f0 0x7fab0019e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.371+0000 7faaff400640 1 --2- 172.21.15.31:0/3455612563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab001123e0 0x7fab001a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.371+0000 7faafea00640 1 -- 172.21.15.31:0/3455612563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab000764f0 msgr2=0x7fab0019e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:59.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.371+0000 7faafea00640 1 --2- 172.21.15.31:0/3455612563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab000764f0 0x7fab0019e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.371+0000 7faafea00640 1 -- 172.21.15.31:0/3455612563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab001123e0 msgr2=0x7fab001a2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:59.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.371+0000 7faafea00640 1 --2- 172.21.15.31:0/3455612563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab001123e0 0x7fab001a2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.371+0000 7faafea00640 1 -- 172.21.15.31:0/3455612563 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab001a3480 con 0x7fab00075bd0 2024-03-21T10:46:59.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.372+0000 7faafea00640 1 --2- 172.21.15.31:0/3455612563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab00075bd0 0x7fab0019e490 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7faaf000ea60 tx=0x7faaf000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:59.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.372+0000 7faaf7400640 1 -- 172.21.15.31:0/3455612563 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf000cde0 con 0x7fab00075bd0 2024-03-21T10:46:59.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.372+0000 7faaf7400640 1 -- 172.21.15.31:0/3455612563 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faaf00035f0 con 0x7fab00075bd0 2024-03-21T10:46:59.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.372+0000 7faaf7400640 1 -- 172.21.15.31:0/3455612563 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf0010670 con 0x7fab00075bd0 2024-03-21T10:46:59.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.372+0000 7faaffe00640 1 -- 172.21.15.31:0/3455612563 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab001a3770 con 0x7fab00075bd0 2024-03-21T10:46:59.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.372+0000 7faaffe00640 1 -- 172.21.15.31:0/3455612563 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab001ab000 con 0x7fab00075bd0 2024-03-21T10:46:59.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.373+0000 7faaffe00640 1 -- 172.21.15.31:0/3455612563 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab0004f470 con 0x7fab00075bd0 2024-03-21T10:46:59.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.375+0000 7faaf7400640 1 -- 172.21.15.31:0/3455612563 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7faaf00037a0 con 0x7fab00075bd0 2024-03-21T10:46:59.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.377+0000 7faaf7400640 1 --2- 172.21.15.31:0/3455612563 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7faae40a3100 0x7faae40a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.377+0000 7faaf7400640 1 -- 172.21.15.31:0/3455612563 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7faaf00cff00 con 0x7fab00075bd0 2024-03-21T10:46:59.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.377+0000 7faaf7e00640 1 --2- 172.21.15.31:0/3455612563 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7faae40a3100 0x7faae40a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.378+0000 7faaf7e00640 1 --2- 172.21.15.31:0/3455612563 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7faae40a3100 0x7faae40a55c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7faae8009900 tx=0x7faae8005a20 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:59.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.381+0000 7faaf7400640 1 -- 172.21.15.31:0/3455612563 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faaf009c5f0 con 0x7fab00075bd0 2024-03-21T10:46:59.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:58 smithi040 ceph-mon[36154]: pgmap v259: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 7 op/s 2024-03-21T10:46:59.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.631+0000 7faaffe00640 1 -- 172.21.15.31:0/3455612563 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab0019f7e0 con 0x7fab00075bd0 2024-03-21T10:46:59.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.633+0000 7faaf7400640 1 -- 172.21.15.31:0/3455612563 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11387 (secure 0 0 0) 0x7faaf009bef0 con 0x7fab00075bd0 2024-03-21T10:46:59.630 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:46:59.630 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:46:51.185295+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24547,24596]},"id":1}]} 2024-03-21T10:46:59.630 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-03-21T10:46:59.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.636+0000 7faaffe00640 1 -- 172.21.15.31:0/3455612563 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7faae40a3100 msgr2=0x7faae40a55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:59.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.636+0000 7faaffe00640 1 --2- 172.21.15.31:0/3455612563 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7faae40a3100 0x7faae40a55c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7faae8009900 tx=0x7faae8005a20 comp rx=0 tx=0).stop 2024-03-21T10:46:59.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.636+0000 7faaffe00640 1 -- 172.21.15.31:0/3455612563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab00075bd0 msgr2=0x7fab0019e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:59.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.636+0000 7faaffe00640 1 --2- 172.21.15.31:0/3455612563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab00075bd0 0x7fab0019e490 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7faaf000ea60 tx=0x7faaf000ef20 comp rx=0 tx=0).stop 2024-03-21T10:46:59.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.636+0000 7faaffe00640 1 -- 172.21.15.31:0/3455612563 shutdown_connections 2024-03-21T10:46:59.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.636+0000 7faaffe00640 1 --2- 172.21.15.31:0/3455612563 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7faae40a3100 0x7faae40a55c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.636+0000 7faaffe00640 1 --2- 172.21.15.31:0/3455612563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab001123e0 0x7fab001a2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.636+0000 7faaffe00640 1 --2- 172.21.15.31:0/3455612563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab000764f0 0x7fab0019e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.636+0000 7faaffe00640 1 --2- 172.21.15.31:0/3455612563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab00075bd0 0x7fab0019e490 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.636+0000 7faaffe00640 1 -- 172.21.15.31:0/3455612563 >> 172.21.15.31:0/3455612563 conn(0x7fab000ff210 msgr2=0x7fab00101600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:59.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.636+0000 7faaffe00640 1 -- 172.21.15.31:0/3455612563 shutdown_connections 2024-03-21T10:46:59.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.636+0000 7faaffe00640 1 -- 172.21.15.31:0/3455612563 wait complete. 2024-03-21T10:46:59.654 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:46:59.654 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-21T10:46:59.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.910+0000 7fe37d600640 1 -- 172.21.15.31:0/1675747266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe378102630 msgr2=0x7fe378102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:59.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.911+0000 7fe37d600640 1 --2- 172.21.15.31:0/1675747266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe378102630 0x7fe378102a10 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe368009900 tx=0x7fe36802bcd0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.911+0000 7fe37d600640 1 -- 172.21.15.31:0/1675747266 shutdown_connections 2024-03-21T10:46:59.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.911+0000 7fe37d600640 1 --2- 172.21.15.31:0/1675747266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe37810ba50 0x7fe37810ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.911+0000 7fe37d600640 1 --2- 172.21.15.31:0/1675747266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe378102f50 0x7fe37810b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.911+0000 7fe37d600640 1 --2- 172.21.15.31:0/1675747266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe378102630 0x7fe378102a10 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.911+0000 7fe37d600640 1 -- 172.21.15.31:0/1675747266 >> 172.21.15.31:0/1675747266 conn(0x7fe378077130 msgr2=0x7fe3780fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:59.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.911+0000 7fe37d600640 1 -- 172.21.15.31:0/1675747266 shutdown_connections 2024-03-21T10:46:59.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.911+0000 7fe37d600640 1 -- 172.21.15.31:0/1675747266 wait complete. 2024-03-21T10:46:59.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.912+0000 7fe37d600640 1 Processor -- start 2024-03-21T10:46:59.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.912+0000 7fe37d600640 1 -- start start 2024-03-21T10:46:59.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.912+0000 7fe37d600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe378102630 0x7fe37819a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.912+0000 7fe37d600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe378102f50 0x7fe37819a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.912+0000 7fe37d600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe37810ba50 0x7fe37819e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.912+0000 7fe37d600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe378100ab0 con 0x7fe378102630 2024-03-21T10:46:59.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.912+0000 7fe37d600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe378100930 con 0x7fe378102f50 2024-03-21T10:46:59.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.912+0000 7fe37d600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe378100c30 con 0x7fe37810ba50 2024-03-21T10:46:59.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.913+0000 7fe377e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe378102630 0x7fe37819a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.913+0000 7fe37cc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe37810ba50 0x7fe37819e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.913+0000 7fe377400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe378102f50 0x7fe37819a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.913+0000 7fe377e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe378102630 0x7fe37819a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42050/0 (socket says 172.21.15.31:42050) 2024-03-21T10:46:59.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.913+0000 7fe37cc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe37810ba50 0x7fe37819e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53756/0 (socket says 172.21.15.31:53756) 2024-03-21T10:46:59.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.913+0000 7fe377400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe378102f50 0x7fe37819a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42430/0 (socket says 172.21.15.31:42430) 2024-03-21T10:46:59.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.913+0000 7fe377e00640 1 -- 172.21.15.31:0/2860803433 learned_addr learned my addr 172.21.15.31:0/2860803433 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:59.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.913+0000 7fe377400640 1 -- 172.21.15.31:0/2860803433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe37810ba50 msgr2=0x7fe37819e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:59.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.913+0000 7fe377400640 1 --2- 172.21.15.31:0/2860803433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe37810ba50 0x7fe37819e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.913+0000 7fe377400640 1 -- 172.21.15.31:0/2860803433 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe378102630 msgr2=0x7fe37819a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:59.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.913+0000 7fe377400640 1 --2- 172.21.15.31:0/2860803433 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe378102630 0x7fe37819a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.913+0000 7fe377400640 1 -- 172.21.15.31:0/2860803433 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe37819efb0 con 0x7fe378102f50 2024-03-21T10:46:59.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.913+0000 7fe377400640 1 --2- 172.21.15.31:0/2860803433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe378102f50 0x7fe37819a7b0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fe36400e9f0 tx=0x7fe36400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:59.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.914+0000 7fe374c00640 1 -- 172.21.15.31:0/2860803433 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36400ccc0 con 0x7fe378102f50 2024-03-21T10:46:59.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.914+0000 7fe37d600640 1 -- 172.21.15.31:0/2860803433 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe37819f2a0 con 0x7fe378102f50 2024-03-21T10:46:59.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.914+0000 7fe374c00640 1 -- 172.21.15.31:0/2860803433 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe364010910 con 0x7fe378102f50 2024-03-21T10:46:59.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.914+0000 7fe374c00640 1 -- 172.21.15.31:0/2860803433 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe364018960 con 0x7fe378102f50 2024-03-21T10:46:59.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.914+0000 7fe37d600640 1 -- 172.21.15.31:0/2860803433 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3781a6af0 con 0x7fe378102f50 2024-03-21T10:46:59.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.918+0000 7fe381965740 1 Processor -- start 2024-03-21T10:46:59.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.918+0000 7fe381965740 1 -- start start 2024-03-21T10:46:59.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.919+0000 7fe381965740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e362b68b0 0x561e362b6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.919+0000 7fe381965740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e362b7260 0x561e362bf810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.919+0000 7fe381965740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e362bfde0 0x561e362c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.919+0000 7fe381965740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e3621a2c0 con 0x561e362b68b0 2024-03-21T10:46:59.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.919+0000 7fe381965740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e361c2c90 con 0x561e362b7260 2024-03-21T10:46:59.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.919+0000 7fe381965740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e36216360 con 0x561e362bfde0 2024-03-21T10:46:59.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.919+0000 7fe37cc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e362b68b0 0x561e362b6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.919+0000 7fe377e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e362b7260 0x561e362bf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.919+0000 7fe374c00640 1 -- 172.21.15.31:0/2860803433 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fe364010ac0 con 0x7fe378102f50 2024-03-21T10:46:59.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.919+0000 7fe377e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e362b7260 0x561e362bf810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42446/0 (socket says 172.21.15.31:42446) 2024-03-21T10:46:59.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.919+0000 7fe377e00640 1 -- 172.21.15.31:0/591718525 learned_addr learned my addr 172.21.15.31:0/591718525 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:59.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.920+0000 7fe377400640 1 --2- 172.21.15.31:0/591718525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e362bfde0 0x561e362c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.920+0000 7fe377e00640 1 -- 172.21.15.31:0/591718525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e362bfde0 msgr2=0x561e362c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:59.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.920+0000 7fe377e00640 1 --2- 172.21.15.31:0/591718525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e362bfde0 0x561e362c2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.920+0000 7fe377e00640 1 -- 172.21.15.31:0/591718525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e362b68b0 msgr2=0x561e362b6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:59.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.920+0000 7fe377e00640 1 --2- 172.21.15.31:0/591718525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e362b68b0 0x561e362b6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.920+0000 7fe377e00640 1 -- 172.21.15.31:0/591718525 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e361e7080 con 0x561e362b7260 2024-03-21T10:46:59.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.920+0000 7fe37cc00640 1 --2- 172.21.15.31:0/591718525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e362b68b0 0x561e362b6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:46:59.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.920+0000 7fe377e00640 1 --2- 172.21.15.31:0/591718525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e362b7260 0x561e362bf810 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fe368005be0 tx=0x7fe368009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b856e579c08f943 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:59.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.920+0000 7fe36d200640 1 -- 172.21.15.31:0/591718525 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe368043070 con 0x561e362b7260 2024-03-21T10:46:59.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.920+0000 7fe36d200640 1 -- 172.21.15.31:0/591718525 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe36802c920 con 0x561e362b7260 2024-03-21T10:46:59.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.920+0000 7fe36d200640 1 -- 172.21.15.31:0/591718525 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe368004bd0 con 0x561e362b7260 2024-03-21T10:46:59.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.921+0000 7fe381965740 1 -- 172.21.15.31:0/591718525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e362b7260 msgr2=0x561e362bf810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:59.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.921+0000 7fe381965740 1 --2- 172.21.15.31:0/591718525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e362b7260 0x561e362bf810 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fe368005be0 tx=0x7fe368009560 comp rx=0 tx=0).stop 2024-03-21T10:46:59.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.921+0000 7fe381965740 1 -- 172.21.15.31:0/591718525 shutdown_connections 2024-03-21T10:46:59.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.921+0000 7fe381965740 1 --2- 172.21.15.31:0/591718525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e362bfde0 0x561e362c2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.921+0000 7fe381965740 1 --2- 172.21.15.31:0/591718525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e362b7260 0x561e362bf810 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.921+0000 7fe381965740 1 --2- 172.21.15.31:0/591718525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e362b68b0 0x561e362b6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.921+0000 7fe381965740 1 -- 172.21.15.31:0/591718525 >> 172.21.15.31:0/591718525 conn(0x561e362b0f80 msgr2=0x561e362b1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:59.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.921+0000 7fe381965740 1 -- 172.21.15.31:0/591718525 shutdown_connections 2024-03-21T10:46:59.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.921+0000 7fe381965740 1 -- 172.21.15.31:0/591718525 wait complete. 2024-03-21T10:46:59.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.921+0000 7fe374c00640 1 --2- 172.21.15.31:0/2860803433 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe35c0a3020 0x7fe35c0a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.922+0000 7fe374c00640 1 -- 172.21.15.31:0/2860803433 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fe3640cfec0 con 0x7fe378102f50 2024-03-21T10:46:59.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.922+0000 7fe381965740 1 -- start start 2024-03-21T10:46:59.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.922+0000 7fe377e00640 1 --2- 172.21.15.31:0/2860803433 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe35c0a3020 0x7fe35c0a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.922+0000 7fe381965740 1 Processor -- start 2024-03-21T10:46:59.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.923+0000 7fe381965740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e362b68b0 0x561e362bf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.923+0000 7fe377e00640 1 --2- 172.21.15.31:0/2860803433 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe35c0a3020 0x7fe35c0a54e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe368006010 tx=0x7fe368000bc0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:59.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.923+0000 7fe381965740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e362b7260 0x561e362bcab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.923+0000 7fe381965740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e362bfde0 0x561e362bd050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.923+0000 7fe381965740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e362c2cf0 con 0x561e362b68b0 2024-03-21T10:46:59.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.923+0000 7fe381965740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e361d60d0 con 0x561e362b7260 2024-03-21T10:46:59.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.923+0000 7fe381965740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e362c2e70 con 0x561e362bfde0 2024-03-21T10:46:59.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.923+0000 7fe381965740 10 client.0 _ll_register_callbacks cb 0x561e361dae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:46:59.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.923+0000 7fe377400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e362b7260 0x561e362bcab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.923+0000 7fe377e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e362b68b0 0x561e362bf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.923+0000 7fe377400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e362b7260 0x561e362bcab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42454/0 (socket says 172.21.15.31:42454) 2024-03-21T10:46:59.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.923+0000 7fe37cc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e362bfde0 0x561e362bd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.923+0000 7fe377400640 1 -- 172.21.15.31:0/2047342649 learned_addr learned my addr 172.21.15.31:0/2047342649 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:59.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.923+0000 7fe377400640 1 -- 172.21.15.31:0/2047342649 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e362bfde0 msgr2=0x561e362bd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:59.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.923+0000 7fe377400640 1 --2- 172.21.15.31:0/2047342649 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e362bfde0 0x561e362bd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.923+0000 7fe377400640 1 -- 172.21.15.31:0/2047342649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e362b68b0 msgr2=0x561e362bf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:59.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.923+0000 7fe377400640 1 --2- 172.21.15.31:0/2047342649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e362b68b0 0x561e362bf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.923+0000 7fe377400640 1 -- 172.21.15.31:0/2047342649 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561e35f2e3f0 con 0x561e362b7260 2024-03-21T10:46:59.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.924+0000 7fe377400640 1 --2- 172.21.15.31:0/2047342649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e362b7260 0x561e362bcab0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fe3640a4390 tx=0x7fe36409be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=865bf3526419a110 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:59.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.924+0000 7fe361600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:46:59.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.924+0000 7fe361600640 1 -- 172.21.15.31:0/2047342649 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3640cd6a0 con 0x561e362b7260 2024-03-21T10:46:59.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.924+0000 7fe381965740 1 -- 172.21.15.31:0/2047342649 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe36802c920 con 0x561e362b7260 2024-03-21T10:46:59.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.924+0000 7fe361600640 1 -- 172.21.15.31:0/2047342649 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fe36409b5b0 con 0x561e362b7260 2024-03-21T10:46:59.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.924+0000 7fe361600640 1 -- 172.21.15.31:0/2047342649 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3640bc0d0 con 0x561e362b7260 2024-03-21T10:46:59.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.924+0000 7fe381965740 10 client.27674 fetch_fsmap learned FSMap version 28 2024-03-21T10:46:59.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.924+0000 7fe381965740 1 -- 172.21.15.31:0/2047342649 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561e35ff87d0 con 0x561e362b7260 2024-03-21T10:46:59.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.924+0000 7fe361600640 1 -- 172.21.15.31:0/2047342649 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe36802c920 con 0x561e362b7260 2024-03-21T10:46:59.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.924+0000 7fe361600640 1 -- 172.21.15.31:0/2047342649 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe364097cb0 con 0x561e362b7260 2024-03-21T10:46:59.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.925+0000 7fe381965740 10 client.27674 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:46:59.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.925+0000 7fe381965740 10 client.27674 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:46:59.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.925+0000 7fe381965740 20 client.27674 populate_metadata read hostname 'smithi031' 2024-03-21T10:46:59.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.925+0000 7fe381965740 1 --2- 172.21.15.31:0/2047342649 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x561e362cfc30 0x561e362d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.925+0000 7fe381965740 4 client.27674 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:46:59.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.925+0000 7fe381965740 1 -- 172.21.15.31:0/2047342649 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561e36227800 con 0x561e362cfc30 2024-03-21T10:46:59.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.925+0000 7fe377e00640 1 --2- 172.21.15.31:0/2047342649 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x561e362cfc30 0x561e362d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.925+0000 7fe377e00640 1 --2- 172.21.15.31:0/2047342649 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x561e362cfc30 0x561e362d2020 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=18caf2d504ed8ba7 server_cookie=b34c901681dfef1c in_seq=0 out_seq=0 2024-03-21T10:46:59.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.925+0000 7fe361600640 10 client.27674 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:46:59.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.926+0000 7fe361600640 1 -- 172.21.15.31:0/2047342649 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561e36227800 con 0x561e362cfc30 2024-03-21T10:46:59.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.926+0000 7fe361600640 10 client.27674 handle_command_reply: tid=0 2024-03-21T10:46:59.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.926+0000 7fe381965740 1 client.27674 shutdown 2024-03-21T10:46:59.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.926+0000 7fe381965740 1 -- 172.21.15.31:0/2047342649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e362b7260 msgr2=0x561e362bcab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:59.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.926+0000 7fe381965740 1 --2- 172.21.15.31:0/2047342649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e362b7260 0x561e362bcab0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fe3640a4390 tx=0x7fe36409be20 comp rx=0 tx=0).stop 2024-03-21T10:46:59.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.926+0000 7fe381965740 1 -- 172.21.15.31:0/2047342649 shutdown_connections 2024-03-21T10:46:59.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.926+0000 7fe381965740 1 --2- 172.21.15.31:0/2047342649 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x561e362cfc30 0x561e362d2020 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.926+0000 7fe381965740 1 --2- 172.21.15.31:0/2047342649 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e362bfde0 0x561e362bd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.926+0000 7fe381965740 1 --2- 172.21.15.31:0/2047342649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e362b7260 0x561e362bcab0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.926+0000 7fe381965740 1 --2- 172.21.15.31:0/2047342649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e362b68b0 0x561e362bf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.926+0000 7fe381965740 1 -- 172.21.15.31:0/2047342649 >> 172.21.15.31:0/2047342649 conn(0x561e362b0f80 msgr2=0x561e362b2cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:59.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.926+0000 7fe361600640 0 client.27674 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:46:59.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.927+0000 7fe381965740 1 -- 172.21.15.31:0/2047342649 shutdown_connections 2024-03-21T10:46:59.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.927+0000 7fe381965740 1 -- 172.21.15.31:0/2047342649 wait complete. 2024-03-21T10:46:59.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.927+0000 7fe381965740 20 client.27674 trim_cache size 0 max 16384 2024-03-21T10:46:59.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.940+0000 7fe381965740 1 Processor -- start 2024-03-21T10:46:59.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.941+0000 7fe381965740 1 -- start start 2024-03-21T10:46:59.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.941+0000 7fe381965740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e362b68b0 0x561e362bbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.941+0000 7fe381965740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e362b7260 0x561e362affe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.941+0000 7fe381965740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e362bfde0 0x561e362b9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.941+0000 7fe381965740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3781b2e50 con 0x561e362bfde0 2024-03-21T10:46:59.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.941+0000 7fe381965740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3781b0950 con 0x561e362b7260 2024-03-21T10:46:59.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.941+0000 7fe381965740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3781aea10 con 0x561e362b68b0 2024-03-21T10:46:59.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.941+0000 7fe377e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e362b68b0 0x561e362bbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.941+0000 7fe377e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e362b68b0 0x561e362bbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53788/0 (socket says 172.21.15.31:53788) 2024-03-21T10:46:59.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.941+0000 7fe377e00640 1 -- 172.21.15.31:0/1448395819 learned_addr learned my addr 172.21.15.31:0/1448395819 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:59.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.941+0000 7fe37cc00640 1 --2- 172.21.15.31:0/1448395819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e362bfde0 0x561e362b9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.941+0000 7fe377400640 1 --2- 172.21.15.31:0/1448395819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e362b7260 0x561e362affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.941+0000 7fe377e00640 1 -- 172.21.15.31:0/1448395819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e362b7260 msgr2=0x561e362affe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:59.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.941+0000 7fe377e00640 1 --2- 172.21.15.31:0/1448395819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e362b7260 0x561e362affe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.941+0000 7fe377e00640 1 -- 172.21.15.31:0/1448395819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e362bfde0 msgr2=0x561e362b9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:59.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.941+0000 7fe377e00640 1 --2- 172.21.15.31:0/1448395819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e362bfde0 0x561e362b9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.942+0000 7fe377e00640 1 -- 172.21.15.31:0/1448395819 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e362b86a0 con 0x561e362b68b0 2024-03-21T10:46:59.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.942+0000 7fe377400640 1 --2- 172.21.15.31:0/1448395819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e362b7260 0x561e362affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:46:59.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.942+0000 7fe377e00640 1 --2- 172.21.15.31:0/1448395819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e362b68b0 0x561e362bbc30 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fe36802c8a0 tx=0x7fe36802c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1c63ec87571d510 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:59.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.942+0000 7fe363e00640 1 -- 172.21.15.31:0/1448395819 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36805d020 con 0x561e362b68b0 2024-03-21T10:46:59.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.942+0000 7fe363e00640 1 -- 172.21.15.31:0/1448395819 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe3680581b0 con 0x561e362b68b0 2024-03-21T10:46:59.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.942+0000 7fe363e00640 1 -- 172.21.15.31:0/1448395819 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe368043040 con 0x561e362b68b0 2024-03-21T10:46:59.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.943+0000 7fe381965740 1 -- 172.21.15.31:0/1448395819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e362b68b0 msgr2=0x561e362bbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:59.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.943+0000 7fe381965740 1 --2- 172.21.15.31:0/1448395819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e362b68b0 0x561e362bbc30 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fe36802c8a0 tx=0x7fe36802c830 comp rx=0 tx=0).stop 2024-03-21T10:46:59.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.943+0000 7fe381965740 1 -- 172.21.15.31:0/1448395819 shutdown_connections 2024-03-21T10:46:59.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.943+0000 7fe381965740 1 --2- 172.21.15.31:0/1448395819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e362bfde0 0x561e362b9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.943+0000 7fe381965740 1 --2- 172.21.15.31:0/1448395819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e362b7260 0x561e362affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.943+0000 7fe381965740 1 --2- 172.21.15.31:0/1448395819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e362b68b0 0x561e362bbc30 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.943+0000 7fe381965740 1 -- 172.21.15.31:0/1448395819 >> 172.21.15.31:0/1448395819 conn(0x561e362b0f80 msgr2=0x561e362af8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:59.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.943+0000 7fe381965740 1 -- 172.21.15.31:0/1448395819 shutdown_connections 2024-03-21T10:46:59.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.943+0000 7fe381965740 1 -- 172.21.15.31:0/1448395819 wait complete. 2024-03-21T10:46:59.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.944+0000 7fe381965740 1 -- start start 2024-03-21T10:46:59.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.944+0000 7fe381965740 1 Processor -- start 2024-03-21T10:46:59.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.944+0000 7fe381965740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e362b68b0 0x561e362c5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.944+0000 7fe381965740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e362b7260 0x561e362c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.944+0000 7fe381965740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e362bfde0 0x561e362c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.944+0000 7fe381965740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3781af010 con 0x561e362b7260 2024-03-21T10:46:59.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.944+0000 7fe381965740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3781b02f0 con 0x561e362bfde0 2024-03-21T10:46:59.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.944+0000 7fe381965740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3781af630 con 0x561e362b68b0 2024-03-21T10:46:59.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.944+0000 7fe381965740 10 client.0 _ll_register_callbacks cb 0x7fe36400ce60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:46:59.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.945+0000 7fe377e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e362b68b0 0x561e362c5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.945+0000 7fe377e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e362b68b0 0x561e362c5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53804/0 (socket says 172.21.15.31:53804) 2024-03-21T10:46:59.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.945+0000 7fe377400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e362b7260 0x561e362c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.945+0000 7fe377e00640 1 -- 172.21.15.31:0/4086574458 learned_addr learned my addr 172.21.15.31:0/4086574458 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:46:59.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.945+0000 7fe37cc00640 1 --2- 172.21.15.31:0/4086574458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e362bfde0 0x561e362c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.945+0000 7fe377e00640 1 -- 172.21.15.31:0/4086574458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e362bfde0 msgr2=0x561e362c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:59.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.945+0000 7fe377e00640 1 --2- 172.21.15.31:0/4086574458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e362bfde0 0x561e362c60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.945+0000 7fe377e00640 1 -- 172.21.15.31:0/4086574458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e362b7260 msgr2=0x561e362c5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:46:59.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.945+0000 7fe377e00640 1 --2- 172.21.15.31:0/4086574458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e362b7260 0x561e362c5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.945+0000 7fe377e00640 1 -- 172.21.15.31:0/4086574458 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561e3632a5f0 con 0x561e362b68b0 2024-03-21T10:46:59.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.945+0000 7fe377e00640 1 --2- 172.21.15.31:0/4086574458 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e362b68b0 0x561e362c5630 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fe368004a60 tx=0x7fe36802bcd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d472650df30bee server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:46:59.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.945+0000 7fe361600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:46:59.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.946+0000 7fe361600640 1 -- 172.21.15.31:0/4086574458 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36805d050 con 0x561e362b68b0 2024-03-21T10:46:59.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.946+0000 7fe381965740 1 -- 172.21.15.31:0/4086574458 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe3680581b0 con 0x561e362b68b0 2024-03-21T10:46:59.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.946+0000 7fe361600640 1 -- 172.21.15.31:0/4086574458 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fe368058dc0 con 0x561e362b68b0 2024-03-21T10:46:59.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.946+0000 7fe361600640 1 -- 172.21.15.31:0/4086574458 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe36806c050 con 0x561e362b68b0 2024-03-21T10:46:59.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.946+0000 7fe381965740 10 client.27601 fetch_fsmap learned FSMap version 28 2024-03-21T10:46:59.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.946+0000 7fe381965740 1 -- 172.21.15.31:0/4086574458 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561e3632b8d0 con 0x561e362b68b0 2024-03-21T10:46:59.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.946+0000 7fe361600640 1 -- 172.21.15.31:0/4086574458 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe3680581b0 con 0x561e362b68b0 2024-03-21T10:46:59.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.946+0000 7fe361600640 1 -- 172.21.15.31:0/4086574458 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe368004d50 con 0x561e362b68b0 2024-03-21T10:46:59.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.946+0000 7fe381965740 10 client.27601 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:46:59.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.946+0000 7fe381965740 10 client.27601 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:46:59.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.946+0000 7fe381965740 20 client.27601 populate_metadata read hostname 'smithi031' 2024-03-21T10:46:59.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.946+0000 7fe381965740 1 --2- 172.21.15.31:0/4086574458 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x561e3632bb50 0x561e3632df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:46:59.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.946+0000 7fe381965740 4 client.27601 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:46:59.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.946+0000 7fe381965740 1 -- 172.21.15.31:0/4086574458 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561e36221b50 con 0x561e3632bb50 2024-03-21T10:46:59.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.947+0000 7fe377400640 1 --2- 172.21.15.31:0/4086574458 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x561e3632bb50 0x561e3632df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:46:59.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.947+0000 7fe377400640 1 --2- 172.21.15.31:0/4086574458 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x561e3632bb50 0x561e3632df40 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=307938fb8b33e038 server_cookie=67242a1d5202b56c in_seq=0 out_seq=0 2024-03-21T10:46:59.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.947+0000 7fe361600640 10 client.27601 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:46:59.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.947+0000 7fe361600640 1 -- 172.21.15.31:0/4086574458 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561e36221b50 con 0x561e3632bb50 2024-03-21T10:46:59.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.947+0000 7fe361600640 10 client.27601 handle_command_reply: tid=0 2024-03-21T10:46:59.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.947+0000 7fe381965740 1 client.27601 shutdown 2024-03-21T10:46:59.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.948+0000 7fe381965740 1 -- 172.21.15.31:0/4086574458 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e362b68b0 msgr2=0x561e362c5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:59.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.948+0000 7fe381965740 1 --2- 172.21.15.31:0/4086574458 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e362b68b0 0x561e362c5630 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fe368004a60 tx=0x7fe36802bcd0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.948+0000 7fe381965740 1 -- 172.21.15.31:0/4086574458 shutdown_connections 2024-03-21T10:46:59.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.948+0000 7fe381965740 1 --2- 172.21.15.31:0/4086574458 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x561e3632bb50 0x561e3632df40 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.948+0000 7fe381965740 1 --2- 172.21.15.31:0/4086574458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e362bfde0 0x561e362c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.948+0000 7fe381965740 1 --2- 172.21.15.31:0/4086574458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e362b7260 0x561e362c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.948+0000 7fe361600640 0 client.27601 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:46:59.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.948+0000 7fe381965740 1 --2- 172.21.15.31:0/4086574458 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561e362b68b0 0x561e362c5630 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.948+0000 7fe381965740 1 -- 172.21.15.31:0/4086574458 >> 172.21.15.31:0/4086574458 conn(0x561e362b0f80 msgr2=0x561e362b7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:59.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.948+0000 7fe381965740 1 -- 172.21.15.31:0/4086574458 shutdown_connections 2024-03-21T10:46:59.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.948+0000 7fe381965740 1 -- 172.21.15.31:0/4086574458 wait complete. 2024-03-21T10:46:59.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.948+0000 7fe381965740 20 client.27601 trim_cache size 0 max 16384 2024-03-21T10:46:59.949 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:46:59.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.949+0000 7fe37d600640 1 -- 172.21.15.31:0/2860803433 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe35c0a3020 msgr2=0x7fe35c0a54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:59.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.949+0000 7fe37d600640 1 --2- 172.21.15.31:0/2860803433 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe35c0a3020 0x7fe35c0a54e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe368006010 tx=0x7fe368000bc0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.949+0000 7fe37d600640 1 -- 172.21.15.31:0/2860803433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe378102f50 msgr2=0x7fe37819a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:46:59.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.949+0000 7fe37d600640 1 --2- 172.21.15.31:0/2860803433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe378102f50 0x7fe37819a7b0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fe36400e9f0 tx=0x7fe36400eeb0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.949+0000 7fe37d600640 1 -- 172.21.15.31:0/2860803433 shutdown_connections 2024-03-21T10:46:59.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.949+0000 7fe37d600640 1 --2- 172.21.15.31:0/2860803433 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe35c0a3020 0x7fe35c0a54e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.949+0000 7fe37d600640 1 --2- 172.21.15.31:0/2860803433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe37810ba50 0x7fe37819e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.949+0000 7fe37d600640 1 --2- 172.21.15.31:0/2860803433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe378102f50 0x7fe37819a7b0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.949+0000 7fe37d600640 1 --2- 172.21.15.31:0/2860803433 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe378102630 0x7fe37819a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:46:59.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.949+0000 7fe37d600640 1 -- 172.21.15.31:0/2860803433 >> 172.21.15.31:0/2860803433 conn(0x7fe378077130 msgr2=0x7fe37810d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:46:59.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.949+0000 7fe37d600640 1 -- 172.21.15.31:0/2860803433 shutdown_connections 2024-03-21T10:46:59.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:46:59.949+0000 7fe37d600640 1 -- 172.21.15.31:0/2860803433 wait complete. 2024-03-21T10:46:59.963 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:46:59.964 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-21T10:47:00.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.187+0000 7f15db200640 1 -- 172.21.15.31:0/3121436180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15dc073a70 msgr2=0x7f15dc073e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:00.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.187+0000 7f15db200640 1 --2- 172.21.15.31:0/3121436180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15dc073a70 0x7f15dc073e50 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f15cc009be0 tx=0x7f15cc02bf80 comp rx=0 tx=0).stop 2024-03-21T10:47:00.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.187+0000 7f15db200640 1 -- 172.21.15.31:0/3121436180 shutdown_connections 2024-03-21T10:47:00.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.187+0000 7f15db200640 1 --2- 172.21.15.31:0/3121436180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15dc10dd20 0x7f15dc110130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.187+0000 7f15db200640 1 --2- 172.21.15.31:0/3121436180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15dc074390 0x7f15dc10d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.187+0000 7f15db200640 1 --2- 172.21.15.31:0/3121436180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15dc073a70 0x7f15dc073e50 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.187+0000 7f15db200640 1 -- 172.21.15.31:0/3121436180 >> 172.21.15.31:0/3121436180 conn(0x7f15dc0fcfb0 msgr2=0x7f15dc0ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:00.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.188+0000 7f15db200640 1 -- 172.21.15.31:0/3121436180 shutdown_connections 2024-03-21T10:47:00.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.188+0000 7f15db200640 1 -- 172.21.15.31:0/3121436180 wait complete. 2024-03-21T10:47:00.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.190+0000 7f15db200640 1 Processor -- start 2024-03-21T10:47:00.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.190+0000 7f15db200640 1 -- start start 2024-03-21T10:47:00.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.191+0000 7f15db200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15dc073a70 0x7f15dc199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.191+0000 7f15db200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15dc074390 0x7f15dc19a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.191+0000 7f15db200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15dc10dd20 0x7f15dc19e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.191+0000 7f15db200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc113ad0 con 0x7f15dc073a70 2024-03-21T10:47:00.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.191+0000 7f15db200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc113950 con 0x7f15dc074390 2024-03-21T10:47:00.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.191+0000 7f15db200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc113c50 con 0x7f15dc10dd20 2024-03-21T10:47:00.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.191+0000 7f15d1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15dc074390 0x7f15dc19a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.191+0000 7f15d1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15dc074390 0x7f15dc19a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42498/0 (socket says 172.21.15.31:42498) 2024-03-21T10:47:00.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.191+0000 7f15d1400640 1 -- 172.21.15.31:0/3077064620 learned_addr learned my addr 172.21.15.31:0/3077064620 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:00.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.191+0000 7f15d9e00640 1 --2- 172.21.15.31:0/3077064620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15dc073a70 0x7f15dc199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.191+0000 7f15da800640 1 --2- 172.21.15.31:0/3077064620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15dc10dd20 0x7f15dc19e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.192+0000 7f15d1400640 1 -- 172.21.15.31:0/3077064620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15dc10dd20 msgr2=0x7f15dc19e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:00.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.192+0000 7f15d1400640 1 --2- 172.21.15.31:0/3077064620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15dc10dd20 0x7f15dc19e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.192+0000 7f15d1400640 1 -- 172.21.15.31:0/3077064620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15dc073a70 msgr2=0x7f15dc199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:00.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.192+0000 7f15d1400640 1 --2- 172.21.15.31:0/3077064620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15dc073a70 0x7f15dc199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.192+0000 7f15d1400640 1 -- 172.21.15.31:0/3077064620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15dc19efa0 con 0x7f15dc074390 2024-03-21T10:47:00.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.192+0000 7f15d1400640 1 --2- 172.21.15.31:0/3077064620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15dc074390 0x7f15dc19a4f0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f15c400e9c0 tx=0x7f15c400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:00.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.192+0000 7f15d3400640 1 -- 172.21.15.31:0/3077064620 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c400ccc0 con 0x7f15dc074390 2024-03-21T10:47:00.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.192+0000 7f15d3400640 1 -- 172.21.15.31:0/3077064620 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15c4005500 con 0x7f15dc074390 2024-03-21T10:47:00.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.192+0000 7f15d3400640 1 -- 172.21.15.31:0/3077064620 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c4010430 con 0x7f15dc074390 2024-03-21T10:47:00.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.192+0000 7f15db200640 1 -- 172.21.15.31:0/3077064620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15dc19f290 con 0x7f15dc074390 2024-03-21T10:47:00.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.192+0000 7f15db200640 1 -- 172.21.15.31:0/3077064620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15dc1a6b20 con 0x7f15dc074390 2024-03-21T10:47:00.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.195+0000 7f15d3400640 1 -- 172.21.15.31:0/3077064620 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f15c40046c0 con 0x7f15dc074390 2024-03-21T10:47:00.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.196+0000 7f15e37ba740 1 Processor -- start 2024-03-21T10:47:00.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.196+0000 7f15e37ba740 1 -- start start 2024-03-21T10:47:00.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.196+0000 7f15e37ba740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdc2a888b0 0x55fdc2a88c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.196+0000 7f15e37ba740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fdc2a89260 0x55fdc2a91810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.196+0000 7f15e37ba740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdc2a91de0 0x55fdc2a94230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.196+0000 7f15e37ba740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdc29ec2c0 con 0x55fdc2a89260 2024-03-21T10:47:00.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.196+0000 7f15e37ba740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdc2994c90 con 0x55fdc2a91de0 2024-03-21T10:47:00.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.196+0000 7f15e37ba740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdc29e8360 con 0x55fdc2a888b0 2024-03-21T10:47:00.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.197+0000 7f15da800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdc2a888b0 0x55fdc2a88c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.197+0000 7f15da800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdc2a888b0 0x55fdc2a88c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53838/0 (socket says 172.21.15.31:53838) 2024-03-21T10:47:00.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.197+0000 7f15da800640 1 -- 172.21.15.31:0/1693640191 learned_addr learned my addr 172.21.15.31:0/1693640191 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:00.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.197+0000 7f15d9e00640 1 --2- 172.21.15.31:0/1693640191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fdc2a89260 0x55fdc2a91810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.197+0000 7f15d1400640 1 --2- 172.21.15.31:0/1693640191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdc2a91de0 0x55fdc2a94230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.197+0000 7f15da800640 1 -- 172.21.15.31:0/1693640191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdc2a91de0 msgr2=0x55fdc2a94230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:00.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.197+0000 7f15da800640 1 --2- 172.21.15.31:0/1693640191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdc2a91de0 0x55fdc2a94230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.197+0000 7f15da800640 1 -- 172.21.15.31:0/1693640191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fdc2a89260 msgr2=0x55fdc2a91810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:00.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.197+0000 7f15da800640 1 --2- 172.21.15.31:0/1693640191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fdc2a89260 0x55fdc2a91810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.197+0000 7f15da800640 1 -- 172.21.15.31:0/1693640191 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fdc29b9080 con 0x55fdc2a888b0 2024-03-21T10:47:00.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.197+0000 7f15d1400640 1 --2- 172.21.15.31:0/1693640191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdc2a91de0 0x55fdc2a94230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:47:00.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.197+0000 7f15da800640 1 --2- 172.21.15.31:0/1693640191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdc2a888b0 0x55fdc2a88c90 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f15d4009b80 tx=0x7f15d4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=586ddc41c778d262 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:00.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.198+0000 7f15cb000640 1 -- 172.21.15.31:0/1693640191 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d4004030 con 0x55fdc2a888b0 2024-03-21T10:47:00.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.198+0000 7f15cb000640 1 -- 172.21.15.31:0/1693640191 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15d40041d0 con 0x55fdc2a888b0 2024-03-21T10:47:00.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.198+0000 7f15cb000640 1 -- 172.21.15.31:0/1693640191 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d40196a0 con 0x55fdc2a888b0 2024-03-21T10:47:00.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.198+0000 7f15d3400640 1 --2- 172.21.15.31:0/3077064620 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f15c00a3110 0x7f15c00a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.198+0000 7f15d3400640 1 -- 172.21.15.31:0/3077064620 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f15c40cfe50 con 0x7f15dc074390 2024-03-21T10:47:00.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.198+0000 7f15d9e00640 1 --2- 172.21.15.31:0/3077064620 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f15c00a3110 0x7f15c00a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.198+0000 7f15e37ba740 1 -- 172.21.15.31:0/1693640191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdc2a888b0 msgr2=0x55fdc2a88c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:00.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.198+0000 7f15e37ba740 1 --2- 172.21.15.31:0/1693640191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdc2a888b0 0x55fdc2a88c90 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f15d4009b80 tx=0x7f15d4011810 comp rx=0 tx=0).stop 2024-03-21T10:47:00.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.198+0000 7f15e37ba740 1 -- 172.21.15.31:0/1693640191 shutdown_connections 2024-03-21T10:47:00.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.198+0000 7f15e37ba740 1 --2- 172.21.15.31:0/1693640191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdc2a91de0 0x55fdc2a94230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.198+0000 7f15e37ba740 1 --2- 172.21.15.31:0/1693640191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fdc2a89260 0x55fdc2a91810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.198+0000 7f15e37ba740 1 --2- 172.21.15.31:0/1693640191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdc2a888b0 0x55fdc2a88c90 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.199+0000 7f15e37ba740 1 -- 172.21.15.31:0/1693640191 >> 172.21.15.31:0/1693640191 conn(0x55fdc2a82f80 msgr2=0x55fdc2a83360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:00.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.199+0000 7f15e37ba740 1 -- 172.21.15.31:0/1693640191 shutdown_connections 2024-03-21T10:47:00.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.199+0000 7f15e37ba740 1 -- 172.21.15.31:0/1693640191 wait complete. 2024-03-21T10:47:00.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.199+0000 7f15d9e00640 1 --2- 172.21.15.31:0/3077064620 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f15c00a3110 0x7f15c00a55d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f15cc004af0 tx=0x7f15cc005fd0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:00.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.199+0000 7f15e37ba740 1 -- start start 2024-03-21T10:47:00.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.200+0000 7f15e37ba740 1 Processor -- start 2024-03-21T10:47:00.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.200+0000 7f15e37ba740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdc2a888b0 0x55fdc2a91740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.200+0000 7f15e37ba740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fdc2a89260 0x55fdc2a8ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.200+0000 7f15e37ba740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdc2a91de0 0x55fdc2a8f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.200+0000 7f15e37ba740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdc2a94d50 con 0x55fdc2a89260 2024-03-21T10:47:00.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.201+0000 7f15e37ba740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdc29a80d0 con 0x55fdc2a888b0 2024-03-21T10:47:00.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.201+0000 7f15e37ba740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdc2a94ed0 con 0x55fdc2a91de0 2024-03-21T10:47:00.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.201+0000 7f15e37ba740 10 client.0 _ll_register_callbacks cb 0x55fdc29ace80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:00.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.201+0000 7f15d9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdc2a888b0 0x55fdc2a91740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.201+0000 7f15d1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fdc2a89260 0x55fdc2a8ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.201+0000 7f15d9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdc2a888b0 0x55fdc2a91740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42516/0 (socket says 172.21.15.31:42516) 2024-03-21T10:47:00.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.201+0000 7f15d9e00640 1 -- 172.21.15.31:0/3488669023 learned_addr learned my addr 172.21.15.31:0/3488669023 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:00.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.201+0000 7f15da800640 1 --2- 172.21.15.31:0/3488669023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdc2a91de0 0x55fdc2a8f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.201+0000 7f15da800640 1 -- 172.21.15.31:0/3488669023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdc2a888b0 msgr2=0x55fdc2a91740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:00.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.201+0000 7f15da800640 1 --2- 172.21.15.31:0/3488669023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdc2a888b0 0x55fdc2a91740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.201+0000 7f15da800640 1 -- 172.21.15.31:0/3488669023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fdc2a89260 msgr2=0x55fdc2a8ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:00.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.201+0000 7f15da800640 1 --2- 172.21.15.31:0/3488669023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fdc2a89260 0x55fdc2a8ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.201+0000 7f15da800640 1 -- 172.21.15.31:0/3488669023 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fdc27003f0 con 0x55fdc2a91de0 2024-03-21T10:47:00.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.202+0000 7f15da800640 1 --2- 172.21.15.31:0/3488669023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdc2a91de0 0x55fdc2a8f0e0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f15d4003e00 tx=0x7f15d4009620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=505ac1982b4e16f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:00.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.202+0000 7f15bf400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:47:00.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.202+0000 7f15bf400640 1 -- 172.21.15.31:0/3488669023 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d4028020 con 0x55fdc2a91de0 2024-03-21T10:47:00.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.202+0000 7f15bf400640 1 -- 172.21.15.31:0/3488669023 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f15d4004380 con 0x55fdc2a91de0 2024-03-21T10:47:00.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.202+0000 7f15e37ba740 1 -- 172.21.15.31:0/3488669023 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f15d40041d0 con 0x55fdc2a91de0 2024-03-21T10:47:00.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.202+0000 7f15bf400640 1 -- 172.21.15.31:0/3488669023 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15d4033050 con 0x55fdc2a91de0 2024-03-21T10:47:00.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.202+0000 7f15e37ba740 10 client.27613 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:00.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.202+0000 7f15e37ba740 1 -- 172.21.15.31:0/3488669023 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55fdc27ca7d0 con 0x55fdc2a91de0 2024-03-21T10:47:00.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.203+0000 7f15bf400640 1 -- 172.21.15.31:0/3488669023 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15d40041d0 con 0x55fdc2a91de0 2024-03-21T10:47:00.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.203+0000 7f15bf400640 1 -- 172.21.15.31:0/3488669023 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f15d4018070 con 0x55fdc2a91de0 2024-03-21T10:47:00.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.203+0000 7f15e37ba740 10 client.27613 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:00.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.203+0000 7f15e37ba740 10 client.27613 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:47:00.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.203+0000 7f15e37ba740 20 client.27613 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:00.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.203+0000 7f15e37ba740 1 --2- 172.21.15.31:0/3488669023 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55fdc2aa1be0 0x55fdc2aa3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.203+0000 7f15e37ba740 4 client.27613 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:47:00.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.203+0000 7f15e37ba740 1 -- 172.21.15.31:0/3488669023 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fdc29f9800 con 0x55fdc2aa1be0 2024-03-21T10:47:00.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.203+0000 7f15d9e00640 1 --2- 172.21.15.31:0/3488669023 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55fdc2aa1be0 0x55fdc2aa3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.204+0000 7f15d9e00640 1 --2- 172.21.15.31:0/3488669023 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55fdc2aa1be0 0x55fdc2aa3fd0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=67f1204b6f3bab30 server_cookie=69550f0d794bf6b1 in_seq=0 out_seq=0 2024-03-21T10:47:00.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.204+0000 7f15bf400640 10 client.27613 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:47:00.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.205+0000 7f15bf400640 1 -- 172.21.15.31:0/3488669023 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55fdc29f9800 con 0x55fdc2aa1be0 2024-03-21T10:47:00.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.205+0000 7f15bf400640 10 client.27613 handle_command_reply: tid=0 2024-03-21T10:47:00.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.205+0000 7f15e37ba740 1 client.27613 shutdown 2024-03-21T10:47:00.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.205+0000 7f15e37ba740 1 -- 172.21.15.31:0/3488669023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdc2a91de0 msgr2=0x55fdc2a8f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:00.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.205+0000 7f15e37ba740 1 --2- 172.21.15.31:0/3488669023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdc2a91de0 0x55fdc2a8f0e0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f15d4003e00 tx=0x7f15d4009620 comp rx=0 tx=0).stop 2024-03-21T10:47:00.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.205+0000 7f15e37ba740 1 -- 172.21.15.31:0/3488669023 shutdown_connections 2024-03-21T10:47:00.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.205+0000 7f15e37ba740 1 --2- 172.21.15.31:0/3488669023 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55fdc2aa1be0 0x55fdc2aa3fd0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.205+0000 7f15e37ba740 1 --2- 172.21.15.31:0/3488669023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdc2a91de0 0x55fdc2a8f0e0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.205+0000 7f15e37ba740 1 --2- 172.21.15.31:0/3488669023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fdc2a89260 0x55fdc2a8ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.205+0000 7f15e37ba740 1 --2- 172.21.15.31:0/3488669023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdc2a888b0 0x55fdc2a91740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.205+0000 7f15bf400640 0 client.27613 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:47:00.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.205+0000 7f15e37ba740 1 -- 172.21.15.31:0/3488669023 >> 172.21.15.31:0/3488669023 conn(0x55fdc2a82f80 msgr2=0x55fdc2a84cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:00.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.205+0000 7f15e37ba740 1 -- 172.21.15.31:0/3488669023 shutdown_connections 2024-03-21T10:47:00.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.205+0000 7f15e37ba740 1 -- 172.21.15.31:0/3488669023 wait complete. 2024-03-21T10:47:00.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.205+0000 7f15e37ba740 20 client.27613 trim_cache size 0 max 16384 2024-03-21T10:47:00.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.231+0000 7f15e37ba740 1 Processor -- start 2024-03-21T10:47:00.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.231+0000 7f15e37ba740 1 -- start start 2024-03-21T10:47:00.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.231+0000 7f15e37ba740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fdc2a888b0 0x55fdc2a91200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.231+0000 7f15e37ba740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdc2a89260 0x55fdc2a91740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.231+0000 7f15e37ba740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdc2a91de0 0x55fdc2a8bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.231+0000 7f15e37ba740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc1b2ed0 con 0x55fdc2a888b0 2024-03-21T10:47:00.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.231+0000 7f15e37ba740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc1b09d0 con 0x55fdc2a89260 2024-03-21T10:47:00.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.231+0000 7f15e37ba740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc1aea90 con 0x55fdc2a91de0 2024-03-21T10:47:00.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.232+0000 7f15d1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdc2a89260 0x55fdc2a91740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.232+0000 7f15da800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdc2a91de0 0x55fdc2a8bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.232+0000 7f15d1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdc2a89260 0x55fdc2a91740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42532/0 (socket says 172.21.15.31:42532) 2024-03-21T10:47:00.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.232+0000 7f15d1400640 1 -- 172.21.15.31:0/2337499281 learned_addr learned my addr 172.21.15.31:0/2337499281 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:00.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.232+0000 7f15d9e00640 1 --2- 172.21.15.31:0/2337499281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fdc2a888b0 0x55fdc2a91200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.232+0000 7f15d1400640 1 -- 172.21.15.31:0/2337499281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdc2a91de0 msgr2=0x55fdc2a8bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:00.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.232+0000 7f15d1400640 1 --2- 172.21.15.31:0/2337499281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdc2a91de0 0x55fdc2a8bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.232+0000 7f15d1400640 1 -- 172.21.15.31:0/2337499281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fdc2a888b0 msgr2=0x55fdc2a91200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:00.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.232+0000 7f15d1400640 1 --2- 172.21.15.31:0/2337499281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fdc2a888b0 0x55fdc2a91200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.232+0000 7f15d1400640 1 -- 172.21.15.31:0/2337499281 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fdc2a87b40 con 0x55fdc2a89260 2024-03-21T10:47:00.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.232+0000 7f15da800640 1 --2- 172.21.15.31:0/2337499281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdc2a91de0 0x55fdc2a8bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:00.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.232+0000 7f15d9e00640 1 --2- 172.21.15.31:0/2337499281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fdc2a888b0 0x55fdc2a91200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:47:00.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.232+0000 7f15d1400640 1 --2- 172.21.15.31:0/2337499281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdc2a89260 0x55fdc2a91740 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f15c40982a0 tx=0x7f15c4004b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3a7f550811873f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:00.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.232+0000 7f15ca600640 1 -- 172.21.15.31:0/2337499281 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c409e070 con 0x55fdc2a89260 2024-03-21T10:47:00.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.233+0000 7f15ca600640 1 -- 172.21.15.31:0/2337499281 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15c409c540 con 0x55fdc2a89260 2024-03-21T10:47:00.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.233+0000 7f15ca600640 1 -- 172.21.15.31:0/2337499281 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c40cd420 con 0x55fdc2a89260 2024-03-21T10:47:00.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.233+0000 7f15e37ba740 1 -- 172.21.15.31:0/2337499281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdc2a89260 msgr2=0x55fdc2a91740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:00.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.233+0000 7f15e37ba740 1 --2- 172.21.15.31:0/2337499281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdc2a89260 0x55fdc2a91740 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f15c40982a0 tx=0x7f15c4004b00 comp rx=0 tx=0).stop 2024-03-21T10:47:00.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.233+0000 7f15e37ba740 1 -- 172.21.15.31:0/2337499281 shutdown_connections 2024-03-21T10:47:00.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.233+0000 7f15e37ba740 1 --2- 172.21.15.31:0/2337499281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdc2a91de0 0x55fdc2a8bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.233+0000 7f15e37ba740 1 --2- 172.21.15.31:0/2337499281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdc2a89260 0x55fdc2a91740 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.233+0000 7f15e37ba740 1 --2- 172.21.15.31:0/2337499281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fdc2a888b0 0x55fdc2a91200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.233+0000 7f15e37ba740 1 -- 172.21.15.31:0/2337499281 >> 172.21.15.31:0/2337499281 conn(0x55fdc2a82f80 msgr2=0x55fdc2a84c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:00.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.233+0000 7f15e37ba740 1 -- 172.21.15.31:0/2337499281 shutdown_connections 2024-03-21T10:47:00.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.233+0000 7f15e37ba740 1 -- 172.21.15.31:0/2337499281 wait complete. 2024-03-21T10:47:00.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.234+0000 7f15e37ba740 1 -- start start 2024-03-21T10:47:00.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.234+0000 7f15e37ba740 1 Processor -- start 2024-03-21T10:47:00.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.234+0000 7f15e37ba740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fdc2a888b0 0x55fdc2a99920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.234+0000 7f15e37ba740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdc2a89260 0x55fdc2a99e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.234+0000 7f15e37ba740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdc2a91de0 0x55fdc2a97da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.234+0000 7f15e37ba740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc1af090 con 0x55fdc2a888b0 2024-03-21T10:47:00.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.234+0000 7f15e37ba740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc1b0370 con 0x55fdc2a89260 2024-03-21T10:47:00.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.234+0000 7f15e37ba740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc1af6b0 con 0x55fdc2a91de0 2024-03-21T10:47:00.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.235+0000 7f15e37ba740 10 client.0 _ll_register_callbacks cb 0x7f15c4005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:00.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.235+0000 7f15d9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fdc2a888b0 0x55fdc2a99920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.235+0000 7f15d1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdc2a89260 0x55fdc2a99e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.235+0000 7f15d9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fdc2a888b0 0x55fdc2a99920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42134/0 (socket says 172.21.15.31:42134) 2024-03-21T10:47:00.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.235+0000 7f15d9e00640 1 -- 172.21.15.31:0/2183096826 learned_addr learned my addr 172.21.15.31:0/2183096826 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:00.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.235+0000 7f15da800640 1 --2- 172.21.15.31:0/2183096826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdc2a91de0 0x55fdc2a97da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.235+0000 7f15d1400640 1 -- 172.21.15.31:0/2183096826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdc2a91de0 msgr2=0x55fdc2a97da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:00.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.235+0000 7f15d1400640 1 --2- 172.21.15.31:0/2183096826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdc2a91de0 0x55fdc2a97da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.235+0000 7f15d1400640 1 -- 172.21.15.31:0/2183096826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fdc2a888b0 msgr2=0x55fdc2a99920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:00.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.235+0000 7f15d1400640 1 --2- 172.21.15.31:0/2183096826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fdc2a888b0 0x55fdc2a99920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.235+0000 7f15d1400640 1 -- 172.21.15.31:0/2183096826 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fdc2a985a0 con 0x55fdc2a89260 2024-03-21T10:47:00.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.235+0000 7f15d1400640 1 --2- 172.21.15.31:0/2183096826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdc2a89260 0x55fdc2a99e60 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x55fdc2a87940 tx=0x7f15c409b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b93deeae3d277ef server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:00.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.235+0000 7f15bf400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:47:00.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.235+0000 7f15bf400640 1 -- 172.21.15.31:0/2183096826 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c40b6070 con 0x55fdc2a89260 2024-03-21T10:47:00.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.236+0000 7f15e37ba740 1 -- 172.21.15.31:0/2183096826 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f15c409c540 con 0x55fdc2a89260 2024-03-21T10:47:00.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.236+0000 7f15bf400640 1 -- 172.21.15.31:0/2183096826 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f15c40a45c0 con 0x55fdc2a89260 2024-03-21T10:47:00.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.236+0000 7f15bf400640 1 -- 172.21.15.31:0/2183096826 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15c409e070 con 0x55fdc2a89260 2024-03-21T10:47:00.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.236+0000 7f15e37ba740 10 client.27698 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:00.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.236+0000 7f15e37ba740 1 -- 172.21.15.31:0/2183096826 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55fdc2afdd50 con 0x55fdc2a89260 2024-03-21T10:47:00.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.236+0000 7f15bf400640 1 -- 172.21.15.31:0/2183096826 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15c409c540 con 0x55fdc2a89260 2024-03-21T10:47:00.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.236+0000 7f15bf400640 1 -- 172.21.15.31:0/2183096826 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f15c409b290 con 0x55fdc2a89260 2024-03-21T10:47:00.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.236+0000 7f15e37ba740 10 client.27698 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:00.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.236+0000 7f15e37ba740 10 client.27698 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:47:00.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.236+0000 7f15e37ba740 20 client.27698 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:00.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.236+0000 7f15e37ba740 1 --2- 172.21.15.31:0/2183096826 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55fdc2afe3e0 0x55fdc2b00800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.236+0000 7f15e37ba740 4 client.27698 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:47:00.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.236+0000 7f15e37ba740 1 -- 172.21.15.31:0/2183096826 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55fdc29f3b50 con 0x55fdc2afe3e0 2024-03-21T10:47:00.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.237+0000 7f15d9e00640 1 --2- 172.21.15.31:0/2183096826 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55fdc2afe3e0 0x55fdc2b00800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.237+0000 7f15bf400640 10 client.27698 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:47:00.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.237+0000 7f15d9e00640 1 --2- 172.21.15.31:0/2183096826 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55fdc2afe3e0 0x55fdc2b00800 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2a79f8f6b1a8c99f server_cookie=433f75b2e993a8ac in_seq=0 out_seq=0 2024-03-21T10:47:00.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.237+0000 7f15bf400640 1 -- 172.21.15.31:0/2183096826 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55fdc29f3b50 con 0x55fdc2afe3e0 2024-03-21T10:47:00.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.237+0000 7f15bf400640 10 client.27698 handle_command_reply: tid=0 2024-03-21T10:47:00.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.237+0000 7f15e37ba740 1 client.27698 shutdown 2024-03-21T10:47:00.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.238+0000 7f15e37ba740 1 -- 172.21.15.31:0/2183096826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdc2a89260 msgr2=0x55fdc2a99e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:00.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.238+0000 7f15e37ba740 1 --2- 172.21.15.31:0/2183096826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdc2a89260 0x55fdc2a99e60 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x55fdc2a87940 tx=0x7f15c409b820 comp rx=0 tx=0).stop 2024-03-21T10:47:00.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.238+0000 7f15e37ba740 1 -- 172.21.15.31:0/2183096826 shutdown_connections 2024-03-21T10:47:00.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.238+0000 7f15e37ba740 1 --2- 172.21.15.31:0/2183096826 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55fdc2afe3e0 0x55fdc2b00800 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.238+0000 7f15e37ba740 1 --2- 172.21.15.31:0/2183096826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdc2a91de0 0x55fdc2a97da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.238+0000 7f15e37ba740 1 --2- 172.21.15.31:0/2183096826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fdc2a89260 0x55fdc2a99e60 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.238+0000 7f15e37ba740 1 --2- 172.21.15.31:0/2183096826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fdc2a888b0 0x55fdc2a99920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.238+0000 7f15bf400640 0 client.27698 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:47:00.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.238+0000 7f15e37ba740 1 -- 172.21.15.31:0/2183096826 >> 172.21.15.31:0/2183096826 conn(0x55fdc2a82f80 msgr2=0x55fdc2a818d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:00.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.238+0000 7f15e37ba740 1 -- 172.21.15.31:0/2183096826 shutdown_connections 2024-03-21T10:47:00.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.238+0000 7f15e37ba740 1 -- 172.21.15.31:0/2183096826 wait complete. 2024-03-21T10:47:00.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.238+0000 7f15e37ba740 20 client.27698 trim_cache size 0 max 16384 2024-03-21T10:47:00.240 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:47:00.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.239+0000 7f15db200640 1 -- 172.21.15.31:0/3077064620 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f15c00a3110 msgr2=0x7f15c00a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:00.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.239+0000 7f15db200640 1 --2- 172.21.15.31:0/3077064620 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f15c00a3110 0x7f15c00a55d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f15cc004af0 tx=0x7f15cc005fd0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.239+0000 7f15db200640 1 -- 172.21.15.31:0/3077064620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15dc074390 msgr2=0x7f15dc19a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:00.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.239+0000 7f15db200640 1 --2- 172.21.15.31:0/3077064620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15dc074390 0x7f15dc19a4f0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f15c400e9c0 tx=0x7f15c400ee80 comp rx=0 tx=0).stop 2024-03-21T10:47:00.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.239+0000 7f15db200640 1 -- 172.21.15.31:0/3077064620 shutdown_connections 2024-03-21T10:47:00.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.239+0000 7f15db200640 1 --2- 172.21.15.31:0/3077064620 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f15c00a3110 0x7f15c00a55d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.239+0000 7f15db200640 1 --2- 172.21.15.31:0/3077064620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15dc10dd20 0x7f15dc19e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.239+0000 7f15db200640 1 --2- 172.21.15.31:0/3077064620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15dc074390 0x7f15dc19a4f0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.239+0000 7f15db200640 1 --2- 172.21.15.31:0/3077064620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15dc073a70 0x7f15dc199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.239+0000 7f15db200640 1 -- 172.21.15.31:0/3077064620 >> 172.21.15.31:0/3077064620 conn(0x7f15dc0fcfb0 msgr2=0x7f15dc0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:00.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.239+0000 7f15db200640 1 -- 172.21.15.31:0/3077064620 shutdown_connections 2024-03-21T10:47:00.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.239+0000 7f15db200640 1 -- 172.21.15.31:0/3077064620 wait complete. 2024-03-21T10:47:00.251 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:46:59 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3455612563' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:47:00.253 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:47:00.253 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-21T10:47:00.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:46:59 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3455612563' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:47:00.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:46:59 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3455612563' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:47:00.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.491+0000 7fd1c9a00640 1 -- 172.21.15.31:0/1817857473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1c4104220 msgr2=0x7fd1c4110730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:00.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.491+0000 7fd1c9a00640 1 --2- 172.21.15.31:0/1817857473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1c4104220 0x7fd1c4110730 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc009ca0 tx=0x7fd1bc02c0f0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.491+0000 7fd1c9a00640 1 -- 172.21.15.31:0/1817857473 shutdown_connections 2024-03-21T10:47:00.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.491+0000 7fd1c9a00640 1 --2- 172.21.15.31:0/1817857473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1c4104220 0x7fd1c4110730 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.491+0000 7fd1c9a00640 1 --2- 172.21.15.31:0/1817857473 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1c4103860 0x7fd1c4103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.491+0000 7fd1c9a00640 1 --2- 172.21.15.31:0/1817857473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1c4109870 0x7fd1c4109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.491+0000 7fd1c9a00640 1 -- 172.21.15.31:0/1817857473 >> 172.21.15.31:0/1817857473 conn(0x7fd1c40ff310 msgr2=0x7fd1c4101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:00.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.491+0000 7fd1c9a00640 1 -- 172.21.15.31:0/1817857473 shutdown_connections 2024-03-21T10:47:00.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.491+0000 7fd1c9a00640 1 -- 172.21.15.31:0/1817857473 wait complete. 2024-03-21T10:47:00.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.492+0000 7fd1c9a00640 1 Processor -- start 2024-03-21T10:47:00.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.493+0000 7fd1c9a00640 1 -- start start 2024-03-21T10:47:00.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.493+0000 7fd1c9a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1c4103860 0x7fd1c419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.493+0000 7fd1c9a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1c4104220 0x7fd1c419a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.493+0000 7fd1c9a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1c4109870 0x7fd1c419e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.493+0000 7fd1c9a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c4113c80 con 0x7fd1c4109870 2024-03-21T10:47:00.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.493+0000 7fd1c9a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c4113b00 con 0x7fd1c4103860 2024-03-21T10:47:00.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.493+0000 7fd1c9a00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c4113e00 con 0x7fd1c4104220 2024-03-21T10:47:00.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.493+0000 7fd1c2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1c4103860 0x7fd1c419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.493+0000 7fd1c3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1c4109870 0x7fd1c419e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.493+0000 7fd1c2000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1c4104220 0x7fd1c419a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.493+0000 7fd1c2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1c4103860 0x7fd1c419a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42566/0 (socket says 172.21.15.31:42566) 2024-03-21T10:47:00.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.493+0000 7fd1c2a00640 1 -- 172.21.15.31:0/2843263955 learned_addr learned my addr 172.21.15.31:0/2843263955 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:00.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.494+0000 7fd1c2a00640 1 -- 172.21.15.31:0/2843263955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1c4104220 msgr2=0x7fd1c419a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:00.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.494+0000 7fd1c2a00640 1 --2- 172.21.15.31:0/2843263955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1c4104220 0x7fd1c419a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.494+0000 7fd1c2a00640 1 -- 172.21.15.31:0/2843263955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1c4109870 msgr2=0x7fd1c419e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:00.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.494+0000 7fd1c2a00640 1 --2- 172.21.15.31:0/2843263955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1c4109870 0x7fd1c419e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.494+0000 7fd1c2a00640 1 -- 172.21.15.31:0/2843263955 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c419f0d0 con 0x7fd1c4103860 2024-03-21T10:47:00.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.494+0000 7fd1c3400640 1 --2- 172.21.15.31:0/2843263955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1c4109870 0x7fd1c419e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:00.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.494+0000 7fd1c2a00640 1 --2- 172.21.15.31:0/2843263955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1c4103860 0x7fd1c419a0e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400e9f0 tx=0x7fd1b400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:00.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.494+0000 7fd1bb400640 1 -- 172.21.15.31:0/2843263955 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b400cd80 con 0x7fd1c4103860 2024-03-21T10:47:00.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.494+0000 7fd1bb400640 1 -- 172.21.15.31:0/2843263955 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd1b4005500 con 0x7fd1c4103860 2024-03-21T10:47:00.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.494+0000 7fd1c9a00640 1 -- 172.21.15.31:0/2843263955 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1c419f3c0 con 0x7fd1c4103860 2024-03-21T10:47:00.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.494+0000 7fd1bb400640 1 -- 172.21.15.31:0/2843263955 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4010690 con 0x7fd1c4103860 2024-03-21T10:47:00.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.495+0000 7fd1c9a00640 1 -- 172.21.15.31:0/2843263955 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1c41a6c50 con 0x7fd1c4103860 2024-03-21T10:47:00.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.498+0000 7fd1bb400640 1 -- 172.21.15.31:0/2843263955 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fd1b40046c0 con 0x7fd1c4103860 2024-03-21T10:47:00.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.498+0000 7fd1cc617740 1 Processor -- start 2024-03-21T10:47:00.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.498+0000 7fd1cc617740 1 -- start start 2024-03-21T10:47:00.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.498+0000 7fd1cc617740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f852eb66d0 0x55f852eb6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.498+0000 7fd1cc617740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f852eb7080 0x55f852ebf630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.498+0000 7fd1cc617740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f852ebfc00 0x55f852ec2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.498+0000 7fd1cc617740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f852e1a2c0 con 0x55f852eb66d0 2024-03-21T10:47:00.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.498+0000 7fd1cc617740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f852dc2c90 con 0x55f852ebfc00 2024-03-21T10:47:00.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.498+0000 7fd1cc617740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f852e16360 con 0x55f852eb7080 2024-03-21T10:47:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.499+0000 7fd1c2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f852eb7080 0x55f852ebf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.499+0000 7fd1c3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f852eb66d0 0x55f852eb6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.499+0000 7fd1c2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f852eb7080 0x55f852ebf630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53904/0 (socket says 172.21.15.31:53904) 2024-03-21T10:47:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.499+0000 7fd1c2a00640 1 -- 172.21.15.31:0/1292027381 learned_addr learned my addr 172.21.15.31:0/1292027381 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.499+0000 7fd1c2000640 1 --2- 172.21.15.31:0/1292027381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f852ebfc00 0x55f852ec2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.499+0000 7fd1c2a00640 1 -- 172.21.15.31:0/1292027381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f852ebfc00 msgr2=0x55f852ec2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.499+0000 7fd1c2a00640 1 --2- 172.21.15.31:0/1292027381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f852ebfc00 0x55f852ec2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.499+0000 7fd1c2a00640 1 -- 172.21.15.31:0/1292027381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f852eb66d0 msgr2=0x55f852eb6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.499+0000 7fd1c2a00640 1 --2- 172.21.15.31:0/1292027381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f852eb66d0 0x55f852eb6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.499+0000 7fd1c2a00640 1 -- 172.21.15.31:0/1292027381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f852de7080 con 0x55f852eb7080 2024-03-21T10:47:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.500+0000 7fd1c2a00640 1 --2- 172.21.15.31:0/1292027381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f852eb7080 0x55f852ebf630 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40cd150 tx=0x7fd1b40d3ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90b511dcfc7da0dc server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:00.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.500+0000 7fd1bb400640 1 --2- 172.21.15.31:0/2843263955 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd1a80a3110 0x7fd1a80a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.500+0000 7fd1afe00640 1 -- 172.21.15.31:0/1292027381 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b40e3070 con 0x55f852eb7080 2024-03-21T10:47:00.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.500+0000 7fd1afe00640 1 -- 172.21.15.31:0/1292027381 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd1b409c5b0 con 0x55f852eb7080 2024-03-21T10:47:00.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.500+0000 7fd1afe00640 1 -- 172.21.15.31:0/1292027381 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4085420 con 0x55f852eb7080 2024-03-21T10:47:00.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.500+0000 7fd1c3400640 1 --2- 172.21.15.31:0/2843263955 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd1a80a3110 0x7fd1a80a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.500+0000 7fd1bb400640 1 -- 172.21.15.31:0/2843263955 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fd1b40cfec0 con 0x7fd1c4103860 2024-03-21T10:47:00.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.501+0000 7fd1cc617740 1 -- 172.21.15.31:0/1292027381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f852eb7080 msgr2=0x55f852ebf630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:00.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.501+0000 7fd1cc617740 1 --2- 172.21.15.31:0/1292027381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f852eb7080 0x55f852ebf630 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40cd150 tx=0x7fd1b40d3ed0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.501+0000 7fd1c3400640 1 --2- 172.21.15.31:0/2843263955 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd1a80a3110 0x7fd1a80a55d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc005e30 tx=0x7fd1bc005d80 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:00.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.501+0000 7fd1cc617740 1 -- 172.21.15.31:0/1292027381 shutdown_connections 2024-03-21T10:47:00.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.501+0000 7fd1cc617740 1 --2- 172.21.15.31:0/1292027381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f852ebfc00 0x55f852ec2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.501+0000 7fd1cc617740 1 --2- 172.21.15.31:0/1292027381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f852eb7080 0x55f852ebf630 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.501+0000 7fd1cc617740 1 --2- 172.21.15.31:0/1292027381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f852eb66d0 0x55f852eb6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.501+0000 7fd1cc617740 1 -- 172.21.15.31:0/1292027381 >> 172.21.15.31:0/1292027381 conn(0x55f852eb0da0 msgr2=0x55f852eb1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:00.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.501+0000 7fd1cc617740 1 -- 172.21.15.31:0/1292027381 shutdown_connections 2024-03-21T10:47:00.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.501+0000 7fd1cc617740 1 -- 172.21.15.31:0/1292027381 wait complete. 2024-03-21T10:47:00.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.501+0000 7fd1cc617740 1 -- start start 2024-03-21T10:47:00.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.502+0000 7fd1cc617740 1 Processor -- start 2024-03-21T10:47:00.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.502+0000 7fd1cc617740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f852eb66d0 0x55f852ebf480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.502+0000 7fd1cc617740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f852eb7080 0x55f852ebc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.502+0000 7fd1cc617740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f852ebfc00 0x55f852ebcd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.502+0000 7fd1cc617740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f852ec2b70 con 0x55f852eb66d0 2024-03-21T10:47:00.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.502+0000 7fd1cc617740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f852dd60d0 con 0x55f852eb7080 2024-03-21T10:47:00.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.502+0000 7fd1cc617740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f852ec2cf0 con 0x55f852ebfc00 2024-03-21T10:47:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.502+0000 7fd1cc617740 10 client.0 _ll_register_callbacks cb 0x55f852ddae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.502+0000 7fd1c2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f852eb66d0 0x55f852ebf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.502+0000 7fd1c2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f852eb7080 0x55f852ebc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.502+0000 7fd1c3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f852ebfc00 0x55f852ebcd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.502+0000 7fd1c2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f852eb66d0 0x55f852ebf480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42180/0 (socket says 172.21.15.31:42180) 2024-03-21T10:47:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.502+0000 7fd1c2a00640 1 -- 172.21.15.31:0/1990363963 learned_addr learned my addr 172.21.15.31:0/1990363963 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.503+0000 7fd1c2000640 1 -- 172.21.15.31:0/1990363963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f852ebfc00 msgr2=0x55f852ebcd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.503+0000 7fd1c2000640 1 --2- 172.21.15.31:0/1990363963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f852ebfc00 0x55f852ebcd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.503+0000 7fd1c2000640 1 -- 172.21.15.31:0/1990363963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f852eb66d0 msgr2=0x55f852ebf480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.503+0000 7fd1c2000640 1 --2- 172.21.15.31:0/1990363963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f852eb66d0 0x55f852ebf480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.503+0000 7fd1c2000640 1 -- 172.21.15.31:0/1990363963 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f852b2e3f0 con 0x55f852eb7080 2024-03-21T10:47:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.503+0000 7fd1c2a00640 1 --2- 172.21.15.31:0/1990363963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f852eb66d0 0x55f852ebf480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.503+0000 7fd1c2000640 1 --2- 172.21.15.31:0/1990363963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f852eb7080 0x55f852ebc810 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd1b00126d0 tx=0x7fd1b0012b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d396aa054295386 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.503+0000 7fd1acc00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:47:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.503+0000 7fd1acc00640 1 -- 172.21.15.31:0/1990363963 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b000f9e0 con 0x55f852eb7080 2024-03-21T10:47:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.503+0000 7fd1cc617740 1 -- 172.21.15.31:0/1990363963 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd1b409c5b0 con 0x55f852eb7080 2024-03-21T10:47:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.503+0000 7fd1acc00640 1 -- 172.21.15.31:0/1990363963 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fd1b000fb80 con 0x55f852eb7080 2024-03-21T10:47:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.504+0000 7fd1acc00640 1 -- 172.21.15.31:0/1990363963 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1b000fd80 con 0x55f852eb7080 2024-03-21T10:47:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.504+0000 7fd1cc617740 10 client.27710 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.504+0000 7fd1cc617740 1 -- 172.21.15.31:0/1990363963 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f852bf87d0 con 0x55f852eb7080 2024-03-21T10:47:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.504+0000 7fd1acc00640 1 -- 172.21.15.31:0/1990363963 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd1b409c5b0 con 0x55f852eb7080 2024-03-21T10:47:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.504+0000 7fd1acc00640 1 -- 172.21.15.31:0/1990363963 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fd1b001c5b0 con 0x55f852eb7080 2024-03-21T10:47:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.504+0000 7fd1cc617740 10 client.27710 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.504+0000 7fd1cc617740 10 client.27710 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:47:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.504+0000 7fd1cc617740 20 client.27710 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.504+0000 7fd1cc617740 1 --2- 172.21.15.31:0/1990363963 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f852ecfa80 0x55f852ed1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.504+0000 7fd1cc617740 4 client.27710 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:47:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.504+0000 7fd1cc617740 1 -- 172.21.15.31:0/1990363963 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f852e27800 con 0x55f852ecfa80 2024-03-21T10:47:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.505+0000 7fd1c2a00640 1 --2- 172.21.15.31:0/1990363963 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f852ecfa80 0x55f852ed1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.505+0000 7fd1c2a00640 1 --2- 172.21.15.31:0/1990363963 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f852ecfa80 0x55f852ed1f00 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17a1801ff93fc324 server_cookie=5feca241d1701612 in_seq=0 out_seq=0 2024-03-21T10:47:00.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.505+0000 7fd1acc00640 10 client.27710 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:00.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.506+0000 7fd1acc00640 1 -- 172.21.15.31:0/1990363963 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f852e27800 con 0x55f852ecfa80 2024-03-21T10:47:00.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.506+0000 7fd1acc00640 10 client.27710 handle_command_reply: tid=0 2024-03-21T10:47:00.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.506+0000 7fd1cc617740 1 client.27710 shutdown 2024-03-21T10:47:00.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.506+0000 7fd1cc617740 1 -- 172.21.15.31:0/1990363963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f852eb7080 msgr2=0x55f852ebc810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:00.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.506+0000 7fd1cc617740 1 --2- 172.21.15.31:0/1990363963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f852eb7080 0x55f852ebc810 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd1b00126d0 tx=0x7fd1b0012b90 comp rx=0 tx=0).stop 2024-03-21T10:47:00.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.506+0000 7fd1cc617740 1 -- 172.21.15.31:0/1990363963 shutdown_connections 2024-03-21T10:47:00.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.506+0000 7fd1cc617740 1 --2- 172.21.15.31:0/1990363963 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f852ecfa80 0x55f852ed1f00 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.506+0000 7fd1cc617740 1 --2- 172.21.15.31:0/1990363963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f852ebfc00 0x55f852ebcd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.506+0000 7fd1cc617740 1 --2- 172.21.15.31:0/1990363963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f852eb7080 0x55f852ebc810 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.506+0000 7fd1cc617740 1 --2- 172.21.15.31:0/1990363963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f852eb66d0 0x55f852ebf480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.506+0000 7fd1acc00640 0 client.27710 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:00.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.506+0000 7fd1cc617740 1 -- 172.21.15.31:0/1990363963 >> 172.21.15.31:0/1990363963 conn(0x55f852eb0da0 msgr2=0x55f852eb2af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:00.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.506+0000 7fd1cc617740 1 -- 172.21.15.31:0/1990363963 shutdown_connections 2024-03-21T10:47:00.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.506+0000 7fd1cc617740 1 -- 172.21.15.31:0/1990363963 wait complete. 2024-03-21T10:47:00.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.506+0000 7fd1cc617740 20 client.27710 trim_cache size 0 max 16384 2024-03-21T10:47:00.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.520+0000 7fd1cc617740 1 Processor -- start 2024-03-21T10:47:00.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.520+0000 7fd1cc617740 1 -- start start 2024-03-21T10:47:00.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.521+0000 7fd1cc617740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f852eb66d0 0x55f852eafd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.521+0000 7fd1cc617740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f852eb7080 0x55f852ebef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.521+0000 7fd1cc617740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f852ebfc00 0x55f852ebf480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.521+0000 7fd1cc617740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c41b3120 con 0x55f852eb66d0 2024-03-21T10:47:00.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.521+0000 7fd1cc617740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c41b0c20 con 0x55f852eb7080 2024-03-21T10:47:00.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.521+0000 7fd1cc617740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c41aece0 con 0x55f852ebfc00 2024-03-21T10:47:00.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.521+0000 7fd1c2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f852eb66d0 0x55f852eafd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.521+0000 7fd1c2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f852eb7080 0x55f852ebef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.521+0000 7fd1c3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f852ebfc00 0x55f852ebf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.521+0000 7fd1c2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f852eb66d0 0x55f852eafd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42188/0 (socket says 172.21.15.31:42188) 2024-03-21T10:47:00.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.521+0000 7fd1c2a00640 1 -- 172.21.15.31:0/3617596455 learned_addr learned my addr 172.21.15.31:0/3617596455 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:00.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.521+0000 7fd1c3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f852ebfc00 0x55f852ebf480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53920/0 (socket says 172.21.15.31:53920) 2024-03-21T10:47:00.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.521+0000 7fd1c2000640 1 -- 172.21.15.31:0/3617596455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f852ebfc00 msgr2=0x55f852ebf480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:00.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.521+0000 7fd1c2000640 1 --2- 172.21.15.31:0/3617596455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f852ebfc00 0x55f852ebf480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.521+0000 7fd1c2000640 1 -- 172.21.15.31:0/3617596455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f852eb66d0 msgr2=0x55f852eafd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:00.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.521+0000 7fd1c2000640 1 --2- 172.21.15.31:0/3617596455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f852eb66d0 0x55f852eafd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.521+0000 7fd1c2000640 1 -- 172.21.15.31:0/3617596455 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f852eac700 con 0x55f852eb7080 2024-03-21T10:47:00.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.522+0000 7fd1c2000640 1 --2- 172.21.15.31:0/3617596455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f852eb7080 0x55f852ebef40 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0009cd0 tx=0x7fd1b0005b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f93535e6441d3a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:00.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.522+0000 7fd1af400640 1 -- 172.21.15.31:0/3617596455 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b001cbe0 con 0x55f852eb7080 2024-03-21T10:47:00.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.522+0000 7fd1af400640 1 -- 172.21.15.31:0/3617596455 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd1b000f9e0 con 0x55f852eb7080 2024-03-21T10:47:00.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.522+0000 7fd1af400640 1 -- 172.21.15.31:0/3617596455 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b0005720 con 0x55f852eb7080 2024-03-21T10:47:00.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.523+0000 7fd1cc617740 1 -- 172.21.15.31:0/3617596455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f852eb7080 msgr2=0x55f852ebef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:00.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.523+0000 7fd1cc617740 1 --2- 172.21.15.31:0/3617596455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f852eb7080 0x55f852ebef40 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0009cd0 tx=0x7fd1b0005b20 comp rx=0 tx=0).stop 2024-03-21T10:47:00.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.523+0000 7fd1cc617740 1 -- 172.21.15.31:0/3617596455 shutdown_connections 2024-03-21T10:47:00.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.523+0000 7fd1cc617740 1 --2- 172.21.15.31:0/3617596455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f852ebfc00 0x55f852ebf480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.523+0000 7fd1cc617740 1 --2- 172.21.15.31:0/3617596455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f852eb7080 0x55f852ebef40 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.523+0000 7fd1cc617740 1 --2- 172.21.15.31:0/3617596455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f852eb66d0 0x55f852eafd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.523+0000 7fd1cc617740 1 -- 172.21.15.31:0/3617596455 >> 172.21.15.31:0/3617596455 conn(0x55f852eb0da0 msgr2=0x55f852eaf620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:00.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.523+0000 7fd1cc617740 1 -- 172.21.15.31:0/3617596455 shutdown_connections 2024-03-21T10:47:00.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.523+0000 7fd1cc617740 1 -- 172.21.15.31:0/3617596455 wait complete. 2024-03-21T10:47:00.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.523+0000 7fd1cc617740 1 -- start start 2024-03-21T10:47:00.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.523+0000 7fd1cc617740 1 Processor -- start 2024-03-21T10:47:00.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.524+0000 7fd1cc617740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f852eb66d0 0x55f852ec3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.524+0000 7fd1cc617740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f852eb7080 0x55f852ec3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.524+0000 7fd1cc617740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f852ebfc00 0x55f852f2a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.524+0000 7fd1cc617740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c41af2e0 con 0x55f852eb7080 2024-03-21T10:47:00.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.524+0000 7fd1cc617740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c41b05c0 con 0x55f852eb66d0 2024-03-21T10:47:00.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.524+0000 7fd1cc617740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c41af900 con 0x55f852ebfc00 2024-03-21T10:47:00.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.524+0000 7fd1cc617740 10 client.0 _ll_register_callbacks cb 0x7fd1b0005a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:00.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.524+0000 7fd1c2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f852eb7080 0x55f852ec3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.524+0000 7fd1c2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f852eb7080 0x55f852ec3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42198/0 (socket says 172.21.15.31:42198) 2024-03-21T10:47:00.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.524+0000 7fd1c2000640 1 -- 172.21.15.31:0/898191676 learned_addr learned my addr 172.21.15.31:0/898191676 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:00.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.524+0000 7fd1c3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f852ebfc00 0x55f852f2a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.524+0000 7fd1c2a00640 1 --2- 172.21.15.31:0/898191676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f852eb66d0 0x55f852ec3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.524+0000 7fd1c2000640 1 -- 172.21.15.31:0/898191676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f852ebfc00 msgr2=0x55f852f2a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:00.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.524+0000 7fd1c2000640 1 --2- 172.21.15.31:0/898191676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f852ebfc00 0x55f852f2a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.524+0000 7fd1c2000640 1 -- 172.21.15.31:0/898191676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f852eb66d0 msgr2=0x55f852ec3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:00.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.524+0000 7fd1c2000640 1 --2- 172.21.15.31:0/898191676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f852eb66d0 0x55f852ec3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.524+0000 7fd1c2000640 1 -- 172.21.15.31:0/898191676 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f852f2a8d0 con 0x55f852eb7080 2024-03-21T10:47:00.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.525+0000 7fd1c2000640 1 --2- 172.21.15.31:0/898191676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f852eb7080 0x55f852ec3cc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd1b001c970 tx=0x7fd1b000ffa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bd6831b7119a8b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:00.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.525+0000 7fd1acc00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:47:00.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.525+0000 7fd1acc00640 1 -- 172.21.15.31:0/898191676 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b000fb90 con 0x55f852eb7080 2024-03-21T10:47:00.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.525+0000 7fd1acc00640 1 -- 172.21.15.31:0/898191676 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fd1b001c630 con 0x55f852eb7080 2024-03-21T10:47:00.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.525+0000 7fd1cc617740 1 -- 172.21.15.31:0/898191676 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd1b000f9e0 con 0x55f852eb7080 2024-03-21T10:47:00.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.525+0000 7fd1acc00640 1 -- 172.21.15.31:0/898191676 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1b0005360 con 0x55f852eb7080 2024-03-21T10:47:00.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.525+0000 7fd1cc617740 10 client.17112 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:00.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.525+0000 7fd1cc617740 1 -- 172.21.15.31:0/898191676 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f852f2ba90 con 0x55f852eb7080 2024-03-21T10:47:00.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.526+0000 7fd1acc00640 1 -- 172.21.15.31:0/898191676 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd1b000f9e0 con 0x55f852eb7080 2024-03-21T10:47:00.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.526+0000 7fd1acc00640 1 -- 172.21.15.31:0/898191676 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fd1b0010430 con 0x55f852eb7080 2024-03-21T10:47:00.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.526+0000 7fd1cc617740 10 client.17112 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:00.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.526+0000 7fd1cc617740 10 client.17112 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:47:00.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.526+0000 7fd1cc617740 20 client.17112 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:00.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.526+0000 7fd1cc617740 1 --2- 172.21.15.31:0/898191676 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f852f2c110 0x55f852f2e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:00.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.526+0000 7fd1cc617740 4 client.17112 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:47:00.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.526+0000 7fd1cc617740 1 -- 172.21.15.31:0/898191676 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f852ebd0e0 con 0x55f852f2c110 2024-03-21T10:47:00.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.526+0000 7fd1c2a00640 1 --2- 172.21.15.31:0/898191676 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f852f2c110 0x55f852f2e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:00.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.527+0000 7fd1acc00640 10 client.17112 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:00.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.527+0000 7fd1c2a00640 1 --2- 172.21.15.31:0/898191676 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f852f2c110 0x55f852f2e530 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0525e3f1e0d43db server_cookie=93b25b2b7638001e in_seq=0 out_seq=0 2024-03-21T10:47:00.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.527+0000 7fd1acc00640 1 -- 172.21.15.31:0/898191676 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f852ebd0e0 con 0x55f852f2c110 2024-03-21T10:47:00.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.527+0000 7fd1acc00640 10 client.17112 handle_command_reply: tid=0 2024-03-21T10:47:00.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.527+0000 7fd1cc617740 1 client.17112 shutdown 2024-03-21T10:47:00.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.527+0000 7fd1cc617740 1 -- 172.21.15.31:0/898191676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f852eb7080 msgr2=0x55f852ec3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:00.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.527+0000 7fd1cc617740 1 --2- 172.21.15.31:0/898191676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f852eb7080 0x55f852ec3cc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd1b001c970 tx=0x7fd1b000ffa0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.527+0000 7fd1cc617740 1 -- 172.21.15.31:0/898191676 shutdown_connections 2024-03-21T10:47:00.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.527+0000 7fd1cc617740 1 --2- 172.21.15.31:0/898191676 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55f852f2c110 0x55f852f2e530 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.527+0000 7fd1cc617740 1 --2- 172.21.15.31:0/898191676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f852ebfc00 0x55f852f2a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.527+0000 7fd1cc617740 1 --2- 172.21.15.31:0/898191676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f852eb7080 0x55f852ec3cc0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.528+0000 7fd1cc617740 1 --2- 172.21.15.31:0/898191676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f852eb66d0 0x55f852ec3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.528+0000 7fd1acc00640 0 client.17112 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:00.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.528+0000 7fd1cc617740 1 -- 172.21.15.31:0/898191676 >> 172.21.15.31:0/898191676 conn(0x55f852eb0da0 msgr2=0x55f852eb7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:00.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.528+0000 7fd1cc617740 1 -- 172.21.15.31:0/898191676 shutdown_connections 2024-03-21T10:47:00.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.528+0000 7fd1cc617740 1 -- 172.21.15.31:0/898191676 wait complete. 2024-03-21T10:47:00.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.528+0000 7fd1cc617740 20 client.17112 trim_cache size 0 max 16384 2024-03-21T10:47:00.528 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:47:00.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.528+0000 7fd1c9a00640 1 -- 172.21.15.31:0/2843263955 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd1a80a3110 msgr2=0x7fd1a80a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:00.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.528+0000 7fd1c9a00640 1 --2- 172.21.15.31:0/2843263955 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd1a80a3110 0x7fd1a80a55d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc005e30 tx=0x7fd1bc005d80 comp rx=0 tx=0).stop 2024-03-21T10:47:00.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.528+0000 7fd1c9a00640 1 -- 172.21.15.31:0/2843263955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1c4103860 msgr2=0x7fd1c419a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:00.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.528+0000 7fd1c9a00640 1 --2- 172.21.15.31:0/2843263955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1c4103860 0x7fd1c419a0e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400e9f0 tx=0x7fd1b400edf0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.529+0000 7fd1c9a00640 1 -- 172.21.15.31:0/2843263955 shutdown_connections 2024-03-21T10:47:00.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.529+0000 7fd1c9a00640 1 --2- 172.21.15.31:0/2843263955 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd1a80a3110 0x7fd1a80a55d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.529+0000 7fd1c9a00640 1 --2- 172.21.15.31:0/2843263955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1c4109870 0x7fd1c419e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.529+0000 7fd1c9a00640 1 --2- 172.21.15.31:0/2843263955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1c4104220 0x7fd1c419a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.529+0000 7fd1c9a00640 1 --2- 172.21.15.31:0/2843263955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1c4103860 0x7fd1c419a0e0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:00.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.529+0000 7fd1c9a00640 1 -- 172.21.15.31:0/2843263955 >> 172.21.15.31:0/2843263955 conn(0x7fd1c40ff310 msgr2=0x7fd1c40ff8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:00.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.529+0000 7fd1c9a00640 1 -- 172.21.15.31:0/2843263955 shutdown_connections 2024-03-21T10:47:00.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:00.529+0000 7fd1c9a00640 1 -- 172.21.15.31:0/2843263955 wait complete. 2024-03-21T10:47:01.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:00 smithi031 ceph-mon[31673]: pgmap v260: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-03-21T10:47:01.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:00 smithi070 ceph-mon[36330]: pgmap v260: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-03-21T10:47:01.331 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:00 smithi040 ceph-mon[36154]: pgmap v260: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-03-21T10:47:01.544 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-21T10:47:01.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:47:01.545 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-21T10:47:01.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.774+0000 7f7a9a400640 1 -- 172.21.15.31:0/3656950192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a94109670 msgr2=0x7f7a94109a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:01.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.774+0000 7f7a9a400640 1 --2- 172.21.15.31:0/3656950192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a94109670 0x7f7a94109a50 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f7a88009930 tx=0x7f7a8802bc30 comp rx=0 tx=0).stop 2024-03-21T10:47:01.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.774+0000 7f7a9a400640 1 -- 172.21.15.31:0/3656950192 shutdown_connections 2024-03-21T10:47:01.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.774+0000 7f7a9a400640 1 --2- 172.21.15.31:0/3656950192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a94104020 0x7f7a94110530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.774+0000 7f7a9a400640 1 --2- 172.21.15.31:0/3656950192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a94103660 0x7f7a94103ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.774+0000 7f7a9a400640 1 --2- 172.21.15.31:0/3656950192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a94109670 0x7f7a94109a50 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.774+0000 7f7a9a400640 1 -- 172.21.15.31:0/3656950192 >> 172.21.15.31:0/3656950192 conn(0x7f7a940ff150 msgr2=0x7f7a94101570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:01.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.774+0000 7f7a9a400640 1 -- 172.21.15.31:0/3656950192 shutdown_connections 2024-03-21T10:47:01.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.774+0000 7f7a9a400640 1 -- 172.21.15.31:0/3656950192 wait complete. 2024-03-21T10:47:01.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.776+0000 7f7a9a400640 1 Processor -- start 2024-03-21T10:47:01.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.776+0000 7f7a9a400640 1 -- start start 2024-03-21T10:47:01.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.776+0000 7f7a9a400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a94103660 0x7f7a94199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:01.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.776+0000 7f7a9a400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a94104020 0x7f7a9419a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:01.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.776+0000 7f7a9a400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a94109670 0x7f7a9419e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:01.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.776+0000 7f7a9a400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a94113a90 con 0x7f7a94103660 2024-03-21T10:47:01.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.776+0000 7f7a9a400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a94113910 con 0x7f7a94109670 2024-03-21T10:47:01.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.776+0000 7f7a9a400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a94113c10 con 0x7f7a94104020 2024-03-21T10:47:01.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.777+0000 7f7a99a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a94109670 0x7f7a9419e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:01.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.777+0000 7f7a99000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a94103660 0x7f7a94199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:01.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.777+0000 7f7a99a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a94109670 0x7f7a9419e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42634/0 (socket says 172.21.15.31:42634) 2024-03-21T10:47:01.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.777+0000 7f7a99a00640 1 -- 172.21.15.31:0/4282155358 learned_addr learned my addr 172.21.15.31:0/4282155358 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:01.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.777+0000 7f7a93e00640 1 --2- 172.21.15.31:0/4282155358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a94104020 0x7f7a9419a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:01.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.777+0000 7f7a99a00640 1 -- 172.21.15.31:0/4282155358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a94104020 msgr2=0x7f7a9419a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:01.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.777+0000 7f7a99a00640 1 --2- 172.21.15.31:0/4282155358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a94104020 0x7f7a9419a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.777+0000 7f7a99a00640 1 -- 172.21.15.31:0/4282155358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a94103660 msgr2=0x7f7a94199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:01.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.777+0000 7f7a99a00640 1 --2- 172.21.15.31:0/4282155358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a94103660 0x7f7a94199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.777+0000 7f7a99a00640 1 -- 172.21.15.31:0/4282155358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a9419efe0 con 0x7f7a94109670 2024-03-21T10:47:01.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.778+0000 7f7a99a00640 1 --2- 172.21.15.31:0/4282155358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a94109670 0x7f7a9419e8b0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c00eb10 tx=0x7f7a8c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:01.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.778+0000 7f7a91600640 1 -- 172.21.15.31:0/4282155358 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c004b20 con 0x7f7a94109670 2024-03-21T10:47:01.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.778+0000 7f7a9a400640 1 -- 172.21.15.31:0/4282155358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a9419f2d0 con 0x7f7a94109670 2024-03-21T10:47:01.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.778+0000 7f7a91600640 1 -- 172.21.15.31:0/4282155358 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a8c004cc0 con 0x7f7a94109670 2024-03-21T10:47:01.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.778+0000 7f7a91600640 1 -- 172.21.15.31:0/4282155358 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c010760 con 0x7f7a94109670 2024-03-21T10:47:01.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.778+0000 7f7a9a400640 1 -- 172.21.15.31:0/4282155358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a941a6b60 con 0x7f7a94109670 2024-03-21T10:47:01.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.781+0000 7f7a9e837740 1 Processor -- start 2024-03-21T10:47:01.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.783+0000 7f7a91600640 1 -- 172.21.15.31:0/4282155358 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f7a8c005020 con 0x7f7a94109670 2024-03-21T10:47:01.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.783+0000 7f7a9e837740 1 -- start start 2024-03-21T10:47:01.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.783+0000 7f7a9e837740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565469482290 0x565469482670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:01.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.783+0000 7f7a9e837740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565469482bb0 0x56546948b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:01.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.784+0000 7f7a9e837740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56546948b790 0x56546948db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:01.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.784+0000 7f7a9e837740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654693e6eb0 con 0x565469482bb0 2024-03-21T10:47:01.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.784+0000 7f7a9e837740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56546938fa60 con 0x56546948b790 2024-03-21T10:47:01.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.784+0000 7f7a9e837740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654693e2f50 con 0x565469482290 2024-03-21T10:47:01.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.784+0000 7f7a93e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565469482290 0x565469482670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:01.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.784+0000 7f7a93e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565469482290 0x565469482670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53944/0 (socket says 172.21.15.31:53944) 2024-03-21T10:47:01.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.784+0000 7f7a93e00640 1 -- 172.21.15.31:0/37838221 learned_addr learned my addr 172.21.15.31:0/37838221 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:01.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.784+0000 7f7a93e00640 1 --2- 172.21.15.31:0/37838221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56546948b790 0x56546948db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:01.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.784+0000 7f7a93e00640 1 -- 172.21.15.31:0/37838221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56546948b790 msgr2=0x56546948db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:01.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.784+0000 7f7a93e00640 1 --2- 172.21.15.31:0/37838221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56546948b790 0x56546948db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.784+0000 7f7a93e00640 1 -- 172.21.15.31:0/37838221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565469482bb0 msgr2=0x56546948b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:01.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.784+0000 7f7a93e00640 1 --2- 172.21.15.31:0/37838221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565469482bb0 0x56546948b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.784+0000 7f7a93e00640 1 -- 172.21.15.31:0/37838221 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654691c37e0 con 0x565469482290 2024-03-21T10:47:01.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.785+0000 7f7a91600640 1 --2- 172.21.15.31:0/4282155358 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7a7c0a3110 0x7f7a7c0a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:01.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.785+0000 7f7a93e00640 1 --2- 172.21.15.31:0/37838221 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565469482290 0x565469482670 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f7a800058c0 tx=0x7f7a80005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee3bc6b573fa1778 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:01.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.785+0000 7f7a91600640 1 -- 172.21.15.31:0/4282155358 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f7a8c0d0010 con 0x7f7a94109670 2024-03-21T10:47:01.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.785+0000 7f7a99000640 1 --2- 172.21.15.31:0/4282155358 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7a7c0a3110 0x7f7a7c0a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:01.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.785+0000 7f7a86000640 1 -- 172.21.15.31:0/37838221 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80004030 con 0x565469482290 2024-03-21T10:47:01.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.785+0000 7f7a86000640 1 -- 172.21.15.31:0/37838221 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a80005500 con 0x565469482290 2024-03-21T10:47:01.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.785+0000 7f7a86000640 1 -- 172.21.15.31:0/37838221 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80004d70 con 0x565469482290 2024-03-21T10:47:01.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.786+0000 7f7a99000640 1 --2- 172.21.15.31:0/4282155358 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7a7c0a3110 0x7f7a7c0a55d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7a88009900 tx=0x7f7a88005fd0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:01.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.786+0000 7f7a9e837740 1 -- 172.21.15.31:0/37838221 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565469482290 msgr2=0x565469482670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:01.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.786+0000 7f7a9e837740 1 --2- 172.21.15.31:0/37838221 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565469482290 0x565469482670 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f7a800058c0 tx=0x7f7a80005850 comp rx=0 tx=0).stop 2024-03-21T10:47:01.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.786+0000 7f7a9e837740 1 -- 172.21.15.31:0/37838221 shutdown_connections 2024-03-21T10:47:01.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.786+0000 7f7a9e837740 1 --2- 172.21.15.31:0/37838221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56546948b790 0x56546948db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.786+0000 7f7a9e837740 1 --2- 172.21.15.31:0/37838221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565469482bb0 0x56546948b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.786+0000 7f7a9e837740 1 --2- 172.21.15.31:0/37838221 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565469482290 0x565469482670 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.786+0000 7f7a9e837740 1 -- 172.21.15.31:0/37838221 >> 172.21.15.31:0/37838221 conn(0x56546947c9c0 msgr2=0x56546947cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:01.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.786+0000 7f7a9e837740 1 -- 172.21.15.31:0/37838221 shutdown_connections 2024-03-21T10:47:01.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.787+0000 7f7a9e837740 1 -- 172.21.15.31:0/37838221 wait complete. 2024-03-21T10:47:01.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.787+0000 7f7a9e837740 1 -- start start 2024-03-21T10:47:01.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.787+0000 7f7a9e837740 1 Processor -- start 2024-03-21T10:47:01.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.788+0000 7f7a9e837740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565469482290 0x56546948b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:01.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.788+0000 7f7a9e837740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565469482bb0 0x5654694884e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:01.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.788+0000 7f7a9e837740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56546948b790 0x565469488af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:01.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.788+0000 7f7a9e837740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56546948e580 con 0x565469482290 2024-03-21T10:47:01.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.788+0000 7f7a9e837740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654693a2eb0 con 0x56546948b790 2024-03-21T10:47:01.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.788+0000 7f7a9e837740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56546948e700 con 0x565469482bb0 2024-03-21T10:47:01.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.788+0000 7f7a9e837740 10 client.0 _ll_register_callbacks cb 0x565469397150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:01.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.788+0000 7f7a99a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56546948b790 0x565469488af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:01.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.788+0000 7f7a93e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565469482bb0 0x5654694884e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:01.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.788+0000 7f7a99a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56546948b790 0x565469488af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42660/0 (socket says 172.21.15.31:42660) 2024-03-21T10:47:01.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.788+0000 7f7a99a00640 1 -- 172.21.15.31:0/3562205575 learned_addr learned my addr 172.21.15.31:0/3562205575 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:01.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.788+0000 7f7a99000640 1 --2- 172.21.15.31:0/3562205575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565469482290 0x56546948b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:01.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.788+0000 7f7a99a00640 1 -- 172.21.15.31:0/3562205575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565469482bb0 msgr2=0x5654694884e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:01.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.788+0000 7f7a99a00640 1 --2- 172.21.15.31:0/3562205575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565469482bb0 0x5654694884e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.788+0000 7f7a99a00640 1 -- 172.21.15.31:0/3562205575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565469482290 msgr2=0x56546948b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:01.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.788+0000 7f7a99a00640 1 --2- 172.21.15.31:0/3562205575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565469482290 0x56546948b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.788+0000 7f7a99a00640 1 -- 172.21.15.31:0/3562205575 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565469223f70 con 0x56546948b790 2024-03-21T10:47:01.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.788+0000 7f7a99000640 1 --2- 172.21.15.31:0/3562205575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565469482290 0x56546948b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:47:01.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.789+0000 7f7a99a00640 1 --2- 172.21.15.31:0/3562205575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56546948b790 0x565469488af0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c098a30 tx=0x7f7a8c09ba80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60902e25314bf0fd server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:01.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.789+0000 7f7a7aa00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:47:01.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.789+0000 7f7a7aa00640 1 -- 172.21.15.31:0/3562205575 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c0bc270 con 0x56546948b790 2024-03-21T10:47:01.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.789+0000 7f7a7aa00640 1 -- 172.21.15.31:0/3562205575 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f7a8c096bd0 con 0x56546948b790 2024-03-21T10:47:01.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.789+0000 7f7a9e837740 1 -- 172.21.15.31:0/3562205575 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7a80005500 con 0x56546948b790 2024-03-21T10:47:01.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.789+0000 7f7a7aa00640 1 -- 172.21.15.31:0/3562205575 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a8c0a01b0 con 0x56546948b790 2024-03-21T10:47:01.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.789+0000 7f7a9e837740 10 client.27728 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:01.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.789+0000 7f7a9e837740 1 -- 172.21.15.31:0/3562205575 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5654690bb170 con 0x56546948b790 2024-03-21T10:47:01.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.790+0000 7f7a7aa00640 1 -- 172.21.15.31:0/3562205575 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a80005500 con 0x56546948b790 2024-03-21T10:47:01.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.790+0000 7f7a7aa00640 1 -- 172.21.15.31:0/3562205575 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f7a8c005310 con 0x56546948b790 2024-03-21T10:47:01.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.790+0000 7f7a9e837740 10 client.27728 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:01.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.790+0000 7f7a9e837740 10 client.27728 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:47:01.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.790+0000 7f7a9e837740 20 client.27728 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:01.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.790+0000 7f7a9e837740 1 --2- 172.21.15.31:0/3562205575 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56546949b830 0x56546949dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:01.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.790+0000 7f7a9e837740 4 client.27728 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:47:01.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.790+0000 7f7a9e837740 1 -- 172.21.15.31:0/3562205575 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5654693f43f0 con 0x56546949b830 2024-03-21T10:47:01.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.790+0000 7f7a99000640 1 --2- 172.21.15.31:0/3562205575 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56546949b830 0x56546949dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:01.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.791+0000 7f7a99000640 1 --2- 172.21.15.31:0/3562205575 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56546949b830 0x56546949dc20 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf290059dd0b8d00 server_cookie=c5a593a55c6d77eb in_seq=0 out_seq=0 2024-03-21T10:47:01.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.791+0000 7f7a7aa00640 10 client.27728 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:01.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.792+0000 7f7a7aa00640 1 -- 172.21.15.31:0/3562205575 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5654693f43f0 con 0x56546949b830 2024-03-21T10:47:01.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.792+0000 7f7a7aa00640 10 client.27728 handle_command_reply: tid=0 2024-03-21T10:47:01.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.792+0000 7f7a9e837740 1 client.27728 shutdown 2024-03-21T10:47:01.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.792+0000 7f7a9e837740 1 -- 172.21.15.31:0/3562205575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56546948b790 msgr2=0x565469488af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:01.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.792+0000 7f7a9e837740 1 --2- 172.21.15.31:0/3562205575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56546948b790 0x565469488af0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c098a30 tx=0x7f7a8c09ba80 comp rx=0 tx=0).stop 2024-03-21T10:47:01.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.792+0000 7f7a9e837740 1 -- 172.21.15.31:0/3562205575 shutdown_connections 2024-03-21T10:47:01.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.792+0000 7f7a9e837740 1 --2- 172.21.15.31:0/3562205575 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56546949b830 0x56546949dc20 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.792+0000 7f7a9e837740 1 --2- 172.21.15.31:0/3562205575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56546948b790 0x565469488af0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.792+0000 7f7a9e837740 1 --2- 172.21.15.31:0/3562205575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565469482bb0 0x5654694884e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.792+0000 7f7a9e837740 1 --2- 172.21.15.31:0/3562205575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565469482290 0x56546948b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.792+0000 7f7a9e837740 1 -- 172.21.15.31:0/3562205575 >> 172.21.15.31:0/3562205575 conn(0x56546947c9c0 msgr2=0x56546947e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:01.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.792+0000 7f7a7aa00640 0 client.27728 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:01.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.792+0000 7f7a9e837740 1 -- 172.21.15.31:0/3562205575 shutdown_connections 2024-03-21T10:47:01.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.793+0000 7f7a9e837740 1 -- 172.21.15.31:0/3562205575 wait complete. 2024-03-21T10:47:01.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.793+0000 7f7a9e837740 20 client.27728 trim_cache size 0 max 16384 2024-03-21T10:47:01.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.827+0000 7f7a9e837740 1 Processor -- start 2024-03-21T10:47:01.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.827+0000 7f7a9e837740 1 -- start start 2024-03-21T10:47:01.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.828+0000 7f7a9e837740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565469482290 0x56546948ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:01.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.828+0000 7f7a9e837740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565469482bb0 0x56546948b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:01.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.828+0000 7f7a9e837740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56546948b790 0x5654694837e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:01.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.828+0000 7f7a9e837740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a941b2f70 con 0x56546948b790 2024-03-21T10:47:01.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.828+0000 7f7a9e837740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a941b0a70 con 0x565469482bb0 2024-03-21T10:47:01.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.828+0000 7f7a9e837740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a941aeb30 con 0x565469482290 2024-03-21T10:47:01.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.828+0000 7f7a99000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565469482290 0x56546948ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:01.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.828+0000 7f7a93e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565469482bb0 0x56546948b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:01.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.828+0000 7f7a99000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565469482290 0x56546948ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53974/0 (socket says 172.21.15.31:53974) 2024-03-21T10:47:01.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.828+0000 7f7a99000640 1 -- 172.21.15.31:0/3531202056 learned_addr learned my addr 172.21.15.31:0/3531202056 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:01.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.828+0000 7f7a93e00640 1 --2- 172.21.15.31:0/3531202056 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565469482bb0 0x56546948b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42668/0 (socket says 172.21.15.31:42668) 2024-03-21T10:47:01.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.828+0000 7f7a99a00640 1 --2- 172.21.15.31:0/3531202056 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56546948b790 0x5654694837e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:01.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.828+0000 7f7a99000640 1 -- 172.21.15.31:0/3531202056 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565469482bb0 msgr2=0x56546948b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:01.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.828+0000 7f7a99000640 1 --2- 172.21.15.31:0/3531202056 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565469482bb0 0x56546948b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.828+0000 7f7a99000640 1 -- 172.21.15.31:0/3531202056 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56546948b790 msgr2=0x5654694837e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:01.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.828+0000 7f7a99000640 1 --2- 172.21.15.31:0/3531202056 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56546948b790 0x5654694837e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.828+0000 7f7a99000640 1 -- 172.21.15.31:0/3531202056 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565469476c40 con 0x565469482290 2024-03-21T10:47:01.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.829+0000 7f7a99000640 1 --2- 172.21.15.31:0/3531202056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565469482290 0x56546948ac10 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f7a880096c0 tx=0x7f7a8803f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=474ef107022990c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:01.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.829+0000 7f7a85600640 1 -- 172.21.15.31:0/3531202056 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8802c880 con 0x565469482290 2024-03-21T10:47:01.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.829+0000 7f7a85600640 1 -- 172.21.15.31:0/3531202056 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a88046040 con 0x565469482290 2024-03-21T10:47:01.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.829+0000 7f7a85600640 1 -- 172.21.15.31:0/3531202056 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a88005060 con 0x565469482290 2024-03-21T10:47:01.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.830+0000 7f7a9e837740 1 -- 172.21.15.31:0/3531202056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565469482290 msgr2=0x56546948ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:01.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.830+0000 7f7a9e837740 1 --2- 172.21.15.31:0/3531202056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565469482290 0x56546948ac10 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f7a880096c0 tx=0x7f7a8803f000 comp rx=0 tx=0).stop 2024-03-21T10:47:01.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.830+0000 7f7a9e837740 1 -- 172.21.15.31:0/3531202056 shutdown_connections 2024-03-21T10:47:01.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.830+0000 7f7a9e837740 1 --2- 172.21.15.31:0/3531202056 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56546948b790 0x5654694837e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.830+0000 7f7a9e837740 1 --2- 172.21.15.31:0/3531202056 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565469482bb0 0x56546948b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.830+0000 7f7a9e837740 1 --2- 172.21.15.31:0/3531202056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565469482290 0x56546948ac10 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.830+0000 7f7a9e837740 1 -- 172.21.15.31:0/3531202056 >> 172.21.15.31:0/3531202056 conn(0x56546947c9c0 msgr2=0x5654694830d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:01.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.830+0000 7f7a9e837740 1 -- 172.21.15.31:0/3531202056 shutdown_connections 2024-03-21T10:47:01.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.830+0000 7f7a9e837740 1 -- 172.21.15.31:0/3531202056 wait complete. 2024-03-21T10:47:01.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.831+0000 7f7a9e837740 1 -- start start 2024-03-21T10:47:01.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.831+0000 7f7a9e837740 1 Processor -- start 2024-03-21T10:47:01.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.831+0000 7f7a9e837740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565469482290 0x565469490920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:01.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.831+0000 7f7a9e837740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565469482bb0 0x565469490e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:01.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.831+0000 7f7a9e837740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56546948b790 0x5654694913a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:01.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.831+0000 7f7a9e837740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a941af130 con 0x565469482290 2024-03-21T10:47:01.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.831+0000 7f7a9e837740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a941b0410 con 0x56546948b790 2024-03-21T10:47:01.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.831+0000 7f7a9e837740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a941af750 con 0x565469482bb0 2024-03-21T10:47:01.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.831+0000 7f7a9e837740 10 client.0 _ll_register_callbacks cb 0x7f7a8c09bea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:01.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.831+0000 7f7a93e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565469482bb0 0x565469490e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:01.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.832+0000 7f7a93e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565469482bb0 0x565469490e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53984/0 (socket says 172.21.15.31:53984) 2024-03-21T10:47:01.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.832+0000 7f7a93e00640 1 -- 172.21.15.31:0/364983348 learned_addr learned my addr 172.21.15.31:0/364983348 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:01.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.832+0000 7f7a99000640 1 --2- 172.21.15.31:0/364983348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565469482290 0x565469490920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:01.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.832+0000 7f7a99a00640 1 --2- 172.21.15.31:0/364983348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56546948b790 0x5654694913a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:01.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.832+0000 7f7a93e00640 1 -- 172.21.15.31:0/364983348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56546948b790 msgr2=0x5654694913a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:01.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.832+0000 7f7a93e00640 1 --2- 172.21.15.31:0/364983348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56546948b790 0x5654694913a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.832+0000 7f7a93e00640 1 -- 172.21.15.31:0/364983348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565469482290 msgr2=0x565469490920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:01.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.832+0000 7f7a93e00640 1 --2- 172.21.15.31:0/364983348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565469482290 0x565469490920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.832+0000 7f7a93e00640 1 -- 172.21.15.31:0/364983348 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5654694f70c0 con 0x565469482bb0 2024-03-21T10:47:01.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.832+0000 7f7a99000640 1 --2- 172.21.15.31:0/364983348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565469482290 0x565469490920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:47:01.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.832+0000 7f7a93e00640 1 --2- 172.21.15.31:0/364983348 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565469482bb0 0x565469490e60 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f7a800058c0 tx=0x7f7a800048b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0bedcbc8223aa99 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:01.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.832+0000 7f7a7aa00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:47:01.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.833+0000 7f7a7aa00640 1 -- 172.21.15.31:0/364983348 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80004590 con 0x565469482bb0 2024-03-21T10:47:01.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.833+0000 7f7a9e837740 1 -- 172.21.15.31:0/364983348 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7a88046040 con 0x565469482bb0 2024-03-21T10:47:01.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.833+0000 7f7a7aa00640 1 -- 172.21.15.31:0/364983348 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f7a80019040 con 0x565469482bb0 2024-03-21T10:47:01.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.833+0000 7f7a7aa00640 1 -- 172.21.15.31:0/364983348 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a80034050 con 0x565469482bb0 2024-03-21T10:47:01.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.833+0000 7f7a9e837740 10 client.27655 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:01.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.833+0000 7f7a9e837740 1 -- 172.21.15.31:0/364983348 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5654694f8330 con 0x565469482bb0 2024-03-21T10:47:01.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.833+0000 7f7a7aa00640 1 -- 172.21.15.31:0/364983348 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a88046040 con 0x565469482bb0 2024-03-21T10:47:01.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.833+0000 7f7a7aa00640 1 -- 172.21.15.31:0/364983348 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f7a800037b0 con 0x565469482bb0 2024-03-21T10:47:01.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.833+0000 7f7a9e837740 10 client.27655 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:01.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.833+0000 7f7a9e837740 10 client.27655 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:47:01.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.833+0000 7f7a9e837740 20 client.27655 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:01.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.833+0000 7f7a9e837740 1 --2- 172.21.15.31:0/364983348 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5654694f88c0 0x5654694face0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:01.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.833+0000 7f7a9e837740 4 client.27655 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-21T10:47:01.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.833+0000 7f7a9e837740 1 -- 172.21.15.31:0/364983348 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5654693ee740 con 0x5654694f88c0 2024-03-21T10:47:01.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.834+0000 7f7a99000640 1 --2- 172.21.15.31:0/364983348 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5654694f88c0 0x5654694face0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:01.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.834+0000 7f7a7aa00640 10 client.27655 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:01.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.834+0000 7f7a99000640 1 --2- 172.21.15.31:0/364983348 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5654694f88c0 0x5654694face0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f2ce48afbef9fff server_cookie=833b3b60be83dd8c in_seq=0 out_seq=0 2024-03-21T10:47:01.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.835+0000 7f7a7aa00640 1 -- 172.21.15.31:0/364983348 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5654693ee740 con 0x5654694f88c0 2024-03-21T10:47:01.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.835+0000 7f7a7aa00640 10 client.27655 handle_command_reply: tid=0 2024-03-21T10:47:01.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.836+0000 7f7a9e837740 1 client.27655 shutdown 2024-03-21T10:47:01.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.836+0000 7f7a9e837740 1 -- 172.21.15.31:0/364983348 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565469482bb0 msgr2=0x565469490e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:01.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.836+0000 7f7a9e837740 1 --2- 172.21.15.31:0/364983348 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565469482bb0 0x565469490e60 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f7a800058c0 tx=0x7f7a800048b0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.836+0000 7f7a9e837740 1 -- 172.21.15.31:0/364983348 shutdown_connections 2024-03-21T10:47:01.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.836+0000 7f7a9e837740 1 --2- 172.21.15.31:0/364983348 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5654694f88c0 0x5654694face0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.836+0000 7f7a9e837740 1 --2- 172.21.15.31:0/364983348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56546948b790 0x5654694913a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.836+0000 7f7a9e837740 1 --2- 172.21.15.31:0/364983348 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565469482bb0 0x565469490e60 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.836+0000 7f7a9e837740 1 --2- 172.21.15.31:0/364983348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565469482290 0x565469490920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.836+0000 7f7a9e837740 1 -- 172.21.15.31:0/364983348 >> 172.21.15.31:0/364983348 conn(0x56546947c9c0 msgr2=0x56546947ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:01.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.836+0000 7f7a7aa00640 0 client.27655 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:01.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.836+0000 7f7a9e837740 1 -- 172.21.15.31:0/364983348 shutdown_connections 2024-03-21T10:47:01.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.836+0000 7f7a9e837740 1 -- 172.21.15.31:0/364983348 wait complete. 2024-03-21T10:47:01.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.836+0000 7f7a9e837740 20 client.27655 trim_cache size 0 max 16384 2024-03-21T10:47:01.838 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:47:01.838 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-03-21T10:47:01.838 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "ba166823-0eba-4caa-832f-34ab7724b69e", 2024-03-21T10:47:01.838 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-03-21T10:47:01.838 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:47:01.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.837+0000 7f7a9a400640 1 -- 172.21.15.31:0/4282155358 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7a7c0a3110 msgr2=0x7f7a7c0a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:01.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.837+0000 7f7a9a400640 1 --2- 172.21.15.31:0/4282155358 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7a7c0a3110 0x7f7a7c0a55d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7a88009900 tx=0x7f7a88005fd0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.837+0000 7f7a9a400640 1 -- 172.21.15.31:0/4282155358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a94109670 msgr2=0x7f7a9419e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:01.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.837+0000 7f7a9a400640 1 --2- 172.21.15.31:0/4282155358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a94109670 0x7f7a9419e8b0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c00eb10 tx=0x7f7a8c00c2d0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.837+0000 7f7a9a400640 1 -- 172.21.15.31:0/4282155358 shutdown_connections 2024-03-21T10:47:01.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.837+0000 7f7a9a400640 1 --2- 172.21.15.31:0/4282155358 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7a7c0a3110 0x7f7a7c0a55d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.837+0000 7f7a9a400640 1 --2- 172.21.15.31:0/4282155358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a94109670 0x7f7a9419e8b0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.837+0000 7f7a9a400640 1 --2- 172.21.15.31:0/4282155358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a94104020 0x7f7a9419a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.837+0000 7f7a9a400640 1 --2- 172.21.15.31:0/4282155358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a94103660 0x7f7a94199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:01.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.837+0000 7f7a9a400640 1 -- 172.21.15.31:0/4282155358 >> 172.21.15.31:0/4282155358 conn(0x7f7a940ff150 msgr2=0x7f7a940ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:01.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.837+0000 7f7a9a400640 1 -- 172.21.15.31:0/4282155358 shutdown_connections 2024-03-21T10:47:01.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:01.837+0000 7f7a9a400640 1 -- 172.21.15.31:0/4282155358 wait complete. 2024-03-21T10:47:01.852 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:47:01.852 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:47:02.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.041+0000 7f36cc400640 1 -- 172.21.15.31:0/2111981810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36c4100440 msgr2=0x7f36c4100820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:02.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.041+0000 7f36cc400640 1 --2- 172.21.15.31:0/2111981810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36c4100440 0x7f36c4100820 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f36b8009900 tx=0x7f36b802bc30 comp rx=0 tx=0).stop 2024-03-21T10:47:02.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.042+0000 7f36cc400640 1 -- 172.21.15.31:0/2111981810 shutdown_connections 2024-03-21T10:47:02.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.042+0000 7f36cc400640 1 --2- 172.21.15.31:0/2111981810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36c410dff0 0x7f36c41103e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.042+0000 7f36cc400640 1 --2- 172.21.15.31:0/2111981810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36c4100d60 0x7f36c410dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.042+0000 7f36cc400640 1 --2- 172.21.15.31:0/2111981810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36c4100440 0x7f36c4100820 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.042+0000 7f36cc400640 1 -- 172.21.15.31:0/2111981810 >> 172.21.15.31:0/2111981810 conn(0x7f36c40776c0 msgr2=0x7f36c4077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:02.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.042+0000 7f36cc400640 1 -- 172.21.15.31:0/2111981810 shutdown_connections 2024-03-21T10:47:02.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.042+0000 7f36cc400640 1 -- 172.21.15.31:0/2111981810 wait complete. 2024-03-21T10:47:02.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.043+0000 7f36cc400640 1 Processor -- start 2024-03-21T10:47:02.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.043+0000 7f36cc400640 1 -- start start 2024-03-21T10:47:02.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.043+0000 7f36cc400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36c4100440 0x7f36c419a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:02.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.043+0000 7f36cc400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36c4100d60 0x7f36c419a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:02.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.044+0000 7f36cc400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36c410dff0 0x7f36c419e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:02.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.044+0000 7f36cc400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c4113ca0 con 0x7f36c4100440 2024-03-21T10:47:02.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.044+0000 7f36cc400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c4113b20 con 0x7f36c410dff0 2024-03-21T10:47:02.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.044+0000 7f36cc400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c4113e20 con 0x7f36c4100d60 2024-03-21T10:47:02.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.044+0000 7f36c9a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36c4100440 0x7f36c419a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:02.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.044+0000 7f36c9a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36c4100440 0x7f36c419a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42292/0 (socket says 172.21.15.31:42292) 2024-03-21T10:47:02.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.044+0000 7f36c9a00640 1 -- 172.21.15.31:0/390518704 learned_addr learned my addr 172.21.15.31:0/390518704 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:02.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.044+0000 7f36c9000640 1 --2- 172.21.15.31:0/390518704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36c4100d60 0x7f36c419a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:02.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.044+0000 7f36ca400640 1 --2- 172.21.15.31:0/390518704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36c410dff0 0x7f36c419e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:02.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.044+0000 7f36c9a00640 1 -- 172.21.15.31:0/390518704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36c4100d60 msgr2=0x7f36c419a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:02.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.044+0000 7f36c9a00640 1 --2- 172.21.15.31:0/390518704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36c4100d60 0x7f36c419a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.044+0000 7f36c9a00640 1 -- 172.21.15.31:0/390518704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36c410dff0 msgr2=0x7f36c419e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:02.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.044+0000 7f36c9a00640 1 --2- 172.21.15.31:0/390518704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36c410dff0 0x7f36c419e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.044+0000 7f36c9a00640 1 -- 172.21.15.31:0/390518704 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c419f0e0 con 0x7f36c4100440 2024-03-21T10:47:02.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.044+0000 7f36c9000640 1 --2- 172.21.15.31:0/390518704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36c4100d60 0x7f36c419a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:02.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.045+0000 7f36c9a00640 1 --2- 172.21.15.31:0/390518704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36c4100440 0x7f36c419a140 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f36b802c240 tx=0x7f36b8004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:02.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.045+0000 7f36c2000640 1 -- 172.21.15.31:0/390518704 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b8043020 con 0x7f36c4100440 2024-03-21T10:47:02.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.045+0000 7f36cc400640 1 -- 172.21.15.31:0/390518704 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c419f370 con 0x7f36c4100440 2024-03-21T10:47:02.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.045+0000 7f36c2000640 1 -- 172.21.15.31:0/390518704 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36b80054e0 con 0x7f36c4100440 2024-03-21T10:47:02.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.045+0000 7f36c2000640 1 -- 172.21.15.31:0/390518704 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b8003c00 con 0x7f36c4100440 2024-03-21T10:47:02.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.045+0000 7f36cc400640 1 -- 172.21.15.31:0/390518704 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c41a6c60 con 0x7f36c4100440 2024-03-21T10:47:02.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.049+0000 7f36cf1bf740 1 Processor -- start 2024-03-21T10:47:02.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.049+0000 7f36cf1bf740 1 -- start start 2024-03-21T10:47:02.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.050+0000 7f36cf1bf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c476b17c00 0x55c476b16950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:02.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.050+0000 7f36cf1bf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c476b16f20 0x55c476b173a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:02.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.050+0000 7f36cf1bf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c476b20290 0x55c476b22680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:02.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.050+0000 7f36cf1bf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c476a7b250 con 0x55c476b20290 2024-03-21T10:47:02.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.050+0000 7f36cf1bf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c476a23a30 con 0x55c476b16f20 2024-03-21T10:47:02.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.050+0000 7f36cf1bf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c476a772f0 con 0x55c476b17c00 2024-03-21T10:47:02.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.050+0000 7f36ca400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c476b17c00 0x55c476b16950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:02.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.050+0000 7f36ca400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c476b17c00 0x55c476b16950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:54012/0 (socket says 172.21.15.31:54012) 2024-03-21T10:47:02.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.050+0000 7f36c2000640 1 -- 172.21.15.31:0/390518704 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f36b8046020 con 0x7f36c4100440 2024-03-21T10:47:02.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.050+0000 7f36ca400640 1 -- 172.21.15.31:0/3415326912 learned_addr learned my addr 172.21.15.31:0/3415326912 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:02.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.050+0000 7f36c9000640 1 --2- 172.21.15.31:0/3415326912 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c476b20290 0x55c476b22680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:02.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.050+0000 7f36ca400640 1 -- 172.21.15.31:0/3415326912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c476b16f20 msgr2=0x55c476b173a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:02.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.050+0000 7f36c9a00640 1 --2- 172.21.15.31:0/3415326912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c476b16f20 0x55c476b173a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:02.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.050+0000 7f36ca400640 1 --2- 172.21.15.31:0/3415326912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c476b16f20 0x55c476b173a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.050+0000 7f36ca400640 1 -- 172.21.15.31:0/3415326912 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c476b20290 msgr2=0x55c476b22680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:02.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.050+0000 7f36ca400640 1 --2- 172.21.15.31:0/3415326912 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c476b20290 0x55c476b22680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.050+0000 7f36ca400640 1 -- 172.21.15.31:0/3415326912 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c476a0e690 con 0x55c476b17c00 2024-03-21T10:47:02.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.050+0000 7f36c9000640 1 --2- 172.21.15.31:0/3415326912 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c476b20290 0x55c476b22680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:47:02.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.050+0000 7f36c9a00640 1 --2- 172.21.15.31:0/3415326912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c476b16f20 0x55c476b173a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:47:02.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.051+0000 7f36ca400640 1 --2- 172.21.15.31:0/3415326912 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c476b17c00 0x55c476b16950 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f36bc009e00 tx=0x7f36bc010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=151844ab8427f35d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:02.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.051+0000 7f36b6a00640 1 -- 172.21.15.31:0/3415326912 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36bc004d20 con 0x55c476b17c00 2024-03-21T10:47:02.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.051+0000 7f36b6a00640 1 -- 172.21.15.31:0/3415326912 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36bc005500 con 0x55c476b17c00 2024-03-21T10:47:02.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.051+0000 7f36b6a00640 1 -- 172.21.15.31:0/3415326912 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36bc022a20 con 0x55c476b17c00 2024-03-21T10:47:02.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.052+0000 7f36cf1bf740 1 -- 172.21.15.31:0/3415326912 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c476b17c00 msgr2=0x55c476b16950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:02.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.052+0000 7f36cf1bf740 1 --2- 172.21.15.31:0/3415326912 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c476b17c00 0x55c476b16950 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f36bc009e00 tx=0x7f36bc010740 comp rx=0 tx=0).stop 2024-03-21T10:47:02.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.052+0000 7f36cf1bf740 1 -- 172.21.15.31:0/3415326912 shutdown_connections 2024-03-21T10:47:02.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.052+0000 7f36cf1bf740 1 --2- 172.21.15.31:0/3415326912 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c476b20290 0x55c476b22680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.052+0000 7f36cf1bf740 1 --2- 172.21.15.31:0/3415326912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c476b16f20 0x55c476b173a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.052+0000 7f36cf1bf740 1 --2- 172.21.15.31:0/3415326912 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c476b17c00 0x55c476b16950 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.052+0000 7f36cf1bf740 1 -- 172.21.15.31:0/3415326912 >> 172.21.15.31:0/3415326912 conn(0x55c476ade200 msgr2=0x55c476b11970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:02.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.052+0000 7f36cf1bf740 1 -- 172.21.15.31:0/3415326912 shutdown_connections 2024-03-21T10:47:02.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.052+0000 7f36cf1bf740 1 -- 172.21.15.31:0/3415326912 wait complete. 2024-03-21T10:47:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.052+0000 7f36cf1bf740 1 -- start start 2024-03-21T10:47:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.053+0000 7f36cf1bf740 1 Processor -- start 2024-03-21T10:47:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.053+0000 7f36c2000640 1 --2- 172.21.15.31:0/390518704 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f36ac0a2fa0 0x7f36ac0a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.053+0000 7f36c2000640 1 -- 172.21.15.31:0/390518704 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f36b80f1ba0 con 0x7f36c4100440 2024-03-21T10:47:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.053+0000 7f36ca400640 1 --2- 172.21.15.31:0/390518704 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f36ac0a2fa0 0x7f36ac0a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.053+0000 7f36cf1bf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c476b16f20 0x55c476b1f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.053+0000 7f36cf1bf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c476b17c00 0x55c476b1cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.053+0000 7f36cf1bf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c476b20290 0x55c476b1d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.053+0000 7f36cf1bf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c476b23100 con 0x55c476b17c00 2024-03-21T10:47:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.053+0000 7f36cf1bf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c476a36e80 con 0x55c476b16f20 2024-03-21T10:47:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.053+0000 7f36cf1bf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c476b23280 con 0x55c476b20290 2024-03-21T10:47:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.053+0000 7f36cf1bf740 10 client.0 _ll_register_callbacks cb 0x55c476a49130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.053+0000 7f36c9a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c476b16f20 0x55c476b1f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.053+0000 7f36c9a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c476b16f20 0x55c476b1f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42730/0 (socket says 172.21.15.31:42730) 2024-03-21T10:47:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.053+0000 7f36c9a00640 1 -- 172.21.15.31:0/926573379 learned_addr learned my addr 172.21.15.31:0/926573379 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.054+0000 7f36c9000640 1 --2- 172.21.15.31:0/926573379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c476b17c00 0x55c476b1cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.054+0000 7f36c9a00640 1 -- 172.21.15.31:0/926573379 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c476b20290 msgr2=0x55c476b1d2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:47:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.054+0000 7f36c9a00640 1 --2- 172.21.15.31:0/926573379 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c476b20290 0x55c476b1d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.054+0000 7f36c9a00640 1 -- 172.21.15.31:0/926573379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c476b17c00 msgr2=0x55c476b1cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.054+0000 7f36c9a00640 1 --2- 172.21.15.31:0/926573379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c476b17c00 0x55c476b1cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.054+0000 7f36c9a00640 1 -- 172.21.15.31:0/926573379 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c4768597d0 con 0x55c476b16f20 2024-03-21T10:47:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.054+0000 7f36ca400640 1 --2- 172.21.15.31:0/390518704 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f36ac0a2fa0 0x7f36ac0a5460 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f36bc0226c0 tx=0x7f36bc01b040 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.054+0000 7f36c9a00640 1 --2- 172.21.15.31:0/926573379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c476b16f20 0x55c476b1f9d0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f36b8000c00 tx=0x7f36b80b9fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b43fab76af4263a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.054+0000 7f36ab400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:47:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.054+0000 7f36ab400640 1 -- 172.21.15.31:0/926573379 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b80bd660 con 0x55c476b16f20 2024-03-21T10:47:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.054+0000 7f36cf1bf740 1 -- 172.21.15.31:0/926573379 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f36bc005500 con 0x55c476b16f20 2024-03-21T10:47:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.054+0000 7f36ab400640 1 -- 172.21.15.31:0/926573379 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f36b80ef1d0 con 0x55c476b16f20 2024-03-21T10:47:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.055+0000 7f36ab400640 1 -- 172.21.15.31:0/926573379 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36b80ef3d0 con 0x55c476b16f20 2024-03-21T10:47:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.055+0000 7f36cf1bf740 10 client.27743 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.055+0000 7f36cf1bf740 1 -- 172.21.15.31:0/926573379 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c47678f3f0 con 0x55c476b16f20 2024-03-21T10:47:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.055+0000 7f36ab400640 1 -- 172.21.15.31:0/926573379 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36bc005500 con 0x55c476b16f20 2024-03-21T10:47:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.055+0000 7f36ab400640 1 -- 172.21.15.31:0/926573379 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f36b80c6280 con 0x55c476b16f20 2024-03-21T10:47:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.055+0000 7f36cf1bf740 10 client.27743 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.055+0000 7f36cf1bf740 10 client.27743 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:47:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.055+0000 7f36cf1bf740 20 client.27743 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.055+0000 7f36cf1bf740 1 --2- 172.21.15.31:0/926573379 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c476b30070 0x55c476b32460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.055+0000 7f36cf1bf740 4 client.27743 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:47:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.055+0000 7f36cf1bf740 1 -- 172.21.15.31:0/926573379 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c476a88790 con 0x55c476b30070 2024-03-21T10:47:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.055+0000 7f36c9000640 1 --2- 172.21.15.31:0/926573379 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c476b30070 0x55c476b32460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:02.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.056+0000 7f36c9000640 1 --2- 172.21.15.31:0/926573379 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c476b30070 0x55c476b32460 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b68887fea111537d server_cookie=e0c1442dddc4b654 in_seq=0 out_seq=0 2024-03-21T10:47:02.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.056+0000 7f36ab400640 10 client.27743 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:02.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.056+0000 7f36ab400640 1 -- 172.21.15.31:0/926573379 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c476a88790 con 0x55c476b30070 2024-03-21T10:47:02.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.057+0000 7f36ab400640 10 client.27743 handle_command_reply: tid=0 2024-03-21T10:47:02.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.057+0000 7f36cf1bf740 1 client.27743 shutdown 2024-03-21T10:47:02.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.057+0000 7f36cf1bf740 1 -- 172.21.15.31:0/926573379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c476b16f20 msgr2=0x55c476b1f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:02.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.057+0000 7f36cf1bf740 1 --2- 172.21.15.31:0/926573379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c476b16f20 0x55c476b1f9d0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f36b8000c00 tx=0x7f36b80b9fa0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.057+0000 7f36cf1bf740 1 -- 172.21.15.31:0/926573379 shutdown_connections 2024-03-21T10:47:02.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.057+0000 7f36cf1bf740 1 --2- 172.21.15.31:0/926573379 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c476b30070 0x55c476b32460 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.057+0000 7f36cf1bf740 1 --2- 172.21.15.31:0/926573379 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c476b20290 0x55c476b1d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.057+0000 7f36cf1bf740 1 --2- 172.21.15.31:0/926573379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c476b17c00 0x55c476b1cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.057+0000 7f36ab400640 0 client.27743 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:02.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.057+0000 7f36cf1bf740 1 --2- 172.21.15.31:0/926573379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c476b16f20 0x55c476b1f9d0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.057+0000 7f36cf1bf740 1 -- 172.21.15.31:0/926573379 >> 172.21.15.31:0/926573379 conn(0x55c476ade200 msgr2=0x55c476b136c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:02.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.057+0000 7f36cf1bf740 1 -- 172.21.15.31:0/926573379 shutdown_connections 2024-03-21T10:47:02.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.057+0000 7f36cf1bf740 1 -- 172.21.15.31:0/926573379 wait complete. 2024-03-21T10:47:02.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.057+0000 7f36cf1bf740 20 client.27743 trim_cache size 0 max 16384 2024-03-21T10:47:02.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.076+0000 7f36cf1bf740 1 Processor -- start 2024-03-21T10:47:02.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.076+0000 7f36cf1bf740 1 -- start start 2024-03-21T10:47:02.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.076+0000 7f36cf1bf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c476b16f20 0x55c476b1f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:02.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.076+0000 7f36cf1bf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c476b17c00 0x55c476b1f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:02.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.076+0000 7f36cf1bf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c476b20290 0x55c476b2b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:02.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.076+0000 7f36cf1bf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c41aeaf0 con 0x55c476b20290 2024-03-21T10:47:02.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.076+0000 7f36cf1bf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c41ac5f0 con 0x55c476b16f20 2024-03-21T10:47:02.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.076+0000 7f36cf1bf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c41aa6b0 con 0x55c476b17c00 2024-03-21T10:47:02.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.076+0000 7f36c9a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c476b16f20 0x55c476b1f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:02.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.076+0000 7f36c9a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c476b16f20 0x55c476b1f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42746/0 (socket says 172.21.15.31:42746) 2024-03-21T10:47:02.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.076+0000 7f36c9000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c476b17c00 0x55c476b1f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:02.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.076+0000 7f36c9a00640 1 -- 172.21.15.31:0/617183325 learned_addr learned my addr 172.21.15.31:0/617183325 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:02.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.077+0000 7f36c9a00640 1 -- 172.21.15.31:0/617183325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c476b17c00 msgr2=0x55c476b1f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:02.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.077+0000 7f36c9a00640 1 --2- 172.21.15.31:0/617183325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c476b17c00 0x55c476b1f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.077+0000 7f36c9a00640 1 -- 172.21.15.31:0/617183325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c476b20290 msgr2=0x55c476b2b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:02.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.077+0000 7f36c9a00640 1 --2- 172.21.15.31:0/617183325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c476b20290 0x55c476b2b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.077+0000 7f36c9a00640 1 -- 172.21.15.31:0/617183325 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c47674f0f0 con 0x55c476b16f20 2024-03-21T10:47:02.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.077+0000 7f36c9a00640 1 --2- 172.21.15.31:0/617183325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c476b16f20 0x55c476b1f490 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f36b80ba6d0 tx=0x7f36b8004ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bbc784816084cf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:02.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.077+0000 7f36b6000640 1 -- 172.21.15.31:0/617183325 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b80db070 con 0x55c476b16f20 2024-03-21T10:47:02.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.077+0000 7f36b6000640 1 -- 172.21.15.31:0/617183325 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36b8005690 con 0x55c476b16f20 2024-03-21T10:47:02.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.077+0000 7f36b6000640 1 -- 172.21.15.31:0/617183325 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b80c1e50 con 0x55c476b16f20 2024-03-21T10:47:02.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.078+0000 7f36cf1bf740 1 -- 172.21.15.31:0/617183325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c476b16f20 msgr2=0x55c476b1f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:02.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.078+0000 7f36cf1bf740 1 --2- 172.21.15.31:0/617183325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c476b16f20 0x55c476b1f490 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f36b80ba6d0 tx=0x7f36b8004ee0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.078+0000 7f36cf1bf740 1 -- 172.21.15.31:0/617183325 shutdown_connections 2024-03-21T10:47:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.078+0000 7f36cf1bf740 1 --2- 172.21.15.31:0/617183325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c476b20290 0x55c476b2b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.078+0000 7f36cf1bf740 1 --2- 172.21.15.31:0/617183325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c476b17c00 0x55c476b1f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.078+0000 7f36cf1bf740 1 --2- 172.21.15.31:0/617183325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c476b16f20 0x55c476b1f490 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.078+0000 7f36cf1bf740 1 -- 172.21.15.31:0/617183325 >> 172.21.15.31:0/617183325 conn(0x55c476ade200 msgr2=0x55c476b114a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.078+0000 7f36cf1bf740 1 -- 172.21.15.31:0/617183325 shutdown_connections 2024-03-21T10:47:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.078+0000 7f36cf1bf740 1 -- 172.21.15.31:0/617183325 wait complete. 2024-03-21T10:47:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.079+0000 7f36cf1bf740 1 -- start start 2024-03-21T10:47:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.079+0000 7f36cf1bf740 1 Processor -- start 2024-03-21T10:47:02.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.080+0000 7f36cf1bf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c476b16f20 0x55c476b1a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:02.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.080+0000 7f36cf1bf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c476b17c00 0x55c476b1a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:02.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.080+0000 7f36cf1bf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c476b20290 0x55c476b20bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:02.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.080+0000 7f36cf1bf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c41aacb0 con 0x55c476b20290 2024-03-21T10:47:02.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.080+0000 7f36cf1bf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c41abf90 con 0x55c476b16f20 2024-03-21T10:47:02.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.080+0000 7f36cf1bf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c41ab2d0 con 0x55c476b17c00 2024-03-21T10:47:02.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.080+0000 7f36cf1bf740 10 client.0 _ll_register_callbacks cb 0x7f36b8005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.080+0000 7f36c9a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c476b16f20 0x55c476b1a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.080+0000 7f36c9000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c476b17c00 0x55c476b1a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.080+0000 7f36c9a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c476b16f20 0x55c476b1a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42752/0 (socket says 172.21.15.31:42752) 2024-03-21T10:47:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.080+0000 7f36c9a00640 1 -- 172.21.15.31:0/1302223293 learned_addr learned my addr 172.21.15.31:0/1302223293 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.080+0000 7f36ca400640 1 --2- 172.21.15.31:0/1302223293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c476b20290 0x55c476b20bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.080+0000 7f36c9000640 1 -- 172.21.15.31:0/1302223293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c476b16f20 msgr2=0x55c476b1a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.080+0000 7f36c9000640 1 --2- 172.21.15.31:0/1302223293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c476b16f20 0x55c476b1a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.080+0000 7f36c9000640 1 -- 172.21.15.31:0/1302223293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c476b20290 msgr2=0x55c476b20bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.080+0000 7f36c9000640 1 --2- 172.21.15.31:0/1302223293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c476b20290 0x55c476b20bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.080+0000 7f36c9000640 1 -- 172.21.15.31:0/1302223293 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c476b1adb0 con 0x55c476b17c00 2024-03-21T10:47:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.081+0000 7f36c9000640 1 --2- 172.21.15.31:0/1302223293 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c476b17c00 0x55c476b1a5d0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x55c476b32cb0 tx=0x7f36b0004b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35d0d8a56d3c0ad7 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.081+0000 7f36ab400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:47:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.081+0000 7f36ab400640 1 -- 172.21.15.31:0/1302223293 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b0012230 con 0x55c476b17c00 2024-03-21T10:47:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.081+0000 7f36ab400640 1 -- 172.21.15.31:0/1302223293 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f36b001fac0 con 0x55c476b17c00 2024-03-21T10:47:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.081+0000 7f36cf1bf740 1 -- 172.21.15.31:0/1302223293 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c476b187b0 con 0x55c476b17c00 2024-03-21T10:47:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.081+0000 7f36ab400640 1 -- 172.21.15.31:0/1302223293 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36b002f050 con 0x55c476b17c00 2024-03-21T10:47:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.081+0000 7f36cf1bf740 10 client.27670 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.081+0000 7f36cf1bf740 1 -- 172.21.15.31:0/1302223293 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c476b8bf70 con 0x55c476b17c00 2024-03-21T10:47:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.082+0000 7f36ab400640 1 -- 172.21.15.31:0/1302223293 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c476b187b0 con 0x55c476b17c00 2024-03-21T10:47:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.082+0000 7f36ab400640 1 -- 172.21.15.31:0/1302223293 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f36b0011440 con 0x55c476b17c00 2024-03-21T10:47:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.082+0000 7f36cf1bf740 10 client.27670 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.082+0000 7f36cf1bf740 10 client.27670 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:47:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.082+0000 7f36cf1bf740 20 client.27670 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.082+0000 7f36cf1bf740 1 --2- 172.21.15.31:0/1302223293 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c476b8c730 0x55c476b8eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.082+0000 7f36cf1bf740 4 client.27670 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:47:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.082+0000 7f36cf1bf740 1 -- 172.21.15.31:0/1302223293 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c476a82ae0 con 0x55c476b8c730 2024-03-21T10:47:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.082+0000 7f36c9a00640 1 --2- 172.21.15.31:0/1302223293 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c476b8c730 0x55c476b8eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.083+0000 7f36c9a00640 1 --2- 172.21.15.31:0/1302223293 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c476b8c730 0x55c476b8eb50 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0f5b4e8829c9b20 server_cookie=e7ed8a2b9861f31c in_seq=0 out_seq=0 2024-03-21T10:47:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.083+0000 7f36ab400640 10 client.27670 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.083+0000 7f36ab400640 1 -- 172.21.15.31:0/1302223293 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55c476a82ae0 con 0x55c476b8c730 2024-03-21T10:47:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.083+0000 7f36ab400640 10 client.27670 handle_command_reply: tid=0 2024-03-21T10:47:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.083+0000 7f36cf1bf740 1 client.27670 shutdown 2024-03-21T10:47:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.083+0000 7f36cf1bf740 1 -- 172.21.15.31:0/1302223293 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c476b17c00 msgr2=0x55c476b1a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.083+0000 7f36cf1bf740 1 --2- 172.21.15.31:0/1302223293 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c476b17c00 0x55c476b1a5d0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x55c476b32cb0 tx=0x7f36b0004b70 comp rx=0 tx=0).stop 2024-03-21T10:47:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.083+0000 7f36cf1bf740 1 -- 172.21.15.31:0/1302223293 shutdown_connections 2024-03-21T10:47:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.083+0000 7f36cf1bf740 1 --2- 172.21.15.31:0/1302223293 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c476b8c730 0x55c476b8eb50 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.083+0000 7f36cf1bf740 1 --2- 172.21.15.31:0/1302223293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c476b20290 0x55c476b20bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.083+0000 7f36ab400640 0 client.27670 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.083+0000 7f36cf1bf740 1 --2- 172.21.15.31:0/1302223293 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c476b17c00 0x55c476b1a5d0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.083+0000 7f36cf1bf740 1 --2- 172.21.15.31:0/1302223293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c476b16f20 0x55c476b1a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.083+0000 7f36cf1bf740 1 -- 172.21.15.31:0/1302223293 >> 172.21.15.31:0/1302223293 conn(0x55c476ade200 msgr2=0x55c476b1b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.084+0000 7f36cf1bf740 1 -- 172.21.15.31:0/1302223293 shutdown_connections 2024-03-21T10:47:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.084+0000 7f36cf1bf740 1 -- 172.21.15.31:0/1302223293 wait complete. 2024-03-21T10:47:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.084+0000 7f36cf1bf740 20 client.27670 trim_cache size 0 max 16384 2024-03-21T10:47:02.085 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:47:02.085 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-21T10:47:02.085 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-03-21T10:47:02.085 INFO:teuthology.orchestra.run.smithi031.stdout: "ba166823-0eba-4caa-832f-34ab7724b69e": { 2024-03-21T10:47:02.085 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-03-21T10:47:02.085 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "ba166823-0eba-4caa-832f-34ab7724b69e", 2024-03-21T10:47:02.085 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-03-21T10:47:02.085 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:47:02.085 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:47:02.085 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:47:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.084+0000 7f36cc400640 1 -- 172.21.15.31:0/390518704 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f36ac0a2fa0 msgr2=0x7f36ac0a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.084+0000 7f36cc400640 1 --2- 172.21.15.31:0/390518704 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f36ac0a2fa0 0x7f36ac0a5460 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f36bc0226c0 tx=0x7f36bc01b040 comp rx=0 tx=0).stop 2024-03-21T10:47:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.084+0000 7f36cc400640 1 -- 172.21.15.31:0/390518704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36c4100440 msgr2=0x7f36c419a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.084+0000 7f36cc400640 1 --2- 172.21.15.31:0/390518704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36c4100440 0x7f36c419a140 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f36b802c240 tx=0x7f36b8004430 comp rx=0 tx=0).stop 2024-03-21T10:47:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.085+0000 7f36cc400640 1 -- 172.21.15.31:0/390518704 shutdown_connections 2024-03-21T10:47:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.085+0000 7f36cc400640 1 --2- 172.21.15.31:0/390518704 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f36ac0a2fa0 0x7f36ac0a5460 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.085+0000 7f36cc400640 1 --2- 172.21.15.31:0/390518704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36c410dff0 0x7f36c419e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.085+0000 7f36cc400640 1 --2- 172.21.15.31:0/390518704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36c4100d60 0x7f36c419a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.085+0000 7f36cc400640 1 --2- 172.21.15.31:0/390518704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36c4100440 0x7f36c419a140 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:02.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.085+0000 7f36cc400640 1 -- 172.21.15.31:0/390518704 >> 172.21.15.31:0/390518704 conn(0x7f36c40776c0 msgr2=0x7f36c410f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:02.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.085+0000 7f36cc400640 1 -- 172.21.15.31:0/390518704 shutdown_connections 2024-03-21T10:47:02.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:02.085+0000 7f36cc400640 1 -- 172.21.15.31:0/390518704 wait complete. 2024-03-21T10:47:02.099 INFO:tasks.cephfs.filesystem:scrub status for tag:ba166823-0eba-4caa-832f-34ab7724b69e - {'path': '/', 'tag': 'ba166823-0eba-4caa-832f-34ab7724b69e', 'options': 'recursive,force'} 2024-03-21T10:47:02.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:01 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:47:02.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:01 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:47:02.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:01 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:47:02.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:01 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:47:02.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:01 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:47:02.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:01 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:47:02.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:01 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:47:02.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:01 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:47:02.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:01 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:47:03.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:02 smithi031 ceph-mon[31673]: scrub queued for path: / 2024-03-21T10:47:03.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:02 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:47:03.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:02 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:47:03.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:02 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:47:03.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:02 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:47:03.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:02 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:47:03.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:02 smithi031 ceph-mon[31673]: pgmap v261: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.6 KiB/s wr, 6 op/s 2024-03-21T10:47:03.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:02 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:47:03.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:02 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:47:03.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:02 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:47:03.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:02 smithi070 ceph-mon[36330]: scrub queued for path: / 2024-03-21T10:47:03.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:02 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:47:03.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:02 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:47:03.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:02 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:47:03.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:02 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:47:03.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:02 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:47:03.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:02 smithi070 ceph-mon[36330]: pgmap v261: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.6 KiB/s wr, 6 op/s 2024-03-21T10:47:03.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:02 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:47:03.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:02 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:47:03.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:02 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:47:03.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:02 smithi040 ceph-mon[36154]: scrub queued for path: / 2024-03-21T10:47:03.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:02 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:47:03.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:02 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:47:03.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:02 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:47:03.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:02 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:47:03.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:02 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:47:03.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:02 smithi040 ceph-mon[36154]: pgmap v261: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.6 KiB/s wr, 6 op/s 2024-03-21T10:47:03.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:02 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:47:03.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:02 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:47:03.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:02 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:47:05.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:05 smithi031 ceph-mon[31673]: pgmap v262: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 KiB/s wr, 21 op/s 2024-03-21T10:47:05.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:05 smithi070 ceph-mon[36330]: pgmap v262: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 KiB/s wr, 21 op/s 2024-03-21T10:47:05.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:05 smithi040 ceph-mon[36154]: pgmap v262: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 KiB/s wr, 21 op/s 2024-03-21T10:47:06.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.194+0000 7f455a800640 1 -- 172.21.15.31:0/1119892171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4554101e60 msgr2=0x7f455410e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:06.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.194+0000 7f455a800640 1 --2- 172.21.15.31:0/1119892171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4554101e60 0x7f455410e320 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f454c009c10 tx=0x7f454c02bee0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.194+0000 7f455a800640 1 -- 172.21.15.31:0/1119892171 shutdown_connections 2024-03-21T10:47:06.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.194+0000 7f455a800640 1 --2- 172.21.15.31:0/1119892171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4554101e60 0x7f455410e320 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.194+0000 7f455a800640 1 --2- 172.21.15.31:0/1119892171 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45541014a0 0x7f4554101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.194+0000 7f455a800640 1 --2- 172.21.15.31:0/1119892171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4554107560 0x7f4554107940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.194+0000 7f455a800640 1 -- 172.21.15.31:0/1119892171 >> 172.21.15.31:0/1119892171 conn(0x7f45540fcfb0 msgr2=0x7f45540ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:06.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.194+0000 7f455a800640 1 -- 172.21.15.31:0/1119892171 shutdown_connections 2024-03-21T10:47:06.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.194+0000 7f455a800640 1 -- 172.21.15.31:0/1119892171 wait complete. 2024-03-21T10:47:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.195+0000 7f455a800640 1 Processor -- start 2024-03-21T10:47:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.195+0000 7f455a800640 1 -- start start 2024-03-21T10:47:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.196+0000 7f455a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45541014a0 0x7f455419d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.196+0000 7f455a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4554101e60 0x7f455419de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.196+0000 7f455a800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4554107560 0x7f4554197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.196+0000 7f455a800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4554111820 con 0x7f45541014a0 2024-03-21T10:47:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.196+0000 7f455a800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45541116a0 con 0x7f4554101e60 2024-03-21T10:47:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.196+0000 7f455a800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45541119a0 con 0x7f4554107560 2024-03-21T10:47:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.196+0000 7f4559400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45541014a0 0x7f455419d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.196+0000 7f4558a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4554101e60 0x7f455419de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.196+0000 7f4559e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4554107560 0x7f4554197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.196+0000 7f4558a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4554101e60 0x7f455419de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42780/0 (socket says 172.21.15.31:42780) 2024-03-21T10:47:06.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.196+0000 7f4559e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4554107560 0x7f4554197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:54056/0 (socket says 172.21.15.31:54056) 2024-03-21T10:47:06.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.196+0000 7f4558a00640 1 -- 172.21.15.31:0/244116789 learned_addr learned my addr 172.21.15.31:0/244116789 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:06.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.197+0000 7f4558a00640 1 -- 172.21.15.31:0/244116789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4554107560 msgr2=0x7f4554197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:06.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.197+0000 7f4558a00640 1 --2- 172.21.15.31:0/244116789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4554107560 0x7f4554197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.197+0000 7f4558a00640 1 -- 172.21.15.31:0/244116789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45541014a0 msgr2=0x7f455419d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:06.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.197+0000 7f4558a00640 1 --2- 172.21.15.31:0/244116789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45541014a0 0x7f455419d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.197+0000 7f4558a00640 1 -- 172.21.15.31:0/244116789 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4554198200 con 0x7f4554101e60 2024-03-21T10:47:06.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.197+0000 7f4558a00640 1 --2- 172.21.15.31:0/244116789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4554101e60 0x7f455419de10 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f454000e9f0 tx=0x7f454000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:06.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.197+0000 7f4552000640 1 -- 172.21.15.31:0/244116789 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454000cd80 con 0x7f4554101e60 2024-03-21T10:47:06.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.197+0000 7f4552000640 1 -- 172.21.15.31:0/244116789 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4540010910 con 0x7f4554101e60 2024-03-21T10:47:06.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.197+0000 7f455a800640 1 -- 172.21.15.31:0/244116789 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45541984f0 con 0x7f4554101e60 2024-03-21T10:47:06.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.197+0000 7f4552000640 1 -- 172.21.15.31:0/244116789 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4540018990 con 0x7f4554101e60 2024-03-21T10:47:06.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.197+0000 7f455a800640 1 -- 172.21.15.31:0/244116789 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45540738d0 con 0x7f4554101e60 2024-03-21T10:47:06.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.201+0000 7f455ebcf740 1 Processor -- start 2024-03-21T10:47:06.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.201+0000 7f455ebcf740 1 -- start start 2024-03-21T10:47:06.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.201+0000 7f4552000640 1 -- 172.21.15.31:0/244116789 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f4540010ac0 con 0x7f4554101e60 2024-03-21T10:47:06.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.201+0000 7f455ebcf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55902bb461e0 0x55902bb465c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:06.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.201+0000 7f455ebcf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55902bb46b00 0x55902bb4f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:06.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.201+0000 7f455ebcf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902bb4f8d0 0x55902bb51cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:06.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.201+0000 7f455ebcf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902bb083b0 con 0x55902bb4f8d0 2024-03-21T10:47:06.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.201+0000 7f455ebcf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902baaa830 con 0x55902bb46b00 2024-03-21T10:47:06.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.201+0000 7f455ebcf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902ba535d0 con 0x55902bb461e0 2024-03-21T10:47:06.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.201+0000 7f4559e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55902bb461e0 0x55902bb465c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:06.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.201+0000 7f4559400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55902bb46b00 0x55902bb4f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:06.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.201+0000 7f4559e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55902bb461e0 0x55902bb465c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:54070/0 (socket says 172.21.15.31:54070) 2024-03-21T10:47:06.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.201+0000 7f4559e00640 1 -- 172.21.15.31:0/20184946 learned_addr learned my addr 172.21.15.31:0/20184946 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:06.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.201+0000 7f4558a00640 1 --2- 172.21.15.31:0/20184946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902bb4f8d0 0x55902bb51cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:06.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.202+0000 7f4559e00640 1 -- 172.21.15.31:0/20184946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55902bb46b00 msgr2=0x55902bb4f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:06.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.202+0000 7f4559e00640 1 --2- 172.21.15.31:0/20184946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55902bb46b00 0x55902bb4f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.202+0000 7f4559e00640 1 -- 172.21.15.31:0/20184946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902bb4f8d0 msgr2=0x55902bb51cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:06.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.202+0000 7f4559e00640 1 --2- 172.21.15.31:0/20184946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902bb4f8d0 0x55902bb51cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.202+0000 7f4559e00640 1 -- 172.21.15.31:0/20184946 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55902ba6b2c0 con 0x55902bb461e0 2024-03-21T10:47:06.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.202+0000 7f4559e00640 1 --2- 172.21.15.31:0/20184946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55902bb461e0 0x55902bb465c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f454c009c10 tx=0x7f454c005eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5472803d00e644b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:06.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.202+0000 7f4546600640 1 -- 172.21.15.31:0/20184946 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454c033e40 con 0x55902bb461e0 2024-03-21T10:47:06.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.202+0000 7f4546600640 1 -- 172.21.15.31:0/20184946 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f454c003970 con 0x55902bb461e0 2024-03-21T10:47:06.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.202+0000 7f4546600640 1 -- 172.21.15.31:0/20184946 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454c02cd00 con 0x55902bb461e0 2024-03-21T10:47:06.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.203+0000 7f4552000640 1 --2- 172.21.15.31:0/244116789 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f453c0a3070 0x7f453c0a5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:06.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.203+0000 7f4552000640 1 -- 172.21.15.31:0/244116789 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f45400cff00 con 0x7f4554101e60 2024-03-21T10:47:06.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.203+0000 7f455ebcf740 1 -- 172.21.15.31:0/20184946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55902bb461e0 msgr2=0x55902bb465c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:06.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.203+0000 7f455ebcf740 1 --2- 172.21.15.31:0/20184946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55902bb461e0 0x55902bb465c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f454c009c10 tx=0x7f454c005eb0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.203+0000 7f4559400640 1 --2- 172.21.15.31:0/244116789 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f453c0a3070 0x7f453c0a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:06.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.203+0000 7f455ebcf740 1 -- 172.21.15.31:0/20184946 shutdown_connections 2024-03-21T10:47:06.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.203+0000 7f455ebcf740 1 --2- 172.21.15.31:0/20184946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902bb4f8d0 0x55902bb51cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.203+0000 7f455ebcf740 1 --2- 172.21.15.31:0/20184946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55902bb46b00 0x55902bb4f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.203+0000 7f455ebcf740 1 --2- 172.21.15.31:0/20184946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55902bb461e0 0x55902bb465c0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.203+0000 7f455ebcf740 1 -- 172.21.15.31:0/20184946 >> 172.21.15.31:0/20184946 conn(0x55902bb40b00 msgr2=0x55902bb40ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:06.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.203+0000 7f455ebcf740 1 -- 172.21.15.31:0/20184946 shutdown_connections 2024-03-21T10:47:06.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.203+0000 7f455ebcf740 1 -- 172.21.15.31:0/20184946 wait complete. 2024-03-21T10:47:06.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.204+0000 7f4559400640 1 --2- 172.21.15.31:0/244116789 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f453c0a3070 0x7f453c0a5530 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x55902bb45c90 tx=0x7f4548012040 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:06.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.204+0000 7f455ebcf740 1 -- start start 2024-03-21T10:47:06.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.204+0000 7f455ebcf740 1 Processor -- start 2024-03-21T10:47:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.205+0000 7f455ebcf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55902bb461e0 0x55902bb4c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.205+0000 7f455ebcf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55902bb46b00 0x55902bb4cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.205+0000 7f455ebcf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902bb4f8d0 0x55902bb4d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.205+0000 7f455ebcf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902ba66a20 con 0x55902bb4f8d0 2024-03-21T10:47:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.205+0000 7f455ebcf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902baa68d0 con 0x55902bb461e0 2024-03-21T10:47:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.205+0000 7f455ebcf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902bb524a0 con 0x55902bb46b00 2024-03-21T10:47:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.205+0000 7f4559400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55902bb461e0 0x55902bb4c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.205+0000 7f4558a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55902bb46b00 0x55902bb4cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.205+0000 7f455ebcf740 10 client.0 _ll_register_callbacks cb 0x55902baa31c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.205+0000 7f4558a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55902bb46b00 0x55902bb4cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:54080/0 (socket says 172.21.15.31:54080) 2024-03-21T10:47:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.205+0000 7f4559400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55902bb461e0 0x55902bb4c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42806/0 (socket says 172.21.15.31:42806) 2024-03-21T10:47:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.205+0000 7f4558a00640 1 -- 172.21.15.31:0/545241011 learned_addr learned my addr 172.21.15.31:0/545241011 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.205+0000 7f4559e00640 1 --2- 172.21.15.31:0/545241011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902bb4f8d0 0x55902bb4d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.206+0000 7f4558a00640 1 -- 172.21.15.31:0/545241011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55902bb461e0 msgr2=0x55902bb4c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.206+0000 7f4558a00640 1 --2- 172.21.15.31:0/545241011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55902bb461e0 0x55902bb4c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.206+0000 7f4558a00640 1 -- 172.21.15.31:0/545241011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902bb4f8d0 msgr2=0x55902bb4d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.206+0000 7f4558a00640 1 --2- 172.21.15.31:0/545241011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902bb4f8d0 0x55902bb4d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.206+0000 7f4558a00640 1 -- 172.21.15.31:0/545241011 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55902b7bf210 con 0x55902bb46b00 2024-03-21T10:47:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.206+0000 7f4558a00640 1 --2- 172.21.15.31:0/545241011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55902bb46b00 0x55902bb4cf00 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f454000e9c0 tx=0x7f454009b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e1f4a1cf7fa0cff server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.206+0000 7f453aa00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:47:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.206+0000 7f453aa00640 1 -- 172.21.15.31:0/545241011 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45400cd6d0 con 0x55902bb46b00 2024-03-21T10:47:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.206+0000 7f453aa00640 1 -- 172.21.15.31:0/545241011 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f4540018b30 con 0x55902bb46b00 2024-03-21T10:47:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.206+0000 7f455ebcf740 1 -- 172.21.15.31:0/545241011 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f454c003970 con 0x55902bb46b00 2024-03-21T10:47:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.207+0000 7f453aa00640 1 -- 172.21.15.31:0/545241011 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45400a0040 con 0x55902bb46b00 2024-03-21T10:47:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.207+0000 7f455ebcf740 10 client.27682 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.207+0000 7f455ebcf740 1 -- 172.21.15.31:0/545241011 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55902b88b620 con 0x55902bb46b00 2024-03-21T10:47:06.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.207+0000 7f453aa00640 1 -- 172.21.15.31:0/545241011 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f454c003970 con 0x55902bb46b00 2024-03-21T10:47:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.207+0000 7f453aa00640 1 -- 172.21.15.31:0/545241011 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f45400b8080 con 0x55902bb46b00 2024-03-21T10:47:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.207+0000 7f455ebcf740 10 client.27682 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.207+0000 7f455ebcf740 10 client.27682 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:47:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.207+0000 7f455ebcf740 20 client.27682 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.207+0000 7f455ebcf740 1 --2- 172.21.15.31:0/545241011 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55902bb5f940 0x55902bb61d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.207+0000 7f455ebcf740 4 client.27682 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:47:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.207+0000 7f455ebcf740 1 -- 172.21.15.31:0/545241011 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55902bab7d70 con 0x55902bb5f940 2024-03-21T10:47:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.208+0000 7f4559400640 1 --2- 172.21.15.31:0/545241011 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55902bb5f940 0x55902bb61d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.208+0000 7f4559400640 1 --2- 172.21.15.31:0/545241011 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55902bb5f940 0x55902bb61d80 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef566f2869e1a2d8 server_cookie=897905f6ab0b354f in_seq=0 out_seq=0 2024-03-21T10:47:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.208+0000 7f453aa00640 10 client.27682 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.209+0000 7f453aa00640 1 -- 172.21.15.31:0/545241011 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55902bab7d70 con 0x55902bb5f940 2024-03-21T10:47:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.209+0000 7f453aa00640 10 client.27682 handle_command_reply: tid=0 2024-03-21T10:47:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.209+0000 7f455ebcf740 1 client.27682 shutdown 2024-03-21T10:47:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.209+0000 7f455ebcf740 1 -- 172.21.15.31:0/545241011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55902bb46b00 msgr2=0x55902bb4cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.209+0000 7f455ebcf740 1 --2- 172.21.15.31:0/545241011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55902bb46b00 0x55902bb4cf00 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f454000e9c0 tx=0x7f454009b900 comp rx=0 tx=0).stop 2024-03-21T10:47:06.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.209+0000 7f455ebcf740 1 -- 172.21.15.31:0/545241011 shutdown_connections 2024-03-21T10:47:06.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.209+0000 7f455ebcf740 1 --2- 172.21.15.31:0/545241011 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55902bb5f940 0x55902bb61d80 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.209+0000 7f455ebcf740 1 --2- 172.21.15.31:0/545241011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902bb4f8d0 0x55902bb4d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.209+0000 7f455ebcf740 1 --2- 172.21.15.31:0/545241011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55902bb46b00 0x55902bb4cf00 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.209+0000 7f453aa00640 0 client.27682 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:06.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.209+0000 7f455ebcf740 1 --2- 172.21.15.31:0/545241011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55902bb461e0 0x55902bb4c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.209+0000 7f455ebcf740 1 -- 172.21.15.31:0/545241011 >> 172.21.15.31:0/545241011 conn(0x55902bb40b00 msgr2=0x55902bb42850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:06.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.209+0000 7f455ebcf740 1 -- 172.21.15.31:0/545241011 shutdown_connections 2024-03-21T10:47:06.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.209+0000 7f455ebcf740 1 -- 172.21.15.31:0/545241011 wait complete. 2024-03-21T10:47:06.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.210+0000 7f455ebcf740 20 client.27682 trim_cache size 0 max 16384 2024-03-21T10:47:06.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.230+0000 7f455ebcf740 1 Processor -- start 2024-03-21T10:47:06.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.230+0000 7f455ebcf740 1 -- start start 2024-03-21T10:47:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.231+0000 7f455ebcf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55902bb461e0 0x55902bb4f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.231+0000 7f455ebcf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55902bb46b00 0x55902bb4d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.231+0000 7f455ebcf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902bb4f8d0 0x55902bb5abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.231+0000 7f455ebcf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45541ae9b0 con 0x55902bb4f8d0 2024-03-21T10:47:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.231+0000 7f455ebcf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45541ac4b0 con 0x55902bb46b00 2024-03-21T10:47:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.231+0000 7f455ebcf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45541aa570 con 0x55902bb461e0 2024-03-21T10:47:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.231+0000 7f4559400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55902bb461e0 0x55902bb4f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.231+0000 7f4559400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55902bb461e0 0x55902bb4f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:54084/0 (socket says 172.21.15.31:54084) 2024-03-21T10:47:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.231+0000 7f4559400640 1 -- 172.21.15.31:0/2699319791 learned_addr learned my addr 172.21.15.31:0/2699319791 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.231+0000 7f4558a00640 1 --2- 172.21.15.31:0/2699319791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55902bb46b00 0x55902bb4d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:06.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.231+0000 7f4559e00640 1 --2- 172.21.15.31:0/2699319791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902bb4f8d0 0x55902bb5abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:06.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.231+0000 7f4559400640 1 -- 172.21.15.31:0/2699319791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55902bb46b00 msgr2=0x55902bb4d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:06.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.231+0000 7f4559400640 1 --2- 172.21.15.31:0/2699319791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55902bb46b00 0x55902bb4d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.231+0000 7f4559400640 1 -- 172.21.15.31:0/2699319791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902bb4f8d0 msgr2=0x55902bb5abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:06.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.231+0000 7f4559400640 1 --2- 172.21.15.31:0/2699319791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902bb4f8d0 0x55902bb5abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.231+0000 7f4559400640 1 -- 172.21.15.31:0/2699319791 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55902bb62400 con 0x55902bb461e0 2024-03-21T10:47:06.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.231+0000 7f4558a00640 1 --2- 172.21.15.31:0/2699319791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55902bb46b00 0x55902bb4d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:06.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.232+0000 7f4559400640 1 --2- 172.21.15.31:0/2699319791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55902bb461e0 0x55902bb4f0f0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f4548009530 tx=0x7f454801a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dafc161a4c48370 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:06.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.232+0000 7f4545c00640 1 -- 172.21.15.31:0/2699319791 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4548030070 con 0x55902bb461e0 2024-03-21T10:47:06.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.232+0000 7f4545c00640 1 -- 172.21.15.31:0/2699319791 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4548021040 con 0x55902bb461e0 2024-03-21T10:47:06.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.232+0000 7f4545c00640 1 -- 172.21.15.31:0/2699319791 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454802a510 con 0x55902bb461e0 2024-03-21T10:47:06.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.233+0000 7f455ebcf740 1 -- 172.21.15.31:0/2699319791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55902bb461e0 msgr2=0x55902bb4f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:06.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.233+0000 7f455ebcf740 1 --2- 172.21.15.31:0/2699319791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55902bb461e0 0x55902bb4f0f0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f4548009530 tx=0x7f454801a000 comp rx=0 tx=0).stop 2024-03-21T10:47:06.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.233+0000 7f455ebcf740 1 -- 172.21.15.31:0/2699319791 shutdown_connections 2024-03-21T10:47:06.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.233+0000 7f455ebcf740 1 --2- 172.21.15.31:0/2699319791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902bb4f8d0 0x55902bb5abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.233+0000 7f455ebcf740 1 --2- 172.21.15.31:0/2699319791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55902bb46b00 0x55902bb4d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.233+0000 7f455ebcf740 1 --2- 172.21.15.31:0/2699319791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55902bb461e0 0x55902bb4f0f0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.233+0000 7f455ebcf740 1 -- 172.21.15.31:0/2699319791 >> 172.21.15.31:0/2699319791 conn(0x55902bb40b00 msgr2=0x55902bb4c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:06.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.233+0000 7f455ebcf740 1 -- 172.21.15.31:0/2699319791 shutdown_connections 2024-03-21T10:47:06.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.233+0000 7f455ebcf740 1 -- 172.21.15.31:0/2699319791 wait complete. 2024-03-21T10:47:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.233+0000 7f455ebcf740 1 -- start start 2024-03-21T10:47:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.234+0000 7f455ebcf740 1 Processor -- start 2024-03-21T10:47:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.234+0000 7f455ebcf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902bb461e0 0x55902bb57320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.234+0000 7f455ebcf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55902bb46b00 0x55902bb49b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.234+0000 7f455ebcf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55902bb4f8d0 0x55902bb4a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.234+0000 7f455ebcf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45541aab70 con 0x55902bb461e0 2024-03-21T10:47:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.234+0000 7f455ebcf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45541abe50 con 0x55902bb46b00 2024-03-21T10:47:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.234+0000 7f455ebcf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45541ab190 con 0x55902bb4f8d0 2024-03-21T10:47:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.234+0000 7f455ebcf740 10 client.0 _ll_register_callbacks cb 0x7f454000cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.235+0000 7f4559400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902bb461e0 0x55902bb57320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.235+0000 7f4558a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55902bb46b00 0x55902bb49b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.235+0000 7f4559e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55902bb4f8d0 0x55902bb4a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.235+0000 7f4558a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55902bb46b00 0x55902bb49b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42832/0 (socket says 172.21.15.31:42832) 2024-03-21T10:47:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.235+0000 7f4558a00640 1 -- 172.21.15.31:0/1734115040 learned_addr learned my addr 172.21.15.31:0/1734115040 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.235+0000 7f4558a00640 1 -- 172.21.15.31:0/1734115040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55902bb4f8d0 msgr2=0x55902bb4a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.235+0000 7f4558a00640 1 --2- 172.21.15.31:0/1734115040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55902bb4f8d0 0x55902bb4a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.235+0000 7f4558a00640 1 -- 172.21.15.31:0/1734115040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902bb461e0 msgr2=0x55902bb57320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.235+0000 7f4558a00640 1 --2- 172.21.15.31:0/1734115040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902bb461e0 0x55902bb57320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.235+0000 7f4558a00640 1 -- 172.21.15.31:0/1734115040 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55902bb523e0 con 0x55902bb46b00 2024-03-21T10:47:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.235+0000 7f4559400640 1 --2- 172.21.15.31:0/1734115040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902bb461e0 0x55902bb57320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.235+0000 7f4558a00640 1 --2- 172.21.15.31:0/1734115040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55902bb46b00 0x55902bb49b80 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f4540098400 tx=0x7f4540005700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d758b9daa6ea2bb server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.235+0000 7f453aa00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:47:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.236+0000 7f453aa00640 1 -- 172.21.15.31:0/1734115040 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45400d3230 con 0x55902bb46b00 2024-03-21T10:47:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.236+0000 7f453aa00640 1 -- 172.21.15.31:0/1734115040 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f4540018b30 con 0x55902bb46b00 2024-03-21T10:47:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.236+0000 7f455ebcf740 1 -- 172.21.15.31:0/1734115040 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4548021040 con 0x55902bb46b00 2024-03-21T10:47:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.236+0000 7f453aa00640 1 -- 172.21.15.31:0/1734115040 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f454009b600 con 0x55902bb46b00 2024-03-21T10:47:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.236+0000 7f455ebcf740 10 client.27767 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.236+0000 7f455ebcf740 1 -- 172.21.15.31:0/1734115040 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55902bbbbe70 con 0x55902bb46b00 2024-03-21T10:47:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.236+0000 7f453aa00640 1 -- 172.21.15.31:0/1734115040 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4548021040 con 0x55902bb46b00 2024-03-21T10:47:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.236+0000 7f453aa00640 1 -- 172.21.15.31:0/1734115040 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f45400d1030 con 0x55902bb46b00 2024-03-21T10:47:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.236+0000 7f455ebcf740 10 client.27767 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.236+0000 7f455ebcf740 10 client.27767 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:47:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.236+0000 7f455ebcf740 20 client.27767 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.236+0000 7f455ebcf740 1 --2- 172.21.15.31:0/1734115040 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55902bbbc5d0 0x55902bbbe9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.236+0000 7f455ebcf740 4 client.27767 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:47:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.236+0000 7f455ebcf740 1 -- 172.21.15.31:0/1734115040 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55902bab20c0 con 0x55902bbbc5d0 2024-03-21T10:47:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.237+0000 7f4559400640 1 --2- 172.21.15.31:0/1734115040 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55902bbbc5d0 0x55902bbbe9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.237+0000 7f4559400640 1 --2- 172.21.15.31:0/1734115040 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55902bbbc5d0 0x55902bbbe9f0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac5a9020a9f108e0 server_cookie=53012101923af99 in_seq=0 out_seq=0 2024-03-21T10:47:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.237+0000 7f453aa00640 10 client.27767 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.245+0000 7f453aa00640 1 -- 172.21.15.31:0/1734115040 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55902bab20c0 con 0x55902bbbc5d0 2024-03-21T10:47:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.245+0000 7f453aa00640 10 client.27767 handle_command_reply: tid=0 2024-03-21T10:47:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.245+0000 7f455ebcf740 1 client.27767 shutdown 2024-03-21T10:47:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.245+0000 7f455ebcf740 1 -- 172.21.15.31:0/1734115040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55902bb46b00 msgr2=0x55902bb49b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.245+0000 7f455ebcf740 1 --2- 172.21.15.31:0/1734115040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55902bb46b00 0x55902bb49b80 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f4540098400 tx=0x7f4540005700 comp rx=0 tx=0).stop 2024-03-21T10:47:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.245+0000 7f455ebcf740 1 -- 172.21.15.31:0/1734115040 shutdown_connections 2024-03-21T10:47:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.245+0000 7f455ebcf740 1 --2- 172.21.15.31:0/1734115040 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55902bbbc5d0 0x55902bbbe9f0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.245+0000 7f455ebcf740 1 --2- 172.21.15.31:0/1734115040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55902bb4f8d0 0x55902bb4a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.245+0000 7f455ebcf740 1 --2- 172.21.15.31:0/1734115040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55902bb46b00 0x55902bb49b80 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.245+0000 7f455ebcf740 1 --2- 172.21.15.31:0/1734115040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902bb461e0 0x55902bb57320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.245+0000 7f455ebcf740 1 -- 172.21.15.31:0/1734115040 >> 172.21.15.31:0/1734115040 conn(0x55902bb40b00 msgr2=0x55902bb42550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.245+0000 7f453aa00640 0 client.27767 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.245+0000 7f455ebcf740 1 -- 172.21.15.31:0/1734115040 shutdown_connections 2024-03-21T10:47:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.245+0000 7f455ebcf740 1 -- 172.21.15.31:0/1734115040 wait complete. 2024-03-21T10:47:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.245+0000 7f455ebcf740 20 client.27767 trim_cache size 0 max 16384 2024-03-21T10:47:06.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:47:06.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:47:06.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:47:06.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:47:06.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.246+0000 7f455a800640 1 -- 172.21.15.31:0/244116789 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f453c0a3070 msgr2=0x7f453c0a5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:06.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.246+0000 7f455a800640 1 --2- 172.21.15.31:0/244116789 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f453c0a3070 0x7f453c0a5530 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x55902bb45c90 tx=0x7f4548012040 comp rx=0 tx=0).stop 2024-03-21T10:47:06.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.246+0000 7f455a800640 1 -- 172.21.15.31:0/244116789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4554101e60 msgr2=0x7f455419de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:06.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.246+0000 7f455a800640 1 --2- 172.21.15.31:0/244116789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4554101e60 0x7f455419de10 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f454000e9f0 tx=0x7f454000eeb0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.246+0000 7f455a800640 1 -- 172.21.15.31:0/244116789 shutdown_connections 2024-03-21T10:47:06.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.246+0000 7f455a800640 1 --2- 172.21.15.31:0/244116789 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f453c0a3070 0x7f453c0a5530 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.246+0000 7f455a800640 1 --2- 172.21.15.31:0/244116789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4554107560 0x7f4554197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.246+0000 7f455a800640 1 --2- 172.21.15.31:0/244116789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4554101e60 0x7f455419de10 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.246+0000 7f455a800640 1 --2- 172.21.15.31:0/244116789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45541014a0 0x7f455419d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:06.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.246+0000 7f455a800640 1 -- 172.21.15.31:0/244116789 >> 172.21.15.31:0/244116789 conn(0x7f45540fcfb0 msgr2=0x7f45540fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:06.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.246+0000 7f455a800640 1 -- 172.21.15.31:0/244116789 shutdown_connections 2024-03-21T10:47:06.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:06.246+0000 7f455a800640 1 -- 172.21.15.31:0/244116789 wait complete. 2024-03-21T10:47:07.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:07 smithi031 ceph-mon[31673]: pgmap v263: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 50 KiB/s wr, 30 op/s 2024-03-21T10:47:07.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:07 smithi070 ceph-mon[36330]: pgmap v263: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 50 KiB/s wr, 30 op/s 2024-03-21T10:47:07.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:07 smithi040 ceph-mon[36154]: pgmap v263: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 50 KiB/s wr, 30 op/s 2024-03-21T10:47:08.303 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:08 smithi031 ceph-mon[31673]: osdmap e99: 12 total, 12 up, 12 in 2024-03-21T10:47:08.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:08 smithi070 ceph-mon[36330]: osdmap e99: 12 total, 12 up, 12 in 2024-03-21T10:47:08.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:08 smithi040 ceph-mon[36154]: osdmap e99: 12 total, 12 up, 12 in 2024-03-21T10:47:09.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:09 smithi031 ceph-mon[31673]: pgmap v265: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 55 KiB/s wr, 31 op/s 2024-03-21T10:47:09.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:09 smithi070 ceph-mon[36330]: pgmap v265: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 55 KiB/s wr, 31 op/s 2024-03-21T10:47:09.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:09 smithi040 ceph-mon[36154]: pgmap v265: 129 pgs: 129 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 55 KiB/s wr, 31 op/s 2024-03-21T10:47:11.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:11 smithi031 ceph-mon[31673]: pgmap v266: 129 pgs: 10 active+clean+snaptrim, 101 active+clean, 18 active+clean+snaptrim_wait; 46 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 58 KiB/s wr, 34 op/s 2024-03-21T10:47:11.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:11 smithi070 ceph-mon[36330]: pgmap v266: 129 pgs: 10 active+clean+snaptrim, 101 active+clean, 18 active+clean+snaptrim_wait; 46 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 58 KiB/s wr, 34 op/s 2024-03-21T10:47:11.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:11 smithi040 ceph-mon[36154]: pgmap v266: 129 pgs: 10 active+clean+snaptrim, 101 active+clean, 18 active+clean+snaptrim_wait; 46 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 58 KiB/s wr, 34 op/s 2024-03-21T10:47:13.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:13 smithi031 ceph-mon[31673]: pgmap v267: 129 pgs: 10 active+clean+snaptrim, 101 active+clean, 18 active+clean+snaptrim_wait; 45 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 58 KiB/s wr, 34 op/s 2024-03-21T10:47:13.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:13 smithi070 ceph-mon[36330]: pgmap v267: 129 pgs: 10 active+clean+snaptrim, 101 active+clean, 18 active+clean+snaptrim_wait; 45 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 58 KiB/s wr, 34 op/s 2024-03-21T10:47:13.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:13 smithi040 ceph-mon[36154]: pgmap v267: 129 pgs: 10 active+clean+snaptrim, 101 active+clean, 18 active+clean+snaptrim_wait; 45 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 58 KiB/s wr, 34 op/s 2024-03-21T10:47:15.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:15 smithi031 ceph-mon[31673]: pgmap v268: 129 pgs: 6 active+clean+snaptrim, 108 active+clean, 15 active+clean+snaptrim_wait; 45 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 33 KiB/s wr, 16 op/s 2024-03-21T10:47:15.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:15 smithi070 ceph-mon[36330]: pgmap v268: 129 pgs: 6 active+clean+snaptrim, 108 active+clean, 15 active+clean+snaptrim_wait; 45 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 33 KiB/s wr, 16 op/s 2024-03-21T10:47:15.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:15 smithi040 ceph-mon[36154]: pgmap v268: 129 pgs: 6 active+clean+snaptrim, 108 active+clean, 15 active+clean+snaptrim_wait; 45 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 33 KiB/s wr, 16 op/s 2024-03-21T10:47:17.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:17 smithi070 ceph-mon[36330]: pgmap v269: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.6 KiB/s wr, 8 op/s 2024-03-21T10:47:17.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:17 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:47:17.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:17 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:47:17.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:17 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:47:17.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:17 smithi070 ceph-mon[36330]: osdmap e100: 12 total, 12 up, 12 in 2024-03-21T10:47:17.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:17 smithi040 ceph-mon[36154]: pgmap v269: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.6 KiB/s wr, 8 op/s 2024-03-21T10:47:17.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:17 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:47:17.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:17 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:47:17.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:17 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:47:17.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:17 smithi040 ceph-mon[36154]: osdmap e100: 12 total, 12 up, 12 in 2024-03-21T10:47:17.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:17 smithi031 ceph-mon[31673]: pgmap v269: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.6 KiB/s wr, 8 op/s 2024-03-21T10:47:17.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:17 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:47:17.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:17 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:47:17.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:17 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:47:17.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:17 smithi031 ceph-mon[31673]: osdmap e100: 12 total, 12 up, 12 in 2024-03-21T10:47:19.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:19 smithi070 ceph-mon[36330]: pgmap v271: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.6 KiB/s wr, 8 op/s 2024-03-21T10:47:19.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.394+0000 7f12d2200640 1 -- 172.21.15.31:0/4109904133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12cc104250 msgr2=0x7f12cc110760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:19.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.394+0000 7f12d2200640 1 --2- 172.21.15.31:0/4109904133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12cc104250 0x7f12cc110760 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f12c4009ca0 tx=0x7f12c402c100 comp rx=0 tx=0).stop 2024-03-21T10:47:19.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.394+0000 7f12d2200640 1 -- 172.21.15.31:0/4109904133 shutdown_connections 2024-03-21T10:47:19.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.394+0000 7f12d2200640 1 --2- 172.21.15.31:0/4109904133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12cc104250 0x7f12cc110760 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.394+0000 7f12d2200640 1 --2- 172.21.15.31:0/4109904133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12cc103890 0x7f12cc103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.394+0000 7f12d2200640 1 --2- 172.21.15.31:0/4109904133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12cc1098a0 0x7f12cc109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.394+0000 7f12d2200640 1 -- 172.21.15.31:0/4109904133 >> 172.21.15.31:0/4109904133 conn(0x7f12cc0ff320 msgr2=0x7f12cc101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:19.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.395+0000 7f12d2200640 1 -- 172.21.15.31:0/4109904133 shutdown_connections 2024-03-21T10:47:19.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.395+0000 7f12d2200640 1 -- 172.21.15.31:0/4109904133 wait complete. 2024-03-21T10:47:19.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.395+0000 7f12d2200640 1 Processor -- start 2024-03-21T10:47:19.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.396+0000 7f12d2200640 1 -- start start 2024-03-21T10:47:19.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.396+0000 7f12d2200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12cc103890 0x7f12cc19a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:19.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.396+0000 7f12d2200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12cc104250 0x7f12cc19a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:19.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.396+0000 7f12d2200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12cc1098a0 0x7f12cc19e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:19.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.396+0000 7f12d2200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc113cb0 con 0x7f12cc1098a0 2024-03-21T10:47:19.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.396+0000 7f12d2200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc113b30 con 0x7f12cc103890 2024-03-21T10:47:19.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.396+0000 7f12d2200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc113e30 con 0x7f12cc104250 2024-03-21T10:47:19.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.396+0000 7f12c2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12cc104250 0x7f12cc19a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:19.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.396+0000 7f12cb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12cc103890 0x7f12cc19a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:19.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.396+0000 7f12c2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12cc104250 0x7f12cc19a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:48776/0 (socket says 172.21.15.31:48776) 2024-03-21T10:47:19.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.396+0000 7f12c2a00640 1 -- 172.21.15.31:0/2876227350 learned_addr learned my addr 172.21.15.31:0/2876227350 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:19.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.396+0000 7f12cbe00640 1 --2- 172.21.15.31:0/2876227350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12cc1098a0 0x7f12cc19e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:19.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.397+0000 7f12c2a00640 1 -- 172.21.15.31:0/2876227350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12cc103890 msgr2=0x7f12cc19a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:19.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.397+0000 7f12c2a00640 1 --2- 172.21.15.31:0/2876227350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12cc103890 0x7f12cc19a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.397+0000 7f12c2a00640 1 -- 172.21.15.31:0/2876227350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12cc1098a0 msgr2=0x7f12cc19e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:19.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.397+0000 7f12c2a00640 1 --2- 172.21.15.31:0/2876227350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12cc1098a0 0x7f12cc19e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.397+0000 7f12c2a00640 1 -- 172.21.15.31:0/2876227350 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12cc19f100 con 0x7f12cc104250 2024-03-21T10:47:19.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.397+0000 7f12c2a00640 1 --2- 172.21.15.31:0/2876227350 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12cc104250 0x7f12cc19a650 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f12b400e9f0 tx=0x7f12b400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:19.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.397+0000 7f12c8c00640 1 -- 172.21.15.31:0/2876227350 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b400cd80 con 0x7f12cc104250 2024-03-21T10:47:19.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.397+0000 7f12c8c00640 1 -- 172.21.15.31:0/2876227350 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12b4010910 con 0x7f12cc104250 2024-03-21T10:47:19.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.397+0000 7f12d2200640 1 -- 172.21.15.31:0/2876227350 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12cc19f390 con 0x7f12cc104250 2024-03-21T10:47:19.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.397+0000 7f12c8c00640 1 -- 172.21.15.31:0/2876227350 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b40189d0 con 0x7f12cc104250 2024-03-21T10:47:19.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.398+0000 7f12d2200640 1 -- 172.21.15.31:0/2876227350 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12cc1a6c80 con 0x7f12cc104250 2024-03-21T10:47:19.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.401+0000 7f12c8c00640 1 -- 172.21.15.31:0/2876227350 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f12b4010ac0 con 0x7f12cc104250 2024-03-21T10:47:19.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.401+0000 7f12d5022740 1 Processor -- start 2024-03-21T10:47:19.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.401+0000 7f12d5022740 1 -- start start 2024-03-21T10:47:19.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.401+0000 7f12d5022740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7873afd0 0x557d7873b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:19.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.401+0000 7f12d5022740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7873b8f0 0x557d78744180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:19.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.401+0000 7f12d5022740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557d787446c0 0x557d78746ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:19.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.401+0000 7f12d5022740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d786fd3b0 con 0x557d7873afd0 2024-03-21T10:47:19.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.401+0000 7f12d5022740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7869f830 con 0x557d787446c0 2024-03-21T10:47:19.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.401+0000 7f12d5022740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d786485d0 con 0x557d7873b8f0 2024-03-21T10:47:19.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.402+0000 7f12cbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7873afd0 0x557d7873b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:19.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.402+0000 7f12c2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557d787446c0 0x557d78746ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:19.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.402+0000 7f12cb400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7873b8f0 0x557d78744180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:19.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.402+0000 7f12cbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7873afd0 0x557d7873b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56130/0 (socket says 172.21.15.31:56130) 2024-03-21T10:47:19.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.402+0000 7f12cb400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7873b8f0 0x557d78744180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:48790/0 (socket says 172.21.15.31:48790) 2024-03-21T10:47:19.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.402+0000 7f12cbe00640 1 -- 172.21.15.31:0/3231617743 learned_addr learned my addr 172.21.15.31:0/3231617743 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.402+0000 7f12cbe00640 1 -- 172.21.15.31:0/3231617743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7873b8f0 msgr2=0x557d78744180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.402+0000 7f12cbe00640 1 --2- 172.21.15.31:0/3231617743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7873b8f0 0x557d78744180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.402+0000 7f12cbe00640 1 -- 172.21.15.31:0/3231617743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557d787446c0 msgr2=0x557d78746ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.402+0000 7f12cbe00640 1 --2- 172.21.15.31:0/3231617743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557d787446c0 0x557d78746ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.402+0000 7f12cbe00640 1 -- 172.21.15.31:0/3231617743 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d786602c0 con 0x557d7873afd0 2024-03-21T10:47:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.402+0000 7f12cb400640 1 --2- 172.21.15.31:0/3231617743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7873b8f0 0x557d78744180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.402+0000 7f12c2a00640 1 --2- 172.21.15.31:0/3231617743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557d787446c0 0x557d78746ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.403+0000 7f12c8c00640 1 --2- 172.21.15.31:0/2876227350 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f12b00a3100 0x7f12b00a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.403+0000 7f12cbe00640 1 --2- 172.21.15.31:0/3231617743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7873afd0 0x557d7873b3b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f12c402c710 tx=0x7f12c4005f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18dfbfb7afc8ee5b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.403+0000 7f12c8c00640 1 -- 172.21.15.31:0/2876227350 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f12b40cfea0 con 0x7f12cc104250 2024-03-21T10:47:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.403+0000 7f12cb400640 1 --2- 172.21.15.31:0/2876227350 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f12b00a3100 0x7f12b00a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:19.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.403+0000 7f12bbe00640 1 -- 172.21.15.31:0/3231617743 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c4045020 con 0x557d7873afd0 2024-03-21T10:47:19.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.403+0000 7f12bbe00640 1 -- 172.21.15.31:0/3231617743 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12c403a040 con 0x557d7873afd0 2024-03-21T10:47:19.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.403+0000 7f12bbe00640 1 -- 172.21.15.31:0/3231617743 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c4039040 con 0x557d7873afd0 2024-03-21T10:47:19.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.404+0000 7f12d5022740 1 -- 172.21.15.31:0/3231617743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7873afd0 msgr2=0x557d7873b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:19.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.404+0000 7f12cb400640 1 --2- 172.21.15.31:0/2876227350 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f12b00a3100 0x7f12b00a55c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x557d7873aa80 tx=0x7f12bc005f90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:19.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.404+0000 7f12d5022740 1 --2- 172.21.15.31:0/3231617743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7873afd0 0x557d7873b3b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f12c402c710 tx=0x7f12c4005f80 comp rx=0 tx=0).stop 2024-03-21T10:47:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.404+0000 7f12d5022740 1 -- 172.21.15.31:0/3231617743 shutdown_connections 2024-03-21T10:47:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.404+0000 7f12d5022740 1 --2- 172.21.15.31:0/3231617743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557d787446c0 0x557d78746ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.404+0000 7f12d5022740 1 --2- 172.21.15.31:0/3231617743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7873b8f0 0x557d78744180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.404+0000 7f12d5022740 1 --2- 172.21.15.31:0/3231617743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7873afd0 0x557d7873b3b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.404+0000 7f12d5022740 1 -- 172.21.15.31:0/3231617743 >> 172.21.15.31:0/3231617743 conn(0x557d787358f0 msgr2=0x557d78735cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.404+0000 7f12d5022740 1 -- 172.21.15.31:0/3231617743 shutdown_connections 2024-03-21T10:47:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.404+0000 7f12d5022740 1 -- 172.21.15.31:0/3231617743 wait complete. 2024-03-21T10:47:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.405+0000 7f12d5022740 1 -- start start 2024-03-21T10:47:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.405+0000 7f12d5022740 1 Processor -- start 2024-03-21T10:47:19.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.406+0000 7f12d5022740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557d7873afd0 0x557d78741760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:19.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.406+0000 7f12d5022740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7873b8f0 0x557d78741ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:19.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.406+0000 7f12d5022740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d787446c0 0x557d787421e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:19.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.406+0000 7f12d5022740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7865ba20 con 0x557d7873b8f0 2024-03-21T10:47:19.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.406+0000 7f12d5022740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7869b8d0 con 0x557d7873afd0 2024-03-21T10:47:19.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.406+0000 7f12d5022740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d78747290 con 0x557d787446c0 2024-03-21T10:47:19.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.406+0000 7f12cb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557d7873afd0 0x557d78741760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:19.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.406+0000 7f12d5022740 10 client.0 _ll_register_callbacks cb 0x557d786981c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:19.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.406+0000 7f12c2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7873b8f0 0x557d78741ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:19.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.406+0000 7f12cb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557d7873afd0 0x557d78741760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39554/0 (socket says 172.21.15.31:39554) 2024-03-21T10:47:19.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.406+0000 7f12cb400640 1 -- 172.21.15.31:0/761600489 learned_addr learned my addr 172.21.15.31:0/761600489 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:19.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.406+0000 7f12cbe00640 1 --2- 172.21.15.31:0/761600489 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d787446c0 0x557d787421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:19.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.406+0000 7f12cb400640 1 -- 172.21.15.31:0/761600489 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d787446c0 msgr2=0x557d787421e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:19.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.406+0000 7f12cb400640 1 --2- 172.21.15.31:0/761600489 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d787446c0 0x557d787421e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.406+0000 7f12cb400640 1 -- 172.21.15.31:0/761600489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7873b8f0 msgr2=0x557d78741ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:19.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.406+0000 7f12cb400640 1 --2- 172.21.15.31:0/761600489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7873b8f0 0x557d78741ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.406+0000 7f12cb400640 1 -- 172.21.15.31:0/761600489 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d783b4210 con 0x557d7873afd0 2024-03-21T10:47:19.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.406+0000 7f12c2a00640 1 --2- 172.21.15.31:0/761600489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7873b8f0 0x557d78741ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:19.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.407+0000 7f12cb400640 1 --2- 172.21.15.31:0/761600489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557d7873afd0 0x557d78741760 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f12bc009cd0 tx=0x7f12bc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f6d617c2e825c62 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:19.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.407+0000 7f12b8c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:47:19.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.407+0000 7f12b8c00640 1 -- 172.21.15.31:0/761600489 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12bc01f640 con 0x557d7873afd0 2024-03-21T10:47:19.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.407+0000 7f12b8c00640 1 -- 172.21.15.31:0/761600489 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f12bc01f7e0 con 0x557d7873afd0 2024-03-21T10:47:19.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.407+0000 7f12d5022740 1 -- 172.21.15.31:0/761600489 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f12c403a040 con 0x557d7873afd0 2024-03-21T10:47:19.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.407+0000 7f12b8c00640 1 -- 172.21.15.31:0/761600489 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f12bc01f9e0 con 0x557d7873afd0 2024-03-21T10:47:19.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.407+0000 7f12d5022740 10 client.27779 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:19.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.407+0000 7f12d5022740 1 -- 172.21.15.31:0/761600489 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x557d78480620 con 0x557d7873afd0 2024-03-21T10:47:19.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.407+0000 7f12b8c00640 1 -- 172.21.15.31:0/761600489 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12c403a040 con 0x557d7873afd0 2024-03-21T10:47:19.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.408+0000 7f12b8c00640 1 -- 172.21.15.31:0/761600489 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f12bc027070 con 0x557d7873afd0 2024-03-21T10:47:19.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.408+0000 7f12d5022740 10 client.27779 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:19.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.408+0000 7f12d5022740 10 client.27779 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:47:19.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.408+0000 7f12d5022740 20 client.27779 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:19.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.408+0000 7f12d5022740 1 --2- 172.21.15.31:0/761600489 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x557d78754810 0x557d78756c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:19.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.408+0000 7f12d5022740 4 client.27779 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:47:19.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.408+0000 7f12d5022740 1 -- 172.21.15.31:0/761600489 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557d786acd70 con 0x557d78754810 2024-03-21T10:47:19.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.408+0000 7f12c2a00640 1 --2- 172.21.15.31:0/761600489 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x557d78754810 0x557d78756c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:19.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.408+0000 7f12c2a00640 1 --2- 172.21.15.31:0/761600489 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x557d78754810 0x557d78756c90 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7c5d0d4590de49f server_cookie=cf99653d812921b6 in_seq=0 out_seq=0 2024-03-21T10:47:19.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.409+0000 7f12b8c00640 10 client.27779 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:19.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.409+0000 7f12b8c00640 1 -- 172.21.15.31:0/761600489 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557d786acd70 con 0x557d78754810 2024-03-21T10:47:19.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.409+0000 7f12b8c00640 10 client.27779 handle_command_reply: tid=0 2024-03-21T10:47:19.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.409+0000 7f12d5022740 1 client.27779 shutdown 2024-03-21T10:47:19.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.409+0000 7f12d5022740 1 -- 172.21.15.31:0/761600489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557d7873afd0 msgr2=0x557d78741760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:19.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.409+0000 7f12d5022740 1 --2- 172.21.15.31:0/761600489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557d7873afd0 0x557d78741760 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f12bc009cd0 tx=0x7f12bc004730 comp rx=0 tx=0).stop 2024-03-21T10:47:19.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.410+0000 7f12d5022740 1 -- 172.21.15.31:0/761600489 shutdown_connections 2024-03-21T10:47:19.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.410+0000 7f12d5022740 1 --2- 172.21.15.31:0/761600489 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x557d78754810 0x557d78756c90 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.410+0000 7f12d5022740 1 --2- 172.21.15.31:0/761600489 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d787446c0 0x557d787421e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.410+0000 7f12d5022740 1 --2- 172.21.15.31:0/761600489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7873b8f0 0x557d78741ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.410+0000 7f12d5022740 1 --2- 172.21.15.31:0/761600489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557d7873afd0 0x557d78741760 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.410+0000 7f12d5022740 1 -- 172.21.15.31:0/761600489 >> 172.21.15.31:0/761600489 conn(0x557d787358f0 msgr2=0x557d78737640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.410+0000 7f12b8c00640 0 client.27779 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.410+0000 7f12d5022740 1 -- 172.21.15.31:0/761600489 shutdown_connections 2024-03-21T10:47:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.410+0000 7f12d5022740 1 -- 172.21.15.31:0/761600489 wait complete. 2024-03-21T10:47:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.410+0000 7f12d5022740 20 client.27779 trim_cache size 0 max 16384 2024-03-21T10:47:19.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.425+0000 7f12d5022740 1 Processor -- start 2024-03-21T10:47:19.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.425+0000 7f12d5022740 1 -- start start 2024-03-21T10:47:19.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.426+0000 7f12d5022740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7873afd0 0x557d78743e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:19.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.426+0000 7f12d5022740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557d7873b8f0 0x557d78751140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:19.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.426+0000 7f12d5022740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d787446c0 0x557d787305d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:19.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.426+0000 7f12d5022740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc1b3040 con 0x557d787446c0 2024-03-21T10:47:19.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.426+0000 7f12d5022740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc1b0b40 con 0x557d7873b8f0 2024-03-21T10:47:19.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.426+0000 7f12d5022740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc1aec00 con 0x557d7873afd0 2024-03-21T10:47:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.426+0000 7f12c2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557d7873b8f0 0x557d78751140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.426+0000 7f12cb400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7873afd0 0x557d78743e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.426+0000 7f12c2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557d7873b8f0 0x557d78751140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39562/0 (socket says 172.21.15.31:39562) 2024-03-21T10:47:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.426+0000 7f12c2a00640 1 -- 172.21.15.31:0/2880007342 learned_addr learned my addr 172.21.15.31:0/2880007342 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.426+0000 7f12cbe00640 1 --2- 172.21.15.31:0/2880007342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d787446c0 0x557d787305d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.426+0000 7f12c2a00640 1 -- 172.21.15.31:0/2880007342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7873afd0 msgr2=0x557d78743e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.426+0000 7f12c2a00640 1 --2- 172.21.15.31:0/2880007342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7873afd0 0x557d78743e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.426+0000 7f12c2a00640 1 -- 172.21.15.31:0/2880007342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d787446c0 msgr2=0x557d787305d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.426+0000 7f12c2a00640 1 --2- 172.21.15.31:0/2880007342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d787446c0 0x557d787305d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.426+0000 7f12c2a00640 1 -- 172.21.15.31:0/2880007342 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d78735420 con 0x557d7873b8f0 2024-03-21T10:47:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.426+0000 7f12cb400640 1 --2- 172.21.15.31:0/2880007342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7873afd0 0x557d78743e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.426+0000 7f12cbe00640 1 --2- 172.21.15.31:0/2880007342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d787446c0 0x557d787305d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:47:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.427+0000 7f12c2a00640 1 --2- 172.21.15.31:0/2880007342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557d7873b8f0 0x557d78751140 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f12b4098400 tx=0x7f12b40d23b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41b3a3336a718fbb server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.427+0000 7f12bb400640 1 -- 172.21.15.31:0/2880007342 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b409b900 con 0x557d7873b8f0 2024-03-21T10:47:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.427+0000 7f12bb400640 1 -- 172.21.15.31:0/2880007342 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12b4010e50 con 0x557d7873b8f0 2024-03-21T10:47:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.427+0000 7f12bb400640 1 -- 172.21.15.31:0/2880007342 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b409a330 con 0x557d7873b8f0 2024-03-21T10:47:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.427+0000 7f12d5022740 1 -- 172.21.15.31:0/2880007342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557d7873b8f0 msgr2=0x557d78751140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.427+0000 7f12d5022740 1 --2- 172.21.15.31:0/2880007342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557d7873b8f0 0x557d78751140 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f12b4098400 tx=0x7f12b40d23b0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.427+0000 7f12d5022740 1 -- 172.21.15.31:0/2880007342 shutdown_connections 2024-03-21T10:47:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.427+0000 7f12d5022740 1 --2- 172.21.15.31:0/2880007342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d787446c0 0x557d787305d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.427+0000 7f12d5022740 1 --2- 172.21.15.31:0/2880007342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557d7873b8f0 0x557d78751140 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.427+0000 7f12d5022740 1 --2- 172.21.15.31:0/2880007342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7873afd0 0x557d78743e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.427+0000 7f12d5022740 1 -- 172.21.15.31:0/2880007342 >> 172.21.15.31:0/2880007342 conn(0x557d787358f0 msgr2=0x557d78736cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.428+0000 7f12d5022740 1 -- 172.21.15.31:0/2880007342 shutdown_connections 2024-03-21T10:47:19.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.428+0000 7f12d5022740 1 -- 172.21.15.31:0/2880007342 wait complete. 2024-03-21T10:47:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.428+0000 7f12d5022740 1 -- start start 2024-03-21T10:47:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.428+0000 7f12d5022740 1 Processor -- start 2024-03-21T10:47:19.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.429+0000 7f12d5022740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557d7873afd0 0x557d7874bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:19.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.429+0000 7f12d5022740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7873b8f0 0x557d7874c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:19.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.429+0000 7f12d5022740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d787446c0 0x557d787474a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:19.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.429+0000 7f12d5022740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc1af200 con 0x557d7873b8f0 2024-03-21T10:47:19.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.429+0000 7f12d5022740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc1b04e0 con 0x557d7873afd0 2024-03-21T10:47:19.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.429+0000 7f12d5022740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc1af820 con 0x557d787446c0 2024-03-21T10:47:19.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.429+0000 7f12d5022740 10 client.0 _ll_register_callbacks cb 0x7f12bc009c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:19.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.429+0000 7f12c2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7873b8f0 0x557d7874c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:19.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.429+0000 7f12cbe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d787446c0 0x557d787474a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:19.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.429+0000 7f12cb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557d7873afd0 0x557d7874bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:19.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.429+0000 7f12c2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7873b8f0 0x557d7874c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56144/0 (socket says 172.21.15.31:56144) 2024-03-21T10:47:19.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.429+0000 7f12c2a00640 1 -- 172.21.15.31:0/2702758417 learned_addr learned my addr 172.21.15.31:0/2702758417 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:19.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.429+0000 7f12cbe00640 1 -- 172.21.15.31:0/2702758417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557d7873afd0 msgr2=0x557d7874bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:19.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.429+0000 7f12cbe00640 1 --2- 172.21.15.31:0/2702758417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557d7873afd0 0x557d7874bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.429+0000 7f12cbe00640 1 -- 172.21.15.31:0/2702758417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7873b8f0 msgr2=0x557d7874c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.429+0000 7f12cbe00640 1 --2- 172.21.15.31:0/2702758417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7873b8f0 0x557d7874c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.429+0000 7f12cbe00640 1 -- 172.21.15.31:0/2702758417 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d78747ca0 con 0x557d787446c0 2024-03-21T10:47:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.430+0000 7f12cbe00640 1 --2- 172.21.15.31:0/2702758417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d787446c0 0x557d787474a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f12c4006010 tx=0x7f12c4003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ec9238bd356524b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.430+0000 7f12b8c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:47:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.430+0000 7f12b8c00640 1 -- 172.21.15.31:0/2702758417 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c4039040 con 0x557d787446c0 2024-03-21T10:47:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.430+0000 7f12b8c00640 1 -- 172.21.15.31:0/2702758417 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f12c4032590 con 0x557d787446c0 2024-03-21T10:47:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.430+0000 7f12d5022740 1 -- 172.21.15.31:0/2702758417 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f12b4010e50 con 0x557d787446c0 2024-03-21T10:47:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.430+0000 7f12b8c00640 1 -- 172.21.15.31:0/2702758417 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f12c4045040 con 0x557d787446c0 2024-03-21T10:47:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.430+0000 7f12d5022740 10 client.27706 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.430+0000 7f12d5022740 1 -- 172.21.15.31:0/2702758417 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x557d787b8450 con 0x557d787446c0 2024-03-21T10:47:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.430+0000 7f12b8c00640 1 -- 172.21.15.31:0/2702758417 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12b4010e50 con 0x557d787446c0 2024-03-21T10:47:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.430+0000 7f12b8c00640 1 -- 172.21.15.31:0/2702758417 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f12c4034ce0 con 0x557d787446c0 2024-03-21T10:47:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.431+0000 7f12d5022740 10 client.27706 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.431+0000 7f12d5022740 10 client.27706 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:47:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.431+0000 7f12d5022740 20 client.27706 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.431+0000 7f12d5022740 1 --2- 172.21.15.31:0/2702758417 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x557d787b8730 0x557d787bab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.431+0000 7f12d5022740 4 client.27706 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:47:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.431+0000 7f12d5022740 1 -- 172.21.15.31:0/2702758417 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557d786a70c0 con 0x557d787b8730 2024-03-21T10:47:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.431+0000 7f12cb400640 1 --2- 172.21.15.31:0/2702758417 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x557d787b8730 0x557d787bab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.431+0000 7f12cb400640 1 --2- 172.21.15.31:0/2702758417 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x557d787b8730 0x557d787bab20 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d44b6de1f70bd394 server_cookie=7875f4f2ae72eb0d in_seq=0 out_seq=0 2024-03-21T10:47:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.431+0000 7f12b8c00640 10 client.27706 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:19.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:19 smithi040 ceph-mon[36154]: pgmap v271: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.6 KiB/s wr, 8 op/s 2024-03-21T10:47:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.437+0000 7f12b8c00640 1 -- 172.21.15.31:0/2702758417 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557d786a70c0 con 0x557d787b8730 2024-03-21T10:47:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.437+0000 7f12b8c00640 10 client.27706 handle_command_reply: tid=0 2024-03-21T10:47:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.437+0000 7f12d5022740 1 client.27706 shutdown 2024-03-21T10:47:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.438+0000 7f12d5022740 1 -- 172.21.15.31:0/2702758417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d787446c0 msgr2=0x557d787474a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.438+0000 7f12d5022740 1 --2- 172.21.15.31:0/2702758417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d787446c0 0x557d787474a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f12c4006010 tx=0x7f12c4003b90 comp rx=0 tx=0).stop 2024-03-21T10:47:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.438+0000 7f12d5022740 1 -- 172.21.15.31:0/2702758417 shutdown_connections 2024-03-21T10:47:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.438+0000 7f12d5022740 1 --2- 172.21.15.31:0/2702758417 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x557d787b8730 0x557d787bab20 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.438+0000 7f12d5022740 1 --2- 172.21.15.31:0/2702758417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d787446c0 0x557d787474a0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.438+0000 7f12d5022740 1 --2- 172.21.15.31:0/2702758417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7873b8f0 0x557d7874c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.438+0000 7f12d5022740 1 --2- 172.21.15.31:0/2702758417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557d7873afd0 0x557d7874bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.438+0000 7f12b8c00640 0 client.27706 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.438+0000 7f12d5022740 1 -- 172.21.15.31:0/2702758417 >> 172.21.15.31:0/2702758417 conn(0x557d787358f0 msgr2=0x557d787340c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.438+0000 7f12d5022740 1 -- 172.21.15.31:0/2702758417 shutdown_connections 2024-03-21T10:47:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.438+0000 7f12d5022740 1 -- 172.21.15.31:0/2702758417 wait complete. 2024-03-21T10:47:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.438+0000 7f12d5022740 20 client.27706 trim_cache size 0 max 16384 2024-03-21T10:47:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:47:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:47:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:47:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:47:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.439+0000 7f12d2200640 1 -- 172.21.15.31:0/2876227350 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f12b00a3100 msgr2=0x7f12b00a55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.439+0000 7f12d2200640 1 --2- 172.21.15.31:0/2876227350 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f12b00a3100 0x7f12b00a55c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x557d7873aa80 tx=0x7f12bc005f90 comp rx=0 tx=0).stop 2024-03-21T10:47:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.439+0000 7f12d2200640 1 -- 172.21.15.31:0/2876227350 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12cc104250 msgr2=0x7f12cc19a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.439+0000 7f12d2200640 1 --2- 172.21.15.31:0/2876227350 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12cc104250 0x7f12cc19a650 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f12b400e9f0 tx=0x7f12b400eeb0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.439+0000 7f12d2200640 1 -- 172.21.15.31:0/2876227350 shutdown_connections 2024-03-21T10:47:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.439+0000 7f12d2200640 1 --2- 172.21.15.31:0/2876227350 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f12b00a3100 0x7f12b00a55c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.439+0000 7f12d2200640 1 --2- 172.21.15.31:0/2876227350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12cc1098a0 0x7f12cc19e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.439+0000 7f12d2200640 1 --2- 172.21.15.31:0/2876227350 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12cc104250 0x7f12cc19a650 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.439+0000 7f12d2200640 1 --2- 172.21.15.31:0/2876227350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12cc103890 0x7f12cc19a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.439+0000 7f12d2200640 1 -- 172.21.15.31:0/2876227350 >> 172.21.15.31:0/2876227350 conn(0x7f12cc0ff320 msgr2=0x7f12cc0ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.439+0000 7f12d2200640 1 -- 172.21.15.31:0/2876227350 shutdown_connections 2024-03-21T10:47:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:19.439+0000 7f12d2200640 1 -- 172.21.15.31:0/2876227350 wait complete. 2024-03-21T10:47:19.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:19 smithi031 ceph-mon[31673]: pgmap v271: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.6 KiB/s wr, 8 op/s 2024-03-21T10:47:21.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:21 smithi070 ceph-mon[36330]: pgmap v272: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-03-21T10:47:21.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:21 smithi040 ceph-mon[36154]: pgmap v272: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-03-21T10:47:21.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:21 smithi031 ceph-mon[31673]: pgmap v272: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-03-21T10:47:23.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:23 smithi070 ceph-mon[36330]: pgmap v273: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-03-21T10:47:23.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:23 smithi040 ceph-mon[36154]: pgmap v273: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-03-21T10:47:23.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:23 smithi031 ceph-mon[31673]: pgmap v273: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-03-21T10:47:25.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:25 smithi040 ceph-mon[36154]: pgmap v274: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-03-21T10:47:25.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:25 smithi031 ceph-mon[31673]: pgmap v274: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-03-21T10:47:25.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:25 smithi070 ceph-mon[36330]: pgmap v274: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-03-21T10:47:27.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:27 smithi040 ceph-mon[36154]: pgmap v275: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-03-21T10:47:27.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:27 smithi031 ceph-mon[31673]: pgmap v275: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-03-21T10:47:27.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:27 smithi070 ceph-mon[36330]: pgmap v275: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-03-21T10:47:29.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:29 smithi040 ceph-mon[36154]: pgmap v276: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2024-03-21T10:47:29.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:29 smithi031 ceph-mon[31673]: pgmap v276: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2024-03-21T10:47:29.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:29 smithi070 ceph-mon[36330]: pgmap v276: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2024-03-21T10:47:31.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:31 smithi040 ceph-mon[36154]: pgmap v277: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-21T10:47:31.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:31 smithi031 ceph-mon[31673]: pgmap v277: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-21T10:47:31.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:31 smithi070 ceph-mon[36330]: pgmap v277: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-21T10:47:32.101 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:47:32.101 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:47:32.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.325+0000 7f99c3000640 1 -- 172.21.15.31:0/1528737605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99bc10dd40 msgr2=0x7f99bc110130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:32.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.325+0000 7f99c3000640 1 --2- 172.21.15.31:0/1528737605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99bc10dd40 0x7f99bc110130 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f99b4009ca0 tx=0x7f99b402c190 comp rx=0 tx=0).stop 2024-03-21T10:47:32.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.325+0000 7f99c3000640 1 -- 172.21.15.31:0/1528737605 shutdown_connections 2024-03-21T10:47:32.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.325+0000 7f99c3000640 1 --2- 172.21.15.31:0/1528737605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99bc10dd40 0x7f99bc110130 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.325+0000 7f99c3000640 1 --2- 172.21.15.31:0/1528737605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99bc074370 0x7f99bc10d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.325+0000 7f99c3000640 1 --2- 172.21.15.31:0/1528737605 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99bc073a50 0x7f99bc073e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.325+0000 7f99c3000640 1 -- 172.21.15.31:0/1528737605 >> 172.21.15.31:0/1528737605 conn(0x7f99bc0fcfb0 msgr2=0x7f99bc0ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:32.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.325+0000 7f99c3000640 1 -- 172.21.15.31:0/1528737605 shutdown_connections 2024-03-21T10:47:32.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.325+0000 7f99c3000640 1 -- 172.21.15.31:0/1528737605 wait complete. 2024-03-21T10:47:32.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.326+0000 7f99c3000640 1 Processor -- start 2024-03-21T10:47:32.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.326+0000 7f99c3000640 1 -- start start 2024-03-21T10:47:32.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.326+0000 7f99c3000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99bc073a50 0x7f99bc199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.326+0000 7f99c3000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99bc074370 0x7f99bc19a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.326+0000 7f99c3000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99bc10dd40 0x7f99bc19e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.326+0000 7f99c3000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99bc113a60 con 0x7f99bc074370 2024-03-21T10:47:32.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.326+0000 7f99c3000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99bc1138e0 con 0x7f99bc073a50 2024-03-21T10:47:32.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.326+0000 7f99c3000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99bc113be0 con 0x7f99bc10dd40 2024-03-21T10:47:32.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.327+0000 7f99c1200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99bc074370 0x7f99bc19a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.327+0000 7f99c1c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99bc073a50 0x7f99bc199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.327+0000 7f99c1c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99bc073a50 0x7f99bc199f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33040/0 (socket says 172.21.15.31:33040) 2024-03-21T10:47:32.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.327+0000 7f99c1c00640 1 -- 172.21.15.31:0/2193944931 learned_addr learned my addr 172.21.15.31:0/2193944931 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:32.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.327+0000 7f99c2600640 1 --2- 172.21.15.31:0/2193944931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99bc10dd40 0x7f99bc19e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.327+0000 7f99c1c00640 1 -- 172.21.15.31:0/2193944931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99bc10dd40 msgr2=0x7f99bc19e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:32.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.327+0000 7f99c1c00640 1 --2- 172.21.15.31:0/2193944931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99bc10dd40 0x7f99bc19e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.327+0000 7f99c1c00640 1 -- 172.21.15.31:0/2193944931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99bc074370 msgr2=0x7f99bc19a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:32.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.327+0000 7f99c1c00640 1 --2- 172.21.15.31:0/2193944931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99bc074370 0x7f99bc19a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.327+0000 7f99c1c00640 1 -- 172.21.15.31:0/2193944931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99bc19ef30 con 0x7f99bc073a50 2024-03-21T10:47:32.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.327+0000 7f99c2600640 1 --2- 172.21.15.31:0/2193944931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99bc10dd40 0x7f99bc19e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:32.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.328+0000 7f99c1c00640 1 --2- 172.21.15.31:0/2193944931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99bc073a50 0x7f99bc199f40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f99b000e9f0 tx=0x7f99b000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:32.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.328+0000 7f99ba000640 1 -- 172.21.15.31:0/2193944931 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b000cd80 con 0x7f99bc073a50 2024-03-21T10:47:32.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.328+0000 7f99ba000640 1 -- 172.21.15.31:0/2193944931 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f99b0010910 con 0x7f99bc073a50 2024-03-21T10:47:32.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.328+0000 7f99ba000640 1 -- 172.21.15.31:0/2193944931 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b0018990 con 0x7f99bc073a50 2024-03-21T10:47:32.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.328+0000 7f99c3000640 1 -- 172.21.15.31:0/2193944931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99bc19f220 con 0x7f99bc073a50 2024-03-21T10:47:32.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.328+0000 7f99c3000640 1 -- 172.21.15.31:0/2193944931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99bc1a6ab0 con 0x7f99bc073a50 2024-03-21T10:47:32.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.331+0000 7f99ba000640 1 -- 172.21.15.31:0/2193944931 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f99b0010ac0 con 0x7f99bc073a50 2024-03-21T10:47:32.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.331+0000 7f99c737f740 1 Processor -- start 2024-03-21T10:47:32.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.331+0000 7f99c737f740 1 -- start start 2024-03-21T10:47:32.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.332+0000 7f99c737f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e528dd0 0x55cd0e527b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.332+0000 7f99c737f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd0e5280f0 0x55cd0e528570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.332+0000 7f99c737f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd0e531460 0x55cd0e533850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.332+0000 7f99c737f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd0e48c250 con 0x55cd0e531460 2024-03-21T10:47:32.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.332+0000 7f99c737f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd0e434a30 con 0x55cd0e528dd0 2024-03-21T10:47:32.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.332+0000 7f99c737f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd0e4882f0 con 0x55cd0e5280f0 2024-03-21T10:47:32.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.332+0000 7f99c2600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e528dd0 0x55cd0e527b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.332+0000 7f99c1c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd0e5280f0 0x55cd0e528570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.332+0000 7f99c2600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e528dd0 0x55cd0e527b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33054/0 (socket says 172.21.15.31:33054) 2024-03-21T10:47:32.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.332+0000 7f99c2600640 1 -- 172.21.15.31:0/711726687 learned_addr learned my addr 172.21.15.31:0/711726687 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:32.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.332+0000 7f99c1200640 1 --2- 172.21.15.31:0/711726687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd0e531460 0x55cd0e533850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.333+0000 7f99c2600640 1 -- 172.21.15.31:0/711726687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd0e5280f0 msgr2=0x55cd0e528570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:32.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.333+0000 7f99c2600640 1 --2- 172.21.15.31:0/711726687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd0e5280f0 0x55cd0e528570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.333+0000 7f99c2600640 1 -- 172.21.15.31:0/711726687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd0e531460 msgr2=0x55cd0e533850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:32.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.333+0000 7f99c2600640 1 --2- 172.21.15.31:0/711726687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd0e531460 0x55cd0e533850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.333+0000 7f99c2600640 1 -- 172.21.15.31:0/711726687 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd0e41f690 con 0x55cd0e528dd0 2024-03-21T10:47:32.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.333+0000 7f99c1c00640 1 --2- 172.21.15.31:0/711726687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd0e5280f0 0x55cd0e528570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:32.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.333+0000 7f99c2600640 1 --2- 172.21.15.31:0/711726687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e528dd0 0x55cd0e527b20 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f99b4009ca0 tx=0x7f99b4009900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76a19629fa8ea812 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:32.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.333+0000 7f99ba000640 1 --2- 172.21.15.31:0/2193944931 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f99a40a3110 0x7f99a40a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.333+0000 7f99aea00640 1 -- 172.21.15.31:0/711726687 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b4043070 con 0x55cd0e528dd0 2024-03-21T10:47:32.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.333+0000 7f99aea00640 1 -- 172.21.15.31:0/711726687 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f99b4003e50 con 0x55cd0e528dd0 2024-03-21T10:47:32.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.333+0000 7f99aea00640 1 -- 172.21.15.31:0/711726687 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b4039040 con 0x55cd0e528dd0 2024-03-21T10:47:32.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.333+0000 7f99ba000640 1 -- 172.21.15.31:0/2193944931 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f99b00cfe70 con 0x7f99bc073a50 2024-03-21T10:47:32.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.334+0000 7f99c1c00640 1 --2- 172.21.15.31:0/2193944931 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f99a40a3110 0x7f99a40a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.334+0000 7f99c737f740 1 -- 172.21.15.31:0/711726687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e528dd0 msgr2=0x55cd0e527b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:32.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.334+0000 7f99c737f740 1 --2- 172.21.15.31:0/711726687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e528dd0 0x55cd0e527b20 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f99b4009ca0 tx=0x7f99b4009900 comp rx=0 tx=0).stop 2024-03-21T10:47:32.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.334+0000 7f99c737f740 1 -- 172.21.15.31:0/711726687 shutdown_connections 2024-03-21T10:47:32.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.334+0000 7f99c737f740 1 --2- 172.21.15.31:0/711726687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd0e531460 0x55cd0e533850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.334+0000 7f99c737f740 1 --2- 172.21.15.31:0/711726687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd0e5280f0 0x55cd0e528570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.334+0000 7f99c737f740 1 --2- 172.21.15.31:0/711726687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e528dd0 0x55cd0e527b20 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.334+0000 7f99c737f740 1 -- 172.21.15.31:0/711726687 >> 172.21.15.31:0/711726687 conn(0x55cd0e4ef200 msgr2=0x55cd0e522b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:32.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.334+0000 7f99c737f740 1 -- 172.21.15.31:0/711726687 shutdown_connections 2024-03-21T10:47:32.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.334+0000 7f99c1c00640 1 --2- 172.21.15.31:0/2193944931 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f99a40a3110 0x7f99a40a55d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x55cd0e529320 tx=0x7f99b00d3920 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:32.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.334+0000 7f99c737f740 1 -- 172.21.15.31:0/711726687 wait complete. 2024-03-21T10:47:32.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.335+0000 7f99c737f740 1 -- start start 2024-03-21T10:47:32.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.335+0000 7f99c737f740 1 Processor -- start 2024-03-21T10:47:32.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.335+0000 7f99c737f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd0e5280f0 0x55cd0e530b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.335+0000 7f99c737f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e528dd0 0x55cd0e52dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.336+0000 7f99c737f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd0e531460 0x55cd0e52e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.336+0000 7f99c737f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd0e534270 con 0x55cd0e531460 2024-03-21T10:47:32.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.336+0000 7f99c737f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd0e447e80 con 0x55cd0e528dd0 2024-03-21T10:47:32.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.336+0000 7f99c737f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd0e5343f0 con 0x55cd0e5280f0 2024-03-21T10:47:32.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.336+0000 7f99c737f740 10 client.0 _ll_register_callbacks cb 0x55cd0e45a130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:32.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.336+0000 7f99c1200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e528dd0 0x55cd0e52dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.336+0000 7f99c1200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e528dd0 0x55cd0e52dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33058/0 (socket says 172.21.15.31:33058) 2024-03-21T10:47:32.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.336+0000 7f99c1200640 1 -- 172.21.15.31:0/1790995118 learned_addr learned my addr 172.21.15.31:0/1790995118 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:32.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.336+0000 7f99c2600640 1 --2- 172.21.15.31:0/1790995118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd0e5280f0 0x55cd0e530b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.336+0000 7f99c2600640 1 --2- 172.21.15.31:0/1790995118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd0e531460 0x55cd0e52e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.336+0000 7f99c1200640 1 -- 172.21.15.31:0/1790995118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd0e5280f0 msgr2=0x55cd0e530b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:32.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.336+0000 7f99c1200640 1 --2- 172.21.15.31:0/1790995118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd0e5280f0 0x55cd0e530b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.336+0000 7f99c1200640 1 -- 172.21.15.31:0/1790995118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd0e531460 msgr2=0x55cd0e52e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:32.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.336+0000 7f99c1200640 1 --2- 172.21.15.31:0/1790995118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd0e531460 0x55cd0e52e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.336+0000 7f99c1200640 1 -- 172.21.15.31:0/1790995118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cd0e26a7d0 con 0x55cd0e528dd0 2024-03-21T10:47:32.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.336+0000 7f99c2600640 1 --2- 172.21.15.31:0/1790995118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd0e531460 0x55cd0e52e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:47:32.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.337+0000 7f99c1200640 1 --2- 172.21.15.31:0/1790995118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e528dd0 0x55cd0e52dee0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f99a800bf40 tx=0x7f99a80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97c801dfb8c891fc server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:32.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.337+0000 7f99a3400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:47:32.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.337+0000 7f99a3400640 1 -- 172.21.15.31:0/1790995118 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a8004020 con 0x55cd0e528dd0 2024-03-21T10:47:32.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.337+0000 7f99a3400640 1 -- 172.21.15.31:0/1790995118 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f99a80041c0 con 0x55cd0e528dd0 2024-03-21T10:47:32.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.337+0000 7f99c737f740 1 -- 172.21.15.31:0/1790995118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f99b4003e50 con 0x55cd0e528dd0 2024-03-21T10:47:32.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.337+0000 7f99a3400640 1 -- 172.21.15.31:0/1790995118 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99a8024050 con 0x55cd0e528dd0 2024-03-21T10:47:32.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.337+0000 7f99c737f740 10 client.27797 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:32.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.337+0000 7f99c737f740 1 -- 172.21.15.31:0/1790995118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55cd0e1a03f0 con 0x55cd0e528dd0 2024-03-21T10:47:32.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.337+0000 7f99a3400640 1 -- 172.21.15.31:0/1790995118 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f99b4003e50 con 0x55cd0e528dd0 2024-03-21T10:47:32.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.338+0000 7f99a3400640 1 -- 172.21.15.31:0/1790995118 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f99a8004c60 con 0x55cd0e528dd0 2024-03-21T10:47:32.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.338+0000 7f99c737f740 10 client.27797 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:32.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.338+0000 7f99c737f740 10 client.27797 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:47:32.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.338+0000 7f99c737f740 20 client.27797 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:32.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.338+0000 7f99c737f740 1 --2- 172.21.15.31:0/1790995118 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55cd0e5413e0 0x55cd0e5437d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.338+0000 7f99c737f740 4 client.27797 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:47:32.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.338+0000 7f99c737f740 1 -- 172.21.15.31:0/1790995118 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cd0e499790 con 0x55cd0e5413e0 2024-03-21T10:47:32.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.338+0000 7f99c2600640 1 --2- 172.21.15.31:0/1790995118 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55cd0e5413e0 0x55cd0e5437d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.338+0000 7f99c2600640 1 --2- 172.21.15.31:0/1790995118 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55cd0e5413e0 0x55cd0e5437d0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3511d8af7940404 server_cookie=6315691e18fc6a8d in_seq=0 out_seq=0 2024-03-21T10:47:32.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.338+0000 7f99a3400640 10 client.27797 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:32.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.339+0000 7f99a3400640 1 -- 172.21.15.31:0/1790995118 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cd0e499790 con 0x55cd0e5413e0 2024-03-21T10:47:32.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.339+0000 7f99a3400640 10 client.27797 handle_command_reply: tid=0 2024-03-21T10:47:32.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.339+0000 7f99c737f740 1 client.27797 shutdown 2024-03-21T10:47:32.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.339+0000 7f99c737f740 1 -- 172.21.15.31:0/1790995118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e528dd0 msgr2=0x55cd0e52dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:32.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.339+0000 7f99c737f740 1 --2- 172.21.15.31:0/1790995118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e528dd0 0x55cd0e52dee0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f99a800bf40 tx=0x7f99a80118d0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.340+0000 7f99c737f740 1 -- 172.21.15.31:0/1790995118 shutdown_connections 2024-03-21T10:47:32.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.340+0000 7f99c737f740 1 --2- 172.21.15.31:0/1790995118 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55cd0e5413e0 0x55cd0e5437d0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.340+0000 7f99c737f740 1 --2- 172.21.15.31:0/1790995118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd0e531460 0x55cd0e52e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.340+0000 7f99c737f740 1 --2- 172.21.15.31:0/1790995118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e528dd0 0x55cd0e52dee0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.340+0000 7f99c737f740 1 --2- 172.21.15.31:0/1790995118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd0e5280f0 0x55cd0e530b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.340+0000 7f99a3400640 0 client.27797 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:32.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.340+0000 7f99c737f740 1 -- 172.21.15.31:0/1790995118 >> 172.21.15.31:0/1790995118 conn(0x55cd0e4ef200 msgr2=0x55cd0e524890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:32.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.340+0000 7f99c737f740 1 -- 172.21.15.31:0/1790995118 shutdown_connections 2024-03-21T10:47:32.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.340+0000 7f99c737f740 1 -- 172.21.15.31:0/1790995118 wait complete. 2024-03-21T10:47:32.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.340+0000 7f99c737f740 20 client.27797 trim_cache size 0 max 16384 2024-03-21T10:47:32.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.357+0000 7f99c737f740 1 Processor -- start 2024-03-21T10:47:32.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.358+0000 7f99c737f740 1 -- start start 2024-03-21T10:47:32.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.358+0000 7f99c737f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd0e5280f0 0x55cd0e530610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.358+0000 7f99c737f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e528dd0 0x55cd0e530b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.358+0000 7f99c737f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd0e531460 0x55cd0e532850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.358+0000 7f99c737f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99bc1b2f90 con 0x55cd0e5280f0 2024-03-21T10:47:32.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.358+0000 7f99c737f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99bc1b0a90 con 0x55cd0e528dd0 2024-03-21T10:47:32.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.358+0000 7f99c737f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99bc1aeb50 con 0x55cd0e531460 2024-03-21T10:47:32.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.358+0000 7f99c1200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e528dd0 0x55cd0e530b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.358+0000 7f99c2600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd0e5280f0 0x55cd0e530610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.358+0000 7f99c1200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e528dd0 0x55cd0e530b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33072/0 (socket says 172.21.15.31:33072) 2024-03-21T10:47:32.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.358+0000 7f99c1200640 1 -- 172.21.15.31:0/1449360143 learned_addr learned my addr 172.21.15.31:0/1449360143 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:32.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.358+0000 7f99c2600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd0e5280f0 0x55cd0e530610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53872/0 (socket says 172.21.15.31:53872) 2024-03-21T10:47:32.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.358+0000 7f99c2600640 1 --2- 172.21.15.31:0/1449360143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd0e531460 0x55cd0e532850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.358+0000 7f99c1200640 1 -- 172.21.15.31:0/1449360143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd0e531460 msgr2=0x55cd0e532850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:32.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.359+0000 7f99c1200640 1 --2- 172.21.15.31:0/1449360143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd0e531460 0x55cd0e532850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.359+0000 7f99c1200640 1 -- 172.21.15.31:0/1449360143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd0e5280f0 msgr2=0x55cd0e530610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:32.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.359+0000 7f99c1200640 1 --2- 172.21.15.31:0/1449360143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd0e5280f0 0x55cd0e530610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.359+0000 7f99c1200640 1 -- 172.21.15.31:0/1449360143 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd0e1600f0 con 0x55cd0e528dd0 2024-03-21T10:47:32.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.359+0000 7f99c2600640 1 --2- 172.21.15.31:0/1449360143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd0e531460 0x55cd0e532850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:47:32.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.359+0000 7f99c1200640 1 --2- 172.21.15.31:0/1449360143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e528dd0 0x55cd0e530b50 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f99a8010780 tx=0x7f99a8011dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f46e15ec7d0e28b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:32.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.359+0000 7f99ae000640 1 -- 172.21.15.31:0/1449360143 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a8010d20 con 0x55cd0e528dd0 2024-03-21T10:47:32.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.359+0000 7f99ae000640 1 -- 172.21.15.31:0/1449360143 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f99a80054e0 con 0x55cd0e528dd0 2024-03-21T10:47:32.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.359+0000 7f99ae000640 1 -- 172.21.15.31:0/1449360143 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a8004ae0 con 0x55cd0e528dd0 2024-03-21T10:47:32.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.360+0000 7f99c737f740 1 -- 172.21.15.31:0/1449360143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e528dd0 msgr2=0x55cd0e530b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:32.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.360+0000 7f99c737f740 1 --2- 172.21.15.31:0/1449360143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e528dd0 0x55cd0e530b50 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f99a8010780 tx=0x7f99a8011dd0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.360+0000 7f99c737f740 1 -- 172.21.15.31:0/1449360143 shutdown_connections 2024-03-21T10:47:32.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.360+0000 7f99c737f740 1 --2- 172.21.15.31:0/1449360143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd0e531460 0x55cd0e532850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.360+0000 7f99c737f740 1 --2- 172.21.15.31:0/1449360143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e528dd0 0x55cd0e530b50 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.360+0000 7f99c737f740 1 --2- 172.21.15.31:0/1449360143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd0e5280f0 0x55cd0e530610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.360+0000 7f99c737f740 1 -- 172.21.15.31:0/1449360143 >> 172.21.15.31:0/1449360143 conn(0x55cd0e4ef200 msgr2=0x55cd0e522670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:32.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.360+0000 7f99c737f740 1 -- 172.21.15.31:0/1449360143 shutdown_connections 2024-03-21T10:47:32.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.360+0000 7f99c737f740 1 -- 172.21.15.31:0/1449360143 wait complete. 2024-03-21T10:47:32.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.360+0000 7f99c737f740 1 -- start start 2024-03-21T10:47:32.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.361+0000 7f99c737f740 1 Processor -- start 2024-03-21T10:47:32.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.361+0000 7f99c737f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd0e5280f0 0x55cd0e539300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.361+0000 7f99c737f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd0e528dd0 0x55cd0e5346c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.361+0000 7f99c737f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e531460 0x55cd0e534c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.361+0000 7f99c737f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99bc1af150 con 0x55cd0e5280f0 2024-03-21T10:47:32.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.361+0000 7f99c737f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99bc1b0430 con 0x55cd0e531460 2024-03-21T10:47:32.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.361+0000 7f99c737f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99bc1af770 con 0x55cd0e528dd0 2024-03-21T10:47:32.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.361+0000 7f99c737f740 10 client.0 _ll_register_callbacks cb 0x7f99a800bb50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:32.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.361+0000 7f99c1200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd0e528dd0 0x55cd0e5346c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.361+0000 7f99c1200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd0e528dd0 0x55cd0e5346c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38398/0 (socket says 172.21.15.31:38398) 2024-03-21T10:47:32.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.361+0000 7f99c1200640 1 -- 172.21.15.31:0/680631607 learned_addr learned my addr 172.21.15.31:0/680631607 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:32.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.362+0000 7f99c2600640 1 --2- 172.21.15.31:0/680631607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd0e5280f0 0x55cd0e539300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.362+0000 7f99c2600640 1 --2- 172.21.15.31:0/680631607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e531460 0x55cd0e534c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.362+0000 7f99c1200640 1 -- 172.21.15.31:0/680631607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e531460 msgr2=0x55cd0e534c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:32.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.362+0000 7f99c1200640 1 --2- 172.21.15.31:0/680631607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e531460 0x55cd0e534c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.362+0000 7f99c1200640 1 -- 172.21.15.31:0/680631607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd0e5280f0 msgr2=0x55cd0e539300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:32.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.362+0000 7f99c1200640 1 --2- 172.21.15.31:0/680631607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd0e5280f0 0x55cd0e539300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.362+0000 7f99c1200640 1 -- 172.21.15.31:0/680631607 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cd0e535240 con 0x55cd0e528dd0 2024-03-21T10:47:32.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.362+0000 7f99c2600640 1 --2- 172.21.15.31:0/680631607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e531460 0x55cd0e534c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:47:32.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.362+0000 7f99c1200640 1 --2- 172.21.15.31:0/680631607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd0e528dd0 0x55cd0e5346c0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f99a8000c00 tx=0x7f99a8010f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65e264cccd07795d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:32.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.362+0000 7f99a3400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:47:32.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.362+0000 7f99a3400640 1 -- 172.21.15.31:0/680631607 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a8005000 con 0x55cd0e528dd0 2024-03-21T10:47:32.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.362+0000 7f99a3400640 1 -- 172.21.15.31:0/680631607 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f99a80051a0 con 0x55cd0e528dd0 2024-03-21T10:47:32.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.362+0000 7f99c737f740 1 -- 172.21.15.31:0/680631607 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f99a80054e0 con 0x55cd0e528dd0 2024-03-21T10:47:32.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.363+0000 7f99a3400640 1 -- 172.21.15.31:0/680631607 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99a8042050 con 0x55cd0e528dd0 2024-03-21T10:47:32.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.363+0000 7f99c737f740 10 client.27724 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:32.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.363+0000 7f99c737f740 1 -- 172.21.15.31:0/680631607 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55cd0e5a5330 con 0x55cd0e528dd0 2024-03-21T10:47:32.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.363+0000 7f99a3400640 1 -- 172.21.15.31:0/680631607 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f99a80054e0 con 0x55cd0e528dd0 2024-03-21T10:47:32.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.363+0000 7f99a3400640 1 -- 172.21.15.31:0/680631607 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f99a801b8d0 con 0x55cd0e528dd0 2024-03-21T10:47:32.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.363+0000 7f99c737f740 10 client.27724 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:32.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.363+0000 7f99c737f740 10 client.27724 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:47:32.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.363+0000 7f99c737f740 20 client.27724 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:32.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.363+0000 7f99c737f740 1 --2- 172.21.15.31:0/680631607 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55cd0e5a59f0 0x55cd0e5a7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.363+0000 7f99c737f740 4 client.27724 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:47:32.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.363+0000 7f99c737f740 1 -- 172.21.15.31:0/680631607 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55cd0e493ae0 con 0x55cd0e5a59f0 2024-03-21T10:47:32.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.364+0000 7f99c2600640 1 --2- 172.21.15.31:0/680631607 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55cd0e5a59f0 0x55cd0e5a7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.364+0000 7f99c2600640 1 --2- 172.21.15.31:0/680631607 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55cd0e5a59f0 0x55cd0e5a7e10 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3aed631a8255f4e7 server_cookie=40e1afcc3e337e1e in_seq=0 out_seq=0 2024-03-21T10:47:32.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.364+0000 7f99a3400640 10 client.27724 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:32.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.364+0000 7f99a3400640 1 -- 172.21.15.31:0/680631607 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55cd0e493ae0 con 0x55cd0e5a59f0 2024-03-21T10:47:32.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.364+0000 7f99a3400640 10 client.27724 handle_command_reply: tid=0 2024-03-21T10:47:32.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.364+0000 7f99c737f740 1 client.27724 shutdown 2024-03-21T10:47:32.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.365+0000 7f99c737f740 1 -- 172.21.15.31:0/680631607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd0e528dd0 msgr2=0x55cd0e5346c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:32.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.365+0000 7f99c737f740 1 --2- 172.21.15.31:0/680631607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd0e528dd0 0x55cd0e5346c0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f99a8000c00 tx=0x7f99a8010f00 comp rx=0 tx=0).stop 2024-03-21T10:47:32.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.365+0000 7f99c737f740 1 -- 172.21.15.31:0/680631607 shutdown_connections 2024-03-21T10:47:32.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.365+0000 7f99c737f740 1 --2- 172.21.15.31:0/680631607 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55cd0e5a59f0 0x55cd0e5a7e10 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.365+0000 7f99c737f740 1 --2- 172.21.15.31:0/680631607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd0e531460 0x55cd0e534c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.365+0000 7f99c737f740 1 --2- 172.21.15.31:0/680631607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cd0e528dd0 0x55cd0e5346c0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.365+0000 7f99a3400640 0 client.27724 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:32.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.365+0000 7f99c737f740 1 --2- 172.21.15.31:0/680631607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd0e5280f0 0x55cd0e539300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.365+0000 7f99c737f740 1 -- 172.21.15.31:0/680631607 >> 172.21.15.31:0/680631607 conn(0x55cd0e4ef200 msgr2=0x55cd0e52c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:32.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.365+0000 7f99c737f740 1 -- 172.21.15.31:0/680631607 shutdown_connections 2024-03-21T10:47:32.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.365+0000 7f99c737f740 1 -- 172.21.15.31:0/680631607 wait complete. 2024-03-21T10:47:32.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.365+0000 7f99c737f740 20 client.27724 trim_cache size 0 max 16384 2024-03-21T10:47:32.367 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:47:32.367 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-03-21T10:47:32.367 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-03-21T10:47:32.367 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:47:32.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.366+0000 7f99c3000640 1 -- 172.21.15.31:0/2193944931 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f99a40a3110 msgr2=0x7f99a40a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:32.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.366+0000 7f99c3000640 1 --2- 172.21.15.31:0/2193944931 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f99a40a3110 0x7f99a40a55d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x55cd0e529320 tx=0x7f99b00d3920 comp rx=0 tx=0).stop 2024-03-21T10:47:32.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.366+0000 7f99c3000640 1 -- 172.21.15.31:0/2193944931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99bc073a50 msgr2=0x7f99bc199f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:32.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.366+0000 7f99c3000640 1 --2- 172.21.15.31:0/2193944931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99bc073a50 0x7f99bc199f40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f99b000e9f0 tx=0x7f99b000eeb0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.366+0000 7f99c3000640 1 -- 172.21.15.31:0/2193944931 shutdown_connections 2024-03-21T10:47:32.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.366+0000 7f99c3000640 1 --2- 172.21.15.31:0/2193944931 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f99a40a3110 0x7f99a40a55d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.366+0000 7f99c3000640 1 --2- 172.21.15.31:0/2193944931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99bc10dd40 0x7f99bc19e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.366+0000 7f99c3000640 1 --2- 172.21.15.31:0/2193944931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99bc074370 0x7f99bc19a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.366+0000 7f99c3000640 1 --2- 172.21.15.31:0/2193944931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99bc073a50 0x7f99bc199f40 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.366+0000 7f99c3000640 1 -- 172.21.15.31:0/2193944931 >> 172.21.15.31:0/2193944931 conn(0x7f99bc0fcfb0 msgr2=0x7f99bc0fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:32.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.366+0000 7f99c3000640 1 -- 172.21.15.31:0/2193944931 shutdown_connections 2024-03-21T10:47:32.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.366+0000 7f99c3000640 1 -- 172.21.15.31:0/2193944931 wait complete. 2024-03-21T10:47:32.383 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-21T10:47:32.384 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:47:32.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.583+0000 7fe585e00640 1 -- 172.21.15.31:0/1986812793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe580075bd0 msgr2=0x7fe580075fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:32.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.583+0000 7fe585e00640 1 --2- 172.21.15.31:0/1986812793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe580075bd0 0x7fe580075fb0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fe570009890 tx=0x7fe57002bb70 comp rx=0 tx=0).stop 2024-03-21T10:47:32.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.583+0000 7fe585e00640 1 -- 172.21.15.31:0/1986812793 shutdown_connections 2024-03-21T10:47:32.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.583+0000 7fe585e00640 1 --2- 172.21.15.31:0/1986812793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe580112500 0x7fe5801148f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.583+0000 7fe585e00640 1 --2- 172.21.15.31:0/1986812793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5800764f0 0x7fe580111fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.583+0000 7fe585e00640 1 --2- 172.21.15.31:0/1986812793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe580075bd0 0x7fe580075fb0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.583+0000 7fe585e00640 1 -- 172.21.15.31:0/1986812793 >> 172.21.15.31:0/1986812793 conn(0x7fe5800ff350 msgr2=0x7fe580101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:32.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.583+0000 7fe585e00640 1 -- 172.21.15.31:0/1986812793 shutdown_connections 2024-03-21T10:47:32.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.583+0000 7fe585e00640 1 -- 172.21.15.31:0/1986812793 wait complete. 2024-03-21T10:47:32.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.584+0000 7fe585e00640 1 Processor -- start 2024-03-21T10:47:32.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.584+0000 7fe585e00640 1 -- start start 2024-03-21T10:47:32.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.585+0000 7fe585e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe580075bd0 0x7fe58019e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.585+0000 7fe585e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5800764f0 0x7fe58019eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.585+0000 7fe585e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe580112500 0x7fe5801a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.585+0000 7fe585e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe580118060 con 0x7fe580112500 2024-03-21T10:47:32.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.585+0000 7fe585e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe580117ee0 con 0x7fe5800764f0 2024-03-21T10:47:32.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.585+0000 7fe585e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5801181e0 con 0x7fe580075bd0 2024-03-21T10:47:32.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.585+0000 7fe57ea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe580075bd0 0x7fe58019e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.585+0000 7fe57e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5800764f0 0x7fe58019eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.585+0000 7fe57ea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe580075bd0 0x7fe58019e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38422/0 (socket says 172.21.15.31:38422) 2024-03-21T10:47:32.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.585+0000 7fe57ea00640 1 -- 172.21.15.31:0/1967592274 learned_addr learned my addr 172.21.15.31:0/1967592274 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:32.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.585+0000 7fe57f400640 1 --2- 172.21.15.31:0/1967592274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe580112500 0x7fe5801a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.586+0000 7fe57ea00640 1 -- 172.21.15.31:0/1967592274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5800764f0 msgr2=0x7fe58019eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:32.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.586+0000 7fe57ea00640 1 --2- 172.21.15.31:0/1967592274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5800764f0 0x7fe58019eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.586+0000 7fe57ea00640 1 -- 172.21.15.31:0/1967592274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe580112500 msgr2=0x7fe5801a2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:32.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.586+0000 7fe57ea00640 1 --2- 172.21.15.31:0/1967592274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe580112500 0x7fe5801a2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.586+0000 7fe57ea00640 1 -- 172.21.15.31:0/1967592274 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5801a3570 con 0x7fe580075bd0 2024-03-21T10:47:32.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.586+0000 7fe57f400640 1 --2- 172.21.15.31:0/1967592274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe580112500 0x7fe5801a2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:47:32.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.586+0000 7fe57e000640 1 --2- 172.21.15.31:0/1967592274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5800764f0 0x7fe58019eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:32.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.586+0000 7fe57ea00640 1 --2- 172.21.15.31:0/1967592274 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe580075bd0 0x7fe58019e580 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fe5700042f0 tx=0x7fe570004320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:32.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.586+0000 7fe584a00640 1 -- 172.21.15.31:0/1967592274 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe570041020 con 0x7fe580075bd0 2024-03-21T10:47:32.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.587+0000 7fe584a00640 1 -- 172.21.15.31:0/1967592274 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe57002c9f0 con 0x7fe580075bd0 2024-03-21T10:47:32.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.587+0000 7fe584a00640 1 -- 172.21.15.31:0/1967592274 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe570039040 con 0x7fe580075bd0 2024-03-21T10:47:32.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.587+0000 7fe585e00640 1 -- 172.21.15.31:0/1967592274 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5801a3800 con 0x7fe580075bd0 2024-03-21T10:47:32.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.587+0000 7fe585e00640 1 -- 172.21.15.31:0/1967592274 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5801ab0f0 con 0x7fe580075bd0 2024-03-21T10:47:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.588+0000 7f8cfde00640 1 -- 172.21.15.31:0/1445255029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cf8102910 msgr2=0x7f8cf8102cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.588+0000 7f8cfde00640 1 --2- 172.21.15.31:0/1445255029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cf8102910 0x7f8cf8102cf0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8009900 tx=0x7f8ce802bcd0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.588+0000 7f8cfde00640 1 -- 172.21.15.31:0/1445255029 shutdown_connections 2024-03-21T10:47:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.588+0000 7f8cfde00640 1 --2- 172.21.15.31:0/1445255029 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cf810bc50 0x7f8cf810e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.588+0000 7f8cfde00640 1 --2- 172.21.15.31:0/1445255029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cf8103230 0x7f8cf810b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.588+0000 7f8cfde00640 1 --2- 172.21.15.31:0/1445255029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cf8102910 0x7f8cf8102cf0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.588+0000 7f8cfde00640 1 -- 172.21.15.31:0/1445255029 >> 172.21.15.31:0/1445255029 conn(0x7f8cf80777b0 msgr2=0x7f8cf8077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.588+0000 7f8cfde00640 1 -- 172.21.15.31:0/1445255029 shutdown_connections 2024-03-21T10:47:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.588+0000 7f8cfde00640 1 -- 172.21.15.31:0/1445255029 wait complete. 2024-03-21T10:47:32.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.588+0000 7fe585e00640 1 -- 172.21.15.31:0/1967592274 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe580077310 con 0x7fe580075bd0 2024-03-21T10:47:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.589+0000 7f8cfde00640 1 Processor -- start 2024-03-21T10:47:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.590+0000 7f8cfde00640 1 -- start start 2024-03-21T10:47:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.590+0000 7f8cfde00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cf8102910 0x7f8cf8197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.590+0000 7f8cfde00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cf8103230 0x7f8cf8198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.590+0000 7f8cfde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cf810bc50 0x7f8cf819c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.590+0000 7f8cfde00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf8111aa0 con 0x7f8cf8102910 2024-03-21T10:47:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.590+0000 7f8cfde00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf8111920 con 0x7f8cf810bc50 2024-03-21T10:47:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.590+0000 7f8cfde00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf8111c20 con 0x7f8cf8103230 2024-03-21T10:47:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.590+0000 7f8cf6000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cf8103230 0x7f8cf8198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.590+0000 7f8cf7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cf810bc50 0x7f8cf819c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.590+0000 7f8cf6000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cf8103230 0x7f8cf8198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38438/0 (socket says 172.21.15.31:38438) 2024-03-21T10:47:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.590+0000 7f8cf6000640 1 -- 172.21.15.31:0/1089756168 learned_addr learned my addr 172.21.15.31:0/1089756168 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.590+0000 7f8cf6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cf8102910 0x7f8cf8197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.591+0000 7f8cf6000640 1 -- 172.21.15.31:0/1089756168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cf810bc50 msgr2=0x7f8cf819c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.591+0000 7f8cf6000640 1 --2- 172.21.15.31:0/1089756168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cf810bc50 0x7f8cf819c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.591+0000 7f8cf6000640 1 -- 172.21.15.31:0/1089756168 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cf8102910 msgr2=0x7f8cf8197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.591+0000 7f8cf6000640 1 --2- 172.21.15.31:0/1089756168 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cf8102910 0x7f8cf8197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.591+0000 7f8cf6000640 1 -- 172.21.15.31:0/1089756168 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf819cef0 con 0x7f8cf8103230 2024-03-21T10:47:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.591+0000 7f8cf6000640 1 --2- 172.21.15.31:0/1089756168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cf8103230 0x7f8cf8198440 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8ce400ea60 tx=0x7f8ce400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:32.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.591+0000 7fe584a00640 1 -- 172.21.15.31:0/1967592274 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fe570044020 con 0x7fe580075bd0 2024-03-21T10:47:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.591+0000 7f8cfca00640 1 -- 172.21.15.31:0/1089756168 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce400cde0 con 0x7f8cf8103230 2024-03-21T10:47:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.591+0000 7f8cfca00640 1 -- 172.21.15.31:0/1089756168 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ce40035f0 con 0x7f8cf8103230 2024-03-21T10:47:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.591+0000 7f8cfde00640 1 -- 172.21.15.31:0/1089756168 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cf819d1e0 con 0x7f8cf8103230 2024-03-21T10:47:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.591+0000 7f8cfca00640 1 -- 172.21.15.31:0/1089756168 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce4010670 con 0x7f8cf8103230 2024-03-21T10:47:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.592+0000 7f8cfde00640 1 -- 172.21.15.31:0/1089756168 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cf81a4a70 con 0x7f8cf8103230 2024-03-21T10:47:32.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.593+0000 7fe584a00640 1 --2- 172.21.15.31:0/1967592274 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe5640a3110 0x7fe5640a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.594+0000 7fe584a00640 1 -- 172.21.15.31:0/1967592274 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fe5700f1420 con 0x7fe580075bd0 2024-03-21T10:47:32.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.594+0000 7fe57e000640 1 --2- 172.21.15.31:0/1967592274 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe5640a3110 0x7fe5640a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.594+0000 7fe57e000640 1 --2- 172.21.15.31:0/1967592274 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe5640a3110 0x7fe5640a55d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe56c0096c0 tx=0x7fe56c009260 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.595+0000 7f8d00a6a740 1 Processor -- start 2024-03-21T10:47:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.596+0000 7f8d00a6a740 1 -- start start 2024-03-21T10:47:32.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.596+0000 7f8d00a6a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f3d02afd0 0x558f3d02b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.596+0000 7f8d00a6a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f3d02b8f0 0x558f3d034180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.596+0000 7f8d00a6a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f3d0346c0 0x558f3d036ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.596+0000 7f8d00a6a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f3cfed3b0 con 0x558f3d02b8f0 2024-03-21T10:47:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.596+0000 7f8d00a6a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f3cf8f830 con 0x558f3d02afd0 2024-03-21T10:47:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.596+0000 7f8d00a6a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f3cf385d0 con 0x558f3d0346c0 2024-03-21T10:47:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.596+0000 7f8cf6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f3d02b8f0 0x558f3d034180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.596+0000 7f8cf7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f3d02afd0 0x558f3d02b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.596+0000 7f8cf6000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f3d0346c0 0x558f3d036ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.596+0000 7f8cf7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f3d02afd0 0x558f3d02b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33118/0 (socket says 172.21.15.31:33118) 2024-03-21T10:47:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.596+0000 7f8cf7400640 1 -- 172.21.15.31:0/4230397652 learned_addr learned my addr 172.21.15.31:0/4230397652 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.597+0000 7f8cf7400640 1 -- 172.21.15.31:0/4230397652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f3d0346c0 msgr2=0x558f3d036ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.597+0000 7f8cf7400640 1 --2- 172.21.15.31:0/4230397652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f3d0346c0 0x558f3d036ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.597+0000 7f8cf7400640 1 -- 172.21.15.31:0/4230397652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f3d02b8f0 msgr2=0x558f3d034180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:32.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.597+0000 7f8cf7400640 1 --2- 172.21.15.31:0/4230397652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f3d02b8f0 0x558f3d034180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.597+0000 7f8cf7400640 1 -- 172.21.15.31:0/4230397652 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f3cf502c0 con 0x558f3d02afd0 2024-03-21T10:47:32.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.597+0000 7f8cf7400640 1 --2- 172.21.15.31:0/4230397652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f3d02afd0 0x558f3d02b3b0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f8cf0009f50 tx=0x7f8cf00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c13b6a314288228 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:32.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.597+0000 7f8ce3e00640 1 -- 172.21.15.31:0/4230397652 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf0004d20 con 0x558f3d02afd0 2024-03-21T10:47:32.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.597+0000 7f8ce3e00640 1 -- 172.21.15.31:0/4230397652 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8cf0004050 con 0x558f3d02afd0 2024-03-21T10:47:32.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.597+0000 7f8ce3e00640 1 -- 172.21.15.31:0/4230397652 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf0005280 con 0x558f3d02afd0 2024-03-21T10:47:32.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.598+0000 7fe584a00640 1 -- 172.21.15.31:0/1967592274 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe57004b020 con 0x7fe580075bd0 2024-03-21T10:47:32.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.598+0000 7f8d00a6a740 1 -- 172.21.15.31:0/4230397652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f3d02afd0 msgr2=0x558f3d02b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:32.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.598+0000 7f8d00a6a740 1 --2- 172.21.15.31:0/4230397652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f3d02afd0 0x558f3d02b3b0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f8cf0009f50 tx=0x7f8cf00118d0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.598+0000 7f8d00a6a740 1 -- 172.21.15.31:0/4230397652 shutdown_connections 2024-03-21T10:47:32.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.598+0000 7f8d00a6a740 1 --2- 172.21.15.31:0/4230397652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f3d0346c0 0x558f3d036ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.598+0000 7f8d00a6a740 1 --2- 172.21.15.31:0/4230397652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f3d02b8f0 0x558f3d034180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.598+0000 7f8d00a6a740 1 --2- 172.21.15.31:0/4230397652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f3d02afd0 0x558f3d02b3b0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.598+0000 7f8d00a6a740 1 -- 172.21.15.31:0/4230397652 >> 172.21.15.31:0/4230397652 conn(0x558f3d0258f0 msgr2=0x558f3d025cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.598+0000 7f8d00a6a740 1 -- 172.21.15.31:0/4230397652 shutdown_connections 2024-03-21T10:47:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.598+0000 7f8d00a6a740 1 -- 172.21.15.31:0/4230397652 wait complete. 2024-03-21T10:47:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.599+0000 7f8d00a6a740 1 -- start start 2024-03-21T10:47:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.599+0000 7f8d00a6a740 1 Processor -- start 2024-03-21T10:47:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.599+0000 7f8d00a6a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f3d02afd0 0x558f3d031740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.599+0000 7f8d00a6a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f3d02b8f0 0x558f3d031c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.599+0000 7f8d00a6a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f3d0346c0 0x558f3d0321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.599+0000 7f8d00a6a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f3cf4ba20 con 0x558f3d0346c0 2024-03-21T10:47:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.599+0000 7f8d00a6a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f3cf8b8d0 con 0x558f3d02afd0 2024-03-21T10:47:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.599+0000 7f8d00a6a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f3d0372f0 con 0x558f3d02b8f0 2024-03-21T10:47:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.600+0000 7f8d00a6a740 10 client.0 _ll_register_callbacks cb 0x558f3cf881c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.600+0000 7f8cf6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f3d02b8f0 0x558f3d031c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.600+0000 7f8cf7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f3d02afd0 0x558f3d031740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.600+0000 7f8cf6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f3d02b8f0 0x558f3d031c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38468/0 (socket says 172.21.15.31:38468) 2024-03-21T10:47:32.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.600+0000 7f8cf6a00640 1 -- 172.21.15.31:0/3833920137 learned_addr learned my addr 172.21.15.31:0/3833920137 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:32.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.600+0000 7f8cf6a00640 1 -- 172.21.15.31:0/3833920137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f3d02afd0 msgr2=0x558f3d031740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:32.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.600+0000 7f8cf6a00640 1 --2- 172.21.15.31:0/3833920137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f3d02afd0 0x558f3d031740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.600+0000 7f8cf6a00640 1 -- 172.21.15.31:0/3833920137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f3d0346c0 msgr2=0x558f3d0321c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:47:32.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.600+0000 7f8cf6a00640 1 --2- 172.21.15.31:0/3833920137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f3d0346c0 0x558f3d0321c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.600+0000 7f8cf6a00640 1 -- 172.21.15.31:0/3833920137 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558f3cca4210 con 0x558f3d02b8f0 2024-03-21T10:47:32.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.600+0000 7f8cf7400640 1 --2- 172.21.15.31:0/3833920137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f3d02afd0 0x558f3d031740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:32.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.600+0000 7f8cf6a00640 1 --2- 172.21.15.31:0/3833920137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f3d02b8f0 0x558f3d031c80 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8009230 tx=0x7f8ce8004610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=957f41ef5589029f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:32.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.600+0000 7f8ce0c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:47:32.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.601+0000 7f8ce0c00640 1 -- 172.21.15.31:0/3833920137 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8044070 con 0x558f3d02b8f0 2024-03-21T10:47:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.601+0000 7f8d00a6a740 1 -- 172.21.15.31:0/3833920137 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8cf0004050 con 0x558f3d02b8f0 2024-03-21T10:47:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.601+0000 7f8ce0c00640 1 -- 172.21.15.31:0/3833920137 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f8ce802cb30 con 0x558f3d02b8f0 2024-03-21T10:47:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.601+0000 7f8ce0c00640 1 -- 172.21.15.31:0/3833920137 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ce8051050 con 0x558f3d02b8f0 2024-03-21T10:47:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.601+0000 7f8d00a6a740 10 client.27742 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.601+0000 7f8d00a6a740 1 -- 172.21.15.31:0/3833920137 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x558f3cd70620 con 0x558f3d02b8f0 2024-03-21T10:47:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.601+0000 7f8ce0c00640 1 -- 172.21.15.31:0/3833920137 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8cf0004050 con 0x558f3d02b8f0 2024-03-21T10:47:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.601+0000 7f8ce0c00640 1 -- 172.21.15.31:0/3833920137 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f8ce80053b0 con 0x558f3d02b8f0 2024-03-21T10:47:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.602+0000 7f8d00a6a740 10 client.27742 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.602+0000 7f8d00a6a740 10 client.27742 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:47:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.602+0000 7f8d00a6a740 20 client.27742 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.602+0000 7f8d00a6a740 1 --2- 172.21.15.31:0/3833920137 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558f3d0447b0 0x558f3d046c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.602+0000 7f8d00a6a740 4 client.27742 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:47:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.602+0000 7f8d00a6a740 1 -- 172.21.15.31:0/3833920137 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558f3cf9cd70 con 0x558f3d0447b0 2024-03-21T10:47:32.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.602+0000 7f8cf7400640 1 --2- 172.21.15.31:0/3833920137 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558f3d0447b0 0x558f3d046c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.602+0000 7f8cfca00640 1 -- 172.21.15.31:0/1089756168 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f8ce40037a0 con 0x7f8cf8103230 2024-03-21T10:47:32.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.603+0000 7f8ce0c00640 10 client.27742 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:32.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.603+0000 7f8cf7400640 1 --2- 172.21.15.31:0/3833920137 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558f3d0447b0 0x558f3d046c30 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26f50a4aa71c133 server_cookie=7eb4cbbba485e28f in_seq=0 out_seq=0 2024-03-21T10:47:32.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.603+0000 7f8ce0c00640 1 -- 172.21.15.31:0/3833920137 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558f3cf9cd70 con 0x558f3d0447b0 2024-03-21T10:47:32.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.603+0000 7f8ce0c00640 10 client.27742 handle_command_reply: tid=0 2024-03-21T10:47:32.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.603+0000 7f8d00a6a740 1 client.27742 shutdown 2024-03-21T10:47:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.604+0000 7f8d00a6a740 1 -- 172.21.15.31:0/3833920137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f3d02b8f0 msgr2=0x558f3d031c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.604+0000 7f8d00a6a740 1 --2- 172.21.15.31:0/3833920137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f3d02b8f0 0x558f3d031c80 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8009230 tx=0x7f8ce8004610 comp rx=0 tx=0).stop 2024-03-21T10:47:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.604+0000 7f8d00a6a740 1 -- 172.21.15.31:0/3833920137 shutdown_connections 2024-03-21T10:47:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.604+0000 7f8d00a6a740 1 --2- 172.21.15.31:0/3833920137 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558f3d0447b0 0x558f3d046c30 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.604+0000 7f8d00a6a740 1 --2- 172.21.15.31:0/3833920137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f3d0346c0 0x558f3d0321c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.604+0000 7f8d00a6a740 1 --2- 172.21.15.31:0/3833920137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f3d02b8f0 0x558f3d031c80 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.604+0000 7f8d00a6a740 1 --2- 172.21.15.31:0/3833920137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f3d02afd0 0x558f3d031740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.604+0000 7f8d00a6a740 1 -- 172.21.15.31:0/3833920137 >> 172.21.15.31:0/3833920137 conn(0x558f3d0258f0 msgr2=0x558f3d027640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.604+0000 7f8ce0c00640 0 client.27742 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.604+0000 7f8cfca00640 1 --2- 172.21.15.31:0/1089756168 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f8cdc0a2fa0 0x7f8cdc0a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.604+0000 7f8d00a6a740 1 -- 172.21.15.31:0/3833920137 shutdown_connections 2024-03-21T10:47:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.604+0000 7f8cfca00640 1 -- 172.21.15.31:0/1089756168 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f8ce40cffb0 con 0x7f8cf8103230 2024-03-21T10:47:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.604+0000 7f8d00a6a740 1 -- 172.21.15.31:0/3833920137 wait complete. 2024-03-21T10:47:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.604+0000 7f8d00a6a740 20 client.27742 trim_cache size 0 max 16384 2024-03-21T10:47:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.604+0000 7f8cf6a00640 1 --2- 172.21.15.31:0/1089756168 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f8cdc0a2fa0 0x7f8cdc0a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.606+0000 7f8cf6a00640 1 --2- 172.21.15.31:0/1089756168 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f8cdc0a2fa0 0x7f8cdc0a5460 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8003760 tx=0x7f8ce8005fd0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:32.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.620+0000 7f8d00a6a740 1 Processor -- start 2024-03-21T10:47:32.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.620+0000 7f8d00a6a740 1 -- start start 2024-03-21T10:47:32.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.620+0000 7f8d00a6a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f3d02afd0 0x558f3d033e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.620+0000 7f8d00a6a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f3d02b8f0 0x558f3d031e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.620+0000 7f8d00a6a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f3d0346c0 0x558f3d03fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.620+0000 7f8d00a6a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf81b10c0 con 0x558f3d02b8f0 2024-03-21T10:47:32.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.620+0000 7f8d00a6a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf81aebc0 con 0x558f3d02afd0 2024-03-21T10:47:32.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.620+0000 7f8d00a6a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf81acc80 con 0x558f3d0346c0 2024-03-21T10:47:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.621+0000 7f8cf6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f3d02b8f0 0x558f3d031e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.621+0000 7f8cf6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f3d02afd0 0x558f3d033e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.621+0000 7f8cf7400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f3d0346c0 0x558f3d03fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.621+0000 7f8cf6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f3d02b8f0 0x558f3d031e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53916/0 (socket says 172.21.15.31:53916) 2024-03-21T10:47:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.621+0000 7f8cf6000640 1 -- 172.21.15.31:0/2359383758 learned_addr learned my addr 172.21.15.31:0/2359383758 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.621+0000 7f8cf6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f3d02afd0 0x558f3d033e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33136/0 (socket says 172.21.15.31:33136) 2024-03-21T10:47:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.621+0000 7f8cf7400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f3d0346c0 0x558f3d03fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38472/0 (socket says 172.21.15.31:38472) 2024-03-21T10:47:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.621+0000 7f8cf7400640 1 -- 172.21.15.31:0/2359383758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f3d02afd0 msgr2=0x558f3d033e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.621+0000 7f8cf7400640 1 --2- 172.21.15.31:0/2359383758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f3d02afd0 0x558f3d033e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.621+0000 7f8cf7400640 1 -- 172.21.15.31:0/2359383758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f3d02b8f0 msgr2=0x558f3d031e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.621+0000 7f8cf7400640 1 --2- 172.21.15.31:0/2359383758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f3d02b8f0 0x558f3d031e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.621+0000 7f8cf7400640 1 -- 172.21.15.31:0/2359383758 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f3d036e20 con 0x558f3d0346c0 2024-03-21T10:47:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.621+0000 7f8cf7400640 1 --2- 172.21.15.31:0/2359383758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f3d0346c0 0x558f3d03fa00 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f8cf0000f80 tx=0x7f8cf00096c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57b8714bbd4c048c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.622+0000 7f8ce3400640 1 -- 172.21.15.31:0/2359383758 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf0003970 con 0x558f3d0346c0 2024-03-21T10:47:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.622+0000 7f8ce3400640 1 -- 172.21.15.31:0/2359383758 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8cf0003ff0 con 0x558f3d0346c0 2024-03-21T10:47:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.622+0000 7f8ce3400640 1 -- 172.21.15.31:0/2359383758 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf002bd30 con 0x558f3d0346c0 2024-03-21T10:47:32.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.622+0000 7f8d00a6a740 1 -- 172.21.15.31:0/2359383758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f3d0346c0 msgr2=0x558f3d03fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.622+0000 7f8d00a6a740 1 --2- 172.21.15.31:0/2359383758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f3d0346c0 0x558f3d03fa00 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f8cf0000f80 tx=0x7f8cf00096c0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.623+0000 7f8d00a6a740 1 -- 172.21.15.31:0/2359383758 shutdown_connections 2024-03-21T10:47:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.623+0000 7f8d00a6a740 1 --2- 172.21.15.31:0/2359383758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f3d0346c0 0x558f3d03fa00 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.623+0000 7f8d00a6a740 1 --2- 172.21.15.31:0/2359383758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f3d02b8f0 0x558f3d031e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.623+0000 7f8d00a6a740 1 --2- 172.21.15.31:0/2359383758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f3d02afd0 0x558f3d033e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.623+0000 7f8d00a6a740 1 -- 172.21.15.31:0/2359383758 >> 172.21.15.31:0/2359383758 conn(0x558f3d0258f0 msgr2=0x558f3d031700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.623+0000 7f8d00a6a740 1 -- 172.21.15.31:0/2359383758 shutdown_connections 2024-03-21T10:47:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.623+0000 7f8d00a6a740 1 -- 172.21.15.31:0/2359383758 wait complete. 2024-03-21T10:47:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.623+0000 7f8d00a6a740 1 -- start start 2024-03-21T10:47:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.624+0000 7f8d00a6a740 1 Processor -- start 2024-03-21T10:47:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.624+0000 7f8d00a6a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f3d02afd0 0x558f3d02e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.624+0000 7f8d00a6a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f3d02b8f0 0x558f3d02eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.624+0000 7f8d00a6a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f3d0346c0 0x558f3d09f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.624+0000 7f8d00a6a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf81ad280 con 0x558f3d02b8f0 2024-03-21T10:47:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.624+0000 7f8d00a6a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf81ae560 con 0x558f3d0346c0 2024-03-21T10:47:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.624+0000 7f8d00a6a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf81ad8a0 con 0x558f3d02afd0 2024-03-21T10:47:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.624+0000 7f8d00a6a740 10 client.0 _ll_register_callbacks cb 0x7f8ce8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.624+0000 7f8cf6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f3d02afd0 0x558f3d02e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.624+0000 7f8cf6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f3d02b8f0 0x558f3d02eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.624+0000 7f8cf6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f3d02afd0 0x558f3d02e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38478/0 (socket says 172.21.15.31:38478) 2024-03-21T10:47:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.624+0000 7f8cf6a00640 1 -- 172.21.15.31:0/239844669 learned_addr learned my addr 172.21.15.31:0/239844669 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.624+0000 7f8cf7400640 1 --2- 172.21.15.31:0/239844669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f3d0346c0 0x558f3d09f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.625+0000 7f8cf6a00640 1 -- 172.21.15.31:0/239844669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f3d0346c0 msgr2=0x558f3d09f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.625+0000 7f8cf6a00640 1 --2- 172.21.15.31:0/239844669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f3d0346c0 0x558f3d09f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.625+0000 7f8cf6a00640 1 -- 172.21.15.31:0/239844669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f3d02b8f0 msgr2=0x558f3d02eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.625+0000 7f8cf6a00640 1 --2- 172.21.15.31:0/239844669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f3d02b8f0 0x558f3d02eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.625+0000 7f8cf6a00640 1 -- 172.21.15.31:0/239844669 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558f3d09f6f0 con 0x558f3d02afd0 2024-03-21T10:47:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.625+0000 7f8cf7400640 1 --2- 172.21.15.31:0/239844669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f3d0346c0 0x558f3d09f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:32.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.625+0000 7f8cf6000640 1 --2- 172.21.15.31:0/239844669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f3d02b8f0 0x558f3d02eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.625+0000 7f8cf6a00640 1 --2- 172.21.15.31:0/239844669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f3d02afd0 0x558f3d02e5d0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8ce802c2e0 tx=0x7f8ce802bcd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83bc3d9d67f3790d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.625+0000 7f8ce0c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:47:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.625+0000 7f8ce0c00640 1 -- 172.21.15.31:0/239844669 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8044040 con 0x558f3d02afd0 2024-03-21T10:47:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.625+0000 7f8d00a6a740 1 -- 172.21.15.31:0/239844669 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8cf0003ff0 con 0x558f3d02afd0 2024-03-21T10:47:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.625+0000 7f8ce0c00640 1 -- 172.21.15.31:0/239844669 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f8ce8003e00 con 0x558f3d02afd0 2024-03-21T10:47:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.626+0000 7f8ce0c00640 1 -- 172.21.15.31:0/239844669 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ce8000e60 con 0x558f3d02afd0 2024-03-21T10:47:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.626+0000 7f8d00a6a740 10 client.27748 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.626+0000 7f8d00a6a740 1 -- 172.21.15.31:0/239844669 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x558f3d0a09e0 con 0x558f3d02afd0 2024-03-21T10:47:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.626+0000 7f8ce0c00640 1 -- 172.21.15.31:0/239844669 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8cf0003ff0 con 0x558f3d02afd0 2024-03-21T10:47:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.626+0000 7f8ce0c00640 1 -- 172.21.15.31:0/239844669 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f8ce8005440 con 0x558f3d02afd0 2024-03-21T10:47:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.626+0000 7f8d00a6a740 10 client.27748 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.626+0000 7f8d00a6a740 10 client.27748 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:47:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.626+0000 7f8d00a6a740 20 client.27748 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.626+0000 7f8d00a6a740 1 --2- 172.21.15.31:0/239844669 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558f3d0a10b0 0x558f3d0a34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.626+0000 7f8d00a6a740 4 client.27748 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:47:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.626+0000 7f8d00a6a740 1 -- 172.21.15.31:0/239844669 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558f3cf970c0 con 0x558f3d0a10b0 2024-03-21T10:47:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.627+0000 7f8cf6000640 1 --2- 172.21.15.31:0/239844669 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558f3d0a10b0 0x558f3d0a34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.627+0000 7f8cf6000640 1 --2- 172.21.15.31:0/239844669 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558f3d0a10b0 0x558f3d0a34d0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef870c005831817a server_cookie=150f9cde93c98b12 in_seq=0 out_seq=0 2024-03-21T10:47:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.627+0000 7f8ce0c00640 10 client.27748 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:32.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.631+0000 7f8ce0c00640 1 -- 172.21.15.31:0/239844669 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558f3cf970c0 con 0x558f3d0a10b0 2024-03-21T10:47:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.631+0000 7f8ce0c00640 10 client.27748 handle_command_reply: tid=0 2024-03-21T10:47:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.631+0000 7f8d00a6a740 1 client.27748 shutdown 2024-03-21T10:47:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.631+0000 7f8d00a6a740 1 -- 172.21.15.31:0/239844669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f3d02afd0 msgr2=0x558f3d02e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.631+0000 7f8d00a6a740 1 --2- 172.21.15.31:0/239844669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f3d02afd0 0x558f3d02e5d0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8ce802c2e0 tx=0x7f8ce802bcd0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.631+0000 7f8d00a6a740 1 -- 172.21.15.31:0/239844669 shutdown_connections 2024-03-21T10:47:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.631+0000 7f8d00a6a740 1 --2- 172.21.15.31:0/239844669 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558f3d0a10b0 0x558f3d0a34d0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.631+0000 7f8d00a6a740 1 --2- 172.21.15.31:0/239844669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f3d0346c0 0x558f3d09f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.631+0000 7f8d00a6a740 1 --2- 172.21.15.31:0/239844669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f3d02b8f0 0x558f3d02eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.631+0000 7f8d00a6a740 1 --2- 172.21.15.31:0/239844669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f3d02afd0 0x558f3d02e5d0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.631+0000 7f8ce0c00640 0 client.27748 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.631+0000 7f8d00a6a740 1 -- 172.21.15.31:0/239844669 >> 172.21.15.31:0/239844669 conn(0x558f3d0258f0 msgr2=0x558f3d0273b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.632+0000 7f8d00a6a740 1 -- 172.21.15.31:0/239844669 shutdown_connections 2024-03-21T10:47:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.632+0000 7f8d00a6a740 1 -- 172.21.15.31:0/239844669 wait complete. 2024-03-21T10:47:32.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.632+0000 7f8d00a6a740 20 client.27748 trim_cache size 0 max 16384 2024-03-21T10:47:32.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:47:32.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:47:32.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:47:32.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:47:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.633+0000 7f8cfde00640 1 -- 172.21.15.31:0/1089756168 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f8cdc0a2fa0 msgr2=0x7f8cdc0a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.633+0000 7f8cfde00640 1 --2- 172.21.15.31:0/1089756168 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f8cdc0a2fa0 0x7f8cdc0a5460 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8003760 tx=0x7f8ce8005fd0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.633+0000 7f8cfde00640 1 -- 172.21.15.31:0/1089756168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cf8103230 msgr2=0x7f8cf8198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.633+0000 7f8cfde00640 1 --2- 172.21.15.31:0/1089756168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cf8103230 0x7f8cf8198440 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8ce400ea60 tx=0x7f8ce400ef20 comp rx=0 tx=0).stop 2024-03-21T10:47:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.633+0000 7f8cfde00640 1 -- 172.21.15.31:0/1089756168 shutdown_connections 2024-03-21T10:47:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.633+0000 7f8cfde00640 1 --2- 172.21.15.31:0/1089756168 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f8cdc0a2fa0 0x7f8cdc0a5460 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.633+0000 7f8cfde00640 1 --2- 172.21.15.31:0/1089756168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cf810bc50 0x7f8cf819c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.633+0000 7f8cfde00640 1 --2- 172.21.15.31:0/1089756168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cf8103230 0x7f8cf8198440 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.633+0000 7f8cfde00640 1 --2- 172.21.15.31:0/1089756168 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cf8102910 0x7f8cf8197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.633+0000 7f8cfde00640 1 -- 172.21.15.31:0/1089756168 >> 172.21.15.31:0/1089756168 conn(0x7f8cf80777b0 msgr2=0x7f8cf80fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.633+0000 7f8cfde00640 1 -- 172.21.15.31:0/1089756168 shutdown_connections 2024-03-21T10:47:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:32.633+0000 7f8cfde00640 1 -- 172.21.15.31:0/1089756168 wait complete. 2024-03-21T10:47:32.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.850+0000 7fe585e00640 1 -- 172.21.15.31:0/1967592274 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe58010d690 con 0x7fe580075bd0 2024-03-21T10:47:32.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.851+0000 7fe584a00640 1 -- 172.21.15.31:0/1967592274 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11387 (secure 0 0 0) 0x7fe5700bdc80 con 0x7fe580075bd0 2024-03-21T10:47:32.848 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:47:32.849 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:46:51.185295+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24547,24596]},"id":1}]} 2024-03-21T10:47:32.849 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-03-21T10:47:32.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.854+0000 7fe585e00640 1 -- 172.21.15.31:0/1967592274 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe5640a3110 msgr2=0x7fe5640a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:32.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.854+0000 7fe585e00640 1 --2- 172.21.15.31:0/1967592274 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe5640a3110 0x7fe5640a55d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe56c0096c0 tx=0x7fe56c009260 comp rx=0 tx=0).stop 2024-03-21T10:47:32.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.854+0000 7fe585e00640 1 -- 172.21.15.31:0/1967592274 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe580075bd0 msgr2=0x7fe58019e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:32.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.854+0000 7fe585e00640 1 --2- 172.21.15.31:0/1967592274 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe580075bd0 0x7fe58019e580 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fe5700042f0 tx=0x7fe570004320 comp rx=0 tx=0).stop 2024-03-21T10:47:32.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.854+0000 7fe585e00640 1 -- 172.21.15.31:0/1967592274 shutdown_connections 2024-03-21T10:47:32.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.854+0000 7fe585e00640 1 --2- 172.21.15.31:0/1967592274 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe5640a3110 0x7fe5640a55d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.854+0000 7fe585e00640 1 --2- 172.21.15.31:0/1967592274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe580112500 0x7fe5801a2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.854+0000 7fe585e00640 1 --2- 172.21.15.31:0/1967592274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5800764f0 0x7fe58019eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.854+0000 7fe585e00640 1 --2- 172.21.15.31:0/1967592274 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe580075bd0 0x7fe58019e580 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:32.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.854+0000 7fe585e00640 1 -- 172.21.15.31:0/1967592274 >> 172.21.15.31:0/1967592274 conn(0x7fe5800ff350 msgr2=0x7fe580113c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:32.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.854+0000 7fe585e00640 1 -- 172.21.15.31:0/1967592274 shutdown_connections 2024-03-21T10:47:32.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:32.854+0000 7fe585e00640 1 -- 172.21.15.31:0/1967592274 wait complete. 2024-03-21T10:47:32.871 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:47:32.871 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-21T10:47:33.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.129+0000 7f9d3be00640 1 -- 172.21.15.31:0/548492089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d3c073a70 msgr2=0x7f9d3c073e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.129+0000 7f9d3be00640 1 --2- 172.21.15.31:0/548492089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d3c073a70 0x7f9d3c073e50 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c009900 tx=0x7f9d2c02bcc0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.129+0000 7f9d3be00640 1 -- 172.21.15.31:0/548492089 shutdown_connections 2024-03-21T10:47:33.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.129+0000 7f9d3be00640 1 --2- 172.21.15.31:0/548492089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d3c10dce0 0x7f9d3c1100f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.129+0000 7f9d3be00640 1 --2- 172.21.15.31:0/548492089 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d3c074390 0x7f9d3c10d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.129+0000 7f9d3be00640 1 --2- 172.21.15.31:0/548492089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d3c073a70 0x7f9d3c073e50 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.129+0000 7f9d3be00640 1 -- 172.21.15.31:0/548492089 >> 172.21.15.31:0/548492089 conn(0x7f9d3c0fcf50 msgr2=0x7f9d3c0ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:33.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.129+0000 7f9d3be00640 1 -- 172.21.15.31:0/548492089 shutdown_connections 2024-03-21T10:47:33.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.129+0000 7f9d3be00640 1 -- 172.21.15.31:0/548492089 wait complete. 2024-03-21T10:47:33.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.130+0000 7f9d3be00640 1 Processor -- start 2024-03-21T10:47:33.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.131+0000 7f9d3be00640 1 -- start start 2024-03-21T10:47:33.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.131+0000 7f9d3be00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d3c073a70 0x7f9d3c19a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.131+0000 7f9d3be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d3c074390 0x7f9d3c19a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.131+0000 7f9d3be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d3c10dce0 0x7f9d3c19e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.131+0000 7f9d3be00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c113a90 con 0x7f9d3c074390 2024-03-21T10:47:33.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.131+0000 7f9d3be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c113910 con 0x7f9d3c10dce0 2024-03-21T10:47:33.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.131+0000 7f9d3be00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c113c10 con 0x7f9d3c073a70 2024-03-21T10:47:33.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.131+0000 7f9d3aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d3c073a70 0x7f9d3c19a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.131+0000 7f9d3aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d3c073a70 0x7f9d3c19a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38500/0 (socket says 172.21.15.31:38500) 2024-03-21T10:47:33.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.131+0000 7f9d3aa00640 1 -- 172.21.15.31:0/1878332050 learned_addr learned my addr 172.21.15.31:0/1878332050 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:33.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.131+0000 7f9d33e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d3c074390 0x7f9d3c19a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.131+0000 7f9d3b400640 1 --2- 172.21.15.31:0/1878332050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d3c10dce0 0x7f9d3c19e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.132+0000 7f9d3aa00640 1 -- 172.21.15.31:0/1878332050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d3c10dce0 msgr2=0x7f9d3c19e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.132+0000 7f9d3aa00640 1 --2- 172.21.15.31:0/1878332050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d3c10dce0 0x7f9d3c19e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.132+0000 7f9d3aa00640 1 -- 172.21.15.31:0/1878332050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d3c074390 msgr2=0x7f9d3c19a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.132+0000 7f9d3aa00640 1 --2- 172.21.15.31:0/1878332050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d3c074390 0x7f9d3c19a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.132+0000 7f9d3aa00640 1 -- 172.21.15.31:0/1878332050 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3c19efd0 con 0x7f9d3c073a70 2024-03-21T10:47:33.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.132+0000 7f9d3aa00640 1 --2- 172.21.15.31:0/1878332050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d3c073a70 0x7f9d3c19a030 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c000c00 tx=0x7f9d2c004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:33.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.132+0000 7f9d33400640 1 -- 172.21.15.31:0/1878332050 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c042020 con 0x7f9d3c073a70 2024-03-21T10:47:33.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.132+0000 7f9d3be00640 1 -- 172.21.15.31:0/1878332050 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d3c19f260 con 0x7f9d3c073a70 2024-03-21T10:47:33.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.132+0000 7f9d33400640 1 -- 172.21.15.31:0/1878332050 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d2c005000 con 0x7f9d3c073a70 2024-03-21T10:47:33.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.132+0000 7f9d33400640 1 -- 172.21.15.31:0/1878332050 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c003b20 con 0x7f9d3c073a70 2024-03-21T10:47:33.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.132+0000 7f9d3be00640 1 -- 172.21.15.31:0/1878332050 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d3c1a6b50 con 0x7f9d3c073a70 2024-03-21T10:47:33.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.135+0000 7f9d33400640 1 -- 172.21.15.31:0/1878332050 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f9d2c045020 con 0x7f9d3c073a70 2024-03-21T10:47:33.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.136+0000 7f9d44446740 1 Processor -- start 2024-03-21T10:47:33.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.136+0000 7f9d44446740 1 -- start start 2024-03-21T10:47:33.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.136+0000 7f9d44446740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626c32bd8b0 0x5626c32bdc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.136+0000 7f9d44446740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626c32be260 0x5626c32c6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.136+0000 7f9d44446740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626c32c6de0 0x5626c32c9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.136+0000 7f9d44446740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626c32212c0 con 0x5626c32c6de0 2024-03-21T10:47:33.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.136+0000 7f9d44446740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626c31c9c90 con 0x5626c32bd8b0 2024-03-21T10:47:33.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.136+0000 7f9d44446740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626c321d360 con 0x5626c32be260 2024-03-21T10:47:33.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.137+0000 7f9d3b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626c32bd8b0 0x5626c32bdc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.137+0000 7f9d3aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626c32be260 0x5626c32c6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.137+0000 7f9d3b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626c32bd8b0 0x5626c32bdc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33180/0 (socket says 172.21.15.31:33180) 2024-03-21T10:47:33.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.137+0000 7f9d3b400640 1 -- 172.21.15.31:0/1931191061 learned_addr learned my addr 172.21.15.31:0/1931191061 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:33.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.137+0000 7f9d33e00640 1 --2- 172.21.15.31:0/1931191061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626c32c6de0 0x5626c32c9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.137+0000 7f9d3b400640 1 -- 172.21.15.31:0/1931191061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626c32be260 msgr2=0x5626c32c6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:33.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.137+0000 7f9d3b400640 1 --2- 172.21.15.31:0/1931191061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626c32be260 0x5626c32c6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.137+0000 7f9d3b400640 1 -- 172.21.15.31:0/1931191061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626c32c6de0 msgr2=0x5626c32c9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:33.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.137+0000 7f9d3b400640 1 --2- 172.21.15.31:0/1931191061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626c32c6de0 0x5626c32c9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.137+0000 7f9d3b400640 1 -- 172.21.15.31:0/1931191061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626c31ee080 con 0x5626c32bd8b0 2024-03-21T10:47:33.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.137+0000 7f9d33400640 1 --2- 172.21.15.31:0/1878332050 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9d200a3110 0x7f9d200a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.137+0000 7f9d33400640 1 -- 172.21.15.31:0/1878332050 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f9d2c0f15d0 con 0x7f9d3c073a70 2024-03-21T10:47:33.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.137+0000 7f9d3b400640 1 --2- 172.21.15.31:0/1931191061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626c32bd8b0 0x5626c32bdc90 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f9d34009f30 tx=0x7f9d340118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9b07d61fd313695 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:33.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.138+0000 7f9d3aa00640 1 --2- 172.21.15.31:0/1878332050 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9d200a3110 0x7f9d200a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.138+0000 7f9d2be00640 1 -- 172.21.15.31:0/1931191061 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d34004d20 con 0x5626c32bd8b0 2024-03-21T10:47:33.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.138+0000 7f9d2be00640 1 -- 172.21.15.31:0/1931191061 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d34004050 con 0x5626c32bd8b0 2024-03-21T10:47:33.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.138+0000 7f9d2be00640 1 -- 172.21.15.31:0/1931191061 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d34005260 con 0x5626c32bd8b0 2024-03-21T10:47:33.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.138+0000 7f9d44446740 1 -- 172.21.15.31:0/1931191061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626c32bd8b0 msgr2=0x5626c32bdc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.138+0000 7f9d44446740 1 --2- 172.21.15.31:0/1931191061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626c32bd8b0 0x5626c32bdc90 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f9d34009f30 tx=0x7f9d340118d0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.138+0000 7f9d3aa00640 1 --2- 172.21.15.31:0/1878332050 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9d200a3110 0x7f9d200a55d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x5626c32bd380 tx=0x7f9d2c0f70e0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:33.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.139+0000 7f9d44446740 1 -- 172.21.15.31:0/1931191061 shutdown_connections 2024-03-21T10:47:33.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.139+0000 7f9d44446740 1 --2- 172.21.15.31:0/1931191061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626c32c6de0 0x5626c32c9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.139+0000 7f9d44446740 1 --2- 172.21.15.31:0/1931191061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626c32be260 0x5626c32c6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.139+0000 7f9d44446740 1 --2- 172.21.15.31:0/1931191061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626c32bd8b0 0x5626c32bdc90 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.139+0000 7f9d44446740 1 -- 172.21.15.31:0/1931191061 >> 172.21.15.31:0/1931191061 conn(0x5626c32b7f80 msgr2=0x5626c32b8360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:33.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.139+0000 7f9d44446740 1 -- 172.21.15.31:0/1931191061 shutdown_connections 2024-03-21T10:47:33.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.139+0000 7f9d44446740 1 -- 172.21.15.31:0/1931191061 wait complete. 2024-03-21T10:47:33.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.139+0000 7f9d44446740 1 -- start start 2024-03-21T10:47:33.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.139+0000 7f9d44446740 1 Processor -- start 2024-03-21T10:47:33.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.140+0000 7f9d44446740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626c32bd8b0 0x5626c32c6750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.140+0000 7f9d44446740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626c32be260 0x5626c32c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.140+0000 7f9d44446740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626c32c6de0 0x5626c32c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.140+0000 7f9d44446740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626c32c9d50 con 0x5626c32be260 2024-03-21T10:47:33.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.140+0000 7f9d44446740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626c31dd0d0 con 0x5626c32bd8b0 2024-03-21T10:47:33.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.140+0000 7f9d44446740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626c32c9ed0 con 0x5626c32c6de0 2024-03-21T10:47:33.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.140+0000 7f9d44446740 10 client.0 _ll_register_callbacks cb 0x5626c31e1e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:33.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.140+0000 7f9d3b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626c32bd8b0 0x5626c32c6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.140+0000 7f9d33e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626c32be260 0x5626c32c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.140+0000 7f9d3b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626c32bd8b0 0x5626c32c6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33188/0 (socket says 172.21.15.31:33188) 2024-03-21T10:47:33.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.141+0000 7f9d3b400640 1 -- 172.21.15.31:0/2767244342 learned_addr learned my addr 172.21.15.31:0/2767244342 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:33.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.141+0000 7f9d3b400640 1 --2- 172.21.15.31:0/2767244342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626c32c6de0 0x5626c32c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.141+0000 7f9d33e00640 1 -- 172.21.15.31:0/2767244342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626c32c6de0 msgr2=0x5626c32c40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:33.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.141+0000 7f9d33e00640 1 --2- 172.21.15.31:0/2767244342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626c32c6de0 0x5626c32c40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.141+0000 7f9d33e00640 1 -- 172.21.15.31:0/2767244342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626c32bd8b0 msgr2=0x5626c32c6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:33.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.141+0000 7f9d33e00640 1 --2- 172.21.15.31:0/2767244342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626c32bd8b0 0x5626c32c6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.141+0000 7f9d33e00640 1 -- 172.21.15.31:0/2767244342 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5626c2f353f0 con 0x5626c32be260 2024-03-21T10:47:33.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.141+0000 7f9d3b400640 1 --2- 172.21.15.31:0/2767244342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626c32bd8b0 0x5626c32c6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:47:33.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.141+0000 7f9d33e00640 1 --2- 172.21.15.31:0/2767244342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626c32be260 0x5626c32c3ae0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9d240126d0 tx=0x7f9d24012b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=537cfd7597d5c0b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:33.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.141+0000 7f9d28c00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:47:33.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.142+0000 7f9d28c00640 1 -- 172.21.15.31:0/2767244342 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2400f920 con 0x5626c32be260 2024-03-21T10:47:33.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.142+0000 7f9d28c00640 1 -- 172.21.15.31:0/2767244342 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f9d2400fac0 con 0x5626c32be260 2024-03-21T10:47:33.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.142+0000 7f9d44446740 1 -- 172.21.15.31:0/2767244342 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9d34004050 con 0x5626c32be260 2024-03-21T10:47:33.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.142+0000 7f9d28c00640 1 -- 172.21.15.31:0/2767244342 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d2400fcc0 con 0x5626c32be260 2024-03-21T10:47:33.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.142+0000 7f9d44446740 10 client.17223 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:33.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.142+0000 7f9d44446740 1 -- 172.21.15.31:0/2767244342 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5626c2fff7d0 con 0x5626c32be260 2024-03-21T10:47:33.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.142+0000 7f9d28c00640 1 -- 172.21.15.31:0/2767244342 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d34004050 con 0x5626c32be260 2024-03-21T10:47:33.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.143+0000 7f9d28c00640 1 -- 172.21.15.31:0/2767244342 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f9d2401c420 con 0x5626c32be260 2024-03-21T10:47:33.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.143+0000 7f9d44446740 10 client.17223 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:33.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.143+0000 7f9d44446740 10 client.17223 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:47:33.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.143+0000 7f9d44446740 20 client.17223 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:33.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.143+0000 7f9d44446740 1 --2- 172.21.15.31:0/2767244342 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5626c32d6d00 0x5626c32d90f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.143+0000 7f9d44446740 4 client.17223 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:47:33.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.143+0000 7f9d44446740 1 -- 172.21.15.31:0/2767244342 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5626c322e800 con 0x5626c32d6d00 2024-03-21T10:47:33.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.143+0000 7f9d3b400640 1 --2- 172.21.15.31:0/2767244342 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5626c32d6d00 0x5626c32d90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.143+0000 7f9d3b400640 1 --2- 172.21.15.31:0/2767244342 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5626c32d6d00 0x5626c32d90f0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=391ae935471e87d7 server_cookie=a05f25c50bd76ba3 in_seq=0 out_seq=0 2024-03-21T10:47:33.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.143+0000 7f9d28c00640 10 client.17223 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:47:33.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.144+0000 7f9d28c00640 1 -- 172.21.15.31:0/2767244342 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5626c322e800 con 0x5626c32d6d00 2024-03-21T10:47:33.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.144+0000 7f9d28c00640 10 client.17223 handle_command_reply: tid=0 2024-03-21T10:47:33.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.144+0000 7f9d44446740 1 client.17223 shutdown 2024-03-21T10:47:33.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.145+0000 7f9d44446740 1 -- 172.21.15.31:0/2767244342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626c32be260 msgr2=0x5626c32c3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.145+0000 7f9d44446740 1 --2- 172.21.15.31:0/2767244342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626c32be260 0x5626c32c3ae0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9d240126d0 tx=0x7f9d24012b90 comp rx=0 tx=0).stop 2024-03-21T10:47:33.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.145+0000 7f9d44446740 1 -- 172.21.15.31:0/2767244342 shutdown_connections 2024-03-21T10:47:33.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.145+0000 7f9d44446740 1 --2- 172.21.15.31:0/2767244342 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5626c32d6d00 0x5626c32d90f0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.145+0000 7f9d44446740 1 --2- 172.21.15.31:0/2767244342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626c32c6de0 0x5626c32c40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.145+0000 7f9d44446740 1 --2- 172.21.15.31:0/2767244342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626c32be260 0x5626c32c3ae0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.145+0000 7f9d44446740 1 --2- 172.21.15.31:0/2767244342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626c32bd8b0 0x5626c32c6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.145+0000 7f9d28c00640 0 client.17223 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:47:33.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.145+0000 7f9d44446740 1 -- 172.21.15.31:0/2767244342 >> 172.21.15.31:0/2767244342 conn(0x5626c32b7f80 msgr2=0x5626c32b9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:33.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.145+0000 7f9d44446740 1 -- 172.21.15.31:0/2767244342 shutdown_connections 2024-03-21T10:47:33.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.145+0000 7f9d44446740 1 -- 172.21.15.31:0/2767244342 wait complete. 2024-03-21T10:47:33.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.145+0000 7f9d44446740 20 client.17223 trim_cache size 0 max 16384 2024-03-21T10:47:33.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.159+0000 7f9d44446740 1 Processor -- start 2024-03-21T10:47:33.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.160+0000 7f9d44446740 1 -- start start 2024-03-21T10:47:33.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.160+0000 7f9d44446740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626c32bd8b0 0x5626c32b73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.160+0000 7f9d44446740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626c32be260 0x5626c32c6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.160+0000 7f9d44446740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626c32c6de0 0x5626c32c6750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.160+0000 7f9d44446740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c1b2f50 con 0x5626c32be260 2024-03-21T10:47:33.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.160+0000 7f9d44446740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c1b0a50 con 0x5626c32c6de0 2024-03-21T10:47:33.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.160+0000 7f9d44446740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c1aeb10 con 0x5626c32bd8b0 2024-03-21T10:47:33.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.160+0000 7f9d33e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626c32be260 0x5626c32c6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.160+0000 7f9d3b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626c32bd8b0 0x5626c32b73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.160+0000 7f9d33e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626c32be260 0x5626c32c6210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53988/0 (socket says 172.21.15.31:53988) 2024-03-21T10:47:33.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.160+0000 7f9d3b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626c32bd8b0 0x5626c32b73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38524/0 (socket says 172.21.15.31:38524) 2024-03-21T10:47:33.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.160+0000 7f9d33e00640 1 -- 172.21.15.31:0/633964855 learned_addr learned my addr 172.21.15.31:0/633964855 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:33.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.160+0000 7f9d3b400640 1 --2- 172.21.15.31:0/633964855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626c32c6de0 0x5626c32c6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.161+0000 7f9d33e00640 1 -- 172.21.15.31:0/633964855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626c32bd8b0 msgr2=0x5626c32b73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:33.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.161+0000 7f9d33e00640 1 --2- 172.21.15.31:0/633964855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626c32bd8b0 0x5626c32b73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.161+0000 7f9d33e00640 1 -- 172.21.15.31:0/633964855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626c32c6de0 msgr2=0x5626c32c6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:33.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.161+0000 7f9d33e00640 1 --2- 172.21.15.31:0/633964855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626c32c6de0 0x5626c32c6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.161+0000 7f9d33e00640 1 -- 172.21.15.31:0/633964855 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626c32c4460 con 0x5626c32be260 2024-03-21T10:47:33.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.161+0000 7f9d3b400640 1 --2- 172.21.15.31:0/633964855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626c32c6de0 0x5626c32c6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:47:33.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.161+0000 7f9d33e00640 1 --2- 172.21.15.31:0/633964855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626c32be260 0x5626c32c6210 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9d24000c00 tx=0x7f9d2400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe9298559a57e220 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:33.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.161+0000 7f9d2b400640 1 -- 172.21.15.31:0/633964855 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2400f9e0 con 0x5626c32be260 2024-03-21T10:47:33.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.161+0000 7f9d2b400640 1 -- 172.21.15.31:0/633964855 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d2401c420 con 0x5626c32be260 2024-03-21T10:47:33.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.161+0000 7f9d2b400640 1 -- 172.21.15.31:0/633964855 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d240035f0 con 0x5626c32be260 2024-03-21T10:47:33.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.162+0000 7f9d44446740 1 -- 172.21.15.31:0/633964855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626c32be260 msgr2=0x5626c32c6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.162+0000 7f9d44446740 1 --2- 172.21.15.31:0/633964855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626c32be260 0x5626c32c6210 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9d24000c00 tx=0x7f9d2400f790 comp rx=0 tx=0).stop 2024-03-21T10:47:33.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.162+0000 7f9d44446740 1 -- 172.21.15.31:0/633964855 shutdown_connections 2024-03-21T10:47:33.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.162+0000 7f9d44446740 1 --2- 172.21.15.31:0/633964855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626c32c6de0 0x5626c32c6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.162+0000 7f9d44446740 1 --2- 172.21.15.31:0/633964855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626c32be260 0x5626c32c6210 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.162+0000 7f9d44446740 1 --2- 172.21.15.31:0/633964855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626c32bd8b0 0x5626c32b73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.162+0000 7f9d44446740 1 -- 172.21.15.31:0/633964855 >> 172.21.15.31:0/633964855 conn(0x5626c32b7f80 msgr2=0x5626c32b9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:33.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.162+0000 7f9d44446740 1 -- 172.21.15.31:0/633964855 shutdown_connections 2024-03-21T10:47:33.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.162+0000 7f9d44446740 1 -- 172.21.15.31:0/633964855 wait complete. 2024-03-21T10:47:33.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.163+0000 7f9d44446740 1 -- start start 2024-03-21T10:47:33.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.163+0000 7f9d44446740 1 Processor -- start 2024-03-21T10:47:33.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.163+0000 7f9d44446740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626c32bd8b0 0x5626c32cec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.163+0000 7f9d44446740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626c32be260 0x5626c32cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.163+0000 7f9d44446740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626c32c6de0 0x5626c32cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.163+0000 7f9d44446740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c1af110 con 0x5626c32be260 2024-03-21T10:47:33.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.163+0000 7f9d44446740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c1b03f0 con 0x5626c32bd8b0 2024-03-21T10:47:33.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.163+0000 7f9d44446740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c1af730 con 0x5626c32c6de0 2024-03-21T10:47:33.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.163+0000 7f9d44446740 10 client.0 _ll_register_callbacks cb 0x7f9d24009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:33.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.163+0000 7f9d3b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626c32bd8b0 0x5626c32cec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.163+0000 7f9d33e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626c32be260 0x5626c32cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.164+0000 7f9d33e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626c32be260 0x5626c32cd0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54000/0 (socket says 172.21.15.31:54000) 2024-03-21T10:47:33.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.164+0000 7f9d3b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626c32bd8b0 0x5626c32cec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33214/0 (socket says 172.21.15.31:33214) 2024-03-21T10:47:33.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.164+0000 7f9d33e00640 1 -- 172.21.15.31:0/2600482396 learned_addr learned my addr 172.21.15.31:0/2600482396 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:33.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.164+0000 7f9d3b400640 1 --2- 172.21.15.31:0/2600482396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626c32c6de0 0x5626c32cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.164+0000 7f9d33e00640 1 -- 172.21.15.31:0/2600482396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626c32c6de0 msgr2=0x5626c32cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:33.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.164+0000 7f9d33e00640 1 --2- 172.21.15.31:0/2600482396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626c32c6de0 0x5626c32cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.164+0000 7f9d33e00640 1 -- 172.21.15.31:0/2600482396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626c32bd8b0 msgr2=0x5626c32cec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:33.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.164+0000 7f9d33e00640 1 --2- 172.21.15.31:0/2600482396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626c32bd8b0 0x5626c32cec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.164+0000 7f9d33e00640 1 -- 172.21.15.31:0/2600482396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5626c32cdc40 con 0x5626c32be260 2024-03-21T10:47:33.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.164+0000 7f9d33e00640 1 --2- 172.21.15.31:0/2600482396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626c32be260 0x5626c32cd0c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9d240126d0 tx=0x7f9d2401c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4427a5cc7d4edf1f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:33.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.164+0000 7f9d28c00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:47:33.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.165+0000 7f9d28c00640 1 -- 172.21.15.31:0/2600482396 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d240055e0 con 0x5626c32be260 2024-03-21T10:47:33.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.165+0000 7f9d44446740 1 -- 172.21.15.31:0/2600482396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9d2401c420 con 0x5626c32be260 2024-03-21T10:47:33.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.165+0000 7f9d28c00640 1 -- 172.21.15.31:0/2600482396 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f9d2402f070 con 0x5626c32be260 2024-03-21T10:47:33.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.165+0000 7f9d28c00640 1 -- 172.21.15.31:0/2600482396 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d24041050 con 0x5626c32be260 2024-03-21T10:47:33.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.165+0000 7f9d44446740 10 client.17229 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:33.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.165+0000 7f9d44446740 1 -- 172.21.15.31:0/2600482396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5626c33333d0 con 0x5626c32be260 2024-03-21T10:47:33.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.165+0000 7f9d28c00640 1 -- 172.21.15.31:0/2600482396 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d2401c420 con 0x5626c32be260 2024-03-21T10:47:33.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.166+0000 7f9d28c00640 1 -- 172.21.15.31:0/2600482396 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f9d24034440 con 0x5626c32be260 2024-03-21T10:47:33.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.166+0000 7f9d44446740 10 client.17229 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:33.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.166+0000 7f9d44446740 10 client.17229 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:47:33.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.166+0000 7f9d44446740 20 client.17229 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:33.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.166+0000 7f9d44446740 1 --2- 172.21.15.31:0/2600482396 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5626c33339e0 0x5626c3335e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.166+0000 7f9d44446740 4 client.17229 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:47:33.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.166+0000 7f9d44446740 1 -- 172.21.15.31:0/2600482396 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5626c3228b50 con 0x5626c33339e0 2024-03-21T10:47:33.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.166+0000 7f9d3b400640 1 --2- 172.21.15.31:0/2600482396 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5626c33339e0 0x5626c3335e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.166+0000 7f9d3b400640 1 --2- 172.21.15.31:0/2600482396 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5626c33339e0 0x5626c3335e00 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=61e51d6f8a4c3d64 server_cookie=39e6904eb87ee287 in_seq=0 out_seq=0 2024-03-21T10:47:33.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.166+0000 7f9d28c00640 10 client.17229 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:47:33.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.167+0000 7f9d28c00640 1 -- 172.21.15.31:0/2600482396 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5626c3228b50 con 0x5626c33339e0 2024-03-21T10:47:33.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.167+0000 7f9d28c00640 10 client.17229 handle_command_reply: tid=0 2024-03-21T10:47:33.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.167+0000 7f9d44446740 1 client.17229 shutdown 2024-03-21T10:47:33.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.167+0000 7f9d44446740 1 -- 172.21.15.31:0/2600482396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626c32be260 msgr2=0x5626c32cd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.167+0000 7f9d44446740 1 --2- 172.21.15.31:0/2600482396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626c32be260 0x5626c32cd0c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9d240126d0 tx=0x7f9d2401c900 comp rx=0 tx=0).stop 2024-03-21T10:47:33.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.167+0000 7f9d44446740 1 -- 172.21.15.31:0/2600482396 shutdown_connections 2024-03-21T10:47:33.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.167+0000 7f9d44446740 1 --2- 172.21.15.31:0/2600482396 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5626c33339e0 0x5626c3335e00 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.167+0000 7f9d44446740 1 --2- 172.21.15.31:0/2600482396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5626c32c6de0 0x5626c32cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.167+0000 7f9d44446740 1 --2- 172.21.15.31:0/2600482396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626c32be260 0x5626c32cd0c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.167+0000 7f9d28c00640 0 client.17229 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:47:33.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.167+0000 7f9d44446740 1 --2- 172.21.15.31:0/2600482396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5626c32bd8b0 0x5626c32cec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.167+0000 7f9d44446740 1 -- 172.21.15.31:0/2600482396 >> 172.21.15.31:0/2600482396 conn(0x5626c32b7f80 msgr2=0x5626c32be640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:33.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.167+0000 7f9d44446740 1 -- 172.21.15.31:0/2600482396 shutdown_connections 2024-03-21T10:47:33.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.167+0000 7f9d44446740 1 -- 172.21.15.31:0/2600482396 wait complete. 2024-03-21T10:47:33.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.167+0000 7f9d44446740 20 client.17229 trim_cache size 0 max 16384 2024-03-21T10:47:33.168 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:47:33.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.168+0000 7f9d3be00640 1 -- 172.21.15.31:0/1878332050 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9d200a3110 msgr2=0x7f9d200a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.168+0000 7f9d3be00640 1 --2- 172.21.15.31:0/1878332050 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9d200a3110 0x7f9d200a55d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x5626c32bd380 tx=0x7f9d2c0f70e0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.168+0000 7f9d3be00640 1 -- 172.21.15.31:0/1878332050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d3c073a70 msgr2=0x7f9d3c19a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.168+0000 7f9d3be00640 1 --2- 172.21.15.31:0/1878332050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d3c073a70 0x7f9d3c19a030 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c000c00 tx=0x7f9d2c004360 comp rx=0 tx=0).stop 2024-03-21T10:47:33.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.168+0000 7f9d3be00640 1 -- 172.21.15.31:0/1878332050 shutdown_connections 2024-03-21T10:47:33.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.168+0000 7f9d3be00640 1 --2- 172.21.15.31:0/1878332050 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9d200a3110 0x7f9d200a55d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.168+0000 7f9d3be00640 1 --2- 172.21.15.31:0/1878332050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d3c10dce0 0x7f9d3c19e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.168+0000 7f9d3be00640 1 --2- 172.21.15.31:0/1878332050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d3c074390 0x7f9d3c19a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.168+0000 7f9d3be00640 1 --2- 172.21.15.31:0/1878332050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d3c073a70 0x7f9d3c19a030 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.168+0000 7f9d3be00640 1 -- 172.21.15.31:0/1878332050 >> 172.21.15.31:0/1878332050 conn(0x7f9d3c0fcf50 msgr2=0x7f9d3c0fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:33.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.168+0000 7f9d3be00640 1 -- 172.21.15.31:0/1878332050 shutdown_connections 2024-03-21T10:47:33.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.168+0000 7f9d3be00640 1 -- 172.21.15.31:0/1878332050 wait complete. 2024-03-21T10:47:33.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:33 smithi031 ceph-mon[31673]: pgmap v278: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-21T10:47:33.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:33 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1967592274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:47:33.182 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:47:33.183 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-21T10:47:33.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.385+0000 7f4c9ca00640 1 -- 172.21.15.31:0/2990419926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c98101e20 msgr2=0x7f4c9810e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.385+0000 7f4c9ca00640 1 --2- 172.21.15.31:0/2990419926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c98101e20 0x7f4c9810e330 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009cd0 tx=0x7f4c9002c0f0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.386+0000 7f4c9ca00640 1 -- 172.21.15.31:0/2990419926 shutdown_connections 2024-03-21T10:47:33.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.386+0000 7f4c9ca00640 1 --2- 172.21.15.31:0/2990419926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c98101e20 0x7f4c9810e330 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.386+0000 7f4c9ca00640 1 --2- 172.21.15.31:0/2990419926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c98101460 0x7f4c981018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.386+0000 7f4c9ca00640 1 --2- 172.21.15.31:0/2990419926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c98107500 0x7f4c981078e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.386+0000 7f4c9ca00640 1 -- 172.21.15.31:0/2990419926 >> 172.21.15.31:0/2990419926 conn(0x7f4c980fcff0 msgr2=0x7f4c980ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:33.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.386+0000 7f4c9ca00640 1 -- 172.21.15.31:0/2990419926 shutdown_connections 2024-03-21T10:47:33.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.386+0000 7f4c9ca00640 1 -- 172.21.15.31:0/2990419926 wait complete. 2024-03-21T10:47:33.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.387+0000 7f4c9ca00640 1 Processor -- start 2024-03-21T10:47:33.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.387+0000 7f4c9ca00640 1 -- start start 2024-03-21T10:47:33.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.387+0000 7f4c9ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c98101460 0x7f4c980739a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.387+0000 7f4c9ca00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c98101e20 0x7f4c98073ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.388+0000 7f4c9ca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c98107500 0x7f4c98074420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.388+0000 7f4c9ca00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c98111810 con 0x7f4c98107500 2024-03-21T10:47:33.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.388+0000 7f4c9ca00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c98111690 con 0x7f4c98101460 2024-03-21T10:47:33.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.388+0000 7f4c9ca00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c98111990 con 0x7f4c98101e20 2024-03-21T10:47:33.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.388+0000 7f4c96a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c98101e20 0x7f4c98073ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.388+0000 7f4c97400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c98101460 0x7f4c980739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.388+0000 7f4c96a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c98101e20 0x7f4c98073ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38552/0 (socket says 172.21.15.31:38552) 2024-03-21T10:47:33.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.388+0000 7f4c96a00640 1 -- 172.21.15.31:0/750755113 learned_addr learned my addr 172.21.15.31:0/750755113 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:33.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.388+0000 7f4c97e00640 1 --2- 172.21.15.31:0/750755113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c98107500 0x7f4c98074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.388+0000 7f4c96a00640 1 -- 172.21.15.31:0/750755113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c98101460 msgr2=0x7f4c980739a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.388+0000 7f4c96a00640 1 --2- 172.21.15.31:0/750755113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c98101460 0x7f4c980739a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.388+0000 7f4c96a00640 1 -- 172.21.15.31:0/750755113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c98107500 msgr2=0x7f4c98074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.388+0000 7f4c96a00640 1 --2- 172.21.15.31:0/750755113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c98107500 0x7f4c98074420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.388+0000 7f4c96a00640 1 -- 172.21.15.31:0/750755113 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c981a6330 con 0x7f4c98101e20 2024-03-21T10:47:33.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.388+0000 7f4c97e00640 1 --2- 172.21.15.31:0/750755113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c98107500 0x7f4c98074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:47:33.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.388+0000 7f4c96a00640 1 --2- 172.21.15.31:0/750755113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c98101e20 0x7f4c98073ee0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f4c8400e9f0 tx=0x7f4c8400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:33.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.389+0000 7f4c8fe00640 1 -- 172.21.15.31:0/750755113 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c8400cd80 con 0x7f4c98101e20 2024-03-21T10:47:33.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.389+0000 7f4c8fe00640 1 -- 172.21.15.31:0/750755113 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c84010910 con 0x7f4c98101e20 2024-03-21T10:47:33.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.389+0000 7f4c8fe00640 1 -- 172.21.15.31:0/750755113 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c84018990 con 0x7f4c98101e20 2024-03-21T10:47:33.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.389+0000 7f4c9ca00640 1 -- 172.21.15.31:0/750755113 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c981a6620 con 0x7f4c98101e20 2024-03-21T10:47:33.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.389+0000 7f4c9ca00640 1 -- 172.21.15.31:0/750755113 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c981a6b20 con 0x7f4c98101e20 2024-03-21T10:47:33.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.392+0000 7f4c8fe00640 1 -- 172.21.15.31:0/750755113 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f4c84010ac0 con 0x7f4c98101e20 2024-03-21T10:47:33.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.394+0000 7f4ca0de5740 1 Processor -- start 2024-03-21T10:47:33.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.394+0000 7f4ca0de5740 1 -- start start 2024-03-21T10:47:33.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.394+0000 7f4c8fe00640 1 --2- 172.21.15.31:0/750755113 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4c7c0a3110 0x7f4c7c0a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.394+0000 7f4ca0de5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d2521b8b0 0x561d2521bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.394+0000 7f4ca0de5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2521c260 0x561d25224810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.394+0000 7f4ca0de5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d25224de0 0x561d25227230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.395+0000 7f4ca0de5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d2517f2c0 con 0x561d2521c260 2024-03-21T10:47:33.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.395+0000 7f4ca0de5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d25127c90 con 0x561d2521b8b0 2024-03-21T10:47:33.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.395+0000 7f4ca0de5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d2517b360 con 0x561d25224de0 2024-03-21T10:47:33.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.395+0000 7f4c97400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d2521b8b0 0x561d2521bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.395+0000 7f4c8fe00640 1 -- 172.21.15.31:0/750755113 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f4c840cff20 con 0x7f4c98101e20 2024-03-21T10:47:33.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.395+0000 7f4c97400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d2521b8b0 0x561d2521bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33250/0 (socket says 172.21.15.31:33250) 2024-03-21T10:47:33.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.395+0000 7f4c97400640 1 -- 172.21.15.31:0/291324343 learned_addr learned my addr 172.21.15.31:0/291324343 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:33.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.395+0000 7f4c96a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2521c260 0x561d25224810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.395+0000 7f4c97e00640 1 --2- 172.21.15.31:0/750755113 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4c7c0a3110 0x7f4c7c0a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.395+0000 7f4c97e00640 1 --2- 172.21.15.31:0/291324343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d25224de0 0x561d25227230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.395+0000 7f4c97400640 1 -- 172.21.15.31:0/291324343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d25224de0 msgr2=0x561d25227230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:33.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.395+0000 7f4c97400640 1 --2- 172.21.15.31:0/291324343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d25224de0 0x561d25227230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.395+0000 7f4c97400640 1 -- 172.21.15.31:0/291324343 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2521c260 msgr2=0x561d25224810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:33.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.395+0000 7f4c97400640 1 --2- 172.21.15.31:0/291324343 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2521c260 0x561d25224810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.395+0000 7f4c97400640 1 -- 172.21.15.31:0/291324343 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561d2514c080 con 0x561d2521b8b0 2024-03-21T10:47:33.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.396+0000 7f4c97400640 1 --2- 172.21.15.31:0/291324343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d2521b8b0 0x561d2521bc90 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f4c88009c70 tx=0x7f4c88011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab5ae8808d43b39a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:33.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.396+0000 7f4c83e00640 1 -- 172.21.15.31:0/291324343 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c88004230 con 0x561d2521b8b0 2024-03-21T10:47:33.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.396+0000 7f4c97e00640 1 --2- 172.21.15.31:0/750755113 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4c7c0a3110 0x7f4c7c0a55d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4c9002c630 tx=0x7f4c9002e040 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:33.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.396+0000 7f4c83e00640 1 -- 172.21.15.31:0/291324343 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c880043d0 con 0x561d2521b8b0 2024-03-21T10:47:33.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.396+0000 7f4c83e00640 1 -- 172.21.15.31:0/291324343 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c88019820 con 0x561d2521b8b0 2024-03-21T10:47:33.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.397+0000 7f4ca0de5740 1 -- 172.21.15.31:0/291324343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d2521b8b0 msgr2=0x561d2521bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.397+0000 7f4ca0de5740 1 --2- 172.21.15.31:0/291324343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d2521b8b0 0x561d2521bc90 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f4c88009c70 tx=0x7f4c88011810 comp rx=0 tx=0).stop 2024-03-21T10:47:33.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.397+0000 7f4ca0de5740 1 -- 172.21.15.31:0/291324343 shutdown_connections 2024-03-21T10:47:33.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.397+0000 7f4ca0de5740 1 --2- 172.21.15.31:0/291324343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d25224de0 0x561d25227230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.397+0000 7f4ca0de5740 1 --2- 172.21.15.31:0/291324343 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2521c260 0x561d25224810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.397+0000 7f4ca0de5740 1 --2- 172.21.15.31:0/291324343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d2521b8b0 0x561d2521bc90 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.397+0000 7f4ca0de5740 1 -- 172.21.15.31:0/291324343 >> 172.21.15.31:0/291324343 conn(0x561d25215f80 msgr2=0x561d25216360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:33.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.397+0000 7f4ca0de5740 1 -- 172.21.15.31:0/291324343 shutdown_connections 2024-03-21T10:47:33.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.397+0000 7f4ca0de5740 1 -- 172.21.15.31:0/291324343 wait complete. 2024-03-21T10:47:33.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.397+0000 7f4ca0de5740 1 -- start start 2024-03-21T10:47:33.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.398+0000 7f4ca0de5740 1 Processor -- start 2024-03-21T10:47:33.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.398+0000 7f4ca0de5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d2521b8b0 0x561d25224740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.398+0000 7f4ca0de5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2521c260 0x561d25221ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.398+0000 7f4ca0de5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d25224de0 0x561d252220e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.398+0000 7f4ca0de5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d25227d50 con 0x561d2521c260 2024-03-21T10:47:33.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.398+0000 7f4ca0de5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d2513b0d0 con 0x561d2521b8b0 2024-03-21T10:47:33.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.398+0000 7f4ca0de5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d25227ed0 con 0x561d25224de0 2024-03-21T10:47:33.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.398+0000 7f4ca0de5740 10 client.0 _ll_register_callbacks cb 0x561d2513fe80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:33.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.398+0000 7f4c97400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d2521b8b0 0x561d25224740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.398+0000 7f4c97400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d2521b8b0 0x561d25224740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33262/0 (socket says 172.21.15.31:33262) 2024-03-21T10:47:33.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.398+0000 7f4c97400640 1 -- 172.21.15.31:0/548302998 learned_addr learned my addr 172.21.15.31:0/548302998 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:33.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.398+0000 7f4c96a00640 1 --2- 172.21.15.31:0/548302998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2521c260 0x561d25221ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.398+0000 7f4c97e00640 1 --2- 172.21.15.31:0/548302998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d25224de0 0x561d252220e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.399+0000 7f4c97400640 1 -- 172.21.15.31:0/548302998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d25224de0 msgr2=0x561d252220e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:33.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.399+0000 7f4c97400640 1 --2- 172.21.15.31:0/548302998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d25224de0 0x561d252220e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.399+0000 7f4c97400640 1 -- 172.21.15.31:0/548302998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2521c260 msgr2=0x561d25221ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:33.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.399+0000 7f4c97400640 1 --2- 172.21.15.31:0/548302998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2521c260 0x561d25221ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.399+0000 7f4c97400640 1 -- 172.21.15.31:0/548302998 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561d24e933f0 con 0x561d2521b8b0 2024-03-21T10:47:33.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.399+0000 7f4c97400640 1 --2- 172.21.15.31:0/548302998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d2521b8b0 0x561d25224740 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f4c88009f70 tx=0x7f4c88009fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fa68c83e00d92de server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:33.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.399+0000 7f4c80c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:47:33.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.399+0000 7f4c80c00640 1 -- 172.21.15.31:0/548302998 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c88027070 con 0x561d2521b8b0 2024-03-21T10:47:33.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.399+0000 7f4c80c00640 1 -- 172.21.15.31:0/548302998 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f4c88003800 con 0x561d2521b8b0 2024-03-21T10:47:33.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.399+0000 7f4ca0de5740 1 -- 172.21.15.31:0/548302998 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4c880043d0 con 0x561d2521b8b0 2024-03-21T10:47:33.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.400+0000 7f4c80c00640 1 -- 172.21.15.31:0/548302998 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c88033050 con 0x561d2521b8b0 2024-03-21T10:47:33.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.400+0000 7f4ca0de5740 10 client.27854 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:33.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.400+0000 7f4ca0de5740 1 -- 172.21.15.31:0/548302998 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561d24f5d7d0 con 0x561d2521b8b0 2024-03-21T10:47:33.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.400+0000 7f4c80c00640 1 -- 172.21.15.31:0/548302998 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c880043d0 con 0x561d2521b8b0 2024-03-21T10:47:33.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.400+0000 7f4c80c00640 1 -- 172.21.15.31:0/548302998 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f4c88018070 con 0x561d2521b8b0 2024-03-21T10:47:33.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.400+0000 7f4ca0de5740 10 client.27854 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:33.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.400+0000 7f4ca0de5740 10 client.27854 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:47:33.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.400+0000 7f4ca0de5740 20 client.27854 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:33.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.400+0000 7f4ca0de5740 1 --2- 172.21.15.31:0/548302998 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x561d25234b30 0x561d25236f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.401+0000 7f4ca0de5740 4 client.27854 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:47:33.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.401+0000 7f4ca0de5740 1 -- 172.21.15.31:0/548302998 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561d2518c800 con 0x561d25234b30 2024-03-21T10:47:33.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.401+0000 7f4c96a00640 1 --2- 172.21.15.31:0/548302998 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x561d25234b30 0x561d25236f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.401+0000 7f4c96a00640 1 --2- 172.21.15.31:0/548302998 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x561d25234b30 0x561d25236f20 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=61e7f0137ce70ccf server_cookie=aab7a3ccd1e9baea in_seq=0 out_seq=0 2024-03-21T10:47:33.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.401+0000 7f4c80c00640 10 client.27854 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:47:33.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.402+0000 7f4c80c00640 1 -- 172.21.15.31:0/548302998 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561d2518c800 con 0x561d25234b30 2024-03-21T10:47:33.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.402+0000 7f4c80c00640 10 client.27854 handle_command_reply: tid=0 2024-03-21T10:47:33.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.402+0000 7f4ca0de5740 1 client.27854 shutdown 2024-03-21T10:47:33.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.402+0000 7f4ca0de5740 1 -- 172.21.15.31:0/548302998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d2521b8b0 msgr2=0x561d25224740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.402+0000 7f4ca0de5740 1 --2- 172.21.15.31:0/548302998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d2521b8b0 0x561d25224740 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f4c88009f70 tx=0x7f4c88009fa0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.402+0000 7f4ca0de5740 1 -- 172.21.15.31:0/548302998 shutdown_connections 2024-03-21T10:47:33.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.402+0000 7f4ca0de5740 1 --2- 172.21.15.31:0/548302998 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x561d25234b30 0x561d25236f20 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.402+0000 7f4ca0de5740 1 --2- 172.21.15.31:0/548302998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d25224de0 0x561d252220e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.402+0000 7f4ca0de5740 1 --2- 172.21.15.31:0/548302998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2521c260 0x561d25221ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.402+0000 7f4ca0de5740 1 --2- 172.21.15.31:0/548302998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d2521b8b0 0x561d25224740 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.402+0000 7f4c80c00640 0 client.27854 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:47:33.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.402+0000 7f4ca0de5740 1 -- 172.21.15.31:0/548302998 >> 172.21.15.31:0/548302998 conn(0x561d25215f80 msgr2=0x561d25217cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:33.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.403+0000 7f4ca0de5740 1 -- 172.21.15.31:0/548302998 shutdown_connections 2024-03-21T10:47:33.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.403+0000 7f4ca0de5740 1 -- 172.21.15.31:0/548302998 wait complete. 2024-03-21T10:47:33.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.403+0000 7f4ca0de5740 20 client.27854 trim_cache size 0 max 16384 2024-03-21T10:47:33.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.417+0000 7f4ca0de5740 1 Processor -- start 2024-03-21T10:47:33.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.417+0000 7f4ca0de5740 1 -- start start 2024-03-21T10:47:33.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.417+0000 7f4ca0de5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d2521b8b0 0x561d25220b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.417+0000 7f4ca0de5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d2521c260 0x561d25224200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.417+0000 7f4ca0de5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d25224de0 0x561d25224740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.417+0000 7f4ca0de5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c981b2fa0 con 0x561d25224de0 2024-03-21T10:47:33.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.417+0000 7f4ca0de5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c981b0aa0 con 0x561d2521c260 2024-03-21T10:47:33.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.417+0000 7f4ca0de5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c981aeb60 con 0x561d2521b8b0 2024-03-21T10:47:33.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.417+0000 7f4c97e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d25224de0 0x561d25224740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.417+0000 7f4c96a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d2521c260 0x561d25224200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.417+0000 7f4c97e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d25224de0 0x561d25224740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54034/0 (socket says 172.21.15.31:54034) 2024-03-21T10:47:33.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.417+0000 7f4c97e00640 1 -- 172.21.15.31:0/279629920 learned_addr learned my addr 172.21.15.31:0/279629920 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:33.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.417+0000 7f4c97400640 1 --2- 172.21.15.31:0/279629920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d2521b8b0 0x561d25220b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.418+0000 7f4c96a00640 1 -- 172.21.15.31:0/279629920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d2521b8b0 msgr2=0x561d25220b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:33.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.418+0000 7f4c96a00640 1 --2- 172.21.15.31:0/279629920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d2521b8b0 0x561d25220b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.418+0000 7f4c96a00640 1 -- 172.21.15.31:0/279629920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d25224de0 msgr2=0x561d25224740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:33.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.418+0000 7f4c96a00640 1 --2- 172.21.15.31:0/279629920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d25224de0 0x561d25224740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.418+0000 7f4c96a00640 1 -- 172.21.15.31:0/279629920 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561d25227c30 con 0x561d2521c260 2024-03-21T10:47:33.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.418+0000 7f4c97e00640 1 --2- 172.21.15.31:0/279629920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d25224de0 0x561d25224740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:33.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.418+0000 7f4c96a00640 1 --2- 172.21.15.31:0/279629920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d2521c260 0x561d25224200 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f4c840cd270 tx=0x7f4c840bc040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d9a47eb8166ac97 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:33.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.418+0000 7f4c83400640 1 -- 172.21.15.31:0/279629920 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c840bc0b0 con 0x561d2521c260 2024-03-21T10:47:33.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.418+0000 7f4c83400640 1 -- 172.21.15.31:0/279629920 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c84099fb0 con 0x561d2521c260 2024-03-21T10:47:33.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.418+0000 7f4c83400640 1 -- 172.21.15.31:0/279629920 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c840cab90 con 0x561d2521c260 2024-03-21T10:47:33.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.419+0000 7f4ca0de5740 1 -- 172.21.15.31:0/279629920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d2521c260 msgr2=0x561d25224200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.419+0000 7f4ca0de5740 1 --2- 172.21.15.31:0/279629920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d2521c260 0x561d25224200 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f4c840cd270 tx=0x7f4c840bc040 comp rx=0 tx=0).stop 2024-03-21T10:47:33.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.419+0000 7f4ca0de5740 1 -- 172.21.15.31:0/279629920 shutdown_connections 2024-03-21T10:47:33.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.419+0000 7f4ca0de5740 1 --2- 172.21.15.31:0/279629920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d25224de0 0x561d25224740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.419+0000 7f4ca0de5740 1 --2- 172.21.15.31:0/279629920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d2521c260 0x561d25224200 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.419+0000 7f4ca0de5740 1 --2- 172.21.15.31:0/279629920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d2521b8b0 0x561d25220b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.419+0000 7f4ca0de5740 1 -- 172.21.15.31:0/279629920 >> 172.21.15.31:0/279629920 conn(0x561d25215f80 msgr2=0x561d25216360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:33.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.419+0000 7f4ca0de5740 1 -- 172.21.15.31:0/279629920 shutdown_connections 2024-03-21T10:47:33.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.419+0000 7f4ca0de5740 1 -- 172.21.15.31:0/279629920 wait complete. 2024-03-21T10:47:33.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.419+0000 7f4ca0de5740 1 -- start start 2024-03-21T10:47:33.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.420+0000 7f4ca0de5740 1 Processor -- start 2024-03-21T10:47:33.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.420+0000 7f4ca0de5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d2521b8b0 0x561d2522c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.420+0000 7f4ca0de5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d2521c260 0x561d2522cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.420+0000 7f4ca0de5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d25224de0 0x561d2522ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.420+0000 7f4ca0de5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c981af160 con 0x561d25224de0 2024-03-21T10:47:33.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.420+0000 7f4ca0de5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c981b0440 con 0x561d2521c260 2024-03-21T10:47:33.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.420+0000 7f4ca0de5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c981af780 con 0x561d2521b8b0 2024-03-21T10:47:33.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.420+0000 7f4ca0de5740 10 client.0 _ll_register_callbacks cb 0x7f4c84097a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:33.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.420+0000 7f4c97e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d25224de0 0x561d2522ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.420+0000 7f4c97400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d2521b8b0 0x561d2522c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.420+0000 7f4c96a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d2521c260 0x561d2522cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.420+0000 7f4c97400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d2521b8b0 0x561d2522c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38576/0 (socket says 172.21.15.31:38576) 2024-03-21T10:47:33.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.420+0000 7f4c97400640 1 -- 172.21.15.31:0/480815947 learned_addr learned my addr 172.21.15.31:0/480815947 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:33.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.421+0000 7f4c97400640 1 -- 172.21.15.31:0/480815947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d2521c260 msgr2=0x561d2522cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:33.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.421+0000 7f4c97400640 1 --2- 172.21.15.31:0/480815947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d2521c260 0x561d2522cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.421+0000 7f4c97400640 1 -- 172.21.15.31:0/480815947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d25224de0 msgr2=0x561d2522ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:33.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.421+0000 7f4c97400640 1 --2- 172.21.15.31:0/480815947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d25224de0 0x561d2522ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.421+0000 7f4c97400640 1 -- 172.21.15.31:0/480815947 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561d2522b380 con 0x561d2521b8b0 2024-03-21T10:47:33.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.421+0000 7f4c96a00640 1 --2- 172.21.15.31:0/480815947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d2521c260 0x561d2522cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:33.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.421+0000 7f4c97400640 1 --2- 172.21.15.31:0/480815947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d2521b8b0 0x561d2522c760 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4c88001eb0 tx=0x7f4c88019e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55dd72b6a3b7eaed server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:33.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.421+0000 7f4c80c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:47:33.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.421+0000 7f4c80c00640 1 -- 172.21.15.31:0/480815947 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c88030070 con 0x561d2521b8b0 2024-03-21T10:47:33.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.421+0000 7f4c80c00640 1 -- 172.21.15.31:0/480815947 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f4c88005040 con 0x561d2521b8b0 2024-03-21T10:47:33.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.421+0000 7f4ca0de5740 1 -- 172.21.15.31:0/480815947 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4c84099fb0 con 0x561d2521b8b0 2024-03-21T10:47:33.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.422+0000 7f4c80c00640 1 -- 172.21.15.31:0/480815947 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c88027070 con 0x561d2521b8b0 2024-03-21T10:47:33.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.422+0000 7f4ca0de5740 10 client.27784 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:33.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.422+0000 7f4ca0de5740 1 -- 172.21.15.31:0/480815947 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561d25290c70 con 0x561d2521b8b0 2024-03-21T10:47:33.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.422+0000 7f4c80c00640 1 -- 172.21.15.31:0/480815947 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c84099fb0 con 0x561d2521b8b0 2024-03-21T10:47:33.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.422+0000 7f4c80c00640 1 -- 172.21.15.31:0/480815947 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f4c88018050 con 0x561d2521b8b0 2024-03-21T10:47:33.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.422+0000 7f4ca0de5740 10 client.27784 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:33.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.422+0000 7f4ca0de5740 10 client.27784 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:47:33.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.422+0000 7f4ca0de5740 20 client.27784 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:33.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.422+0000 7f4ca0de5740 1 --2- 172.21.15.31:0/480815947 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x561d252912d0 0x561d252936f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.422+0000 7f4ca0de5740 4 client.27784 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:47:33.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.422+0000 7f4ca0de5740 1 -- 172.21.15.31:0/480815947 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561d25186b50 con 0x561d252912d0 2024-03-21T10:47:33.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.422+0000 7f4c96a00640 1 --2- 172.21.15.31:0/480815947 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x561d252912d0 0x561d252936f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.423+0000 7f4c96a00640 1 --2- 172.21.15.31:0/480815947 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x561d252912d0 0x561d252936f0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7ac579d80a70869d server_cookie=a2c425a96bfb11ff in_seq=0 out_seq=0 2024-03-21T10:47:33.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.423+0000 7f4c80c00640 10 client.27784 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:47:33.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.423+0000 7f4c80c00640 1 -- 172.21.15.31:0/480815947 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561d25186b50 con 0x561d252912d0 2024-03-21T10:47:33.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.423+0000 7f4c80c00640 10 client.27784 handle_command_reply: tid=0 2024-03-21T10:47:33.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.423+0000 7f4ca0de5740 1 client.27784 shutdown 2024-03-21T10:47:33.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.423+0000 7f4ca0de5740 1 -- 172.21.15.31:0/480815947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d2521b8b0 msgr2=0x561d2522c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.423+0000 7f4ca0de5740 1 --2- 172.21.15.31:0/480815947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d2521b8b0 0x561d2522c760 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4c88001eb0 tx=0x7f4c88019e40 comp rx=0 tx=0).stop 2024-03-21T10:47:33.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.424+0000 7f4ca0de5740 1 -- 172.21.15.31:0/480815947 shutdown_connections 2024-03-21T10:47:33.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.424+0000 7f4ca0de5740 1 --2- 172.21.15.31:0/480815947 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x561d252912d0 0x561d252936f0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.424+0000 7f4ca0de5740 1 --2- 172.21.15.31:0/480815947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d25224de0 0x561d2522ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.424+0000 7f4c80c00640 0 client.27784 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:47:33.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.424+0000 7f4ca0de5740 1 --2- 172.21.15.31:0/480815947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d2521c260 0x561d2522cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.424+0000 7f4ca0de5740 1 --2- 172.21.15.31:0/480815947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d2521b8b0 0x561d2522c760 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.424+0000 7f4ca0de5740 1 -- 172.21.15.31:0/480815947 >> 172.21.15.31:0/480815947 conn(0x561d25215f80 msgr2=0x561d25220b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:33.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.424+0000 7f4ca0de5740 1 -- 172.21.15.31:0/480815947 shutdown_connections 2024-03-21T10:47:33.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.424+0000 7f4ca0de5740 1 -- 172.21.15.31:0/480815947 wait complete. 2024-03-21T10:47:33.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.424+0000 7f4ca0de5740 20 client.27784 trim_cache size 0 max 16384 2024-03-21T10:47:33.425 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:47:33.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.425+0000 7f4c9ca00640 1 -- 172.21.15.31:0/750755113 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4c7c0a3110 msgr2=0x7f4c7c0a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.425+0000 7f4c9ca00640 1 --2- 172.21.15.31:0/750755113 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4c7c0a3110 0x7f4c7c0a55d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4c9002c630 tx=0x7f4c9002e040 comp rx=0 tx=0).stop 2024-03-21T10:47:33.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.425+0000 7f4c9ca00640 1 -- 172.21.15.31:0/750755113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c98101e20 msgr2=0x7f4c98073ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.425+0000 7f4c9ca00640 1 --2- 172.21.15.31:0/750755113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c98101e20 0x7f4c98073ee0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f4c8400e9f0 tx=0x7f4c8400eeb0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.425+0000 7f4c9ca00640 1 -- 172.21.15.31:0/750755113 shutdown_connections 2024-03-21T10:47:33.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.425+0000 7f4c9ca00640 1 --2- 172.21.15.31:0/750755113 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4c7c0a3110 0x7f4c7c0a55d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.425+0000 7f4c9ca00640 1 --2- 172.21.15.31:0/750755113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c98107500 0x7f4c98074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.425+0000 7f4c9ca00640 1 --2- 172.21.15.31:0/750755113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c98101e20 0x7f4c98073ee0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.425+0000 7f4c9ca00640 1 --2- 172.21.15.31:0/750755113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c98101460 0x7f4c980739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.425+0000 7f4c9ca00640 1 -- 172.21.15.31:0/750755113 >> 172.21.15.31:0/750755113 conn(0x7f4c980fcff0 msgr2=0x7f4c980fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:33.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.425+0000 7f4c9ca00640 1 -- 172.21.15.31:0/750755113 shutdown_connections 2024-03-21T10:47:33.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.425+0000 7f4c9ca00640 1 -- 172.21.15.31:0/750755113 wait complete. 2024-03-21T10:47:33.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:33 smithi040 ceph-mon[36154]: pgmap v278: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-21T10:47:33.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:33 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1967592274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:47:33.440 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:47:33.440 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-21T10:47:33.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:33 smithi070 ceph-mon[36330]: pgmap v278: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-21T10:47:33.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:33 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1967592274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:47:33.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.659+0000 7fcbb4600640 1 -- 172.21.15.31:0/3098500872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbac103130 msgr2=0x7fcbac10b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.659+0000 7fcbb4600640 1 --2- 172.21.15.31:0/3098500872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbac103130 0x7fcbac10b610 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fcb98009900 tx=0x7fcb9802bcd0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.659+0000 7fcbb4600640 1 -- 172.21.15.31:0/3098500872 shutdown_connections 2024-03-21T10:47:33.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.659+0000 7fcbb4600640 1 --2- 172.21.15.31:0/3098500872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbac10bc30 0x7fcbac10e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.659+0000 7fcbb4600640 1 --2- 172.21.15.31:0/3098500872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbac103130 0x7fcbac10b610 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.659+0000 7fcbb4600640 1 --2- 172.21.15.31:0/3098500872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbac102810 0x7fcbac102bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.659+0000 7fcbb4600640 1 -- 172.21.15.31:0/3098500872 >> 172.21.15.31:0/3098500872 conn(0x7fcbac0fbff0 msgr2=0x7fcbac0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:33.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.660+0000 7fcbb4600640 1 -- 172.21.15.31:0/3098500872 shutdown_connections 2024-03-21T10:47:33.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.660+0000 7fcbb4600640 1 -- 172.21.15.31:0/3098500872 wait complete. 2024-03-21T10:47:33.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.662+0000 7fcbb4600640 1 Processor -- start 2024-03-21T10:47:33.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.662+0000 7fcbb4600640 1 -- start start 2024-03-21T10:47:33.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.663+0000 7fcbb4600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbac102810 0x7fcbac103680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.663+0000 7fcbb4600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbac103130 0x7fcbac103bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.663+0000 7fcbb4600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbac10bc30 0x7fcbac104100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.663+0000 7fcbb4600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac111a30 con 0x7fcbac103130 2024-03-21T10:47:33.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.663+0000 7fcbb4600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac1118b0 con 0x7fcbac102810 2024-03-21T10:47:33.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.663+0000 7fcbb4600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac111bb0 con 0x7fcbac10bc30 2024-03-21T10:47:33.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.663+0000 7fcbb1c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbac102810 0x7fcbac103680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.663+0000 7fcbb2600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbac10bc30 0x7fcbac104100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.663+0000 7fcbb1c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbac102810 0x7fcbac103680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33308/0 (socket says 172.21.15.31:33308) 2024-03-21T10:47:33.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.663+0000 7fcbb1c00640 1 -- 172.21.15.31:0/3049242427 learned_addr learned my addr 172.21.15.31:0/3049242427 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:33.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.663+0000 7fcbb1200640 1 --2- 172.21.15.31:0/3049242427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbac103130 0x7fcbac103bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.663+0000 7fcbb1c00640 1 -- 172.21.15.31:0/3049242427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbac10bc30 msgr2=0x7fcbac104100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.663+0000 7fcbb1c00640 1 --2- 172.21.15.31:0/3049242427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbac10bc30 0x7fcbac104100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.663+0000 7fcbb1c00640 1 -- 172.21.15.31:0/3049242427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbac103130 msgr2=0x7fcbac103bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.663+0000 7fcbb1c00640 1 --2- 172.21.15.31:0/3049242427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbac103130 0x7fcbac103bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.663+0000 7fcbb1c00640 1 -- 172.21.15.31:0/3049242427 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbac1aa7e0 con 0x7fcbac102810 2024-03-21T10:47:33.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.664+0000 7fcbb2600640 1 --2- 172.21.15.31:0/3049242427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbac10bc30 0x7fcbac104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:33.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.664+0000 7fcbb1200640 1 --2- 172.21.15.31:0/3049242427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbac103130 0x7fcbac103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:47:33.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.664+0000 7fcbb1c00640 1 --2- 172.21.15.31:0/3049242427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbac102810 0x7fcbac103680 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fcba000ea20 tx=0x7fcba000ee20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:33.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.664+0000 7fcbaa000640 1 -- 172.21.15.31:0/3049242427 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba000ccc0 con 0x7fcbac102810 2024-03-21T10:47:33.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.664+0000 7fcbaa000640 1 -- 172.21.15.31:0/3049242427 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcba000ce60 con 0x7fcbac102810 2024-03-21T10:47:33.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.664+0000 7fcbb4600640 1 -- 172.21.15.31:0/3049242427 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbac1aaad0 con 0x7fcbac102810 2024-03-21T10:47:33.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.664+0000 7fcbaa000640 1 -- 172.21.15.31:0/3049242427 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba00187e0 con 0x7fcbac102810 2024-03-21T10:47:33.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.664+0000 7fcbb4600640 1 -- 172.21.15.31:0/3049242427 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbac1ab040 con 0x7fcbac102810 2024-03-21T10:47:33.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.668+0000 7fcbb7396740 1 Processor -- start 2024-03-21T10:47:33.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.668+0000 7fcbb7396740 1 -- start start 2024-03-21T10:47:33.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.668+0000 7fcbb7396740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e123f56d0 0x560e123f5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.668+0000 7fcbb7396740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e123f6080 0x560e123fe630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.668+0000 7fcbb7396740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e123fec00 0x560e12401050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.668+0000 7fcbb7396740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e123592c0 con 0x560e123f56d0 2024-03-21T10:47:33.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.668+0000 7fcbb7396740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e12301c90 con 0x560e123fec00 2024-03-21T10:47:33.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.668+0000 7fcbb7396740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e12355360 con 0x560e123f6080 2024-03-21T10:47:33.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.668+0000 7fcbb2600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e123f56d0 0x560e123f5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.668+0000 7fcbb2600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e123f56d0 0x560e123f5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54076/0 (socket says 172.21.15.31:54076) 2024-03-21T10:47:33.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.668+0000 7fcbb2600640 1 -- 172.21.15.31:0/915104899 learned_addr learned my addr 172.21.15.31:0/915104899 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:33.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.668+0000 7fcbb1200640 1 --2- 172.21.15.31:0/915104899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e123fec00 0x560e12401050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.669+0000 7fcbb2600640 1 -- 172.21.15.31:0/915104899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e123f6080 msgr2=0x560e123fe630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:47:33.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.669+0000 7fcbb2600640 1 --2- 172.21.15.31:0/915104899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e123f6080 0x560e123fe630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.669+0000 7fcbb2600640 1 -- 172.21.15.31:0/915104899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e123fec00 msgr2=0x560e12401050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:33.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.669+0000 7fcbb2600640 1 --2- 172.21.15.31:0/915104899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e123fec00 0x560e12401050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.669+0000 7fcbb2600640 1 -- 172.21.15.31:0/915104899 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560e12326080 con 0x560e123f56d0 2024-03-21T10:47:33.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.669+0000 7fcbb1200640 1 --2- 172.21.15.31:0/915104899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e123fec00 0x560e12401050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:33.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.669+0000 7fcbaa000640 1 -- 172.21.15.31:0/3049242427 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fcba0010430 con 0x7fcbac102810 2024-03-21T10:47:33.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.669+0000 7fcbb2600640 1 --2- 172.21.15.31:0/915104899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e123f56d0 0x560e123f5ab0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fcba4011790 tx=0x7fcba4011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f9b08fad86cb0b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:33.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.670+0000 7fcb9ea00640 1 -- 172.21.15.31:0/915104899 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba4004d20 con 0x560e123f56d0 2024-03-21T10:47:33.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.670+0000 7fcb9ea00640 1 -- 172.21.15.31:0/915104899 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcba4003970 con 0x560e123f56d0 2024-03-21T10:47:33.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.670+0000 7fcb9ea00640 1 -- 172.21.15.31:0/915104899 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba40044f0 con 0x560e123f56d0 2024-03-21T10:47:33.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.670+0000 7fcbb7396740 1 -- 172.21.15.31:0/915104899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e123f56d0 msgr2=0x560e123f5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.670+0000 7fcbb7396740 1 --2- 172.21.15.31:0/915104899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e123f56d0 0x560e123f5ab0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fcba4011790 tx=0x7fcba4011c50 comp rx=0 tx=0).stop 2024-03-21T10:47:33.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.670+0000 7fcbb7396740 1 -- 172.21.15.31:0/915104899 shutdown_connections 2024-03-21T10:47:33.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.670+0000 7fcbb7396740 1 --2- 172.21.15.31:0/915104899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e123fec00 0x560e12401050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.670+0000 7fcbb7396740 1 --2- 172.21.15.31:0/915104899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e123f6080 0x560e123fe630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.670+0000 7fcbb7396740 1 --2- 172.21.15.31:0/915104899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e123f56d0 0x560e123f5ab0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.670+0000 7fcbb7396740 1 -- 172.21.15.31:0/915104899 >> 172.21.15.31:0/915104899 conn(0x560e123efda0 msgr2=0x560e123f0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:33.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.671+0000 7fcbb7396740 1 -- 172.21.15.31:0/915104899 shutdown_connections 2024-03-21T10:47:33.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.671+0000 7fcbb7396740 1 -- 172.21.15.31:0/915104899 wait complete. 2024-03-21T10:47:33.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.671+0000 7fcbaa000640 1 --2- 172.21.15.31:0/3049242427 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fcb940a3020 0x7fcb940a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.671+0000 7fcbaa000640 1 -- 172.21.15.31:0/3049242427 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fcba00cff00 con 0x7fcbac102810 2024-03-21T10:47:33.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.671+0000 7fcbb7396740 1 -- start start 2024-03-21T10:47:33.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.671+0000 7fcbb1200640 1 --2- 172.21.15.31:0/3049242427 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fcb940a3020 0x7fcb940a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.671+0000 7fcbb7396740 1 Processor -- start 2024-03-21T10:47:33.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.672+0000 7fcbb1200640 1 --2- 172.21.15.31:0/3049242427 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fcb940a3020 0x7fcb940a54e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcbac104ed0 tx=0x7fcb98005bd0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:33.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.672+0000 7fcbb7396740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e123f56d0 0x560e123fe520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.672+0000 7fcbb7396740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e123f6080 0x560e123fb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.672+0000 7fcbb7396740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e123fec00 0x560e123fbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.672+0000 7fcbb7396740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e12401b70 con 0x560e123fec00 2024-03-21T10:47:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.672+0000 7fcbb7396740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e123150d0 con 0x560e123f6080 2024-03-21T10:47:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.672+0000 7fcbb7396740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e12401cf0 con 0x560e123f56d0 2024-03-21T10:47:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.672+0000 7fcbb1200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e123f6080 0x560e123fb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.672+0000 7fcbb1c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e123f56d0 0x560e123fe520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.672+0000 7fcbb1200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e123f6080 0x560e123fb8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33328/0 (socket says 172.21.15.31:33328) 2024-03-21T10:47:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.672+0000 7fcbb1c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e123f56d0 0x560e123fe520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38612/0 (socket says 172.21.15.31:38612) 2024-03-21T10:47:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.672+0000 7fcbb1c00640 1 -- 172.21.15.31:0/825214461 learned_addr learned my addr 172.21.15.31:0/825214461 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.672+0000 7fcbb7396740 10 client.0 _ll_register_callbacks cb 0x560e12319e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.672+0000 7fcbb2600640 1 --2- 172.21.15.31:0/825214461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e123fec00 0x560e123fbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.673+0000 7fcbb1c00640 1 -- 172.21.15.31:0/825214461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e123f6080 msgr2=0x560e123fb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.673+0000 7fcbb1c00640 1 --2- 172.21.15.31:0/825214461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e123f6080 0x560e123fb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.673+0000 7fcbb1c00640 1 -- 172.21.15.31:0/825214461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e123fec00 msgr2=0x560e123fbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:33.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.673+0000 7fcbb1c00640 1 --2- 172.21.15.31:0/825214461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e123fec00 0x560e123fbe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.673+0000 7fcbb1c00640 1 -- 172.21.15.31:0/825214461 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560e1206d3f0 con 0x560e123f56d0 2024-03-21T10:47:33.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.673+0000 7fcbb2600640 1 --2- 172.21.15.31:0/825214461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e123fec00 0x560e123fbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:47:33.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.673+0000 7fcbb1c00640 1 --2- 172.21.15.31:0/825214461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e123f56d0 0x560e123fe520 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fcba00cd310 tx=0x7fcba009c710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c9afe9e751a0a7e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:33.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.673+0000 7fcb93400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:47:33.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.673+0000 7fcb93400640 1 -- 172.21.15.31:0/825214461 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba00a4960 con 0x560e123f56d0 2024-03-21T10:47:33.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.673+0000 7fcb93400640 1 -- 172.21.15.31:0/825214461 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fcba00985c0 con 0x560e123f56d0 2024-03-21T10:47:33.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.673+0000 7fcbb7396740 1 -- 172.21.15.31:0/825214461 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcba4003970 con 0x560e123f56d0 2024-03-21T10:47:33.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.674+0000 7fcb93400640 1 -- 172.21.15.31:0/825214461 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcba0010720 con 0x560e123f56d0 2024-03-21T10:47:33.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.674+0000 7fcbb7396740 10 client.27793 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:33.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.674+0000 7fcbb7396740 1 -- 172.21.15.31:0/825214461 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560e121377d0 con 0x560e123f56d0 2024-03-21T10:47:33.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.674+0000 7fcb93400640 1 -- 172.21.15.31:0/825214461 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcba4003970 con 0x560e123f56d0 2024-03-21T10:47:33.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.674+0000 7fcb93400640 1 -- 172.21.15.31:0/825214461 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fcba0018a10 con 0x560e123f56d0 2024-03-21T10:47:33.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.674+0000 7fcbb7396740 10 client.27793 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:33.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.674+0000 7fcbb7396740 10 client.27793 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:47:33.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.674+0000 7fcbb7396740 20 client.27793 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:33.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.674+0000 7fcbb7396740 1 --2- 172.21.15.31:0/825214461 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560e1240ea80 0x560e12410e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.674+0000 7fcbb7396740 4 client.27793 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:47:33.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.674+0000 7fcbb7396740 1 -- 172.21.15.31:0/825214461 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560e12366800 con 0x560e1240ea80 2024-03-21T10:47:33.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.674+0000 7fcbb1200640 1 --2- 172.21.15.31:0/825214461 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560e1240ea80 0x560e12410e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.675+0000 7fcbb1200640 1 --2- 172.21.15.31:0/825214461 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560e1240ea80 0x560e12410e90 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e570b408fec4709 server_cookie=eaf4bb6a26a92dc8 in_seq=0 out_seq=0 2024-03-21T10:47:33.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.675+0000 7fcb93400640 10 client.27793 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:33.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.676+0000 7fcb93400640 1 -- 172.21.15.31:0/825214461 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560e12366800 con 0x560e1240ea80 2024-03-21T10:47:33.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.676+0000 7fcb93400640 10 client.27793 handle_command_reply: tid=0 2024-03-21T10:47:33.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.676+0000 7fcbb7396740 1 client.27793 shutdown 2024-03-21T10:47:33.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.676+0000 7fcbb7396740 1 -- 172.21.15.31:0/825214461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e123f56d0 msgr2=0x560e123fe520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.676+0000 7fcbb7396740 1 --2- 172.21.15.31:0/825214461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e123f56d0 0x560e123fe520 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fcba00cd310 tx=0x7fcba009c710 comp rx=0 tx=0).stop 2024-03-21T10:47:33.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.676+0000 7fcbb7396740 1 -- 172.21.15.31:0/825214461 shutdown_connections 2024-03-21T10:47:33.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.676+0000 7fcbb7396740 1 --2- 172.21.15.31:0/825214461 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560e1240ea80 0x560e12410e90 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.676+0000 7fcbb7396740 1 --2- 172.21.15.31:0/825214461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e123fec00 0x560e123fbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.676+0000 7fcbb7396740 1 --2- 172.21.15.31:0/825214461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e123f6080 0x560e123fb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.676+0000 7fcbb7396740 1 --2- 172.21.15.31:0/825214461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e123f56d0 0x560e123fe520 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.676+0000 7fcbb7396740 1 -- 172.21.15.31:0/825214461 >> 172.21.15.31:0/825214461 conn(0x560e123efda0 msgr2=0x560e123f1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:33.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.676+0000 7fcb93400640 0 client.27793 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:33.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.676+0000 7fcbb7396740 1 -- 172.21.15.31:0/825214461 shutdown_connections 2024-03-21T10:47:33.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.676+0000 7fcbb7396740 1 -- 172.21.15.31:0/825214461 wait complete. 2024-03-21T10:47:33.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.676+0000 7fcbb7396740 20 client.27793 trim_cache size 0 max 16384 2024-03-21T10:47:33.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.690+0000 7fcbb7396740 1 Processor -- start 2024-03-21T10:47:33.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.690+0000 7fcbb7396740 1 -- start start 2024-03-21T10:47:33.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.690+0000 7fcbb7396740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e123f56d0 0x560e123fdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.690+0000 7fcbb7396740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e123f6080 0x560e123fe520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.690+0000 7fcbb7396740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e123fec00 0x560e123f8b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.690+0000 7fcbb7396740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac1b75b0 con 0x560e123f6080 2024-03-21T10:47:33.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.690+0000 7fcbb7396740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac1b50b0 con 0x560e123fec00 2024-03-21T10:47:33.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.690+0000 7fcbb7396740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac1b3170 con 0x560e123f56d0 2024-03-21T10:47:33.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.691+0000 7fcbb1c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e123f56d0 0x560e123fdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.691+0000 7fcbb1c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e123f56d0 0x560e123fdfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38624/0 (socket says 172.21.15.31:38624) 2024-03-21T10:47:33.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.691+0000 7fcbb1c00640 1 -- 172.21.15.31:0/2422044724 learned_addr learned my addr 172.21.15.31:0/2422044724 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:33.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.691+0000 7fcbb2600640 1 --2- 172.21.15.31:0/2422044724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e123fec00 0x560e123f8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.691+0000 7fcbb1200640 1 --2- 172.21.15.31:0/2422044724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e123f6080 0x560e123fe520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.691+0000 7fcbb1c00640 1 -- 172.21.15.31:0/2422044724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e123fec00 msgr2=0x560e123f8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:33.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.691+0000 7fcbb1c00640 1 --2- 172.21.15.31:0/2422044724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e123fec00 0x560e123f8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.691+0000 7fcbb1c00640 1 -- 172.21.15.31:0/2422044724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e123f6080 msgr2=0x560e123fe520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:33.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.691+0000 7fcbb1c00640 1 --2- 172.21.15.31:0/2422044724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e123f6080 0x560e123fe520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.691+0000 7fcbb1c00640 1 -- 172.21.15.31:0/2422044724 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560e123f0b30 con 0x560e123f56d0 2024-03-21T10:47:33.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.691+0000 7fcbb1c00640 1 --2- 172.21.15.31:0/2422044724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e123f56d0 0x560e123fdfe0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fcba0098920 tx=0x7fcba0000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=994ffe14c02699e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:33.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.692+0000 7fcb9e000640 1 -- 172.21.15.31:0/2422044724 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba00981f0 con 0x560e123f56d0 2024-03-21T10:47:33.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.692+0000 7fcb9e000640 1 -- 172.21.15.31:0/2422044724 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcba0096ac0 con 0x560e123f56d0 2024-03-21T10:47:33.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.692+0000 7fcb9e000640 1 -- 172.21.15.31:0/2422044724 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba009b600 con 0x560e123f56d0 2024-03-21T10:47:33.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.692+0000 7fcbb7396740 1 -- 172.21.15.31:0/2422044724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e123f56d0 msgr2=0x560e123fdfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.692+0000 7fcbb7396740 1 --2- 172.21.15.31:0/2422044724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e123f56d0 0x560e123fdfe0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fcba0098920 tx=0x7fcba0000bc0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.692+0000 7fcbb7396740 1 -- 172.21.15.31:0/2422044724 shutdown_connections 2024-03-21T10:47:33.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.692+0000 7fcbb7396740 1 --2- 172.21.15.31:0/2422044724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e123fec00 0x560e123f8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.692+0000 7fcbb7396740 1 --2- 172.21.15.31:0/2422044724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e123f6080 0x560e123fe520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.692+0000 7fcbb7396740 1 --2- 172.21.15.31:0/2422044724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e123f56d0 0x560e123fdfe0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.692+0000 7fcbb7396740 1 -- 172.21.15.31:0/2422044724 >> 172.21.15.31:0/2422044724 conn(0x560e123efda0 msgr2=0x560e123eeeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:33.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.692+0000 7fcbb7396740 1 -- 172.21.15.31:0/2422044724 shutdown_connections 2024-03-21T10:47:33.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.692+0000 7fcbb7396740 1 -- 172.21.15.31:0/2422044724 wait complete. 2024-03-21T10:47:33.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.693+0000 7fcbb7396740 1 -- start start 2024-03-21T10:47:33.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.693+0000 7fcbb7396740 1 Processor -- start 2024-03-21T10:47:33.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.693+0000 7fcbb7396740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e123f56d0 0x560e12404300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.693+0000 7fcbb7396740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e123f6080 0x560e12404840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.693+0000 7fcbb7396740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e123fec00 0x560e12404d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.693+0000 7fcbb7396740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac1b3770 con 0x560e123f6080 2024-03-21T10:47:33.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.693+0000 7fcbb7396740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac1b4a50 con 0x560e123fec00 2024-03-21T10:47:33.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.693+0000 7fcbb7396740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac1b3d90 con 0x560e123f56d0 2024-03-21T10:47:33.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.693+0000 7fcbb7396740 10 client.0 _ll_register_callbacks cb 0x7fcba0010d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:33.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.694+0000 7fcbb1c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e123f56d0 0x560e12404300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.694+0000 7fcbb2600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e123fec00 0x560e12404d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.694+0000 7fcbb1c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e123f56d0 0x560e12404300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38630/0 (socket says 172.21.15.31:38630) 2024-03-21T10:47:33.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.694+0000 7fcbb1c00640 1 -- 172.21.15.31:0/3730483574 learned_addr learned my addr 172.21.15.31:0/3730483574 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:33.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.694+0000 7fcbb1200640 1 --2- 172.21.15.31:0/3730483574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e123f6080 0x560e12404840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.694+0000 7fcbb1c00640 1 -- 172.21.15.31:0/3730483574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e123fec00 msgr2=0x560e12404d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:33.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.694+0000 7fcbb1c00640 1 --2- 172.21.15.31:0/3730483574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e123fec00 0x560e12404d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.694+0000 7fcbb1c00640 1 -- 172.21.15.31:0/3730483574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e123f6080 msgr2=0x560e12404840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:33.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.694+0000 7fcbb1c00640 1 --2- 172.21.15.31:0/3730483574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e123f6080 0x560e12404840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.694+0000 7fcbb1c00640 1 -- 172.21.15.31:0/3730483574 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560e12469330 con 0x560e123f56d0 2024-03-21T10:47:33.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.694+0000 7fcbb1c00640 1 --2- 172.21.15.31:0/3730483574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e123f56d0 0x560e12404300 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fcba00983b0 tx=0x7fcba009c710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c4b8c9bd6cad7c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:33.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.694+0000 7fcb93400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:47:33.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.694+0000 7fcb93400640 1 -- 172.21.15.31:0/3730483574 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba00981f0 con 0x560e123f56d0 2024-03-21T10:47:33.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.695+0000 7fcb93400640 1 -- 172.21.15.31:0/3730483574 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fcba0018b80 con 0x560e123f56d0 2024-03-21T10:47:33.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.695+0000 7fcbb7396740 1 -- 172.21.15.31:0/3730483574 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcba0096ac0 con 0x560e123f56d0 2024-03-21T10:47:33.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.695+0000 7fcb93400640 1 -- 172.21.15.31:0/3730483574 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcba009b3a0 con 0x560e123f56d0 2024-03-21T10:47:33.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.695+0000 7fcbb7396740 10 client.27799 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:33.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.695+0000 7fcbb7396740 1 -- 172.21.15.31:0/3730483574 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560e1246a590 con 0x560e123f56d0 2024-03-21T10:47:33.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.695+0000 7fcb93400640 1 -- 172.21.15.31:0/3730483574 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcba0096ac0 con 0x560e123f56d0 2024-03-21T10:47:33.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.695+0000 7fcb93400640 1 -- 172.21.15.31:0/3730483574 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fcba00a1230 con 0x560e123f56d0 2024-03-21T10:47:33.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.695+0000 7fcbb7396740 10 client.27799 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:33.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.695+0000 7fcbb7396740 10 client.27799 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:47:33.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.695+0000 7fcbb7396740 20 client.27799 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:33.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.695+0000 7fcbb7396740 1 --2- 172.21.15.31:0/3730483574 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560e1246a920 0x560e1246cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:33.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.695+0000 7fcbb7396740 4 client.27799 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:47:33.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.695+0000 7fcbb7396740 1 -- 172.21.15.31:0/3730483574 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560e12360b50 con 0x560e1246a920 2024-03-21T10:47:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.696+0000 7fcbb1200640 1 --2- 172.21.15.31:0/3730483574 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560e1246a920 0x560e1246cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.696+0000 7fcbb1200640 1 --2- 172.21.15.31:0/3730483574 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560e1246a920 0x560e1246cd40 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d52c35aabe416f95 server_cookie=d59eff3d1c956a3e in_seq=0 out_seq=0 2024-03-21T10:47:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.696+0000 7fcb93400640 10 client.27799 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.696+0000 7fcb93400640 1 -- 172.21.15.31:0/3730483574 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560e12360b50 con 0x560e1246a920 2024-03-21T10:47:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.696+0000 7fcb93400640 10 client.27799 handle_command_reply: tid=0 2024-03-21T10:47:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.696+0000 7fcbb7396740 1 client.27799 shutdown 2024-03-21T10:47:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.696+0000 7fcbb7396740 1 -- 172.21.15.31:0/3730483574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e123f56d0 msgr2=0x560e12404300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.697+0000 7fcbb7396740 1 --2- 172.21.15.31:0/3730483574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e123f56d0 0x560e12404300 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fcba00983b0 tx=0x7fcba009c710 comp rx=0 tx=0).stop 2024-03-21T10:47:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.697+0000 7fcbb7396740 1 -- 172.21.15.31:0/3730483574 shutdown_connections 2024-03-21T10:47:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.697+0000 7fcbb7396740 1 --2- 172.21.15.31:0/3730483574 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x560e1246a920 0x560e1246cd40 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.697+0000 7fcbb7396740 1 --2- 172.21.15.31:0/3730483574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560e123fec00 0x560e12404d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.697+0000 7fcb93400640 0 client.27799 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:33.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.697+0000 7fcbb7396740 1 --2- 172.21.15.31:0/3730483574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e123f6080 0x560e12404840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.697+0000 7fcbb7396740 1 --2- 172.21.15.31:0/3730483574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e123f56d0 0x560e12404300 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.697+0000 7fcbb7396740 1 -- 172.21.15.31:0/3730483574 >> 172.21.15.31:0/3730483574 conn(0x560e123efda0 msgr2=0x560e123f6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:33.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.697+0000 7fcbb7396740 1 -- 172.21.15.31:0/3730483574 shutdown_connections 2024-03-21T10:47:33.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.697+0000 7fcbb7396740 1 -- 172.21.15.31:0/3730483574 wait complete. 2024-03-21T10:47:33.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.697+0000 7fcbb7396740 20 client.27799 trim_cache size 0 max 16384 2024-03-21T10:47:33.700 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:47:33.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.697+0000 7fcbb4600640 1 -- 172.21.15.31:0/3049242427 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fcb940a3020 msgr2=0x7fcb940a54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.697+0000 7fcbb4600640 1 --2- 172.21.15.31:0/3049242427 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fcb940a3020 0x7fcb940a54e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcbac104ed0 tx=0x7fcb98005bd0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.698+0000 7fcbb4600640 1 -- 172.21.15.31:0/3049242427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbac102810 msgr2=0x7fcbac103680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:33.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.698+0000 7fcbb4600640 1 --2- 172.21.15.31:0/3049242427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbac102810 0x7fcbac103680 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fcba000ea20 tx=0x7fcba000ee20 comp rx=0 tx=0).stop 2024-03-21T10:47:33.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.698+0000 7fcbb4600640 1 -- 172.21.15.31:0/3049242427 shutdown_connections 2024-03-21T10:47:33.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.698+0000 7fcbb4600640 1 --2- 172.21.15.31:0/3049242427 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fcb940a3020 0x7fcb940a54e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.698+0000 7fcbb4600640 1 --2- 172.21.15.31:0/3049242427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbac10bc30 0x7fcbac104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.698+0000 7fcbb4600640 1 --2- 172.21.15.31:0/3049242427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbac103130 0x7fcbac103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.698+0000 7fcbb4600640 1 --2- 172.21.15.31:0/3049242427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbac102810 0x7fcbac103680 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:33.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.698+0000 7fcbb4600640 1 -- 172.21.15.31:0/3049242427 >> 172.21.15.31:0/3049242427 conn(0x7fcbac0fbff0 msgr2=0x7fcbac0fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:33.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.698+0000 7fcbb4600640 1 -- 172.21.15.31:0/3049242427 shutdown_connections 2024-03-21T10:47:33.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:33.698+0000 7fcbb4600640 1 -- 172.21.15.31:0/3049242427 wait complete. 2024-03-21T10:47:34.714 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-21T10:47:34.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:47:34.714 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-21T10:47:34.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.969+0000 7f2527000640 1 -- 172.21.15.31:0/2077314972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25200fff20 msgr2=0x7f25201003a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:34.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.969+0000 7f2527000640 1 --2- 172.21.15.31:0/2077314972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25200fff20 0x7f25201003a0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f2508009930 tx=0x7f250802bcb0 comp rx=0 tx=0).stop 2024-03-21T10:47:34.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.969+0000 7f2527000640 1 -- 172.21.15.31:0/2077314972 shutdown_connections 2024-03-21T10:47:34.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.969+0000 7f2527000640 1 --2- 172.21.15.31:0/2077314972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25201008e0 0x7f252010e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:34.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.969+0000 7f2527000640 1 --2- 172.21.15.31:0/2077314972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25200fff20 0x7f25201003a0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:34.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.969+0000 7f2527000640 1 --2- 172.21.15.31:0/2077314972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2520107860 0x7f2520107c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:34.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.969+0000 7f2527000640 1 -- 172.21.15.31:0/2077314972 >> 172.21.15.31:0/2077314972 conn(0x7f2520075860 msgr2=0x7f2520075c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:34.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.969+0000 7f2527000640 1 -- 172.21.15.31:0/2077314972 shutdown_connections 2024-03-21T10:47:34.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.969+0000 7f2527000640 1 -- 172.21.15.31:0/2077314972 wait complete. 2024-03-21T10:47:34.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.970+0000 7f2527000640 1 Processor -- start 2024-03-21T10:47:34.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.971+0000 7f2527000640 1 -- start start 2024-03-21T10:47:34.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.971+0000 7f2527000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25200fff20 0x7f2520197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:34.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.971+0000 7f2527000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25201008e0 0x7f2520198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:34.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.971+0000 7f2527000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2520107860 0x7f252019c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:34.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.971+0000 7f2527000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2520111b10 con 0x7f2520107860 2024-03-21T10:47:34.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.971+0000 7f2527000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2520111990 con 0x7f25200fff20 2024-03-21T10:47:34.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.971+0000 7f2527000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2520111c90 con 0x7f25201008e0 2024-03-21T10:47:34.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.972+0000 7f2517e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25201008e0 0x7f2520198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:34.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.972+0000 7f251fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25200fff20 0x7f2520197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:34.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.972+0000 7f251fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25200fff20 0x7f2520197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33374/0 (socket says 172.21.15.31:33374) 2024-03-21T10:47:34.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.972+0000 7f2517e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25201008e0 0x7f2520198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38644/0 (socket says 172.21.15.31:38644) 2024-03-21T10:47:34.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.972+0000 7f251fe00640 1 -- 172.21.15.31:0/445422007 learned_addr learned my addr 172.21.15.31:0/445422007 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:34.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.972+0000 7f2525000640 1 --2- 172.21.15.31:0/445422007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2520107860 0x7f252019c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:34.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.972+0000 7f2517e00640 1 -- 172.21.15.31:0/445422007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25200fff20 msgr2=0x7f2520197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:34.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.972+0000 7f2517e00640 1 --2- 172.21.15.31:0/445422007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25200fff20 0x7f2520197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:34.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.972+0000 7f2517e00640 1 -- 172.21.15.31:0/445422007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2520107860 msgr2=0x7f252019c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:34.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.972+0000 7f2517e00640 1 --2- 172.21.15.31:0/445422007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2520107860 0x7f252019c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:34.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.972+0000 7f2517e00640 1 -- 172.21.15.31:0/445422007 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f252019cf40 con 0x7f25201008e0 2024-03-21T10:47:34.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.972+0000 7f2517e00640 1 --2- 172.21.15.31:0/445422007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25201008e0 0x7f2520198490 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f2508036040 tx=0x7f25080042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:34.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.973+0000 7f251d600640 1 -- 172.21.15.31:0/445422007 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2508042020 con 0x7f25201008e0 2024-03-21T10:47:34.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.973+0000 7f251d600640 1 -- 172.21.15.31:0/445422007 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f25080054e0 con 0x7f25201008e0 2024-03-21T10:47:34.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.973+0000 7f251d600640 1 -- 172.21.15.31:0/445422007 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2508003950 con 0x7f25201008e0 2024-03-21T10:47:34.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.973+0000 7f2527000640 1 -- 172.21.15.31:0/445422007 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f252019d1d0 con 0x7f25201008e0 2024-03-21T10:47:34.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.973+0000 7f2527000640 1 -- 172.21.15.31:0/445422007 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25201a4ac0 con 0x7f25201008e0 2024-03-21T10:47:34.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.977+0000 7f2529d96740 1 Processor -- start 2024-03-21T10:47:34.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.977+0000 7f2529d96740 1 -- start start 2024-03-21T10:47:34.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.978+0000 7f251d600640 1 -- 172.21.15.31:0/445422007 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f2508045020 con 0x7f25201008e0 2024-03-21T10:47:34.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.978+0000 7f2529d96740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ec2fd3db0 0x558ec2fd4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:34.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.978+0000 7f2529d96740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ec2fd46d0 0x558ec2fe1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:34.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.978+0000 7f2529d96740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ec2fe1990 0x558ec2fe3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:34.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.978+0000 7f2529d96740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ec2f38eb0 con 0x558ec2fd3db0 2024-03-21T10:47:34.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.978+0000 7f2529d96740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ec2ee1a60 con 0x558ec2fd46d0 2024-03-21T10:47:34.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.978+0000 7f2529d96740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ec2f34f50 con 0x558ec2fe1990 2024-03-21T10:47:34.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.978+0000 7f251fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ec2fd46d0 0x558ec2fe1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:34.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.978+0000 7f2517e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ec2fe1990 0x558ec2fe3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:34.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.978+0000 7f2525000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ec2fd3db0 0x558ec2fd4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:34.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.978+0000 7f251fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ec2fd46d0 0x558ec2fe1450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33382/0 (socket says 172.21.15.31:33382) 2024-03-21T10:47:34.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.978+0000 7f251fe00640 1 -- 172.21.15.31:0/592411503 learned_addr learned my addr 172.21.15.31:0/592411503 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:34.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.978+0000 7f251fe00640 1 -- 172.21.15.31:0/592411503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ec2fe1990 msgr2=0x558ec2fe3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:34.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.978+0000 7f251fe00640 1 --2- 172.21.15.31:0/592411503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ec2fe1990 0x558ec2fe3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:34.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.978+0000 7f251fe00640 1 -- 172.21.15.31:0/592411503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ec2fd3db0 msgr2=0x558ec2fd4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:34.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.978+0000 7f251fe00640 1 --2- 172.21.15.31:0/592411503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ec2fd3db0 0x558ec2fd4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:34.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.978+0000 7f251fe00640 1 -- 172.21.15.31:0/592411503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558ec2d157e0 con 0x558ec2fd46d0 2024-03-21T10:47:34.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.979+0000 7f251fe00640 1 --2- 172.21.15.31:0/592411503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ec2fd46d0 0x558ec2fe1450 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f2510009e10 tx=0x7f2510011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5043157f87b613 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:34.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.979+0000 7f2515600640 1 -- 172.21.15.31:0/592411503 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251001e070 con 0x558ec2fd46d0 2024-03-21T10:47:34.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.979+0000 7f2515600640 1 -- 172.21.15.31:0/592411503 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2510019b10 con 0x558ec2fd46d0 2024-03-21T10:47:34.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.979+0000 7f2515600640 1 -- 172.21.15.31:0/592411503 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251000fc30 con 0x558ec2fd46d0 2024-03-21T10:47:34.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.980+0000 7f2529d96740 1 -- 172.21.15.31:0/592411503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ec2fd46d0 msgr2=0x558ec2fe1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:34.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.980+0000 7f2529d96740 1 --2- 172.21.15.31:0/592411503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ec2fd46d0 0x558ec2fe1450 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f2510009e10 tx=0x7f2510011810 comp rx=0 tx=0).stop 2024-03-21T10:47:34.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.980+0000 7f2529d96740 1 -- 172.21.15.31:0/592411503 shutdown_connections 2024-03-21T10:47:34.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.980+0000 7f2529d96740 1 --2- 172.21.15.31:0/592411503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ec2fe1990 0x558ec2fe3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:34.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.980+0000 7f2529d96740 1 --2- 172.21.15.31:0/592411503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ec2fd46d0 0x558ec2fe1450 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:34.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.980+0000 7f2529d96740 1 --2- 172.21.15.31:0/592411503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ec2fd3db0 0x558ec2fd4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:34.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.980+0000 7f2529d96740 1 -- 172.21.15.31:0/592411503 >> 172.21.15.31:0/592411503 conn(0x558ec2fce7b0 msgr2=0x558ec2fceb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:34.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.980+0000 7f251d600640 1 --2- 172.21.15.31:0/445422007 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f25040a3110 0x7f25040a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:34.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.980+0000 7f2529d96740 1 -- 172.21.15.31:0/592411503 shutdown_connections 2024-03-21T10:47:34.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.980+0000 7f251d600640 1 -- 172.21.15.31:0/445422007 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f25080f1570 con 0x7f25201008e0 2024-03-21T10:47:34.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.980+0000 7f2529d96740 1 -- 172.21.15.31:0/592411503 wait complete. 2024-03-21T10:47:34.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.980+0000 7f2525000640 1 --2- 172.21.15.31:0/445422007 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f25040a3110 0x7f25040a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:34.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.981+0000 7f2529d96740 1 -- start start 2024-03-21T10:47:34.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.981+0000 7f2529d96740 1 Processor -- start 2024-03-21T10:47:34.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.981+0000 7f2525000640 1 --2- 172.21.15.31:0/445422007 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f25040a3110 0x7f25040a55d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2518005fd0 tx=0x7f2518009500 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:34.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.981+0000 7f2529d96740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ec2fd3db0 0x558ec2fe1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:34.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.981+0000 7f2529d96740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ec2fd46d0 0x558ec2fde6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:34.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.982+0000 7f2529d96740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ec2fe1990 0x558ec2fdec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:34.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.982+0000 7f2529d96740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ec2fe4750 con 0x558ec2fe1990 2024-03-21T10:47:34.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.982+0000 7f2529d96740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ec2ef4eb0 con 0x558ec2fd3db0 2024-03-21T10:47:34.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.982+0000 7f2529d96740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ec2fe48d0 con 0x558ec2fd46d0 2024-03-21T10:47:34.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.982+0000 7f2529d96740 10 client.0 _ll_register_callbacks cb 0x558ec2ee9150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:34.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.982+0000 7f251fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ec2fd3db0 0x558ec2fe1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:34.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.982+0000 7f2517e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ec2fd46d0 0x558ec2fde6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:34.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.982+0000 7f251fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ec2fd3db0 0x558ec2fe1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33394/0 (socket says 172.21.15.31:33394) 2024-03-21T10:47:34.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.982+0000 7f251fe00640 1 -- 172.21.15.31:0/3952988199 learned_addr learned my addr 172.21.15.31:0/3952988199 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:34.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.982+0000 7f2525000640 1 --2- 172.21.15.31:0/3952988199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ec2fe1990 0x558ec2fdec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:34.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.982+0000 7f251fe00640 1 -- 172.21.15.31:0/3952988199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ec2fd46d0 msgr2=0x558ec2fde6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:34.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.982+0000 7f251fe00640 1 --2- 172.21.15.31:0/3952988199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ec2fd46d0 0x558ec2fde6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:34.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.982+0000 7f251fe00640 1 -- 172.21.15.31:0/3952988199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ec2fe1990 msgr2=0x558ec2fdec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:34.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.982+0000 7f251fe00640 1 --2- 172.21.15.31:0/3952988199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ec2fe1990 0x558ec2fdec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:34.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.982+0000 7f251fe00640 1 -- 172.21.15.31:0/3952988199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558ec2d75f70 con 0x558ec2fd3db0 2024-03-21T10:47:34.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.982+0000 7f2525000640 1 --2- 172.21.15.31:0/3952988199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ec2fe1990 0x558ec2fdec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:47:34.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.982+0000 7f2517e00640 1 --2- 172.21.15.31:0/3952988199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ec2fd46d0 0x558ec2fde6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:34.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.983+0000 7f251fe00640 1 --2- 172.21.15.31:0/3952988199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ec2fd3db0 0x558ec2fe1310 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f2510000c00 tx=0x7f251000f3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8db385c7d35c60b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:34.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.983+0000 7f250e000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:47:34.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.983+0000 7f250e000640 1 -- 172.21.15.31:0/3952988199 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2510027070 con 0x558ec2fd3db0 2024-03-21T10:47:34.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.983+0000 7f250e000640 1 -- 172.21.15.31:0/3952988199 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f251000fd80 con 0x558ec2fd3db0 2024-03-21T10:47:34.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.983+0000 7f2529d96740 1 -- 172.21.15.31:0/3952988199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2510019b10 con 0x558ec2fd3db0 2024-03-21T10:47:34.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.983+0000 7f250e000640 1 -- 172.21.15.31:0/3952988199 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2510033050 con 0x558ec2fd3db0 2024-03-21T10:47:34.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.983+0000 7f2529d96740 10 client.27890 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:34.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.983+0000 7f2529d96740 1 -- 172.21.15.31:0/3952988199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x558ec2c0d170 con 0x558ec2fd3db0 2024-03-21T10:47:34.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.983+0000 7f250e000640 1 -- 172.21.15.31:0/3952988199 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2510019b10 con 0x558ec2fd3db0 2024-03-21T10:47:34.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.984+0000 7f250e000640 1 -- 172.21.15.31:0/3952988199 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f25100046c0 con 0x558ec2fd3db0 2024-03-21T10:47:34.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.984+0000 7f2529d96740 10 client.27890 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:34.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.984+0000 7f2529d96740 10 client.27890 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:47:34.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.984+0000 7f2529d96740 20 client.27890 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:34.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.984+0000 7f2529d96740 1 --2- 172.21.15.31:0/3952988199 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558ec2ff1900 0x558ec2ff3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:34.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.984+0000 7f2529d96740 4 client.27890 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:47:34.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.984+0000 7f2529d96740 1 -- 172.21.15.31:0/3952988199 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558ec2f463f0 con 0x558ec2ff1900 2024-03-21T10:47:34.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.984+0000 7f2517e00640 1 --2- 172.21.15.31:0/3952988199 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558ec2ff1900 0x558ec2ff3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:34.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.985+0000 7f2517e00640 1 --2- 172.21.15.31:0/3952988199 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558ec2ff1900 0x558ec2ff3cf0 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7693244512198cd1 server_cookie=78a73e3c7532b6fe in_seq=0 out_seq=0 2024-03-21T10:47:34.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.985+0000 7f250e000640 10 client.27890 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:34.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.985+0000 7f250e000640 1 -- 172.21.15.31:0/3952988199 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558ec2f463f0 con 0x558ec2ff1900 2024-03-21T10:47:34.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.985+0000 7f250e000640 10 client.27890 handle_command_reply: tid=0 2024-03-21T10:47:34.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.985+0000 7f2529d96740 1 client.27890 shutdown 2024-03-21T10:47:34.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.986+0000 7f2529d96740 1 -- 172.21.15.31:0/3952988199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ec2fd3db0 msgr2=0x558ec2fe1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:34.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.986+0000 7f2529d96740 1 --2- 172.21.15.31:0/3952988199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ec2fd3db0 0x558ec2fe1310 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f2510000c00 tx=0x7f251000f3c0 comp rx=0 tx=0).stop 2024-03-21T10:47:34.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.986+0000 7f2529d96740 1 -- 172.21.15.31:0/3952988199 shutdown_connections 2024-03-21T10:47:34.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.986+0000 7f2529d96740 1 --2- 172.21.15.31:0/3952988199 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558ec2ff1900 0x558ec2ff3cf0 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:34.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.986+0000 7f2529d96740 1 --2- 172.21.15.31:0/3952988199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ec2fe1990 0x558ec2fdec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:34.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.986+0000 7f2529d96740 1 --2- 172.21.15.31:0/3952988199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ec2fd46d0 0x558ec2fde6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:34.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.986+0000 7f2529d96740 1 --2- 172.21.15.31:0/3952988199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ec2fd3db0 0x558ec2fe1310 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:34.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.986+0000 7f250e000640 0 client.27890 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:34.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.986+0000 7f2529d96740 1 -- 172.21.15.31:0/3952988199 >> 172.21.15.31:0/3952988199 conn(0x558ec2fce7b0 msgr2=0x558ec2fd0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:34.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.986+0000 7f2529d96740 1 -- 172.21.15.31:0/3952988199 shutdown_connections 2024-03-21T10:47:34.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.986+0000 7f2529d96740 1 -- 172.21.15.31:0/3952988199 wait complete. 2024-03-21T10:47:34.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:34.986+0000 7f2529d96740 20 client.27890 trim_cache size 0 max 16384 2024-03-21T10:47:35.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.003+0000 7f2529d96740 1 Processor -- start 2024-03-21T10:47:35.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.004+0000 7f2529d96740 1 -- start start 2024-03-21T10:47:35.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.004+0000 7f2529d96740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ec2fce7b0 0x558ec2fd9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:35.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.004+0000 7f2529d96740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ec2fd3db0 0x558ec2fe0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:35.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.004+0000 7f2529d96740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ec2fd46d0 0x558ec2fe1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:35.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.004+0000 7f2529d96740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25201aec80 con 0x558ec2fce7b0 2024-03-21T10:47:35.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.004+0000 7f2529d96740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25201ac780 con 0x558ec2fd46d0 2024-03-21T10:47:35.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.004+0000 7f2529d96740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25201aa840 con 0x558ec2fd3db0 2024-03-21T10:47:35.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.004+0000 7f251fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ec2fce7b0 0x558ec2fd9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:35.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.004+0000 7f2517e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ec2fd3db0 0x558ec2fe0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:35.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.004+0000 7f251fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ec2fce7b0 0x558ec2fd9b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54142/0 (socket says 172.21.15.31:54142) 2024-03-21T10:47:35.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.004+0000 7f251fe00640 1 -- 172.21.15.31:0/1986172090 learned_addr learned my addr 172.21.15.31:0/1986172090 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:35.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.004+0000 7f2525000640 1 --2- 172.21.15.31:0/1986172090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ec2fd46d0 0x558ec2fe1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:35.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.005+0000 7f2517e00640 1 -- 172.21.15.31:0/1986172090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ec2fd46d0 msgr2=0x558ec2fe1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:35.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.005+0000 7f2517e00640 1 --2- 172.21.15.31:0/1986172090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ec2fd46d0 0x558ec2fe1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.005+0000 7f2517e00640 1 -- 172.21.15.31:0/1986172090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ec2fce7b0 msgr2=0x558ec2fd9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:35.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.005+0000 7f2517e00640 1 --2- 172.21.15.31:0/1986172090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ec2fce7b0 0x558ec2fd9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.005+0000 7f2517e00640 1 -- 172.21.15.31:0/1986172090 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558ec2fdef90 con 0x558ec2fd3db0 2024-03-21T10:47:35.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.005+0000 7f251fe00640 1 --2- 172.21.15.31:0/1986172090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ec2fce7b0 0x558ec2fd9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:35.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.005+0000 7f2517e00640 1 --2- 172.21.15.31:0/1986172090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ec2fd3db0 0x558ec2fe0dd0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f25080eea30 tx=0x7f25080f3040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=757cc459ce8c90c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:35.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.005+0000 7f2514c00640 1 -- 172.21.15.31:0/1986172090 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25080bd4a0 con 0x558ec2fd3db0 2024-03-21T10:47:35.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.005+0000 7f2514c00640 1 -- 172.21.15.31:0/1986172090 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f25080a80b0 con 0x558ec2fd3db0 2024-03-21T10:47:35.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.005+0000 7f2514c00640 1 -- 172.21.15.31:0/1986172090 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25080044e0 con 0x558ec2fd3db0 2024-03-21T10:47:35.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.006+0000 7f2529d96740 1 -- 172.21.15.31:0/1986172090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ec2fd3db0 msgr2=0x558ec2fe0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:35.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.006+0000 7f2529d96740 1 --2- 172.21.15.31:0/1986172090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ec2fd3db0 0x558ec2fe0dd0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f25080eea30 tx=0x7f25080f3040 comp rx=0 tx=0).stop 2024-03-21T10:47:35.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.006+0000 7f2529d96740 1 -- 172.21.15.31:0/1986172090 shutdown_connections 2024-03-21T10:47:35.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.006+0000 7f2529d96740 1 --2- 172.21.15.31:0/1986172090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ec2fd46d0 0x558ec2fe1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.006+0000 7f2529d96740 1 --2- 172.21.15.31:0/1986172090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ec2fd3db0 0x558ec2fe0dd0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.006+0000 7f2529d96740 1 --2- 172.21.15.31:0/1986172090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ec2fce7b0 0x558ec2fd9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.006+0000 7f2529d96740 1 -- 172.21.15.31:0/1986172090 >> 172.21.15.31:0/1986172090 conn(0x558ec2ff4060 msgr2=0x558ec2fd93f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:35.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.006+0000 7f2529d96740 1 -- 172.21.15.31:0/1986172090 shutdown_connections 2024-03-21T10:47:35.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.006+0000 7f2529d96740 1 -- 172.21.15.31:0/1986172090 wait complete. 2024-03-21T10:47:35.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.006+0000 7f2529d96740 1 -- start start 2024-03-21T10:47:35.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.006+0000 7f2529d96740 1 Processor -- start 2024-03-21T10:47:35.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.007+0000 7f2529d96740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ec2fce7b0 0x558ec2fe6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:35.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.007+0000 7f2529d96740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ec2fd3db0 0x558ec2fe7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:35.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.007+0000 7f2529d96740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ec2fd46d0 0x558ec2fe75b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:35.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.007+0000 7f2529d96740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25201aae40 con 0x558ec2fce7b0 2024-03-21T10:47:35.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.007+0000 7f2529d96740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25201ac120 con 0x558ec2fd3db0 2024-03-21T10:47:35.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.007+0000 7f2529d96740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25201ab460 con 0x558ec2fd46d0 2024-03-21T10:47:35.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.007+0000 7f2529d96740 10 client.0 _ll_register_callbacks cb 0x7f251001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:35.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.007+0000 7f2517e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ec2fd3db0 0x558ec2fe7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:35.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.007+0000 7f2517e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ec2fd3db0 0x558ec2fe7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33410/0 (socket says 172.21.15.31:33410) 2024-03-21T10:47:35.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.007+0000 7f2517e00640 1 -- 172.21.15.31:0/1476553907 learned_addr learned my addr 172.21.15.31:0/1476553907 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:35.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.007+0000 7f2525000640 1 --2- 172.21.15.31:0/1476553907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ec2fd46d0 0x558ec2fe75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:35.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.007+0000 7f251fe00640 1 --2- 172.21.15.31:0/1476553907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ec2fce7b0 0x558ec2fe6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:35.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.007+0000 7f2517e00640 1 -- 172.21.15.31:0/1476553907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ec2fd46d0 msgr2=0x558ec2fe75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:35.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.007+0000 7f2517e00640 1 --2- 172.21.15.31:0/1476553907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ec2fd46d0 0x558ec2fe75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.007+0000 7f2517e00640 1 -- 172.21.15.31:0/1476553907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ec2fce7b0 msgr2=0x558ec2fe6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:35.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.007+0000 7f2517e00640 1 --2- 172.21.15.31:0/1476553907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ec2fce7b0 0x558ec2fe6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.007+0000 7f2517e00640 1 -- 172.21.15.31:0/1476553907 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558ec304d0c0 con 0x558ec2fd3db0 2024-03-21T10:47:35.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.008+0000 7f2525000640 1 --2- 172.21.15.31:0/1476553907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ec2fd46d0 0x558ec2fe75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:35.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.008+0000 7f2517e00640 1 --2- 172.21.15.31:0/1476553907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ec2fd3db0 0x558ec2fe7070 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f2508009930 tx=0x7f2508004560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a94c2e0ceb9d4f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:35.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.008+0000 7f250e000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:47:35.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.008+0000 7f250e000640 1 -- 172.21.15.31:0/1476553907 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25080bd030 con 0x558ec2fd3db0 2024-03-21T10:47:35.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.008+0000 7f250e000640 1 -- 172.21.15.31:0/1476553907 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f25080b9d40 con 0x558ec2fd3db0 2024-03-21T10:47:35.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.008+0000 7f2529d96740 1 -- 172.21.15.31:0/1476553907 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558ec2f40740 con 0x558ec2fd3db0 2024-03-21T10:47:35.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.008+0000 7f250e000640 1 -- 172.21.15.31:0/1476553907 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f25080bd210 con 0x558ec2fd3db0 2024-03-21T10:47:35.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.008+0000 7f2529d96740 10 client.27896 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:35.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.008+0000 7f2529d96740 1 -- 172.21.15.31:0/1476553907 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x558ec304e400 con 0x558ec2fd3db0 2024-03-21T10:47:35.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.008+0000 7f250e000640 1 -- 172.21.15.31:0/1476553907 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x558ec2f40740 con 0x558ec2fd3db0 2024-03-21T10:47:35.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.009+0000 7f250e000640 1 -- 172.21.15.31:0/1476553907 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f25080f1180 con 0x558ec2fd3db0 2024-03-21T10:47:35.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.009+0000 7f2529d96740 10 client.27896 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:35.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.009+0000 7f2529d96740 10 client.27896 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:47:35.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.009+0000 7f2529d96740 20 client.27896 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:35.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.009+0000 7f2529d96740 1 --2- 172.21.15.31:0/1476553907 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558ec2fe1990 0x558ec3050af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:35.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.009+0000 7f2529d96740 4 client.27896 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-21T10:47:35.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.009+0000 7f2529d96740 1 -- 172.21.15.31:0/1476553907 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x558ec2ebea40 con 0x558ec2fe1990 2024-03-21T10:47:35.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.009+0000 7f251fe00640 1 --2- 172.21.15.31:0/1476553907 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558ec2fe1990 0x558ec3050af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:35.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.009+0000 7f251fe00640 1 --2- 172.21.15.31:0/1476553907 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558ec2fe1990 0x558ec3050af0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=235d01d3d0d26935 server_cookie=1dc9a81e4d529486 in_seq=0 out_seq=0 2024-03-21T10:47:35.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.009+0000 7f250e000640 10 client.27896 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:35.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.010+0000 7f250e000640 1 -- 172.21.15.31:0/1476553907 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x558ec2ebea40 con 0x558ec2fe1990 2024-03-21T10:47:35.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.010+0000 7f250e000640 10 client.27896 handle_command_reply: tid=0 2024-03-21T10:47:35.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.010+0000 7f2529d96740 1 client.27896 shutdown 2024-03-21T10:47:35.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.011+0000 7f2529d96740 1 -- 172.21.15.31:0/1476553907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ec2fd3db0 msgr2=0x558ec2fe7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:35.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.011+0000 7f2529d96740 1 --2- 172.21.15.31:0/1476553907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ec2fd3db0 0x558ec2fe7070 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f2508009930 tx=0x7f2508004560 comp rx=0 tx=0).stop 2024-03-21T10:47:35.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.011+0000 7f2529d96740 1 -- 172.21.15.31:0/1476553907 shutdown_connections 2024-03-21T10:47:35.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.011+0000 7f2529d96740 1 --2- 172.21.15.31:0/1476553907 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x558ec2fe1990 0x558ec3050af0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.011+0000 7f2529d96740 1 --2- 172.21.15.31:0/1476553907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ec2fd46d0 0x558ec2fe75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.011+0000 7f2529d96740 1 --2- 172.21.15.31:0/1476553907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ec2fd3db0 0x558ec2fe7070 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.011+0000 7f2529d96740 1 --2- 172.21.15.31:0/1476553907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ec2fce7b0 0x558ec2fe6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.011+0000 7f2529d96740 1 -- 172.21.15.31:0/1476553907 >> 172.21.15.31:0/1476553907 conn(0x558ec2ff4060 msgr2=0x558ec2fd9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:35.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.011+0000 7f250e000640 0 client.27896 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:35.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.011+0000 7f2529d96740 1 -- 172.21.15.31:0/1476553907 shutdown_connections 2024-03-21T10:47:35.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.011+0000 7f2529d96740 1 -- 172.21.15.31:0/1476553907 wait complete. 2024-03-21T10:47:35.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.011+0000 7f2529d96740 20 client.27896 trim_cache size 0 max 16384 2024-03-21T10:47:35.022 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:47:35.022 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-03-21T10:47:35.022 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "101713a7-dbb2-4155-8c8e-6264a24e3d86", 2024-03-21T10:47:35.022 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-03-21T10:47:35.022 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:47:35.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.012+0000 7f2527000640 1 -- 172.21.15.31:0/445422007 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f25040a3110 msgr2=0x7f25040a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:35.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.012+0000 7f2527000640 1 --2- 172.21.15.31:0/445422007 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f25040a3110 0x7f25040a55d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2518005fd0 tx=0x7f2518009500 comp rx=0 tx=0).stop 2024-03-21T10:47:35.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.012+0000 7f2527000640 1 -- 172.21.15.31:0/445422007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25201008e0 msgr2=0x7f2520198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:35.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.012+0000 7f2527000640 1 --2- 172.21.15.31:0/445422007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25201008e0 0x7f2520198490 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f2508036040 tx=0x7f25080042f0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.012+0000 7f2527000640 1 -- 172.21.15.31:0/445422007 shutdown_connections 2024-03-21T10:47:35.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.012+0000 7f2527000640 1 --2- 172.21.15.31:0/445422007 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f25040a3110 0x7f25040a55d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.012+0000 7f2527000640 1 --2- 172.21.15.31:0/445422007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2520107860 0x7f252019c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.012+0000 7f2527000640 1 --2- 172.21.15.31:0/445422007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25201008e0 0x7f2520198490 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.012+0000 7f2527000640 1 --2- 172.21.15.31:0/445422007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25200fff20 0x7f2520197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.012+0000 7f2527000640 1 -- 172.21.15.31:0/445422007 >> 172.21.15.31:0/445422007 conn(0x7f2520075860 msgr2=0x7f25200fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:35.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.012+0000 7f2527000640 1 -- 172.21.15.31:0/445422007 shutdown_connections 2024-03-21T10:47:35.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.012+0000 7f2527000640 1 -- 172.21.15.31:0/445422007 wait complete. 2024-03-21T10:47:35.028 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:47:35.028 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:47:35.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.250+0000 7fc654000640 1 -- 172.21.15.31:0/649526391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc64c105380 msgr2=0x7fc64c1057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:35.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.250+0000 7fc654000640 1 --2- 172.21.15.31:0/649526391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc64c105380 0x7fc64c1057e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fc638009890 tx=0x7fc63802bc40 comp rx=0 tx=0).stop 2024-03-21T10:47:35.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.250+0000 7fc654000640 1 -- 172.21.15.31:0/649526391 shutdown_connections 2024-03-21T10:47:35.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.250+0000 7fc654000640 1 --2- 172.21.15.31:0/649526391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc64c105d20 0x7fc64c110850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.251+0000 7fc654000640 1 --2- 172.21.15.31:0/649526391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc64c105380 0x7fc64c1057e0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.251+0000 7fc654000640 1 --2- 172.21.15.31:0/649526391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64c103c40 0x7fc64c104020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.251+0000 7fc654000640 1 -- 172.21.15.31:0/649526391 >> 172.21.15.31:0/649526391 conn(0x7fc64c0ff350 msgr2=0x7fc64c101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:35.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.251+0000 7fc654000640 1 -- 172.21.15.31:0/649526391 shutdown_connections 2024-03-21T10:47:35.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.251+0000 7fc654000640 1 -- 172.21.15.31:0/649526391 wait complete. 2024-03-21T10:47:35.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.252+0000 7fc654000640 1 Processor -- start 2024-03-21T10:47:35.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.253+0000 7fc654000640 1 -- start start 2024-03-21T10:47:35.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.253+0000 7fc654000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc64c103c40 0x7fc64c19a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:35.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.253+0000 7fc654000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc64c105380 0x7fc64c19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:35.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.253+0000 7fc654000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64c105d20 0x7fc64c19eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:35.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.253+0000 7fc654000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c113d90 con 0x7fc64c103c40 2024-03-21T10:47:35.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.253+0000 7fc654000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c113c10 con 0x7fc64c105d20 2024-03-21T10:47:35.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.253+0000 7fc654000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c113f10 con 0x7fc64c105380 2024-03-21T10:47:35.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.253+0000 7fc651600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc64c103c40 0x7fc64c19a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:35.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.253+0000 7fc650c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc64c105380 0x7fc64c19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:35.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.253+0000 7fc651600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc64c103c40 0x7fc64c19a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54170/0 (socket says 172.21.15.31:54170) 2024-03-21T10:47:35.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.253+0000 7fc651600640 1 -- 172.21.15.31:0/3080611435 learned_addr learned my addr 172.21.15.31:0/3080611435 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:35.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.253+0000 7fc650c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc64c105380 0x7fc64c19a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38686/0 (socket says 172.21.15.31:38686) 2024-03-21T10:47:35.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.253+0000 7fc652000640 1 --2- 172.21.15.31:0/3080611435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64c105d20 0x7fc64c19eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:35.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.254+0000 7fc650c00640 1 -- 172.21.15.31:0/3080611435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64c105d20 msgr2=0x7fc64c19eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:35.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.254+0000 7fc650c00640 1 --2- 172.21.15.31:0/3080611435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64c105d20 0x7fc64c19eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.254+0000 7fc650c00640 1 -- 172.21.15.31:0/3080611435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc64c103c40 msgr2=0x7fc64c19a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:35.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.254+0000 7fc650c00640 1 --2- 172.21.15.31:0/3080611435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc64c103c40 0x7fc64c19a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.254+0000 7fc650c00640 1 -- 172.21.15.31:0/3080611435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc64c19f240 con 0x7fc64c105380 2024-03-21T10:47:35.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.254+0000 7fc650c00640 1 --2- 172.21.15.31:0/3080611435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc64c105380 0x7fc64c19a790 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fc6380057c0 tx=0x7fc638005d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:35.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.254+0000 7fc64a000640 1 -- 172.21.15.31:0/3080611435 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc638041020 con 0x7fc64c105380 2024-03-21T10:47:35.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.254+0000 7fc64a000640 1 -- 172.21.15.31:0/3080611435 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc63802ca60 con 0x7fc64c105380 2024-03-21T10:47:35.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.254+0000 7fc64a000640 1 -- 172.21.15.31:0/3080611435 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc638039040 con 0x7fc64c105380 2024-03-21T10:47:35.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.254+0000 7fc654000640 1 -- 172.21.15.31:0/3080611435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc64c19f4d0 con 0x7fc64c105380 2024-03-21T10:47:35.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.255+0000 7fc654000640 1 -- 172.21.15.31:0/3080611435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc64c1a6dc0 con 0x7fc64c105380 2024-03-21T10:47:35.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.258+0000 7fc64a000640 1 -- 172.21.15.31:0/3080611435 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fc638044020 con 0x7fc64c105380 2024-03-21T10:47:35.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.258+0000 7fc656c22740 1 Processor -- start 2024-03-21T10:47:35.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.258+0000 7fc656c22740 1 -- start start 2024-03-21T10:47:35.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.259+0000 7fc656c22740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631461c9c00 0x5631461c8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:35.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.259+0000 7fc656c22740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631461c8f20 0x5631461c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:35.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.259+0000 7fc656c22740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631461d2290 0x5631461d4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:35.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.259+0000 7fc656c22740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56314612d250 con 0x5631461c8f20 2024-03-21T10:47:35.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.259+0000 7fc656c22740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631460d5a30 con 0x5631461c9c00 2024-03-21T10:47:35.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.259+0000 7fc656c22740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631461292f0 con 0x5631461d2290 2024-03-21T10:47:35.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.259+0000 7fc652000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631461c9c00 0x5631461c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:35.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.259+0000 7fc652000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631461c9c00 0x5631461c8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33436/0 (socket says 172.21.15.31:33436) 2024-03-21T10:47:35.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.259+0000 7fc652000640 1 -- 172.21.15.31:0/3341937033 learned_addr learned my addr 172.21.15.31:0/3341937033 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:35.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.259+0000 7fc651600640 1 --2- 172.21.15.31:0/3341937033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631461c8f20 0x5631461c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:35.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.259+0000 7fc650c00640 1 --2- 172.21.15.31:0/3341937033 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631461d2290 0x5631461d4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:35.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.259+0000 7fc652000640 1 -- 172.21.15.31:0/3341937033 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631461d2290 msgr2=0x5631461d4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:35.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.259+0000 7fc652000640 1 --2- 172.21.15.31:0/3341937033 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631461d2290 0x5631461d4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.259+0000 7fc652000640 1 -- 172.21.15.31:0/3341937033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631461c8f20 msgr2=0x5631461c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:35.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.259+0000 7fc652000640 1 --2- 172.21.15.31:0/3341937033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631461c8f20 0x5631461c93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.259+0000 7fc652000640 1 -- 172.21.15.31:0/3341937033 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5631460c0690 con 0x5631461c9c00 2024-03-21T10:47:35.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.259+0000 7fc651600640 1 --2- 172.21.15.31:0/3341937033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631461c8f20 0x5631461c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:47:35.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.260+0000 7fc652000640 1 --2- 172.21.15.31:0/3341937033 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631461c9c00 0x5631461c8950 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fc644010370 tx=0x7fc644010830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dba75c1c22912b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:35.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.260+0000 7fc64a000640 1 --2- 172.21.15.31:0/3080611435 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc6340a3110 0x7fc6340a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:35.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.260+0000 7fc63e600640 1 -- 172.21.15.31:0/3341937033 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc644016070 con 0x5631461c9c00 2024-03-21T10:47:35.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.260+0000 7fc63e600640 1 -- 172.21.15.31:0/3341937033 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc644004d20 con 0x5631461c9c00 2024-03-21T10:47:35.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.260+0000 7fc63e600640 1 -- 172.21.15.31:0/3341937033 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc644005480 con 0x5631461c9c00 2024-03-21T10:47:35.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.260+0000 7fc64a000640 1 -- 172.21.15.31:0/3080611435 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fc6380f0fd0 con 0x7fc64c105380 2024-03-21T10:47:35.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.260+0000 7fc651600640 1 --2- 172.21.15.31:0/3080611435 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc6340a3110 0x7fc6340a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:35.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.261+0000 7fc656c22740 1 -- 172.21.15.31:0/3341937033 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631461c9c00 msgr2=0x5631461c8950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:35.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.261+0000 7fc656c22740 1 --2- 172.21.15.31:0/3341937033 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631461c9c00 0x5631461c8950 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fc644010370 tx=0x7fc644010830 comp rx=0 tx=0).stop 2024-03-21T10:47:35.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.261+0000 7fc656c22740 1 -- 172.21.15.31:0/3341937033 shutdown_connections 2024-03-21T10:47:35.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.261+0000 7fc656c22740 1 --2- 172.21.15.31:0/3341937033 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631461d2290 0x5631461d4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.261+0000 7fc656c22740 1 --2- 172.21.15.31:0/3341937033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631461c8f20 0x5631461c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.261+0000 7fc656c22740 1 --2- 172.21.15.31:0/3341937033 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631461c9c00 0x5631461c8950 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.261+0000 7fc656c22740 1 -- 172.21.15.31:0/3341937033 >> 172.21.15.31:0/3341937033 conn(0x563146190200 msgr2=0x5631461c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:35.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.261+0000 7fc651600640 1 --2- 172.21.15.31:0/3080611435 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc6340a3110 0x7fc6340a55d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x5631461ca150 tx=0x7fc640009260 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:35.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.261+0000 7fc656c22740 1 -- 172.21.15.31:0/3341937033 shutdown_connections 2024-03-21T10:47:35.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.261+0000 7fc656c22740 1 -- 172.21.15.31:0/3341937033 wait complete. 2024-03-21T10:47:35.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.262+0000 7fc656c22740 1 -- start start 2024-03-21T10:47:35.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.262+0000 7fc656c22740 1 Processor -- start 2024-03-21T10:47:35.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.262+0000 7fc656c22740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631461c8f20 0x5631461d1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:35.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.262+0000 7fc656c22740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631461c9c00 0x5631461cecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:35.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.262+0000 7fc656c22740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631461d2290 0x5631461cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:35.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.262+0000 7fc656c22740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631461d5100 con 0x5631461c9c00 2024-03-21T10:47:35.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.262+0000 7fc656c22740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631460e8e80 con 0x5631461d2290 2024-03-21T10:47:35.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.262+0000 7fc656c22740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631461d5280 con 0x5631461c8f20 2024-03-21T10:47:35.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.263+0000 7fc656c22740 10 client.0 _ll_register_callbacks cb 0x5631460fb130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:35.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.263+0000 7fc651600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631461c8f20 0x5631461d1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:35.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.263+0000 7fc651600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631461c8f20 0x5631461d1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38712/0 (socket says 172.21.15.31:38712) 2024-03-21T10:47:35.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.263+0000 7fc651600640 1 -- 172.21.15.31:0/1471719611 learned_addr learned my addr 172.21.15.31:0/1471719611 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:35.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.263+0000 7fc652000640 1 --2- 172.21.15.31:0/1471719611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631461d2290 0x5631461cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:35.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.263+0000 7fc650c00640 1 --2- 172.21.15.31:0/1471719611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631461c9c00 0x5631461cecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:35.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.263+0000 7fc651600640 1 -- 172.21.15.31:0/1471719611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631461d2290 msgr2=0x5631461cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:35.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.263+0000 7fc651600640 1 --2- 172.21.15.31:0/1471719611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631461d2290 0x5631461cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.263+0000 7fc651600640 1 -- 172.21.15.31:0/1471719611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631461c9c00 msgr2=0x5631461cecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:35.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.263+0000 7fc651600640 1 --2- 172.21.15.31:0/1471719611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631461c9c00 0x5631461cecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.263+0000 7fc651600640 1 -- 172.21.15.31:0/1471719611 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563145f0b7d0 con 0x5631461c8f20 2024-03-21T10:47:35.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.263+0000 7fc652000640 1 --2- 172.21.15.31:0/1471719611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631461d2290 0x5631461cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:35.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.263+0000 7fc650c00640 1 --2- 172.21.15.31:0/1471719611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631461c9c00 0x5631461cecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:47:35.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.264+0000 7fc651600640 1 --2- 172.21.15.31:0/1471719611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631461c8f20 0x5631461d1960 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc64001d960 tx=0x7fc640004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6953d10c563545c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:35.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.264+0000 7fc632a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:47:35.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.264+0000 7fc632a00640 1 -- 172.21.15.31:0/1471719611 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64002d060 con 0x5631461c8f20 2024-03-21T10:47:35.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.264+0000 7fc656c22740 1 -- 172.21.15.31:0/1471719611 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc644004d20 con 0x5631461c8f20 2024-03-21T10:47:35.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.264+0000 7fc632a00640 1 -- 172.21.15.31:0/1471719611 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fc640018030 con 0x5631461c8f20 2024-03-21T10:47:35.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.264+0000 7fc632a00640 1 -- 172.21.15.31:0/1471719611 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc640034050 con 0x5631461c8f20 2024-03-21T10:47:35.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.264+0000 7fc656c22740 10 client.27829 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:35.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.264+0000 7fc656c22740 1 -- 172.21.15.31:0/1471719611 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563145e413f0 con 0x5631461c8f20 2024-03-21T10:47:35.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.264+0000 7fc632a00640 1 -- 172.21.15.31:0/1471719611 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc644004d20 con 0x5631461c8f20 2024-03-21T10:47:35.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.265+0000 7fc632a00640 1 -- 172.21.15.31:0/1471719611 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc64000f5f0 con 0x5631461c8f20 2024-03-21T10:47:35.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.265+0000 7fc656c22740 10 client.27829 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:35.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.265+0000 7fc656c22740 10 client.27829 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:47:35.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.265+0000 7fc656c22740 20 client.27829 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:35.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.265+0000 7fc656c22740 1 --2- 172.21.15.31:0/1471719611 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5631461e2220 0x5631461e4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:35.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.265+0000 7fc656c22740 4 client.27829 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:47:35.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.265+0000 7fc656c22740 1 -- 172.21.15.31:0/1471719611 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56314613a790 con 0x5631461e2220 2024-03-21T10:47:35.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.265+0000 7fc650c00640 1 --2- 172.21.15.31:0/1471719611 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5631461e2220 0x5631461e4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:35.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.266+0000 7fc632a00640 10 client.27829 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:35.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.266+0000 7fc650c00640 1 --2- 172.21.15.31:0/1471719611 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5631461e2220 0x5631461e4650 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9d9591ab17002c1 server_cookie=1633bf52e5df6e5 in_seq=0 out_seq=0 2024-03-21T10:47:35.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.266+0000 7fc632a00640 1 -- 172.21.15.31:0/1471719611 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56314613a790 con 0x5631461e2220 2024-03-21T10:47:35.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.266+0000 7fc632a00640 10 client.27829 handle_command_reply: tid=0 2024-03-21T10:47:35.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.267+0000 7fc656c22740 1 client.27829 shutdown 2024-03-21T10:47:35.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.267+0000 7fc656c22740 1 -- 172.21.15.31:0/1471719611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631461c8f20 msgr2=0x5631461d1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:35.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.267+0000 7fc656c22740 1 --2- 172.21.15.31:0/1471719611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631461c8f20 0x5631461d1960 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc64001d960 tx=0x7fc640004730 comp rx=0 tx=0).stop 2024-03-21T10:47:35.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.267+0000 7fc656c22740 1 -- 172.21.15.31:0/1471719611 shutdown_connections 2024-03-21T10:47:35.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.267+0000 7fc656c22740 1 --2- 172.21.15.31:0/1471719611 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5631461e2220 0x5631461e4650 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.267+0000 7fc656c22740 1 --2- 172.21.15.31:0/1471719611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631461d2290 0x5631461cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.267+0000 7fc632a00640 0 client.27829 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:35.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.267+0000 7fc656c22740 1 --2- 172.21.15.31:0/1471719611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631461c9c00 0x5631461cecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.267+0000 7fc656c22740 1 --2- 172.21.15.31:0/1471719611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631461c8f20 0x5631461d1960 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.267+0000 7fc656c22740 1 -- 172.21.15.31:0/1471719611 >> 172.21.15.31:0/1471719611 conn(0x563146190200 msgr2=0x5631461c56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:35.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.267+0000 7fc656c22740 1 -- 172.21.15.31:0/1471719611 shutdown_connections 2024-03-21T10:47:35.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.267+0000 7fc656c22740 1 -- 172.21.15.31:0/1471719611 wait complete. 2024-03-21T10:47:35.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.267+0000 7fc656c22740 20 client.27829 trim_cache size 0 max 16384 2024-03-21T10:47:35.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.285+0000 7fc656c22740 1 Processor -- start 2024-03-21T10:47:35.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.285+0000 7fc656c22740 1 -- start start 2024-03-21T10:47:35.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.285+0000 7fc656c22740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631461c8f20 0x5631461d1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:35.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.285+0000 7fc656c22740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631461c9c00 0x5631461d1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:35.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.285+0000 7fc656c22740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631461d2290 0x5631461d3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:35.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.285+0000 7fc656c22740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c1b30e0 con 0x5631461d2290 2024-03-21T10:47:35.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.285+0000 7fc656c22740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c1b0be0 con 0x5631461c9c00 2024-03-21T10:47:35.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.285+0000 7fc656c22740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c1aeca0 con 0x5631461c8f20 2024-03-21T10:47:35.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.286+0000 7fc651600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631461c8f20 0x5631461d1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:35.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.286+0000 7fc650c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631461c9c00 0x5631461d1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:35.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.286+0000 7fc651600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631461c8f20 0x5631461d1420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:38718/0 (socket says 172.21.15.31:38718) 2024-03-21T10:47:35.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.286+0000 7fc651600640 1 -- 172.21.15.31:0/1108797640 learned_addr learned my addr 172.21.15.31:0/1108797640 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:35.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.286+0000 7fc652000640 1 --2- 172.21.15.31:0/1108797640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631461d2290 0x5631461d3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:35.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.286+0000 7fc651600640 1 -- 172.21.15.31:0/1108797640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631461c9c00 msgr2=0x5631461d1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:35.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.286+0000 7fc651600640 1 --2- 172.21.15.31:0/1108797640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631461c9c00 0x5631461d1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.286+0000 7fc651600640 1 -- 172.21.15.31:0/1108797640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631461d2290 msgr2=0x5631461d3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:35.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.286+0000 7fc651600640 1 --2- 172.21.15.31:0/1108797640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631461d2290 0x5631461d3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.286+0000 7fc651600640 1 -- 172.21.15.31:0/1108797640 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563145e010f0 con 0x5631461c8f20 2024-03-21T10:47:35.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.286+0000 7fc651600640 1 --2- 172.21.15.31:0/1108797640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631461c8f20 0x5631461d1420 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fc6400095d0 tx=0x7fc640009d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bfc37832751bb77 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:35.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.286+0000 7fc63dc00640 1 -- 172.21.15.31:0/1108797640 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64003d070 con 0x5631461c8f20 2024-03-21T10:47:35.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.287+0000 7fc63dc00640 1 -- 172.21.15.31:0/1108797640 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc64000fbb0 con 0x5631461c8f20 2024-03-21T10:47:35.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.287+0000 7fc63dc00640 1 -- 172.21.15.31:0/1108797640 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc640004a00 con 0x5631461c8f20 2024-03-21T10:47:35.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.287+0000 7fc656c22740 1 -- 172.21.15.31:0/1108797640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631461c8f20 msgr2=0x5631461d1420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:35.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.287+0000 7fc656c22740 1 --2- 172.21.15.31:0/1108797640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631461c8f20 0x5631461d1420 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fc6400095d0 tx=0x7fc640009d90 comp rx=0 tx=0).stop 2024-03-21T10:47:35.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.287+0000 7fc656c22740 1 -- 172.21.15.31:0/1108797640 shutdown_connections 2024-03-21T10:47:35.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.287+0000 7fc656c22740 1 --2- 172.21.15.31:0/1108797640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631461d2290 0x5631461d3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.287+0000 7fc656c22740 1 --2- 172.21.15.31:0/1108797640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631461c9c00 0x5631461d1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.287+0000 7fc656c22740 1 --2- 172.21.15.31:0/1108797640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631461c8f20 0x5631461d1420 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.287+0000 7fc656c22740 1 -- 172.21.15.31:0/1108797640 >> 172.21.15.31:0/1108797640 conn(0x563146190200 msgr2=0x5631461cd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:35.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.287+0000 7fc656c22740 1 -- 172.21.15.31:0/1108797640 shutdown_connections 2024-03-21T10:47:35.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.287+0000 7fc656c22740 1 -- 172.21.15.31:0/1108797640 wait complete. 2024-03-21T10:47:35.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.288+0000 7fc656c22740 1 -- start start 2024-03-21T10:47:35.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.288+0000 7fc656c22740 1 Processor -- start 2024-03-21T10:47:35.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.288+0000 7fc656c22740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631461c8f20 0x5631461cba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:35.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.288+0000 7fc656c22740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631461c9c00 0x5631461cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:35.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.288+0000 7fc656c22740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631461d2290 0x56314623cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:35.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.288+0000 7fc656c22740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c1af2a0 con 0x5631461c8f20 2024-03-21T10:47:35.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.288+0000 7fc656c22740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c1b0580 con 0x5631461d2290 2024-03-21T10:47:35.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.289+0000 7fc656c22740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c1af8c0 con 0x5631461c9c00 2024-03-21T10:47:35.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.289+0000 7fc656c22740 10 client.0 _ll_register_callbacks cb 0x7fc640009ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:35.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.289+0000 7fc652000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631461d2290 0x56314623cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:35.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.289+0000 7fc650c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631461c9c00 0x5631461cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:35.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.289+0000 7fc652000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631461d2290 0x56314623cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33452/0 (socket says 172.21.15.31:33452) 2024-03-21T10:47:35.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.289+0000 7fc652000640 1 -- 172.21.15.31:0/522409451 learned_addr learned my addr 172.21.15.31:0/522409451 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:35.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.289+0000 7fc651600640 1 --2- 172.21.15.31:0/522409451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631461c8f20 0x5631461cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:35.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.289+0000 7fc652000640 1 -- 172.21.15.31:0/522409451 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631461c9c00 msgr2=0x5631461cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:35.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.289+0000 7fc652000640 1 --2- 172.21.15.31:0/522409451 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631461c9c00 0x5631461cbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.289+0000 7fc652000640 1 -- 172.21.15.31:0/522409451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631461c8f20 msgr2=0x5631461cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:35.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.289+0000 7fc652000640 1 --2- 172.21.15.31:0/522409451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631461c8f20 0x5631461cba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.289+0000 7fc652000640 1 -- 172.21.15.31:0/522409451 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56314623d070 con 0x5631461d2290 2024-03-21T10:47:35.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.289+0000 7fc651600640 1 --2- 172.21.15.31:0/522409451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631461c8f20 0x5631461cba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:47:35.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.289+0000 7fc650c00640 1 --2- 172.21.15.31:0/522409451 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631461c9c00 0x5631461cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:35.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.289+0000 7fc652000640 1 --2- 172.21.15.31:0/522409451 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631461d2290 0x56314623cae0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fc644010370 tx=0x7fc644005380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57732cebb7492085 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:35.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.289+0000 7fc632a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:47:35.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.290+0000 7fc632a00640 1 -- 172.21.15.31:0/522409451 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc644029060 con 0x5631461d2290 2024-03-21T10:47:35.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.290+0000 7fc656c22740 1 -- 172.21.15.31:0/522409451 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc64000fbb0 con 0x5631461d2290 2024-03-21T10:47:35.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.290+0000 7fc632a00640 1 -- 172.21.15.31:0/522409451 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fc644005900 con 0x5631461d2290 2024-03-21T10:47:35.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.290+0000 7fc632a00640 1 -- 172.21.15.31:0/522409451 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc644030050 con 0x5631461d2290 2024-03-21T10:47:35.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.290+0000 7fc656c22740 10 client.27914 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:35.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.290+0000 7fc656c22740 1 -- 172.21.15.31:0/522409451 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56314623e3b0 con 0x5631461d2290 2024-03-21T10:47:35.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.290+0000 7fc632a00640 1 -- 172.21.15.31:0/522409451 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc64000fbb0 con 0x5631461d2290 2024-03-21T10:47:35.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.290+0000 7fc632a00640 1 -- 172.21.15.31:0/522409451 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc64400f040 con 0x5631461d2290 2024-03-21T10:47:35.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.291+0000 7fc656c22740 10 client.27914 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:35.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.291+0000 7fc656c22740 10 client.27914 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:47:35.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.291+0000 7fc656c22740 20 client.27914 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:35.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.291+0000 7fc656c22740 1 --2- 172.21.15.31:0/522409451 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56314623e7b0 0x563146240ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:35.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.291+0000 7fc656c22740 4 client.27914 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:47:35.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.291+0000 7fc656c22740 1 -- 172.21.15.31:0/522409451 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563146134ae0 con 0x56314623e7b0 2024-03-21T10:47:35.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.291+0000 7fc651600640 1 --2- 172.21.15.31:0/522409451 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56314623e7b0 0x563146240ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:35.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.291+0000 7fc651600640 1 --2- 172.21.15.31:0/522409451 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56314623e7b0 0x563146240ba0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34086a9ad8fcda7a server_cookie=6f1afd0aed4cc8b in_seq=0 out_seq=0 2024-03-21T10:47:35.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.291+0000 7fc632a00640 10 client.27914 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:35.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.291+0000 7fc632a00640 1 -- 172.21.15.31:0/522409451 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x563146134ae0 con 0x56314623e7b0 2024-03-21T10:47:35.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.292+0000 7fc632a00640 10 client.27914 handle_command_reply: tid=0 2024-03-21T10:47:35.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.292+0000 7fc656c22740 1 client.27914 shutdown 2024-03-21T10:47:35.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.292+0000 7fc656c22740 1 -- 172.21.15.31:0/522409451 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631461d2290 msgr2=0x56314623cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:35.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.292+0000 7fc656c22740 1 --2- 172.21.15.31:0/522409451 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631461d2290 0x56314623cae0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fc644010370 tx=0x7fc644005380 comp rx=0 tx=0).stop 2024-03-21T10:47:35.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.292+0000 7fc656c22740 1 -- 172.21.15.31:0/522409451 shutdown_connections 2024-03-21T10:47:35.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.292+0000 7fc656c22740 1 --2- 172.21.15.31:0/522409451 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56314623e7b0 0x563146240ba0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.292+0000 7fc656c22740 1 --2- 172.21.15.31:0/522409451 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631461d2290 0x56314623cae0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.292+0000 7fc656c22740 1 --2- 172.21.15.31:0/522409451 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631461c9c00 0x5631461cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.292+0000 7fc656c22740 1 --2- 172.21.15.31:0/522409451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631461c8f20 0x5631461cba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.292+0000 7fc632a00640 0 client.27914 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:35.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.292+0000 7fc656c22740 1 -- 172.21.15.31:0/522409451 >> 172.21.15.31:0/522409451 conn(0x563146190200 msgr2=0x5631461c4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:35.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.292+0000 7fc656c22740 1 -- 172.21.15.31:0/522409451 shutdown_connections 2024-03-21T10:47:35.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.292+0000 7fc656c22740 1 -- 172.21.15.31:0/522409451 wait complete. 2024-03-21T10:47:35.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.292+0000 7fc656c22740 20 client.27914 trim_cache size 0 max 16384 2024-03-21T10:47:35.294 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:47:35.294 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-21T10:47:35.294 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-03-21T10:47:35.294 INFO:teuthology.orchestra.run.smithi031.stdout: "101713a7-dbb2-4155-8c8e-6264a24e3d86": { 2024-03-21T10:47:35.294 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-03-21T10:47:35.294 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "101713a7-dbb2-4155-8c8e-6264a24e3d86", 2024-03-21T10:47:35.294 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-03-21T10:47:35.294 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:47:35.294 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:47:35.295 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:47:35.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.293+0000 7fc654000640 1 -- 172.21.15.31:0/3080611435 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc6340a3110 msgr2=0x7fc6340a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:35.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.293+0000 7fc654000640 1 --2- 172.21.15.31:0/3080611435 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc6340a3110 0x7fc6340a55d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x5631461ca150 tx=0x7fc640009260 comp rx=0 tx=0).stop 2024-03-21T10:47:35.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.293+0000 7fc654000640 1 -- 172.21.15.31:0/3080611435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc64c105380 msgr2=0x7fc64c19a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:35.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.293+0000 7fc654000640 1 --2- 172.21.15.31:0/3080611435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc64c105380 0x7fc64c19a790 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fc6380057c0 tx=0x7fc638005d00 comp rx=0 tx=0).stop 2024-03-21T10:47:35.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.293+0000 7fc654000640 1 -- 172.21.15.31:0/3080611435 shutdown_connections 2024-03-21T10:47:35.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.293+0000 7fc654000640 1 --2- 172.21.15.31:0/3080611435 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc6340a3110 0x7fc6340a55d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.293+0000 7fc654000640 1 --2- 172.21.15.31:0/3080611435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64c105d20 0x7fc64c19eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.293+0000 7fc654000640 1 --2- 172.21.15.31:0/3080611435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc64c105380 0x7fc64c19a790 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.293+0000 7fc654000640 1 --2- 172.21.15.31:0/3080611435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc64c103c40 0x7fc64c19a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:35.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.293+0000 7fc654000640 1 -- 172.21.15.31:0/3080611435 >> 172.21.15.31:0/3080611435 conn(0x7fc64c0ff350 msgr2=0x7fc64c1076d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:35.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.293+0000 7fc654000640 1 -- 172.21.15.31:0/3080611435 shutdown_connections 2024-03-21T10:47:35.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:47:35.293+0000 7fc654000640 1 -- 172.21.15.31:0/3080611435 wait complete. 2024-03-21T10:47:35.307 INFO:tasks.cephfs.filesystem:scrub status for tag:101713a7-dbb2-4155-8c8e-6264a24e3d86 - {'path': '/', 'tag': '101713a7-dbb2-4155-8c8e-6264a24e3d86', 'options': 'recursive,force'} 2024-03-21T10:47:35.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:35 smithi040 ceph-mon[36154]: pgmap v279: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-21T10:47:35.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:35 smithi031 ceph-mon[31673]: pgmap v279: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-21T10:47:35.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:35 smithi070 ceph-mon[36330]: pgmap v279: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-21T10:47:36.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:36 smithi040 ceph-mon[36154]: scrub queued for path: / 2024-03-21T10:47:36.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:36 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:47:36.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:36 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:47:36.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:36 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:47:36.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:36 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:47:36.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:36 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:47:36.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:36 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:47:36.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:36 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:47:36.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:36 smithi031 ceph-mon[31673]: scrub queued for path: / 2024-03-21T10:47:36.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:36 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:47:36.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:36 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:47:36.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:36 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:47:36.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:36 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:47:36.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:36 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:47:36.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:36 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:47:36.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:36 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:47:36.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:36 smithi070 ceph-mon[36330]: scrub queued for path: / 2024-03-21T10:47:36.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:36 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:47:36.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:36 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:47:36.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:36 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:47:36.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:36 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:47:36.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:36 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:47:36.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:36 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:47:36.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:36 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:47:37.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:37 smithi040 ceph-mon[36154]: pgmap v280: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-21T10:47:37.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:37 smithi031 ceph-mon[31673]: pgmap v280: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-21T10:47:37.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:37 smithi070 ceph-mon[36330]: pgmap v280: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-21T10:47:39.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:39 smithi040 ceph-mon[36154]: pgmap v281: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-21T10:47:39.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:39 smithi031 ceph-mon[31673]: pgmap v281: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-21T10:47:39.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:39 smithi070 ceph-mon[36330]: pgmap v281: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-21T10:47:41.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:41 smithi040 ceph-mon[36154]: pgmap v282: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-03-21T10:47:41.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:41 smithi031 ceph-mon[31673]: pgmap v282: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-03-21T10:47:41.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:41 smithi070 ceph-mon[36330]: pgmap v282: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-03-21T10:47:43.374 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:43 smithi070 ceph-mon[36330]: pgmap v283: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-21T10:47:43.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:43 smithi040 ceph-mon[36154]: pgmap v283: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-21T10:47:43.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:43 smithi031 ceph-mon[31673]: pgmap v283: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-21T10:47:45.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:45 smithi040 ceph-mon[36154]: pgmap v284: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-21T10:47:45.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:45 smithi031 ceph-mon[31673]: pgmap v284: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-21T10:47:45.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:45 smithi070 ceph-mon[36330]: pgmap v284: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-21T10:47:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.777+0000 7f3c57e00640 1 -- 172.21.15.31:0/2535347424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c58102430 msgr2=0x7f3c58068e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.777+0000 7f3c57e00640 1 --2- 172.21.15.31:0/2535347424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c58102430 0x7f3c58068e90 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f3c48009900 tx=0x7f3c4802bcd0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.777+0000 7f3c57e00640 1 -- 172.21.15.31:0/2535347424 shutdown_connections 2024-03-21T10:47:45.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.777+0000 7f3c57e00640 1 --2- 172.21.15.31:0/2535347424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c5810dc90 0x7f3c58110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.777+0000 7f3c57e00640 1 --2- 172.21.15.31:0/2535347424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c580693d0 0x7f3c58069810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.777+0000 7f3c57e00640 1 --2- 172.21.15.31:0/2535347424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c58102430 0x7f3c58068e90 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.777+0000 7f3c57e00640 1 -- 172.21.15.31:0/2535347424 >> 172.21.15.31:0/2535347424 conn(0x7f3c58075f00 msgr2=0x7f3c58076310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:45.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.778+0000 7f3c57e00640 1 -- 172.21.15.31:0/2535347424 shutdown_connections 2024-03-21T10:47:45.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.778+0000 7f3c57e00640 1 -- 172.21.15.31:0/2535347424 wait complete. 2024-03-21T10:47:45.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.779+0000 7f3c57e00640 1 Processor -- start 2024-03-21T10:47:45.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.779+0000 7f3c57e00640 1 -- start start 2024-03-21T10:47:45.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.779+0000 7f3c57e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c580693d0 0x7f3c58199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:45.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.779+0000 7f3c57e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c58102430 0x7f3c5819a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:45.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.779+0000 7f3c57e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c5810dc90 0x7f3c5819e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:45.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.779+0000 7f3c57e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c58113ab0 con 0x7f3c5810dc90 2024-03-21T10:47:45.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.779+0000 7f3c57e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c58113930 con 0x7f3c58102430 2024-03-21T10:47:45.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.779+0000 7f3c57e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c58113c30 con 0x7f3c580693d0 2024-03-21T10:47:45.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.779+0000 7f3c56000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c58102430 0x7f3c5819a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:45.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.779+0000 7f3c56a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c580693d0 0x7f3c58199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:45.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.779+0000 7f3c56000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c58102430 0x7f3c5819a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:54870/0 (socket says 172.21.15.31:54870) 2024-03-21T10:47:45.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.780+0000 7f3c56000640 1 -- 172.21.15.31:0/1929793805 learned_addr learned my addr 172.21.15.31:0/1929793805 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:45.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.780+0000 7f3c57400640 1 --2- 172.21.15.31:0/1929793805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c5810dc90 0x7f3c5819e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:45.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.780+0000 7f3c56a00640 1 -- 172.21.15.31:0/1929793805 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c58102430 msgr2=0x7f3c5819a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:45.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.780+0000 7f3c56a00640 1 --2- 172.21.15.31:0/1929793805 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c58102430 0x7f3c5819a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.780+0000 7f3c56a00640 1 -- 172.21.15.31:0/1929793805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c5810dc90 msgr2=0x7f3c5819e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:45.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.780+0000 7f3c56a00640 1 --2- 172.21.15.31:0/1929793805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c5810dc90 0x7f3c5819e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.780+0000 7f3c56a00640 1 -- 172.21.15.31:0/1929793805 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5819ee90 con 0x7f3c580693d0 2024-03-21T10:47:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.780+0000 7f3c56a00640 1 --2- 172.21.15.31:0/1929793805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c580693d0 0x7f3c58199ef0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f3c4802c210 tx=0x7f3c48005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.780+0000 7f3c4f400640 1 -- 172.21.15.31:0/1929793805 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c48042020 con 0x7f3c580693d0 2024-03-21T10:47:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.780+0000 7f3c4f400640 1 -- 172.21.15.31:0/1929793805 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c4802ce00 con 0x7f3c580693d0 2024-03-21T10:47:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.781+0000 7f3c4f400640 1 -- 172.21.15.31:0/1929793805 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c48003a50 con 0x7f3c580693d0 2024-03-21T10:47:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.781+0000 7f3c57e00640 1 -- 172.21.15.31:0/1929793805 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c5819f120 con 0x7f3c580693d0 2024-03-21T10:47:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.781+0000 7f3c57e00640 1 -- 172.21.15.31:0/1929793805 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c581a6a10 con 0x7f3c580693d0 2024-03-21T10:47:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.784+0000 7f3c4f400640 1 -- 172.21.15.31:0/1929793805 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f3c48045020 con 0x7f3c580693d0 2024-03-21T10:47:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.784+0000 7f3c6081d740 1 Processor -- start 2024-03-21T10:47:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.784+0000 7f3c6081d740 1 -- start start 2024-03-21T10:47:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.784+0000 7f3c6081d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e81e0 0x5622014e85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.785+0000 7f3c6081d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5622014e8b00 0x5622014f1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.785+0000 7f3c6081d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622014f18d0 0x5622014f3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.785+0000 7f3c6081d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622014aa3b0 con 0x5622014e8b00 2024-03-21T10:47:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.785+0000 7f3c6081d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56220144c830 con 0x5622014e81e0 2024-03-21T10:47:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.785+0000 7f3c6081d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622013f55d0 con 0x5622014f18d0 2024-03-21T10:47:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.785+0000 7f3c57400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e81e0 0x5622014e85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.785+0000 7f3c57400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e81e0 0x5622014e85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:54876/0 (socket says 172.21.15.31:54876) 2024-03-21T10:47:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.785+0000 7f3c57400640 1 -- 172.21.15.31:0/3090612397 learned_addr learned my addr 172.21.15.31:0/3090612397 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.785+0000 7f3c56000640 1 --2- 172.21.15.31:0/3090612397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622014f18d0 0x5622014f3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.785+0000 7f3c56a00640 1 --2- 172.21.15.31:0/3090612397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5622014e8b00 0x5622014f1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.785+0000 7f3c57400640 1 -- 172.21.15.31:0/3090612397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622014f18d0 msgr2=0x5622014f3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.785+0000 7f3c57400640 1 --2- 172.21.15.31:0/3090612397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622014f18d0 0x5622014f3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.785+0000 7f3c57400640 1 -- 172.21.15.31:0/3090612397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5622014e8b00 msgr2=0x5622014f1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.785+0000 7f3c57400640 1 --2- 172.21.15.31:0/3090612397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5622014e8b00 0x5622014f1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.785+0000 7f3c57400640 1 -- 172.21.15.31:0/3090612397 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56220140d2c0 con 0x5622014e81e0 2024-03-21T10:47:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.786+0000 7f3c57400640 1 --2- 172.21.15.31:0/3090612397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e81e0 0x5622014e85c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3c50009fc0 tx=0x7f3c50010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8990e7f26a967c40 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.786+0000 7f3c4f400640 1 --2- 172.21.15.31:0/1929793805 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3c3c0a30f0 0x7f3c3c0a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.786+0000 7f3c43e00640 1 -- 172.21.15.31:0/3090612397 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c50003b70 con 0x5622014e81e0 2024-03-21T10:47:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.786+0000 7f3c43e00640 1 -- 172.21.15.31:0/3090612397 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c50003d10 con 0x5622014e81e0 2024-03-21T10:47:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.786+0000 7f3c43e00640 1 -- 172.21.15.31:0/3090612397 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c50011aa0 con 0x5622014e81e0 2024-03-21T10:47:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.786+0000 7f3c4f400640 1 -- 172.21.15.31:0/1929793805 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f3c480f1550 con 0x7f3c580693d0 2024-03-21T10:47:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.786+0000 7f3c56a00640 1 --2- 172.21.15.31:0/1929793805 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3c3c0a30f0 0x7f3c3c0a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.787+0000 7f3c6081d740 1 -- 172.21.15.31:0/3090612397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e81e0 msgr2=0x5622014e85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.787+0000 7f3c6081d740 1 --2- 172.21.15.31:0/3090612397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e81e0 0x5622014e85c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3c50009fc0 tx=0x7f3c50010860 comp rx=0 tx=0).stop 2024-03-21T10:47:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.787+0000 7f3c6081d740 1 -- 172.21.15.31:0/3090612397 shutdown_connections 2024-03-21T10:47:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.787+0000 7f3c6081d740 1 --2- 172.21.15.31:0/3090612397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622014f18d0 0x5622014f3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.787+0000 7f3c6081d740 1 --2- 172.21.15.31:0/3090612397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5622014e8b00 0x5622014f1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.787+0000 7f3c6081d740 1 --2- 172.21.15.31:0/3090612397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e81e0 0x5622014e85c0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.787+0000 7f3c6081d740 1 -- 172.21.15.31:0/3090612397 >> 172.21.15.31:0/3090612397 conn(0x5622014e2b00 msgr2=0x5622014e2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.787+0000 7f3c6081d740 1 -- 172.21.15.31:0/3090612397 shutdown_connections 2024-03-21T10:47:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.787+0000 7f3c56a00640 1 --2- 172.21.15.31:0/1929793805 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3c3c0a30f0 0x7f3c3c0a55b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3c480eeac0 tx=0x7f3c4802bcd0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.787+0000 7f3c6081d740 1 -- 172.21.15.31:0/3090612397 wait complete. 2024-03-21T10:47:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.787+0000 7f3c6081d740 1 -- start start 2024-03-21T10:47:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.788+0000 7f3c6081d740 1 Processor -- start 2024-03-21T10:47:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.788+0000 7f3c6081d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5622014e81e0 0x5622014ee9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.788+0000 7f3c6081d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e8b00 0x5622014eef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.788+0000 7f3c6081d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622014f18d0 0x5622014ef470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.788+0000 7f3c6081d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562201408a20 con 0x5622014e81e0 2024-03-21T10:47:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.788+0000 7f3c6081d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622014488d0 con 0x5622014e8b00 2024-03-21T10:47:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.788+0000 7f3c6081d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622014f4500 con 0x5622014f18d0 2024-03-21T10:47:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.788+0000 7f3c6081d740 10 client.0 _ll_register_callbacks cb 0x5622014451c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.788+0000 7f3c56000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e8b00 0x5622014eef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.788+0000 7f3c57400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5622014e81e0 0x5622014ee9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.788+0000 7f3c56000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e8b00 0x5622014eef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:54890/0 (socket says 172.21.15.31:54890) 2024-03-21T10:47:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.788+0000 7f3c56000640 1 -- 172.21.15.31:0/1607032836 learned_addr learned my addr 172.21.15.31:0/1607032836 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.789+0000 7f3c57400640 1 --2- 172.21.15.31:0/1607032836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622014f18d0 0x5622014ef470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.789+0000 7f3c56000640 1 -- 172.21.15.31:0/1607032836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622014f18d0 msgr2=0x5622014ef470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.789+0000 7f3c56000640 1 --2- 172.21.15.31:0/1607032836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622014f18d0 0x5622014ef470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.789+0000 7f3c56000640 1 -- 172.21.15.31:0/1607032836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5622014e81e0 msgr2=0x5622014ee9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.789+0000 7f3c56000640 1 --2- 172.21.15.31:0/1607032836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5622014e81e0 0x5622014ee9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.789+0000 7f3c56000640 1 -- 172.21.15.31:0/1607032836 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562201161210 con 0x5622014e8b00 2024-03-21T10:47:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.789+0000 7f3c57400640 1 --2- 172.21.15.31:0/1607032836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5622014e81e0 0x5622014ee9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:47:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.789+0000 7f3c56000640 1 --2- 172.21.15.31:0/1607032836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e8b00 0x5622014eef30 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f3c44012770 tx=0x7f3c44012c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b61763051a70f66a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.789+0000 7f3c40c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:47:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.789+0000 7f3c40c00640 1 -- 172.21.15.31:0/1607032836 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c44004050 con 0x5622014e8b00 2024-03-21T10:47:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.789+0000 7f3c6081d740 1 -- 172.21.15.31:0/1607032836 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3c50003d10 con 0x5622014e8b00 2024-03-21T10:47:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.789+0000 7f3c40c00640 1 -- 172.21.15.31:0/1607032836 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f3c440041f0 con 0x5622014e8b00 2024-03-21T10:47:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.790+0000 7f3c40c00640 1 -- 172.21.15.31:0/1607032836 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c44025050 con 0x5622014e8b00 2024-03-21T10:47:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.790+0000 7f3c6081d740 10 client.27926 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.790+0000 7f3c6081d740 1 -- 172.21.15.31:0/1607032836 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56220122d620 con 0x5622014e8b00 2024-03-21T10:47:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.790+0000 7f3c40c00640 1 -- 172.21.15.31:0/1607032836 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c50003d10 con 0x5622014e8b00 2024-03-21T10:47:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.790+0000 7f3c40c00640 1 -- 172.21.15.31:0/1607032836 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f3c440039f0 con 0x5622014e8b00 2024-03-21T10:47:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.790+0000 7f3c6081d740 10 client.27926 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.790+0000 7f3c6081d740 10 client.27926 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:47:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.790+0000 7f3c6081d740 20 client.27926 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.790+0000 7f3c6081d740 1 --2- 172.21.15.31:0/1607032836 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562201501a70 0x562201503ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.790+0000 7f3c6081d740 4 client.27926 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:47:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.790+0000 7f3c6081d740 1 -- 172.21.15.31:0/1607032836 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562201459d70 con 0x562201501a70 2024-03-21T10:47:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.791+0000 7f3c57400640 1 --2- 172.21.15.31:0/1607032836 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562201501a70 0x562201503ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.791+0000 7f3c57400640 1 --2- 172.21.15.31:0/1607032836 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562201501a70 0x562201503ea0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f079fc676f6fb539 server_cookie=58a4aeece51f3908 in_seq=0 out_seq=0 2024-03-21T10:47:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.791+0000 7f3c40c00640 10 client.27926 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.792+0000 7f3c40c00640 1 -- 172.21.15.31:0/1607032836 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562201459d70 con 0x562201501a70 2024-03-21T10:47:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.792+0000 7f3c40c00640 10 client.27926 handle_command_reply: tid=0 2024-03-21T10:47:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.792+0000 7f3c6081d740 1 client.27926 shutdown 2024-03-21T10:47:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.792+0000 7f3c6081d740 1 -- 172.21.15.31:0/1607032836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e8b00 msgr2=0x5622014eef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.792+0000 7f3c6081d740 1 --2- 172.21.15.31:0/1607032836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e8b00 0x5622014eef30 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f3c44012770 tx=0x7f3c44012c30 comp rx=0 tx=0).stop 2024-03-21T10:47:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.793+0000 7f3c6081d740 1 -- 172.21.15.31:0/1607032836 shutdown_connections 2024-03-21T10:47:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.793+0000 7f3c6081d740 1 --2- 172.21.15.31:0/1607032836 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x562201501a70 0x562201503ea0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.793+0000 7f3c6081d740 1 --2- 172.21.15.31:0/1607032836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622014f18d0 0x5622014ef470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.793+0000 7f3c6081d740 1 --2- 172.21.15.31:0/1607032836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e8b00 0x5622014eef30 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.793+0000 7f3c6081d740 1 --2- 172.21.15.31:0/1607032836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5622014e81e0 0x5622014ee9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.793+0000 7f3c6081d740 1 -- 172.21.15.31:0/1607032836 >> 172.21.15.31:0/1607032836 conn(0x5622014e2b00 msgr2=0x5622014e4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.793+0000 7f3c40c00640 0 client.27926 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.793+0000 7f3c6081d740 1 -- 172.21.15.31:0/1607032836 shutdown_connections 2024-03-21T10:47:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.793+0000 7f3c6081d740 1 -- 172.21.15.31:0/1607032836 wait complete. 2024-03-21T10:47:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.793+0000 7f3c6081d740 20 client.27926 trim_cache size 0 max 16384 2024-03-21T10:47:45.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.808+0000 7f3c6081d740 1 Processor -- start 2024-03-21T10:47:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.808+0000 7f3c6081d740 1 -- start start 2024-03-21T10:47:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.809+0000 7f3c6081d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e81e0 0x5622014e19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.809+0000 7f3c6081d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622014e8b00 0x5622014fcc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.809+0000 7f3c6081d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5622014f18d0 0x5622014fd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.809+0000 7f3c6081d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c581b2e70 con 0x5622014f18d0 2024-03-21T10:47:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.809+0000 7f3c6081d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c581b0970 con 0x5622014e81e0 2024-03-21T10:47:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.809+0000 7f3c6081d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c581aea30 con 0x5622014e8b00 2024-03-21T10:47:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.809+0000 7f3c57400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e81e0 0x5622014e19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.809+0000 7f3c57400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e81e0 0x5622014e19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:54892/0 (socket says 172.21.15.31:54892) 2024-03-21T10:47:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.809+0000 7f3c57400640 1 -- 172.21.15.31:0/98250344 learned_addr learned my addr 172.21.15.31:0/98250344 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.809+0000 7f3c56000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622014e8b00 0x5622014fcc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.809+0000 7f3c57400640 1 --2- 172.21.15.31:0/98250344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5622014f18d0 0x5622014fd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.809+0000 7f3c56000640 1 -- 172.21.15.31:0/98250344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e81e0 msgr2=0x5622014e19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.809+0000 7f3c56000640 1 --2- 172.21.15.31:0/98250344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e81e0 0x5622014e19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.809+0000 7f3c56000640 1 -- 172.21.15.31:0/98250344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5622014f18d0 msgr2=0x5622014fd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.809+0000 7f3c56000640 1 --2- 172.21.15.31:0/98250344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5622014f18d0 0x5622014fd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.809+0000 7f3c56000640 1 -- 172.21.15.31:0/98250344 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622014e0f80 con 0x5622014e8b00 2024-03-21T10:47:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.809+0000 7f3c57400640 1 --2- 172.21.15.31:0/98250344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e81e0 0x5622014e19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.810+0000 7f3c56000640 1 --2- 172.21.15.31:0/98250344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622014e8b00 0x5622014fcc10 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f3c44003fb0 tx=0x7f3c44009f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2afe7000a1180a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.810+0000 7f3c43400640 1 -- 172.21.15.31:0/98250344 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4402d020 con 0x5622014e8b00 2024-03-21T10:47:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.810+0000 7f3c43400640 1 -- 172.21.15.31:0/98250344 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c44010530 con 0x5622014e8b00 2024-03-21T10:47:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.810+0000 7f3c43400640 1 -- 172.21.15.31:0/98250344 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c44000e60 con 0x5622014e8b00 2024-03-21T10:47:45.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.810+0000 7f3c6081d740 1 -- 172.21.15.31:0/98250344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622014e8b00 msgr2=0x5622014fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:45.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.810+0000 7f3c6081d740 1 --2- 172.21.15.31:0/98250344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622014e8b00 0x5622014fcc10 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f3c44003fb0 tx=0x7f3c44009f60 comp rx=0 tx=0).stop 2024-03-21T10:47:45.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.810+0000 7f3c6081d740 1 -- 172.21.15.31:0/98250344 shutdown_connections 2024-03-21T10:47:45.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.810+0000 7f3c6081d740 1 --2- 172.21.15.31:0/98250344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5622014f18d0 0x5622014fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.810+0000 7f3c6081d740 1 --2- 172.21.15.31:0/98250344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622014e8b00 0x5622014fcc10 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.810+0000 7f3c6081d740 1 --2- 172.21.15.31:0/98250344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e81e0 0x5622014e19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.810+0000 7f3c6081d740 1 -- 172.21.15.31:0/98250344 >> 172.21.15.31:0/98250344 conn(0x5622014e2b00 msgr2=0x5622014e12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:45.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.811+0000 7f3c6081d740 1 -- 172.21.15.31:0/98250344 shutdown_connections 2024-03-21T10:47:45.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.811+0000 7f3c6081d740 1 -- 172.21.15.31:0/98250344 wait complete. 2024-03-21T10:47:45.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.811+0000 7f3c6081d740 1 -- start start 2024-03-21T10:47:45.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.811+0000 7f3c6081d740 1 Processor -- start 2024-03-21T10:47:45.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.812+0000 7f3c6081d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622014e81e0 0x5622014eb300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:45.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.812+0000 7f3c6081d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e8b00 0x5622014eb840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:45.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.812+0000 7f3c6081d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5622014f18d0 0x5622014ebd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:45.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.812+0000 7f3c6081d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c581af030 con 0x5622014f18d0 2024-03-21T10:47:45.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.812+0000 7f3c6081d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c581b0310 con 0x5622014e8b00 2024-03-21T10:47:45.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.812+0000 7f3c6081d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c581af650 con 0x5622014e81e0 2024-03-21T10:47:45.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.812+0000 7f3c6081d740 10 client.0 _ll_register_callbacks cb 0x7f3c440099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:45.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.812+0000 7f3c56000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e8b00 0x5622014eb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:45.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.812+0000 7f3c57400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622014e81e0 0x5622014eb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:45.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.812+0000 7f3c56000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e8b00 0x5622014eb840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:54904/0 (socket says 172.21.15.31:54904) 2024-03-21T10:47:45.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.812+0000 7f3c56000640 1 -- 172.21.15.31:0/2315604838 learned_addr learned my addr 172.21.15.31:0/2315604838 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:45.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.812+0000 7f3c57400640 1 --2- 172.21.15.31:0/2315604838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5622014f18d0 0x5622014ebd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:45.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.812+0000 7f3c56000640 1 -- 172.21.15.31:0/2315604838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622014e81e0 msgr2=0x5622014eb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:45.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.812+0000 7f3c56000640 1 --2- 172.21.15.31:0/2315604838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622014e81e0 0x5622014eb300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.812+0000 7f3c56000640 1 -- 172.21.15.31:0/2315604838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5622014f18d0 msgr2=0x5622014ebd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:45.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.812+0000 7f3c56000640 1 --2- 172.21.15.31:0/2315604838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5622014f18d0 0x5622014ebd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.812+0000 7f3c56000640 1 -- 172.21.15.31:0/2315604838 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56220155c4b0 con 0x5622014e8b00 2024-03-21T10:47:45.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.812+0000 7f3c57400640 1 --2- 172.21.15.31:0/2315604838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622014e81e0 0x5622014eb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:45.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.813+0000 7f3c56000640 1 --2- 172.21.15.31:0/2315604838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e8b00 0x5622014eb840 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f3c44000c00 tx=0x7f3c44005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19eac98922fffa7c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:45.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.813+0000 7f3c40c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:47:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.813+0000 7f3c40c00640 1 -- 172.21.15.31:0/2315604838 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c44036060 con 0x5622014e8b00 2024-03-21T10:47:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.813+0000 7f3c40c00640 1 -- 172.21.15.31:0/2315604838 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f3c44022040 con 0x5622014e8b00 2024-03-21T10:47:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.813+0000 7f3c6081d740 1 -- 172.21.15.31:0/2315604838 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3c44010530 con 0x5622014e8b00 2024-03-21T10:47:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.813+0000 7f3c40c00640 1 -- 172.21.15.31:0/2315604838 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c4403d050 con 0x5622014e8b00 2024-03-21T10:47:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.813+0000 7f3c6081d740 10 client.27932 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.813+0000 7f3c6081d740 1 -- 172.21.15.31:0/2315604838 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56220155d7a0 con 0x5622014e8b00 2024-03-21T10:47:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.814+0000 7f3c40c00640 1 -- 172.21.15.31:0/2315604838 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c44010530 con 0x5622014e8b00 2024-03-21T10:47:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.814+0000 7f3c40c00640 1 -- 172.21.15.31:0/2315604838 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f3c440106e0 con 0x5622014e8b00 2024-03-21T10:47:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.814+0000 7f3c6081d740 10 client.27932 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.814+0000 7f3c6081d740 10 client.27932 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:47:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.814+0000 7f3c6081d740 20 client.27932 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.814+0000 7f3c6081d740 1 --2- 172.21.15.31:0/2315604838 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56220155de30 0x562201560250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.814+0000 7f3c6081d740 4 client.27932 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:47:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.814+0000 7f3c6081d740 1 -- 172.21.15.31:0/2315604838 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5622014540c0 con 0x56220155de30 2024-03-21T10:47:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.814+0000 7f3c57400640 1 --2- 172.21.15.31:0/2315604838 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56220155de30 0x562201560250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.815+0000 7f3c57400640 1 --2- 172.21.15.31:0/2315604838 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56220155de30 0x562201560250 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10bad2946fc32f3d server_cookie=23a6996c8eab86bb in_seq=0 out_seq=0 2024-03-21T10:47:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.815+0000 7f3c40c00640 10 client.27932 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.818+0000 7f3c40c00640 1 -- 172.21.15.31:0/2315604838 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5622014540c0 con 0x56220155de30 2024-03-21T10:47:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.818+0000 7f3c40c00640 10 client.27932 handle_command_reply: tid=0 2024-03-21T10:47:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.818+0000 7f3c6081d740 1 client.27932 shutdown 2024-03-21T10:47:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.818+0000 7f3c6081d740 1 -- 172.21.15.31:0/2315604838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e8b00 msgr2=0x5622014eb840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.818+0000 7f3c6081d740 1 --2- 172.21.15.31:0/2315604838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e8b00 0x5622014eb840 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f3c44000c00 tx=0x7f3c44005f10 comp rx=0 tx=0).stop 2024-03-21T10:47:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.819+0000 7f3c6081d740 1 -- 172.21.15.31:0/2315604838 shutdown_connections 2024-03-21T10:47:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.819+0000 7f3c6081d740 1 --2- 172.21.15.31:0/2315604838 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56220155de30 0x562201560250 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.819+0000 7f3c6081d740 1 --2- 172.21.15.31:0/2315604838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5622014f18d0 0x5622014ebd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.819+0000 7f3c6081d740 1 --2- 172.21.15.31:0/2315604838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622014e8b00 0x5622014eb840 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.819+0000 7f3c6081d740 1 --2- 172.21.15.31:0/2315604838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622014e81e0 0x5622014eb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.819+0000 7f3c6081d740 1 -- 172.21.15.31:0/2315604838 >> 172.21.15.31:0/2315604838 conn(0x5622014e2b00 msgr2=0x5622014ef240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.819+0000 7f3c40c00640 0 client.27932 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.819+0000 7f3c6081d740 1 -- 172.21.15.31:0/2315604838 shutdown_connections 2024-03-21T10:47:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.819+0000 7f3c6081d740 1 -- 172.21.15.31:0/2315604838 wait complete. 2024-03-21T10:47:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.819+0000 7f3c6081d740 20 client.27932 trim_cache size 0 max 16384 2024-03-21T10:47:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:47:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:47:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:47:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:47:45.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.820+0000 7f3c57e00640 1 -- 172.21.15.31:0/1929793805 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3c3c0a30f0 msgr2=0x7f3c3c0a55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:45.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.820+0000 7f3c57e00640 1 --2- 172.21.15.31:0/1929793805 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3c3c0a30f0 0x7f3c3c0a55b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3c480eeac0 tx=0x7f3c4802bcd0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.820+0000 7f3c57e00640 1 -- 172.21.15.31:0/1929793805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c580693d0 msgr2=0x7f3c58199ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:45.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.820+0000 7f3c57e00640 1 --2- 172.21.15.31:0/1929793805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c580693d0 0x7f3c58199ef0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f3c4802c210 tx=0x7f3c48005e80 comp rx=0 tx=0).stop 2024-03-21T10:47:45.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.820+0000 7f3c57e00640 1 -- 172.21.15.31:0/1929793805 shutdown_connections 2024-03-21T10:47:45.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.820+0000 7f3c57e00640 1 --2- 172.21.15.31:0/1929793805 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3c3c0a30f0 0x7f3c3c0a55b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.820+0000 7f3c57e00640 1 --2- 172.21.15.31:0/1929793805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c5810dc90 0x7f3c5819e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.820+0000 7f3c57e00640 1 --2- 172.21.15.31:0/1929793805 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c58102430 0x7f3c5819a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.820+0000 7f3c57e00640 1 --2- 172.21.15.31:0/1929793805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c580693d0 0x7f3c58199ef0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:45.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.820+0000 7f3c57e00640 1 -- 172.21.15.31:0/1929793805 >> 172.21.15.31:0/1929793805 conn(0x7f3c58075f00 msgr2=0x7f3c580ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:45.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.820+0000 7f3c57e00640 1 -- 172.21.15.31:0/1929793805 shutdown_connections 2024-03-21T10:47:45.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:45.820+0000 7f3c57e00640 1 -- 172.21.15.31:0/1929793805 wait complete. 2024-03-21T10:47:47.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:47 smithi040 ceph-mon[36154]: pgmap v285: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-03-21T10:47:47.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:47 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:47:47.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:47 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:47:47.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:47 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:47:47.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:47 smithi031 ceph-mon[31673]: pgmap v285: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-03-21T10:47:47.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:47 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:47:47.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:47 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:47:47.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:47 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:47:47.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:47 smithi070 ceph-mon[36330]: pgmap v285: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-03-21T10:47:47.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:47 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:47:47.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:47 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:47:47.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:47 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:47:49.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:49 smithi040 ceph-mon[36154]: pgmap v286: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 7 op/s 2024-03-21T10:47:49.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:49 smithi031 ceph-mon[31673]: pgmap v286: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 7 op/s 2024-03-21T10:47:49.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:49 smithi070 ceph-mon[36330]: pgmap v286: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 7 op/s 2024-03-21T10:47:51.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:51 smithi040 ceph-mon[36154]: pgmap v287: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-03-21T10:47:51.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:51 smithi031 ceph-mon[31673]: pgmap v287: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-03-21T10:47:51.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:51 smithi070 ceph-mon[36330]: pgmap v287: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-03-21T10:47:53.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:53 smithi040 ceph-mon[36154]: pgmap v288: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-21T10:47:53.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:53 smithi031 ceph-mon[31673]: pgmap v288: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-21T10:47:53.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:53 smithi070 ceph-mon[36330]: pgmap v288: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-21T10:47:55.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:55 smithi040 ceph-mon[36154]: pgmap v289: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-21T10:47:55.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:55 smithi031 ceph-mon[31673]: pgmap v289: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-21T10:47:55.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:55 smithi070 ceph-mon[36330]: pgmap v289: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-21T10:47:57.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:57 smithi040 ceph-mon[36154]: pgmap v290: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-21T10:47:57.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:57 smithi031 ceph-mon[31673]: pgmap v290: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-21T10:47:57.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:57 smithi070 ceph-mon[36330]: pgmap v290: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-21T10:47:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.970+0000 7f8df0400640 1 -- 172.21.15.31:0/2819873987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8de8107f40 msgr2=0x7f8de8108320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.970+0000 7f8df0400640 1 --2- 172.21.15.31:0/2819873987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8de8107f40 0x7f8de8108320 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc009860 tx=0x7f8ddc02bb70 comp rx=0 tx=0).stop 2024-03-21T10:47:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.971+0000 7f8df0400640 1 -- 172.21.15.31:0/2819873987 shutdown_connections 2024-03-21T10:47:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.971+0000 7f8df0400640 1 --2- 172.21.15.31:0/2819873987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8de810cfc0 0x7f8de8114b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.971+0000 7f8df0400640 1 --2- 172.21.15.31:0/2819873987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8de81088f0 0x7f8de810c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.971+0000 7f8df0400640 1 --2- 172.21.15.31:0/2819873987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8de8107f40 0x7f8de8108320 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.971+0000 7f8df0400640 1 -- 172.21.15.31:0/2819873987 >> 172.21.15.31:0/2819873987 conn(0x7f8de80774e0 msgr2=0x7f8de81064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.971+0000 7f8df0400640 1 -- 172.21.15.31:0/2819873987 shutdown_connections 2024-03-21T10:47:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.971+0000 7f8df0400640 1 -- 172.21.15.31:0/2819873987 wait complete. 2024-03-21T10:47:58.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.972+0000 7f8df0400640 1 Processor -- start 2024-03-21T10:47:58.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.972+0000 7f8df0400640 1 -- start start 2024-03-21T10:47:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.973+0000 7f8df0400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8de8107f40 0x7f8de819e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.973+0000 7f8df0400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8de810cfc0 0x7f8de819ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.973+0000 7f8df0400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8de81a2e50 0x7f8de81a3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.973+0000 7f8df0400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de8118070 con 0x7f8de8107f40 2024-03-21T10:47:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.973+0000 7f8df0400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de8117ef0 con 0x7f8de810cfc0 2024-03-21T10:47:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.973+0000 7f8df0400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de81181f0 con 0x7f8de81a2e50 2024-03-21T10:47:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.973+0000 7f8deda00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8de8107f40 0x7f8de819e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.973+0000 7f8ded000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8de810cfc0 0x7f8de819ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.973+0000 7f8deda00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8de8107f40 0x7f8de819e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57876/0 (socket says 172.21.15.31:57876) 2024-03-21T10:47:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.973+0000 7f8dee400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8de81a2e50 0x7f8de81a3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.973+0000 7f8deda00640 1 -- 172.21.15.31:0/3835316886 learned_addr learned my addr 172.21.15.31:0/3835316886 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.974+0000 7f8ded000640 1 -- 172.21.15.31:0/3835316886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8de81a2e50 msgr2=0x7f8de81a3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.974+0000 7f8ded000640 1 --2- 172.21.15.31:0/3835316886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8de81a2e50 0x7f8de81a3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.974+0000 7f8ded000640 1 -- 172.21.15.31:0/3835316886 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8de8107f40 msgr2=0x7f8de819e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.974+0000 7f8ded000640 1 --2- 172.21.15.31:0/3835316886 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8de8107f40 0x7f8de819e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.974+0000 7f8ded000640 1 -- 172.21.15.31:0/3835316886 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de81a3940 con 0x7f8de810cfc0 2024-03-21T10:47:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.974+0000 7f8ded000640 1 --2- 172.21.15.31:0/3835316886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8de810cfc0 0x7f8de819ead0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f8de806ac90 tx=0x7f8dd400ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.974+0000 7f8de6000640 1 -- 172.21.15.31:0/3835316886 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd400ccd0 con 0x7f8de810cfc0 2024-03-21T10:47:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.974+0000 7f8de6000640 1 -- 172.21.15.31:0/3835316886 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8dd40035f0 con 0x7f8de810cfc0 2024-03-21T10:47:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.974+0000 7f8df0400640 1 -- 172.21.15.31:0/3835316886 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de81a3c30 con 0x7f8de810cfc0 2024-03-21T10:47:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.974+0000 7f8de6000640 1 -- 172.21.15.31:0/3835316886 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd4010670 con 0x7f8de810cfc0 2024-03-21T10:47:58.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.974+0000 7f8df0400640 1 -- 172.21.15.31:0/3835316886 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8de81ab420 con 0x7f8de810cfc0 2024-03-21T10:47:58.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.979+0000 7f8df31ba740 1 Processor -- start 2024-03-21T10:47:58.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.979+0000 7f8df31ba740 1 -- start start 2024-03-21T10:47:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.979+0000 7f8df31ba740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f7a51fd0 0x5634f7a523b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.980+0000 7f8df31ba740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f7a528f0 0x5634f7a5b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.980+0000 7f8df31ba740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634f7a5b6c0 0x5634f7a5dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.980+0000 7f8df31ba740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634f7a143b0 con 0x5634f7a5b6c0 2024-03-21T10:47:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.980+0000 7f8df31ba740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634f79b6830 con 0x5634f7a528f0 2024-03-21T10:47:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.980+0000 7f8df31ba740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634f795f5d0 con 0x5634f7a51fd0 2024-03-21T10:47:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.980+0000 7f8dee400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f7a51fd0 0x5634f7a523b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.980+0000 7f8dee400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f7a51fd0 0x5634f7a523b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53184/0 (socket says 172.21.15.31:53184) 2024-03-21T10:47:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.980+0000 7f8dee400640 1 -- 172.21.15.31:0/576153287 learned_addr learned my addr 172.21.15.31:0/576153287 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.980+0000 7f8deda00640 1 --2- 172.21.15.31:0/576153287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f7a528f0 0x5634f7a5b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.980+0000 7f8de6000640 1 -- 172.21.15.31:0/3835316886 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f8dd40037a0 con 0x7f8de810cfc0 2024-03-21T10:47:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.980+0000 7f8dee400640 1 -- 172.21.15.31:0/576153287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f7a528f0 msgr2=0x5634f7a5b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.980+0000 7f8dee400640 1 --2- 172.21.15.31:0/576153287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f7a528f0 0x5634f7a5b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.980+0000 7f8dee400640 1 -- 172.21.15.31:0/576153287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634f7a5b6c0 msgr2=0x5634f7a5dab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:47:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.980+0000 7f8ded000640 1 --2- 172.21.15.31:0/576153287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634f7a5b6c0 0x5634f7a5dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.981+0000 7f8dee400640 1 --2- 172.21.15.31:0/576153287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634f7a5b6c0 0x5634f7a5dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.981+0000 7f8dee400640 1 -- 172.21.15.31:0/576153287 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5634f79772c0 con 0x5634f7a51fd0 2024-03-21T10:47:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.981+0000 7f8ded000640 1 --2- 172.21.15.31:0/576153287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634f7a5b6c0 0x5634f7a5dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:47:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.981+0000 7f8dee400640 1 --2- 172.21.15.31:0/576153287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f7a51fd0 0x5634f7a523b0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f8de0009f40 tx=0x7f8de00118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=705055f31335ecb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.981+0000 7f8ddaa00640 1 -- 172.21.15.31:0/576153287 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de0004d20 con 0x5634f7a51fd0 2024-03-21T10:47:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.981+0000 7f8ddaa00640 1 -- 172.21.15.31:0/576153287 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8de0004050 con 0x5634f7a51fd0 2024-03-21T10:47:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.981+0000 7f8ddaa00640 1 -- 172.21.15.31:0/576153287 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de0005280 con 0x5634f7a51fd0 2024-03-21T10:47:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.982+0000 7f8de6000640 1 --2- 172.21.15.31:0/3835316886 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f8dd00a3080 0x7f8dd00a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.982+0000 7f8df31ba740 1 -- 172.21.15.31:0/576153287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f7a51fd0 msgr2=0x5634f7a523b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.982+0000 7f8df31ba740 1 --2- 172.21.15.31:0/576153287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f7a51fd0 0x5634f7a523b0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f8de0009f40 tx=0x7f8de00118d0 comp rx=0 tx=0).stop 2024-03-21T10:47:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.982+0000 7f8de6000640 1 -- 172.21.15.31:0/3835316886 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f8dd40cfd90 con 0x7f8de810cfc0 2024-03-21T10:47:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.982+0000 7f8df31ba740 1 -- 172.21.15.31:0/576153287 shutdown_connections 2024-03-21T10:47:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.982+0000 7f8df31ba740 1 --2- 172.21.15.31:0/576153287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634f7a5b6c0 0x5634f7a5dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.982+0000 7f8df31ba740 1 --2- 172.21.15.31:0/576153287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f7a528f0 0x5634f7a5b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.982+0000 7f8df31ba740 1 --2- 172.21.15.31:0/576153287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f7a51fd0 0x5634f7a523b0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.982+0000 7f8df31ba740 1 -- 172.21.15.31:0/576153287 >> 172.21.15.31:0/576153287 conn(0x5634f7a4c8f0 msgr2=0x5634f7a4ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.982+0000 7f8deda00640 1 --2- 172.21.15.31:0/3835316886 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f8dd00a3080 0x7f8dd00a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.982+0000 7f8df31ba740 1 -- 172.21.15.31:0/576153287 shutdown_connections 2024-03-21T10:47:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.982+0000 7f8df31ba740 1 -- 172.21.15.31:0/576153287 wait complete. 2024-03-21T10:47:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.983+0000 7f8deda00640 1 --2- 172.21.15.31:0/3835316886 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f8dd00a3080 0x7f8dd00a5540 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc0057c0 tx=0x7f8ddc0056d0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.983+0000 7f8df31ba740 1 -- start start 2024-03-21T10:47:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.983+0000 7f8df31ba740 1 Processor -- start 2024-03-21T10:47:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.984+0000 7f8df31ba740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f7a51fd0 0x5634f7a58730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.984+0000 7f8df31ba740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634f7a528f0 0x5634f7a58c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.984+0000 7f8df31ba740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f7a5b6c0 0x5634f7a591b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.984+0000 7f8df31ba740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634f7972a20 con 0x5634f7a528f0 2024-03-21T10:47:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.984+0000 7f8df31ba740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634f79b28d0 con 0x5634f7a5b6c0 2024-03-21T10:47:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.984+0000 7f8df31ba740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634f7a5e2f0 con 0x5634f7a51fd0 2024-03-21T10:47:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.984+0000 7f8df31ba740 10 client.0 _ll_register_callbacks cb 0x5634f79af1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.984+0000 7f8deda00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f7a51fd0 0x5634f7a58730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.984+0000 7f8ded000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634f7a528f0 0x5634f7a58c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.984+0000 7f8dee400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f7a5b6c0 0x5634f7a591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.984+0000 7f8deda00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f7a51fd0 0x5634f7a58730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53190/0 (socket says 172.21.15.31:53190) 2024-03-21T10:47:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.984+0000 7f8deda00640 1 -- 172.21.15.31:0/3272186375 learned_addr learned my addr 172.21.15.31:0/3272186375 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:58.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.985+0000 7f8deda00640 1 -- 172.21.15.31:0/3272186375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f7a5b6c0 msgr2=0x5634f7a591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:58.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.985+0000 7f8deda00640 1 --2- 172.21.15.31:0/3272186375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f7a5b6c0 0x5634f7a591b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.985+0000 7f8deda00640 1 -- 172.21.15.31:0/3272186375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634f7a528f0 msgr2=0x5634f7a58c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.985+0000 7f8deda00640 1 --2- 172.21.15.31:0/3272186375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634f7a528f0 0x5634f7a58c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.985+0000 7f8deda00640 1 -- 172.21.15.31:0/3272186375 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5634f76cb210 con 0x5634f7a51fd0 2024-03-21T10:47:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.985+0000 7f8deda00640 1 --2- 172.21.15.31:0/3272186375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f7a51fd0 0x5634f7a58730 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc004180 tx=0x7f8ddc02bb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a5cf80d11d0b43a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.985+0000 7f8dcf400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:47:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.985+0000 7f8dcf400640 1 -- 172.21.15.31:0/3272186375 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc04f020 con 0x5634f7a51fd0 2024-03-21T10:47:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.986+0000 7f8dcf400640 1 -- 172.21.15.31:0/3272186375 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f8ddc02c710 con 0x5634f7a51fd0 2024-03-21T10:47:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.986+0000 7f8df31ba740 1 -- 172.21.15.31:0/3272186375 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8de0004050 con 0x5634f7a51fd0 2024-03-21T10:47:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.986+0000 7f8dcf400640 1 -- 172.21.15.31:0/3272186375 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ddc05a050 con 0x5634f7a51fd0 2024-03-21T10:47:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.986+0000 7f8df31ba740 10 client.27865 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.986+0000 7f8df31ba740 1 -- 172.21.15.31:0/3272186375 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5634f7797620 con 0x5634f7a51fd0 2024-03-21T10:47:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.986+0000 7f8dcf400640 1 -- 172.21.15.31:0/3272186375 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8de0004050 con 0x5634f7a51fd0 2024-03-21T10:47:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.986+0000 7f8dcf400640 1 -- 172.21.15.31:0/3272186375 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f8ddc0035f0 con 0x5634f7a51fd0 2024-03-21T10:47:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.987+0000 7f8df31ba740 10 client.27865 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.987+0000 7f8df31ba740 10 client.27865 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:47:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.987+0000 7f8df31ba740 20 client.27865 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.987+0000 7f8df31ba740 1 --2- 172.21.15.31:0/3272186375 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5634f7a6b6d0 0x5634f7a6db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.987+0000 7f8df31ba740 4 client.27865 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:47:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.987+0000 7f8df31ba740 1 -- 172.21.15.31:0/3272186375 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5634f79c3d70 con 0x5634f7a6b6d0 2024-03-21T10:47:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.987+0000 7f8ded000640 1 --2- 172.21.15.31:0/3272186375 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5634f7a6b6d0 0x5634f7a6db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.988+0000 7f8ded000640 1 --2- 172.21.15.31:0/3272186375 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5634f7a6b6d0 0x5634f7a6db40 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a77eaad73c0c2025 server_cookie=ff843b2f8686841c in_seq=0 out_seq=0 2024-03-21T10:47:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.988+0000 7f8dcf400640 10 client.27865 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.988+0000 7f8dcf400640 1 -- 172.21.15.31:0/3272186375 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5634f79c3d70 con 0x5634f7a6b6d0 2024-03-21T10:47:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.988+0000 7f8dcf400640 10 client.27865 handle_command_reply: tid=0 2024-03-21T10:47:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.988+0000 7f8df31ba740 1 client.27865 shutdown 2024-03-21T10:47:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.988+0000 7f8df31ba740 1 -- 172.21.15.31:0/3272186375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f7a51fd0 msgr2=0x5634f7a58730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.988+0000 7f8df31ba740 1 --2- 172.21.15.31:0/3272186375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f7a51fd0 0x5634f7a58730 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc004180 tx=0x7f8ddc02bb70 comp rx=0 tx=0).stop 2024-03-21T10:47:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.989+0000 7f8df31ba740 1 -- 172.21.15.31:0/3272186375 shutdown_connections 2024-03-21T10:47:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.989+0000 7f8df31ba740 1 --2- 172.21.15.31:0/3272186375 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5634f7a6b6d0 0x5634f7a6db40 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.989+0000 7f8df31ba740 1 --2- 172.21.15.31:0/3272186375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f7a5b6c0 0x5634f7a591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.989+0000 7f8dcf400640 0 client.27865 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.989+0000 7f8df31ba740 1 --2- 172.21.15.31:0/3272186375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634f7a528f0 0x5634f7a58c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.989+0000 7f8df31ba740 1 --2- 172.21.15.31:0/3272186375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f7a51fd0 0x5634f7a58730 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.989+0000 7f8df31ba740 1 -- 172.21.15.31:0/3272186375 >> 172.21.15.31:0/3272186375 conn(0x5634f7a4c8f0 msgr2=0x5634f7a4e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.989+0000 7f8df31ba740 1 -- 172.21.15.31:0/3272186375 shutdown_connections 2024-03-21T10:47:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.989+0000 7f8df31ba740 1 -- 172.21.15.31:0/3272186375 wait complete. 2024-03-21T10:47:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:58.989+0000 7f8df31ba740 20 client.27865 trim_cache size 0 max 16384 2024-03-21T10:47:59.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.004+0000 7f8df31ba740 1 Processor -- start 2024-03-21T10:47:59.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.004+0000 7f8df31ba740 1 -- start start 2024-03-21T10:47:59.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.004+0000 7f8df31ba740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f7a51fd0 0x5634f7a5ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:59.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.004+0000 7f8df31ba740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634f7a528f0 0x5634f7a66a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:59.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.004+0000 7f8df31ba740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f7a5b6c0 0x5634f7a66f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:59.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.004+0000 7f8df31ba740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de81b7600 con 0x5634f7a528f0 2024-03-21T10:47:59.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.004+0000 7f8df31ba740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de81b5100 con 0x5634f7a51fd0 2024-03-21T10:47:59.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.004+0000 7f8df31ba740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de81b31c0 con 0x5634f7a5b6c0 2024-03-21T10:47:59.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.005+0000 7f8deda00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f7a51fd0 0x5634f7a5ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:59.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.005+0000 7f8deda00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f7a51fd0 0x5634f7a5ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58876/0 (socket says 172.21.15.31:58876) 2024-03-21T10:47:59.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.005+0000 7f8deda00640 1 -- 172.21.15.31:0/1305712144 learned_addr learned my addr 172.21.15.31:0/1305712144 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:59.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.005+0000 7f8dee400640 1 --2- 172.21.15.31:0/1305712144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f7a5b6c0 0x5634f7a66f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:59.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.005+0000 7f8ded000640 1 --2- 172.21.15.31:0/1305712144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634f7a528f0 0x5634f7a66a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:59.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.005+0000 7f8deda00640 1 -- 172.21.15.31:0/1305712144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f7a5b6c0 msgr2=0x5634f7a66f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:59.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.005+0000 7f8deda00640 1 --2- 172.21.15.31:0/1305712144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f7a5b6c0 0x5634f7a66f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:59.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.005+0000 7f8deda00640 1 -- 172.21.15.31:0/1305712144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634f7a528f0 msgr2=0x5634f7a66a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:59.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.005+0000 7f8deda00640 1 --2- 172.21.15.31:0/1305712144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634f7a528f0 0x5634f7a66a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:59.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.005+0000 7f8deda00640 1 -- 172.21.15.31:0/1305712144 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5634f7a52720 con 0x5634f7a51fd0 2024-03-21T10:47:59.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.005+0000 7f8deda00640 1 --2- 172.21.15.31:0/1305712144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f7a51fd0 0x5634f7a5ae60 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc0095f0 tx=0x7f8ddc0097b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=808e98fda1844130 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:59.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.006+0000 7f8dda000640 1 -- 172.21.15.31:0/1305712144 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc063070 con 0x5634f7a51fd0 2024-03-21T10:47:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.006+0000 7f8dda000640 1 -- 172.21.15.31:0/1305712144 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ddc02c510 con 0x5634f7a51fd0 2024-03-21T10:47:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.006+0000 7f8dda000640 1 -- 172.21.15.31:0/1305712144 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc04f070 con 0x5634f7a51fd0 2024-03-21T10:47:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.006+0000 7f8df31ba740 1 -- 172.21.15.31:0/1305712144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f7a51fd0 msgr2=0x5634f7a5ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.006+0000 7f8df31ba740 1 --2- 172.21.15.31:0/1305712144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f7a51fd0 0x5634f7a5ae60 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc0095f0 tx=0x7f8ddc0097b0 comp rx=0 tx=0).stop 2024-03-21T10:47:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.006+0000 7f8df31ba740 1 -- 172.21.15.31:0/1305712144 shutdown_connections 2024-03-21T10:47:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.006+0000 7f8df31ba740 1 --2- 172.21.15.31:0/1305712144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f7a5b6c0 0x5634f7a66f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.006+0000 7f8df31ba740 1 --2- 172.21.15.31:0/1305712144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634f7a528f0 0x5634f7a66a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.006+0000 7f8df31ba740 1 --2- 172.21.15.31:0/1305712144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f7a51fd0 0x5634f7a5ae60 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.006+0000 7f8df31ba740 1 -- 172.21.15.31:0/1305712144 >> 172.21.15.31:0/1305712144 conn(0x5634f7a4c8f0 msgr2=0x5634f7a4b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.006+0000 7f8df31ba740 1 -- 172.21.15.31:0/1305712144 shutdown_connections 2024-03-21T10:47:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.006+0000 7f8df31ba740 1 -- 172.21.15.31:0/1305712144 wait complete. 2024-03-21T10:47:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.007+0000 7f8df31ba740 1 -- start start 2024-03-21T10:47:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.007+0000 7f8df31ba740 1 Processor -- start 2024-03-21T10:47:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.007+0000 7f8df31ba740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f7a51fd0 0x5634f7a5f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.007+0000 7f8df31ba740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f7a528f0 0x5634f7a5f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.007+0000 7f8df31ba740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634f7a5b6c0 0x5634f7ac0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.007+0000 7f8df31ba740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de81b37c0 con 0x5634f7a5b6c0 2024-03-21T10:47:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.007+0000 7f8df31ba740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de81b4aa0 con 0x5634f7a51fd0 2024-03-21T10:47:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.007+0000 7f8df31ba740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de81b3de0 con 0x5634f7a528f0 2024-03-21T10:47:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.007+0000 7f8df31ba740 10 client.0 _ll_register_callbacks cb 0x7f8dd4005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:47:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.008+0000 7f8deda00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f7a51fd0 0x5634f7a5f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.008+0000 7f8ded000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f7a528f0 0x5634f7a5f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.008+0000 7f8ded000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f7a528f0 0x5634f7a5f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53208/0 (socket says 172.21.15.31:53208) 2024-03-21T10:47:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.008+0000 7f8ded000640 1 -- 172.21.15.31:0/2611552233 learned_addr learned my addr 172.21.15.31:0/2611552233 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:47:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.008+0000 7f8deda00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f7a51fd0 0x5634f7a5f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58882/0 (socket says 172.21.15.31:58882) 2024-03-21T10:47:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.008+0000 7f8dee400640 1 --2- 172.21.15.31:0/2611552233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634f7a5b6c0 0x5634f7ac0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.008+0000 7f8ded000640 1 -- 172.21.15.31:0/2611552233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f7a51fd0 msgr2=0x5634f7a5f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.008+0000 7f8ded000640 1 --2- 172.21.15.31:0/2611552233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f7a51fd0 0x5634f7a5f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.008+0000 7f8ded000640 1 -- 172.21.15.31:0/2611552233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634f7a5b6c0 msgr2=0x5634f7ac0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:47:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.008+0000 7f8ded000640 1 --2- 172.21.15.31:0/2611552233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634f7a5b6c0 0x5634f7ac0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.008+0000 7f8ded000640 1 -- 172.21.15.31:0/2611552233 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5634f7ac0650 con 0x5634f7a528f0 2024-03-21T10:47:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.008+0000 7f8deda00640 1 --2- 172.21.15.31:0/2611552233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f7a51fd0 0x5634f7a5f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:47:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.008+0000 7f8dee400640 1 --2- 172.21.15.31:0/2611552233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634f7a5b6c0 0x5634f7ac0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:47:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.008+0000 7f8ded000640 1 --2- 172.21.15.31:0/2611552233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f7a528f0 0x5634f7a5f7a0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f8dd409bd10 tx=0x7f8dd40d21e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c06858ad03b165b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:47:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.008+0000 7f8dcf400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:47:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.009+0000 7f8dcf400640 1 -- 172.21.15.31:0/2611552233 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd40a4790 con 0x5634f7a528f0 2024-03-21T10:47:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.009+0000 7f8df31ba740 1 -- 172.21.15.31:0/2611552233 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ddc02c510 con 0x5634f7a528f0 2024-03-21T10:47:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.009+0000 7f8dcf400640 1 -- 172.21.15.31:0/2611552233 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f8dd40983f0 con 0x5634f7a528f0 2024-03-21T10:47:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.009+0000 7f8dcf400640 1 -- 172.21.15.31:0/2611552233 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8dd40cd360 con 0x5634f7a528f0 2024-03-21T10:47:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.009+0000 7f8df31ba740 10 client.27871 fetch_fsmap learned FSMap version 28 2024-03-21T10:47:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.009+0000 7f8df31ba740 1 -- 172.21.15.31:0/2611552233 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5634f7ac7a70 con 0x5634f7a528f0 2024-03-21T10:47:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.009+0000 7f8dcf400640 1 -- 172.21.15.31:0/2611552233 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ddc02c510 con 0x5634f7a528f0 2024-03-21T10:47:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.009+0000 7f8dcf400640 1 -- 172.21.15.31:0/2611552233 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f8dd40baaf0 con 0x5634f7a528f0 2024-03-21T10:47:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.009+0000 7f8df31ba740 10 client.27871 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:47:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.009+0000 7f8df31ba740 10 client.27871 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:47:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.009+0000 7f8df31ba740 20 client.27871 populate_metadata read hostname 'smithi031' 2024-03-21T10:47:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.009+0000 7f8df31ba740 1 --2- 172.21.15.31:0/2611552233 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5634f7ac81b0 0x5634f7aca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:47:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.009+0000 7f8df31ba740 4 client.27871 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:47:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.009+0000 7f8df31ba740 1 -- 172.21.15.31:0/2611552233 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5634f79be0c0 con 0x5634f7ac81b0 2024-03-21T10:47:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.010+0000 7f8deda00640 1 --2- 172.21.15.31:0/2611552233 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5634f7ac81b0 0x5634f7aca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:47:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.010+0000 7f8deda00640 1 --2- 172.21.15.31:0/2611552233 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5634f7ac81b0 0x5634f7aca5d0 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bde76aa0507fb70 server_cookie=5ad84e0ef279a11e in_seq=0 out_seq=0 2024-03-21T10:47:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.010+0000 7f8dcf400640 10 client.27871 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.014+0000 7f8dcf400640 1 -- 172.21.15.31:0/2611552233 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5634f79be0c0 con 0x5634f7ac81b0 2024-03-21T10:47:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.014+0000 7f8dcf400640 10 client.27871 handle_command_reply: tid=0 2024-03-21T10:47:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.014+0000 7f8df31ba740 1 client.27871 shutdown 2024-03-21T10:47:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.014+0000 7f8df31ba740 1 -- 172.21.15.31:0/2611552233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f7a528f0 msgr2=0x5634f7a5f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.014+0000 7f8df31ba740 1 --2- 172.21.15.31:0/2611552233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f7a528f0 0x5634f7a5f7a0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f8dd409bd10 tx=0x7f8dd40d21e0 comp rx=0 tx=0).stop 2024-03-21T10:47:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.014+0000 7f8df31ba740 1 -- 172.21.15.31:0/2611552233 shutdown_connections 2024-03-21T10:47:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.014+0000 7f8df31ba740 1 --2- 172.21.15.31:0/2611552233 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5634f7ac81b0 0x5634f7aca5d0 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.014+0000 7f8df31ba740 1 --2- 172.21.15.31:0/2611552233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634f7a5b6c0 0x5634f7ac0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.014+0000 7f8df31ba740 1 --2- 172.21.15.31:0/2611552233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f7a528f0 0x5634f7a5f7a0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.014+0000 7f8df31ba740 1 --2- 172.21.15.31:0/2611552233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f7a51fd0 0x5634f7a5f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.014+0000 7f8df31ba740 1 -- 172.21.15.31:0/2611552233 >> 172.21.15.31:0/2611552233 conn(0x5634f7a4c8f0 msgr2=0x5634f7a586d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.014+0000 7f8dcf400640 0 client.27871 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:47:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.014+0000 7f8df31ba740 1 -- 172.21.15.31:0/2611552233 shutdown_connections 2024-03-21T10:47:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.014+0000 7f8df31ba740 1 -- 172.21.15.31:0/2611552233 wait complete. 2024-03-21T10:47:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.015+0000 7f8df31ba740 20 client.27871 trim_cache size 0 max 16384 2024-03-21T10:47:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:47:59.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:47:59.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:47:59.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:47:59.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.015+0000 7f8df0400640 1 -- 172.21.15.31:0/3835316886 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f8dd00a3080 msgr2=0x7f8dd00a5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.015+0000 7f8df0400640 1 --2- 172.21.15.31:0/3835316886 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f8dd00a3080 0x7f8dd00a5540 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc0057c0 tx=0x7f8ddc0056d0 comp rx=0 tx=0).stop 2024-03-21T10:47:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.015+0000 7f8df0400640 1 -- 172.21.15.31:0/3835316886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8de810cfc0 msgr2=0x7f8de819ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:47:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.015+0000 7f8df0400640 1 --2- 172.21.15.31:0/3835316886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8de810cfc0 0x7f8de819ead0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f8de806ac90 tx=0x7f8dd400ee50 comp rx=0 tx=0).stop 2024-03-21T10:47:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.015+0000 7f8df0400640 1 -- 172.21.15.31:0/3835316886 shutdown_connections 2024-03-21T10:47:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.015+0000 7f8df0400640 1 --2- 172.21.15.31:0/3835316886 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f8dd00a3080 0x7f8dd00a5540 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.015+0000 7f8df0400640 1 --2- 172.21.15.31:0/3835316886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8de81a2e50 0x7f8de81a3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.015+0000 7f8df0400640 1 --2- 172.21.15.31:0/3835316886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8de810cfc0 0x7f8de819ead0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.015+0000 7f8df0400640 1 --2- 172.21.15.31:0/3835316886 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8de8107f40 0x7f8de819e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:47:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.015+0000 7f8df0400640 1 -- 172.21.15.31:0/3835316886 >> 172.21.15.31:0/3835316886 conn(0x7f8de80774e0 msgr2=0x7f8de8107350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:47:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.016+0000 7f8df0400640 1 -- 172.21.15.31:0/3835316886 shutdown_connections 2024-03-21T10:47:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:47:59.016+0000 7f8df0400640 1 -- 172.21.15.31:0/3835316886 wait complete. 2024-03-21T10:47:59.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:47:59 smithi040 ceph-mon[36154]: pgmap v291: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-21T10:47:59.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:47:59 smithi031 ceph-mon[31673]: pgmap v291: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-21T10:47:59.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:47:59 smithi070 ceph-mon[36330]: pgmap v291: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-21T10:48:01.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:01 smithi040 ceph-mon[36154]: pgmap v292: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 7.1 KiB/s wr, 8 op/s 2024-03-21T10:48:01.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:01 smithi031 ceph-mon[31673]: pgmap v292: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 7.1 KiB/s wr, 8 op/s 2024-03-21T10:48:01.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:01 smithi070 ceph-mon[36330]: pgmap v292: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 7.1 KiB/s wr, 8 op/s 2024-03-21T10:48:03.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:03 smithi040 ceph-mon[36154]: pgmap v293: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.7 KiB/s wr, 5 op/s 2024-03-21T10:48:03.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:03 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:48:03.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:03 smithi031 ceph-mon[31673]: pgmap v293: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.7 KiB/s wr, 5 op/s 2024-03-21T10:48:03.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:03 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:48:03.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:03 smithi070 ceph-mon[36330]: pgmap v293: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.7 KiB/s wr, 5 op/s 2024-03-21T10:48:03.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:03 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:48:04.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:04 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:48:04.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:04 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:48:04.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:04 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:48:04.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:04 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:48:04.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:04 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:48:04.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:04 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:48:04.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:04 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:48:04.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:04 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:48:04.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:04 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:48:04.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:04 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:48:04.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:04 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:48:04.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:04 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:48:04.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:04 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:48:04.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:04 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:48:04.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:04 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:48:04.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:04 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:48:04.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:04 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:48:04.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:04 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:48:05.308 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:48:05.308 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:48:05.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.531+0000 7fa1f6a00640 1 -- 172.21.15.31:0/2853149320 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e802cc70 con 0x7fa1f8109850 2024-03-21T10:48:05.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.531+0000 7fa1fec00640 1 -- 172.21.15.31:0/2853149320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1f8109850 msgr2=0x7fa1f8109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:05.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.531+0000 7fa1fec00640 1 --2- 172.21.15.31:0/2853149320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1f8109850 0x7fa1f8109c30 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa1e8009930 tx=0x7fa1e802bc20 comp rx=0 tx=0).stop 2024-03-21T10:48:05.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.531+0000 7fa1fec00640 1 -- 172.21.15.31:0/2853149320 shutdown_connections 2024-03-21T10:48:05.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.531+0000 7fa1fec00640 1 --2- 172.21.15.31:0/2853149320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1f8104200 0x7fa1f8110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.532+0000 7fa1fec00640 1 --2- 172.21.15.31:0/2853149320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1f8103840 0x7fa1f8103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.532+0000 7fa1fec00640 1 --2- 172.21.15.31:0/2853149320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1f8109850 0x7fa1f8109c30 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.532+0000 7fa1fec00640 1 -- 172.21.15.31:0/2853149320 >> 172.21.15.31:0/2853149320 conn(0x7fa1f80ff2f0 msgr2=0x7fa1f8101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:05.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.532+0000 7fa1fec00640 1 -- 172.21.15.31:0/2853149320 shutdown_connections 2024-03-21T10:48:05.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.532+0000 7fa1fec00640 1 -- 172.21.15.31:0/2853149320 wait complete. 2024-03-21T10:48:05.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.533+0000 7fa1fec00640 1 Processor -- start 2024-03-21T10:48:05.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.534+0000 7fa1fec00640 1 -- start start 2024-03-21T10:48:05.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.534+0000 7fa1fec00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1f8103840 0x7fa1f819a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:05.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.534+0000 7fa1fec00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1f8104200 0x7fa1f819a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:05.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.534+0000 7fa1fec00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1f8109850 0x7fa1f819ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:05.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.534+0000 7fa1fec00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f8113c20 con 0x7fa1f8103840 2024-03-21T10:48:05.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.534+0000 7fa1fec00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f8113aa0 con 0x7fa1f8104200 2024-03-21T10:48:05.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.534+0000 7fa1fec00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f8113da0 con 0x7fa1f8109850 2024-03-21T10:48:05.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.535+0000 7fa1f7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1f8103840 0x7fa1f819a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:05.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.535+0000 7fa1f7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1f8104200 0x7fa1f819a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:05.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.535+0000 7fa1f7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1f8104200 0x7fa1f819a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58900/0 (socket says 172.21.15.31:58900) 2024-03-21T10:48:05.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.535+0000 7fa1f7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1f8103840 0x7fa1f819a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57926/0 (socket says 172.21.15.31:57926) 2024-03-21T10:48:05.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.535+0000 7fa1f7400640 1 -- 172.21.15.31:0/3549976739 learned_addr learned my addr 172.21.15.31:0/3549976739 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:05.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.535+0000 7fa1fcc00640 1 --2- 172.21.15.31:0/3549976739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1f8109850 0x7fa1f819ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:05.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.535+0000 7fa1f7400640 1 -- 172.21.15.31:0/3549976739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1f8109850 msgr2=0x7fa1f819ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:05.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.535+0000 7fa1f7400640 1 --2- 172.21.15.31:0/3549976739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1f8109850 0x7fa1f819ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.535+0000 7fa1f7400640 1 -- 172.21.15.31:0/3549976739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1f8103840 msgr2=0x7fa1f819a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:05.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.535+0000 7fa1f7400640 1 --2- 172.21.15.31:0/3549976739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1f8103840 0x7fa1f819a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.535+0000 7fa1f7400640 1 -- 172.21.15.31:0/3549976739 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f819f130 con 0x7fa1f8104200 2024-03-21T10:48:05.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.535+0000 7fa1f7e00640 1 --2- 172.21.15.31:0/3549976739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1f8103840 0x7fa1f819a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:05.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.536+0000 7fa1f7400640 1 --2- 172.21.15.31:0/3549976739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1f8104200 0x7fa1f819a680 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fa1e400e9f0 tx=0x7fa1e400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:05.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.536+0000 7fa1f4c00640 1 -- 172.21.15.31:0/3549976739 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e400cd80 con 0x7fa1f8104200 2024-03-21T10:48:05.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.536+0000 7fa1fec00640 1 -- 172.21.15.31:0/3549976739 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f819f420 con 0x7fa1f8104200 2024-03-21T10:48:05.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.536+0000 7fa1f4c00640 1 -- 172.21.15.31:0/3549976739 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1e4010910 con 0x7fa1f8104200 2024-03-21T10:48:05.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.536+0000 7fa1f4c00640 1 -- 172.21.15.31:0/3549976739 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e40188f0 con 0x7fa1f8104200 2024-03-21T10:48:05.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.536+0000 7fa1fec00640 1 -- 172.21.15.31:0/3549976739 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f81a6cb0 con 0x7fa1f8104200 2024-03-21T10:48:05.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.540+0000 7fa1f4c00640 1 -- 172.21.15.31:0/3549976739 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fa1e4010ac0 con 0x7fa1f8104200 2024-03-21T10:48:05.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.540+0000 7fa201806740 1 Processor -- start 2024-03-21T10:48:05.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.540+0000 7fa201806740 1 -- start start 2024-03-21T10:48:05.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.541+0000 7fa201806740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8df0d2c00 0x55b8df0d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:05.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.541+0000 7fa201806740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8df0d1f20 0x55b8df0d23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:05.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.541+0000 7fa201806740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8df0db290 0x55b8df0dd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:05.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.541+0000 7fa201806740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8df036250 con 0x55b8df0db290 2024-03-21T10:48:05.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.541+0000 7fa201806740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8defdea30 con 0x55b8df0d2c00 2024-03-21T10:48:05.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.541+0000 7fa201806740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8df0322f0 con 0x55b8df0d1f20 2024-03-21T10:48:05.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.541+0000 7fa1fcc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8df0d2c00 0x55b8df0d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:05.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.541+0000 7fa1f7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8df0d1f20 0x55b8df0d23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:05.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.541+0000 7fa1fcc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8df0d2c00 0x55b8df0d1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58906/0 (socket says 172.21.15.31:58906) 2024-03-21T10:48:05.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.541+0000 7fa1fcc00640 1 -- 172.21.15.31:0/2604120515 learned_addr learned my addr 172.21.15.31:0/2604120515 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:05.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.541+0000 7fa1f7400640 1 --2- 172.21.15.31:0/2604120515 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8df0db290 0x55b8df0dd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:05.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.542+0000 7fa1fcc00640 1 -- 172.21.15.31:0/2604120515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8df0d1f20 msgr2=0x55b8df0d23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:05.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.542+0000 7fa1fcc00640 1 --2- 172.21.15.31:0/2604120515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8df0d1f20 0x55b8df0d23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.542+0000 7fa1fcc00640 1 -- 172.21.15.31:0/2604120515 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8df0db290 msgr2=0x55b8df0dd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:05.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.542+0000 7fa1f4c00640 1 --2- 172.21.15.31:0/3549976739 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa1dc0a3090 0x7fa1dc0a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:05.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.542+0000 7fa1fcc00640 1 --2- 172.21.15.31:0/2604120515 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8df0db290 0x55b8df0dd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.542+0000 7fa1fcc00640 1 -- 172.21.15.31:0/2604120515 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b8defc9690 con 0x55b8df0d2c00 2024-03-21T10:48:05.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.542+0000 7fa1f7e00640 1 --2- 172.21.15.31:0/2604120515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8df0d1f20 0x55b8df0d23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:05.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.542+0000 7fa1f4c00640 1 -- 172.21.15.31:0/3549976739 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fa1e40cffa0 con 0x7fa1f8104200 2024-03-21T10:48:05.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.542+0000 7fa1fcc00640 1 --2- 172.21.15.31:0/2604120515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8df0d2c00 0x55b8df0d1950 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa1f0009f50 tx=0x7fa1f00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb1a4f58ec393e05 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:05.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.542+0000 7fa1fcc00640 1 --2- 172.21.15.31:0/3549976739 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa1dc0a3090 0x7fa1dc0a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:05.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.542+0000 7fa1ed200640 1 -- 172.21.15.31:0/2604120515 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f001e070 con 0x55b8df0d2c00 2024-03-21T10:48:05.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.543+0000 7fa1ed200640 1 -- 172.21.15.31:0/2604120515 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1f0003d70 con 0x55b8df0d2c00 2024-03-21T10:48:05.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.543+0000 7fa1ed200640 1 -- 172.21.15.31:0/2604120515 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f000fca0 con 0x55b8df0d2c00 2024-03-21T10:48:05.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.543+0000 7fa201806740 1 -- 172.21.15.31:0/2604120515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8df0d2c00 msgr2=0x55b8df0d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:05.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.543+0000 7fa201806740 1 --2- 172.21.15.31:0/2604120515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8df0d2c00 0x55b8df0d1950 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa1f0009f50 tx=0x7fa1f00118d0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.543+0000 7fa1fcc00640 1 --2- 172.21.15.31:0/3549976739 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa1dc0a3090 0x7fa1dc0a5550 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa1f0010010 tx=0x7fa1f000ff20 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:05.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.543+0000 7fa201806740 1 -- 172.21.15.31:0/2604120515 shutdown_connections 2024-03-21T10:48:05.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.543+0000 7fa201806740 1 --2- 172.21.15.31:0/2604120515 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8df0db290 0x55b8df0dd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.543+0000 7fa201806740 1 --2- 172.21.15.31:0/2604120515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8df0d1f20 0x55b8df0d23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.543+0000 7fa201806740 1 --2- 172.21.15.31:0/2604120515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8df0d2c00 0x55b8df0d1950 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.543+0000 7fa201806740 1 -- 172.21.15.31:0/2604120515 >> 172.21.15.31:0/2604120515 conn(0x55b8df099200 msgr2=0x55b8df0cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:05.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.544+0000 7fa201806740 1 -- 172.21.15.31:0/2604120515 shutdown_connections 2024-03-21T10:48:05.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.544+0000 7fa201806740 1 -- 172.21.15.31:0/2604120515 wait complete. 2024-03-21T10:48:05.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.544+0000 7fa201806740 1 -- start start 2024-03-21T10:48:05.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.544+0000 7fa201806740 1 Processor -- start 2024-03-21T10:48:05.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.545+0000 7fa201806740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8df0d1f20 0x55b8df0da8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:05.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.545+0000 7fa201806740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8df0d2c00 0x55b8df0d7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:05.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.545+0000 7fa201806740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8df0db290 0x55b8df0d81a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:05.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.545+0000 7fa201806740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8df0de100 con 0x55b8df0d2c00 2024-03-21T10:48:05.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.545+0000 7fa201806740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8deff1e80 con 0x55b8df0db290 2024-03-21T10:48:05.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.545+0000 7fa201806740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8df0de280 con 0x55b8df0d1f20 2024-03-21T10:48:05.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.545+0000 7fa201806740 10 client.0 _ll_register_callbacks cb 0x55b8df004130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:05.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.545+0000 7fa1f7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8df0d1f20 0x55b8df0da8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:05.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.545+0000 7fa1f7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8df0d1f20 0x55b8df0da8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53248/0 (socket says 172.21.15.31:53248) 2024-03-21T10:48:05.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.545+0000 7fa1f7e00640 1 -- 172.21.15.31:0/558475070 learned_addr learned my addr 172.21.15.31:0/558475070 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:05.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.545+0000 7fa1fcc00640 1 --2- 172.21.15.31:0/558475070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8df0db290 0x55b8df0d81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:05.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.545+0000 7fa1f7400640 1 --2- 172.21.15.31:0/558475070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8df0d2c00 0x55b8df0d7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:05.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.545+0000 7fa1f7e00640 1 -- 172.21.15.31:0/558475070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8df0db290 msgr2=0x55b8df0d81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:05.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.545+0000 7fa1f7e00640 1 --2- 172.21.15.31:0/558475070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8df0db290 0x55b8df0d81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.545+0000 7fa1f7e00640 1 -- 172.21.15.31:0/558475070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8df0d2c00 msgr2=0x55b8df0d7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:05.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.545+0000 7fa1f7e00640 1 --2- 172.21.15.31:0/558475070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8df0d2c00 0x55b8df0d7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.545+0000 7fa1f7e00640 1 -- 172.21.15.31:0/558475070 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b8dee147d0 con 0x55b8df0d1f20 2024-03-21T10:48:05.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.546+0000 7fa1f7e00640 1 --2- 172.21.15.31:0/558475070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8df0d1f20 0x55b8df0da8d0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa1e8006010 tx=0x7fa1e802c870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52dd8c2a52c59f62 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:05.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.546+0000 7fa1e1600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:48:05.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.546+0000 7fa1e1600640 1 -- 172.21.15.31:0/558475070 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e802cc60 con 0x55b8df0d1f20 2024-03-21T10:48:05.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.546+0000 7fa1e1600640 1 -- 172.21.15.31:0/558475070 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fa1e8004040 con 0x55b8df0d1f20 2024-03-21T10:48:05.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.546+0000 7fa201806740 1 -- 172.21.15.31:0/558475070 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa1f0003d70 con 0x55b8df0d1f20 2024-03-21T10:48:05.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.547+0000 7fa1e1600640 1 -- 172.21.15.31:0/558475070 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1e8004240 con 0x55b8df0d1f20 2024-03-21T10:48:05.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.547+0000 7fa201806740 10 client.27883 fetch_fsmap learned FSMap version 28 2024-03-21T10:48:05.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.547+0000 7fa201806740 1 -- 172.21.15.31:0/558475070 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55b8ded4a3f0 con 0x55b8df0d1f20 2024-03-21T10:48:05.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.547+0000 7fa1e1600640 1 -- 172.21.15.31:0/558475070 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1f0003d70 con 0x55b8df0d1f20 2024-03-21T10:48:05.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.547+0000 7fa1e1600640 1 -- 172.21.15.31:0/558475070 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fa1e8003850 con 0x55b8df0d1f20 2024-03-21T10:48:05.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.547+0000 7fa201806740 10 client.27883 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:48:05.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.547+0000 7fa201806740 10 client.27883 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:48:05.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.547+0000 7fa201806740 20 client.27883 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:05.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.547+0000 7fa201806740 1 --2- 172.21.15.31:0/558475070 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55b8df0eb0f0 0x55b8df0ed590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:05.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.547+0000 7fa201806740 4 client.27883 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:48:05.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.547+0000 7fa201806740 1 -- 172.21.15.31:0/558475070 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b8df043790 con 0x55b8df0eb0f0 2024-03-21T10:48:05.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.548+0000 7fa1f7400640 1 --2- 172.21.15.31:0/558475070 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55b8df0eb0f0 0x55b8df0ed590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:05.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.548+0000 7fa1f7400640 1 --2- 172.21.15.31:0/558475070 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55b8df0eb0f0 0x55b8df0ed590 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9dcec0bbea6657c2 server_cookie=88992edddf457dd5 in_seq=0 out_seq=0 2024-03-21T10:48:05.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.548+0000 7fa1e1600640 10 client.27883 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:05.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.549+0000 7fa1e1600640 1 -- 172.21.15.31:0/558475070 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b8df043790 con 0x55b8df0eb0f0 2024-03-21T10:48:05.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.549+0000 7fa1e1600640 10 client.27883 handle_command_reply: tid=0 2024-03-21T10:48:05.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.549+0000 7fa201806740 1 client.27883 shutdown 2024-03-21T10:48:05.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.549+0000 7fa201806740 1 -- 172.21.15.31:0/558475070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8df0d1f20 msgr2=0x55b8df0da8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:05.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.549+0000 7fa201806740 1 --2- 172.21.15.31:0/558475070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8df0d1f20 0x55b8df0da8d0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa1e8006010 tx=0x7fa1e802c870 comp rx=0 tx=0).stop 2024-03-21T10:48:05.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.549+0000 7fa201806740 1 -- 172.21.15.31:0/558475070 shutdown_connections 2024-03-21T10:48:05.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.549+0000 7fa201806740 1 --2- 172.21.15.31:0/558475070 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55b8df0eb0f0 0x55b8df0ed590 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.549+0000 7fa201806740 1 --2- 172.21.15.31:0/558475070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8df0db290 0x55b8df0d81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.549+0000 7fa201806740 1 --2- 172.21.15.31:0/558475070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8df0d2c00 0x55b8df0d7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.549+0000 7fa201806740 1 --2- 172.21.15.31:0/558475070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8df0d1f20 0x55b8df0da8d0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.549+0000 7fa201806740 1 -- 172.21.15.31:0/558475070 >> 172.21.15.31:0/558475070 conn(0x55b8df099200 msgr2=0x55b8df0ce6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:05.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.549+0000 7fa1e1600640 0 client.27883 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:05.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.549+0000 7fa201806740 1 -- 172.21.15.31:0/558475070 shutdown_connections 2024-03-21T10:48:05.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.549+0000 7fa201806740 1 -- 172.21.15.31:0/558475070 wait complete. 2024-03-21T10:48:05.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.550+0000 7fa201806740 20 client.27883 trim_cache size 0 max 16384 2024-03-21T10:48:05.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.576+0000 7fa201806740 1 Processor -- start 2024-03-21T10:48:05.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.576+0000 7fa201806740 1 -- start start 2024-03-21T10:48:05.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.577+0000 7fa201806740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8df0d1f20 0x55b8df0dc680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:05.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.577+0000 7fa201806740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8df0d2c00 0x55b8df0dcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:05.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.577+0000 7fa201806740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8df0db290 0x55b8df0d6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:05.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.577+0000 7fa201806740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f81b3210 con 0x55b8df0db290 2024-03-21T10:48:05.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.577+0000 7fa201806740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f81b0d10 con 0x55b8df0d2c00 2024-03-21T10:48:05.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.577+0000 7fa201806740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f81aedd0 con 0x55b8df0d1f20 2024-03-21T10:48:05.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.577+0000 7fa1f7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8df0d1f20 0x55b8df0dc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:05.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.577+0000 7fa1f7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8df0d1f20 0x55b8df0dc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53258/0 (socket says 172.21.15.31:53258) 2024-03-21T10:48:05.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.577+0000 7fa1fcc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8df0db290 0x55b8df0d6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:05.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.577+0000 7fa1f7e00640 1 -- 172.21.15.31:0/3755361578 learned_addr learned my addr 172.21.15.31:0/3755361578 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:05.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.577+0000 7fa1f7400640 1 --2- 172.21.15.31:0/3755361578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8df0d2c00 0x55b8df0dcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:05.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.578+0000 7fa1f7e00640 1 -- 172.21.15.31:0/3755361578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8df0d2c00 msgr2=0x55b8df0dcbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:05.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.578+0000 7fa1f7e00640 1 --2- 172.21.15.31:0/3755361578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8df0d2c00 0x55b8df0dcbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.578+0000 7fa1f7e00640 1 -- 172.21.15.31:0/3755361578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8df0db290 msgr2=0x55b8df0d6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:05.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.578+0000 7fa1f7e00640 1 --2- 172.21.15.31:0/3755361578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8df0db290 0x55b8df0d6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.578+0000 7fa1f7e00640 1 -- 172.21.15.31:0/3755361578 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b8ded0a0f0 con 0x55b8df0d1f20 2024-03-21T10:48:05.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.578+0000 7fa1f7400640 1 --2- 172.21.15.31:0/3755361578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8df0d2c00 0x55b8df0dcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:48:05.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.578+0000 7fa1f7e00640 1 --2- 172.21.15.31:0/3755361578 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8df0d1f20 0x55b8df0dc680 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fa1e8005e90 tx=0x7fa1e802c5c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90ce45a56946326b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:05.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.579+0000 7fa1e3e00640 1 -- 172.21.15.31:0/3755361578 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e8009520 con 0x55b8df0d1f20 2024-03-21T10:48:05.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.579+0000 7fa1e3e00640 1 -- 172.21.15.31:0/3755361578 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1e805a070 con 0x55b8df0d1f20 2024-03-21T10:48:05.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.579+0000 7fa1e3e00640 1 -- 172.21.15.31:0/3755361578 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e8004ef0 con 0x55b8df0d1f20 2024-03-21T10:48:05.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.579+0000 7fa201806740 1 -- 172.21.15.31:0/3755361578 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8df0d1f20 msgr2=0x55b8df0dc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:05.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.580+0000 7fa201806740 1 --2- 172.21.15.31:0/3755361578 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8df0d1f20 0x55b8df0dc680 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fa1e8005e90 tx=0x7fa1e802c5c0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.580+0000 7fa201806740 1 -- 172.21.15.31:0/3755361578 shutdown_connections 2024-03-21T10:48:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.580+0000 7fa201806740 1 --2- 172.21.15.31:0/3755361578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8df0db290 0x55b8df0d6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.580+0000 7fa201806740 1 --2- 172.21.15.31:0/3755361578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8df0d2c00 0x55b8df0dcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.580+0000 7fa201806740 1 --2- 172.21.15.31:0/3755361578 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8df0d1f20 0x55b8df0dc680 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.580+0000 7fa201806740 1 -- 172.21.15.31:0/3755361578 >> 172.21.15.31:0/3755361578 conn(0x55b8df099200 msgr2=0x55b8df0cc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.580+0000 7fa201806740 1 -- 172.21.15.31:0/3755361578 shutdown_connections 2024-03-21T10:48:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.580+0000 7fa201806740 1 -- 172.21.15.31:0/3755361578 wait complete. 2024-03-21T10:48:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.580+0000 7fa201806740 1 -- start start 2024-03-21T10:48:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.580+0000 7fa201806740 1 Processor -- start 2024-03-21T10:48:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.581+0000 7fa201806740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8df0d1f20 0x55b8df0d4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.581+0000 7fa201806740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8df0d2c00 0x55b8df0d4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.581+0000 7fa201806740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8df0db290 0x55b8df0d4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.581+0000 7fa201806740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f81af3d0 con 0x55b8df0d2c00 2024-03-21T10:48:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.581+0000 7fa201806740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f81b06b0 con 0x55b8df0d1f20 2024-03-21T10:48:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.581+0000 7fa201806740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f81af9f0 con 0x55b8df0db290 2024-03-21T10:48:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.581+0000 7fa201806740 10 client.0 _ll_register_callbacks cb 0x7fa1e8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.581+0000 7fa1f7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8df0d1f20 0x55b8df0d4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.581+0000 7fa1fcc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8df0db290 0x55b8df0d4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.581+0000 7fa1f7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8df0d1f20 0x55b8df0d4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58930/0 (socket says 172.21.15.31:58930) 2024-03-21T10:48:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.581+0000 7fa1fcc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8df0db290 0x55b8df0d4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53266/0 (socket says 172.21.15.31:53266) 2024-03-21T10:48:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.581+0000 7fa1f7e00640 1 -- 172.21.15.31:0/1652399712 learned_addr learned my addr 172.21.15.31:0/1652399712 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.582+0000 7fa1f7400640 1 --2- 172.21.15.31:0/1652399712 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8df0d2c00 0x55b8df0d4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.582+0000 7fa1f7e00640 1 -- 172.21.15.31:0/1652399712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8df0db290 msgr2=0x55b8df0d4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.582+0000 7fa1f7e00640 1 --2- 172.21.15.31:0/1652399712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8df0db290 0x55b8df0d4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.582+0000 7fa1f7e00640 1 -- 172.21.15.31:0/1652399712 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8df0d2c00 msgr2=0x55b8df0d4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.582+0000 7fa1f7e00640 1 --2- 172.21.15.31:0/1652399712 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8df0d2c00 0x55b8df0d4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.582+0000 7fa1f7e00640 1 -- 172.21.15.31:0/1652399712 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b8df145bf0 con 0x55b8df0d1f20 2024-03-21T10:48:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.582+0000 7fa1f7e00640 1 --2- 172.21.15.31:0/1652399712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8df0d1f20 0x55b8df0d4550 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fa1e8003d90 tx=0x7fa1e8000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6233671cb0a1941 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.582+0000 7fa1e1600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:48:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.582+0000 7fa1e1600640 1 -- 172.21.15.31:0/1652399712 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e8062020 con 0x55b8df0d1f20 2024-03-21T10:48:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.582+0000 7fa1e1600640 1 -- 172.21.15.31:0/1652399712 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fa1e8003e60 con 0x55b8df0d1f20 2024-03-21T10:48:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.583+0000 7fa201806740 1 -- 172.21.15.31:0/1652399712 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa1e805a070 con 0x55b8df0d1f20 2024-03-21T10:48:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.583+0000 7fa1e1600640 1 -- 172.21.15.31:0/1652399712 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1e8004080 con 0x55b8df0d1f20 2024-03-21T10:48:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.583+0000 7fa201806740 10 client.27968 fetch_fsmap learned FSMap version 28 2024-03-21T10:48:05.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.583+0000 7fa201806740 1 -- 172.21.15.31:0/1652399712 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55b8df146ee0 con 0x55b8df0d1f20 2024-03-21T10:48:05.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.583+0000 7fa1e1600640 1 -- 172.21.15.31:0/1652399712 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1e805a070 con 0x55b8df0d1f20 2024-03-21T10:48:05.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.583+0000 7fa1e1600640 1 -- 172.21.15.31:0/1652399712 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fa1e8034500 con 0x55b8df0d1f20 2024-03-21T10:48:05.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.583+0000 7fa201806740 10 client.27968 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:48:05.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.583+0000 7fa201806740 10 client.27968 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:48:05.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.583+0000 7fa201806740 20 client.27968 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:05.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.583+0000 7fa201806740 1 --2- 172.21.15.31:0/1652399712 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55b8df1473c0 0x55b8df1497e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:05.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.583+0000 7fa201806740 4 client.27968 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:48:05.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.583+0000 7fa201806740 1 -- 172.21.15.31:0/1652399712 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b8df03dae0 con 0x55b8df1473c0 2024-03-21T10:48:05.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.584+0000 7fa1f7400640 1 --2- 172.21.15.31:0/1652399712 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55b8df1473c0 0x55b8df1497e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:05.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.584+0000 7fa1f7400640 1 --2- 172.21.15.31:0/1652399712 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55b8df1473c0 0x55b8df1497e0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7de40f9117e5bdf0 server_cookie=efc1a5be65ce2ca2 in_seq=0 out_seq=0 2024-03-21T10:48:05.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.584+0000 7fa1e1600640 10 client.27968 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:05.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.584+0000 7fa1e1600640 1 -- 172.21.15.31:0/1652399712 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55b8df03dae0 con 0x55b8df1473c0 2024-03-21T10:48:05.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.584+0000 7fa1e1600640 10 client.27968 handle_command_reply: tid=0 2024-03-21T10:48:05.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.584+0000 7fa201806740 1 client.27968 shutdown 2024-03-21T10:48:05.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.585+0000 7fa201806740 1 -- 172.21.15.31:0/1652399712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8df0d1f20 msgr2=0x55b8df0d4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:05.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.585+0000 7fa201806740 1 --2- 172.21.15.31:0/1652399712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8df0d1f20 0x55b8df0d4550 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fa1e8003d90 tx=0x7fa1e8000bc0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.585+0000 7fa201806740 1 -- 172.21.15.31:0/1652399712 shutdown_connections 2024-03-21T10:48:05.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.585+0000 7fa201806740 1 --2- 172.21.15.31:0/1652399712 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55b8df1473c0 0x55b8df1497e0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.585+0000 7fa201806740 1 --2- 172.21.15.31:0/1652399712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8df0db290 0x55b8df0d4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.585+0000 7fa201806740 1 --2- 172.21.15.31:0/1652399712 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8df0d2c00 0x55b8df0d4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.585+0000 7fa201806740 1 --2- 172.21.15.31:0/1652399712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8df0d1f20 0x55b8df0d4550 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.585+0000 7fa201806740 1 -- 172.21.15.31:0/1652399712 >> 172.21.15.31:0/1652399712 conn(0x55b8df099200 msgr2=0x55b8df0d6130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:05.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.585+0000 7fa1e1600640 0 client.27968 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:05.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.585+0000 7fa201806740 1 -- 172.21.15.31:0/1652399712 shutdown_connections 2024-03-21T10:48:05.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.585+0000 7fa201806740 1 -- 172.21.15.31:0/1652399712 wait complete. 2024-03-21T10:48:05.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.585+0000 7fa201806740 20 client.27968 trim_cache size 0 max 16384 2024-03-21T10:48:05.586 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:48:05.586 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-03-21T10:48:05.586 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-03-21T10:48:05.586 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:48:05.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.586+0000 7fa1fec00640 1 -- 172.21.15.31:0/3549976739 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa1dc0a3090 msgr2=0x7fa1dc0a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:05.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.586+0000 7fa1fec00640 1 --2- 172.21.15.31:0/3549976739 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa1dc0a3090 0x7fa1dc0a5550 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa1f0010010 tx=0x7fa1f000ff20 comp rx=0 tx=0).stop 2024-03-21T10:48:05.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.586+0000 7fa1fec00640 1 -- 172.21.15.31:0/3549976739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1f8104200 msgr2=0x7fa1f819a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:05.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.586+0000 7fa1fec00640 1 --2- 172.21.15.31:0/3549976739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1f8104200 0x7fa1f819a680 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fa1e400e9f0 tx=0x7fa1e400eeb0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.586+0000 7fa1fec00640 1 -- 172.21.15.31:0/3549976739 shutdown_connections 2024-03-21T10:48:05.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.586+0000 7fa1fec00640 1 --2- 172.21.15.31:0/3549976739 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa1dc0a3090 0x7fa1dc0a5550 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.586+0000 7fa1fec00640 1 --2- 172.21.15.31:0/3549976739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1f8109850 0x7fa1f819ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.586+0000 7fa1fec00640 1 --2- 172.21.15.31:0/3549976739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1f8104200 0x7fa1f819a680 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.586+0000 7fa1fec00640 1 --2- 172.21.15.31:0/3549976739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1f8103840 0x7fa1f819a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.586+0000 7fa1fec00640 1 -- 172.21.15.31:0/3549976739 >> 172.21.15.31:0/3549976739 conn(0x7fa1f80ff2f0 msgr2=0x7fa1f80ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:05.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.586+0000 7fa1fec00640 1 -- 172.21.15.31:0/3549976739 shutdown_connections 2024-03-21T10:48:05.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.586+0000 7fa1fec00640 1 -- 172.21.15.31:0/3549976739 wait complete. 2024-03-21T10:48:05.603 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-21T10:48:05.603 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:48:05.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:05 smithi040 ceph-mon[36154]: pgmap v294: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-21T10:48:05.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:05 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:48:05.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:05 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:48:05.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:05 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:48:05.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:05 smithi031 ceph-mon[31673]: pgmap v294: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-21T10:48:05.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:05 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:48:05.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:05 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:48:05.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:05 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:48:05.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:05 smithi070 ceph-mon[36330]: pgmap v294: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 27 KiB/s wr, 20 op/s 2024-03-21T10:48:05.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:05 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:48:05.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:05 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:48:05.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:05 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:48:05.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.830+0000 7fafd7e00640 1 -- 172.21.15.31:0/749453268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafd8073a40 msgr2=0x7fafd8073e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:05.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.830+0000 7fafd7e00640 1 --2- 172.21.15.31:0/749453268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafd8073a40 0x7fafd8073e20 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009ca0 tx=0x7fafc802c190 comp rx=0 tx=0).stop 2024-03-21T10:48:05.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.830+0000 7fafd7e00640 1 -- 172.21.15.31:0/749453268 shutdown_connections 2024-03-21T10:48:05.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.830+0000 7fafd7e00640 1 --2- 172.21.15.31:0/749453268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fafd8112530 0x7fafd8114920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.830+0000 7fafd7e00640 1 --2- 172.21.15.31:0/749453268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafd8074360 0x7fafd8111ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.830+0000 7fafd7e00640 1 --2- 172.21.15.31:0/749453268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafd8073a40 0x7fafd8073e20 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.830+0000 7fafd7e00640 1 -- 172.21.15.31:0/749453268 >> 172.21.15.31:0/749453268 conn(0x7fafd80fd070 msgr2=0x7fafd80ff490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:05.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.830+0000 7fafd7e00640 1 -- 172.21.15.31:0/749453268 shutdown_connections 2024-03-21T10:48:05.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.830+0000 7fafd7e00640 1 -- 172.21.15.31:0/749453268 wait complete. 2024-03-21T10:48:05.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.832+0000 7fafd7e00640 1 Processor -- start 2024-03-21T10:48:05.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.832+0000 7fafd7e00640 1 -- start start 2024-03-21T10:48:05.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.833+0000 7fafd7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fafd8073a40 0x7fafd819e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:05.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.833+0000 7fafd7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafd8074360 0x7fafd819ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:05.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.833+0000 7fafd7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafd8112530 0x7fafd81a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:05.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.833+0000 7fafd7e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd8117f30 con 0x7fafd8112530 2024-03-21T10:48:05.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.833+0000 7fafd7e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd8117db0 con 0x7fafd8074360 2024-03-21T10:48:05.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.833+0000 7fafd7e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd81180b0 con 0x7fafd8073a40 2024-03-21T10:48:05.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.833+0000 7fafd6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fafd8073a40 0x7fafd819e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:05.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.833+0000 7fafd6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fafd8073a40 0x7fafd819e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53274/0 (socket says 172.21.15.31:53274) 2024-03-21T10:48:05.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.833+0000 7fafd6a00640 1 -- 172.21.15.31:0/1033384950 learned_addr learned my addr 172.21.15.31:0/1033384950 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:05.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.833+0000 7fafd6000640 1 --2- 172.21.15.31:0/1033384950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafd8074360 0x7fafd819ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:05.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.833+0000 7fafd7400640 1 --2- 172.21.15.31:0/1033384950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafd8112530 0x7fafd81a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:05.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.833+0000 7fafd6a00640 1 -- 172.21.15.31:0/1033384950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafd8074360 msgr2=0x7fafd819ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:05.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.833+0000 7fafd6a00640 1 --2- 172.21.15.31:0/1033384950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafd8074360 0x7fafd819ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.833+0000 7fafd6a00640 1 -- 172.21.15.31:0/1033384950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafd8112530 msgr2=0x7fafd81a2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:05.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.833+0000 7fafd6a00640 1 --2- 172.21.15.31:0/1033384950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafd8112530 0x7fafd81a2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:05.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.833+0000 7fafd6a00640 1 -- 172.21.15.31:0/1033384950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd81a3470 con 0x7fafd8073a40 2024-03-21T10:48:05.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.834+0000 7fafd6a00640 1 --2- 172.21.15.31:0/1033384950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fafd8073a40 0x7fafd819e4d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fafc8005e30 tx=0x7fafc8035d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:05.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.834+0000 7fafdca00640 1 -- 172.21.15.31:0/1033384950 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc8039040 con 0x7fafd8073a40 2024-03-21T10:48:05.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.834+0000 7fafdca00640 1 -- 172.21.15.31:0/1033384950 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafc8044070 con 0x7fafd8073a40 2024-03-21T10:48:05.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.834+0000 7fafdca00640 1 -- 172.21.15.31:0/1033384950 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc80328d0 con 0x7fafd8073a40 2024-03-21T10:48:05.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.834+0000 7fafd7e00640 1 -- 172.21.15.31:0/1033384950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd81a3700 con 0x7fafd8073a40 2024-03-21T10:48:05.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.834+0000 7fafd7e00640 1 -- 172.21.15.31:0/1033384950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd81aaff0 con 0x7fafd8073a40 2024-03-21T10:48:05.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.835+0000 7fafd7e00640 1 -- 172.21.15.31:0/1033384950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd804f470 con 0x7fafd8073a40 2024-03-21T10:48:05.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.837+0000 7fafdca00640 1 -- 172.21.15.31:0/1033384950 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fafc8032a70 con 0x7fafd8073a40 2024-03-21T10:48:05.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.838+0000 7fafdca00640 1 --2- 172.21.15.31:0/1033384950 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fafbc0a3100 0x7fafbc0a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:05.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.838+0000 7fafdca00640 1 -- 172.21.15.31:0/1033384950 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fafc80f1f50 con 0x7fafd8073a40 2024-03-21T10:48:05.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.839+0000 7fafd6000640 1 --2- 172.21.15.31:0/1033384950 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fafbc0a3100 0x7fafbc0a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:05.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.840+0000 7fafd6000640 1 --2- 172.21.15.31:0/1033384950 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fafbc0a3100 0x7fafbc0a55c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fafd80ff7d0 tx=0x7fafc400b390 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:05.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:05.842+0000 7fafdca00640 1 -- 172.21.15.31:0/1033384950 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fafc80be750 con 0x7fafd8073a40 2024-03-21T10:48:06.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.128+0000 7fafd7e00640 1 -- 172.21.15.31:0/1033384950 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafd81049e0 con 0x7fafd8073a40 2024-03-21T10:48:06.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.129+0000 7fafdca00640 1 -- 172.21.15.31:0/1033384950 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11387 (secure 0 0 0) 0x7fafc80be050 con 0x7fafd8073a40 2024-03-21T10:48:06.126 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:48:06.126 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:46:51.185295+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24547,24596]},"id":1}]} 2024-03-21T10:48:06.126 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-03-21T10:48:06.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.132+0000 7fafd7e00640 1 -- 172.21.15.31:0/1033384950 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fafbc0a3100 msgr2=0x7fafbc0a55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:06.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.132+0000 7fafd7e00640 1 --2- 172.21.15.31:0/1033384950 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fafbc0a3100 0x7fafbc0a55c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fafd80ff7d0 tx=0x7fafc400b390 comp rx=0 tx=0).stop 2024-03-21T10:48:06.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.132+0000 7fafd7e00640 1 -- 172.21.15.31:0/1033384950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fafd8073a40 msgr2=0x7fafd819e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:06.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.132+0000 7fafd7e00640 1 --2- 172.21.15.31:0/1033384950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fafd8073a40 0x7fafd819e4d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fafc8005e30 tx=0x7fafc8035d70 comp rx=0 tx=0).stop 2024-03-21T10:48:06.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.133+0000 7fafd7e00640 1 -- 172.21.15.31:0/1033384950 shutdown_connections 2024-03-21T10:48:06.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.133+0000 7fafd7e00640 1 --2- 172.21.15.31:0/1033384950 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fafbc0a3100 0x7fafbc0a55c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.133+0000 7fafd7e00640 1 --2- 172.21.15.31:0/1033384950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafd8112530 0x7fafd81a2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.133+0000 7fafd7e00640 1 --2- 172.21.15.31:0/1033384950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafd8074360 0x7fafd819ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.133+0000 7fafd7e00640 1 --2- 172.21.15.31:0/1033384950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fafd8073a40 0x7fafd819e4d0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.133+0000 7fafd7e00640 1 -- 172.21.15.31:0/1033384950 >> 172.21.15.31:0/1033384950 conn(0x7fafd80fd070 msgr2=0x7fafd80fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:06.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.133+0000 7fafd7e00640 1 -- 172.21.15.31:0/1033384950 shutdown_connections 2024-03-21T10:48:06.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.133+0000 7fafd7e00640 1 -- 172.21.15.31:0/1033384950 wait complete. 2024-03-21T10:48:06.149 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:48:06.149 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-21T10:48:06.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.406+0000 7fc777e00640 1 -- 172.21.15.31:0/374137550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7700ffae0 msgr2=0x7fc7700ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:06.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.406+0000 7fc777e00640 1 --2- 172.21.15.31:0/374137550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7700ffae0 0x7fc7700ffec0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fc764009930 tx=0x7fc76402bc50 comp rx=0 tx=0).stop 2024-03-21T10:48:06.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.408+0000 7fc777e00640 1 -- 172.21.15.31:0/374137550 shutdown_connections 2024-03-21T10:48:06.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.408+0000 7fc777e00640 1 --2- 172.21.15.31:0/374137550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc77010e020 0x7fc770110410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.408+0000 7fc777e00640 1 --2- 172.21.15.31:0/374137550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc770100400 0x7fc77010dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.408+0000 7fc777e00640 1 --2- 172.21.15.31:0/374137550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7700ffae0 0x7fc7700ffec0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.408+0000 7fc777e00640 1 -- 172.21.15.31:0/374137550 >> 172.21.15.31:0/374137550 conn(0x7fc7700753e0 msgr2=0x7fc7700757f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:06.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.409+0000 7fc777e00640 1 -- 172.21.15.31:0/374137550 shutdown_connections 2024-03-21T10:48:06.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.409+0000 7fc777e00640 1 -- 172.21.15.31:0/374137550 wait complete. 2024-03-21T10:48:06.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.411+0000 7fc777e00640 1 Processor -- start 2024-03-21T10:48:06.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.412+0000 7fc777e00640 1 -- start start 2024-03-21T10:48:06.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.413+0000 7fc777e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7700ffae0 0x7fc77019a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.413+0000 7fc777e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc770100400 0x7fc77019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.413+0000 7fc777e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc77010e020 0x7fc77019eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.413+0000 7fc777e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc770113d40 con 0x7fc770100400 2024-03-21T10:48:06.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.413+0000 7fc777e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc770113bc0 con 0x7fc7700ffae0 2024-03-21T10:48:06.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.413+0000 7fc777e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc770113ec0 con 0x7fc77010e020 2024-03-21T10:48:06.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.414+0000 7fc775e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc77010e020 0x7fc77019eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.414+0000 7fc775e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc77010e020 0x7fc77019eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53298/0 (socket says 172.21.15.31:53298) 2024-03-21T10:48:06.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.414+0000 7fc775e00640 1 -- 172.21.15.31:0/981307983 learned_addr learned my addr 172.21.15.31:0/981307983 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:06.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.414+0000 7fc774a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc770100400 0x7fc77019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.414+0000 7fc775400640 1 --2- 172.21.15.31:0/981307983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7700ffae0 0x7fc77019a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.414+0000 7fc775e00640 1 -- 172.21.15.31:0/981307983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7700ffae0 msgr2=0x7fc77019a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:06.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.414+0000 7fc775e00640 1 --2- 172.21.15.31:0/981307983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7700ffae0 0x7fc77019a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.415+0000 7fc775e00640 1 -- 172.21.15.31:0/981307983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc770100400 msgr2=0x7fc77019a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:06.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.415+0000 7fc775e00640 1 --2- 172.21.15.31:0/981307983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc770100400 0x7fc77019a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.415+0000 7fc775e00640 1 -- 172.21.15.31:0/981307983 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc77019f1f0 con 0x7fc77010e020 2024-03-21T10:48:06.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.415+0000 7fc774a00640 1 --2- 172.21.15.31:0/981307983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc770100400 0x7fc77019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:48:06.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.416+0000 7fc775e00640 1 --2- 172.21.15.31:0/981307983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc77010e020 0x7fc77019eac0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fc76800b9e0 tx=0x7fc76800bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:06.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.416+0000 7fc76e000640 1 -- 172.21.15.31:0/981307983 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc768019070 con 0x7fc77010e020 2024-03-21T10:48:06.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.417+0000 7fc76e000640 1 -- 172.21.15.31:0/981307983 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc768014440 con 0x7fc77010e020 2024-03-21T10:48:06.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.417+0000 7fc777e00640 1 -- 172.21.15.31:0/981307983 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc77019f4e0 con 0x7fc77010e020 2024-03-21T10:48:06.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.417+0000 7fc76e000640 1 -- 172.21.15.31:0/981307983 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7680125c0 con 0x7fc77010e020 2024-03-21T10:48:06.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.417+0000 7fc777e00640 1 -- 172.21.15.31:0/981307983 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc770103690 con 0x7fc77010e020 2024-03-21T10:48:06.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.426+0000 7fc76e000640 1 -- 172.21.15.31:0/981307983 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fc76801c070 con 0x7fc77010e020 2024-03-21T10:48:06.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.429+0000 7fc76e000640 1 --2- 172.21.15.31:0/981307983 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc7580a3260 0x7fc7580a5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.429+0000 7fc77ac19740 1 Processor -- start 2024-03-21T10:48:06.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.429+0000 7fc76e000640 1 -- 172.21.15.31:0/981307983 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9534+0+0 (secure 0 0 0) 0x7fc7680cef90 con 0x7fc77010e020 2024-03-21T10:48:06.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.430+0000 7fc775400640 1 --2- 172.21.15.31:0/981307983 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc7580a3260 0x7fc7580a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.431+0000 7fc77ac19740 1 -- start start 2024-03-21T10:48:06.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.431+0000 7fc77ac19740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636bc5e96d0 0x5636bc5e9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.431+0000 7fc77ac19740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636bc5ea080 0x5636bc5f2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.432+0000 7fc77ac19740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636bc5f2c00 0x5636bc5f5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.432+0000 7fc77ac19740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bc54d2c0 con 0x5636bc5f2c00 2024-03-21T10:48:06.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.432+0000 7fc77ac19740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bc4f5c90 con 0x5636bc5e96d0 2024-03-21T10:48:06.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.432+0000 7fc77ac19740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bc549360 con 0x5636bc5ea080 2024-03-21T10:48:06.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.432+0000 7fc774a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636bc5ea080 0x5636bc5f2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.432+0000 7fc774a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636bc5ea080 0x5636bc5f2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53308/0 (socket says 172.21.15.31:53308) 2024-03-21T10:48:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.432+0000 7fc774a00640 1 -- 172.21.15.31:0/2794273723 learned_addr learned my addr 172.21.15.31:0/2794273723 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.432+0000 7fc775e00640 1 --2- 172.21.15.31:0/2794273723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636bc5f2c00 0x5636bc5f5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.433+0000 7fc774a00640 1 -- 172.21.15.31:0/2794273723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636bc5e96d0 msgr2=0x5636bc5e9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.433+0000 7fc774a00640 1 --2- 172.21.15.31:0/2794273723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636bc5e96d0 0x5636bc5e9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.433+0000 7fc774a00640 1 -- 172.21.15.31:0/2794273723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636bc5f2c00 msgr2=0x5636bc5f5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.433+0000 7fc774a00640 1 --2- 172.21.15.31:0/2794273723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636bc5f2c00 0x5636bc5f5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.433+0000 7fc774a00640 1 -- 172.21.15.31:0/2794273723 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636bc51a080 con 0x5636bc5ea080 2024-03-21T10:48:06.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.434+0000 7fc775400640 1 --2- 172.21.15.31:0/981307983 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc7580a3260 0x7fc7580a5720 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc764009900 tx=0x7fc764000bc0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:06.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.434+0000 7fc774a00640 1 --2- 172.21.15.31:0/2794273723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636bc5ea080 0x5636bc5f2630 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc75c007990 tx=0x7fc75c007e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a14abe99df265781 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:06.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.435+0000 7fc762600640 1 -- 172.21.15.31:0/2794273723 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75c027020 con 0x5636bc5ea080 2024-03-21T10:48:06.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.435+0000 7fc762600640 1 -- 172.21.15.31:0/2794273723 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc75c0233f0 con 0x5636bc5ea080 2024-03-21T10:48:06.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.435+0000 7fc762600640 1 -- 172.21.15.31:0/2794273723 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75c01f040 con 0x5636bc5ea080 2024-03-21T10:48:06.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.438+0000 7fc77ac19740 1 -- 172.21.15.31:0/2794273723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636bc5ea080 msgr2=0x5636bc5f2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:06.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.438+0000 7fc77ac19740 1 --2- 172.21.15.31:0/2794273723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636bc5ea080 0x5636bc5f2630 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc75c007990 tx=0x7fc75c007e50 comp rx=0 tx=0).stop 2024-03-21T10:48:06.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.439+0000 7fc77ac19740 1 -- 172.21.15.31:0/2794273723 shutdown_connections 2024-03-21T10:48:06.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.439+0000 7fc77ac19740 1 --2- 172.21.15.31:0/2794273723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636bc5f2c00 0x5636bc5f5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.439+0000 7fc77ac19740 1 --2- 172.21.15.31:0/2794273723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636bc5ea080 0x5636bc5f2630 secure :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc75c007990 tx=0x7fc75c007e50 comp rx=0 tx=0).stop 2024-03-21T10:48:06.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.439+0000 7fc77ac19740 1 --2- 172.21.15.31:0/2794273723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636bc5e96d0 0x5636bc5e9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.439+0000 7fc77ac19740 1 -- 172.21.15.31:0/2794273723 >> 172.21.15.31:0/2794273723 conn(0x5636bc5e3da0 msgr2=0x5636bc5e4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:06.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.440+0000 7fc77ac19740 1 -- 172.21.15.31:0/2794273723 shutdown_connections 2024-03-21T10:48:06.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.442+0000 7fc77ac19740 1 -- 172.21.15.31:0/2794273723 wait complete. 2024-03-21T10:48:06.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.442+0000 7fc77ac19740 1 -- start start 2024-03-21T10:48:06.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.443+0000 7fc77ac19740 1 Processor -- start 2024-03-21T10:48:06.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.445+0000 7fc77ac19740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636bc5e96d0 0x5636bc5efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.445+0000 7fc77ac19740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636bc5f2c00 0x5636bc5eff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.445+0000 7fc77ac19740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636bc5f0490 0x5636bc5fefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.446+0000 7fc775400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636bc5e96d0 0x5636bc5efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.446+0000 7fc774a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636bc5f2c00 0x5636bc5eff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.446+0000 7fc774a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636bc5f2c00 0x5636bc5eff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53322/0 (socket says 172.21.15.31:53322) 2024-03-21T10:48:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.446+0000 7fc775400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636bc5e96d0 0x5636bc5efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58994/0 (socket says 172.21.15.31:58994) 2024-03-21T10:48:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.446+0000 7fc774a00640 1 -- 172.21.15.31:0/3176757053 learned_addr learned my addr 172.21.15.31:0/3176757053 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.446+0000 7fc77ac19740 1 -- 172.21.15.31:0/3176757053 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bc5f5b70 con 0x5636bc5f0490 2024-03-21T10:48:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.446+0000 7fc77ac19740 1 -- 172.21.15.31:0/3176757053 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bc5090d0 con 0x5636bc5e96d0 2024-03-21T10:48:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.446+0000 7fc77ac19740 1 -- 172.21.15.31:0/3176757053 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bc5f5cf0 con 0x5636bc5f2c00 2024-03-21T10:48:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.446+0000 7fc77ac19740 10 client.0 _ll_register_callbacks cb 0x5636bc50de80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:06.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.446+0000 7fc775e00640 1 --2- 172.21.15.31:0/3176757053 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636bc5f0490 0x5636bc5fefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.447+0000 7fc774a00640 1 -- 172.21.15.31:0/3176757053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636bc5e96d0 msgr2=0x5636bc5efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:06.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.447+0000 7fc774a00640 1 --2- 172.21.15.31:0/3176757053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636bc5e96d0 0x5636bc5efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.447+0000 7fc774a00640 1 -- 172.21.15.31:0/3176757053 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636bc5f0490 msgr2=0x5636bc5fefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:06.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.447+0000 7fc774a00640 1 --2- 172.21.15.31:0/3176757053 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636bc5f0490 0x5636bc5fefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.447+0000 7fc774a00640 1 -- 172.21.15.31:0/3176757053 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5636bc2613f0 con 0x5636bc5f2c00 2024-03-21T10:48:06.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.448+0000 7fc774a00640 1 --2- 172.21.15.31:0/3176757053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636bc5f2c00 0x5636bc5eff50 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc75c023dc0 tx=0x7fc75c023be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d58f4afd8eced153 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:06.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.448+0000 7fc756a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:48:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.449+0000 7fc756a00640 1 -- 172.21.15.31:0/3176757053 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75c027040 con 0x5636bc5f2c00 2024-03-21T10:48:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.449+0000 7fc77ac19740 1 -- 172.21.15.31:0/3176757053 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc75c0233f0 con 0x5636bc5f2c00 2024-03-21T10:48:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.449+0000 7fc756a00640 1 -- 172.21.15.31:0/3176757053 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9534+0+0 (secure 0 0 0) 0x7fc75c01ec00 con 0x5636bc5f2c00 2024-03-21T10:48:06.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.450+0000 7fc756a00640 1 -- 172.21.15.31:0/3176757053 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc75c03f050 con 0x5636bc5f2c00 2024-03-21T10:48:06.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.450+0000 7fc77ac19740 10 client.27907 fetch_fsmap learned FSMap version 28 2024-03-21T10:48:06.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.450+0000 7fc77ac19740 1 -- 172.21.15.31:0/3176757053 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5636bc32b7d0 con 0x5636bc5f2c00 2024-03-21T10:48:06.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.452+0000 7fc756a00640 1 -- 172.21.15.31:0/3176757053 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc75c0233f0 con 0x5636bc5f2c00 2024-03-21T10:48:06.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.452+0000 7fc756a00640 1 -- 172.21.15.31:0/3176757053 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc75c022030 con 0x5636bc5f2c00 2024-03-21T10:48:06.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.452+0000 7fc77ac19740 10 client.27907 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:48:06.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.452+0000 7fc77ac19740 10 client.27907 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:48:06.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.452+0000 7fc77ac19740 20 client.27907 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:06.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.452+0000 7fc77ac19740 1 --2- 172.21.15.31:0/3176757053 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5636bc602ee0 0x5636bc6052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.452+0000 7fc77ac19740 4 client.27907 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:48:06.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.452+0000 7fc77ac19740 1 -- 172.21.15.31:0/3176757053 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5636bc55a800 con 0x5636bc602ee0 2024-03-21T10:48:06.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.453+0000 7fc775400640 1 --2- 172.21.15.31:0/3176757053 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5636bc602ee0 0x5636bc6052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.454+0000 7fc756a00640 10 client.27907 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:48:06.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.456+0000 7fc775400640 1 --2- 172.21.15.31:0/3176757053 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5636bc602ee0 0x5636bc6052d0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f53dda8e0abba4bd server_cookie=d25615d7da1340ae in_seq=0 out_seq=0 2024-03-21T10:48:06.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.457+0000 7fc756a00640 1 -- 172.21.15.31:0/3176757053 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5636bc55a800 con 0x5636bc602ee0 2024-03-21T10:48:06.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.457+0000 7fc756a00640 10 client.27907 handle_command_reply: tid=0 2024-03-21T10:48:06.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.458+0000 7fc77ac19740 1 client.27907 shutdown 2024-03-21T10:48:06.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.458+0000 7fc77ac19740 1 -- 172.21.15.31:0/3176757053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636bc5f2c00 msgr2=0x5636bc5eff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:06.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.458+0000 7fc77ac19740 1 --2- 172.21.15.31:0/3176757053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636bc5f2c00 0x5636bc5eff50 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc75c023dc0 tx=0x7fc75c023be0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.458+0000 7fc77ac19740 1 -- 172.21.15.31:0/3176757053 shutdown_connections 2024-03-21T10:48:06.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.458+0000 7fc77ac19740 1 --2- 172.21.15.31:0/3176757053 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5636bc602ee0 0x5636bc6052d0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.458+0000 7fc77ac19740 1 --2- 172.21.15.31:0/3176757053 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636bc5f0490 0x5636bc5fefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.458+0000 7fc77ac19740 1 --2- 172.21.15.31:0/3176757053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636bc5f2c00 0x5636bc5eff50 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.458+0000 7fc77ac19740 1 --2- 172.21.15.31:0/3176757053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636bc5e96d0 0x5636bc5efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.458+0000 7fc77ac19740 1 -- 172.21.15.31:0/3176757053 >> 172.21.15.31:0/3176757053 conn(0x5636bc5e3da0 msgr2=0x5636bc5e5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:06.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.458+0000 7fc756a00640 0 client.27907 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:48:06.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.459+0000 7fc77ac19740 1 -- 172.21.15.31:0/3176757053 shutdown_connections 2024-03-21T10:48:06.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.459+0000 7fc77ac19740 1 -- 172.21.15.31:0/3176757053 wait complete. 2024-03-21T10:48:06.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.459+0000 7fc77ac19740 20 client.27907 trim_cache size 0 max 16384 2024-03-21T10:48:06.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.492+0000 7fc77ac19740 1 Processor -- start 2024-03-21T10:48:06.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.493+0000 7fc77ac19740 1 -- start start 2024-03-21T10:48:06.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.494+0000 7fc77ac19740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636bc5e96d0 0x5636bc5fef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.494+0000 7fc77ac19740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636bc5f0490 0x5636bc5ff490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.494+0000 7fc77ac19740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636bc5f2c00 0x5636bc5ecc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.494+0000 7fc77ac19740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc770073060 con 0x5636bc5f2c00 2024-03-21T10:48:06.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.494+0000 7fc77ac19740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc770070b60 con 0x5636bc5e96d0 2024-03-21T10:48:06.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.494+0000 7fc77ac19740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77006ec20 con 0x5636bc5f0490 2024-03-21T10:48:06.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.495+0000 7fc775400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636bc5e96d0 0x5636bc5fef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.495+0000 7fc775400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636bc5e96d0 0x5636bc5fef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:59006/0 (socket says 172.21.15.31:59006) 2024-03-21T10:48:06.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.495+0000 7fc775400640 1 -- 172.21.15.31:0/501071108 learned_addr learned my addr 172.21.15.31:0/501071108 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:06.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.495+0000 7fc774a00640 1 --2- 172.21.15.31:0/501071108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636bc5f0490 0x5636bc5ff490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.495+0000 7fc775e00640 1 --2- 172.21.15.31:0/501071108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636bc5f2c00 0x5636bc5ecc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.496+0000 7fc775400640 1 -- 172.21.15.31:0/501071108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636bc5f0490 msgr2=0x5636bc5ff490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:06.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.496+0000 7fc775400640 1 --2- 172.21.15.31:0/501071108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636bc5f0490 0x5636bc5ff490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.497+0000 7fc775400640 1 -- 172.21.15.31:0/501071108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636bc5f2c00 msgr2=0x5636bc5ecc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:06.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.497+0000 7fc775400640 1 --2- 172.21.15.31:0/501071108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636bc5f2c00 0x5636bc5ecc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.497+0000 7fc775400640 1 -- 172.21.15.31:0/501071108 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636bc5eb5e0 con 0x5636bc5e96d0 2024-03-21T10:48:06.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.499+0000 7fc775e00640 1 --2- 172.21.15.31:0/501071108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636bc5f2c00 0x5636bc5ecc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:48:06.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.499+0000 7fc774a00640 1 --2- 172.21.15.31:0/501071108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636bc5f0490 0x5636bc5ff490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:06.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.499+0000 7fc775400640 1 --2- 172.21.15.31:0/501071108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636bc5e96d0 0x5636bc5fef50 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fc76402c840 tx=0x7fc76402e050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b9c081d9342731e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:06.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.500+0000 7fc761c00640 1 -- 172.21.15.31:0/501071108 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76402cd30 con 0x5636bc5e96d0 2024-03-21T10:48:06.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.500+0000 7fc761c00640 1 -- 172.21.15.31:0/501071108 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc764004ba0 con 0x5636bc5e96d0 2024-03-21T10:48:06.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.501+0000 7fc761c00640 1 -- 172.21.15.31:0/501071108 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc764055860 con 0x5636bc5e96d0 2024-03-21T10:48:06.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.502+0000 7fc77ac19740 1 -- 172.21.15.31:0/501071108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636bc5e96d0 msgr2=0x5636bc5fef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:06.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.502+0000 7fc77ac19740 1 --2- 172.21.15.31:0/501071108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636bc5e96d0 0x5636bc5fef50 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fc76402c840 tx=0x7fc76402e050 comp rx=0 tx=0).stop 2024-03-21T10:48:06.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.502+0000 7fc77ac19740 1 -- 172.21.15.31:0/501071108 shutdown_connections 2024-03-21T10:48:06.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.502+0000 7fc77ac19740 1 --2- 172.21.15.31:0/501071108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636bc5f2c00 0x5636bc5ecc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.502+0000 7fc77ac19740 1 --2- 172.21.15.31:0/501071108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636bc5f0490 0x5636bc5ff490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.502+0000 7fc77ac19740 1 --2- 172.21.15.31:0/501071108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636bc5e96d0 0x5636bc5fef50 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.502+0000 7fc77ac19740 1 -- 172.21.15.31:0/501071108 >> 172.21.15.31:0/501071108 conn(0x5636bc5e3da0 msgr2=0x5636bc5e2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:06.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.503+0000 7fc77ac19740 1 -- 172.21.15.31:0/501071108 shutdown_connections 2024-03-21T10:48:06.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.505+0000 7fc77ac19740 1 -- 172.21.15.31:0/501071108 wait complete. 2024-03-21T10:48:06.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.505+0000 7fc77ac19740 1 -- start start 2024-03-21T10:48:06.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.506+0000 7fc77ac19740 1 Processor -- start 2024-03-21T10:48:06.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.507+0000 7fc77ac19740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636bc5e96d0 0x5636bc605940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.507+0000 7fc77ac19740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636bc5f0490 0x5636bc605e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.507+0000 7fc77ac19740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636bc5f2c00 0x5636bc6063c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.507+0000 7fc774a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636bc5f0490 0x5636bc605e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.508+0000 7fc774a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636bc5f0490 0x5636bc605e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:59022/0 (socket says 172.21.15.31:59022) 2024-03-21T10:48:06.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.508+0000 7fc774a00640 1 -- 172.21.15.31:0/2582380001 learned_addr learned my addr 172.21.15.31:0/2582380001 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:06.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.508+0000 7fc775e00640 1 --2- 172.21.15.31:0/2582380001 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636bc5f2c00 0x5636bc6063c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.508+0000 7fc77ac19740 1 -- 172.21.15.31:0/2582380001 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77006f220 con 0x5636bc5e96d0 2024-03-21T10:48:06.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.508+0000 7fc77ac19740 1 -- 172.21.15.31:0/2582380001 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc770070500 con 0x5636bc5f0490 2024-03-21T10:48:06.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.508+0000 7fc77ac19740 1 -- 172.21.15.31:0/2582380001 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77006f840 con 0x5636bc5f2c00 2024-03-21T10:48:06.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.508+0000 7fc77ac19740 10 client.0 _ll_register_callbacks cb 0x7fc76809aec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:06.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.509+0000 7fc775e00640 1 -- 172.21.15.31:0/2582380001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636bc5f0490 msgr2=0x5636bc605e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:06.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.509+0000 7fc775e00640 1 --2- 172.21.15.31:0/2582380001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636bc5f0490 0x5636bc605e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.509+0000 7fc775e00640 1 -- 172.21.15.31:0/2582380001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636bc5e96d0 msgr2=0x5636bc605940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:06.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.509+0000 7fc775e00640 1 --2- 172.21.15.31:0/2582380001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636bc5e96d0 0x5636bc605940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.509+0000 7fc775e00640 1 -- 172.21.15.31:0/2582380001 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5636bc65d660 con 0x5636bc5f2c00 2024-03-21T10:48:06.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.511+0000 7fc775e00640 1 --2- 172.21.15.31:0/2582380001 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636bc5f2c00 0x5636bc6063c0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc7680cc3a0 tx=0x7fc7680bb040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba6935c6b8f12ced server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:06.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.511+0000 7fc756a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:48:06.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.511+0000 7fc756a00640 1 -- 172.21.15.31:0/2582380001 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7680cc510 con 0x5636bc5f2c00 2024-03-21T10:48:06.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.511+0000 7fc756a00640 1 -- 172.21.15.31:0/2582380001 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9534+0+0 (secure 0 0 0) 0x7fc7680cc6b0 con 0x5636bc5f2c00 2024-03-21T10:48:06.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.511+0000 7fc77ac19740 1 -- 172.21.15.31:0/2582380001 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc764004ba0 con 0x5636bc5f2c00 2024-03-21T10:48:06.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.512+0000 7fc756a00640 1 -- 172.21.15.31:0/2582380001 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc7680ab050 con 0x5636bc5f2c00 2024-03-21T10:48:06.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.513+0000 7fc77ac19740 10 client.27913 fetch_fsmap learned FSMap version 28 2024-03-21T10:48:06.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.513+0000 7fc77ac19740 1 -- 172.21.15.31:0/2582380001 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5636bc65e910 con 0x5636bc5f2c00 2024-03-21T10:48:06.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.514+0000 7fc756a00640 1 -- 172.21.15.31:0/2582380001 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc764004ba0 con 0x5636bc5f2c00 2024-03-21T10:48:06.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.516+0000 7fc756a00640 1 -- 172.21.15.31:0/2582380001 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc76809a540 con 0x5636bc5f2c00 2024-03-21T10:48:06.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.516+0000 7fc77ac19740 10 client.27913 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:48:06.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.516+0000 7fc77ac19740 10 client.27913 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:48:06.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.516+0000 7fc77ac19740 20 client.27913 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:06.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.516+0000 7fc77ac19740 1 --2- 172.21.15.31:0/2582380001 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5636bc65eff0 0x5636bc661410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.516+0000 7fc77ac19740 4 client.27913 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:48:06.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.516+0000 7fc77ac19740 1 -- 172.21.15.31:0/2582380001 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5636bc5f02e0 con 0x5636bc65eff0 2024-03-21T10:48:06.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.518+0000 7fc775400640 1 --2- 172.21.15.31:0/2582380001 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5636bc65eff0 0x5636bc661410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.519+0000 7fc756a00640 10 client.27913 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:48:06.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.519+0000 7fc775400640 1 --2- 172.21.15.31:0/2582380001 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5636bc65eff0 0x5636bc661410 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7d90f323f8537d73 server_cookie=ecb483524b64b0a3 in_seq=0 out_seq=0 2024-03-21T10:48:06.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.519+0000 7fc756a00640 1 -- 172.21.15.31:0/2582380001 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5636bc5f02e0 con 0x5636bc65eff0 2024-03-21T10:48:06.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.519+0000 7fc756a00640 10 client.27913 handle_command_reply: tid=0 2024-03-21T10:48:06.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.519+0000 7fc77ac19740 1 client.27913 shutdown 2024-03-21T10:48:06.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.520+0000 7fc77ac19740 1 -- 172.21.15.31:0/2582380001 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636bc5f2c00 msgr2=0x5636bc6063c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:06.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.520+0000 7fc77ac19740 1 --2- 172.21.15.31:0/2582380001 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636bc5f2c00 0x5636bc6063c0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc7680cc3a0 tx=0x7fc7680bb040 comp rx=0 tx=0).stop 2024-03-21T10:48:06.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.520+0000 7fc77ac19740 1 -- 172.21.15.31:0/2582380001 shutdown_connections 2024-03-21T10:48:06.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.520+0000 7fc77ac19740 1 --2- 172.21.15.31:0/2582380001 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5636bc65eff0 0x5636bc661410 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.520+0000 7fc77ac19740 1 --2- 172.21.15.31:0/2582380001 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636bc5f2c00 0x5636bc6063c0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.520+0000 7fc77ac19740 1 --2- 172.21.15.31:0/2582380001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636bc5f0490 0x5636bc605e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.520+0000 7fc77ac19740 1 --2- 172.21.15.31:0/2582380001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636bc5e96d0 0x5636bc605940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.520+0000 7fc756a00640 0 client.27913 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:48:06.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.520+0000 7fc77ac19740 1 -- 172.21.15.31:0/2582380001 >> 172.21.15.31:0/2582380001 conn(0x5636bc5e3da0 msgr2=0x5636bc5e2620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:06.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.521+0000 7fc77ac19740 1 -- 172.21.15.31:0/2582380001 shutdown_connections 2024-03-21T10:48:06.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.521+0000 7fc77ac19740 1 -- 172.21.15.31:0/2582380001 wait complete. 2024-03-21T10:48:06.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.521+0000 7fc77ac19740 20 client.27913 trim_cache size 0 max 16384 2024-03-21T10:48:06.519 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:48:06.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.522+0000 7fc777e00640 1 -- 172.21.15.31:0/981307983 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc7580a3260 msgr2=0x7fc7580a5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:06.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.522+0000 7fc777e00640 1 --2- 172.21.15.31:0/981307983 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc7580a3260 0x7fc7580a5720 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc764009900 tx=0x7fc764000bc0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.522+0000 7fc777e00640 1 -- 172.21.15.31:0/981307983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc77010e020 msgr2=0x7fc77019eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:06.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.522+0000 7fc777e00640 1 --2- 172.21.15.31:0/981307983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc77010e020 0x7fc77019eac0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fc76800b9e0 tx=0x7fc76800bea0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.522+0000 7fc777e00640 1 -- 172.21.15.31:0/981307983 shutdown_connections 2024-03-21T10:48:06.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.522+0000 7fc777e00640 1 --2- 172.21.15.31:0/981307983 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc7580a3260 0x7fc7580a5720 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.522+0000 7fc777e00640 1 --2- 172.21.15.31:0/981307983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc77010e020 0x7fc77019eac0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.522+0000 7fc777e00640 1 --2- 172.21.15.31:0/981307983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc770100400 0x7fc77019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.522+0000 7fc777e00640 1 --2- 172.21.15.31:0/981307983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7700ffae0 0x7fc77019a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.522+0000 7fc777e00640 1 -- 172.21.15.31:0/981307983 >> 172.21.15.31:0/981307983 conn(0x7fc7700753e0 msgr2=0x7fc77010e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:06.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.523+0000 7fc777e00640 1 -- 172.21.15.31:0/981307983 shutdown_connections 2024-03-21T10:48:06.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.523+0000 7fc777e00640 1 -- 172.21.15.31:0/981307983 wait complete. 2024-03-21T10:48:06.547 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:06 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1033384950' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:48:06.550 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:48:06.550 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-21T10:48:06.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:06 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1033384950' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:48:06.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:06 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1033384950' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:48:06.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.884+0000 7f4e87200640 1 -- 172.21.15.31:0/2564906131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e8810fc90 msgr2=0x7f4e88110070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:06.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.884+0000 7f4e87200640 1 --2- 172.21.15.31:0/2564906131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e8810fc90 0x7f4e88110070 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4e70009930 tx=0x7f4e7002bc20 comp rx=0 tx=0).stop 2024-03-21T10:48:06.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.884+0000 7f4e87200640 1 -- 172.21.15.31:0/2564906131 shutdown_connections 2024-03-21T10:48:06.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.884+0000 7f4e87200640 1 --2- 172.21.15.31:0/2564906131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e8810e310 0x7f4e8811ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.884+0000 7f4e87200640 1 --2- 172.21.15.31:0/2564906131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e8810d780 0x7f4e8810dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.885+0000 7f4e87200640 1 --2- 172.21.15.31:0/2564906131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e8810fc90 0x7f4e88110070 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.885+0000 7f4e87200640 1 -- 172.21.15.31:0/2564906131 >> 172.21.15.31:0/2564906131 conn(0x7f4e8806c580 msgr2=0x7f4e8806c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:06.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.885+0000 7f4e87200640 1 -- 172.21.15.31:0/2564906131 shutdown_connections 2024-03-21T10:48:06.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.885+0000 7f4e87200640 1 -- 172.21.15.31:0/2564906131 wait complete. 2024-03-21T10:48:06.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.886+0000 7f4e87200640 1 Processor -- start 2024-03-21T10:48:06.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.887+0000 7f4e87200640 1 -- start start 2024-03-21T10:48:06.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.887+0000 7f4e87200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e8810d780 0x7f4e881ac710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.887+0000 7f4e87200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e8810e310 0x7f4e881acc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.887+0000 7f4e87200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e8810fc90 0x7f4e881a68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.887+0000 7f4e87200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e88120660 con 0x7f4e8810e310 2024-03-21T10:48:06.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.887+0000 7f4e87200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e881204e0 con 0x7f4e8810d780 2024-03-21T10:48:06.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.887+0000 7f4e87200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e881207e0 con 0x7f4e8810fc90 2024-03-21T10:48:06.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.888+0000 7f4e85400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e8810e310 0x7f4e881acc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.888+0000 7f4e85e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e8810d780 0x7f4e881ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.888+0000 7f4e86800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e8810fc90 0x7f4e881a68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.888+0000 7f4e86800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e8810fc90 0x7f4e881a68c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53362/0 (socket says 172.21.15.31:53362) 2024-03-21T10:48:06.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.888+0000 7f4e86800640 1 -- 172.21.15.31:0/4091030972 learned_addr learned my addr 172.21.15.31:0/4091030972 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:06.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.889+0000 7f4e86800640 1 -- 172.21.15.31:0/4091030972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e8810d780 msgr2=0x7f4e881ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:06.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.889+0000 7f4e86800640 1 --2- 172.21.15.31:0/4091030972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e8810d780 0x7f4e881ac710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.889+0000 7f4e86800640 1 -- 172.21.15.31:0/4091030972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e8810e310 msgr2=0x7f4e881acc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:06.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.889+0000 7f4e86800640 1 --2- 172.21.15.31:0/4091030972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e8810e310 0x7f4e881acc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.889+0000 7f4e86800640 1 -- 172.21.15.31:0/4091030972 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e881a7040 con 0x7f4e8810fc90 2024-03-21T10:48:06.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.889+0000 7f4e85400640 1 --2- 172.21.15.31:0/4091030972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e8810e310 0x7f4e881acc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:48:06.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.890+0000 7f4e86800640 1 --2- 172.21.15.31:0/4091030972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e8810fc90 0x7f4e881a68c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f4e8000eaa0 tx=0x7f4e8000ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:06.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.891+0000 7f4e7ea00640 1 -- 172.21.15.31:0/4091030972 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8000cdb0 con 0x7f4e8810fc90 2024-03-21T10:48:06.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.892+0000 7f4e7ea00640 1 -- 172.21.15.31:0/4091030972 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e80005020 con 0x7f4e8810fc90 2024-03-21T10:48:06.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.892+0000 7f4e87200640 1 -- 172.21.15.31:0/4091030972 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e881a7330 con 0x7f4e8810fc90 2024-03-21T10:48:06.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.892+0000 7f4e87200640 1 -- 172.21.15.31:0/4091030972 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e881b35d0 con 0x7f4e8810fc90 2024-03-21T10:48:06.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.892+0000 7f4e7ea00640 1 -- 172.21.15.31:0/4091030972 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e800116b0 con 0x7f4e8810fc90 2024-03-21T10:48:06.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.899+0000 7f4e7ea00640 1 -- 172.21.15.31:0/4091030972 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f4e80004a40 con 0x7f4e8810fc90 2024-03-21T10:48:06.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.900+0000 7f4e7ea00640 1 --2- 172.21.15.31:0/4091030972 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4e6c0a3220 0x7f4e6c0a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.900+0000 7f4e7ea00640 1 -- 172.21.15.31:0/4091030972 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f4e800cfcc0 con 0x7f4e8810fc90 2024-03-21T10:48:06.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.901+0000 7f4e85e00640 1 --2- 172.21.15.31:0/4091030972 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4e6c0a3220 0x7f4e6c0a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.902+0000 7f4e85e00640 1 --2- 172.21.15.31:0/4091030972 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4e6c0a3220 0x7f4e6c0a56e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4e70006010 tx=0x7f4e70000bc0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:06.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.904+0000 7f4e8f77f740 1 Processor -- start 2024-03-21T10:48:06.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.906+0000 7f4e8f77f740 1 -- start start 2024-03-21T10:48:06.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.907+0000 7f4e8f77f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d278b0 0x55d3c6d27c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.907+0000 7f4e8f77f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3c6d28260 0x55d3c6d30810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.907+0000 7f4e8f77f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3c6d30de0 0x55d3c6d33230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.907+0000 7f4e8f77f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3c6c8b2c0 con 0x55d3c6d28260 2024-03-21T10:48:06.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.907+0000 7f4e8f77f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3c6c33c90 con 0x55d3c6d30de0 2024-03-21T10:48:06.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.907+0000 7f4e8f77f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3c6c87360 con 0x55d3c6d278b0 2024-03-21T10:48:06.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.907+0000 7f4e85e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d278b0 0x55d3c6d27c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.908+0000 7f4e85e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d278b0 0x55d3c6d27c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53376/0 (socket says 172.21.15.31:53376) 2024-03-21T10:48:06.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.908+0000 7f4e85e00640 1 -- 172.21.15.31:0/3588384745 learned_addr learned my addr 172.21.15.31:0/3588384745 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:06.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.908+0000 7f4e86800640 1 --2- 172.21.15.31:0/3588384745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3c6d30de0 0x55d3c6d33230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.908+0000 7f4e85e00640 1 -- 172.21.15.31:0/3588384745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3c6d30de0 msgr2=0x55d3c6d33230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:06.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.908+0000 7f4e85e00640 1 --2- 172.21.15.31:0/3588384745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3c6d30de0 0x55d3c6d33230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.908+0000 7f4e85e00640 1 -- 172.21.15.31:0/3588384745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3c6d28260 msgr2=0x55d3c6d30810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:06.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.908+0000 7f4e85400640 1 --2- 172.21.15.31:0/3588384745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3c6d28260 0x55d3c6d30810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.909+0000 7f4e85e00640 1 --2- 172.21.15.31:0/3588384745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3c6d28260 0x55d3c6d30810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.909+0000 7f4e85e00640 1 -- 172.21.15.31:0/3588384745 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3c6c58080 con 0x55d3c6d278b0 2024-03-21T10:48:06.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.909+0000 7f4e85400640 1 --2- 172.21.15.31:0/3588384745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3c6d28260 0x55d3c6d30810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:48:06.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.909+0000 7f4e86800640 1 --2- 172.21.15.31:0/3588384745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3c6d30de0 0x55d3c6d33230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:48:06.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.910+0000 7f4e85e00640 1 --2- 172.21.15.31:0/3588384745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d278b0 0x55d3c6d27c90 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f4e70035000 tx=0x7f4e7003f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=820829cbc3c1883f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:06.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.910+0000 7f4e77000640 1 -- 172.21.15.31:0/3588384745 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7004e070 con 0x55d3c6d278b0 2024-03-21T10:48:06.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.911+0000 7f4e77000640 1 -- 172.21.15.31:0/3588384745 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e70004ba0 con 0x55d3c6d278b0 2024-03-21T10:48:06.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.911+0000 7f4e77000640 1 -- 172.21.15.31:0/3588384745 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e70005650 con 0x55d3c6d278b0 2024-03-21T10:48:06.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.914+0000 7f4e8f77f740 1 -- 172.21.15.31:0/3588384745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d278b0 msgr2=0x55d3c6d27c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:06.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.914+0000 7f4e8f77f740 1 --2- 172.21.15.31:0/3588384745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d278b0 0x55d3c6d27c90 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f4e70035000 tx=0x7f4e7003f000 comp rx=0 tx=0).stop 2024-03-21T10:48:06.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.915+0000 7f4e8f77f740 1 -- 172.21.15.31:0/3588384745 shutdown_connections 2024-03-21T10:48:06.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.915+0000 7f4e8f77f740 1 --2- 172.21.15.31:0/3588384745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3c6d30de0 0x55d3c6d33230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.915+0000 7f4e8f77f740 1 --2- 172.21.15.31:0/3588384745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3c6d28260 0x55d3c6d30810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.915+0000 7f4e8f77f740 1 --2- 172.21.15.31:0/3588384745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d278b0 0x55d3c6d27c90 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.915+0000 7f4e8f77f740 1 -- 172.21.15.31:0/3588384745 >> 172.21.15.31:0/3588384745 conn(0x55d3c6d21f80 msgr2=0x55d3c6d22360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:06.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.915+0000 7f4e8f77f740 1 -- 172.21.15.31:0/3588384745 shutdown_connections 2024-03-21T10:48:06.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.915+0000 7f4e8f77f740 1 -- 172.21.15.31:0/3588384745 wait complete. 2024-03-21T10:48:06.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.916+0000 7f4e8f77f740 1 -- start start 2024-03-21T10:48:06.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.917+0000 7f4e8f77f740 1 Processor -- start 2024-03-21T10:48:06.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.919+0000 7f4e8f77f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3c6d278b0 0x55d3c6d306b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.919+0000 7f4e8f77f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3c6d28260 0x55d3c6d2da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.919+0000 7f4e8f77f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d30de0 0x55d3c6d2dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.920+0000 7f4e8f77f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3c6d33cf0 con 0x55d3c6d28260 2024-03-21T10:48:06.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.920+0000 7f4e8f77f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3c6c470d0 con 0x55d3c6d278b0 2024-03-21T10:48:06.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.920+0000 7f4e8f77f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3c6d33e70 con 0x55d3c6d30de0 2024-03-21T10:48:06.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.920+0000 7f4e8f77f740 10 client.0 _ll_register_callbacks cb 0x55d3c6c4be80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:06.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.920+0000 7f4e86800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d30de0 0x55d3c6d2dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.920+0000 7f4e86800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d30de0 0x55d3c6d2dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:48490/0 (socket says 172.21.15.31:48490) 2024-03-21T10:48:06.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.920+0000 7f4e86800640 1 -- 172.21.15.31:0/2300786739 learned_addr learned my addr 172.21.15.31:0/2300786739 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:06.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.921+0000 7f4e85e00640 1 --2- 172.21.15.31:0/2300786739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3c6d278b0 0x55d3c6d306b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.921+0000 7f4e85400640 1 --2- 172.21.15.31:0/2300786739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3c6d28260 0x55d3c6d2da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.922+0000 7f4e86800640 1 -- 172.21.15.31:0/2300786739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3c6d278b0 msgr2=0x55d3c6d306b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:06.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.922+0000 7f4e86800640 1 --2- 172.21.15.31:0/2300786739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3c6d278b0 0x55d3c6d306b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.922+0000 7f4e86800640 1 -- 172.21.15.31:0/2300786739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3c6d28260 msgr2=0x55d3c6d2da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:06.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.922+0000 7f4e86800640 1 --2- 172.21.15.31:0/2300786739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3c6d28260 0x55d3c6d2da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.922+0000 7f4e86800640 1 -- 172.21.15.31:0/2300786739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d3c699f3f0 con 0x55d3c6d30de0 2024-03-21T10:48:06.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.923+0000 7f4e86800640 1 --2- 172.21.15.31:0/2300786739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d30de0 0x55d3c6d2dfa0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f4e800cd0d0 tx=0x7f4e8009be00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=123e785abda5afac server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:06.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.923+0000 7f4e6b400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:48:06.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.923+0000 7f4e6b400640 1 -- 172.21.15.31:0/2300786739 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8001f040 con 0x55d3c6d30de0 2024-03-21T10:48:06.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.924+0000 7f4e6b400640 1 -- 172.21.15.31:0/2300786739 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f4e8009b110 con 0x55d3c6d30de0 2024-03-21T10:48:06.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.924+0000 7f4e8f77f740 1 -- 172.21.15.31:0/2300786739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4e70004ba0 con 0x55d3c6d30de0 2024-03-21T10:48:06.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.924+0000 7f4e6b400640 1 -- 172.21.15.31:0/2300786739 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e8009b310 con 0x55d3c6d30de0 2024-03-21T10:48:06.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.924+0000 7f4e8f77f740 10 client.27925 fetch_fsmap learned FSMap version 28 2024-03-21T10:48:06.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.924+0000 7f4e8f77f740 1 -- 172.21.15.31:0/2300786739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d3c6a697d0 con 0x55d3c6d30de0 2024-03-21T10:48:06.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.925+0000 7f4e6b400640 1 -- 172.21.15.31:0/2300786739 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e70004ba0 con 0x55d3c6d30de0 2024-03-21T10:48:06.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.926+0000 7f4e6b400640 1 -- 172.21.15.31:0/2300786739 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f4e8000f040 con 0x55d3c6d30de0 2024-03-21T10:48:06.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.927+0000 7f4e8f77f740 10 client.27925 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:48:06.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.927+0000 7f4e8f77f740 10 client.27925 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:48:06.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.927+0000 7f4e8f77f740 20 client.27925 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:06.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.927+0000 7f4e8f77f740 1 --2- 172.21.15.31:0/2300786739 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55d3c6d41080 0x55d3c6d434a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.928+0000 7f4e8f77f740 4 client.27925 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:48:06.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.928+0000 7f4e8f77f740 1 -- 172.21.15.31:0/2300786739 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d3c6c98800 con 0x55d3c6d41080 2024-03-21T10:48:06.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.928+0000 7f4e85e00640 1 --2- 172.21.15.31:0/2300786739 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55d3c6d41080 0x55d3c6d434a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.931+0000 7f4e85e00640 1 --2- 172.21.15.31:0/2300786739 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55d3c6d41080 0x55d3c6d434a0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dfee0bd90ae32f8f server_cookie=4b3bd81e5ee56816 in_seq=0 out_seq=0 2024-03-21T10:48:06.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.931+0000 7f4e6b400640 10 client.27925 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:48:06.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.935+0000 7f4e6b400640 1 -- 172.21.15.31:0/2300786739 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d3c6c98800 con 0x55d3c6d41080 2024-03-21T10:48:06.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.935+0000 7f4e6b400640 10 client.27925 handle_command_reply: tid=0 2024-03-21T10:48:06.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.935+0000 7f4e8f77f740 1 client.27925 shutdown 2024-03-21T10:48:06.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.936+0000 7f4e8f77f740 1 -- 172.21.15.31:0/2300786739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d30de0 msgr2=0x55d3c6d2dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:06.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.936+0000 7f4e8f77f740 1 --2- 172.21.15.31:0/2300786739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d30de0 0x55d3c6d2dfa0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f4e800cd0d0 tx=0x7f4e8009be00 comp rx=0 tx=0).stop 2024-03-21T10:48:06.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.936+0000 7f4e8f77f740 1 -- 172.21.15.31:0/2300786739 shutdown_connections 2024-03-21T10:48:06.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.936+0000 7f4e8f77f740 1 --2- 172.21.15.31:0/2300786739 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55d3c6d41080 0x55d3c6d434a0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.936+0000 7f4e8f77f740 1 --2- 172.21.15.31:0/2300786739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d30de0 0x55d3c6d2dfa0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.936+0000 7f4e8f77f740 1 --2- 172.21.15.31:0/2300786739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3c6d28260 0x55d3c6d2da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.936+0000 7f4e8f77f740 1 --2- 172.21.15.31:0/2300786739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3c6d278b0 0x55d3c6d306b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.936+0000 7f4e8f77f740 1 -- 172.21.15.31:0/2300786739 >> 172.21.15.31:0/2300786739 conn(0x55d3c6d21f80 msgr2=0x55d3c6d23cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:06.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.936+0000 7f4e6b400640 0 client.27925 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:48:06.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.936+0000 7f4e8f77f740 1 -- 172.21.15.31:0/2300786739 shutdown_connections 2024-03-21T10:48:06.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.937+0000 7f4e8f77f740 1 -- 172.21.15.31:0/2300786739 wait complete. 2024-03-21T10:48:06.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.937+0000 7f4e8f77f740 20 client.27925 trim_cache size 0 max 16384 2024-03-21T10:48:06.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.969+0000 7f4e8f77f740 1 Processor -- start 2024-03-21T10:48:06.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.969+0000 7f4e8f77f740 1 -- start start 2024-03-21T10:48:06.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.970+0000 7f4e8f77f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3c6d278b0 0x55d3c6d213a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.970+0000 7f4e8f77f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d28260 0x55d3c6d30170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.970+0000 7f4e8f77f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3c6d30de0 0x55d3c6d306b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.971+0000 7f4e8f77f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e64012470 con 0x55d3c6d30de0 2024-03-21T10:48:06.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.971+0000 7f4e8f77f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6400ff70 con 0x55d3c6d278b0 2024-03-21T10:48:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.971+0000 7f4e8f77f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6400e030 con 0x55d3c6d28260 2024-03-21T10:48:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.971+0000 7f4e85e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3c6d278b0 0x55d3c6d213a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.971+0000 7f4e85400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d28260 0x55d3c6d30170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.971+0000 7f4e85e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3c6d278b0 0x55d3c6d213a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:40622/0 (socket says 172.21.15.31:40622) 2024-03-21T10:48:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.972+0000 7f4e85e00640 1 -- 172.21.15.31:0/1531035970 learned_addr learned my addr 172.21.15.31:0/1531035970 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.972+0000 7f4e86800640 1 --2- 172.21.15.31:0/1531035970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3c6d30de0 0x55d3c6d306b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.972+0000 7f4e85400640 1 -- 172.21.15.31:0/1531035970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3c6d278b0 msgr2=0x55d3c6d213a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:06.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.972+0000 7f4e85400640 1 --2- 172.21.15.31:0/1531035970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3c6d278b0 0x55d3c6d213a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.972+0000 7f4e85400640 1 -- 172.21.15.31:0/1531035970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3c6d30de0 msgr2=0x55d3c6d306b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:06.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.972+0000 7f4e85400640 1 --2- 172.21.15.31:0/1531035970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3c6d30de0 0x55d3c6d306b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.972+0000 7f4e85400640 1 -- 172.21.15.31:0/1531035970 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3c6d28000 con 0x55d3c6d28260 2024-03-21T10:48:06.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.972+0000 7f4e86800640 1 --2- 172.21.15.31:0/1531035970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3c6d30de0 0x55d3c6d306b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:06.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.975+0000 7f4e85400640 1 --2- 172.21.15.31:0/1531035970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d28260 0x55d3c6d30170 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x55d3c6d22ff0 tx=0x7f4e780127c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84b020ddfbbd6dd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:06.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.975+0000 7f4e76600640 1 -- 172.21.15.31:0/1531035970 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e78004030 con 0x55d3c6d28260 2024-03-21T10:48:06.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.975+0000 7f4e76600640 1 -- 172.21.15.31:0/1531035970 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e780041d0 con 0x55d3c6d28260 2024-03-21T10:48:06.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.975+0000 7f4e76600640 1 -- 172.21.15.31:0/1531035970 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7801b730 con 0x55d3c6d28260 2024-03-21T10:48:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.976+0000 7f4e8f77f740 1 -- 172.21.15.31:0/1531035970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d28260 msgr2=0x55d3c6d30170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.977+0000 7f4e8f77f740 1 --2- 172.21.15.31:0/1531035970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d28260 0x55d3c6d30170 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x55d3c6d22ff0 tx=0x7f4e780127c0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.977+0000 7f4e8f77f740 1 -- 172.21.15.31:0/1531035970 shutdown_connections 2024-03-21T10:48:06.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.977+0000 7f4e8f77f740 1 --2- 172.21.15.31:0/1531035970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3c6d30de0 0x55d3c6d306b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.977+0000 7f4e8f77f740 1 --2- 172.21.15.31:0/1531035970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d28260 0x55d3c6d30170 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.977+0000 7f4e8f77f740 1 --2- 172.21.15.31:0/1531035970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3c6d278b0 0x55d3c6d213a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.977+0000 7f4e8f77f740 1 -- 172.21.15.31:0/1531035970 >> 172.21.15.31:0/1531035970 conn(0x55d3c6d21f80 msgr2=0x55d3c6d28f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:06.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.978+0000 7f4e8f77f740 1 -- 172.21.15.31:0/1531035970 shutdown_connections 2024-03-21T10:48:06.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.978+0000 7f4e8f77f740 1 -- 172.21.15.31:0/1531035970 wait complete. 2024-03-21T10:48:06.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.979+0000 7f4e8f77f740 1 -- start start 2024-03-21T10:48:06.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.979+0000 7f4e8f77f740 1 Processor -- start 2024-03-21T10:48:06.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.980+0000 7f4e8f77f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3c6d278b0 0x55d3c6d38f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.981+0000 7f4e8f77f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3c6d28260 0x55d3c6d34270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.981+0000 7f4e8f77f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d30de0 0x55d3c6d347b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.981+0000 7f4e8f77f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6400e630 con 0x55d3c6d278b0 2024-03-21T10:48:06.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.981+0000 7f4e8f77f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6400f910 con 0x55d3c6d28260 2024-03-21T10:48:06.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.981+0000 7f4e8f77f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6400ec50 con 0x55d3c6d30de0 2024-03-21T10:48:06.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.981+0000 7f4e86800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d30de0 0x55d3c6d347b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.981+0000 7f4e86800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d30de0 0x55d3c6d347b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:48514/0 (socket says 172.21.15.31:48514) 2024-03-21T10:48:06.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.981+0000 7f4e86800640 1 -- 172.21.15.31:0/1821027500 learned_addr learned my addr 172.21.15.31:0/1821027500 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:06.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.981+0000 7f4e8f77f740 10 client.0 _ll_register_callbacks cb 0x7f4e80005e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:06.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.982+0000 7f4e85400640 1 --2- 172.21.15.31:0/1821027500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3c6d28260 0x55d3c6d34270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.982+0000 7f4e86800640 1 -- 172.21.15.31:0/1821027500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3c6d28260 msgr2=0x55d3c6d34270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:06.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.982+0000 7f4e86800640 1 --2- 172.21.15.31:0/1821027500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3c6d28260 0x55d3c6d34270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.982+0000 7f4e86800640 1 -- 172.21.15.31:0/1821027500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3c6d278b0 msgr2=0x55d3c6d38f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:06.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.982+0000 7f4e86800640 1 --2- 172.21.15.31:0/1821027500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3c6d278b0 0x55d3c6d38f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.982+0000 7f4e86800640 1 -- 172.21.15.31:0/1821027500 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d3c6d34fb0 con 0x55d3c6d30de0 2024-03-21T10:48:06.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.983+0000 7f4e86800640 1 --2- 172.21.15.31:0/1821027500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d30de0 0x55d3c6d347b0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f4e80098690 tx=0x7f4e80097c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cee9ecdd03d6f39d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:06.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.983+0000 7f4e6b400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:48:06.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.983+0000 7f4e6b400640 1 -- 172.21.15.31:0/1821027500 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8001f070 con 0x55d3c6d30de0 2024-03-21T10:48:06.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.984+0000 7f4e6b400640 1 -- 172.21.15.31:0/1821027500 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f4e8009ad90 con 0x55d3c6d30de0 2024-03-21T10:48:06.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.984+0000 7f4e8f77f740 1 -- 172.21.15.31:0/1821027500 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4e780041d0 con 0x55d3c6d30de0 2024-03-21T10:48:06.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.985+0000 7f4e6b400640 1 -- 172.21.15.31:0/1821027500 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e800ba040 con 0x55d3c6d30de0 2024-03-21T10:48:06.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.985+0000 7f4e8f77f740 10 client.27931 fetch_fsmap learned FSMap version 28 2024-03-21T10:48:06.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.985+0000 7f4e8f77f740 1 -- 172.21.15.31:0/1821027500 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d3c6d9c9a0 con 0x55d3c6d30de0 2024-03-21T10:48:06.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.985+0000 7f4e6b400640 1 -- 172.21.15.31:0/1821027500 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e780041d0 con 0x55d3c6d30de0 2024-03-21T10:48:06.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.986+0000 7f4e6b400640 1 -- 172.21.15.31:0/1821027500 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f4e800ac020 con 0x55d3c6d30de0 2024-03-21T10:48:06.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.986+0000 7f4e8f77f740 10 client.27931 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:48:06.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.986+0000 7f4e8f77f740 10 client.27931 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:48:06.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.986+0000 7f4e8f77f740 20 client.27931 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:06.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.986+0000 7f4e8f77f740 1 --2- 172.21.15.31:0/1821027500 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55d3c6d9d0a0 0x55d3c6d9f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:06.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.986+0000 7f4e8f77f740 4 client.27931 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:48:06.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.986+0000 7f4e8f77f740 1 -- 172.21.15.31:0/1821027500 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d3c6c92b50 con 0x55d3c6d9d0a0 2024-03-21T10:48:06.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.987+0000 7f4e85e00640 1 --2- 172.21.15.31:0/1821027500 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55d3c6d9d0a0 0x55d3c6d9f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:06.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.991+0000 7f4e85e00640 1 --2- 172.21.15.31:0/1821027500 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55d3c6d9d0a0 0x55d3c6d9f4c0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ec214b754b40732a server_cookie=a30460c75ccc82ed in_seq=0 out_seq=0 2024-03-21T10:48:06.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.991+0000 7f4e6b400640 10 client.27931 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:48:06.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.993+0000 7f4e6b400640 1 -- 172.21.15.31:0/1821027500 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d3c6c92b50 con 0x55d3c6d9d0a0 2024-03-21T10:48:06.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.993+0000 7f4e6b400640 10 client.27931 handle_command_reply: tid=0 2024-03-21T10:48:06.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.993+0000 7f4e8f77f740 1 client.27931 shutdown 2024-03-21T10:48:06.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.996+0000 7f4e8f77f740 1 -- 172.21.15.31:0/1821027500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d30de0 msgr2=0x55d3c6d347b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:06.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.996+0000 7f4e8f77f740 1 --2- 172.21.15.31:0/1821027500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d30de0 0x55d3c6d347b0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f4e80098690 tx=0x7f4e80097c40 comp rx=0 tx=0).stop 2024-03-21T10:48:06.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.998+0000 7f4e8f77f740 1 -- 172.21.15.31:0/1821027500 shutdown_connections 2024-03-21T10:48:06.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.998+0000 7f4e8f77f740 1 --2- 172.21.15.31:0/1821027500 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x55d3c6d9d0a0 0x55d3c6d9f4c0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.998+0000 7f4e8f77f740 1 --2- 172.21.15.31:0/1821027500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3c6d30de0 0x55d3c6d347b0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.998+0000 7f4e8f77f740 1 --2- 172.21.15.31:0/1821027500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d3c6d28260 0x55d3c6d34270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.998+0000 7f4e8f77f740 1 --2- 172.21.15.31:0/1821027500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3c6d278b0 0x55d3c6d38f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.998+0000 7f4e8f77f740 1 -- 172.21.15.31:0/1821027500 >> 172.21.15.31:0/1821027500 conn(0x55d3c6d21f80 msgr2=0x55d3c6d23280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:06.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.998+0000 7f4e6b400640 0 client.27931 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:48:06.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.998+0000 7f4e8f77f740 1 -- 172.21.15.31:0/1821027500 shutdown_connections 2024-03-21T10:48:06.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.998+0000 7f4e8f77f740 1 -- 172.21.15.31:0/1821027500 wait complete. 2024-03-21T10:48:06.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:06.999+0000 7f4e8f77f740 20 client.27931 trim_cache size 0 max 16384 2024-03-21T10:48:06.997 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:48:06.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.001+0000 7f4e87200640 1 -- 172.21.15.31:0/4091030972 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4e6c0a3220 msgr2=0x7f4e6c0a56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:06.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.001+0000 7f4e87200640 1 --2- 172.21.15.31:0/4091030972 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4e6c0a3220 0x7f4e6c0a56e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4e70006010 tx=0x7f4e70000bc0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.001+0000 7f4e87200640 1 -- 172.21.15.31:0/4091030972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e8810fc90 msgr2=0x7f4e881a68c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:06.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.001+0000 7f4e87200640 1 --2- 172.21.15.31:0/4091030972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e8810fc90 0x7f4e881a68c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f4e8000eaa0 tx=0x7f4e8000ef60 comp rx=0 tx=0).stop 2024-03-21T10:48:06.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.001+0000 7f4e87200640 1 -- 172.21.15.31:0/4091030972 shutdown_connections 2024-03-21T10:48:06.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.001+0000 7f4e87200640 1 --2- 172.21.15.31:0/4091030972 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4e6c0a3220 0x7f4e6c0a56e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.001+0000 7f4e87200640 1 --2- 172.21.15.31:0/4091030972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e8810fc90 0x7f4e881a68c0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.001+0000 7f4e87200640 1 --2- 172.21.15.31:0/4091030972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e8810e310 0x7f4e881acc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.001+0000 7f4e87200640 1 --2- 172.21.15.31:0/4091030972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e8810d780 0x7f4e881ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:06.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.001+0000 7f4e87200640 1 -- 172.21.15.31:0/4091030972 >> 172.21.15.31:0/4091030972 conn(0x7f4e8806c580 msgr2=0x7f4e880709e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:06.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.002+0000 7f4e87200640 1 -- 172.21.15.31:0/4091030972 shutdown_connections 2024-03-21T10:48:06.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.003+0000 7f4e87200640 1 -- 172.21.15.31:0/4091030972 wait complete. 2024-03-21T10:48:07.032 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:48:07.032 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-21T10:48:07.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.378+0000 7f235f800640 1 -- 172.21.15.31:0/4041436680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2358075cb0 msgr2=0x7f2358076090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:07.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.378+0000 7f235f800640 1 --2- 172.21.15.31:0/4041436680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2358075cb0 0x7f2358076090 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f2350009ca0 tx=0x7f235002c120 comp rx=0 tx=0).stop 2024-03-21T10:48:07.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.379+0000 7f235f800640 1 -- 172.21.15.31:0/4041436680 shutdown_connections 2024-03-21T10:48:07.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.379+0000 7f235f800640 1 --2- 172.21.15.31:0/4041436680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2358112400 0x7f23581147f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.379+0000 7f235f800640 1 --2- 172.21.15.31:0/4041436680 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23580765d0 0x7f2358111ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.379+0000 7f235f800640 1 --2- 172.21.15.31:0/4041436680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2358075cb0 0x7f2358076090 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.379+0000 7f235f800640 1 -- 172.21.15.31:0/4041436680 >> 172.21.15.31:0/4041436680 conn(0x7f23580ff2f0 msgr2=0x7f2358101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:07.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.379+0000 7f235f800640 1 -- 172.21.15.31:0/4041436680 shutdown_connections 2024-03-21T10:48:07.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.379+0000 7f235f800640 1 -- 172.21.15.31:0/4041436680 wait complete. 2024-03-21T10:48:07.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.381+0000 7f235f800640 1 Processor -- start 2024-03-21T10:48:07.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.381+0000 7f235f800640 1 -- start start 2024-03-21T10:48:07.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.381+0000 7f235f800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2358075cb0 0x7f2358072ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:07.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.381+0000 7f235f800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23580765d0 0x7f235806c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:07.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.382+0000 7f235f800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2358112400 0x7f235806c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:07.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.382+0000 7f2357e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23580765d0 0x7f235806c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:07.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.382+0000 7f235ce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2358075cb0 0x7f2358072ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:07.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.382+0000 7f235f800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23581180b0 con 0x7f2358075cb0 2024-03-21T10:48:07.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.382+0000 7f235ce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2358075cb0 0x7f2358072ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46088/0 (socket says 172.21.15.31:46088) 2024-03-21T10:48:07.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.382+0000 7f235d800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2358112400 0x7f235806c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:07.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.382+0000 7f235ce00640 1 -- 172.21.15.31:0/3125467969 learned_addr learned my addr 172.21.15.31:0/3125467969 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:07.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.382+0000 7f235f800640 1 -- 172.21.15.31:0/3125467969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2358117f30 con 0x7f2358112400 2024-03-21T10:48:07.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.382+0000 7f235f800640 1 -- 172.21.15.31:0/3125467969 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2358118230 con 0x7f23580765d0 2024-03-21T10:48:07.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.383+0000 7f235d800640 1 -- 172.21.15.31:0/3125467969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23580765d0 msgr2=0x7f235806c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:07.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.383+0000 7f235d800640 1 --2- 172.21.15.31:0/3125467969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23580765d0 0x7f235806c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.383+0000 7f235d800640 1 -- 172.21.15.31:0/3125467969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2358075cb0 msgr2=0x7f2358072ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:07.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.383+0000 7f235d800640 1 --2- 172.21.15.31:0/3125467969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2358075cb0 0x7f2358072ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.383+0000 7f235d800640 1 -- 172.21.15.31:0/3125467969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f235806ccb0 con 0x7f2358112400 2024-03-21T10:48:07.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.383+0000 7f235d800640 1 --2- 172.21.15.31:0/3125467969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2358112400 0x7f235806c5b0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f234400d650 tx=0x7f234400db10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:07.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.383+0000 7f2357e00640 1 --2- 172.21.15.31:0/3125467969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23580765d0 0x7f235806c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:48:07.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.383+0000 7f2355600640 1 -- 172.21.15.31:0/3125467969 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2344017070 con 0x7f2358112400 2024-03-21T10:48:07.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.384+0000 7f2355600640 1 -- 172.21.15.31:0/3125467969 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2344009820 con 0x7f2358112400 2024-03-21T10:48:07.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.384+0000 7f2355600640 1 -- 172.21.15.31:0/3125467969 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2344004d00 con 0x7f2358112400 2024-03-21T10:48:07.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.384+0000 7f235f800640 1 -- 172.21.15.31:0/3125467969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2358070b10 con 0x7f2358112400 2024-03-21T10:48:07.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.384+0000 7f235f800640 1 -- 172.21.15.31:0/3125467969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2358071010 con 0x7f2358112400 2024-03-21T10:48:07.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.394+0000 7f2362589740 1 Processor -- start 2024-03-21T10:48:07.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.396+0000 7f2355600640 1 -- 172.21.15.31:0/3125467969 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f23440099d0 con 0x7f2358112400 2024-03-21T10:48:07.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.397+0000 7f2362589740 1 -- start start 2024-03-21T10:48:07.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.397+0000 7f2362589740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595e920b6d0 0x5595e920bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:07.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.397+0000 7f2362589740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595e920c080 0x5595e9214630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:07.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.397+0000 7f2362589740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595e9214c00 0x5595e9217050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:07.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.397+0000 7f2362589740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595e916f2c0 con 0x5595e9214c00 2024-03-21T10:48:07.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.397+0000 7f2362589740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595e9117c90 con 0x5595e920c080 2024-03-21T10:48:07.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.397+0000 7f2362589740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595e916b360 con 0x5595e920b6d0 2024-03-21T10:48:07.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.398+0000 7f2355600640 1 --2- 172.21.15.31:0/3125467969 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f233c0a3090 0x7f233c0a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:07.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.398+0000 7f2355600640 1 -- 172.21.15.31:0/3125467969 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f23440cf4e0 con 0x7f2358112400 2024-03-21T10:48:07.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.398+0000 7f235d800640 1 --2- 172.21.15.31:0/3125467969 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f233c0a3090 0x7f233c0a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:07.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.399+0000 7f235ce00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595e920c080 0x5595e9214630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:07.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.399+0000 7f235ce00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595e920c080 0x5595e9214630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:40668/0 (socket says 172.21.15.31:40668) 2024-03-21T10:48:07.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.399+0000 7f235ce00640 1 -- 172.21.15.31:0/3758064272 learned_addr learned my addr 172.21.15.31:0/3758064272 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:07.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.399+0000 7f2357e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595e920b6d0 0x5595e920bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:07.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.399+0000 7f235d800640 1 --2- 172.21.15.31:0/3125467969 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f233c0a3090 0x7f233c0a5550 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f23440cc990 tx=0x7f2344097160 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:07.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.400+0000 7f2357e00640 1 --2- 172.21.15.31:0/3758064272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595e9214c00 0x5595e9217050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:07.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.401+0000 7f235ce00640 1 -- 172.21.15.31:0/3758064272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595e920b6d0 msgr2=0x5595e920bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:07.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.401+0000 7f235ce00640 1 --2- 172.21.15.31:0/3758064272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595e920b6d0 0x5595e920bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.401+0000 7f235ce00640 1 -- 172.21.15.31:0/3758064272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595e9214c00 msgr2=0x5595e9217050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:07.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.401+0000 7f235ce00640 1 --2- 172.21.15.31:0/3758064272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595e9214c00 0x5595e9217050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.401+0000 7f235ce00640 1 -- 172.21.15.31:0/3758064272 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595e913c080 con 0x5595e920c080 2024-03-21T10:48:07.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.402+0000 7f235ce00640 1 --2- 172.21.15.31:0/3758064272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595e920c080 0x5595e9214630 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f2350037040 tx=0x7f23500099d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccaea9e30b6fa70f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:07.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.402+0000 7f234e000640 1 -- 172.21.15.31:0/3758064272 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2350043070 con 0x5595e920c080 2024-03-21T10:48:07.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.404+0000 7f234e000640 1 -- 172.21.15.31:0/3758064272 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2350003970 con 0x5595e920c080 2024-03-21T10:48:07.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.404+0000 7f234e000640 1 -- 172.21.15.31:0/3758064272 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2350039040 con 0x5595e920c080 2024-03-21T10:48:07.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.405+0000 7f2362589740 1 -- 172.21.15.31:0/3758064272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595e920c080 msgr2=0x5595e9214630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:07.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.405+0000 7f2362589740 1 --2- 172.21.15.31:0/3758064272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595e920c080 0x5595e9214630 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f2350037040 tx=0x7f23500099d0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.406+0000 7f2362589740 1 -- 172.21.15.31:0/3758064272 shutdown_connections 2024-03-21T10:48:07.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.406+0000 7f2362589740 1 --2- 172.21.15.31:0/3758064272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595e9214c00 0x5595e9217050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.406+0000 7f2362589740 1 --2- 172.21.15.31:0/3758064272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595e920c080 0x5595e9214630 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.407+0000 7f2362589740 1 --2- 172.21.15.31:0/3758064272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595e920b6d0 0x5595e920bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.407+0000 7f2362589740 1 -- 172.21.15.31:0/3758064272 >> 172.21.15.31:0/3758064272 conn(0x5595e9205da0 msgr2=0x5595e9206180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:07.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.407+0000 7f2362589740 1 -- 172.21.15.31:0/3758064272 shutdown_connections 2024-03-21T10:48:07.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.407+0000 7f2362589740 1 -- 172.21.15.31:0/3758064272 wait complete. 2024-03-21T10:48:07.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.408+0000 7f2362589740 1 -- start start 2024-03-21T10:48:07.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.408+0000 7f2362589740 1 Processor -- start 2024-03-21T10:48:07.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.409+0000 7f2362589740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595e920b6d0 0x5595e9214500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:07.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.409+0000 7f2362589740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595e920c080 0x5595e9211890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:07.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.409+0000 7f2362589740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595e9214c00 0x5595e9211e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:07.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.409+0000 7f2362589740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595e9217b60 con 0x5595e9214c00 2024-03-21T10:48:07.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.409+0000 7f2362589740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595e912b0d0 con 0x5595e920c080 2024-03-21T10:48:07.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.409+0000 7f2362589740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595e9217ce0 con 0x5595e920b6d0 2024-03-21T10:48:07.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.409+0000 7f2362589740 10 client.0 _ll_register_callbacks cb 0x5595e912fe80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:07.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.410+0000 7f2357e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595e920b6d0 0x5595e9214500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:07.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.411+0000 7f2357e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595e9214c00 0x5595e9211e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:07.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.411+0000 7f2357e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595e9214c00 0x5595e9211e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46104/0 (socket says 172.21.15.31:46104) 2024-03-21T10:48:07.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.411+0000 7f2357e00640 1 -- 172.21.15.31:0/3736886020 learned_addr learned my addr 172.21.15.31:0/3736886020 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:07.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.411+0000 7f235ce00640 1 --2- 172.21.15.31:0/3736886020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595e920c080 0x5595e9211890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:07.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.411+0000 7f2357e00640 1 -- 172.21.15.31:0/3736886020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595e920c080 msgr2=0x5595e9211890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:07.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.411+0000 7f2357e00640 1 --2- 172.21.15.31:0/3736886020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595e920c080 0x5595e9211890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.411+0000 7f2357e00640 1 -- 172.21.15.31:0/3736886020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595e9214c00 msgr2=0x5595e9211e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:07.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.411+0000 7f2357e00640 1 --2- 172.21.15.31:0/3736886020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595e9214c00 0x5595e9211e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.411+0000 7f2357e00640 1 -- 172.21.15.31:0/3736886020 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5595e8e833f0 con 0x5595e920b6d0 2024-03-21T10:48:07.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.412+0000 7f2357e00640 1 --2- 172.21.15.31:0/3736886020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595e920b6d0 0x5595e9214500 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f2348018f70 tx=0x7f2348006ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6f0a991c85a272b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:07.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.412+0000 7f2342a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:48:07.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.412+0000 7f2342a00640 1 -- 172.21.15.31:0/3736886020 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f234800f040 con 0x5595e920b6d0 2024-03-21T10:48:07.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.412+0000 7f2342a00640 1 -- 172.21.15.31:0/3736886020 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f2348004020 con 0x5595e920b6d0 2024-03-21T10:48:07.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.413+0000 7f2362589740 1 -- 172.21.15.31:0/3736886020 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2350003970 con 0x5595e920b6d0 2024-03-21T10:48:07.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.415+0000 7f2342a00640 1 -- 172.21.15.31:0/3736886020 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2348018540 con 0x5595e920b6d0 2024-03-21T10:48:07.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.415+0000 7f2362589740 10 client.27937 fetch_fsmap learned FSMap version 28 2024-03-21T10:48:07.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.415+0000 7f2362589740 1 -- 172.21.15.31:0/3736886020 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5595e8f4d7d0 con 0x5595e920b6d0 2024-03-21T10:48:07.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.416+0000 7f2342a00640 1 -- 172.21.15.31:0/3736886020 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2350003970 con 0x5595e920b6d0 2024-03-21T10:48:07.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.416+0000 7f2342a00640 1 -- 172.21.15.31:0/3736886020 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f2348015070 con 0x5595e920b6d0 2024-03-21T10:48:07.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.416+0000 7f2362589740 10 client.27937 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:48:07.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.416+0000 7f2362589740 10 client.27937 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:48:07.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.416+0000 7f2362589740 20 client.27937 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:07.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.416+0000 7f2362589740 1 --2- 172.21.15.31:0/3736886020 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5595e9224af0 0x5595e9226ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:07.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.417+0000 7f2362589740 4 client.27937 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:48:07.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.417+0000 7f2362589740 1 -- 172.21.15.31:0/3736886020 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5595e917c800 con 0x5595e9224af0 2024-03-21T10:48:07.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.418+0000 7f235ce00640 1 --2- 172.21.15.31:0/3736886020 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5595e9224af0 0x5595e9226ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:07.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.419+0000 7f235ce00640 1 --2- 172.21.15.31:0/3736886020 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5595e9224af0 0x5595e9226ee0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9b69b2e80346237 server_cookie=aca7a15bb00976f in_seq=0 out_seq=0 2024-03-21T10:48:07.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.419+0000 7f2342a00640 10 client.27937 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:07.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.420+0000 7f2342a00640 1 -- 172.21.15.31:0/3736886020 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5595e917c800 con 0x5595e9224af0 2024-03-21T10:48:07.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.420+0000 7f2342a00640 10 client.27937 handle_command_reply: tid=0 2024-03-21T10:48:07.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.420+0000 7f2362589740 1 client.27937 shutdown 2024-03-21T10:48:07.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.423+0000 7f2362589740 1 -- 172.21.15.31:0/3736886020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595e920b6d0 msgr2=0x5595e9214500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:07.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.423+0000 7f2362589740 1 --2- 172.21.15.31:0/3736886020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595e920b6d0 0x5595e9214500 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f2348018f70 tx=0x7f2348006ca0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.426+0000 7f2362589740 1 -- 172.21.15.31:0/3736886020 shutdown_connections 2024-03-21T10:48:07.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.426+0000 7f2362589740 1 --2- 172.21.15.31:0/3736886020 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5595e9224af0 0x5595e9226ee0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.426+0000 7f2362589740 1 --2- 172.21.15.31:0/3736886020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595e9214c00 0x5595e9211e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.426+0000 7f2362589740 1 --2- 172.21.15.31:0/3736886020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595e920c080 0x5595e9211890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.426+0000 7f2362589740 1 --2- 172.21.15.31:0/3736886020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595e920b6d0 0x5595e9214500 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.426+0000 7f2342a00640 0 client.27937 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:07.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.426+0000 7f2362589740 1 -- 172.21.15.31:0/3736886020 >> 172.21.15.31:0/3736886020 conn(0x5595e9205da0 msgr2=0x5595e9207af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:07.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.427+0000 7f2362589740 1 -- 172.21.15.31:0/3736886020 shutdown_connections 2024-03-21T10:48:07.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.428+0000 7f2362589740 1 -- 172.21.15.31:0/3736886020 wait complete. 2024-03-21T10:48:07.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.428+0000 7f2362589740 20 client.27937 trim_cache size 0 max 16384 2024-03-21T10:48:07.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.463+0000 7f2362589740 1 Processor -- start 2024-03-21T10:48:07.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.463+0000 7f2362589740 1 -- start start 2024-03-21T10:48:07.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.464+0000 7f2362589740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595e920b6d0 0x5595e9213fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:07.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.464+0000 7f2362589740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595e920c080 0x5595e9214500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:07.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.464+0000 7f2362589740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595e9214c00 0x5595e920eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:07.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.464+0000 7f2362589740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23581bfee0 con 0x5595e9214c00 2024-03-21T10:48:07.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.464+0000 7f2362589740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23581bd9e0 con 0x5595e920b6d0 2024-03-21T10:48:07.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.464+0000 7f2362589740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23581bbaa0 con 0x5595e920c080 2024-03-21T10:48:07.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.465+0000 7f235ce00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595e920c080 0x5595e9214500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:07.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.465+0000 7f2357e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595e920b6d0 0x5595e9213fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:07.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.465+0000 7f235ce00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595e920c080 0x5595e9214500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:48562/0 (socket says 172.21.15.31:48562) 2024-03-21T10:48:07.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.465+0000 7f235ce00640 1 -- 172.21.15.31:0/4040274485 learned_addr learned my addr 172.21.15.31:0/4040274485 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:07.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.465+0000 7f2357e00640 1 -- 172.21.15.31:0/4040274485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595e920c080 msgr2=0x5595e9214500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:07.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.465+0000 7f2357e00640 1 --2- 172.21.15.31:0/4040274485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595e920c080 0x5595e9214500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.465+0000 7f2357e00640 1 -- 172.21.15.31:0/4040274485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595e9214c00 msgr2=0x5595e920eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:07.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.465+0000 7f2357e00640 1 --2- 172.21.15.31:0/4040274485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595e9214c00 0x5595e920eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.465+0000 7f2357e00640 1 -- 172.21.15.31:0/4040274485 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595e9212180 con 0x5595e920b6d0 2024-03-21T10:48:07.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.466+0000 7f2357e00640 1 --2- 172.21.15.31:0/4040274485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595e920b6d0 0x5595e9213fc0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f2348006c70 tx=0x7f2348004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48706906f63e41ed server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:07.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.466+0000 7f234d600640 1 -- 172.21.15.31:0/4040274485 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2348018ab0 con 0x5595e920b6d0 2024-03-21T10:48:07.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.467+0000 7f234d600640 1 -- 172.21.15.31:0/4040274485 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2348013730 con 0x5595e920b6d0 2024-03-21T10:48:07.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.467+0000 7f234d600640 1 -- 172.21.15.31:0/4040274485 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2348003910 con 0x5595e920b6d0 2024-03-21T10:48:07.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.468+0000 7f2362589740 1 -- 172.21.15.31:0/4040274485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595e920b6d0 msgr2=0x5595e9213fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:07.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.468+0000 7f2362589740 1 --2- 172.21.15.31:0/4040274485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595e920b6d0 0x5595e9213fc0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f2348006c70 tx=0x7f2348004770 comp rx=0 tx=0).stop 2024-03-21T10:48:07.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.468+0000 7f2362589740 1 -- 172.21.15.31:0/4040274485 shutdown_connections 2024-03-21T10:48:07.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.468+0000 7f2362589740 1 --2- 172.21.15.31:0/4040274485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595e9214c00 0x5595e920eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.468+0000 7f2362589740 1 --2- 172.21.15.31:0/4040274485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595e920c080 0x5595e9214500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.468+0000 7f2362589740 1 --2- 172.21.15.31:0/4040274485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595e920b6d0 0x5595e9213fc0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.468+0000 7f2362589740 1 -- 172.21.15.31:0/4040274485 >> 172.21.15.31:0/4040274485 conn(0x5595e9205da0 msgr2=0x5595e9204eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:07.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.468+0000 7f2362589740 1 -- 172.21.15.31:0/4040274485 shutdown_connections 2024-03-21T10:48:07.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.468+0000 7f2362589740 1 -- 172.21.15.31:0/4040274485 wait complete. 2024-03-21T10:48:07.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.471+0000 7f2362589740 1 -- start start 2024-03-21T10:48:07.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.471+0000 7f2362589740 1 Processor -- start 2024-03-21T10:48:07.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.472+0000 7f2362589740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595e920b6d0 0x5595e921ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:07.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.473+0000 7f2362589740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595e920c080 0x5595e921b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:07.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.473+0000 7f2362589740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595e9214c00 0x5595e927f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:07.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.473+0000 7f2362589740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23581bc0a0 con 0x5595e9214c00 2024-03-21T10:48:07.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.473+0000 7f2362589740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23581bd380 con 0x5595e920b6d0 2024-03-21T10:48:07.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.473+0000 7f2362589740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23581bc6c0 con 0x5595e920c080 2024-03-21T10:48:07.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.473+0000 7f2357e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595e920b6d0 0x5595e921ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:07.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.473+0000 7f2362589740 10 client.0 _ll_register_callbacks cb 0x7f2348018390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:07.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.473+0000 7f2357e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595e920b6d0 0x5595e921ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:40706/0 (socket says 172.21.15.31:40706) 2024-03-21T10:48:07.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.474+0000 7f2357e00640 1 -- 172.21.15.31:0/1467028543 learned_addr learned my addr 172.21.15.31:0/1467028543 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:07.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.474+0000 7f235ce00640 1 --2- 172.21.15.31:0/1467028543 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595e920c080 0x5595e921b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:07.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.474+0000 7f2357e00640 1 --2- 172.21.15.31:0/1467028543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595e9214c00 0x5595e927f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:07.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.475+0000 7f235ce00640 1 -- 172.21.15.31:0/1467028543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595e920b6d0 msgr2=0x5595e921ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:07.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.475+0000 7f235ce00640 1 --2- 172.21.15.31:0/1467028543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595e920b6d0 0x5595e921ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.475+0000 7f235ce00640 1 -- 172.21.15.31:0/1467028543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595e9214c00 msgr2=0x5595e927f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:07.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.475+0000 7f235ce00640 1 --2- 172.21.15.31:0/1467028543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595e9214c00 0x5595e927f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.476+0000 7f235ce00640 1 -- 172.21.15.31:0/1467028543 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5595e927f970 con 0x5595e920c080 2024-03-21T10:48:07.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.480+0000 7f235ce00640 1 --2- 172.21.15.31:0/1467028543 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595e920c080 0x5595e921b0c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f2350004af0 tx=0x7f2350009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c81834fa3a35b06 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:07.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.480+0000 7f2342a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:48:07.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.480+0000 7f2342a00640 1 -- 172.21.15.31:0/1467028543 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2350054060 con 0x5595e920c080 2024-03-21T10:48:07.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.481+0000 7f2342a00640 1 -- 172.21.15.31:0/1467028543 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f2350032800 con 0x5595e920c080 2024-03-21T10:48:07.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.481+0000 7f2362589740 1 -- 172.21.15.31:0/1467028543 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2348013730 con 0x5595e920c080 2024-03-21T10:48:07.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.482+0000 7f2342a00640 1 -- 172.21.15.31:0/1467028543 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2350039040 con 0x5595e920c080 2024-03-21T10:48:07.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.482+0000 7f2362589740 10 client.27943 fetch_fsmap learned FSMap version 28 2024-03-21T10:48:07.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.482+0000 7f2362589740 1 -- 172.21.15.31:0/1467028543 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5595e9280ca0 con 0x5595e920c080 2024-03-21T10:48:07.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.484+0000 7f2342a00640 1 -- 172.21.15.31:0/1467028543 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2348013730 con 0x5595e920c080 2024-03-21T10:48:07.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.484+0000 7f2342a00640 1 -- 172.21.15.31:0/1467028543 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f2350004490 con 0x5595e920c080 2024-03-21T10:48:07.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.484+0000 7f2362589740 10 client.27943 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:48:07.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.484+0000 7f2362589740 10 client.27943 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:48:07.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.484+0000 7f2362589740 20 client.27943 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:07.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.484+0000 7f2362589740 1 --2- 172.21.15.31:0/1467028543 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5595e9280f80 0x5595e9283370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:07.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.484+0000 7f2362589740 4 client.27943 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:48:07.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.484+0000 7f2362589740 1 -- 172.21.15.31:0/1467028543 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5595e9176b50 con 0x5595e9280f80 2024-03-21T10:48:07.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.485+0000 7f2357e00640 1 --2- 172.21.15.31:0/1467028543 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5595e9280f80 0x5595e9283370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:07.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.485+0000 7f2357e00640 1 --2- 172.21.15.31:0/1467028543 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5595e9280f80 0x5595e9283370 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a0655ed43cec218 server_cookie=740d7aff71d80ba4 in_seq=0 out_seq=0 2024-03-21T10:48:07.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.486+0000 7f2342a00640 10 client.27943 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:07.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.489+0000 7f2342a00640 1 -- 172.21.15.31:0/1467028543 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5595e9176b50 con 0x5595e9280f80 2024-03-21T10:48:07.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.489+0000 7f2342a00640 10 client.27943 handle_command_reply: tid=0 2024-03-21T10:48:07.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.489+0000 7f2362589740 1 client.27943 shutdown 2024-03-21T10:48:07.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.490+0000 7f2362589740 1 -- 172.21.15.31:0/1467028543 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595e920c080 msgr2=0x5595e921b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:07.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.490+0000 7f2362589740 1 --2- 172.21.15.31:0/1467028543 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595e920c080 0x5595e921b0c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f2350004af0 tx=0x7f2350009780 comp rx=0 tx=0).stop 2024-03-21T10:48:07.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.490+0000 7f2362589740 1 -- 172.21.15.31:0/1467028543 shutdown_connections 2024-03-21T10:48:07.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.490+0000 7f2362589740 1 --2- 172.21.15.31:0/1467028543 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5595e9280f80 0x5595e9283370 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.490+0000 7f2362589740 1 --2- 172.21.15.31:0/1467028543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595e9214c00 0x5595e927f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.490+0000 7f2362589740 1 --2- 172.21.15.31:0/1467028543 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595e920c080 0x5595e921b0c0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.490+0000 7f2362589740 1 --2- 172.21.15.31:0/1467028543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595e920b6d0 0x5595e921ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.490+0000 7f2362589740 1 -- 172.21.15.31:0/1467028543 >> 172.21.15.31:0/1467028543 conn(0x5595e9205da0 msgr2=0x5595e9204620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:07.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.490+0000 7f2342a00640 0 client.27943 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:07.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.490+0000 7f2362589740 1 -- 172.21.15.31:0/1467028543 shutdown_connections 2024-03-21T10:48:07.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.490+0000 7f2362589740 1 -- 172.21.15.31:0/1467028543 wait complete. 2024-03-21T10:48:07.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.491+0000 7f2362589740 20 client.27943 trim_cache size 0 max 16384 2024-03-21T10:48:07.488 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:48:07.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.492+0000 7f235f800640 1 -- 172.21.15.31:0/3125467969 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f233c0a3090 msgr2=0x7f233c0a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:07.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.492+0000 7f235f800640 1 --2- 172.21.15.31:0/3125467969 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f233c0a3090 0x7f233c0a5550 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f23440cc990 tx=0x7f2344097160 comp rx=0 tx=0).stop 2024-03-21T10:48:07.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.492+0000 7f235f800640 1 -- 172.21.15.31:0/3125467969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2358112400 msgr2=0x7f235806c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:07.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.492+0000 7f235f800640 1 --2- 172.21.15.31:0/3125467969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2358112400 0x7f235806c5b0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f234400d650 tx=0x7f234400db10 comp rx=0 tx=0).stop 2024-03-21T10:48:07.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.492+0000 7f235f800640 1 -- 172.21.15.31:0/3125467969 shutdown_connections 2024-03-21T10:48:07.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.492+0000 7f235f800640 1 --2- 172.21.15.31:0/3125467969 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f233c0a3090 0x7f233c0a5550 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.492+0000 7f235f800640 1 --2- 172.21.15.31:0/3125467969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2358112400 0x7f235806c5b0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.492+0000 7f235f800640 1 --2- 172.21.15.31:0/3125467969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23580765d0 0x7f235806c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.492+0000 7f235f800640 1 --2- 172.21.15.31:0/3125467969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2358075cb0 0x7f2358072ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:07.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.492+0000 7f235f800640 1 -- 172.21.15.31:0/3125467969 >> 172.21.15.31:0/3125467969 conn(0x7f23580ff2f0 msgr2=0x7f2358113b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:07.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.493+0000 7f235f800640 1 -- 172.21.15.31:0/3125467969 shutdown_connections 2024-03-21T10:48:07.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:07.493+0000 7f235f800640 1 -- 172.21.15.31:0/3125467969 wait complete. 2024-03-21T10:48:07.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:07 smithi040 ceph-mon[36154]: pgmap v295: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 53 KiB/s wr, 25 op/s 2024-03-21T10:48:07.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:07 smithi040 ceph-mon[36154]: osdmap e101: 12 total, 12 up, 12 in 2024-03-21T10:48:07.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:07 smithi031 ceph-mon[31673]: pgmap v295: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 53 KiB/s wr, 25 op/s 2024-03-21T10:48:07.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:07 smithi031 ceph-mon[31673]: osdmap e101: 12 total, 12 up, 12 in 2024-03-21T10:48:07.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:07 smithi070 ceph-mon[36330]: pgmap v295: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 53 KiB/s wr, 25 op/s 2024-03-21T10:48:07.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:07 smithi070 ceph-mon[36330]: osdmap e101: 12 total, 12 up, 12 in 2024-03-21T10:48:08.519 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-21T10:48:08.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:48:08.519 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-21T10:48:08.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.845+0000 7fb56d600640 1 -- 172.21.15.31:0/2442541591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5681147f0 msgr2=0x7fb568116be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:08.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.845+0000 7fb56d600640 1 --2- 172.21.15.31:0/2442541591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5681147f0 0x7fb568116be0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fb554009860 tx=0x7fb55402bb70 comp rx=0 tx=0).stop 2024-03-21T10:48:08.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.846+0000 7fb56d600640 1 -- 172.21.15.31:0/2442541591 shutdown_connections 2024-03-21T10:48:08.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.846+0000 7fb56d600640 1 --2- 172.21.15.31:0/2442541591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5681147f0 0x7fb568116be0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.846+0000 7fb56d600640 1 --2- 172.21.15.31:0/2442541591 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb568108fc0 0x7fb5681142b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.846+0000 7fb56d600640 1 --2- 172.21.15.31:0/2442541591 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5681086a0 0x7fb568108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.846+0000 7fb56d600640 1 -- 172.21.15.31:0/2442541591 >> 172.21.15.31:0/2442541591 conn(0x7fb56806c710 msgr2=0x7fb56806cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:08.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.846+0000 7fb56d600640 1 -- 172.21.15.31:0/2442541591 shutdown_connections 2024-03-21T10:48:08.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.846+0000 7fb56d600640 1 -- 172.21.15.31:0/2442541591 wait complete. 2024-03-21T10:48:08.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.850+0000 7fb56d600640 1 Processor -- start 2024-03-21T10:48:08.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.850+0000 7fb56d600640 1 -- start start 2024-03-21T10:48:08.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.850+0000 7fb56d600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5681086a0 0x7fb568114210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:08.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.850+0000 7fb56d600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb568108fc0 0x7fb5681112b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:08.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.850+0000 7fb56d600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5681147f0 0x7fb5681118a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:08.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.851+0000 7fb56d600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56810bd20 con 0x7fb5681147f0 2024-03-21T10:48:08.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.851+0000 7fb56d600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56810bba0 con 0x7fb5681086a0 2024-03-21T10:48:08.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.851+0000 7fb56d600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56810bea0 con 0x7fb568108fc0 2024-03-21T10:48:08.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.851+0000 7fb566000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb568108fc0 0x7fb5681112b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:08.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.851+0000 7fb566000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb568108fc0 0x7fb5681112b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:48586/0 (socket says 172.21.15.31:48586) 2024-03-21T10:48:08.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.851+0000 7fb566000640 1 -- 172.21.15.31:0/566612630 learned_addr learned my addr 172.21.15.31:0/566612630 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:08.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.851+0000 7fb567400640 1 --2- 172.21.15.31:0/566612630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5681147f0 0x7fb5681118a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:08.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.851+0000 7fb566000640 1 -- 172.21.15.31:0/566612630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5681086a0 msgr2=0x7fb568114210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:08.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.851+0000 7fb566a00640 1 --2- 172.21.15.31:0/566612630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5681086a0 0x7fb568114210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:08.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.852+0000 7fb566000640 1 --2- 172.21.15.31:0/566612630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5681086a0 0x7fb568114210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.852+0000 7fb566000640 1 -- 172.21.15.31:0/566612630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5681147f0 msgr2=0x7fb5681118a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:08.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.852+0000 7fb566000640 1 --2- 172.21.15.31:0/566612630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5681147f0 0x7fb5681118a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.852+0000 7fb566000640 1 -- 172.21.15.31:0/566612630 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5681b2e60 con 0x7fb568108fc0 2024-03-21T10:48:08.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.852+0000 7fb567400640 1 --2- 172.21.15.31:0/566612630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5681147f0 0x7fb5681118a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:08.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.852+0000 7fb566a00640 1 --2- 172.21.15.31:0/566612630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5681086a0 0x7fb568114210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:48:08.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.852+0000 7fb566000640 1 --2- 172.21.15.31:0/566612630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb568108fc0 0x7fb5681112b0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fb55800e9c0 tx=0x7fb55800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:08.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.853+0000 7fb55f400640 1 -- 172.21.15.31:0/566612630 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55800ccc0 con 0x7fb568108fc0 2024-03-21T10:48:08.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.853+0000 7fb56d600640 1 -- 172.21.15.31:0/566612630 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5681b3030 con 0x7fb568108fc0 2024-03-21T10:48:08.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.853+0000 7fb56d600640 1 -- 172.21.15.31:0/566612630 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5681b35a0 con 0x7fb568108fc0 2024-03-21T10:48:08.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.853+0000 7fb55f400640 1 -- 172.21.15.31:0/566612630 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb558005500 con 0x7fb568108fc0 2024-03-21T10:48:08.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.855+0000 7fb55f400640 1 -- 172.21.15.31:0/566612630 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55800ccc0 con 0x7fb568108fc0 2024-03-21T10:48:08.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.862+0000 7fb5703c1740 1 Processor -- start 2024-03-21T10:48:08.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.863+0000 7fb5703c1740 1 -- start start 2024-03-21T10:48:08.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.863+0000 7fb55f400640 1 -- 172.21.15.31:0/566612630 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fb5580046c0 con 0x7fb568108fc0 2024-03-21T10:48:08.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.863+0000 7fb5703c1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dcab10bd30 0x55dcab10c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:08.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.864+0000 7fb5703c1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dcab10c650 0x55dcab126050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:08.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.864+0000 7fb5703c1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcab126590 0x55dcab128980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:08.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.864+0000 7fb5703c1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcab070eb0 con 0x55dcab10bd30 2024-03-21T10:48:08.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.864+0000 7fb5703c1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcab019a60 con 0x55dcab10c650 2024-03-21T10:48:08.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.864+0000 7fb5703c1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcab06cf50 con 0x55dcab126590 2024-03-21T10:48:08.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.864+0000 7fb567400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dcab10bd30 0x55dcab10c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:08.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.865+0000 7fb567400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dcab10bd30 0x55dcab10c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46156/0 (socket says 172.21.15.31:46156) 2024-03-21T10:48:08.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.865+0000 7fb567400640 1 -- 172.21.15.31:0/501533618 learned_addr learned my addr 172.21.15.31:0/501533618 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:08.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.865+0000 7fb55f400640 1 --2- 172.21.15.31:0/566612630 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb54c0a3310 0x7fb54c0a57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:08.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.866+0000 7fb566000640 1 --2- 172.21.15.31:0/501533618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcab126590 0x55dcab128980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:08.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.865+0000 7fb55f400640 1 -- 172.21.15.31:0/566612630 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9534+0+0 (secure 0 0 0) 0x7fb5580cfba0 con 0x7fb568108fc0 2024-03-21T10:48:08.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.866+0000 7fb567400640 1 --2- 172.21.15.31:0/566612630 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb54c0a3310 0x7fb54c0a57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:08.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.866+0000 7fb566a00640 1 --2- 172.21.15.31:0/501533618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dcab10c650 0x55dcab126050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:08.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.866+0000 7fb566000640 1 -- 172.21.15.31:0/501533618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dcab10c650 msgr2=0x55dcab126050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:08.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.866+0000 7fb566000640 1 --2- 172.21.15.31:0/501533618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dcab10c650 0x55dcab126050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.866+0000 7fb566000640 1 -- 172.21.15.31:0/501533618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dcab10bd30 msgr2=0x55dcab10c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:08.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.866+0000 7fb566000640 1 --2- 172.21.15.31:0/501533618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dcab10bd30 0x55dcab10c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.867+0000 7fb566000640 1 -- 172.21.15.31:0/501533618 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dcaae4d7e0 con 0x55dcab126590 2024-03-21T10:48:08.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.867+0000 7fb566a00640 1 --2- 172.21.15.31:0/501533618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dcab10c650 0x55dcab126050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:48:08.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.867+0000 7fb566000640 1 --2- 172.21.15.31:0/501533618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcab126590 0x55dcab128980 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb5580ccea0 tx=0x7fb55809b560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a63989c5fe24dcee server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:08.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.867+0000 7fb553400640 1 -- 172.21.15.31:0/501533618 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55809bb40 con 0x55dcab126590 2024-03-21T10:48:08.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.867+0000 7fb553400640 1 -- 172.21.15.31:0/501533618 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb55809fc90 con 0x55dcab126590 2024-03-21T10:48:08.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.867+0000 7fb553400640 1 -- 172.21.15.31:0/501533618 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55809af90 con 0x55dcab126590 2024-03-21T10:48:08.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.869+0000 7fb5703c1740 1 -- 172.21.15.31:0/501533618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcab126590 msgr2=0x55dcab128980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:08.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.869+0000 7fb5703c1740 1 --2- 172.21.15.31:0/501533618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcab126590 0x55dcab128980 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb5580ccea0 tx=0x7fb55809b560 comp rx=0 tx=0).stop 2024-03-21T10:48:08.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.870+0000 7fb567400640 1 --2- 172.21.15.31:0/566612630 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb54c0a3310 0x7fb54c0a57d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb554034a30 tx=0x7fb554035e40 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:08.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.870+0000 7fb5703c1740 1 -- 172.21.15.31:0/501533618 shutdown_connections 2024-03-21T10:48:08.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.870+0000 7fb5703c1740 1 --2- 172.21.15.31:0/501533618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcab126590 0x55dcab128980 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.870+0000 7fb5703c1740 1 --2- 172.21.15.31:0/501533618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dcab10c650 0x55dcab126050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.870+0000 7fb5703c1740 1 --2- 172.21.15.31:0/501533618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dcab10bd30 0x55dcab10c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.870+0000 7fb5703c1740 1 -- 172.21.15.31:0/501533618 >> 172.21.15.31:0/501533618 conn(0x55dcab1067b0 msgr2=0x55dcab106b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:08.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.871+0000 7fb5703c1740 1 -- 172.21.15.31:0/501533618 shutdown_connections 2024-03-21T10:48:08.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.871+0000 7fb5703c1740 1 -- 172.21.15.31:0/501533618 wait complete. 2024-03-21T10:48:08.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.872+0000 7fb5703c1740 1 -- start start 2024-03-21T10:48:08.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.873+0000 7fb5703c1740 1 Processor -- start 2024-03-21T10:48:08.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.873+0000 7fb5703c1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcab10bd30 0x55dcab117440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:08.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.873+0000 7fb5703c1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dcab10c650 0x55dcab117980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:08.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.873+0000 7fb5703c1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dcab126590 0x55dcab112840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:08.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.873+0000 7fb5703c1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcab10d220 con 0x55dcab126590 2024-03-21T10:48:08.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.873+0000 7fb5703c1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcab02ceb0 con 0x55dcab10c650 2024-03-21T10:48:08.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.873+0000 7fb5703c1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcab10d3a0 con 0x55dcab10bd30 2024-03-21T10:48:08.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.874+0000 7fb5703c1740 10 client.0 _ll_register_callbacks cb 0x55dcab021150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:08.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.874+0000 7fb567400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dcab126590 0x55dcab112840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:08.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.874+0000 7fb567400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dcab126590 0x55dcab112840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46172/0 (socket says 172.21.15.31:46172) 2024-03-21T10:48:08.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.874+0000 7fb567400640 1 -- 172.21.15.31:0/3193090836 learned_addr learned my addr 172.21.15.31:0/3193090836 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:08.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.874+0000 7fb566000640 1 --2- 172.21.15.31:0/3193090836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dcab10c650 0x55dcab117980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:08.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.874+0000 7fb566a00640 1 --2- 172.21.15.31:0/3193090836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcab10bd30 0x55dcab117440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:08.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.875+0000 7fb567400640 1 -- 172.21.15.31:0/3193090836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcab10bd30 msgr2=0x55dcab117440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:08.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.875+0000 7fb567400640 1 --2- 172.21.15.31:0/3193090836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcab10bd30 0x55dcab117440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.875+0000 7fb567400640 1 -- 172.21.15.31:0/3193090836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dcab10c650 msgr2=0x55dcab117980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:08.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.875+0000 7fb567400640 1 --2- 172.21.15.31:0/3193090836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dcab10c650 0x55dcab117980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.875+0000 7fb567400640 1 -- 172.21.15.31:0/3193090836 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dcaaeadf70 con 0x55dcab126590 2024-03-21T10:48:08.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.876+0000 7fb566000640 1 --2- 172.21.15.31:0/3193090836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dcab10c650 0x55dcab117980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:08.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.876+0000 7fb567400640 1 --2- 172.21.15.31:0/3193090836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dcab126590 0x55dcab112840 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb55402c180 tx=0x7fb554009490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbc1bca0a98a5b72 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:08.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.877+0000 7fb547e00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:48:08.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.877+0000 7fb547e00640 1 -- 172.21.15.31:0/3193090836 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb554039040 con 0x55dcab126590 2024-03-21T10:48:08.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.877+0000 7fb547e00640 1 -- 172.21.15.31:0/3193090836 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9534+0+0 (secure 0 0 0) 0x7fb554004ad0 con 0x55dcab126590 2024-03-21T10:48:08.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.877+0000 7fb5703c1740 1 -- 172.21.15.31:0/3193090836 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb55809fc90 con 0x55dcab126590 2024-03-21T10:48:08.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.880+0000 7fb547e00640 1 -- 172.21.15.31:0/3193090836 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb554004cd0 con 0x55dcab126590 2024-03-21T10:48:08.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.880+0000 7fb5703c1740 10 client.17403 fetch_fsmap learned FSMap version 28 2024-03-21T10:48:08.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.880+0000 7fb5703c1740 1 -- 172.21.15.31:0/3193090836 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55dcaad45170 con 0x55dcab126590 2024-03-21T10:48:08.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.881+0000 7fb547e00640 1 -- 172.21.15.31:0/3193090836 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb55809fc90 con 0x55dcab126590 2024-03-21T10:48:08.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.881+0000 7fb547e00640 1 -- 172.21.15.31:0/3193090836 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fb554061070 con 0x55dcab126590 2024-03-21T10:48:08.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.881+0000 7fb5703c1740 10 client.17403 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:48:08.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.881+0000 7fb5703c1740 10 client.17403 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:48:08.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.881+0000 7fb5703c1740 20 client.17403 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:08.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.881+0000 7fb5703c1740 1 --2- 172.21.15.31:0/3193090836 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55dcab122ad0 0x55dcab124ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:08.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.881+0000 7fb5703c1740 4 client.17403 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:48:08.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.881+0000 7fb5703c1740 1 -- 172.21.15.31:0/3193090836 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dcab07e3f0 con 0x55dcab122ad0 2024-03-21T10:48:08.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.883+0000 7fb566a00640 1 --2- 172.21.15.31:0/3193090836 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55dcab122ad0 0x55dcab124ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:08.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.883+0000 7fb566a00640 1 --2- 172.21.15.31:0/3193090836 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55dcab122ad0 0x55dcab124ef0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7450d9947eb7a1fb server_cookie=743140ed4a780c95 in_seq=0 out_seq=0 2024-03-21T10:48:08.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.883+0000 7fb547e00640 10 client.17403 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:08.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.884+0000 7fb547e00640 1 -- 172.21.15.31:0/3193090836 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55dcab07e3f0 con 0x55dcab122ad0 2024-03-21T10:48:08.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.884+0000 7fb547e00640 10 client.17403 handle_command_reply: tid=0 2024-03-21T10:48:08.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.884+0000 7fb5703c1740 1 client.17403 shutdown 2024-03-21T10:48:08.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.885+0000 7fb5703c1740 1 -- 172.21.15.31:0/3193090836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dcab126590 msgr2=0x55dcab112840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:08.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.885+0000 7fb5703c1740 1 --2- 172.21.15.31:0/3193090836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dcab126590 0x55dcab112840 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb55402c180 tx=0x7fb554009490 comp rx=0 tx=0).stop 2024-03-21T10:48:08.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.885+0000 7fb5703c1740 1 -- 172.21.15.31:0/3193090836 shutdown_connections 2024-03-21T10:48:08.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.885+0000 7fb5703c1740 1 --2- 172.21.15.31:0/3193090836 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55dcab122ad0 0x55dcab124ef0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.885+0000 7fb5703c1740 1 --2- 172.21.15.31:0/3193090836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dcab126590 0x55dcab112840 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.885+0000 7fb5703c1740 1 --2- 172.21.15.31:0/3193090836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dcab10c650 0x55dcab117980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.885+0000 7fb5703c1740 1 --2- 172.21.15.31:0/3193090836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcab10bd30 0x55dcab117440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.885+0000 7fb547e00640 0 client.17403 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:08.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.885+0000 7fb5703c1740 1 -- 172.21.15.31:0/3193090836 >> 172.21.15.31:0/3193090836 conn(0x55dcab1067b0 msgr2=0x55dcab108500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:08.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.886+0000 7fb5703c1740 1 -- 172.21.15.31:0/3193090836 shutdown_connections 2024-03-21T10:48:08.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.886+0000 7fb5703c1740 1 -- 172.21.15.31:0/3193090836 wait complete. 2024-03-21T10:48:08.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.886+0000 7fb5703c1740 20 client.17403 trim_cache size 0 max 16384 2024-03-21T10:48:08.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.928+0000 7fb5703c1740 1 Processor -- start 2024-03-21T10:48:08.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.929+0000 7fb5703c1740 1 -- start start 2024-03-21T10:48:08.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.929+0000 7fb5703c1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dcab1067b0 0x55dcab108310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:08.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.929+0000 7fb5703c1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dcab10bd30 0x55dcab10fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:08.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.929+0000 7fb5703c1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcab10c650 0x55dcab110000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:08.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.929+0000 7fb5703c1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5480123e0 con 0x55dcab1067b0 2024-03-21T10:48:08.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.929+0000 7fb5703c1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54800fee0 con 0x55dcab10bd30 2024-03-21T10:48:08.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.929+0000 7fb5703c1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54800df60 con 0x55dcab10c650 2024-03-21T10:48:08.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.930+0000 7fb567400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcab10c650 0x55dcab110000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:08.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.930+0000 7fb567400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcab10c650 0x55dcab110000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:48604/0 (socket says 172.21.15.31:48604) 2024-03-21T10:48:08.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.930+0000 7fb567400640 1 -- 172.21.15.31:0/3435347141 learned_addr learned my addr 172.21.15.31:0/3435347141 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:08.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.930+0000 7fb566000640 1 --2- 172.21.15.31:0/3435347141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dcab10bd30 0x55dcab10fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:08.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.930+0000 7fb566a00640 1 --2- 172.21.15.31:0/3435347141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dcab1067b0 0x55dcab108310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:08.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.930+0000 7fb566000640 1 -- 172.21.15.31:0/3435347141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcab10c650 msgr2=0x55dcab110000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:08.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.930+0000 7fb566000640 1 --2- 172.21.15.31:0/3435347141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcab10c650 0x55dcab110000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.930+0000 7fb566000640 1 -- 172.21.15.31:0/3435347141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dcab1067b0 msgr2=0x55dcab108310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:08.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.930+0000 7fb566000640 1 --2- 172.21.15.31:0/3435347141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dcab1067b0 0x55dcab108310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.930+0000 7fb566000640 1 -- 172.21.15.31:0/3435347141 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dcab125550 con 0x55dcab10bd30 2024-03-21T10:48:08.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.931+0000 7fb566000640 1 --2- 172.21.15.31:0/3435347141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dcab10bd30 0x55dcab10fac0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fb558009470 tx=0x7fb558009400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29039d0d3469ba20 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:08.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.931+0000 7fb552a00640 1 -- 172.21.15.31:0/3435347141 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55809bb40 con 0x55dcab10bd30 2024-03-21T10:48:08.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.931+0000 7fb552a00640 1 -- 172.21.15.31:0/3435347141 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5580049b0 con 0x55dcab10bd30 2024-03-21T10:48:08.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.931+0000 7fb552a00640 1 -- 172.21.15.31:0/3435347141 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55809b280 con 0x55dcab10bd30 2024-03-21T10:48:08.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.932+0000 7fb5703c1740 1 -- 172.21.15.31:0/3435347141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dcab10bd30 msgr2=0x55dcab10fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:08.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.932+0000 7fb5703c1740 1 --2- 172.21.15.31:0/3435347141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dcab10bd30 0x55dcab10fac0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fb558009470 tx=0x7fb558009400 comp rx=0 tx=0).stop 2024-03-21T10:48:08.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.932+0000 7fb5703c1740 1 -- 172.21.15.31:0/3435347141 shutdown_connections 2024-03-21T10:48:08.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.932+0000 7fb5703c1740 1 --2- 172.21.15.31:0/3435347141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcab10c650 0x55dcab110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.932+0000 7fb5703c1740 1 --2- 172.21.15.31:0/3435347141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dcab10bd30 0x55dcab10fac0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.932+0000 7fb5703c1740 1 --2- 172.21.15.31:0/3435347141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dcab1067b0 0x55dcab108310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.932+0000 7fb5703c1740 1 -- 172.21.15.31:0/3435347141 >> 172.21.15.31:0/3435347141 conn(0x55dcab1129a0 msgr2=0x55dcab107c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:08.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.933+0000 7fb5703c1740 1 -- 172.21.15.31:0/3435347141 shutdown_connections 2024-03-21T10:48:08.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.933+0000 7fb5703c1740 1 -- 172.21.15.31:0/3435347141 wait complete. 2024-03-21T10:48:08.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.934+0000 7fb5703c1740 1 -- start start 2024-03-21T10:48:08.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.934+0000 7fb5703c1740 1 Processor -- start 2024-03-21T10:48:08.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.935+0000 7fb5703c1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dcab1067b0 0x55dcab11cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:08.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.935+0000 7fb5703c1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dcab10bd30 0x55dcab11d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:08.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.935+0000 7fb5703c1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcab10c650 0x55dcab17ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:08.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.935+0000 7fb5703c1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54800e560 con 0x55dcab1067b0 2024-03-21T10:48:08.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.936+0000 7fb5703c1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54800f880 con 0x55dcab10bd30 2024-03-21T10:48:08.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.936+0000 7fb5703c1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54800ebc0 con 0x55dcab10c650 2024-03-21T10:48:08.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.936+0000 7fb566a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dcab1067b0 0x55dcab11cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:08.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.936+0000 7fb567400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcab10c650 0x55dcab17ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:08.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.936+0000 7fb566000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dcab10bd30 0x55dcab11d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:08.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.936+0000 7fb566000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dcab10bd30 0x55dcab11d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:40746/0 (socket says 172.21.15.31:40746) 2024-03-21T10:48:08.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.936+0000 7fb566000640 1 -- 172.21.15.31:0/285600535 learned_addr learned my addr 172.21.15.31:0/285600535 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:08.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.936+0000 7fb5703c1740 10 client.0 _ll_register_callbacks cb 0x7fb5540096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:08.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.936+0000 7fb567400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcab10c650 0x55dcab17ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:48612/0 (socket says 172.21.15.31:48612) 2024-03-21T10:48:08.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.936+0000 7fb567400640 1 -- 172.21.15.31:0/285600535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dcab10bd30 msgr2=0x55dcab11d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:08.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.936+0000 7fb567400640 1 --2- 172.21.15.31:0/285600535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dcab10bd30 0x55dcab11d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.936+0000 7fb567400640 1 -- 172.21.15.31:0/285600535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dcab1067b0 msgr2=0x55dcab11cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:08.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.936+0000 7fb567400640 1 --2- 172.21.15.31:0/285600535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dcab1067b0 0x55dcab11cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.936+0000 7fb567400640 1 -- 172.21.15.31:0/285600535 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dcab10cb10 con 0x55dcab10c650 2024-03-21T10:48:08.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.937+0000 7fb567400640 1 --2- 172.21.15.31:0/285600535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcab10c650 0x55dcab17ad60 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb554005e00 tx=0x7fb554004b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7da017933c969fe server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:08.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.937+0000 7fb547e00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:48:08.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.939+0000 7fb547e00640 1 -- 172.21.15.31:0/285600535 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb554004cd0 con 0x55dcab10c650 2024-03-21T10:48:08.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.939+0000 7fb547e00640 1 -- 172.21.15.31:0/285600535 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9534+0+0 (secure 0 0 0) 0x7fb554004370 con 0x55dcab10c650 2024-03-21T10:48:08.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.939+0000 7fb5703c1740 1 -- 172.21.15.31:0/285600535 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb5580049b0 con 0x55dcab10c650 2024-03-21T10:48:08.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.940+0000 7fb547e00640 1 -- 172.21.15.31:0/285600535 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb554005600 con 0x55dcab10c650 2024-03-21T10:48:08.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.940+0000 7fb5703c1740 10 client.27961 fetch_fsmap learned FSMap version 28 2024-03-21T10:48:08.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.940+0000 7fb5703c1740 1 -- 172.21.15.31:0/285600535 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55dcab101c20 con 0x55dcab10c650 2024-03-21T10:48:08.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.941+0000 7fb547e00640 1 -- 172.21.15.31:0/285600535 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5580049b0 con 0x55dcab10c650 2024-03-21T10:48:08.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.941+0000 7fb547e00640 1 -- 172.21.15.31:0/285600535 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fb55408c480 con 0x55dcab10c650 2024-03-21T10:48:08.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.941+0000 7fb5703c1740 10 client.27961 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:48:08.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.941+0000 7fb5703c1740 10 client.27961 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:48:08.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.941+0000 7fb5703c1740 20 client.27961 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:08.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.941+0000 7fb5703c1740 1 --2- 172.21.15.31:0/285600535 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55dcab126590 0x55dcab184a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:08.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.941+0000 7fb5703c1740 4 client.27961 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-21T10:48:08.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.941+0000 7fb5703c1740 1 -- 172.21.15.31:0/285600535 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55dcab078740 con 0x55dcab126590 2024-03-21T10:48:08.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.941+0000 7fb566a00640 1 --2- 172.21.15.31:0/285600535 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55dcab126590 0x55dcab184a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:08.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.942+0000 7fb547e00640 10 client.27961 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:08.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.944+0000 7fb566a00640 1 --2- 172.21.15.31:0/285600535 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55dcab126590 0x55dcab184a30 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ce7a6bc85495959 server_cookie=98bcf771d941452b in_seq=0 out_seq=0 2024-03-21T10:48:08.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.946+0000 7fb547e00640 1 -- 172.21.15.31:0/285600535 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55dcab078740 con 0x55dcab126590 2024-03-21T10:48:08.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.946+0000 7fb547e00640 10 client.27961 handle_command_reply: tid=0 2024-03-21T10:48:08.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.946+0000 7fb5703c1740 1 client.27961 shutdown 2024-03-21T10:48:08.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.947+0000 7fb5703c1740 1 -- 172.21.15.31:0/285600535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcab10c650 msgr2=0x55dcab17ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:08.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.947+0000 7fb5703c1740 1 --2- 172.21.15.31:0/285600535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcab10c650 0x55dcab17ad60 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb554005e00 tx=0x7fb554004b80 comp rx=0 tx=0).stop 2024-03-21T10:48:08.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.947+0000 7fb5703c1740 1 -- 172.21.15.31:0/285600535 shutdown_connections 2024-03-21T10:48:08.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.947+0000 7fb5703c1740 1 --2- 172.21.15.31:0/285600535 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55dcab126590 0x55dcab184a30 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.947+0000 7fb5703c1740 1 --2- 172.21.15.31:0/285600535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dcab10c650 0x55dcab17ad60 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.947+0000 7fb5703c1740 1 --2- 172.21.15.31:0/285600535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dcab10bd30 0x55dcab11d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.947+0000 7fb547e00640 0 client.27961 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:08.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.947+0000 7fb5703c1740 1 --2- 172.21.15.31:0/285600535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dcab1067b0 0x55dcab11cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.947+0000 7fb5703c1740 1 -- 172.21.15.31:0/285600535 >> 172.21.15.31:0/285600535 conn(0x55dcab1129a0 msgr2=0x55dcab108490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:08.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.948+0000 7fb5703c1740 1 -- 172.21.15.31:0/285600535 shutdown_connections 2024-03-21T10:48:08.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.948+0000 7fb5703c1740 1 -- 172.21.15.31:0/285600535 wait complete. 2024-03-21T10:48:08.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.948+0000 7fb5703c1740 20 client.27961 trim_cache size 0 max 16384 2024-03-21T10:48:08.946 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:48:08.946 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-03-21T10:48:08.946 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "ff07db78-f943-4bf3-9384-0bb095dfb72d", 2024-03-21T10:48:08.946 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-03-21T10:48:08.946 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:48:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.950+0000 7fb56d600640 1 -- 172.21.15.31:0/566612630 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb54c0a3310 msgr2=0x7fb54c0a57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.950+0000 7fb56d600640 1 --2- 172.21.15.31:0/566612630 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb54c0a3310 0x7fb54c0a57d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb554034a30 tx=0x7fb554035e40 comp rx=0 tx=0).stop 2024-03-21T10:48:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.950+0000 7fb56d600640 1 -- 172.21.15.31:0/566612630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb568108fc0 msgr2=0x7fb5681112b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.950+0000 7fb56d600640 1 --2- 172.21.15.31:0/566612630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb568108fc0 0x7fb5681112b0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fb55800e9c0 tx=0x7fb55800ee80 comp rx=0 tx=0).stop 2024-03-21T10:48:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.950+0000 7fb56d600640 1 -- 172.21.15.31:0/566612630 shutdown_connections 2024-03-21T10:48:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.950+0000 7fb56d600640 1 --2- 172.21.15.31:0/566612630 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb54c0a3310 0x7fb54c0a57d0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.950+0000 7fb56d600640 1 --2- 172.21.15.31:0/566612630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb5681147f0 0x7fb5681118a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.950+0000 7fb56d600640 1 --2- 172.21.15.31:0/566612630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb568108fc0 0x7fb5681112b0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.950+0000 7fb56d600640 1 --2- 172.21.15.31:0/566612630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5681086a0 0x7fb568114210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:08.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.950+0000 7fb56d600640 1 -- 172.21.15.31:0/566612630 >> 172.21.15.31:0/566612630 conn(0x7fb56806c710 msgr2=0x7fb5680700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:08.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.950+0000 7fb56d600640 1 -- 172.21.15.31:0/566612630 shutdown_connections 2024-03-21T10:48:08.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:08.950+0000 7fb56d600640 1 -- 172.21.15.31:0/566612630 wait complete. 2024-03-21T10:48:08.978 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:48:08.978 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:48:09.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.284+0000 7f4955400640 1 -- 172.21.15.31:0/2421545340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4950104040 msgr2=0x7f4950110550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:09.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.284+0000 7f4955400640 1 --2- 172.21.15.31:0/2421545340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4950104040 0x7f4950110550 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4948009cd0 tx=0x7f494802c0e0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.284+0000 7f4955400640 1 -- 172.21.15.31:0/2421545340 shutdown_connections 2024-03-21T10:48:09.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.284+0000 7f4955400640 1 --2- 172.21.15.31:0/2421545340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4950104040 0x7f4950110550 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.284+0000 7f4955400640 1 --2- 172.21.15.31:0/2421545340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4950103680 0x7f4950103b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.284+0000 7f4955400640 1 --2- 172.21.15.31:0/2421545340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4950109690 0x7f4950109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.284+0000 7f4955400640 1 -- 172.21.15.31:0/2421545340 >> 172.21.15.31:0/2421545340 conn(0x7f49500ff110 msgr2=0x7f4950101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:09.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.285+0000 7f4955400640 1 -- 172.21.15.31:0/2421545340 shutdown_connections 2024-03-21T10:48:09.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.285+0000 7f4955400640 1 -- 172.21.15.31:0/2421545340 wait complete. 2024-03-21T10:48:09.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.287+0000 7f4955400640 1 Processor -- start 2024-03-21T10:48:09.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.287+0000 7f4955400640 1 -- start start 2024-03-21T10:48:09.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.287+0000 7f4955400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4950103680 0x7f4950110450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:09.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.287+0000 7f4955400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4950104040 0x7f495010d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:09.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.287+0000 7f4955400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4950109690 0x7f495010dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:09.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.287+0000 7f4955400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4950113a60 con 0x7f4950104040 2024-03-21T10:48:09.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.287+0000 7f4955400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49501138e0 con 0x7f4950109690 2024-03-21T10:48:09.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.287+0000 7f4955400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4950113be0 con 0x7f4950103680 2024-03-21T10:48:09.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.288+0000 7f4954a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4950109690 0x7f495010dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:09.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.288+0000 7f4954a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4950109690 0x7f495010dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:40772/0 (socket says 172.21.15.31:40772) 2024-03-21T10:48:09.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.288+0000 7f4954a00640 1 -- 172.21.15.31:0/3904468857 learned_addr learned my addr 172.21.15.31:0/3904468857 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:09.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.288+0000 7f494fe00640 1 --2- 172.21.15.31:0/3904468857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4950103680 0x7f4950110450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:09.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.289+0000 7f494f400640 1 --2- 172.21.15.31:0/3904468857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4950104040 0x7f495010d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:09.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.289+0000 7f4954a00640 1 -- 172.21.15.31:0/3904468857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4950103680 msgr2=0x7f4950110450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:09.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.289+0000 7f4954a00640 1 --2- 172.21.15.31:0/3904468857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4950103680 0x7f4950110450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.289+0000 7f4954a00640 1 -- 172.21.15.31:0/3904468857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4950104040 msgr2=0x7f495010d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:09.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.289+0000 7f4954a00640 1 --2- 172.21.15.31:0/3904468857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4950104040 0x7f495010d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.289+0000 7f4954a00640 1 -- 172.21.15.31:0/3904468857 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f495019ff40 con 0x7f4950109690 2024-03-21T10:48:09.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.289+0000 7f4954a00640 1 --2- 172.21.15.31:0/3904468857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4950109690 0x7f495010dae0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f4948037040 tx=0x7f4948032460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:09.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.290+0000 7f494cc00640 1 -- 172.21.15.31:0/3904468857 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4948039040 con 0x7f4950109690 2024-03-21T10:48:09.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.290+0000 7f4955400640 1 -- 172.21.15.31:0/3904468857 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49501a01d0 con 0x7f4950109690 2024-03-21T10:48:09.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.290+0000 7f494cc00640 1 -- 172.21.15.31:0/3904468857 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4948044070 con 0x7f4950109690 2024-03-21T10:48:09.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.290+0000 7f4955400640 1 -- 172.21.15.31:0/3904468857 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49501a0710 con 0x7f4950109690 2024-03-21T10:48:09.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.293+0000 7f494cc00640 1 -- 172.21.15.31:0/3904468857 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4948039040 con 0x7f4950109690 2024-03-21T10:48:09.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.295+0000 7f49597b8740 1 Processor -- start 2024-03-21T10:48:09.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.296+0000 7f494cc00640 1 -- 172.21.15.31:0/3904468857 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f4948004040 con 0x7f4950109690 2024-03-21T10:48:09.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.296+0000 7f49597b8740 1 -- start start 2024-03-21T10:48:09.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.297+0000 7f49597b8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccf6934f20 0x55ccf6933320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:09.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.297+0000 7f49597b8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccf6933860 0x55ccf6933ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:09.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.297+0000 7f49597b8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ccf693d5d0 0x55ccf693f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:09.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.297+0000 7f49597b8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccf6898250 con 0x55ccf6933860 2024-03-21T10:48:09.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.297+0000 7f49597b8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccf6840a30 con 0x55ccf693d5d0 2024-03-21T10:48:09.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.297+0000 7f49597b8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccf68942f0 con 0x55ccf6934f20 2024-03-21T10:48:09.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.297+0000 7f494f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccf6934f20 0x55ccf6933320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:09.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.297+0000 7f494f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccf6934f20 0x55ccf6933320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:48624/0 (socket says 172.21.15.31:48624) 2024-03-21T10:48:09.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.297+0000 7f494f400640 1 -- 172.21.15.31:0/582527914 learned_addr learned my addr 172.21.15.31:0/582527914 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:09.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.297+0000 7f494f400640 1 --2- 172.21.15.31:0/582527914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ccf693d5d0 0x55ccf693f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:09.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.298+0000 7f494f400640 1 -- 172.21.15.31:0/582527914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ccf693d5d0 msgr2=0x55ccf693f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:09.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.298+0000 7f494f400640 1 --2- 172.21.15.31:0/582527914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ccf693d5d0 0x55ccf693f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.298+0000 7f494f400640 1 -- 172.21.15.31:0/582527914 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccf6933860 msgr2=0x55ccf6933ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:48:09.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.298+0000 7f494f400640 1 --2- 172.21.15.31:0/582527914 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccf6933860 0x55ccf6933ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.298+0000 7f494f400640 1 -- 172.21.15.31:0/582527914 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ccf682b690 con 0x55ccf6934f20 2024-03-21T10:48:09.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.300+0000 7f494f400640 1 --2- 172.21.15.31:0/582527914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccf6934f20 0x55ccf6933320 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f49400157e0 tx=0x7f4940009670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3821005f9cbbb6fa server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:09.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.300+0000 7f494cc00640 1 --2- 172.21.15.31:0/3904468857 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f49340a32c0 0x7f49340a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:09.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.300+0000 7f4955400640 1 -- 172.21.15.31:0/582527914 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4940015e20 con 0x55ccf6934f20 2024-03-21T10:48:09.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.300+0000 7f4955400640 1 -- 172.21.15.31:0/582527914 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4940005020 con 0x55ccf6934f20 2024-03-21T10:48:09.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.300+0000 7f4955400640 1 -- 172.21.15.31:0/582527914 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4940004a00 con 0x55ccf6934f20 2024-03-21T10:48:09.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.301+0000 7f494cc00640 1 -- 172.21.15.31:0/3904468857 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f49480f2000 con 0x7f4950109690 2024-03-21T10:48:09.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.301+0000 7f49597b8740 1 -- 172.21.15.31:0/582527914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccf6934f20 msgr2=0x55ccf6933320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:09.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.301+0000 7f49597b8740 1 --2- 172.21.15.31:0/582527914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccf6934f20 0x55ccf6933320 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f49400157e0 tx=0x7f4940009670 comp rx=0 tx=0).stop 2024-03-21T10:48:09.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.301+0000 7f494fe00640 1 --2- 172.21.15.31:0/3904468857 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f49340a32c0 0x7f49340a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:09.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.302+0000 7f49597b8740 1 -- 172.21.15.31:0/582527914 shutdown_connections 2024-03-21T10:48:09.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.302+0000 7f49597b8740 1 --2- 172.21.15.31:0/582527914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ccf693d5d0 0x55ccf693f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.302+0000 7f49597b8740 1 --2- 172.21.15.31:0/582527914 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccf6933860 0x55ccf6933ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.302+0000 7f49597b8740 1 --2- 172.21.15.31:0/582527914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccf6934f20 0x55ccf6933320 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.302+0000 7f49597b8740 1 -- 172.21.15.31:0/582527914 >> 172.21.15.31:0/582527914 conn(0x55ccf68fb200 msgr2=0x55ccf692ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:09.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.303+0000 7f49597b8740 1 -- 172.21.15.31:0/582527914 shutdown_connections 2024-03-21T10:48:09.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.304+0000 7f494fe00640 1 --2- 172.21.15.31:0/3904468857 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f49340a32c0 0x7f49340a5780 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x55ccf69344e0 tx=0x7f4938005f90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:09.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.304+0000 7f49597b8740 1 -- 172.21.15.31:0/582527914 wait complete. 2024-03-21T10:48:09.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.304+0000 7f49597b8740 1 -- start start 2024-03-21T10:48:09.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.305+0000 7f49597b8740 1 Processor -- start 2024-03-21T10:48:09.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.306+0000 7f49597b8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccf6933860 0x55ccf693cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:09.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.306+0000 7f49597b8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccf6934f20 0x55ccf693a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:09.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.306+0000 7f49597b8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ccf693d5d0 0x55ccf693a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:09.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.306+0000 7f49597b8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccf69402a0 con 0x55ccf6933860 2024-03-21T10:48:09.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.306+0000 7f49597b8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccf6853e80 con 0x55ccf693d5d0 2024-03-21T10:48:09.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.306+0000 7f49597b8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccf6940420 con 0x55ccf6934f20 2024-03-21T10:48:09.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.306+0000 7f49597b8740 10 client.0 _ll_register_callbacks cb 0x55ccf6866130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:09.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.307+0000 7f494fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccf6933860 0x55ccf693cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:09.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.307+0000 7f4954a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ccf693d5d0 0x55ccf693a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:09.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.307+0000 7f494f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccf6934f20 0x55ccf693a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:09.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.307+0000 7f494f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccf6934f20 0x55ccf693a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:48630/0 (socket says 172.21.15.31:48630) 2024-03-21T10:48:09.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.307+0000 7f494f400640 1 -- 172.21.15.31:0/3461611949 learned_addr learned my addr 172.21.15.31:0/3461611949 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:09.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.308+0000 7f494f400640 1 -- 172.21.15.31:0/3461611949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ccf693d5d0 msgr2=0x55ccf693a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:09.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.308+0000 7f494f400640 1 --2- 172.21.15.31:0/3461611949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ccf693d5d0 0x55ccf693a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.308+0000 7f494f400640 1 -- 172.21.15.31:0/3461611949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccf6933860 msgr2=0x55ccf693cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:09.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.308+0000 7f494f400640 1 --2- 172.21.15.31:0/3461611949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccf6933860 0x55ccf693cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.308+0000 7f494f400640 1 -- 172.21.15.31:0/3461611949 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ccf66767d0 con 0x55ccf6934f20 2024-03-21T10:48:09.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.308+0000 7f4954a00640 1 --2- 172.21.15.31:0/3461611949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ccf693d5d0 0x55ccf693a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:09.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.309+0000 7f494f400640 1 --2- 172.21.15.31:0/3461611949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccf6934f20 0x55ccf693a0e0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f49400058c0 tx=0x7f4940005480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0a7b1eea089e0f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:09.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.309+0000 7f494fe00640 1 --2- 172.21.15.31:0/3461611949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccf6933860 0x55ccf693cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:09.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.309+0000 7f493d600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:48:09.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.309+0000 7f493d600640 1 -- 172.21.15.31:0/3461611949 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4940025070 con 0x55ccf6934f20 2024-03-21T10:48:09.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.310+0000 7f493d600640 1 -- 172.21.15.31:0/3461611949 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f4940004220 con 0x55ccf6934f20 2024-03-21T10:48:09.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.311+0000 7f49597b8740 1 -- 172.21.15.31:0/3461611949 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4940005020 con 0x55ccf6934f20 2024-03-21T10:48:09.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.311+0000 7f493d600640 1 -- 172.21.15.31:0/3461611949 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4940004460 con 0x55ccf6934f20 2024-03-21T10:48:09.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.311+0000 7f49597b8740 10 client.27973 fetch_fsmap learned FSMap version 28 2024-03-21T10:48:09.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.311+0000 7f49597b8740 1 -- 172.21.15.31:0/3461611949 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ccf65ac3f0 con 0x55ccf6934f20 2024-03-21T10:48:09.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.312+0000 7f493d600640 1 -- 172.21.15.31:0/3461611949 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4940005020 con 0x55ccf6934f20 2024-03-21T10:48:09.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.314+0000 7f493d600640 1 -- 172.21.15.31:0/3461611949 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f494000f040 con 0x55ccf6934f20 2024-03-21T10:48:09.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.314+0000 7f49597b8740 10 client.27973 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:48:09.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.315+0000 7f49597b8740 10 client.27973 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:48:09.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.315+0000 7f49597b8740 20 client.27973 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:09.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.315+0000 7f49597b8740 1 --2- 172.21.15.31:0/3461611949 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ccf695e1e0 0x55ccf69605d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:09.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.315+0000 7f49597b8740 4 client.27973 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:48:09.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.315+0000 7f49597b8740 1 -- 172.21.15.31:0/3461611949 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ccf68a5790 con 0x55ccf695e1e0 2024-03-21T10:48:09.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.316+0000 7f494fe00640 1 --2- 172.21.15.31:0/3461611949 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ccf695e1e0 0x55ccf69605d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:09.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.316+0000 7f493d600640 10 client.27973 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:09.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.316+0000 7f494fe00640 1 --2- 172.21.15.31:0/3461611949 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ccf695e1e0 0x55ccf69605d0 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bab9264db821c945 server_cookie=96d81a9830161fdb in_seq=0 out_seq=0 2024-03-21T10:48:09.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.317+0000 7f493d600640 1 -- 172.21.15.31:0/3461611949 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ccf68a5790 con 0x55ccf695e1e0 2024-03-21T10:48:09.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.317+0000 7f493d600640 10 client.27973 handle_command_reply: tid=0 2024-03-21T10:48:09.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.317+0000 7f49597b8740 1 client.27973 shutdown 2024-03-21T10:48:09.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.319+0000 7f49597b8740 1 -- 172.21.15.31:0/3461611949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccf6934f20 msgr2=0x55ccf693a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:09.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.319+0000 7f49597b8740 1 --2- 172.21.15.31:0/3461611949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccf6934f20 0x55ccf693a0e0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f49400058c0 tx=0x7f4940005480 comp rx=0 tx=0).stop 2024-03-21T10:48:09.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.319+0000 7f49597b8740 1 -- 172.21.15.31:0/3461611949 shutdown_connections 2024-03-21T10:48:09.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.319+0000 7f49597b8740 1 --2- 172.21.15.31:0/3461611949 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ccf695e1e0 0x55ccf69605d0 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.319+0000 7f49597b8740 1 --2- 172.21.15.31:0/3461611949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ccf693d5d0 0x55ccf693a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.319+0000 7f49597b8740 1 --2- 172.21.15.31:0/3461611949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccf6934f20 0x55ccf693a0e0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.319+0000 7f493d600640 0 client.27973 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:09.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.319+0000 7f49597b8740 1 --2- 172.21.15.31:0/3461611949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccf6933860 0x55ccf693cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.319+0000 7f49597b8740 1 -- 172.21.15.31:0/3461611949 >> 172.21.15.31:0/3461611949 conn(0x55ccf68fb200 msgr2=0x55ccf6930200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:09.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.321+0000 7f49597b8740 1 -- 172.21.15.31:0/3461611949 shutdown_connections 2024-03-21T10:48:09.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.321+0000 7f49597b8740 1 -- 172.21.15.31:0/3461611949 wait complete. 2024-03-21T10:48:09.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.321+0000 7f49597b8740 20 client.27973 trim_cache size 0 max 16384 2024-03-21T10:48:09.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.356+0000 7f49597b8740 1 Processor -- start 2024-03-21T10:48:09.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.358+0000 7f49597b8740 1 -- start start 2024-03-21T10:48:09.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.358+0000 7f49597b8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccf6933860 0x55ccf692e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:09.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.358+0000 7f49597b8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ccf6934f20 0x55ccf6930750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:09.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.358+0000 7f49597b8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccf693d5d0 0x55ccf693c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:09.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.358+0000 7f49597b8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49501b7520 con 0x55ccf6933860 2024-03-21T10:48:09.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.358+0000 7f49597b8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49501b5020 con 0x55ccf6934f20 2024-03-21T10:48:09.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.358+0000 7f49597b8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4950072240 con 0x55ccf693d5d0 2024-03-21T10:48:09.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.359+0000 7f494f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ccf6934f20 0x55ccf6930750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:09.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.359+0000 7f494f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ccf6934f20 0x55ccf6930750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:40798/0 (socket says 172.21.15.31:40798) 2024-03-21T10:48:09.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.359+0000 7f494f400640 1 -- 172.21.15.31:0/3756130739 learned_addr learned my addr 172.21.15.31:0/3756130739 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:09.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.359+0000 7f4954a00640 1 --2- 172.21.15.31:0/3756130739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccf693d5d0 0x55ccf693c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:09.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.359+0000 7f494f400640 1 -- 172.21.15.31:0/3756130739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccf693d5d0 msgr2=0x55ccf693c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:09.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.359+0000 7f494f400640 1 --2- 172.21.15.31:0/3756130739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccf693d5d0 0x55ccf693c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.359+0000 7f494f400640 1 -- 172.21.15.31:0/3756130739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccf6933860 msgr2=0x55ccf692e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:09.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.359+0000 7f494f400640 1 --2- 172.21.15.31:0/3756130739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccf6933860 0x55ccf692e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.359+0000 7f494f400640 1 -- 172.21.15.31:0/3756130739 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ccf656c0f0 con 0x55ccf6934f20 2024-03-21T10:48:09.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.360+0000 7f494f400640 1 --2- 172.21.15.31:0/3756130739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ccf6934f20 0x55ccf6930750 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f4940018010 tx=0x7f4940015050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf13ca8e5c651c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:09.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.361+0000 7f493fe00640 1 -- 172.21.15.31:0/3756130739 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4940036070 con 0x55ccf6934f20 2024-03-21T10:48:09.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.361+0000 7f493fe00640 1 -- 172.21.15.31:0/3756130739 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f494000e8c0 con 0x55ccf6934f20 2024-03-21T10:48:09.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.361+0000 7f493fe00640 1 -- 172.21.15.31:0/3756130739 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4940004460 con 0x55ccf6934f20 2024-03-21T10:48:09.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.362+0000 7f49597b8740 1 -- 172.21.15.31:0/3756130739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ccf6934f20 msgr2=0x55ccf6930750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:09.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.362+0000 7f49597b8740 1 --2- 172.21.15.31:0/3756130739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ccf6934f20 0x55ccf6930750 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f4940018010 tx=0x7f4940015050 comp rx=0 tx=0).stop 2024-03-21T10:48:09.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.363+0000 7f49597b8740 1 -- 172.21.15.31:0/3756130739 shutdown_connections 2024-03-21T10:48:09.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.363+0000 7f49597b8740 1 --2- 172.21.15.31:0/3756130739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccf693d5d0 0x55ccf693c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.363+0000 7f49597b8740 1 --2- 172.21.15.31:0/3756130739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ccf6934f20 0x55ccf6930750 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.363+0000 7f49597b8740 1 --2- 172.21.15.31:0/3756130739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccf6933860 0x55ccf692e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.363+0000 7f49597b8740 1 -- 172.21.15.31:0/3756130739 >> 172.21.15.31:0/3756130739 conn(0x55ccf68fb200 msgr2=0x55ccf6930040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:09.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.363+0000 7f49597b8740 1 -- 172.21.15.31:0/3756130739 shutdown_connections 2024-03-21T10:48:09.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.363+0000 7f49597b8740 1 -- 172.21.15.31:0/3756130739 wait complete. 2024-03-21T10:48:09.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.364+0000 7f49597b8740 1 -- start start 2024-03-21T10:48:09.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.364+0000 7f49597b8740 1 Processor -- start 2024-03-21T10:48:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.365+0000 7f49597b8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccf6933860 0x55ccf6960c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.365+0000 7f49597b8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccf6934f20 0x55ccf6961170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.365+0000 7f49597b8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ccf693d5d0 0x55ccf69616b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.365+0000 7f49597b8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4950072840 con 0x55ccf6933860 2024-03-21T10:48:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.365+0000 7f49597b8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49501b49c0 con 0x55ccf693d5d0 2024-03-21T10:48:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.365+0000 7f49597b8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4950072ea0 con 0x55ccf6934f20 2024-03-21T10:48:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.365+0000 7f49597b8740 10 client.0 _ll_register_callbacks cb 0x7f4940009d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:09.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.366+0000 7f494f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccf6934f20 0x55ccf6961170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:09.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.366+0000 7f494f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccf6934f20 0x55ccf6961170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:48648/0 (socket says 172.21.15.31:48648) 2024-03-21T10:48:09.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.366+0000 7f494f400640 1 -- 172.21.15.31:0/437916327 learned_addr learned my addr 172.21.15.31:0/437916327 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:09.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.366+0000 7f4954a00640 1 --2- 172.21.15.31:0/437916327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ccf693d5d0 0x55ccf69616b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:09.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.367+0000 7f494f400640 1 -- 172.21.15.31:0/437916327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ccf693d5d0 msgr2=0x55ccf69616b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:09.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.367+0000 7f494f400640 1 --2- 172.21.15.31:0/437916327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ccf693d5d0 0x55ccf69616b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.367+0000 7f494f400640 1 -- 172.21.15.31:0/437916327 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccf6933860 msgr2=0x55ccf6960c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:09.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.367+0000 7f494f400640 1 --2- 172.21.15.31:0/437916327 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccf6933860 0x55ccf6960c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.367+0000 7f494f400640 1 -- 172.21.15.31:0/437916327 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ccf69b8960 con 0x55ccf6934f20 2024-03-21T10:48:09.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.367+0000 7f4954a00640 1 --2- 172.21.15.31:0/437916327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ccf693d5d0 0x55ccf69616b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:48:09.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.368+0000 7f494f400640 1 --2- 172.21.15.31:0/437916327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccf6934f20 0x55ccf6961170 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x55ccf692a2b0 tx=0x7f494000eae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=113718f3abb22432 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:09.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.368+0000 7f493d600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:48:09.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.369+0000 7f493d600640 1 -- 172.21.15.31:0/437916327 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4940036070 con 0x55ccf6934f20 2024-03-21T10:48:09.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.369+0000 7f49597b8740 1 -- 172.21.15.31:0/437916327 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f494000e8c0 con 0x55ccf6934f20 2024-03-21T10:48:09.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.369+0000 7f493d600640 1 -- 172.21.15.31:0/437916327 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f4940004ae0 con 0x55ccf6934f20 2024-03-21T10:48:09.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.369+0000 7f493d600640 1 -- 172.21.15.31:0/437916327 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4940005000 con 0x55ccf6934f20 2024-03-21T10:48:09.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.370+0000 7f49597b8740 10 client.27979 fetch_fsmap learned FSMap version 28 2024-03-21T10:48:09.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.370+0000 7f49597b8740 1 -- 172.21.15.31:0/437916327 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ccf69b8960 con 0x55ccf6934f20 2024-03-21T10:48:09.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.370+0000 7f493d600640 1 -- 172.21.15.31:0/437916327 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f494000e8c0 con 0x55ccf6934f20 2024-03-21T10:48:09.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.370+0000 7f493d600640 1 -- 172.21.15.31:0/437916327 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f4940047900 con 0x55ccf6934f20 2024-03-21T10:48:09.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.371+0000 7f49597b8740 10 client.27979 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:48:09.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.371+0000 7f49597b8740 10 client.27979 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:48:09.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.371+0000 7f49597b8740 20 client.27979 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:09.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.371+0000 7f49597b8740 1 --2- 172.21.15.31:0/437916327 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ccf69b9ea0 0x55ccf69bc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:09.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.371+0000 7f49597b8740 4 client.27979 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:48:09.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.371+0000 7f49597b8740 1 -- 172.21.15.31:0/437916327 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ccf689fae0 con 0x55ccf69b9ea0 2024-03-21T10:48:09.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.372+0000 7f494fe00640 1 --2- 172.21.15.31:0/437916327 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ccf69b9ea0 0x55ccf69bc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:09.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.372+0000 7f494fe00640 1 --2- 172.21.15.31:0/437916327 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ccf69b9ea0 0x55ccf69bc2c0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d96f68fedd0ae48 server_cookie=68de6d982264fbf in_seq=0 out_seq=0 2024-03-21T10:48:09.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.373+0000 7f493d600640 10 client.27979 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:09.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.373+0000 7f493d600640 1 -- 172.21.15.31:0/437916327 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55ccf689fae0 con 0x55ccf69b9ea0 2024-03-21T10:48:09.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.373+0000 7f493d600640 10 client.27979 handle_command_reply: tid=0 2024-03-21T10:48:09.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.373+0000 7f49597b8740 1 client.27979 shutdown 2024-03-21T10:48:09.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.373+0000 7f49597b8740 1 -- 172.21.15.31:0/437916327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccf6934f20 msgr2=0x55ccf6961170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:09.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.373+0000 7f49597b8740 1 --2- 172.21.15.31:0/437916327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccf6934f20 0x55ccf6961170 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x55ccf692a2b0 tx=0x7f494000eae0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.374+0000 7f49597b8740 1 -- 172.21.15.31:0/437916327 shutdown_connections 2024-03-21T10:48:09.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.374+0000 7f49597b8740 1 --2- 172.21.15.31:0/437916327 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55ccf69b9ea0 0x55ccf69bc2c0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.374+0000 7f49597b8740 1 --2- 172.21.15.31:0/437916327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ccf693d5d0 0x55ccf69616b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.374+0000 7f49597b8740 1 --2- 172.21.15.31:0/437916327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccf6934f20 0x55ccf6961170 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.374+0000 7f49597b8740 1 --2- 172.21.15.31:0/437916327 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccf6933860 0x55ccf6960c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.374+0000 7f49597b8740 1 -- 172.21.15.31:0/437916327 >> 172.21.15.31:0/437916327 conn(0x55ccf68fb200 msgr2=0x55ccf69308d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:09.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.375+0000 7f493d600640 0 client.27979 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:09.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.375+0000 7f49597b8740 1 -- 172.21.15.31:0/437916327 shutdown_connections 2024-03-21T10:48:09.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.375+0000 7f49597b8740 1 -- 172.21.15.31:0/437916327 wait complete. 2024-03-21T10:48:09.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.375+0000 7f49597b8740 20 client.27979 trim_cache size 0 max 16384 2024-03-21T10:48:09.373 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:48:09.373 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-21T10:48:09.373 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-03-21T10:48:09.373 INFO:teuthology.orchestra.run.smithi031.stdout: "ff07db78-f943-4bf3-9384-0bb095dfb72d": { 2024-03-21T10:48:09.373 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-03-21T10:48:09.373 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "ff07db78-f943-4bf3-9384-0bb095dfb72d", 2024-03-21T10:48:09.373 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-03-21T10:48:09.373 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:48:09.373 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:48:09.374 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:48:09.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.376+0000 7f4945c00640 1 -- 172.21.15.31:0/3904468857 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f49340a32c0 msgr2=0x7f49340a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:09.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.376+0000 7f4945c00640 1 --2- 172.21.15.31:0/3904468857 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f49340a32c0 0x7f49340a5780 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x55ccf69344e0 tx=0x7f4938005f90 comp rx=0 tx=0).stop 2024-03-21T10:48:09.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.376+0000 7f4945c00640 1 -- 172.21.15.31:0/3904468857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4950109690 msgr2=0x7f495010dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:09.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.376+0000 7f4945c00640 1 --2- 172.21.15.31:0/3904468857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4950109690 0x7f495010dae0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f4948037040 tx=0x7f4948032460 comp rx=0 tx=0).stop 2024-03-21T10:48:09.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.377+0000 7f4945c00640 1 -- 172.21.15.31:0/3904468857 shutdown_connections 2024-03-21T10:48:09.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.377+0000 7f4945c00640 1 --2- 172.21.15.31:0/3904468857 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f49340a32c0 0x7f49340a5780 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.377+0000 7f4945c00640 1 --2- 172.21.15.31:0/3904468857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4950109690 0x7f495010dae0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.377+0000 7f4945c00640 1 --2- 172.21.15.31:0/3904468857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4950104040 0x7f495010d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.377+0000 7f4945c00640 1 --2- 172.21.15.31:0/3904468857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4950103680 0x7f4950110450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:09.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.377+0000 7f4945c00640 1 -- 172.21.15.31:0/3904468857 >> 172.21.15.31:0/3904468857 conn(0x7f49500ff110 msgr2=0x7f49500ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:09.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.377+0000 7f4945c00640 1 -- 172.21.15.31:0/3904468857 shutdown_connections 2024-03-21T10:48:09.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:09.377+0000 7f4945c00640 1 -- 172.21.15.31:0/3904468857 wait complete. 2024-03-21T10:48:09.404 INFO:tasks.cephfs.filesystem:scrub status for tag:ff07db78-f943-4bf3-9384-0bb095dfb72d - {'path': '/', 'tag': 'ff07db78-f943-4bf3-9384-0bb095dfb72d', 'options': 'recursive,force'} 2024-03-21T10:48:09.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:09 smithi040 ceph-mon[36154]: pgmap v297: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 63 KiB/s wr, 27 op/s 2024-03-21T10:48:09.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:09 smithi040 ceph-mon[36154]: scrub queued for path: / 2024-03-21T10:48:09.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:09 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:48:09.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:09 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:48:09.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:09 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:48:09.681 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:09 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:48:09.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:09 smithi031 ceph-mon[31673]: pgmap v297: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 63 KiB/s wr, 27 op/s 2024-03-21T10:48:09.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:09 smithi031 ceph-mon[31673]: scrub queued for path: / 2024-03-21T10:48:09.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:09 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:48:09.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:09 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:48:09.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:09 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:48:09.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:09 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:48:09.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:09 smithi070 ceph-mon[36330]: pgmap v297: 129 pgs: 129 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 63 KiB/s wr, 27 op/s 2024-03-21T10:48:09.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:09 smithi070 ceph-mon[36330]: scrub queued for path: / 2024-03-21T10:48:09.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:09 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:48:09.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:09 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:48:09.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:09 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:48:09.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:09 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:48:10.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:10 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:48:10.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:10 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:48:10.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:10 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:48:10.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:10 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:48:10.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:10 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:48:10.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:10 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:48:10.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:10 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:48:10.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:10 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:48:10.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:10 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:48:11.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:11 smithi040 ceph-mon[36154]: pgmap v298: 129 pgs: 10 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 103 active+clean; 37 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 59 KiB/s wr, 30 op/s 2024-03-21T10:48:11.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:11 smithi031 ceph-mon[31673]: pgmap v298: 129 pgs: 10 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 103 active+clean; 37 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 59 KiB/s wr, 30 op/s 2024-03-21T10:48:11.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:11 smithi070 ceph-mon[36330]: pgmap v298: 129 pgs: 10 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 103 active+clean; 37 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 59 KiB/s wr, 30 op/s 2024-03-21T10:48:12.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.176+0000 7f0324400640 1 -- 172.21.15.31:0/2851202589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f031c074390 msgr2=0x7f031c10d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:12.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.176+0000 7f0324400640 1 --2- 172.21.15.31:0/2851202589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f031c074390 0x7f031c10d9b0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f0308009900 tx=0x7f030802bcd0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.177+0000 7f0324400640 1 -- 172.21.15.31:0/2851202589 shutdown_connections 2024-03-21T10:48:12.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.177+0000 7f0324400640 1 --2- 172.21.15.31:0/2851202589 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f031c10def0 0x7f031c110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.177+0000 7f0324400640 1 --2- 172.21.15.31:0/2851202589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f031c074390 0x7f031c10d9b0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.177+0000 7f0324400640 1 --2- 172.21.15.31:0/2851202589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f031c073a70 0x7f031c073e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.177+0000 7f0324400640 1 -- 172.21.15.31:0/2851202589 >> 172.21.15.31:0/2851202589 conn(0x7f031c0fd120 msgr2=0x7f031c0ff540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:12.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.177+0000 7f0324400640 1 -- 172.21.15.31:0/2851202589 shutdown_connections 2024-03-21T10:48:12.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.177+0000 7f0324400640 1 -- 172.21.15.31:0/2851202589 wait complete. 2024-03-21T10:48:12.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.178+0000 7f0324400640 1 Processor -- start 2024-03-21T10:48:12.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.178+0000 7f0324400640 1 -- start start 2024-03-21T10:48:12.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.178+0000 7f0324400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f031c073a70 0x7f031c19a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:12.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.178+0000 7f0324400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f031c074390 0x7f031c19a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:12.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.178+0000 7f0324400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f031c10def0 0x7f031c19ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:12.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.178+0000 7f0324400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031c113cc0 con 0x7f031c073a70 2024-03-21T10:48:12.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.178+0000 7f0324400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031c113b40 con 0x7f031c074390 2024-03-21T10:48:12.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.178+0000 7f0324400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031c113e40 con 0x7f031c10def0 2024-03-21T10:48:12.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.178+0000 7f0321a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f031c073a70 0x7f031c19a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:12.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.178+0000 7f0321000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f031c074390 0x7f031c19a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:12.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.179+0000 7f0321000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f031c074390 0x7f031c19a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:40824/0 (socket says 172.21.15.31:40824) 2024-03-21T10:48:12.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.179+0000 7f0321000640 1 -- 172.21.15.31:0/3469089428 learned_addr learned my addr 172.21.15.31:0/3469089428 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:12.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.179+0000 7f0322400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f031c10def0 0x7f031c19ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:12.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.179+0000 7f0321000640 1 -- 172.21.15.31:0/3469089428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f031c10def0 msgr2=0x7f031c19ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:12.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.179+0000 7f0321000640 1 --2- 172.21.15.31:0/3469089428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f031c10def0 0x7f031c19ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.179+0000 7f0321000640 1 -- 172.21.15.31:0/3469089428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f031c073a70 msgr2=0x7f031c19a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:12.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.179+0000 7f0321000640 1 --2- 172.21.15.31:0/3469089428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f031c073a70 0x7f031c19a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.179+0000 7f0321000640 1 -- 172.21.15.31:0/3469089428 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f031c19f1c0 con 0x7f031c074390 2024-03-21T10:48:12.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.179+0000 7f0321000640 1 --2- 172.21.15.31:0/3469089428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f031c074390 0x7f031c19a760 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f0308000c00 tx=0x7f0308004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:12.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.179+0000 7f031a000640 1 -- 172.21.15.31:0/3469089428 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0308042020 con 0x7f031c074390 2024-03-21T10:48:12.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.180+0000 7f031a000640 1 -- 172.21.15.31:0/3469089428 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f03080054e0 con 0x7f031c074390 2024-03-21T10:48:12.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.180+0000 7f0324400640 1 -- 172.21.15.31:0/3469089428 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f031c19f450 con 0x7f031c074390 2024-03-21T10:48:12.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.180+0000 7f031a000640 1 -- 172.21.15.31:0/3469089428 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0308003b50 con 0x7f031c074390 2024-03-21T10:48:12.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.180+0000 7f0324400640 1 -- 172.21.15.31:0/3469089428 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f031c1a6d40 con 0x7f031c074390 2024-03-21T10:48:12.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.183+0000 7f0326ffa740 1 Processor -- start 2024-03-21T10:48:12.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.183+0000 7f031a000640 1 -- 172.21.15.31:0/3469089428 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f0308045020 con 0x7f031c074390 2024-03-21T10:48:12.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.183+0000 7f0326ffa740 1 -- start start 2024-03-21T10:48:12.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.183+0000 7f0326ffa740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555860ff4fd0 0x555860ff53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:12.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.183+0000 7f0326ffa740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555860ff58f0 0x555860ffe180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:12.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.183+0000 7f0326ffa740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ffe6c0 0x555861000ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:12.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.183+0000 7f0326ffa740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555860fb73b0 con 0x555860ff58f0 2024-03-21T10:48:12.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.183+0000 7f0326ffa740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555860f59830 con 0x555860ffe6c0 2024-03-21T10:48:12.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.183+0000 7f0326ffa740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x555860f025d0 con 0x555860ff4fd0 2024-03-21T10:48:12.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.184+0000 7f0321000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ffe6c0 0x555861000ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:12.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.184+0000 7f0321a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555860ff58f0 0x555860ffe180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:12.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.184+0000 7f0321000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ffe6c0 0x555861000ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:40828/0 (socket says 172.21.15.31:40828) 2024-03-21T10:48:12.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.184+0000 7f0321000640 1 -- 172.21.15.31:0/208052967 learned_addr learned my addr 172.21.15.31:0/208052967 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:12.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.184+0000 7f0322400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555860ff4fd0 0x555860ff53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:12.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.184+0000 7f0321a00640 1 -- 172.21.15.31:0/208052967 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555860ff4fd0 msgr2=0x555860ff53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:12.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.184+0000 7f0321a00640 1 --2- 172.21.15.31:0/208052967 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555860ff4fd0 0x555860ff53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.184+0000 7f0321a00640 1 -- 172.21.15.31:0/208052967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ffe6c0 msgr2=0x555861000ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:12.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.184+0000 7f0321a00640 1 --2- 172.21.15.31:0/208052967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ffe6c0 0x555861000ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.184+0000 7f0321a00640 1 -- 172.21.15.31:0/208052967 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555860f1a2c0 con 0x555860ff58f0 2024-03-21T10:48:12.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.184+0000 7f0321000640 1 --2- 172.21.15.31:0/208052967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ffe6c0 0x555861000ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:12.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.185+0000 7f0321a00640 1 --2- 172.21.15.31:0/208052967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555860ff58f0 0x555860ffe180 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0310009c00 tx=0x7f0310011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0a84f8c515ef893 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:12.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.185+0000 7f031a000640 1 --2- 172.21.15.31:0/3469089428 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f03040a3100 0x7f03040a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:12.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.185+0000 7f030ea00640 1 -- 172.21.15.31:0/208052967 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0310004230 con 0x555860ff58f0 2024-03-21T10:48:12.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.185+0000 7f031a000640 1 -- 172.21.15.31:0/3469089428 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f03080f1600 con 0x7f031c074390 2024-03-21T10:48:12.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.185+0000 7f030ea00640 1 -- 172.21.15.31:0/208052967 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f03100043d0 con 0x555860ff58f0 2024-03-21T10:48:12.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.185+0000 7f030ea00640 1 -- 172.21.15.31:0/208052967 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0310004d70 con 0x555860ff58f0 2024-03-21T10:48:12.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.185+0000 7f0322400640 1 --2- 172.21.15.31:0/3469089428 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f03040a3100 0x7f03040a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:12.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.186+0000 7f0326ffa740 1 -- 172.21.15.31:0/208052967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555860ff58f0 msgr2=0x555860ffe180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:12.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.186+0000 7f0326ffa740 1 --2- 172.21.15.31:0/208052967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555860ff58f0 0x555860ffe180 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0310009c00 tx=0x7f0310011810 comp rx=0 tx=0).stop 2024-03-21T10:48:12.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.186+0000 7f0326ffa740 1 -- 172.21.15.31:0/208052967 shutdown_connections 2024-03-21T10:48:12.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.186+0000 7f0326ffa740 1 --2- 172.21.15.31:0/208052967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ffe6c0 0x555861000ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.186+0000 7f0326ffa740 1 --2- 172.21.15.31:0/208052967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555860ff58f0 0x555860ffe180 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.186+0000 7f0326ffa740 1 --2- 172.21.15.31:0/208052967 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555860ff4fd0 0x555860ff53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.186+0000 7f0326ffa740 1 -- 172.21.15.31:0/208052967 >> 172.21.15.31:0/208052967 conn(0x555860fef8f0 msgr2=0x555860fefcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.186+0000 7f0326ffa740 1 -- 172.21.15.31:0/208052967 shutdown_connections 2024-03-21T10:48:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.186+0000 7f0322400640 1 --2- 172.21.15.31:0/3469089428 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f03040a3100 0x7f03040a55c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f031c06ad70 tx=0x7f0314009500 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.187+0000 7f0326ffa740 1 -- 172.21.15.31:0/208052967 wait complete. 2024-03-21T10:48:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.187+0000 7f0326ffa740 1 -- start start 2024-03-21T10:48:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.187+0000 7f0326ffa740 1 Processor -- start 2024-03-21T10:48:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.188+0000 7f0326ffa740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ff4fd0 0x555860ffb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.188+0000 7f0326ffa740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555860ff58f0 0x555860ffbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.188+0000 7f0326ffa740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555860ffe6c0 0x555860ffc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.188+0000 7f0326ffa740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555860f15a20 con 0x555860ffe6c0 2024-03-21T10:48:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.188+0000 7f0326ffa740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555860f558d0 con 0x555860ff4fd0 2024-03-21T10:48:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.188+0000 7f0326ffa740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558610012f0 con 0x555860ff58f0 2024-03-21T10:48:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.188+0000 7f0326ffa740 10 client.0 _ll_register_callbacks cb 0x555860f521c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.188+0000 7f0321a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ff4fd0 0x555860ffb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.188+0000 7f0321000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555860ff58f0 0x555860ffbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.188+0000 7f0321a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ff4fd0 0x555860ffb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:40830/0 (socket says 172.21.15.31:40830) 2024-03-21T10:48:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.188+0000 7f0321a00640 1 -- 172.21.15.31:0/731859063 learned_addr learned my addr 172.21.15.31:0/731859063 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.188+0000 7f0322400640 1 --2- 172.21.15.31:0/731859063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555860ffe6c0 0x555860ffc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.188+0000 7f0321000640 1 -- 172.21.15.31:0/731859063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ff4fd0 msgr2=0x555860ffb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.188+0000 7f0321000640 1 --2- 172.21.15.31:0/731859063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ff4fd0 0x555860ffb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.188+0000 7f0321000640 1 -- 172.21.15.31:0/731859063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555860ffe6c0 msgr2=0x555860ffc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.188+0000 7f0321000640 1 --2- 172.21.15.31:0/731859063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555860ffe6c0 0x555860ffc0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.188+0000 7f0321000640 1 -- 172.21.15.31:0/731859063 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555860c6e210 con 0x555860ff58f0 2024-03-21T10:48:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.189+0000 7f0321a00640 1 --2- 172.21.15.31:0/731859063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ff4fd0 0x555860ffb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.189+0000 7f0322400640 1 --2- 172.21.15.31:0/731859063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555860ffe6c0 0x555860ffc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:48:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.189+0000 7f0321000640 1 --2- 172.21.15.31:0/731859063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555860ff58f0 0x555860ffbba0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f03080eef60 tx=0x7f0308005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3d0a27a1d761a7d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.189+0000 7f0303400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:48:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.189+0000 7f0303400640 1 -- 172.21.15.31:0/731859063 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03080b99b0 con 0x555860ff58f0 2024-03-21T10:48:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.189+0000 7f0303400640 1 -- 172.21.15.31:0/731859063 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f03080c15b0 con 0x555860ff58f0 2024-03-21T10:48:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.189+0000 7f0326ffa740 1 -- 172.21.15.31:0/731859063 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f03100043d0 con 0x555860ff58f0 2024-03-21T10:48:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.189+0000 7f0303400640 1 -- 172.21.15.31:0/731859063 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03080bd070 con 0x555860ff58f0 2024-03-21T10:48:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.189+0000 7f0326ffa740 10 client.27991 fetch_fsmap learned FSMap version 28 2024-03-21T10:48:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.189+0000 7f0326ffa740 1 -- 172.21.15.31:0/731859063 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555860d3a620 con 0x555860ff58f0 2024-03-21T10:48:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.190+0000 7f0303400640 1 -- 172.21.15.31:0/731859063 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f03100043d0 con 0x555860ff58f0 2024-03-21T10:48:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.190+0000 7f0303400640 1 -- 172.21.15.31:0/731859063 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f03080eebe0 con 0x555860ff58f0 2024-03-21T10:48:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.190+0000 7f0326ffa740 10 client.27991 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:48:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.190+0000 7f0326ffa740 10 client.27991 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:48:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.190+0000 7f0326ffa740 20 client.27991 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.190+0000 7f0326ffa740 1 --2- 172.21.15.31:0/731859063 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55586100e6d0 0x555861010ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.190+0000 7f0326ffa740 4 client.27991 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:48:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.190+0000 7f0326ffa740 1 -- 172.21.15.31:0/731859063 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555860f66d70 con 0x55586100e6d0 2024-03-21T10:48:12.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.191+0000 7f0321a00640 1 --2- 172.21.15.31:0/731859063 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55586100e6d0 0x555861010ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:12.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.191+0000 7f0303400640 10 client.27991 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.191+0000 7f0321a00640 1 --2- 172.21.15.31:0/731859063 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55586100e6d0 0x555861010ac0 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ad4c5dde2c28bd1 server_cookie=f02cfb9ff1c764bf in_seq=0 out_seq=0 2024-03-21T10:48:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.192+0000 7f0303400640 1 -- 172.21.15.31:0/731859063 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555860f66d70 con 0x55586100e6d0 2024-03-21T10:48:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.192+0000 7f0303400640 10 client.27991 handle_command_reply: tid=0 2024-03-21T10:48:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.192+0000 7f0326ffa740 1 client.27991 shutdown 2024-03-21T10:48:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.192+0000 7f0326ffa740 1 -- 172.21.15.31:0/731859063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555860ff58f0 msgr2=0x555860ffbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.192+0000 7f0326ffa740 1 --2- 172.21.15.31:0/731859063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555860ff58f0 0x555860ffbba0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f03080eef60 tx=0x7f0308005b50 comp rx=0 tx=0).stop 2024-03-21T10:48:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.192+0000 7f0326ffa740 1 -- 172.21.15.31:0/731859063 shutdown_connections 2024-03-21T10:48:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.192+0000 7f0326ffa740 1 --2- 172.21.15.31:0/731859063 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55586100e6d0 0x555861010ac0 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.192+0000 7f0326ffa740 1 --2- 172.21.15.31:0/731859063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555860ffe6c0 0x555860ffc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.192+0000 7f0326ffa740 1 --2- 172.21.15.31:0/731859063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555860ff58f0 0x555860ffbba0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.192+0000 7f0326ffa740 1 --2- 172.21.15.31:0/731859063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ff4fd0 0x555860ffb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.192+0000 7f0303400640 0 client.27991 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.192+0000 7f0326ffa740 1 -- 172.21.15.31:0/731859063 >> 172.21.15.31:0/731859063 conn(0x555860fef8f0 msgr2=0x555860ff1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.193+0000 7f0326ffa740 1 -- 172.21.15.31:0/731859063 shutdown_connections 2024-03-21T10:48:12.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.193+0000 7f0326ffa740 1 -- 172.21.15.31:0/731859063 wait complete. 2024-03-21T10:48:12.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.193+0000 7f0326ffa740 20 client.27991 trim_cache size 0 max 16384 2024-03-21T10:48:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.209+0000 7f0326ffa740 1 Processor -- start 2024-03-21T10:48:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.209+0000 7f0326ffa740 1 -- start start 2024-03-21T10:48:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.209+0000 7f0326ffa740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555860ff4fd0 0x555860ffdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.209+0000 7f0326ffa740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555860ff58f0 0x555860fefcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.209+0000 7f0326ffa740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ffe6c0 0x555860ffbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.209+0000 7f0326ffa740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031c1b3220 con 0x555860ff4fd0 2024-03-21T10:48:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.209+0000 7f0326ffa740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031c1b0d20 con 0x555860ffe6c0 2024-03-21T10:48:12.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.209+0000 7f0326ffa740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031c1aede0 con 0x555860ff58f0 2024-03-21T10:48:12.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.209+0000 7f0322400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ffe6c0 0x555860ffbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:12.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.209+0000 7f0321000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555860ff58f0 0x555860fefcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:12.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.209+0000 7f0322400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ffe6c0 0x555860ffbf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:40836/0 (socket says 172.21.15.31:40836) 2024-03-21T10:48:12.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.209+0000 7f0322400640 1 -- 172.21.15.31:0/3266313206 learned_addr learned my addr 172.21.15.31:0/3266313206 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:12.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.209+0000 7f0321a00640 1 --2- 172.21.15.31:0/3266313206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555860ff4fd0 0x555860ffdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:12.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.210+0000 7f0322400640 1 -- 172.21.15.31:0/3266313206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555860ff58f0 msgr2=0x555860fefcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:12.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.210+0000 7f0322400640 1 --2- 172.21.15.31:0/3266313206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555860ff58f0 0x555860fefcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.210+0000 7f0322400640 1 -- 172.21.15.31:0/3266313206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555860ff4fd0 msgr2=0x555860ffdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:12.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.210+0000 7f0322400640 1 --2- 172.21.15.31:0/3266313206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555860ff4fd0 0x555860ffdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.210+0000 7f0322400640 1 -- 172.21.15.31:0/3266313206 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555860ffa610 con 0x555860ffe6c0 2024-03-21T10:48:12.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.210+0000 7f0321000640 1 --2- 172.21.15.31:0/3266313206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555860ff58f0 0x555860fefcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:12.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.210+0000 7f0321a00640 1 --2- 172.21.15.31:0/3266313206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555860ff4fd0 0x555860ffdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.210+0000 7f0322400640 1 --2- 172.21.15.31:0/3266313206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ffe6c0 0x555860ffbf00 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f0314003d10 tx=0x7f0314016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc66b87a061b964d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.210+0000 7f030e000640 1 -- 172.21.15.31:0/3266313206 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03140045c0 con 0x555860ffe6c0 2024-03-21T10:48:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.211+0000 7f030e000640 1 -- 172.21.15.31:0/3266313206 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0314004760 con 0x555860ffe6c0 2024-03-21T10:48:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.211+0000 7f030e000640 1 -- 172.21.15.31:0/3266313206 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f031401e5f0 con 0x555860ffe6c0 2024-03-21T10:48:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.211+0000 7f0326ffa740 1 -- 172.21.15.31:0/3266313206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ffe6c0 msgr2=0x555860ffbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.211+0000 7f0326ffa740 1 --2- 172.21.15.31:0/3266313206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ffe6c0 0x555860ffbf00 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f0314003d10 tx=0x7f0314016000 comp rx=0 tx=0).stop 2024-03-21T10:48:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.211+0000 7f0326ffa740 1 -- 172.21.15.31:0/3266313206 shutdown_connections 2024-03-21T10:48:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.211+0000 7f0326ffa740 1 --2- 172.21.15.31:0/3266313206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ffe6c0 0x555860ffbf00 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.211+0000 7f0326ffa740 1 --2- 172.21.15.31:0/3266313206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555860ff58f0 0x555860fefcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.211+0000 7f0326ffa740 1 --2- 172.21.15.31:0/3266313206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555860ff4fd0 0x555860ffdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.211+0000 7f0326ffa740 1 -- 172.21.15.31:0/3266313206 >> 172.21.15.31:0/3266313206 conn(0x555860fef8f0 msgr2=0x555860fee950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.211+0000 7f0326ffa740 1 -- 172.21.15.31:0/3266313206 shutdown_connections 2024-03-21T10:48:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.211+0000 7f0326ffa740 1 -- 172.21.15.31:0/3266313206 wait complete. 2024-03-21T10:48:12.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.212+0000 7f0326ffa740 1 -- start start 2024-03-21T10:48:12.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.212+0000 7f0326ffa740 1 Processor -- start 2024-03-21T10:48:12.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.212+0000 7f0326ffa740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555860ff4fd0 0x555861005d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:12.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.212+0000 7f0326ffa740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555860ff58f0 0x555861006240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:12.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.212+0000 7f0326ffa740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ffe6c0 0x555861002110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:12.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.212+0000 7f0326ffa740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031c1af3e0 con 0x555860ff4fd0 2024-03-21T10:48:12.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.213+0000 7f0326ffa740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031c1b06c0 con 0x555860ffe6c0 2024-03-21T10:48:12.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.213+0000 7f0326ffa740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031c1afa00 con 0x555860ff58f0 2024-03-21T10:48:12.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.213+0000 7f0326ffa740 10 client.0 _ll_register_callbacks cb 0x7f03080f7240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:12.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.213+0000 7f0321a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555860ff4fd0 0x555861005d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:12.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.213+0000 7f0322400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ffe6c0 0x555861002110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:12.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.213+0000 7f0321a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555860ff4fd0 0x555861005d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46314/0 (socket says 172.21.15.31:46314) 2024-03-21T10:48:12.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.213+0000 7f0321a00640 1 -- 172.21.15.31:0/4028443120 learned_addr learned my addr 172.21.15.31:0/4028443120 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:12.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.213+0000 7f0321000640 1 --2- 172.21.15.31:0/4028443120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555860ff58f0 0x555861006240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:12.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.213+0000 7f0321a00640 1 -- 172.21.15.31:0/4028443120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555860ff58f0 msgr2=0x555861006240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:12.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.213+0000 7f0321a00640 1 --2- 172.21.15.31:0/4028443120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555860ff58f0 0x555861006240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.213+0000 7f0321a00640 1 -- 172.21.15.31:0/4028443120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ffe6c0 msgr2=0x555861002110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:12.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.213+0000 7f0321a00640 1 --2- 172.21.15.31:0/4028443120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ffe6c0 0x555861002110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.213+0000 7f0321a00640 1 -- 172.21.15.31:0/4028443120 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555860ff5720 con 0x555860ff4fd0 2024-03-21T10:48:12.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.213+0000 7f0322400640 1 --2- 172.21.15.31:0/4028443120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ffe6c0 0x555861002110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:12.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.213+0000 7f0321000640 1 --2- 172.21.15.31:0/4028443120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555860ff58f0 0x555861006240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:12.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.214+0000 7f0321a00640 1 --2- 172.21.15.31:0/4028443120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555860ff4fd0 0x555861005d00 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f03100058c0 tx=0x7f0310004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca8bbad58d02a277 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:12.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.214+0000 7f0303400640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:48:12.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.214+0000 7f0303400640 1 -- 172.21.15.31:0/4028443120 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f031001e040 con 0x555860ff4fd0 2024-03-21T10:48:12.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.214+0000 7f0326ffa740 1 -- 172.21.15.31:0/4028443120 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0314004760 con 0x555860ff4fd0 2024-03-21T10:48:12.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.214+0000 7f0303400640 1 -- 172.21.15.31:0/4028443120 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f0310019900 con 0x555860ff4fd0 2024-03-21T10:48:12.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.214+0000 7f0303400640 1 -- 172.21.15.31:0/4028443120 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0310019b00 con 0x555860ff4fd0 2024-03-21T10:48:12.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.214+0000 7f0326ffa740 10 client.17433 fetch_fsmap learned FSMap version 28 2024-03-21T10:48:12.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.214+0000 7f0326ffa740 1 -- 172.21.15.31:0/4028443120 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55586106a870 con 0x555860ff4fd0 2024-03-21T10:48:12.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.215+0000 7f0303400640 1 -- 172.21.15.31:0/4028443120 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0314004760 con 0x555860ff4fd0 2024-03-21T10:48:12.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.215+0000 7f0303400640 1 -- 172.21.15.31:0/4028443120 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f0310024040 con 0x555860ff4fd0 2024-03-21T10:48:12.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.215+0000 7f0326ffa740 10 client.17433 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:48:12.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.215+0000 7f0326ffa740 10 client.17433 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:48:12.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.215+0000 7f0326ffa740 20 client.17433 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:12.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.215+0000 7f0326ffa740 1 --2- 172.21.15.31:0/4028443120 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55586106ae00 0x55586106d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:12.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.215+0000 7f0326ffa740 4 client.17433 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:48:12.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.215+0000 7f0326ffa740 1 -- 172.21.15.31:0/4028443120 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555860f610c0 con 0x55586106ae00 2024-03-21T10:48:12.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.215+0000 7f0321000640 1 --2- 172.21.15.31:0/4028443120 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55586106ae00 0x55586106d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:12.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.216+0000 7f0321000640 1 --2- 172.21.15.31:0/4028443120 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55586106ae00 0x55586106d220 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3802142940ebcd3f server_cookie=b7fd62a4d8bbbc9d in_seq=0 out_seq=0 2024-03-21T10:48:12.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.216+0000 7f0303400640 10 client.17433 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:12.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.223+0000 7f0303400640 1 -- 172.21.15.31:0/4028443120 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555860f610c0 con 0x55586106ae00 2024-03-21T10:48:12.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.223+0000 7f0303400640 10 client.17433 handle_command_reply: tid=0 2024-03-21T10:48:12.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.224+0000 7f0326ffa740 1 client.17433 shutdown 2024-03-21T10:48:12.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.224+0000 7f0326ffa740 1 -- 172.21.15.31:0/4028443120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555860ff4fd0 msgr2=0x555861005d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:12.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.224+0000 7f0326ffa740 1 --2- 172.21.15.31:0/4028443120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555860ff4fd0 0x555861005d00 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f03100058c0 tx=0x7f0310004690 comp rx=0 tx=0).stop 2024-03-21T10:48:12.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.224+0000 7f0326ffa740 1 -- 172.21.15.31:0/4028443120 shutdown_connections 2024-03-21T10:48:12.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.224+0000 7f0326ffa740 1 --2- 172.21.15.31:0/4028443120 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55586106ae00 0x55586106d220 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.224+0000 7f0326ffa740 1 --2- 172.21.15.31:0/4028443120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555860ffe6c0 0x555861002110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.224+0000 7f0326ffa740 1 --2- 172.21.15.31:0/4028443120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555860ff58f0 0x555861006240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.224+0000 7f0326ffa740 1 --2- 172.21.15.31:0/4028443120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555860ff4fd0 0x555861005d00 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.224+0000 7f0303400640 0 client.17433 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:12.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.224+0000 7f0326ffa740 1 -- 172.21.15.31:0/4028443120 >> 172.21.15.31:0/4028443120 conn(0x555860fef8f0 msgr2=0x555860fee950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:12.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.224+0000 7f0326ffa740 1 -- 172.21.15.31:0/4028443120 shutdown_connections 2024-03-21T10:48:12.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.224+0000 7f0326ffa740 1 -- 172.21.15.31:0/4028443120 wait complete. 2024-03-21T10:48:12.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.224+0000 7f0326ffa740 20 client.17433 trim_cache size 0 max 16384 2024-03-21T10:48:12.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:48:12.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:48:12.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:48:12.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:48:12.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.225+0000 7f0324400640 1 -- 172.21.15.31:0/3469089428 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f03040a3100 msgr2=0x7f03040a55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:12.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.225+0000 7f0324400640 1 --2- 172.21.15.31:0/3469089428 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f03040a3100 0x7f03040a55c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f031c06ad70 tx=0x7f0314009500 comp rx=0 tx=0).stop 2024-03-21T10:48:12.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.225+0000 7f0324400640 1 -- 172.21.15.31:0/3469089428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f031c074390 msgr2=0x7f031c19a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:12.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.225+0000 7f0324400640 1 --2- 172.21.15.31:0/3469089428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f031c074390 0x7f031c19a760 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f0308000c00 tx=0x7f0308004360 comp rx=0 tx=0).stop 2024-03-21T10:48:12.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.225+0000 7f0324400640 1 -- 172.21.15.31:0/3469089428 shutdown_connections 2024-03-21T10:48:12.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.225+0000 7f0324400640 1 --2- 172.21.15.31:0/3469089428 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f03040a3100 0x7f03040a55c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.225+0000 7f0324400640 1 --2- 172.21.15.31:0/3469089428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f031c10def0 0x7f031c19ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.225+0000 7f0324400640 1 --2- 172.21.15.31:0/3469089428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f031c074390 0x7f031c19a760 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.225+0000 7f0324400640 1 --2- 172.21.15.31:0/3469089428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f031c073a70 0x7f031c19a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:12.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.225+0000 7f0324400640 1 -- 172.21.15.31:0/3469089428 >> 172.21.15.31:0/3469089428 conn(0x7f031c0fd120 msgr2=0x7f031c0fe950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:12.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.225+0000 7f0324400640 1 -- 172.21.15.31:0/3469089428 shutdown_connections 2024-03-21T10:48:12.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:12.225+0000 7f0324400640 1 -- 172.21.15.31:0/3469089428 wait complete. 2024-03-21T10:48:13.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:13 smithi040 ceph-mon[36154]: pgmap v299: 129 pgs: 10 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 103 active+clean; 36 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 59 KiB/s wr, 30 op/s 2024-03-21T10:48:13.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:13 smithi031 ceph-mon[31673]: pgmap v299: 129 pgs: 10 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 103 active+clean; 36 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 59 KiB/s wr, 30 op/s 2024-03-21T10:48:13.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:13 smithi070 ceph-mon[36330]: pgmap v299: 129 pgs: 10 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 103 active+clean; 36 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 59 KiB/s wr, 30 op/s 2024-03-21T10:48:15.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:15 smithi040 ceph-mon[36154]: pgmap v300: 129 pgs: 7 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 110 active+clean; 36 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 35 KiB/s wr, 14 op/s 2024-03-21T10:48:15.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:15 smithi031 ceph-mon[31673]: pgmap v300: 129 pgs: 7 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 110 active+clean; 36 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 35 KiB/s wr, 14 op/s 2024-03-21T10:48:15.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:15 smithi070 ceph-mon[36330]: pgmap v300: 129 pgs: 7 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 110 active+clean; 36 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 35 KiB/s wr, 14 op/s 2024-03-21T10:48:17.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:17 smithi040 ceph-mon[36154]: pgmap v301: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 KiB/s wr, 12 op/s 2024-03-21T10:48:17.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:17 smithi031 ceph-mon[31673]: pgmap v301: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 KiB/s wr, 12 op/s 2024-03-21T10:48:17.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:17 smithi070 ceph-mon[36330]: pgmap v301: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 KiB/s wr, 12 op/s 2024-03-21T10:48:19.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:19 smithi040 ceph-mon[36154]: pgmap v302: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 6.2 KiB/s wr, 10 op/s 2024-03-21T10:48:19.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:19 smithi031 ceph-mon[31673]: pgmap v302: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 6.2 KiB/s wr, 10 op/s 2024-03-21T10:48:19.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:19 smithi070 ceph-mon[36330]: pgmap v302: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 6.2 KiB/s wr, 10 op/s 2024-03-21T10:48:21.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:21 smithi040 ceph-mon[36154]: pgmap v303: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-03-21T10:48:21.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:21 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:48:21.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:21 smithi040 ceph-mon[36154]: osdmap e102: 12 total, 12 up, 12 in 2024-03-21T10:48:21.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:21 smithi031 ceph-mon[31673]: pgmap v303: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-03-21T10:48:21.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:21 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:48:21.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:21 smithi031 ceph-mon[31673]: osdmap e102: 12 total, 12 up, 12 in 2024-03-21T10:48:21.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:21 smithi070 ceph-mon[36330]: pgmap v303: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-03-21T10:48:21.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:21 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:48:21.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:21 smithi070 ceph-mon[36330]: osdmap e102: 12 total, 12 up, 12 in 2024-03-21T10:48:22.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:22 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:48:22.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:22 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:48:22.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:22 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:48:22.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:22 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:48:22.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:22 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:48:22.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:22 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:48:23.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:23 smithi040 ceph-mon[36154]: pgmap v305: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.4 KiB/s wr, 8 op/s 2024-03-21T10:48:23.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:23 smithi031 ceph-mon[31673]: pgmap v305: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.4 KiB/s wr, 8 op/s 2024-03-21T10:48:23.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:23 smithi070 ceph-mon[36330]: pgmap v305: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.4 KiB/s wr, 8 op/s 2024-03-21T10:48:24.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:24 smithi040 ceph-mon[36154]: pgmap v306: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 3.6 KiB/s wr, 7 op/s 2024-03-21T10:48:24.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:24 smithi031 ceph-mon[31673]: pgmap v306: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 3.6 KiB/s wr, 7 op/s 2024-03-21T10:48:24.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:24 smithi070 ceph-mon[36330]: pgmap v306: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 3.6 KiB/s wr, 7 op/s 2024-03-21T10:48:25.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.376+0000 7f7a6aa00640 1 -- 172.21.15.31:0/119129053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a64068f40 msgr2=0x7f7a64069320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:25.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.376+0000 7f7a6aa00640 1 --2- 172.21.15.31:0/119129053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a64068f40 0x7f7a64069320 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f7a54009900 tx=0x7f7a5402bc40 comp rx=0 tx=0).stop 2024-03-21T10:48:25.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.376+0000 7f7a6aa00640 1 -- 172.21.15.31:0/119129053 shutdown_connections 2024-03-21T10:48:25.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.376+0000 7f7a6aa00640 1 --2- 172.21.15.31:0/119129053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a6410e040 0x7f7a64110430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.376+0000 7f7a6aa00640 1 --2- 172.21.15.31:0/119129053 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a64069860 0x7f7a6410db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.376+0000 7f7a6aa00640 1 --2- 172.21.15.31:0/119129053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a64068f40 0x7f7a64069320 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.376+0000 7f7a6aa00640 1 -- 172.21.15.31:0/119129053 >> 172.21.15.31:0/119129053 conn(0x7f7a64077690 msgr2=0x7f7a64077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:25.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.376+0000 7f7a6aa00640 1 -- 172.21.15.31:0/119129053 shutdown_connections 2024-03-21T10:48:25.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.376+0000 7f7a6aa00640 1 -- 172.21.15.31:0/119129053 wait complete. 2024-03-21T10:48:25.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.377+0000 7f7a6aa00640 1 Processor -- start 2024-03-21T10:48:25.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.378+0000 7f7a6aa00640 1 -- start start 2024-03-21T10:48:25.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.378+0000 7f7a6aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a64068f40 0x7f7a6419a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:25.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.378+0000 7f7a6aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a64069860 0x7f7a6419a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:25.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.378+0000 7f7a6aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a6410e040 0x7f7a6419e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:25.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.378+0000 7f7a6aa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a64113c70 con 0x7f7a6410e040 2024-03-21T10:48:25.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.378+0000 7f7a6aa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a64113af0 con 0x7f7a64068f40 2024-03-21T10:48:25.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.378+0000 7f7a6aa00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a64113df0 con 0x7f7a64069860 2024-03-21T10:48:25.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.378+0000 7f7a63e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a64068f40 0x7f7a6419a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:25.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.378+0000 7f7a63e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a64068f40 0x7f7a6419a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46148/0 (socket says 172.21.15.31:46148) 2024-03-21T10:48:25.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.378+0000 7f7a63e00640 1 -- 172.21.15.31:0/2069408769 learned_addr learned my addr 172.21.15.31:0/2069408769 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:25.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.378+0000 7f7a63400640 1 --2- 172.21.15.31:0/2069408769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a64069860 0x7f7a6419a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:25.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.378+0000 7f7a68a00640 1 --2- 172.21.15.31:0/2069408769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a6410e040 0x7f7a6419e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:25.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.379+0000 7f7a63e00640 1 -- 172.21.15.31:0/2069408769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a64069860 msgr2=0x7f7a6419a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:25.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.379+0000 7f7a63e00640 1 --2- 172.21.15.31:0/2069408769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a64069860 0x7f7a6419a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.379+0000 7f7a63e00640 1 -- 172.21.15.31:0/2069408769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a6410e040 msgr2=0x7f7a6419e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:25.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.379+0000 7f7a63e00640 1 --2- 172.21.15.31:0/2069408769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a6410e040 0x7f7a6419e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.379+0000 7f7a63e00640 1 -- 172.21.15.31:0/2069408769 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6419f100 con 0x7f7a64068f40 2024-03-21T10:48:25.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.379+0000 7f7a63e00640 1 --2- 172.21.15.31:0/2069408769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a64068f40 0x7f7a6419a110 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f7a54005ab0 tx=0x7f7a540042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:25.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.379+0000 7f7a60c00640 1 -- 172.21.15.31:0/2069408769 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a54041070 con 0x7f7a64068f40 2024-03-21T10:48:25.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.379+0000 7f7a60c00640 1 -- 172.21.15.31:0/2069408769 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a5402cca0 con 0x7f7a64068f40 2024-03-21T10:48:25.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.379+0000 7f7a6aa00640 1 -- 172.21.15.31:0/2069408769 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a6419f390 con 0x7f7a64068f40 2024-03-21T10:48:25.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.379+0000 7f7a60c00640 1 -- 172.21.15.31:0/2069408769 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a54003d90 con 0x7f7a64068f40 2024-03-21T10:48:25.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.380+0000 7f7a6aa00640 1 -- 172.21.15.31:0/2069408769 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a641a6c80 con 0x7f7a64068f40 2024-03-21T10:48:25.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.384+0000 7f7a6d815740 1 Processor -- start 2024-03-21T10:48:25.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.384+0000 7f7a60c00640 1 -- 172.21.15.31:0/2069408769 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f7a54044070 con 0x7f7a64068f40 2024-03-21T10:48:25.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.384+0000 7f7a6d815740 1 -- start start 2024-03-21T10:48:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.385+0000 7f7a6d815740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ea4805fd0 0x563ea48063b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.385+0000 7f7a6d815740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ea48068f0 0x563ea480f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.385+0000 7f7a6d815740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ea480f6c0 0x563ea4811ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.385+0000 7f7a6d815740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ea47c83b0 con 0x563ea4805fd0 2024-03-21T10:48:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.385+0000 7f7a6d815740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ea476a830 con 0x563ea480f6c0 2024-03-21T10:48:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.385+0000 7f7a6d815740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ea47135d0 con 0x563ea48068f0 2024-03-21T10:48:25.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.385+0000 7f7a63e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ea48068f0 0x563ea480f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:25.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.385+0000 7f7a68a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ea4805fd0 0x563ea48063b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:25.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.385+0000 7f7a63400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ea480f6c0 0x563ea4811ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:25.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.385+0000 7f7a63e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ea48068f0 0x563ea480f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53828/0 (socket says 172.21.15.31:53828) 2024-03-21T10:48:25.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.385+0000 7f7a68a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ea4805fd0 0x563ea48063b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54410/0 (socket says 172.21.15.31:54410) 2024-03-21T10:48:25.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.385+0000 7f7a63e00640 1 -- 172.21.15.31:0/1422457383 learned_addr learned my addr 172.21.15.31:0/1422457383 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.386+0000 7f7a63e00640 1 -- 172.21.15.31:0/1422457383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ea480f6c0 msgr2=0x563ea4811ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.386+0000 7f7a63e00640 1 --2- 172.21.15.31:0/1422457383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ea480f6c0 0x563ea4811ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.386+0000 7f7a63e00640 1 -- 172.21.15.31:0/1422457383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ea4805fd0 msgr2=0x563ea48063b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.386+0000 7f7a63e00640 1 --2- 172.21.15.31:0/1422457383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ea4805fd0 0x563ea48063b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.386+0000 7f7a63e00640 1 -- 172.21.15.31:0/1422457383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ea472b2c0 con 0x563ea48068f0 2024-03-21T10:48:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.386+0000 7f7a63e00640 1 --2- 172.21.15.31:0/1422457383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ea48068f0 0x563ea480f180 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f7a540ee800 tx=0x7f7a540f94d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2da37b8dd9b91b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.386+0000 7f7a59200640 1 -- 172.21.15.31:0/1422457383 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a540f32d0 con 0x563ea48068f0 2024-03-21T10:48:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.386+0000 7f7a59200640 1 -- 172.21.15.31:0/1422457383 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a540fe030 con 0x563ea48068f0 2024-03-21T10:48:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.386+0000 7f7a59200640 1 -- 172.21.15.31:0/1422457383 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a54103840 con 0x563ea48068f0 2024-03-21T10:48:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.387+0000 7f7a60c00640 1 --2- 172.21.15.31:0/2069408769 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7a480a3080 0x7f7a480a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.387+0000 7f7a6d815740 1 -- 172.21.15.31:0/1422457383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ea48068f0 msgr2=0x563ea480f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.387+0000 7f7a6d815740 1 --2- 172.21.15.31:0/1422457383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ea48068f0 0x563ea480f180 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f7a540ee800 tx=0x7f7a540f94d0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.387+0000 7f7a6d815740 1 -- 172.21.15.31:0/1422457383 shutdown_connections 2024-03-21T10:48:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.387+0000 7f7a6d815740 1 --2- 172.21.15.31:0/1422457383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ea480f6c0 0x563ea4811ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.387+0000 7f7a6d815740 1 --2- 172.21.15.31:0/1422457383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ea48068f0 0x563ea480f180 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.387+0000 7f7a6d815740 1 --2- 172.21.15.31:0/1422457383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ea4805fd0 0x563ea48063b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.387+0000 7f7a6d815740 1 -- 172.21.15.31:0/1422457383 >> 172.21.15.31:0/1422457383 conn(0x563ea48008f0 msgr2=0x563ea4800cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.387+0000 7f7a60c00640 1 -- 172.21.15.31:0/2069408769 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f7a540f1460 con 0x7f7a64068f40 2024-03-21T10:48:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.387+0000 7f7a68a00640 1 --2- 172.21.15.31:0/2069408769 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7a480a3080 0x7f7a480a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.387+0000 7f7a6d815740 1 -- 172.21.15.31:0/1422457383 shutdown_connections 2024-03-21T10:48:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.387+0000 7f7a6d815740 1 -- 172.21.15.31:0/1422457383 wait complete. 2024-03-21T10:48:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.388+0000 7f7a6d815740 1 -- start start 2024-03-21T10:48:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.388+0000 7f7a68a00640 1 --2- 172.21.15.31:0/2069408769 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7a480a3080 0x7f7a480a5540 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c005fd0 tx=0x7f7a5c009500 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.388+0000 7f7a6d815740 1 Processor -- start 2024-03-21T10:48:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.389+0000 7f7a6d815740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ea4805fd0 0x563ea480c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.389+0000 7f7a6d815740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ea48068f0 0x563ea480cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.389+0000 7f7a6d815740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ea480f6c0 0x563ea480d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.389+0000 7f7a6d815740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ea4726a20 con 0x563ea480f6c0 2024-03-21T10:48:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.389+0000 7f7a6d815740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ea47668d0 con 0x563ea48068f0 2024-03-21T10:48:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.389+0000 7f7a6d815740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ea48120b0 con 0x563ea4805fd0 2024-03-21T10:48:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.389+0000 7f7a63e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ea4805fd0 0x563ea480c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.389+0000 7f7a6d815740 10 client.0 _ll_register_callbacks cb 0x563ea47631c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.389+0000 7f7a63e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ea4805fd0 0x563ea480c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53832/0 (socket says 172.21.15.31:53832) 2024-03-21T10:48:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.389+0000 7f7a63e00640 1 -- 172.21.15.31:0/4082756237 learned_addr learned my addr 172.21.15.31:0/4082756237 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.389+0000 7f7a68a00640 1 --2- 172.21.15.31:0/4082756237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ea480f6c0 0x563ea480d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.389+0000 7f7a63400640 1 --2- 172.21.15.31:0/4082756237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ea48068f0 0x563ea480cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.390+0000 7f7a63e00640 1 -- 172.21.15.31:0/4082756237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ea48068f0 msgr2=0x563ea480cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.390+0000 7f7a63e00640 1 --2- 172.21.15.31:0/4082756237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ea48068f0 0x563ea480cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.390+0000 7f7a63e00640 1 -- 172.21.15.31:0/4082756237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ea480f6c0 msgr2=0x563ea480d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.390+0000 7f7a63e00640 1 --2- 172.21.15.31:0/4082756237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ea480f6c0 0x563ea480d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.390+0000 7f7a63e00640 1 -- 172.21.15.31:0/4082756237 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563ea447f210 con 0x563ea4805fd0 2024-03-21T10:48:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.390+0000 7f7a63e00640 1 --2- 172.21.15.31:0/4082756237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ea4805fd0 0x563ea480c6e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f7a540f99d0 tx=0x7f7a540f9f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=679783547a01e66e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.390+0000 7f7a4d600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:48:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.390+0000 7f7a4d600640 1 -- 172.21.15.31:0/4082756237 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5404b070 con 0x563ea4805fd0 2024-03-21T10:48:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.390+0000 7f7a6d815740 1 -- 172.21.15.31:0/4082756237 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7a540fe030 con 0x563ea4805fd0 2024-03-21T10:48:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.390+0000 7f7a4d600640 1 -- 172.21.15.31:0/4082756237 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f7a540035f0 con 0x563ea4805fd0 2024-03-21T10:48:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.390+0000 7f7a4d600640 1 -- 172.21.15.31:0/4082756237 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a540b98b0 con 0x563ea4805fd0 2024-03-21T10:48:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.390+0000 7f7a6d815740 10 client.28009 fetch_fsmap learned FSMap version 28 2024-03-21T10:48:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.390+0000 7f7a6d815740 1 -- 172.21.15.31:0/4082756237 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563ea454b620 con 0x563ea4805fd0 2024-03-21T10:48:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.391+0000 7f7a4d600640 1 -- 172.21.15.31:0/4082756237 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a540fe030 con 0x563ea4805fd0 2024-03-21T10:48:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.391+0000 7f7a4d600640 1 -- 172.21.15.31:0/4082756237 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f7a540b9c80 con 0x563ea4805fd0 2024-03-21T10:48:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.391+0000 7f7a6d815740 10 client.28009 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:48:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.391+0000 7f7a6d815740 10 client.28009 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:48:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.391+0000 7f7a6d815740 20 client.28009 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.391+0000 7f7a6d815740 1 --2- 172.21.15.31:0/4082756237 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563ea481f340 0x563ea4821750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.391+0000 7f7a6d815740 4 client.28009 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:48:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.391+0000 7f7a6d815740 1 -- 172.21.15.31:0/4082756237 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563ea4777d70 con 0x563ea481f340 2024-03-21T10:48:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.391+0000 7f7a63400640 1 --2- 172.21.15.31:0/4082756237 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563ea481f340 0x563ea4821750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.392+0000 7f7a63400640 1 --2- 172.21.15.31:0/4082756237 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563ea481f340 0x563ea4821750 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f50d78a93aa76aff server_cookie=f891fbfc0e289bf3 in_seq=0 out_seq=0 2024-03-21T10:48:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.392+0000 7f7a4d600640 10 client.28009 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.392+0000 7f7a4d600640 1 -- 172.21.15.31:0/4082756237 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563ea4777d70 con 0x563ea481f340 2024-03-21T10:48:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.392+0000 7f7a4d600640 10 client.28009 handle_command_reply: tid=0 2024-03-21T10:48:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.393+0000 7f7a6d815740 1 client.28009 shutdown 2024-03-21T10:48:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.393+0000 7f7a6d815740 1 -- 172.21.15.31:0/4082756237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ea4805fd0 msgr2=0x563ea480c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:25.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.393+0000 7f7a6d815740 1 --2- 172.21.15.31:0/4082756237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ea4805fd0 0x563ea480c6e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f7a540f99d0 tx=0x7f7a540f9f10 comp rx=0 tx=0).stop 2024-03-21T10:48:25.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.393+0000 7f7a6d815740 1 -- 172.21.15.31:0/4082756237 shutdown_connections 2024-03-21T10:48:25.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.393+0000 7f7a6d815740 1 --2- 172.21.15.31:0/4082756237 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563ea481f340 0x563ea4821750 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.393+0000 7f7a6d815740 1 --2- 172.21.15.31:0/4082756237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ea480f6c0 0x563ea480d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.393+0000 7f7a6d815740 1 --2- 172.21.15.31:0/4082756237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ea48068f0 0x563ea480cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.393+0000 7f7a6d815740 1 --2- 172.21.15.31:0/4082756237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ea4805fd0 0x563ea480c6e0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.393+0000 7f7a6d815740 1 -- 172.21.15.31:0/4082756237 >> 172.21.15.31:0/4082756237 conn(0x563ea48008f0 msgr2=0x563ea4802640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:25.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.393+0000 7f7a4d600640 0 client.28009 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:25.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.393+0000 7f7a6d815740 1 -- 172.21.15.31:0/4082756237 shutdown_connections 2024-03-21T10:48:25.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.393+0000 7f7a6d815740 1 -- 172.21.15.31:0/4082756237 wait complete. 2024-03-21T10:48:25.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.393+0000 7f7a6d815740 20 client.28009 trim_cache size 0 max 16384 2024-03-21T10:48:25.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.408+0000 7f7a6d815740 1 Processor -- start 2024-03-21T10:48:25.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.408+0000 7f7a6d815740 1 -- start start 2024-03-21T10:48:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.408+0000 7f7a6d815740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ea4805fd0 0x563ea480ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.409+0000 7f7a6d815740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ea48068f0 0x563ea480b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.409+0000 7f7a6d815740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ea480f6c0 0x563ea4806e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.409+0000 7f7a6d815740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a641b3040 con 0x563ea48068f0 2024-03-21T10:48:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.409+0000 7f7a6d815740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a641b0b40 con 0x563ea480f6c0 2024-03-21T10:48:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.409+0000 7f7a6d815740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a641aec00 con 0x563ea4805fd0 2024-03-21T10:48:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.409+0000 7f7a63e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ea4805fd0 0x563ea480ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.409+0000 7f7a63e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ea4805fd0 0x563ea480ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53838/0 (socket says 172.21.15.31:53838) 2024-03-21T10:48:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.409+0000 7f7a63e00640 1 -- 172.21.15.31:0/1364404764 learned_addr learned my addr 172.21.15.31:0/1364404764 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.409+0000 7f7a63400640 1 --2- 172.21.15.31:0/1364404764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ea48068f0 0x563ea480b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.409+0000 7f7a68a00640 1 --2- 172.21.15.31:0/1364404764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ea480f6c0 0x563ea4806e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:25.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.409+0000 7f7a63e00640 1 -- 172.21.15.31:0/1364404764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ea480f6c0 msgr2=0x563ea4806e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:25.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.409+0000 7f7a63e00640 1 --2- 172.21.15.31:0/1364404764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ea480f6c0 0x563ea4806e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.409+0000 7f7a63e00640 1 -- 172.21.15.31:0/1364404764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ea48068f0 msgr2=0x563ea480b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:25.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.409+0000 7f7a63e00640 1 --2- 172.21.15.31:0/1364404764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ea48068f0 0x563ea480b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.409+0000 7f7a63e00640 1 -- 172.21.15.31:0/1364404764 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ea47fed40 con 0x563ea4805fd0 2024-03-21T10:48:25.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.409+0000 7f7a63400640 1 --2- 172.21.15.31:0/1364404764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ea48068f0 0x563ea480b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:48:25.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.409+0000 7f7a63e00640 1 --2- 172.21.15.31:0/1364404764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ea4805fd0 0x563ea480ee10 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f7a5402c830 tx=0x7f7a5402cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2ef62d818c9cec2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:25.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.410+0000 7f7a4fe00640 1 -- 172.21.15.31:0/1364404764 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5404b070 con 0x563ea4805fd0 2024-03-21T10:48:25.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.410+0000 7f7a4fe00640 1 -- 172.21.15.31:0/1364404764 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a540fe020 con 0x563ea4805fd0 2024-03-21T10:48:25.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.410+0000 7f7a4fe00640 1 -- 172.21.15.31:0/1364404764 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a540bcf70 con 0x563ea4805fd0 2024-03-21T10:48:25.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.410+0000 7f7a6d815740 1 -- 172.21.15.31:0/1364404764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ea4805fd0 msgr2=0x563ea480ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:25.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.410+0000 7f7a6d815740 1 --2- 172.21.15.31:0/1364404764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ea4805fd0 0x563ea480ee10 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f7a5402c830 tx=0x7f7a5402cf90 comp rx=0 tx=0).stop 2024-03-21T10:48:25.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.410+0000 7f7a6d815740 1 -- 172.21.15.31:0/1364404764 shutdown_connections 2024-03-21T10:48:25.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.410+0000 7f7a6d815740 1 --2- 172.21.15.31:0/1364404764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ea480f6c0 0x563ea4806e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.410+0000 7f7a6d815740 1 --2- 172.21.15.31:0/1364404764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ea48068f0 0x563ea480b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.410+0000 7f7a6d815740 1 --2- 172.21.15.31:0/1364404764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ea4805fd0 0x563ea480ee10 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.410+0000 7f7a6d815740 1 -- 172.21.15.31:0/1364404764 >> 172.21.15.31:0/1364404764 conn(0x563ea48008f0 msgr2=0x563ea4800cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:25.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.410+0000 7f7a6d815740 1 -- 172.21.15.31:0/1364404764 shutdown_connections 2024-03-21T10:48:25.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.410+0000 7f7a6d815740 1 -- 172.21.15.31:0/1364404764 wait complete. 2024-03-21T10:48:25.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.411+0000 7f7a6d815740 1 -- start start 2024-03-21T10:48:25.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.411+0000 7f7a6d815740 1 Processor -- start 2024-03-21T10:48:25.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.411+0000 7f7a6d815740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ea4805fd0 0x563ea4812c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:25.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.411+0000 7f7a6d815740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ea48068f0 0x563ea48131c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:25.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.411+0000 7f7a6d815740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ea480f6c0 0x563ea4879be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:25.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.411+0000 7f7a6d815740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a641af200 con 0x563ea48068f0 2024-03-21T10:48:25.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.411+0000 7f7a6d815740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a641b04e0 con 0x563ea4805fd0 2024-03-21T10:48:25.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.411+0000 7f7a6d815740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a641af820 con 0x563ea480f6c0 2024-03-21T10:48:25.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.412+0000 7f7a6d815740 10 client.0 _ll_register_callbacks cb 0x7f7a5c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:25.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.412+0000 7f7a63e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ea4805fd0 0x563ea4812c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:25.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.412+0000 7f7a63e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ea4805fd0 0x563ea4812c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46184/0 (socket says 172.21.15.31:46184) 2024-03-21T10:48:25.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.412+0000 7f7a63e00640 1 -- 172.21.15.31:0/4236079953 learned_addr learned my addr 172.21.15.31:0/4236079953 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:25.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.412+0000 7f7a68a00640 1 --2- 172.21.15.31:0/4236079953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ea480f6c0 0x563ea4879be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:25.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.412+0000 7f7a63400640 1 --2- 172.21.15.31:0/4236079953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ea48068f0 0x563ea48131c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:25.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.412+0000 7f7a63e00640 1 -- 172.21.15.31:0/4236079953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ea480f6c0 msgr2=0x563ea4879be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:25.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.412+0000 7f7a63e00640 1 --2- 172.21.15.31:0/4236079953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ea480f6c0 0x563ea4879be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.412+0000 7f7a63e00640 1 -- 172.21.15.31:0/4236079953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ea48068f0 msgr2=0x563ea48131c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:25.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.412+0000 7f7a63e00640 1 --2- 172.21.15.31:0/4236079953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ea48068f0 0x563ea48131c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.412+0000 7f7a63e00640 1 -- 172.21.15.31:0/4236079953 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563ea487a120 con 0x563ea4805fd0 2024-03-21T10:48:25.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.412+0000 7f7a63400640 1 --2- 172.21.15.31:0/4236079953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ea48068f0 0x563ea48131c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:25.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.412+0000 7f7a63e00640 1 --2- 172.21.15.31:0/4236079953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ea4805fd0 0x563ea4812c80 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f7a540b9fe0 tx=0x7f7a540f4f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a09ab341968ce0b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:25.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.412+0000 7f7a4d600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:48:25.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.413+0000 7f7a4d600640 1 -- 172.21.15.31:0/4236079953 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5404b070 con 0x563ea4805fd0 2024-03-21T10:48:25.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.413+0000 7f7a4d600640 1 -- 172.21.15.31:0/4236079953 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f7a540b9c80 con 0x563ea4805fd0 2024-03-21T10:48:25.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.413+0000 7f7a6d815740 1 -- 172.21.15.31:0/4236079953 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7a540fe020 con 0x563ea4805fd0 2024-03-21T10:48:25.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.413+0000 7f7a4d600640 1 -- 172.21.15.31:0/4236079953 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a540bd150 con 0x563ea4805fd0 2024-03-21T10:48:25.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.413+0000 7f7a6d815740 10 client.28085 fetch_fsmap learned FSMap version 28 2024-03-21T10:48:25.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.413+0000 7f7a6d815740 1 -- 172.21.15.31:0/4236079953 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563ea487a120 con 0x563ea4805fd0 2024-03-21T10:48:25.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.413+0000 7f7a4d600640 1 -- 172.21.15.31:0/4236079953 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a540fe020 con 0x563ea4805fd0 2024-03-21T10:48:25.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.413+0000 7f7a4d600640 1 -- 172.21.15.31:0/4236079953 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f7a540c5ae0 con 0x563ea4805fd0 2024-03-21T10:48:25.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.413+0000 7f7a6d815740 10 client.28085 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:48:25.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.413+0000 7f7a6d815740 10 client.28085 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:48:25.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.413+0000 7f7a6d815740 20 client.28085 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:25.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.413+0000 7f7a6d815740 1 --2- 172.21.15.31:0/4236079953 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563ea487b540 0x563ea487d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:25.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.413+0000 7f7a6d815740 4 client.28085 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:48:25.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.413+0000 7f7a6d815740 1 -- 172.21.15.31:0/4236079953 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563ea47720c0 con 0x563ea487b540 2024-03-21T10:48:25.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.414+0000 7f7a63400640 1 --2- 172.21.15.31:0/4236079953 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563ea487b540 0x563ea487d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:25.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.414+0000 7f7a63400640 1 --2- 172.21.15.31:0/4236079953 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563ea487b540 0x563ea487d960 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d90d6093964cd444 server_cookie=7bfea560a63e3879 in_seq=0 out_seq=0 2024-03-21T10:48:25.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.414+0000 7f7a4d600640 10 client.28085 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:25.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.418+0000 7f7a4d600640 1 -- 172.21.15.31:0/4236079953 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563ea47720c0 con 0x563ea487b540 2024-03-21T10:48:25.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.418+0000 7f7a4d600640 10 client.28085 handle_command_reply: tid=0 2024-03-21T10:48:25.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.418+0000 7f7a6d815740 1 client.28085 shutdown 2024-03-21T10:48:25.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.418+0000 7f7a6d815740 1 -- 172.21.15.31:0/4236079953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ea4805fd0 msgr2=0x563ea4812c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:25.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.418+0000 7f7a6d815740 1 --2- 172.21.15.31:0/4236079953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ea4805fd0 0x563ea4812c80 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f7a540b9fe0 tx=0x7f7a540f4f10 comp rx=0 tx=0).stop 2024-03-21T10:48:25.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.418+0000 7f7a6d815740 1 -- 172.21.15.31:0/4236079953 shutdown_connections 2024-03-21T10:48:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.418+0000 7f7a6d815740 1 --2- 172.21.15.31:0/4236079953 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x563ea487b540 0x563ea487d960 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.418+0000 7f7a6d815740 1 --2- 172.21.15.31:0/4236079953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ea480f6c0 0x563ea4879be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.418+0000 7f7a6d815740 1 --2- 172.21.15.31:0/4236079953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ea48068f0 0x563ea48131c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.418+0000 7f7a4d600640 0 client.28085 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.418+0000 7f7a6d815740 1 --2- 172.21.15.31:0/4236079953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ea4805fd0 0x563ea4812c80 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.418+0000 7f7a6d815740 1 -- 172.21.15.31:0/4236079953 >> 172.21.15.31:0/4236079953 conn(0x563ea48008f0 msgr2=0x563ea4806e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.418+0000 7f7a6d815740 1 -- 172.21.15.31:0/4236079953 shutdown_connections 2024-03-21T10:48:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.418+0000 7f7a6d815740 1 -- 172.21.15.31:0/4236079953 wait complete. 2024-03-21T10:48:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.418+0000 7f7a6d815740 20 client.28085 trim_cache size 0 max 16384 2024-03-21T10:48:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:48:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:48:25.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:48:25.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:48:25.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.419+0000 7f7a6aa00640 1 -- 172.21.15.31:0/2069408769 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7a480a3080 msgr2=0x7f7a480a5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:25.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.419+0000 7f7a6aa00640 1 --2- 172.21.15.31:0/2069408769 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7a480a3080 0x7f7a480a5540 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c005fd0 tx=0x7f7a5c009500 comp rx=0 tx=0).stop 2024-03-21T10:48:25.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.419+0000 7f7a6aa00640 1 -- 172.21.15.31:0/2069408769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a64068f40 msgr2=0x7f7a6419a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:25.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.419+0000 7f7a6aa00640 1 --2- 172.21.15.31:0/2069408769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a64068f40 0x7f7a6419a110 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f7a54005ab0 tx=0x7f7a540042f0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.419+0000 7f7a6aa00640 1 -- 172.21.15.31:0/2069408769 shutdown_connections 2024-03-21T10:48:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.419+0000 7f7a6aa00640 1 --2- 172.21.15.31:0/2069408769 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f7a480a3080 0x7f7a480a5540 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.419+0000 7f7a6aa00640 1 --2- 172.21.15.31:0/2069408769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a6410e040 0x7f7a6419e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.419+0000 7f7a6aa00640 1 --2- 172.21.15.31:0/2069408769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a64069860 0x7f7a6419a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.419+0000 7f7a6aa00640 1 --2- 172.21.15.31:0/2069408769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a64068f40 0x7f7a6419a110 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.419+0000 7f7a6aa00640 1 -- 172.21.15.31:0/2069408769 >> 172.21.15.31:0/2069408769 conn(0x7f7a64077690 msgr2=0x7f7a6410f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.419+0000 7f7a6aa00640 1 -- 172.21.15.31:0/2069408769 shutdown_connections 2024-03-21T10:48:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:25.419+0000 7f7a6aa00640 1 -- 172.21.15.31:0/2069408769 wait complete. 2024-03-21T10:48:27.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:26 smithi031 ceph-mon[31673]: pgmap v307: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-21T10:48:27.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:26 smithi070 ceph-mon[36330]: pgmap v307: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-21T10:48:27.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:26 smithi040 ceph-mon[36154]: pgmap v307: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-21T10:48:29.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:28 smithi031 ceph-mon[31673]: pgmap v308: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-21T10:48:29.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:28 smithi070 ceph-mon[36330]: pgmap v308: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-21T10:48:29.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:28 smithi040 ceph-mon[36154]: pgmap v308: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-21T10:48:31.231 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:30 smithi040 ceph-mon[36154]: pgmap v309: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-03-21T10:48:31.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:30 smithi031 ceph-mon[31673]: pgmap v309: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-03-21T10:48:31.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:30 smithi070 ceph-mon[36330]: pgmap v309: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-03-21T10:48:33.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:32 smithi031 ceph-mon[31673]: pgmap v310: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-21T10:48:33.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:32 smithi070 ceph-mon[36330]: pgmap v310: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-21T10:48:33.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:32 smithi040 ceph-mon[36154]: pgmap v310: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-21T10:48:35.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:34 smithi031 ceph-mon[31673]: pgmap v311: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:48:35.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:34 smithi070 ceph-mon[36330]: pgmap v311: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:48:35.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:34 smithi040 ceph-mon[36154]: pgmap v311: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:48:37.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:36 smithi031 ceph-mon[31673]: pgmap v312: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-21T10:48:37.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:36 smithi070 ceph-mon[36330]: pgmap v312: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-21T10:48:37.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:36 smithi040 ceph-mon[36154]: pgmap v312: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-21T10:48:38.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.572+0000 7fe531600640 1 -- 172.21.15.31:0/1504167333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe52c069900 msgr2=0x7fe52c10e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:38.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.572+0000 7fe531600640 1 --2- 172.21.15.31:0/1504167333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe52c069900 0x7fe52c10e510 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe524009ca0 tx=0x7fe52402c100 comp rx=0 tx=0).stop 2024-03-21T10:48:38.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.572+0000 7fe531600640 1 -- 172.21.15.31:0/1504167333 shutdown_connections 2024-03-21T10:48:38.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.572+0000 7fe531600640 1 --2- 172.21.15.31:0/1504167333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe52c069900 0x7fe52c10e510 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.572+0000 7fe531600640 1 --2- 172.21.15.31:0/1504167333 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe52c068f40 0x7fe52c0693c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.572+0000 7fe531600640 1 --2- 172.21.15.31:0/1504167333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe52c107650 0x7fe52c107a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.572+0000 7fe531600640 1 -- 172.21.15.31:0/1504167333 >> 172.21.15.31:0/1504167333 conn(0x7fe52c077690 msgr2=0x7fe52c077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:38.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.572+0000 7fe531600640 1 -- 172.21.15.31:0/1504167333 shutdown_connections 2024-03-21T10:48:38.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.572+0000 7fe531600640 1 -- 172.21.15.31:0/1504167333 wait complete. 2024-03-21T10:48:38.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.573+0000 7fe531600640 1 Processor -- start 2024-03-21T10:48:38.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.573+0000 7fe531600640 1 -- start start 2024-03-21T10:48:38.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.573+0000 7fe531600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe52c068f40 0x7fe52c197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:38.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.573+0000 7fe531600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe52c069900 0x7fe52c1983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:38.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.573+0000 7fe531600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe52c107650 0x7fe52c19c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.573+0000 7fe531600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c111a60 con 0x7fe52c068f40 2024-03-21T10:48:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.573+0000 7fe531600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c1118e0 con 0x7fe52c069900 2024-03-21T10:48:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.573+0000 7fe531600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c111be0 con 0x7fe52c107650 2024-03-21T10:48:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.574+0000 7fe523e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe52c069900 0x7fe52c1983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.574+0000 7fe52aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe52c068f40 0x7fe52c197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.574+0000 7fe523e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe52c069900 0x7fe52c1983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:47802/0 (socket says 172.21.15.31:47802) 2024-03-21T10:48:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.574+0000 7fe52b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe52c107650 0x7fe52c19c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.574+0000 7fe523e00640 1 -- 172.21.15.31:0/1605460475 learned_addr learned my addr 172.21.15.31:0/1605460475 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.574+0000 7fe523e00640 1 -- 172.21.15.31:0/1605460475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe52c107650 msgr2=0x7fe52c19c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.574+0000 7fe523e00640 1 --2- 172.21.15.31:0/1605460475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe52c107650 0x7fe52c19c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.574+0000 7fe523e00640 1 -- 172.21.15.31:0/1605460475 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe52c068f40 msgr2=0x7fe52c197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.574+0000 7fe523e00640 1 --2- 172.21.15.31:0/1605460475 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe52c068f40 0x7fe52c197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.574+0000 7fe523e00640 1 -- 172.21.15.31:0/1605460475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52c19ce90 con 0x7fe52c069900 2024-03-21T10:48:38.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.574+0000 7fe523e00640 1 --2- 172.21.15.31:0/1605460475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe52c069900 0x7fe52c1983e0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fe51400e9f0 tx=0x7fe51400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:38.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.575+0000 7fe523400640 1 -- 172.21.15.31:0/1605460475 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51400cd80 con 0x7fe52c069900 2024-03-21T10:48:38.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.575+0000 7fe531600640 1 -- 172.21.15.31:0/1605460475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52c19d180 con 0x7fe52c069900 2024-03-21T10:48:38.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.575+0000 7fe523400640 1 -- 172.21.15.31:0/1605460475 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe514005500 con 0x7fe52c069900 2024-03-21T10:48:38.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.575+0000 7fe523400640 1 -- 172.21.15.31:0/1605460475 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe514010690 con 0x7fe52c069900 2024-03-21T10:48:38.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.575+0000 7fe531600640 1 -- 172.21.15.31:0/1605460475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe52c1a4a10 con 0x7fe52c069900 2024-03-21T10:48:38.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.578+0000 7fe534278740 1 Processor -- start 2024-03-21T10:48:38.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.578+0000 7fe534278740 1 -- start start 2024-03-21T10:48:38.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.579+0000 7fe534278740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bb261a3e0 0x555bb261a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:38.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.579+0000 7fe534278740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bb261ad00 0x555bb2623590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:38.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.579+0000 7fe534278740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bb2623ad0 0x555bb2625ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:38.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.579+0000 7fe534278740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bb25d83b0 con 0x555bb261a3e0 2024-03-21T10:48:38.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.579+0000 7fe534278740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bb257a830 con 0x555bb261ad00 2024-03-21T10:48:38.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.579+0000 7fe534278740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bb25235d0 con 0x555bb2623ad0 2024-03-21T10:48:38.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.579+0000 7fe52aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bb261ad00 0x555bb2623590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:38.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.579+0000 7fe52b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bb261a3e0 0x555bb261a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:38.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.579+0000 7fe52aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bb261ad00 0x555bb2623590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:47810/0 (socket says 172.21.15.31:47810) 2024-03-21T10:48:38.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.579+0000 7fe52aa00640 1 -- 172.21.15.31:0/2940778605 learned_addr learned my addr 172.21.15.31:0/2940778605 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:38.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.579+0000 7fe52aa00640 1 -- 172.21.15.31:0/2940778605 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bb2623ad0 msgr2=0x555bb2625ec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:48:38.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.579+0000 7fe52aa00640 1 --2- 172.21.15.31:0/2940778605 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bb2623ad0 0x555bb2625ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.579+0000 7fe52aa00640 1 -- 172.21.15.31:0/2940778605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bb261a3e0 msgr2=0x555bb261a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:38.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.579+0000 7fe52aa00640 1 --2- 172.21.15.31:0/2940778605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bb261a3e0 0x555bb261a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.579+0000 7fe52aa00640 1 -- 172.21.15.31:0/2940778605 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555bb253b2c0 con 0x555bb261ad00 2024-03-21T10:48:38.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.580+0000 7fe52aa00640 1 --2- 172.21.15.31:0/2940778605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bb261ad00 0x555bb2623590 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fe51c009c70 tx=0x7fe51c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fb66aafa3535978 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:38.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.580+0000 7fe51b400640 1 -- 172.21.15.31:0/2940778605 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c004230 con 0x555bb261ad00 2024-03-21T10:48:38.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.580+0000 7fe51b400640 1 -- 172.21.15.31:0/2940778605 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe51c0043d0 con 0x555bb261ad00 2024-03-21T10:48:38.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.580+0000 7fe51b400640 1 -- 172.21.15.31:0/2940778605 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c019800 con 0x555bb261ad00 2024-03-21T10:48:38.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.580+0000 7fe534278740 1 -- 172.21.15.31:0/2940778605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bb261ad00 msgr2=0x555bb2623590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:38.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.580+0000 7fe534278740 1 --2- 172.21.15.31:0/2940778605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bb261ad00 0x555bb2623590 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fe51c009c70 tx=0x7fe51c011810 comp rx=0 tx=0).stop 2024-03-21T10:48:38.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.581+0000 7fe534278740 1 -- 172.21.15.31:0/2940778605 shutdown_connections 2024-03-21T10:48:38.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.581+0000 7fe534278740 1 --2- 172.21.15.31:0/2940778605 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bb2623ad0 0x555bb2625ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.581+0000 7fe534278740 1 --2- 172.21.15.31:0/2940778605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bb261ad00 0x555bb2623590 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.581+0000 7fe534278740 1 --2- 172.21.15.31:0/2940778605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bb261a3e0 0x555bb261a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.581+0000 7fe534278740 1 -- 172.21.15.31:0/2940778605 >> 172.21.15.31:0/2940778605 conn(0x555bb2614d00 msgr2=0x555bb26150e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:38.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.582+0000 7fe523400640 1 -- 172.21.15.31:0/1605460475 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fe5140046c0 con 0x7fe52c069900 2024-03-21T10:48:38.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.582+0000 7fe534278740 1 -- 172.21.15.31:0/2940778605 shutdown_connections 2024-03-21T10:48:38.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.582+0000 7fe534278740 1 -- 172.21.15.31:0/2940778605 wait complete. 2024-03-21T10:48:38.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.582+0000 7fe534278740 1 -- start start 2024-03-21T10:48:38.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.583+0000 7fe534278740 1 Processor -- start 2024-03-21T10:48:38.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.583+0000 7fe534278740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bb261a3e0 0x555bb2620bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:38.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.583+0000 7fe534278740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bb261ad00 0x555bb2621130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:38.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.583+0000 7fe534278740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bb2621670 0x555bb262f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:38.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.583+0000 7fe534278740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bb2536a20 con 0x555bb261ad00 2024-03-21T10:48:38.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.583+0000 7fe534278740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bb25768d0 con 0x555bb261a3e0 2024-03-21T10:48:38.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.583+0000 7fe534278740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bb2626700 con 0x555bb2621670 2024-03-21T10:48:38.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.583+0000 7fe534278740 10 client.0 _ll_register_callbacks cb 0x555bb25731c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:38.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.583+0000 7fe52b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bb261a3e0 0x555bb2620bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:38.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.583+0000 7fe52b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bb261a3e0 0x555bb2620bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:47812/0 (socket says 172.21.15.31:47812) 2024-03-21T10:48:38.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.583+0000 7fe52aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bb261ad00 0x555bb2621130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:38.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.583+0000 7fe52b400640 1 -- 172.21.15.31:0/1673388267 learned_addr learned my addr 172.21.15.31:0/1673388267 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:38.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.583+0000 7fe523e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bb2621670 0x555bb262f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:38.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.583+0000 7fe523400640 1 --2- 172.21.15.31:0/1605460475 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe5100a2f90 0x7fe5100a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:38.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.584+0000 7fe523400640 1 -- 172.21.15.31:0/1605460475 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fe5140cff70 con 0x7fe52c069900 2024-03-21T10:48:38.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.584+0000 7fe52b400640 1 --2- 172.21.15.31:0/1605460475 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe5100a2f90 0x7fe5100a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:38.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.584+0000 7fe52aa00640 1 -- 172.21.15.31:0/1673388267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bb2621670 msgr2=0x555bb262f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:38.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.584+0000 7fe52aa00640 1 --2- 172.21.15.31:0/1673388267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bb2621670 0x555bb262f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.584+0000 7fe52aa00640 1 -- 172.21.15.31:0/1673388267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bb261a3e0 msgr2=0x555bb2620bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:38.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.584+0000 7fe52aa00640 1 --2- 172.21.15.31:0/1673388267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bb261a3e0 0x555bb2620bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.584+0000 7fe52aa00640 1 -- 172.21.15.31:0/1673388267 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555bb228f210 con 0x555bb261ad00 2024-03-21T10:48:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.584+0000 7fe523e00640 1 --2- 172.21.15.31:0/1673388267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bb2621670 0x555bb262f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.584+0000 7fe52aa00640 1 --2- 172.21.15.31:0/1673388267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bb261ad00 0x555bb2621130 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe51c019420 tx=0x7fe51c005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc047f5b667c151f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.585+0000 7fe50fe00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:48:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.585+0000 7fe50fe00640 1 -- 172.21.15.31:0/1673388267 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c027070 con 0x555bb261ad00 2024-03-21T10:48:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.585+0000 7fe52b400640 1 --2- 172.21.15.31:0/1605460475 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe5100a2f90 0x7fe5100a5450 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe524009a60 tx=0x7fe5240095c0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.585+0000 7fe50fe00640 1 -- 172.21.15.31:0/1673388267 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fe51c003800 con 0x555bb261ad00 2024-03-21T10:48:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.585+0000 7fe534278740 1 -- 172.21.15.31:0/1673388267 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe51c0043d0 con 0x555bb261ad00 2024-03-21T10:48:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.585+0000 7fe50fe00640 1 -- 172.21.15.31:0/1673388267 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe51c033050 con 0x555bb261ad00 2024-03-21T10:48:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.585+0000 7fe534278740 10 client.17463 fetch_fsmap learned FSMap version 28 2024-03-21T10:48:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.585+0000 7fe534278740 1 -- 172.21.15.31:0/1673388267 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555bb235b620 con 0x555bb261ad00 2024-03-21T10:48:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.586+0000 7fe50fe00640 1 -- 172.21.15.31:0/1673388267 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe51c0043d0 con 0x555bb261ad00 2024-03-21T10:48:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.586+0000 7fe50fe00640 1 -- 172.21.15.31:0/1673388267 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe51c018070 con 0x555bb261ad00 2024-03-21T10:48:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.586+0000 7fe534278740 10 client.17463 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:48:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.586+0000 7fe534278740 10 client.17463 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:48:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.586+0000 7fe534278740 20 client.17463 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.586+0000 7fe534278740 1 --2- 172.21.15.31:0/1673388267 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x555bb2633f30 0x555bb2636320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.586+0000 7fe534278740 4 client.17463 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:48:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.586+0000 7fe534278740 1 -- 172.21.15.31:0/1673388267 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555bb2587d70 con 0x555bb2633f30 2024-03-21T10:48:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.586+0000 7fe523e00640 1 --2- 172.21.15.31:0/1673388267 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x555bb2633f30 0x555bb2636320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.587+0000 7fe523e00640 1 --2- 172.21.15.31:0/1673388267 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x555bb2633f30 0x555bb2636320 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5529284f2d87b36f server_cookie=745f8f0389cfe120 in_seq=0 out_seq=0 2024-03-21T10:48:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.587+0000 7fe50fe00640 10 client.17463 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.587+0000 7fe50fe00640 1 -- 172.21.15.31:0/1673388267 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555bb2587d70 con 0x555bb2633f30 2024-03-21T10:48:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.588+0000 7fe50fe00640 10 client.17463 handle_command_reply: tid=0 2024-03-21T10:48:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.588+0000 7fe534278740 1 client.17463 shutdown 2024-03-21T10:48:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.588+0000 7fe534278740 1 -- 172.21.15.31:0/1673388267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bb261ad00 msgr2=0x555bb2621130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.588+0000 7fe534278740 1 --2- 172.21.15.31:0/1673388267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bb261ad00 0x555bb2621130 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe51c019420 tx=0x7fe51c005dc0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.588+0000 7fe534278740 1 -- 172.21.15.31:0/1673388267 shutdown_connections 2024-03-21T10:48:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.588+0000 7fe534278740 1 --2- 172.21.15.31:0/1673388267 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x555bb2633f30 0x555bb2636320 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.588+0000 7fe534278740 1 --2- 172.21.15.31:0/1673388267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bb2621670 0x555bb262f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.588+0000 7fe534278740 1 --2- 172.21.15.31:0/1673388267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bb261ad00 0x555bb2621130 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.588+0000 7fe534278740 1 --2- 172.21.15.31:0/1673388267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bb261a3e0 0x555bb2620bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.588+0000 7fe50fe00640 0 client.17463 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.588+0000 7fe534278740 1 -- 172.21.15.31:0/1673388267 >> 172.21.15.31:0/1673388267 conn(0x555bb2614d00 msgr2=0x555bb2616a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.588+0000 7fe534278740 1 -- 172.21.15.31:0/1673388267 shutdown_connections 2024-03-21T10:48:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.588+0000 7fe534278740 1 -- 172.21.15.31:0/1673388267 wait complete. 2024-03-21T10:48:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.588+0000 7fe534278740 20 client.17463 trim_cache size 0 max 16384 2024-03-21T10:48:38.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.604+0000 7fe534278740 1 Processor -- start 2024-03-21T10:48:38.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.604+0000 7fe534278740 1 -- start start 2024-03-21T10:48:38.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.604+0000 7fe534278740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bb261a3e0 0x555bb261de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:38.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.604+0000 7fe534278740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bb261ad00 0x555bb261e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:38.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.604+0000 7fe534278740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bb2621670 0x555bb26160c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:38.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.604+0000 7fe534278740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c1aec50 con 0x555bb261a3e0 2024-03-21T10:48:38.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.604+0000 7fe534278740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c1ac750 con 0x555bb2621670 2024-03-21T10:48:38.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.604+0000 7fe534278740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c1aa810 con 0x555bb261ad00 2024-03-21T10:48:38.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.605+0000 7fe52aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bb261a3e0 0x555bb261de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:38.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.605+0000 7fe52aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bb261a3e0 0x555bb261de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57236/0 (socket says 172.21.15.31:57236) 2024-03-21T10:48:38.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.605+0000 7fe52aa00640 1 -- 172.21.15.31:0/713533117 learned_addr learned my addr 172.21.15.31:0/713533117 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:38.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.605+0000 7fe52b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bb2621670 0x555bb26160c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:38.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.605+0000 7fe523e00640 1 --2- 172.21.15.31:0/713533117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bb261ad00 0x555bb261e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:38.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.605+0000 7fe52aa00640 1 -- 172.21.15.31:0/713533117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bb261ad00 msgr2=0x555bb261e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:38.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.605+0000 7fe52aa00640 1 --2- 172.21.15.31:0/713533117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bb261ad00 0x555bb261e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.605+0000 7fe52aa00640 1 -- 172.21.15.31:0/713533117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bb2621670 msgr2=0x555bb26160c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:38.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.605+0000 7fe52aa00640 1 --2- 172.21.15.31:0/713533117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bb2621670 0x555bb26160c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.605+0000 7fe52aa00640 1 -- 172.21.15.31:0/713533117 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555bb260ed40 con 0x555bb261a3e0 2024-03-21T10:48:38.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.605+0000 7fe52aa00640 1 --2- 172.21.15.31:0/713533117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bb261a3e0 0x555bb261de70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe51c0095f0 tx=0x7fe51c01a700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f26b747f8fb0b627 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:38.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.606+0000 7fe51aa00640 1 -- 172.21.15.31:0/713533117 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c027030 con 0x555bb261a3e0 2024-03-21T10:48:38.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.606+0000 7fe51aa00640 1 -- 172.21.15.31:0/713533117 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe51c009e20 con 0x555bb261a3e0 2024-03-21T10:48:38.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.606+0000 7fe51aa00640 1 -- 172.21.15.31:0/713533117 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c005400 con 0x555bb261a3e0 2024-03-21T10:48:38.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.606+0000 7fe534278740 1 -- 172.21.15.31:0/713533117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bb261a3e0 msgr2=0x555bb261de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:38.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.606+0000 7fe534278740 1 --2- 172.21.15.31:0/713533117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bb261a3e0 0x555bb261de70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe51c0095f0 tx=0x7fe51c01a700 comp rx=0 tx=0).stop 2024-03-21T10:48:38.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.606+0000 7fe534278740 1 -- 172.21.15.31:0/713533117 shutdown_connections 2024-03-21T10:48:38.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.606+0000 7fe534278740 1 --2- 172.21.15.31:0/713533117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bb2621670 0x555bb26160c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.606+0000 7fe534278740 1 --2- 172.21.15.31:0/713533117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bb261ad00 0x555bb261e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.606+0000 7fe534278740 1 --2- 172.21.15.31:0/713533117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bb261a3e0 0x555bb261de70 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.606+0000 7fe534278740 1 -- 172.21.15.31:0/713533117 >> 172.21.15.31:0/713533117 conn(0x555bb2614d00 msgr2=0x555bb261f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:38.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.607+0000 7fe534278740 1 -- 172.21.15.31:0/713533117 shutdown_connections 2024-03-21T10:48:38.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.607+0000 7fe534278740 1 -- 172.21.15.31:0/713533117 wait complete. 2024-03-21T10:48:38.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.607+0000 7fe534278740 1 -- start start 2024-03-21T10:48:38.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.607+0000 7fe534278740 1 Processor -- start 2024-03-21T10:48:38.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.608+0000 7fe534278740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bb261a3e0 0x555bb2629110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:38.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.608+0000 7fe534278740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bb261ad00 0x555bb2629650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:38.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.608+0000 7fe534278740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bb2621670 0x555bb268e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:38.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.608+0000 7fe534278740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c1aae10 con 0x555bb261ad00 2024-03-21T10:48:38.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.608+0000 7fe534278740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c1ac0f0 con 0x555bb2621670 2024-03-21T10:48:38.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.608+0000 7fe534278740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c1ab430 con 0x555bb261a3e0 2024-03-21T10:48:38.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.608+0000 7fe534278740 10 client.0 _ll_register_callbacks cb 0x7fe51c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:38.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.608+0000 7fe52aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bb261a3e0 0x555bb2629110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:38.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.608+0000 7fe523e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bb261ad00 0x555bb2629650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:38.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.608+0000 7fe52aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bb261a3e0 0x555bb2629110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:39018/0 (socket says 172.21.15.31:39018) 2024-03-21T10:48:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.608+0000 7fe52aa00640 1 -- 172.21.15.31:0/4176301977 learned_addr learned my addr 172.21.15.31:0/4176301977 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.608+0000 7fe52b400640 1 --2- 172.21.15.31:0/4176301977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bb2621670 0x555bb268e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.608+0000 7fe52aa00640 1 -- 172.21.15.31:0/4176301977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bb2621670 msgr2=0x555bb268e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.608+0000 7fe52aa00640 1 --2- 172.21.15.31:0/4176301977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bb2621670 0x555bb268e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.608+0000 7fe52aa00640 1 -- 172.21.15.31:0/4176301977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bb261ad00 msgr2=0x555bb2629650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.608+0000 7fe52aa00640 1 --2- 172.21.15.31:0/4176301977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bb261ad00 0x555bb2629650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.608+0000 7fe52aa00640 1 -- 172.21.15.31:0/4176301977 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555bb268eae0 con 0x555bb261a3e0 2024-03-21T10:48:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.609+0000 7fe52aa00640 1 --2- 172.21.15.31:0/4176301977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bb261a3e0 0x555bb2629110 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fe51c009460 tx=0x7fe51c005630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=615ec6c5a51e31b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.609+0000 7fe50fe00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:48:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.609+0000 7fe50fe00640 1 -- 172.21.15.31:0/4176301977 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c03d020 con 0x555bb261a3e0 2024-03-21T10:48:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.609+0000 7fe534278740 1 -- 172.21.15.31:0/4176301977 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe51c009e20 con 0x555bb261a3e0 2024-03-21T10:48:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.609+0000 7fe50fe00640 1 -- 172.21.15.31:0/4176301977 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fe51c003a10 con 0x555bb261a3e0 2024-03-21T10:48:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.609+0000 7fe50fe00640 1 -- 172.21.15.31:0/4176301977 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe51c048050 con 0x555bb261a3e0 2024-03-21T10:48:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.609+0000 7fe534278740 10 client.28030 fetch_fsmap learned FSMap version 28 2024-03-21T10:48:38.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.609+0000 7fe534278740 1 -- 172.21.15.31:0/4176301977 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555bb268fc50 con 0x555bb261a3e0 2024-03-21T10:48:38.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.610+0000 7fe50fe00640 1 -- 172.21.15.31:0/4176301977 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe51c009e20 con 0x555bb261a3e0 2024-03-21T10:48:38.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.610+0000 7fe50fe00640 1 -- 172.21.15.31:0/4176301977 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe51c018070 con 0x555bb261a3e0 2024-03-21T10:48:38.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.610+0000 7fe534278740 10 client.28030 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:48:38.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.610+0000 7fe534278740 10 client.28030 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:48:38.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.610+0000 7fe534278740 20 client.28030 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:38.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.610+0000 7fe534278740 1 --2- 172.21.15.31:0/4176301977 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x555bb26902f0 0x555bb2692710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:38.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.610+0000 7fe534278740 4 client.28030 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:48:38.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.610+0000 7fe534278740 1 -- 172.21.15.31:0/4176301977 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555bb25820c0 con 0x555bb26902f0 2024-03-21T10:48:38.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.610+0000 7fe523e00640 1 --2- 172.21.15.31:0/4176301977 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x555bb26902f0 0x555bb2692710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:38.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.610+0000 7fe523e00640 1 --2- 172.21.15.31:0/4176301977 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x555bb26902f0 0x555bb2692710 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=318a1e6aff7e420b server_cookie=ee50ed17ccbb9787 in_seq=0 out_seq=0 2024-03-21T10:48:38.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.610+0000 7fe50fe00640 10 client.28030 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.614+0000 7fe50fe00640 1 -- 172.21.15.31:0/4176301977 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555bb25820c0 con 0x555bb26902f0 2024-03-21T10:48:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.614+0000 7fe50fe00640 10 client.28030 handle_command_reply: tid=0 2024-03-21T10:48:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.614+0000 7fe534278740 1 client.28030 shutdown 2024-03-21T10:48:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.614+0000 7fe534278740 1 -- 172.21.15.31:0/4176301977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bb261a3e0 msgr2=0x555bb2629110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.614+0000 7fe534278740 1 --2- 172.21.15.31:0/4176301977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bb261a3e0 0x555bb2629110 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fe51c009460 tx=0x7fe51c005630 comp rx=0 tx=0).stop 2024-03-21T10:48:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.614+0000 7fe534278740 1 -- 172.21.15.31:0/4176301977 shutdown_connections 2024-03-21T10:48:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.614+0000 7fe534278740 1 --2- 172.21.15.31:0/4176301977 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x555bb26902f0 0x555bb2692710 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.614+0000 7fe534278740 1 --2- 172.21.15.31:0/4176301977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555bb2621670 0x555bb268e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.614+0000 7fe534278740 1 --2- 172.21.15.31:0/4176301977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555bb261ad00 0x555bb2629650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.614+0000 7fe50fe00640 0 client.28030 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.614+0000 7fe534278740 1 --2- 172.21.15.31:0/4176301977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bb261a3e0 0x555bb2629110 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.614+0000 7fe534278740 1 -- 172.21.15.31:0/4176301977 >> 172.21.15.31:0/4176301977 conn(0x555bb2614d00 msgr2=0x555bb260f950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.615+0000 7fe534278740 1 -- 172.21.15.31:0/4176301977 shutdown_connections 2024-03-21T10:48:38.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.615+0000 7fe534278740 1 -- 172.21.15.31:0/4176301977 wait complete. 2024-03-21T10:48:38.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.615+0000 7fe534278740 20 client.28030 trim_cache size 0 max 16384 2024-03-21T10:48:38.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:48:38.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:48:38.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:48:38.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:48:38.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.615+0000 7fe531600640 1 -- 172.21.15.31:0/1605460475 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe5100a2f90 msgr2=0x7fe5100a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:38.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.615+0000 7fe531600640 1 --2- 172.21.15.31:0/1605460475 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe5100a2f90 0x7fe5100a5450 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe524009a60 tx=0x7fe5240095c0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.615+0000 7fe531600640 1 -- 172.21.15.31:0/1605460475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe52c069900 msgr2=0x7fe52c1983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:38.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.615+0000 7fe531600640 1 --2- 172.21.15.31:0/1605460475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe52c069900 0x7fe52c1983e0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fe51400e9f0 tx=0x7fe51400edf0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.616+0000 7fe531600640 1 -- 172.21.15.31:0/1605460475 shutdown_connections 2024-03-21T10:48:38.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.616+0000 7fe531600640 1 --2- 172.21.15.31:0/1605460475 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe5100a2f90 0x7fe5100a5450 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.616+0000 7fe531600640 1 --2- 172.21.15.31:0/1605460475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe52c107650 0x7fe52c19c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.616+0000 7fe531600640 1 --2- 172.21.15.31:0/1605460475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe52c069900 0x7fe52c1983e0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.616+0000 7fe531600640 1 --2- 172.21.15.31:0/1605460475 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe52c068f40 0x7fe52c197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:38.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.616+0000 7fe531600640 1 -- 172.21.15.31:0/1605460475 >> 172.21.15.31:0/1605460475 conn(0x7fe52c077690 msgr2=0x7fe52c0fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:38.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.616+0000 7fe531600640 1 -- 172.21.15.31:0/1605460475 shutdown_connections 2024-03-21T10:48:38.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:38.616+0000 7fe531600640 1 -- 172.21.15.31:0/1605460475 wait complete. 2024-03-21T10:48:39.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:38 smithi031 ceph-mon[31673]: pgmap v313: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:48:39.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:38 smithi070 ceph-mon[36330]: pgmap v313: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:48:39.405 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:48:39.405 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:48:39.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:38 smithi040 ceph-mon[36154]: pgmap v313: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:48:39.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.635+0000 7f28b7200640 1 -- 172.21.15.31:0/2894478317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28b8101e00 msgr2=0x7f28b810e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:39.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.635+0000 7f28b7200640 1 --2- 172.21.15.31:0/2894478317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28b8101e00 0x7f28b810e310 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f28b0009900 tx=0x7f28b002bc40 comp rx=0 tx=0).stop 2024-03-21T10:48:39.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.635+0000 7f28b7200640 1 -- 172.21.15.31:0/2894478317 shutdown_connections 2024-03-21T10:48:39.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.635+0000 7f28b7200640 1 --2- 172.21.15.31:0/2894478317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28b8101e00 0x7f28b810e310 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.635+0000 7f28b7200640 1 --2- 172.21.15.31:0/2894478317 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f28b8101440 0x7f28b81018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.635+0000 7f28b7200640 1 --2- 172.21.15.31:0/2894478317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28b81074e0 0x7f28b81078c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.635+0000 7f28b7200640 1 -- 172.21.15.31:0/2894478317 >> 172.21.15.31:0/2894478317 conn(0x7f28b80fcf30 msgr2=0x7f28b80ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:39.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.635+0000 7f28b7200640 1 -- 172.21.15.31:0/2894478317 shutdown_connections 2024-03-21T10:48:39.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.635+0000 7f28b7200640 1 -- 172.21.15.31:0/2894478317 wait complete. 2024-03-21T10:48:39.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.637+0000 7f28b7200640 1 Processor -- start 2024-03-21T10:48:39.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.637+0000 7f28b7200640 1 -- start start 2024-03-21T10:48:39.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.638+0000 7f28b7200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f28b8101440 0x7f28b8199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:39.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.638+0000 7f28b7200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28b8101e00 0x7f28b819a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:39.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.638+0000 7f28b7200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28b81074e0 0x7f28b819e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:39.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.638+0000 7f28b7200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b8111810 con 0x7f28b8101e00 2024-03-21T10:48:39.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.638+0000 7f28b7200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b8111690 con 0x7f28b81074e0 2024-03-21T10:48:39.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.638+0000 7f28b7200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b8111990 con 0x7f28b8101440 2024-03-21T10:48:39.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.638+0000 7f28b5e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f28b8101440 0x7f28b8199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:39.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.638+0000 7f28ad400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28b8101e00 0x7f28b819a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:39.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.638+0000 7f28b5e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f28b8101440 0x7f28b8199ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:39034/0 (socket says 172.21.15.31:39034) 2024-03-21T10:48:39.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.638+0000 7f28b5e00640 1 -- 172.21.15.31:0/2750714465 learned_addr learned my addr 172.21.15.31:0/2750714465 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:39.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.638+0000 7f28b6800640 1 --2- 172.21.15.31:0/2750714465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28b81074e0 0x7f28b819e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:39.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.638+0000 7f28b5e00640 1 -- 172.21.15.31:0/2750714465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28b81074e0 msgr2=0x7f28b819e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:39.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.638+0000 7f28b5e00640 1 --2- 172.21.15.31:0/2750714465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28b81074e0 0x7f28b819e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.639+0000 7f28b5e00640 1 -- 172.21.15.31:0/2750714465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28b8101e00 msgr2=0x7f28b819a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:39.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.639+0000 7f28b5e00640 1 --2- 172.21.15.31:0/2750714465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28b8101e00 0x7f28b819a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.639+0000 7f28b5e00640 1 -- 172.21.15.31:0/2750714465 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b819eed0 con 0x7f28b8101440 2024-03-21T10:48:39.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.639+0000 7f28ad400640 1 --2- 172.21.15.31:0/2750714465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28b8101e00 0x7f28b819a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:39.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.639+0000 7f28b5e00640 1 --2- 172.21.15.31:0/2750714465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f28b8101440 0x7f28b8199ee0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f28a800e9f0 tx=0x7f28a800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:39.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.639+0000 7f28af400640 1 -- 172.21.15.31:0/2750714465 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a800cd80 con 0x7f28b8101440 2024-03-21T10:48:39.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.639+0000 7f28af400640 1 -- 172.21.15.31:0/2750714465 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f28a8010910 con 0x7f28b8101440 2024-03-21T10:48:39.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.639+0000 7f28af400640 1 -- 172.21.15.31:0/2750714465 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a8018990 con 0x7f28b8101440 2024-03-21T10:48:39.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.639+0000 7f28b7200640 1 -- 172.21.15.31:0/2750714465 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28b819f1c0 con 0x7f28b8101440 2024-03-21T10:48:39.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.640+0000 7f28b7200640 1 -- 172.21.15.31:0/2750714465 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28b81a6a50 con 0x7f28b8101440 2024-03-21T10:48:39.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.644+0000 7f28af400640 1 -- 172.21.15.31:0/2750714465 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f28a8010ac0 con 0x7f28b8101440 2024-03-21T10:48:39.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.645+0000 7f28bfb7f740 1 Processor -- start 2024-03-21T10:48:39.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.645+0000 7f28bfb7f740 1 -- start start 2024-03-21T10:48:39.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.645+0000 7f28bfb7f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5de525dd0 0x55d5de524b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:39.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.645+0000 7f28bfb7f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d5de5250f0 0x55d5de525570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:39.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.645+0000 7f28bfb7f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d5de52e460 0x55d5de530850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:39.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.645+0000 7f28bfb7f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5de489250 con 0x55d5de525dd0 2024-03-21T10:48:39.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.645+0000 7f28bfb7f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5de431a30 con 0x55d5de5250f0 2024-03-21T10:48:39.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.645+0000 7f28bfb7f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5de4852f0 con 0x55d5de52e460 2024-03-21T10:48:39.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.646+0000 7f28b5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d5de5250f0 0x55d5de525570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:39.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.646+0000 7f28b5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d5de5250f0 0x55d5de525570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:47854/0 (socket says 172.21.15.31:47854) 2024-03-21T10:48:39.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.646+0000 7f28ad400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d5de52e460 0x55d5de530850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:39.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.646+0000 7f28b6800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5de525dd0 0x55d5de524b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:39.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.646+0000 7f28b5e00640 1 -- 172.21.15.31:0/1741955594 learned_addr learned my addr 172.21.15.31:0/1741955594 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:39.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.646+0000 7f28ad400640 1 -- 172.21.15.31:0/1741955594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d5de5250f0 msgr2=0x55d5de525570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:39.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.646+0000 7f28ad400640 1 --2- 172.21.15.31:0/1741955594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d5de5250f0 0x55d5de525570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.646+0000 7f28ad400640 1 -- 172.21.15.31:0/1741955594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5de525dd0 msgr2=0x55d5de524b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:39.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.646+0000 7f28ad400640 1 --2- 172.21.15.31:0/1741955594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5de525dd0 0x55d5de524b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.646+0000 7f28ad400640 1 -- 172.21.15.31:0/1741955594 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d5de41c690 con 0x55d5de52e460 2024-03-21T10:48:39.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.646+0000 7f28af400640 1 --2- 172.21.15.31:0/2750714465 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f289c0a3190 0x7f289c0a5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:39.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.646+0000 7f28ad400640 1 --2- 172.21.15.31:0/1741955594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d5de52e460 0x55d5de530850 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f28b810e6d0 tx=0x7f28a0011c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad435e798ed43a89 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:39.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.647+0000 7f28af400640 1 -- 172.21.15.31:0/2750714465 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f28a80cfe40 con 0x7f28b8101440 2024-03-21T10:48:39.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.647+0000 7f28b6800640 1 --2- 172.21.15.31:0/2750714465 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f289c0a3190 0x7f289c0a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:39.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.647+0000 7f28a7400640 1 -- 172.21.15.31:0/1741955594 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a0004d20 con 0x55d5de52e460 2024-03-21T10:48:39.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.647+0000 7f28a7400640 1 -- 172.21.15.31:0/1741955594 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f28a000f970 con 0x55d5de52e460 2024-03-21T10:48:39.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.647+0000 7f28a7400640 1 -- 172.21.15.31:0/1741955594 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a0004510 con 0x55d5de52e460 2024-03-21T10:48:39.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.648+0000 7f28bfb7f740 1 -- 172.21.15.31:0/1741955594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d5de52e460 msgr2=0x55d5de530850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:39.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.648+0000 7f28bfb7f740 1 --2- 172.21.15.31:0/1741955594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d5de52e460 0x55d5de530850 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f28b810e6d0 tx=0x7f28a0011c70 comp rx=0 tx=0).stop 2024-03-21T10:48:39.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.648+0000 7f28b6800640 1 --2- 172.21.15.31:0/2750714465 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f289c0a3190 0x7f289c0a5650 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f28b0006010 tx=0x7f28b0005b50 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:39.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.648+0000 7f28bfb7f740 1 -- 172.21.15.31:0/1741955594 shutdown_connections 2024-03-21T10:48:39.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.648+0000 7f28bfb7f740 1 --2- 172.21.15.31:0/1741955594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d5de52e460 0x55d5de530850 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.648+0000 7f28bfb7f740 1 --2- 172.21.15.31:0/1741955594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d5de5250f0 0x55d5de525570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.648+0000 7f28bfb7f740 1 --2- 172.21.15.31:0/1741955594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5de525dd0 0x55d5de524b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.648+0000 7f28bfb7f740 1 -- 172.21.15.31:0/1741955594 >> 172.21.15.31:0/1741955594 conn(0x55d5de4ec200 msgr2=0x55d5de51fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:39.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.648+0000 7f28bfb7f740 1 -- 172.21.15.31:0/1741955594 shutdown_connections 2024-03-21T10:48:39.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.648+0000 7f28bfb7f740 1 -- 172.21.15.31:0/1741955594 wait complete. 2024-03-21T10:48:39.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.648+0000 7f28bfb7f740 1 -- start start 2024-03-21T10:48:39.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.648+0000 7f28bfb7f740 1 Processor -- start 2024-03-21T10:48:39.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.649+0000 7f28bfb7f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d5de5250f0 0x55d5de52dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:39.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.649+0000 7f28bfb7f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d5de525dd0 0x55d5de52ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:39.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.649+0000 7f28bfb7f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5de52e460 0x55d5de52b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:39.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.649+0000 7f28bfb7f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5de5312d0 con 0x55d5de52e460 2024-03-21T10:48:39.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.649+0000 7f28bfb7f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5de444e80 con 0x55d5de525dd0 2024-03-21T10:48:39.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.649+0000 7f28bfb7f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5de531450 con 0x55d5de5250f0 2024-03-21T10:48:39.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.649+0000 7f28bfb7f740 10 client.0 _ll_register_callbacks cb 0x55d5de457130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:39.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.649+0000 7f28b5e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d5de5250f0 0x55d5de52dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:39.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.649+0000 7f28b6800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5de52e460 0x55d5de52b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:39.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.649+0000 7f28b5e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d5de5250f0 0x55d5de52dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:39046/0 (socket says 172.21.15.31:39046) 2024-03-21T10:48:39.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.649+0000 7f28ad400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d5de525dd0 0x55d5de52ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:39.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.649+0000 7f28b5e00640 1 -- 172.21.15.31:0/1212411789 learned_addr learned my addr 172.21.15.31:0/1212411789 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:39.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.650+0000 7f28b5e00640 1 -- 172.21.15.31:0/1212411789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d5de525dd0 msgr2=0x55d5de52ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:39.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.650+0000 7f28b5e00640 1 --2- 172.21.15.31:0/1212411789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d5de525dd0 0x55d5de52ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.650+0000 7f28b5e00640 1 -- 172.21.15.31:0/1212411789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5de52e460 msgr2=0x55d5de52b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:39.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.650+0000 7f28b5e00640 1 --2- 172.21.15.31:0/1212411789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5de52e460 0x55d5de52b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.650+0000 7f28b5e00640 1 -- 172.21.15.31:0/1212411789 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d5de2677d0 con 0x55d5de5250f0 2024-03-21T10:48:39.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.650+0000 7f28b5e00640 1 --2- 172.21.15.31:0/1212411789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d5de5250f0 0x55d5de52dac0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f28a8098340 tx=0x7f28a8098180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9446c40892c76798 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:39.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.650+0000 7f289be00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:48:39.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.650+0000 7f289be00640 1 -- 172.21.15.31:0/1212411789 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a809b840 con 0x55d5de5250f0 2024-03-21T10:48:39.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.650+0000 7f28bfb7f740 1 -- 172.21.15.31:0/1212411789 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f28a000f970 con 0x55d5de5250f0 2024-03-21T10:48:39.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.650+0000 7f289be00640 1 -- 172.21.15.31:0/1212411789 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f28a80a45b0 con 0x55d5de5250f0 2024-03-21T10:48:39.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.651+0000 7f289be00640 1 -- 172.21.15.31:0/1212411789 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28a809bee0 con 0x55d5de5250f0 2024-03-21T10:48:39.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.651+0000 7f28bfb7f740 10 client.28042 fetch_fsmap learned FSMap version 28 2024-03-21T10:48:39.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.651+0000 7f28bfb7f740 1 -- 172.21.15.31:0/1212411789 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d5de19d3f0 con 0x55d5de5250f0 2024-03-21T10:48:39.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.651+0000 7f289be00640 1 -- 172.21.15.31:0/1212411789 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f28a000f970 con 0x55d5de5250f0 2024-03-21T10:48:39.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.651+0000 7f289be00640 1 -- 172.21.15.31:0/1212411789 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f28a8018b30 con 0x55d5de5250f0 2024-03-21T10:48:39.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.651+0000 7f28bfb7f740 10 client.28042 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:48:39.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.651+0000 7f28bfb7f740 10 client.28042 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:48:39.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.651+0000 7f28bfb7f740 20 client.28042 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:39.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.651+0000 7f28bfb7f740 1 --2- 172.21.15.31:0/1212411789 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d5de53e400 0x55d5de540880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:39.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.651+0000 7f28bfb7f740 4 client.28042 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:48:39.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.651+0000 7f28bfb7f740 1 -- 172.21.15.31:0/1212411789 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d5de496790 con 0x55d5de53e400 2024-03-21T10:48:39.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.652+0000 7f28ad400640 1 --2- 172.21.15.31:0/1212411789 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d5de53e400 0x55d5de540880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:39.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.652+0000 7f28ad400640 1 --2- 172.21.15.31:0/1212411789 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d5de53e400 0x55d5de540880 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1f9b4fef9d03695 server_cookie=a611fb57b4d32554 in_seq=0 out_seq=0 2024-03-21T10:48:39.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.652+0000 7f289be00640 10 client.28042 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:39.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.653+0000 7f289be00640 1 -- 172.21.15.31:0/1212411789 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d5de496790 con 0x55d5de53e400 2024-03-21T10:48:39.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.653+0000 7f289be00640 10 client.28042 handle_command_reply: tid=0 2024-03-21T10:48:39.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.653+0000 7f28bfb7f740 1 client.28042 shutdown 2024-03-21T10:48:39.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.653+0000 7f28bfb7f740 1 -- 172.21.15.31:0/1212411789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d5de5250f0 msgr2=0x55d5de52dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:39.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.653+0000 7f28bfb7f740 1 --2- 172.21.15.31:0/1212411789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d5de5250f0 0x55d5de52dac0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f28a8098340 tx=0x7f28a8098180 comp rx=0 tx=0).stop 2024-03-21T10:48:39.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.654+0000 7f28bfb7f740 1 -- 172.21.15.31:0/1212411789 shutdown_connections 2024-03-21T10:48:39.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.654+0000 7f28bfb7f740 1 --2- 172.21.15.31:0/1212411789 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d5de53e400 0x55d5de540880 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.654+0000 7f28bfb7f740 1 --2- 172.21.15.31:0/1212411789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5de52e460 0x55d5de52b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.654+0000 7f28bfb7f740 1 --2- 172.21.15.31:0/1212411789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d5de525dd0 0x55d5de52ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.654+0000 7f28bfb7f740 1 --2- 172.21.15.31:0/1212411789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d5de5250f0 0x55d5de52dac0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.654+0000 7f28bfb7f740 1 -- 172.21.15.31:0/1212411789 >> 172.21.15.31:0/1212411789 conn(0x55d5de4ec200 msgr2=0x55d5de521890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:39.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.654+0000 7f289be00640 0 client.28042 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:39.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.654+0000 7f28bfb7f740 1 -- 172.21.15.31:0/1212411789 shutdown_connections 2024-03-21T10:48:39.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.654+0000 7f28bfb7f740 1 -- 172.21.15.31:0/1212411789 wait complete. 2024-03-21T10:48:39.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.654+0000 7f28bfb7f740 20 client.28042 trim_cache size 0 max 16384 2024-03-21T10:48:39.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.671+0000 7f28bfb7f740 1 Processor -- start 2024-03-21T10:48:39.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.672+0000 7f28bfb7f740 1 -- start start 2024-03-21T10:48:39.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.672+0000 7f28bfb7f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d5de5250f0 0x55d5de52a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:39.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.672+0000 7f28bfb7f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5de525dd0 0x55d5de5216a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:39.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.672+0000 7f28bfb7f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d5de52e460 0x55d5de52d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:39.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.672+0000 7f28bfb7f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b81b2e40 con 0x55d5de525dd0 2024-03-21T10:48:39.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.672+0000 7f28bfb7f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b81b0940 con 0x55d5de52e460 2024-03-21T10:48:39.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.672+0000 7f28bfb7f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b81aea00 con 0x55d5de5250f0 2024-03-21T10:48:39.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.672+0000 7f28ad400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5de525dd0 0x55d5de5216a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:39.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.672+0000 7f28b6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d5de52e460 0x55d5de52d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:39.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.672+0000 7f28b5e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d5de5250f0 0x55d5de52a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:39.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.672+0000 7f28ad400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5de525dd0 0x55d5de5216a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57282/0 (socket says 172.21.15.31:57282) 2024-03-21T10:48:39.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.672+0000 7f28ad400640 1 -- 172.21.15.31:0/458131928 learned_addr learned my addr 172.21.15.31:0/458131928 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:39.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.673+0000 7f28b5e00640 1 -- 172.21.15.31:0/458131928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d5de52e460 msgr2=0x55d5de52d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:39.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.673+0000 7f28b5e00640 1 --2- 172.21.15.31:0/458131928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d5de52e460 0x55d5de52d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.673+0000 7f28b5e00640 1 -- 172.21.15.31:0/458131928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5de525dd0 msgr2=0x55d5de5216a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:39.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.673+0000 7f28b5e00640 1 --2- 172.21.15.31:0/458131928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5de525dd0 0x55d5de5216a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.673+0000 7f28b5e00640 1 -- 172.21.15.31:0/458131928 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d5de15d0f0 con 0x55d5de5250f0 2024-03-21T10:48:39.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.673+0000 7f28b5e00640 1 --2- 172.21.15.31:0/458131928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d5de5250f0 0x55d5de52a790 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f28a80097b0 tx=0x7f28a80982b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2faccfd73687317 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:39.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.673+0000 7f28a6a00640 1 -- 172.21.15.31:0/458131928 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a809bcb0 con 0x55d5de5250f0 2024-03-21T10:48:39.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.673+0000 7f28a6a00640 1 -- 172.21.15.31:0/458131928 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f28a80d3560 con 0x55d5de5250f0 2024-03-21T10:48:39.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.673+0000 7f28a6a00640 1 -- 172.21.15.31:0/458131928 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a80a4610 con 0x55d5de5250f0 2024-03-21T10:48:39.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.674+0000 7f28bfb7f740 1 -- 172.21.15.31:0/458131928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d5de5250f0 msgr2=0x55d5de52a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:39.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.674+0000 7f28bfb7f740 1 --2- 172.21.15.31:0/458131928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d5de5250f0 0x55d5de52a790 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f28a80097b0 tx=0x7f28a80982b0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.674+0000 7f28bfb7f740 1 -- 172.21.15.31:0/458131928 shutdown_connections 2024-03-21T10:48:39.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.674+0000 7f28bfb7f740 1 --2- 172.21.15.31:0/458131928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d5de52e460 0x55d5de52d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.674+0000 7f28bfb7f740 1 --2- 172.21.15.31:0/458131928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5de525dd0 0x55d5de5216a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.674+0000 7f28bfb7f740 1 --2- 172.21.15.31:0/458131928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d5de5250f0 0x55d5de52a790 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.674+0000 7f28bfb7f740 1 -- 172.21.15.31:0/458131928 >> 172.21.15.31:0/458131928 conn(0x55d5de4ec200 msgr2=0x55d5de520f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:39.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.674+0000 7f28bfb7f740 1 -- 172.21.15.31:0/458131928 shutdown_connections 2024-03-21T10:48:39.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.674+0000 7f28bfb7f740 1 -- 172.21.15.31:0/458131928 wait complete. 2024-03-21T10:48:39.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.674+0000 7f28bfb7f740 1 -- start start 2024-03-21T10:48:39.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.675+0000 7f28bfb7f740 1 Processor -- start 2024-03-21T10:48:39.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.675+0000 7f28bfb7f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d5de5250f0 0x55d5de528af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:39.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.675+0000 7f28bfb7f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d5de525dd0 0x55d5de527530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:39.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.675+0000 7f28bfb7f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5de52e460 0x55d5de527b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:39.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.675+0000 7f28bfb7f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b81af000 con 0x55d5de52e460 2024-03-21T10:48:39.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.675+0000 7f28bfb7f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b81b02e0 con 0x55d5de5250f0 2024-03-21T10:48:39.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.675+0000 7f28bfb7f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b81af620 con 0x55d5de525dd0 2024-03-21T10:48:39.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.675+0000 7f28bfb7f740 10 client.0 _ll_register_callbacks cb 0x7f28a800cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:39.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.675+0000 7f28b5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d5de5250f0 0x55d5de528af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:39.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.675+0000 7f28b6800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5de52e460 0x55d5de527b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:39.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.675+0000 7f28ad400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d5de525dd0 0x55d5de527530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:39.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.675+0000 7f28b5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d5de5250f0 0x55d5de528af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:47886/0 (socket says 172.21.15.31:47886) 2024-03-21T10:48:39.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.675+0000 7f28b5e00640 1 -- 172.21.15.31:0/654898865 learned_addr learned my addr 172.21.15.31:0/654898865 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:39.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.676+0000 7f28b5e00640 1 -- 172.21.15.31:0/654898865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d5de525dd0 msgr2=0x55d5de527530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:39.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.676+0000 7f28b5e00640 1 --2- 172.21.15.31:0/654898865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d5de525dd0 0x55d5de527530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.676+0000 7f28b5e00640 1 -- 172.21.15.31:0/654898865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5de52e460 msgr2=0x55d5de527b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:39.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.676+0000 7f28b5e00640 1 --2- 172.21.15.31:0/654898865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5de52e460 0x55d5de527b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.676+0000 7f28b5e00640 1 -- 172.21.15.31:0/654898865 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d5de528340 con 0x55d5de5250f0 2024-03-21T10:48:39.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.676+0000 7f28ad400640 1 --2- 172.21.15.31:0/654898865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d5de525dd0 0x55d5de527530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:39.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.676+0000 7f28b5e00640 1 --2- 172.21.15.31:0/654898865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d5de5250f0 0x55d5de528af0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f28a800c6a0 tx=0x7f28a80bc040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec464f7d4703b5a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:39.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.676+0000 7f289be00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:48:39.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.676+0000 7f289be00640 1 -- 172.21.15.31:0/654898865 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a809bcb0 con 0x55d5de5250f0 2024-03-21T10:48:39.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.676+0000 7f289be00640 1 -- 172.21.15.31:0/654898865 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f28a80baba0 con 0x55d5de5250f0 2024-03-21T10:48:39.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.676+0000 7f28bfb7f740 1 -- 172.21.15.31:0/654898865 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f28a80d3560 con 0x55d5de5250f0 2024-03-21T10:48:39.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.677+0000 7f289be00640 1 -- 172.21.15.31:0/654898865 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28a80ab050 con 0x55d5de5250f0 2024-03-21T10:48:39.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.677+0000 7f28bfb7f740 10 client.28121 fetch_fsmap learned FSMap version 28 2024-03-21T10:48:39.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.677+0000 7f28bfb7f740 1 -- 172.21.15.31:0/654898865 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d5de599e20 con 0x55d5de5250f0 2024-03-21T10:48:39.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.677+0000 7f289be00640 1 -- 172.21.15.31:0/654898865 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f28a80d3560 con 0x55d5de5250f0 2024-03-21T10:48:39.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.677+0000 7f289be00640 1 -- 172.21.15.31:0/654898865 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f28a809ada0 con 0x55d5de5250f0 2024-03-21T10:48:39.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.677+0000 7f28bfb7f740 10 client.28121 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:48:39.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.677+0000 7f28bfb7f740 10 client.28121 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:48:39.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.677+0000 7f28bfb7f740 20 client.28121 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:39.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.677+0000 7f28bfb7f740 1 --2- 172.21.15.31:0/654898865 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d5de59a300 0x55d5de59c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:39.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.677+0000 7f28bfb7f740 4 client.28121 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:48:39.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.677+0000 7f28bfb7f740 1 -- 172.21.15.31:0/654898865 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d5de490ae0 con 0x55d5de59a300 2024-03-21T10:48:39.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.677+0000 7f28ad400640 1 --2- 172.21.15.31:0/654898865 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d5de59a300 0x55d5de59c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:39.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.678+0000 7f28ad400640 1 --2- 172.21.15.31:0/654898865 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d5de59a300 0x55d5de59c720 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c604cf3ab3723204 server_cookie=5603d5095320d013 in_seq=0 out_seq=0 2024-03-21T10:48:39.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.678+0000 7f289be00640 10 client.28121 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:39.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.678+0000 7f289be00640 1 -- 172.21.15.31:0/654898865 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55d5de490ae0 con 0x55d5de59a300 2024-03-21T10:48:39.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.678+0000 7f289be00640 10 client.28121 handle_command_reply: tid=0 2024-03-21T10:48:39.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.678+0000 7f28bfb7f740 1 client.28121 shutdown 2024-03-21T10:48:39.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.678+0000 7f28bfb7f740 1 -- 172.21.15.31:0/654898865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d5de5250f0 msgr2=0x55d5de528af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:39.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.678+0000 7f28bfb7f740 1 --2- 172.21.15.31:0/654898865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d5de5250f0 0x55d5de528af0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f28a800c6a0 tx=0x7f28a80bc040 comp rx=0 tx=0).stop 2024-03-21T10:48:39.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.678+0000 7f28bfb7f740 1 -- 172.21.15.31:0/654898865 shutdown_connections 2024-03-21T10:48:39.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.678+0000 7f28bfb7f740 1 --2- 172.21.15.31:0/654898865 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55d5de59a300 0x55d5de59c720 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.678+0000 7f28bfb7f740 1 --2- 172.21.15.31:0/654898865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5de52e460 0x55d5de527b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.678+0000 7f289be00640 0 client.28121 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:39.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.678+0000 7f28bfb7f740 1 --2- 172.21.15.31:0/654898865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d5de525dd0 0x55d5de527530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.678+0000 7f28bfb7f740 1 --2- 172.21.15.31:0/654898865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d5de5250f0 0x55d5de528af0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.678+0000 7f28bfb7f740 1 -- 172.21.15.31:0/654898865 >> 172.21.15.31:0/654898865 conn(0x55d5de4ec200 msgr2=0x55d5de5215f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:39.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.679+0000 7f28bfb7f740 1 -- 172.21.15.31:0/654898865 shutdown_connections 2024-03-21T10:48:39.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.679+0000 7f28bfb7f740 1 -- 172.21.15.31:0/654898865 wait complete. 2024-03-21T10:48:39.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.679+0000 7f28bfb7f740 20 client.28121 trim_cache size 0 max 16384 2024-03-21T10:48:39.680 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:48:39.680 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-03-21T10:48:39.680 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-03-21T10:48:39.680 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:48:39.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.679+0000 7f28b7200640 1 -- 172.21.15.31:0/2750714465 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f289c0a3190 msgr2=0x7f289c0a5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:39.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.679+0000 7f28b7200640 1 --2- 172.21.15.31:0/2750714465 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f289c0a3190 0x7f289c0a5650 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f28b0006010 tx=0x7f28b0005b50 comp rx=0 tx=0).stop 2024-03-21T10:48:39.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.680+0000 7f28b7200640 1 -- 172.21.15.31:0/2750714465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f28b8101440 msgr2=0x7f28b8199ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:39.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.680+0000 7f28b7200640 1 --2- 172.21.15.31:0/2750714465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f28b8101440 0x7f28b8199ee0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f28a800e9f0 tx=0x7f28a800eeb0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.680+0000 7f28b7200640 1 -- 172.21.15.31:0/2750714465 shutdown_connections 2024-03-21T10:48:39.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.680+0000 7f28b7200640 1 --2- 172.21.15.31:0/2750714465 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f289c0a3190 0x7f289c0a5650 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.680+0000 7f28b7200640 1 --2- 172.21.15.31:0/2750714465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28b81074e0 0x7f28b819e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.680+0000 7f28b7200640 1 --2- 172.21.15.31:0/2750714465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28b8101e00 0x7f28b819a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.680+0000 7f28b7200640 1 --2- 172.21.15.31:0/2750714465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f28b8101440 0x7f28b8199ee0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.680+0000 7f28b7200640 1 -- 172.21.15.31:0/2750714465 >> 172.21.15.31:0/2750714465 conn(0x7f28b80fcf30 msgr2=0x7f28b810cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:39.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.680+0000 7f28b7200640 1 -- 172.21.15.31:0/2750714465 shutdown_connections 2024-03-21T10:48:39.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.680+0000 7f28b7200640 1 -- 172.21.15.31:0/2750714465 wait complete. 2024-03-21T10:48:39.702 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-21T10:48:39.702 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:48:39.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.932+0000 7f9cd7c00640 1 -- 172.21.15.31:0/1163599185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cd0102800 msgr2=0x7f9cd0102be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:39.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.932+0000 7f9cd7c00640 1 --2- 172.21.15.31:0/1163599185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cd0102800 0x7f9cd0102be0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4009900 tx=0x7f9cc402bcd0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.932+0000 7f9cd7c00640 1 -- 172.21.15.31:0/1163599185 shutdown_connections 2024-03-21T10:48:39.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.932+0000 7f9cd7c00640 1 --2- 172.21.15.31:0/1163599185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cd010bc20 0x7f9cd010e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.932+0000 7f9cd7c00640 1 --2- 172.21.15.31:0/1163599185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cd0103120 0x7f9cd010b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.932+0000 7f9cd7c00640 1 --2- 172.21.15.31:0/1163599185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cd0102800 0x7f9cd0102be0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.932+0000 7f9cd7c00640 1 -- 172.21.15.31:0/1163599185 >> 172.21.15.31:0/1163599185 conn(0x7f9cd00fc020 msgr2=0x7f9cd00fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:39.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.933+0000 7f9cd7c00640 1 -- 172.21.15.31:0/1163599185 shutdown_connections 2024-03-21T10:48:39.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.933+0000 7f9cd7c00640 1 -- 172.21.15.31:0/1163599185 wait complete. 2024-03-21T10:48:39.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.934+0000 7f9cd7c00640 1 Processor -- start 2024-03-21T10:48:39.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.934+0000 7f9cd7c00640 1 -- start start 2024-03-21T10:48:39.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.935+0000 7f9cd7c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cd0102800 0x7f9cd0103670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:39.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.935+0000 7f9cd7c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cd0103120 0x7f9cd0103bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:39.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.935+0000 7f9cd7c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cd010bc20 0x7f9cd01040f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:39.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.935+0000 7f9cd7c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd0111a70 con 0x7f9cd0102800 2024-03-21T10:48:39.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.935+0000 7f9cd7c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd01118f0 con 0x7f9cd010bc20 2024-03-21T10:48:39.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.935+0000 7f9cd7c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd0111bf0 con 0x7f9cd0103120 2024-03-21T10:48:39.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.935+0000 7f9ccfe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cd0103120 0x7f9cd0103bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:39.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.935+0000 7f9ccfe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cd0103120 0x7f9cd0103bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:39090/0 (socket says 172.21.15.31:39090) 2024-03-21T10:48:39.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.935+0000 7f9ccfe00640 1 -- 172.21.15.31:0/1238222827 learned_addr learned my addr 172.21.15.31:0/1238222827 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:39.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.935+0000 7f9cd5200640 1 --2- 172.21.15.31:0/1238222827 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cd0102800 0x7f9cd0103670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:39.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.935+0000 7f9cd5c00640 1 --2- 172.21.15.31:0/1238222827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cd010bc20 0x7f9cd01040f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:39.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.935+0000 7f9ccfe00640 1 -- 172.21.15.31:0/1238222827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cd010bc20 msgr2=0x7f9cd01040f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:39.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.935+0000 7f9ccfe00640 1 --2- 172.21.15.31:0/1238222827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cd010bc20 0x7f9cd01040f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.935+0000 7f9ccfe00640 1 -- 172.21.15.31:0/1238222827 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cd0102800 msgr2=0x7f9cd0103670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:39.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.935+0000 7f9ccfe00640 1 --2- 172.21.15.31:0/1238222827 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cd0102800 0x7f9cd0103670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:39.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.935+0000 7f9ccfe00640 1 -- 172.21.15.31:0/1238222827 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd01aa800 con 0x7f9cd0103120 2024-03-21T10:48:39.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.935+0000 7f9cd5c00640 1 --2- 172.21.15.31:0/1238222827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cd010bc20 0x7f9cd01040f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:39.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.936+0000 7f9ccfe00640 1 --2- 172.21.15.31:0/1238222827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cd0103120 0x7f9cd0103bb0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc00e9f0 tx=0x7f9cbc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:39.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.936+0000 7f9ccd600640 1 -- 172.21.15.31:0/1238222827 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc00ccc0 con 0x7f9cd0103120 2024-03-21T10:48:39.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.936+0000 7f9ccd600640 1 -- 172.21.15.31:0/1238222827 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9cbc010910 con 0x7f9cd0103120 2024-03-21T10:48:39.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.936+0000 7f9cd7c00640 1 -- 172.21.15.31:0/1238222827 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cd01aaaf0 con 0x7f9cd0103120 2024-03-21T10:48:39.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.936+0000 7f9ccd600640 1 -- 172.21.15.31:0/1238222827 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc018920 con 0x7f9cd0103120 2024-03-21T10:48:39.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.936+0000 7f9cd7c00640 1 -- 172.21.15.31:0/1238222827 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cd01ab090 con 0x7f9cd0103120 2024-03-21T10:48:39.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.937+0000 7f9cd7c00640 1 -- 172.21.15.31:0/1238222827 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cd006a660 con 0x7f9cd0103120 2024-03-21T10:48:39.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.940+0000 7f9ccd600640 1 -- 172.21.15.31:0/1238222827 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f9cbc010430 con 0x7f9cd0103120 2024-03-21T10:48:39.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.941+0000 7f9ccd600640 1 --2- 172.21.15.31:0/1238222827 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9cb80a3110 0x7f9cb80a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:39.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.942+0000 7f9ccd600640 1 -- 172.21.15.31:0/1238222827 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f9cbc0d0280 con 0x7f9cd0103120 2024-03-21T10:48:39.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.942+0000 7f9cd5200640 1 --2- 172.21.15.31:0/1238222827 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9cb80a3110 0x7f9cb80a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:39.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.942+0000 7f9cd5200640 1 --2- 172.21.15.31:0/1238222827 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9cb80a3110 0x7f9cb80a55d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4005ab0 tx=0x7f9cc4005a20 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:39.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:39.943+0000 7f9ccd600640 1 -- 172.21.15.31:0/1238222827 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9cbc09c320 con 0x7f9cd0103120 2024-03-21T10:48:40.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.189+0000 7f9cd7c00640 1 -- 172.21.15.31:0/1238222827 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cd01ab380 con 0x7f9cd0103120 2024-03-21T10:48:40.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.190+0000 7f9ccd600640 1 -- 172.21.15.31:0/1238222827 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11387 (secure 0 0 0) 0x7f9cbc0a03c0 con 0x7f9cd0103120 2024-03-21T10:48:40.187 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:48:40.187 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:46:51.185295+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24547,24596]},"id":1}]} 2024-03-21T10:48:40.187 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-03-21T10:48:40.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.193+0000 7f9cd7c00640 1 -- 172.21.15.31:0/1238222827 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9cb80a3110 msgr2=0x7f9cb80a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:40.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.193+0000 7f9cd7c00640 1 --2- 172.21.15.31:0/1238222827 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9cb80a3110 0x7f9cb80a55d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4005ab0 tx=0x7f9cc4005a20 comp rx=0 tx=0).stop 2024-03-21T10:48:40.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.193+0000 7f9cd7c00640 1 -- 172.21.15.31:0/1238222827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cd0103120 msgr2=0x7f9cd0103bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:40.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.193+0000 7f9cd7c00640 1 --2- 172.21.15.31:0/1238222827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cd0103120 0x7f9cd0103bb0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc00e9f0 tx=0x7f9cbc00eeb0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.193+0000 7f9cd7c00640 1 -- 172.21.15.31:0/1238222827 shutdown_connections 2024-03-21T10:48:40.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.193+0000 7f9cd7c00640 1 --2- 172.21.15.31:0/1238222827 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9cb80a3110 0x7f9cb80a55d0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.193+0000 7f9cd7c00640 1 --2- 172.21.15.31:0/1238222827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cd010bc20 0x7f9cd01040f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.193+0000 7f9cd7c00640 1 --2- 172.21.15.31:0/1238222827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cd0103120 0x7f9cd0103bb0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.193+0000 7f9cd7c00640 1 --2- 172.21.15.31:0/1238222827 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cd0102800 0x7f9cd0103670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.193+0000 7f9cd7c00640 1 -- 172.21.15.31:0/1238222827 >> 172.21.15.31:0/1238222827 conn(0x7f9cd00fc020 msgr2=0x7f9cd0100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:40.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.193+0000 7f9cd7c00640 1 -- 172.21.15.31:0/1238222827 shutdown_connections 2024-03-21T10:48:40.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.193+0000 7f9cd7c00640 1 -- 172.21.15.31:0/1238222827 wait complete. 2024-03-21T10:48:40.210 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:48:40.211 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-21T10:48:40.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.434+0000 7f147f400640 1 -- 172.21.15.31:0/950322867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1478108760 msgr2=0x7f1478114c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:40.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.434+0000 7f147f400640 1 --2- 172.21.15.31:0/950322867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1478108760 0x7f1478114c30 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f1470009cd0 tx=0x7f147002c0f0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.434+0000 7f147f400640 1 -- 172.21.15.31:0/950322867 shutdown_connections 2024-03-21T10:48:40.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.435+0000 7f147f400640 1 --2- 172.21.15.31:0/950322867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1478108760 0x7f1478114c30 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.435+0000 7f147f400640 1 --2- 172.21.15.31:0/950322867 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1478107dc0 0x7f1478108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.435+0000 7f147f400640 1 --2- 172.21.15.31:0/950322867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1478109b70 0x7f1478109f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.435+0000 7f147f400640 1 -- 172.21.15.31:0/950322867 >> 172.21.15.31:0/950322867 conn(0x7f1478077540 msgr2=0x7f14781064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:40.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.435+0000 7f147f400640 1 -- 172.21.15.31:0/950322867 shutdown_connections 2024-03-21T10:48:40.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.435+0000 7f147f400640 1 -- 172.21.15.31:0/950322867 wait complete. 2024-03-21T10:48:40.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.436+0000 7f147f400640 1 Processor -- start 2024-03-21T10:48:40.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.436+0000 7f147f400640 1 -- start start 2024-03-21T10:48:40.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.437+0000 7f147f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1478107dc0 0x7f147819e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.437+0000 7f147f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1478108760 0x7f147819eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.437+0000 7f147f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1478109b70 0x7f14781a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.437+0000 7f147f400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1478118100 con 0x7f1478109b70 2024-03-21T10:48:40.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.437+0000 7f147f400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1478117f80 con 0x7f1478107dc0 2024-03-21T10:48:40.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.437+0000 7f147f400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1478118280 con 0x7f1478108760 2024-03-21T10:48:40.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.437+0000 7f147ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1478107dc0 0x7f147819e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.437+0000 7f147ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1478107dc0 0x7f147819e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:47932/0 (socket says 172.21.15.31:47932) 2024-03-21T10:48:40.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.437+0000 7f147ca00640 1 -- 172.21.15.31:0/3955722314 learned_addr learned my addr 172.21.15.31:0/3955722314 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:40.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.437+0000 7f1477e00640 1 --2- 172.21.15.31:0/3955722314 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1478108760 0x7f147819eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.437+0000 7f147d400640 1 --2- 172.21.15.31:0/3955722314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1478109b70 0x7f14781a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.437+0000 7f147ca00640 1 -- 172.21.15.31:0/3955722314 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1478108760 msgr2=0x7f147819eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:40.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.437+0000 7f147ca00640 1 --2- 172.21.15.31:0/3955722314 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1478108760 0x7f147819eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.437+0000 7f147ca00640 1 -- 172.21.15.31:0/3955722314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1478109b70 msgr2=0x7f14781a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:40.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.437+0000 7f147ca00640 1 --2- 172.21.15.31:0/3955722314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1478109b70 0x7f14781a2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.437+0000 7f147ca00640 1 -- 172.21.15.31:0/3955722314 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14781a3590 con 0x7f1478107dc0 2024-03-21T10:48:40.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.438+0000 7f147ca00640 1 --2- 172.21.15.31:0/3955722314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1478107dc0 0x7f147819e5a0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f146c00e9f0 tx=0x7f146c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:40.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.438+0000 7f1475600640 1 -- 172.21.15.31:0/3955722314 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c00cd80 con 0x7f1478107dc0 2024-03-21T10:48:40.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.438+0000 7f147f400640 1 -- 172.21.15.31:0/3955722314 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14781a3880 con 0x7f1478107dc0 2024-03-21T10:48:40.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.438+0000 7f1475600640 1 -- 172.21.15.31:0/3955722314 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f146c010910 con 0x7f1478107dc0 2024-03-21T10:48:40.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.438+0000 7f1475600640 1 -- 172.21.15.31:0/3955722314 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c0188f0 con 0x7f1478107dc0 2024-03-21T10:48:40.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.438+0000 7f147f400640 1 -- 172.21.15.31:0/3955722314 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14781ab110 con 0x7f1478107dc0 2024-03-21T10:48:40.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.443+0000 7f1482057740 1 Processor -- start 2024-03-21T10:48:40.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.443+0000 7f1475600640 1 -- 172.21.15.31:0/3955722314 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f146c010ac0 con 0x7f1478107dc0 2024-03-21T10:48:40.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.443+0000 7f1482057740 1 -- start start 2024-03-21T10:48:40.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.443+0000 7f1482057740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612b9b3b6d0 0x5612b9b3bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.443+0000 7f1482057740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612b9b3c080 0x5612b9b44630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.443+0000 7f1482057740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612b9b44c00 0x5612b9b47050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.443+0000 7f1482057740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612b9a9f2c0 con 0x5612b9b44c00 2024-03-21T10:48:40.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.443+0000 7f1482057740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612b9a47c90 con 0x5612b9b3c080 2024-03-21T10:48:40.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.443+0000 7f1482057740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612b9a9b360 con 0x5612b9b3b6d0 2024-03-21T10:48:40.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.443+0000 7f147d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612b9b3b6d0 0x5612b9b3bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.443+0000 7f147ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612b9b3c080 0x5612b9b44630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.443+0000 7f1477e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612b9b44c00 0x5612b9b47050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.443+0000 7f147d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612b9b3b6d0 0x5612b9b3bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:39120/0 (socket says 172.21.15.31:39120) 2024-03-21T10:48:40.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.443+0000 7f147d400640 1 -- 172.21.15.31:0/669557769 learned_addr learned my addr 172.21.15.31:0/669557769 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:40.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.444+0000 7f147d400640 1 -- 172.21.15.31:0/669557769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612b9b3c080 msgr2=0x5612b9b44630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:40.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.444+0000 7f147d400640 1 --2- 172.21.15.31:0/669557769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612b9b3c080 0x5612b9b44630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.444+0000 7f147d400640 1 -- 172.21.15.31:0/669557769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612b9b44c00 msgr2=0x5612b9b47050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:40.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.444+0000 7f147d400640 1 --2- 172.21.15.31:0/669557769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612b9b44c00 0x5612b9b47050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.444+0000 7f147d400640 1 -- 172.21.15.31:0/669557769 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5612b9a6c080 con 0x5612b9b3b6d0 2024-03-21T10:48:40.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.444+0000 7f147d400640 1 --2- 172.21.15.31:0/669557769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612b9b3b6d0 0x5612b9b3bab0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f1470004af0 tx=0x7f14700098a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15230fe28d43d64e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:40.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.444+0000 7f1469c00640 1 -- 172.21.15.31:0/669557769 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1470043070 con 0x5612b9b3b6d0 2024-03-21T10:48:40.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.444+0000 7f1469c00640 1 -- 172.21.15.31:0/669557769 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1470032c90 con 0x5612b9b3b6d0 2024-03-21T10:48:40.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.445+0000 7f1469c00640 1 -- 172.21.15.31:0/669557769 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1470039040 con 0x5612b9b3b6d0 2024-03-21T10:48:40.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.445+0000 7f1475600640 1 --2- 172.21.15.31:0/3955722314 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f14600a3090 0x7f14600a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.445+0000 7f1475600640 1 -- 172.21.15.31:0/3955722314 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f146c0cffa0 con 0x7f1478107dc0 2024-03-21T10:48:40.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.445+0000 7f147ca00640 1 --2- 172.21.15.31:0/3955722314 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f14600a3090 0x7f14600a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.445+0000 7f1482057740 1 -- 172.21.15.31:0/669557769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612b9b3b6d0 msgr2=0x5612b9b3bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:40.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.445+0000 7f1482057740 1 --2- 172.21.15.31:0/669557769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612b9b3b6d0 0x5612b9b3bab0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f1470004af0 tx=0x7f14700098a0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.445+0000 7f1482057740 1 -- 172.21.15.31:0/669557769 shutdown_connections 2024-03-21T10:48:40.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.445+0000 7f1482057740 1 --2- 172.21.15.31:0/669557769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612b9b44c00 0x5612b9b47050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.445+0000 7f1482057740 1 --2- 172.21.15.31:0/669557769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612b9b3c080 0x5612b9b44630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.445+0000 7f1482057740 1 --2- 172.21.15.31:0/669557769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612b9b3b6d0 0x5612b9b3bab0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.445+0000 7f1482057740 1 -- 172.21.15.31:0/669557769 >> 172.21.15.31:0/669557769 conn(0x5612b9b35da0 msgr2=0x5612b9b36180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:40.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.445+0000 7f1482057740 1 -- 172.21.15.31:0/669557769 shutdown_connections 2024-03-21T10:48:40.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.446+0000 7f1482057740 1 -- 172.21.15.31:0/669557769 wait complete. 2024-03-21T10:48:40.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.446+0000 7f147ca00640 1 --2- 172.21.15.31:0/3955722314 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f14600a3090 0x7f14600a5550 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x5612b9b3b1a0 tx=0x7f146c0d3a50 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:40.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.446+0000 7f1482057740 1 -- start start 2024-03-21T10:48:40.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.446+0000 7f1482057740 1 Processor -- start 2024-03-21T10:48:40.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.447+0000 7f1482057740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612b9b3b6d0 0x5612b9b444c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.447+0000 7f1482057740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612b9b3c080 0x5612b9b41850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.447+0000 7f1482057740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612b9b44c00 0x5612b9b41db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.447+0000 7f1482057740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612b9b47b10 con 0x5612b9b44c00 2024-03-21T10:48:40.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.447+0000 7f1482057740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612b9a5b0d0 con 0x5612b9b3c080 2024-03-21T10:48:40.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.447+0000 7f1482057740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612b9b47c90 con 0x5612b9b3b6d0 2024-03-21T10:48:40.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.447+0000 7f1482057740 10 client.0 _ll_register_callbacks cb 0x5612b9a5fe80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:40.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.447+0000 7f1477e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612b9b3c080 0x5612b9b41850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.447+0000 7f1477e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612b9b3c080 0x5612b9b41850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:47954/0 (socket says 172.21.15.31:47954) 2024-03-21T10:48:40.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.447+0000 7f1477e00640 1 -- 172.21.15.31:0/2710601085 learned_addr learned my addr 172.21.15.31:0/2710601085 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:40.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.447+0000 7f147d400640 1 --2- 172.21.15.31:0/2710601085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612b9b3b6d0 0x5612b9b444c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.447+0000 7f147d400640 1 --2- 172.21.15.31:0/2710601085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612b9b44c00 0x5612b9b41db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.447+0000 7f1477e00640 1 -- 172.21.15.31:0/2710601085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612b9b3b6d0 msgr2=0x5612b9b444c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:40.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.447+0000 7f1477e00640 1 --2- 172.21.15.31:0/2710601085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612b9b3b6d0 0x5612b9b444c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.447+0000 7f1477e00640 1 -- 172.21.15.31:0/2710601085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612b9b44c00 msgr2=0x5612b9b41db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:40.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.447+0000 7f1477e00640 1 --2- 172.21.15.31:0/2710601085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612b9b44c00 0x5612b9b41db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.447+0000 7f1477e00640 1 -- 172.21.15.31:0/2710601085 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5612b97b33f0 con 0x5612b9b3c080 2024-03-21T10:48:40.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.448+0000 7f1477e00640 1 --2- 172.21.15.31:0/2710601085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612b9b3c080 0x5612b9b41850 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f1464012700 tx=0x7f1464012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17c1bac40092a2ad server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:40.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.448+0000 7f145e000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:48:40.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.448+0000 7f145e000640 1 -- 172.21.15.31:0/2710601085 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146400f920 con 0x5612b9b3c080 2024-03-21T10:48:40.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.448+0000 7f145e000640 1 -- 172.21.15.31:0/2710601085 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f146400fac0 con 0x5612b9b3c080 2024-03-21T10:48:40.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.448+0000 7f1482057740 1 -- 172.21.15.31:0/2710601085 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1470032c90 con 0x5612b9b3c080 2024-03-21T10:48:40.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.448+0000 7f145e000640 1 -- 172.21.15.31:0/2710601085 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f146400fcc0 con 0x5612b9b3c080 2024-03-21T10:48:40.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.448+0000 7f1482057740 10 client.28139 fetch_fsmap learned FSMap version 28 2024-03-21T10:48:40.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.448+0000 7f1482057740 1 -- 172.21.15.31:0/2710601085 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5612b987d7d0 con 0x5612b9b3c080 2024-03-21T10:48:40.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.449+0000 7f145e000640 1 -- 172.21.15.31:0/2710601085 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1470032c90 con 0x5612b9b3c080 2024-03-21T10:48:40.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.449+0000 7f145e000640 1 -- 172.21.15.31:0/2710601085 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f146401c420 con 0x5612b9b3c080 2024-03-21T10:48:40.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.449+0000 7f1482057740 10 client.28139 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:48:40.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.449+0000 7f1482057740 10 client.28139 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:48:40.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.449+0000 7f1482057740 20 client.28139 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:40.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.449+0000 7f1482057740 1 --2- 172.21.15.31:0/2710601085 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5612b9b54a90 0x5612b9b56eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.449+0000 7f1482057740 4 client.28139 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:48:40.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.449+0000 7f1482057740 1 -- 172.21.15.31:0/2710601085 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5612b9aac800 con 0x5612b9b54a90 2024-03-21T10:48:40.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.449+0000 7f147d400640 1 --2- 172.21.15.31:0/2710601085 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5612b9b54a90 0x5612b9b56eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.450+0000 7f147d400640 1 --2- 172.21.15.31:0/2710601085 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5612b9b54a90 0x5612b9b56eb0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=44a29bfe8cd5a011 server_cookie=93a9ab6181c0d1b1 in_seq=0 out_seq=0 2024-03-21T10:48:40.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.450+0000 7f145e000640 10 client.28139 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:48:40.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.451+0000 7f145e000640 1 -- 172.21.15.31:0/2710601085 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5612b9aac800 con 0x5612b9b54a90 2024-03-21T10:48:40.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.451+0000 7f145e000640 10 client.28139 handle_command_reply: tid=0 2024-03-21T10:48:40.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.451+0000 7f1482057740 1 client.28139 shutdown 2024-03-21T10:48:40.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.451+0000 7f1482057740 1 -- 172.21.15.31:0/2710601085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612b9b3c080 msgr2=0x5612b9b41850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:40.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.451+0000 7f1482057740 1 --2- 172.21.15.31:0/2710601085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612b9b3c080 0x5612b9b41850 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f1464012700 tx=0x7f1464012bc0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.451+0000 7f1482057740 1 -- 172.21.15.31:0/2710601085 shutdown_connections 2024-03-21T10:48:40.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.451+0000 7f1482057740 1 --2- 172.21.15.31:0/2710601085 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5612b9b54a90 0x5612b9b56eb0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.451+0000 7f1482057740 1 --2- 172.21.15.31:0/2710601085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612b9b44c00 0x5612b9b41db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.451+0000 7f1482057740 1 --2- 172.21.15.31:0/2710601085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612b9b3c080 0x5612b9b41850 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.451+0000 7f1482057740 1 --2- 172.21.15.31:0/2710601085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612b9b3b6d0 0x5612b9b444c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.451+0000 7f1482057740 1 -- 172.21.15.31:0/2710601085 >> 172.21.15.31:0/2710601085 conn(0x5612b9b35da0 msgr2=0x5612b9b37af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:40.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.451+0000 7f145e000640 0 client.28139 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:48:40.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.451+0000 7f1482057740 1 -- 172.21.15.31:0/2710601085 shutdown_connections 2024-03-21T10:48:40.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.452+0000 7f1482057740 1 -- 172.21.15.31:0/2710601085 wait complete. 2024-03-21T10:48:40.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.452+0000 7f1482057740 20 client.28139 trim_cache size 0 max 16384 2024-03-21T10:48:40.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.465+0000 7f1482057740 1 Processor -- start 2024-03-21T10:48:40.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.465+0000 7f1482057740 1 -- start start 2024-03-21T10:48:40.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.466+0000 7f1482057740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612b9b3b6d0 0x5612b9b37860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.466+0000 7f1482057740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612b9b3c080 0x5612b9b43f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.466+0000 7f1482057740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612b9b44c00 0x5612b9b444c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.466+0000 7f1482057740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14781b7440 con 0x5612b9b3c080 2024-03-21T10:48:40.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.466+0000 7f1482057740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14781b4f40 con 0x5612b9b3b6d0 2024-03-21T10:48:40.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.466+0000 7f1482057740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14781b3000 con 0x5612b9b44c00 2024-03-21T10:48:40.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.466+0000 7f147d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612b9b3b6d0 0x5612b9b37860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.466+0000 7f147d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612b9b3b6d0 0x5612b9b37860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:47956/0 (socket says 172.21.15.31:47956) 2024-03-21T10:48:40.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.466+0000 7f147d400640 1 -- 172.21.15.31:0/1851073149 learned_addr learned my addr 172.21.15.31:0/1851073149 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:40.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.466+0000 7f147d400640 1 --2- 172.21.15.31:0/1851073149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612b9b44c00 0x5612b9b444c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.466+0000 7f1477e00640 1 --2- 172.21.15.31:0/1851073149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612b9b3c080 0x5612b9b43f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.466+0000 7f147d400640 1 -- 172.21.15.31:0/1851073149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612b9b44c00 msgr2=0x5612b9b444c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:40.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.466+0000 7f147d400640 1 --2- 172.21.15.31:0/1851073149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612b9b44c00 0x5612b9b444c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.466+0000 7f147d400640 1 -- 172.21.15.31:0/1851073149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612b9b3c080 msgr2=0x5612b9b43f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:40.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.466+0000 7f147d400640 1 --2- 172.21.15.31:0/1851073149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612b9b3c080 0x5612b9b43f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.466+0000 7f147d400640 1 -- 172.21.15.31:0/1851073149 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5612b9b57510 con 0x5612b9b3b6d0 2024-03-21T10:48:40.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.466+0000 7f1477e00640 1 --2- 172.21.15.31:0/1851073149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612b9b3c080 0x5612b9b43f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:48:40.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.466+0000 7f147d400640 1 --2- 172.21.15.31:0/1851073149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612b9b3b6d0 0x5612b9b37860 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f1470032f80 tx=0x7f147002c0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a80a49c4d6bb99db server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:40.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.467+0000 7f1469200640 1 -- 172.21.15.31:0/1851073149 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1470052040 con 0x5612b9b3b6d0 2024-03-21T10:48:40.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.467+0000 7f1469200640 1 -- 172.21.15.31:0/1851073149 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f147004e030 con 0x5612b9b3b6d0 2024-03-21T10:48:40.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.467+0000 7f1469200640 1 -- 172.21.15.31:0/1851073149 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1470039040 con 0x5612b9b3b6d0 2024-03-21T10:48:40.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.467+0000 7f1482057740 1 -- 172.21.15.31:0/1851073149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612b9b3b6d0 msgr2=0x5612b9b37860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:40.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.467+0000 7f1482057740 1 --2- 172.21.15.31:0/1851073149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612b9b3b6d0 0x5612b9b37860 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f1470032f80 tx=0x7f147002c0f0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.467+0000 7f1482057740 1 -- 172.21.15.31:0/1851073149 shutdown_connections 2024-03-21T10:48:40.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.467+0000 7f1482057740 1 --2- 172.21.15.31:0/1851073149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612b9b44c00 0x5612b9b444c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.467+0000 7f1482057740 1 --2- 172.21.15.31:0/1851073149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612b9b3c080 0x5612b9b43f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.467+0000 7f1482057740 1 --2- 172.21.15.31:0/1851073149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612b9b3b6d0 0x5612b9b37860 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.467+0000 7f1482057740 1 -- 172.21.15.31:0/1851073149 >> 172.21.15.31:0/1851073149 conn(0x5612b9b35da0 msgr2=0x5612b9b37150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:40.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.467+0000 7f1482057740 1 -- 172.21.15.31:0/1851073149 shutdown_connections 2024-03-21T10:48:40.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.468+0000 7f1482057740 1 -- 172.21.15.31:0/1851073149 wait complete. 2024-03-21T10:48:40.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.468+0000 7f1482057740 1 -- start start 2024-03-21T10:48:40.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.468+0000 7f1482057740 1 Processor -- start 2024-03-21T10:48:40.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.469+0000 7f1482057740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612b9b3b6d0 0x5612b9b4c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.469+0000 7f1482057740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612b9b3c080 0x5612b9b4cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.469+0000 7f1482057740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612b9b44c00 0x5612b9b48e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.469+0000 7f1482057740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14781b3600 con 0x5612b9b3c080 2024-03-21T10:48:40.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.469+0000 7f1482057740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14781b48e0 con 0x5612b9b44c00 2024-03-21T10:48:40.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.469+0000 7f1482057740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14781b3c20 con 0x5612b9b3b6d0 2024-03-21T10:48:40.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.469+0000 7f1482057740 10 client.0 _ll_register_callbacks cb 0x7f1464009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:40.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.469+0000 7f147d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612b9b3b6d0 0x5612b9b4c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.469+0000 7f1477e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612b9b3c080 0x5612b9b4cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.469+0000 7f147d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612b9b3b6d0 0x5612b9b4c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:39136/0 (socket says 172.21.15.31:39136) 2024-03-21T10:48:40.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.469+0000 7f147d400640 1 -- 172.21.15.31:0/3485554552 learned_addr learned my addr 172.21.15.31:0/3485554552 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:40.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.469+0000 7f147d400640 1 --2- 172.21.15.31:0/3485554552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612b9b44c00 0x5612b9b48e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.470+0000 7f147d400640 1 -- 172.21.15.31:0/3485554552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612b9b44c00 msgr2=0x5612b9b48e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:40.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.470+0000 7f147d400640 1 --2- 172.21.15.31:0/3485554552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612b9b44c00 0x5612b9b48e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.470+0000 7f147d400640 1 -- 172.21.15.31:0/3485554552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612b9b3c080 msgr2=0x5612b9b4cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:40.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.470+0000 7f147d400640 1 --2- 172.21.15.31:0/3485554552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612b9b3c080 0x5612b9b4cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.470+0000 7f147d400640 1 -- 172.21.15.31:0/3485554552 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5612b9b49610 con 0x5612b9b3b6d0 2024-03-21T10:48:40.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.470+0000 7f1477e00640 1 --2- 172.21.15.31:0/3485554552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612b9b3c080 0x5612b9b4cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:40.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.470+0000 7f147d400640 1 --2- 172.21.15.31:0/3485554552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612b9b3b6d0 0x5612b9b4c890 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f1470009bf0 tx=0x7f1470009620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4b4ca1b3fb14500 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:40.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.470+0000 7f145e000640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:48:40.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.470+0000 7f145e000640 1 -- 172.21.15.31:0/3485554552 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1470039070 con 0x5612b9b3b6d0 2024-03-21T10:48:40.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.470+0000 7f1482057740 1 -- 172.21.15.31:0/3485554552 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f147004e030 con 0x5612b9b3b6d0 2024-03-21T10:48:40.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.470+0000 7f145e000640 1 -- 172.21.15.31:0/3485554552 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f1470004b40 con 0x5612b9b3b6d0 2024-03-21T10:48:40.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.471+0000 7f145e000640 1 -- 172.21.15.31:0/3485554552 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1470043040 con 0x5612b9b3b6d0 2024-03-21T10:48:40.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.471+0000 7f1482057740 10 client.28072 fetch_fsmap learned FSMap version 28 2024-03-21T10:48:40.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.471+0000 7f1482057740 1 -- 172.21.15.31:0/3485554552 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5612b9bb0f40 con 0x5612b9b3b6d0 2024-03-21T10:48:40.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.471+0000 7f145e000640 1 -- 172.21.15.31:0/3485554552 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f147004e030 con 0x5612b9b3b6d0 2024-03-21T10:48:40.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.471+0000 7f145e000640 1 -- 172.21.15.31:0/3485554552 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f14700537c0 con 0x5612b9b3b6d0 2024-03-21T10:48:40.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.471+0000 7f1482057740 10 client.28072 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:48:40.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.471+0000 7f1482057740 10 client.28072 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:48:40.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.471+0000 7f1482057740 20 client.28072 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:40.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.471+0000 7f1482057740 1 --2- 172.21.15.31:0/3485554552 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5612b9bb1110 0x5612b9bb3500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.471+0000 7f1482057740 4 client.28072 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:48:40.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.471+0000 7f1482057740 1 -- 172.21.15.31:0/3485554552 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5612b9aa6b50 con 0x5612b9bb1110 2024-03-21T10:48:40.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.472+0000 7f1477e00640 1 --2- 172.21.15.31:0/3485554552 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5612b9bb1110 0x5612b9bb3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.472+0000 7f1477e00640 1 --2- 172.21.15.31:0/3485554552 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5612b9bb1110 0x5612b9bb3500 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2c33e50d5536cf0b server_cookie=bec0491d09020be2 in_seq=0 out_seq=0 2024-03-21T10:48:40.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.472+0000 7f145e000640 10 client.28072 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:48:40.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.472+0000 7f145e000640 1 -- 172.21.15.31:0/3485554552 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5612b9aa6b50 con 0x5612b9bb1110 2024-03-21T10:48:40.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.472+0000 7f145e000640 10 client.28072 handle_command_reply: tid=0 2024-03-21T10:48:40.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.472+0000 7f1482057740 1 client.28072 shutdown 2024-03-21T10:48:40.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.472+0000 7f1482057740 1 -- 172.21.15.31:0/3485554552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612b9b3b6d0 msgr2=0x5612b9b4c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:40.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.472+0000 7f1482057740 1 --2- 172.21.15.31:0/3485554552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612b9b3b6d0 0x5612b9b4c890 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f1470009bf0 tx=0x7f1470009620 comp rx=0 tx=0).stop 2024-03-21T10:48:40.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.472+0000 7f1482057740 1 -- 172.21.15.31:0/3485554552 shutdown_connections 2024-03-21T10:48:40.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.472+0000 7f1482057740 1 --2- 172.21.15.31:0/3485554552 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x5612b9bb1110 0x5612b9bb3500 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.472+0000 7f1482057740 1 --2- 172.21.15.31:0/3485554552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612b9b44c00 0x5612b9b48e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.472+0000 7f1482057740 1 --2- 172.21.15.31:0/3485554552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612b9b3c080 0x5612b9b4cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.472+0000 7f1482057740 1 --2- 172.21.15.31:0/3485554552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5612b9b3b6d0 0x5612b9b4c890 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.472+0000 7f145e000640 0 client.28072 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:48:40.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.472+0000 7f1482057740 1 -- 172.21.15.31:0/3485554552 >> 172.21.15.31:0/3485554552 conn(0x5612b9b35da0 msgr2=0x5612b9b34eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:40.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.473+0000 7f1482057740 1 -- 172.21.15.31:0/3485554552 shutdown_connections 2024-03-21T10:48:40.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.473+0000 7f1482057740 1 -- 172.21.15.31:0/3485554552 wait complete. 2024-03-21T10:48:40.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.473+0000 7f1482057740 20 client.28072 trim_cache size 0 max 16384 2024-03-21T10:48:40.473 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:48:40.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.473+0000 7f147f400640 1 -- 172.21.15.31:0/3955722314 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f14600a3090 msgr2=0x7f14600a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:40.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.473+0000 7f147f400640 1 --2- 172.21.15.31:0/3955722314 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f14600a3090 0x7f14600a5550 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x5612b9b3b1a0 tx=0x7f146c0d3a50 comp rx=0 tx=0).stop 2024-03-21T10:48:40.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.473+0000 7f147f400640 1 -- 172.21.15.31:0/3955722314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1478107dc0 msgr2=0x7f147819e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:40.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.473+0000 7f147f400640 1 --2- 172.21.15.31:0/3955722314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1478107dc0 0x7f147819e5a0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f146c00e9f0 tx=0x7f146c00eeb0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.474+0000 7f147f400640 1 -- 172.21.15.31:0/3955722314 shutdown_connections 2024-03-21T10:48:40.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.474+0000 7f147f400640 1 --2- 172.21.15.31:0/3955722314 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f14600a3090 0x7f14600a5550 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.474+0000 7f147f400640 1 --2- 172.21.15.31:0/3955722314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1478109b70 0x7f14781a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.474+0000 7f147f400640 1 --2- 172.21.15.31:0/3955722314 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1478108760 0x7f147819eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.474+0000 7f147f400640 1 --2- 172.21.15.31:0/3955722314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1478107dc0 0x7f147819e5a0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.474+0000 7f147f400640 1 -- 172.21.15.31:0/3955722314 >> 172.21.15.31:0/3955722314 conn(0x7f1478077540 msgr2=0x7f1478077a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:40.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.474+0000 7f147f400640 1 -- 172.21.15.31:0/3955722314 shutdown_connections 2024-03-21T10:48:40.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.474+0000 7f147f400640 1 -- 172.21.15.31:0/3955722314 wait complete. 2024-03-21T10:48:40.487 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:48:40.488 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-21T10:48:40.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.702+0000 7fe337800640 1 -- 172.21.15.31:0/914383274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe330104220 msgr2=0x7fe330110730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:40.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.702+0000 7fe337800640 1 --2- 172.21.15.31:0/914383274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe330104220 0x7fe330110730 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fe328009ca0 tx=0x7fe32802c100 comp rx=0 tx=0).stop 2024-03-21T10:48:40.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.702+0000 7fe337800640 1 -- 172.21.15.31:0/914383274 shutdown_connections 2024-03-21T10:48:40.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.702+0000 7fe337800640 1 --2- 172.21.15.31:0/914383274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe330104220 0x7fe330110730 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.702+0000 7fe337800640 1 --2- 172.21.15.31:0/914383274 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe330103860 0x7fe330103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.702+0000 7fe337800640 1 --2- 172.21.15.31:0/914383274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe330109870 0x7fe330109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.702+0000 7fe337800640 1 -- 172.21.15.31:0/914383274 >> 172.21.15.31:0/914383274 conn(0x7fe3300ff310 msgr2=0x7fe330101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:40.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.702+0000 7fe337800640 1 -- 172.21.15.31:0/914383274 shutdown_connections 2024-03-21T10:48:40.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.702+0000 7fe337800640 1 -- 172.21.15.31:0/914383274 wait complete. 2024-03-21T10:48:40.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.703+0000 7fe337800640 1 Processor -- start 2024-03-21T10:48:40.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.703+0000 7fe337800640 1 -- start start 2024-03-21T10:48:40.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.704+0000 7fe337800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe330103860 0x7fe33019a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.704+0000 7fe337800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe330104220 0x7fe33019a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.704+0000 7fe337800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe330109870 0x7fe33019e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.704+0000 7fe337800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe330113c40 con 0x7fe330103860 2024-03-21T10:48:40.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.704+0000 7fe337800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe330113ac0 con 0x7fe330109870 2024-03-21T10:48:40.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.704+0000 7fe337800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe330113dc0 con 0x7fe330104220 2024-03-21T10:48:40.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.704+0000 7fe334e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe330103860 0x7fe33019a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.704+0000 7fe32fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe330104220 0x7fe33019a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.704+0000 7fe32fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe330104220 0x7fe33019a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:39158/0 (socket says 172.21.15.31:39158) 2024-03-21T10:48:40.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.704+0000 7fe32fe00640 1 -- 172.21.15.31:0/2300560793 learned_addr learned my addr 172.21.15.31:0/2300560793 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:40.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.704+0000 7fe335800640 1 --2- 172.21.15.31:0/2300560793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe330109870 0x7fe33019e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.705+0000 7fe32fe00640 1 -- 172.21.15.31:0/2300560793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe330109870 msgr2=0x7fe33019e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:40.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.705+0000 7fe32fe00640 1 --2- 172.21.15.31:0/2300560793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe330109870 0x7fe33019e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.705+0000 7fe32fe00640 1 -- 172.21.15.31:0/2300560793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe330103860 msgr2=0x7fe33019a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:40.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.705+0000 7fe32fe00640 1 --2- 172.21.15.31:0/2300560793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe330103860 0x7fe33019a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.705+0000 7fe32fe00640 1 -- 172.21.15.31:0/2300560793 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33019f0d0 con 0x7fe330104220 2024-03-21T10:48:40.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.705+0000 7fe335800640 1 --2- 172.21.15.31:0/2300560793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe330109870 0x7fe33019e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:40.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.705+0000 7fe32fe00640 1 --2- 172.21.15.31:0/2300560793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe330104220 0x7fe33019a620 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fe31c00ea60 tx=0x7fe31c00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:40.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.705+0000 7fe32d600640 1 -- 172.21.15.31:0/2300560793 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31c00cde0 con 0x7fe330104220 2024-03-21T10:48:40.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.705+0000 7fe32d600640 1 -- 172.21.15.31:0/2300560793 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe31c0035f0 con 0x7fe330104220 2024-03-21T10:48:40.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.705+0000 7fe337800640 1 -- 172.21.15.31:0/2300560793 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe33019f3c0 con 0x7fe330104220 2024-03-21T10:48:40.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.705+0000 7fe32d600640 1 -- 172.21.15.31:0/2300560793 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31c010670 con 0x7fe330104220 2024-03-21T10:48:40.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.705+0000 7fe337800640 1 -- 172.21.15.31:0/2300560793 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3301a6c50 con 0x7fe330104220 2024-03-21T10:48:40.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.709+0000 7fe33a5e4740 1 Processor -- start 2024-03-21T10:48:40.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.709+0000 7fe33a5e4740 1 -- start start 2024-03-21T10:48:40.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.709+0000 7fe33a5e4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d3614f6d0 0x558d3614fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.709+0000 7fe33a5e4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d36150080 0x558d36158630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.709+0000 7fe33a5e4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558d36158c00 0x558d3615b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.709+0000 7fe33a5e4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d360b32c0 con 0x558d36150080 2024-03-21T10:48:40.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.709+0000 7fe33a5e4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d3605bc90 con 0x558d3614f6d0 2024-03-21T10:48:40.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.709+0000 7fe33a5e4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d360af360 con 0x558d36158c00 2024-03-21T10:48:40.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.709+0000 7fe335800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d3614f6d0 0x558d3614fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.710+0000 7fe335800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d3614f6d0 0x558d3614fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:47968/0 (socket says 172.21.15.31:47968) 2024-03-21T10:48:40.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.710+0000 7fe335800640 1 -- 172.21.15.31:0/417118323 learned_addr learned my addr 172.21.15.31:0/417118323 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:40.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.710+0000 7fe334e00640 1 --2- 172.21.15.31:0/417118323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d36150080 0x558d36158630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.710+0000 7fe335800640 1 -- 172.21.15.31:0/417118323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558d36158c00 msgr2=0x558d3615b050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:48:40.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.710+0000 7fe335800640 1 --2- 172.21.15.31:0/417118323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558d36158c00 0x558d3615b050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.710+0000 7fe335800640 1 -- 172.21.15.31:0/417118323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d36150080 msgr2=0x558d36158630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:40.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.710+0000 7fe335800640 1 --2- 172.21.15.31:0/417118323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d36150080 0x558d36158630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.710+0000 7fe335800640 1 -- 172.21.15.31:0/417118323 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558d36080080 con 0x558d3614f6d0 2024-03-21T10:48:40.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.710+0000 7fe32d600640 1 -- 172.21.15.31:0/2300560793 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fe31c0037a0 con 0x7fe330104220 2024-03-21T10:48:40.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.710+0000 7fe335800640 1 --2- 172.21.15.31:0/417118323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d3614f6d0 0x558d3614fab0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe328009ca0 tx=0x7fe328009900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b736b62ff50e02ff server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:40.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.711+0000 7fe322000640 1 -- 172.21.15.31:0/417118323 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe328045020 con 0x558d3614f6d0 2024-03-21T10:48:40.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.711+0000 7fe322000640 1 -- 172.21.15.31:0/417118323 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe32803a040 con 0x558d3614f6d0 2024-03-21T10:48:40.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.711+0000 7fe322000640 1 -- 172.21.15.31:0/417118323 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe328039040 con 0x558d3614f6d0 2024-03-21T10:48:40.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.711+0000 7fe33a5e4740 1 -- 172.21.15.31:0/417118323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d3614f6d0 msgr2=0x558d3614fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:40.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.711+0000 7fe33a5e4740 1 --2- 172.21.15.31:0/417118323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d3614f6d0 0x558d3614fab0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe328009ca0 tx=0x7fe328009900 comp rx=0 tx=0).stop 2024-03-21T10:48:40.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.712+0000 7fe33a5e4740 1 -- 172.21.15.31:0/417118323 shutdown_connections 2024-03-21T10:48:40.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.712+0000 7fe33a5e4740 1 --2- 172.21.15.31:0/417118323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558d36158c00 0x558d3615b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.712+0000 7fe33a5e4740 1 --2- 172.21.15.31:0/417118323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d36150080 0x558d36158630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.712+0000 7fe33a5e4740 1 --2- 172.21.15.31:0/417118323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d3614f6d0 0x558d3614fab0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.712+0000 7fe33a5e4740 1 -- 172.21.15.31:0/417118323 >> 172.21.15.31:0/417118323 conn(0x558d36149da0 msgr2=0x558d3614a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:40.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.712+0000 7fe33a5e4740 1 -- 172.21.15.31:0/417118323 shutdown_connections 2024-03-21T10:48:40.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.712+0000 7fe33a5e4740 1 -- 172.21.15.31:0/417118323 wait complete. 2024-03-21T10:48:40.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.712+0000 7fe32d600640 1 --2- 172.21.15.31:0/2300560793 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe3180a2fa0 0x7fe3180a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.712+0000 7fe33a5e4740 1 -- start start 2024-03-21T10:48:40.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.712+0000 7fe32d600640 1 -- 172.21.15.31:0/2300560793 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fe31c0cfe60 con 0x7fe330104220 2024-03-21T10:48:40.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.712+0000 7fe33a5e4740 1 Processor -- start 2024-03-21T10:48:40.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.713+0000 7fe335800640 1 --2- 172.21.15.31:0/2300560793 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe3180a2fa0 0x7fe3180a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.713+0000 7fe33a5e4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d3614f6d0 0x558d361585c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.713+0000 7fe33a5e4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558d36150080 0x558d361559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.713+0000 7fe33a5e4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d36158c00 0x558d361560f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.713+0000 7fe33a5e4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d3615bb10 con 0x558d3614f6d0 2024-03-21T10:48:40.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.713+0000 7fe33a5e4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d3606f0d0 con 0x558d36158c00 2024-03-21T10:48:40.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.713+0000 7fe33a5e4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d3615bc90 con 0x558d36150080 2024-03-21T10:48:40.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.713+0000 7fe33a5e4740 10 client.0 _ll_register_callbacks cb 0x558d36073e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:40.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.713+0000 7fe334e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d3614f6d0 0x558d361585c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.713+0000 7fe32fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558d36150080 0x558d361559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.713+0000 7fe334e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d3614f6d0 0x558d361585c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57418/0 (socket says 172.21.15.31:57418) 2024-03-21T10:48:40.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.714+0000 7fe334e00640 1 -- 172.21.15.31:0/45092335 learned_addr learned my addr 172.21.15.31:0/45092335 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:40.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.714+0000 7fe335800640 1 --2- 172.21.15.31:0/45092335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d36158c00 0x558d361560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.714+0000 7fe335800640 1 --2- 172.21.15.31:0/2300560793 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe3180a2fa0 0x7fe3180a5460 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe328034670 tx=0x7fe32802c100 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:40.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.714+0000 7fe32fe00640 1 -- 172.21.15.31:0/45092335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d36158c00 msgr2=0x558d361560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:40.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.714+0000 7fe32fe00640 1 --2- 172.21.15.31:0/45092335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d36158c00 0x558d361560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.714+0000 7fe32fe00640 1 -- 172.21.15.31:0/45092335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d3614f6d0 msgr2=0x558d361585c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:40.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.714+0000 7fe32fe00640 1 --2- 172.21.15.31:0/45092335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d3614f6d0 0x558d361585c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.714+0000 7fe32fe00640 1 -- 172.21.15.31:0/45092335 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558d35dc73f0 con 0x558d36150080 2024-03-21T10:48:40.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.714+0000 7fe334e00640 1 --2- 172.21.15.31:0/45092335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d3614f6d0 0x558d361585c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:40.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.714+0000 7fe32fe00640 1 --2- 172.21.15.31:0/45092335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558d36150080 0x558d361559b0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fe31c0cd210 tx=0x7fe31c09bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e186e5c33894ecd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:40.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.714+0000 7fe316a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:48:40.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.714+0000 7fe316a00640 1 -- 172.21.15.31:0/45092335 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31c0a0000 con 0x558d36150080 2024-03-21T10:48:40.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.715+0000 7fe316a00640 1 -- 172.21.15.31:0/45092335 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fe31c003b30 con 0x558d36150080 2024-03-21T10:48:40.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.715+0000 7fe33a5e4740 1 -- 172.21.15.31:0/45092335 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe32803a040 con 0x558d36150080 2024-03-21T10:48:40.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.715+0000 7fe316a00640 1 -- 172.21.15.31:0/45092335 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe31c0a48c0 con 0x558d36150080 2024-03-21T10:48:40.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.715+0000 7fe33a5e4740 10 client.28081 fetch_fsmap learned FSMap version 28 2024-03-21T10:48:40.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.715+0000 7fe33a5e4740 1 -- 172.21.15.31:0/45092335 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x558d35e917d0 con 0x558d36150080 2024-03-21T10:48:40.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.715+0000 7fe316a00640 1 -- 172.21.15.31:0/45092335 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe32803a040 con 0x558d36150080 2024-03-21T10:48:40.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.715+0000 7fe316a00640 1 -- 172.21.15.31:0/45092335 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe31c09b490 con 0x558d36150080 2024-03-21T10:48:40.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.715+0000 7fe33a5e4740 10 client.28081 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:48:40.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.715+0000 7fe33a5e4740 10 client.28081 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:48:40.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.715+0000 7fe33a5e4740 20 client.28081 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:40.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.715+0000 7fe33a5e4740 1 --2- 172.21.15.31:0/45092335 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x558d36168b10 0x558d3616af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.715+0000 7fe33a5e4740 4 client.28081 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:48:40.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.715+0000 7fe33a5e4740 1 -- 172.21.15.31:0/45092335 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558d360c0800 con 0x558d36168b10 2024-03-21T10:48:40.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.716+0000 7fe334e00640 1 --2- 172.21.15.31:0/45092335 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x558d36168b10 0x558d3616af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.716+0000 7fe334e00640 1 --2- 172.21.15.31:0/45092335 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x558d36168b10 0x558d3616af00 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8c22dfafda8e142a server_cookie=ac08a50ba6f7557a in_seq=0 out_seq=0 2024-03-21T10:48:40.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.716+0000 7fe316a00640 10 client.28081 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:48:40.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.717+0000 7fe316a00640 1 -- 172.21.15.31:0/45092335 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558d360c0800 con 0x558d36168b10 2024-03-21T10:48:40.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.717+0000 7fe316a00640 10 client.28081 handle_command_reply: tid=0 2024-03-21T10:48:40.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.717+0000 7fe33a5e4740 1 client.28081 shutdown 2024-03-21T10:48:40.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.717+0000 7fe33a5e4740 1 -- 172.21.15.31:0/45092335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558d36150080 msgr2=0x558d361559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:40.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.717+0000 7fe33a5e4740 1 --2- 172.21.15.31:0/45092335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558d36150080 0x558d361559b0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fe31c0cd210 tx=0x7fe31c09bab0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.717+0000 7fe33a5e4740 1 -- 172.21.15.31:0/45092335 shutdown_connections 2024-03-21T10:48:40.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.717+0000 7fe33a5e4740 1 --2- 172.21.15.31:0/45092335 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x558d36168b10 0x558d3616af00 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.717+0000 7fe33a5e4740 1 --2- 172.21.15.31:0/45092335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d36158c00 0x558d361560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.717+0000 7fe33a5e4740 1 --2- 172.21.15.31:0/45092335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558d36150080 0x558d361559b0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.717+0000 7fe33a5e4740 1 --2- 172.21.15.31:0/45092335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d3614f6d0 0x558d361585c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.717+0000 7fe316a00640 0 client.28081 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:48:40.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.717+0000 7fe33a5e4740 1 -- 172.21.15.31:0/45092335 >> 172.21.15.31:0/45092335 conn(0x558d36149da0 msgr2=0x558d3614baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:40.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.718+0000 7fe33a5e4740 1 -- 172.21.15.31:0/45092335 shutdown_connections 2024-03-21T10:48:40.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.718+0000 7fe33a5e4740 1 -- 172.21.15.31:0/45092335 wait complete. 2024-03-21T10:48:40.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.718+0000 7fe33a5e4740 20 client.28081 trim_cache size 0 max 16384 2024-03-21T10:48:40.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.731+0000 7fe33a5e4740 1 Processor -- start 2024-03-21T10:48:40.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.731+0000 7fe33a5e4740 1 -- start start 2024-03-21T10:48:40.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.732+0000 7fe33a5e4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d36149da0 0x558d36155270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.732+0000 7fe33a5e4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d3614f6d0 0x558d36152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.732+0000 7fe33a5e4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558d36150080 0x558d36153130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.732+0000 7fe33a5e4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3301b3000 con 0x558d3614f6d0 2024-03-21T10:48:40.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.732+0000 7fe33a5e4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3301b0b00 con 0x558d36149da0 2024-03-21T10:48:40.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.732+0000 7fe33a5e4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3301aebc0 con 0x558d36150080 2024-03-21T10:48:40.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.732+0000 7fe334e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d36149da0 0x558d36155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.732+0000 7fe32fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d3614f6d0 0x558d36152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.732+0000 7fe334e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d36149da0 0x558d36155270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:47988/0 (socket says 172.21.15.31:47988) 2024-03-21T10:48:40.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.732+0000 7fe334e00640 1 -- 172.21.15.31:0/3779889387 learned_addr learned my addr 172.21.15.31:0/3779889387 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:40.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.732+0000 7fe335800640 1 --2- 172.21.15.31:0/3779889387 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558d36150080 0x558d36153130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.732+0000 7fe334e00640 1 -- 172.21.15.31:0/3779889387 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558d36150080 msgr2=0x558d36153130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:40.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.732+0000 7fe334e00640 1 --2- 172.21.15.31:0/3779889387 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558d36150080 0x558d36153130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.732+0000 7fe334e00640 1 -- 172.21.15.31:0/3779889387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d3614f6d0 msgr2=0x558d36152bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:40.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.732+0000 7fe334e00640 1 --2- 172.21.15.31:0/3779889387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d3614f6d0 0x558d36152bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.732+0000 7fe334e00640 1 -- 172.21.15.31:0/3779889387 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558d3614c0a0 con 0x558d36149da0 2024-03-21T10:48:40.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.733+0000 7fe334e00640 1 --2- 172.21.15.31:0/3779889387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d36149da0 0x558d36155270 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x558d3616b680 tx=0x7fe32400b390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aca1c14f2cb00c1a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:40.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.733+0000 7fe321600640 1 -- 172.21.15.31:0/3779889387 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe324026070 con 0x558d36149da0 2024-03-21T10:48:40.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.733+0000 7fe321600640 1 -- 172.21.15.31:0/3779889387 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe324020c70 con 0x558d36149da0 2024-03-21T10:48:40.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.733+0000 7fe321600640 1 -- 172.21.15.31:0/3779889387 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3240117a0 con 0x558d36149da0 2024-03-21T10:48:40.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.733+0000 7fe33a5e4740 1 -- 172.21.15.31:0/3779889387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d36149da0 msgr2=0x558d36155270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:40.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.733+0000 7fe33a5e4740 1 --2- 172.21.15.31:0/3779889387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d36149da0 0x558d36155270 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x558d3616b680 tx=0x7fe32400b390 comp rx=0 tx=0).stop 2024-03-21T10:48:40.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.733+0000 7fe33a5e4740 1 -- 172.21.15.31:0/3779889387 shutdown_connections 2024-03-21T10:48:40.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.734+0000 7fe33a5e4740 1 --2- 172.21.15.31:0/3779889387 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558d36150080 0x558d36153130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.734+0000 7fe33a5e4740 1 --2- 172.21.15.31:0/3779889387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d3614f6d0 0x558d36152bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.734+0000 7fe33a5e4740 1 --2- 172.21.15.31:0/3779889387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d36149da0 0x558d36155270 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.734+0000 7fe33a5e4740 1 -- 172.21.15.31:0/3779889387 >> 172.21.15.31:0/3779889387 conn(0x558d36158720 msgr2=0x558d36157940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:40.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.734+0000 7fe33a5e4740 1 -- 172.21.15.31:0/3779889387 shutdown_connections 2024-03-21T10:48:40.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.734+0000 7fe33a5e4740 1 -- 172.21.15.31:0/3779889387 wait complete. 2024-03-21T10:48:40.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.734+0000 7fe33a5e4740 1 -- start start 2024-03-21T10:48:40.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.734+0000 7fe33a5e4740 1 Processor -- start 2024-03-21T10:48:40.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.735+0000 7fe33a5e4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558d36149da0 0x558d3615ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.735+0000 7fe33a5e4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d3614f6d0 0x558d3615efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.735+0000 7fe33a5e4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d36150080 0x558d361c32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.735+0000 7fe33a5e4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3301af1c0 con 0x558d3614f6d0 2024-03-21T10:48:40.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.735+0000 7fe33a5e4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3301b04a0 con 0x558d36150080 2024-03-21T10:48:40.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.735+0000 7fe33a5e4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3301af7e0 con 0x558d36149da0 2024-03-21T10:48:40.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.735+0000 7fe33a5e4740 10 client.0 _ll_register_callbacks cb 0x7fe31c003e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:40.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.735+0000 7fe334e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558d36149da0 0x558d3615ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.735+0000 7fe335800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d36150080 0x558d361c32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.735+0000 7fe334e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558d36149da0 0x558d3615ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:39194/0 (socket says 172.21.15.31:39194) 2024-03-21T10:48:40.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.735+0000 7fe334e00640 1 -- 172.21.15.31:0/3847521484 learned_addr learned my addr 172.21.15.31:0/3847521484 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:40.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.735+0000 7fe32fe00640 1 --2- 172.21.15.31:0/3847521484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d3614f6d0 0x558d3615efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.735+0000 7fe334e00640 1 -- 172.21.15.31:0/3847521484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d36150080 msgr2=0x558d361c32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:40.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.735+0000 7fe334e00640 1 --2- 172.21.15.31:0/3847521484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d36150080 0x558d361c32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.735+0000 7fe334e00640 1 -- 172.21.15.31:0/3847521484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d3614f6d0 msgr2=0x558d3615efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:40.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.735+0000 7fe334e00640 1 --2- 172.21.15.31:0/3847521484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d3614f6d0 0x558d3615efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.735+0000 7fe334e00640 1 -- 172.21.15.31:0/3847521484 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558d36156460 con 0x558d36149da0 2024-03-21T10:48:40.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.735+0000 7fe334e00640 1 --2- 172.21.15.31:0/3847521484 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558d36149da0 0x558d3615ea70 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fe324024040 tx=0x7fe3240044f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=beba5016f8d9bc7a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:40.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.735+0000 7fe316a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:48:40.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.736+0000 7fe316a00640 1 -- 172.21.15.31:0/3847521484 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe324026040 con 0x558d36149da0 2024-03-21T10:48:40.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.736+0000 7fe316a00640 1 -- 172.21.15.31:0/3847521484 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fe324005040 con 0x558d36149da0 2024-03-21T10:48:40.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.736+0000 7fe33a5e4740 1 -- 172.21.15.31:0/3847521484 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe324020c70 con 0x558d36149da0 2024-03-21T10:48:40.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.736+0000 7fe316a00640 1 -- 172.21.15.31:0/3847521484 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe324035050 con 0x558d36149da0 2024-03-21T10:48:40.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.736+0000 7fe33a5e4740 10 client.28087 fetch_fsmap learned FSMap version 28 2024-03-21T10:48:40.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.736+0000 7fe33a5e4740 1 -- 172.21.15.31:0/3847521484 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x558d361c4920 con 0x558d36149da0 2024-03-21T10:48:40.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.736+0000 7fe316a00640 1 -- 172.21.15.31:0/3847521484 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe324020c70 con 0x558d36149da0 2024-03-21T10:48:40.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.736+0000 7fe316a00640 1 -- 172.21.15.31:0/3847521484 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe324003710 con 0x558d36149da0 2024-03-21T10:48:40.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.736+0000 7fe33a5e4740 10 client.28087 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:48:40.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.737+0000 7fe33a5e4740 10 client.28087 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:48:40.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.737+0000 7fe33a5e4740 20 client.28087 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:40.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.737+0000 7fe33a5e4740 1 --2- 172.21.15.31:0/3847521484 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x558d36158c00 0x558d361c6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.737+0000 7fe33a5e4740 4 client.28087 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:48:40.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.737+0000 7fe33a5e4740 1 -- 172.21.15.31:0/3847521484 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558d360bab50 con 0x558d36158c00 2024-03-21T10:48:40.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.737+0000 7fe32fe00640 1 --2- 172.21.15.31:0/3847521484 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x558d36158c00 0x558d361c6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.737+0000 7fe32fe00640 1 --2- 172.21.15.31:0/3847521484 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x558d36158c00 0x558d361c6fa0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d324d7ae63080d03 server_cookie=f24c2c9404d86974 in_seq=0 out_seq=0 2024-03-21T10:48:40.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.737+0000 7fe316a00640 10 client.28087 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:48:40.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.738+0000 7fe316a00640 1 -- 172.21.15.31:0/3847521484 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558d360bab50 con 0x558d36158c00 2024-03-21T10:48:40.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.738+0000 7fe316a00640 10 client.28087 handle_command_reply: tid=0 2024-03-21T10:48:40.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.738+0000 7fe33a5e4740 1 client.28087 shutdown 2024-03-21T10:48:40.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.738+0000 7fe33a5e4740 1 -- 172.21.15.31:0/3847521484 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558d36149da0 msgr2=0x558d3615ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:40.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.738+0000 7fe33a5e4740 1 --2- 172.21.15.31:0/3847521484 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558d36149da0 0x558d3615ea70 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fe324024040 tx=0x7fe3240044f0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.738+0000 7fe33a5e4740 1 -- 172.21.15.31:0/3847521484 shutdown_connections 2024-03-21T10:48:40.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.738+0000 7fe33a5e4740 1 --2- 172.21.15.31:0/3847521484 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x558d36158c00 0x558d361c6fa0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.738+0000 7fe33a5e4740 1 --2- 172.21.15.31:0/3847521484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d36150080 0x558d361c32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.738+0000 7fe33a5e4740 1 --2- 172.21.15.31:0/3847521484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d3614f6d0 0x558d3615efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.738+0000 7fe316a00640 0 client.28087 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:48:40.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.738+0000 7fe33a5e4740 1 --2- 172.21.15.31:0/3847521484 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558d36149da0 0x558d3615ea70 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.738+0000 7fe33a5e4740 1 -- 172.21.15.31:0/3847521484 >> 172.21.15.31:0/3847521484 conn(0x558d36158720 msgr2=0x558d36157040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:40.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.738+0000 7fe33a5e4740 1 -- 172.21.15.31:0/3847521484 shutdown_connections 2024-03-21T10:48:40.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.738+0000 7fe33a5e4740 1 -- 172.21.15.31:0/3847521484 wait complete. 2024-03-21T10:48:40.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.738+0000 7fe33a5e4740 20 client.28087 trim_cache size 0 max 16384 2024-03-21T10:48:40.739 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:48:40.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.739+0000 7fe337800640 1 -- 172.21.15.31:0/2300560793 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe3180a2fa0 msgr2=0x7fe3180a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:40.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.739+0000 7fe337800640 1 --2- 172.21.15.31:0/2300560793 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe3180a2fa0 0x7fe3180a5460 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe328034670 tx=0x7fe32802c100 comp rx=0 tx=0).stop 2024-03-21T10:48:40.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.739+0000 7fe337800640 1 -- 172.21.15.31:0/2300560793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe330104220 msgr2=0x7fe33019a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:40.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.739+0000 7fe337800640 1 --2- 172.21.15.31:0/2300560793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe330104220 0x7fe33019a620 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fe31c00ea60 tx=0x7fe31c00ef20 comp rx=0 tx=0).stop 2024-03-21T10:48:40.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.739+0000 7fe337800640 1 -- 172.21.15.31:0/2300560793 shutdown_connections 2024-03-21T10:48:40.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.739+0000 7fe337800640 1 --2- 172.21.15.31:0/2300560793 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe3180a2fa0 0x7fe3180a5460 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.739+0000 7fe337800640 1 --2- 172.21.15.31:0/2300560793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe330109870 0x7fe33019e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.739+0000 7fe337800640 1 --2- 172.21.15.31:0/2300560793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe330104220 0x7fe33019a620 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.739+0000 7fe337800640 1 --2- 172.21.15.31:0/2300560793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe330103860 0x7fe33019a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.739+0000 7fe337800640 1 -- 172.21.15.31:0/2300560793 >> 172.21.15.31:0/2300560793 conn(0x7fe3300ff310 msgr2=0x7fe3300ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:40.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.739+0000 7fe337800640 1 -- 172.21.15.31:0/2300560793 shutdown_connections 2024-03-21T10:48:40.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.739+0000 7fe337800640 1 -- 172.21.15.31:0/2300560793 wait complete. 2024-03-21T10:48:40.754 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:48:40.754 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-21T10:48:40.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.973+0000 7f8f13200640 1 -- 172.21.15.31:0/3522848051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f14103660 msgr2=0x7f8f14103ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:40.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.973+0000 7f8f13200640 1 --2- 172.21.15.31:0/3522848051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f14103660 0x7f8f14103ae0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8f00009900 tx=0x7f8f0002bc40 comp rx=0 tx=0).stop 2024-03-21T10:48:40.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.974+0000 7f8f13200640 1 -- 172.21.15.31:0/3522848051 shutdown_connections 2024-03-21T10:48:40.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.974+0000 7f8f13200640 1 --2- 172.21.15.31:0/3522848051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f14104020 0x7f8f14110530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.974+0000 7f8f13200640 1 --2- 172.21.15.31:0/3522848051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f14103660 0x7f8f14103ae0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.974+0000 7f8f13200640 1 --2- 172.21.15.31:0/3522848051 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f14109670 0x7f8f14109a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.974+0000 7f8f13200640 1 -- 172.21.15.31:0/3522848051 >> 172.21.15.31:0/3522848051 conn(0x7f8f140ff110 msgr2=0x7f8f14101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:40.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.974+0000 7f8f13200640 1 -- 172.21.15.31:0/3522848051 shutdown_connections 2024-03-21T10:48:40.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.974+0000 7f8f13200640 1 -- 172.21.15.31:0/3522848051 wait complete. 2024-03-21T10:48:40.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.975+0000 7f8f13200640 1 Processor -- start 2024-03-21T10:48:40.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.975+0000 7f8f13200640 1 -- start start 2024-03-21T10:48:40.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.976+0000 7f8f13200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f14103660 0x7f8f14199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.976+0000 7f8f13200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f14104020 0x7f8f1419a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.976+0000 7f8f13200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f14109670 0x7f8f1419e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.976+0000 7f8f13200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f14113a80 con 0x7f8f14103660 2024-03-21T10:48:40.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.976+0000 7f8f13200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f14113900 con 0x7f8f14104020 2024-03-21T10:48:40.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.976+0000 7f8f13200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f14113c00 con 0x7f8f14109670 2024-03-21T10:48:40.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.976+0000 7f8f11400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f14104020 0x7f8f1419a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.976+0000 7f8f11400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f14104020 0x7f8f1419a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:48010/0 (socket says 172.21.15.31:48010) 2024-03-21T10:48:40.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.976+0000 7f8f11400640 1 -- 172.21.15.31:0/3286325777 learned_addr learned my addr 172.21.15.31:0/3286325777 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:40.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.976+0000 7f8f12800640 1 --2- 172.21.15.31:0/3286325777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f14109670 0x7f8f1419e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.976+0000 7f8f11e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f14103660 0x7f8f14199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.976+0000 7f8f11400640 1 -- 172.21.15.31:0/3286325777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f14109670 msgr2=0x7f8f1419e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:40.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.976+0000 7f8f11400640 1 --2- 172.21.15.31:0/3286325777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f14109670 0x7f8f1419e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.976+0000 7f8f11400640 1 -- 172.21.15.31:0/3286325777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f14103660 msgr2=0x7f8f14199f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:40.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.976+0000 7f8f11400640 1 --2- 172.21.15.31:0/3286325777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f14103660 0x7f8f14199f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.976+0000 7f8f11400640 1 -- 172.21.15.31:0/3286325777 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1419ef70 con 0x7f8f14104020 2024-03-21T10:48:40.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.977+0000 7f8f11400640 1 --2- 172.21.15.31:0/3286325777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f14104020 0x7f8f1419a4c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f8f00006010 tx=0x7f8f00004430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:40.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.977+0000 7f8f0aa00640 1 -- 172.21.15.31:0/3286325777 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f00041070 con 0x7f8f14104020 2024-03-21T10:48:40.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.977+0000 7f8f0aa00640 1 -- 172.21.15.31:0/3286325777 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f000054e0 con 0x7f8f14104020 2024-03-21T10:48:40.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.977+0000 7f8f13200640 1 -- 172.21.15.31:0/3286325777 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f1419f200 con 0x7f8f14104020 2024-03-21T10:48:40.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.977+0000 7f8f0aa00640 1 -- 172.21.15.31:0/3286325777 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f00003e20 con 0x7f8f14104020 2024-03-21T10:48:40.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.977+0000 7f8f13200640 1 -- 172.21.15.31:0/3286325777 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f141a6af0 con 0x7f8f14104020 2024-03-21T10:48:40.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.980+0000 7f8f0aa00640 1 -- 172.21.15.31:0/3286325777 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f8f00044070 con 0x7f8f14104020 2024-03-21T10:48:40.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.981+0000 7f8f1bd6c740 1 Processor -- start 2024-03-21T10:48:40.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.981+0000 7f8f1bd6c740 1 -- start start 2024-03-21T10:48:40.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.982+0000 7f8f1bd6c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c3094a8b0 0x564c3094ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.982+0000 7f8f1bd6c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c3094b260 0x564c30953810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.982+0000 7f8f1bd6c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c30953de0 0x564c30956230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.982+0000 7f8f1bd6c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c308ae2c0 con 0x564c3094b260 2024-03-21T10:48:40.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.982+0000 7f8f1bd6c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c30856c90 con 0x564c3094a8b0 2024-03-21T10:48:40.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.982+0000 7f8f1bd6c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c308aa360 con 0x564c30953de0 2024-03-21T10:48:40.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.982+0000 7f8f12800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c3094a8b0 0x564c3094ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.982+0000 7f8f11400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c30953de0 0x564c30956230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.982+0000 7f8f12800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c3094a8b0 0x564c3094ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:48026/0 (socket says 172.21.15.31:48026) 2024-03-21T10:48:40.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.982+0000 7f8f12800640 1 -- 172.21.15.31:0/1274041919 learned_addr learned my addr 172.21.15.31:0/1274041919 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:40.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.982+0000 7f8f11e00640 1 --2- 172.21.15.31:0/1274041919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c3094b260 0x564c30953810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.983+0000 7f8f12800640 1 -- 172.21.15.31:0/1274041919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c30953de0 msgr2=0x564c30956230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:40.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.983+0000 7f8f12800640 1 --2- 172.21.15.31:0/1274041919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c30953de0 0x564c30956230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.983+0000 7f8f12800640 1 -- 172.21.15.31:0/1274041919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c3094b260 msgr2=0x564c30953810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:40.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.983+0000 7f8f12800640 1 --2- 172.21.15.31:0/1274041919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c3094b260 0x564c30953810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.983+0000 7f8f12800640 1 -- 172.21.15.31:0/1274041919 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c3087b080 con 0x564c3094a8b0 2024-03-21T10:48:40.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.983+0000 7f8f0aa00640 1 --2- 172.21.15.31:0/3286325777 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f8ef80a3090 0x7f8ef80a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.983+0000 7f8f0aa00640 1 -- 172.21.15.31:0/3286325777 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f8f000f1610 con 0x7f8f14104020 2024-03-21T10:48:40.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.983+0000 7f8f12800640 1 --2- 172.21.15.31:0/1274041919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c3094a8b0 0x564c3094ac90 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c010370 tx=0x7f8f0c010830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c234dc152fe9b4c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:40.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.983+0000 7f8f11e00640 1 --2- 172.21.15.31:0/3286325777 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f8ef80a3090 0x7f8ef80a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.983+0000 7f8eff400640 1 -- 172.21.15.31:0/1274041919 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c016070 con 0x564c3094a8b0 2024-03-21T10:48:40.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.983+0000 7f8eff400640 1 -- 172.21.15.31:0/1274041919 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f0c004d20 con 0x564c3094a8b0 2024-03-21T10:48:40.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.983+0000 7f8eff400640 1 -- 172.21.15.31:0/1274041919 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c005480 con 0x564c3094a8b0 2024-03-21T10:48:40.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.984+0000 7f8f1bd6c740 1 -- 172.21.15.31:0/1274041919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c3094a8b0 msgr2=0x564c3094ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:40.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.984+0000 7f8f11e00640 1 --2- 172.21.15.31:0/3286325777 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f8ef80a3090 0x7f8ef80a5550 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x564c30945ff0 tx=0x7f8f0400b410 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:40.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.984+0000 7f8f1bd6c740 1 --2- 172.21.15.31:0/1274041919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c3094a8b0 0x564c3094ac90 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c010370 tx=0x7f8f0c010830 comp rx=0 tx=0).stop 2024-03-21T10:48:40.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.984+0000 7f8f1bd6c740 1 -- 172.21.15.31:0/1274041919 shutdown_connections 2024-03-21T10:48:40.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.984+0000 7f8f1bd6c740 1 --2- 172.21.15.31:0/1274041919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c30953de0 0x564c30956230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.984+0000 7f8f1bd6c740 1 --2- 172.21.15.31:0/1274041919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c3094b260 0x564c30953810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.984+0000 7f8f1bd6c740 1 --2- 172.21.15.31:0/1274041919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c3094a8b0 0x564c3094ac90 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.984+0000 7f8f1bd6c740 1 -- 172.21.15.31:0/1274041919 >> 172.21.15.31:0/1274041919 conn(0x564c30944f80 msgr2=0x564c30945360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:40.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.984+0000 7f8f1bd6c740 1 -- 172.21.15.31:0/1274041919 shutdown_connections 2024-03-21T10:48:40.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.984+0000 7f8f1bd6c740 1 -- 172.21.15.31:0/1274041919 wait complete. 2024-03-21T10:48:40.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.985+0000 7f8f1bd6c740 1 -- start start 2024-03-21T10:48:40.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.985+0000 7f8f1bd6c740 1 Processor -- start 2024-03-21T10:48:40.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.986+0000 7f8f1bd6c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c3094a8b0 0x564c30953770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.986+0000 7f8f1bd6c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c3094b260 0x564c30950b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.986+0000 7f8f1bd6c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c30953de0 0x564c30951110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.986+0000 7f8f1bd6c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c30956d50 con 0x564c30953de0 2024-03-21T10:48:40.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.986+0000 7f8f1bd6c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c3086a0d0 con 0x564c3094b260 2024-03-21T10:48:40.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.986+0000 7f8f1bd6c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c30956ed0 con 0x564c3094a8b0 2024-03-21T10:48:40.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.986+0000 7f8f1bd6c740 10 client.0 _ll_register_callbacks cb 0x564c3086ee80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:40.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.986+0000 7f8f11e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c3094a8b0 0x564c30953770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.986+0000 7f8f11e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c3094a8b0 0x564c30953770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:39216/0 (socket says 172.21.15.31:39216) 2024-03-21T10:48:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.986+0000 7f8f11e00640 1 -- 172.21.15.31:0/921608637 learned_addr learned my addr 172.21.15.31:0/921608637 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.986+0000 7f8f11400640 1 --2- 172.21.15.31:0/921608637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c3094b260 0x564c30950b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.986+0000 7f8f12800640 1 --2- 172.21.15.31:0/921608637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c30953de0 0x564c30951110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.986+0000 7f8f11e00640 1 -- 172.21.15.31:0/921608637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c3094b260 msgr2=0x564c30950b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.986+0000 7f8f11e00640 1 --2- 172.21.15.31:0/921608637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c3094b260 0x564c30950b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.986+0000 7f8f11e00640 1 -- 172.21.15.31:0/921608637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c30953de0 msgr2=0x564c30951110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.986+0000 7f8f11e00640 1 --2- 172.21.15.31:0/921608637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c30953de0 0x564c30951110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.986+0000 7f8f11e00640 1 -- 172.21.15.31:0/921608637 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564c305c23f0 con 0x564c3094a8b0 2024-03-21T10:48:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.986+0000 7f8f11400640 1 --2- 172.21.15.31:0/921608637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c3094b260 0x564c30950b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.987+0000 7f8f11e00640 1 --2- 172.21.15.31:0/921608637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c3094a8b0 0x564c30953770 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8f0401f430 tx=0x7f8f04004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=877405378639cf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.987+0000 7f8ef3e00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:48:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.987+0000 7f8ef3e00640 1 -- 172.21.15.31:0/921608637 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0402c060 con 0x564c3094a8b0 2024-03-21T10:48:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.987+0000 7f8f1bd6c740 1 -- 172.21.15.31:0/921608637 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8f0c004d20 con 0x564c3094a8b0 2024-03-21T10:48:40.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.987+0000 7f8ef3e00640 1 -- 172.21.15.31:0/921608637 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f8f04017030 con 0x564c3094a8b0 2024-03-21T10:48:40.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.987+0000 7f8ef3e00640 1 -- 172.21.15.31:0/921608637 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f04033050 con 0x564c3094a8b0 2024-03-21T10:48:40.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.988+0000 7f8f1bd6c740 10 client.28099 fetch_fsmap learned FSMap version 28 2024-03-21T10:48:40.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.988+0000 7f8f1bd6c740 1 -- 172.21.15.31:0/921608637 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x564c3068c7d0 con 0x564c3094a8b0 2024-03-21T10:48:40.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.988+0000 7f8ef3e00640 1 -- 172.21.15.31:0/921608637 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f0c004d20 con 0x564c3094a8b0 2024-03-21T10:48:40.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.988+0000 7f8ef3e00640 1 -- 172.21.15.31:0/921608637 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f8f04010700 con 0x564c3094a8b0 2024-03-21T10:48:40.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.988+0000 7f8f1bd6c740 10 client.28099 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:48:40.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.988+0000 7f8f1bd6c740 10 client.28099 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:48:40.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.988+0000 7f8f1bd6c740 20 client.28099 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:40.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.988+0000 7f8f1bd6c740 1 --2- 172.21.15.31:0/921608637 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564c30963d10 0x564c30966120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:40.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.988+0000 7f8f1bd6c740 4 client.28099 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:48:40.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.988+0000 7f8f1bd6c740 1 -- 172.21.15.31:0/921608637 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564c308bb800 con 0x564c30963d10 2024-03-21T10:48:40.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.989+0000 7f8f11400640 1 --2- 172.21.15.31:0/921608637 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564c30963d10 0x564c30966120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:40.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.989+0000 7f8f11400640 1 --2- 172.21.15.31:0/921608637 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564c30963d10 0x564c30966120 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=107a1391bb4d6f1c server_cookie=d89a1e2379fc4f6e in_seq=0 out_seq=0 2024-03-21T10:48:40.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.989+0000 7f8ef3e00640 10 client.28099 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:40.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.990+0000 7f8ef3e00640 1 -- 172.21.15.31:0/921608637 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564c308bb800 con 0x564c30963d10 2024-03-21T10:48:40.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.990+0000 7f8ef3e00640 10 client.28099 handle_command_reply: tid=0 2024-03-21T10:48:40.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.990+0000 7f8f1bd6c740 1 client.28099 shutdown 2024-03-21T10:48:40.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.991+0000 7f8f1bd6c740 1 -- 172.21.15.31:0/921608637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c3094a8b0 msgr2=0x564c30953770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:40.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.991+0000 7f8f1bd6c740 1 --2- 172.21.15.31:0/921608637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c3094a8b0 0x564c30953770 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8f0401f430 tx=0x7f8f04004730 comp rx=0 tx=0).stop 2024-03-21T10:48:40.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.991+0000 7f8f1bd6c740 1 -- 172.21.15.31:0/921608637 shutdown_connections 2024-03-21T10:48:40.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.991+0000 7f8f1bd6c740 1 --2- 172.21.15.31:0/921608637 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564c30963d10 0x564c30966120 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.991+0000 7f8f1bd6c740 1 --2- 172.21.15.31:0/921608637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c30953de0 0x564c30951110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.991+0000 7f8f1bd6c740 1 --2- 172.21.15.31:0/921608637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c3094b260 0x564c30950b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.991+0000 7f8ef3e00640 0 client.28099 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:40.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.991+0000 7f8f1bd6c740 1 --2- 172.21.15.31:0/921608637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c3094a8b0 0x564c30953770 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:40.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.991+0000 7f8f1bd6c740 1 -- 172.21.15.31:0/921608637 >> 172.21.15.31:0/921608637 conn(0x564c30944f80 msgr2=0x564c30946cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:40.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.991+0000 7f8f1bd6c740 1 -- 172.21.15.31:0/921608637 shutdown_connections 2024-03-21T10:48:40.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.991+0000 7f8f1bd6c740 1 -- 172.21.15.31:0/921608637 wait complete. 2024-03-21T10:48:40.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:40.991+0000 7f8f1bd6c740 20 client.28099 trim_cache size 0 max 16384 2024-03-21T10:48:41.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.007+0000 7f8f1bd6c740 1 Processor -- start 2024-03-21T10:48:41.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.007+0000 7f8f1bd6c740 1 -- start start 2024-03-21T10:48:41.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.007+0000 7f8f1bd6c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c3094a8b0 0x564c309400a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:41.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.007+0000 7f8f1bd6c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c3094b260 0x564c30953230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:41.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.007+0000 7f8f1bd6c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c30953de0 0x564c30953770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:41.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.007+0000 7f8f1bd6c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f141b2f80 con 0x564c3094a8b0 2024-03-21T10:48:41.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.007+0000 7f8f1bd6c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f141b0a80 con 0x564c3094b260 2024-03-21T10:48:41.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.007+0000 7f8f1bd6c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f141aeb40 con 0x564c30953de0 2024-03-21T10:48:41.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.007+0000 7f8f12800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c30953de0 0x564c30953770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:41.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.007+0000 7f8f12800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c30953de0 0x564c30953770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:39218/0 (socket says 172.21.15.31:39218) 2024-03-21T10:48:41.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.007+0000 7f8f12800640 1 -- 172.21.15.31:0/34447030 learned_addr learned my addr 172.21.15.31:0/34447030 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:41.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.008+0000 7f8f12800640 1 -- 172.21.15.31:0/34447030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c3094b260 msgr2=0x564c30953230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:41.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.008+0000 7f8f12800640 1 --2- 172.21.15.31:0/34447030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c3094b260 0x564c30953230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:41.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.008+0000 7f8f11e00640 1 --2- 172.21.15.31:0/34447030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c3094a8b0 0x564c309400a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:41.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.008+0000 7f8f12800640 1 -- 172.21.15.31:0/34447030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c3094a8b0 msgr2=0x564c309400a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:41.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.008+0000 7f8f12800640 1 --2- 172.21.15.31:0/34447030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c3094a8b0 0x564c309400a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:41.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.008+0000 7f8f12800640 1 -- 172.21.15.31:0/34447030 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c30956600 con 0x564c30953de0 2024-03-21T10:48:41.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.008+0000 7f8f11e00640 1 --2- 172.21.15.31:0/34447030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c3094a8b0 0x564c309400a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:48:41.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.008+0000 7f8f12800640 1 --2- 172.21.15.31:0/34447030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c30953de0 0x564c30953770 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c010370 tx=0x7f8f0c005c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4d8dcb7bfa5c27a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:41.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.009+0000 7f8efea00640 1 -- 172.21.15.31:0/34447030 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c016040 con 0x564c30953de0 2024-03-21T10:48:41.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.009+0000 7f8efea00640 1 -- 172.21.15.31:0/34447030 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f0c003b70 con 0x564c30953de0 2024-03-21T10:48:41.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.009+0000 7f8efea00640 1 -- 172.21.15.31:0/34447030 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c012770 con 0x564c30953de0 2024-03-21T10:48:41.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.009+0000 7f8f1bd6c740 1 -- 172.21.15.31:0/34447030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c30953de0 msgr2=0x564c30953770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:41.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.009+0000 7f8f1bd6c740 1 --2- 172.21.15.31:0/34447030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c30953de0 0x564c30953770 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c010370 tx=0x7f8f0c005c30 comp rx=0 tx=0).stop 2024-03-21T10:48:41.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.010+0000 7f8f1bd6c740 1 -- 172.21.15.31:0/34447030 shutdown_connections 2024-03-21T10:48:41.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.010+0000 7f8f1bd6c740 1 --2- 172.21.15.31:0/34447030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c30953de0 0x564c30953770 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:41.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.010+0000 7f8f1bd6c740 1 --2- 172.21.15.31:0/34447030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c3094b260 0x564c30953230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:41.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.010+0000 7f8f1bd6c740 1 --2- 172.21.15.31:0/34447030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c3094a8b0 0x564c309400a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:41.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.010+0000 7f8f1bd6c740 1 -- 172.21.15.31:0/34447030 >> 172.21.15.31:0/34447030 conn(0x564c30944f80 msgr2=0x564c30944160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:41.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.010+0000 7f8f1bd6c740 1 -- 172.21.15.31:0/34447030 shutdown_connections 2024-03-21T10:48:41.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.010+0000 7f8f1bd6c740 1 -- 172.21.15.31:0/34447030 wait complete. 2024-03-21T10:48:41.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.010+0000 7f8f1bd6c740 1 -- start start 2024-03-21T10:48:41.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.011+0000 7f8f1bd6c740 1 Processor -- start 2024-03-21T10:48:41.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.011+0000 7f8f1bd6c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c3094a8b0 0x564c3095bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:41.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.011+0000 7f8f1bd6c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c3094b260 0x564c30959530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:41.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.011+0000 7f8f1bd6c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c30953de0 0x564c30959b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:41.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.011+0000 7f8f1bd6c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f141af140 con 0x564c3094b260 2024-03-21T10:48:41.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.011+0000 7f8f1bd6c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f141b0420 con 0x564c30953de0 2024-03-21T10:48:41.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.011+0000 7f8f1bd6c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f141af760 con 0x564c3094a8b0 2024-03-21T10:48:41.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.011+0000 7f8f1bd6c740 10 client.0 _ll_register_callbacks cb 0x7f8f0400bb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:41.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.012+0000 7f8f11e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c3094a8b0 0x564c3095bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:41.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.012+0000 7f8f11e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c3094a8b0 0x564c3095bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:39222/0 (socket says 172.21.15.31:39222) 2024-03-21T10:48:41.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.012+0000 7f8f11e00640 1 -- 172.21.15.31:0/2028555233 learned_addr learned my addr 172.21.15.31:0/2028555233 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:41.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.012+0000 7f8f12800640 1 --2- 172.21.15.31:0/2028555233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c30953de0 0x564c30959b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:41.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.012+0000 7f8f11400640 1 --2- 172.21.15.31:0/2028555233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c3094b260 0x564c30959530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:41.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.012+0000 7f8f11e00640 1 -- 172.21.15.31:0/2028555233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c30953de0 msgr2=0x564c30959b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:41.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.012+0000 7f8f11e00640 1 --2- 172.21.15.31:0/2028555233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c30953de0 0x564c30959b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.012+0000 7f8f11e00640 1 -- 172.21.15.31:0/2028555233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c3094b260 msgr2=0x564c30959530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.012+0000 7f8f11e00640 1 --2- 172.21.15.31:0/2028555233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c3094b260 0x564c30959530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.012+0000 7f8f11e00640 1 -- 172.21.15.31:0/2028555233 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564c3095a360 con 0x564c3094a8b0 2024-03-21T10:48:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.012+0000 7f8f11400640 1 --2- 172.21.15.31:0/2028555233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c3094b260 0x564c30959530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:48:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.012+0000 7f8f11e00640 1 --2- 172.21.15.31:0/2028555233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c3094a8b0 0x564c3095bfa0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f8f04011010 tx=0x7f8f04010ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddc453ba919c5906 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.012+0000 7f8ef3e00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:48:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.012+0000 7f8ef3e00640 1 -- 172.21.15.31:0/2028555233 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f04010840 con 0x564c3094a8b0 2024-03-21T10:48:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.013+0000 7f8ef3e00640 1 -- 172.21.15.31:0/2028555233 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f8f0401c600 con 0x564c3094a8b0 2024-03-21T10:48:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.013+0000 7f8f1bd6c740 1 -- 172.21.15.31:0/2028555233 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8f0c003b70 con 0x564c3094a8b0 2024-03-21T10:48:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.013+0000 7f8ef3e00640 1 -- 172.21.15.31:0/2028555233 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f04045050 con 0x564c3094a8b0 2024-03-21T10:48:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.013+0000 7f8f1bd6c740 10 client.28105 fetch_fsmap learned FSMap version 28 2024-03-21T10:48:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.013+0000 7f8f1bd6c740 1 -- 172.21.15.31:0/2028555233 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x564c309bf660 con 0x564c3094a8b0 2024-03-21T10:48:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.013+0000 7f8ef3e00640 1 -- 172.21.15.31:0/2028555233 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f0c003b70 con 0x564c3094a8b0 2024-03-21T10:48:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.013+0000 7f8ef3e00640 1 -- 172.21.15.31:0/2028555233 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f8f04037670 con 0x564c3094a8b0 2024-03-21T10:48:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.013+0000 7f8f1bd6c740 10 client.28105 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T10:48:41.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.013+0000 7f8f1bd6c740 10 client.28105 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:48:41.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.013+0000 7f8f1bd6c740 20 client.28105 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:41.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.013+0000 7f8f1bd6c740 1 --2- 172.21.15.31:0/2028555233 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564c309bfdc0 0x564c309c21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:41.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.013+0000 7f8f1bd6c740 4 client.28105 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:48:41.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.013+0000 7f8f1bd6c740 1 -- 172.21.15.31:0/2028555233 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564c308b5b50 con 0x564c309bfdc0 2024-03-21T10:48:41.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.014+0000 7f8f11400640 1 --2- 172.21.15.31:0/2028555233 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564c309bfdc0 0x564c309c21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:41.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.014+0000 7f8f11400640 1 --2- 172.21.15.31:0/2028555233 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564c309bfdc0 0x564c309c21e0 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c671e23123add135 server_cookie=c984adc325486290 in_seq=0 out_seq=0 2024-03-21T10:48:41.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.014+0000 7f8ef3e00640 10 client.28105 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:41.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.014+0000 7f8ef3e00640 1 -- 172.21.15.31:0/2028555233 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564c308b5b50 con 0x564c309bfdc0 2024-03-21T10:48:41.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.014+0000 7f8ef3e00640 10 client.28105 handle_command_reply: tid=0 2024-03-21T10:48:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.014+0000 7f8f1bd6c740 1 client.28105 shutdown 2024-03-21T10:48:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.015+0000 7f8f1bd6c740 1 -- 172.21.15.31:0/2028555233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c3094a8b0 msgr2=0x564c3095bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.015+0000 7f8f1bd6c740 1 --2- 172.21.15.31:0/2028555233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c3094a8b0 0x564c3095bfa0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f8f04011010 tx=0x7f8f04010ec0 comp rx=0 tx=0).stop 2024-03-21T10:48:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.015+0000 7f8f1bd6c740 1 -- 172.21.15.31:0/2028555233 shutdown_connections 2024-03-21T10:48:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.015+0000 7f8f1bd6c740 1 --2- 172.21.15.31:0/2028555233 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564c309bfdc0 0x564c309c21e0 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.015+0000 7f8f1bd6c740 1 --2- 172.21.15.31:0/2028555233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c30953de0 0x564c30959b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.015+0000 7f8f1bd6c740 1 --2- 172.21.15.31:0/2028555233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c3094b260 0x564c30959530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.015+0000 7f8ef3e00640 0 client.28105 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.015+0000 7f8f1bd6c740 1 --2- 172.21.15.31:0/2028555233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c3094a8b0 0x564c3095bfa0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.015+0000 7f8f1bd6c740 1 -- 172.21.15.31:0/2028555233 >> 172.21.15.31:0/2028555233 conn(0x564c30944f80 msgr2=0x564c30944160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.015+0000 7f8f1bd6c740 1 -- 172.21.15.31:0/2028555233 shutdown_connections 2024-03-21T10:48:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.015+0000 7f8f1bd6c740 1 -- 172.21.15.31:0/2028555233 wait complete. 2024-03-21T10:48:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.015+0000 7f8f1bd6c740 20 client.28105 trim_cache size 0 max 16384 2024-03-21T10:48:41.016 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:48:41.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.016+0000 7f8f13200640 1 -- 172.21.15.31:0/3286325777 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f8ef80a3090 msgr2=0x7f8ef80a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:41.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.016+0000 7f8f13200640 1 --2- 172.21.15.31:0/3286325777 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f8ef80a3090 0x7f8ef80a5550 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x564c30945ff0 tx=0x7f8f0400b410 comp rx=0 tx=0).stop 2024-03-21T10:48:41.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.016+0000 7f8f13200640 1 -- 172.21.15.31:0/3286325777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f14104020 msgr2=0x7f8f1419a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.016+0000 7f8f13200640 1 --2- 172.21.15.31:0/3286325777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f14104020 0x7f8f1419a4c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f8f00006010 tx=0x7f8f00004430 comp rx=0 tx=0).stop 2024-03-21T10:48:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.016+0000 7f8f13200640 1 -- 172.21.15.31:0/3286325777 shutdown_connections 2024-03-21T10:48:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.016+0000 7f8f13200640 1 --2- 172.21.15.31:0/3286325777 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f8ef80a3090 0x7f8ef80a5550 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.016+0000 7f8f13200640 1 --2- 172.21.15.31:0/3286325777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f14109670 0x7f8f1419e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.016+0000 7f8f13200640 1 --2- 172.21.15.31:0/3286325777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f14104020 0x7f8f1419a4c0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.016+0000 7f8f13200640 1 --2- 172.21.15.31:0/3286325777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f14103660 0x7f8f14199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.016+0000 7f8f13200640 1 -- 172.21.15.31:0/3286325777 >> 172.21.15.31:0/3286325777 conn(0x7f8f140ff110 msgr2=0x7f8f140ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.016+0000 7f8f13200640 1 -- 172.21.15.31:0/3286325777 shutdown_connections 2024-03-21T10:48:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:41.016+0000 7f8f13200640 1 -- 172.21.15.31:0/3286325777 wait complete. 2024-03-21T10:48:41.028 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:40 smithi031 ceph-mon[31673]: pgmap v314: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 3.0 KiB/s wr, 8 op/s 2024-03-21T10:48:41.028 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:40 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1238222827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:48:41.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:40 smithi070 ceph-mon[36330]: pgmap v314: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 3.0 KiB/s wr, 8 op/s 2024-03-21T10:48:41.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:40 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1238222827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:48:41.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:40 smithi040 ceph-mon[36154]: pgmap v314: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 3.0 KiB/s wr, 8 op/s 2024-03-21T10:48:41.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:40 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1238222827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:48:42.031 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-21T10:48:42.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:48:42.031 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-21T10:48:42.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.288+0000 7ff085000640 1 -- 172.21.15.31:0/1041761277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff080073a40 msgr2=0x7ff080073e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:42.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.288+0000 7ff085000640 1 --2- 172.21.15.31:0/1041761277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff080073a40 0x7ff080073e20 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff070009900 tx=0x7ff07002bcd0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.288+0000 7ff085000640 1 -- 172.21.15.31:0/1041761277 shutdown_connections 2024-03-21T10:48:42.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.288+0000 7ff085000640 1 --2- 172.21.15.31:0/1041761277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff080112470 0x7ff080114860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.288+0000 7ff085000640 1 --2- 172.21.15.31:0/1041761277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff080074360 0x7ff080111f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.288+0000 7ff085000640 1 --2- 172.21.15.31:0/1041761277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff080073a40 0x7ff080073e20 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.288+0000 7ff085000640 1 -- 172.21.15.31:0/1041761277 >> 172.21.15.31:0/1041761277 conn(0x7ff0800fcff0 msgr2=0x7ff0800ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:42.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.288+0000 7ff085000640 1 -- 172.21.15.31:0/1041761277 shutdown_connections 2024-03-21T10:48:42.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.288+0000 7ff085000640 1 -- 172.21.15.31:0/1041761277 wait complete. 2024-03-21T10:48:42.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.290+0000 7ff085000640 1 Processor -- start 2024-03-21T10:48:42.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.290+0000 7ff085000640 1 -- start start 2024-03-21T10:48:42.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.290+0000 7ff085000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff080073a40 0x7ff08019e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.290+0000 7ff085000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff080074360 0x7ff08019e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.290+0000 7ff085000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff080112470 0x7ff0801a2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.290+0000 7ff085000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff080117ec0 con 0x7ff080112470 2024-03-21T10:48:42.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.290+0000 7ff085000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff080117d40 con 0x7ff080074360 2024-03-21T10:48:42.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.290+0000 7ff085000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff080118040 con 0x7ff080073a40 2024-03-21T10:48:42.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.291+0000 7ff07f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff080073a40 0x7ff08019e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.291+0000 7ff07ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff080074360 0x7ff08019e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.291+0000 7ff07f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff080073a40 0x7ff08019e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:39242/0 (socket says 172.21.15.31:39242) 2024-03-21T10:48:42.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.291+0000 7ff07f400640 1 -- 172.21.15.31:0/1818239228 learned_addr learned my addr 172.21.15.31:0/1818239228 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:42.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.291+0000 7ff07fe00640 1 --2- 172.21.15.31:0/1818239228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff080112470 0x7ff0801a2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.291+0000 7ff07f400640 1 -- 172.21.15.31:0/1818239228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff080074360 msgr2=0x7ff08019e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:42.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.291+0000 7ff07f400640 1 --2- 172.21.15.31:0/1818239228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff080074360 0x7ff08019e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.291+0000 7ff07f400640 1 -- 172.21.15.31:0/1818239228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff080112470 msgr2=0x7ff0801a2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:42.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.291+0000 7ff07f400640 1 --2- 172.21.15.31:0/1818239228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff080112470 0x7ff0801a2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.291+0000 7ff07f400640 1 -- 172.21.15.31:0/1818239228 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0801a3350 con 0x7ff080073a40 2024-03-21T10:48:42.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.291+0000 7ff07fe00640 1 --2- 172.21.15.31:0/1818239228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff080112470 0x7ff0801a2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:48:42.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.291+0000 7ff07ea00640 1 --2- 172.21.15.31:0/1818239228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff080074360 0x7ff08019e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:42.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.292+0000 7ff07f400640 1 --2- 172.21.15.31:0/1818239228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff080073a40 0x7ff08019e360 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff070000c00 tx=0x7ff070004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:42.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.292+0000 7ff077e00640 1 -- 172.21.15.31:0/1818239228 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff070042020 con 0x7ff080073a40 2024-03-21T10:48:42.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.292+0000 7ff077e00640 1 -- 172.21.15.31:0/1818239228 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0700054e0 con 0x7ff080073a40 2024-03-21T10:48:42.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.292+0000 7ff085000640 1 -- 172.21.15.31:0/1818239228 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0801a35e0 con 0x7ff080073a40 2024-03-21T10:48:42.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.292+0000 7ff077e00640 1 -- 172.21.15.31:0/1818239228 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff070003af0 con 0x7ff080073a40 2024-03-21T10:48:42.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.292+0000 7ff085000640 1 -- 172.21.15.31:0/1818239228 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0801aaed0 con 0x7ff080073a40 2024-03-21T10:48:42.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.295+0000 7ff077e00640 1 -- 172.21.15.31:0/1818239228 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7ff070045020 con 0x7ff080073a40 2024-03-21T10:48:42.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.297+0000 7ff077e00640 1 --2- 172.21.15.31:0/1818239228 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff0640a3090 0x7ff0640a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.297+0000 7ff089377740 1 Processor -- start 2024-03-21T10:48:42.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.297+0000 7ff089377740 1 -- start start 2024-03-21T10:48:42.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.297+0000 7ff077e00640 1 -- 172.21.15.31:0/1818239228 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7ff0700f1760 con 0x7ff080073a40 2024-03-21T10:48:42.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.297+0000 7ff07ea00640 1 --2- 172.21.15.31:0/1818239228 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff0640a3090 0x7ff0640a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.298+0000 7ff089377740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56353b08c290 0x56353b08c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.298+0000 7ff089377740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56353b08cbb0 0x56353b095250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.298+0000 7ff089377740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56353b095790 0x56353b097b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.298+0000 7ff089377740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56353aff0eb0 con 0x56353b08cbb0 2024-03-21T10:48:42.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.298+0000 7ff089377740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56353af99a60 con 0x56353b095790 2024-03-21T10:48:42.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.298+0000 7ff089377740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56353afecf50 con 0x56353b08c290 2024-03-21T10:48:42.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.298+0000 7ff07f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56353b08c290 0x56353b08c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.298+0000 7ff07f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56353b08c290 0x56353b08c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:39252/0 (socket says 172.21.15.31:39252) 2024-03-21T10:48:42.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.298+0000 7ff07f400640 1 -- 172.21.15.31:0/895526424 learned_addr learned my addr 172.21.15.31:0/895526424 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:42.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.298+0000 7ff07fe00640 1 --2- 172.21.15.31:0/895526424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56353b095790 0x56353b097b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.298+0000 7ff07ea00640 1 --2- 172.21.15.31:0/895526424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56353b08cbb0 0x56353b095250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.298+0000 7ff07f400640 1 -- 172.21.15.31:0/895526424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56353b095790 msgr2=0x56353b097b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:42.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.298+0000 7ff07f400640 1 --2- 172.21.15.31:0/895526424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56353b095790 0x56353b097b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.298+0000 7ff07f400640 1 -- 172.21.15.31:0/895526424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56353b08cbb0 msgr2=0x56353b095250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:42.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.298+0000 7ff07f400640 1 --2- 172.21.15.31:0/895526424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56353b08cbb0 0x56353b095250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.298+0000 7ff07f400640 1 -- 172.21.15.31:0/895526424 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56353adcd7e0 con 0x56353b08c290 2024-03-21T10:48:42.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.299+0000 7ff07f400640 1 --2- 172.21.15.31:0/895526424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56353b08c290 0x56353b08c670 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7ff070005ec0 tx=0x7ff0700035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5e0aa492a17ef90 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:42.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.299+0000 7ff07ea00640 1 --2- 172.21.15.31:0/1818239228 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff0640a3090 0x7ff0640a5550 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff0801052c0 tx=0x7ff06c013c30 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:42.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.299+0000 7ff074c00640 1 -- 172.21.15.31:0/895526424 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0700bdf60 con 0x56353b08c290 2024-03-21T10:48:42.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.299+0000 7ff074c00640 1 -- 172.21.15.31:0/895526424 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0700bd860 con 0x56353b08c290 2024-03-21T10:48:42.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.299+0000 7ff074c00640 1 -- 172.21.15.31:0/895526424 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0700b9ed0 con 0x56353b08c290 2024-03-21T10:48:42.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.299+0000 7ff089377740 1 -- 172.21.15.31:0/895526424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56353b08c290 msgr2=0x56353b08c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:42.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.299+0000 7ff089377740 1 --2- 172.21.15.31:0/895526424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56353b08c290 0x56353b08c670 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7ff070005ec0 tx=0x7ff0700035f0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.300+0000 7ff089377740 1 -- 172.21.15.31:0/895526424 shutdown_connections 2024-03-21T10:48:42.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.300+0000 7ff089377740 1 --2- 172.21.15.31:0/895526424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56353b095790 0x56353b097b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.300+0000 7ff089377740 1 --2- 172.21.15.31:0/895526424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56353b08cbb0 0x56353b095250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.300+0000 7ff089377740 1 --2- 172.21.15.31:0/895526424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56353b08c290 0x56353b08c670 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.300+0000 7ff089377740 1 -- 172.21.15.31:0/895526424 >> 172.21.15.31:0/895526424 conn(0x56353b0869c0 msgr2=0x56353b086da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:42.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.300+0000 7ff089377740 1 -- 172.21.15.31:0/895526424 shutdown_connections 2024-03-21T10:48:42.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.300+0000 7ff089377740 1 -- 172.21.15.31:0/895526424 wait complete. 2024-03-21T10:48:42.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.300+0000 7ff089377740 1 -- start start 2024-03-21T10:48:42.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.300+0000 7ff089377740 1 Processor -- start 2024-03-21T10:48:42.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.301+0000 7ff089377740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56353b08c290 0x56353b0951a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.301+0000 7ff089377740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56353b08cbb0 0x56353b092530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.301+0000 7ff089377740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56353b095790 0x56353b092b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.301+0000 7ff089377740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56353b098340 con 0x56353b08cbb0 2024-03-21T10:48:42.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.301+0000 7ff089377740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56353afaceb0 con 0x56353b095790 2024-03-21T10:48:42.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.301+0000 7ff089377740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56353b0984c0 con 0x56353b08c290 2024-03-21T10:48:42.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.301+0000 7ff089377740 10 client.0 _ll_register_callbacks cb 0x56353afa1150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:42.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.301+0000 7ff07f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56353b08c290 0x56353b0951a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.301+0000 7ff07f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56353b08c290 0x56353b0951a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:39268/0 (socket says 172.21.15.31:39268) 2024-03-21T10:48:42.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.302+0000 7ff07f400640 1 -- 172.21.15.31:0/2926288634 learned_addr learned my addr 172.21.15.31:0/2926288634 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:42.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.302+0000 7ff07fe00640 1 --2- 172.21.15.31:0/2926288634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56353b095790 0x56353b092b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.302+0000 7ff07ea00640 1 --2- 172.21.15.31:0/2926288634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56353b08cbb0 0x56353b092530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.302+0000 7ff07f400640 1 -- 172.21.15.31:0/2926288634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56353b095790 msgr2=0x56353b092b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:42.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.302+0000 7ff07f400640 1 --2- 172.21.15.31:0/2926288634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56353b095790 0x56353b092b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.302+0000 7ff07f400640 1 -- 172.21.15.31:0/2926288634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56353b08cbb0 msgr2=0x56353b092530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:42.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.302+0000 7ff07f400640 1 --2- 172.21.15.31:0/2926288634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56353b08cbb0 0x56353b092530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.302+0000 7ff07f400640 1 -- 172.21.15.31:0/2926288634 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56353ae2df70 con 0x56353b08c290 2024-03-21T10:48:42.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.302+0000 7ff07ea00640 1 --2- 172.21.15.31:0/2926288634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56353b08cbb0 0x56353b092530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:48:42.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.302+0000 7ff07f400640 1 --2- 172.21.15.31:0/2926288634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56353b08c290 0x56353b0951a0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7ff0700bd560 tx=0x7ff070005bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59b336ee22eb4f0d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:42.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.302+0000 7ff069600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:48:42.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.302+0000 7ff069600640 1 -- 172.21.15.31:0/2926288634 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0700da070 con 0x56353b08c290 2024-03-21T10:48:42.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.302+0000 7ff069600640 1 -- 172.21.15.31:0/2926288634 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7ff0700dc2c0 con 0x56353b08c290 2024-03-21T10:48:42.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.302+0000 7ff089377740 1 -- 172.21.15.31:0/2926288634 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff0700bd860 con 0x56353b08c290 2024-03-21T10:48:42.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.303+0000 7ff069600640 1 -- 172.21.15.31:0/2926288634 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0700bdf60 con 0x56353b08c290 2024-03-21T10:48:42.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.303+0000 7ff089377740 10 client.28117 fetch_fsmap learned FSMap version 29 2024-03-21T10:48:42.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.303+0000 7ff089377740 1 -- 172.21.15.31:0/2926288634 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x56353acc5170 con 0x56353b08c290 2024-03-21T10:48:42.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.303+0000 7ff069600640 1 -- 172.21.15.31:0/2926288634 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0700bd860 con 0x56353b08c290 2024-03-21T10:48:42.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.303+0000 7ff069600640 1 -- 172.21.15.31:0/2926288634 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) v1 ==== 7930+0+0 (secure 0 0 0) 0x7ff0700f1310 con 0x56353b08c290 2024-03-21T10:48:42.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.303+0000 7ff089377740 10 client.28117 fetch_fsmap finished waiting for FSMap version 29 2024-03-21T10:48:42.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.303+0000 7ff089377740 10 client.28117 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:48:42.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.303+0000 7ff089377740 20 client.28117 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:42.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.303+0000 7ff089377740 1 --2- 172.21.15.31:0/2926288634 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56353b0a5460 0x56353b0a7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.303+0000 7ff089377740 4 client.28117 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:48:42.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.303+0000 7ff089377740 1 -- 172.21.15.31:0/2926288634 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56353affe3f0 con 0x56353b0a5460 2024-03-21T10:48:42.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.303+0000 7ff07ea00640 1 --2- 172.21.15.31:0/2926288634 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56353b0a5460 0x56353b0a7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.304+0000 7ff07ea00640 1 --2- 172.21.15.31:0/2926288634 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56353b0a5460 0x56353b0a7850 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7ea233879b6e1ac server_cookie=88765cab03a4a1f3 in_seq=0 out_seq=0 2024-03-21T10:48:42.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.304+0000 7ff069600640 10 client.28117 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:42.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.305+0000 7ff069600640 1 -- 172.21.15.31:0/2926288634 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56353affe3f0 con 0x56353b0a5460 2024-03-21T10:48:42.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.305+0000 7ff069600640 10 client.28117 handle_command_reply: tid=0 2024-03-21T10:48:42.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.305+0000 7ff089377740 1 client.28117 shutdown 2024-03-21T10:48:42.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.305+0000 7ff089377740 1 -- 172.21.15.31:0/2926288634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56353b08c290 msgr2=0x56353b0951a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.305+0000 7ff089377740 1 --2- 172.21.15.31:0/2926288634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56353b08c290 0x56353b0951a0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7ff0700bd560 tx=0x7ff070005bc0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.305+0000 7ff089377740 1 -- 172.21.15.31:0/2926288634 shutdown_connections 2024-03-21T10:48:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.305+0000 7ff089377740 1 --2- 172.21.15.31:0/2926288634 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56353b0a5460 0x56353b0a7850 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.305+0000 7ff089377740 1 --2- 172.21.15.31:0/2926288634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56353b095790 0x56353b092b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.305+0000 7ff089377740 1 --2- 172.21.15.31:0/2926288634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56353b08cbb0 0x56353b092530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.305+0000 7ff089377740 1 --2- 172.21.15.31:0/2926288634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56353b08c290 0x56353b0951a0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.305+0000 7ff089377740 1 -- 172.21.15.31:0/2926288634 >> 172.21.15.31:0/2926288634 conn(0x56353b0869c0 msgr2=0x56353b088710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.305+0000 7ff069600640 0 client.28117 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.305+0000 7ff089377740 1 -- 172.21.15.31:0/2926288634 shutdown_connections 2024-03-21T10:48:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.305+0000 7ff089377740 1 -- 172.21.15.31:0/2926288634 wait complete. 2024-03-21T10:48:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.305+0000 7ff089377740 20 client.28117 trim_cache size 0 max 16384 2024-03-21T10:48:42.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.323+0000 7ff089377740 1 Processor -- start 2024-03-21T10:48:42.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.324+0000 7ff089377740 1 -- start start 2024-03-21T10:48:42.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.324+0000 7ff089377740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56353b08c290 0x56353b094c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.324+0000 7ff089377740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56353b08cbb0 0x56353b0951a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.324+0000 7ff089377740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56353b095790 0x56353b0a0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.324+0000 7ff089377740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0801b72d0 con 0x56353b08cbb0 2024-03-21T10:48:42.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.324+0000 7ff089377740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0801b4dd0 con 0x56353b08c290 2024-03-21T10:48:42.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.324+0000 7ff089377740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0801b2e90 con 0x56353b095790 2024-03-21T10:48:42.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.324+0000 7ff07f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56353b08c290 0x56353b094c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.324+0000 7ff07fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56353b095790 0x56353b0a0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.324+0000 7ff07ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56353b08cbb0 0x56353b0951a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.324+0000 7ff07f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56353b08c290 0x56353b094c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:48106/0 (socket says 172.21.15.31:48106) 2024-03-21T10:48:42.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.324+0000 7ff07f400640 1 -- 172.21.15.31:0/84929782 learned_addr learned my addr 172.21.15.31:0/84929782 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:42.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.325+0000 7ff07fe00640 1 -- 172.21.15.31:0/84929782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56353b08c290 msgr2=0x56353b094c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:42.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.325+0000 7ff07fe00640 1 --2- 172.21.15.31:0/84929782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56353b08c290 0x56353b094c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.325+0000 7ff07fe00640 1 -- 172.21.15.31:0/84929782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56353b08cbb0 msgr2=0x56353b0951a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:42.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.325+0000 7ff07fe00640 1 --2- 172.21.15.31:0/84929782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56353b08cbb0 0x56353b0951a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.325+0000 7ff07fe00640 1 -- 172.21.15.31:0/84929782 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56353b092eb0 con 0x56353b095790 2024-03-21T10:48:42.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.325+0000 7ff07ea00640 1 --2- 172.21.15.31:0/84929782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56353b08cbb0 0x56353b0951a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:48:42.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.325+0000 7ff07fe00640 1 --2- 172.21.15.31:0/84929782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56353b095790 0x56353b0a0b40 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7ff078005fd0 tx=0x7ff0780129e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2d0a6815587df14 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:42.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.325+0000 7ff06be00640 1 -- 172.21.15.31:0/84929782 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078003a30 con 0x56353b095790 2024-03-21T10:48:42.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.325+0000 7ff06be00640 1 -- 172.21.15.31:0/84929782 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff078003bd0 con 0x56353b095790 2024-03-21T10:48:42.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.325+0000 7ff06be00640 1 -- 172.21.15.31:0/84929782 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078010a50 con 0x56353b095790 2024-03-21T10:48:42.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.326+0000 7ff089377740 1 -- 172.21.15.31:0/84929782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56353b095790 msgr2=0x56353b0a0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:42.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.326+0000 7ff089377740 1 --2- 172.21.15.31:0/84929782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56353b095790 0x56353b0a0b40 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7ff078005fd0 tx=0x7ff0780129e0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.326+0000 7ff089377740 1 -- 172.21.15.31:0/84929782 shutdown_connections 2024-03-21T10:48:42.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.326+0000 7ff089377740 1 --2- 172.21.15.31:0/84929782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56353b095790 0x56353b0a0b40 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.326+0000 7ff089377740 1 --2- 172.21.15.31:0/84929782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56353b08cbb0 0x56353b0951a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.326+0000 7ff089377740 1 --2- 172.21.15.31:0/84929782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56353b08c290 0x56353b094c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.326+0000 7ff089377740 1 -- 172.21.15.31:0/84929782 >> 172.21.15.31:0/84929782 conn(0x56353b0869c0 msgr2=0x56353b088550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:42.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.326+0000 7ff089377740 1 -- 172.21.15.31:0/84929782 shutdown_connections 2024-03-21T10:48:42.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.326+0000 7ff089377740 1 -- 172.21.15.31:0/84929782 wait complete. 2024-03-21T10:48:42.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.326+0000 7ff089377740 1 -- start start 2024-03-21T10:48:42.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.327+0000 7ff089377740 1 Processor -- start 2024-03-21T10:48:42.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.327+0000 7ff089377740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56353b08c290 0x56353b09d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.327+0000 7ff089377740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56353b08cbb0 0x56353b099760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.327+0000 7ff089377740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56353b095790 0x56353b099ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.327+0000 7ff089377740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0801b3490 con 0x56353b095790 2024-03-21T10:48:42.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.327+0000 7ff089377740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0801b4770 con 0x56353b08c290 2024-03-21T10:48:42.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.327+0000 7ff089377740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0801b3ab0 con 0x56353b08cbb0 2024-03-21T10:48:42.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.327+0000 7ff089377740 10 client.0 _ll_register_callbacks cb 0x7ff0780098e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:42.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.327+0000 7ff07f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56353b08c290 0x56353b09d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.327+0000 7ff07ea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56353b08cbb0 0x56353b099760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.327+0000 7ff07f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56353b08c290 0x56353b09d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:48110/0 (socket says 172.21.15.31:48110) 2024-03-21T10:48:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.328+0000 7ff07f400640 1 -- 172.21.15.31:0/3687449062 learned_addr learned my addr 172.21.15.31:0/3687449062 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.328+0000 7ff07fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56353b095790 0x56353b099ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.328+0000 7ff07f400640 1 -- 172.21.15.31:0/3687449062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56353b08cbb0 msgr2=0x56353b099760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.328+0000 7ff07f400640 1 --2- 172.21.15.31:0/3687449062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56353b08cbb0 0x56353b099760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.328+0000 7ff07f400640 1 -- 172.21.15.31:0/3687449062 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56353b095790 msgr2=0x56353b099ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.328+0000 7ff07f400640 1 --2- 172.21.15.31:0/3687449062 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56353b095790 0x56353b099ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.328+0000 7ff07f400640 1 -- 172.21.15.31:0/3687449062 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56353b09a4b0 con 0x56353b08c290 2024-03-21T10:48:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.328+0000 7ff07ea00640 1 --2- 172.21.15.31:0/3687449062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56353b08cbb0 0x56353b099760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.328+0000 7ff07f400640 1 --2- 172.21.15.31:0/3687449062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56353b08c290 0x56353b09d3e0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff0700bd560 tx=0x7ff0700bd630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e98db8e708ab9e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.328+0000 7ff069600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:48:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.328+0000 7ff069600640 1 -- 172.21.15.31:0/3687449062 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0700da070 con 0x56353b08c290 2024-03-21T10:48:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.328+0000 7ff069600640 1 -- 172.21.15.31:0/3687449062 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7ff0700dc2c0 con 0x56353b08c290 2024-03-21T10:48:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.328+0000 7ff089377740 1 -- 172.21.15.31:0/3687449062 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff078003bd0 con 0x56353b08c290 2024-03-21T10:48:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.329+0000 7ff069600640 1 -- 172.21.15.31:0/3687449062 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0700bd860 con 0x56353b08c290 2024-03-21T10:48:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.329+0000 7ff089377740 10 client.28196 fetch_fsmap learned FSMap version 29 2024-03-21T10:48:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.329+0000 7ff089377740 1 -- 172.21.15.31:0/3687449062 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x56353b09a4b0 con 0x56353b08c290 2024-03-21T10:48:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.329+0000 7ff069600640 1 -- 172.21.15.31:0/3687449062 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff078003bd0 con 0x56353b08c290 2024-03-21T10:48:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.329+0000 7ff069600640 1 -- 172.21.15.31:0/3687449062 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) v1 ==== 7930+0+0 (secure 0 0 0) 0x7ff070003c90 con 0x56353b08c290 2024-03-21T10:48:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.329+0000 7ff089377740 10 client.28196 fetch_fsmap finished waiting for FSMap version 29 2024-03-21T10:48:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.329+0000 7ff089377740 10 client.28196 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:48:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.329+0000 7ff089377740 20 client.28196 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.329+0000 7ff089377740 1 --2- 172.21.15.31:0/3687449062 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56353b101c80 0x56353b104070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.329+0000 7ff089377740 4 client.28196 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-21T10:48:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.329+0000 7ff089377740 1 -- 172.21.15.31:0/3687449062 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56353aff8740 con 0x56353b101c80 2024-03-21T10:48:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.330+0000 7ff07ea00640 1 --2- 172.21.15.31:0/3687449062 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56353b101c80 0x56353b104070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.330+0000 7ff07ea00640 1 --2- 172.21.15.31:0/3687449062 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56353b101c80 0x56353b104070 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca84fd6c98232716 server_cookie=381af77c0470bab1 in_seq=0 out_seq=0 2024-03-21T10:48:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.330+0000 7ff069600640 10 client.28196 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.331+0000 7ff069600640 1 -- 172.21.15.31:0/3687449062 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56353aff8740 con 0x56353b101c80 2024-03-21T10:48:42.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.331+0000 7ff069600640 10 client.28196 handle_command_reply: tid=0 2024-03-21T10:48:42.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.331+0000 7ff089377740 1 client.28196 shutdown 2024-03-21T10:48:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.331+0000 7ff089377740 1 -- 172.21.15.31:0/3687449062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56353b08c290 msgr2=0x56353b09d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.331+0000 7ff089377740 1 --2- 172.21.15.31:0/3687449062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56353b08c290 0x56353b09d3e0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff0700bd560 tx=0x7ff0700bd630 comp rx=0 tx=0).stop 2024-03-21T10:48:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.332+0000 7ff089377740 1 -- 172.21.15.31:0/3687449062 shutdown_connections 2024-03-21T10:48:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.332+0000 7ff089377740 1 --2- 172.21.15.31:0/3687449062 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56353b101c80 0x56353b104070 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.332+0000 7ff089377740 1 --2- 172.21.15.31:0/3687449062 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56353b095790 0x56353b099ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.332+0000 7ff089377740 1 --2- 172.21.15.31:0/3687449062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56353b08cbb0 0x56353b099760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.332+0000 7ff089377740 1 --2- 172.21.15.31:0/3687449062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56353b08c290 0x56353b09d3e0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.332+0000 7ff089377740 1 -- 172.21.15.31:0/3687449062 >> 172.21.15.31:0/3687449062 conn(0x56353b0869c0 msgr2=0x56353b0851f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.332+0000 7ff069600640 0 client.28196 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.332+0000 7ff089377740 1 -- 172.21.15.31:0/3687449062 shutdown_connections 2024-03-21T10:48:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.332+0000 7ff089377740 1 -- 172.21.15.31:0/3687449062 wait complete. 2024-03-21T10:48:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.332+0000 7ff089377740 20 client.28196 trim_cache size 0 max 16384 2024-03-21T10:48:42.333 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:48:42.333 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-03-21T10:48:42.333 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "2660d39d-26fb-48b7-92a0-ac6f0dbb9278", 2024-03-21T10:48:42.333 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-03-21T10:48:42.333 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:48:42.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.333+0000 7ff085000640 1 -- 172.21.15.31:0/1818239228 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff0640a3090 msgr2=0x7ff0640a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:42.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.333+0000 7ff085000640 1 --2- 172.21.15.31:0/1818239228 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff0640a3090 0x7ff0640a5550 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff0801052c0 tx=0x7ff06c013c30 comp rx=0 tx=0).stop 2024-03-21T10:48:42.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.333+0000 7ff085000640 1 -- 172.21.15.31:0/1818239228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff080073a40 msgr2=0x7ff08019e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:42.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.333+0000 7ff085000640 1 --2- 172.21.15.31:0/1818239228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff080073a40 0x7ff08019e360 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff070000c00 tx=0x7ff070004360 comp rx=0 tx=0).stop 2024-03-21T10:48:42.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.333+0000 7ff085000640 1 -- 172.21.15.31:0/1818239228 shutdown_connections 2024-03-21T10:48:42.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.333+0000 7ff085000640 1 --2- 172.21.15.31:0/1818239228 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff0640a3090 0x7ff0640a5550 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.333+0000 7ff085000640 1 --2- 172.21.15.31:0/1818239228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff080112470 0x7ff0801a2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.333+0000 7ff085000640 1 --2- 172.21.15.31:0/1818239228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff080074360 0x7ff08019e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.333+0000 7ff085000640 1 --2- 172.21.15.31:0/1818239228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff080073a40 0x7ff08019e360 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.333+0000 7ff085000640 1 -- 172.21.15.31:0/1818239228 >> 172.21.15.31:0/1818239228 conn(0x7ff0800fcff0 msgr2=0x7ff0800fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:42.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.333+0000 7ff085000640 1 -- 172.21.15.31:0/1818239228 shutdown_connections 2024-03-21T10:48:42.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.333+0000 7ff085000640 1 -- 172.21.15.31:0/1818239228 wait complete. 2024-03-21T10:48:42.355 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:48:42.356 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:48:42.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.565+0000 7fc36a200640 1 -- 172.21.15.31:0/5743717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc364074370 msgr2=0x7fc36410d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:42.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.565+0000 7fc36a200640 1 --2- 172.21.15.31:0/5743717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc364074370 0x7fc36410d800 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc350009900 tx=0x7fc35002bcc0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.565+0000 7fc36a200640 1 -- 172.21.15.31:0/5743717 shutdown_connections 2024-03-21T10:48:42.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.565+0000 7fc36a200640 1 --2- 172.21.15.31:0/5743717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc36410dd40 0x7fc364110130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.565+0000 7fc36a200640 1 --2- 172.21.15.31:0/5743717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc364074370 0x7fc36410d800 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.565+0000 7fc36a200640 1 --2- 172.21.15.31:0/5743717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc364073a50 0x7fc364073e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.565+0000 7fc36a200640 1 -- 172.21.15.31:0/5743717 >> 172.21.15.31:0/5743717 conn(0x7fc3640fcfb0 msgr2=0x7fc3640ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:42.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.565+0000 7fc36a200640 1 -- 172.21.15.31:0/5743717 shutdown_connections 2024-03-21T10:48:42.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.565+0000 7fc36a200640 1 -- 172.21.15.31:0/5743717 wait complete. 2024-03-21T10:48:42.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.566+0000 7fc36a200640 1 Processor -- start 2024-03-21T10:48:42.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.566+0000 7fc36a200640 1 -- start start 2024-03-21T10:48:42.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.567+0000 7fc36a200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc364073a50 0x7fc36419a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.567+0000 7fc36a200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc364074370 0x7fc36419a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.567+0000 7fc36a200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc36410dd40 0x7fc36419e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.567+0000 7fc36a200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc364113ab0 con 0x7fc364074370 2024-03-21T10:48:42.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.567+0000 7fc36a200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc364113930 con 0x7fc36410dd40 2024-03-21T10:48:42.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.567+0000 7fc36a200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc364113c30 con 0x7fc364073a50 2024-03-21T10:48:42.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.567+0000 7fc368e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc364073a50 0x7fc36419a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.567+0000 7fc368e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc364073a50 0x7fc36419a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:39312/0 (socket says 172.21.15.31:39312) 2024-03-21T10:48:42.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.567+0000 7fc368e00640 1 -- 172.21.15.31:0/1193802442 learned_addr learned my addr 172.21.15.31:0/1193802442 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:42.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.567+0000 7fc363e00640 1 --2- 172.21.15.31:0/1193802442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc364074370 0x7fc36419a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.567+0000 7fc369800640 1 --2- 172.21.15.31:0/1193802442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc36410dd40 0x7fc36419e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.567+0000 7fc368e00640 1 -- 172.21.15.31:0/1193802442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc36410dd40 msgr2=0x7fc36419e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:42.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.567+0000 7fc368e00640 1 --2- 172.21.15.31:0/1193802442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc36410dd40 0x7fc36419e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.568+0000 7fc368e00640 1 -- 172.21.15.31:0/1193802442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc364074370 msgr2=0x7fc36419a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:42.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.568+0000 7fc368e00640 1 --2- 172.21.15.31:0/1193802442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc364074370 0x7fc36419a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.568+0000 7fc368e00640 1 -- 172.21.15.31:0/1193802442 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36419ef60 con 0x7fc364073a50 2024-03-21T10:48:42.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.568+0000 7fc368e00640 1 --2- 172.21.15.31:0/1193802442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc364073a50 0x7fc36419a010 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fc35800ea60 tx=0x7fc35800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:42.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.568+0000 7fc361600640 1 -- 172.21.15.31:0/1193802442 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35800cde0 con 0x7fc364073a50 2024-03-21T10:48:42.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.568+0000 7fc36a200640 1 -- 172.21.15.31:0/1193802442 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc36419f250 con 0x7fc364073a50 2024-03-21T10:48:42.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.568+0000 7fc361600640 1 -- 172.21.15.31:0/1193802442 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3580035f0 con 0x7fc364073a50 2024-03-21T10:48:42.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.568+0000 7fc361600640 1 -- 172.21.15.31:0/1193802442 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc358010670 con 0x7fc364073a50 2024-03-21T10:48:42.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.568+0000 7fc36a200640 1 -- 172.21.15.31:0/1193802442 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3641a6ae0 con 0x7fc364073a50 2024-03-21T10:48:42.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.571+0000 7fc36e5a1740 1 Processor -- start 2024-03-21T10:48:42.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.571+0000 7fc361600640 1 -- 172.21.15.31:0/1193802442 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fc3580037a0 con 0x7fc364073a50 2024-03-21T10:48:42.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.572+0000 7fc36e5a1740 1 -- start start 2024-03-21T10:48:42.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.572+0000 7fc36e5a1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55666509add0 0x556665099b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.572+0000 7fc36e5a1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55666509a0f0 0x55666509a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.572+0000 7fc36e5a1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566650a3460 0x5566650a5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.572+0000 7fc36e5a1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556664ffe250 con 0x5566650a3460 2024-03-21T10:48:42.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.572+0000 7fc36e5a1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556664fa6a30 con 0x55666509a0f0 2024-03-21T10:48:42.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.572+0000 7fc36e5a1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x556664ffa2f0 con 0x55666509add0 2024-03-21T10:48:42.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.572+0000 7fc369800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55666509add0 0x556665099b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.572+0000 7fc369800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55666509add0 0x556665099b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:39328/0 (socket says 172.21.15.31:39328) 2024-03-21T10:48:42.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.572+0000 7fc368e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55666509a0f0 0x55666509a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.572+0000 7fc369800640 1 -- 172.21.15.31:0/4091109248 learned_addr learned my addr 172.21.15.31:0/4091109248 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:42.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.572+0000 7fc363e00640 1 --2- 172.21.15.31:0/4091109248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566650a3460 0x5566650a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.573+0000 7fc369800640 1 -- 172.21.15.31:0/4091109248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55666509a0f0 msgr2=0x55666509a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:42.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.573+0000 7fc369800640 1 --2- 172.21.15.31:0/4091109248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55666509a0f0 0x55666509a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.573+0000 7fc369800640 1 -- 172.21.15.31:0/4091109248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566650a3460 msgr2=0x5566650a5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:42.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.573+0000 7fc369800640 1 --2- 172.21.15.31:0/4091109248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566650a3460 0x5566650a5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.573+0000 7fc369800640 1 -- 172.21.15.31:0/4091109248 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556664f91690 con 0x55666509add0 2024-03-21T10:48:42.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.573+0000 7fc363e00640 1 --2- 172.21.15.31:0/4091109248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566650a3460 0x5566650a5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:48:42.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.573+0000 7fc369800640 1 --2- 172.21.15.31:0/4091109248 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55666509add0 0x556665099b20 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fc35c009f50 tx=0x7fc35c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a751d60d3e333f93 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:42.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.573+0000 7fc361600640 1 --2- 172.21.15.31:0/1193802442 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc34c0a3090 0x7fc34c0a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.573+0000 7fc356000640 1 -- 172.21.15.31:0/4091109248 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c004d20 con 0x55666509add0 2024-03-21T10:48:42.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.573+0000 7fc356000640 1 -- 172.21.15.31:0/4091109248 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc35c004050 con 0x55666509add0 2024-03-21T10:48:42.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.573+0000 7fc356000640 1 -- 172.21.15.31:0/4091109248 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c005280 con 0x55666509add0 2024-03-21T10:48:42.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.573+0000 7fc361600640 1 -- 172.21.15.31:0/1193802442 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fc3580cfe50 con 0x7fc364073a50 2024-03-21T10:48:42.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.574+0000 7fc368e00640 1 --2- 172.21.15.31:0/1193802442 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc34c0a3090 0x7fc34c0a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.574+0000 7fc36e5a1740 1 -- 172.21.15.31:0/4091109248 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55666509add0 msgr2=0x556665099b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:42.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.574+0000 7fc36e5a1740 1 --2- 172.21.15.31:0/4091109248 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55666509add0 0x556665099b20 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fc35c009f50 tx=0x7fc35c0118d0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.574+0000 7fc36e5a1740 1 -- 172.21.15.31:0/4091109248 shutdown_connections 2024-03-21T10:48:42.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.574+0000 7fc36e5a1740 1 --2- 172.21.15.31:0/4091109248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566650a3460 0x5566650a5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.574+0000 7fc36e5a1740 1 --2- 172.21.15.31:0/4091109248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55666509a0f0 0x55666509a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.574+0000 7fc36e5a1740 1 --2- 172.21.15.31:0/4091109248 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55666509add0 0x556665099b20 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.574+0000 7fc36e5a1740 1 -- 172.21.15.31:0/4091109248 >> 172.21.15.31:0/4091109248 conn(0x556665061200 msgr2=0x556665094b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:42.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.574+0000 7fc36e5a1740 1 -- 172.21.15.31:0/4091109248 shutdown_connections 2024-03-21T10:48:42.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.574+0000 7fc368e00640 1 --2- 172.21.15.31:0/1193802442 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc34c0a3090 0x7fc34c0a5550 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x55666509b320 tx=0x7fc3580d3900 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:42.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.574+0000 7fc36e5a1740 1 -- 172.21.15.31:0/4091109248 wait complete. 2024-03-21T10:48:42.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.575+0000 7fc36e5a1740 1 -- start start 2024-03-21T10:48:42.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.575+0000 7fc36e5a1740 1 Processor -- start 2024-03-21T10:48:42.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.575+0000 7fc36e5a1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55666509a0f0 0x5566650a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.575+0000 7fc36e5a1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55666509add0 0x55666509ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.575+0000 7fc36e5a1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566650a3460 0x5566650a0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.575+0000 7fc36e5a1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566650a62d0 con 0x55666509add0 2024-03-21T10:48:42.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.576+0000 7fc36e5a1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556664fb9e80 con 0x55666509a0f0 2024-03-21T10:48:42.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.576+0000 7fc36e5a1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566650a6450 con 0x5566650a3460 2024-03-21T10:48:42.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.576+0000 7fc36e5a1740 10 client.0 _ll_register_callbacks cb 0x556664fcc130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:42.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.576+0000 7fc369800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55666509a0f0 0x5566650a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.576+0000 7fc369800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55666509a0f0 0x5566650a2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:48150/0 (socket says 172.21.15.31:48150) 2024-03-21T10:48:42.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.576+0000 7fc363e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55666509add0 0x55666509ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.576+0000 7fc369800640 1 -- 172.21.15.31:0/3012610826 learned_addr learned my addr 172.21.15.31:0/3012610826 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:42.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.576+0000 7fc369800640 1 --2- 172.21.15.31:0/3012610826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566650a3460 0x5566650a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.576+0000 7fc363e00640 1 -- 172.21.15.31:0/3012610826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566650a3460 msgr2=0x5566650a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:42.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.576+0000 7fc363e00640 1 --2- 172.21.15.31:0/3012610826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566650a3460 0x5566650a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.576+0000 7fc363e00640 1 -- 172.21.15.31:0/3012610826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55666509a0f0 msgr2=0x5566650a2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:42.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.576+0000 7fc363e00640 1 --2- 172.21.15.31:0/3012610826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55666509a0f0 0x5566650a2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.576+0000 7fc363e00640 1 -- 172.21.15.31:0/3012610826 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556664ddc7d0 con 0x55666509add0 2024-03-21T10:48:42.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.577+0000 7fc363e00640 1 --2- 172.21.15.31:0/3012610826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55666509add0 0x55666509ff10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc35002c200 tx=0x7fc350005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bea95cec4784b2a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:42.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.577+0000 7fc34aa00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:48:42.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.577+0000 7fc34aa00640 1 -- 172.21.15.31:0/3012610826 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc350043070 con 0x55666509add0 2024-03-21T10:48:42.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.577+0000 7fc34aa00640 1 -- 172.21.15.31:0/3012610826 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fc35002cbe0 con 0x55666509add0 2024-03-21T10:48:42.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.577+0000 7fc36e5a1740 1 -- 172.21.15.31:0/3012610826 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc35c004050 con 0x55666509add0 2024-03-21T10:48:42.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.577+0000 7fc34aa00640 1 -- 172.21.15.31:0/3012610826 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc350050050 con 0x55666509add0 2024-03-21T10:48:42.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.577+0000 7fc36e5a1740 10 client.17568 fetch_fsmap learned FSMap version 29 2024-03-21T10:48:42.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.577+0000 7fc36e5a1740 1 -- 172.21.15.31:0/3012610826 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x556664d123f0 con 0x55666509add0 2024-03-21T10:48:42.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.578+0000 7fc34aa00640 1 -- 172.21.15.31:0/3012610826 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc35c004050 con 0x55666509add0 2024-03-21T10:48:42.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.578+0000 7fc34aa00640 1 -- 172.21.15.31:0/3012610826 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fc350005260 con 0x55666509add0 2024-03-21T10:48:42.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.578+0000 7fc36e5a1740 10 client.17568 fetch_fsmap finished waiting for FSMap version 29 2024-03-21T10:48:42.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.578+0000 7fc36e5a1740 10 client.17568 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:48:42.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.578+0000 7fc36e5a1740 20 client.17568 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:42.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.578+0000 7fc36e5a1740 1 --2- 172.21.15.31:0/3012610826 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5566650b33a0 0x5566650b5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.578+0000 7fc36e5a1740 4 client.17568 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:48:42.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.578+0000 7fc36e5a1740 1 -- 172.21.15.31:0/3012610826 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55666500b790 con 0x5566650b33a0 2024-03-21T10:48:42.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.578+0000 7fc369800640 1 --2- 172.21.15.31:0/3012610826 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5566650b33a0 0x5566650b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.579+0000 7fc369800640 1 --2- 172.21.15.31:0/3012610826 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5566650b33a0 0x5566650b5790 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a840469d86c338d server_cookie=e1e380dd36b810b9 in_seq=0 out_seq=0 2024-03-21T10:48:42.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.579+0000 7fc34aa00640 10 client.17568 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:42.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.580+0000 7fc34aa00640 1 -- 172.21.15.31:0/3012610826 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55666500b790 con 0x5566650b33a0 2024-03-21T10:48:42.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.580+0000 7fc34aa00640 10 client.17568 handle_command_reply: tid=0 2024-03-21T10:48:42.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.580+0000 7fc36e5a1740 1 client.17568 shutdown 2024-03-21T10:48:42.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.580+0000 7fc36e5a1740 1 -- 172.21.15.31:0/3012610826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55666509add0 msgr2=0x55666509ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:42.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.580+0000 7fc36e5a1740 1 --2- 172.21.15.31:0/3012610826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55666509add0 0x55666509ff10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc35002c200 tx=0x7fc350005dd0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.580+0000 7fc36e5a1740 1 -- 172.21.15.31:0/3012610826 shutdown_connections 2024-03-21T10:48:42.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.580+0000 7fc36e5a1740 1 --2- 172.21.15.31:0/3012610826 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5566650b33a0 0x5566650b5790 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.580+0000 7fc36e5a1740 1 --2- 172.21.15.31:0/3012610826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566650a3460 0x5566650a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.580+0000 7fc36e5a1740 1 --2- 172.21.15.31:0/3012610826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55666509add0 0x55666509ff10 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.580+0000 7fc34aa00640 0 client.17568 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:42.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.580+0000 7fc36e5a1740 1 --2- 172.21.15.31:0/3012610826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55666509a0f0 0x5566650a2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.580+0000 7fc36e5a1740 1 -- 172.21.15.31:0/3012610826 >> 172.21.15.31:0/3012610826 conn(0x556665061200 msgr2=0x556665096890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:42.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.580+0000 7fc36e5a1740 1 -- 172.21.15.31:0/3012610826 shutdown_connections 2024-03-21T10:48:42.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.580+0000 7fc36e5a1740 1 -- 172.21.15.31:0/3012610826 wait complete. 2024-03-21T10:48:42.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.580+0000 7fc36e5a1740 20 client.17568 trim_cache size 0 max 16384 2024-03-21T10:48:42.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.599+0000 7fc36e5a1740 1 Processor -- start 2024-03-21T10:48:42.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.599+0000 7fc36e5a1740 1 -- start start 2024-03-21T10:48:42.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.599+0000 7fc36e5a1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55666509a0f0 0x5566650a2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.599+0000 7fc36e5a1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55666509add0 0x5566650a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.599+0000 7fc36e5a1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566650a3460 0x5566650a4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.599+0000 7fc36e5a1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3641b2f70 con 0x55666509a0f0 2024-03-21T10:48:42.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.599+0000 7fc36e5a1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3641b0a70 con 0x55666509add0 2024-03-21T10:48:42.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.599+0000 7fc36e5a1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3641aeb30 con 0x5566650a3460 2024-03-21T10:48:42.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.599+0000 7fc369800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55666509a0f0 0x5566650a2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.599+0000 7fc363e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55666509add0 0x5566650a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.599+0000 7fc369800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55666509a0f0 0x5566650a2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57568/0 (socket says 172.21.15.31:57568) 2024-03-21T10:48:42.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.599+0000 7fc369800640 1 -- 172.21.15.31:0/3208115838 learned_addr learned my addr 172.21.15.31:0/3208115838 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:42.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.599+0000 7fc369800640 1 --2- 172.21.15.31:0/3208115838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566650a3460 0x5566650a4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.600+0000 7fc363e00640 1 -- 172.21.15.31:0/3208115838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566650a3460 msgr2=0x5566650a4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:42.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.600+0000 7fc363e00640 1 --2- 172.21.15.31:0/3208115838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566650a3460 0x5566650a4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.600+0000 7fc363e00640 1 -- 172.21.15.31:0/3208115838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55666509a0f0 msgr2=0x5566650a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:42.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.600+0000 7fc363e00640 1 --2- 172.21.15.31:0/3208115838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55666509a0f0 0x5566650a2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.600+0000 7fc363e00640 1 -- 172.21.15.31:0/3208115838 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556664cd20f0 con 0x55666509add0 2024-03-21T10:48:42.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.600+0000 7fc363e00640 1 --2- 172.21.15.31:0/3208115838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55666509add0 0x5566650a2b80 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fc350005ab0 tx=0x7fc350005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fe01f7146e183d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:42.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.600+0000 7fc355600640 1 -- 172.21.15.31:0/3208115838 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc350043040 con 0x55666509add0 2024-03-21T10:48:42.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.600+0000 7fc355600640 1 -- 172.21.15.31:0/3208115838 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc350009520 con 0x55666509add0 2024-03-21T10:48:42.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.600+0000 7fc355600640 1 -- 172.21.15.31:0/3208115838 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc350004920 con 0x55666509add0 2024-03-21T10:48:42.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.601+0000 7fc36e5a1740 1 -- 172.21.15.31:0/3208115838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55666509add0 msgr2=0x5566650a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:42.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.601+0000 7fc36e5a1740 1 --2- 172.21.15.31:0/3208115838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55666509add0 0x5566650a2b80 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fc350005ab0 tx=0x7fc350005fd0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.601+0000 7fc36e5a1740 1 -- 172.21.15.31:0/3208115838 shutdown_connections 2024-03-21T10:48:42.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.601+0000 7fc36e5a1740 1 --2- 172.21.15.31:0/3208115838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566650a3460 0x5566650a4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.601+0000 7fc36e5a1740 1 --2- 172.21.15.31:0/3208115838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55666509add0 0x5566650a2b80 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.601+0000 7fc36e5a1740 1 --2- 172.21.15.31:0/3208115838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55666509a0f0 0x5566650a2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.601+0000 7fc36e5a1740 1 -- 172.21.15.31:0/3208115838 >> 172.21.15.31:0/3208115838 conn(0x556665061200 msgr2=0x556665094670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:42.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.601+0000 7fc36e5a1740 1 -- 172.21.15.31:0/3208115838 shutdown_connections 2024-03-21T10:48:42.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.601+0000 7fc36e5a1740 1 -- 172.21.15.31:0/3208115838 wait complete. 2024-03-21T10:48:42.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.602+0000 7fc36e5a1740 1 -- start start 2024-03-21T10:48:42.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.602+0000 7fc36e5a1740 1 Processor -- start 2024-03-21T10:48:42.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.602+0000 7fc36e5a1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55666509a0f0 0x55666509d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.602+0000 7fc36e5a1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55666509add0 0x55666509d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.602+0000 7fc36e5a1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566650a3460 0x55666509dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.602+0000 7fc36e5a1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3641af130 con 0x5566650a3460 2024-03-21T10:48:42.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.602+0000 7fc36e5a1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3641b0410 con 0x55666509add0 2024-03-21T10:48:42.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.602+0000 7fc36e5a1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3641af750 con 0x55666509a0f0 2024-03-21T10:48:42.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.603+0000 7fc36e5a1740 10 client.0 _ll_register_callbacks cb 0x7fc358005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:42.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.603+0000 7fc363e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55666509add0 0x55666509d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.603+0000 7fc363e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55666509add0 0x55666509d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:48164/0 (socket says 172.21.15.31:48164) 2024-03-21T10:48:42.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.603+0000 7fc363e00640 1 -- 172.21.15.31:0/593210832 learned_addr learned my addr 172.21.15.31:0/593210832 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:42.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.603+0000 7fc369800640 1 --2- 172.21.15.31:0/593210832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55666509a0f0 0x55666509d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.603+0000 7fc363e00640 1 -- 172.21.15.31:0/593210832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55666509a0f0 msgr2=0x55666509d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:42.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.603+0000 7fc363e00640 1 --2- 172.21.15.31:0/593210832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55666509a0f0 0x55666509d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.603+0000 7fc363e00640 1 -- 172.21.15.31:0/593210832 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566650a3460 msgr2=0x55666509dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:42.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.603+0000 7fc363e00640 1 --2- 172.21.15.31:0/593210832 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566650a3460 0x55666509dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.603+0000 7fc363e00640 1 -- 172.21.15.31:0/593210832 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5566650a1790 con 0x55666509add0 2024-03-21T10:48:42.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.603+0000 7fc363e00640 1 --2- 172.21.15.31:0/593210832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55666509add0 0x55666509d6d0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fc350000c00 tx=0x7fc35002ce90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2133b1fd0d33b1a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:42.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.603+0000 7fc34aa00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:48:42.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.604+0000 7fc34aa00640 1 -- 172.21.15.31:0/593210832 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc350043030 con 0x55666509add0 2024-03-21T10:48:42.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.604+0000 7fc34aa00640 1 -- 172.21.15.31:0/593210832 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fc3500052f0 con 0x55666509add0 2024-03-21T10:48:42.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.604+0000 7fc36e5a1740 1 -- 172.21.15.31:0/593210832 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc350009520 con 0x55666509add0 2024-03-21T10:48:42.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.604+0000 7fc34aa00640 1 -- 172.21.15.31:0/593210832 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc3500056e0 con 0x55666509add0 2024-03-21T10:48:42.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.604+0000 7fc36e5a1740 10 client.28214 fetch_fsmap learned FSMap version 29 2024-03-21T10:48:42.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.604+0000 7fc36e5a1740 1 -- 172.21.15.31:0/593210832 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55666510f010 con 0x55666509add0 2024-03-21T10:48:42.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.604+0000 7fc34aa00640 1 -- 172.21.15.31:0/593210832 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc350009520 con 0x55666509add0 2024-03-21T10:48:42.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.604+0000 7fc34aa00640 1 -- 172.21.15.31:0/593210832 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fc350003c60 con 0x55666509add0 2024-03-21T10:48:42.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.604+0000 7fc36e5a1740 10 client.28214 fetch_fsmap finished waiting for FSMap version 29 2024-03-21T10:48:42.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.604+0000 7fc36e5a1740 10 client.28214 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:48:42.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.604+0000 7fc36e5a1740 20 client.28214 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:42.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.604+0000 7fc36e5a1740 1 --2- 172.21.15.31:0/593210832 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55666510f650 0x556665111a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:42.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.604+0000 7fc36e5a1740 4 client.28214 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:48:42.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.604+0000 7fc36e5a1740 1 -- 172.21.15.31:0/593210832 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556665005ae0 con 0x55666510f650 2024-03-21T10:48:42.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.605+0000 7fc369800640 1 --2- 172.21.15.31:0/593210832 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55666510f650 0x556665111a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:42.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.605+0000 7fc369800640 1 --2- 172.21.15.31:0/593210832 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55666510f650 0x556665111a70 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5dcf46492ee35492 server_cookie=645c75eb5313061e in_seq=0 out_seq=0 2024-03-21T10:48:42.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.605+0000 7fc34aa00640 10 client.28214 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:42.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.605+0000 7fc34aa00640 1 -- 172.21.15.31:0/593210832 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x556665005ae0 con 0x55666510f650 2024-03-21T10:48:42.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.605+0000 7fc34aa00640 10 client.28214 handle_command_reply: tid=0 2024-03-21T10:48:42.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.605+0000 7fc36e5a1740 1 client.28214 shutdown 2024-03-21T10:48:42.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.606+0000 7fc36e5a1740 1 -- 172.21.15.31:0/593210832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55666509add0 msgr2=0x55666509d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:42.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.606+0000 7fc36e5a1740 1 --2- 172.21.15.31:0/593210832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55666509add0 0x55666509d6d0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fc350000c00 tx=0x7fc35002ce90 comp rx=0 tx=0).stop 2024-03-21T10:48:42.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.606+0000 7fc36e5a1740 1 -- 172.21.15.31:0/593210832 shutdown_connections 2024-03-21T10:48:42.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.606+0000 7fc36e5a1740 1 --2- 172.21.15.31:0/593210832 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55666510f650 0x556665111a70 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.606+0000 7fc36e5a1740 1 --2- 172.21.15.31:0/593210832 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566650a3460 0x55666509dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.606+0000 7fc36e5a1740 1 --2- 172.21.15.31:0/593210832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55666509add0 0x55666509d6d0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.606+0000 7fc36e5a1740 1 --2- 172.21.15.31:0/593210832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55666509a0f0 0x55666509d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.606+0000 7fc36e5a1740 1 -- 172.21.15.31:0/593210832 >> 172.21.15.31:0/593210832 conn(0x556665061200 msgr2=0x55666509e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:42.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.606+0000 7fc34aa00640 0 client.28214 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:42.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.606+0000 7fc36e5a1740 1 -- 172.21.15.31:0/593210832 shutdown_connections 2024-03-21T10:48:42.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.606+0000 7fc36e5a1740 1 -- 172.21.15.31:0/593210832 wait complete. 2024-03-21T10:48:42.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.606+0000 7fc36e5a1740 20 client.28214 trim_cache size 0 max 16384 2024-03-21T10:48:42.608 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:48:42.608 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-21T10:48:42.608 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-03-21T10:48:42.608 INFO:teuthology.orchestra.run.smithi031.stdout: "2660d39d-26fb-48b7-92a0-ac6f0dbb9278": { 2024-03-21T10:48:42.608 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-03-21T10:48:42.608 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "2660d39d-26fb-48b7-92a0-ac6f0dbb9278", 2024-03-21T10:48:42.608 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-03-21T10:48:42.608 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:48:42.608 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:48:42.608 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:48:42.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.607+0000 7fc36a200640 1 -- 172.21.15.31:0/1193802442 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc34c0a3090 msgr2=0x7fc34c0a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:42.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.607+0000 7fc36a200640 1 --2- 172.21.15.31:0/1193802442 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc34c0a3090 0x7fc34c0a5550 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x55666509b320 tx=0x7fc3580d3900 comp rx=0 tx=0).stop 2024-03-21T10:48:42.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.607+0000 7fc36a200640 1 -- 172.21.15.31:0/1193802442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc364073a50 msgr2=0x7fc36419a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:42.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.607+0000 7fc36a200640 1 --2- 172.21.15.31:0/1193802442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc364073a50 0x7fc36419a010 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fc35800ea60 tx=0x7fc35800ef20 comp rx=0 tx=0).stop 2024-03-21T10:48:42.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.607+0000 7fc36a200640 1 -- 172.21.15.31:0/1193802442 shutdown_connections 2024-03-21T10:48:42.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.607+0000 7fc36a200640 1 --2- 172.21.15.31:0/1193802442 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc34c0a3090 0x7fc34c0a5550 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.607+0000 7fc36a200640 1 --2- 172.21.15.31:0/1193802442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc36410dd40 0x7fc36419e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.607+0000 7fc36a200640 1 --2- 172.21.15.31:0/1193802442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc364074370 0x7fc36419a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.607+0000 7fc36a200640 1 --2- 172.21.15.31:0/1193802442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc364073a50 0x7fc36419a010 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:42.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.607+0000 7fc36a200640 1 -- 172.21.15.31:0/1193802442 >> 172.21.15.31:0/1193802442 conn(0x7fc3640fcfb0 msgr2=0x7fc3640fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:42.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.607+0000 7fc36a200640 1 -- 172.21.15.31:0/1193802442 shutdown_connections 2024-03-21T10:48:42.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:48:42.607+0000 7fc36a200640 1 -- 172.21.15.31:0/1193802442 wait complete. 2024-03-21T10:48:42.622 INFO:tasks.cephfs.filesystem:scrub status for tag:2660d39d-26fb-48b7-92a0-ac6f0dbb9278 - {'path': '/', 'tag': '2660d39d-26fb-48b7-92a0-ac6f0dbb9278', 'options': 'recursive,force'} 2024-03-21T10:48:43.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:43 smithi031 ceph-mon[31673]: pgmap v315: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:48:43.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:43 smithi031 ceph-mon[31673]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:48:43.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:43 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:48:43.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:43 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:48:43.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:43 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:48:43.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:43 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:48:43.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:43 smithi070 ceph-mon[36330]: pgmap v315: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:48:43.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:43 smithi070 ceph-mon[36330]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:48:43.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:43 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:48:43.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:43 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:48:43.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:43 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:48:43.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:43 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:48:43.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:43 smithi040 ceph-mon[36154]: pgmap v315: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:48:43.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:43 smithi040 ceph-mon[36154]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:48:43.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:43 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:48:43.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:43 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:48:43.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:43 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:48:43.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:43 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:48:44.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:44 smithi031 ceph-mon[31673]: scrub queued for path: / 2024-03-21T10:48:44.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:44 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:48:44.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:44 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:48:44.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:44 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:48:44.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:44 smithi070 ceph-mon[36330]: scrub queued for path: / 2024-03-21T10:48:44.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:44 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:48:44.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:44 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:48:44.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:44 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:48:44.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:44 smithi040 ceph-mon[36154]: scrub queued for path: / 2024-03-21T10:48:44.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:44 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:48:44.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:44 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:48:44.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:44 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:48:45.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:45 smithi031 ceph-mon[31673]: pgmap v316: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:48:45.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:45 smithi070 ceph-mon[36330]: pgmap v316: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:48:45.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:45 smithi040 ceph-mon[36154]: pgmap v316: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:48:47.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:47 smithi031 ceph-mon[31673]: pgmap v317: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-21T10:48:47.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:47 smithi070 ceph-mon[36330]: pgmap v317: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-21T10:48:47.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:47 smithi040 ceph-mon[36154]: pgmap v317: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-21T10:48:49.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:49 smithi031 ceph-mon[31673]: pgmap v318: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-21T10:48:49.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:49 smithi070 ceph-mon[36330]: pgmap v318: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-21T10:48:49.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:49 smithi040 ceph-mon[36154]: pgmap v318: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-21T10:48:51.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:51 smithi031 ceph-mon[31673]: pgmap v319: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-21T10:48:51.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:51 smithi070 ceph-mon[36330]: pgmap v319: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-21T10:48:51.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:51 smithi040 ceph-mon[36154]: pgmap v319: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-21T10:48:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.765+0000 7f4c18200640 1 -- 172.21.15.31:0/3251028218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c10103870 msgr2=0x7f4c10103cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.765+0000 7f4c18200640 1 --2- 172.21.15.31:0/3251028218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c10103870 0x7f4c10103cf0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc009900 tx=0x7f4bfc02bc30 comp rx=0 tx=0).stop 2024-03-21T10:48:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.765+0000 7f4c18200640 1 -- 172.21.15.31:0/3251028218 shutdown_connections 2024-03-21T10:48:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.765+0000 7f4c18200640 1 --2- 172.21.15.31:0/3251028218 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c10104230 0x7f4c10110740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.765+0000 7f4c18200640 1 --2- 172.21.15.31:0/3251028218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c10103870 0x7f4c10103cf0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.765+0000 7f4c18200640 1 --2- 172.21.15.31:0/3251028218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c10109880 0x7f4c10109c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.765+0000 7f4c18200640 1 -- 172.21.15.31:0/3251028218 >> 172.21.15.31:0/3251028218 conn(0x7f4c100ff320 msgr2=0x7f4c10101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.765+0000 7f4c18200640 1 -- 172.21.15.31:0/3251028218 shutdown_connections 2024-03-21T10:48:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.765+0000 7f4c18200640 1 -- 172.21.15.31:0/3251028218 wait complete. 2024-03-21T10:48:51.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.766+0000 7f4c18200640 1 Processor -- start 2024-03-21T10:48:51.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.766+0000 7f4c18200640 1 -- start start 2024-03-21T10:48:51.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.766+0000 7f4c18200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c10103870 0x7f4c1019a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:51.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.766+0000 7f4c18200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c10104230 0x7f4c1019a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:51.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.766+0000 7f4c18200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c10109880 0x7f4c1019ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:51.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.766+0000 7f4c18200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c10113ca0 con 0x7f4c10104230 2024-03-21T10:48:51.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.766+0000 7f4c18200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c10113b20 con 0x7f4c10103870 2024-03-21T10:48:51.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.766+0000 7f4c18200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c10113e20 con 0x7f4c10109880 2024-03-21T10:48:51.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.767+0000 7f4c14e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c10104230 0x7f4c1019a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.767+0000 7f4c15800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c10103870 0x7f4c1019a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.767+0000 7f4c16200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c10109880 0x7f4c1019ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.767+0000 7f4c15800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c10103870 0x7f4c1019a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39736/0 (socket says 172.21.15.31:39736) 2024-03-21T10:48:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.767+0000 7f4c15800640 1 -- 172.21.15.31:0/1151066356 learned_addr learned my addr 172.21.15.31:0/1151066356 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.767+0000 7f4c15800640 1 -- 172.21.15.31:0/1151066356 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c10109880 msgr2=0x7f4c1019ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.767+0000 7f4c15800640 1 --2- 172.21.15.31:0/1151066356 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c10109880 0x7f4c1019ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.767+0000 7f4c15800640 1 -- 172.21.15.31:0/1151066356 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c10104230 msgr2=0x7f4c1019a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.767+0000 7f4c15800640 1 --2- 172.21.15.31:0/1151066356 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c10104230 0x7f4c1019a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.767+0000 7f4c15800640 1 -- 172.21.15.31:0/1151066356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c1019f140 con 0x7f4c10103870 2024-03-21T10:48:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.767+0000 7f4c15800640 1 --2- 172.21.15.31:0/1151066356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c10103870 0x7f4c1019a1a0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f4c0400c890 tx=0x7f4c0400cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.768+0000 7f4c0e000640 1 -- 172.21.15.31:0/1151066356 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c04004220 con 0x7f4c10103870 2024-03-21T10:48:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.768+0000 7f4c18200640 1 -- 172.21.15.31:0/1151066356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c1019f430 con 0x7f4c10103870 2024-03-21T10:48:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.768+0000 7f4c0e000640 1 -- 172.21.15.31:0/1151066356 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c040043c0 con 0x7f4c10103870 2024-03-21T10:48:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.768+0000 7f4c0e000640 1 -- 172.21.15.31:0/1151066356 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c04004a00 con 0x7f4c10103870 2024-03-21T10:48:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.768+0000 7f4c18200640 1 -- 172.21.15.31:0/1151066356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c101a6cc0 con 0x7f4c10103870 2024-03-21T10:48:51.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.772+0000 7f4c1ae64740 1 Processor -- start 2024-03-21T10:48:51.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.773+0000 7f4c1ae64740 1 -- start start 2024-03-21T10:48:51.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.773+0000 7f4c0e000640 1 -- 172.21.15.31:0/1151066356 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f4c040079b0 con 0x7f4c10103870 2024-03-21T10:48:51.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.773+0000 7f4c1ae64740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561801d72fd0 0x561801d733b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:51.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.773+0000 7f4c1ae64740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d738f0 0x561801d7c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:51.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.773+0000 7f4c1ae64740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561801d7c6c0 0x561801d7eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:51.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.773+0000 7f4c1ae64740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561801d353b0 con 0x561801d72fd0 2024-03-21T10:48:51.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.773+0000 7f4c1ae64740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561801cd7830 con 0x561801d738f0 2024-03-21T10:48:51.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.773+0000 7f4c1ae64740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x561801c805d0 con 0x561801d7c6c0 2024-03-21T10:48:51.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.773+0000 7f4c15800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d738f0 0x561801d7c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:51.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.773+0000 7f4c16200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561801d72fd0 0x561801d733b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:51.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.773+0000 7f4c14e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561801d7c6c0 0x561801d7eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:51.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.774+0000 7f4c15800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d738f0 0x561801d7c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39742/0 (socket says 172.21.15.31:39742) 2024-03-21T10:48:51.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.774+0000 7f4c15800640 1 -- 172.21.15.31:0/2843137821 learned_addr learned my addr 172.21.15.31:0/2843137821 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:51.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.774+0000 7f4c16200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561801d72fd0 0x561801d733b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43458/0 (socket says 172.21.15.31:43458) 2024-03-21T10:48:51.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.774+0000 7f4c15800640 1 -- 172.21.15.31:0/2843137821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561801d7c6c0 msgr2=0x561801d7eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:51.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.774+0000 7f4c15800640 1 --2- 172.21.15.31:0/2843137821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561801d7c6c0 0x561801d7eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.774+0000 7f4c15800640 1 -- 172.21.15.31:0/2843137821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561801d72fd0 msgr2=0x561801d733b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:51.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.774+0000 7f4c15800640 1 --2- 172.21.15.31:0/2843137821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561801d72fd0 0x561801d733b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.774+0000 7f4c15800640 1 -- 172.21.15.31:0/2843137821 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561801c982c0 con 0x561801d738f0 2024-03-21T10:48:51.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.774+0000 7f4c15800640 1 --2- 172.21.15.31:0/2843137821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d738f0 0x561801d7c180 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f4c040cca90 tx=0x7f4c040d7690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c567a02e96ef73eb server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:51.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.775+0000 7f4c0e000640 1 --2- 172.21.15.31:0/1151066356 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4bf80a3080 0x7f4bf80a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:51.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.775+0000 7f4c02a00640 1 -- 172.21.15.31:0/2843137821 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c040de070 con 0x561801d738f0 2024-03-21T10:48:51.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.775+0000 7f4c16200640 1 --2- 172.21.15.31:0/1151066356 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4bf80a3080 0x7f4bf80a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:51.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.775+0000 7f4c0e000640 1 -- 172.21.15.31:0/1151066356 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f4c040cf770 con 0x7f4c10103870 2024-03-21T10:48:51.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.775+0000 7f4c02a00640 1 -- 172.21.15.31:0/2843137821 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c0409b870 con 0x561801d738f0 2024-03-21T10:48:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.775+0000 7f4c02a00640 1 -- 172.21.15.31:0/2843137821 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c04097b10 con 0x561801d738f0 2024-03-21T10:48:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.776+0000 7f4c1ae64740 1 -- 172.21.15.31:0/2843137821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d738f0 msgr2=0x561801d7c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.776+0000 7f4c1ae64740 1 --2- 172.21.15.31:0/2843137821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d738f0 0x561801d7c180 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f4c040cca90 tx=0x7f4c040d7690 comp rx=0 tx=0).stop 2024-03-21T10:48:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.776+0000 7f4c1ae64740 1 -- 172.21.15.31:0/2843137821 shutdown_connections 2024-03-21T10:48:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.776+0000 7f4c1ae64740 1 --2- 172.21.15.31:0/2843137821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561801d7c6c0 0x561801d7eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.776+0000 7f4c1ae64740 1 --2- 172.21.15.31:0/2843137821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d738f0 0x561801d7c180 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.776+0000 7f4c1ae64740 1 --2- 172.21.15.31:0/2843137821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561801d72fd0 0x561801d733b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.776+0000 7f4c1ae64740 1 -- 172.21.15.31:0/2843137821 >> 172.21.15.31:0/2843137821 conn(0x561801d6d8f0 msgr2=0x561801d6dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.776+0000 7f4c16200640 1 --2- 172.21.15.31:0/1151066356 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4bf80a3080 0x7f4bf80a5540 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4c08005fd0 tx=0x7f4c08009500 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.776+0000 7f4c1ae64740 1 -- 172.21.15.31:0/2843137821 shutdown_connections 2024-03-21T10:48:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.776+0000 7f4c1ae64740 1 -- 172.21.15.31:0/2843137821 wait complete. 2024-03-21T10:48:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.776+0000 7f4c1ae64740 1 -- start start 2024-03-21T10:48:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.777+0000 7f4c1ae64740 1 Processor -- start 2024-03-21T10:48:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.777+0000 7f4c1ae64740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d72fd0 0x561801d79750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.777+0000 7f4c1ae64740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561801d738f0 0x561801d79c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.777+0000 7f4c1ae64740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561801d7c6c0 0x561801d7a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.777+0000 7f4c1ae64740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561801c93a20 con 0x561801d7c6c0 2024-03-21T10:48:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.777+0000 7f4c1ae64740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561801cd38d0 con 0x561801d72fd0 2024-03-21T10:48:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.777+0000 7f4c1ae64740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x561801d7f2f0 con 0x561801d738f0 2024-03-21T10:48:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.777+0000 7f4c1ae64740 10 client.0 _ll_register_callbacks cb 0x561801cd01c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.777+0000 7f4c15800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d72fd0 0x561801d79750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.777+0000 7f4c14e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561801d738f0 0x561801d79c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.777+0000 7f4c15800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d72fd0 0x561801d79750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39750/0 (socket says 172.21.15.31:39750) 2024-03-21T10:48:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.777+0000 7f4c15800640 1 -- 172.21.15.31:0/1654376448 learned_addr learned my addr 172.21.15.31:0/1654376448 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.777+0000 7f4c16200640 1 --2- 172.21.15.31:0/1654376448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561801d7c6c0 0x561801d7a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.778+0000 7f4c15800640 1 -- 172.21.15.31:0/1654376448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561801d738f0 msgr2=0x561801d79c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.778+0000 7f4c15800640 1 --2- 172.21.15.31:0/1654376448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561801d738f0 0x561801d79c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.778+0000 7f4c15800640 1 -- 172.21.15.31:0/1654376448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561801d7c6c0 msgr2=0x561801d7a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.778+0000 7f4c15800640 1 --2- 172.21.15.31:0/1654376448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561801d7c6c0 0x561801d7a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.778+0000 7f4c15800640 1 -- 172.21.15.31:0/1654376448 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5618019ec210 con 0x561801d72fd0 2024-03-21T10:48:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.778+0000 7f4c14e00640 1 --2- 172.21.15.31:0/1654376448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561801d738f0 0x561801d79c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.778+0000 7f4c16200640 1 --2- 172.21.15.31:0/1654376448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561801d7c6c0 0x561801d7a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:48:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.778+0000 7f4c15800640 1 --2- 172.21.15.31:0/1654376448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d72fd0 0x561801d79750 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f4c040d51b0 tx=0x7f4c04004570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45cf4089bbf05273 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.778+0000 7f4bf7400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:48:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.778+0000 7f4bf7400640 1 -- 172.21.15.31:0/1654376448 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c0401d030 con 0x561801d72fd0 2024-03-21T10:48:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.778+0000 7f4bf7400640 1 -- 172.21.15.31:0/1654376448 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f4c04097ee0 con 0x561801d72fd0 2024-03-21T10:48:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.778+0000 7f4c1ae64740 1 -- 172.21.15.31:0/1654376448 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4c0409b870 con 0x561801d72fd0 2024-03-21T10:48:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.779+0000 7f4bf7400640 1 -- 172.21.15.31:0/1654376448 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c040de070 con 0x561801d72fd0 2024-03-21T10:48:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.779+0000 7f4c1ae64740 10 client.28226 fetch_fsmap learned FSMap version 29 2024-03-21T10:48:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.779+0000 7f4c1ae64740 1 -- 172.21.15.31:0/1654376448 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x561801ab8620 con 0x561801d72fd0 2024-03-21T10:48:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.779+0000 7f4bf7400640 1 -- 172.21.15.31:0/1654376448 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c0409b870 con 0x561801d72fd0 2024-03-21T10:48:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.779+0000 7f4bf7400640 1 -- 172.21.15.31:0/1654376448 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f4c04007ca0 con 0x561801d72fd0 2024-03-21T10:48:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.779+0000 7f4c1ae64740 10 client.28226 fetch_fsmap finished waiting for FSMap version 29 2024-03-21T10:48:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.779+0000 7f4c1ae64740 10 client.28226 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:48:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.779+0000 7f4c1ae64740 20 client.28226 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.779+0000 7f4c1ae64740 1 --2- 172.21.15.31:0/1654376448 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561801d8c5b0 0x561801d8e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.779+0000 7f4c1ae64740 4 client.28226 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:48:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.779+0000 7f4c1ae64740 1 -- 172.21.15.31:0/1654376448 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561801ce4d70 con 0x561801d8c5b0 2024-03-21T10:48:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.780+0000 7f4c14e00640 1 --2- 172.21.15.31:0/1654376448 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561801d8c5b0 0x561801d8e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:51.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.780+0000 7f4c14e00640 1 --2- 172.21.15.31:0/1654376448 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561801d8c5b0 0x561801d8e9a0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b13e3326102d11d server_cookie=da3764746c6e1780 in_seq=0 out_seq=0 2024-03-21T10:48:51.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.780+0000 7f4bf7400640 10 client.28226 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:51.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.781+0000 7f4bf7400640 1 -- 172.21.15.31:0/1654376448 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561801ce4d70 con 0x561801d8c5b0 2024-03-21T10:48:51.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.781+0000 7f4bf7400640 10 client.28226 handle_command_reply: tid=0 2024-03-21T10:48:51.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.781+0000 7f4c1ae64740 1 client.28226 shutdown 2024-03-21T10:48:51.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.781+0000 7f4c1ae64740 1 -- 172.21.15.31:0/1654376448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d72fd0 msgr2=0x561801d79750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:51.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.781+0000 7f4c1ae64740 1 --2- 172.21.15.31:0/1654376448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d72fd0 0x561801d79750 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f4c040d51b0 tx=0x7f4c04004570 comp rx=0 tx=0).stop 2024-03-21T10:48:51.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.781+0000 7f4c1ae64740 1 -- 172.21.15.31:0/1654376448 shutdown_connections 2024-03-21T10:48:51.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.781+0000 7f4c1ae64740 1 --2- 172.21.15.31:0/1654376448 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561801d8c5b0 0x561801d8e9a0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.781+0000 7f4c1ae64740 1 --2- 172.21.15.31:0/1654376448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561801d7c6c0 0x561801d7a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.781+0000 7f4c1ae64740 1 --2- 172.21.15.31:0/1654376448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561801d738f0 0x561801d79c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.781+0000 7f4c1ae64740 1 --2- 172.21.15.31:0/1654376448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d72fd0 0x561801d79750 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.781+0000 7f4bf7400640 0 client.28226 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:51.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.781+0000 7f4c1ae64740 1 -- 172.21.15.31:0/1654376448 >> 172.21.15.31:0/1654376448 conn(0x561801d6d8f0 msgr2=0x561801d6f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:51.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.782+0000 7f4c1ae64740 1 -- 172.21.15.31:0/1654376448 shutdown_connections 2024-03-21T10:48:51.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.782+0000 7f4c1ae64740 1 -- 172.21.15.31:0/1654376448 wait complete. 2024-03-21T10:48:51.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.782+0000 7f4c1ae64740 20 client.28226 trim_cache size 0 max 16384 2024-03-21T10:48:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.797+0000 7f4c1ae64740 1 Processor -- start 2024-03-21T10:48:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.797+0000 7f4c1ae64740 1 -- start start 2024-03-21T10:48:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.797+0000 7f4c1ae64740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561801d72fd0 0x561801d7be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.797+0000 7f4c1ae64740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d738f0 0x561801d6c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.797+0000 7f4c1ae64740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561801d7c6c0 0x561801d769b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.797+0000 7f4c1ae64740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c101b3190 con 0x561801d72fd0 2024-03-21T10:48:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.797+0000 7f4c1ae64740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c101b0c90 con 0x561801d738f0 2024-03-21T10:48:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.797+0000 7f4c1ae64740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c101aed50 con 0x561801d7c6c0 2024-03-21T10:48:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.798+0000 7f4c15800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561801d72fd0 0x561801d7be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.798+0000 7f4c14e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d738f0 0x561801d6c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.798+0000 7f4c16200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561801d7c6c0 0x561801d769b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.798+0000 7f4c15800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561801d72fd0 0x561801d7be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43474/0 (socket says 172.21.15.31:43474) 2024-03-21T10:48:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.798+0000 7f4c15800640 1 -- 172.21.15.31:0/3150060897 learned_addr learned my addr 172.21.15.31:0/3150060897 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.798+0000 7f4c15800640 1 -- 172.21.15.31:0/3150060897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561801d7c6c0 msgr2=0x561801d769b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.798+0000 7f4c15800640 1 --2- 172.21.15.31:0/3150060897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561801d7c6c0 0x561801d769b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.798+0000 7f4c15800640 1 -- 172.21.15.31:0/3150060897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d738f0 msgr2=0x561801d6c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.798+0000 7f4c15800640 1 --2- 172.21.15.31:0/3150060897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d738f0 0x561801d6c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.798+0000 7f4c15800640 1 -- 172.21.15.31:0/3150060897 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561801d7eab0 con 0x561801d72fd0 2024-03-21T10:48:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.798+0000 7f4c14e00640 1 --2- 172.21.15.31:0/3150060897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d738f0 0x561801d6c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.798+0000 7f4c15800640 1 --2- 172.21.15.31:0/3150060897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561801d72fd0 0x561801d7be80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4c0409b570 tx=0x7f4c040d7690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db6f4b1ce19efef5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.799+0000 7f4c02000640 1 -- 172.21.15.31:0/3150060897 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c0401d070 con 0x561801d72fd0 2024-03-21T10:48:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.799+0000 7f4c02000640 1 -- 172.21.15.31:0/3150060897 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c040d7b90 con 0x561801d72fd0 2024-03-21T10:48:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.799+0000 7f4c02000640 1 -- 172.21.15.31:0/3150060897 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c0409b1d0 con 0x561801d72fd0 2024-03-21T10:48:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.799+0000 7f4c1ae64740 1 -- 172.21.15.31:0/3150060897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561801d72fd0 msgr2=0x561801d7be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.799+0000 7f4c1ae64740 1 --2- 172.21.15.31:0/3150060897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561801d72fd0 0x561801d7be80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4c0409b570 tx=0x7f4c040d7690 comp rx=0 tx=0).stop 2024-03-21T10:48:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.800+0000 7f4c1ae64740 1 -- 172.21.15.31:0/3150060897 shutdown_connections 2024-03-21T10:48:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.800+0000 7f4c1ae64740 1 --2- 172.21.15.31:0/3150060897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561801d7c6c0 0x561801d769b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.800+0000 7f4c1ae64740 1 --2- 172.21.15.31:0/3150060897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d738f0 0x561801d6c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.800+0000 7f4c1ae64740 1 --2- 172.21.15.31:0/3150060897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561801d72fd0 0x561801d7be80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.800+0000 7f4c1ae64740 1 -- 172.21.15.31:0/3150060897 >> 172.21.15.31:0/3150060897 conn(0x561801d6d8f0 msgr2=0x561801d6c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.800+0000 7f4c1ae64740 1 -- 172.21.15.31:0/3150060897 shutdown_connections 2024-03-21T10:48:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.800+0000 7f4c1ae64740 1 -- 172.21.15.31:0/3150060897 wait complete. 2024-03-21T10:48:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.800+0000 7f4c1ae64740 1 -- start start 2024-03-21T10:48:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.800+0000 7f4c1ae64740 1 Processor -- start 2024-03-21T10:48:51.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.801+0000 7f4c1ae64740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561801d72fd0 0x561801d8eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:51.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.801+0000 7f4c1ae64740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d738f0 0x561801d8f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:51.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.801+0000 7f4c1ae64740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561801d7c6c0 0x561801d8f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:51.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.801+0000 7f4c1ae64740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c101af350 con 0x561801d72fd0 2024-03-21T10:48:51.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.801+0000 7f4c1ae64740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c101b0630 con 0x561801d738f0 2024-03-21T10:48:51.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.801+0000 7f4c1ae64740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c101af970 con 0x561801d7c6c0 2024-03-21T10:48:51.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.801+0000 7f4c1ae64740 10 client.0 _ll_register_callbacks cb 0x7f4c08009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:48:51.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.801+0000 7f4c15800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561801d72fd0 0x561801d8eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:51.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.801+0000 7f4c14e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d738f0 0x561801d8f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:51.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.801+0000 7f4c15800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561801d72fd0 0x561801d8eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43488/0 (socket says 172.21.15.31:43488) 2024-03-21T10:48:51.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.801+0000 7f4c15800640 1 -- 172.21.15.31:0/379005142 learned_addr learned my addr 172.21.15.31:0/379005142 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:48:51.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.801+0000 7f4c14e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d738f0 0x561801d8f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39772/0 (socket says 172.21.15.31:39772) 2024-03-21T10:48:51.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.801+0000 7f4c16200640 1 --2- 172.21.15.31:0/379005142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561801d7c6c0 0x561801d8f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:51.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.802+0000 7f4c14e00640 1 -- 172.21.15.31:0/379005142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561801d7c6c0 msgr2=0x561801d8f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:51.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.802+0000 7f4c14e00640 1 --2- 172.21.15.31:0/379005142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561801d7c6c0 0x561801d8f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.802+0000 7f4c14e00640 1 -- 172.21.15.31:0/379005142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561801d72fd0 msgr2=0x561801d8eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:48:51.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.802+0000 7f4c14e00640 1 --2- 172.21.15.31:0/379005142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561801d72fd0 0x561801d8eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.802+0000 7f4c14e00640 1 -- 172.21.15.31:0/379005142 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561801def3f0 con 0x561801d738f0 2024-03-21T10:48:51.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.802+0000 7f4c15800640 1 --2- 172.21.15.31:0/379005142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561801d72fd0 0x561801d8eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:51.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.802+0000 7f4c16200640 1 --2- 172.21.15.31:0/379005142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561801d7c6c0 0x561801d8f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:48:51.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.802+0000 7f4c14e00640 1 --2- 172.21.15.31:0/379005142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d738f0 0x561801d8f430 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc036040 tx=0x7f4bfc004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fda3bd4e1b4b37d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:48:51.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.802+0000 7f4bf7400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:48:51.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.802+0000 7f4bf7400640 1 -- 172.21.15.31:0/379005142 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc02cbb0 con 0x561801d738f0 2024-03-21T10:48:51.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.802+0000 7f4c1ae64740 1 -- 172.21.15.31:0/379005142 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4c040d7b90 con 0x561801d738f0 2024-03-21T10:48:51.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.802+0000 7f4bf7400640 1 -- 172.21.15.31:0/379005142 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f4bfc0046c0 con 0x561801d738f0 2024-03-21T10:48:51.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.803+0000 7f4bf7400640 1 -- 172.21.15.31:0/379005142 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4bfc04a300 con 0x561801d738f0 2024-03-21T10:48:51.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.803+0000 7f4c1ae64740 10 client.28232 fetch_fsmap learned FSMap version 29 2024-03-21T10:48:51.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.803+0000 7f4c1ae64740 1 -- 172.21.15.31:0/379005142 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x561801df0760 con 0x561801d738f0 2024-03-21T10:48:51.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.803+0000 7f4bf7400640 1 -- 172.21.15.31:0/379005142 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c040d7b90 con 0x561801d738f0 2024-03-21T10:48:51.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.803+0000 7f4bf7400640 1 -- 172.21.15.31:0/379005142 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f4bfc04a6f0 con 0x561801d738f0 2024-03-21T10:48:51.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.803+0000 7f4c1ae64740 10 client.28232 fetch_fsmap finished waiting for FSMap version 29 2024-03-21T10:48:51.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.803+0000 7f4c1ae64740 10 client.28232 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:48:51.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.803+0000 7f4c1ae64740 20 client.28232 populate_metadata read hostname 'smithi031' 2024-03-21T10:48:51.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.803+0000 7f4c1ae64740 1 --2- 172.21.15.31:0/379005142 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561801df0ef0 0x561801df3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:48:51.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.803+0000 7f4c1ae64740 4 client.28232 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:48:51.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.803+0000 7f4c1ae64740 1 -- 172.21.15.31:0/379005142 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x561801cdf0c0 con 0x561801df0ef0 2024-03-21T10:48:51.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.803+0000 7f4c15800640 1 --2- 172.21.15.31:0/379005142 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561801df0ef0 0x561801df3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:48:51.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.804+0000 7f4c15800640 1 --2- 172.21.15.31:0/379005142 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561801df0ef0 0x561801df3310 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fc5bf42e00b01cc server_cookie=f832fa3054c6f43f in_seq=0 out_seq=0 2024-03-21T10:48:51.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.804+0000 7f4bf7400640 10 client.28232 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:51.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.808+0000 7f4bf7400640 1 -- 172.21.15.31:0/379005142 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x561801cdf0c0 con 0x561801df0ef0 2024-03-21T10:48:51.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.808+0000 7f4bf7400640 10 client.28232 handle_command_reply: tid=0 2024-03-21T10:48:51.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.808+0000 7f4c1ae64740 1 client.28232 shutdown 2024-03-21T10:48:51.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.808+0000 7f4c1ae64740 1 -- 172.21.15.31:0/379005142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d738f0 msgr2=0x561801d8f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:51.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.808+0000 7f4c1ae64740 1 --2- 172.21.15.31:0/379005142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d738f0 0x561801d8f430 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc036040 tx=0x7f4bfc004610 comp rx=0 tx=0).stop 2024-03-21T10:48:51.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.808+0000 7f4c1ae64740 1 -- 172.21.15.31:0/379005142 shutdown_connections 2024-03-21T10:48:51.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.808+0000 7f4c1ae64740 1 --2- 172.21.15.31:0/379005142 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561801df0ef0 0x561801df3310 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.808+0000 7f4c1ae64740 1 --2- 172.21.15.31:0/379005142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561801d7c6c0 0x561801d8f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.808+0000 7f4c1ae64740 1 --2- 172.21.15.31:0/379005142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561801d738f0 0x561801d8f430 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.808+0000 7f4c1ae64740 1 --2- 172.21.15.31:0/379005142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561801d72fd0 0x561801d8eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.808+0000 7f4bf7400640 0 client.28232 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:48:51.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.808+0000 7f4c1ae64740 1 -- 172.21.15.31:0/379005142 >> 172.21.15.31:0/379005142 conn(0x561801d6d8f0 msgr2=0x561801d6c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:51.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.808+0000 7f4c1ae64740 1 -- 172.21.15.31:0/379005142 shutdown_connections 2024-03-21T10:48:51.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.808+0000 7f4c1ae64740 1 -- 172.21.15.31:0/379005142 wait complete. 2024-03-21T10:48:51.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.808+0000 7f4c1ae64740 20 client.28232 trim_cache size 0 max 16384 2024-03-21T10:48:51.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:48:51.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:48:51.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:48:51.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:48:51.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.809+0000 7f4c18200640 1 -- 172.21.15.31:0/1151066356 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4bf80a3080 msgr2=0x7f4bf80a5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:51.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.809+0000 7f4c18200640 1 --2- 172.21.15.31:0/1151066356 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4bf80a3080 0x7f4bf80a5540 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4c08005fd0 tx=0x7f4c08009500 comp rx=0 tx=0).stop 2024-03-21T10:48:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.809+0000 7f4c18200640 1 -- 172.21.15.31:0/1151066356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c10103870 msgr2=0x7f4c1019a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:48:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.809+0000 7f4c18200640 1 --2- 172.21.15.31:0/1151066356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c10103870 0x7f4c1019a1a0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f4c0400c890 tx=0x7f4c0400cd50 comp rx=0 tx=0).stop 2024-03-21T10:48:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.809+0000 7f4c18200640 1 -- 172.21.15.31:0/1151066356 shutdown_connections 2024-03-21T10:48:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.809+0000 7f4c18200640 1 --2- 172.21.15.31:0/1151066356 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4bf80a3080 0x7f4bf80a5540 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.809+0000 7f4c18200640 1 --2- 172.21.15.31:0/1151066356 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4c10109880 0x7f4c1019ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.809+0000 7f4c18200640 1 --2- 172.21.15.31:0/1151066356 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c10104230 0x7f4c1019a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.809+0000 7f4c18200640 1 --2- 172.21.15.31:0/1151066356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c10103870 0x7f4c1019a1a0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:48:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.809+0000 7f4c18200640 1 -- 172.21.15.31:0/1151066356 >> 172.21.15.31:0/1151066356 conn(0x7f4c100ff320 msgr2=0x7f4c100ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:48:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.809+0000 7f4c18200640 1 -- 172.21.15.31:0/1151066356 shutdown_connections 2024-03-21T10:48:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:48:51.809+0000 7f4c18200640 1 -- 172.21.15.31:0/1151066356 wait complete. 2024-03-21T10:48:53.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:53 smithi031 ceph-mon[31673]: pgmap v320: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 7 op/s 2024-03-21T10:48:53.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:53 smithi070 ceph-mon[36330]: pgmap v320: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 7 op/s 2024-03-21T10:48:53.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:53 smithi040 ceph-mon[36154]: pgmap v320: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 7 op/s 2024-03-21T10:48:55.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:55 smithi031 ceph-mon[31673]: pgmap v321: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-21T10:48:55.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:55 smithi070 ceph-mon[36330]: pgmap v321: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-21T10:48:55.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:55 smithi040 ceph-mon[36154]: pgmap v321: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-21T10:48:57.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:57 smithi031 ceph-mon[31673]: pgmap v322: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-21T10:48:57.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:57 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:48:57.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:57 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:48:57.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:57 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:48:57.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:57 smithi070 ceph-mon[36330]: pgmap v322: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-21T10:48:57.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:57 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:48:57.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:57 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:48:57.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:57 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:48:57.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:57 smithi040 ceph-mon[36154]: pgmap v322: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-21T10:48:57.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:57 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:48:57.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:57 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:48:57.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:57 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:48:59.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:48:59 smithi031 ceph-mon[31673]: pgmap v323: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:48:59.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:48:59 smithi070 ceph-mon[36330]: pgmap v323: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:48:59.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:48:59 smithi040 ceph-mon[36154]: pgmap v323: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:49:01.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:01 smithi031 ceph-mon[31673]: pgmap v324: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-21T10:49:01.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:01 smithi070 ceph-mon[36330]: pgmap v324: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-21T10:49:01.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:01 smithi040 ceph-mon[36154]: pgmap v324: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-21T10:49:03.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:03 smithi070 ceph-mon[36330]: pgmap v325: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:49:03.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:03 smithi040 ceph-mon[36154]: pgmap v325: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:49:03.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:03 smithi031 ceph-mon[31673]: pgmap v325: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:49:04.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.957+0000 7f3f47400640 1 -- 172.21.15.31:0/2415619703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f40103890 msgr2=0x7f3f40103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:04.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.957+0000 7f3f47400640 1 --2- 172.21.15.31:0/2415619703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f40103890 0x7f3f40103d10 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f3f28009940 tx=0x7f3f2802bc20 comp rx=0 tx=0).stop 2024-03-21T10:49:04.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.957+0000 7f3f47400640 1 -- 172.21.15.31:0/2415619703 shutdown_connections 2024-03-21T10:49:04.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.957+0000 7f3f47400640 1 --2- 172.21.15.31:0/2415619703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f40104250 0x7f3f40110760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.957+0000 7f3f47400640 1 --2- 172.21.15.31:0/2415619703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f40103890 0x7f3f40103d10 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.957+0000 7f3f47400640 1 --2- 172.21.15.31:0/2415619703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f401098a0 0x7f3f40109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.957+0000 7f3f47400640 1 -- 172.21.15.31:0/2415619703 >> 172.21.15.31:0/2415619703 conn(0x7f3f400ff320 msgr2=0x7f3f40101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:04.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.957+0000 7f3f47400640 1 -- 172.21.15.31:0/2415619703 shutdown_connections 2024-03-21T10:49:04.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.957+0000 7f3f47400640 1 -- 172.21.15.31:0/2415619703 wait complete. 2024-03-21T10:49:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.958+0000 7f3f47400640 1 Processor -- start 2024-03-21T10:49:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.958+0000 7f3f47400640 1 -- start start 2024-03-21T10:49:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.958+0000 7f3f47400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f40103890 0x7f3f4019a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.958+0000 7f3f47400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f40104250 0x7f3f4019a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.958+0000 7f3f47400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f401098a0 0x7f3f4019ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.959+0000 7f3f47400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f40113cb0 con 0x7f3f401098a0 2024-03-21T10:49:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.959+0000 7f3f47400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f40113b30 con 0x7f3f40103890 2024-03-21T10:49:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.959+0000 7f3f47400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f40113e30 con 0x7f3f40104250 2024-03-21T10:49:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.959+0000 7f3f44a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f40103890 0x7f3f4019a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.959+0000 7f3f37e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f40104250 0x7f3f4019a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.959+0000 7f3f44a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f40103890 0x7f3f4019a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44294/0 (socket says 172.21.15.31:44294) 2024-03-21T10:49:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.959+0000 7f3f44a00640 1 -- 172.21.15.31:0/2452676978 learned_addr learned my addr 172.21.15.31:0/2452676978 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.959+0000 7f3f45400640 1 --2- 172.21.15.31:0/2452676978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f401098a0 0x7f3f4019ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.959+0000 7f3f37e00640 1 -- 172.21.15.31:0/2452676978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f40103890 msgr2=0x7f3f4019a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.959+0000 7f3f37e00640 1 --2- 172.21.15.31:0/2452676978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f40103890 0x7f3f4019a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.959+0000 7f3f37e00640 1 -- 172.21.15.31:0/2452676978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f401098a0 msgr2=0x7f3f4019ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.959+0000 7f3f37e00640 1 --2- 172.21.15.31:0/2452676978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f401098a0 0x7f3f4019ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.959+0000 7f3f37e00640 1 -- 172.21.15.31:0/2452676978 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4019f190 con 0x7f3f40104250 2024-03-21T10:49:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.959+0000 7f3f44a00640 1 --2- 172.21.15.31:0/2452676978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f40103890 0x7f3f4019a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.960+0000 7f3f37e00640 1 --2- 172.21.15.31:0/2452676978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f40104250 0x7f3f4019a730 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f3f28037040 tx=0x7f3f28004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.960+0000 7f3f3e000640 1 -- 172.21.15.31:0/2452676978 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f28041020 con 0x7f3f40104250 2024-03-21T10:49:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.960+0000 7f3f3e000640 1 -- 172.21.15.31:0/2452676978 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3f2802ce50 con 0x7f3f40104250 2024-03-21T10:49:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.960+0000 7f3f47400640 1 -- 172.21.15.31:0/2452676978 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f4019f420 con 0x7f3f40104250 2024-03-21T10:49:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.960+0000 7f3f3e000640 1 -- 172.21.15.31:0/2452676978 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f28039040 con 0x7f3f40104250 2024-03-21T10:49:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.960+0000 7f3f47400640 1 -- 172.21.15.31:0/2452676978 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f401a6d10 con 0x7f3f40104250 2024-03-21T10:49:04.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.963+0000 7f3f4a22e740 1 Processor -- start 2024-03-21T10:49:04.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.963+0000 7f3f4a22e740 1 -- start start 2024-03-21T10:49:04.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.963+0000 7f3f3e000640 1 -- 172.21.15.31:0/2452676978 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f3f28044020 con 0x7f3f40104250 2024-03-21T10:49:04.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.964+0000 7f3f4a22e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a57ca0dfd0 0x55a57ca0e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:04.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.964+0000 7f3f4a22e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a57ca0e8f0 0x55a57ca17180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:04.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.964+0000 7f3f4a22e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a57ca176c0 0x55a57ca19ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:04.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.964+0000 7f3f4a22e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a57c9d03b0 con 0x55a57ca0e8f0 2024-03-21T10:49:04.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.964+0000 7f3f4a22e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a57c972830 con 0x55a57ca176c0 2024-03-21T10:49:04.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.964+0000 7f3f4a22e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a57c91b5d0 con 0x55a57ca0dfd0 2024-03-21T10:49:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.964+0000 7f3f45400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a57ca0dfd0 0x55a57ca0e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.964+0000 7f3f44a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a57ca0e8f0 0x55a57ca17180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.964+0000 7f3f45400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a57ca0dfd0 0x55a57ca0e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34536/0 (socket says 172.21.15.31:34536) 2024-03-21T10:49:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.964+0000 7f3f45400640 1 -- 172.21.15.31:0/1657806209 learned_addr learned my addr 172.21.15.31:0/1657806209 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.964+0000 7f3f37e00640 1 --2- 172.21.15.31:0/1657806209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a57ca176c0 0x55a57ca19ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.964+0000 7f3f45400640 1 -- 172.21.15.31:0/1657806209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a57ca176c0 msgr2=0x55a57ca19ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.964+0000 7f3f45400640 1 --2- 172.21.15.31:0/1657806209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a57ca176c0 0x55a57ca19ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.964+0000 7f3f45400640 1 -- 172.21.15.31:0/1657806209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a57ca0e8f0 msgr2=0x55a57ca17180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.964+0000 7f3f45400640 1 --2- 172.21.15.31:0/1657806209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a57ca0e8f0 0x55a57ca17180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.964+0000 7f3f45400640 1 -- 172.21.15.31:0/1657806209 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a57c9332c0 con 0x55a57ca0dfd0 2024-03-21T10:49:04.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.964+0000 7f3f44a00640 1 --2- 172.21.15.31:0/1657806209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a57ca0e8f0 0x55a57ca17180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:04.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.965+0000 7f3f45400640 1 --2- 172.21.15.31:0/1657806209 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a57ca0dfd0 0x55a57ca0e3b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f3f38009fd0 tx=0x7f3f380118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5dda72f169f8352d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:04.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.965+0000 7f3f35c00640 1 -- 172.21.15.31:0/1657806209 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f38004d20 con 0x55a57ca0dfd0 2024-03-21T10:49:04.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.965+0000 7f3f35c00640 1 -- 172.21.15.31:0/1657806209 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3f38004050 con 0x55a57ca0dfd0 2024-03-21T10:49:04.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.965+0000 7f3f35c00640 1 -- 172.21.15.31:0/1657806209 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f38005260 con 0x55a57ca0dfd0 2024-03-21T10:49:04.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.966+0000 7f3f4a22e740 1 -- 172.21.15.31:0/1657806209 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a57ca0dfd0 msgr2=0x55a57ca0e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:04.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.966+0000 7f3f4a22e740 1 --2- 172.21.15.31:0/1657806209 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a57ca0dfd0 0x55a57ca0e3b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f3f38009fd0 tx=0x7f3f380118d0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.966+0000 7f3f4a22e740 1 -- 172.21.15.31:0/1657806209 shutdown_connections 2024-03-21T10:49:04.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.966+0000 7f3f4a22e740 1 --2- 172.21.15.31:0/1657806209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a57ca176c0 0x55a57ca19ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.966+0000 7f3f4a22e740 1 --2- 172.21.15.31:0/1657806209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a57ca0e8f0 0x55a57ca17180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.966+0000 7f3f4a22e740 1 --2- 172.21.15.31:0/1657806209 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a57ca0dfd0 0x55a57ca0e3b0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.966+0000 7f3f4a22e740 1 -- 172.21.15.31:0/1657806209 >> 172.21.15.31:0/1657806209 conn(0x55a57ca088f0 msgr2=0x55a57ca08cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:04.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.966+0000 7f3f4a22e740 1 -- 172.21.15.31:0/1657806209 shutdown_connections 2024-03-21T10:49:04.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.966+0000 7f3f4a22e740 1 -- 172.21.15.31:0/1657806209 wait complete. 2024-03-21T10:49:04.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.966+0000 7f3f3e000640 1 --2- 172.21.15.31:0/2452676978 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3f240a3010 0x7f3f240a54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:04.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.966+0000 7f3f3e000640 1 -- 172.21.15.31:0/2452676978 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f3f280f0de0 con 0x7f3f40104250 2024-03-21T10:49:04.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.966+0000 7f3f44a00640 1 --2- 172.21.15.31:0/2452676978 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3f240a3010 0x7f3f240a54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:04.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.966+0000 7f3f4a22e740 1 -- start start 2024-03-21T10:49:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.967+0000 7f3f4a22e740 1 Processor -- start 2024-03-21T10:49:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.967+0000 7f3f44a00640 1 --2- 172.21.15.31:0/2452676978 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3f240a3010 0x7f3f240a54d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x55a57ca0da80 tx=0x7f3f30005f90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.967+0000 7f3f4a22e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a57ca0dfd0 0x55a57ca14770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.968+0000 7f3f4a22e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a57ca0e8f0 0x55a57ca14cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.968+0000 7f3f4a22e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a57ca176c0 0x55a57ca151f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.968+0000 7f3f4a22e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a57c92ea20 con 0x55a57ca0e8f0 2024-03-21T10:49:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.968+0000 7f3f4a22e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a57c96e8d0 con 0x55a57ca0dfd0 2024-03-21T10:49:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.968+0000 7f3f4a22e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a57ca1a2f0 con 0x55a57ca176c0 2024-03-21T10:49:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.968+0000 7f3f4a22e740 10 client.0 _ll_register_callbacks cb 0x55a57c96b1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.968+0000 7f3f44a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a57ca0dfd0 0x55a57ca14770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.968+0000 7f3f37e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a57ca0e8f0 0x55a57ca14cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.968+0000 7f3f44a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a57ca0dfd0 0x55a57ca14770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44312/0 (socket says 172.21.15.31:44312) 2024-03-21T10:49:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.968+0000 7f3f44a00640 1 -- 172.21.15.31:0/2908319420 learned_addr learned my addr 172.21.15.31:0/2908319420 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.968+0000 7f3f45400640 1 --2- 172.21.15.31:0/2908319420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a57ca176c0 0x55a57ca151f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.968+0000 7f3f44a00640 1 -- 172.21.15.31:0/2908319420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a57ca176c0 msgr2=0x55a57ca151f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.968+0000 7f3f44a00640 1 --2- 172.21.15.31:0/2908319420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a57ca176c0 0x55a57ca151f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.968+0000 7f3f44a00640 1 -- 172.21.15.31:0/2908319420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a57ca0e8f0 msgr2=0x55a57ca14cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.968+0000 7f3f44a00640 1 --2- 172.21.15.31:0/2908319420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a57ca0e8f0 0x55a57ca14cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.968+0000 7f3f44a00640 1 -- 172.21.15.31:0/2908319420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a57c687210 con 0x55a57ca0dfd0 2024-03-21T10:49:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.968+0000 7f3f37e00640 1 --2- 172.21.15.31:0/2908319420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a57ca0e8f0 0x55a57ca14cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.969+0000 7f3f44a00640 1 --2- 172.21.15.31:0/2908319420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a57ca0dfd0 0x55a57ca14770 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f3f30003670 tx=0x7f3f30004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d18b69d8e9909c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.969+0000 7f3f2e000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:49:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.969+0000 7f3f2e000640 1 -- 172.21.15.31:0/2908319420 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3000f830 con 0x55a57ca0dfd0 2024-03-21T10:49:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.969+0000 7f3f2e000640 1 -- 172.21.15.31:0/2908319420 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f3f3000f9d0 con 0x55a57ca0dfd0 2024-03-21T10:49:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.969+0000 7f3f4a22e740 1 -- 172.21.15.31:0/2908319420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3f38004050 con 0x55a57ca0dfd0 2024-03-21T10:49:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.970+0000 7f3f2e000640 1 -- 172.21.15.31:0/2908319420 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f3000fbd0 con 0x55a57ca0dfd0 2024-03-21T10:49:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.970+0000 7f3f4a22e740 10 client.28244 fetch_fsmap learned FSMap version 29 2024-03-21T10:49:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.970+0000 7f3f4a22e740 1 -- 172.21.15.31:0/2908319420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55a57c753620 con 0x55a57ca0dfd0 2024-03-21T10:49:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.970+0000 7f3f2e000640 1 -- 172.21.15.31:0/2908319420 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3f38004050 con 0x55a57ca0dfd0 2024-03-21T10:49:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.970+0000 7f3f2e000640 1 -- 172.21.15.31:0/2908319420 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f3f30027070 con 0x55a57ca0dfd0 2024-03-21T10:49:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.970+0000 7f3f4a22e740 10 client.28244 fetch_fsmap finished waiting for FSMap version 29 2024-03-21T10:49:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.970+0000 7f3f4a22e740 10 client.28244 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:49:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.970+0000 7f3f4a22e740 20 client.28244 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.970+0000 7f3f4a22e740 1 --2- 172.21.15.31:0/2908319420 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a57ca27800 0x55a57ca29bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.970+0000 7f3f4a22e740 4 client.28244 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:49:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.970+0000 7f3f4a22e740 1 -- 172.21.15.31:0/2908319420 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a57c97fd70 con 0x55a57ca27800 2024-03-21T10:49:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.971+0000 7f3f37e00640 1 --2- 172.21.15.31:0/2908319420 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a57ca27800 0x55a57ca29bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:04.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.971+0000 7f3f37e00640 1 --2- 172.21.15.31:0/2908319420 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a57ca27800 0x55a57ca29bf0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb6f23090d118fa5 server_cookie=bb76c375765c6df2 in_seq=0 out_seq=0 2024-03-21T10:49:04.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.971+0000 7f3f2e000640 10 client.28244 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:04.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.972+0000 7f3f2e000640 1 -- 172.21.15.31:0/2908319420 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a57c97fd70 con 0x55a57ca27800 2024-03-21T10:49:04.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.972+0000 7f3f2e000640 10 client.28244 handle_command_reply: tid=0 2024-03-21T10:49:04.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.972+0000 7f3f4a22e740 1 client.28244 shutdown 2024-03-21T10:49:04.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.972+0000 7f3f4a22e740 1 -- 172.21.15.31:0/2908319420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a57ca0dfd0 msgr2=0x55a57ca14770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:04.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.972+0000 7f3f4a22e740 1 --2- 172.21.15.31:0/2908319420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a57ca0dfd0 0x55a57ca14770 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f3f30003670 tx=0x7f3f30004730 comp rx=0 tx=0).stop 2024-03-21T10:49:04.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.972+0000 7f3f4a22e740 1 -- 172.21.15.31:0/2908319420 shutdown_connections 2024-03-21T10:49:04.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.972+0000 7f3f4a22e740 1 --2- 172.21.15.31:0/2908319420 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a57ca27800 0x55a57ca29bf0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.972+0000 7f3f4a22e740 1 --2- 172.21.15.31:0/2908319420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a57ca176c0 0x55a57ca151f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.972+0000 7f3f4a22e740 1 --2- 172.21.15.31:0/2908319420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a57ca0e8f0 0x55a57ca14cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.972+0000 7f3f2e000640 0 client.28244 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.972+0000 7f3f4a22e740 1 --2- 172.21.15.31:0/2908319420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a57ca0dfd0 0x55a57ca14770 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.972+0000 7f3f4a22e740 1 -- 172.21.15.31:0/2908319420 >> 172.21.15.31:0/2908319420 conn(0x55a57ca088f0 msgr2=0x55a57ca0a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.972+0000 7f3f4a22e740 1 -- 172.21.15.31:0/2908319420 shutdown_connections 2024-03-21T10:49:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.972+0000 7f3f4a22e740 1 -- 172.21.15.31:0/2908319420 wait complete. 2024-03-21T10:49:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.973+0000 7f3f4a22e740 20 client.28244 trim_cache size 0 max 16384 2024-03-21T10:49:04.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.988+0000 7f3f4a22e740 1 Processor -- start 2024-03-21T10:49:04.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.988+0000 7f3f4a22e740 1 -- start start 2024-03-21T10:49:04.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.988+0000 7f3f4a22e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a57ca0dfd0 0x55a57ca16ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:04.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.988+0000 7f3f4a22e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a57ca0e8f0 0x55a57ca13080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:04.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.988+0000 7f3f4a22e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a57ca176c0 0x55a57ca22a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:04.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.988+0000 7f3f4a22e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f401b3100 con 0x55a57ca0e8f0 2024-03-21T10:49:04.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.988+0000 7f3f4a22e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f401b0c00 con 0x55a57ca176c0 2024-03-21T10:49:04.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.988+0000 7f3f4a22e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f401aecc0 con 0x55a57ca0dfd0 2024-03-21T10:49:04.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.989+0000 7f3f44a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a57ca0dfd0 0x55a57ca16ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:04.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.989+0000 7f3f44a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a57ca0dfd0 0x55a57ca16ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34564/0 (socket says 172.21.15.31:34564) 2024-03-21T10:49:04.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.989+0000 7f3f44a00640 1 -- 172.21.15.31:0/2851661352 learned_addr learned my addr 172.21.15.31:0/2851661352 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:04.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.989+0000 7f3f45400640 1 --2- 172.21.15.31:0/2851661352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a57ca176c0 0x55a57ca22a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:04.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.989+0000 7f3f37e00640 1 --2- 172.21.15.31:0/2851661352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a57ca0e8f0 0x55a57ca13080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:04.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.989+0000 7f3f44a00640 1 -- 172.21.15.31:0/2851661352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a57ca176c0 msgr2=0x55a57ca22a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:04.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.989+0000 7f3f44a00640 1 --2- 172.21.15.31:0/2851661352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a57ca176c0 0x55a57ca22a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.989+0000 7f3f44a00640 1 -- 172.21.15.31:0/2851661352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a57ca0e8f0 msgr2=0x55a57ca13080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:04.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.989+0000 7f3f44a00640 1 --2- 172.21.15.31:0/2851661352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a57ca0e8f0 0x55a57ca13080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.989+0000 7f3f44a00640 1 -- 172.21.15.31:0/2851661352 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a57ca29f60 con 0x55a57ca0dfd0 2024-03-21T10:49:04.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.989+0000 7f3f45400640 1 --2- 172.21.15.31:0/2851661352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a57ca176c0 0x55a57ca22a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:04.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.989+0000 7f3f44a00640 1 --2- 172.21.15.31:0/2851661352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a57ca0dfd0 0x55a57ca16ea0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f3f30009f80 tx=0x7f3f30000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18cdb2b36de7dd7b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:04.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.990+0000 7f3f35200640 1 -- 172.21.15.31:0/2851661352 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3001f800 con 0x55a57ca0dfd0 2024-03-21T10:49:04.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.990+0000 7f3f35200640 1 -- 172.21.15.31:0/2851661352 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3f3000f830 con 0x55a57ca0dfd0 2024-03-21T10:49:04.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.990+0000 7f3f35200640 1 -- 172.21.15.31:0/2851661352 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f30029770 con 0x55a57ca0dfd0 2024-03-21T10:49:04.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.990+0000 7f3f4a22e740 1 -- 172.21.15.31:0/2851661352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a57ca0dfd0 msgr2=0x55a57ca16ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:04.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.990+0000 7f3f4a22e740 1 --2- 172.21.15.31:0/2851661352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a57ca0dfd0 0x55a57ca16ea0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f3f30009f80 tx=0x7f3f30000bc0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.990+0000 7f3f4a22e740 1 -- 172.21.15.31:0/2851661352 shutdown_connections 2024-03-21T10:49:04.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.990+0000 7f3f4a22e740 1 --2- 172.21.15.31:0/2851661352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a57ca176c0 0x55a57ca22a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.990+0000 7f3f4a22e740 1 --2- 172.21.15.31:0/2851661352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a57ca0e8f0 0x55a57ca13080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.990+0000 7f3f4a22e740 1 --2- 172.21.15.31:0/2851661352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a57ca0dfd0 0x55a57ca16ea0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.990+0000 7f3f4a22e740 1 -- 172.21.15.31:0/2851661352 >> 172.21.15.31:0/2851661352 conn(0x55a57ca088f0 msgr2=0x55a57ca08cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:04.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.991+0000 7f3f4a22e740 1 -- 172.21.15.31:0/2851661352 shutdown_connections 2024-03-21T10:49:04.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.991+0000 7f3f4a22e740 1 -- 172.21.15.31:0/2851661352 wait complete. 2024-03-21T10:49:04.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.991+0000 7f3f4a22e740 1 -- start start 2024-03-21T10:49:04.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.991+0000 7f3f4a22e740 1 Processor -- start 2024-03-21T10:49:04.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.992+0000 7f3f4a22e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a57ca0dfd0 0x55a57ca11020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:04.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.992+0000 7f3f4a22e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a57ca0e8f0 0x55a57ca11560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:04.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.992+0000 7f3f4a22e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a57ca176c0 0x55a57ca11aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:04.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.992+0000 7f3f4a22e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f401af2c0 con 0x55a57ca176c0 2024-03-21T10:49:04.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.992+0000 7f3f4a22e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f401b05a0 con 0x55a57ca0dfd0 2024-03-21T10:49:04.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.992+0000 7f3f4a22e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f401af8e0 con 0x55a57ca0e8f0 2024-03-21T10:49:04.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.992+0000 7f3f4a22e740 10 client.0 _ll_register_callbacks cb 0x7f3f30009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:04.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.992+0000 7f3f44a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a57ca0dfd0 0x55a57ca11020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:04.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.992+0000 7f3f37e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a57ca0e8f0 0x55a57ca11560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:04.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.992+0000 7f3f45400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a57ca176c0 0x55a57ca11aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:04.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.992+0000 7f3f44a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a57ca0dfd0 0x55a57ca11020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44320/0 (socket says 172.21.15.31:44320) 2024-03-21T10:49:04.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.992+0000 7f3f37e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a57ca0e8f0 0x55a57ca11560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34570/0 (socket says 172.21.15.31:34570) 2024-03-21T10:49:04.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.992+0000 7f3f44a00640 1 -- 172.21.15.31:0/339030588 learned_addr learned my addr 172.21.15.31:0/339030588 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:04.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.992+0000 7f3f37e00640 1 -- 172.21.15.31:0/339030588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a57ca0dfd0 msgr2=0x55a57ca11020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:04.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.992+0000 7f3f37e00640 1 --2- 172.21.15.31:0/339030588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a57ca0dfd0 0x55a57ca11020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.992+0000 7f3f37e00640 1 -- 172.21.15.31:0/339030588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a57ca176c0 msgr2=0x55a57ca11aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.992+0000 7f3f37e00640 1 --2- 172.21.15.31:0/339030588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a57ca176c0 0x55a57ca11aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.992+0000 7f3f37e00640 1 -- 172.21.15.31:0/339030588 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a57ca821c0 con 0x55a57ca0e8f0 2024-03-21T10:49:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.992+0000 7f3f45400640 1 --2- 172.21.15.31:0/339030588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a57ca176c0 0x55a57ca11aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.992+0000 7f3f44a00640 1 --2- 172.21.15.31:0/339030588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a57ca0dfd0 0x55a57ca11020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.993+0000 7f3f37e00640 1 --2- 172.21.15.31:0/339030588 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a57ca0e8f0 0x55a57ca11560 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f3f280ee240 tx=0x7f3f2802bc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f718d24b20d1d9a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.993+0000 7f3f2e000640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:49:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.993+0000 7f3f2e000640 1 -- 172.21.15.31:0/339030588 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f280097b0 con 0x55a57ca0e8f0 2024-03-21T10:49:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.993+0000 7f3f2e000640 1 -- 172.21.15.31:0/339030588 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f3f280dd0b0 con 0x55a57ca0e8f0 2024-03-21T10:49:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.993+0000 7f3f4a22e740 1 -- 172.21.15.31:0/339030588 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3f3000f830 con 0x55a57ca0e8f0 2024-03-21T10:49:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.993+0000 7f3f2e000640 1 -- 172.21.15.31:0/339030588 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f280bd5e0 con 0x55a57ca0e8f0 2024-03-21T10:49:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.993+0000 7f3f4a22e740 10 client.28177 fetch_fsmap learned FSMap version 29 2024-03-21T10:49:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.993+0000 7f3f4a22e740 1 -- 172.21.15.31:0/339030588 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55a57ca834d0 con 0x55a57ca0e8f0 2024-03-21T10:49:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.993+0000 7f3f2e000640 1 -- 172.21.15.31:0/339030588 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3f3000f830 con 0x55a57ca0e8f0 2024-03-21T10:49:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.993+0000 7f3f2e000640 1 -- 172.21.15.31:0/339030588 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f3f2803c030 con 0x55a57ca0e8f0 2024-03-21T10:49:04.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.994+0000 7f3f4a22e740 10 client.28177 fetch_fsmap finished waiting for FSMap version 29 2024-03-21T10:49:04.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.994+0000 7f3f4a22e740 10 client.28177 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:49:04.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.994+0000 7f3f4a22e740 20 client.28177 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:04.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.994+0000 7f3f4a22e740 1 --2- 172.21.15.31:0/339030588 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a57ca83be0 0x55a57ca86000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:04.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.994+0000 7f3f4a22e740 4 client.28177 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:49:04.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.994+0000 7f3f4a22e740 1 -- 172.21.15.31:0/339030588 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a57c97a0c0 con 0x55a57ca83be0 2024-03-21T10:49:04.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.994+0000 7f3f44a00640 1 --2- 172.21.15.31:0/339030588 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a57ca83be0 0x55a57ca86000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:04.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.995+0000 7f3f2e000640 10 client.28177 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:04.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:04.995+0000 7f3f44a00640 1 --2- 172.21.15.31:0/339030588 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a57ca83be0 0x55a57ca86000 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb2894a2bdcde1d2 server_cookie=ac2c86400d944c4c in_seq=0 out_seq=0 2024-03-21T10:49:04.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.000+0000 7f3f2e000640 1 -- 172.21.15.31:0/339030588 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a57c97a0c0 con 0x55a57ca83be0 2024-03-21T10:49:04.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.000+0000 7f3f2e000640 10 client.28177 handle_command_reply: tid=0 2024-03-21T10:49:04.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.000+0000 7f3f4a22e740 1 client.28177 shutdown 2024-03-21T10:49:04.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.000+0000 7f3f4a22e740 1 -- 172.21.15.31:0/339030588 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a57ca0e8f0 msgr2=0x55a57ca11560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:04.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.000+0000 7f3f4a22e740 1 --2- 172.21.15.31:0/339030588 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a57ca0e8f0 0x55a57ca11560 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f3f280ee240 tx=0x7f3f2802bc20 comp rx=0 tx=0).stop 2024-03-21T10:49:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.000+0000 7f3f4a22e740 1 -- 172.21.15.31:0/339030588 shutdown_connections 2024-03-21T10:49:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.000+0000 7f3f4a22e740 1 --2- 172.21.15.31:0/339030588 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a57ca83be0 0x55a57ca86000 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.000+0000 7f3f4a22e740 1 --2- 172.21.15.31:0/339030588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a57ca176c0 0x55a57ca11aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.000+0000 7f3f4a22e740 1 --2- 172.21.15.31:0/339030588 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a57ca0e8f0 0x55a57ca11560 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.000+0000 7f3f2e000640 0 client.28177 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.000+0000 7f3f4a22e740 1 --2- 172.21.15.31:0/339030588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a57ca0dfd0 0x55a57ca11020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.000+0000 7f3f4a22e740 1 -- 172.21.15.31:0/339030588 >> 172.21.15.31:0/339030588 conn(0x55a57ca088f0 msgr2=0x55a57ca12eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.001+0000 7f3f4a22e740 1 -- 172.21.15.31:0/339030588 shutdown_connections 2024-03-21T10:49:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.001+0000 7f3f4a22e740 1 -- 172.21.15.31:0/339030588 wait complete. 2024-03-21T10:49:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.001+0000 7f3f4a22e740 20 client.28177 trim_cache size 0 max 16384 2024-03-21T10:49:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:49:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:49:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:49:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:49:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.001+0000 7f3f47400640 1 -- 172.21.15.31:0/2452676978 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3f240a3010 msgr2=0x7f3f240a54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.001+0000 7f3f47400640 1 --2- 172.21.15.31:0/2452676978 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3f240a3010 0x7f3f240a54d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x55a57ca0da80 tx=0x7f3f30005f90 comp rx=0 tx=0).stop 2024-03-21T10:49:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.001+0000 7f3f47400640 1 -- 172.21.15.31:0/2452676978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f40104250 msgr2=0x7f3f4019a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:05.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.001+0000 7f3f47400640 1 --2- 172.21.15.31:0/2452676978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f40104250 0x7f3f4019a730 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f3f28037040 tx=0x7f3f28004360 comp rx=0 tx=0).stop 2024-03-21T10:49:05.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.002+0000 7f3f47400640 1 -- 172.21.15.31:0/2452676978 shutdown_connections 2024-03-21T10:49:05.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.002+0000 7f3f47400640 1 --2- 172.21.15.31:0/2452676978 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3f240a3010 0x7f3f240a54d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:05.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.002+0000 7f3f47400640 1 --2- 172.21.15.31:0/2452676978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f401098a0 0x7f3f4019ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:05.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.002+0000 7f3f47400640 1 --2- 172.21.15.31:0/2452676978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f40104250 0x7f3f4019a730 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:05.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.002+0000 7f3f47400640 1 --2- 172.21.15.31:0/2452676978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f40103890 0x7f3f4019a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:05.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.002+0000 7f3f47400640 1 -- 172.21.15.31:0/2452676978 >> 172.21.15.31:0/2452676978 conn(0x7f3f400ff320 msgr2=0x7f3f400ffb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:05.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.002+0000 7f3f47400640 1 -- 172.21.15.31:0/2452676978 shutdown_connections 2024-03-21T10:49:05.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:05.002+0000 7f3f47400640 1 -- 172.21.15.31:0/2452676978 wait complete. 2024-03-21T10:49:05.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:05 smithi070 ceph-mon[36330]: pgmap v326: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 19 op/s 2024-03-21T10:49:05.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:05 smithi040 ceph-mon[36154]: pgmap v326: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 19 op/s 2024-03-21T10:49:05.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:05 smithi031 ceph-mon[31673]: pgmap v326: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 19 op/s 2024-03-21T10:49:06.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:06 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:49:06.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:06 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:49:06.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:06 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:49:07.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:07 smithi070 ceph-mon[36330]: pgmap v327: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 46 KiB/s wr, 25 op/s 2024-03-21T10:49:07.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:07 smithi040 ceph-mon[36154]: pgmap v327: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 46 KiB/s wr, 25 op/s 2024-03-21T10:49:07.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:07 smithi031 ceph-mon[31673]: pgmap v327: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 46 KiB/s wr, 25 op/s 2024-03-21T10:49:08.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:08 smithi040 ceph-mon[36154]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:49:08.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:08 smithi040 ceph-mon[36154]: osdmap e103: 12 total, 12 up, 12 in 2024-03-21T10:49:08.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:08 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:49:08.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:08 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:49:08.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:08 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:49:08.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:08 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:49:08.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:08 smithi031 ceph-mon[31673]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:49:08.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:08 smithi031 ceph-mon[31673]: osdmap e103: 12 total, 12 up, 12 in 2024-03-21T10:49:08.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:08 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:49:08.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:08 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:49:08.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:08 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:49:08.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:08 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:49:08.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:08 smithi070 ceph-mon[36330]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:49:08.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:08 smithi070 ceph-mon[36330]: osdmap e103: 12 total, 12 up, 12 in 2024-03-21T10:49:08.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:08 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:49:08.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:08 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:49:08.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:08 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:49:08.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:08 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:49:09.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:09 smithi040 ceph-mon[36154]: pgmap v329: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-21T10:49:09.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:09 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:09 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:09 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:09 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:09 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:09 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:09 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:09 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:09 smithi040 ceph-mon[36154]: Adjusting osd_memory_target on smithi070 to 174.1M 2024-03-21T10:49:09.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:09 smithi040 ceph-mon[36154]: Unable to set osd_memory_target on smithi070 to 182619136: error parsing value: Value '182619136' is below minimum 939524096 2024-03-21T10:49:09.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:09 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:09 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:09 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:09 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:09 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:09 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:09 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.432 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:09 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:09 smithi031 ceph-mon[31673]: pgmap v329: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-21T10:49:09.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:09 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:09 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:09 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:09 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:09 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:09 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:09 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:09 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:09 smithi031 ceph-mon[31673]: Adjusting osd_memory_target on smithi070 to 174.1M 2024-03-21T10:49:09.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:09 smithi031 ceph-mon[31673]: Unable to set osd_memory_target on smithi070 to 182619136: error parsing value: Value '182619136' is below minimum 939524096 2024-03-21T10:49:09.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:09 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:09 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:09 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:09 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:09 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:09 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:09 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.556 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:09 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:09 smithi070 ceph-mon[36330]: pgmap v329: 129 pgs: 129 active+clean; 35 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-21T10:49:09.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:09 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:09 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:09 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:09 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:09 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:09 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:09 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:09 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:09 smithi070 ceph-mon[36330]: Adjusting osd_memory_target on smithi070 to 174.1M 2024-03-21T10:49:09.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:09 smithi070 ceph-mon[36330]: Unable to set osd_memory_target on smithi070 to 182619136: error parsing value: Value '182619136' is below minimum 939524096 2024-03-21T10:49:09.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:09 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:09 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:09 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:09 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:09 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:09 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:09 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:09.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:09 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:10.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:10 smithi040 ceph-mon[36154]: Adjusting osd_memory_target on smithi040 to 175.2M 2024-03-21T10:49:10.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:10 smithi040 ceph-mon[36154]: Unable to set osd_memory_target on smithi040 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-21T10:49:10.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:10 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:49:10.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:10 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:49:10.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:10 smithi031 ceph-mon[31673]: Adjusting osd_memory_target on smithi040 to 175.2M 2024-03-21T10:49:10.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:10 smithi031 ceph-mon[31673]: Unable to set osd_memory_target on smithi040 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-21T10:49:10.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:10 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:49:10.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:10 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:49:10.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:10 smithi070 ceph-mon[36330]: Adjusting osd_memory_target on smithi040 to 175.2M 2024-03-21T10:49:10.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:10 smithi070 ceph-mon[36330]: Unable to set osd_memory_target on smithi040 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-21T10:49:10.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:10 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:49:10.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:10 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:49:11.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:11 smithi040 ceph-mon[36154]: pgmap v330: 129 pgs: 15 active+clean+snaptrim, 43 active+clean+snaptrim_wait, 71 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 56 KiB/s wr, 27 op/s 2024-03-21T10:49:11.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:11 smithi031 ceph-mon[31673]: pgmap v330: 129 pgs: 15 active+clean+snaptrim, 43 active+clean+snaptrim_wait, 71 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 56 KiB/s wr, 27 op/s 2024-03-21T10:49:11.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:11 smithi070 ceph-mon[36330]: pgmap v330: 129 pgs: 15 active+clean+snaptrim, 43 active+clean+snaptrim_wait, 71 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 56 KiB/s wr, 27 op/s 2024-03-21T10:49:12.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:12 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:12.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:12 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:12.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:12 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:12.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:12 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:12.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:12 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:12.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:12 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:12.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:12 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:12.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:12 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:12.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:12 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:49:12.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:12 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:49:12.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:12 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:49:12.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:12 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:12.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:12 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:12.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:12 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:12.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:12 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:12.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:12 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:12.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:12 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:12.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:12 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:12.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:12 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:12.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:12 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:49:12.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:12 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:49:12.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:12 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:49:12.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:12 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:12.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:12 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:12.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:12 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:12.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:12 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:12.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:12 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:12.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:12 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:12.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:12 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:12.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:12 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-21T10:49:12.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:12 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:49:12.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:12 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:49:12.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:12 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:49:12.624 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:49:12.624 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:49:12.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.867+0000 7f2428200640 1 -- 172.21.15.31:0/341618685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2420105380 msgr2=0x7f24201057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:12.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.867+0000 7f2428200640 1 --2- 172.21.15.31:0/341618685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2420105380 0x7f24201057e0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f240c009930 tx=0x7f240c02bd10 comp rx=0 tx=0).stop 2024-03-21T10:49:12.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.868+0000 7f2428200640 1 -- 172.21.15.31:0/341618685 shutdown_connections 2024-03-21T10:49:12.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.868+0000 7f2428200640 1 --2- 172.21.15.31:0/341618685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2420105d20 0x7f2420110850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.868+0000 7f2428200640 1 --2- 172.21.15.31:0/341618685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2420105380 0x7f24201057e0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.868+0000 7f2428200640 1 --2- 172.21.15.31:0/341618685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2420103c40 0x7f2420104020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.868+0000 7f2428200640 1 -- 172.21.15.31:0/341618685 >> 172.21.15.31:0/341618685 conn(0x7f24200ff350 msgr2=0x7f2420101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:12.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.868+0000 7f2428200640 1 -- 172.21.15.31:0/341618685 shutdown_connections 2024-03-21T10:49:12.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.868+0000 7f2428200640 1 -- 172.21.15.31:0/341618685 wait complete. 2024-03-21T10:49:12.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.869+0000 7f2428200640 1 Processor -- start 2024-03-21T10:49:12.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.869+0000 7f2428200640 1 -- start start 2024-03-21T10:49:12.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.869+0000 7f2428200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2420103c40 0x7f242019a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:12.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.869+0000 7f2428200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2420105380 0x7f242019a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:12.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.869+0000 7f2428200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2420105d20 0x7f242019eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.869+0000 7f2428200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2420113db0 con 0x7f2420105d20 2024-03-21T10:49:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.869+0000 7f2428200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2420113c30 con 0x7f2420105380 2024-03-21T10:49:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.869+0000 7f2428200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2420113f30 con 0x7f2420103c40 2024-03-21T10:49:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.869+0000 7f2425800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2420103c40 0x7f242019a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.869+0000 7f2424e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2420105380 0x7f242019a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.869+0000 7f2425800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2420103c40 0x7f242019a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:52976/0 (socket says 172.21.15.31:52976) 2024-03-21T10:49:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.869+0000 7f2426200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2420105d20 0x7f242019eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.869+0000 7f2425800640 1 -- 172.21.15.31:0/602267381 learned_addr learned my addr 172.21.15.31:0/602267381 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.870+0000 7f2424e00640 1 -- 172.21.15.31:0/602267381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2420103c40 msgr2=0x7f242019a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.870+0000 7f2424e00640 1 --2- 172.21.15.31:0/602267381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2420103c40 0x7f242019a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.870+0000 7f2424e00640 1 -- 172.21.15.31:0/602267381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2420105d20 msgr2=0x7f242019eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.870+0000 7f2424e00640 1 --2- 172.21.15.31:0/602267381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2420105d20 0x7f242019eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.870+0000 7f2424e00640 1 -- 172.21.15.31:0/602267381 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f242019f210 con 0x7f2420105380 2024-03-21T10:49:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.870+0000 7f2424e00640 1 --2- 172.21.15.31:0/602267381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2420105380 0x7f242019a7b0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f240c036040 tx=0x7f240c0042d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:12.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.870+0000 7f241e000640 1 -- 172.21.15.31:0/602267381 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c042020 con 0x7f2420105380 2024-03-21T10:49:12.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.870+0000 7f241e000640 1 -- 172.21.15.31:0/602267381 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f240c0044c0 con 0x7f2420105380 2024-03-21T10:49:12.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.870+0000 7f2428200640 1 -- 172.21.15.31:0/602267381 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f242019f4a0 con 0x7f2420105380 2024-03-21T10:49:12.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.871+0000 7f241e000640 1 -- 172.21.15.31:0/602267381 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c003950 con 0x7f2420105380 2024-03-21T10:49:12.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.871+0000 7f2428200640 1 -- 172.21.15.31:0/602267381 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24201a6d90 con 0x7f2420105380 2024-03-21T10:49:12.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.874+0000 7f241e000640 1 -- 172.21.15.31:0/602267381 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f240c045020 con 0x7f2420105380 2024-03-21T10:49:12.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.874+0000 7f242af68740 1 Processor -- start 2024-03-21T10:49:12.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.874+0000 7f242af68740 1 -- start start 2024-03-21T10:49:12.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.874+0000 7f242af68740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b4b5c0c00 0x564b4b5bf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:12.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.874+0000 7f242af68740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b4b5bff20 0x564b4b5c03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:12.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.874+0000 7f242af68740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b4b5c9290 0x564b4b5cb680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:12.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.874+0000 7f242af68740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b4b524250 con 0x564b4b5c0c00 2024-03-21T10:49:12.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.874+0000 7f242af68740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b4b4cca30 con 0x564b4b5bff20 2024-03-21T10:49:12.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.874+0000 7f242af68740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b4b5202f0 con 0x564b4b5c9290 2024-03-21T10:49:12.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.875+0000 7f2425800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b4b5bff20 0x564b4b5c03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:12.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.875+0000 7f2426200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b4b5c0c00 0x564b4b5bf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:12.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.875+0000 7f2425800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b4b5bff20 0x564b4b5c03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39218/0 (socket says 172.21.15.31:39218) 2024-03-21T10:49:12.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.875+0000 7f2425800640 1 -- 172.21.15.31:0/2045500360 learned_addr learned my addr 172.21.15.31:0/2045500360 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:12.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.875+0000 7f2424e00640 1 --2- 172.21.15.31:0/2045500360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b4b5c9290 0x564b4b5cb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:12.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.875+0000 7f2425800640 1 -- 172.21.15.31:0/2045500360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b4b5c9290 msgr2=0x564b4b5cb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:12.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.875+0000 7f2425800640 1 --2- 172.21.15.31:0/2045500360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b4b5c9290 0x564b4b5cb680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.875+0000 7f2425800640 1 -- 172.21.15.31:0/2045500360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b4b5c0c00 msgr2=0x564b4b5bf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:12.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.875+0000 7f2425800640 1 --2- 172.21.15.31:0/2045500360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b4b5c0c00 0x564b4b5bf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.875+0000 7f2425800640 1 -- 172.21.15.31:0/2045500360 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564b4b4b7690 con 0x564b4b5bff20 2024-03-21T10:49:12.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.875+0000 7f2424e00640 1 --2- 172.21.15.31:0/2045500360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b4b5c9290 0x564b4b5cb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:49:12.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.876+0000 7f2425800640 1 --2- 172.21.15.31:0/2045500360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b4b5bff20 0x564b4b5c03a0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f2414009dc0 tx=0x7f2414011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e68b1eeed50ebb0e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:12.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.876+0000 7f241e000640 1 --2- 172.21.15.31:0/602267381 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24080a3090 0x7f24080a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:12.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.876+0000 7f241e000640 1 -- 172.21.15.31:0/602267381 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f240c0f15a0 con 0x7f2420105380 2024-03-21T10:49:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.876+0000 7f2412a00640 1 -- 172.21.15.31:0/2045500360 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2414005220 con 0x564b4b5bff20 2024-03-21T10:49:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.876+0000 7f2412a00640 1 -- 172.21.15.31:0/2045500360 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2414004c80 con 0x564b4b5bff20 2024-03-21T10:49:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.876+0000 7f2412a00640 1 -- 172.21.15.31:0/2045500360 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2414019970 con 0x564b4b5bff20 2024-03-21T10:49:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.876+0000 7f2426200640 1 --2- 172.21.15.31:0/602267381 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24080a3090 0x7f24080a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.876+0000 7f242af68740 1 -- 172.21.15.31:0/2045500360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b4b5bff20 msgr2=0x564b4b5c03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.876+0000 7f242af68740 1 --2- 172.21.15.31:0/2045500360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b4b5bff20 0x564b4b5c03a0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f2414009dc0 tx=0x7f2414011810 comp rx=0 tx=0).stop 2024-03-21T10:49:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.877+0000 7f242af68740 1 -- 172.21.15.31:0/2045500360 shutdown_connections 2024-03-21T10:49:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.877+0000 7f242af68740 1 --2- 172.21.15.31:0/2045500360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b4b5c9290 0x564b4b5cb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.877+0000 7f242af68740 1 --2- 172.21.15.31:0/2045500360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b4b5bff20 0x564b4b5c03a0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.877+0000 7f242af68740 1 --2- 172.21.15.31:0/2045500360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b4b5c0c00 0x564b4b5bf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.877+0000 7f2426200640 1 --2- 172.21.15.31:0/602267381 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24080a3090 0x7f24080a5550 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2418005fd0 tx=0x7f2418009500 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.877+0000 7f242af68740 1 -- 172.21.15.31:0/2045500360 >> 172.21.15.31:0/2045500360 conn(0x564b4b587200 msgr2=0x564b4b5ba970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.877+0000 7f242af68740 1 -- 172.21.15.31:0/2045500360 shutdown_connections 2024-03-21T10:49:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.877+0000 7f242af68740 1 -- 172.21.15.31:0/2045500360 wait complete. 2024-03-21T10:49:12.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.877+0000 7f242af68740 1 -- start start 2024-03-21T10:49:12.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.878+0000 7f242af68740 1 Processor -- start 2024-03-21T10:49:12.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.878+0000 7f242af68740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b4b5bff20 0x564b4b5c8930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:12.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.878+0000 7f242af68740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b4b5c0c00 0x564b4b5c5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:12.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.879+0000 7f242af68740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b4b5c9290 0x564b4b5c6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:12.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.879+0000 7f242af68740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b4b5cc100 con 0x564b4b5c9290 2024-03-21T10:49:12.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.879+0000 7f242af68740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b4b4dfe80 con 0x564b4b5c0c00 2024-03-21T10:49:12.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.879+0000 7f242af68740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b4b5cc280 con 0x564b4b5bff20 2024-03-21T10:49:12.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.879+0000 7f242af68740 10 client.0 _ll_register_callbacks cb 0x564b4b4f2130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:12.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.879+0000 7f2424e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b4b5c0c00 0x564b4b5c5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:12.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.879+0000 7f2425800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b4b5bff20 0x564b4b5c8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:12.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.879+0000 7f2424e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b4b5c0c00 0x564b4b5c5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39220/0 (socket says 172.21.15.31:39220) 2024-03-21T10:49:12.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.879+0000 7f2424e00640 1 -- 172.21.15.31:0/121645960 learned_addr learned my addr 172.21.15.31:0/121645960 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:12.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.879+0000 7f2426200640 1 --2- 172.21.15.31:0/121645960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b4b5c9290 0x564b4b5c6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:12.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.879+0000 7f2425800640 1 -- 172.21.15.31:0/121645960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b4b5c0c00 msgr2=0x564b4b5c5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:12.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.879+0000 7f2425800640 1 --2- 172.21.15.31:0/121645960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b4b5c0c00 0x564b4b5c5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.879+0000 7f2425800640 1 -- 172.21.15.31:0/121645960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b4b5c9290 msgr2=0x564b4b5c6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:12.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.879+0000 7f2425800640 1 --2- 172.21.15.31:0/121645960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b4b5c9290 0x564b4b5c6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.879+0000 7f2425800640 1 -- 172.21.15.31:0/121645960 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564b4b3027d0 con 0x564b4b5bff20 2024-03-21T10:49:12.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.879+0000 7f2426200640 1 --2- 172.21.15.31:0/121645960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b4b5c9290 0x564b4b5c6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:49:12.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.879+0000 7f2424e00640 1 --2- 172.21.15.31:0/121645960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b4b5c0c00 0x564b4b5c5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:12.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.880+0000 7f2425800640 1 --2- 172.21.15.31:0/121645960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b4b5bff20 0x564b4b5c8930 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f2414019470 tx=0x7f2414004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66e549c6930998ab server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:12.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.880+0000 7f2407400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:49:12.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.880+0000 7f2407400640 1 -- 172.21.15.31:0/121645960 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2414005030 con 0x564b4b5bff20 2024-03-21T10:49:12.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.880+0000 7f2407400640 1 -- 172.21.15.31:0/121645960 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f24140051d0 con 0x564b4b5bff20 2024-03-21T10:49:12.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.880+0000 7f242af68740 1 -- 172.21.15.31:0/121645960 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2414004c80 con 0x564b4b5bff20 2024-03-21T10:49:12.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.880+0000 7f2407400640 1 -- 172.21.15.31:0/121645960 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2414032050 con 0x564b4b5bff20 2024-03-21T10:49:12.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.880+0000 7f242af68740 10 client.28189 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:12.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.880+0000 7f242af68740 1 -- 172.21.15.31:0/121645960 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x564b4b2383f0 con 0x564b4b5bff20 2024-03-21T10:49:12.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.880+0000 7f2407400640 1 -- 172.21.15.31:0/121645960 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2414004c80 con 0x564b4b5bff20 2024-03-21T10:49:12.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.880+0000 7f2407400640 1 -- 172.21.15.31:0/121645960 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f2414003680 con 0x564b4b5bff20 2024-03-21T10:49:12.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.881+0000 7f242af68740 10 client.28189 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:12.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.881+0000 7f242af68740 10 client.28189 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:49:12.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.881+0000 7f242af68740 20 client.28189 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:12.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.881+0000 7f242af68740 1 --2- 172.21.15.31:0/121645960 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564b4b5d9070 0x564b4b5db460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:12.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.881+0000 7f242af68740 4 client.28189 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:49:12.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.881+0000 7f242af68740 1 -- 172.21.15.31:0/121645960 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564b4b531790 con 0x564b4b5d9070 2024-03-21T10:49:12.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.881+0000 7f2424e00640 1 --2- 172.21.15.31:0/121645960 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564b4b5d9070 0x564b4b5db460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:12.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.881+0000 7f2424e00640 1 --2- 172.21.15.31:0/121645960 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564b4b5d9070 0x564b4b5db460 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=665759be482ddc24 server_cookie=f7789184bb54ef6d in_seq=0 out_seq=0 2024-03-21T10:49:12.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.881+0000 7f2407400640 10 client.28189 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:12.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.882+0000 7f2407400640 1 -- 172.21.15.31:0/121645960 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564b4b531790 con 0x564b4b5d9070 2024-03-21T10:49:12.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.882+0000 7f2407400640 10 client.28189 handle_command_reply: tid=0 2024-03-21T10:49:12.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.882+0000 7f242af68740 1 client.28189 shutdown 2024-03-21T10:49:12.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.883+0000 7f242af68740 1 -- 172.21.15.31:0/121645960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b4b5bff20 msgr2=0x564b4b5c8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:12.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.883+0000 7f242af68740 1 --2- 172.21.15.31:0/121645960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b4b5bff20 0x564b4b5c8930 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f2414019470 tx=0x7f2414004270 comp rx=0 tx=0).stop 2024-03-21T10:49:12.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.883+0000 7f242af68740 1 -- 172.21.15.31:0/121645960 shutdown_connections 2024-03-21T10:49:12.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.883+0000 7f242af68740 1 --2- 172.21.15.31:0/121645960 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564b4b5d9070 0x564b4b5db460 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.883+0000 7f242af68740 1 --2- 172.21.15.31:0/121645960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b4b5c9290 0x564b4b5c6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.883+0000 7f242af68740 1 --2- 172.21.15.31:0/121645960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b4b5c0c00 0x564b4b5c5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.883+0000 7f2407400640 0 client.28189 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:12.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.883+0000 7f242af68740 1 --2- 172.21.15.31:0/121645960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b4b5bff20 0x564b4b5c8930 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.883+0000 7f242af68740 1 -- 172.21.15.31:0/121645960 >> 172.21.15.31:0/121645960 conn(0x564b4b587200 msgr2=0x564b4b5bc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:12.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.883+0000 7f242af68740 1 -- 172.21.15.31:0/121645960 shutdown_connections 2024-03-21T10:49:12.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.883+0000 7f242af68740 1 -- 172.21.15.31:0/121645960 wait complete. 2024-03-21T10:49:12.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.883+0000 7f242af68740 20 client.28189 trim_cache size 0 max 16384 2024-03-21T10:49:12.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.910+0000 7f242af68740 1 Processor -- start 2024-03-21T10:49:12.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.910+0000 7f242af68740 1 -- start start 2024-03-21T10:49:12.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.910+0000 7f242af68740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b4b5bff20 0x564b4b5c5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:12.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.910+0000 7f242af68740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b4b5c0c00 0x564b4b5bc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:12.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.910+0000 7f242af68740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b4b5c9290 0x564b4b5ca680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:12.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.910+0000 7f242af68740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24201b30e0 con 0x564b4b5bff20 2024-03-21T10:49:12.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.910+0000 7f242af68740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24201b0be0 con 0x564b4b5c0c00 2024-03-21T10:49:12.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.910+0000 7f242af68740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24201aeca0 con 0x564b4b5c9290 2024-03-21T10:49:12.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.910+0000 7f2426200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b4b5c9290 0x564b4b5ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:12.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.910+0000 7f2424e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b4b5c0c00 0x564b4b5bc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:12.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.910+0000 7f2426200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b4b5c9290 0x564b4b5ca680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53002/0 (socket says 172.21.15.31:53002) 2024-03-21T10:49:12.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.910+0000 7f2424e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b4b5c0c00 0x564b4b5bc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39232/0 (socket says 172.21.15.31:39232) 2024-03-21T10:49:12.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.910+0000 7f2425800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b4b5bff20 0x564b4b5c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:12.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.910+0000 7f2426200640 1 -- 172.21.15.31:0/2193649215 learned_addr learned my addr 172.21.15.31:0/2193649215 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:12.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.911+0000 7f2424e00640 1 -- 172.21.15.31:0/2193649215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b4b5c9290 msgr2=0x564b4b5ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:12.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.911+0000 7f2424e00640 1 --2- 172.21.15.31:0/2193649215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b4b5c9290 0x564b4b5ca680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.911+0000 7f2424e00640 1 -- 172.21.15.31:0/2193649215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b4b5bff20 msgr2=0x564b4b5c5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:12.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.911+0000 7f2424e00640 1 --2- 172.21.15.31:0/2193649215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b4b5bff20 0x564b4b5c5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.911+0000 7f2424e00640 1 -- 172.21.15.31:0/2193649215 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564b4b1f80f0 con 0x564b4b5c0c00 2024-03-21T10:49:12.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.911+0000 7f2426200640 1 --2- 172.21.15.31:0/2193649215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b4b5c9290 0x564b4b5ca680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:12.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.911+0000 7f2424e00640 1 --2- 172.21.15.31:0/2193649215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b4b5c0c00 0x564b4b5bc4d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f240c006010 tx=0x7f240c02bd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=567f220e0f2e615e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:12.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.911+0000 7f2412000640 1 -- 172.21.15.31:0/2193649215 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c0b98f0 con 0x564b4b5c0c00 2024-03-21T10:49:12.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.911+0000 7f2412000640 1 -- 172.21.15.31:0/2193649215 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f240c0bdd50 con 0x564b4b5c0c00 2024-03-21T10:49:12.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.911+0000 7f2412000640 1 -- 172.21.15.31:0/2193649215 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c0bcfb0 con 0x564b4b5c0c00 2024-03-21T10:49:12.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.912+0000 7f242af68740 1 -- 172.21.15.31:0/2193649215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b4b5c0c00 msgr2=0x564b4b5bc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:12.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.912+0000 7f242af68740 1 --2- 172.21.15.31:0/2193649215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b4b5c0c00 0x564b4b5bc4d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f240c006010 tx=0x7f240c02bd10 comp rx=0 tx=0).stop 2024-03-21T10:49:12.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.912+0000 7f242af68740 1 -- 172.21.15.31:0/2193649215 shutdown_connections 2024-03-21T10:49:12.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.912+0000 7f242af68740 1 --2- 172.21.15.31:0/2193649215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b4b5c9290 0x564b4b5ca680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.912+0000 7f242af68740 1 --2- 172.21.15.31:0/2193649215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b4b5c0c00 0x564b4b5bc4d0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.912+0000 7f242af68740 1 --2- 172.21.15.31:0/2193649215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b4b5bff20 0x564b4b5c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.912+0000 7f242af68740 1 -- 172.21.15.31:0/2193649215 >> 172.21.15.31:0/2193649215 conn(0x564b4b587200 msgr2=0x564b4b5bbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:12.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.912+0000 7f242af68740 1 -- 172.21.15.31:0/2193649215 shutdown_connections 2024-03-21T10:49:12.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.912+0000 7f242af68740 1 -- 172.21.15.31:0/2193649215 wait complete. 2024-03-21T10:49:12.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.913+0000 7f242af68740 1 -- start start 2024-03-21T10:49:12.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.913+0000 7f242af68740 1 Processor -- start 2024-03-21T10:49:12.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.913+0000 7f242af68740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b4b5bff20 0x564b4b5c2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:12.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.913+0000 7f242af68740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b4b5c0c00 0x564b4b5c2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:12.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.913+0000 7f242af68740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b4b5c9290 0x564b4b5c2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:12.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.913+0000 7f242af68740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24201af2a0 con 0x564b4b5c9290 2024-03-21T10:49:12.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.913+0000 7f242af68740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24201b0580 con 0x564b4b5c0c00 2024-03-21T10:49:12.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.913+0000 7f242af68740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24201af8c0 con 0x564b4b5bff20 2024-03-21T10:49:12.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.913+0000 7f242af68740 10 client.0 _ll_register_callbacks cb 0x7f2418009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:12.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.913+0000 7f2425800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b4b5bff20 0x564b4b5c2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:12.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.914+0000 7f2425800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b4b5bff20 0x564b4b5c2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53004/0 (socket says 172.21.15.31:53004) 2024-03-21T10:49:12.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.914+0000 7f2425800640 1 -- 172.21.15.31:0/2849503564 learned_addr learned my addr 172.21.15.31:0/2849503564 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:12.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.914+0000 7f2424e00640 1 --2- 172.21.15.31:0/2849503564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b4b5c0c00 0x564b4b5c2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:12.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.914+0000 7f2426200640 1 --2- 172.21.15.31:0/2849503564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b4b5c9290 0x564b4b5c2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:12.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.914+0000 7f2425800640 1 -- 172.21.15.31:0/2849503564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b4b5c0c00 msgr2=0x564b4b5c2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:12.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.914+0000 7f2425800640 1 --2- 172.21.15.31:0/2849503564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b4b5c0c00 0x564b4b5c2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.914+0000 7f2425800640 1 -- 172.21.15.31:0/2849503564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b4b5c9290 msgr2=0x564b4b5c2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:12.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.914+0000 7f2425800640 1 --2- 172.21.15.31:0/2849503564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b4b5c9290 0x564b4b5c2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.914+0000 7f2425800640 1 -- 172.21.15.31:0/2849503564 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564b4b6336e0 con 0x564b4b5bff20 2024-03-21T10:49:12.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.914+0000 7f2425800640 1 --2- 172.21.15.31:0/2849503564 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b4b5bff20 0x564b4b5c2570 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f2414003a80 tx=0x7f2414009a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3af17a509b90a32 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:12.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.914+0000 7f2407400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:49:12.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.914+0000 7f2407400640 1 -- 172.21.15.31:0/2849503564 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241403b060 con 0x564b4b5bff20 2024-03-21T10:49:12.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.914+0000 7f2407400640 1 -- 172.21.15.31:0/2849503564 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f2414019880 con 0x564b4b5bff20 2024-03-21T10:49:12.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.914+0000 7f242af68740 1 -- 172.21.15.31:0/2849503564 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f240c0bdd50 con 0x564b4b5bff20 2024-03-21T10:49:12.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.915+0000 7f2407400640 1 -- 172.21.15.31:0/2849503564 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2414042050 con 0x564b4b5bff20 2024-03-21T10:49:12.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.915+0000 7f242af68740 10 client.28195 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:12.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.915+0000 7f242af68740 1 -- 172.21.15.31:0/2849503564 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x564b4b6349f0 con 0x564b4b5bff20 2024-03-21T10:49:12.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.915+0000 7f2407400640 1 -- 172.21.15.31:0/2849503564 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f240c0bdd50 con 0x564b4b5bff20 2024-03-21T10:49:12.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.915+0000 7f2407400640 1 -- 172.21.15.31:0/2849503564 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f2414004d10 con 0x564b4b5bff20 2024-03-21T10:49:12.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.915+0000 7f242af68740 10 client.28195 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:12.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.915+0000 7f242af68740 10 client.28195 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:49:12.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.915+0000 7f242af68740 20 client.28195 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:12.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.915+0000 7f242af68740 1 --2- 172.21.15.31:0/2849503564 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564b4b635000 0x564b4b637420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:12.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.915+0000 7f242af68740 4 client.28195 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:49:12.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.915+0000 7f242af68740 1 -- 172.21.15.31:0/2849503564 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564b4b52bae0 con 0x564b4b635000 2024-03-21T10:49:12.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.916+0000 7f2424e00640 1 --2- 172.21.15.31:0/2849503564 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564b4b635000 0x564b4b637420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:12.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.916+0000 7f2424e00640 1 --2- 172.21.15.31:0/2849503564 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564b4b635000 0x564b4b637420 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35a4adbf4683f22e server_cookie=10ce858f0a0b1b3a in_seq=0 out_seq=0 2024-03-21T10:49:12.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.916+0000 7f2407400640 10 client.28195 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:12.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.916+0000 7f2407400640 1 -- 172.21.15.31:0/2849503564 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x564b4b52bae0 con 0x564b4b635000 2024-03-21T10:49:12.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.916+0000 7f2407400640 10 client.28195 handle_command_reply: tid=0 2024-03-21T10:49:12.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.916+0000 7f242af68740 1 client.28195 shutdown 2024-03-21T10:49:12.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.916+0000 7f242af68740 1 -- 172.21.15.31:0/2849503564 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b4b5bff20 msgr2=0x564b4b5c2570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:12.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.916+0000 7f242af68740 1 --2- 172.21.15.31:0/2849503564 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b4b5bff20 0x564b4b5c2570 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f2414003a80 tx=0x7f2414009a90 comp rx=0 tx=0).stop 2024-03-21T10:49:12.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.916+0000 7f242af68740 1 -- 172.21.15.31:0/2849503564 shutdown_connections 2024-03-21T10:49:12.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.916+0000 7f242af68740 1 --2- 172.21.15.31:0/2849503564 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x564b4b635000 0x564b4b637420 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.916+0000 7f242af68740 1 --2- 172.21.15.31:0/2849503564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b4b5c9290 0x564b4b5c2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.916+0000 7f242af68740 1 --2- 172.21.15.31:0/2849503564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b4b5c0c00 0x564b4b5c2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.916+0000 7f242af68740 1 --2- 172.21.15.31:0/2849503564 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564b4b5bff20 0x564b4b5c2570 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.916+0000 7f2407400640 0 client.28195 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:12.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.916+0000 7f242af68740 1 -- 172.21.15.31:0/2849503564 >> 172.21.15.31:0/2849503564 conn(0x564b4b587200 msgr2=0x564b4b5b9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:12.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.917+0000 7f242af68740 1 -- 172.21.15.31:0/2849503564 shutdown_connections 2024-03-21T10:49:12.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.917+0000 7f242af68740 1 -- 172.21.15.31:0/2849503564 wait complete. 2024-03-21T10:49:12.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.917+0000 7f242af68740 20 client.28195 trim_cache size 0 max 16384 2024-03-21T10:49:12.923 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:49:12.923 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-03-21T10:49:12.923 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-03-21T10:49:12.923 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:49:12.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.917+0000 7f2428200640 1 -- 172.21.15.31:0/602267381 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24080a3090 msgr2=0x7f24080a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:12.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.917+0000 7f2428200640 1 --2- 172.21.15.31:0/602267381 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24080a3090 0x7f24080a5550 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2418005fd0 tx=0x7f2418009500 comp rx=0 tx=0).stop 2024-03-21T10:49:12.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.917+0000 7f2428200640 1 -- 172.21.15.31:0/602267381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2420105380 msgr2=0x7f242019a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:12.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.917+0000 7f2428200640 1 --2- 172.21.15.31:0/602267381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2420105380 0x7f242019a7b0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f240c036040 tx=0x7f240c0042d0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.917+0000 7f2428200640 1 -- 172.21.15.31:0/602267381 shutdown_connections 2024-03-21T10:49:12.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.917+0000 7f2428200640 1 --2- 172.21.15.31:0/602267381 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24080a3090 0x7f24080a5550 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.917+0000 7f2428200640 1 --2- 172.21.15.31:0/602267381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2420105d20 0x7f242019eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.917+0000 7f2428200640 1 --2- 172.21.15.31:0/602267381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2420105380 0x7f242019a7b0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.917+0000 7f2428200640 1 --2- 172.21.15.31:0/602267381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2420103c40 0x7f242019a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:12.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.918+0000 7f2428200640 1 -- 172.21.15.31:0/602267381 >> 172.21.15.31:0/602267381 conn(0x7f24200ff350 msgr2=0x7f2420107770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:12.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.918+0000 7f2428200640 1 -- 172.21.15.31:0/602267381 shutdown_connections 2024-03-21T10:49:12.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:12.918+0000 7f2428200640 1 -- 172.21.15.31:0/602267381 wait complete. 2024-03-21T10:49:12.932 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-21T10:49:12.932 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:49:13.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.133+0000 7f3dffe00640 1 -- 172.21.15.31:0/4144185851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e00109750 msgr2=0x7f3e00109b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:13.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.133+0000 7f3dffe00640 1 --2- 172.21.15.31:0/4144185851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e00109750 0x7f3e00109b30 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f3df0009900 tx=0x7f3df002bc30 comp rx=0 tx=0).stop 2024-03-21T10:49:13.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.133+0000 7f3dffe00640 1 -- 172.21.15.31:0/4144185851 shutdown_connections 2024-03-21T10:49:13.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.133+0000 7f3dffe00640 1 --2- 172.21.15.31:0/4144185851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e00104100 0x7f3e00110610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.133+0000 7f3dffe00640 1 --2- 172.21.15.31:0/4144185851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e00103740 0x7f3e00103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.133+0000 7f3dffe00640 1 --2- 172.21.15.31:0/4144185851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e00109750 0x7f3e00109b30 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.133+0000 7f3dffe00640 1 -- 172.21.15.31:0/4144185851 >> 172.21.15.31:0/4144185851 conn(0x7f3e000ff210 msgr2=0x7f3e00101630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:13.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.134+0000 7f3dffe00640 1 -- 172.21.15.31:0/4144185851 shutdown_connections 2024-03-21T10:49:13.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.134+0000 7f3dffe00640 1 -- 172.21.15.31:0/4144185851 wait complete. 2024-03-21T10:49:13.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.135+0000 7f3dffe00640 1 Processor -- start 2024-03-21T10:49:13.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.135+0000 7f3dffe00640 1 -- start start 2024-03-21T10:49:13.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.135+0000 7f3dffe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e00103740 0x7f3e001104f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.135+0000 7f3dffe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e00104100 0x7f3e0010d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.135+0000 7f3dffe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e00109750 0x7f3e0010db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.135+0000 7f3dffe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e00113b60 con 0x7f3e00103740 2024-03-21T10:49:13.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.135+0000 7f3dffe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e001139e0 con 0x7f3e00104100 2024-03-21T10:49:13.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.135+0000 7f3dffe00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e00113ce0 con 0x7f3e00109750 2024-03-21T10:49:13.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.136+0000 7f3df6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e00104100 0x7f3e0010d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.136+0000 7f3dff400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e00109750 0x7f3e0010db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.136+0000 7f3df6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e00104100 0x7f3e0010d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39262/0 (socket says 172.21.15.31:39262) 2024-03-21T10:49:13.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.136+0000 7f3df6000640 1 -- 172.21.15.31:0/1963094256 learned_addr learned my addr 172.21.15.31:0/1963094256 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:13.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.136+0000 7f3dfea00640 1 --2- 172.21.15.31:0/1963094256 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e00103740 0x7f3e001104f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.136+0000 7f3dff400640 1 -- 172.21.15.31:0/1963094256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e00104100 msgr2=0x7f3e0010d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:13.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.136+0000 7f3dff400640 1 --2- 172.21.15.31:0/1963094256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e00104100 0x7f3e0010d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.136+0000 7f3dff400640 1 -- 172.21.15.31:0/1963094256 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e00103740 msgr2=0x7f3e001104f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:13.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.136+0000 7f3dff400640 1 --2- 172.21.15.31:0/1963094256 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e00103740 0x7f3e001104f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.136+0000 7f3dff400640 1 -- 172.21.15.31:0/1963094256 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e0010e3d0 con 0x7f3e00109750 2024-03-21T10:49:13.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.136+0000 7f3df6000640 1 --2- 172.21.15.31:0/1963094256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e00104100 0x7f3e0010d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:13.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.137+0000 7f3dff400640 1 --2- 172.21.15.31:0/1963094256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e00109750 0x7f3e0010db10 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f3df8009fd0 tx=0x7f3df800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:13.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.137+0000 7f3e04a00640 1 -- 172.21.15.31:0/1963094256 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df800ce20 con 0x7f3e00109750 2024-03-21T10:49:13.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.137+0000 7f3e04a00640 1 -- 172.21.15.31:0/1963094256 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3df8005020 con 0x7f3e00109750 2024-03-21T10:49:13.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.137+0000 7f3e04a00640 1 -- 172.21.15.31:0/1963094256 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df8010720 con 0x7f3e00109750 2024-03-21T10:49:13.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.137+0000 7f3dffe00640 1 -- 172.21.15.31:0/1963094256 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e001aa960 con 0x7f3e00109750 2024-03-21T10:49:13.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.137+0000 7f3dffe00640 1 -- 172.21.15.31:0/1963094256 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e001aae60 con 0x7f3e00109750 2024-03-21T10:49:13.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.139+0000 7f3dffe00640 1 -- 172.21.15.31:0/1963094256 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e00104e80 con 0x7f3e00109750 2024-03-21T10:49:13.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.142+0000 7f3e04a00640 1 -- 172.21.15.31:0/1963094256 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f3df8005540 con 0x7f3e00109750 2024-03-21T10:49:13.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.144+0000 7f3e04a00640 1 --2- 172.21.15.31:0/1963094256 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3de40a3100 0x7f3de40a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.144+0000 7f3e04a00640 1 -- 172.21.15.31:0/1963094256 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f3df80d0150 con 0x7f3e00109750 2024-03-21T10:49:13.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.144+0000 7f3dfea00640 1 --2- 172.21.15.31:0/1963094256 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3de40a3100 0x7f3de40a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.145+0000 7f3dfea00640 1 --2- 172.21.15.31:0/1963094256 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3de40a3100 0x7f3de40a55c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3df0009230 tx=0x7f3df0005fd0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:13.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.148+0000 7f3e04a00640 1 -- 172.21.15.31:0/1963094256 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3df809c7f0 con 0x7f3e00109750 2024-03-21T10:49:13.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:13 smithi031 ceph-mon[31673]: Adjusting osd_memory_target on smithi031 to 177.8M 2024-03-21T10:49:13.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:13 smithi031 ceph-mon[31673]: Unable to set osd_memory_target on smithi031 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-21T10:49:13.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:13 smithi031 ceph-mon[31673]: pgmap v331: 129 pgs: 15 active+clean+snaptrim, 43 active+clean+snaptrim_wait, 71 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 56 KiB/s wr, 27 op/s 2024-03-21T10:49:13.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.403+0000 7f3dffe00640 1 -- 172.21.15.31:0/1963094256 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e0002d070 con 0x7f3e00109750 2024-03-21T10:49:13.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.404+0000 7f3e04a00640 1 -- 172.21.15.31:0/1963094256 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+11384 (secure 0 0 0) 0x7f3df809c0f0 con 0x7f3e00109750 2024-03-21T10:49:13.401 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:49:13.401 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:49:06.188518+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24547,24596]},"id":1}]} 2024-03-21T10:49:13.401 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 30 2024-03-21T10:49:13.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.406+0000 7f3dffe00640 1 -- 172.21.15.31:0/1963094256 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3de40a3100 msgr2=0x7f3de40a55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:13.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.406+0000 7f3dffe00640 1 --2- 172.21.15.31:0/1963094256 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3de40a3100 0x7f3de40a55c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3df0009230 tx=0x7f3df0005fd0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.406+0000 7f3dffe00640 1 -- 172.21.15.31:0/1963094256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e00109750 msgr2=0x7f3e0010db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:13.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.406+0000 7f3dffe00640 1 --2- 172.21.15.31:0/1963094256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e00109750 0x7f3e0010db10 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f3df8009fd0 tx=0x7f3df800ef10 comp rx=0 tx=0).stop 2024-03-21T10:49:13.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.407+0000 7f3dffe00640 1 -- 172.21.15.31:0/1963094256 shutdown_connections 2024-03-21T10:49:13.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.407+0000 7f3dffe00640 1 --2- 172.21.15.31:0/1963094256 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3de40a3100 0x7f3de40a55c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.407+0000 7f3dffe00640 1 --2- 172.21.15.31:0/1963094256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e00109750 0x7f3e0010db10 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.407+0000 7f3dffe00640 1 --2- 172.21.15.31:0/1963094256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e00104100 0x7f3e0010d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.407+0000 7f3dffe00640 1 --2- 172.21.15.31:0/1963094256 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e00103740 0x7f3e001104f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.407+0000 7f3dffe00640 1 -- 172.21.15.31:0/1963094256 >> 172.21.15.31:0/1963094256 conn(0x7f3e000ff210 msgr2=0x7f3e000ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:13.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.407+0000 7f3dffe00640 1 -- 172.21.15.31:0/1963094256 shutdown_connections 2024-03-21T10:49:13.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.407+0000 7f3dffe00640 1 -- 172.21.15.31:0/1963094256 wait complete. 2024-03-21T10:49:13.423 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:49:13.423 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-21T10:49:13.429 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:13 smithi040 ceph-mon[36154]: Adjusting osd_memory_target on smithi031 to 177.8M 2024-03-21T10:49:13.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:13 smithi040 ceph-mon[36154]: Unable to set osd_memory_target on smithi031 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-21T10:49:13.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:13 smithi040 ceph-mon[36154]: pgmap v331: 129 pgs: 15 active+clean+snaptrim, 43 active+clean+snaptrim_wait, 71 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 56 KiB/s wr, 27 op/s 2024-03-21T10:49:13.473 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:13 smithi070 ceph-mon[36330]: Adjusting osd_memory_target on smithi031 to 177.8M 2024-03-21T10:49:13.473 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:13 smithi070 ceph-mon[36330]: Unable to set osd_memory_target on smithi031 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-21T10:49:13.474 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:13 smithi070 ceph-mon[36330]: pgmap v331: 129 pgs: 15 active+clean+snaptrim, 43 active+clean+snaptrim_wait, 71 active+clean; 30 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 56 KiB/s wr, 27 op/s 2024-03-21T10:49:13.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.633+0000 7f2835000640 1 -- 172.21.15.31:0/2699209337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2830074680 msgr2=0x7f283010d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:13.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.633+0000 7f2835000640 1 --2- 172.21.15.31:0/2699209337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2830074680 0x7f283010d940 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f281c009900 tx=0x7f281c02bc50 comp rx=0 tx=0).stop 2024-03-21T10:49:13.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.633+0000 7f2835000640 1 -- 172.21.15.31:0/2699209337 shutdown_connections 2024-03-21T10:49:13.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.633+0000 7f2835000640 1 --2- 172.21.15.31:0/2699209337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f283010dfb0 0x7f28301103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.633+0000 7f2835000640 1 --2- 172.21.15.31:0/2699209337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2830074680 0x7f283010d940 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.633+0000 7f2835000640 1 --2- 172.21.15.31:0/2699209337 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2830073cd0 0x7f28300740b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.633+0000 7f2835000640 1 -- 172.21.15.31:0/2699209337 >> 172.21.15.31:0/2699209337 conn(0x7f28300fd130 msgr2=0x7f28300ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:13.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.633+0000 7f2835000640 1 -- 172.21.15.31:0/2699209337 shutdown_connections 2024-03-21T10:49:13.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.633+0000 7f2835000640 1 -- 172.21.15.31:0/2699209337 wait complete. 2024-03-21T10:49:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.634+0000 7f2835000640 1 Processor -- start 2024-03-21T10:49:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.634+0000 7f2835000640 1 -- start start 2024-03-21T10:49:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.634+0000 7f2835000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2830073cd0 0x7f283019a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.634+0000 7f2835000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2830074680 0x7f283019a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.634+0000 7f2835000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f283010dfb0 0x7f283019ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.634+0000 7f2835000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2830113d20 con 0x7f2830074680 2024-03-21T10:49:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.634+0000 7f2835000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2830113ba0 con 0x7f2830073cd0 2024-03-21T10:49:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.634+0000 7f2835000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2830113ea0 con 0x7f283010dfb0 2024-03-21T10:49:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.635+0000 7f282de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2830073cd0 0x7f283019a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.635+0000 7f282de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2830073cd0 0x7f283019a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39292/0 (socket says 172.21.15.31:39292) 2024-03-21T10:49:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.635+0000 7f282e800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f283010dfb0 0x7f283019ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.635+0000 7f282de00640 1 -- 172.21.15.31:0/525094927 learned_addr learned my addr 172.21.15.31:0/525094927 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.635+0000 7f282d400640 1 --2- 172.21.15.31:0/525094927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2830074680 0x7f283019a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.635+0000 7f282de00640 1 -- 172.21.15.31:0/525094927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f283010dfb0 msgr2=0x7f283019ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.635+0000 7f282de00640 1 --2- 172.21.15.31:0/525094927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f283010dfb0 0x7f283019ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.635+0000 7f282de00640 1 -- 172.21.15.31:0/525094927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2830074680 msgr2=0x7f283019a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.635+0000 7f282de00640 1 --2- 172.21.15.31:0/525094927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2830074680 0x7f283019a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.635+0000 7f282de00640 1 -- 172.21.15.31:0/525094927 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f283019f360 con 0x7f2830073cd0 2024-03-21T10:49:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.635+0000 7f282e800640 1 --2- 172.21.15.31:0/525094927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f283010dfb0 0x7f283019ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.635+0000 7f282d400640 1 --2- 172.21.15.31:0/525094927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2830074680 0x7f283019a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.635+0000 7f282de00640 1 --2- 172.21.15.31:0/525094927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2830073cd0 0x7f283019a370 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f282000e9f0 tx=0x7f282000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.636+0000 7f2826a00640 1 -- 172.21.15.31:0/525094927 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282000cd80 con 0x7f2830073cd0 2024-03-21T10:49:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.636+0000 7f2826a00640 1 -- 172.21.15.31:0/525094927 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2820010910 con 0x7f2830073cd0 2024-03-21T10:49:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.636+0000 7f2835000640 1 -- 172.21.15.31:0/525094927 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f283019f650 con 0x7f2830073cd0 2024-03-21T10:49:13.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.636+0000 7f2826a00640 1 -- 172.21.15.31:0/525094927 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2820018990 con 0x7f2830073cd0 2024-03-21T10:49:13.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.636+0000 7f2835000640 1 -- 172.21.15.31:0/525094927 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28301a6ee0 con 0x7f2830073cd0 2024-03-21T10:49:13.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.639+0000 7f2837e1c740 1 Processor -- start 2024-03-21T10:49:13.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.639+0000 7f2837e1c740 1 -- start start 2024-03-21T10:49:13.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.639+0000 7f2826a00640 1 -- 172.21.15.31:0/525094927 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f2820010ac0 con 0x7f2830073cd0 2024-03-21T10:49:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.639+0000 7f2837e1c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d95c7f6d0 0x563d95c7fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.639+0000 7f2837e1c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d95c80080 0x563d95c88630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.640+0000 7f2837e1c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d95c88c00 0x563d95c8b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.640+0000 7f2837e1c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d95be32c0 con 0x563d95c88c00 2024-03-21T10:49:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.640+0000 7f2837e1c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d95b8bc90 con 0x563d95c80080 2024-03-21T10:49:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.640+0000 7f2837e1c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d95bdf360 con 0x563d95c7f6d0 2024-03-21T10:49:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.640+0000 7f282de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d95c80080 0x563d95c88630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.640+0000 7f282e800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d95c7f6d0 0x563d95c7fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.640+0000 7f282e800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d95c7f6d0 0x563d95c7fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53042/0 (socket says 172.21.15.31:53042) 2024-03-21T10:49:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.640+0000 7f282d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d95c88c00 0x563d95c8b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.640+0000 7f282e800640 1 -- 172.21.15.31:0/745486617 learned_addr learned my addr 172.21.15.31:0/745486617 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:13.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.640+0000 7f282de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d95c80080 0x563d95c88630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39294/0 (socket says 172.21.15.31:39294) 2024-03-21T10:49:13.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.640+0000 7f282e800640 1 -- 172.21.15.31:0/745486617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d95c80080 msgr2=0x563d95c88630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:13.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.640+0000 7f282e800640 1 --2- 172.21.15.31:0/745486617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d95c80080 0x563d95c88630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.640+0000 7f282e800640 1 -- 172.21.15.31:0/745486617 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d95c88c00 msgr2=0x563d95c8b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.640+0000 7f282e800640 1 --2- 172.21.15.31:0/745486617 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d95c88c00 0x563d95c8b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.640+0000 7f282e800640 1 -- 172.21.15.31:0/745486617 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d95bb0080 con 0x563d95c7f6d0 2024-03-21T10:49:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.641+0000 7f282e800640 1 --2- 172.21.15.31:0/745486617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d95c7f6d0 0x563d95c7fab0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f2828011790 tx=0x7f2828011c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ce24cb026aae8fd server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.641+0000 7f281b400640 1 -- 172.21.15.31:0/745486617 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2828003c30 con 0x563d95c7f6d0 2024-03-21T10:49:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.641+0000 7f2826a00640 1 --2- 172.21.15.31:0/525094927 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f28140a3110 0x7f28140a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.641+0000 7f281b400640 1 -- 172.21.15.31:0/745486617 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f282800fe80 con 0x563d95c7f6d0 2024-03-21T10:49:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.641+0000 7f281b400640 1 -- 172.21.15.31:0/745486617 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28280193e0 con 0x563d95c7f6d0 2024-03-21T10:49:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.641+0000 7f2826a00640 1 -- 172.21.15.31:0/525094927 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f28200cfe60 con 0x7f2830073cd0 2024-03-21T10:49:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.641+0000 7f282de00640 1 --2- 172.21.15.31:0/525094927 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f28140a3110 0x7f28140a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.642+0000 7f2837e1c740 1 -- 172.21.15.31:0/745486617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d95c7f6d0 msgr2=0x563d95c7fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.642+0000 7f2837e1c740 1 --2- 172.21.15.31:0/745486617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d95c7f6d0 0x563d95c7fab0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f2828011790 tx=0x7f2828011c50 comp rx=0 tx=0).stop 2024-03-21T10:49:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.642+0000 7f2837e1c740 1 -- 172.21.15.31:0/745486617 shutdown_connections 2024-03-21T10:49:13.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.642+0000 7f2837e1c740 1 --2- 172.21.15.31:0/745486617 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d95c88c00 0x563d95c8b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.642+0000 7f2837e1c740 1 --2- 172.21.15.31:0/745486617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d95c80080 0x563d95c88630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.642+0000 7f2837e1c740 1 --2- 172.21.15.31:0/745486617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d95c7f6d0 0x563d95c7fab0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.642+0000 7f2837e1c740 1 -- 172.21.15.31:0/745486617 >> 172.21.15.31:0/745486617 conn(0x563d95c79da0 msgr2=0x563d95c7a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:13.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.642+0000 7f2837e1c740 1 -- 172.21.15.31:0/745486617 shutdown_connections 2024-03-21T10:49:13.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.642+0000 7f282de00640 1 --2- 172.21.15.31:0/525094927 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f28140a3110 0x7f28140a55d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f28200cd270 tx=0x7f28200d39b0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:13.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.642+0000 7f2837e1c740 1 -- 172.21.15.31:0/745486617 wait complete. 2024-03-21T10:49:13.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.643+0000 7f2837e1c740 1 -- start start 2024-03-21T10:49:13.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.643+0000 7f2837e1c740 1 Processor -- start 2024-03-21T10:49:13.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.644+0000 7f2837e1c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d95c7f6d0 0x563d95c88490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.644+0000 7f2837e1c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d95c80080 0x563d95c85820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.644+0000 7f2837e1c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d95c88c00 0x563d95c85d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.644+0000 7f2837e1c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d95c8bb70 con 0x563d95c7f6d0 2024-03-21T10:49:13.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.644+0000 7f2837e1c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d95b9f0d0 con 0x563d95c80080 2024-03-21T10:49:13.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.644+0000 7f2837e1c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d95c8bcf0 con 0x563d95c88c00 2024-03-21T10:49:13.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.644+0000 7f2837e1c740 10 client.0 _ll_register_callbacks cb 0x563d95ba3e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.644+0000 7f282e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d95c7f6d0 0x563d95c88490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.644+0000 7f282d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d95c80080 0x563d95c85820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.644+0000 7f282e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d95c7f6d0 0x563d95c88490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45596/0 (socket says 172.21.15.31:45596) 2024-03-21T10:49:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.644+0000 7f282d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d95c80080 0x563d95c85820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39304/0 (socket says 172.21.15.31:39304) 2024-03-21T10:49:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.644+0000 7f282e800640 1 -- 172.21.15.31:0/2005892813 learned_addr learned my addr 172.21.15.31:0/2005892813 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.644+0000 7f282e800640 1 --2- 172.21.15.31:0/2005892813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d95c88c00 0x563d95c85d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.645+0000 7f282d400640 1 -- 172.21.15.31:0/2005892813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d95c88c00 msgr2=0x563d95c85d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.645+0000 7f282d400640 1 --2- 172.21.15.31:0/2005892813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d95c88c00 0x563d95c85d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.645+0000 7f282d400640 1 -- 172.21.15.31:0/2005892813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d95c7f6d0 msgr2=0x563d95c88490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.645+0000 7f282d400640 1 --2- 172.21.15.31:0/2005892813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d95c7f6d0 0x563d95c88490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.645+0000 7f282d400640 1 -- 172.21.15.31:0/2005892813 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563d958f73f0 con 0x563d95c80080 2024-03-21T10:49:13.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.645+0000 7f282d400640 1 --2- 172.21.15.31:0/2005892813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d95c80080 0x563d95c85820 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f281c036040 tx=0x7f281c02c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=208a7f51c73026bc server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.645+0000 7f280fe00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:49:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.645+0000 7f280fe00640 1 -- 172.21.15.31:0/2005892813 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281c004240 con 0x563d95c80080 2024-03-21T10:49:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.645+0000 7f280fe00640 1 -- 172.21.15.31:0/2005892813 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f281c0043e0 con 0x563d95c80080 2024-03-21T10:49:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.645+0000 7f2837e1c740 1 -- 172.21.15.31:0/2005892813 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563d95bf0800 con 0x563d95c80080 2024-03-21T10:49:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.646+0000 7f280fe00640 1 -- 172.21.15.31:0/2005892813 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f281c051050 con 0x563d95c80080 2024-03-21T10:49:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.646+0000 7f2837e1c740 10 client.28286 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.646+0000 7f2837e1c740 1 -- 172.21.15.31:0/2005892813 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x563d959c17d0 con 0x563d95c80080 2024-03-21T10:49:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.646+0000 7f280fe00640 1 -- 172.21.15.31:0/2005892813 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563d95bf0800 con 0x563d95c80080 2024-03-21T10:49:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.646+0000 7f280fe00640 1 -- 172.21.15.31:0/2005892813 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f281c003890 con 0x563d95c80080 2024-03-21T10:49:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.646+0000 7f2837e1c740 10 client.28286 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.646+0000 7f2837e1c740 10 client.28286 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:49:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.646+0000 7f2837e1c740 20 client.28286 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.646+0000 7f2837e1c740 1 --2- 172.21.15.31:0/2005892813 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x563d95c989f0 0x563d95c9ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.646+0000 7f2837e1c740 4 client.28286 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:49:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.646+0000 7f2837e1c740 1 -- 172.21.15.31:0/2005892813 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563d95beab50 con 0x563d95c989f0 2024-03-21T10:49:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.647+0000 7f282e800640 1 --2- 172.21.15.31:0/2005892813 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x563d95c989f0 0x563d95c9ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.647+0000 7f280fe00640 10 client.28286 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:49:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.647+0000 7f282e800640 1 --2- 172.21.15.31:0/2005892813 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x563d95c989f0 0x563d95c9ade0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8cbd3a2c0379b0a0 server_cookie=dcf5d36b2a4292b7 in_seq=0 out_seq=0 2024-03-21T10:49:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.648+0000 7f280fe00640 1 -- 172.21.15.31:0/2005892813 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563d95beab50 con 0x563d95c989f0 2024-03-21T10:49:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.648+0000 7f280fe00640 10 client.28286 handle_command_reply: tid=0 2024-03-21T10:49:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.648+0000 7f2837e1c740 1 client.28286 shutdown 2024-03-21T10:49:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.648+0000 7f2837e1c740 1 -- 172.21.15.31:0/2005892813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d95c80080 msgr2=0x563d95c85820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.648+0000 7f2837e1c740 1 --2- 172.21.15.31:0/2005892813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d95c80080 0x563d95c85820 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f281c036040 tx=0x7f281c02c8a0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.648+0000 7f2837e1c740 1 -- 172.21.15.31:0/2005892813 shutdown_connections 2024-03-21T10:49:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.648+0000 7f2837e1c740 1 --2- 172.21.15.31:0/2005892813 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x563d95c989f0 0x563d95c9ade0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.648+0000 7f2837e1c740 1 --2- 172.21.15.31:0/2005892813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d95c88c00 0x563d95c85d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.648+0000 7f2837e1c740 1 --2- 172.21.15.31:0/2005892813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d95c80080 0x563d95c85820 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.648+0000 7f280fe00640 0 client.28286 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:49:13.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.648+0000 7f2837e1c740 1 --2- 172.21.15.31:0/2005892813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d95c7f6d0 0x563d95c88490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.648+0000 7f2837e1c740 1 -- 172.21.15.31:0/2005892813 >> 172.21.15.31:0/2005892813 conn(0x563d95c79da0 msgr2=0x563d95c7baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:13.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.648+0000 7f2837e1c740 1 -- 172.21.15.31:0/2005892813 shutdown_connections 2024-03-21T10:49:13.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.648+0000 7f2837e1c740 1 -- 172.21.15.31:0/2005892813 wait complete. 2024-03-21T10:49:13.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.648+0000 7f2837e1c740 20 client.28286 trim_cache size 0 max 16384 2024-03-21T10:49:13.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.663+0000 7f2837e1c740 1 Processor -- start 2024-03-21T10:49:13.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.663+0000 7f2837e1c740 1 -- start start 2024-03-21T10:49:13.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.664+0000 7f2837e1c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d95c7f6d0 0x563d95c83fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.664+0000 7f2837e1c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d95c80080 0x563d95c87f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.664+0000 7f2837e1c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d95c88c00 0x563d95c88490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.664+0000 7f2837e1c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28301b3380 con 0x563d95c7f6d0 2024-03-21T10:49:13.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.664+0000 7f2837e1c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28301b0e80 con 0x563d95c88c00 2024-03-21T10:49:13.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.664+0000 7f2837e1c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28301aef40 con 0x563d95c80080 2024-03-21T10:49:13.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.664+0000 7f282d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d95c80080 0x563d95c87f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.664+0000 7f282e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d95c7f6d0 0x563d95c83fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.664+0000 7f282d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d95c80080 0x563d95c87f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53048/0 (socket says 172.21.15.31:53048) 2024-03-21T10:49:13.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.664+0000 7f282d400640 1 -- 172.21.15.31:0/1073658541 learned_addr learned my addr 172.21.15.31:0/1073658541 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:13.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.664+0000 7f282e800640 1 --2- 172.21.15.31:0/1073658541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d95c88c00 0x563d95c88490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.664+0000 7f282d400640 1 -- 172.21.15.31:0/1073658541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d95c88c00 msgr2=0x563d95c88490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:13.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.664+0000 7f282d400640 1 --2- 172.21.15.31:0/1073658541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d95c88c00 0x563d95c88490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.664+0000 7f282d400640 1 -- 172.21.15.31:0/1073658541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d95c7f6d0 msgr2=0x563d95c83fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:13.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.664+0000 7f282d400640 1 --2- 172.21.15.31:0/1073658541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d95c7f6d0 0x563d95c83fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.664+0000 7f282d400640 1 -- 172.21.15.31:0/1073658541 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d95c81400 con 0x563d95c80080 2024-03-21T10:49:13.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.664+0000 7f282e800640 1 --2- 172.21.15.31:0/1073658541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d95c88c00 0x563d95c88490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:49:13.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.665+0000 7f282d400640 1 --2- 172.21.15.31:0/1073658541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d95c80080 0x563d95c87f50 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f281c02c840 tx=0x7f281c02bc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5aec365d28422fa server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:13.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.665+0000 7f281aa00640 1 -- 172.21.15.31:0/1073658541 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281c05a070 con 0x563d95c80080 2024-03-21T10:49:13.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.665+0000 7f281aa00640 1 -- 172.21.15.31:0/1073658541 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f281c043070 con 0x563d95c80080 2024-03-21T10:49:13.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.665+0000 7f281aa00640 1 -- 172.21.15.31:0/1073658541 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281c003a50 con 0x563d95c80080 2024-03-21T10:49:13.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.666+0000 7f2837e1c740 1 -- 172.21.15.31:0/1073658541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d95c80080 msgr2=0x563d95c87f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:13.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.666+0000 7f2837e1c740 1 --2- 172.21.15.31:0/1073658541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d95c80080 0x563d95c87f50 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f281c02c840 tx=0x7f281c02bc50 comp rx=0 tx=0).stop 2024-03-21T10:49:13.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.666+0000 7f2837e1c740 1 -- 172.21.15.31:0/1073658541 shutdown_connections 2024-03-21T10:49:13.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.666+0000 7f2837e1c740 1 --2- 172.21.15.31:0/1073658541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d95c88c00 0x563d95c88490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.666+0000 7f2837e1c740 1 --2- 172.21.15.31:0/1073658541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d95c80080 0x563d95c87f50 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.666+0000 7f2837e1c740 1 --2- 172.21.15.31:0/1073658541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d95c7f6d0 0x563d95c83fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.666+0000 7f2837e1c740 1 -- 172.21.15.31:0/1073658541 >> 172.21.15.31:0/1073658541 conn(0x563d95c79da0 msgr2=0x563d95c80cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:13.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.666+0000 7f2837e1c740 1 -- 172.21.15.31:0/1073658541 shutdown_connections 2024-03-21T10:49:13.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.666+0000 7f2837e1c740 1 -- 172.21.15.31:0/1073658541 wait complete. 2024-03-21T10:49:13.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.666+0000 7f2837e1c740 1 -- start start 2024-03-21T10:49:13.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.667+0000 7f2837e1c740 1 Processor -- start 2024-03-21T10:49:13.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.667+0000 7f2837e1c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d95c7f6d0 0x563d95c8cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.667+0000 7f2837e1c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d95c80080 0x563d95c8d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.667+0000 7f2837e1c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d95c88c00 0x563d95cf3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.667+0000 7f2837e1c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28301af540 con 0x563d95c7f6d0 2024-03-21T10:49:13.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.667+0000 7f2837e1c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28301b0820 con 0x563d95c88c00 2024-03-21T10:49:13.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.667+0000 7f2837e1c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28301afb60 con 0x563d95c80080 2024-03-21T10:49:13.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.667+0000 7f2837e1c740 10 client.0 _ll_register_callbacks cb 0x7f281c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:13.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.668+0000 7f282d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d95c80080 0x563d95c8d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.668+0000 7f282d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d95c80080 0x563d95c8d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53052/0 (socket says 172.21.15.31:53052) 2024-03-21T10:49:13.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.668+0000 7f282e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d95c7f6d0 0x563d95c8cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.668+0000 7f282d400640 1 -- 172.21.15.31:0/4245328447 learned_addr learned my addr 172.21.15.31:0/4245328447 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:13.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.668+0000 7f282e800640 1 --2- 172.21.15.31:0/4245328447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d95c88c00 0x563d95cf3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.668+0000 7f282d400640 1 -- 172.21.15.31:0/4245328447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d95c88c00 msgr2=0x563d95cf3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:13.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.668+0000 7f282d400640 1 --2- 172.21.15.31:0/4245328447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d95c88c00 0x563d95cf3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.668+0000 7f282d400640 1 -- 172.21.15.31:0/4245328447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d95c7f6d0 msgr2=0x563d95c8cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:13.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.668+0000 7f282d400640 1 --2- 172.21.15.31:0/4245328447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d95c7f6d0 0x563d95c8cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.668+0000 7f282d400640 1 -- 172.21.15.31:0/4245328447 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563d95cf3860 con 0x563d95c80080 2024-03-21T10:49:13.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.668+0000 7f282e800640 1 --2- 172.21.15.31:0/4245328447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d95c7f6d0 0x563d95c8cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:49:13.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.668+0000 7f282d400640 1 --2- 172.21.15.31:0/4245328447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d95c80080 0x563d95c8d050 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f281c02c190 tx=0x7f281c0054a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45b09b8a84a6239e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:13.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.668+0000 7f280fe00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:49:13.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.669+0000 7f280fe00640 1 -- 172.21.15.31:0/4245328447 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281c062060 con 0x563d95c80080 2024-03-21T10:49:13.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.669+0000 7f2837e1c740 1 -- 172.21.15.31:0/4245328447 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f281c043070 con 0x563d95c80080 2024-03-21T10:49:13.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.669+0000 7f280fe00640 1 -- 172.21.15.31:0/4245328447 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f281c003800 con 0x563d95c80080 2024-03-21T10:49:13.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.669+0000 7f280fe00640 1 -- 172.21.15.31:0/4245328447 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f281c069050 con 0x563d95c80080 2024-03-21T10:49:13.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.669+0000 7f2837e1c740 10 client.28219 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:13.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.669+0000 7f2837e1c740 1 -- 172.21.15.31:0/4245328447 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x563d95cf4ab0 con 0x563d95c80080 2024-03-21T10:49:13.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.669+0000 7f280fe00640 1 -- 172.21.15.31:0/4245328447 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f281c043070 con 0x563d95c80080 2024-03-21T10:49:13.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.669+0000 7f280fe00640 1 -- 172.21.15.31:0/4245328447 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f281c004260 con 0x563d95c80080 2024-03-21T10:49:13.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.669+0000 7f2837e1c740 10 client.28219 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:13.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.669+0000 7f2837e1c740 10 client.28219 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:49:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.669+0000 7f2837e1c740 20 client.28219 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.669+0000 7f2837e1c740 1 --2- 172.21.15.31:0/4245328447 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x563d95cf5110 0x563d95cf7530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.669+0000 7f2837e1c740 4 client.28219 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:49:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.669+0000 7f2837e1c740 1 -- 172.21.15.31:0/4245328447 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563d95c860f0 con 0x563d95cf5110 2024-03-21T10:49:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.670+0000 7f282e800640 1 --2- 172.21.15.31:0/4245328447 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x563d95cf5110 0x563d95cf7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.670+0000 7f282e800640 1 --2- 172.21.15.31:0/4245328447 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x563d95cf5110 0x563d95cf7530 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1ca9c7db2a5b844 server_cookie=b2d0c0c202fdc2b0 in_seq=0 out_seq=0 2024-03-21T10:49:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.670+0000 7f280fe00640 10 client.28219 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:49:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.670+0000 7f280fe00640 1 -- 172.21.15.31:0/4245328447 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563d95c860f0 con 0x563d95cf5110 2024-03-21T10:49:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.670+0000 7f280fe00640 10 client.28219 handle_command_reply: tid=0 2024-03-21T10:49:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.670+0000 7f2837e1c740 1 client.28219 shutdown 2024-03-21T10:49:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.670+0000 7f2837e1c740 1 -- 172.21.15.31:0/4245328447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d95c80080 msgr2=0x563d95c8d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.670+0000 7f2837e1c740 1 --2- 172.21.15.31:0/4245328447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d95c80080 0x563d95c8d050 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f281c02c190 tx=0x7f281c0054a0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.671+0000 7f2837e1c740 1 -- 172.21.15.31:0/4245328447 shutdown_connections 2024-03-21T10:49:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.671+0000 7f2837e1c740 1 --2- 172.21.15.31:0/4245328447 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x563d95cf5110 0x563d95cf7530 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.671+0000 7f2837e1c740 1 --2- 172.21.15.31:0/4245328447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d95c88c00 0x563d95cf3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.671+0000 7f2837e1c740 1 --2- 172.21.15.31:0/4245328447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d95c80080 0x563d95c8d050 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.671+0000 7f2837e1c740 1 --2- 172.21.15.31:0/4245328447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d95c7f6d0 0x563d95c8cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.671+0000 7f2837e1c740 1 -- 172.21.15.31:0/4245328447 >> 172.21.15.31:0/4245328447 conn(0x563d95c79da0 msgr2=0x563d95c78620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:13.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.671+0000 7f280fe00640 0 client.28219 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:49:13.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.671+0000 7f2837e1c740 1 -- 172.21.15.31:0/4245328447 shutdown_connections 2024-03-21T10:49:13.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.671+0000 7f2837e1c740 1 -- 172.21.15.31:0/4245328447 wait complete. 2024-03-21T10:49:13.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.671+0000 7f2837e1c740 20 client.28219 trim_cache size 0 max 16384 2024-03-21T10:49:13.672 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:49:13.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.671+0000 7f2835000640 1 -- 172.21.15.31:0/525094927 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f28140a3110 msgr2=0x7f28140a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:13.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.671+0000 7f2835000640 1 --2- 172.21.15.31:0/525094927 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f28140a3110 0x7f28140a55d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f28200cd270 tx=0x7f28200d39b0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.671+0000 7f2835000640 1 -- 172.21.15.31:0/525094927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2830073cd0 msgr2=0x7f283019a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:13.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.671+0000 7f2835000640 1 --2- 172.21.15.31:0/525094927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2830073cd0 0x7f283019a370 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f282000e9f0 tx=0x7f282000eeb0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.672+0000 7f2835000640 1 -- 172.21.15.31:0/525094927 shutdown_connections 2024-03-21T10:49:13.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.672+0000 7f2835000640 1 --2- 172.21.15.31:0/525094927 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f28140a3110 0x7f28140a55d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.672+0000 7f2835000640 1 --2- 172.21.15.31:0/525094927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f283010dfb0 0x7f283019ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.672+0000 7f2835000640 1 --2- 172.21.15.31:0/525094927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2830074680 0x7f283019a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.672+0000 7f2835000640 1 --2- 172.21.15.31:0/525094927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2830073cd0 0x7f283019a370 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.672+0000 7f2835000640 1 -- 172.21.15.31:0/525094927 >> 172.21.15.31:0/525094927 conn(0x7f28300fd130 msgr2=0x7f28300fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:13.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.672+0000 7f2835000640 1 -- 172.21.15.31:0/525094927 shutdown_connections 2024-03-21T10:49:13.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.672+0000 7f2835000640 1 -- 172.21.15.31:0/525094927 wait complete. 2024-03-21T10:49:13.693 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:49:13.693 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-21T10:49:13.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.898+0000 7f2488e00640 1 -- 172.21.15.31:0/2797094047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f248410ba50 msgr2=0x7f248410ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:13.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.898+0000 7f2488e00640 1 --2- 172.21.15.31:0/2797094047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f248410ba50 0x7f248410ded0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f247c009be0 tx=0x7f247c02bf80 comp rx=0 tx=0).stop 2024-03-21T10:49:13.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.898+0000 7f2488e00640 1 -- 172.21.15.31:0/2797094047 shutdown_connections 2024-03-21T10:49:13.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.898+0000 7f2488e00640 1 --2- 172.21.15.31:0/2797094047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f248410ba50 0x7f248410ded0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.898+0000 7f2488e00640 1 --2- 172.21.15.31:0/2797094047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2484102f50 0x7f248410b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.898+0000 7f2488e00640 1 --2- 172.21.15.31:0/2797094047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2484102630 0x7f2484102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.898+0000 7f2488e00640 1 -- 172.21.15.31:0/2797094047 >> 172.21.15.31:0/2797094047 conn(0x7f2484077130 msgr2=0x7f24840fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:13.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.898+0000 7f2488e00640 1 -- 172.21.15.31:0/2797094047 shutdown_connections 2024-03-21T10:49:13.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.898+0000 7f2488e00640 1 -- 172.21.15.31:0/2797094047 wait complete. 2024-03-21T10:49:13.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.899+0000 7f2488e00640 1 Processor -- start 2024-03-21T10:49:13.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.899+0000 7f2488e00640 1 -- start start 2024-03-21T10:49:13.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.900+0000 7f2488e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2484102630 0x7f2484100280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.900+0000 7f2488e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2484102f50 0x7f24841007c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.900+0000 7f2488e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f248410ba50 0x7f2484100d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.900+0000 7f2488e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2484111840 con 0x7f2484102630 2024-03-21T10:49:13.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.900+0000 7f2488e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24841116c0 con 0x7f2484102f50 2024-03-21T10:49:13.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.900+0000 7f2488e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24841119c0 con 0x7f248410ba50 2024-03-21T10:49:13.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.900+0000 7f2483400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2484102630 0x7f2484100280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.900+0000 7f2482a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2484102f50 0x7f24841007c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.900+0000 7f2482a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2484102f50 0x7f24841007c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39328/0 (socket says 172.21.15.31:39328) 2024-03-21T10:49:13.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.900+0000 7f2482a00640 1 -- 172.21.15.31:0/604701063 learned_addr learned my addr 172.21.15.31:0/604701063 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:13.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.900+0000 7f2483e00640 1 --2- 172.21.15.31:0/604701063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f248410ba50 0x7f2484100d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.900+0000 7f2482a00640 1 -- 172.21.15.31:0/604701063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f248410ba50 msgr2=0x7f2484100d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:13.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.900+0000 7f2482a00640 1 --2- 172.21.15.31:0/604701063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f248410ba50 0x7f2484100d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.900+0000 7f2482a00640 1 -- 172.21.15.31:0/604701063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2484102630 msgr2=0x7f2484100280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:13.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.900+0000 7f2482a00640 1 --2- 172.21.15.31:0/604701063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2484102630 0x7f2484100280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.900+0000 7f2482a00640 1 -- 172.21.15.31:0/604701063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24841a6180 con 0x7f2484102f50 2024-03-21T10:49:13.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.901+0000 7f2482a00640 1 --2- 172.21.15.31:0/604701063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2484102f50 0x7f24841007c0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f247000e9c0 tx=0x7f247000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:13.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.901+0000 7f247be00640 1 -- 172.21.15.31:0/604701063 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247000cde0 con 0x7f2484102f50 2024-03-21T10:49:13.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.901+0000 7f247be00640 1 -- 172.21.15.31:0/604701063 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f24700035f0 con 0x7f2484102f50 2024-03-21T10:49:13.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.901+0000 7f2488e00640 1 -- 172.21.15.31:0/604701063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24841a6470 con 0x7f2484102f50 2024-03-21T10:49:13.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.901+0000 7f247be00640 1 -- 172.21.15.31:0/604701063 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2470010690 con 0x7f2484102f50 2024-03-21T10:49:13.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.901+0000 7f2488e00640 1 -- 172.21.15.31:0/604701063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24841a6a10 con 0x7f2484102f50 2024-03-21T10:49:13.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.905+0000 7f248d1e2740 1 Processor -- start 2024-03-21T10:49:13.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.906+0000 7f248d1e2740 1 -- start start 2024-03-21T10:49:13.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.906+0000 7f248d1e2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdea8b0 0x556b6cdeac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.906+0000 7f248d1e2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b6cdeb260 0x556b6cdf3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.906+0000 7f248d1e2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b6cdf3de0 0x556b6cdf6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.906+0000 7f248d1e2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6cd4e2c0 con 0x556b6cdeb260 2024-03-21T10:49:13.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.906+0000 7f248d1e2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6ccf6c90 con 0x556b6cdea8b0 2024-03-21T10:49:13.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.906+0000 7f248d1e2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6cd4a360 con 0x556b6cdf3de0 2024-03-21T10:49:13.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.906+0000 7f247be00640 1 -- 172.21.15.31:0/604701063 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f24700037a0 con 0x7f2484102f50 2024-03-21T10:49:13.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.907+0000 7f2483e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdea8b0 0x556b6cdeac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.907+0000 7f2483e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdea8b0 0x556b6cdeac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39336/0 (socket says 172.21.15.31:39336) 2024-03-21T10:49:13.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.907+0000 7f2483e00640 1 -- 172.21.15.31:0/2943452826 learned_addr learned my addr 172.21.15.31:0/2943452826 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:13.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.907+0000 7f2483400640 1 --2- 172.21.15.31:0/2943452826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b6cdeb260 0x556b6cdf3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.907+0000 7f2482a00640 1 --2- 172.21.15.31:0/2943452826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b6cdf3de0 0x556b6cdf6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.907+0000 7f2483e00640 1 -- 172.21.15.31:0/2943452826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b6cdf3de0 msgr2=0x556b6cdf6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:13.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.907+0000 7f2483e00640 1 --2- 172.21.15.31:0/2943452826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b6cdf3de0 0x556b6cdf6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.907+0000 7f2483e00640 1 -- 172.21.15.31:0/2943452826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b6cdeb260 msgr2=0x556b6cdf3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:13.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.907+0000 7f2483e00640 1 --2- 172.21.15.31:0/2943452826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b6cdeb260 0x556b6cdf3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.907+0000 7f2483e00640 1 -- 172.21.15.31:0/2943452826 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b6cd1b080 con 0x556b6cdea8b0 2024-03-21T10:49:13.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.907+0000 7f2482a00640 1 --2- 172.21.15.31:0/2943452826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b6cdf3de0 0x556b6cdf6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:49:13.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.907+0000 7f2483400640 1 --2- 172.21.15.31:0/2943452826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b6cdeb260 0x556b6cdf3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:49:13.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.907+0000 7f2483e00640 1 --2- 172.21.15.31:0/2943452826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdea8b0 0x556b6cdeac90 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f247c02c590 tx=0x7f247c0098f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=222416bdd6123198 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:13.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.908+0000 7f2478c00640 1 -- 172.21.15.31:0/2943452826 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247c043030 con 0x556b6cdea8b0 2024-03-21T10:49:13.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.908+0000 7f2478c00640 1 -- 172.21.15.31:0/2943452826 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f247c02c920 con 0x556b6cdea8b0 2024-03-21T10:49:13.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.908+0000 7f2478c00640 1 -- 172.21.15.31:0/2943452826 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247c004410 con 0x556b6cdea8b0 2024-03-21T10:49:13.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.908+0000 7f247be00640 1 --2- 172.21.15.31:0/604701063 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24680a3140 0x7f24680a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.909+0000 7f247be00640 1 -- 172.21.15.31:0/604701063 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f24700cfe50 con 0x7f2484102f50 2024-03-21T10:49:13.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.909+0000 7f2483400640 1 --2- 172.21.15.31:0/604701063 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24680a3140 0x7f24680a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.909+0000 7f248d1e2740 1 -- 172.21.15.31:0/2943452826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdea8b0 msgr2=0x556b6cdeac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:13.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.909+0000 7f248d1e2740 1 --2- 172.21.15.31:0/2943452826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdea8b0 0x556b6cdeac90 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f247c02c590 tx=0x7f247c0098f0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.909+0000 7f248d1e2740 1 -- 172.21.15.31:0/2943452826 shutdown_connections 2024-03-21T10:49:13.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.909+0000 7f248d1e2740 1 --2- 172.21.15.31:0/2943452826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b6cdf3de0 0x556b6cdf6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.909+0000 7f248d1e2740 1 --2- 172.21.15.31:0/2943452826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b6cdeb260 0x556b6cdf3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.909+0000 7f248d1e2740 1 --2- 172.21.15.31:0/2943452826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdea8b0 0x556b6cdeac90 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.909+0000 7f248d1e2740 1 -- 172.21.15.31:0/2943452826 >> 172.21.15.31:0/2943452826 conn(0x556b6cde4f80 msgr2=0x556b6cde5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:13.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.909+0000 7f248d1e2740 1 -- 172.21.15.31:0/2943452826 shutdown_connections 2024-03-21T10:49:13.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.909+0000 7f248d1e2740 1 -- 172.21.15.31:0/2943452826 wait complete. 2024-03-21T10:49:13.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.909+0000 7f2483400640 1 --2- 172.21.15.31:0/604701063 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24680a3140 0x7f24680a5600 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x556b6cde5ff0 tx=0x7f247400b410 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:13.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.910+0000 7f248d1e2740 1 -- start start 2024-03-21T10:49:13.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.910+0000 7f248d1e2740 1 Processor -- start 2024-03-21T10:49:13.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.910+0000 7f248d1e2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b6cdea8b0 0x556b6cdf36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.910+0000 7f248d1e2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdeb260 0x556b6cdf0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.910+0000 7f248d1e2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b6cdf3de0 0x556b6cdf0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.910+0000 7f248d1e2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6cdf6cf0 con 0x556b6cdea8b0 2024-03-21T10:49:13.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.910+0000 7f248d1e2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6cd0a0d0 con 0x556b6cdeb260 2024-03-21T10:49:13.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.910+0000 7f248d1e2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6cdf6e70 con 0x556b6cdf3de0 2024-03-21T10:49:13.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.911+0000 7f248d1e2740 10 client.0 _ll_register_callbacks cb 0x556b6cd0ee80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:13.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.911+0000 7f2483400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b6cdea8b0 0x556b6cdf36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.911+0000 7f2482a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdeb260 0x556b6cdf0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.911+0000 7f2483e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b6cdf3de0 0x556b6cdf0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.911+0000 7f2482a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdeb260 0x556b6cdf0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39340/0 (socket says 172.21.15.31:39340) 2024-03-21T10:49:13.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.911+0000 7f2482a00640 1 -- 172.21.15.31:0/2609841809 learned_addr learned my addr 172.21.15.31:0/2609841809 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:13.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.911+0000 7f2482a00640 1 -- 172.21.15.31:0/2609841809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b6cdf3de0 msgr2=0x556b6cdf0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:13.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.911+0000 7f2482a00640 1 --2- 172.21.15.31:0/2609841809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b6cdf3de0 0x556b6cdf0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.911+0000 7f2482a00640 1 -- 172.21.15.31:0/2609841809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b6cdea8b0 msgr2=0x556b6cdf36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:13.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.911+0000 7f2482a00640 1 --2- 172.21.15.31:0/2609841809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b6cdea8b0 0x556b6cdf36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.911+0000 7f2482a00640 1 -- 172.21.15.31:0/2609841809 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556b6ca623f0 con 0x556b6cdeb260 2024-03-21T10:49:13.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.911+0000 7f2483e00640 1 --2- 172.21.15.31:0/2609841809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b6cdf3de0 0x556b6cdf0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:13.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.911+0000 7f2482a00640 1 --2- 172.21.15.31:0/2609841809 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdeb260 0x556b6cdf0a50 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f2470000c00 tx=0x7f247009c4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a05052a7719904f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:13.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.911+0000 7f246d600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:49:13.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.912+0000 7f246d600640 1 -- 172.21.15.31:0/2609841809 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24700cd5d0 con 0x556b6cdeb260 2024-03-21T10:49:13.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.912+0000 7f248d1e2740 1 -- 172.21.15.31:0/2609841809 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f247c02c920 con 0x556b6cdeb260 2024-03-21T10:49:13.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.912+0000 7f246d600640 1 -- 172.21.15.31:0/2609841809 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f24700bab60 con 0x556b6cdeb260 2024-03-21T10:49:13.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.912+0000 7f246d600640 1 -- 172.21.15.31:0/2609841809 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f247009ff40 con 0x556b6cdeb260 2024-03-21T10:49:13.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.912+0000 7f248d1e2740 10 client.28301 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:13.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.912+0000 7f248d1e2740 1 -- 172.21.15.31:0/2609841809 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x556b6cb2c7d0 con 0x556b6cdeb260 2024-03-21T10:49:13.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.912+0000 7f246d600640 1 -- 172.21.15.31:0/2609841809 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f247c02c920 con 0x556b6cdeb260 2024-03-21T10:49:13.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.913+0000 7f246d600640 1 -- 172.21.15.31:0/2609841809 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f2470097b40 con 0x556b6cdeb260 2024-03-21T10:49:13.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.913+0000 7f248d1e2740 10 client.28301 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:13.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.913+0000 7f248d1e2740 10 client.28301 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:49:13.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.913+0000 7f248d1e2740 20 client.28301 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:13.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.913+0000 7f248d1e2740 1 --2- 172.21.15.31:0/2609841809 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x556b6ce03ab0 0x556b6ce05ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.913+0000 7f248d1e2740 4 client.28301 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:49:13.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.913+0000 7f248d1e2740 1 -- 172.21.15.31:0/2609841809 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556b6cd5b800 con 0x556b6ce03ab0 2024-03-21T10:49:13.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.913+0000 7f2483400640 1 --2- 172.21.15.31:0/2609841809 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x556b6ce03ab0 0x556b6ce05ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.914+0000 7f2483400640 1 --2- 172.21.15.31:0/2609841809 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x556b6ce03ab0 0x556b6ce05ea0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=19cc21d8ea542fd7 server_cookie=b95841a53d1d75a6 in_seq=0 out_seq=0 2024-03-21T10:49:13.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.914+0000 7f246d600640 10 client.28301 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:49:13.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.915+0000 7f246d600640 1 -- 172.21.15.31:0/2609841809 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556b6cd5b800 con 0x556b6ce03ab0 2024-03-21T10:49:13.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.915+0000 7f246d600640 10 client.28301 handle_command_reply: tid=0 2024-03-21T10:49:13.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.915+0000 7f248d1e2740 1 client.28301 shutdown 2024-03-21T10:49:13.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.915+0000 7f248d1e2740 1 -- 172.21.15.31:0/2609841809 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdeb260 msgr2=0x556b6cdf0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:13.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.915+0000 7f248d1e2740 1 --2- 172.21.15.31:0/2609841809 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdeb260 0x556b6cdf0a50 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f2470000c00 tx=0x7f247009c4f0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.915+0000 7f248d1e2740 1 -- 172.21.15.31:0/2609841809 shutdown_connections 2024-03-21T10:49:13.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.915+0000 7f248d1e2740 1 --2- 172.21.15.31:0/2609841809 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x556b6ce03ab0 0x556b6ce05ea0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.915+0000 7f248d1e2740 1 --2- 172.21.15.31:0/2609841809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b6cdf3de0 0x556b6cdf0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.915+0000 7f248d1e2740 1 --2- 172.21.15.31:0/2609841809 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdeb260 0x556b6cdf0a50 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.915+0000 7f248d1e2740 1 --2- 172.21.15.31:0/2609841809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b6cdea8b0 0x556b6cdf36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.915+0000 7f248d1e2740 1 -- 172.21.15.31:0/2609841809 >> 172.21.15.31:0/2609841809 conn(0x556b6cde4f80 msgr2=0x556b6cde6cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.915+0000 7f246d600640 0 client.28301 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:49:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.915+0000 7f248d1e2740 1 -- 172.21.15.31:0/2609841809 shutdown_connections 2024-03-21T10:49:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.915+0000 7f248d1e2740 1 -- 172.21.15.31:0/2609841809 wait complete. 2024-03-21T10:49:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.915+0000 7f248d1e2740 20 client.28301 trim_cache size 0 max 16384 2024-03-21T10:49:13.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.930+0000 7f248d1e2740 1 Processor -- start 2024-03-21T10:49:13.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.930+0000 7f248d1e2740 1 -- start start 2024-03-21T10:49:13.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.931+0000 7f248d1e2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b6cdea8b0 0x556b6cdf3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.931+0000 7f248d1e2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b6cdeb260 0x556b6cdf36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.931+0000 7f248d1e2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdf3de0 0x556b6cdedcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.931+0000 7f248d1e2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24841b2f10 con 0x556b6cdea8b0 2024-03-21T10:49:13.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.931+0000 7f248d1e2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24841b0a10 con 0x556b6cdf3de0 2024-03-21T10:49:13.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.931+0000 7f248d1e2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24841aead0 con 0x556b6cdeb260 2024-03-21T10:49:13.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.931+0000 7f2482a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b6cdeb260 0x556b6cdf36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.931+0000 7f2482a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b6cdeb260 0x556b6cdf36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53096/0 (socket says 172.21.15.31:53096) 2024-03-21T10:49:13.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.931+0000 7f2482a00640 1 -- 172.21.15.31:0/3127386291 learned_addr learned my addr 172.21.15.31:0/3127386291 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:13.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.931+0000 7f2483e00640 1 --2- 172.21.15.31:0/3127386291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdf3de0 0x556b6cdedcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.931+0000 7f2483400640 1 --2- 172.21.15.31:0/3127386291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b6cdea8b0 0x556b6cdf3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.931+0000 7f2482a00640 1 -- 172.21.15.31:0/3127386291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdf3de0 msgr2=0x556b6cdedcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:13.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.931+0000 7f2482a00640 1 --2- 172.21.15.31:0/3127386291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdf3de0 0x556b6cdedcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.931+0000 7f2482a00640 1 -- 172.21.15.31:0/3127386291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b6cdea8b0 msgr2=0x556b6cdf3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:13.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.931+0000 7f2482a00640 1 --2- 172.21.15.31:0/3127386291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b6cdea8b0 0x556b6cdf3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.931+0000 7f2482a00640 1 -- 172.21.15.31:0/3127386291 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b6cdec650 con 0x556b6cdeb260 2024-03-21T10:49:13.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.931+0000 7f2483e00640 1 --2- 172.21.15.31:0/3127386291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdf3de0 0x556b6cdedcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:49:13.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.932+0000 7f2482a00640 1 --2- 172.21.15.31:0/3127386291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b6cdeb260 0x556b6cdf36c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f247009a0a0 tx=0x7f2470003d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a43804dea2b1f974 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:13.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.932+0000 7f246fe00640 1 -- 172.21.15.31:0/3127386291 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24700b7030 con 0x556b6cdeb260 2024-03-21T10:49:13.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.932+0000 7f246fe00640 1 -- 172.21.15.31:0/3127386291 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f247007f070 con 0x556b6cdeb260 2024-03-21T10:49:13.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.932+0000 7f246fe00640 1 -- 172.21.15.31:0/3127386291 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24700cd3d0 con 0x556b6cdeb260 2024-03-21T10:49:13.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.933+0000 7f248d1e2740 1 -- 172.21.15.31:0/3127386291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b6cdeb260 msgr2=0x556b6cdf36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:13.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.933+0000 7f248d1e2740 1 --2- 172.21.15.31:0/3127386291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b6cdeb260 0x556b6cdf36c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f247009a0a0 tx=0x7f2470003d20 comp rx=0 tx=0).stop 2024-03-21T10:49:13.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.933+0000 7f248d1e2740 1 -- 172.21.15.31:0/3127386291 shutdown_connections 2024-03-21T10:49:13.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.933+0000 7f248d1e2740 1 --2- 172.21.15.31:0/3127386291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdf3de0 0x556b6cdedcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.933+0000 7f248d1e2740 1 --2- 172.21.15.31:0/3127386291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b6cdeb260 0x556b6cdf36c0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.933+0000 7f248d1e2740 1 --2- 172.21.15.31:0/3127386291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b6cdea8b0 0x556b6cdf3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.933+0000 7f248d1e2740 1 -- 172.21.15.31:0/3127386291 >> 172.21.15.31:0/3127386291 conn(0x556b6cde4f80 msgr2=0x556b6cde4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:13.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.933+0000 7f248d1e2740 1 -- 172.21.15.31:0/3127386291 shutdown_connections 2024-03-21T10:49:13.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.933+0000 7f248d1e2740 1 -- 172.21.15.31:0/3127386291 wait complete. 2024-03-21T10:49:13.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.933+0000 7f248d1e2740 1 -- start start 2024-03-21T10:49:13.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.934+0000 7f248d1e2740 1 Processor -- start 2024-03-21T10:49:13.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.934+0000 7f248d1e2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b6cdea8b0 0x556b6cdf77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.934+0000 7f248d1e2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdeb260 0x556b6cdf7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.934+0000 7f248d1e2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b6cdf3de0 0x556b6cdf8250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.934+0000 7f248d1e2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24841af0d0 con 0x556b6cdea8b0 2024-03-21T10:49:13.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.934+0000 7f248d1e2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24841b03b0 con 0x556b6cdeb260 2024-03-21T10:49:13.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.934+0000 7f248d1e2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24841af6f0 con 0x556b6cdf3de0 2024-03-21T10:49:13.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.934+0000 7f248d1e2740 10 client.0 _ll_register_callbacks cb 0x7f24700046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:13.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.934+0000 7f2483400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b6cdea8b0 0x556b6cdf77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.934+0000 7f2483e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b6cdf3de0 0x556b6cdf8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.934+0000 7f2482a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdeb260 0x556b6cdf7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.934+0000 7f2483e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b6cdf3de0 0x556b6cdf8250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53098/0 (socket says 172.21.15.31:53098) 2024-03-21T10:49:13.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.934+0000 7f2483400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b6cdea8b0 0x556b6cdf77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45634/0 (socket says 172.21.15.31:45634) 2024-03-21T10:49:13.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.934+0000 7f2483e00640 1 -- 172.21.15.31:0/3233579968 learned_addr learned my addr 172.21.15.31:0/3233579968 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:13.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.934+0000 7f2482a00640 1 --2- 172.21.15.31:0/3233579968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdeb260 0x556b6cdf7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39364/0 (socket says 172.21.15.31:39364) 2024-03-21T10:49:13.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.935+0000 7f2483e00640 1 -- 172.21.15.31:0/3233579968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdeb260 msgr2=0x556b6cdf7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:13.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.935+0000 7f2483e00640 1 --2- 172.21.15.31:0/3233579968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdeb260 0x556b6cdf7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.935+0000 7f2483e00640 1 -- 172.21.15.31:0/3233579968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b6cdea8b0 msgr2=0x556b6cdf77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:13.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.935+0000 7f2483e00640 1 --2- 172.21.15.31:0/3233579968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b6cdea8b0 0x556b6cdf77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.935+0000 7f2483e00640 1 -- 172.21.15.31:0/3233579968 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556b6ce5e5c0 con 0x556b6cdf3de0 2024-03-21T10:49:13.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.935+0000 7f2482a00640 1 --2- 172.21.15.31:0/3233579968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdeb260 0x556b6cdf7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:13.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.935+0000 7f2483e00640 1 --2- 172.21.15.31:0/3233579968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b6cdf3de0 0x556b6cdf8250 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f247c009be0 tx=0x7f247c009960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ca389be36c414a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:13.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.935+0000 7f246d600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:49:13.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.935+0000 7f246d600640 1 -- 172.21.15.31:0/3233579968 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247c043070 con 0x556b6cdf3de0 2024-03-21T10:49:13.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.935+0000 7f248d1e2740 1 -- 172.21.15.31:0/3233579968 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f247007f070 con 0x556b6cdf3de0 2024-03-21T10:49:13.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.935+0000 7f246d600640 1 -- 172.21.15.31:0/3233579968 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f247c0047b0 con 0x556b6cdf3de0 2024-03-21T10:49:13.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.936+0000 7f246d600640 1 -- 172.21.15.31:0/3233579968 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f247c033520 con 0x556b6cdf3de0 2024-03-21T10:49:13.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.936+0000 7f248d1e2740 10 client.28234 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:13.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.936+0000 7f248d1e2740 1 -- 172.21.15.31:0/3233579968 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x556b6ce5f8c0 con 0x556b6cdf3de0 2024-03-21T10:49:13.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.936+0000 7f246d600640 1 -- 172.21.15.31:0/3233579968 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f247007f070 con 0x556b6cdf3de0 2024-03-21T10:49:13.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.936+0000 7f246d600640 1 -- 172.21.15.31:0/3233579968 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f247c03f900 con 0x556b6cdf3de0 2024-03-21T10:49:13.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.936+0000 7f248d1e2740 10 client.28234 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:13.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.936+0000 7f248d1e2740 10 client.28234 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:49:13.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.936+0000 7f248d1e2740 20 client.28234 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:13.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.936+0000 7f248d1e2740 1 --2- 172.21.15.31:0/3233579968 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x556b6ce5fba0 0x556b6ce61f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:13.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.936+0000 7f248d1e2740 4 client.28234 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:49:13.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.936+0000 7f248d1e2740 1 -- 172.21.15.31:0/3233579968 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556b6cd55b50 con 0x556b6ce5fba0 2024-03-21T10:49:13.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.936+0000 7f2483400640 1 --2- 172.21.15.31:0/3233579968 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x556b6ce5fba0 0x556b6ce61f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:13.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.937+0000 7f2483400640 1 --2- 172.21.15.31:0/3233579968 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x556b6ce5fba0 0x556b6ce61f90 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=570cb1fa28c001cd server_cookie=74243261183ff26a in_seq=0 out_seq=0 2024-03-21T10:49:13.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.937+0000 7f246d600640 10 client.28234 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:49:13.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.937+0000 7f246d600640 1 -- 172.21.15.31:0/3233579968 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556b6cd55b50 con 0x556b6ce5fba0 2024-03-21T10:49:13.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.937+0000 7f246d600640 10 client.28234 handle_command_reply: tid=0 2024-03-21T10:49:13.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.937+0000 7f248d1e2740 1 client.28234 shutdown 2024-03-21T10:49:13.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.937+0000 7f248d1e2740 1 -- 172.21.15.31:0/3233579968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b6cdf3de0 msgr2=0x556b6cdf8250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:13.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.937+0000 7f248d1e2740 1 --2- 172.21.15.31:0/3233579968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b6cdf3de0 0x556b6cdf8250 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f247c009be0 tx=0x7f247c009960 comp rx=0 tx=0).stop 2024-03-21T10:49:13.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.938+0000 7f248d1e2740 1 -- 172.21.15.31:0/3233579968 shutdown_connections 2024-03-21T10:49:13.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.938+0000 7f248d1e2740 1 --2- 172.21.15.31:0/3233579968 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x556b6ce5fba0 0x556b6ce61f90 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.938+0000 7f248d1e2740 1 --2- 172.21.15.31:0/3233579968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b6cdf3de0 0x556b6cdf8250 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.938+0000 7f248d1e2740 1 --2- 172.21.15.31:0/3233579968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b6cdeb260 0x556b6cdf7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.938+0000 7f248d1e2740 1 --2- 172.21.15.31:0/3233579968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556b6cdea8b0 0x556b6cdf77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.938+0000 7f246d600640 0 client.28234 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:49:13.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.938+0000 7f248d1e2740 1 -- 172.21.15.31:0/3233579968 >> 172.21.15.31:0/3233579968 conn(0x556b6cde4f80 msgr2=0x556b6cdeb640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:13.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.938+0000 7f248d1e2740 1 -- 172.21.15.31:0/3233579968 shutdown_connections 2024-03-21T10:49:13.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.938+0000 7f248d1e2740 1 -- 172.21.15.31:0/3233579968 wait complete. 2024-03-21T10:49:13.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.938+0000 7f248d1e2740 20 client.28234 trim_cache size 0 max 16384 2024-03-21T10:49:13.943 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:49:13.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.939+0000 7f2488e00640 1 -- 172.21.15.31:0/604701063 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24680a3140 msgr2=0x7f24680a5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:13.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.939+0000 7f2488e00640 1 --2- 172.21.15.31:0/604701063 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24680a3140 0x7f24680a5600 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x556b6cde5ff0 tx=0x7f247400b410 comp rx=0 tx=0).stop 2024-03-21T10:49:13.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.939+0000 7f2488e00640 1 -- 172.21.15.31:0/604701063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2484102f50 msgr2=0x7f24841007c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:13.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.939+0000 7f2488e00640 1 --2- 172.21.15.31:0/604701063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2484102f50 0x7f24841007c0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f247000e9c0 tx=0x7f247000ee80 comp rx=0 tx=0).stop 2024-03-21T10:49:13.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.939+0000 7f2488e00640 1 -- 172.21.15.31:0/604701063 shutdown_connections 2024-03-21T10:49:13.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.939+0000 7f2488e00640 1 --2- 172.21.15.31:0/604701063 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24680a3140 0x7f24680a5600 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.939+0000 7f2488e00640 1 --2- 172.21.15.31:0/604701063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f248410ba50 0x7f2484100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.939+0000 7f2488e00640 1 --2- 172.21.15.31:0/604701063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2484102f50 0x7f24841007c0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.939+0000 7f2488e00640 1 --2- 172.21.15.31:0/604701063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2484102630 0x7f2484100280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:13.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.939+0000 7f2488e00640 1 -- 172.21.15.31:0/604701063 >> 172.21.15.31:0/604701063 conn(0x7f2484077130 msgr2=0x7f24840fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:13.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.939+0000 7f2488e00640 1 -- 172.21.15.31:0/604701063 shutdown_connections 2024-03-21T10:49:13.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:13.939+0000 7f2488e00640 1 -- 172.21.15.31:0/604701063 wait complete. 2024-03-21T10:49:13.956 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:49:13.956 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-21T10:49:14.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.178+0000 7f809e800640 1 -- 172.21.15.31:0/2737470506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8098105820 msgr2=0x7f8098105c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:14.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.178+0000 7f809e800640 1 --2- 172.21.15.31:0/2737470506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8098105820 0x7f8098105c90 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f8090009be0 tx=0x7f809002bf80 comp rx=0 tx=0).stop 2024-03-21T10:49:14.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.179+0000 7f809e800640 1 -- 172.21.15.31:0/2737470506 shutdown_connections 2024-03-21T10:49:14.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.179+0000 7f809e800640 1 --2- 172.21.15.31:0/2737470506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8098105820 0x7f8098105c90 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.179+0000 7f809e800640 1 --2- 172.21.15.31:0/2737470506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8098100d40 0x7f8098105200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.179+0000 7f809e800640 1 --2- 172.21.15.31:0/2737470506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8098100420 0x7f8098100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.179+0000 7f809e800640 1 -- 172.21.15.31:0/2737470506 >> 172.21.15.31:0/2737470506 conn(0x7f8098077130 msgr2=0x7f80980fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:14.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.179+0000 7f809e800640 1 -- 172.21.15.31:0/2737470506 shutdown_connections 2024-03-21T10:49:14.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.179+0000 7f809e800640 1 -- 172.21.15.31:0/2737470506 wait complete. 2024-03-21T10:49:14.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.180+0000 7f809e800640 1 Processor -- start 2024-03-21T10:49:14.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.180+0000 7f809e800640 1 -- start start 2024-03-21T10:49:14.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.181+0000 7f809e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8098100420 0x7f809819e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:14.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.181+0000 7f809e800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8098100d40 0x7f809819e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:14.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.181+0000 7f809e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8098105820 0x7f80981a2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:14.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.181+0000 7f809e800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8098117e80 con 0x7f8098100420 2024-03-21T10:49:14.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.181+0000 7f809e800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8098117d00 con 0x7f8098105820 2024-03-21T10:49:14.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.181+0000 7f809e800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8098118000 con 0x7f8098100d40 2024-03-21T10:49:14.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.181+0000 7f809d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8098100420 0x7f809819e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:14.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.181+0000 7f809ca00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8098100d40 0x7f809819e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:14.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.181+0000 7f809ca00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8098100d40 0x7f809819e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53114/0 (socket says 172.21.15.31:53114) 2024-03-21T10:49:14.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.181+0000 7f809ca00640 1 -- 172.21.15.31:0/2683371758 learned_addr learned my addr 172.21.15.31:0/2683371758 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:14.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.181+0000 7f809de00640 1 --2- 172.21.15.31:0/2683371758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8098105820 0x7f80981a2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:14.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.182+0000 7f809ca00640 1 -- 172.21.15.31:0/2683371758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8098105820 msgr2=0x7f80981a2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:14.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.182+0000 7f809ca00640 1 --2- 172.21.15.31:0/2683371758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8098105820 0x7f80981a2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.182+0000 7f809ca00640 1 -- 172.21.15.31:0/2683371758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8098100420 msgr2=0x7f809819e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:14.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.182+0000 7f809ca00640 1 --2- 172.21.15.31:0/2683371758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8098100420 0x7f809819e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.182+0000 7f809ca00640 1 -- 172.21.15.31:0/2683371758 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80981a3350 con 0x7f8098100d40 2024-03-21T10:49:14.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.182+0000 7f809ca00640 1 --2- 172.21.15.31:0/2683371758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8098100d40 0x7f809819e8a0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f808400e9c0 tx=0x7f808400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:14.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.182+0000 7f8096000640 1 -- 172.21.15.31:0/2683371758 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808400ccc0 con 0x7f8098100d40 2024-03-21T10:49:14.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.182+0000 7f8096000640 1 -- 172.21.15.31:0/2683371758 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8084005500 con 0x7f8098100d40 2024-03-21T10:49:14.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.182+0000 7f809e800640 1 -- 172.21.15.31:0/2683371758 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80981a3640 con 0x7f8098100d40 2024-03-21T10:49:14.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.182+0000 7f8096000640 1 -- 172.21.15.31:0/2683371758 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8084010430 con 0x7f8098100d40 2024-03-21T10:49:14.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.182+0000 7f809e800640 1 -- 172.21.15.31:0/2683371758 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80981aaed0 con 0x7f8098100d40 2024-03-21T10:49:14.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.187+0000 7f8096000640 1 -- 172.21.15.31:0/2683371758 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f80840046c0 con 0x7f8098100d40 2024-03-21T10:49:14.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.187+0000 7f80a2b71740 1 Processor -- start 2024-03-21T10:49:14.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.187+0000 7f80a2b71740 1 -- start start 2024-03-21T10:49:14.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.187+0000 7f80a2b71740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c211a838b0 0x55c211a83c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:14.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.187+0000 7f80a2b71740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c211a84260 0x55c211a8c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:14.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.187+0000 7f80a2b71740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c211a8cde0 0x55c211a8f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:14.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.187+0000 7f80a2b71740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2119e72c0 con 0x55c211a84260 2024-03-21T10:49:14.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.187+0000 7f80a2b71740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c21198fc90 con 0x55c211a838b0 2024-03-21T10:49:14.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.187+0000 7f80a2b71740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2119e3360 con 0x55c211a8cde0 2024-03-21T10:49:14.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.188+0000 7f809d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c211a84260 0x55c211a8c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:14.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.188+0000 7f809ca00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c211a8cde0 0x55c211a8f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:14.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.188+0000 7f809de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c211a838b0 0x55c211a83c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:14.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.188+0000 7f809d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c211a84260 0x55c211a8c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45672/0 (socket says 172.21.15.31:45672) 2024-03-21T10:49:14.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.188+0000 7f809de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c211a838b0 0x55c211a83c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39398/0 (socket says 172.21.15.31:39398) 2024-03-21T10:49:14.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.188+0000 7f809d400640 1 -- 172.21.15.31:0/1764577606 learned_addr learned my addr 172.21.15.31:0/1764577606 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:14.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.188+0000 7f809ca00640 1 -- 172.21.15.31:0/1764577606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c211a838b0 msgr2=0x55c211a83c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:14.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.188+0000 7f809ca00640 1 --2- 172.21.15.31:0/1764577606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c211a838b0 0x55c211a83c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.188+0000 7f809ca00640 1 -- 172.21.15.31:0/1764577606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c211a84260 msgr2=0x55c211a8c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:14.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.188+0000 7f809ca00640 1 --2- 172.21.15.31:0/1764577606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c211a84260 0x55c211a8c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.188+0000 7f809ca00640 1 -- 172.21.15.31:0/1764577606 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2119b4080 con 0x55c211a8cde0 2024-03-21T10:49:14.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.188+0000 7f809d400640 1 --2- 172.21.15.31:0/1764577606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c211a84260 0x55c211a8c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:14.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.189+0000 7f8096000640 1 --2- 172.21.15.31:0/2683371758 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f80800a3110 0x7f80800a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.189+0000 7f8096000640 1 -- 172.21.15.31:0/2683371758 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f80840cfde0 con 0x7f8098100d40 2024-03-21T10:49:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.189+0000 7f809ca00640 1 --2- 172.21.15.31:0/1764577606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c211a8cde0 0x55c211a8f230 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f80840cd140 tx=0x7f80840d3e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6ca48ab19abef3d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.189+0000 7f809de00640 1 --2- 172.21.15.31:0/2683371758 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f80800a3110 0x7f80800a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.189+0000 7f808a600640 1 -- 172.21.15.31:0/1764577606 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8084098080 con 0x55c211a8cde0 2024-03-21T10:49:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.189+0000 7f808a600640 1 -- 172.21.15.31:0/1764577606 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f808409c4e0 con 0x55c211a8cde0 2024-03-21T10:49:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.189+0000 7f808a600640 1 -- 172.21.15.31:0/1764577606 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808401b160 con 0x55c211a8cde0 2024-03-21T10:49:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.190+0000 7f809de00640 1 --2- 172.21.15.31:0/2683371758 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f80800a3110 0x7f80800a55d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8090006010 tx=0x7f8090009280 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.190+0000 7f80a2b71740 1 -- 172.21.15.31:0/1764577606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c211a8cde0 msgr2=0x55c211a8f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.190+0000 7f80a2b71740 1 --2- 172.21.15.31:0/1764577606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c211a8cde0 0x55c211a8f230 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f80840cd140 tx=0x7f80840d3e90 comp rx=0 tx=0).stop 2024-03-21T10:49:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.190+0000 7f80a2b71740 1 -- 172.21.15.31:0/1764577606 shutdown_connections 2024-03-21T10:49:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.190+0000 7f80a2b71740 1 --2- 172.21.15.31:0/1764577606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c211a8cde0 0x55c211a8f230 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.190+0000 7f80a2b71740 1 --2- 172.21.15.31:0/1764577606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c211a84260 0x55c211a8c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.190+0000 7f80a2b71740 1 --2- 172.21.15.31:0/1764577606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c211a838b0 0x55c211a83c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.190+0000 7f80a2b71740 1 -- 172.21.15.31:0/1764577606 >> 172.21.15.31:0/1764577606 conn(0x55c211a7df80 msgr2=0x55c211a7e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:14.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.190+0000 7f80a2b71740 1 -- 172.21.15.31:0/1764577606 shutdown_connections 2024-03-21T10:49:14.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.190+0000 7f80a2b71740 1 -- 172.21.15.31:0/1764577606 wait complete. 2024-03-21T10:49:14.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.191+0000 7f80a2b71740 1 -- start start 2024-03-21T10:49:14.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.191+0000 7f80a2b71740 1 Processor -- start 2024-03-21T10:49:14.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.192+0000 7f80a2b71740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c211a838b0 0x55c211a8c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:14.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.192+0000 7f80a2b71740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c211a84260 0x55c211a89af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:14.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.192+0000 7f80a2b71740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c211a8cde0 0x55c211a8a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:14.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.192+0000 7f80a2b71740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c211a8fd50 con 0x55c211a838b0 2024-03-21T10:49:14.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.192+0000 7f80a2b71740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2119a30d0 con 0x55c211a84260 2024-03-21T10:49:14.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.192+0000 7f80a2b71740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c211a8fed0 con 0x55c211a8cde0 2024-03-21T10:49:14.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.192+0000 7f80a2b71740 10 client.0 _ll_register_callbacks cb 0x55c2119a7e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:14.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.192+0000 7f809ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c211a84260 0x55c211a89af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:14.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.192+0000 7f809ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c211a84260 0x55c211a89af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39400/0 (socket says 172.21.15.31:39400) 2024-03-21T10:49:14.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.192+0000 7f809ca00640 1 -- 172.21.15.31:0/4168901186 learned_addr learned my addr 172.21.15.31:0/4168901186 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:14.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.192+0000 7f809de00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c211a8cde0 0x55c211a8a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:14.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.192+0000 7f809d400640 1 --2- 172.21.15.31:0/4168901186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c211a838b0 0x55c211a8c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:14.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.192+0000 7f809ca00640 1 -- 172.21.15.31:0/4168901186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c211a8cde0 msgr2=0x55c211a8a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:14.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.192+0000 7f809ca00640 1 --2- 172.21.15.31:0/4168901186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c211a8cde0 0x55c211a8a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.192+0000 7f809ca00640 1 -- 172.21.15.31:0/4168901186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c211a838b0 msgr2=0x55c211a8c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:14.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.192+0000 7f809ca00640 1 --2- 172.21.15.31:0/4168901186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c211a838b0 0x55c211a8c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.192+0000 7f809ca00640 1 -- 172.21.15.31:0/4168901186 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c2116fb3f0 con 0x55c211a84260 2024-03-21T10:49:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.193+0000 7f809ca00640 1 --2- 172.21.15.31:0/4168901186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c211a84260 0x55c211a89af0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f8084000c00 tx=0x7f8084004b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33166a00bcb9defe server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.193+0000 7f807ea00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:49:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.193+0000 7f807ea00640 1 -- 172.21.15.31:0/4168901186 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80840b7070 con 0x55c211a84260 2024-03-21T10:49:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.193+0000 7f80a2b71740 1 -- 172.21.15.31:0/4168901186 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f808409c4e0 con 0x55c211a84260 2024-03-21T10:49:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.193+0000 7f807ea00640 1 -- 172.21.15.31:0/4168901186 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f808409c250 con 0x55c211a84260 2024-03-21T10:49:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.193+0000 7f807ea00640 1 -- 172.21.15.31:0/4168901186 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80840b4070 con 0x55c211a84260 2024-03-21T10:49:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.193+0000 7f80a2b71740 10 client.28319 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.193+0000 7f80a2b71740 1 -- 172.21.15.31:0/4168901186 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55c2117c57d0 con 0x55c211a84260 2024-03-21T10:49:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.193+0000 7f807ea00640 1 -- 172.21.15.31:0/4168901186 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f808409c4e0 con 0x55c211a84260 2024-03-21T10:49:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.194+0000 7f807ea00640 1 -- 172.21.15.31:0/4168901186 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f80840ba020 con 0x55c211a84260 2024-03-21T10:49:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.194+0000 7f80a2b71740 10 client.28319 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.194+0000 7f80a2b71740 10 client.28319 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:49:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.194+0000 7f80a2b71740 20 client.28319 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.194+0000 7f80a2b71740 1 --2- 172.21.15.31:0/4168901186 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c211a9cb70 0x55c211a9ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:14.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.194+0000 7f80a2b71740 4 client.28319 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:49:14.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.194+0000 7f80a2b71740 1 -- 172.21.15.31:0/4168901186 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c2119f4800 con 0x55c211a9cb70 2024-03-21T10:49:14.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.194+0000 7f809d400640 1 --2- 172.21.15.31:0/4168901186 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c211a9cb70 0x55c211a9ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:14.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.195+0000 7f809d400640 1 --2- 172.21.15.31:0/4168901186 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c211a9cb70 0x55c211a9ef60 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f79c38cc845cae server_cookie=e1c52573a52aa79b in_seq=0 out_seq=0 2024-03-21T10:49:14.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.195+0000 7f807ea00640 10 client.28319 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:14.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.196+0000 7f807ea00640 1 -- 172.21.15.31:0/4168901186 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c2119f4800 con 0x55c211a9cb70 2024-03-21T10:49:14.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.196+0000 7f807ea00640 10 client.28319 handle_command_reply: tid=0 2024-03-21T10:49:14.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.196+0000 7f80a2b71740 1 client.28319 shutdown 2024-03-21T10:49:14.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.196+0000 7f80a2b71740 1 -- 172.21.15.31:0/4168901186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c211a84260 msgr2=0x55c211a89af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:14.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.196+0000 7f80a2b71740 1 --2- 172.21.15.31:0/4168901186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c211a84260 0x55c211a89af0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f8084000c00 tx=0x7f8084004b00 comp rx=0 tx=0).stop 2024-03-21T10:49:14.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.196+0000 7f80a2b71740 1 -- 172.21.15.31:0/4168901186 shutdown_connections 2024-03-21T10:49:14.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.196+0000 7f80a2b71740 1 --2- 172.21.15.31:0/4168901186 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c211a9cb70 0x55c211a9ef60 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.196+0000 7f80a2b71740 1 --2- 172.21.15.31:0/4168901186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c211a8cde0 0x55c211a8a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.196+0000 7f80a2b71740 1 --2- 172.21.15.31:0/4168901186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c211a84260 0x55c211a89af0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.196+0000 7f807ea00640 0 client.28319 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:14.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.196+0000 7f80a2b71740 1 --2- 172.21.15.31:0/4168901186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c211a838b0 0x55c211a8c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.196+0000 7f80a2b71740 1 -- 172.21.15.31:0/4168901186 >> 172.21.15.31:0/4168901186 conn(0x55c211a7df80 msgr2=0x55c211a7fcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:14.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.196+0000 7f80a2b71740 1 -- 172.21.15.31:0/4168901186 shutdown_connections 2024-03-21T10:49:14.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.196+0000 7f80a2b71740 1 -- 172.21.15.31:0/4168901186 wait complete. 2024-03-21T10:49:14.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.196+0000 7f80a2b71740 20 client.28319 trim_cache size 0 max 16384 2024-03-21T10:49:14.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.210+0000 7f80a2b71740 1 Processor -- start 2024-03-21T10:49:14.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.210+0000 7f80a2b71740 1 -- start start 2024-03-21T10:49:14.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.210+0000 7f80a2b71740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c211a838b0 0x55c211a99130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:14.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.210+0000 7f80a2b71740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c211a84260 0x55c211a8c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:14.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.210+0000 7f80a2b71740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c211a8cde0 0x55c211a8c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:14.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.210+0000 7f80a2b71740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80981b7400 con 0x55c211a8cde0 2024-03-21T10:49:14.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.210+0000 7f80a2b71740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80981b4f00 con 0x55c211a84260 2024-03-21T10:49:14.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.210+0000 7f80a2b71740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80981b2fc0 con 0x55c211a838b0 2024-03-21T10:49:14.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.211+0000 7f809d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c211a838b0 0x55c211a99130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:14.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.211+0000 7f809ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c211a84260 0x55c211a8c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:14.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.211+0000 7f809d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c211a838b0 0x55c211a99130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53144/0 (socket says 172.21.15.31:53144) 2024-03-21T10:49:14.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.211+0000 7f809d400640 1 -- 172.21.15.31:0/1625861994 learned_addr learned my addr 172.21.15.31:0/1625861994 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:14.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.211+0000 7f809ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c211a84260 0x55c211a8c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39412/0 (socket says 172.21.15.31:39412) 2024-03-21T10:49:14.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.211+0000 7f809de00640 1 --2- 172.21.15.31:0/1625861994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c211a8cde0 0x55c211a8c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:14.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.211+0000 7f809d400640 1 -- 172.21.15.31:0/1625861994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c211a84260 msgr2=0x55c211a8c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:14.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.211+0000 7f809d400640 1 --2- 172.21.15.31:0/1625861994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c211a84260 0x55c211a8c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.211+0000 7f809d400640 1 -- 172.21.15.31:0/1625861994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c211a8cde0 msgr2=0x55c211a8c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:14.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.211+0000 7f809d400640 1 --2- 172.21.15.31:0/1625861994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c211a8cde0 0x55c211a8c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.211+0000 7f809d400640 1 -- 172.21.15.31:0/1625861994 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c211a8fe70 con 0x55c211a838b0 2024-03-21T10:49:14.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.211+0000 7f809d400640 1 --2- 172.21.15.31:0/1625861994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c211a838b0 0x55c211a99130 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f808c005b90 tx=0x7f808c005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cd091ee2ef7aa10 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:14.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.212+0000 7f8089c00640 1 -- 172.21.15.31:0/1625861994 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808c004020 con 0x55c211a838b0 2024-03-21T10:49:14.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.212+0000 7f8089c00640 1 -- 172.21.15.31:0/1625861994 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f808c011ba0 con 0x55c211a838b0 2024-03-21T10:49:14.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.212+0000 7f8089c00640 1 -- 172.21.15.31:0/1625861994 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808c0039e0 con 0x55c211a838b0 2024-03-21T10:49:14.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.212+0000 7f80a2b71740 1 -- 172.21.15.31:0/1625861994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c211a838b0 msgr2=0x55c211a99130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:14.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.212+0000 7f80a2b71740 1 --2- 172.21.15.31:0/1625861994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c211a838b0 0x55c211a99130 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f808c005b90 tx=0x7f808c005f90 comp rx=0 tx=0).stop 2024-03-21T10:49:14.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.213+0000 7f80a2b71740 1 -- 172.21.15.31:0/1625861994 shutdown_connections 2024-03-21T10:49:14.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.213+0000 7f80a2b71740 1 --2- 172.21.15.31:0/1625861994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c211a8cde0 0x55c211a8c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.213+0000 7f80a2b71740 1 --2- 172.21.15.31:0/1625861994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c211a84260 0x55c211a8c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.213+0000 7f80a2b71740 1 --2- 172.21.15.31:0/1625861994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c211a838b0 0x55c211a99130 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.213+0000 7f80a2b71740 1 -- 172.21.15.31:0/1625861994 >> 172.21.15.31:0/1625861994 conn(0x55c211a7df80 msgr2=0x55c211a7d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:14.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.213+0000 7f80a2b71740 1 -- 172.21.15.31:0/1625861994 shutdown_connections 2024-03-21T10:49:14.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.213+0000 7f80a2b71740 1 -- 172.21.15.31:0/1625861994 wait complete. 2024-03-21T10:49:14.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.213+0000 7f80a2b71740 1 -- start start 2024-03-21T10:49:14.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.213+0000 7f80a2b71740 1 Processor -- start 2024-03-21T10:49:14.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.214+0000 7f80a2b71740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c211a838b0 0x55c211a9fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:14.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.214+0000 7f80a2b71740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c211a84260 0x55c211a9ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:14.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.214+0000 7f80a2b71740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c211a8cde0 0x55c211af70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:14.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.214+0000 7f80a2b71740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80981b35c0 con 0x55c211a8cde0 2024-03-21T10:49:14.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.214+0000 7f80a2b71740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80981b48a0 con 0x55c211a838b0 2024-03-21T10:49:14.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.214+0000 7f80a2b71740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80981b3be0 con 0x55c211a84260 2024-03-21T10:49:14.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.214+0000 7f80a2b71740 10 client.0 _ll_register_callbacks cb 0x7f808c0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:14.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.214+0000 7f809d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c211a838b0 0x55c211a9fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:14.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.214+0000 7f809ca00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c211a84260 0x55c211a9ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:14.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.214+0000 7f809de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c211a8cde0 0x55c211af70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:14.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.214+0000 7f809d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c211a838b0 0x55c211a9fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39428/0 (socket says 172.21.15.31:39428) 2024-03-21T10:49:14.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.214+0000 7f809ca00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c211a84260 0x55c211a9ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53152/0 (socket says 172.21.15.31:53152) 2024-03-21T10:49:14.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.214+0000 7f809d400640 1 -- 172.21.15.31:0/2287980321 learned_addr learned my addr 172.21.15.31:0/2287980321 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:14.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.215+0000 7f809d400640 1 -- 172.21.15.31:0/2287980321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c211a84260 msgr2=0x55c211a9ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:14.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.215+0000 7f809d400640 1 --2- 172.21.15.31:0/2287980321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c211a84260 0x55c211a9ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.215+0000 7f809d400640 1 -- 172.21.15.31:0/2287980321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c211a8cde0 msgr2=0x55c211af70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:14.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.215+0000 7f809d400640 1 --2- 172.21.15.31:0/2287980321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c211a8cde0 0x55c211af70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.215+0000 7f809d400640 1 -- 172.21.15.31:0/2287980321 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c211af75e0 con 0x55c211a838b0 2024-03-21T10:49:14.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.215+0000 7f809ca00640 1 --2- 172.21.15.31:0/2287980321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c211a84260 0x55c211a9ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:14.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.215+0000 7f809d400640 1 --2- 172.21.15.31:0/2287980321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c211a838b0 0x55c211a9fa10 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f808c009b30 tx=0x7f808c009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8a18eff0ae29d49 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:14.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.215+0000 7f807ea00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:49:14.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.215+0000 7f807ea00640 1 -- 172.21.15.31:0/2287980321 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808c02d020 con 0x55c211a838b0 2024-03-21T10:49:14.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.215+0000 7f80a2b71740 1 -- 172.21.15.31:0/2287980321 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f808c011ba0 con 0x55c211a838b0 2024-03-21T10:49:14.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.215+0000 7f807ea00640 1 -- 172.21.15.31:0/2287980321 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f808c0116c0 con 0x55c211a838b0 2024-03-21T10:49:14.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.216+0000 7f807ea00640 1 -- 172.21.15.31:0/2287980321 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f808c0118c0 con 0x55c211a838b0 2024-03-21T10:49:14.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.216+0000 7f80a2b71740 10 client.28325 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:14.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.216+0000 7f80a2b71740 1 -- 172.21.15.31:0/2287980321 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55c211af8890 con 0x55c211a838b0 2024-03-21T10:49:14.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.216+0000 7f807ea00640 1 -- 172.21.15.31:0/2287980321 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f808c011ba0 con 0x55c211a838b0 2024-03-21T10:49:14.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.216+0000 7f807ea00640 1 -- 172.21.15.31:0/2287980321 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f808c029450 con 0x55c211a838b0 2024-03-21T10:49:14.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.216+0000 7f80a2b71740 10 client.28325 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:14.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.216+0000 7f80a2b71740 10 client.28325 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:49:14.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.216+0000 7f80a2b71740 20 client.28325 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:14.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.216+0000 7f80a2b71740 1 --2- 172.21.15.31:0/2287980321 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c211af8f90 0x55c211afb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:14.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.216+0000 7f80a2b71740 4 client.28325 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:49:14.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.216+0000 7f80a2b71740 1 -- 172.21.15.31:0/2287980321 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c2119eeb50 con 0x55c211af8f90 2024-03-21T10:49:14.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.217+0000 7f809ca00640 1 --2- 172.21.15.31:0/2287980321 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c211af8f90 0x55c211afb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:14.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.217+0000 7f809ca00640 1 --2- 172.21.15.31:0/2287980321 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c211af8f90 0x55c211afb3b0 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aaee3b24d1238292 server_cookie=5c57f37db30741be in_seq=0 out_seq=0 2024-03-21T10:49:14.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.217+0000 7f807ea00640 10 client.28325 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:14.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.217+0000 7f807ea00640 1 -- 172.21.15.31:0/2287980321 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c2119eeb50 con 0x55c211af8f90 2024-03-21T10:49:14.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.217+0000 7f807ea00640 10 client.28325 handle_command_reply: tid=0 2024-03-21T10:49:14.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.217+0000 7f80a2b71740 1 client.28325 shutdown 2024-03-21T10:49:14.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.217+0000 7f80a2b71740 1 -- 172.21.15.31:0/2287980321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c211a838b0 msgr2=0x55c211a9fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:14.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.217+0000 7f80a2b71740 1 --2- 172.21.15.31:0/2287980321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c211a838b0 0x55c211a9fa10 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f808c009b30 tx=0x7f808c009b60 comp rx=0 tx=0).stop 2024-03-21T10:49:14.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.217+0000 7f80a2b71740 1 -- 172.21.15.31:0/2287980321 shutdown_connections 2024-03-21T10:49:14.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.217+0000 7f80a2b71740 1 --2- 172.21.15.31:0/2287980321 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c211af8f90 0x55c211afb3b0 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.217+0000 7f80a2b71740 1 --2- 172.21.15.31:0/2287980321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c211a8cde0 0x55c211af70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.217+0000 7f80a2b71740 1 --2- 172.21.15.31:0/2287980321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c211a84260 0x55c211a9ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.217+0000 7f80a2b71740 1 --2- 172.21.15.31:0/2287980321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c211a838b0 0x55c211a9fa10 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.217+0000 7f807ea00640 0 client.28325 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:14.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.217+0000 7f80a2b71740 1 -- 172.21.15.31:0/2287980321 >> 172.21.15.31:0/2287980321 conn(0x55c211a7df80 msgr2=0x55c211a84640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:14.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.218+0000 7f80a2b71740 1 -- 172.21.15.31:0/2287980321 shutdown_connections 2024-03-21T10:49:14.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.218+0000 7f80a2b71740 1 -- 172.21.15.31:0/2287980321 wait complete. 2024-03-21T10:49:14.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.218+0000 7f80a2b71740 20 client.28325 trim_cache size 0 max 16384 2024-03-21T10:49:14.218 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:49:14.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.218+0000 7f809e800640 1 -- 172.21.15.31:0/2683371758 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f80800a3110 msgr2=0x7f80800a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:14.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.218+0000 7f809e800640 1 --2- 172.21.15.31:0/2683371758 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f80800a3110 0x7f80800a55d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8090006010 tx=0x7f8090009280 comp rx=0 tx=0).stop 2024-03-21T10:49:14.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.218+0000 7f809e800640 1 -- 172.21.15.31:0/2683371758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8098100d40 msgr2=0x7f809819e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:14.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.218+0000 7f809e800640 1 --2- 172.21.15.31:0/2683371758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8098100d40 0x7f809819e8a0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f808400e9c0 tx=0x7f808400ee80 comp rx=0 tx=0).stop 2024-03-21T10:49:14.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.219+0000 7f809e800640 1 -- 172.21.15.31:0/2683371758 shutdown_connections 2024-03-21T10:49:14.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.219+0000 7f809e800640 1 --2- 172.21.15.31:0/2683371758 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f80800a3110 0x7f80800a55d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.219+0000 7f809e800640 1 --2- 172.21.15.31:0/2683371758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8098105820 0x7f80981a2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.219+0000 7f809e800640 1 --2- 172.21.15.31:0/2683371758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8098100d40 0x7f809819e8a0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.219+0000 7f809e800640 1 --2- 172.21.15.31:0/2683371758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8098100420 0x7f809819e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:14.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.219+0000 7f809e800640 1 -- 172.21.15.31:0/2683371758 >> 172.21.15.31:0/2683371758 conn(0x7f8098077130 msgr2=0x7f80981031d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:14.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.219+0000 7f809e800640 1 -- 172.21.15.31:0/2683371758 shutdown_connections 2024-03-21T10:49:14.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:14.219+0000 7f809e800640 1 -- 172.21.15.31:0/2683371758 wait complete. 2024-03-21T10:49:14.230 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:14 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1963094256' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:49:14.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:14 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1963094256' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:49:14.484 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:14 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1963094256' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:49:15.234 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-21T10:49:15.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:49:15.235 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-21T10:49:15.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:15 smithi040 ceph-mon[36154]: pgmap v332: 129 pgs: 11 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 85 active+clean; 29 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 30 KiB/s wr, 10 op/s 2024-03-21T10:49:15.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.468+0000 7f5914e00640 1 -- 172.21.15.31:0/3590896470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5910102630 msgr2=0x7f5910102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:15.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.468+0000 7f5914e00640 1 --2- 172.21.15.31:0/3590896470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5910102630 0x7f5910102a10 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f5900009900 tx=0x7f590002bcc0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.468+0000 7f5914e00640 1 -- 172.21.15.31:0/3590896470 shutdown_connections 2024-03-21T10:49:15.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.468+0000 7f5914e00640 1 --2- 172.21.15.31:0/3590896470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f591010ba50 0x7f591010ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.468+0000 7f5914e00640 1 --2- 172.21.15.31:0/3590896470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5910102f50 0x7f591010b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.468+0000 7f5914e00640 1 --2- 172.21.15.31:0/3590896470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5910102630 0x7f5910102a10 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.468+0000 7f5914e00640 1 -- 172.21.15.31:0/3590896470 >> 172.21.15.31:0/3590896470 conn(0x7f5910077130 msgr2=0x7f59100fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:15.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.468+0000 7f5914e00640 1 -- 172.21.15.31:0/3590896470 shutdown_connections 2024-03-21T10:49:15.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.468+0000 7f5914e00640 1 -- 172.21.15.31:0/3590896470 wait complete. 2024-03-21T10:49:15.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.470+0000 7f5914e00640 1 Processor -- start 2024-03-21T10:49:15.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.470+0000 7f5914e00640 1 -- start start 2024-03-21T10:49:15.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.470+0000 7f5914e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5910102630 0x7f5910199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.470+0000 7f5914e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5910102f50 0x7f591019a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.470+0000 7f5914e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f591010ba50 0x7f591019e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.470+0000 7f5914e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5910111820 con 0x7f5910102f50 2024-03-21T10:49:15.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.470+0000 7f5914e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59101116a0 con 0x7f5910102630 2024-03-21T10:49:15.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.470+0000 7f5914e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59101119a0 con 0x7f591010ba50 2024-03-21T10:49:15.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.471+0000 7f590f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5910102630 0x7f5910199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.471+0000 7f590fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f591010ba50 0x7f591019e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.471+0000 7f590f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5910102630 0x7f5910199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39446/0 (socket says 172.21.15.31:39446) 2024-03-21T10:49:15.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.471+0000 7f590ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5910102f50 0x7f591019a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.471+0000 7f590f400640 1 -- 172.21.15.31:0/4115741546 learned_addr learned my addr 172.21.15.31:0/4115741546 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:15.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.471+0000 7f590fe00640 1 -- 172.21.15.31:0/4115741546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5910102630 msgr2=0x7f5910199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:15.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.471+0000 7f590fe00640 1 --2- 172.21.15.31:0/4115741546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5910102630 0x7f5910199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.471+0000 7f590fe00640 1 -- 172.21.15.31:0/4115741546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5910102f50 msgr2=0x7f591019a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:15.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.471+0000 7f590fe00640 1 --2- 172.21.15.31:0/4115741546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5910102f50 0x7f591019a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.471+0000 7f590fe00640 1 -- 172.21.15.31:0/4115741546 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f591019ef50 con 0x7f591010ba50 2024-03-21T10:49:15.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.471+0000 7f590fe00640 1 --2- 172.21.15.31:0/4115741546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f591010ba50 0x7f591019e820 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f5908009fd0 tx=0x7f590800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:15.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.472+0000 7f5907e00640 1 -- 172.21.15.31:0/4115741546 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590800ce20 con 0x7f591010ba50 2024-03-21T10:49:15.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.472+0000 7f5907e00640 1 -- 172.21.15.31:0/4115741546 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5908005020 con 0x7f591010ba50 2024-03-21T10:49:15.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.472+0000 7f5907e00640 1 -- 172.21.15.31:0/4115741546 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5908010720 con 0x7f591010ba50 2024-03-21T10:49:15.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.472+0000 7f5914e00640 1 -- 172.21.15.31:0/4115741546 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f591019f240 con 0x7f591010ba50 2024-03-21T10:49:15.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.472+0000 7f5914e00640 1 -- 172.21.15.31:0/4115741546 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59101a6ad0 con 0x7f591010ba50 2024-03-21T10:49:15.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.475+0000 7f591923a740 1 Processor -- start 2024-03-21T10:49:15.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.475+0000 7f5907e00640 1 -- 172.21.15.31:0/4115741546 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f5908005540 con 0x7f591010ba50 2024-03-21T10:49:15.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.476+0000 7f591923a740 1 -- start start 2024-03-21T10:49:15.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.476+0000 7f591923a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5de58e290 0x55c5de58e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.476+0000 7f591923a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5de58ebb0 0x55c5de597250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.476+0000 7f591923a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5de597790 0x55c5de599b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.476+0000 7f591923a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5de4f2eb0 con 0x55c5de58e290 2024-03-21T10:49:15.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.476+0000 7f591923a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5de49ba60 con 0x55c5de58ebb0 2024-03-21T10:49:15.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.476+0000 7f591923a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5de4eef50 con 0x55c5de597790 2024-03-21T10:49:15.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.476+0000 7f590f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5de58ebb0 0x55c5de597250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.476+0000 7f590f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5de58ebb0 0x55c5de597250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39450/0 (socket says 172.21.15.31:39450) 2024-03-21T10:49:15.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.476+0000 7f590f400640 1 -- 172.21.15.31:0/2190380096 learned_addr learned my addr 172.21.15.31:0/2190380096 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:15.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.476+0000 7f590ea00640 1 --2- 172.21.15.31:0/2190380096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5de58e290 0x55c5de58e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.476+0000 7f590ea00640 1 --2- 172.21.15.31:0/2190380096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5de597790 0x55c5de599b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.477+0000 7f590f400640 1 -- 172.21.15.31:0/2190380096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5de597790 msgr2=0x55c5de599b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:15.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.477+0000 7f590f400640 1 --2- 172.21.15.31:0/2190380096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5de597790 0x55c5de599b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.477+0000 7f590f400640 1 -- 172.21.15.31:0/2190380096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5de58e290 msgr2=0x55c5de58e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:15.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.477+0000 7f590f400640 1 --2- 172.21.15.31:0/2190380096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5de58e290 0x55c5de58e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.477+0000 7f590f400640 1 -- 172.21.15.31:0/2190380096 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5de2cf7e0 con 0x55c5de58ebb0 2024-03-21T10:49:15.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.477+0000 7f590f400640 1 --2- 172.21.15.31:0/2190380096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5de58ebb0 0x55c5de597250 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f59000095e0 tx=0x7f5900009610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=879b4e332beb0a62 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:15.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.477+0000 7f5904c00640 1 -- 172.21.15.31:0/2190380096 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5900045070 con 0x55c5de58ebb0 2024-03-21T10:49:15.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.477+0000 7f5904c00640 1 -- 172.21.15.31:0/2190380096 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5900004040 con 0x55c5de58ebb0 2024-03-21T10:49:15.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.477+0000 7f5904c00640 1 -- 172.21.15.31:0/2190380096 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5900004ee0 con 0x55c5de58ebb0 2024-03-21T10:49:15.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.478+0000 7f5907e00640 1 --2- 172.21.15.31:0/4115741546 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f58f40a3110 0x7f58f40a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.478+0000 7f5907e00640 1 -- 172.21.15.31:0/4115741546 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f59080d0150 con 0x7f591010ba50 2024-03-21T10:49:15.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.478+0000 7f590ea00640 1 --2- 172.21.15.31:0/4115741546 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f58f40a3110 0x7f58f40a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.478+0000 7f591923a740 1 -- 172.21.15.31:0/2190380096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5de58ebb0 msgr2=0x55c5de597250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:15.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.478+0000 7f591923a740 1 --2- 172.21.15.31:0/2190380096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5de58ebb0 0x55c5de597250 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f59000095e0 tx=0x7f5900009610 comp rx=0 tx=0).stop 2024-03-21T10:49:15.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.478+0000 7f591923a740 1 -- 172.21.15.31:0/2190380096 shutdown_connections 2024-03-21T10:49:15.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.478+0000 7f591923a740 1 --2- 172.21.15.31:0/2190380096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5de597790 0x55c5de599b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.478+0000 7f591923a740 1 --2- 172.21.15.31:0/2190380096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5de58ebb0 0x55c5de597250 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.478+0000 7f591923a740 1 --2- 172.21.15.31:0/2190380096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5de58e290 0x55c5de58e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.478+0000 7f591923a740 1 -- 172.21.15.31:0/2190380096 >> 172.21.15.31:0/2190380096 conn(0x55c5de5889c0 msgr2=0x55c5de588da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:15.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.479+0000 7f591923a740 1 -- 172.21.15.31:0/2190380096 shutdown_connections 2024-03-21T10:49:15.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.479+0000 7f591923a740 1 -- 172.21.15.31:0/2190380096 wait complete. 2024-03-21T10:49:15.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.479+0000 7f590ea00640 1 --2- 172.21.15.31:0/4115741546 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f58f40a3110 0x7f58f40a55d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f58fc0058a0 tx=0x7f58fc005c90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:15.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.479+0000 7f591923a740 1 -- start start 2024-03-21T10:49:15.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.480+0000 7f591923a740 1 Processor -- start 2024-03-21T10:49:15.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.480+0000 7f591923a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5de58e290 0x55c5de597100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.480+0000 7f591923a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5de58ebb0 0x55c5de594490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.480+0000 7f591923a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5de597790 0x55c5de5949f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.480+0000 7f591923a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5de59a520 con 0x55c5de597790 2024-03-21T10:49:15.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.480+0000 7f591923a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5de4aeeb0 con 0x55c5de58ebb0 2024-03-21T10:49:15.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.480+0000 7f591923a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5de59a6a0 con 0x55c5de58e290 2024-03-21T10:49:15.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.480+0000 7f591923a740 10 client.0 _ll_register_callbacks cb 0x55c5de4a3150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:15.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.481+0000 7f590ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5de58ebb0 0x55c5de594490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.480+0000 7f590f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5de58e290 0x55c5de597100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.481+0000 7f590f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5de58e290 0x55c5de597100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53196/0 (socket says 172.21.15.31:53196) 2024-03-21T10:49:15.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.481+0000 7f590f400640 1 -- 172.21.15.31:0/1409731041 learned_addr learned my addr 172.21.15.31:0/1409731041 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:15.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.481+0000 7f590fe00640 1 --2- 172.21.15.31:0/1409731041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5de597790 0x55c5de5949f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.481+0000 7f590f400640 1 -- 172.21.15.31:0/1409731041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5de58ebb0 msgr2=0x55c5de594490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:15.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.481+0000 7f590f400640 1 --2- 172.21.15.31:0/1409731041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5de58ebb0 0x55c5de594490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.481+0000 7f590f400640 1 -- 172.21.15.31:0/1409731041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5de597790 msgr2=0x55c5de5949f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:15.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.481+0000 7f590f400640 1 --2- 172.21.15.31:0/1409731041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5de597790 0x55c5de5949f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.481+0000 7f590f400640 1 -- 172.21.15.31:0/1409731041 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c5de32ff70 con 0x55c5de58e290 2024-03-21T10:49:15.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.481+0000 7f590f400640 1 --2- 172.21.15.31:0/1409731041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5de58e290 0x55c5de597100 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f5900009850 tx=0x7f5900005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3a7038b4519dc3c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:15.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.481+0000 7f58f9600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:49:15.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.482+0000 7f58f9600640 1 -- 172.21.15.31:0/1409731041 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5900045070 con 0x55c5de58e290 2024-03-21T10:49:15.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.482+0000 7f58f9600640 1 -- 172.21.15.31:0/1409731041 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f5900004ec0 con 0x55c5de58e290 2024-03-21T10:49:15.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.482+0000 7f591923a740 1 -- 172.21.15.31:0/1409731041 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5900004040 con 0x55c5de58e290 2024-03-21T10:49:15.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.482+0000 7f58f9600640 1 -- 172.21.15.31:0/1409731041 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f590005b050 con 0x55c5de58e290 2024-03-21T10:49:15.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.482+0000 7f591923a740 10 client.28264 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:15.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.482+0000 7f591923a740 1 -- 172.21.15.31:0/1409731041 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55c5de1c7170 con 0x55c5de58e290 2024-03-21T10:49:15.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.482+0000 7f58f9600640 1 -- 172.21.15.31:0/1409731041 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5900004040 con 0x55c5de58e290 2024-03-21T10:49:15.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.482+0000 7f58f9600640 1 -- 172.21.15.31:0/1409731041 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f5900003790 con 0x55c5de58e290 2024-03-21T10:49:15.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.483+0000 7f591923a740 10 client.28264 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:15.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.483+0000 7f591923a740 10 client.28264 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:49:15.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.483+0000 7f591923a740 20 client.28264 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:15.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.483+0000 7f591923a740 1 --2- 172.21.15.31:0/1409731041 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c5de5a7640 0x55c5de5a9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.483+0000 7f591923a740 4 client.28264 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:49:15.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.483+0000 7f591923a740 1 -- 172.21.15.31:0/1409731041 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c5de5003f0 con 0x55c5de5a7640 2024-03-21T10:49:15.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.483+0000 7f590ea00640 1 --2- 172.21.15.31:0/1409731041 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c5de5a7640 0x55c5de5a9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.483+0000 7f590ea00640 1 --2- 172.21.15.31:0/1409731041 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c5de5a7640 0x55c5de5a9a30 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8bd1ebc8d6918db server_cookie=906cbe6664dfdcd1 in_seq=0 out_seq=0 2024-03-21T10:49:15.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.483+0000 7f58f9600640 10 client.28264 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:15.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.484+0000 7f58f9600640 1 -- 172.21.15.31:0/1409731041 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c5de5003f0 con 0x55c5de5a7640 2024-03-21T10:49:15.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.484+0000 7f58f9600640 10 client.28264 handle_command_reply: tid=0 2024-03-21T10:49:15.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.484+0000 7f591923a740 1 client.28264 shutdown 2024-03-21T10:49:15.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.484+0000 7f591923a740 1 -- 172.21.15.31:0/1409731041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5de58e290 msgr2=0x55c5de597100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:15.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.484+0000 7f591923a740 1 --2- 172.21.15.31:0/1409731041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5de58e290 0x55c5de597100 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f5900009850 tx=0x7f5900005fd0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.484+0000 7f591923a740 1 -- 172.21.15.31:0/1409731041 shutdown_connections 2024-03-21T10:49:15.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.484+0000 7f591923a740 1 --2- 172.21.15.31:0/1409731041 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c5de5a7640 0x55c5de5a9a30 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.484+0000 7f591923a740 1 --2- 172.21.15.31:0/1409731041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5de597790 0x55c5de5949f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.484+0000 7f591923a740 1 --2- 172.21.15.31:0/1409731041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5de58ebb0 0x55c5de594490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.484+0000 7f58f9600640 0 client.28264 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:15.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.484+0000 7f591923a740 1 --2- 172.21.15.31:0/1409731041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5de58e290 0x55c5de597100 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.484+0000 7f591923a740 1 -- 172.21.15.31:0/1409731041 >> 172.21.15.31:0/1409731041 conn(0x55c5de5889c0 msgr2=0x55c5de58a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:15.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.485+0000 7f591923a740 1 -- 172.21.15.31:0/1409731041 shutdown_connections 2024-03-21T10:49:15.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.485+0000 7f591923a740 1 -- 172.21.15.31:0/1409731041 wait complete. 2024-03-21T10:49:15.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.485+0000 7f591923a740 20 client.28264 trim_cache size 0 max 16384 2024-03-21T10:49:15.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.502+0000 7f591923a740 1 Processor -- start 2024-03-21T10:49:15.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.502+0000 7f591923a740 1 -- start start 2024-03-21T10:49:15.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.503+0000 7f591923a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5de58e290 0x55c5de596bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.503+0000 7f591923a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5de58ebb0 0x55c5de597100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.503+0000 7f591923a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5de597790 0x55c5de5a2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.503+0000 7f591923a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59101b2f80 con 0x55c5de58ebb0 2024-03-21T10:49:15.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.503+0000 7f591923a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59101b0a80 con 0x55c5de597790 2024-03-21T10:49:15.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.503+0000 7f591923a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59101aeb40 con 0x55c5de58e290 2024-03-21T10:49:15.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.503+0000 7f590f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5de58e290 0x55c5de596bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.503+0000 7f590fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5de597790 0x55c5de5a2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.503+0000 7f590f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5de58e290 0x55c5de596bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53204/0 (socket says 172.21.15.31:53204) 2024-03-21T10:49:15.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.503+0000 7f590f400640 1 -- 172.21.15.31:0/282364160 learned_addr learned my addr 172.21.15.31:0/282364160 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:15.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.503+0000 7f590ea00640 1 --2- 172.21.15.31:0/282364160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5de58ebb0 0x55c5de597100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.503+0000 7f590f400640 1 -- 172.21.15.31:0/282364160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5de597790 msgr2=0x55c5de5a2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:15.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.503+0000 7f590f400640 1 --2- 172.21.15.31:0/282364160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5de597790 0x55c5de5a2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.503+0000 7f590f400640 1 -- 172.21.15.31:0/282364160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5de58ebb0 msgr2=0x55c5de597100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:15.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.503+0000 7f590f400640 1 --2- 172.21.15.31:0/282364160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5de58ebb0 0x55c5de597100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.503+0000 7f590f400640 1 -- 172.21.15.31:0/282364160 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5de594d60 con 0x55c5de58e290 2024-03-21T10:49:15.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.504+0000 7f590f400640 1 --2- 172.21.15.31:0/282364160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5de58e290 0x55c5de596bc0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f5900005ec0 tx=0x7f5900005870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6d99ad295243747 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:15.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.504+0000 7f58fbe00640 1 -- 172.21.15.31:0/282364160 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5900045070 con 0x55c5de58e290 2024-03-21T10:49:15.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.504+0000 7f58fbe00640 1 -- 172.21.15.31:0/282364160 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5900004400 con 0x55c5de58e290 2024-03-21T10:49:15.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.504+0000 7f58fbe00640 1 -- 172.21.15.31:0/282364160 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5900003850 con 0x55c5de58e290 2024-03-21T10:49:15.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.504+0000 7f591923a740 1 -- 172.21.15.31:0/282364160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5de58e290 msgr2=0x55c5de596bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:15.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.504+0000 7f591923a740 1 --2- 172.21.15.31:0/282364160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5de58e290 0x55c5de596bc0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f5900005ec0 tx=0x7f5900005870 comp rx=0 tx=0).stop 2024-03-21T10:49:15.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.504+0000 7f591923a740 1 -- 172.21.15.31:0/282364160 shutdown_connections 2024-03-21T10:49:15.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.504+0000 7f591923a740 1 --2- 172.21.15.31:0/282364160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5de597790 0x55c5de5a2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.504+0000 7f591923a740 1 --2- 172.21.15.31:0/282364160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5de58ebb0 0x55c5de597100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.504+0000 7f591923a740 1 --2- 172.21.15.31:0/282364160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5de58e290 0x55c5de596bc0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.504+0000 7f591923a740 1 -- 172.21.15.31:0/282364160 >> 172.21.15.31:0/282364160 conn(0x55c5de5889c0 msgr2=0x55c5de58a630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:15.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.504+0000 7f591923a740 1 -- 172.21.15.31:0/282364160 shutdown_connections 2024-03-21T10:49:15.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.505+0000 7f591923a740 1 -- 172.21.15.31:0/282364160 wait complete. 2024-03-21T10:49:15.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.505+0000 7f591923a740 1 -- start start 2024-03-21T10:49:15.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.505+0000 7f591923a740 1 Processor -- start 2024-03-21T10:49:15.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.506+0000 7f591923a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5de58e290 0x55c5de59c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.506+0000 7f591923a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5de58ebb0 0x55c5de59ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.506+0000 7f591923a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5de597790 0x55c5de59d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.506+0000 7f591923a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59101af140 con 0x55c5de597790 2024-03-21T10:49:15.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.506+0000 7f591923a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59101b0420 con 0x55c5de58e290 2024-03-21T10:49:15.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.506+0000 7f591923a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59101af760 con 0x55c5de58ebb0 2024-03-21T10:49:15.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.506+0000 7f590f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5de58e290 0x55c5de59c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.506+0000 7f591923a740 10 client.0 _ll_register_callbacks cb 0x7f58fc00ba30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:15.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.506+0000 7f590f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5de58e290 0x55c5de59c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39492/0 (socket says 172.21.15.31:39492) 2024-03-21T10:49:15.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.506+0000 7f590f400640 1 -- 172.21.15.31:0/580380493 learned_addr learned my addr 172.21.15.31:0/580380493 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:15.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.506+0000 7f590ea00640 1 --2- 172.21.15.31:0/580380493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5de58ebb0 0x55c5de59ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.506+0000 7f590fe00640 1 --2- 172.21.15.31:0/580380493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5de597790 0x55c5de59d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.506+0000 7f590f400640 1 -- 172.21.15.31:0/580380493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5de58ebb0 msgr2=0x55c5de59ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:15.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.506+0000 7f590f400640 1 --2- 172.21.15.31:0/580380493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5de58ebb0 0x55c5de59ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.506+0000 7f590f400640 1 -- 172.21.15.31:0/580380493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5de597790 msgr2=0x55c5de59d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:15.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.506+0000 7f590f400640 1 --2- 172.21.15.31:0/580380493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5de597790 0x55c5de59d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.506+0000 7f590f400640 1 -- 172.21.15.31:0/580380493 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c5de599ef0 con 0x55c5de58e290 2024-03-21T10:49:15.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.506+0000 7f590ea00640 1 --2- 172.21.15.31:0/580380493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5de58ebb0 0x55c5de59ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:15.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.507+0000 7f590f400640 1 --2- 172.21.15.31:0/580380493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5de58e290 0x55c5de59c8f0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f59000049f0 tx=0x7f5900005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b78df654b41e1ac server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:15.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.507+0000 7f58f9600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:49:15.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.507+0000 7f58f9600640 1 -- 172.21.15.31:0/580380493 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5900064060 con 0x55c5de58e290 2024-03-21T10:49:15.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.507+0000 7f591923a740 1 -- 172.21.15.31:0/580380493 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5900004400 con 0x55c5de58e290 2024-03-21T10:49:15.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.507+0000 7f58f9600640 1 -- 172.21.15.31:0/580380493 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f5900003f10 con 0x55c5de58e290 2024-03-21T10:49:15.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.507+0000 7f58f9600640 1 -- 172.21.15.31:0/580380493 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f590006b050 con 0x55c5de58e290 2024-03-21T10:49:15.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.507+0000 7f591923a740 10 client.28343 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:15.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.507+0000 7f591923a740 1 -- 172.21.15.31:0/580380493 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55c5de603d00 con 0x55c5de58e290 2024-03-21T10:49:15.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.508+0000 7f58f9600640 1 -- 172.21.15.31:0/580380493 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5900004400 con 0x55c5de58e290 2024-03-21T10:49:15.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.508+0000 7f58f9600640 1 -- 172.21.15.31:0/580380493 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f59000053e0 con 0x55c5de58e290 2024-03-21T10:49:15.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.508+0000 7f591923a740 10 client.28343 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:15.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.508+0000 7f591923a740 10 client.28343 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:49:15.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.508+0000 7f591923a740 20 client.28343 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:15.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.508+0000 7f591923a740 1 --2- 172.21.15.31:0/580380493 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c5de6042b0 0x55c5de6066a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.508+0000 7f591923a740 4 client.28343 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-21T10:49:15.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.508+0000 7f591923a740 1 -- 172.21.15.31:0/580380493 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55c5de4fa740 con 0x55c5de6042b0 2024-03-21T10:49:15.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.508+0000 7f590ea00640 1 --2- 172.21.15.31:0/580380493 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c5de6042b0 0x55c5de6066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.509+0000 7f590ea00640 1 --2- 172.21.15.31:0/580380493 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c5de6042b0 0x55c5de6066a0 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e63e86468cf0890 server_cookie=fa9255a45023c236 in_seq=0 out_seq=0 2024-03-21T10:49:15.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.509+0000 7f58f9600640 10 client.28343 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:15.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.510+0000 7f58f9600640 1 -- 172.21.15.31:0/580380493 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55c5de4fa740 con 0x55c5de6042b0 2024-03-21T10:49:15.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.510+0000 7f58f9600640 10 client.28343 handle_command_reply: tid=0 2024-03-21T10:49:15.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.510+0000 7f591923a740 1 client.28343 shutdown 2024-03-21T10:49:15.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.510+0000 7f591923a740 1 -- 172.21.15.31:0/580380493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5de58e290 msgr2=0x55c5de59c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:15.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.510+0000 7f591923a740 1 --2- 172.21.15.31:0/580380493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5de58e290 0x55c5de59c8f0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f59000049f0 tx=0x7f5900005fd0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.510+0000 7f591923a740 1 -- 172.21.15.31:0/580380493 shutdown_connections 2024-03-21T10:49:15.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.510+0000 7f591923a740 1 --2- 172.21.15.31:0/580380493 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55c5de6042b0 0x55c5de6066a0 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.510+0000 7f591923a740 1 --2- 172.21.15.31:0/580380493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c5de597790 0x55c5de59d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.510+0000 7f591923a740 1 --2- 172.21.15.31:0/580380493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c5de58ebb0 0x55c5de59ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.510+0000 7f591923a740 1 --2- 172.21.15.31:0/580380493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5de58e290 0x55c5de59c8f0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.510+0000 7f591923a740 1 -- 172.21.15.31:0/580380493 >> 172.21.15.31:0/580380493 conn(0x55c5de5889c0 msgr2=0x55c5de589b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:15.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.510+0000 7f58f9600640 0 client.28343 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:15.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.510+0000 7f591923a740 1 -- 172.21.15.31:0/580380493 shutdown_connections 2024-03-21T10:49:15.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.510+0000 7f591923a740 1 -- 172.21.15.31:0/580380493 wait complete. 2024-03-21T10:49:15.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.510+0000 7f591923a740 20 client.28343 trim_cache size 0 max 16384 2024-03-21T10:49:15.510 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:49:15.510 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-03-21T10:49:15.510 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "fac962b9-bcf3-4145-8bf4-bb77d33f7983", 2024-03-21T10:49:15.511 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-03-21T10:49:15.511 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:49:15.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.511+0000 7f5914e00640 1 -- 172.21.15.31:0/4115741546 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f58f40a3110 msgr2=0x7f58f40a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:15.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.511+0000 7f5914e00640 1 --2- 172.21.15.31:0/4115741546 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f58f40a3110 0x7f58f40a55d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f58fc0058a0 tx=0x7f58fc005c90 comp rx=0 tx=0).stop 2024-03-21T10:49:15.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.511+0000 7f5914e00640 1 -- 172.21.15.31:0/4115741546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f591010ba50 msgr2=0x7f591019e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:15.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.511+0000 7f5914e00640 1 --2- 172.21.15.31:0/4115741546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f591010ba50 0x7f591019e820 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f5908009fd0 tx=0x7f590800ef10 comp rx=0 tx=0).stop 2024-03-21T10:49:15.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.511+0000 7f5914e00640 1 -- 172.21.15.31:0/4115741546 shutdown_connections 2024-03-21T10:49:15.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.511+0000 7f5914e00640 1 --2- 172.21.15.31:0/4115741546 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f58f40a3110 0x7f58f40a55d0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.511+0000 7f5914e00640 1 --2- 172.21.15.31:0/4115741546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f591010ba50 0x7f591019e820 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.511+0000 7f5914e00640 1 --2- 172.21.15.31:0/4115741546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5910102f50 0x7f591019a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.511+0000 7f5914e00640 1 --2- 172.21.15.31:0/4115741546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5910102630 0x7f5910199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.511+0000 7f5914e00640 1 -- 172.21.15.31:0/4115741546 >> 172.21.15.31:0/4115741546 conn(0x7f5910077130 msgr2=0x7f5910100230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:15.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.511+0000 7f5914e00640 1 -- 172.21.15.31:0/4115741546 shutdown_connections 2024-03-21T10:49:15.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.511+0000 7f5914e00640 1 -- 172.21.15.31:0/4115741546 wait complete. 2024-03-21T10:49:15.526 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:15 smithi031 ceph-mon[31673]: pgmap v332: 129 pgs: 11 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 85 active+clean; 29 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 30 KiB/s wr, 10 op/s 2024-03-21T10:49:15.530 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:49:15.530 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:49:15.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:15 smithi070 ceph-mon[36330]: pgmap v332: 129 pgs: 11 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 85 active+clean; 29 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 30 KiB/s wr, 10 op/s 2024-03-21T10:49:15.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.747+0000 7f87f3200640 1 -- 172.21.15.31:0/2818046974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87f4074370 msgr2=0x7f87f410d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:15.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.747+0000 7f87f3200640 1 --2- 172.21.15.31:0/2818046974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87f4074370 0x7f87f410d800 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f87dc009900 tx=0x7f87dc02bcd0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.747+0000 7f87f3200640 1 -- 172.21.15.31:0/2818046974 shutdown_connections 2024-03-21T10:49:15.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.747+0000 7f87f3200640 1 --2- 172.21.15.31:0/2818046974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87f410dd40 0x7f87f4110130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.747+0000 7f87f3200640 1 --2- 172.21.15.31:0/2818046974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87f4074370 0x7f87f410d800 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.747+0000 7f87f3200640 1 --2- 172.21.15.31:0/2818046974 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87f4073a50 0x7f87f4073e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.747+0000 7f87f3200640 1 -- 172.21.15.31:0/2818046974 >> 172.21.15.31:0/2818046974 conn(0x7f87f40fcfb0 msgr2=0x7f87f40ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:15.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.747+0000 7f87f3200640 1 -- 172.21.15.31:0/2818046974 shutdown_connections 2024-03-21T10:49:15.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.747+0000 7f87f3200640 1 -- 172.21.15.31:0/2818046974 wait complete. 2024-03-21T10:49:15.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.748+0000 7f87f3200640 1 Processor -- start 2024-03-21T10:49:15.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.748+0000 7f87f3200640 1 -- start start 2024-03-21T10:49:15.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.749+0000 7f87f3200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87f4073a50 0x7f87f4101460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.749+0000 7f87f3200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87f4074370 0x7f87f41019a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.749+0000 7f87f3200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87f410dd40 0x7f87f4101ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.749+0000 7f87f3200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f4113a70 con 0x7f87f410dd40 2024-03-21T10:49:15.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.749+0000 7f87f3200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f41138f0 con 0x7f87f4074370 2024-03-21T10:49:15.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.749+0000 7f87f3200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f4113bf0 con 0x7f87f4073a50 2024-03-21T10:49:15.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.749+0000 7f87f1e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87f4073a50 0x7f87f4101460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.749+0000 7f87f1e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87f4073a50 0x7f87f4101460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53246/0 (socket says 172.21.15.31:53246) 2024-03-21T10:49:15.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.749+0000 7f87f1e00640 1 -- 172.21.15.31:0/3661815777 learned_addr learned my addr 172.21.15.31:0/3661815777 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:15.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.749+0000 7f87e9400640 1 --2- 172.21.15.31:0/3661815777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87f4074370 0x7f87f41019a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.749+0000 7f87f2800640 1 --2- 172.21.15.31:0/3661815777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87f410dd40 0x7f87f4101ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.749+0000 7f87f1e00640 1 -- 172.21.15.31:0/3661815777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87f4074370 msgr2=0x7f87f41019a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:15.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.749+0000 7f87f1e00640 1 --2- 172.21.15.31:0/3661815777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87f4074370 0x7f87f41019a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.749+0000 7f87f1e00640 1 -- 172.21.15.31:0/3661815777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87f410dd40 msgr2=0x7f87f4101ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:15.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.750+0000 7f87f1e00640 1 --2- 172.21.15.31:0/3661815777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87f410dd40 0x7f87f4101ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.750+0000 7f87f1e00640 1 -- 172.21.15.31:0/3661815777 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f41aa690 con 0x7f87f4073a50 2024-03-21T10:49:15.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.750+0000 7f87f1e00640 1 --2- 172.21.15.31:0/3661815777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87f4073a50 0x7f87f4101460 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f87e400ea20 tx=0x7f87e400ee20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:15.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.750+0000 7f87eb400640 1 -- 172.21.15.31:0/3661815777 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e400ccc0 con 0x7f87f4073a50 2024-03-21T10:49:15.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.750+0000 7f87eb400640 1 -- 172.21.15.31:0/3661815777 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87e400ce60 con 0x7f87f4073a50 2024-03-21T10:49:15.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.750+0000 7f87f3200640 1 -- 172.21.15.31:0/3661815777 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f41aa980 con 0x7f87f4073a50 2024-03-21T10:49:15.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.750+0000 7f87eb400640 1 -- 172.21.15.31:0/3661815777 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e40187e0 con 0x7f87f4073a50 2024-03-21T10:49:15.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.750+0000 7f87f3200640 1 -- 172.21.15.31:0/3661815777 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f41aaef0 con 0x7f87f4073a50 2024-03-21T10:49:15.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.755+0000 7f87fbc17740 1 Processor -- start 2024-03-21T10:49:15.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.755+0000 7f87fbc17740 1 -- start start 2024-03-21T10:49:15.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.755+0000 7f87eb400640 1 -- 172.21.15.31:0/3661815777 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f87e4010430 con 0x7f87f4073a50 2024-03-21T10:49:15.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.755+0000 7f87fbc17740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce702dd0 0x55fbce701b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.755+0000 7f87fbc17740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbce7020f0 0x55fbce702570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.755+0000 7f87fbc17740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbce70b460 0x55fbce70d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.755+0000 7f87fbc17740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbce666250 con 0x55fbce7020f0 2024-03-21T10:49:15.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.755+0000 7f87fbc17740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbce60ea30 con 0x55fbce70b460 2024-03-21T10:49:15.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.755+0000 7f87fbc17740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbce6622f0 con 0x55fbce702dd0 2024-03-21T10:49:15.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.756+0000 7f87f2800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce702dd0 0x55fbce701b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.756+0000 7f87f2800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce702dd0 0x55fbce701b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53256/0 (socket says 172.21.15.31:53256) 2024-03-21T10:49:15.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.756+0000 7f87f2800640 1 -- 172.21.15.31:0/2075290297 learned_addr learned my addr 172.21.15.31:0/2075290297 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:15.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.756+0000 7f87e9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbce70b460 0x55fbce70d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.756+0000 7f87f1e00640 1 --2- 172.21.15.31:0/2075290297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbce7020f0 0x55fbce702570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.756+0000 7f87f2800640 1 -- 172.21.15.31:0/2075290297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbce70b460 msgr2=0x55fbce70d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:15.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.756+0000 7f87f2800640 1 --2- 172.21.15.31:0/2075290297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbce70b460 0x55fbce70d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.756+0000 7f87f2800640 1 -- 172.21.15.31:0/2075290297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbce7020f0 msgr2=0x55fbce702570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:15.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.756+0000 7f87f2800640 1 --2- 172.21.15.31:0/2075290297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbce7020f0 0x55fbce702570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.756+0000 7f87f2800640 1 -- 172.21.15.31:0/2075290297 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fbce5f9690 con 0x55fbce702dd0 2024-03-21T10:49:15.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.756+0000 7f87f2800640 1 --2- 172.21.15.31:0/2075290297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce702dd0 0x55fbce701b20 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f87ec009f60 tx=0x7f87ec009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58e9a4a142c99cfc server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:15.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.757+0000 7f87e3400640 1 -- 172.21.15.31:0/2075290297 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec01e070 con 0x55fbce702dd0 2024-03-21T10:49:15.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.757+0000 7f87e3400640 1 -- 172.21.15.31:0/2075290297 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87ec003c30 con 0x55fbce702dd0 2024-03-21T10:49:15.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.757+0000 7f87e3400640 1 -- 172.21.15.31:0/2075290297 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec00fa20 con 0x55fbce702dd0 2024-03-21T10:49:15.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.757+0000 7f87eb400640 1 --2- 172.21.15.31:0/3661815777 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f87d80a3110 0x7f87d80a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.757+0000 7f87eb400640 1 -- 172.21.15.31:0/3661815777 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f87e40cfea0 con 0x7f87f4073a50 2024-03-21T10:49:15.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.757+0000 7f87f1e00640 1 --2- 172.21.15.31:0/3661815777 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f87d80a3110 0x7f87d80a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.757+0000 7f87fbc17740 1 -- 172.21.15.31:0/2075290297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce702dd0 msgr2=0x55fbce701b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:15.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.757+0000 7f87fbc17740 1 --2- 172.21.15.31:0/2075290297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce702dd0 0x55fbce701b20 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f87ec009f60 tx=0x7f87ec009f90 comp rx=0 tx=0).stop 2024-03-21T10:49:15.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.758+0000 7f87fbc17740 1 -- 172.21.15.31:0/2075290297 shutdown_connections 2024-03-21T10:49:15.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.758+0000 7f87fbc17740 1 --2- 172.21.15.31:0/2075290297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbce70b460 0x55fbce70d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.758+0000 7f87fbc17740 1 --2- 172.21.15.31:0/2075290297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbce7020f0 0x55fbce702570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.758+0000 7f87fbc17740 1 --2- 172.21.15.31:0/2075290297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce702dd0 0x55fbce701b20 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.758+0000 7f87fbc17740 1 -- 172.21.15.31:0/2075290297 >> 172.21.15.31:0/2075290297 conn(0x55fbce6c9200 msgr2=0x55fbce6fcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:15.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.758+0000 7f87fbc17740 1 -- 172.21.15.31:0/2075290297 shutdown_connections 2024-03-21T10:49:15.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.758+0000 7f87fbc17740 1 -- 172.21.15.31:0/2075290297 wait complete. 2024-03-21T10:49:15.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.758+0000 7f87f1e00640 1 --2- 172.21.15.31:0/3661815777 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f87d80a3110 0x7f87d80a55d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f87e40cd2b0 tx=0x7f87e4000bc0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:15.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.758+0000 7f87fbc17740 1 -- start start 2024-03-21T10:49:15.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.759+0000 7f87fbc17740 1 Processor -- start 2024-03-21T10:49:15.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.759+0000 7f87fbc17740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce7020f0 0x55fbce70ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.759+0000 7f87fbc17740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbce702dd0 0x55fbce707ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.759+0000 7f87fbc17740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbce70b460 0x55fbce708410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.759+0000 7f87fbc17740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbce70e2d0 con 0x55fbce702dd0 2024-03-21T10:49:15.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.759+0000 7f87fbc17740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbce621e80 con 0x55fbce70b460 2024-03-21T10:49:15.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.759+0000 7f87fbc17740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbce70e450 con 0x55fbce7020f0 2024-03-21T10:49:15.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.759+0000 7f87fbc17740 10 client.0 _ll_register_callbacks cb 0x55fbce634130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:15.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.759+0000 7f87f2800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce7020f0 0x55fbce70ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.759+0000 7f87e9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbce702dd0 0x55fbce707ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.759+0000 7f87f2800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce7020f0 0x55fbce70ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53268/0 (socket says 172.21.15.31:53268) 2024-03-21T10:49:15.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.759+0000 7f87f2800640 1 -- 172.21.15.31:0/1356349629 learned_addr learned my addr 172.21.15.31:0/1356349629 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:15.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.760+0000 7f87f2800640 1 --2- 172.21.15.31:0/1356349629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbce70b460 0x55fbce708410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.760+0000 7f87e9400640 1 -- 172.21.15.31:0/1356349629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce7020f0 msgr2=0x55fbce70ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:15.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.760+0000 7f87e9400640 1 --2- 172.21.15.31:0/1356349629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce7020f0 0x55fbce70ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.760+0000 7f87e9400640 1 -- 172.21.15.31:0/1356349629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbce70b460 msgr2=0x55fbce708410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:15.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.760+0000 7f87e9400640 1 --2- 172.21.15.31:0/1356349629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbce70b460 0x55fbce708410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.760+0000 7f87e9400640 1 -- 172.21.15.31:0/1356349629 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fbce4447d0 con 0x55fbce702dd0 2024-03-21T10:49:15.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.760+0000 7f87f2800640 1 --2- 172.21.15.31:0/1356349629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbce70b460 0x55fbce708410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:49:15.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.760+0000 7f87e9400640 1 --2- 172.21.15.31:0/1356349629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbce702dd0 0x55fbce707ed0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f87dc02c210 tx=0x7f87dc005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bacdf907e5fbe11e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:15.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.760+0000 7f87d7e00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:49:15.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.761+0000 7f87d7e00640 1 -- 172.21.15.31:0/1356349629 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc044070 con 0x55fbce702dd0 2024-03-21T10:49:15.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.761+0000 7f87d7e00640 1 -- 172.21.15.31:0/1356349629 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f87dc0046c0 con 0x55fbce702dd0 2024-03-21T10:49:15.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.761+0000 7f87fbc17740 1 -- 172.21.15.31:0/1356349629 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f87ec003c30 con 0x55fbce702dd0 2024-03-21T10:49:15.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.761+0000 7f87d7e00640 1 -- 172.21.15.31:0/1356349629 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87dc051050 con 0x55fbce702dd0 2024-03-21T10:49:15.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.761+0000 7f87fbc17740 10 client.17712 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:15.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.761+0000 7f87fbc17740 1 -- 172.21.15.31:0/1356349629 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55fbce37a3f0 con 0x55fbce702dd0 2024-03-21T10:49:15.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.761+0000 7f87d7e00640 1 -- 172.21.15.31:0/1356349629 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87ec003c30 con 0x55fbce702dd0 2024-03-21T10:49:15.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.762+0000 7f87d7e00640 1 -- 172.21.15.31:0/1356349629 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f87dc0035f0 con 0x55fbce702dd0 2024-03-21T10:49:15.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.762+0000 7f87fbc17740 10 client.17712 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:15.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.762+0000 7f87fbc17740 10 client.17712 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:49:15.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.762+0000 7f87fbc17740 20 client.17712 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:15.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.762+0000 7f87fbc17740 1 --2- 172.21.15.31:0/1356349629 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55fbce71b370 0x55fbce71d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.762+0000 7f87fbc17740 4 client.17712 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:49:15.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.762+0000 7f87fbc17740 1 -- 172.21.15.31:0/1356349629 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fbce673790 con 0x55fbce71b370 2024-03-21T10:49:15.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.762+0000 7f87f2800640 1 --2- 172.21.15.31:0/1356349629 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55fbce71b370 0x55fbce71d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.763+0000 7f87f2800640 1 --2- 172.21.15.31:0/1356349629 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55fbce71b370 0x55fbce71d760 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e08b9aae588ef5c server_cookie=f17424e73a78eb3a in_seq=0 out_seq=0 2024-03-21T10:49:15.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.763+0000 7f87d7e00640 10 client.17712 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:15.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.763+0000 7f87d7e00640 1 -- 172.21.15.31:0/1356349629 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55fbce673790 con 0x55fbce71b370 2024-03-21T10:49:15.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.763+0000 7f87d7e00640 10 client.17712 handle_command_reply: tid=0 2024-03-21T10:49:15.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.763+0000 7f87fbc17740 1 client.17712 shutdown 2024-03-21T10:49:15.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.764+0000 7f87fbc17740 1 -- 172.21.15.31:0/1356349629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbce702dd0 msgr2=0x55fbce707ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:15.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.764+0000 7f87fbc17740 1 --2- 172.21.15.31:0/1356349629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbce702dd0 0x55fbce707ed0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f87dc02c210 tx=0x7f87dc005dd0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.764+0000 7f87fbc17740 1 -- 172.21.15.31:0/1356349629 shutdown_connections 2024-03-21T10:49:15.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.764+0000 7f87fbc17740 1 --2- 172.21.15.31:0/1356349629 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55fbce71b370 0x55fbce71d760 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.764+0000 7f87fbc17740 1 --2- 172.21.15.31:0/1356349629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbce70b460 0x55fbce708410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.764+0000 7f87fbc17740 1 --2- 172.21.15.31:0/1356349629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbce702dd0 0x55fbce707ed0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.764+0000 7f87fbc17740 1 --2- 172.21.15.31:0/1356349629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce7020f0 0x55fbce70ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.764+0000 7f87d7e00640 0 client.17712 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:15.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.764+0000 7f87fbc17740 1 -- 172.21.15.31:0/1356349629 >> 172.21.15.31:0/1356349629 conn(0x55fbce6c9200 msgr2=0x55fbce6fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:15.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.764+0000 7f87fbc17740 1 -- 172.21.15.31:0/1356349629 shutdown_connections 2024-03-21T10:49:15.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.764+0000 7f87fbc17740 1 -- 172.21.15.31:0/1356349629 wait complete. 2024-03-21T10:49:15.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.764+0000 7f87fbc17740 20 client.17712 trim_cache size 0 max 16384 2024-03-21T10:49:15.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.782+0000 7f87fbc17740 1 Processor -- start 2024-03-21T10:49:15.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.782+0000 7f87fbc17740 1 -- start start 2024-03-21T10:49:15.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.782+0000 7f87fbc17740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce7020f0 0x55fbce70a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.782+0000 7f87fbc17740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbce702dd0 0x55fbce70ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.782+0000 7f87fbc17740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbce70b460 0x55fbce70c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.782+0000 7f87fbc17740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f41b7570 con 0x55fbce702dd0 2024-03-21T10:49:15.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.782+0000 7f87fbc17740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f41b5070 con 0x55fbce70b460 2024-03-21T10:49:15.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.782+0000 7f87fbc17740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f41b3130 con 0x55fbce7020f0 2024-03-21T10:49:15.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.783+0000 7f87f2800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce7020f0 0x55fbce70a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.783+0000 7f87f2800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce7020f0 0x55fbce70a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53274/0 (socket says 172.21.15.31:53274) 2024-03-21T10:49:15.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.783+0000 7f87f2800640 1 -- 172.21.15.31:0/2449036592 learned_addr learned my addr 172.21.15.31:0/2449036592 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:15.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.783+0000 7f87e9400640 1 --2- 172.21.15.31:0/2449036592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbce702dd0 0x55fbce70ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.783+0000 7f87f2800640 1 --2- 172.21.15.31:0/2449036592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbce70b460 0x55fbce70c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.783+0000 7f87f2800640 1 -- 172.21.15.31:0/2449036592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbce70b460 msgr2=0x55fbce70c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:15.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.783+0000 7f87f2800640 1 --2- 172.21.15.31:0/2449036592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbce70b460 0x55fbce70c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.783+0000 7f87f2800640 1 -- 172.21.15.31:0/2449036592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbce702dd0 msgr2=0x55fbce70ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:15.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.783+0000 7f87f2800640 1 --2- 172.21.15.31:0/2449036592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbce702dd0 0x55fbce70ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.783+0000 7f87f2800640 1 -- 172.21.15.31:0/2449036592 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fbce33a0f0 con 0x55fbce7020f0 2024-03-21T10:49:15.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.783+0000 7f87f2800640 1 --2- 172.21.15.31:0/2449036592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce7020f0 0x55fbce70a600 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f87ec00f4f0 tx=0x7f87ec009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86c6e437bc48b455 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:15.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.784+0000 7f87e2a00640 1 -- 172.21.15.31:0/2449036592 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec01e040 con 0x55fbce7020f0 2024-03-21T10:49:15.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.784+0000 7f87e2a00640 1 -- 172.21.15.31:0/2449036592 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87ec02cda0 con 0x55fbce7020f0 2024-03-21T10:49:15.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.784+0000 7f87e2a00640 1 -- 172.21.15.31:0/2449036592 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec00fd20 con 0x55fbce7020f0 2024-03-21T10:49:15.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.784+0000 7f87fbc17740 1 -- 172.21.15.31:0/2449036592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce7020f0 msgr2=0x55fbce70a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:15.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.784+0000 7f87fbc17740 1 --2- 172.21.15.31:0/2449036592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce7020f0 0x55fbce70a600 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f87ec00f4f0 tx=0x7f87ec009f90 comp rx=0 tx=0).stop 2024-03-21T10:49:15.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.784+0000 7f87fbc17740 1 -- 172.21.15.31:0/2449036592 shutdown_connections 2024-03-21T10:49:15.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.784+0000 7f87fbc17740 1 --2- 172.21.15.31:0/2449036592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbce70b460 0x55fbce70c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.784+0000 7f87fbc17740 1 --2- 172.21.15.31:0/2449036592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbce702dd0 0x55fbce70ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.784+0000 7f87fbc17740 1 --2- 172.21.15.31:0/2449036592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce7020f0 0x55fbce70a600 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.784+0000 7f87fbc17740 1 -- 172.21.15.31:0/2449036592 >> 172.21.15.31:0/2449036592 conn(0x55fbce6c9200 msgr2=0x55fbce7067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:15.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.784+0000 7f87fbc17740 1 -- 172.21.15.31:0/2449036592 shutdown_connections 2024-03-21T10:49:15.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.784+0000 7f87fbc17740 1 -- 172.21.15.31:0/2449036592 wait complete. 2024-03-21T10:49:15.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.785+0000 7f87fbc17740 1 -- start start 2024-03-21T10:49:15.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.785+0000 7f87fbc17740 1 Processor -- start 2024-03-21T10:49:15.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.785+0000 7f87fbc17740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbce7020f0 0x55fbce713130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.785+0000 7f87fbc17740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce702dd0 0x55fbce704f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.785+0000 7f87fbc17740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbce70b460 0x55fbce705450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.785+0000 7f87fbc17740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f41b3730 con 0x55fbce7020f0 2024-03-21T10:49:15.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.785+0000 7f87fbc17740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f41b4a10 con 0x55fbce70b460 2024-03-21T10:49:15.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.785+0000 7f87fbc17740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f41b3d50 con 0x55fbce702dd0 2024-03-21T10:49:15.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.785+0000 7f87fbc17740 10 client.0 _ll_register_callbacks cb 0x7f87dc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:15.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.786+0000 7f87e9400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce702dd0 0x55fbce704f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.786+0000 7f87f2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbce7020f0 0x55fbce713130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.786+0000 7f87e9400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce702dd0 0x55fbce704f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:53280/0 (socket says 172.21.15.31:53280) 2024-03-21T10:49:15.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.786+0000 7f87e9400640 1 -- 172.21.15.31:0/1308961972 learned_addr learned my addr 172.21.15.31:0/1308961972 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:15.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.786+0000 7f87f2800640 1 --2- 172.21.15.31:0/1308961972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbce70b460 0x55fbce705450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.786+0000 7f87e9400640 1 -- 172.21.15.31:0/1308961972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbce70b460 msgr2=0x55fbce705450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:15.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.786+0000 7f87e9400640 1 --2- 172.21.15.31:0/1308961972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbce70b460 0x55fbce705450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.786+0000 7f87e9400640 1 -- 172.21.15.31:0/1308961972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbce7020f0 msgr2=0x55fbce713130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:15.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.786+0000 7f87e9400640 1 --2- 172.21.15.31:0/1308961972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbce7020f0 0x55fbce713130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.786+0000 7f87e9400640 1 -- 172.21.15.31:0/1308961972 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fbce705990 con 0x55fbce702dd0 2024-03-21T10:49:15.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.786+0000 7f87e9400640 1 --2- 172.21.15.31:0/1308961972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce702dd0 0x55fbce704f10 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f87dc001eb0 tx=0x7f87dc0058c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1eaad74649b32ce server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:15.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.786+0000 7f87d7e00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:49:15.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.787+0000 7f87d7e00640 1 -- 172.21.15.31:0/1308961972 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc059070 con 0x55fbce702dd0 2024-03-21T10:49:15.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.787+0000 7f87d7e00640 1 -- 172.21.15.31:0/1308961972 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f87dc0046c0 con 0x55fbce702dd0 2024-03-21T10:49:15.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.787+0000 7f87fbc17740 1 -- 172.21.15.31:0/1308961972 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f87ec02cda0 con 0x55fbce702dd0 2024-03-21T10:49:15.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.787+0000 7f87d7e00640 1 -- 172.21.15.31:0/1308961972 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87dc044070 con 0x55fbce702dd0 2024-03-21T10:49:15.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.787+0000 7f87fbc17740 10 client.28288 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:15.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.787+0000 7f87fbc17740 1 -- 172.21.15.31:0/1308961972 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55fbce777820 con 0x55fbce702dd0 2024-03-21T10:49:15.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.787+0000 7f87d7e00640 1 -- 172.21.15.31:0/1308961972 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87ec02cda0 con 0x55fbce702dd0 2024-03-21T10:49:15.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.787+0000 7f87d7e00640 1 -- 172.21.15.31:0/1308961972 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f87dc005050 con 0x55fbce702dd0 2024-03-21T10:49:15.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.787+0000 7f87fbc17740 10 client.28288 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:15.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.788+0000 7f87fbc17740 10 client.28288 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:49:15.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.788+0000 7f87fbc17740 20 client.28288 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:15.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.788+0000 7f87fbc17740 1 --2- 172.21.15.31:0/1308961972 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55fbce777f20 0x55fbce77a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:15.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.788+0000 7f87fbc17740 4 client.28288 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:49:15.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.788+0000 7f87fbc17740 1 -- 172.21.15.31:0/1308961972 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55fbce66dae0 con 0x55fbce777f20 2024-03-21T10:49:15.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.788+0000 7f87f2800640 1 --2- 172.21.15.31:0/1308961972 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55fbce777f20 0x55fbce77a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:15.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.788+0000 7f87f2800640 1 --2- 172.21.15.31:0/1308961972 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55fbce777f20 0x55fbce77a340 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cda25135807d3d62 server_cookie=f01f1eaad5b426df in_seq=0 out_seq=0 2024-03-21T10:49:15.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.788+0000 7f87d7e00640 10 client.28288 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:15.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.788+0000 7f87d7e00640 1 -- 172.21.15.31:0/1308961972 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55fbce66dae0 con 0x55fbce777f20 2024-03-21T10:49:15.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.788+0000 7f87d7e00640 10 client.28288 handle_command_reply: tid=0 2024-03-21T10:49:15.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.788+0000 7f87fbc17740 1 client.28288 shutdown 2024-03-21T10:49:15.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.789+0000 7f87fbc17740 1 -- 172.21.15.31:0/1308961972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce702dd0 msgr2=0x55fbce704f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:15.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.789+0000 7f87fbc17740 1 --2- 172.21.15.31:0/1308961972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce702dd0 0x55fbce704f10 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f87dc001eb0 tx=0x7f87dc0058c0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.789+0000 7f87fbc17740 1 -- 172.21.15.31:0/1308961972 shutdown_connections 2024-03-21T10:49:15.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.789+0000 7f87fbc17740 1 --2- 172.21.15.31:0/1308961972 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55fbce777f20 0x55fbce77a340 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.789+0000 7f87fbc17740 1 --2- 172.21.15.31:0/1308961972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbce70b460 0x55fbce705450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.789+0000 7f87fbc17740 1 --2- 172.21.15.31:0/1308961972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbce702dd0 0x55fbce704f10 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.789+0000 7f87fbc17740 1 --2- 172.21.15.31:0/1308961972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbce7020f0 0x55fbce713130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.789+0000 7f87fbc17740 1 -- 172.21.15.31:0/1308961972 >> 172.21.15.31:0/1308961972 conn(0x55fbce6c9200 msgr2=0x55fbce6fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:15.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.789+0000 7f87d7e00640 0 client.28288 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:15.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.789+0000 7f87fbc17740 1 -- 172.21.15.31:0/1308961972 shutdown_connections 2024-03-21T10:49:15.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.789+0000 7f87fbc17740 1 -- 172.21.15.31:0/1308961972 wait complete. 2024-03-21T10:49:15.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.789+0000 7f87fbc17740 20 client.28288 trim_cache size 0 max 16384 2024-03-21T10:49:15.790 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:49:15.790 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-21T10:49:15.790 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-03-21T10:49:15.790 INFO:teuthology.orchestra.run.smithi031.stdout: "fac962b9-bcf3-4145-8bf4-bb77d33f7983": { 2024-03-21T10:49:15.790 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-03-21T10:49:15.790 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "fac962b9-bcf3-4145-8bf4-bb77d33f7983", 2024-03-21T10:49:15.790 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-03-21T10:49:15.791 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:49:15.791 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:49:15.791 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:49:15.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.790+0000 7f87f3200640 1 -- 172.21.15.31:0/3661815777 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f87d80a3110 msgr2=0x7f87d80a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:15.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.790+0000 7f87f3200640 1 --2- 172.21.15.31:0/3661815777 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f87d80a3110 0x7f87d80a55d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f87e40cd2b0 tx=0x7f87e4000bc0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.790+0000 7f87f3200640 1 -- 172.21.15.31:0/3661815777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87f4073a50 msgr2=0x7f87f4101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:15.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.790+0000 7f87f3200640 1 --2- 172.21.15.31:0/3661815777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87f4073a50 0x7f87f4101460 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f87e400ea20 tx=0x7f87e400ee20 comp rx=0 tx=0).stop 2024-03-21T10:49:15.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.790+0000 7f87f3200640 1 -- 172.21.15.31:0/3661815777 shutdown_connections 2024-03-21T10:49:15.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.790+0000 7f87f3200640 1 --2- 172.21.15.31:0/3661815777 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f87d80a3110 0x7f87d80a55d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.790+0000 7f87f3200640 1 --2- 172.21.15.31:0/3661815777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f87f410dd40 0x7f87f4101ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.790+0000 7f87f3200640 1 --2- 172.21.15.31:0/3661815777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87f4074370 0x7f87f41019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.790+0000 7f87f3200640 1 --2- 172.21.15.31:0/3661815777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87f4073a50 0x7f87f4101460 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:15.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.790+0000 7f87f3200640 1 -- 172.21.15.31:0/3661815777 >> 172.21.15.31:0/3661815777 conn(0x7f87f40fcfb0 msgr2=0x7f87f40fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:15.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.790+0000 7f87f3200640 1 -- 172.21.15.31:0/3661815777 shutdown_connections 2024-03-21T10:49:15.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:15.790+0000 7f87f3200640 1 -- 172.21.15.31:0/3661815777 wait complete. 2024-03-21T10:49:15.804 INFO:tasks.cephfs.filesystem:scrub status for tag:fac962b9-bcf3-4145-8bf4-bb77d33f7983 - {'path': '/', 'tag': 'fac962b9-bcf3-4145-8bf4-bb77d33f7983', 'options': 'recursive,force'} 2024-03-21T10:49:16.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:16 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:49:16.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:16 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:49:16.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:16 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:49:16.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:16 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:49:16.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:16 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:49:16.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:16 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:49:16.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:16 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:49:16.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:16 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:49:16.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:16 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:49:16.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:16 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:49:16.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:16 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:49:16.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:16 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:49:17.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:17 smithi040 ceph-mon[36154]: scrub queued for path: / 2024-03-21T10:49:17.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:17 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:49:17.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:17 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:49:17.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:17 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:49:17.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:17 smithi040 ceph-mon[36154]: pgmap v333: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-03-21T10:49:17.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:17 smithi040 ceph-mon[36154]: osdmap e104: 12 total, 12 up, 12 in 2024-03-21T10:49:17.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:17 smithi031 ceph-mon[31673]: scrub queued for path: / 2024-03-21T10:49:17.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:17 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:49:17.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:17 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:49:17.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:17 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:49:17.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:17 smithi031 ceph-mon[31673]: pgmap v333: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-03-21T10:49:17.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:17 smithi031 ceph-mon[31673]: osdmap e104: 12 total, 12 up, 12 in 2024-03-21T10:49:17.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:17 smithi070 ceph-mon[36330]: scrub queued for path: / 2024-03-21T10:49:17.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:17 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:49:17.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:17 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:49:17.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:17 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:49:17.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:17 smithi070 ceph-mon[36330]: pgmap v333: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-03-21T10:49:17.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:17 smithi070 ceph-mon[36330]: osdmap e104: 12 total, 12 up, 12 in 2024-03-21T10:49:18.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.151+0000 7f9f5cc00640 1 -- 172.21.15.31:0/45780599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f581014a0 msgr2=0x7f9f58101920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:18.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.151+0000 7f9f5cc00640 1 --2- 172.21.15.31:0/45780599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f581014a0 0x7f9f58101920 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f9f44009930 tx=0x7f9f4402bc30 comp rx=0 tx=0).stop 2024-03-21T10:49:18.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.151+0000 7f9f5cc00640 1 -- 172.21.15.31:0/45780599 shutdown_connections 2024-03-21T10:49:18.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.151+0000 7f9f5cc00640 1 --2- 172.21.15.31:0/45780599 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f58101e60 0x7f9f5810e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.151+0000 7f9f5cc00640 1 --2- 172.21.15.31:0/45780599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f581014a0 0x7f9f58101920 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.151+0000 7f9f5cc00640 1 --2- 172.21.15.31:0/45780599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f58107560 0x7f9f58107940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.151+0000 7f9f5cc00640 1 -- 172.21.15.31:0/45780599 >> 172.21.15.31:0/45780599 conn(0x7f9f580fcfb0 msgr2=0x7f9f580ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:18.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.151+0000 7f9f5cc00640 1 -- 172.21.15.31:0/45780599 shutdown_connections 2024-03-21T10:49:18.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.151+0000 7f9f5cc00640 1 -- 172.21.15.31:0/45780599 wait complete. 2024-03-21T10:49:18.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.153+0000 7f9f5cc00640 1 Processor -- start 2024-03-21T10:49:18.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.153+0000 7f9f5cc00640 1 -- start start 2024-03-21T10:49:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.154+0000 7f9f5cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f581014a0 0x7f9f58199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.154+0000 7f9f5cc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f58101e60 0x7f9f5819a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.154+0000 7f9f5cc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f58107560 0x7f9f5819e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.154+0000 7f9f5cc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f58111890 con 0x7f9f58101e60 2024-03-21T10:49:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.154+0000 7f9f5cc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f58111710 con 0x7f9f581014a0 2024-03-21T10:49:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.154+0000 7f9f5cc00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f58111a10 con 0x7f9f58107560 2024-03-21T10:49:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.154+0000 7f9f56a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f58101e60 0x7f9f5819a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.154+0000 7f9f57e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f58107560 0x7f9f5819e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.154+0000 7f9f57400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f581014a0 0x7f9f58199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.154+0000 7f9f57400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f581014a0 0x7f9f58199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49818/0 (socket says 172.21.15.31:49818) 2024-03-21T10:49:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.154+0000 7f9f56a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f58101e60 0x7f9f5819a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51782/0 (socket says 172.21.15.31:51782) 2024-03-21T10:49:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.154+0000 7f9f57400640 1 -- 172.21.15.31:0/1554874127 learned_addr learned my addr 172.21.15.31:0/1554874127 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:18.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.155+0000 7f9f57e00640 1 -- 172.21.15.31:0/1554874127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f581014a0 msgr2=0x7f9f58199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:18.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.155+0000 7f9f57e00640 1 --2- 172.21.15.31:0/1554874127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f581014a0 0x7f9f58199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.155+0000 7f9f57e00640 1 -- 172.21.15.31:0/1554874127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f58101e60 msgr2=0x7f9f5819a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:18.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.155+0000 7f9f57e00640 1 --2- 172.21.15.31:0/1554874127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f58101e60 0x7f9f5819a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.155+0000 7f9f57e00640 1 -- 172.21.15.31:0/1554874127 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f5819ef40 con 0x7f9f58107560 2024-03-21T10:49:18.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.155+0000 7f9f57400640 1 --2- 172.21.15.31:0/1554874127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f581014a0 0x7f9f58199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:18.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.155+0000 7f9f57e00640 1 --2- 172.21.15.31:0/1554874127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f58107560 0x7f9f5819e810 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f9f50009fd0 tx=0x7f9f5000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:18.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.155+0000 7f9f4fe00640 1 -- 172.21.15.31:0/1554874127 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5000ce20 con 0x7f9f58107560 2024-03-21T10:49:18.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.155+0000 7f9f4fe00640 1 -- 172.21.15.31:0/1554874127 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f50005020 con 0x7f9f58107560 2024-03-21T10:49:18.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.155+0000 7f9f4fe00640 1 -- 172.21.15.31:0/1554874127 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f50010720 con 0x7f9f58107560 2024-03-21T10:49:18.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.155+0000 7f9f5cc00640 1 -- 172.21.15.31:0/1554874127 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f5819f230 con 0x7f9f58107560 2024-03-21T10:49:18.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.156+0000 7f9f5cc00640 1 -- 172.21.15.31:0/1554874127 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f581a6ac0 con 0x7f9f58107560 2024-03-21T10:49:18.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.159+0000 7f9f4fe00640 1 -- 172.21.15.31:0/1554874127 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f9f50005540 con 0x7f9f58107560 2024-03-21T10:49:18.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.159+0000 7f9f60fa8740 1 Processor -- start 2024-03-21T10:49:18.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.160+0000 7f9f60fa8740 1 -- start start 2024-03-21T10:49:18.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.160+0000 7f9f60fa8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618955041e0 0x5618955045c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:18.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.160+0000 7f9f60fa8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561895504b00 0x56189550d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:18.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.160+0000 7f9f60fa8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56189550d8d0 0x56189550fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:18.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.160+0000 7f9f60fa8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618954c63b0 con 0x56189550d8d0 2024-03-21T10:49:18.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.160+0000 7f9f60fa8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561895468830 con 0x5618955041e0 2024-03-21T10:49:18.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.160+0000 7f9f60fa8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618954115d0 con 0x561895504b00 2024-03-21T10:49:18.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.160+0000 7f9f57400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561895504b00 0x56189550d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:18.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.160+0000 7f9f57400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561895504b00 0x56189550d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:42992/0 (socket says 172.21.15.31:42992) 2024-03-21T10:49:18.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.160+0000 7f9f57400640 1 -- 172.21.15.31:0/1857109176 learned_addr learned my addr 172.21.15.31:0/1857109176 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:18.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.160+0000 7f9f56a00640 1 --2- 172.21.15.31:0/1857109176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618955041e0 0x5618955045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:18.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.161+0000 7f9f57400640 1 -- 172.21.15.31:0/1857109176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618955041e0 msgr2=0x5618955045c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:18.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.161+0000 7f9f57400640 1 --2- 172.21.15.31:0/1857109176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618955041e0 0x5618955045c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.161+0000 7f9f56a00640 1 --2- 172.21.15.31:0/1857109176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56189550d8d0 0x56189550fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:18.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.161+0000 7f9f57400640 1 -- 172.21.15.31:0/1857109176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56189550d8d0 msgr2=0x56189550fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:18.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.161+0000 7f9f57400640 1 --2- 172.21.15.31:0/1857109176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56189550d8d0 0x56189550fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.161+0000 7f9f57400640 1 -- 172.21.15.31:0/1857109176 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5618954292c0 con 0x561895504b00 2024-03-21T10:49:18.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.161+0000 7f9f56a00640 1 --2- 172.21.15.31:0/1857109176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56189550d8d0 0x56189550fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:49:18.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.161+0000 7f9f57400640 1 --2- 172.21.15.31:0/1857109176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561895504b00 0x56189550d390 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f9f48011630 tx=0x7f9f48011af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e0c2ab0495f05c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:18.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.161+0000 7f9f4fe00640 1 --2- 172.21.15.31:0/1554874127 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9f3c0a30f0 0x7f9f3c0a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:18.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.161+0000 7f9f43e00640 1 -- 172.21.15.31:0/1857109176 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4801e070 con 0x561895504b00 2024-03-21T10:49:18.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.161+0000 7f9f43e00640 1 -- 172.21.15.31:0/1857109176 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f4800fbd0 con 0x561895504b00 2024-03-21T10:49:18.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.161+0000 7f9f43e00640 1 -- 172.21.15.31:0/1857109176 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f48019d00 con 0x561895504b00 2024-03-21T10:49:18.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.161+0000 7f9f4fe00640 1 -- 172.21.15.31:0/1554874127 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f9f500d0260 con 0x7f9f58107560 2024-03-21T10:49:18.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.162+0000 7f9f56a00640 1 --2- 172.21.15.31:0/1554874127 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9f3c0a30f0 0x7f9f3c0a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:18.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.162+0000 7f9f60fa8740 1 -- 172.21.15.31:0/1857109176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561895504b00 msgr2=0x56189550d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:18.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.162+0000 7f9f60fa8740 1 --2- 172.21.15.31:0/1857109176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561895504b00 0x56189550d390 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f9f48011630 tx=0x7f9f48011af0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.162+0000 7f9f60fa8740 1 -- 172.21.15.31:0/1857109176 shutdown_connections 2024-03-21T10:49:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.162+0000 7f9f60fa8740 1 --2- 172.21.15.31:0/1857109176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56189550d8d0 0x56189550fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.162+0000 7f9f60fa8740 1 --2- 172.21.15.31:0/1857109176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561895504b00 0x56189550d390 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.162+0000 7f9f60fa8740 1 --2- 172.21.15.31:0/1857109176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618955041e0 0x5618955045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.162+0000 7f9f60fa8740 1 -- 172.21.15.31:0/1857109176 >> 172.21.15.31:0/1857109176 conn(0x5618954feb00 msgr2=0x5618954feee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.162+0000 7f9f60fa8740 1 -- 172.21.15.31:0/1857109176 shutdown_connections 2024-03-21T10:49:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.162+0000 7f9f56a00640 1 --2- 172.21.15.31:0/1554874127 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9f3c0a30f0 0x7f9f3c0a55b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9f44009900 tx=0x7f9f4402df90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.162+0000 7f9f60fa8740 1 -- 172.21.15.31:0/1857109176 wait complete. 2024-03-21T10:49:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.163+0000 7f9f60fa8740 1 -- start start 2024-03-21T10:49:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.163+0000 7f9f60fa8740 1 Processor -- start 2024-03-21T10:49:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.163+0000 7f9f60fa8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618955041e0 0x56189550aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.163+0000 7f9f60fa8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561895504b00 0x56189550af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.164+0000 7f9f60fa8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56189550d8d0 0x56189550b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.164+0000 7f9f60fa8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561895424a20 con 0x561895504b00 2024-03-21T10:49:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.164+0000 7f9f60fa8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618954648d0 con 0x56189550d8d0 2024-03-21T10:49:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.164+0000 7f9f60fa8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x561895510500 con 0x5618955041e0 2024-03-21T10:49:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.164+0000 7f9f60fa8740 10 client.0 _ll_register_callbacks cb 0x5618954611c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.164+0000 7f9f57400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618955041e0 0x56189550aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.164+0000 7f9f57400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618955041e0 0x56189550aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:43008/0 (socket says 172.21.15.31:43008) 2024-03-21T10:49:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.164+0000 7f9f57400640 1 -- 172.21.15.31:0/3804529327 learned_addr learned my addr 172.21.15.31:0/3804529327 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.164+0000 7f9f57e00640 1 --2- 172.21.15.31:0/3804529327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56189550d8d0 0x56189550b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.164+0000 7f9f56a00640 1 --2- 172.21.15.31:0/3804529327 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561895504b00 0x56189550af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.164+0000 7f9f57400640 1 -- 172.21.15.31:0/3804529327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56189550d8d0 msgr2=0x56189550b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.164+0000 7f9f57400640 1 --2- 172.21.15.31:0/3804529327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56189550d8d0 0x56189550b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.164+0000 7f9f57400640 1 -- 172.21.15.31:0/3804529327 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561895504b00 msgr2=0x56189550af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.164+0000 7f9f57400640 1 --2- 172.21.15.31:0/3804529327 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561895504b00 0x56189550af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.164+0000 7f9f57400640 1 -- 172.21.15.31:0/3804529327 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56189517d210 con 0x5618955041e0 2024-03-21T10:49:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.164+0000 7f9f56a00640 1 --2- 172.21.15.31:0/3804529327 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561895504b00 0x56189550af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:49:18.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.164+0000 7f9f57400640 1 --2- 172.21.15.31:0/3804529327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618955041e0 0x56189550aa20 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f9f4801c040 tx=0x7f9f480196a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a530e94f47899478 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:18.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.165+0000 7f9f40c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:49:18.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.165+0000 7f9f40c00640 1 -- 172.21.15.31:0/3804529327 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f48026020 con 0x5618955041e0 2024-03-21T10:49:18.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.165+0000 7f9f40c00640 1 -- 172.21.15.31:0/3804529327 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f9f4800fd80 con 0x5618955041e0 2024-03-21T10:49:18.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.165+0000 7f9f60fa8740 1 -- 172.21.15.31:0/3804529327 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9f4800fbd0 con 0x5618955041e0 2024-03-21T10:49:18.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.165+0000 7f9f40c00640 1 -- 172.21.15.31:0/3804529327 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f48031050 con 0x5618955041e0 2024-03-21T10:49:18.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.165+0000 7f9f60fa8740 10 client.28300 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:18.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.165+0000 7f9f60fa8740 1 -- 172.21.15.31:0/3804529327 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x561895249620 con 0x5618955041e0 2024-03-21T10:49:18.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.165+0000 7f9f40c00640 1 -- 172.21.15.31:0/3804529327 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f4800fbd0 con 0x5618955041e0 2024-03-21T10:49:18.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.166+0000 7f9f40c00640 1 -- 172.21.15.31:0/3804529327 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f9f48021050 con 0x5618955041e0 2024-03-21T10:49:18.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.166+0000 7f9f60fa8740 10 client.28300 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:18.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.166+0000 7f9f60fa8740 10 client.28300 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:49:18.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.166+0000 7f9f60fa8740 20 client.28300 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:18.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.166+0000 7f9f60fa8740 1 --2- 172.21.15.31:0/3804529327 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56189551d8b0 0x56189551fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:18.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.166+0000 7f9f60fa8740 4 client.28300 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:49:18.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.166+0000 7f9f60fa8740 1 -- 172.21.15.31:0/3804529327 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561895475d70 con 0x56189551d8b0 2024-03-21T10:49:18.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.166+0000 7f9f56a00640 1 --2- 172.21.15.31:0/3804529327 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56189551d8b0 0x56189551fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:18.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.167+0000 7f9f56a00640 1 --2- 172.21.15.31:0/3804529327 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56189551d8b0 0x56189551fca0 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8ea1814de0113a6 server_cookie=c89b015ace8311eb in_seq=0 out_seq=0 2024-03-21T10:49:18.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.167+0000 7f9f40c00640 10 client.28300 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:18.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.167+0000 7f9f40c00640 1 -- 172.21.15.31:0/3804529327 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561895475d70 con 0x56189551d8b0 2024-03-21T10:49:18.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.167+0000 7f9f40c00640 10 client.28300 handle_command_reply: tid=0 2024-03-21T10:49:18.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.167+0000 7f9f60fa8740 1 client.28300 shutdown 2024-03-21T10:49:18.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.167+0000 7f9f60fa8740 1 -- 172.21.15.31:0/3804529327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618955041e0 msgr2=0x56189550aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:18.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.167+0000 7f9f60fa8740 1 --2- 172.21.15.31:0/3804529327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618955041e0 0x56189550aa20 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f9f4801c040 tx=0x7f9f480196a0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.168+0000 7f9f60fa8740 1 -- 172.21.15.31:0/3804529327 shutdown_connections 2024-03-21T10:49:18.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.168+0000 7f9f60fa8740 1 --2- 172.21.15.31:0/3804529327 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56189551d8b0 0x56189551fca0 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.168+0000 7f9f60fa8740 1 --2- 172.21.15.31:0/3804529327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56189550d8d0 0x56189550b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.168+0000 7f9f60fa8740 1 --2- 172.21.15.31:0/3804529327 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561895504b00 0x56189550af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.168+0000 7f9f40c00640 0 client.28300 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:18.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.168+0000 7f9f60fa8740 1 --2- 172.21.15.31:0/3804529327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618955041e0 0x56189550aa20 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.168+0000 7f9f60fa8740 1 -- 172.21.15.31:0/3804529327 >> 172.21.15.31:0/3804529327 conn(0x5618954feb00 msgr2=0x561895500850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:18.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.168+0000 7f9f60fa8740 1 -- 172.21.15.31:0/3804529327 shutdown_connections 2024-03-21T10:49:18.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.168+0000 7f9f60fa8740 1 -- 172.21.15.31:0/3804529327 wait complete. 2024-03-21T10:49:18.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.168+0000 7f9f60fa8740 20 client.28300 trim_cache size 0 max 16384 2024-03-21T10:49:18.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.188+0000 7f9f60fa8740 1 Processor -- start 2024-03-21T10:49:18.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.188+0000 7f9f60fa8740 1 -- start start 2024-03-21T10:49:18.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.189+0000 7f9f60fa8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5618955041e0 0x56189550d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:18.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.189+0000 7f9f60fa8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561895504b00 0x561895518c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:18.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.189+0000 7f9f60fa8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56189550d8d0 0x561895519150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:18.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.189+0000 7f9f60fa8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f581b2e70 con 0x5618955041e0 2024-03-21T10:49:18.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.189+0000 7f9f60fa8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f581b0970 con 0x561895504b00 2024-03-21T10:49:18.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.189+0000 7f9f60fa8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f581aea30 con 0x56189550d8d0 2024-03-21T10:49:18.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.189+0000 7f9f57400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5618955041e0 0x56189550d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:18.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.189+0000 7f9f56a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561895504b00 0x561895518c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:18.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.189+0000 7f9f57400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5618955041e0 0x56189550d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51816/0 (socket says 172.21.15.31:51816) 2024-03-21T10:49:18.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.189+0000 7f9f57400640 1 -- 172.21.15.31:0/1290783016 learned_addr learned my addr 172.21.15.31:0/1290783016 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:18.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.189+0000 7f9f56a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561895504b00 0x561895518c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49848/0 (socket says 172.21.15.31:49848) 2024-03-21T10:49:18.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.189+0000 7f9f57e00640 1 --2- 172.21.15.31:0/1290783016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56189550d8d0 0x561895519150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:18.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.189+0000 7f9f56a00640 1 -- 172.21.15.31:0/1290783016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56189550d8d0 msgr2=0x561895519150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:18.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.189+0000 7f9f56a00640 1 --2- 172.21.15.31:0/1290783016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56189550d8d0 0x561895519150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.189+0000 7f9f56a00640 1 -- 172.21.15.31:0/1290783016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5618955041e0 msgr2=0x56189550d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:18.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.189+0000 7f9f56a00640 1 --2- 172.21.15.31:0/1290783016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5618955041e0 0x56189550d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.189+0000 7f9f56a00640 1 -- 172.21.15.31:0/1290783016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5618954f9d10 con 0x561895504b00 2024-03-21T10:49:18.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.190+0000 7f9f56a00640 1 --2- 172.21.15.31:0/1290783016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561895504b00 0x561895518c10 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f9f44036040 tx=0x7f9f44045000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6402eff4e9a52901 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:18.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.190+0000 7f9f43400640 1 -- 172.21.15.31:0/1290783016 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4405c070 con 0x561895504b00 2024-03-21T10:49:18.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.190+0000 7f9f43400640 1 -- 172.21.15.31:0/1290783016 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f44048040 con 0x561895504b00 2024-03-21T10:49:18.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.190+0000 7f9f43400640 1 -- 172.21.15.31:0/1290783016 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f440055c0 con 0x561895504b00 2024-03-21T10:49:18.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.191+0000 7f9f60fa8740 1 -- 172.21.15.31:0/1290783016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561895504b00 msgr2=0x561895518c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:18.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.191+0000 7f9f60fa8740 1 --2- 172.21.15.31:0/1290783016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561895504b00 0x561895518c10 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f9f44036040 tx=0x7f9f44045000 comp rx=0 tx=0).stop 2024-03-21T10:49:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.191+0000 7f9f60fa8740 1 -- 172.21.15.31:0/1290783016 shutdown_connections 2024-03-21T10:49:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.191+0000 7f9f60fa8740 1 --2- 172.21.15.31:0/1290783016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56189550d8d0 0x561895519150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.191+0000 7f9f60fa8740 1 --2- 172.21.15.31:0/1290783016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561895504b00 0x561895518c10 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.191+0000 7f9f60fa8740 1 --2- 172.21.15.31:0/1290783016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5618955041e0 0x56189550d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.191+0000 7f9f60fa8740 1 -- 172.21.15.31:0/1290783016 >> 172.21.15.31:0/1290783016 conn(0x5618954feb00 msgr2=0x5618955092f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.191+0000 7f9f60fa8740 1 -- 172.21.15.31:0/1290783016 shutdown_connections 2024-03-21T10:49:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.191+0000 7f9f60fa8740 1 -- 172.21.15.31:0/1290783016 wait complete. 2024-03-21T10:49:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.191+0000 7f9f60fa8740 1 -- start start 2024-03-21T10:49:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.191+0000 7f9f60fa8740 1 Processor -- start 2024-03-21T10:49:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.192+0000 7f9f60fa8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618955041e0 0x561895507370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.192+0000 7f9f60fa8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561895504b00 0x5618955078b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.192+0000 7f9f60fa8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56189550d8d0 0x561895507df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.192+0000 7f9f60fa8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f581af030 con 0x56189550d8d0 2024-03-21T10:49:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.192+0000 7f9f60fa8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f581b0310 con 0x5618955041e0 2024-03-21T10:49:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.192+0000 7f9f60fa8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f581af650 con 0x561895504b00 2024-03-21T10:49:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.192+0000 7f9f60fa8740 10 client.0 _ll_register_callbacks cb 0x7f9f4801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.192+0000 7f9f57400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618955041e0 0x561895507370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.192+0000 7f9f57400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618955041e0 0x561895507370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49850/0 (socket says 172.21.15.31:49850) 2024-03-21T10:49:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.192+0000 7f9f57400640 1 -- 172.21.15.31:0/761233351 learned_addr learned my addr 172.21.15.31:0/761233351 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.192+0000 7f9f56a00640 1 --2- 172.21.15.31:0/761233351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561895504b00 0x5618955078b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.192+0000 7f9f57e00640 1 --2- 172.21.15.31:0/761233351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56189550d8d0 0x561895507df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.192+0000 7f9f57400640 1 -- 172.21.15.31:0/761233351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561895504b00 msgr2=0x5618955078b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.192+0000 7f9f57400640 1 --2- 172.21.15.31:0/761233351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561895504b00 0x5618955078b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.192+0000 7f9f57400640 1 -- 172.21.15.31:0/761233351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56189550d8d0 msgr2=0x561895507df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.192+0000 7f9f57400640 1 --2- 172.21.15.31:0/761233351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56189550d8d0 0x561895507df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.192+0000 7f9f57400640 1 -- 172.21.15.31:0/761233351 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5618955781f0 con 0x5618955041e0 2024-03-21T10:49:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.193+0000 7f9f57400640 1 --2- 172.21.15.31:0/761233351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618955041e0 0x561895507370 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f9f48011b30 tx=0x7f9f48004bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2a30250ad5134e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.193+0000 7f9f40c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:49:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.193+0000 7f9f40c00640 1 -- 172.21.15.31:0/761233351 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f48015020 con 0x5618955041e0 2024-03-21T10:49:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.193+0000 7f9f60fa8740 1 -- 172.21.15.31:0/761233351 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9f44048040 con 0x5618955041e0 2024-03-21T10:49:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.193+0000 7f9f40c00640 1 -- 172.21.15.31:0/761233351 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f9f48009e30 con 0x5618955041e0 2024-03-21T10:49:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.193+0000 7f9f40c00640 1 -- 172.21.15.31:0/761233351 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f4803e050 con 0x5618955041e0 2024-03-21T10:49:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.193+0000 7f9f60fa8740 10 client.28376 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.193+0000 7f9f60fa8740 1 -- 172.21.15.31:0/761233351 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x561895579500 con 0x5618955041e0 2024-03-21T10:49:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.194+0000 7f9f40c00640 1 -- 172.21.15.31:0/761233351 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f44048040 con 0x5618955041e0 2024-03-21T10:49:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.194+0000 7f9f40c00640 1 -- 172.21.15.31:0/761233351 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f9f48021040 con 0x5618955041e0 2024-03-21T10:49:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.194+0000 7f9f60fa8740 10 client.28376 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.194+0000 7f9f60fa8740 10 client.28376 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:49:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.194+0000 7f9f60fa8740 20 client.28376 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.194+0000 7f9f60fa8740 1 --2- 172.21.15.31:0/761233351 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561895579b40 0x56189557bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:18.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.194+0000 7f9f60fa8740 4 client.28376 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:49:18.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.194+0000 7f9f60fa8740 1 -- 172.21.15.31:0/761233351 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5618954700c0 con 0x561895579b40 2024-03-21T10:49:18.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.194+0000 7f9f56a00640 1 --2- 172.21.15.31:0/761233351 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561895579b40 0x56189557bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:18.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.195+0000 7f9f56a00640 1 --2- 172.21.15.31:0/761233351 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561895579b40 0x56189557bf60 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c53ce8592eab0a2 server_cookie=d328f175fd689732 in_seq=0 out_seq=0 2024-03-21T10:49:18.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.195+0000 7f9f40c00640 10 client.28376 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:18.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.201+0000 7f9f40c00640 1 -- 172.21.15.31:0/761233351 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5618954700c0 con 0x561895579b40 2024-03-21T10:49:18.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.201+0000 7f9f40c00640 10 client.28376 handle_command_reply: tid=0 2024-03-21T10:49:18.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.201+0000 7f9f60fa8740 1 client.28376 shutdown 2024-03-21T10:49:18.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.201+0000 7f9f60fa8740 1 -- 172.21.15.31:0/761233351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618955041e0 msgr2=0x561895507370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.201+0000 7f9f60fa8740 1 --2- 172.21.15.31:0/761233351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618955041e0 0x561895507370 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f9f48011b30 tx=0x7f9f48004bb0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.201+0000 7f9f60fa8740 1 -- 172.21.15.31:0/761233351 shutdown_connections 2024-03-21T10:49:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.201+0000 7f9f60fa8740 1 --2- 172.21.15.31:0/761233351 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561895579b40 0x56189557bf60 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.201+0000 7f9f60fa8740 1 --2- 172.21.15.31:0/761233351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56189550d8d0 0x561895507df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.201+0000 7f9f60fa8740 1 --2- 172.21.15.31:0/761233351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561895504b00 0x5618955078b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.201+0000 7f9f60fa8740 1 --2- 172.21.15.31:0/761233351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618955041e0 0x561895507370 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.201+0000 7f9f60fa8740 1 -- 172.21.15.31:0/761233351 >> 172.21.15.31:0/761233351 conn(0x5618954feb00 msgr2=0x5618954fd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.201+0000 7f9f40c00640 0 client.28376 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.202+0000 7f9f60fa8740 1 -- 172.21.15.31:0/761233351 shutdown_connections 2024-03-21T10:49:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.202+0000 7f9f60fa8740 1 -- 172.21.15.31:0/761233351 wait complete. 2024-03-21T10:49:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.202+0000 7f9f60fa8740 20 client.28376 trim_cache size 0 max 16384 2024-03-21T10:49:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:49:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:49:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:49:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:49:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.202+0000 7f9f5cc00640 1 -- 172.21.15.31:0/1554874127 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9f3c0a30f0 msgr2=0x7f9f3c0a55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.202+0000 7f9f5cc00640 1 --2- 172.21.15.31:0/1554874127 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9f3c0a30f0 0x7f9f3c0a55b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9f44009900 tx=0x7f9f4402df90 comp rx=0 tx=0).stop 2024-03-21T10:49:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.202+0000 7f9f5cc00640 1 -- 172.21.15.31:0/1554874127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f58107560 msgr2=0x7f9f5819e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.202+0000 7f9f5cc00640 1 --2- 172.21.15.31:0/1554874127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f58107560 0x7f9f5819e810 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f9f50009fd0 tx=0x7f9f5000ef10 comp rx=0 tx=0).stop 2024-03-21T10:49:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.203+0000 7f9f5cc00640 1 -- 172.21.15.31:0/1554874127 shutdown_connections 2024-03-21T10:49:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.203+0000 7f9f5cc00640 1 --2- 172.21.15.31:0/1554874127 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9f3c0a30f0 0x7f9f3c0a55b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.203+0000 7f9f5cc00640 1 --2- 172.21.15.31:0/1554874127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f58107560 0x7f9f5819e810 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.203+0000 7f9f5cc00640 1 --2- 172.21.15.31:0/1554874127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f58101e60 0x7f9f5819a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.203+0000 7f9f5cc00640 1 --2- 172.21.15.31:0/1554874127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f581014a0 0x7f9f58199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.203+0000 7f9f5cc00640 1 -- 172.21.15.31:0/1554874127 >> 172.21.15.31:0/1554874127 conn(0x7f9f580fcfb0 msgr2=0x7f9f580fe420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.203+0000 7f9f5cc00640 1 -- 172.21.15.31:0/1554874127 shutdown_connections 2024-03-21T10:49:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:18.203+0000 7f9f5cc00640 1 -- 172.21.15.31:0/1554874127 wait complete. 2024-03-21T10:49:19.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:19 smithi040 ceph-mon[36154]: pgmap v335: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-03-21T10:49:19.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:19 smithi031 ceph-mon[31673]: pgmap v335: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-03-21T10:49:19.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:19 smithi070 ceph-mon[36330]: pgmap v335: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-03-21T10:49:21.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:21 smithi040 ceph-mon[36154]: pgmap v336: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-03-21T10:49:21.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:21 smithi031 ceph-mon[31673]: pgmap v336: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-03-21T10:49:21.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:21 smithi070 ceph-mon[36330]: pgmap v336: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-03-21T10:49:23.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:23 smithi040 ceph-mon[36154]: pgmap v337: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-03-21T10:49:23.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:23 smithi031 ceph-mon[31673]: pgmap v337: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-03-21T10:49:23.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:23 smithi070 ceph-mon[36330]: pgmap v337: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-03-21T10:49:25.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:25 smithi040 ceph-mon[36154]: pgmap v338: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-03-21T10:49:25.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:25 smithi031 ceph-mon[31673]: pgmap v338: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-03-21T10:49:25.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:25 smithi070 ceph-mon[36330]: pgmap v338: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-03-21T10:49:27.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:27 smithi040 ceph-mon[36154]: pgmap v339: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-03-21T10:49:27.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:27 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:49:27.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:27 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:49:27.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:27 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:49:27.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:27 smithi031 ceph-mon[31673]: pgmap v339: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-03-21T10:49:27.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:27 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:49:27.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:27 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:49:27.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:27 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:49:27.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:27 smithi070 ceph-mon[36330]: pgmap v339: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-03-21T10:49:27.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:27 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:49:27.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:27 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:49:27.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:27 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:49:29.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:29 smithi040 ceph-mon[36154]: pgmap v340: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-03-21T10:49:29.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:29 smithi031 ceph-mon[31673]: pgmap v340: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-03-21T10:49:29.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:29 smithi070 ceph-mon[36330]: pgmap v340: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-03-21T10:49:31.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.367+0000 7f9b43e00640 1 -- 172.21.15.31:0/4107302732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b44104060 msgr2=0x7f9b44110570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.367+0000 7f9b43e00640 1 --2- 172.21.15.31:0/4107302732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b44104060 0x7f9b44110570 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009cd0 tx=0x7f9b3c02c0e0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.367+0000 7f9b43e00640 1 -- 172.21.15.31:0/4107302732 shutdown_connections 2024-03-21T10:49:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.367+0000 7f9b43e00640 1 --2- 172.21.15.31:0/4107302732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b44104060 0x7f9b44110570 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.367+0000 7f9b43e00640 1 --2- 172.21.15.31:0/4107302732 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b441036a0 0x7f9b44103b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.367+0000 7f9b43e00640 1 --2- 172.21.15.31:0/4107302732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b441096b0 0x7f9b44109a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.367+0000 7f9b43e00640 1 -- 172.21.15.31:0/4107302732 >> 172.21.15.31:0/4107302732 conn(0x7f9b440ff150 msgr2=0x7f9b44101570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.367+0000 7f9b43e00640 1 -- 172.21.15.31:0/4107302732 shutdown_connections 2024-03-21T10:49:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.367+0000 7f9b43e00640 1 -- 172.21.15.31:0/4107302732 wait complete. 2024-03-21T10:49:31.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.368+0000 7f9b43e00640 1 Processor -- start 2024-03-21T10:49:31.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.368+0000 7f9b43e00640 1 -- start start 2024-03-21T10:49:31.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.368+0000 7f9b43e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b441036a0 0x7f9b44199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:31.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.368+0000 7f9b43e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b44104060 0x7f9b4419a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:31.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.368+0000 7f9b43e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b441096b0 0x7f9b4419e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:31.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.368+0000 7f9b43e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b44113a80 con 0x7f9b44104060 2024-03-21T10:49:31.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.368+0000 7f9b43e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b44113900 con 0x7f9b441096b0 2024-03-21T10:49:31.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.368+0000 7f9b43e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b44113c00 con 0x7f9b441036a0 2024-03-21T10:49:31.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.369+0000 7f9b43400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b441096b0 0x7f9b4419e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:31.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.369+0000 7f9b42a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b441036a0 0x7f9b44199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:31.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.369+0000 7f9b43400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b441096b0 0x7f9b4419e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:43504/0 (socket says 172.21.15.31:43504) 2024-03-21T10:49:31.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.369+0000 7f9b42000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b44104060 0x7f9b4419a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:31.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.369+0000 7f9b43400640 1 -- 172.21.15.31:0/307044906 learned_addr learned my addr 172.21.15.31:0/307044906 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:31.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.369+0000 7f9b43400640 1 -- 172.21.15.31:0/307044906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b441036a0 msgr2=0x7f9b44199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:31.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.369+0000 7f9b43400640 1 --2- 172.21.15.31:0/307044906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b441036a0 0x7f9b44199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.369+0000 7f9b43400640 1 -- 172.21.15.31:0/307044906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b44104060 msgr2=0x7f9b4419a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:31.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.369+0000 7f9b43400640 1 --2- 172.21.15.31:0/307044906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b44104060 0x7f9b4419a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.369+0000 7f9b43400640 1 -- 172.21.15.31:0/307044906 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b4419ef30 con 0x7f9b441096b0 2024-03-21T10:49:31.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.369+0000 7f9b43400640 1 --2- 172.21.15.31:0/307044906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b441096b0 0x7f9b4419e800 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c005e30 tx=0x7f9b3c035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:31.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.370+0000 7f9b3b400640 1 -- 172.21.15.31:0/307044906 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c039040 con 0x7f9b441096b0 2024-03-21T10:49:31.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.370+0000 7f9b43e00640 1 -- 172.21.15.31:0/307044906 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b4419f1c0 con 0x7f9b441096b0 2024-03-21T10:49:31.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.370+0000 7f9b3b400640 1 -- 172.21.15.31:0/307044906 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b3c044070 con 0x7f9b441096b0 2024-03-21T10:49:31.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.370+0000 7f9b3b400640 1 -- 172.21.15.31:0/307044906 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c004460 con 0x7f9b441096b0 2024-03-21T10:49:31.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.370+0000 7f9b43e00640 1 -- 172.21.15.31:0/307044906 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b441a6ab0 con 0x7f9b441096b0 2024-03-21T10:49:31.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.373+0000 7f9b4c96e740 1 Processor -- start 2024-03-21T10:49:31.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.373+0000 7f9b3b400640 1 -- 172.21.15.31:0/307044906 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f9b3c032790 con 0x7f9b441096b0 2024-03-21T10:49:31.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.373+0000 7f9b4c96e740 1 -- start start 2024-03-21T10:49:31.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.374+0000 7f9b4c96e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bf1e0 0x5615557bf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:31.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.374+0000 7f9b4c96e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615557bfb00 0x5615557c8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:31.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.374+0000 7f9b4c96e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615557c88d0 0x5615557cacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:31.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.374+0000 7f9b4c96e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615557813b0 con 0x5615557bfb00 2024-03-21T10:49:31.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.374+0000 7f9b4c96e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561555723830 con 0x5615557c88d0 2024-03-21T10:49:31.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.374+0000 7f9b4c96e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615556cc5d0 con 0x5615557bf1e0 2024-03-21T10:49:31.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.374+0000 7f9b42000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bf1e0 0x5615557bf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:31.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.374+0000 7f9b42a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615557bfb00 0x5615557c8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:31.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.374+0000 7f9b42000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bf1e0 0x5615557bf5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:51078/0 (socket says 172.21.15.31:51078) 2024-03-21T10:49:31.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.374+0000 7f9b42000640 1 -- 172.21.15.31:0/758891532 learned_addr learned my addr 172.21.15.31:0/758891532 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:31.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.374+0000 7f9b42000640 1 --2- 172.21.15.31:0/758891532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615557c88d0 0x5615557cacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:31.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.374+0000 7f9b42000640 1 -- 172.21.15.31:0/758891532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615557c88d0 msgr2=0x5615557cacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:31.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.374+0000 7f9b42000640 1 --2- 172.21.15.31:0/758891532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615557c88d0 0x5615557cacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.374+0000 7f9b42000640 1 -- 172.21.15.31:0/758891532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615557bfb00 msgr2=0x5615557c8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:31.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.375+0000 7f9b42000640 1 --2- 172.21.15.31:0/758891532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615557bfb00 0x5615557c8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.375+0000 7f9b42000640 1 -- 172.21.15.31:0/758891532 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5615556e42c0 con 0x5615557bf1e0 2024-03-21T10:49:31.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.375+0000 7f9b42000640 1 --2- 172.21.15.31:0/758891532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bf1e0 0x5615557bf5c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9b30012790 tx=0x7f9b3000b7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8adc8449fb72755f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:31.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.375+0000 7f9b3b400640 1 --2- 172.21.15.31:0/307044906 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9b280a30f0 0x7f9b280a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:31.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.375+0000 7f9b2fe00640 1 -- 172.21.15.31:0/758891532 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b30012dd0 con 0x5615557bf1e0 2024-03-21T10:49:31.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.375+0000 7f9b2fe00640 1 -- 172.21.15.31:0/758891532 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b30010c50 con 0x5615557bf1e0 2024-03-21T10:49:31.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.375+0000 7f9b2fe00640 1 -- 172.21.15.31:0/758891532 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3001ea70 con 0x5615557bf1e0 2024-03-21T10:49:31.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.375+0000 7f9b3b400640 1 -- 172.21.15.31:0/307044906 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f9b3c0f1ff0 con 0x7f9b441096b0 2024-03-21T10:49:31.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.375+0000 7f9b42a00640 1 --2- 172.21.15.31:0/307044906 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9b280a30f0 0x7f9b280a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:31.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.376+0000 7f9b4c96e740 1 -- 172.21.15.31:0/758891532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bf1e0 msgr2=0x5615557bf5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:31.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.376+0000 7f9b4c96e740 1 --2- 172.21.15.31:0/758891532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bf1e0 0x5615557bf5c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9b30012790 tx=0x7f9b3000b7c0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.376+0000 7f9b4c96e740 1 -- 172.21.15.31:0/758891532 shutdown_connections 2024-03-21T10:49:31.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.376+0000 7f9b4c96e740 1 --2- 172.21.15.31:0/758891532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615557c88d0 0x5615557cacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.376+0000 7f9b4c96e740 1 --2- 172.21.15.31:0/758891532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615557bfb00 0x5615557c8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.376+0000 7f9b4c96e740 1 --2- 172.21.15.31:0/758891532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bf1e0 0x5615557bf5c0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.376+0000 7f9b4c96e740 1 -- 172.21.15.31:0/758891532 >> 172.21.15.31:0/758891532 conn(0x5615557b9b00 msgr2=0x5615557b9ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:31.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.376+0000 7f9b4c96e740 1 -- 172.21.15.31:0/758891532 shutdown_connections 2024-03-21T10:49:31.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.376+0000 7f9b4c96e740 1 -- 172.21.15.31:0/758891532 wait complete. 2024-03-21T10:49:31.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.376+0000 7f9b42a00640 1 --2- 172.21.15.31:0/307044906 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9b280a30f0 0x7f9b280a55b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x5615557bec90 tx=0x7f9b3400b390 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:31.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.377+0000 7f9b4c96e740 1 -- start start 2024-03-21T10:49:31.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.377+0000 7f9b4c96e740 1 Processor -- start 2024-03-21T10:49:31.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.377+0000 7f9b4c96e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615557bf1e0 0x5615557c58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:31.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.377+0000 7f9b4c96e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bfb00 0x5615557c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:31.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.377+0000 7f9b4c96e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615557c88d0 0x5615557c6320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:31.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.377+0000 7f9b4c96e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615556dfa20 con 0x5615557bf1e0 2024-03-21T10:49:31.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.377+0000 7f9b4c96e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56155571f8d0 con 0x5615557c88d0 2024-03-21T10:49:31.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.377+0000 7f9b4c96e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615557cb500 con 0x5615557bfb00 2024-03-21T10:49:31.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.377+0000 7f9b4c96e740 10 client.0 _ll_register_callbacks cb 0x56155571c1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:31.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.378+0000 7f9b42000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bfb00 0x5615557c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:31.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.378+0000 7f9b42a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615557bf1e0 0x5615557c58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:31.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.378+0000 7f9b42000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bfb00 0x5615557c5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:51084/0 (socket says 172.21.15.31:51084) 2024-03-21T10:49:31.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.378+0000 7f9b42000640 1 -- 172.21.15.31:0/4229400030 learned_addr learned my addr 172.21.15.31:0/4229400030 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:31.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.378+0000 7f9b43400640 1 --2- 172.21.15.31:0/4229400030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615557c88d0 0x5615557c6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:31.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.378+0000 7f9b42000640 1 -- 172.21.15.31:0/4229400030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615557c88d0 msgr2=0x5615557c6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:31.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.378+0000 7f9b42000640 1 --2- 172.21.15.31:0/4229400030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615557c88d0 0x5615557c6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.378+0000 7f9b42000640 1 -- 172.21.15.31:0/4229400030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615557bf1e0 msgr2=0x5615557c58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:31.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.378+0000 7f9b42000640 1 --2- 172.21.15.31:0/4229400030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615557bf1e0 0x5615557c58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.378+0000 7f9b42000640 1 -- 172.21.15.31:0/4229400030 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561555438210 con 0x5615557bfb00 2024-03-21T10:49:31.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.378+0000 7f9b42000640 1 --2- 172.21.15.31:0/4229400030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bfb00 0x5615557c5de0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f9b3000b6b0 tx=0x7f9b300109a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=772a08287454d7af server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:31.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.378+0000 7f9b2cc00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:49:31.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.379+0000 7f9b2cc00640 1 -- 172.21.15.31:0/4229400030 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3002c060 con 0x5615557bfb00 2024-03-21T10:49:31.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.379+0000 7f9b4c96e740 1 -- 172.21.15.31:0/4229400030 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9b30010c50 con 0x5615557bfb00 2024-03-21T10:49:31.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.379+0000 7f9b2cc00640 1 -- 172.21.15.31:0/4229400030 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f9b30004250 con 0x5615557bfb00 2024-03-21T10:49:31.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.379+0000 7f9b2cc00640 1 -- 172.21.15.31:0/4229400030 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b30033050 con 0x5615557bfb00 2024-03-21T10:49:31.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.379+0000 7f9b4c96e740 10 client.28318 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:31.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.379+0000 7f9b4c96e740 1 -- 172.21.15.31:0/4229400030 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x561555504620 con 0x5615557bfb00 2024-03-21T10:49:31.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.379+0000 7f9b2cc00640 1 -- 172.21.15.31:0/4229400030 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b30010c50 con 0x5615557bfb00 2024-03-21T10:49:31.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.380+0000 7f9b2cc00640 1 -- 172.21.15.31:0/4229400030 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f9b3001e860 con 0x5615557bfb00 2024-03-21T10:49:31.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.380+0000 7f9b4c96e740 10 client.28318 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:31.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.380+0000 7f9b4c96e740 10 client.28318 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:49:31.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.380+0000 7f9b4c96e740 20 client.28318 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:31.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.380+0000 7f9b4c96e740 1 --2- 172.21.15.31:0/4229400030 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5615557d8830 0x5615557dac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:31.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.380+0000 7f9b4c96e740 4 client.28318 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:49:31.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.380+0000 7f9b4c96e740 1 -- 172.21.15.31:0/4229400030 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561555730d70 con 0x5615557d8830 2024-03-21T10:49:31.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.380+0000 7f9b42a00640 1 --2- 172.21.15.31:0/4229400030 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5615557d8830 0x5615557dac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:31.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.380+0000 7f9b42a00640 1 --2- 172.21.15.31:0/4229400030 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5615557d8830 0x5615557dac20 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b36a0fc722c6e1d server_cookie=14745226efaa0ae8 in_seq=0 out_seq=0 2024-03-21T10:49:31.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.380+0000 7f9b2cc00640 10 client.28318 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:31.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.381+0000 7f9b2cc00640 1 -- 172.21.15.31:0/4229400030 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561555730d70 con 0x5615557d8830 2024-03-21T10:49:31.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.381+0000 7f9b2cc00640 10 client.28318 handle_command_reply: tid=0 2024-03-21T10:49:31.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.381+0000 7f9b4c96e740 1 client.28318 shutdown 2024-03-21T10:49:31.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.382+0000 7f9b4c96e740 1 -- 172.21.15.31:0/4229400030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bfb00 msgr2=0x5615557c5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:31.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.382+0000 7f9b4c96e740 1 --2- 172.21.15.31:0/4229400030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bfb00 0x5615557c5de0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f9b3000b6b0 tx=0x7f9b300109a0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.382+0000 7f9b4c96e740 1 -- 172.21.15.31:0/4229400030 shutdown_connections 2024-03-21T10:49:31.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.382+0000 7f9b4c96e740 1 --2- 172.21.15.31:0/4229400030 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5615557d8830 0x5615557dac20 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.382+0000 7f9b4c96e740 1 --2- 172.21.15.31:0/4229400030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615557c88d0 0x5615557c6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.382+0000 7f9b2cc00640 0 client.28318 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:31.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.382+0000 7f9b4c96e740 1 --2- 172.21.15.31:0/4229400030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bfb00 0x5615557c5de0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.382+0000 7f9b4c96e740 1 --2- 172.21.15.31:0/4229400030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615557bf1e0 0x5615557c58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.382+0000 7f9b4c96e740 1 -- 172.21.15.31:0/4229400030 >> 172.21.15.31:0/4229400030 conn(0x5615557b9b00 msgr2=0x5615557bb850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:31.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.382+0000 7f9b4c96e740 1 -- 172.21.15.31:0/4229400030 shutdown_connections 2024-03-21T10:49:31.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.382+0000 7f9b4c96e740 1 -- 172.21.15.31:0/4229400030 wait complete. 2024-03-21T10:49:31.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.382+0000 7f9b4c96e740 20 client.28318 trim_cache size 0 max 16384 2024-03-21T10:49:31.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.397+0000 7f9b4c96e740 1 Processor -- start 2024-03-21T10:49:31.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.397+0000 7f9b4c96e740 1 -- start start 2024-03-21T10:49:31.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.397+0000 7f9b4c96e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bf1e0 0x5615557c7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:31.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.398+0000 7f9b4c96e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615557bfb00 0x5615557d3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:31.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.398+0000 7f9b4c96e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615557c88d0 0x5615557d4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:31.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.398+0000 7f9b4c96e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b441b2ef0 con 0x5615557bfb00 2024-03-21T10:49:31.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.398+0000 7f9b4c96e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b441b09f0 con 0x5615557c88d0 2024-03-21T10:49:31.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.398+0000 7f9b4c96e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b441aeab0 con 0x5615557bf1e0 2024-03-21T10:49:31.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.398+0000 7f9b42a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bf1e0 0x5615557c7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:31.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.398+0000 7f9b43400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615557c88d0 0x5615557d4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:31.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.398+0000 7f9b42a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bf1e0 0x5615557c7fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:51088/0 (socket says 172.21.15.31:51088) 2024-03-21T10:49:31.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.398+0000 7f9b42a00640 1 -- 172.21.15.31:0/628287863 learned_addr learned my addr 172.21.15.31:0/628287863 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:31.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.398+0000 7f9b42000640 1 --2- 172.21.15.31:0/628287863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615557bfb00 0x5615557d3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:31.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.398+0000 7f9b42a00640 1 -- 172.21.15.31:0/628287863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615557c88d0 msgr2=0x5615557d4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:31.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.398+0000 7f9b42a00640 1 --2- 172.21.15.31:0/628287863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615557c88d0 0x5615557d4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.398+0000 7f9b42a00640 1 -- 172.21.15.31:0/628287863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615557bfb00 msgr2=0x5615557d3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:31.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.398+0000 7f9b42a00640 1 --2- 172.21.15.31:0/628287863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615557bfb00 0x5615557d3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.398+0000 7f9b42a00640 1 -- 172.21.15.31:0/628287863 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5615557b92f0 con 0x5615557bf1e0 2024-03-21T10:49:31.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.398+0000 7f9b42a00640 1 --2- 172.21.15.31:0/628287863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bf1e0 0x5615557c7fd0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f9b34003d00 tx=0x7f9b34021040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94f62c4687d34438 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:31.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.399+0000 7f9b2f400640 1 -- 172.21.15.31:0/628287863 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3401ea70 con 0x5615557bf1e0 2024-03-21T10:49:31.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.399+0000 7f9b2f400640 1 -- 172.21.15.31:0/628287863 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b3402f070 con 0x5615557bf1e0 2024-03-21T10:49:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.399+0000 7f9b2f400640 1 -- 172.21.15.31:0/628287863 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b34011730 con 0x5615557bf1e0 2024-03-21T10:49:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.399+0000 7f9b4c96e740 1 -- 172.21.15.31:0/628287863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bf1e0 msgr2=0x5615557c7fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.399+0000 7f9b4c96e740 1 --2- 172.21.15.31:0/628287863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bf1e0 0x5615557c7fd0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f9b34003d00 tx=0x7f9b34021040 comp rx=0 tx=0).stop 2024-03-21T10:49:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.399+0000 7f9b4c96e740 1 -- 172.21.15.31:0/628287863 shutdown_connections 2024-03-21T10:49:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.399+0000 7f9b4c96e740 1 --2- 172.21.15.31:0/628287863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615557c88d0 0x5615557d4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.399+0000 7f9b4c96e740 1 --2- 172.21.15.31:0/628287863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615557bfb00 0x5615557d3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.399+0000 7f9b4c96e740 1 --2- 172.21.15.31:0/628287863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bf1e0 0x5615557c7fd0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.399+0000 7f9b4c96e740 1 -- 172.21.15.31:0/628287863 >> 172.21.15.31:0/628287863 conn(0x5615557b9b00 msgr2=0x5615557c4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.400+0000 7f9b4c96e740 1 -- 172.21.15.31:0/628287863 shutdown_connections 2024-03-21T10:49:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.400+0000 7f9b4c96e740 1 -- 172.21.15.31:0/628287863 wait complete. 2024-03-21T10:49:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.400+0000 7f9b4c96e740 1 -- start start 2024-03-21T10:49:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.400+0000 7f9b4c96e740 1 Processor -- start 2024-03-21T10:49:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.401+0000 7f9b4c96e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615557bf1e0 0x5615557c2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.401+0000 7f9b4c96e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bfb00 0x5615557c3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.401+0000 7f9b4c96e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615557c88d0 0x56155582be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.401+0000 7f9b4c96e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b441af0b0 con 0x5615557c88d0 2024-03-21T10:49:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.401+0000 7f9b4c96e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b441b0390 con 0x5615557bf1e0 2024-03-21T10:49:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.401+0000 7f9b4c96e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b441af6d0 con 0x5615557bfb00 2024-03-21T10:49:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.401+0000 7f9b4c96e740 10 client.0 _ll_register_callbacks cb 0x7f9b3400b690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.401+0000 7f9b42a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615557bf1e0 0x5615557c2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.401+0000 7f9b42a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615557bf1e0 0x5615557c2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:43560/0 (socket says 172.21.15.31:43560) 2024-03-21T10:49:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.401+0000 7f9b42000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bfb00 0x5615557c3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.401+0000 7f9b42a00640 1 -- 172.21.15.31:0/1282671964 learned_addr learned my addr 172.21.15.31:0/1282671964 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.401+0000 7f9b43400640 1 --2- 172.21.15.31:0/1282671964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615557c88d0 0x56155582be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.401+0000 7f9b42a00640 1 -- 172.21.15.31:0/1282671964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bfb00 msgr2=0x5615557c3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.401+0000 7f9b42a00640 1 --2- 172.21.15.31:0/1282671964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bfb00 0x5615557c3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.401+0000 7f9b42a00640 1 -- 172.21.15.31:0/1282671964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615557c88d0 msgr2=0x56155582be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.401+0000 7f9b42a00640 1 --2- 172.21.15.31:0/1282671964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615557c88d0 0x56155582be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.401+0000 7f9b42a00640 1 -- 172.21.15.31:0/1282671964 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56155582c630 con 0x5615557bf1e0 2024-03-21T10:49:31.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.401+0000 7f9b42000640 1 --2- 172.21.15.31:0/1282671964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bfb00 0x5615557c3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:31.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.402+0000 7f9b42a00640 1 --2- 172.21.15.31:0/1282671964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615557bf1e0 0x5615557c2b50 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f9b340058c0 tx=0x7f9b3400b5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93207d3627793611 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:31.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.402+0000 7f9b2cc00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:49:31.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.402+0000 7f9b2cc00640 1 -- 172.21.15.31:0/1282671964 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3401edb0 con 0x5615557bf1e0 2024-03-21T10:49:31.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.402+0000 7f9b2cc00640 1 -- 172.21.15.31:0/1282671964 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f9b34005040 con 0x5615557bf1e0 2024-03-21T10:49:31.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.402+0000 7f9b4c96e740 1 -- 172.21.15.31:0/1282671964 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9b3402f070 con 0x5615557bf1e0 2024-03-21T10:49:31.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.402+0000 7f9b2cc00640 1 -- 172.21.15.31:0/1282671964 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b34045050 con 0x5615557bf1e0 2024-03-21T10:49:31.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.402+0000 7f9b4c96e740 10 client.28394 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:31.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.402+0000 7f9b4c96e740 1 -- 172.21.15.31:0/1282671964 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5615558349c0 con 0x5615557bf1e0 2024-03-21T10:49:31.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.403+0000 7f9b2cc00640 1 -- 172.21.15.31:0/1282671964 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b3402f070 con 0x5615557bf1e0 2024-03-21T10:49:31.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.403+0000 7f9b2cc00640 1 -- 172.21.15.31:0/1282671964 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f9b340046c0 con 0x5615557bf1e0 2024-03-21T10:49:31.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.403+0000 7f9b4c96e740 10 client.28394 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:31.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.403+0000 7f9b4c96e740 10 client.28394 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:49:31.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.403+0000 7f9b4c96e740 20 client.28394 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:31.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.403+0000 7f9b4c96e740 1 --2- 172.21.15.31:0/1282671964 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561555835060 0x561555837480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:31.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.403+0000 7f9b4c96e740 4 client.28394 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:49:31.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.403+0000 7f9b4c96e740 1 -- 172.21.15.31:0/1282671964 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56155572b0c0 con 0x561555835060 2024-03-21T10:49:31.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.403+0000 7f9b42000640 1 --2- 172.21.15.31:0/1282671964 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561555835060 0x561555837480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:31.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.404+0000 7f9b42000640 1 --2- 172.21.15.31:0/1282671964 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561555835060 0x561555837480 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b43ab88a2571ea2 server_cookie=177d4504e487dc31 in_seq=0 out_seq=0 2024-03-21T10:49:31.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.404+0000 7f9b2cc00640 10 client.28394 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:31.404 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.407+0000 7f9b2cc00640 1 -- 172.21.15.31:0/1282671964 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56155572b0c0 con 0x561555835060 2024-03-21T10:49:31.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.407+0000 7f9b2cc00640 10 client.28394 handle_command_reply: tid=0 2024-03-21T10:49:31.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.407+0000 7f9b4c96e740 1 client.28394 shutdown 2024-03-21T10:49:31.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.408+0000 7f9b4c96e740 1 -- 172.21.15.31:0/1282671964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615557bf1e0 msgr2=0x5615557c2b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:31.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.408+0000 7f9b4c96e740 1 --2- 172.21.15.31:0/1282671964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615557bf1e0 0x5615557c2b50 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f9b340058c0 tx=0x7f9b3400b5b0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.408+0000 7f9b4c96e740 1 -- 172.21.15.31:0/1282671964 shutdown_connections 2024-03-21T10:49:31.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.408+0000 7f9b4c96e740 1 --2- 172.21.15.31:0/1282671964 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x561555835060 0x561555837480 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.408+0000 7f9b4c96e740 1 --2- 172.21.15.31:0/1282671964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615557c88d0 0x56155582be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.408+0000 7f9b4c96e740 1 --2- 172.21.15.31:0/1282671964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5615557bfb00 0x5615557c3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.408+0000 7f9b4c96e740 1 --2- 172.21.15.31:0/1282671964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615557bf1e0 0x5615557c2b50 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.408+0000 7f9b4c96e740 1 -- 172.21.15.31:0/1282671964 >> 172.21.15.31:0/1282671964 conn(0x5615557b9b00 msgr2=0x5615557c5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:31.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.408+0000 7f9b2cc00640 0 client.28394 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:31.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.408+0000 7f9b4c96e740 1 -- 172.21.15.31:0/1282671964 shutdown_connections 2024-03-21T10:49:31.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.408+0000 7f9b4c96e740 1 -- 172.21.15.31:0/1282671964 wait complete. 2024-03-21T10:49:31.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.408+0000 7f9b4c96e740 20 client.28394 trim_cache size 0 max 16384 2024-03-21T10:49:31.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:49:31.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:49:31.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:49:31.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:49:31.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.409+0000 7f9b43e00640 1 -- 172.21.15.31:0/307044906 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9b280a30f0 msgr2=0x7f9b280a55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:31.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.409+0000 7f9b43e00640 1 --2- 172.21.15.31:0/307044906 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9b280a30f0 0x7f9b280a55b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x5615557bec90 tx=0x7f9b3400b390 comp rx=0 tx=0).stop 2024-03-21T10:49:31.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.409+0000 7f9b43e00640 1 -- 172.21.15.31:0/307044906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b441096b0 msgr2=0x7f9b4419e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:31.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.409+0000 7f9b43e00640 1 --2- 172.21.15.31:0/307044906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b441096b0 0x7f9b4419e800 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c005e30 tx=0x7f9b3c035d10 comp rx=0 tx=0).stop 2024-03-21T10:49:31.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.409+0000 7f9b43e00640 1 -- 172.21.15.31:0/307044906 shutdown_connections 2024-03-21T10:49:31.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.409+0000 7f9b43e00640 1 --2- 172.21.15.31:0/307044906 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9b280a30f0 0x7f9b280a55b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.409+0000 7f9b43e00640 1 --2- 172.21.15.31:0/307044906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b441096b0 0x7f9b4419e800 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.409+0000 7f9b43e00640 1 --2- 172.21.15.31:0/307044906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b44104060 0x7f9b4419a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.409+0000 7f9b43e00640 1 --2- 172.21.15.31:0/307044906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b441036a0 0x7f9b44199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:31.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.409+0000 7f9b43e00640 1 -- 172.21.15.31:0/307044906 >> 172.21.15.31:0/307044906 conn(0x7f9b440ff150 msgr2=0x7f9b440ff8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:31.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.409+0000 7f9b43e00640 1 -- 172.21.15.31:0/307044906 shutdown_connections 2024-03-21T10:49:31.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:31.409+0000 7f9b43e00640 1 -- 172.21.15.31:0/307044906 wait complete. 2024-03-21T10:49:31.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:31 smithi040 ceph-mon[36154]: pgmap v341: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 1.6 KiB/s wr, 10 op/s 2024-03-21T10:49:31.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:31 smithi031 ceph-mon[31673]: pgmap v341: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 1.6 KiB/s wr, 10 op/s 2024-03-21T10:49:31.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:31 smithi070 ceph-mon[36330]: pgmap v341: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 1.6 KiB/s wr, 10 op/s 2024-03-21T10:49:33.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:33 smithi040 ceph-mon[36154]: pgmap v342: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:49:33.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:33 smithi031 ceph-mon[31673]: pgmap v342: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:49:33.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:33 smithi070 ceph-mon[36330]: pgmap v342: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:49:35.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:35 smithi040 ceph-mon[36154]: pgmap v343: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-21T10:49:35.553 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:35 smithi031 ceph-mon[31673]: pgmap v343: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-21T10:49:35.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:35 smithi070 ceph-mon[36330]: pgmap v343: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-21T10:49:37.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:37 smithi031 ceph-mon[31673]: pgmap v344: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-21T10:49:37.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:37 smithi070 ceph-mon[36330]: pgmap v344: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-21T10:49:37.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:37 smithi040 ceph-mon[36154]: pgmap v344: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-21T10:49:39.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:39 smithi031 ceph-mon[31673]: pgmap v345: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:49:39.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:39 smithi070 ceph-mon[36330]: pgmap v345: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:49:39.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:39 smithi040 ceph-mon[36154]: pgmap v345: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:49:41.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:41 smithi031 ceph-mon[31673]: pgmap v346: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-21T10:49:41.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:41 smithi070 ceph-mon[36330]: pgmap v346: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-21T10:49:41.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:41 smithi040 ceph-mon[36154]: pgmap v346: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-21T10:49:43.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:43 smithi031 ceph-mon[31673]: pgmap v347: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:49:43.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:43 smithi070 ceph-mon[36330]: pgmap v347: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:49:43.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:43 smithi040 ceph-mon[36154]: pgmap v347: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:49:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.557+0000 7f512a800640 1 -- 172.21.15.31:0/843838301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5124068f40 msgr2=0x7f51240693c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.557+0000 7f512a800640 1 --2- 172.21.15.31:0/843838301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5124068f40 0x7f51240693c0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f5110009900 tx=0x7f511002bc30 comp rx=0 tx=0).stop 2024-03-21T10:49:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.557+0000 7f512a800640 1 -- 172.21.15.31:0/843838301 shutdown_connections 2024-03-21T10:49:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.557+0000 7f512a800640 1 --2- 172.21.15.31:0/843838301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5124069900 0x7f512410e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.557+0000 7f512a800640 1 --2- 172.21.15.31:0/843838301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5124068f40 0x7f51240693c0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.557+0000 7f512a800640 1 --2- 172.21.15.31:0/843838301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5124107650 0x7f5124107a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.557+0000 7f512a800640 1 -- 172.21.15.31:0/843838301 >> 172.21.15.31:0/843838301 conn(0x7f5124077690 msgr2=0x7f5124077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.558+0000 7f512a800640 1 -- 172.21.15.31:0/843838301 shutdown_connections 2024-03-21T10:49:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.558+0000 7f512a800640 1 -- 172.21.15.31:0/843838301 wait complete. 2024-03-21T10:49:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.559+0000 7f512a800640 1 Processor -- start 2024-03-21T10:49:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.559+0000 7f512a800640 1 -- start start 2024-03-21T10:49:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.559+0000 7f512a800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5124068f40 0x7f5124197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.559+0000 7f512a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5124069900 0x7f5124198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.560+0000 7f512a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5124107650 0x7f512419c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.560+0000 7f512a800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5124111a60 con 0x7f5124107650 2024-03-21T10:49:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.560+0000 7f512a800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51241118e0 con 0x7f5124069900 2024-03-21T10:49:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.560+0000 7f512a800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5124111be0 con 0x7f5124068f40 2024-03-21T10:49:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.560+0000 7f5123400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5124068f40 0x7f5124197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.560+0000 7f5122a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5124069900 0x7f5124198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.560+0000 7f5123400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5124068f40 0x7f5124197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:44256/0 (socket says 172.21.15.31:44256) 2024-03-21T10:49:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.560+0000 7f5123400640 1 -- 172.21.15.31:0/3857747755 learned_addr learned my addr 172.21.15.31:0/3857747755 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.560+0000 7f5123e00640 1 --2- 172.21.15.31:0/3857747755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5124107650 0x7f512419c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.560+0000 7f5123400640 1 -- 172.21.15.31:0/3857747755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5124069900 msgr2=0x7f5124198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.560+0000 7f5123400640 1 --2- 172.21.15.31:0/3857747755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5124069900 0x7f5124198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.560+0000 7f5123400640 1 -- 172.21.15.31:0/3857747755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5124107650 msgr2=0x7f512419c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.560+0000 7f5123400640 1 --2- 172.21.15.31:0/3857747755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5124107650 0x7f512419c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.560+0000 7f5123400640 1 -- 172.21.15.31:0/3857747755 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f512419cf10 con 0x7f5124068f40 2024-03-21T10:49:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.560+0000 7f5122a00640 1 --2- 172.21.15.31:0/3857747755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5124069900 0x7f5124198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.560+0000 7f5123e00640 1 --2- 172.21.15.31:0/3857747755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5124107650 0x7f512419c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:49:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.561+0000 7f5123400640 1 --2- 172.21.15.31:0/3857747755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5124068f40 0x7f5124197f20 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f511400e9f0 tx=0x7f511400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.561+0000 7f511be00640 1 -- 172.21.15.31:0/3857747755 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511400cd80 con 0x7f5124068f40 2024-03-21T10:49:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.561+0000 7f512a800640 1 -- 172.21.15.31:0/3857747755 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f512419d200 con 0x7f5124068f40 2024-03-21T10:49:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.561+0000 7f511be00640 1 -- 172.21.15.31:0/3857747755 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5114010910 con 0x7f5124068f40 2024-03-21T10:49:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.561+0000 7f511be00640 1 -- 172.21.15.31:0/3857747755 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51140188f0 con 0x7f5124068f40 2024-03-21T10:49:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.561+0000 7f512a800640 1 -- 172.21.15.31:0/3857747755 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51241a4a90 con 0x7f5124068f40 2024-03-21T10:49:44.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.564+0000 7f512d122740 1 Processor -- start 2024-03-21T10:49:44.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.564+0000 7f512d122740 1 -- start start 2024-03-21T10:49:44.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.565+0000 7f512d122740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56245e9bd990 0x56245e9bdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:44.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.565+0000 7f512d122740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56245e9be2b0 0x56245e9cb590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:44.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.565+0000 7f512d122740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56245e9cbad0 0x56245e9cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:44.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.565+0000 7f512d122740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56245e9803b0 con 0x56245e9cbad0 2024-03-21T10:49:44.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.565+0000 7f512d122740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56245e922830 con 0x56245e9bd990 2024-03-21T10:49:44.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.565+0000 7f512d122740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56245e8cb5d0 con 0x56245e9be2b0 2024-03-21T10:49:44.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.565+0000 7f5123e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56245e9bd990 0x56245e9bdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:44.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.565+0000 7f5123e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56245e9bd990 0x56245e9bdd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44116/0 (socket says 172.21.15.31:44116) 2024-03-21T10:49:44.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.565+0000 7f5123e00640 1 -- 172.21.15.31:0/528466192 learned_addr learned my addr 172.21.15.31:0/528466192 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:44.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.565+0000 7f5122a00640 1 --2- 172.21.15.31:0/528466192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56245e9cbad0 0x56245e9cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:44.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.565+0000 7f5123e00640 1 -- 172.21.15.31:0/528466192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56245e9be2b0 msgr2=0x56245e9cb590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:49:44.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.565+0000 7f5123e00640 1 --2- 172.21.15.31:0/528466192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56245e9be2b0 0x56245e9cb590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.566+0000 7f5123e00640 1 -- 172.21.15.31:0/528466192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56245e9cbad0 msgr2=0x56245e9cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:44.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.566+0000 7f5123e00640 1 --2- 172.21.15.31:0/528466192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56245e9cbad0 0x56245e9cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.566+0000 7f5123e00640 1 -- 172.21.15.31:0/528466192 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56245e8e32c0 con 0x56245e9bd990 2024-03-21T10:49:44.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.566+0000 7f5123e00640 1 --2- 172.21.15.31:0/528466192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56245e9bd990 0x56245e9bdd70 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f511c009fd0 tx=0x7f511c0108e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0f32f285770c207 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:44.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.566+0000 7f511be00640 1 -- 172.21.15.31:0/3857747755 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f5114010ac0 con 0x7f5124068f40 2024-03-21T10:49:44.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.566+0000 7f5118c00640 1 -- 172.21.15.31:0/528466192 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511c011760 con 0x56245e9bd990 2024-03-21T10:49:44.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.566+0000 7f5118c00640 1 -- 172.21.15.31:0/528466192 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f511c011de0 con 0x56245e9bd990 2024-03-21T10:49:44.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.566+0000 7f5118c00640 1 -- 172.21.15.31:0/528466192 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511c00d340 con 0x56245e9bd990 2024-03-21T10:49:44.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.567+0000 7f512d122740 1 -- 172.21.15.31:0/528466192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56245e9bd990 msgr2=0x56245e9bdd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:44.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.567+0000 7f512d122740 1 --2- 172.21.15.31:0/528466192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56245e9bd990 0x56245e9bdd70 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f511c009fd0 tx=0x7f511c0108e0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.567+0000 7f512d122740 1 -- 172.21.15.31:0/528466192 shutdown_connections 2024-03-21T10:49:44.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.567+0000 7f512d122740 1 --2- 172.21.15.31:0/528466192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56245e9cbad0 0x56245e9cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.567+0000 7f512d122740 1 --2- 172.21.15.31:0/528466192 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56245e9be2b0 0x56245e9cb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.567+0000 7f512d122740 1 --2- 172.21.15.31:0/528466192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56245e9bd990 0x56245e9bdd70 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.567+0000 7f512d122740 1 -- 172.21.15.31:0/528466192 >> 172.21.15.31:0/528466192 conn(0x56245e9b88f0 msgr2=0x56245e9b8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:44.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.567+0000 7f512d122740 1 -- 172.21.15.31:0/528466192 shutdown_connections 2024-03-21T10:49:44.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.567+0000 7f512d122740 1 -- 172.21.15.31:0/528466192 wait complete. 2024-03-21T10:49:44.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.567+0000 7f511be00640 1 --2- 172.21.15.31:0/3857747755 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f51080a2f90 0x7f51080a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:44.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.567+0000 7f512d122740 1 -- start start 2024-03-21T10:49:44.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.568+0000 7f512d122740 1 Processor -- start 2024-03-21T10:49:44.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.568+0000 7f511be00640 1 -- 172.21.15.31:0/3857747755 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f51140cff40 con 0x7f5124068f40 2024-03-21T10:49:44.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.568+0000 7f5123e00640 1 --2- 172.21.15.31:0/3857747755 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f51080a2f90 0x7f51080a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:44.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.568+0000 7f512d122740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56245e9bd990 0x56245e9c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:44.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.568+0000 7f512d122740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56245e9be2b0 0x56245e9c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:44.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.568+0000 7f512d122740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56245e9cbad0 0x56245e9c96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:44.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.568+0000 7f512d122740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56245e8dea20 con 0x56245e9be2b0 2024-03-21T10:49:44.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.568+0000 7f512d122740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56245e91e8d0 con 0x56245e9cbad0 2024-03-21T10:49:44.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.568+0000 7f512d122740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x56245e9ce700 con 0x56245e9bd990 2024-03-21T10:49:44.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.569+0000 7f512d122740 10 client.0 _ll_register_callbacks cb 0x56245e91b1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:44.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.569+0000 7f5123400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56245e9bd990 0x56245e9c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:44.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.569+0000 7f5123400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56245e9bd990 0x56245e9c8c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:44258/0 (socket says 172.21.15.31:44258) 2024-03-21T10:49:44.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.569+0000 7f5123400640 1 -- 172.21.15.31:0/4168829386 learned_addr learned my addr 172.21.15.31:0/4168829386 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:44.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.569+0000 7f5122a00640 1 --2- 172.21.15.31:0/4168829386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56245e9be2b0 0x56245e9c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:44.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.569+0000 7f5123e00640 1 --2- 172.21.15.31:0/3857747755 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f51080a2f90 0x7f51080a5450 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f511c010de0 tx=0x7f511c011f90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:44.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.569+0000 7f5123e00640 1 --2- 172.21.15.31:0/4168829386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56245e9cbad0 0x56245e9c96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:44.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.569+0000 7f5123400640 1 -- 172.21.15.31:0/4168829386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56245e9cbad0 msgr2=0x56245e9c96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:44.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.569+0000 7f5123400640 1 --2- 172.21.15.31:0/4168829386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56245e9cbad0 0x56245e9c96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.569+0000 7f5123400640 1 -- 172.21.15.31:0/4168829386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56245e9be2b0 msgr2=0x56245e9c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:44.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.569+0000 7f5123400640 1 --2- 172.21.15.31:0/4168829386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56245e9be2b0 0x56245e9c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.569+0000 7f5123400640 1 -- 172.21.15.31:0/4168829386 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56245e637210 con 0x56245e9bd990 2024-03-21T10:49:44.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.569+0000 7f5123e00640 1 --2- 172.21.15.31:0/4168829386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56245e9cbad0 0x56245e9c96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:49:44.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.569+0000 7f5123400640 1 --2- 172.21.15.31:0/4168829386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56245e9bd990 0x56245e9c8c30 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f51140cd350 tx=0x7f511409b9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ffa4dc4542877f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:44.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.569+0000 7f510d600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:49:44.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.570+0000 7f510d600640 1 -- 172.21.15.31:0/4168829386 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51140a49a0 con 0x56245e9bd990 2024-03-21T10:49:44.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.570+0000 7f510d600640 1 -- 172.21.15.31:0/4168829386 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f511409b3e0 con 0x56245e9bd990 2024-03-21T10:49:44.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.570+0000 7f512d122740 1 -- 172.21.15.31:0/4168829386 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f511c011de0 con 0x56245e9bd990 2024-03-21T10:49:44.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.570+0000 7f510d600640 1 -- 172.21.15.31:0/4168829386 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f511409b5e0 con 0x56245e9bd990 2024-03-21T10:49:44.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.570+0000 7f512d122740 10 client.28333 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:44.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.570+0000 7f512d122740 1 -- 172.21.15.31:0/4168829386 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x56245e703620 con 0x56245e9bd990 2024-03-21T10:49:44.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.570+0000 7f510d600640 1 -- 172.21.15.31:0/4168829386 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f511c011de0 con 0x56245e9bd990 2024-03-21T10:49:44.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.570+0000 7f510d600640 1 -- 172.21.15.31:0/4168829386 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f5114097e00 con 0x56245e9bd990 2024-03-21T10:49:44.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.570+0000 7f512d122740 10 client.28333 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:44.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.570+0000 7f512d122740 10 client.28333 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:49:44.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.570+0000 7f512d122740 20 client.28333 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:44.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.571+0000 7f512d122740 1 --2- 172.21.15.31:0/4168829386 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56245e9dbbe0 0x56245e9ddfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:44.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.571+0000 7f512d122740 4 client.28333 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:49:44.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.571+0000 7f512d122740 1 -- 172.21.15.31:0/4168829386 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56245e92fd70 con 0x56245e9dbbe0 2024-03-21T10:49:44.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.571+0000 7f5122a00640 1 --2- 172.21.15.31:0/4168829386 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56245e9dbbe0 0x56245e9ddfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:44.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.571+0000 7f5122a00640 1 --2- 172.21.15.31:0/4168829386 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56245e9dbbe0 0x56245e9ddfd0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d821df29cf26e5a server_cookie=c3d5c3deff0b221 in_seq=0 out_seq=0 2024-03-21T10:49:44.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.571+0000 7f510d600640 10 client.28333 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:44.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.572+0000 7f510d600640 1 -- 172.21.15.31:0/4168829386 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56245e92fd70 con 0x56245e9dbbe0 2024-03-21T10:49:44.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.572+0000 7f510d600640 10 client.28333 handle_command_reply: tid=0 2024-03-21T10:49:44.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.572+0000 7f512d122740 1 client.28333 shutdown 2024-03-21T10:49:44.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.573+0000 7f512d122740 1 -- 172.21.15.31:0/4168829386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56245e9bd990 msgr2=0x56245e9c8c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:44.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.573+0000 7f512d122740 1 --2- 172.21.15.31:0/4168829386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56245e9bd990 0x56245e9c8c30 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f51140cd350 tx=0x7f511409b9e0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.573+0000 7f512d122740 1 -- 172.21.15.31:0/4168829386 shutdown_connections 2024-03-21T10:49:44.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.573+0000 7f512d122740 1 --2- 172.21.15.31:0/4168829386 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56245e9dbbe0 0x56245e9ddfd0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.573+0000 7f512d122740 1 --2- 172.21.15.31:0/4168829386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56245e9cbad0 0x56245e9c96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.573+0000 7f512d122740 1 --2- 172.21.15.31:0/4168829386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56245e9be2b0 0x56245e9c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.573+0000 7f510d600640 0 client.28333 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:44.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.573+0000 7f512d122740 1 --2- 172.21.15.31:0/4168829386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56245e9bd990 0x56245e9c8c30 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.573+0000 7f512d122740 1 -- 172.21.15.31:0/4168829386 >> 172.21.15.31:0/4168829386 conn(0x56245e9b88f0 msgr2=0x56245e9ba640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:44.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.573+0000 7f512d122740 1 -- 172.21.15.31:0/4168829386 shutdown_connections 2024-03-21T10:49:44.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.573+0000 7f512d122740 1 -- 172.21.15.31:0/4168829386 wait complete. 2024-03-21T10:49:44.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.573+0000 7f512d122740 20 client.28333 trim_cache size 0 max 16384 2024-03-21T10:49:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.588+0000 7f512d122740 1 Processor -- start 2024-03-21T10:49:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.588+0000 7f512d122740 1 -- start start 2024-03-21T10:49:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.588+0000 7f512d122740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56245e9bd990 0x56245e9cb360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.589+0000 7f512d122740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56245e9be2b0 0x56245e9b8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:44.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.589+0000 7f512d122740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56245e9cbad0 0x56245e9d6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:44.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.589+0000 7f512d122740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51241aed50 con 0x56245e9bd990 2024-03-21T10:49:44.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.589+0000 7f512d122740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51241ac850 con 0x56245e9be2b0 2024-03-21T10:49:44.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.589+0000 7f512d122740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51241aa910 con 0x56245e9cbad0 2024-03-21T10:49:44.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.589+0000 7f5123e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56245e9cbad0 0x56245e9d6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:44.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.589+0000 7f5123400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56245e9bd990 0x56245e9cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:44.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.589+0000 7f5122a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56245e9be2b0 0x56245e9b8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:44.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.589+0000 7f5123e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56245e9cbad0 0x56245e9d6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:44262/0 (socket says 172.21.15.31:44262) 2024-03-21T10:49:44.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.589+0000 7f5123e00640 1 -- 172.21.15.31:0/2978362239 learned_addr learned my addr 172.21.15.31:0/2978362239 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:44.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.589+0000 7f5123e00640 1 -- 172.21.15.31:0/2978362239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56245e9be2b0 msgr2=0x56245e9b8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:44.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.589+0000 7f5123e00640 1 --2- 172.21.15.31:0/2978362239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56245e9be2b0 0x56245e9b8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.589+0000 7f5123e00640 1 -- 172.21.15.31:0/2978362239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56245e9bd990 msgr2=0x56245e9cb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:44.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.589+0000 7f5123e00640 1 --2- 172.21.15.31:0/2978362239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56245e9bd990 0x56245e9cb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.589+0000 7f5123e00640 1 -- 172.21.15.31:0/2978362239 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56245e9c7c00 con 0x56245e9cbad0 2024-03-21T10:49:44.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.589+0000 7f5122a00640 1 --2- 172.21.15.31:0/2978362239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56245e9be2b0 0x56245e9b8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:44.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.590+0000 7f5123e00640 1 --2- 172.21.15.31:0/2978362239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56245e9cbad0 0x56245e9d6e10 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f511c009fa0 tx=0x7f511c009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=215f8681077189d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:44.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.590+0000 7f510fe00640 1 -- 172.21.15.31:0/2978362239 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511c004e50 con 0x56245e9cbad0 2024-03-21T10:49:44.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.590+0000 7f510fe00640 1 -- 172.21.15.31:0/2978362239 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f511c00e910 con 0x56245e9cbad0 2024-03-21T10:49:44.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.590+0000 7f510fe00640 1 -- 172.21.15.31:0/2978362239 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511c005680 con 0x56245e9cbad0 2024-03-21T10:49:44.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.590+0000 7f512d122740 1 -- 172.21.15.31:0/2978362239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56245e9cbad0 msgr2=0x56245e9d6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:44.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.591+0000 7f512d122740 1 --2- 172.21.15.31:0/2978362239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56245e9cbad0 0x56245e9d6e10 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f511c009fa0 tx=0x7f511c009500 comp rx=0 tx=0).stop 2024-03-21T10:49:44.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.591+0000 7f512d122740 1 -- 172.21.15.31:0/2978362239 shutdown_connections 2024-03-21T10:49:44.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.591+0000 7f512d122740 1 --2- 172.21.15.31:0/2978362239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56245e9cbad0 0x56245e9d6e10 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.591+0000 7f512d122740 1 --2- 172.21.15.31:0/2978362239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56245e9be2b0 0x56245e9b8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.591+0000 7f512d122740 1 --2- 172.21.15.31:0/2978362239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56245e9bd990 0x56245e9cb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.591+0000 7f512d122740 1 -- 172.21.15.31:0/2978362239 >> 172.21.15.31:0/2978362239 conn(0x56245e9b88f0 msgr2=0x56245e9b7950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:44.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.591+0000 7f512d122740 1 -- 172.21.15.31:0/2978362239 shutdown_connections 2024-03-21T10:49:44.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.591+0000 7f512d122740 1 -- 172.21.15.31:0/2978362239 wait complete. 2024-03-21T10:49:44.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.591+0000 7f512d122740 1 -- start start 2024-03-21T10:49:44.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.591+0000 7f512d122740 1 Processor -- start 2024-03-21T10:49:44.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.592+0000 7f512d122740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56245e9bd990 0x56245e9d3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:44.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.592+0000 7f512d122740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56245e9be2b0 0x56245e9cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:44.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.592+0000 7f512d122740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56245e9cbad0 0x56245e9cfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:44.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.592+0000 7f512d122740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51241aaf10 con 0x56245e9cbad0 2024-03-21T10:49:44.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.592+0000 7f512d122740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51241ac1f0 con 0x56245e9bd990 2024-03-21T10:49:44.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.592+0000 7f512d122740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51241ab530 con 0x56245e9be2b0 2024-03-21T10:49:44.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.592+0000 7f512d122740 10 client.0 _ll_register_callbacks cb 0x7f5114009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:44.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.592+0000 7f5123400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56245e9bd990 0x56245e9d3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:44.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.592+0000 7f5123400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56245e9bd990 0x56245e9d3600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44150/0 (socket says 172.21.15.31:44150) 2024-03-21T10:49:44.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.592+0000 7f5123400640 1 -- 172.21.15.31:0/3606663621 learned_addr learned my addr 172.21.15.31:0/3606663621 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.592+0000 7f5122a00640 1 --2- 172.21.15.31:0/3606663621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56245e9be2b0 0x56245e9cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.592+0000 7f5123e00640 1 --2- 172.21.15.31:0/3606663621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56245e9cbad0 0x56245e9cfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.593+0000 7f5123400640 1 -- 172.21.15.31:0/3606663621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56245e9be2b0 msgr2=0x56245e9cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.593+0000 7f5123400640 1 --2- 172.21.15.31:0/3606663621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56245e9be2b0 0x56245e9cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.593+0000 7f5123400640 1 -- 172.21.15.31:0/3606663621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56245e9cbad0 msgr2=0x56245e9cfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.593+0000 7f5123400640 1 --2- 172.21.15.31:0/3606663621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56245e9cbad0 0x56245e9cfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.593+0000 7f5123400640 1 -- 172.21.15.31:0/3606663621 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56245e9ce230 con 0x56245e9bd990 2024-03-21T10:49:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.593+0000 7f5123e00640 1 --2- 172.21.15.31:0/3606663621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56245e9cbad0 0x56245e9cfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:49:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.593+0000 7f5123400640 1 --2- 172.21.15.31:0/3606663621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56245e9bd990 0x56245e9d3600 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f511409a240 tx=0x7f5114097f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf56fe3cf34219cd server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.593+0000 7f510d600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:49:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.593+0000 7f510d600640 1 -- 172.21.15.31:0/3606663621 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51140a49a0 con 0x56245e9bd990 2024-03-21T10:49:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.593+0000 7f510d600640 1 -- 172.21.15.31:0/3606663621 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f5114010640 con 0x56245e9bd990 2024-03-21T10:49:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.593+0000 7f512d122740 1 -- 172.21.15.31:0/3606663621 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f511c00e910 con 0x56245e9bd990 2024-03-21T10:49:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.593+0000 7f510d600640 1 -- 172.21.15.31:0/3606663621 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f511409b0c0 con 0x56245e9bd990 2024-03-21T10:49:44.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.593+0000 7f512d122740 10 client.28412 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:44.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.593+0000 7f512d122740 1 -- 172.21.15.31:0/3606663621 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x56245ea38120 con 0x56245e9bd990 2024-03-21T10:49:44.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.594+0000 7f510d600640 1 -- 172.21.15.31:0/3606663621 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f511c00e910 con 0x56245e9bd990 2024-03-21T10:49:44.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.594+0000 7f510d600640 1 -- 172.21.15.31:0/3606663621 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f5114018bb0 con 0x56245e9bd990 2024-03-21T10:49:44.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.594+0000 7f512d122740 10 client.28412 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:44.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.594+0000 7f512d122740 10 client.28412 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:49:44.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.594+0000 7f512d122740 20 client.28412 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:44.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.594+0000 7f512d122740 1 --2- 172.21.15.31:0/3606663621 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56245ea38790 0x56245ea3abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:44.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.594+0000 7f512d122740 4 client.28412 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:49:44.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.594+0000 7f512d122740 1 -- 172.21.15.31:0/3606663621 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56245e92a0c0 con 0x56245ea38790 2024-03-21T10:49:44.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.594+0000 7f5122a00640 1 --2- 172.21.15.31:0/3606663621 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56245ea38790 0x56245ea3abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:44.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.595+0000 7f5122a00640 1 --2- 172.21.15.31:0/3606663621 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56245ea38790 0x56245ea3abb0 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8788444e685a40c2 server_cookie=c3ea446402ba4a5b in_seq=0 out_seq=0 2024-03-21T10:49:44.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.595+0000 7f510d600640 10 client.28412 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:44.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.598+0000 7f510d600640 1 -- 172.21.15.31:0/3606663621 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56245e92a0c0 con 0x56245ea38790 2024-03-21T10:49:44.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.598+0000 7f510d600640 10 client.28412 handle_command_reply: tid=0 2024-03-21T10:49:44.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.598+0000 7f512d122740 1 client.28412 shutdown 2024-03-21T10:49:44.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.598+0000 7f512d122740 1 -- 172.21.15.31:0/3606663621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56245e9bd990 msgr2=0x56245e9d3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:44.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.598+0000 7f512d122740 1 --2- 172.21.15.31:0/3606663621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56245e9bd990 0x56245e9d3600 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f511409a240 tx=0x7f5114097f10 comp rx=0 tx=0).stop 2024-03-21T10:49:44.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.599+0000 7f512d122740 1 -- 172.21.15.31:0/3606663621 shutdown_connections 2024-03-21T10:49:44.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.599+0000 7f512d122740 1 --2- 172.21.15.31:0/3606663621 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56245ea38790 0x56245ea3abb0 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.599+0000 7f512d122740 1 --2- 172.21.15.31:0/3606663621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56245e9cbad0 0x56245e9cfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.599+0000 7f512d122740 1 --2- 172.21.15.31:0/3606663621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56245e9be2b0 0x56245e9cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.599+0000 7f512d122740 1 --2- 172.21.15.31:0/3606663621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56245e9bd990 0x56245e9d3600 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.599+0000 7f512d122740 1 -- 172.21.15.31:0/3606663621 >> 172.21.15.31:0/3606663621 conn(0x56245e9b88f0 msgr2=0x56245e9b7950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:44.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.599+0000 7f510d600640 0 client.28412 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:44.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.599+0000 7f512d122740 1 -- 172.21.15.31:0/3606663621 shutdown_connections 2024-03-21T10:49:44.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.599+0000 7f512d122740 1 -- 172.21.15.31:0/3606663621 wait complete. 2024-03-21T10:49:44.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.599+0000 7f512d122740 20 client.28412 trim_cache size 0 max 16384 2024-03-21T10:49:44.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:49:44.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:49:44.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:49:44.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:49:44.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.600+0000 7f512a800640 1 -- 172.21.15.31:0/3857747755 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f51080a2f90 msgr2=0x7f51080a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:44.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.600+0000 7f512a800640 1 --2- 172.21.15.31:0/3857747755 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f51080a2f90 0x7f51080a5450 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f511c010de0 tx=0x7f511c011f90 comp rx=0 tx=0).stop 2024-03-21T10:49:44.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.600+0000 7f512a800640 1 -- 172.21.15.31:0/3857747755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5124068f40 msgr2=0x7f5124197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:44.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.600+0000 7f512a800640 1 --2- 172.21.15.31:0/3857747755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5124068f40 0x7f5124197f20 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f511400e9f0 tx=0x7f511400eeb0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.600+0000 7f512a800640 1 -- 172.21.15.31:0/3857747755 shutdown_connections 2024-03-21T10:49:44.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.600+0000 7f512a800640 1 --2- 172.21.15.31:0/3857747755 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f51080a2f90 0x7f51080a5450 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.600+0000 7f512a800640 1 --2- 172.21.15.31:0/3857747755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5124107650 0x7f512419c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.600+0000 7f512a800640 1 --2- 172.21.15.31:0/3857747755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5124069900 0x7f5124198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.600+0000 7f512a800640 1 --2- 172.21.15.31:0/3857747755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5124068f40 0x7f5124197f20 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:44.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.600+0000 7f512a800640 1 -- 172.21.15.31:0/3857747755 >> 172.21.15.31:0/3857747755 conn(0x7f5124077690 msgr2=0x7f51240fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:44.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.600+0000 7f512a800640 1 -- 172.21.15.31:0/3857747755 shutdown_connections 2024-03-21T10:49:44.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:44.600+0000 7f512a800640 1 -- 172.21.15.31:0/3857747755 wait complete. 2024-03-21T10:49:45.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:45 smithi031 ceph-mon[31673]: pgmap v348: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:49:45.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:45 smithi070 ceph-mon[36330]: pgmap v348: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:49:45.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:45 smithi040 ceph-mon[36154]: pgmap v348: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:49:45.805 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:49:45.805 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:49:46.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.040+0000 7f4d52000640 1 -- 172.21.15.31:0/3173688741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d4c102730 msgr2=0x7f4d4c102b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:46.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.040+0000 7f4d52000640 1 --2- 172.21.15.31:0/3173688741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d4c102730 0x7f4d4c102b10 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f4d40009900 tx=0x7f4d4002bcc0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.040+0000 7f4d52000640 1 -- 172.21.15.31:0/3173688741 shutdown_connections 2024-03-21T10:49:46.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.040+0000 7f4d52000640 1 --2- 172.21.15.31:0/3173688741 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d4c10ba70 0x7f4d4c10dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.040+0000 7f4d52000640 1 --2- 172.21.15.31:0/3173688741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d4c103050 0x7f4d4c10b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.040+0000 7f4d52000640 1 --2- 172.21.15.31:0/3173688741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d4c102730 0x7f4d4c102b10 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.040+0000 7f4d52000640 1 -- 172.21.15.31:0/3173688741 >> 172.21.15.31:0/3173688741 conn(0x7f4d4c077130 msgr2=0x7f4d4c0fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:46.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.040+0000 7f4d52000640 1 -- 172.21.15.31:0/3173688741 shutdown_connections 2024-03-21T10:49:46.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.040+0000 7f4d52000640 1 -- 172.21.15.31:0/3173688741 wait complete. 2024-03-21T10:49:46.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.041+0000 7f4d52000640 1 Processor -- start 2024-03-21T10:49:46.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.041+0000 7f4d52000640 1 -- start start 2024-03-21T10:49:46.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.042+0000 7f4d52000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d4c102730 0x7f4d4c199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.042+0000 7f4d52000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d4c103050 0x7f4d4c19a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.042+0000 7f4d52000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d4c10ba70 0x7f4d4c19e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.042+0000 7f4d52000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c111870 con 0x7f4d4c10ba70 2024-03-21T10:49:46.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.042+0000 7f4d52000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c1116f0 con 0x7f4d4c103050 2024-03-21T10:49:46.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.042+0000 7f4d52000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c1119f0 con 0x7f4d4c102730 2024-03-21T10:49:46.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.042+0000 7f4d50c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d4c102730 0x7f4d4c199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.042+0000 7f4d50c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d4c102730 0x7f4d4c199ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:44298/0 (socket says 172.21.15.31:44298) 2024-03-21T10:49:46.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.042+0000 7f4d50c00640 1 -- 172.21.15.31:0/908897532 learned_addr learned my addr 172.21.15.31:0/908897532 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:46.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.042+0000 7f4d4be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d4c103050 0x7f4d4c19a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.042+0000 7f4d51600640 1 --2- 172.21.15.31:0/908897532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d4c10ba70 0x7f4d4c19e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.042+0000 7f4d50c00640 1 -- 172.21.15.31:0/908897532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d4c103050 msgr2=0x7f4d4c19a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:46.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.042+0000 7f4d50c00640 1 --2- 172.21.15.31:0/908897532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d4c103050 0x7f4d4c19a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.042+0000 7f4d50c00640 1 -- 172.21.15.31:0/908897532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d4c10ba70 msgr2=0x7f4d4c19e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:46.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.042+0000 7f4d50c00640 1 --2- 172.21.15.31:0/908897532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d4c10ba70 0x7f4d4c19e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.042+0000 7f4d50c00640 1 -- 172.21.15.31:0/908897532 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4c19eec0 con 0x7f4d4c102730 2024-03-21T10:49:46.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.043+0000 7f4d50c00640 1 --2- 172.21.15.31:0/908897532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d4c102730 0x7f4d4c199ed0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f4d40000c00 tx=0x7f4d400043c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:46.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.043+0000 7f4d49600640 1 -- 172.21.15.31:0/908897532 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d40042020 con 0x7f4d4c102730 2024-03-21T10:49:46.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.043+0000 7f4d49600640 1 -- 172.21.15.31:0/908897532 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d40005000 con 0x7f4d4c102730 2024-03-21T10:49:46.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.043+0000 7f4d49600640 1 -- 172.21.15.31:0/908897532 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d40003a90 con 0x7f4d4c102730 2024-03-21T10:49:46.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.043+0000 7f4d52000640 1 -- 172.21.15.31:0/908897532 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d4c19f150 con 0x7f4d4c102730 2024-03-21T10:49:46.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.043+0000 7f4d52000640 1 -- 172.21.15.31:0/908897532 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d4c1a6a40 con 0x7f4d4c102730 2024-03-21T10:49:46.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.046+0000 7f4d49600640 1 -- 172.21.15.31:0/908897532 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f4d40045020 con 0x7f4d4c102730 2024-03-21T10:49:46.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.046+0000 7f4d56419740 1 Processor -- start 2024-03-21T10:49:46.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.046+0000 7f4d56419740 1 -- start start 2024-03-21T10:49:46.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.047+0000 7f4d56419740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a85ce67dd0 0x55a85ce66b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.047+0000 7f4d56419740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85ce670f0 0x55a85ce67570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.047+0000 7f4d56419740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce70460 0x55a85ce72850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.047+0000 7f4d56419740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a85cdcb250 con 0x55a85ce67dd0 2024-03-21T10:49:46.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.047+0000 7f4d56419740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a85cd73a30 con 0x55a85ce670f0 2024-03-21T10:49:46.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.047+0000 7f4d56419740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a85cdc72f0 con 0x55a85ce70460 2024-03-21T10:49:46.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.047+0000 7f4d50c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85ce670f0 0x55a85ce67570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.047+0000 7f4d51600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a85ce67dd0 0x55a85ce66b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.047+0000 7f4d50c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85ce670f0 0x55a85ce67570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44170/0 (socket says 172.21.15.31:44170) 2024-03-21T10:49:46.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.047+0000 7f4d4be00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce70460 0x55a85ce72850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.047+0000 7f4d50c00640 1 -- 172.21.15.31:0/2372032498 learned_addr learned my addr 172.21.15.31:0/2372032498 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:46.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.047+0000 7f4d4be00640 1 -- 172.21.15.31:0/2372032498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85ce670f0 msgr2=0x55a85ce67570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:46.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.048+0000 7f4d4be00640 1 --2- 172.21.15.31:0/2372032498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85ce670f0 0x55a85ce67570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.048+0000 7f4d4be00640 1 -- 172.21.15.31:0/2372032498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a85ce67dd0 msgr2=0x55a85ce66b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:46.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.048+0000 7f4d4be00640 1 --2- 172.21.15.31:0/2372032498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a85ce67dd0 0x55a85ce66b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.048+0000 7f4d4be00640 1 -- 172.21.15.31:0/2372032498 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a85cd5e690 con 0x55a85ce70460 2024-03-21T10:49:46.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.048+0000 7f4d4be00640 1 --2- 172.21.15.31:0/2372032498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce70460 0x55a85ce72850 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f4d3800abc0 tx=0x7f4d3800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffd3dd1fdd6923cd server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:46.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.048+0000 7f4d49600640 1 --2- 172.21.15.31:0/908897532 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4d340a3090 0x7f4d340a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.048+0000 7f4d3dc00640 1 -- 172.21.15.31:0/2372032498 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d38005220 con 0x55a85ce70460 2024-03-21T10:49:46.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.048+0000 7f4d3dc00640 1 -- 172.21.15.31:0/2372032498 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d38004c80 con 0x55a85ce70460 2024-03-21T10:49:46.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.048+0000 7f4d3dc00640 1 -- 172.21.15.31:0/2372032498 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d38018900 con 0x55a85ce70460 2024-03-21T10:49:46.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.048+0000 7f4d49600640 1 -- 172.21.15.31:0/908897532 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f4d400f17b0 con 0x7f4d4c102730 2024-03-21T10:49:46.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.048+0000 7f4d51600640 1 --2- 172.21.15.31:0/908897532 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4d340a3090 0x7f4d340a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.049+0000 7f4d56419740 1 -- 172.21.15.31:0/2372032498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce70460 msgr2=0x55a85ce72850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:46.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.049+0000 7f4d56419740 1 --2- 172.21.15.31:0/2372032498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce70460 0x55a85ce72850 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f4d3800abc0 tx=0x7f4d3800f790 comp rx=0 tx=0).stop 2024-03-21T10:49:46.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.049+0000 7f4d56419740 1 -- 172.21.15.31:0/2372032498 shutdown_connections 2024-03-21T10:49:46.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.049+0000 7f4d56419740 1 --2- 172.21.15.31:0/2372032498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce70460 0x55a85ce72850 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.049+0000 7f4d56419740 1 --2- 172.21.15.31:0/2372032498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85ce670f0 0x55a85ce67570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.049+0000 7f4d56419740 1 --2- 172.21.15.31:0/2372032498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a85ce67dd0 0x55a85ce66b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.049+0000 7f4d56419740 1 -- 172.21.15.31:0/2372032498 >> 172.21.15.31:0/2372032498 conn(0x55a85ce2e200 msgr2=0x55a85ce61b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:46.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.049+0000 7f4d51600640 1 --2- 172.21.15.31:0/908897532 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4d340a3090 0x7f4d340a5550 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4d44005fd0 tx=0x7f4d44009500 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:46.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.049+0000 7f4d56419740 1 -- 172.21.15.31:0/2372032498 shutdown_connections 2024-03-21T10:49:46.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.049+0000 7f4d56419740 1 -- 172.21.15.31:0/2372032498 wait complete. 2024-03-21T10:49:46.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.050+0000 7f4d56419740 1 -- start start 2024-03-21T10:49:46.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.050+0000 7f4d56419740 1 Processor -- start 2024-03-21T10:49:46.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.050+0000 7f4d56419740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce670f0 0x55a85ce6faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.050+0000 7f4d56419740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85ce67dd0 0x55a85ce6ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.050+0000 7f4d56419740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a85ce70460 0x55a85ce6d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.050+0000 7f4d56419740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a85ce732d0 con 0x55a85ce70460 2024-03-21T10:49:46.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.050+0000 7f4d56419740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a85cd86e80 con 0x55a85ce67dd0 2024-03-21T10:49:46.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.050+0000 7f4d56419740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a85ce73450 con 0x55a85ce670f0 2024-03-21T10:49:46.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.051+0000 7f4d56419740 10 client.0 _ll_register_callbacks cb 0x55a85cd99130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:46.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.051+0000 7f4d50c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce670f0 0x55a85ce6faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.051+0000 7f4d50c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce670f0 0x55a85ce6faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:44314/0 (socket says 172.21.15.31:44314) 2024-03-21T10:49:46.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.051+0000 7f4d50c00640 1 -- 172.21.15.31:0/804764305 learned_addr learned my addr 172.21.15.31:0/804764305 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:46.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.051+0000 7f4d4be00640 1 --2- 172.21.15.31:0/804764305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85ce67dd0 0x55a85ce6ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.051+0000 7f4d51600640 1 --2- 172.21.15.31:0/804764305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a85ce70460 0x55a85ce6d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.051+0000 7f4d50c00640 1 -- 172.21.15.31:0/804764305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85ce67dd0 msgr2=0x55a85ce6ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:46.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.051+0000 7f4d50c00640 1 --2- 172.21.15.31:0/804764305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85ce67dd0 0x55a85ce6ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.051+0000 7f4d50c00640 1 -- 172.21.15.31:0/804764305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a85ce70460 msgr2=0x55a85ce6d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:46.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.051+0000 7f4d50c00640 1 --2- 172.21.15.31:0/804764305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a85ce70460 0x55a85ce6d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.051+0000 7f4d50c00640 1 -- 172.21.15.31:0/804764305 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a85cba97d0 con 0x55a85ce670f0 2024-03-21T10:49:46.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.051+0000 7f4d50c00640 1 --2- 172.21.15.31:0/804764305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce670f0 0x55a85ce6faf0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f4d40005ab0 tx=0x7f4d40005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfa77e525c123140 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:46.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.051+0000 7f4d32000640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:49:46.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.052+0000 7f4d32000640 1 -- 172.21.15.31:0/804764305 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d400bd3f0 con 0x55a85ce670f0 2024-03-21T10:49:46.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.052+0000 7f4d32000640 1 -- 172.21.15.31:0/804764305 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f4d400bbbb0 con 0x55a85ce670f0 2024-03-21T10:49:46.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.052+0000 7f4d56419740 1 -- 172.21.15.31:0/804764305 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4d38004c80 con 0x55a85ce670f0 2024-03-21T10:49:46.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.052+0000 7f4d32000640 1 -- 172.21.15.31:0/804764305 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d400bd210 con 0x55a85ce670f0 2024-03-21T10:49:46.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.052+0000 7f4d56419740 10 client.28351 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:46.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.052+0000 7f4d56419740 1 -- 172.21.15.31:0/804764305 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55a85cadf3f0 con 0x55a85ce670f0 2024-03-21T10:49:46.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.052+0000 7f4d32000640 1 -- 172.21.15.31:0/804764305 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d38004c80 con 0x55a85ce670f0 2024-03-21T10:49:46.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.052+0000 7f4d32000640 1 -- 172.21.15.31:0/804764305 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f4d400f1070 con 0x55a85ce670f0 2024-03-21T10:49:46.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.052+0000 7f4d56419740 10 client.28351 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:46.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.052+0000 7f4d56419740 10 client.28351 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:49:46.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.052+0000 7f4d56419740 20 client.28351 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:46.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.052+0000 7f4d56419740 1 --2- 172.21.15.31:0/804764305 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a85ce80280 0x55a85ce826d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.052+0000 7f4d56419740 4 client.28351 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:49:46.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.052+0000 7f4d56419740 1 -- 172.21.15.31:0/804764305 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a85cdd8790 con 0x55a85ce80280 2024-03-21T10:49:46.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.053+0000 7f4d4be00640 1 --2- 172.21.15.31:0/804764305 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a85ce80280 0x55a85ce826d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.053+0000 7f4d4be00640 1 --2- 172.21.15.31:0/804764305 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a85ce80280 0x55a85ce826d0 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da4826fd04d22c86 server_cookie=92a25dc8f8ef590e in_seq=0 out_seq=0 2024-03-21T10:49:46.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.053+0000 7f4d32000640 10 client.28351 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:46.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.054+0000 7f4d32000640 1 -- 172.21.15.31:0/804764305 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a85cdd8790 con 0x55a85ce80280 2024-03-21T10:49:46.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.054+0000 7f4d32000640 10 client.28351 handle_command_reply: tid=0 2024-03-21T10:49:46.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.054+0000 7f4d56419740 1 client.28351 shutdown 2024-03-21T10:49:46.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.054+0000 7f4d56419740 1 -- 172.21.15.31:0/804764305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce670f0 msgr2=0x55a85ce6faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:46.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.054+0000 7f4d56419740 1 --2- 172.21.15.31:0/804764305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce670f0 0x55a85ce6faf0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f4d40005ab0 tx=0x7f4d40005b50 comp rx=0 tx=0).stop 2024-03-21T10:49:46.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.054+0000 7f4d56419740 1 -- 172.21.15.31:0/804764305 shutdown_connections 2024-03-21T10:49:46.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.054+0000 7f4d56419740 1 --2- 172.21.15.31:0/804764305 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a85ce80280 0x55a85ce826d0 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.054+0000 7f4d56419740 1 --2- 172.21.15.31:0/804764305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a85ce70460 0x55a85ce6d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.054+0000 7f4d56419740 1 --2- 172.21.15.31:0/804764305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85ce67dd0 0x55a85ce6ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.054+0000 7f4d32000640 0 client.28351 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:46.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.054+0000 7f4d56419740 1 --2- 172.21.15.31:0/804764305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce670f0 0x55a85ce6faf0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.054+0000 7f4d56419740 1 -- 172.21.15.31:0/804764305 >> 172.21.15.31:0/804764305 conn(0x55a85ce2e200 msgr2=0x55a85ce63890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:46.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.054+0000 7f4d56419740 1 -- 172.21.15.31:0/804764305 shutdown_connections 2024-03-21T10:49:46.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.054+0000 7f4d56419740 1 -- 172.21.15.31:0/804764305 wait complete. 2024-03-21T10:49:46.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.055+0000 7f4d56419740 20 client.28351 trim_cache size 0 max 16384 2024-03-21T10:49:46.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.085+0000 7f4d56419740 1 Processor -- start 2024-03-21T10:49:46.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.085+0000 7f4d56419740 1 -- start start 2024-03-21T10:49:46.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.085+0000 7f4d56419740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce670f0 0x55a85ce6f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.085+0000 7f4d56419740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85ce67dd0 0x55a85ce6fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.085+0000 7f4d56419740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a85ce70460 0x55a85ce71850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.085+0000 7f4d56419740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c1b2e50 con 0x55a85ce70460 2024-03-21T10:49:46.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.085+0000 7f4d56419740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c1b0950 con 0x55a85ce67dd0 2024-03-21T10:49:46.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.085+0000 7f4d56419740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c1aea10 con 0x55a85ce670f0 2024-03-21T10:49:46.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.086+0000 7f4d50c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce670f0 0x55a85ce6f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.086+0000 7f4d50c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce670f0 0x55a85ce6f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:44316/0 (socket says 172.21.15.31:44316) 2024-03-21T10:49:46.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.086+0000 7f4d50c00640 1 -- 172.21.15.31:0/842055142 learned_addr learned my addr 172.21.15.31:0/842055142 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:46.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.086+0000 7f4d4be00640 1 --2- 172.21.15.31:0/842055142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85ce67dd0 0x55a85ce6fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.086+0000 7f4d51600640 1 --2- 172.21.15.31:0/842055142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a85ce70460 0x55a85ce71850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.086+0000 7f4d50c00640 1 -- 172.21.15.31:0/842055142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85ce67dd0 msgr2=0x55a85ce6fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:46.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.086+0000 7f4d50c00640 1 --2- 172.21.15.31:0/842055142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85ce67dd0 0x55a85ce6fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.086+0000 7f4d50c00640 1 -- 172.21.15.31:0/842055142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a85ce70460 msgr2=0x55a85ce71850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:46.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.086+0000 7f4d50c00640 1 --2- 172.21.15.31:0/842055142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a85ce70460 0x55a85ce71850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.086+0000 7f4d50c00640 1 -- 172.21.15.31:0/842055142 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a85ca9f0f0 con 0x55a85ce670f0 2024-03-21T10:49:46.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.086+0000 7f4d51600640 1 --2- 172.21.15.31:0/842055142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a85ce70460 0x55a85ce71850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:49:46.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.086+0000 7f4d50c00640 1 --2- 172.21.15.31:0/842055142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce670f0 0x55a85ce6f540 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f4d400ba280 tx=0x7f4d4002bcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d576d51983691152 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:46.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.087+0000 7f4d3d200640 1 -- 172.21.15.31:0/842055142 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d400bd3f0 con 0x55a85ce670f0 2024-03-21T10:49:46.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.087+0000 7f4d3d200640 1 -- 172.21.15.31:0/842055142 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d400d5030 con 0x55a85ce670f0 2024-03-21T10:49:46.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.087+0000 7f4d3d200640 1 -- 172.21.15.31:0/842055142 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d400bbbb0 con 0x55a85ce670f0 2024-03-21T10:49:46.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.087+0000 7f4d56419740 1 -- 172.21.15.31:0/842055142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce670f0 msgr2=0x55a85ce6f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:46.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.087+0000 7f4d56419740 1 --2- 172.21.15.31:0/842055142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce670f0 0x55a85ce6f540 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f4d400ba280 tx=0x7f4d4002bcc0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.087+0000 7f4d56419740 1 -- 172.21.15.31:0/842055142 shutdown_connections 2024-03-21T10:49:46.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.087+0000 7f4d56419740 1 --2- 172.21.15.31:0/842055142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a85ce70460 0x55a85ce71850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.087+0000 7f4d56419740 1 --2- 172.21.15.31:0/842055142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85ce67dd0 0x55a85ce6fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.087+0000 7f4d56419740 1 --2- 172.21.15.31:0/842055142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce670f0 0x55a85ce6f540 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.087+0000 7f4d56419740 1 -- 172.21.15.31:0/842055142 >> 172.21.15.31:0/842055142 conn(0x55a85ce2e200 msgr2=0x55a85ce61670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:46.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.088+0000 7f4d56419740 1 -- 172.21.15.31:0/842055142 shutdown_connections 2024-03-21T10:49:46.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.088+0000 7f4d56419740 1 -- 172.21.15.31:0/842055142 wait complete. 2024-03-21T10:49:46.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.088+0000 7f4d56419740 1 -- start start 2024-03-21T10:49:46.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.088+0000 7f4d56419740 1 Processor -- start 2024-03-21T10:49:46.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.089+0000 7f4d56419740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce670f0 0x55a85ce6ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.089+0000 7f4d56419740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85ce67dd0 0x55a85ce694c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.089+0000 7f4d56419740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a85ce70460 0x55a85ce69bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.089+0000 7f4d56419740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c1af010 con 0x55a85ce70460 2024-03-21T10:49:46.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.089+0000 7f4d56419740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c1b02f0 con 0x55a85ce67dd0 2024-03-21T10:49:46.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.089+0000 7f4d56419740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c1af630 con 0x55a85ce670f0 2024-03-21T10:49:46.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.089+0000 7f4d56419740 10 client.0 _ll_register_callbacks cb 0x7f4d40005770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:46.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.089+0000 7f4d50c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce670f0 0x55a85ce6ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.089+0000 7f4d4be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85ce67dd0 0x55a85ce694c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.089+0000 7f4d50c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce670f0 0x55a85ce6ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:44318/0 (socket says 172.21.15.31:44318) 2024-03-21T10:49:46.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.089+0000 7f4d50c00640 1 -- 172.21.15.31:0/737150660 learned_addr learned my addr 172.21.15.31:0/737150660 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:46.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.089+0000 7f4d51600640 1 --2- 172.21.15.31:0/737150660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a85ce70460 0x55a85ce69bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.090+0000 7f4d50c00640 1 -- 172.21.15.31:0/737150660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85ce67dd0 msgr2=0x55a85ce694c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:46.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.090+0000 7f4d50c00640 1 --2- 172.21.15.31:0/737150660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85ce67dd0 0x55a85ce694c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.090+0000 7f4d50c00640 1 -- 172.21.15.31:0/737150660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a85ce70460 msgr2=0x55a85ce69bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:46.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.090+0000 7f4d50c00640 1 --2- 172.21.15.31:0/737150660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a85ce70460 0x55a85ce69bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.090+0000 7f4d50c00640 1 -- 172.21.15.31:0/737150660 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a85cedab60 con 0x55a85ce670f0 2024-03-21T10:49:46.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.090+0000 7f4d4be00640 1 --2- 172.21.15.31:0/737150660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85ce67dd0 0x55a85ce694c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:46.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.090+0000 7f4d50c00640 1 --2- 172.21.15.31:0/737150660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce670f0 0x55a85ce6ab90 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f4d400c5d90 tx=0x7f4d40005e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9c6e5d97b208fb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:46.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.090+0000 7f4d32000640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:49:46.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.090+0000 7f4d32000640 1 -- 172.21.15.31:0/737150660 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d400da020 con 0x55a85ce670f0 2024-03-21T10:49:46.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.090+0000 7f4d32000640 1 -- 172.21.15.31:0/737150660 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f4d400a80e0 con 0x55a85ce670f0 2024-03-21T10:49:46.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.090+0000 7f4d56419740 1 -- 172.21.15.31:0/737150660 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4d400d5030 con 0x55a85ce670f0 2024-03-21T10:49:46.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.090+0000 7f4d32000640 1 -- 172.21.15.31:0/737150660 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d400bd3f0 con 0x55a85ce670f0 2024-03-21T10:49:46.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.091+0000 7f4d56419740 10 client.28357 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:46.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.091+0000 7f4d56419740 1 -- 172.21.15.31:0/737150660 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55a85cedab60 con 0x55a85ce670f0 2024-03-21T10:49:46.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.091+0000 7f4d32000640 1 -- 172.21.15.31:0/737150660 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d400d5030 con 0x55a85ce670f0 2024-03-21T10:49:46.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.091+0000 7f4d32000640 1 -- 172.21.15.31:0/737150660 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f4d400f1040 con 0x55a85ce670f0 2024-03-21T10:49:46.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.091+0000 7f4d56419740 10 client.28357 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:46.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.091+0000 7f4d56419740 10 client.28357 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:49:46.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.091+0000 7f4d56419740 20 client.28357 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:46.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.091+0000 7f4d56419740 1 --2- 172.21.15.31:0/737150660 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a85cedbf40 0x55a85cede360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.091+0000 7f4d56419740 4 client.28357 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:49:46.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.091+0000 7f4d56419740 1 -- 172.21.15.31:0/737150660 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a85cdd2ae0 con 0x55a85cedbf40 2024-03-21T10:49:46.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.091+0000 7f4d4be00640 1 --2- 172.21.15.31:0/737150660 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a85cedbf40 0x55a85cede360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.092+0000 7f4d4be00640 1 --2- 172.21.15.31:0/737150660 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a85cedbf40 0x55a85cede360 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2738c34e625a0a13 server_cookie=4ea298d4ef3683d4 in_seq=0 out_seq=0 2024-03-21T10:49:46.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.092+0000 7f4d32000640 10 client.28357 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:46.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.092+0000 7f4d32000640 1 -- 172.21.15.31:0/737150660 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55a85cdd2ae0 con 0x55a85cedbf40 2024-03-21T10:49:46.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.092+0000 7f4d32000640 10 client.28357 handle_command_reply: tid=0 2024-03-21T10:49:46.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.092+0000 7f4d56419740 1 client.28357 shutdown 2024-03-21T10:49:46.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.092+0000 7f4d56419740 1 -- 172.21.15.31:0/737150660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce670f0 msgr2=0x55a85ce6ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:46.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.092+0000 7f4d56419740 1 --2- 172.21.15.31:0/737150660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce670f0 0x55a85ce6ab90 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f4d400c5d90 tx=0x7f4d40005e60 comp rx=0 tx=0).stop 2024-03-21T10:49:46.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.092+0000 7f4d56419740 1 -- 172.21.15.31:0/737150660 shutdown_connections 2024-03-21T10:49:46.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.092+0000 7f4d56419740 1 --2- 172.21.15.31:0/737150660 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55a85cedbf40 0x55a85cede360 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.092+0000 7f4d56419740 1 --2- 172.21.15.31:0/737150660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a85ce70460 0x55a85ce69bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.092+0000 7f4d56419740 1 --2- 172.21.15.31:0/737150660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a85ce67dd0 0x55a85ce694c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.092+0000 7f4d56419740 1 --2- 172.21.15.31:0/737150660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a85ce670f0 0x55a85ce6ab90 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.092+0000 7f4d32000640 0 client.28357 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:46.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.092+0000 7f4d56419740 1 -- 172.21.15.31:0/737150660 >> 172.21.15.31:0/737150660 conn(0x55a85ce2e200 msgr2=0x55a85ce6b6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:46.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.092+0000 7f4d56419740 1 -- 172.21.15.31:0/737150660 shutdown_connections 2024-03-21T10:49:46.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.092+0000 7f4d56419740 1 -- 172.21.15.31:0/737150660 wait complete. 2024-03-21T10:49:46.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.093+0000 7f4d56419740 20 client.28357 trim_cache size 0 max 16384 2024-03-21T10:49:46.093 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:49:46.093 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-03-21T10:49:46.093 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-03-21T10:49:46.093 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:49:46.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.093+0000 7f4d52000640 1 -- 172.21.15.31:0/908897532 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4d340a3090 msgr2=0x7f4d340a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:46.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.093+0000 7f4d52000640 1 --2- 172.21.15.31:0/908897532 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4d340a3090 0x7f4d340a5550 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4d44005fd0 tx=0x7f4d44009500 comp rx=0 tx=0).stop 2024-03-21T10:49:46.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.093+0000 7f4d52000640 1 -- 172.21.15.31:0/908897532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d4c102730 msgr2=0x7f4d4c199ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:46.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.093+0000 7f4d52000640 1 --2- 172.21.15.31:0/908897532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d4c102730 0x7f4d4c199ed0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f4d40000c00 tx=0x7f4d400043c0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.093+0000 7f4d52000640 1 -- 172.21.15.31:0/908897532 shutdown_connections 2024-03-21T10:49:46.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.093+0000 7f4d52000640 1 --2- 172.21.15.31:0/908897532 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4d340a3090 0x7f4d340a5550 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.093+0000 7f4d52000640 1 --2- 172.21.15.31:0/908897532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d4c10ba70 0x7f4d4c19e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.093+0000 7f4d52000640 1 --2- 172.21.15.31:0/908897532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d4c103050 0x7f4d4c19a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.093+0000 7f4d52000640 1 --2- 172.21.15.31:0/908897532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d4c102730 0x7f4d4c199ed0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.093+0000 7f4d52000640 1 -- 172.21.15.31:0/908897532 >> 172.21.15.31:0/908897532 conn(0x7f4d4c077130 msgr2=0x7f4d4c0fc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:46.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.094+0000 7f4d52000640 1 -- 172.21.15.31:0/908897532 shutdown_connections 2024-03-21T10:49:46.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.094+0000 7f4d52000640 1 -- 172.21.15.31:0/908897532 wait complete. 2024-03-21T10:49:46.109 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-21T10:49:46.109 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:49:46.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.333+0000 7fe6b8000640 1 -- 172.21.15.31:0/3861561682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6b0109850 msgr2=0x7fe6b0109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:46.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.333+0000 7fe6b8000640 1 --2- 172.21.15.31:0/3861561682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6b0109850 0x7fe6b0109c30 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4009930 tx=0x7fe6a402bc30 comp rx=0 tx=0).stop 2024-03-21T10:49:46.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.333+0000 7fe6b8000640 1 -- 172.21.15.31:0/3861561682 shutdown_connections 2024-03-21T10:49:46.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.333+0000 7fe6b8000640 1 --2- 172.21.15.31:0/3861561682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6b0104200 0x7fe6b0110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.333+0000 7fe6b8000640 1 --2- 172.21.15.31:0/3861561682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6b0103840 0x7fe6b0103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.333+0000 7fe6b8000640 1 --2- 172.21.15.31:0/3861561682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6b0109850 0x7fe6b0109c30 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.333+0000 7fe6b8000640 1 -- 172.21.15.31:0/3861561682 >> 172.21.15.31:0/3861561682 conn(0x7fe6b00ff350 msgr2=0x7fe6b0101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:46.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.334+0000 7fe6b8000640 1 -- 172.21.15.31:0/3861561682 shutdown_connections 2024-03-21T10:49:46.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.334+0000 7fe6b8000640 1 -- 172.21.15.31:0/3861561682 wait complete. 2024-03-21T10:49:46.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.335+0000 7fe6b8000640 1 Processor -- start 2024-03-21T10:49:46.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.335+0000 7fe6b8000640 1 -- start start 2024-03-21T10:49:46.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.335+0000 7fe6b8000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6b0103840 0x7fe6b019a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.335+0000 7fe6b8000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6b0104200 0x7fe6b019a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.335+0000 7fe6b8000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6b0109850 0x7fe6b019ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.335+0000 7fe6b8000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b0113c70 con 0x7fe6b0104200 2024-03-21T10:49:46.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.335+0000 7fe6b8000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b0113af0 con 0x7fe6b0103840 2024-03-21T10:49:46.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.335+0000 7fe6b8000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b0113df0 con 0x7fe6b0109850 2024-03-21T10:49:46.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.335+0000 7fe6b5600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6b0103840 0x7fe6b019a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.335+0000 7fe6b5600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6b0103840 0x7fe6b019a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44212/0 (socket says 172.21.15.31:44212) 2024-03-21T10:49:46.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.335+0000 7fe6b5600640 1 -- 172.21.15.31:0/1367492836 learned_addr learned my addr 172.21.15.31:0/1367492836 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:46.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.335+0000 7fe6b6000640 1 --2- 172.21.15.31:0/1367492836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6b0109850 0x7fe6b019ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.336+0000 7fe6b4c00640 1 --2- 172.21.15.31:0/1367492836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6b0104200 0x7fe6b019a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.336+0000 7fe6b5600640 1 -- 172.21.15.31:0/1367492836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6b0109850 msgr2=0x7fe6b019ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:46.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.336+0000 7fe6b5600640 1 --2- 172.21.15.31:0/1367492836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6b0109850 0x7fe6b019ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.336+0000 7fe6b5600640 1 -- 172.21.15.31:0/1367492836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6b0104200 msgr2=0x7fe6b019a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:46.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.336+0000 7fe6b5600640 1 --2- 172.21.15.31:0/1367492836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6b0104200 0x7fe6b019a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.336+0000 7fe6b5600640 1 -- 172.21.15.31:0/1367492836 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b019f140 con 0x7fe6b0103840 2024-03-21T10:49:46.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.336+0000 7fe6b4c00640 1 --2- 172.21.15.31:0/1367492836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6b0104200 0x7fe6b019a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:49:46.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.336+0000 7fe6b5600640 1 --2- 172.21.15.31:0/1367492836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6b0103840 0x7fe6b019a150 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4009900 tx=0x7fe6a4004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:46.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.336+0000 7fe6ae000640 1 -- 172.21.15.31:0/1367492836 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a4043020 con 0x7fe6b0103840 2024-03-21T10:49:46.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.336+0000 7fe6ae000640 1 -- 172.21.15.31:0/1367492836 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6a40054e0 con 0x7fe6b0103840 2024-03-21T10:49:46.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.336+0000 7fe6b8000640 1 -- 172.21.15.31:0/1367492836 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b019f3d0 con 0x7fe6b0103840 2024-03-21T10:49:46.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.337+0000 7fe6ae000640 1 -- 172.21.15.31:0/1367492836 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a4003bb0 con 0x7fe6b0103840 2024-03-21T10:49:46.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.337+0000 7fe6b8000640 1 -- 172.21.15.31:0/1367492836 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b01a6cc0 con 0x7fe6b0103840 2024-03-21T10:49:46.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.338+0000 7fe6b8000640 1 -- 172.21.15.31:0/1367492836 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b0068970 con 0x7fe6b0103840 2024-03-21T10:49:46.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.340+0000 7fe6ae000640 1 -- 172.21.15.31:0/1367492836 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fe6a4046020 con 0x7fe6b0103840 2024-03-21T10:49:46.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.341+0000 7fe6ae000640 1 --2- 172.21.15.31:0/1367492836 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe6980a3110 0x7fe6980a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.341+0000 7fe6ae000640 1 -- 172.21.15.31:0/1367492836 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fe6a40f1b30 con 0x7fe6b0103840 2024-03-21T10:49:46.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.341+0000 7fe6b4c00640 1 --2- 172.21.15.31:0/1367492836 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe6980a3110 0x7fe6980a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.342+0000 7fe6b4c00640 1 --2- 172.21.15.31:0/1367492836 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe6980a3110 0x7fe6980a55d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe6b01019a0 tx=0x7fe6a0005f90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:46.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.345+0000 7fe6ae000640 1 -- 172.21.15.31:0/1367492836 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe6a404d020 con 0x7fe6b0103840 2024-03-21T10:49:46.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.600+0000 7fe6b8000640 1 -- 172.21.15.31:0/1367492836 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6b010bdb0 con 0x7fe6b0103840 2024-03-21T10:49:46.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.601+0000 7fe6ae000640 1 -- 172.21.15.31:0/1367492836 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+11384 (secure 0 0 0) 0x7fe6a40be330 con 0x7fe6b0103840 2024-03-21T10:49:46.598 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:49:46.598 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:49:06.188518+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24547,24596]},"id":1}]} 2024-03-21T10:49:46.598 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 30 2024-03-21T10:49:46.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.604+0000 7fe6b8000640 1 -- 172.21.15.31:0/1367492836 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe6980a3110 msgr2=0x7fe6980a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:46.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.604+0000 7fe6b8000640 1 --2- 172.21.15.31:0/1367492836 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe6980a3110 0x7fe6980a55d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe6b01019a0 tx=0x7fe6a0005f90 comp rx=0 tx=0).stop 2024-03-21T10:49:46.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.604+0000 7fe6b8000640 1 -- 172.21.15.31:0/1367492836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6b0103840 msgr2=0x7fe6b019a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:46.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.604+0000 7fe6b8000640 1 --2- 172.21.15.31:0/1367492836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6b0103840 0x7fe6b019a150 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4009900 tx=0x7fe6a4004360 comp rx=0 tx=0).stop 2024-03-21T10:49:46.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.604+0000 7fe6b8000640 1 -- 172.21.15.31:0/1367492836 shutdown_connections 2024-03-21T10:49:46.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.604+0000 7fe6b8000640 1 --2- 172.21.15.31:0/1367492836 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe6980a3110 0x7fe6980a55d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.604+0000 7fe6b8000640 1 --2- 172.21.15.31:0/1367492836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6b0109850 0x7fe6b019ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.604+0000 7fe6b8000640 1 --2- 172.21.15.31:0/1367492836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6b0104200 0x7fe6b019a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.604+0000 7fe6b8000640 1 --2- 172.21.15.31:0/1367492836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6b0103840 0x7fe6b019a150 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.604+0000 7fe6b8000640 1 -- 172.21.15.31:0/1367492836 >> 172.21.15.31:0/1367492836 conn(0x7fe6b00ff350 msgr2=0x7fe6b00ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:46.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.605+0000 7fe6b8000640 1 -- 172.21.15.31:0/1367492836 shutdown_connections 2024-03-21T10:49:46.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.605+0000 7fe6b8000640 1 -- 172.21.15.31:0/1367492836 wait complete. 2024-03-21T10:49:46.627 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:49:46.628 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-21T10:49:46.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.846+0000 7fd966c00640 1 -- 172.21.15.31:0/1405795023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd960101e60 msgr2=0x7fd96010e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:46.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.846+0000 7fd966c00640 1 --2- 172.21.15.31:0/1405795023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd960101e60 0x7fd96010e320 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fd958009c10 tx=0x7fd95802bef0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.846+0000 7fd966c00640 1 -- 172.21.15.31:0/1405795023 shutdown_connections 2024-03-21T10:49:46.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.846+0000 7fd966c00640 1 --2- 172.21.15.31:0/1405795023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd960101e60 0x7fd96010e320 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.846+0000 7fd966c00640 1 --2- 172.21.15.31:0/1405795023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9601014a0 0x7fd960101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.846+0000 7fd966c00640 1 --2- 172.21.15.31:0/1405795023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd960107560 0x7fd960107940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.846+0000 7fd966c00640 1 -- 172.21.15.31:0/1405795023 >> 172.21.15.31:0/1405795023 conn(0x7fd9600fcfb0 msgr2=0x7fd9600ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:46.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.846+0000 7fd966c00640 1 -- 172.21.15.31:0/1405795023 shutdown_connections 2024-03-21T10:49:46.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.846+0000 7fd966c00640 1 -- 172.21.15.31:0/1405795023 wait complete. 2024-03-21T10:49:46.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.848+0000 7fd966c00640 1 Processor -- start 2024-03-21T10:49:46.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.848+0000 7fd966c00640 1 -- start start 2024-03-21T10:49:46.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.848+0000 7fd966c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9601014a0 0x7fd960199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.848+0000 7fd966c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd960101e60 0x7fd96019a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.848+0000 7fd966c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd960107560 0x7fd96019e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.848+0000 7fd966c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd960111870 con 0x7fd9601014a0 2024-03-21T10:49:46.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.848+0000 7fd966c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9601116f0 con 0x7fd960107560 2024-03-21T10:49:46.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.848+0000 7fd966c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9601119f0 con 0x7fd960101e60 2024-03-21T10:49:46.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.849+0000 7fd965800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9601014a0 0x7fd960199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.849+0000 7fd964e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd960101e60 0x7fd96019a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.849+0000 7fd964e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd960101e60 0x7fd96019a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:44346/0 (socket says 172.21.15.31:44346) 2024-03-21T10:49:46.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.849+0000 7fd965800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9601014a0 0x7fd960199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46176/0 (socket says 172.21.15.31:46176) 2024-03-21T10:49:46.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.849+0000 7fd964e00640 1 -- 172.21.15.31:0/1681218791 learned_addr learned my addr 172.21.15.31:0/1681218791 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:46.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.849+0000 7fd966200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd960107560 0x7fd96019e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.849+0000 7fd964e00640 1 -- 172.21.15.31:0/1681218791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd960107560 msgr2=0x7fd96019e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:46.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.849+0000 7fd964e00640 1 --2- 172.21.15.31:0/1681218791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd960107560 0x7fd96019e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.849+0000 7fd964e00640 1 -- 172.21.15.31:0/1681218791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9601014a0 msgr2=0x7fd960199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:46.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.849+0000 7fd964e00640 1 --2- 172.21.15.31:0/1681218791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9601014a0 0x7fd960199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.849+0000 7fd964e00640 1 -- 172.21.15.31:0/1681218791 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96019ef50 con 0x7fd960101e60 2024-03-21T10:49:46.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.849+0000 7fd964e00640 1 --2- 172.21.15.31:0/1681218791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd960101e60 0x7fd96019a4a0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fd94c00e9f0 tx=0x7fd94c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:46.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.850+0000 7fd95e000640 1 -- 172.21.15.31:0/1681218791 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94c00cd80 con 0x7fd960101e60 2024-03-21T10:49:46.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.850+0000 7fd95e000640 1 -- 172.21.15.31:0/1681218791 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd94c010910 con 0x7fd960101e60 2024-03-21T10:49:46.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.850+0000 7fd966c00640 1 -- 172.21.15.31:0/1681218791 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd96019f240 con 0x7fd960101e60 2024-03-21T10:49:46.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.850+0000 7fd95e000640 1 -- 172.21.15.31:0/1681218791 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94c0188f0 con 0x7fd960101e60 2024-03-21T10:49:46.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.850+0000 7fd966c00640 1 -- 172.21.15.31:0/1681218791 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9601a6ad0 con 0x7fd960101e60 2024-03-21T10:49:46.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.854+0000 7fd96adf9740 1 Processor -- start 2024-03-21T10:49:46.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.854+0000 7fd96adf9740 1 -- start start 2024-03-21T10:49:46.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.855+0000 7fd96adf9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe326c88b0 0x55fe326c8c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.855+0000 7fd95e000640 1 -- 172.21.15.31:0/1681218791 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fd94c010ac0 con 0x7fd960101e60 2024-03-21T10:49:46.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.855+0000 7fd96adf9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c9260 0x55fe326d1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.855+0000 7fd96adf9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe326d1de0 0x55fe326d4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.855+0000 7fd96adf9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe3262c2c0 con 0x55fe326c88b0 2024-03-21T10:49:46.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.855+0000 7fd96adf9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe325d4c90 con 0x55fe326c9260 2024-03-21T10:49:46.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.855+0000 7fd96adf9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe32628360 con 0x55fe326d1de0 2024-03-21T10:49:46.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.855+0000 7fd965800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c9260 0x55fe326d1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.855+0000 7fd966200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe326c88b0 0x55fe326c8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.855+0000 7fd964e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe326d1de0 0x55fe326d4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.855+0000 7fd965800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c9260 0x55fe326d1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44246/0 (socket says 172.21.15.31:44246) 2024-03-21T10:49:46.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.855+0000 7fd965800640 1 -- 172.21.15.31:0/3094658890 learned_addr learned my addr 172.21.15.31:0/3094658890 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:46.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.855+0000 7fd965800640 1 -- 172.21.15.31:0/3094658890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe326d1de0 msgr2=0x55fe326d4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:46.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.855+0000 7fd965800640 1 --2- 172.21.15.31:0/3094658890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe326d1de0 0x55fe326d4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.855+0000 7fd965800640 1 -- 172.21.15.31:0/3094658890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe326c88b0 msgr2=0x55fe326c8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:46.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.855+0000 7fd965800640 1 --2- 172.21.15.31:0/3094658890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe326c88b0 0x55fe326c8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.855+0000 7fd965800640 1 -- 172.21.15.31:0/3094658890 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe325f9080 con 0x55fe326c9260 2024-03-21T10:49:46.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.856+0000 7fd965800640 1 --2- 172.21.15.31:0/3094658890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c9260 0x55fe326d1810 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fd954009c60 tx=0x7fd954011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=650e827b22f85875 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:46.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.856+0000 7fd952a00640 1 -- 172.21.15.31:0/3094658890 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd954005220 con 0x55fe326c9260 2024-03-21T10:49:46.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.856+0000 7fd952a00640 1 -- 172.21.15.31:0/3094658890 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd954009e10 con 0x55fe326c9260 2024-03-21T10:49:46.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.856+0000 7fd952a00640 1 -- 172.21.15.31:0/3094658890 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd954019870 con 0x55fe326c9260 2024-03-21T10:49:46.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.856+0000 7fd95e000640 1 --2- 172.21.15.31:0/1681218791 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd9480a3110 0x7fd9480a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.856+0000 7fd95e000640 1 -- 172.21.15.31:0/1681218791 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fd94c0cfec0 con 0x7fd960101e60 2024-03-21T10:49:46.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.857+0000 7fd966200640 1 --2- 172.21.15.31:0/1681218791 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd9480a3110 0x7fd9480a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.857+0000 7fd96adf9740 1 -- 172.21.15.31:0/3094658890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c9260 msgr2=0x55fe326d1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:46.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.857+0000 7fd96adf9740 1 --2- 172.21.15.31:0/3094658890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c9260 0x55fe326d1810 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fd954009c60 tx=0x7fd954011810 comp rx=0 tx=0).stop 2024-03-21T10:49:46.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.857+0000 7fd96adf9740 1 -- 172.21.15.31:0/3094658890 shutdown_connections 2024-03-21T10:49:46.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.857+0000 7fd96adf9740 1 --2- 172.21.15.31:0/3094658890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe326d1de0 0x55fe326d4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.857+0000 7fd96adf9740 1 --2- 172.21.15.31:0/3094658890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c9260 0x55fe326d1810 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.857+0000 7fd96adf9740 1 --2- 172.21.15.31:0/3094658890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe326c88b0 0x55fe326c8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.857+0000 7fd96adf9740 1 -- 172.21.15.31:0/3094658890 >> 172.21.15.31:0/3094658890 conn(0x55fe326c2f80 msgr2=0x55fe326c3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:46.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.857+0000 7fd96adf9740 1 -- 172.21.15.31:0/3094658890 shutdown_connections 2024-03-21T10:49:46.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.857+0000 7fd96adf9740 1 -- 172.21.15.31:0/3094658890 wait complete. 2024-03-21T10:49:46.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.858+0000 7fd966200640 1 --2- 172.21.15.31:0/1681218791 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd9480a3110 0x7fd9480a55d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd958033420 tx=0x7fd958005fd0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:46.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.858+0000 7fd96adf9740 1 -- start start 2024-03-21T10:49:46.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.858+0000 7fd96adf9740 1 Processor -- start 2024-03-21T10:49:46.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.858+0000 7fd96adf9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c88b0 0x55fe326d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.859+0000 7fd96adf9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe326c9260 0x55fe326cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.859+0000 7fd96adf9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe326d1de0 0x55fe326cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.859+0000 7fd96adf9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe326d4d50 con 0x55fe326c9260 2024-03-21T10:49:46.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.859+0000 7fd96adf9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe325e80d0 con 0x55fe326c88b0 2024-03-21T10:49:46.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.859+0000 7fd96adf9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe326d4ed0 con 0x55fe326d1de0 2024-03-21T10:49:46.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.859+0000 7fd96adf9740 10 client.0 _ll_register_callbacks cb 0x55fe325ece80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:46.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.859+0000 7fd965800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c88b0 0x55fe326d16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.859+0000 7fd965800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c88b0 0x55fe326d16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44248/0 (socket says 172.21.15.31:44248) 2024-03-21T10:49:46.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.859+0000 7fd965800640 1 -- 172.21.15.31:0/752725988 learned_addr learned my addr 172.21.15.31:0/752725988 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:46.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.859+0000 7fd966200640 1 --2- 172.21.15.31:0/752725988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe326d1de0 0x55fe326cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.859+0000 7fd964e00640 1 --2- 172.21.15.31:0/752725988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe326c9260 0x55fe326cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.859+0000 7fd965800640 1 -- 172.21.15.31:0/752725988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe326d1de0 msgr2=0x55fe326cefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:46.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.859+0000 7fd965800640 1 --2- 172.21.15.31:0/752725988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe326d1de0 0x55fe326cefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.859+0000 7fd965800640 1 -- 172.21.15.31:0/752725988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe326c9260 msgr2=0x55fe326cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:46.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.859+0000 7fd965800640 1 --2- 172.21.15.31:0/752725988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe326c9260 0x55fe326cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.859+0000 7fd965800640 1 -- 172.21.15.31:0/752725988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe323403f0 con 0x55fe326c88b0 2024-03-21T10:49:46.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.859+0000 7fd966200640 1 --2- 172.21.15.31:0/752725988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe326d1de0 0x55fe326cefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:49:46.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.860+0000 7fd965800640 1 --2- 172.21.15.31:0/752725988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c88b0 0x55fe326d16b0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd95401b040 tx=0x7fd954005780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ed9f58793a7099e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:46.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.860+0000 7fd947400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:49:46.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.860+0000 7fd947400640 1 -- 172.21.15.31:0/752725988 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd954029070 con 0x55fe326c88b0 2024-03-21T10:49:46.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.860+0000 7fd947400640 1 -- 172.21.15.31:0/752725988 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fd954004020 con 0x55fe326c88b0 2024-03-21T10:49:46.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.860+0000 7fd96adf9740 1 -- 172.21.15.31:0/752725988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd954009e10 con 0x55fe326c88b0 2024-03-21T10:49:46.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.860+0000 7fd947400640 1 -- 172.21.15.31:0/752725988 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd954034050 con 0x55fe326c88b0 2024-03-21T10:49:46.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.860+0000 7fd96adf9740 10 client.28448 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:46.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.860+0000 7fd96adf9740 1 -- 172.21.15.31:0/752725988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55fe3240a7d0 con 0x55fe326c88b0 2024-03-21T10:49:46.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.860+0000 7fd947400640 1 -- 172.21.15.31:0/752725988 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd954009e10 con 0x55fe326c88b0 2024-03-21T10:49:46.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.861+0000 7fd947400640 1 -- 172.21.15.31:0/752725988 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd954003ad0 con 0x55fe326c88b0 2024-03-21T10:49:46.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.861+0000 7fd96adf9740 10 client.28448 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:46.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.861+0000 7fd96adf9740 10 client.28448 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:49:46.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.861+0000 7fd96adf9740 20 client.28448 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:46.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.861+0000 7fd96adf9740 1 --2- 172.21.15.31:0/752725988 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55fe326e1ab0 0x55fe326e3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.861+0000 7fd96adf9740 4 client.28448 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:49:46.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.861+0000 7fd96adf9740 1 -- 172.21.15.31:0/752725988 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fe32639800 con 0x55fe326e1ab0 2024-03-21T10:49:46.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.861+0000 7fd964e00640 1 --2- 172.21.15.31:0/752725988 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55fe326e1ab0 0x55fe326e3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.862+0000 7fd964e00640 1 --2- 172.21.15.31:0/752725988 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55fe326e1ab0 0x55fe326e3ea0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6b01f65f6fa5cc1d server_cookie=64f9c938822dc3a7 in_seq=0 out_seq=0 2024-03-21T10:49:46.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.862+0000 7fd947400640 10 client.28448 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:49:46.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.862+0000 7fd947400640 1 -- 172.21.15.31:0/752725988 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55fe32639800 con 0x55fe326e1ab0 2024-03-21T10:49:46.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.862+0000 7fd947400640 10 client.28448 handle_command_reply: tid=0 2024-03-21T10:49:46.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.862+0000 7fd96adf9740 1 client.28448 shutdown 2024-03-21T10:49:46.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.862+0000 7fd96adf9740 1 -- 172.21.15.31:0/752725988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c88b0 msgr2=0x55fe326d16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:46.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.862+0000 7fd96adf9740 1 --2- 172.21.15.31:0/752725988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c88b0 0x55fe326d16b0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd95401b040 tx=0x7fd954005780 comp rx=0 tx=0).stop 2024-03-21T10:49:46.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.862+0000 7fd96adf9740 1 -- 172.21.15.31:0/752725988 shutdown_connections 2024-03-21T10:49:46.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.863+0000 7fd96adf9740 1 --2- 172.21.15.31:0/752725988 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55fe326e1ab0 0x55fe326e3ea0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.863+0000 7fd96adf9740 1 --2- 172.21.15.31:0/752725988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe326d1de0 0x55fe326cefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.863+0000 7fd96adf9740 1 --2- 172.21.15.31:0/752725988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe326c9260 0x55fe326cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.863+0000 7fd96adf9740 1 --2- 172.21.15.31:0/752725988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c88b0 0x55fe326d16b0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.863+0000 7fd947400640 0 client.28448 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:49:46.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.863+0000 7fd96adf9740 1 -- 172.21.15.31:0/752725988 >> 172.21.15.31:0/752725988 conn(0x55fe326c2f80 msgr2=0x55fe326c4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:46.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.863+0000 7fd96adf9740 1 -- 172.21.15.31:0/752725988 shutdown_connections 2024-03-21T10:49:46.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.863+0000 7fd96adf9740 1 -- 172.21.15.31:0/752725988 wait complete. 2024-03-21T10:49:46.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.863+0000 7fd96adf9740 20 client.28448 trim_cache size 0 max 16384 2024-03-21T10:49:46.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.876+0000 7fd96adf9740 1 Processor -- start 2024-03-21T10:49:46.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.876+0000 7fd96adf9740 1 -- start start 2024-03-21T10:49:46.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.877+0000 7fd96adf9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c88b0 0x55fe326ce320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.877+0000 7fd96adf9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe326c9260 0x55fe326c1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.877+0000 7fd96adf9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe326d1de0 0x55fe326d1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.877+0000 7fd96adf9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9601b2fc0 con 0x55fe326c9260 2024-03-21T10:49:46.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.877+0000 7fd96adf9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9601b0ac0 con 0x55fe326c88b0 2024-03-21T10:49:46.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.877+0000 7fd96adf9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9601aeb80 con 0x55fe326d1de0 2024-03-21T10:49:46.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.877+0000 7fd964e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe326c9260 0x55fe326c1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.877+0000 7fd966200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe326d1de0 0x55fe326d1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.877+0000 7fd965800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c88b0 0x55fe326ce320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.877+0000 7fd966200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe326d1de0 0x55fe326d1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:44372/0 (socket says 172.21.15.31:44372) 2024-03-21T10:49:46.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.877+0000 7fd966200640 1 -- 172.21.15.31:0/2381216211 learned_addr learned my addr 172.21.15.31:0/2381216211 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:46.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.877+0000 7fd965800640 1 -- 172.21.15.31:0/2381216211 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe326d1de0 msgr2=0x55fe326d1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:46.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.877+0000 7fd965800640 1 --2- 172.21.15.31:0/2381216211 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe326d1de0 0x55fe326d1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.878+0000 7fd965800640 1 -- 172.21.15.31:0/2381216211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe326c9260 msgr2=0x55fe326c1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:46.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.878+0000 7fd965800640 1 --2- 172.21.15.31:0/2381216211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe326c9260 0x55fe326c1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.878+0000 7fd965800640 1 -- 172.21.15.31:0/2381216211 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe326d4c30 con 0x55fe326c88b0 2024-03-21T10:49:46.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.878+0000 7fd966200640 1 --2- 172.21.15.31:0/2381216211 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe326d1de0 0x55fe326d1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:46.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.878+0000 7fd965800640 1 --2- 172.21.15.31:0/2381216211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c88b0 0x55fe326ce320 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fd954011850 tx=0x7fd9540196a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1969afc46d7eb805 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:46.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.878+0000 7fd952000640 1 -- 172.21.15.31:0/2381216211 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd954029070 con 0x55fe326c88b0 2024-03-21T10:49:46.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.878+0000 7fd952000640 1 -- 172.21.15.31:0/2381216211 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd954009870 con 0x55fe326c88b0 2024-03-21T10:49:46.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.878+0000 7fd952000640 1 -- 172.21.15.31:0/2381216211 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd954005590 con 0x55fe326c88b0 2024-03-21T10:49:46.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.879+0000 7fd96adf9740 1 -- 172.21.15.31:0/2381216211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c88b0 msgr2=0x55fe326ce320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:46.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.879+0000 7fd96adf9740 1 --2- 172.21.15.31:0/2381216211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c88b0 0x55fe326ce320 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fd954011850 tx=0x7fd9540196a0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.879+0000 7fd96adf9740 1 -- 172.21.15.31:0/2381216211 shutdown_connections 2024-03-21T10:49:46.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.879+0000 7fd96adf9740 1 --2- 172.21.15.31:0/2381216211 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe326d1de0 0x55fe326d1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.879+0000 7fd96adf9740 1 --2- 172.21.15.31:0/2381216211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe326c9260 0x55fe326c1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.879+0000 7fd96adf9740 1 --2- 172.21.15.31:0/2381216211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c88b0 0x55fe326ce320 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.879+0000 7fd96adf9740 1 -- 172.21.15.31:0/2381216211 >> 172.21.15.31:0/2381216211 conn(0x55fe326c2f80 msgr2=0x55fe326c18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:46.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.879+0000 7fd96adf9740 1 -- 172.21.15.31:0/2381216211 shutdown_connections 2024-03-21T10:49:46.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.879+0000 7fd96adf9740 1 -- 172.21.15.31:0/2381216211 wait complete. 2024-03-21T10:49:46.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.879+0000 7fd96adf9740 1 -- start start 2024-03-21T10:49:46.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.879+0000 7fd96adf9740 1 Processor -- start 2024-03-21T10:49:46.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.880+0000 7fd96adf9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c88b0 0x55fe326da040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.880+0000 7fd96adf9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe326c9260 0x55fe326d54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.880+0000 7fd96adf9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe326d1de0 0x55fe326d5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.880+0000 7fd96adf9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9601af180 con 0x55fe326d1de0 2024-03-21T10:49:46.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.880+0000 7fd96adf9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9601b0460 con 0x55fe326c88b0 2024-03-21T10:49:46.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.880+0000 7fd96adf9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9601af7a0 con 0x55fe326c9260 2024-03-21T10:49:46.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.880+0000 7fd96adf9740 10 client.0 _ll_register_callbacks cb 0x7fd94c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:46.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.880+0000 7fd965800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c88b0 0x55fe326da040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.880+0000 7fd964e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe326c9260 0x55fe326d54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.880+0000 7fd966200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe326d1de0 0x55fe326d5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.880+0000 7fd965800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c88b0 0x55fe326da040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44272/0 (socket says 172.21.15.31:44272) 2024-03-21T10:49:46.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.880+0000 7fd964e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe326c9260 0x55fe326d54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:44380/0 (socket says 172.21.15.31:44380) 2024-03-21T10:49:46.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.880+0000 7fd965800640 1 -- 172.21.15.31:0/1707376829 learned_addr learned my addr 172.21.15.31:0/1707376829 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:46.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.881+0000 7fd964e00640 1 -- 172.21.15.31:0/1707376829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c88b0 msgr2=0x55fe326da040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:46.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.881+0000 7fd964e00640 1 --2- 172.21.15.31:0/1707376829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c88b0 0x55fe326da040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.881+0000 7fd964e00640 1 -- 172.21.15.31:0/1707376829 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe326d1de0 msgr2=0x55fe326d5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:46.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.881+0000 7fd964e00640 1 --2- 172.21.15.31:0/1707376829 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe326d1de0 0x55fe326d5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.881+0000 7fd964e00640 1 -- 172.21.15.31:0/1707376829 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe3273c150 con 0x55fe326c9260 2024-03-21T10:49:46.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.881+0000 7fd965800640 1 --2- 172.21.15.31:0/1707376829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c88b0 0x55fe326da040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:46.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.881+0000 7fd964e00640 1 --2- 172.21.15.31:0/1707376829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe326c9260 0x55fe326d54f0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fd94c0cd270 tx=0x7fd94c005f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=717618693aa2c7a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:46.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.881+0000 7fd947400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:49:46.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.881+0000 7fd947400640 1 -- 172.21.15.31:0/1707376829 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94c09bfc0 con 0x55fe326c9260 2024-03-21T10:49:46.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.881+0000 7fd947400640 1 -- 172.21.15.31:0/1707376829 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fd94c0cd4f0 con 0x55fe326c9260 2024-03-21T10:49:46.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.881+0000 7fd96adf9740 1 -- 172.21.15.31:0/1707376829 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd954009870 con 0x55fe326c9260 2024-03-21T10:49:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.881+0000 7fd947400640 1 -- 172.21.15.31:0/1707376829 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd94c0a0060 con 0x55fe326c9260 2024-03-21T10:49:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.881+0000 7fd96adf9740 10 client.28381 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.881+0000 7fd96adf9740 1 -- 172.21.15.31:0/1707376829 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55fe3273d2f0 con 0x55fe326c9260 2024-03-21T10:49:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.881+0000 7fd947400640 1 -- 172.21.15.31:0/1707376829 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd954009870 con 0x55fe326c9260 2024-03-21T10:49:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.882+0000 7fd947400640 1 -- 172.21.15.31:0/1707376829 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd94c0b8070 con 0x55fe326c9260 2024-03-21T10:49:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.882+0000 7fd96adf9740 10 client.28381 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.882+0000 7fd96adf9740 10 client.28381 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:49:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.882+0000 7fd96adf9740 20 client.28381 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.882+0000 7fd96adf9740 1 --2- 172.21.15.31:0/1707376829 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55fe3273d980 0x55fe3273fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.882+0000 7fd96adf9740 4 client.28381 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:49:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.882+0000 7fd96adf9740 1 -- 172.21.15.31:0/1707376829 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55fe32633b50 con 0x55fe3273d980 2024-03-21T10:49:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.882+0000 7fd965800640 1 --2- 172.21.15.31:0/1707376829 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55fe3273d980 0x55fe3273fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.882+0000 7fd965800640 1 --2- 172.21.15.31:0/1707376829 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55fe3273d980 0x55fe3273fda0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a883cd99a851b4ea server_cookie=b0687362f13fc5c2 in_seq=0 out_seq=0 2024-03-21T10:49:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.882+0000 7fd947400640 10 client.28381 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:49:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.883+0000 7fd947400640 1 -- 172.21.15.31:0/1707376829 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55fe32633b50 con 0x55fe3273d980 2024-03-21T10:49:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.883+0000 7fd947400640 10 client.28381 handle_command_reply: tid=0 2024-03-21T10:49:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.883+0000 7fd96adf9740 1 client.28381 shutdown 2024-03-21T10:49:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.883+0000 7fd96adf9740 1 -- 172.21.15.31:0/1707376829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe326c9260 msgr2=0x55fe326d54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.883+0000 7fd96adf9740 1 --2- 172.21.15.31:0/1707376829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe326c9260 0x55fe326d54f0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fd94c0cd270 tx=0x7fd94c005f10 comp rx=0 tx=0).stop 2024-03-21T10:49:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.883+0000 7fd96adf9740 1 -- 172.21.15.31:0/1707376829 shutdown_connections 2024-03-21T10:49:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.883+0000 7fd96adf9740 1 --2- 172.21.15.31:0/1707376829 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55fe3273d980 0x55fe3273fda0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.883+0000 7fd96adf9740 1 --2- 172.21.15.31:0/1707376829 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe326d1de0 0x55fe326d5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.883+0000 7fd96adf9740 1 --2- 172.21.15.31:0/1707376829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe326c9260 0x55fe326d54f0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.883+0000 7fd947400640 0 client.28381 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:49:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.883+0000 7fd96adf9740 1 --2- 172.21.15.31:0/1707376829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe326c88b0 0x55fe326da040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.883+0000 7fd96adf9740 1 -- 172.21.15.31:0/1707376829 >> 172.21.15.31:0/1707376829 conn(0x55fe326c2f80 msgr2=0x55fe326c2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.883+0000 7fd96adf9740 1 -- 172.21.15.31:0/1707376829 shutdown_connections 2024-03-21T10:49:46.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.883+0000 7fd96adf9740 1 -- 172.21.15.31:0/1707376829 wait complete. 2024-03-21T10:49:46.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.883+0000 7fd96adf9740 20 client.28381 trim_cache size 0 max 16384 2024-03-21T10:49:46.885 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:49:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.884+0000 7fd966c00640 1 -- 172.21.15.31:0/1681218791 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd9480a3110 msgr2=0x7fd9480a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.884+0000 7fd966c00640 1 --2- 172.21.15.31:0/1681218791 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd9480a3110 0x7fd9480a55d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd958033420 tx=0x7fd958005fd0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.884+0000 7fd966c00640 1 -- 172.21.15.31:0/1681218791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd960101e60 msgr2=0x7fd96019a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.884+0000 7fd966c00640 1 --2- 172.21.15.31:0/1681218791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd960101e60 0x7fd96019a4a0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fd94c00e9f0 tx=0x7fd94c00eeb0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.884+0000 7fd966c00640 1 -- 172.21.15.31:0/1681218791 shutdown_connections 2024-03-21T10:49:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.884+0000 7fd966c00640 1 --2- 172.21.15.31:0/1681218791 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fd9480a3110 0x7fd9480a55d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.884+0000 7fd966c00640 1 --2- 172.21.15.31:0/1681218791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd960107560 0x7fd96019e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.884+0000 7fd966c00640 1 --2- 172.21.15.31:0/1681218791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd960101e60 0x7fd96019a4a0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.884+0000 7fd966c00640 1 --2- 172.21.15.31:0/1681218791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9601014a0 0x7fd960199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.884+0000 7fd966c00640 1 -- 172.21.15.31:0/1681218791 >> 172.21.15.31:0/1681218791 conn(0x7fd9600fcfb0 msgr2=0x7fd9600fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.884+0000 7fd966c00640 1 -- 172.21.15.31:0/1681218791 shutdown_connections 2024-03-21T10:49:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:46.884+0000 7fd966c00640 1 -- 172.21.15.31:0/1681218791 wait complete. 2024-03-21T10:49:46.901 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:49:46.901 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-21T10:49:47.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.122+0000 7fc92fe00640 1 -- 172.21.15.31:0/74557554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc928069e20 msgr2=0x7fc92806a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:47.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.122+0000 7fc92fe00640 1 --2- 172.21.15.31:0/74557554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc928069e20 0x7fc92806a2a0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fc914009960 tx=0x7fc91402bc10 comp rx=0 tx=0).stop 2024-03-21T10:49:47.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.122+0000 7fc92fe00640 1 -- 172.21.15.31:0/74557554 shutdown_connections 2024-03-21T10:49:47.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.122+0000 7fc92fe00640 1 --2- 172.21.15.31:0/74557554 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc92806a7e0 0x7fc92810e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.122+0000 7fc92fe00640 1 --2- 172.21.15.31:0/74557554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc928069e20 0x7fc92806a2a0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.122+0000 7fc92fe00640 1 --2- 172.21.15.31:0/74557554 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9281076f0 0x7fc928107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.122+0000 7fc92fe00640 1 -- 172.21.15.31:0/74557554 >> 172.21.15.31:0/74557554 conn(0x7fc928077640 msgr2=0x7fc928077a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:47.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.122+0000 7fc92fe00640 1 -- 172.21.15.31:0/74557554 shutdown_connections 2024-03-21T10:49:47.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.122+0000 7fc92fe00640 1 -- 172.21.15.31:0/74557554 wait complete. 2024-03-21T10:49:47.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.123+0000 7fc92fe00640 1 Processor -- start 2024-03-21T10:49:47.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.124+0000 7fc92fe00640 1 -- start start 2024-03-21T10:49:47.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.124+0000 7fc92fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc928069e20 0x7fc92819a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.124+0000 7fc92fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc92806a7e0 0x7fc92819a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.124+0000 7fc92fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9281076f0 0x7fc92819e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.124+0000 7fc92fe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc928111a40 con 0x7fc9281076f0 2024-03-21T10:49:47.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.124+0000 7fc92fe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9281118c0 con 0x7fc928069e20 2024-03-21T10:49:47.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.124+0000 7fc92fe00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc928111bc0 con 0x7fc92806a7e0 2024-03-21T10:49:47.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.124+0000 7fc92d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc928069e20 0x7fc92819a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.124+0000 7fc92ca00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc92806a7e0 0x7fc92819a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.124+0000 7fc92d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc928069e20 0x7fc92819a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33162/0 (socket says 172.21.15.31:33162) 2024-03-21T10:49:47.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.124+0000 7fc92d400640 1 -- 172.21.15.31:0/216966609 learned_addr learned my addr 172.21.15.31:0/216966609 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:47.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.124+0000 7fc92de00640 1 --2- 172.21.15.31:0/216966609 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9281076f0 0x7fc92819e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.125+0000 7fc92d400640 1 -- 172.21.15.31:0/216966609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc92806a7e0 msgr2=0x7fc92819a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:47.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.125+0000 7fc92d400640 1 --2- 172.21.15.31:0/216966609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc92806a7e0 0x7fc92819a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.125+0000 7fc92d400640 1 -- 172.21.15.31:0/216966609 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9281076f0 msgr2=0x7fc92819e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:47.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.125+0000 7fc92d400640 1 --2- 172.21.15.31:0/216966609 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9281076f0 0x7fc92819e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.125+0000 7fc92d400640 1 -- 172.21.15.31:0/216966609 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc92819f0f0 con 0x7fc928069e20 2024-03-21T10:49:47.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.125+0000 7fc92ca00640 1 --2- 172.21.15.31:0/216966609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc92806a7e0 0x7fc92819a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:47.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.125+0000 7fc92d400640 1 --2- 172.21.15.31:0/216966609 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc928069e20 0x7fc92819a100 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fc91c00ebc0 tx=0x7fc91c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:47.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.125+0000 7fc926000640 1 -- 172.21.15.31:0/216966609 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c010630 con 0x7fc928069e20 2024-03-21T10:49:47.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.125+0000 7fc926000640 1 -- 172.21.15.31:0/216966609 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc91c010cb0 con 0x7fc928069e20 2024-03-21T10:49:47.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.125+0000 7fc92fe00640 1 -- 172.21.15.31:0/216966609 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc92819f3e0 con 0x7fc928069e20 2024-03-21T10:49:47.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.125+0000 7fc926000640 1 -- 172.21.15.31:0/216966609 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c018bd0 con 0x7fc928069e20 2024-03-21T10:49:47.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.125+0000 7fc92fe00640 1 -- 172.21.15.31:0/216966609 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9281a6c70 con 0x7fc928069e20 2024-03-21T10:49:47.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.128+0000 7fc926000640 1 -- 172.21.15.31:0/216966609 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fc91c0107d0 con 0x7fc928069e20 2024-03-21T10:49:47.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.129+0000 7fc932a5d740 1 Processor -- start 2024-03-21T10:49:47.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.129+0000 7fc932a5d740 1 -- start start 2024-03-21T10:49:47.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.129+0000 7fc932a5d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d0f7166d0 0x564d0f716ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.129+0000 7fc932a5d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d0f717080 0x564d0f71f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.129+0000 7fc932a5d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d0f71fc00 0x564d0f722050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.129+0000 7fc932a5d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0f67a2c0 con 0x564d0f7166d0 2024-03-21T10:49:47.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.129+0000 7fc932a5d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0f622c90 con 0x564d0f71fc00 2024-03-21T10:49:47.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.129+0000 7fc932a5d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0f676360 con 0x564d0f717080 2024-03-21T10:49:47.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.129+0000 7fc92de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d0f7166d0 0x564d0f716ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.130+0000 7fc92ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d0f71fc00 0x564d0f722050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.129+0000 7fc92d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d0f717080 0x564d0f71f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.130+0000 7fc92de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d0f7166d0 0x564d0f716ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39954/0 (socket says 172.21.15.31:39954) 2024-03-21T10:49:47.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.130+0000 7fc92de00640 1 -- 172.21.15.31:0/388495194 learned_addr learned my addr 172.21.15.31:0/388495194 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:47.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.130+0000 7fc92ca00640 1 -- 172.21.15.31:0/388495194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d0f717080 msgr2=0x564d0f71f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:47.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.130+0000 7fc92ca00640 1 --2- 172.21.15.31:0/388495194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d0f717080 0x564d0f71f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.130+0000 7fc92ca00640 1 -- 172.21.15.31:0/388495194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d0f7166d0 msgr2=0x564d0f716ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:47.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.130+0000 7fc92ca00640 1 --2- 172.21.15.31:0/388495194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d0f7166d0 0x564d0f716ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.130+0000 7fc92ca00640 1 -- 172.21.15.31:0/388495194 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d0f647080 con 0x564d0f71fc00 2024-03-21T10:49:47.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.130+0000 7fc92ca00640 1 --2- 172.21.15.31:0/388495194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d0f71fc00 0x564d0f722050 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fc914006010 tx=0x7fc914009670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e37d3af2d760dd14 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:47.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.131+0000 7fc91a600640 1 -- 172.21.15.31:0/388495194 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc914042070 con 0x564d0f71fc00 2024-03-21T10:49:47.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.131+0000 7fc91a600640 1 -- 172.21.15.31:0/388495194 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc914004a70 con 0x564d0f71fc00 2024-03-21T10:49:47.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.131+0000 7fc91a600640 1 -- 172.21.15.31:0/388495194 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc914005110 con 0x564d0f71fc00 2024-03-21T10:49:47.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.131+0000 7fc926000640 1 --2- 172.21.15.31:0/216966609 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc9100a3110 0x7fc9100a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.131+0000 7fc926000640 1 -- 172.21.15.31:0/216966609 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fc91c0d0180 con 0x7fc928069e20 2024-03-21T10:49:47.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.131+0000 7fc92de00640 1 --2- 172.21.15.31:0/216966609 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc9100a3110 0x7fc9100a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.131+0000 7fc932a5d740 1 -- 172.21.15.31:0/388495194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d0f71fc00 msgr2=0x564d0f722050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:47.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.131+0000 7fc932a5d740 1 --2- 172.21.15.31:0/388495194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d0f71fc00 0x564d0f722050 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fc914006010 tx=0x7fc914009670 comp rx=0 tx=0).stop 2024-03-21T10:49:47.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.131+0000 7fc932a5d740 1 -- 172.21.15.31:0/388495194 shutdown_connections 2024-03-21T10:49:47.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.131+0000 7fc932a5d740 1 --2- 172.21.15.31:0/388495194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d0f71fc00 0x564d0f722050 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.131+0000 7fc932a5d740 1 --2- 172.21.15.31:0/388495194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d0f717080 0x564d0f71f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.131+0000 7fc932a5d740 1 --2- 172.21.15.31:0/388495194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d0f7166d0 0x564d0f716ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.131+0000 7fc932a5d740 1 -- 172.21.15.31:0/388495194 >> 172.21.15.31:0/388495194 conn(0x564d0f710da0 msgr2=0x564d0f711180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:47.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.132+0000 7fc932a5d740 1 -- 172.21.15.31:0/388495194 shutdown_connections 2024-03-21T10:49:47.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.132+0000 7fc932a5d740 1 -- 172.21.15.31:0/388495194 wait complete. 2024-03-21T10:49:47.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.132+0000 7fc92de00640 1 --2- 172.21.15.31:0/216966609 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc9100a3110 0x7fc9100a55d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc920009dc0 tx=0x7fc920005f90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:47.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.132+0000 7fc932a5d740 1 -- start start 2024-03-21T10:49:47.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.133+0000 7fc932a5d740 1 Processor -- start 2024-03-21T10:49:47.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.133+0000 7fc932a5d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d0f7166d0 0x564d0f71f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.133+0000 7fc932a5d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d0f717080 0x564d0f71c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.133+0000 7fc932a5d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d0f71fc00 0x564d0f71cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.133+0000 7fc932a5d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0f722b10 con 0x564d0f7166d0 2024-03-21T10:49:47.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.133+0000 7fc932a5d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0f6360d0 con 0x564d0f717080 2024-03-21T10:49:47.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.133+0000 7fc932a5d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0f722c90 con 0x564d0f71fc00 2024-03-21T10:49:47.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.133+0000 7fc932a5d740 10 client.0 _ll_register_callbacks cb 0x564d0f63ae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:47.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.134+0000 7fc92de00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d0f71fc00 0x564d0f71cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.134+0000 7fc92ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d0f717080 0x564d0f71c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.134+0000 7fc92ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d0f717080 0x564d0f71c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33190/0 (socket says 172.21.15.31:33190) 2024-03-21T10:49:47.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.134+0000 7fc92ca00640 1 -- 172.21.15.31:0/3698785676 learned_addr learned my addr 172.21.15.31:0/3698785676 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:47.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.134+0000 7fc92d400640 1 --2- 172.21.15.31:0/3698785676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d0f7166d0 0x564d0f71f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.134+0000 7fc92ca00640 1 -- 172.21.15.31:0/3698785676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d0f71fc00 msgr2=0x564d0f71cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:47.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.134+0000 7fc92ca00640 1 --2- 172.21.15.31:0/3698785676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d0f71fc00 0x564d0f71cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.134+0000 7fc92ca00640 1 -- 172.21.15.31:0/3698785676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d0f7166d0 msgr2=0x564d0f71f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:47.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.134+0000 7fc92ca00640 1 --2- 172.21.15.31:0/3698785676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d0f7166d0 0x564d0f71f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.134+0000 7fc92ca00640 1 -- 172.21.15.31:0/3698785676 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564d0f38e3f0 con 0x564d0f717080 2024-03-21T10:49:47.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.134+0000 7fc92ca00640 1 --2- 172.21.15.31:0/3698785676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d0f717080 0x564d0f71c8d0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fc91402ce50 tx=0x7fc91402c310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1acd5538d22cc33 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:47.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.134+0000 7fc90ea00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:49:47.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.135+0000 7fc90ea00640 1 -- 172.21.15.31:0/3698785676 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc914042040 con 0x564d0f717080 2024-03-21T10:49:47.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.135+0000 7fc90ea00640 1 -- 172.21.15.31:0/3698785676 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fc9140050d0 con 0x564d0f717080 2024-03-21T10:49:47.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.135+0000 7fc932a5d740 1 -- 172.21.15.31:0/3698785676 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc914004a70 con 0x564d0f717080 2024-03-21T10:49:47.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.135+0000 7fc90ea00640 1 -- 172.21.15.31:0/3698785676 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc914058050 con 0x564d0f717080 2024-03-21T10:49:47.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.135+0000 7fc932a5d740 10 client.28466 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:47.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.135+0000 7fc932a5d740 1 -- 172.21.15.31:0/3698785676 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x564d0f4587d0 con 0x564d0f717080 2024-03-21T10:49:47.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.135+0000 7fc90ea00640 1 -- 172.21.15.31:0/3698785676 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc914004a70 con 0x564d0f717080 2024-03-21T10:49:47.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.135+0000 7fc90ea00640 1 -- 172.21.15.31:0/3698785676 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fc91404fc60 con 0x564d0f717080 2024-03-21T10:49:47.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.135+0000 7fc932a5d740 10 client.28466 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:47.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.135+0000 7fc932a5d740 10 client.28466 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:49:47.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.136+0000 7fc932a5d740 20 client.28466 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:47.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.136+0000 7fc932a5d740 1 --2- 172.21.15.31:0/3698785676 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x564d0f72f8d0 0x564d0f731cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.136+0000 7fc932a5d740 4 client.28466 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:49:47.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.136+0000 7fc932a5d740 1 -- 172.21.15.31:0/3698785676 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564d0f687800 con 0x564d0f72f8d0 2024-03-21T10:49:47.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.136+0000 7fc92d400640 1 --2- 172.21.15.31:0/3698785676 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x564d0f72f8d0 0x564d0f731cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.136+0000 7fc92d400640 1 --2- 172.21.15.31:0/3698785676 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x564d0f72f8d0 0x564d0f731cc0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=adeb6315b9c1644d server_cookie=f16bdeb0fc26bbf0 in_seq=0 out_seq=0 2024-03-21T10:49:47.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.136+0000 7fc90ea00640 10 client.28466 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:49:47.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.137+0000 7fc90ea00640 1 -- 172.21.15.31:0/3698785676 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564d0f687800 con 0x564d0f72f8d0 2024-03-21T10:49:47.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.137+0000 7fc90ea00640 10 client.28466 handle_command_reply: tid=0 2024-03-21T10:49:47.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.137+0000 7fc932a5d740 1 client.28466 shutdown 2024-03-21T10:49:47.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.138+0000 7fc932a5d740 1 -- 172.21.15.31:0/3698785676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d0f717080 msgr2=0x564d0f71c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:47.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.138+0000 7fc932a5d740 1 --2- 172.21.15.31:0/3698785676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d0f717080 0x564d0f71c8d0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fc91402ce50 tx=0x7fc91402c310 comp rx=0 tx=0).stop 2024-03-21T10:49:47.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.138+0000 7fc932a5d740 1 -- 172.21.15.31:0/3698785676 shutdown_connections 2024-03-21T10:49:47.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.138+0000 7fc932a5d740 1 --2- 172.21.15.31:0/3698785676 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x564d0f72f8d0 0x564d0f731cc0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.138+0000 7fc932a5d740 1 --2- 172.21.15.31:0/3698785676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d0f71fc00 0x564d0f71cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.138+0000 7fc932a5d740 1 --2- 172.21.15.31:0/3698785676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d0f717080 0x564d0f71c8d0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.138+0000 7fc90ea00640 0 client.28466 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:49:47.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.138+0000 7fc932a5d740 1 --2- 172.21.15.31:0/3698785676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d0f7166d0 0x564d0f71f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.138+0000 7fc932a5d740 1 -- 172.21.15.31:0/3698785676 >> 172.21.15.31:0/3698785676 conn(0x564d0f710da0 msgr2=0x564d0f712af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:47.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.138+0000 7fc932a5d740 1 -- 172.21.15.31:0/3698785676 shutdown_connections 2024-03-21T10:49:47.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.138+0000 7fc932a5d740 1 -- 172.21.15.31:0/3698785676 wait complete. 2024-03-21T10:49:47.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.138+0000 7fc932a5d740 20 client.28466 trim_cache size 0 max 16384 2024-03-21T10:49:47.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.161+0000 7fc932a5d740 1 Processor -- start 2024-03-21T10:49:47.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.161+0000 7fc932a5d740 1 -- start start 2024-03-21T10:49:47.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.161+0000 7fc932a5d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d0f7166d0 0x564d0f70f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.161+0000 7fc932a5d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d0f717080 0x564d0f71f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.161+0000 7fc932a5d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d0f71fc00 0x564d0f71f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.161+0000 7fc932a5d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9281b3050 con 0x564d0f7166d0 2024-03-21T10:49:47.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.161+0000 7fc932a5d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9281b0b50 con 0x564d0f717080 2024-03-21T10:49:47.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.161+0000 7fc932a5d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9281aec10 con 0x564d0f71fc00 2024-03-21T10:49:47.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.162+0000 7fc92ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d0f717080 0x564d0f71f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.162+0000 7fc92d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d0f7166d0 0x564d0f70f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.162+0000 7fc92ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d0f717080 0x564d0f71f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33204/0 (socket says 172.21.15.31:33204) 2024-03-21T10:49:47.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.162+0000 7fc92de00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d0f71fc00 0x564d0f71f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.162+0000 7fc92ca00640 1 -- 172.21.15.31:0/2432444496 learned_addr learned my addr 172.21.15.31:0/2432444496 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:47.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.162+0000 7fc92ca00640 1 -- 172.21.15.31:0/2432444496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d0f71fc00 msgr2=0x564d0f71f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:47.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.162+0000 7fc92ca00640 1 --2- 172.21.15.31:0/2432444496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d0f71fc00 0x564d0f71f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.162+0000 7fc92ca00640 1 -- 172.21.15.31:0/2432444496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d0f7166d0 msgr2=0x564d0f70f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:47.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.162+0000 7fc92ca00640 1 --2- 172.21.15.31:0/2432444496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d0f7166d0 0x564d0f70f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.162+0000 7fc92ca00640 1 -- 172.21.15.31:0/2432444496 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d0f71d250 con 0x564d0f717080 2024-03-21T10:49:47.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.162+0000 7fc92d400640 1 --2- 172.21.15.31:0/2432444496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d0f7166d0 0x564d0f70f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:47.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.162+0000 7fc92de00640 1 --2- 172.21.15.31:0/2432444496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d0f71fc00 0x564d0f71f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:47.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.162+0000 7fc92ca00640 1 --2- 172.21.15.31:0/2432444496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d0f717080 0x564d0f71f000 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fc91402bc50 tx=0x7fc914001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58b01348bcc54fac server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:47.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.162+0000 7fc919c00640 1 -- 172.21.15.31:0/2432444496 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc914042070 con 0x564d0f717080 2024-03-21T10:49:47.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.163+0000 7fc919c00640 1 -- 172.21.15.31:0/2432444496 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc914004a70 con 0x564d0f717080 2024-03-21T10:49:47.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.163+0000 7fc919c00640 1 -- 172.21.15.31:0/2432444496 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc914005250 con 0x564d0f717080 2024-03-21T10:49:47.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.163+0000 7fc932a5d740 1 -- 172.21.15.31:0/2432444496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d0f717080 msgr2=0x564d0f71f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:47.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.163+0000 7fc932a5d740 1 --2- 172.21.15.31:0/2432444496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d0f717080 0x564d0f71f000 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fc91402bc50 tx=0x7fc914001e70 comp rx=0 tx=0).stop 2024-03-21T10:49:47.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.163+0000 7fc932a5d740 1 -- 172.21.15.31:0/2432444496 shutdown_connections 2024-03-21T10:49:47.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.163+0000 7fc932a5d740 1 --2- 172.21.15.31:0/2432444496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d0f71fc00 0x564d0f71f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.163+0000 7fc932a5d740 1 --2- 172.21.15.31:0/2432444496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d0f717080 0x564d0f71f000 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.163+0000 7fc932a5d740 1 --2- 172.21.15.31:0/2432444496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d0f7166d0 0x564d0f70f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.163+0000 7fc932a5d740 1 -- 172.21.15.31:0/2432444496 >> 172.21.15.31:0/2432444496 conn(0x564d0f710da0 msgr2=0x564d0f717cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:47.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.163+0000 7fc932a5d740 1 -- 172.21.15.31:0/2432444496 shutdown_connections 2024-03-21T10:49:47.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.163+0000 7fc932a5d740 1 -- 172.21.15.31:0/2432444496 wait complete. 2024-03-21T10:49:47.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.164+0000 7fc932a5d740 1 -- start start 2024-03-21T10:49:47.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.164+0000 7fc932a5d740 1 Processor -- start 2024-03-21T10:49:47.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.164+0000 7fc932a5d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d0f7166d0 0x564d0f7348c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.164+0000 7fc932a5d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d0f717080 0x564d0f732cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.164+0000 7fc932a5d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d0f71fc00 0x564d0f7331f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.164+0000 7fc932a5d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9281af210 con 0x564d0f71fc00 2024-03-21T10:49:47.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.164+0000 7fc932a5d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9281b04f0 con 0x564d0f717080 2024-03-21T10:49:47.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.164+0000 7fc932a5d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9281af830 con 0x564d0f7166d0 2024-03-21T10:49:47.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.164+0000 7fc932a5d740 10 client.0 _ll_register_callbacks cb 0x7fc920009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:47.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.165+0000 7fc92d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d0f7166d0 0x564d0f7348c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.165+0000 7fc92d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d0f7166d0 0x564d0f7348c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34516/0 (socket says 172.21.15.31:34516) 2024-03-21T10:49:47.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.165+0000 7fc92d400640 1 -- 172.21.15.31:0/3458317485 learned_addr learned my addr 172.21.15.31:0/3458317485 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:47.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.165+0000 7fc92ca00640 1 --2- 172.21.15.31:0/3458317485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d0f717080 0x564d0f732cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.165+0000 7fc92de00640 1 --2- 172.21.15.31:0/3458317485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d0f71fc00 0x564d0f7331f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.165+0000 7fc92d400640 1 -- 172.21.15.31:0/3458317485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d0f717080 msgr2=0x564d0f732cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:47.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.165+0000 7fc92d400640 1 --2- 172.21.15.31:0/3458317485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d0f717080 0x564d0f732cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.165+0000 7fc92d400640 1 -- 172.21.15.31:0/3458317485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d0f71fc00 msgr2=0x564d0f7331f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:47.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.165+0000 7fc92d400640 1 --2- 172.21.15.31:0/3458317485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d0f71fc00 0x564d0f7331f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.165+0000 7fc92d400640 1 -- 172.21.15.31:0/3458317485 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564d0f733860 con 0x564d0f7166d0 2024-03-21T10:49:47.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.165+0000 7fc92d400640 1 --2- 172.21.15.31:0/3458317485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d0f7166d0 0x564d0f7348c0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fc91c098680 tx=0x7fc91c0d25d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1425b1d79ce7051c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:47.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.165+0000 7fc90ea00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:49:47.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.165+0000 7fc90ea00640 1 -- 172.21.15.31:0/3458317485 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c010e60 con 0x564d0f7166d0 2024-03-21T10:49:47.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.166+0000 7fc90ea00640 1 -- 172.21.15.31:0/3458317485 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fc91c098890 con 0x564d0f7166d0 2024-03-21T10:49:47.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.166+0000 7fc932a5d740 1 -- 172.21.15.31:0/3458317485 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc914004a70 con 0x564d0f7166d0 2024-03-21T10:49:47.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.166+0000 7fc90ea00640 1 -- 172.21.15.31:0/3458317485 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc91c0a02c0 con 0x564d0f7166d0 2024-03-21T10:49:47.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.166+0000 7fc932a5d740 10 client.28399 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:47.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.166+0000 7fc932a5d740 1 -- 172.21.15.31:0/3458317485 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x564d0f794530 con 0x564d0f7166d0 2024-03-21T10:49:47.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.166+0000 7fc90ea00640 1 -- 172.21.15.31:0/3458317485 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc914004a70 con 0x564d0f7166d0 2024-03-21T10:49:47.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.166+0000 7fc90ea00640 1 -- 172.21.15.31:0/3458317485 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fc91c0d3ca0 con 0x564d0f7166d0 2024-03-21T10:49:47.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.166+0000 7fc932a5d740 10 client.28399 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:47.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.166+0000 7fc932a5d740 10 client.28399 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:49:47.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.166+0000 7fc932a5d740 20 client.28399 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:47.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.166+0000 7fc932a5d740 1 --2- 172.21.15.31:0/3458317485 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x564d0f794bb0 0x564d0f796fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.166+0000 7fc932a5d740 4 client.28399 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:49:47.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.166+0000 7fc932a5d740 1 -- 172.21.15.31:0/3458317485 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564d0f681b50 con 0x564d0f794bb0 2024-03-21T10:49:47.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.167+0000 7fc92ca00640 1 --2- 172.21.15.31:0/3458317485 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x564d0f794bb0 0x564d0f796fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.167+0000 7fc92ca00640 1 --2- 172.21.15.31:0/3458317485 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x564d0f794bb0 0x564d0f796fd0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9facc64a916c2203 server_cookie=f8f33befedfef4df in_seq=0 out_seq=0 2024-03-21T10:49:47.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.167+0000 7fc90ea00640 10 client.28399 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:49:47.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.167+0000 7fc90ea00640 1 -- 172.21.15.31:0/3458317485 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564d0f681b50 con 0x564d0f794bb0 2024-03-21T10:49:47.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.167+0000 7fc90ea00640 10 client.28399 handle_command_reply: tid=0 2024-03-21T10:49:47.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.167+0000 7fc932a5d740 1 client.28399 shutdown 2024-03-21T10:49:47.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.168+0000 7fc932a5d740 1 -- 172.21.15.31:0/3458317485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d0f7166d0 msgr2=0x564d0f7348c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:47.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.168+0000 7fc932a5d740 1 --2- 172.21.15.31:0/3458317485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d0f7166d0 0x564d0f7348c0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fc91c098680 tx=0x7fc91c0d25d0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.168+0000 7fc932a5d740 1 -- 172.21.15.31:0/3458317485 shutdown_connections 2024-03-21T10:49:47.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.168+0000 7fc932a5d740 1 --2- 172.21.15.31:0/3458317485 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x564d0f794bb0 0x564d0f796fd0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.168+0000 7fc932a5d740 1 --2- 172.21.15.31:0/3458317485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d0f71fc00 0x564d0f7331f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.168+0000 7fc932a5d740 1 --2- 172.21.15.31:0/3458317485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564d0f717080 0x564d0f732cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.168+0000 7fc90ea00640 0 client.28399 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:49:47.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.168+0000 7fc932a5d740 1 --2- 172.21.15.31:0/3458317485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d0f7166d0 0x564d0f7348c0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.168+0000 7fc932a5d740 1 -- 172.21.15.31:0/3458317485 >> 172.21.15.31:0/3458317485 conn(0x564d0f710da0 msgr2=0x564d0f711da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:47.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.168+0000 7fc932a5d740 1 -- 172.21.15.31:0/3458317485 shutdown_connections 2024-03-21T10:49:47.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.168+0000 7fc932a5d740 1 -- 172.21.15.31:0/3458317485 wait complete. 2024-03-21T10:49:47.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.168+0000 7fc932a5d740 20 client.28399 trim_cache size 0 max 16384 2024-03-21T10:49:47.171 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:49:47.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.169+0000 7fc92fe00640 1 -- 172.21.15.31:0/216966609 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc9100a3110 msgr2=0x7fc9100a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:47.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.169+0000 7fc92fe00640 1 --2- 172.21.15.31:0/216966609 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc9100a3110 0x7fc9100a55d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc920009dc0 tx=0x7fc920005f90 comp rx=0 tx=0).stop 2024-03-21T10:49:47.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.169+0000 7fc92fe00640 1 -- 172.21.15.31:0/216966609 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc928069e20 msgr2=0x7fc92819a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:47.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.169+0000 7fc92fe00640 1 --2- 172.21.15.31:0/216966609 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc928069e20 0x7fc92819a100 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fc91c00ebc0 tx=0x7fc91c00c6a0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.169+0000 7fc92fe00640 1 -- 172.21.15.31:0/216966609 shutdown_connections 2024-03-21T10:49:47.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.169+0000 7fc92fe00640 1 --2- 172.21.15.31:0/216966609 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc9100a3110 0x7fc9100a55d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.169+0000 7fc92fe00640 1 --2- 172.21.15.31:0/216966609 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9281076f0 0x7fc92819e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.169+0000 7fc92fe00640 1 --2- 172.21.15.31:0/216966609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc92806a7e0 0x7fc92819a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.169+0000 7fc92fe00640 1 --2- 172.21.15.31:0/216966609 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc928069e20 0x7fc92819a100 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.169+0000 7fc92fe00640 1 -- 172.21.15.31:0/216966609 >> 172.21.15.31:0/216966609 conn(0x7fc928077640 msgr2=0x7fc92810ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:47.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.169+0000 7fc92fe00640 1 -- 172.21.15.31:0/216966609 shutdown_connections 2024-03-21T10:49:47.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.169+0000 7fc92fe00640 1 -- 172.21.15.31:0/216966609 wait complete. 2024-03-21T10:49:47.184 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:49:47.184 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-21T10:49:47.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.407+0000 7f6e73000640 1 -- 172.21.15.31:0/1419101118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e6c109830 msgr2=0x7f6e6c109c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:47.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.407+0000 7f6e73000640 1 --2- 172.21.15.31:0/1419101118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e6c109830 0x7f6e6c109c10 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c009c10 tx=0x7f6e5c02bef0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.407+0000 7f6e73000640 1 -- 172.21.15.31:0/1419101118 shutdown_connections 2024-03-21T10:49:47.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.407+0000 7f6e73000640 1 --2- 172.21.15.31:0/1419101118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e6c1041e0 0x7f6e6c1106f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.407+0000 7f6e73000640 1 --2- 172.21.15.31:0/1419101118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e6c103820 0x7f6e6c103ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.407+0000 7f6e73000640 1 --2- 172.21.15.31:0/1419101118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e6c109830 0x7f6e6c109c10 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.407+0000 7f6e73000640 1 -- 172.21.15.31:0/1419101118 >> 172.21.15.31:0/1419101118 conn(0x7f6e6c0ff310 msgr2=0x7f6e6c101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:47.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.407+0000 7f6e73000640 1 -- 172.21.15.31:0/1419101118 shutdown_connections 2024-03-21T10:49:47.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.407+0000 7f6e73000640 1 -- 172.21.15.31:0/1419101118 wait complete. 2024-03-21T10:49:47.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.408+0000 7f6e73000640 1 Processor -- start 2024-03-21T10:49:47.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.409+0000 7f6e73000640 1 -- start start 2024-03-21T10:49:47.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.409+0000 7f6e73000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e6c103820 0x7f6e6c19a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.409+0000 7f6e73000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e6c1041e0 0x7f6e6c19a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.409+0000 7f6e73000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e6c109830 0x7f6e6c19ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.409+0000 7f6e73000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c113c90 con 0x7f6e6c103820 2024-03-21T10:49:47.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.409+0000 7f6e73000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c113b10 con 0x7f6e6c1041e0 2024-03-21T10:49:47.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.409+0000 7f6e73000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c113e10 con 0x7f6e6c109830 2024-03-21T10:49:47.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.410+0000 7f6e71000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e6c109830 0x7f6e6c19ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.410+0000 7f6e63400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e6c1041e0 0x7f6e6c19a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.410+0000 7f6e63400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e6c1041e0 0x7f6e6c19a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33220/0 (socket says 172.21.15.31:33220) 2024-03-21T10:49:47.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.410+0000 7f6e63400640 1 -- 172.21.15.31:0/2013628095 learned_addr learned my addr 172.21.15.31:0/2013628095 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:47.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.410+0000 7f6e71000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e6c109830 0x7f6e6c19ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34538/0 (socket says 172.21.15.31:34538) 2024-03-21T10:49:47.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.410+0000 7f6e6be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e6c103820 0x7f6e6c19a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.410+0000 7f6e63400640 1 -- 172.21.15.31:0/2013628095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e6c109830 msgr2=0x7f6e6c19ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:47.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.410+0000 7f6e63400640 1 --2- 172.21.15.31:0/2013628095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e6c109830 0x7f6e6c19ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.410+0000 7f6e63400640 1 -- 172.21.15.31:0/2013628095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e6c103820 msgr2=0x7f6e6c19a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:47.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.410+0000 7f6e63400640 1 --2- 172.21.15.31:0/2013628095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e6c103820 0x7f6e6c19a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.410+0000 7f6e63400640 1 -- 172.21.15.31:0/2013628095 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6c19f180 con 0x7f6e6c1041e0 2024-03-21T10:49:47.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.410+0000 7f6e63400640 1 --2- 172.21.15.31:0/2013628095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e6c1041e0 0x7f6e6c19a6d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6e5400d950 tx=0x7f6e5400dd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:47.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.410+0000 7f6e69600640 1 -- 172.21.15.31:0/2013628095 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e54017030 con 0x7f6e6c1041e0 2024-03-21T10:49:47.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.411+0000 7f6e69600640 1 -- 172.21.15.31:0/2013628095 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e54004020 con 0x7f6e6c1041e0 2024-03-21T10:49:47.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.411+0000 7f6e69600640 1 -- 172.21.15.31:0/2013628095 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e54005650 con 0x7f6e6c1041e0 2024-03-21T10:49:47.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.411+0000 7f6e73000640 1 -- 172.21.15.31:0/2013628095 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e6c19f470 con 0x7f6e6c1041e0 2024-03-21T10:49:47.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.411+0000 7f6e73000640 1 -- 172.21.15.31:0/2013628095 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e6c1a6d00 con 0x7f6e6c1041e0 2024-03-21T10:49:47.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.414+0000 7f6e69600640 1 -- 172.21.15.31:0/2013628095 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f6e540041d0 con 0x7f6e6c1041e0 2024-03-21T10:49:47.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.414+0000 7f6e75d76740 1 Processor -- start 2024-03-21T10:49:47.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.414+0000 7f6e75d76740 1 -- start start 2024-03-21T10:49:47.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.414+0000 7f6e75d76740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da5334d6d0 0x55da5334dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.414+0000 7f6e75d76740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da5334e080 0x55da53356630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.414+0000 7f6e75d76740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da53356c00 0x55da53359050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.414+0000 7f6e75d76740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da532b12c0 con 0x55da5334e080 2024-03-21T10:49:47.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.414+0000 7f6e75d76740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da53259c90 con 0x55da5334d6d0 2024-03-21T10:49:47.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.414+0000 7f6e75d76740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da532ad360 con 0x55da53356c00 2024-03-21T10:49:47.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.415+0000 7f6e71000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da5334d6d0 0x55da5334dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.415+0000 7f6e63400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da53356c00 0x55da53359050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.415+0000 7f6e71000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da5334d6d0 0x55da5334dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33224/0 (socket says 172.21.15.31:33224) 2024-03-21T10:49:47.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.415+0000 7f6e71000640 1 -- 172.21.15.31:0/1997928688 learned_addr learned my addr 172.21.15.31:0/1997928688 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:47.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.415+0000 7f6e6be00640 1 --2- 172.21.15.31:0/1997928688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da5334e080 0x55da53356630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.415+0000 7f6e71000640 1 -- 172.21.15.31:0/1997928688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da53356c00 msgr2=0x55da53359050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:47.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.415+0000 7f6e71000640 1 --2- 172.21.15.31:0/1997928688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da53356c00 0x55da53359050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.415+0000 7f6e71000640 1 -- 172.21.15.31:0/1997928688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da5334e080 msgr2=0x55da53356630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:47.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.415+0000 7f6e71000640 1 --2- 172.21.15.31:0/1997928688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da5334e080 0x55da53356630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.415+0000 7f6e71000640 1 -- 172.21.15.31:0/1997928688 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da5327e080 con 0x55da5334d6d0 2024-03-21T10:49:47.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.415+0000 7f6e71000640 1 --2- 172.21.15.31:0/1997928688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da5334d6d0 0x55da5334dab0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6e64009be0 tx=0x7f6e64011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71686912b5f4936b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:47.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.415+0000 7f6e69600640 1 --2- 172.21.15.31:0/2013628095 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6e500a3110 0x7f6e500a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.416+0000 7f6e69600640 1 -- 172.21.15.31:0/2013628095 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f6e540cf4f0 con 0x7f6e6c1041e0 2024-03-21T10:49:47.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.416+0000 7f6e61600640 1 -- 172.21.15.31:0/1997928688 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6401e070 con 0x55da5334d6d0 2024-03-21T10:49:47.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.416+0000 7f6e61600640 1 -- 172.21.15.31:0/1997928688 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e64009e50 con 0x55da5334d6d0 2024-03-21T10:49:47.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.416+0000 7f6e61600640 1 -- 172.21.15.31:0/1997928688 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6400ea20 con 0x55da5334d6d0 2024-03-21T10:49:47.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.416+0000 7f6e6be00640 1 --2- 172.21.15.31:0/2013628095 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6e500a3110 0x7f6e500a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.416+0000 7f6e75d76740 1 -- 172.21.15.31:0/1997928688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da5334d6d0 msgr2=0x55da5334dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:47.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.416+0000 7f6e75d76740 1 --2- 172.21.15.31:0/1997928688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da5334d6d0 0x55da5334dab0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6e64009be0 tx=0x7f6e64011810 comp rx=0 tx=0).stop 2024-03-21T10:49:47.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.417+0000 7f6e75d76740 1 -- 172.21.15.31:0/1997928688 shutdown_connections 2024-03-21T10:49:47.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.417+0000 7f6e75d76740 1 --2- 172.21.15.31:0/1997928688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da53356c00 0x55da53359050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.417+0000 7f6e75d76740 1 --2- 172.21.15.31:0/1997928688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da5334e080 0x55da53356630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.417+0000 7f6e75d76740 1 --2- 172.21.15.31:0/1997928688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da5334d6d0 0x55da5334dab0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.417+0000 7f6e75d76740 1 -- 172.21.15.31:0/1997928688 >> 172.21.15.31:0/1997928688 conn(0x55da53347da0 msgr2=0x55da53348180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.417+0000 7f6e6be00640 1 --2- 172.21.15.31:0/2013628095 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6e500a3110 0x7f6e500a55d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c000f80 tx=0x7f6e5c0333e0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.417+0000 7f6e75d76740 1 -- 172.21.15.31:0/1997928688 shutdown_connections 2024-03-21T10:49:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.417+0000 7f6e75d76740 1 -- 172.21.15.31:0/1997928688 wait complete. 2024-03-21T10:49:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.417+0000 7f6e75d76740 1 -- start start 2024-03-21T10:49:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.417+0000 7f6e75d76740 1 Processor -- start 2024-03-21T10:49:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.418+0000 7f6e75d76740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da5334d6d0 0x55da53356510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.418+0000 7f6e75d76740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da5334e080 0x55da533538a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.418+0000 7f6e75d76740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da53356c00 0x55da53353e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.418+0000 7f6e75d76740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da53359b70 con 0x55da5334e080 2024-03-21T10:49:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.418+0000 7f6e75d76740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da5326d0d0 con 0x55da53356c00 2024-03-21T10:49:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.418+0000 7f6e75d76740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da53359cf0 con 0x55da5334d6d0 2024-03-21T10:49:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.418+0000 7f6e75d76740 10 client.0 _ll_register_callbacks cb 0x55da53271e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.418+0000 7f6e6be00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da5334d6d0 0x55da53356510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.418+0000 7f6e6be00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da5334d6d0 0x55da53356510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34568/0 (socket says 172.21.15.31:34568) 2024-03-21T10:49:47.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.418+0000 7f6e6be00640 1 -- 172.21.15.31:0/945892484 learned_addr learned my addr 172.21.15.31:0/945892484 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:47.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.418+0000 7f6e71000640 1 --2- 172.21.15.31:0/945892484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da53356c00 0x55da53353e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.418+0000 7f6e63400640 1 --2- 172.21.15.31:0/945892484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da5334e080 0x55da533538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.419+0000 7f6e6be00640 1 -- 172.21.15.31:0/945892484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da53356c00 msgr2=0x55da53353e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:47.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.419+0000 7f6e6be00640 1 --2- 172.21.15.31:0/945892484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da53356c00 0x55da53353e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.419+0000 7f6e6be00640 1 -- 172.21.15.31:0/945892484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da5334e080 msgr2=0x55da533538a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:47.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.419+0000 7f6e6be00640 1 --2- 172.21.15.31:0/945892484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da5334e080 0x55da533538a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.419+0000 7f6e6be00640 1 -- 172.21.15.31:0/945892484 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55da52fc53f0 con 0x55da5334d6d0 2024-03-21T10:49:47.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.419+0000 7f6e71000640 1 --2- 172.21.15.31:0/945892484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da53356c00 0x55da53353e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:47.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.419+0000 7f6e63400640 1 --2- 172.21.15.31:0/945892484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da5334e080 0x55da533538a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:49:47.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.419+0000 7f6e6be00640 1 --2- 172.21.15.31:0/945892484 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da5334d6d0 0x55da53356510 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c009370 tx=0x7f6e5c009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa3b63c2993a1987 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:47.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.419+0000 7f6e5a000640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:49:47.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.419+0000 7f6e5a000640 1 -- 172.21.15.31:0/945892484 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c051070 con 0x55da5334d6d0 2024-03-21T10:49:47.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.419+0000 7f6e5a000640 1 -- 172.21.15.31:0/945892484 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f6e5c04d450 con 0x55da5334d6d0 2024-03-21T10:49:47.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.419+0000 7f6e75d76740 1 -- 172.21.15.31:0/945892484 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6e64009e50 con 0x55da5334d6d0 2024-03-21T10:49:47.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.420+0000 7f6e5a000640 1 -- 172.21.15.31:0/945892484 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e5c05d050 con 0x55da5334d6d0 2024-03-21T10:49:47.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.420+0000 7f6e75d76740 10 client.28411 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:47.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.420+0000 7f6e75d76740 1 -- 172.21.15.31:0/945892484 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55da5308f7d0 con 0x55da5334d6d0 2024-03-21T10:49:47.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.420+0000 7f6e5a000640 1 -- 172.21.15.31:0/945892484 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e64009e50 con 0x55da5334d6d0 2024-03-21T10:49:47.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.420+0000 7f6e5a000640 1 -- 172.21.15.31:0/945892484 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f6e5c056ac0 con 0x55da5334d6d0 2024-03-21T10:49:47.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.420+0000 7f6e75d76740 10 client.28411 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:47.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.420+0000 7f6e75d76740 10 client.28411 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:49:47.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.420+0000 7f6e75d76740 20 client.28411 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:47.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.420+0000 7f6e75d76740 1 --2- 172.21.15.31:0/945892484 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55da533669d0 0x55da53368dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.420+0000 7f6e75d76740 4 client.28411 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:49:47.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.420+0000 7f6e75d76740 1 -- 172.21.15.31:0/945892484 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55da532be800 con 0x55da533669d0 2024-03-21T10:49:47.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.421+0000 7f6e63400640 1 --2- 172.21.15.31:0/945892484 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55da533669d0 0x55da53368dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.421+0000 7f6e63400640 1 --2- 172.21.15.31:0/945892484 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55da533669d0 0x55da53368dc0 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68b71d20f8d80232 server_cookie=839fcb82d156edc7 in_seq=0 out_seq=0 2024-03-21T10:49:47.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.421+0000 7f6e5a000640 10 client.28411 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:47.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.422+0000 7f6e5a000640 1 -- 172.21.15.31:0/945892484 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55da532be800 con 0x55da533669d0 2024-03-21T10:49:47.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.422+0000 7f6e5a000640 10 client.28411 handle_command_reply: tid=0 2024-03-21T10:49:47.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.422+0000 7f6e75d76740 1 client.28411 shutdown 2024-03-21T10:49:47.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.422+0000 7f6e75d76740 1 -- 172.21.15.31:0/945892484 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da5334d6d0 msgr2=0x55da53356510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:47.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.422+0000 7f6e75d76740 1 --2- 172.21.15.31:0/945892484 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da5334d6d0 0x55da53356510 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c009370 tx=0x7f6e5c009500 comp rx=0 tx=0).stop 2024-03-21T10:49:47.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.422+0000 7f6e75d76740 1 -- 172.21.15.31:0/945892484 shutdown_connections 2024-03-21T10:49:47.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.422+0000 7f6e75d76740 1 --2- 172.21.15.31:0/945892484 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55da533669d0 0x55da53368dc0 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.422+0000 7f6e75d76740 1 --2- 172.21.15.31:0/945892484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da53356c00 0x55da53353e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.422+0000 7f6e75d76740 1 --2- 172.21.15.31:0/945892484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da5334e080 0x55da533538a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.422+0000 7f6e75d76740 1 --2- 172.21.15.31:0/945892484 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da5334d6d0 0x55da53356510 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.422+0000 7f6e75d76740 1 -- 172.21.15.31:0/945892484 >> 172.21.15.31:0/945892484 conn(0x55da53347da0 msgr2=0x55da53349af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:47.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.422+0000 7f6e5a000640 0 client.28411 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:47.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.422+0000 7f6e75d76740 1 -- 172.21.15.31:0/945892484 shutdown_connections 2024-03-21T10:49:47.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.423+0000 7f6e75d76740 1 -- 172.21.15.31:0/945892484 wait complete. 2024-03-21T10:49:47.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.423+0000 7f6e75d76740 20 client.28411 trim_cache size 0 max 16384 2024-03-21T10:49:47.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.438+0000 7f6e75d76740 1 Processor -- start 2024-03-21T10:49:47.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.438+0000 7f6e75d76740 1 -- start start 2024-03-21T10:49:47.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.439+0000 7f6e75d76740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da5334d6d0 0x55da53346eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.439+0000 7f6e75d76740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da5334e080 0x55da53355fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.439+0000 7f6e75d76740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da53356c00 0x55da53356510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.439+0000 7f6e75d76740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c1b3110 con 0x55da5334d6d0 2024-03-21T10:49:47.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.439+0000 7f6e75d76740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c1b0c10 con 0x55da53356c00 2024-03-21T10:49:47.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.439+0000 7f6e75d76740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c1aecd0 con 0x55da5334e080 2024-03-21T10:49:47.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.439+0000 7f6e63400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da5334e080 0x55da53355fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.439+0000 7f6e63400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da5334e080 0x55da53355fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34572/0 (socket says 172.21.15.31:34572) 2024-03-21T10:49:47.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.439+0000 7f6e63400640 1 -- 172.21.15.31:0/1719276442 learned_addr learned my addr 172.21.15.31:0/1719276442 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:47.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.439+0000 7f6e71000640 1 --2- 172.21.15.31:0/1719276442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da53356c00 0x55da53356510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.439+0000 7f6e6be00640 1 --2- 172.21.15.31:0/1719276442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da5334d6d0 0x55da53346eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.440+0000 7f6e63400640 1 -- 172.21.15.31:0/1719276442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da53356c00 msgr2=0x55da53356510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:47.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.440+0000 7f6e63400640 1 --2- 172.21.15.31:0/1719276442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da53356c00 0x55da53356510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.440+0000 7f6e63400640 1 -- 172.21.15.31:0/1719276442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da5334d6d0 msgr2=0x55da53346eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:47.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.440+0000 7f6e63400640 1 --2- 172.21.15.31:0/1719276442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da5334d6d0 0x55da53346eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.440+0000 7f6e63400640 1 -- 172.21.15.31:0/1719276442 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da533541b0 con 0x55da5334e080 2024-03-21T10:49:47.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.440+0000 7f6e6be00640 1 --2- 172.21.15.31:0/1719276442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da5334d6d0 0x55da53346eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:49:47.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.440+0000 7f6e63400640 1 --2- 172.21.15.31:0/1719276442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da5334e080 0x55da53355fd0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f6e54009610 tx=0x7f6e540d5060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25a1a8ab9a0b2df9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:47.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.440+0000 7f6e60c00640 1 -- 172.21.15.31:0/1719276442 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5409b020 con 0x55da5334e080 2024-03-21T10:49:47.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.440+0000 7f6e60c00640 1 -- 172.21.15.31:0/1719276442 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e5409bbe0 con 0x55da5334e080 2024-03-21T10:49:47.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.440+0000 7f6e60c00640 1 -- 172.21.15.31:0/1719276442 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5409ae40 con 0x55da5334e080 2024-03-21T10:49:47.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.441+0000 7f6e75d76740 1 -- 172.21.15.31:0/1719276442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da5334e080 msgr2=0x55da53355fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:47.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.441+0000 7f6e75d76740 1 --2- 172.21.15.31:0/1719276442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da5334e080 0x55da53355fd0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f6e54009610 tx=0x7f6e540d5060 comp rx=0 tx=0).stop 2024-03-21T10:49:47.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.441+0000 7f6e75d76740 1 -- 172.21.15.31:0/1719276442 shutdown_connections 2024-03-21T10:49:47.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.441+0000 7f6e75d76740 1 --2- 172.21.15.31:0/1719276442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da53356c00 0x55da53356510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.441+0000 7f6e75d76740 1 --2- 172.21.15.31:0/1719276442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da5334e080 0x55da53355fd0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.441+0000 7f6e75d76740 1 --2- 172.21.15.31:0/1719276442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da5334d6d0 0x55da53346eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.441+0000 7f6e75d76740 1 -- 172.21.15.31:0/1719276442 >> 172.21.15.31:0/1719276442 conn(0x55da53347da0 msgr2=0x55da533490a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:47.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.441+0000 7f6e75d76740 1 -- 172.21.15.31:0/1719276442 shutdown_connections 2024-03-21T10:49:47.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.441+0000 7f6e75d76740 1 -- 172.21.15.31:0/1719276442 wait complete. 2024-03-21T10:49:47.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.442+0000 7f6e75d76740 1 -- start start 2024-03-21T10:49:47.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.442+0000 7f6e75d76740 1 Processor -- start 2024-03-21T10:49:47.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.442+0000 7f6e75d76740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da5334d6d0 0x55da5335e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.442+0000 7f6e75d76740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da5334e080 0x55da5335ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.442+0000 7f6e75d76740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da53356c00 0x55da5335cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.442+0000 7f6e75d76740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c1af2d0 con 0x55da5334d6d0 2024-03-21T10:49:47.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.442+0000 7f6e75d76740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c1b05b0 con 0x55da53356c00 2024-03-21T10:49:47.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.442+0000 7f6e75d76740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c1af8f0 con 0x55da5334e080 2024-03-21T10:49:47.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.442+0000 7f6e75d76740 10 client.0 _ll_register_callbacks cb 0x7f6e5c0092c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:47.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.442+0000 7f6e6be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da5334d6d0 0x55da5335e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.442+0000 7f6e71000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da53356c00 0x55da5335cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.442+0000 7f6e63400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da5334e080 0x55da5335ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.443+0000 7f6e6be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da5334d6d0 0x55da5335e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40050/0 (socket says 172.21.15.31:40050) 2024-03-21T10:49:47.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.443+0000 7f6e6be00640 1 -- 172.21.15.31:0/2933695516 learned_addr learned my addr 172.21.15.31:0/2933695516 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:47.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.443+0000 7f6e63400640 1 -- 172.21.15.31:0/2933695516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da53356c00 msgr2=0x55da5335cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:47.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.443+0000 7f6e63400640 1 --2- 172.21.15.31:0/2933695516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da53356c00 0x55da5335cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.443+0000 7f6e63400640 1 -- 172.21.15.31:0/2933695516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da5334d6d0 msgr2=0x55da5335e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:47.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.443+0000 7f6e63400640 1 --2- 172.21.15.31:0/2933695516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da5334d6d0 0x55da5335e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.443+0000 7f6e63400640 1 -- 172.21.15.31:0/2933695516 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55da5334de20 con 0x55da5334e080 2024-03-21T10:49:47.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.443+0000 7f6e63400640 1 --2- 172.21.15.31:0/2933695516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da5334e080 0x55da5335ee30 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x55da53352080 tx=0x7f6e540d2d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa370c8a5f573b53 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:47.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.443+0000 7f6e5a000640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:49:47.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.443+0000 7f6e5a000640 1 -- 172.21.15.31:0/2933695516 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e54084070 con 0x55da5334e080 2024-03-21T10:49:47.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.443+0000 7f6e5a000640 1 -- 172.21.15.31:0/2933695516 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f6e540ccac0 con 0x55da5334e080 2024-03-21T10:49:47.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.443+0000 7f6e75d76740 1 -- 172.21.15.31:0/2933695516 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6e5409bbe0 con 0x55da5334e080 2024-03-21T10:49:47.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.444+0000 7f6e5a000640 1 -- 172.21.15.31:0/2933695516 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e5409ae40 con 0x55da5334e080 2024-03-21T10:49:47.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.444+0000 7f6e75d76740 10 client.28417 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:47.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.444+0000 7f6e75d76740 1 -- 172.21.15.31:0/2933695516 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55da533c2c70 con 0x55da5334e080 2024-03-21T10:49:47.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.444+0000 7f6e5a000640 1 -- 172.21.15.31:0/2933695516 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e5409bbe0 con 0x55da5334e080 2024-03-21T10:49:47.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.444+0000 7f6e5a000640 1 -- 172.21.15.31:0/2933695516 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f6e5409f130 con 0x55da5334e080 2024-03-21T10:49:47.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.444+0000 7f6e75d76740 10 client.28417 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:47.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.444+0000 7f6e75d76740 10 client.28417 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:49:47.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.444+0000 7f6e75d76740 20 client.28417 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:47.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.444+0000 7f6e75d76740 1 --2- 172.21.15.31:0/2933695516 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55da533c3310 0x55da533c5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:47.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.444+0000 7f6e75d76740 4 client.28417 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:49:47.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.444+0000 7f6e75d76740 1 -- 172.21.15.31:0/2933695516 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55da533475b0 con 0x55da533c3310 2024-03-21T10:49:47.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.444+0000 7f6e6be00640 1 --2- 172.21.15.31:0/2933695516 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55da533c3310 0x55da533c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:47.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.445+0000 7f6e6be00640 1 --2- 172.21.15.31:0/2933695516 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55da533c3310 0x55da533c5730 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=434a17a55d859050 server_cookie=cf70456769fb6bb0 in_seq=0 out_seq=0 2024-03-21T10:49:47.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.445+0000 7f6e5a000640 10 client.28417 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:47.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.445+0000 7f6e5a000640 1 -- 172.21.15.31:0/2933695516 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55da533475b0 con 0x55da533c3310 2024-03-21T10:49:47.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.445+0000 7f6e5a000640 10 client.28417 handle_command_reply: tid=0 2024-03-21T10:49:47.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.445+0000 7f6e75d76740 1 client.28417 shutdown 2024-03-21T10:49:47.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.445+0000 7f6e75d76740 1 -- 172.21.15.31:0/2933695516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da5334e080 msgr2=0x55da5335ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:47.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.445+0000 7f6e75d76740 1 --2- 172.21.15.31:0/2933695516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da5334e080 0x55da5335ee30 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x55da53352080 tx=0x7f6e540d2d10 comp rx=0 tx=0).stop 2024-03-21T10:49:47.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.445+0000 7f6e75d76740 1 -- 172.21.15.31:0/2933695516 shutdown_connections 2024-03-21T10:49:47.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.445+0000 7f6e75d76740 1 --2- 172.21.15.31:0/2933695516 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55da533c3310 0x55da533c5730 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.445+0000 7f6e75d76740 1 --2- 172.21.15.31:0/2933695516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da53356c00 0x55da5335cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.445+0000 7f6e75d76740 1 --2- 172.21.15.31:0/2933695516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da5334e080 0x55da5335ee30 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.445+0000 7f6e5a000640 0 client.28417 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:47.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.445+0000 7f6e75d76740 1 --2- 172.21.15.31:0/2933695516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da5334d6d0 0x55da5335e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.445+0000 7f6e75d76740 1 -- 172.21.15.31:0/2933695516 >> 172.21.15.31:0/2933695516 conn(0x55da53347da0 msgr2=0x55da53346d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:47.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.445+0000 7f6e75d76740 1 -- 172.21.15.31:0/2933695516 shutdown_connections 2024-03-21T10:49:47.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.446+0000 7f6e75d76740 1 -- 172.21.15.31:0/2933695516 wait complete. 2024-03-21T10:49:47.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.446+0000 7f6e75d76740 20 client.28417 trim_cache size 0 max 16384 2024-03-21T10:49:47.448 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:49:47.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.446+0000 7f6e73000640 1 -- 172.21.15.31:0/2013628095 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6e500a3110 msgr2=0x7f6e500a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:47.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.446+0000 7f6e73000640 1 --2- 172.21.15.31:0/2013628095 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6e500a3110 0x7f6e500a55d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c000f80 tx=0x7f6e5c0333e0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.446+0000 7f6e73000640 1 -- 172.21.15.31:0/2013628095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e6c1041e0 msgr2=0x7f6e6c19a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:47.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.446+0000 7f6e73000640 1 --2- 172.21.15.31:0/2013628095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e6c1041e0 0x7f6e6c19a6d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6e5400d950 tx=0x7f6e5400dd50 comp rx=0 tx=0).stop 2024-03-21T10:49:47.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.447+0000 7f6e73000640 1 -- 172.21.15.31:0/2013628095 shutdown_connections 2024-03-21T10:49:47.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.447+0000 7f6e73000640 1 --2- 172.21.15.31:0/2013628095 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f6e500a3110 0x7f6e500a55d0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.447+0000 7f6e73000640 1 --2- 172.21.15.31:0/2013628095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e6c109830 0x7f6e6c19ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.447+0000 7f6e73000640 1 --2- 172.21.15.31:0/2013628095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e6c1041e0 0x7f6e6c19a6d0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.447+0000 7f6e73000640 1 --2- 172.21.15.31:0/2013628095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e6c103820 0x7f6e6c19a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:47.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.447+0000 7f6e73000640 1 -- 172.21.15.31:0/2013628095 >> 172.21.15.31:0/2013628095 conn(0x7f6e6c0ff310 msgr2=0x7f6e6c0ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:47.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.447+0000 7f6e73000640 1 -- 172.21.15.31:0/2013628095 shutdown_connections 2024-03-21T10:49:47.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:47.447+0000 7f6e73000640 1 -- 172.21.15.31:0/2013628095 wait complete. 2024-03-21T10:49:47.460 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:47 smithi031 ceph-mon[31673]: pgmap v349: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 3.0 KiB/s wr, 8 op/s 2024-03-21T10:49:47.461 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:47 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1367492836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:49:47.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:47 smithi040 ceph-mon[36154]: pgmap v349: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 3.0 KiB/s wr, 8 op/s 2024-03-21T10:49:47.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:47 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1367492836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:49:47.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:47 smithi070 ceph-mon[36330]: pgmap v349: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 3.0 KiB/s wr, 8 op/s 2024-03-21T10:49:47.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:47 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1367492836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:49:48.465 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-21T10:49:48.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:49:48.465 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-21T10:49:48.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.735+0000 7f00b1000640 1 -- 172.21.15.31:0/3065273156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00ac10c980 msgr2=0x7f00ac10cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:48.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.735+0000 7f00b1000640 1 --2- 172.21.15.31:0/3065273156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00ac10c980 0x7f00ac10cd60 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f009c009900 tx=0x7f009c02bcc0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.735+0000 7f00b1000640 1 -- 172.21.15.31:0/3065273156 shutdown_connections 2024-03-21T10:49:48.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.735+0000 7f00b1000640 1 --2- 172.21.15.31:0/3065273156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00ac108440 0x7f00ac114900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.735+0000 7f00b1000640 1 --2- 172.21.15.31:0/3065273156 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ac107aa0 0x7f00ac107f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.735+0000 7f00b1000640 1 --2- 172.21.15.31:0/3065273156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00ac10c980 0x7f00ac10cd60 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.735+0000 7f00b1000640 1 -- 172.21.15.31:0/3065273156 >> 172.21.15.31:0/3065273156 conn(0x7f00ac0772a0 msgr2=0x7f00ac0796c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:48.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.735+0000 7f00b1000640 1 -- 172.21.15.31:0/3065273156 shutdown_connections 2024-03-21T10:49:48.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.735+0000 7f00b1000640 1 -- 172.21.15.31:0/3065273156 wait complete. 2024-03-21T10:49:48.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.736+0000 7f00b1000640 1 Processor -- start 2024-03-21T10:49:48.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.736+0000 7f00b1000640 1 -- start start 2024-03-21T10:49:48.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.737+0000 7f00b1000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ac107aa0 0x7f00ac19e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:48.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.737+0000 7f00b1000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00ac108440 0x7f00ac19e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:48.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.737+0000 7f00b1000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00ac10c980 0x7f00ac1a2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:48.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.737+0000 7f00b1000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac117ec0 con 0x7f00ac107aa0 2024-03-21T10:49:48.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.737+0000 7f00b1000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac117d40 con 0x7f00ac10c980 2024-03-21T10:49:48.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.737+0000 7f00b1000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac118040 con 0x7f00ac108440 2024-03-21T10:49:48.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.737+0000 7f00aaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00ac108440 0x7f00ac19e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:48.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.737+0000 7f00ab400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ac107aa0 0x7f00ac19e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:48.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.737+0000 7f00aaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00ac108440 0x7f00ac19e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34604/0 (socket says 172.21.15.31:34604) 2024-03-21T10:49:48.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.737+0000 7f00aaa00640 1 -- 172.21.15.31:0/224910847 learned_addr learned my addr 172.21.15.31:0/224910847 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:48.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.737+0000 7f00abe00640 1 --2- 172.21.15.31:0/224910847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00ac10c980 0x7f00ac1a2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:48.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.738+0000 7f00aaa00640 1 -- 172.21.15.31:0/224910847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00ac10c980 msgr2=0x7f00ac1a2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:48.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.738+0000 7f00aaa00640 1 --2- 172.21.15.31:0/224910847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00ac10c980 0x7f00ac1a2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.738+0000 7f00aaa00640 1 -- 172.21.15.31:0/224910847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ac107aa0 msgr2=0x7f00ac19e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:48.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.738+0000 7f00aaa00640 1 --2- 172.21.15.31:0/224910847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ac107aa0 0x7f00ac19e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.738+0000 7f00aaa00640 1 -- 172.21.15.31:0/224910847 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ac1a3350 con 0x7f00ac108440 2024-03-21T10:49:48.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.738+0000 7f00ab400640 1 --2- 172.21.15.31:0/224910847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ac107aa0 0x7f00ac19e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:48.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.738+0000 7f00aaa00640 1 --2- 172.21.15.31:0/224910847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00ac108440 0x7f00ac19e8a0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f009800ea60 tx=0x7f009800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:48.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.738+0000 7f00a3e00640 1 -- 172.21.15.31:0/224910847 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009800cde0 con 0x7f00ac108440 2024-03-21T10:49:48.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.738+0000 7f00a3e00640 1 -- 172.21.15.31:0/224910847 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00980035f0 con 0x7f00ac108440 2024-03-21T10:49:48.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.738+0000 7f00a3e00640 1 -- 172.21.15.31:0/224910847 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0098010670 con 0x7f00ac108440 2024-03-21T10:49:48.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.738+0000 7f00b1000640 1 -- 172.21.15.31:0/224910847 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00ac1a3640 con 0x7f00ac108440 2024-03-21T10:49:48.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.738+0000 7f00b1000640 1 -- 172.21.15.31:0/224910847 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00ac1aaed0 con 0x7f00ac108440 2024-03-21T10:49:48.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.742+0000 7f00b520c740 1 Processor -- start 2024-03-21T10:49:48.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.742+0000 7f00b520c740 1 -- start start 2024-03-21T10:49:48.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.742+0000 7f00a3e00640 1 -- 172.21.15.31:0/224910847 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f00980037a0 con 0x7f00ac108440 2024-03-21T10:49:48.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.742+0000 7f00b520c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55878bd06290 0x55878bd06670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:48.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.742+0000 7f00b520c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55878bd06bb0 0x55878bd0f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:48.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.742+0000 7f00b520c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55878bd0f790 0x55878bd11b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:48.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.742+0000 7f00b520c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55878bc6aeb0 con 0x55878bd0f790 2024-03-21T10:49:48.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.742+0000 7f00b520c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55878bc13a60 con 0x55878bd06bb0 2024-03-21T10:49:48.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.742+0000 7f00b520c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55878bc66f50 con 0x55878bd06290 2024-03-21T10:49:48.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.742+0000 7f00ab400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55878bd06bb0 0x55878bd0f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:48.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.742+0000 7f00abe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55878bd06290 0x55878bd06670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:48.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.743+0000 7f00abe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55878bd06290 0x55878bd06670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34614/0 (socket says 172.21.15.31:34614) 2024-03-21T10:49:48.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.743+0000 7f00abe00640 1 -- 172.21.15.31:0/917118634 learned_addr learned my addr 172.21.15.31:0/917118634 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:48.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.743+0000 7f00aaa00640 1 --2- 172.21.15.31:0/917118634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55878bd0f790 0x55878bd11b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:48.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.743+0000 7f00abe00640 1 -- 172.21.15.31:0/917118634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55878bd06bb0 msgr2=0x55878bd0f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:48.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.743+0000 7f00abe00640 1 --2- 172.21.15.31:0/917118634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55878bd06bb0 0x55878bd0f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.743+0000 7f00abe00640 1 -- 172.21.15.31:0/917118634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55878bd0f790 msgr2=0x55878bd11b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:48.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.743+0000 7f00abe00640 1 --2- 172.21.15.31:0/917118634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55878bd0f790 0x55878bd11b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.743+0000 7f00abe00640 1 -- 172.21.15.31:0/917118634 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55878ba477e0 con 0x55878bd06290 2024-03-21T10:49:48.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.743+0000 7f00abe00640 1 --2- 172.21.15.31:0/917118634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55878bd06290 0x55878bd06670 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009f50 tx=0x7f00a40118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbe7ab4a5da27790 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:48.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.744+0000 7f00a0c00640 1 -- 172.21.15.31:0/917118634 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a4004d20 con 0x55878bd06290 2024-03-21T10:49:48.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.744+0000 7f00a0c00640 1 -- 172.21.15.31:0/917118634 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00a4004050 con 0x55878bd06290 2024-03-21T10:49:48.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.744+0000 7f00a0c00640 1 -- 172.21.15.31:0/917118634 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a4005280 con 0x55878bd06290 2024-03-21T10:49:48.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.744+0000 7f00a3e00640 1 --2- 172.21.15.31:0/224910847 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f00900a3110 0x7f00900a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:48.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.744+0000 7f00a3e00640 1 -- 172.21.15.31:0/224910847 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f00980cfe40 con 0x7f00ac108440 2024-03-21T10:49:48.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.744+0000 7f00ab400640 1 --2- 172.21.15.31:0/224910847 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f00900a3110 0x7f00900a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:48.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.745+0000 7f00b520c740 1 -- 172.21.15.31:0/917118634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55878bd06290 msgr2=0x55878bd06670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:48.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.745+0000 7f00b520c740 1 --2- 172.21.15.31:0/917118634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55878bd06290 0x55878bd06670 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009f50 tx=0x7f00a40118d0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.745+0000 7f00b520c740 1 -- 172.21.15.31:0/917118634 shutdown_connections 2024-03-21T10:49:48.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.745+0000 7f00b520c740 1 --2- 172.21.15.31:0/917118634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55878bd0f790 0x55878bd11b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.745+0000 7f00b520c740 1 --2- 172.21.15.31:0/917118634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55878bd06bb0 0x55878bd0f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.745+0000 7f00b520c740 1 --2- 172.21.15.31:0/917118634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55878bd06290 0x55878bd06670 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.745+0000 7f00b520c740 1 -- 172.21.15.31:0/917118634 >> 172.21.15.31:0/917118634 conn(0x55878bd009c0 msgr2=0x55878bd00da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:48.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.745+0000 7f00b520c740 1 -- 172.21.15.31:0/917118634 shutdown_connections 2024-03-21T10:49:48.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.745+0000 7f00b520c740 1 -- 172.21.15.31:0/917118634 wait complete. 2024-03-21T10:49:48.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.745+0000 7f00ab400640 1 --2- 172.21.15.31:0/224910847 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f00900a3110 0x7f00900a55d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f009c005ab0 tx=0x7f009c02e040 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:48.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.746+0000 7f00b520c740 1 -- start start 2024-03-21T10:49:48.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.746+0000 7f00b520c740 1 Processor -- start 2024-03-21T10:49:48.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.746+0000 7f00b520c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55878bd06290 0x55878bd0f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:48.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.746+0000 7f00b520c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55878bd06bb0 0x55878bd0c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:48.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.746+0000 7f00b520c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55878bd0f790 0x55878bd0c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:48.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.746+0000 7f00b520c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55878bd12580 con 0x55878bd06290 2024-03-21T10:49:48.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.746+0000 7f00b520c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55878bc26eb0 con 0x55878bd0f790 2024-03-21T10:49:48.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.746+0000 7f00b520c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55878bd12700 con 0x55878bd06bb0 2024-03-21T10:49:48.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.747+0000 7f00b520c740 10 client.0 _ll_register_callbacks cb 0x55878bc1b150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:48.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.747+0000 7f00aaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55878bd06bb0 0x55878bd0c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:48.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.747+0000 7f00aaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55878bd06bb0 0x55878bd0c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34622/0 (socket says 172.21.15.31:34622) 2024-03-21T10:49:48.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.747+0000 7f00abe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55878bd0f790 0x55878bd0c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:48.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.747+0000 7f00aaa00640 1 -- 172.21.15.31:0/240023329 learned_addr learned my addr 172.21.15.31:0/240023329 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:48.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.747+0000 7f00ab400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55878bd06290 0x55878bd0f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:48.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.747+0000 7f00aaa00640 1 -- 172.21.15.31:0/240023329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55878bd0f790 msgr2=0x55878bd0c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:48.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.747+0000 7f00aaa00640 1 --2- 172.21.15.31:0/240023329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55878bd0f790 0x55878bd0c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.747+0000 7f00aaa00640 1 -- 172.21.15.31:0/240023329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55878bd06290 msgr2=0x55878bd0f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:48.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.747+0000 7f00aaa00640 1 --2- 172.21.15.31:0/240023329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55878bd06290 0x55878bd0f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.747+0000 7f00aaa00640 1 -- 172.21.15.31:0/240023329 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55878baa7f70 con 0x55878bd06bb0 2024-03-21T10:49:48.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.747+0000 7f00ab400640 1 --2- 172.21.15.31:0/240023329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55878bd06290 0x55878bd0f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:49:48.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.747+0000 7f00aaa00640 1 --2- 172.21.15.31:0/240023329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55878bd06bb0 0x55878bd0c480 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f009801f000 tx=0x7f00980095a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cb97cbef8759464 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:48.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.748+0000 7f0095600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:49:48.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.748+0000 7f0095600640 1 -- 172.21.15.31:0/240023329 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009809b790 con 0x55878bd06bb0 2024-03-21T10:49:48.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.748+0000 7f0095600640 1 -- 172.21.15.31:0/240023329 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f00980cd410 con 0x55878bd06bb0 2024-03-21T10:49:48.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.748+0000 7f00b520c740 1 -- 172.21.15.31:0/240023329 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f00a4004050 con 0x55878bd06bb0 2024-03-21T10:49:48.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.748+0000 7f0095600640 1 -- 172.21.15.31:0/240023329 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00980cd610 con 0x55878bd06bb0 2024-03-21T10:49:48.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.748+0000 7f00b520c740 10 client.28429 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:48.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.748+0000 7f00b520c740 1 -- 172.21.15.31:0/240023329 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55878b93f170 con 0x55878bd06bb0 2024-03-21T10:49:48.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.748+0000 7f0095600640 1 -- 172.21.15.31:0/240023329 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00a4004050 con 0x55878bd06bb0 2024-03-21T10:49:48.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.748+0000 7f0095600640 1 -- 172.21.15.31:0/240023329 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f00980baba0 con 0x55878bd06bb0 2024-03-21T10:49:48.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.748+0000 7f00b520c740 10 client.28429 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:48.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.748+0000 7f00b520c740 10 client.28429 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:49:48.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.749+0000 7f00b520c740 20 client.28429 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:48.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.749+0000 7f00b520c740 1 --2- 172.21.15.31:0/240023329 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55878bd1f840 0x55878bd21c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:48.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.749+0000 7f00b520c740 4 client.28429 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:49:48.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.749+0000 7f00b520c740 1 -- 172.21.15.31:0/240023329 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55878bc783f0 con 0x55878bd1f840 2024-03-21T10:49:48.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.749+0000 7f00ab400640 1 --2- 172.21.15.31:0/240023329 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55878bd1f840 0x55878bd21c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:48.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.749+0000 7f00ab400640 1 --2- 172.21.15.31:0/240023329 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55878bd1f840 0x55878bd21c30 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c8c2a837a46fa0a server_cookie=b21d1da7629a0ccf in_seq=0 out_seq=0 2024-03-21T10:49:48.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.749+0000 7f0095600640 10 client.28429 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:48.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.750+0000 7f0095600640 1 -- 172.21.15.31:0/240023329 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55878bc783f0 con 0x55878bd1f840 2024-03-21T10:49:48.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.750+0000 7f0095600640 10 client.28429 handle_command_reply: tid=0 2024-03-21T10:49:48.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.750+0000 7f00b520c740 1 client.28429 shutdown 2024-03-21T10:49:48.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.750+0000 7f00b520c740 1 -- 172.21.15.31:0/240023329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55878bd06bb0 msgr2=0x55878bd0c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:48.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.750+0000 7f00b520c740 1 --2- 172.21.15.31:0/240023329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55878bd06bb0 0x55878bd0c480 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f009801f000 tx=0x7f00980095a0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.750+0000 7f00b520c740 1 -- 172.21.15.31:0/240023329 shutdown_connections 2024-03-21T10:49:48.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.750+0000 7f00b520c740 1 --2- 172.21.15.31:0/240023329 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55878bd1f840 0x55878bd21c30 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.750+0000 7f00b520c740 1 --2- 172.21.15.31:0/240023329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55878bd0f790 0x55878bd0c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.750+0000 7f00b520c740 1 --2- 172.21.15.31:0/240023329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55878bd06bb0 0x55878bd0c480 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.750+0000 7f0095600640 0 client.28429 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:48.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.750+0000 7f00b520c740 1 --2- 172.21.15.31:0/240023329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55878bd06290 0x55878bd0f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.750+0000 7f00b520c740 1 -- 172.21.15.31:0/240023329 >> 172.21.15.31:0/240023329 conn(0x55878bd009c0 msgr2=0x55878bd02710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:48.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.751+0000 7f00b520c740 1 -- 172.21.15.31:0/240023329 shutdown_connections 2024-03-21T10:49:48.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.751+0000 7f00b520c740 1 -- 172.21.15.31:0/240023329 wait complete. 2024-03-21T10:49:48.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.751+0000 7f00b520c740 20 client.28429 trim_cache size 0 max 16384 2024-03-21T10:49:48.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.768+0000 7f00b520c740 1 Processor -- start 2024-03-21T10:49:48.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.769+0000 7f00b520c740 1 -- start start 2024-03-21T10:49:48.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.769+0000 7f00b520c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55878bd06290 0x55878bd0ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:48.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.769+0000 7f00b520c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55878bd06bb0 0x55878bd0f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:48.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.769+0000 7f00b520c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55878bd0f790 0x55878bd023d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:48.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.769+0000 7f00b520c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac1b7250 con 0x55878bd06bb0 2024-03-21T10:49:48.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.769+0000 7f00b520c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac1b4d50 con 0x55878bd06290 2024-03-21T10:49:48.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.769+0000 7f00b520c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac1b2e10 con 0x55878bd0f790 2024-03-21T10:49:48.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.769+0000 7f00aaa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55878bd06bb0 0x55878bd0f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:48.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.769+0000 7f00ab400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55878bd06290 0x55878bd0ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:48.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.769+0000 7f00abe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55878bd0f790 0x55878bd023d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:48.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.769+0000 7f00ab400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55878bd06290 0x55878bd0ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33290/0 (socket says 172.21.15.31:33290) 2024-03-21T10:49:48.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.769+0000 7f00ab400640 1 -- 172.21.15.31:0/44024009 learned_addr learned my addr 172.21.15.31:0/44024009 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:48.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.770+0000 7f00ab400640 1 -- 172.21.15.31:0/44024009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55878bd0f790 msgr2=0x55878bd023d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:48.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.770+0000 7f00ab400640 1 --2- 172.21.15.31:0/44024009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55878bd0f790 0x55878bd023d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.770+0000 7f00ab400640 1 -- 172.21.15.31:0/44024009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55878bd06bb0 msgr2=0x55878bd0f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:48.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.770+0000 7f00ab400640 1 --2- 172.21.15.31:0/44024009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55878bd06bb0 0x55878bd0f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.770+0000 7f00ab400640 1 -- 172.21.15.31:0/44024009 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55878bd22290 con 0x55878bd06290 2024-03-21T10:49:48.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.770+0000 7f00ab400640 1 --2- 172.21.15.31:0/44024009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55878bd06290 0x55878bd0ebb0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f009c004130 tx=0x7f009c041000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61d02881cae7a682 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:48.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.770+0000 7f0097e00640 1 -- 172.21.15.31:0/44024009 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c02c910 con 0x55878bd06290 2024-03-21T10:49:48.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.770+0000 7f0097e00640 1 -- 172.21.15.31:0/44024009 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f009c05b070 con 0x55878bd06290 2024-03-21T10:49:48.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.770+0000 7f0097e00640 1 -- 172.21.15.31:0/44024009 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c005630 con 0x55878bd06290 2024-03-21T10:49:48.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.771+0000 7f00b520c740 1 -- 172.21.15.31:0/44024009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55878bd06290 msgr2=0x55878bd0ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:48.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.771+0000 7f00b520c740 1 --2- 172.21.15.31:0/44024009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55878bd06290 0x55878bd0ebb0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f009c004130 tx=0x7f009c041000 comp rx=0 tx=0).stop 2024-03-21T10:49:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.771+0000 7f00b520c740 1 -- 172.21.15.31:0/44024009 shutdown_connections 2024-03-21T10:49:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.771+0000 7f00b520c740 1 --2- 172.21.15.31:0/44024009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55878bd0f790 0x55878bd023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.771+0000 7f00b520c740 1 --2- 172.21.15.31:0/44024009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55878bd06bb0 0x55878bd0f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.771+0000 7f00b520c740 1 --2- 172.21.15.31:0/44024009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55878bd06290 0x55878bd0ebb0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.771+0000 7f00b520c740 1 -- 172.21.15.31:0/44024009 >> 172.21.15.31:0/44024009 conn(0x55878bd009c0 msgr2=0x55878bd01cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.771+0000 7f00b520c740 1 -- 172.21.15.31:0/44024009 shutdown_connections 2024-03-21T10:49:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.771+0000 7f00b520c740 1 -- 172.21.15.31:0/44024009 wait complete. 2024-03-21T10:49:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.771+0000 7f00b520c740 1 -- start start 2024-03-21T10:49:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.772+0000 7f00b520c740 1 Processor -- start 2024-03-21T10:49:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.772+0000 7f00b520c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55878bd06290 0x55878bd14920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.772+0000 7f00b520c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55878bd06bb0 0x55878bd14e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.772+0000 7f00b520c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55878bd0f790 0x55878bd153a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.772+0000 7f00b520c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac1b3410 con 0x55878bd0f790 2024-03-21T10:49:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.772+0000 7f00b520c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac1b46f0 con 0x55878bd06bb0 2024-03-21T10:49:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.772+0000 7f00b520c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac1b3a30 con 0x55878bd06290 2024-03-21T10:49:48.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.772+0000 7f00b520c740 10 client.0 _ll_register_callbacks cb 0x7f0098003e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:48.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.772+0000 7f00ab400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55878bd06290 0x55878bd14920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:48.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.772+0000 7f00aaa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55878bd06bb0 0x55878bd14e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:48.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.772+0000 7f00abe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55878bd0f790 0x55878bd153a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:48.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.772+0000 7f00ab400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55878bd06290 0x55878bd14920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34640/0 (socket says 172.21.15.31:34640) 2024-03-21T10:49:48.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.772+0000 7f00ab400640 1 -- 172.21.15.31:0/2085486043 learned_addr learned my addr 172.21.15.31:0/2085486043 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:48.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.773+0000 7f00ab400640 1 -- 172.21.15.31:0/2085486043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55878bd06bb0 msgr2=0x55878bd14e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:48.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.773+0000 7f00ab400640 1 --2- 172.21.15.31:0/2085486043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55878bd06bb0 0x55878bd14e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.773+0000 7f00ab400640 1 -- 172.21.15.31:0/2085486043 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55878bd0f790 msgr2=0x55878bd153a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:48.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.773+0000 7f00ab400640 1 --2- 172.21.15.31:0/2085486043 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55878bd0f790 0x55878bd153a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.773+0000 7f00ab400640 1 -- 172.21.15.31:0/2085486043 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55878bd7b190 con 0x55878bd06290 2024-03-21T10:49:48.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.773+0000 7f00aaa00640 1 --2- 172.21.15.31:0/2085486043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55878bd06bb0 0x55878bd14e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:48.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.773+0000 7f00ab400640 1 --2- 172.21.15.31:0/2085486043 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55878bd06290 0x55878bd14920 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f009c058040 tx=0x7f009c005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b66fb836e7922d39 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:48.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.773+0000 7f0095600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:49:48.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.773+0000 7f0095600640 1 -- 172.21.15.31:0/2085486043 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c02cbd0 con 0x55878bd06290 2024-03-21T10:49:48.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.773+0000 7f0095600640 1 -- 172.21.15.31:0/2085486043 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f009c02cd70 con 0x55878bd06290 2024-03-21T10:49:48.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.773+0000 7f00b520c740 1 -- 172.21.15.31:0/2085486043 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f009c05b070 con 0x55878bd06290 2024-03-21T10:49:48.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.774+0000 7f0095600640 1 -- 172.21.15.31:0/2085486043 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f009c06d050 con 0x55878bd06290 2024-03-21T10:49:48.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.774+0000 7f00b520c740 10 client.28435 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:48.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.774+0000 7f00b520c740 1 -- 172.21.15.31:0/2085486043 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55878bd7c4b0 con 0x55878bd06290 2024-03-21T10:49:48.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.774+0000 7f0095600640 1 -- 172.21.15.31:0/2085486043 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f009c05b070 con 0x55878bd06290 2024-03-21T10:49:48.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.774+0000 7f0095600640 1 -- 172.21.15.31:0/2085486043 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f009c003ab0 con 0x55878bd06290 2024-03-21T10:49:48.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.774+0000 7f00b520c740 10 client.28435 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:48.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.774+0000 7f00b520c740 10 client.28435 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:49:48.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.774+0000 7f00b520c740 20 client.28435 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:48.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.774+0000 7f00b520c740 1 --2- 172.21.15.31:0/2085486043 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55878bd7cb10 0x55878bd7ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:48.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.774+0000 7f00b520c740 4 client.28435 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-21T10:49:48.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.774+0000 7f00b520c740 1 -- 172.21.15.31:0/2085486043 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55878bc72740 con 0x55878bd7cb10 2024-03-21T10:49:48.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.775+0000 7f00aaa00640 1 --2- 172.21.15.31:0/2085486043 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55878bd7cb10 0x55878bd7ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:48.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.775+0000 7f00aaa00640 1 --2- 172.21.15.31:0/2085486043 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55878bd7cb10 0x55878bd7ef00 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3d83b3ee2d5c700 server_cookie=96cb4cc21f0ddc4c in_seq=0 out_seq=0 2024-03-21T10:49:48.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.775+0000 7f0095600640 10 client.28435 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:48.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.776+0000 7f0095600640 1 -- 172.21.15.31:0/2085486043 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55878bc72740 con 0x55878bd7cb10 2024-03-21T10:49:48.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.776+0000 7f0095600640 10 client.28435 handle_command_reply: tid=0 2024-03-21T10:49:48.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.776+0000 7f00b520c740 1 client.28435 shutdown 2024-03-21T10:49:48.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.777+0000 7f00b520c740 1 -- 172.21.15.31:0/2085486043 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55878bd06290 msgr2=0x55878bd14920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:48.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.777+0000 7f00b520c740 1 --2- 172.21.15.31:0/2085486043 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55878bd06290 0x55878bd14920 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f009c058040 tx=0x7f009c005c00 comp rx=0 tx=0).stop 2024-03-21T10:49:48.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.777+0000 7f00b520c740 1 -- 172.21.15.31:0/2085486043 shutdown_connections 2024-03-21T10:49:48.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.777+0000 7f00b520c740 1 --2- 172.21.15.31:0/2085486043 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55878bd7cb10 0x55878bd7ef00 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.777+0000 7f00b520c740 1 --2- 172.21.15.31:0/2085486043 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55878bd0f790 0x55878bd153a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.777+0000 7f00b520c740 1 --2- 172.21.15.31:0/2085486043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55878bd06bb0 0x55878bd14e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.777+0000 7f00b520c740 1 --2- 172.21.15.31:0/2085486043 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55878bd06290 0x55878bd14920 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.777+0000 7f00b520c740 1 -- 172.21.15.31:0/2085486043 >> 172.21.15.31:0/2085486043 conn(0x55878bd009c0 msgr2=0x55878bd02550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:48.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.777+0000 7f0095600640 0 client.28435 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:48.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.777+0000 7f00b520c740 1 -- 172.21.15.31:0/2085486043 shutdown_connections 2024-03-21T10:49:48.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.777+0000 7f00b520c740 1 -- 172.21.15.31:0/2085486043 wait complete. 2024-03-21T10:49:48.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.777+0000 7f00b520c740 20 client.28435 trim_cache size 0 max 16384 2024-03-21T10:49:48.777 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:49:48.777 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-03-21T10:49:48.777 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "d263959a-5444-4a72-8e01-b7ef83302148", 2024-03-21T10:49:48.777 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-03-21T10:49:48.777 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:49:48.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.778+0000 7f00b1000640 1 -- 172.21.15.31:0/224910847 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f00900a3110 msgr2=0x7f00900a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:48.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.778+0000 7f00b1000640 1 --2- 172.21.15.31:0/224910847 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f00900a3110 0x7f00900a55d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f009c005ab0 tx=0x7f009c02e040 comp rx=0 tx=0).stop 2024-03-21T10:49:48.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.778+0000 7f00b1000640 1 -- 172.21.15.31:0/224910847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00ac108440 msgr2=0x7f00ac19e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:48.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.778+0000 7f00b1000640 1 --2- 172.21.15.31:0/224910847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00ac108440 0x7f00ac19e8a0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f009800ea60 tx=0x7f009800ef20 comp rx=0 tx=0).stop 2024-03-21T10:49:48.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.778+0000 7f00b1000640 1 -- 172.21.15.31:0/224910847 shutdown_connections 2024-03-21T10:49:48.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.778+0000 7f00b1000640 1 --2- 172.21.15.31:0/224910847 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f00900a3110 0x7f00900a55d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.778+0000 7f00b1000640 1 --2- 172.21.15.31:0/224910847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00ac10c980 0x7f00ac1a2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.778+0000 7f00b1000640 1 --2- 172.21.15.31:0/224910847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00ac108440 0x7f00ac19e8a0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.778+0000 7f00b1000640 1 --2- 172.21.15.31:0/224910847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ac107aa0 0x7f00ac19e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:48.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.778+0000 7f00b1000640 1 -- 172.21.15.31:0/224910847 >> 172.21.15.31:0/224910847 conn(0x7f00ac0772a0 msgr2=0x7f00ac077a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:48.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.778+0000 7f00b1000640 1 -- 172.21.15.31:0/224910847 shutdown_connections 2024-03-21T10:49:48.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:48.778+0000 7f00b1000640 1 -- 172.21.15.31:0/224910847 wait complete. 2024-03-21T10:49:48.794 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:49:48.794 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:49:49.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.040+0000 7f38f7e00640 1 -- 172.21.15.31:0/1442801926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38f810c980 msgr2=0x7f38f810cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:49.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.040+0000 7f38f7e00640 1 --2- 172.21.15.31:0/1442801926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38f810c980 0x7f38f810cd60 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f38e8009900 tx=0x7f38e802bcd0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.041+0000 7f38f7e00640 1 -- 172.21.15.31:0/1442801926 shutdown_connections 2024-03-21T10:49:49.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.041+0000 7f38f7e00640 1 --2- 172.21.15.31:0/1442801926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38f8108440 0x7f38f8114900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.041+0000 7f38f7e00640 1 --2- 172.21.15.31:0/1442801926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38f8107aa0 0x7f38f8107f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.041+0000 7f38f7e00640 1 --2- 172.21.15.31:0/1442801926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38f810c980 0x7f38f810cd60 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.041+0000 7f38f7e00640 1 -- 172.21.15.31:0/1442801926 >> 172.21.15.31:0/1442801926 conn(0x7f38f80772a0 msgr2=0x7f38f80796c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:49.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.041+0000 7f38f7e00640 1 -- 172.21.15.31:0/1442801926 shutdown_connections 2024-03-21T10:49:49.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.041+0000 7f38f7e00640 1 -- 172.21.15.31:0/1442801926 wait complete. 2024-03-21T10:49:49.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.042+0000 7f38f7e00640 1 Processor -- start 2024-03-21T10:49:49.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.042+0000 7f38f7e00640 1 -- start start 2024-03-21T10:49:49.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.043+0000 7f38f7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38f8107aa0 0x7f38f819e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:49.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.043+0000 7f38f7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38f8108440 0x7f38f819e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:49.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.043+0000 7f38f7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38f810c980 0x7f38f81a2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:49.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.043+0000 7f38f7e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f8117ec0 con 0x7f38f8108440 2024-03-21T10:49:49.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.043+0000 7f38f7e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f8117d40 con 0x7f38f8107aa0 2024-03-21T10:49:49.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.043+0000 7f38f7e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f8118040 con 0x7f38f810c980 2024-03-21T10:49:49.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.043+0000 7f38f6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38f8107aa0 0x7f38f819e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:49.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.043+0000 7f38f6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38f8107aa0 0x7f38f819e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33328/0 (socket says 172.21.15.31:33328) 2024-03-21T10:49:49.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.043+0000 7f38f6a00640 1 -- 172.21.15.31:0/2965283825 learned_addr learned my addr 172.21.15.31:0/2965283825 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:49.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.043+0000 7f38f7400640 1 --2- 172.21.15.31:0/2965283825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38f810c980 0x7f38f81a2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:49.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.043+0000 7f38ee000640 1 --2- 172.21.15.31:0/2965283825 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38f8108440 0x7f38f819e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:49.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.044+0000 7f38f6a00640 1 -- 172.21.15.31:0/2965283825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38f810c980 msgr2=0x7f38f81a2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:49.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.044+0000 7f38f6a00640 1 --2- 172.21.15.31:0/2965283825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38f810c980 0x7f38f81a2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.044+0000 7f38f6a00640 1 -- 172.21.15.31:0/2965283825 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38f8108440 msgr2=0x7f38f819e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:49.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.044+0000 7f38f6a00640 1 --2- 172.21.15.31:0/2965283825 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38f8108440 0x7f38f819e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.044+0000 7f38f6a00640 1 -- 172.21.15.31:0/2965283825 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f81a3350 con 0x7f38f8107aa0 2024-03-21T10:49:49.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.044+0000 7f38f6a00640 1 --2- 172.21.15.31:0/2965283825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38f8107aa0 0x7f38f819e360 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f38e8000c00 tx=0x7f38e80043c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:49.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.044+0000 7f38fca00640 1 -- 172.21.15.31:0/2965283825 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e8042020 con 0x7f38f8107aa0 2024-03-21T10:49:49.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.044+0000 7f38fca00640 1 -- 172.21.15.31:0/2965283825 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38e80054e0 con 0x7f38f8107aa0 2024-03-21T10:49:49.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.044+0000 7f38f7e00640 1 -- 172.21.15.31:0/2965283825 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38f81a35e0 con 0x7f38f8107aa0 2024-03-21T10:49:49.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.044+0000 7f38fca00640 1 -- 172.21.15.31:0/2965283825 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e8003b20 con 0x7f38f8107aa0 2024-03-21T10:49:49.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.044+0000 7f38f7e00640 1 -- 172.21.15.31:0/2965283825 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38f81aaed0 con 0x7f38f8107aa0 2024-03-21T10:49:49.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.048+0000 7f3900bbc740 1 Processor -- start 2024-03-21T10:49:49.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.048+0000 7f3900bbc740 1 -- start start 2024-03-21T10:49:49.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.048+0000 7f38fca00640 1 -- 172.21.15.31:0/2965283825 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f38e8045020 con 0x7f38f8107aa0 2024-03-21T10:49:49.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.048+0000 7f3900bbc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbc7b53dd0 0x55bbc7b52b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:49.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.048+0000 7f3900bbc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbc7b530f0 0x55bbc7b53570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:49.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.048+0000 7f3900bbc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbc7b5c460 0x55bbc7b5e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:49.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.048+0000 7f3900bbc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbc7ab7250 con 0x55bbc7b53dd0 2024-03-21T10:49:49.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.048+0000 7f3900bbc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbc7a5fa30 con 0x55bbc7b530f0 2024-03-21T10:49:49.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.048+0000 7f3900bbc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbc7ab32f0 con 0x55bbc7b5c460 2024-03-21T10:49:49.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.048+0000 7f38f7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbc7b53dd0 0x55bbc7b52b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:49.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.048+0000 7f38f6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbc7b530f0 0x55bbc7b53570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:49.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.048+0000 7f38ee000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbc7b5c460 0x55bbc7b5e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:49.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.048+0000 7f38f6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbc7b530f0 0x55bbc7b53570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33342/0 (socket says 172.21.15.31:33342) 2024-03-21T10:49:49.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.049+0000 7f38f7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbc7b53dd0 0x55bbc7b52b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40148/0 (socket says 172.21.15.31:40148) 2024-03-21T10:49:49.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.049+0000 7f38f6a00640 1 -- 172.21.15.31:0/146082313 learned_addr learned my addr 172.21.15.31:0/146082313 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:49.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.049+0000 7f38ee000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbc7b5c460 0x55bbc7b5e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34652/0 (socket says 172.21.15.31:34652) 2024-03-21T10:49:49.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.049+0000 7f38ee000640 1 -- 172.21.15.31:0/146082313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbc7b530f0 msgr2=0x55bbc7b53570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:49.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.049+0000 7f38ee000640 1 --2- 172.21.15.31:0/146082313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbc7b530f0 0x55bbc7b53570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.049+0000 7f38ee000640 1 -- 172.21.15.31:0/146082313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbc7b53dd0 msgr2=0x55bbc7b52b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:49.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.049+0000 7f38ee000640 1 --2- 172.21.15.31:0/146082313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbc7b53dd0 0x55bbc7b52b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.049+0000 7f38ee000640 1 -- 172.21.15.31:0/146082313 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bbc7a4a690 con 0x55bbc7b5c460 2024-03-21T10:49:49.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.049+0000 7f38ee000640 1 --2- 172.21.15.31:0/146082313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbc7b5c460 0x55bbc7b5e850 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f38e0005fd0 tx=0x7f38e000f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8588fbc3b2702587 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:49.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.049+0000 7f38e7e00640 1 -- 172.21.15.31:0/146082313 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e000dea0 con 0x55bbc7b5c460 2024-03-21T10:49:49.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.049+0000 7f38e7e00640 1 -- 172.21.15.31:0/146082313 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38e0005500 con 0x55bbc7b5c460 2024-03-21T10:49:49.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.049+0000 7f38e7e00640 1 -- 172.21.15.31:0/146082313 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e0011680 con 0x55bbc7b5c460 2024-03-21T10:49:49.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.050+0000 7f38fca00640 1 --2- 172.21.15.31:0/2965283825 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f38dc0a3090 0x7f38dc0a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:49.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.050+0000 7f38fca00640 1 -- 172.21.15.31:0/2965283825 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f38e80f18e0 con 0x7f38f8107aa0 2024-03-21T10:49:49.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.050+0000 7f38f7400640 1 --2- 172.21.15.31:0/2965283825 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f38dc0a3090 0x7f38dc0a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:49.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.050+0000 7f3900bbc740 1 -- 172.21.15.31:0/146082313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbc7b5c460 msgr2=0x55bbc7b5e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:49.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.050+0000 7f3900bbc740 1 --2- 172.21.15.31:0/146082313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbc7b5c460 0x55bbc7b5e850 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f38e0005fd0 tx=0x7f38e000f730 comp rx=0 tx=0).stop 2024-03-21T10:49:49.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.050+0000 7f3900bbc740 1 -- 172.21.15.31:0/146082313 shutdown_connections 2024-03-21T10:49:49.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.050+0000 7f3900bbc740 1 --2- 172.21.15.31:0/146082313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbc7b5c460 0x55bbc7b5e850 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.050+0000 7f3900bbc740 1 --2- 172.21.15.31:0/146082313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbc7b530f0 0x55bbc7b53570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.050+0000 7f3900bbc740 1 --2- 172.21.15.31:0/146082313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbc7b53dd0 0x55bbc7b52b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.050+0000 7f3900bbc740 1 -- 172.21.15.31:0/146082313 >> 172.21.15.31:0/146082313 conn(0x55bbc7b1a200 msgr2=0x55bbc7b4db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:49.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.050+0000 7f3900bbc740 1 -- 172.21.15.31:0/146082313 shutdown_connections 2024-03-21T10:49:49.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.051+0000 7f3900bbc740 1 -- 172.21.15.31:0/146082313 wait complete. 2024-03-21T10:49:49.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.051+0000 7f38f7400640 1 --2- 172.21.15.31:0/2965283825 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f38dc0a3090 0x7f38dc0a5550 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f38f0005fd0 tx=0x7f38f0009500 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:49.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.051+0000 7f3900bbc740 1 -- start start 2024-03-21T10:49:49.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.051+0000 7f3900bbc740 1 Processor -- start 2024-03-21T10:49:49.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.052+0000 7f3900bbc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbc7b530f0 0x55bbc7b5bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:49.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.052+0000 7f3900bbc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbc7b53dd0 0x55bbc7b58eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:49.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.052+0000 7f3900bbc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbc7b5c460 0x55bbc7b593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:49.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.052+0000 7f3900bbc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbc7b5f2d0 con 0x55bbc7b53dd0 2024-03-21T10:49:49.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.052+0000 7f3900bbc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbc7a72e80 con 0x55bbc7b530f0 2024-03-21T10:49:49.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.052+0000 7f3900bbc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbc7b5f450 con 0x55bbc7b5c460 2024-03-21T10:49:49.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.052+0000 7f3900bbc740 10 client.0 _ll_register_callbacks cb 0x55bbc7a85130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:49.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.052+0000 7f38f6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbc7b530f0 0x55bbc7b5bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:49.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.052+0000 7f38f6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbc7b530f0 0x55bbc7b5bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33346/0 (socket says 172.21.15.31:33346) 2024-03-21T10:49:49.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.052+0000 7f38f6a00640 1 -- 172.21.15.31:0/818003991 learned_addr learned my addr 172.21.15.31:0/818003991 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:49.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.052+0000 7f38f7400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbc7b5c460 0x55bbc7b593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:49.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.052+0000 7f38ee000640 1 --2- 172.21.15.31:0/818003991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbc7b53dd0 0x55bbc7b58eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:49.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.052+0000 7f38f6a00640 1 -- 172.21.15.31:0/818003991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbc7b5c460 msgr2=0x55bbc7b593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:49.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.052+0000 7f38f6a00640 1 --2- 172.21.15.31:0/818003991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbc7b5c460 0x55bbc7b593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.052+0000 7f38f6a00640 1 -- 172.21.15.31:0/818003991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbc7b53dd0 msgr2=0x55bbc7b58eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:49.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.052+0000 7f38f6a00640 1 --2- 172.21.15.31:0/818003991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbc7b53dd0 0x55bbc7b58eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.052+0000 7f38f6a00640 1 -- 172.21.15.31:0/818003991 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bbc78957d0 con 0x55bbc7b530f0 2024-03-21T10:49:49.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.052+0000 7f38ee000640 1 --2- 172.21.15.31:0/818003991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbc7b53dd0 0x55bbc7b58eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:49:49.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.053+0000 7f38f6a00640 1 --2- 172.21.15.31:0/818003991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbc7b530f0 0x55bbc7b5bab0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f38e80c5ec0 tx=0x7f38e80f5040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c5c23c858f25033 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:49.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.053+0000 7f38e4c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:49:49.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.053+0000 7f38e4c00640 1 -- 172.21.15.31:0/818003991 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e80bd7b0 con 0x55bbc7b530f0 2024-03-21T10:49:49.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.053+0000 7f38e4c00640 1 -- 172.21.15.31:0/818003991 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f38e80eeeb0 con 0x55bbc7b530f0 2024-03-21T10:49:49.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.053+0000 7f3900bbc740 1 -- 172.21.15.31:0/818003991 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f38e0005500 con 0x55bbc7b530f0 2024-03-21T10:49:49.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.053+0000 7f38e4c00640 1 -- 172.21.15.31:0/818003991 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38e80f7020 con 0x55bbc7b530f0 2024-03-21T10:49:49.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.053+0000 7f3900bbc740 10 client.28520 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:49.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.053+0000 7f3900bbc740 1 -- 172.21.15.31:0/818003991 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55bbc77cb3f0 con 0x55bbc7b530f0 2024-03-21T10:49:49.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.053+0000 7f38e4c00640 1 -- 172.21.15.31:0/818003991 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38e0005500 con 0x55bbc7b530f0 2024-03-21T10:49:49.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.053+0000 7f38e4c00640 1 -- 172.21.15.31:0/818003991 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f38e80c5f60 con 0x55bbc7b530f0 2024-03-21T10:49:49.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.053+0000 7f3900bbc740 10 client.28520 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:49.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.053+0000 7f3900bbc740 10 client.28520 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:49:49.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.054+0000 7f3900bbc740 20 client.28520 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:49.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.054+0000 7f3900bbc740 1 --2- 172.21.15.31:0/818003991 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55bbc7b6c2c0 0x55bbc7b6e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:49.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.054+0000 7f3900bbc740 4 client.28520 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:49:49.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.054+0000 7f3900bbc740 1 -- 172.21.15.31:0/818003991 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bbc7ac4790 con 0x55bbc7b6c2c0 2024-03-21T10:49:49.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.054+0000 7f38ee000640 1 --2- 172.21.15.31:0/818003991 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55bbc7b6c2c0 0x55bbc7b6e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:49.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.054+0000 7f38ee000640 1 --2- 172.21.15.31:0/818003991 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55bbc7b6c2c0 0x55bbc7b6e740 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfc6bfab438b8fb1 server_cookie=8d0821426d6100ee in_seq=0 out_seq=0 2024-03-21T10:49:49.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.054+0000 7f38e4c00640 10 client.28520 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:49.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.055+0000 7f38e4c00640 1 -- 172.21.15.31:0/818003991 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55bbc7ac4790 con 0x55bbc7b6c2c0 2024-03-21T10:49:49.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.055+0000 7f38e4c00640 10 client.28520 handle_command_reply: tid=0 2024-03-21T10:49:49.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.055+0000 7f3900bbc740 1 client.28520 shutdown 2024-03-21T10:49:49.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.055+0000 7f3900bbc740 1 -- 172.21.15.31:0/818003991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbc7b530f0 msgr2=0x55bbc7b5bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:49.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.055+0000 7f3900bbc740 1 --2- 172.21.15.31:0/818003991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbc7b530f0 0x55bbc7b5bab0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f38e80c5ec0 tx=0x7f38e80f5040 comp rx=0 tx=0).stop 2024-03-21T10:49:49.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.056+0000 7f3900bbc740 1 -- 172.21.15.31:0/818003991 shutdown_connections 2024-03-21T10:49:49.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.056+0000 7f3900bbc740 1 --2- 172.21.15.31:0/818003991 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55bbc7b6c2c0 0x55bbc7b6e740 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.056+0000 7f3900bbc740 1 --2- 172.21.15.31:0/818003991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbc7b5c460 0x55bbc7b593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.056+0000 7f3900bbc740 1 --2- 172.21.15.31:0/818003991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbc7b53dd0 0x55bbc7b58eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.056+0000 7f3900bbc740 1 --2- 172.21.15.31:0/818003991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbc7b530f0 0x55bbc7b5bab0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.056+0000 7f38e4c00640 0 client.28520 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:49.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.056+0000 7f3900bbc740 1 -- 172.21.15.31:0/818003991 >> 172.21.15.31:0/818003991 conn(0x55bbc7b1a200 msgr2=0x55bbc7b4f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:49.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.056+0000 7f3900bbc740 1 -- 172.21.15.31:0/818003991 shutdown_connections 2024-03-21T10:49:49.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.056+0000 7f3900bbc740 1 -- 172.21.15.31:0/818003991 wait complete. 2024-03-21T10:49:49.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.056+0000 7f3900bbc740 20 client.28520 trim_cache size 0 max 16384 2024-03-21T10:49:49.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.073+0000 7f3900bbc740 1 Processor -- start 2024-03-21T10:49:49.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.074+0000 7f3900bbc740 1 -- start start 2024-03-21T10:49:49.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.074+0000 7f3900bbc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbc7b530f0 0x55bbc7b5b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:49.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.074+0000 7f3900bbc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbc7b53dd0 0x55bbc7b5bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:49.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.074+0000 7f3900bbc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbc7b5c460 0x55bbc7b5d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:49.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.074+0000 7f3900bbc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f81b7300 con 0x55bbc7b5c460 2024-03-21T10:49:49.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.074+0000 7f3900bbc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f81b4e00 con 0x55bbc7b53dd0 2024-03-21T10:49:49.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.074+0000 7f3900bbc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f81b2ec0 con 0x55bbc7b530f0 2024-03-21T10:49:49.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.074+0000 7f38f6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbc7b530f0 0x55bbc7b5b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:49.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.074+0000 7f38f6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbc7b530f0 0x55bbc7b5b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34668/0 (socket says 172.21.15.31:34668) 2024-03-21T10:49:49.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.074+0000 7f38f6a00640 1 -- 172.21.15.31:0/3804199318 learned_addr learned my addr 172.21.15.31:0/3804199318 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:49.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.074+0000 7f38ee000640 1 --2- 172.21.15.31:0/3804199318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbc7b53dd0 0x55bbc7b5bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:49.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.074+0000 7f38f7400640 1 --2- 172.21.15.31:0/3804199318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbc7b5c460 0x55bbc7b5d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:49.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.074+0000 7f38f6a00640 1 -- 172.21.15.31:0/3804199318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbc7b53dd0 msgr2=0x55bbc7b5bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:49.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.074+0000 7f38f6a00640 1 --2- 172.21.15.31:0/3804199318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbc7b53dd0 0x55bbc7b5bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.074+0000 7f38f6a00640 1 -- 172.21.15.31:0/3804199318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbc7b5c460 msgr2=0x55bbc7b5d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:49.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.074+0000 7f38f6a00640 1 --2- 172.21.15.31:0/3804199318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbc7b5c460 0x55bbc7b5d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.074+0000 7f38f6a00640 1 -- 172.21.15.31:0/3804199318 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bbc778b0f0 con 0x55bbc7b530f0 2024-03-21T10:49:49.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.074+0000 7f38f7400640 1 --2- 172.21.15.31:0/3804199318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbc7b5c460 0x55bbc7b5d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:49:49.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.075+0000 7f38f6a00640 1 --2- 172.21.15.31:0/3804199318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbc7b530f0 0x55bbc7b5b570 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f38e80ba3b0 tx=0x7f38e802cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7760cba38a314a75 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:49.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.075+0000 7f38e7400640 1 -- 172.21.15.31:0/3804199318 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e80bd340 con 0x55bbc7b530f0 2024-03-21T10:49:49.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.075+0000 7f38e7400640 1 -- 172.21.15.31:0/3804199318 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38e8004c80 con 0x55bbc7b530f0 2024-03-21T10:49:49.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.075+0000 7f38e7400640 1 -- 172.21.15.31:0/3804199318 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e80bd520 con 0x55bbc7b530f0 2024-03-21T10:49:49.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.076+0000 7f3900bbc740 1 -- 172.21.15.31:0/3804199318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbc7b530f0 msgr2=0x55bbc7b5b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:49.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.076+0000 7f3900bbc740 1 --2- 172.21.15.31:0/3804199318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbc7b530f0 0x55bbc7b5b570 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f38e80ba3b0 tx=0x7f38e802cbc0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.076+0000 7f3900bbc740 1 -- 172.21.15.31:0/3804199318 shutdown_connections 2024-03-21T10:49:49.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.076+0000 7f3900bbc740 1 --2- 172.21.15.31:0/3804199318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbc7b5c460 0x55bbc7b5d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.076+0000 7f3900bbc740 1 --2- 172.21.15.31:0/3804199318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbc7b53dd0 0x55bbc7b5bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.076+0000 7f3900bbc740 1 --2- 172.21.15.31:0/3804199318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbc7b530f0 0x55bbc7b5b570 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.076+0000 7f3900bbc740 1 -- 172.21.15.31:0/3804199318 >> 172.21.15.31:0/3804199318 conn(0x55bbc7b1a200 msgr2=0x55bbc7b4d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:49.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.076+0000 7f3900bbc740 1 -- 172.21.15.31:0/3804199318 shutdown_connections 2024-03-21T10:49:49.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.076+0000 7f3900bbc740 1 -- 172.21.15.31:0/3804199318 wait complete. 2024-03-21T10:49:49.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.076+0000 7f3900bbc740 1 -- start start 2024-03-21T10:49:49.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.076+0000 7f3900bbc740 1 Processor -- start 2024-03-21T10:49:49.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.077+0000 7f3900bbc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbc7b530f0 0x55bbc7b55820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:49.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.077+0000 7f3900bbc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbc7b53dd0 0x55bbc7b55d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:49.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.077+0000 7f3900bbc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbc7b5c460 0x55bbc7b562a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:49.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.077+0000 7f3900bbc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f81b34c0 con 0x55bbc7b53dd0 2024-03-21T10:49:49.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.077+0000 7f3900bbc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f81b47a0 con 0x55bbc7b5c460 2024-03-21T10:49:49.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.077+0000 7f3900bbc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f81b3ae0 con 0x55bbc7b530f0 2024-03-21T10:49:49.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.077+0000 7f3900bbc740 10 client.0 _ll_register_callbacks cb 0x7f38e80eece0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:49.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.077+0000 7f38f6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbc7b530f0 0x55bbc7b55820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:49.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.077+0000 7f38f6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbc7b530f0 0x55bbc7b55820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34670/0 (socket says 172.21.15.31:34670) 2024-03-21T10:49:49.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.077+0000 7f38f6a00640 1 -- 172.21.15.31:0/653375773 learned_addr learned my addr 172.21.15.31:0/653375773 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:49.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.077+0000 7f38f7400640 1 --2- 172.21.15.31:0/653375773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbc7b5c460 0x55bbc7b562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:49.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.077+0000 7f38ee000640 1 --2- 172.21.15.31:0/653375773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbc7b53dd0 0x55bbc7b55d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:49.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.077+0000 7f38f6a00640 1 -- 172.21.15.31:0/653375773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbc7b5c460 msgr2=0x55bbc7b562a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:49.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.077+0000 7f38f6a00640 1 --2- 172.21.15.31:0/653375773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbc7b5c460 0x55bbc7b562a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.077+0000 7f38f6a00640 1 -- 172.21.15.31:0/653375773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbc7b53dd0 msgr2=0x55bbc7b55d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:49.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.077+0000 7f38f6a00640 1 --2- 172.21.15.31:0/653375773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbc7b53dd0 0x55bbc7b55d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.077+0000 7f38f6a00640 1 -- 172.21.15.31:0/653375773 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bbc7bc6df0 con 0x55bbc7b530f0 2024-03-21T10:49:49.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.077+0000 7f38ee000640 1 --2- 172.21.15.31:0/653375773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbc7b53dd0 0x55bbc7b55d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:49:49.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.078+0000 7f38f6a00640 1 --2- 172.21.15.31:0/653375773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbc7b530f0 0x55bbc7b55820 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f38e80bd6e0 tx=0x7f38e802cb30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59bb8a0367d71227 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:49.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.078+0000 7f38e4c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:49:49.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.078+0000 7f38e4c00640 1 -- 172.21.15.31:0/653375773 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e80bd340 con 0x55bbc7b530f0 2024-03-21T10:49:49.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.078+0000 7f38e4c00640 1 -- 172.21.15.31:0/653375773 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f38e80ef020 con 0x55bbc7b530f0 2024-03-21T10:49:49.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.078+0000 7f3900bbc740 1 -- 172.21.15.31:0/653375773 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f38e8004c80 con 0x55bbc7b530f0 2024-03-21T10:49:49.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.078+0000 7f38e4c00640 1 -- 172.21.15.31:0/653375773 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38e80bd7b0 con 0x55bbc7b530f0 2024-03-21T10:49:49.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.078+0000 7f3900bbc740 10 client.28453 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:49.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.078+0000 7f3900bbc740 1 -- 172.21.15.31:0/653375773 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55bbc7bc6df0 con 0x55bbc7b530f0 2024-03-21T10:49:49.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.078+0000 7f38e4c00640 1 -- 172.21.15.31:0/653375773 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38e8004c80 con 0x55bbc7b530f0 2024-03-21T10:49:49.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.078+0000 7f38e4c00640 1 -- 172.21.15.31:0/653375773 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f38e80a8210 con 0x55bbc7b530f0 2024-03-21T10:49:49.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.078+0000 7f3900bbc740 10 client.28453 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:49.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.078+0000 7f3900bbc740 10 client.28453 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:49:49.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.078+0000 7f3900bbc740 20 client.28453 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:49.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.078+0000 7f3900bbc740 1 --2- 172.21.15.31:0/653375773 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55bbc7bc8390 0x55bbc7bca780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:49.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.079+0000 7f3900bbc740 4 client.28453 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:49:49.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.079+0000 7f3900bbc740 1 -- 172.21.15.31:0/653375773 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55bbc7abeae0 con 0x55bbc7bc8390 2024-03-21T10:49:49.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.079+0000 7f38ee000640 1 --2- 172.21.15.31:0/653375773 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55bbc7bc8390 0x55bbc7bca780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:49.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.079+0000 7f38ee000640 1 --2- 172.21.15.31:0/653375773 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55bbc7bc8390 0x55bbc7bca780 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c23a944c1ee4fff server_cookie=b5d9b55778bcf984 in_seq=0 out_seq=0 2024-03-21T10:49:49.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.079+0000 7f38e4c00640 10 client.28453 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:49.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.079+0000 7f38e4c00640 1 -- 172.21.15.31:0/653375773 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55bbc7abeae0 con 0x55bbc7bc8390 2024-03-21T10:49:49.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.079+0000 7f38e4c00640 10 client.28453 handle_command_reply: tid=0 2024-03-21T10:49:49.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.079+0000 7f3900bbc740 1 client.28453 shutdown 2024-03-21T10:49:49.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.080+0000 7f3900bbc740 1 -- 172.21.15.31:0/653375773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbc7b530f0 msgr2=0x55bbc7b55820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:49.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.080+0000 7f3900bbc740 1 --2- 172.21.15.31:0/653375773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbc7b530f0 0x55bbc7b55820 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f38e80bd6e0 tx=0x7f38e802cb30 comp rx=0 tx=0).stop 2024-03-21T10:49:49.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.080+0000 7f3900bbc740 1 -- 172.21.15.31:0/653375773 shutdown_connections 2024-03-21T10:49:49.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.080+0000 7f3900bbc740 1 --2- 172.21.15.31:0/653375773 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55bbc7bc8390 0x55bbc7bca780 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.080+0000 7f3900bbc740 1 --2- 172.21.15.31:0/653375773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbc7b5c460 0x55bbc7b562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.080+0000 7f3900bbc740 1 --2- 172.21.15.31:0/653375773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbc7b53dd0 0x55bbc7b55d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.080+0000 7f3900bbc740 1 --2- 172.21.15.31:0/653375773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbc7b530f0 0x55bbc7b55820 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.080+0000 7f38e4c00640 0 client.28453 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:49.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.080+0000 7f3900bbc740 1 -- 172.21.15.31:0/653375773 >> 172.21.15.31:0/653375773 conn(0x55bbc7b1a200 msgr2=0x55bbc7b57760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:49.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.080+0000 7f3900bbc740 1 -- 172.21.15.31:0/653375773 shutdown_connections 2024-03-21T10:49:49.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.080+0000 7f3900bbc740 1 -- 172.21.15.31:0/653375773 wait complete. 2024-03-21T10:49:49.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.080+0000 7f3900bbc740 20 client.28453 trim_cache size 0 max 16384 2024-03-21T10:49:49.082 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:49:49.082 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-21T10:49:49.082 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-03-21T10:49:49.082 INFO:teuthology.orchestra.run.smithi031.stdout: "d263959a-5444-4a72-8e01-b7ef83302148": { 2024-03-21T10:49:49.082 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-03-21T10:49:49.082 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "d263959a-5444-4a72-8e01-b7ef83302148", 2024-03-21T10:49:49.082 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-03-21T10:49:49.082 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:49:49.082 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:49:49.082 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:49:49.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.081+0000 7f38f7e00640 1 -- 172.21.15.31:0/2965283825 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f38dc0a3090 msgr2=0x7f38dc0a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:49.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.081+0000 7f38f7e00640 1 --2- 172.21.15.31:0/2965283825 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f38dc0a3090 0x7f38dc0a5550 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f38f0005fd0 tx=0x7f38f0009500 comp rx=0 tx=0).stop 2024-03-21T10:49:49.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.081+0000 7f38f7e00640 1 -- 172.21.15.31:0/2965283825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38f8107aa0 msgr2=0x7f38f819e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:49.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.081+0000 7f38f7e00640 1 --2- 172.21.15.31:0/2965283825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38f8107aa0 0x7f38f819e360 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f38e8000c00 tx=0x7f38e80043c0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.081+0000 7f38f7e00640 1 -- 172.21.15.31:0/2965283825 shutdown_connections 2024-03-21T10:49:49.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.081+0000 7f38f7e00640 1 --2- 172.21.15.31:0/2965283825 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f38dc0a3090 0x7f38dc0a5550 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.081+0000 7f38f7e00640 1 --2- 172.21.15.31:0/2965283825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38f810c980 0x7f38f81a2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.081+0000 7f38f7e00640 1 --2- 172.21.15.31:0/2965283825 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38f8108440 0x7f38f819e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.081+0000 7f38f7e00640 1 --2- 172.21.15.31:0/2965283825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38f8107aa0 0x7f38f819e360 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:49.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.081+0000 7f38f7e00640 1 -- 172.21.15.31:0/2965283825 >> 172.21.15.31:0/2965283825 conn(0x7f38f80772a0 msgr2=0x7f38f8077a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:49.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.081+0000 7f38f7e00640 1 -- 172.21.15.31:0/2965283825 shutdown_connections 2024-03-21T10:49:49.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:49:49.081+0000 7f38f7e00640 1 -- 172.21.15.31:0/2965283825 wait complete. 2024-03-21T10:49:49.104 INFO:tasks.cephfs.filesystem:scrub status for tag:d263959a-5444-4a72-8e01-b7ef83302148 - {'path': '/', 'tag': 'd263959a-5444-4a72-8e01-b7ef83302148', 'options': 'recursive,force'} 2024-03-21T10:49:49.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:49 smithi040 ceph-mon[36154]: pgmap v350: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:49:49.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:49 smithi040 ceph-mon[36154]: scrub queued for path: / 2024-03-21T10:49:49.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:49 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:49:49.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:49 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:49:49.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:49 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:49:49.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:49 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:49:49.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:49 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:49:49.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:49 smithi031 ceph-mon[31673]: pgmap v350: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:49:49.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:49 smithi031 ceph-mon[31673]: scrub queued for path: / 2024-03-21T10:49:49.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:49 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:49:49.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:49 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:49:49.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:49 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:49:49.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:49 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:49:49.805 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:49 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:49:49.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:49 smithi070 ceph-mon[36330]: pgmap v350: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:49:49.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:49 smithi070 ceph-mon[36330]: scrub queued for path: / 2024-03-21T10:49:49.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:49 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:49:49.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:49 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:49:49.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:49 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:49:49.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:49 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:49:49.823 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:49 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:49:50.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:50 smithi040 ceph-mon[36154]: scrub summary: active paths [/] 2024-03-21T10:49:50.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:50 smithi040 ceph-mon[36154]: scrub summary: idle+waiting paths [/] 2024-03-21T10:49:50.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:50 smithi031 ceph-mon[31673]: scrub summary: active paths [/] 2024-03-21T10:49:50.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:50 smithi031 ceph-mon[31673]: scrub summary: idle+waiting paths [/] 2024-03-21T10:49:50.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:50 smithi070 ceph-mon[36330]: scrub summary: active paths [/] 2024-03-21T10:49:50.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:50 smithi070 ceph-mon[36330]: scrub summary: idle+waiting paths [/] 2024-03-21T10:49:51.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:51 smithi040 ceph-mon[36154]: pgmap v351: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-21T10:49:51.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:51 smithi031 ceph-mon[31673]: pgmap v351: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-21T10:49:51.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:51 smithi070 ceph-mon[36330]: pgmap v351: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-21T10:49:53.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:53 smithi031 ceph-mon[31673]: pgmap v352: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-21T10:49:53.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:53 smithi070 ceph-mon[36330]: pgmap v352: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-21T10:49:53.930 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:53 smithi040 ceph-mon[36154]: pgmap v352: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-21T10:49:55.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:55 smithi031 ceph-mon[31673]: pgmap v353: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-21T10:49:55.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:55 smithi070 ceph-mon[36330]: pgmap v353: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-21T10:49:55.930 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:55 smithi040 ceph-mon[36154]: pgmap v353: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-21T10:49:56.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:56 smithi031 ceph-mon[31673]: pgmap v354: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-21T10:49:56.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:56 smithi070 ceph-mon[36330]: pgmap v354: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-21T10:49:56.930 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:56 smithi040 ceph-mon[36154]: pgmap v354: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-21T10:49:57.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.745+0000 7fc79b800640 1 -- 172.21.15.31:0/680494601 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc794109880 msgr2=0x7fc794109c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:57.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.745+0000 7fc79b800640 1 --2- 172.21.15.31:0/680494601 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc794109880 0x7fc794109c60 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fc788009930 tx=0x7fc78802bc30 comp rx=0 tx=0).stop 2024-03-21T10:49:57.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.746+0000 7fc79b800640 1 -- 172.21.15.31:0/680494601 shutdown_connections 2024-03-21T10:49:57.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.746+0000 7fc79b800640 1 --2- 172.21.15.31:0/680494601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc794104230 0x7fc794110740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.746+0000 7fc79b800640 1 --2- 172.21.15.31:0/680494601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc794103870 0x7fc794103cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.746+0000 7fc79b800640 1 --2- 172.21.15.31:0/680494601 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc794109880 0x7fc794109c60 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.746+0000 7fc79b800640 1 -- 172.21.15.31:0/680494601 >> 172.21.15.31:0/680494601 conn(0x7fc7940ff320 msgr2=0x7fc794101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:57.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.746+0000 7fc79b800640 1 -- 172.21.15.31:0/680494601 shutdown_connections 2024-03-21T10:49:57.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.746+0000 7fc79b800640 1 -- 172.21.15.31:0/680494601 wait complete. 2024-03-21T10:49:57.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.747+0000 7fc79b800640 1 Processor -- start 2024-03-21T10:49:57.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.747+0000 7fc79b800640 1 -- start start 2024-03-21T10:49:57.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.747+0000 7fc79b800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc794103870 0x7fc79419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:57.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.747+0000 7fc79b800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc794104230 0x7fc79419a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:57.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.747+0000 7fc79b800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc794109880 0x7fc79419ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:57.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.747+0000 7fc79b800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc794113c90 con 0x7fc794104230 2024-03-21T10:49:57.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.747+0000 7fc79b800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc794113b10 con 0x7fc794109880 2024-03-21T10:49:57.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.747+0000 7fc79b800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc794113e10 con 0x7fc794103870 2024-03-21T10:49:57.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.748+0000 7fc798e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc794103870 0x7fc79419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:57.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.748+0000 7fc798e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc794103870 0x7fc79419a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:49894/0 (socket says 172.21.15.31:49894) 2024-03-21T10:49:57.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.748+0000 7fc798e00640 1 -- 172.21.15.31:0/1421872700 learned_addr learned my addr 172.21.15.31:0/1421872700 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:57.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.748+0000 7fc793e00640 1 --2- 172.21.15.31:0/1421872700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc794104230 0x7fc79419a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:57.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.748+0000 7fc799800640 1 --2- 172.21.15.31:0/1421872700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc794109880 0x7fc79419ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:57.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.748+0000 7fc798e00640 1 -- 172.21.15.31:0/1421872700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc794109880 msgr2=0x7fc79419ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:57.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.748+0000 7fc798e00640 1 --2- 172.21.15.31:0/1421872700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc794109880 0x7fc79419ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.748+0000 7fc798e00640 1 -- 172.21.15.31:0/1421872700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc794104230 msgr2=0x7fc79419a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:57.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.748+0000 7fc798e00640 1 --2- 172.21.15.31:0/1421872700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc794104230 0x7fc79419a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.748+0000 7fc798e00640 1 -- 172.21.15.31:0/1421872700 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79419f1c0 con 0x7fc794103870 2024-03-21T10:49:57.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.748+0000 7fc798e00640 1 --2- 172.21.15.31:0/1421872700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc794103870 0x7fc79419a1d0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fc788009230 tx=0x7fc78802cb30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:57.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.749+0000 7fc791600640 1 -- 172.21.15.31:0/1421872700 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc788041070 con 0x7fc794103870 2024-03-21T10:49:57.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.749+0000 7fc791600640 1 -- 172.21.15.31:0/1421872700 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc788004450 con 0x7fc794103870 2024-03-21T10:49:57.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.749+0000 7fc791600640 1 -- 172.21.15.31:0/1421872700 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc788003e50 con 0x7fc794103870 2024-03-21T10:49:57.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.749+0000 7fc79b800640 1 -- 172.21.15.31:0/1421872700 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc79419f450 con 0x7fc794103870 2024-03-21T10:49:57.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.749+0000 7fc79b800640 1 -- 172.21.15.31:0/1421872700 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7941a6d40 con 0x7fc794103870 2024-03-21T10:49:57.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.753+0000 7fc79e593740 1 Processor -- start 2024-03-21T10:49:57.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.753+0000 7fc79e593740 1 -- start start 2024-03-21T10:49:57.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.753+0000 7fc791600640 1 -- 172.21.15.31:0/1421872700 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fc788044070 con 0x7fc794103870 2024-03-21T10:49:57.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.754+0000 7fc79e593740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600dd80afd0 0x5600dd80b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.754+0000 7fc79e593740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80b8f0 0x5600dd814180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.754+0000 7fc79e593740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600dd8146c0 0x5600dd816ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.754+0000 7fc79e593740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600dd7cd3b0 con 0x5600dd80afd0 2024-03-21T10:49:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.754+0000 7fc79e593740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600dd76f830 con 0x5600dd8146c0 2024-03-21T10:49:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.754+0000 7fc79e593740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600dd7185d0 con 0x5600dd80b8f0 2024-03-21T10:49:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.754+0000 7fc798e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80b8f0 0x5600dd814180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.754+0000 7fc793e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600dd8146c0 0x5600dd816ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.754+0000 7fc798e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80b8f0 0x5600dd814180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:49910/0 (socket says 172.21.15.31:49910) 2024-03-21T10:49:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.754+0000 7fc798e00640 1 -- 172.21.15.31:0/3469710776 learned_addr learned my addr 172.21.15.31:0/3469710776 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.754+0000 7fc799800640 1 --2- 172.21.15.31:0/3469710776 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600dd80afd0 0x5600dd80b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.755+0000 7fc798e00640 1 -- 172.21.15.31:0/3469710776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600dd8146c0 msgr2=0x5600dd816ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.755+0000 7fc798e00640 1 --2- 172.21.15.31:0/3469710776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600dd8146c0 0x5600dd816ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.755+0000 7fc798e00640 1 -- 172.21.15.31:0/3469710776 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600dd80afd0 msgr2=0x5600dd80b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.755+0000 7fc798e00640 1 --2- 172.21.15.31:0/3469710776 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600dd80afd0 0x5600dd80b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.755+0000 7fc798e00640 1 -- 172.21.15.31:0/3469710776 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600dd7302c0 con 0x5600dd80b8f0 2024-03-21T10:49:57.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.755+0000 7fc798e00640 1 --2- 172.21.15.31:0/3469710776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80b8f0 0x5600dd814180 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fc7880ee760 tx=0x7fc7880f9490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7d1210f466198c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:57.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.755+0000 7fc786000640 1 -- 172.21.15.31:0/3469710776 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7880f3310 con 0x5600dd80b8f0 2024-03-21T10:49:57.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.755+0000 7fc786000640 1 -- 172.21.15.31:0/3469710776 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7880f34b0 con 0x5600dd80b8f0 2024-03-21T10:49:57.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.755+0000 7fc786000640 1 -- 172.21.15.31:0/3469710776 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc788103a40 con 0x5600dd80b8f0 2024-03-21T10:49:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.756+0000 7fc791600640 1 --2- 172.21.15.31:0/1421872700 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc77c0a3220 0x7fc77c0a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.756+0000 7fc79e593740 1 -- 172.21.15.31:0/3469710776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80b8f0 msgr2=0x5600dd814180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.756+0000 7fc79e593740 1 --2- 172.21.15.31:0/3469710776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80b8f0 0x5600dd814180 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fc7880ee760 tx=0x7fc7880f9490 comp rx=0 tx=0).stop 2024-03-21T10:49:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.756+0000 7fc791600640 1 -- 172.21.15.31:0/1421872700 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fc7880f1390 con 0x7fc794103870 2024-03-21T10:49:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.756+0000 7fc799800640 1 --2- 172.21.15.31:0/1421872700 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc77c0a3220 0x7fc77c0a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.756+0000 7fc79e593740 1 -- 172.21.15.31:0/3469710776 shutdown_connections 2024-03-21T10:49:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.756+0000 7fc79e593740 1 --2- 172.21.15.31:0/3469710776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600dd8146c0 0x5600dd816ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.756+0000 7fc79e593740 1 --2- 172.21.15.31:0/3469710776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80b8f0 0x5600dd814180 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.756+0000 7fc79e593740 1 --2- 172.21.15.31:0/3469710776 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600dd80afd0 0x5600dd80b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.756+0000 7fc79e593740 1 -- 172.21.15.31:0/3469710776 >> 172.21.15.31:0/3469710776 conn(0x5600dd8058f0 msgr2=0x5600dd805cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.756+0000 7fc79e593740 1 -- 172.21.15.31:0/3469710776 shutdown_connections 2024-03-21T10:49:57.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.756+0000 7fc79e593740 1 -- 172.21.15.31:0/3469710776 wait complete. 2024-03-21T10:49:57.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.757+0000 7fc79e593740 1 -- start start 2024-03-21T10:49:57.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.757+0000 7fc799800640 1 --2- 172.21.15.31:0/1421872700 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc77c0a3220 0x7fc77c0a56e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc78c005fd0 tx=0x7fc78c009500 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:57.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.757+0000 7fc79e593740 1 Processor -- start 2024-03-21T10:49:57.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.757+0000 7fc79e593740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80afd0 0x5600dd811700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:57.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.757+0000 7fc79e593740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600dd80b8f0 0x5600dd811c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:57.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.757+0000 7fc79e593740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600dd8146c0 0x5600dd812180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:57.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.757+0000 7fc79e593740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600dd72ba20 con 0x5600dd80b8f0 2024-03-21T10:49:57.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.757+0000 7fc79e593740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600dd76b8d0 con 0x5600dd8146c0 2024-03-21T10:49:57.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.757+0000 7fc79e593740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600dd8170b0 con 0x5600dd80afd0 2024-03-21T10:49:57.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.758+0000 7fc79e593740 10 client.0 _ll_register_callbacks cb 0x5600dd7681c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:57.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.758+0000 7fc798e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80afd0 0x5600dd811700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:57.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.758+0000 7fc799800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600dd8146c0 0x5600dd812180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:57.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.758+0000 7fc798e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80afd0 0x5600dd811700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:49916/0 (socket says 172.21.15.31:49916) 2024-03-21T10:49:57.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.758+0000 7fc798e00640 1 -- 172.21.15.31:0/3293856610 learned_addr learned my addr 172.21.15.31:0/3293856610 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:57.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.758+0000 7fc793e00640 1 --2- 172.21.15.31:0/3293856610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600dd80b8f0 0x5600dd811c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:57.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.758+0000 7fc798e00640 1 -- 172.21.15.31:0/3293856610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600dd8146c0 msgr2=0x5600dd812180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:57.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.758+0000 7fc798e00640 1 --2- 172.21.15.31:0/3293856610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600dd8146c0 0x5600dd812180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.758+0000 7fc798e00640 1 -- 172.21.15.31:0/3293856610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600dd80b8f0 msgr2=0x5600dd811c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:57.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.758+0000 7fc798e00640 1 --2- 172.21.15.31:0/3293856610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600dd80b8f0 0x5600dd811c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.758+0000 7fc798e00640 1 -- 172.21.15.31:0/3293856610 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5600dd484210 con 0x5600dd80afd0 2024-03-21T10:49:57.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.758+0000 7fc793e00640 1 --2- 172.21.15.31:0/3293856610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600dd80b8f0 0x5600dd811c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:49:57.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.758+0000 7fc798e00640 1 --2- 172.21.15.31:0/3293856610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80afd0 0x5600dd811700 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fc7880b99a0 tx=0x7fc788005c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55ffba6c3852c7ad server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:57.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.758+0000 7fc77aa00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:49:57.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.759+0000 7fc77aa00640 1 -- 172.21.15.31:0/3293856610 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78804b030 con 0x5600dd80afd0 2024-03-21T10:49:57.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.759+0000 7fc77aa00640 1 -- 172.21.15.31:0/3293856610 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fc7880b9bb0 con 0x5600dd80afd0 2024-03-21T10:49:57.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.759+0000 7fc79e593740 1 -- 172.21.15.31:0/3293856610 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc7880f34b0 con 0x5600dd80afd0 2024-03-21T10:49:57.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.759+0000 7fc77aa00640 1 -- 172.21.15.31:0/3293856610 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc7880b97e0 con 0x5600dd80afd0 2024-03-21T10:49:57.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.759+0000 7fc79e593740 10 client.28465 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:57.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.759+0000 7fc79e593740 1 -- 172.21.15.31:0/3293856610 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5600dd550620 con 0x5600dd80afd0 2024-03-21T10:49:57.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.759+0000 7fc77aa00640 1 -- 172.21.15.31:0/3293856610 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7880f34b0 con 0x5600dd80afd0 2024-03-21T10:49:57.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.759+0000 7fc77aa00640 1 -- 172.21.15.31:0/3293856610 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fc7880f3110 con 0x5600dd80afd0 2024-03-21T10:49:57.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.759+0000 7fc79e593740 10 client.28465 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:57.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.759+0000 7fc79e593740 10 client.28465 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:49:57.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.759+0000 7fc79e593740 20 client.28465 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:57.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.759+0000 7fc79e593740 1 --2- 172.21.15.31:0/3293856610 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5600dd8242e0 0x5600dd8266d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:57.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.759+0000 7fc79e593740 4 client.28465 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:49:57.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.759+0000 7fc79e593740 1 -- 172.21.15.31:0/3293856610 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5600dd77cd70 con 0x5600dd8242e0 2024-03-21T10:49:57.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.760+0000 7fc793e00640 1 --2- 172.21.15.31:0/3293856610 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5600dd8242e0 0x5600dd8266d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:57.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.760+0000 7fc793e00640 1 --2- 172.21.15.31:0/3293856610 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5600dd8242e0 0x5600dd8266d0 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9c096ce5edaa2a1 server_cookie=e934ef5fffcf1319 in_seq=0 out_seq=0 2024-03-21T10:49:57.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.760+0000 7fc77aa00640 10 client.28465 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.761+0000 7fc77aa00640 1 -- 172.21.15.31:0/3293856610 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5600dd77cd70 con 0x5600dd8242e0 2024-03-21T10:49:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.761+0000 7fc77aa00640 10 client.28465 handle_command_reply: tid=0 2024-03-21T10:49:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.761+0000 7fc79e593740 1 client.28465 shutdown 2024-03-21T10:49:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.761+0000 7fc79e593740 1 -- 172.21.15.31:0/3293856610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80afd0 msgr2=0x5600dd811700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.761+0000 7fc79e593740 1 --2- 172.21.15.31:0/3293856610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80afd0 0x5600dd811700 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fc7880b99a0 tx=0x7fc788005c90 comp rx=0 tx=0).stop 2024-03-21T10:49:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.761+0000 7fc79e593740 1 -- 172.21.15.31:0/3293856610 shutdown_connections 2024-03-21T10:49:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.761+0000 7fc79e593740 1 --2- 172.21.15.31:0/3293856610 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5600dd8242e0 0x5600dd8266d0 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.761+0000 7fc79e593740 1 --2- 172.21.15.31:0/3293856610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600dd8146c0 0x5600dd812180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.761+0000 7fc77aa00640 0 client.28465 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.761+0000 7fc79e593740 1 --2- 172.21.15.31:0/3293856610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600dd80b8f0 0x5600dd811c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.761+0000 7fc79e593740 1 --2- 172.21.15.31:0/3293856610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80afd0 0x5600dd811700 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.761+0000 7fc79e593740 1 -- 172.21.15.31:0/3293856610 >> 172.21.15.31:0/3293856610 conn(0x5600dd8058f0 msgr2=0x5600dd807640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:57.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.762+0000 7fc79e593740 1 -- 172.21.15.31:0/3293856610 shutdown_connections 2024-03-21T10:49:57.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.762+0000 7fc79e593740 1 -- 172.21.15.31:0/3293856610 wait complete. 2024-03-21T10:49:57.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.762+0000 7fc79e593740 20 client.28465 trim_cache size 0 max 16384 2024-03-21T10:49:57.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.777+0000 7fc79e593740 1 Processor -- start 2024-03-21T10:49:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.777+0000 7fc79e593740 1 -- start start 2024-03-21T10:49:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.777+0000 7fc79e593740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80afd0 0x5600dd813e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.777+0000 7fc79e593740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600dd80b8f0 0x5600dd80e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.777+0000 7fc79e593740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600dd8146c0 0x5600dd80eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.777+0000 7fc79e593740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7941b3120 con 0x5600dd80b8f0 2024-03-21T10:49:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.777+0000 7fc79e593740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7941b0c20 con 0x5600dd8146c0 2024-03-21T10:49:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.777+0000 7fc79e593740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7941aece0 con 0x5600dd80afd0 2024-03-21T10:49:57.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.777+0000 7fc799800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600dd8146c0 0x5600dd80eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:57.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.777+0000 7fc798e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80afd0 0x5600dd813e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:57.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.777+0000 7fc793e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600dd80b8f0 0x5600dd80e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:57.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.777+0000 7fc799800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600dd8146c0 0x5600dd80eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:54518/0 (socket says 172.21.15.31:54518) 2024-03-21T10:49:57.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.777+0000 7fc798e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80afd0 0x5600dd813e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:49930/0 (socket says 172.21.15.31:49930) 2024-03-21T10:49:57.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.777+0000 7fc799800640 1 -- 172.21.15.31:0/3809749056 learned_addr learned my addr 172.21.15.31:0/3809749056 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:57.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.778+0000 7fc798e00640 1 -- 172.21.15.31:0/3809749056 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600dd8146c0 msgr2=0x5600dd80eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:57.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.778+0000 7fc798e00640 1 --2- 172.21.15.31:0/3809749056 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600dd8146c0 0x5600dd80eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.778+0000 7fc798e00640 1 -- 172.21.15.31:0/3809749056 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600dd80b8f0 msgr2=0x5600dd80e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.778+0000 7fc798e00640 1 --2- 172.21.15.31:0/3809749056 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600dd80b8f0 0x5600dd80e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.778+0000 7fc798e00640 1 -- 172.21.15.31:0/3809749056 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600dd80b720 con 0x5600dd80afd0 2024-03-21T10:49:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.778+0000 7fc798e00640 1 --2- 172.21.15.31:0/3809749056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80afd0 0x5600dd813e30 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fc788005d00 tx=0x7fc788005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbf74b0f60e87592 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.778+0000 7fc785600640 1 -- 172.21.15.31:0/3809749056 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78804b070 con 0x5600dd80afd0 2024-03-21T10:49:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.778+0000 7fc785600640 1 -- 172.21.15.31:0/3809749056 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7880f1150 con 0x5600dd80afd0 2024-03-21T10:49:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.778+0000 7fc785600640 1 -- 172.21.15.31:0/3809749056 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7880f9ea0 con 0x5600dd80afd0 2024-03-21T10:49:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.779+0000 7fc79e593740 1 -- 172.21.15.31:0/3809749056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80afd0 msgr2=0x5600dd813e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.779+0000 7fc79e593740 1 --2- 172.21.15.31:0/3809749056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80afd0 0x5600dd813e30 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fc788005d00 tx=0x7fc788005fd0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.779+0000 7fc79e593740 1 -- 172.21.15.31:0/3809749056 shutdown_connections 2024-03-21T10:49:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.779+0000 7fc79e593740 1 --2- 172.21.15.31:0/3809749056 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600dd8146c0 0x5600dd80eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.779+0000 7fc79e593740 1 --2- 172.21.15.31:0/3809749056 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600dd80b8f0 0x5600dd80e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.779+0000 7fc79e593740 1 --2- 172.21.15.31:0/3809749056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80afd0 0x5600dd813e30 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.779+0000 7fc79e593740 1 -- 172.21.15.31:0/3809749056 >> 172.21.15.31:0/3809749056 conn(0x5600dd8058f0 msgr2=0x5600dd805cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:57.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.779+0000 7fc79e593740 1 -- 172.21.15.31:0/3809749056 shutdown_connections 2024-03-21T10:49:57.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.779+0000 7fc79e593740 1 -- 172.21.15.31:0/3809749056 wait complete. 2024-03-21T10:49:57.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.779+0000 7fc79e593740 1 -- start start 2024-03-21T10:49:57.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.780+0000 7fc79e593740 1 Processor -- start 2024-03-21T10:49:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.780+0000 7fc79e593740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80afd0 0x5600dd828e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.780+0000 7fc79e593740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600dd80b8f0 0x5600dd8293c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.780+0000 7fc79e593740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600dd8146c0 0x5600dd8272b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.780+0000 7fc79e593740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7941af2e0 con 0x5600dd8146c0 2024-03-21T10:49:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.780+0000 7fc79e593740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7941b05c0 con 0x5600dd80b8f0 2024-03-21T10:49:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.780+0000 7fc79e593740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7941af900 con 0x5600dd80afd0 2024-03-21T10:49:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.780+0000 7fc79e593740 10 client.0 _ll_register_callbacks cb 0x7fc78c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:49:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.780+0000 7fc798e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80afd0 0x5600dd828e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.780+0000 7fc793e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600dd80b8f0 0x5600dd8293c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.780+0000 7fc798e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80afd0 0x5600dd828e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:49946/0 (socket says 172.21.15.31:49946) 2024-03-21T10:49:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.780+0000 7fc798e00640 1 -- 172.21.15.31:0/2577920677 learned_addr learned my addr 172.21.15.31:0/2577920677 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:49:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.780+0000 7fc799800640 1 --2- 172.21.15.31:0/2577920677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600dd8146c0 0x5600dd8272b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.781+0000 7fc798e00640 1 -- 172.21.15.31:0/2577920677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600dd80b8f0 msgr2=0x5600dd8293c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.781+0000 7fc798e00640 1 --2- 172.21.15.31:0/2577920677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600dd80b8f0 0x5600dd8293c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.781+0000 7fc798e00640 1 -- 172.21.15.31:0/2577920677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600dd8146c0 msgr2=0x5600dd8272b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:49:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.781+0000 7fc798e00640 1 --2- 172.21.15.31:0/2577920677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600dd8146c0 0x5600dd8272b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.781+0000 7fc798e00640 1 -- 172.21.15.31:0/2577920677 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5600dd827ab0 con 0x5600dd80afd0 2024-03-21T10:49:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.781+0000 7fc793e00640 1 --2- 172.21.15.31:0/2577920677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600dd80b8f0 0x5600dd8293c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:49:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.781+0000 7fc798e00640 1 --2- 172.21.15.31:0/2577920677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80afd0 0x5600dd828e80 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fc788009430 tx=0x7fc788000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13d63e605dd9bf65 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:49:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.781+0000 7fc77aa00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:49:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.781+0000 7fc77aa00640 1 -- 172.21.15.31:0/2577920677 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78804b070 con 0x5600dd80afd0 2024-03-21T10:49:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.781+0000 7fc77aa00640 1 -- 172.21.15.31:0/2577920677 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fc7880c5a10 con 0x5600dd80afd0 2024-03-21T10:49:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.781+0000 7fc79e593740 1 -- 172.21.15.31:0/2577920677 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc7880f1150 con 0x5600dd80afd0 2024-03-21T10:49:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.781+0000 7fc77aa00640 1 -- 172.21.15.31:0/2577920677 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc7880bcea0 con 0x5600dd80afd0 2024-03-21T10:49:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.781+0000 7fc79e593740 10 client.28471 fetch_fsmap learned FSMap version 30 2024-03-21T10:49:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.781+0000 7fc79e593740 1 -- 172.21.15.31:0/2577920677 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5600dd827ab0 con 0x5600dd80afd0 2024-03-21T10:49:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.782+0000 7fc77aa00640 1 -- 172.21.15.31:0/2577920677 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7880f1150 con 0x5600dd80afd0 2024-03-21T10:49:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.782+0000 7fc77aa00640 1 -- 172.21.15.31:0/2577920677 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fc7880bd7a0 con 0x5600dd80afd0 2024-03-21T10:49:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.782+0000 7fc79e593740 10 client.28471 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:49:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.782+0000 7fc79e593740 10 client.28471 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:49:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.782+0000 7fc79e593740 20 client.28471 populate_metadata read hostname 'smithi031' 2024-03-21T10:49:57.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.782+0000 7fc79e593740 1 --2- 172.21.15.31:0/2577920677 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5600dd8242e0 0x5600dd8824d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:49:57.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.782+0000 7fc79e593740 4 client.28471 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:49:57.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.782+0000 7fc79e593740 1 -- 172.21.15.31:0/2577920677 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5600dd7770c0 con 0x5600dd8242e0 2024-03-21T10:49:57.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.782+0000 7fc793e00640 1 --2- 172.21.15.31:0/2577920677 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5600dd8242e0 0x5600dd8824d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:49:57.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.782+0000 7fc793e00640 1 --2- 172.21.15.31:0/2577920677 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5600dd8242e0 0x5600dd8824d0 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5690ab690db331d server_cookie=a07bf72ad352e397 in_seq=0 out_seq=0 2024-03-21T10:49:57.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.782+0000 7fc77aa00640 10 client.28471 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:57.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.786+0000 7fc77aa00640 1 -- 172.21.15.31:0/2577920677 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5600dd7770c0 con 0x5600dd8242e0 2024-03-21T10:49:57.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.786+0000 7fc77aa00640 10 client.28471 handle_command_reply: tid=0 2024-03-21T10:49:57.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.786+0000 7fc79e593740 1 client.28471 shutdown 2024-03-21T10:49:57.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.786+0000 7fc79e593740 1 -- 172.21.15.31:0/2577920677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80afd0 msgr2=0x5600dd828e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:57.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.786+0000 7fc79e593740 1 --2- 172.21.15.31:0/2577920677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80afd0 0x5600dd828e80 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fc788009430 tx=0x7fc788000bc0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.786+0000 7fc79e593740 1 -- 172.21.15.31:0/2577920677 shutdown_connections 2024-03-21T10:49:57.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.786+0000 7fc79e593740 1 --2- 172.21.15.31:0/2577920677 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5600dd8242e0 0x5600dd8824d0 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.786+0000 7fc79e593740 1 --2- 172.21.15.31:0/2577920677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600dd8146c0 0x5600dd8272b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.786+0000 7fc79e593740 1 --2- 172.21.15.31:0/2577920677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600dd80b8f0 0x5600dd8293c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.786+0000 7fc79e593740 1 --2- 172.21.15.31:0/2577920677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dd80afd0 0x5600dd828e80 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.786+0000 7fc77aa00640 0 client.28471 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:49:57.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.786+0000 7fc79e593740 1 -- 172.21.15.31:0/2577920677 >> 172.21.15.31:0/2577920677 conn(0x5600dd8058f0 msgr2=0x5600dd806d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:57.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.786+0000 7fc79e593740 1 -- 172.21.15.31:0/2577920677 shutdown_connections 2024-03-21T10:49:57.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.786+0000 7fc79e593740 1 -- 172.21.15.31:0/2577920677 wait complete. 2024-03-21T10:49:57.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.787+0000 7fc79e593740 20 client.28471 trim_cache size 0 max 16384 2024-03-21T10:49:57.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:49:57.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:49:57.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:49:57.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:49:57.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.787+0000 7fc79b800640 1 -- 172.21.15.31:0/1421872700 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc77c0a3220 msgr2=0x7fc77c0a56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:57.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.787+0000 7fc79b800640 1 --2- 172.21.15.31:0/1421872700 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc77c0a3220 0x7fc77c0a56e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc78c005fd0 tx=0x7fc78c009500 comp rx=0 tx=0).stop 2024-03-21T10:49:57.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.787+0000 7fc79b800640 1 -- 172.21.15.31:0/1421872700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc794103870 msgr2=0x7fc79419a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:49:57.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.787+0000 7fc79b800640 1 --2- 172.21.15.31:0/1421872700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc794103870 0x7fc79419a1d0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fc788009230 tx=0x7fc78802cb30 comp rx=0 tx=0).stop 2024-03-21T10:49:57.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.787+0000 7fc79b800640 1 -- 172.21.15.31:0/1421872700 shutdown_connections 2024-03-21T10:49:57.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.787+0000 7fc79b800640 1 --2- 172.21.15.31:0/1421872700 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc77c0a3220 0x7fc77c0a56e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.787+0000 7fc79b800640 1 --2- 172.21.15.31:0/1421872700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc794109880 0x7fc79419ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.787+0000 7fc79b800640 1 --2- 172.21.15.31:0/1421872700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc794104230 0x7fc79419a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.787+0000 7fc79b800640 1 --2- 172.21.15.31:0/1421872700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc794103870 0x7fc79419a1d0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:49:57.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.787+0000 7fc79b800640 1 -- 172.21.15.31:0/1421872700 >> 172.21.15.31:0/1421872700 conn(0x7fc7940ff320 msgr2=0x7fc7940ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:49:57.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.788+0000 7fc79b800640 1 -- 172.21.15.31:0/1421872700 shutdown_connections 2024-03-21T10:49:57.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:49:57.788+0000 7fc79b800640 1 -- 172.21.15.31:0/1421872700 wait complete. 2024-03-21T10:49:59.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:49:59 smithi031 ceph-mon[31673]: pgmap v355: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-21T10:49:59.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:49:59 smithi070 ceph-mon[36330]: pgmap v355: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-21T10:49:59.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:49:59 smithi040 ceph-mon[36154]: pgmap v355: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-21T10:50:00.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:00 smithi031 ceph-mon[31673]: overall HEALTH_OK 2024-03-21T10:50:00.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:00 smithi070 ceph-mon[36330]: overall HEALTH_OK 2024-03-21T10:50:00.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:00 smithi040 ceph-mon[36154]: overall HEALTH_OK 2024-03-21T10:50:01.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:01 smithi031 ceph-mon[31673]: pgmap v356: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-21T10:50:01.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:01 smithi070 ceph-mon[36330]: pgmap v356: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-21T10:50:01.331 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:01 smithi040 ceph-mon[36154]: pgmap v356: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-21T10:50:02.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:02 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:50:02.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:02 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:50:02.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:02 smithi031 ceph-mon[31673]: scrub summary: idle 2024-03-21T10:50:02.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:02 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:50:02.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:02 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:50:02.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:02 smithi070 ceph-mon[36330]: scrub summary: idle 2024-03-21T10:50:02.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:02 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:50:02.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:02 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:50:02.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:02 smithi040 ceph-mon[36154]: scrub summary: idle 2024-03-21T10:50:03.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:03 smithi031 ceph-mon[31673]: pgmap v357: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:50:03.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:03 smithi070 ceph-mon[36330]: pgmap v357: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:50:03.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:03 smithi040 ceph-mon[36154]: pgmap v357: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-21T10:50:05.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:05 smithi031 ceph-mon[31673]: pgmap v358: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-21T10:50:05.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:05 smithi070 ceph-mon[36330]: pgmap v358: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-21T10:50:05.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:05 smithi040 ceph-mon[36154]: pgmap v358: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-21T10:50:07.303 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:07 smithi031 ceph-mon[31673]: pgmap v359: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 46 KiB/s wr, 25 op/s 2024-03-21T10:50:07.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:07 smithi070 ceph-mon[36330]: pgmap v359: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 46 KiB/s wr, 25 op/s 2024-03-21T10:50:07.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:07 smithi040 ceph-mon[36154]: pgmap v359: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 46 KiB/s wr, 25 op/s 2024-03-21T10:50:08.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:08 smithi070 ceph-mon[36330]: osdmap e105: 12 total, 12 up, 12 in 2024-03-21T10:50:08.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:08 smithi040 ceph-mon[36154]: osdmap e105: 12 total, 12 up, 12 in 2024-03-21T10:50:08.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:08 smithi031 ceph-mon[31673]: osdmap e105: 12 total, 12 up, 12 in 2024-03-21T10:50:09.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:09 smithi070 ceph-mon[36330]: pgmap v361: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 55 KiB/s wr, 27 op/s 2024-03-21T10:50:09.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:09 smithi040 ceph-mon[36154]: pgmap v361: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 55 KiB/s wr, 27 op/s 2024-03-21T10:50:09.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:09 smithi031 ceph-mon[31673]: pgmap v361: 129 pgs: 129 active+clean; 26 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 55 KiB/s wr, 27 op/s 2024-03-21T10:50:10.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.969+0000 7f72ca400640 1 -- 172.21.15.31:0/343462961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72c4106450 msgr2=0x7f72c4068e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:10.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.969+0000 7f72ca400640 1 --2- 172.21.15.31:0/343462961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72c4106450 0x7f72c4068e90 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f72b8009900 tx=0x7f72b802bc30 comp rx=0 tx=0).stop 2024-03-21T10:50:10.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.969+0000 7f72ca400640 1 -- 172.21.15.31:0/343462961 shutdown_connections 2024-03-21T10:50:10.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.969+0000 7f72ca400640 1 --2- 172.21.15.31:0/343462961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72c4103ee0 0x7f72c4104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:10.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.969+0000 7f72ca400640 1 --2- 172.21.15.31:0/343462961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72c40693d0 0x7f72c4069830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:10.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.969+0000 7f72ca400640 1 --2- 172.21.15.31:0/343462961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72c4106450 0x7f72c4068e90 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:10.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.969+0000 7f72ca400640 1 -- 172.21.15.31:0/343462961 >> 172.21.15.31:0/343462961 conn(0x7f72c4075f00 msgr2=0x7f72c4076310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:10.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.969+0000 7f72ca400640 1 -- 172.21.15.31:0/343462961 shutdown_connections 2024-03-21T10:50:10.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.969+0000 7f72ca400640 1 -- 172.21.15.31:0/343462961 wait complete. 2024-03-21T10:50:10.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.970+0000 7f72ca400640 1 Processor -- start 2024-03-21T10:50:10.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.971+0000 7f72ca400640 1 -- start start 2024-03-21T10:50:10.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.971+0000 7f72ca400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72c40693d0 0x7f72c419e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:10.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.971+0000 7f72ca400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72c4103ee0 0x7f72c419e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:10.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.971+0000 7f72ca400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72c4106450 0x7f72c41a2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:10.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.971+0000 7f72ca400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c4117e30 con 0x7f72c4103ee0 2024-03-21T10:50:10.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.971+0000 7f72ca400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c4117cb0 con 0x7f72c40693d0 2024-03-21T10:50:10.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.971+0000 7f72ca400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c4117fb0 con 0x7f72c4106450 2024-03-21T10:50:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.971+0000 7f72c9000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72c40693d0 0x7f72c419e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.971+0000 7f72c9000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72c40693d0 0x7f72c419e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:48122/0 (socket says 172.21.15.31:48122) 2024-03-21T10:50:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.971+0000 7f72c9000640 1 -- 172.21.15.31:0/1615792274 learned_addr learned my addr 172.21.15.31:0/1615792274 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.971+0000 7f72c9a00640 1 --2- 172.21.15.31:0/1615792274 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72c4106450 0x7f72c41a2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.971+0000 7f72c3e00640 1 --2- 172.21.15.31:0/1615792274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72c4103ee0 0x7f72c419e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.972+0000 7f72c9000640 1 -- 172.21.15.31:0/1615792274 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72c4106450 msgr2=0x7f72c41a2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.972+0000 7f72c9000640 1 --2- 172.21.15.31:0/1615792274 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72c4106450 0x7f72c41a2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.972+0000 7f72c9000640 1 -- 172.21.15.31:0/1615792274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72c4103ee0 msgr2=0x7f72c419e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.972+0000 7f72c9000640 1 --2- 172.21.15.31:0/1615792274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72c4103ee0 0x7f72c419e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.972+0000 7f72c9000640 1 -- 172.21.15.31:0/1615792274 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c41a3200 con 0x7f72c40693d0 2024-03-21T10:50:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.972+0000 7f72c9a00640 1 --2- 172.21.15.31:0/1615792274 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72c4106450 0x7f72c41a2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:50:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.972+0000 7f72c3e00640 1 --2- 172.21.15.31:0/1615792274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72c4103ee0 0x7f72c419e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:50:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.972+0000 7f72c9000640 1 --2- 172.21.15.31:0/1615792274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72c40693d0 0x7f72c419e210 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f72b8036040 tx=0x7f72b80042d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.972+0000 7f72c1600640 1 -- 172.21.15.31:0/1615792274 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b8043020 con 0x7f72c40693d0 2024-03-21T10:50:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.972+0000 7f72c1600640 1 -- 172.21.15.31:0/1615792274 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72b80054e0 con 0x7f72c40693d0 2024-03-21T10:50:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.973+0000 7f72c1600640 1 -- 172.21.15.31:0/1615792274 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b8003be0 con 0x7f72c40693d0 2024-03-21T10:50:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.973+0000 7f72ca400640 1 -- 172.21.15.31:0/1615792274 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72c41a34f0 con 0x7f72c40693d0 2024-03-21T10:50:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.973+0000 7f72ca400640 1 -- 172.21.15.31:0/1615792274 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72c41aad80 con 0x7f72c40693d0 2024-03-21T10:50:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.977+0000 7f72ce76f740 1 Processor -- start 2024-03-21T10:50:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.977+0000 7f72c1600640 1 -- 172.21.15.31:0/1615792274 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f72b8046020 con 0x7f72c40693d0 2024-03-21T10:50:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.977+0000 7f72ce76f740 1 -- start start 2024-03-21T10:50:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.978+0000 7f72ce76f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56440261a1e0 0x56440261a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.978+0000 7f72ce76f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56440261ab00 0x564402623390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:10.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.978+0000 7f72ce76f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644026238d0 0x564402625cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:10.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.978+0000 7f72ce76f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644025dc3b0 con 0x56440261ab00 2024-03-21T10:50:10.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.978+0000 7f72ce76f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56440257e830 con 0x56440261a1e0 2024-03-21T10:50:10.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.978+0000 7f72ce76f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644025275d0 con 0x5644026238d0 2024-03-21T10:50:10.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.978+0000 7f72c9a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56440261a1e0 0x56440261a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:10.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.978+0000 7f72c3e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644026238d0 0x564402625cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:10.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.978+0000 7f72c3e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644026238d0 0x564402625cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:55014/0 (socket says 172.21.15.31:55014) 2024-03-21T10:50:10.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.978+0000 7f72c3e00640 1 -- 172.21.15.31:0/2382016602 learned_addr learned my addr 172.21.15.31:0/2382016602 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:10.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.978+0000 7f72c9a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56440261a1e0 0x56440261a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:48132/0 (socket says 172.21.15.31:48132) 2024-03-21T10:50:10.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.978+0000 7f72c9000640 1 --2- 172.21.15.31:0/2382016602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56440261ab00 0x564402623390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:10.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.978+0000 7f72c3e00640 1 -- 172.21.15.31:0/2382016602 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56440261a1e0 msgr2=0x56440261a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:10.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.978+0000 7f72c3e00640 1 --2- 172.21.15.31:0/2382016602 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56440261a1e0 0x56440261a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.978+0000 7f72c3e00640 1 -- 172.21.15.31:0/2382016602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56440261ab00 msgr2=0x564402623390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.978+0000 7f72c3e00640 1 --2- 172.21.15.31:0/2382016602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56440261ab00 0x564402623390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.978+0000 7f72c3e00640 1 -- 172.21.15.31:0/2382016602 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56440253f2c0 con 0x5644026238d0 2024-03-21T10:50:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.978+0000 7f72c9000640 1 --2- 172.21.15.31:0/2382016602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56440261ab00 0x564402623390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:50:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.979+0000 7f72c3e00640 1 --2- 172.21.15.31:0/2382016602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644026238d0 0x564402625cc0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f72b4009cf0 tx=0x7f72b40117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db6e67c16d057401 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.979+0000 7f72c1600640 1 --2- 172.21.15.31:0/1615792274 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f72ac0a30f0 0x7f72ac0a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.979+0000 7f72b2000640 1 -- 172.21.15.31:0/2382016602 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b4012450 con 0x5644026238d0 2024-03-21T10:50:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.979+0000 7f72b2000640 1 -- 172.21.15.31:0/2382016602 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72b4012ad0 con 0x5644026238d0 2024-03-21T10:50:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.979+0000 7f72c1600640 1 -- 172.21.15.31:0/1615792274 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f72b80f1ba0 con 0x7f72c40693d0 2024-03-21T10:50:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.979+0000 7f72b2000640 1 -- 172.21.15.31:0/2382016602 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b400fca0 con 0x5644026238d0 2024-03-21T10:50:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.979+0000 7f72c9a00640 1 --2- 172.21.15.31:0/1615792274 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f72ac0a30f0 0x7f72ac0a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.980+0000 7f72ce76f740 1 -- 172.21.15.31:0/2382016602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644026238d0 msgr2=0x564402625cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.980+0000 7f72ce76f740 1 --2- 172.21.15.31:0/2382016602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644026238d0 0x564402625cc0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f72b4009cf0 tx=0x7f72b40117b0 comp rx=0 tx=0).stop 2024-03-21T10:50:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.980+0000 7f72c9a00640 1 --2- 172.21.15.31:0/1615792274 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f72ac0a30f0 0x7f72ac0a55b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f72bc005fd0 tx=0x7f72bc0096a0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:10.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.981+0000 7f72ce76f740 1 -- 172.21.15.31:0/2382016602 shutdown_connections 2024-03-21T10:50:10.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.981+0000 7f72ce76f740 1 --2- 172.21.15.31:0/2382016602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644026238d0 0x564402625cc0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:10.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.981+0000 7f72ce76f740 1 --2- 172.21.15.31:0/2382016602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56440261ab00 0x564402623390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:10.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.981+0000 7f72ce76f740 1 --2- 172.21.15.31:0/2382016602 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56440261a1e0 0x56440261a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:10.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.981+0000 7f72ce76f740 1 -- 172.21.15.31:0/2382016602 >> 172.21.15.31:0/2382016602 conn(0x564402614b00 msgr2=0x564402614ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:10.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.981+0000 7f72ce76f740 1 -- 172.21.15.31:0/2382016602 shutdown_connections 2024-03-21T10:50:10.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.981+0000 7f72ce76f740 1 -- 172.21.15.31:0/2382016602 wait complete. 2024-03-21T10:50:10.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.982+0000 7f72ce76f740 1 -- start start 2024-03-21T10:50:10.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.982+0000 7f72ce76f740 1 Processor -- start 2024-03-21T10:50:10.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.982+0000 7f72ce76f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56440261a1e0 0x564402620950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:10.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.982+0000 7f72ce76f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56440261ab00 0x564402620e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:10.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.982+0000 7f72ce76f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5644026238d0 0x5644026213d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:10.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.982+0000 7f72ce76f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56440253aa20 con 0x5644026238d0 2024-03-21T10:50:10.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.982+0000 7f72ce76f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56440257a8d0 con 0x56440261a1e0 2024-03-21T10:50:10.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.982+0000 7f72ce76f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x564402626500 con 0x56440261ab00 2024-03-21T10:50:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.982+0000 7f72ce76f740 10 client.0 _ll_register_callbacks cb 0x5644025771c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:50:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.983+0000 7f72c3e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56440261ab00 0x564402620e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.983+0000 7f72c3e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56440261ab00 0x564402620e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:55026/0 (socket says 172.21.15.31:55026) 2024-03-21T10:50:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.983+0000 7f72c3e00640 1 -- 172.21.15.31:0/3637941236 learned_addr learned my addr 172.21.15.31:0/3637941236 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.983+0000 7f72c9000640 1 --2- 172.21.15.31:0/3637941236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56440261a1e0 0x564402620950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.983+0000 7f72c9a00640 1 --2- 172.21.15.31:0/3637941236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5644026238d0 0x5644026213d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.983+0000 7f72c3e00640 1 -- 172.21.15.31:0/3637941236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56440261a1e0 msgr2=0x564402620950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.983+0000 7f72c3e00640 1 --2- 172.21.15.31:0/3637941236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56440261a1e0 0x564402620950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.983+0000 7f72c3e00640 1 -- 172.21.15.31:0/3637941236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5644026238d0 msgr2=0x5644026213d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.983+0000 7f72c3e00640 1 --2- 172.21.15.31:0/3637941236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5644026238d0 0x5644026213d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.983+0000 7f72c3e00640 1 -- 172.21.15.31:0/3637941236 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564402293210 con 0x56440261ab00 2024-03-21T10:50:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.983+0000 7f72c9000640 1 --2- 172.21.15.31:0/3637941236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56440261a1e0 0x564402620950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:50:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.984+0000 7f72c3e00640 1 --2- 172.21.15.31:0/3637941236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56440261ab00 0x564402620e90 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f72b401c040 tx=0x7f72b4009a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=875302ff1889b110 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.984+0000 7f72a6a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:50:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.984+0000 7f72a6a00640 1 -- 172.21.15.31:0/3637941236 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b4028020 con 0x56440261ab00 2024-03-21T10:50:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.984+0000 7f72a6a00640 1 -- 172.21.15.31:0/3637941236 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f72b4004310 con 0x56440261ab00 2024-03-21T10:50:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.984+0000 7f72ce76f740 1 -- 172.21.15.31:0/3637941236 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f72b4012ad0 con 0x56440261ab00 2024-03-21T10:50:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.985+0000 7f72a6a00640 1 -- 172.21.15.31:0/3637941236 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f72b4033050 con 0x56440261ab00 2024-03-21T10:50:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.985+0000 7f72ce76f740 10 client.28483 fetch_fsmap learned FSMap version 30 2024-03-21T10:50:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.985+0000 7f72ce76f740 1 -- 172.21.15.31:0/3637941236 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x56440235f620 con 0x56440261ab00 2024-03-21T10:50:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.985+0000 7f72a6a00640 1 -- 172.21.15.31:0/3637941236 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72b4012ad0 con 0x56440261ab00 2024-03-21T10:50:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.985+0000 7f72a6a00640 1 -- 172.21.15.31:0/3637941236 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f72b400f8a0 con 0x56440261ab00 2024-03-21T10:50:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.985+0000 7f72ce76f740 10 client.28483 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:50:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.985+0000 7f72ce76f740 10 client.28483 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:50:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.985+0000 7f72ce76f740 20 client.28483 populate_metadata read hostname 'smithi031' 2024-03-21T10:50:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.985+0000 7f72ce76f740 1 --2- 172.21.15.31:0/3637941236 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5644026335d0 0x5644026359c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.985+0000 7f72ce76f740 4 client.28483 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:50:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.985+0000 7f72ce76f740 1 -- 172.21.15.31:0/3637941236 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56440258bd70 con 0x5644026335d0 2024-03-21T10:50:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.986+0000 7f72c9000640 1 --2- 172.21.15.31:0/3637941236 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5644026335d0 0x5644026359c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.986+0000 7f72c9000640 1 --2- 172.21.15.31:0/3637941236 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5644026335d0 0x5644026359c0 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d70434994e2c1a7 server_cookie=2ea887bf3a2ead5e in_seq=0 out_seq=0 2024-03-21T10:50:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.986+0000 7f72a6a00640 10 client.28483 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:50:10.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.987+0000 7f72a6a00640 1 -- 172.21.15.31:0/3637941236 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56440258bd70 con 0x5644026335d0 2024-03-21T10:50:10.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.987+0000 7f72a6a00640 10 client.28483 handle_command_reply: tid=0 2024-03-21T10:50:10.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.987+0000 7f72ce76f740 1 client.28483 shutdown 2024-03-21T10:50:10.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.988+0000 7f72ce76f740 1 -- 172.21.15.31:0/3637941236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56440261ab00 msgr2=0x564402620e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:10.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.988+0000 7f72ce76f740 1 --2- 172.21.15.31:0/3637941236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56440261ab00 0x564402620e90 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f72b401c040 tx=0x7f72b4009a10 comp rx=0 tx=0).stop 2024-03-21T10:50:10.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.988+0000 7f72ce76f740 1 -- 172.21.15.31:0/3637941236 shutdown_connections 2024-03-21T10:50:10.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.988+0000 7f72ce76f740 1 --2- 172.21.15.31:0/3637941236 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x5644026335d0 0x5644026359c0 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.988+0000 7f72ce76f740 1 --2- 172.21.15.31:0/3637941236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5644026238d0 0x5644026213d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.988+0000 7f72ce76f740 1 --2- 172.21.15.31:0/3637941236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56440261ab00 0x564402620e90 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.988+0000 7f72ce76f740 1 --2- 172.21.15.31:0/3637941236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56440261a1e0 0x564402620950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.988+0000 7f72ce76f740 1 -- 172.21.15.31:0/3637941236 >> 172.21.15.31:0/3637941236 conn(0x564402614b00 msgr2=0x564402616850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.988+0000 7f72a6a00640 0 client.28483 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:50:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.988+0000 7f72ce76f740 1 -- 172.21.15.31:0/3637941236 shutdown_connections 2024-03-21T10:50:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.989+0000 7f72ce76f740 1 -- 172.21.15.31:0/3637941236 wait complete. 2024-03-21T10:50:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:10.989+0000 7f72ce76f740 20 client.28483 trim_cache size 0 max 16384 2024-03-21T10:50:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.010+0000 7f72ce76f740 1 Processor -- start 2024-03-21T10:50:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.010+0000 7f72ce76f740 1 -- start start 2024-03-21T10:50:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.010+0000 7f72ce76f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56440261a1e0 0x564402623080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.010+0000 7f72ce76f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56440261ab00 0x5644026211f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.010+0000 7f72ce76f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5644026238d0 0x56440262ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.010+0000 7f72ce76f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c41b73c0 con 0x56440261a1e0 2024-03-21T10:50:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.010+0000 7f72ce76f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c41b4ec0 con 0x5644026238d0 2024-03-21T10:50:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.010+0000 7f72ce76f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c41b2f80 con 0x56440261ab00 2024-03-21T10:50:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.011+0000 7f72c9a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5644026238d0 0x56440262ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.011+0000 7f72c9a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5644026238d0 0x56440262ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:48150/0 (socket says 172.21.15.31:48150) 2024-03-21T10:50:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.011+0000 7f72c9a00640 1 -- 172.21.15.31:0/1523714455 learned_addr learned my addr 172.21.15.31:0/1523714455 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.011+0000 7f72c3e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56440261ab00 0x5644026211f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.011+0000 7f72c9000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56440261a1e0 0x564402623080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:11.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.011+0000 7f72c9000640 1 -- 172.21.15.31:0/1523714455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56440261ab00 msgr2=0x5644026211f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:11.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.011+0000 7f72c9000640 1 --2- 172.21.15.31:0/1523714455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56440261ab00 0x5644026211f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:11.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.011+0000 7f72c9000640 1 -- 172.21.15.31:0/1523714455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5644026238d0 msgr2=0x56440262ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:11.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.011+0000 7f72c9000640 1 --2- 172.21.15.31:0/1523714455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5644026238d0 0x56440262ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:11.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.011+0000 7f72c9000640 1 -- 172.21.15.31:0/1523714455 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564402612f80 con 0x56440261a1e0 2024-03-21T10:50:11.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.012+0000 7f72c9000640 1 --2- 172.21.15.31:0/1523714455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56440261a1e0 0x564402623080 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f72b80eefb0 tx=0x7f72b80f3f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cb400d6e877a8d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:11.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.012+0000 7f72b1600640 1 -- 172.21.15.31:0/1523714455 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b8004c80 con 0x56440261a1e0 2024-03-21T10:50:11.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.012+0000 7f72b1600640 1 -- 172.21.15.31:0/1523714455 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72b8005690 con 0x56440261a1e0 2024-03-21T10:50:11.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.012+0000 7f72b1600640 1 -- 172.21.15.31:0/1523714455 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b8003d80 con 0x56440261a1e0 2024-03-21T10:50:11.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.013+0000 7f72ce76f740 1 -- 172.21.15.31:0/1523714455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56440261a1e0 msgr2=0x564402623080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:11.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.013+0000 7f72ce76f740 1 --2- 172.21.15.31:0/1523714455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56440261a1e0 0x564402623080 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f72b80eefb0 tx=0x7f72b80f3f70 comp rx=0 tx=0).stop 2024-03-21T10:50:11.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.013+0000 7f72ce76f740 1 -- 172.21.15.31:0/1523714455 shutdown_connections 2024-03-21T10:50:11.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.013+0000 7f72ce76f740 1 --2- 172.21.15.31:0/1523714455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5644026238d0 0x56440262ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:11.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.013+0000 7f72ce76f740 1 --2- 172.21.15.31:0/1523714455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56440261ab00 0x5644026211f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:11.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.013+0000 7f72ce76f740 1 --2- 172.21.15.31:0/1523714455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56440261a1e0 0x564402623080 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:11.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.013+0000 7f72ce76f740 1 -- 172.21.15.31:0/1523714455 >> 172.21.15.31:0/1523714455 conn(0x564402614b00 msgr2=0x56440261bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:11.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.013+0000 7f72ce76f740 1 -- 172.21.15.31:0/1523714455 shutdown_connections 2024-03-21T10:50:11.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.013+0000 7f72ce76f740 1 -- 172.21.15.31:0/1523714455 wait complete. 2024-03-21T10:50:11.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.014+0000 7f72ce76f740 1 -- start start 2024-03-21T10:50:11.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.014+0000 7f72ce76f740 1 Processor -- start 2024-03-21T10:50:11.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.014+0000 7f72ce76f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56440261a1e0 0x56440261e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:11.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.014+0000 7f72ce76f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56440261ab00 0x56440261cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:11.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.014+0000 7f72ce76f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5644026238d0 0x56440261d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:11.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.014+0000 7f72ce76f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c41b3580 con 0x5644026238d0 2024-03-21T10:50:11.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.015+0000 7f72ce76f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c41b4860 con 0x56440261ab00 2024-03-21T10:50:11.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.015+0000 7f72ce76f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c41b3ba0 con 0x56440261a1e0 2024-03-21T10:50:11.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.015+0000 7f72ce76f740 10 client.0 _ll_register_callbacks cb 0x7f72bc009e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:50:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.015+0000 7f72c9000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56440261a1e0 0x56440261e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.015+0000 7f72c3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56440261ab00 0x56440261cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.015+0000 7f72c9000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56440261a1e0 0x56440261e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:55042/0 (socket says 172.21.15.31:55042) 2024-03-21T10:50:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.015+0000 7f72c9000640 1 -- 172.21.15.31:0/3568797116 learned_addr learned my addr 172.21.15.31:0/3568797116 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.015+0000 7f72c9a00640 1 --2- 172.21.15.31:0/3568797116 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5644026238d0 0x56440261d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.015+0000 7f72c9000640 1 -- 172.21.15.31:0/3568797116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56440261ab00 msgr2=0x56440261cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.015+0000 7f72c9000640 1 --2- 172.21.15.31:0/3568797116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56440261ab00 0x56440261cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.015+0000 7f72c9000640 1 -- 172.21.15.31:0/3568797116 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5644026238d0 msgr2=0x56440261d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.015+0000 7f72c9000640 1 --2- 172.21.15.31:0/3568797116 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5644026238d0 0x56440261d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.015+0000 7f72c9000640 1 -- 172.21.15.31:0/3568797116 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56440261d730 con 0x56440261a1e0 2024-03-21T10:50:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.016+0000 7f72c9000640 1 --2- 172.21.15.31:0/3568797116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56440261a1e0 0x56440261e4d0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f72b80bdd00 tx=0x7f72b80bd790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5ac4a1eadbf1819 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.016+0000 7f72a6a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:50:11.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.016+0000 7f72a6a00640 1 -- 172.21.15.31:0/3568797116 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b80be470 con 0x56440261a1e0 2024-03-21T10:50:11.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.016+0000 7f72a6a00640 1 -- 172.21.15.31:0/3568797116 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f72b80ef120 con 0x56440261a1e0 2024-03-21T10:50:11.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.016+0000 7f72ce76f740 1 -- 172.21.15.31:0/3568797116 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5644025860c0 con 0x56440261a1e0 2024-03-21T10:50:11.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.017+0000 7f72a6a00640 1 -- 172.21.15.31:0/3568797116 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f72b80a1050 con 0x56440261a1e0 2024-03-21T10:50:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.017+0000 7f72ce76f740 10 client.28489 fetch_fsmap learned FSMap version 30 2024-03-21T10:50:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.017+0000 7f72ce76f740 1 -- 172.21.15.31:0/3568797116 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x56440268e980 con 0x56440261a1e0 2024-03-21T10:50:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.017+0000 7f72a6a00640 1 -- 172.21.15.31:0/3568797116 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5644025860c0 con 0x56440261a1e0 2024-03-21T10:50:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.017+0000 7f72a6a00640 1 -- 172.21.15.31:0/3568797116 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 30) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f72b802cb90 con 0x56440261a1e0 2024-03-21T10:50:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.017+0000 7f72ce76f740 10 client.28489 fetch_fsmap finished waiting for FSMap version 30 2024-03-21T10:50:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.017+0000 7f72ce76f740 10 client.28489 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-21T10:50:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.017+0000 7f72ce76f740 20 client.28489 populate_metadata read hostname 'smithi031' 2024-03-21T10:50:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.017+0000 7f72ce76f740 1 --2- 172.21.15.31:0/3568797116 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56440268ef40 0x564402691360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.017+0000 7f72ce76f740 4 client.28489 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-21T10:50:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.017+0000 7f72ce76f740 1 -- 172.21.15.31:0/3568797116 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5644025045d0 con 0x56440268ef40 2024-03-21T10:50:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.018+0000 7f72c3e00640 1 --2- 172.21.15.31:0/3568797116 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56440268ef40 0x564402691360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.019+0000 7f72c3e00640 1 --2- 172.21.15.31:0/3568797116 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56440268ef40 0x564402691360 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76939eb950e68fff server_cookie=3a2dd84646ed56db in_seq=0 out_seq=0 2024-03-21T10:50:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.019+0000 7f72a6a00640 10 client.28489 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:50:11.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.025+0000 7f72a6a00640 1 -- 172.21.15.31:0/3568797116 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5644025045d0 con 0x56440268ef40 2024-03-21T10:50:11.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.025+0000 7f72a6a00640 10 client.28489 handle_command_reply: tid=0 2024-03-21T10:50:11.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.025+0000 7f72ce76f740 1 client.28489 shutdown 2024-03-21T10:50:11.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.025+0000 7f72ce76f740 1 -- 172.21.15.31:0/3568797116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56440261a1e0 msgr2=0x56440261e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:11.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.025+0000 7f72ce76f740 1 --2- 172.21.15.31:0/3568797116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56440261a1e0 0x56440261e4d0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f72b80bdd00 tx=0x7f72b80bd790 comp rx=0 tx=0).stop 2024-03-21T10:50:11.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.026+0000 7f72ce76f740 1 -- 172.21.15.31:0/3568797116 shutdown_connections 2024-03-21T10:50:11.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.026+0000 7f72ce76f740 1 --2- 172.21.15.31:0/3568797116 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x56440268ef40 0x564402691360 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:11.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.026+0000 7f72ce76f740 1 --2- 172.21.15.31:0/3568797116 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5644026238d0 0x56440261d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:11.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.026+0000 7f72ce76f740 1 --2- 172.21.15.31:0/3568797116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56440261ab00 0x56440261cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:11.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.026+0000 7f72ce76f740 1 --2- 172.21.15.31:0/3568797116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56440261a1e0 0x56440261e4d0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:11.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.026+0000 7f72ce76f740 1 -- 172.21.15.31:0/3568797116 >> 172.21.15.31:0/3568797116 conn(0x564402614b00 msgr2=0x564402615e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:11.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.026+0000 7f72a6a00640 0 client.28489 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:50:11.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.026+0000 7f72ce76f740 1 -- 172.21.15.31:0/3568797116 shutdown_connections 2024-03-21T10:50:11.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.026+0000 7f72ce76f740 1 -- 172.21.15.31:0/3568797116 wait complete. 2024-03-21T10:50:11.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.026+0000 7f72ce76f740 20 client.28489 trim_cache size 0 max 16384 2024-03-21T10:50:11.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-03-21T10:50:11.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-03-21T10:50:11.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-03-21T10:50:11.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-03-21T10:50:11.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.027+0000 7f72ca400640 1 -- 172.21.15.31:0/1615792274 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f72ac0a30f0 msgr2=0x7f72ac0a55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:11.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.027+0000 7f72ca400640 1 --2- 172.21.15.31:0/1615792274 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f72ac0a30f0 0x7f72ac0a55b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f72bc005fd0 tx=0x7f72bc0096a0 comp rx=0 tx=0).stop 2024-03-21T10:50:11.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.027+0000 7f72ca400640 1 -- 172.21.15.31:0/1615792274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72c40693d0 msgr2=0x7f72c419e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:11.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.027+0000 7f72ca400640 1 --2- 172.21.15.31:0/1615792274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72c40693d0 0x7f72c419e210 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f72b8036040 tx=0x7f72b80042d0 comp rx=0 tx=0).stop 2024-03-21T10:50:11.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.027+0000 7f72ca400640 1 -- 172.21.15.31:0/1615792274 shutdown_connections 2024-03-21T10:50:11.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.027+0000 7f72ca400640 1 --2- 172.21.15.31:0/1615792274 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f72ac0a30f0 0x7f72ac0a55b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:11.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.027+0000 7f72ca400640 1 --2- 172.21.15.31:0/1615792274 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72c4106450 0x7f72c41a2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:11.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.027+0000 7f72ca400640 1 --2- 172.21.15.31:0/1615792274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72c4103ee0 0x7f72c419e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:11.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.027+0000 7f72ca400640 1 --2- 172.21.15.31:0/1615792274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72c40693d0 0x7f72c419e210 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:11.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.027+0000 7f72ca400640 1 -- 172.21.15.31:0/1615792274 >> 172.21.15.31:0/1615792274 conn(0x7f72c4075f00 msgr2=0x7f72c4102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:11.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.028+0000 7f72ca400640 1 -- 172.21.15.31:0/1615792274 shutdown_connections 2024-03-21T10:50:11.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-03-21T10:50:11.028+0000 7f72ca400640 1 -- 172.21.15.31:0/1615792274 wait complete. 2024-03-21T10:50:11.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:11 smithi040 ceph-mon[36154]: pgmap v362: 129 pgs: 14 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 75 active+clean; 21 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-21T10:50:11.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:11 smithi031 ceph-mon[31673]: pgmap v362: 129 pgs: 14 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 75 active+clean; 21 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-21T10:50:11.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:11 smithi070 ceph-mon[36330]: pgmap v362: 129 pgs: 14 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 75 active+clean; 21 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-21T10:50:12.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:12 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:50:12.553 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:12 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:50:12.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:12 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:50:12.602 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo 'stopping iogen' 2024-03-21T10:50:12.602 INFO:tasks.workunit.client.0.smithi031.stdout:stopping iogen 2024-03-21T10:50:12.603 INFO:tasks.workunit.client.0.smithi031.stderr:+ ./iogen -k 2024-03-21T10:50:12.607 INFO:tasks.workunit.client.0.smithi031.stdout:OK 2024-03-21T10:50:12.608 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo OK 2024-03-21T10:50:13.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:13 smithi040 ceph-mon[36154]: pgmap v363: 129 pgs: 14 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 75 active+clean; 21 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-21T10:50:13.473 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:13 smithi070 ceph-mon[36330]: pgmap v363: 129 pgs: 14 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 75 active+clean; 21 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-21T10:50:13.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:13 smithi031 ceph-mon[31673]: pgmap v363: 129 pgs: 14 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 75 active+clean; 21 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-21T10:50:14.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:14 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:50:14.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:14 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:50:14.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:14 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:50:14.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:14 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:50:14.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:14 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:50:14.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:14 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:50:14.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:14 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:50:14.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:14 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:50:14.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:14 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:50:14.645 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-21T10:50:14.645 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-21T10:50:15.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:15 smithi031 ceph-mon[31673]: pgmap v364: 129 pgs: 10 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 91 active+clean; 20 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 28 KiB/s wr, 9 op/s 2024-03-21T10:50:15.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:15 smithi070 ceph-mon[36330]: pgmap v364: 129 pgs: 10 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 91 active+clean; 20 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 28 KiB/s wr, 9 op/s 2024-03-21T10:50:15.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:15 smithi040 ceph-mon[36154]: pgmap v364: 129 pgs: 10 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 91 active+clean; 20 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 28 KiB/s wr, 9 op/s 2024-03-21T10:50:17.353 INFO:tasks.workunit:Stopping ['suites/iogen.sh'] on client.0... 2024-03-21T10:50:17.353 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-21T10:50:17.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:17 smithi031 ceph-mon[31673]: pgmap v365: 129 pgs: 129 active+clean; 18 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7.7 KiB/s wr, 7 op/s 2024-03-21T10:50:17.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:17 smithi031 ceph-mon[31673]: osdmap e106: 12 total, 12 up, 12 in 2024-03-21T10:50:17.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:17 smithi070 ceph-mon[36330]: pgmap v365: 129 pgs: 129 active+clean; 18 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7.7 KiB/s wr, 7 op/s 2024-03-21T10:50:17.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:17 smithi070 ceph-mon[36330]: osdmap e106: 12 total, 12 up, 12 in 2024-03-21T10:50:17.930 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:17 smithi040 ceph-mon[36154]: pgmap v365: 129 pgs: 129 active+clean; 18 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7.7 KiB/s wr, 7 op/s 2024-03-21T10:50:17.930 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:17 smithi040 ceph-mon[36154]: osdmap e106: 12 total, 12 up, 12 in 2024-03-21T10:50:19.105 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-21T10:50:19.105 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-21T10:50:19.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.440+0000 7f3297e00640 1 -- 172.21.15.31:0/2635142250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32900a4fc0 msgr2=0x7f32900a53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:19.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.440+0000 7f3297e00640 1 --2- 172.21.15.31:0/2635142250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32900a4fc0 0x7f32900a53a0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f32980674e0 tx=0x7f329808f290 comp rx=0 tx=0).stop 2024-03-21T10:50:19.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.441+0000 7f3297e00640 1 -- 172.21.15.31:0/2635142250 shutdown_connections 2024-03-21T10:50:19.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.441+0000 7f3297e00640 1 --2- 172.21.15.31:0/2635142250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32900ac100 0x7f32900b3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.441+0000 7f3297e00640 1 --2- 172.21.15.31:0/2635142250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32900a5970 0x7f32900ab9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.441+0000 7f3297e00640 1 --2- 172.21.15.31:0/2635142250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32900a4fc0 0x7f32900a53a0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.441+0000 7f3297e00640 1 -- 172.21.15.31:0/2635142250 >> 172.21.15.31:0/2635142250 conn(0x7f329001a640 msgr2=0x7f329001aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:19.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.442+0000 7f3297e00640 1 -- 172.21.15.31:0/2635142250 shutdown_connections 2024-03-21T10:50:19.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.442+0000 7f3297e00640 1 -- 172.21.15.31:0/2635142250 wait complete. 2024-03-21T10:50:19.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.444+0000 7f3297e00640 1 Processor -- start 2024-03-21T10:50:19.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.444+0000 7f3297e00640 1 -- start start 2024-03-21T10:50:19.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.446+0000 7f3297e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32900a4fc0 0x7f3290144110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:19.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.446+0000 7f3297e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32900ac100 0x7f3290144650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:19.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.447+0000 7f3297e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32900b2d10 0x7f32900b3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:19.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.447+0000 7f3297e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32900b7060 con 0x7f32900b2d10 2024-03-21T10:50:19.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.447+0000 7f3297e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32900b6ee0 con 0x7f32900ac100 2024-03-21T10:50:19.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.447+0000 7f3297e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32900b71e0 con 0x7f32900a4fc0 2024-03-21T10:50:19.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.447+0000 7f3296a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32900a4fc0 0x7f3290144110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:19.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.447+0000 7f3296a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32900a4fc0 0x7f3290144110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34794/0 (socket says 172.21.15.31:34794) 2024-03-21T10:50:19.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.447+0000 7f3296a00640 1 -- 172.21.15.31:0/3284224601 learned_addr learned my addr 172.21.15.31:0/3284224601 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:19.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.447+0000 7f3296000640 1 --2- 172.21.15.31:0/3284224601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32900ac100 0x7f3290144650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:19.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.447+0000 7f3296a00640 1 -- 172.21.15.31:0/3284224601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32900ac100 msgr2=0x7f3290144650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:19.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.447+0000 7f3296a00640 1 --2- 172.21.15.31:0/3284224601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32900ac100 0x7f3290144650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.447+0000 7f3296a00640 1 -- 172.21.15.31:0/3284224601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32900b2d10 msgr2=0x7f32900b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:19.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.448+0000 7f3297400640 1 --2- 172.21.15.31:0/3284224601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32900b2d10 0x7f32900b3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:19.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.448+0000 7f3296a00640 1 --2- 172.21.15.31:0/3284224601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32900b2d10 0x7f32900b3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.448+0000 7f3296a00640 1 -- 172.21.15.31:0/3284224601 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32900b3a00 con 0x7f32900a4fc0 2024-03-21T10:50:19.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.448+0000 7f3296000640 1 --2- 172.21.15.31:0/3284224601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32900ac100 0x7f3290144650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:50:19.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.448+0000 7f3296a00640 1 --2- 172.21.15.31:0/3284224601 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32900a4fc0 0x7f3290144110 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f3298050670 tx=0x7f32980986a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:19.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.452+0000 7f329ca00640 1 -- 172.21.15.31:0/3284224601 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32980a6070 con 0x7f32900a4fc0 2024-03-21T10:50:19.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.452+0000 7f329ca00640 1 -- 172.21.15.31:0/3284224601 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32980a1410 con 0x7f32900a4fc0 2024-03-21T10:50:19.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.452+0000 7f329ca00640 1 -- 172.21.15.31:0/3284224601 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3298097630 con 0x7f32900a4fc0 2024-03-21T10:50:19.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.452+0000 7f3297e00640 1 -- 172.21.15.31:0/3284224601 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f329013fc40 con 0x7f32900a4fc0 2024-03-21T10:50:19.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.452+0000 7f3297e00640 1 -- 172.21.15.31:0/3284224601 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3290140110 con 0x7f32900a4fc0 2024-03-21T10:50:19.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.458+0000 7f329ca00640 1 -- 172.21.15.31:0/3284224601 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f32980977d0 con 0x7f32900a4fc0 2024-03-21T10:50:19.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.459+0000 7f32a0a84740 1 Processor -- start 2024-03-21T10:50:19.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.460+0000 7f32a0a84740 1 -- start start 2024-03-21T10:50:19.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.461+0000 7f329ca00640 1 --2- 172.21.15.31:0/3284224601 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f327c0a3260 0x7f327c0a5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:19.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.461+0000 7f329ca00640 1 -- 172.21.15.31:0/3284224601 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f3298155b60 con 0x7f32900a4fc0 2024-03-21T10:50:19.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.462+0000 7f32a0a84740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5498bdd0 0x556d5498ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:19.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.462+0000 7f32a0a84740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5498b0f0 0x556d5498b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:19.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.462+0000 7f32a0a84740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d54994460 0x556d54996850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:19.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.462+0000 7f32a0a84740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d548ef250 con 0x556d5498bdd0 2024-03-21T10:50:19.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.462+0000 7f32a0a84740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d54897a30 con 0x556d54994460 2024-03-21T10:50:19.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.462+0000 7f32a0a84740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d548eb2f0 con 0x556d5498b0f0 2024-03-21T10:50:19.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.462+0000 7f3296000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5498b0f0 0x556d5498b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:19.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.463+0000 7f3296000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5498b0f0 0x556d5498b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34804/0 (socket says 172.21.15.31:34804) 2024-03-21T10:50:19.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.463+0000 7f3296000640 1 -- 172.21.15.31:0/491673257 learned_addr learned my addr 172.21.15.31:0/491673257 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:19.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.463+0000 7f3297400640 1 --2- 172.21.15.31:0/491673257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d54994460 0x556d54996850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:19.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.464+0000 7f3296a00640 1 --2- 172.21.15.31:0/491673257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5498bdd0 0x556d5498ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:19.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.464+0000 7f3296000640 1 -- 172.21.15.31:0/491673257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d54994460 msgr2=0x556d54996850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:19.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.464+0000 7f3296000640 1 --2- 172.21.15.31:0/491673257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d54994460 0x556d54996850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.464+0000 7f3296000640 1 -- 172.21.15.31:0/491673257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5498bdd0 msgr2=0x556d5498ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:19.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.464+0000 7f3296000640 1 --2- 172.21.15.31:0/491673257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5498bdd0 0x556d5498ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.464+0000 7f3296000640 1 -- 172.21.15.31:0/491673257 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d54882690 con 0x556d5498b0f0 2024-03-21T10:50:19.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.464+0000 7f3297400640 1 --2- 172.21.15.31:0/3284224601 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f327c0a3260 0x7f327c0a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:19.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.464+0000 7f3296000640 1 --2- 172.21.15.31:0/491673257 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5498b0f0 0x556d5498b570 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f328800bae0 tx=0x7f328800d340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32ec76b8391b725e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:19.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.465+0000 7f3283e00640 1 -- 172.21.15.31:0/491673257 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3288017420 con 0x556d5498b0f0 2024-03-21T10:50:19.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.465+0000 7f3283e00640 1 -- 172.21.15.31:0/491673257 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3288017aa0 con 0x556d5498b0f0 2024-03-21T10:50:19.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.465+0000 7f3283e00640 1 -- 172.21.15.31:0/491673257 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328800fbd0 con 0x556d5498b0f0 2024-03-21T10:50:19.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.466+0000 7f32a0a84740 1 -- 172.21.15.31:0/491673257 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5498b0f0 msgr2=0x556d5498b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:19.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.466+0000 7f32a0a84740 1 --2- 172.21.15.31:0/491673257 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5498b0f0 0x556d5498b570 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f328800bae0 tx=0x7f328800d340 comp rx=0 tx=0).stop 2024-03-21T10:50:19.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.466+0000 7f32a0a84740 1 -- 172.21.15.31:0/491673257 shutdown_connections 2024-03-21T10:50:19.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.466+0000 7f32a0a84740 1 --2- 172.21.15.31:0/491673257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d54994460 0x556d54996850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.466+0000 7f32a0a84740 1 --2- 172.21.15.31:0/491673257 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5498b0f0 0x556d5498b570 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.466+0000 7f32a0a84740 1 --2- 172.21.15.31:0/491673257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5498bdd0 0x556d5498ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.466+0000 7f32a0a84740 1 -- 172.21.15.31:0/491673257 >> 172.21.15.31:0/491673257 conn(0x556d54952200 msgr2=0x556d54985b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:19.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.467+0000 7f3297400640 1 --2- 172.21.15.31:0/3284224601 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f327c0a3260 0x7f327c0a5720 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3284005dc0 tx=0x7f3284004730 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:19.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.468+0000 7f32a0a84740 1 -- 172.21.15.31:0/491673257 shutdown_connections 2024-03-21T10:50:19.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.468+0000 7f32a0a84740 1 -- 172.21.15.31:0/491673257 wait complete. 2024-03-21T10:50:19.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.469+0000 7f32a0a84740 1 -- start start 2024-03-21T10:50:19.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.470+0000 7f32a0a84740 1 Processor -- start 2024-03-21T10:50:19.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.470+0000 7f32a0a84740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5498b0f0 0x556d54993b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:19.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.471+0000 7f32a0a84740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5498bdd0 0x556d54990f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:19.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.471+0000 7f32a0a84740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d54994460 0x556d54991450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:19.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.471+0000 7f3296a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5498b0f0 0x556d54993b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:19.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.471+0000 7f3296a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5498b0f0 0x556d54993b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37090/0 (socket says 172.21.15.31:37090) 2024-03-21T10:50:19.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.471+0000 7f3296000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5498bdd0 0x556d54990f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:19.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.471+0000 7f3296a00640 1 -- 172.21.15.31:0/936488725 learned_addr learned my addr 172.21.15.31:0/936488725 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:19.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.471+0000 7f32a0a84740 1 -- 172.21.15.31:0/936488725 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d549972d0 con 0x556d5498b0f0 2024-03-21T10:50:19.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.471+0000 7f32a0a84740 1 -- 172.21.15.31:0/936488725 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d548aae80 con 0x556d54994460 2024-03-21T10:50:19.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.471+0000 7f32a0a84740 1 -- 172.21.15.31:0/936488725 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d54997450 con 0x556d5498bdd0 2024-03-21T10:50:19.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.471+0000 7f32a0a84740 10 client.0 _ll_register_callbacks cb 0x556d548bd130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:50:19.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.472+0000 7f3296000640 1 -- 172.21.15.31:0/936488725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d54994460 msgr2=0x556d54991450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:19.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.472+0000 7f3296000640 1 --2- 172.21.15.31:0/936488725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d54994460 0x556d54991450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.472+0000 7f3296000640 1 -- 172.21.15.31:0/936488725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5498b0f0 msgr2=0x556d54993b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:19.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.472+0000 7f3296000640 1 --2- 172.21.15.31:0/936488725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5498b0f0 0x556d54993b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.472+0000 7f3296000640 1 -- 172.21.15.31:0/936488725 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556d546cd7d0 con 0x556d5498bdd0 2024-03-21T10:50:19.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.472+0000 7f3280c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:50:19.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.472+0000 7f3296000640 1 --2- 172.21.15.31:0/936488725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5498bdd0 0x556d54990f10 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f3288000c00 tx=0x7f328800dfc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e4bd0544d873fb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:19.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.473+0000 7f3280c00640 1 -- 172.21.15.31:0/936488725 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3288027020 con 0x556d5498bdd0 2024-03-21T10:50:19.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.473+0000 7f32a0a84740 1 -- 172.21.15.31:0/936488725 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3288017aa0 con 0x556d5498bdd0 2024-03-21T10:50:19.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.473+0000 7f3280c00640 1 -- 172.21.15.31:0/936488725 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f3288017630 con 0x556d5498bdd0 2024-03-21T10:50:19.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.473+0000 7f3280c00640 1 -- 172.21.15.31:0/936488725 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3288027020 con 0x556d5498bdd0 2024-03-21T10:50:19.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.473+0000 7f32a0a84740 10 client.28498 fetch_fsmap learned FSMap version 31 2024-03-21T10:50:19.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.474+0000 7f32a0a84740 1 -- 172.21.15.31:0/936488725 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x556d546033f0 con 0x556d5498bdd0 2024-03-21T10:50:19.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.474+0000 7f3280c00640 1 -- 172.21.15.31:0/936488725 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3288017aa0 con 0x556d5498bdd0 2024-03-21T10:50:19.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.474+0000 7f3280c00640 1 -- 172.21.15.31:0/936488725 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f32880236d0 con 0x556d5498bdd0 2024-03-21T10:50:19.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.474+0000 7f32a0a84740 10 client.28498 fetch_fsmap finished waiting for FSMap version 31 2024-03-21T10:50:19.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.474+0000 7f32a0a84740 10 client.28498 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:50:19.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.474+0000 7f32a0a84740 20 client.28498 populate_metadata read hostname 'smithi031' 2024-03-21T10:50:19.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.474+0000 7f32a0a84740 1 --2- 172.21.15.31:0/936488725 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556d549a42d0 0x556d549a66c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:19.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.474+0000 7f32a0a84740 4 client.28498 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:50:19.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.474+0000 7f32a0a84740 1 -- 172.21.15.31:0/936488725 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556d548fc790 con 0x556d549a42d0 2024-03-21T10:50:19.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.475+0000 7f3296a00640 1 --2- 172.21.15.31:0/936488725 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556d549a42d0 0x556d549a66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:19.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.478+0000 7f3280c00640 10 client.28498 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:50:19.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.480+0000 7f3296a00640 1 --2- 172.21.15.31:0/936488725 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556d549a42d0 0x556d549a66c0 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e8a5e5d624ae992 server_cookie=b5916b7e660d9580 in_seq=0 out_seq=0 2024-03-21T10:50:19.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.481+0000 7f3280c00640 1 -- 172.21.15.31:0/936488725 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556d548fc790 con 0x556d549a42d0 2024-03-21T10:50:19.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.481+0000 7f3280c00640 10 client.28498 handle_command_reply: tid=0 2024-03-21T10:50:19.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.481+0000 7f32a0a84740 1 client.28498 shutdown 2024-03-21T10:50:19.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.482+0000 7f32a0a84740 1 -- 172.21.15.31:0/936488725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5498bdd0 msgr2=0x556d54990f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:19.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.482+0000 7f32a0a84740 1 --2- 172.21.15.31:0/936488725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5498bdd0 0x556d54990f10 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f3288000c00 tx=0x7f328800dfc0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.482+0000 7f32a0a84740 1 -- 172.21.15.31:0/936488725 shutdown_connections 2024-03-21T10:50:19.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.482+0000 7f32a0a84740 1 --2- 172.21.15.31:0/936488725 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556d549a42d0 0x556d549a66c0 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.482+0000 7f32a0a84740 1 --2- 172.21.15.31:0/936488725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d54994460 0x556d54991450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.482+0000 7f32a0a84740 1 --2- 172.21.15.31:0/936488725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5498bdd0 0x556d54990f10 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.482+0000 7f32a0a84740 1 --2- 172.21.15.31:0/936488725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5498b0f0 0x556d54993b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.482+0000 7f32a0a84740 1 -- 172.21.15.31:0/936488725 >> 172.21.15.31:0/936488725 conn(0x556d54952200 msgr2=0x556d54987890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:19.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.482+0000 7f3280c00640 0 client.28498 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:50:19.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.482+0000 7f32a0a84740 1 -- 172.21.15.31:0/936488725 shutdown_connections 2024-03-21T10:50:19.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.482+0000 7f32a0a84740 1 -- 172.21.15.31:0/936488725 wait complete. 2024-03-21T10:50:19.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.482+0000 7f32a0a84740 20 client.28498 trim_cache size 0 max 16384 2024-03-21T10:50:19.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.523+0000 7f32a0a84740 1 Processor -- start 2024-03-21T10:50:19.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.524+0000 7f32a0a84740 1 -- start start 2024-03-21T10:50:19.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.524+0000 7f32a0a84740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5498b0f0 0x556d54993640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:19.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.524+0000 7f32a0a84740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5498bdd0 0x556d54993b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:19.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.525+0000 7f3296a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5498b0f0 0x556d54993640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:19.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.525+0000 7f32a0a84740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d54994460 0x556d54995850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:19.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.525+0000 7f32a0a84740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32780123c0 con 0x556d5498bdd0 2024-03-21T10:50:19.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.525+0000 7f32a0a84740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327800fe90 con 0x556d5498b0f0 2024-03-21T10:50:19.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.525+0000 7f32a0a84740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327800def0 con 0x556d54994460 2024-03-21T10:50:19.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.525+0000 7f3296a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5498b0f0 0x556d54993640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:52680/0 (socket says 172.21.15.31:52680) 2024-03-21T10:50:19.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.525+0000 7f3296a00640 1 -- 172.21.15.31:0/16351495 learned_addr learned my addr 172.21.15.31:0/16351495 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:19.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.525+0000 7f3297400640 1 --2- 172.21.15.31:0/16351495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d54994460 0x556d54995850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:19.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.525+0000 7f3296a00640 1 -- 172.21.15.31:0/16351495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d54994460 msgr2=0x556d54995850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:19.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.525+0000 7f3296a00640 1 --2- 172.21.15.31:0/16351495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d54994460 0x556d54995850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.525+0000 7f3296a00640 1 -- 172.21.15.31:0/16351495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5498bdd0 msgr2=0x556d54993b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:19.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.525+0000 7f3296000640 1 --2- 172.21.15.31:0/16351495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5498bdd0 0x556d54993b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:19.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.526+0000 7f3296a00640 1 --2- 172.21.15.31:0/16351495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5498bdd0 0x556d54993b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.526+0000 7f3296a00640 1 -- 172.21.15.31:0/16351495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d54991f00 con 0x556d5498b0f0 2024-03-21T10:50:19.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.526+0000 7f3296000640 1 --2- 172.21.15.31:0/16351495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5498bdd0 0x556d54993b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:50:19.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.526+0000 7f3296a00640 1 --2- 172.21.15.31:0/16351495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5498b0f0 0x556d54993640 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f32980676f0 tx=0x7f329809a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac626811afb5c792 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:19.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.527+0000 7f3283400640 1 -- 172.21.15.31:0/16351495 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329812a670 con 0x556d5498b0f0 2024-03-21T10:50:19.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.527+0000 7f3283400640 1 -- 172.21.15.31:0/16351495 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3298093030 con 0x556d5498b0f0 2024-03-21T10:50:19.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.527+0000 7f3283400640 1 -- 172.21.15.31:0/16351495 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32980a18a0 con 0x556d5498b0f0 2024-03-21T10:50:19.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.529+0000 7f32a0a84740 1 -- 172.21.15.31:0/16351495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5498b0f0 msgr2=0x556d54993640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:19.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.529+0000 7f32a0a84740 1 --2- 172.21.15.31:0/16351495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5498b0f0 0x556d54993640 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f32980676f0 tx=0x7f329809a040 comp rx=0 tx=0).stop 2024-03-21T10:50:19.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.529+0000 7f32a0a84740 1 -- 172.21.15.31:0/16351495 shutdown_connections 2024-03-21T10:50:19.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.529+0000 7f32a0a84740 1 --2- 172.21.15.31:0/16351495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d54994460 0x556d54995850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.529+0000 7f32a0a84740 1 --2- 172.21.15.31:0/16351495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5498bdd0 0x556d54993b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.529+0000 7f32a0a84740 1 --2- 172.21.15.31:0/16351495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d5498b0f0 0x556d54993640 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.529+0000 7f32a0a84740 1 -- 172.21.15.31:0/16351495 >> 172.21.15.31:0/16351495 conn(0x556d54952200 msgr2=0x556d54985670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:19.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.529+0000 7f32a0a84740 1 -- 172.21.15.31:0/16351495 shutdown_connections 2024-03-21T10:50:19.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.530+0000 7f32a0a84740 1 -- 172.21.15.31:0/16351495 wait complete. 2024-03-21T10:50:19.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.530+0000 7f32a0a84740 1 -- start start 2024-03-21T10:50:19.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.530+0000 7f32a0a84740 1 Processor -- start 2024-03-21T10:50:19.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.532+0000 7f32a0a84740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5498b0f0 0x556d5498e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:19.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.532+0000 7f32a0a84740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5498bdd0 0x556d5498e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:19.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.532+0000 7f32a0a84740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d54994460 0x556d5499eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:19.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.533+0000 7f3296000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5498bdd0 0x556d5498e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:19.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.533+0000 7f3296000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5498bdd0 0x556d5498e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34828/0 (socket says 172.21.15.31:34828) 2024-03-21T10:50:19.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.533+0000 7f3297400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d54994460 0x556d5499eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:19.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.533+0000 7f3297400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d54994460 0x556d5499eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:52688/0 (socket says 172.21.15.31:52688) 2024-03-21T10:50:19.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.533+0000 7f3296a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5498b0f0 0x556d5498e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:19.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.533+0000 7f3296a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5498b0f0 0x556d5498e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37106/0 (socket says 172.21.15.31:37106) 2024-03-21T10:50:19.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.534+0000 7f32a0a84740 1 -- 172.21.15.31:0/293421073 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327800e510 con 0x556d5498b0f0 2024-03-21T10:50:19.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.534+0000 7f32a0a84740 1 -- 172.21.15.31:0/293421073 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327800f830 con 0x556d54994460 2024-03-21T10:50:19.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.534+0000 7f32a0a84740 1 -- 172.21.15.31:0/293421073 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327800eb70 con 0x556d5498bdd0 2024-03-21T10:50:19.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.534+0000 7f32a0a84740 10 client.0 _ll_register_callbacks cb 0x7f328800b930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:50:19.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.534+0000 7f3297400640 1 -- 172.21.15.31:0/293421073 learned_addr learned my addr 172.21.15.31:0/293421073 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:19.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.535+0000 7f3297400640 1 -- 172.21.15.31:0/293421073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5498bdd0 msgr2=0x556d5498e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:19.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.535+0000 7f3297400640 1 --2- 172.21.15.31:0/293421073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5498bdd0 0x556d5498e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.535+0000 7f3297400640 1 -- 172.21.15.31:0/293421073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5498b0f0 msgr2=0x556d5498e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:19.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.535+0000 7f3297400640 1 --2- 172.21.15.31:0/293421073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5498b0f0 0x556d5498e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.535+0000 7f3297400640 1 -- 172.21.15.31:0/293421073 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556d545c30f0 con 0x556d54994460 2024-03-21T10:50:19.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.535+0000 7f3296000640 1 --2- 172.21.15.31:0/293421073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5498bdd0 0x556d5498e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:50:19.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.536+0000 7f3296a00640 1 --2- 172.21.15.31:0/293421073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5498b0f0 0x556d5498e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:50:19.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.536+0000 7f3297400640 1 --2- 172.21.15.31:0/293421073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d54994460 0x556d5499eff0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f328401cb50 tx=0x7f328400e610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9eeb97f3fede3716 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:19.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.536+0000 7f3280c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:50:19.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.536+0000 7f3280c00640 1 -- 172.21.15.31:0/293421073 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3284010a70 con 0x556d54994460 2024-03-21T10:50:19.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.536+0000 7f32a0a84740 1 -- 172.21.15.31:0/293421073 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3298093030 con 0x556d54994460 2024-03-21T10:50:19.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.537+0000 7f3280c00640 1 -- 172.21.15.31:0/293421073 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f3284010c10 con 0x556d54994460 2024-03-21T10:50:19.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.540+0000 7f3280c00640 1 -- 172.21.15.31:0/293421073 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3284032050 con 0x556d54994460 2024-03-21T10:50:19.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.540+0000 7f32a0a84740 10 client.28574 fetch_fsmap learned FSMap version 31 2024-03-21T10:50:19.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.540+0000 7f32a0a84740 1 -- 172.21.15.31:0/293421073 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x556d549ffe50 con 0x556d54994460 2024-03-21T10:50:19.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.540+0000 7f3280c00640 1 -- 172.21.15.31:0/293421073 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3298093030 con 0x556d54994460 2024-03-21T10:50:19.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.540+0000 7f3280c00640 1 -- 172.21.15.31:0/293421073 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 31) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f328402d030 con 0x556d54994460 2024-03-21T10:50:19.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.540+0000 7f32a0a84740 10 client.28574 fetch_fsmap finished waiting for FSMap version 31 2024-03-21T10:50:19.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.540+0000 7f32a0a84740 10 client.28574 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:50:19.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.540+0000 7f32a0a84740 20 client.28574 populate_metadata read hostname 'smithi031' 2024-03-21T10:50:19.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.540+0000 7f32a0a84740 1 --2- 172.21.15.31:0/293421073 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556d54a005b0 0x556d54a029d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:19.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.540+0000 7f32a0a84740 4 client.28574 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-21T10:50:19.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.540+0000 7f32a0a84740 1 -- 172.21.15.31:0/293421073 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556d548f6ae0 con 0x556d54a005b0 2024-03-21T10:50:19.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.541+0000 7f3296a00640 1 --2- 172.21.15.31:0/293421073 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556d54a005b0 0x556d54a029d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:19.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.542+0000 7f3296a00640 1 --2- 172.21.15.31:0/293421073 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556d54a005b0 0x556d54a029d0 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9c361cd8d89293a server_cookie=66928dc2a4eabd2 in_seq=0 out_seq=0 2024-03-21T10:50:19.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.542+0000 7f3280c00640 10 client.28574 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:50:19.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.542+0000 7f3280c00640 1 -- 172.21.15.31:0/293421073 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x556d548f6ae0 con 0x556d54a005b0 2024-03-21T10:50:19.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.542+0000 7f3280c00640 10 client.28574 handle_command_reply: tid=0 2024-03-21T10:50:19.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.543+0000 7f32a0a84740 1 client.28574 shutdown 2024-03-21T10:50:19.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.543+0000 7f32a0a84740 1 -- 172.21.15.31:0/293421073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d54994460 msgr2=0x556d5499eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:19.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.543+0000 7f32a0a84740 1 --2- 172.21.15.31:0/293421073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d54994460 0x556d5499eff0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f328401cb50 tx=0x7f328400e610 comp rx=0 tx=0).stop 2024-03-21T10:50:19.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.543+0000 7f32a0a84740 1 -- 172.21.15.31:0/293421073 shutdown_connections 2024-03-21T10:50:19.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.543+0000 7f32a0a84740 1 --2- 172.21.15.31:0/293421073 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x556d54a005b0 0x556d54a029d0 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.543+0000 7f3280c00640 0 client.28574 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:50:19.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.543+0000 7f32a0a84740 1 --2- 172.21.15.31:0/293421073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d54994460 0x556d5499eff0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.543+0000 7f32a0a84740 1 --2- 172.21.15.31:0/293421073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556d5498bdd0 0x556d5498e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.543+0000 7f32a0a84740 1 --2- 172.21.15.31:0/293421073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d5498b0f0 0x556d5498e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.543+0000 7f32a0a84740 1 -- 172.21.15.31:0/293421073 >> 172.21.15.31:0/293421073 conn(0x556d54952200 msgr2=0x556d54986e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:19.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.546+0000 7f32a0a84740 1 -- 172.21.15.31:0/293421073 shutdown_connections 2024-03-21T10:50:19.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.546+0000 7f32a0a84740 1 -- 172.21.15.31:0/293421073 wait complete. 2024-03-21T10:50:19.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.546+0000 7f32a0a84740 20 client.28574 trim_cache size 0 max 16384 2024-03-21T10:50:19.544 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:50:19.544 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-03-21T10:50:19.544 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-03-21T10:50:19.544 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:50:19.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.548+0000 7f3297e00640 1 -- 172.21.15.31:0/3284224601 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f327c0a3260 msgr2=0x7f327c0a5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:19.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.548+0000 7f3297e00640 1 --2- 172.21.15.31:0/3284224601 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f327c0a3260 0x7f327c0a5720 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3284005dc0 tx=0x7f3284004730 comp rx=0 tx=0).stop 2024-03-21T10:50:19.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.548+0000 7f3297e00640 1 -- 172.21.15.31:0/3284224601 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32900a4fc0 msgr2=0x7f3290144110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:19.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.548+0000 7f3297e00640 1 --2- 172.21.15.31:0/3284224601 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32900a4fc0 0x7f3290144110 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f3298050670 tx=0x7f32980986a0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.548+0000 7f3297e00640 1 -- 172.21.15.31:0/3284224601 shutdown_connections 2024-03-21T10:50:19.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.548+0000 7f3297e00640 1 --2- 172.21.15.31:0/3284224601 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f327c0a3260 0x7f327c0a5720 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.548+0000 7f3297e00640 1 --2- 172.21.15.31:0/3284224601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32900b2d10 0x7f32900b3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.548+0000 7f3297e00640 1 --2- 172.21.15.31:0/3284224601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32900ac100 0x7f3290144650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.548+0000 7f3297e00640 1 --2- 172.21.15.31:0/3284224601 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32900a4fc0 0x7f3290144110 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.548+0000 7f3297e00640 1 -- 172.21.15.31:0/3284224601 >> 172.21.15.31:0/3284224601 conn(0x7f329001a640 msgr2=0x7f32900ad230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:19.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.549+0000 7f3297e00640 1 -- 172.21.15.31:0/3284224601 shutdown_connections 2024-03-21T10:50:19.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.549+0000 7f3297e00640 1 -- 172.21.15.31:0/3284224601 wait complete. 2024-03-21T10:50:19.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:19 smithi031 ceph-mon[31673]: pgmap v367: 129 pgs: 129 active+clean; 18 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7.7 KiB/s wr, 7 op/s 2024-03-21T10:50:19.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:19 smithi031 ceph-mon[31673]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:50:19.581 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-21T10:50:19.581 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:50:19.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:19 smithi070 ceph-mon[36330]: pgmap v367: 129 pgs: 129 active+clean; 18 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7.7 KiB/s wr, 7 op/s 2024-03-21T10:50:19.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:19 smithi070 ceph-mon[36330]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:50:19.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.903+0000 7f3dce000640 1 -- 172.21.15.31:0/857797683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dc8107eb0 msgr2=0x7f3dc8108290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:19.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.903+0000 7f3dce000640 1 --2- 172.21.15.31:0/857797683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dc8107eb0 0x7f3dc8108290 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0009ca0 tx=0x7f3dc002c100 comp rx=0 tx=0).stop 2024-03-21T10:50:19.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.904+0000 7f3dce000640 1 -- 172.21.15.31:0/857797683 shutdown_connections 2024-03-21T10:50:19.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.904+0000 7f3dce000640 1 --2- 172.21.15.31:0/857797683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dc810cf80 0x7f3dc8114b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.904+0000 7f3dce000640 1 --2- 172.21.15.31:0/857797683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3dc8108860 0x7f3dc810c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.904+0000 7f3dce000640 1 --2- 172.21.15.31:0/857797683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dc8107eb0 0x7f3dc8108290 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.904+0000 7f3dce000640 1 -- 172.21.15.31:0/857797683 >> 172.21.15.31:0/857797683 conn(0x7f3dc80773c0 msgr2=0x7f3dc80797e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:19.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.904+0000 7f3dce000640 1 -- 172.21.15.31:0/857797683 shutdown_connections 2024-03-21T10:50:19.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.904+0000 7f3dce000640 1 -- 172.21.15.31:0/857797683 wait complete. 2024-03-21T10:50:19.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.906+0000 7f3dce000640 1 Processor -- start 2024-03-21T10:50:19.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.906+0000 7f3dce000640 1 -- start start 2024-03-21T10:50:19.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.907+0000 7f3dce000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dc8107eb0 0x7f3dc819e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:19.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.908+0000 7f3dce000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dc810cf80 0x7f3dc819ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:19.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.908+0000 7f3dce000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3dc81a2d90 0x7f3dc81a3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:19.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.908+0000 7f3dce000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc8118030 con 0x7f3dc8107eb0 2024-03-21T10:50:19.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.908+0000 7f3dce000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc8117eb0 con 0x7f3dc81a2d90 2024-03-21T10:50:19.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.908+0000 7f3dce000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc81181b0 con 0x7f3dc810cf80 2024-03-21T10:50:19.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.909+0000 7f3dc7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dc8107eb0 0x7f3dc819e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:19.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.909+0000 7f3dc6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dc810cf80 0x7f3dc819ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:19.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.909+0000 7f3dc6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dc810cf80 0x7f3dc819ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34844/0 (socket says 172.21.15.31:34844) 2024-03-21T10:50:19.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.909+0000 7f3dc6a00640 1 -- 172.21.15.31:0/1757712777 learned_addr learned my addr 172.21.15.31:0/1757712777 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:19.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.909+0000 7f3dc7e00640 1 --2- 172.21.15.31:0/1757712777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3dc81a2d90 0x7f3dc81a3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:19.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.909+0000 7f3dc6a00640 1 -- 172.21.15.31:0/1757712777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3dc81a2d90 msgr2=0x7f3dc81a3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:19.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.909+0000 7f3dc6a00640 1 --2- 172.21.15.31:0/1757712777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3dc81a2d90 0x7f3dc81a3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.909+0000 7f3dc6a00640 1 -- 172.21.15.31:0/1757712777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dc8107eb0 msgr2=0x7f3dc819e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:19.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.909+0000 7f3dc6a00640 1 --2- 172.21.15.31:0/1757712777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dc8107eb0 0x7f3dc819e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:19.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.910+0000 7f3dc6a00640 1 -- 172.21.15.31:0/1757712777 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc81a3880 con 0x7f3dc810cf80 2024-03-21T10:50:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.910+0000 7f3dc7e00640 1 --2- 172.21.15.31:0/1757712777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3dc81a2d90 0x7f3dc81a3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:50:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.910+0000 7f3dc7400640 1 --2- 172.21.15.31:0/1757712777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dc8107eb0 0x7f3dc819e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:50:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.910+0000 7f3dc6a00640 1 --2- 172.21.15.31:0/1757712777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dc810cf80 0x7f3dc819ea10 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f3db400e9a0 tx=0x7f3db400ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.911+0000 7f3dbfe00640 1 -- 172.21.15.31:0/1757712777 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db400cd80 con 0x7f3dc810cf80 2024-03-21T10:50:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.911+0000 7f3dbfe00640 1 -- 172.21.15.31:0/1757712777 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3db40035f0 con 0x7f3dc810cf80 2024-03-21T10:50:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.911+0000 7f3dbfe00640 1 -- 172.21.15.31:0/1757712777 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db4010670 con 0x7f3dc810cf80 2024-03-21T10:50:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.911+0000 7f3dce000640 1 -- 172.21.15.31:0/1757712777 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc81a3b70 con 0x7f3dc810cf80 2024-03-21T10:50:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.911+0000 7f3dce000640 1 -- 172.21.15.31:0/1757712777 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc81ab310 con 0x7f3dc810cf80 2024-03-21T10:50:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.913+0000 7f3dce000640 1 -- 172.21.15.31:0/1757712777 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dc8068970 con 0x7f3dc810cf80 2024-03-21T10:50:19.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.917+0000 7f3dbfe00640 1 -- 172.21.15.31:0/1757712777 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f3db40037a0 con 0x7f3dc810cf80 2024-03-21T10:50:19.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.920+0000 7f3dbfe00640 1 --2- 172.21.15.31:0/1757712777 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3dac0a3110 0x7f3dac0a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:19.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.920+0000 7f3dbfe00640 1 -- 172.21.15.31:0/1757712777 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f3db40cfeb0 con 0x7f3dc810cf80 2024-03-21T10:50:19.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.921+0000 7f3dc7400640 1 --2- 172.21.15.31:0/1757712777 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3dac0a3110 0x7f3dac0a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:19.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.924+0000 7f3dc7400640 1 --2- 172.21.15.31:0/1757712777 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3dac0a3110 0x7f3dac0a55d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0006010 tx=0x7f3dc0005d80 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:19.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:19.929+0000 7f3dbfe00640 1 -- 172.21.15.31:0/1757712777 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3db409c600 con 0x7f3dc810cf80 2024-03-21T10:50:19.929 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:19 smithi040 ceph-mon[36154]: pgmap v367: 129 pgs: 129 active+clean; 18 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7.7 KiB/s wr, 7 op/s 2024-03-21T10:50:19.930 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:19 smithi040 ceph-mon[36154]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:50:20.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.390+0000 7f3dce000640 1 -- 172.21.15.31:0/1757712777 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3dc8108860 con 0x7f3dc810cf80 2024-03-21T10:50:20.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.391+0000 7f3dbfe00640 1 -- 172.21.15.31:0/1757712777 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+11385 (secure 0 0 0) 0x7f3db409bf00 con 0x7f3dc810cf80 2024-03-21T10:50:20.389 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:50:20.389 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:50:18.332570+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24596,24547]},"id":1}]} 2024-03-21T10:50:20.389 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 31 2024-03-21T10:50:20.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.395+0000 7f3dce000640 1 -- 172.21.15.31:0/1757712777 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3dac0a3110 msgr2=0x7f3dac0a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:20.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.395+0000 7f3dce000640 1 --2- 172.21.15.31:0/1757712777 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3dac0a3110 0x7f3dac0a55d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0006010 tx=0x7f3dc0005d80 comp rx=0 tx=0).stop 2024-03-21T10:50:20.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.395+0000 7f3dce000640 1 -- 172.21.15.31:0/1757712777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dc810cf80 msgr2=0x7f3dc819ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:20.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.395+0000 7f3dce000640 1 --2- 172.21.15.31:0/1757712777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dc810cf80 0x7f3dc819ea10 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f3db400e9a0 tx=0x7f3db400ee60 comp rx=0 tx=0).stop 2024-03-21T10:50:20.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.397+0000 7f3dce000640 1 -- 172.21.15.31:0/1757712777 shutdown_connections 2024-03-21T10:50:20.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.397+0000 7f3dce000640 1 --2- 172.21.15.31:0/1757712777 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3dac0a3110 0x7f3dac0a55d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.397+0000 7f3dce000640 1 --2- 172.21.15.31:0/1757712777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3dc81a2d90 0x7f3dc81a3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.397+0000 7f3dce000640 1 --2- 172.21.15.31:0/1757712777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3dc810cf80 0x7f3dc819ea10 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.397+0000 7f3dce000640 1 --2- 172.21.15.31:0/1757712777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dc8107eb0 0x7f3dc819e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.397+0000 7f3dce000640 1 -- 172.21.15.31:0/1757712777 >> 172.21.15.31:0/1757712777 conn(0x7f3dc80773c0 msgr2=0x7f3dc80796f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:20.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.397+0000 7f3dce000640 1 -- 172.21.15.31:0/1757712777 shutdown_connections 2024-03-21T10:50:20.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.397+0000 7f3dce000640 1 -- 172.21.15.31:0/1757712777 wait complete. 2024-03-21T10:50:20.425 DEBUG:teuthology.parallel:result is None 2024-03-21T10:50:20.425 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-21T10:50:20.427 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:50:20.428 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-21T10:50:20.571 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-21T10:50:20.572 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-21T10:50:20.583 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi031.front.sepia.ceph.com 2024-03-21T10:50:20.605 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-21T10:50:20.618 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-21T10:50:20.618 INFO:teuthology.task.exec:Executing custom commands... 2024-03-21T10:50:20.618 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2024-03-21T10:50:20.618 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-21T10:50:20.758 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:20 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1757712777' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:50:20.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.800+0000 7fb3f7e00640 1 -- 172.21.15.31:0/339254163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3f00055d0 msgr2=0x7fb3f00a24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:20.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.800+0000 7fb3f7e00640 1 --2- 172.21.15.31:0/339254163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3f00055d0 0x7fb3f00a24c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009930 tx=0x7fb3e402bc20 comp rx=0 tx=0).stop 2024-03-21T10:50:20.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.800+0000 7fb3f7e00640 1 -- 172.21.15.31:0/339254163 shutdown_connections 2024-03-21T10:50:20.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.800+0000 7fb3f7e00640 1 --2- 172.21.15.31:0/339254163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3f00a2a00 0x7fb3f00a4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.800+0000 7fb3f7e00640 1 --2- 172.21.15.31:0/339254163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3f00055d0 0x7fb3f00a24c0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.800+0000 7fb3f7e00640 1 --2- 172.21.15.31:0/339254163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3f0004cb0 0x7fb3f0005090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.800+0000 7fb3f7e00640 1 -- 172.21.15.31:0/339254163 >> 172.21.15.31:0/339254163 conn(0x7fb3f000a740 msgr2=0x7fb3f000ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:20.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.801+0000 7fb3f7e00640 1 -- 172.21.15.31:0/339254163 shutdown_connections 2024-03-21T10:50:20.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.801+0000 7fb3f7e00640 1 -- 172.21.15.31:0/339254163 wait complete. 2024-03-21T10:50:20.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.803+0000 7fb3f7e00640 1 Processor -- start 2024-03-21T10:50:20.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.803+0000 7fb3f7e00640 1 -- start start 2024-03-21T10:50:20.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.804+0000 7fb3f7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3f0004cb0 0x7fb3f013f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:20.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.804+0000 7fb3f7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3f00055d0 0x7fb3f013fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:20.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.804+0000 7fb3f7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3f00a2a00 0x7fb3f0144010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:20.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.805+0000 7fb3f6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3f0004cb0 0x7fb3f013f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:20.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.805+0000 7fb3f6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3f0004cb0 0x7fb3f013f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34866/0 (socket says 172.21.15.31:34866) 2024-03-21T10:50:20.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.805+0000 7fb3f6a00640 1 -- 172.21.15.31:0/3448835957 learned_addr learned my addr 172.21.15.31:0/3448835957 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:20.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.805+0000 7fb3f7e00640 1 -- 172.21.15.31:0/3448835957 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f00a82e0 con 0x7fb3f00a2a00 2024-03-21T10:50:20.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.805+0000 7fb3f7e00640 1 -- 172.21.15.31:0/3448835957 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f00a8160 con 0x7fb3f00055d0 2024-03-21T10:50:20.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.805+0000 7fb3f7e00640 1 -- 172.21.15.31:0/3448835957 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f00a8460 con 0x7fb3f0004cb0 2024-03-21T10:50:20.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.805+0000 7fb3f7400640 1 --2- 172.21.15.31:0/3448835957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3f00a2a00 0x7fb3f0144010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:20.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.805+0000 7fb3f6000640 1 --2- 172.21.15.31:0/3448835957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3f00055d0 0x7fb3f013fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:20.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.806+0000 7fb3f6a00640 1 -- 172.21.15.31:0/3448835957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3f00055d0 msgr2=0x7fb3f013fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:20.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.806+0000 7fb3f6a00640 1 --2- 172.21.15.31:0/3448835957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3f00055d0 0x7fb3f013fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.806+0000 7fb3f6a00640 1 -- 172.21.15.31:0/3448835957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3f00a2a00 msgr2=0x7fb3f0144010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:20.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.806+0000 7fb3f6a00640 1 --2- 172.21.15.31:0/3448835957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3f00a2a00 0x7fb3f0144010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.806+0000 7fb3f6a00640 1 -- 172.21.15.31:0/3448835957 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f0144740 con 0x7fb3f0004cb0 2024-03-21T10:50:20.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.806+0000 7fb3f6a00640 1 --2- 172.21.15.31:0/3448835957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3f0004cb0 0x7fb3f013f750 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fb3e800d950 tx=0x7fb3e800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:20.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.807+0000 7fb3ef400640 1 -- 172.21.15.31:0/3448835957 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e8017070 con 0x7fb3f0004cb0 2024-03-21T10:50:20.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.807+0000 7fb3ef400640 1 -- 172.21.15.31:0/3448835957 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3e8009c30 con 0x7fb3f0004cb0 2024-03-21T10:50:20.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.807+0000 7fb3ef400640 1 -- 172.21.15.31:0/3448835957 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e800f6b0 con 0x7fb3f0004cb0 2024-03-21T10:50:20.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.807+0000 7fb3f7e00640 1 -- 172.21.15.31:0/3448835957 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f0144a30 con 0x7fb3f0004cb0 2024-03-21T10:50:20.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.807+0000 7fb3f7e00640 1 -- 172.21.15.31:0/3448835957 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f014c2c0 con 0x7fb3f0004cb0 2024-03-21T10:50:20.805 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-21T10:50:20.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.818+0000 7fb3ef400640 1 -- 172.21.15.31:0/3448835957 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fb3e80109d0 con 0x7fb3f0004cb0 2024-03-21T10:50:20.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.820+0000 7fb3ef400640 1 --2- 172.21.15.31:0/3448835957 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb3dc0a3290 0x7fb3dc0a5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:20.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.821+0000 7fb3ef400640 1 -- 172.21.15.31:0/3448835957 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fb3e80cf750 con 0x7fb3f0004cb0 2024-03-21T10:50:20.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.822+0000 7fb4005f5740 1 Processor -- start 2024-03-21T10:50:20.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.822+0000 7fb3f6000640 1 --2- 172.21.15.31:0/3448835957 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb3dc0a3290 0x7fb3dc0a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:20.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.824+0000 7fb3f6000640 1 --2- 172.21.15.31:0/3448835957 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb3dc0a3290 0x7fb3dc0a5750 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009900 tx=0x7fb3e4005fd0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:20.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:20 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1757712777' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:50:20.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.826+0000 7fb4005f5740 1 -- start start 2024-03-21T10:50:20.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.827+0000 7fb4005f5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e44ef3d850 0x55e44ef3dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:20.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.827+0000 7fb4005f5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e44ef3e200 0x55e44ef4f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:20.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.827+0000 7fb4005f5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e44ef4f5f0 0x55e44ef51a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:20.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.827+0000 7fb4005f5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e44eea12c0 con 0x55e44ef4f5f0 2024-03-21T10:50:20.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.827+0000 7fb4005f5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e44ee49c90 con 0x55e44ef3d850 2024-03-21T10:50:20.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.827+0000 7fb4005f5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e44ee9d360 con 0x55e44ef3e200 2024-03-21T10:50:20.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.827+0000 7fb3f6000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e44ef3e200 0x55e44ef4f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:20.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.827+0000 7fb3f6000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e44ef3e200 0x55e44ef4f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34880/0 (socket says 172.21.15.31:34880) 2024-03-21T10:50:20.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.827+0000 7fb3f6000640 1 -- 172.21.15.31:0/2799737000 learned_addr learned my addr 172.21.15.31:0/2799737000 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:20.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.828+0000 7fb3f6000640 1 -- 172.21.15.31:0/2799737000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e44ef3d850 msgr2=0x55e44ef3dc30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:50:20.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.828+0000 7fb3f6000640 1 --2- 172.21.15.31:0/2799737000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e44ef3d850 0x55e44ef3dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.828+0000 7fb3f6000640 1 -- 172.21.15.31:0/2799737000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e44ef4f5f0 msgr2=0x55e44ef51a00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:50:20.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.828+0000 7fb3f6000640 1 --2- 172.21.15.31:0/2799737000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e44ef4f5f0 0x55e44ef51a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.828+0000 7fb3f6000640 1 -- 172.21.15.31:0/2799737000 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e44ee6e080 con 0x55e44ef3e200 2024-03-21T10:50:20.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.828+0000 7fb3f6000640 1 --2- 172.21.15.31:0/2799737000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e44ef3e200 0x55e44ef4f020 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009930 tx=0x7fb3e403d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84fbe323560ce511 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:20.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.828+0000 7fb3e3e00640 1 -- 172.21.15.31:0/2799737000 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e402ccf0 con 0x55e44ef3e200 2024-03-21T10:50:20.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.829+0000 7fb3e3e00640 1 -- 172.21.15.31:0/2799737000 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3e404f070 con 0x55e44ef3e200 2024-03-21T10:50:20.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.830+0000 7fb4005f5740 1 -- 172.21.15.31:0/2799737000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e44ef3e200 msgr2=0x55e44ef4f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:20.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.830+0000 7fb4005f5740 1 --2- 172.21.15.31:0/2799737000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e44ef3e200 0x55e44ef4f020 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009930 tx=0x7fb3e403d000 comp rx=0 tx=0).stop 2024-03-21T10:50:20.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.831+0000 7fb4005f5740 1 -- 172.21.15.31:0/2799737000 shutdown_connections 2024-03-21T10:50:20.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.831+0000 7fb4005f5740 1 --2- 172.21.15.31:0/2799737000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e44ef4f5f0 0x55e44ef51a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.831+0000 7fb4005f5740 1 --2- 172.21.15.31:0/2799737000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e44ef3e200 0x55e44ef4f020 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.831+0000 7fb4005f5740 1 --2- 172.21.15.31:0/2799737000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e44ef3d850 0x55e44ef3dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.831+0000 7fb4005f5740 1 -- 172.21.15.31:0/2799737000 >> 172.21.15.31:0/2799737000 conn(0x55e44ef37f80 msgr2=0x55e44ef38360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:20.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.831+0000 7fb4005f5740 1 -- 172.21.15.31:0/2799737000 shutdown_connections 2024-03-21T10:50:20.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.832+0000 7fb4005f5740 1 -- 172.21.15.31:0/2799737000 wait complete. 2024-03-21T10:50:20.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.832+0000 7fb4005f5740 1 -- start start 2024-03-21T10:50:20.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.834+0000 7fb4005f5740 1 Processor -- start 2024-03-21T10:50:20.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.835+0000 7fb4005f5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e44ef3d850 0x55e44ef48ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:20.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.835+0000 7fb4005f5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e44ef3e200 0x55e44ef49420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:20.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.835+0000 7fb4005f5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e44ef4f5f0 0x55e44ef4ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:20.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.835+0000 7fb4005f5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e44ef3ebf0 con 0x55e44ef4f5f0 2024-03-21T10:50:20.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.835+0000 7fb4005f5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e44ee5d0d0 con 0x55e44ef3d850 2024-03-21T10:50:20.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.835+0000 7fb4005f5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e44ef3ed70 con 0x55e44ef3e200 2024-03-21T10:50:20.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.835+0000 7fb4005f5740 10 client.0 _ll_register_callbacks cb 0x55e44ee61e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:50:20.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.838+0000 7fb3f6000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e44ef3e200 0x55e44ef49420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:20.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.838+0000 7fb3f6000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e44ef3e200 0x55e44ef49420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34886/0 (socket says 172.21.15.31:34886) 2024-03-21T10:50:20.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.838+0000 7fb3f6000640 1 -- 172.21.15.31:0/3606784206 learned_addr learned my addr 172.21.15.31:0/3606784206 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:20.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.838+0000 7fb3f7400640 1 --2- 172.21.15.31:0/3606784206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e44ef4f5f0 0x55e44ef4ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:20.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.839+0000 7fb3f6a00640 1 --2- 172.21.15.31:0/3606784206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e44ef3d850 0x55e44ef48ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:20.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.840+0000 7fb3f7400640 1 -- 172.21.15.31:0/3606784206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e44ef3e200 msgr2=0x55e44ef49420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:20.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.840+0000 7fb3f7400640 1 --2- 172.21.15.31:0/3606784206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e44ef3e200 0x55e44ef49420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.840+0000 7fb3f7400640 1 -- 172.21.15.31:0/3606784206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e44ef3d850 msgr2=0x55e44ef48ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:20.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.840+0000 7fb3f7400640 1 --2- 172.21.15.31:0/3606784206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e44ef3d850 0x55e44ef48ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.840+0000 7fb3f7400640 1 -- 172.21.15.31:0/3606784206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e44ebb53f0 con 0x55e44ef4f5f0 2024-03-21T10:50:20.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.841+0000 7fb3f7400640 1 --2- 172.21.15.31:0/3606784206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e44ef4f5f0 0x55e44ef4ec00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb3f80674e0 tx=0x7fb3f8069770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87a5b2dd806f6336 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:20.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.841+0000 7fb3e0c00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-21T10:50:20.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.841+0000 7fb3e0c00640 1 -- 172.21.15.31:0/3606784206 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8073700 con 0x55e44ef4f5f0 2024-03-21T10:50:20.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.842+0000 7fb3e0c00640 1 -- 172.21.15.31:0/3606784206 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fb3f807d3e0 con 0x55e44ef4f5f0 2024-03-21T10:50:20.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.842+0000 7fb4005f5740 1 -- 172.21.15.31:0/3606784206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb3e404f070 con 0x55e44ef4f5f0 2024-03-21T10:50:20.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.843+0000 7fb3e0c00640 1 -- 172.21.15.31:0/3606784206 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3f807d5e0 con 0x55e44ef4f5f0 2024-03-21T10:50:20.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.843+0000 7fb4005f5740 10 client.17931 fetch_fsmap learned FSMap version 31 2024-03-21T10:50:20.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.843+0000 7fb4005f5740 1 -- 172.21.15.31:0/3606784206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55e44ec7f7d0 con 0x55e44ef4f5f0 2024-03-21T10:50:20.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.844+0000 7fb3e0c00640 1 -- 172.21.15.31:0/3606784206 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3e404f070 con 0x55e44ef4f5f0 2024-03-21T10:50:20.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.845+0000 7fb3e0c00640 1 -- 172.21.15.31:0/3606784206 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 31) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fb3f80888e0 con 0x55e44ef4f5f0 2024-03-21T10:50:20.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.846+0000 7fb4005f5740 10 client.17931 fetch_fsmap finished waiting for FSMap version 31 2024-03-21T10:50:20.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.846+0000 7fb4005f5740 10 client.17931 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:50:20.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.846+0000 7fb4005f5740 20 client.17931 populate_metadata read hostname 'smithi031' 2024-03-21T10:50:20.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.846+0000 7fb4005f5740 1 --2- 172.21.15.31:0/3606784206 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55e44ef56cf0 0x55e44ef590e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:20.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.846+0000 7fb4005f5740 4 client.17931 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:50:20.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.846+0000 7fb4005f5740 1 -- 172.21.15.31:0/3606784206 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e44eeae800 con 0x55e44ef56cf0 2024-03-21T10:50:20.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.847+0000 7fb3f6a00640 1 --2- 172.21.15.31:0/3606784206 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55e44ef56cf0 0x55e44ef590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:20.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.850+0000 7fb3f6a00640 1 --2- 172.21.15.31:0/3606784206 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55e44ef56cf0 0x55e44ef590e0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=919190b91124e156 server_cookie=714e716be4b98eb in_seq=0 out_seq=0 2024-03-21T10:50:20.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.850+0000 7fb3e0c00640 10 client.17931 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:50:20.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.851+0000 7fb3e0c00640 1 -- 172.21.15.31:0/3606784206 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e44eeae800 con 0x55e44ef56cf0 2024-03-21T10:50:20.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.851+0000 7fb3e0c00640 10 client.17931 handle_command_reply: tid=0 2024-03-21T10:50:20.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.851+0000 7fb4005f5740 1 client.17931 shutdown 2024-03-21T10:50:20.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.851+0000 7fb4005f5740 1 -- 172.21.15.31:0/3606784206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e44ef4f5f0 msgr2=0x55e44ef4ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:20.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.851+0000 7fb4005f5740 1 --2- 172.21.15.31:0/3606784206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e44ef4f5f0 0x55e44ef4ec00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb3f80674e0 tx=0x7fb3f8069770 comp rx=0 tx=0).stop 2024-03-21T10:50:20.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.851+0000 7fb4005f5740 1 -- 172.21.15.31:0/3606784206 shutdown_connections 2024-03-21T10:50:20.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.851+0000 7fb4005f5740 1 --2- 172.21.15.31:0/3606784206 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55e44ef56cf0 0x55e44ef590e0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.851+0000 7fb3e0c00640 0 client.17931 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:50:20.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.851+0000 7fb4005f5740 1 --2- 172.21.15.31:0/3606784206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e44ef4f5f0 0x55e44ef4ec00 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.851+0000 7fb4005f5740 1 --2- 172.21.15.31:0/3606784206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e44ef3e200 0x55e44ef49420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.851+0000 7fb4005f5740 1 --2- 172.21.15.31:0/3606784206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e44ef3d850 0x55e44ef48ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.851+0000 7fb4005f5740 1 -- 172.21.15.31:0/3606784206 >> 172.21.15.31:0/3606784206 conn(0x55e44ef37f80 msgr2=0x55e44ef39cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:20.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.852+0000 7fb4005f5740 1 -- 172.21.15.31:0/3606784206 shutdown_connections 2024-03-21T10:50:20.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.852+0000 7fb4005f5740 1 -- 172.21.15.31:0/3606784206 wait complete. 2024-03-21T10:50:20.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.852+0000 7fb4005f5740 20 client.17931 trim_cache size 0 max 16384 2024-03-21T10:50:20.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.880+0000 7fb4005f5740 1 Processor -- start 2024-03-21T10:50:20.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.880+0000 7fb4005f5740 1 -- start start 2024-03-21T10:50:20.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.881+0000 7fb4005f5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e44ef3d850 0x55e44ef3fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:20.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.881+0000 7fb4005f5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e44ef3e200 0x55e44ef4ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:20.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.881+0000 7fb4005f5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e44ef4f5f0 0x55e44ef41540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:20.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.881+0000 7fb4005f5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f0176340 con 0x55e44ef4f5f0 2024-03-21T10:50:20.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.881+0000 7fb4005f5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f0173e10 con 0x55e44ef3e200 2024-03-21T10:50:20.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.881+0000 7fb4005f5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f0171f10 con 0x55e44ef3d850 2024-03-21T10:50:20.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.882+0000 7fb3f6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e44ef3e200 0x55e44ef4ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:20.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.882+0000 7fb3f6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e44ef3d850 0x55e44ef3fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:20.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.882+0000 7fb3f6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e44ef3e200 0x55e44ef4ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:52734/0 (socket says 172.21.15.31:52734) 2024-03-21T10:50:20.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.882+0000 7fb3f6000640 1 -- 172.21.15.31:0/1285497068 learned_addr learned my addr 172.21.15.31:0/1285497068 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:20.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.882+0000 7fb3f7400640 1 --2- 172.21.15.31:0/1285497068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e44ef4f5f0 0x55e44ef41540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:20.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.882+0000 7fb3f6a00640 1 -- 172.21.15.31:0/1285497068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e44ef3e200 msgr2=0x55e44ef4ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:20.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.882+0000 7fb3f6a00640 1 --2- 172.21.15.31:0/1285497068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e44ef3e200 0x55e44ef4ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.882+0000 7fb3f6a00640 1 -- 172.21.15.31:0/1285497068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e44ef4f5f0 msgr2=0x55e44ef41540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:20.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.882+0000 7fb3f6a00640 1 --2- 172.21.15.31:0/1285497068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e44ef4f5f0 0x55e44ef41540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.883+0000 7fb3f6a00640 1 -- 172.21.15.31:0/1285497068 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e44ef37db0 con 0x55e44ef3d850 2024-03-21T10:50:20.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.883+0000 7fb3f6a00640 1 --2- 172.21.15.31:0/1285497068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e44ef3d850 0x55e44ef3fb50 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fb3e80ccaa0 tx=0x7fb3e800bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51443b8ce833b7b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:20.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.884+0000 7fb3e3400640 1 -- 172.21.15.31:0/1285497068 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e8020040 con 0x55e44ef3d850 2024-03-21T10:50:20.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.884+0000 7fb3e3400640 1 -- 172.21.15.31:0/1285497068 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3e809b740 con 0x55e44ef3d850 2024-03-21T10:50:20.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.884+0000 7fb3e3400640 1 -- 172.21.15.31:0/1285497068 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e80a3cc0 con 0x55e44ef3d850 2024-03-21T10:50:20.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.887+0000 7fb4005f5740 1 -- 172.21.15.31:0/1285497068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e44ef3d850 msgr2=0x55e44ef3fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:20.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.887+0000 7fb4005f5740 1 --2- 172.21.15.31:0/1285497068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e44ef3d850 0x55e44ef3fb50 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fb3e80ccaa0 tx=0x7fb3e800bf60 comp rx=0 tx=0).stop 2024-03-21T10:50:20.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.888+0000 7fb4005f5740 1 -- 172.21.15.31:0/1285497068 shutdown_connections 2024-03-21T10:50:20.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.888+0000 7fb4005f5740 1 --2- 172.21.15.31:0/1285497068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e44ef4f5f0 0x55e44ef41540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.888+0000 7fb4005f5740 1 --2- 172.21.15.31:0/1285497068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e44ef3e200 0x55e44ef4ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.888+0000 7fb4005f5740 1 --2- 172.21.15.31:0/1285497068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e44ef3d850 0x55e44ef3fb50 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.888+0000 7fb4005f5740 1 -- 172.21.15.31:0/1285497068 >> 172.21.15.31:0/1285497068 conn(0x55e44ef37f80 msgr2=0x55e44ef37160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:20.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.889+0000 7fb4005f5740 1 -- 172.21.15.31:0/1285497068 shutdown_connections 2024-03-21T10:50:20.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.889+0000 7fb4005f5740 1 -- 172.21.15.31:0/1285497068 wait complete. 2024-03-21T10:50:20.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.890+0000 7fb4005f5740 1 -- start start 2024-03-21T10:50:20.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.890+0000 7fb4005f5740 1 Processor -- start 2024-03-21T10:50:20.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.891+0000 7fb4005f5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e44ef3d850 0x55e44efa6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:20.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.891+0000 7fb4005f5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e44ef3e200 0x55e44efa6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:20.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.891+0000 7fb4005f5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e44ef4f5f0 0x55e44efb15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:20.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.892+0000 7fb4005f5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f0172510 con 0x55e44ef4f5f0 2024-03-21T10:50:20.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.892+0000 7fb4005f5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f01737b0 con 0x55e44ef3e200 2024-03-21T10:50:20.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.892+0000 7fb4005f5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f0172b10 con 0x55e44ef3d850 2024-03-21T10:50:20.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.892+0000 7fb3f6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e44ef3d850 0x55e44efa6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:20.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.892+0000 7fb4005f5740 10 client.0 _ll_register_callbacks cb 0x7fb3f8062af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:50:20.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.892+0000 7fb3f6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e44ef3d850 0x55e44efa6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34906/0 (socket says 172.21.15.31:34906) 2024-03-21T10:50:20.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.892+0000 7fb3f6a00640 1 -- 172.21.15.31:0/3727387144 learned_addr learned my addr 172.21.15.31:0/3727387144 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:20.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.892+0000 7fb3f6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e44ef3e200 0x55e44efa6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:20.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.892+0000 7fb3f7400640 1 --2- 172.21.15.31:0/3727387144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e44ef4f5f0 0x55e44efb15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:20.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.892+0000 7fb3f6a00640 1 -- 172.21.15.31:0/3727387144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e44ef3e200 msgr2=0x55e44efa6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:20.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.892+0000 7fb3f6a00640 1 --2- 172.21.15.31:0/3727387144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e44ef3e200 0x55e44efa6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.892+0000 7fb3f6a00640 1 -- 172.21.15.31:0/3727387144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e44ef4f5f0 msgr2=0x55e44efb15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:20.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.892+0000 7fb3f6a00640 1 --2- 172.21.15.31:0/3727387144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e44ef4f5f0 0x55e44efb15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.892+0000 7fb3f6a00640 1 -- 172.21.15.31:0/3727387144 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e44efb1b20 con 0x55e44ef3d850 2024-03-21T10:50:20.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.893+0000 7fb3f6a00640 1 --2- 172.21.15.31:0/3727387144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e44ef3d850 0x55e44efa6720 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8097ba0 tx=0x7fb3e809f960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfe1614a6bf2a4c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:20.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.893+0000 7fb3e0c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:50:20.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.893+0000 7fb3e0c00640 1 -- 172.21.15.31:0/3727387144 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e80b9060 con 0x55e44ef3d850 2024-03-21T10:50:20.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.893+0000 7fb3e0c00640 1 -- 172.21.15.31:0/3727387144 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fb3e80a3cc0 con 0x55e44ef3d850 2024-03-21T10:50:20.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.894+0000 7fb4005f5740 1 -- 172.21.15.31:0/3727387144 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb3e809b740 con 0x55e44ef3d850 2024-03-21T10:50:20.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.895+0000 7fb3e0c00640 1 -- 172.21.15.31:0/3727387144 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3e8020070 con 0x55e44ef3d850 2024-03-21T10:50:20.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.896+0000 7fb4005f5740 10 client.28528 fetch_fsmap learned FSMap version 31 2024-03-21T10:50:20.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.897+0000 7fb4005f5740 1 -- 172.21.15.31:0/3727387144 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55e44efb2e20 con 0x55e44ef3d850 2024-03-21T10:50:20.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.899+0000 7fb3e0c00640 1 -- 172.21.15.31:0/3727387144 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3e809b740 con 0x55e44ef3d850 2024-03-21T10:50:20.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.899+0000 7fb3e0c00640 1 -- 172.21.15.31:0/3727387144 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) v1 ==== 7930+0+0 (secure 0 0 0) 0x7fb3e80cf270 con 0x55e44ef3d850 2024-03-21T10:50:20.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.899+0000 7fb4005f5740 10 client.28528 fetch_fsmap finished waiting for FSMap version 31 2024-03-21T10:50:20.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.899+0000 7fb4005f5740 10 client.28528 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-21T10:50:20.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.899+0000 7fb4005f5740 20 client.28528 populate_metadata read hostname 'smithi031' 2024-03-21T10:50:20.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.899+0000 7fb4005f5740 1 --2- 172.21.15.31:0/3727387144 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55e44ef56cf0 0x55e44efb5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:20.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.899+0000 7fb4005f5740 4 client.28528 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:50:20.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.899+0000 7fb4005f5740 1 -- 172.21.15.31:0/3727387144 --> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e44eea8b50 con 0x55e44ef56cf0 2024-03-21T10:50:20.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.900+0000 7fb3f6000640 1 --2- 172.21.15.31:0/3727387144 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55e44ef56cf0 0x55e44efb5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:20.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.900+0000 7fb3f6000640 1 --2- 172.21.15.31:0/3727387144 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55e44ef56cf0 0x55e44efb5380 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a9986596020bda83 server_cookie=22c62d4176724067 in_seq=0 out_seq=0 2024-03-21T10:50:20.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.900+0000 7fb3e0c00640 10 client.28528 ms_handle_connect on v2:172.21.15.70:6838/3204327153 2024-03-21T10:50:20.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.901+0000 7fb3e0c00640 1 -- 172.21.15.31:0/3727387144 <== mds.2 v2:172.21.15.70:6838/3204327153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e44eea8b50 con 0x55e44ef56cf0 2024-03-21T10:50:20.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.901+0000 7fb3e0c00640 10 client.28528 handle_command_reply: tid=0 2024-03-21T10:50:20.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.901+0000 7fb4005f5740 1 client.28528 shutdown 2024-03-21T10:50:20.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.902+0000 7fb4005f5740 1 -- 172.21.15.31:0/3727387144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e44ef3d850 msgr2=0x55e44efa6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:20.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.902+0000 7fb4005f5740 1 --2- 172.21.15.31:0/3727387144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e44ef3d850 0x55e44efa6720 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8097ba0 tx=0x7fb3e809f960 comp rx=0 tx=0).stop 2024-03-21T10:50:20.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.902+0000 7fb4005f5740 1 -- 172.21.15.31:0/3727387144 shutdown_connections 2024-03-21T10:50:20.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.902+0000 7fb4005f5740 1 --2- 172.21.15.31:0/3727387144 >> [v2:172.21.15.70:6838/3204327153,v1:172.21.15.70:6839/3204327153] conn(0x55e44ef56cf0 0x55e44efb5380 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.902+0000 7fb4005f5740 1 --2- 172.21.15.31:0/3727387144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e44ef4f5f0 0x55e44efb15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.902+0000 7fb4005f5740 1 --2- 172.21.15.31:0/3727387144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e44ef3e200 0x55e44efa6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.902+0000 7fb4005f5740 1 --2- 172.21.15.31:0/3727387144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e44ef3d850 0x55e44efa6720 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.902+0000 7fb4005f5740 1 -- 172.21.15.31:0/3727387144 >> 172.21.15.31:0/3727387144 conn(0x55e44ef37f80 msgr2=0x55e44ef390b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:20.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.902+0000 7fb3e0c00640 0 client.28528 ms_handle_reset on v2:172.21.15.70:6838/3204327153 2024-03-21T10:50:20.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.903+0000 7fb4005f5740 1 -- 172.21.15.31:0/3727387144 shutdown_connections 2024-03-21T10:50:20.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.903+0000 7fb4005f5740 1 -- 172.21.15.31:0/3727387144 wait complete. 2024-03-21T10:50:20.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.903+0000 7fb4005f5740 20 client.28528 trim_cache size 0 max 16384 2024-03-21T10:50:20.903 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:50:20.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.904+0000 7fb3f7e00640 1 -- 172.21.15.31:0/3448835957 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb3dc0a3290 msgr2=0x7fb3dc0a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:20.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.904+0000 7fb3f7e00640 1 --2- 172.21.15.31:0/3448835957 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb3dc0a3290 0x7fb3dc0a5750 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009900 tx=0x7fb3e4005fd0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.904+0000 7fb3f7e00640 1 -- 172.21.15.31:0/3448835957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3f0004cb0 msgr2=0x7fb3f013f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:20.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.904+0000 7fb3f7e00640 1 --2- 172.21.15.31:0/3448835957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3f0004cb0 0x7fb3f013f750 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fb3e800d950 tx=0x7fb3e800de10 comp rx=0 tx=0).stop 2024-03-21T10:50:20.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.905+0000 7fb3f7e00640 1 -- 172.21.15.31:0/3448835957 shutdown_connections 2024-03-21T10:50:20.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.905+0000 7fb3f7e00640 1 --2- 172.21.15.31:0/3448835957 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fb3dc0a3290 0x7fb3dc0a5750 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.905+0000 7fb3f7e00640 1 --2- 172.21.15.31:0/3448835957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3f00a2a00 0x7fb3f0144010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.905+0000 7fb3f7e00640 1 --2- 172.21.15.31:0/3448835957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3f00055d0 0x7fb3f013fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.905+0000 7fb3f7e00640 1 --2- 172.21.15.31:0/3448835957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3f0004cb0 0x7fb3f013f750 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:20.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.905+0000 7fb3f7e00640 1 -- 172.21.15.31:0/3448835957 >> 172.21.15.31:0/3448835957 conn(0x7fb3f000a740 msgr2=0x7fb3f00a3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:20.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.906+0000 7fb3f7e00640 1 -- 172.21.15.31:0/3448835957 shutdown_connections 2024-03-21T10:50:20.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:20.906+0000 7fb3f7e00640 1 -- 172.21.15.31:0/3448835957 wait complete. 2024-03-21T10:50:20.929 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:20 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1757712777' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:50:20.940 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:50:20.940 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-21T10:50:20.963 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-21T10:50:21.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.324+0000 7f4b5ca00640 1 -- 172.21.15.31:0/1282894755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b500a6b90 msgr2=0x7f4b500a8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:21.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.324+0000 7f4b5ca00640 1 --2- 172.21.15.31:0/1282894755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b500a6b90 0x7f4b500a8f80 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f4b48009890 tx=0x7f4b4802bb70 comp rx=0 tx=0).stop 2024-03-21T10:50:21.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.325+0000 7f4b5ca00640 1 -- 172.21.15.31:0/1282894755 shutdown_connections 2024-03-21T10:50:21.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.325+0000 7f4b5ca00640 1 --2- 172.21.15.31:0/1282894755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b500a6b90 0x7f4b500a8f80 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.325+0000 7f4b5ca00640 1 --2- 172.21.15.31:0/1282894755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b500099d0 0x7f4b500a6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.325+0000 7f4b5ca00640 1 --2- 172.21.15.31:0/1282894755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b500090b0 0x7f4b50009490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.325+0000 7f4b5ca00640 1 -- 172.21.15.31:0/1282894755 >> 172.21.15.31:0/1282894755 conn(0x7f4b500939a0 msgr2=0x7f4b50095dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:21.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.325+0000 7f4b5ca00640 1 -- 172.21.15.31:0/1282894755 shutdown_connections 2024-03-21T10:50:21.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.325+0000 7f4b5ca00640 1 -- 172.21.15.31:0/1282894755 wait complete. 2024-03-21T10:50:21.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.328+0000 7f4b5ca00640 1 Processor -- start 2024-03-21T10:50:21.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.328+0000 7f4b5ca00640 1 -- start start 2024-03-21T10:50:21.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.329+0000 7f4b5ca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b500090b0 0x7f4b500a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.329+0000 7f4b5ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b500099d0 0x7f4b500a23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.329+0000 7f4b5ca00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b500a6b90 0x7f4b500a28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.329+0000 7f4b5ca00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5009a790 con 0x7f4b500090b0 2024-03-21T10:50:21.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.329+0000 7f4b5ca00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5009a610 con 0x7f4b500099d0 2024-03-21T10:50:21.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.329+0000 7f4b5ca00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5009a910 con 0x7f4b500a6b90 2024-03-21T10:50:21.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.330+0000 7f4b57200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b500a6b90 0x7f4b500a28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.330+0000 7f4b57200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b500a6b90 0x7f4b500a28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34924/0 (socket says 172.21.15.31:34924) 2024-03-21T10:50:21.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.330+0000 7f4b57200640 1 -- 172.21.15.31:0/392737335 learned_addr learned my addr 172.21.15.31:0/392737335 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:21.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.330+0000 7f4b57200640 1 -- 172.21.15.31:0/392737335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b500099d0 msgr2=0x7f4b500a23b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T10:50:21.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.330+0000 7f4b57200640 1 --2- 172.21.15.31:0/392737335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b500099d0 0x7f4b500a23b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.330+0000 7f4b57200640 1 -- 172.21.15.31:0/392737335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b500090b0 msgr2=0x7f4b500a6280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T10:50:21.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.330+0000 7f4b57200640 1 --2- 172.21.15.31:0/392737335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b500090b0 0x7f4b500a6280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.330+0000 7f4b57200640 1 -- 172.21.15.31:0/392737335 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b500a3020 con 0x7f4b500a6b90 2024-03-21T10:50:21.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.331+0000 7f4b57200640 1 --2- 172.21.15.31:0/392737335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b500a6b90 0x7f4b500a28f0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f4b48037040 tx=0x7f4b480045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:21.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.331+0000 7f4b4f400640 1 -- 172.21.15.31:0/392737335 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b48041020 con 0x7f4b500a6b90 2024-03-21T10:50:21.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.331+0000 7f4b4f400640 1 -- 172.21.15.31:0/392737335 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b4802c9f0 con 0x7f4b500a6b90 2024-03-21T10:50:21.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.331+0000 7f4b4f400640 1 -- 172.21.15.31:0/392737335 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b48039040 con 0x7f4b500a6b90 2024-03-21T10:50:21.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.334+0000 7f4b5ca00640 1 -- 172.21.15.31:0/392737335 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b50147ad0 con 0x7f4b500a6b90 2024-03-21T10:50:21.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.334+0000 7f4b5ca00640 1 -- 172.21.15.31:0/392737335 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b50147e80 con 0x7f4b500a6b90 2024-03-21T10:50:21.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.336+0000 7f245ce00640 1 -- 172.21.15.31:0/2269089073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2458107650 msgr2=0x7f2458107a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:21.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.336+0000 7f245ce00640 1 --2- 172.21.15.31:0/2269089073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2458107650 0x7f2458107a30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f244001a940 tx=0x7f244003cca0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.336+0000 7f245ce00640 1 -- 172.21.15.31:0/2269089073 shutdown_connections 2024-03-21T10:50:21.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.336+0000 7f245ce00640 1 --2- 172.21.15.31:0/2269089073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2458102000 0x7f245810e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.336+0000 7f245ce00640 1 --2- 172.21.15.31:0/2269089073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2458101640 0x7f2458101ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.336+0000 7f245ce00640 1 --2- 172.21.15.31:0/2269089073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2458107650 0x7f2458107a30 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.336+0000 7f245ce00640 1 -- 172.21.15.31:0/2269089073 >> 172.21.15.31:0/2269089073 conn(0x7f24580fd130 msgr2=0x7f24580ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:21.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.338+0000 7f245ce00640 1 -- 172.21.15.31:0/2269089073 shutdown_connections 2024-03-21T10:50:21.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.339+0000 7f245ce00640 1 -- 172.21.15.31:0/2269089073 wait complete. 2024-03-21T10:50:21.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.340+0000 7f245ce00640 1 Processor -- start 2024-03-21T10:50:21.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.341+0000 7f245ce00640 1 -- start start 2024-03-21T10:50:21.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.340+0000 7f4b4f400640 1 -- 172.21.15.31:0/392737335 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f4b48044020 con 0x7f4b500a6b90 2024-03-21T10:50:21.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.340+0000 7f4b5ca00640 1 -- 172.21.15.31:0/392737335 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b5000a7f0 con 0x7f4b500a6b90 2024-03-21T10:50:21.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.342+0000 7f245ce00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2458101640 0x7f245810e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.342+0000 7f245ce00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2458102000 0x7f245810a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.343+0000 7f4b4f400640 1 --2- 172.21.15.31:0/392737335 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4b3c0a3150 0x7f4b3c0a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.342+0000 7f245ce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2458107650 0x7f245810a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.342+0000 7f245ce00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f245806db30 con 0x7f2458107650 2024-03-21T10:50:21.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.342+0000 7f245ce00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f245806d9b0 con 0x7f2458102000 2024-03-21T10:50:21.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.342+0000 7f245ce00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f245806dcb0 con 0x7f2458101640 2024-03-21T10:50:21.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.343+0000 7f2455e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2458101640 0x7f245810e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.343+0000 7f2455e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2458101640 0x7f245810e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34932/0 (socket says 172.21.15.31:34932) 2024-03-21T10:50:21.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.343+0000 7f2456800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2458107650 0x7f245810a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.343+0000 7f2455e00640 1 -- 172.21.15.31:0/1153931125 learned_addr learned my addr 172.21.15.31:0/1153931125 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:21.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.343+0000 7f244fe00640 1 --2- 172.21.15.31:0/1153931125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2458102000 0x7f245810a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.343+0000 7f4b4f400640 1 -- 172.21.15.31:0/392737335 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f4b480f0a90 con 0x7f4b500a6b90 2024-03-21T10:50:21.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.343+0000 7f4b56800640 1 --2- 172.21.15.31:0/392737335 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4b3c0a3150 0x7f4b3c0a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.344+0000 7f4b56800640 1 --2- 172.21.15.31:0/392737335 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4b3c0a3150 0x7f4b3c0a5610 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4b580674f0 tx=0x7f4b5804faa0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:21.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.344+0000 7f2455e00640 1 -- 172.21.15.31:0/1153931125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2458102000 msgr2=0x7f245810a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:21.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.344+0000 7f2455e00640 1 --2- 172.21.15.31:0/1153931125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2458102000 0x7f245810a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.344+0000 7f2455e00640 1 -- 172.21.15.31:0/1153931125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2458107650 msgr2=0x7f245810a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:21.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.344+0000 7f2455e00640 1 --2- 172.21.15.31:0/1153931125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2458107650 0x7f245810a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.344+0000 7f2455e00640 1 -- 172.21.15.31:0/1153931125 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f245810b180 con 0x7f2458101640 2024-03-21T10:50:21.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.344+0000 7f244fe00640 1 --2- 172.21.15.31:0/1153931125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2458102000 0x7f245810a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:50:21.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.344+0000 7f2456800640 1 --2- 172.21.15.31:0/1153931125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2458107650 0x7f245810a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T10:50:21.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.346+0000 7f2455e00640 1 --2- 172.21.15.31:0/1153931125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2458101640 0x7f245810e2d0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f244001a910 tx=0x7f2440004e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:21.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.350+0000 7f244ea00640 1 -- 172.21.15.31:0/1153931125 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2440004410 con 0x7f2458101640 2024-03-21T10:50:21.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.350+0000 7f244ea00640 1 -- 172.21.15.31:0/1153931125 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2440053070 con 0x7f2458101640 2024-03-21T10:50:21.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.350+0000 7f244ea00640 1 -- 172.21.15.31:0/1153931125 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2440003de0 con 0x7f2458101640 2024-03-21T10:50:21.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.350+0000 7f245ce00640 1 -- 172.21.15.31:0/1153931125 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24580738d0 con 0x7f2458101640 2024-03-21T10:50:21.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.351+0000 7f245ce00640 1 -- 172.21.15.31:0/1153931125 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2458073d70 con 0x7f2458101640 2024-03-21T10:50:21.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.351+0000 7f4b4f400640 1 -- 172.21.15.31:0/392737335 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4b4804b020 con 0x7f4b500a6b90 2024-03-21T10:50:21.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.359+0000 7f244ea00640 1 -- 172.21.15.31:0/1153931125 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f244003d850 con 0x7f2458101640 2024-03-21T10:50:21.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.363+0000 7f244ea00640 1 --2- 172.21.15.31:0/1153931125 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f243c0a3110 0x7f243c0a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.363+0000 7f245fa00740 1 Processor -- start 2024-03-21T10:50:21.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.363+0000 7f244ea00640 1 -- 172.21.15.31:0/1153931125 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f24401022b0 con 0x7f2458101640 2024-03-21T10:50:21.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.364+0000 7f244fe00640 1 --2- 172.21.15.31:0/1153931125 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f243c0a3110 0x7f243c0a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.364+0000 7f245fa00740 1 -- start start 2024-03-21T10:50:21.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.364+0000 7f245fa00740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f4123e450 0x558f4123e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.364+0000 7f245fa00740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f4123ee00 0x558f4124fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.364+0000 7f245fa00740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f41250410 0x558f41252820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.364+0000 7f245fa00740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f411a22c0 con 0x558f41250410 2024-03-21T10:50:21.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.364+0000 7f245fa00740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f4114ac90 con 0x558f4123e450 2024-03-21T10:50:21.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.364+0000 7f245fa00740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f4119e360 con 0x558f4123ee00 2024-03-21T10:50:21.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.365+0000 7f2455e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f4123e450 0x558f4123e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.365+0000 7f2455e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f4123e450 0x558f4123e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:52788/0 (socket says 172.21.15.31:52788) 2024-03-21T10:50:21.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.365+0000 7f2455e00640 1 -- 172.21.15.31:0/4225232928 learned_addr learned my addr 172.21.15.31:0/4225232928 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:21.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.365+0000 7f2455e00640 1 -- 172.21.15.31:0/4225232928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f4123ee00 msgr2=0x558f4124fe40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T10:50:21.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.365+0000 7f2455e00640 1 --2- 172.21.15.31:0/4225232928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f4123ee00 0x558f4124fe40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.365+0000 7f2455e00640 1 -- 172.21.15.31:0/4225232928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f41250410 msgr2=0x558f41252820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:21.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.365+0000 7f2456800640 1 --2- 172.21.15.31:0/4225232928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f41250410 0x558f41252820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.365+0000 7f2455e00640 1 --2- 172.21.15.31:0/4225232928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f41250410 0x558f41252820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.365+0000 7f2455e00640 1 -- 172.21.15.31:0/4225232928 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f4116f080 con 0x558f4123e450 2024-03-21T10:50:21.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.365+0000 7f2456800640 1 --2- 172.21.15.31:0/4225232928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f41250410 0x558f41252820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:50:21.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.366+0000 7f2455e00640 1 --2- 172.21.15.31:0/4225232928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f4123e450 0x558f4123e830 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f24400cade0 tx=0x7f24400ca7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fe6b342466b3b19 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:21.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.366+0000 7f244fe00640 1 --2- 172.21.15.31:0/1153931125 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f243c0a3110 0x7f243c0a55d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2448006fd0 tx=0x7f2448008040 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:21.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.366+0000 7f2447400640 1 -- 172.21.15.31:0/4225232928 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24400d2560 con 0x558f4123e450 2024-03-21T10:50:21.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.366+0000 7f2447400640 1 -- 172.21.15.31:0/4225232928 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f24400ee040 con 0x558f4123e450 2024-03-21T10:50:21.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.366+0000 7f2447400640 1 -- 172.21.15.31:0/4225232928 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24400caa80 con 0x558f4123e450 2024-03-21T10:50:21.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.367+0000 7f245fa00740 1 -- 172.21.15.31:0/4225232928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f4123e450 msgr2=0x558f4123e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:21.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.367+0000 7f245fa00740 1 --2- 172.21.15.31:0/4225232928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f4123e450 0x558f4123e830 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f24400cade0 tx=0x7f24400ca7e0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.367+0000 7f245fa00740 1 -- 172.21.15.31:0/4225232928 shutdown_connections 2024-03-21T10:50:21.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.367+0000 7f245fa00740 1 --2- 172.21.15.31:0/4225232928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f41250410 0x558f41252820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.367+0000 7f245fa00740 1 --2- 172.21.15.31:0/4225232928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f4123ee00 0x558f4124fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.367+0000 7f245fa00740 1 --2- 172.21.15.31:0/4225232928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f4123e450 0x558f4123e830 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.367+0000 7f245fa00740 1 -- 172.21.15.31:0/4225232928 >> 172.21.15.31:0/4225232928 conn(0x558f41238da0 msgr2=0x558f41239180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:21.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.369+0000 7f245fa00740 1 -- 172.21.15.31:0/4225232928 shutdown_connections 2024-03-21T10:50:21.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.369+0000 7f245fa00740 1 -- 172.21.15.31:0/4225232928 wait complete. 2024-03-21T10:50:21.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.369+0000 7f245fa00740 1 -- start start 2024-03-21T10:50:21.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.370+0000 7f245fa00740 1 Processor -- start 2024-03-21T10:50:21.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.370+0000 7f245fa00740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f4123e450 0x558f4124d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.370+0000 7f245fa00740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f4123ee00 0x558f4124d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.370+0000 7f245fa00740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f41250410 0x558f4124dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.370+0000 7f245fa00740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f4114ac90 con 0x558f4123ee00 2024-03-21T10:50:21.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.370+0000 7f245fa00740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f4115e0d0 con 0x558f4123e450 2024-03-21T10:50:21.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.370+0000 7f245fa00740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f41253100 con 0x558f41250410 2024-03-21T10:50:21.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.370+0000 7f245fa00740 10 client.0 _ll_register_callbacks cb 0x558f41162e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:50:21.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.371+0000 7f244fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f4123ee00 0x558f4124d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.371+0000 7f2456800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f41250410 0x558f4124dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.371+0000 7f2456800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f41250410 0x558f4124dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34962/0 (socket says 172.21.15.31:34962) 2024-03-21T10:50:21.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.371+0000 7f2456800640 1 -- 172.21.15.31:0/453877116 learned_addr learned my addr 172.21.15.31:0/453877116 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:21.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.371+0000 7f2455e00640 1 --2- 172.21.15.31:0/453877116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f4123e450 0x558f4124d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.372+0000 7f2456800640 1 -- 172.21.15.31:0/453877116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f4123e450 msgr2=0x558f4124d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:21.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.372+0000 7f2456800640 1 --2- 172.21.15.31:0/453877116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f4123e450 0x558f4124d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.372+0000 7f2456800640 1 -- 172.21.15.31:0/453877116 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f4123ee00 msgr2=0x558f4124d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:21.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.372+0000 7f2456800640 1 --2- 172.21.15.31:0/453877116 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f4123ee00 0x558f4124d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.372+0000 7f2456800640 1 -- 172.21.15.31:0/453877116 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558f40eb63f0 con 0x558f41250410 2024-03-21T10:50:21.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.372+0000 7f2456800640 1 --2- 172.21.15.31:0/453877116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f41250410 0x558f4124dc30 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f245000dbb0 tx=0x7f245000a3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86431f1b2854e91 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:21.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.372+0000 7f243be00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:50:21.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.373+0000 7f243be00640 1 -- 172.21.15.31:0/453877116 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245001b060 con 0x558f41250410 2024-03-21T10:50:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.373+0000 7f243be00640 1 -- 172.21.15.31:0/453877116 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f245000ac10 con 0x558f41250410 2024-03-21T10:50:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.373+0000 7f245fa00740 1 -- 172.21.15.31:0/453877116 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f24400ee040 con 0x558f41250410 2024-03-21T10:50:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.373+0000 7f243be00640 1 -- 172.21.15.31:0/453877116 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f245000ae50 con 0x558f41250410 2024-03-21T10:50:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.373+0000 7f245fa00740 10 client.28540 fetch_fsmap learned FSMap version 31 2024-03-21T10:50:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.373+0000 7f245fa00740 1 -- 172.21.15.31:0/453877116 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x558f40f807d0 con 0x558f41250410 2024-03-21T10:50:21.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.373+0000 7f243be00640 1 -- 172.21.15.31:0/453877116 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f24400ee040 con 0x558f41250410 2024-03-21T10:50:21.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.374+0000 7f243be00640 1 -- 172.21.15.31:0/453877116 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f2450016530 con 0x558f41250410 2024-03-21T10:50:21.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.374+0000 7f245fa00740 10 client.28540 fetch_fsmap finished waiting for FSMap version 31 2024-03-21T10:50:21.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.374+0000 7f245fa00740 10 client.28540 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:50:21.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.374+0000 7f245fa00740 20 client.28540 populate_metadata read hostname 'smithi031' 2024-03-21T10:50:21.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.374+0000 7f245fa00740 1 --2- 172.21.15.31:0/453877116 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x558f4125ff00 0x558f412622f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.374+0000 7f245fa00740 4 client.28540 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:50:21.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.375+0000 7f245fa00740 1 -- 172.21.15.31:0/453877116 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558f411af800 con 0x558f4125ff00 2024-03-21T10:50:21.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.375+0000 7f2455e00640 1 --2- 172.21.15.31:0/453877116 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x558f4125ff00 0x558f412622f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.377+0000 7f2455e00640 1 --2- 172.21.15.31:0/453877116 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x558f4125ff00 0x558f412622f0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a7fec5206e875668 server_cookie=b0aedc9b0d71d83a in_seq=0 out_seq=0 2024-03-21T10:50:21.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.377+0000 7f243be00640 10 client.28540 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:50:21.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.378+0000 7f243be00640 1 -- 172.21.15.31:0/453877116 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558f411af800 con 0x558f4125ff00 2024-03-21T10:50:21.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.378+0000 7f243be00640 10 client.28540 handle_command_reply: tid=0 2024-03-21T10:50:21.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.379+0000 7f245fa00740 1 client.28540 shutdown 2024-03-21T10:50:21.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.379+0000 7f245fa00740 1 -- 172.21.15.31:0/453877116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f41250410 msgr2=0x558f4124dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:21.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.379+0000 7f245fa00740 1 --2- 172.21.15.31:0/453877116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f41250410 0x558f4124dc30 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f245000dbb0 tx=0x7f245000a3e0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.379+0000 7f245fa00740 1 -- 172.21.15.31:0/453877116 shutdown_connections 2024-03-21T10:50:21.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.379+0000 7f245fa00740 1 --2- 172.21.15.31:0/453877116 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x558f4125ff00 0x558f412622f0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.379+0000 7f245fa00740 1 --2- 172.21.15.31:0/453877116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f41250410 0x558f4124dc30 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.379+0000 7f243be00640 0 client.28540 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:50:21.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.379+0000 7f245fa00740 1 --2- 172.21.15.31:0/453877116 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f4123ee00 0x558f4124d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.379+0000 7f245fa00740 1 --2- 172.21.15.31:0/453877116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f4123e450 0x558f4124d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.379+0000 7f245fa00740 1 -- 172.21.15.31:0/453877116 >> 172.21.15.31:0/453877116 conn(0x558f41238da0 msgr2=0x558f4123aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:21.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.380+0000 7f245fa00740 1 -- 172.21.15.31:0/453877116 shutdown_connections 2024-03-21T10:50:21.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.380+0000 7f245fa00740 1 -- 172.21.15.31:0/453877116 wait complete. 2024-03-21T10:50:21.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.380+0000 7f245fa00740 20 client.28540 trim_cache size 0 max 16384 2024-03-21T10:50:21.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.427+0000 7f245fa00740 1 Processor -- start 2024-03-21T10:50:21.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.427+0000 7f245fa00740 1 -- start start 2024-03-21T10:50:21.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.428+0000 7f245fa00740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f4123e450 0x558f412488b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.428+0000 7f245fa00740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f4123ee00 0x558f4124da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.428+0000 7f245fa00740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f41250410 0x558f4125b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.428+0000 7f245fa00740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24581a66f0 con 0x558f41250410 2024-03-21T10:50:21.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.428+0000 7f245fa00740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24581a41c0 con 0x558f4123e450 2024-03-21T10:50:21.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.428+0000 7f245fa00740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24581a2280 con 0x558f4123ee00 2024-03-21T10:50:21.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.429+0000 7f244fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f4123ee00 0x558f4124da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.429+0000 7f244fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f4123ee00 0x558f4124da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34966/0 (socket says 172.21.15.31:34966) 2024-03-21T10:50:21.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.429+0000 7f244fe00640 1 -- 172.21.15.31:0/4162898207 learned_addr learned my addr 172.21.15.31:0/4162898207 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:21.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.429+0000 7f2455e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f4123e450 0x558f412488b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.429+0000 7f2456800640 1 --2- 172.21.15.31:0/4162898207 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f41250410 0x558f4125b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.431+0000 7f244fe00640 1 -- 172.21.15.31:0/4162898207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f4123e450 msgr2=0x558f412488b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:21.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.431+0000 7f244fe00640 1 --2- 172.21.15.31:0/4162898207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f4123e450 0x558f412488b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.431+0000 7f244fe00640 1 -- 172.21.15.31:0/4162898207 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f41250410 msgr2=0x558f4125b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:21.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.432+0000 7f244fe00640 1 --2- 172.21.15.31:0/4162898207 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f41250410 0x558f4125b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.432+0000 7f244fe00640 1 -- 172.21.15.31:0/4162898207 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f41262660 con 0x558f4123ee00 2024-03-21T10:50:21.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.432+0000 7f244fe00640 1 --2- 172.21.15.31:0/4162898207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f4123ee00 0x558f4124da50 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f245810bbb0 tx=0x7f2448010000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e5ac0365698261a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:21.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.433+0000 7f2446a00640 1 -- 172.21.15.31:0/4162898207 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2448014bc0 con 0x558f4123ee00 2024-03-21T10:50:21.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.433+0000 7f2446a00640 1 -- 172.21.15.31:0/4162898207 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2448025070 con 0x558f4123ee00 2024-03-21T10:50:21.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.433+0000 7f2446a00640 1 -- 172.21.15.31:0/4162898207 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24480123e0 con 0x558f4123ee00 2024-03-21T10:50:21.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.434+0000 7f245fa00740 1 -- 172.21.15.31:0/4162898207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f4123ee00 msgr2=0x558f4124da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:21.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.434+0000 7f245fa00740 1 --2- 172.21.15.31:0/4162898207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f4123ee00 0x558f4124da50 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f245810bbb0 tx=0x7f2448010000 comp rx=0 tx=0).stop 2024-03-21T10:50:21.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.436+0000 7f245fa00740 1 -- 172.21.15.31:0/4162898207 shutdown_connections 2024-03-21T10:50:21.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.436+0000 7f245fa00740 1 --2- 172.21.15.31:0/4162898207 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f41250410 0x558f4125b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.436+0000 7f245fa00740 1 --2- 172.21.15.31:0/4162898207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f4123ee00 0x558f4124da50 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.436+0000 7f245fa00740 1 --2- 172.21.15.31:0/4162898207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f4123e450 0x558f412488b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.436+0000 7f245fa00740 1 -- 172.21.15.31:0/4162898207 >> 172.21.15.31:0/4162898207 conn(0x558f41238da0 msgr2=0x558f4123a840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:21.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.436+0000 7f245fa00740 1 -- 172.21.15.31:0/4162898207 shutdown_connections 2024-03-21T10:50:21.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.437+0000 7f245fa00740 1 -- 172.21.15.31:0/4162898207 wait complete. 2024-03-21T10:50:21.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.438+0000 7f245fa00740 1 -- start start 2024-03-21T10:50:21.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.438+0000 7f245fa00740 1 Processor -- start 2024-03-21T10:50:21.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.440+0000 7f245fa00740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f4123e450 0x558f41255320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.440+0000 7f245fa00740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f4123ee00 0x558f41255860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.440+0000 7f245fa00740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f41250410 0x558f41255da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.440+0000 7f245fa00740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24581a2880 con 0x558f4123ee00 2024-03-21T10:50:21.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.440+0000 7f245fa00740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24581a3b60 con 0x558f4123e450 2024-03-21T10:50:21.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.440+0000 7f245fa00740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24581a2ea0 con 0x558f41250410 2024-03-21T10:50:21.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.440+0000 7f245fa00740 10 client.0 _ll_register_callbacks cb 0x7f24500097c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:50:21.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.441+0000 7f2455e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f4123e450 0x558f41255320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.441+0000 7f2455e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f4123e450 0x558f41255320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:52824/0 (socket says 172.21.15.31:52824) 2024-03-21T10:50:21.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.441+0000 7f2455e00640 1 -- 172.21.15.31:0/3807833950 learned_addr learned my addr 172.21.15.31:0/3807833950 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:21.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.441+0000 7f2456800640 1 --2- 172.21.15.31:0/3807833950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f41250410 0x558f41255da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.442+0000 7f244fe00640 1 --2- 172.21.15.31:0/3807833950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f4123ee00 0x558f41255860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.443+0000 7f2455e00640 1 -- 172.21.15.31:0/3807833950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f41250410 msgr2=0x558f41255da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:21.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.443+0000 7f2455e00640 1 --2- 172.21.15.31:0/3807833950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f41250410 0x558f41255da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.443+0000 7f2455e00640 1 -- 172.21.15.31:0/3807833950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f4123ee00 msgr2=0x558f41255860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:21.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.443+0000 7f2455e00640 1 --2- 172.21.15.31:0/3807833950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f4123ee00 0x558f41255860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.443+0000 7f2455e00640 1 -- 172.21.15.31:0/3807833950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558f4124b180 con 0x558f4123e450 2024-03-21T10:50:21.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.444+0000 7f2455e00640 1 --2- 172.21.15.31:0/3807833950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f4123e450 0x558f41255320 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f24400cade0 tx=0x7f244001a590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b262e92f2ded38c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:21.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.444+0000 7f243be00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:50:21.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.444+0000 7f243be00640 1 -- 172.21.15.31:0/3807833950 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24400e2060 con 0x558f4123e450 2024-03-21T10:50:21.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.445+0000 7f243be00640 1 -- 172.21.15.31:0/3807833950 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f24400ff8e0 con 0x558f4123e450 2024-03-21T10:50:21.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.445+0000 7f245fa00740 1 -- 172.21.15.31:0/3807833950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2448025070 con 0x558f4123e450 2024-03-21T10:50:21.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.447+0000 7f243be00640 1 -- 172.21.15.31:0/3807833950 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24400ec050 con 0x558f4123e450 2024-03-21T10:50:21.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.447+0000 7f245fa00740 10 client.28610 fetch_fsmap learned FSMap version 31 2024-03-21T10:50:21.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.447+0000 7f245fa00740 1 -- 172.21.15.31:0/3807833950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x558f412bc740 con 0x558f4123e450 2024-03-21T10:50:21.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.449+0000 7f243be00640 1 -- 172.21.15.31:0/3807833950 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2448025070 con 0x558f4123e450 2024-03-21T10:50:21.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.449+0000 7f243be00640 1 -- 172.21.15.31:0/3807833950 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 31) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f24400ce670 con 0x558f4123e450 2024-03-21T10:50:21.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.450+0000 7f245fa00740 10 client.28610 fetch_fsmap finished waiting for FSMap version 31 2024-03-21T10:50:21.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.450+0000 7f245fa00740 10 client.28610 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-21T10:50:21.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.450+0000 7f245fa00740 20 client.28610 populate_metadata read hostname 'smithi031' 2024-03-21T10:50:21.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.450+0000 7f245fa00740 1 --2- 172.21.15.31:0/3807833950 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x558f412bcdc0 0x558f412bf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.450+0000 7f245fa00740 4 client.28610 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:50:21.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.450+0000 7f245fa00740 1 -- 172.21.15.31:0/3807833950 --> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558f411a9b50 con 0x558f412bcdc0 2024-03-21T10:50:21.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.451+0000 7f244fe00640 1 --2- 172.21.15.31:0/3807833950 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x558f412bcdc0 0x558f412bf1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.453+0000 7f244fe00640 1 --2- 172.21.15.31:0/3807833950 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x558f412bcdc0 0x558f412bf1e0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1c08c3e316c668ac server_cookie=db9d84ce168180f4 in_seq=0 out_seq=0 2024-03-21T10:50:21.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.453+0000 7f243be00640 10 client.28610 ms_handle_connect on v2:172.21.15.31:6840/1752599123 2024-03-21T10:50:21.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.454+0000 7f243be00640 1 -- 172.21.15.31:0/3807833950 <== mds.1 v2:172.21.15.31:6840/1752599123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558f411a9b50 con 0x558f412bcdc0 2024-03-21T10:50:21.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.454+0000 7f243be00640 10 client.28610 handle_command_reply: tid=0 2024-03-21T10:50:21.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.454+0000 7f245fa00740 1 client.28610 shutdown 2024-03-21T10:50:21.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.454+0000 7f245fa00740 1 -- 172.21.15.31:0/3807833950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f4123e450 msgr2=0x558f41255320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:21.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.454+0000 7f245fa00740 1 --2- 172.21.15.31:0/3807833950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f4123e450 0x558f41255320 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f24400cade0 tx=0x7f244001a590 comp rx=0 tx=0).stop 2024-03-21T10:50:21.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.455+0000 7f245fa00740 1 -- 172.21.15.31:0/3807833950 shutdown_connections 2024-03-21T10:50:21.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.455+0000 7f245fa00740 1 --2- 172.21.15.31:0/3807833950 >> [v2:172.21.15.31:6840/1752599123,v1:172.21.15.31:6841/1752599123] conn(0x558f412bcdc0 0x558f412bf1e0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.455+0000 7f245fa00740 1 --2- 172.21.15.31:0/3807833950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f41250410 0x558f41255da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.455+0000 7f243be00640 0 client.28610 ms_handle_reset on v2:172.21.15.31:6840/1752599123 2024-03-21T10:50:21.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.455+0000 7f245fa00740 1 --2- 172.21.15.31:0/3807833950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f4123ee00 0x558f41255860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.456+0000 7f245fa00740 1 --2- 172.21.15.31:0/3807833950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558f4123e450 0x558f41255320 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.456+0000 7f245fa00740 1 -- 172.21.15.31:0/3807833950 >> 172.21.15.31:0/3807833950 conn(0x558f41238da0 msgr2=0x558f41266b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:21.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.456+0000 7f245fa00740 1 -- 172.21.15.31:0/3807833950 shutdown_connections 2024-03-21T10:50:21.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.456+0000 7f245fa00740 1 -- 172.21.15.31:0/3807833950 wait complete. 2024-03-21T10:50:21.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.457+0000 7f245fa00740 20 client.28610 trim_cache size 0 max 16384 2024-03-21T10:50:21.456 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:50:21.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.461+0000 7f245ce00640 1 -- 172.21.15.31:0/1153931125 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f243c0a3110 msgr2=0x7f243c0a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:21.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.461+0000 7f245ce00640 1 --2- 172.21.15.31:0/1153931125 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f243c0a3110 0x7f243c0a55d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2448006fd0 tx=0x7f2448008040 comp rx=0 tx=0).stop 2024-03-21T10:50:21.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.461+0000 7f245ce00640 1 -- 172.21.15.31:0/1153931125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2458101640 msgr2=0x7f245810e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:21.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.461+0000 7f245ce00640 1 --2- 172.21.15.31:0/1153931125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2458101640 0x7f245810e2d0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f244001a910 tx=0x7f2440004e50 comp rx=0 tx=0).stop 2024-03-21T10:50:21.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.463+0000 7f245ce00640 1 -- 172.21.15.31:0/1153931125 shutdown_connections 2024-03-21T10:50:21.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.463+0000 7f245ce00640 1 --2- 172.21.15.31:0/1153931125 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f243c0a3110 0x7f243c0a55d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.463+0000 7f245ce00640 1 --2- 172.21.15.31:0/1153931125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2458107650 0x7f245810a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.463+0000 7f245ce00640 1 --2- 172.21.15.31:0/1153931125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2458102000 0x7f245810a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.463+0000 7f245ce00640 1 --2- 172.21.15.31:0/1153931125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2458101640 0x7f245810e2d0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.463+0000 7f245ce00640 1 -- 172.21.15.31:0/1153931125 >> 172.21.15.31:0/1153931125 conn(0x7f24580fd130 msgr2=0x7f24580fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:21.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.463+0000 7f245ce00640 1 -- 172.21.15.31:0/1153931125 shutdown_connections 2024-03-21T10:50:21.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.463+0000 7f245ce00640 1 -- 172.21.15.31:0/1153931125 wait complete. 2024-03-21T10:50:21.491 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-21T10:50:21.491 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-21T10:50:21.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:21 smithi031 ceph-mon[31673]: pgmap v368: 129 pgs: 129 active+clean; 18 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 44 op/s 2024-03-21T10:50:21.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.813+0000 7f45f1800640 1 -- 172.21.15.31:0/2075512432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45ec105d20 msgr2=0x7f45ec110850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:21.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.813+0000 7f45f1800640 1 --2- 172.21.15.31:0/2075512432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45ec105d20 0x7f45ec110850 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f45dc009890 tx=0x7f45dc02bc40 comp rx=0 tx=0).stop 2024-03-21T10:50:21.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.815+0000 7f45f1800640 1 -- 172.21.15.31:0/2075512432 shutdown_connections 2024-03-21T10:50:21.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.815+0000 7f45f1800640 1 --2- 172.21.15.31:0/2075512432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45ec105d20 0x7f45ec110850 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.815+0000 7f45f1800640 1 --2- 172.21.15.31:0/2075512432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45ec105380 0x7f45ec1057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.815+0000 7f45f1800640 1 --2- 172.21.15.31:0/2075512432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45ec103c40 0x7f45ec104020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.815+0000 7f45f1800640 1 -- 172.21.15.31:0/2075512432 >> 172.21.15.31:0/2075512432 conn(0x7f45ec0ff350 msgr2=0x7f45ec101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:21.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.815+0000 7f45f1800640 1 -- 172.21.15.31:0/2075512432 shutdown_connections 2024-03-21T10:50:21.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.815+0000 7f45f1800640 1 -- 172.21.15.31:0/2075512432 wait complete. 2024-03-21T10:50:21.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.817+0000 7f45f1800640 1 Processor -- start 2024-03-21T10:50:21.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.817+0000 7f45f1800640 1 -- start start 2024-03-21T10:50:21.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.818+0000 7f45f1800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45ec103c40 0x7f45ec06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.818+0000 7f45f1800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45ec105380 0x7f45ec06d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.818+0000 7f45f1800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45ec105d20 0x7f45ec0728b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.818+0000 7f45f1800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec06b7e0 con 0x7f45ec103c40 2024-03-21T10:50:21.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.818+0000 7f45f1800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec06b660 con 0x7f45ec105380 2024-03-21T10:50:21.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.818+0000 7f45f1800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec06b960 con 0x7f45ec105d20 2024-03-21T10:50:21.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.818+0000 7f45ea000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45ec105380 0x7f45ec06d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.818+0000 7f45eb400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45ec105d20 0x7f45ec0728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.818+0000 7f45ea000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45ec105380 0x7f45ec06d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:52842/0 (socket says 172.21.15.31:52842) 2024-03-21T10:50:21.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.818+0000 7f45ea000640 1 -- 172.21.15.31:0/1171921032 learned_addr learned my addr 172.21.15.31:0/1171921032 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:21.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.818+0000 7f45eaa00640 1 --2- 172.21.15.31:0/1171921032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45ec103c40 0x7f45ec06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.819+0000 7f45eb400640 1 -- 172.21.15.31:0/1171921032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45ec105380 msgr2=0x7f45ec06d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:21.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.819+0000 7f45eb400640 1 --2- 172.21.15.31:0/1171921032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45ec105380 0x7f45ec06d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.819+0000 7f45eb400640 1 -- 172.21.15.31:0/1171921032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45ec103c40 msgr2=0x7f45ec06d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:21.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.819+0000 7f45eb400640 1 --2- 172.21.15.31:0/1171921032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45ec103c40 0x7f45ec06d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.819+0000 7f45eb400640 1 -- 172.21.15.31:0/1171921032 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ec073150 con 0x7f45ec105d20 2024-03-21T10:50:21.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.821+0000 7f45eb400640 1 --2- 172.21.15.31:0/1171921032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45ec105d20 0x7f45ec0728b0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f45dc02c180 tx=0x7f45dc0045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:21.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.821+0000 7f45e3400640 1 -- 172.21.15.31:0/1171921032 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45dc041020 con 0x7f45ec105d20 2024-03-21T10:50:21.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.822+0000 7f45e3400640 1 -- 172.21.15.31:0/1171921032 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45dc02ca60 con 0x7f45ec105d20 2024-03-21T10:50:21.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.822+0000 7f45f1800640 1 -- 172.21.15.31:0/1171921032 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45ec0733e0 con 0x7f45ec105d20 2024-03-21T10:50:21.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.822+0000 7f45e3400640 1 -- 172.21.15.31:0/1171921032 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45dc039040 con 0x7f45ec105d20 2024-03-21T10:50:21.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.822+0000 7f45f1800640 1 -- 172.21.15.31:0/1171921032 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45ec1af3c0 con 0x7f45ec105d20 2024-03-21T10:50:21.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:21 smithi070 ceph-mon[36330]: pgmap v368: 129 pgs: 129 active+clean; 18 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 44 op/s 2024-03-21T10:50:21.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.827+0000 7f45e3400640 1 -- 172.21.15.31:0/1171921032 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f45dc044020 con 0x7f45ec105d20 2024-03-21T10:50:21.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.829+0000 7f45e3400640 1 --2- 172.21.15.31:0/1171921032 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f45d00a31e0 0x7f45d00a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.829+0000 7f45e3400640 1 -- 172.21.15.31:0/1171921032 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f45dc0f0ed0 con 0x7f45ec105d20 2024-03-21T10:50:21.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.830+0000 7f45eaa00640 1 --2- 172.21.15.31:0/1171921032 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f45d00a31e0 0x7f45d00a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.831+0000 7f45eaa00640 1 --2- 172.21.15.31:0/1171921032 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f45d00a31e0 0x7f45d00a56a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f45e4005fd0 tx=0x7f45e4009500 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:21.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.835+0000 7f45f4592740 1 Processor -- start 2024-03-21T10:50:21.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.839+0000 7f45f4592740 1 -- start start 2024-03-21T10:50:21.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.840+0000 7f45f4592740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd5376d0 0x55fbfd537ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.840+0000 7f45f4592740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbfd538080 0x55fbfd540630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.840+0000 7f45f4592740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbfd540c00 0x55fbfd543050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.840+0000 7f45f4592740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbfd49b2c0 con 0x55fbfd538080 2024-03-21T10:50:21.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.840+0000 7f45f4592740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbfd443c90 con 0x55fbfd540c00 2024-03-21T10:50:21.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.840+0000 7f45f4592740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbfd497360 con 0x55fbfd5376d0 2024-03-21T10:50:21.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.841+0000 7f45eaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd5376d0 0x55fbfd537ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.841+0000 7f45eaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd5376d0 0x55fbfd537ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34982/0 (socket says 172.21.15.31:34982) 2024-03-21T10:50:21.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.841+0000 7f45eaa00640 1 -- 172.21.15.31:0/1450024924 learned_addr learned my addr 172.21.15.31:0/1450024924 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:21.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.841+0000 7f45eb400640 1 --2- 172.21.15.31:0/1450024924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbfd540c00 0x55fbfd543050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.841+0000 7f45ea000640 1 --2- 172.21.15.31:0/1450024924 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbfd538080 0x55fbfd540630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.841+0000 7f45eaa00640 1 -- 172.21.15.31:0/1450024924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbfd540c00 msgr2=0x55fbfd543050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:21.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.842+0000 7f45eaa00640 1 --2- 172.21.15.31:0/1450024924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbfd540c00 0x55fbfd543050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.842+0000 7f45eaa00640 1 -- 172.21.15.31:0/1450024924 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbfd538080 msgr2=0x55fbfd540630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:21.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.842+0000 7f45eaa00640 1 --2- 172.21.15.31:0/1450024924 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbfd538080 0x55fbfd540630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.842+0000 7f45eaa00640 1 -- 172.21.15.31:0/1450024924 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fbfd468080 con 0x55fbfd5376d0 2024-03-21T10:50:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.842+0000 7f45eaa00640 1 --2- 172.21.15.31:0/1450024924 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd5376d0 0x55fbfd537ab0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f45e4011540 tx=0x7f45e4012000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22a5d4e5dfe1a1ad server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.843+0000 7f45d7e00640 1 -- 172.21.15.31:0/1450024924 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e4044070 con 0x55fbfd5376d0 2024-03-21T10:50:21.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.843+0000 7f45d7e00640 1 -- 172.21.15.31:0/1450024924 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45e4040b40 con 0x55fbfd5376d0 2024-03-21T10:50:21.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.843+0000 7f45d7e00640 1 -- 172.21.15.31:0/1450024924 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e4010c70 con 0x55fbfd5376d0 2024-03-21T10:50:21.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.844+0000 7f45f4592740 1 -- 172.21.15.31:0/1450024924 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd5376d0 msgr2=0x55fbfd537ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:21.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.844+0000 7f45f4592740 1 --2- 172.21.15.31:0/1450024924 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd5376d0 0x55fbfd537ab0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f45e4011540 tx=0x7f45e4012000 comp rx=0 tx=0).stop 2024-03-21T10:50:21.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.844+0000 7f45f4592740 1 -- 172.21.15.31:0/1450024924 shutdown_connections 2024-03-21T10:50:21.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.844+0000 7f45f4592740 1 --2- 172.21.15.31:0/1450024924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbfd540c00 0x55fbfd543050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.844+0000 7f45f4592740 1 --2- 172.21.15.31:0/1450024924 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbfd538080 0x55fbfd540630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.844+0000 7f45f4592740 1 --2- 172.21.15.31:0/1450024924 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd5376d0 0x55fbfd537ab0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.844+0000 7f45f4592740 1 -- 172.21.15.31:0/1450024924 >> 172.21.15.31:0/1450024924 conn(0x55fbfd531da0 msgr2=0x55fbfd532180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:21.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.845+0000 7f45f4592740 1 -- 172.21.15.31:0/1450024924 shutdown_connections 2024-03-21T10:50:21.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.845+0000 7f45f4592740 1 -- 172.21.15.31:0/1450024924 wait complete. 2024-03-21T10:50:21.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.845+0000 7f45f4592740 1 -- start start 2024-03-21T10:50:21.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.846+0000 7f45f4592740 1 Processor -- start 2024-03-21T10:50:21.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.846+0000 7f45f4592740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbfd5376d0 0x55fbfd540560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.846+0000 7f45f4592740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd538080 0x55fbfd53d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.846+0000 7f45f4592740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbfd540c00 0x55fbfd53dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.847+0000 7f45ea000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd538080 0x55fbfd53d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.847+0000 7f45ea000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd538080 0x55fbfd53d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34986/0 (socket says 172.21.15.31:34986) 2024-03-21T10:50:21.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.847+0000 7f45eaa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbfd5376d0 0x55fbfd540560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.847+0000 7f45ea000640 1 -- 172.21.15.31:0/764969404 learned_addr learned my addr 172.21.15.31:0/764969404 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:21.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.847+0000 7f45f4592740 1 -- 172.21.15.31:0/764969404 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbfd543b70 con 0x55fbfd5376d0 2024-03-21T10:50:21.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.847+0000 7f45f4592740 1 -- 172.21.15.31:0/764969404 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbfd4570d0 con 0x55fbfd540c00 2024-03-21T10:50:21.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.847+0000 7f45f4592740 1 -- 172.21.15.31:0/764969404 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbfd543cf0 con 0x55fbfd538080 2024-03-21T10:50:21.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.848+0000 7f45eb400640 1 --2- 172.21.15.31:0/764969404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbfd540c00 0x55fbfd53dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.848+0000 7f45f4592740 10 client.0 _ll_register_callbacks cb 0x55fbfd45be80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:50:21.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.848+0000 7f45eb400640 1 -- 172.21.15.31:0/764969404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd538080 msgr2=0x55fbfd53d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:21.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.848+0000 7f45eb400640 1 --2- 172.21.15.31:0/764969404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd538080 0x55fbfd53d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.848+0000 7f45eb400640 1 -- 172.21.15.31:0/764969404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbfd5376d0 msgr2=0x55fbfd540560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:21.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.848+0000 7f45eb400640 1 --2- 172.21.15.31:0/764969404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbfd5376d0 0x55fbfd540560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.848+0000 7f45eb400640 1 -- 172.21.15.31:0/764969404 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fbfd1af3f0 con 0x55fbfd540c00 2024-03-21T10:50:21.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.849+0000 7f45eb400640 1 --2- 172.21.15.31:0/764969404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbfd540c00 0x55fbfd53dfe0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f45dc0ee390 tx=0x7f45dc0bd730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8e0ae76ca3bcc2b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:21.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.850+0000 7f45d4c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-21T10:50:21.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.850+0000 7f45d4c00640 1 -- 172.21.15.31:0/764969404 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45dc0c5a40 con 0x55fbfd540c00 2024-03-21T10:50:21.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.850+0000 7f45d4c00640 1 -- 172.21.15.31:0/764969404 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f45dc0346c0 con 0x55fbfd540c00 2024-03-21T10:50:21.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.850+0000 7f45f4592740 1 -- 172.21.15.31:0/764969404 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f45e4040b40 con 0x55fbfd540c00 2024-03-21T10:50:21.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.850+0000 7f45d4c00640 1 -- 172.21.15.31:0/764969404 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45dc0b96a0 con 0x55fbfd540c00 2024-03-21T10:50:21.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.851+0000 7f45f4592740 10 client.28622 fetch_fsmap learned FSMap version 31 2024-03-21T10:50:21.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.851+0000 7f45f4592740 1 -- 172.21.15.31:0/764969404 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55fbfd2797d0 con 0x55fbfd540c00 2024-03-21T10:50:21.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.851+0000 7f45d4c00640 1 -- 172.21.15.31:0/764969404 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45e4040b40 con 0x55fbfd540c00 2024-03-21T10:50:21.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.852+0000 7f45d4c00640 1 -- 172.21.15.31:0/764969404 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 31) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f45dc03c030 con 0x55fbfd540c00 2024-03-21T10:50:21.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.852+0000 7f45f4592740 10 client.28622 fetch_fsmap finished waiting for FSMap version 31 2024-03-21T10:50:21.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.852+0000 7f45f4592740 10 client.28622 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:50:21.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.852+0000 7f45f4592740 20 client.28622 populate_metadata read hostname 'smithi031' 2024-03-21T10:50:21.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.852+0000 7f45f4592740 1 --2- 172.21.15.31:0/764969404 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55fbfd550a40 0x55fbfd552e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.852+0000 7f45f4592740 4 client.28622 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T10:50:21.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.852+0000 7f45f4592740 1 -- 172.21.15.31:0/764969404 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fbfd4a8800 con 0x55fbfd550a40 2024-03-21T10:50:21.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.852+0000 7f45eaa00640 1 --2- 172.21.15.31:0/764969404 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55fbfd550a40 0x55fbfd552e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.853+0000 7f45eaa00640 1 --2- 172.21.15.31:0/764969404 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55fbfd550a40 0x55fbfd552e50 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3d7be9e82eb8eb1 server_cookie=410f5a8c29c719cd in_seq=0 out_seq=0 2024-03-21T10:50:21.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.853+0000 7f45d4c00640 10 client.28622 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:50:21.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.855+0000 7f45d4c00640 1 -- 172.21.15.31:0/764969404 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55fbfd4a8800 con 0x55fbfd550a40 2024-03-21T10:50:21.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.855+0000 7f45d4c00640 10 client.28622 handle_command_reply: tid=0 2024-03-21T10:50:21.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.855+0000 7f45f4592740 1 client.28622 shutdown 2024-03-21T10:50:21.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.856+0000 7f45f4592740 1 -- 172.21.15.31:0/764969404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbfd540c00 msgr2=0x55fbfd53dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:21.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.856+0000 7f45f4592740 1 --2- 172.21.15.31:0/764969404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbfd540c00 0x55fbfd53dfe0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f45dc0ee390 tx=0x7f45dc0bd730 comp rx=0 tx=0).stop 2024-03-21T10:50:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.857+0000 7f45f4592740 1 -- 172.21.15.31:0/764969404 shutdown_connections 2024-03-21T10:50:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.857+0000 7f45f4592740 1 --2- 172.21.15.31:0/764969404 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55fbfd550a40 0x55fbfd552e50 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.857+0000 7f45f4592740 1 --2- 172.21.15.31:0/764969404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbfd540c00 0x55fbfd53dfe0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.857+0000 7f45f4592740 1 --2- 172.21.15.31:0/764969404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd538080 0x55fbfd53d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.857+0000 7f45d4c00640 0 client.28622 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:50:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.857+0000 7f45f4592740 1 --2- 172.21.15.31:0/764969404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbfd5376d0 0x55fbfd540560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.857+0000 7f45f4592740 1 -- 172.21.15.31:0/764969404 >> 172.21.15.31:0/764969404 conn(0x55fbfd531da0 msgr2=0x55fbfd533af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:21.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.860+0000 7f45f4592740 1 -- 172.21.15.31:0/764969404 shutdown_connections 2024-03-21T10:50:21.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.860+0000 7f45f4592740 1 -- 172.21.15.31:0/764969404 wait complete. 2024-03-21T10:50:21.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.860+0000 7f4b5ca00640 1 -- 172.21.15.31:0/392737335 --> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b5009af10 con 0x7f4b3c0a3150 2024-03-21T10:50:21.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.861+0000 7f45f4592740 20 client.28622 trim_cache size 0 max 16384 2024-03-21T10:50:21.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.895+0000 7f45f4592740 1 Processor -- start 2024-03-21T10:50:21.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.896+0000 7f45f4592740 1 -- start start 2024-03-21T10:50:21.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.896+0000 7f45f4592740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd5376d0 0x55fbfd533b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.896+0000 7f45f4592740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbfd538080 0x55fbfd540090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.896+0000 7f45f4592740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbfd540c00 0x55fbfd53ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.896+0000 7f45f4592740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec11f950 con 0x55fbfd538080 2024-03-21T10:50:21.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.896+0000 7f45f4592740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec11d420 con 0x55fbfd540c00 2024-03-21T10:50:21.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.896+0000 7f45f4592740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec11b4e0 con 0x55fbfd5376d0 2024-03-21T10:50:21.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.897+0000 7f45eaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd5376d0 0x55fbfd533b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.897+0000 7f45eaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd5376d0 0x55fbfd533b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34994/0 (socket says 172.21.15.31:34994) 2024-03-21T10:50:21.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.897+0000 7f45eb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbfd540c00 0x55fbfd53ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.897+0000 7f45eaa00640 1 -- 172.21.15.31:0/431077137 learned_addr learned my addr 172.21.15.31:0/431077137 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:21.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.898+0000 7f45ea000640 1 --2- 172.21.15.31:0/431077137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbfd538080 0x55fbfd540090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.898+0000 7f45eb400640 1 -- 172.21.15.31:0/431077137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd5376d0 msgr2=0x55fbfd533b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:21.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.898+0000 7f45eb400640 1 --2- 172.21.15.31:0/431077137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd5376d0 0x55fbfd533b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.898+0000 7f45eb400640 1 -- 172.21.15.31:0/431077137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbfd538080 msgr2=0x55fbfd540090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:21.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.898+0000 7f45eb400640 1 --2- 172.21.15.31:0/431077137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbfd538080 0x55fbfd540090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.898+0000 7f45eb400640 1 -- 172.21.15.31:0/431077137 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fbfd52d510 con 0x55fbfd540c00 2024-03-21T10:50:21.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.898+0000 7f45eaa00640 1 --2- 172.21.15.31:0/431077137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd5376d0 0x55fbfd533b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:50:21.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.899+0000 7f45eb400640 1 --2- 172.21.15.31:0/431077137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbfd540c00 0x55fbfd53ac00 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f45dc0b9490 tx=0x7f45dc0bb540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e24910664c86a9bb server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:21.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.900+0000 7f45d7400640 1 -- 172.21.15.31:0/431077137 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45dc0c5a40 con 0x55fbfd540c00 2024-03-21T10:50:21.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.900+0000 7f45d7400640 1 -- 172.21.15.31:0/431077137 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45dc0035f0 con 0x55fbfd540c00 2024-03-21T10:50:21.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.900+0000 7f45d7400640 1 -- 172.21.15.31:0/431077137 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45dc0b9840 con 0x55fbfd540c00 2024-03-21T10:50:21.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.902+0000 7f45f4592740 1 -- 172.21.15.31:0/431077137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbfd540c00 msgr2=0x55fbfd53ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:21.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.902+0000 7f45f4592740 1 --2- 172.21.15.31:0/431077137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbfd540c00 0x55fbfd53ac00 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f45dc0b9490 tx=0x7f45dc0bb540 comp rx=0 tx=0).stop 2024-03-21T10:50:21.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.902+0000 7f45f4592740 1 -- 172.21.15.31:0/431077137 shutdown_connections 2024-03-21T10:50:21.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.902+0000 7f45f4592740 1 --2- 172.21.15.31:0/431077137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbfd540c00 0x55fbfd53ac00 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.902+0000 7f45f4592740 1 --2- 172.21.15.31:0/431077137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbfd538080 0x55fbfd540090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.902+0000 7f45f4592740 1 --2- 172.21.15.31:0/431077137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd5376d0 0x55fbfd533b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.902+0000 7f45f4592740 1 -- 172.21.15.31:0/431077137 >> 172.21.15.31:0/431077137 conn(0x55fbfd531da0 msgr2=0x55fbfd538cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:21.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.903+0000 7f45f4592740 1 -- 172.21.15.31:0/431077137 shutdown_connections 2024-03-21T10:50:21.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.903+0000 7f45f4592740 1 -- 172.21.15.31:0/431077137 wait complete. 2024-03-21T10:50:21.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.904+0000 7f45f4592740 1 -- start start 2024-03-21T10:50:21.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.904+0000 7f45f4592740 1 Processor -- start 2024-03-21T10:50:21.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.908+0000 7f4b4f400640 1 -- 172.21.15.31:0/392737335 <== mgr.25564 v2:172.21.15.31:6800/3298970029 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+350 (secure 0 0 0) 0x7f4b5009af10 con 0x7f4b3c0a3150 2024-03-21T10:50:21.904 INFO:teuthology.orchestra.run.smithi031.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-21T00:00:00", "created": "2024-03-21T10:38:34", "first": "2024-03-21T10:39:00", "last": "2024-03-21T10:50:00", "last_pruned": "2024-03-21T10:50:01", "created_count": 12, "pruned_count": 6, "active": true} 2024-03-21T10:50:21.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.908+0000 7f45f4592740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd5376d0 0x55fbfd546c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.909+0000 7f45f4592740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbfd538080 0x55fbfd547160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.909+0000 7f45eaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd5376d0 0x55fbfd546c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.909+0000 7f45eaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd5376d0 0x55fbfd546c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:34998/0 (socket says 172.21.15.31:34998) 2024-03-21T10:50:21.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.909+0000 7f45f4592740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbfd540c00 0x55fbfd5adb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.909+0000 7f45eaa00640 1 -- 172.21.15.31:0/367065716 learned_addr learned my addr 172.21.15.31:0/367065716 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:21.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.909+0000 7f45f4592740 1 -- 172.21.15.31:0/367065716 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec11bae0 con 0x55fbfd540c00 2024-03-21T10:50:21.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.909+0000 7f45f4592740 1 -- 172.21.15.31:0/367065716 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec11cdc0 con 0x55fbfd538080 2024-03-21T10:50:21.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.909+0000 7f45f4592740 1 -- 172.21.15.31:0/367065716 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec11c100 con 0x55fbfd5376d0 2024-03-21T10:50:21.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.910+0000 7f45f4592740 10 client.0 _ll_register_callbacks cb 0x7f45dc0bce00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T10:50:21.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.910+0000 7f45eaa00640 1 -- 172.21.15.31:0/367065716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbfd538080 msgr2=0x55fbfd547160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T10:50:21.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.910+0000 7f45eaa00640 1 --2- 172.21.15.31:0/367065716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbfd538080 0x55fbfd547160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.910+0000 7f45eaa00640 1 -- 172.21.15.31:0/367065716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbfd540c00 msgr2=0x55fbfd5adb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T10:50:21.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.910+0000 7f45eaa00640 1 --2- 172.21.15.31:0/367065716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbfd540c00 0x55fbfd5adb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.910+0000 7f45eaa00640 1 -- 172.21.15.31:0/367065716 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fbfd5ae100 con 0x55fbfd5376d0 2024-03-21T10:50:21.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.911+0000 7f45eaa00640 1 --2- 172.21.15.31:0/367065716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd5376d0 0x55fbfd546c20 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f45e4011d10 tx=0x7f45e400ffd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84b27140ab8b11cd server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:21.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.911+0000 7f4b5ca00640 1 -- 172.21.15.31:0/392737335 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4b3c0a3150 msgr2=0x7f4b3c0a5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:21.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.911+0000 7f4b5ca00640 1 --2- 172.21.15.31:0/392737335 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4b3c0a3150 0x7f4b3c0a5610 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4b580674f0 tx=0x7f4b5804faa0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.911+0000 7f4b5ca00640 1 -- 172.21.15.31:0/392737335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b500a6b90 msgr2=0x7f4b500a28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:21.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.911+0000 7f4b5ca00640 1 --2- 172.21.15.31:0/392737335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b500a6b90 0x7f4b500a28f0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f4b48037040 tx=0x7f4b480045e0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.911+0000 7f45d4c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-03-21T10:50:21.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.912+0000 7f45d4c00640 1 -- 172.21.15.31:0/367065716 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e4044070 con 0x55fbfd5376d0 2024-03-21T10:50:21.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.912+0000 7f45d4c00640 1 -- 172.21.15.31:0/367065716 <== mon.1 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f45e4056030 con 0x55fbfd5376d0 2024-03-21T10:50:21.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.912+0000 7f45f4592740 1 -- 172.21.15.31:0/367065716 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f45dc0035f0 con 0x55fbfd5376d0 2024-03-21T10:50:21.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.913+0000 7f4b5ca00640 1 -- 172.21.15.31:0/392737335 shutdown_connections 2024-03-21T10:50:21.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.913+0000 7f4b5ca00640 1 --2- 172.21.15.31:0/392737335 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f4b3c0a3150 0x7f4b3c0a5610 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.913+0000 7f4b5ca00640 1 --2- 172.21.15.31:0/392737335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4b500a6b90 0x7f4b500a28f0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.913+0000 7f4b5ca00640 1 --2- 172.21.15.31:0/392737335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b500099d0 0x7f4b500a23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.913+0000 7f4b5ca00640 1 --2- 172.21.15.31:0/392737335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b500090b0 0x7f4b500a6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.913+0000 7f4b5ca00640 1 -- 172.21.15.31:0/392737335 >> 172.21.15.31:0/392737335 conn(0x7f4b500939a0 msgr2=0x7f4b500a8320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:21.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.913+0000 7f45d4c00640 1 -- 172.21.15.31:0/367065716 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45e4056230 con 0x55fbfd5376d0 2024-03-21T10:50:21.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.913+0000 7f45f4592740 10 client.28564 fetch_fsmap learned FSMap version 31 2024-03-21T10:50:21.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.913+0000 7f45f4592740 1 -- 172.21.15.31:0/367065716 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55fbfd5b5450 con 0x55fbfd5376d0 2024-03-21T10:50:21.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.913+0000 7f4b5ca00640 1 -- 172.21.15.31:0/392737335 shutdown_connections 2024-03-21T10:50:21.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.913+0000 7f4b5ca00640 1 -- 172.21.15.31:0/392737335 wait complete. 2024-03-21T10:50:21.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.916+0000 7f45d4c00640 1 -- 172.21.15.31:0/367065716 <== mon.1 v2:172.21.15.70:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45dc0035f0 con 0x55fbfd5376d0 2024-03-21T10:50:21.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.918+0000 7f45d4c00640 1 -- 172.21.15.31:0/367065716 <== mon.1 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) v1 ==== 7930+0+0 (secure 0 0 0) 0x7f45e4040680 con 0x55fbfd5376d0 2024-03-21T10:50:21.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.918+0000 7f45f4592740 10 client.28564 fetch_fsmap finished waiting for FSMap version 31 2024-03-21T10:50:21.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.918+0000 7f45f4592740 10 client.28564 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-21T10:50:21.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.918+0000 7f45f4592740 20 client.28564 populate_metadata read hostname 'smithi031' 2024-03-21T10:50:21.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.918+0000 7f45f4592740 1 --2- 172.21.15.31:0/367065716 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55fbfd5b5920 0x55fbfd5b7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:21.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.918+0000 7f45f4592740 4 client.28564 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-21T10:50:21.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.918+0000 7f45f4592740 1 -- 172.21.15.31:0/367065716 --> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55fbfd4a2b50 con 0x55fbfd5b5920 2024-03-21T10:50:21.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.918+0000 7f45ea000640 1 --2- 172.21.15.31:0/367065716 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55fbfd5b5920 0x55fbfd5b7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:21.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.919+0000 7f45d4c00640 10 client.28564 ms_handle_connect on v2:172.21.15.40:6838/2539201148 2024-03-21T10:50:21.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.919+0000 7f45ea000640 1 --2- 172.21.15.31:0/367065716 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55fbfd5b5920 0x55fbfd5b7d80 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd81a62e1f86a485 server_cookie=63860a145be8d41c in_seq=0 out_seq=0 2024-03-21T10:50:21.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.919+0000 7f45d4c00640 1 -- 172.21.15.31:0/367065716 <== mds.0 v2:172.21.15.40:6838/2539201148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55fbfd4a2b50 con 0x55fbfd5b5920 2024-03-21T10:50:21.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.919+0000 7f45d4c00640 10 client.28564 handle_command_reply: tid=0 2024-03-21T10:50:21.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.919+0000 7f45f4592740 1 client.28564 shutdown 2024-03-21T10:50:21.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.920+0000 7f45f4592740 1 -- 172.21.15.31:0/367065716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd5376d0 msgr2=0x55fbfd546c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:21.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.920+0000 7f45f4592740 1 --2- 172.21.15.31:0/367065716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd5376d0 0x55fbfd546c20 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f45e4011d10 tx=0x7f45e400ffd0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.921+0000 7f45f4592740 1 -- 172.21.15.31:0/367065716 shutdown_connections 2024-03-21T10:50:21.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.921+0000 7f45f4592740 1 --2- 172.21.15.31:0/367065716 >> [v2:172.21.15.40:6838/2539201148,v1:172.21.15.40:6839/2539201148] conn(0x55fbfd5b5920 0x55fbfd5b7d80 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.921+0000 7f45f4592740 1 --2- 172.21.15.31:0/367065716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fbfd540c00 0x55fbfd5adb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.921+0000 7f45f4592740 1 --2- 172.21.15.31:0/367065716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbfd538080 0x55fbfd547160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.921+0000 7f45d4c00640 0 client.28564 ms_handle_reset on v2:172.21.15.40:6838/2539201148 2024-03-21T10:50:21.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.921+0000 7f45f4592740 1 --2- 172.21.15.31:0/367065716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd5376d0 0x55fbfd546c20 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.921+0000 7f45f4592740 1 -- 172.21.15.31:0/367065716 >> 172.21.15.31:0/367065716 conn(0x55fbfd531da0 msgr2=0x55fbfd557710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:21.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.921+0000 7f45f4592740 1 -- 172.21.15.31:0/367065716 shutdown_connections 2024-03-21T10:50:21.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.921+0000 7f45f4592740 1 -- 172.21.15.31:0/367065716 wait complete. 2024-03-21T10:50:21.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.921+0000 7f45f4592740 20 client.28564 trim_cache size 0 max 16384 2024-03-21T10:50:21.921 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-03-21T10:50:21.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.922+0000 7f45f1800640 1 -- 172.21.15.31:0/1171921032 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f45d00a31e0 msgr2=0x7f45d00a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:21.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.922+0000 7f45f1800640 1 --2- 172.21.15.31:0/1171921032 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f45d00a31e0 0x7f45d00a56a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f45e4005fd0 tx=0x7f45e4009500 comp rx=0 tx=0).stop 2024-03-21T10:50:21.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.922+0000 7f45f1800640 1 -- 172.21.15.31:0/1171921032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45ec105d20 msgr2=0x7f45ec0728b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:21.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.922+0000 7f45f1800640 1 --2- 172.21.15.31:0/1171921032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45ec105d20 0x7f45ec0728b0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f45dc02c180 tx=0x7f45dc0045e0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.923+0000 7f45f1800640 1 -- 172.21.15.31:0/1171921032 shutdown_connections 2024-03-21T10:50:21.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.923+0000 7f45f1800640 1 --2- 172.21.15.31:0/1171921032 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f45d00a31e0 0x7f45d00a56a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.923+0000 7f45f1800640 1 --2- 172.21.15.31:0/1171921032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45ec105d20 0x7f45ec0728b0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.923+0000 7f45f1800640 1 --2- 172.21.15.31:0/1171921032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45ec105380 0x7f45ec06d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.923+0000 7f45f1800640 1 --2- 172.21.15.31:0/1171921032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45ec103c40 0x7f45ec06d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:21.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.923+0000 7f45f1800640 1 -- 172.21.15.31:0/1171921032 >> 172.21.15.31:0/1171921032 conn(0x7f45ec0ff350 msgr2=0x7f45ec107690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:21.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.923+0000 7f45f1800640 1 -- 172.21.15.31:0/1171921032 shutdown_connections 2024-03-21T10:50:21.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:21.923+0000 7f45f1800640 1 -- 172.21.15.31:0/1171921032 wait complete. 2024-03-21T10:50:21.929 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:21 smithi040 ceph-mon[36154]: pgmap v368: 129 pgs: 129 active+clean; 18 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 44 op/s 2024-03-21T10:50:21.947 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-21T10:50:22.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.283+0000 7f1eeb400640 1 -- 172.21.15.31:0/1461234117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ee410cfc0 msgr2=0x7f1ee4114b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:22.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.283+0000 7f1eeb400640 1 --2- 172.21.15.31:0/1461234117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ee410cfc0 0x7f1ee4114b90 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1edc009ca0 tx=0x7f1edc02c0f0 comp rx=0 tx=0).stop 2024-03-21T10:50:22.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.283+0000 7f1eeb400640 1 -- 172.21.15.31:0/1461234117 shutdown_connections 2024-03-21T10:50:22.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.283+0000 7f1eeb400640 1 --2- 172.21.15.31:0/1461234117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ee410cfc0 0x7f1ee4114b90 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:22.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.283+0000 7f1eeb400640 1 --2- 172.21.15.31:0/1461234117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ee41088f0 0x7f1ee410c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:22.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.283+0000 7f1eeb400640 1 --2- 172.21.15.31:0/1461234117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ee4107f40 0x7f1ee4108320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:22.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.283+0000 7f1eeb400640 1 -- 172.21.15.31:0/1461234117 >> 172.21.15.31:0/1461234117 conn(0x7f1ee40774e0 msgr2=0x7f1ee41064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:22.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.283+0000 7f1eeb400640 1 -- 172.21.15.31:0/1461234117 shutdown_connections 2024-03-21T10:50:22.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.283+0000 7f1eeb400640 1 -- 172.21.15.31:0/1461234117 wait complete. 2024-03-21T10:50:22.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.285+0000 7f1eeb400640 1 Processor -- start 2024-03-21T10:50:22.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.285+0000 7f1eeb400640 1 -- start start 2024-03-21T10:50:22.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.285+0000 7f1eeb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ee4107f40 0x7f1ee419e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:22.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.286+0000 7f1eeb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ee410cfc0 0x7f1ee419eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:22.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.286+0000 7f1eeb400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ee41a2eb0 0x7f1ee41a3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:22.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.286+0000 7f1eeb400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee4118150 con 0x7f1ee4107f40 2024-03-21T10:50:22.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.286+0000 7f1eeb400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee4117fd0 con 0x7f1ee410cfc0 2024-03-21T10:50:22.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.286+0000 7f1eeb400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee41182d0 con 0x7f1ee41a2eb0 2024-03-21T10:50:22.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.286+0000 7f1ee9400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ee41a2eb0 0x7f1ee41a3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:22.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.286+0000 7f1ee9400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ee41a2eb0 0x7f1ee41a3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:35012/0 (socket says 172.21.15.31:35012) 2024-03-21T10:50:22.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.286+0000 7f1ee3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ee410cfc0 0x7f1ee419eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:22.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.286+0000 7f1ee9400640 1 -- 172.21.15.31:0/299980455 learned_addr learned my addr 172.21.15.31:0/299980455 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:22.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.286+0000 7f1ee8a00640 1 --2- 172.21.15.31:0/299980455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ee4107f40 0x7f1ee419e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:22.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.287+0000 7f1ee9400640 1 -- 172.21.15.31:0/299980455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ee410cfc0 msgr2=0x7f1ee419eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:22.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.287+0000 7f1ee9400640 1 --2- 172.21.15.31:0/299980455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ee410cfc0 0x7f1ee419eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:22.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.287+0000 7f1ee9400640 1 -- 172.21.15.31:0/299980455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ee4107f40 msgr2=0x7f1ee419e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:22.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.287+0000 7f1ee9400640 1 --2- 172.21.15.31:0/299980455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ee4107f40 0x7f1ee419e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:22.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.287+0000 7f1ee9400640 1 -- 172.21.15.31:0/299980455 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee41a39a0 con 0x7f1ee41a2eb0 2024-03-21T10:50:22.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.289+0000 7f1ee9400640 1 --2- 172.21.15.31:0/299980455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ee41a2eb0 0x7f1ee41a3360 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f1edc037040 tx=0x7f1edc032ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:22.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.289+0000 7f1ee1600640 1 -- 172.21.15.31:0/299980455 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1edc039040 con 0x7f1ee41a2eb0 2024-03-21T10:50:22.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.289+0000 7f1eeb400640 1 -- 172.21.15.31:0/299980455 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee41a3c30 con 0x7f1ee41a2eb0 2024-03-21T10:50:22.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.290+0000 7f1ee1600640 1 -- 172.21.15.31:0/299980455 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1edc044070 con 0x7f1ee41a2eb0 2024-03-21T10:50:22.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.290+0000 7f1eeb400640 1 -- 172.21.15.31:0/299980455 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee41ab450 con 0x7f1ee41a2eb0 2024-03-21T10:50:22.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.290+0000 7f1ee1600640 1 -- 172.21.15.31:0/299980455 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1edc0042a0 con 0x7f1ee41a2eb0 2024-03-21T10:50:22.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.291+0000 7f1eeb400640 1 -- 172.21.15.31:0/299980455 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ec40052c0 con 0x7f1ee41a2eb0 2024-03-21T10:50:22.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.295+0000 7f1ee1600640 1 -- 172.21.15.31:0/299980455 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f1edc032590 con 0x7f1ee41a2eb0 2024-03-21T10:50:22.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.297+0000 7f1ee1600640 1 --2- 172.21.15.31:0/299980455 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f1ecc0a3100 0x7f1ecc0a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:22.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.297+0000 7f1ee1600640 1 -- 172.21.15.31:0/299980455 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f1edc0f1b20 con 0x7f1ee41a2eb0 2024-03-21T10:50:22.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.298+0000 7f1ee8a00640 1 --2- 172.21.15.31:0/299980455 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f1ecc0a3100 0x7f1ecc0a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:22.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.299+0000 7f1ee8a00640 1 --2- 172.21.15.31:0/299980455 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f1ecc0a3100 0x7f1ecc0a55c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1ed80099e0 tx=0x7f1ed8009110 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:22.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.304+0000 7f1ee1600640 1 -- 172.21.15.31:0/299980455 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1edc04e020 con 0x7f1ee41a2eb0 2024-03-21T10:50:22.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.754+0000 7f1eeb400640 1 -- 172.21.15.31:0/299980455 --> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f1ec4000bf0 con 0x7f1ecc0a3100 2024-03-21T10:50:22.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:22 smithi031 ceph-mon[31673]: from='client.28534 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:50:22.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:22 smithi031 ceph-mon[31673]: pgmap v369: 129 pgs: 129 active+clean; 18 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 48 op/s 2024-03-21T10:50:22.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.813+0000 7f1ee1600640 1 -- 172.21.15.31:0/299980455 <== mgr.25564 v2:172.21.15.31:6800/3298970029 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f1ec4000bf0 con 0x7f1ecc0a3100 2024-03-21T10:50:22.809 INFO:teuthology.orchestra.run.smithi031.stdout:/ 1m 6m3h 2024-03-21T10:50:22.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.816+0000 7f1eeb400640 1 -- 172.21.15.31:0/299980455 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f1ecc0a3100 msgr2=0x7f1ecc0a55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:22.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.816+0000 7f1eeb400640 1 --2- 172.21.15.31:0/299980455 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f1ecc0a3100 0x7f1ecc0a55c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1ed80099e0 tx=0x7f1ed8009110 comp rx=0 tx=0).stop 2024-03-21T10:50:22.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.816+0000 7f1eeb400640 1 -- 172.21.15.31:0/299980455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ee41a2eb0 msgr2=0x7f1ee41a3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:22.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.816+0000 7f1eeb400640 1 --2- 172.21.15.31:0/299980455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ee41a2eb0 0x7f1ee41a3360 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f1edc037040 tx=0x7f1edc032ec0 comp rx=0 tx=0).stop 2024-03-21T10:50:22.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.817+0000 7f1eeb400640 1 -- 172.21.15.31:0/299980455 shutdown_connections 2024-03-21T10:50:22.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.817+0000 7f1eeb400640 1 --2- 172.21.15.31:0/299980455 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f1ecc0a3100 0x7f1ecc0a55c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:22.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.817+0000 7f1eeb400640 1 --2- 172.21.15.31:0/299980455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ee41a2eb0 0x7f1ee41a3360 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:22.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.817+0000 7f1eeb400640 1 --2- 172.21.15.31:0/299980455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ee410cfc0 0x7f1ee419eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:22.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.817+0000 7f1eeb400640 1 --2- 172.21.15.31:0/299980455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ee4107f40 0x7f1ee419e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:22.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.817+0000 7f1eeb400640 1 -- 172.21.15.31:0/299980455 >> 172.21.15.31:0/299980455 conn(0x7f1ee40774e0 msgr2=0x7f1ee40790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:22.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.817+0000 7f1eeb400640 1 -- 172.21.15.31:0/299980455 shutdown_connections 2024-03-21T10:50:22.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:22.817+0000 7f1eeb400640 1 -- 172.21.15.31:0/299980455 wait complete. 2024-03-21T10:50:22.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:22 smithi070 ceph-mon[36330]: from='client.28534 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:50:22.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:22 smithi070 ceph-mon[36330]: pgmap v369: 129 pgs: 129 active+clean; 18 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 48 op/s 2024-03-21T10:50:22.841 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-21T10:50:22.852 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-21T10:50:22.930 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:22 smithi040 ceph-mon[36154]: from='client.28534 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:50:22.930 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:22 smithi040 ceph-mon[36154]: pgmap v369: 129 pgs: 129 active+clean; 18 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 48 op/s 2024-03-21T10:50:22.949 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-21T10:50:22.950 INFO:tasks.fwd_scrub:done joining 2024-03-21T10:50:22.950 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-21T10:50:22.963 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T10:50:23.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.265+0000 7f321b800640 1 -- 172.21.15.31:0/2713385021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3214107eb0 msgr2=0x7f3214108290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:23.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.265+0000 7f321b800640 1 --2- 172.21.15.31:0/2713385021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3214107eb0 0x7f3214108290 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f3208009900 tx=0x7f320802bc60 comp rx=0 tx=0).stop 2024-03-21T10:50:23.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.266+0000 7f321b800640 1 -- 172.21.15.31:0/2713385021 shutdown_connections 2024-03-21T10:50:23.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.266+0000 7f321b800640 1 --2- 172.21.15.31:0/2713385021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f321410cf80 0x7f3214114b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:23.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.266+0000 7f321b800640 1 --2- 172.21.15.31:0/2713385021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3214108860 0x7f321410c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:23.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.266+0000 7f321b800640 1 --2- 172.21.15.31:0/2713385021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3214107eb0 0x7f3214108290 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:23.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.266+0000 7f321b800640 1 -- 172.21.15.31:0/2713385021 >> 172.21.15.31:0/2713385021 conn(0x7f32140773c0 msgr2=0x7f32140797e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:23.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.268+0000 7f321b800640 1 -- 172.21.15.31:0/2713385021 shutdown_connections 2024-03-21T10:50:23.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.268+0000 7f321b800640 1 -- 172.21.15.31:0/2713385021 wait complete. 2024-03-21T10:50:23.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.270+0000 7f321b800640 1 Processor -- start 2024-03-21T10:50:23.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.273+0000 7f321b800640 1 -- start start 2024-03-21T10:50:23.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.274+0000 7f321b800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3214107eb0 0x7f321419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:23.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.275+0000 7f321b800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f321410cf80 0x7f321419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:23.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.275+0000 7f321b800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32141a2f60 0x7f32141a3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:23.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.275+0000 7f321b800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3214118160 con 0x7f32141a2f60 2024-03-21T10:50:23.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.275+0000 7f321b800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3214117fe0 con 0x7f321410cf80 2024-03-21T10:50:23.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.275+0000 7f321b800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32141182e0 con 0x7f3214107eb0 2024-03-21T10:50:23.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.275+0000 7f3213e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f321410cf80 0x7f321419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:23.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.275+0000 7f3218e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3214107eb0 0x7f321419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:23.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.275+0000 7f3213e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f321410cf80 0x7f321419ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:52910/0 (socket says 172.21.15.31:52910) 2024-03-21T10:50:23.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.275+0000 7f3213e00640 1 -- 172.21.15.31:0/1594817453 learned_addr learned my addr 172.21.15.31:0/1594817453 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:23.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.275+0000 7f3219800640 1 --2- 172.21.15.31:0/1594817453 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32141a2f60 0x7f32141a3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:23.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.276+0000 7f3218e00640 1 -- 172.21.15.31:0/1594817453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f321410cf80 msgr2=0x7f321419ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:23.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.276+0000 7f3218e00640 1 --2- 172.21.15.31:0/1594817453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f321410cf80 0x7f321419ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:23.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.276+0000 7f3218e00640 1 -- 172.21.15.31:0/1594817453 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32141a2f60 msgr2=0x7f32141a3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:23.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.276+0000 7f3218e00640 1 --2- 172.21.15.31:0/1594817453 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32141a2f60 0x7f32141a3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:23.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.276+0000 7f3218e00640 1 -- 172.21.15.31:0/1594817453 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32141a3a50 con 0x7f3214107eb0 2024-03-21T10:50:23.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.276+0000 7f3218e00640 1 --2- 172.21.15.31:0/1594817453 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3214107eb0 0x7f321419e6a0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f320802c270 tx=0x7f320802ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:23.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.277+0000 7f3211600640 1 -- 172.21.15.31:0/1594817453 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3208041020 con 0x7f3214107eb0 2024-03-21T10:50:23.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.277+0000 7f3211600640 1 -- 172.21.15.31:0/1594817453 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f320802e2e0 con 0x7f3214107eb0 2024-03-21T10:50:23.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.277+0000 7f3211600640 1 -- 172.21.15.31:0/1594817453 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3208004fc0 con 0x7f3214107eb0 2024-03-21T10:50:23.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.279+0000 7f321b800640 1 -- 172.21.15.31:0/1594817453 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32141a3ce0 con 0x7f3214107eb0 2024-03-21T10:50:23.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.279+0000 7f321b800640 1 -- 172.21.15.31:0/1594817453 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32141ab3f0 con 0x7f3214107eb0 2024-03-21T10:50:23.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.281+0000 7f321b800640 1 -- 172.21.15.31:0/1594817453 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3214068970 con 0x7f3214107eb0 2024-03-21T10:50:23.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.282+0000 7f3211600640 1 -- 172.21.15.31:0/1594817453 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f3208044020 con 0x7f3214107eb0 2024-03-21T10:50:23.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.286+0000 7f3211600640 1 --2- 172.21.15.31:0/1594817453 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f31fc0a3150 0x7f31fc0a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:23.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.286+0000 7f3211600640 1 -- 172.21.15.31:0/1594817453 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f32080f0ce0 con 0x7f3214107eb0 2024-03-21T10:50:23.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.286+0000 7f3213e00640 1 --2- 172.21.15.31:0/1594817453 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f31fc0a3150 0x7f31fc0a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:23.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.288+0000 7f3213e00640 1 --2- 172.21.15.31:0/1594817453 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f31fc0a3150 0x7f31fc0a5610 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f321410e560 tx=0x7f3200009190 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:23.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.292+0000 7f3211600640 1 -- 172.21.15.31:0/1594817453 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f320804b020 con 0x7f3214107eb0 2024-03-21T10:50:23.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.638+0000 7f321b800640 1 -- 172.21.15.31:0/1594817453 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3214110210 con 0x7f3214107eb0 2024-03-21T10:50:23.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.639+0000 7f3211600640 1 -- 172.21.15.31:0/1594817453 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+25117 (secure 0 0 0) 0x7f32080bd570 con 0x7f3214107eb0 2024-03-21T10:50:23.636 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:50:23.637 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":106,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","created":"2024-03-21T10:25:41.971117+0000","modified":"2024-03-21T10:50:16.429735+0000","last_up_change":"2024-03-21T10:34:07.757196+0000","last_in_change":"2024-03-21T10:33:47.460406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T10:29:44.553968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T10:35:04.945449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T10:35:05.970589+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":105,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T10:35:09.688711+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":105,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e0244d33-00e2-47d5-9f66-34efc138c462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6803","nonce":3089885125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6805","nonce":3089885125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6809","nonce":3089885125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6807","nonce":3089885125}]},"public_addr":"172.21.15.31:6803/3089885125","cluster_addr":"172.21.15.31:6805/3089885125","heartbeat_back_addr":"172.21.15.31:6809/3089885125","heartbeat_front_addr":"172.21.15.31:6807/3089885125","state":["exists","up"]},{"osd":1,"uuid":"76a63504-beab-4354-a6c3-eebc3b16ad03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6801","nonce":618840037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6803","nonce":618840037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6807","nonce":618840037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6805","nonce":618840037}]},"public_addr":"172.21.15.40:6801/618840037","cluster_addr":"172.21.15.40:6803/618840037","heartbeat_back_addr":"172.21.15.40:6807/618840037","heartbeat_front_addr":"172.21.15.40:6805/618840037","state":["exists","up"]},{"osd":2,"uuid":"fa191bd6-9a21-4ee2-8adf-8a63c9cfd821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6801","nonce":2524184117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6803","nonce":2524184117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6807","nonce":2524184117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6805","nonce":2524184117}]},"public_addr":"172.21.15.70:6801/2524184117","cluster_addr":"172.21.15.70:6803/2524184117","heartbeat_back_addr":"172.21.15.70:6807/2524184117","heartbeat_front_addr":"172.21.15.70:6805/2524184117","state":["exists","up"]},{"osd":3,"uuid":"edb44507-ffb0-42a4-827b-08461e1bf952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6811","nonce":2823036296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6813","nonce":2823036296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6817","nonce":2823036296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6815","nonce":2823036296}]},"public_addr":"172.21.15.31:6811/2823036296","cluster_addr":"172.21.15.31:6813/2823036296","heartbeat_back_addr":"172.21.15.31:6817/2823036296","heartbeat_front_addr":"172.21.15.31:6815/2823036296","state":["exists","up"]},{"osd":4,"uuid":"8cb89153-8c3b-431f-b57c-8863fada76dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6809","nonce":4215186309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6811","nonce":4215186309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6815","nonce":4215186309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6813","nonce":4215186309}]},"public_addr":"172.21.15.40:6809/4215186309","cluster_addr":"172.21.15.40:6811/4215186309","heartbeat_back_addr":"172.21.15.40:6815/4215186309","heartbeat_front_addr":"172.21.15.40:6813/4215186309","state":["exists","up"]},{"osd":5,"uuid":"9409505a-d463-408c-a3cd-218380c04d68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6809","nonce":3402830452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6811","nonce":3402830452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6815","nonce":3402830452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6813","nonce":3402830452}]},"public_addr":"172.21.15.70:6809/3402830452","cluster_addr":"172.21.15.70:6811/3402830452","heartbeat_back_addr":"172.21.15.70:6815/3402830452","heartbeat_front_addr":"172.21.15.70:6813/3402830452","state":["exists","up"]},{"osd":6,"uuid":"f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6819","nonce":1193497058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6821","nonce":1193497058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6825","nonce":1193497058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6823","nonce":1193497058}]},"public_addr":"172.21.15.31:6819/1193497058","cluster_addr":"172.21.15.31:6821/1193497058","heartbeat_back_addr":"172.21.15.31:6825/1193497058","heartbeat_front_addr":"172.21.15.31:6823/1193497058","state":["exists","up"]},{"osd":7,"uuid":"db228a87-29a5-45d4-8fe6-a0a554262eb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6817","nonce":3825628119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6819","nonce":3825628119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6823","nonce":3825628119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6821","nonce":3825628119}]},"public_addr":"172.21.15.40:6817/3825628119","cluster_addr":"172.21.15.40:6819/3825628119","heartbeat_back_addr":"172.21.15.40:6823/3825628119","heartbeat_front_addr":"172.21.15.40:6821/3825628119","state":["exists","up"]},{"osd":8,"uuid":"59b9fde0-095d-4e0b-87da-ead37bd8ff1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6817","nonce":534880586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6819","nonce":534880586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6823","nonce":534880586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6821","nonce":534880586}]},"public_addr":"172.21.15.70:6817/534880586","cluster_addr":"172.21.15.70:6819/534880586","heartbeat_back_addr":"172.21.15.70:6823/534880586","heartbeat_front_addr":"172.21.15.70:6821/534880586","state":["exists","up"]},{"osd":9,"uuid":"77f47061-0c5f-4159-b276-e814935d6c1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6827","nonce":587047469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6829","nonce":587047469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6833","nonce":587047469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6831","nonce":587047469}]},"public_addr":"172.21.15.31:6827/587047469","cluster_addr":"172.21.15.31:6829/587047469","heartbeat_back_addr":"172.21.15.31:6833/587047469","heartbeat_front_addr":"172.21.15.31:6831/587047469","state":["exists","up"]},{"osd":10,"uuid":"8853dc43-7bd1-4c93-8063-f7982fd7da38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6825","nonce":512108711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6827","nonce":512108711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6831","nonce":512108711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6829","nonce":512108711}]},"public_addr":"172.21.15.40:6825/512108711","cluster_addr":"172.21.15.40:6827/512108711","heartbeat_back_addr":"172.21.15.40:6831/512108711","heartbeat_front_addr":"172.21.15.40:6829/512108711","state":["exists","up"]},{"osd":11,"uuid":"84b7c2de-c424-45a6-9583-60097e372a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6825","nonce":540305718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6827","nonce":540305718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6831","nonce":540305718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6829","nonce":540305718}]},"public_addr":"172.21.15.70:6825/540305718","cluster_addr":"172.21.15.70:6827/540305718","heartbeat_back_addr":"172.21.15.70:6831/540305718","heartbeat_front_addr":"172.21.15.70:6829/540305718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:28:51.477358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:17.405045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:41.626268+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:15.339568+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:42.676996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:07.516999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:42.412273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:10.365041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:35.699120+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:11.733639+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:39.505442+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:34:05.449301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:6800/3875791078":"2024-03-22T10:38:33.502014+0000","172.21.15.31:6801/3875791078":"2024-03-22T10:38:33.502014+0000","172.21.15.31:0/3488063077":"2024-03-22T10:38:33.502014+0000","172.21.15.31:0/1451930554":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/3633519331":"2024-03-22T10:38:33.502014+0000","172.21.15.31:0/3342588843":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/1729503305":"2024-03-22T10:38:33.502014+0000","172.21.15.31:6801/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/2171929076":"2024-03-22T10:38:33.502014+0000","172.21.15.31:0/1626448389":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/1910059351":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/344822770":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/3811368512":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/1981437651":"2024-03-22T10:26:16.959165+0000","172.21.15.31:6801/1981437651":"2024-03-22T10:26:16.959165+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]},{"pool":4,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T10:50:23.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.643+0000 7f321b800640 1 -- 172.21.15.31:0/1594817453 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f31fc0a3150 msgr2=0x7f31fc0a5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:23.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.643+0000 7f321b800640 1 --2- 172.21.15.31:0/1594817453 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f31fc0a3150 0x7f31fc0a5610 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f321410e560 tx=0x7f3200009190 comp rx=0 tx=0).stop 2024-03-21T10:50:23.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.643+0000 7f321b800640 1 -- 172.21.15.31:0/1594817453 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3214107eb0 msgr2=0x7f321419e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:23.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.643+0000 7f321b800640 1 --2- 172.21.15.31:0/1594817453 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3214107eb0 0x7f321419e6a0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f320802c270 tx=0x7f320802ec50 comp rx=0 tx=0).stop 2024-03-21T10:50:23.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.643+0000 7f321b800640 1 -- 172.21.15.31:0/1594817453 shutdown_connections 2024-03-21T10:50:23.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.643+0000 7f321b800640 1 --2- 172.21.15.31:0/1594817453 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f31fc0a3150 0x7f31fc0a5610 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:23.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.643+0000 7f321b800640 1 --2- 172.21.15.31:0/1594817453 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32141a2f60 0x7f32141a3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:23.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.643+0000 7f321b800640 1 --2- 172.21.15.31:0/1594817453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f321410cf80 0x7f321419ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:23.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.643+0000 7f321b800640 1 --2- 172.21.15.31:0/1594817453 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3214107eb0 0x7f321419e6a0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:23.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.643+0000 7f321b800640 1 -- 172.21.15.31:0/1594817453 >> 172.21.15.31:0/1594817453 conn(0x7f32140773c0 msgr2=0x7f3214079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:23.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.643+0000 7f321b800640 1 -- 172.21.15.31:0/1594817453 shutdown_connections 2024-03-21T10:50:23.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:23.643+0000 7f321b800640 1 -- 172.21.15.31:0/1594817453 wait complete. 2024-03-21T10:50:23.672 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T10:29:44.553968+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T10:35:04.945449+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T10:35:05.970589+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 19, 'snap_epoch': 105, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T10:35:09.688711+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 19, 'snap_epoch': 105, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T10:50:23.673 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T10:50:23.778 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:23 smithi031 ceph-mon[31673]: from='client.28567 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:50:23.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:23 smithi070 ceph-mon[36330]: from='client.28567 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:50:23.930 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:23 smithi040 ceph-mon[36154]: from='client.28567 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T10:50:24.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.004+0000 7f5891000640 1 -- 172.21.15.31:0/3326896100 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f588c108be0 msgr2=0x7f588c109040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:24.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.004+0000 7f5891000640 1 --2- 172.21.15.31:0/3326896100 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f588c108be0 0x7f588c109040 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f5878009900 tx=0x7f587802bc60 comp rx=0 tx=0).stop 2024-03-21T10:50:24.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.005+0000 7f5891000640 1 -- 172.21.15.31:0/3326896100 shutdown_connections 2024-03-21T10:50:24.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.005+0000 7f5891000640 1 --2- 172.21.15.31:0/3326896100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f588c11ae80 0x7f588c11d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:24.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.005+0000 7f5891000640 1 --2- 172.21.15.31:0/3326896100 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f588c108be0 0x7f588c109040 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:24.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.005+0000 7f5891000640 1 --2- 172.21.15.31:0/3326896100 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f588c10fe30 0x7f588c1086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:24.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.005+0000 7f5891000640 1 -- 172.21.15.31:0/3326896100 >> 172.21.15.31:0/3326896100 conn(0x7f588c06c710 msgr2=0x7f588c06cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:24.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.006+0000 7f5891000640 1 -- 172.21.15.31:0/3326896100 shutdown_connections 2024-03-21T10:50:24.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.006+0000 7f5891000640 1 -- 172.21.15.31:0/3326896100 wait complete. 2024-03-21T10:50:24.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.007+0000 7f5891000640 1 Processor -- start 2024-03-21T10:50:24.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.008+0000 7f5891000640 1 -- start start 2024-03-21T10:50:24.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.008+0000 7f5891000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f588c108be0 0x7f588c1a6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:24.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.008+0000 7f5891000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f588c10fe30 0x7f588c1a7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:24.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.008+0000 7f5891000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f588c11ae80 0x7f588c1ab710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:24.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.009+0000 7f5891000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c120970 con 0x7f588c10fe30 2024-03-21T10:50:24.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.009+0000 7f5891000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c1207f0 con 0x7f588c11ae80 2024-03-21T10:50:24.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.009+0000 7f5891000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c120af0 con 0x7f588c108be0 2024-03-21T10:50:24.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.009+0000 7f5889e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f588c108be0 0x7f588c1a6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:24.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.009+0000 7f5889e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f588c108be0 0x7f588c1a6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:35040/0 (socket says 172.21.15.31:35040) 2024-03-21T10:50:24.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.009+0000 7f5889e00640 1 -- 172.21.15.31:0/2610952826 learned_addr learned my addr 172.21.15.31:0/2610952826 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:24.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.010+0000 7f5889400640 1 --2- 172.21.15.31:0/2610952826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f588c10fe30 0x7f588c1a7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:24.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.010+0000 7f5889e00640 1 -- 172.21.15.31:0/2610952826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f588c11ae80 msgr2=0x7f588c1ab710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:24.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.010+0000 7f5889e00640 1 --2- 172.21.15.31:0/2610952826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f588c11ae80 0x7f588c1ab710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:24.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.010+0000 7f5889e00640 1 -- 172.21.15.31:0/2610952826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f588c10fe30 msgr2=0x7f588c1a7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:24.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.010+0000 7f5889e00640 1 --2- 172.21.15.31:0/2610952826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f588c10fe30 0x7f588c1a7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:24.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.010+0000 7f5889e00640 1 -- 172.21.15.31:0/2610952826 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588c1abe40 con 0x7f588c108be0 2024-03-21T10:50:24.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.010+0000 7f5889400640 1 --2- 172.21.15.31:0/2610952826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f588c10fe30 0x7f588c1a7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T10:50:24.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.010+0000 7f5889e00640 1 --2- 172.21.15.31:0/2610952826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f588c108be0 0x7f588c1a6e50 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f587c00ba50 tx=0x7f587c00bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:24.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.010+0000 7f5882a00640 1 -- 172.21.15.31:0/2610952826 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587c019070 con 0x7f588c108be0 2024-03-21T10:50:24.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.011+0000 7f5891000640 1 -- 172.21.15.31:0/2610952826 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f588c1ac130 con 0x7f588c108be0 2024-03-21T10:50:24.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.011+0000 7f5882a00640 1 -- 172.21.15.31:0/2610952826 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f587c00cc10 con 0x7f588c108be0 2024-03-21T10:50:24.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.011+0000 7f5882a00640 1 -- 172.21.15.31:0/2610952826 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587c0076a0 con 0x7f588c108be0 2024-03-21T10:50:24.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.011+0000 7f5891000640 1 -- 172.21.15.31:0/2610952826 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f588c1b39c0 con 0x7f588c108be0 2024-03-21T10:50:24.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.012+0000 7f5891000640 1 -- 172.21.15.31:0/2610952826 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f588c068520 con 0x7f588c108be0 2024-03-21T10:50:24.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.017+0000 7f5882a00640 1 -- 172.21.15.31:0/2610952826 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f587c01c070 con 0x7f588c108be0 2024-03-21T10:50:24.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.019+0000 7f5882a00640 1 --2- 172.21.15.31:0/2610952826 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f58700a31d0 0x7f58700a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:24.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.019+0000 7f5882a00640 1 -- 172.21.15.31:0/2610952826 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f587c0cf140 con 0x7f588c108be0 2024-03-21T10:50:24.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.022+0000 7f5889400640 1 --2- 172.21.15.31:0/2610952826 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f58700a31d0 0x7f58700a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:24.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.022+0000 7f5882a00640 1 -- 172.21.15.31:0/2610952826 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f587c09b940 con 0x7f588c108be0 2024-03-21T10:50:24.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.024+0000 7f5889400640 1 --2- 172.21.15.31:0/2610952826 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f58700a31d0 0x7f58700a5690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f587802efd0 tx=0x7f5878005fd0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:24.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.383+0000 7f5891000640 1 -- 172.21.15.31:0/2610952826 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f588c1b3d20 con 0x7f588c108be0 2024-03-21T10:50:24.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.385+0000 7f5882a00640 1 -- 172.21.15.31:0/2610952826 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) v1 ==== 93+0+10 (secure 0 0 0) 0x7f587c09b240 con 0x7f588c108be0 2024-03-21T10:50:24.383 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-03-21T10:50:24.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.390+0000 7f5891000640 1 -- 172.21.15.31:0/2610952826 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f58700a31d0 msgr2=0x7f58700a5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:24.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.390+0000 7f5891000640 1 --2- 172.21.15.31:0/2610952826 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f58700a31d0 0x7f58700a5690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f587802efd0 tx=0x7f5878005fd0 comp rx=0 tx=0).stop 2024-03-21T10:50:24.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.390+0000 7f5891000640 1 -- 172.21.15.31:0/2610952826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f588c108be0 msgr2=0x7f588c1a6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:24.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.390+0000 7f5891000640 1 --2- 172.21.15.31:0/2610952826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f588c108be0 0x7f588c1a6e50 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f587c00ba50 tx=0x7f587c00bf10 comp rx=0 tx=0).stop 2024-03-21T10:50:24.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.391+0000 7f5891000640 1 -- 172.21.15.31:0/2610952826 shutdown_connections 2024-03-21T10:50:24.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.391+0000 7f5891000640 1 --2- 172.21.15.31:0/2610952826 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f58700a31d0 0x7f58700a5690 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:24.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.391+0000 7f5891000640 1 --2- 172.21.15.31:0/2610952826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f588c11ae80 0x7f588c1ab710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:24.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.391+0000 7f5891000640 1 --2- 172.21.15.31:0/2610952826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f588c10fe30 0x7f588c1a7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:24.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.391+0000 7f5891000640 1 --2- 172.21.15.31:0/2610952826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f588c108be0 0x7f588c1a6e50 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:24.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.391+0000 7f5891000640 1 -- 172.21.15.31:0/2610952826 >> 172.21.15.31:0/2610952826 conn(0x7f588c06c710 msgr2=0x7f588c118e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:24.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.391+0000 7f5891000640 1 -- 172.21.15.31:0/2610952826 shutdown_connections 2024-03-21T10:50:24.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.391+0000 7f5891000640 1 -- 172.21.15.31:0/2610952826 wait complete. 2024-03-21T10:50:24.415 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T10:50:24.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.714+0000 7fc302800640 1 -- 172.21.15.31:0/1577040673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2fc101630 msgr2=0x7fc2fc101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:24.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.714+0000 7fc302800640 1 --2- 172.21.15.31:0/1577040673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2fc101630 0x7fc2fc101ab0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec009cd0 tx=0x7fc2ec02c0f0 comp rx=0 tx=0).stop 2024-03-21T10:50:24.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.715+0000 7fc302800640 1 -- 172.21.15.31:0/1577040673 shutdown_connections 2024-03-21T10:50:24.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.715+0000 7fc302800640 1 --2- 172.21.15.31:0/1577040673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2fc101ff0 0x7fc2fc10e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:24.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.715+0000 7fc302800640 1 --2- 172.21.15.31:0/1577040673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2fc101630 0x7fc2fc101ab0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:24.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.715+0000 7fc302800640 1 --2- 172.21.15.31:0/1577040673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2fc107640 0x7fc2fc107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:24.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.715+0000 7fc302800640 1 -- 172.21.15.31:0/1577040673 >> 172.21.15.31:0/1577040673 conn(0x7fc2fc0fd120 msgr2=0x7fc2fc0ff540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:24.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.715+0000 7fc302800640 1 -- 172.21.15.31:0/1577040673 shutdown_connections 2024-03-21T10:50:24.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.715+0000 7fc302800640 1 -- 172.21.15.31:0/1577040673 wait complete. 2024-03-21T10:50:24.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.717+0000 7fc302800640 1 Processor -- start 2024-03-21T10:50:24.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.719+0000 7fc302800640 1 -- start start 2024-03-21T10:50:24.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.719+0000 7fc302800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2fc101630 0x7fc2fc116fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:24.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.719+0000 7fc302800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2fc101ff0 0x7fc2fc110020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:24.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.719+0000 7fc302800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2fc107640 0x7fc2fc110560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:24.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.720+0000 7fc302800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc118080 con 0x7fc2fc107640 2024-03-21T10:50:24.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.720+0000 7fc2fbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2fc107640 0x7fc2fc110560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:24.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.720+0000 7fc2fbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2fc107640 0x7fc2fc110560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37380/0 (socket says 172.21.15.31:37380) 2024-03-21T10:50:24.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.720+0000 7fc2fbe00640 1 -- 172.21.15.31:0/602978659 learned_addr learned my addr 172.21.15.31:0/602978659 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:24.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.720+0000 7fc2fb400640 1 --2- 172.21.15.31:0/602978659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2fc101630 0x7fc2fc116fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:24.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.720+0000 7fc302800640 1 -- 172.21.15.31:0/602978659 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc117f00 con 0x7fc2fc101ff0 2024-03-21T10:50:24.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.720+0000 7fc302800640 1 -- 172.21.15.31:0/602978659 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc118200 con 0x7fc2fc101630 2024-03-21T10:50:24.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.721+0000 7fc2fbe00640 1 -- 172.21.15.31:0/602978659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2fc101630 msgr2=0x7fc2fc116fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:24.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.721+0000 7fc2fbe00640 1 --2- 172.21.15.31:0/602978659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2fc101630 0x7fc2fc116fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:24.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.721+0000 7fc2fbe00640 1 -- 172.21.15.31:0/602978659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2fc101ff0 msgr2=0x7fc2fc110020 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-21T10:50:24.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.721+0000 7fc2fbe00640 1 --2- 172.21.15.31:0/602978659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2fc101ff0 0x7fc2fc110020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:24.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.721+0000 7fc2fbe00640 1 -- 172.21.15.31:0/602978659 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2fc110c00 con 0x7fc2fc107640 2024-03-21T10:50:24.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.721+0000 7fc2fb400640 1 --2- 172.21.15.31:0/602978659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2fc101630 0x7fc2fc116fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:50:24.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.722+0000 7fc2fbe00640 1 --2- 172.21.15.31:0/602978659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2fc107640 0x7fc2fc110560 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc2e400e9b0 tx=0x7fc2e400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:24.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.723+0000 7fc2f3e00640 1 -- 172.21.15.31:0/602978659 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e400cd80 con 0x7fc2fc107640 2024-03-21T10:50:24.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.723+0000 7fc2f3e00640 1 -- 172.21.15.31:0/602978659 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2e40035f0 con 0x7fc2fc107640 2024-03-21T10:50:24.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.723+0000 7fc2f3e00640 1 -- 172.21.15.31:0/602978659 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e4010670 con 0x7fc2fc107640 2024-03-21T10:50:24.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.723+0000 7fc302800640 1 -- 172.21.15.31:0/602978659 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2fc114ac0 con 0x7fc2fc107640 2024-03-21T10:50:24.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.723+0000 7fc302800640 1 -- 172.21.15.31:0/602978659 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2fc114fc0 con 0x7fc2fc107640 2024-03-21T10:50:24.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.727+0000 7fc302800640 1 -- 172.21.15.31:0/602978659 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2fc102d70 con 0x7fc2fc107640 2024-03-21T10:50:24.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.732+0000 7fc2f3e00640 1 -- 172.21.15.31:0/602978659 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fc2e40037a0 con 0x7fc2fc107640 2024-03-21T10:50:24.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.735+0000 7fc2f3e00640 1 --2- 172.21.15.31:0/602978659 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc2e00a3220 0x7fc2e00a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:24.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.735+0000 7fc2f3e00640 1 -- 172.21.15.31:0/602978659 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fc2e40d01c0 con 0x7fc2fc107640 2024-03-21T10:50:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.736+0000 7fc2fb400640 1 --2- 172.21.15.31:0/602978659 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc2e00a3220 0x7fc2e00a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:24.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.738+0000 7fc2fb400640 1 --2- 172.21.15.31:0/602978659 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc2e00a3220 0x7fc2e00a56e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4005d00 tx=0x7fc2f400b410 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:24.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:24.749+0000 7fc2f3e00640 1 -- 172.21.15.31:0/602978659 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc2e401d060 con 0x7fc2fc107640 2024-03-21T10:50:24.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:24 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1594817453' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:50:24.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:24 smithi031 ceph-mon[31673]: pgmap v370: 129 pgs: 129 active+clean; 18 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 79 op/s 2024-03-21T10:50:24.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:24 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2610952826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:50:24.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:24 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1594817453' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:50:24.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:24 smithi070 ceph-mon[36330]: pgmap v370: 129 pgs: 129 active+clean; 18 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 79 op/s 2024-03-21T10:50:24.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:24 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2610952826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:50:24.930 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:24 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1594817453' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:50:24.930 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:24 smithi040 ceph-mon[36154]: pgmap v370: 129 pgs: 129 active+clean; 18 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 79 op/s 2024-03-21T10:50:24.930 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:24 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2610952826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:50:25.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.110+0000 7fc302800640 1 -- 172.21.15.31:0/602978659 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc2fc109630 con 0x7fc2fc107640 2024-03-21T10:50:25.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.111+0000 7fc2f3e00640 1 -- 172.21.15.31:0/602978659 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v106) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc2e409c910 con 0x7fc2fc107640 2024-03-21T10:50:25.107 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-03-21T10:50:25.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.114+0000 7fc302800640 1 -- 172.21.15.31:0/602978659 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc2e00a3220 msgr2=0x7fc2e00a56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:25.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.114+0000 7fc302800640 1 --2- 172.21.15.31:0/602978659 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc2e00a3220 0x7fc2e00a56e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4005d00 tx=0x7fc2f400b410 comp rx=0 tx=0).stop 2024-03-21T10:50:25.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.114+0000 7fc302800640 1 -- 172.21.15.31:0/602978659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2fc107640 msgr2=0x7fc2fc110560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:25.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.114+0000 7fc302800640 1 --2- 172.21.15.31:0/602978659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2fc107640 0x7fc2fc110560 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc2e400e9b0 tx=0x7fc2e400ee70 comp rx=0 tx=0).stop 2024-03-21T10:50:25.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.114+0000 7fc302800640 1 -- 172.21.15.31:0/602978659 shutdown_connections 2024-03-21T10:50:25.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.114+0000 7fc302800640 1 --2- 172.21.15.31:0/602978659 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc2e00a3220 0x7fc2e00a56e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:25.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.114+0000 7fc302800640 1 --2- 172.21.15.31:0/602978659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2fc107640 0x7fc2fc110560 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:25.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.114+0000 7fc302800640 1 --2- 172.21.15.31:0/602978659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2fc101ff0 0x7fc2fc110020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:25.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.114+0000 7fc302800640 1 --2- 172.21.15.31:0/602978659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2fc101630 0x7fc2fc116fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:25.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.114+0000 7fc302800640 1 -- 172.21.15.31:0/602978659 >> 172.21.15.31:0/602978659 conn(0x7fc2fc0fd120 msgr2=0x7fc2fc0fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:25.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.115+0000 7fc302800640 1 -- 172.21.15.31:0/602978659 shutdown_connections 2024-03-21T10:50:25.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.115+0000 7fc302800640 1 -- 172.21.15.31:0/602978659 wait complete. 2024-03-21T10:50:25.138 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T10:50:25.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.406+0000 7f88dbc00640 1 -- 172.21.15.31:0/3112390078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d4073a40 msgr2=0x7f88d4073e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:25.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.406+0000 7f88dbc00640 1 --2- 172.21.15.31:0/3112390078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d4073a40 0x7f88d4073e20 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f88c8009900 tx=0x7f88c802bcc0 comp rx=0 tx=0).stop 2024-03-21T10:50:25.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.406+0000 7f88dbc00640 1 -- 172.21.15.31:0/3112390078 shutdown_connections 2024-03-21T10:50:25.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.406+0000 7f88dbc00640 1 --2- 172.21.15.31:0/3112390078 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88d4112640 0x7f88d4114a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:25.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.406+0000 7f88dbc00640 1 --2- 172.21.15.31:0/3112390078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88d4074360 0x7f88d4112100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:25.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.406+0000 7f88dbc00640 1 --2- 172.21.15.31:0/3112390078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d4073a40 0x7f88d4073e20 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:25.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.406+0000 7f88dbc00640 1 -- 172.21.15.31:0/3112390078 >> 172.21.15.31:0/3112390078 conn(0x7f88d40fd140 msgr2=0x7f88d40ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:25.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.407+0000 7f88dbc00640 1 -- 172.21.15.31:0/3112390078 shutdown_connections 2024-03-21T10:50:25.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.407+0000 7f88dbc00640 1 -- 172.21.15.31:0/3112390078 wait complete. 2024-03-21T10:50:25.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.407+0000 7f88dbc00640 1 Processor -- start 2024-03-21T10:50:25.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.408+0000 7f88dbc00640 1 -- start start 2024-03-21T10:50:25.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.408+0000 7f88dbc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88d4073a40 0x7f88d419e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:25.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.408+0000 7f88dbc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d4074360 0x7f88d419eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:25.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.408+0000 7f88dbc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88d4112640 0x7f88d41a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:25.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.408+0000 7f88dbc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d4118040 con 0x7f88d4112640 2024-03-21T10:50:25.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.408+0000 7f88dbc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d4117ec0 con 0x7f88d4074360 2024-03-21T10:50:25.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.408+0000 7f88dbc00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d41181c0 con 0x7f88d4073a40 2024-03-21T10:50:25.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.408+0000 7f88d9200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88d4073a40 0x7f88d419e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:25.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.408+0000 7f88d3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d4074360 0x7f88d419eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:25.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.408+0000 7f88d9200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88d4073a40 0x7f88d419e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:35074/0 (socket says 172.21.15.31:35074) 2024-03-21T10:50:25.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.408+0000 7f88d9200640 1 -- 172.21.15.31:0/226216118 learned_addr learned my addr 172.21.15.31:0/226216118 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:25.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.408+0000 7f88d9c00640 1 --2- 172.21.15.31:0/226216118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88d4112640 0x7f88d41a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:25.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.408+0000 7f88d9200640 1 -- 172.21.15.31:0/226216118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d4074360 msgr2=0x7f88d419eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:25.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.408+0000 7f88d9200640 1 --2- 172.21.15.31:0/226216118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d4074360 0x7f88d419eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:25.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.408+0000 7f88d9200640 1 -- 172.21.15.31:0/226216118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88d4112640 msgr2=0x7f88d41a2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:25.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.408+0000 7f88d9200640 1 --2- 172.21.15.31:0/226216118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88d4112640 0x7f88d41a2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:25.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.408+0000 7f88d9200640 1 -- 172.21.15.31:0/226216118 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d41a35b0 con 0x7f88d4073a40 2024-03-21T10:50:25.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.409+0000 7f88d9200640 1 --2- 172.21.15.31:0/226216118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88d4073a40 0x7f88d419e5c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f88c8036040 tx=0x7f88c802cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:25.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.409+0000 7f88d1600640 1 -- 172.21.15.31:0/226216118 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c8043020 con 0x7f88d4073a40 2024-03-21T10:50:25.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.409+0000 7f88d1600640 1 -- 172.21.15.31:0/226216118 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88c80054e0 con 0x7f88d4073a40 2024-03-21T10:50:25.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.409+0000 7f88d1600640 1 -- 172.21.15.31:0/226216118 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c8003bb0 con 0x7f88d4073a40 2024-03-21T10:50:25.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.409+0000 7f88dbc00640 1 -- 172.21.15.31:0/226216118 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d41a3840 con 0x7f88d4073a40 2024-03-21T10:50:25.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.409+0000 7f88dbc00640 1 -- 172.21.15.31:0/226216118 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d41ab130 con 0x7f88d4073a40 2024-03-21T10:50:25.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.411+0000 7f88dbc00640 1 -- 172.21.15.31:0/226216118 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88d4068970 con 0x7f88d4073a40 2024-03-21T10:50:25.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.412+0000 7f88d1600640 1 -- 172.21.15.31:0/226216118 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f88c8046020 con 0x7f88d4073a40 2024-03-21T10:50:25.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.413+0000 7f88d1600640 1 --2- 172.21.15.31:0/226216118 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f88bc0a3110 0x7f88bc0a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:25.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.414+0000 7f88d1600640 1 -- 172.21.15.31:0/226216118 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f88c80f1a10 con 0x7f88d4073a40 2024-03-21T10:50:25.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.414+0000 7f88d3e00640 1 --2- 172.21.15.31:0/226216118 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f88bc0a3110 0x7f88bc0a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:25.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.414+0000 7f88d3e00640 1 --2- 172.21.15.31:0/226216118 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f88bc0a3110 0x7f88bc0a55d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f88c00095f0 tx=0x7f88c0005f90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:25.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.417+0000 7f88d1600640 1 -- 172.21.15.31:0/226216118 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f88c804d020 con 0x7f88d4073a40 2024-03-21T10:50:25.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.694+0000 7f88dbc00640 1 -- 172.21.15.31:0/226216118 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f88d4104ab0 con 0x7f88d4073a40 2024-03-21T10:50:25.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.694+0000 7f88d1600640 1 -- 172.21.15.31:0/226216118 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v106) v1 ==== 100+0+11 (secure 0 0 0) 0x7f88c80be210 con 0x7f88d4073a40 2024-03-21T10:50:25.691 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-03-21T10:50:25.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.697+0000 7f88dbc00640 1 -- 172.21.15.31:0/226216118 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f88bc0a3110 msgr2=0x7f88bc0a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:25.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.697+0000 7f88dbc00640 1 --2- 172.21.15.31:0/226216118 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f88bc0a3110 0x7f88bc0a55d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f88c00095f0 tx=0x7f88c0005f90 comp rx=0 tx=0).stop 2024-03-21T10:50:25.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.697+0000 7f88dbc00640 1 -- 172.21.15.31:0/226216118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88d4073a40 msgr2=0x7f88d419e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:25.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.697+0000 7f88dbc00640 1 --2- 172.21.15.31:0/226216118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88d4073a40 0x7f88d419e5c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f88c8036040 tx=0x7f88c802cbc0 comp rx=0 tx=0).stop 2024-03-21T10:50:25.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.697+0000 7f88dbc00640 1 -- 172.21.15.31:0/226216118 shutdown_connections 2024-03-21T10:50:25.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.697+0000 7f88dbc00640 1 --2- 172.21.15.31:0/226216118 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f88bc0a3110 0x7f88bc0a55d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:25.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.697+0000 7f88dbc00640 1 --2- 172.21.15.31:0/226216118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88d4112640 0x7f88d41a2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:25.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.697+0000 7f88dbc00640 1 --2- 172.21.15.31:0/226216118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d4074360 0x7f88d419eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:25.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.697+0000 7f88dbc00640 1 --2- 172.21.15.31:0/226216118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88d4073a40 0x7f88d419e5c0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:25.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.697+0000 7f88dbc00640 1 -- 172.21.15.31:0/226216118 >> 172.21.15.31:0/226216118 conn(0x7f88d40fd140 msgr2=0x7f88d40fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:25.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.697+0000 7f88dbc00640 1 -- 172.21.15.31:0/226216118 shutdown_connections 2024-03-21T10:50:25.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.697+0000 7f88dbc00640 1 -- 172.21.15.31:0/226216118 wait complete. 2024-03-21T10:50:25.712 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T10:50:25.792 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:25 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/602978659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:50:25.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:25 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/602978659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:50:25.930 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:25 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/602978659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:50:25.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.934+0000 7f3d1f200640 1 -- 172.21.15.31:0/2905797860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d2010ba50 msgr2=0x7f3d2010ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:25.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.934+0000 7f3d1f200640 1 --2- 172.21.15.31:0/2905797860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d2010ba50 0x7f3d2010ded0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f3d18009ca0 tx=0x7f3d1802c180 comp rx=0 tx=0).stop 2024-03-21T10:50:25.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.934+0000 7f3d1f200640 1 -- 172.21.15.31:0/2905797860 shutdown_connections 2024-03-21T10:50:25.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.934+0000 7f3d1f200640 1 --2- 172.21.15.31:0/2905797860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d2010ba50 0x7f3d2010ded0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:25.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.934+0000 7f3d1f200640 1 --2- 172.21.15.31:0/2905797860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d20102f50 0x7f3d2010b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:25.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.934+0000 7f3d1f200640 1 --2- 172.21.15.31:0/2905797860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d20102630 0x7f3d20102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:25.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.934+0000 7f3d1f200640 1 -- 172.21.15.31:0/2905797860 >> 172.21.15.31:0/2905797860 conn(0x7f3d20077130 msgr2=0x7f3d200fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:25.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.934+0000 7f3d1f200640 1 -- 172.21.15.31:0/2905797860 shutdown_connections 2024-03-21T10:50:25.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.934+0000 7f3d1f200640 1 -- 172.21.15.31:0/2905797860 wait complete. 2024-03-21T10:50:25.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.935+0000 7f3d1f200640 1 Processor -- start 2024-03-21T10:50:25.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.935+0000 7f3d1f200640 1 -- start start 2024-03-21T10:50:25.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.935+0000 7f3d1f200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d20102630 0x7f3d20199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:25.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.935+0000 7f3d1f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d20102f50 0x7f3d2019a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:25.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.935+0000 7f3d1f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d2010ba50 0x7f3d2019e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:25.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.935+0000 7f3d1f200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d20111810 con 0x7f3d2010ba50 2024-03-21T10:50:25.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.935+0000 7f3d1f200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d20111690 con 0x7f3d20102f50 2024-03-21T10:50:25.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.935+0000 7f3d1f200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d20111990 con 0x7f3d20102630 2024-03-21T10:50:25.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.936+0000 7f3d1de00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d20102630 0x7f3d20199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:25.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.936+0000 7f3d1e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d2010ba50 0x7f3d2019e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:25.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.936+0000 7f3d1de00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d20102630 0x7f3d20199ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:35100/0 (socket says 172.21.15.31:35100) 2024-03-21T10:50:25.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.936+0000 7f3d1de00640 1 -- 172.21.15.31:0/3762587956 learned_addr learned my addr 172.21.15.31:0/3762587956 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:25.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.936+0000 7f3d1d400640 1 --2- 172.21.15.31:0/3762587956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d20102f50 0x7f3d2019a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:25.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.936+0000 7f3d1de00640 1 -- 172.21.15.31:0/3762587956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d20102f50 msgr2=0x7f3d2019a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:25.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.936+0000 7f3d1de00640 1 --2- 172.21.15.31:0/3762587956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d20102f50 0x7f3d2019a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:25.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.936+0000 7f3d1de00640 1 -- 172.21.15.31:0/3762587956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d2010ba50 msgr2=0x7f3d2019e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:25.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.936+0000 7f3d1de00640 1 --2- 172.21.15.31:0/3762587956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d2010ba50 0x7f3d2019e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:25.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.936+0000 7f3d1de00640 1 -- 172.21.15.31:0/3762587956 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d2019eec0 con 0x7f3d20102630 2024-03-21T10:50:25.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.936+0000 7f3d1e800640 1 --2- 172.21.15.31:0/3762587956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d2010ba50 0x7f3d2019e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:50:25.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.936+0000 7f3d1de00640 1 --2- 172.21.15.31:0/3762587956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d20102630 0x7f3d20199ed0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f3d1000e9f0 tx=0x7f3d1000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:25.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.937+0000 7f3d16a00640 1 -- 172.21.15.31:0/3762587956 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1000cd80 con 0x7f3d20102630 2024-03-21T10:50:25.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.937+0000 7f3d16a00640 1 -- 172.21.15.31:0/3762587956 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d10005500 con 0x7f3d20102630 2024-03-21T10:50:25.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.937+0000 7f3d1f200640 1 -- 172.21.15.31:0/3762587956 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d2019f1b0 con 0x7f3d20102630 2024-03-21T10:50:25.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.937+0000 7f3d16a00640 1 -- 172.21.15.31:0/3762587956 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d10010690 con 0x7f3d20102630 2024-03-21T10:50:25.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.937+0000 7f3d1f200640 1 -- 172.21.15.31:0/3762587956 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d201a6a40 con 0x7f3d20102630 2024-03-21T10:50:25.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.938+0000 7f3d1f200640 1 -- 172.21.15.31:0/3762587956 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d2004f400 con 0x7f3d20102630 2024-03-21T10:50:25.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.941+0000 7f3d16a00640 1 -- 172.21.15.31:0/3762587956 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f3d100046c0 con 0x7f3d20102630 2024-03-21T10:50:25.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.942+0000 7f3d16a00640 1 --2- 172.21.15.31:0/3762587956 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3d040a3110 0x7f3d040a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:25.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.943+0000 7f3d16a00640 1 -- 172.21.15.31:0/3762587956 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f3d100cff70 con 0x7f3d20102630 2024-03-21T10:50:25.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.943+0000 7f3d1d400640 1 --2- 172.21.15.31:0/3762587956 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3d040a3110 0x7f3d040a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:25.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.944+0000 7f3d1d400640 1 --2- 172.21.15.31:0/3762587956 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3d040a3110 0x7f3d040a55d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3d20102210 tx=0x7f3d0800b390 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:25.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:25.946+0000 7f3d16a00640 1 -- 172.21.15.31:0/3762587956 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3d1009c660 con 0x7f3d20102630 2024-03-21T10:50:26.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.154+0000 7f3d1f200640 1 -- 172.21.15.31:0/3762587956 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f3d201a6dd0 con 0x7f3d20102630 2024-03-21T10:50:26.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.155+0000 7f3d16a00640 1 -- 172.21.15.31:0/3762587956 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v106) v1 ==== 103+0+11 (secure 0 0 0) 0x7f3d1009bf60 con 0x7f3d20102630 2024-03-21T10:50:26.151 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-03-21T10:50:26.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.157+0000 7f3d1f200640 1 -- 172.21.15.31:0/3762587956 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3d040a3110 msgr2=0x7f3d040a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:26.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.157+0000 7f3d1f200640 1 --2- 172.21.15.31:0/3762587956 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3d040a3110 0x7f3d040a55d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3d20102210 tx=0x7f3d0800b390 comp rx=0 tx=0).stop 2024-03-21T10:50:26.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.157+0000 7f3d1f200640 1 -- 172.21.15.31:0/3762587956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d20102630 msgr2=0x7f3d20199ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:26.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.157+0000 7f3d1f200640 1 --2- 172.21.15.31:0/3762587956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d20102630 0x7f3d20199ed0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f3d1000e9f0 tx=0x7f3d1000edf0 comp rx=0 tx=0).stop 2024-03-21T10:50:26.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.157+0000 7f3d1f200640 1 -- 172.21.15.31:0/3762587956 shutdown_connections 2024-03-21T10:50:26.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.157+0000 7f3d1f200640 1 --2- 172.21.15.31:0/3762587956 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f3d040a3110 0x7f3d040a55d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:26.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.157+0000 7f3d1f200640 1 --2- 172.21.15.31:0/3762587956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d2010ba50 0x7f3d2019e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:26.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.157+0000 7f3d1f200640 1 --2- 172.21.15.31:0/3762587956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d20102f50 0x7f3d2019a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:26.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.157+0000 7f3d1f200640 1 --2- 172.21.15.31:0/3762587956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d20102630 0x7f3d20199ed0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:26.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.157+0000 7f3d1f200640 1 -- 172.21.15.31:0/3762587956 >> 172.21.15.31:0/3762587956 conn(0x7f3d20077130 msgr2=0x7f3d200fc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:26.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.157+0000 7f3d1f200640 1 -- 172.21.15.31:0/3762587956 shutdown_connections 2024-03-21T10:50:26.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.157+0000 7f3d1f200640 1 -- 172.21.15.31:0/3762587956 wait complete. 2024-03-21T10:50:26.171 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-21T10:50:26.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.387+0000 7fa305e00640 1 -- 172.21.15.31:0/1370393489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3000ffac0 msgr2=0x7fa3000fff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:26.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.387+0000 7fa305e00640 1 --2- 172.21.15.31:0/1370393489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3000ffac0 0x7fa3000fff40 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec009930 tx=0x7fa2ec02bc30 comp rx=0 tx=0).stop 2024-03-21T10:50:26.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.388+0000 7fa305e00640 1 -- 172.21.15.31:0/1370393489 shutdown_connections 2024-03-21T10:50:26.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.388+0000 7fa305e00640 1 --2- 172.21.15.31:0/1370393489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa300100480 0x7fa30010e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:26.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.388+0000 7fa305e00640 1 --2- 172.21.15.31:0/1370393489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3000ffac0 0x7fa3000fff40 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:26.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.388+0000 7fa305e00640 1 --2- 172.21.15.31:0/1370393489 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa300106610 0x7fa3001069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:26.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.388+0000 7fa305e00640 1 -- 172.21.15.31:0/1370393489 >> 172.21.15.31:0/1370393489 conn(0x7fa300077b30 msgr2=0x7fa3000fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:26.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.388+0000 7fa305e00640 1 -- 172.21.15.31:0/1370393489 shutdown_connections 2024-03-21T10:50:26.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.388+0000 7fa305e00640 1 -- 172.21.15.31:0/1370393489 wait complete. 2024-03-21T10:50:26.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.389+0000 7fa305e00640 1 Processor -- start 2024-03-21T10:50:26.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.389+0000 7fa305e00640 1 -- start start 2024-03-21T10:50:26.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.389+0000 7fa305e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3000ffac0 0x7fa30019a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:26.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.389+0000 7fa305e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa300100480 0x7fa30019a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:26.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.389+0000 7fa305e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa300106610 0x7fa30019e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:26.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.389+0000 7fa305e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa300111a60 con 0x7fa300100480 2024-03-21T10:50:26.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.389+0000 7fa305e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3001118e0 con 0x7fa3000ffac0 2024-03-21T10:50:26.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.389+0000 7fa305e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa300111be0 con 0x7fa300106610 2024-03-21T10:50:26.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.389+0000 7fa2fea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3000ffac0 0x7fa30019a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:26.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.389+0000 7fa2fe000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa300100480 0x7fa30019a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:26.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.390+0000 7fa2ff400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa300106610 0x7fa30019e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:26.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.390+0000 7fa2fe000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa300100480 0x7fa30019a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37418/0 (socket says 172.21.15.31:37418) 2024-03-21T10:50:26.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.390+0000 7fa2fe000640 1 -- 172.21.15.31:0/2745794237 learned_addr learned my addr 172.21.15.31:0/2745794237 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:50:26.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.390+0000 7fa2fe000640 1 -- 172.21.15.31:0/2745794237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa300106610 msgr2=0x7fa30019e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:26.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.390+0000 7fa2fe000640 1 --2- 172.21.15.31:0/2745794237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa300106610 0x7fa30019e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:26.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.390+0000 7fa2fe000640 1 -- 172.21.15.31:0/2745794237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3000ffac0 msgr2=0x7fa30019a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:26.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.390+0000 7fa2fe000640 1 --2- 172.21.15.31:0/2745794237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3000ffac0 0x7fa30019a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:26.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.390+0000 7fa2fe000640 1 -- 172.21.15.31:0/2745794237 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa30019f110 con 0x7fa300100480 2024-03-21T10:50:26.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.390+0000 7fa2fea00640 1 --2- 172.21.15.31:0/2745794237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3000ffac0 0x7fa30019a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:50:26.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.390+0000 7fa2fe000640 1 --2- 172.21.15.31:0/2745794237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa300100480 0x7fa30019a660 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec009900 tx=0x7fa2ec004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:26.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.390+0000 7fa304a00640 1 -- 172.21.15.31:0/2745794237 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec043020 con 0x7fa300100480 2024-03-21T10:50:26.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.391+0000 7fa305e00640 1 -- 172.21.15.31:0/2745794237 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa30019f3a0 con 0x7fa300100480 2024-03-21T10:50:26.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.391+0000 7fa304a00640 1 -- 172.21.15.31:0/2745794237 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2ec0054e0 con 0x7fa300100480 2024-03-21T10:50:26.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.391+0000 7fa304a00640 1 -- 172.21.15.31:0/2745794237 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec003b80 con 0x7fa300100480 2024-03-21T10:50:26.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.391+0000 7fa305e00640 1 -- 172.21.15.31:0/2745794237 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3001a6c90 con 0x7fa300100480 2024-03-21T10:50:26.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.392+0000 7fa305e00640 1 -- 172.21.15.31:0/2745794237 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa300068970 con 0x7fa300100480 2024-03-21T10:50:26.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.394+0000 7fa304a00640 1 -- 172.21.15.31:0/2745794237 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fa2ec046020 con 0x7fa300100480 2024-03-21T10:50:26.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.396+0000 7fa304a00640 1 --2- 172.21.15.31:0/2745794237 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa2e40a3090 0x7fa2e40a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:50:26.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.396+0000 7fa304a00640 1 -- 172.21.15.31:0/2745794237 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9446+0+0 (secure 0 0 0) 0x7fa2ec0f1bd0 con 0x7fa300100480 2024-03-21T10:50:26.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.396+0000 7fa2fea00640 1 --2- 172.21.15.31:0/2745794237 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa2e40a3090 0x7fa2e40a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:50:26.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.397+0000 7fa2fea00640 1 --2- 172.21.15.31:0/2745794237 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa2e40a3090 0x7fa2e40a5550 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0009f40 tx=0x7fa2f0005f90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:50:26.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.400+0000 7fa304a00640 1 -- 172.21.15.31:0/2745794237 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa2ec04d020 con 0x7fa300100480 2024-03-21T10:50:26.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:26 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/226216118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:50:26.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:26 smithi031 ceph-mon[31673]: pgmap v371: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 23 KiB/s wr, 242 op/s 2024-03-21T10:50:26.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:26 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3762587956' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:50:26.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.675+0000 7fa305e00640 1 -- 172.21.15.31:0/2745794237 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7fa3001a6f20 con 0x7fa300100480 2024-03-21T10:50:26.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.702+0000 7fa304a00640 1 -- 172.21.15.31:0/2745794237 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v27) v1 ==== 81+0+742373 (secure 0 0 0) 0x7fa2ec0be3d0 con 0x7fa300100480 2024-03-21T10:50:26.712 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:50:26.712 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:50:26.712 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 27, 2024-03-21T10:50:26.712 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.712 INFO:teuthology.orchestra.run.smithi031.stdout: "active_gid": 25564, 2024-03-21T10:50:26.712 INFO:teuthology.orchestra.run.smithi031.stdout: "active_name": "x", 2024-03-21T10:50:26.713 INFO:teuthology.orchestra.run.smithi031.stdout: "active_addrs": { 2024-03-21T10:50:26.713 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-03-21T10:50:26.713 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:26.713 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-03-21T10:50:26.713 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6800", 2024-03-21T10:50:26.713 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3298970029 2024-03-21T10:50:26.713 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.713 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:26.713 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-03-21T10:50:26.713 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6801", 2024-03-21T10:50:26.713 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3298970029 2024-03-21T10:50:26.713 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.713 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-03-21T10:50:26.713 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.714 INFO:teuthology.orchestra.run.smithi031.stdout: "active_addr": "172.21.15.31:6801/3298970029", 2024-03-21T10:50:26.714 INFO:teuthology.orchestra.run.smithi031.stdout: "active_change": "2024-03-21T10:38:33.502306+0000", 2024-03-21T10:50:26.714 INFO:teuthology.orchestra.run.smithi031.stdout: "active_mgr_features": 4540701547738038271, 2024-03-21T10:50:26.714 INFO:teuthology.orchestra.run.smithi031.stdout: "available": true, 2024-03-21T10:50:26.714 INFO:teuthology.orchestra.run.smithi031.stdout: "standbys": [ 2024-03-21T10:50:26.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:26.714 INFO:teuthology.orchestra.run.smithi031.stdout: "gid": 25555, 2024-03-21T10:50:26.714 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "y", 2024-03-21T10:50:26.714 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_features": 4540701547738038271, 2024-03-21T10:50:26.714 INFO:teuthology.orchestra.run.smithi031.stdout: "available_modules": [ 2024-03-21T10:50:26.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:26.714 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "alerts", 2024-03-21T10:50:26.715 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:26.715 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:26.715 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:26.715 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-03-21T10:50:26.715 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-03-21T10:50:26.715 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:26.715 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.715 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.715 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-03-21T10:50:26.715 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.715 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.715 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.715 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How frequently to reexamine health status", 2024-03-21T10:50:26.715 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.716 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.716 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.716 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.716 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:26.716 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:26.716 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.716 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.716 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.716 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.716 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.716 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.716 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.716 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.716 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.717 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.717 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.717 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.717 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.717 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.717 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.717 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.717 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.717 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.717 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.717 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:26.717 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:26.717 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.717 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.718 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.718 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.718 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.718 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.718 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.718 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.718 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.718 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.718 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.718 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.718 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:26.718 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:26.718 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.718 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.719 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.719 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:26.719 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.719 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.719 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.719 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.719 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.719 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.719 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.719 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.719 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.719 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.719 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.719 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.720 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.720 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.720 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.720 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:26.720 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:26.720 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.720 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.720 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.720 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.720 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.720 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.720 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.720 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.720 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.720 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.721 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.721 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_destination": { 2024-03-21T10:50:26.721 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_destination", 2024-03-21T10:50:26.721 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.721 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.721 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.721 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.721 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.721 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.721 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.721 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email address to send alerts to", 2024-03-21T10:50:26.721 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.721 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.721 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.722 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.722 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_from_name": { 2024-03-21T10:50:26.722 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_from_name", 2024-03-21T10:50:26.722 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.722 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.722 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.722 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "Ceph", 2024-03-21T10:50:26.722 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.722 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.722 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.722 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email From: name", 2024-03-21T10:50:26.722 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.722 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.722 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.722 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.723 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_host": { 2024-03-21T10:50:26.723 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_host", 2024-03-21T10:50:26.723 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.723 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.723 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.723 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.723 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.723 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.723 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.723 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP server", 2024-03-21T10:50:26.723 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.723 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.723 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.724 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_password": { 2024-03-21T10:50:26.724 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_password", 2024-03-21T10:50:26.724 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.724 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.724 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.724 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.724 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.724 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.724 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.724 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Password to authenticate with", 2024-03-21T10:50:26.724 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.724 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.724 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.724 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.724 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_port": { 2024-03-21T10:50:26.725 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_port", 2024-03-21T10:50:26.725 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.725 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.725 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.725 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "465", 2024-03-21T10:50:26.725 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.725 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.725 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.725 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP port", 2024-03-21T10:50:26.725 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.725 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.725 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.725 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.725 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_sender": { 2024-03-21T10:50:26.726 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_sender", 2024-03-21T10:50:26.726 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.726 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.726 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.726 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.726 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.726 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.726 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.726 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP envelope sender", 2024-03-21T10:50:26.726 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.726 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.726 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.727 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_ssl": { 2024-03-21T10:50:26.727 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_ssl", 2024-03-21T10:50:26.727 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.727 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.727 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.727 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.727 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.727 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.727 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.727 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-21T10:50:26.727 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.727 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.727 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.728 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_user": { 2024-03-21T10:50:26.728 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_user", 2024-03-21T10:50:26.728 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.728 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.728 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.728 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.728 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.728 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.728 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.728 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "User to authenticate as", 2024-03-21T10:50:26.728 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.728 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.728 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.728 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.729 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.729 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:26.729 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "balancer", 2024-03-21T10:50:26.729 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:26.729 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:26.729 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:26.729 INFO:teuthology.orchestra.run.smithi031.stdout: "active": { 2024-03-21T10:50:26.729 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "active", 2024-03-21T10:50:26.729 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.729 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.729 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.729 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.729 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.730 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.730 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.730 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically balance PGs across cluster", 2024-03-21T10:50:26.730 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.730 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.730 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.730 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_time": { 2024-03-21T10:50:26.730 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_time", 2024-03-21T10:50:26.730 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.731 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.731 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.731 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0000", 2024-03-21T10:50:26.731 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.731 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.731 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.731 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "beginning time of day to automatically balance", 2024-03-21T10:50:26.731 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-21T10:50:26.731 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.731 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.731 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_weekday": { 2024-03-21T10:50:26.732 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_weekday", 2024-03-21T10:50:26.732 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-03-21T10:50:26.732 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.732 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.732 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-03-21T10:50:26.732 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-03-21T10:50:26.732 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-03-21T10:50:26.732 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.732 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-21T10:50:26.732 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-21T10:50:26.732 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.733 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.733 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.733 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_max_iterations": { 2024-03-21T10:50:26.733 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_max_iterations", 2024-03-21T10:50:26.733 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-03-21T10:50:26.733 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.733 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.733 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "25", 2024-03-21T10:50:26.733 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-03-21T10:50:26.733 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "250", 2024-03-21T10:50:26.733 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.733 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-21T10:50:26.733 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.733 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.733 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.734 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_metrics": { 2024-03-21T10:50:26.734 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_metrics", 2024-03-21T10:50:26.734 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.734 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.734 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.734 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "pgs,objects,bytes", 2024-03-21T10:50:26.734 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.734 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.734 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.735 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-21T10:50:26.735 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-21T10:50:26.735 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.735 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.735 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_step": { 2024-03-21T10:50:26.735 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_step", 2024-03-21T10:50:26.735 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-03-21T10:50:26.735 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.735 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.735 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.5", 2024-03-21T10:50:26.735 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0.001", 2024-03-21T10:50:26.735 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "0.999", 2024-03-21T10:50:26.735 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.736 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "aggressiveness of optimization", 2024-03-21T10:50:26.736 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-21T10:50:26.736 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.736 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.736 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.736 INFO:teuthology.orchestra.run.smithi031.stdout: "end_time": { 2024-03-21T10:50:26.736 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_time", 2024-03-21T10:50:26.736 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.736 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.736 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.736 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2359", 2024-03-21T10:50:26.737 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.737 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.737 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.737 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ending time of day to automatically balance", 2024-03-21T10:50:26.737 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-21T10:50:26.737 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.737 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.737 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.737 INFO:teuthology.orchestra.run.smithi031.stdout: "end_weekday": { 2024-03-21T10:50:26.737 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_weekday", 2024-03-21T10:50:26.737 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-03-21T10:50:26.737 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.737 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.737 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-03-21T10:50:26.738 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-03-21T10:50:26.738 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-03-21T10:50:26.738 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.738 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-21T10:50:26.738 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-21T10:50:26.738 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.738 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.738 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.738 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:26.738 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:26.738 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.738 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.738 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.738 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.739 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.739 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.739 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.739 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.739 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.739 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.739 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.739 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.739 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.739 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.739 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.739 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.739 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.739 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.740 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:26.740 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:26.740 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.740 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.740 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.740 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.740 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.740 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.740 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.740 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.741 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.741 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.741 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.741 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:26.741 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:26.741 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.741 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.741 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.741 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:26.741 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.741 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.741 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.741 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.742 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.742 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.742 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.742 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.742 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.742 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.742 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.742 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.742 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.742 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.742 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:26.742 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:26.742 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.743 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.743 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.743 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.743 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.743 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.743 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.743 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.743 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.743 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.743 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.743 INFO:teuthology.orchestra.run.smithi031.stdout: "min_score": { 2024-03-21T10:50:26.743 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_score", 2024-03-21T10:50:26.743 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-03-21T10:50:26.744 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.744 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.744 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-03-21T10:50:26.744 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.744 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.744 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.744 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-21T10:50:26.744 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.744 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.744 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.744 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.744 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-03-21T10:50:26.744 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-03-21T10:50:26.744 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.745 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.745 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.745 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "upmap", 2024-03-21T10:50:26.745 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.745 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.745 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.745 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-compat", 2024-03-21T10:50:26.745 INFO:teuthology.orchestra.run.smithi031.stdout: "none", 2024-03-21T10:50:26.745 INFO:teuthology.orchestra.run.smithi031.stdout: "read", 2024-03-21T10:50:26.745 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap", 2024-03-21T10:50:26.745 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap-read" 2024-03-21T10:50:26.745 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.745 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Balancer mode", 2024-03-21T10:50:26.745 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.745 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.746 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.746 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.746 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_ids": { 2024-03-21T10:50:26.746 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_ids", 2024-03-21T10:50:26.746 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.746 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.746 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.746 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.746 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.746 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.746 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.746 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-21T10:50:26.746 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.746 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.747 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.747 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.747 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-03-21T10:50:26.747 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-03-21T10:50:26.747 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:26.747 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.747 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.747 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-03-21T10:50:26.747 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.747 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.747 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.747 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-21T10:50:26.747 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.747 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.747 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.748 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.748 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_deviation": { 2024-03-21T10:50:26.748 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_deviation", 2024-03-21T10:50:26.748 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.748 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.748 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.748 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-03-21T10:50:26.748 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-03-21T10:50:26.748 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.748 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.748 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-21T10:50:26.748 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-21T10:50:26.749 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.749 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.749 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_optimizations": { 2024-03-21T10:50:26.749 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_optimizations", 2024-03-21T10:50:26.749 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-03-21T10:50:26.749 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.749 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.749 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-03-21T10:50:26.749 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.749 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.749 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.749 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-21T10:50:26.749 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.750 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.750 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.750 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.750 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.750 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:26.750 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm", 2024-03-21T10:50:26.750 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:26.750 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:26.750 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:26.750 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_down_multiplier": { 2024-03-21T10:50:26.750 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_down_multiplier", 2024-03-21T10:50:26.750 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-03-21T10:50:26.751 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.751 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.751 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-03-21T10:50:26.751 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.751 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.751 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.751 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-21T10:50:26.751 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.751 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.751 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.751 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_refresh_rate": { 2024-03-21T10:50:26.751 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_refresh_rate", 2024-03-21T10:50:26.751 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:26.752 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.752 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.752 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "20", 2024-03-21T10:50:26.752 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.752 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.752 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.752 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-21T10:50:26.752 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.752 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.752 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.752 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_starting_port": { 2024-03-21T10:50:26.752 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_starting_port", 2024-03-21T10:50:26.752 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.752 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.753 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.753 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4721", 2024-03-21T10:50:26.753 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.753 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.753 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.753 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-21T10:50:26.753 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.753 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.753 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.753 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_ptrace": { 2024-03-21T10:50:26.753 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_ptrace", 2024-03-21T10:50:26.754 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.754 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.754 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.754 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.754 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.754 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.754 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.754 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-21T10:50:26.754 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-21T10:50:26.754 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.754 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.754 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_interval": { 2024-03-21T10:50:26.754 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_interval", 2024-03-21T10:50:26.755 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:26.755 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.755 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.755 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-03-21T10:50:26.755 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.755 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.755 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.755 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-21T10:50:26.755 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.755 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.755 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.755 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_memory_target_ratio": { 2024-03-21T10:50:26.755 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_memory_target_ratio", 2024-03-21T10:50:26.756 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-03-21T10:50:26.756 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.756 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.756 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.7", 2024-03-21T10:50:26.756 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.756 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.756 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.756 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-21T10:50:26.756 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.756 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.756 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.756 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.757 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm_log_destination": { 2024-03-21T10:50:26.757 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm_log_destination", 2024-03-21T10:50:26.757 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.757 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.757 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.757 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.757 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.757 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.757 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.757 INFO:teuthology.orchestra.run.smithi031.stdout: "file", 2024-03-21T10:50:26.757 INFO:teuthology.orchestra.run.smithi031.stdout: "file,syslog", 2024-03-21T10:50:26.757 INFO:teuthology.orchestra.run.smithi031.stdout: "syslog" 2024-03-21T10:50:26.757 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.757 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-21T10:50:26.758 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.758 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.758 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.758 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.758 INFO:teuthology.orchestra.run.smithi031.stdout: "cgroups_split": { 2024-03-21T10:50:26.758 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cgroups_split", 2024-03-21T10:50:26.758 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.758 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.758 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.758 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.758 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.758 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.758 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.758 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-21T10:50:26.759 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.759 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.759 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.759 INFO:teuthology.orchestra.run.smithi031.stdout: "config_checks_enabled": { 2024-03-21T10:50:26.759 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_checks_enabled", 2024-03-21T10:50:26.759 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.759 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.759 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.759 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.759 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.759 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.759 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.759 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-21T10:50:26.759 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.760 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.760 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.760 INFO:teuthology.orchestra.run.smithi031.stdout: "config_dashboard": { 2024-03-21T10:50:26.760 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_dashboard", 2024-03-21T10:50:26.760 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.760 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.760 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.760 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.760 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.760 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.760 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.760 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-21T10:50:26.760 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.761 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.761 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.761 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_alertmanager": { 2024-03-21T10:50:26.761 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_alertmanager", 2024-03-21T10:50:26.761 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.761 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.761 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.761 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-21T10:50:26.761 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.761 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.761 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.761 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-03-21T10:50:26.761 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.762 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.762 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.762 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_base": { 2024-03-21T10:50:26.762 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_base", 2024-03-21T10:50:26.762 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.762 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.762 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.762 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-21T10:50:26.762 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.762 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.762 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.762 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Container image name, without the tag", 2024-03-21T10:50:26.762 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.763 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.763 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.763 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.763 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_elasticsearch": { 2024-03-21T10:50:26.763 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_elasticsearch", 2024-03-21T10:50:26.763 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.763 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.763 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.763 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-21T10:50:26.763 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.763 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.763 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.763 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "elasticsearch container image", 2024-03-21T10:50:26.763 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.763 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.764 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.764 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_grafana": { 2024-03-21T10:50:26.764 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_grafana", 2024-03-21T10:50:26.764 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.764 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.764 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.764 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-03-21T10:50:26.764 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.764 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.764 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.764 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-03-21T10:50:26.764 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.764 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.764 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.765 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.765 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_haproxy": { 2024-03-21T10:50:26.765 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_haproxy", 2024-03-21T10:50:26.765 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.765 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.765 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.765 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-21T10:50:26.765 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.765 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.765 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.765 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "HAproxy container image", 2024-03-21T10:50:26.765 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.766 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.766 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.766 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_agent": { 2024-03-21T10:50:26.766 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_agent", 2024-03-21T10:50:26.766 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.766 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.766 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.766 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-21T10:50:26.766 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.766 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.766 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.766 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger agent container image", 2024-03-21T10:50:26.767 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.767 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.767 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.767 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_collector": { 2024-03-21T10:50:26.767 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_collector", 2024-03-21T10:50:26.767 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.767 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.767 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.767 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-21T10:50:26.767 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.767 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.767 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.767 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger collector container image", 2024-03-21T10:50:26.767 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.768 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.768 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.768 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.768 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_query": { 2024-03-21T10:50:26.768 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_query", 2024-03-21T10:50:26.768 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.768 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.768 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.768 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-21T10:50:26.768 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.768 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.768 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.768 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger query container image", 2024-03-21T10:50:26.768 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.769 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.769 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.769 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_keepalived": { 2024-03-21T10:50:26.769 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_keepalived", 2024-03-21T10:50:26.769 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.769 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.769 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.769 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-21T10:50:26.769 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.769 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.769 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.769 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Keepalived container image", 2024-03-21T10:50:26.770 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.770 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.770 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.770 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_loki": { 2024-03-21T10:50:26.770 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_loki", 2024-03-21T10:50:26.770 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.770 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.770 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.770 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-21T10:50:26.770 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.771 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.771 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.771 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Loki container image", 2024-03-21T10:50:26.771 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.771 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.771 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.771 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_node_exporter": { 2024-03-21T10:50:26.771 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_node_exporter", 2024-03-21T10:50:26.771 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.771 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.772 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.772 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-21T10:50:26.772 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.772 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.772 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.772 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-03-21T10:50:26.772 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.772 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.772 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.772 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_nvmeof": { 2024-03-21T10:50:26.772 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_nvmeof", 2024-03-21T10:50:26.773 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.773 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.773 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.773 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-21T10:50:26.773 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.773 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.773 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.773 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Nvme-of container image", 2024-03-21T10:50:26.773 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.773 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.773 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.774 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.774 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_prometheus": { 2024-03-21T10:50:26.774 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_prometheus", 2024-03-21T10:50:26.774 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.774 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.774 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.774 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-21T10:50:26.774 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.774 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.774 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.775 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-03-21T10:50:26.775 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.775 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.775 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.775 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_promtail": { 2024-03-21T10:50:26.775 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_promtail", 2024-03-21T10:50:26.775 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.775 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.775 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.775 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-21T10:50:26.776 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.776 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.776 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.776 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Promtail container image", 2024-03-21T10:50:26.776 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.776 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.776 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.776 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_snmp_gateway": { 2024-03-21T10:50:26.776 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_snmp_gateway", 2024-03-21T10:50:26.776 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.776 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.777 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.777 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-21T10:50:26.777 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.777 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.777 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.777 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SNMP Gateway container image", 2024-03-21T10:50:26.777 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.777 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.777 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.777 INFO:teuthology.orchestra.run.smithi031.stdout: "container_init": { 2024-03-21T10:50:26.777 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_init", 2024-03-21T10:50:26.778 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.778 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.778 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.778 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.778 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.778 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.778 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.778 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Run podman/docker with `--init`", 2024-03-21T10:50:26.778 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.778 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.778 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.779 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.779 INFO:teuthology.orchestra.run.smithi031.stdout: "daemon_cache_timeout": { 2024-03-21T10:50:26.779 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "daemon_cache_timeout", 2024-03-21T10:50:26.779 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:26.779 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.779 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.779 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-03-21T10:50:26.779 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.779 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.779 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.780 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-21T10:50:26.780 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.780 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.780 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.780 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.780 INFO:teuthology.orchestra.run.smithi031.stdout: "default_cephadm_command_timeout": { 2024-03-21T10:50:26.780 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_cephadm_command_timeout", 2024-03-21T10:50:26.780 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.780 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.780 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.780 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "900", 2024-03-21T10:50:26.781 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.781 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.781 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.781 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-21T10:50:26.781 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.781 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.781 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.781 INFO:teuthology.orchestra.run.smithi031.stdout: "default_registry": { 2024-03-21T10:50:26.781 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_registry", 2024-03-21T10:50:26.781 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.781 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.782 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.782 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io", 2024-03-21T10:50:26.782 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.782 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.782 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.782 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-21T10:50:26.782 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.782 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.782 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.782 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.782 INFO:teuthology.orchestra.run.smithi031.stdout: "device_cache_timeout": { 2024-03-21T10:50:26.782 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_cache_timeout", 2024-03-21T10:50:26.783 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:26.783 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.783 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.783 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1800", 2024-03-21T10:50:26.783 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.783 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.783 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.783 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache device inventory", 2024-03-21T10:50:26.783 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.783 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.784 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.784 INFO:teuthology.orchestra.run.smithi031.stdout: "device_enhanced_scan": { 2024-03-21T10:50:26.784 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_enhanced_scan", 2024-03-21T10:50:26.784 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.784 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.784 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.784 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.784 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.784 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.784 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.784 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-21T10:50:26.784 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.784 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.785 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.785 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.785 INFO:teuthology.orchestra.run.smithi031.stdout: "facts_cache_timeout": { 2024-03-21T10:50:26.785 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "facts_cache_timeout", 2024-03-21T10:50:26.785 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:26.785 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.785 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.785 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-03-21T10:50:26.785 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.785 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.785 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.785 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache host facts data", 2024-03-21T10:50:26.785 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.785 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.785 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.786 INFO:teuthology.orchestra.run.smithi031.stdout: "grafana_dashboards_path": { 2024-03-21T10:50:26.786 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "grafana_dashboards_path", 2024-03-21T10:50:26.786 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.786 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.786 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.786 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-03-21T10:50:26.786 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.786 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.786 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.786 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-03-21T10:50:26.786 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.786 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.786 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.787 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.787 INFO:teuthology.orchestra.run.smithi031.stdout: "host_check_interval": { 2024-03-21T10:50:26.787 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "host_check_interval", 2024-03-21T10:50:26.787 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:26.787 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.787 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.787 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-03-21T10:50:26.787 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.787 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.787 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.787 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to perform a host check", 2024-03-21T10:50:26.787 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.787 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.787 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.787 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.788 INFO:teuthology.orchestra.run.smithi031.stdout: "hw_monitoring": { 2024-03-21T10:50:26.788 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hw_monitoring", 2024-03-21T10:50:26.788 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.788 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.788 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.788 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.788 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.788 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.788 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.788 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-21T10:50:26.788 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.788 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.788 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.788 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.789 INFO:teuthology.orchestra.run.smithi031.stdout: "inventory_list_all": { 2024-03-21T10:50:26.789 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "inventory_list_all", 2024-03-21T10:50:26.789 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.789 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.789 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.789 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.789 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.789 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.789 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.789 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-21T10:50:26.789 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.789 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.789 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.789 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:26.790 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:26.790 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.790 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.790 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.790 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.790 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.790 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.790 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.790 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.790 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.790 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.790 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.790 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.791 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.791 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.791 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.791 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.791 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.791 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.791 INFO:teuthology.orchestra.run.smithi031.stdout: "log_refresh_metadata": { 2024-03-21T10:50:26.791 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_refresh_metadata", 2024-03-21T10:50:26.791 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.791 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.791 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.792 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.792 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.792 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.792 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.792 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-21T10:50:26.792 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.792 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.792 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.792 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:26.792 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:26.793 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.793 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.793 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.793 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.793 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.793 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.793 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.793 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-21T10:50:26.793 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.793 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.793 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.794 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:26.794 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:26.794 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.794 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.794 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.794 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:26.794 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.794 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.794 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.794 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.794 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.794 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.794 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.795 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.795 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.795 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.795 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.795 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.795 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.795 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.795 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.795 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:26.795 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:26.795 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.795 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.795 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.795 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.796 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.796 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.796 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.796 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.796 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.796 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.796 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.796 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-21T10:50:26.796 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-21T10:50:26.796 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.796 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.796 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.796 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.797 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.797 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.797 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.797 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-21T10:50:26.797 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.797 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.797 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.797 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-21T10:50:26.797 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-21T10:50:26.797 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.797 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.797 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.797 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "*", 2024-03-21T10:50:26.798 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.798 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.798 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.798 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-21T10:50:26.798 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.798 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.798 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.798 INFO:teuthology.orchestra.run.smithi031.stdout: "max_count_per_host": { 2024-03-21T10:50:26.798 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_count_per_host", 2024-03-21T10:50:26.798 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.798 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.798 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.798 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-03-21T10:50:26.799 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.799 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.799 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.799 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of daemons per service per host", 2024-03-21T10:50:26.799 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.799 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.799 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.799 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd_draining_count": { 2024-03-21T10:50:26.799 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_osd_draining_count", 2024-03-21T10:50:26.799 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.799 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.799 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.799 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-03-21T10:50:26.800 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.800 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.800 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.800 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-21T10:50:26.800 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.800 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.800 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.800 INFO:teuthology.orchestra.run.smithi031.stdout: "migration_current": { 2024-03-21T10:50:26.800 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "migration_current", 2024-03-21T10:50:26.800 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.800 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.800 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.800 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.801 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.801 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.801 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.801 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "internal - do not modify", 2024-03-21T10:50:26.801 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.801 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.801 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.801 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.801 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-03-21T10:50:26.801 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-03-21T10:50:26.801 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.801 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.801 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.801 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "root", 2024-03-21T10:50:26.801 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.802 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.802 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.802 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm-package", 2024-03-21T10:50:26.802 INFO:teuthology.orchestra.run.smithi031.stdout: "root" 2024-03-21T10:50:26.802 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.802 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "mode for remote execution of cephadm", 2024-03-21T10:50:26.802 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.802 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.802 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.802 INFO:teuthology.orchestra.run.smithi031.stdout: "oob_default_addr": { 2024-03-21T10:50:26.802 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "oob_default_addr", 2024-03-21T10:50:26.802 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.802 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.803 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.803 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "169.254.1.1", 2024-03-21T10:50:26.803 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.803 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.803 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.803 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-21T10:50:26.803 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.803 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.803 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.803 INFO:teuthology.orchestra.run.smithi031.stdout: "prometheus_alerts_path": { 2024-03-21T10:50:26.803 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus_alerts_path", 2024-03-21T10:50:26.803 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.803 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.804 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.804 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-21T10:50:26.804 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.804 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.804 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.804 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-21T10:50:26.804 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.804 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.804 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.804 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_insecure": { 2024-03-21T10:50:26.804 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_insecure", 2024-03-21T10:50:26.804 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.805 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.805 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.805 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.805 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.805 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.805 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.805 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-21T10:50:26.805 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.805 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.805 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.806 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_password": { 2024-03-21T10:50:26.806 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_password", 2024-03-21T10:50:26.806 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.806 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.806 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.806 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.806 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.806 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.806 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.807 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-21T10:50:26.807 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.807 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.807 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.807 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_url": { 2024-03-21T10:50:26.807 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_url", 2024-03-21T10:50:26.807 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.807 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.807 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.807 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.807 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.808 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.808 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.808 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-21T10:50:26.808 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.808 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.808 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.808 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_username": { 2024-03-21T10:50:26.808 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_username", 2024-03-21T10:50:26.808 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.808 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.809 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.809 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.809 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.809 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.809 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.809 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-21T10:50:26.809 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.809 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.809 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.809 INFO:teuthology.orchestra.run.smithi031.stdout: "secure_monitoring_stack": { 2024-03-21T10:50:26.810 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "secure_monitoring_stack", 2024-03-21T10:50:26.810 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.810 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.810 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.810 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.810 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.810 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.810 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.810 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-21T10:50:26.810 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.810 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.811 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.811 INFO:teuthology.orchestra.run.smithi031.stdout: "service_discovery_port": { 2024-03-21T10:50:26.811 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "service_discovery_port", 2024-03-21T10:50:26.811 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.811 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.811 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.811 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8765", 2024-03-21T10:50:26.811 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.811 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.811 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.811 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "cephadm service discovery port", 2024-03-21T10:50:26.811 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.812 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.812 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.812 INFO:teuthology.orchestra.run.smithi031.stdout: "ssh_config_file": { 2024-03-21T10:50:26.812 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssh_config_file", 2024-03-21T10:50:26.812 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.812 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.812 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.812 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.812 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.812 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.812 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.812 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-21T10:50:26.812 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.813 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.813 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.813 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.813 INFO:teuthology.orchestra.run.smithi031.stdout: "use_agent": { 2024-03-21T10:50:26.813 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_agent", 2024-03-21T10:50:26.813 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.813 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.813 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.813 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.813 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.813 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.814 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.814 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-21T10:50:26.814 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.814 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.814 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.814 INFO:teuthology.orchestra.run.smithi031.stdout: "use_repo_digest": { 2024-03-21T10:50:26.814 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_repo_digest", 2024-03-21T10:50:26.814 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.814 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.814 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.815 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.815 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.815 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.815 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.815 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-21T10:50:26.815 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.815 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.815 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.815 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.815 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_failed_host_check": { 2024-03-21T10:50:26.815 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_failed_host_check", 2024-03-21T10:50:26.816 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.816 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.816 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.816 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.816 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.816 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.816 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.816 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if the host check fails", 2024-03-21T10:50:26.816 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.816 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.816 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.817 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.817 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_daemons": { 2024-03-21T10:50:26.817 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_daemons", 2024-03-21T10:50:26.817 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.817 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.817 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.817 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.817 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.817 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.817 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.817 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-21T10:50:26.817 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.817 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.817 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.818 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_hosts": { 2024-03-21T10:50:26.818 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_hosts", 2024-03-21T10:50:26.818 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.818 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.818 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.818 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.818 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.818 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.818 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.818 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-21T10:50:26.818 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.818 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.818 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.819 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.819 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.819 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:26.819 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crash", 2024-03-21T10:50:26.819 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:26.819 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:26.819 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:26.819 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:26.819 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:26.819 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.819 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.819 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.820 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.820 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.820 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.820 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.820 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.820 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.820 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.820 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.820 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.820 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.820 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.821 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.821 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.821 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.821 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.821 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:26.821 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:26.821 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.821 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.821 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.821 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.821 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.821 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.821 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.822 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.822 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.822 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.822 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.822 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:26.822 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:26.822 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.822 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.822 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.822 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:26.822 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.822 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.822 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.823 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.825 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.825 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.825 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.825 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.825 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.825 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.825 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.825 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.825 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.825 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.826 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.826 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:26.826 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:26.826 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.826 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.826 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.826 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.826 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.826 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.826 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.826 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.826 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.826 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.827 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.827 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.827 INFO:teuthology.orchestra.run.smithi031.stdout: "retain_interval": { 2024-03-21T10:50:26.827 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retain_interval", 2024-03-21T10:50:26.827 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:26.827 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.827 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.827 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "31536000", 2024-03-21T10:50:26.827 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.827 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.827 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.827 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-21T10:50:26.827 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.827 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.828 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.828 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_recent_interval": { 2024-03-21T10:50:26.828 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_recent_interval", 2024-03-21T10:50:26.828 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:26.828 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.828 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.828 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1209600", 2024-03-21T10:50:26.828 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.828 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.828 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.828 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-21T10:50:26.828 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.828 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.829 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.829 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.829 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.829 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:26.829 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dashboard", 2024-03-21T10:50:26.829 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:26.829 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:26.829 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:26.829 INFO:teuthology.orchestra.run.smithi031.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-21T10:50:26.829 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-21T10:50:26.829 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.829 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.829 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.829 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-03-21T10:50:26.830 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.830 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.830 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.830 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.830 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.830 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.830 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.830 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.830 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_HOST": { 2024-03-21T10:50:26.830 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-21T10:50:26.830 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.830 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.830 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.830 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.831 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.831 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.831 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.831 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.831 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.831 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.831 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.831 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-21T10:50:26.831 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-21T10:50:26.831 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.831 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.831 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.831 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.831 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.832 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.832 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.832 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.832 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.832 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.832 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.832 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_ENABLED": { 2024-03-21T10:50:26.832 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_ENABLED", 2024-03-21T10:50:26.832 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.832 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.832 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.832 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.832 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.832 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.833 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.833 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.833 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.833 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.833 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.833 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-21T10:50:26.833 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-21T10:50:26.833 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.833 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.833 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.833 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.833 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.833 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.833 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.834 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.834 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.834 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.834 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.834 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.834 INFO:teuthology.orchestra.run.smithi031.stdout: "ENABLE_BROWSABLE_API": { 2024-03-21T10:50:26.834 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-21T10:50:26.834 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.834 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.834 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.834 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.834 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.834 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.834 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.835 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.835 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.835 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.835 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.835 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-21T10:50:26.835 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-21T10:50:26.835 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.835 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.835 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.835 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.835 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.835 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.835 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.836 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.836 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.836 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.836 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.836 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.836 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-21T10:50:26.836 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-21T10:50:26.836 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.836 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.836 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.836 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.836 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.837 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.837 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.837 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.837 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.837 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.837 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.837 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-21T10:50:26.837 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-21T10:50:26.837 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.837 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.837 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.837 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.837 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.838 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.838 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.838 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.838 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.838 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.838 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.838 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-21T10:50:26.838 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-21T10:50:26.838 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.838 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.838 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.838 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.838 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-21T10:50:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-21T10:50:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-21T10:50:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-21T10:50:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-21T10:50:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-21T10:50:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-21T10:50:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-21T10:50:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_PASSWORD": { 2024-03-21T10:50:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-21T10:50:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-03-21T10:50:26.843 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-21T10:50:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-21T10:50:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.844 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_URL": { 2024-03-21T10:50:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_URL", 2024-03-21T10:50:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.845 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_USERNAME": { 2024-03-21T10:50:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-21T10:50:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-03-21T10:50:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.846 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-21T10:50:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-21T10:50:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.847 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.848 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.848 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.848 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.848 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.848 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-21T10:50:26.848 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-21T10:50:26.848 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.848 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.848 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.848 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.848 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.849 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.849 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.849 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.849 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.849 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.849 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.849 INFO:teuthology.orchestra.run.smithi031.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-21T10:50:26.849 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-21T10:50:26.849 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.849 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.849 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-21T10:50:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-21T10:50:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: "MULTICLUSTER_CONFIG": { 2024-03-21T10:50:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-21T10:50:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.851 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "{}", 2024-03-21T10:50:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_HOST": { 2024-03-21T10:50:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-21T10:50:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.852 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.853 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.853 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.853 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.853 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.853 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.853 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.853 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.853 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.853 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-21T10:50:26.853 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-21T10:50:26.853 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.853 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.854 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.854 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.854 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.854 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.854 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.854 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.854 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.854 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.854 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.854 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.854 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-21T10:50:26.854 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-21T10:50:26.854 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.854 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-21T10:50:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-21T10:50:26.855 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.856 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.856 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.856 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.856 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.856 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.856 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.856 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.856 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.856 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.856 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.856 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.856 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-21T10:50:26.856 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-21T10:50:26.856 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-21T10:50:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-21T10:50:26.857 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.858 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.858 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.858 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.858 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.858 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.858 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.858 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.858 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.858 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.858 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.858 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-21T10:50:26.859 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-21T10:50:26.859 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.859 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.859 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.859 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.859 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.859 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.859 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.859 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.859 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.860 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.860 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.860 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-21T10:50:26.860 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-21T10:50:26.860 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.860 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.860 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.860 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.860 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.860 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-21T10:50:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-21T10:50:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.861 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.862 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.862 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.862 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.862 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.862 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.862 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.862 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.862 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_ENABLED": { 2024-03-21T10:50:26.862 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-21T10:50:26.862 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.862 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.862 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.862 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-21T10:50:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-21T10:50:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.863 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-21T10:50:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-21T10:50:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-21T10:50:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-03-21T10:50:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-21T10:50:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-21T10:50:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8", 2024-03-21T10:50:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-21T10:50:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-21T10:50:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "45", 2024-03-21T10:50:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ACCESS_KEY": { 2024-03-21T10:50:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-21T10:50:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-21T10:50:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-21T10:50:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-03-21T10:50:26.868 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SECRET_KEY": { 2024-03-21T10:50:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-21T10:50:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.869 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SSL_VERIFY": { 2024-03-21T10:50:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-21T10:50:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.870 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: "UNSAFE_TLS_v1_2": { 2024-03-21T10:50:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-21T10:50:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.871 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-21T10:50:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-21T10:50:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-03-21T10:50:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.872 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-21T10:50:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-21T10:50:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-03-21T10:50:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.873 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-21T10:50:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-21T10:50:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-03-21T10:50:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.874 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: "cross_origin_url": { 2024-03-21T10:50:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cross_origin_url", 2024-03-21T10:50:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.875 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: "crt_file": { 2024-03-21T10:50:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crt_file", 2024-03-21T10:50:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.876 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: "debug": { 2024-03-21T10:50:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "debug", 2024-03-21T10:50:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable/disable debug options", 2024-03-21T10:50:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.877 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: "jwt_token_ttl": { 2024-03-21T10:50:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "jwt_token_ttl", 2024-03-21T10:50:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "28800", 2024-03-21T10:50:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.878 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-03-21T10:50:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-03-21T10:50:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.879 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.880 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.881 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.882 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.883 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.884 INFO:teuthology.orchestra.run.smithi031.stdout: "motd": { 2024-03-21T10:50:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "motd", 2024-03-21T10:50:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "The message of the day", 2024-03-21T10:50:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect_resolve_ip_addr": { 2024-03-21T10:50:26.885 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "redirect_resolve_ip_addr", 2024-03-21T10:50:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-03-21T10:50:26.886 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-03-21T10:50:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-03-21T10:50:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-03-21T10:50:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-03-21T10:50:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8080", 2024-03-21T10:50:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-03-21T10:50:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-03-21T10:50:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl_server_port": { 2024-03-21T10:50:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl_server_port", 2024-03-21T10:50:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8443", 2024-03-21T10:50:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-03-21T10:50:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-03-21T10:50:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "redirect", 2024-03-21T10:50:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect" 2024-03-21T10:50:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-03-21T10:50:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-03-21T10:50:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-03-21T10:50:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-03-21T10:50:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-03-21T10:50:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "url_prefix": { 2024-03-21T10:50:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url_prefix", 2024-03-21T10:50:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "devicehealth", 2024-03-21T10:50:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_monitoring": { 2024-03-21T10:50:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_monitoring", 2024-03-21T10:50:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "monitor device health metrics", 2024-03-21T10:50:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.895 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:26.896 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.897 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.898 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.899 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: "mark_out_threshold": { 2024-03-21T10:50:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mark_out_threshold", 2024-03-21T10:50:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2419200", 2024-03-21T10:50:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-21T10:50:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.900 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": { 2024-03-21T10:50:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_name", 2024-03-21T10:50:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "device_health_metrics", 2024-03-21T10:50:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-21T10:50:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.901 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.902 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.902 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.902 INFO:teuthology.orchestra.run.smithi031.stdout: "retention_period": { 2024-03-21T10:50:26.902 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retention_period", 2024-03-21T10:50:26.902 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:26.902 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.902 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.902 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15552000", 2024-03-21T10:50:26.902 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.902 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.902 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.902 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain device health metrics", 2024-03-21T10:50:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_frequency": { 2024-03-21T10:50:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_frequency", 2024-03-21T10:50:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-03-21T10:50:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.903 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-21T10:50:26.904 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.904 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.904 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.904 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.904 INFO:teuthology.orchestra.run.smithi031.stdout: "self_heal": { 2024-03-21T10:50:26.904 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "self_heal", 2024-03-21T10:50:26.904 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.904 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.904 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.904 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:26.904 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.904 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.904 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.904 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-21T10:50:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-03-21T10:50:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-03-21T10:50:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-03-21T10:50:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and check device health", 2024-03-21T10:50:26.905 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_threshold": { 2024-03-21T10:50:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_threshold", 2024-03-21T10:50:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7257600", 2024-03-21T10:50:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-21T10:50:26.906 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.907 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.907 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.907 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.907 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.907 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.907 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:26.907 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "diskprediction_local", 2024-03-21T10:50:26.907 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:26.907 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:26.907 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:26.907 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:26.907 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:26.907 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.907 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.908 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.909 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.910 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.911 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: "predict_interval": { 2024-03-21T10:50:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predict_interval", 2024-03-21T10:50:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-03-21T10:50:26.912 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: "predictor_model": { 2024-03-21T10:50:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predictor_model", 2024-03-21T10:50:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "prophetstor", 2024-03-21T10:50:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.913 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-03-21T10:50:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-03-21T10:50:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-03-21T10:50:26.914 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "influx", 2024-03-21T10:50:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": false, 2024-03-21T10:50:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "influxdb python module not found", 2024-03-21T10:50:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:26.915 INFO:teuthology.orchestra.run.smithi031.stdout: "batch_size": { 2024-03-21T10:50:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "batch_size", 2024-03-21T10:50:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5000", 2024-03-21T10:50:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-21T10:50:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.916 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: "database": { 2024-03-21T10:50:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "database", 2024-03-21T10:50:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "ceph", 2024-03-21T10:50:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-21T10:50:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.917 INFO:teuthology.orchestra.run.smithi031.stdout: "hostname": { 2024-03-21T10:50:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hostname", 2024-03-21T10:50:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server hostname", 2024-03-21T10:50:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.918 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-03-21T10:50:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-03-21T10:50:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "30", 2024-03-21T10:50:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "5", 2024-03-21T10:50:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-21T10:50:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.919 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.920 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.921 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.921 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.921 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.921 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.921 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.921 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.921 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:26.921 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:26.921 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.921 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.921 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.921 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.921 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.921 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.922 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:26.923 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:26.924 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.924 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.924 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.924 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.924 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.924 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.924 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.924 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.924 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.924 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.924 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.924 INFO:teuthology.orchestra.run.smithi031.stdout: "password": { 2024-03-21T10:50:26.924 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "password", 2024-03-21T10:50:26.924 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "password of InfluxDB server user", 2024-03-21T10:50:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: "port": { 2024-03-21T10:50:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "port", 2024-03-21T10:50:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.925 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8086", 2024-03-21T10:50:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server port", 2024-03-21T10:50:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-03-21T10:50:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-03-21T10:50:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.926 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "false", 2024-03-21T10:50:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-21T10:50:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: "threads": { 2024-03-21T10:50:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threads", 2024-03-21T10:50:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.927 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-03-21T10:50:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-03-21T10:50:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "32", 2024-03-21T10:50:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-21T10:50:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: "username": { 2024-03-21T10:50:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "username", 2024-03-21T10:50:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.928 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.929 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.929 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.929 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.929 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.929 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.929 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "username of InfluxDB server user", 2024-03-21T10:50:26.929 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.929 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.929 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.929 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.929 INFO:teuthology.orchestra.run.smithi031.stdout: "verify_ssl": { 2024-03-21T10:50:26.929 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "verify_ssl", 2024-03-21T10:50:26.929 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.929 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.930 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.930 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "true", 2024-03-21T10:50:26.930 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.930 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.930 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.930 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-21T10:50:26.930 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.930 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.930 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.930 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.930 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.930 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.930 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:26.930 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "insights", 2024-03-21T10:50:26.931 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:26.931 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:26.931 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:26.931 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:26.931 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:26.931 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.931 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.931 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.931 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.931 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.931 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.931 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.931 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.931 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.931 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.932 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.932 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.932 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.932 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.932 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.932 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.932 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.932 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.932 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.932 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:26.932 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:26.932 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.932 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.932 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.933 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.933 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.933 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.933 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.933 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.933 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.933 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.933 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.933 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.933 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:26.933 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:26.933 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.933 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.933 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.933 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:26.934 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.934 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.934 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.934 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.934 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.934 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.934 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.934 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.934 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.934 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.934 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.934 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.934 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.934 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.935 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.935 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:26.935 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:26.935 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.935 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.935 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.935 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.935 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.935 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.935 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.935 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.935 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.935 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.935 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.935 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.936 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.936 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:26.936 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "iostat", 2024-03-21T10:50:26.936 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:26.936 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:26.936 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:26.936 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:26.936 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:26.936 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.936 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.936 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.936 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.936 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.936 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.937 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.937 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.937 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.937 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.937 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.937 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.937 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.937 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.937 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.937 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.937 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.937 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.937 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:26.937 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:26.938 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.938 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.938 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.938 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.938 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.938 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.938 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.938 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.938 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.938 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.938 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.938 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.938 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:26.938 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:26.938 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.939 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.939 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.939 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:26.939 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.939 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.939 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.939 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.939 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.939 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.939 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.939 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.939 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.939 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.939 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.939 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.940 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.940 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.940 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.940 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:26.940 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:26.940 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.940 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.940 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.940 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.940 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.940 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.940 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.940 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.940 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.940 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.941 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.941 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.941 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.941 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.941 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:26.941 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "k8sevents", 2024-03-21T10:50:26.941 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:26.941 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:26.941 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:26.941 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_event_retention_days": { 2024-03-21T10:50:26.941 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ceph_event_retention_days", 2024-03-21T10:50:26.941 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.941 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.941 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.941 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7", 2024-03-21T10:50:26.942 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.942 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.942 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.942 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-21T10:50:26.942 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.942 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.942 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.942 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.942 INFO:teuthology.orchestra.run.smithi031.stdout: "config_check_secs": { 2024-03-21T10:50:26.942 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_check_secs", 2024-03-21T10:50:26.942 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.942 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.942 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.943 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-03-21T10:50:26.943 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "10", 2024-03-21T10:50:26.943 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.943 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.943 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-21T10:50:26.943 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.943 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.943 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.943 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.943 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:26.943 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:26.943 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.943 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.943 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.944 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.944 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.944 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.944 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.944 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.944 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.944 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.944 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.944 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.944 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.944 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.944 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.944 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.944 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.944 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.945 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:26.945 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:26.945 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.945 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.945 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.945 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.945 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.945 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.945 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.945 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.945 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.945 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.945 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.946 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:26.946 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:26.946 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.946 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.946 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.946 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:26.946 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.946 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.946 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.946 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.946 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.946 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.946 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.946 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.947 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.947 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.947 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.947 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.947 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.947 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.947 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.947 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:26.947 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:26.947 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.947 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.947 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.947 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.947 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.947 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.948 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.948 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.948 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.948 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.948 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.948 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.948 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.948 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.948 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:26.948 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "localpool", 2024-03-21T10:50:26.948 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:26.948 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:26.948 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:26.948 INFO:teuthology.orchestra.run.smithi031.stdout: "failure_domain": { 2024-03-21T10:50:26.949 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "failure_domain", 2024-03-21T10:50:26.949 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.949 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.949 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.949 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "host", 2024-03-21T10:50:26.949 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.949 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.949 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.949 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "failure domain for any created local pool", 2024-03-21T10:50:26.949 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-21T10:50:26.949 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.949 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.949 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.949 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:26.949 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:26.950 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.950 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.950 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.950 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.950 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.950 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.950 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.950 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.950 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.950 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.950 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.950 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.950 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.950 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.951 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.951 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.951 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.951 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.951 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.951 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:26.951 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:26.951 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.951 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.951 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.951 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.951 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.951 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.951 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.951 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.952 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.952 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.952 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.952 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.952 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:26.952 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:26.952 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.952 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.952 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.952 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:26.952 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.952 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.952 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.952 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.953 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.953 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.953 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.953 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.953 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.953 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.953 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.953 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.953 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.953 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.953 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.953 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:26.953 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:26.953 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.953 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.954 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.954 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.954 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.954 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.954 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.954 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.954 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.954 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.954 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.954 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.954 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": { 2024-03-21T10:50:26.954 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_size", 2024-03-21T10:50:26.954 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.954 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.955 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.955 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.955 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.955 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.955 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.955 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default min_size for any created local pool", 2024-03-21T10:50:26.955 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-21T10:50:26.955 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.955 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.955 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.955 INFO:teuthology.orchestra.run.smithi031.stdout: "num_rep": { 2024-03-21T10:50:26.955 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "num_rep", 2024-03-21T10:50:26.956 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.956 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.956 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.956 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3", 2024-03-21T10:50:26.956 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.956 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.956 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.956 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default replica count for any created local pool", 2024-03-21T10:50:26.956 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.956 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.956 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.956 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.956 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": { 2024-03-21T10:50:26.956 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_num", 2024-03-21T10:50:26.956 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:26.957 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.957 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.957 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "128", 2024-03-21T10:50:26.957 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.957 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.957 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.957 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default pg_num for any created local pool", 2024-03-21T10:50:26.957 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.957 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.957 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.957 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.957 INFO:teuthology.orchestra.run.smithi031.stdout: "prefix": { 2024-03-21T10:50:26.957 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prefix", 2024-03-21T10:50:26.957 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.958 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.958 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.958 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.958 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.958 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.958 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.958 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name prefix for any created local pool", 2024-03-21T10:50:26.958 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.958 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.958 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.958 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.958 INFO:teuthology.orchestra.run.smithi031.stdout: "subtree": { 2024-03-21T10:50:26.959 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "subtree", 2024-03-21T10:50:26.959 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.959 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.959 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.959 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "rack", 2024-03-21T10:50:26.959 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.959 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.959 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.959 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-21T10:50:26.959 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-21T10:50:26.959 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.959 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.960 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.960 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.960 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:26.960 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mds_autoscaler", 2024-03-21T10:50:26.960 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:26.960 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:26.960 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:26.960 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:26.960 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:26.960 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.960 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.960 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.960 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.961 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.961 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.961 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.961 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.961 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.961 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.961 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.961 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.961 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.961 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.961 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.961 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.962 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.962 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.962 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.962 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:26.962 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:26.962 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.962 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.962 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.962 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.962 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.962 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.962 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.962 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.963 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.963 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.963 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.963 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.963 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:26.963 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:26.963 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.963 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.963 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.963 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:26.963 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.963 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.963 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.963 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.964 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.964 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.964 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.964 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.964 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.964 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.964 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.964 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.964 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.964 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.964 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.964 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:26.964 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:26.965 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.965 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.965 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.965 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.965 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.965 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.965 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.965 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.965 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.965 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.965 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.965 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.965 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.965 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.966 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:26.966 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirroring", 2024-03-21T10:50:26.966 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:26.966 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:26.966 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:26.966 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:26.966 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:26.966 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.966 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.966 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.966 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.966 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.966 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.966 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.967 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.967 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.967 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.967 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.967 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.967 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.967 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.967 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.967 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.967 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.967 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.967 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.967 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:26.967 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:26.968 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.968 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.968 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.968 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.968 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.968 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.968 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.968 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.968 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.968 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.968 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.968 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.968 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:26.968 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:26.969 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.969 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.969 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.969 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:26.969 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.969 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.969 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.969 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.969 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.969 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.969 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.969 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.969 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.970 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.970 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.970 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.970 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.970 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.970 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.970 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:26.970 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:26.970 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.970 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.970 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.970 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.970 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.971 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.971 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.971 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.971 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.971 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.971 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.971 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.971 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.971 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.971 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:26.971 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "nfs", 2024-03-21T10:50:26.971 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:26.971 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:26.971 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:26.972 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:26.972 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:26.972 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.972 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.972 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.972 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.972 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.972 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.972 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.972 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.972 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.972 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.972 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.972 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.972 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.973 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.973 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.973 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.973 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.973 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.973 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.973 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:26.973 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:26.973 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.973 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.973 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.973 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.973 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.973 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.974 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.974 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.974 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.974 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.974 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.974 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.974 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:26.974 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:26.974 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.974 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.974 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.974 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:26.974 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.974 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.975 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.975 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.975 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.975 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.975 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.975 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.975 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.975 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.975 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.975 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.975 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.975 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.975 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.975 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:26.976 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:26.976 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.976 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.976 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.976 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.976 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.976 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.976 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.976 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.976 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.976 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.976 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.976 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.976 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.976 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.977 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:26.977 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-03-21T10:50:26.977 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:26.977 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:26.977 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:26.977 INFO:teuthology.orchestra.run.smithi031.stdout: "fail_fs": { 2024-03-21T10:50:26.977 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "fail_fs", 2024-03-21T10:50:26.977 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.977 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.977 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:26.977 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.977 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.977 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.977 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.978 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-21T10:50:26.978 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.978 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.978 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.978 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.978 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:26.978 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:26.978 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.978 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.978 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.978 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.978 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.978 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.978 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.979 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.979 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.979 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.979 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.979 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.979 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.979 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.979 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.979 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.979 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.979 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.979 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.979 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:26.979 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:26.979 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.980 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.980 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.980 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.980 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.980 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.980 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.980 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.980 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.980 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.980 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.980 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.980 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:26.980 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:26.980 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.981 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.981 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.981 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:26.981 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.981 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.981 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.981 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.981 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.981 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.981 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.981 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.981 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.981 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.981 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.982 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.982 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.982 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.982 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.982 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:26.982 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:26.982 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.982 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.982 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.982 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.982 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.982 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.982 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.982 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.983 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.983 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.983 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.983 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator": { 2024-03-21T10:50:26.983 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-03-21T10:50:26.983 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.983 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.983 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.983 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.983 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.983 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.983 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.984 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm", 2024-03-21T10:50:26.984 INFO:teuthology.orchestra.run.smithi031.stdout: "rook", 2024-03-21T10:50:26.984 INFO:teuthology.orchestra.run.smithi031.stdout: "test_orchestrator" 2024-03-21T10:50:26.984 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.984 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Orchestrator backend", 2024-03-21T10:50:26.984 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.984 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.984 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.984 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.984 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.984 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:26.984 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_perf_query", 2024-03-21T10:50:26.984 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:26.985 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:26.985 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:26.985 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:26.985 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:26.985 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.985 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.985 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.985 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.985 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.985 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.985 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.985 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.985 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.985 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.986 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.986 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.986 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.986 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.986 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.986 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.986 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.986 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.986 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.986 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:26.986 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:26.986 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.986 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.987 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.987 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.987 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.987 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.987 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.987 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.987 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.987 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.987 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.987 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.987 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:26.987 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:26.987 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.987 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.987 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.988 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:26.988 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.988 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.988 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.988 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.988 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.988 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.988 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.988 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.988 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.988 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.988 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.988 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.988 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.988 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.989 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.989 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:26.989 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:26.989 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.989 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.989 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.989 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.989 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.989 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.989 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.989 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.989 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.989 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.989 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.990 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.990 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.990 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.990 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:26.990 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_support", 2024-03-21T10:50:26.990 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:26.990 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:26.990 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:26.990 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:26.990 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:26.990 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.990 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.990 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.990 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.990 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.991 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.991 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.991 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.991 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.991 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.991 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.991 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.991 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.991 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.991 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.991 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.991 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.991 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.991 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.991 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:26.992 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:26.992 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.992 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.992 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.992 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.992 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.992 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.992 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.992 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.992 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.992 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.992 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.992 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.992 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:26.992 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:26.993 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.993 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.993 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.993 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:26.993 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.993 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.993 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.993 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.993 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.993 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.993 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.993 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.993 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.993 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.993 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.994 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.994 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.994 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.994 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.994 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:26.994 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:26.994 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.994 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.994 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.994 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.994 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.994 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.994 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.994 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.994 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.995 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.995 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.995 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.995 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:26.995 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.995 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:26.995 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_autoscaler", 2024-03-21T10:50:26.995 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:26.995 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:26.995 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:26.995 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:26.995 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:26.995 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.995 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.995 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.996 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:26.996 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.996 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.996 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.996 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.996 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.996 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.996 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.996 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.996 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.996 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.996 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.996 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.996 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.996 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.997 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.997 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:26.997 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:26.997 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:26.997 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.997 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.997 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:26.997 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.997 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.997 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:26.997 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.997 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.997 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.998 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.998 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.998 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:26.998 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:26.998 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:26.998 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:26.998 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:26.998 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:26.998 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:26.998 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:26.998 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:26.998 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:26.999 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:26.999 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:26.999 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:26.999 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:26.999 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:26.999 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:26.999 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:26.999 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:26.999 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:26.999 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:26.999 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:26.999 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:26.999 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:26.999 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.000 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.000 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.000 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.000 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.000 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.000 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.000 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.000 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.000 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.000 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.000 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.000 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-03-21T10:50:27.000 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-03-21T10:50:27.000 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.000 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.001 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.001 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-03-21T10:50:27.001 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.001 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.001 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.001 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.001 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.001 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.001 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.001 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.001 INFO:teuthology.orchestra.run.smithi031.stdout: "threshold": { 2024-03-21T10:50:27.001 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threshold", 2024-03-21T10:50:27.001 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-03-21T10:50:27.001 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.002 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.002 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-03-21T10:50:27.002 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1.0", 2024-03-21T10:50:27.002 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.002 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.002 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "scaling threshold", 2024-03-21T10:50:27.002 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-21T10:50:27.002 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.002 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.002 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.002 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.002 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.002 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.003 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "progress", 2024-03-21T10:50:27.003 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.003 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.003 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.003 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_pg_recovery_event": { 2024-03-21T10:50:27.003 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_pg_recovery_event", 2024-03-21T10:50:27.003 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.003 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.003 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.003 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.003 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.003 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.003 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.003 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-21T10:50:27.003 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.004 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.004 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.004 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.004 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-03-21T10:50:27.004 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-03-21T10:50:27.004 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.004 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.004 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.004 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.004 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.004 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.004 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.004 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.004 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.005 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.005 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.005 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.005 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.005 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.005 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.005 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.005 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.005 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.005 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.005 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.005 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.005 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.005 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.005 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.006 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.006 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.006 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.006 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.006 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.006 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.006 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.006 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.006 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.006 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.006 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.006 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.006 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.006 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.006 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.007 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.007 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.007 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.007 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.007 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.007 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.007 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.007 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.007 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.007 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.007 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.007 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.007 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.009 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.009 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.009 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.009 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.009 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.009 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.009 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.009 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.009 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.009 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.009 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.009 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.009 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.009 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.009 INFO:teuthology.orchestra.run.smithi031.stdout: "max_completed_events": { 2024-03-21T10:50:27.010 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_completed_events", 2024-03-21T10:50:27.010 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.010 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.010 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.010 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "50", 2024-03-21T10:50:27.010 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.010 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.010 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.010 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "number of past completed events to remember", 2024-03-21T10:50:27.010 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.010 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.010 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.011 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-03-21T10:50:27.011 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-03-21T10:50:27.011 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.011 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.011 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.011 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-03-21T10:50:27.011 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.011 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.011 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.011 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long the module is going to sleep", 2024-03-21T10:50:27.011 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.011 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.011 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.011 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.012 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.012 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.012 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.012 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus", 2024-03-21T10:50:27.012 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.012 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.012 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.012 INFO:teuthology.orchestra.run.smithi031.stdout: "cache": { 2024-03-21T10:50:27.012 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cache", 2024-03-21T10:50:27.012 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.012 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.012 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.012 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.012 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.013 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.013 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.013 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.013 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.013 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.013 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.013 INFO:teuthology.orchestra.run.smithi031.stdout: "exclude_perf_counters": { 2024-03-21T10:50:27.013 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "exclude_perf_counters", 2024-03-21T10:50:27.013 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.013 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.013 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.013 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.013 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.013 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.014 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.014 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-21T10:50:27.014 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-21T10:50:27.014 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.014 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.014 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.014 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.014 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.014 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.014 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.014 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.014 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.015 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.015 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.015 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.015 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.015 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.015 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.015 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.015 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.015 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.015 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.015 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.015 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.015 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.015 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.015 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.016 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.016 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.016 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.016 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.016 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.016 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.016 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.016 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.016 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.016 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.016 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.016 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.016 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.016 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.017 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.017 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.017 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.017 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.017 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.017 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.017 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.017 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.017 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.017 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.017 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.017 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.017 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.017 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.017 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.018 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.018 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.018 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.018 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.018 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.018 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.018 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.018 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.018 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.018 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.018 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.018 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.018 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.018 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.019 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.019 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.019 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.019 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.019 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.019 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.019 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools": { 2024-03-21T10:50:27.019 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools", 2024-03-21T10:50:27.019 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.019 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.019 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.019 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.019 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.019 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.019 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.020 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.020 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.020 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.020 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.020 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.020 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-21T10:50:27.020 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-21T10:50:27.020 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.020 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.020 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.020 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "300", 2024-03-21T10:50:27.020 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.020 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.020 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.020 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.021 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.021 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.021 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.021 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.021 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_interval": { 2024-03-21T10:50:27.021 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_interval", 2024-03-21T10:50:27.021 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-03-21T10:50:27.021 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.021 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.021 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15.0", 2024-03-21T10:50:27.021 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.021 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.021 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.021 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.022 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.022 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.022 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.022 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.022 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-03-21T10:50:27.022 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-03-21T10:50:27.022 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.022 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.022 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.022 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-03-21T10:50:27.022 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.022 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.022 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.023 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-21T10:50:27.023 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.023 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.023 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.023 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.023 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-03-21T10:50:27.023 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-03-21T10:50:27.023 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.023 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.023 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.023 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "9283", 2024-03-21T10:50:27.024 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.024 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.024 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.024 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-21T10:50:27.024 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.024 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.024 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.024 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.024 INFO:teuthology.orchestra.run.smithi031.stdout: "stale_cache_strategy": { 2024-03-21T10:50:27.024 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stale_cache_strategy", 2024-03-21T10:50:27.024 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.024 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.024 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.024 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "log", 2024-03-21T10:50:27.024 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.025 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.025 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.025 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.025 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.025 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.025 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.025 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.025 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-03-21T10:50:27.025 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-03-21T10:50:27.025 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.025 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.025 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.025 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "default", 2024-03-21T10:50:27.025 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.026 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.026 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.026 INFO:teuthology.orchestra.run.smithi031.stdout: "default", 2024-03-21T10:50:27.026 INFO:teuthology.orchestra.run.smithi031.stdout: "error" 2024-03-21T10:50:27.026 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.026 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.026 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.026 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.026 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.026 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.026 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-03-21T10:50:27.026 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-03-21T10:50:27.026 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.026 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.027 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.027 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-03-21T10:50:27.027 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-03-21T10:50:27.027 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-03-21T10:50:27.027 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.027 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.027 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.027 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.027 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.027 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.027 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.027 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.027 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.027 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_support", 2024-03-21T10:50:27.028 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.028 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.028 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.028 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.028 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.028 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.028 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.028 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.028 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.028 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.028 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.028 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.028 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.028 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.029 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.029 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.029 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.029 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.029 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.029 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.029 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.029 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.029 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.029 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.029 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.029 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.029 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.029 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.030 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.030 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.030 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.030 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.030 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.030 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.030 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.030 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.030 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.030 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.030 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.030 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.030 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.030 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.031 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.031 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.031 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.031 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.031 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.031 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.031 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.031 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.031 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.031 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.031 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.031 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.031 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.031 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.031 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.032 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.032 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.032 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.032 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.032 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.032 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.032 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.032 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.032 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.032 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.032 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.032 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.032 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.032 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.032 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.033 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.033 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_snap_create": { 2024-03-21T10:50:27.033 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_snap_create", 2024-03-21T10:50:27.033 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.033 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.033 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.033 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-03-21T10:50:27.033 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.033 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.033 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.033 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.033 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.033 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.033 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.034 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.034 INFO:teuthology.orchestra.run.smithi031.stdout: "mirror_snapshot_schedule": { 2024-03-21T10:50:27.034 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirror_snapshot_schedule", 2024-03-21T10:50:27.034 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.034 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.034 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.034 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.034 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.034 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.034 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.034 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.034 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.034 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.034 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.035 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.035 INFO:teuthology.orchestra.run.smithi031.stdout: "trash_purge_schedule": { 2024-03-21T10:50:27.035 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "trash_purge_schedule", 2024-03-21T10:50:27.035 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.035 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.035 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.035 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.035 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.035 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.035 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.035 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.035 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.035 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.035 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.035 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.036 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.036 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.036 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.036 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "restful", 2024-03-21T10:50:27.036 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.036 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.036 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.036 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_auth": { 2024-03-21T10:50:27.036 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_auth", 2024-03-21T10:50:27.036 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.036 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.036 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.036 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.036 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.037 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.037 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.037 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.037 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.037 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.037 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.037 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.037 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-03-21T10:50:27.037 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-03-21T10:50:27.038 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.038 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.038 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.038 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.038 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.038 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.038 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.038 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.038 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.038 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.038 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.038 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.038 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.038 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.039 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.039 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.039 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.039 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.039 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.039 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.039 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.039 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.039 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.039 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.039 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.039 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.039 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.039 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.039 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.040 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.040 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.040 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.040 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.040 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.040 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.040 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.040 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.040 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.040 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.040 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.040 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.040 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.040 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.041 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.041 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.041 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.041 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.041 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.041 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.041 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.041 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.041 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.041 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.041 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.041 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.041 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.041 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.041 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.042 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.042 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.042 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.042 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.042 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.042 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.042 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.042 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.042 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.042 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.042 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.042 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.042 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.042 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.043 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.043 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.043 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.043 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.043 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.043 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.043 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.043 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.043 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.043 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.043 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-03-21T10:50:27.043 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-03-21T10:50:27.043 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.043 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.043 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.044 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.044 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.044 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.044 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.044 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.044 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.044 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.044 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.044 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.044 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-03-21T10:50:27.044 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-03-21T10:50:27.044 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.044 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.044 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.045 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.045 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.045 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.045 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.045 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.045 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.045 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.045 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.045 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.045 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.045 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.045 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.045 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rgw", 2024-03-21T10:50:27.045 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.045 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.046 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.046 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.046 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.046 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.046 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.046 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.046 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.046 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.046 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.046 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.046 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.046 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.046 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.046 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.047 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.047 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.047 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.047 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.047 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.047 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.047 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.047 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.047 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.047 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.047 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.047 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.047 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.047 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.047 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.048 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.048 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.048 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.048 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.048 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.048 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.048 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.048 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.048 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.048 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.048 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.048 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.048 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.049 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.049 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.049 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.049 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.049 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.049 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.049 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.049 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.049 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.049 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.049 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.049 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.049 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.049 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.050 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.050 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.050 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.050 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.050 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.050 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.050 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.050 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.050 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.050 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.050 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.050 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.050 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.051 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.051 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.051 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.051 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.051 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rook", 2024-03-21T10:50:27.051 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.051 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.051 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.051 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.051 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.051 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.051 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.051 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.052 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.052 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.052 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.052 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.052 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.052 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.052 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.052 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.052 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.052 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.052 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.052 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.052 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.052 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.053 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.053 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.053 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.053 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.053 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.053 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.053 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.053 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.053 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.053 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.053 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.053 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.053 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.053 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.053 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.054 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.054 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.054 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.054 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.054 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.054 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.054 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.054 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.054 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.054 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.054 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.054 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.054 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.055 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.055 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.055 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.055 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.055 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.055 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.055 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.055 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.055 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.055 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.055 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.055 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.055 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.055 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.055 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.056 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.056 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.056 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.056 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.056 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.056 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.056 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.056 INFO:teuthology.orchestra.run.smithi031.stdout: "storage_class": { 2024-03-21T10:50:27.056 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "storage_class", 2024-03-21T10:50:27.056 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.056 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.056 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.056 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "local", 2024-03-21T10:50:27.056 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.057 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.057 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.057 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-21T10:50:27.057 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.057 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.057 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.057 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.057 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.057 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.057 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.057 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "selftest", 2024-03-21T10:50:27.057 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.057 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.057 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.058 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.058 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.058 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.058 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.058 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.058 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.058 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.058 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.058 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.058 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.058 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.058 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.058 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.058 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.058 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.059 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.059 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.059 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.059 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.059 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.059 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.059 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.059 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.059 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.059 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.059 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.059 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.059 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.059 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.059 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.060 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.060 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.060 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.060 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.060 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.060 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.060 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.060 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.060 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.060 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.060 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.060 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.060 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.060 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.061 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.061 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.061 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.061 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.061 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.061 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.061 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.061 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.061 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.061 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.061 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.061 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.061 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.061 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.061 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.062 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.062 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.062 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.062 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.062 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.062 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.062 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.062 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.062 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.062 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.062 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.062 INFO:teuthology.orchestra.run.smithi031.stdout: "roption1": { 2024-03-21T10:50:27.062 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption1", 2024-03-21T10:50:27.062 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.063 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.063 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.063 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.063 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.063 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.063 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.063 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.063 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.063 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.063 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.063 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.063 INFO:teuthology.orchestra.run.smithi031.stdout: "roption2": { 2024-03-21T10:50:27.063 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption2", 2024-03-21T10:50:27.063 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.063 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.064 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.064 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "xyz", 2024-03-21T10:50:27.064 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.064 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.064 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.064 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.064 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.064 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.064 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.064 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.064 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption1": { 2024-03-21T10:50:27.064 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption1", 2024-03-21T10:50:27.064 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.064 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.064 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.065 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.065 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.065 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.065 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.065 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.065 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.065 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.065 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.065 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.065 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption2": { 2024-03-21T10:50:27.065 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption2", 2024-03-21T10:50:27.065 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.066 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.066 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.066 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.066 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.066 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.066 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.066 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.066 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.066 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.066 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.066 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.066 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption3": { 2024-03-21T10:50:27.066 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption3", 2024-03-21T10:50:27.066 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-03-21T10:50:27.067 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.067 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.067 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.067 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.067 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.067 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.067 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.067 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.067 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.067 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.067 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.067 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption4": { 2024-03-21T10:50:27.067 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption4", 2024-03-21T10:50:27.067 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.068 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.068 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.068 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.068 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.068 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.068 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.068 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.068 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.068 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.068 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.068 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.068 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption5": { 2024-03-21T10:50:27.068 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption5", 2024-03-21T10:50:27.068 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.068 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.069 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.069 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.069 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.069 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.069 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.069 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.069 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.069 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.069 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.069 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.069 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption6": { 2024-03-21T10:50:27.069 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption6", 2024-03-21T10:50:27.069 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.069 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.070 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.070 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.070 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.070 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.070 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.070 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.071 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.071 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.071 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.071 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.071 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption7": { 2024-03-21T10:50:27.071 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption7", 2024-03-21T10:50:27.071 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.071 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.071 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.071 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.071 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-03-21T10:50:27.071 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "42", 2024-03-21T10:50:27.071 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.072 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.072 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.072 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.072 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.072 INFO:teuthology.orchestra.run.smithi031.stdout: "testkey": { 2024-03-21T10:50:27.072 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testkey", 2024-03-21T10:50:27.072 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.072 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.072 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.072 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.072 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.072 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.072 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.072 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.073 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.073 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.073 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.073 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.073 INFO:teuthology.orchestra.run.smithi031.stdout: "testlkey": { 2024-03-21T10:50:27.073 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testlkey", 2024-03-21T10:50:27.073 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.073 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.073 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.073 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.073 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.073 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.073 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.073 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.073 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.074 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.074 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.074 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.074 INFO:teuthology.orchestra.run.smithi031.stdout: "testnewline": { 2024-03-21T10:50:27.074 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testnewline", 2024-03-21T10:50:27.074 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.074 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.074 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.074 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.074 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.074 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.074 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.074 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.074 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.074 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.075 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.075 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.075 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.075 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.075 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.075 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snap_schedule", 2024-03-21T10:50:27.075 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.075 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.075 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.075 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_m_granularity": { 2024-03-21T10:50:27.075 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_m_granularity", 2024-03-21T10:50:27.075 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.075 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.075 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.076 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.076 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.076 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.076 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.076 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow minute scheduled snapshots", 2024-03-21T10:50:27.076 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.076 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.076 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.076 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.076 INFO:teuthology.orchestra.run.smithi031.stdout: "dump_on_update": { 2024-03-21T10:50:27.076 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dump_on_update", 2024-03-21T10:50:27.076 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.076 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.076 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.076 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.077 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.077 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.077 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.077 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "dump database to debug log on update", 2024-03-21T10:50:27.077 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.077 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.077 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.077 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.077 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.077 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.077 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.077 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.077 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.077 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.078 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.078 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.078 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.078 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.078 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.078 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.078 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.078 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.078 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.078 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.078 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.078 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.078 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.078 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.079 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.079 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.079 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.079 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.079 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.079 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.079 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.079 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.079 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.079 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.079 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.079 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.079 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.080 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.080 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.080 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.080 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.080 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.080 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.080 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.080 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.080 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.080 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.080 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.080 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.080 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.080 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.081 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.081 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.081 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.081 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.081 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.081 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.081 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.081 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.081 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.081 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.081 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.081 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.081 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.081 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.082 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.082 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.082 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.082 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.082 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.082 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.082 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.082 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.082 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.082 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.082 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.082 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.082 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stats", 2024-03-21T10:50:27.082 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.083 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.083 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.083 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.083 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.083 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.083 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.083 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.083 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.083 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.083 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.083 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.083 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.083 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.083 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.084 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.084 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.084 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.084 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.084 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.084 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.084 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.084 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.084 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.084 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.084 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.084 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.084 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.084 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.085 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.085 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.085 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.085 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.085 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.085 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.085 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.085 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.085 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.085 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.085 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.085 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.085 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.085 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.085 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.086 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.086 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.086 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.086 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.086 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.086 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.086 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.086 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.086 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.086 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.086 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.086 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.086 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.086 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.087 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.087 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.087 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.087 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.087 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.087 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.087 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.087 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.087 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.087 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.087 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.087 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.087 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.087 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.087 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.088 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.088 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.088 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.088 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "status", 2024-03-21T10:50:27.088 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.088 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.088 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.088 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.088 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.088 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.088 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.088 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.088 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.088 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.088 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.089 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.089 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.089 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.089 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.089 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.089 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.089 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.089 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.089 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.089 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.089 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.089 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.089 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.089 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.090 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.090 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.090 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.090 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.090 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.090 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.090 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.090 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.090 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.090 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.090 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.090 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.090 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.090 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.090 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.091 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.091 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.091 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.091 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.091 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.091 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.091 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.091 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.091 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.091 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.091 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.091 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.091 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.091 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.091 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.092 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.092 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.092 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.092 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.092 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.092 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.092 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.092 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.092 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.092 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.092 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.092 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.092 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.092 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.093 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.093 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.093 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.093 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.093 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.093 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.093 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.093 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telegraf", 2024-03-21T10:50:27.093 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.093 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.093 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.093 INFO:teuthology.orchestra.run.smithi031.stdout: "address": { 2024-03-21T10:50:27.093 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "address", 2024-03-21T10:50:27.094 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.094 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.094 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.094 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-21T10:50:27.094 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.094 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.094 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.094 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.094 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.094 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.094 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.094 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.094 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-03-21T10:50:27.094 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-03-21T10:50:27.095 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.095 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.095 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.095 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15", 2024-03-21T10:50:27.095 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.095 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.095 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.095 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.095 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.095 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.095 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.095 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.095 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.096 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.096 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.096 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.096 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.096 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.096 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.096 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.096 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.096 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.096 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.096 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.096 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.096 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.097 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.097 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.097 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.097 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.097 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.097 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.097 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.097 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.097 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.097 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.097 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.097 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.097 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.097 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.098 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.098 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.098 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.098 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.098 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.098 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.098 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.098 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.098 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.098 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.098 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.098 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.098 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.098 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.099 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.099 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.099 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.099 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.099 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.099 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.099 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.099 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.099 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.099 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.099 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.099 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.099 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.099 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.100 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.100 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.100 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.100 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.100 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.100 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.100 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.100 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.100 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.100 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.100 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.100 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.100 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.100 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.100 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.101 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.101 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.101 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telemetry", 2024-03-21T10:50:27.101 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.101 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.101 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.101 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_basic": { 2024-03-21T10:50:27.101 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_basic", 2024-03-21T10:50:27.101 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.101 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.101 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.101 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.101 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.101 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.101 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.102 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-21T10:50:27.102 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.102 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.102 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.102 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.102 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_crash": { 2024-03-21T10:50:27.102 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_crash", 2024-03-21T10:50:27.102 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.102 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.102 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.102 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.102 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.102 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.102 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.103 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-21T10:50:27.103 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.103 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.103 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.103 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.103 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_device": { 2024-03-21T10:50:27.103 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_device", 2024-03-21T10:50:27.103 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.103 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.103 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.103 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.103 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.103 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.103 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.104 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-21T10:50:27.104 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.104 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.104 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.104 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.104 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_ident": { 2024-03-21T10:50:27.104 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_ident", 2024-03-21T10:50:27.104 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.104 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.104 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.104 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.104 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.104 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.104 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.104 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-21T10:50:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_perf": { 2024-03-21T10:50:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_perf", 2024-03-21T10:50:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-21T10:50:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.106 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.106 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.106 INFO:teuthology.orchestra.run.smithi031.stdout: "contact": { 2024-03-21T10:50:27.106 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "contact", 2024-03-21T10:50:27.106 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.106 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.106 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.106 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.106 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.106 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.106 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.106 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.107 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.107 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.107 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.107 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.107 INFO:teuthology.orchestra.run.smithi031.stdout: "description": { 2024-03-21T10:50:27.107 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "description", 2024-03-21T10:50:27.107 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.107 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.107 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.107 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.107 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.107 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.107 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.107 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.107 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.108 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.108 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.108 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.108 INFO:teuthology.orchestra.run.smithi031.stdout: "device_url": { 2024-03-21T10:50:27.108 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_url", 2024-03-21T10:50:27.108 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.108 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.108 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.108 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-21T10:50:27.108 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.108 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.108 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.108 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.108 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.109 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.109 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.109 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.109 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-03-21T10:50:27.109 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-03-21T10:50:27.109 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.109 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.109 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.109 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.109 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.109 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.109 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.109 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.109 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.109 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.110 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.110 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.110 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-03-21T10:50:27.110 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-03-21T10:50:27.110 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.110 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.110 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.110 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "24", 2024-03-21T10:50:27.110 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "8", 2024-03-21T10:50:27.110 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.110 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.110 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.110 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.110 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.111 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.111 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.111 INFO:teuthology.orchestra.run.smithi031.stdout: "last_opt_revision": { 2024-03-21T10:50:27.111 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "last_opt_revision", 2024-03-21T10:50:27.111 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.111 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.111 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.111 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1", 2024-03-21T10:50:27.111 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.111 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.111 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.111 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.111 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.112 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.112 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.112 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.112 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard": { 2024-03-21T10:50:27.112 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard", 2024-03-21T10:50:27.112 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.112 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.112 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.112 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.112 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.112 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.112 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.112 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.112 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.112 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.113 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.113 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.113 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard_description": { 2024-03-21T10:50:27.113 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard_description", 2024-03-21T10:50:27.113 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.113 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.113 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.113 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.113 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.113 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.113 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.113 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.113 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.113 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.113 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.114 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.114 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.114 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.114 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.114 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.114 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.114 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.114 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.114 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.114 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.114 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.114 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.114 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.114 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.115 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.115 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.115 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.115 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.115 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.115 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.115 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.115 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.115 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.115 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.115 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.115 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.115 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.115 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.115 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.116 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.116 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.116 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.116 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.116 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.116 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.116 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.116 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.116 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.116 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.116 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.116 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.116 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.116 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.117 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.117 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.117 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.117 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.117 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.117 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.117 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.117 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.117 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.117 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.117 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.117 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.117 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.117 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.118 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.118 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.118 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.118 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.118 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.118 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.118 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.118 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.118 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.118 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.118 INFO:teuthology.orchestra.run.smithi031.stdout: "organization": { 2024-03-21T10:50:27.118 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "organization", 2024-03-21T10:50:27.118 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.118 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.119 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.119 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.119 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.119 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.119 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.119 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.119 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.119 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.119 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.119 INFO:teuthology.orchestra.run.smithi031.stdout: "proxy": { 2024-03-21T10:50:27.119 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "proxy", 2024-03-21T10:50:27.119 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.120 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.120 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.120 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.120 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.120 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.120 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.120 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.120 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.120 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.120 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.120 INFO:teuthology.orchestra.run.smithi031.stdout: "url": { 2024-03-21T10:50:27.120 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url", 2024-03-21T10:50:27.120 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.120 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.121 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.121 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-21T10:50:27.121 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.121 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.121 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.121 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.121 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.121 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.121 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.121 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.121 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.121 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.121 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test_orchestrator", 2024-03-21T10:50:27.122 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.122 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.122 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.122 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.122 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.122 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.122 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.122 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.122 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.122 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.122 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.122 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.122 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.122 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.123 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.123 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.123 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.123 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.123 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.123 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.123 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.123 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.123 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.123 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.123 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.123 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.123 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.124 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.124 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.124 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.124 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.124 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.124 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.124 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.124 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.124 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.124 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.124 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.124 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.124 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.125 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.125 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.125 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.125 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.125 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.125 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.125 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.125 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.125 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.125 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.125 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.125 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.125 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.125 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.126 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.126 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.126 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.126 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.126 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.126 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.126 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.126 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.126 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.126 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.126 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.126 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.126 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.126 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.127 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.127 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.127 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.127 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2024-03-21T10:50:27.127 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.127 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.127 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.127 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.127 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.127 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.127 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.127 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.127 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.128 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.128 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.128 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.128 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.128 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.128 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.128 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.128 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.128 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.128 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.128 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.128 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.128 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.128 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.129 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.129 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.129 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.129 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.129 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.129 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.129 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.129 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.129 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.129 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.129 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.129 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.129 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.129 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.130 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.130 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.130 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.130 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.130 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.130 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.130 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.130 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.130 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.130 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.130 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.130 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.130 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.130 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.131 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.131 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.131 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.131 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.131 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.131 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.131 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.131 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.131 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.131 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.131 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.131 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.131 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.131 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.131 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.132 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.132 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.132 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.132 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_clones": { 2024-03-21T10:50:27.132 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_clones", 2024-03-21T10:50:27.132 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.132 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.132 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.132 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4", 2024-03-21T10:50:27.132 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.132 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.132 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.132 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-21T10:50:27.132 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.133 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.133 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.133 INFO:teuthology.orchestra.run.smithi031.stdout: "periodic_async_work": { 2024-03-21T10:50:27.133 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "periodic_async_work", 2024-03-21T10:50:27.133 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.133 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.133 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.133 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.133 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.133 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.133 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.134 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Periodically check for async work", 2024-03-21T10:50:27.134 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.134 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.134 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.134 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_delay": { 2024-03-21T10:50:27.134 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_delay", 2024-03-21T10:50:27.134 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.134 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.134 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.134 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-03-21T10:50:27.134 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.135 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.135 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.135 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-21T10:50:27.135 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.135 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.135 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.135 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_no_wait": { 2024-03-21T10:50:27.135 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_no_wait", 2024-03-21T10:50:27.135 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.135 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.135 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.135 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.135 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.136 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.136 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.136 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-21T10:50:27.136 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.136 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.136 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.136 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.136 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.136 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix", 2024-03-21T10:50:27.136 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.136 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.136 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.136 INFO:teuthology.orchestra.run.smithi031.stdout: "discovery_interval": { 2024-03-21T10:50:27.137 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "discovery_interval", 2024-03-21T10:50:27.137 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-03-21T10:50:27.137 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.137 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.137 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "100", 2024-03-21T10:50:27.137 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.137 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.137 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.137 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.137 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.137 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.137 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.137 INFO:teuthology.orchestra.run.smithi031.stdout: "identifier": { 2024-03-21T10:50:27.137 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "identifier", 2024-03-21T10:50:27.138 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.138 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.138 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.138 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.138 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.138 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.138 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.138 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.138 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.138 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.138 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.138 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-03-21T10:50:27.139 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-03-21T10:50:27.139 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.139 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.139 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.139 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-03-21T10:50:27.139 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.139 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.139 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.139 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.139 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.139 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.139 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.139 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.139 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.140 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.140 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.140 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.140 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.140 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.140 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.140 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.140 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.140 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.140 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.140 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.140 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.140 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.140 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.141 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.141 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.141 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.141 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.141 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.141 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.141 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.141 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.141 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.141 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.141 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.141 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.141 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.141 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.142 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.142 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.142 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.142 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.142 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.142 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.142 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.142 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.142 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.142 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.142 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.142 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.142 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.143 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.143 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.143 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.143 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.143 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.143 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.143 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.143 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.143 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.143 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.143 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.143 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.143 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.143 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.144 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.144 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.144 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.144 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.144 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.144 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.144 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.144 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.144 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.144 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_host": { 2024-03-21T10:50:27.144 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_host", 2024-03-21T10:50:27.144 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.144 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.144 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.145 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.145 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.145 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.145 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.145 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.145 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.145 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.145 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.145 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_port": { 2024-03-21T10:50:27.145 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_port", 2024-03-21T10:50:27.145 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.145 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.145 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.145 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10051", 2024-03-21T10:50:27.146 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.146 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.146 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.146 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.146 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.146 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.146 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.146 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_sender": { 2024-03-21T10:50:27.146 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_sender", 2024-03-21T10:50:27.146 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.146 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.147 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.147 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-21T10:50:27.147 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.147 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.147 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.147 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.147 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.147 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.147 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.147 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.147 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.147 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.147 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-03-21T10:50:27.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.148 INFO:teuthology.orchestra.run.smithi031.stdout: "gid": 25561, 2024-03-21T10:50:27.148 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "z", 2024-03-21T10:50:27.148 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_features": 4540701547738038271, 2024-03-21T10:50:27.148 INFO:teuthology.orchestra.run.smithi031.stdout: "available_modules": [ 2024-03-21T10:50:27.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.148 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "alerts", 2024-03-21T10:50:27.148 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.148 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.148 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.148 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-03-21T10:50:27.148 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-03-21T10:50:27.148 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.148 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.148 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.149 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-03-21T10:50:27.149 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.149 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.149 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.149 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How frequently to reexamine health status", 2024-03-21T10:50:27.149 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.149 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.149 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.149 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.149 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.149 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.149 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.150 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.150 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.150 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.150 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.150 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.150 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.150 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.150 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.150 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.150 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.150 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.150 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.150 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.150 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.151 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.151 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.151 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.151 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.151 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.151 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.151 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.151 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.151 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.151 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.151 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.151 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.151 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.152 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.152 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.152 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.152 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.152 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.152 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.152 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.152 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.152 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.152 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.152 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.152 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.153 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.153 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.153 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.153 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.153 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.153 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.153 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.153 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.153 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.153 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.153 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.153 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.153 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.153 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.154 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.154 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.154 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.154 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.154 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.154 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.154 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.154 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.154 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.154 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_destination": { 2024-03-21T10:50:27.154 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_destination", 2024-03-21T10:50:27.154 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.154 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.155 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.155 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.155 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.155 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.155 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.155 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email address to send alerts to", 2024-03-21T10:50:27.155 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.155 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.155 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.155 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_from_name": { 2024-03-21T10:50:27.155 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_from_name", 2024-03-21T10:50:27.155 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.155 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.156 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.156 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "Ceph", 2024-03-21T10:50:27.156 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.156 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.156 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.156 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email From: name", 2024-03-21T10:50:27.156 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.156 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.156 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.156 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_host": { 2024-03-21T10:50:27.156 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_host", 2024-03-21T10:50:27.156 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.156 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.156 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.157 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.157 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.157 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.157 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.157 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP server", 2024-03-21T10:50:27.157 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.157 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.157 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.157 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_password": { 2024-03-21T10:50:27.157 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_password", 2024-03-21T10:50:27.157 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.157 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.157 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.157 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.158 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.158 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.158 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.158 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Password to authenticate with", 2024-03-21T10:50:27.158 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.158 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.158 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.158 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_port": { 2024-03-21T10:50:27.158 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_port", 2024-03-21T10:50:27.158 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.158 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.158 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.158 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "465", 2024-03-21T10:50:27.159 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.159 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.159 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.159 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP port", 2024-03-21T10:50:27.159 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.159 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.159 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.159 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_sender": { 2024-03-21T10:50:27.159 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_sender", 2024-03-21T10:50:27.159 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.159 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.159 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.159 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.159 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.160 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.160 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.160 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP envelope sender", 2024-03-21T10:50:27.160 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.160 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.160 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.160 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_ssl": { 2024-03-21T10:50:27.160 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_ssl", 2024-03-21T10:50:27.160 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.160 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.160 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.160 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.161 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.161 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.161 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.161 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-21T10:50:27.161 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.161 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.161 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.161 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_user": { 2024-03-21T10:50:27.161 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_user", 2024-03-21T10:50:27.161 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.161 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.161 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.161 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.161 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.162 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.162 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.162 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "User to authenticate as", 2024-03-21T10:50:27.162 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.162 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.162 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.162 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.162 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.162 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "balancer", 2024-03-21T10:50:27.162 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.162 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: "active": { 2024-03-21T10:50:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "active", 2024-03-21T10:50:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically balance PGs across cluster", 2024-03-21T10:50:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.164 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_time": { 2024-03-21T10:50:27.164 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_time", 2024-03-21T10:50:27.164 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.164 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.164 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.164 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0000", 2024-03-21T10:50:27.164 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.164 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.164 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.164 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "beginning time of day to automatically balance", 2024-03-21T10:50:27.164 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-21T10:50:27.164 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.164 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_weekday": { 2024-03-21T10:50:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_weekday", 2024-03-21T10:50:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-03-21T10:50:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-03-21T10:50:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-03-21T10:50:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-03-21T10:50:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-21T10:50:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-21T10:50:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_max_iterations": { 2024-03-21T10:50:27.166 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_max_iterations", 2024-03-21T10:50:27.166 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-03-21T10:50:27.166 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.166 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.166 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "25", 2024-03-21T10:50:27.166 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-03-21T10:50:27.166 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "250", 2024-03-21T10:50:27.166 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.166 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-21T10:50:27.166 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.166 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.166 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.167 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_metrics": { 2024-03-21T10:50:27.167 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_metrics", 2024-03-21T10:50:27.167 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.167 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.167 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.167 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "pgs,objects,bytes", 2024-03-21T10:50:27.167 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.167 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.167 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.167 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-21T10:50:27.167 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-21T10:50:27.167 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.167 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.167 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_step": { 2024-03-21T10:50:27.168 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_step", 2024-03-21T10:50:27.168 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-03-21T10:50:27.168 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.168 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.168 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.5", 2024-03-21T10:50:27.168 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0.001", 2024-03-21T10:50:27.168 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "0.999", 2024-03-21T10:50:27.168 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.168 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "aggressiveness of optimization", 2024-03-21T10:50:27.168 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-21T10:50:27.168 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.168 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.168 INFO:teuthology.orchestra.run.smithi031.stdout: "end_time": { 2024-03-21T10:50:27.168 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_time", 2024-03-21T10:50:27.169 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.169 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.169 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.169 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2359", 2024-03-21T10:50:27.169 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.169 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.169 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.169 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ending time of day to automatically balance", 2024-03-21T10:50:27.169 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-21T10:50:27.169 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.169 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.169 INFO:teuthology.orchestra.run.smithi031.stdout: "end_weekday": { 2024-03-21T10:50:27.169 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_weekday", 2024-03-21T10:50:27.169 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-03-21T10:50:27.170 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.170 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.170 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-03-21T10:50:27.170 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-03-21T10:50:27.170 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-03-21T10:50:27.170 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.170 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-21T10:50:27.170 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-21T10:50:27.170 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.170 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.170 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.170 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.170 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.170 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.171 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.171 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.171 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.171 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.171 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.171 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.171 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.171 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.171 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.171 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.171 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.171 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.171 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.171 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.171 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.172 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.172 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.172 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.172 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.172 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.172 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.172 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.172 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.172 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.172 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.172 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.172 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.173 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.173 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.173 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.173 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.173 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.173 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.173 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.173 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.173 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.173 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.173 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.173 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.173 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.174 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.174 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.174 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.174 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.174 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.174 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.174 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.174 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.174 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.174 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.174 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.174 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.174 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.175 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.175 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.175 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.175 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.175 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.175 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.175 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.175 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.175 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.175 INFO:teuthology.orchestra.run.smithi031.stdout: "min_score": { 2024-03-21T10:50:27.175 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_score", 2024-03-21T10:50:27.175 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-03-21T10:50:27.175 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.175 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.176 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-03-21T10:50:27.176 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.176 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.176 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.176 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-21T10:50:27.176 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.176 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.176 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.176 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-03-21T10:50:27.176 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-03-21T10:50:27.176 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.176 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.176 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.176 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "upmap", 2024-03-21T10:50:27.177 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.177 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.177 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.177 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-compat", 2024-03-21T10:50:27.177 INFO:teuthology.orchestra.run.smithi031.stdout: "none", 2024-03-21T10:50:27.177 INFO:teuthology.orchestra.run.smithi031.stdout: "read", 2024-03-21T10:50:27.177 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap", 2024-03-21T10:50:27.177 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap-read" 2024-03-21T10:50:27.177 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.177 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Balancer mode", 2024-03-21T10:50:27.177 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.177 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.177 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.178 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_ids": { 2024-03-21T10:50:27.178 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_ids", 2024-03-21T10:50:27.178 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.178 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.178 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.178 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.178 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.178 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.178 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.178 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-21T10:50:27.178 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.178 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.178 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.178 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-03-21T10:50:27.179 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-03-21T10:50:27.179 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.179 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.179 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.179 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-03-21T10:50:27.179 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.179 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.179 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.179 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-21T10:50:27.179 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.179 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.179 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.179 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_deviation": { 2024-03-21T10:50:27.179 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_deviation", 2024-03-21T10:50:27.180 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.180 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.180 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.180 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-03-21T10:50:27.180 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-03-21T10:50:27.180 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.180 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.180 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-21T10:50:27.180 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-21T10:50:27.180 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.180 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.180 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_optimizations": { 2024-03-21T10:50:27.180 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_optimizations", 2024-03-21T10:50:27.180 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-03-21T10:50:27.181 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.181 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.181 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-03-21T10:50:27.181 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.181 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.181 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.181 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-21T10:50:27.181 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.181 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.181 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.181 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.181 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.182 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm", 2024-03-21T10:50:27.182 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.182 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.182 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.182 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_down_multiplier": { 2024-03-21T10:50:27.182 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_down_multiplier", 2024-03-21T10:50:27.182 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-03-21T10:50:27.182 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.182 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.182 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-03-21T10:50:27.182 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.182 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.182 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.182 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-21T10:50:27.182 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.183 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.183 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.183 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_refresh_rate": { 2024-03-21T10:50:27.183 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_refresh_rate", 2024-03-21T10:50:27.183 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.183 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.183 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.183 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "20", 2024-03-21T10:50:27.183 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.183 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.183 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.183 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-21T10:50:27.183 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.184 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.184 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.184 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_starting_port": { 2024-03-21T10:50:27.184 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_starting_port", 2024-03-21T10:50:27.184 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.184 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.184 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.184 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4721", 2024-03-21T10:50:27.184 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.184 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.184 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.184 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-21T10:50:27.184 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.185 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.185 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.185 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_ptrace": { 2024-03-21T10:50:27.185 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_ptrace", 2024-03-21T10:50:27.185 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.185 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.185 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.185 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.185 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.185 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.185 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.185 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-21T10:50:27.185 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-21T10:50:27.186 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.186 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.186 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_interval": { 2024-03-21T10:50:27.186 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_interval", 2024-03-21T10:50:27.186 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.186 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.186 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.186 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-03-21T10:50:27.186 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.186 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.186 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.186 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-21T10:50:27.186 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.187 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.187 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.187 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_memory_target_ratio": { 2024-03-21T10:50:27.187 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_memory_target_ratio", 2024-03-21T10:50:27.187 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-03-21T10:50:27.187 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.187 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.187 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.7", 2024-03-21T10:50:27.187 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.187 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.187 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.187 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-21T10:50:27.187 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.188 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.188 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.188 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm_log_destination": { 2024-03-21T10:50:27.188 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm_log_destination", 2024-03-21T10:50:27.188 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.188 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.188 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.188 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.188 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.188 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.188 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.188 INFO:teuthology.orchestra.run.smithi031.stdout: "file", 2024-03-21T10:50:27.188 INFO:teuthology.orchestra.run.smithi031.stdout: "file,syslog", 2024-03-21T10:50:27.188 INFO:teuthology.orchestra.run.smithi031.stdout: "syslog" 2024-03-21T10:50:27.189 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.189 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-21T10:50:27.189 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.189 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.189 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.189 INFO:teuthology.orchestra.run.smithi031.stdout: "cgroups_split": { 2024-03-21T10:50:27.189 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cgroups_split", 2024-03-21T10:50:27.189 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.189 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.189 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.189 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.189 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.190 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.190 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.190 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-21T10:50:27.190 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.190 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.190 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.190 INFO:teuthology.orchestra.run.smithi031.stdout: "config_checks_enabled": { 2024-03-21T10:50:27.190 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_checks_enabled", 2024-03-21T10:50:27.190 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.190 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.191 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.191 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.191 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.191 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.191 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.191 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-21T10:50:27.191 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.191 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.191 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.191 INFO:teuthology.orchestra.run.smithi031.stdout: "config_dashboard": { 2024-03-21T10:50:27.191 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_dashboard", 2024-03-21T10:50:27.191 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.191 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.192 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.192 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.192 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.192 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.192 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.192 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-21T10:50:27.192 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.192 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.192 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.192 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_alertmanager": { 2024-03-21T10:50:27.192 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_alertmanager", 2024-03-21T10:50:27.192 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.192 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.193 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.193 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-21T10:50:27.193 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.193 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.193 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.193 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-03-21T10:50:27.193 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.193 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.193 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.193 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_base": { 2024-03-21T10:50:27.193 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_base", 2024-03-21T10:50:27.193 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.193 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.194 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.194 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-21T10:50:27.194 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.194 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.194 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.194 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Container image name, without the tag", 2024-03-21T10:50:27.194 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.194 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.194 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.194 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_elasticsearch": { 2024-03-21T10:50:27.194 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_elasticsearch", 2024-03-21T10:50:27.194 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.194 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.195 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.195 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-21T10:50:27.195 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.195 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.195 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.195 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "elasticsearch container image", 2024-03-21T10:50:27.195 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.195 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.195 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.195 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_grafana": { 2024-03-21T10:50:27.195 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_grafana", 2024-03-21T10:50:27.196 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.196 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.196 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.196 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-03-21T10:50:27.196 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.196 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.196 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.196 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-03-21T10:50:27.196 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.196 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.196 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.197 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_haproxy": { 2024-03-21T10:50:27.197 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_haproxy", 2024-03-21T10:50:27.197 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.197 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.197 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.197 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-21T10:50:27.197 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.197 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.197 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.197 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "HAproxy container image", 2024-03-21T10:50:27.197 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.197 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.198 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.198 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_agent": { 2024-03-21T10:50:27.198 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_agent", 2024-03-21T10:50:27.198 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.198 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.198 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.198 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-21T10:50:27.198 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.198 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.199 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.199 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger agent container image", 2024-03-21T10:50:27.199 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.199 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.199 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.199 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_collector": { 2024-03-21T10:50:27.199 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_collector", 2024-03-21T10:50:27.199 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.199 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.199 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.199 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-21T10:50:27.200 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.200 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.200 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.200 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger collector container image", 2024-03-21T10:50:27.200 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.200 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.200 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.200 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_query": { 2024-03-21T10:50:27.200 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_query", 2024-03-21T10:50:27.200 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.200 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.200 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.200 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-21T10:50:27.200 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.201 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.201 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.201 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger query container image", 2024-03-21T10:50:27.201 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.201 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.201 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.201 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_keepalived": { 2024-03-21T10:50:27.201 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_keepalived", 2024-03-21T10:50:27.201 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.201 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.201 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.201 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-21T10:50:27.201 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.202 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.202 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.202 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Keepalived container image", 2024-03-21T10:50:27.202 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.202 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.202 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.202 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_loki": { 2024-03-21T10:50:27.202 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_loki", 2024-03-21T10:50:27.202 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.202 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.202 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.202 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-21T10:50:27.202 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.203 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.203 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.203 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Loki container image", 2024-03-21T10:50:27.203 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.203 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.203 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.203 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_node_exporter": { 2024-03-21T10:50:27.203 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_node_exporter", 2024-03-21T10:50:27.203 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.203 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.203 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.203 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-21T10:50:27.203 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.203 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.204 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.204 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-03-21T10:50:27.204 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.204 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.204 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.204 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_nvmeof": { 2024-03-21T10:50:27.204 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_nvmeof", 2024-03-21T10:50:27.204 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.204 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.204 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.204 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-21T10:50:27.204 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.204 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.205 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.205 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Nvme-of container image", 2024-03-21T10:50:27.205 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.205 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.205 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.205 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_prometheus": { 2024-03-21T10:50:27.205 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_prometheus", 2024-03-21T10:50:27.205 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.205 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.205 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.205 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-21T10:50:27.205 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.205 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.205 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.206 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-03-21T10:50:27.206 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.206 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.206 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.206 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_promtail": { 2024-03-21T10:50:27.206 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_promtail", 2024-03-21T10:50:27.206 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.206 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.206 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.206 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-21T10:50:27.206 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.206 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.206 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.207 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Promtail container image", 2024-03-21T10:50:27.207 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.207 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.207 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.207 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_snmp_gateway": { 2024-03-21T10:50:27.207 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_snmp_gateway", 2024-03-21T10:50:27.207 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.207 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.207 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.207 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-21T10:50:27.207 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.207 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.207 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.207 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SNMP Gateway container image", 2024-03-21T10:50:27.208 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.208 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.208 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.208 INFO:teuthology.orchestra.run.smithi031.stdout: "container_init": { 2024-03-21T10:50:27.208 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_init", 2024-03-21T10:50:27.208 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.208 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.208 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.208 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.208 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.208 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.208 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.208 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Run podman/docker with `--init`", 2024-03-21T10:50:27.209 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.209 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.209 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.209 INFO:teuthology.orchestra.run.smithi031.stdout: "daemon_cache_timeout": { 2024-03-21T10:50:27.209 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "daemon_cache_timeout", 2024-03-21T10:50:27.209 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.209 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.209 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.209 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-03-21T10:50:27.209 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.209 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.209 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.210 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-21T10:50:27.210 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.210 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.210 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.210 INFO:teuthology.orchestra.run.smithi031.stdout: "default_cephadm_command_timeout": { 2024-03-21T10:50:27.210 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_cephadm_command_timeout", 2024-03-21T10:50:27.210 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.210 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.210 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.210 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "900", 2024-03-21T10:50:27.210 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.210 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.210 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.211 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-21T10:50:27.211 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.211 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.211 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.211 INFO:teuthology.orchestra.run.smithi031.stdout: "default_registry": { 2024-03-21T10:50:27.211 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_registry", 2024-03-21T10:50:27.211 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.211 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.211 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.211 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io", 2024-03-21T10:50:27.211 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.211 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.211 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.211 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-21T10:50:27.212 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.212 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.212 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.212 INFO:teuthology.orchestra.run.smithi031.stdout: "device_cache_timeout": { 2024-03-21T10:50:27.212 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_cache_timeout", 2024-03-21T10:50:27.212 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.212 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.212 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.212 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1800", 2024-03-21T10:50:27.212 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.212 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.212 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.212 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache device inventory", 2024-03-21T10:50:27.213 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.213 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.213 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.213 INFO:teuthology.orchestra.run.smithi031.stdout: "device_enhanced_scan": { 2024-03-21T10:50:27.213 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_enhanced_scan", 2024-03-21T10:50:27.213 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.213 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.213 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.213 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.213 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.213 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.213 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.213 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-21T10:50:27.213 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.214 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.214 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.214 INFO:teuthology.orchestra.run.smithi031.stdout: "facts_cache_timeout": { 2024-03-21T10:50:27.214 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "facts_cache_timeout", 2024-03-21T10:50:27.214 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.214 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.214 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.214 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-03-21T10:50:27.214 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.214 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.214 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.214 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache host facts data", 2024-03-21T10:50:27.215 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.215 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.215 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.215 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.215 INFO:teuthology.orchestra.run.smithi031.stdout: "grafana_dashboards_path": { 2024-03-21T10:50:27.215 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "grafana_dashboards_path", 2024-03-21T10:50:27.215 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.215 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.215 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.215 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-03-21T10:50:27.215 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.215 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.215 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.215 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-03-21T10:50:27.215 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.216 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.216 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.216 INFO:teuthology.orchestra.run.smithi031.stdout: "host_check_interval": { 2024-03-21T10:50:27.216 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "host_check_interval", 2024-03-21T10:50:27.216 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.216 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.216 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.216 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-03-21T10:50:27.216 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.216 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.216 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.216 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to perform a host check", 2024-03-21T10:50:27.217 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.217 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.217 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.217 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.217 INFO:teuthology.orchestra.run.smithi031.stdout: "hw_monitoring": { 2024-03-21T10:50:27.217 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hw_monitoring", 2024-03-21T10:50:27.217 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.217 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.217 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.217 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.217 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.217 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.217 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.217 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-21T10:50:27.217 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.218 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.218 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.218 INFO:teuthology.orchestra.run.smithi031.stdout: "inventory_list_all": { 2024-03-21T10:50:27.218 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "inventory_list_all", 2024-03-21T10:50:27.218 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.218 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.218 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.218 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.218 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.218 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.218 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.218 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-21T10:50:27.219 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.219 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.219 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.219 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.219 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.219 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.219 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.219 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.219 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.219 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.219 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.219 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.219 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.220 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.220 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.220 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.220 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.220 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.220 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.220 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.220 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.220 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.220 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.220 INFO:teuthology.orchestra.run.smithi031.stdout: "log_refresh_metadata": { 2024-03-21T10:50:27.220 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_refresh_metadata", 2024-03-21T10:50:27.221 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.221 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.221 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.221 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.221 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.221 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.221 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.221 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-21T10:50:27.221 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.221 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.221 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.221 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.221 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.222 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.222 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.222 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.222 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.222 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.222 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.222 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.222 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-21T10:50:27.222 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.222 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.222 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.222 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.223 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.223 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.223 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.223 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.223 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.223 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.223 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.223 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.223 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.223 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.223 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.223 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.224 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.224 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.224 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.224 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.224 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.224 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.224 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.224 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.224 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.224 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.224 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.224 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.224 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.225 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.225 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.225 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.225 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.225 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.225 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.225 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.225 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-21T10:50:27.225 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-21T10:50:27.225 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.225 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.225 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.225 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.226 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.226 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.226 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.226 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-21T10:50:27.226 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.226 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.226 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.226 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-21T10:50:27.226 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-21T10:50:27.226 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.226 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.226 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.226 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "*", 2024-03-21T10:50:27.227 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.227 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.227 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.227 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-21T10:50:27.227 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.227 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.227 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.227 INFO:teuthology.orchestra.run.smithi031.stdout: "max_count_per_host": { 2024-03-21T10:50:27.227 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_count_per_host", 2024-03-21T10:50:27.227 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.227 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.227 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.227 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-03-21T10:50:27.228 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.228 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.228 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.228 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of daemons per service per host", 2024-03-21T10:50:27.228 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.228 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.228 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.228 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd_draining_count": { 2024-03-21T10:50:27.228 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_osd_draining_count", 2024-03-21T10:50:27.229 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.229 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.229 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.229 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-03-21T10:50:27.229 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.229 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.229 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.230 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-21T10:50:27.230 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.230 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.230 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.230 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.230 INFO:teuthology.orchestra.run.smithi031.stdout: "migration_current": { 2024-03-21T10:50:27.231 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "migration_current", 2024-03-21T10:50:27.231 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.231 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.231 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.231 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.231 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.231 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.232 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.232 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "internal - do not modify", 2024-03-21T10:50:27.232 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.232 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.232 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.232 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-03-21T10:50:27.232 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-03-21T10:50:27.232 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.232 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.232 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.232 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "root", 2024-03-21T10:50:27.232 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.233 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.233 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.233 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm-package", 2024-03-21T10:50:27.233 INFO:teuthology.orchestra.run.smithi031.stdout: "root" 2024-03-21T10:50:27.233 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.233 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "mode for remote execution of cephadm", 2024-03-21T10:50:27.233 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.233 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.233 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.233 INFO:teuthology.orchestra.run.smithi031.stdout: "oob_default_addr": { 2024-03-21T10:50:27.233 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "oob_default_addr", 2024-03-21T10:50:27.233 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.233 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.233 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.234 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "169.254.1.1", 2024-03-21T10:50:27.234 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.234 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.234 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.234 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-21T10:50:27.234 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.234 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.234 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.234 INFO:teuthology.orchestra.run.smithi031.stdout: "prometheus_alerts_path": { 2024-03-21T10:50:27.234 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus_alerts_path", 2024-03-21T10:50:27.234 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.234 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.234 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.234 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-21T10:50:27.235 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.235 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.235 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.235 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-21T10:50:27.235 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.235 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.235 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.235 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_insecure": { 2024-03-21T10:50:27.235 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_insecure", 2024-03-21T10:50:27.235 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.235 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.235 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.235 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.236 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.236 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.236 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.236 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-21T10:50:27.236 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.236 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.236 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.236 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_password": { 2024-03-21T10:50:27.236 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_password", 2024-03-21T10:50:27.236 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.236 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.236 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.236 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.237 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.237 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.237 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.237 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-21T10:50:27.237 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.237 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.237 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.237 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_url": { 2024-03-21T10:50:27.237 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_url", 2024-03-21T10:50:27.237 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.237 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.237 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.237 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.238 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.238 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.238 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.238 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-21T10:50:27.238 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.238 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.238 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.238 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_username": { 2024-03-21T10:50:27.238 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_username", 2024-03-21T10:50:27.238 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.238 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.238 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.238 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.239 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.239 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.239 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.239 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-21T10:50:27.239 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.239 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.239 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.239 INFO:teuthology.orchestra.run.smithi031.stdout: "secure_monitoring_stack": { 2024-03-21T10:50:27.239 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "secure_monitoring_stack", 2024-03-21T10:50:27.239 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.239 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.239 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.239 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.240 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.240 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.240 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.240 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-21T10:50:27.240 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.240 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.240 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.240 INFO:teuthology.orchestra.run.smithi031.stdout: "service_discovery_port": { 2024-03-21T10:50:27.240 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "service_discovery_port", 2024-03-21T10:50:27.240 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.240 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.240 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.240 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8765", 2024-03-21T10:50:27.240 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.241 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.241 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.241 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "cephadm service discovery port", 2024-03-21T10:50:27.241 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.241 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.241 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.241 INFO:teuthology.orchestra.run.smithi031.stdout: "ssh_config_file": { 2024-03-21T10:50:27.241 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssh_config_file", 2024-03-21T10:50:27.241 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.241 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.241 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.241 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.241 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.242 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.242 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.242 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-21T10:50:27.242 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.242 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.242 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.242 INFO:teuthology.orchestra.run.smithi031.stdout: "use_agent": { 2024-03-21T10:50:27.242 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_agent", 2024-03-21T10:50:27.242 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.242 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.242 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.242 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.242 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.243 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.243 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.243 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-21T10:50:27.243 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.243 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.243 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.243 INFO:teuthology.orchestra.run.smithi031.stdout: "use_repo_digest": { 2024-03-21T10:50:27.243 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_repo_digest", 2024-03-21T10:50:27.243 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.243 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.243 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.243 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.243 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.243 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.244 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.244 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-21T10:50:27.244 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.244 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.244 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.244 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.244 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_failed_host_check": { 2024-03-21T10:50:27.244 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_failed_host_check", 2024-03-21T10:50:27.244 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.244 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.244 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.244 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.244 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.244 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.244 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.245 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if the host check fails", 2024-03-21T10:50:27.245 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.245 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.245 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.245 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_daemons": { 2024-03-21T10:50:27.245 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_daemons", 2024-03-21T10:50:27.245 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.245 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.245 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.245 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.245 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.245 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.246 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.246 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-21T10:50:27.246 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.246 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.246 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.246 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_hosts": { 2024-03-21T10:50:27.246 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_hosts", 2024-03-21T10:50:27.246 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.246 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.246 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.246 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.246 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.246 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.247 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.247 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-21T10:50:27.247 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.247 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.247 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.247 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.247 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.247 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.247 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crash", 2024-03-21T10:50:27.247 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.247 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.247 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.247 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.247 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.248 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.248 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.248 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.248 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.248 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.248 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.248 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.248 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.248 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.248 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.248 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.248 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.248 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.249 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.249 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.249 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.249 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.249 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.249 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.249 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.249 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.249 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.249 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.249 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.249 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.249 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.250 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.250 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.250 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.250 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.250 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.250 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.250 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.250 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.250 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.250 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.250 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.250 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.250 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.251 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.251 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.251 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.251 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.251 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.251 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.251 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.251 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.251 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.251 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.251 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.251 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.251 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.252 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.252 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.252 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.252 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.252 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.252 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.252 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.252 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.252 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.252 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.252 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.252 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.252 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.252 INFO:teuthology.orchestra.run.smithi031.stdout: "retain_interval": { 2024-03-21T10:50:27.252 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retain_interval", 2024-03-21T10:50:27.253 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.253 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.253 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.253 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "31536000", 2024-03-21T10:50:27.253 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.253 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.253 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.253 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-21T10:50:27.253 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.253 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.253 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.253 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_recent_interval": { 2024-03-21T10:50:27.253 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_recent_interval", 2024-03-21T10:50:27.254 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.254 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.254 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.254 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1209600", 2024-03-21T10:50:27.254 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.254 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.254 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.254 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-21T10:50:27.254 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.254 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.254 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.254 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.254 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.255 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dashboard", 2024-03-21T10:50:27.255 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.255 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.255 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.255 INFO:teuthology.orchestra.run.smithi031.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-21T10:50:27.255 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-21T10:50:27.255 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.255 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.255 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.255 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-03-21T10:50:27.255 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.255 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.256 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.256 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.256 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.256 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.256 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.256 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_HOST": { 2024-03-21T10:50:27.256 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-21T10:50:27.256 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.256 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.256 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.256 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.256 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.257 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.257 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.257 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.257 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.257 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.257 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.257 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.257 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-21T10:50:27.257 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-21T10:50:27.257 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.257 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.258 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.258 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.258 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.258 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.258 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.258 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.258 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.258 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.258 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.258 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.258 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_ENABLED": { 2024-03-21T10:50:27.259 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_ENABLED", 2024-03-21T10:50:27.259 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.259 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.259 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.259 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.259 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.259 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.259 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.259 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.259 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.259 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.259 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.260 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-21T10:50:27.260 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-21T10:50:27.260 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.260 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.260 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.260 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.260 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.260 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.260 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.260 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.260 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.260 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.260 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.261 INFO:teuthology.orchestra.run.smithi031.stdout: "ENABLE_BROWSABLE_API": { 2024-03-21T10:50:27.261 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-21T10:50:27.261 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.261 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.261 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.261 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.261 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.261 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.261 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.261 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.261 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.261 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.261 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.261 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-21T10:50:27.262 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-21T10:50:27.262 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.262 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.262 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.262 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.262 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.262 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.262 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.262 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.262 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.262 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.262 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.263 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-21T10:50:27.263 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-21T10:50:27.263 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.263 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.263 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.263 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.263 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.263 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.263 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.263 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.263 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.263 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.263 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.264 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-21T10:50:27.264 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-21T10:50:27.264 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.264 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.264 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.264 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.264 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.264 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.264 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.264 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.264 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.264 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.264 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.264 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.264 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-21T10:50:27.265 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-21T10:50:27.265 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.265 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.265 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.265 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.265 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.265 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.265 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.265 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.265 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.265 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.265 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.265 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-21T10:50:27.266 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-21T10:50:27.266 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.266 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.266 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.266 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.266 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.266 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.266 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.266 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.266 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.266 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.266 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.266 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-21T10:50:27.267 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-21T10:50:27.267 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.267 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.267 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.267 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.267 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.267 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.267 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.267 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.267 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.267 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.267 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.267 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-21T10:50:27.267 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-21T10:50:27.268 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.268 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.268 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.268 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.268 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.268 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.268 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.268 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.268 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.268 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.268 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.268 INFO:teuthology.orchestra.run.smithi031.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-21T10:50:27.268 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-21T10:50:27.269 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.269 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.269 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.269 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.269 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.269 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.269 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.269 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.269 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.269 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.269 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.269 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_PASSWORD": { 2024-03-21T10:50:27.269 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-21T10:50:27.269 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.270 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.270 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.270 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-03-21T10:50:27.270 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.270 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.270 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.270 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.270 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.270 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.270 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.270 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-21T10:50:27.270 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-21T10:50:27.270 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.271 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.271 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.271 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.271 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.271 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.271 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.271 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.271 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.271 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.271 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.271 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_URL": { 2024-03-21T10:50:27.271 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_URL", 2024-03-21T10:50:27.271 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.271 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.272 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.272 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.272 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.272 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.272 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.272 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.272 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.272 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.272 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.272 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_USERNAME": { 2024-03-21T10:50:27.272 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-21T10:50:27.272 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.272 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.272 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.273 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-03-21T10:50:27.273 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.273 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.273 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.273 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.273 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.273 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.273 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.273 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-21T10:50:27.273 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-21T10:50:27.273 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.273 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.273 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.273 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.274 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.274 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.274 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.274 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.274 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.274 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.274 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.274 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.274 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-21T10:50:27.274 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-21T10:50:27.274 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.274 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.274 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.274 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.275 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.275 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.275 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.275 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.275 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.275 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.275 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.275 INFO:teuthology.orchestra.run.smithi031.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-21T10:50:27.275 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-21T10:50:27.275 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.275 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.275 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.275 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.276 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.276 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.276 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.276 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.276 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.276 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.276 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.276 INFO:teuthology.orchestra.run.smithi031.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-21T10:50:27.276 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-21T10:50:27.276 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.276 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.276 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.276 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.276 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.277 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.277 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.277 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.277 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.277 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.277 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.277 INFO:teuthology.orchestra.run.smithi031.stdout: "MULTICLUSTER_CONFIG": { 2024-03-21T10:50:27.277 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-21T10:50:27.277 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.277 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.277 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.277 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "{}", 2024-03-21T10:50:27.277 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.277 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.278 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.278 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.278 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.278 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.278 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.278 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_HOST": { 2024-03-21T10:50:27.278 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-21T10:50:27.278 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.278 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.278 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.278 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.278 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.278 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.279 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.279 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.279 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.279 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.279 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.279 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.279 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-21T10:50:27.279 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-21T10:50:27.279 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.279 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.279 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.279 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.279 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.279 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.280 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.280 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.280 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.280 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.280 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.280 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.280 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-21T10:50:27.280 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-21T10:50:27.280 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.280 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.280 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.280 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.280 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.280 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.280 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.281 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.281 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.281 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.281 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.281 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.281 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-21T10:50:27.281 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-21T10:50:27.281 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.281 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.281 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.281 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.281 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.281 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.281 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.281 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.282 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.282 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.282 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.282 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.282 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-21T10:50:27.282 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-21T10:50:27.282 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.282 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.282 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.282 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.282 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.282 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.282 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.283 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.283 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.283 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.283 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.283 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.283 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-21T10:50:27.283 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-21T10:50:27.283 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.283 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.283 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.283 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.283 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.283 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.283 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.283 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.284 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.284 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.284 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.284 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.284 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-21T10:50:27.284 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-21T10:50:27.284 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.284 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.284 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.284 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.284 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.284 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.284 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.284 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.284 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.285 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.285 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.285 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.285 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-21T10:50:27.285 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-21T10:50:27.285 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.285 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.285 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.285 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.285 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.285 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.285 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.285 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.285 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.285 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.286 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.286 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.286 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-21T10:50:27.286 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-21T10:50:27.286 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.286 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.286 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.286 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.286 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.286 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.286 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.286 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.286 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.286 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.287 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.287 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.287 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_ENABLED": { 2024-03-21T10:50:27.287 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-21T10:50:27.287 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.287 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.287 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.287 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.287 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.287 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.287 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.287 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.287 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.287 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.287 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.288 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.288 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-21T10:50:27.288 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-21T10:50:27.288 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.288 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.288 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.288 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-21T10:50:27.288 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.288 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.288 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.288 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.288 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.288 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.289 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.289 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.289 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-21T10:50:27.289 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-21T10:50:27.289 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.289 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.289 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.289 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-03-21T10:50:27.289 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.289 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.289 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.289 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.289 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.289 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.290 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.290 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.290 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-21T10:50:27.290 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-21T10:50:27.290 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.290 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.290 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.290 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8", 2024-03-21T10:50:27.290 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.290 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.290 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.290 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.290 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.291 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.291 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.291 INFO:teuthology.orchestra.run.smithi031.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-21T10:50:27.291 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-21T10:50:27.291 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.291 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.291 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.291 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "45", 2024-03-21T10:50:27.291 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.291 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.291 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.291 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ACCESS_KEY": { 2024-03-21T10:50:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-21T10:50:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.293 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.293 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.293 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.293 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-21T10:50:27.293 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-21T10:50:27.293 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.293 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.293 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.293 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-03-21T10:50:27.293 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.293 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.293 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.294 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.294 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.294 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.294 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.294 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SECRET_KEY": { 2024-03-21T10:50:27.294 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-21T10:50:27.294 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.294 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.294 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.294 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.294 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.294 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.295 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.295 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.295 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.295 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.295 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.295 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SSL_VERIFY": { 2024-03-21T10:50:27.295 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-21T10:50:27.295 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.295 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.295 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.295 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.295 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.296 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.296 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.296 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.296 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.296 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.296 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.296 INFO:teuthology.orchestra.run.smithi031.stdout: "UNSAFE_TLS_v1_2": { 2024-03-21T10:50:27.296 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-21T10:50:27.296 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.296 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.296 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.296 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.296 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.297 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.297 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.297 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.297 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.297 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.297 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.297 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-21T10:50:27.297 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-21T10:50:27.297 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.297 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.297 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.297 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-03-21T10:50:27.298 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.298 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.298 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.298 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.298 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.298 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.298 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.298 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-21T10:50:27.298 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-21T10:50:27.298 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.298 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.299 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.299 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-03-21T10:50:27.299 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.299 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.299 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.299 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.299 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.299 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.299 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.299 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-21T10:50:27.299 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-21T10:50:27.299 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.299 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.300 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.300 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-03-21T10:50:27.300 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.300 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.300 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.300 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.300 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.300 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.300 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.300 INFO:teuthology.orchestra.run.smithi031.stdout: "cross_origin_url": { 2024-03-21T10:50:27.300 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cross_origin_url", 2024-03-21T10:50:27.300 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.300 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.300 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.301 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.301 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.301 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.301 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.301 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.301 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.301 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.301 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.301 INFO:teuthology.orchestra.run.smithi031.stdout: "crt_file": { 2024-03-21T10:50:27.301 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crt_file", 2024-03-21T10:50:27.301 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.301 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.301 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.302 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.302 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.302 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.302 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.302 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.302 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.302 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.302 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.302 INFO:teuthology.orchestra.run.smithi031.stdout: "debug": { 2024-03-21T10:50:27.302 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "debug", 2024-03-21T10:50:27.302 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.302 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.302 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.303 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.303 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.303 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.303 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.303 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable/disable debug options", 2024-03-21T10:50:27.303 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.303 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.303 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.303 INFO:teuthology.orchestra.run.smithi031.stdout: "jwt_token_ttl": { 2024-03-21T10:50:27.303 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "jwt_token_ttl", 2024-03-21T10:50:27.303 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.303 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.303 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.304 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "28800", 2024-03-21T10:50:27.304 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.304 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.304 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.304 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.304 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.304 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.304 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.304 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-03-21T10:50:27.304 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-03-21T10:50:27.304 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.304 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.304 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.304 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.305 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.305 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.305 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.305 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.305 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.305 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.305 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.305 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.305 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.305 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.305 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.305 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.305 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.306 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.306 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.306 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.306 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.306 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.306 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.306 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.306 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.306 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.306 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.306 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.306 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.306 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.306 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.307 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.307 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.307 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.307 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.307 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.307 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.307 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.307 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.307 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.307 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.307 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.307 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.307 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.308 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.308 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.308 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.308 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.308 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.308 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.308 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.308 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.308 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.308 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.308 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.308 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.308 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.309 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.309 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.309 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.309 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.309 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.309 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.309 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.309 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.309 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.309 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.309 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.309 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.309 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.309 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.310 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.310 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.310 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.310 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.310 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.310 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.310 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.310 INFO:teuthology.orchestra.run.smithi031.stdout: "motd": { 2024-03-21T10:50:27.310 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "motd", 2024-03-21T10:50:27.310 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.310 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.310 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.310 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.311 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.311 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.311 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.311 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "The message of the day", 2024-03-21T10:50:27.311 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.311 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.311 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.311 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect_resolve_ip_addr": { 2024-03-21T10:50:27.311 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "redirect_resolve_ip_addr", 2024-03-21T10:50:27.311 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.311 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.311 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.311 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.311 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.312 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.312 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.312 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.312 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.312 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.312 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.312 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-03-21T10:50:27.312 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-03-21T10:50:27.312 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.312 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.312 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.312 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-03-21T10:50:27.312 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.312 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.312 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.313 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.313 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.313 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.313 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.313 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-03-21T10:50:27.313 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-03-21T10:50:27.313 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.313 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.313 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.313 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8080", 2024-03-21T10:50:27.313 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.313 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.313 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.313 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.314 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.314 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.314 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.314 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-03-21T10:50:27.314 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-03-21T10:50:27.314 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.314 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.314 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.314 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.314 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.314 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.314 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.315 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.315 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.315 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.315 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.315 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl_server_port": { 2024-03-21T10:50:27.315 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl_server_port", 2024-03-21T10:50:27.315 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.315 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.315 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.315 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8443", 2024-03-21T10:50:27.315 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.315 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.315 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.316 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.316 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.316 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.316 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.316 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-03-21T10:50:27.316 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-03-21T10:50:27.316 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.316 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.316 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.316 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "redirect", 2024-03-21T10:50:27.316 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.316 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.316 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.317 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.317 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect" 2024-03-21T10:50:27.317 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.317 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.317 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.317 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.317 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.317 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-03-21T10:50:27.317 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-03-21T10:50:27.317 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.317 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.317 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.317 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-03-21T10:50:27.318 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-03-21T10:50:27.318 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-03-21T10:50:27.318 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.318 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.318 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.318 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.318 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.318 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.318 INFO:teuthology.orchestra.run.smithi031.stdout: "url_prefix": { 2024-03-21T10:50:27.318 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url_prefix", 2024-03-21T10:50:27.318 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.318 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.318 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.318 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.319 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.319 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.319 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.319 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.319 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.319 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.319 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.319 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.319 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.319 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "devicehealth", 2024-03-21T10:50:27.319 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.319 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.320 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.320 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_monitoring": { 2024-03-21T10:50:27.320 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_monitoring", 2024-03-21T10:50:27.320 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.320 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.320 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.320 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.320 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.320 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.320 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.320 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "monitor device health metrics", 2024-03-21T10:50:27.320 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.320 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.320 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.321 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.321 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.321 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.321 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.321 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.321 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.321 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.321 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.321 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.321 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.321 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.321 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.321 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.321 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.322 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.322 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.322 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.322 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.322 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.322 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.322 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.322 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.322 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.322 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.322 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.322 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.322 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.323 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.323 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.323 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.323 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.323 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.323 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.323 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.323 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.323 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.323 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.323 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.323 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.323 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.323 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.324 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.324 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.324 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.324 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.324 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.324 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.324 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.324 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.324 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.324 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.324 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.324 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.324 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.325 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.325 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.325 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.325 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.325 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.325 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.325 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.325 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.325 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.325 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.325 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.325 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.325 INFO:teuthology.orchestra.run.smithi031.stdout: "mark_out_threshold": { 2024-03-21T10:50:27.325 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mark_out_threshold", 2024-03-21T10:50:27.326 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.326 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.326 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.326 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2419200", 2024-03-21T10:50:27.326 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.326 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.326 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.326 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-21T10:50:27.326 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.326 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.326 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.326 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": { 2024-03-21T10:50:27.326 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_name", 2024-03-21T10:50:27.327 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.327 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.327 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.327 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "device_health_metrics", 2024-03-21T10:50:27.327 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.327 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.327 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.327 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-21T10:50:27.327 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.327 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.327 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.327 INFO:teuthology.orchestra.run.smithi031.stdout: "retention_period": { 2024-03-21T10:50:27.327 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retention_period", 2024-03-21T10:50:27.327 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.328 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.328 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.328 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15552000", 2024-03-21T10:50:27.328 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.328 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.328 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.328 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain device health metrics", 2024-03-21T10:50:27.328 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.328 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.328 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.328 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_frequency": { 2024-03-21T10:50:27.328 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_frequency", 2024-03-21T10:50:27.328 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.329 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.329 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.329 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-03-21T10:50:27.329 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.329 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.329 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.329 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-21T10:50:27.329 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.329 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.329 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.329 INFO:teuthology.orchestra.run.smithi031.stdout: "self_heal": { 2024-03-21T10:50:27.329 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "self_heal", 2024-03-21T10:50:27.329 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.329 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.330 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.330 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.330 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.330 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.330 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.330 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-21T10:50:27.330 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.330 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.330 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.330 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-03-21T10:50:27.330 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-03-21T10:50:27.330 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.330 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.331 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.331 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-03-21T10:50:27.331 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.331 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.331 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.331 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and check device health", 2024-03-21T10:50:27.331 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.331 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.331 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.331 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_threshold": { 2024-03-21T10:50:27.331 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_threshold", 2024-03-21T10:50:27.331 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.331 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.331 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.332 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7257600", 2024-03-21T10:50:27.332 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.332 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.332 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.332 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-21T10:50:27.332 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.332 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.332 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.332 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.332 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.332 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.332 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "diskprediction_local", 2024-03-21T10:50:27.332 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.332 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.333 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.333 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.333 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.333 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.333 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.333 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.333 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.333 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.333 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.333 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.333 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.333 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.333 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.333 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.334 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.334 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.334 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.334 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.334 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.334 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.334 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.334 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.334 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.334 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.334 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.334 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.334 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.335 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.335 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.335 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.335 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.335 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.335 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.335 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.335 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.335 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.335 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.335 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.335 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.335 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.335 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.336 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.336 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.336 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.336 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.336 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.336 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.336 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.336 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.336 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.336 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.336 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.336 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.336 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.336 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.337 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.337 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.337 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.337 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.337 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.337 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.337 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.337 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.337 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.337 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.337 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.337 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.337 INFO:teuthology.orchestra.run.smithi031.stdout: "predict_interval": { 2024-03-21T10:50:27.338 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predict_interval", 2024-03-21T10:50:27.338 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.338 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.338 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.338 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-03-21T10:50:27.338 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.338 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.338 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.338 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.338 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.338 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.338 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.338 INFO:teuthology.orchestra.run.smithi031.stdout: "predictor_model": { 2024-03-21T10:50:27.339 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predictor_model", 2024-03-21T10:50:27.339 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.339 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.339 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.339 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "prophetstor", 2024-03-21T10:50:27.339 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.339 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.339 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.339 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.339 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.339 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.339 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.339 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.339 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-03-21T10:50:27.339 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-03-21T10:50:27.340 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.340 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.340 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.340 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-03-21T10:50:27.340 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.340 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.340 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.340 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.340 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.340 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.340 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.340 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.340 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.341 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.341 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.341 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "influx", 2024-03-21T10:50:27.341 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": false, 2024-03-21T10:50:27.341 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "influxdb python module not found", 2024-03-21T10:50:27.341 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.341 INFO:teuthology.orchestra.run.smithi031.stdout: "batch_size": { 2024-03-21T10:50:27.341 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "batch_size", 2024-03-21T10:50:27.341 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.341 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.341 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.341 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5000", 2024-03-21T10:50:27.341 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.341 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.342 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.342 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-21T10:50:27.342 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.342 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.342 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.342 INFO:teuthology.orchestra.run.smithi031.stdout: "database": { 2024-03-21T10:50:27.342 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "database", 2024-03-21T10:50:27.342 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.342 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.342 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.342 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "ceph", 2024-03-21T10:50:27.342 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.343 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.343 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.343 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-21T10:50:27.343 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.343 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.343 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.343 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.343 INFO:teuthology.orchestra.run.smithi031.stdout: "hostname": { 2024-03-21T10:50:27.343 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hostname", 2024-03-21T10:50:27.343 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.343 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.343 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.343 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.343 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.344 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.344 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.344 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server hostname", 2024-03-21T10:50:27.344 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.344 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.344 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.344 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.344 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-03-21T10:50:27.344 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-03-21T10:50:27.344 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.344 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.344 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.344 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "30", 2024-03-21T10:50:27.344 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "5", 2024-03-21T10:50:27.345 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.345 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.345 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-21T10:50:27.345 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.345 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.345 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.345 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.345 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.345 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.345 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.345 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.345 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.345 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.345 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.346 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.346 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.346 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.346 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.346 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.346 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.346 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.346 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.346 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.346 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.346 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.346 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.346 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.347 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.347 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.347 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.347 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.347 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.347 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.347 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.347 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.347 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.347 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.347 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.347 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.348 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.348 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.348 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.348 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.348 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.348 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.348 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.348 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.348 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.348 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.348 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.348 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.348 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.348 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.349 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.349 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.349 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.349 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.349 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.349 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.349 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.349 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.349 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.349 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.349 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.349 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.350 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.350 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.350 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.350 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.350 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.350 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.350 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.350 INFO:teuthology.orchestra.run.smithi031.stdout: "password": { 2024-03-21T10:50:27.350 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "password", 2024-03-21T10:50:27.350 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.350 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.350 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.350 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.351 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.351 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.351 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.351 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "password of InfluxDB server user", 2024-03-21T10:50:27.351 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.351 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.351 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.351 INFO:teuthology.orchestra.run.smithi031.stdout: "port": { 2024-03-21T10:50:27.351 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "port", 2024-03-21T10:50:27.351 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.351 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.351 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.351 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8086", 2024-03-21T10:50:27.352 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.352 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.352 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.352 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server port", 2024-03-21T10:50:27.352 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.352 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.352 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.352 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-03-21T10:50:27.352 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-03-21T10:50:27.352 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.352 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.352 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.352 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "false", 2024-03-21T10:50:27.353 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.353 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.353 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.353 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-21T10:50:27.353 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.353 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.353 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.353 INFO:teuthology.orchestra.run.smithi031.stdout: "threads": { 2024-03-21T10:50:27.353 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threads", 2024-03-21T10:50:27.353 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.353 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.353 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.354 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-03-21T10:50:27.354 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-03-21T10:50:27.354 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "32", 2024-03-21T10:50:27.354 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.354 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-21T10:50:27.354 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.354 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.354 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.354 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.354 INFO:teuthology.orchestra.run.smithi031.stdout: "username": { 2024-03-21T10:50:27.354 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "username", 2024-03-21T10:50:27.354 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.354 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.355 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.355 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.355 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.355 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.355 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.355 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "username of InfluxDB server user", 2024-03-21T10:50:27.355 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.355 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.355 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.355 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.355 INFO:teuthology.orchestra.run.smithi031.stdout: "verify_ssl": { 2024-03-21T10:50:27.355 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "verify_ssl", 2024-03-21T10:50:27.355 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.355 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.356 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.356 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "true", 2024-03-21T10:50:27.356 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.356 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.356 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.356 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-21T10:50:27.356 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.356 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.356 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.356 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.356 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.356 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.356 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.356 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "insights", 2024-03-21T10:50:27.357 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.357 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.357 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.357 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.357 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.357 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.357 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.357 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.357 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.357 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.357 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.357 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.357 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.357 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.357 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.358 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.358 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.358 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.358 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.358 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.358 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.358 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.358 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.358 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.358 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.358 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.358 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.358 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.358 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.359 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.359 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.359 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.359 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.359 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.359 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.359 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.359 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.359 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.359 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.359 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.359 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.359 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.359 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.360 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.360 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.360 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.360 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.360 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.360 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.360 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.360 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.360 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.360 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.360 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.360 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.360 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.360 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.360 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.361 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.361 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.361 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.361 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.361 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.361 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.361 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.361 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.361 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.361 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.361 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.361 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.361 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.361 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.361 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.362 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.362 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.362 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.362 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "iostat", 2024-03-21T10:50:27.362 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.362 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.362 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.362 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.362 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.362 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.362 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.362 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.362 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.362 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.363 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.363 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.363 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.363 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.363 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.363 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.363 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.363 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.363 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.363 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.363 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.363 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.363 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.363 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.363 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.364 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.364 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.364 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.364 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.364 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.364 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.364 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.364 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.364 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.364 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.364 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.364 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.364 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.364 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.364 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.365 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.365 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.365 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.365 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.365 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.365 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.365 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.365 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.365 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.365 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.365 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.365 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.365 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.365 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.365 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.365 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.366 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.366 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.366 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.366 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.366 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.366 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.366 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.366 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.366 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.366 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.366 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.366 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.366 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.366 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.367 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.367 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.367 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.367 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.367 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.367 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.367 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "k8sevents", 2024-03-21T10:50:27.367 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.367 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.367 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.367 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_event_retention_days": { 2024-03-21T10:50:27.367 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ceph_event_retention_days", 2024-03-21T10:50:27.367 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.367 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.367 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.368 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7", 2024-03-21T10:50:27.368 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.368 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.368 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.368 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-21T10:50:27.368 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.368 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.368 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.368 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.368 INFO:teuthology.orchestra.run.smithi031.stdout: "config_check_secs": { 2024-03-21T10:50:27.368 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_check_secs", 2024-03-21T10:50:27.368 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.368 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.369 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.369 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-03-21T10:50:27.369 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "10", 2024-03-21T10:50:27.369 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.369 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.369 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-21T10:50:27.369 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.369 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.369 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.369 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.369 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.369 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.369 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.370 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.370 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.370 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.370 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.370 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.370 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.370 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.370 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.370 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.370 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.370 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.370 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.370 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.371 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.371 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.371 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.371 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.371 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.371 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.371 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.371 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.371 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.371 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.371 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.371 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.371 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.371 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.372 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.372 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.372 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.372 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.372 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.372 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.372 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.372 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.372 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.373 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.373 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.373 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.373 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.373 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.373 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.373 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.373 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.373 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.373 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.373 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.373 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.373 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.374 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.374 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.374 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.374 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.374 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.374 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.374 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.374 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.374 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.374 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.374 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.374 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.374 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.375 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.375 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.375 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.375 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.375 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.375 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.375 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "localpool", 2024-03-21T10:50:27.375 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.375 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.375 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.375 INFO:teuthology.orchestra.run.smithi031.stdout: "failure_domain": { 2024-03-21T10:50:27.375 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "failure_domain", 2024-03-21T10:50:27.375 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.375 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.375 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.376 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "host", 2024-03-21T10:50:27.376 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.376 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.376 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.376 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "failure domain for any created local pool", 2024-03-21T10:50:27.376 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-21T10:50:27.376 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.376 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.376 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.376 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.376 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.376 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.376 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.376 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.376 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.377 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.377 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.377 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.377 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.377 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.377 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.377 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.377 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.377 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.377 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.377 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.377 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.377 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.377 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.377 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.377 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.378 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.378 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.378 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.378 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.378 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.378 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.378 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.378 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.378 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.378 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.378 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.378 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.378 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.379 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.379 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.379 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.379 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.379 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.379 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.379 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.379 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.379 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.379 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.379 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.379 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.379 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.379 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.379 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.380 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.380 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.380 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.380 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.380 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.380 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.380 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.380 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.380 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.380 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.380 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.380 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.380 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.380 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.380 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.381 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.381 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.381 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.381 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.381 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.381 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": { 2024-03-21T10:50:27.381 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_size", 2024-03-21T10:50:27.381 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.381 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.381 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.381 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.381 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.381 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.381 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.381 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default min_size for any created local pool", 2024-03-21T10:50:27.382 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-21T10:50:27.382 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.382 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.382 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.382 INFO:teuthology.orchestra.run.smithi031.stdout: "num_rep": { 2024-03-21T10:50:27.382 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "num_rep", 2024-03-21T10:50:27.382 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.382 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.382 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.382 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3", 2024-03-21T10:50:27.383 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.383 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.383 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.383 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default replica count for any created local pool", 2024-03-21T10:50:27.383 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.383 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.383 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.383 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.383 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": { 2024-03-21T10:50:27.383 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_num", 2024-03-21T10:50:27.383 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.383 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.383 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.383 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "128", 2024-03-21T10:50:27.384 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.384 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.384 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.384 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default pg_num for any created local pool", 2024-03-21T10:50:27.384 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.384 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.384 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.384 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.384 INFO:teuthology.orchestra.run.smithi031.stdout: "prefix": { 2024-03-21T10:50:27.384 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prefix", 2024-03-21T10:50:27.384 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.384 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.384 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.384 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.385 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.385 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.385 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.385 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name prefix for any created local pool", 2024-03-21T10:50:27.385 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.385 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.385 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.385 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.385 INFO:teuthology.orchestra.run.smithi031.stdout: "subtree": { 2024-03-21T10:50:27.385 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "subtree", 2024-03-21T10:50:27.385 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.385 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.385 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.385 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "rack", 2024-03-21T10:50:27.385 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.386 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.386 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.386 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-21T10:50:27.386 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-21T10:50:27.386 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.386 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.386 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.386 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.386 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.386 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.386 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mds_autoscaler", 2024-03-21T10:50:27.386 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.386 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.387 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.387 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.387 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.387 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.387 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.387 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.387 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.387 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.387 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.387 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.387 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.387 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.387 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.387 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.387 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.388 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.388 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.388 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.388 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.388 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.388 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.388 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.388 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.388 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.388 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.388 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.388 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.388 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.388 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.388 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.389 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.389 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.389 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.389 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.389 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.389 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.389 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.389 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.389 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.389 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.389 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.389 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.389 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.389 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.390 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.390 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.390 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.390 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.390 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.390 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.390 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.390 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.390 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.390 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.390 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.390 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.390 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.391 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.391 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.391 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.391 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.391 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.391 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.391 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.391 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.391 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.391 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.391 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.391 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.391 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.391 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.391 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.392 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.392 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.392 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirroring", 2024-03-21T10:50:27.392 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.392 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.392 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.392 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.392 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.392 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.392 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.392 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.392 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.392 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.392 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.393 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.393 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.393 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.393 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.393 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.393 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.393 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.393 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.393 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.393 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.393 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.393 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.393 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.393 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.394 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.394 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.394 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.394 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.394 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.394 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.394 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.394 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.394 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.394 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.394 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.394 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.394 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.394 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.395 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.395 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.395 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.395 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.395 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.395 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.395 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.395 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.395 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.395 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.395 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.395 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.395 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.395 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.396 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.396 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.396 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.396 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.396 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.396 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.396 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.396 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.396 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.396 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.396 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.396 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.396 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.397 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.397 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.397 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.397 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.397 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.397 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.397 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.397 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "nfs", 2024-03-21T10:50:27.397 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.397 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.397 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.397 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.397 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.397 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.398 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.398 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.398 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.398 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.398 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.398 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.398 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.398 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.398 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.398 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.398 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.398 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.398 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.399 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.399 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.399 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.399 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.399 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.399 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.399 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.399 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.399 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.399 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.399 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.399 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.399 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.399 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.400 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.400 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.400 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.400 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.400 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.400 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.400 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.400 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.400 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.400 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.400 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.400 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.400 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.400 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.400 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.401 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.401 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.401 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.401 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.401 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.401 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.401 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.401 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.401 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.401 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.401 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.401 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.401 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.401 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.402 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.402 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.402 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.402 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.402 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.402 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.402 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.402 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.402 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.402 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.402 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.402 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.402 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.402 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.403 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-03-21T10:50:27.403 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.403 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.403 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.403 INFO:teuthology.orchestra.run.smithi031.stdout: "fail_fs": { 2024-03-21T10:50:27.403 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "fail_fs", 2024-03-21T10:50:27.403 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.403 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.403 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.403 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.403 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.403 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.403 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.403 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-21T10:50:27.404 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.404 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.404 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.404 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.404 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.404 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.404 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.404 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.404 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.404 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.404 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.404 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.404 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.405 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.405 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.405 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.405 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.405 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.405 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.405 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.405 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.405 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.405 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.405 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.405 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.405 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.405 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.406 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.406 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.406 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.406 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.406 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.406 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.406 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.406 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.406 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.406 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.406 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.406 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.406 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.406 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.407 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.407 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.407 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.407 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.407 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.407 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.407 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.407 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.407 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.407 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.407 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.407 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.407 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.407 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.408 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.408 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.408 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.408 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.408 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.408 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.408 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.408 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.408 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.408 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.408 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.408 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.408 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.409 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.409 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.409 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator": { 2024-03-21T10:50:27.409 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-03-21T10:50:27.409 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.409 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.409 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.409 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.409 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.409 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.409 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.409 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm", 2024-03-21T10:50:27.409 INFO:teuthology.orchestra.run.smithi031.stdout: "rook", 2024-03-21T10:50:27.410 INFO:teuthology.orchestra.run.smithi031.stdout: "test_orchestrator" 2024-03-21T10:50:27.410 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.410 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Orchestrator backend", 2024-03-21T10:50:27.410 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.410 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.410 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.410 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.410 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.410 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.410 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.410 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_perf_query", 2024-03-21T10:50:27.410 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.410 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.410 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.411 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.411 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.411 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.411 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.411 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.411 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.411 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.411 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.411 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.411 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.411 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.411 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.411 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.411 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.411 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.412 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.412 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.412 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.412 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.412 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.412 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.412 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.412 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.412 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.412 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.412 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.412 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.413 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.413 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.413 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.413 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.413 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.413 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.413 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.413 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.413 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.413 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.413 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.413 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.413 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.414 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.414 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.414 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.414 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.414 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.414 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.414 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.414 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.414 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.414 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.414 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.414 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.414 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.414 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.415 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.415 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.415 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.415 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.415 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.415 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.415 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.415 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.415 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.415 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.415 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.415 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.415 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.415 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.416 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.416 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.416 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_support", 2024-03-21T10:50:27.416 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.416 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.416 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.416 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.416 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.416 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.416 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.416 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.416 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.416 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.416 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.417 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.417 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.417 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.417 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.417 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.417 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.417 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.417 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.417 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.417 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.417 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.417 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.417 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.417 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.417 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.418 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.418 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.418 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.418 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.418 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.418 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.418 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.418 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.418 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.418 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.418 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.418 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.418 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.419 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.419 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.419 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.419 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.419 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.419 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.419 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.419 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.419 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.419 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.419 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.419 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.419 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.419 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.419 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.420 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.420 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.420 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.420 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.420 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.420 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.420 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.420 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.420 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.420 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.420 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.420 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.420 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.421 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.421 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.421 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.421 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.421 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.421 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.421 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_autoscaler", 2024-03-21T10:50:27.421 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.421 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.421 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.421 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.421 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.421 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.421 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.422 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.422 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.422 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.422 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.422 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.422 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.422 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.422 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.422 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.422 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.422 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.422 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.422 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.422 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.423 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.423 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.423 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.423 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.423 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.423 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.423 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.423 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.423 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.423 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.423 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.423 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.424 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.424 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.424 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.424 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.424 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.424 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.424 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.424 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.424 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.424 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.424 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.424 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.424 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.424 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.425 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.425 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.425 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.425 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.425 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.425 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.425 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.425 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.425 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.425 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.425 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.425 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.425 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.426 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.426 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.426 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.426 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.426 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.426 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.426 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.426 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.426 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.426 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.426 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-03-21T10:50:27.426 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-03-21T10:50:27.426 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.426 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.427 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.427 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-03-21T10:50:27.427 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.427 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.427 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.427 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.427 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.427 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.427 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.427 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.427 INFO:teuthology.orchestra.run.smithi031.stdout: "threshold": { 2024-03-21T10:50:27.427 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threshold", 2024-03-21T10:50:27.427 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-03-21T10:50:27.427 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.428 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.428 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-03-21T10:50:27.428 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1.0", 2024-03-21T10:50:27.428 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.428 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.428 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "scaling threshold", 2024-03-21T10:50:27.428 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-21T10:50:27.428 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.428 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.428 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.428 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.428 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.428 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.428 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "progress", 2024-03-21T10:50:27.429 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.429 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.429 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.429 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_pg_recovery_event": { 2024-03-21T10:50:27.429 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_pg_recovery_event", 2024-03-21T10:50:27.429 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.429 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.429 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.429 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.429 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.429 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.429 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.429 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-21T10:50:27.429 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.429 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.429 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.430 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.430 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-03-21T10:50:27.430 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-03-21T10:50:27.430 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.430 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.430 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.430 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.430 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.430 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.430 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.430 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.430 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.430 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.431 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.431 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.431 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.431 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.431 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.431 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.431 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.431 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.431 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.431 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.431 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.431 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.431 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.431 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.431 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.431 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.432 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.432 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.432 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.432 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.432 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.432 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.432 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.432 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.432 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.432 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.432 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.432 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.432 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.432 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.432 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.433 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.433 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.433 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.433 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.433 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.433 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.433 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.433 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.433 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.433 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.433 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.433 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.433 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.434 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.434 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.434 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.434 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.434 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.434 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.434 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.434 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.434 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.434 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.434 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.434 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.434 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.434 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.434 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.435 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.435 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.435 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.435 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.435 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.435 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.435 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.435 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.435 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.435 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.435 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.435 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.435 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.435 INFO:teuthology.orchestra.run.smithi031.stdout: "max_completed_events": { 2024-03-21T10:50:27.435 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_completed_events", 2024-03-21T10:50:27.435 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.436 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.436 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.436 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "50", 2024-03-21T10:50:27.436 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.436 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.436 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.436 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "number of past completed events to remember", 2024-03-21T10:50:27.436 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.436 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.436 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.436 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.436 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-03-21T10:50:27.436 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-03-21T10:50:27.436 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.436 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.437 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.437 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-03-21T10:50:27.437 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.437 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.437 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.437 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long the module is going to sleep", 2024-03-21T10:50:27.437 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.437 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.437 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.437 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.437 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.437 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.437 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus", 2024-03-21T10:50:27.438 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.438 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.438 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.438 INFO:teuthology.orchestra.run.smithi031.stdout: "cache": { 2024-03-21T10:50:27.438 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cache", 2024-03-21T10:50:27.438 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.438 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.438 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.438 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.438 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.438 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.438 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.438 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.439 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.439 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.439 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.439 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.439 INFO:teuthology.orchestra.run.smithi031.stdout: "exclude_perf_counters": { 2024-03-21T10:50:27.439 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "exclude_perf_counters", 2024-03-21T10:50:27.439 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.439 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.439 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.439 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.439 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.439 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.439 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.439 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-21T10:50:27.441 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-21T10:50:27.441 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.441 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.441 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.441 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.441 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.441 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.441 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.441 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.442 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.442 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.442 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.442 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.442 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.442 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.442 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.442 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.442 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.442 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.442 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.442 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.443 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.443 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.443 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.443 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.443 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.443 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.443 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.443 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.443 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.443 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.443 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.443 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.443 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.443 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.444 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.444 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.444 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.444 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.444 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.444 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.444 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.444 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.444 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.444 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.444 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.444 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.444 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.445 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.445 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.445 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.445 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.445 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.445 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.445 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.445 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.445 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.445 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.445 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.445 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.445 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.445 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.446 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.446 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.446 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.446 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.446 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.446 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.446 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.446 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.446 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.446 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.446 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools": { 2024-03-21T10:50:27.446 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools", 2024-03-21T10:50:27.446 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.446 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.447 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.447 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.447 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.447 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.447 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.447 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.447 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.447 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.447 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.447 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.447 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-21T10:50:27.447 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-21T10:50:27.447 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.447 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.447 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.448 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "300", 2024-03-21T10:50:27.448 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.448 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.448 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.448 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.448 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.448 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.448 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.448 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.448 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_interval": { 2024-03-21T10:50:27.448 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_interval", 2024-03-21T10:50:27.448 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-03-21T10:50:27.448 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.448 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.448 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15.0", 2024-03-21T10:50:27.449 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.449 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.449 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.449 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.449 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.449 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.449 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.449 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.449 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-03-21T10:50:27.449 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-03-21T10:50:27.449 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.449 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.449 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.449 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-03-21T10:50:27.449 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.450 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.450 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.450 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-21T10:50:27.450 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.450 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.450 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.450 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.450 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-03-21T10:50:27.450 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-03-21T10:50:27.450 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.450 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.450 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.450 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "9283", 2024-03-21T10:50:27.451 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.451 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.451 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.451 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-21T10:50:27.451 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.451 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.451 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.451 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.451 INFO:teuthology.orchestra.run.smithi031.stdout: "stale_cache_strategy": { 2024-03-21T10:50:27.451 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stale_cache_strategy", 2024-03-21T10:50:27.451 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.451 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.451 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.451 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "log", 2024-03-21T10:50:27.451 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.452 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.452 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.452 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.452 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.452 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.452 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.452 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.452 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-03-21T10:50:27.452 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-03-21T10:50:27.452 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.452 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.452 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.452 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "default", 2024-03-21T10:50:27.452 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: "default", 2024-03-21T10:50:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: "error" 2024-03-21T10:50:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-03-21T10:50:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-03-21T10:50:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.453 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-03-21T10:50:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-03-21T10:50:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-03-21T10:50:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_support", 2024-03-21T10:50:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.455 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.456 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.457 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_snap_create": { 2024-03-21T10:50:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_snap_create", 2024-03-21T10:50:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-03-21T10:50:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: "mirror_snapshot_schedule": { 2024-03-21T10:50:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirror_snapshot_schedule", 2024-03-21T10:50:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.461 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.461 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.461 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.461 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.461 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.461 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.461 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.461 INFO:teuthology.orchestra.run.smithi031.stdout: "trash_purge_schedule": { 2024-03-21T10:50:27.461 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "trash_purge_schedule", 2024-03-21T10:50:27.461 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.461 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.461 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.461 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.461 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.462 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.462 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.462 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.462 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.462 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.462 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.462 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.462 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.462 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "restful", 2024-03-21T10:50:27.462 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.462 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.462 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.462 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_auth": { 2024-03-21T10:50:27.463 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_auth", 2024-03-21T10:50:27.463 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.463 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.463 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.463 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.463 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.463 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.463 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.463 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.463 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.463 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.463 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.463 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-03-21T10:50:27.464 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-03-21T10:50:27.464 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.464 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.464 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.464 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.464 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.464 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.464 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.464 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.464 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.464 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.464 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.464 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.464 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.464 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.465 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.465 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.465 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.465 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.465 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.465 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.465 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.465 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.465 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.465 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.465 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.465 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.465 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.465 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.465 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.466 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.466 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.466 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.466 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.466 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.466 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.466 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.466 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.466 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.466 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.466 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.466 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.467 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.467 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.467 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.467 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.467 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.467 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.467 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.467 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.467 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.467 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.467 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.467 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.467 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.467 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.468 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.468 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.468 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.468 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.468 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.468 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.468 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.468 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.468 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.468 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.468 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.468 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.468 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.468 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.469 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.469 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.469 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.469 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.469 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.469 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.469 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.469 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.469 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.469 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-03-21T10:50:27.469 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-03-21T10:50:27.469 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.469 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.470 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.470 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.470 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.470 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.470 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.470 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.470 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.470 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.470 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.470 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-03-21T10:50:27.470 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-03-21T10:50:27.470 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.470 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.471 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.471 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.471 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.471 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.471 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.471 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.471 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.471 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.471 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.471 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.471 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.471 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.471 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rgw", 2024-03-21T10:50:27.471 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.474 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.474 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.474 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.474 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.474 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.474 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.474 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.474 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.474 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.474 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.474 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.474 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.474 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.475 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.475 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.475 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.475 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.475 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.475 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.475 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.475 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.475 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.475 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.475 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.475 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.475 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.476 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.476 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.476 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.476 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.476 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.476 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.476 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.476 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.476 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.476 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.476 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.476 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.476 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.476 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.476 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.477 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.477 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rook", 2024-03-21T10:50:27.477 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.477 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.477 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.477 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.477 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.477 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.477 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.477 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.477 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.477 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.477 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.478 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.478 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.478 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.478 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.478 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.478 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.478 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.478 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.478 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.478 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.478 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.478 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.478 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.478 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.479 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.479 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.479 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.479 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.479 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.479 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.479 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.479 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.479 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.479 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.479 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.479 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.479 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.479 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.480 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.480 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.480 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.480 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.480 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.480 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.480 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.480 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.480 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.480 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.480 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.480 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.480 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.480 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.481 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.481 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.481 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.481 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.481 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.481 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.481 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.481 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.481 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.481 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.481 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.481 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.481 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.481 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.482 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.482 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.482 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.482 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.482 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.482 INFO:teuthology.orchestra.run.smithi031.stdout: "storage_class": { 2024-03-21T10:50:27.482 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "storage_class", 2024-03-21T10:50:27.482 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.482 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.482 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.482 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "local", 2024-03-21T10:50:27.482 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.482 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.482 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.482 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-21T10:50:27.483 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.483 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.483 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.483 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.483 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.483 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.483 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.483 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "selftest", 2024-03-21T10:50:27.483 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.483 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.483 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.483 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.483 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.483 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.483 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.484 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.484 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.484 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.484 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.484 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.484 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.484 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.484 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.484 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.484 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.484 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.484 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.484 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.484 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.484 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.484 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.485 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.485 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.485 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.485 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.485 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.485 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.485 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.485 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.485 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.485 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.485 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.485 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.485 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.486 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.486 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.486 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.486 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.486 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.486 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.486 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.486 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.486 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.486 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.486 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.486 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.486 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.486 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.486 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.487 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.487 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.487 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.487 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.487 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.487 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.487 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.487 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.487 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.487 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.487 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.487 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.487 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.487 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.488 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.488 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.488 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.488 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.488 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.488 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.488 INFO:teuthology.orchestra.run.smithi031.stdout: "roption1": { 2024-03-21T10:50:27.488 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption1", 2024-03-21T10:50:27.488 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.488 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.488 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.488 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.488 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.488 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.488 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.488 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.489 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.489 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.489 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.489 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.489 INFO:teuthology.orchestra.run.smithi031.stdout: "roption2": { 2024-03-21T10:50:27.489 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption2", 2024-03-21T10:50:27.489 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.489 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.489 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.489 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "xyz", 2024-03-21T10:50:27.489 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.489 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.489 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.489 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.490 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.490 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.490 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.490 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.490 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption1": { 2024-03-21T10:50:27.490 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption1", 2024-03-21T10:50:27.490 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.490 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.490 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.490 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.490 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.490 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.490 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.490 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.491 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.491 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.491 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.491 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption2": { 2024-03-21T10:50:27.491 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption2", 2024-03-21T10:50:27.491 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.491 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.491 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.491 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.491 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.491 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.491 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.491 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.492 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.492 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.492 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.492 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.492 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption3": { 2024-03-21T10:50:27.492 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption3", 2024-03-21T10:50:27.492 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-03-21T10:50:27.492 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.492 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.492 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.492 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.492 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.492 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.492 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.492 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.493 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.493 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.493 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.493 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption4": { 2024-03-21T10:50:27.493 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption4", 2024-03-21T10:50:27.493 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.493 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.493 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.493 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.493 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.493 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.493 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.493 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.493 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.494 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.494 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.494 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.494 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption5": { 2024-03-21T10:50:27.494 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption5", 2024-03-21T10:50:27.494 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.494 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.494 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.494 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.494 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.494 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.494 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.494 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.494 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.495 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.495 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.495 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.495 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption6": { 2024-03-21T10:50:27.495 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption6", 2024-03-21T10:50:27.495 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.495 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.495 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.495 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.495 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.495 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.495 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.495 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.495 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.495 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.496 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.496 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.496 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption7": { 2024-03-21T10:50:27.496 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption7", 2024-03-21T10:50:27.496 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.496 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.496 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.496 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.496 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-03-21T10:50:27.496 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "42", 2024-03-21T10:50:27.496 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.496 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.496 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.496 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.496 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.497 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.497 INFO:teuthology.orchestra.run.smithi031.stdout: "testkey": { 2024-03-21T10:50:27.497 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testkey", 2024-03-21T10:50:27.497 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.497 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.497 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.497 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.497 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.497 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.497 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.497 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.497 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.497 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.497 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.497 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.498 INFO:teuthology.orchestra.run.smithi031.stdout: "testlkey": { 2024-03-21T10:50:27.498 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testlkey", 2024-03-21T10:50:27.498 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.498 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.498 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.498 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.498 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.498 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.498 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.498 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.498 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.498 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.498 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.498 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.498 INFO:teuthology.orchestra.run.smithi031.stdout: "testnewline": { 2024-03-21T10:50:27.499 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testnewline", 2024-03-21T10:50:27.499 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.499 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.499 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.499 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.499 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.499 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.499 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.499 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.499 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.499 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.499 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.499 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.499 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.499 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.500 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.500 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snap_schedule", 2024-03-21T10:50:27.500 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.500 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.500 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.500 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_m_granularity": { 2024-03-21T10:50:27.500 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_m_granularity", 2024-03-21T10:50:27.500 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.500 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.500 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.500 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.500 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.500 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.500 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.500 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow minute scheduled snapshots", 2024-03-21T10:50:27.501 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.501 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.501 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.501 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.501 INFO:teuthology.orchestra.run.smithi031.stdout: "dump_on_update": { 2024-03-21T10:50:27.501 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dump_on_update", 2024-03-21T10:50:27.501 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.501 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.501 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.501 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.501 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.501 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.501 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.501 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "dump database to debug log on update", 2024-03-21T10:50:27.502 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.502 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.502 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.502 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.502 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.502 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.502 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.502 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.502 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.502 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.502 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.502 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.502 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.502 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.502 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.503 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.503 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.503 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.503 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.503 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.503 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.503 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.503 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.503 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.503 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.503 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.503 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.503 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.503 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.504 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.504 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.504 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.504 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.504 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.504 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.504 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.504 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.504 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.504 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.504 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.504 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.504 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.504 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.504 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.505 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.505 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.505 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.505 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.505 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.505 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.505 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.505 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.505 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.505 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.505 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.505 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.505 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.505 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.505 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.506 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.506 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.506 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.506 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.506 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.506 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.506 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.506 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.506 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.506 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.506 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.506 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.506 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.506 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.507 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.507 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.507 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.507 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.507 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stats", 2024-03-21T10:50:27.507 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.507 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.507 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.507 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.507 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.507 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.507 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.507 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.507 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.507 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.508 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.508 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.508 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.508 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.508 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.508 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.508 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.508 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.508 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.508 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.508 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.508 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.508 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.508 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.509 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.509 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.509 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.509 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.509 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.509 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.509 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.509 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.509 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.509 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.509 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.509 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.509 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.509 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.509 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.510 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.510 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.510 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.510 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.510 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.510 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.510 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.510 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.510 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.510 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.510 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.510 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.510 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.510 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.510 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.511 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.511 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.511 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.511 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.511 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.511 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.511 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.511 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.511 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.511 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.511 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.511 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.511 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.511 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.511 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.512 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.512 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.512 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.512 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.512 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.512 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "status", 2024-03-21T10:50:27.512 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.512 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.512 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.512 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.512 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.512 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.512 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.512 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.512 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.513 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.513 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.513 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.513 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.513 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.513 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.513 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.513 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.513 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.513 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.513 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.513 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.513 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.513 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.514 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.514 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.514 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.514 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.514 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.514 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.514 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.514 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.514 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.514 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.514 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.514 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.514 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.514 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.515 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.515 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.515 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.515 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.515 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.515 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.515 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.515 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.515 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.515 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.515 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.515 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.515 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.515 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.515 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.515 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.516 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.516 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.516 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.516 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.516 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.516 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.516 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.516 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.516 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.516 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.516 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.516 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.516 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.516 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.516 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.516 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.516 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.517 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.517 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.517 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.517 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.517 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.517 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.517 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telegraf", 2024-03-21T10:50:27.517 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.517 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.517 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.517 INFO:teuthology.orchestra.run.smithi031.stdout: "address": { 2024-03-21T10:50:27.517 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "address", 2024-03-21T10:50:27.517 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.517 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.517 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.517 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-21T10:50:27.518 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.518 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.518 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.518 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.518 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.518 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.518 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.518 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.518 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-03-21T10:50:27.518 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-03-21T10:50:27.518 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.518 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.518 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.519 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15", 2024-03-21T10:50:27.519 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.519 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.519 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.519 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.519 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.519 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.519 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.519 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.519 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.519 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.519 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.519 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.519 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.519 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.520 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.520 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.520 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.520 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.520 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.520 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.520 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.520 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.520 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.520 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.520 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.520 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.520 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.520 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.521 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.521 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.521 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.521 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.521 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.521 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.521 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.521 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.521 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.521 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.521 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.521 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.521 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.521 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.521 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.522 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.522 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.522 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.522 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.522 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.522 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.522 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.522 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.522 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.522 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.522 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.522 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.522 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.523 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.523 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.523 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.523 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.523 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.523 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.523 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.523 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.523 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.523 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.523 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.523 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.523 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.523 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.524 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.524 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.524 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.524 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.524 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.524 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.524 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.524 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.524 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.524 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.524 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.524 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telemetry", 2024-03-21T10:50:27.524 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.524 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.524 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.524 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_basic": { 2024-03-21T10:50:27.525 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_basic", 2024-03-21T10:50:27.525 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.525 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.525 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.525 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.525 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.525 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.525 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.525 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-21T10:50:27.525 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.525 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.525 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.525 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.525 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_crash": { 2024-03-21T10:50:27.525 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_crash", 2024-03-21T10:50:27.526 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.526 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.526 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.526 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.526 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.526 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.526 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.526 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-21T10:50:27.526 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.526 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.526 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.526 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.526 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_device": { 2024-03-21T10:50:27.526 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_device", 2024-03-21T10:50:27.526 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.527 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.527 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.527 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.527 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.527 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.527 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.527 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-21T10:50:27.527 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.527 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.527 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.527 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.527 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_ident": { 2024-03-21T10:50:27.527 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_ident", 2024-03-21T10:50:27.527 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.528 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.528 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.528 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.528 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.528 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.528 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.528 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-21T10:50:27.528 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.528 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.528 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.528 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.528 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_perf": { 2024-03-21T10:50:27.528 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_perf", 2024-03-21T10:50:27.528 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.528 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.528 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.529 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.529 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.529 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.529 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.529 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-21T10:50:27.529 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.529 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.529 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.529 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.529 INFO:teuthology.orchestra.run.smithi031.stdout: "contact": { 2024-03-21T10:50:27.529 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "contact", 2024-03-21T10:50:27.529 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.529 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.529 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.529 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.530 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.530 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.530 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.530 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.530 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.530 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.530 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.530 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.530 INFO:teuthology.orchestra.run.smithi031.stdout: "description": { 2024-03-21T10:50:27.530 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "description", 2024-03-21T10:50:27.530 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.530 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.530 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.530 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.531 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.531 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.531 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.531 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.531 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.531 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.531 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.531 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.531 INFO:teuthology.orchestra.run.smithi031.stdout: "device_url": { 2024-03-21T10:50:27.531 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_url", 2024-03-21T10:50:27.531 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.531 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.531 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.531 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-21T10:50:27.531 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.532 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.532 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.532 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.532 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.532 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.532 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.532 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.532 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-03-21T10:50:27.532 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-03-21T10:50:27.532 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.532 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.532 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.532 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.532 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.532 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.532 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.533 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.533 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.533 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.533 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.533 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.533 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-03-21T10:50:27.533 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-03-21T10:50:27.533 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.533 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.533 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.533 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "24", 2024-03-21T10:50:27.533 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "8", 2024-03-21T10:50:27.533 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.533 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.533 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.534 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.534 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.534 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.534 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.534 INFO:teuthology.orchestra.run.smithi031.stdout: "last_opt_revision": { 2024-03-21T10:50:27.534 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "last_opt_revision", 2024-03-21T10:50:27.534 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.534 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.534 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.534 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1", 2024-03-21T10:50:27.534 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.534 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.534 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.534 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.535 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.535 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.535 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.535 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.535 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard": { 2024-03-21T10:50:27.535 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard", 2024-03-21T10:50:27.535 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.535 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.535 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.535 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.535 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.535 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.535 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.535 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.535 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.535 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.536 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.536 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.536 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard_description": { 2024-03-21T10:50:27.536 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard_description", 2024-03-21T10:50:27.536 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.536 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.536 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.536 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.536 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.536 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.536 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.536 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.536 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.536 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.536 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.537 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.537 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.537 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.537 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.537 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.537 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.537 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.537 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.537 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.537 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.537 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.537 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.537 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.537 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.537 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.537 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.538 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.538 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.538 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.538 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.538 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.538 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.538 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.538 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.538 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.538 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.538 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.538 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.538 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.538 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.538 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.539 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.539 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.539 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.539 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.539 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.539 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.539 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.539 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.539 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.539 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.539 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.539 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.539 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.539 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.539 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.540 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.540 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.540 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.540 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.540 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.540 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.540 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.540 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.540 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.540 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.540 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.540 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.540 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.540 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.540 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.541 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.541 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.541 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.541 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.541 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.541 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.541 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.541 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.541 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.541 INFO:teuthology.orchestra.run.smithi031.stdout: "organization": { 2024-03-21T10:50:27.541 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "organization", 2024-03-21T10:50:27.541 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.541 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.541 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.542 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.542 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.542 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.542 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.542 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.542 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.542 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.542 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.542 INFO:teuthology.orchestra.run.smithi031.stdout: "proxy": { 2024-03-21T10:50:27.542 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "proxy", 2024-03-21T10:50:27.542 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.542 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.542 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.543 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.543 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.543 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.543 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.543 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.543 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.543 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.543 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.543 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.543 INFO:teuthology.orchestra.run.smithi031.stdout: "url": { 2024-03-21T10:50:27.543 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url", 2024-03-21T10:50:27.543 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.543 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.543 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.543 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-21T10:50:27.544 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.544 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.544 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.544 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.544 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.544 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.544 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.544 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.544 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.544 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.544 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.544 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test_orchestrator", 2024-03-21T10:50:27.544 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.544 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.544 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.545 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.545 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.545 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.545 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.545 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.545 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.545 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.545 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.545 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.545 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.545 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.545 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.545 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.545 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.545 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.546 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.546 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.546 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.546 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.546 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.546 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.546 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.546 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.546 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.546 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.546 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.546 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.546 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.546 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.547 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.547 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.547 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.547 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.547 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.547 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.547 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.547 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.547 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.547 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.547 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.547 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.547 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.547 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.548 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.548 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.548 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.548 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.548 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.548 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.548 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.548 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.548 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.548 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.548 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.548 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.548 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.548 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.549 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.549 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.549 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.549 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.549 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.549 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.549 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.549 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.549 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.549 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.549 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.549 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.549 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.549 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.549 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2024-03-21T10:50:27.549 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.550 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.550 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.550 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.550 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.550 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.550 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.550 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.550 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.550 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.550 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.550 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.550 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.550 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.550 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.550 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.551 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.551 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.551 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.551 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.551 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.551 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.551 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.551 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.551 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.551 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.551 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.551 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.551 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.552 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.553 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.553 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.553 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.553 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.553 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.553 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.553 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.553 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.553 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.554 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.554 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.554 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.554 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.554 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.554 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.554 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.554 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.554 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.554 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.554 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.554 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.554 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.554 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.555 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.555 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.555 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.555 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.555 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.555 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.555 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.555 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.555 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.555 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.555 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.555 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.555 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.555 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.556 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.556 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.556 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.556 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_clones": { 2024-03-21T10:50:27.556 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_clones", 2024-03-21T10:50:27.556 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.556 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.556 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.556 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4", 2024-03-21T10:50:27.556 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.556 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.556 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.556 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-21T10:50:27.556 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.557 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.557 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.557 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.557 INFO:teuthology.orchestra.run.smithi031.stdout: "periodic_async_work": { 2024-03-21T10:50:27.557 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "periodic_async_work", 2024-03-21T10:50:27.557 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.557 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.557 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.557 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.557 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.557 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.557 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.557 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Periodically check for async work", 2024-03-21T10:50:27.558 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.558 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.558 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.558 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.558 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_delay": { 2024-03-21T10:50:27.558 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_delay", 2024-03-21T10:50:27.558 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.558 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.558 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.558 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-03-21T10:50:27.558 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.558 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.558 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.559 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-21T10:50:27.559 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.559 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.559 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.559 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_no_wait": { 2024-03-21T10:50:27.559 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_no_wait", 2024-03-21T10:50:27.559 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.559 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.559 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.559 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.559 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.559 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.559 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.560 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-21T10:50:27.560 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.560 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.560 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.560 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.560 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.560 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.560 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix", 2024-03-21T10:50:27.560 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.560 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.560 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.560 INFO:teuthology.orchestra.run.smithi031.stdout: "discovery_interval": { 2024-03-21T10:50:27.560 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "discovery_interval", 2024-03-21T10:50:27.560 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-03-21T10:50:27.561 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.561 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.561 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "100", 2024-03-21T10:50:27.561 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.561 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.561 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.561 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.561 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.561 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.561 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.561 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.561 INFO:teuthology.orchestra.run.smithi031.stdout: "identifier": { 2024-03-21T10:50:27.561 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "identifier", 2024-03-21T10:50:27.561 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.562 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.562 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.562 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.562 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.562 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.562 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.562 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.562 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.562 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.562 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.562 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-03-21T10:50:27.562 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-03-21T10:50:27.562 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.563 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.563 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.563 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-03-21T10:50:27.563 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.563 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.563 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.563 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.563 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.563 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.563 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.563 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.563 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.563 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.563 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.564 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.564 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.564 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.564 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.564 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.564 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.564 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.564 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.564 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.564 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.564 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.564 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.564 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.564 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.564 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.565 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.565 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.565 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.565 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.565 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.565 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.565 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.565 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.565 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.565 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.565 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.565 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.565 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.565 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.566 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.566 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.566 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.566 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.566 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.566 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.566 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.566 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.566 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.566 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.566 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.566 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.566 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.566 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.566 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.567 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.567 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.567 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.567 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.567 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.567 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.567 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.567 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.567 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.567 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.567 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.567 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.567 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.567 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_host": { 2024-03-21T10:50:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_host", 2024-03-21T10:50:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_port": { 2024-03-21T10:50:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_port", 2024-03-21T10:50:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10051", 2024-03-21T10:50:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.570 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.570 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.570 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.570 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.570 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.570 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_sender": { 2024-03-21T10:50:27.570 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_sender", 2024-03-21T10:50:27.570 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.570 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.570 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.570 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-21T10:50:27.570 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-03-21T10:50:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: "modules": [ 2024-03-21T10:50:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm", 2024-03-21T10:50:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: "dashboard", 2024-03-21T10:50:27.572 INFO:teuthology.orchestra.run.smithi031.stdout: "iostat", 2024-03-21T10:50:27.572 INFO:teuthology.orchestra.run.smithi031.stdout: "nfs", 2024-03-21T10:50:27.572 INFO:teuthology.orchestra.run.smithi031.stdout: "restful", 2024-03-21T10:50:27.572 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_schedule" 2024-03-21T10:50:27.572 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.572 INFO:teuthology.orchestra.run.smithi031.stdout: "available_modules": [ 2024-03-21T10:50:27.572 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.572 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "alerts", 2024-03-21T10:50:27.572 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.572 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.572 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.572 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-03-21T10:50:27.572 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-03-21T10:50:27.572 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.572 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.573 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.573 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-03-21T10:50:27.573 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.573 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.573 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.573 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How frequently to reexamine health status", 2024-03-21T10:50:27.573 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.573 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.573 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.573 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.573 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.573 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.573 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.573 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.574 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.574 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.574 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.574 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.574 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.574 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.574 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.574 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.574 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.574 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.574 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.574 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.574 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.574 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.575 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.575 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.575 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.575 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.575 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.575 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.575 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.575 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.575 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.575 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.575 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.575 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.575 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.575 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.576 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.576 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.576 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.576 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.576 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.576 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.576 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.576 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.576 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.576 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.576 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.576 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.576 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.576 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.577 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.577 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.577 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.577 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.577 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.577 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.577 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.577 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.577 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.577 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.577 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.577 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.577 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.577 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.578 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.578 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.578 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.578 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.578 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.578 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_destination": { 2024-03-21T10:50:27.578 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_destination", 2024-03-21T10:50:27.578 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.578 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.578 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.578 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.578 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.578 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.578 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.579 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email address to send alerts to", 2024-03-21T10:50:27.579 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.579 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.579 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.579 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_from_name": { 2024-03-21T10:50:27.579 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_from_name", 2024-03-21T10:50:27.579 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.579 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.579 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.579 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "Ceph", 2024-03-21T10:50:27.579 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.579 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.579 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.579 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email From: name", 2024-03-21T10:50:27.580 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.580 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.580 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.580 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_host": { 2024-03-21T10:50:27.580 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_host", 2024-03-21T10:50:27.580 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.580 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.580 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.580 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.580 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.580 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.580 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.580 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP server", 2024-03-21T10:50:27.580 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.580 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.581 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.581 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_password": { 2024-03-21T10:50:27.581 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_password", 2024-03-21T10:50:27.581 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.581 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.581 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.581 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.581 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.581 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.581 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.581 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Password to authenticate with", 2024-03-21T10:50:27.581 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.581 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.581 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.582 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_port": { 2024-03-21T10:50:27.582 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_port", 2024-03-21T10:50:27.582 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.582 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.582 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.582 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "465", 2024-03-21T10:50:27.582 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.582 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.582 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.582 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP port", 2024-03-21T10:50:27.582 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.582 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.582 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.583 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_sender": { 2024-03-21T10:50:27.583 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_sender", 2024-03-21T10:50:27.583 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.583 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.583 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.583 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.583 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.583 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.583 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.583 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP envelope sender", 2024-03-21T10:50:27.583 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.583 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.583 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.584 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_ssl": { 2024-03-21T10:50:27.584 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_ssl", 2024-03-21T10:50:27.584 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.584 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.584 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.584 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.584 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.584 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.584 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.584 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-21T10:50:27.584 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.584 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.584 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.584 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_user": { 2024-03-21T10:50:27.584 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_user", 2024-03-21T10:50:27.585 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.585 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.585 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.585 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.585 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.585 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.585 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.585 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "User to authenticate as", 2024-03-21T10:50:27.585 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.585 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.585 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.585 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.585 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.586 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.586 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "balancer", 2024-03-21T10:50:27.586 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.586 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.586 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.586 INFO:teuthology.orchestra.run.smithi031.stdout: "active": { 2024-03-21T10:50:27.586 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "active", 2024-03-21T10:50:27.586 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.586 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.586 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.586 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.586 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.586 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.587 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.587 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically balance PGs across cluster", 2024-03-21T10:50:27.587 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.587 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.587 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.587 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_time": { 2024-03-21T10:50:27.587 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_time", 2024-03-21T10:50:27.587 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.587 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.587 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.587 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0000", 2024-03-21T10:50:27.587 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.587 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.588 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.588 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "beginning time of day to automatically balance", 2024-03-21T10:50:27.588 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-21T10:50:27.588 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.588 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.588 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_weekday": { 2024-03-21T10:50:27.588 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_weekday", 2024-03-21T10:50:27.588 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-03-21T10:50:27.588 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.588 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.588 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-03-21T10:50:27.588 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-03-21T10:50:27.588 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-03-21T10:50:27.589 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.589 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-21T10:50:27.589 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-21T10:50:27.589 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.589 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.589 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_max_iterations": { 2024-03-21T10:50:27.589 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_max_iterations", 2024-03-21T10:50:27.589 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-03-21T10:50:27.589 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.589 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.589 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "25", 2024-03-21T10:50:27.589 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-03-21T10:50:27.590 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "250", 2024-03-21T10:50:27.590 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.590 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-21T10:50:27.590 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.590 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.590 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.590 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.590 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_metrics": { 2024-03-21T10:50:27.590 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_metrics", 2024-03-21T10:50:27.590 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.590 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.590 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.590 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "pgs,objects,bytes", 2024-03-21T10:50:27.590 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.591 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.591 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.591 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-21T10:50:27.591 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-21T10:50:27.591 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.591 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.591 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_step": { 2024-03-21T10:50:27.591 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_step", 2024-03-21T10:50:27.591 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-03-21T10:50:27.591 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.591 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.591 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.5", 2024-03-21T10:50:27.591 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0.001", 2024-03-21T10:50:27.592 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "0.999", 2024-03-21T10:50:27.592 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.592 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "aggressiveness of optimization", 2024-03-21T10:50:27.592 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-21T10:50:27.592 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.592 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.592 INFO:teuthology.orchestra.run.smithi031.stdout: "end_time": { 2024-03-21T10:50:27.592 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_time", 2024-03-21T10:50:27.592 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.592 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.592 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.592 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2359", 2024-03-21T10:50:27.592 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.592 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.593 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.593 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ending time of day to automatically balance", 2024-03-21T10:50:27.593 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-21T10:50:27.593 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.593 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.593 INFO:teuthology.orchestra.run.smithi031.stdout: "end_weekday": { 2024-03-21T10:50:27.593 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_weekday", 2024-03-21T10:50:27.593 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-03-21T10:50:27.593 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.593 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.593 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-03-21T10:50:27.593 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-03-21T10:50:27.593 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-03-21T10:50:27.594 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.594 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-21T10:50:27.594 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-21T10:50:27.594 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.594 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.594 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.594 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.594 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.594 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.594 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.594 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.594 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.594 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.595 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.595 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.595 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.595 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.595 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.595 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.595 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.595 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.595 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.595 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.595 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.595 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.595 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.596 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.596 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.596 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.596 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.596 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.596 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.596 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.596 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.596 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.596 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.596 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.596 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.597 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.597 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.597 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.597 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.597 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.597 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.597 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.597 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.597 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.597 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.597 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.597 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.598 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.598 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.598 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.598 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.598 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.598 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.598 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.598 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.598 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.598 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.598 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.598 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.598 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.598 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.599 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.599 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.599 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.599 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.599 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.599 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.599 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.599 INFO:teuthology.orchestra.run.smithi031.stdout: "min_score": { 2024-03-21T10:50:27.599 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_score", 2024-03-21T10:50:27.599 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-03-21T10:50:27.599 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.599 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.599 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-03-21T10:50:27.600 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.600 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.600 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.600 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-21T10:50:27.600 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.600 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.600 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.600 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-03-21T10:50:27.600 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-03-21T10:50:27.600 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.600 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.600 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.600 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "upmap", 2024-03-21T10:50:27.600 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.601 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.601 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.601 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-compat", 2024-03-21T10:50:27.601 INFO:teuthology.orchestra.run.smithi031.stdout: "none", 2024-03-21T10:50:27.601 INFO:teuthology.orchestra.run.smithi031.stdout: "read", 2024-03-21T10:50:27.601 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap", 2024-03-21T10:50:27.601 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap-read" 2024-03-21T10:50:27.601 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.601 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Balancer mode", 2024-03-21T10:50:27.601 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.601 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.601 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.601 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_ids": { 2024-03-21T10:50:27.601 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_ids", 2024-03-21T10:50:27.602 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.602 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.602 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.602 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.602 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.602 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.602 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.602 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-21T10:50:27.602 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.602 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.602 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.602 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.603 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-03-21T10:50:27.603 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-03-21T10:50:27.603 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.603 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.603 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.603 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-03-21T10:50:27.603 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.603 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.603 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.603 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-21T10:50:27.603 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.603 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.603 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.603 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_deviation": { 2024-03-21T10:50:27.604 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_deviation", 2024-03-21T10:50:27.604 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.604 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.604 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.604 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-03-21T10:50:27.604 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-03-21T10:50:27.604 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.604 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.604 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-21T10:50:27.604 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-21T10:50:27.604 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.604 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.604 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.605 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_optimizations": { 2024-03-21T10:50:27.605 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_optimizations", 2024-03-21T10:50:27.605 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-03-21T10:50:27.605 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.605 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.605 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-03-21T10:50:27.605 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.605 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.605 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.605 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-21T10:50:27.605 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.605 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.605 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.605 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.605 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.606 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.606 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.606 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm", 2024-03-21T10:50:27.606 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.606 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.606 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.606 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_down_multiplier": { 2024-03-21T10:50:27.606 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_down_multiplier", 2024-03-21T10:50:27.606 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-03-21T10:50:27.606 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.606 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.606 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-03-21T10:50:27.606 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.606 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.607 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.607 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-21T10:50:27.607 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.607 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.607 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.607 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_refresh_rate": { 2024-03-21T10:50:27.607 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_refresh_rate", 2024-03-21T10:50:27.607 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.607 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.607 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.607 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "20", 2024-03-21T10:50:27.607 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.608 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.608 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.608 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-21T10:50:27.608 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.608 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.608 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.608 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_starting_port": { 2024-03-21T10:50:27.608 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_starting_port", 2024-03-21T10:50:27.608 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.608 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.608 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.608 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4721", 2024-03-21T10:50:27.608 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.609 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.609 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.609 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-21T10:50:27.609 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.609 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.609 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.609 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_ptrace": { 2024-03-21T10:50:27.609 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_ptrace", 2024-03-21T10:50:27.609 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.609 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.609 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.609 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.609 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.609 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.609 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.610 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-21T10:50:27.610 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-21T10:50:27.610 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.610 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.610 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_interval": { 2024-03-21T10:50:27.610 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_interval", 2024-03-21T10:50:27.610 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.610 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.610 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.610 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-03-21T10:50:27.610 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.610 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.610 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.611 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-21T10:50:27.611 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.611 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.611 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.611 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_memory_target_ratio": { 2024-03-21T10:50:27.611 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_memory_target_ratio", 2024-03-21T10:50:27.611 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-03-21T10:50:27.611 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.611 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.611 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.7", 2024-03-21T10:50:27.611 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.611 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.611 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.611 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-21T10:50:27.612 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.612 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.612 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.612 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm_log_destination": { 2024-03-21T10:50:27.612 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm_log_destination", 2024-03-21T10:50:27.612 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.612 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.612 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.612 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.612 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.612 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.612 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.612 INFO:teuthology.orchestra.run.smithi031.stdout: "file", 2024-03-21T10:50:27.612 INFO:teuthology.orchestra.run.smithi031.stdout: "file,syslog", 2024-03-21T10:50:27.613 INFO:teuthology.orchestra.run.smithi031.stdout: "syslog" 2024-03-21T10:50:27.613 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.613 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-21T10:50:27.613 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.613 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.613 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.613 INFO:teuthology.orchestra.run.smithi031.stdout: "cgroups_split": { 2024-03-21T10:50:27.613 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cgroups_split", 2024-03-21T10:50:27.613 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.613 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.613 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.613 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.613 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.613 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.613 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.614 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-21T10:50:27.614 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.614 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.614 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.614 INFO:teuthology.orchestra.run.smithi031.stdout: "config_checks_enabled": { 2024-03-21T10:50:27.614 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_checks_enabled", 2024-03-21T10:50:27.614 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.614 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.614 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.614 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.614 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.614 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.614 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.614 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-21T10:50:27.615 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.615 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.615 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.615 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.615 INFO:teuthology.orchestra.run.smithi031.stdout: "config_dashboard": { 2024-03-21T10:50:27.615 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_dashboard", 2024-03-21T10:50:27.615 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.615 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.615 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.615 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.615 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.615 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.615 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.615 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-21T10:50:27.616 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.616 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.616 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.616 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_alertmanager": { 2024-03-21T10:50:27.616 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_alertmanager", 2024-03-21T10:50:27.616 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.616 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.616 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.616 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-21T10:50:27.616 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.616 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.616 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.616 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-03-21T10:50:27.616 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.617 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.617 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.617 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_base": { 2024-03-21T10:50:27.617 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_base", 2024-03-21T10:50:27.617 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.617 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.617 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.617 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-21T10:50:27.617 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.617 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.617 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.617 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Container image name, without the tag", 2024-03-21T10:50:27.617 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.617 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.618 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.618 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_elasticsearch": { 2024-03-21T10:50:27.618 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_elasticsearch", 2024-03-21T10:50:27.618 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.618 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.618 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.618 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-21T10:50:27.618 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.618 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.618 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.618 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "elasticsearch container image", 2024-03-21T10:50:27.618 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.618 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.619 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.619 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_grafana": { 2024-03-21T10:50:27.619 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_grafana", 2024-03-21T10:50:27.619 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.619 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.619 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.619 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-03-21T10:50:27.619 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.619 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.619 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.619 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-03-21T10:50:27.619 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.619 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.619 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.620 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_haproxy": { 2024-03-21T10:50:27.620 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_haproxy", 2024-03-21T10:50:27.620 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.620 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.620 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.620 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-21T10:50:27.620 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.620 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.620 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.620 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "HAproxy container image", 2024-03-21T10:50:27.620 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.620 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.620 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.621 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_agent": { 2024-03-21T10:50:27.621 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_agent", 2024-03-21T10:50:27.621 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.621 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.621 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.621 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-21T10:50:27.621 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.621 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.621 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.621 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger agent container image", 2024-03-21T10:50:27.621 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.621 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.621 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.622 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_collector": { 2024-03-21T10:50:27.622 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_collector", 2024-03-21T10:50:27.622 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.622 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.622 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.622 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-21T10:50:27.622 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.622 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.622 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.622 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger collector container image", 2024-03-21T10:50:27.622 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.622 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.622 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.622 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_query": { 2024-03-21T10:50:27.623 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_query", 2024-03-21T10:50:27.623 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.623 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.623 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.623 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-21T10:50:27.623 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.623 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.623 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.623 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger query container image", 2024-03-21T10:50:27.623 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.623 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.623 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.623 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_keepalived": { 2024-03-21T10:50:27.623 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_keepalived", 2024-03-21T10:50:27.624 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.624 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.624 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.624 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-21T10:50:27.624 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.624 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.624 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.624 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Keepalived container image", 2024-03-21T10:50:27.624 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.624 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.624 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.624 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_loki": { 2024-03-21T10:50:27.624 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_loki", 2024-03-21T10:50:27.624 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.624 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.625 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.625 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-21T10:50:27.625 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.625 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.625 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.625 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Loki container image", 2024-03-21T10:50:27.625 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.625 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.625 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.625 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_node_exporter": { 2024-03-21T10:50:27.625 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_node_exporter", 2024-03-21T10:50:27.625 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.625 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.625 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.626 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-21T10:50:27.626 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.626 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.626 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.626 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-03-21T10:50:27.626 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.626 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.626 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.626 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_nvmeof": { 2024-03-21T10:50:27.626 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_nvmeof", 2024-03-21T10:50:27.626 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.626 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.626 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.626 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-21T10:50:27.627 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.627 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.627 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.627 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Nvme-of container image", 2024-03-21T10:50:27.627 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.627 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.627 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.627 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_prometheus": { 2024-03-21T10:50:27.627 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_prometheus", 2024-03-21T10:50:27.627 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.627 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.627 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.627 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-21T10:50:27.627 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.628 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.628 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.628 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-03-21T10:50:27.628 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.628 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.628 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.628 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_promtail": { 2024-03-21T10:50:27.628 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_promtail", 2024-03-21T10:50:27.628 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.628 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.628 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.628 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-21T10:50:27.628 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.628 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.628 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.629 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Promtail container image", 2024-03-21T10:50:27.629 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.629 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.629 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.629 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_snmp_gateway": { 2024-03-21T10:50:27.629 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_snmp_gateway", 2024-03-21T10:50:27.629 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.629 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.629 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.629 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-21T10:50:27.629 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.629 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.629 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.630 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SNMP Gateway container image", 2024-03-21T10:50:27.630 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.630 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.630 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.630 INFO:teuthology.orchestra.run.smithi031.stdout: "container_init": { 2024-03-21T10:50:27.630 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_init", 2024-03-21T10:50:27.630 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.630 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.630 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.630 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.630 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.630 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.630 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.630 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Run podman/docker with `--init`", 2024-03-21T10:50:27.631 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.631 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.631 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.631 INFO:teuthology.orchestra.run.smithi031.stdout: "daemon_cache_timeout": { 2024-03-21T10:50:27.631 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "daemon_cache_timeout", 2024-03-21T10:50:27.631 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.631 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.631 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.631 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-03-21T10:50:27.631 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.631 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.631 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.631 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-21T10:50:27.632 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.632 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.632 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.632 INFO:teuthology.orchestra.run.smithi031.stdout: "default_cephadm_command_timeout": { 2024-03-21T10:50:27.632 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_cephadm_command_timeout", 2024-03-21T10:50:27.632 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.632 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.632 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.632 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "900", 2024-03-21T10:50:27.632 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.632 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.632 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.632 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-21T10:50:27.632 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.633 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.633 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.633 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.633 INFO:teuthology.orchestra.run.smithi031.stdout: "default_registry": { 2024-03-21T10:50:27.633 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_registry", 2024-03-21T10:50:27.633 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.633 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.633 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.633 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io", 2024-03-21T10:50:27.633 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.633 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.633 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.633 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-21T10:50:27.633 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.633 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.634 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.634 INFO:teuthology.orchestra.run.smithi031.stdout: "device_cache_timeout": { 2024-03-21T10:50:27.634 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_cache_timeout", 2024-03-21T10:50:27.634 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.634 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.634 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.634 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1800", 2024-03-21T10:50:27.634 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.634 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.634 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.634 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache device inventory", 2024-03-21T10:50:27.634 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.634 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.635 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.635 INFO:teuthology.orchestra.run.smithi031.stdout: "device_enhanced_scan": { 2024-03-21T10:50:27.635 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_enhanced_scan", 2024-03-21T10:50:27.635 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.635 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.635 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.635 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.635 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.635 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.635 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.635 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-21T10:50:27.635 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.635 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.635 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.636 INFO:teuthology.orchestra.run.smithi031.stdout: "facts_cache_timeout": { 2024-03-21T10:50:27.636 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "facts_cache_timeout", 2024-03-21T10:50:27.636 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.636 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.636 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.636 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-03-21T10:50:27.636 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.636 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.636 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.636 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache host facts data", 2024-03-21T10:50:27.636 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.636 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.636 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.637 INFO:teuthology.orchestra.run.smithi031.stdout: "grafana_dashboards_path": { 2024-03-21T10:50:27.637 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "grafana_dashboards_path", 2024-03-21T10:50:27.637 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.637 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.637 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.637 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-03-21T10:50:27.637 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.637 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.637 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.637 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-03-21T10:50:27.637 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.637 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.637 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.637 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.638 INFO:teuthology.orchestra.run.smithi031.stdout: "host_check_interval": { 2024-03-21T10:50:27.638 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "host_check_interval", 2024-03-21T10:50:27.638 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.638 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.638 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.638 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-03-21T10:50:27.638 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.638 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.638 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.638 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to perform a host check", 2024-03-21T10:50:27.638 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.638 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.638 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.638 INFO:teuthology.orchestra.run.smithi031.stdout: "hw_monitoring": { 2024-03-21T10:50:27.639 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hw_monitoring", 2024-03-21T10:50:27.639 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.639 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.639 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.639 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.639 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.639 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.639 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.639 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-21T10:50:27.639 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.639 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.639 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.639 INFO:teuthology.orchestra.run.smithi031.stdout: "inventory_list_all": { 2024-03-21T10:50:27.639 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "inventory_list_all", 2024-03-21T10:50:27.640 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.640 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.640 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.640 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.640 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.640 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.640 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.640 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-21T10:50:27.640 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.640 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.640 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.640 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.640 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.640 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.641 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.641 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.641 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.641 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.641 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.641 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.641 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.641 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.641 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.641 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.641 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.641 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.641 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.641 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.641 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.642 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.642 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.642 INFO:teuthology.orchestra.run.smithi031.stdout: "log_refresh_metadata": { 2024-03-21T10:50:27.642 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_refresh_metadata", 2024-03-21T10:50:27.642 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.642 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.642 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.642 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.642 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.642 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.642 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.642 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-21T10:50:27.642 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.642 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.643 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.643 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.643 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.643 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.643 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.643 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.643 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.643 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.643 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.643 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.643 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-21T10:50:27.643 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.643 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.644 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.644 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.644 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.644 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.644 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.644 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.644 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.644 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.644 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.644 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.644 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.644 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.644 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.644 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.645 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.645 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.645 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.645 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.645 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.645 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.645 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.645 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.645 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.645 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.645 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.645 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.645 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.646 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.646 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.646 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.646 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.646 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.646 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.646 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.646 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-21T10:50:27.646 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-21T10:50:27.646 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.646 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.646 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.646 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.646 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.647 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.647 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.647 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-21T10:50:27.647 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.647 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.647 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.647 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-21T10:50:27.647 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-21T10:50:27.647 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.647 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.647 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.647 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "*", 2024-03-21T10:50:27.647 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.648 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.648 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.648 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-21T10:50:27.648 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.648 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.648 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.648 INFO:teuthology.orchestra.run.smithi031.stdout: "max_count_per_host": { 2024-03-21T10:50:27.648 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_count_per_host", 2024-03-21T10:50:27.648 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.648 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.648 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.648 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-03-21T10:50:27.648 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.648 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.649 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.649 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of daemons per service per host", 2024-03-21T10:50:27.649 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.649 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.649 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.649 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd_draining_count": { 2024-03-21T10:50:27.649 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_osd_draining_count", 2024-03-21T10:50:27.649 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.649 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.649 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.649 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-03-21T10:50:27.649 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.649 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.649 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.650 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-21T10:50:27.650 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.650 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.650 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.650 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.650 INFO:teuthology.orchestra.run.smithi031.stdout: "migration_current": { 2024-03-21T10:50:27.650 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "migration_current", 2024-03-21T10:50:27.650 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.650 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.650 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.650 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.650 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.650 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.650 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.650 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "internal - do not modify", 2024-03-21T10:50:27.651 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.651 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.651 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.651 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-03-21T10:50:27.651 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-03-21T10:50:27.651 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.651 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.651 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.651 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "root", 2024-03-21T10:50:27.651 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.651 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.651 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.651 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm-package", 2024-03-21T10:50:27.651 INFO:teuthology.orchestra.run.smithi031.stdout: "root" 2024-03-21T10:50:27.652 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.652 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "mode for remote execution of cephadm", 2024-03-21T10:50:27.652 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.652 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.652 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.652 INFO:teuthology.orchestra.run.smithi031.stdout: "oob_default_addr": { 2024-03-21T10:50:27.652 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "oob_default_addr", 2024-03-21T10:50:27.652 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.652 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.652 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.652 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "169.254.1.1", 2024-03-21T10:50:27.652 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.652 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.652 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.653 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-21T10:50:27.653 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.653 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.653 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.653 INFO:teuthology.orchestra.run.smithi031.stdout: "prometheus_alerts_path": { 2024-03-21T10:50:27.653 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus_alerts_path", 2024-03-21T10:50:27.653 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.653 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.653 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.653 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-21T10:50:27.653 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.653 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.653 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.653 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-21T10:50:27.653 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.654 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.654 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.654 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_insecure": { 2024-03-21T10:50:27.654 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_insecure", 2024-03-21T10:50:27.654 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.654 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.654 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.654 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.654 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.654 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.654 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.654 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-21T10:50:27.654 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.655 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.655 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.655 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_password": { 2024-03-21T10:50:27.655 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_password", 2024-03-21T10:50:27.655 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.655 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.655 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.655 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.655 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.655 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.655 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.655 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-21T10:50:27.655 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.655 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.655 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.656 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_url": { 2024-03-21T10:50:27.656 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_url", 2024-03-21T10:50:27.656 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.656 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.656 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.656 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.656 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.656 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.656 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.656 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-21T10:50:27.656 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.656 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.656 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.657 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_username": { 2024-03-21T10:50:27.657 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_username", 2024-03-21T10:50:27.657 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.657 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.657 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.657 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.657 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.657 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.657 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.657 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-21T10:50:27.657 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.657 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.657 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.658 INFO:teuthology.orchestra.run.smithi031.stdout: "secure_monitoring_stack": { 2024-03-21T10:50:27.658 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "secure_monitoring_stack", 2024-03-21T10:50:27.658 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.658 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.658 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.658 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.658 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.658 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.658 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.658 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-21T10:50:27.658 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.658 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.658 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.659 INFO:teuthology.orchestra.run.smithi031.stdout: "service_discovery_port": { 2024-03-21T10:50:27.659 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "service_discovery_port", 2024-03-21T10:50:27.659 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.659 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.659 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.659 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8765", 2024-03-21T10:50:27.659 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.659 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.659 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.659 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "cephadm service discovery port", 2024-03-21T10:50:27.659 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.659 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.659 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.659 INFO:teuthology.orchestra.run.smithi031.stdout: "ssh_config_file": { 2024-03-21T10:50:27.660 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssh_config_file", 2024-03-21T10:50:27.660 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.660 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.660 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.660 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.660 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.660 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.660 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.660 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-21T10:50:27.660 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.660 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.660 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.660 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.660 INFO:teuthology.orchestra.run.smithi031.stdout: "use_agent": { 2024-03-21T10:50:27.660 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_agent", 2024-03-21T10:50:27.660 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.661 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.661 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.661 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.661 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.661 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.661 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.661 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-21T10:50:27.661 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.661 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.661 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.661 INFO:teuthology.orchestra.run.smithi031.stdout: "use_repo_digest": { 2024-03-21T10:50:27.661 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_repo_digest", 2024-03-21T10:50:27.661 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.661 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.662 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.662 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.662 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.662 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.662 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.662 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-21T10:50:27.662 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.662 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.662 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.662 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_failed_host_check": { 2024-03-21T10:50:27.662 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_failed_host_check", 2024-03-21T10:50:27.662 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.662 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.663 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.663 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.663 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.663 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.663 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.663 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if the host check fails", 2024-03-21T10:50:27.663 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.663 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.663 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.663 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_daemons": { 2024-03-21T10:50:27.663 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_daemons", 2024-03-21T10:50:27.663 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.663 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.663 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.664 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.664 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.664 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.664 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.664 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-21T10:50:27.664 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.664 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.664 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.664 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_hosts": { 2024-03-21T10:50:27.664 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_hosts", 2024-03-21T10:50:27.664 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.664 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.664 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.664 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.665 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.665 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.665 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.665 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-21T10:50:27.665 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.665 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.665 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.665 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.665 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.665 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.665 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crash", 2024-03-21T10:50:27.665 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.665 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.666 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.666 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.666 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.666 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.666 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.666 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.666 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.666 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.666 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.666 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.666 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.666 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.666 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.666 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.666 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.667 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.667 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.667 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.667 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.667 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.667 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.667 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.667 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.667 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.667 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.667 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.667 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.667 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.667 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.668 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.668 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.668 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.668 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.668 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.668 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.668 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.668 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.668 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.668 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.668 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.668 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.668 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.668 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.669 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.669 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.669 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.669 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.669 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.669 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.669 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.669 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.669 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.669 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.669 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.669 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.669 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.669 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.669 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.670 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.670 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.670 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.670 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.670 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.670 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.670 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.670 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.670 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.670 INFO:teuthology.orchestra.run.smithi031.stdout: "retain_interval": { 2024-03-21T10:50:27.670 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retain_interval", 2024-03-21T10:50:27.670 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.670 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.670 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.671 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "31536000", 2024-03-21T10:50:27.671 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.671 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.671 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.671 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-21T10:50:27.671 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.671 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.671 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.671 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_recent_interval": { 2024-03-21T10:50:27.671 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_recent_interval", 2024-03-21T10:50:27.671 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.671 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.671 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.671 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1209600", 2024-03-21T10:50:27.672 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.672 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.672 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.672 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-21T10:50:27.672 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.672 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.672 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.672 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.672 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.672 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.672 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dashboard", 2024-03-21T10:50:27.672 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.672 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.672 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.673 INFO:teuthology.orchestra.run.smithi031.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-21T10:50:27.673 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-21T10:50:27.673 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.673 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.673 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.673 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-03-21T10:50:27.673 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.673 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.673 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.673 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.673 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.673 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.673 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.673 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_HOST": { 2024-03-21T10:50:27.674 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-21T10:50:27.674 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.674 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.674 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.674 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.674 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.674 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.674 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.674 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.674 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.674 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.674 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.674 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-21T10:50:27.675 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-21T10:50:27.675 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.675 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.675 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.675 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.675 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.675 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.675 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.675 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.675 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.675 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.675 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.675 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_ENABLED": { 2024-03-21T10:50:27.675 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_ENABLED", 2024-03-21T10:50:27.676 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.676 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.676 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.676 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.676 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.676 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.676 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.676 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.676 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.676 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.676 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.676 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-21T10:50:27.676 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-21T10:50:27.676 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.676 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.677 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.677 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.677 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.677 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.677 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.677 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.677 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.677 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.677 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.677 INFO:teuthology.orchestra.run.smithi031.stdout: "ENABLE_BROWSABLE_API": { 2024-03-21T10:50:27.677 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-21T10:50:27.677 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.677 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.677 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.678 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.678 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.678 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.678 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.678 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.678 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.678 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.678 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.678 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.678 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-21T10:50:27.678 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-21T10:50:27.678 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.678 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.678 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.679 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.679 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.679 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.679 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.679 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.679 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.679 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.679 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.679 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-21T10:50:27.679 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-21T10:50:27.679 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.679 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.679 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.679 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.680 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.680 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.680 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.680 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.680 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.680 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.680 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.680 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-21T10:50:27.680 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-21T10:50:27.680 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.680 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.680 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.680 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.680 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.681 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.681 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.681 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.681 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.681 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.681 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.681 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.681 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-21T10:50:27.681 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-21T10:50:27.681 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.681 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.681 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.681 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.681 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.681 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.681 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.682 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.682 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.682 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.682 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.682 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-21T10:50:27.682 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-21T10:50:27.682 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.682 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.682 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.682 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.682 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.682 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.682 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.683 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.683 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.683 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.683 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.683 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-21T10:50:27.683 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-21T10:50:27.683 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.683 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.683 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.683 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.683 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.683 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.683 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.683 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.683 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.684 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.684 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.684 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-21T10:50:27.684 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-21T10:50:27.684 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.684 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.684 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.684 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.684 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.684 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.684 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.684 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.684 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.684 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.684 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.685 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.685 INFO:teuthology.orchestra.run.smithi031.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-21T10:50:27.685 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-21T10:50:27.685 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.685 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.685 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.685 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.685 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.685 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.685 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.685 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.685 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.685 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.685 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.685 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.686 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_PASSWORD": { 2024-03-21T10:50:27.686 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-21T10:50:27.686 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.686 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.686 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.686 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-03-21T10:50:27.686 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.686 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.686 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.686 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.686 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.686 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.686 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.686 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.686 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-21T10:50:27.687 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-21T10:50:27.687 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.687 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.687 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.687 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.687 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.687 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.687 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.687 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.687 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.687 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.687 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.687 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_URL": { 2024-03-21T10:50:27.687 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_URL", 2024-03-21T10:50:27.688 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.688 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.688 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.688 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.688 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.688 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.688 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.688 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.688 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.688 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.688 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.688 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.688 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_USERNAME": { 2024-03-21T10:50:27.688 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-21T10:50:27.688 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.689 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.689 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.689 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-03-21T10:50:27.689 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.689 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.689 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.689 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.689 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.689 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.689 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.689 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.689 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-21T10:50:27.689 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-21T10:50:27.689 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.689 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.690 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.690 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.690 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.690 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.690 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.690 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.690 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.690 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.690 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.690 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-21T10:50:27.690 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-21T10:50:27.690 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.690 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.690 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.691 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.691 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.691 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.691 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.691 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.691 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.691 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.691 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.691 INFO:teuthology.orchestra.run.smithi031.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-21T10:50:27.691 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-21T10:50:27.691 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.691 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.691 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.691 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.692 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.692 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.692 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.692 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.692 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.692 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.692 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.692 INFO:teuthology.orchestra.run.smithi031.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-21T10:50:27.692 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-21T10:50:27.692 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.692 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.692 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.692 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.692 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.693 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.693 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.693 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.693 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.693 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.693 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.693 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.693 INFO:teuthology.orchestra.run.smithi031.stdout: "MULTICLUSTER_CONFIG": { 2024-03-21T10:50:27.693 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-21T10:50:27.693 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.693 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.693 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.693 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "{}", 2024-03-21T10:50:27.693 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.693 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.694 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.694 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.694 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.694 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.694 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.694 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_HOST": { 2024-03-21T10:50:27.694 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-21T10:50:27.694 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.694 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.694 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.694 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.694 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.694 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.695 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.695 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.695 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.695 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.695 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.695 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-21T10:50:27.695 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-21T10:50:27.695 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.695 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.695 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.695 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.695 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.695 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.695 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.696 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.696 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.696 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.696 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.696 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-21T10:50:27.696 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-21T10:50:27.696 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.696 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.696 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.696 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.696 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.696 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.696 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.696 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.697 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.697 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.697 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.697 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-21T10:50:27.697 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-21T10:50:27.697 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.697 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.697 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.697 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.697 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.697 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.697 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.697 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.697 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.698 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.698 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.698 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-21T10:50:27.698 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-21T10:50:27.698 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.698 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.698 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.698 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.698 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.698 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.698 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.698 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.698 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.698 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.699 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.699 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-21T10:50:27.699 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-21T10:50:27.699 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.699 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.699 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.699 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.699 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.699 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.699 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.699 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.699 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.699 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.699 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.700 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-21T10:50:27.700 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-21T10:50:27.700 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.700 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.700 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.700 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.700 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.700 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.700 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.700 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.700 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.700 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.700 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.700 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-21T10:50:27.701 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-21T10:50:27.701 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.701 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.701 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.701 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.701 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.701 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.701 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.701 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.701 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.701 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.701 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.701 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.701 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-21T10:50:27.701 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-21T10:50:27.701 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.702 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.702 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.702 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.702 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.702 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.702 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.702 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.702 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.702 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.702 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.702 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_ENABLED": { 2024-03-21T10:50:27.702 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-21T10:50:27.702 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.703 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.703 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.703 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.703 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.703 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.703 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.703 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.703 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.703 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.703 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.703 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.703 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-21T10:50:27.703 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-21T10:50:27.703 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.703 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.704 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.704 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-21T10:50:27.704 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.704 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.704 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.704 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.704 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.704 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.704 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.704 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-21T10:50:27.704 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-21T10:50:27.704 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.704 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.704 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.705 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-03-21T10:50:27.705 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.705 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.705 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.705 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.705 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.705 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.705 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.705 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-21T10:50:27.705 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-21T10:50:27.705 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.705 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.705 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.705 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8", 2024-03-21T10:50:27.706 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.706 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.706 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.706 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.706 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.706 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.706 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.706 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.706 INFO:teuthology.orchestra.run.smithi031.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-21T10:50:27.706 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-21T10:50:27.706 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.706 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.706 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.706 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "45", 2024-03-21T10:50:27.706 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.707 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.707 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.707 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.707 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.707 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.707 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.707 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ACCESS_KEY": { 2024-03-21T10:50:27.707 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-21T10:50:27.707 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.707 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.707 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.707 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.707 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.707 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.707 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.708 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.708 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.708 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.708 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.708 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-21T10:50:27.708 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-21T10:50:27.708 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.708 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.708 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.708 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-03-21T10:50:27.708 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.708 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.708 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.708 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.709 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.709 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.709 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.709 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SECRET_KEY": { 2024-03-21T10:50:27.709 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-21T10:50:27.709 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.709 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.709 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.709 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.709 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.709 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.709 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.709 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.709 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.709 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.710 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.710 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SSL_VERIFY": { 2024-03-21T10:50:27.710 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-21T10:50:27.710 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.710 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.710 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.710 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.710 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.710 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.710 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.710 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.710 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.710 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.710 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.711 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.711 INFO:teuthology.orchestra.run.smithi031.stdout: "UNSAFE_TLS_v1_2": { 2024-03-21T10:50:27.711 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-21T10:50:27.711 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.711 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.711 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.711 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.711 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.711 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.711 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.711 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.711 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.711 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.711 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.711 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.712 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-21T10:50:27.712 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-21T10:50:27.712 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.712 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.712 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.712 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-03-21T10:50:27.712 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.712 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.712 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.712 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.712 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.712 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.712 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.712 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.712 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-21T10:50:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-21T10:50:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-03-21T10:50:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-21T10:50:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-21T10:50:27.714 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.714 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.714 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.714 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-03-21T10:50:27.714 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.714 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.714 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.714 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.714 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.714 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.714 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.714 INFO:teuthology.orchestra.run.smithi031.stdout: "cross_origin_url": { 2024-03-21T10:50:27.715 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cross_origin_url", 2024-03-21T10:50:27.715 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.715 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.715 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.715 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.715 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.715 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.715 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.715 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.715 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.715 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.715 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.715 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.715 INFO:teuthology.orchestra.run.smithi031.stdout: "crt_file": { 2024-03-21T10:50:27.715 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crt_file", 2024-03-21T10:50:27.716 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.716 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.716 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.716 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.716 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.716 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.716 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.716 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.716 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.716 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.716 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.716 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.716 INFO:teuthology.orchestra.run.smithi031.stdout: "debug": { 2024-03-21T10:50:27.716 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "debug", 2024-03-21T10:50:27.717 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.717 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.717 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.717 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.717 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.717 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.717 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.717 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable/disable debug options", 2024-03-21T10:50:27.717 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.717 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.718 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.718 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.718 INFO:teuthology.orchestra.run.smithi031.stdout: "jwt_token_ttl": { 2024-03-21T10:50:27.718 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "jwt_token_ttl", 2024-03-21T10:50:27.718 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.718 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.718 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.718 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "28800", 2024-03-21T10:50:27.718 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.718 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.718 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.719 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.719 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.719 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.719 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.719 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-03-21T10:50:27.719 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-03-21T10:50:27.719 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.719 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.719 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.719 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.720 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.720 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.720 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.720 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.720 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.720 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.720 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.720 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.720 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.720 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.720 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.720 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.720 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.720 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.721 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.721 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.721 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.721 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.721 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.721 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.721 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.721 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.721 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.721 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.721 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.721 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.721 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.721 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.722 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.722 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.722 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.722 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.722 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.722 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.722 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.722 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.722 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.722 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.722 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.722 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.722 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.722 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.722 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.723 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.723 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.723 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.723 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.723 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.723 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.723 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.723 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.723 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.723 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.723 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.723 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.723 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.723 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.724 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.724 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.724 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.724 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.724 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.724 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.724 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.724 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.724 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.724 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.724 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.724 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.724 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.724 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.725 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.725 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.725 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.725 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.725 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.725 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.725 INFO:teuthology.orchestra.run.smithi031.stdout: "motd": { 2024-03-21T10:50:27.725 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "motd", 2024-03-21T10:50:27.725 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.725 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.725 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.725 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.725 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.725 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.725 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.725 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "The message of the day", 2024-03-21T10:50:27.726 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.726 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.726 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.726 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect_resolve_ip_addr": { 2024-03-21T10:50:27.726 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "redirect_resolve_ip_addr", 2024-03-21T10:50:27.726 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.726 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.726 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.726 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.726 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.726 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.726 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.726 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.727 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.727 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.727 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.727 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-03-21T10:50:27.727 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-03-21T10:50:27.727 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.727 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.727 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.727 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-03-21T10:50:27.727 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.727 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.727 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.727 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.727 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.728 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.728 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.728 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-03-21T10:50:27.728 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-03-21T10:50:27.728 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.728 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.728 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.728 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8080", 2024-03-21T10:50:27.728 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.728 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.728 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.728 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.728 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.728 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.729 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.729 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-03-21T10:50:27.729 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-03-21T10:50:27.729 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.729 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.729 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.729 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.729 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.729 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.729 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.729 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.729 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.729 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.729 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.730 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl_server_port": { 2024-03-21T10:50:27.730 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl_server_port", 2024-03-21T10:50:27.730 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.730 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.730 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.730 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8443", 2024-03-21T10:50:27.730 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.730 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.730 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.730 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.730 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.730 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.730 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.731 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-03-21T10:50:27.731 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-03-21T10:50:27.731 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.731 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.731 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.731 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "redirect", 2024-03-21T10:50:27.731 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.731 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.731 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.731 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.731 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect" 2024-03-21T10:50:27.731 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.731 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.731 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.732 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.732 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.732 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-03-21T10:50:27.732 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-03-21T10:50:27.732 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.732 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.732 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.732 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-03-21T10:50:27.732 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-03-21T10:50:27.732 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-03-21T10:50:27.732 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.732 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.732 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.733 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.733 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.733 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.733 INFO:teuthology.orchestra.run.smithi031.stdout: "url_prefix": { 2024-03-21T10:50:27.733 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url_prefix", 2024-03-21T10:50:27.733 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.733 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.733 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.733 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.733 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.733 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.733 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.733 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.733 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.734 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.734 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.734 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.734 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.734 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.734 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "devicehealth", 2024-03-21T10:50:27.734 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.734 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.734 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.734 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_monitoring": { 2024-03-21T10:50:27.734 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_monitoring", 2024-03-21T10:50:27.734 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.735 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.735 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.735 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.735 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.735 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.735 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.735 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "monitor device health metrics", 2024-03-21T10:50:27.735 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.735 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.735 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.735 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.735 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.735 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.736 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.736 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.736 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.736 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.736 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.736 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.736 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.736 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.736 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.736 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.736 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.736 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.736 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.736 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.737 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.737 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.737 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.737 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.737 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.737 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.737 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.737 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.737 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.737 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.737 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.737 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.737 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.737 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.738 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.738 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.738 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.738 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.738 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.738 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.738 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.738 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.738 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.738 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.738 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.738 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.738 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.738 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.739 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.739 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.739 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.739 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.739 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.739 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.739 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.739 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.739 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.739 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.739 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.739 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.739 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.740 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.740 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.740 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.740 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.740 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.740 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.740 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.740 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.740 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.740 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.740 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.740 INFO:teuthology.orchestra.run.smithi031.stdout: "mark_out_threshold": { 2024-03-21T10:50:27.740 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mark_out_threshold", 2024-03-21T10:50:27.740 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.741 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.741 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.741 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2419200", 2024-03-21T10:50:27.741 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.741 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.741 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.741 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-21T10:50:27.741 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.741 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.741 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.741 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": { 2024-03-21T10:50:27.741 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_name", 2024-03-21T10:50:27.741 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.742 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.742 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.742 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "device_health_metrics", 2024-03-21T10:50:27.742 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.742 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.742 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.742 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-21T10:50:27.742 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.742 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.742 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.742 INFO:teuthology.orchestra.run.smithi031.stdout: "retention_period": { 2024-03-21T10:50:27.742 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retention_period", 2024-03-21T10:50:27.742 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.742 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.743 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.743 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15552000", 2024-03-21T10:50:27.743 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.743 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.743 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.743 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain device health metrics", 2024-03-21T10:50:27.743 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.743 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.743 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.743 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_frequency": { 2024-03-21T10:50:27.743 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_frequency", 2024-03-21T10:50:27.743 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.743 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.744 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.744 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-03-21T10:50:27.744 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.744 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.744 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.744 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-21T10:50:27.744 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.744 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.744 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.744 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.744 INFO:teuthology.orchestra.run.smithi031.stdout: "self_heal": { 2024-03-21T10:50:27.744 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "self_heal", 2024-03-21T10:50:27.744 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.744 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.745 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.745 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.745 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.745 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.745 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.745 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-21T10:50:27.745 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.745 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.745 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.745 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.745 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-03-21T10:50:27.745 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-03-21T10:50:27.745 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.746 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.746 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.746 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-03-21T10:50:27.746 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.746 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.746 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.746 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and check device health", 2024-03-21T10:50:27.746 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.746 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.746 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.746 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.746 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_threshold": { 2024-03-21T10:50:27.746 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_threshold", 2024-03-21T10:50:27.746 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.746 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7257600", 2024-03-21T10:50:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-21T10:50:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "diskprediction_local", 2024-03-21T10:50:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.750 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.750 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.750 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.750 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.750 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.750 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.750 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.750 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.750 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.750 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.751 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.751 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.751 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.751 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.751 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.751 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.751 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.751 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.751 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.751 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.751 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.751 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.751 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.751 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.753 INFO:teuthology.orchestra.run.smithi031.stdout: "predict_interval": { 2024-03-21T10:50:27.753 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predict_interval", 2024-03-21T10:50:27.753 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.753 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.753 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.753 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-03-21T10:50:27.753 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.753 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.753 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.753 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.753 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.753 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.753 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.753 INFO:teuthology.orchestra.run.smithi031.stdout: "predictor_model": { 2024-03-21T10:50:27.754 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predictor_model", 2024-03-21T10:50:27.754 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.754 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.754 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.754 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "prophetstor", 2024-03-21T10:50:27.754 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.754 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.754 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.754 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.754 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.754 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.754 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.754 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-03-21T10:50:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-03-21T10:50:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-03-21T10:50:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "influx", 2024-03-21T10:50:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": false, 2024-03-21T10:50:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "influxdb python module not found", 2024-03-21T10:50:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: "batch_size": { 2024-03-21T10:50:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "batch_size", 2024-03-21T10:50:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5000", 2024-03-21T10:50:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.756 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-21T10:50:27.757 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.757 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.757 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.757 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.757 INFO:teuthology.orchestra.run.smithi031.stdout: "database": { 2024-03-21T10:50:27.757 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "database", 2024-03-21T10:50:27.757 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.757 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.757 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.757 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "ceph", 2024-03-21T10:50:27.757 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.757 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.757 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.757 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-21T10:50:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: "hostname": { 2024-03-21T10:50:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hostname", 2024-03-21T10:50:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server hostname", 2024-03-21T10:50:27.758 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-03-21T10:50:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-03-21T10:50:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "30", 2024-03-21T10:50:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "5", 2024-03-21T10:50:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-21T10:50:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.759 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.760 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.760 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.760 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.760 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.760 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.760 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.760 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.760 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.760 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.760 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.760 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.760 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.760 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.760 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.761 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.762 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.762 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.762 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.762 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.762 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.762 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.762 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.762 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.762 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.762 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.762 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.762 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.762 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.763 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: "password": { 2024-03-21T10:50:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "password", 2024-03-21T10:50:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.764 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.765 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.765 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.765 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.765 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.765 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.765 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "password of InfluxDB server user", 2024-03-21T10:50:27.765 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.765 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.765 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.765 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.765 INFO:teuthology.orchestra.run.smithi031.stdout: "port": { 2024-03-21T10:50:27.765 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "port", 2024-03-21T10:50:27.765 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.765 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.765 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.766 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8086", 2024-03-21T10:50:27.766 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.766 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.766 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.766 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server port", 2024-03-21T10:50:27.766 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.766 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.766 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.766 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-03-21T10:50:27.766 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-03-21T10:50:27.766 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.766 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.766 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.767 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "false", 2024-03-21T10:50:27.767 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.767 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.767 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.767 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-21T10:50:27.767 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.767 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.767 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.767 INFO:teuthology.orchestra.run.smithi031.stdout: "threads": { 2024-03-21T10:50:27.767 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threads", 2024-03-21T10:50:27.767 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.767 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.767 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.768 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-03-21T10:50:27.768 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-03-21T10:50:27.768 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "32", 2024-03-21T10:50:27.768 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.768 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-21T10:50:27.768 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.768 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.768 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.768 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.768 INFO:teuthology.orchestra.run.smithi031.stdout: "username": { 2024-03-21T10:50:27.768 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "username", 2024-03-21T10:50:27.768 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.768 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.768 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.768 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.769 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.769 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.769 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.769 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "username of InfluxDB server user", 2024-03-21T10:50:27.769 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.769 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.769 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.769 INFO:teuthology.orchestra.run.smithi031.stdout: "verify_ssl": { 2024-03-21T10:50:27.769 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "verify_ssl", 2024-03-21T10:50:27.769 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.769 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.769 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.769 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "true", 2024-03-21T10:50:27.770 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.770 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.770 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.770 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-21T10:50:27.770 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.770 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.770 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.770 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.770 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.770 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.770 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "insights", 2024-03-21T10:50:27.770 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.770 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.770 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.771 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.771 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.771 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.771 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.771 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.771 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.771 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.771 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.771 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.771 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.771 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.771 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.771 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.771 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.772 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.772 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.772 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.772 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.772 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.772 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.772 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.772 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.772 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.772 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.772 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.772 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.772 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.772 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.773 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.773 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.773 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.773 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.773 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.773 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.773 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.773 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.773 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.773 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.773 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.773 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.773 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.773 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.773 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.774 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.774 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.774 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.774 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.774 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.774 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.774 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.774 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.774 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.774 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.774 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.774 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.774 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.774 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.775 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.775 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.775 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.775 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.775 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.775 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.775 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.775 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.775 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.775 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.775 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.775 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.775 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.775 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.776 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "iostat", 2024-03-21T10:50:27.776 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.776 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.776 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.776 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.776 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.776 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.776 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.776 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.776 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.776 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.776 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.776 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.776 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.776 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.776 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.777 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.777 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.777 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.777 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.777 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.777 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.777 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.777 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.777 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.777 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.777 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.777 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.777 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.777 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.778 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.778 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.778 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.778 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.778 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.778 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.778 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.778 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.778 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.778 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.778 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.778 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.778 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.778 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.778 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.779 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.779 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.779 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.779 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.779 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.779 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.779 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.779 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.779 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.779 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.779 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.779 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.779 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.779 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.779 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.780 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.780 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.780 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.780 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.780 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.780 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.780 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.780 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.780 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.780 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.780 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.780 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.780 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.780 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.780 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.781 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.781 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "k8sevents", 2024-03-21T10:50:27.781 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.781 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.781 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.781 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_event_retention_days": { 2024-03-21T10:50:27.781 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ceph_event_retention_days", 2024-03-21T10:50:27.781 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.781 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.781 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.781 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7", 2024-03-21T10:50:27.781 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.781 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.781 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.782 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-21T10:50:27.782 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.782 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.782 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.782 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.782 INFO:teuthology.orchestra.run.smithi031.stdout: "config_check_secs": { 2024-03-21T10:50:27.782 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_check_secs", 2024-03-21T10:50:27.782 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.782 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.782 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.782 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-03-21T10:50:27.782 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "10", 2024-03-21T10:50:27.782 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.783 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.783 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-21T10:50:27.783 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.783 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.783 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.783 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.783 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.783 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.783 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.783 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.783 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.783 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.783 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.783 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.784 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.784 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.784 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.784 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.784 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.784 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.784 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.784 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.784 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.784 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.784 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.784 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.784 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.784 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "localpool", 2024-03-21T10:50:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: "failure_domain": { 2024-03-21T10:50:27.789 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "failure_domain", 2024-03-21T10:50:27.789 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.789 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.789 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.789 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "host", 2024-03-21T10:50:27.789 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.789 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.789 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.789 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "failure domain for any created local pool", 2024-03-21T10:50:27.789 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-21T10:50:27.789 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.789 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.790 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.790 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.790 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.790 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.790 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.790 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.790 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.790 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.790 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.791 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.791 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.791 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.791 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.791 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.791 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.791 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.791 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.791 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.791 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.791 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.791 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.791 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.791 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.793 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.793 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.793 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.793 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.793 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.793 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.793 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.793 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.793 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.793 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.793 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.793 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.793 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.793 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.793 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.794 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.794 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.794 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.794 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.794 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.794 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.794 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.794 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.794 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.794 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.794 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.794 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.794 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.795 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.795 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": { 2024-03-21T10:50:27.795 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_size", 2024-03-21T10:50:27.795 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.795 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.795 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.795 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.795 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.795 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.795 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.795 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default min_size for any created local pool", 2024-03-21T10:50:27.795 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-21T10:50:27.795 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.795 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.795 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.796 INFO:teuthology.orchestra.run.smithi031.stdout: "num_rep": { 2024-03-21T10:50:27.796 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "num_rep", 2024-03-21T10:50:27.796 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.796 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.796 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.796 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3", 2024-03-21T10:50:27.796 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.796 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.796 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.796 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default replica count for any created local pool", 2024-03-21T10:50:27.796 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.796 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.796 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.796 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": { 2024-03-21T10:50:27.797 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_num", 2024-03-21T10:50:27.797 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.797 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.797 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.797 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "128", 2024-03-21T10:50:27.797 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.797 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.797 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.797 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default pg_num for any created local pool", 2024-03-21T10:50:27.797 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.797 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.797 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.797 INFO:teuthology.orchestra.run.smithi031.stdout: "prefix": { 2024-03-21T10:50:27.797 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prefix", 2024-03-21T10:50:27.798 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.798 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.798 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.798 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.798 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.798 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.798 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.798 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name prefix for any created local pool", 2024-03-21T10:50:27.798 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.798 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.798 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.798 INFO:teuthology.orchestra.run.smithi031.stdout: "subtree": { 2024-03-21T10:50:27.798 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "subtree", 2024-03-21T10:50:27.799 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.799 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.799 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.799 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "rack", 2024-03-21T10:50:27.799 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.799 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.799 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.799 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-21T10:50:27.799 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-21T10:50:27.799 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.799 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.799 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.799 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.800 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.800 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mds_autoscaler", 2024-03-21T10:50:27.800 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.800 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.800 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.800 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.800 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.800 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.800 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.800 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.800 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.800 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.800 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.800 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.800 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.801 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.801 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.801 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.801 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.801 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.801 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.801 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.801 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.801 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.801 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.801 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.801 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.801 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.801 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.801 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.802 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.802 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.802 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.802 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.802 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.802 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.802 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.802 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.802 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.802 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.802 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.802 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.802 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.802 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.803 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.803 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.803 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.803 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.803 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.803 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.803 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.803 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.803 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.803 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.803 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.803 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.803 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.803 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.804 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.804 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.804 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.804 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.804 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.804 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.804 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.804 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.804 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.804 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.804 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.804 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.804 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.804 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.804 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.805 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.805 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.805 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirroring", 2024-03-21T10:50:27.805 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.805 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.805 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.805 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.805 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.805 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.805 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.805 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.805 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.805 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.805 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.806 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.806 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.806 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.806 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.806 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.806 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.806 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.806 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.806 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.806 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.806 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.806 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.806 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.806 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.806 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.807 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.807 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.807 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.807 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.807 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.807 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.807 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.807 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.807 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.807 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.807 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.807 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.807 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.808 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.808 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.808 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.808 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.808 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.808 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.808 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.808 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.808 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.808 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.808 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.808 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.808 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.808 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.808 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.808 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.809 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.809 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.809 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.809 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.809 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.809 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.809 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.809 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.809 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.809 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.809 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.809 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.809 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.810 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.810 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.810 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.810 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.810 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.810 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "nfs", 2024-03-21T10:50:27.810 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.810 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.810 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.810 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.810 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.810 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.810 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.810 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.811 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.811 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.811 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.811 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.811 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.811 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.811 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.811 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.811 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.811 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.811 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.811 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.811 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.811 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.811 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.812 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.812 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.812 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.812 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.812 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.812 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.812 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.812 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.812 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.812 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.812 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.812 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.812 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.813 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.813 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.813 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.813 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.813 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.813 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.813 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.813 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.813 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.813 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.813 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.813 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.813 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.813 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.813 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.813 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.814 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.814 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.814 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.814 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.814 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.814 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.814 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.814 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.814 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.814 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.814 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.814 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.815 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.815 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.815 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.815 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.815 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.815 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.815 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.815 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.815 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.815 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-03-21T10:50:27.815 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.815 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.815 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.815 INFO:teuthology.orchestra.run.smithi031.stdout: "fail_fs": { 2024-03-21T10:50:27.815 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "fail_fs", 2024-03-21T10:50:27.816 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.816 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.816 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.816 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.816 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.816 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.816 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.816 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-21T10:50:27.816 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.816 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.816 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.816 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.816 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.816 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.816 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.817 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.817 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.817 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.817 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.817 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.817 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.817 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.817 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.817 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.817 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.817 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.817 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.817 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.817 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.817 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.818 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.818 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.818 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.818 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.818 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.818 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.818 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.818 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.818 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.818 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.818 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.818 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.818 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.819 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.819 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.819 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.819 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.819 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.819 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.819 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.819 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.819 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.819 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.819 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.819 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.819 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.820 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.820 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.820 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.820 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.820 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.820 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.820 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.820 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.820 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.820 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.820 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.820 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.820 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.820 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.821 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.821 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.821 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.821 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.821 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.821 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.821 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.821 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.821 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator": { 2024-03-21T10:50:27.821 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-03-21T10:50:27.821 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.821 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.821 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.822 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.822 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.822 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.822 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.822 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm", 2024-03-21T10:50:27.822 INFO:teuthology.orchestra.run.smithi031.stdout: "rook", 2024-03-21T10:50:27.822 INFO:teuthology.orchestra.run.smithi031.stdout: "test_orchestrator" 2024-03-21T10:50:27.822 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.822 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Orchestrator backend", 2024-03-21T10:50:27.822 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.822 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.822 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.822 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.822 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.823 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.823 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_perf_query", 2024-03-21T10:50:27.823 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.823 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.823 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.823 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.823 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.823 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.823 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.823 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.823 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.823 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.823 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.823 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.824 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.824 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.824 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.824 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.824 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.824 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.824 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.824 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.824 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.824 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.824 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.824 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.824 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.824 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.824 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.825 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.825 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.825 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.825 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.825 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.825 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.825 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.825 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.825 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.825 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.825 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.825 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.825 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.825 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.825 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.826 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.826 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.826 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.826 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.826 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.826 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.826 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.826 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.826 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.826 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.826 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.826 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.826 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.826 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.826 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.827 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.827 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.827 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.827 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.827 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.827 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.827 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.827 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.827 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.827 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.827 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.827 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.827 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.827 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.827 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.828 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.828 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.828 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_support", 2024-03-21T10:50:27.828 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.828 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.828 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.828 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.828 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.828 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.828 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.828 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.828 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.828 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.828 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.829 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.829 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.829 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.829 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.829 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.829 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.829 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.829 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.829 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.829 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.829 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.829 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.829 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.829 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.830 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.830 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.830 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.830 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.830 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.830 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.830 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.830 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.830 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.830 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.830 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.830 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.830 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.830 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.830 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.831 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.831 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.831 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.831 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.831 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.831 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.831 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.831 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.831 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.831 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.831 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.831 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.831 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.831 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.831 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.832 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.832 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.832 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.832 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.832 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.832 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.832 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.832 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.832 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.832 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.832 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.832 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.832 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.832 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.833 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.833 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.833 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.833 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.833 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_autoscaler", 2024-03-21T10:50:27.833 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.833 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.833 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.833 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.833 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.833 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.833 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.833 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.833 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.834 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.834 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.834 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.834 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.834 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.834 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.834 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.834 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.834 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.834 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.834 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.834 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.834 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.834 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.834 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.835 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.835 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.835 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.835 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.835 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.835 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.835 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.835 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.835 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.835 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.835 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.835 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.835 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.835 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.836 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.836 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.836 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.836 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.836 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.836 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.836 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.836 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.836 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.836 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.836 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.836 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.836 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.836 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.836 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.837 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.837 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.837 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.837 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.837 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.837 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.837 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.837 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.837 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.837 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.837 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.837 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.837 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.837 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.838 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.838 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.838 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.838 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-03-21T10:50:27.838 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-03-21T10:50:27.838 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.838 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.838 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.838 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-03-21T10:50:27.838 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.838 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.838 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.838 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.839 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.839 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.839 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.839 INFO:teuthology.orchestra.run.smithi031.stdout: "threshold": { 2024-03-21T10:50:27.839 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threshold", 2024-03-21T10:50:27.839 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-03-21T10:50:27.839 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.839 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.839 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-03-21T10:50:27.839 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1.0", 2024-03-21T10:50:27.839 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.839 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.839 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "scaling threshold", 2024-03-21T10:50:27.839 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-21T10:50:27.840 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.840 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.840 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.840 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.840 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.840 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "progress", 2024-03-21T10:50:27.840 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.840 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.840 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.840 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_pg_recovery_event": { 2024-03-21T10:50:27.840 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_pg_recovery_event", 2024-03-21T10:50:27.840 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.840 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.840 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.841 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.841 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.841 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.841 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.841 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-21T10:50:27.841 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.841 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.841 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.841 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.841 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-03-21T10:50:27.841 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-03-21T10:50:27.841 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.841 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.841 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.841 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.841 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.842 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.842 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.842 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.842 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.842 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.842 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.842 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.842 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.842 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.842 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.842 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.842 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.842 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.842 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.842 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.843 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.843 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.843 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.843 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.843 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.843 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.843 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.843 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.843 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.843 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.843 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.843 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.843 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.843 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.844 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.844 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.844 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.844 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.844 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.844 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.844 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.844 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.844 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.844 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.844 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.844 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.844 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.844 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.845 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.845 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.845 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.845 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.845 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.845 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.845 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.845 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.845 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.845 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.845 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.845 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.845 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.845 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.845 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.845 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.846 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.846 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.846 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.846 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.846 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.846 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.846 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.846 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.846 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.846 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.846 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.846 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.846 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.846 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.847 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.847 INFO:teuthology.orchestra.run.smithi031.stdout: "max_completed_events": { 2024-03-21T10:50:27.847 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_completed_events", 2024-03-21T10:50:27.847 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.847 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.847 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.847 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "50", 2024-03-21T10:50:27.847 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.847 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.847 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.847 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "number of past completed events to remember", 2024-03-21T10:50:27.847 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.847 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.849 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.849 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-03-21T10:50:27.849 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-03-21T10:50:27.849 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.849 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.849 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.849 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-03-21T10:50:27.849 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.849 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.850 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.850 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long the module is going to sleep", 2024-03-21T10:50:27.850 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.850 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.850 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.850 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.850 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.850 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.850 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus", 2024-03-21T10:50:27.850 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.850 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.850 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.850 INFO:teuthology.orchestra.run.smithi031.stdout: "cache": { 2024-03-21T10:50:27.851 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cache", 2024-03-21T10:50:27.851 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.851 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.851 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.851 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.851 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.851 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.851 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.851 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.851 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.851 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.851 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.851 INFO:teuthology.orchestra.run.smithi031.stdout: "exclude_perf_counters": { 2024-03-21T10:50:27.851 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "exclude_perf_counters", 2024-03-21T10:50:27.851 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.852 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.852 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.852 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.852 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.852 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.852 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.852 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-21T10:50:27.852 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-21T10:50:27.852 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.852 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.852 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.852 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.852 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.853 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.853 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.853 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.853 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.853 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.853 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.853 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.853 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.853 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.853 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.853 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.853 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.853 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.853 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.853 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.854 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.854 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.854 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.854 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.854 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.854 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.854 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.854 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.854 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.854 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.854 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.854 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.854 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.854 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.855 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.855 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.855 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.855 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.855 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.855 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.855 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.855 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.855 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.855 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.855 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.855 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.855 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.855 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.855 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.856 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.856 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.856 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.856 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.856 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.856 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.856 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.856 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.856 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.856 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.856 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.856 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.856 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.856 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.856 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.856 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.857 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.857 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.857 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.857 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.857 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.857 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.857 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.857 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools": { 2024-03-21T10:50:27.857 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools", 2024-03-21T10:50:27.857 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.857 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.857 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.857 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.857 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.857 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.858 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.858 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.858 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.858 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.858 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.858 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-21T10:50:27.858 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-21T10:50:27.858 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.858 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.858 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.858 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "300", 2024-03-21T10:50:27.858 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.858 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.858 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.859 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.859 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.859 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.859 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.859 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.859 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_interval": { 2024-03-21T10:50:27.859 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_interval", 2024-03-21T10:50:27.859 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-03-21T10:50:27.859 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.859 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.859 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15.0", 2024-03-21T10:50:27.859 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.859 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.859 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.859 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.860 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.860 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.860 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.860 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-03-21T10:50:27.860 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-03-21T10:50:27.860 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.860 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.860 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.860 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-03-21T10:50:27.860 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.860 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.860 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.860 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-21T10:50:27.860 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.860 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.861 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.861 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-03-21T10:50:27.861 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-03-21T10:50:27.861 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.861 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.861 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.861 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "9283", 2024-03-21T10:50:27.861 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.861 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.861 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.861 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-21T10:50:27.861 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.861 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.861 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.862 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.862 INFO:teuthology.orchestra.run.smithi031.stdout: "stale_cache_strategy": { 2024-03-21T10:50:27.862 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stale_cache_strategy", 2024-03-21T10:50:27.862 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.862 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.862 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.862 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "log", 2024-03-21T10:50:27.862 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.862 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.862 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.862 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.862 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.862 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.862 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.862 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.862 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-03-21T10:50:27.863 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-03-21T10:50:27.863 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.863 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.863 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.863 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "default", 2024-03-21T10:50:27.863 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.863 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.863 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.863 INFO:teuthology.orchestra.run.smithi031.stdout: "default", 2024-03-21T10:50:27.863 INFO:teuthology.orchestra.run.smithi031.stdout: "error" 2024-03-21T10:50:27.863 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.863 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.863 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.863 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.864 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.864 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.864 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-03-21T10:50:27.864 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-03-21T10:50:27.864 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.864 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.864 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.864 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-03-21T10:50:27.864 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-03-21T10:50:27.864 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-03-21T10:50:27.864 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.864 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.864 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.864 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.864 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.864 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.865 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.865 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.865 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_support", 2024-03-21T10:50:27.865 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.865 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.865 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.865 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.865 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.865 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.865 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.865 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.865 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.865 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.865 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.866 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.866 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.866 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.866 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.866 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.866 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.866 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.866 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.866 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.866 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.866 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.866 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.866 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.866 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.866 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.867 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.867 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.867 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.867 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.867 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.867 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.867 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.867 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.867 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.867 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.867 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.867 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.867 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.867 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.867 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.868 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.868 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.868 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.868 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.868 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.868 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.868 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.868 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.868 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.868 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.868 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.868 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.868 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.868 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.868 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.869 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.869 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.869 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.869 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.869 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.869 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.869 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.869 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.869 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.869 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.869 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.869 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.869 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.869 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.869 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.870 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.870 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.870 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_snap_create": { 2024-03-21T10:50:27.870 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_snap_create", 2024-03-21T10:50:27.870 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.870 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.870 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.870 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-03-21T10:50:27.870 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.870 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.870 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.870 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.870 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.870 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.870 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.871 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.871 INFO:teuthology.orchestra.run.smithi031.stdout: "mirror_snapshot_schedule": { 2024-03-21T10:50:27.871 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirror_snapshot_schedule", 2024-03-21T10:50:27.871 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.871 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.871 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.871 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.871 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.871 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.871 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.871 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.871 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.871 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.871 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.871 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.872 INFO:teuthology.orchestra.run.smithi031.stdout: "trash_purge_schedule": { 2024-03-21T10:50:27.872 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "trash_purge_schedule", 2024-03-21T10:50:27.872 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.872 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.872 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.872 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.872 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.872 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.872 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.872 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.872 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.872 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.872 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.872 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.872 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.873 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.873 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.873 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "restful", 2024-03-21T10:50:27.873 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.873 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.873 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.873 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_auth": { 2024-03-21T10:50:27.873 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_auth", 2024-03-21T10:50:27.873 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.873 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.873 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.873 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.873 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.873 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.873 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.874 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.874 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.874 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.874 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.874 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.874 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-03-21T10:50:27.874 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-03-21T10:50:27.874 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.874 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.874 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.874 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.874 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.874 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.874 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.874 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.875 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.875 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.875 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.875 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.875 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.875 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.875 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.875 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.875 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.875 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.875 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.875 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.875 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.875 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.875 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.875 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.876 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.876 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.876 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.876 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.876 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.876 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.876 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.876 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.876 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.876 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.876 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.876 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.876 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.876 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.876 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.877 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.877 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.877 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.877 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.877 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.877 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.877 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.877 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.877 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.877 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.877 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.877 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.877 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.877 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.877 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.878 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.878 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.878 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.878 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.878 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.878 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.878 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.878 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.878 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.878 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.878 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.878 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.878 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.878 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.878 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.879 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.879 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.879 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.879 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.879 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.879 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.879 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.879 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.879 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.879 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.879 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.879 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.879 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.879 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-03-21T10:50:27.880 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-03-21T10:50:27.880 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.880 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.880 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.880 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.880 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.880 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.880 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.880 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.880 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.880 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.880 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.880 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.880 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-03-21T10:50:27.880 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-03-21T10:50:27.880 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.881 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.881 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.881 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.881 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.881 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.881 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.881 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.881 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.881 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.881 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.881 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.881 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.881 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.882 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rgw", 2024-03-21T10:50:27.882 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.882 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.882 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.882 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.882 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.882 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.882 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.882 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.882 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.882 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.882 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.882 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.882 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.882 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.883 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.883 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.883 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.883 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.883 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.883 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.883 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.883 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.883 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.883 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.883 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.883 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.883 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.883 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.883 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.884 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.884 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.884 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.884 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.884 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.884 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.884 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.884 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.884 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.884 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.884 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.884 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.884 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.884 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.885 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.885 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.885 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.885 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.885 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.885 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.885 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.885 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.885 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.885 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.885 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.885 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.885 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.885 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.886 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.886 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.886 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.886 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.886 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.886 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.886 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.886 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.886 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.886 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.886 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.886 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.886 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.886 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.886 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.887 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.887 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.887 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rook", 2024-03-21T10:50:27.887 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.887 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.887 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.887 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.887 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.887 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.887 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.887 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.887 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.887 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.887 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.887 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.888 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.888 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.888 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.888 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.888 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.888 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.888 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.888 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.888 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.888 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.888 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.888 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.888 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.888 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.888 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.888 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.889 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.889 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.889 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.889 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.889 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.889 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.889 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.889 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.889 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.889 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.889 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.889 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.889 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.889 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.889 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.890 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.890 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.890 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.890 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.890 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.890 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.890 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.890 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.890 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.890 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.890 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.890 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.890 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.890 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.890 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.891 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.891 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.891 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.891 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.891 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.891 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.891 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.891 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.891 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.891 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.891 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.891 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.891 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.891 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.891 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.892 INFO:teuthology.orchestra.run.smithi031.stdout: "storage_class": { 2024-03-21T10:50:27.892 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "storage_class", 2024-03-21T10:50:27.892 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.892 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.892 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.892 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "local", 2024-03-21T10:50:27.892 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.892 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.892 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.892 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-21T10:50:27.892 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.892 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.892 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.892 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.892 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.893 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.893 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.893 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "selftest", 2024-03-21T10:50:27.893 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.893 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.893 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.893 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.893 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.893 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.893 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.893 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.893 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.893 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.893 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.893 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.894 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.894 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.894 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.894 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.894 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.894 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.894 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.894 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.894 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.894 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.894 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.894 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.894 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.895 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.895 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.895 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.895 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.895 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.895 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.895 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.895 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.895 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.895 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.895 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.895 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.895 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.895 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.896 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.896 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.896 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.896 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.896 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.896 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.896 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.896 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.896 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.896 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.896 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.896 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.896 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.896 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.897 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.897 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.897 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.897 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.897 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.897 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.897 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.897 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.897 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.897 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.897 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.897 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.897 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.897 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.897 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.898 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.898 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.898 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.898 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.898 INFO:teuthology.orchestra.run.smithi031.stdout: "roption1": { 2024-03-21T10:50:27.898 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption1", 2024-03-21T10:50:27.898 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.898 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.898 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.898 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.898 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.898 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.898 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.898 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.899 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.899 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.899 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.899 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.899 INFO:teuthology.orchestra.run.smithi031.stdout: "roption2": { 2024-03-21T10:50:27.899 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption2", 2024-03-21T10:50:27.899 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.899 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.899 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.899 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "xyz", 2024-03-21T10:50:27.899 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.899 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.899 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.899 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.899 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.900 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.900 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.900 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.900 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption1": { 2024-03-21T10:50:27.900 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption1", 2024-03-21T10:50:27.900 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.900 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.900 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.900 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.900 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.900 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.900 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.900 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.900 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.900 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.901 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.901 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.901 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption2": { 2024-03-21T10:50:27.901 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption2", 2024-03-21T10:50:27.901 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.901 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.901 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.901 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.901 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.901 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.901 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.901 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.901 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.901 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.901 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.902 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.902 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption3": { 2024-03-21T10:50:27.902 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption3", 2024-03-21T10:50:27.902 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-03-21T10:50:27.902 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.902 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.902 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.902 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.902 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.902 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.902 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.902 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.902 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.902 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.903 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.903 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption4": { 2024-03-21T10:50:27.903 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption4", 2024-03-21T10:50:27.903 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.903 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.903 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.903 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.903 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.903 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.903 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.903 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.903 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.903 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.903 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.903 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.904 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption5": { 2024-03-21T10:50:27.904 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption5", 2024-03-21T10:50:27.904 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.904 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.904 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.904 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.904 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.904 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.904 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.904 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.904 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.904 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.904 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.904 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.904 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption6": { 2024-03-21T10:50:27.904 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption6", 2024-03-21T10:50:27.905 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.905 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.905 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.905 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.905 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.905 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.905 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.905 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.905 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.905 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.905 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.905 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.905 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption7": { 2024-03-21T10:50:27.905 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption7", 2024-03-21T10:50:27.905 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.906 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.906 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.906 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.906 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-03-21T10:50:27.906 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "42", 2024-03-21T10:50:27.906 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.906 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.906 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.906 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.906 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.906 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.906 INFO:teuthology.orchestra.run.smithi031.stdout: "testkey": { 2024-03-21T10:50:27.906 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testkey", 2024-03-21T10:50:27.906 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.906 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.907 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.907 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.907 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.907 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.907 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.907 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.907 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.907 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.907 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.907 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.907 INFO:teuthology.orchestra.run.smithi031.stdout: "testlkey": { 2024-03-21T10:50:27.907 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testlkey", 2024-03-21T10:50:27.907 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.907 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.907 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: "testnewline": { 2024-03-21T10:50:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testnewline", 2024-03-21T10:50:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.908 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snap_schedule", 2024-03-21T10:50:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.909 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_m_granularity": { 2024-03-21T10:50:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_m_granularity", 2024-03-21T10:50:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow minute scheduled snapshots", 2024-03-21T10:50:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.910 INFO:teuthology.orchestra.run.smithi031.stdout: "dump_on_update": { 2024-03-21T10:50:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dump_on_update", 2024-03-21T10:50:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "dump database to debug log on update", 2024-03-21T10:50:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.911 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.912 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.913 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.914 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.915 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.916 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.916 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.916 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.916 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.916 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.916 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.916 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.916 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.916 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.916 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.916 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.916 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.916 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stats", 2024-03-21T10:50:27.916 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.916 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.917 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.917 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.917 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.917 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.917 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.917 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.917 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.917 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.917 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.917 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.917 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.917 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.917 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.917 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.917 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.917 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.918 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.918 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.918 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.918 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.918 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.918 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.918 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.918 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.918 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.918 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.918 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.918 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.918 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.919 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.919 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.919 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.919 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.919 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.919 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.919 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.919 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.919 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.919 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.919 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.919 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.919 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.919 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.919 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.920 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.920 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.920 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.920 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.920 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.920 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.920 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.920 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.920 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.920 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.920 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.920 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.920 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.920 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.921 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.921 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.921 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.921 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.921 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.921 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.921 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.921 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.921 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.921 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.921 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.921 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.921 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.921 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.921 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "status", 2024-03-21T10:50:27.922 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.922 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.922 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.922 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.922 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.922 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.922 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.922 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.922 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.922 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.922 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.922 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.922 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.922 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.922 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.923 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.923 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.923 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.923 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.923 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.923 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.923 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.923 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.923 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.923 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.923 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.923 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.923 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.923 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.923 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.924 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.924 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.924 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.924 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.924 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.924 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.924 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.924 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.924 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.924 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.924 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.924 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.924 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.924 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telegraf", 2024-03-21T10:50:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: "address": { 2024-03-21T10:50:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "address", 2024-03-21T10:50:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-21T10:50:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-03-21T10:50:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-03-21T10:50:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15", 2024-03-21T10:50:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.930 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.930 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.930 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.930 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.930 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.930 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.930 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.930 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.930 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.930 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.930 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.930 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.930 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.930 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.930 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.931 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.931 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.931 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.931 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.931 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.931 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.931 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.931 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.931 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.931 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.931 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.931 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.931 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.931 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.931 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.932 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.932 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.932 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.932 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.932 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.932 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.932 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.932 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.932 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.932 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.932 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.932 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.932 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.932 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.932 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.933 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.933 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.933 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.933 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.933 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.933 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.933 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.933 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.933 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.933 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.933 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.933 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.933 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.933 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.933 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.933 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.934 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telemetry", 2024-03-21T10:50:27.934 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.934 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.934 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.934 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_basic": { 2024-03-21T10:50:27.934 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_basic", 2024-03-21T10:50:27.934 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.934 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.934 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.934 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.934 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.934 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.934 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.934 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-21T10:50:27.934 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.935 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.935 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.935 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.935 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_crash": { 2024-03-21T10:50:27.935 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_crash", 2024-03-21T10:50:27.935 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.935 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.935 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.935 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.935 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.935 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.935 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.935 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-21T10:50:27.935 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.935 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.936 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.936 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_device": { 2024-03-21T10:50:27.936 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_device", 2024-03-21T10:50:27.936 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.936 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.936 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.936 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.936 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.936 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.936 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.936 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-21T10:50:27.936 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.936 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.936 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.937 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_ident": { 2024-03-21T10:50:27.937 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_ident", 2024-03-21T10:50:27.937 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.937 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.937 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.937 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.937 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.937 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.937 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.937 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-21T10:50:27.937 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.937 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.937 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.938 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_perf": { 2024-03-21T10:50:27.938 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_perf", 2024-03-21T10:50:27.938 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.938 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.938 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.938 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.938 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.938 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.938 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.938 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-21T10:50:27.938 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.938 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.938 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.938 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.938 INFO:teuthology.orchestra.run.smithi031.stdout: "contact": { 2024-03-21T10:50:27.939 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "contact", 2024-03-21T10:50:27.939 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.939 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.939 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.939 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.939 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.939 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.939 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.939 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.939 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.939 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.939 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.939 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.939 INFO:teuthology.orchestra.run.smithi031.stdout: "description": { 2024-03-21T10:50:27.939 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "description", 2024-03-21T10:50:27.939 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.940 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.940 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.940 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.940 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.940 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.940 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.940 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.940 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.940 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.940 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.940 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.940 INFO:teuthology.orchestra.run.smithi031.stdout: "device_url": { 2024-03-21T10:50:27.940 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_url", 2024-03-21T10:50:27.940 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.940 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.940 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.941 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-21T10:50:27.941 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.941 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.941 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.941 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.941 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.941 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.941 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.941 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.941 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-03-21T10:50:27.941 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-03-21T10:50:27.941 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.941 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.941 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.941 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.942 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.942 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.942 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.942 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.942 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.942 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.942 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.942 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.942 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-03-21T10:50:27.942 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-03-21T10:50:27.942 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.942 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.942 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.942 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "24", 2024-03-21T10:50:27.942 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "8", 2024-03-21T10:50:27.943 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.943 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.943 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.943 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.943 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.943 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.943 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.943 INFO:teuthology.orchestra.run.smithi031.stdout: "last_opt_revision": { 2024-03-21T10:50:27.943 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "last_opt_revision", 2024-03-21T10:50:27.943 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.943 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.943 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.943 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1", 2024-03-21T10:50:27.943 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.944 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.944 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.944 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.944 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.944 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.944 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.944 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.944 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard": { 2024-03-21T10:50:27.944 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard", 2024-03-21T10:50:27.944 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.944 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.944 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.944 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.944 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.944 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.944 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.945 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.945 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.945 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.945 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.945 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard_description": { 2024-03-21T10:50:27.945 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard_description", 2024-03-21T10:50:27.945 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.945 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.945 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.945 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.945 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.945 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.945 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.945 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.946 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.946 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.946 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.946 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.946 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.946 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.946 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.946 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.946 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.946 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.946 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.946 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.946 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.946 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.946 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.947 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.947 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.947 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.947 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.947 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.947 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.947 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.947 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.947 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.947 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.947 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.947 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.947 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.947 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.947 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.948 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.948 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.948 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.948 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.948 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.948 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.948 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.948 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.948 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.948 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.948 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.948 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.948 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.948 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.949 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.949 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.949 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.949 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.949 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.949 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.949 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.949 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.949 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.949 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.949 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.949 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.949 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.949 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.949 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.949 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.950 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.950 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.950 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.950 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.950 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.950 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.950 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.950 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.950 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.950 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.950 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.950 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.950 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.950 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.951 INFO:teuthology.orchestra.run.smithi031.stdout: "organization": { 2024-03-21T10:50:27.951 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "organization", 2024-03-21T10:50:27.951 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.951 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.951 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.951 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.951 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.951 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.951 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.951 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.951 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.951 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.951 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.951 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.951 INFO:teuthology.orchestra.run.smithi031.stdout: "proxy": { 2024-03-21T10:50:27.952 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "proxy", 2024-03-21T10:50:27.952 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.952 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.952 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.952 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.952 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.952 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.952 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.952 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.952 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.952 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.952 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.952 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.952 INFO:teuthology.orchestra.run.smithi031.stdout: "url": { 2024-03-21T10:50:27.952 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url", 2024-03-21T10:50:27.953 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.953 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.953 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.953 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-21T10:50:27.953 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.953 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.953 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.953 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.953 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.953 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.953 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.953 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.953 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.953 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.953 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.954 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test_orchestrator", 2024-03-21T10:50:27.954 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.954 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.954 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.954 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.954 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.954 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.954 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.954 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.954 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.954 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.954 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.954 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.954 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.954 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.955 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.955 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.955 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.955 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.955 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.955 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.955 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.955 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.955 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.955 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.955 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.955 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.955 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.955 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.955 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.955 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.956 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.956 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.956 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.956 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.956 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.956 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.956 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.956 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.956 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.956 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.956 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.956 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.956 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.956 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.956 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.956 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.957 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.957 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.957 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.957 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.957 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.957 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.957 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.957 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.957 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.957 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.957 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.957 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.957 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.957 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.957 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.958 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.958 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.958 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.958 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.958 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.958 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.958 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.958 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.958 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.958 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.958 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.958 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.958 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.958 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.958 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.958 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2024-03-21T10:50:27.959 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.959 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.959 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.959 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.959 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.959 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.959 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.959 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.959 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.959 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.959 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.959 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.959 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.959 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.960 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.960 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.960 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.960 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.960 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.960 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.960 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.960 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.960 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.960 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.960 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.960 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.960 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.960 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.960 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.961 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.961 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.961 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.961 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.961 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.961 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.961 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.961 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.961 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.961 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.961 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.961 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.961 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.961 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.961 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.962 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.962 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.962 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.962 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.962 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.962 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.962 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.962 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.962 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.962 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.962 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.962 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.962 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.962 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.962 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.963 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.963 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.963 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.963 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.963 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.963 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.963 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.963 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.963 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.963 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.963 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.963 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.963 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.963 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_clones": { 2024-03-21T10:50:27.963 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_clones", 2024-03-21T10:50:27.964 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.964 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.964 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.964 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4", 2024-03-21T10:50:27.964 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.964 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.964 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.964 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-21T10:50:27.964 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.964 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.964 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.964 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.964 INFO:teuthology.orchestra.run.smithi031.stdout: "periodic_async_work": { 2024-03-21T10:50:27.964 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "periodic_async_work", 2024-03-21T10:50:27.964 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.965 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.965 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.965 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.965 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.965 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.965 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.965 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Periodically check for async work", 2024-03-21T10:50:27.965 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.965 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.965 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.965 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.965 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_delay": { 2024-03-21T10:50:27.965 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_delay", 2024-03-21T10:50:27.965 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.965 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.966 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.966 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-03-21T10:50:27.966 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.966 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.966 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.966 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-21T10:50:27.966 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.966 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.966 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.966 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.966 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_no_wait": { 2024-03-21T10:50:27.966 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_no_wait", 2024-03-21T10:50:27.966 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.966 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.966 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.967 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-03-21T10:50:27.967 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.967 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.967 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.967 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-21T10:50:27.967 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.967 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.967 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.967 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.967 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.967 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.967 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.967 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix", 2024-03-21T10:50:27.967 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-03-21T10:50:27.967 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-03-21T10:50:27.968 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-03-21T10:50:27.968 INFO:teuthology.orchestra.run.smithi031.stdout: "discovery_interval": { 2024-03-21T10:50:27.968 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "discovery_interval", 2024-03-21T10:50:27.968 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-03-21T10:50:27.968 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.968 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.968 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "100", 2024-03-21T10:50:27.968 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.968 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.968 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.968 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.968 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.968 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.968 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.968 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.968 INFO:teuthology.orchestra.run.smithi031.stdout: "identifier": { 2024-03-21T10:50:27.969 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "identifier", 2024-03-21T10:50:27.969 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.969 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.969 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.969 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.969 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.969 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.969 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.969 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.969 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.969 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.969 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.969 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.969 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-03-21T10:50:27.969 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-03-21T10:50:27.970 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-03-21T10:50:27.970 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.970 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.970 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-03-21T10:50:27.970 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.970 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.970 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.970 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.970 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.970 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.970 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.970 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.970 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-03-21T10:50:27.970 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-03-21T10:50:27.970 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.971 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.971 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.971 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.971 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.971 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.971 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.971 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.971 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.971 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.971 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.971 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.971 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.971 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.971 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.971 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.971 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.972 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.972 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.972 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-03-21T10:50:27.972 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-03-21T10:50:27.972 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.972 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.972 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.972 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.972 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.972 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.972 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.972 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.972 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.972 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.972 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.973 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.973 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-03-21T10:50:27.973 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-03-21T10:50:27.973 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.973 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.973 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.973 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-03-21T10:50:27.973 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.973 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.973 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-03-21T10:50:27.973 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-03-21T10:50:27.973 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-03-21T10:50:27.973 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-03-21T10:50:27.973 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-03-21T10:50:27.973 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-03-21T10:50:27.974 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-03-21T10:50:27.974 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.974 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.974 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.974 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.974 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.974 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.974 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-03-21T10:50:27.974 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-03-21T10:50:27.974 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-03-21T10:50:27.974 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.974 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-03-21T10:50:27.974 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-03-21T10:50:27.974 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.974 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.975 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.975 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.975 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.975 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.975 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.975 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.975 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_host": { 2024-03-21T10:50:27.975 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_host", 2024-03-21T10:50:27.975 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.975 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.975 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.975 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-03-21T10:50:27.975 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.975 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.975 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.976 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.976 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.976 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.976 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.976 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.976 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_port": { 2024-03-21T10:50:27.976 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_port", 2024-03-21T10:50:27.976 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-03-21T10:50:27.976 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.976 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.976 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10051", 2024-03-21T10:50:27.976 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.976 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.976 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.976 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.976 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.977 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.977 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.977 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.977 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_sender": { 2024-03-21T10:50:27.977 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_sender", 2024-03-21T10:50:27.977 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-03-21T10:50:27.977 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-03-21T10:50:27.977 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-03-21T10:50:27.977 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-21T10:50:27.977 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-03-21T10:50:27.977 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-03-21T10:50:27.977 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-03-21T10:50:27.977 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-03-21T10:50:27.977 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-03-21T10:50:27.977 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-03-21T10:50:27.978 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-03-21T10:50:27.978 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.978 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.978 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.978 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.978 INFO:teuthology.orchestra.run.smithi031.stdout: "services": { 2024-03-21T10:50:27.978 INFO:teuthology.orchestra.run.smithi031.stdout: "dashboard": "https://172.21.15.31:8443/" 2024-03-21T10:50:27.978 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.978 INFO:teuthology.orchestra.run.smithi031.stdout: "always_on_modules": { 2024-03-21T10:50:27.978 INFO:teuthology.orchestra.run.smithi031.stdout: "octopus": [ 2024-03-21T10:50:27.978 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2024-03-21T10:50:27.978 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-03-21T10:50:27.978 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2024-03-21T10:50:27.978 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2024-03-21T10:50:27.979 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2024-03-21T10:50:27.979 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2024-03-21T10:50:27.979 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2024-03-21T10:50:27.979 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2024-03-21T10:50:27.979 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2024-03-21T10:50:27.979 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2024-03-21T10:50:27.979 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.979 INFO:teuthology.orchestra.run.smithi031.stdout: "pacific": [ 2024-03-21T10:50:27.979 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2024-03-21T10:50:27.979 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-03-21T10:50:27.979 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2024-03-21T10:50:27.979 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2024-03-21T10:50:27.979 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2024-03-21T10:50:27.979 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2024-03-21T10:50:27.979 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2024-03-21T10:50:27.980 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2024-03-21T10:50:27.980 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2024-03-21T10:50:27.980 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2024-03-21T10:50:27.980 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.980 INFO:teuthology.orchestra.run.smithi031.stdout: "quincy": [ 2024-03-21T10:50:27.980 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2024-03-21T10:50:27.980 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-03-21T10:50:27.980 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2024-03-21T10:50:27.980 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2024-03-21T10:50:27.980 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2024-03-21T10:50:27.980 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2024-03-21T10:50:27.980 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2024-03-21T10:50:27.981 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2024-03-21T10:50:27.981 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2024-03-21T10:50:27.981 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2024-03-21T10:50:27.981 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.981 INFO:teuthology.orchestra.run.smithi031.stdout: "reef": [ 2024-03-21T10:50:27.981 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2024-03-21T10:50:27.981 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-03-21T10:50:27.981 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2024-03-21T10:50:27.981 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2024-03-21T10:50:27.981 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2024-03-21T10:50:27.981 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2024-03-21T10:50:27.981 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2024-03-21T10:50:27.981 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2024-03-21T10:50:27.981 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2024-03-21T10:50:27.981 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2024-03-21T10:50:27.981 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-03-21T10:50:27.982 INFO:teuthology.orchestra.run.smithi031.stdout: "squid": [ 2024-03-21T10:50:27.982 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2024-03-21T10:50:27.982 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-03-21T10:50:27.982 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2024-03-21T10:50:27.982 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2024-03-21T10:50:27.982 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2024-03-21T10:50:27.982 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2024-03-21T10:50:27.982 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2024-03-21T10:50:27.982 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2024-03-21T10:50:27.982 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2024-03-21T10:50:27.982 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2024-03-21T10:50:27.982 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-03-21T10:50:27.982 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.982 INFO:teuthology.orchestra.run.smithi031.stdout: "last_failure_osd_epoch": 94, 2024-03-21T10:50:27.982 INFO:teuthology.orchestra.run.smithi031.stdout: "active_clients": [ 2024-03-21T10:50:27.983 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.983 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "devicehealth", 2024-03-21T10:50:27.983 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-03-21T10:50:27.983 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.983 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-03-21T10:50:27.983 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2024-03-21T10:50:27.983 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 458710496 2024-03-21T10:50:27.983 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.983 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-03-21T10:50:27.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.983 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.983 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "libcephsqlite", 2024-03-21T10:50:27.983 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-03-21T10:50:27.983 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.983 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-03-21T10:50:27.984 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2024-03-21T10:50:27.984 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3998712420 2024-03-21T10:50:27.984 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.984 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-03-21T10:50:27.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.984 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.984 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_support", 2024-03-21T10:50:27.984 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-03-21T10:50:27.984 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.984 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-03-21T10:50:27.984 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2024-03-21T10:50:27.984 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 666825391 2024-03-21T10:50:27.984 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.984 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-03-21T10:50:27.985 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.985 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.985 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snap_schedule", 2024-03-21T10:50:27.985 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-03-21T10:50:27.985 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.985 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-03-21T10:50:27.985 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2024-03-21T10:50:27.985 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1094718579 2024-03-21T10:50:27.985 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.985 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-03-21T10:50:27.985 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.985 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.985 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snap_schedule", 2024-03-21T10:50:27.985 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-03-21T10:50:27.985 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.985 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-03-21T10:50:27.986 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2024-03-21T10:50:27.986 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3588638850 2024-03-21T10:50:27.986 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.986 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-03-21T10:50:27.986 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:27.986 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.986 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2024-03-21T10:50:27.986 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-03-21T10:50:27.986 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-03-21T10:50:27.986 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-03-21T10:50:27.986 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2024-03-21T10:50:27.986 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4033328463 2024-03-21T10:50:27.986 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.986 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-03-21T10:50:27.987 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:27.987 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-03-21T10:50:27.987 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:50:27.988 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:26 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/226216118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:50:27.988 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:26 smithi040 ceph-mon[36154]: pgmap v371: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 23 KiB/s wr, 242 op/s 2024-03-21T10:50:27.988 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:26 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3762587956' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:50:27.988 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:27 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2745794237' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-21T10:50:27.988 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:26 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/226216118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:50:27.989 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:26 smithi070 ceph-mon[36330]: pgmap v371: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 23 KiB/s wr, 242 op/s 2024-03-21T10:50:27.989 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:26 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3762587956' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:50:27.989 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:27 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2745794237' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-21T10:50:27.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.716+0000 7fa305e00640 1 -- 172.21.15.31:0/2745794237 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa2e40a3090 msgr2=0x7fa2e40a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:27.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.716+0000 7fa305e00640 1 --2- 172.21.15.31:0/2745794237 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa2e40a3090 0x7fa2e40a5550 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0009f40 tx=0x7fa2f0005f90 comp rx=0 tx=0).stop 2024-03-21T10:50:27.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.716+0000 7fa305e00640 1 -- 172.21.15.31:0/2745794237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa300100480 msgr2=0x7fa30019a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:50:27.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.716+0000 7fa305e00640 1 --2- 172.21.15.31:0/2745794237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa300100480 0x7fa30019a660 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec009900 tx=0x7fa2ec004360 comp rx=0 tx=0).stop 2024-03-21T10:50:27.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.716+0000 7fa305e00640 1 -- 172.21.15.31:0/2745794237 shutdown_connections 2024-03-21T10:50:27.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.716+0000 7fa305e00640 1 --2- 172.21.15.31:0/2745794237 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa2e40a3090 0x7fa2e40a5550 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:27.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.716+0000 7fa305e00640 1 --2- 172.21.15.31:0/2745794237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa300106610 0x7fa30019e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:27.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.716+0000 7fa305e00640 1 --2- 172.21.15.31:0/2745794237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa300100480 0x7fa30019a660 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:27.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.716+0000 7fa305e00640 1 --2- 172.21.15.31:0/2745794237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3000ffac0 0x7fa30019a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:50:27.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.716+0000 7fa305e00640 1 -- 172.21.15.31:0/2745794237 >> 172.21.15.31:0/2745794237 conn(0x7fa300077b30 msgr2=0x7fa3001045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:50:27.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.716+0000 7fa305e00640 1 -- 172.21.15.31:0/2745794237 shutdown_connections 2024-03-21T10:50:27.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:50:26.716+0000 7fa305e00640 1 -- 172.21.15.31:0/2745794237 wait complete. 2024-03-21T10:50:27.996 DEBUG:tasks.check_counter:Getting stats from a 2024-03-21T10:50:27.996 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-21T10:50:27.997 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:27 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2745794237' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-21T10:50:28.470 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:50:28.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:28 smithi031 ceph-mon[31673]: pgmap v372: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 20 KiB/s wr, 210 op/s 2024-03-21T10:50:28.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:28 smithi070 ceph-mon[36330]: pgmap v372: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 20 KiB/s wr, 210 op/s 2024-03-21T10:50:28.930 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:28 smithi040 ceph-mon[36154]: pgmap v372: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 20 KiB/s wr, 210 op/s 2024-03-21T10:50:31.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:30 smithi031 ceph-mon[31673]: pgmap v373: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 21 KiB/s wr, 230 op/s 2024-03-21T10:50:31.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:30 smithi031 ceph-mon[31673]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:50:31.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:30 smithi070 ceph-mon[36330]: pgmap v373: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 21 KiB/s wr, 230 op/s 2024-03-21T10:50:31.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:30 smithi070 ceph-mon[36330]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:50:31.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:30 smithi040 ceph-mon[36154]: pgmap v373: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 21 KiB/s wr, 230 op/s 2024-03-21T10:50:31.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:30 smithi040 ceph-mon[36154]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-21T10:50:33.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:32 smithi031 ceph-mon[31673]: pgmap v374: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.0 KiB/s wr, 197 op/s 2024-03-21T10:50:33.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:32 smithi070 ceph-mon[36330]: pgmap v374: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.0 KiB/s wr, 197 op/s 2024-03-21T10:50:33.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:32 smithi040 ceph-mon[36154]: pgmap v374: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.0 KiB/s wr, 197 op/s 2024-03-21T10:50:34.767 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:50:34.767 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-0": { 2024-03-21T10:50:34.767 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 1617, 2024-03-21T10:50:34.767 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 1270, 2024-03-21T10:50:34.768 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 443169, 2024-03-21T10:50:34.768 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 912342, 2024-03-21T10:50:34.768 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 11, 2024-03-21T10:50:34.768 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 4, 2024-03-21T10:50:34.768 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.279008422, 2024-03-21T10:50:34.768 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.132090996, 2024-03-21T10:50:34.768 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.172144860, 2024-03-21T10:50:34.768 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.008854560, 2024-03-21T10:50:34.768 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:50:34.768 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 1269, 2024-03-21T10:50:34.768 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.526119253, 2024-03-21T10:50:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000414593 2024-03-21T10:50:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:50:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 2445, 2024-03-21T10:50:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.001715020, 2024-03-21T10:50:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000701 2024-03-21T10:50:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-21T10:50:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 346208 2024-03-21T10:50:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-1": { 2024-03-21T10:50:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 226, 2024-03-21T10:50:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 269, 2024-03-21T10:50:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 115405, 2024-03-21T10:50:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 327351, 2024-03-21T10:50:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 21, 2024-03-21T10:50:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 5, 2024-03-21T10:50:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.067995067, 2024-03-21T10:50:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.026938186, 2024-03-21T10:50:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.041582574, 2024-03-21T10:50:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.001425036, 2024-03-21T10:50:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:50:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 266, 2024-03-21T10:50:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.112351880, 2024-03-21T10:50:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000422375 2024-03-21T10:50:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:50:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 358, 2024-03-21T10:50:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000148621, 2024-03-21T10:50:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000415 2024-03-21T10:50:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 5968, 2024-03-21T10:50:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 161312 2024-03-21T10:50:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-2": { 2024-03-21T10:50:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 457, 2024-03-21T10:50:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 462, 2024-03-21T10:50:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 1839497, 2024-03-21T10:50:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 602939, 2024-03-21T10:50:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 45, 2024-03-21T10:50:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 5, 2024-03-21T10:50:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.224688317, 2024-03-21T10:50:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.064124519, 2024-03-21T10:50:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.192897841, 2024-03-21T10:50:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.015423581, 2024-03-21T10:50:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:50:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 461, 2024-03-21T10:50:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.146758523, 2024-03-21T10:50:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000318348 2024-03-21T10:50:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.775 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:50:34.775 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 84, 2024-03-21T10:50:34.775 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000041795, 2024-03-21T10:50:34.775 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000497 2024-03-21T10:50:34.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.775 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 1813136, 2024-03-21T10:50:34.775 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 176848 2024-03-21T10:50:34.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.775 INFO:teuthology.orchestra.run.smithi031.stdout: "cct": { 2024-03-21T10:50:34.775 INFO:teuthology.orchestra.run.smithi031.stdout: "total_workers": 1, 2024-03-21T10:50:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: "unhealthy_workers": 0 2024-03-21T10:50:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: "finisher-MDSRank": { 2024-03-21T10:50:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: "queue_len": 0, 2024-03-21T10:50:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: "complete_latency": { 2024-03-21T10:50:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 127, 2024-03-21T10:50:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.018720055, 2024-03-21T10:50:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000147402 2024-03-21T10:50:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: "finisher-PurgeQueue": { 2024-03-21T10:50:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: "queue_len": 0, 2024-03-21T10:50:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: "complete_latency": { 2024-03-21T10:50:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 15, 2024-03-21T10:50:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.001245082, 2024-03-21T10:50:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000083005 2024-03-21T10:50:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-21T10:50:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: "request": 14, 2024-03-21T10:50:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: "reply": 11, 2024-03-21T10:50:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: "reply_latency": { 2024-03-21T10:50:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 11, 2024-03-21T10:50:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.008043653, 2024-03-21T10:50:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000731241 2024-03-21T10:50:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: "slow_reply": 0, 2024-03-21T10:50:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: "forward": 3, 2024-03-21T10:50:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_fetch_complete": 0, 2024-03-21T10:50:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_fetch_keys": 0, 2024-03-21T10:50:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_commit": 11, 2024-03-21T10:50:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_split": 0, 2024-03-21T10:50:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_merge": 0, 2024-03-21T10:50:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes": 15, 2024-03-21T10:50:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_top": 0, 2024-03-21T10:50:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_bottom": 1, 2024-03-21T10:50:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_pin_tail": 14, 2024-03-21T10:50:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_pinned": 14, 2024-03-21T10:50:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_expired": 11, 2024-03-21T10:50:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_with_caps": 1, 2024-03-21T10:50:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": 1, 2024-03-21T10:50:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "subtrees": 3, 2024-03-21T10:50:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse": 28, 2024-03-21T10:50:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_hit": 18, 2024-03-21T10:50:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_forward": 0, 2024-03-21T10:50:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_discover": 3, 2024-03-21T10:50:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_dir_fetch": 0, 2024-03-21T10:50:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_remote_ino": 0, 2024-03-21T10:50:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_lock": 0, 2024-03-21T10:50:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: "load_cent": 2000, 2024-03-21T10:50:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: "q": 2, 2024-03-21T10:50:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: "exported": 1, 2024-03-21T10:50:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: "exported_inodes": 6, 2024-03-21T10:50:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: "imported": 1, 2024-03-21T10:50:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: "imported_inodes": 2, 2024-03-21T10:50:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_dir_fetch": 0, 2024-03-21T10:50:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_backtrace_fetch": 34, 2024-03-21T10:50:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_peer_discover": 0, 2024-03-21T10:50:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rfiles": 22, 2024-03-21T10:50:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rbytes": 10738149278, 2024-03-21T10:50:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rsnaps": 6, 2024-03-21T10:50:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_backtrace_fetch": 34, 2024-03-21T10:50:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_set_tag": 0, 2024-03-21T10:50:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_backtrace_repaired": 0, 2024-03-21T10:50:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_inotable_repaired": 0, 2024-03-21T10:50:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dir_inodes": 0, 2024-03-21T10:50:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dir_base_inodes": 0, 2024-03-21T10:50:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dirfrag_rstats": 0, 2024-03-21T10:50:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_file_inodes": 34, 2024-03-21T10:50:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_inode_file_caps": 0, 2024-03-21T10:50:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_revoke": 2, 2024-03-21T10:50:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_grant": 3, 2024-03-21T10:50:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_trunc": 0, 2024-03-21T10:50:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-21T10:50:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-21T10:50:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_caps": 1, 2024-03-21T10:50:34.783 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_caps_dirty": 0, 2024-03-21T10:50:34.783 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_cap_release": 2, 2024-03-21T10:50:34.783 INFO:teuthology.orchestra.run.smithi031.stdout: "process_request_cap_release": 5 2024-03-21T10:50:34.783 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.783 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache": { 2024-03-21T10:50:34.783 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_update": 0, 2024-03-21T10:50:34.783 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_update_receipt": 0, 2024-03-21T10:50:34.783 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_try_discover": 0, 2024-03-21T10:50:34.783 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_send_discover": 4, 2024-03-21T10:50:34.783 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_handle_discover": 1, 2024-03-21T10:50:34.783 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays": 0, 2024-03-21T10:50:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays_delayed": 0, 2024-03-21T10:50:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays_enqueuing": 0, 2024-03-21T10:50:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_created": 2, 2024-03-21T10:50:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_enqueued": 2, 2024-03-21T10:50:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_reintegrated": 0, 2024-03-21T10:50:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_migrated": 0, 2024-03-21T10:50:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_processing": 0, 2024-03-21T10:50:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_enqueued": 0, 2024-03-21T10:50:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_prioritized": 0, 2024-03-21T10:50:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_started": 0, 2024-03-21T10:50:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_completed": 0, 2024-03-21T10:50:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_enqueue_scrub": 0, 2024-03-21T10:50:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_exportdir": 1, 2024-03-21T10:50:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_flush": 0, 2024-03-21T10:50:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_fragmentdir": 0, 2024-03-21T10:50:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_fragstats": 0, 2024-03-21T10:50:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_inodestats": 0 2024-03-21T10:50:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log": { 2024-03-21T10:50:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "evlrg": 0, 2024-03-21T10:50:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "evadd": 23, 2024-03-21T10:50:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "evex": 0, 2024-03-21T10:50:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "evtrm": 0, 2024-03-21T10:50:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: "ev": 23, 2024-03-21T10:50:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: "evexg": 0, 2024-03-21T10:50:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: "evexd": 0, 2024-03-21T10:50:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: "segadd": 2, 2024-03-21T10:50:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: "segex": 0, 2024-03-21T10:50:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: "segtrm": 0, 2024-03-21T10:50:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: "seg": 2, 2024-03-21T10:50:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: "segmjr": 2, 2024-03-21T10:50:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: "segexg": 0, 2024-03-21T10:50:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: "segexd": 0, 2024-03-21T10:50:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: "expos": 4194304, 2024-03-21T10:50:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: "wrpos": 4232679, 2024-03-21T10:50:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: "rdpos": 0, 2024-03-21T10:50:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: "jlat": { 2024-03-21T10:50:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 18, 2024-03-21T10:50:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.042731088, 2024-03-21T10:50:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.002373949 2024-03-21T10:50:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: "replayed": 0 2024-03-21T10:50:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_mem": { 2024-03-21T10:50:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: "ino": 20, 2024-03-21T10:50:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "ino+": 29, 2024-03-21T10:50:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "ino-": 9, 2024-03-21T10:50:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "dir": 18, 2024-03-21T10:50:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "dir+": 23, 2024-03-21T10:50:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "dir-": 5, 2024-03-21T10:50:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "dn": 18, 2024-03-21T10:50:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "dn+": 31, 2024-03-21T10:50:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "dn-": 13, 2024-03-21T10:50:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "cap": 1, 2024-03-21T10:50:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "cap+": 9, 2024-03-21T10:50:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "cap-": 8, 2024-03-21T10:50:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "rss": 52480, 2024-03-21T10:50:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "heap": 223612 2024-03-21T10:50:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_server": { 2024-03-21T10:50:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "dispatch_client_request": 19, 2024-03-21T10:50:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "dispatch_server_request": 1, 2024-03-21T10:50:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_request": 14, 2024-03-21T10:50:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_session": 30, 2024-03-21T10:50:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_peer_request": 4, 2024-03-21T10:50:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "req_create_latency": { 2024-03-21T10:50:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getattr_latency": { 2024-03-21T10:50:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 5, 2024-03-21T10:50:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.001470578, 2024-03-21T10:50:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000294115 2024-03-21T10:50:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getfilelock_latency": { 2024-03-21T10:50:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: "req_link_latency": { 2024-03-21T10:50:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookup_latency": { 2024-03-21T10:50:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 3, 2024-03-21T10:50:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.001590247, 2024-03-21T10:50:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000530082 2024-03-21T10:50:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookuphash_latency": { 2024-03-21T10:50:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupino_latency": { 2024-03-21T10:50:34.792 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.792 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.792 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.792 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupname_latency": { 2024-03-21T10:50:34.792 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.792 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.792 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.792 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupparent_latency": { 2024-03-21T10:50:34.792 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.792 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.792 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.793 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupsnap_latency": { 2024-03-21T10:50:34.793 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.793 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.793 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.793 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lssnap_latency": { 2024-03-21T10:50:34.793 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.793 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.793 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.793 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mkdir_latency": { 2024-03-21T10:50:34.793 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.793 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.794 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.794 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mknod_latency": { 2024-03-21T10:50:34.794 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.794 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.794 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.795 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mksnap_latency": { 2024-03-21T10:50:34.795 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.797 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.797 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.797 INFO:teuthology.orchestra.run.smithi031.stdout: "req_open_latency": { 2024-03-21T10:50:34.797 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.797 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.797 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.797 INFO:teuthology.orchestra.run.smithi031.stdout: "req_readdir_latency": { 2024-03-21T10:50:34.797 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 1, 2024-03-21T10:50:34.797 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000776349, 2024-03-21T10:50:34.798 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000776349 2024-03-21T10:50:34.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.798 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rename_latency": { 2024-03-21T10:50:34.798 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.798 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.798 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.798 INFO:teuthology.orchestra.run.smithi031.stdout: "req_renamesnap_latency": { 2024-03-21T10:50:34.798 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.798 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.798 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.798 INFO:teuthology.orchestra.run.smithi031.stdout: "req_snapdiff_latency": { 2024-03-21T10:50:34.798 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.799 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.799 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.799 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmdir_latency": { 2024-03-21T10:50:34.799 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.799 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.799 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.799 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmsnap_latency": { 2024-03-21T10:50:34.799 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.799 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.799 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.799 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmxattr_latency": { 2024-03-21T10:50:34.800 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.800 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.800 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.800 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setattr_latency": { 2024-03-21T10:50:34.800 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.800 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.800 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.800 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setdirlayout_latency": { 2024-03-21T10:50:34.800 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.800 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.800 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.800 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setfilelock_latency": { 2024-03-21T10:50:34.801 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.801 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.801 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.801 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.801 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setlayout_latency": { 2024-03-21T10:50:34.801 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.802 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.802 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.802 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setxattr_latency": { 2024-03-21T10:50:34.802 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.802 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.802 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.803 INFO:teuthology.orchestra.run.smithi031.stdout: "req_symlink_latency": { 2024-03-21T10:50:34.803 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.803 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.803 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.803 INFO:teuthology.orchestra.run.smithi031.stdout: "req_unlink_latency": { 2024-03-21T10:50:34.803 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 2, 2024-03-21T10:50:34.803 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.004206479, 2024-03-21T10:50:34.803 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.002103239 2024-03-21T10:50:34.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.803 INFO:teuthology.orchestra.run.smithi031.stdout: "cap_revoke_eviction": 0, 2024-03-21T10:50:34.803 INFO:teuthology.orchestra.run.smithi031.stdout: "cap_acquisition_throttle": 0, 2024-03-21T10:50:34.803 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getvxattr_latency": { 2024-03-21T10:50:34.803 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.804 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.804 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.804 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:34.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.804 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_sessions": { 2024-03-21T10:50:34.804 INFO:teuthology.orchestra.run.smithi031.stdout: "session_count": 2, 2024-03-21T10:50:34.804 INFO:teuthology.orchestra.run.smithi031.stdout: "session_add": 5, 2024-03-21T10:50:34.804 INFO:teuthology.orchestra.run.smithi031.stdout: "session_remove": 3, 2024-03-21T10:50:34.804 INFO:teuthology.orchestra.run.smithi031.stdout: "sessions_open": 2, 2024-03-21T10:50:34.804 INFO:teuthology.orchestra.run.smithi031.stdout: "sessions_stale": 0, 2024-03-21T10:50:34.805 INFO:teuthology.orchestra.run.smithi031.stdout: "total_load": 4, 2024-03-21T10:50:34.805 INFO:teuthology.orchestra.run.smithi031.stdout: "average_load": 2, 2024-03-21T10:50:34.805 INFO:teuthology.orchestra.run.smithi031.stdout: "avg_session_uptime": 693, 2024-03-21T10:50:34.805 INFO:teuthology.orchestra.run.smithi031.stdout: "mdthresh_evicted": 0 2024-03-21T10:50:34.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.805 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool": { 2024-03-21T10:50:34.805 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_bytes": 0, 2024-03-21T10:50:34.805 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_items": 0, 2024-03-21T10:50:34.805 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_bytes": 0, 2024-03-21T10:50:34.805 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_items": 0, 2024-03-21T10:50:34.805 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_bytes": 0, 2024-03-21T10:50:34.806 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_items": 0, 2024-03-21T10:50:34.806 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-21T10:50:34.806 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_items": 0, 2024-03-21T10:50:34.806 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-21T10:50:34.806 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_items": 0, 2024-03-21T10:50:34.806 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_bytes": 0, 2024-03-21T10:50:34.806 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_items": 0, 2024-03-21T10:50:34.806 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-21T10:50:34.806 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_items": 0, 2024-03-21T10:50:34.806 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_bytes": 0, 2024-03-21T10:50:34.806 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_items": 0, 2024-03-21T10:50:34.806 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_bytes": 0, 2024-03-21T10:50:34.806 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_items": 0, 2024-03-21T10:50:34.807 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-21T10:50:34.807 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_items": 0, 2024-03-21T10:50:34.807 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-21T10:50:34.807 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_items": 0, 2024-03-21T10:50:34.807 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_bytes": 0, 2024-03-21T10:50:34.807 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_items": 0, 2024-03-21T10:50:34.807 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_bytes": 0, 2024-03-21T10:50:34.807 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_items": 0, 2024-03-21T10:50:34.807 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-21T10:50:34.807 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_items": 0, 2024-03-21T10:50:34.807 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_bytes": 0, 2024-03-21T10:50:34.807 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_items": 0, 2024-03-21T10:50:34.807 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_bytes": 0, 2024-03-21T10:50:34.807 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_items": 0, 2024-03-21T10:50:34.807 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_bytes": 0, 2024-03-21T10:50:34.807 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_items": 0, 2024-03-21T10:50:34.808 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_bytes": 0, 2024-03-21T10:50:34.808 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_items": 0, 2024-03-21T10:50:34.808 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_bytes": 296326, 2024-03-21T10:50:34.808 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_items": 105, 2024-03-21T10:50:34.808 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_bytes": 0, 2024-03-21T10:50:34.808 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_items": 0, 2024-03-21T10:50:34.808 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bytes": 0, 2024-03-21T10:50:34.808 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_items": 0, 2024-03-21T10:50:34.808 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_bytes": 0, 2024-03-21T10:50:34.808 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_items": 0, 2024-03-21T10:50:34.808 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_bytes": 0, 2024-03-21T10:50:34.808 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_items": 0, 2024-03-21T10:50:34.808 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_bytes": 9328, 2024-03-21T10:50:34.808 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_items": 179, 2024-03-21T10:50:34.808 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_bytes": 0, 2024-03-21T10:50:34.809 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_items": 0, 2024-03-21T10:50:34.809 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_bytes": 0, 2024-03-21T10:50:34.809 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_items": 0, 2024-03-21T10:50:34.809 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_bytes": 88006, 2024-03-21T10:50:34.809 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_items": 412, 2024-03-21T10:50:34.809 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_bytes": 0, 2024-03-21T10:50:34.809 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_items": 0, 2024-03-21T10:50:34.809 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_bytes": 0, 2024-03-21T10:50:34.809 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_items": 0 2024-03-21T10:50:34.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.809 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter": { 2024-03-21T10:50:34.809 INFO:teuthology.orchestra.run.smithi031.stdout: "op_active": 0, 2024-03-21T10:50:34.809 INFO:teuthology.orchestra.run.smithi031.stdout: "op_laggy": 0, 2024-03-21T10:50:34.809 INFO:teuthology.orchestra.run.smithi031.stdout: "op_send": 113, 2024-03-21T10:50:34.809 INFO:teuthology.orchestra.run.smithi031.stdout: "op_send_bytes": 51297, 2024-03-21T10:50:34.810 INFO:teuthology.orchestra.run.smithi031.stdout: "op_resend": 30, 2024-03-21T10:50:34.810 INFO:teuthology.orchestra.run.smithi031.stdout: "op_reply": 113, 2024-03-21T10:50:34.810 INFO:teuthology.orchestra.run.smithi031.stdout: "op_latency": { 2024-03-21T10:50:34.810 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 113, 2024-03-21T10:50:34.810 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 1.839969093, 2024-03-21T10:50:34.810 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.016282912 2024-03-21T10:50:34.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.810 INFO:teuthology.orchestra.run.smithi031.stdout: "op_inflight": 0, 2024-03-21T10:50:34.810 INFO:teuthology.orchestra.run.smithi031.stdout: "oplen_avg": { 2024-03-21T10:50:34.810 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 113, 2024-03-21T10:50:34.810 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 141 2024-03-21T10:50:34.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.810 INFO:teuthology.orchestra.run.smithi031.stdout: "op": 113, 2024-03-21T10:50:34.810 INFO:teuthology.orchestra.run.smithi031.stdout: "op_r": 35, 2024-03-21T10:50:34.810 INFO:teuthology.orchestra.run.smithi031.stdout: "op_w": 78, 2024-03-21T10:50:34.811 INFO:teuthology.orchestra.run.smithi031.stdout: "op_rmw": 0, 2024-03-21T10:50:34.811 INFO:teuthology.orchestra.run.smithi031.stdout: "op_pg": 0, 2024-03-21T10:50:34.811 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_stat": 0, 2024-03-21T10:50:34.811 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_create": 10, 2024-03-21T10:50:34.811 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_read": 1, 2024-03-21T10:50:34.811 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_write": 19, 2024-03-21T10:50:34.811 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_writefull": 14, 2024-03-21T10:50:34.811 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_writesame": 0, 2024-03-21T10:50:34.811 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_append": 0, 2024-03-21T10:50:34.811 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_zero": 0, 2024-03-21T10:50:34.811 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_truncate": 0, 2024-03-21T10:50:34.811 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_delete": 16, 2024-03-21T10:50:34.811 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_mapext": 0, 2024-03-21T10:50:34.811 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_sparse_read": 0, 2024-03-21T10:50:34.811 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_clonerange": 0, 2024-03-21T10:50:34.812 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_getxattr": 34, 2024-03-21T10:50:34.812 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_setxattr": 20, 2024-03-21T10:50:34.812 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_cmpxattr": 0, 2024-03-21T10:50:34.812 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_rmxattr": 0, 2024-03-21T10:50:34.812 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_resetxattrs": 0, 2024-03-21T10:50:34.812 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_call": 0, 2024-03-21T10:50:34.812 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_watch": 0, 2024-03-21T10:50:34.812 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_notify": 0, 2024-03-21T10:50:34.812 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_src_cmpxattr": 0, 2024-03-21T10:50:34.812 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_pgls": 0, 2024-03-21T10:50:34.812 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_pgls_filter": 0, 2024-03-21T10:50:34.812 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_other": 0, 2024-03-21T10:50:34.812 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_active": 0, 2024-03-21T10:50:34.812 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_send": 0, 2024-03-21T10:50:34.812 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_resend": 0, 2024-03-21T10:50:34.812 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_ping": 0, 2024-03-21T10:50:34.813 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_active": 0, 2024-03-21T10:50:34.813 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_send": 0, 2024-03-21T10:50:34.813 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_resend": 0, 2024-03-21T10:50:34.813 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_active": 0, 2024-03-21T10:50:34.813 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_send": 0, 2024-03-21T10:50:34.813 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_resend": 0, 2024-03-21T10:50:34.813 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_active": 0, 2024-03-21T10:50:34.813 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_send": 0, 2024-03-21T10:50:34.813 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_resend": 0, 2024-03-21T10:50:34.813 INFO:teuthology.orchestra.run.smithi031.stdout: "command_active": 0, 2024-03-21T10:50:34.813 INFO:teuthology.orchestra.run.smithi031.stdout: "command_send": 0, 2024-03-21T10:50:34.813 INFO:teuthology.orchestra.run.smithi031.stdout: "command_resend": 0, 2024-03-21T10:50:34.813 INFO:teuthology.orchestra.run.smithi031.stdout: "map_epoch": 106, 2024-03-21T10:50:34.813 INFO:teuthology.orchestra.run.smithi031.stdout: "map_full": 0, 2024-03-21T10:50:34.813 INFO:teuthology.orchestra.run.smithi031.stdout: "map_inc": 17, 2024-03-21T10:50:34.814 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_sessions": 11, 2024-03-21T10:50:34.814 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_session_open": 11, 2024-03-21T10:50:34.814 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_session_close": 0, 2024-03-21T10:50:34.814 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_laggy": 0, 2024-03-21T10:50:34.814 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_wr": 23, 2024-03-21T10:50:34.814 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_rd": 0, 2024-03-21T10:50:34.814 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_del": 4 2024-03-21T10:50:34.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.814 INFO:teuthology.orchestra.run.smithi031.stdout: "oft": { 2024-03-21T10:50:34.814 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_objs": 1, 2024-03-21T10:50:34.814 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_kv_pairs": 0, 2024-03-21T10:50:34.814 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_updates": 0, 2024-03-21T10:50:34.814 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_removes": 0 2024-03-21T10:50:34.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.814 INFO:teuthology.orchestra.run.smithi031.stdout: "osd-slow-ops": { 2024-03-21T10:50:34.815 INFO:teuthology.orchestra.run.smithi031.stdout: "slow_ops_count": 0 2024-03-21T10:50:34.815 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.815 INFO:teuthology.orchestra.run.smithi031.stdout: "purge_queue": { 2024-03-21T10:50:34.815 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_ops": 0, 2024-03-21T10:50:34.815 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_ops_high_water": 4, 2024-03-21T10:50:34.815 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing": 0, 2024-03-21T10:50:34.815 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_high_water": 2, 2024-03-21T10:50:34.815 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executed_ops": 4, 2024-03-21T10:50:34.815 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executed": 2, 2024-03-21T10:50:34.815 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_item_in_journal": 0 2024-03-21T10:50:34.815 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.815 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-21T10:50:34.815 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-21T10:50:34.815 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-03-21T10:50:34.815 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-21T10:50:34.815 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 2296, 2024-03-21T10:50:34.816 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 2169487, 2024-03-21T10:50:34.816 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-21T10:50:34.816 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 2296, 2024-03-21T10:50:34.816 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-03-21T10:50:34.816 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-03-21T10:50:34.816 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 2296, 2024-03-21T10:50:34.816 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 2169487, 2024-03-21T10:50:34.816 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-21T10:50:34.816 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.816 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.816 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.816 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:34.816 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.816 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-objecter_bytes": { 2024-03-21T10:50:34.816 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-21T10:50:34.817 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-03-21T10:50:34.817 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-21T10:50:34.817 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-03-21T10:50:34.817 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-03-21T10:50:34.817 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-21T10:50:34.817 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-03-21T10:50:34.817 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 113, 2024-03-21T10:50:34.817 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 51611, 2024-03-21T10:50:34.817 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 113, 2024-03-21T10:50:34.817 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 51611, 2024-03-21T10:50:34.817 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-21T10:50:34.817 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.817 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.817 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.817 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:34.817 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.818 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-objecter_ops": { 2024-03-21T10:50:34.818 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-21T10:50:34.818 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 1024, 2024-03-21T10:50:34.818 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-21T10:50:34.818 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-03-21T10:50:34.818 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-03-21T10:50:34.818 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-21T10:50:34.818 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-03-21T10:50:34.818 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 113, 2024-03-21T10:50:34.818 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 113, 2024-03-21T10:50:34.818 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 113, 2024-03-21T10:50:34.818 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 113, 2024-03-21T10:50:34.818 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-21T10:50:34.818 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.818 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-write_buf_throttle": { 2024-03-21T10:50:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-21T10:50:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 3758096384, 2024-03-21T10:50:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-21T10:50:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 2, 2024-03-21T10:50:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 314, 2024-03-21T10:50:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-21T10:50:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 2, 2024-03-21T10:50:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-03-21T10:50:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-03-21T10:50:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 1, 2024-03-21T10:50:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 314, 2024-03-21T10:50:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-21T10:50:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-write_buf_throttle-0x5575c255c460": { 2024-03-21T10:50:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-21T10:50:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 3758096384, 2024-03-21T10:50:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-21T10:50:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 23, 2024-03-21T10:50:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 38375, 2024-03-21T10:50:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-21T10:50:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 23, 2024-03-21T10:50:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-03-21T10:50:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-03-21T10:50:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 18, 2024-03-21T10:50:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 38375, 2024-03-21T10:50:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-21T10:50:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:34.821 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:50:35.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:34 smithi031 ceph-mon[31673]: pgmap v375: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.0 KiB/s wr, 193 op/s 2024-03-21T10:50:35.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:34 smithi070 ceph-mon[36330]: pgmap v375: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.0 KiB/s wr, 193 op/s 2024-03-21T10:50:35.429 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:34 smithi040 ceph-mon[36154]: pgmap v375: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.0 KiB/s wr, 193 op/s 2024-03-21T10:50:35.865 INFO:tasks.check_counter:Daemon mds.a mds.exported=1 2024-03-21T10:50:35.865 INFO:tasks.check_counter:Daemon mds.a mds.imported=1 2024-03-21T10:50:35.865 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=6 2024-03-21T10:50:35.865 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-03-21T10:50:35.866 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-03-21T10:50:35.866 DEBUG:tasks.check_counter:Getting stats from d 2024-03-21T10:50:35.866 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-21T10:50:36.347 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:50:37.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:36 smithi031 ceph-mon[31673]: pgmap v376: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.5 KiB/s wr, 170 op/s 2024-03-21T10:50:37.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:36 smithi070 ceph-mon[36330]: pgmap v376: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.5 KiB/s wr, 170 op/s 2024-03-21T10:50:37.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:36 smithi040 ceph-mon[36154]: pgmap v376: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.5 KiB/s wr, 170 op/s 2024-03-21T10:50:38.726 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:50:38.726 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-0": { 2024-03-21T10:50:38.726 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 125, 2024-03-21T10:50:38.727 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 126, 2024-03-21T10:50:38.727 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 175424, 2024-03-21T10:50:38.727 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 74656, 2024-03-21T10:50:38.727 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 5, 2024-03-21T10:50:38.727 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 0, 2024-03-21T10:50:38.727 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.038842807, 2024-03-21T10:50:38.727 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.013058647, 2024-03-21T10:50:38.727 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.012794202, 2024-03-21T10:50:38.727 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.006231428, 2024-03-21T10:50:38.727 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:50:38.727 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 124, 2024-03-21T10:50:38.727 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.034006489, 2024-03-21T10:50:38.727 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000274245 2024-03-21T10:50:38.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:38.728 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:50:38.728 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:38.728 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:38.728 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:38.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:38.728 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 175424, 2024-03-21T10:50:38.728 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 74656 2024-03-21T10:50:38.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:38.728 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-1": { 2024-03-21T10:50:38.728 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 150, 2024-03-21T10:50:38.728 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 138, 2024-03-21T10:50:38.728 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 1699600, 2024-03-21T10:50:38.729 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 80416, 2024-03-21T10:50:38.729 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 4, 2024-03-21T10:50:38.729 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 2, 2024-03-21T10:50:38.729 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.083494722, 2024-03-21T10:50:38.729 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.026476357, 2024-03-21T10:50:38.729 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.075328107, 2024-03-21T10:50:38.729 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.005247833, 2024-03-21T10:50:38.729 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:50:38.729 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 136, 2024-03-21T10:50:38.729 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.038195184, 2024-03-21T10:50:38.729 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000280846 2024-03-21T10:50:38.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:38.730 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:50:38.730 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:38.730 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:38.730 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:38.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:38.730 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 1699600, 2024-03-21T10:50:38.730 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 80416 2024-03-21T10:50:38.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:38.730 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-2": { 2024-03-21T10:50:38.730 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 2, 2024-03-21T10:50:38.730 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 196, 2024-03-21T10:50:38.730 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 320, 2024-03-21T10:50:38.730 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 191744, 2024-03-21T10:50:38.731 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 5, 2024-03-21T10:50:38.731 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 2, 2024-03-21T10:50:38.731 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.036884263, 2024-03-21T10:50:38.731 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.027621986, 2024-03-21T10:50:38.731 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.003811988, 2024-03-21T10:50:38.731 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-21T10:50:38.731 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:50:38.731 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 193, 2024-03-21T10:50:38.731 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.038818137, 2024-03-21T10:50:38.731 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000201130 2024-03-21T10:50:38.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:38.731 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:50:38.731 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:38.732 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:38.732 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:38.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:38.732 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-21T10:50:38.732 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 191744 2024-03-21T10:50:38.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:38.732 INFO:teuthology.orchestra.run.smithi031.stdout: "cct": { 2024-03-21T10:50:38.732 INFO:teuthology.orchestra.run.smithi031.stdout: "total_workers": 0, 2024-03-21T10:50:38.732 INFO:teuthology.orchestra.run.smithi031.stdout: "unhealthy_workers": 0 2024-03-21T10:50:38.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:38.732 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool": { 2024-03-21T10:50:38.732 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_bytes": 0, 2024-03-21T10:50:38.732 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_items": 0, 2024-03-21T10:50:38.733 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_bytes": 0, 2024-03-21T10:50:38.733 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_items": 0, 2024-03-21T10:50:38.733 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_bytes": 0, 2024-03-21T10:50:38.733 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_items": 0, 2024-03-21T10:50:38.733 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-21T10:50:38.733 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_items": 0, 2024-03-21T10:50:38.733 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-21T10:50:38.733 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_items": 0, 2024-03-21T10:50:38.733 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_bytes": 0, 2024-03-21T10:50:38.733 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_items": 0, 2024-03-21T10:50:38.733 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-21T10:50:38.733 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_items": 0, 2024-03-21T10:50:38.733 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_bytes": 0, 2024-03-21T10:50:38.733 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_items": 0, 2024-03-21T10:50:38.734 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_bytes": 0, 2024-03-21T10:50:38.734 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_items": 0, 2024-03-21T10:50:38.734 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-21T10:50:38.734 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_items": 0, 2024-03-21T10:50:38.734 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-21T10:50:38.734 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_items": 0, 2024-03-21T10:50:38.734 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_bytes": 0, 2024-03-21T10:50:38.734 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_items": 0, 2024-03-21T10:50:38.734 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_bytes": 0, 2024-03-21T10:50:38.734 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_items": 0, 2024-03-21T10:50:38.734 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-21T10:50:38.734 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_items": 0, 2024-03-21T10:50:38.734 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_bytes": 0, 2024-03-21T10:50:38.735 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_items": 0, 2024-03-21T10:50:38.735 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_bytes": 0, 2024-03-21T10:50:38.735 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_items": 0, 2024-03-21T10:50:38.735 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_bytes": 0, 2024-03-21T10:50:38.735 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_items": 0, 2024-03-21T10:50:38.735 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_bytes": 0, 2024-03-21T10:50:38.735 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_items": 0, 2024-03-21T10:50:38.735 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_bytes": 148738, 2024-03-21T10:50:38.735 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_items": 30, 2024-03-21T10:50:38.735 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_bytes": 0, 2024-03-21T10:50:38.735 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_items": 0, 2024-03-21T10:50:38.735 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bytes": 0, 2024-03-21T10:50:38.735 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_items": 0, 2024-03-21T10:50:38.736 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_bytes": 0, 2024-03-21T10:50:38.736 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_items": 0, 2024-03-21T10:50:38.736 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_bytes": 0, 2024-03-21T10:50:38.736 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_items": 0, 2024-03-21T10:50:38.736 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_bytes": 0, 2024-03-21T10:50:38.736 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_items": 0, 2024-03-21T10:50:38.736 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_bytes": 0, 2024-03-21T10:50:38.736 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_items": 0, 2024-03-21T10:50:38.736 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_bytes": 0, 2024-03-21T10:50:38.736 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_items": 0, 2024-03-21T10:50:38.736 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_bytes": 928, 2024-03-21T10:50:38.736 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_items": 2, 2024-03-21T10:50:38.736 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_bytes": 0, 2024-03-21T10:50:38.737 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_items": 0, 2024-03-21T10:50:38.737 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_bytes": 0, 2024-03-21T10:50:38.737 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_items": 0 2024-03-21T10:50:38.737 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:38.737 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-21T10:50:38.737 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-21T10:50:38.737 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-03-21T10:50:38.737 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-21T10:50:38.737 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 273, 2024-03-21T10:50:38.737 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 1833799, 2024-03-21T10:50:38.737 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-21T10:50:38.737 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 273, 2024-03-21T10:50:38.737 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-03-21T10:50:38.737 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-03-21T10:50:38.738 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 273, 2024-03-21T10:50:38.738 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 1833799, 2024-03-21T10:50:38.738 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-21T10:50:38.738 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:38.738 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:38.738 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:38.738 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:38.738 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:38.738 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:50:39.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:38 smithi031 ceph-mon[31673]: pgmap v377: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.8 KiB/s wr, 31 op/s 2024-03-21T10:50:39.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:38 smithi070 ceph-mon[36330]: pgmap v377: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.8 KiB/s wr, 31 op/s 2024-03-21T10:50:39.429 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:38 smithi040 ceph-mon[36154]: pgmap v377: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.8 KiB/s wr, 31 op/s 2024-03-21T10:50:39.607 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-21T10:50:39.607 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-21T10:50:39.607 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-03-21T10:50:39.607 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-03-21T10:50:39.607 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-03-21T10:50:39.607 DEBUG:tasks.check_counter:Getting stats from g 2024-03-21T10:50:39.607 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-21T10:50:40.105 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:50:41.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:40 smithi031 ceph-mon[31673]: pgmap v378: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 5.7 KiB/s wr, 34 op/s 2024-03-21T10:50:41.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:40 smithi070 ceph-mon[36330]: pgmap v378: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 5.7 KiB/s wr, 34 op/s 2024-03-21T10:50:41.429 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:40 smithi040 ceph-mon[36154]: pgmap v378: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 5.7 KiB/s wr, 34 op/s 2024-03-21T10:50:43.022 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:50:43.022 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-0": { 2024-03-21T10:50:43.023 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 0, 2024-03-21T10:50:43.023 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 2, 2024-03-21T10:50:43.023 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 0, 2024-03-21T10:50:43.023 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 0, 2024-03-21T10:50:43.023 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 4, 2024-03-21T10:50:43.023 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 0, 2024-03-21T10:50:43.023 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.006389881, 2024-03-21T10:50:43.023 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.000000000, 2024-03-21T10:50:43.023 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.001021905, 2024-03-21T10:50:43.023 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-21T10:50:43.024 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:50:43.024 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:43.024 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:43.024 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:43.024 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:43.024 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:50:43.024 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:43.024 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:43.024 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:43.024 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:43.024 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-21T10:50:43.025 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 0 2024-03-21T10:50:43.025 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:43.025 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-1": { 2024-03-21T10:50:43.025 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 266, 2024-03-21T10:50:43.025 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 254, 2024-03-21T10:50:43.025 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 1731344, 2024-03-21T10:50:43.025 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 152736, 2024-03-21T10:50:43.025 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 2, 2024-03-21T10:50:43.025 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 1, 2024-03-21T10:50:43.025 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.116630351, 2024-03-21T10:50:43.025 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.044465741, 2024-03-21T10:50:43.026 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.077582102, 2024-03-21T10:50:43.026 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.011285598, 2024-03-21T10:50:43.026 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:50:43.026 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 253, 2024-03-21T10:50:43.026 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.062360259, 2024-03-21T10:50:43.026 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000246483 2024-03-21T10:50:43.026 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:43.026 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:50:43.026 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:43.026 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:43.026 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:43.027 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:43.027 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 1731344, 2024-03-21T10:50:43.027 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 152736 2024-03-21T10:50:43.027 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:43.027 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-2": { 2024-03-21T10:50:43.027 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 6, 2024-03-21T10:50:43.027 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 195, 2024-03-21T10:50:43.027 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 3312, 2024-03-21T10:50:43.027 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 189504, 2024-03-21T10:50:43.027 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 5, 2024-03-21T10:50:43.027 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 3, 2024-03-21T10:50:43.027 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.040418217, 2024-03-21T10:50:43.028 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.028723578, 2024-03-21T10:50:43.028 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.006373709, 2024-03-21T10:50:43.028 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-21T10:50:43.028 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:50:43.028 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 193, 2024-03-21T10:50:43.028 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.048640287, 2024-03-21T10:50:43.028 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000252022 2024-03-21T10:50:43.028 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:43.028 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:50:43.028 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:43.028 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:43.028 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:43.028 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:43.029 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-21T10:50:43.029 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 189504 2024-03-21T10:50:43.029 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:43.029 INFO:teuthology.orchestra.run.smithi031.stdout: "cct": { 2024-03-21T10:50:43.029 INFO:teuthology.orchestra.run.smithi031.stdout: "total_workers": 0, 2024-03-21T10:50:43.029 INFO:teuthology.orchestra.run.smithi031.stdout: "unhealthy_workers": 0 2024-03-21T10:50:43.029 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:43.029 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool": { 2024-03-21T10:50:43.029 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_bytes": 0, 2024-03-21T10:50:43.029 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_items": 0, 2024-03-21T10:50:43.029 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_bytes": 0, 2024-03-21T10:50:43.029 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_items": 0, 2024-03-21T10:50:43.029 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_bytes": 0, 2024-03-21T10:50:43.029 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_items": 0, 2024-03-21T10:50:43.029 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-21T10:50:43.030 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_items": 0, 2024-03-21T10:50:43.030 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-21T10:50:43.030 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_items": 0, 2024-03-21T10:50:43.030 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_bytes": 0, 2024-03-21T10:50:43.030 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_items": 0, 2024-03-21T10:50:43.030 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-21T10:50:43.030 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_items": 0, 2024-03-21T10:50:43.030 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_bytes": 0, 2024-03-21T10:50:43.030 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_items": 0, 2024-03-21T10:50:43.030 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_bytes": 0, 2024-03-21T10:50:43.030 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_items": 0, 2024-03-21T10:50:43.030 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-21T10:50:43.030 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_items": 0, 2024-03-21T10:50:43.030 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-21T10:50:43.031 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_items": 0, 2024-03-21T10:50:43.031 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_bytes": 0, 2024-03-21T10:50:43.031 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_items": 0, 2024-03-21T10:50:43.031 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_bytes": 0, 2024-03-21T10:50:43.031 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_items": 0, 2024-03-21T10:50:43.031 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-21T10:50:43.031 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_items": 0, 2024-03-21T10:50:43.031 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_bytes": 0, 2024-03-21T10:50:43.031 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_items": 0, 2024-03-21T10:50:43.031 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_bytes": 0, 2024-03-21T10:50:43.031 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_items": 0, 2024-03-21T10:50:43.031 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_bytes": 0, 2024-03-21T10:50:43.031 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_items": 0, 2024-03-21T10:50:43.032 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_bytes": 0, 2024-03-21T10:50:43.032 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_items": 0, 2024-03-21T10:50:43.032 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_bytes": 148714, 2024-03-21T10:50:43.032 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_items": 30, 2024-03-21T10:50:43.032 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_bytes": 0, 2024-03-21T10:50:43.032 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_items": 0, 2024-03-21T10:50:43.032 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bytes": 0, 2024-03-21T10:50:43.032 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_items": 0, 2024-03-21T10:50:43.032 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_bytes": 0, 2024-03-21T10:50:43.032 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_items": 0, 2024-03-21T10:50:43.032 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_bytes": 0, 2024-03-21T10:50:43.032 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_items": 0, 2024-03-21T10:50:43.032 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_bytes": 0, 2024-03-21T10:50:43.032 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_items": 0, 2024-03-21T10:50:43.032 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_bytes": 0, 2024-03-21T10:50:43.033 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_items": 0, 2024-03-21T10:50:43.033 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_bytes": 0, 2024-03-21T10:50:43.033 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_items": 0, 2024-03-21T10:50:43.033 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_bytes": 928, 2024-03-21T10:50:43.033 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_items": 2, 2024-03-21T10:50:43.033 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_bytes": 0, 2024-03-21T10:50:43.033 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_items": 0, 2024-03-21T10:50:43.033 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_bytes": 0, 2024-03-21T10:50:43.033 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_items": 0 2024-03-21T10:50:43.033 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:43.033 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-21T10:50:43.033 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-21T10:50:43.033 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-03-21T10:50:43.033 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-21T10:50:43.033 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 268, 2024-03-21T10:50:43.034 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 1693819, 2024-03-21T10:50:43.034 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-21T10:50:43.034 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 268, 2024-03-21T10:50:43.034 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-03-21T10:50:43.034 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-03-21T10:50:43.034 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 268, 2024-03-21T10:50:43.034 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 1693819, 2024-03-21T10:50:43.034 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-21T10:50:43.034 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:43.034 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:43.034 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:43.034 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:43.034 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:43.034 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:50:43.053 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:43 smithi031 ceph-mon[31673]: pgmap v379: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.3 KiB/s wr, 6 op/s 2024-03-21T10:50:43.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:43 smithi070 ceph-mon[36330]: pgmap v379: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.3 KiB/s wr, 6 op/s 2024-03-21T10:50:43.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:43 smithi040 ceph-mon[36154]: pgmap v379: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.3 KiB/s wr, 6 op/s 2024-03-21T10:50:44.043 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-03-21T10:50:44.043 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-03-21T10:50:44.043 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-03-21T10:50:44.043 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-03-21T10:50:44.044 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-03-21T10:50:44.044 DEBUG:tasks.check_counter:Getting stats from j 2024-03-21T10:50:44.044 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-21T10:50:44.510 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T10:50:45.215 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:45 smithi031 ceph-mon[31673]: pgmap v380: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.3 KiB/s wr, 6 op/s 2024-03-21T10:50:45.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:45 smithi070 ceph-mon[36330]: pgmap v380: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.3 KiB/s wr, 6 op/s 2024-03-21T10:50:45.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:45 smithi040 ceph-mon[36154]: pgmap v380: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.3 KiB/s wr, 6 op/s 2024-03-21T10:50:47.007 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-21T10:50:47.007 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-0": { 2024-03-21T10:50:47.008 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 4, 2024-03-21T10:50:47.008 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 4, 2024-03-21T10:50:47.008 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 2992, 2024-03-21T10:50:47.008 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 448, 2024-03-21T10:50:47.008 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 4, 2024-03-21T10:50:47.008 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 0, 2024-03-21T10:50:47.008 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.004366058, 2024-03-21T10:50:47.008 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.000073213, 2024-03-21T10:50:47.008 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.001461160, 2024-03-21T10:50:47.008 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-21T10:50:47.008 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:50:47.008 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 3, 2024-03-21T10:50:47.008 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.001674558, 2024-03-21T10:50:47.008 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000558186 2024-03-21T10:50:47.009 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:47.009 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:50:47.009 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:47.009 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:47.009 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:47.009 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:47.009 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-21T10:50:47.009 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 448 2024-03-21T10:50:47.009 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:47.009 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-1": { 2024-03-21T10:50:47.009 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 2, 2024-03-21T10:50:47.009 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 199, 2024-03-21T10:50:47.009 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 320, 2024-03-21T10:50:47.009 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 190112, 2024-03-21T10:50:47.010 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 5, 2024-03-21T10:50:47.010 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 3, 2024-03-21T10:50:47.010 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.037345879, 2024-03-21T10:50:47.010 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.027183109, 2024-03-21T10:50:47.010 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.003924360, 2024-03-21T10:50:47.010 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-21T10:50:47.010 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:50:47.010 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 196, 2024-03-21T10:50:47.010 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.078756595, 2024-03-21T10:50:47.010 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000401819 2024-03-21T10:50:47.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:47.011 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:50:47.011 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:47.011 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:47.011 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:47.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:47.011 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-21T10:50:47.011 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 190112 2024-03-21T10:50:47.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:47.011 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-2": { 2024-03-21T10:50:47.011 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 273, 2024-03-21T10:50:47.011 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 261, 2024-03-21T10:50:47.012 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 1733248, 2024-03-21T10:50:47.012 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 157104, 2024-03-21T10:50:47.012 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 2, 2024-03-21T10:50:47.012 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 1, 2024-03-21T10:50:47.012 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.123438891, 2024-03-21T10:50:47.012 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.045788491, 2024-03-21T10:50:47.012 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.087886455, 2024-03-21T10:50:47.012 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.014653676, 2024-03-21T10:50:47.012 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:50:47.012 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 260, 2024-03-21T10:50:47.012 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.054498155, 2024-03-21T10:50:47.012 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000209608 2024-03-21T10:50:47.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:47.013 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:50:47.013 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:47.013 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:47.013 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:47.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:47.013 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 1733248, 2024-03-21T10:50:47.013 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 157104 2024-03-21T10:50:47.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:47.013 INFO:teuthology.orchestra.run.smithi031.stdout: "cct": { 2024-03-21T10:50:47.013 INFO:teuthology.orchestra.run.smithi031.stdout: "total_workers": 0, 2024-03-21T10:50:47.013 INFO:teuthology.orchestra.run.smithi031.stdout: "unhealthy_workers": 0 2024-03-21T10:50:47.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:47.013 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool": { 2024-03-21T10:50:47.014 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_bytes": 0, 2024-03-21T10:50:47.014 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_items": 0, 2024-03-21T10:50:47.014 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_bytes": 0, 2024-03-21T10:50:47.014 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_items": 0, 2024-03-21T10:50:47.014 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_bytes": 0, 2024-03-21T10:50:47.014 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_items": 0, 2024-03-21T10:50:47.014 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-21T10:50:47.014 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_items": 0, 2024-03-21T10:50:47.014 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-21T10:50:47.015 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_items": 0, 2024-03-21T10:50:47.015 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_bytes": 0, 2024-03-21T10:50:47.015 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_items": 0, 2024-03-21T10:50:47.015 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-21T10:50:47.015 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_items": 0, 2024-03-21T10:50:47.015 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_bytes": 0, 2024-03-21T10:50:47.015 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_items": 0, 2024-03-21T10:50:47.015 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_bytes": 0, 2024-03-21T10:50:47.016 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_items": 0, 2024-03-21T10:50:47.016 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-21T10:50:47.016 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_items": 0, 2024-03-21T10:50:47.016 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-21T10:50:47.016 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_items": 0, 2024-03-21T10:50:47.016 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_bytes": 0, 2024-03-21T10:50:47.017 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_items": 0, 2024-03-21T10:50:47.017 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_bytes": 0, 2024-03-21T10:50:47.017 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_items": 0, 2024-03-21T10:50:47.017 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-21T10:50:47.017 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_items": 0, 2024-03-21T10:50:47.017 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_bytes": 0, 2024-03-21T10:50:47.017 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_items": 0, 2024-03-21T10:50:47.018 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_bytes": 0, 2024-03-21T10:50:47.018 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_items": 0, 2024-03-21T10:50:47.018 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_bytes": 0, 2024-03-21T10:50:47.018 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_items": 0, 2024-03-21T10:50:47.018 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_bytes": 0, 2024-03-21T10:50:47.018 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_items": 0, 2024-03-21T10:50:47.019 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_bytes": 148714, 2024-03-21T10:50:47.019 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_items": 30, 2024-03-21T10:50:47.019 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_bytes": 0, 2024-03-21T10:50:47.019 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_items": 0, 2024-03-21T10:50:47.019 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bytes": 0, 2024-03-21T10:50:47.019 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_items": 0, 2024-03-21T10:50:47.019 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_bytes": 0, 2024-03-21T10:50:47.020 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_items": 0, 2024-03-21T10:50:47.020 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_bytes": 0, 2024-03-21T10:50:47.020 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_items": 0, 2024-03-21T10:50:47.020 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_bytes": 0, 2024-03-21T10:50:47.020 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_items": 0, 2024-03-21T10:50:47.020 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_bytes": 0, 2024-03-21T10:50:47.021 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_items": 0, 2024-03-21T10:50:47.021 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_bytes": 0, 2024-03-21T10:50:47.021 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_items": 0, 2024-03-21T10:50:47.021 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_bytes": 928, 2024-03-21T10:50:47.021 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_items": 2, 2024-03-21T10:50:47.021 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_bytes": 0, 2024-03-21T10:50:47.021 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_items": 0, 2024-03-21T10:50:47.022 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_bytes": 0, 2024-03-21T10:50:47.022 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_items": 0 2024-03-21T10:50:47.022 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-21T10:50:47.022 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-21T10:50:47.022 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-21T10:50:47.022 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-03-21T10:50:47.022 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-21T10:50:47.023 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 275, 2024-03-21T10:50:47.023 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 1694729, 2024-03-21T10:50:47.023 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-21T10:50:47.023 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 275, 2024-03-21T10:50:47.023 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-03-21T10:50:47.023 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-03-21T10:50:47.024 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 275, 2024-03-21T10:50:47.024 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 1694729, 2024-03-21T10:50:47.024 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-21T10:50:47.024 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-21T10:50:47.024 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-21T10:50:47.024 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-21T10:50:47.025 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:47.025 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-21T10:50:47.025 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-21T10:50:47.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:47 smithi031 ceph-mon[31673]: pgmap v381: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 4.3 KiB/s wr, 8 op/s 2024-03-21T10:50:47.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:47 smithi070 ceph-mon[36330]: pgmap v381: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 4.3 KiB/s wr, 8 op/s 2024-03-21T10:50:47.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:47 smithi040 ceph-mon[36154]: pgmap v381: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 4.3 KiB/s wr, 8 op/s 2024-03-21T10:50:48.052 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-21T10:50:48.053 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-21T10:50:48.053 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-03-21T10:50:48.053 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-03-21T10:50:48.053 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-03-21T10:50:48.053 DEBUG:tasks.check_counter:Getting stats from b 2024-03-21T10:50:48.053 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-21T10:50:48.327 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T10:50:49.303 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:49 smithi031 ceph-mon[31673]: pgmap v382: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-03-21T10:50:49.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:49 smithi070 ceph-mon[36330]: pgmap v382: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-03-21T10:50:49.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:49 smithi040 ceph-mon[36154]: pgmap v382: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-03-21T10:50:50.390 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-21T10:50:50.390 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-03-21T10:50:50.391 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 2615, 2024-03-21T10:50:50.391 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 3018, 2024-03-21T10:50:50.391 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 973377, 2024-03-21T10:50:50.391 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 12154505, 2024-03-21T10:50:50.391 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 270, 2024-03-21T10:50:50.391 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 5, 2024-03-21T10:50:50.391 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.294161829, 2024-03-21T10:50:50.391 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.082199456, 2024-03-21T10:50:50.391 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.141944836, 2024-03-21T10:50:50.391 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.049224184, 2024-03-21T10:50:50.391 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:50:50.391 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 3019, 2024-03-21T10:50:50.391 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.288298784, 2024-03-21T10:50:50.392 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000095494 2024-03-21T10:50:50.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.392 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:50:50.392 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2566, 2024-03-21T10:50:50.392 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000535365, 2024-03-21T10:50:50.392 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000208 2024-03-21T10:50:50.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.392 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 42064, 2024-03-21T10:50:50.392 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 1056 2024-03-21T10:50:50.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.392 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-03-21T10:50:50.392 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 2817, 2024-03-21T10:50:50.392 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 2407, 2024-03-21T10:50:50.393 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 2706991, 2024-03-21T10:50:50.393 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 1490839, 2024-03-21T10:50:50.393 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 14, 2024-03-21T10:50:50.393 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 6, 2024-03-21T10:50:50.393 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.239674924, 2024-03-21T10:50:50.393 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.068181551, 2024-03-21T10:50:50.393 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.142466226, 2024-03-21T10:50:50.393 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.053518443, 2024-03-21T10:50:50.393 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:50:50.393 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2407, 2024-03-21T10:50:50.393 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.383828368, 2024-03-21T10:50:50.393 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000159463 2024-03-21T10:50:50.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.394 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:50:50.394 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1157, 2024-03-21T10:50:50.394 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000468349, 2024-03-21T10:50:50.394 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000404 2024-03-21T10:50:50.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.394 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 1879056, 2024-03-21T10:50:50.394 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 574128 2024-03-21T10:50:50.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.394 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-03-21T10:50:50.394 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 2834, 2024-03-21T10:50:50.394 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 2322, 2024-03-21T10:50:50.394 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 1086450, 2024-03-21T10:50:50.394 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 4882914, 2024-03-21T10:50:50.395 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 27, 2024-03-21T10:50:50.395 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 7, 2024-03-21T10:50:50.395 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.240568756, 2024-03-21T10:50:50.395 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.070594990, 2024-03-21T10:50:50.395 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.094504674, 2024-03-21T10:50:50.395 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.068867241, 2024-03-21T10:50:50.395 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:50:50.395 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2316, 2024-03-21T10:50:50.395 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.192024970, 2024-03-21T10:50:50.395 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000082912 2024-03-21T10:50:50.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.395 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:50:50.395 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 3393, 2024-03-21T10:50:50.395 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000618772, 2024-03-21T10:50:50.396 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000182 2024-03-21T10:50:50.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.396 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 416080, 2024-03-21T10:50:50.396 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 267536 2024-03-21T10:50:50.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.396 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-03-21T10:50:50.396 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 1, 2024-03-21T10:50:50.396 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-03-21T10:50:50.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.396 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-MDSRank": { 2024-03-21T10:50:50.396 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-03-21T10:50:50.396 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-03-21T10:50:50.396 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2332, 2024-03-21T10:50:50.397 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.983575685, 2024-03-21T10:50:50.397 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000421773 2024-03-21T10:50:50.397 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-21T10:50:50.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.397 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-PurgeQueue": { 2024-03-21T10:50:50.397 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-03-21T10:50:50.397 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-03-21T10:50:50.397 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2, 2024-03-21T10:50:50.397 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000006419, 2024-03-21T10:50:50.397 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000003209 2024-03-21T10:50:50.397 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-21T10:50:50.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.397 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": { 2024-03-21T10:50:50.398 INFO:teuthology.orchestra.run.smithi040.stdout: "request": 204, 2024-03-21T10:50:50.398 INFO:teuthology.orchestra.run.smithi040.stdout: "reply": 204, 2024-03-21T10:50:50.398 INFO:teuthology.orchestra.run.smithi040.stdout: "reply_latency": { 2024-03-21T10:50:50.398 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 204, 2024-03-21T10:50:50.398 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.224771531, 2024-03-21T10:50:50.398 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.001101821 2024-03-21T10:50:50.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.398 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_reply": 0, 2024-03-21T10:50:50.398 INFO:teuthology.orchestra.run.smithi040.stdout: "forward": 0, 2024-03-21T10:50:50.398 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_complete": 3, 2024-03-21T10:50:50.398 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_keys": 2, 2024-03-21T10:50:50.398 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_commit": 276, 2024-03-21T10:50:50.398 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_split": 0, 2024-03-21T10:50:50.399 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_merge": 0, 2024-03-21T10:50:50.399 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes": 26, 2024-03-21T10:50:50.399 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_top": 7, 2024-03-21T10:50:50.399 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_bottom": 7, 2024-03-21T10:50:50.399 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pin_tail": 12, 2024-03-21T10:50:50.399 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pinned": 17, 2024-03-21T10:50:50.399 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_expired": 85, 2024-03-21T10:50:50.399 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_with_caps": 2, 2024-03-21T10:50:50.399 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": 2, 2024-03-21T10:50:50.399 INFO:teuthology.orchestra.run.smithi040.stdout: "subtrees": 3, 2024-03-21T10:50:50.399 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse": 275, 2024-03-21T10:50:50.400 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_hit": 268, 2024-03-21T10:50:50.400 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_forward": 0, 2024-03-21T10:50:50.400 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_discover": 0, 2024-03-21T10:50:50.400 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_dir_fetch": 0, 2024-03-21T10:50:50.400 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_remote_ino": 0, 2024-03-21T10:50:50.400 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_lock": 0, 2024-03-21T10:50:50.401 INFO:teuthology.orchestra.run.smithi040.stdout: "load_cent": 11, 2024-03-21T10:50:50.401 INFO:teuthology.orchestra.run.smithi040.stdout: "q": 0, 2024-03-21T10:50:50.401 INFO:teuthology.orchestra.run.smithi040.stdout: "exported": 2, 2024-03-21T10:50:50.401 INFO:teuthology.orchestra.run.smithi040.stdout: "exported_inodes": 77, 2024-03-21T10:50:50.401 INFO:teuthology.orchestra.run.smithi040.stdout: "imported": 1, 2024-03-21T10:50:50.402 INFO:teuthology.orchestra.run.smithi040.stdout: "imported_inodes": 2, 2024-03-21T10:50:50.402 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_dir_fetch": 0, 2024-03-21T10:50:50.402 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_backtrace_fetch": 454, 2024-03-21T10:50:50.402 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_peer_discover": 0, 2024-03-21T10:50:50.402 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rfiles": 4, 2024-03-21T10:50:50.402 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rbytes": 234, 2024-03-21T10:50:50.403 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rsnaps": 6, 2024-03-21T10:50:50.403 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_fetch": 454, 2024-03-21T10:50:50.403 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_set_tag": 0, 2024-03-21T10:50:50.403 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_repaired": 0, 2024-03-21T10:50:50.403 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_inotable_repaired": 0, 2024-03-21T10:50:50.403 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_inodes": 194, 2024-03-21T10:50:50.403 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_base_inodes": 21, 2024-03-21T10:50:50.403 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dirfrag_rstats": 173, 2024-03-21T10:50:50.403 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_file_inodes": 260, 2024-03-21T10:50:50.403 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_inode_file_caps": 1, 2024-03-21T10:50:50.403 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_revoke": 66, 2024-03-21T10:50:50.404 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_grant": 210, 2024-03-21T10:50:50.404 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_trunc": 0, 2024-03-21T10:50:50.404 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-21T10:50:50.404 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-21T10:50:50.404 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps": 1099, 2024-03-21T10:50:50.404 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps_dirty": 1023, 2024-03-21T10:50:50.404 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_cap_release": 0, 2024-03-21T10:50:50.404 INFO:teuthology.orchestra.run.smithi040.stdout: "process_request_cap_release": 21 2024-03-21T10:50:50.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.404 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache": { 2024-03-21T10:50:50.404 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update": 0, 2024-03-21T10:50:50.404 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update_receipt": 0, 2024-03-21T10:50:50.404 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_try_discover": 0, 2024-03-21T10:50:50.404 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_send_discover": 0, 2024-03-21T10:50:50.405 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_handle_discover": 8, 2024-03-21T10:50:50.405 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays": 0, 2024-03-21T10:50:50.405 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_delayed": 0, 2024-03-21T10:50:50.405 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_enqueuing": 0, 2024-03-21T10:50:50.405 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_created": 0, 2024-03-21T10:50:50.405 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_enqueued": 0, 2024-03-21T10:50:50.405 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_reintegrated": 0, 2024-03-21T10:50:50.405 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_migrated": 0, 2024-03-21T10:50:50.405 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_processing": 0, 2024-03-21T10:50:50.405 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_enqueued": 0, 2024-03-21T10:50:50.405 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_prioritized": 0, 2024-03-21T10:50:50.405 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_started": 0, 2024-03-21T10:50:50.405 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_completed": 0, 2024-03-21T10:50:50.406 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_enqueue_scrub": 21, 2024-03-21T10:50:50.406 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_exportdir": 2, 2024-03-21T10:50:50.406 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_flush": 0, 2024-03-21T10:50:50.406 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragmentdir": 0, 2024-03-21T10:50:50.406 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragstats": 0, 2024-03-21T10:50:50.406 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_inodestats": 0 2024-03-21T10:50:50.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.406 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log": { 2024-03-21T10:50:50.406 INFO:teuthology.orchestra.run.smithi040.stdout: "evlrg": 0, 2024-03-21T10:50:50.406 INFO:teuthology.orchestra.run.smithi040.stdout: "evadd": 1631, 2024-03-21T10:50:50.406 INFO:teuthology.orchestra.run.smithi040.stdout: "evex": 1624, 2024-03-21T10:50:50.406 INFO:teuthology.orchestra.run.smithi040.stdout: "evtrm": 1624, 2024-03-21T10:50:50.406 INFO:teuthology.orchestra.run.smithi040.stdout: "ev": 7, 2024-03-21T10:50:50.406 INFO:teuthology.orchestra.run.smithi040.stdout: "evexg": 0, 2024-03-21T10:50:50.407 INFO:teuthology.orchestra.run.smithi040.stdout: "evexd": 10, 2024-03-21T10:50:50.407 INFO:teuthology.orchestra.run.smithi040.stdout: "segadd": 56, 2024-03-21T10:50:50.407 INFO:teuthology.orchestra.run.smithi040.stdout: "segex": 55, 2024-03-21T10:50:50.407 INFO:teuthology.orchestra.run.smithi040.stdout: "segtrm": 55, 2024-03-21T10:50:50.407 INFO:teuthology.orchestra.run.smithi040.stdout: "seg": 1, 2024-03-21T10:50:50.407 INFO:teuthology.orchestra.run.smithi040.stdout: "segmjr": 1, 2024-03-21T10:50:50.407 INFO:teuthology.orchestra.run.smithi040.stdout: "segexg": 0, 2024-03-21T10:50:50.407 INFO:teuthology.orchestra.run.smithi040.stdout: "segexd": 1, 2024-03-21T10:50:50.407 INFO:teuthology.orchestra.run.smithi040.stdout: "expos": 16221209, 2024-03-21T10:50:50.407 INFO:teuthology.orchestra.run.smithi040.stdout: "wrpos": 16299517, 2024-03-21T10:50:50.407 INFO:teuthology.orchestra.run.smithi040.stdout: "rdpos": 0, 2024-03-21T10:50:50.407 INFO:teuthology.orchestra.run.smithi040.stdout: "jlat": { 2024-03-21T10:50:50.407 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 738, 2024-03-21T10:50:50.407 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 2.292820814, 2024-03-21T10:50:50.408 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.003106803 2024-03-21T10:50:50.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.408 INFO:teuthology.orchestra.run.smithi040.stdout: "replayed": 0 2024-03-21T10:50:50.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.408 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_mem": { 2024-03-21T10:50:50.408 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 25, 2024-03-21T10:50:50.408 INFO:teuthology.orchestra.run.smithi040.stdout: "ino+": 126, 2024-03-21T10:50:50.408 INFO:teuthology.orchestra.run.smithi040.stdout: "ino-": 101, 2024-03-21T10:50:50.408 INFO:teuthology.orchestra.run.smithi040.stdout: "dir": 20, 2024-03-21T10:50:50.408 INFO:teuthology.orchestra.run.smithi040.stdout: "dir+": 26, 2024-03-21T10:50:50.408 INFO:teuthology.orchestra.run.smithi040.stdout: "dir-": 6, 2024-03-21T10:50:50.408 INFO:teuthology.orchestra.run.smithi040.stdout: "dn": 26, 2024-03-21T10:50:50.408 INFO:teuthology.orchestra.run.smithi040.stdout: "dn+": 111, 2024-03-21T10:50:50.408 INFO:teuthology.orchestra.run.smithi040.stdout: "dn-": 85, 2024-03-21T10:50:50.409 INFO:teuthology.orchestra.run.smithi040.stdout: "cap": 2, 2024-03-21T10:50:50.409 INFO:teuthology.orchestra.run.smithi040.stdout: "cap+": 36, 2024-03-21T10:50:50.409 INFO:teuthology.orchestra.run.smithi040.stdout: "cap-": 34, 2024-03-21T10:50:50.409 INFO:teuthology.orchestra.run.smithi040.stdout: "rss": 57156, 2024-03-21T10:50:50.409 INFO:teuthology.orchestra.run.smithi040.stdout: "heap": 231804 2024-03-21T10:50:50.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.409 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_server": { 2024-03-21T10:50:50.409 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_client_request": 254, 2024-03-21T10:50:50.409 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_server_request": 3, 2024-03-21T10:50:50.409 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_request": 204, 2024-03-21T10:50:50.409 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_session": 39, 2024-03-21T10:50:50.409 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_peer_request": 6, 2024-03-21T10:50:50.409 INFO:teuthology.orchestra.run.smithi040.stdout: "req_create_latency": { 2024-03-21T10:50:50.409 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 22, 2024-03-21T10:50:50.410 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.028733225, 2024-03-21T10:50:50.410 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.001306055 2024-03-21T10:50:50.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.410 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getattr_latency": { 2024-03-21T10:50:50.410 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 129, 2024-03-21T10:50:50.410 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.023968353, 2024-03-21T10:50:50.410 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000185801 2024-03-21T10:50:50.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.410 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getfilelock_latency": { 2024-03-21T10:50:50.410 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:50.410 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:50.410 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:50.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.411 INFO:teuthology.orchestra.run.smithi040.stdout: "req_link_latency": { 2024-03-21T10:50:50.411 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:50.411 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:50.411 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:50.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.411 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookup_latency": { 2024-03-21T10:50:50.411 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:50.411 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:50.411 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:50.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.411 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookuphash_latency": { 2024-03-21T10:50:50.411 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:50.411 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:50.412 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:50.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.412 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupino_latency": { 2024-03-21T10:50:50.412 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:50.412 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:50.413 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:50.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.413 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupname_latency": { 2024-03-21T10:50:50.413 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:50.413 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:50.413 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:50.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.414 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupparent_latency": { 2024-03-21T10:50:50.414 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:50.414 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:50.414 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:50.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.415 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupsnap_latency": { 2024-03-21T10:50:50.415 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:50.415 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:50.415 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:50.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.415 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lssnap_latency": { 2024-03-21T10:50:50.416 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 12, 2024-03-21T10:50:50.416 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.002106546, 2024-03-21T10:50:50.416 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000175545 2024-03-21T10:50:50.416 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.416 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mkdir_latency": { 2024-03-21T10:50:50.417 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 10, 2024-03-21T10:50:50.417 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.012869765, 2024-03-21T10:50:50.417 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.001286976 2024-03-21T10:50:50.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.417 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mknod_latency": { 2024-03-21T10:50:50.417 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:50.417 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:50.417 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:50.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.417 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mksnap_latency": { 2024-03-21T10:50:50.417 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 12, 2024-03-21T10:50:50.418 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.096902707, 2024-03-21T10:50:50.418 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.008075225 2024-03-21T10:50:50.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.418 INFO:teuthology.orchestra.run.smithi040.stdout: "req_open_latency": { 2024-03-21T10:50:50.418 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:50.418 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:50.418 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:50.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.418 INFO:teuthology.orchestra.run.smithi040.stdout: "req_readdir_latency": { 2024-03-21T10:50:50.418 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1, 2024-03-21T10:50:50.419 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.006243853, 2024-03-21T10:50:50.419 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.006243853 2024-03-21T10:50:50.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.419 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rename_latency": { 2024-03-21T10:50:50.419 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2, 2024-03-21T10:50:50.419 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.002291433, 2024-03-21T10:50:50.419 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.001145716 2024-03-21T10:50:50.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.419 INFO:teuthology.orchestra.run.smithi040.stdout: "req_renamesnap_latency": { 2024-03-21T10:50:50.419 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:50.419 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:50.419 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:50.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.419 INFO:teuthology.orchestra.run.smithi040.stdout: "req_snapdiff_latency": { 2024-03-21T10:50:50.420 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:50.420 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:50.420 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:50.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.420 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmdir_latency": { 2024-03-21T10:50:50.420 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:50.420 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:50.420 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:50.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.421 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmsnap_latency": { 2024-03-21T10:50:50.421 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 6, 2024-03-21T10:50:50.421 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.032087073, 2024-03-21T10:50:50.421 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.005347845 2024-03-21T10:50:50.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.422 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmxattr_latency": { 2024-03-21T10:50:50.422 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:50.422 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:50.422 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:50.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.422 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setattr_latency": { 2024-03-21T10:50:50.422 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2, 2024-03-21T10:50:50.422 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.001284418, 2024-03-21T10:50:50.423 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000642209 2024-03-21T10:50:50.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.423 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setdirlayout_latency": { 2024-03-21T10:50:50.423 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:50.423 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:50.423 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:50.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.423 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setfilelock_latency": { 2024-03-21T10:50:50.423 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:50.423 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:50.423 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:50.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.423 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setlayout_latency": { 2024-03-21T10:50:50.424 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:50.424 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:50.424 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:50.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.424 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setxattr_latency": { 2024-03-21T10:50:50.424 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 8, 2024-03-21T10:50:50.424 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.018284158, 2024-03-21T10:50:50.424 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.002285519 2024-03-21T10:50:50.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.424 INFO:teuthology.orchestra.run.smithi040.stdout: "req_symlink_latency": { 2024-03-21T10:50:50.424 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:50.424 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:50.424 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:50.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.424 INFO:teuthology.orchestra.run.smithi040.stdout: "req_unlink_latency": { 2024-03-21T10:50:50.425 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:50.425 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:50.425 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:50.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.425 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_revoke_eviction": 0, 2024-03-21T10:50:50.425 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_acquisition_throttle": 0, 2024-03-21T10:50:50.425 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getvxattr_latency": { 2024-03-21T10:50:50.425 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:50.425 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:50.425 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:50.425 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-21T10:50:50.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.425 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_sessions": { 2024-03-21T10:50:50.425 INFO:teuthology.orchestra.run.smithi040.stdout: "session_count": 2, 2024-03-21T10:50:50.426 INFO:teuthology.orchestra.run.smithi040.stdout: "session_add": 6, 2024-03-21T10:50:50.426 INFO:teuthology.orchestra.run.smithi040.stdout: "session_remove": 4, 2024-03-21T10:50:50.426 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_open": 2, 2024-03-21T10:50:50.426 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_stale": 0, 2024-03-21T10:50:50.426 INFO:teuthology.orchestra.run.smithi040.stdout: "total_load": 27, 2024-03-21T10:50:50.426 INFO:teuthology.orchestra.run.smithi040.stdout: "average_load": 13, 2024-03-21T10:50:50.426 INFO:teuthology.orchestra.run.smithi040.stdout: "avg_session_uptime": 730, 2024-03-21T10:50:50.426 INFO:teuthology.orchestra.run.smithi040.stdout: "mdthresh_evicted": 0 2024-03-21T10:50:50.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.426 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-03-21T10:50:50.426 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-03-21T10:50:50.426 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-03-21T10:50:50.426 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-03-21T10:50:50.426 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-03-21T10:50:50.427 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-03-21T10:50:50.427 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-03-21T10:50:50.427 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-21T10:50:50.427 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-03-21T10:50:50.427 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-21T10:50:50.427 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-03-21T10:50:50.427 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-03-21T10:50:50.427 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-03-21T10:50:50.427 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-21T10:50:50.427 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-03-21T10:50:50.427 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-03-21T10:50:50.427 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-03-21T10:50:50.427 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-03-21T10:50:50.427 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-03-21T10:50:50.427 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-21T10:50:50.428 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-03-21T10:50:50.428 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-21T10:50:50.428 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-03-21T10:50:50.428 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-03-21T10:50:50.428 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-03-21T10:50:50.428 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-03-21T10:50:50.428 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-03-21T10:50:50.428 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-21T10:50:50.428 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-03-21T10:50:50.428 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-03-21T10:50:50.428 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-03-21T10:50:50.428 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-03-21T10:50:50.428 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-03-21T10:50:50.428 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-03-21T10:50:50.428 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-03-21T10:50:50.429 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-03-21T10:50:50.429 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-03-21T10:50:50.429 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 332845, 2024-03-21T10:50:50.429 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 121, 2024-03-21T10:50:50.429 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 352, 2024-03-21T10:50:50.429 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 4, 2024-03-21T10:50:50.429 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-03-21T10:50:50.429 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-03-21T10:50:50.429 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-03-21T10:50:50.429 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-03-21T10:50:50.429 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-03-21T10:50:50.429 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-03-21T10:50:50.429 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 9328, 2024-03-21T10:50:50.429 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 179, 2024-03-21T10:50:50.429 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-03-21T10:50:50.429 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-03-21T10:50:50.430 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-03-21T10:50:50.430 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-03-21T10:50:50.430 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 128626, 2024-03-21T10:50:50.430 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 657, 2024-03-21T10:50:50.430 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-03-21T10:50:50.430 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-03-21T10:50:50.430 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-03-21T10:50:50.430 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-03-21T10:50:50.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.430 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter": { 2024-03-21T10:50:50.430 INFO:teuthology.orchestra.run.smithi040.stdout: "op_active": 0, 2024-03-21T10:50:50.430 INFO:teuthology.orchestra.run.smithi040.stdout: "op_laggy": 0, 2024-03-21T10:50:50.430 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send": 2384, 2024-03-21T10:50:50.430 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send_bytes": 13135807, 2024-03-21T10:50:50.431 INFO:teuthology.orchestra.run.smithi040.stdout: "op_resend": 31, 2024-03-21T10:50:50.431 INFO:teuthology.orchestra.run.smithi040.stdout: "op_reply": 2384, 2024-03-21T10:50:50.431 INFO:teuthology.orchestra.run.smithi040.stdout: "op_latency": { 2024-03-21T10:50:50.431 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2384, 2024-03-21T10:50:50.431 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 39.195523040, 2024-03-21T10:50:50.431 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.016441075 2024-03-21T10:50:50.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.431 INFO:teuthology.orchestra.run.smithi040.stdout: "op_inflight": 0, 2024-03-21T10:50:50.431 INFO:teuthology.orchestra.run.smithi040.stdout: "oplen_avg": { 2024-03-21T10:50:50.431 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2384, 2024-03-21T10:50:50.431 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 3052 2024-03-21T10:50:50.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.431 INFO:teuthology.orchestra.run.smithi040.stdout: "op": 2384, 2024-03-21T10:50:50.431 INFO:teuthology.orchestra.run.smithi040.stdout: "op_r": 501, 2024-03-21T10:50:50.431 INFO:teuthology.orchestra.run.smithi040.stdout: "op_w": 1883, 2024-03-21T10:50:50.432 INFO:teuthology.orchestra.run.smithi040.stdout: "op_rmw": 0, 2024-03-21T10:50:50.432 INFO:teuthology.orchestra.run.smithi040.stdout: "op_pg": 0, 2024-03-21T10:50:50.432 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_stat": 254, 2024-03-21T10:50:50.432 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_create": 66, 2024-03-21T10:50:50.432 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_read": 21, 2024-03-21T10:50:50.432 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_write": 738, 2024-03-21T10:50:50.432 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writefull": 175, 2024-03-21T10:50:50.432 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writesame": 0, 2024-03-21T10:50:50.432 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_append": 0, 2024-03-21T10:50:50.432 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_zero": 0, 2024-03-21T10:50:50.432 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_truncate": 0, 2024-03-21T10:50:50.432 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_delete": 510, 2024-03-21T10:50:50.432 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_mapext": 0, 2024-03-21T10:50:50.432 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_sparse_read": 0, 2024-03-21T10:50:50.432 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_clonerange": 0, 2024-03-21T10:50:50.432 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_getxattr": 480, 2024-03-21T10:50:50.433 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_setxattr": 110, 2024-03-21T10:50:50.433 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_cmpxattr": 0, 2024-03-21T10:50:50.433 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_rmxattr": 0, 2024-03-21T10:50:50.433 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_resetxattrs": 0, 2024-03-21T10:50:50.433 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_call": 0, 2024-03-21T10:50:50.433 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_watch": 0, 2024-03-21T10:50:50.433 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_notify": 0, 2024-03-21T10:50:50.433 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_src_cmpxattr": 0, 2024-03-21T10:50:50.433 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls": 0, 2024-03-21T10:50:50.433 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls_filter": 0, 2024-03-21T10:50:50.433 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_other": 0, 2024-03-21T10:50:50.433 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_active": 0, 2024-03-21T10:50:50.433 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_send": 0, 2024-03-21T10:50:50.433 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_resend": 0, 2024-03-21T10:50:50.433 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_ping": 0, 2024-03-21T10:50:50.434 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_active": 0, 2024-03-21T10:50:50.434 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_send": 0, 2024-03-21T10:50:50.434 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_resend": 0, 2024-03-21T10:50:50.434 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_active": 0, 2024-03-21T10:50:50.434 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_send": 0, 2024-03-21T10:50:50.434 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_resend": 0, 2024-03-21T10:50:50.434 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_active": 0, 2024-03-21T10:50:50.434 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_send": 0, 2024-03-21T10:50:50.434 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_resend": 0, 2024-03-21T10:50:50.434 INFO:teuthology.orchestra.run.smithi040.stdout: "command_active": 0, 2024-03-21T10:50:50.434 INFO:teuthology.orchestra.run.smithi040.stdout: "command_send": 0, 2024-03-21T10:50:50.434 INFO:teuthology.orchestra.run.smithi040.stdout: "command_resend": 0, 2024-03-21T10:50:50.434 INFO:teuthology.orchestra.run.smithi040.stdout: "map_epoch": 106, 2024-03-21T10:50:50.434 INFO:teuthology.orchestra.run.smithi040.stdout: "map_full": 0, 2024-03-21T10:50:50.434 INFO:teuthology.orchestra.run.smithi040.stdout: "map_inc": 22, 2024-03-21T10:50:50.435 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_sessions": 12, 2024-03-21T10:50:50.435 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_open": 12, 2024-03-21T10:50:50.435 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_close": 0, 2024-03-21T10:50:50.435 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_laggy": 0, 2024-03-21T10:50:50.435 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_wr": 673, 2024-03-21T10:50:50.435 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_rd": 10, 2024-03-21T10:50:50.435 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_del": 15 2024-03-21T10:50:50.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.435 INFO:teuthology.orchestra.run.smithi040.stdout: "oft": { 2024-03-21T10:50:50.435 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_objs": 1, 2024-03-21T10:50:50.435 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_kv_pairs": 0, 2024-03-21T10:50:50.435 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_updates": 0, 2024-03-21T10:50:50.435 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_removes": 0 2024-03-21T10:50:50.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.435 INFO:teuthology.orchestra.run.smithi040.stdout: "osd-slow-ops": { 2024-03-21T10:50:50.436 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_ops_count": 0 2024-03-21T10:50:50.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.436 INFO:teuthology.orchestra.run.smithi040.stdout: "purge_queue": { 2024-03-21T10:50:50.436 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops": 0, 2024-03-21T10:50:50.436 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops_high_water": 0, 2024-03-21T10:50:50.436 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing": 0, 2024-03-21T10:50:50.436 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_high_water": 0, 2024-03-21T10:50:50.436 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed_ops": 0, 2024-03-21T10:50:50.436 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed": 0, 2024-03-21T10:50:50.436 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_item_in_journal": 0 2024-03-21T10:50:50.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.436 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-21T10:50:50.436 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-21T10:50:50.436 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-03-21T10:50:50.436 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-21T10:50:50.437 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 8262, 2024-03-21T10:50:50.437 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 3978047, 2024-03-21T10:50:50.437 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-21T10:50:50.437 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 8262, 2024-03-21T10:50:50.437 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-21T10:50:50.437 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-21T10:50:50.437 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 8262, 2024-03-21T10:50:50.437 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 3978047, 2024-03-21T10:50:50.437 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-21T10:50:50.437 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:50.437 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:50.437 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:50.437 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-21T10:50:50.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.437 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_bytes": { 2024-03-21T10:50:50.437 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-21T10:50:50.438 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-03-21T10:50:50.438 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-21T10:50:50.438 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-21T10:50:50.438 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-21T10:50:50.438 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-21T10:50:50.438 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-21T10:50:50.438 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 2384, 2024-03-21T10:50:50.438 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 13135751, 2024-03-21T10:50:50.438 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 2384, 2024-03-21T10:50:50.438 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 13135751, 2024-03-21T10:50:50.438 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-21T10:50:50.438 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:50.438 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:50.438 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:50.439 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-21T10:50:50.439 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.439 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_ops": { 2024-03-21T10:50:50.439 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-21T10:50:50.439 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 1024, 2024-03-21T10:50:50.439 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-21T10:50:50.439 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-21T10:50:50.439 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-21T10:50:50.439 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-21T10:50:50.439 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-21T10:50:50.439 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 2384, 2024-03-21T10:50:50.439 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 2384, 2024-03-21T10:50:50.439 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 2384, 2024-03-21T10:50:50.439 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 2384, 2024-03-21T10:50:50.439 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-21T10:50:50.439 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:50.440 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:50.440 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:50.440 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-21T10:50:50.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.440 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle": { 2024-03-21T10:50:50.440 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-21T10:50:50.440 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-03-21T10:50:50.440 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-21T10:50:50.440 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-21T10:50:50.440 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-21T10:50:50.440 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-21T10:50:50.440 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-21T10:50:50.440 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-21T10:50:50.440 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-21T10:50:50.441 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 0, 2024-03-21T10:50:50.441 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 0, 2024-03-21T10:50:50.441 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-21T10:50:50.441 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:50.441 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:50.441 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:50.441 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-21T10:50:50.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:50.441 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle-0x5581699c4460": { 2024-03-21T10:50:50.441 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-21T10:50:50.441 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-03-21T10:50:50.441 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-21T10:50:50.441 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 1631, 2024-03-21T10:50:50.441 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 12105213, 2024-03-21T10:50:50.441 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-21T10:50:50.442 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 1631, 2024-03-21T10:50:50.442 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-21T10:50:50.442 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-21T10:50:50.442 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 738, 2024-03-21T10:50:50.442 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 12105213, 2024-03-21T10:50:50.442 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-21T10:50:50.442 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:50.442 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:50.442 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:50.442 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-21T10:50:50.442 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-21T10:50:50.442 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-21T10:50:51.081 INFO:tasks.check_counter:Daemon mds.b mds.exported=2 2024-03-21T10:50:51.081 INFO:tasks.check_counter:Daemon mds.b mds.imported=1 2024-03-21T10:50:51.081 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-03-21T10:50:51.081 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=12 2024-03-21T10:50:51.081 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=6 2024-03-21T10:50:51.081 DEBUG:tasks.check_counter:Getting stats from e 2024-03-21T10:50:51.081 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-21T10:50:51.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:51 smithi031 ceph-mon[31673]: pgmap v383: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-03-21T10:50:51.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:51 smithi070 ceph-mon[36330]: pgmap v383: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-03-21T10:50:51.413 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T10:50:51.429 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:51 smithi040 ceph-mon[36154]: pgmap v383: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-03-21T10:50:53.242 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:53 smithi040 ceph-mon[36154]: pgmap v384: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:50:53.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:53 smithi031 ceph-mon[31673]: pgmap v384: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:50:53.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:53 smithi070 ceph-mon[36330]: pgmap v384: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:50:53.373 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-21T10:50:53.373 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-03-21T10:50:53.373 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 272, 2024-03-21T10:50:53.373 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 260, 2024-03-21T10:50:53.373 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 1732976, 2024-03-21T10:50:53.373 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 156480, 2024-03-21T10:50:53.373 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 4, 2024-03-21T10:50:53.374 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 0, 2024-03-21T10:50:53.374 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.047222026, 2024-03-21T10:50:53.374 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.010974020, 2024-03-21T10:50:53.374 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.041797683, 2024-03-21T10:50:53.374 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.005219798, 2024-03-21T10:50:53.374 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:50:53.374 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 259, 2024-03-21T10:50:53.374 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.008523393, 2024-03-21T10:50:53.374 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000032908 2024-03-21T10:50:53.374 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:53.374 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:50:53.374 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:53.374 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:53.374 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:53.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:53.375 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 1732976, 2024-03-21T10:50:53.375 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 156480 2024-03-21T10:50:53.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:53.375 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-03-21T10:50:53.375 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 2, 2024-03-21T10:50:53.375 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 199, 2024-03-21T10:50:53.375 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 320, 2024-03-21T10:50:53.375 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 190112, 2024-03-21T10:50:53.375 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 5, 2024-03-21T10:50:53.375 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 3, 2024-03-21T10:50:53.375 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.009204635, 2024-03-21T10:50:53.375 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.006162704, 2024-03-21T10:50:53.375 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.001344364, 2024-03-21T10:50:53.376 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-21T10:50:53.376 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:50:53.376 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 196, 2024-03-21T10:50:53.376 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.021725748, 2024-03-21T10:50:53.376 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000110845 2024-03-21T10:50:53.376 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:53.376 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:50:53.376 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:53.376 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:53.376 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:53.376 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:53.376 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-21T10:50:53.376 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 190112 2024-03-21T10:50:53.376 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:53.377 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-03-21T10:50:53.377 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 4, 2024-03-21T10:50:53.377 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 4, 2024-03-21T10:50:53.377 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 2992, 2024-03-21T10:50:53.377 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 448, 2024-03-21T10:50:53.377 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 2, 2024-03-21T10:50:53.377 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 1, 2024-03-21T10:50:53.377 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.000562786, 2024-03-21T10:50:53.377 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.000028377, 2024-03-21T10:50:53.377 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.000500334, 2024-03-21T10:50:53.377 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-21T10:50:53.377 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:50:53.377 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 3, 2024-03-21T10:50:53.377 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000735761, 2024-03-21T10:50:53.378 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000245253 2024-03-21T10:50:53.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:53.378 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:50:53.378 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:53.378 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:53.378 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:53.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:53.378 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-21T10:50:53.378 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 448 2024-03-21T10:50:53.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:53.378 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-03-21T10:50:53.378 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 0, 2024-03-21T10:50:53.378 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-03-21T10:50:53.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:53.379 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-03-21T10:50:53.379 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-03-21T10:50:53.379 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-03-21T10:50:53.379 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-03-21T10:50:53.379 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-03-21T10:50:53.379 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-03-21T10:50:53.379 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-03-21T10:50:53.379 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-21T10:50:53.379 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-03-21T10:50:53.379 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-21T10:50:53.380 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-03-21T10:50:53.380 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-03-21T10:50:53.380 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-03-21T10:50:53.380 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-21T10:50:53.380 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-03-21T10:50:53.380 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-03-21T10:50:53.380 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-03-21T10:50:53.380 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-03-21T10:50:53.380 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-03-21T10:50:53.380 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-21T10:50:53.380 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-03-21T10:50:53.381 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-21T10:50:53.381 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-03-21T10:50:53.381 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-03-21T10:50:53.381 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-03-21T10:50:53.381 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-03-21T10:50:53.381 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-03-21T10:50:53.381 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-21T10:50:53.381 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-03-21T10:50:53.381 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-03-21T10:50:53.381 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-03-21T10:50:53.382 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-03-21T10:50:53.382 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-03-21T10:50:53.382 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-03-21T10:50:53.382 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-03-21T10:50:53.382 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-03-21T10:50:53.382 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-03-21T10:50:53.382 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 148692, 2024-03-21T10:50:53.382 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 30, 2024-03-21T10:50:53.382 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2024-03-21T10:50:53.383 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2024-03-21T10:50:53.383 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-03-21T10:50:53.383 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-03-21T10:50:53.383 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-03-21T10:50:53.383 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-03-21T10:50:53.383 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-03-21T10:50:53.383 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-03-21T10:50:53.383 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 0, 2024-03-21T10:50:53.383 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 0, 2024-03-21T10:50:53.383 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-03-21T10:50:53.383 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-03-21T10:50:53.383 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-03-21T10:50:53.383 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-03-21T10:50:53.383 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 928, 2024-03-21T10:50:53.384 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 2, 2024-03-21T10:50:53.384 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-03-21T10:50:53.384 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-03-21T10:50:53.384 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-03-21T10:50:53.384 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-03-21T10:50:53.384 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:53.384 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-21T10:50:53.384 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-21T10:50:53.384 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-03-21T10:50:53.384 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-21T10:50:53.384 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 274, 2024-03-21T10:50:53.384 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 1694599, 2024-03-21T10:50:53.384 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-21T10:50:53.384 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 274, 2024-03-21T10:50:53.385 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-21T10:50:53.385 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-21T10:50:53.385 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 274, 2024-03-21T10:50:53.385 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 1694599, 2024-03-21T10:50:53.385 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-21T10:50:53.385 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:53.385 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:53.385 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:53.385 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-21T10:50:53.385 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-21T10:50:53.385 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-21T10:50:54.117 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.e 2024-03-21T10:50:54.117 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.e 2024-03-21T10:50:54.118 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.e 2024-03-21T10:50:54.118 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.e 2024-03-21T10:50:54.118 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.e 2024-03-21T10:50:54.118 DEBUG:tasks.check_counter:Getting stats from h 2024-03-21T10:50:54.118 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-21T10:50:54.385 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T10:50:55.130 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:55 smithi040 ceph-mon[36154]: pgmap v385: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:50:55.303 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:55 smithi031 ceph-mon[31673]: pgmap v385: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:50:55.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:55 smithi070 ceph-mon[36330]: pgmap v385: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:50:56.029 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-21T10:50:56.030 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-03-21T10:50:56.030 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 1473, 2024-03-21T10:50:56.030 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 1473, 2024-03-21T10:50:56.030 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 4648487, 2024-03-21T10:50:56.030 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 467600, 2024-03-21T10:50:56.030 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 6, 2024-03-21T10:50:56.030 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 2, 2024-03-21T10:50:56.030 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.111393125, 2024-03-21T10:50:56.030 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.037362219, 2024-03-21T10:50:56.031 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.029777991, 2024-03-21T10:50:56.031 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.034427638, 2024-03-21T10:50:56.031 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:50:56.031 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1472, 2024-03-21T10:50:56.031 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.041625172, 2024-03-21T10:50:56.031 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000028277 2024-03-21T10:50:56.031 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.031 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:50:56.031 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.031 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.031 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.031 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.031 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-21T10:50:56.032 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 448 2024-03-21T10:50:56.032 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.032 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-03-21T10:50:56.032 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 639, 2024-03-21T10:50:56.032 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 618, 2024-03-21T10:50:56.032 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 1880736, 2024-03-21T10:50:56.032 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 266754, 2024-03-21T10:50:56.032 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 5, 2024-03-21T10:50:56.032 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 3, 2024-03-21T10:50:56.032 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.067166258, 2024-03-21T10:50:56.032 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.017339378, 2024-03-21T10:50:56.032 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.044733468, 2024-03-21T10:50:56.032 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.015120872, 2024-03-21T10:50:56.032 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:50:56.033 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 617, 2024-03-21T10:50:56.033 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.018095744, 2024-03-21T10:50:56.033 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000029328 2024-03-21T10:50:56.033 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.033 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:50:56.033 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.033 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.033 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.033 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.033 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 1797408, 2024-03-21T10:50:56.033 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 159840 2024-03-21T10:50:56.034 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.034 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-03-21T10:50:56.034 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 1138, 2024-03-21T10:50:56.034 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 1334, 2024-03-21T10:50:56.034 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 8217094, 2024-03-21T10:50:56.034 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 860593, 2024-03-21T10:50:56.034 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 7, 2024-03-21T10:50:56.034 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 5, 2024-03-21T10:50:56.034 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.086291512, 2024-03-21T10:50:56.035 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.025832478, 2024-03-21T10:50:56.035 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.025485017, 2024-03-21T10:50:56.035 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.026840083, 2024-03-21T10:50:56.035 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:50:56.035 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1331, 2024-03-21T10:50:56.035 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.056485348, 2024-03-21T10:50:56.035 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000042438 2024-03-21T10:50:56.035 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.036 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:50:56.036 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.036 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.036 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.036 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.037 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-21T10:50:56.037 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 499344 2024-03-21T10:50:56.037 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.037 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-03-21T10:50:56.037 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 1, 2024-03-21T10:50:56.037 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-03-21T10:50:56.038 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.038 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-MDSRank": { 2024-03-21T10:50:56.038 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-03-21T10:50:56.038 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-03-21T10:50:56.039 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 4927, 2024-03-21T10:50:56.039 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.110920101, 2024-03-21T10:50:56.039 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000022512 2024-03-21T10:50:56.039 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-21T10:50:56.039 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.039 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-PurgeQueue": { 2024-03-21T10:50:56.040 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-03-21T10:50:56.040 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-03-21T10:50:56.040 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.040 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.040 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.040 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-21T10:50:56.041 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.041 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": { 2024-03-21T10:50:56.041 INFO:teuthology.orchestra.run.smithi040.stdout: "request": 0, 2024-03-21T10:50:56.041 INFO:teuthology.orchestra.run.smithi040.stdout: "reply": 0, 2024-03-21T10:50:56.041 INFO:teuthology.orchestra.run.smithi040.stdout: "reply_latency": { 2024-03-21T10:50:56.041 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.042 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.042 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.042 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.042 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_reply": 0, 2024-03-21T10:50:56.042 INFO:teuthology.orchestra.run.smithi040.stdout: "forward": 0, 2024-03-21T10:50:56.043 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_complete": 0, 2024-03-21T10:50:56.043 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_keys": 0, 2024-03-21T10:50:56.043 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_commit": 0, 2024-03-21T10:50:56.043 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_split": 0, 2024-03-21T10:50:56.043 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_merge": 0, 2024-03-21T10:50:56.043 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes": 14, 2024-03-21T10:50:56.044 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_top": 5, 2024-03-21T10:50:56.044 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_bottom": 9, 2024-03-21T10:50:56.044 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pin_tail": 0, 2024-03-21T10:50:56.044 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pinned": 7, 2024-03-21T10:50:56.044 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_expired": 0, 2024-03-21T10:50:56.044 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_with_caps": 0, 2024-03-21T10:50:56.045 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": 0, 2024-03-21T10:50:56.045 INFO:teuthology.orchestra.run.smithi040.stdout: "subtrees": 2, 2024-03-21T10:50:56.045 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse": 0, 2024-03-21T10:50:56.045 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_hit": 0, 2024-03-21T10:50:56.045 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_forward": 0, 2024-03-21T10:50:56.045 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_discover": 0, 2024-03-21T10:50:56.045 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_dir_fetch": 0, 2024-03-21T10:50:56.046 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_remote_ino": 0, 2024-03-21T10:50:56.046 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_lock": 0, 2024-03-21T10:50:56.046 INFO:teuthology.orchestra.run.smithi040.stdout: "load_cent": 0, 2024-03-21T10:50:56.046 INFO:teuthology.orchestra.run.smithi040.stdout: "q": 0, 2024-03-21T10:50:56.046 INFO:teuthology.orchestra.run.smithi040.stdout: "exported": 0, 2024-03-21T10:50:56.046 INFO:teuthology.orchestra.run.smithi040.stdout: "exported_inodes": 0, 2024-03-21T10:50:56.046 INFO:teuthology.orchestra.run.smithi040.stdout: "imported": 0, 2024-03-21T10:50:56.047 INFO:teuthology.orchestra.run.smithi040.stdout: "imported_inodes": 0, 2024-03-21T10:50:56.047 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_dir_fetch": 0, 2024-03-21T10:50:56.047 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_backtrace_fetch": 0, 2024-03-21T10:50:56.047 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_peer_discover": 0, 2024-03-21T10:50:56.047 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rfiles": 4, 2024-03-21T10:50:56.047 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rbytes": 234, 2024-03-21T10:50:56.047 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rsnaps": 6, 2024-03-21T10:50:56.048 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_fetch": 0, 2024-03-21T10:50:56.048 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_set_tag": 0, 2024-03-21T10:50:56.048 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_repaired": 0, 2024-03-21T10:50:56.048 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_inotable_repaired": 0, 2024-03-21T10:50:56.048 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_inodes": 0, 2024-03-21T10:50:56.048 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_base_inodes": 0, 2024-03-21T10:50:56.048 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dirfrag_rstats": 0, 2024-03-21T10:50:56.048 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_file_inodes": 0, 2024-03-21T10:50:56.049 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_inode_file_caps": 0, 2024-03-21T10:50:56.049 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_revoke": 0, 2024-03-21T10:50:56.049 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_grant": 0, 2024-03-21T10:50:56.049 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_trunc": 0, 2024-03-21T10:50:56.049 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-21T10:50:56.049 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-21T10:50:56.049 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps": 0, 2024-03-21T10:50:56.049 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps_dirty": 0, 2024-03-21T10:50:56.050 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_cap_release": 0, 2024-03-21T10:50:56.050 INFO:teuthology.orchestra.run.smithi040.stdout: "process_request_cap_release": 0 2024-03-21T10:50:56.050 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.050 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache": { 2024-03-21T10:50:56.050 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update": 0, 2024-03-21T10:50:56.050 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update_receipt": 0, 2024-03-21T10:50:56.051 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_try_discover": 0, 2024-03-21T10:50:56.051 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_send_discover": 0, 2024-03-21T10:50:56.051 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_handle_discover": 0, 2024-03-21T10:50:56.051 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays": 0, 2024-03-21T10:50:56.051 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_delayed": 0, 2024-03-21T10:50:56.051 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_enqueuing": 0, 2024-03-21T10:50:56.052 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_created": 0, 2024-03-21T10:50:56.052 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_enqueued": 0, 2024-03-21T10:50:56.052 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_reintegrated": 0, 2024-03-21T10:50:56.052 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_migrated": 0, 2024-03-21T10:50:56.052 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_processing": 0, 2024-03-21T10:50:56.052 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_enqueued": 0, 2024-03-21T10:50:56.053 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_prioritized": 0, 2024-03-21T10:50:56.053 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_started": 0, 2024-03-21T10:50:56.053 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_completed": 0, 2024-03-21T10:50:56.053 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_enqueue_scrub": 0, 2024-03-21T10:50:56.053 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_exportdir": 0, 2024-03-21T10:50:56.053 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_flush": 0, 2024-03-21T10:50:56.053 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragmentdir": 0, 2024-03-21T10:50:56.053 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragstats": 0, 2024-03-21T10:50:56.053 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_inodestats": 0 2024-03-21T10:50:56.054 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.054 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log": { 2024-03-21T10:50:56.054 INFO:teuthology.orchestra.run.smithi040.stdout: "evlrg": 0, 2024-03-21T10:50:56.054 INFO:teuthology.orchestra.run.smithi040.stdout: "evadd": 0, 2024-03-21T10:50:56.054 INFO:teuthology.orchestra.run.smithi040.stdout: "evex": 0, 2024-03-21T10:50:56.054 INFO:teuthology.orchestra.run.smithi040.stdout: "evtrm": 0, 2024-03-21T10:50:56.054 INFO:teuthology.orchestra.run.smithi040.stdout: "ev": 1631, 2024-03-21T10:50:56.054 INFO:teuthology.orchestra.run.smithi040.stdout: "evexg": 0, 2024-03-21T10:50:56.055 INFO:teuthology.orchestra.run.smithi040.stdout: "evexd": 0, 2024-03-21T10:50:56.055 INFO:teuthology.orchestra.run.smithi040.stdout: "segadd": 0, 2024-03-21T10:50:56.055 INFO:teuthology.orchestra.run.smithi040.stdout: "segex": 0, 2024-03-21T10:50:56.055 INFO:teuthology.orchestra.run.smithi040.stdout: "segtrm": 0, 2024-03-21T10:50:56.055 INFO:teuthology.orchestra.run.smithi040.stdout: "seg": 2, 2024-03-21T10:50:56.055 INFO:teuthology.orchestra.run.smithi040.stdout: "segmjr": 54, 2024-03-21T10:50:56.055 INFO:teuthology.orchestra.run.smithi040.stdout: "segexg": 0, 2024-03-21T10:50:56.055 INFO:teuthology.orchestra.run.smithi040.stdout: "segexd": 0, 2024-03-21T10:50:56.055 INFO:teuthology.orchestra.run.smithi040.stdout: "expos": 16221209, 2024-03-21T10:50:56.056 INFO:teuthology.orchestra.run.smithi040.stdout: "wrpos": 16299517, 2024-03-21T10:50:56.056 INFO:teuthology.orchestra.run.smithi040.stdout: "rdpos": 16287187, 2024-03-21T10:50:56.056 INFO:teuthology.orchestra.run.smithi040.stdout: "jlat": { 2024-03-21T10:50:56.056 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.056 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.056 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.056 INFO:teuthology.orchestra.run.smithi040.stdout: "replayed": 1631 2024-03-21T10:50:56.057 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.057 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_mem": { 2024-03-21T10:50:56.057 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 13, 2024-03-21T10:50:56.057 INFO:teuthology.orchestra.run.smithi040.stdout: "ino+": 90, 2024-03-21T10:50:56.057 INFO:teuthology.orchestra.run.smithi040.stdout: "ino-": 77, 2024-03-21T10:50:56.057 INFO:teuthology.orchestra.run.smithi040.stdout: "dir": 8, 2024-03-21T10:50:56.057 INFO:teuthology.orchestra.run.smithi040.stdout: "dir+": 12, 2024-03-21T10:50:56.058 INFO:teuthology.orchestra.run.smithi040.stdout: "dir-": 4, 2024-03-21T10:50:56.058 INFO:teuthology.orchestra.run.smithi040.stdout: "dn": 14, 2024-03-21T10:50:56.058 INFO:teuthology.orchestra.run.smithi040.stdout: "dn+": 91, 2024-03-21T10:50:56.058 INFO:teuthology.orchestra.run.smithi040.stdout: "dn-": 77, 2024-03-21T10:50:56.058 INFO:teuthology.orchestra.run.smithi040.stdout: "cap": 0, 2024-03-21T10:50:56.058 INFO:teuthology.orchestra.run.smithi040.stdout: "cap+": 0, 2024-03-21T10:50:56.059 INFO:teuthology.orchestra.run.smithi040.stdout: "cap-": 0, 2024-03-21T10:50:56.059 INFO:teuthology.orchestra.run.smithi040.stdout: "rss": 48040, 2024-03-21T10:50:56.059 INFO:teuthology.orchestra.run.smithi040.stdout: "heap": 215420 2024-03-21T10:50:56.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.059 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_server": { 2024-03-21T10:50:56.059 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_client_request": 0, 2024-03-21T10:50:56.060 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_server_request": 0, 2024-03-21T10:50:56.061 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_request": 0, 2024-03-21T10:50:56.061 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_session": 0, 2024-03-21T10:50:56.062 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_peer_request": 0, 2024-03-21T10:50:56.062 INFO:teuthology.orchestra.run.smithi040.stdout: "req_create_latency": { 2024-03-21T10:50:56.062 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.062 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.062 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.063 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getattr_latency": { 2024-03-21T10:50:56.063 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.063 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.063 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.063 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getfilelock_latency": { 2024-03-21T10:50:56.063 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.063 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.063 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.064 INFO:teuthology.orchestra.run.smithi040.stdout: "req_link_latency": { 2024-03-21T10:50:56.064 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.064 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.064 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.064 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookup_latency": { 2024-03-21T10:50:56.064 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.064 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.064 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.064 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookuphash_latency": { 2024-03-21T10:50:56.064 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.064 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.065 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.065 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupino_latency": { 2024-03-21T10:50:56.065 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.065 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.065 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.065 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupname_latency": { 2024-03-21T10:50:56.065 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.065 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.065 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.065 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupparent_latency": { 2024-03-21T10:50:56.065 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.066 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.066 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.066 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.066 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupsnap_latency": { 2024-03-21T10:50:56.066 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.066 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.066 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.066 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.066 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lssnap_latency": { 2024-03-21T10:50:56.066 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.066 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.066 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.067 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.067 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mkdir_latency": { 2024-03-21T10:50:56.067 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.067 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.067 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.068 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mknod_latency": { 2024-03-21T10:50:56.068 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.068 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.068 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.068 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mksnap_latency": { 2024-03-21T10:50:56.068 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.068 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.068 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.068 INFO:teuthology.orchestra.run.smithi040.stdout: "req_open_latency": { 2024-03-21T10:50:56.068 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.068 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.069 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.069 INFO:teuthology.orchestra.run.smithi040.stdout: "req_readdir_latency": { 2024-03-21T10:50:56.069 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.069 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.069 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.069 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rename_latency": { 2024-03-21T10:50:56.069 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.069 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.069 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.069 INFO:teuthology.orchestra.run.smithi040.stdout: "req_renamesnap_latency": { 2024-03-21T10:50:56.069 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.070 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.070 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.070 INFO:teuthology.orchestra.run.smithi040.stdout: "req_snapdiff_latency": { 2024-03-21T10:50:56.070 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.070 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.070 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.070 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmdir_latency": { 2024-03-21T10:50:56.070 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.071 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.071 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.071 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmsnap_latency": { 2024-03-21T10:50:56.071 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.071 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.071 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.071 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmxattr_latency": { 2024-03-21T10:50:56.071 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.071 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.071 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.071 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setattr_latency": { 2024-03-21T10:50:56.072 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.072 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.072 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.072 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setdirlayout_latency": { 2024-03-21T10:50:56.072 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.072 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.072 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.072 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setfilelock_latency": { 2024-03-21T10:50:56.072 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.072 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.072 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.073 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setlayout_latency": { 2024-03-21T10:50:56.073 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.073 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.073 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.073 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setxattr_latency": { 2024-03-21T10:50:56.073 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.074 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.074 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.074 INFO:teuthology.orchestra.run.smithi040.stdout: "req_symlink_latency": { 2024-03-21T10:50:56.074 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.074 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.074 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.075 INFO:teuthology.orchestra.run.smithi040.stdout: "req_unlink_latency": { 2024-03-21T10:50:56.075 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.075 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.075 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.075 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_revoke_eviction": 0, 2024-03-21T10:50:56.075 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_acquisition_throttle": 0, 2024-03-21T10:50:56.075 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getvxattr_latency": { 2024-03-21T10:50:56.075 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.075 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.075 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.075 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-21T10:50:56.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.076 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_sessions": { 2024-03-21T10:50:56.076 INFO:teuthology.orchestra.run.smithi040.stdout: "session_count": 2, 2024-03-21T10:50:56.076 INFO:teuthology.orchestra.run.smithi040.stdout: "session_add": 6, 2024-03-21T10:50:56.076 INFO:teuthology.orchestra.run.smithi040.stdout: "session_remove": 4, 2024-03-21T10:50:56.076 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_open": 4, 2024-03-21T10:50:56.076 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_stale": 0, 2024-03-21T10:50:56.076 INFO:teuthology.orchestra.run.smithi040.stdout: "total_load": 0, 2024-03-21T10:50:56.076 INFO:teuthology.orchestra.run.smithi040.stdout: "average_load": 0, 2024-03-21T10:50:56.076 INFO:teuthology.orchestra.run.smithi040.stdout: "avg_session_uptime": 4070, 2024-03-21T10:50:56.076 INFO:teuthology.orchestra.run.smithi040.stdout: "mdthresh_evicted": 0 2024-03-21T10:50:56.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.076 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-03-21T10:50:56.076 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-03-21T10:50:56.076 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-03-21T10:50:56.077 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-03-21T10:50:56.077 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-03-21T10:50:56.077 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-03-21T10:50:56.077 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-03-21T10:50:56.077 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-21T10:50:56.077 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-03-21T10:50:56.077 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-21T10:50:56.077 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-03-21T10:50:56.077 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-03-21T10:50:56.077 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-03-21T10:50:56.077 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-21T10:50:56.077 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-03-21T10:50:56.077 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-03-21T10:50:56.077 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-03-21T10:50:56.078 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-03-21T10:50:56.078 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-03-21T10:50:56.078 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-21T10:50:56.078 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-03-21T10:50:56.078 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-21T10:50:56.078 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-03-21T10:50:56.078 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-03-21T10:50:56.078 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-03-21T10:50:56.078 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-03-21T10:50:56.078 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-03-21T10:50:56.078 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-21T10:50:56.078 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-03-21T10:50:56.078 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-03-21T10:50:56.079 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-03-21T10:50:56.079 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-03-21T10:50:56.079 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-03-21T10:50:56.079 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-03-21T10:50:56.079 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-03-21T10:50:56.079 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-03-21T10:50:56.079 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-03-21T10:50:56.079 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 199043, 2024-03-21T10:50:56.079 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 59, 2024-03-21T10:50:56.079 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2024-03-21T10:50:56.079 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2024-03-21T10:50:56.079 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-03-21T10:50:56.079 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-03-21T10:50:56.079 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-03-21T10:50:56.080 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-03-21T10:50:56.080 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-03-21T10:50:56.080 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-03-21T10:50:56.080 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 9328, 2024-03-21T10:50:56.080 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 179, 2024-03-21T10:50:56.080 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-03-21T10:50:56.080 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-03-21T10:50:56.080 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-03-21T10:50:56.080 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-03-21T10:50:56.080 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 73530, 2024-03-21T10:50:56.080 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 272, 2024-03-21T10:50:56.080 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-03-21T10:50:56.080 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-03-21T10:50:56.081 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-03-21T10:50:56.081 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-03-21T10:50:56.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.081 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter": { 2024-03-21T10:50:56.081 INFO:teuthology.orchestra.run.smithi040.stdout: "op_active": 0, 2024-03-21T10:50:56.081 INFO:teuthology.orchestra.run.smithi040.stdout: "op_laggy": 0, 2024-03-21T10:50:56.081 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send": 2941, 2024-03-21T10:50:56.081 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send_bytes": 26, 2024-03-21T10:50:56.081 INFO:teuthology.orchestra.run.smithi040.stdout: "op_resend": 4, 2024-03-21T10:50:56.081 INFO:teuthology.orchestra.run.smithi040.stdout: "op_reply": 2941, 2024-03-21T10:50:56.081 INFO:teuthology.orchestra.run.smithi040.stdout: "op_latency": { 2024-03-21T10:50:56.081 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2941, 2024-03-21T10:50:56.081 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 5.112941834, 2024-03-21T10:50:56.081 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.001738504 2024-03-21T10:50:56.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.082 INFO:teuthology.orchestra.run.smithi040.stdout: "op_inflight": 0, 2024-03-21T10:50:56.082 INFO:teuthology.orchestra.run.smithi040.stdout: "oplen_avg": { 2024-03-21T10:50:56.082 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2941, 2024-03-21T10:50:56.082 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 2942 2024-03-21T10:50:56.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.082 INFO:teuthology.orchestra.run.smithi040.stdout: "op": 2941, 2024-03-21T10:50:56.082 INFO:teuthology.orchestra.run.smithi040.stdout: "op_r": 2941, 2024-03-21T10:50:56.082 INFO:teuthology.orchestra.run.smithi040.stdout: "op_w": 0, 2024-03-21T10:50:56.082 INFO:teuthology.orchestra.run.smithi040.stdout: "op_rmw": 0, 2024-03-21T10:50:56.082 INFO:teuthology.orchestra.run.smithi040.stdout: "op_pg": 0, 2024-03-21T10:50:56.082 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_stat": 1820, 2024-03-21T10:50:56.082 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_create": 0, 2024-03-21T10:50:56.082 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_read": 1118, 2024-03-21T10:50:56.082 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_write": 0, 2024-03-21T10:50:56.082 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writefull": 0, 2024-03-21T10:50:56.083 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writesame": 0, 2024-03-21T10:50:56.083 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_append": 0, 2024-03-21T10:50:56.083 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_zero": 0, 2024-03-21T10:50:56.083 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_truncate": 0, 2024-03-21T10:50:56.083 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_delete": 0, 2024-03-21T10:50:56.083 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_mapext": 0, 2024-03-21T10:50:56.083 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_sparse_read": 0, 2024-03-21T10:50:56.083 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_clonerange": 0, 2024-03-21T10:50:56.083 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_getxattr": 2, 2024-03-21T10:50:56.083 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_setxattr": 0, 2024-03-21T10:50:56.083 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_cmpxattr": 0, 2024-03-21T10:50:56.083 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_rmxattr": 0, 2024-03-21T10:50:56.083 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_resetxattrs": 0, 2024-03-21T10:50:56.083 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_call": 0, 2024-03-21T10:50:56.083 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_watch": 0, 2024-03-21T10:50:56.084 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_notify": 0, 2024-03-21T10:50:56.084 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_src_cmpxattr": 0, 2024-03-21T10:50:56.084 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls": 0, 2024-03-21T10:50:56.084 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls_filter": 0, 2024-03-21T10:50:56.084 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_other": 0, 2024-03-21T10:50:56.084 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_active": 0, 2024-03-21T10:50:56.084 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_send": 0, 2024-03-21T10:50:56.084 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_resend": 0, 2024-03-21T10:50:56.084 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_ping": 0, 2024-03-21T10:50:56.084 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_active": 0, 2024-03-21T10:50:56.084 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_send": 0, 2024-03-21T10:50:56.084 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_resend": 0, 2024-03-21T10:50:56.084 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_active": 0, 2024-03-21T10:50:56.084 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_send": 0, 2024-03-21T10:50:56.084 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_resend": 0, 2024-03-21T10:50:56.085 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_active": 0, 2024-03-21T10:50:56.085 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_send": 0, 2024-03-21T10:50:56.085 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_resend": 0, 2024-03-21T10:50:56.085 INFO:teuthology.orchestra.run.smithi040.stdout: "command_active": 0, 2024-03-21T10:50:56.085 INFO:teuthology.orchestra.run.smithi040.stdout: "command_send": 0, 2024-03-21T10:50:56.085 INFO:teuthology.orchestra.run.smithi040.stdout: "command_resend": 0, 2024-03-21T10:50:56.085 INFO:teuthology.orchestra.run.smithi040.stdout: "map_epoch": 106, 2024-03-21T10:50:56.085 INFO:teuthology.orchestra.run.smithi040.stdout: "map_full": 0, 2024-03-21T10:50:56.085 INFO:teuthology.orchestra.run.smithi040.stdout: "map_inc": 17, 2024-03-21T10:50:56.085 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_sessions": 6, 2024-03-21T10:50:56.085 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_open": 7, 2024-03-21T10:50:56.085 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_close": 1, 2024-03-21T10:50:56.085 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_laggy": 0, 2024-03-21T10:50:56.085 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_wr": 0, 2024-03-21T10:50:56.086 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_rd": 2, 2024-03-21T10:50:56.086 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_del": 0 2024-03-21T10:50:56.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.086 INFO:teuthology.orchestra.run.smithi040.stdout: "oft": { 2024-03-21T10:50:56.086 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_objs": 0, 2024-03-21T10:50:56.086 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_kv_pairs": 0, 2024-03-21T10:50:56.086 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_updates": 0, 2024-03-21T10:50:56.086 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_removes": 0 2024-03-21T10:50:56.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.086 INFO:teuthology.orchestra.run.smithi040.stdout: "osd-slow-ops": { 2024-03-21T10:50:56.086 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_ops_count": 0 2024-03-21T10:50:56.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.086 INFO:teuthology.orchestra.run.smithi040.stdout: "purge_queue": { 2024-03-21T10:50:56.086 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops": 0, 2024-03-21T10:50:56.086 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops_high_water": 0, 2024-03-21T10:50:56.087 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing": 0, 2024-03-21T10:50:56.087 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_high_water": 0, 2024-03-21T10:50:56.087 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed_ops": 0, 2024-03-21T10:50:56.087 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed": 0, 2024-03-21T10:50:56.087 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_item_in_journal": 0 2024-03-21T10:50:56.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.087 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-21T10:50:56.087 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-21T10:50:56.087 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-03-21T10:50:56.087 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-21T10:50:56.087 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 3246, 2024-03-21T10:50:56.087 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 14435724, 2024-03-21T10:50:56.087 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-21T10:50:56.087 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 3246, 2024-03-21T10:50:56.087 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-21T10:50:56.088 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-21T10:50:56.088 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 3246, 2024-03-21T10:50:56.088 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 14435724, 2024-03-21T10:50:56.088 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-21T10:50:56.088 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.088 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.088 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.088 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-21T10:50:56.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.088 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_bytes": { 2024-03-21T10:50:56.088 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-21T10:50:56.088 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-03-21T10:50:56.088 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-21T10:50:56.088 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-21T10:50:56.088 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-21T10:50:56.089 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-21T10:50:56.089 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-21T10:50:56.089 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 2941, 2024-03-21T10:50:56.089 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 12124073, 2024-03-21T10:50:56.089 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 2941, 2024-03-21T10:50:56.089 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 12124073, 2024-03-21T10:50:56.089 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-21T10:50:56.089 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.089 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.089 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.089 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-21T10:50:56.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.089 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_ops": { 2024-03-21T10:50:56.089 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-21T10:50:56.089 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 1024, 2024-03-21T10:50:56.090 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-21T10:50:56.090 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-21T10:50:56.090 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-21T10:50:56.090 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-21T10:50:56.090 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-21T10:50:56.090 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 2941, 2024-03-21T10:50:56.090 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 2941, 2024-03-21T10:50:56.090 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 2941, 2024-03-21T10:50:56.090 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 2941, 2024-03-21T10:50:56.090 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-21T10:50:56.090 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.090 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.090 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.090 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-21T10:50:56.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.091 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle": { 2024-03-21T10:50:56.091 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-21T10:50:56.091 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-03-21T10:50:56.091 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-21T10:50:56.091 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-21T10:50:56.091 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-21T10:50:56.091 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-21T10:50:56.091 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-21T10:50:56.091 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-21T10:50:56.091 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-21T10:50:56.091 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 0, 2024-03-21T10:50:56.091 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 0, 2024-03-21T10:50:56.091 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-21T10:50:56.091 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.092 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.092 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.092 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-21T10:50:56.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:56.092 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle-0x560e93c586e0": { 2024-03-21T10:50:56.092 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-21T10:50:56.092 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-03-21T10:50:56.092 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-21T10:50:56.092 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-21T10:50:56.092 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-21T10:50:56.092 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-21T10:50:56.092 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-21T10:50:56.092 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-21T10:50:56.092 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-21T10:50:56.092 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 0, 2024-03-21T10:50:56.093 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 0, 2024-03-21T10:50:56.093 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-21T10:50:56.093 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:56.093 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:56.093 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:56.093 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-21T10:50:56.093 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-21T10:50:56.093 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-21T10:50:56.750 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-21T10:50:56.750 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-03-21T10:50:56.750 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-03-21T10:50:56.750 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-03-21T10:50:56.750 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-03-21T10:50:56.750 DEBUG:tasks.check_counter:Getting stats from k 2024-03-21T10:50:56.750 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-21T10:50:57.030 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T10:50:57.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:57 smithi031 ceph-mon[31673]: pgmap v386: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:50:57.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:57 smithi070 ceph-mon[36330]: pgmap v386: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:50:57.355 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:57 smithi040 ceph-mon[36154]: pgmap v386: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:50:58.842 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-21T10:50:58.842 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-03-21T10:50:58.842 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 4, 2024-03-21T10:50:58.842 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 4, 2024-03-21T10:50:58.842 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 2992, 2024-03-21T10:50:58.842 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 448, 2024-03-21T10:50:58.843 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 4, 2024-03-21T10:50:58.843 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 0, 2024-03-21T10:50:58.843 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.001737109, 2024-03-21T10:50:58.843 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.000036890, 2024-03-21T10:50:58.843 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.000738006, 2024-03-21T10:50:58.843 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-21T10:50:58.843 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:50:58.843 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 3, 2024-03-21T10:50:58.843 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000794648, 2024-03-21T10:50:58.843 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000264882 2024-03-21T10:50:58.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:58.843 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:50:58.843 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:58.843 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:58.843 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:58.844 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:58.844 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-21T10:50:58.844 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 448 2024-03-21T10:50:58.844 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:58.844 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-03-21T10:50:58.844 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 3, 2024-03-21T10:50:58.844 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 204, 2024-03-21T10:50:58.844 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 480, 2024-03-21T10:50:58.844 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 268800, 2024-03-21T10:50:58.844 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 6, 2024-03-21T10:50:58.844 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 3, 2024-03-21T10:50:58.844 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.009067652, 2024-03-21T10:50:58.844 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.005837434, 2024-03-21T10:50:58.844 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.001504112, 2024-03-21T10:50:58.845 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-21T10:50:58.845 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:50:58.845 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 201, 2024-03-21T10:50:58.845 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.030296678, 2024-03-21T10:50:58.845 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000150729 2024-03-21T10:50:58.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:58.845 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:50:58.845 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:58.845 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:58.845 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:58.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:58.845 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 480, 2024-03-21T10:50:58.845 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 268800 2024-03-21T10:50:58.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:58.845 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-03-21T10:50:58.846 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 278, 2024-03-21T10:50:58.846 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 266, 2024-03-21T10:50:58.846 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 1734704, 2024-03-21T10:50:58.846 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 159904, 2024-03-21T10:50:58.846 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 2, 2024-03-21T10:50:58.846 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 1, 2024-03-21T10:50:58.846 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.044218782, 2024-03-21T10:50:58.846 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.010784274, 2024-03-21T10:50:58.846 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.039486101, 2024-03-21T10:50:58.846 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.005110404, 2024-03-21T10:50:58.846 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:50:58.846 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 265, 2024-03-21T10:50:58.846 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.008356643, 2024-03-21T10:50:58.846 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000031534 2024-03-21T10:50:58.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:58.847 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:50:58.847 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:58.847 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:58.847 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:58.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:58.847 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 1734704, 2024-03-21T10:50:58.847 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 159904 2024-03-21T10:50:58.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:58.847 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-03-21T10:50:58.847 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 0, 2024-03-21T10:50:58.847 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-03-21T10:50:58.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:58.847 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-03-21T10:50:58.847 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-03-21T10:50:58.848 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-03-21T10:50:58.848 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-03-21T10:50:58.848 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-03-21T10:50:58.848 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-03-21T10:50:58.848 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-03-21T10:50:58.848 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-21T10:50:58.848 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-03-21T10:50:58.848 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-21T10:50:58.848 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-03-21T10:50:58.848 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-03-21T10:50:58.848 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-03-21T10:50:58.848 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-21T10:50:58.848 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-03-21T10:50:58.848 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-03-21T10:50:58.849 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-03-21T10:50:58.849 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-03-21T10:50:58.849 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-03-21T10:50:58.849 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-21T10:50:58.849 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-03-21T10:50:58.849 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-21T10:50:58.849 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-03-21T10:50:58.849 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-03-21T10:50:58.849 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-03-21T10:50:58.849 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-03-21T10:50:58.849 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-03-21T10:50:58.849 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-21T10:50:58.849 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-03-21T10:50:58.849 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-03-21T10:50:58.849 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-03-21T10:50:58.850 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-03-21T10:50:58.850 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-03-21T10:50:58.850 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-03-21T10:50:58.850 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-03-21T10:50:58.850 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-03-21T10:50:58.850 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-03-21T10:50:58.850 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 148771, 2024-03-21T10:50:58.850 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 32, 2024-03-21T10:50:58.850 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2024-03-21T10:50:58.850 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2024-03-21T10:50:58.850 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-03-21T10:50:58.850 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-03-21T10:50:58.850 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-03-21T10:50:58.850 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-03-21T10:50:58.851 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-03-21T10:50:58.851 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-03-21T10:50:58.851 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 0, 2024-03-21T10:50:58.851 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 0, 2024-03-21T10:50:58.851 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-03-21T10:50:58.851 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-03-21T10:50:58.851 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-03-21T10:50:58.851 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-03-21T10:50:58.851 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 928, 2024-03-21T10:50:58.852 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 2, 2024-03-21T10:50:58.852 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-03-21T10:50:58.852 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-03-21T10:50:58.852 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-03-21T10:50:58.852 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-03-21T10:50:58.852 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-21T10:50:58.852 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-21T10:50:58.852 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-21T10:50:58.852 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-03-21T10:50:58.852 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-21T10:50:58.852 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 281, 2024-03-21T10:50:58.852 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 1695509, 2024-03-21T10:50:58.852 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-21T10:50:58.853 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 281, 2024-03-21T10:50:58.853 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-21T10:50:58.853 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-21T10:50:58.853 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 281, 2024-03-21T10:50:58.853 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 1695509, 2024-03-21T10:50:58.853 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-21T10:50:58.853 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-21T10:50:58.853 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-21T10:50:58.853 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-21T10:50:58.853 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-21T10:50:58.853 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-21T10:50:58.853 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-21T10:50:59.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:50:59 smithi031 ceph-mon[31673]: pgmap v387: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:50:59.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:50:59 smithi070 ceph-mon[36330]: pgmap v387: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:50:59.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:50:59 smithi040 ceph-mon[36154]: pgmap v387: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:50:59.526 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-03-21T10:50:59.526 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-03-21T10:50:59.526 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-03-21T10:50:59.526 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-03-21T10:50:59.527 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-03-21T10:50:59.527 DEBUG:tasks.check_counter:Getting stats from c 2024-03-21T10:50:59.527 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-21T10:50:59.781 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.c/config 2024-03-21T10:51:01.230 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:01 smithi070 ceph-mon[36330]: pgmap v388: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:51:01.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:01 smithi040 ceph-mon[36154]: pgmap v388: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:51:01.555 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:01 smithi031 ceph-mon[31673]: pgmap v388: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-21T10:51:01.698 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-03-21T10:51:01.698 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-0": { 2024-03-21T10:51:01.699 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 2702, 2024-03-21T10:51:01.699 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 2704, 2024-03-21T10:51:01.699 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 801210, 2024-03-21T10:51:01.699 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 859243, 2024-03-21T10:51:01.699 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 6, 2024-03-21T10:51:01.699 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 2, 2024-03-21T10:51:01.699 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.159984813, 2024-03-21T10:51:01.699 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.047561187, 2024-03-21T10:51:01.699 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.040252923, 2024-03-21T10:51:01.699 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.060927089, 2024-03-21T10:51:01.699 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:51:01.699 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 2702, 2024-03-21T10:51:01.699 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.058264602, 2024-03-21T10:51:01.699 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000021563 2024-03-21T10:51:01.700 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.700 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:51:01.700 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.700 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.700 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.700 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.700 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-21T10:51:01.700 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 0 2024-03-21T10:51:01.700 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.700 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-1": { 2024-03-21T10:51:01.700 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 306, 2024-03-21T10:51:01.700 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 285, 2024-03-21T10:51:01.700 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 1798348, 2024-03-21T10:51:01.700 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 161626, 2024-03-21T10:51:01.701 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 3, 2024-03-21T10:51:01.701 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 2, 2024-03-21T10:51:01.701 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.042618517, 2024-03-21T10:51:01.701 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.011245013, 2024-03-21T10:51:01.701 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.037378216, 2024-03-21T10:51:01.701 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.005016192, 2024-03-21T10:51:01.701 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:51:01.701 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 284, 2024-03-21T10:51:01.701 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.009345431, 2024-03-21T10:51:01.701 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000032906 2024-03-21T10:51:01.701 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.701 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:51:01.701 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.702 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.702 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.702 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.702 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 1797792, 2024-03-21T10:51:01.702 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 160928 2024-03-21T10:51:01.702 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.702 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-2": { 2024-03-21T10:51:01.702 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 7, 2024-03-21T10:51:01.702 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 203, 2024-03-21T10:51:01.702 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 3580, 2024-03-21T10:51:01.702 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 578270, 2024-03-21T10:51:01.702 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 8, 2024-03-21T10:51:01.702 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 4, 2024-03-21T10:51:01.702 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.010647928, 2024-03-21T10:51:01.702 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.006290098, 2024-03-21T10:51:01.703 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.002609885, 2024-03-21T10:51:01.703 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.000036685, 2024-03-21T10:51:01.703 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:51:01.703 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 201, 2024-03-21T10:51:01.703 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.025236941, 2024-03-21T10:51:01.703 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000125556 2024-03-21T10:51:01.703 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.703 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:51:01.703 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.703 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.703 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.703 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.703 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-21T10:51:01.704 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 577952 2024-03-21T10:51:01.704 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.704 INFO:teuthology.orchestra.run.smithi070.stdout: "cct": { 2024-03-21T10:51:01.704 INFO:teuthology.orchestra.run.smithi070.stdout: "total_workers": 1, 2024-03-21T10:51:01.704 INFO:teuthology.orchestra.run.smithi070.stdout: "unhealthy_workers": 0 2024-03-21T10:51:01.704 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.704 INFO:teuthology.orchestra.run.smithi070.stdout: "finisher-MDSRank": { 2024-03-21T10:51:01.704 INFO:teuthology.orchestra.run.smithi070.stdout: "queue_len": 0, 2024-03-21T10:51:01.704 INFO:teuthology.orchestra.run.smithi070.stdout: "complete_latency": { 2024-03-21T10:51:01.704 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 4551, 2024-03-21T10:51:01.704 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.091557118, 2024-03-21T10:51:01.704 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000020118 2024-03-21T10:51:01.704 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:01.704 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.704 INFO:teuthology.orchestra.run.smithi070.stdout: "finisher-PurgeQueue": { 2024-03-21T10:51:01.705 INFO:teuthology.orchestra.run.smithi070.stdout: "queue_len": 0, 2024-03-21T10:51:01.705 INFO:teuthology.orchestra.run.smithi070.stdout: "complete_latency": { 2024-03-21T10:51:01.705 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.705 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.705 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.705 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:01.705 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.705 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-03-21T10:51:01.705 INFO:teuthology.orchestra.run.smithi070.stdout: "request": 0, 2024-03-21T10:51:01.705 INFO:teuthology.orchestra.run.smithi070.stdout: "reply": 0, 2024-03-21T10:51:01.705 INFO:teuthology.orchestra.run.smithi070.stdout: "reply_latency": { 2024-03-21T10:51:01.705 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.705 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.705 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.706 INFO:teuthology.orchestra.run.smithi070.stdout: "slow_reply": 0, 2024-03-21T10:51:01.706 INFO:teuthology.orchestra.run.smithi070.stdout: "forward": 0, 2024-03-21T10:51:01.706 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_fetch_complete": 0, 2024-03-21T10:51:01.706 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_fetch_keys": 0, 2024-03-21T10:51:01.706 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_commit": 0, 2024-03-21T10:51:01.706 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_split": 0, 2024-03-21T10:51:01.706 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_merge": 0, 2024-03-21T10:51:01.706 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes": 3, 2024-03-21T10:51:01.706 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_top": 0, 2024-03-21T10:51:01.706 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_bottom": 3, 2024-03-21T10:51:01.706 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_pin_tail": 0, 2024-03-21T10:51:01.706 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_pinned": 3, 2024-03-21T10:51:01.706 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_expired": 0, 2024-03-21T10:51:01.707 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_with_caps": 0, 2024-03-21T10:51:01.707 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": 0, 2024-03-21T10:51:01.707 INFO:teuthology.orchestra.run.smithi070.stdout: "subtrees": 1, 2024-03-21T10:51:01.707 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse": 0, 2024-03-21T10:51:01.707 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_hit": 0, 2024-03-21T10:51:01.707 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_forward": 0, 2024-03-21T10:51:01.707 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_discover": 0, 2024-03-21T10:51:01.707 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_dir_fetch": 0, 2024-03-21T10:51:01.707 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_remote_ino": 0, 2024-03-21T10:51:01.707 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_lock": 0, 2024-03-21T10:51:01.707 INFO:teuthology.orchestra.run.smithi070.stdout: "load_cent": 0, 2024-03-21T10:51:01.707 INFO:teuthology.orchestra.run.smithi070.stdout: "q": 0, 2024-03-21T10:51:01.707 INFO:teuthology.orchestra.run.smithi070.stdout: "exported": 0, 2024-03-21T10:51:01.707 INFO:teuthology.orchestra.run.smithi070.stdout: "exported_inodes": 0, 2024-03-21T10:51:01.707 INFO:teuthology.orchestra.run.smithi070.stdout: "imported": 0, 2024-03-21T10:51:01.708 INFO:teuthology.orchestra.run.smithi070.stdout: "imported_inodes": 0, 2024-03-21T10:51:01.708 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_dir_fetch": 0, 2024-03-21T10:51:01.708 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_backtrace_fetch": 0, 2024-03-21T10:51:01.708 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_peer_discover": 0, 2024-03-21T10:51:01.708 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rfiles": 0, 2024-03-21T10:51:01.708 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rbytes": 0, 2024-03-21T10:51:01.708 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rsnaps": 0, 2024-03-21T10:51:01.708 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_backtrace_fetch": 0, 2024-03-21T10:51:01.708 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_set_tag": 0, 2024-03-21T10:51:01.708 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_backtrace_repaired": 0, 2024-03-21T10:51:01.708 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_inotable_repaired": 0, 2024-03-21T10:51:01.708 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dir_inodes": 0, 2024-03-21T10:51:01.708 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dir_base_inodes": 0, 2024-03-21T10:51:01.708 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dirfrag_rstats": 0, 2024-03-21T10:51:01.708 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_file_inodes": 0, 2024-03-21T10:51:01.709 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_inode_file_caps": 0, 2024-03-21T10:51:01.709 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_revoke": 0, 2024-03-21T10:51:01.709 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_grant": 0, 2024-03-21T10:51:01.709 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_trunc": 0, 2024-03-21T10:51:01.709 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-21T10:51:01.709 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-21T10:51:01.709 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_caps": 0, 2024-03-21T10:51:01.709 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_caps_dirty": 0, 2024-03-21T10:51:01.709 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_cap_release": 0, 2024-03-21T10:51:01.709 INFO:teuthology.orchestra.run.smithi070.stdout: "process_request_cap_release": 0 2024-03-21T10:51:01.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.709 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache": { 2024-03-21T10:51:01.709 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_update": 0, 2024-03-21T10:51:01.709 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_update_receipt": 0, 2024-03-21T10:51:01.709 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_try_discover": 0, 2024-03-21T10:51:01.710 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_send_discover": 0, 2024-03-21T10:51:01.710 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_handle_discover": 0, 2024-03-21T10:51:01.710 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays": 0, 2024-03-21T10:51:01.710 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays_delayed": 0, 2024-03-21T10:51:01.710 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays_enqueuing": 0, 2024-03-21T10:51:01.710 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_created": 0, 2024-03-21T10:51:01.710 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_enqueued": 0, 2024-03-21T10:51:01.710 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_reintegrated": 0, 2024-03-21T10:51:01.710 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_migrated": 0, 2024-03-21T10:51:01.710 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_processing": 0, 2024-03-21T10:51:01.710 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_enqueued": 0, 2024-03-21T10:51:01.710 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_prioritized": 0, 2024-03-21T10:51:01.710 INFO:teuthology.orchestra.run.smithi070.stdout: "recovery_started": 0, 2024-03-21T10:51:01.710 INFO:teuthology.orchestra.run.smithi070.stdout: "recovery_completed": 0, 2024-03-21T10:51:01.710 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_enqueue_scrub": 0, 2024-03-21T10:51:01.711 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_exportdir": 0, 2024-03-21T10:51:01.711 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_flush": 0, 2024-03-21T10:51:01.711 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_fragmentdir": 0, 2024-03-21T10:51:01.711 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_fragstats": 0, 2024-03-21T10:51:01.711 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_inodestats": 0 2024-03-21T10:51:01.711 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.711 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log": { 2024-03-21T10:51:01.711 INFO:teuthology.orchestra.run.smithi070.stdout: "evlrg": 0, 2024-03-21T10:51:01.711 INFO:teuthology.orchestra.run.smithi070.stdout: "evadd": 0, 2024-03-21T10:51:01.711 INFO:teuthology.orchestra.run.smithi070.stdout: "evex": 0, 2024-03-21T10:51:01.711 INFO:teuthology.orchestra.run.smithi070.stdout: "evtrm": 0, 2024-03-21T10:51:01.711 INFO:teuthology.orchestra.run.smithi070.stdout: "ev": 23, 2024-03-21T10:51:01.711 INFO:teuthology.orchestra.run.smithi070.stdout: "evexg": 0, 2024-03-21T10:51:01.711 INFO:teuthology.orchestra.run.smithi070.stdout: "evexd": 0, 2024-03-21T10:51:01.711 INFO:teuthology.orchestra.run.smithi070.stdout: "segadd": 0, 2024-03-21T10:51:01.712 INFO:teuthology.orchestra.run.smithi070.stdout: "segex": 0, 2024-03-21T10:51:01.712 INFO:teuthology.orchestra.run.smithi070.stdout: "segtrm": 0, 2024-03-21T10:51:01.712 INFO:teuthology.orchestra.run.smithi070.stdout: "seg": 2, 2024-03-21T10:51:01.712 INFO:teuthology.orchestra.run.smithi070.stdout: "segmjr": 2, 2024-03-21T10:51:01.712 INFO:teuthology.orchestra.run.smithi070.stdout: "segexg": 0, 2024-03-21T10:51:01.712 INFO:teuthology.orchestra.run.smithi070.stdout: "segexd": 0, 2024-03-21T10:51:01.712 INFO:teuthology.orchestra.run.smithi070.stdout: "expos": 4194304, 2024-03-21T10:51:01.712 INFO:teuthology.orchestra.run.smithi070.stdout: "wrpos": 4232679, 2024-03-21T10:51:01.712 INFO:teuthology.orchestra.run.smithi070.stdout: "rdpos": 4230074, 2024-03-21T10:51:01.712 INFO:teuthology.orchestra.run.smithi070.stdout: "jlat": { 2024-03-21T10:51:01.712 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.712 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.712 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.712 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.712 INFO:teuthology.orchestra.run.smithi070.stdout: "replayed": 23 2024-03-21T10:51:01.713 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.713 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_mem": { 2024-03-21T10:51:01.713 INFO:teuthology.orchestra.run.smithi070.stdout: "ino": 4, 2024-03-21T10:51:01.713 INFO:teuthology.orchestra.run.smithi070.stdout: "ino+": 20, 2024-03-21T10:51:01.713 INFO:teuthology.orchestra.run.smithi070.stdout: "ino-": 16, 2024-03-21T10:51:01.713 INFO:teuthology.orchestra.run.smithi070.stdout: "dir": 2, 2024-03-21T10:51:01.713 INFO:teuthology.orchestra.run.smithi070.stdout: "dir+": 13, 2024-03-21T10:51:01.713 INFO:teuthology.orchestra.run.smithi070.stdout: "dir-": 11, 2024-03-21T10:51:01.713 INFO:teuthology.orchestra.run.smithi070.stdout: "dn": 3, 2024-03-21T10:51:01.713 INFO:teuthology.orchestra.run.smithi070.stdout: "dn+": 19, 2024-03-21T10:51:01.713 INFO:teuthology.orchestra.run.smithi070.stdout: "dn-": 16, 2024-03-21T10:51:01.713 INFO:teuthology.orchestra.run.smithi070.stdout: "cap": 0, 2024-03-21T10:51:01.713 INFO:teuthology.orchestra.run.smithi070.stdout: "cap+": 0, 2024-03-21T10:51:01.713 INFO:teuthology.orchestra.run.smithi070.stdout: "cap-": 0, 2024-03-21T10:51:01.713 INFO:teuthology.orchestra.run.smithi070.stdout: "rss": 48296, 2024-03-21T10:51:01.713 INFO:teuthology.orchestra.run.smithi070.stdout: "heap": 215420 2024-03-21T10:51:01.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.714 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_server": { 2024-03-21T10:51:01.714 INFO:teuthology.orchestra.run.smithi070.stdout: "dispatch_client_request": 0, 2024-03-21T10:51:01.714 INFO:teuthology.orchestra.run.smithi070.stdout: "dispatch_server_request": 0, 2024-03-21T10:51:01.714 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_request": 0, 2024-03-21T10:51:01.714 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_session": 0, 2024-03-21T10:51:01.714 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_peer_request": 0, 2024-03-21T10:51:01.714 INFO:teuthology.orchestra.run.smithi070.stdout: "req_create_latency": { 2024-03-21T10:51:01.714 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.714 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.714 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.714 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getattr_latency": { 2024-03-21T10:51:01.714 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.714 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.715 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.715 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getfilelock_latency": { 2024-03-21T10:51:01.715 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.715 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.715 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.715 INFO:teuthology.orchestra.run.smithi070.stdout: "req_link_latency": { 2024-03-21T10:51:01.715 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.715 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.715 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.715 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookup_latency": { 2024-03-21T10:51:01.715 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.715 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.716 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.716 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.716 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookuphash_latency": { 2024-03-21T10:51:01.716 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.716 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.716 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.716 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.716 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupino_latency": { 2024-03-21T10:51:01.716 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.716 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.716 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.716 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.716 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupname_latency": { 2024-03-21T10:51:01.716 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.716 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.716 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.717 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupparent_latency": { 2024-03-21T10:51:01.717 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.717 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.717 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.717 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupsnap_latency": { 2024-03-21T10:51:01.717 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.717 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.717 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.717 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lssnap_latency": { 2024-03-21T10:51:01.717 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.718 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.718 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.718 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.718 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mkdir_latency": { 2024-03-21T10:51:01.718 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.718 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.718 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.718 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.719 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mknod_latency": { 2024-03-21T10:51:01.719 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.719 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.719 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.719 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.719 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mksnap_latency": { 2024-03-21T10:51:01.719 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.719 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.719 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.719 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.719 INFO:teuthology.orchestra.run.smithi070.stdout: "req_open_latency": { 2024-03-21T10:51:01.719 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.719 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.720 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.720 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.720 INFO:teuthology.orchestra.run.smithi070.stdout: "req_readdir_latency": { 2024-03-21T10:51:01.720 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.720 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.720 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.720 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.720 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rename_latency": { 2024-03-21T10:51:01.720 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.720 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.720 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.720 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.720 INFO:teuthology.orchestra.run.smithi070.stdout: "req_renamesnap_latency": { 2024-03-21T10:51:01.720 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.721 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.721 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.721 INFO:teuthology.orchestra.run.smithi070.stdout: "req_snapdiff_latency": { 2024-03-21T10:51:01.721 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.721 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.721 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.721 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmdir_latency": { 2024-03-21T10:51:01.721 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.721 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.721 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.721 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmsnap_latency": { 2024-03-21T10:51:01.721 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.721 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.722 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.722 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmxattr_latency": { 2024-03-21T10:51:01.722 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.722 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.722 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.722 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setattr_latency": { 2024-03-21T10:51:01.722 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.722 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.722 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.722 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setdirlayout_latency": { 2024-03-21T10:51:01.722 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setfilelock_latency": { 2024-03-21T10:51:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setlayout_latency": { 2024-03-21T10:51:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setxattr_latency": { 2024-03-21T10:51:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: "req_symlink_latency": { 2024-03-21T10:51:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: "req_unlink_latency": { 2024-03-21T10:51:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: "cap_revoke_eviction": 0, 2024-03-21T10:51:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: "cap_acquisition_throttle": 0, 2024-03-21T10:51:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getvxattr_latency": { 2024-03-21T10:51:01.725 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.725 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.725 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.725 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:01.725 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.725 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_sessions": { 2024-03-21T10:51:01.726 INFO:teuthology.orchestra.run.smithi070.stdout: "session_count": 2, 2024-03-21T10:51:01.726 INFO:teuthology.orchestra.run.smithi070.stdout: "session_add": 5, 2024-03-21T10:51:01.726 INFO:teuthology.orchestra.run.smithi070.stdout: "session_remove": 3, 2024-03-21T10:51:01.726 INFO:teuthology.orchestra.run.smithi070.stdout: "sessions_open": 2, 2024-03-21T10:51:01.726 INFO:teuthology.orchestra.run.smithi070.stdout: "sessions_stale": 0, 2024-03-21T10:51:01.726 INFO:teuthology.orchestra.run.smithi070.stdout: "total_load": 0, 2024-03-21T10:51:01.727 INFO:teuthology.orchestra.run.smithi070.stdout: "average_load": 0, 2024-03-21T10:51:01.727 INFO:teuthology.orchestra.run.smithi070.stdout: "avg_session_uptime": 5771, 2024-03-21T10:51:01.727 INFO:teuthology.orchestra.run.smithi070.stdout: "mdthresh_evicted": 0 2024-03-21T10:51:01.727 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.727 INFO:teuthology.orchestra.run.smithi070.stdout: "mempool": { 2024-03-21T10:51:01.727 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_bytes": 0, 2024-03-21T10:51:01.727 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_items": 0, 2024-03-21T10:51:01.727 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_bytes": 0, 2024-03-21T10:51:01.727 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_items": 0, 2024-03-21T10:51:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_bytes": 0, 2024-03-21T10:51:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_items": 0, 2024-03-21T10:51:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-21T10:51:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_items": 0, 2024-03-21T10:51:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-21T10:51:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_items": 0, 2024-03-21T10:51:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_bytes": 0, 2024-03-21T10:51:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_items": 0, 2024-03-21T10:51:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-21T10:51:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_items": 0, 2024-03-21T10:51:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_bytes": 0, 2024-03-21T10:51:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_items": 0, 2024-03-21T10:51:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_bytes": 0, 2024-03-21T10:51:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_items": 0, 2024-03-21T10:51:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-21T10:51:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_items": 0, 2024-03-21T10:51:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-21T10:51:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_items": 0, 2024-03-21T10:51:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_bytes": 0, 2024-03-21T10:51:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_items": 0, 2024-03-21T10:51:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_bytes": 0, 2024-03-21T10:51:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_items": 0, 2024-03-21T10:51:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-21T10:51:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_items": 0, 2024-03-21T10:51:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_bytes": 0, 2024-03-21T10:51:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_items": 0, 2024-03-21T10:51:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_bytes": 0, 2024-03-21T10:51:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_items": 0, 2024-03-21T10:51:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_bytes": 0, 2024-03-21T10:51:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_items": 0, 2024-03-21T10:51:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_bytes": 0, 2024-03-21T10:51:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_items": 0, 2024-03-21T10:51:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_bytes": 182059, 2024-03-21T10:51:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_items": 48, 2024-03-21T10:51:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_bytes": 0, 2024-03-21T10:51:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_items": 0, 2024-03-21T10:51:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_bytes": 0, 2024-03-21T10:51:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_items": 0, 2024-03-21T10:51:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_bytes": 0, 2024-03-21T10:51:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_items": 0, 2024-03-21T10:51:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_bytes": 0, 2024-03-21T10:51:01.731 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_items": 0, 2024-03-21T10:51:01.731 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_bytes": 9328, 2024-03-21T10:51:01.731 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_items": 179, 2024-03-21T10:51:01.731 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_bytes": 0, 2024-03-21T10:51:01.731 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_items": 0, 2024-03-21T10:51:01.731 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_bytes": 0, 2024-03-21T10:51:01.731 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_items": 0, 2024-03-21T10:51:01.731 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_bytes": 15584, 2024-03-21T10:51:01.731 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_items": 44, 2024-03-21T10:51:01.731 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_bytes": 0, 2024-03-21T10:51:01.731 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_items": 0, 2024-03-21T10:51:01.731 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_bytes": 0, 2024-03-21T10:51:01.731 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_items": 0 2024-03-21T10:51:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: "objecter": { 2024-03-21T10:51:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: "op_active": 0, 2024-03-21T10:51:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: "op_laggy": 0, 2024-03-21T10:51:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: "op_send": 2705, 2024-03-21T10:51:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: "op_send_bytes": 21, 2024-03-21T10:51:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: "op_resend": 3, 2024-03-21T10:51:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: "op_reply": 2705, 2024-03-21T10:51:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: "op_latency": { 2024-03-21T10:51:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 2705, 2024-03-21T10:51:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 1.230963875, 2024-03-21T10:51:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000455069 2024-03-21T10:51:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.733 INFO:teuthology.orchestra.run.smithi070.stdout: "op_inflight": 0, 2024-03-21T10:51:01.733 INFO:teuthology.orchestra.run.smithi070.stdout: "oplen_avg": { 2024-03-21T10:51:01.733 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 2705, 2024-03-21T10:51:01.733 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 2706 2024-03-21T10:51:01.733 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.734 INFO:teuthology.orchestra.run.smithi070.stdout: "op": 2705, 2024-03-21T10:51:01.734 INFO:teuthology.orchestra.run.smithi070.stdout: "op_r": 2705, 2024-03-21T10:51:01.734 INFO:teuthology.orchestra.run.smithi070.stdout: "op_w": 0, 2024-03-21T10:51:01.734 INFO:teuthology.orchestra.run.smithi070.stdout: "op_rmw": 0, 2024-03-21T10:51:01.734 INFO:teuthology.orchestra.run.smithi070.stdout: "op_pg": 0, 2024-03-21T10:51:01.735 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_stat": 1767, 2024-03-21T10:51:01.735 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_create": 0, 2024-03-21T10:51:01.735 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_read": 936, 2024-03-21T10:51:01.735 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_write": 0, 2024-03-21T10:51:01.735 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_writefull": 0, 2024-03-21T10:51:01.735 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_writesame": 0, 2024-03-21T10:51:01.735 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_append": 0, 2024-03-21T10:51:01.735 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_zero": 0, 2024-03-21T10:51:01.735 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_truncate": 0, 2024-03-21T10:51:01.735 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_delete": 0, 2024-03-21T10:51:01.735 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_mapext": 0, 2024-03-21T10:51:01.735 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_sparse_read": 0, 2024-03-21T10:51:01.735 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_clonerange": 0, 2024-03-21T10:51:01.736 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_getxattr": 1, 2024-03-21T10:51:01.736 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_setxattr": 0, 2024-03-21T10:51:01.736 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_cmpxattr": 0, 2024-03-21T10:51:01.736 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_rmxattr": 0, 2024-03-21T10:51:01.736 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_resetxattrs": 0, 2024-03-21T10:51:01.736 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_call": 0, 2024-03-21T10:51:01.736 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_watch": 0, 2024-03-21T10:51:01.736 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_notify": 0, 2024-03-21T10:51:01.736 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_src_cmpxattr": 0, 2024-03-21T10:51:01.736 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_pgls": 0, 2024-03-21T10:51:01.736 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_pgls_filter": 0, 2024-03-21T10:51:01.736 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_other": 0, 2024-03-21T10:51:01.736 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_active": 0, 2024-03-21T10:51:01.736 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_send": 0, 2024-03-21T10:51:01.737 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_resend": 0, 2024-03-21T10:51:01.737 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_ping": 0, 2024-03-21T10:51:01.737 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_active": 0, 2024-03-21T10:51:01.737 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_send": 0, 2024-03-21T10:51:01.737 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_resend": 0, 2024-03-21T10:51:01.737 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_active": 0, 2024-03-21T10:51:01.737 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_send": 0, 2024-03-21T10:51:01.737 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_resend": 0, 2024-03-21T10:51:01.737 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_active": 0, 2024-03-21T10:51:01.737 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_send": 0, 2024-03-21T10:51:01.737 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_resend": 0, 2024-03-21T10:51:01.737 INFO:teuthology.orchestra.run.smithi070.stdout: "command_active": 0, 2024-03-21T10:51:01.737 INFO:teuthology.orchestra.run.smithi070.stdout: "command_send": 0, 2024-03-21T10:51:01.738 INFO:teuthology.orchestra.run.smithi070.stdout: "command_resend": 0, 2024-03-21T10:51:01.738 INFO:teuthology.orchestra.run.smithi070.stdout: "map_epoch": 106, 2024-03-21T10:51:01.738 INFO:teuthology.orchestra.run.smithi070.stdout: "map_full": 0, 2024-03-21T10:51:01.738 INFO:teuthology.orchestra.run.smithi070.stdout: "map_inc": 17, 2024-03-21T10:51:01.738 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_sessions": 4, 2024-03-21T10:51:01.738 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_session_open": 5, 2024-03-21T10:51:01.738 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_session_close": 1, 2024-03-21T10:51:01.738 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_laggy": 0, 2024-03-21T10:51:01.738 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_wr": 0, 2024-03-21T10:51:01.738 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_rd": 2, 2024-03-21T10:51:01.738 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_del": 0 2024-03-21T10:51:01.738 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.738 INFO:teuthology.orchestra.run.smithi070.stdout: "oft": { 2024-03-21T10:51:01.739 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_objs": 0, 2024-03-21T10:51:01.739 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_kv_pairs": 0, 2024-03-21T10:51:01.739 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_updates": 0, 2024-03-21T10:51:01.739 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_removes": 0 2024-03-21T10:51:01.739 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.739 INFO:teuthology.orchestra.run.smithi070.stdout: "osd-slow-ops": { 2024-03-21T10:51:01.739 INFO:teuthology.orchestra.run.smithi070.stdout: "slow_ops_count": 0 2024-03-21T10:51:01.739 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.739 INFO:teuthology.orchestra.run.smithi070.stdout: "purge_queue": { 2024-03-21T10:51:01.739 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_ops": 0, 2024-03-21T10:51:01.739 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_ops_high_water": 0, 2024-03-21T10:51:01.739 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing": 0, 2024-03-21T10:51:01.739 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_high_water": 0, 2024-03-21T10:51:01.740 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executed_ops": 0, 2024-03-21T10:51:01.740 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executed": 0, 2024-03-21T10:51:01.740 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_item_in_journal": 0 2024-03-21T10:51:01.740 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.740 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-21T10:51:01.740 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-03-21T10:51:01.740 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 104857600, 2024-03-21T10:51:01.740 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-03-21T10:51:01.740 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 3011, 2024-03-21T10:51:01.740 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 2313637, 2024-03-21T10:51:01.740 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-03-21T10:51:01.741 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 3011, 2024-03-21T10:51:01.741 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-03-21T10:51:01.741 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-03-21T10:51:01.741 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 3011, 2024-03-21T10:51:01.741 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 2313637, 2024-03-21T10:51:01.741 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-03-21T10:51:01.741 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.741 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.741 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.741 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:01.741 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.741 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-objecter_bytes": { 2024-03-21T10:51:01.742 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-03-21T10:51:01.742 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 104857600, 2024-03-21T10:51:01.742 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-03-21T10:51:01.742 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-03-21T10:51:01.742 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-03-21T10:51:01.742 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-03-21T10:51:01.742 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-03-21T10:51:01.742 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 2705, 2024-03-21T10:51:01.742 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 38380, 2024-03-21T10:51:01.742 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 2705, 2024-03-21T10:51:01.743 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 38380, 2024-03-21T10:51:01.743 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-03-21T10:51:01.743 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.743 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.743 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.743 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:01.743 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.743 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-objecter_ops": { 2024-03-21T10:51:01.743 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-03-21T10:51:01.743 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 1024, 2024-03-21T10:51:01.743 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-03-21T10:51:01.743 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-03-21T10:51:01.744 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-03-21T10:51:01.744 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-03-21T10:51:01.744 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-03-21T10:51:01.744 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 2705, 2024-03-21T10:51:01.744 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 2705, 2024-03-21T10:51:01.744 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 2705, 2024-03-21T10:51:01.744 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 2705, 2024-03-21T10:51:01.744 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-03-21T10:51:01.744 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.744 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.744 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.745 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:01.745 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.745 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-write_buf_throttle": { 2024-03-21T10:51:01.745 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-03-21T10:51:01.745 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 3758096384, 2024-03-21T10:51:01.745 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-03-21T10:51:01.745 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-03-21T10:51:01.745 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-03-21T10:51:01.745 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-03-21T10:51:01.745 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-03-21T10:51:01.745 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-03-21T10:51:01.745 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-03-21T10:51:01.745 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 0, 2024-03-21T10:51:01.746 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 0, 2024-03-21T10:51:01.746 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-03-21T10:51:01.746 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.746 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.746 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.746 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:01.746 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:01.746 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-write_buf_throttle-0x55e64bbee6e0": { 2024-03-21T10:51:01.746 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-03-21T10:51:01.746 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 3758096384, 2024-03-21T10:51:01.746 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-03-21T10:51:01.746 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-03-21T10:51:01.747 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-03-21T10:51:01.747 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-03-21T10:51:01.747 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-03-21T10:51:01.747 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-03-21T10:51:01.747 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-03-21T10:51:01.747 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 0, 2024-03-21T10:51:01.747 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 0, 2024-03-21T10:51:01.747 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-03-21T10:51:01.747 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:01.747 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:01.747 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:01.747 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:01.748 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:01.748 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-03-21T10:51:02.416 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-21T10:51:02.416 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-21T10:51:02.416 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=0 2024-03-21T10:51:02.416 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-03-21T10:51:02.416 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-03-21T10:51:02.416 DEBUG:tasks.check_counter:Getting stats from f 2024-03-21T10:51:02.417 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-21T10:51:02.666 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.c/config 2024-03-21T10:51:03.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:03 smithi070 ceph-mon[36330]: pgmap v389: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:51:03.356 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:03 smithi031 ceph-mon[31673]: pgmap v389: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:51:03.429 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:03 smithi040 ceph-mon[36154]: pgmap v389: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-21T10:51:04.423 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-03-21T10:51:04.423 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-0": { 2024-03-21T10:51:04.423 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 935, 2024-03-21T10:51:04.424 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 937, 2024-03-21T10:51:04.424 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 625626, 2024-03-21T10:51:04.424 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 297394, 2024-03-21T10:51:04.424 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 7, 2024-03-21T10:51:04.424 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 2, 2024-03-21T10:51:04.424 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.059205077, 2024-03-21T10:51:04.424 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.017677158, 2024-03-21T10:51:04.424 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.015659455, 2024-03-21T10:51:04.424 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.019211892, 2024-03-21T10:51:04.424 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:51:04.424 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 935, 2024-03-21T10:51:04.424 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.024017441, 2024-03-21T10:51:04.424 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000025687 2024-03-21T10:51:04.425 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.425 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:51:04.425 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.425 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.425 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.425 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.425 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-21T10:51:04.425 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 0 2024-03-21T10:51:04.425 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.425 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-1": { 2024-03-21T10:51:04.425 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 768, 2024-03-21T10:51:04.425 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 966, 2024-03-21T10:51:04.425 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 188778, 2024-03-21T10:51:04.425 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 744889, 2024-03-21T10:51:04.425 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 7, 2024-03-21T10:51:04.426 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 5, 2024-03-21T10:51:04.426 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.062683016, 2024-03-21T10:51:04.426 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.019240267, 2024-03-21T10:51:04.426 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.014098598, 2024-03-21T10:51:04.426 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.023218909, 2024-03-21T10:51:04.426 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:51:04.426 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 963, 2024-03-21T10:51:04.426 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.037823093, 2024-03-21T10:51:04.426 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000039276 2024-03-21T10:51:04.426 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.426 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:51:04.426 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.426 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.426 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.427 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.427 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-21T10:51:04.427 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 501296 2024-03-21T10:51:04.427 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.427 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-2": { 2024-03-21T10:51:04.427 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 1224, 2024-03-21T10:51:04.427 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 1201, 2024-03-21T10:51:04.427 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 2108496, 2024-03-21T10:51:04.427 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 452970, 2024-03-21T10:51:04.427 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 3, 2024-03-21T10:51:04.427 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 2, 2024-03-21T10:51:04.427 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.116243449, 2024-03-21T10:51:04.427 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.032341163, 2024-03-21T10:51:04.427 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.056473150, 2024-03-21T10:51:04.427 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.031552878, 2024-03-21T10:51:04.428 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:51:04.428 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 1201, 2024-03-21T10:51:04.428 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.043233231, 2024-03-21T10:51:04.428 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000035997 2024-03-21T10:51:04.428 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.428 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:51:04.428 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.428 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.428 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.428 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.428 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 1801056, 2024-03-21T10:51:04.428 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 162000 2024-03-21T10:51:04.428 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.428 INFO:teuthology.orchestra.run.smithi070.stdout: "cct": { 2024-03-21T10:51:04.429 INFO:teuthology.orchestra.run.smithi070.stdout: "total_workers": 1, 2024-03-21T10:51:04.429 INFO:teuthology.orchestra.run.smithi070.stdout: "unhealthy_workers": 0 2024-03-21T10:51:04.429 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.429 INFO:teuthology.orchestra.run.smithi070.stdout: "finisher-MDSRank": { 2024-03-21T10:51:04.429 INFO:teuthology.orchestra.run.smithi070.stdout: "queue_len": 0, 2024-03-21T10:51:04.429 INFO:teuthology.orchestra.run.smithi070.stdout: "complete_latency": { 2024-03-21T10:51:04.429 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 4440, 2024-03-21T10:51:04.429 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.096978514, 2024-03-21T10:51:04.429 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000021842 2024-03-21T10:51:04.429 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:04.429 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.429 INFO:teuthology.orchestra.run.smithi070.stdout: "finisher-PurgeQueue": { 2024-03-21T10:51:04.429 INFO:teuthology.orchestra.run.smithi070.stdout: "queue_len": 0, 2024-03-21T10:51:04.429 INFO:teuthology.orchestra.run.smithi070.stdout: "complete_latency": { 2024-03-21T10:51:04.429 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.430 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.430 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.430 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:04.430 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.430 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-03-21T10:51:04.430 INFO:teuthology.orchestra.run.smithi070.stdout: "request": 0, 2024-03-21T10:51:04.430 INFO:teuthology.orchestra.run.smithi070.stdout: "reply": 0, 2024-03-21T10:51:04.430 INFO:teuthology.orchestra.run.smithi070.stdout: "reply_latency": { 2024-03-21T10:51:04.430 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.430 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.430 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.430 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.430 INFO:teuthology.orchestra.run.smithi070.stdout: "slow_reply": 0, 2024-03-21T10:51:04.431 INFO:teuthology.orchestra.run.smithi070.stdout: "forward": 0, 2024-03-21T10:51:04.431 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_fetch_complete": 0, 2024-03-21T10:51:04.431 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_fetch_keys": 0, 2024-03-21T10:51:04.431 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_commit": 0, 2024-03-21T10:51:04.431 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_split": 0, 2024-03-21T10:51:04.431 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_merge": 0, 2024-03-21T10:51:04.431 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes": 120, 2024-03-21T10:51:04.431 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_top": 38, 2024-03-21T10:51:04.431 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_bottom": 82, 2024-03-21T10:51:04.431 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_pin_tail": 0, 2024-03-21T10:51:04.431 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_pinned": 65, 2024-03-21T10:51:04.431 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_expired": 0, 2024-03-21T10:51:04.431 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_with_caps": 0, 2024-03-21T10:51:04.431 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": 0, 2024-03-21T10:51:04.431 INFO:teuthology.orchestra.run.smithi070.stdout: "subtrees": 1, 2024-03-21T10:51:04.432 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse": 0, 2024-03-21T10:51:04.432 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_hit": 0, 2024-03-21T10:51:04.432 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_forward": 0, 2024-03-21T10:51:04.432 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_discover": 0, 2024-03-21T10:51:04.432 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_dir_fetch": 0, 2024-03-21T10:51:04.432 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_remote_ino": 0, 2024-03-21T10:51:04.432 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_lock": 0, 2024-03-21T10:51:04.432 INFO:teuthology.orchestra.run.smithi070.stdout: "load_cent": 0, 2024-03-21T10:51:04.432 INFO:teuthology.orchestra.run.smithi070.stdout: "q": 0, 2024-03-21T10:51:04.432 INFO:teuthology.orchestra.run.smithi070.stdout: "exported": 0, 2024-03-21T10:51:04.432 INFO:teuthology.orchestra.run.smithi070.stdout: "exported_inodes": 0, 2024-03-21T10:51:04.432 INFO:teuthology.orchestra.run.smithi070.stdout: "imported": 0, 2024-03-21T10:51:04.432 INFO:teuthology.orchestra.run.smithi070.stdout: "imported_inodes": 0, 2024-03-21T10:51:04.432 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_dir_fetch": 0, 2024-03-21T10:51:04.432 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_backtrace_fetch": 0, 2024-03-21T10:51:04.432 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_peer_discover": 0, 2024-03-21T10:51:04.433 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rfiles": 0, 2024-03-21T10:51:04.433 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rbytes": 0, 2024-03-21T10:51:04.433 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rsnaps": 0, 2024-03-21T10:51:04.433 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_backtrace_fetch": 0, 2024-03-21T10:51:04.433 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_set_tag": 0, 2024-03-21T10:51:04.433 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_backtrace_repaired": 0, 2024-03-21T10:51:04.433 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_inotable_repaired": 0, 2024-03-21T10:51:04.433 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dir_inodes": 0, 2024-03-21T10:51:04.433 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dir_base_inodes": 0, 2024-03-21T10:51:04.433 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dirfrag_rstats": 0, 2024-03-21T10:51:04.433 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_file_inodes": 0, 2024-03-21T10:51:04.433 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_inode_file_caps": 0, 2024-03-21T10:51:04.433 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_revoke": 0, 2024-03-21T10:51:04.433 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_grant": 0, 2024-03-21T10:51:04.434 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_trunc": 0, 2024-03-21T10:51:04.434 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-21T10:51:04.434 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-21T10:51:04.434 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_caps": 0, 2024-03-21T10:51:04.434 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_caps_dirty": 0, 2024-03-21T10:51:04.434 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_cap_release": 0, 2024-03-21T10:51:04.434 INFO:teuthology.orchestra.run.smithi070.stdout: "process_request_cap_release": 0 2024-03-21T10:51:04.434 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.434 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache": { 2024-03-21T10:51:04.434 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_update": 0, 2024-03-21T10:51:04.434 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_update_receipt": 0, 2024-03-21T10:51:04.434 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_try_discover": 0, 2024-03-21T10:51:04.434 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_send_discover": 0, 2024-03-21T10:51:04.434 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_handle_discover": 0, 2024-03-21T10:51:04.434 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays": 0, 2024-03-21T10:51:04.435 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays_delayed": 0, 2024-03-21T10:51:04.435 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays_enqueuing": 0, 2024-03-21T10:51:04.435 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_created": 0, 2024-03-21T10:51:04.435 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_enqueued": 0, 2024-03-21T10:51:04.435 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_reintegrated": 0, 2024-03-21T10:51:04.435 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_migrated": 0, 2024-03-21T10:51:04.435 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_processing": 0, 2024-03-21T10:51:04.435 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_enqueued": 0, 2024-03-21T10:51:04.435 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_prioritized": 0, 2024-03-21T10:51:04.435 INFO:teuthology.orchestra.run.smithi070.stdout: "recovery_started": 0, 2024-03-21T10:51:04.435 INFO:teuthology.orchestra.run.smithi070.stdout: "recovery_completed": 0, 2024-03-21T10:51:04.435 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_enqueue_scrub": 0, 2024-03-21T10:51:04.435 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_exportdir": 0, 2024-03-21T10:51:04.435 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_flush": 0, 2024-03-21T10:51:04.435 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_fragmentdir": 0, 2024-03-21T10:51:04.436 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_fragstats": 0, 2024-03-21T10:51:04.436 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_inodestats": 0 2024-03-21T10:51:04.436 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.436 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log": { 2024-03-21T10:51:04.436 INFO:teuthology.orchestra.run.smithi070.stdout: "evlrg": 0, 2024-03-21T10:51:04.436 INFO:teuthology.orchestra.run.smithi070.stdout: "evadd": 0, 2024-03-21T10:51:04.436 INFO:teuthology.orchestra.run.smithi070.stdout: "evex": 0, 2024-03-21T10:51:04.436 INFO:teuthology.orchestra.run.smithi070.stdout: "evtrm": 0, 2024-03-21T10:51:04.436 INFO:teuthology.orchestra.run.smithi070.stdout: "ev": 81, 2024-03-21T10:51:04.436 INFO:teuthology.orchestra.run.smithi070.stdout: "evexg": 0, 2024-03-21T10:51:04.436 INFO:teuthology.orchestra.run.smithi070.stdout: "evexd": 0, 2024-03-21T10:51:04.436 INFO:teuthology.orchestra.run.smithi070.stdout: "segadd": 0, 2024-03-21T10:51:04.436 INFO:teuthology.orchestra.run.smithi070.stdout: "segex": 0, 2024-03-21T10:51:04.436 INFO:teuthology.orchestra.run.smithi070.stdout: "segtrm": 0, 2024-03-21T10:51:04.436 INFO:teuthology.orchestra.run.smithi070.stdout: "seg": 2, 2024-03-21T10:51:04.436 INFO:teuthology.orchestra.run.smithi070.stdout: "segmjr": 2, 2024-03-21T10:51:04.437 INFO:teuthology.orchestra.run.smithi070.stdout: "segexg": 0, 2024-03-21T10:51:04.437 INFO:teuthology.orchestra.run.smithi070.stdout: "segexd": 0, 2024-03-21T10:51:04.437 INFO:teuthology.orchestra.run.smithi070.stdout: "expos": 4194304, 2024-03-21T10:51:04.437 INFO:teuthology.orchestra.run.smithi070.stdout: "wrpos": 4574593, 2024-03-21T10:51:04.437 INFO:teuthology.orchestra.run.smithi070.stdout: "rdpos": 4570574, 2024-03-21T10:51:04.437 INFO:teuthology.orchestra.run.smithi070.stdout: "jlat": { 2024-03-21T10:51:04.437 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.437 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.437 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.437 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.437 INFO:teuthology.orchestra.run.smithi070.stdout: "replayed": 81 2024-03-21T10:51:04.437 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.437 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_mem": { 2024-03-21T10:51:04.437 INFO:teuthology.orchestra.run.smithi070.stdout: "ino": 83, 2024-03-21T10:51:04.437 INFO:teuthology.orchestra.run.smithi070.stdout: "ino+": 103, 2024-03-21T10:51:04.438 INFO:teuthology.orchestra.run.smithi070.stdout: "ino-": 20, 2024-03-21T10:51:04.438 INFO:teuthology.orchestra.run.smithi070.stdout: "dir": 6, 2024-03-21T10:51:04.438 INFO:teuthology.orchestra.run.smithi070.stdout: "dir+": 11, 2024-03-21T10:51:04.438 INFO:teuthology.orchestra.run.smithi070.stdout: "dir-": 5, 2024-03-21T10:51:04.438 INFO:teuthology.orchestra.run.smithi070.stdout: "dn": 120, 2024-03-21T10:51:04.438 INFO:teuthology.orchestra.run.smithi070.stdout: "dn+": 126, 2024-03-21T10:51:04.438 INFO:teuthology.orchestra.run.smithi070.stdout: "dn-": 6, 2024-03-21T10:51:04.438 INFO:teuthology.orchestra.run.smithi070.stdout: "cap": 0, 2024-03-21T10:51:04.438 INFO:teuthology.orchestra.run.smithi070.stdout: "cap+": 0, 2024-03-21T10:51:04.438 INFO:teuthology.orchestra.run.smithi070.stdout: "cap-": 0, 2024-03-21T10:51:04.438 INFO:teuthology.orchestra.run.smithi070.stdout: "rss": 50652, 2024-03-21T10:51:04.438 INFO:teuthology.orchestra.run.smithi070.stdout: "heap": 215420 2024-03-21T10:51:04.438 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.438 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_server": { 2024-03-21T10:51:04.438 INFO:teuthology.orchestra.run.smithi070.stdout: "dispatch_client_request": 0, 2024-03-21T10:51:04.439 INFO:teuthology.orchestra.run.smithi070.stdout: "dispatch_server_request": 0, 2024-03-21T10:51:04.439 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_request": 0, 2024-03-21T10:51:04.439 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_session": 0, 2024-03-21T10:51:04.439 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_peer_request": 0, 2024-03-21T10:51:04.439 INFO:teuthology.orchestra.run.smithi070.stdout: "req_create_latency": { 2024-03-21T10:51:04.439 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.439 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.439 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.439 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.439 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getattr_latency": { 2024-03-21T10:51:04.439 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.439 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.439 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.439 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.439 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getfilelock_latency": { 2024-03-21T10:51:04.440 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.440 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.440 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.440 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.440 INFO:teuthology.orchestra.run.smithi070.stdout: "req_link_latency": { 2024-03-21T10:51:04.440 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.440 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.440 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.440 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.440 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookup_latency": { 2024-03-21T10:51:04.440 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.440 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.440 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.440 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.440 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookuphash_latency": { 2024-03-21T10:51:04.441 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.441 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.441 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.441 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.441 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupino_latency": { 2024-03-21T10:51:04.441 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.441 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.441 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.441 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.441 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupname_latency": { 2024-03-21T10:51:04.441 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.441 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.441 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.441 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.441 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupparent_latency": { 2024-03-21T10:51:04.441 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.442 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.442 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.442 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.442 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupsnap_latency": { 2024-03-21T10:51:04.442 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.442 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.442 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.442 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.442 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lssnap_latency": { 2024-03-21T10:51:04.442 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.442 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.442 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.442 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.443 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mkdir_latency": { 2024-03-21T10:51:04.443 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.443 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.443 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.444 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.444 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mknod_latency": { 2024-03-21T10:51:04.444 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.444 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.444 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.444 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.444 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mksnap_latency": { 2024-03-21T10:51:04.444 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.444 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.444 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.444 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.444 INFO:teuthology.orchestra.run.smithi070.stdout: "req_open_latency": { 2024-03-21T10:51:04.444 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.444 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.445 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.445 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.445 INFO:teuthology.orchestra.run.smithi070.stdout: "req_readdir_latency": { 2024-03-21T10:51:04.445 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.445 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.445 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.445 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.445 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rename_latency": { 2024-03-21T10:51:04.445 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.445 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.445 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.445 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.445 INFO:teuthology.orchestra.run.smithi070.stdout: "req_renamesnap_latency": { 2024-03-21T10:51:04.445 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.446 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.446 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.446 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.446 INFO:teuthology.orchestra.run.smithi070.stdout: "req_snapdiff_latency": { 2024-03-21T10:51:04.446 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.446 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.446 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.446 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.446 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmdir_latency": { 2024-03-21T10:51:04.446 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.446 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.446 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.447 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.447 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmsnap_latency": { 2024-03-21T10:51:04.447 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.447 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.447 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.447 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.447 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmxattr_latency": { 2024-03-21T10:51:04.447 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.447 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.447 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.447 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.447 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setattr_latency": { 2024-03-21T10:51:04.447 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.447 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.448 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.448 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.448 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setdirlayout_latency": { 2024-03-21T10:51:04.448 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.448 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.448 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.448 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.448 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setfilelock_latency": { 2024-03-21T10:51:04.448 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.448 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.448 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.448 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.448 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setlayout_latency": { 2024-03-21T10:51:04.448 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.448 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.449 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.449 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.449 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setxattr_latency": { 2024-03-21T10:51:04.449 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.449 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.449 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.449 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.449 INFO:teuthology.orchestra.run.smithi070.stdout: "req_symlink_latency": { 2024-03-21T10:51:04.449 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.449 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.449 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.449 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.449 INFO:teuthology.orchestra.run.smithi070.stdout: "req_unlink_latency": { 2024-03-21T10:51:04.449 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.449 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.449 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.450 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.450 INFO:teuthology.orchestra.run.smithi070.stdout: "cap_revoke_eviction": 0, 2024-03-21T10:51:04.450 INFO:teuthology.orchestra.run.smithi070.stdout: "cap_acquisition_throttle": 0, 2024-03-21T10:51:04.450 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getvxattr_latency": { 2024-03-21T10:51:04.450 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.450 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.450 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.450 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:04.450 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.450 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_sessions": { 2024-03-21T10:51:04.450 INFO:teuthology.orchestra.run.smithi070.stdout: "session_count": 1, 2024-03-21T10:51:04.450 INFO:teuthology.orchestra.run.smithi070.stdout: "session_add": 3, 2024-03-21T10:51:04.450 INFO:teuthology.orchestra.run.smithi070.stdout: "session_remove": 2, 2024-03-21T10:51:04.450 INFO:teuthology.orchestra.run.smithi070.stdout: "sessions_open": 3, 2024-03-21T10:51:04.451 INFO:teuthology.orchestra.run.smithi070.stdout: "sessions_stale": 0, 2024-03-21T10:51:04.451 INFO:teuthology.orchestra.run.smithi070.stdout: "total_load": 0, 2024-03-21T10:51:04.451 INFO:teuthology.orchestra.run.smithi070.stdout: "average_load": 0, 2024-03-21T10:51:04.451 INFO:teuthology.orchestra.run.smithi070.stdout: "avg_session_uptime": 5773, 2024-03-21T10:51:04.451 INFO:teuthology.orchestra.run.smithi070.stdout: "mdthresh_evicted": 0 2024-03-21T10:51:04.451 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.451 INFO:teuthology.orchestra.run.smithi070.stdout: "mempool": { 2024-03-21T10:51:04.451 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_bytes": 0, 2024-03-21T10:51:04.451 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_items": 0, 2024-03-21T10:51:04.451 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_bytes": 0, 2024-03-21T10:51:04.451 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_items": 0, 2024-03-21T10:51:04.451 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_bytes": 0, 2024-03-21T10:51:04.451 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_items": 0, 2024-03-21T10:51:04.451 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-21T10:51:04.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_items": 0, 2024-03-21T10:51:04.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-21T10:51:04.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_items": 0, 2024-03-21T10:51:04.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_bytes": 0, 2024-03-21T10:51:04.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_items": 0, 2024-03-21T10:51:04.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-21T10:51:04.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_items": 0, 2024-03-21T10:51:04.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_bytes": 0, 2024-03-21T10:51:04.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_items": 0, 2024-03-21T10:51:04.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_bytes": 0, 2024-03-21T10:51:04.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_items": 0, 2024-03-21T10:51:04.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-21T10:51:04.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_items": 0, 2024-03-21T10:51:04.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-21T10:51:04.452 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_items": 0, 2024-03-21T10:51:04.453 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_bytes": 0, 2024-03-21T10:51:04.453 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_items": 0, 2024-03-21T10:51:04.453 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_bytes": 0, 2024-03-21T10:51:04.453 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_items": 0, 2024-03-21T10:51:04.453 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-21T10:51:04.453 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_items": 0, 2024-03-21T10:51:04.453 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_bytes": 0, 2024-03-21T10:51:04.453 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_items": 0, 2024-03-21T10:51:04.453 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_bytes": 0, 2024-03-21T10:51:04.453 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_items": 0, 2024-03-21T10:51:04.454 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_bytes": 0, 2024-03-21T10:51:04.454 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_items": 0, 2024-03-21T10:51:04.454 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_bytes": 0, 2024-03-21T10:51:04.454 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_items": 0, 2024-03-21T10:51:04.454 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_bytes": 193895, 2024-03-21T10:51:04.454 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_items": 154, 2024-03-21T10:51:04.454 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_bytes": 0, 2024-03-21T10:51:04.454 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_items": 0, 2024-03-21T10:51:04.454 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_bytes": 0, 2024-03-21T10:51:04.454 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_items": 0, 2024-03-21T10:51:04.454 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_bytes": 0, 2024-03-21T10:51:04.454 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_items": 0, 2024-03-21T10:51:04.454 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_bytes": 0, 2024-03-21T10:51:04.454 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_items": 0, 2024-03-21T10:51:04.454 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_bytes": 9328, 2024-03-21T10:51:04.455 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_items": 179, 2024-03-21T10:51:04.455 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_bytes": 0, 2024-03-21T10:51:04.455 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_items": 0, 2024-03-21T10:51:04.455 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_bytes": 0, 2024-03-21T10:51:04.455 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_items": 0, 2024-03-21T10:51:04.455 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_bytes": 299528, 2024-03-21T10:51:04.455 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_items": 2887, 2024-03-21T10:51:04.455 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_bytes": 0, 2024-03-21T10:51:04.455 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_items": 0, 2024-03-21T10:51:04.455 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_bytes": 0, 2024-03-21T10:51:04.455 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_items": 0 2024-03-21T10:51:04.455 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.455 INFO:teuthology.orchestra.run.smithi070.stdout: "objecter": { 2024-03-21T10:51:04.455 INFO:teuthology.orchestra.run.smithi070.stdout: "op_active": 0, 2024-03-21T10:51:04.456 INFO:teuthology.orchestra.run.smithi070.stdout: "op_laggy": 0, 2024-03-21T10:51:04.456 INFO:teuthology.orchestra.run.smithi070.stdout: "op_send": 2616, 2024-03-21T10:51:04.456 INFO:teuthology.orchestra.run.smithi070.stdout: "op_send_bytes": 21, 2024-03-21T10:51:04.456 INFO:teuthology.orchestra.run.smithi070.stdout: "op_resend": 3, 2024-03-21T10:51:04.456 INFO:teuthology.orchestra.run.smithi070.stdout: "op_reply": 2616, 2024-03-21T10:51:04.456 INFO:teuthology.orchestra.run.smithi070.stdout: "op_latency": { 2024-03-21T10:51:04.456 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 2616, 2024-03-21T10:51:04.456 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 9.097733497, 2024-03-21T10:51:04.456 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.003477726 2024-03-21T10:51:04.456 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.456 INFO:teuthology.orchestra.run.smithi070.stdout: "op_inflight": 0, 2024-03-21T10:51:04.456 INFO:teuthology.orchestra.run.smithi070.stdout: "oplen_avg": { 2024-03-21T10:51:04.456 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 2616, 2024-03-21T10:51:04.456 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 2617 2024-03-21T10:51:04.456 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.456 INFO:teuthology.orchestra.run.smithi070.stdout: "op": 2616, 2024-03-21T10:51:04.457 INFO:teuthology.orchestra.run.smithi070.stdout: "op_r": 2616, 2024-03-21T10:51:04.457 INFO:teuthology.orchestra.run.smithi070.stdout: "op_w": 0, 2024-03-21T10:51:04.457 INFO:teuthology.orchestra.run.smithi070.stdout: "op_rmw": 0, 2024-03-21T10:51:04.457 INFO:teuthology.orchestra.run.smithi070.stdout: "op_pg": 0, 2024-03-21T10:51:04.457 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_stat": 1679, 2024-03-21T10:51:04.457 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_create": 0, 2024-03-21T10:51:04.457 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_read": 935, 2024-03-21T10:51:04.457 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_write": 0, 2024-03-21T10:51:04.457 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_writefull": 0, 2024-03-21T10:51:04.457 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_writesame": 0, 2024-03-21T10:51:04.457 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_append": 0, 2024-03-21T10:51:04.457 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_zero": 0, 2024-03-21T10:51:04.457 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_truncate": 0, 2024-03-21T10:51:04.457 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_delete": 0, 2024-03-21T10:51:04.457 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_mapext": 0, 2024-03-21T10:51:04.458 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_sparse_read": 0, 2024-03-21T10:51:04.458 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_clonerange": 0, 2024-03-21T10:51:04.458 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_getxattr": 1, 2024-03-21T10:51:04.458 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_setxattr": 0, 2024-03-21T10:51:04.458 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_cmpxattr": 0, 2024-03-21T10:51:04.458 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_rmxattr": 0, 2024-03-21T10:51:04.458 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_resetxattrs": 0, 2024-03-21T10:51:04.458 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_call": 0, 2024-03-21T10:51:04.458 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_watch": 0, 2024-03-21T10:51:04.458 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_notify": 0, 2024-03-21T10:51:04.458 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_src_cmpxattr": 0, 2024-03-21T10:51:04.458 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_pgls": 0, 2024-03-21T10:51:04.458 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_pgls_filter": 0, 2024-03-21T10:51:04.458 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_other": 0, 2024-03-21T10:51:04.458 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_active": 0, 2024-03-21T10:51:04.459 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_send": 0, 2024-03-21T10:51:04.459 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_resend": 0, 2024-03-21T10:51:04.459 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_ping": 0, 2024-03-21T10:51:04.459 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_active": 0, 2024-03-21T10:51:04.459 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_send": 0, 2024-03-21T10:51:04.459 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_resend": 0, 2024-03-21T10:51:04.459 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_active": 0, 2024-03-21T10:51:04.459 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_send": 0, 2024-03-21T10:51:04.459 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_resend": 0, 2024-03-21T10:51:04.459 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_active": 0, 2024-03-21T10:51:04.459 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_send": 0, 2024-03-21T10:51:04.459 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_resend": 0, 2024-03-21T10:51:04.459 INFO:teuthology.orchestra.run.smithi070.stdout: "command_active": 0, 2024-03-21T10:51:04.459 INFO:teuthology.orchestra.run.smithi070.stdout: "command_send": 0, 2024-03-21T10:51:04.459 INFO:teuthology.orchestra.run.smithi070.stdout: "command_resend": 0, 2024-03-21T10:51:04.460 INFO:teuthology.orchestra.run.smithi070.stdout: "map_epoch": 106, 2024-03-21T10:51:04.460 INFO:teuthology.orchestra.run.smithi070.stdout: "map_full": 0, 2024-03-21T10:51:04.460 INFO:teuthology.orchestra.run.smithi070.stdout: "map_inc": 17, 2024-03-21T10:51:04.460 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_sessions": 5, 2024-03-21T10:51:04.460 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_session_open": 6, 2024-03-21T10:51:04.460 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_session_close": 1, 2024-03-21T10:51:04.460 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_laggy": 0, 2024-03-21T10:51:04.460 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_wr": 0, 2024-03-21T10:51:04.460 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_rd": 2, 2024-03-21T10:51:04.460 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_del": 0 2024-03-21T10:51:04.460 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.460 INFO:teuthology.orchestra.run.smithi070.stdout: "oft": { 2024-03-21T10:51:04.460 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_objs": 0, 2024-03-21T10:51:04.460 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_kv_pairs": 0, 2024-03-21T10:51:04.460 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_updates": 0, 2024-03-21T10:51:04.460 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_removes": 0 2024-03-21T10:51:04.461 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.461 INFO:teuthology.orchestra.run.smithi070.stdout: "osd-slow-ops": { 2024-03-21T10:51:04.461 INFO:teuthology.orchestra.run.smithi070.stdout: "slow_ops_count": 0 2024-03-21T10:51:04.461 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.461 INFO:teuthology.orchestra.run.smithi070.stdout: "purge_queue": { 2024-03-21T10:51:04.461 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_ops": 0, 2024-03-21T10:51:04.461 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_ops_high_water": 0, 2024-03-21T10:51:04.461 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing": 0, 2024-03-21T10:51:04.461 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_high_water": 0, 2024-03-21T10:51:04.461 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executed_ops": 0, 2024-03-21T10:51:04.461 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executed": 0, 2024-03-21T10:51:04.461 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_item_in_journal": 0 2024-03-21T10:51:04.462 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.462 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-21T10:51:04.462 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-03-21T10:51:04.462 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 104857600, 2024-03-21T10:51:04.462 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-03-21T10:51:04.462 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 2923, 2024-03-21T10:51:04.462 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 2641267, 2024-03-21T10:51:04.462 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-03-21T10:51:04.462 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 2923, 2024-03-21T10:51:04.462 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-03-21T10:51:04.462 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-03-21T10:51:04.462 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 2923, 2024-03-21T10:51:04.462 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 2641267, 2024-03-21T10:51:04.462 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-03-21T10:51:04.462 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.463 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.463 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.463 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:04.463 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.463 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-objecter_bytes": { 2024-03-21T10:51:04.463 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-03-21T10:51:04.463 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 104857600, 2024-03-21T10:51:04.463 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-03-21T10:51:04.463 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-03-21T10:51:04.463 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-03-21T10:51:04.463 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-03-21T10:51:04.463 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-03-21T10:51:04.463 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 2616, 2024-03-21T10:51:04.463 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 380294, 2024-03-21T10:51:04.463 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 2616, 2024-03-21T10:51:04.464 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 380294, 2024-03-21T10:51:04.464 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-03-21T10:51:04.464 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.464 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.464 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.464 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:04.464 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.464 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-objecter_ops": { 2024-03-21T10:51:04.464 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-03-21T10:51:04.464 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 1024, 2024-03-21T10:51:04.464 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-03-21T10:51:04.465 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-03-21T10:51:04.465 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-03-21T10:51:04.465 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-03-21T10:51:04.465 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-03-21T10:51:04.465 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 2616, 2024-03-21T10:51:04.465 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 2616, 2024-03-21T10:51:04.465 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 2616, 2024-03-21T10:51:04.465 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 2616, 2024-03-21T10:51:04.465 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-03-21T10:51:04.465 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.465 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.465 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.465 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:04.465 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.466 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-write_buf_throttle": { 2024-03-21T10:51:04.466 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-03-21T10:51:04.466 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 3758096384, 2024-03-21T10:51:04.466 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-03-21T10:51:04.466 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-03-21T10:51:04.466 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-03-21T10:51:04.466 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-03-21T10:51:04.466 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-03-21T10:51:04.466 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-03-21T10:51:04.466 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-03-21T10:51:04.466 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 0, 2024-03-21T10:51:04.466 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 0, 2024-03-21T10:51:04.466 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-03-21T10:51:04.466 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.466 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.467 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.467 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:04.467 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:04.467 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-write_buf_throttle-0x5572b95626e0": { 2024-03-21T10:51:04.467 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-03-21T10:51:04.467 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 3758096384, 2024-03-21T10:51:04.467 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-03-21T10:51:04.467 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-03-21T10:51:04.467 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-03-21T10:51:04.467 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-03-21T10:51:04.467 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-03-21T10:51:04.467 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-03-21T10:51:04.467 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-03-21T10:51:04.467 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 0, 2024-03-21T10:51:04.467 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 0, 2024-03-21T10:51:04.468 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-03-21T10:51:04.468 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:04.468 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:04.468 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:04.468 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:04.468 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:04.468 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-03-21T10:51:05.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:05 smithi031 ceph-mon[31673]: pgmap v390: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 21 KiB/s wr, 19 op/s 2024-03-21T10:51:05.211 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-21T10:51:05.211 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-21T10:51:05.211 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-03-21T10:51:05.211 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-03-21T10:51:05.211 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-03-21T10:51:05.211 DEBUG:tasks.check_counter:Getting stats from i 2024-03-21T10:51:05.212 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-21T10:51:05.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:05 smithi070 ceph-mon[36330]: pgmap v390: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 21 KiB/s wr, 19 op/s 2024-03-21T10:51:05.429 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:05 smithi040 ceph-mon[36154]: pgmap v390: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 21 KiB/s wr, 19 op/s 2024-03-21T10:51:05.459 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.c/config 2024-03-21T10:51:07.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:07 smithi070 ceph-mon[36330]: pgmap v391: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-21T10:51:07.336 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-03-21T10:51:07.336 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-0": { 2024-03-21T10:51:07.336 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 2415, 2024-03-21T10:51:07.337 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 2358, 2024-03-21T10:51:07.337 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 742902, 2024-03-21T10:51:07.337 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 1556833, 2024-03-21T10:51:07.337 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 11, 2024-03-21T10:51:07.337 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 5, 2024-03-21T10:51:07.337 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.156844226, 2024-03-21T10:51:07.337 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.052792882, 2024-03-21T10:51:07.338 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.071892281, 2024-03-21T10:51:07.338 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.030105669, 2024-03-21T10:51:07.338 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:51:07.338 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 2360, 2024-03-21T10:51:07.338 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.113045158, 2024-03-21T10:51:07.338 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000047900 2024-03-21T10:51:07.338 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.338 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:51:07.339 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 2269, 2024-03-21T10:51:07.339 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000441192, 2024-03-21T10:51:07.339 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000194 2024-03-21T10:51:07.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.339 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-21T10:51:07.339 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 355712 2024-03-21T10:51:07.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.339 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-1": { 2024-03-21T10:51:07.340 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 1394, 2024-03-21T10:51:07.340 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 1448, 2024-03-21T10:51:07.340 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 373463, 2024-03-21T10:51:07.340 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 990575, 2024-03-21T10:51:07.340 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 43, 2024-03-21T10:51:07.340 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 7, 2024-03-21T10:51:07.340 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.114080564, 2024-03-21T10:51:07.341 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.032819616, 2024-03-21T10:51:07.341 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.039043192, 2024-03-21T10:51:07.341 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.033849769, 2024-03-21T10:51:07.341 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:51:07.341 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 1443, 2024-03-21T10:51:07.341 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.057366030, 2024-03-21T10:51:07.341 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000039754 2024-03-21T10:51:07.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.342 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:51:07.342 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 55, 2024-03-21T10:51:07.342 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000006896, 2024-03-21T10:51:07.342 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000125 2024-03-21T10:51:07.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.342 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-21T10:51:07.343 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 150416 2024-03-21T10:51:07.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.343 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-2": { 2024-03-21T10:51:07.343 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 890, 2024-03-21T10:51:07.343 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 1006, 2024-03-21T10:51:07.343 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 1991531, 2024-03-21T10:51:07.343 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 618934, 2024-03-21T10:51:07.343 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 22, 2024-03-21T10:51:07.344 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 7, 2024-03-21T10:51:07.344 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.085434741, 2024-03-21T10:51:07.344 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.025615438, 2024-03-21T10:51:07.344 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.057942436, 2024-03-21T10:51:07.344 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.012674901, 2024-03-21T10:51:07.344 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:51:07.344 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 1004, 2024-03-21T10:51:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.045540248, 2024-03-21T10:51:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000045358 2024-03-21T10:51:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:51:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 386, 2024-03-21T10:51:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000089714, 2024-03-21T10:51:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000232 2024-03-21T10:51:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 1811136, 2024-03-21T10:51:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 171360 2024-03-21T10:51:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: "cct": { 2024-03-21T10:51:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: "total_workers": 1, 2024-03-21T10:51:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: "unhealthy_workers": 0 2024-03-21T10:51:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "finisher-MDSRank": { 2024-03-21T10:51:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "queue_len": 0, 2024-03-21T10:51:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "complete_latency": { 2024-03-21T10:51:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 195, 2024-03-21T10:51:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.043982091, 2024-03-21T10:51:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000225549 2024-03-21T10:51:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "finisher-PurgeQueue": { 2024-03-21T10:51:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "queue_len": 0, 2024-03-21T10:51:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "complete_latency": { 2024-03-21T10:51:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 2514, 2024-03-21T10:51:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.083901770, 2024-03-21T10:51:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000033373 2024-03-21T10:51:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-03-21T10:51:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "request": 48, 2024-03-21T10:51:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "reply": 45, 2024-03-21T10:51:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "reply_latency": { 2024-03-21T10:51:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 45, 2024-03-21T10:51:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 2.166032472, 2024-03-21T10:51:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.048134054 2024-03-21T10:51:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "slow_reply": 0, 2024-03-21T10:51:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "forward": 3, 2024-03-21T10:51:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_fetch_complete": 0, 2024-03-21T10:51:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_fetch_keys": 0, 2024-03-21T10:51:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_commit": 11, 2024-03-21T10:51:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_split": 0, 2024-03-21T10:51:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_merge": 0, 2024-03-21T10:51:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes": 116, 2024-03-21T10:51:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_top": 37, 2024-03-21T10:51:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_bottom": 61, 2024-03-21T10:51:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_pin_tail": 18, 2024-03-21T10:51:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_pinned": 62, 2024-03-21T10:51:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_expired": 4, 2024-03-21T10:51:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_with_caps": 3, 2024-03-21T10:51:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": 3, 2024-03-21T10:51:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "subtrees": 2, 2024-03-21T10:51:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse": 75, 2024-03-21T10:51:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_hit": 61, 2024-03-21T10:51:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_forward": 1, 2024-03-21T10:51:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_discover": 2, 2024-03-21T10:51:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_dir_fetch": 0, 2024-03-21T10:51:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_remote_ino": 0, 2024-03-21T10:51:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_lock": 0, 2024-03-21T10:51:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "load_cent": 0, 2024-03-21T10:51:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "q": 0, 2024-03-21T10:51:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "exported": 1, 2024-03-21T10:51:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "exported_inodes": 2, 2024-03-21T10:51:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "imported": 2, 2024-03-21T10:51:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "imported_inodes": 81, 2024-03-21T10:51:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_dir_fetch": 0, 2024-03-21T10:51:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_backtrace_fetch": 120, 2024-03-21T10:51:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_peer_discover": 0, 2024-03-21T10:51:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rfiles": 22, 2024-03-21T10:51:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rbytes": 10738149278, 2024-03-21T10:51:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rsnaps": 6, 2024-03-21T10:51:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_backtrace_fetch": 120, 2024-03-21T10:51:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_set_tag": 0, 2024-03-21T10:51:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_backtrace_repaired": 0, 2024-03-21T10:51:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_inotable_repaired": 0, 2024-03-21T10:51:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dir_inodes": 24, 2024-03-21T10:51:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dir_base_inodes": 0, 2024-03-21T10:51:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dirfrag_rstats": 24, 2024-03-21T10:51:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_file_inodes": 96, 2024-03-21T10:51:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_inode_file_caps": 2, 2024-03-21T10:51:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_revoke": 14, 2024-03-21T10:51:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_grant": 32, 2024-03-21T10:51:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_trunc": 0, 2024-03-21T10:51:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-21T10:51:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-21T10:51:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_caps": 22, 2024-03-21T10:51:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_caps_dirty": 0, 2024-03-21T10:51:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_cap_release": 3, 2024-03-21T10:51:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: "process_request_cap_release": 27 2024-03-21T10:51:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache": { 2024-03-21T10:51:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_update": 0, 2024-03-21T10:51:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_update_receipt": 0, 2024-03-21T10:51:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_try_discover": 0, 2024-03-21T10:51:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_send_discover": 5, 2024-03-21T10:51:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_handle_discover": 0, 2024-03-21T10:51:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays": 4, 2024-03-21T10:51:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays_delayed": 0, 2024-03-21T10:51:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays_enqueuing": 0, 2024-03-21T10:51:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_created": 20, 2024-03-21T10:51:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_enqueued": 16, 2024-03-21T10:51:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_reintegrated": 0, 2024-03-21T10:51:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_migrated": 0, 2024-03-21T10:51:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_processing": 0, 2024-03-21T10:51:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_enqueued": 0, 2024-03-21T10:51:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_prioritized": 0, 2024-03-21T10:51:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "recovery_started": 0, 2024-03-21T10:51:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "recovery_completed": 0, 2024-03-21T10:51:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_enqueue_scrub": 0, 2024-03-21T10:51:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_exportdir": 1, 2024-03-21T10:51:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_flush": 0, 2024-03-21T10:51:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_fragmentdir": 0, 2024-03-21T10:51:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_fragstats": 0, 2024-03-21T10:51:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_inodestats": 0 2024-03-21T10:51:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log": { 2024-03-21T10:51:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "evlrg": 0, 2024-03-21T10:51:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "evadd": 81, 2024-03-21T10:51:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "evex": 0, 2024-03-21T10:51:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "evtrm": 0, 2024-03-21T10:51:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "ev": 81, 2024-03-21T10:51:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "evexg": 0, 2024-03-21T10:51:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "evexd": 0, 2024-03-21T10:51:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "segadd": 2, 2024-03-21T10:51:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "segex": 0, 2024-03-21T10:51:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "segtrm": 0, 2024-03-21T10:51:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "seg": 2, 2024-03-21T10:51:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "segmjr": 2, 2024-03-21T10:51:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "segexg": 0, 2024-03-21T10:51:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "segexd": 0, 2024-03-21T10:51:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "expos": 4194304, 2024-03-21T10:51:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "wrpos": 4574593, 2024-03-21T10:51:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "rdpos": 0, 2024-03-21T10:51:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "jlat": { 2024-03-21T10:51:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 25, 2024-03-21T10:51:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.060408114, 2024-03-21T10:51:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.002416324 2024-03-21T10:51:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "replayed": 0 2024-03-21T10:51:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_mem": { 2024-03-21T10:51:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "ino": 95, 2024-03-21T10:51:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "ino+": 112, 2024-03-21T10:51:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "ino-": 17, 2024-03-21T10:51:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "dir": 20, 2024-03-21T10:51:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "dir+": 21, 2024-03-21T10:51:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "dir-": 1, 2024-03-21T10:51:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "dn": 116, 2024-03-21T10:51:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "dn+": 136, 2024-03-21T10:51:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "dn-": 20, 2024-03-21T10:51:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "cap": 3, 2024-03-21T10:51:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "cap+": 25, 2024-03-21T10:51:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "cap-": 22, 2024-03-21T10:51:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "rss": 53512, 2024-03-21T10:51:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: "heap": 223612 2024-03-21T10:51:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_server": { 2024-03-21T10:51:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: "dispatch_client_request": 78, 2024-03-21T10:51:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: "dispatch_server_request": 2, 2024-03-21T10:51:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_request": 48, 2024-03-21T10:51:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_session": 18, 2024-03-21T10:51:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_peer_request": 10, 2024-03-21T10:51:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: "req_create_latency": { 2024-03-21T10:51:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getattr_latency": { 2024-03-21T10:51:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 2, 2024-03-21T10:51:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.001077185, 2024-03-21T10:51:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000538592 2024-03-21T10:51:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getfilelock_latency": { 2024-03-21T10:51:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "req_link_latency": { 2024-03-21T10:51:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookup_latency": { 2024-03-21T10:51:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 18, 2024-03-21T10:51:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.005908681, 2024-03-21T10:51:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000328260 2024-03-21T10:51:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookuphash_latency": { 2024-03-21T10:51:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupino_latency": { 2024-03-21T10:51:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 2, 2024-03-21T10:51:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000254489, 2024-03-21T10:51:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000127244 2024-03-21T10:51:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupname_latency": { 2024-03-21T10:51:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupparent_latency": { 2024-03-21T10:51:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupsnap_latency": { 2024-03-21T10:51:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lssnap_latency": { 2024-03-21T10:51:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mkdir_latency": { 2024-03-21T10:51:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mknod_latency": { 2024-03-21T10:51:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mksnap_latency": { 2024-03-21T10:51:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "req_open_latency": { 2024-03-21T10:51:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "req_readdir_latency": { 2024-03-21T10:51:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 3, 2024-03-21T10:51:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.001716655, 2024-03-21T10:51:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000572218 2024-03-21T10:51:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rename_latency": { 2024-03-21T10:51:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "req_renamesnap_latency": { 2024-03-21T10:51:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "req_snapdiff_latency": { 2024-03-21T10:51:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmdir_latency": { 2024-03-21T10:51:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 4, 2024-03-21T10:51:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 2.139058796, 2024-03-21T10:51:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.534764699 2024-03-21T10:51:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmsnap_latency": { 2024-03-21T10:51:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmxattr_latency": { 2024-03-21T10:51:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setattr_latency": { 2024-03-21T10:51:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setdirlayout_latency": { 2024-03-21T10:51:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setfilelock_latency": { 2024-03-21T10:51:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setlayout_latency": { 2024-03-21T10:51:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setxattr_latency": { 2024-03-21T10:51:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "req_symlink_latency": { 2024-03-21T10:51:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "req_unlink_latency": { 2024-03-21T10:51:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 16, 2024-03-21T10:51:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.018016666, 2024-03-21T10:51:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.001126041 2024-03-21T10:51:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "cap_revoke_eviction": 0, 2024-03-21T10:51:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "cap_acquisition_throttle": 0, 2024-03-21T10:51:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getvxattr_latency": { 2024-03-21T10:51:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_sessions": { 2024-03-21T10:51:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "session_count": 1, 2024-03-21T10:51:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "session_add": 3, 2024-03-21T10:51:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "session_remove": 2, 2024-03-21T10:51:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "sessions_open": 1, 2024-03-21T10:51:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "sessions_stale": 0, 2024-03-21T10:51:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "total_load": 36, 2024-03-21T10:51:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "average_load": 36, 2024-03-21T10:51:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "avg_session_uptime": 773, 2024-03-21T10:51:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "mdthresh_evicted": 0 2024-03-21T10:51:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "mempool": { 2024-03-21T10:51:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_bytes": 0, 2024-03-21T10:51:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_items": 0, 2024-03-21T10:51:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_bytes": 0, 2024-03-21T10:51:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_items": 0, 2024-03-21T10:51:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_bytes": 0, 2024-03-21T10:51:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_items": 0, 2024-03-21T10:51:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-21T10:51:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_items": 0, 2024-03-21T10:51:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-21T10:51:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_items": 0, 2024-03-21T10:51:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_bytes": 0, 2024-03-21T10:51:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_items": 0, 2024-03-21T10:51:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-21T10:51:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_items": 0, 2024-03-21T10:51:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_bytes": 0, 2024-03-21T10:51:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_items": 0, 2024-03-21T10:51:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_bytes": 0, 2024-03-21T10:51:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_items": 0, 2024-03-21T10:51:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-21T10:51:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_items": 0, 2024-03-21T10:51:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-21T10:51:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_items": 0, 2024-03-21T10:51:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_bytes": 0, 2024-03-21T10:51:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_items": 0, 2024-03-21T10:51:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_bytes": 0, 2024-03-21T10:51:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_items": 0, 2024-03-21T10:51:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-21T10:51:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_items": 0, 2024-03-21T10:51:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_bytes": 0, 2024-03-21T10:51:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_items": 0, 2024-03-21T10:51:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_bytes": 0, 2024-03-21T10:51:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_items": 0, 2024-03-21T10:51:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_bytes": 0, 2024-03-21T10:51:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_items": 0, 2024-03-21T10:51:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_bytes": 0, 2024-03-21T10:51:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_items": 0, 2024-03-21T10:51:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_bytes": 327961, 2024-03-21T10:51:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_items": 211, 2024-03-21T10:51:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_bytes": 0, 2024-03-21T10:51:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_items": 0, 2024-03-21T10:51:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_bytes": 0, 2024-03-21T10:51:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_items": 0, 2024-03-21T10:51:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_bytes": 0, 2024-03-21T10:51:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_items": 0, 2024-03-21T10:51:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_bytes": 0, 2024-03-21T10:51:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_items": 0, 2024-03-21T10:51:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_bytes": 9328, 2024-03-21T10:51:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_items": 179, 2024-03-21T10:51:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_bytes": 0, 2024-03-21T10:51:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_items": 0, 2024-03-21T10:51:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_bytes": 0, 2024-03-21T10:51:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_items": 0, 2024-03-21T10:51:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_bytes": 339036, 2024-03-21T10:51:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_items": 3362, 2024-03-21T10:51:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_bytes": 0, 2024-03-21T10:51:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_items": 0, 2024-03-21T10:51:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_bytes": 0, 2024-03-21T10:51:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_items": 0 2024-03-21T10:51:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "objecter": { 2024-03-21T10:51:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "op_active": 0, 2024-03-21T10:51:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "op_laggy": 0, 2024-03-21T10:51:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "op_send": 2801, 2024-03-21T10:51:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "op_send_bytes": 397461, 2024-03-21T10:51:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "op_resend": 31, 2024-03-21T10:51:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "op_reply": 2801, 2024-03-21T10:51:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "op_latency": { 2024-03-21T10:51:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 2801, 2024-03-21T10:51:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 67.667053545, 2024-03-21T10:51:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.024158176 2024-03-21T10:51:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "op_inflight": 0, 2024-03-21T10:51:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "oplen_avg": { 2024-03-21T10:51:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 2801, 2024-03-21T10:51:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 2837 2024-03-21T10:51:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "op": 2801, 2024-03-21T10:51:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "op_r": 121, 2024-03-21T10:51:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "op_w": 2680, 2024-03-21T10:51:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "op_rmw": 0, 2024-03-21T10:51:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "op_pg": 0, 2024-03-21T10:51:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_stat": 0, 2024-03-21T10:51:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_create": 10, 2024-03-21T10:51:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_read": 1, 2024-03-21T10:51:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_write": 26, 2024-03-21T10:51:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_writefull": 15, 2024-03-21T10:51:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_writesame": 0, 2024-03-21T10:51:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_append": 0, 2024-03-21T10:51:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_zero": 0, 2024-03-21T10:51:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_truncate": 0, 2024-03-21T10:51:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_delete": 2604, 2024-03-21T10:51:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_mapext": 0, 2024-03-21T10:51:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_sparse_read": 0, 2024-03-21T10:51:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_clonerange": 0, 2024-03-21T10:51:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_getxattr": 120, 2024-03-21T10:51:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_setxattr": 20, 2024-03-21T10:51:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_cmpxattr": 0, 2024-03-21T10:51:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_rmxattr": 0, 2024-03-21T10:51:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_resetxattrs": 0, 2024-03-21T10:51:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_call": 0, 2024-03-21T10:51:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_watch": 0, 2024-03-21T10:51:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_notify": 0, 2024-03-21T10:51:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_src_cmpxattr": 0, 2024-03-21T10:51:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_pgls": 0, 2024-03-21T10:51:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_pgls_filter": 0, 2024-03-21T10:51:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_other": 0, 2024-03-21T10:51:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_active": 0, 2024-03-21T10:51:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_send": 0, 2024-03-21T10:51:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_resend": 0, 2024-03-21T10:51:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_ping": 0, 2024-03-21T10:51:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_active": 0, 2024-03-21T10:51:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_send": 0, 2024-03-21T10:51:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_resend": 0, 2024-03-21T10:51:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_active": 0, 2024-03-21T10:51:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_send": 0, 2024-03-21T10:51:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_resend": 0, 2024-03-21T10:51:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_active": 0, 2024-03-21T10:51:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_send": 0, 2024-03-21T10:51:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_resend": 0, 2024-03-21T10:51:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "command_active": 0, 2024-03-21T10:51:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "command_send": 0, 2024-03-21T10:51:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "command_resend": 0, 2024-03-21T10:51:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "map_epoch": 107, 2024-03-21T10:51:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "map_full": 0, 2024-03-21T10:51:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "map_inc": 18, 2024-03-21T10:51:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_sessions": 12, 2024-03-21T10:51:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_session_open": 12, 2024-03-21T10:51:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_session_close": 0, 2024-03-21T10:51:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_laggy": 0, 2024-03-21T10:51:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_wr": 36, 2024-03-21T10:51:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_rd": 0, 2024-03-21T10:51:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_del": 5 2024-03-21T10:51:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "oft": { 2024-03-21T10:51:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_objs": 1, 2024-03-21T10:51:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_kv_pairs": 0, 2024-03-21T10:51:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_updates": 0, 2024-03-21T10:51:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_removes": 0 2024-03-21T10:51:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "osd-slow-ops": { 2024-03-21T10:51:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "slow_ops_count": 0 2024-03-21T10:51:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "purge_queue": { 2024-03-21T10:51:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_ops": 0, 2024-03-21T10:51:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_ops_high_water": 524, 2024-03-21T10:51:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing": 0, 2024-03-21T10:51:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_high_water": 6, 2024-03-21T10:51:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executed_ops": 2592, 2024-03-21T10:51:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executed": 16, 2024-03-21T10:51:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_item_in_journal": 0 2024-03-21T10:51:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-21T10:51:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-03-21T10:51:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 104857600, 2024-03-21T10:51:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-03-21T10:51:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 4695, 2024-03-21T10:51:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 2665720, 2024-03-21T10:51:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-03-21T10:51:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 4695, 2024-03-21T10:51:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-03-21T10:51:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-03-21T10:51:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 4695, 2024-03-21T10:51:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 2665720, 2024-03-21T10:51:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-03-21T10:51:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-objecter_bytes": { 2024-03-21T10:51:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-03-21T10:51:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 104857600, 2024-03-21T10:51:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-03-21T10:51:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-03-21T10:51:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-03-21T10:51:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-03-21T10:51:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-03-21T10:51:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 2801, 2024-03-21T10:51:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 399973, 2024-03-21T10:51:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 2801, 2024-03-21T10:51:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 399973, 2024-03-21T10:51:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-03-21T10:51:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-objecter_ops": { 2024-03-21T10:51:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-03-21T10:51:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 1024, 2024-03-21T10:51:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-03-21T10:51:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-03-21T10:51:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-03-21T10:51:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-03-21T10:51:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-03-21T10:51:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 2801, 2024-03-21T10:51:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 2801, 2024-03-21T10:51:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 2801, 2024-03-21T10:51:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 2801, 2024-03-21T10:51:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-03-21T10:51:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-write_buf_throttle": { 2024-03-21T10:51:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-03-21T10:51:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 3758096384, 2024-03-21T10:51:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-03-21T10:51:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 16, 2024-03-21T10:51:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 2512, 2024-03-21T10:51:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-03-21T10:51:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 16, 2024-03-21T10:51:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-03-21T10:51:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-03-21T10:51:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 1, 2024-03-21T10:51:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 2512, 2024-03-21T10:51:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-03-21T10:51:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-write_buf_throttle-0x55c097756460": { 2024-03-21T10:51:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-03-21T10:51:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 3758096384, 2024-03-21T10:51:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-03-21T10:51:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 81, 2024-03-21T10:51:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 380289, 2024-03-21T10:51:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-03-21T10:51:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 81, 2024-03-21T10:51:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-03-21T10:51:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-03-21T10:51:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 25, 2024-03-21T10:51:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 380289, 2024-03-21T10:51:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-03-21T10:51:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:07.389 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-03-21T10:51:07.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:07 smithi040 ceph-mon[36154]: pgmap v391: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-21T10:51:07.553 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:07 smithi031 ceph-mon[31673]: pgmap v391: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-21T10:51:08.208 INFO:tasks.check_counter:Daemon mds.i mds.exported=1 2024-03-21T10:51:08.284 INFO:tasks.check_counter:Daemon mds.i mds.imported=2 2024-03-21T10:51:08.284 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-03-21T10:51:08.284 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-03-21T10:51:08.284 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-03-21T10:51:08.284 DEBUG:tasks.check_counter:Getting stats from l 2024-03-21T10:51:08.284 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11 shell --fsid 3847408c-e76d-11ee-95cd-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-21T10:51:08.429 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:08 smithi040 ceph-mon[36154]: osdmap e107: 12 total, 12 up, 12 in 2024-03-21T10:51:08.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:08 smithi031 ceph-mon[31673]: osdmap e107: 12 total, 12 up, 12 in 2024-03-21T10:51:08.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:08 smithi070 ceph-mon[36330]: osdmap e107: 12 total, 12 up, 12 in 2024-03-21T10:51:08.681 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/mon.c/config 2024-03-21T10:51:09.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:09 smithi040 ceph-mon[36154]: pgmap v393: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-21T10:51:09.553 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:09 smithi031 ceph-mon[31673]: pgmap v393: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-21T10:51:09.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:09 smithi070 ceph-mon[36330]: pgmap v393: 129 pgs: 129 active+clean; 18 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-21T10:51:10.849 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-03-21T10:51:10.849 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-0": { 2024-03-21T10:51:10.849 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 285, 2024-03-21T10:51:10.849 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 271, 2024-03-21T10:51:10.849 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 1738512, 2024-03-21T10:51:10.849 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 162224, 2024-03-21T10:51:10.849 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 4, 2024-03-21T10:51:10.849 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 0, 2024-03-21T10:51:10.849 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.042947557, 2024-03-21T10:51:10.850 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.011172280, 2024-03-21T10:51:10.850 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.036817770, 2024-03-21T10:51:10.850 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.005322225, 2024-03-21T10:51:10.850 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:51:10.850 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 271, 2024-03-21T10:51:10.850 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.009190690, 2024-03-21T10:51:10.850 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000033913 2024-03-21T10:51:10.850 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:10.850 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:51:10.850 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:10.850 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:10.850 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:10.850 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:10.850 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 1738512, 2024-03-21T10:51:10.850 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 162224 2024-03-21T10:51:10.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:10.851 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-1": { 2024-03-21T10:51:10.851 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 2, 2024-03-21T10:51:10.851 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 206, 2024-03-21T10:51:10.851 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 320, 2024-03-21T10:51:10.851 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 191344, 2024-03-21T10:51:10.851 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 5, 2024-03-21T10:51:10.851 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 3, 2024-03-21T10:51:10.851 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.008996790, 2024-03-21T10:51:10.851 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.005945913, 2024-03-21T10:51:10.851 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.001197589, 2024-03-21T10:51:10.851 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-21T10:51:10.851 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:51:10.851 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 203, 2024-03-21T10:51:10.852 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.031732783, 2024-03-21T10:51:10.852 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000156319 2024-03-21T10:51:10.852 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:10.852 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:51:10.852 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:10.852 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:10.852 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:10.852 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:10.852 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-21T10:51:10.852 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 191344 2024-03-21T10:51:10.852 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:10.852 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-2": { 2024-03-21T10:51:10.852 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 0, 2024-03-21T10:51:10.852 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 2, 2024-03-21T10:51:10.852 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 0, 2024-03-21T10:51:10.853 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 0, 2024-03-21T10:51:10.853 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 2, 2024-03-21T10:51:10.853 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 1, 2024-03-21T10:51:10.853 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.000164586, 2024-03-21T10:51:10.853 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.000000000, 2024-03-21T10:51:10.853 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.000110706, 2024-03-21T10:51:10.853 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-21T10:51:10.853 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-03-21T10:51:10.853 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:10.853 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:10.853 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:10.853 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:10.853 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-03-21T10:51:10.853 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:10.853 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:10.854 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:10.854 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:10.854 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-21T10:51:10.854 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 0 2024-03-21T10:51:10.854 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:10.854 INFO:teuthology.orchestra.run.smithi070.stdout: "cct": { 2024-03-21T10:51:10.854 INFO:teuthology.orchestra.run.smithi070.stdout: "total_workers": 0, 2024-03-21T10:51:10.854 INFO:teuthology.orchestra.run.smithi070.stdout: "unhealthy_workers": 0 2024-03-21T10:51:10.854 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:10.854 INFO:teuthology.orchestra.run.smithi070.stdout: "mempool": { 2024-03-21T10:51:10.854 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_bytes": 0, 2024-03-21T10:51:10.854 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_items": 0, 2024-03-21T10:51:10.854 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_bytes": 0, 2024-03-21T10:51:10.854 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_items": 0, 2024-03-21T10:51:10.854 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_bytes": 0, 2024-03-21T10:51:10.855 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_items": 0, 2024-03-21T10:51:10.855 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-21T10:51:10.855 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_items": 0, 2024-03-21T10:51:10.855 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-21T10:51:10.855 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_items": 0, 2024-03-21T10:51:10.855 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_bytes": 0, 2024-03-21T10:51:10.855 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_items": 0, 2024-03-21T10:51:10.855 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-21T10:51:10.855 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_items": 0, 2024-03-21T10:51:10.855 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_bytes": 0, 2024-03-21T10:51:10.855 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_items": 0, 2024-03-21T10:51:10.855 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_bytes": 0, 2024-03-21T10:51:10.855 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_items": 0, 2024-03-21T10:51:10.855 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-21T10:51:10.855 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_items": 0, 2024-03-21T10:51:10.856 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-21T10:51:10.856 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_items": 0, 2024-03-21T10:51:10.856 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_bytes": 0, 2024-03-21T10:51:10.856 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_items": 0, 2024-03-21T10:51:10.856 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_bytes": 0, 2024-03-21T10:51:10.856 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_items": 0, 2024-03-21T10:51:10.856 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-21T10:51:10.856 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_items": 0, 2024-03-21T10:51:10.856 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_bytes": 0, 2024-03-21T10:51:10.856 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_items": 0, 2024-03-21T10:51:10.856 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_bytes": 0, 2024-03-21T10:51:10.856 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_items": 0, 2024-03-21T10:51:10.856 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_bytes": 0, 2024-03-21T10:51:10.856 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_items": 0, 2024-03-21T10:51:10.856 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_bytes": 0, 2024-03-21T10:51:10.856 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_items": 0, 2024-03-21T10:51:10.857 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_bytes": 148670, 2024-03-21T10:51:10.857 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_items": 30, 2024-03-21T10:51:10.857 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_bytes": 0, 2024-03-21T10:51:10.857 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_items": 0, 2024-03-21T10:51:10.857 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_bytes": 0, 2024-03-21T10:51:10.857 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_items": 0, 2024-03-21T10:51:10.857 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_bytes": 0, 2024-03-21T10:51:10.857 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_items": 0, 2024-03-21T10:51:10.857 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_bytes": 0, 2024-03-21T10:51:10.857 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_items": 0, 2024-03-21T10:51:10.857 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_bytes": 0, 2024-03-21T10:51:10.857 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_items": 0, 2024-03-21T10:51:10.857 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_bytes": 0, 2024-03-21T10:51:10.857 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_items": 0, 2024-03-21T10:51:10.857 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_bytes": 0, 2024-03-21T10:51:10.858 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_items": 0, 2024-03-21T10:51:10.858 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_bytes": 928, 2024-03-21T10:51:10.858 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_items": 2, 2024-03-21T10:51:10.858 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_bytes": 0, 2024-03-21T10:51:10.858 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_items": 0, 2024-03-21T10:51:10.858 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_bytes": 0, 2024-03-21T10:51:10.858 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_items": 0 2024-03-21T10:51:10.858 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-03-21T10:51:10.858 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-21T10:51:10.858 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-03-21T10:51:10.858 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 104857600, 2024-03-21T10:51:10.858 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-03-21T10:51:10.858 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 283, 2024-03-21T10:51:10.858 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 1695872, 2024-03-21T10:51:10.858 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-03-21T10:51:10.859 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 283, 2024-03-21T10:51:10.859 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-03-21T10:51:10.859 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-03-21T10:51:10.859 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 283, 2024-03-21T10:51:10.859 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 1695872, 2024-03-21T10:51:10.859 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-03-21T10:51:10.859 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-03-21T10:51:10.859 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-03-21T10:51:10.859 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-03-21T10:51:10.859 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:10.859 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-03-21T10:51:10.859 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-03-21T10:51:11.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:11 smithi040 ceph-mon[36154]: pgmap v394: 129 pgs: 15 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 75 active+clean; 13 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-21T10:51:11.553 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:11 smithi031 ceph-mon[31673]: pgmap v394: 129 pgs: 15 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 75 active+clean; 13 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-21T10:51:11.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:11 smithi070 ceph-mon[36330]: pgmap v394: 129 pgs: 15 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 75 active+clean; 13 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-21T10:51:12.360 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-03-21T10:51:12.361 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-03-21T10:51:12.361 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-03-21T10:51:12.361 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-03-21T10:51:12.361 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-03-21T10:51:12.361 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-21T10:51:12.374 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-21T10:51:12.374 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-21T10:51:12.374 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T10:51:12.444 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-21T10:51:12.444 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T10:51:12.500 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T10:51:12.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.737+0000 7f9023e00640 1 -- 172.21.15.31:0/2005751389 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9024106540 msgr2=0x7f9024106920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:12.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.737+0000 7f9023e00640 1 --2- 172.21.15.31:0/2005751389 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9024106540 0x7f9024106920 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f9014009890 tx=0x7f901402bb70 comp rx=0 tx=0).stop 2024-03-21T10:51:12.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.737+0000 7f9023e00640 1 -- 172.21.15.31:0/2005751389 shutdown_connections 2024-03-21T10:51:12.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.737+0000 7f9023e00640 1 --2- 172.21.15.31:0/2005751389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9024100820 0x7f902410e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:12.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.737+0000 7f9023e00640 1 --2- 172.21.15.31:0/2005751389 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90240ffe60 0x7f90241002e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:12.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.737+0000 7f9023e00640 1 --2- 172.21.15.31:0/2005751389 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9024106540 0x7f9024106920 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:12.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.737+0000 7f9023e00640 1 -- 172.21.15.31:0/2005751389 >> 172.21.15.31:0/2005751389 conn(0x7f90240756e0 msgr2=0x7f9024075af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:12.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.737+0000 7f9023e00640 1 -- 172.21.15.31:0/2005751389 shutdown_connections 2024-03-21T10:51:12.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.737+0000 7f9023e00640 1 -- 172.21.15.31:0/2005751389 wait complete. 2024-03-21T10:51:12.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.739+0000 7f9023e00640 1 Processor -- start 2024-03-21T10:51:12.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.739+0000 7f9023e00640 1 -- start start 2024-03-21T10:51:12.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.739+0000 7f9023e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90240ffe60 0x7f9024197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:12.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.740+0000 7f9023e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9024100820 0x7f9024198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:12.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.740+0000 7f9023e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9024106540 0x7f902419c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.740+0000 7f9023e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9024111a10 con 0x7f90240ffe60 2024-03-21T10:51:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.740+0000 7f9023e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9024111890 con 0x7f9024100820 2024-03-21T10:51:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.740+0000 7f9023e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9024111b90 con 0x7f9024106540 2024-03-21T10:51:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.740+0000 7f9023400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9024106540 0x7f902419c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.740+0000 7f9022000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9024100820 0x7f9024198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.740+0000 7f9022a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90240ffe60 0x7f9024197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.740+0000 7f9023400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9024106540 0x7f902419c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:40958/0 (socket says 172.21.15.31:40958) 2024-03-21T10:51:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.740+0000 7f9022000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9024100820 0x7f9024198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:45086/0 (socket says 172.21.15.31:45086) 2024-03-21T10:51:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.740+0000 7f9023400640 1 -- 172.21.15.31:0/2136420872 learned_addr learned my addr 172.21.15.31:0/2136420872 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:51:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.740+0000 7f9023400640 1 -- 172.21.15.31:0/2136420872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9024100820 msgr2=0x7f9024198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.740+0000 7f9023400640 1 --2- 172.21.15.31:0/2136420872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9024100820 0x7f9024198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.740+0000 7f9023400640 1 -- 172.21.15.31:0/2136420872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90240ffe60 msgr2=0x7f9024197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.740+0000 7f9023400640 1 --2- 172.21.15.31:0/2136420872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90240ffe60 0x7f9024197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:12.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.740+0000 7f9023400640 1 -- 172.21.15.31:0/2136420872 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902419ceb0 con 0x7f9024106540 2024-03-21T10:51:12.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.741+0000 7f9023400640 1 --2- 172.21.15.31:0/2136420872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9024106540 0x7f902419c780 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f901c00efc0 tx=0x7f901c00c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:12.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.741+0000 7f9028a00640 1 -- 172.21.15.31:0/2136420872 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901c003970 con 0x7f9024106540 2024-03-21T10:51:12.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.741+0000 7f9028a00640 1 -- 172.21.15.31:0/2136420872 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f901c004020 con 0x7f9024106540 2024-03-21T10:51:12.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.741+0000 7f9023e00640 1 -- 172.21.15.31:0/2136420872 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f902419d1a0 con 0x7f9024106540 2024-03-21T10:51:12.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.741+0000 7f9028a00640 1 -- 172.21.15.31:0/2136420872 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901c00b9f0 con 0x7f9024106540 2024-03-21T10:51:12.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.741+0000 7f9023e00640 1 -- 172.21.15.31:0/2136420872 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90241a4a30 con 0x7f9024106540 2024-03-21T10:51:12.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.742+0000 7f9023e00640 1 -- 172.21.15.31:0/2136420872 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f902404f470 con 0x7f9024106540 2024-03-21T10:51:12.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.745+0000 7f9028a00640 1 -- 172.21.15.31:0/2136420872 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f901c01d020 con 0x7f9024106540 2024-03-21T10:51:12.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.746+0000 7f9028a00640 1 --2- 172.21.15.31:0/2136420872 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f90080a3080 0x7f90080a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:12.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.746+0000 7f9028a00640 1 -- 172.21.15.31:0/2136420872 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f901c0cfaf0 con 0x7f9024106540 2024-03-21T10:51:12.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.746+0000 7f9022a00640 1 --2- 172.21.15.31:0/2136420872 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f90080a3080 0x7f90080a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:12.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.747+0000 7f9022a00640 1 --2- 172.21.15.31:0/2136420872 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f90080a3080 0x7f90080a5540 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f901402c180 tx=0x7f90140056d0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:12.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.750+0000 7f9028a00640 1 -- 172.21.15.31:0/2136420872 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f901c09c190 con 0x7f9024106540 2024-03-21T10:51:12.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.955+0000 7f9023e00640 1 -- 172.21.15.31:0/2136420872 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9024109a20 con 0x7f9024106540 2024-03-21T10:51:12.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.957+0000 7f9028a00640 1 -- 172.21.15.31:0/2136420872 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+25252 (secure 0 0 0) 0x7f901c09ba90 con 0x7f9024106540 2024-03-21T10:51:12.954 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:51:12.954 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":107,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","created":"2024-03-21T10:25:41.971117+0000","modified":"2024-03-21T10:51:07.072213+0000","last_up_change":"2024-03-21T10:34:07.757196+0000","last_in_change":"2024-03-21T10:33:47.460406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T10:29:44.553968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T10:35:04.945449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T10:35:05.970589+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":21,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T10:35:09.688711+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":21,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e0244d33-00e2-47d5-9f66-34efc138c462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6803","nonce":3089885125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6805","nonce":3089885125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6809","nonce":3089885125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6807","nonce":3089885125}]},"public_addr":"172.21.15.31:6803/3089885125","cluster_addr":"172.21.15.31:6805/3089885125","heartbeat_back_addr":"172.21.15.31:6809/3089885125","heartbeat_front_addr":"172.21.15.31:6807/3089885125","state":["exists","up"]},{"osd":1,"uuid":"76a63504-beab-4354-a6c3-eebc3b16ad03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6801","nonce":618840037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6803","nonce":618840037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6807","nonce":618840037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6805","nonce":618840037}]},"public_addr":"172.21.15.40:6801/618840037","cluster_addr":"172.21.15.40:6803/618840037","heartbeat_back_addr":"172.21.15.40:6807/618840037","heartbeat_front_addr":"172.21.15.40:6805/618840037","state":["exists","up"]},{"osd":2,"uuid":"fa191bd6-9a21-4ee2-8adf-8a63c9cfd821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6801","nonce":2524184117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6803","nonce":2524184117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6807","nonce":2524184117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6805","nonce":2524184117}]},"public_addr":"172.21.15.70:6801/2524184117","cluster_addr":"172.21.15.70:6803/2524184117","heartbeat_back_addr":"172.21.15.70:6807/2524184117","heartbeat_front_addr":"172.21.15.70:6805/2524184117","state":["exists","up"]},{"osd":3,"uuid":"edb44507-ffb0-42a4-827b-08461e1bf952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6811","nonce":2823036296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6813","nonce":2823036296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6817","nonce":2823036296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6815","nonce":2823036296}]},"public_addr":"172.21.15.31:6811/2823036296","cluster_addr":"172.21.15.31:6813/2823036296","heartbeat_back_addr":"172.21.15.31:6817/2823036296","heartbeat_front_addr":"172.21.15.31:6815/2823036296","state":["exists","up"]},{"osd":4,"uuid":"8cb89153-8c3b-431f-b57c-8863fada76dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6809","nonce":4215186309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6811","nonce":4215186309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6815","nonce":4215186309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6813","nonce":4215186309}]},"public_addr":"172.21.15.40:6809/4215186309","cluster_addr":"172.21.15.40:6811/4215186309","heartbeat_back_addr":"172.21.15.40:6815/4215186309","heartbeat_front_addr":"172.21.15.40:6813/4215186309","state":["exists","up"]},{"osd":5,"uuid":"9409505a-d463-408c-a3cd-218380c04d68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6809","nonce":3402830452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6811","nonce":3402830452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6815","nonce":3402830452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6813","nonce":3402830452}]},"public_addr":"172.21.15.70:6809/3402830452","cluster_addr":"172.21.15.70:6811/3402830452","heartbeat_back_addr":"172.21.15.70:6815/3402830452","heartbeat_front_addr":"172.21.15.70:6813/3402830452","state":["exists","up"]},{"osd":6,"uuid":"f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6819","nonce":1193497058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6821","nonce":1193497058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6825","nonce":1193497058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6823","nonce":1193497058}]},"public_addr":"172.21.15.31:6819/1193497058","cluster_addr":"172.21.15.31:6821/1193497058","heartbeat_back_addr":"172.21.15.31:6825/1193497058","heartbeat_front_addr":"172.21.15.31:6823/1193497058","state":["exists","up"]},{"osd":7,"uuid":"db228a87-29a5-45d4-8fe6-a0a554262eb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6817","nonce":3825628119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6819","nonce":3825628119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6823","nonce":3825628119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6821","nonce":3825628119}]},"public_addr":"172.21.15.40:6817/3825628119","cluster_addr":"172.21.15.40:6819/3825628119","heartbeat_back_addr":"172.21.15.40:6823/3825628119","heartbeat_front_addr":"172.21.15.40:6821/3825628119","state":["exists","up"]},{"osd":8,"uuid":"59b9fde0-095d-4e0b-87da-ead37bd8ff1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6817","nonce":534880586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6819","nonce":534880586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6823","nonce":534880586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6821","nonce":534880586}]},"public_addr":"172.21.15.70:6817/534880586","cluster_addr":"172.21.15.70:6819/534880586","heartbeat_back_addr":"172.21.15.70:6823/534880586","heartbeat_front_addr":"172.21.15.70:6821/534880586","state":["exists","up"]},{"osd":9,"uuid":"77f47061-0c5f-4159-b276-e814935d6c1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6827","nonce":587047469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6829","nonce":587047469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6833","nonce":587047469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6831","nonce":587047469}]},"public_addr":"172.21.15.31:6827/587047469","cluster_addr":"172.21.15.31:6829/587047469","heartbeat_back_addr":"172.21.15.31:6833/587047469","heartbeat_front_addr":"172.21.15.31:6831/587047469","state":["exists","up"]},{"osd":10,"uuid":"8853dc43-7bd1-4c93-8063-f7982fd7da38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6825","nonce":512108711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6827","nonce":512108711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6831","nonce":512108711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6829","nonce":512108711}]},"public_addr":"172.21.15.40:6825/512108711","cluster_addr":"172.21.15.40:6827/512108711","heartbeat_back_addr":"172.21.15.40:6831/512108711","heartbeat_front_addr":"172.21.15.40:6829/512108711","state":["exists","up"]},{"osd":11,"uuid":"84b7c2de-c424-45a6-9583-60097e372a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6825","nonce":540305718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6827","nonce":540305718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6831","nonce":540305718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6829","nonce":540305718}]},"public_addr":"172.21.15.70:6825/540305718","cluster_addr":"172.21.15.70:6827/540305718","heartbeat_back_addr":"172.21.15.70:6831/540305718","heartbeat_front_addr":"172.21.15.70:6829/540305718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:28:51.477358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:17.405045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:41.626268+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:15.339568+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:42.676996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:07.516999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:42.412273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:10.365041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:35.699120+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:11.733639+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:39.505442+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:34:05.449301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:6800/3875791078":"2024-03-22T10:38:33.502014+0000","172.21.15.31:6801/3875791078":"2024-03-22T10:38:33.502014+0000","172.21.15.31:0/3488063077":"2024-03-22T10:38:33.502014+0000","172.21.15.31:0/1451930554":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/3633519331":"2024-03-22T10:38:33.502014+0000","172.21.15.31:0/3342588843":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/1729503305":"2024-03-22T10:38:33.502014+0000","172.21.15.31:6801/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/2171929076":"2024-03-22T10:38:33.502014+0000","172.21.15.31:0/1626448389":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/1910059351":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/344822770":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/3811368512":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/1981437651":"2024-03-22T10:26:16.959165+0000","172.21.15.31:6801/1981437651":"2024-03-22T10:26:16.959165+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]},{"pool":4,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]},{"pool":4,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T10:51:12.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.960+0000 7f9023e00640 1 -- 172.21.15.31:0/2136420872 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f90080a3080 msgr2=0x7f90080a5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:12.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.960+0000 7f9023e00640 1 --2- 172.21.15.31:0/2136420872 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f90080a3080 0x7f90080a5540 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f901402c180 tx=0x7f90140056d0 comp rx=0 tx=0).stop 2024-03-21T10:51:12.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.960+0000 7f9023e00640 1 -- 172.21.15.31:0/2136420872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9024106540 msgr2=0x7f902419c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:12.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.960+0000 7f9023e00640 1 --2- 172.21.15.31:0/2136420872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9024106540 0x7f902419c780 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f901c00efc0 tx=0x7f901c00c800 comp rx=0 tx=0).stop 2024-03-21T10:51:12.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.960+0000 7f9023e00640 1 -- 172.21.15.31:0/2136420872 shutdown_connections 2024-03-21T10:51:12.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.960+0000 7f9023e00640 1 --2- 172.21.15.31:0/2136420872 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f90080a3080 0x7f90080a5540 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:12.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.960+0000 7f9023e00640 1 --2- 172.21.15.31:0/2136420872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9024106540 0x7f902419c780 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:12.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.960+0000 7f9023e00640 1 --2- 172.21.15.31:0/2136420872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9024100820 0x7f9024198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:12.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.960+0000 7f9023e00640 1 --2- 172.21.15.31:0/2136420872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90240ffe60 0x7f9024197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:12.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.960+0000 7f9023e00640 1 -- 172.21.15.31:0/2136420872 >> 172.21.15.31:0/2136420872 conn(0x7f90240756e0 msgr2=0x7f90240fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:12.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.960+0000 7f9023e00640 1 -- 172.21.15.31:0/2136420872 shutdown_connections 2024-03-21T10:51:12.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:12.960+0000 7f9023e00640 1 -- 172.21.15.31:0/2136420872 wait complete. 2024-03-21T10:51:12.980 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T10:29:44.553968+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T10:35:04.945449+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T10:35:05.970589+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 21, 'snap_epoch': 107, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T10:35:09.688711+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 21, 'snap_epoch': 107, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T10:51:12.981 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T10:51:13.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.211+0000 7f8433e00640 1 -- 172.21.15.31:0/73757442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8434102f50 msgr2=0x7f843410b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:13.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.211+0000 7f8433e00640 1 --2- 172.21.15.31:0/73757442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8434102f50 0x7f843410b430 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f841c009900 tx=0x7f841c02bcc0 comp rx=0 tx=0).stop 2024-03-21T10:51:13.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.211+0000 7f8433e00640 1 -- 172.21.15.31:0/73757442 shutdown_connections 2024-03-21T10:51:13.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.211+0000 7f8433e00640 1 --2- 172.21.15.31:0/73757442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f843410ba50 0x7f843410ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:13.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.211+0000 7f8433e00640 1 --2- 172.21.15.31:0/73757442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8434102f50 0x7f843410b430 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:13.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.211+0000 7f8433e00640 1 --2- 172.21.15.31:0/73757442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8434102630 0x7f8434102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:13.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.211+0000 7f8433e00640 1 -- 172.21.15.31:0/73757442 >> 172.21.15.31:0/73757442 conn(0x7f8434077130 msgr2=0x7f84340fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:13.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.211+0000 7f8433e00640 1 -- 172.21.15.31:0/73757442 shutdown_connections 2024-03-21T10:51:13.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.211+0000 7f8433e00640 1 -- 172.21.15.31:0/73757442 wait complete. 2024-03-21T10:51:13.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.212+0000 7f8433e00640 1 Processor -- start 2024-03-21T10:51:13.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.212+0000 7f8433e00640 1 -- start start 2024-03-21T10:51:13.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.213+0000 7f8433e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8434102630 0x7f8434199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:13.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.213+0000 7f8433e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8434102f50 0x7f843419a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:13.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.213+0000 7f8433e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f843410ba50 0x7f843419e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:13.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.213+0000 7f8433e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8434111850 con 0x7f8434102f50 2024-03-21T10:51:13.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.213+0000 7f8433e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84341116d0 con 0x7f8434102630 2024-03-21T10:51:13.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.213+0000 7f8433e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84341119d0 con 0x7f843410ba50 2024-03-21T10:51:13.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.213+0000 7f8432a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8434102630 0x7f8434199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:13.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.213+0000 7f842be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8434102f50 0x7f843419a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:13.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.213+0000 7f8432a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8434102630 0x7f8434199fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:45102/0 (socket says 172.21.15.31:45102) 2024-03-21T10:51:13.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.213+0000 7f8432a00640 1 -- 172.21.15.31:0/462741567 learned_addr learned my addr 172.21.15.31:0/462741567 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:51:13.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.213+0000 7f8433400640 1 --2- 172.21.15.31:0/462741567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f843410ba50 0x7f843419e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:13.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.213+0000 7f8432a00640 1 -- 172.21.15.31:0/462741567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f843410ba50 msgr2=0x7f843419e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:13.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.213+0000 7f8432a00640 1 --2- 172.21.15.31:0/462741567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f843410ba50 0x7f843419e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:13.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.213+0000 7f8432a00640 1 -- 172.21.15.31:0/462741567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8434102f50 msgr2=0x7f843419a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:13.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.213+0000 7f8432a00640 1 --2- 172.21.15.31:0/462741567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8434102f50 0x7f843419a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:13.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.213+0000 7f8432a00640 1 -- 172.21.15.31:0/462741567 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843419efa0 con 0x7f8434102630 2024-03-21T10:51:13.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.214+0000 7f8432a00640 1 --2- 172.21.15.31:0/462741567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8434102630 0x7f8434199fb0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f842400d950 tx=0x7f842400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:13.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.214+0000 7f842b400640 1 -- 172.21.15.31:0/462741567 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8424017070 con 0x7f8434102630 2024-03-21T10:51:13.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.214+0000 7f8433e00640 1 -- 172.21.15.31:0/462741567 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f843419f290 con 0x7f8434102630 2024-03-21T10:51:13.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.214+0000 7f842b400640 1 -- 172.21.15.31:0/462741567 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8424009c30 con 0x7f8434102630 2024-03-21T10:51:13.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.214+0000 7f842b400640 1 -- 172.21.15.31:0/462741567 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842400f6b0 con 0x7f8434102630 2024-03-21T10:51:13.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.214+0000 7f8433e00640 1 -- 172.21.15.31:0/462741567 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84341a6b20 con 0x7f8434102630 2024-03-21T10:51:13.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.216+0000 7f8433e00640 1 -- 172.21.15.31:0/462741567 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f843404f400 con 0x7f8434102630 2024-03-21T10:51:13.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.217+0000 7f842b400640 1 -- 172.21.15.31:0/462741567 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f84240109d0 con 0x7f8434102630 2024-03-21T10:51:13.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.218+0000 7f842b400640 1 --2- 172.21.15.31:0/462741567 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f84180a3110 0x7f84180a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:13.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.219+0000 7f842b400640 1 -- 172.21.15.31:0/462741567 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9534+0+0 (secure 0 0 0) 0x7f84240cf770 con 0x7f8434102630 2024-03-21T10:51:13.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.219+0000 7f842be00640 1 --2- 172.21.15.31:0/462741567 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f84180a3110 0x7f84180a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:13.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.220+0000 7f842be00640 1 --2- 172.21.15.31:0/462741567 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f84180a3110 0x7f84180a55d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f841c005ab0 tx=0x7f841c005a20 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:13.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.222+0000 7f842b400640 1 -- 172.21.15.31:0/462741567 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8424009de0 con 0x7f8434102630 2024-03-21T10:51:13.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:13 smithi031 ceph-mon[31673]: pgmap v395: 129 pgs: 15 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 75 active+clean; 13 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-21T10:51:13.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:13 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2136420872' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:51:13.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.427+0000 7f8433e00640 1 -- 172.21.15.31:0/462741567 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f84341a6cf0 con 0x7f8434102630 2024-03-21T10:51:13.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.428+0000 7f842b400640 1 -- 172.21.15.31:0/462741567 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) v1 ==== 93+0+10 (secure 0 0 0) 0x7f842409b7c0 con 0x7f8434102630 2024-03-21T10:51:13.425 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-03-21T10:51:13.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.431+0000 7f8433e00640 1 -- 172.21.15.31:0/462741567 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f84180a3110 msgr2=0x7f84180a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:13.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.431+0000 7f8433e00640 1 --2- 172.21.15.31:0/462741567 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f84180a3110 0x7f84180a55d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f841c005ab0 tx=0x7f841c005a20 comp rx=0 tx=0).stop 2024-03-21T10:51:13.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.431+0000 7f8433e00640 1 -- 172.21.15.31:0/462741567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8434102630 msgr2=0x7f8434199fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:13.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.431+0000 7f8433e00640 1 --2- 172.21.15.31:0/462741567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8434102630 0x7f8434199fb0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f842400d950 tx=0x7f842400de10 comp rx=0 tx=0).stop 2024-03-21T10:51:13.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.431+0000 7f8433e00640 1 -- 172.21.15.31:0/462741567 shutdown_connections 2024-03-21T10:51:13.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.431+0000 7f8433e00640 1 --2- 172.21.15.31:0/462741567 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f84180a3110 0x7f84180a55d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:13.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.431+0000 7f8433e00640 1 --2- 172.21.15.31:0/462741567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f843410ba50 0x7f843419e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:13.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.431+0000 7f8433e00640 1 --2- 172.21.15.31:0/462741567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8434102f50 0x7f843419a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:13.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.431+0000 7f8433e00640 1 --2- 172.21.15.31:0/462741567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8434102630 0x7f8434199fb0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:13.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.431+0000 7f8433e00640 1 -- 172.21.15.31:0/462741567 >> 172.21.15.31:0/462741567 conn(0x7f8434077130 msgr2=0x7f8434100250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:13.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.431+0000 7f8433e00640 1 -- 172.21.15.31:0/462741567 shutdown_connections 2024-03-21T10:51:13.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.431+0000 7f8433e00640 1 -- 172.21.15.31:0/462741567 wait complete. 2024-03-21T10:51:13.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:13 smithi040 ceph-mon[36154]: pgmap v395: 129 pgs: 15 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 75 active+clean; 13 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-21T10:51:13.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:13 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2136420872' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:51:13.447 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T10:51:13.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:13 smithi070 ceph-mon[36330]: pgmap v395: 129 pgs: 15 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 75 active+clean; 13 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-21T10:51:13.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:13 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2136420872' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:51:13.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.661+0000 7efcbd600640 1 -- 172.21.15.31:0/862566648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcb8105380 msgr2=0x7efcb81057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:13.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.661+0000 7efcbd600640 1 --2- 172.21.15.31:0/862566648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcb8105380 0x7efcb81057e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7efca4009890 tx=0x7efca402bc40 comp rx=0 tx=0).stop 2024-03-21T10:51:13.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.661+0000 7efcbd600640 1 -- 172.21.15.31:0/862566648 shutdown_connections 2024-03-21T10:51:13.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.661+0000 7efcbd600640 1 --2- 172.21.15.31:0/862566648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb8105d20 0x7efcb8110850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:13.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.661+0000 7efcbd600640 1 --2- 172.21.15.31:0/862566648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcb8105380 0x7efcb81057e0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:13.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.661+0000 7efcbd600640 1 --2- 172.21.15.31:0/862566648 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcb8103c40 0x7efcb8104020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:13.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.661+0000 7efcbd600640 1 -- 172.21.15.31:0/862566648 >> 172.21.15.31:0/862566648 conn(0x7efcb80ff350 msgr2=0x7efcb8101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:13.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.661+0000 7efcbd600640 1 -- 172.21.15.31:0/862566648 shutdown_connections 2024-03-21T10:51:13.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.661+0000 7efcbd600640 1 -- 172.21.15.31:0/862566648 wait complete. 2024-03-21T10:51:13.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.662+0000 7efcbd600640 1 Processor -- start 2024-03-21T10:51:13.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.662+0000 7efcbd600640 1 -- start start 2024-03-21T10:51:13.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.663+0000 7efcbd600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcb8103c40 0x7efcb819a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:13.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.663+0000 7efcbd600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb8105380 0x7efcb819a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:13.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.663+0000 7efcbd600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcb8105d20 0x7efcb819eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:13.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.663+0000 7efcbd600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb8113d90 con 0x7efcb8105d20 2024-03-21T10:51:13.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.663+0000 7efcbd600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb8113c10 con 0x7efcb8105380 2024-03-21T10:51:13.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.663+0000 7efcbd600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb8113f10 con 0x7efcb8103c40 2024-03-21T10:51:13.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.663+0000 7efcb6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb8105380 0x7efcb819a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:13.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.663+0000 7efcb6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcb8103c40 0x7efcb819a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:13.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.663+0000 7efcb6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcb8103c40 0x7efcb819a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:40994/0 (socket says 172.21.15.31:40994) 2024-03-21T10:51:13.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.663+0000 7efcb6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb8105380 0x7efcb819a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:45130/0 (socket says 172.21.15.31:45130) 2024-03-21T10:51:13.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.663+0000 7efcb6a00640 1 -- 172.21.15.31:0/426711891 learned_addr learned my addr 172.21.15.31:0/426711891 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:51:13.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.663+0000 7efcb7400640 1 --2- 172.21.15.31:0/426711891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcb8105d20 0x7efcb819eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:13.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.663+0000 7efcb6000640 1 -- 172.21.15.31:0/426711891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcb8103c40 msgr2=0x7efcb819a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:13.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.663+0000 7efcb6000640 1 --2- 172.21.15.31:0/426711891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcb8103c40 0x7efcb819a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:13.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.663+0000 7efcb6000640 1 -- 172.21.15.31:0/426711891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcb8105d20 msgr2=0x7efcb819eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:13.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.663+0000 7efcb6000640 1 --2- 172.21.15.31:0/426711891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcb8105d20 0x7efcb819eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:13.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.663+0000 7efcb6000640 1 -- 172.21.15.31:0/426711891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb819f220 con 0x7efcb8105380 2024-03-21T10:51:13.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.663+0000 7efcb6a00640 1 --2- 172.21.15.31:0/426711891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcb8103c40 0x7efcb819a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:51:13.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.664+0000 7efcb6000640 1 --2- 172.21.15.31:0/426711891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb8105380 0x7efcb819a770 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7efca40057c0 tx=0x7efca40045d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:13.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.664+0000 7efcaf400640 1 -- 172.21.15.31:0/426711891 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca4041020 con 0x7efcb8105380 2024-03-21T10:51:13.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.664+0000 7efcaf400640 1 -- 172.21.15.31:0/426711891 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efca402ca60 con 0x7efcb8105380 2024-03-21T10:51:13.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.664+0000 7efcaf400640 1 -- 172.21.15.31:0/426711891 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca4039040 con 0x7efcb8105380 2024-03-21T10:51:13.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.664+0000 7efcbd600640 1 -- 172.21.15.31:0/426711891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb819f4b0 con 0x7efcb8105380 2024-03-21T10:51:13.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.664+0000 7efcbd600640 1 -- 172.21.15.31:0/426711891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb81a6da0 con 0x7efcb8105380 2024-03-21T10:51:13.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.666+0000 7efcbd600640 1 -- 172.21.15.31:0/426711891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb8068970 con 0x7efcb8105380 2024-03-21T10:51:13.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.668+0000 7efcaf400640 1 -- 172.21.15.31:0/426711891 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7efca4044020 con 0x7efcb8105380 2024-03-21T10:51:13.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.670+0000 7efcaf400640 1 --2- 172.21.15.31:0/426711891 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7efc9c0a3110 0x7efc9c0a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:13.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.670+0000 7efcaf400640 1 -- 172.21.15.31:0/426711891 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9534+0+0 (secure 0 0 0) 0x7efca40f0f50 con 0x7efcb8105380 2024-03-21T10:51:13.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.670+0000 7efcb6a00640 1 --2- 172.21.15.31:0/426711891 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7efc9c0a3110 0x7efc9c0a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:13.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.671+0000 7efcb6a00640 1 --2- 172.21.15.31:0/426711891 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7efc9c0a3110 0x7efc9c0a55d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7efca80096c0 tx=0x7efca8009260 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:13.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.673+0000 7efcaf400640 1 -- 172.21.15.31:0/426711891 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efca404b020 con 0x7efcb8105380 2024-03-21T10:51:13.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.886+0000 7efcbd600640 1 -- 172.21.15.31:0/426711891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efcb810bea0 con 0x7efcb8105380 2024-03-21T10:51:13.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.886+0000 7efcaf400640 1 -- 172.21.15.31:0/426711891 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v107) v1 ==== 104+0+11 (secure 0 0 0) 0x7efca40bd700 con 0x7efcb8105380 2024-03-21T10:51:13.883 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-03-21T10:51:13.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.889+0000 7efcbd600640 1 -- 172.21.15.31:0/426711891 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7efc9c0a3110 msgr2=0x7efc9c0a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:13.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.889+0000 7efcbd600640 1 --2- 172.21.15.31:0/426711891 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7efc9c0a3110 0x7efc9c0a55d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7efca80096c0 tx=0x7efca8009260 comp rx=0 tx=0).stop 2024-03-21T10:51:13.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.889+0000 7efcbd600640 1 -- 172.21.15.31:0/426711891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb8105380 msgr2=0x7efcb819a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:13.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.889+0000 7efcbd600640 1 --2- 172.21.15.31:0/426711891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb8105380 0x7efcb819a770 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7efca40057c0 tx=0x7efca40045d0 comp rx=0 tx=0).stop 2024-03-21T10:51:13.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.889+0000 7efcbd600640 1 -- 172.21.15.31:0/426711891 shutdown_connections 2024-03-21T10:51:13.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.889+0000 7efcbd600640 1 --2- 172.21.15.31:0/426711891 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7efc9c0a3110 0x7efc9c0a55d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:13.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.889+0000 7efcbd600640 1 --2- 172.21.15.31:0/426711891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcb8105d20 0x7efcb819eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:13.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.889+0000 7efcbd600640 1 --2- 172.21.15.31:0/426711891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb8105380 0x7efcb819a770 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:13.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.889+0000 7efcbd600640 1 --2- 172.21.15.31:0/426711891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efcb8103c40 0x7efcb819a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:13.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.889+0000 7efcbd600640 1 -- 172.21.15.31:0/426711891 >> 172.21.15.31:0/426711891 conn(0x7efcb80ff350 msgr2=0x7efcb8107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:13.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.890+0000 7efcbd600640 1 -- 172.21.15.31:0/426711891 shutdown_connections 2024-03-21T10:51:13.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:13.890+0000 7efcbd600640 1 -- 172.21.15.31:0/426711891 wait complete. 2024-03-21T10:51:13.906 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T10:51:14.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.156+0000 7fe49b800640 1 -- 172.21.15.31:0/1218579555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe494104220 msgr2=0x7fe494110730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:14.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.156+0000 7fe49b800640 1 --2- 172.21.15.31:0/1218579555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe494104220 0x7fe494110730 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fe48c009cd0 tx=0x7fe48c02c0f0 comp rx=0 tx=0).stop 2024-03-21T10:51:14.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.156+0000 7fe49b800640 1 -- 172.21.15.31:0/1218579555 shutdown_connections 2024-03-21T10:51:14.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.156+0000 7fe49b800640 1 --2- 172.21.15.31:0/1218579555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe494104220 0x7fe494110730 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:14.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.156+0000 7fe49b800640 1 --2- 172.21.15.31:0/1218579555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe494103860 0x7fe494103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:14.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.156+0000 7fe49b800640 1 --2- 172.21.15.31:0/1218579555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe494109870 0x7fe494109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:14.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.156+0000 7fe49b800640 1 -- 172.21.15.31:0/1218579555 >> 172.21.15.31:0/1218579555 conn(0x7fe4940ff310 msgr2=0x7fe494101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:14.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.157+0000 7fe49b800640 1 -- 172.21.15.31:0/1218579555 shutdown_connections 2024-03-21T10:51:14.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.157+0000 7fe49b800640 1 -- 172.21.15.31:0/1218579555 wait complete. 2024-03-21T10:51:14.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.158+0000 7fe49b800640 1 Processor -- start 2024-03-21T10:51:14.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.158+0000 7fe49b800640 1 -- start start 2024-03-21T10:51:14.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.159+0000 7fe49b800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe494103860 0x7fe49419a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:14.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.159+0000 7fe49b800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe494104220 0x7fe49419a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:14.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.159+0000 7fe49b800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe494109870 0x7fe49419e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:14.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.159+0000 7fe49b800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe494113c80 con 0x7fe494103860 2024-03-21T10:51:14.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.159+0000 7fe49b800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe494113b00 con 0x7fe494109870 2024-03-21T10:51:14.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.159+0000 7fe49b800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe494113e00 con 0x7fe494104220 2024-03-21T10:51:14.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.159+0000 7fe498e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe494103860 0x7fe49419a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:14.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.159+0000 7fe493e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe494104220 0x7fe49419a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:14.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.159+0000 7fe498e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe494103860 0x7fe49419a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53408/0 (socket says 172.21.15.31:53408) 2024-03-21T10:51:14.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.159+0000 7fe498e00640 1 -- 172.21.15.31:0/1906315008 learned_addr learned my addr 172.21.15.31:0/1906315008 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:51:14.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.159+0000 7fe499800640 1 --2- 172.21.15.31:0/1906315008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe494109870 0x7fe49419e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:14.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.160+0000 7fe493e00640 1 -- 172.21.15.31:0/1906315008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe494109870 msgr2=0x7fe49419e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:14.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.160+0000 7fe493e00640 1 --2- 172.21.15.31:0/1906315008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe494109870 0x7fe49419e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:14.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.160+0000 7fe493e00640 1 -- 172.21.15.31:0/1906315008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe494103860 msgr2=0x7fe49419a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:14.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.160+0000 7fe493e00640 1 --2- 172.21.15.31:0/1906315008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe494103860 0x7fe49419a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:14.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.160+0000 7fe493e00640 1 -- 172.21.15.31:0/1906315008 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe49419f110 con 0x7fe494104220 2024-03-21T10:51:14.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.160+0000 7fe493e00640 1 --2- 172.21.15.31:0/1906315008 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe494104220 0x7fe49419a660 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fe48400e9f0 tx=0x7fe48400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:14.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.160+0000 7fe491600640 1 -- 172.21.15.31:0/1906315008 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48400ccc0 con 0x7fe494104220 2024-03-21T10:51:14.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.160+0000 7fe49b800640 1 -- 172.21.15.31:0/1906315008 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe49419f400 con 0x7fe494104220 2024-03-21T10:51:14.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.160+0000 7fe491600640 1 -- 172.21.15.31:0/1906315008 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe484005020 con 0x7fe494104220 2024-03-21T10:51:14.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.160+0000 7fe491600640 1 -- 172.21.15.31:0/1906315008 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe484004b20 con 0x7fe494104220 2024-03-21T10:51:14.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.161+0000 7fe49b800640 1 -- 172.21.15.31:0/1906315008 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4941a6c90 con 0x7fe494104220 2024-03-21T10:51:14.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.162+0000 7fe49b800640 1 -- 172.21.15.31:0/1906315008 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe494068970 con 0x7fe494104220 2024-03-21T10:51:14.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.164+0000 7fe491600640 1 -- 172.21.15.31:0/1906315008 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fe484004cc0 con 0x7fe494104220 2024-03-21T10:51:14.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.165+0000 7fe491600640 1 --2- 172.21.15.31:0/1906315008 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe47c0a3110 0x7fe47c0a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:14.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.165+0000 7fe491600640 1 -- 172.21.15.31:0/1906315008 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9534+0+0 (secure 0 0 0) 0x7fe4840cfb00 con 0x7fe494104220 2024-03-21T10:51:14.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.166+0000 7fe498e00640 1 --2- 172.21.15.31:0/1906315008 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe47c0a3110 0x7fe47c0a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:14.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.166+0000 7fe498e00640 1 --2- 172.21.15.31:0/1906315008 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe47c0a3110 0x7fe47c0a55d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe488005fd0 tx=0x7fe48800b480 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:14.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.169+0000 7fe491600640 1 -- 172.21.15.31:0/1906315008 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe48409c1a0 con 0x7fe494104220 2024-03-21T10:51:14.303 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:14 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/462741567' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:51:14.303 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:14 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:51:14.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:14 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/426711891' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:51:14.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:14 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/462741567' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:51:14.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:14 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:51:14.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:14 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/426711891' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:51:14.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.440+0000 7fe49b800640 1 -- 172.21.15.31:0/1906315008 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe49402d070 con 0x7fe494104220 2024-03-21T10:51:14.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.441+0000 7fe491600640 1 -- 172.21.15.31:0/1906315008 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v107) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe48409baa0 con 0x7fe494104220 2024-03-21T10:51:14.437 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-03-21T10:51:14.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.443+0000 7fe49b800640 1 -- 172.21.15.31:0/1906315008 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe47c0a3110 msgr2=0x7fe47c0a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:14.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.443+0000 7fe49b800640 1 --2- 172.21.15.31:0/1906315008 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe47c0a3110 0x7fe47c0a55d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe488005fd0 tx=0x7fe48800b480 comp rx=0 tx=0).stop 2024-03-21T10:51:14.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.443+0000 7fe49b800640 1 -- 172.21.15.31:0/1906315008 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe494104220 msgr2=0x7fe49419a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:14.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.443+0000 7fe49b800640 1 --2- 172.21.15.31:0/1906315008 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe494104220 0x7fe49419a660 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fe48400e9f0 tx=0x7fe48400eeb0 comp rx=0 tx=0).stop 2024-03-21T10:51:14.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.443+0000 7fe49b800640 1 -- 172.21.15.31:0/1906315008 shutdown_connections 2024-03-21T10:51:14.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.443+0000 7fe49b800640 1 --2- 172.21.15.31:0/1906315008 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fe47c0a3110 0x7fe47c0a55d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:14.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.443+0000 7fe49b800640 1 --2- 172.21.15.31:0/1906315008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe494109870 0x7fe49419e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:14.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.443+0000 7fe49b800640 1 --2- 172.21.15.31:0/1906315008 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe494104220 0x7fe49419a660 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:14.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.443+0000 7fe49b800640 1 --2- 172.21.15.31:0/1906315008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe494103860 0x7fe49419a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:14.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.443+0000 7fe49b800640 1 -- 172.21.15.31:0/1906315008 >> 172.21.15.31:0/1906315008 conn(0x7fe4940ff310 msgr2=0x7fe4940ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:14.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.444+0000 7fe49b800640 1 -- 172.21.15.31:0/1906315008 shutdown_connections 2024-03-21T10:51:14.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.444+0000 7fe49b800640 1 -- 172.21.15.31:0/1906315008 wait complete. 2024-03-21T10:51:14.460 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T10:51:14.519 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:14 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/462741567' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T10:51:14.519 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:14 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T10:51:14.519 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:14 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/426711891' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-21T10:51:14.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.664+0000 7feac7200640 1 -- 172.21.15.31:0/3118686154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feac01028c0 msgr2=0x7feac0102ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:14.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.664+0000 7feac7200640 1 --2- 172.21.15.31:0/3118686154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feac01028c0 0x7feac0102ca0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7feab0009900 tx=0x7feab002bcd0 comp rx=0 tx=0).stop 2024-03-21T10:51:14.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.664+0000 7feac7200640 1 -- 172.21.15.31:0/3118686154 shutdown_connections 2024-03-21T10:51:14.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.664+0000 7feac7200640 1 --2- 172.21.15.31:0/3118686154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feac010bc90 0x7feac010e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:14.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.664+0000 7feac7200640 1 --2- 172.21.15.31:0/3118686154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feac01031e0 0x7feac010b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:14.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.664+0000 7feac7200640 1 --2- 172.21.15.31:0/3118686154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feac01028c0 0x7feac0102ca0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:14.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.664+0000 7feac7200640 1 -- 172.21.15.31:0/3118686154 >> 172.21.15.31:0/3118686154 conn(0x7feac00776a0 msgr2=0x7feac0077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:14.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.665+0000 7feac7200640 1 -- 172.21.15.31:0/3118686154 shutdown_connections 2024-03-21T10:51:14.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.665+0000 7feac7200640 1 -- 172.21.15.31:0/3118686154 wait complete. 2024-03-21T10:51:14.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.666+0000 7feac7200640 1 Processor -- start 2024-03-21T10:51:14.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.666+0000 7feac7200640 1 -- start start 2024-03-21T10:51:14.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.667+0000 7feac7200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feac01028c0 0x7feac0197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:14.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.667+0000 7feac7200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feac01031e0 0x7feac0198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:14.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.667+0000 7feac7200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feac010bc90 0x7feac019c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:14.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.667+0000 7feac7200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac0111a70 con 0x7feac01028c0 2024-03-21T10:51:14.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.667+0000 7feac7200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac01118f0 con 0x7feac010bc90 2024-03-21T10:51:14.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.667+0000 7feac7200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac0111bf0 con 0x7feac01031e0 2024-03-21T10:51:14.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.667+0000 7feabf400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feac01031e0 0x7feac0198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:14.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.667+0000 7feabf400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feac01031e0 0x7feac0198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:41018/0 (socket says 172.21.15.31:41018) 2024-03-21T10:51:14.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.667+0000 7feabf400640 1 -- 172.21.15.31:0/1898975082 learned_addr learned my addr 172.21.15.31:0/1898975082 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:51:14.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.667+0000 7feabfe00640 1 --2- 172.21.15.31:0/1898975082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feac01028c0 0x7feac0197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:14.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.667+0000 7feac5200640 1 --2- 172.21.15.31:0/1898975082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feac010bc90 0x7feac019c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:14.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.668+0000 7feabf400640 1 -- 172.21.15.31:0/1898975082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feac010bc90 msgr2=0x7feac019c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:14.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.668+0000 7feabf400640 1 --2- 172.21.15.31:0/1898975082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feac010bc90 0x7feac019c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:14.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.668+0000 7feabf400640 1 -- 172.21.15.31:0/1898975082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feac01028c0 msgr2=0x7feac0197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:14.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.668+0000 7feabf400640 1 --2- 172.21.15.31:0/1898975082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feac01028c0 0x7feac0197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:14.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.668+0000 7feabf400640 1 -- 172.21.15.31:0/1898975082 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac019cf00 con 0x7feac01031e0 2024-03-21T10:51:14.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.668+0000 7feac5200640 1 --2- 172.21.15.31:0/1898975082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feac010bc90 0x7feac019c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:51:14.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.668+0000 7feabfe00640 1 --2- 172.21.15.31:0/1898975082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feac01028c0 0x7feac0197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:51:14.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.668+0000 7feabf400640 1 --2- 172.21.15.31:0/1898975082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feac01031e0 0x7feac0198450 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7feaac00e9f0 tx=0x7feaac00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:14.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.668+0000 7feabcc00640 1 -- 172.21.15.31:0/1898975082 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac00cd80 con 0x7feac01031e0 2024-03-21T10:51:14.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.668+0000 7feabcc00640 1 -- 172.21.15.31:0/1898975082 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feaac010910 con 0x7feac01031e0 2024-03-21T10:51:14.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.668+0000 7feac7200640 1 -- 172.21.15.31:0/1898975082 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac019d1f0 con 0x7feac01031e0 2024-03-21T10:51:14.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.668+0000 7feabcc00640 1 -- 172.21.15.31:0/1898975082 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac018990 con 0x7feac01031e0 2024-03-21T10:51:14.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.669+0000 7feac7200640 1 -- 172.21.15.31:0/1898975082 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac01a4a80 con 0x7feac01031e0 2024-03-21T10:51:14.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.670+0000 7feac7200640 1 -- 172.21.15.31:0/1898975082 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac0068970 con 0x7feac01031e0 2024-03-21T10:51:14.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.675+0000 7feabcc00640 1 -- 172.21.15.31:0/1898975082 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7feaac010ac0 con 0x7feac01031e0 2024-03-21T10:51:14.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.676+0000 7feabcc00640 1 --2- 172.21.15.31:0/1898975082 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7feaa40a3110 0x7feaa40a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:14.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.677+0000 7feabcc00640 1 -- 172.21.15.31:0/1898975082 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9534+0+0 (secure 0 0 0) 0x7feaac0d0310 con 0x7feac01031e0 2024-03-21T10:51:14.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.677+0000 7feabfe00640 1 --2- 172.21.15.31:0/1898975082 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7feaa40a3110 0x7feaa40a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:14.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.677+0000 7feabcc00640 1 -- 172.21.15.31:0/1898975082 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7feaac0d0800 con 0x7feac01031e0 2024-03-21T10:51:14.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.678+0000 7feabfe00640 1 --2- 172.21.15.31:0/1898975082 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7feaa40a3110 0x7feaa40a55d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7feab0005ab0 tx=0x7feab0005a20 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:14.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.919+0000 7feac7200640 1 -- 172.21.15.31:0/1898975082 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7feac01067f0 con 0x7feac01031e0 2024-03-21T10:51:14.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.920+0000 7feabcc00640 1 -- 172.21.15.31:0/1898975082 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v107) v1 ==== 103+0+11 (secure 0 0 0) 0x7feaac09c3c0 con 0x7feac01031e0 2024-03-21T10:51:14.916 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-03-21T10:51:14.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.922+0000 7feac7200640 1 -- 172.21.15.31:0/1898975082 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7feaa40a3110 msgr2=0x7feaa40a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:14.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.922+0000 7feac7200640 1 --2- 172.21.15.31:0/1898975082 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7feaa40a3110 0x7feaa40a55d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7feab0005ab0 tx=0x7feab0005a20 comp rx=0 tx=0).stop 2024-03-21T10:51:14.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.922+0000 7feac7200640 1 -- 172.21.15.31:0/1898975082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feac01031e0 msgr2=0x7feac0198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:14.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.922+0000 7feac7200640 1 --2- 172.21.15.31:0/1898975082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feac01031e0 0x7feac0198450 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7feaac00e9f0 tx=0x7feaac00eeb0 comp rx=0 tx=0).stop 2024-03-21T10:51:14.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.923+0000 7feac7200640 1 -- 172.21.15.31:0/1898975082 shutdown_connections 2024-03-21T10:51:14.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.923+0000 7feac7200640 1 --2- 172.21.15.31:0/1898975082 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7feaa40a3110 0x7feaa40a55d0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:14.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.923+0000 7feac7200640 1 --2- 172.21.15.31:0/1898975082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feac010bc90 0x7feac019c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:14.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.923+0000 7feac7200640 1 --2- 172.21.15.31:0/1898975082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feac01031e0 0x7feac0198450 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:14.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.923+0000 7feac7200640 1 --2- 172.21.15.31:0/1898975082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feac01028c0 0x7feac0197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:14.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.923+0000 7feac7200640 1 -- 172.21.15.31:0/1898975082 >> 172.21.15.31:0/1898975082 conn(0x7feac00776a0 msgr2=0x7feac00fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:14.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.923+0000 7feac7200640 1 -- 172.21.15.31:0/1898975082 shutdown_connections 2024-03-21T10:51:14.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:14.923+0000 7feac7200640 1 -- 172.21.15.31:0/1898975082 wait complete. 2024-03-21T10:51:14.945 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-21T10:51:15.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.157+0000 7fa40f000640 1 -- 172.21.15.31:0/3466682386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa408109b70 msgr2=0x7fa408109f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:15.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.157+0000 7fa40f000640 1 --2- 172.21.15.31:0/3466682386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa408109b70 0x7fa408109f50 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8009930 tx=0x7fa3f802bc20 comp rx=0 tx=0).stop 2024-03-21T10:51:15.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.157+0000 7fa40f000640 1 -- 172.21.15.31:0/3466682386 shutdown_connections 2024-03-21T10:51:15.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.157+0000 7fa40f000640 1 --2- 172.21.15.31:0/3466682386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa408108760 0x7fa408114c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:15.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.157+0000 7fa40f000640 1 --2- 172.21.15.31:0/3466682386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa408107dc0 0x7fa408108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:15.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.157+0000 7fa40f000640 1 --2- 172.21.15.31:0/3466682386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa408109b70 0x7fa408109f50 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:15.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.157+0000 7fa40f000640 1 -- 172.21.15.31:0/3466682386 >> 172.21.15.31:0/3466682386 conn(0x7fa408077540 msgr2=0x7fa4081064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:15.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.157+0000 7fa40f000640 1 -- 172.21.15.31:0/3466682386 shutdown_connections 2024-03-21T10:51:15.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.157+0000 7fa40f000640 1 -- 172.21.15.31:0/3466682386 wait complete. 2024-03-21T10:51:15.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.158+0000 7fa40f000640 1 Processor -- start 2024-03-21T10:51:15.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.158+0000 7fa40f000640 1 -- start start 2024-03-21T10:51:15.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.159+0000 7fa40f000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa408107dc0 0x7fa40819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:15.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.159+0000 7fa40f000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa408108760 0x7fa40819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:15.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.159+0000 7fa40f000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa408109b70 0x7fa4081a2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:15.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.159+0000 7fa40f000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa408118170 con 0x7fa408109b70 2024-03-21T10:51:15.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.159+0000 7fa40f000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa408117ff0 con 0x7fa408107dc0 2024-03-21T10:51:15.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.159+0000 7fa40f000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4081182f0 con 0x7fa408108760 2024-03-21T10:51:15.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.159+0000 7fa407400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa408108760 0x7fa40819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:15.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.159+0000 7fa407400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa408108760 0x7fa40819eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:41036/0 (socket says 172.21.15.31:41036) 2024-03-21T10:51:15.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.159+0000 7fa407400640 1 -- 172.21.15.31:0/4052938838 learned_addr learned my addr 172.21.15.31:0/4052938838 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:51:15.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.159+0000 7fa40d000640 1 --2- 172.21.15.31:0/4052938838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa408109b70 0x7fa4081a2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:15.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.159+0000 7fa407e00640 1 --2- 172.21.15.31:0/4052938838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa408107dc0 0x7fa40819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:15.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.159+0000 7fa407400640 1 -- 172.21.15.31:0/4052938838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa408107dc0 msgr2=0x7fa40819e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:15.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.159+0000 7fa407400640 1 --2- 172.21.15.31:0/4052938838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa408107dc0 0x7fa40819e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:15.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.159+0000 7fa407400640 1 -- 172.21.15.31:0/4052938838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa408109b70 msgr2=0x7fa4081a2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:15.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.159+0000 7fa407400640 1 --2- 172.21.15.31:0/4052938838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa408109b70 0x7fa4081a2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:15.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.159+0000 7fa407400640 1 -- 172.21.15.31:0/4052938838 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4081a3640 con 0x7fa408108760 2024-03-21T10:51:15.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.160+0000 7fa407400640 1 --2- 172.21.15.31:0/4052938838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa408108760 0x7fa40819eb90 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400d950 tx=0x7fa3f400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:15.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.160+0000 7fa404c00640 1 -- 172.21.15.31:0/4052938838 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4017030 con 0x7fa408108760 2024-03-21T10:51:15.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.160+0000 7fa404c00640 1 -- 172.21.15.31:0/4052938838 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa3f4004020 con 0x7fa408108760 2024-03-21T10:51:15.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.160+0000 7fa404c00640 1 -- 172.21.15.31:0/4052938838 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f40047a0 con 0x7fa408108760 2024-03-21T10:51:15.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.160+0000 7fa40f000640 1 -- 172.21.15.31:0/4052938838 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4081a3930 con 0x7fa408108760 2024-03-21T10:51:15.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.160+0000 7fa40f000640 1 -- 172.21.15.31:0/4052938838 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4081ab1c0 con 0x7fa408108760 2024-03-21T10:51:15.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.161+0000 7fa40f000640 1 -- 172.21.15.31:0/4052938838 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa408068970 con 0x7fa408108760 2024-03-21T10:51:15.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.166+0000 7fa404c00640 1 -- 172.21.15.31:0/4052938838 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fa3f40041d0 con 0x7fa408108760 2024-03-21T10:51:15.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.167+0000 7fa404c00640 1 --2- 172.21.15.31:0/4052938838 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa3ec0a3110 0x7fa3ec0a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:15.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.168+0000 7fa404c00640 1 -- 172.21.15.31:0/4052938838 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9534+0+0 (secure 0 0 0) 0x7fa3f40cf530 con 0x7fa408108760 2024-03-21T10:51:15.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.168+0000 7fa407e00640 1 --2- 172.21.15.31:0/4052938838 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa3ec0a3110 0x7fa3ec0a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:15.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.168+0000 7fa407e00640 1 --2- 172.21.15.31:0/4052938838 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa3ec0a3110 0x7fa3ec0a55d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8009900 tx=0x7fa3f8005a20 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:15.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.172+0000 7fa404c00640 1 -- 172.21.15.31:0/4052938838 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa3f409bbd0 con 0x7fa408108760 2024-03-21T10:51:15.303 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:15 smithi031 ceph-mon[31673]: pgmap v396: 129 pgs: 11 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 88 active+clean; 12 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 28 KiB/s wr, 9 op/s 2024-03-21T10:51:15.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:15 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1906315008' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:51:15.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:15 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:51:15.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:15 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:51:15.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:15 smithi031 ceph-mon[31673]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:51:15.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:15 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1898975082' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:51:15.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.396+0000 7fa40f000640 1 -- 172.21.15.31:0/4052938838 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa4081102d0 con 0x7fa408108760 2024-03-21T10:51:15.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.396+0000 7fa404c00640 1 -- 172.21.15.31:0/4052938838 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v107) v1 ==== 81+0+945 (secure 0 0 0) 0x7fa3f409b4d0 con 0x7fa408108760 2024-03-21T10:51:15.393 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6800/3875791078 2024-03-22T10:38:33.502014+0000 2024-03-21T10:51:15.393 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6801/3875791078 2024-03-22T10:38:33.502014+0000 2024-03-21T10:51:15.393 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/3488063077 2024-03-22T10:38:33.502014+0000 2024-03-21T10:51:15.394 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/1451930554 2024-03-22T10:26:53.842623+0000 2024-03-21T10:51:15.394 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/3633519331 2024-03-22T10:38:33.502014+0000 2024-03-21T10:51:15.394 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/3342588843 2024-03-22T10:26:53.842623+0000 2024-03-21T10:51:15.394 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/1729503305 2024-03-22T10:38:33.502014+0000 2024-03-21T10:51:15.394 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6801/4085461895 2024-03-22T10:26:53.842623+0000 2024-03-21T10:51:15.394 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6800/4085461895 2024-03-22T10:26:53.842623+0000 2024-03-21T10:51:15.394 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/2171929076 2024-03-22T10:38:33.502014+0000 2024-03-21T10:51:15.394 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/1626448389 2024-03-22T10:26:16.959165+0000 2024-03-21T10:51:15.394 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/1910059351 2024-03-22T10:26:16.959165+0000 2024-03-21T10:51:15.395 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/344822770 2024-03-22T10:26:16.959165+0000 2024-03-21T10:51:15.395 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/3811368512 2024-03-22T10:26:53.842623+0000 2024-03-21T10:51:15.395 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6800/1981437651 2024-03-22T10:26:16.959165+0000 2024-03-21T10:51:15.395 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6801/1981437651 2024-03-22T10:26:16.959165+0000 2024-03-21T10:51:15.395 INFO:teuthology.orchestra.run.smithi031.stderr:listed 16 entries 2024-03-21T10:51:15.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.400+0000 7fa40f000640 1 -- 172.21.15.31:0/4052938838 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa3ec0a3110 msgr2=0x7fa3ec0a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:15.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.400+0000 7fa40f000640 1 --2- 172.21.15.31:0/4052938838 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa3ec0a3110 0x7fa3ec0a55d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8009900 tx=0x7fa3f8005a20 comp rx=0 tx=0).stop 2024-03-21T10:51:15.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.400+0000 7fa40f000640 1 -- 172.21.15.31:0/4052938838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa408108760 msgr2=0x7fa40819eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:15.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.400+0000 7fa40f000640 1 --2- 172.21.15.31:0/4052938838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa408108760 0x7fa40819eb90 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400d950 tx=0x7fa3f400de10 comp rx=0 tx=0).stop 2024-03-21T10:51:15.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.400+0000 7fa40f000640 1 -- 172.21.15.31:0/4052938838 shutdown_connections 2024-03-21T10:51:15.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.400+0000 7fa40f000640 1 --2- 172.21.15.31:0/4052938838 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fa3ec0a3110 0x7fa3ec0a55d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:15.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.400+0000 7fa40f000640 1 --2- 172.21.15.31:0/4052938838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa408109b70 0x7fa4081a2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:15.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.400+0000 7fa40f000640 1 --2- 172.21.15.31:0/4052938838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa408108760 0x7fa40819eb90 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:15.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.400+0000 7fa40f000640 1 --2- 172.21.15.31:0/4052938838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa408107dc0 0x7fa40819e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:15.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.400+0000 7fa40f000640 1 -- 172.21.15.31:0/4052938838 >> 172.21.15.31:0/4052938838 conn(0x7fa408077540 msgr2=0x7fa408077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:15.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.400+0000 7fa40f000640 1 -- 172.21.15.31:0/4052938838 shutdown_connections 2024-03-21T10:51:15.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:15.400+0000 7fa40f000640 1 -- 172.21.15.31:0/4052938838 wait complete. 2024-03-21T10:51:15.415 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-21T10:51:15.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:51:15.416 DEBUG:teuthology.orchestra.run.smithi031:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-21T10:51:15.429 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:15 smithi040 ceph-mon[36154]: pgmap v396: 129 pgs: 11 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 88 active+clean; 12 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 28 KiB/s wr, 9 op/s 2024-03-21T10:51:15.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:15 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1906315008' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:51:15.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:15 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:51:15.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:15 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:51:15.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:15 smithi040 ceph-mon[36154]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:51:15.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:15 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1898975082' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:51:15.540 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi031.front.sepia.ceph.com 2024-03-21T10:51:15.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:51:15.540 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-21T10:51:15.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:15 smithi070 ceph-mon[36330]: pgmap v396: 129 pgs: 11 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 88 active+clean; 12 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 28 KiB/s wr, 9 op/s 2024-03-21T10:51:15.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:15 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1906315008' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-21T10:51:15.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:15 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T10:51:15.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:15 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T10:51:15.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:15 smithi070 ceph-mon[36330]: from='mgr.25564 ' entity='mgr.x' 2024-03-21T10:51:15.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:15 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1898975082' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-21T10:51:15.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:51:15.624 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-03-21T10:51:15.706 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-21T10:51:15.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:51:15.707 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T10:51:15.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T10:51:15.795 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link delete ceph-brx 2024-03-21T10:51:15.920 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-21T10:51:15.931 INFO:tasks.cephadm:Teardown begin 2024-03-21T10:51:15.932 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-21T10:51:15.932 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:51:16.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.148+0000 7ff426400640 1 -- 172.21.15.31:0/2718125971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff420073a40 msgr2=0x7ff420073e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:16.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.148+0000 7ff426400640 1 --2- 172.21.15.31:0/2718125971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff420073a40 0x7ff420073e20 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7ff410009900 tx=0x7ff41002bcd0 comp rx=0 tx=0).stop 2024-03-21T10:51:16.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.148+0000 7ff426400640 1 -- 172.21.15.31:0/2718125971 shutdown_connections 2024-03-21T10:51:16.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.148+0000 7ff426400640 1 --2- 172.21.15.31:0/2718125971 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff420112650 0x7ff420114a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:16.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.148+0000 7ff426400640 1 --2- 172.21.15.31:0/2718125971 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff420074360 0x7ff420112110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:16.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.148+0000 7ff426400640 1 --2- 172.21.15.31:0/2718125971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff420073a40 0x7ff420073e20 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:16.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.148+0000 7ff426400640 1 -- 172.21.15.31:0/2718125971 >> 172.21.15.31:0/2718125971 conn(0x7ff4200fd150 msgr2=0x7ff4200ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:16.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.148+0000 7ff426400640 1 -- 172.21.15.31:0/2718125971 shutdown_connections 2024-03-21T10:51:16.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.148+0000 7ff426400640 1 -- 172.21.15.31:0/2718125971 wait complete. 2024-03-21T10:51:16.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.150+0000 7ff426400640 1 Processor -- start 2024-03-21T10:51:16.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.150+0000 7ff426400640 1 -- start start 2024-03-21T10:51:16.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.150+0000 7ff426400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff420073a40 0x7ff42019e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:16.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.150+0000 7ff426400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff420074360 0x7ff42019ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:16.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.150+0000 7ff426400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff420112650 0x7ff4201a2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:16.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.150+0000 7ff426400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff420118050 con 0x7ff420112650 2024-03-21T10:51:16.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.150+0000 7ff426400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff420117ed0 con 0x7ff420074360 2024-03-21T10:51:16.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.150+0000 7ff426400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4201181d0 con 0x7ff420073a40 2024-03-21T10:51:16.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.151+0000 7ff41f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff420073a40 0x7ff42019e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:16.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.151+0000 7ff417e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff420074360 0x7ff42019ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.151+0000 7ff41f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff420073a40 0x7ff42019e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:41056/0 (socket says 172.21.15.31:41056) 2024-03-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.151+0000 7ff41f400640 1 -- 172.21.15.31:0/387121759 learned_addr learned my addr 172.21.15.31:0/387121759 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.151+0000 7ff41fe00640 1 --2- 172.21.15.31:0/387121759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff420112650 0x7ff4201a2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.151+0000 7ff41f400640 1 -- 172.21.15.31:0/387121759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff420074360 msgr2=0x7ff42019ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.151+0000 7ff41f400640 1 --2- 172.21.15.31:0/387121759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff420074360 0x7ff42019ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.151+0000 7ff41f400640 1 -- 172.21.15.31:0/387121759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff420112650 msgr2=0x7ff4201a2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.151+0000 7ff41f400640 1 --2- 172.21.15.31:0/387121759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff420112650 0x7ff4201a2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.151+0000 7ff41f400640 1 -- 172.21.15.31:0/387121759 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4201a3580 con 0x7ff420073a40 2024-03-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.151+0000 7ff417e00640 1 --2- 172.21.15.31:0/387121759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff420074360 0x7ff42019ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.151+0000 7ff41f400640 1 --2- 172.21.15.31:0/387121759 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff420073a40 0x7ff42019e590 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7ff410005940 tx=0x7ff41002cbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.152+0000 7ff41cc00640 1 -- 172.21.15.31:0/387121759 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff410041070 con 0x7ff420073a40 2024-03-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.152+0000 7ff41cc00640 1 -- 172.21.15.31:0/387121759 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff410004450 con 0x7ff420073a40 2024-03-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.152+0000 7ff426400640 1 -- 172.21.15.31:0/387121759 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4201a3810 con 0x7ff420073a40 2024-03-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.152+0000 7ff41cc00640 1 -- 172.21.15.31:0/387121759 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff410003d80 con 0x7ff420073a40 2024-03-21T10:51:16.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.152+0000 7ff426400640 1 -- 172.21.15.31:0/387121759 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4201ab100 con 0x7ff420073a40 2024-03-21T10:51:16.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.153+0000 7ff426400640 1 -- 172.21.15.31:0/387121759 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff420068970 con 0x7ff420073a40 2024-03-21T10:51:16.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.154+0000 7ff41cc00640 1 -- 172.21.15.31:0/387121759 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7ff410044070 con 0x7ff420073a40 2024-03-21T10:51:16.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.157+0000 7ff41cc00640 1 --2- 172.21.15.31:0/387121759 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff4040a3110 0x7ff4040a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:16.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.157+0000 7ff41cc00640 1 -- 172.21.15.31:0/387121759 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9534+0+0 (secure 0 0 0) 0x7ff4100f1550 con 0x7ff420073a40 2024-03-21T10:51:16.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.157+0000 7ff417e00640 1 --2- 172.21.15.31:0/387121759 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff4040a3110 0x7ff4040a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:16.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.158+0000 7ff417e00640 1 --2- 172.21.15.31:0/387121759 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff4040a3110 0x7ff4040a55d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff4200ff850 tx=0x7ff408005f90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:16.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.160+0000 7ff41cc00640 1 -- 172.21.15.31:0/387121759 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff4100bd6b0 con 0x7ff420073a40 2024-03-21T10:51:16.351 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:16 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/4052938838' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T10:51:16.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:16 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/4052938838' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T10:51:16.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.435+0000 7ff426400640 1 -- 172.21.15.31:0/387121759 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff420104ac0 con 0x7ff420073a40 2024-03-21T10:51:16.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.435+0000 7ff41cc00640 1 -- 172.21.15.31:0/387121759 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+11386 (secure 0 0 0) 0x7ff4100c1750 con 0x7ff420073a40 2024-03-21T10:51:16.432 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:51:16.432 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:50:29.918324+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24547,24596]},"id":1}]} 2024-03-21T10:51:16.433 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 32 2024-03-21T10:51:16.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.438+0000 7ff426400640 1 -- 172.21.15.31:0/387121759 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff4040a3110 msgr2=0x7ff4040a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:16.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.438+0000 7ff426400640 1 --2- 172.21.15.31:0/387121759 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff4040a3110 0x7ff4040a55d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff4200ff850 tx=0x7ff408005f90 comp rx=0 tx=0).stop 2024-03-21T10:51:16.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.438+0000 7ff426400640 1 -- 172.21.15.31:0/387121759 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff420073a40 msgr2=0x7ff42019e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:16.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.438+0000 7ff426400640 1 --2- 172.21.15.31:0/387121759 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff420073a40 0x7ff42019e590 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7ff410005940 tx=0x7ff41002cbd0 comp rx=0 tx=0).stop 2024-03-21T10:51:16.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.438+0000 7ff426400640 1 -- 172.21.15.31:0/387121759 shutdown_connections 2024-03-21T10:51:16.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.438+0000 7ff426400640 1 --2- 172.21.15.31:0/387121759 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff4040a3110 0x7ff4040a55d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:16.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.438+0000 7ff426400640 1 --2- 172.21.15.31:0/387121759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff420112650 0x7ff4201a2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:16.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.438+0000 7ff426400640 1 --2- 172.21.15.31:0/387121759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff420074360 0x7ff42019ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:16.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.438+0000 7ff426400640 1 --2- 172.21.15.31:0/387121759 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff420073a40 0x7ff42019e590 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:16.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.438+0000 7ff426400640 1 -- 172.21.15.31:0/387121759 >> 172.21.15.31:0/387121759 conn(0x7ff4200fd150 msgr2=0x7ff4200fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:16.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.438+0000 7ff426400640 1 -- 172.21.15.31:0/387121759 shutdown_connections 2024-03-21T10:51:16.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.438+0000 7ff426400640 1 -- 172.21.15.31:0/387121759 wait complete. 2024-03-21T10:51:16.453 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T10:51:16.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:16 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/4052938838' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T10:51:16.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.661+0000 7f3231400640 1 -- 172.21.15.31:0/2093302754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f322c107610 msgr2=0x7f322c1079f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:16.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.661+0000 7f3231400640 1 --2- 172.21.15.31:0/2093302754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f322c107610 0x7f322c1079f0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f321c009930 tx=0x7f321c02bc20 comp rx=0 tx=0).stop 2024-03-21T10:51:16.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.661+0000 7f3231400640 1 -- 172.21.15.31:0/2093302754 shutdown_connections 2024-03-21T10:51:16.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.661+0000 7f3231400640 1 --2- 172.21.15.31:0/2093302754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f322c101f10 0x7f322c10e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:16.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.661+0000 7f3231400640 1 --2- 172.21.15.31:0/2093302754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f322c101550 0x7f322c1019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:16.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.661+0000 7f3231400640 1 --2- 172.21.15.31:0/2093302754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f322c107610 0x7f322c1079f0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:16.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.661+0000 7f3231400640 1 -- 172.21.15.31:0/2093302754 >> 172.21.15.31:0/2093302754 conn(0x7f322c0fd060 msgr2=0x7f322c0ff480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:16.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.661+0000 7f3231400640 1 -- 172.21.15.31:0/2093302754 shutdown_connections 2024-03-21T10:51:16.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.661+0000 7f3231400640 1 -- 172.21.15.31:0/2093302754 wait complete. 2024-03-21T10:51:16.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.662+0000 7f3231400640 1 Processor -- start 2024-03-21T10:51:16.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.662+0000 7f3231400640 1 -- start start 2024-03-21T10:51:16.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.662+0000 7f3231400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f322c101550 0x7f322c197dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:16.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.663+0000 7f3231400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f322c101f10 0x7f322c198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:16.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.663+0000 7f3231400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f322c107610 0x7f322c19c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:16.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.663+0000 7f3231400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c1118f0 con 0x7f322c107610 2024-03-21T10:51:16.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.663+0000 7f3231400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c111770 con 0x7f322c101f10 2024-03-21T10:51:16.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.663+0000 7f3231400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c111a70 con 0x7f322c101550 2024-03-21T10:51:16.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.663+0000 7f322be00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f322c101550 0x7f322c197dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:16.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.663+0000 7f322be00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f322c101550 0x7f322c197dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:41070/0 (socket says 172.21.15.31:41070) 2024-03-21T10:51:16.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.663+0000 7f322be00640 1 -- 172.21.15.31:0/256113716 learned_addr learned my addr 172.21.15.31:0/256113716 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:51:16.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.663+0000 7f3223e00640 1 --2- 172.21.15.31:0/256113716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f322c101f10 0x7f322c198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:16.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.663+0000 7f3230a00640 1 --2- 172.21.15.31:0/256113716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f322c107610 0x7f322c19c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:16.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.663+0000 7f322be00640 1 -- 172.21.15.31:0/256113716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f322c101f10 msgr2=0x7f322c198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:16.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.663+0000 7f322be00640 1 --2- 172.21.15.31:0/256113716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f322c101f10 0x7f322c198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:16.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.663+0000 7f322be00640 1 -- 172.21.15.31:0/256113716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f322c107610 msgr2=0x7f322c19c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:16.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.663+0000 7f322be00640 1 --2- 172.21.15.31:0/256113716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f322c107610 0x7f322c19c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:16.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.663+0000 7f322be00640 1 -- 172.21.15.31:0/256113716 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322c19cdc0 con 0x7f322c101550 2024-03-21T10:51:16.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.663+0000 7f3223e00640 1 --2- 172.21.15.31:0/256113716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f322c101f10 0x7f322c198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:51:16.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.664+0000 7f322be00640 1 --2- 172.21.15.31:0/256113716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f322c101550 0x7f322c197dd0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f321c009430 tx=0x7f321c02cb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:16.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.664+0000 7f3229600640 1 -- 172.21.15.31:0/256113716 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c041070 con 0x7f322c101550 2024-03-21T10:51:16.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.664+0000 7f3229600640 1 -- 172.21.15.31:0/256113716 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f321c004450 con 0x7f322c101550 2024-03-21T10:51:16.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.664+0000 7f3231400640 1 -- 172.21.15.31:0/256113716 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f322c19d050 con 0x7f322c101550 2024-03-21T10:51:16.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.664+0000 7f3229600640 1 -- 172.21.15.31:0/256113716 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c003dc0 con 0x7f322c101550 2024-03-21T10:51:16.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.664+0000 7f3231400640 1 -- 172.21.15.31:0/256113716 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f322c1a4940 con 0x7f322c101550 2024-03-21T10:51:16.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.665+0000 7f3231400640 1 -- 172.21.15.31:0/256113716 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f322c04f470 con 0x7f322c101550 2024-03-21T10:51:16.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.667+0000 7f3229600640 1 -- 172.21.15.31:0/256113716 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f321c044070 con 0x7f322c101550 2024-03-21T10:51:16.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.669+0000 7f3229600640 1 --2- 172.21.15.31:0/256113716 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f32100a3100 0x7f32100a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:16.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.670+0000 7f3229600640 1 -- 172.21.15.31:0/256113716 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f321c0f14e0 con 0x7f322c101550 2024-03-21T10:51:16.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.670+0000 7f3223e00640 1 --2- 172.21.15.31:0/256113716 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f32100a3100 0x7f32100a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:16.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.670+0000 7f3223e00640 1 --2- 172.21.15.31:0/256113716 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f32100a3100 0x7f32100a55c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f322c106c00 tx=0x7f3214005f90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:16.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.672+0000 7f3229600640 1 -- 172.21.15.31:0/256113716 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f321c0bdce0 con 0x7f322c101550 2024-03-21T10:51:16.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.980+0000 7f3231400640 1 -- 172.21.15.31:0/256113716 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f322c109a50 con 0x7f322c101550 2024-03-21T10:51:16.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.981+0000 7f3229600640 1 -- 172.21.15.31:0/256113716 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+11386 (secure 0 0 0) 0x7f321c0bd5e0 con 0x7f322c101550 2024-03-21T10:51:16.977 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:51:16.977 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2099524212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2099524212},{"type":"v1","addr":"172.21.15.31:6835","nonce":2099524212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/1193249174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":1193249174},{"type":"v1","addr":"172.21.15.31:6837","nonce":1193249174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/4261892640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":4261892640},{"type":"v1","addr":"172.21.15.31:6839","nonce":4261892640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/4158670810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":4158670810},{"type":"v1","addr":"172.21.15.70:6833","nonce":4158670810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2806510034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2806510034},{"type":"v1","addr":"172.21.15.40:6833","nonce":2806510034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2081017532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2081017532},{"type":"v1","addr":"172.21.15.40:6835","nonce":2081017532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-21T10:35:06.492115+0000","modified":"2024-03-21T10:50:29.918324+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24596,"mds_1":24584,"mds_2":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/729664889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":729664889},{"type":"v1","addr":"172.21.15.70:6835","nonce":729664889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/3650805915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3650805915},{"type":"v1","addr":"172.21.15.70:6837","nonce":3650805915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6837/4121707657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":4121707657},{"type":"v1","addr":"172.21.15.40:6837","nonce":4121707657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.70:6839/3204327153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":3204327153},{"type":"v1","addr":"172.21.15.70:6839","nonce":3204327153}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24584":{"gid":24584,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6841/1752599123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":1752599123},{"type":"v1","addr":"172.21.15.31:6841","nonce":1752599123}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/2539201148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2539201148},{"type":"v1","addr":"172.21.15.40:6839","nonce":2539201148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24584,24547,24596]},"id":1}]} 2024-03-21T10:51:16.978 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 32 2024-03-21T10:51:16.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.984+0000 7f3231400640 1 -- 172.21.15.31:0/256113716 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f32100a3100 msgr2=0x7f32100a55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:16.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.984+0000 7f3231400640 1 --2- 172.21.15.31:0/256113716 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f32100a3100 0x7f32100a55c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f322c106c00 tx=0x7f3214005f90 comp rx=0 tx=0).stop 2024-03-21T10:51:16.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.984+0000 7f3231400640 1 -- 172.21.15.31:0/256113716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f322c101550 msgr2=0x7f322c197dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:16.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.984+0000 7f3231400640 1 --2- 172.21.15.31:0/256113716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f322c101550 0x7f322c197dd0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f321c009430 tx=0x7f321c02cb10 comp rx=0 tx=0).stop 2024-03-21T10:51:16.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.984+0000 7f3231400640 1 -- 172.21.15.31:0/256113716 shutdown_connections 2024-03-21T10:51:16.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.984+0000 7f3231400640 1 --2- 172.21.15.31:0/256113716 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f32100a3100 0x7f32100a55c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:16.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.984+0000 7f3231400640 1 --2- 172.21.15.31:0/256113716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f322c107610 0x7f322c19c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:16.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.984+0000 7f3231400640 1 --2- 172.21.15.31:0/256113716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f322c101f10 0x7f322c198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:16.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.984+0000 7f3231400640 1 --2- 172.21.15.31:0/256113716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f322c101550 0x7f322c197dd0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:16.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.984+0000 7f3231400640 1 -- 172.21.15.31:0/256113716 >> 172.21.15.31:0/256113716 conn(0x7f322c0fd060 msgr2=0x7f322c0fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:16.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.984+0000 7f3231400640 1 -- 172.21.15.31:0/256113716 shutdown_connections 2024-03-21T10:51:16.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:16.984+0000 7f3231400640 1 -- 172.21.15.31:0/256113716 wait complete. 2024-03-21T10:51:17.002 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T10:51:17.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.202+0000 7ff4f7600640 1 -- 172.21.15.31:0/1483194626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4f0103860 msgr2=0x7ff4f0103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:17.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.202+0000 7ff4f7600640 1 --2- 172.21.15.31:0/1483194626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4f0103860 0x7ff4f0103ce0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8009930 tx=0x7ff4d802bc30 comp rx=0 tx=0).stop 2024-03-21T10:51:17.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.202+0000 7ff4f7600640 1 -- 172.21.15.31:0/1483194626 shutdown_connections 2024-03-21T10:51:17.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.202+0000 7ff4f7600640 1 --2- 172.21.15.31:0/1483194626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4f0104220 0x7ff4f0110730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:17.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.202+0000 7ff4f7600640 1 --2- 172.21.15.31:0/1483194626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4f0103860 0x7ff4f0103ce0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:17.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.202+0000 7ff4f7600640 1 --2- 172.21.15.31:0/1483194626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4f0109870 0x7ff4f0109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:17.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.202+0000 7ff4f7600640 1 -- 172.21.15.31:0/1483194626 >> 172.21.15.31:0/1483194626 conn(0x7ff4f00ff2f0 msgr2=0x7ff4f0101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:17.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.202+0000 7ff4f7600640 1 -- 172.21.15.31:0/1483194626 shutdown_connections 2024-03-21T10:51:17.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.202+0000 7ff4f7600640 1 -- 172.21.15.31:0/1483194626 wait complete. 2024-03-21T10:51:17.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.203+0000 7ff4f7600640 1 Processor -- start 2024-03-21T10:51:17.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.203+0000 7ff4f7600640 1 -- start start 2024-03-21T10:51:17.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.204+0000 7ff4f7600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4f0103860 0x7ff4f019a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:17.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.204+0000 7ff4f7600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4f0104220 0x7ff4f019a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:17.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.204+0000 7ff4f7600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4f0109870 0x7ff4f019ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:17.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.204+0000 7ff4f7600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f0113c90 con 0x7ff4f0109870 2024-03-21T10:51:17.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.204+0000 7ff4f7600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f0113b10 con 0x7ff4f0104220 2024-03-21T10:51:17.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.204+0000 7ff4f7600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f0113e10 con 0x7ff4f0103860 2024-03-21T10:51:17.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.204+0000 7ff4e7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4f0104220 0x7ff4f019a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:17.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.204+0000 7ff4f4c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4f0103860 0x7ff4f019a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:17.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.204+0000 7ff4e7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4f0104220 0x7ff4f019a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:52090/0 (socket says 172.21.15.31:52090) 2024-03-21T10:51:17.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.204+0000 7ff4e7e00640 1 -- 172.21.15.31:0/3248329835 learned_addr learned my addr 172.21.15.31:0/3248329835 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:51:17.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.204+0000 7ff4f5600640 1 --2- 172.21.15.31:0/3248329835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4f0109870 0x7ff4f019ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:17.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.204+0000 7ff4e7e00640 1 -- 172.21.15.31:0/3248329835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4f0103860 msgr2=0x7ff4f019a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:17.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.204+0000 7ff4e7e00640 1 --2- 172.21.15.31:0/3248329835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4f0103860 0x7ff4f019a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:17.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.204+0000 7ff4e7e00640 1 -- 172.21.15.31:0/3248329835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4f0109870 msgr2=0x7ff4f019ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:17.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.204+0000 7ff4e7e00640 1 --2- 172.21.15.31:0/3248329835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4f0109870 0x7ff4f019ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:17.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.204+0000 7ff4e7e00640 1 -- 172.21.15.31:0/3248329835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f019f180 con 0x7ff4f0104220 2024-03-21T10:51:17.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.204+0000 7ff4f4c00640 1 --2- 172.21.15.31:0/3248329835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4f0103860 0x7ff4f019a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:51:17.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.205+0000 7ff4e7e00640 1 --2- 172.21.15.31:0/3248329835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4f0104220 0x7ff4f019a6d0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8009900 tx=0x7ff4d8004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:17.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.205+0000 7ff4ee000640 1 -- 172.21.15.31:0/3248329835 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d8043020 con 0x7ff4f0104220 2024-03-21T10:51:17.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.205+0000 7ff4f7600640 1 -- 172.21.15.31:0/3248329835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4f019f410 con 0x7ff4f0104220 2024-03-21T10:51:17.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.205+0000 7ff4ee000640 1 -- 172.21.15.31:0/3248329835 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4d80054e0 con 0x7ff4f0104220 2024-03-21T10:51:17.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.205+0000 7ff4ee000640 1 -- 172.21.15.31:0/3248329835 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d8003bb0 con 0x7ff4f0104220 2024-03-21T10:51:17.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.205+0000 7ff4f7600640 1 -- 172.21.15.31:0/3248329835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4f01a6d00 con 0x7ff4f0104220 2024-03-21T10:51:17.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.206+0000 7ff4f7600640 1 -- 172.21.15.31:0/3248329835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4f0068970 con 0x7ff4f0104220 2024-03-21T10:51:17.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.209+0000 7ff4ee000640 1 -- 172.21.15.31:0/3248329835 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7ff4d8046020 con 0x7ff4f0104220 2024-03-21T10:51:17.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.210+0000 7ff4ee000640 1 --2- 172.21.15.31:0/3248329835 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff4d40a3090 0x7ff4d40a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:17.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.210+0000 7ff4ee000640 1 -- 172.21.15.31:0/3248329835 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 9446+0+0 (secure 0 0 0) 0x7ff4d80f1cd0 con 0x7ff4f0104220 2024-03-21T10:51:17.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.210+0000 7ff4f4c00640 1 --2- 172.21.15.31:0/3248329835 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff4d40a3090 0x7ff4d40a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:17.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.211+0000 7ff4f4c00640 1 --2- 172.21.15.31:0/3248329835 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff4d40a3090 0x7ff4d40a5550 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0009f40 tx=0x7ff4e0005f90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:17.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.212+0000 7ff4ee000640 1 -- 172.21.15.31:0/3248329835 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff4d804d020 con 0x7ff4f0104220 2024-03-21T10:51:17.303 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:17 smithi031 ceph-mon[31673]: pgmap v397: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-21T10:51:17.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:17 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/387121759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:51:17.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:17 smithi031 ceph-mon[31673]: osdmap e108: 12 total, 12 up, 12 in 2024-03-21T10:51:17.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:17 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/256113716' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:51:17.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:17 smithi040 ceph-mon[36154]: pgmap v397: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-21T10:51:17.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:17 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/387121759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:51:17.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:17 smithi040 ceph-mon[36154]: osdmap e108: 12 total, 12 up, 12 in 2024-03-21T10:51:17.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:17 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/256113716' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:51:17.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.501+0000 7ff4f7600640 1 -- 172.21.15.31:0/3248329835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4f010be20 con 0x7ff4f0104220 2024-03-21T10:51:17.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.503+0000 7ff4ee000640 1 -- 172.21.15.31:0/3248329835 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+25117 (secure 0 0 0) 0x7ff4d80be4d0 con 0x7ff4f0104220 2024-03-21T10:51:17.500 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-21T10:51:17.501 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":108,"fsid":"3847408c-e76d-11ee-95cd-87774f69a715","created":"2024-03-21T10:25:41.971117+0000","modified":"2024-03-21T10:51:16.455071+0000","last_up_change":"2024-03-21T10:34:07.757196+0000","last_in_change":"2024-03-21T10:33:47.460406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T10:29:44.553968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T10:35:04.945449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T10:35:05.970589+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":21,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T10:35:09.688711+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":21,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e0244d33-00e2-47d5-9f66-34efc138c462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6803","nonce":3089885125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6805","nonce":3089885125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6809","nonce":3089885125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3089885125},{"type":"v1","addr":"172.21.15.31:6807","nonce":3089885125}]},"public_addr":"172.21.15.31:6803/3089885125","cluster_addr":"172.21.15.31:6805/3089885125","heartbeat_back_addr":"172.21.15.31:6809/3089885125","heartbeat_front_addr":"172.21.15.31:6807/3089885125","state":["exists","up"]},{"osd":1,"uuid":"76a63504-beab-4354-a6c3-eebc3b16ad03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6801","nonce":618840037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6803","nonce":618840037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6807","nonce":618840037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":618840037},{"type":"v1","addr":"172.21.15.40:6805","nonce":618840037}]},"public_addr":"172.21.15.40:6801/618840037","cluster_addr":"172.21.15.40:6803/618840037","heartbeat_back_addr":"172.21.15.40:6807/618840037","heartbeat_front_addr":"172.21.15.40:6805/618840037","state":["exists","up"]},{"osd":2,"uuid":"fa191bd6-9a21-4ee2-8adf-8a63c9cfd821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6801","nonce":2524184117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6803","nonce":2524184117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6807","nonce":2524184117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":2524184117},{"type":"v1","addr":"172.21.15.70:6805","nonce":2524184117}]},"public_addr":"172.21.15.70:6801/2524184117","cluster_addr":"172.21.15.70:6803/2524184117","heartbeat_back_addr":"172.21.15.70:6807/2524184117","heartbeat_front_addr":"172.21.15.70:6805/2524184117","state":["exists","up"]},{"osd":3,"uuid":"edb44507-ffb0-42a4-827b-08461e1bf952","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6811","nonce":2823036296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6813","nonce":2823036296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6817","nonce":2823036296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2823036296},{"type":"v1","addr":"172.21.15.31:6815","nonce":2823036296}]},"public_addr":"172.21.15.31:6811/2823036296","cluster_addr":"172.21.15.31:6813/2823036296","heartbeat_back_addr":"172.21.15.31:6817/2823036296","heartbeat_front_addr":"172.21.15.31:6815/2823036296","state":["exists","up"]},{"osd":4,"uuid":"8cb89153-8c3b-431f-b57c-8863fada76dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6809","nonce":4215186309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6811","nonce":4215186309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6815","nonce":4215186309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4215186309},{"type":"v1","addr":"172.21.15.40:6813","nonce":4215186309}]},"public_addr":"172.21.15.40:6809/4215186309","cluster_addr":"172.21.15.40:6811/4215186309","heartbeat_back_addr":"172.21.15.40:6815/4215186309","heartbeat_front_addr":"172.21.15.40:6813/4215186309","state":["exists","up"]},{"osd":5,"uuid":"9409505a-d463-408c-a3cd-218380c04d68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6809","nonce":3402830452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6811","nonce":3402830452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6815","nonce":3402830452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3402830452},{"type":"v1","addr":"172.21.15.70:6813","nonce":3402830452}]},"public_addr":"172.21.15.70:6809/3402830452","cluster_addr":"172.21.15.70:6811/3402830452","heartbeat_back_addr":"172.21.15.70:6815/3402830452","heartbeat_front_addr":"172.21.15.70:6813/3402830452","state":["exists","up"]},{"osd":6,"uuid":"f415f9ea-f9c6-4fd3-b9b1-2a7cec48ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6819","nonce":1193497058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6821","nonce":1193497058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6825","nonce":1193497058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1193497058},{"type":"v1","addr":"172.21.15.31:6823","nonce":1193497058}]},"public_addr":"172.21.15.31:6819/1193497058","cluster_addr":"172.21.15.31:6821/1193497058","heartbeat_back_addr":"172.21.15.31:6825/1193497058","heartbeat_front_addr":"172.21.15.31:6823/1193497058","state":["exists","up"]},{"osd":7,"uuid":"db228a87-29a5-45d4-8fe6-a0a554262eb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6817","nonce":3825628119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6819","nonce":3825628119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6823","nonce":3825628119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3825628119},{"type":"v1","addr":"172.21.15.40:6821","nonce":3825628119}]},"public_addr":"172.21.15.40:6817/3825628119","cluster_addr":"172.21.15.40:6819/3825628119","heartbeat_back_addr":"172.21.15.40:6823/3825628119","heartbeat_front_addr":"172.21.15.40:6821/3825628119","state":["exists","up"]},{"osd":8,"uuid":"59b9fde0-095d-4e0b-87da-ead37bd8ff1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6817","nonce":534880586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6819","nonce":534880586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6823","nonce":534880586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":534880586},{"type":"v1","addr":"172.21.15.70:6821","nonce":534880586}]},"public_addr":"172.21.15.70:6817/534880586","cluster_addr":"172.21.15.70:6819/534880586","heartbeat_back_addr":"172.21.15.70:6823/534880586","heartbeat_front_addr":"172.21.15.70:6821/534880586","state":["exists","up"]},{"osd":9,"uuid":"77f47061-0c5f-4159-b276-e814935d6c1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6827","nonce":587047469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6829","nonce":587047469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6833","nonce":587047469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":587047469},{"type":"v1","addr":"172.21.15.31:6831","nonce":587047469}]},"public_addr":"172.21.15.31:6827/587047469","cluster_addr":"172.21.15.31:6829/587047469","heartbeat_back_addr":"172.21.15.31:6833/587047469","heartbeat_front_addr":"172.21.15.31:6831/587047469","state":["exists","up"]},{"osd":10,"uuid":"8853dc43-7bd1-4c93-8063-f7982fd7da38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6825","nonce":512108711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6827","nonce":512108711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6831","nonce":512108711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":512108711},{"type":"v1","addr":"172.21.15.40:6829","nonce":512108711}]},"public_addr":"172.21.15.40:6825/512108711","cluster_addr":"172.21.15.40:6827/512108711","heartbeat_back_addr":"172.21.15.40:6831/512108711","heartbeat_front_addr":"172.21.15.40:6829/512108711","state":["exists","up"]},{"osd":11,"uuid":"84b7c2de-c424-45a6-9583-60097e372a86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6825","nonce":540305718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6827","nonce":540305718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6831","nonce":540305718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":540305718},{"type":"v1","addr":"172.21.15.70:6829","nonce":540305718}]},"public_addr":"172.21.15.70:6825/540305718","cluster_addr":"172.21.15.70:6827/540305718","heartbeat_back_addr":"172.21.15.70:6831/540305718","heartbeat_front_addr":"172.21.15.70:6829/540305718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:28:51.477358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:17.405045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:29:41.626268+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:15.339568+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:30:42.676996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:07.516999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:31:42.412273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:10.365041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:32:35.699120+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:11.733639+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:33:39.505442+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T10:34:05.449301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:6800/3875791078":"2024-03-22T10:38:33.502014+0000","172.21.15.31:6801/3875791078":"2024-03-22T10:38:33.502014+0000","172.21.15.31:0/3488063077":"2024-03-22T10:38:33.502014+0000","172.21.15.31:0/1451930554":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/3633519331":"2024-03-22T10:38:33.502014+0000","172.21.15.31:0/3342588843":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/1729503305":"2024-03-22T10:38:33.502014+0000","172.21.15.31:6801/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/4085461895":"2024-03-22T10:26:53.842623+0000","172.21.15.31:0/2171929076":"2024-03-22T10:38:33.502014+0000","172.21.15.31:0/1626448389":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/1910059351":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/344822770":"2024-03-22T10:26:16.959165+0000","172.21.15.31:0/3811368512":"2024-03-22T10:26:53.842623+0000","172.21.15.31:6800/1981437651":"2024-03-22T10:26:16.959165+0000","172.21.15.31:6801/1981437651":"2024-03-22T10:26:16.959165+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]},{"pool":4,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T10:51:17.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.507+0000 7ff4f7600640 1 -- 172.21.15.31:0/3248329835 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff4d40a3090 msgr2=0x7ff4d40a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:17.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.507+0000 7ff4f7600640 1 --2- 172.21.15.31:0/3248329835 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff4d40a3090 0x7ff4d40a5550 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0009f40 tx=0x7ff4e0005f90 comp rx=0 tx=0).stop 2024-03-21T10:51:17.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.507+0000 7ff4f7600640 1 -- 172.21.15.31:0/3248329835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4f0104220 msgr2=0x7ff4f019a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:17.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.507+0000 7ff4f7600640 1 --2- 172.21.15.31:0/3248329835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4f0104220 0x7ff4f019a6d0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8009900 tx=0x7ff4d8004360 comp rx=0 tx=0).stop 2024-03-21T10:51:17.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.507+0000 7ff4f7600640 1 -- 172.21.15.31:0/3248329835 shutdown_connections 2024-03-21T10:51:17.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.507+0000 7ff4f7600640 1 --2- 172.21.15.31:0/3248329835 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7ff4d40a3090 0x7ff4d40a5550 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:17.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.507+0000 7ff4f7600640 1 --2- 172.21.15.31:0/3248329835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4f0109870 0x7ff4f019ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:17.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.507+0000 7ff4f7600640 1 --2- 172.21.15.31:0/3248329835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4f0104220 0x7ff4f019a6d0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:17.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.507+0000 7ff4f7600640 1 --2- 172.21.15.31:0/3248329835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4f0103860 0x7ff4f019a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:17.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.507+0000 7ff4f7600640 1 -- 172.21.15.31:0/3248329835 >> 172.21.15.31:0/3248329835 conn(0x7ff4f00ff2f0 msgr2=0x7ff4f00ffa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:17.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.507+0000 7ff4f7600640 1 -- 172.21.15.31:0/3248329835 shutdown_connections 2024-03-21T10:51:17.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.507+0000 7ff4f7600640 1 -- 172.21.15.31:0/3248329835 wait complete. 2024-03-21T10:51:17.523 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-21T10:51:17.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:17 smithi070 ceph-mon[36330]: pgmap v397: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-21T10:51:17.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:17 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/387121759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:51:17.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:17 smithi070 ceph-mon[36330]: osdmap e108: 12 total, 12 up, 12 in 2024-03-21T10:51:17.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:17 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/256113716' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-21T10:51:17.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.743+0000 7f9d55c00640 1 -- 172.21.15.31:0/2271669680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d50104020 msgr2=0x7f9d50110530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:17.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.743+0000 7f9d55c00640 1 --2- 172.21.15.31:0/2271669680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d50104020 0x7f9d50110530 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f9d48009ca0 tx=0x7f9d4802c100 comp rx=0 tx=0).stop 2024-03-21T10:51:17.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.743+0000 7f9d55c00640 1 -- 172.21.15.31:0/2271669680 shutdown_connections 2024-03-21T10:51:17.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.743+0000 7f9d55c00640 1 --2- 172.21.15.31:0/2271669680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d50104020 0x7f9d50110530 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:17.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.743+0000 7f9d55c00640 1 --2- 172.21.15.31:0/2271669680 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d50103660 0x7f9d50103ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:17.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.743+0000 7f9d55c00640 1 --2- 172.21.15.31:0/2271669680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d50109670 0x7f9d50109a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:17.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.743+0000 7f9d55c00640 1 -- 172.21.15.31:0/2271669680 >> 172.21.15.31:0/2271669680 conn(0x7f9d500ff110 msgr2=0x7f9d50101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:17.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.743+0000 7f9d55c00640 1 -- 172.21.15.31:0/2271669680 shutdown_connections 2024-03-21T10:51:17.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.743+0000 7f9d55c00640 1 -- 172.21.15.31:0/2271669680 wait complete. 2024-03-21T10:51:17.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.744+0000 7f9d55c00640 1 Processor -- start 2024-03-21T10:51:17.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.744+0000 7f9d55c00640 1 -- start start 2024-03-21T10:51:17.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.744+0000 7f9d55c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d50103660 0x7f9d50199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:17.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.744+0000 7f9d55c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d50104020 0x7f9d5019a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:17.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.745+0000 7f9d55c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d50109670 0x7f9d5019e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:17.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.745+0000 7f9d55c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d50113a80 con 0x7f9d50109670 2024-03-21T10:51:17.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.745+0000 7f9d55c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d50113900 con 0x7f9d50104020 2024-03-21T10:51:17.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.745+0000 7f9d55c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d50113c00 con 0x7f9d50103660 2024-03-21T10:51:17.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.745+0000 7f9d4fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d50103660 0x7f9d50199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:17.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.745+0000 7f9d4fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d50103660 0x7f9d50199f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:40202/0 (socket says 172.21.15.31:40202) 2024-03-21T10:51:17.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.745+0000 7f9d4fe00640 1 -- 172.21.15.31:0/403406465 learned_addr learned my addr 172.21.15.31:0/403406465 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:51:17.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.745+0000 7f9d4f400640 1 --2- 172.21.15.31:0/403406465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d50104020 0x7f9d5019a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:17.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.745+0000 7f9d55200640 1 --2- 172.21.15.31:0/403406465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d50109670 0x7f9d5019e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:17.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.745+0000 7f9d4fe00640 1 -- 172.21.15.31:0/403406465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d50104020 msgr2=0x7f9d5019a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:17.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.745+0000 7f9d4fe00640 1 --2- 172.21.15.31:0/403406465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d50104020 0x7f9d5019a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:17.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.745+0000 7f9d4fe00640 1 -- 172.21.15.31:0/403406465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d50109670 msgr2=0x7f9d5019e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:17.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.745+0000 7f9d4fe00640 1 --2- 172.21.15.31:0/403406465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d50109670 0x7f9d5019e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:17.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.745+0000 7f9d4fe00640 1 -- 172.21.15.31:0/403406465 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d5019ef10 con 0x7f9d50103660 2024-03-21T10:51:17.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.746+0000 7f9d4fe00640 1 --2- 172.21.15.31:0/403406465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d50103660 0x7f9d50199f20 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f9d4000d950 tx=0x7f9d4000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:17.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.746+0000 7f9d4cc00640 1 -- 172.21.15.31:0/403406465 <== mon.1 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d40017070 con 0x7f9d50103660 2024-03-21T10:51:17.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.746+0000 7f9d4cc00640 1 -- 172.21.15.31:0/403406465 <== mon.1 v2:172.21.15.70:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d40009c30 con 0x7f9d50103660 2024-03-21T10:51:17.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.746+0000 7f9d55c00640 1 -- 172.21.15.31:0/403406465 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d5019f200 con 0x7f9d50103660 2024-03-21T10:51:17.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.746+0000 7f9d4cc00640 1 -- 172.21.15.31:0/403406465 <== mon.1 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4000f6b0 con 0x7f9d50103660 2024-03-21T10:51:17.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.746+0000 7f9d55c00640 1 -- 172.21.15.31:0/403406465 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d501a6a90 con 0x7f9d50103660 2024-03-21T10:51:17.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.747+0000 7f9d55c00640 1 -- 172.21.15.31:0/403406465 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d5004f400 con 0x7f9d50103660 2024-03-21T10:51:17.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.749+0000 7f9d4cc00640 1 -- 172.21.15.31:0/403406465 <== mon.1 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f9d400109d0 con 0x7f9d50103660 2024-03-21T10:51:17.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.750+0000 7f9d4cc00640 1 --2- 172.21.15.31:0/403406465 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9d340a3090 0x7f9d340a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:17.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.750+0000 7f9d4cc00640 1 -- 172.21.15.31:0/403406465 <== mon.1 v2:172.21.15.70:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 9446+0+0 (secure 0 0 0) 0x7f9d400cf7b0 con 0x7f9d50103660 2024-03-21T10:51:17.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.751+0000 7f9d4f400640 1 --2- 172.21.15.31:0/403406465 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9d340a3090 0x7f9d340a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:17.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.751+0000 7f9d4f400640 1 --2- 172.21.15.31:0/403406465 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9d340a3090 0x7f9d340a5550 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9d501016f0 tx=0x7f9d3c005f90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:17.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:17.754+0000 7f9d4cc00640 1 -- 172.21.15.31:0/403406465 <== mon.1 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9d40009de0 con 0x7f9d50103660 2024-03-21T10:51:17.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.002+0000 7f9d55c00640 1 -- 172.21.15.31:0/403406465 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f9d5010b640 con 0x7f9d50103660 2024-03-21T10:51:18.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.010+0000 7f9d4cc00640 1 -- 172.21.15.31:0/403406465 <== mon.1 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v33) v1 ==== 161+0+0 (secure 0 0 0) 0x7f9d4009bea0 con 0x7f9d50103660 2024-03-21T10:51:18.007 INFO:teuthology.orchestra.run.smithi031.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-21T10:51:18.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.014+0000 7f9d55c00640 1 -- 172.21.15.31:0/403406465 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9d340a3090 msgr2=0x7f9d340a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:18.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.014+0000 7f9d55c00640 1 --2- 172.21.15.31:0/403406465 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9d340a3090 0x7f9d340a5550 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9d501016f0 tx=0x7f9d3c005f90 comp rx=0 tx=0).stop 2024-03-21T10:51:18.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.014+0000 7f9d55c00640 1 -- 172.21.15.31:0/403406465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d50103660 msgr2=0x7f9d50199f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:18.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.014+0000 7f9d55c00640 1 --2- 172.21.15.31:0/403406465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d50103660 0x7f9d50199f20 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f9d4000d950 tx=0x7f9d4000de10 comp rx=0 tx=0).stop 2024-03-21T10:51:18.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.014+0000 7f9d55c00640 1 -- 172.21.15.31:0/403406465 shutdown_connections 2024-03-21T10:51:18.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.014+0000 7f9d55c00640 1 --2- 172.21.15.31:0/403406465 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f9d340a3090 0x7f9d340a5550 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:18.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.014+0000 7f9d55c00640 1 --2- 172.21.15.31:0/403406465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d50109670 0x7f9d5019e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:18.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.014+0000 7f9d55c00640 1 --2- 172.21.15.31:0/403406465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d50104020 0x7f9d5019a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:18.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.014+0000 7f9d55c00640 1 --2- 172.21.15.31:0/403406465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d50103660 0x7f9d50199f20 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:18.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.014+0000 7f9d55c00640 1 -- 172.21.15.31:0/403406465 >> 172.21.15.31:0/403406465 conn(0x7f9d500ff110 msgr2=0x7f9d500ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:18.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.014+0000 7f9d55c00640 1 -- 172.21.15.31:0/403406465 shutdown_connections 2024-03-21T10:51:18.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.014+0000 7f9d55c00640 1 -- 172.21.15.31:0/403406465 wait complete. 2024-03-21T10:51:18.031 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-21T10:51:18.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.237+0000 7fc662400640 1 -- 172.21.15.31:0/1742164738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc65c107640 msgr2=0x7fc65c107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:18.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.237+0000 7fc662400640 1 --2- 172.21.15.31:0/1742164738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc65c107640 0x7fc65c107a20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc64c009cd0 tx=0x7fc64c02c0f0 comp rx=0 tx=0).stop 2024-03-21T10:51:18.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.237+0000 7fc662400640 1 -- 172.21.15.31:0/1742164738 shutdown_connections 2024-03-21T10:51:18.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.237+0000 7fc662400640 1 --2- 172.21.15.31:0/1742164738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc65c101ff0 0x7fc65c10e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:18.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.237+0000 7fc662400640 1 --2- 172.21.15.31:0/1742164738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc65c101630 0x7fc65c101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:18.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.237+0000 7fc662400640 1 --2- 172.21.15.31:0/1742164738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc65c107640 0x7fc65c107a20 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:18.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.237+0000 7fc662400640 1 -- 172.21.15.31:0/1742164738 >> 172.21.15.31:0/1742164738 conn(0x7fc65c0fd140 msgr2=0x7fc65c0ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:18.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.238+0000 7fc662400640 1 -- 172.21.15.31:0/1742164738 shutdown_connections 2024-03-21T10:51:18.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.238+0000 7fc662400640 1 -- 172.21.15.31:0/1742164738 wait complete. 2024-03-21T10:51:18.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.238+0000 7fc662400640 1 Processor -- start 2024-03-21T10:51:18.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.239+0000 7fc662400640 1 -- start start 2024-03-21T10:51:18.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.239+0000 7fc662400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc65c101630 0x7fc65c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:18.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.239+0000 7fc662400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc65c101ff0 0x7fc65c1984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:18.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.239+0000 7fc662400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc65c107640 0x7fc65c19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:18.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.239+0000 7fc662400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c111a10 con 0x7fc65c101ff0 2024-03-21T10:51:18.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.239+0000 7fc662400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c111890 con 0x7fc65c107640 2024-03-21T10:51:18.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.239+0000 7fc662400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c111b90 con 0x7fc65c101630 2024-03-21T10:51:18.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.239+0000 7fc65b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc65c101630 0x7fc65c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:18.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.239+0000 7fc65aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc65c101ff0 0x7fc65c1984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:18.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.239+0000 7fc65b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc65c101630 0x7fc65c197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.31:40234/0 (socket says 172.21.15.31:40234) 2024-03-21T10:51:18.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.239+0000 7fc65b400640 1 -- 172.21.15.31:0/1325560965 learned_addr learned my addr 172.21.15.31:0/1325560965 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:51:18.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.240+0000 7fc65be00640 1 --2- 172.21.15.31:0/1325560965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc65c107640 0x7fc65c19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:18.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.240+0000 7fc65be00640 1 -- 172.21.15.31:0/1325560965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc65c101630 msgr2=0x7fc65c197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:18.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.240+0000 7fc65be00640 1 --2- 172.21.15.31:0/1325560965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc65c101630 0x7fc65c197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:18.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.240+0000 7fc65be00640 1 -- 172.21.15.31:0/1325560965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc65c101ff0 msgr2=0x7fc65c1984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:18.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.240+0000 7fc65be00640 1 --2- 172.21.15.31:0/1325560965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc65c101ff0 0x7fc65c1984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:18.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.240+0000 7fc65be00640 1 -- 172.21.15.31:0/1325560965 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65c19cf60 con 0x7fc65c107640 2024-03-21T10:51:18.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.240+0000 7fc65b400640 1 --2- 172.21.15.31:0/1325560965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc65c101630 0x7fc65c197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:51:18.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.240+0000 7fc65aa00640 1 --2- 172.21.15.31:0/1325560965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc65c101ff0 0x7fc65c1984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:51:18.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.240+0000 7fc65be00640 1 --2- 172.21.15.31:0/1325560965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc65c107640 0x7fc65c19c830 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fc65400e970 tx=0x7fc65400ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:18.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.241+0000 7fc653e00640 1 -- 172.21.15.31:0/1325560965 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65400ccc0 con 0x7fc65c107640 2024-03-21T10:51:18.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.241+0000 7fc653e00640 1 -- 172.21.15.31:0/1325560965 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc6540035f0 con 0x7fc65c107640 2024-03-21T10:51:18.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.241+0000 7fc662400640 1 -- 172.21.15.31:0/1325560965 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc65c19d250 con 0x7fc65c107640 2024-03-21T10:51:18.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.241+0000 7fc653e00640 1 -- 172.21.15.31:0/1325560965 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65400ce60 con 0x7fc65c107640 2024-03-21T10:51:18.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.241+0000 7fc662400640 1 -- 172.21.15.31:0/1325560965 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc65c1a4ae0 con 0x7fc65c107640 2024-03-21T10:51:18.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.242+0000 7fc662400640 1 -- 172.21.15.31:0/1325560965 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc65c068970 con 0x7fc65c107640 2024-03-21T10:51:18.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.244+0000 7fc653e00640 1 -- 172.21.15.31:0/1325560965 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fc6540037a0 con 0x7fc65c107640 2024-03-21T10:51:18.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.245+0000 7fc653e00640 1 --2- 172.21.15.31:0/1325560965 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc6400a3110 0x7fc6400a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:18.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.245+0000 7fc653e00640 1 -- 172.21.15.31:0/1325560965 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 9616+0+0 (secure 0 0 0) 0x7fc6540cfda0 con 0x7fc65c107640 2024-03-21T10:51:18.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.246+0000 7fc65b400640 1 --2- 172.21.15.31:0/1325560965 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc6400a3110 0x7fc6400a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:18.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.246+0000 7fc65b400640 1 --2- 172.21.15.31:0/1325560965 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc6400a3110 0x7fc6400a55d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc64c006010 tx=0x7fc64c009880 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:18.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.250+0000 7fc653e00640 1 -- 172.21.15.31:0/1325560965 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc65409c410 con 0x7fc65c107640 2024-03-21T10:51:18.304 INFO:journalctl@ceph.mds.a.smithi031.stdout:Mar 21 10:51:18 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-a[72739]: ignoring --setuser ceph since I am not root 2024-03-21T10:51:18.304 INFO:journalctl@ceph.mds.a.smithi031.stdout:Mar 21 10:51:18 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-a[72739]: ignoring --setgroup ceph since I am not root 2024-03-21T10:51:18.304 INFO:journalctl@ceph.mds.a.smithi031.stdout:Mar 21 10:51:18 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-a[72739]: starting mds.a at 2024-03-21T10:51:18.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:18 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-a[31669]: 2024-03-21T10:51:18.003+0000 7fd51ca00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-21T10:51:18.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:18 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3248329835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:51:18.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:18 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/403406465' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-21T10:51:18.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:18 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-21T10:51:18.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:18 smithi031 ceph-mon[31673]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-21T10:51:18.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:18 smithi031 ceph-mon[31673]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-21T10:51:18.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:18 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-21T10:51:18.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:18 smithi031 ceph-mon[31673]: osdmap e109: 12 total, 12 up, 12 in 2024-03-21T10:51:18.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:18 smithi031 ceph-mon[31673]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-21T10:51:18.322 INFO:journalctl@ceph.mds.c.smithi070.stdout:Mar 21 10:51:18 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-c[67043]: ignoring --setuser ceph since I am not root 2024-03-21T10:51:18.322 INFO:journalctl@ceph.mds.c.smithi070.stdout:Mar 21 10:51:18 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-c[67043]: ignoring --setgroup ceph since I am not root 2024-03-21T10:51:18.322 INFO:journalctl@ceph.mds.c.smithi070.stdout:Mar 21 10:51:18 smithi070 ceph-mds[67047]: -- 172.21.15.70:0/972303815 <== mon.1 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ba8a173a40 con 0x55ba8a147000 2024-03-21T10:51:18.322 INFO:journalctl@ceph.mds.c.smithi070.stdout:Mar 21 10:51:18 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-c[67043]: starting mds.c at 2024-03-21T10:51:18.323 INFO:journalctl@ceph.mds.f.smithi070.stdout:Mar 21 10:51:18 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-f[66734]: ignoring --setuser ceph since I am not root 2024-03-21T10:51:18.323 INFO:journalctl@ceph.mds.f.smithi070.stdout:Mar 21 10:51:18 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-f[66734]: ignoring --setgroup ceph since I am not root 2024-03-21T10:51:18.323 INFO:journalctl@ceph.mds.f.smithi070.stdout:Mar 21 10:51:18 smithi070 ceph-mds[66738]: -- 172.21.15.70:0/3070833791 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5601f833da40 con 0x5601f8310800 2024-03-21T10:51:18.323 INFO:journalctl@ceph.mds.f.smithi070.stdout:Mar 21 10:51:18 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-f[66734]: starting mds.f at 2024-03-21T10:51:18.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:18 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3248329835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:51:18.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:18 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/403406465' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-21T10:51:18.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:18 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-21T10:51:18.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:18 smithi070 ceph-mon[36330]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-21T10:51:18.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:18 smithi070 ceph-mon[36330]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-21T10:51:18.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:18 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-21T10:51:18.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:18 smithi070 ceph-mon[36330]: osdmap e109: 12 total, 12 up, 12 in 2024-03-21T10:51:18.324 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:18 smithi070 ceph-mon[36330]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-21T10:51:18.324 INFO:journalctl@ceph.mds.i.smithi070.stdout:Mar 21 10:51:18 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-i[67353]: ignoring --setuser ceph since I am not root 2024-03-21T10:51:18.324 INFO:journalctl@ceph.mds.i.smithi070.stdout:Mar 21 10:51:18 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-i[67353]: ignoring --setgroup ceph since I am not root 2024-03-21T10:51:18.324 INFO:journalctl@ceph.mds.i.smithi070.stdout:Mar 21 10:51:18 smithi070 ceph-mds[67357]: -- 172.21.15.70:0/1695343401 <== mon.1 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564054b1ba40 con 0x564054aef000 2024-03-21T10:51:18.325 INFO:journalctl@ceph.mds.i.smithi070.stdout:Mar 21 10:51:18 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-i[67353]: starting mds.i at 2024-03-21T10:51:18.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:18 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3248329835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T10:51:18.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:18 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/403406465' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-21T10:51:18.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:18 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-21T10:51:18.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:18 smithi040 ceph-mon[36154]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-21T10:51:18.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:18 smithi040 ceph-mon[36154]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-21T10:51:18.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:18 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-21T10:51:18.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:18 smithi040 ceph-mon[36154]: osdmap e109: 12 total, 12 up, 12 in 2024-03-21T10:51:18.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:18 smithi040 ceph-mon[36154]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-21T10:51:18.431 INFO:journalctl@ceph.mds.b.smithi040.stdout:Mar 21 10:51:18 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-b[66460]: ignoring --setuser ceph since I am not root 2024-03-21T10:51:18.431 INFO:journalctl@ceph.mds.b.smithi040.stdout:Mar 21 10:51:18 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-b[66460]: ignoring --setgroup ceph since I am not root 2024-03-21T10:51:18.431 INFO:journalctl@ceph.mds.b.smithi040.stdout:Mar 21 10:51:18 smithi040 ceph-mds[66464]: -- 172.21.15.40:0/604312809 <== mon.1 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5650b9b51a40 con 0x5650b9b24800 2024-03-21T10:51:18.431 INFO:journalctl@ceph.mds.b.smithi040.stdout:Mar 21 10:51:18 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-b[66460]: starting mds.b at 2024-03-21T10:51:18.431 INFO:journalctl@ceph.mds.h.smithi040.stdout:Mar 21 10:51:18 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-h[66136]: ignoring --setuser ceph since I am not root 2024-03-21T10:51:18.431 INFO:journalctl@ceph.mds.h.smithi040.stdout:Mar 21 10:51:18 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-h[66136]: ignoring --setgroup ceph since I am not root 2024-03-21T10:51:18.431 INFO:journalctl@ceph.mds.h.smithi040.stdout:Mar 21 10:51:18 smithi040 ceph-mds[66140]: -- 172.21.15.40:0/384815540 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b84a7fda40 con 0x55b84a7d0800 2024-03-21T10:51:18.432 INFO:journalctl@ceph.mds.h.smithi040.stdout:Mar 21 10:51:18 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-h[66136]: starting mds.h at 2024-03-21T10:51:18.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:18.498+0000 7fc662400640 1 -- 172.21.15.31:0/1325560965 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc65c02d070 con 0x7fc65c107640 2024-03-21T10:51:19.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.013+0000 7fc653e00640 1 -- 172.21.15.31:0/1325560965 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v34) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc65409bd10 con 0x7fc65c107640 2024-03-21T10:51:19.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.019+0000 7fc662400640 1 -- 172.21.15.31:0/1325560965 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc6400a3110 msgr2=0x7fc6400a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:19.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.019+0000 7fc662400640 1 --2- 172.21.15.31:0/1325560965 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc6400a3110 0x7fc6400a55d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc64c006010 tx=0x7fc64c009880 comp rx=0 tx=0).stop 2024-03-21T10:51:19.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.019+0000 7fc662400640 1 -- 172.21.15.31:0/1325560965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc65c107640 msgr2=0x7fc65c19c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:19.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.019+0000 7fc662400640 1 --2- 172.21.15.31:0/1325560965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc65c107640 0x7fc65c19c830 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fc65400e970 tx=0x7fc65400ee30 comp rx=0 tx=0).stop 2024-03-21T10:51:19.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.019+0000 7fc662400640 1 -- 172.21.15.31:0/1325560965 shutdown_connections 2024-03-21T10:51:19.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.019+0000 7fc662400640 1 --2- 172.21.15.31:0/1325560965 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc6400a3110 0x7fc6400a55d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:19.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.019+0000 7fc662400640 1 --2- 172.21.15.31:0/1325560965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc65c107640 0x7fc65c19c830 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:19.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.019+0000 7fc662400640 1 --2- 172.21.15.31:0/1325560965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc65c101ff0 0x7fc65c1984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:19.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.019+0000 7fc662400640 1 --2- 172.21.15.31:0/1325560965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc65c101630 0x7fc65c197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:19.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.019+0000 7fc662400640 1 -- 172.21.15.31:0/1325560965 >> 172.21.15.31:0/1325560965 conn(0x7fc65c0fd140 msgr2=0x7fc65c0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:19.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.020+0000 7fc662400640 1 -- 172.21.15.31:0/1325560965 shutdown_connections 2024-03-21T10:51:19.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.020+0000 7fc662400640 1 -- 172.21.15.31:0/1325560965 wait complete. 2024-03-21T10:51:19.038 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-21T10:51:19.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.265+0000 7f16a6c00640 1 -- 172.21.15.31:0/4041879661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16a0109670 msgr2=0x7f16a0109a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:19.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.265+0000 7f16a6c00640 1 --2- 172.21.15.31:0/4041879661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16a0109670 0x7f16a0109a50 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f1694009ca0 tx=0x7f169402c0f0 comp rx=0 tx=0).stop 2024-03-21T10:51:19.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.265+0000 7f16a6c00640 1 -- 172.21.15.31:0/4041879661 shutdown_connections 2024-03-21T10:51:19.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.265+0000 7f16a6c00640 1 --2- 172.21.15.31:0/4041879661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16a0104020 0x7f16a0110530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:19.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.265+0000 7f16a6c00640 1 --2- 172.21.15.31:0/4041879661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16a0103660 0x7f16a0103ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:19.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.265+0000 7f16a6c00640 1 --2- 172.21.15.31:0/4041879661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16a0109670 0x7f16a0109a50 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:19.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.265+0000 7f16a6c00640 1 -- 172.21.15.31:0/4041879661 >> 172.21.15.31:0/4041879661 conn(0x7f16a00ff150 msgr2=0x7f16a0101570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:19.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.265+0000 7f16a6c00640 1 -- 172.21.15.31:0/4041879661 shutdown_connections 2024-03-21T10:51:19.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.265+0000 7f16a6c00640 1 -- 172.21.15.31:0/4041879661 wait complete. 2024-03-21T10:51:19.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.266+0000 7f16a6c00640 1 Processor -- start 2024-03-21T10:51:19.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.266+0000 7f16a6c00640 1 -- start start 2024-03-21T10:51:19.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.267+0000 7f16a6c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16a0103660 0x7f16a0075bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:19.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.267+0000 7f16a6c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16a0104020 0x7f16a00760f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:19.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.267+0000 7f16a6c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16a0109670 0x7f16a0076630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:19.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.267+0000 7f16a6c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a0113a40 con 0x7f16a0109670 2024-03-21T10:51:19.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.267+0000 7f16a6c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a01138c0 con 0x7f16a0103660 2024-03-21T10:51:19.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.267+0000 7f16a6c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a0113bc0 con 0x7f16a0104020 2024-03-21T10:51:19.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.267+0000 7f16a5800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16a0103660 0x7f16a0075bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:19.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.267+0000 7f16a5800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16a0103660 0x7f16a0075bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:52162/0 (socket says 172.21.15.31:52162) 2024-03-21T10:51:19.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.267+0000 7f16a5800640 1 -- 172.21.15.31:0/454833039 learned_addr learned my addr 172.21.15.31:0/454833039 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:51:19.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.267+0000 7f16a6200640 1 --2- 172.21.15.31:0/454833039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16a0109670 0x7f16a0076630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:19.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.267+0000 7f16a4e00640 1 --2- 172.21.15.31:0/454833039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16a0104020 0x7f16a00760f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:19.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.268+0000 7f16a5800640 1 -- 172.21.15.31:0/454833039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16a0104020 msgr2=0x7f16a00760f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:19.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.268+0000 7f16a5800640 1 --2- 172.21.15.31:0/454833039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16a0104020 0x7f16a00760f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:19.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.268+0000 7f16a5800640 1 -- 172.21.15.31:0/454833039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16a0109670 msgr2=0x7f16a0076630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:19.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.268+0000 7f16a5800640 1 --2- 172.21.15.31:0/454833039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16a0109670 0x7f16a0076630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:19.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.268+0000 7f16a5800640 1 -- 172.21.15.31:0/454833039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a01aa630 con 0x7f16a0103660 2024-03-21T10:51:19.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.268+0000 7f16a6200640 1 --2- 172.21.15.31:0/454833039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16a0109670 0x7f16a0076630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:51:19.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.268+0000 7f16a5800640 1 --2- 172.21.15.31:0/454833039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16a0103660 0x7f16a0075bb0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f169402c700 tx=0x7f1694032f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:19.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.268+0000 7f169e000640 1 -- 172.21.15.31:0/454833039 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1694042020 con 0x7f16a0103660 2024-03-21T10:51:19.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.268+0000 7f16a6c00640 1 -- 172.21.15.31:0/454833039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16a01aa8c0 con 0x7f16a0103660 2024-03-21T10:51:19.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.269+0000 7f169e000640 1 -- 172.21.15.31:0/454833039 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1694034640 con 0x7f16a0103660 2024-03-21T10:51:19.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.269+0000 7f169e000640 1 -- 172.21.15.31:0/454833039 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1694039040 con 0x7f16a0103660 2024-03-21T10:51:19.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.269+0000 7f16a6c00640 1 -- 172.21.15.31:0/454833039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16a01aae00 con 0x7f16a0103660 2024-03-21T10:51:19.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.270+0000 7f16a6c00640 1 -- 172.21.15.31:0/454833039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16a0104da0 con 0x7f16a0103660 2024-03-21T10:51:19.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.272+0000 7f169e000640 1 -- 172.21.15.31:0/454833039 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f1694045020 con 0x7f16a0103660 2024-03-21T10:51:19.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.273+0000 7f169e000640 1 --2- 172.21.15.31:0/454833039 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f16880a3090 0x7f16880a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:19.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.273+0000 7f169e000640 1 -- 172.21.15.31:0/454833039 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 9616+0+0 (secure 0 0 0) 0x7f16940f1720 con 0x7f16a0103660 2024-03-21T10:51:19.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.273+0000 7f16a4e00640 1 --2- 172.21.15.31:0/454833039 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f16880a3090 0x7f16880a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:19.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.274+0000 7f16a4e00640 1 --2- 172.21.15.31:0/454833039 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f16880a3090 0x7f16880a5550 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f168c0095f0 tx=0x7f168c005f90 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:19.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.277+0000 7f169e000640 1 -- 172.21.15.31:0/454833039 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f169404c020 con 0x7f16a0103660 2024-03-21T10:51:19.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:19 smithi031 ceph-mon[31673]: pgmap v399: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-21T10:51:19.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:19 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/1325560965' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T10:51:19.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:19 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T10:51:19.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:19 smithi031 ceph-mon[31673]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-21T10:51:19.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:19 smithi031 ceph-mon[31673]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-21T10:51:19.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:19 smithi031 ceph-mon[31673]: Cluster is now healthy 2024-03-21T10:51:19.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:19 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-21T10:51:19.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:19 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-21T10:51:19.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:19 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-21T10:51:19.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:19 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-21T10:51:19.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:19 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-21T10:51:19.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:19 smithi031 ceph-mon[31673]: mds.? [v2:172.21.15.40:6836/2763613456,v1:172.21.15.40:6838/2763613456] up:boot 2024-03-21T10:51:19.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:19 smithi031 ceph-mon[31673]: mds.? [v2:172.21.15.70:6836/4269676585,v1:172.21.15.70:6839/4269676585] up:boot 2024-03-21T10:51:19.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:19 smithi031 ceph-mon[31673]: mds.? [v2:172.21.15.70:6835/32076223,v1:172.21.15.70:6838/32076223] up:boot 2024-03-21T10:51:19.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:19 smithi031 ceph-mon[31673]: mds.? [v2:172.21.15.70:6834/3969584359,v1:172.21.15.70:6837/3969584359] up:boot 2024-03-21T10:51:19.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:19 smithi031 ceph-mon[31673]: mds.? [v2:172.21.15.40:6837/466860031,v1:172.21.15.40:6839/466860031] up:boot 2024-03-21T10:51:19.305 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:19 smithi031 ceph-mon[31673]: mds.? [v2:172.21.15.31:6840/135313925,v1:172.21.15.31:6841/135313925] up:boot 2024-03-21T10:51:19.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:19 smithi031 ceph-mon[31673]: fsmap 12 up:standby 2024-03-21T10:51:19.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:19 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-21T10:51:19.306 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:19 smithi031 ceph-mon[31673]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-21T10:51:19.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:19 smithi040 ceph-mon[36154]: pgmap v399: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-21T10:51:19.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:19 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/1325560965' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T10:51:19.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:19 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T10:51:19.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:19 smithi040 ceph-mon[36154]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-21T10:51:19.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:19 smithi040 ceph-mon[36154]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-21T10:51:19.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:19 smithi040 ceph-mon[36154]: Cluster is now healthy 2024-03-21T10:51:19.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:19 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-21T10:51:19.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:19 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-21T10:51:19.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:19 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-21T10:51:19.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:19 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-21T10:51:19.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:19 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-21T10:51:19.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:19 smithi040 ceph-mon[36154]: mds.? [v2:172.21.15.40:6836/2763613456,v1:172.21.15.40:6838/2763613456] up:boot 2024-03-21T10:51:19.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:19 smithi040 ceph-mon[36154]: mds.? [v2:172.21.15.70:6836/4269676585,v1:172.21.15.70:6839/4269676585] up:boot 2024-03-21T10:51:19.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:19 smithi040 ceph-mon[36154]: mds.? [v2:172.21.15.70:6835/32076223,v1:172.21.15.70:6838/32076223] up:boot 2024-03-21T10:51:19.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:19 smithi040 ceph-mon[36154]: mds.? [v2:172.21.15.70:6834/3969584359,v1:172.21.15.70:6837/3969584359] up:boot 2024-03-21T10:51:19.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:19 smithi040 ceph-mon[36154]: mds.? [v2:172.21.15.40:6837/466860031,v1:172.21.15.40:6839/466860031] up:boot 2024-03-21T10:51:19.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:19 smithi040 ceph-mon[36154]: mds.? [v2:172.21.15.31:6840/135313925,v1:172.21.15.31:6841/135313925] up:boot 2024-03-21T10:51:19.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:19 smithi040 ceph-mon[36154]: fsmap 12 up:standby 2024-03-21T10:51:19.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:19 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-21T10:51:19.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:19 smithi040 ceph-mon[36154]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-21T10:51:19.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:19.473+0000 7f16a6c00640 1 -- 172.21.15.31:0/454833039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f16a010bbd0 con 0x7f16a0103660 2024-03-21T10:51:19.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:19 smithi070 ceph-mon[36330]: pgmap v399: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-21T10:51:19.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:19 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/1325560965' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T10:51:19.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:19 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T10:51:19.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:19 smithi070 ceph-mon[36330]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-21T10:51:19.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:19 smithi070 ceph-mon[36330]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-21T10:51:19.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:19 smithi070 ceph-mon[36330]: Cluster is now healthy 2024-03-21T10:51:19.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:19 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-21T10:51:19.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:19 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-21T10:51:19.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:19 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-21T10:51:19.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:19 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-21T10:51:19.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:19 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-21T10:51:19.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:19 smithi070 ceph-mon[36330]: mds.? [v2:172.21.15.40:6836/2763613456,v1:172.21.15.40:6838/2763613456] up:boot 2024-03-21T10:51:19.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:19 smithi070 ceph-mon[36330]: mds.? [v2:172.21.15.70:6836/4269676585,v1:172.21.15.70:6839/4269676585] up:boot 2024-03-21T10:51:19.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:19 smithi070 ceph-mon[36330]: mds.? [v2:172.21.15.70:6835/32076223,v1:172.21.15.70:6838/32076223] up:boot 2024-03-21T10:51:19.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:19 smithi070 ceph-mon[36330]: mds.? [v2:172.21.15.70:6834/3969584359,v1:172.21.15.70:6837/3969584359] up:boot 2024-03-21T10:51:19.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:19 smithi070 ceph-mon[36330]: mds.? [v2:172.21.15.40:6837/466860031,v1:172.21.15.40:6839/466860031] up:boot 2024-03-21T10:51:19.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:19 smithi070 ceph-mon[36330]: mds.? [v2:172.21.15.31:6840/135313925,v1:172.21.15.31:6841/135313925] up:boot 2024-03-21T10:51:19.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:19 smithi070 ceph-mon[36330]: fsmap 12 up:standby 2024-03-21T10:51:19.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:19 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-21T10:51:19.574 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:19 smithi070 ceph-mon[36330]: from='mgr.25564 172.21.15.31:0/3611906474' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-21T10:51:20.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.158+0000 7f169e000640 1 -- 172.21.15.31:0/454833039 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v110) v1 ==== 181+0+0 (secure 0 0 0) 0x7f16940bdea0 con 0x7f16a0103660 2024-03-21T10:51:20.155 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' removed 2024-03-21T10:51:20.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.163+0000 7f16a6c00640 1 -- 172.21.15.31:0/454833039 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f16880a3090 msgr2=0x7f16880a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:20.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.163+0000 7f16a6c00640 1 --2- 172.21.15.31:0/454833039 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f16880a3090 0x7f16880a5550 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f168c0095f0 tx=0x7f168c005f90 comp rx=0 tx=0).stop 2024-03-21T10:51:20.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.163+0000 7f16a6c00640 1 -- 172.21.15.31:0/454833039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16a0103660 msgr2=0x7f16a0075bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:20.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.163+0000 7f16a6c00640 1 --2- 172.21.15.31:0/454833039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16a0103660 0x7f16a0075bb0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f169402c700 tx=0x7f1694032f20 comp rx=0 tx=0).stop 2024-03-21T10:51:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.163+0000 7f16a6c00640 1 -- 172.21.15.31:0/454833039 shutdown_connections 2024-03-21T10:51:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.163+0000 7f16a6c00640 1 --2- 172.21.15.31:0/454833039 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f16880a3090 0x7f16880a5550 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.163+0000 7f16a6c00640 1 --2- 172.21.15.31:0/454833039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16a0109670 0x7f16a0076630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.163+0000 7f16a6c00640 1 --2- 172.21.15.31:0/454833039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16a0104020 0x7f16a00760f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.163+0000 7f16a6c00640 1 --2- 172.21.15.31:0/454833039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16a0103660 0x7f16a0075bb0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.163+0000 7f16a6c00640 1 -- 172.21.15.31:0/454833039 >> 172.21.15.31:0/454833039 conn(0x7f16a00ff150 msgr2=0x7f16a00ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:20.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.164+0000 7f16a6c00640 1 -- 172.21.15.31:0/454833039 shutdown_connections 2024-03-21T10:51:20.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.164+0000 7f16a6c00640 1 -- 172.21.15.31:0/454833039 wait complete. 2024-03-21T10:51:20.185 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-21T10:51:20.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.412+0000 7f247a400640 1 -- 172.21.15.31:0/2890808527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2474073a40 msgr2=0x7f2474073e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:20.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.412+0000 7f247a400640 1 --2- 172.21.15.31:0/2890808527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2474073a40 0x7f2474073e20 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f2464009ca0 tx=0x7f246402c180 comp rx=0 tx=0).stop 2024-03-21T10:51:20.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.412+0000 7f247a400640 1 -- 172.21.15.31:0/2890808527 shutdown_connections 2024-03-21T10:51:20.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.412+0000 7f247a400640 1 --2- 172.21.15.31:0/2890808527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2474112470 0x7f2474114860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:20.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.412+0000 7f247a400640 1 --2- 172.21.15.31:0/2890808527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2474074360 0x7f2474111f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:20.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.412+0000 7f247a400640 1 --2- 172.21.15.31:0/2890808527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2474073a40 0x7f2474073e20 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:20.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.412+0000 7f247a400640 1 -- 172.21.15.31:0/2890808527 >> 172.21.15.31:0/2890808527 conn(0x7f24740fcff0 msgr2=0x7f24740ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:20.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.412+0000 7f247a400640 1 -- 172.21.15.31:0/2890808527 shutdown_connections 2024-03-21T10:51:20.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.412+0000 7f247a400640 1 -- 172.21.15.31:0/2890808527 wait complete. 2024-03-21T10:51:20.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.413+0000 7f247a400640 1 Processor -- start 2024-03-21T10:51:20.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.414+0000 7f247a400640 1 -- start start 2024-03-21T10:51:20.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.414+0000 7f247a400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2474073a40 0x7f247419e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:20.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.414+0000 7f247a400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2474074360 0x7f247419e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:20.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.414+0000 7f247a400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2474112470 0x7f24741a2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:20.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.414+0000 7f247a400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2474117e70 con 0x7f2474112470 2024-03-21T10:51:20.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.414+0000 7f247a400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2474117cf0 con 0x7f2474073a40 2024-03-21T10:51:20.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.414+0000 7f247a400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2474117ff0 con 0x7f2474074360 2024-03-21T10:51:20.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.414+0000 7f2479000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2474073a40 0x7f247419e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:20.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.415+0000 7f2479a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2474112470 0x7f24741a2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:20.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.414+0000 7f2479000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2474073a40 0x7f247419e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:52172/0 (socket says 172.21.15.31:52172) 2024-03-21T10:51:20.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.415+0000 7f2479000640 1 -- 172.21.15.31:0/3061373506 learned_addr learned my addr 172.21.15.31:0/3061373506 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:51:20.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.415+0000 7f2473e00640 1 --2- 172.21.15.31:0/3061373506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2474074360 0x7f247419e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:20.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.415+0000 7f2479a00640 1 -- 172.21.15.31:0/3061373506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2474074360 msgr2=0x7f247419e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:20.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.415+0000 7f2479a00640 1 --2- 172.21.15.31:0/3061373506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2474074360 0x7f247419e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:20.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.415+0000 7f2479a00640 1 -- 172.21.15.31:0/3061373506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2474073a40 msgr2=0x7f247419e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:20.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.415+0000 7f2479a00640 1 --2- 172.21.15.31:0/3061373506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2474073a40 0x7f247419e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:20.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.415+0000 7f2479a00640 1 -- 172.21.15.31:0/3061373506 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24741a3350 con 0x7f2474112470 2024-03-21T10:51:20.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.415+0000 7f2479000640 1 --2- 172.21.15.31:0/3061373506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2474073a40 0x7f247419e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T10:51:20.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.415+0000 7f2479a00640 1 --2- 172.21.15.31:0/3061373506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2474112470 0x7f24741a2c20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f246c00e940 tx=0x7f246c00ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:20.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.416+0000 7f2471600640 1 -- 172.21.15.31:0/3061373506 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c00ccc0 con 0x7f2474112470 2024-03-21T10:51:20.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.416+0000 7f2471600640 1 -- 172.21.15.31:0/3061373506 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f246c005020 con 0x7f2474112470 2024-03-21T10:51:20.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.416+0000 7f247a400640 1 -- 172.21.15.31:0/3061373506 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24741a3640 con 0x7f2474112470 2024-03-21T10:51:20.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.416+0000 7f2471600640 1 -- 172.21.15.31:0/3061373506 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c004a00 con 0x7f2474112470 2024-03-21T10:51:20.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.416+0000 7f247a400640 1 -- 172.21.15.31:0/3061373506 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24741aaed0 con 0x7f2474112470 2024-03-21T10:51:20.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.417+0000 7f247a400640 1 -- 172.21.15.31:0/3061373506 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f247404f400 con 0x7f2474112470 2024-03-21T10:51:20.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.419+0000 7f2471600640 1 -- 172.21.15.31:0/3061373506 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7f246c0051d0 con 0x7f2474112470 2024-03-21T10:51:20.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.420+0000 7f2471600640 1 --2- 172.21.15.31:0/3061373506 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24580a3110 0x7f24580a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:20.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.420+0000 7f2471600640 1 -- 172.21.15.31:0/3061373506 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 9120+0+0 (secure 0 0 0) 0x7f246c0cf920 con 0x7f2474112470 2024-03-21T10:51:20.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.421+0000 7f2479000640 1 --2- 172.21.15.31:0/3061373506 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24580a3110 0x7f24580a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:20.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.422+0000 7f2479000640 1 --2- 172.21.15.31:0/3061373506 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24580a3110 0x7f24580a55d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2464009a60 tx=0x7f2464005d80 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:20.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.425+0000 7f2471600640 1 -- 172.21.15.31:0/3061373506 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f246c09c180 con 0x7f2474112470 2024-03-21T10:51:20.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:20 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/454833039' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T10:51:20.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:20 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T10:51:20.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:20 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/454833039' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T10:51:20.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:20 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T10:51:20.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:20 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/454833039' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T10:51:20.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:20 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T10:51:20.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:20.629+0000 7f247a400640 1 -- 172.21.15.31:0/3061373506 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f24741043d0 con 0x7f2474112470 2024-03-21T10:51:21.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.158+0000 7f2471600640 1 -- 172.21.15.31:0/3061373506 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v111) v1 ==== 169+0+0 (secure 0 0 0) 0x7f246c09ba80 con 0x7f2474112470 2024-03-21T10:51:21.154 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' removed 2024-03-21T10:51:21.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.161+0000 7f247a400640 1 -- 172.21.15.31:0/3061373506 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24580a3110 msgr2=0x7f24580a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:21.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.161+0000 7f247a400640 1 --2- 172.21.15.31:0/3061373506 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24580a3110 0x7f24580a55d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2464009a60 tx=0x7f2464005d80 comp rx=0 tx=0).stop 2024-03-21T10:51:21.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.161+0000 7f247a400640 1 -- 172.21.15.31:0/3061373506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2474112470 msgr2=0x7f24741a2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:21.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.161+0000 7f247a400640 1 --2- 172.21.15.31:0/3061373506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2474112470 0x7f24741a2c20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f246c00e940 tx=0x7f246c00ee00 comp rx=0 tx=0).stop 2024-03-21T10:51:21.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.164+0000 7f247a400640 1 -- 172.21.15.31:0/3061373506 shutdown_connections 2024-03-21T10:51:21.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.164+0000 7f247a400640 1 --2- 172.21.15.31:0/3061373506 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7f24580a3110 0x7f24580a55d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:21.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.164+0000 7f247a400640 1 --2- 172.21.15.31:0/3061373506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2474112470 0x7f24741a2c20 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:21.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.164+0000 7f247a400640 1 --2- 172.21.15.31:0/3061373506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2474074360 0x7f247419e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:21.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.164+0000 7f247a400640 1 --2- 172.21.15.31:0/3061373506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2474073a40 0x7f247419e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:21.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.164+0000 7f247a400640 1 -- 172.21.15.31:0/3061373506 >> 172.21.15.31:0/3061373506 conn(0x7f24740fcff0 msgr2=0x7f24740fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:21.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.165+0000 7f247a400640 1 -- 172.21.15.31:0/3061373506 shutdown_connections 2024-03-21T10:51:21.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.165+0000 7f247a400640 1 -- 172.21.15.31:0/3061373506 wait complete. 2024-03-21T10:51:21.189 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-21T10:51:21.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.412+0000 7fc1dca00640 1 -- 172.21.15.31:0/2331066643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1d810c980 msgr2=0x7fc1d810cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:21.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.412+0000 7fc1dca00640 1 --2- 172.21.15.31:0/2331066643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1d810c980 0x7fc1d810cd60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc1c8009860 tx=0x7fc1c802bc20 comp rx=0 tx=0).stop 2024-03-21T10:51:21.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.412+0000 7fc1dca00640 1 -- 172.21.15.31:0/2331066643 shutdown_connections 2024-03-21T10:51:21.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.412+0000 7fc1dca00640 1 --2- 172.21.15.31:0/2331066643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1d8108440 0x7fc1d8114900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:21.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.412+0000 7fc1dca00640 1 --2- 172.21.15.31:0/2331066643 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1d8107aa0 0x7fc1d8107f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:21.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.412+0000 7fc1dca00640 1 --2- 172.21.15.31:0/2331066643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1d810c980 0x7fc1d810cd60 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:21.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.412+0000 7fc1dca00640 1 -- 172.21.15.31:0/2331066643 >> 172.21.15.31:0/2331066643 conn(0x7fc1d80772a0 msgr2=0x7fc1d80796c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:21.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.413+0000 7fc1dca00640 1 -- 172.21.15.31:0/2331066643 shutdown_connections 2024-03-21T10:51:21.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.413+0000 7fc1dca00640 1 -- 172.21.15.31:0/2331066643 wait complete. 2024-03-21T10:51:21.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.414+0000 7fc1dca00640 1 Processor -- start 2024-03-21T10:51:21.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.414+0000 7fc1dca00640 1 -- start start 2024-03-21T10:51:21.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.414+0000 7fc1dca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1d8107aa0 0x7fc1d819e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:21.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.414+0000 7fc1dca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1d8108440 0x7fc1d819e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:21.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.414+0000 7fc1dca00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1d810c980 0x7fc1d81a2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:21.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.414+0000 7fc1dca00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d8117e70 con 0x7fc1d8108440 2024-03-21T10:51:21.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.414+0000 7fc1dca00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d8117cf0 con 0x7fc1d8107aa0 2024-03-21T10:51:21.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.414+0000 7fc1dca00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d8117ff0 con 0x7fc1d810c980 2024-03-21T10:51:21.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.414+0000 7fc1d6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1d8108440 0x7fc1d819e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:21.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.414+0000 7fc1d6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1d8108440 0x7fc1d819e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47120/0 (socket says 172.21.15.31:47120) 2024-03-21T10:51:21.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.414+0000 7fc1d6a00640 1 -- 172.21.15.31:0/2809934184 learned_addr learned my addr 172.21.15.31:0/2809934184 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-21T10:51:21.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.414+0000 7fc1d7400640 1 --2- 172.21.15.31:0/2809934184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1d8107aa0 0x7fc1d819e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:21.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.415+0000 7fc1d7e00640 1 --2- 172.21.15.31:0/2809934184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1d810c980 0x7fc1d81a2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:21.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.415+0000 7fc1d6a00640 1 -- 172.21.15.31:0/2809934184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1d810c980 msgr2=0x7fc1d81a2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:21.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.415+0000 7fc1d6a00640 1 --2- 172.21.15.31:0/2809934184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1d810c980 0x7fc1d81a2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:21.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.415+0000 7fc1d6a00640 1 -- 172.21.15.31:0/2809934184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1d8107aa0 msgr2=0x7fc1d819e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:21.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.415+0000 7fc1d6a00640 1 --2- 172.21.15.31:0/2809934184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1d8107aa0 0x7fc1d819e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:21.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.415+0000 7fc1d6a00640 1 -- 172.21.15.31:0/2809934184 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d81a3380 con 0x7fc1d8108440 2024-03-21T10:51:21.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.415+0000 7fc1d6a00640 1 --2- 172.21.15.31:0/2809934184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1d8108440 0x7fc1d819e8d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc1c000ebc0 tx=0x7fc1c000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:21.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.415+0000 7fc1cfe00640 1 -- 172.21.15.31:0/2809934184 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c0010630 con 0x7fc1d8108440 2024-03-21T10:51:21.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.415+0000 7fc1cfe00640 1 -- 172.21.15.31:0/2809934184 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc1c0010cb0 con 0x7fc1d8108440 2024-03-21T10:51:21.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.415+0000 7fc1cfe00640 1 -- 172.21.15.31:0/2809934184 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c0018bd0 con 0x7fc1d8108440 2024-03-21T10:51:21.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.416+0000 7fc1dca00640 1 -- 172.21.15.31:0/2809934184 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1d81a3670 con 0x7fc1d8108440 2024-03-21T10:51:21.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.416+0000 7fc1dca00640 1 -- 172.21.15.31:0/2809934184 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1d81aaf00 con 0x7fc1d8108440 2024-03-21T10:51:21.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.417+0000 7fc1dca00640 1 -- 172.21.15.31:0/2809934184 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1d804f400 con 0x7fc1d8108440 2024-03-21T10:51:21.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.419+0000 7fc1cfe00640 1 -- 172.21.15.31:0/2809934184 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 27) v1 ==== 137985+0+0 (secure 0 0 0) 0x7fc1c00107d0 con 0x7fc1d8108440 2024-03-21T10:51:21.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.420+0000 7fc1cfe00640 1 --2- 172.21.15.31:0/2809934184 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc1b80a3110 0x7fc1b80a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T10:51:21.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.420+0000 7fc1cfe00640 1 -- 172.21.15.31:0/2809934184 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8582+0+0 (secure 0 0 0) 0x7fc1c00cfe20 con 0x7fc1d8108440 2024-03-21T10:51:21.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.420+0000 7fc1d7400640 1 --2- 172.21.15.31:0/2809934184 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc1b80a3110 0x7fc1b80a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T10:51:21.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.421+0000 7fc1d7400640 1 --2- 172.21.15.31:0/2809934184 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc1b80a3110 0x7fc1b80a55d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc1c802c230 tx=0x7fc1c80058d0 comp rx=0 tx=0).ready entity=mgr.25564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T10:51:21.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.424+0000 7fc1cfe00640 1 -- 172.21.15.31:0/2809934184 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc1c009c220 con 0x7fc1d8108440 2024-03-21T10:51:21.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:21 smithi040 ceph-mon[36154]: pgmap v401: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1023 B/s wr, 8 op/s 2024-03-21T10:51:21.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:21 smithi040 ceph-mon[36154]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T10:51:21.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:21 smithi040 ceph-mon[36154]: osdmap e110: 12 total, 12 up, 12 in 2024-03-21T10:51:21.431 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:21 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3061373506' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T10:51:21.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:21 smithi031 ceph-mon[31673]: pgmap v401: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1023 B/s wr, 8 op/s 2024-03-21T10:51:21.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:21 smithi031 ceph-mon[31673]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T10:51:21.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:21 smithi031 ceph-mon[31673]: osdmap e110: 12 total, 12 up, 12 in 2024-03-21T10:51:21.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:21 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3061373506' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T10:51:21.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:21 smithi070 ceph-mon[36330]: pgmap v401: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1023 B/s wr, 8 op/s 2024-03-21T10:51:21.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:21 smithi070 ceph-mon[36330]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T10:51:21.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:21 smithi070 ceph-mon[36330]: osdmap e110: 12 total, 12 up, 12 in 2024-03-21T10:51:21.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:21 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3061373506' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T10:51:21.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:21.629+0000 7fc1dca00640 1 -- 172.21.15.31:0/2809934184 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc1d802d070 con 0x7fc1d8108440 2024-03-21T10:51:22.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:22.183+0000 7fc1cfe00640 1 -- 172.21.15.31:0/2809934184 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v112) v1 ==== 178+0+0 (secure 0 0 0) 0x7fc1c00a02c0 con 0x7fc1d8108440 2024-03-21T10:51:22.180 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data_ec' removed 2024-03-21T10:51:22.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:22.189+0000 7fc1dca00640 1 -- 172.21.15.31:0/2809934184 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc1b80a3110 msgr2=0x7fc1b80a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:22.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:22.189+0000 7fc1dca00640 1 --2- 172.21.15.31:0/2809934184 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc1b80a3110 0x7fc1b80a55d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc1c802c230 tx=0x7fc1c80058d0 comp rx=0 tx=0).stop 2024-03-21T10:51:22.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:22.189+0000 7fc1dca00640 1 -- 172.21.15.31:0/2809934184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1d8108440 msgr2=0x7fc1d819e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T10:51:22.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:22.189+0000 7fc1dca00640 1 --2- 172.21.15.31:0/2809934184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1d8108440 0x7fc1d819e8d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc1c000ebc0 tx=0x7fc1c000c6a0 comp rx=0 tx=0).stop 2024-03-21T10:51:22.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:22.191+0000 7fc1dca00640 1 -- 172.21.15.31:0/2809934184 shutdown_connections 2024-03-21T10:51:22.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:22.191+0000 7fc1dca00640 1 --2- 172.21.15.31:0/2809934184 >> [v2:172.21.15.31:6800/3298970029,v1:172.21.15.31:6801/3298970029] conn(0x7fc1b80a3110 0x7fc1b80a55d0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:22.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:22.191+0000 7fc1dca00640 1 --2- 172.21.15.31:0/2809934184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1d810c980 0x7fc1d81a2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:22.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:22.191+0000 7fc1dca00640 1 --2- 172.21.15.31:0/2809934184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1d8108440 0x7fc1d819e8d0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:22.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:22.191+0000 7fc1dca00640 1 --2- 172.21.15.31:0/2809934184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1d8107aa0 0x7fc1d819e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T10:51:22.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:22.191+0000 7fc1dca00640 1 -- 172.21.15.31:0/2809934184 >> 172.21.15.31:0/2809934184 conn(0x7fc1d80772a0 msgr2=0x7fc1d8077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T10:51:22.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:22.191+0000 7fc1dca00640 1 -- 172.21.15.31:0/2809934184 shutdown_connections 2024-03-21T10:51:22.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-21T10:51:22.191+0000 7fc1dca00640 1 -- 172.21.15.31:0/2809934184 wait complete. 2024-03-21T10:51:22.217 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-21T10:51:22.320 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-21T10:51:22.363 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-21T10:51:22.411 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-21T10:51:22.411 DEBUG:teuthology.orchestra.run.smithi031:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-21T10:51:22.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:22 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/3061373506' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T10:51:22.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:22 smithi040 ceph-mon[36154]: osdmap e111: 12 total, 12 up, 12 in 2024-03-21T10:51:22.430 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:22 smithi040 ceph-mon[36154]: from='client.? 172.21.15.31:0/2809934184' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T10:51:22.468 DEBUG:teuthology.orchestra.run.smithi040:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-21T10:51:22.492 DEBUG:teuthology.orchestra.run.smithi070:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-21T10:51:22.513 INFO:tasks.cephadm:Stopping all daemons... 2024-03-21T10:51:22.513 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-21T10:51:22.513 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@mon.a 2024-03-21T10:51:22.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:22 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/3061373506' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T10:51:22.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:22 smithi031 ceph-mon[31673]: osdmap e111: 12 total, 12 up, 12 in 2024-03-21T10:51:22.554 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:22 smithi031 ceph-mon[31673]: from='client.? 172.21.15.31:0/2809934184' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T10:51:22.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:22 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/3061373506' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T10:51:22.572 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:22 smithi070 ceph-mon[36330]: osdmap e111: 12 total, 12 up, 12 in 2024-03-21T10:51:22.573 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:22 smithi070 ceph-mon[36330]: from='client.? 172.21.15.31:0/2809934184' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T10:51:23.054 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:22 smithi031 systemd[1]: Stopping Ceph mon.a for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:51:23.440 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:23 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-a[31669]: 2024-03-21T10:51:23.068+0000 7fd523800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:51:23.440 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:23 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-a[31669]: 2024-03-21T10:51:23.068+0000 7fd523800700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-21T10:51:23.440 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:23 smithi031 podman[103612]: 2024-03-21 10:51:23.120693994 +0000 UTC m=+0.210577075 container died aeb8a37c34e6eca49d0cb7d648fd98e198456cb3a6bb14e092ee36f2b2d41433 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-a, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T10:51:23.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:23 smithi031 podman[103612]: 2024-03-21 10:51:23.64854653 +0000 UTC m=+0.738429629 container cleanup aeb8a37c34e6eca49d0cb7d648fd98e198456cb3a6bb14e092ee36f2b2d41433 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-21T10:51:23.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:23 smithi031 bash[103612]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-a 2024-03-21T10:51:24.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Mar 21 10:51:23 smithi031 podman[103625]: 2024-03-21 10:51:23.883456709 +0000 UTC m=+0.751158365 container remove aeb8a37c34e6eca49d0cb7d648fd98e198456cb3a6bb14e092ee36f2b2d41433 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319) 2024-03-21T10:51:24.445 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mon.a.service' 2024-03-21T10:51:24.625 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:51:24.625 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-21T10:51:24.625 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-21T10:51:24.625 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@mon.b 2024-03-21T10:51:24.930 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:24 smithi040 systemd[1]: Stopping Ceph mon.b for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:51:25.364 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:24 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-b[36150]: 2024-03-21T10:51:24.949+0000 7f1fbf0ed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:51:25.364 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:24 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-b[36150]: 2024-03-21T10:51:24.949+0000 7f1fbf0ed700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-21T10:51:25.364 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:25 smithi040 podman[72860]: 2024-03-21 10:51:25.062373127 +0000 UTC m=+0.232085726 container died 5897a61780c0a15ddb1d0ae2703d1b63c5db2466a06c05ec1a434fb317463b41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-21T10:51:25.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:25 smithi040 podman[72860]: 2024-03-21 10:51:25.467160565 +0000 UTC m=+0.636873154 container cleanup 5897a61780c0a15ddb1d0ae2703d1b63c5db2466a06c05ec1a434fb317463b41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T10:51:25.680 INFO:journalctl@ceph.mon.b.smithi040.stdout:Mar 21 10:51:25 smithi040 bash[72860]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-b 2024-03-21T10:51:25.831 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mon.b.service' 2024-03-21T10:51:25.886 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:51:25.886 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-21T10:51:25.886 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-21T10:51:25.886 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@mon.c 2024-03-21T10:51:26.181 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:25 smithi070 systemd[1]: Stopping Ceph mon.c for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:51:26.436 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:26 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-c[36312]: 2024-03-21T10:51:26.184+0000 7f9e7e7b4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:51:26.436 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:26 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-c[36312]: 2024-03-21T10:51:26.184+0000 7f9e7e7b4700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-21T10:51:26.707 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:26 smithi070 podman[73642]: 2024-03-21 10:51:26.44039608 +0000 UTC m=+0.361688121 container died 0db0cc4126096526f2e65f4b6a927ec5c3d1baac342cf8c5852a69151fb64b3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T10:51:27.072 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:26 smithi070 podman[73642]: 2024-03-21 10:51:26.810194106 +0000 UTC m=+0.731486155 container cleanup 0db0cc4126096526f2e65f4b6a927ec5c3d1baac342cf8c5852a69151fb64b3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-21T10:51:27.072 INFO:journalctl@ceph.mon.c.smithi070.stdout:Mar 21 10:51:26 smithi070 bash[73642]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-mon-c 2024-03-21T10:51:27.269 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mon.c.service' 2024-03-21T10:51:27.328 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:51:27.329 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-21T10:51:27.329 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-21T10:51:27.329 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@mgr.x 2024-03-21T10:51:27.803 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:51:27 smithi031 systemd[1]: Stopping Ceph mgr.x for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:51:28.304 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:51:27 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:51:27.902+0000 7f461ba00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:51:28.304 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:51:27 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x[31899]: 2024-03-21T10:51:27.902+0000 7f461ba00700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-21T10:51:28.304 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:51:27 smithi031 podman[103731]: 2024-03-21 10:51:27.996535836 +0000 UTC m=+0.255333792 container died d2fad9ce65f9c1e0d84494b8af7e9fc45f2757f4157fb5acc46f57c4bdd6287c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-21T10:51:28.577 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:51:28 smithi031 podman[103731]: 2024-03-21 10:51:28.570771776 +0000 UTC m=+0.829569771 container cleanup d2fad9ce65f9c1e0d84494b8af7e9fc45f2757f4157fb5acc46f57c4bdd6287c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-21T10:51:28.577 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:51:28 smithi031 bash[103731]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x 2024-03-21T10:51:29.206 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Mar 21 10:51:28 smithi031 podman[103744]: 2024-03-21 10:51:28.853954808 +0000 UTC m=+0.831655447 container remove d2fad9ce65f9c1e0d84494b8af7e9fc45f2757f4157fb5acc46f57c4bdd6287c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2edfee68ad7243bed422a049b6f5f58619300f11, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-x, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-21T10:51:29.236 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mgr.x.service' 2024-03-21T10:51:29.359 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:51:29.359 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-21T10:51:29.359 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-21T10:51:29.359 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@mgr.y 2024-03-21T10:51:29.680 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:51:29 smithi040 systemd[1]: Stopping Ceph mgr.y for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:51:31.345 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:51:31 smithi040 podman[72981]: 2024-03-21 10:51:31.005231952 +0000 UTC m=+1.414857575 container died 37bcf80653ae1042d392fa5c4deda22bfdb3ee8509c6baeb027242ee05c4cb51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-21T10:51:31.637 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:51:31 smithi040 podman[72981]: 2024-03-21 10:51:31.456777294 +0000 UTC m=+1.866402917 container cleanup 37bcf80653ae1042d392fa5c4deda22bfdb3ee8509c6baeb027242ee05c4cb51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-21T10:51:31.638 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:51:31 smithi040 bash[72981]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y 2024-03-21T10:51:31.878 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mgr.y.service' 2024-03-21T10:51:31.906 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:51:31 smithi040 podman[72995]: 2024-03-21 10:51:31.641939169 +0000 UTC m=+0.639585516 container remove 37bcf80653ae1042d392fa5c4deda22bfdb3ee8509c6baeb027242ee05c4cb51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-y, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-21T10:51:31.906 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:51:31 smithi040 systemd[1]: ceph-3847408c-e76d-11ee-95cd-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-21T10:51:31.907 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:51:31 smithi040 systemd[1]: ceph-3847408c-e76d-11ee-95cd-87774f69a715@mgr.y.service: Failed with result 'exit-code'. 2024-03-21T10:51:31.907 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:51:31 smithi040 systemd[1]: Stopped Ceph mgr.y for 3847408c-e76d-11ee-95cd-87774f69a715. 2024-03-21T10:51:31.907 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Mar 21 10:51:31 smithi040 systemd[1]: ceph-3847408c-e76d-11ee-95cd-87774f69a715@mgr.y.service: Consumed 21.091s CPU time. 2024-03-21T10:51:31.920 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:51:31.920 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-21T10:51:31.921 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-21T10:51:31.921 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@mgr.z 2024-03-21T10:51:32.322 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:51:31 smithi070 systemd[1]: Stopping Ceph mgr.z for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:51:32.795 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:51:32 smithi070 podman[73765]: 2024-03-21 10:51:32.513443512 +0000 UTC m=+0.392446108 container died 7aeba34584304e32690e724124e328e3002535b235d5629cb6fd57d4a9989b60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6) 2024-03-21T10:51:33.072 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:51:32 smithi070 podman[73765]: 2024-03-21 10:51:32.899062263 +0000 UTC m=+0.778064921 container cleanup 7aeba34584304e32690e724124e328e3002535b235d5629cb6fd57d4a9989b60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T10:51:33.072 INFO:journalctl@ceph.mgr.z.smithi070.stdout:Mar 21 10:51:32 smithi070 bash[73765]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-mgr-z 2024-03-21T10:51:33.309 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mgr.z.service' 2024-03-21T10:51:33.346 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:51:33.346 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-21T10:51:33.346 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-21T10:51:33.346 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.0 2024-03-21T10:51:33.803 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 21 10:51:33 smithi031 systemd[1]: Stopping Ceph osd.0 for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:51:34.304 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 21 10:51:34 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-0[46368]: 2024-03-21T10:51:34.024+0000 7f479ce00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:51:34.304 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 21 10:51:34 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-0[46368]: 2024-03-21T10:51:34.024+0000 7f479ce00700 -1 osd.0 112 *** Got signal Terminated *** 2024-03-21T10:51:34.304 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 21 10:51:34 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-0[46368]: 2024-03-21T10:51:34.024+0000 7f479ce00700 -1 osd.0 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T10:51:40.054 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 21 10:51:39 smithi031 podman[103849]: 2024-03-21 10:51:39.741635154 +0000 UTC m=+5.908405265 container died 5a69b9fd1361c73cbe6e713e0a0b738cb88cee71bcdf8a189958f4709d07c2e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-21T10:51:40.554 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 21 10:51:40 smithi031 podman[103849]: 2024-03-21 10:51:40.268408282 +0000 UTC m=+6.435178425 container cleanup 5a69b9fd1361c73cbe6e713e0a0b738cb88cee71bcdf8a189958f4709d07c2e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-0, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T10:51:40.554 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 21 10:51:40 smithi031 bash[103849]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-0 2024-03-21T10:51:41.303 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 21 10:51:40 smithi031 podman[103885]: 2024-03-21 10:51:40.953776794 +0000 UTC m=+1.193148789 container remove 5a69b9fd1361c73cbe6e713e0a0b738cb88cee71bcdf8a189958f4709d07c2e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T10:51:42.151 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 21 10:51:41 smithi031 podman[103973]: 2024-03-21 10:51:41.770800812 +0000 UTC m=+0.035654567 image pull 237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232 2024-03-21T10:51:42.151 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 21 10:51:41 smithi031 podman[103973]: 2024-03-21 10:51:41.929862774 +0000 UTC m=+0.194716549 container create df2b7f439926127e540204ccb078f75da9ddbb92c1e0e808a97025a5f1b81cdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-21T10:51:42.804 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 21 10:51:42 smithi031 podman[103973]: 2024-03-21 10:51:42.42794491 +0000 UTC m=+0.692798695 container init df2b7f439926127e540204ccb078f75da9ddbb92c1e0e808a97025a5f1b81cdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T10:51:42.804 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 21 10:51:42 smithi031 podman[103973]: 2024-03-21 10:51:42.439405752 +0000 UTC m=+0.704259534 container start df2b7f439926127e540204ccb078f75da9ddbb92c1e0e808a97025a5f1b81cdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, ceph=True) 2024-03-21T10:51:42.804 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 21 10:51:42 smithi031 podman[103973]: 2024-03-21 10:51:42.541351955 +0000 UTC m=+0.806205797 container attach df2b7f439926127e540204ccb078f75da9ddbb92c1e0e808a97025a5f1b81cdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-0-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-21T10:51:43.554 INFO:journalctl@ceph.osd.0.smithi031.stdout:Mar 21 10:51:43 smithi031 podman[103973]: 2024-03-21 10:51:43.271943487 +0000 UTC m=+1.536797276 container died df2b7f439926127e540204ccb078f75da9ddbb92c1e0e808a97025a5f1b81cdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0) 2024-03-21T10:51:44.292 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.0.service' 2024-03-21T10:51:44.420 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:51:44.421 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-21T10:51:44.421 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-21T10:51:44.421 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.1 2024-03-21T10:51:44.741 INFO:journalctl@ceph.osd.1.smithi040.stdout:Mar 21 10:51:44 smithi040 systemd[1]: Stopping Ceph osd.1 for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:51:45.180 INFO:journalctl@ceph.osd.1.smithi040.stdout:Mar 21 10:51:44 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-1[40744]: 2024-03-21T10:51:44.744+0000 7fee99b9c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:51:45.180 INFO:journalctl@ceph.osd.1.smithi040.stdout:Mar 21 10:51:44 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-1[40744]: 2024-03-21T10:51:44.744+0000 7fee99b9c700 -1 osd.1 112 *** Got signal Terminated *** 2024-03-21T10:51:45.180 INFO:journalctl@ceph.osd.1.smithi040.stdout:Mar 21 10:51:44 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-1[40744]: 2024-03-21T10:51:44.744+0000 7fee99b9c700 -1 osd.1 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T10:51:50.680 INFO:journalctl@ceph.osd.1.smithi040.stdout:Mar 21 10:51:50 smithi040 podman[73115]: 2024-03-21 10:51:50.400817621 +0000 UTC m=+5.770834311 container died e878d4a7bd93141345fbe0ab69bc9b38c754400faae1f955cd99b396a216b524 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T10:51:51.058 INFO:journalctl@ceph.osd.1.smithi040.stdout:Mar 21 10:51:50 smithi040 podman[73115]: 2024-03-21 10:51:50.80316624 +0000 UTC m=+6.173182922 container cleanup e878d4a7bd93141345fbe0ab69bc9b38c754400faae1f955cd99b396a216b524 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=) 2024-03-21T10:51:51.059 INFO:journalctl@ceph.osd.1.smithi040.stdout:Mar 21 10:51:50 smithi040 bash[73115]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-1 2024-03-21T10:51:51.417 INFO:journalctl@ceph.osd.1.smithi040.stdout:Mar 21 10:51:51 smithi040 podman[73132]: 2024-03-21 10:51:51.063207887 +0000 UTC m=+0.659005527 container remove e878d4a7bd93141345fbe0ab69bc9b38c754400faae1f955cd99b396a216b524 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-21T10:51:51.680 INFO:journalctl@ceph.osd.1.smithi040.stdout:Mar 21 10:51:51 smithi040 podman[73219]: 2024-03-21 10:51:51.325041567 +0000 UTC m=+0.018257139 image pull 237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232 2024-03-21T10:51:51.680 INFO:journalctl@ceph.osd.1.smithi040.stdout:Mar 21 10:51:51 smithi040 podman[73219]: 2024-03-21 10:51:51.43920207 +0000 UTC m=+0.132417628 container create 3ced2f7b37f962c19a8994c79ffe33439fcb2a3b0aa66381dd47ce9fc685359f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-21T10:51:51.680 INFO:journalctl@ceph.osd.1.smithi040.stdout:Mar 21 10:51:51 smithi040 podman[73219]: 2024-03-21 10:51:51.642532036 +0000 UTC m=+0.335747608 container init 3ced2f7b37f962c19a8994c79ffe33439fcb2a3b0aa66381dd47ce9fc685359f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6) 2024-03-21T10:51:51.680 INFO:journalctl@ceph.osd.1.smithi040.stdout:Mar 21 10:51:51 smithi040 podman[73219]: 2024-03-21 10:51:51.645809177 +0000 UTC m=+0.339024734 container start 3ced2f7b37f962c19a8994c79ffe33439fcb2a3b0aa66381dd47ce9fc685359f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-1-deactivate, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T10:51:51.939 INFO:journalctl@ceph.osd.1.smithi040.stdout:Mar 21 10:51:51 smithi040 podman[73219]: 2024-03-21 10:51:51.76731234 +0000 UTC m=+0.460527914 container attach 3ced2f7b37f962c19a8994c79ffe33439fcb2a3b0aa66381dd47ce9fc685359f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6) 2024-03-21T10:51:52.250 INFO:journalctl@ceph.osd.1.smithi040.stdout:Mar 21 10:51:51 smithi040 podman[73219]: 2024-03-21 10:51:51.943760021 +0000 UTC m=+0.636975583 container died 3ced2f7b37f962c19a8994c79ffe33439fcb2a3b0aa66381dd47ce9fc685359f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T10:51:52.703 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.1.service' 2024-03-21T10:51:52.757 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:51:52.757 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-21T10:51:52.757 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-21T10:51:52.757 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.2 2024-03-21T10:51:53.072 INFO:journalctl@ceph.osd.2.smithi070.stdout:Mar 21 10:51:52 smithi070 systemd[1]: Stopping Ceph osd.2 for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:51:53.572 INFO:journalctl@ceph.osd.2.smithi070.stdout:Mar 21 10:51:53 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-2[41483]: 2024-03-21T10:51:53.083+0000 7f05dea82700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:51:53.572 INFO:journalctl@ceph.osd.2.smithi070.stdout:Mar 21 10:51:53 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-2[41483]: 2024-03-21T10:51:53.083+0000 7f05dea82700 -1 osd.2 112 *** Got signal Terminated *** 2024-03-21T10:51:53.572 INFO:journalctl@ceph.osd.2.smithi070.stdout:Mar 21 10:51:53 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-2[41483]: 2024-03-21T10:51:53.083+0000 7f05dea82700 -1 osd.2 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T10:51:58.981 INFO:journalctl@ceph.osd.2.smithi070.stdout:Mar 21 10:51:58 smithi070 podman[73888]: 2024-03-21 10:51:58.716610895 +0000 UTC m=+5.756126550 container died cee1732e04fc962f00fb9d3175cc5a9d22717278a5ba7c41b6b43be7908dbb4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T10:51:59.322 INFO:journalctl@ceph.osd.2.smithi070.stdout:Mar 21 10:51:59 smithi070 podman[73888]: 2024-03-21 10:51:59.076079894 +0000 UTC m=+6.115595548 container cleanup cee1732e04fc962f00fb9d3175cc5a9d22717278a5ba7c41b6b43be7908dbb4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True) 2024-03-21T10:51:59.322 INFO:journalctl@ceph.osd.2.smithi070.stdout:Mar 21 10:51:59 smithi070 bash[73888]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-2 2024-03-21T10:51:59.725 INFO:journalctl@ceph.osd.2.smithi070.stdout:Mar 21 10:51:59 smithi070 podman[73917]: 2024-03-21 10:51:59.377824294 +0000 UTC m=+0.664997586 container remove cee1732e04fc962f00fb9d3175cc5a9d22717278a5ba7c41b6b43be7908dbb4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6) 2024-03-21T10:52:00.002 INFO:journalctl@ceph.osd.2.smithi070.stdout:Mar 21 10:51:59 smithi070 podman[74009]: 2024-03-21 10:51:59.63258137 +0000 UTC m=+0.017939412 image pull 237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232 2024-03-21T10:52:00.002 INFO:journalctl@ceph.osd.2.smithi070.stdout:Mar 21 10:51:59 smithi070 podman[74009]: 2024-03-21 10:51:59.753636123 +0000 UTC m=+0.138994145 container create e3aeb5c2f5be2d7edde97f5baa258d995bb9a52c1178fffe40441f28109ba142 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-21T10:52:00.002 INFO:journalctl@ceph.osd.2.smithi070.stdout:Mar 21 10:52:00 smithi070 podman[74009]: 2024-03-21 10:52:00.006594655 +0000 UTC m=+0.391952705 container init e3aeb5c2f5be2d7edde97f5baa258d995bb9a52c1178fffe40441f28109ba142 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-21T10:52:00.305 INFO:journalctl@ceph.osd.2.smithi070.stdout:Mar 21 10:52:00 smithi070 podman[74009]: 2024-03-21 10:52:00.009796829 +0000 UTC m=+0.395154851 container start e3aeb5c2f5be2d7edde97f5baa258d995bb9a52c1178fffe40441f28109ba142 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T10:52:00.305 INFO:journalctl@ceph.osd.2.smithi070.stdout:Mar 21 10:52:00 smithi070 podman[74009]: 2024-03-21 10:52:00.114833305 +0000 UTC m=+0.500191335 container attach e3aeb5c2f5be2d7edde97f5baa258d995bb9a52c1178fffe40441f28109ba142 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-21T10:52:00.572 INFO:journalctl@ceph.osd.2.smithi070.stdout:Mar 21 10:52:00 smithi070 podman[74009]: 2024-03-21 10:52:00.309121758 +0000 UTC m=+0.694479792 container died e3aeb5c2f5be2d7edde97f5baa258d995bb9a52c1178fffe40441f28109ba142 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS) 2024-03-21T10:52:00.914 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:00 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:00.809+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:35.470719+0000 front 2024-03-21T10:51:35.470881+0000 (oldest deadline 2024-03-21T10:52:00.671294+0000) 2024-03-21T10:52:00.934 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.2.service' 2024-03-21T10:52:00.975 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:52:00.976 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-21T10:52:00.976 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-21T10:52:00.976 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.3 2024-03-21T10:52:01.495 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 21 10:52:01 smithi031 systemd[1]: Stopping Ceph osd.3 for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:52:01.752 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:01 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:01.498+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:01.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:01 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:01.447+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:02.054 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 21 10:52:01 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-3[52880]: 2024-03-21T10:52:01.754+0000 7efedde00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:52:02.054 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 21 10:52:01 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-3[52880]: 2024-03-21T10:52:01.754+0000 7efedde00700 -1 osd.3 112 *** Got signal Terminated *** 2024-03-21T10:52:02.054 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 21 10:52:01 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-3[52880]: 2024-03-21T10:52:01.754+0000 7efedde00700 -1 osd.3 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T10:52:02.322 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:01 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:01.831+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:35.470719+0000 front 2024-03-21T10:51:35.470881+0000 (oldest deadline 2024-03-21T10:52:00.671294+0000) 2024-03-21T10:52:02.484 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 21 10:52:02 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-3[52880]: 2024-03-21T10:52:02.094+0000 7efed9600700 -1 osd.3 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:38.011763+0000 front 2024-03-21T10:51:38.011604+0000 (oldest deadline 2024-03-21T10:52:01.412086+0000) 2024-03-21T10:52:02.803 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:02 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:02.488+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:02.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:02 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:02.437+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:03.322 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:02 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:02.870+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:35.470719+0000 front 2024-03-21T10:51:35.470881+0000 (oldest deadline 2024-03-21T10:52:00.671294+0000) 2024-03-21T10:52:03.435 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 21 10:52:03 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-3[52880]: 2024-03-21T10:52:03.059+0000 7efed9600700 -1 osd.3 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:38.011763+0000 front 2024-03-21T10:51:38.011604+0000 (oldest deadline 2024-03-21T10:52:01.412086+0000) 2024-03-21T10:52:03.804 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:03 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:03.439+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:03.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:03 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:03.438+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:04.322 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:03 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:03.843+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:35.470719+0000 front 2024-03-21T10:51:35.470881+0000 (oldest deadline 2024-03-21T10:52:00.671294+0000) 2024-03-21T10:52:04.449 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 21 10:52:04 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-3[52880]: 2024-03-21T10:52:04.096+0000 7efed9600700 -1 osd.3 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:38.011763+0000 front 2024-03-21T10:51:38.011604+0000 (oldest deadline 2024-03-21T10:52:01.412086+0000) 2024-03-21T10:52:04.803 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:04 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:04.451+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:04.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:04 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:04.418+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:04.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:04 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:04.555+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:05.322 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:04 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:04.838+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:35.470719+0000 front 2024-03-21T10:51:35.470881+0000 (oldest deadline 2024-03-21T10:52:00.671294+0000) 2024-03-21T10:52:05.473 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 21 10:52:05 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-3[52880]: 2024-03-21T10:52:05.103+0000 7efed9600700 -1 osd.3 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:38.011763+0000 front 2024-03-21T10:51:38.011604+0000 (oldest deadline 2024-03-21T10:52:01.412086+0000) 2024-03-21T10:52:05.801 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:05 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:05.454+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:05.803 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:05 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:05.477+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:05.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:05 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:05.523+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:06.072 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:05 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:05.805+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:35.470719+0000 front 2024-03-21T10:51:35.470881+0000 (oldest deadline 2024-03-21T10:52:00.671294+0000) 2024-03-21T10:52:06.515 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 21 10:52:06 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-3[52880]: 2024-03-21T10:52:06.090+0000 7efed9600700 -1 osd.3 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:38.011763+0000 front 2024-03-21T10:51:38.011604+0000 (oldest deadline 2024-03-21T10:52:01.412086+0000) 2024-03-21T10:52:06.786 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:06 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:06.489+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:06.803 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:06 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:06.518+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:06.929 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:06 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:06.558+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:07.072 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:06 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:06.789+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:35.470719+0000 front 2024-03-21T10:51:35.470881+0000 (oldest deadline 2024-03-21T10:52:00.671294+0000) 2024-03-21T10:52:07.760 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:07 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:07.461+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:07.803 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 21 10:52:07 smithi031 podman[104077]: 2024-03-21 10:52:07.614274362 +0000 UTC m=+6.124708608 container died 1c9db06e79bc6e2a208d7545f81d93e76508934ae703f14182a13dd7a9f3049d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-3, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T10:52:07.804 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:07 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:07.512+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:07.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:07 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:07.590+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:08.072 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:07 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:07.764+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:35.470719+0000 front 2024-03-21T10:51:35.470881+0000 (oldest deadline 2024-03-21T10:52:00.671294+0000) 2024-03-21T10:52:08.548 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 21 10:52:08 smithi031 podman[104077]: 2024-03-21 10:52:08.272634401 +0000 UTC m=+6.783068633 container cleanup 1c9db06e79bc6e2a208d7545f81d93e76508934ae703f14182a13dd7a9f3049d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-3, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True) 2024-03-21T10:52:08.549 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 21 10:52:08 smithi031 bash[104077]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-3 2024-03-21T10:52:08.804 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 21 10:52:08 smithi031 podman[104107]: 2024-03-21 10:52:08.599323504 +0000 UTC m=+0.955808981 container remove 1c9db06e79bc6e2a208d7545f81d93e76508934ae703f14182a13dd7a9f3049d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-3, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T10:52:08.804 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:08 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:08.549+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:08.824 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:08 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:08.491+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:08.824 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:08 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:08.735+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:35.470719+0000 front 2024-03-21T10:51:35.470881+0000 (oldest deadline 2024-03-21T10:52:00.671294+0000) 2024-03-21T10:52:08.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:08 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:08.548+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:09.692 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:09 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:09.515+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:09.693 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 21 10:52:09 smithi031 podman[104194]: 2024-03-21 10:52:09.336456543 +0000 UTC m=+0.039109239 image pull 237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232 2024-03-21T10:52:09.693 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 21 10:52:09 smithi031 podman[104194]: 2024-03-21 10:52:09.516901133 +0000 UTC m=+0.219553871 container create e6a23a48ace1d382d55ea6877a68df0ed144d8a41cd46f755d32237b1968c9ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.schema-version=1.0) 2024-03-21T10:52:09.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:09 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:09.535+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:09.822 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:09 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:09.740+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:35.470719+0000 front 2024-03-21T10:51:35.470881+0000 (oldest deadline 2024-03-21T10:52:00.671294+0000) 2024-03-21T10:52:09.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:09 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:09.578+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:09.974 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 21 10:52:09 smithi031 podman[104194]: 2024-03-21 10:52:09.870989744 +0000 UTC m=+0.573642485 container init e6a23a48ace1d382d55ea6877a68df0ed144d8a41cd46f755d32237b1968c9ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6) 2024-03-21T10:52:09.974 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 21 10:52:09 smithi031 podman[104194]: 2024-03-21 10:52:09.883027231 +0000 UTC m=+0.585680002 container start e6a23a48ace1d382d55ea6877a68df0ed144d8a41cd46f755d32237b1968c9ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-21T10:52:09.975 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 21 10:52:09 smithi031 podman[104194]: 2024-03-21 10:52:09.97802599 +0000 UTC m=+0.680678736 container attach e6a23a48ace1d382d55ea6877a68df0ed144d8a41cd46f755d32237b1968c9ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-21T10:52:10.699 INFO:journalctl@ceph.osd.3.smithi031.stdout:Mar 21 10:52:10 smithi031 podman[104194]: 2024-03-21 10:52:10.334621407 +0000 UTC m=+1.037274112 container died e6a23a48ace1d382d55ea6877a68df0ed144d8a41cd46f755d32237b1968c9ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-21T10:52:10.699 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:10 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:10.530+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:10.770 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:10 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:10.489+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:10.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:10 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:10.559+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:11.072 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:10 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:10.774+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:35.470719+0000 front 2024-03-21T10:51:35.470881+0000 (oldest deadline 2024-03-21T10:52:00.671294+0000) 2024-03-21T10:52:11.115 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.3.service' 2024-03-21T10:52:11.249 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:52:11.250 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-21T10:52:11.250 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-21T10:52:11.250 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.4 2024-03-21T10:52:11.554 INFO:journalctl@ceph.osd.4.smithi040.stdout:Mar 21 10:52:11 smithi040 systemd[1]: Stopping Ceph osd.4 for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:52:11.808 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:11 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:11.537+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:11.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:11 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:11.557+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:11.930 INFO:journalctl@ceph.osd.4.smithi040.stdout:Mar 21 10:52:11 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-4[47697]: 2024-03-21T10:52:11.573+0000 7fd04a014700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:52:11.930 INFO:journalctl@ceph.osd.4.smithi040.stdout:Mar 21 10:52:11 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-4[47697]: 2024-03-21T10:52:11.573+0000 7fd04a014700 -1 osd.4 112 *** Got signal Terminated *** 2024-03-21T10:52:11.930 INFO:journalctl@ceph.osd.4.smithi040.stdout:Mar 21 10:52:11 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-4[47697]: 2024-03-21T10:52:11.573+0000 7fd04a014700 -1 osd.4 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T10:52:12.053 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:11 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:11.557+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:12.072 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:11 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:11.812+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:35.470719+0000 front 2024-03-21T10:51:35.470881+0000 (oldest deadline 2024-03-21T10:52:00.671294+0000) 2024-03-21T10:52:12.519 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:12 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:12.257+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:45.648272+0000 front 2024-03-21T10:51:45.648197+0000 (oldest deadline 2024-03-21T10:52:11.548371+0000) 2024-03-21T10:52:12.680 INFO:journalctl@ceph.osd.4.smithi040.stdout:Mar 21 10:52:12 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-4[47697]: 2024-03-21T10:52:12.305+0000 7fd046608700 -1 osd.4 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:46.767246+0000 front 2024-03-21T10:51:46.767302+0000 (oldest deadline 2024-03-21T10:52:11.467408+0000) 2024-03-21T10:52:12.680 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:12 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:12.554+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:12.680 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:12 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:12.554+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:12.804 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:12 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:12.521+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:12.804 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:12 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:12.521+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:12.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:12 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:12.487+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:13.322 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:12 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:12.855+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:35.470719+0000 front 2024-03-21T10:51:35.470881+0000 (oldest deadline 2024-03-21T10:52:00.671294+0000) 2024-03-21T10:52:13.322 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:12 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:12.976+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:13.546 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:13 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:13.236+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:45.648272+0000 front 2024-03-21T10:51:45.648197+0000 (oldest deadline 2024-03-21T10:52:11.548371+0000) 2024-03-21T10:52:13.577 INFO:journalctl@ceph.osd.4.smithi040.stdout:Mar 21 10:52:13 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-4[47697]: 2024-03-21T10:52:13.264+0000 7fd046608700 -1 osd.4 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:46.767246+0000 front 2024-03-21T10:51:46.767302+0000 (oldest deadline 2024-03-21T10:52:11.467408+0000) 2024-03-21T10:52:13.803 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:13 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:13.549+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:13.804 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:13 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:13.549+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:13.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:13 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:13.466+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:13.929 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:13 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:13.581+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:13.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:13 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:13.581+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:14.322 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:13 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:13.956+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:14.322 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:13 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:13.847+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:35.470719+0000 front 2024-03-21T10:51:35.470881+0000 (oldest deadline 2024-03-21T10:52:00.671294+0000) 2024-03-21T10:52:14.322 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:13 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:13.847+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:46.973714+0000 front 2024-03-21T10:51:46.973823+0000 (oldest deadline 2024-03-21T10:52:12.873717+0000) 2024-03-21T10:52:14.502 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:14 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:14.225+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:45.648272+0000 front 2024-03-21T10:51:45.648197+0000 (oldest deadline 2024-03-21T10:52:11.548371+0000) 2024-03-21T10:52:14.560 INFO:journalctl@ceph.osd.4.smithi040.stdout:Mar 21 10:52:14 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-4[47697]: 2024-03-21T10:52:14.234+0000 7fd046608700 -1 osd.4 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:46.767246+0000 front 2024-03-21T10:51:46.767302+0000 (oldest deadline 2024-03-21T10:52:11.467408+0000) 2024-03-21T10:52:14.796 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:14 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:14.459+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:14.797 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:14 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:14.459+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:14.804 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:14 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:14.505+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:14.804 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:14 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:14.505+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:14.927 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:14 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:14.564+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:14.927 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:14 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:14.564+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:15.072 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:15 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:15.005+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:15.072 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:14 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:14.800+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:35.470719+0000 front 2024-03-21T10:51:35.470881+0000 (oldest deadline 2024-03-21T10:52:00.671294+0000) 2024-03-21T10:52:15.073 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:14 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:14.800+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:46.973714+0000 front 2024-03-21T10:51:46.973823+0000 (oldest deadline 2024-03-21T10:52:12.873717+0000) 2024-03-21T10:52:15.179 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:14 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:14.931+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:15.468 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:15 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:15.205+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:45.648272+0000 front 2024-03-21T10:51:45.648197+0000 (oldest deadline 2024-03-21T10:52:11.548371+0000) 2024-03-21T10:52:15.514 INFO:journalctl@ceph.osd.4.smithi040.stdout:Mar 21 10:52:15 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-4[47697]: 2024-03-21T10:52:15.248+0000 7fd046608700 -1 osd.4 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:46.767246+0000 front 2024-03-21T10:51:46.767302+0000 (oldest deadline 2024-03-21T10:52:11.467408+0000) 2024-03-21T10:52:15.777 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:15 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:15.466+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:15.777 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:15 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:15.466+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:15.803 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:15 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:15.471+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:15.804 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:15 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:15.471+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:15.912 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:15 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:15.517+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:15.912 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:15 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:15.517+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:16.072 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:16 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:16.030+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:16.073 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:15 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:15.780+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:35.470719+0000 front 2024-03-21T10:51:35.470881+0000 (oldest deadline 2024-03-21T10:52:00.671294+0000) 2024-03-21T10:52:16.073 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:15 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:15.780+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:46.973714+0000 front 2024-03-21T10:51:46.973823+0000 (oldest deadline 2024-03-21T10:52:12.873717+0000) 2024-03-21T10:52:16.179 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:15 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:15.915+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:16.483 INFO:journalctl@ceph.osd.4.smithi040.stdout:Mar 21 10:52:16 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-4[47697]: 2024-03-21T10:52:16.229+0000 7fd046608700 -1 osd.4 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:46.767246+0000 front 2024-03-21T10:51:46.767302+0000 (oldest deadline 2024-03-21T10:52:11.467408+0000) 2024-03-21T10:52:16.553 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:16 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:16.211+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:45.648272+0000 front 2024-03-21T10:51:45.648197+0000 (oldest deadline 2024-03-21T10:52:11.548371+0000) 2024-03-21T10:52:16.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:16 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:16.438+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:16.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:16 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:16.438+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:16.812 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:16 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:16.446+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:16.812 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:16 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:16.446+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:16.874 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:16 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:16.487+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:16.874 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:16 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:16.487+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:17.072 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:16 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:16.816+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:35.470719+0000 front 2024-03-21T10:51:35.470881+0000 (oldest deadline 2024-03-21T10:52:00.671294+0000) 2024-03-21T10:52:17.073 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:16 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:16.816+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:46.973714+0000 front 2024-03-21T10:51:46.973823+0000 (oldest deadline 2024-03-21T10:52:12.873717+0000) 2024-03-21T10:52:17.073 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:17 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:17.038+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:17.179 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:16 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:16.877+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:17.462 INFO:journalctl@ceph.osd.4.smithi040.stdout:Mar 21 10:52:17 smithi040 podman[73336]: 2024-03-21 10:52:17.216124917 +0000 UTC m=+5.759705384 container died 4aff6cfa99dcc01ca6aee824ecb8062ddf8ab86ee5aaebe97be1dab76dbd93f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-4, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-21T10:52:17.462 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:17 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:17.453+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:17.462 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:17 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:17.453+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:17.553 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:17 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:17.225+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:45.648272+0000 front 2024-03-21T10:51:45.648197+0000 (oldest deadline 2024-03-21T10:52:11.548371+0000) 2024-03-21T10:52:17.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:17 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:17.414+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:17.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:17 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:17.414+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:17.776 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:17 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:17.444+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:17.776 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:17 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:17.444+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:17.887 INFO:journalctl@ceph.osd.4.smithi040.stdout:Mar 21 10:52:17 smithi040 podman[73336]: 2024-03-21 10:52:17.565801609 +0000 UTC m=+6.109382114 container cleanup 4aff6cfa99dcc01ca6aee824ecb8062ddf8ab86ee5aaebe97be1dab76dbd93f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T10:52:17.888 INFO:journalctl@ceph.osd.4.smithi040.stdout:Mar 21 10:52:17 smithi040 bash[73336]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-4 2024-03-21T10:52:18.045 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:17 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:17.779+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:35.470719+0000 front 2024-03-21T10:51:35.470881+0000 (oldest deadline 2024-03-21T10:52:00.671294+0000) 2024-03-21T10:52:18.045 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:17 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:17.779+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:46.973714+0000 front 2024-03-21T10:51:46.973823+0000 (oldest deadline 2024-03-21T10:52:12.873717+0000) 2024-03-21T10:52:18.180 INFO:journalctl@ceph.osd.4.smithi040.stdout:Mar 21 10:52:17 smithi040 podman[73355]: 2024-03-21 10:52:17.892139715 +0000 UTC m=+0.679521745 container remove 4aff6cfa99dcc01ca6aee824ecb8062ddf8ab86ee5aaebe97be1dab76dbd93f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-4, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-21T10:52:18.181 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:17 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:17.897+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:18.322 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:18 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:18.049+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:18.487 INFO:journalctl@ceph.osd.4.smithi040.stdout:Mar 21 10:52:18 smithi040 podman[73446]: 2024-03-21 10:52:18.142163679 +0000 UTC m=+0.018584969 image pull 237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232 2024-03-21T10:52:18.487 INFO:journalctl@ceph.osd.4.smithi040.stdout:Mar 21 10:52:18 smithi040 podman[73446]: 2024-03-21 10:52:18.251582833 +0000 UTC m=+0.128004110 container create 0341e161af1a3031df05a28c73d0397a4f2596ec6dee427234992b8e2b69ae06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-21T10:52:18.487 INFO:journalctl@ceph.osd.4.smithi040.stdout:Mar 21 10:52:18 smithi040 podman[73446]: 2024-03-21 10:52:18.487584753 +0000 UTC m=+0.364006035 container init 0341e161af1a3031df05a28c73d0397a4f2596ec6dee427234992b8e2b69ae06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-21T10:52:18.487 INFO:journalctl@ceph.osd.4.smithi040.stdout:Mar 21 10:52:18 smithi040 podman[73446]: 2024-03-21 10:52:18.49147629 +0000 UTC m=+0.367897570 container start 0341e161af1a3031df05a28c73d0397a4f2596ec6dee427234992b8e2b69ae06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-21T10:52:18.553 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:18 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:18.233+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:45.648272+0000 front 2024-03-21T10:51:45.648197+0000 (oldest deadline 2024-03-21T10:52:11.548371+0000) 2024-03-21T10:52:18.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:18 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:18.379+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:18.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:18 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:18.379+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:18.790 INFO:journalctl@ceph.osd.4.smithi040.stdout:Mar 21 10:52:18 smithi040 podman[73446]: 2024-03-21 10:52:18.587644804 +0000 UTC m=+0.464066102 container attach 0341e161af1a3031df05a28c73d0397a4f2596ec6dee427234992b8e2b69ae06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6) 2024-03-21T10:52:18.791 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:18 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:18.490+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:18.791 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:18 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:18.490+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:18.802 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:18 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:18.464+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:18.802 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:18 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:18.464+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:19.072 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:18 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:18.806+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:35.470719+0000 front 2024-03-21T10:51:35.470881+0000 (oldest deadline 2024-03-21T10:52:00.671294+0000) 2024-03-21T10:52:19.072 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:18 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:18.806+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:46.973714+0000 front 2024-03-21T10:51:46.973823+0000 (oldest deadline 2024-03-21T10:52:12.873717+0000) 2024-03-21T10:52:19.072 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:19 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:19.000+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:19.152 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:18 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:18.912+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:19.153 INFO:journalctl@ceph.osd.4.smithi040.stdout:Mar 21 10:52:18 smithi040 podman[73446]: 2024-03-21 10:52:18.79549401 +0000 UTC m=+0.671915300 container died 0341e161af1a3031df05a28c73d0397a4f2596ec6dee427234992b8e2b69ae06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True) 2024-03-21T10:52:19.475 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.4.service' 2024-03-21T10:52:19.525 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:52:19.525 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-21T10:52:19.525 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-21T10:52:19.525 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.5 2024-03-21T10:52:19.553 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:19 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:19.205+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:45.648272+0000 front 2024-03-21T10:51:45.648197+0000 (oldest deadline 2024-03-21T10:52:11.548371+0000) 2024-03-21T10:52:19.554 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:19 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:19.206+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.151525+0000 front 2024-03-21T10:51:56.151582+0000 (oldest deadline 2024-03-21T10:52:19.051761+0000) 2024-03-21T10:52:19.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:19 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:19.350+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:19.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:19 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:19.350+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:19.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:19 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:19.350+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:19.772 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:19 smithi070 systemd[1]: Stopping Ceph osd.5 for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:52:19.772 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:19 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:19.422+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:19.772 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:19 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:19.422+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:19.883 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:19 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:19.484+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:19.883 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:19 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:19.484+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:19.883 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:19 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:19.484+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:20.072 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:20 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:20.018+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:20.072 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:20 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:20.018+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:20.072 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:19 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:19.775+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:35.470719+0000 front 2024-03-21T10:51:35.470881+0000 (oldest deadline 2024-03-21T10:52:00.671294+0000) 2024-03-21T10:52:20.073 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:19 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:19.775+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:46.973714+0000 front 2024-03-21T10:51:46.973823+0000 (oldest deadline 2024-03-21T10:52:12.873717+0000) 2024-03-21T10:52:20.073 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:19 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:19.853+0000 7fc52b024700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:52:20.073 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:19 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:19.853+0000 7fc52b024700 -1 osd.5 112 *** Got signal Terminated *** 2024-03-21T10:52:20.073 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:19 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:19.853+0000 7fc52b024700 -1 osd.5 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T10:52:20.179 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:19 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:19.886+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:20.554 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:20 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:20.178+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:45.648272+0000 front 2024-03-21T10:51:45.648197+0000 (oldest deadline 2024-03-21T10:52:11.548371+0000) 2024-03-21T10:52:20.554 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:20 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:20.178+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.151525+0000 front 2024-03-21T10:51:56.151582+0000 (oldest deadline 2024-03-21T10:52:19.051761+0000) 2024-03-21T10:52:20.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:20 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:20.365+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:20.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:20 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:20.365+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:20.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:20 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:20.365+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:20.767 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:20 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:20.415+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:20.767 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:20 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:20.415+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:20.902 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:20 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:20.521+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:20.902 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:20 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:20.521+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:20.902 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:20 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:20.521+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:21.072 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:20 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:20.771+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:35.470719+0000 front 2024-03-21T10:51:35.470881+0000 (oldest deadline 2024-03-21T10:52:00.671294+0000) 2024-03-21T10:52:21.072 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:20 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:20.771+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:46.973714+0000 front 2024-03-21T10:51:46.973823+0000 (oldest deadline 2024-03-21T10:52:12.873717+0000) 2024-03-21T10:52:21.072 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:21 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:21.006+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:21.073 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:21 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:21.006+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:21.179 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:20 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:20.906+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:21.553 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:21 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:21.208+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:45.648272+0000 front 2024-03-21T10:51:45.648197+0000 (oldest deadline 2024-03-21T10:52:11.548371+0000) 2024-03-21T10:52:21.554 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:21 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:21.208+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.151525+0000 front 2024-03-21T10:51:56.151582+0000 (oldest deadline 2024-03-21T10:52:19.051761+0000) 2024-03-21T10:52:21.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:21 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:21.388+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:21.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:21 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:21.388+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:21.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:21 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:21.388+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:21.775 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:21 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:21.435+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:21.775 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:21 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:21.435+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:21.890 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:21 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:21.502+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:21.890 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:21 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:21.502+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:21.890 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:21 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:21.502+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:22.072 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:22 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:22.013+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:22.072 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:22 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:22.013+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:22.072 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:21 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:21.779+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:35.470719+0000 front 2024-03-21T10:51:35.470881+0000 (oldest deadline 2024-03-21T10:52:00.671294+0000) 2024-03-21T10:52:22.072 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:21 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:21.779+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:46.973714+0000 front 2024-03-21T10:51:46.973823+0000 (oldest deadline 2024-03-21T10:52:12.873717+0000) 2024-03-21T10:52:22.073 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:21 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:21.779+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.374724+0000 front 2024-03-21T10:51:56.374752+0000 (oldest deadline 2024-03-21T10:52:21.074862+0000) 2024-03-21T10:52:22.180 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:21 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:21.894+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:22.554 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:22 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:22.199+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:45.648272+0000 front 2024-03-21T10:51:45.648197+0000 (oldest deadline 2024-03-21T10:52:11.548371+0000) 2024-03-21T10:52:22.554 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:22 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:22.199+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.151525+0000 front 2024-03-21T10:51:56.151582+0000 (oldest deadline 2024-03-21T10:52:19.051761+0000) 2024-03-21T10:52:22.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:22 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:22.396+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:22.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:22 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:22.396+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:22.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:22 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:22.396+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:22.810 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:22 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:22.414+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:22.810 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:22 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:22.414+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:22.845 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:22 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:22.486+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:22.845 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:22 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:22.486+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:22.845 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:22 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:22.486+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:23.072 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:23 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:23.040+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:23.072 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:23 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:23.040+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:23.072 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:22 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:22.814+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:35.470719+0000 front 2024-03-21T10:51:35.470881+0000 (oldest deadline 2024-03-21T10:52:00.671294+0000) 2024-03-21T10:52:23.072 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:22 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:22.814+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:46.973714+0000 front 2024-03-21T10:51:46.973823+0000 (oldest deadline 2024-03-21T10:52:12.873717+0000) 2024-03-21T10:52:23.072 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:22 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:22.814+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.374724+0000 front 2024-03-21T10:51:56.374752+0000 (oldest deadline 2024-03-21T10:52:21.074862+0000) 2024-03-21T10:52:23.180 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:22 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:22.849+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:23.180 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:22 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:22.849+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.999254+0000 front 2024-03-21T10:51:56.999105+0000 (oldest deadline 2024-03-21T10:52:22.199976+0000) 2024-03-21T10:52:23.435 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:23 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:23.153+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:45.648272+0000 front 2024-03-21T10:51:45.648197+0000 (oldest deadline 2024-03-21T10:52:11.548371+0000) 2024-03-21T10:52:23.435 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:23 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:23.153+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.151525+0000 front 2024-03-21T10:51:56.151582+0000 (oldest deadline 2024-03-21T10:52:19.051761+0000) 2024-03-21T10:52:23.781 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:23 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:23.380+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:23.781 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:23 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:23.380+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:23.781 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:23 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:23.380+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:23.803 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:23 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:23.438+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:23.804 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:23 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:23.438+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:23.804 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:23 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:23.438+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:23.819 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:23 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:23.522+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:23.820 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:23 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:23.522+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:23.820 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:23 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:23.522+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:24.072 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:24 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:24.032+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:24.072 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:24 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:24.032+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:24.072 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:23 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:23.785+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:35.470719+0000 front 2024-03-21T10:51:35.470881+0000 (oldest deadline 2024-03-21T10:52:00.671294+0000) 2024-03-21T10:52:24.072 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:23 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:23.785+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:46.973714+0000 front 2024-03-21T10:51:46.973823+0000 (oldest deadline 2024-03-21T10:52:12.873717+0000) 2024-03-21T10:52:24.073 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:23 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:23.785+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.374724+0000 front 2024-03-21T10:51:56.374752+0000 (oldest deadline 2024-03-21T10:52:21.074862+0000) 2024-03-21T10:52:24.180 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:23 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:23.823+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:24.180 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:23 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:23.823+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.999254+0000 front 2024-03-21T10:51:56.999105+0000 (oldest deadline 2024-03-21T10:52:22.199976+0000) 2024-03-21T10:52:24.426 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:24 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:24.135+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:45.648272+0000 front 2024-03-21T10:51:45.648197+0000 (oldest deadline 2024-03-21T10:52:11.548371+0000) 2024-03-21T10:52:24.426 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:24 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:24.135+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.151525+0000 front 2024-03-21T10:51:56.151582+0000 (oldest deadline 2024-03-21T10:52:19.051761+0000) 2024-03-21T10:52:24.743 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:24 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:24.356+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:24.743 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:24 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:24.356+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:24.743 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:24 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:24.356+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:24.803 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:24 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:24.429+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:24.803 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:24 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:24.429+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:24.804 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:24 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:24.429+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:24.826 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:24 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:24.572+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:24.826 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:24 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:24.572+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:24.826 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:24 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:24.572+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:25.072 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:24 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:24.985+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:25.072 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:24 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:24.985+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:25.072 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:24 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:24.747+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:35.470719+0000 front 2024-03-21T10:51:35.470881+0000 (oldest deadline 2024-03-21T10:52:00.671294+0000) 2024-03-21T10:52:25.073 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:24 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:24.747+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:46.973714+0000 front 2024-03-21T10:51:46.973823+0000 (oldest deadline 2024-03-21T10:52:12.873717+0000) 2024-03-21T10:52:25.073 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:24 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5[48191]: 2024-03-21T10:52:24.747+0000 7fc527618700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.374724+0000 front 2024-03-21T10:51:56.374752+0000 (oldest deadline 2024-03-21T10:52:21.074862+0000) 2024-03-21T10:52:25.180 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:24 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:24.829+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:25.180 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:24 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:24.829+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.999254+0000 front 2024-03-21T10:51:56.999105+0000 (oldest deadline 2024-03-21T10:52:22.199976+0000) 2024-03-21T10:52:25.427 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:25 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:25.166+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:45.648272+0000 front 2024-03-21T10:51:45.648197+0000 (oldest deadline 2024-03-21T10:52:11.548371+0000) 2024-03-21T10:52:25.428 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:25 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:25.166+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.151525+0000 front 2024-03-21T10:51:56.151582+0000 (oldest deadline 2024-03-21T10:52:19.051761+0000) 2024-03-21T10:52:25.716 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:25 smithi070 podman[74113]: 2024-03-21 10:52:25.458623347 +0000 UTC m=+5.737882276 container died 8b449ee2c4ded6605169a9c5f55ebf570205383aae2ca411f3303b96795a4ea0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-21T10:52:25.717 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:25 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:25.357+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:25.717 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:25 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:25.357+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:25.717 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:25 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:25.357+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:25.804 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:25 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:25.430+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:25.804 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:25 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:25.430+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:25.804 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:25 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:25.430+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:25.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:25 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:25.852+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:25.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:25 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:25.852+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.999254+0000 front 2024-03-21T10:51:56.999105+0000 (oldest deadline 2024-03-21T10:52:22.199976+0000) 2024-03-21T10:52:25.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:25 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:25.607+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:25.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:25 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:25.607+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:25.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:25 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:25.607+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:26.072 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:25 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:25.956+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:26.072 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:25 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:25.956+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:26.072 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:25 smithi070 podman[74113]: 2024-03-21 10:52:25.819653896 +0000 UTC m=+6.098912823 container cleanup 8b449ee2c4ded6605169a9c5f55ebf570205383aae2ca411f3303b96795a4ea0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.build-date=20240319) 2024-03-21T10:52:26.072 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:25 smithi070 bash[74113]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5 2024-03-21T10:52:26.423 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:26 smithi070 podman[74130]: 2024-03-21 10:52:26.11260466 +0000 UTC m=+0.646771388 container remove 8b449ee2c4ded6605169a9c5f55ebf570205383aae2ca411f3303b96795a4ea0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-21T10:52:26.424 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:26 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:26.359+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:26.424 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:26 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:26.359+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:26.424 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:26 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:26.359+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:26.553 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:26 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:26.193+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:45.648272+0000 front 2024-03-21T10:51:45.648197+0000 (oldest deadline 2024-03-21T10:52:11.548371+0000) 2024-03-21T10:52:26.553 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:26 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:26.193+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.151525+0000 front 2024-03-21T10:51:56.151582+0000 (oldest deadline 2024-03-21T10:52:19.051761+0000) 2024-03-21T10:52:26.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:26 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:26.385+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:26.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:26 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:26.385+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:26.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:26 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:26.385+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:26.695 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:26 smithi070 podman[74222]: 2024-03-21 10:52:26.331317522 +0000 UTC m=+0.017175280 image pull 237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232 2024-03-21T10:52:26.695 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:26 smithi070 podman[74222]: 2024-03-21 10:52:26.464045999 +0000 UTC m=+0.149903758 container create ba2da766f879743edcbac5391bb7acb9582d9401f49c638e456bc34e31aad7d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T10:52:26.695 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:26 smithi070 podman[74222]: 2024-03-21 10:52:26.699830708 +0000 UTC m=+0.385688476 container init ba2da766f879743edcbac5391bb7acb9582d9401f49c638e456bc34e31aad7d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.vendor=CentOS) 2024-03-21T10:52:26.862 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:26 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:26.591+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:26.862 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:26 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:26.591+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:26.862 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:26 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:26.591+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:26.963 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:26 smithi070 podman[74222]: 2024-03-21 10:52:26.702860575 +0000 UTC m=+0.388718340 container start ba2da766f879743edcbac5391bb7acb9582d9401f49c638e456bc34e31aad7d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T10:52:26.963 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:26 smithi070 podman[74222]: 2024-03-21 10:52:26.782953436 +0000 UTC m=+0.468811205 container attach ba2da766f879743edcbac5391bb7acb9582d9401f49c638e456bc34e31aad7d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-03-21T10:52:27.179 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:26 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:26.866+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:27.180 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:26 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:26.866+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.999254+0000 front 2024-03-21T10:51:56.999105+0000 (oldest deadline 2024-03-21T10:52:22.199976+0000) 2024-03-21T10:52:27.290 INFO:journalctl@ceph.osd.5.smithi070.stdout:Mar 21 10:52:26 smithi070 podman[74222]: 2024-03-21 10:52:26.989681258 +0000 UTC m=+0.675539028 container died ba2da766f879743edcbac5391bb7acb9582d9401f49c638e456bc34e31aad7d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-5-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-21T10:52:27.290 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:26 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:26.966+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:27.290 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:26 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:26.966+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:27.553 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:27 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:27.224+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:45.648272+0000 front 2024-03-21T10:51:45.648197+0000 (oldest deadline 2024-03-21T10:52:11.548371+0000) 2024-03-21T10:52:27.554 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:27 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:27.224+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.151525+0000 front 2024-03-21T10:51:56.151582+0000 (oldest deadline 2024-03-21T10:52:19.051761+0000) 2024-03-21T10:52:27.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:27 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:27.365+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:27.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:27 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:27.365+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:27.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:27 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:27.365+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:27.572 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:27 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:27.392+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:27.572 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:27 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:27.392+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:27.572 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:27 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:27.392+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:27.594 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.5.service' 2024-03-21T10:52:27.634 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:52:27.635 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-21T10:52:27.635 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-21T10:52:27.635 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.6 2024-03-21T10:52:27.878 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:27 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:27.629+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:27.879 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:27 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:27.629+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:27.879 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:27 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:27.629+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:28.053 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:27 smithi031 systemd[1]: Stopping Ceph osd.6 for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:52:28.180 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:27 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:27.881+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:28.180 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:27 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:27.881+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.999254+0000 front 2024-03-21T10:51:56.999105+0000 (oldest deadline 2024-03-21T10:52:22.199976+0000) 2024-03-21T10:52:28.322 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:28 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:28.000+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:28.322 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:28 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:28.000+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:28.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:28 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:28.358+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:28.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:28 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:28.358+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:28.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:28 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:28.358+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:28.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:28 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:28.358+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:28.555 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:28 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:28.180+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:45.648272+0000 front 2024-03-21T10:51:45.648197+0000 (oldest deadline 2024-03-21T10:52:11.548371+0000) 2024-03-21T10:52:28.555 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:28 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:28.180+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.151525+0000 front 2024-03-21T10:51:56.151582+0000 (oldest deadline 2024-03-21T10:52:19.051761+0000) 2024-03-21T10:52:28.555 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:28 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:28.267+0000 7f63b3000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:52:28.555 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:28 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:28.267+0000 7f63b3000700 -1 osd.6 112 *** Got signal Terminated *** 2024-03-21T10:52:28.555 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:28 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:28.267+0000 7f63b3000700 -1 osd.6 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T10:52:28.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:28 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:28.359+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:28.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:28 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:28.359+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:28.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:28 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:28.359+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:28.878 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:28 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:28.631+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:28.878 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:28 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:28.631+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:28.878 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:28 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:28.631+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:28.878 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:28 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:28.881+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:28.878 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:28 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:28.881+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.999254+0000 front 2024-03-21T10:51:56.999105+0000 (oldest deadline 2024-03-21T10:52:22.199976+0000) 2024-03-21T10:52:28.878 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:28 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:28.881+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.200862+0000 front 2024-03-21T10:52:02.200915+0000 (oldest deadline 2024-03-21T10:52:28.100759+0000) 2024-03-21T10:52:29.322 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:29 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:29.024+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:29.322 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:29 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:29.024+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:29.554 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:29 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:29.148+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:45.648272+0000 front 2024-03-21T10:51:45.648197+0000 (oldest deadline 2024-03-21T10:52:11.548371+0000) 2024-03-21T10:52:29.554 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:29 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:29.148+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.151525+0000 front 2024-03-21T10:51:56.151582+0000 (oldest deadline 2024-03-21T10:52:19.051761+0000) 2024-03-21T10:52:29.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:29 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:29.374+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:29.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:29 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:29.374+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:29.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:29 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:29.374+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:29.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:29 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:29.374+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:29.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:29 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:29.354+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:29.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:29 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:29.354+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:29.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:29 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:29.354+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:29.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:29 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:29.844+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:29.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:29 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:29.844+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.999254+0000 front 2024-03-21T10:51:56.999105+0000 (oldest deadline 2024-03-21T10:52:22.199976+0000) 2024-03-21T10:52:29.931 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:29 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:29.844+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.200862+0000 front 2024-03-21T10:52:02.200915+0000 (oldest deadline 2024-03-21T10:52:28.100759+0000) 2024-03-21T10:52:29.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:29 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:29.656+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:29.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:29 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:29.656+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:29.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:29 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:29.656+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:29.932 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:29 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:29.656+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:30.322 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:30 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:30.039+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:30.322 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:30 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:30.039+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:30.554 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:30 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:30.157+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:45.648272+0000 front 2024-03-21T10:51:45.648197+0000 (oldest deadline 2024-03-21T10:52:11.548371+0000) 2024-03-21T10:52:30.554 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:30 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:30.157+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.151525+0000 front 2024-03-21T10:51:56.151582+0000 (oldest deadline 2024-03-21T10:52:19.051761+0000) 2024-03-21T10:52:30.554 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:30 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:30.157+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:06.055570+0000 front 2024-03-21T10:52:06.055973+0000 (oldest deadline 2024-03-21T10:52:29.555677+0000) 2024-03-21T10:52:30.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:30 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:30.331+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:30.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:30 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:30.331+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:30.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:30 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:30.331+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:30.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:30 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:30.331+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:30.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:30 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:30.337+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:30.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:30 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:30.337+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:30.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:30 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:30.337+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:30.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:30 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:30.796+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:30.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:30 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:30.796+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.999254+0000 front 2024-03-21T10:51:56.999105+0000 (oldest deadline 2024-03-21T10:52:22.199976+0000) 2024-03-21T10:52:30.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:30 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:30.796+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.200862+0000 front 2024-03-21T10:52:02.200915+0000 (oldest deadline 2024-03-21T10:52:28.100759+0000) 2024-03-21T10:52:30.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:30 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:30.651+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:30.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:30 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:30.651+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:30.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:30 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:30.651+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:30.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:30 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:30.651+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:31.322 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:31 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:31.046+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:31.322 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:31 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:31.046+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:31.554 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:31 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:31.114+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:45.648272+0000 front 2024-03-21T10:51:45.648197+0000 (oldest deadline 2024-03-21T10:52:11.548371+0000) 2024-03-21T10:52:31.554 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:31 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:31.114+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.151525+0000 front 2024-03-21T10:51:56.151582+0000 (oldest deadline 2024-03-21T10:52:19.051761+0000) 2024-03-21T10:52:31.554 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:31 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:31.114+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:06.055570+0000 front 2024-03-21T10:52:06.055973+0000 (oldest deadline 2024-03-21T10:52:29.555677+0000) 2024-03-21T10:52:31.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:31 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:31.312+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:31.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:31 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:31.312+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:31.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:31 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:31.312+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:31.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:31 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:31.312+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:31.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:31 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:31.357+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:31.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:31 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:31.357+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:31.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:31 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:31.357+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:31.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:31 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:31.824+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:31.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:31 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:31.824+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.999254+0000 front 2024-03-21T10:51:56.999105+0000 (oldest deadline 2024-03-21T10:52:22.199976+0000) 2024-03-21T10:52:31.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:31 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:31.824+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.200862+0000 front 2024-03-21T10:52:02.200915+0000 (oldest deadline 2024-03-21T10:52:28.100759+0000) 2024-03-21T10:52:31.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:31 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:31.664+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:31.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:31 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:31.664+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:31.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:31 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:31.664+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:31.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:31 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:31.664+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:32.554 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:32 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:32.134+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:45.648272+0000 front 2024-03-21T10:51:45.648197+0000 (oldest deadline 2024-03-21T10:52:11.548371+0000) 2024-03-21T10:52:32.554 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:32 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:32.134+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.151525+0000 front 2024-03-21T10:51:56.151582+0000 (oldest deadline 2024-03-21T10:52:19.051761+0000) 2024-03-21T10:52:32.554 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:32 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:32.134+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:06.055570+0000 front 2024-03-21T10:52:06.055973+0000 (oldest deadline 2024-03-21T10:52:29.555677+0000) 2024-03-21T10:52:32.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:32 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:32.315+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:32.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:32 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:32.315+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:32.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:32 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:32.315+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:32.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:32 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:32.315+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:32.572 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:32 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:32.084+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:32.572 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:32 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:32.084+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:32.572 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:32 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:32.084+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:06.747711+0000 front 2024-03-21T10:52:06.747844+0000 (oldest deadline 2024-03-21T10:52:32.047564+0000) 2024-03-21T10:52:32.572 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:32 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:32.311+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:32.573 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:32 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:32.311+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:32.573 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:32 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:32.311+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:32.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:32 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:32.818+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:32.931 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:32 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:32.818+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.999254+0000 front 2024-03-21T10:51:56.999105+0000 (oldest deadline 2024-03-21T10:52:22.199976+0000) 2024-03-21T10:52:32.931 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:32 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:32.818+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.200862+0000 front 2024-03-21T10:52:02.200915+0000 (oldest deadline 2024-03-21T10:52:28.100759+0000) 2024-03-21T10:52:32.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:32 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:32.667+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:32.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:32 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:32.667+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:32.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:32 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:32.667+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:32.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:32 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:32.667+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:33.456 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:33 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:33.109+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:45.648272+0000 front 2024-03-21T10:51:45.648197+0000 (oldest deadline 2024-03-21T10:52:11.548371+0000) 2024-03-21T10:52:33.456 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:33 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:33.109+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.151525+0000 front 2024-03-21T10:51:56.151582+0000 (oldest deadline 2024-03-21T10:52:19.051761+0000) 2024-03-21T10:52:33.456 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:33 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6[59443]: 2024-03-21T10:52:33.109+0000 7f63ae800700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:06.055570+0000 front 2024-03-21T10:52:06.055973+0000 (oldest deadline 2024-03-21T10:52:29.555677+0000) 2024-03-21T10:52:33.456 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:33 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:33.323+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:33.456 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:33 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:33.324+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:33.456 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:33 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:33.324+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:33.456 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:33 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:33.324+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:33.572 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:33 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:33.116+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:33.573 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:33 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:33.116+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:33.573 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:33 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:33.116+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:06.747711+0000 front 2024-03-21T10:52:06.747844+0000 (oldest deadline 2024-03-21T10:52:32.047564+0000) 2024-03-21T10:52:33.573 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:33 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:33.346+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:33.573 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:33 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:33.346+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:33.573 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:33 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:33.346+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:33.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:33 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:33.843+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:33.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:33 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:33.843+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.999254+0000 front 2024-03-21T10:51:56.999105+0000 (oldest deadline 2024-03-21T10:52:22.199976+0000) 2024-03-21T10:52:33.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:33 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:33.843+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.200862+0000 front 2024-03-21T10:52:02.200915+0000 (oldest deadline 2024-03-21T10:52:28.100759+0000) 2024-03-21T10:52:33.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:33 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:33.622+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:33.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:33 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:33.622+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:33.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:33 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:33.622+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:33.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:33 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:33.622+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:34.304 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:34 smithi031 podman[104297]: 2024-03-21 10:52:34.04530515 +0000 UTC m=+5.936532837 container died d0481594607dacd95faeb3d13b59fd591c2ee6247f911bb0de5373c2809df1ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T10:52:34.374 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:34 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:34.109+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:34.374 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:34 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:34.109+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:34.374 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:34 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:34.109+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:06.747711+0000 front 2024-03-21T10:52:06.747844+0000 (oldest deadline 2024-03-21T10:52:32.047564+0000) 2024-03-21T10:52:34.653 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:34 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:34.308+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:34.653 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:34 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:34.308+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:34.653 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:34 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:34.308+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:34.653 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:34 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:34.308+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:34.654 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:34 smithi031 podman[104297]: 2024-03-21 10:52:34.467004798 +0000 UTC m=+6.358232450 container cleanup d0481594607dacd95faeb3d13b59fd591c2ee6247f911bb0de5373c2809df1ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T10:52:34.654 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:34 smithi031 bash[104297]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6 2024-03-21T10:52:34.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:34 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:34.377+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:34.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:34 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:34.377+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:34.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:34 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:34.377+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:34.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:34 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:34.815+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:34.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:34 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:34.815+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.999254+0000 front 2024-03-21T10:51:56.999105+0000 (oldest deadline 2024-03-21T10:52:22.199976+0000) 2024-03-21T10:52:34.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:34 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:34.815+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.200862+0000 front 2024-03-21T10:52:02.200915+0000 (oldest deadline 2024-03-21T10:52:28.100759+0000) 2024-03-21T10:52:34.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:34 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:34.572+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:34.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:34 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:34.572+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:34.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:34 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:34.572+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:34.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:34 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:34.572+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:34.975 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:34 smithi031 podman[104316]: 2024-03-21 10:52:34.827067623 +0000 UTC m=+0.743819130 container remove d0481594607dacd95faeb3d13b59fd591c2ee6247f911bb0de5373c2809df1ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T10:52:35.572 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:35 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:35.141+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:35.573 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:35 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:35.141+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:35.573 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:35 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:35.141+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:06.747711+0000 front 2024-03-21T10:52:06.747844+0000 (oldest deadline 2024-03-21T10:52:32.047564+0000) 2024-03-21T10:52:35.573 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:35 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:35.371+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:35.574 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:35 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:35.371+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:35.574 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:35 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:35.371+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:35.642 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:35 smithi031 podman[104411]: 2024-03-21 10:52:35.3338295 +0000 UTC m=+0.045183072 image pull 237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232 2024-03-21T10:52:35.642 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:35 smithi031 podman[104411]: 2024-03-21 10:52:35.494461971 +0000 UTC m=+0.205815565 container create 88ea630af37d34e8b782420495810e57c2e731ebd8e934ca2ea34576cf6644e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6) 2024-03-21T10:52:35.642 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:35 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:35.318+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:35.643 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:35 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:35.318+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:35.643 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:35 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:35.318+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:35.643 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:35 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:35.318+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:35.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:35 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:35.782+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:35.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:35 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:35.782+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.999254+0000 front 2024-03-21T10:51:56.999105+0000 (oldest deadline 2024-03-21T10:52:22.199976+0000) 2024-03-21T10:52:35.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:35 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:35.782+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.200862+0000 front 2024-03-21T10:52:02.200915+0000 (oldest deadline 2024-03-21T10:52:28.100759+0000) 2024-03-21T10:52:35.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:35 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:35.613+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:35.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:35 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:35.613+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:35.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:35 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:35.613+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:35.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:35 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:35.613+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:35.931 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:35 smithi031 podman[104411]: 2024-03-21 10:52:35.840732811 +0000 UTC m=+0.552086409 container init 88ea630af37d34e8b782420495810e57c2e731ebd8e934ca2ea34576cf6644e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True) 2024-03-21T10:52:35.931 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:35 smithi031 podman[104411]: 2024-03-21 10:52:35.851871599 +0000 UTC m=+0.563225162 container start 88ea630af37d34e8b782420495810e57c2e731ebd8e934ca2ea34576cf6644e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-21T10:52:36.291 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:35 smithi031 podman[104411]: 2024-03-21 10:52:35.956261363 +0000 UTC m=+0.667614980 container attach 88ea630af37d34e8b782420495810e57c2e731ebd8e934ca2ea34576cf6644e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-21T10:52:36.348 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:36 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:36.098+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:36.348 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:36 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:36.098+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:36.348 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:36 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:36.098+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:06.747711+0000 front 2024-03-21T10:52:06.747844+0000 (oldest deadline 2024-03-21T10:52:32.047564+0000) 2024-03-21T10:52:36.554 INFO:journalctl@ceph.osd.6.smithi031.stdout:Mar 21 10:52:36 smithi031 podman[104411]: 2024-03-21 10:52:36.295716774 +0000 UTC m=+1.007070439 container died 88ea630af37d34e8b782420495810e57c2e731ebd8e934ca2ea34576cf6644e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-03-21T10:52:36.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:36 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:36.323+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:36.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:36 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:36.323+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:36.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:36 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:36.323+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:36.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:36 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:36.323+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:36.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:36 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:36.352+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:36.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:36 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:36.352+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:36.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:36 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:36.352+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:36.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:36 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:36.743+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:36.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:36 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:36.743+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.999254+0000 front 2024-03-21T10:51:56.999105+0000 (oldest deadline 2024-03-21T10:52:22.199976+0000) 2024-03-21T10:52:36.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:36 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:36.743+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.200862+0000 front 2024-03-21T10:52:02.200915+0000 (oldest deadline 2024-03-21T10:52:28.100759+0000) 2024-03-21T10:52:36.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:36 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:36.648+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:36.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:36 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:36.648+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:36.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:36 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:36.648+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:36.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:36 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:36.648+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:37.214 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.6.service' 2024-03-21T10:52:37.333 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:37 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:37.083+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:37.333 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:37 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:37.083+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:37.333 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:37 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:37.083+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:06.747711+0000 front 2024-03-21T10:52:06.747844+0000 (oldest deadline 2024-03-21T10:52:32.047564+0000) 2024-03-21T10:52:37.347 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:52:37.347 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-21T10:52:37.347 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-21T10:52:37.347 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.7 2024-03-21T10:52:37.553 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:37 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:37.279+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:37.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:37 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:37.279+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:37.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:37 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:37.279+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:37.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:37 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:37.279+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:37.674 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:37 smithi040 systemd[1]: Stopping Ceph osd.7 for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:52:37.674 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:37 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:37.605+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:37.674 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:37 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:37.605+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:37.674 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:37 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:37.605+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:37.675 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:37 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:37.605+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:37.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:37 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:37.337+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:37.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:37 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:37.337+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:37.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:37 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:37.337+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:37.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:37 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:37.677+0000 7f38aef76700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:52:37.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:37 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:37.677+0000 7f38aef76700 -1 osd.7 112 *** Got signal Terminated *** 2024-03-21T10:52:37.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:37 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:37.677+0000 7f38aef76700 -1 osd.7 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T10:52:37.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:37 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:37.694+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:37.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:37 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:37.694+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.999254+0000 front 2024-03-21T10:51:56.999105+0000 (oldest deadline 2024-03-21T10:52:22.199976+0000) 2024-03-21T10:52:37.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:37 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:37.694+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.200862+0000 front 2024-03-21T10:52:02.200915+0000 (oldest deadline 2024-03-21T10:52:28.100759+0000) 2024-03-21T10:52:38.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:38 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:38.279+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:38.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:38 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:38.279+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:38.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:38 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:38.279+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:38.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:38 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:38.279+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:38.572 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:38 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:38.356+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:38.572 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:38 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:38.356+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:38.572 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:38 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:38.356+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:38.572 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:38 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:38.356+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:52:38.572 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:38 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:38.115+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:38.573 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:38 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:38.115+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:38.573 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:38 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:38.115+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:06.747711+0000 front 2024-03-21T10:52:06.747844+0000 (oldest deadline 2024-03-21T10:52:32.047564+0000) 2024-03-21T10:52:38.573 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:38 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:38.115+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.148710+0000 front 2024-03-21T10:52:13.148775+0000 (oldest deadline 2024-03-21T10:52:37.848711+0000) 2024-03-21T10:52:38.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:38 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:38.729+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:38.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:38 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:38.729+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.999254+0000 front 2024-03-21T10:51:56.999105+0000 (oldest deadline 2024-03-21T10:52:22.199976+0000) 2024-03-21T10:52:38.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:38 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:38.729+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.200862+0000 front 2024-03-21T10:52:02.200915+0000 (oldest deadline 2024-03-21T10:52:28.100759+0000) 2024-03-21T10:52:38.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:38 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:38.626+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:38.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:38 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:38.626+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:38.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:38 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:38.626+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:38.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:38 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:38.626+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:38.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:38 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:38.626+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:52:39.361 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:39 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:39.109+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:39.445 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:39 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:39.109+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:39.445 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:39 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:39.109+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:06.747711+0000 front 2024-03-21T10:52:06.747844+0000 (oldest deadline 2024-03-21T10:52:32.047564+0000) 2024-03-21T10:52:39.445 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:39 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:39.109+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.148710+0000 front 2024-03-21T10:52:13.148775+0000 (oldest deadline 2024-03-21T10:52:37.848711+0000) 2024-03-21T10:52:39.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:39 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:39.265+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:39.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:39 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:39.265+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:39.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:39 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:39.265+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:39.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:39 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:39.265+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:39.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:39 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:39.365+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:39.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:39 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:39.365+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:39.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:39 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:39.365+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:39.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:39 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:39.365+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:52:39.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:39 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:39.727+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:39.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:39 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:39.727+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.999254+0000 front 2024-03-21T10:51:56.999105+0000 (oldest deadline 2024-03-21T10:52:22.199976+0000) 2024-03-21T10:52:39.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:39 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:39.727+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.200862+0000 front 2024-03-21T10:52:02.200915+0000 (oldest deadline 2024-03-21T10:52:28.100759+0000) 2024-03-21T10:52:39.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:39 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:39.727+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:14.402838+0000 front 2024-03-21T10:52:14.402877+0000 (oldest deadline 2024-03-21T10:52:39.703053+0000) 2024-03-21T10:52:39.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:39 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:39.582+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:39.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:39 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:39.582+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:39.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:39 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:39.582+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:39.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:39 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:39.582+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:39.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:39 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:39.582+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:52:40.553 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:40 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:40.244+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:40.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:40 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:40.244+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:40.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:40 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:40.244+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:40.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:40 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:40.244+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:40.572 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:40 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:40.403+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:40.572 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:40 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:40.403+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:40.572 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:40 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:40.403+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:40.572 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:40 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:40.403+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:52:40.573 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:40 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:40.158+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:40.573 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:40 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:40.158+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:40.573 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:40 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:40.158+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:06.747711+0000 front 2024-03-21T10:52:06.747844+0000 (oldest deadline 2024-03-21T10:52:32.047564+0000) 2024-03-21T10:52:40.573 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:40 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:40.158+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.148710+0000 front 2024-03-21T10:52:13.148775+0000 (oldest deadline 2024-03-21T10:52:37.848711+0000) 2024-03-21T10:52:40.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:40 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:40.741+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:40.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:40 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:40.741+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.999254+0000 front 2024-03-21T10:51:56.999105+0000 (oldest deadline 2024-03-21T10:52:22.199976+0000) 2024-03-21T10:52:40.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:40 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:40.741+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.200862+0000 front 2024-03-21T10:52:02.200915+0000 (oldest deadline 2024-03-21T10:52:28.100759+0000) 2024-03-21T10:52:40.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:40 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:40.741+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:14.402838+0000 front 2024-03-21T10:52:14.402877+0000 (oldest deadline 2024-03-21T10:52:39.703053+0000) 2024-03-21T10:52:40.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:40 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:40.588+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:40.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:40 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:40.588+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:40.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:40 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:40.588+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:40.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:40 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:40.588+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:40.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:40 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:40.588+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:52:41.553 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:41 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:41.221+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:41.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:41 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:41.221+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:41.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:41 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:41.221+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:41.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:41 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:41.221+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:41.572 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:41 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:41.386+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:41.572 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:41 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:41.386+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:41.572 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:41 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:41.386+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:41.572 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:41 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:41.386+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:52:41.573 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:41 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:41.162+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:41.573 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:41 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:41.162+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:41.573 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:41 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:41.162+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:06.747711+0000 front 2024-03-21T10:52:06.747844+0000 (oldest deadline 2024-03-21T10:52:32.047564+0000) 2024-03-21T10:52:41.573 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:41 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:41.162+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.148710+0000 front 2024-03-21T10:52:13.148775+0000 (oldest deadline 2024-03-21T10:52:37.848711+0000) 2024-03-21T10:52:41.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:41 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:41.698+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.297240+0000 front 2024-03-21T10:51:48.297366+0000 (oldest deadline 2024-03-21T10:52:14.197431+0000) 2024-03-21T10:52:41.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:41 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:41.698+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.999254+0000 front 2024-03-21T10:51:56.999105+0000 (oldest deadline 2024-03-21T10:52:22.199976+0000) 2024-03-21T10:52:41.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:41 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:41.698+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.200862+0000 front 2024-03-21T10:52:02.200915+0000 (oldest deadline 2024-03-21T10:52:28.100759+0000) 2024-03-21T10:52:41.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:41 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7[54261]: 2024-03-21T10:52:41.698+0000 7f38a7562700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:14.402838+0000 front 2024-03-21T10:52:14.402877+0000 (oldest deadline 2024-03-21T10:52:39.703053+0000) 2024-03-21T10:52:41.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:41 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:41.614+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:41.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:41 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:41.614+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:41.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:41 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:41.614+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:41.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:41 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:41.614+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:41.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:41 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:41.614+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:52:42.397 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:42 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:42.125+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:42.397 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:42 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:42.125+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:42.397 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:42 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:42.125+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:06.747711+0000 front 2024-03-21T10:52:06.747844+0000 (oldest deadline 2024-03-21T10:52:32.047564+0000) 2024-03-21T10:52:42.397 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:42 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:42.125+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.148710+0000 front 2024-03-21T10:52:13.148775+0000 (oldest deadline 2024-03-21T10:52:37.848711+0000) 2024-03-21T10:52:42.553 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:42 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:42.187+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:42.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:42 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:42.187+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:42.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:42 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:42.187+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:42.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:42 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:42.187+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:42.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:42 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:42.400+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:42.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:42 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:42.400+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:42.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:42 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:42.400+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:42.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:42 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:42.400+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:52:42.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:42 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:42.613+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:42.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:42 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:42.613+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:42.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:42 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:42.613+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:42.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:42 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:42.613+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:42.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:42 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:42.613+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:52:43.443 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:43 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:43.101+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:43.443 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:43 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:43.101+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:43.443 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:43 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:43.101+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:06.747711+0000 front 2024-03-21T10:52:06.747844+0000 (oldest deadline 2024-03-21T10:52:32.047564+0000) 2024-03-21T10:52:43.443 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:43 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:43.101+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.148710+0000 front 2024-03-21T10:52:13.148775+0000 (oldest deadline 2024-03-21T10:52:37.848711+0000) 2024-03-21T10:52:43.498 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:43 smithi040 podman[73564]: 2024-03-21 10:52:43.206828292 +0000 UTC m=+5.654570340 container died 72f416a9c71feb21be2e4a7e79b567c82a38f57ee2c85e86ae967ba6a5921e00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6) 2024-03-21T10:52:43.553 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:43 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:43.139+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:43.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:43 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:43.140+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:43.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:43 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:43.140+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:43.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:43 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:43.140+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:43.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:43 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:43.446+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:43.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:43 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:43.446+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:43.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:43 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:43.446+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:43.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:43 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:43.446+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:52:43.858 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:43 smithi040 podman[73564]: 2024-03-21 10:52:43.59429321 +0000 UTC m=+6.042035256 container cleanup 72f416a9c71feb21be2e4a7e79b567c82a38f57ee2c85e86ae967ba6a5921e00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T10:52:43.858 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:43 smithi040 bash[73564]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7 2024-03-21T10:52:43.858 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:43 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:43.639+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:43.858 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:43 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:43.639+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:43.858 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:43 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:43.639+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:43.858 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:43 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:43.639+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:43.858 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:43 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:43.639+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:52:44.180 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:43 smithi040 podman[73581]: 2024-03-21 10:52:43.862334209 +0000 UTC m=+0.656120562 container remove 72f416a9c71feb21be2e4a7e79b567c82a38f57ee2c85e86ae967ba6a5921e00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.license=GPLv2) 2024-03-21T10:52:44.419 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:44 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:44.147+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:44.420 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:44 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:44.147+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:44.420 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:44 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:44.147+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:06.747711+0000 front 2024-03-21T10:52:06.747844+0000 (oldest deadline 2024-03-21T10:52:32.047564+0000) 2024-03-21T10:52:44.420 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:44 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:44.147+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.148710+0000 front 2024-03-21T10:52:13.148775+0000 (oldest deadline 2024-03-21T10:52:37.848711+0000) 2024-03-21T10:52:44.529 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:44 smithi040 podman[73669]: 2024-03-21 10:52:44.114201263 +0000 UTC m=+0.018229941 image pull 237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232 2024-03-21T10:52:44.530 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:44 smithi040 podman[73669]: 2024-03-21 10:52:44.213367597 +0000 UTC m=+0.117396293 container create 2d70857411c15ab2af6dd2253a1f21126501bc6abe11359fd5a96b2d0e73b173 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-21T10:52:44.530 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:44 smithi040 podman[73669]: 2024-03-21 10:52:44.457610805 +0000 UTC m=+0.361639496 container init 2d70857411c15ab2af6dd2253a1f21126501bc6abe11359fd5a96b2d0e73b173 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-21T10:52:44.530 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:44 smithi040 podman[73669]: 2024-03-21 10:52:44.460990559 +0000 UTC m=+0.365019228 container start 2d70857411c15ab2af6dd2253a1f21126501bc6abe11359fd5a96b2d0e73b173 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-21T10:52:44.530 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:44 smithi040 podman[73669]: 2024-03-21 10:52:44.53266591 +0000 UTC m=+0.436694601 container attach 2d70857411c15ab2af6dd2253a1f21126501bc6abe11359fd5a96b2d0e73b173 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, ceph=True, GIT_CLEAN=True) 2024-03-21T10:52:44.553 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:44 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:44.187+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:44.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:44 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:44.187+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:44.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:44 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:44.187+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:44.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:44 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:44.187+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:44.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:44 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:44.423+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:44.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:44 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:44.423+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:44.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:44 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:44.423+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:44.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:44 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:44.423+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:52:44.930 INFO:journalctl@ceph.osd.7.smithi040.stdout:Mar 21 10:52:44 smithi040 podman[73669]: 2024-03-21 10:52:44.767709004 +0000 UTC m=+0.671737681 container died 2d70857411c15ab2af6dd2253a1f21126501bc6abe11359fd5a96b2d0e73b173 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-21T10:52:44.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:44 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:44.626+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:44.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:44 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:44.626+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:44.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:44 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:44.626+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:44.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:44 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:44.626+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:44.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:44 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:44.626+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:52:45.384 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:45 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:45.126+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:45.384 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:45 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:45.126+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:45.384 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:45 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:45.126+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:06.747711+0000 front 2024-03-21T10:52:06.747844+0000 (oldest deadline 2024-03-21T10:52:32.047564+0000) 2024-03-21T10:52:45.384 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:45 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:45.126+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.148710+0000 front 2024-03-21T10:52:13.148775+0000 (oldest deadline 2024-03-21T10:52:37.848711+0000) 2024-03-21T10:52:45.473 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.7.service' 2024-03-21T10:52:45.518 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:52:45.518 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-21T10:52:45.519 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-21T10:52:45.519 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.8 2024-03-21T10:52:45.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:45 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:45.185+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:45.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:45 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:45.185+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:45.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:45 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:45.185+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:45.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:45 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:45.185+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:45.680 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:45 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:45.585+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:45.680 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:45 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:45.585+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:45.680 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:45 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:45.585+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:45.680 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:45 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:45.585+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:45.681 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:45 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:45.585+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:52:45.822 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:45 smithi070 systemd[1]: Stopping Ceph osd.8 for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:52:45.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:45 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:45.387+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:45.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:45 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:45.387+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:45.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:45 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:45.387+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:45.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:45 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:45.388+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:52:46.165 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:45 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:45.836+0000 7f45e40c7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:52:46.165 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:45 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:45.836+0000 7f45e40c7700 -1 osd.8 112 *** Got signal Terminated *** 2024-03-21T10:52:46.165 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:45 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:45.836+0000 7f45e40c7700 -1 osd.8 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T10:52:46.165 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:46 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:46.169+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:46.166 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:46 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:46.169+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:46.166 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:46 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:46.169+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:06.747711+0000 front 2024-03-21T10:52:06.747844+0000 (oldest deadline 2024-03-21T10:52:32.047564+0000) 2024-03-21T10:52:46.166 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:46 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:46.169+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.148710+0000 front 2024-03-21T10:52:13.148775+0000 (oldest deadline 2024-03-21T10:52:37.848711+0000) 2024-03-21T10:52:46.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:46 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:46.187+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:46.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:46 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:46.187+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:46.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:46 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:46.187+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:46.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:46 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:46.187+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:46.572 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:46 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:46.406+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:46.572 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:46 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:46.406+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:46.572 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:46 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:46.406+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:46.572 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:46 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:46.406+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:52:46.680 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:46 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:46.569+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:46.680 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:46 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:46.569+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:46.680 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:46 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:46.569+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:46.680 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:46 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:46.569+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:46.680 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:46 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:46.569+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:52:46.680 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:46 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:46.569+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:22.871708+0000 front 2024-03-21T10:52:22.871647+0000 (oldest deadline 2024-03-21T10:52:45.771600+0000) 2024-03-21T10:52:47.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:47 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:47.166+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:47.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:47 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:47.166+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:47.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:47 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:47.166+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:47.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:47 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:47.166+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:47.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:47 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:47.166+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:20.471066+0000 front 2024-03-21T10:52:20.471022+0000 (oldest deadline 2024-03-21T10:52:46.371284+0000) 2024-03-21T10:52:47.572 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:47 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:47.166+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:47.572 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:47 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:47.166+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:47.572 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:47 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:47.166+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:06.747711+0000 front 2024-03-21T10:52:06.747844+0000 (oldest deadline 2024-03-21T10:52:32.047564+0000) 2024-03-21T10:52:47.573 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:47 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:47.166+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.148710+0000 front 2024-03-21T10:52:13.148775+0000 (oldest deadline 2024-03-21T10:52:37.848711+0000) 2024-03-21T10:52:47.573 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:47 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:47.395+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:47.573 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:47 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:47.395+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:47.573 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:47 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:47.395+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:47.573 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:47 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:47.395+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:52:47.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:47 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:47.551+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:47.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:47 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:47.551+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:47.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:47 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:47.551+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:47.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:47 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:47.551+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:47.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:47 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:47.551+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:52:47.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:47 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:47.551+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:22.871708+0000 front 2024-03-21T10:52:22.871647+0000 (oldest deadline 2024-03-21T10:52:45.771600+0000) 2024-03-21T10:52:48.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:48 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:48.126+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:48.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:48 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:48.126+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:48.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:48 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:48.126+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:48.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:48 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:48.126+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:48.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:48 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:48.126+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:20.471066+0000 front 2024-03-21T10:52:20.471022+0000 (oldest deadline 2024-03-21T10:52:46.371284+0000) 2024-03-21T10:52:48.572 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:48 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:48.121+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:48.572 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:48 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:48.121+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:48.572 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:48 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:48.121+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:06.747711+0000 front 2024-03-21T10:52:06.747844+0000 (oldest deadline 2024-03-21T10:52:32.047564+0000) 2024-03-21T10:52:48.572 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:48 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:48.121+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.148710+0000 front 2024-03-21T10:52:13.148775+0000 (oldest deadline 2024-03-21T10:52:37.848711+0000) 2024-03-21T10:52:48.573 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:48 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:48.121+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:21.949631+0000 front 2024-03-21T10:52:21.949653+0000 (oldest deadline 2024-03-21T10:52:47.849634+0000) 2024-03-21T10:52:48.573 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:48 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:48.346+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:48.573 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:48 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:48.346+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:48.573 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:48 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:48.346+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:48.573 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:48 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:48.346+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:52:48.573 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:48 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:48.346+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:52:48.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:48 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:48.509+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:48.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:48 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:48.509+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:48.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:48 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:48.509+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:48.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:48 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:48.509+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:48.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:48 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:48.509+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:52:48.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:48 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:48.509+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:22.871708+0000 front 2024-03-21T10:52:22.871647+0000 (oldest deadline 2024-03-21T10:52:45.771600+0000) 2024-03-21T10:52:49.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:49 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:49.088+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:49.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:49 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:49.088+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:49.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:49 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:49.088+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:49.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:49 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:49.088+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:49.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:49 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:49.088+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:20.471066+0000 front 2024-03-21T10:52:20.471022+0000 (oldest deadline 2024-03-21T10:52:46.371284+0000) 2024-03-21T10:52:49.572 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:49 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:49.361+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:49.572 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:49 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:49.361+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:49.572 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:49 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:49.361+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:49.572 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:49 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:49.361+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:52:49.572 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:49 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:49.361+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:52:49.572 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:49 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:49.112+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:49.573 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:49 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:49.112+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:49.573 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:49 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:49.112+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:06.747711+0000 front 2024-03-21T10:52:06.747844+0000 (oldest deadline 2024-03-21T10:52:32.047564+0000) 2024-03-21T10:52:49.573 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:49 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:49.112+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.148710+0000 front 2024-03-21T10:52:13.148775+0000 (oldest deadline 2024-03-21T10:52:37.848711+0000) 2024-03-21T10:52:49.573 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:49 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:49.112+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:21.949631+0000 front 2024-03-21T10:52:21.949653+0000 (oldest deadline 2024-03-21T10:52:47.849634+0000) 2024-03-21T10:52:49.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:49 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:49.508+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:49.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:49 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:49.508+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:49.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:49 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:49.508+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:49.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:49 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:49.508+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:49.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:49 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:49.508+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:52:49.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:49 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:49.508+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:22.871708+0000 front 2024-03-21T10:52:22.871647+0000 (oldest deadline 2024-03-21T10:52:45.771600+0000) 2024-03-21T10:52:50.322 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:50 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:50.063+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.843768+0000 front 2024-03-21T10:51:48.843906+0000 (oldest deadline 2024-03-21T10:52:12.943880+0000) 2024-03-21T10:52:50.322 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:50 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:50.063+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:53.945274+0000 front 2024-03-21T10:51:53.945305+0000 (oldest deadline 2024-03-21T10:52:19.845506+0000) 2024-03-21T10:52:50.323 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:50 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:50.063+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:06.747711+0000 front 2024-03-21T10:52:06.747844+0000 (oldest deadline 2024-03-21T10:52:32.047564+0000) 2024-03-21T10:52:50.323 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:50 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:50.063+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.148710+0000 front 2024-03-21T10:52:13.148775+0000 (oldest deadline 2024-03-21T10:52:37.848711+0000) 2024-03-21T10:52:50.323 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:50 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8[55218]: 2024-03-21T10:52:50.063+0000 7f45dc6b3700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:21.949631+0000 front 2024-03-21T10:52:21.949653+0000 (oldest deadline 2024-03-21T10:52:47.849634+0000) 2024-03-21T10:52:50.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:50 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:50.113+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:50.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:50 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:50.113+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:50.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:50 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:50.113+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:50.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:50 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:50.113+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:50.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:50 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:50.113+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:20.471066+0000 front 2024-03-21T10:52:20.471022+0000 (oldest deadline 2024-03-21T10:52:46.371284+0000) 2024-03-21T10:52:50.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:50 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:50.408+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:50.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:50 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:50.408+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:50.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:50 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:50.408+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:50.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:50 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:50.408+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:52:50.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:50 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:50.408+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:52:50.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:50 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:50.555+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:50.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:50 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:50.555+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:50.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:50 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:50.555+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:50.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:50 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:50.555+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:50.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:50 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:50.555+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:52:50.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:50 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:50.555+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:22.871708+0000 front 2024-03-21T10:52:22.871647+0000 (oldest deadline 2024-03-21T10:52:45.771600+0000) 2024-03-21T10:52:51.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:51 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:51.134+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:51.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:51 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:51.134+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:51.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:51 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:51.134+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:51.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:51 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:51.134+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:51.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:51 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:51.134+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:20.471066+0000 front 2024-03-21T10:52:20.471022+0000 (oldest deadline 2024-03-21T10:52:46.371284+0000) 2024-03-21T10:52:51.782 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:51 smithi070 podman[74328]: 2024-03-21 10:52:51.478903624 +0000 UTC m=+5.761052383 container died 8e799f5de3323d60242fc48a2fa24bf240cc282a917ebe23b2cd56fa7761b123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True) 2024-03-21T10:52:51.783 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:51 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:51.431+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:51.783 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:51 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:51.431+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:51.783 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:51 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:51.431+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:51.783 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:51 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:51.431+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:52:51.783 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:51 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:51.431+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:52:51.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:51 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:51.586+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:51.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:51 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:51.586+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:51.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:51 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:51.586+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:51.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:51 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:51.586+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:51.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:51 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:51.586+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:52:51.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:51 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:51.586+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:22.871708+0000 front 2024-03-21T10:52:22.871647+0000 (oldest deadline 2024-03-21T10:52:45.771600+0000) 2024-03-21T10:52:52.072 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:51 smithi070 podman[74328]: 2024-03-21 10:52:51.869572924 +0000 UTC m=+6.151721676 container cleanup 8e799f5de3323d60242fc48a2fa24bf240cc282a917ebe23b2cd56fa7761b123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-21T10:52:52.072 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:51 smithi070 bash[74328]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8 2024-03-21T10:52:52.403 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:52 smithi070 podman[74344]: 2024-03-21 10:52:52.112302731 +0000 UTC m=+0.628264534 container remove 8e799f5de3323d60242fc48a2fa24bf240cc282a917ebe23b2cd56fa7761b123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T10:52:52.553 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:52 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:52.121+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:52.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:52 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:52.121+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:52.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:52 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:52.121+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:52.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:52 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:52.121+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:52.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:52 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:52.121+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:20.471066+0000 front 2024-03-21T10:52:20.471022+0000 (oldest deadline 2024-03-21T10:52:46.371284+0000) 2024-03-21T10:52:52.670 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:52 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:52.407+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:52.670 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:52 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:52.407+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:52.670 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:52 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:52.407+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:52.670 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:52 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:52.407+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:52:52.670 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:52 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:52.407+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:52:52.670 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:52 smithi070 podman[74440]: 2024-03-21 10:52:52.339935642 +0000 UTC m=+0.018287143 image pull 237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232 2024-03-21T10:52:52.670 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:52 smithi070 podman[74440]: 2024-03-21 10:52:52.4418634 +0000 UTC m=+0.120214882 container create 06096f78ada20fbdd8eac7a5f7b96e1cff7ba9369415c74be78308ded806f374 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.build-date=20240319) 2024-03-21T10:52:52.671 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:52 smithi070 podman[74440]: 2024-03-21 10:52:52.674077806 +0000 UTC m=+0.352429314 container init 06096f78ada20fbdd8eac7a5f7b96e1cff7ba9369415c74be78308ded806f374 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-21T10:52:52.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:52 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:52.623+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:52.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:52 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:52.623+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:52.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:52 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:52.623+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:52.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:52 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:52.623+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:52.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:52 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:52.623+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:52:52.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:52 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:52.623+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:22.871708+0000 front 2024-03-21T10:52:22.871647+0000 (oldest deadline 2024-03-21T10:52:45.771600+0000) 2024-03-21T10:52:52.958 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:52 smithi070 podman[74440]: 2024-03-21 10:52:52.67758914 +0000 UTC m=+0.355940638 container start 06096f78ada20fbdd8eac7a5f7b96e1cff7ba9369415c74be78308ded806f374 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-21T10:52:52.959 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:52 smithi070 podman[74440]: 2024-03-21 10:52:52.765612254 +0000 UTC m=+0.443963755 container attach 06096f78ada20fbdd8eac7a5f7b96e1cff7ba9369415c74be78308ded806f374 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-21T10:52:53.247 INFO:journalctl@ceph.osd.8.smithi070.stdout:Mar 21 10:52:52 smithi070 podman[74440]: 2024-03-21 10:52:52.963422129 +0000 UTC m=+0.641773649 container died 06096f78ada20fbdd8eac7a5f7b96e1cff7ba9369415c74be78308ded806f374 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-8-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6) 2024-03-21T10:52:53.523 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:53 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:53.414+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:53.523 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:53 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:53.414+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:53.523 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:53 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:53.414+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:53.523 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:53 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:53.414+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:52:53.523 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:53 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:53.414+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:52:53.544 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.8.service' 2024-03-21T10:52:53.553 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:53 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:53.074+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:53.553 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:53 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:53.074+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:53.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:53 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:53.074+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:53.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:53 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:53.074+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:53.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:53 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:53.074+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:20.471066+0000 front 2024-03-21T10:52:20.471022+0000 (oldest deadline 2024-03-21T10:52:46.371284+0000) 2024-03-21T10:52:53.587 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:52:53.587 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-21T10:52:53.587 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-21T10:52:53.587 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.9 2024-03-21T10:52:53.929 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:53 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:53.656+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:53.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:53 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:53.656+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:53.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:53 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:53.656+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:53.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:53 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:53.656+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:53.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:53 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:53.656+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:52:53.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:53 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:53.656+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:22.871708+0000 front 2024-03-21T10:52:22.871647+0000 (oldest deadline 2024-03-21T10:52:45.771600+0000) 2024-03-21T10:52:54.053 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:53 smithi031 systemd[1]: Stopping Ceph osd.9 for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:52:54.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:54 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:54.088+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:54.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:54 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:54.088+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:54.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:54 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:54.088+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:54.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:54 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:54.088+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:54.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:54 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:54.088+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:20.471066+0000 front 2024-03-21T10:52:20.471022+0000 (oldest deadline 2024-03-21T10:52:46.371284+0000) 2024-03-21T10:52:54.555 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:54 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:54.294+0000 7f746a200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:52:54.555 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:54 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:54.294+0000 7f746a200700 -1 osd.9 112 *** Got signal Terminated *** 2024-03-21T10:52:54.555 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:54 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:54.294+0000 7f746a200700 -1 osd.9 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T10:52:54.728 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:54 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:54.462+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:54.728 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:54 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:54.462+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:54.728 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:54 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:54.462+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:54.728 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:54 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:54.462+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:52:54.729 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:54 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:54.462+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:52:54.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:54 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:54.655+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:54.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:54 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:54.655+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:54.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:54 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:54.655+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:54.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:54 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:54.655+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:54.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:54 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:54.655+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:52:54.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:54 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:54.655+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:22.871708+0000 front 2024-03-21T10:52:22.871647+0000 (oldest deadline 2024-03-21T10:52:45.771600+0000) 2024-03-21T10:52:54.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:54 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:54.655+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.773577+0000 front 2024-03-21T10:52:32.773473+0000 (oldest deadline 2024-03-21T10:52:54.373581+0000) 2024-03-21T10:52:55.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:55 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:55.077+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:55.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:55 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:55.077+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:55.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:55 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:55.077+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:55.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:55 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:55.077+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:55.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:55 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:55.077+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:20.471066+0000 front 2024-03-21T10:52:20.471022+0000 (oldest deadline 2024-03-21T10:52:46.371284+0000) 2024-03-21T10:52:55.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:55 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:55.483+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:55.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:55 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:55.483+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:55.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:55 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:55.483+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:55.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:55 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:55.483+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:52:55.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:55 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:55.483+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:52:55.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:55 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:55.642+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:55.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:55 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:55.642+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:55.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:55 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:55.642+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:55.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:55 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:55.642+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:55.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:55 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:55.642+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:52:55.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:55 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:55.642+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:22.871708+0000 front 2024-03-21T10:52:22.871647+0000 (oldest deadline 2024-03-21T10:52:45.771600+0000) 2024-03-21T10:52:55.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:55 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:55.642+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.773577+0000 front 2024-03-21T10:52:32.773473+0000 (oldest deadline 2024-03-21T10:52:54.373581+0000) 2024-03-21T10:52:56.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:56 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:56.103+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:56.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:56 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:56.103+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:56.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:56 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:56.103+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:56.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:56 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:56.103+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:56.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:56 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:56.103+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:20.471066+0000 front 2024-03-21T10:52:20.471022+0000 (oldest deadline 2024-03-21T10:52:46.371284+0000) 2024-03-21T10:52:56.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:56 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:56.455+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:56.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:56 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:56.455+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:56.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:56 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:56.455+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:56.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:56 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:56.455+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:52:56.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:56 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:56.455+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:52:56.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:56 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:56.626+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:56.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:56 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:56.626+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:56.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:56 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:56.626+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:56.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:56 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:56.626+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:56.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:56 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:56.626+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:52:56.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:56 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:56.626+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:22.871708+0000 front 2024-03-21T10:52:22.871647+0000 (oldest deadline 2024-03-21T10:52:45.771600+0000) 2024-03-21T10:52:56.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:56 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:56.626+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.773577+0000 front 2024-03-21T10:52:32.773473+0000 (oldest deadline 2024-03-21T10:52:54.373581+0000) 2024-03-21T10:52:57.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:57 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:57.094+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:57.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:57 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:57.094+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:57.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:57 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:57.094+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:57.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:57 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:57.094+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:57.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:57 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:57.094+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:20.471066+0000 front 2024-03-21T10:52:20.471022+0000 (oldest deadline 2024-03-21T10:52:46.371284+0000) 2024-03-21T10:52:57.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:57 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:57.094+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:30.974649+0000 front 2024-03-21T10:52:30.975084+0000 (oldest deadline 2024-03-21T10:52:56.274717+0000) 2024-03-21T10:52:57.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:57 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:57.433+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:57.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:57 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:57.433+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:57.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:57 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:57.433+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:57.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:57 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:57.433+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:52:57.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:57 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:57.433+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:52:57.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:57 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:57.671+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:57.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:57 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:57.671+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:57.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:57 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:57.671+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:57.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:57 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:57.671+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:57.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:57 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:57.671+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:52:57.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:57 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:57.671+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:22.871708+0000 front 2024-03-21T10:52:22.871647+0000 (oldest deadline 2024-03-21T10:52:45.771600+0000) 2024-03-21T10:52:57.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:57 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:57.671+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.773577+0000 front 2024-03-21T10:52:32.773473+0000 (oldest deadline 2024-03-21T10:52:54.373581+0000) 2024-03-21T10:52:58.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:58 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:58.080+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:58.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:58 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:58.080+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:58.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:58 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:58.080+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:58.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:58 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:58.080+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:58.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:58 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:58.080+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:20.471066+0000 front 2024-03-21T10:52:20.471022+0000 (oldest deadline 2024-03-21T10:52:46.371284+0000) 2024-03-21T10:52:58.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:58 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:58.080+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:30.974649+0000 front 2024-03-21T10:52:30.975084+0000 (oldest deadline 2024-03-21T10:52:56.274717+0000) 2024-03-21T10:52:58.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:58 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:58.481+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:58.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:58 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:58.481+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:58.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:58 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:58.481+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:58.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:58 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:58.481+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:52:58.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:58 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:58.481+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:52:58.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:58 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:58.481+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.508171+0000 front 2024-03-21T10:52:32.508006+0000 (oldest deadline 2024-03-21T10:52:57.807721+0000) 2024-03-21T10:52:59.179 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:58 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:58.687+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:59.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:58 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:58.687+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:59.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:58 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:58.687+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:59.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:58 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:58.687+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:59.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:58 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:58.687+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:52:59.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:58 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:58.687+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:22.871708+0000 front 2024-03-21T10:52:22.871647+0000 (oldest deadline 2024-03-21T10:52:45.771600+0000) 2024-03-21T10:52:59.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:58 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:58.687+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.773577+0000 front 2024-03-21T10:52:32.773473+0000 (oldest deadline 2024-03-21T10:52:54.373581+0000) 2024-03-21T10:52:59.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:59 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:59.087+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.258760+0000 front 2024-03-21T10:51:37.258400+0000 (oldest deadline 2024-03-21T10:52:00.758567+0000) 2024-03-21T10:52:59.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:59 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:59.087+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:47.161878+0000 front 2024-03-21T10:51:47.162009+0000 (oldest deadline 2024-03-21T10:52:11.861745+0000) 2024-03-21T10:52:59.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:59 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:59.087+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.764301+0000 front 2024-03-21T10:51:57.764589+0000 (oldest deadline 2024-03-21T10:52:18.864101+0000) 2024-03-21T10:52:59.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:59 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:59.087+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:02.366734+0000 front 2024-03-21T10:52:02.366128+0000 (oldest deadline 2024-03-21T10:52:27.666113+0000) 2024-03-21T10:52:59.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:59 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:59.087+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:20.471066+0000 front 2024-03-21T10:52:20.471022+0000 (oldest deadline 2024-03-21T10:52:46.371284+0000) 2024-03-21T10:52:59.554 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:52:59 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9[66175]: 2024-03-21T10:52:59.087+0000 7f7465a00700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:30.974649+0000 front 2024-03-21T10:52:30.975084+0000 (oldest deadline 2024-03-21T10:52:56.274717+0000) 2024-03-21T10:52:59.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:59 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:59.516+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:52:59.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:59 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:59.516+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:52:59.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:59 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:59.516+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:52:59.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:59 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:59.516+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:52:59.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:59 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:59.516+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:52:59.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:52:59 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:52:59.516+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.508171+0000 front 2024-03-21T10:52:32.508006+0000 (oldest deadline 2024-03-21T10:52:57.807721+0000) 2024-03-21T10:52:59.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:59 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:59.659+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:52:59.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:59 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:59.659+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:52:59.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:59 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:59.659+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:52:59.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:59 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:59.659+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:52:59.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:59 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:59.659+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:52:59.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:59 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:59.659+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:22.871708+0000 front 2024-03-21T10:52:22.871647+0000 (oldest deadline 2024-03-21T10:52:45.771600+0000) 2024-03-21T10:52:59.931 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:52:59 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:52:59.659+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.773577+0000 front 2024-03-21T10:52:32.773473+0000 (oldest deadline 2024-03-21T10:52:54.373581+0000) 2024-03-21T10:53:00.498 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:53:00 smithi031 podman[104518]: 2024-03-21 10:53:00.148989429 +0000 UTC m=+6.138446774 container died 78e358f696b22ab2d7cb69db63b9ccb0a0509dca9e6323ec620b7aa4f03ea2f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-21T10:53:00.803 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:53:00 smithi031 podman[104518]: 2024-03-21 10:53:00.620329867 +0000 UTC m=+6.609787206 container cleanup 78e358f696b22ab2d7cb69db63b9ccb0a0509dca9e6323ec620b7aa4f03ea2f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6) 2024-03-21T10:53:00.803 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:53:00 smithi031 bash[104518]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9 2024-03-21T10:53:00.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:00 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:00.509+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:53:00.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:00 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:00.509+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:53:00.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:00 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:00.509+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:53:00.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:00 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:00.509+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:53:00.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:00 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:00.509+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:53:00.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:00 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:00.509+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.508171+0000 front 2024-03-21T10:52:32.508006+0000 (oldest deadline 2024-03-21T10:52:57.807721+0000) 2024-03-21T10:53:01.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:00 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:00.700+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:53:01.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:00 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:00.700+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:53:01.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:00 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:00.700+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:53:01.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:00 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:00.700+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:53:01.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:00 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:00.700+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:53:01.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:00 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:00.700+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:22.871708+0000 front 2024-03-21T10:52:22.871647+0000 (oldest deadline 2024-03-21T10:52:45.771600+0000) 2024-03-21T10:53:01.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:00 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:00.700+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.773577+0000 front 2024-03-21T10:52:32.773473+0000 (oldest deadline 2024-03-21T10:52:54.373581+0000) 2024-03-21T10:53:01.303 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:53:00 smithi031 podman[104536]: 2024-03-21 10:53:00.905227715 +0000 UTC m=+0.723581253 container remove 78e358f696b22ab2d7cb69db63b9ccb0a0509dca9e6323ec620b7aa4f03ea2f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-21T10:53:01.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:01 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:01.474+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:53:01.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:01 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:01.474+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:53:01.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:01 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:01.474+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:53:01.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:01 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:01.474+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:53:01.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:01 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:01.474+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:53:01.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:01 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:01.474+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.508171+0000 front 2024-03-21T10:52:32.508006+0000 (oldest deadline 2024-03-21T10:52:57.807721+0000) 2024-03-21T10:53:02.040 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:53:01 smithi031 podman[104619]: 2024-03-21 10:53:01.643789622 +0000 UTC m=+0.038481139 image pull 237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232 2024-03-21T10:53:02.041 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:53:01 smithi031 podman[104619]: 2024-03-21 10:53:01.764264891 +0000 UTC m=+0.158956421 container create 0d4edda74ce89d2315e0b37b12999934f8003d100cecc18587b0ca1a5495a019 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-21T10:53:02.041 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:53:02 smithi031 podman[104619]: 2024-03-21 10:53:02.042520774 +0000 UTC m=+0.437212311 container init 0d4edda74ce89d2315e0b37b12999934f8003d100cecc18587b0ca1a5495a019 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-21T10:53:02.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:01 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:01.718+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:53:02.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:01 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:01.718+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:53:02.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:01 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:01.718+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:53:02.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:01 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:01.718+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:53:02.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:01 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:01.718+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:53:02.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:01 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:01.718+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:22.871708+0000 front 2024-03-21T10:52:22.871647+0000 (oldest deadline 2024-03-21T10:52:45.771600+0000) 2024-03-21T10:53:02.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:01 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:01.718+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.773577+0000 front 2024-03-21T10:52:32.773473+0000 (oldest deadline 2024-03-21T10:52:54.373581+0000) 2024-03-21T10:53:02.303 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:53:02 smithi031 podman[104619]: 2024-03-21 10:53:02.055843643 +0000 UTC m=+0.450535146 container start 0d4edda74ce89d2315e0b37b12999934f8003d100cecc18587b0ca1a5495a019 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-21T10:53:02.303 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:53:02 smithi031 podman[104619]: 2024-03-21 10:53:02.208439674 +0000 UTC m=+0.603131234 container attach 0d4edda74ce89d2315e0b37b12999934f8003d100cecc18587b0ca1a5495a019 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9-deactivate, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-21T10:53:02.802 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:53:02 smithi031 podman[104619]: 2024-03-21 10:53:02.513504645 +0000 UTC m=+0.908196182 container died 0d4edda74ce89d2315e0b37b12999934f8003d100cecc18587b0ca1a5495a019 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-21T10:53:02.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:02 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:02.487+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:53:02.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:02 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:02.487+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:53:02.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:02 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:02.487+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:53:02.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:02 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:02.487+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:53:02.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:02 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:02.487+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:53:02.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:02 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:02.487+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.508171+0000 front 2024-03-21T10:52:32.508006+0000 (oldest deadline 2024-03-21T10:52:57.807721+0000) 2024-03-21T10:53:03.081 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:02 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:02.761+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:53:03.081 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:02 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:02.761+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:53:03.081 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:02 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:02.761+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:53:03.081 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:02 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:02.761+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:53:03.081 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:02 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:02.761+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:53:03.082 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:02 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:02.761+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:22.871708+0000 front 2024-03-21T10:52:22.871647+0000 (oldest deadline 2024-03-21T10:52:45.771600+0000) 2024-03-21T10:53:03.082 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:02 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:02.761+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.773577+0000 front 2024-03-21T10:52:32.773473+0000 (oldest deadline 2024-03-21T10:52:54.373581+0000) 2024-03-21T10:53:03.243 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.9.service' 2024-03-21T10:53:03.328 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:53:03 smithi031 podman[104645]: 2024-03-21 10:53:03.208549777 +0000 UTC m=+0.680538060 container remove 0d4edda74ce89d2315e0b37b12999934f8003d100cecc18587b0ca1a5495a019 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-9-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-21T10:53:03.328 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:53:03 smithi031 systemd[1]: ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.9.service: Deactivated successfully. 2024-03-21T10:53:03.329 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:53:03 smithi031 systemd[1]: Stopped Ceph osd.9 for 3847408c-e76d-11ee-95cd-87774f69a715. 2024-03-21T10:53:03.329 INFO:journalctl@ceph.osd.9.smithi031.stdout:Mar 21 10:53:03 smithi031 systemd[1]: ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.9.service: Consumed 6min 31.726s CPU time. 2024-03-21T10:53:03.365 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:53:03.366 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-21T10:53:03.366 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-21T10:53:03.366 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.10 2024-03-21T10:53:03.679 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:03 smithi040 systemd[1]: Stopping Ceph osd.10 for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:53:03.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:03 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:03.504+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:53:03.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:03 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:03.504+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:53:03.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:03 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:03.504+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:53:03.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:03 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:03.505+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:53:03.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:03 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:03.505+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:53:03.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:03 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:03.505+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.508171+0000 front 2024-03-21T10:52:32.508006+0000 (oldest deadline 2024-03-21T10:52:57.807721+0000) 2024-03-21T10:53:04.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:03 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:03.724+0000 7f43ed5ff700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:53:04.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:03 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:03.724+0000 7f43ed5ff700 -1 osd.10 112 *** Got signal Terminated *** 2024-03-21T10:53:04.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:03 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:03.724+0000 7f43ed5ff700 -1 osd.10 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T10:53:04.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:03 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:03.791+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:53:04.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:03 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:03.791+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:53:04.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:03 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:03.791+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:53:04.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:03 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:03.791+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:53:04.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:03 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:03.791+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:53:04.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:03 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:03.791+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:22.871708+0000 front 2024-03-21T10:52:22.871647+0000 (oldest deadline 2024-03-21T10:52:45.771600+0000) 2024-03-21T10:53:04.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:03 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:03.791+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.773577+0000 front 2024-03-21T10:52:32.773473+0000 (oldest deadline 2024-03-21T10:52:54.373581+0000) 2024-03-21T10:53:04.798 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:04 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:04.546+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:53:04.799 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:04 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:04.546+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:53:04.799 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:04 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:04.546+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:53:04.799 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:04 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:04.546+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:53:04.799 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:04 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:04.546+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:53:04.799 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:04 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:04.546+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.508171+0000 front 2024-03-21T10:52:32.508006+0000 (oldest deadline 2024-03-21T10:52:57.807721+0000) 2024-03-21T10:53:04.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:04 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:04.805+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:53:04.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:04 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:04.805+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:53:04.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:04 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:04.805+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:53:04.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:04 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:04.805+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:53:04.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:04 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:04.805+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:53:04.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:04 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:04.805+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:22.871708+0000 front 2024-03-21T10:52:22.871647+0000 (oldest deadline 2024-03-21T10:52:45.771600+0000) 2024-03-21T10:53:04.930 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:04 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:04.805+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.773577+0000 front 2024-03-21T10:52:32.773473+0000 (oldest deadline 2024-03-21T10:52:54.373581+0000) 2024-03-21T10:53:05.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:05 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:05.520+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:53:05.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:05 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:05.520+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:53:05.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:05 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:05.520+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:53:05.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:05 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:05.520+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:53:05.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:05 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:05.520+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:53:05.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:05 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:05.520+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.508171+0000 front 2024-03-21T10:52:32.508006+0000 (oldest deadline 2024-03-21T10:52:57.807721+0000) 2024-03-21T10:53:05.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:05 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:05.520+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-03-21T10:52:39.508873+0000 front 2024-03-21T10:52:39.509004+0000 (oldest deadline 2024-03-21T10:53:05.408890+0000) 2024-03-21T10:53:06.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:05 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:05.839+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:53:06.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:05 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:05.839+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:53:06.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:05 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:05.839+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:53:06.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:05 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:05.839+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:53:06.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:05 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:05.839+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:53:06.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:05 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:05.839+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:22.871708+0000 front 2024-03-21T10:52:22.871647+0000 (oldest deadline 2024-03-21T10:52:45.771600+0000) 2024-03-21T10:53:06.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:05 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:05.839+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.773577+0000 front 2024-03-21T10:52:32.773473+0000 (oldest deadline 2024-03-21T10:52:54.373581+0000) 2024-03-21T10:53:06.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:06 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:06.473+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:53:06.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:06 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:06.473+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:53:06.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:06 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:06.473+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:53:06.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:06 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:06.473+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:53:06.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:06 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:06.473+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:53:06.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:06 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:06.473+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.508171+0000 front 2024-03-21T10:52:32.508006+0000 (oldest deadline 2024-03-21T10:52:57.807721+0000) 2024-03-21T10:53:06.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:06 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:06.473+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-03-21T10:52:39.508873+0000 front 2024-03-21T10:52:39.509004+0000 (oldest deadline 2024-03-21T10:53:05.408890+0000) 2024-03-21T10:53:07.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:06 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:06.824+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:53:07.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:06 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:06.824+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:53:07.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:06 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:06.824+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:53:07.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:06 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:06.824+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:53:07.181 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:06 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:06.824+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:53:07.181 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:06 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:06.824+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:22.871708+0000 front 2024-03-21T10:52:22.871647+0000 (oldest deadline 2024-03-21T10:52:45.771600+0000) 2024-03-21T10:53:07.181 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:06 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:06.824+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.773577+0000 front 2024-03-21T10:52:32.773473+0000 (oldest deadline 2024-03-21T10:52:54.373581+0000) 2024-03-21T10:53:07.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:07 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:07.473+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:53:07.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:07 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:07.473+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:53:07.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:07 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:07.473+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:53:07.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:07 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:07.473+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:53:07.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:07 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:07.473+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:53:07.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:07 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:07.473+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.508171+0000 front 2024-03-21T10:52:32.508006+0000 (oldest deadline 2024-03-21T10:52:57.807721+0000) 2024-03-21T10:53:07.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:07 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:07.473+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-03-21T10:52:39.508873+0000 front 2024-03-21T10:52:39.509004+0000 (oldest deadline 2024-03-21T10:53:05.408890+0000) 2024-03-21T10:53:08.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:07 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:07.804+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:36.160812+0000 front 2024-03-21T10:51:36.160872+0000 (oldest deadline 2024-03-21T10:52:04.361314+0000) 2024-03-21T10:53:08.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:07 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:07.804+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:49.563011+0000 front 2024-03-21T10:51:49.563064+0000 (oldest deadline 2024-03-21T10:52:11.863115+0000) 2024-03-21T10:53:08.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:07 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:07.804+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:56.464768+0000 front 2024-03-21T10:51:56.464804+0000 (oldest deadline 2024-03-21T10:52:18.764995+0000) 2024-03-21T10:53:08.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:07 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:07.804+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-03-21T10:52:04.666344+0000 front 2024-03-21T10:52:04.666522+0000 (oldest deadline 2024-03-21T10:52:28.766206+0000) 2024-03-21T10:53:08.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:07 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:07.804+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:16.268513+0000 front 2024-03-21T10:52:16.268546+0000 (oldest deadline 2024-03-21T10:52:37.869183+0000) 2024-03-21T10:53:08.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:07 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:07.804+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:22.871708+0000 front 2024-03-21T10:52:22.871647+0000 (oldest deadline 2024-03-21T10:52:45.771600+0000) 2024-03-21T10:53:08.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:07 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10[61203]: 2024-03-21T10:53:07.804+0000 7f43e5beb700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.773577+0000 front 2024-03-21T10:52:32.773473+0000 (oldest deadline 2024-03-21T10:52:54.373581+0000) 2024-03-21T10:53:08.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:08 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:08.516+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:53:08.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:08 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:08.516+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:53:08.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:08 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:08.516+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:53:08.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:08 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:08.516+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:53:08.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:08 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:08.516+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:53:08.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:08 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:08.516+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.508171+0000 front 2024-03-21T10:52:32.508006+0000 (oldest deadline 2024-03-21T10:52:57.807721+0000) 2024-03-21T10:53:08.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:08 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:08.516+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-03-21T10:52:39.508873+0000 front 2024-03-21T10:52:39.509004+0000 (oldest deadline 2024-03-21T10:53:05.408890+0000) 2024-03-21T10:53:09.679 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:09 smithi040 podman[73788]: 2024-03-21 10:53:09.388359467 +0000 UTC m=+5.784056668 container died ef51dcba7b291f792230cd43233972e392a486ff295e62cea47e0332823bc3d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6) 2024-03-21T10:53:09.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:09 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:09.481+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:53:09.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:09 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:09.481+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:53:09.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:09 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:09.481+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:53:09.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:09 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:09.481+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:53:09.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:09 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:09.481+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:53:09.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:09 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:09.481+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.508171+0000 front 2024-03-21T10:52:32.508006+0000 (oldest deadline 2024-03-21T10:52:57.807721+0000) 2024-03-21T10:53:09.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:09 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:09.481+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-03-21T10:52:39.508873+0000 front 2024-03-21T10:52:39.509004+0000 (oldest deadline 2024-03-21T10:53:05.408890+0000) 2024-03-21T10:53:10.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:09 smithi040 podman[73788]: 2024-03-21 10:53:09.726385588 +0000 UTC m=+6.122082789 container cleanup ef51dcba7b291f792230cd43233972e392a486ff295e62cea47e0332823bc3d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-21T10:53:10.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:09 smithi040 bash[73788]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10 2024-03-21T10:53:10.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:09 smithi040 podman[73816]: 2024-03-21 10:53:09.917207426 +0000 UTC m=+0.526827167 container remove ef51dcba7b291f792230cd43233972e392a486ff295e62cea47e0332823bc3d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-21T10:53:10.594 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:10 smithi040 podman[73908]: 2024-03-21 10:53:10.251988233 +0000 UTC m=+0.017834807 image pull 237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232 2024-03-21T10:53:10.594 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:10 smithi040 podman[73908]: 2024-03-21 10:53:10.351514102 +0000 UTC m=+0.117360667 container create 46aa06e68d2e07c7b6b78bbdf9d8297db0ce17e97d3741a3955674dd428b5706 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6) 2024-03-21T10:53:10.594 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:10 smithi040 podman[73908]: 2024-03-21 10:53:10.595781813 +0000 UTC m=+0.361628392 container init 46aa06e68d2e07c7b6b78bbdf9d8297db0ce17e97d3741a3955674dd428b5706 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-21T10:53:10.595 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:10 smithi040 podman[73908]: 2024-03-21 10:53:10.599048065 +0000 UTC m=+0.364894637 container start 46aa06e68d2e07c7b6b78bbdf9d8297db0ce17e97d3741a3955674dd428b5706 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-21T10:53:10.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:10 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:10.494+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:53:10.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:10 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:10.494+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:53:10.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:10 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:10.494+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:53:10.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:10 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:10.494+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:53:10.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:10 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:10.494+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:53:10.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:10 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:10.494+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.508171+0000 front 2024-03-21T10:52:32.508006+0000 (oldest deadline 2024-03-21T10:52:57.807721+0000) 2024-03-21T10:53:10.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:10 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:10.494+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-03-21T10:52:39.508873+0000 front 2024-03-21T10:52:39.509004+0000 (oldest deadline 2024-03-21T10:53:05.408890+0000) 2024-03-21T10:53:10.892 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:10 smithi040 podman[73908]: 2024-03-21 10:53:10.712356307 +0000 UTC m=+0.478202885 container attach 46aa06e68d2e07c7b6b78bbdf9d8297db0ce17e97d3741a3955674dd428b5706 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-21T10:53:11.180 INFO:journalctl@ceph.osd.10.smithi040.stdout:Mar 21 10:53:10 smithi040 podman[73908]: 2024-03-21 10:53:10.897062237 +0000 UTC m=+0.662908822 container died 46aa06e68d2e07c7b6b78bbdf9d8297db0ce17e97d3741a3955674dd428b5706 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-10-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-21T10:53:11.468 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.10.service' 2024-03-21T10:53:11.516 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:53:11.516 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-21T10:53:11.516 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-21T10:53:11.517 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.11 2024-03-21T10:53:11.800 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:11 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:11.469+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:53:11.800 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:11 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:11.469+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:53:11.800 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:11 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:11.469+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:53:11.800 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:11 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:11.469+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:53:11.800 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:11 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:11.469+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:53:11.800 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:11 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:11.469+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.508171+0000 front 2024-03-21T10:52:32.508006+0000 (oldest deadline 2024-03-21T10:52:57.807721+0000) 2024-03-21T10:53:11.800 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:11 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:11.469+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-03-21T10:52:39.508873+0000 front 2024-03-21T10:52:39.509004+0000 (oldest deadline 2024-03-21T10:53:05.408890+0000) 2024-03-21T10:53:11.801 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:11 smithi070 systemd[1]: Stopping Ceph osd.11 for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:53:11.801 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:11 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:11.803+0000 7f1160b2e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:53:12.072 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:11 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:11.803+0000 7f1160b2e700 -1 osd.11 112 *** Got signal Terminated *** 2024-03-21T10:53:12.072 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:11 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:11.803+0000 7f1160b2e700 -1 osd.11 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T10:53:12.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:12 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:12.501+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:53:12.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:12 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:12.501+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:53:12.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:12 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:12.501+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:53:12.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:12 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:12.501+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:53:12.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:12 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:12.501+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:53:12.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:12 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:12.501+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.508171+0000 front 2024-03-21T10:52:32.508006+0000 (oldest deadline 2024-03-21T10:52:57.807721+0000) 2024-03-21T10:53:12.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:12 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:12.501+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-03-21T10:52:39.508873+0000 front 2024-03-21T10:52:39.509004+0000 (oldest deadline 2024-03-21T10:53:05.408890+0000) 2024-03-21T10:53:13.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:13 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:13.476+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:53:13.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:13 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:13.476+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:53:13.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:13 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:13.476+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:53:13.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:13 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:13.476+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:53:13.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:13 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:13.476+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:53:13.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:13 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:13.476+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.508171+0000 front 2024-03-21T10:52:32.508006+0000 (oldest deadline 2024-03-21T10:52:57.807721+0000) 2024-03-21T10:53:13.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:13 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:13.476+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-03-21T10:52:39.508873+0000 front 2024-03-21T10:52:39.509004+0000 (oldest deadline 2024-03-21T10:53:05.408890+0000) 2024-03-21T10:53:14.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:14 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:14.501+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:53:14.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:14 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:14.501+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:53:14.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:14 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:14.501+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:53:14.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:14 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:14.501+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:53:14.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:14 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:14.501+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:53:14.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:14 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:14.501+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.508171+0000 front 2024-03-21T10:52:32.508006+0000 (oldest deadline 2024-03-21T10:52:57.807721+0000) 2024-03-21T10:53:14.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:14 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:14.501+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-03-21T10:52:39.508873+0000 front 2024-03-21T10:52:39.509004+0000 (oldest deadline 2024-03-21T10:53:05.408890+0000) 2024-03-21T10:53:15.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:15 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:15.496+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:53:15.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:15 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:15.496+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:53:15.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:15 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:15.496+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:53:15.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:15 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:15.496+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:53:15.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:15 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:15.496+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:53:15.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:15 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:15.496+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.508171+0000 front 2024-03-21T10:52:32.508006+0000 (oldest deadline 2024-03-21T10:52:57.807721+0000) 2024-03-21T10:53:15.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:15 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:15.496+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-03-21T10:52:39.508873+0000 front 2024-03-21T10:52:39.509004+0000 (oldest deadline 2024-03-21T10:53:05.408890+0000) 2024-03-21T10:53:16.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:16 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:16.525+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-03-21T10:51:37.897742+0000 front 2024-03-21T10:51:37.897613+0000 (oldest deadline 2024-03-21T10:52:00.797590+0000) 2024-03-21T10:53:16.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:16 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:16.525+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-03-21T10:51:48.800674+0000 front 2024-03-21T10:51:48.800600+0000 (oldest deadline 2024-03-21T10:52:13.500607+0000) 2024-03-21T10:53:16.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:16 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:16.525+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6804 osd.2 since back 2024-03-21T10:51:57.502067+0000 front 2024-03-21T10:51:57.502086+0000 (oldest deadline 2024-03-21T10:52:22.802172+0000) 2024-03-21T10:53:16.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:16 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:16.525+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-03-21T10:52:13.205080+0000 front 2024-03-21T10:52:13.204962+0000 (oldest deadline 2024-03-21T10:52:37.905015+0000) 2024-03-21T10:53:16.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:16 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:16.525+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6812 osd.5 since back 2024-03-21T10:52:24.306759+0000 front 2024-03-21T10:52:24.306903+0000 (oldest deadline 2024-03-21T10:52:47.806881+0000) 2024-03-21T10:53:16.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:16 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:16.525+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-03-21T10:52:32.508171+0000 front 2024-03-21T10:52:32.508006+0000 (oldest deadline 2024-03-21T10:52:57.807721+0000) 2024-03-21T10:53:16.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:16 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:16.525+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-03-21T10:52:39.508873+0000 front 2024-03-21T10:52:39.509004+0000 (oldest deadline 2024-03-21T10:53:05.408890+0000) 2024-03-21T10:53:16.823 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:16 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11[62015]: 2024-03-21T10:53:16.525+0000 7f115911a700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.70:6820 osd.8 since back 2024-03-21T10:52:45.409438+0000 front 2024-03-21T10:52:45.409420+0000 (oldest deadline 2024-03-21T10:53:16.009913+0000) 2024-03-21T10:53:17.507 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:17 smithi070 podman[74559]: 2024-03-21 10:53:17.24869651 +0000 UTC m=+5.535964462 container died 2553e8cac7c5d81f7f502678b658fe61972a2632a25f1850bead9d994ebec360 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-21T10:53:17.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:17 smithi070 podman[74559]: 2024-03-21 10:53:17.610389606 +0000 UTC m=+5.897657573 container cleanup 2553e8cac7c5d81f7f502678b658fe61972a2632a25f1850bead9d994ebec360 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-21T10:53:17.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:17 smithi070 bash[74559]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11 2024-03-21T10:53:18.207 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:17 smithi070 podman[74576]: 2024-03-21 10:53:17.886494673 +0000 UTC m=+0.630693731 container remove 2553e8cac7c5d81f7f502678b658fe61972a2632a25f1850bead9d994ebec360 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-21T10:53:18.494 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:18 smithi070 podman[74666]: 2024-03-21 10:53:18.11466583 +0000 UTC m=+0.017160035 image pull 237e43c7a65ff48534d94b97c1bfdbe4b7be5672e3d9a613745083ea3702aa54 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232 2024-03-21T10:53:18.494 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:18 smithi070 podman[74666]: 2024-03-21 10:53:18.220522436 +0000 UTC m=+0.123016619 container create 7959396d3b809a52e89180b1d00d5f5645af891b9c508862be15ec8676bce937 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-21T10:53:18.494 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:18 smithi070 podman[74666]: 2024-03-21 10:53:18.43966835 +0000 UTC m=+0.342162535 container init 7959396d3b809a52e89180b1d00d5f5645af891b9c508862be15ec8676bce937 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-21T10:53:18.494 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:18 smithi070 podman[74666]: 2024-03-21 10:53:18.442749049 +0000 UTC m=+0.345243235 container start 7959396d3b809a52e89180b1d00d5f5645af891b9c508862be15ec8676bce937 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T10:53:18.494 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:18 smithi070 podman[74666]: 2024-03-21 10:53:18.497858918 +0000 UTC m=+0.400353130 container attach 7959396d3b809a52e89180b1d00d5f5645af891b9c508862be15ec8676bce937 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T10:53:18.822 INFO:journalctl@ceph.osd.11.smithi070.stdout:Mar 21 10:53:18 smithi070 podman[74666]: 2024-03-21 10:53:18.737621873 +0000 UTC m=+0.640116068 container died 7959396d3b809a52e89180b1d00d5f5645af891b9c508862be15ec8676bce937 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T10:53:19.341 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@osd.11.service' 2024-03-21T10:53:19.387 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:53:19.388 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-21T10:53:19.388 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-21T10:53:19.388 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.a 2024-03-21T10:53:19.803 INFO:journalctl@ceph.mds.a.smithi031.stdout:Mar 21 10:53:19 smithi031 systemd[1]: Stopping Ceph mds.a for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:53:20.303 INFO:journalctl@ceph.mds.a.smithi031.stdout:Mar 21 10:53:19 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-a[72739]: 2024-03-21T10:53:19.969+0000 7f1215200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:53:20.304 INFO:journalctl@ceph.mds.a.smithi031.stdout:Mar 21 10:53:19 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-a[72739]: 2024-03-21T10:53:19.969+0000 7f1215200700 -1 mds.a *** got signal Terminated *** 2024-03-21T10:53:22.489 INFO:journalctl@ceph.mds.a.smithi031.stdout:Mar 21 10:53:22 smithi031 podman[104736]: 2024-03-21 10:53:22.198488063 +0000 UTC m=+2.348322256 container died 0700393bf215ae26e97f128e64ea04d5dfeb35f6a5640c6b70227e6d9ca3cfa8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T10:53:22.803 INFO:journalctl@ceph.mds.a.smithi031.stdout:Mar 21 10:53:22 smithi031 podman[104736]: 2024-03-21 10:53:22.638025998 +0000 UTC m=+2.787860184 container cleanup 0700393bf215ae26e97f128e64ea04d5dfeb35f6a5640c6b70227e6d9ca3cfa8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-a, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.vendor=CentOS) 2024-03-21T10:53:22.804 INFO:journalctl@ceph.mds.a.smithi031.stdout:Mar 21 10:53:22 smithi031 bash[104736]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-a 2024-03-21T10:53:23.292 INFO:journalctl@ceph.mds.a.smithi031.stdout:Mar 21 10:53:22 smithi031 podman[104750]: 2024-03-21 10:53:22.814298474 +0000 UTC m=+0.554984090 container remove 0700393bf215ae26e97f128e64ea04d5dfeb35f6a5640c6b70227e6d9ca3cfa8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-a, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6) 2024-03-21T10:53:23.323 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.a.service' 2024-03-21T10:53:23.451 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:53:23.451 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-21T10:53:23.451 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-21T10:53:23.451 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.d 2024-03-21T10:53:24.053 INFO:journalctl@ceph.mds.d.smithi031.stdout:Mar 21 10:53:23 smithi031 systemd[1]: Stopping Ceph mds.d for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:53:24.553 INFO:journalctl@ceph.mds.d.smithi031.stdout:Mar 21 10:53:24 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-d[71752]: 2024-03-21T10:53:24.081+0000 7f87dc000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:53:24.553 INFO:journalctl@ceph.mds.d.smithi031.stdout:Mar 21 10:53:24 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-d[71752]: 2024-03-21T10:53:24.081+0000 7f87dc000700 -1 mds.d *** got signal Terminated *** 2024-03-21T10:53:27.536 INFO:journalctl@ceph.mds.d.smithi031.stdout:Mar 21 10:53:27 smithi031 podman[104856]: 2024-03-21 10:53:27.218432616 +0000 UTC m=+3.273301479 container died 283c28155b7b50696faf1f76bd1e40b8c400850ae75187485d33059a95ddb84f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-d, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6) 2024-03-21T10:53:27.803 INFO:journalctl@ceph.mds.d.smithi031.stdout:Mar 21 10:53:27 smithi031 podman[104856]: 2024-03-21 10:53:27.641432909 +0000 UTC m=+3.696301768 container cleanup 283c28155b7b50696faf1f76bd1e40b8c400850ae75187485d33059a95ddb84f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-d, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-21T10:53:27.803 INFO:journalctl@ceph.mds.d.smithi031.stdout:Mar 21 10:53:27 smithi031 bash[104856]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-d 2024-03-21T10:53:28.303 INFO:journalctl@ceph.mds.d.smithi031.stdout:Mar 21 10:53:27 smithi031 podman[104869]: 2024-03-21 10:53:27.909198495 +0000 UTC m=+0.672815600 container remove 283c28155b7b50696faf1f76bd1e40b8c400850ae75187485d33059a95ddb84f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-d, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-21T10:53:28.408 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.d.service' 2024-03-21T10:53:28.533 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:53:28.533 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-21T10:53:28.533 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-21T10:53:28.534 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.g 2024-03-21T10:53:29.053 INFO:journalctl@ceph.mds.g.smithi031.stdout:Mar 21 10:53:28 smithi031 systemd[1]: Stopping Ceph mds.g for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:53:29.554 INFO:journalctl@ceph.mds.g.smithi031.stdout:Mar 21 10:53:29 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-g[72344]: 2024-03-21T10:53:29.101+0000 7feee0e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:53:29.554 INFO:journalctl@ceph.mds.g.smithi031.stdout:Mar 21 10:53:29 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-g[72344]: 2024-03-21T10:53:29.101+0000 7feee0e00700 -1 mds.g *** got signal Terminated *** 2024-03-21T10:53:30.447 INFO:journalctl@ceph.mds.g.smithi031.stdout:Mar 21 10:53:30 smithi031 podman[104975]: 2024-03-21 10:53:30.156269728 +0000 UTC m=+1.197432718 container died cf9cdadc0cb09bcee953696058461416392348d9c278831c3d0d2923976241b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-g, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, maintainer=Guillaume Abrioux ) 2024-03-21T10:53:30.803 INFO:journalctl@ceph.mds.g.smithi031.stdout:Mar 21 10:53:30 smithi031 podman[104975]: 2024-03-21 10:53:30.603430303 +0000 UTC m=+1.644593286 container cleanup cf9cdadc0cb09bcee953696058461416392348d9c278831c3d0d2923976241b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6) 2024-03-21T10:53:30.804 INFO:journalctl@ceph.mds.g.smithi031.stdout:Mar 21 10:53:30 smithi031 bash[104975]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-g 2024-03-21T10:53:31.303 INFO:journalctl@ceph.mds.g.smithi031.stdout:Mar 21 10:53:30 smithi031 podman[104989]: 2024-03-21 10:53:30.929731314 +0000 UTC m=+0.750606551 container remove cf9cdadc0cb09bcee953696058461416392348d9c278831c3d0d2923976241b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-g, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-21T10:53:31.487 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.g.service' 2024-03-21T10:53:31.612 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:53:31.612 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-21T10:53:31.613 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-21T10:53:31.613 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.j 2024-03-21T10:53:32.053 INFO:journalctl@ceph.mds.j.smithi031.stdout:Mar 21 10:53:31 smithi031 systemd[1]: Stopping Ceph mds.j for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:53:32.553 INFO:journalctl@ceph.mds.j.smithi031.stdout:Mar 21 10:53:32 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-j[71373]: 2024-03-21T10:53:32.213+0000 7f7cefe00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:53:32.553 INFO:journalctl@ceph.mds.j.smithi031.stdout:Mar 21 10:53:32 smithi031 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-j[71373]: 2024-03-21T10:53:32.213+0000 7f7cefe00700 -1 mds.j *** got signal Terminated *** 2024-03-21T10:53:35.176 INFO:journalctl@ceph.mds.j.smithi031.stdout:Mar 21 10:53:34 smithi031 podman[105094]: 2024-03-21 10:53:34.893708168 +0000 UTC m=+2.801210764 container died 64416c35f4077144393ca8cb51fe82295bbfa5b2c1ae0cd73521021fb35a3172 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-21T10:53:35.512 INFO:journalctl@ceph.mds.j.smithi031.stdout:Mar 21 10:53:35 smithi031 podman[105094]: 2024-03-21 10:53:35.281955246 +0000 UTC m=+3.189457782 container cleanup 64416c35f4077144393ca8cb51fe82295bbfa5b2c1ae0cd73521021fb35a3172 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-j, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True) 2024-03-21T10:53:35.512 INFO:journalctl@ceph.mds.j.smithi031.stdout:Mar 21 10:53:35 smithi031 bash[105094]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-j 2024-03-21T10:53:35.803 INFO:journalctl@ceph.mds.j.smithi031.stdout:Mar 21 10:53:35 smithi031 podman[105108]: 2024-03-21 10:53:35.516169192 +0000 UTC m=+0.610161840 container remove 64416c35f4077144393ca8cb51fe82295bbfa5b2c1ae0cd73521021fb35a3172 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T10:53:36.004 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.j.service' 2024-03-21T10:53:36.123 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:53:36.124 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-21T10:53:36.124 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-21T10:53:36.124 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.b 2024-03-21T10:53:36.429 INFO:journalctl@ceph.mds.b.smithi040.stdout:Mar 21 10:53:36 smithi040 systemd[1]: Stopping Ceph mds.b for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:53:36.704 INFO:journalctl@ceph.mds.b.smithi040.stdout:Mar 21 10:53:36 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-b[66460]: 2024-03-21T10:53:36.441+0000 7f61162ee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:53:36.704 INFO:journalctl@ceph.mds.b.smithi040.stdout:Mar 21 10:53:36 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-b[66460]: 2024-03-21T10:53:36.441+0000 7f61162ee700 -1 mds.b *** got signal Terminated *** 2024-03-21T10:53:38.330 INFO:journalctl@ceph.mds.b.smithi040.stdout:Mar 21 10:53:38 smithi040 podman[74013]: 2024-03-21 10:53:38.071440848 +0000 UTC m=+1.737514663 container died 68b79ae43eaf7ebd472efd6e9c885aed2da03fc7dce370b53c0b0b21bf254912 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-21T10:53:38.680 INFO:journalctl@ceph.mds.b.smithi040.stdout:Mar 21 10:53:38 smithi040 podman[74013]: 2024-03-21 10:53:38.417410824 +0000 UTC m=+2.083484640 container cleanup 68b79ae43eaf7ebd472efd6e9c885aed2da03fc7dce370b53c0b0b21bf254912 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0) 2024-03-21T10:53:38.680 INFO:journalctl@ceph.mds.b.smithi040.stdout:Mar 21 10:53:38 smithi040 bash[74013]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-b 2024-03-21T10:53:38.841 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.b.service' 2024-03-21T10:53:38.883 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:53:38.884 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-21T10:53:38.884 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-21T10:53:38.884 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.e 2024-03-21T10:53:39.317 INFO:journalctl@ceph.mds.e.smithi040.stdout:Mar 21 10:53:38 smithi040 systemd[1]: Stopping Ceph mds.e for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:53:39.679 INFO:journalctl@ceph.mds.e.smithi040.stdout:Mar 21 10:53:39 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-e[65830]: 2024-03-21T10:53:39.320+0000 7f27f27d3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:53:39.680 INFO:journalctl@ceph.mds.e.smithi040.stdout:Mar 21 10:53:39 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-e[65830]: 2024-03-21T10:53:39.320+0000 7f27f27d3700 -1 mds.e *** got signal Terminated *** 2024-03-21T10:53:42.376 INFO:journalctl@ceph.mds.e.smithi040.stdout:Mar 21 10:53:42 smithi040 podman[74150]: 2024-03-21 10:53:42.123209477 +0000 UTC m=+3.013817061 container died 4a091e9b018ebe5a412245e78c446eb19b2cbded9bef23e677e11b519645ac74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-e, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-21T10:53:42.679 INFO:journalctl@ceph.mds.e.smithi040.stdout:Mar 21 10:53:42 smithi040 podman[74150]: 2024-03-21 10:53:42.47987809 +0000 UTC m=+3.370485662 container cleanup 4a091e9b018ebe5a412245e78c446eb19b2cbded9bef23e677e11b519645ac74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-e, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T10:53:42.680 INFO:journalctl@ceph.mds.e.smithi040.stdout:Mar 21 10:53:42 smithi040 bash[74150]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-e 2024-03-21T10:53:42.922 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.e.service' 2024-03-21T10:53:42.962 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:53:42.962 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-21T10:53:42.962 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-21T10:53:42.962 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.h 2024-03-21T10:53:43.279 INFO:journalctl@ceph.mds.h.smithi040.stdout:Mar 21 10:53:43 smithi040 systemd[1]: Stopping Ceph mds.h for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:53:43.680 INFO:journalctl@ceph.mds.h.smithi040.stdout:Mar 21 10:53:43 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-h[66136]: 2024-03-21T10:53:43.283+0000 7f600693e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:53:43.680 INFO:journalctl@ceph.mds.h.smithi040.stdout:Mar 21 10:53:43 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-h[66136]: 2024-03-21T10:53:43.283+0000 7f600693e700 -1 mds.h *** got signal Terminated *** 2024-03-21T10:53:46.356 INFO:journalctl@ceph.mds.h.smithi040.stdout:Mar 21 10:53:46 smithi040 podman[74274]: 2024-03-21 10:53:46.066427543 +0000 UTC m=+2.877218734 container died 101a42563d4bad5bc0ab4fbc3519b5f8d062e4b74bcbdc7253d8ee39afc0da1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-h, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-21T10:53:46.664 INFO:journalctl@ceph.mds.h.smithi040.stdout:Mar 21 10:53:46 smithi040 podman[74274]: 2024-03-21 10:53:46.467976614 +0000 UTC m=+3.278767793 container cleanup 101a42563d4bad5bc0ab4fbc3519b5f8d062e4b74bcbdc7253d8ee39afc0da1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True) 2024-03-21T10:53:46.664 INFO:journalctl@ceph.mds.h.smithi040.stdout:Mar 21 10:53:46 smithi040 bash[74274]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-h 2024-03-21T10:53:46.765 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.h.service' 2024-03-21T10:53:46.804 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:53:46.804 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-21T10:53:46.805 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-21T10:53:46.805 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.k 2024-03-21T10:53:46.929 INFO:journalctl@ceph.mds.k.smithi040.stdout:Mar 21 10:53:46 smithi040 systemd[1]: Stopping Ceph mds.k for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:53:47.429 INFO:journalctl@ceph.mds.k.smithi040.stdout:Mar 21 10:53:47 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-k[65509]: 2024-03-21T10:53:47.121+0000 7fb47e964700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:53:47.430 INFO:journalctl@ceph.mds.k.smithi040.stdout:Mar 21 10:53:47 smithi040 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-k[65509]: 2024-03-21T10:53:47.121+0000 7fb47e964700 -1 mds.k *** got signal Terminated *** 2024-03-21T10:53:50.800 INFO:journalctl@ceph.mds.k.smithi040.stdout:Mar 21 10:53:50 smithi040 podman[74395]: 2024-03-21 10:53:50.496711835 +0000 UTC m=+3.467023809 container died 2858ae883013fbfe5841c3e00e41102dd97ec8068d2b02cc6ebb79b2e49f96db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-21T10:53:51.143 INFO:journalctl@ceph.mds.k.smithi040.stdout:Mar 21 10:53:50 smithi040 podman[74395]: 2024-03-21 10:53:50.80463807 +0000 UTC m=+3.774950027 container cleanup 2858ae883013fbfe5841c3e00e41102dd97ec8068d2b02cc6ebb79b2e49f96db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-k, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T10:53:51.143 INFO:journalctl@ceph.mds.k.smithi040.stdout:Mar 21 10:53:50 smithi040 bash[74395]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-k 2024-03-21T10:53:51.296 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.k.service' 2024-03-21T10:53:51.335 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:53:51.335 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-21T10:53:51.335 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-21T10:53:51.335 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.c 2024-03-21T10:53:51.635 INFO:journalctl@ceph.mds.c.smithi070.stdout:Mar 21 10:53:51 smithi070 systemd[1]: Stopping Ceph mds.c for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:53:51.899 INFO:journalctl@ceph.mds.c.smithi070.stdout:Mar 21 10:53:51 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-c[67043]: 2024-03-21T10:53:51.639+0000 7f185da41700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:53:51.899 INFO:journalctl@ceph.mds.c.smithi070.stdout:Mar 21 10:53:51 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-c[67043]: 2024-03-21T10:53:51.639+0000 7f185da41700 -1 mds.c *** got signal Terminated *** 2024-03-21T10:53:54.322 INFO:journalctl@ceph.mds.c.smithi070.stdout:Mar 21 10:53:54 smithi070 podman[74770]: 2024-03-21 10:53:54.066249679 +0000 UTC m=+2.535627876 container died 5b129a97f390ccc86b4010dc69cb0a2af7be775fdd4e579bb54d2124a45b12be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-21T10:53:54.705 INFO:journalctl@ceph.mds.c.smithi070.stdout:Mar 21 10:53:54 smithi070 podman[74770]: 2024-03-21 10:53:54.424868893 +0000 UTC m=+2.894247084 container cleanup 5b129a97f390ccc86b4010dc69cb0a2af7be775fdd4e579bb54d2124a45b12be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-c, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-21T10:53:54.705 INFO:journalctl@ceph.mds.c.smithi070.stdout:Mar 21 10:53:54 smithi070 bash[74770]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-c 2024-03-21T10:53:54.853 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.c.service' 2024-03-21T10:53:54.891 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:53:54.892 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-21T10:53:54.892 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-21T10:53:54.892 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.f 2024-03-21T10:53:55.322 INFO:journalctl@ceph.mds.f.smithi070.stdout:Mar 21 10:53:54 smithi070 systemd[1]: Stopping Ceph mds.f for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:53:55.322 INFO:journalctl@ceph.mds.f.smithi070.stdout:Mar 21 10:53:55 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-f[66734]: 2024-03-21T10:53:55.203+0000 7f8b1dfaf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:53:55.322 INFO:journalctl@ceph.mds.f.smithi070.stdout:Mar 21 10:53:55 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-f[66734]: 2024-03-21T10:53:55.203+0000 7f8b1dfaf700 -1 mds.f *** got signal Terminated *** 2024-03-21T10:53:58.322 INFO:journalctl@ceph.mds.f.smithi070.stdout:Mar 21 10:53:58 smithi070 podman[74904]: 2024-03-21 10:53:58.073146292 +0000 UTC m=+2.956004499 container died 33d6518933b04d0140568e257e4cca6c1a51418966afb818e6655c4986e02b4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-f, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T10:53:58.683 INFO:journalctl@ceph.mds.f.smithi070.stdout:Mar 21 10:53:58 smithi070 podman[74904]: 2024-03-21 10:53:58.42826491 +0000 UTC m=+3.311123166 container cleanup 33d6518933b04d0140568e257e4cca6c1a51418966afb818e6655c4986e02b4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-f, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.vendor=CentOS, ceph=True) 2024-03-21T10:53:58.683 INFO:journalctl@ceph.mds.f.smithi070.stdout:Mar 21 10:53:58 smithi070 bash[74904]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-f 2024-03-21T10:53:58.835 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.f.service' 2024-03-21T10:53:58.870 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:53:58.870 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-21T10:53:58.870 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-21T10:53:58.871 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.i 2024-03-21T10:53:59.186 INFO:journalctl@ceph.mds.i.smithi070.stdout:Mar 21 10:53:58 smithi070 systemd[1]: Stopping Ceph mds.i for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:53:59.572 INFO:journalctl@ceph.mds.i.smithi070.stdout:Mar 21 10:53:59 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-i[67353]: 2024-03-21T10:53:59.189+0000 7fcdd0d65700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:53:59.572 INFO:journalctl@ceph.mds.i.smithi070.stdout:Mar 21 10:53:59 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-i[67353]: 2024-03-21T10:53:59.189+0000 7fcdd0d65700 -1 mds.i *** got signal Terminated *** 2024-03-21T10:54:02.319 INFO:journalctl@ceph.mds.i.smithi070.stdout:Mar 21 10:54:02 smithi070 podman[75025]: 2024-03-21 10:54:02.067015735 +0000 UTC m=+2.981564904 container died e44ffbadd2e579bbcb9f8ff915329f86170c2d03232233ac461891ecb6c68c48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-i, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-21T10:54:02.572 INFO:journalctl@ceph.mds.i.smithi070.stdout:Mar 21 10:54:02 smithi070 podman[75025]: 2024-03-21 10:54:02.406420415 +0000 UTC m=+3.320969583 container cleanup e44ffbadd2e579bbcb9f8ff915329f86170c2d03232233ac461891ecb6c68c48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-i, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T10:54:02.572 INFO:journalctl@ceph.mds.i.smithi070.stdout:Mar 21 10:54:02 smithi070 bash[75025]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-i 2024-03-21T10:54:02.819 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.i.service' 2024-03-21T10:54:02.855 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:54:02.855 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-21T10:54:02.855 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-21T10:54:02.855 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.l 2024-03-21T10:54:03.181 INFO:journalctl@ceph.mds.l.smithi070.stdout:Mar 21 10:54:02 smithi070 systemd[1]: Stopping Ceph mds.l for 3847408c-e76d-11ee-95cd-87774f69a715... 2024-03-21T10:54:03.572 INFO:journalctl@ceph.mds.l.smithi070.stdout:Mar 21 10:54:03 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-l[66415]: 2024-03-21T10:54:03.184+0000 7f5c3bb78700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T10:54:03.572 INFO:journalctl@ceph.mds.l.smithi070.stdout:Mar 21 10:54:03 smithi070 ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-l[66415]: 2024-03-21T10:54:03.184+0000 7f5c3bb78700 -1 mds.l *** got signal Terminated *** 2024-03-21T10:54:07.448 INFO:journalctl@ceph.mds.l.smithi070.stdout:Mar 21 10:54:07 smithi070 podman[75147]: 2024-03-21 10:54:07.192199407 +0000 UTC m=+4.128665810 container died 0f729b3e58ba0e4aaa0e36ea461dedc41a5215d250eb1d786f074ef07801da03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-l, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T10:54:07.822 INFO:journalctl@ceph.mds.l.smithi070.stdout:Mar 21 10:54:07 smithi070 podman[75147]: 2024-03-21 10:54:07.551567383 +0000 UTC m=+4.488033829 container cleanup 0f729b3e58ba0e4aaa0e36ea461dedc41a5215d250eb1d786f074ef07801da03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:87c941b756786d43881503fb78c47c26d07136cca67af598d52ba7d6d9d06232, name=ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-l, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240320.053002-2edfee6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-21T10:54:07.822 INFO:journalctl@ceph.mds.l.smithi070.stdout:Mar 21 10:54:07 smithi070 bash[75147]: ceph-3847408c-e76d-11ee-95cd-87774f69a715-mds-l 2024-03-21T10:54:07.957 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3847408c-e76d-11ee-95cd-87774f69a715@mds.l.service' 2024-03-21T10:54:07.990 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T10:54:07.990 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-21T10:54:07.991 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3847408c-e76d-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-21T10:54:08.327 INFO:teuthology.orchestra.run.smithi031.stdout:Deleting cluster with fsid: 3847408c-e76d-11ee-95cd-87774f69a715 2024-03-21T10:54:13.696 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3847408c-e76d-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-21T10:54:13.898 INFO:teuthology.orchestra.run.smithi040.stdout:Deleting cluster with fsid: 3847408c-e76d-11ee-95cd-87774f69a715 2024-03-21T10:54:16.556 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3847408c-e76d-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-21T10:54:16.745 INFO:teuthology.orchestra.run.smithi070.stdout:Deleting cluster with fsid: 3847408c-e76d-11ee-95cd-87774f69a715 2024-03-21T10:54:19.199 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-21T10:54:19.311 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-21T10:54:19.349 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-21T10:54:19.378 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-21T10:54:19.381 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612235/remote/smithi031/crash 2024-03-21T10:54:19.382 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/crash -- . 2024-03-21T10:54:19.482 INFO:teuthology.orchestra.run.smithi031.stderr:tar: /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-21T10:54:19.483 INFO:teuthology.orchestra.run.smithi031.stderr:tar: Error is not recoverable: exiting now 2024-03-21T10:54:19.488 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612235/remote/smithi040/crash 2024-03-21T10:54:19.489 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/crash -- . 2024-03-21T10:54:19.539 INFO:teuthology.orchestra.run.smithi040.stderr:tar: /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-21T10:54:19.540 INFO:teuthology.orchestra.run.smithi040.stderr:tar: Error is not recoverable: exiting now 2024-03-21T10:54:19.541 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612235/remote/smithi070/crash 2024-03-21T10:54:19.542 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/crash -- . 2024-03-21T10:54:19.635 INFO:teuthology.orchestra.run.smithi070.stderr:tar: /var/lib/ceph/3847408c-e76d-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-21T10:54:19.635 INFO:teuthology.orchestra.run.smithi070.stderr:tar: Error is not recoverable: exiting now 2024-03-21T10:54:19.636 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-21T10:54:19.637 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-21T10:54:19.810 INFO:teuthology.orchestra.run.smithi031.stdout:2024-03-21T10:28:06.339620+0000 mon.a (mon.0) 187 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-21T10:54:19.812 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-21T10:54:19.812 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[SEC\]' /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-21T10:54:19.919 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]' /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-21T10:54:20.018 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[WRN\]' /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-21T10:54:20.113 INFO:teuthology.orchestra.run.smithi031.stdout:2024-03-21T10:28:06.339620+0000 mon.a (mon.0) 187 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-21T10:54:20.114 INFO:tasks.cephadm:Compressing logs... 2024-03-21T10:54:20.115 DEBUG:teuthology.orchestra.run.smithi031:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-21T10:54:20.119 DEBUG:teuthology.orchestra.run.smithi040:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-21T10:54:20.121 DEBUG:teuthology.orchestra.run.smithi070:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-21T10:54:20.151 INFO:teuthology.orchestra.run.smithi070.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log: No such file or directory 2024-03-21T10:54:20.151 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-21T10:54:20.151 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.l.log 2024-03-21T10:54:20.152 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-21T10:54:20.152 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.i.log 2024-03-21T10:54:20.152 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.11.log 2024-03-21T10:54:20.153 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.audit.log: /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mgr.z.log 2024-03-21T10:54:20.153 INFO:teuthology.orchestra.run.smithi070.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-21T10:54:20.153 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.8.log 2024-03-21T10:54:20.153 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.5.log 2024-03-21T10:54:20.154 INFO:teuthology.orchestra.run.smithi040.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose --: No such file or directory /var/log/ceph/cephadm.log 2024-03-21T10:54:20.155 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-03-21T10:54:20.155 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.10.log 2024-03-21T10:54:20.157 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-21T10:54:20.157 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.b.log 2024-03-21T10:54:20.157 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.4.log 2024-03-21T10:54:20.157 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-21T10:54:20.157 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.b.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-21T10:54:20.157 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.k.log 2024-03-21T10:54:20.158 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.h.log 2024-03-21T10:54:20.158 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.7.log 2024-03-21T10:54:20.159 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.e.log 2024-03-21T10:54:20.162 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.f.log 2024-03-21T10:54:20.162 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-21T10:54:20.162 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.f.log: 90.2% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-21T10:54:20.162 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.log 2024-03-21T10:54:20.162 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.cephadm.log: 83.4% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-21T10:54:20.162 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-21T10:54:20.162 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.log: 94.6% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.l.log.gz 2024-03-21T10:54:20.162 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.c.log 2024-03-21T10:54:20.162 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.2.log: 87.1% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-21T10:54:20.162 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-21T10:54:20.165 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.c.log: 92.1% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mgr.z.log.gz 2024-03-21T10:54:20.165 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mon.c.log 2024-03-21T10:54:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.7.log: 90.2% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-21T10:54:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mgr.y.log 2024-03-21T10:54:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.e.log: 94.5% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.k.log.gz 2024-03-21T10:54:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-21T10:54:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mgr.y.log: 94.5% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.e.log.gz 2024-03-21T10:54:20.168 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.log 2024-03-21T10:54:20.168 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.cephadm.log: 83.5% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-21T10:54:20.168 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-21T10:54:20.172 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.log: 87.1% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-21T10:54:20.172 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mon.b.log 2024-03-21T10:54:20.174 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-volume.log: 92.1% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mgr.y.log.gz 2024-03-21T10:54:20.177 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mon.c.log: 91.0% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-21T10:54:20.187 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mon.b.log: 90.9% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-21T10:54:20.195 INFO:teuthology.orchestra.run.smithi031.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-21T10:54:20.195 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-21T10:54:20.207 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.6.log 2024-03-21T10:54:20.208 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mon.a.log 2024-03-21T10:54:20.209 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-21T10:54:20.210 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.g.log 2024-03-21T10:54:20.212 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.j.log 2024-03-21T10:54:20.213 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.3.log 2024-03-21T10:54:20.214 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mgr.x.log 2024-03-21T10:54:20.215 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-21T10:54:20.217 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-21T10:54:20.217 INFO:teuthology.orchestra.run.smithi031.stderr: 94.4% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.g.log.gz 2024-03-21T10:54:20.218 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.log 2024-03-21T10:54:20.219 INFO:teuthology.orchestra.run.smithi031.stderr: 94.4%/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.cephadm.log: -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.j.log.gz 2024-03-21T10:54:20.219 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-21T10:54:20.220 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.9.log 2024-03-21T10:54:20.221 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.d.log 2024-03-21T10:54:20.223 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.9.log: 87.1%gzip -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.log.gz -5 2024-03-21T10:54:20.223 INFO:teuthology.orchestra.run.smithi031.stderr: --verbose -- /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.a.log 2024-03-21T10:54:20.223 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-21T10:54:20.226 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-21T10:54:20.228 INFO:teuthology.orchestra.run.smithi031.stderr: 94.2% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.d.log.gz 2024-03-21T10:54:20.242 INFO:teuthology.orchestra.run.smithi070.stderr: 94.3% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.c.log.gz 2024-03-21T10:54:20.246 INFO:teuthology.orchestra.run.smithi070.stderr: 94.1% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.f.log.gz 2024-03-21T10:54:20.273 INFO:teuthology.orchestra.run.smithi031.stderr: 90.1% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-21T10:54:20.304 INFO:teuthology.orchestra.run.smithi031.stderr: 84.0% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-21T10:54:20.338 INFO:teuthology.orchestra.run.smithi031.stderr: 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-21T10:54:20.384 INFO:teuthology.orchestra.run.smithi040.stderr: 93.7% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.h.log.gz 2024-03-21T10:54:20.386 INFO:teuthology.orchestra.run.smithi070.stderr: 92.7% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.i.log.gz 2024-03-21T10:54:20.461 INFO:teuthology.orchestra.run.smithi031.stderr: 90.8% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-21T10:54:20.590 INFO:teuthology.orchestra.run.smithi031.stderr: 92.0% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.a.log.gz 2024-03-21T10:54:20.665 INFO:teuthology.orchestra.run.smithi070.stderr: 91.6% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mon.c.log.gz 2024-03-21T10:54:20.832 INFO:teuthology.orchestra.run.smithi040.stderr: 91.6% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mon.b.log.gz 2024-03-21T10:54:21.288 INFO:teuthology.orchestra.run.smithi031.stderr: 90.0% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mgr.x.log.gz 2024-03-21T10:54:21.374 INFO:teuthology.orchestra.run.smithi040.stderr: 93.2% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mds.b.log.gz 2024-03-21T10:54:24.073 INFO:teuthology.orchestra.run.smithi031.stderr: 90.4% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-mon.a.log.gz 2024-03-21T10:54:51.043 INFO:teuthology.orchestra.run.smithi040.stderr: 92.3% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.10.log.gz 2024-03-21T10:54:54.430 INFO:teuthology.orchestra.run.smithi070.stderr: 92.3% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.8.log.gz 2024-03-21T10:54:55.003 INFO:teuthology.orchestra.run.smithi070.stderr: 92.3% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.11.log.gz 2024-03-21T10:54:57.752 INFO:teuthology.orchestra.run.smithi040.stderr: 92.3% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-21T10:55:02.501 INFO:teuthology.orchestra.run.smithi070.stderr: 92.2% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-21T10:55:08.497 INFO:teuthology.orchestra.run.smithi040.stderr: 92.3% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.7.log.gz 2024-03-21T10:55:13.854 INFO:teuthology.orchestra.run.smithi040.stderr: 92.2% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.4.log.gz 2024-03-21T10:55:13.857 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-03-21T10:55:13.857 INFO:teuthology.orchestra.run.smithi040.stderr:real 0m53.720s 2024-03-21T10:55:13.857 INFO:teuthology.orchestra.run.smithi040.stderr:user 2m47.239s 2024-03-21T10:55:13.857 INFO:teuthology.orchestra.run.smithi040.stderr:sys 0m5.027s 2024-03-21T10:55:19.570 INFO:teuthology.orchestra.run.smithi070.stderr: 92.2% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.5.log.gz 2024-03-21T10:55:19.572 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-21T10:55:19.572 INFO:teuthology.orchestra.run.smithi070.stderr:real 0m59.435s 2024-03-21T10:55:19.572 INFO:teuthology.orchestra.run.smithi070.stderr:user 2m46.125s 2024-03-21T10:55:19.572 INFO:teuthology.orchestra.run.smithi070.stderr:sys 0m5.380s 2024-03-21T10:56:32.089 INFO:teuthology.orchestra.run.smithi031.stderr: 92.3% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-21T10:56:46.500 INFO:teuthology.orchestra.run.smithi031.stderr: 92.2% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.3.log.gz 2024-03-21T10:56:47.162 INFO:teuthology.orchestra.run.smithi031.stderr: 92.2% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.6.log.gz 2024-03-21T10:57:01.831 INFO:teuthology.orchestra.run.smithi031.stderr: 92.2% -- replaced with /var/log/ceph/3847408c-e76d-11ee-95cd-87774f69a715/ceph-osd.9.log.gz 2024-03-21T10:57:01.836 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-03-21T10:57:01.836 INFO:teuthology.orchestra.run.smithi031.stderr:real 2m41.686s 2024-03-21T10:57:01.836 INFO:teuthology.orchestra.run.smithi031.stderr:user 5m7.487s 2024-03-21T10:57:01.836 INFO:teuthology.orchestra.run.smithi031.stderr:sys 0m32.729s 2024-03-21T10:57:01.837 INFO:tasks.cephadm:Archiving logs... 2024-03-21T10:57:01.838 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612235/remote/smithi031/log 2024-03-21T10:57:01.839 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-21T10:57:35.708 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612235/remote/smithi040/log 2024-03-21T10:57:35.709 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-21T10:58:03.414 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612235/remote/smithi070/log 2024-03-21T10:58:03.415 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-21T10:58:32.497 INFO:tasks.cephadm:Removing cluster... 2024-03-21T10:58:32.497 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3847408c-e76d-11ee-95cd-87774f69a715 --force 2024-03-21T10:58:32.831 INFO:teuthology.orchestra.run.smithi031.stdout:Deleting cluster with fsid: 3847408c-e76d-11ee-95cd-87774f69a715 2024-03-21T10:58:34.140 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3847408c-e76d-11ee-95cd-87774f69a715 --force 2024-03-21T10:58:34.343 INFO:teuthology.orchestra.run.smithi040.stdout:Deleting cluster with fsid: 3847408c-e76d-11ee-95cd-87774f69a715 2024-03-21T10:58:34.853 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3847408c-e76d-11ee-95cd-87774f69a715 --force 2024-03-21T10:58:35.047 INFO:teuthology.orchestra.run.smithi070.stdout:Deleting cluster with fsid: 3847408c-e76d-11ee-95cd-87774f69a715 2024-03-21T10:58:35.588 INFO:tasks.cephadm:Removing cephadm ... 2024-03-21T10:58:35.588 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-21T10:58:35.642 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-21T10:58:35.661 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-21T10:58:35.676 INFO:tasks.cephadm:Teardown complete 2024-03-21T10:58:35.676 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-21T10:58:35.688 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-21T10:58:35.688 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-21T10:58:35.694 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-21T10:58:35.704 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-21T10:58:35.854 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-21T10:58:35.866 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-21T10:58:35.866 DEBUG:teuthology.orchestra.run.smithi031:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T10:58:35.874 DEBUG:teuthology.orchestra.run.smithi040:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T10:58:35.876 DEBUG:teuthology.orchestra.run.smithi070:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T10:58:35.890 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: ntpq: command not found 2024-03-21T10:58:35.890 INFO:teuthology.orchestra.run.smithi070.stderr:bash: line 1: ntpq: command not found 2024-03-21T10:58:35.896 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-21T10:58:35.896 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2024-03-21T10:58:35.896 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 42 +706us[ +728us] +/- 88ms 2024-03-21T10:58:35.896 INFO:teuthology.orchestra.run.smithi070.stdout:^* hv02.front.sepia.ceph.com 3 6 377 42 -973us[ -950us] +/- 63ms 2024-03-21T10:58:35.896 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 +642us[ +642us] +/- 78ms 2024-03-21T10:58:35.897 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T10:58:35.897 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-21T10:58:35.897 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2024-03-21T10:58:35.897 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 42 +702us[ +713us] +/- 88ms 2024-03-21T10:58:35.897 INFO:teuthology.orchestra.run.smithi040.stdout:^* hv02.front.sepia.ceph.com 3 6 377 40 -982us[ -971us] +/- 63ms 2024-03-21T10:58:35.898 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 42 +649us[ +660us] +/- 78ms 2024-03-21T10:58:35.898 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T10:58:35.923 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2024-03-21T10:58:35.933 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-21T10:58:35.933 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-03-21T10:58:35.933 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 40 +687us[ +687us] +/- 88ms 2024-03-21T10:58:35.933 INFO:teuthology.orchestra.run.smithi031.stdout:^* hv02.front.sepia.ceph.com 3 6 377 41 -979us[ -960us] +/- 63ms 2024-03-21T10:58:35.933 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 40 +651us[ +651us] +/- 78ms 2024-03-21T10:58:35.933 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T10:58:35.935 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-21T10:58:35.947 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-21T10:58:35.947 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-21T10:58:35.958 DEBUG:teuthology.orchestra.run.smithi031:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-21T10:58:36.255 DEBUG:teuthology.orchestra.run.smithi040:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-21T10:58:36.343 DEBUG:teuthology.orchestra.run.smithi070:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-21T10:58:36.422 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-21T10:58:36.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T10:58:36.535 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-21T10:58:36.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T10:58:36.572 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-21T10:58:36.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T10:58:36.606 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-21T10:58:36.618 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-21T10:58:36.628 INFO:teuthology.task.internal:Duration was 2563.904053 seconds 2024-03-21T10:58:36.629 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-21T10:58:36.639 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-21T10:58:36.640 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-21T10:58:36.648 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-21T10:58:36.650 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-21T10:58:36.696 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-21T10:58:36.702 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-21T10:58:36.866 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-21T10:58:37.173 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-21T10:58:37.173 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com 2024-03-21T10:58:37.173 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-21T10:58:37.242 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi040.front.sepia.ceph.com 2024-03-21T10:58:37.242 DEBUG:teuthology.orchestra.run.smithi040:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-21T10:58:37.265 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi070.front.sepia.ceph.com 2024-03-21T10:58:37.266 DEBUG:teuthology.orchestra.run.smithi070:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-21T10:58:37.295 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-21T10:58:37.295 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-21T10:58:37.304 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-21T10:58:37.308 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-21T10:58:37.403 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-21T10:58:37.404 DEBUG:teuthology.orchestra.run.smithi031:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-21T10:58:37.409 DEBUG:teuthology.orchestra.run.smithi040:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-21T10:58:37.411 DEBUG:teuthology.orchestra.run.smithi070:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-21T10:58:37.946 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-21T10:58:37.957 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-21T10:58:37.957 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-21T10:58:38.104 DEBUG:teuthology.orchestra.run.smithi040:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-21T10:58:38.136 DEBUG:teuthology.orchestra.run.smithi070:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-21T10:58:38.165 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-21T10:58:38.178 DEBUG:teuthology.orchestra.run.smithi031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-21T10:58:38.186 DEBUG:teuthology.orchestra.run.smithi040:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-21T10:58:38.188 DEBUG:teuthology.orchestra.run.smithi070:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-21T10:58:38.216 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = core 2024-03-21T10:58:38.231 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = core 2024-03-21T10:58:38.412 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = core 2024-03-21T10:58:38.468 DEBUG:teuthology.orchestra.run.smithi031:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-21T10:58:38.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T10:58:38.521 DEBUG:teuthology.orchestra.run.smithi040:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-21T10:58:38.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T10:58:38.537 DEBUG:teuthology.orchestra.run.smithi070:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-21T10:58:38.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T10:58:38.553 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-21T10:58:38.564 INFO:teuthology.task.internal:Transferring archived files... 2024-03-21T10:58:38.566 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612235/remote/smithi031 2024-03-21T10:58:38.566 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-21T10:58:38.689 DEBUG:teuthology.misc:Transferring archived files from smithi040:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612235/remote/smithi040 2024-03-21T10:58:38.690 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-21T10:58:38.739 DEBUG:teuthology.misc:Transferring archived files from smithi070:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612235/remote/smithi070 2024-03-21T10:58:38.739 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-21T10:58:38.780 INFO:teuthology.task.internal:Removing archive directory... 2024-03-21T10:58:38.781 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-21T10:58:38.788 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-21T10:58:38.791 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-21T10:58:38.841 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-21T10:58:38.853 INFO:teuthology.task.internal:Not uploading archives. 2024-03-21T10:58:38.853 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-21T10:58:38.867 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-21T10:58:38.867 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-21T10:58:38.873 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-21T10:58:38.876 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-21T10:58:38.891 INFO:teuthology.orchestra.run.smithi070.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 21 10:58 /home/ubuntu/cephtest 2024-03-21T10:58:38.891 INFO:teuthology.orchestra.run.smithi040.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 21 10:58 /home/ubuntu/cephtest 2024-03-21T10:58:38.924 INFO:teuthology.orchestra.run.smithi031.stdout: 394384 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 21 10:58 /home/ubuntu/cephtest 2024-03-21T10:58:38.928 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-21T10:58:38.941 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-21T10:58:38.967 INFO:teuthology.nuke:Checking targets against current locks 2024-03-21T10:58:38.993 DEBUG:teuthology.nuke:shortname: smithi031 2024-03-21T10:58:38.993 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-21T10:58:39.017 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612235', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 10:06:48.832840', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAHtE/wE/RvnJ0NC6xiejpORRY5Lgaz/LyoHHQv/b5T0xb1IbwCQ6a4X06dQtR/RLGmqaliachMDeain5quaeyo='} 2024-03-21T10:58:39.021 DEBUG:teuthology.nuke:shortname: smithi040 2024-03-21T10:58:39.021 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-21T10:58:39.046 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612235', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 10:06:48.834260', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHNbw+RtJDgDGG+sGMwMbSkwF8iXd5VS40DfjBQeH1mxRs+wPxILdy8NbI9qL1xHjRxQkukrrLQQMh1SiHejrEs='} 2024-03-21T10:58:39.051 DEBUG:teuthology.nuke:shortname: smithi070 2024-03-21T10:58:39.051 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-21T10:58:39.078 INFO:teuthology.orchestra.console.smithi031:Power off 2024-03-21T10:58:39.078 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-21T10:58:39.089 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612235', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 10:06:48.831191', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHY8qxMgdgd6aVrQl/VWAAFd1U+hRQgCkw1lEhtjt/nL77gWzBfZsYNKriRLWt9xAUs+INvo0t5wbKBh0ArgyR0='} 2024-03-21T10:58:39.120 INFO:teuthology.orchestra.console.smithi040:Power off 2024-03-21T10:58:39.120 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-21T10:58:39.132 DEBUG:teuthology.orchestra.console.smithi031:power off output: Chassis Power Control: Down/Off 2024-03-21T10:58:39.133 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T10:58:39.142 DEBUG:teuthology.orchestra.console.smithi040:power off output: Chassis Power Control: Down/Off 2024-03-21T10:58:39.143 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T10:58:39.157 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2024-03-21T10:58:39.169 INFO:teuthology.orchestra.console.smithi070:Power off 2024-03-21T10:58:39.169 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-21T10:58:39.178 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-03-21T10:58:39.190 DEBUG:teuthology.orchestra.console.smithi070:power off output: Chassis Power Control: Down/Off 2024-03-21T10:58:39.191 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T10:58:39.215 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is on 2024-03-21T10:58:43.158 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T10:58:43.179 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T10:58:43.216 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T10:58:43.271 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2024-03-21T10:58:43.289 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-03-21T10:58:43.327 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is on 2024-03-21T10:58:47.274 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T10:58:47.290 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T10:58:47.328 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T10:58:47.386 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2024-03-21T10:58:47.401 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-03-21T10:58:47.439 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is off 2024-03-21T10:58:47.501 INFO:teuthology.orchestra.console.smithi040:Power off completed 2024-03-21T10:58:47.540 INFO:teuthology.orchestra.console.smithi070:Power off completed 2024-03-21T10:58:51.390 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T10:58:51.501 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is off 2024-03-21T10:58:51.602 INFO:teuthology.orchestra.console.smithi031:Power off completed 2024-03-21T10:58:51.707 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} duration: 2563.9040529727936 failure_reason: '"2024-03-21T10:28:06.339620+0000 mon.a (mon.0) 187 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log' flavor: default owner: scheduled_vshankar@teuthology success: false 2024-03-21T10:58:51.707 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-21T10:58:51.788 INFO:teuthology.run:FAIL